2024-05-09T08:11:46.886 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-09T08:11:46.886 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_teuthology@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} --name teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698666 -- /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698666/orig.config.yaml 2024-05-09T08:11:46.946 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T08:11:47.063 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698666 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7698666' kernel: kdb: 1 sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: be85c202969d25af6e130678e7d73a20e9c4ca11 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 2334 sha1: 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc sleep_before_teardown: 0 subset: 359/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: be85c202969d25af6e130678e7d73a20e9c4ca11 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNTfg1XPFLU1mWmcgkhwDM0wOrHeCV+W9MG1su8XIWr2Q2wAFTJqd+HxDveORoN9zcKCDdcUCLEgNqN7gkMtt9M= smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFzzL/fzoy6+IjGnSK6nMnXifOxjdgrhzFTcxoEeNJzoqC9H6NaCj2SciAuWhy3W5OSmewkD4iM+XF4K/1G5ODA= smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNBb3yoSZZG7FSG2fxkSjLq96QZ+4R8JGaBCyJ2rJi046ILYXvIjZkwdDbhS7nUGMpfcB14BSBUuic4kFwdDIhI= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-08_21:24:04 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-09T08:11:47.063 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa; will attempt to use it 2024-05-09T08:11:47.064 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa/tasks 2024-05-09T08:11:47.064 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-09T08:11:47.065 INFO:teuthology.task.internal:Checking packages... 2024-05-09T08:11:47.088 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc' 2024-05-09T08:11:47.089 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-09T08:11:47.089 INFO:teuthology.packaging:ref: None 2024-05-09T08:11:47.089 INFO:teuthology.packaging:tag: None 2024-05-09T08:11:47.089 INFO:teuthology.packaging:branch: squid 2024-05-09T08:11:47.089 INFO:teuthology.packaging:sha1: 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:11:47.089 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-05-09T08:11:47.296 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2529.gbf77c77e 2024-05-09T08:11:47.298 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-09T08:11:47.304 INFO:teuthology.task.internal:no buildpackages task found 2024-05-09T08:11:47.304 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-09T08:11:47.343 INFO:teuthology.task.internal:Saving configuration 2024-05-09T08:11:47.364 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-09T08:11:47.370 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T08:11:47.395 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698666', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 08:06:08.228960', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNTfg1XPFLU1mWmcgkhwDM0wOrHeCV+W9MG1su8XIWr2Q2wAFTJqd+HxDveORoN9zcKCDdcUCLEgNqN7gkMtt9M='} 2024-05-09T08:11:47.421 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698666', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 08:06:08.225995', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFzzL/fzoy6+IjGnSK6nMnXifOxjdgrhzFTcxoEeNJzoqC9H6NaCj2SciAuWhy3W5OSmewkD4iM+XF4K/1G5ODA='} 2024-05-09T08:11:47.445 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698666', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 08:06:08.227496', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNBb3yoSZZG7FSG2fxkSjLq96QZ+4R8JGaBCyJ2rJi046ILYXvIjZkwdDbhS7nUGMpfcB14BSBUuic4kFwdDIhI='} 2024-05-09T08:11:47.446 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-09T08:11:47.451 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-05-09T08:11:47.451 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-05-09T08:11:47.451 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-05-09T08:11:47.451 INFO:teuthology.run_tasks:Running task console_log... 2024-05-09T08:11:47.593 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc4e300b160>, signals=[15]) 2024-05-09T08:11:47.594 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-09T08:11:47.601 INFO:teuthology.task.internal:Opening connections... 2024-05-09T08:11:47.601 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-05-09T08:11:47.604 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:11:47.682 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2024-05-09T08:11:47.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:11:47.757 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2024-05-09T08:11:47.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:11:47.832 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-09T08:11:47.838 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-05-09T08:11:47.856 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-05-09T08:11:47.856 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-05-09T08:11:47.912 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-05-09T08:11:47.913 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-05-09T08:11:47.913 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-05-09T08:11:47.913 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-05-09T08:11:47.913 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-05-09T08:11:47.913 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-05-09T08:11:47.913 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T08:11:47.913 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-05-09T08:11:47.913 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-05-09T08:11:47.913 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T08:11:47.913 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-05-09T08:11:47.913 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T08:11:47.913 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T08:11:47.913 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T08:11:47.914 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-05-09T08:11:47.942 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2024-05-09T08:11:47.960 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2024-05-09T08:11:47.961 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2024-05-09T08:11:48.017 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="CentOS Stream" 2024-05-09T08:11:48.017 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="9" 2024-05-09T08:11:48.017 INFO:teuthology.orchestra.run.smithi110.stdout:ID="centos" 2024-05-09T08:11:48.018 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="rhel fedora" 2024-05-09T08:11:48.018 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="9" 2024-05-09T08:11:48.018 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el9" 2024-05-09T08:11:48.018 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T08:11:48.018 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2024-05-09T08:11:48.018 INFO:teuthology.orchestra.run.smithi110.stdout:LOGO="fedora-logo-icon" 2024-05-09T08:11:48.018 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T08:11:48.018 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://centos.org/" 2024-05-09T08:11:48.018 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T08:11:48.018 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T08:11:48.018 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T08:11:48.019 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2024-05-09T08:11:48.042 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2024-05-09T08:11:48.060 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2024-05-09T08:11:48.061 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2024-05-09T08:11:48.118 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="CentOS Stream" 2024-05-09T08:11:48.118 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="9" 2024-05-09T08:11:48.118 INFO:teuthology.orchestra.run.smithi157.stdout:ID="centos" 2024-05-09T08:11:48.118 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="rhel fedora" 2024-05-09T08:11:48.118 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="9" 2024-05-09T08:11:48.118 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el9" 2024-05-09T08:11:48.118 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T08:11:48.118 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31" 2024-05-09T08:11:48.118 INFO:teuthology.orchestra.run.smithi157.stdout:LOGO="fedora-logo-icon" 2024-05-09T08:11:48.119 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T08:11:48.119 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://centos.org/" 2024-05-09T08:11:48.119 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T08:11:48.119 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T08:11:48.119 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T08:11:48.119 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2024-05-09T08:11:48.150 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-09T08:11:48.160 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-09T08:11:48.234 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-09T08:11:48.234 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-05-09T08:11:48.237 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2024-05-09T08:11:48.239 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2024-05-09T08:11:48.256 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-09T08:11:48.263 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-09T08:11:48.263 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-05-09T08:11:48.293 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2024-05-09T08:11:48.295 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2024-05-09T08:11:48.373 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-09T08:11:48.454 INFO:teuthology.run_tasks:Running task kernel... 2024-05-09T08:11:48.469 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-09T08:11:48.469 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-09T08:11:48.469 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T08:11:48.470 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T08:11:48.470 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T08:11:48.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:11:48.487 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-05-09T08:11:48.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:11:48.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:11:48.489 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-05-09T08:11:48.489 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-05-09T08:11:48.544 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-437.el9.x86_64 2024-05-09T08:11:48.545 INFO:teuthology.task.kernel:Running kernel on smithi110: 5.14.0-437.el9.x86_64 2024-05-09T08:11:48.545 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum install -y kernel 2024-05-09T08:11:48.549 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-437.el9.x86_64 2024-05-09T08:11:48.550 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-437.el9.x86_64 2024-05-09T08:11:48.550 INFO:teuthology.task.kernel:Running kernel on smithi107: 5.14.0-437.el9.x86_64 2024-05-09T08:11:48.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2024-05-09T08:11:48.550 INFO:teuthology.task.kernel:Running kernel on smithi157: 5.14.0-437.el9.x86_64 2024-05-09T08:11:48.550 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum install -y kernel 2024-05-09T08:11:52.004 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.1 MB 00:01 2024-05-09T08:11:52.206 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.1 MB 00:02 2024-05-09T08:11:54.367 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.1 MB 00:04 2024-05-09T08:11:56.246 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 19 MB 00:03 2024-05-09T08:11:59.612 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 3.0 MB/s | 19 MB 00:06 2024-05-09T08:11:59.984 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 4.2 MB/s | 19 MB 00:04 2024-05-09T08:12:00.538 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-05-09T08:12:02.006 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 16 kB 00:00 2024-05-09T08:12:04.531 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.2 MB 00:01 2024-05-09T08:12:05.776 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.2 MB 00:01 2024-05-09T08:12:06.126 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 5.3 MB/s | 21 MB 00:03 2024-05-09T08:12:06.131 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 16 kB 00:00 2024-05-09T08:12:07.552 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 16 kB 00:00 2024-05-09T08:12:07.993 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-05-09T08:12:09.211 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-05-09T08:12:11.240 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-05-09T08:12:13.190 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-05-09T08:12:13.246 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-09T08:12:13.247 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-09T08:12:13.338 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-05-09T08:12:13.350 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-05-09T08:12:13.350 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-05-09T08:12:13.350 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-05-09T08:12:13.350 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-05-09T08:12:13.351 INFO:teuthology.orchestra.run.smithi110.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-09T08:12:13.351 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-05-09T08:12:13.351 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-09T08:12:13.351 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-09T08:12:13.351 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-09T08:12:13.351 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:12:13.351 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-05-09T08:12:13.351 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-05-09T08:12:13.351 INFO:teuthology.orchestra.run.smithi110.stdout:Install 4 Packages 2024-05-09T08:12:13.351 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:12:13.352 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 78 M 2024-05-09T08:12:13.352 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 123 M 2024-05-09T08:12:13.352 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-05-09T08:12:13.858 INFO:teuthology.orchestra.run.smithi110.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 2.0 MB/s | 484 kB 00:00 2024-05-09T08:12:14.486 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-05-09T08:12:15.171 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-09T08:12:15.171 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-09T08:12:15.266 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-05-09T08:12:15.278 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-05-09T08:12:15.278 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-05-09T08:12:15.278 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-05-09T08:12:15.278 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-05-09T08:12:15.278 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-09T08:12:15.278 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-05-09T08:12:15.278 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-09T08:12:15.278 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-09T08:12:15.278 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-09T08:12:15.279 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:12:15.279 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-05-09T08:12:15.279 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-05-09T08:12:15.279 INFO:teuthology.orchestra.run.smithi107.stdout:Install 4 Packages 2024-05-09T08:12:15.279 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:12:15.279 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 78 M 2024-05-09T08:12:15.279 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 123 M 2024-05-09T08:12:15.280 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-05-09T08:12:16.550 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-09T08:12:16.551 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-09T08:12:16.633 INFO:teuthology.orchestra.run.smithi107.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 452 kB/s | 484 kB 00:01 2024-05-09T08:12:16.649 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-05-09T08:12:16.656 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-05-09T08:12:16.656 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-05-09T08:12:16.656 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-05-09T08:12:16.656 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-05-09T08:12:16.656 INFO:teuthology.orchestra.run.smithi157.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-09T08:12:16.656 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-05-09T08:12:16.657 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-09T08:12:16.657 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-09T08:12:16.657 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-09T08:12:16.657 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:12:16.657 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-05-09T08:12:16.657 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-05-09T08:12:16.657 INFO:teuthology.orchestra.run.smithi157.stdout:Install 4 Packages 2024-05-09T08:12:16.657 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:12:16.658 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 78 M 2024-05-09T08:12:16.658 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 123 M 2024-05-09T08:12:16.658 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-05-09T08:12:16.792 INFO:teuthology.orchestra.run.smithi110.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 4.9 MB/s | 15 MB 00:03 2024-05-09T08:12:17.679 INFO:teuthology.orchestra.run.smithi157.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 735 kB/s | 484 kB 00:00 2024-05-09T08:12:17.683 INFO:teuthology.orchestra.run.smithi107.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 7.3 MB/s | 15 MB 00:02 2024-05-09T08:12:18.641 INFO:teuthology.orchestra.run.smithi107.stdout:(3/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 11 MB/s | 34 MB 00:03 2024-05-09T08:12:19.491 INFO:teuthology.orchestra.run.smithi107.stdout:(4/4): kernel-modules-core-5.14.0-444.el9.x86_6 9.8 MB/s | 28 MB 00:02 2024-05-09T08:12:19.492 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:12:19.492 INFO:teuthology.orchestra.run.smithi107.stdout:Total 18 MB/s | 78 MB 00:04 2024-05-09T08:12:20.170 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-05-09T08:12:20.503 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-05-09T08:12:20.503 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-05-09T08:12:20.892 INFO:teuthology.orchestra.run.smithi110.stdout:(3/4): kernel-modules-core-5.14.0-444.el9.x86_6 4.0 MB/s | 28 MB 00:07 2024-05-09T08:12:21.435 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-05-09T08:12:21.435 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-05-09T08:12:22.050 INFO:teuthology.orchestra.run.smithi110.stdout:(4/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 4.0 MB/s | 34 MB 00:08 2024-05-09T08:12:22.051 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:12:22.051 INFO:teuthology.orchestra.run.smithi110.stdout:Total 8.9 MB/s | 78 MB 00:08 2024-05-09T08:12:22.731 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-05-09T08:12:23.078 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-05-09T08:12:23.078 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-05-09T08:12:23.787 INFO:teuthology.orchestra.run.smithi157.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 2.3 MB/s | 15 MB 00:06 2024-05-09T08:12:23.861 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-05-09T08:12:24.021 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-05-09T08:12:24.022 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-05-09T08:12:24.466 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-09T08:12:24.542 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:12:26.414 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-05-09T08:12:27.120 INFO:teuthology.orchestra.run.smithi157.stdout:(3/4): kernel-modules-core-5.14.0-444.el9.x86_6 3.0 MB/s | 28 MB 00:09 2024-05-09T08:12:27.132 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-09T08:12:27.189 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:12:27.209 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:12:27.320 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:12:29.876 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:12:30.023 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:12:30.911 INFO:teuthology.orchestra.run.smithi157.stdout:(4/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 2.4 MB/s | 34 MB 00:13 2024-05-09T08:12:30.911 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:12:30.911 INFO:teuthology.orchestra.run.smithi157.stdout:Total 5.5 MB/s | 78 MB 00:14 2024-05-09T08:12:31.594 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-05-09T08:12:31.926 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-05-09T08:12:31.926 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-05-09T08:12:32.032 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:12:32.247 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:12:32.821 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-05-09T08:12:32.821 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-05-09T08:12:34.709 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:12:34.750 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:12:35.216 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-05-09T08:12:35.853 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-09T08:12:35.936 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:12:37.002 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:12:38.616 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:12:38.766 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:12:39.482 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:12:43.465 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:12:43.510 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:12:48.269 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:13:06.447 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:13:06.482 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:13:06.813 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:13:06.813 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-09T08:13:06.813 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:13:06.813 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:13:07.585 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:13:07.585 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:13:07.585 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-05-09T08:13:07.585 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-09T08:13:07.585 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-09T08:13:07.585 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-09T08:13:07.585 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-09T08:13:07.586 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:13:07.586 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-05-09T08:13:07.946 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T08:13:08.022 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T08:13:08.023 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T08:13:08.023 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-09T08:13:08.023 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-09T08:13:08.023 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-09T08:13:08.023 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-09T08:13:08.023 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2024-05-09T08:13:08.023 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T08:13:08.071 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T08:13:08.072 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T08:13:08.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2024-05-09T08:13:08.112 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2024-05-09T08:13:08.112 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2024-05-09T08:13:08.112 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.06 2024-05-09T08:13:08.112 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 68.el9 2024-05-09T08:13:08.112 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2024-05-09T08:13:08.113 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T08:13:08.113 INFO:teuthology.orchestra.run.smithi107.stdout:Group : Unspecified 2024-05-09T08:13:08.113 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 8264254 2024-05-09T08:13:08.113 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2024-05-09T08:13:08.113 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T08:13:08.113 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T08:13:08.113 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T08:13:08.113 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T08:13:08.113 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : builder@centos.org 2024-05-09T08:13:08.113 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2024-05-09T08:13:08.113 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T08:13:08.113 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2024-05-09T08:13:08.113 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2024-05-09T08:13:08.113 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:13:08.113 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T08:13:08.114 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T08:13:08.114 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T08:13:08.114 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2024-05-09T08:13:08.114 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:13:08.114 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T08:13:08.115 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T08:13:08.115 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 5.14.0-444.el9.x86_64 2024-05-09T08:13:08.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T08:13:08.756 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:13:08.769 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2024-05-09T08:13:08.793 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:13:09.176 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:13:09.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-09T08:13:09.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:13:09.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:13:10.008 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:13:10.008 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:13:10.008 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-05-09T08:13:10.008 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-09T08:13:10.008 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-09T08:13:10.008 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-09T08:13:10.008 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-09T08:13:10.009 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:13:10.009 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-05-09T08:13:10.140 INFO:teuthology.orchestra.run.smithi107.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T08:13:10.174 INFO:teuthology.orchestra.run.smithi107.stderr:done 2024-05-09T08:13:10.176 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-05-09T08:13:10.194 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.ASyF6e30pl 2024-05-09T08:13:10.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ASyF6e30pl 2024-05-09T08:13:10.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.ASyF6e30pl 2024-05-09T08:13:10.360 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T08:13:10.396 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.ASyF6e30pl is 6KB 2024-05-09T08:13:10.432 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T08:13:10.433 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T08:13:10.433 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-09T08:13:10.433 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-09T08:13:10.433 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-09T08:13:10.433 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-05-09T08:13:10.433 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2024-05-09T08:13:10.433 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T08:13:10.444 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.ASyF6e30pl 2024-05-09T08:13:10.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T08:13:10.482 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T08:13:10.483 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T08:13:10.483 DEBUG:teuthology.orchestra.run.smithi110:> sudo rpm -qi grub2-tools 2024-05-09T08:13:10.526 INFO:teuthology.orchestra.run.smithi110.stdout:Name : grub2-tools 2024-05-09T08:13:10.526 INFO:teuthology.orchestra.run.smithi110.stdout:Epoch : 1 2024-05-09T08:13:10.527 INFO:teuthology.orchestra.run.smithi110.stdout:Version : 2.06 2024-05-09T08:13:10.527 INFO:teuthology.orchestra.run.smithi110.stdout:Release : 68.el9 2024-05-09T08:13:10.527 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: x86_64 2024-05-09T08:13:10.527 INFO:teuthology.orchestra.run.smithi110.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T08:13:10.527 INFO:teuthology.orchestra.run.smithi110.stdout:Group : Unspecified 2024-05-09T08:13:10.527 INFO:teuthology.orchestra.run.smithi110.stdout:Size : 8264254 2024-05-09T08:13:10.527 INFO:teuthology.orchestra.run.smithi110.stdout:License : GPLv3+ 2024-05-09T08:13:10.527 INFO:teuthology.orchestra.run.smithi110.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T08:13:10.527 INFO:teuthology.orchestra.run.smithi110.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T08:13:10.527 INFO:teuthology.orchestra.run.smithi110.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T08:13:10.527 INFO:teuthology.orchestra.run.smithi110.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T08:13:10.527 INFO:teuthology.orchestra.run.smithi110.stdout:Packager : builder@centos.org 2024-05-09T08:13:10.527 INFO:teuthology.orchestra.run.smithi110.stdout:Vendor : CentOS 2024-05-09T08:13:10.527 INFO:teuthology.orchestra.run.smithi110.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T08:13:10.528 INFO:teuthology.orchestra.run.smithi110.stdout:Summary : Support tools for GRUB. 2024-05-09T08:13:10.528 INFO:teuthology.orchestra.run.smithi110.stdout:Description : 2024-05-09T08:13:10.528 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:13:10.528 INFO:teuthology.orchestra.run.smithi110.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T08:13:10.528 INFO:teuthology.orchestra.run.smithi110.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T08:13:10.528 INFO:teuthology.orchestra.run.smithi110.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T08:13:10.528 INFO:teuthology.orchestra.run.smithi110.stdout:hardware devices. 2024-05-09T08:13:10.528 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:13:10.528 INFO:teuthology.orchestra.run.smithi110.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T08:13:10.530 INFO:teuthology.orchestra.run.smithi107.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T08:13:10.530 INFO:teuthology.orchestra.run.smithi107.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T08:13:10.530 INFO:teuthology.orchestra.run.smithi107.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T08:13:10.530 INFO:teuthology.orchestra.run.smithi107.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-09T08:13:10.530 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T08:13:10.530 INFO:teuthology.task.kernel:Updating grub on smithi110 to boot 5.14.0-444.el9.x86_64 2024-05-09T08:13:10.530 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T08:13:10.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-09T08:13:10.681 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2024-05-09T08:13:11.136 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2024-05-09T08:13:12.470 INFO:teuthology.orchestra.run.smithi110.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T08:13:12.502 INFO:teuthology.orchestra.run.smithi110.stderr:done 2024-05-09T08:13:12.505 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-05-09T08:13:12.519 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.o2A0ONoGK9 2024-05-09T08:13:12.520 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.o2A0ONoGK9 2024-05-09T08:13:12.597 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.o2A0ONoGK9 2024-05-09T08:13:12.733 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.o2A0ONoGK9 is 6KB 2024-05-09T08:13:12.786 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.o2A0ONoGK9 2024-05-09T08:13:12.800 DEBUG:teuthology.orchestra.run.smithi110:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T08:13:12.870 INFO:teuthology.orchestra.run.smithi110.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T08:13:12.871 INFO:teuthology.orchestra.run.smithi110.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T08:13:12.871 INFO:teuthology.orchestra.run.smithi110.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T08:13:12.871 INFO:teuthology.orchestra.run.smithi110.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-09T08:13:12.872 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-09T08:13:13.023 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2024-05-09T08:13:17.484 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:13:17.522 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:13:17.844 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:13:17.845 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-09T08:13:17.845 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:13:17.845 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:13:18.632 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:13:18.632 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:13:18.632 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-05-09T08:13:18.632 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-09T08:13:18.632 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-09T08:13:18.632 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-09T08:13:18.632 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-09T08:13:18.632 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:13:18.632 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-05-09T08:13:18.997 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T08:13:19.086 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T08:13:19.087 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T08:13:19.088 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-09T08:13:19.088 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-09T08:13:19.088 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-09T08:13:19.088 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-05-09T08:13:19.088 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2024-05-09T08:13:19.088 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T08:13:19.138 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T08:13:19.139 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T08:13:19.139 DEBUG:teuthology.orchestra.run.smithi157:> sudo rpm -qi grub2-tools 2024-05-09T08:13:19.176 INFO:teuthology.orchestra.run.smithi157.stdout:Name : grub2-tools 2024-05-09T08:13:19.176 INFO:teuthology.orchestra.run.smithi157.stdout:Epoch : 1 2024-05-09T08:13:19.176 INFO:teuthology.orchestra.run.smithi157.stdout:Version : 2.06 2024-05-09T08:13:19.176 INFO:teuthology.orchestra.run.smithi157.stdout:Release : 68.el9 2024-05-09T08:13:19.176 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: x86_64 2024-05-09T08:13:19.176 INFO:teuthology.orchestra.run.smithi157.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T08:13:19.176 INFO:teuthology.orchestra.run.smithi157.stdout:Group : Unspecified 2024-05-09T08:13:19.176 INFO:teuthology.orchestra.run.smithi157.stdout:Size : 8264254 2024-05-09T08:13:19.176 INFO:teuthology.orchestra.run.smithi157.stdout:License : GPLv3+ 2024-05-09T08:13:19.176 INFO:teuthology.orchestra.run.smithi157.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T08:13:19.177 INFO:teuthology.orchestra.run.smithi157.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T08:13:19.177 INFO:teuthology.orchestra.run.smithi157.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T08:13:19.177 INFO:teuthology.orchestra.run.smithi157.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T08:13:19.177 INFO:teuthology.orchestra.run.smithi157.stdout:Packager : builder@centos.org 2024-05-09T08:13:19.177 INFO:teuthology.orchestra.run.smithi157.stdout:Vendor : CentOS 2024-05-09T08:13:19.177 INFO:teuthology.orchestra.run.smithi157.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T08:13:19.177 INFO:teuthology.orchestra.run.smithi157.stdout:Summary : Support tools for GRUB. 2024-05-09T08:13:19.177 INFO:teuthology.orchestra.run.smithi157.stdout:Description : 2024-05-09T08:13:19.177 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:13:19.177 INFO:teuthology.orchestra.run.smithi157.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T08:13:19.177 INFO:teuthology.orchestra.run.smithi157.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T08:13:19.177 INFO:teuthology.orchestra.run.smithi157.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T08:13:19.177 INFO:teuthology.orchestra.run.smithi157.stdout:hardware devices. 2024-05-09T08:13:19.177 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:13:19.178 INFO:teuthology.orchestra.run.smithi157.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T08:13:19.178 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T08:13:19.179 INFO:teuthology.task.kernel:Updating grub on smithi157 to boot 5.14.0-444.el9.x86_64 2024-05-09T08:13:19.179 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T08:13:19.808 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2024-05-09T08:13:21.243 INFO:teuthology.orchestra.run.smithi157.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T08:13:21.262 INFO:teuthology.orchestra.run.smithi157.stderr:done 2024-05-09T08:13:21.264 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-05-09T08:13:21.279 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.xd4NyL8hm3 2024-05-09T08:13:21.279 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xd4NyL8hm3 2024-05-09T08:13:21.344 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.xd4NyL8hm3 2024-05-09T08:13:21.477 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.xd4NyL8hm3 is 6KB 2024-05-09T08:13:21.526 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.xd4NyL8hm3 2024-05-09T08:13:21.540 DEBUG:teuthology.orchestra.run.smithi157:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T08:13:21.602 INFO:teuthology.orchestra.run.smithi157.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T08:13:21.602 INFO:teuthology.orchestra.run.smithi157.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T08:13:21.602 INFO:teuthology.orchestra.run.smithi157.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T08:13:21.603 INFO:teuthology.orchestra.run.smithi157.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-09T08:13:21.604 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-09T08:13:21.755 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2024-05-09T08:13:40.703 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-09T08:13:40.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-05-09T08:13:40.705 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:13:43.026 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-09T08:13:43.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-05-09T08:13:43.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:13:51.766 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-05-09T08:13:51.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-05-09T08:13:51.767 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:13:59.199 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-05-09T08:14:01.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-05-09T08:14:08.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-05-09T08:14:08.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:14:10.210 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-05-09T08:14:10.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-05-09T08:14:10.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:14:11.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-05-09T08:14:13.566 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-05-09T08:14:19.213 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-05-09T08:14:19.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:14:22.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-05-09T08:14:23.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-05-09T08:14:23.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:14:25.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-05-09T08:14:25.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:14:34.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-05-09T08:14:34.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:14:37.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-05-09T08:14:43.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-05-09T08:14:52.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-05-09T08:14:52.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:14:59.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-05-09T08:14:59.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:14:59.958 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-05-09T08:15:00.317 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-05-09T08:15:00.317 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-444.el9.x86_64"... 2024-05-09T08:15:00.318 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-05-09T08:15:00.333 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-444.el9.x86_64 2024-05-09T08:15:00.333 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-09T08:15:00.333 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T08:15:00.333 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T08:15:01.334 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-09T08:15:01.335 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-09T08:15:01.409 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2024-05-09T08:15:01.444 DEBUG:teuthology.parallel:result is None 2024-05-09T08:15:10.878 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-05-09T08:15:23.267 DEBUG:teuthology.orchestra.remote:timed out 2024-05-09T08:15:28.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-05-09T08:15:28.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:15:29.251 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-05-09T08:15:29.596 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-05-09T08:15:29.597 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-444.el9.x86_64"... 2024-05-09T08:15:29.597 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-05-09T08:15:29.611 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-444.el9.x86_64 2024-05-09T08:15:29.612 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-09T08:15:29.612 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T08:15:29.612 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T08:15:30.613 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-05-09T08:15:30.613 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-09T08:15:30.676 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2024-05-09T08:15:30.703 DEBUG:teuthology.parallel:result is None 2024-05-09T08:15:38.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-05-09T08:15:38.269 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:15:38.643 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-05-09T08:15:39.011 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-05-09T08:15:39.011 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-444.el9.x86_64"... 2024-05-09T08:15:39.011 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-05-09T08:15:39.027 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-444.el9.x86_64 2024-05-09T08:15:39.028 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-09T08:15:39.028 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T08:15:39.028 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T08:15:40.028 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-09T08:15:40.029 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-09T08:15:40.102 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2024-05-09T08:15:40.329 DEBUG:teuthology.parallel:result is None 2024-05-09T08:15:40.329 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-09T08:15:40.337 INFO:teuthology.task.internal:Creating test directory... 2024-05-09T08:15:40.337 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T08:15:40.340 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T08:15:40.343 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T08:15:40.361 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-09T08:15:40.368 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-09T08:15:40.411 INFO:teuthology.task.internal:Creating archive directory... 2024-05-09T08:15:40.411 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T08:15:40.414 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T08:15:40.417 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T08:15:40.464 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-09T08:15:40.471 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-09T08:15:40.472 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T08:15:40.488 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T08:15:40.499 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T08:15:40.520 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:15:40.530 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:15:40.531 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:15:40.536 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:15:40.541 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:15:40.548 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T08:15:40.549 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-09T08:15:40.556 INFO:teuthology.task.internal:Configuring sudo... 2024-05-09T08:15:40.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T08:15:40.576 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T08:15:40.585 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T08:15:40.619 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-09T08:15:40.628 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-09T08:15:40.628 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T08:15:40.645 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T08:15:40.653 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T08:15:40.676 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:15:40.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:15:40.786 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:15:40.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:15:40.911 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:15:40.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T08:15:40.978 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:15:41.018 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:15:41.073 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:15:41.158 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:15:41.199 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-09T08:15:41.199 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T08:15:41.266 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:15:41.312 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T08:15:41.358 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:15:41.444 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T08:15:41.488 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-05-09T08:15:41.488 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T08:15:41.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-05-09T08:15:41.559 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2024-05-09T08:15:41.561 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2024-05-09T08:15:41.598 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T08:15:41.607 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T08:15:41.640 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T08:15:42.138 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-09T08:15:42.146 INFO:teuthology.task.internal:Starting timer... 2024-05-09T08:15:42.146 INFO:teuthology.run_tasks:Running task pcp... 2024-05-09T08:15:42.181 INFO:teuthology.run_tasks:Running task selinux... 2024-05-09T08:15:42.265 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-09T08:15:42.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-05-09T08:15:42.308 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-05-09T08:15:42.311 DEBUG:teuthology.orchestra.run.smithi110:> sudo service auditd rotate 2024-05-09T08:15:42.369 INFO:teuthology.orchestra.run.smithi110.stdout:Rotating logs: 2024-05-09T08:15:42.371 DEBUG:teuthology.orchestra.run.smithi157:> sudo service auditd rotate 2024-05-09T08:15:42.426 INFO:teuthology.orchestra.run.smithi157.stdout:Rotating logs: 2024-05-09T08:15:42.428 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-09T08:15:42.428 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-05-09T08:15:42.462 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-05-09T08:15:42.462 DEBUG:teuthology.orchestra.run.smithi110:> /usr/sbin/getenforce 2024-05-09T08:15:42.551 INFO:teuthology.orchestra.run.smithi110.stdout:Permissive 2024-05-09T08:15:42.551 DEBUG:teuthology.orchestra.run.smithi157:> /usr/sbin/getenforce 2024-05-09T08:15:42.579 INFO:teuthology.orchestra.run.smithi157.stdout:Permissive 2024-05-09T08:15:42.579 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi110.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi157.front.sepia.ceph.com': 'permissive'} 2024-05-09T08:15:42.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T08:15:42.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:15:42.609 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T08:15:42.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:15:42.638 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T08:15:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:15:42.669 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-09T08:15:42.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-05-09T08:15:42.696 DEBUG:teuthology.orchestra.run.smithi110:> sudo /usr/sbin/setenforce permissive 2024-05-09T08:15:42.724 DEBUG:teuthology.orchestra.run.smithi157:> sudo /usr/sbin/setenforce permissive 2024-05-09T08:15:42.752 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-09T08:15:42.763 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-09T08:15:42.783 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-09T08:15:42.784 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com,smithi110.front.sepia.ceph.com,smithi157.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-09T08:19:53.751 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi110.front.sepia.ceph.com'), Remote(name='ubuntu@smithi157.front.sepia.ceph.com')] 2024-05-09T08:19:53.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-05-09T08:19:53.754 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:19:53.833 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-05-09T08:19:53.909 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-05-09T08:19:53.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-05-09T08:19:53.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:19:53.991 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-05-09T08:19:54.069 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-05-09T08:19:54.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-05-09T08:19:54.070 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:19:54.148 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-05-09T08:19:54.226 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-05-09T08:19:54.226 INFO:teuthology.run_tasks:Running task clock... 2024-05-09T08:19:54.237 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-09T08:19:54.237 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T08:19:54.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T08:19:54.240 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T08:19:54.240 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T08:19:54.242 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T08:19:54.243 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T08:19:54.270 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T08:19:54.270 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T08:19:54.284 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T08:19:54.285 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T08:19:54.298 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T08:19:54.309 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: ntpd: command not found 2024-05-09T08:19:54.311 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2024-05-09T08:19:54.313 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T08:19:54.320 INFO:teuthology.orchestra.run.smithi110.stdout:506 Cannot talk to daemon 2024-05-09T08:19:54.322 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2024-05-09T08:19:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T08:19:54.337 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T08:19:54.338 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: ntpd: command not found 2024-05-09T08:19:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T08:19:54.349 INFO:teuthology.orchestra.run.smithi157.stdout:506 Cannot talk to daemon 2024-05-09T08:19:54.351 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T08:19:54.364 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T08:19:54.377 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T08:19:54.404 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-05-09T08:19:54.407 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T08:19:54.407 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-05-09T08:19:54.407 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:19:54.407 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:19:54.407 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:19:54.407 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:19:54.411 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-05-09T08:19:54.415 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T08:19:54.415 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-05-09T08:19:54.415 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:19:54.415 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:19:54.415 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:19:54.415 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:19:54.432 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-05-09T08:19:54.437 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T08:19:54.437 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-05-09T08:19:54.438 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:19:54.438 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:19:54.438 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:19:54.438 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T08:19:54.438 INFO:teuthology.run_tasks:Running task install... 2024-05-09T08:19:54.447 DEBUG:teuthology.task.install:project ceph 2024-05-09T08:19:54.447 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc'}} 2024-05-09T08:19:54.447 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc'} 2024-05-09T08:19:54.447 INFO:teuthology.task.install:Using flavor: default 2024-05-09T08:19:54.453 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-09T08:19:54.454 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-09T08:19:54.454 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'tag': None, 'wait_for_package': False} 2024-05-09T08:19:54.454 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:19:54.457 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'tag': None, 'wait_for_package': False} 2024-05-09T08:19:54.457 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:19:54.458 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'tag': None, 'wait_for_package': False} 2024-05-09T08:19:54.458 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:19:54.609 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/ 2024-05-09T08:19:54.609 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2515.g96805dcf 2024-05-09T08:19:54.682 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/ 2024-05-09T08:19:54.682 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2515.g96805dcf 2024-05-09T08:19:54.686 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/ 2024-05-09T08:19:54.686 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2515.g96805dcf 2024-05-09T08:19:54.718 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T08:19:54.718 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-05-09T08:19:54.718 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T08:19:54.745 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T08:19:54.745 DEBUG:teuthology.orchestra.run.smithi157:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T08:19:54.811 DEBUG:teuthology.orchestra.run.smithi157:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T08:19:54.846 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T08:19:54.846 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:19:54.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T08:19:54.854 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T08:19:54.854 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-09T08:19:54.854 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T08:19:54.872 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T08:19:54.872 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T08:19:54.883 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T08:19:54.883 DEBUG:teuthology.orchestra.run.smithi110:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T08:19:54.887 DEBUG:teuthology.orchestra.run.smithi157:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T08:19:54.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T08:19:54.951 DEBUG:teuthology.orchestra.run.smithi110:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T08:19:54.952 INFO:teuthology.orchestra.run.smithi157.stdout:check_obsoletes = 1 2024-05-09T08:19:54.953 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum clean all 2024-05-09T08:19:55.014 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T08:19:55.027 DEBUG:teuthology.orchestra.run.smithi110:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T08:19:55.079 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2024-05-09T08:19:55.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2024-05-09T08:19:55.092 INFO:teuthology.orchestra.run.smithi110.stdout:check_obsoletes = 1 2024-05-09T08:19:55.094 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum clean all 2024-05-09T08:19:55.188 INFO:teuthology.orchestra.run.smithi157.stdout:54 files removed 2024-05-09T08:19:55.215 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T08:19:55.316 INFO:teuthology.orchestra.run.smithi107.stdout:54 files removed 2024-05-09T08:19:55.331 INFO:teuthology.orchestra.run.smithi110.stdout:54 files removed 2024-05-09T08:19:55.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T08:19:55.358 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T08:19:55.954 INFO:teuthology.orchestra.run.smithi157.stdout:ceph packages for x86_64 180 kB/s | 83 kB 00:00 2024-05-09T08:19:55.988 INFO:teuthology.orchestra.run.smithi110.stdout:ceph packages for x86_64 234 kB/s | 83 kB 00:00 2024-05-09T08:19:56.093 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 175 kB/s | 83 kB 00:00 2024-05-09T08:19:56.313 INFO:teuthology.orchestra.run.smithi157.stdout:ceph noarch packages 38 kB/s | 11 kB 00:00 2024-05-09T08:19:56.439 INFO:teuthology.orchestra.run.smithi110.stdout:ceph noarch packages 29 kB/s | 11 kB 00:00 2024-05-09T08:19:56.460 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 37 kB/s | 11 kB 00:00 2024-05-09T08:19:56.639 INFO:teuthology.orchestra.run.smithi157.stdout:ceph source packages 6.7 kB/s | 1.9 kB 00:00 2024-05-09T08:19:56.773 INFO:teuthology.orchestra.run.smithi110.stdout:ceph source packages 6.5 kB/s | 1.9 kB 00:00 2024-05-09T08:19:56.803 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 6.3 kB/s | 1.9 kB 00:00 2024-05-09T08:19:57.473 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.1 MB 00:00 2024-05-09T08:19:57.547 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 9.3 MB/s | 8.1 MB 00:00 2024-05-09T08:19:58.394 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.1 MB 00:01 2024-05-09T08:19:59.258 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-05-09T08:20:00.228 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-05-09T08:20:03.250 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-05-09T08:20:05.052 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-05-09T08:20:06.306 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-05-09T08:20:07.037 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 1.9 MB/s | 6.2 MB 00:03 2024-05-09T08:20:08.621 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 16 kB 00:00 2024-05-09T08:20:10.300 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-05-09T08:20:11.420 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-05-09T08:20:13.756 INFO:teuthology.orchestra.run.smithi110.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:20:13.756 INFO:teuthology.orchestra.run.smithi110.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:20:13.767 INFO:teuthology.orchestra.run.smithi110.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T08:20:13.768 INFO:teuthology.orchestra.run.smithi110.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T08:20:13.769 INFO:teuthology.orchestra.run.smithi110.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T08:20:13.769 INFO:teuthology.orchestra.run.smithi110.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T08:20:13.771 INFO:teuthology.orchestra.run.smithi110.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T08:20:13.774 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:15 2024-05-09T08:20:13.775 INFO:teuthology.orchestra.run.smithi110.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T08:20:13.775 INFO:teuthology.orchestra.run.smithi110.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T08:20:13.835 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-05-09T08:20:13.854 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-05-09T08:20:13.854 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-05-09T08:20:13.855 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-05-09T08:20:13.855 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-05-09T08:20:13.855 INFO:teuthology.orchestra.run.smithi110.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T08:20:13.855 INFO:teuthology.orchestra.run.smithi110.stdout: ceph x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 6.6 k 2024-05-09T08:20:13.855 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.4 M 2024-05-09T08:20:13.855 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 889 k 2024-05-09T08:20:13.855 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 145 k 2024-05-09T08:20:13.855 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 1.5 M 2024-05-09T08:20:13.855 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 142 k 2024-05-09T08:20:13.855 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 3.5 M 2024-05-09T08:20:13.855 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 7.4 M 2024-05-09T08:20:13.855 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 49 k 2024-05-09T08:20:13.856 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 10 M 2024-05-09T08:20:13.856 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 48 M 2024-05-09T08:20:13.856 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 264 k 2024-05-09T08:20:13.856 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 763 k 2024-05-09T08:20:13.856 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 222 k 2024-05-09T08:20:13.856 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-shell x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 27 k 2024-05-09T08:20:13.856 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-top noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 18 k 2024-05-09T08:20:13.856 INFO:teuthology.orchestra.run.smithi110.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T08:20:13.856 INFO:teuthology.orchestra.run.smithi110.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T08:20:13.856 INFO:teuthology.orchestra.run.smithi110.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T08:20:13.857 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 32 k 2024-05-09T08:20:13.857 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 746 k 2024-05-09T08:20:13.857 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 127 k 2024-05-09T08:20:13.857 INFO:teuthology.orchestra.run.smithi110.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T08:20:13.857 INFO:teuthology.orchestra.run.smithi110.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T08:20:13.857 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T08:20:13.857 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T08:20:13.857 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T08:20:13.857 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T08:20:13.857 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 162 k 2024-05-09T08:20:13.857 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 321 k 2024-05-09T08:20:13.858 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 298 k 2024-05-09T08:20:13.858 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 99 k 2024-05-09T08:20:13.858 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 85 k 2024-05-09T08:20:13.858 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:20:13.858 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 171 k 2024-05-09T08:20:13.858 INFO:teuthology.orchestra.run.smithi110.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T08:20:13.858 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrading: 2024-05-09T08:20:13.858 INFO:teuthology.orchestra.run.smithi110.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T08:20:13.858 INFO:teuthology.orchestra.run.smithi110.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T08:20:13.858 INFO:teuthology.orchestra.run.smithi110.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T08:20:13.859 INFO:teuthology.orchestra.run.smithi110.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T08:20:13.859 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.4 M 2024-05-09T08:20:13.859 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:20:13.859 INFO:teuthology.orchestra.run.smithi110.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T08:20:13.859 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T08:20:13.859 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-05-09T08:20:13.859 INFO:teuthology.orchestra.run.smithi110.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T08:20:13.859 INFO:teuthology.orchestra.run.smithi110.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T08:20:13.859 INFO:teuthology.orchestra.run.smithi110.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T08:20:13.859 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 21 M 2024-05-09T08:20:13.860 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 32 k 2024-05-09T08:20:13.860 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 2.4 M 2024-05-09T08:20:13.860 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 249 k 2024-05-09T08:20:13.860 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 4.7 M 2024-05-09T08:20:13.860 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 17 M 2024-05-09T08:20:13.860 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 16 k 2024-05-09T08:20:13.860 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 25 k 2024-05-09T08:20:13.860 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T08:20:13.860 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T08:20:13.860 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T08:20:13.860 INFO:teuthology.orchestra.run.smithi110.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T08:20:13.861 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T08:20:13.861 INFO:teuthology.orchestra.run.smithi110.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T08:20:13.861 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T08:20:13.861 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T08:20:13.861 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 163 k 2024-05-09T08:20:13.861 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T08:20:13.861 INFO:teuthology.orchestra.run.smithi110.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T08:20:13.861 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T08:20:13.861 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T08:20:13.861 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T08:20:13.861 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 500 k 2024-05-09T08:20:13.861 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T08:20:13.862 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.3 M 2024-05-09T08:20:13.862 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T08:20:13.862 INFO:teuthology.orchestra.run.smithi110.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T08:20:13.862 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T08:20:13.862 INFO:teuthology.orchestra.run.smithi110.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T08:20:13.862 INFO:teuthology.orchestra.run.smithi110.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T08:20:13.862 INFO:teuthology.orchestra.run.smithi110.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T08:20:13.862 INFO:teuthology.orchestra.run.smithi110.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T08:20:13.862 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T08:20:13.862 INFO:teuthology.orchestra.run.smithi110.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T08:20:13.862 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T08:20:13.862 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T08:20:13.862 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T08:20:13.863 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T08:20:13.863 INFO:teuthology.orchestra.run.smithi110.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T08:20:13.863 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T08:20:13.863 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T08:20:13.863 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T08:20:13.863 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 45 k 2024-05-09T08:20:13.863 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 135 k 2024-05-09T08:20:13.863 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T08:20:13.863 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T08:20:13.863 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T08:20:13.863 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T08:20:13.863 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T08:20:13.863 INFO:teuthology.orchestra.run.smithi110.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T08:20:13.864 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T08:20:13.864 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T08:20:13.864 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T08:20:13.864 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T08:20:13.864 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T08:20:13.864 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T08:20:13.864 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T08:20:13.864 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T08:20:13.864 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T08:20:13.864 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T08:20:13.864 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T08:20:13.864 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T08:20:13.864 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T08:20:13.865 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T08:20:13.865 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T08:20:13.865 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T08:20:13.865 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T08:20:13.865 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T08:20:13.865 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T08:20:13.865 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T08:20:13.865 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T08:20:13.865 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T08:20:13.865 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T08:20:13.865 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T08:20:13.865 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T08:20:13.865 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T08:20:13.865 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T08:20:13.866 INFO:teuthology.orchestra.run.smithi110.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T08:20:13.866 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T08:20:13.866 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T08:20:13.866 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T08:20:13.866 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T08:20:13.866 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T08:20:13.866 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T08:20:13.866 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T08:20:13.866 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T08:20:13.866 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T08:20:13.866 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T08:20:13.866 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T08:20:13.866 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T08:20:13.867 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T08:20:13.867 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T08:20:13.867 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T08:20:13.867 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T08:20:13.867 INFO:teuthology.orchestra.run.smithi110.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T08:20:13.867 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T08:20:13.867 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T08:20:13.867 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T08:20:13.867 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T08:20:13.867 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T08:20:13.867 INFO:teuthology.orchestra.run.smithi110.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T08:20:13.867 INFO:teuthology.orchestra.run.smithi110.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T08:20:13.867 INFO:teuthology.orchestra.run.smithi110.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T08:20:13.868 INFO:teuthology.orchestra.run.smithi110.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T08:20:13.868 INFO:teuthology.orchestra.run.smithi110.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T08:20:13.868 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T08:20:13.868 INFO:teuthology.orchestra.run.smithi110.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T08:20:13.868 INFO:teuthology.orchestra.run.smithi110.stdout:Installing weak dependencies: 2024-05-09T08:20:13.868 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T08:20:13.868 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:20:13.868 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-05-09T08:20:13.868 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-05-09T08:20:13.868 INFO:teuthology.orchestra.run.smithi110.stdout:Install 148 Packages 2024-05-09T08:20:13.868 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrade 8 Packages 2024-05-09T08:20:13.868 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:20:13.869 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 225 M 2024-05-09T08:20:13.869 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-05-09T08:20:14.747 INFO:teuthology.orchestra.run.smithi110.stdout:(1/156): ceph-19.0.0-2515.g96805dcf.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-05-09T08:20:15.022 INFO:teuthology.orchestra.run.smithi110.stdout:(2/156): ceph-fuse-19.0.0-2515.g96805dcf.el9.x8 3.2 MB/s | 889 kB 00:00 2024-05-09T08:20:15.089 INFO:teuthology.orchestra.run.smithi110.stdout:(3/156): ceph-immutable-object-cache-19.0.0-251 2.1 MB/s | 145 kB 00:00 2024-05-09T08:20:15.423 INFO:teuthology.orchestra.run.smithi110.stdout:(4/156): ceph-mds-19.0.0-2515.g96805dcf.el9.x86 7.1 MB/s | 2.4 MB 00:00 2024-05-09T08:20:15.438 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-05-09T08:20:15.581 INFO:teuthology.orchestra.run.smithi110.stdout:(5/156): ceph-mgr-19.0.0-2515.g96805dcf.el9.x86 9.4 MB/s | 1.5 MB 00:00 2024-05-09T08:20:15.932 INFO:teuthology.orchestra.run.smithi110.stdout:(6/156): ceph-mon-19.0.0-2515.g96805dcf.el9.x86 13 MB/s | 4.7 MB 00:00 2024-05-09T08:20:16.132 INFO:teuthology.orchestra.run.smithi110.stdout:(7/156): ceph-base-19.0.0-2515.g96805dcf.el9.x8 3.4 MB/s | 5.4 MB 00:01 2024-05-09T08:20:17.049 INFO:teuthology.orchestra.run.smithi110.stdout:(8/156): ceph-osd-19.0.0-2515.g96805dcf.el9.x86 15 MB/s | 17 MB 00:01 2024-05-09T08:20:17.107 INFO:teuthology.orchestra.run.smithi110.stdout:(9/156): ceph-selinux-19.0.0-2515.g96805dcf.el9 432 kB/s | 25 kB 00:00 2024-05-09T08:20:17.823 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:20:17.824 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:20:17.834 INFO:teuthology.orchestra.run.smithi107.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T08:20:17.835 INFO:teuthology.orchestra.run.smithi107.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T08:20:17.836 INFO:teuthology.orchestra.run.smithi107.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T08:20:17.836 INFO:teuthology.orchestra.run.smithi107.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T08:20:17.838 INFO:teuthology.orchestra.run.smithi107.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T08:20:17.841 INFO:teuthology.orchestra.run.smithi107.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T08:20:17.842 INFO:teuthology.orchestra.run.smithi107.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T08:20:17.902 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-05-09T08:20:17.912 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-05-09T08:20:17.913 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-05-09T08:20:17.913 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-05-09T08:20:17.913 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-05-09T08:20:17.913 INFO:teuthology.orchestra.run.smithi107.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T08:20:17.913 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 6.6 k 2024-05-09T08:20:17.913 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.4 M 2024-05-09T08:20:17.914 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 889 k 2024-05-09T08:20:17.914 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 145 k 2024-05-09T08:20:17.914 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 1.5 M 2024-05-09T08:20:17.914 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 142 k 2024-05-09T08:20:17.914 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 3.5 M 2024-05-09T08:20:17.914 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 7.4 M 2024-05-09T08:20:17.915 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 49 k 2024-05-09T08:20:17.915 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 10 M 2024-05-09T08:20:17.915 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 48 M 2024-05-09T08:20:17.915 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 264 k 2024-05-09T08:20:17.915 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 763 k 2024-05-09T08:20:17.915 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 222 k 2024-05-09T08:20:17.915 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-shell x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 27 k 2024-05-09T08:20:17.915 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-top noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 18 k 2024-05-09T08:20:17.915 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T08:20:17.916 INFO:teuthology.orchestra.run.smithi107.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T08:20:17.916 INFO:teuthology.orchestra.run.smithi107.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T08:20:17.916 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 32 k 2024-05-09T08:20:17.916 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 746 k 2024-05-09T08:20:17.916 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 127 k 2024-05-09T08:20:17.917 INFO:teuthology.orchestra.run.smithi107.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T08:20:17.917 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T08:20:17.917 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T08:20:17.917 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T08:20:17.917 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T08:20:17.917 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T08:20:17.917 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 162 k 2024-05-09T08:20:17.917 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 321 k 2024-05-09T08:20:17.917 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 298 k 2024-05-09T08:20:17.917 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 99 k 2024-05-09T08:20:17.917 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 85 k 2024-05-09T08:20:17.917 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:20:17.918 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 171 k 2024-05-09T08:20:17.918 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T08:20:17.918 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2024-05-09T08:20:17.918 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T08:20:17.918 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T08:20:17.918 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T08:20:17.918 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T08:20:17.918 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.4 M 2024-05-09T08:20:17.918 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:20:17.918 INFO:teuthology.orchestra.run.smithi107.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T08:20:17.918 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T08:20:17.919 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-05-09T08:20:17.919 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T08:20:17.919 INFO:teuthology.orchestra.run.smithi107.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T08:20:17.919 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T08:20:17.919 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 21 M 2024-05-09T08:20:17.919 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 32 k 2024-05-09T08:20:17.919 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 2.4 M 2024-05-09T08:20:17.919 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 249 k 2024-05-09T08:20:17.920 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 4.7 M 2024-05-09T08:20:17.920 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 17 M 2024-05-09T08:20:17.920 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 16 k 2024-05-09T08:20:17.920 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 25 k 2024-05-09T08:20:17.920 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T08:20:17.920 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T08:20:17.920 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T08:20:17.920 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T08:20:17.920 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T08:20:17.920 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T08:20:17.920 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T08:20:17.921 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T08:20:17.921 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 163 k 2024-05-09T08:20:17.921 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T08:20:17.921 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T08:20:17.921 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T08:20:17.921 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T08:20:17.921 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T08:20:17.921 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 500 k 2024-05-09T08:20:17.922 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T08:20:17.922 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.3 M 2024-05-09T08:20:17.922 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T08:20:17.922 INFO:teuthology.orchestra.run.smithi107.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T08:20:17.922 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T08:20:17.922 INFO:teuthology.orchestra.run.smithi107.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T08:20:17.922 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T08:20:17.922 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T08:20:17.922 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T08:20:17.923 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T08:20:17.923 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T08:20:17.923 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T08:20:17.923 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T08:20:17.923 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T08:20:17.923 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T08:20:17.923 INFO:teuthology.orchestra.run.smithi107.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T08:20:17.923 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T08:20:17.923 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T08:20:17.923 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T08:20:17.923 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 45 k 2024-05-09T08:20:17.924 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 135 k 2024-05-09T08:20:17.924 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T08:20:17.924 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T08:20:17.924 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T08:20:17.924 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T08:20:17.924 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T08:20:17.924 INFO:teuthology.orchestra.run.smithi107.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T08:20:17.924 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T08:20:17.925 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T08:20:17.925 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T08:20:17.925 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T08:20:17.925 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T08:20:17.925 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T08:20:17.925 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T08:20:17.926 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T08:20:17.926 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T08:20:17.926 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T08:20:17.926 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T08:20:17.926 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T08:20:17.926 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T08:20:17.927 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T08:20:17.927 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T08:20:17.927 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T08:20:17.927 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T08:20:17.927 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T08:20:17.928 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T08:20:17.928 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T08:20:17.928 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T08:20:17.928 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T08:20:17.928 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T08:20:17.928 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T08:20:17.928 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T08:20:17.929 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T08:20:17.929 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T08:20:17.929 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T08:20:17.929 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T08:20:17.929 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T08:20:17.929 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T08:20:17.930 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T08:20:17.930 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T08:20:17.930 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T08:20:17.930 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T08:20:17.930 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T08:20:17.930 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T08:20:17.930 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T08:20:17.931 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T08:20:17.931 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T08:20:17.931 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T08:20:17.931 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T08:20:17.931 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T08:20:17.931 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T08:20:17.931 INFO:teuthology.orchestra.run.smithi107.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T08:20:17.932 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T08:20:17.932 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T08:20:17.932 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T08:20:17.932 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T08:20:17.932 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T08:20:17.932 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T08:20:17.933 INFO:teuthology.orchestra.run.smithi107.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T08:20:17.933 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T08:20:17.933 INFO:teuthology.orchestra.run.smithi107.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T08:20:17.933 INFO:teuthology.orchestra.run.smithi107.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T08:20:17.933 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T08:20:17.933 INFO:teuthology.orchestra.run.smithi107.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T08:20:17.934 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2024-05-09T08:20:17.934 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T08:20:17.934 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:20:17.936 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-05-09T08:20:17.936 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-05-09T08:20:17.936 INFO:teuthology.orchestra.run.smithi107.stdout:Install 148 Packages 2024-05-09T08:20:17.936 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 8 Packages 2024-05-09T08:20:17.937 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:20:17.938 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 225 M 2024-05-09T08:20:17.938 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-05-09T08:20:17.972 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.2 MB 00:00 2024-05-09T08:20:18.093 INFO:teuthology.orchestra.run.smithi110.stdout:(10/156): ceph-radosgw-19.0.0-2515.g96805dcf.el 5.2 MB/s | 10 MB 00:01 2024-05-09T08:20:18.158 INFO:teuthology.orchestra.run.smithi110.stdout:(11/156): cephfs-mirror-19.0.0-2515.g96805dcf.e 3.3 MB/s | 222 kB 00:00 2024-05-09T08:20:18.250 INFO:teuthology.orchestra.run.smithi110.stdout:(12/156): cephfs-shell-19.0.0-2515.g96805dcf.el 295 kB/s | 27 kB 00:00 2024-05-09T08:20:18.333 INFO:teuthology.orchestra.run.smithi110.stdout:(13/156): libcephfs-devel-19.0.0-2515.g96805dcf 380 kB/s | 32 kB 00:00 2024-05-09T08:20:18.432 INFO:teuthology.orchestra.run.smithi107.stdout:(1/156): ceph-19.0.0-2515.g96805dcf.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-05-09T08:20:18.567 INFO:teuthology.orchestra.run.smithi110.stdout:(14/156): ceph-common-19.0.0-2515.g96805dcf.el9 5.2 MB/s | 21 MB 00:03 2024-05-09T08:20:18.610 INFO:teuthology.orchestra.run.smithi110.stdout:(15/156): libcephfs2-19.0.0-2515.g96805dcf.el9. 2.6 MB/s | 746 kB 00:00 2024-05-09T08:20:18.635 INFO:teuthology.orchestra.run.smithi110.stdout:(16/156): libcephsqlite-19.0.0-2515.g96805dcf.e 2.4 MB/s | 163 kB 00:00 2024-05-09T08:20:18.676 INFO:teuthology.orchestra.run.smithi110.stdout:(17/156): librados-devel-19.0.0-2515.g96805dcf. 1.9 MB/s | 127 kB 00:00 2024-05-09T08:20:18.710 INFO:teuthology.orchestra.run.smithi110.stdout:(18/156): libradosstriper1-19.0.0-2515.g96805dc 6.5 MB/s | 500 kB 00:00 2024-05-09T08:20:18.810 INFO:teuthology.orchestra.run.smithi110.stdout:(19/156): python3-ceph-argparse-19.0.0-2515.g96 451 kB/s | 45 kB 00:00 2024-05-09T08:20:18.911 INFO:teuthology.orchestra.run.smithi110.stdout:(20/156): python3-ceph-common-19.0.0-2515.g9680 1.3 MB/s | 135 kB 00:00 2024-05-09T08:20:18.915 INFO:teuthology.orchestra.run.smithi107.stdout:(2/156): ceph-fuse-19.0.0-2515.g96805dcf.el9.x8 1.8 MB/s | 889 kB 00:00 2024-05-09T08:20:19.020 INFO:teuthology.orchestra.run.smithi110.stdout:(21/156): python3-cephfs-19.0.0-2515.g96805dcf. 1.5 MB/s | 162 kB 00:00 2024-05-09T08:20:19.074 INFO:teuthology.orchestra.run.smithi107.stdout:(3/156): ceph-immutable-object-cache-19.0.0-251 917 kB/s | 145 kB 00:00 2024-05-09T08:20:19.136 INFO:teuthology.orchestra.run.smithi110.stdout:(22/156): python3-rados-19.0.0-2515.g96805dcf.e 2.7 MB/s | 321 kB 00:00 2024-05-09T08:20:19.245 INFO:teuthology.orchestra.run.smithi110.stdout:(23/156): python3-rbd-19.0.0-2515.g96805dcf.el9 2.7 MB/s | 298 kB 00:00 2024-05-09T08:20:19.337 INFO:teuthology.orchestra.run.smithi110.stdout:(24/156): python3-rgw-19.0.0-2515.g96805dcf.el9 1.1 MB/s | 99 kB 00:00 2024-05-09T08:20:19.437 INFO:teuthology.orchestra.run.smithi110.stdout:(25/156): rbd-fuse-19.0.0-2515.g96805dcf.el9.x8 854 kB/s | 85 kB 00:00 2024-05-09T08:20:19.473 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 16 kB 00:00 2024-05-09T08:20:19.613 INFO:teuthology.orchestra.run.smithi110.stdout:(26/156): librgw2-19.0.0-2515.g96805dcf.el9.x86 5.7 MB/s | 5.3 MB 00:00 2024-05-09T08:20:19.713 INFO:teuthology.orchestra.run.smithi110.stdout:(27/156): rbd-nbd-19.0.0-2515.g96805dcf.el9.x86 1.7 MB/s | 171 kB 00:00 2024-05-09T08:20:19.813 INFO:teuthology.orchestra.run.smithi110.stdout:(28/156): ceph-grafana-dashboards-19.0.0-2515.g 317 kB/s | 32 kB 00:00 2024-05-09T08:20:19.947 INFO:teuthology.orchestra.run.smithi110.stdout:(29/156): rbd-mirror-19.0.0-2515.g96805dcf.el9. 6.1 MB/s | 3.1 MB 00:00 2024-05-09T08:20:19.972 INFO:teuthology.orchestra.run.smithi110.stdout:(30/156): ceph-mgr-cephadm-19.0.0-2515.g96805dc 893 kB/s | 142 kB 00:00 2024-05-09T08:20:20.406 INFO:teuthology.orchestra.run.smithi110.stdout:(31/156): ceph-test-19.0.0-2515.g96805dcf.el9.x 15 MB/s | 48 MB 00:03 2024-05-09T08:20:20.416 INFO:teuthology.orchestra.run.smithi107.stdout:(4/156): ceph-mds-19.0.0-2515.g96805dcf.el9.x86 1.8 MB/s | 2.4 MB 00:01 2024-05-09T08:20:20.481 INFO:teuthology.orchestra.run.smithi110.stdout:(32/156): ceph-mgr-dashboard-19.0.0-2515.g96805 6.6 MB/s | 3.5 MB 00:00 2024-05-09T08:20:20.491 INFO:teuthology.orchestra.run.smithi107.stdout:(5/156): ceph-base-19.0.0-2515.g96805dcf.el9.x8 2.3 MB/s | 5.4 MB 00:02 2024-05-09T08:20:20.506 INFO:teuthology.orchestra.run.smithi110.stdout:(33/156): ceph-mgr-modules-core-19.0.0-2515.g96 2.4 MB/s | 249 kB 00:00 2024-05-09T08:20:20.549 INFO:teuthology.orchestra.run.smithi110.stdout:(34/156): ceph-mgr-rook-19.0.0-2515.g96805dcf.e 739 kB/s | 49 kB 00:00 2024-05-09T08:20:20.574 INFO:teuthology.orchestra.run.smithi110.stdout:(35/156): ceph-prometheus-alerts-19.0.0-2515.g9 245 kB/s | 16 kB 00:00 2024-05-09T08:20:20.632 INFO:teuthology.orchestra.run.smithi110.stdout:(36/156): ceph-volume-19.0.0-2515.g96805dcf.el9 3.1 MB/s | 264 kB 00:00 2024-05-09T08:20:20.724 INFO:teuthology.orchestra.run.smithi110.stdout:(37/156): cephadm-19.0.0-2515.g96805dcf.el9.noa 5.0 MB/s | 763 kB 00:00 2024-05-09T08:20:20.749 INFO:teuthology.orchestra.run.smithi110.stdout:(38/156): cephfs-top-19.0.0-2515.g96805dcf.el9. 156 kB/s | 18 kB 00:00 2024-05-09T08:20:20.891 INFO:teuthology.orchestra.run.smithi110.stdout:(39/156): fuse-2.9.9-15.el9.x86_64.rpm 481 kB/s | 80 kB 00:00 2024-05-09T08:20:20.936 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-05-09T08:20:21.010 INFO:teuthology.orchestra.run.smithi107.stdout:(6/156): ceph-mgr-19.0.0-2515.g96805dcf.el9.x86 2.5 MB/s | 1.5 MB 00:00 2024-05-09T08:20:21.050 INFO:teuthology.orchestra.run.smithi110.stdout:(40/156): ceph-mgr-diskprediction-local-19.0.0- 6.9 MB/s | 7.4 MB 00:01 2024-05-09T08:20:21.075 INFO:teuthology.orchestra.run.smithi110.stdout:(41/156): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-05-09T08:20:21.158 INFO:teuthology.orchestra.run.smithi107.stdout:(7/156): ceph-mon-19.0.0-2515.g96805dcf.el9.x86 7.0 MB/s | 4.7 MB 00:00 2024-05-09T08:20:21.243 INFO:teuthology.orchestra.run.smithi110.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-05-09T08:20:21.293 INFO:teuthology.orchestra.run.smithi110.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-05-09T08:20:21.343 INFO:teuthology.orchestra.run.smithi110.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-05-09T08:20:21.393 INFO:teuthology.orchestra.run.smithi110.stdout:(45/156): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-05-09T08:20:21.422 INFO:teuthology.orchestra.run.smithi110.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-05-09T08:20:21.519 INFO:teuthology.orchestra.run.smithi110.stdout:(47/156): automake-1.16.2-8.el9.noarch.rpm 6.7 MB/s | 689 kB 00:00 2024-05-09T08:20:21.586 INFO:teuthology.orchestra.run.smithi110.stdout:(48/156): python3-urllib3-1.26.5-5.el9.noarch.r 1.1 MB/s | 215 kB 00:00 2024-05-09T08:20:21.645 INFO:teuthology.orchestra.run.smithi110.stdout:(49/156): bison-3.7.4-5.el9.x86_64.rpm 7.4 MB/s | 943 kB 00:00 2024-05-09T08:20:21.678 INFO:teuthology.orchestra.run.smithi110.stdout:(50/156): python3-cryptography-36.0.1-4.el9.x86 2.0 MB/s | 1.2 MB 00:00 2024-05-09T08:20:21.695 INFO:teuthology.orchestra.run.smithi110.stdout:(51/156): boost-program-options-1.75.0-8.el9.x8 982 kB/s | 107 kB 00:00 2024-05-09T08:20:21.720 INFO:teuthology.orchestra.run.smithi110.stdout:(52/156): elfutils-libelf-devel-0.190-2.el9.x86 284 kB/s | 21 kB 00:00 2024-05-09T08:20:21.762 INFO:teuthology.orchestra.run.smithi110.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 488 kB/s | 33 kB 00:00 2024-05-09T08:20:21.788 INFO:teuthology.orchestra.run.smithi110.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-05-09T08:20:21.830 INFO:teuthology.orchestra.run.smithi110.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 273 kB/s | 18 kB 00:00 2024-05-09T08:20:21.855 INFO:teuthology.orchestra.run.smithi110.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 167 kB/s | 11 kB 00:00 2024-05-09T08:20:21.914 INFO:teuthology.orchestra.run.smithi110.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-05-09T08:20:21.939 INFO:teuthology.orchestra.run.smithi110.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 541 kB/s | 45 kB 00:00 2024-05-09T08:20:21.989 INFO:teuthology.orchestra.run.smithi110.stdout:(59/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-05-09T08:20:22.039 INFO:teuthology.orchestra.run.smithi110.stdout:(60/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 510 kB/s | 25 kB 00:00 2024-05-09T08:20:22.041 INFO:teuthology.orchestra.run.smithi107.stdout:(8/156): ceph-common-19.0.0-2515.g96805dcf.el9. 5.4 MB/s | 21 MB 00:03 2024-05-09T08:20:22.082 INFO:teuthology.orchestra.run.smithi110.stdout:(61/156): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 662 kB 00:00 2024-05-09T08:20:22.107 INFO:teuthology.orchestra.run.smithi110.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 708 kB/s | 47 kB 00:00 2024-05-09T08:20:22.132 INFO:teuthology.orchestra.run.smithi110.stdout:(63/156): libtool-2.4.6-45.el9.x86_64.rpm 3.0 MB/s | 582 kB 00:00 2024-05-09T08:20:22.158 INFO:teuthology.orchestra.run.smithi107.stdout:(9/156): ceph-radosgw-19.0.0-2515.g96805dcf.el9 10 MB/s | 10 MB 00:00 2024-05-09T08:20:22.175 INFO:teuthology.orchestra.run.smithi107.stdout:(10/156): ceph-selinux-19.0.0-2515.g96805dcf.el 188 kB/s | 25 kB 00:00 2024-05-09T08:20:22.182 INFO:teuthology.orchestra.run.smithi110.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-05-09T08:20:22.208 INFO:teuthology.orchestra.run.smithi110.stdout:(65/156): openblas-0.3.21-2.el9.x86_64.rpm 468 kB/s | 35 kB 00:00 2024-05-09T08:20:22.233 INFO:teuthology.orchestra.run.smithi110.stdout:(66/156): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-05-09T08:20:22.259 INFO:teuthology.orchestra.run.smithi107.stdout:(11/156): cephfs-mirror-19.0.0-2515.g96805dcf.e 2.6 MB/s | 222 kB 00:00 2024-05-09T08:20:22.334 INFO:teuthology.orchestra.run.smithi107.stdout:(12/156): cephfs-shell-19.0.0-2515.g96805dcf.el 360 kB/s | 27 kB 00:00 2024-05-09T08:20:22.366 INFO:teuthology.orchestra.run.smithi110.stdout:(67/156): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-05-09T08:20:22.409 INFO:teuthology.orchestra.run.smithi107.stdout:(13/156): libcephfs-devel-19.0.0-2515.g96805dcf 422 kB/s | 32 kB 00:00 2024-05-09T08:20:22.467 INFO:teuthology.orchestra.run.smithi110.stdout:(68/156): openblas-openmp-0.3.21-2.el9.x86_64.r 17 MB/s | 4.7 MB 00:00 2024-05-09T08:20:22.526 INFO:teuthology.orchestra.run.smithi107.stdout:(14/156): libcephfs2-19.0.0-2515.g96805dcf.el9. 6.3 MB/s | 746 kB 00:00 2024-05-09T08:20:22.542 INFO:teuthology.orchestra.run.smithi110.stdout:(69/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-05-09T08:20:22.592 INFO:teuthology.orchestra.run.smithi110.stdout:(70/156): postgresql-contrib-13.14-1.el9.x86_64 3.9 MB/s | 911 kB 00:00 2024-05-09T08:20:22.618 INFO:teuthology.orchestra.run.smithi107.stdout:(15/156): libcephsqlite-19.0.0-2515.g96805dcf.e 1.7 MB/s | 163 kB 00:00 2024-05-09T08:20:22.627 INFO:teuthology.orchestra.run.smithi110.stdout:(71/156): postgresql-private-libs-13.14-1.el9.x 838 kB/s | 133 kB 00:00 2024-05-09T08:20:22.693 INFO:teuthology.orchestra.run.smithi110.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-05-09T08:20:22.718 INFO:teuthology.orchestra.run.smithi107.stdout:(16/156): librados-devel-19.0.0-2515.g96805dcf. 1.2 MB/s | 127 kB 00:00 2024-05-09T08:20:22.735 INFO:teuthology.orchestra.run.smithi110.stdout:(73/156): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-05-09T08:20:22.836 INFO:teuthology.orchestra.run.smithi107.stdout:(17/156): libradosstriper1-19.0.0-2515.g96805dc 4.2 MB/s | 500 kB 00:00 2024-05-09T08:20:22.860 INFO:teuthology.orchestra.run.smithi110.stdout:(74/156): postgresql-server-13.14-1.el9.x86_64. 18 MB/s | 5.8 MB 00:00 2024-05-09T08:20:22.886 INFO:teuthology.orchestra.run.smithi110.stdout:(75/156): python3-devel-3.9.18-3.el9.x86_64.rpm 1.6 MB/s | 244 kB 00:00 2024-05-09T08:20:22.927 INFO:teuthology.orchestra.run.smithi110.stdout:(76/156): python3-jsonpatch-1.21-16.el9.noarch. 619 kB/s | 26 kB 00:00 2024-05-09T08:20:22.961 INFO:teuthology.orchestra.run.smithi110.stdout:(77/156): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-05-09T08:20:23.003 INFO:teuthology.orchestra.run.smithi110.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 254 kB/s | 19 kB 00:00 2024-05-09T08:20:23.045 INFO:teuthology.orchestra.run.smithi110.stdout:(79/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-05-09T08:20:23.120 INFO:teuthology.orchestra.run.smithi110.stdout:(80/156): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-05-09T08:20:23.146 INFO:teuthology.orchestra.run.smithi110.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 244 kB/s | 35 kB 00:00 2024-05-09T08:20:23.213 INFO:teuthology.orchestra.run.smithi110.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.7 MB/s | 250 kB 00:00 2024-05-09T08:20:23.238 INFO:teuthology.orchestra.run.smithi110.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-05-09T08:20:23.297 INFO:teuthology.orchestra.run.smithi110.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 924 kB/s | 77 kB 00:00 2024-05-09T08:20:23.322 INFO:teuthology.orchestra.run.smithi110.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 505 kB/s | 42 kB 00:00 2024-05-09T08:20:23.414 INFO:teuthology.orchestra.run.smithi110.stdout:(86/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 5.1 MB 00:00 2024-05-09T08:20:23.427 INFO:teuthology.orchestra.run.smithi107.stdout:(18/156): librgw2-19.0.0-2515.g96805dcf.el9.x86 9.0 MB/s | 5.3 MB 00:00 2024-05-09T08:20:23.456 INFO:teuthology.orchestra.run.smithi110.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-05-09T08:20:23.481 INFO:teuthology.orchestra.run.smithi110.stdout:(88/156): python3-protobuf-3.14.0-13.el9.noarch 1.4 MB/s | 269 kB 00:00 2024-05-09T08:20:23.528 INFO:teuthology.orchestra.run.smithi107.stdout:(19/156): python3-ceph-argparse-19.0.0-2515.g96 451 kB/s | 45 kB 00:00 2024-05-09T08:20:23.531 INFO:teuthology.orchestra.run.smithi110.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 279 kB 00:00 2024-05-09T08:20:23.603 INFO:teuthology.orchestra.run.smithi107.stdout:(20/156): python3-ceph-common-19.0.0-2515.g9680 1.8 MB/s | 135 kB 00:00 2024-05-09T08:20:23.606 INFO:teuthology.orchestra.run.smithi110.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 339 kB/s | 51 kB 00:00 2024-05-09T08:20:23.632 INFO:teuthology.orchestra.run.smithi110.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 356 kB/s | 54 kB 00:00 2024-05-09T08:20:23.678 INFO:teuthology.orchestra.run.smithi107.stdout:(21/156): python3-cephfs-19.0.0-2515.g96805dcf. 2.1 MB/s | 162 kB 00:00 2024-05-09T08:20:23.682 INFO:teuthology.orchestra.run.smithi110.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-05-09T08:20:23.707 INFO:teuthology.orchestra.run.smithi110.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 587 kB/s | 44 kB 00:00 2024-05-09T08:20:23.758 INFO:teuthology.orchestra.run.smithi110.stdout:(94/156): qatlib-service-24.02.0-1.el9.x86_64.r 718 kB/s | 36 kB 00:00 2024-05-09T08:20:23.762 INFO:teuthology.orchestra.run.smithi107.stdout:(22/156): python3-rados-19.0.0-2515.g96805dcf.e 3.8 MB/s | 321 kB 00:00 2024-05-09T08:20:23.783 INFO:teuthology.orchestra.run.smithi110.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-05-09T08:20:23.833 INFO:teuthology.orchestra.run.smithi110.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 631 kB/s | 47 kB 00:00 2024-05-09T08:20:23.846 INFO:teuthology.orchestra.run.smithi107.stdout:(23/156): python3-rbd-19.0.0-2515.g96805dcf.el9 3.5 MB/s | 298 kB 00:00 2024-05-09T08:20:23.892 INFO:teuthology.orchestra.run.smithi110.stdout:(97/156): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 305 kB 00:00 2024-05-09T08:20:23.917 INFO:teuthology.orchestra.run.smithi110.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 691 kB/s | 58 kB 00:00 2024-05-09T08:20:23.929 INFO:teuthology.orchestra.run.smithi107.stdout:(24/156): python3-rgw-19.0.0-2515.g96805dcf.el9 1.2 MB/s | 99 kB 00:00 2024-05-09T08:20:23.985 INFO:teuthology.orchestra.run.smithi110.stdout:(99/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.0 MB/s | 94 kB 00:00 2024-05-09T08:20:24.010 INFO:teuthology.orchestra.run.smithi110.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 690 kB/s | 64 kB 00:00 2024-05-09T08:20:24.013 INFO:teuthology.orchestra.run.smithi107.stdout:(25/156): rbd-fuse-19.0.0-2515.g96805dcf.el9.x8 1.0 MB/s | 85 kB 00:00 2024-05-09T08:20:24.060 INFO:teuthology.orchestra.run.smithi110.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 596 kB/s | 45 kB 00:00 2024-05-09T08:20:24.086 INFO:teuthology.orchestra.run.smithi110.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 296 kB/s | 22 kB 00:00 2024-05-09T08:20:24.344 INFO:teuthology.orchestra.run.smithi110.stdout:(103/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-05-09T08:20:24.388 INFO:teuthology.orchestra.run.smithi107.stdout:(26/156): rbd-mirror-19.0.0-2515.g96805dcf.el9. 8.2 MB/s | 3.1 MB 00:00 2024-05-09T08:20:24.461 INFO:teuthology.orchestra.run.smithi110.stdout:(104/156): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-05-09T08:20:24.463 INFO:teuthology.orchestra.run.smithi107.stdout:(27/156): rbd-nbd-19.0.0-2515.g96805dcf.el9.x86 2.2 MB/s | 171 kB 00:00 2024-05-09T08:20:24.486 INFO:teuthology.orchestra.run.smithi110.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-05-09T08:20:24.512 INFO:teuthology.orchestra.run.smithi110.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 388 kB/s | 19 kB 00:00 2024-05-09T08:20:24.539 INFO:teuthology.orchestra.run.smithi107.stdout:(28/156): ceph-grafana-dashboards-19.0.0-2515.g 424 kB/s | 32 kB 00:00 2024-05-09T08:20:24.554 INFO:teuthology.orchestra.run.smithi110.stdout:(107/156): gperftools-libs-2.9.1-2.el9.x86_64.r 1.4 MB/s | 309 kB 00:00 2024-05-09T08:20:24.579 INFO:teuthology.orchestra.run.smithi110.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 388 kB/s | 26 kB 00:00 2024-05-09T08:20:24.604 INFO:teuthology.orchestra.run.smithi110.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-05-09T08:20:24.629 INFO:teuthology.orchestra.run.smithi110.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-05-09T08:20:24.654 INFO:teuthology.orchestra.run.smithi110.stdout:(111/156): luarocks-3.9.2-1.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-05-09T08:20:24.763 INFO:teuthology.orchestra.run.smithi110.stdout:(112/156): python3-asyncssh-2.13.2-5.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-05-09T08:20:24.814 INFO:teuthology.orchestra.run.smithi110.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.5 MB/s | 839 kB 00:00 2024-05-09T08:20:24.831 INFO:teuthology.orchestra.run.smithi110.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 648 kB/s | 43 kB 00:00 2024-05-09T08:20:24.847 INFO:teuthology.orchestra.run.smithi110.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 963 kB/s | 32 kB 00:00 2024-05-09T08:20:24.864 INFO:teuthology.orchestra.run.smithi110.stdout:(116/156): python3-certifi-2021.10.8-2.el9.noar 443 kB/s | 15 kB 00:00 2024-05-09T08:20:24.890 INFO:teuthology.orchestra.run.smithi110.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-05-09T08:20:24.948 INFO:teuthology.orchestra.run.smithi107.stdout:(29/156): ceph-test-19.0.0-2515.g96805dcf.el9.x 17 MB/s | 48 MB 00:02 2024-05-09T08:20:24.948 INFO:teuthology.orchestra.run.smithi110.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-05-09T08:20:24.964 INFO:teuthology.orchestra.run.smithi107.stdout:(30/156): ceph-mgr-cephadm-19.0.0-2515.g96805dc 333 kB/s | 142 kB 00:00 2024-05-09T08:20:24.974 INFO:teuthology.orchestra.run.smithi110.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-05-09T08:20:24.999 INFO:teuthology.orchestra.run.smithi110.stdout:(120/156): python3-colorama-0.4.4-12.el9.noarch 699 kB/s | 35 kB 00:00 2024-05-09T08:20:25.041 INFO:teuthology.orchestra.run.smithi110.stdout:(121/156): python3-google-auth-2.29.0-1.el9.noa 3.0 MB/s | 206 kB 00:00 2024-05-09T08:20:25.099 INFO:teuthology.orchestra.run.smithi110.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-05-09T08:20:25.142 INFO:teuthology.orchestra.run.smithi110.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 256 kB/s | 11 kB 00:00 2024-05-09T08:20:25.184 INFO:teuthology.orchestra.run.smithi110.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 425 kB/s | 18 kB 00:00 2024-05-09T08:20:25.225 INFO:teuthology.orchestra.run.smithi110.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 557 kB/s | 23 kB 00:00 2024-05-09T08:20:25.240 INFO:teuthology.orchestra.run.smithi107.stdout:(31/156): ceph-mgr-dashboard-19.0.0-2515.g96805 12 MB/s | 3.5 MB 00:00 2024-05-09T08:20:25.268 INFO:teuthology.orchestra.run.smithi110.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 463 kB/s | 19 kB 00:00 2024-05-09T08:20:25.306 INFO:teuthology.orchestra.run.smithi107.stdout:(32/156): ceph-mgr-modules-core-19.0.0-2515.g96 3.7 MB/s | 249 kB 00:00 2024-05-09T08:20:25.309 INFO:teuthology.orchestra.run.smithi110.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 472 kB/s | 20 kB 00:00 2024-05-09T08:20:25.399 INFO:teuthology.orchestra.run.smithi107.stdout:(33/156): ceph-mgr-rook-19.0.0-2515.g96805dcf.e 537 kB/s | 49 kB 00:00 2024-05-09T08:20:25.460 INFO:teuthology.orchestra.run.smithi110.stdout:(128/156): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-05-09T08:20:25.474 INFO:teuthology.orchestra.run.smithi107.stdout:(34/156): ceph-prometheus-alerts-19.0.0-2515.g9 220 kB/s | 16 kB 00:00 2024-05-09T08:20:25.502 INFO:teuthology.orchestra.run.smithi110.stdout:(129/156): python3-grpcio-1.46.7-10.el9.x86_64. 4.1 MB/s | 2.0 MB 00:00 2024-05-09T08:20:25.527 INFO:teuthology.orchestra.run.smithi110.stdout:(130/156): python3-logutils-0.3.5-21.el9.noarch 693 kB/s | 46 kB 00:00 2024-05-09T08:20:25.576 INFO:teuthology.orchestra.run.smithi107.stdout:(35/156): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-05-09T08:20:25.608 INFO:teuthology.orchestra.run.smithi107.stdout:(36/156): ceph-volume-19.0.0-2515.g96805dcf.el9 1.9 MB/s | 264 kB 00:00 2024-05-09T08:20:25.611 INFO:teuthology.orchestra.run.smithi110.stdout:(131/156): libarrow-9.0.0-11.el9.x86_64.rpm 3.9 MB/s | 4.4 MB 00:01 2024-05-09T08:20:25.636 INFO:teuthology.orchestra.run.smithi110.stdout:(132/156): python3-more-itertools-8.12.0-2.el9. 588 kB/s | 79 kB 00:00 2024-05-09T08:20:25.658 INFO:teuthology.orchestra.run.smithi107.stdout:(37/156): cephadm-19.0.0-2515.g96805dcf.el9.noa 9.1 MB/s | 763 kB 00:00 2024-05-09T08:20:25.661 INFO:teuthology.orchestra.run.smithi110.stdout:(133/156): python3-natsort-7.1.1-5.el9.noarch.r 432 kB/s | 58 kB 00:00 2024-05-09T08:20:25.675 INFO:teuthology.orchestra.run.smithi107.stdout:(38/156): cephfs-top-19.0.0-2515.g96805dcf.el9. 272 kB/s | 18 kB 00:00 2024-05-09T08:20:25.720 INFO:teuthology.orchestra.run.smithi110.stdout:(134/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-05-09T08:20:25.745 INFO:teuthology.orchestra.run.smithi110.stdout:(135/156): python3-portend-3.1.0-2.el9.noarch.r 197 kB/s | 16 kB 00:00 2024-05-09T08:20:25.771 INFO:teuthology.orchestra.run.smithi110.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-05-09T08:20:25.796 INFO:teuthology.orchestra.run.smithi110.stdout:(137/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 671 kB/s | 90 kB 00:00 2024-05-09T08:20:25.821 INFO:teuthology.orchestra.run.smithi110.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-05-09T08:20:25.863 INFO:teuthology.orchestra.run.smithi110.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-05-09T08:20:25.867 INFO:teuthology.orchestra.run.smithi107.stdout:(39/156): fuse-2.9.9-15.el9.x86_64.rpm 385 kB/s | 80 kB 00:00 2024-05-09T08:20:25.889 INFO:teuthology.orchestra.run.smithi110.stdout:(140/156): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-05-09T08:20:25.915 INFO:teuthology.orchestra.run.smithi110.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 390 kB/s | 36 kB 00:00 2024-05-09T08:20:25.939 INFO:teuthology.orchestra.run.smithi110.stdout:(142/156): python3-typing-extensions-4.4.0-2.el 678 kB/s | 51 kB 00:00 2024-05-09T08:20:25.959 INFO:teuthology.orchestra.run.smithi107.stdout:(40/156): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-05-09T08:20:25.964 INFO:teuthology.orchestra.run.smithi110.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-05-09T08:20:25.983 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-05-09T08:20:25.984 INFO:teuthology.orchestra.run.smithi107.stdout:(41/156): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-05-09T08:20:25.989 INFO:teuthology.orchestra.run.smithi110.stdout:(144/156): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-05-09T08:20:26.031 INFO:teuthology.orchestra.run.smithi110.stdout:(145/156): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-05-09T08:20:26.043 INFO:teuthology.orchestra.run.smithi107.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-05-09T08:20:26.057 INFO:teuthology.orchestra.run.smithi110.stdout:(146/156): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-05-09T08:20:26.082 INFO:teuthology.orchestra.run.smithi110.stdout:(147/156): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-05-09T08:20:26.101 INFO:teuthology.orchestra.run.smithi107.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-05-09T08:20:26.169 INFO:teuthology.orchestra.run.smithi107.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-05-09T08:20:26.210 INFO:teuthology.orchestra.run.smithi107.stdout:(45/156): python3-cryptography-36.0.1-4.el9.x86 5.4 MB/s | 1.2 MB 00:00 2024-05-09T08:20:26.228 INFO:teuthology.orchestra.run.smithi107.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-05-09T08:20:26.249 INFO:teuthology.orchestra.run.smithi110.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-05-09T08:20:26.269 INFO:teuthology.orchestra.run.smithi107.stdout:(47/156): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-05-09T08:20:26.461 INFO:teuthology.orchestra.run.smithi107.stdout:(48/156): bison-3.7.4-5.el9.x86_64.rpm 4.8 MB/s | 943 kB 00:00 2024-05-09T08:20:26.487 INFO:teuthology.orchestra.run.smithi107.stdout:(49/156): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 689 kB 00:00 2024-05-09T08:20:26.520 INFO:teuthology.orchestra.run.smithi107.stdout:(50/156): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-05-09T08:20:26.546 INFO:teuthology.orchestra.run.smithi107.stdout:(51/156): elfutils-libelf-devel-0.190-2.el9.x86 366 kB/s | 21 kB 00:00 2024-05-09T08:20:26.571 INFO:teuthology.orchestra.run.smithi107.stdout:(52/156): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-05-09T08:20:26.596 INFO:teuthology.orchestra.run.smithi107.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 653 kB/s | 33 kB 00:00 2024-05-09T08:20:26.607 INFO:teuthology.orchestra.run.smithi110.stdout:(149/156): librados2-19.0.0-2515.g96805dcf.el9. 6.2 MB/s | 3.4 MB 00:00 2024-05-09T08:20:26.688 INFO:teuthology.orchestra.run.smithi107.stdout:(54/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-05-09T08:20:26.713 INFO:teuthology.orchestra.run.smithi107.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 156 kB/s | 18 kB 00:00 2024-05-09T08:20:26.808 INFO:teuthology.orchestra.run.smithi110.stdout:(150/156): NetworkManager-1.47.5-1.el9.x86_64.r 4.1 MB/s | 2.3 MB 00:00 2024-05-09T08:20:26.858 INFO:teuthology.orchestra.run.smithi110.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 838 kB/s | 42 kB 00:00 2024-05-09T08:20:26.880 INFO:teuthology.orchestra.run.smithi107.stdout:(56/156): ceph-osd-19.0.0-2515.g96805dcf.el9.x8 2.9 MB/s | 17 MB 00:05 2024-05-09T08:20:26.906 INFO:teuthology.orchestra.run.smithi107.stdout:(57/156): libaio-devel-0.3.111-13.el9.x86_64.rp 51 kB/s | 11 kB 00:00 2024-05-09T08:20:26.934 INFO:teuthology.orchestra.run.smithi107.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 907 kB/s | 45 kB 00:00 2024-05-09T08:20:26.941 INFO:teuthology.orchestra.run.smithi110.stdout:(152/156): NetworkManager-tui-1.47.5-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-05-09T08:20:26.956 INFO:teuthology.orchestra.run.smithi107.stdout:(59/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 660 kB/s | 160 kB 00:00 2024-05-09T08:20:26.990 INFO:teuthology.orchestra.run.smithi107.stdout:(60/156): libtool-2.4.6-45.el9.x86_64.rpm 9.7 MB/s | 582 kB 00:00 2024-05-09T08:20:27.000 INFO:teuthology.orchestra.run.smithi110.stdout:(153/156): librbd1-19.0.0-2515.g96805dcf.el9.x8 3.4 MB/s | 3.1 MB 00:00 2024-05-09T08:20:27.015 INFO:teuthology.orchestra.run.smithi107.stdout:(61/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 435 kB/s | 25 kB 00:00 2024-05-09T08:20:27.040 INFO:teuthology.orchestra.run.smithi107.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-05-09T08:20:27.042 INFO:teuthology.orchestra.run.smithi110.stdout:(154/156): NetworkManager-libnm-1.47.5-1.el9.x8 4.2 MB/s | 1.8 MB 00:00 2024-05-09T08:20:27.082 INFO:teuthology.orchestra.run.smithi107.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-05-09T08:20:27.133 INFO:teuthology.orchestra.run.smithi107.stdout:(64/156): librdkafka-1.6.1-102.el9.x86_64.rpm 2.9 MB/s | 662 kB 00:00 2024-05-09T08:20:27.158 INFO:teuthology.orchestra.run.smithi107.stdout:(65/156): openblas-0.3.21-2.el9.x86_64.rpm 468 kB/s | 35 kB 00:00 2024-05-09T08:20:27.183 INFO:teuthology.orchestra.run.smithi107.stdout:(66/156): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-05-09T08:20:27.250 INFO:teuthology.orchestra.run.smithi107.stdout:(67/156): postgresql-13.14-1.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-05-09T08:20:27.284 INFO:teuthology.orchestra.run.smithi110.stdout:(155/156): openssl-3.2.1-1.el9.x86_64.rpm 3.9 MB/s | 1.3 MB 00:00 2024-05-09T08:20:27.334 INFO:teuthology.orchestra.run.smithi107.stdout:(68/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2024-05-09T08:20:27.376 INFO:teuthology.orchestra.run.smithi107.stdout:(69/156): postgresql-contrib-13.14-1.el9.x86_64 7.1 MB/s | 911 kB 00:00 2024-05-09T08:20:27.401 INFO:teuthology.orchestra.run.smithi107.stdout:(70/156): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-05-09T08:20:27.476 INFO:teuthology.orchestra.run.smithi107.stdout:(71/156): openblas-openmp-0.3.21-2.el9.x86_64.r 14 MB/s | 4.7 MB 00:00 2024-05-09T08:20:27.518 INFO:teuthology.orchestra.run.smithi107.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 8.6 MB/s | 1.0 MB 00:00 2024-05-09T08:20:27.546 INFO:teuthology.orchestra.run.smithi107.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-05-09T08:20:27.592 INFO:teuthology.orchestra.run.smithi110.stdout:(156/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 4.0 MB/s | 2.4 MB 00:00 2024-05-09T08:20:27.596 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:20:27.596 INFO:teuthology.orchestra.run.smithi110.stdout:Total 16 MB/s | 225 MB 00:13 2024-05-09T08:20:27.627 INFO:teuthology.orchestra.run.smithi107.stdout:(74/156): postgresql-server-13.14-1.el9.x86_64. 23 MB/s | 5.8 MB 00:00 2024-05-09T08:20:27.652 INFO:teuthology.orchestra.run.smithi107.stdout:(75/156): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-05-09T08:20:27.678 INFO:teuthology.orchestra.run.smithi107.stdout:(76/156): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-05-09T08:20:27.703 INFO:teuthology.orchestra.run.smithi107.stdout:(77/156): python3-jsonpatch-1.21-16.el9.noarch. 510 kB/s | 26 kB 00:00 2024-05-09T08:20:27.728 INFO:teuthology.orchestra.run.smithi107.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 380 kB/s | 19 kB 00:00 2024-05-09T08:20:27.754 INFO:teuthology.orchestra.run.smithi107.stdout:(79/156): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-05-09T08:20:27.779 INFO:teuthology.orchestra.run.smithi107.stdout:(80/156): python3-markupsafe-1.1.1-12.el9.x86_6 693 kB/s | 35 kB 00:00 2024-05-09T08:20:27.913 INFO:teuthology.orchestra.run.smithi107.stdout:(81/156): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-05-09T08:20:27.938 INFO:teuthology.orchestra.run.smithi107.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 250 kB 00:00 2024-05-09T08:20:28.013 INFO:teuthology.orchestra.run.smithi107.stdout:(83/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-05-09T08:20:28.038 INFO:teuthology.orchestra.run.smithi107.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 1.7 MB/s | 222 kB 00:00 2024-05-09T08:20:28.064 INFO:teuthology.orchestra.run.smithi107.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-05-09T08:20:28.089 INFO:teuthology.orchestra.run.smithi107.stdout:(86/156): python3-packaging-20.9-5.el9.noarch.r 512 kB/s | 77 kB 00:00 2024-05-09T08:20:28.114 INFO:teuthology.orchestra.run.smithi107.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-05-09T08:20:28.148 INFO:teuthology.orchestra.run.smithi107.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-05-09T08:20:28.173 INFO:teuthology.orchestra.run.smithi107.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 870 kB/s | 51 kB 00:00 2024-05-09T08:20:28.198 INFO:teuthology.orchestra.run.smithi107.stdout:(90/156): python3-protobuf-3.14.0-13.el9.noarch 1.6 MB/s | 269 kB 00:00 2024-05-09T08:20:28.224 INFO:teuthology.orchestra.run.smithi107.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 710 kB/s | 54 kB 00:00 2024-05-09T08:20:28.249 INFO:teuthology.orchestra.run.smithi107.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 828 kB/s | 42 kB 00:00 2024-05-09T08:20:28.291 INFO:teuthology.orchestra.run.smithi107.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 662 kB/s | 44 kB 00:00 2024-05-09T08:20:28.316 INFO:teuthology.orchestra.run.smithi107.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-05-09T08:20:28.334 INFO:teuthology.orchestra.run.smithi157.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:20:28.335 INFO:teuthology.orchestra.run.smithi157.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T08:20:28.346 INFO:teuthology.orchestra.run.smithi157.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T08:20:28.346 INFO:teuthology.orchestra.run.smithi157.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T08:20:28.347 INFO:teuthology.orchestra.run.smithi157.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T08:20:28.348 INFO:teuthology.orchestra.run.smithi157.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T08:20:28.349 INFO:teuthology.orchestra.run.smithi157.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T08:20:28.350 INFO:teuthology.orchestra.run.smithi107.stdout:(95/156): qatlib-service-24.02.0-1.el9.x86_64.r 613 kB/s | 36 kB 00:00 2024-05-09T08:20:28.352 INFO:teuthology.orchestra.run.smithi157.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T08:20:28.352 INFO:teuthology.orchestra.run.smithi157.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T08:20:28.375 INFO:teuthology.orchestra.run.smithi107.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 809 kB/s | 47 kB 00:00 2024-05-09T08:20:28.409 INFO:teuthology.orchestra.run.smithi107.stdout:(97/156): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-05-09T08:20:28.414 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-05-09T08:20:28.434 INFO:teuthology.orchestra.run.smithi107.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 991 kB/s | 58 kB 00:00 2024-05-09T08:20:28.439 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================= 2024-05-09T08:20:28.439 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-05-09T08:20:28.439 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================= 2024-05-09T08:20:28.439 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-05-09T08:20:28.439 INFO:teuthology.orchestra.run.smithi157.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T08:20:28.439 INFO:teuthology.orchestra.run.smithi157.stdout: ceph x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 6.6 k 2024-05-09T08:20:28.440 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.4 M 2024-05-09T08:20:28.440 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 889 k 2024-05-09T08:20:28.440 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 145 k 2024-05-09T08:20:28.440 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 1.5 M 2024-05-09T08:20:28.440 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 142 k 2024-05-09T08:20:28.440 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 3.5 M 2024-05-09T08:20:28.440 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 7.4 M 2024-05-09T08:20:28.440 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 49 k 2024-05-09T08:20:28.440 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 10 M 2024-05-09T08:20:28.440 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 48 M 2024-05-09T08:20:28.440 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 264 k 2024-05-09T08:20:28.440 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 763 k 2024-05-09T08:20:28.440 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 222 k 2024-05-09T08:20:28.441 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-shell x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 27 k 2024-05-09T08:20:28.441 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-top noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 18 k 2024-05-09T08:20:28.441 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T08:20:28.441 INFO:teuthology.orchestra.run.smithi157.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T08:20:28.441 INFO:teuthology.orchestra.run.smithi157.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T08:20:28.441 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 32 k 2024-05-09T08:20:28.441 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 746 k 2024-05-09T08:20:28.441 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 127 k 2024-05-09T08:20:28.441 INFO:teuthology.orchestra.run.smithi157.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T08:20:28.441 INFO:teuthology.orchestra.run.smithi157.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T08:20:28.441 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T08:20:28.441 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T08:20:28.441 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T08:20:28.442 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T08:20:28.442 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 162 k 2024-05-09T08:20:28.442 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 321 k 2024-05-09T08:20:28.442 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 298 k 2024-05-09T08:20:28.442 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 99 k 2024-05-09T08:20:28.442 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 85 k 2024-05-09T08:20:28.442 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:20:28.442 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 171 k 2024-05-09T08:20:28.442 INFO:teuthology.orchestra.run.smithi157.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T08:20:28.442 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrading: 2024-05-09T08:20:28.442 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T08:20:28.442 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T08:20:28.442 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T08:20:28.443 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T08:20:28.443 INFO:teuthology.orchestra.run.smithi157.stdout: librados2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.4 M 2024-05-09T08:20:28.443 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T08:20:28.443 INFO:teuthology.orchestra.run.smithi157.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T08:20:28.443 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T08:20:28.443 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-05-09T08:20:28.443 INFO:teuthology.orchestra.run.smithi157.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T08:20:28.443 INFO:teuthology.orchestra.run.smithi157.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T08:20:28.443 INFO:teuthology.orchestra.run.smithi157.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T08:20:28.443 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 21 M 2024-05-09T08:20:28.443 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 32 k 2024-05-09T08:20:28.443 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 2.4 M 2024-05-09T08:20:28.444 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 249 k 2024-05-09T08:20:28.444 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 4.7 M 2024-05-09T08:20:28.444 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 17 M 2024-05-09T08:20:28.444 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 16 k 2024-05-09T08:20:28.444 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 25 k 2024-05-09T08:20:28.444 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T08:20:28.444 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T08:20:28.444 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T08:20:28.444 INFO:teuthology.orchestra.run.smithi157.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T08:20:28.445 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T08:20:28.445 INFO:teuthology.orchestra.run.smithi157.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T08:20:28.445 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T08:20:28.445 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T08:20:28.445 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 163 k 2024-05-09T08:20:28.445 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T08:20:28.445 INFO:teuthology.orchestra.run.smithi157.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T08:20:28.445 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T08:20:28.445 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T08:20:28.445 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T08:20:28.445 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 500 k 2024-05-09T08:20:28.445 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T08:20:28.446 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.3 M 2024-05-09T08:20:28.446 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T08:20:28.446 INFO:teuthology.orchestra.run.smithi157.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T08:20:28.446 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T08:20:28.446 INFO:teuthology.orchestra.run.smithi157.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T08:20:28.446 INFO:teuthology.orchestra.run.smithi157.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T08:20:28.446 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T08:20:28.446 INFO:teuthology.orchestra.run.smithi157.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T08:20:28.446 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T08:20:28.446 INFO:teuthology.orchestra.run.smithi157.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T08:20:28.446 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T08:20:28.446 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T08:20:28.447 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T08:20:28.447 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T08:20:28.447 INFO:teuthology.orchestra.run.smithi157.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T08:20:28.447 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T08:20:28.447 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T08:20:28.447 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T08:20:28.447 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 45 k 2024-05-09T08:20:28.447 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 135 k 2024-05-09T08:20:28.447 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T08:20:28.447 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T08:20:28.447 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T08:20:28.447 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T08:20:28.447 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T08:20:28.448 INFO:teuthology.orchestra.run.smithi157.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T08:20:28.448 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T08:20:28.448 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T08:20:28.448 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T08:20:28.448 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T08:20:28.448 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T08:20:28.448 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T08:20:28.448 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T08:20:28.448 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T08:20:28.448 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T08:20:28.448 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T08:20:28.448 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T08:20:28.448 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T08:20:28.449 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T08:20:28.449 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T08:20:28.449 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T08:20:28.449 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T08:20:28.449 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T08:20:28.449 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T08:20:28.449 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T08:20:28.449 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T08:20:28.449 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T08:20:28.449 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T08:20:28.449 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T08:20:28.449 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T08:20:28.449 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T08:20:28.450 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T08:20:28.450 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T08:20:28.450 INFO:teuthology.orchestra.run.smithi157.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T08:20:28.450 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T08:20:28.450 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T08:20:28.450 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T08:20:28.450 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T08:20:28.450 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T08:20:28.450 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T08:20:28.450 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T08:20:28.450 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T08:20:28.450 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T08:20:28.450 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T08:20:28.451 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T08:20:28.451 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T08:20:28.451 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T08:20:28.451 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T08:20:28.451 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T08:20:28.451 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T08:20:28.451 INFO:teuthology.orchestra.run.smithi157.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T08:20:28.451 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T08:20:28.451 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T08:20:28.451 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T08:20:28.451 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T08:20:28.451 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T08:20:28.451 INFO:teuthology.orchestra.run.smithi157.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T08:20:28.452 INFO:teuthology.orchestra.run.smithi157.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T08:20:28.452 INFO:teuthology.orchestra.run.smithi157.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T08:20:28.452 INFO:teuthology.orchestra.run.smithi157.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T08:20:28.452 INFO:teuthology.orchestra.run.smithi157.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T08:20:28.452 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T08:20:28.452 INFO:teuthology.orchestra.run.smithi157.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T08:20:28.452 INFO:teuthology.orchestra.run.smithi157.stdout:Installing weak dependencies: 2024-05-09T08:20:28.452 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T08:20:28.452 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:20:28.452 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-05-09T08:20:28.452 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================= 2024-05-09T08:20:28.452 INFO:teuthology.orchestra.run.smithi157.stdout:Install 148 Packages 2024-05-09T08:20:28.452 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrade 8 Packages 2024-05-09T08:20:28.452 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:20:28.453 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 225 M 2024-05-09T08:20:28.453 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-05-09T08:20:28.459 INFO:teuthology.orchestra.run.smithi107.stdout:(99/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-05-09T08:20:28.476 INFO:teuthology.orchestra.run.smithi107.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-05-09T08:20:28.502 INFO:teuthology.orchestra.run.smithi107.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.0 MB/s | 45 kB 00:00 2024-05-09T08:20:28.798 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-05-09T08:20:28.947 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-05-09T08:20:28.948 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-05-09T08:20:29.018 INFO:teuthology.orchestra.run.smithi157.stdout:(1/156): ceph-19.0.0-2515.g96805dcf.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-05-09T08:20:29.060 INFO:teuthology.orchestra.run.smithi107.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-05-09T08:20:29.219 INFO:teuthology.orchestra.run.smithi107.stdout:(103/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:01 2024-05-09T08:20:29.385 INFO:teuthology.orchestra.run.smithi157.stdout:(2/156): ceph-fuse-19.0.0-2515.g96805dcf.el9.x8 2.4 MB/s | 889 kB 00:00 2024-05-09T08:20:29.452 INFO:teuthology.orchestra.run.smithi157.stdout:(3/156): ceph-immutable-object-cache-19.0.0-251 2.1 MB/s | 145 kB 00:00 2024-05-09T08:20:29.478 INFO:teuthology.orchestra.run.smithi107.stdout:(104/156): protobuf-compiler-3.14.0-13.el9.x86_ 885 kB/s | 863 kB 00:00 2024-05-09T08:20:29.503 INFO:teuthology.orchestra.run.smithi107.stdout:(105/156): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-05-09T08:20:29.536 INFO:teuthology.orchestra.run.smithi107.stdout:(106/156): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-05-09T08:20:29.587 INFO:teuthology.orchestra.run.smithi107.stdout:(107/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 521 kB/s | 26 kB 00:00 2024-05-09T08:20:29.620 INFO:teuthology.orchestra.run.smithi107.stdout:(108/156): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-05-09T08:20:29.637 INFO:teuthology.orchestra.run.smithi107.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 966 kB/s | 49 kB 00:00 2024-05-09T08:20:29.704 INFO:teuthology.orchestra.run.smithi107.stdout:(110/156): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-05-09T08:20:29.763 INFO:teuthology.orchestra.run.smithi107.stdout:(111/156): libarrow-9.0.0-11.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-05-09T08:20:29.788 INFO:teuthology.orchestra.run.smithi107.stdout:(112/156): libunwind-1.6.2-1.el9.x86_64.rpm 402 kB/s | 67 kB 00:00 2024-05-09T08:20:29.832 INFO:teuthology.orchestra.run.smithi107.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-05-09T08:20:29.855 INFO:teuthology.orchestra.run.smithi107.stdout:(114/156): python3-asyncssh-2.13.2-5.el9.noarch 5.8 MB/s | 548 kB 00:00 2024-05-09T08:20:29.881 INFO:teuthology.orchestra.run.smithi107.stdout:(115/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 472 kB/s | 43 kB 00:00 2024-05-09T08:20:29.906 INFO:teuthology.orchestra.run.smithi107.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 427 kB/s | 32 kB 00:00 2024-05-09T08:20:29.931 INFO:teuthology.orchestra.run.smithi107.stdout:(117/156): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-05-09T08:20:29.973 INFO:teuthology.orchestra.run.smithi107.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-05-09T08:20:29.998 INFO:teuthology.orchestra.run.smithi107.stdout:(119/156): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-05-09T08:20:30.024 INFO:teuthology.orchestra.run.smithi107.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-05-09T08:20:30.049 INFO:teuthology.orchestra.run.smithi107.stdout:(121/156): python3-colorama-0.4.4-12.el9.noarch 468 kB/s | 35 kB 00:00 2024-05-09T08:20:30.083 INFO:teuthology.orchestra.run.smithi107.stdout:(122/156): python3-google-auth-2.29.0-1.el9.noa 2.4 MB/s | 206 kB 00:00 2024-05-09T08:20:30.108 INFO:teuthology.orchestra.run.smithi107.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-05-09T08:20:30.152 INFO:teuthology.orchestra.run.smithi157.stdout:(4/156): ceph-mds-19.0.0-2515.g96805dcf.el9.x86 3.4 MB/s | 2.4 MB 00:00 2024-05-09T08:20:30.167 INFO:teuthology.orchestra.run.smithi107.stdout:(124/156): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-05-09T08:20:30.192 INFO:teuthology.orchestra.run.smithi107.stdout:(125/156): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-05-09T08:20:30.217 INFO:teuthology.orchestra.run.smithi107.stdout:(126/156): python3-jaraco-classes-3.2.1-5.el9.n 162 kB/s | 18 kB 00:00 2024-05-09T08:20:30.243 INFO:teuthology.orchestra.run.smithi107.stdout:(127/156): python3-jaraco-collections-3.0.0-8.e 307 kB/s | 23 kB 00:00 2024-05-09T08:20:30.270 INFO:teuthology.orchestra.run.smithi107.stdout:(128/156): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-05-09T08:20:30.293 INFO:teuthology.orchestra.run.smithi107.stdout:(129/156): python3-jaraco-text-3.2.0-6.el9.noar 260 kB/s | 20 kB 00:00 2024-05-09T08:20:30.327 INFO:teuthology.orchestra.run.smithi107.stdout:(130/156): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-05-09T08:20:30.352 INFO:teuthology.orchestra.run.smithi107.stdout:(131/156): python3-logutils-0.3.5-21.el9.noarch 550 kB/s | 46 kB 00:00 2024-05-09T08:20:30.377 INFO:teuthology.orchestra.run.smithi107.stdout:(132/156): python3-more-itertools-8.12.0-2.el9. 939 kB/s | 79 kB 00:00 2024-05-09T08:20:30.402 INFO:teuthology.orchestra.run.smithi107.stdout:(133/156): python3-natsort-7.1.1-5.el9.noarch.r 766 kB/s | 58 kB 00:00 2024-05-09T08:20:30.428 INFO:teuthology.orchestra.run.smithi107.stdout:(134/156): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-05-09T08:20:30.453 INFO:teuthology.orchestra.run.smithi107.stdout:(135/156): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-05-09T08:20:30.478 INFO:teuthology.orchestra.run.smithi107.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-05-09T08:20:30.504 INFO:teuthology.orchestra.run.smithi107.stdout:(137/156): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-05-09T08:20:30.529 INFO:teuthology.orchestra.run.smithi107.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-05-09T08:20:30.554 INFO:teuthology.orchestra.run.smithi107.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-05-09T08:20:30.579 INFO:teuthology.orchestra.run.smithi107.stdout:(140/156): python3-rsa-4.9-2.el9.noarch.rpm 783 kB/s | 59 kB 00:00 2024-05-09T08:20:30.621 INFO:teuthology.orchestra.run.smithi107.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 389 kB/s | 36 kB 00:00 2024-05-09T08:20:30.647 INFO:teuthology.orchestra.run.smithi107.stdout:(142/156): python3-typing-extensions-4.4.0-2.el 553 kB/s | 51 kB 00:00 2024-05-09T08:20:30.661 INFO:teuthology.orchestra.run.smithi157.stdout:(5/156): ceph-mgr-19.0.0-2515.g96805dcf.el9.x86 2.9 MB/s | 1.5 MB 00:00 2024-05-09T08:20:30.726 INFO:teuthology.orchestra.run.smithi107.stdout:(143/156): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-05-09T08:20:30.756 INFO:teuthology.orchestra.run.smithi107.stdout:(144/156): python3-websocket-client-1.2.3-2.el9 668 kB/s | 90 kB 00:00 2024-05-09T08:20:30.781 INFO:teuthology.orchestra.run.smithi107.stdout:(145/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-05-09T08:20:30.806 INFO:teuthology.orchestra.run.smithi107.stdout:(146/156): python3-zc-lockfile-2.0-10.el9.noarc 250 kB/s | 20 kB 00:00 2024-05-09T08:20:30.832 INFO:teuthology.orchestra.run.smithi107.stdout:(147/156): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-05-09T08:20:30.874 INFO:teuthology.orchestra.run.smithi107.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-05-09T08:20:31.165 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-05-09T08:20:31.165 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-05-09T08:20:31.397 INFO:teuthology.orchestra.run.smithi107.stdout:(149/156): NetworkManager-1.47.5-1.el9.x86_64.r 4.4 MB/s | 2.3 MB 00:00 2024-05-09T08:20:31.453 INFO:teuthology.orchestra.run.smithi157.stdout:(6/156): ceph-base-19.0.0-2515.g96805dcf.el9.x8 2.0 MB/s | 5.4 MB 00:02 2024-05-09T08:20:31.533 INFO:teuthology.orchestra.run.smithi107.stdout:(150/156): librbd1-19.0.0-2515.g96805dcf.el9.x8 4.5 MB/s | 3.1 MB 00:00 2024-05-09T08:20:31.691 INFO:teuthology.orchestra.run.smithi107.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 264 kB/s | 42 kB 00:00 2024-05-09T08:20:31.733 INFO:teuthology.orchestra.run.smithi107.stdout:(152/156): NetworkManager-libnm-1.47.5-1.el9.x8 5.4 MB/s | 1.8 MB 00:00 2024-05-09T08:20:31.833 INFO:teuthology.orchestra.run.smithi107.stdout:(153/156): NetworkManager-tui-1.47.5-1.el9.x86_ 1.7 MB/s | 248 kB 00:00 2024-05-09T08:20:31.909 INFO:teuthology.orchestra.run.smithi107.stdout:(154/156): openssl-3.2.1-1.el9.x86_64.rpm 7.7 MB/s | 1.3 MB 00:00 2024-05-09T08:20:32.103 INFO:teuthology.orchestra.run.smithi157.stdout:(7/156): ceph-mon-19.0.0-2515.g96805dcf.el9.x86 3.3 MB/s | 4.7 MB 00:01 2024-05-09T08:20:32.192 INFO:teuthology.orchestra.run.smithi107.stdout:(155/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 6.7 MB/s | 2.4 MB 00:00 2024-05-09T08:20:32.242 INFO:teuthology.orchestra.run.smithi107.stdout:(156/156): librados2-19.0.0-2515.g96805dcf.el9. 2.4 MB/s | 3.4 MB 00:01 2024-05-09T08:20:32.245 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:20:32.245 INFO:teuthology.orchestra.run.smithi107.stdout:Total 16 MB/s | 225 MB 00:14 2024-05-09T08:20:33.476 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-05-09T08:20:33.627 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-05-09T08:20:33.627 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-05-09T08:20:34.419 INFO:teuthology.orchestra.run.smithi157.stdout:(8/156): ceph-radosgw-19.0.0-2515.g96805dcf.el9 4.4 MB/s | 10 MB 00:02 2024-05-09T08:20:34.470 INFO:teuthology.orchestra.run.smithi157.stdout:(9/156): ceph-selinux-19.0.0-2515.g96805dcf.el9 505 kB/s | 25 kB 00:00 2024-05-09T08:20:35.767 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-05-09T08:20:35.768 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-05-09T08:20:36.653 INFO:teuthology.orchestra.run.smithi157.stdout:(10/156): ceph-osd-19.0.0-2515.g96805dcf.el9.x8 3.2 MB/s | 17 MB 00:05 2024-05-09T08:20:36.811 INFO:teuthology.orchestra.run.smithi157.stdout:(11/156): cephfs-mirror-19.0.0-2515.g96805dcf.e 1.4 MB/s | 222 kB 00:00 2024-05-09T08:20:36.895 INFO:teuthology.orchestra.run.smithi157.stdout:(12/156): cephfs-shell-19.0.0-2515.g96805dcf.el 325 kB/s | 27 kB 00:00 2024-05-09T08:20:36.979 INFO:teuthology.orchestra.run.smithi157.stdout:(13/156): libcephfs-devel-19.0.0-2515.g96805dcf 381 kB/s | 32 kB 00:00 2024-05-09T08:20:37.235 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-05-09T08:20:37.262 INFO:teuthology.orchestra.run.smithi157.stdout:(14/156): libcephfs2-19.0.0-2515.g96805dcf.el9. 2.6 MB/s | 746 kB 00:00 2024-05-09T08:20:37.323 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T08:20:37.361 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T08:20:37.362 INFO:teuthology.orchestra.run.smithi157.stdout:(15/156): libcephsqlite-19.0.0-2515.g96805dcf.e 1.6 MB/s | 163 kB 00:00 2024-05-09T08:20:37.433 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T08:20:37.463 INFO:teuthology.orchestra.run.smithi157.stdout:(16/156): librados-devel-19.0.0-2515.g96805dcf. 1.2 MB/s | 127 kB 00:00 2024-05-09T08:20:37.473 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T08:20:37.499 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T08:20:37.552 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T08:20:37.705 INFO:teuthology.orchestra.run.smithi157.stdout:(17/156): libradosstriper1-19.0.0-2515.g96805dc 2.0 MB/s | 500 kB 00:00 2024-05-09T08:20:37.954 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T08:20:38.142 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:20:38.247 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:20:38.276 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:20:38.351 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:20:38.380 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:20:38.545 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T08:20:38.602 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T08:20:38.630 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:20:38.734 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:20:38.756 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:20:38.819 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:20:38.849 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:20:38.871 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:20:39.068 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:20:39.155 INFO:teuthology.orchestra.run.smithi157.stdout:(18/156): ceph-common-19.0.0-2515.g96805dcf.el9 2.0 MB/s | 21 MB 00:10 2024-05-09T08:20:39.230 INFO:teuthology.orchestra.run.smithi157.stdout:(19/156): librgw2-19.0.0-2515.g96805dcf.el9.x86 3.5 MB/s | 5.3 MB 00:01 2024-05-09T08:20:39.245 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:20:39.256 INFO:teuthology.orchestra.run.smithi157.stdout:(20/156): python3-ceph-argparse-19.0.0-2515.g96 451 kB/s | 45 kB 00:00 2024-05-09T08:20:39.331 INFO:teuthology.orchestra.run.smithi157.stdout:(21/156): python3-ceph-common-19.0.0-2515.g9680 1.3 MB/s | 135 kB 00:00 2024-05-09T08:20:39.356 INFO:teuthology.orchestra.run.smithi157.stdout:(22/156): python3-cephfs-19.0.0-2515.g96805dcf. 1.6 MB/s | 162 kB 00:00 2024-05-09T08:20:39.388 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:20:39.432 INFO:teuthology.orchestra.run.smithi157.stdout:(23/156): python3-rados-19.0.0-2515.g96805dcf.e 3.1 MB/s | 321 kB 00:00 2024-05-09T08:20:39.448 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:20:39.457 INFO:teuthology.orchestra.run.smithi157.stdout:(24/156): python3-rbd-19.0.0-2515.g96805dcf.el9 2.9 MB/s | 298 kB 00:00 2024-05-09T08:20:39.499 INFO:teuthology.orchestra.run.smithi157.stdout:(25/156): python3-rgw-19.0.0-2515.g96805dcf.el9 1.4 MB/s | 99 kB 00:00 2024-05-09T08:20:39.524 INFO:teuthology.orchestra.run.smithi157.stdout:(26/156): rbd-fuse-19.0.0-2515.g96805dcf.el9.x8 1.2 MB/s | 85 kB 00:00 2024-05-09T08:20:39.569 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:20:39.591 INFO:teuthology.orchestra.run.smithi157.stdout:(27/156): rbd-nbd-19.0.0-2515.g96805dcf.el9.x86 2.5 MB/s | 171 kB 00:00 2024-05-09T08:20:39.641 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:20:39.658 INFO:teuthology.orchestra.run.smithi157.stdout:(28/156): ceph-grafana-dashboards-19.0.0-2515.g 477 kB/s | 32 kB 00:00 2024-05-09T08:20:39.725 INFO:teuthology.orchestra.run.smithi157.stdout:(29/156): ceph-mgr-cephadm-19.0.0-2515.g96805dc 2.1 MB/s | 142 kB 00:00 2024-05-09T08:20:39.840 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:20:39.951 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T08:20:40.044 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T08:20:40.075 INFO:teuthology.orchestra.run.smithi157.stdout:(30/156): rbd-mirror-19.0.0-2515.g96805dcf.el9. 5.4 MB/s | 3.1 MB 00:00 2024-05-09T08:20:40.089 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-09T08:20:40.124 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-09T08:20:40.197 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-09T08:20:40.232 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-09T08:20:40.283 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-09T08:20:40.316 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-09T08:20:40.344 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-09T08:20:40.413 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-09T08:20:40.466 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-09T08:20:40.505 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-09T08:20:40.509 INFO:teuthology.orchestra.run.smithi157.stdout:(31/156): ceph-mgr-dashboard-19.0.0-2515.g96805 4.5 MB/s | 3.5 MB 00:00 2024-05-09T08:20:40.549 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-09T08:20:40.609 INFO:teuthology.orchestra.run.smithi157.stdout:(32/156): ceph-mgr-modules-core-19.0.0-2515.g96 2.4 MB/s | 249 kB 00:00 2024-05-09T08:20:40.625 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-09T08:20:40.669 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 33/164 2024-05-09T08:20:40.676 INFO:teuthology.orchestra.run.smithi157.stdout:(33/156): ceph-mgr-rook-19.0.0-2515.g96805dcf.e 741 kB/s | 49 kB 00:00 2024-05-09T08:20:40.703 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:20:40.731 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 35/164 2024-05-09T08:20:40.743 INFO:teuthology.orchestra.run.smithi157.stdout:(34/156): ceph-prometheus-alerts-19.0.0-2515.g9 247 kB/s | 16 kB 00:00 2024-05-09T08:20:40.800 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-09T08:20:40.844 INFO:teuthology.orchestra.run.smithi157.stdout:(35/156): ceph-volume-19.0.0-2515.g96805dcf.el9 2.6 MB/s | 264 kB 00:00 2024-05-09T08:20:40.846 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-09T08:20:40.919 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-09T08:20:41.052 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-09T08:20:41.106 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-09T08:20:41.254 INFO:teuthology.orchestra.run.smithi157.stdout:(36/156): ceph-test-19.0.0-2515.g96805dcf.el9.x 7.1 MB/s | 48 MB 00:06 2024-05-09T08:20:41.286 INFO:teuthology.orchestra.run.smithi157.stdout:(37/156): cephadm-19.0.0-2515.g96805dcf.el9.noa 1.7 MB/s | 763 kB 00:00 2024-05-09T08:20:41.303 INFO:teuthology.orchestra.run.smithi157.stdout:(38/156): cephfs-top-19.0.0-2515.g96805dcf.el9. 365 kB/s | 18 kB 00:00 2024-05-09T08:20:41.528 INFO:teuthology.orchestra.run.smithi157.stdout:(39/156): ceph-mgr-diskprediction-local-19.0.0- 5.1 MB/s | 7.4 MB 00:01 2024-05-09T08:20:41.553 INFO:teuthology.orchestra.run.smithi157.stdout:(40/156): fuse-2.9.9-15.el9.x86_64.rpm 301 kB/s | 80 kB 00:00 2024-05-09T08:20:41.619 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-09T08:20:41.679 INFO:teuthology.orchestra.run.smithi157.stdout:(41/156): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-05-09T08:20:41.704 INFO:teuthology.orchestra.run.smithi157.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-05-09T08:20:41.710 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-09T08:20:41.740 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-09T08:20:41.763 INFO:teuthology.orchestra.run.smithi157.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-05-09T08:20:41.776 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-09T08:20:41.788 INFO:teuthology.orchestra.run.smithi157.stdout:(44/156): libquadmath-11.4.1-3.el9.x86_64.rpm 753 kB/s | 195 kB 00:00 2024-05-09T08:20:41.813 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 45/164 2024-05-09T08:20:41.830 INFO:teuthology.orchestra.run.smithi157.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-05-09T08:20:41.855 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-09T08:20:41.872 INFO:teuthology.orchestra.run.smithi157.stdout:(46/156): python3-cryptography-36.0.1-4.el9.x86 6.4 MB/s | 1.2 MB 00:00 2024-05-09T08:20:41.897 INFO:teuthology.orchestra.run.smithi157.stdout:(47/156): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-05-09T08:20:41.922 INFO:teuthology.orchestra.run.smithi157.stdout:(48/156): python3-urllib3-1.26.5-5.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-05-09T08:20:42.008 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-09T08:20:42.058 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-09T08:20:42.074 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-05-09T08:20:42.112 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-09T08:20:42.155 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T08:20:42.184 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T08:20:42.198 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T08:20:42.217 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T08:20:42.253 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T08:20:42.265 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T08:20:42.299 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T08:20:42.305 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T08:20:42.331 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T08:20:42.336 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T08:20:42.348 INFO:teuthology.orchestra.run.smithi157.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 251 kB/s | 107 kB 00:00 2024-05-09T08:20:42.379 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T08:20:42.391 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T08:20:42.421 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T08:20:42.456 INFO:teuthology.orchestra.run.smithi157.stdout:(50/156): elfutils-libelf-devel-0.190-2.el9.x86 198 kB/s | 21 kB 00:00 2024-05-09T08:20:42.555 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T08:20:42.590 INFO:teuthology.orchestra.run.smithi157.stdout:(51/156): automake-1.16.2-8.el9.noarch.rpm 960 kB/s | 689 kB 00:00 2024-05-09T08:20:42.673 INFO:teuthology.orchestra.run.smithi157.stdout:(52/156): bison-3.7.4-5.el9.x86_64.rpm 1.2 MB/s | 943 kB 00:00 2024-05-09T08:20:42.707 INFO:teuthology.orchestra.run.smithi157.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-05-09T08:20:42.752 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T08:20:42.757 INFO:teuthology.orchestra.run.smithi157.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 1.0 MB/s | 313 kB 00:00 2024-05-09T08:20:42.782 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T08:20:42.786 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T08:20:42.824 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T08:20:42.831 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:20:42.841 INFO:teuthology.orchestra.run.smithi157.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 136 kB/s | 18 kB 00:00 2024-05-09T08:20:42.875 INFO:teuthology.orchestra.run.smithi157.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 95 kB/s | 11 kB 00:00 2024-05-09T08:20:42.928 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T08:20:42.953 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:20:42.966 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T08:20:42.967 INFO:teuthology.orchestra.run.smithi157.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-05-09T08:20:42.991 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:20:42.992 INFO:teuthology.orchestra.run.smithi157.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-05-09T08:20:43.007 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T08:20:43.040 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T08:20:43.064 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T08:20:43.084 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:20:43.112 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:20:43.125 INFO:teuthology.orchestra.run.smithi157.stdout:(59/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.6 MB/s | 3.0 MB 00:00 2024-05-09T08:20:43.159 INFO:teuthology.orchestra.run.smithi157.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 662 kB 00:00 2024-05-09T08:20:43.160 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T08:20:43.226 INFO:teuthology.orchestra.run.smithi157.stdout:(61/156): libtool-2.4.6-45.el9.x86_64.rpm 2.4 MB/s | 582 kB 00:00 2024-05-09T08:20:43.251 INFO:teuthology.orchestra.run.smithi157.stdout:(62/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 203 kB/s | 25 kB 00:00 2024-05-09T08:20:43.277 INFO:teuthology.orchestra.run.smithi157.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 405 kB/s | 47 kB 00:00 2024-05-09T08:20:43.290 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T08:20:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T08:20:43.330 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T08:20:43.335 INFO:teuthology.orchestra.run.smithi157.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-05-09T08:20:43.359 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T08:20:43.362 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T08:20:43.377 INFO:teuthology.orchestra.run.smithi157.stdout:(65/156): openblas-0.3.21-2.el9.x86_64.rpm 352 kB/s | 35 kB 00:00 2024-05-09T08:20:43.387 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:20:43.393 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T08:20:43.432 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T08:20:43.472 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T08:20:43.490 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:20:43.510 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T08:20:43.519 INFO:teuthology.orchestra.run.smithi157.stdout:(66/156): lua-5.4.4-4.el9.x86_64.rpm 705 kB/s | 188 kB 00:00 2024-05-09T08:20:43.530 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:20:43.593 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:20:43.614 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:20:43.638 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:20:43.658 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T08:20:43.803 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T08:20:43.832 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:20:43.928 INFO:teuthology.orchestra.run.smithi157.stdout:(67/156): openblas-openmp-0.3.21-2.el9.x86_64.r 8.0 MB/s | 4.7 MB 00:00 2024-05-09T08:20:43.928 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T08:20:43.962 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T08:20:44.024 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:20:44.049 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T08:20:44.061 INFO:teuthology.orchestra.run.smithi157.stdout:(68/156): postgresql-contrib-13.14-1.el9.x86_64 6.7 MB/s | 911 kB 00:00 2024-05-09T08:20:44.093 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T08:20:44.124 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T08:20:44.170 INFO:teuthology.orchestra.run.smithi157.stdout:(69/156): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-05-09T08:20:44.170 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:20:44.171 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T08:20:44.230 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:20:44.277 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T08:20:44.312 INFO:teuthology.orchestra.run.smithi157.stdout:(70/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 4.7 MB/s | 4.4 MB 00:00 2024-05-09T08:20:44.334 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T08:20:44.352 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:20:44.423 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:20:44.631 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:20:44.737 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T08:20:44.779 INFO:teuthology.orchestra.run.smithi157.stdout:(71/156): postgresql-server-13.14-1.el9.x86_64. 9.5 MB/s | 5.8 MB 00:00 2024-05-09T08:20:44.831 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T08:20:44.877 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-09T08:20:44.896 INFO:teuthology.orchestra.run.smithi157.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 750 kB/s | 87 kB 00:00 2024-05-09T08:20:44.914 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-09T08:20:44.979 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-09T08:20:45.022 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-09T08:20:45.048 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T08:20:45.064 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-09T08:20:45.097 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-09T08:20:45.098 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T08:20:45.125 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-09T08:20:45.132 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T08:20:45.194 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-09T08:20:45.230 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T08:20:45.255 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-09T08:20:45.287 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-09T08:20:45.290 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T08:20:45.330 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-09T08:20:45.378 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T08:20:45.406 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-09T08:20:45.409 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T08:20:45.441 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 33/164 2024-05-09T08:20:45.515 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:20:45.587 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T08:20:45.623 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T08:20:45.671 INFO:teuthology.orchestra.run.smithi157.stdout:(73/156): postgresql-13.14-1.el9.x86_64.rpm 756 kB/s | 1.6 MB 00:02 2024-05-09T08:20:45.689 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 35/164 2024-05-09T08:20:45.704 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T08:20:45.730 INFO:teuthology.orchestra.run.smithi157.stdout:(74/156): protobuf-3.14.0-13.el9.x86_64.rpm 729 kB/s | 1.0 MB 00:01 2024-05-09T08:20:45.849 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-09T08:20:45.969 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-09T08:20:46.013 INFO:teuthology.orchestra.run.smithi157.stdout:(75/156): python3-devel-3.9.18-3.el9.x86_64.rpm 714 kB/s | 244 kB 00:00 2024-05-09T08:20:46.043 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-09T08:20:46.072 INFO:teuthology.orchestra.run.smithi157.stdout:(76/156): python3-jinja2-2.11.3-5.el9.noarch.rp 725 kB/s | 248 kB 00:00 2024-05-09T08:20:46.114 INFO:teuthology.orchestra.run.smithi157.stdout:(77/156): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-05-09T08:20:46.172 INFO:teuthology.orchestra.run.smithi157.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 191 kB/s | 19 kB 00:00 2024-05-09T08:20:46.181 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-09T08:20:46.237 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-09T08:20:46.281 INFO:teuthology.orchestra.run.smithi157.stdout:(79/156): python3-markupsafe-1.1.1-12.el9.x86_6 321 kB/s | 35 kB 00:00 2024-05-09T08:20:46.366 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T08:20:46.398 INFO:teuthology.orchestra.run.smithi157.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 606 kB/s | 172 kB 00:00 2024-05-09T08:20:46.488 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T08:20:46.755 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-09T08:20:46.773 INFO:teuthology.orchestra.run.smithi157.stdout:(81/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 668 kB/s | 250 kB 00:00 2024-05-09T08:20:46.841 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-09T08:20:46.879 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-09T08:20:46.914 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-09T08:20:46.952 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 45/164 2024-05-09T08:20:46.993 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-09T08:20:47.115 INFO:teuthology.orchestra.run.smithi157.stdout:(82/156): python3-oauthlib-3.1.1-5.el9.noarch.r 650 kB/s | 222 kB 00:00 2024-05-09T08:20:47.151 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-09T08:20:47.205 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-09T08:20:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-09T08:20:47.299 INFO:teuthology.orchestra.run.smithi157.stdout:(83/156): python3-packaging-20.9-5.el9.noarch.r 421 kB/s | 77 kB 00:00 2024-05-09T08:20:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T08:20:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T08:20:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T08:20:47.407 INFO:teuthology.orchestra.run.smithi157.stdout:(84/156): python3-prettytable-0.7.2-27.el9.noar 390 kB/s | 42 kB 00:00 2024-05-09T08:20:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T08:20:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T08:20:47.526 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T08:20:47.569 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T08:20:47.738 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T08:20:47.828 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T08:20:47.832 INFO:teuthology.orchestra.run.smithi157.stdout:(85/156): python3-protobuf-3.14.0-13.el9.noarch 634 kB/s | 269 kB 00:00 2024-05-09T08:20:47.856 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T08:20:47.876 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T08:20:47.889 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T08:20:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T08:20:47.954 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T08:20:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T08:20:48.091 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T08:20:48.178 INFO:teuthology.orchestra.run.smithi157.stdout:(86/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 595 kB/s | 159 kB 00:00 2024-05-09T08:20:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T08:20:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T08:20:48.204 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T08:20:48.207 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T08:20:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T08:20:48.244 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:20:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T08:20:48.318 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:20:48.349 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 102/164 2024-05-09T08:20:48.385 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T08:20:48.438 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T08:20:48.477 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T08:20:48.505 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T08:20:48.533 INFO:teuthology.orchestra.run.smithi157.stdout:(87/156): python3-babel-2.9.1-2.el9.noarch.rpm 1.6 MB/s | 6.0 MB 00:03 2024-05-09T08:20:48.539 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T08:20:48.558 INFO:teuthology.orchestra.run.smithi157.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 608 kB/s | 279 kB 00:00 2024-05-09T08:20:48.578 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T08:20:48.610 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T08:20:48.640 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T08:20:48.642 INFO:teuthology.orchestra.run.smithi157.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 471 kB/s | 51 kB 00:00 2024-05-09T08:20:48.667 INFO:teuthology.orchestra.run.smithi157.stdout:(90/156): python3-requests-oauthlib-1.3.0-12.el 495 kB/s | 54 kB 00:00 2024-05-09T08:20:48.674 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T08:20:48.707 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T08:20:48.748 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T08:20:48.768 INFO:teuthology.orchestra.run.smithi157.stdout:(91/156): python3-toml-0.10.2-6.el9.noarch.rpm 416 kB/s | 42 kB 00:00 2024-05-09T08:20:48.783 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T08:20:48.864 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T08:20:48.868 INFO:teuthology.orchestra.run.smithi157.stdout:(92/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 442 kB/s | 44 kB 00:00 2024-05-09T08:20:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T08:20:48.889 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:20:48.914 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T08:20:48.953 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T08:20:48.960 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:20:48.982 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 106/164 2024-05-09T08:20:49.018 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T08:20:49.123 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T08:20:49.171 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T08:20:49.218 INFO:teuthology.orchestra.run.smithi157.stdout:(93/156): qatlib-24.02.0-1.el9.x86_64.rpm 631 kB/s | 221 kB 00:00 2024-05-09T08:20:49.318 INFO:teuthology.orchestra.run.smithi157.stdout:(94/156): qatlib-service-24.02.0-1.el9.x86_64.r 358 kB/s | 36 kB 00:00 2024-05-09T08:20:49.419 INFO:teuthology.orchestra.run.smithi157.stdout:(95/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 475 kB/s | 47 kB 00:00 2024-05-09T08:20:49.844 INFO:teuthology.orchestra.run.smithi157.stdout:(96/156): socat-1.7.4.1-5.el9.x86_64.rpm 717 kB/s | 305 kB 00:00 2024-05-09T08:20:49.902 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T08:20:49.952 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T08:20:49.986 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T08:20:50.085 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T08:20:50.136 INFO:teuthology.orchestra.run.smithi157.stdout:(97/156): uuid-1.6.2-55.el9.x86_64.rpm 198 kB/s | 58 kB 00:00 2024-05-09T08:20:50.144 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T08:20:50.234 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T08:20:50.263 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T08:20:50.320 INFO:teuthology.orchestra.run.smithi157.stdout:(98/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 510 kB/s | 94 kB 00:00 2024-05-09T08:20:50.429 INFO:teuthology.orchestra.run.smithi157.stdout:(99/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 589 kB/s | 64 kB 00:00 2024-05-09T08:20:50.433 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T08:20:50.471 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T08:20:50.537 INFO:teuthology.orchestra.run.smithi157.stdout:(100/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 413 kB/s | 45 kB 00:00 2024-05-09T08:20:50.548 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T08:20:50.621 INFO:teuthology.orchestra.run.smithi157.stdout:(101/156): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-05-09T08:20:50.880 INFO:teuthology.orchestra.run.smithi157.stdout:(102/156): protobuf-compiler-3.14.0-13.el9.x86_ 3.3 MB/s | 863 kB 00:00 2024-05-09T08:20:51.088 INFO:teuthology.orchestra.run.smithi157.stdout:(103/156): abseil-cpp-20211102.0-3.el9.x86_64.r 2.6 MB/s | 548 kB 00:00 2024-05-09T08:20:51.147 INFO:teuthology.orchestra.run.smithi157.stdout:(104/156): gperftools-libs-2.9.1-2.el9.x86_64.r 5.2 MB/s | 309 kB 00:00 2024-05-09T08:20:51.189 INFO:teuthology.orchestra.run.smithi157.stdout:(105/156): grpc-data-1.46.7-10.el9.noarch.rpm 466 kB/s | 19 kB 00:00 2024-05-09T08:20:51.203 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T08:20:51.281 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:20:51.312 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:20:51.325 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T08:20:51.373 INFO:teuthology.orchestra.run.smithi157.stdout:(106/156): libarrow-9.0.0-11.el9.x86_64.rpm 24 MB/s | 4.4 MB 00:00 2024-05-09T08:20:51.414 INFO:teuthology.orchestra.run.smithi157.stdout:(107/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 627 kB/s | 26 kB 00:00 2024-05-09T08:20:51.456 INFO:teuthology.orchestra.run.smithi157.stdout:(108/156): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-05-09T08:20:51.507 INFO:teuthology.orchestra.run.smithi157.stdout:(109/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-05-09T08:20:51.557 INFO:teuthology.orchestra.run.smithi157.stdout:(110/156): luarocks-3.9.2-1.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-05-09T08:20:51.624 INFO:teuthology.orchestra.run.smithi157.stdout:(111/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-05-09T08:20:51.674 INFO:teuthology.orchestra.run.smithi157.stdout:(112/156): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-05-09T08:20:51.724 INFO:teuthology.orchestra.run.smithi157.stdout:(113/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 868 kB/s | 43 kB 00:00 2024-05-09T08:20:51.775 INFO:teuthology.orchestra.run.smithi157.stdout:(114/156): python3-cachetools-4.2.4-1.el9.noarc 646 kB/s | 32 kB 00:00 2024-05-09T08:20:51.825 INFO:teuthology.orchestra.run.smithi157.stdout:(115/156): python3-certifi-2021.10.8-2.el9.noar 300 kB/s | 15 kB 00:00 2024-05-09T08:20:51.857 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:20:51.892 INFO:teuthology.orchestra.run.smithi157.stdout:(116/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-05-09T08:20:51.899 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:20:51.942 INFO:teuthology.orchestra.run.smithi157.stdout:(117/156): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-05-09T08:20:51.976 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:20:51.976 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T08:20:51.977 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T08:20:51.977 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:20:52.017 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:20:52.017 INFO:teuthology.orchestra.run.smithi157.stdout:(118/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-05-09T08:20:52.068 INFO:teuthology.orchestra.run.smithi157.stdout:(119/156): python3-colorama-0.4.4-12.el9.noarch 706 kB/s | 35 kB 00:00 2024-05-09T08:20:52.118 INFO:teuthology.orchestra.run.smithi157.stdout:(120/156): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-05-09T08:20:52.185 INFO:teuthology.orchestra.run.smithi157.stdout:(121/156): python3-grpcio-1.46.7-10.el9.x86_64. 31 MB/s | 2.0 MB 00:00 2024-05-09T08:20:52.235 INFO:teuthology.orchestra.run.smithi157.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-05-09T08:20:52.286 INFO:teuthology.orchestra.run.smithi157.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-05-09T08:20:52.336 INFO:teuthology.orchestra.run.smithi157.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-05-09T08:20:52.395 INFO:teuthology.orchestra.run.smithi157.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 395 kB/s | 23 kB 00:00 2024-05-09T08:20:52.445 INFO:teuthology.orchestra.run.smithi157.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 389 kB/s | 19 kB 00:00 2024-05-09T08:20:52.495 INFO:teuthology.orchestra.run.smithi157.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 396 kB/s | 20 kB 00:00 2024-05-09T08:20:52.554 INFO:teuthology.orchestra.run.smithi157.stdout:(128/156): python3-kubernetes-26.1.0-2.el9.noar 18 MB/s | 1.0 MB 00:00 2024-05-09T08:20:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T08:20:52.604 INFO:teuthology.orchestra.run.smithi157.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 926 kB/s | 46 kB 00:00 2024-05-09T08:20:52.654 INFO:teuthology.orchestra.run.smithi157.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-05-09T08:20:52.682 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T08:20:52.705 INFO:teuthology.orchestra.run.smithi157.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-05-09T08:20:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T08:20:52.742 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T08:20:52.755 INFO:teuthology.orchestra.run.smithi157.stdout:(132/156): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-05-09T08:20:52.805 INFO:teuthology.orchestra.run.smithi157.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 329 kB/s | 16 kB 00:00 2024-05-09T08:20:52.847 INFO:teuthology.orchestra.run.smithi157.stdout:(134/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-05-09T08:20:52.898 INFO:teuthology.orchestra.run.smithi157.stdout:(135/156): python3-pyperclip-1.8.2-2.el9.noarch 564 kB/s | 28 kB 00:00 2024-05-09T08:20:52.965 INFO:teuthology.orchestra.run.smithi157.stdout:(136/156): python3-repoze-lru-0.7-16.el9.noarch 462 kB/s | 31 kB 00:00 2024-05-09T08:20:53.015 INFO:teuthology.orchestra.run.smithi157.stdout:(137/156): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-05-09T08:20:53.063 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T08:20:53.065 INFO:teuthology.orchestra.run.smithi157.stdout:(138/156): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-05-09T08:20:53.105 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:20:53.115 INFO:teuthology.orchestra.run.smithi157.stdout:(139/156): python3-tempora-5.0.0-2.el9.noarch.r 716 kB/s | 36 kB 00:00 2024-05-09T08:20:53.166 INFO:teuthology.orchestra.run.smithi157.stdout:(140/156): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-05-09T08:20:53.195 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:20:53.216 INFO:teuthology.orchestra.run.smithi157.stdout:(141/156): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-05-09T08:20:53.227 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 102/164 2024-05-09T08:20:53.263 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T08:20:53.266 INFO:teuthology.orchestra.run.smithi157.stdout:(142/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-05-09T08:20:53.317 INFO:teuthology.orchestra.run.smithi157.stdout:(143/156): python3-werkzeug-2.0.3-3.el9.1.noarc 8.4 MB/s | 427 kB 00:00 2024-05-09T08:20:53.359 INFO:teuthology.orchestra.run.smithi157.stdout:(144/156): python3-zc-lockfile-2.0-10.el9.noarc 478 kB/s | 20 kB 00:00 2024-05-09T08:20:53.409 INFO:teuthology.orchestra.run.smithi157.stdout:(145/156): re2-20211101-3.el9.x86_64.rpm 3.8 MB/s | 192 kB 00:00 2024-05-09T08:20:53.476 INFO:teuthology.orchestra.run.smithi157.stdout:(146/156): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-05-09T08:20:53.741 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T08:20:53.753 INFO:teuthology.orchestra.run.smithi157.stdout:(147/156): python3-numpy-1.20.1-5.el9.x86_64.rp 699 kB/s | 5.1 MB 00:07 2024-05-09T08:20:53.767 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:20:53.837 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:20:53.860 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 106/164 2024-05-09T08:20:53.870 INFO:teuthology.orchestra.run.smithi157.stdout:(148/156): librados2-19.0.0-2515.g96805dcf.el9. 8.6 MB/s | 3.4 MB 00:00 2024-05-09T08:20:54.360 INFO:teuthology.orchestra.run.smithi157.stdout:(149/156): NetworkManager-1.47.5-1.el9.x86_64.r 4.7 MB/s | 2.3 MB 00:00 2024-05-09T08:20:54.411 INFO:teuthology.orchestra.run.smithi157.stdout:(150/156): librbd1-19.0.0-2515.g96805dcf.el9.x8 4.8 MB/s | 3.1 MB 00:00 2024-05-09T08:20:54.578 INFO:teuthology.orchestra.run.smithi157.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 250 kB/s | 42 kB 00:00 2024-05-09T08:20:54.620 INFO:teuthology.orchestra.run.smithi157.stdout:(152/156): NetworkManager-libnm-1.47.5-1.el9.x8 7.1 MB/s | 1.8 MB 00:00 2024-05-09T08:20:54.778 INFO:teuthology.orchestra.run.smithi157.stdout:(153/156): NetworkManager-tui-1.47.5-1.el9.x86_ 1.2 MB/s | 248 kB 00:00 2024-05-09T08:20:54.854 INFO:teuthology.orchestra.run.smithi157.stdout:(154/156): openssl-3.2.1-1.el9.x86_64.rpm 5.8 MB/s | 1.3 MB 00:00 2024-05-09T08:20:55.129 INFO:teuthology.orchestra.run.smithi157.stdout:(155/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 6.8 MB/s | 2.4 MB 00:00 2024-05-09T08:20:56.134 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:20:56.181 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:20:56.733 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:20:56.768 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:20:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:20:56.862 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T08:20:56.862 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T08:20:56.862 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:20:56.902 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:21:03.968 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:21:03.968 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /sys 2024-05-09T08:21:03.968 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /proc 2024-05-09T08:21:03.968 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /mnt 2024-05-09T08:21:03.968 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /var/tmp 2024-05-09T08:21:03.968 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /home 2024-05-09T08:21:03.968 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /root 2024-05-09T08:21:03.969 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /tmp 2024-05-09T08:21:03.969 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:21:04.141 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:21:04.160 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:21:04.161 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:04.161 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T08:21:04.161 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:21:05.479 INFO:teuthology.orchestra.run.smithi157.stdout:(156/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 988 kB/s | 16 MB 00:16 2024-05-09T08:21:05.482 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:21:05.482 INFO:teuthology.orchestra.run.smithi157.stdout:Total 6.1 MB/s | 225 MB 00:37 2024-05-09T08:21:05.636 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:21:05.664 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:21:05.664 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:05.664 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T08:21:05.664 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:21:05.664 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:21:05.664 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:21:05.895 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:21:05.919 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:21:05.919 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:05.919 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T08:21:05.919 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:21:05.919 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:21:05.920 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:21:06.362 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:21:06.386 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:21:06.386 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:06.386 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T08:21:06.386 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:21:06.386 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:21:06.386 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:21:06.492 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T08:21:06.549 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T08:21:06.604 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T08:21:06.703 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-05-09T08:21:06.797 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T08:21:06.846 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T08:21:06.855 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-05-09T08:21:06.855 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-05-09T08:21:06.923 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T08:21:07.011 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T08:21:07.153 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T08:21:07.196 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T08:21:07.239 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T08:21:07.264 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T08:21:07.640 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:21:07.671 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:21:07.702 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 126/164 2024-05-09T08:21:07.764 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 127/164 2024-05-09T08:21:07.794 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:21:08.823 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:21:08.841 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:21:08.841 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2024-05-09T08:21:08.841 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2024-05-09T08:21:08.841 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2024-05-09T08:21:08.841 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2024-05-09T08:21:08.841 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2024-05-09T08:21:08.841 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /root 2024-05-09T08:21:08.841 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2024-05-09T08:21:08.842 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:21:08.862 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:21:09.008 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:21:09.025 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:21:09.025 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:09.025 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T08:21:09.026 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:21:09.110 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-05-09T08:21:09.111 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-05-09T08:21:09.821 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:21:09.855 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:21:09.957 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:21:10.117 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 131/164 2024-05-09T08:21:10.157 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:21:10.180 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:21:10.180 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:10.180 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T08:21:10.180 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:21:10.180 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:21:10.181 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:21:10.232 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:21:10.244 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:21:10.341 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 134/164 2024-05-09T08:21:10.398 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 135/164 2024-05-09T08:21:10.519 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:21:10.546 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:21:10.547 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:10.547 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T08:21:10.547 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:21:10.547 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:21:10.547 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:21:10.761 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:21:10.785 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:21:10.785 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:10.785 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T08:21:10.785 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:21:10.786 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:21:10.786 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:21:11.245 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:21:11.269 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:21:11.269 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:11.269 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T08:21:11.269 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:21:11.269 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:21:11.269 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:21:11.291 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T08:21:11.318 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:21:11.341 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:21:11.341 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:11.342 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T08:21:11.342 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:21:11.342 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:21:11.342 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:21:11.375 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T08:21:11.385 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:21:11.408 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:21:11.408 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:11.408 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T08:21:11.409 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:21:11.423 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T08:21:11.461 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:21:11.478 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T08:21:11.484 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:21:11.484 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:11.484 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T08:21:11.484 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:21:11.484 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:21:11.484 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:21:11.671 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T08:21:11.720 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T08:21:11.787 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:21:11.789 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T08:21:11.810 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:21:11.810 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:11.810 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T08:21:11.810 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:21:11.810 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:21:11.811 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:21:11.878 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T08:21:12.018 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T08:21:12.070 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T08:21:12.106 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T08:21:12.139 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T08:21:12.575 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:21:12.606 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:21:12.637 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 126/164 2024-05-09T08:21:12.698 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 127/164 2024-05-09T08:21:12.729 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:21:13.761 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:21:13.805 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:21:14.769 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:21:14.797 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:21:14.898 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:21:15.061 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 131/164 2024-05-09T08:21:15.099 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:21:15.123 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:21:15.123 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:15.123 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T08:21:15.123 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:21:15.123 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:21:15.123 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:21:15.167 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:21:15.178 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:21:15.202 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-05-09T08:21:15.274 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T08:21:15.275 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 134/164 2024-05-09T08:21:15.312 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T08:21:15.332 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 135/164 2024-05-09T08:21:15.368 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T08:21:15.408 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T08:21:15.434 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T08:21:15.493 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T08:21:15.828 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T08:21:15.935 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:21:16.057 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:21:16.086 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:21:16.177 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:21:16.207 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:21:16.228 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T08:21:16.260 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:21:16.284 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:21:16.284 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:16.284 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T08:21:16.285 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:21:16.285 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:21:16.285 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:21:16.328 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:21:16.340 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 141/164 2024-05-09T08:21:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:21:16.351 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:16.351 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T08:21:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:21:16.381 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 142/164 2024-05-09T08:21:16.392 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T08:21:16.395 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:21:16.418 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:21:16.418 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:16.419 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T08:21:16.419 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:21:16.419 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:21:16.419 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:21:16.422 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 143/164 2024-05-09T08:21:16.454 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T08:21:16.455 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T08:21:16.481 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T08:21:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:21:16.568 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:21:16.591 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:21:16.654 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T08:21:16.682 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 146/164 2024-05-09T08:21:16.684 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:21:16.706 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T08:21:16.720 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:21:16.728 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T08:21:16.744 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:21:16.744 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:16.744 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T08:21:16.745 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:21:16.745 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:21:16.745 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:21:16.758 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 148/164 2024-05-09T08:21:16.787 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T08:21:16.834 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T08:21:16.894 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:21:17.062 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T08:21:17.214 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:21:17.258 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T08:21:17.386 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:21:17.443 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:21:17.464 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:21:17.507 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:21:17.527 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:21:17.649 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T08:21:17.766 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T08:21:17.860 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T08:21:17.894 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-09T08:21:17.934 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-09T08:21:17.998 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-09T08:21:18.033 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-09T08:21:18.076 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-09T08:21:18.118 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-09T08:21:18.145 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-09T08:21:18.213 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-09T08:21:18.267 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-09T08:21:18.298 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-09T08:21:18.342 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-09T08:21:18.360 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 152/164 2024-05-09T08:21:18.418 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-09T08:21:18.444 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T08:21:18.452 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 33/164 2024-05-09T08:21:18.495 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T08:21:18.499 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:21:18.525 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 35/164 2024-05-09T08:21:18.594 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-09T08:21:18.602 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T08:21:18.631 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T08:21:18.648 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-09T08:21:18.657 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T08:21:18.657 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:21:18.722 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-09T08:21:18.733 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:21:18.766 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T08:21:18.766 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:21:18.850 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:21:18.855 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-09T08:21:18.884 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T08:21:18.890 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:21:18.890 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:21:18.908 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-09T08:21:19.069 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:21:19.069 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:21:19.153 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:21:19.218 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:21:19.436 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-09T08:21:19.522 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-09T08:21:19.558 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-09T08:21:19.593 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-09T08:21:19.631 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 45/164 2024-05-09T08:21:19.673 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-09T08:21:19.835 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-09T08:21:19.900 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-09T08:21:19.956 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-09T08:21:20.028 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T08:21:20.059 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T08:21:20.096 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T08:21:20.146 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T08:21:20.178 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T08:21:20.221 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T08:21:20.264 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T08:21:20.400 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T08:21:20.444 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T08:21:20.483 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T08:21:20.525 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T08:21:20.620 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T08:21:20.659 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T08:21:20.708 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T08:21:20.740 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T08:21:20.765 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T08:21:20.854 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T08:21:20.983 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T08:21:21.022 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T08:21:21.056 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T08:21:21.098 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T08:21:21.141 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T08:21:21.172 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T08:21:21.202 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T08:21:21.228 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T08:21:21.261 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T08:21:21.265 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 141/164 2024-05-09T08:21:21.302 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T08:21:21.306 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 142/164 2024-05-09T08:21:21.337 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T08:21:21.351 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 143/164 2024-05-09T08:21:21.379 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T08:21:21.407 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T08:21:21.424 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T08:21:21.476 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T08:21:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T08:21:21.547 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T08:21:21.598 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 146/164 2024-05-09T08:21:21.636 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T08:21:21.645 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T08:21:21.666 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 148/164 2024-05-09T08:21:21.692 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T08:21:21.695 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T08:21:21.742 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T08:21:22.297 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:21:22.339 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:21:22.359 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:21:22.412 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T08:21:22.464 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T08:21:22.498 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T08:21:22.597 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T08:21:22.656 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T08:21:22.744 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T08:21:22.786 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T08:21:22.962 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T08:21:22.989 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T08:21:23.078 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T08:21:23.190 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 152/164 2024-05-09T08:21:23.277 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T08:21:23.330 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T08:21:23.425 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T08:21:23.455 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T08:21:23.489 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T08:21:23.489 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:21:23.573 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:21:23.606 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T08:21:23.606 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:21:23.690 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:21:23.715 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T08:21:23.721 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:21:23.721 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:21:23.730 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T08:21:23.846 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T08:21:23.909 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:21:23.910 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:21:24.002 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:21:24.067 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:21:24.870 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:21:24.870 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 1/164 2024-05-09T08:21:24.870 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2/164 2024-05-09T08:21:24.870 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 3/164 2024-05-09T08:21:24.871 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 4/164 2024-05-09T08:21:24.871 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2515.g96805 5/164 2024-05-09T08:21:24.871 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 6/164 2024-05-09T08:21:24.871 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 7/164 2024-05-09T08:21:24.871 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:21:24.871 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:21:24.871 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:21:24.871 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 11/164 2024-05-09T08:21:24.871 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 12/164 2024-05-09T08:21:24.871 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:21:24.871 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 14/164 2024-05-09T08:21:24.871 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 15/164 2024-05-09T08:21:24.872 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 16/164 2024-05-09T08:21:24.872 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 17/164 2024-05-09T08:21:24.872 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 18/164 2024-05-09T08:21:24.872 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 19/164 2024-05-09T08:21:24.872 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 20/164 2024-05-09T08:21:24.872 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 21/164 2024-05-09T08:21:24.872 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 22/164 2024-05-09T08:21:24.872 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 23/164 2024-05-09T08:21:24.872 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 24/164 2024-05-09T08:21:24.872 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 25/164 2024-05-09T08:21:24.872 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 26/164 2024-05-09T08:21:24.872 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 27/164 2024-05-09T08:21:24.872 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 28/164 2024-05-09T08:21:24.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 29/164 2024-05-09T08:21:24.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 30/164 2024-05-09T08:21:24.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 31/164 2024-05-09T08:21:24.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 32/164 2024-05-09T08:21:24.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 33/164 2024-05-09T08:21:24.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:21:24.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 35/164 2024-05-09T08:21:24.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 36/164 2024-05-09T08:21:24.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 37/164 2024-05-09T08:21:24.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 38/164 2024-05-09T08:21:24.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 39/164 2024-05-09T08:21:24.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T08:21:24.873 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T08:21:24.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T08:21:24.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T08:21:24.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T08:21:24.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T08:21:24.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T08:21:24.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T08:21:24.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T08:21:24.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T08:21:24.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T08:21:24.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T08:21:24.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T08:21:24.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T08:21:24.874 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T08:21:24.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T08:21:24.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T08:21:24.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T08:21:24.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T08:21:24.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T08:21:24.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T08:21:24.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T08:21:24.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T08:21:24.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T08:21:24.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T08:21:24.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T08:21:24.875 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T08:21:24.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T08:21:24.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T08:21:24.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T08:21:24.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T08:21:24.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T08:21:24.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T08:21:24.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T08:21:24.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T08:21:24.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T08:21:24.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T08:21:24.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T08:21:24.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T08:21:24.876 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T08:21:24.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T08:21:24.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T08:21:24.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T08:21:24.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T08:21:24.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T08:21:24.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T08:21:24.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T08:21:24.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T08:21:24.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T08:21:24.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T08:21:24.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T08:21:24.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T08:21:24.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T08:21:24.877 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T08:21:24.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T08:21:24.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T08:21:24.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T08:21:24.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T08:21:24.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T08:21:24.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T08:21:24.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T08:21:24.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T08:21:24.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T08:21:24.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T08:21:24.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T08:21:24.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T08:21:24.878 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T08:21:24.879 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T08:21:24.879 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T08:21:24.879 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T08:21:24.879 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T08:21:24.879 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T08:21:24.879 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T08:21:24.879 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T08:21:24.879 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T08:21:24.879 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T08:21:24.879 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T08:21:24.879 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T08:21:24.879 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T08:21:24.879 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T08:21:24.880 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T08:21:24.880 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T08:21:24.880 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T08:21:24.880 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T08:21:24.880 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T08:21:24.880 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T08:21:24.880 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T08:21:24.880 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T08:21:24.880 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T08:21:24.880 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T08:21:24.880 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T08:21:24.880 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T08:21:24.880 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T08:21:24.880 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T08:21:24.881 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T08:21:24.881 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T08:21:24.881 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T08:21:24.881 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T08:21:24.881 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T08:21:24.881 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T08:21:24.881 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T08:21:24.881 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T08:21:24.881 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T08:21:24.881 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T08:21:24.881 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T08:21:24.881 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T08:21:24.881 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T08:21:24.882 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T08:21:24.882 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T08:21:24.882 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 149/164 2024-05-09T08:21:24.882 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T08:21:24.882 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 151/164 2024-05-09T08:21:24.882 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T08:21:24.882 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T08:21:24.882 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T08:21:24.882 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T08:21:24.882 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T08:21:24.882 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T08:21:24.882 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T08:21:24.882 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T08:21:24.883 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T08:21:24.883 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T08:21:24.883 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T08:21:24.883 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T08:21:25.091 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T08:21:25.177 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T08:21:25.205 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T08:21:25.238 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T08:21:25.558 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T08:21:25.609 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:21:25.699 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T08:21:25.730 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 102/164 2024-05-09T08:21:25.766 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T08:21:26.244 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T08:21:26.270 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:21:26.333 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T08:21:26.355 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 106/164 2024-05-09T08:21:27.557 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:21:27.557 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:21:27.557 INFO:teuthology.orchestra.run.smithi110.stdout:Upgraded: 2024-05-09T08:21:27.557 INFO:teuthology.orchestra.run.smithi110.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T08:21:27.557 INFO:teuthology.orchestra.run.smithi110.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T08:21:27.557 INFO:teuthology.orchestra.run.smithi110.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T08:21:27.557 INFO:teuthology.orchestra.run.smithi110.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T08:21:27.557 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.557 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.557 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T08:21:27.557 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T08:21:27.557 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-05-09T08:21:27.558 INFO:teuthology.orchestra.run.smithi110.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T08:21:27.558 INFO:teuthology.orchestra.run.smithi110.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T08:21:27.558 INFO:teuthology.orchestra.run.smithi110.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T08:21:27.558 INFO:teuthology.orchestra.run.smithi110.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T08:21:27.558 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.558 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.558 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.558 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.558 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:27.558 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.558 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.558 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.558 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:27.559 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:27.559 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:27.559 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:27.559 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:27.559 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.559 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.559 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:27.559 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.559 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.559 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.559 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:27.559 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:27.559 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.560 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.560 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:27.560 INFO:teuthology.orchestra.run.smithi110.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T08:21:27.560 INFO:teuthology.orchestra.run.smithi110.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T08:21:27.560 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T08:21:27.560 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T08:21:27.560 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T08:21:27.560 INFO:teuthology.orchestra.run.smithi110.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T08:21:27.560 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T08:21:27.561 INFO:teuthology.orchestra.run.smithi110.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T08:21:27.561 INFO:teuthology.orchestra.run.smithi110.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T08:21:27.561 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T08:21:27.561 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T08:21:27.561 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.561 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.561 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.562 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T08:21:27.562 INFO:teuthology.orchestra.run.smithi110.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T08:21:27.562 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T08:21:27.562 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T08:21:27.562 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T08:21:27.562 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.562 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.562 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T08:21:27.562 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.562 INFO:teuthology.orchestra.run.smithi110.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T08:21:27.562 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T08:21:27.562 INFO:teuthology.orchestra.run.smithi110.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T08:21:27.562 INFO:teuthology.orchestra.run.smithi110.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T08:21:27.563 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T08:21:27.563 INFO:teuthology.orchestra.run.smithi110.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T08:21:27.563 INFO:teuthology.orchestra.run.smithi110.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T08:21:27.563 INFO:teuthology.orchestra.run.smithi110.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T08:21:27.563 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T08:21:27.563 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T08:21:27.563 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T08:21:27.563 INFO:teuthology.orchestra.run.smithi110.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T08:21:27.563 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T08:21:27.563 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T08:21:27.563 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T08:21:27.563 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T08:21:27.563 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T08:21:27.563 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T08:21:27.564 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T08:21:27.564 INFO:teuthology.orchestra.run.smithi110.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T08:21:27.564 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T08:21:27.564 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T08:21:27.564 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T08:21:27.564 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.564 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.564 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.564 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T08:21:27.564 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T08:21:27.564 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T08:21:27.564 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T08:21:27.565 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T08:21:27.565 INFO:teuthology.orchestra.run.smithi110.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T08:21:27.565 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T08:21:27.565 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T08:21:27.565 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T08:21:27.565 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T08:21:27.565 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T08:21:27.565 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T08:21:27.565 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T08:21:27.565 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T08:21:27.565 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T08:21:27.565 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T08:21:27.565 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T08:21:27.565 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T08:21:27.566 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T08:21:27.566 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T08:21:27.566 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T08:21:27.566 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T08:21:27.566 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T08:21:27.566 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T08:21:27.566 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T08:21:27.566 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T08:21:27.566 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T08:21:27.566 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T08:21:27.566 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T08:21:27.566 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T08:21:27.567 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T08:21:27.567 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T08:21:27.567 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T08:21:27.567 INFO:teuthology.orchestra.run.smithi110.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T08:21:27.567 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T08:21:27.567 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T08:21:27.567 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T08:21:27.567 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T08:21:27.567 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T08:21:27.567 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T08:21:27.567 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.567 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.567 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T08:21:27.567 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T08:21:27.568 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T08:21:27.568 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.568 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T08:21:27.568 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T08:21:27.568 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T08:21:27.568 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T08:21:27.568 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T08:21:27.568 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T08:21:27.568 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T08:21:27.568 INFO:teuthology.orchestra.run.smithi110.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T08:21:27.568 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T08:21:27.568 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T08:21:27.568 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T08:21:27.569 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T08:21:27.569 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T08:21:27.569 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T08:21:27.569 INFO:teuthology.orchestra.run.smithi110.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T08:21:27.569 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.569 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.569 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:27.569 INFO:teuthology.orchestra.run.smithi110.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T08:21:27.569 INFO:teuthology.orchestra.run.smithi110.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T08:21:27.569 INFO:teuthology.orchestra.run.smithi110.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T08:21:27.569 INFO:teuthology.orchestra.run.smithi110.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T08:21:27.569 INFO:teuthology.orchestra.run.smithi110.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T08:21:27.569 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T08:21:27.570 INFO:teuthology.orchestra.run.smithi110.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T08:21:27.570 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-05-09T08:21:27.570 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-05-09T08:21:27.994 DEBUG:teuthology.parallel:result is None 2024-05-09T08:21:28.686 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:21:28.720 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:21:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T08:21:29.307 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:21:29.389 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T08:21:29.389 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T08:21:29.390 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T08:21:29.390 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:21:29.424 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:21:29.623 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:21:29.623 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 1/164 2024-05-09T08:21:29.623 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2/164 2024-05-09T08:21:29.623 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 3/164 2024-05-09T08:21:29.624 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 4/164 2024-05-09T08:21:29.624 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2515.g96805 5/164 2024-05-09T08:21:29.624 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 6/164 2024-05-09T08:21:29.624 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 7/164 2024-05-09T08:21:29.624 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:21:29.624 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:21:29.624 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:21:29.624 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 11/164 2024-05-09T08:21:29.624 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 12/164 2024-05-09T08:21:29.624 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:21:29.624 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 14/164 2024-05-09T08:21:29.624 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 15/164 2024-05-09T08:21:29.624 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 16/164 2024-05-09T08:21:29.625 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 17/164 2024-05-09T08:21:29.625 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 18/164 2024-05-09T08:21:29.625 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 19/164 2024-05-09T08:21:29.625 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 20/164 2024-05-09T08:21:29.625 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 21/164 2024-05-09T08:21:29.625 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 22/164 2024-05-09T08:21:29.625 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 23/164 2024-05-09T08:21:29.625 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 24/164 2024-05-09T08:21:29.625 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 25/164 2024-05-09T08:21:29.625 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 26/164 2024-05-09T08:21:29.625 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 27/164 2024-05-09T08:21:29.638 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 28/164 2024-05-09T08:21:29.639 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 29/164 2024-05-09T08:21:29.639 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 30/164 2024-05-09T08:21:29.639 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 31/164 2024-05-09T08:21:29.639 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 32/164 2024-05-09T08:21:29.639 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 33/164 2024-05-09T08:21:29.639 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:21:29.639 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 35/164 2024-05-09T08:21:29.639 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 36/164 2024-05-09T08:21:29.639 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 37/164 2024-05-09T08:21:29.639 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 38/164 2024-05-09T08:21:29.639 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 39/164 2024-05-09T08:21:29.639 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T08:21:29.640 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T08:21:29.640 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T08:21:29.640 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T08:21:29.640 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T08:21:29.640 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T08:21:29.640 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T08:21:29.640 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T08:21:29.640 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T08:21:29.640 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T08:21:29.640 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T08:21:29.640 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T08:21:29.640 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T08:21:29.640 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T08:21:29.640 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T08:21:29.641 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T08:21:29.641 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T08:21:29.641 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T08:21:29.641 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T08:21:29.641 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T08:21:29.641 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T08:21:29.641 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T08:21:29.641 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T08:21:29.641 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T08:21:29.641 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T08:21:29.642 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T08:21:29.642 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T08:21:29.642 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T08:21:29.642 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T08:21:29.642 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T08:21:29.642 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T08:21:29.642 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T08:21:29.642 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T08:21:29.642 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T08:21:29.642 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T08:21:29.643 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T08:21:29.643 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T08:21:29.643 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T08:21:29.643 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T08:21:29.643 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T08:21:29.643 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T08:21:29.643 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T08:21:29.643 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T08:21:29.643 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T08:21:29.643 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T08:21:29.643 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T08:21:29.643 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T08:21:29.643 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T08:21:29.643 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T08:21:29.644 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T08:21:29.644 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T08:21:29.644 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T08:21:29.644 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T08:21:29.644 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T08:21:29.644 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T08:21:29.644 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T08:21:29.644 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T08:21:29.644 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T08:21:29.644 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T08:21:29.644 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T08:21:29.644 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T08:21:29.644 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T08:21:29.645 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T08:21:29.645 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T08:21:29.645 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T08:21:29.645 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T08:21:29.645 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T08:21:29.645 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T08:21:29.645 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T08:21:29.645 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T08:21:29.645 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T08:21:29.645 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T08:21:29.645 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T08:21:29.645 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T08:21:29.645 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T08:21:29.645 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T08:21:29.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T08:21:29.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T08:21:29.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T08:21:29.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T08:21:29.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T08:21:29.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T08:21:29.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T08:21:29.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T08:21:29.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T08:21:29.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T08:21:29.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T08:21:29.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T08:21:29.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T08:21:29.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T08:21:29.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T08:21:29.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T08:21:29.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T08:21:29.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T08:21:29.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T08:21:29.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T08:21:29.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T08:21:29.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 149/164 2024-05-09T08:21:29.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T08:21:29.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 151/164 2024-05-09T08:21:29.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T08:21:29.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T08:21:29.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T08:21:29.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T08:21:29.649 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T08:21:29.649 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T08:21:29.649 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T08:21:29.649 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T08:21:29.649 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T08:21:29.649 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T08:21:29.649 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T08:21:29.649 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T08:21:32.004 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:21:32.004 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:21:32.004 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2024-05-09T08:21:32.004 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T08:21:32.004 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T08:21:32.004 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T08:21:32.004 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T08:21:32.004 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.005 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.005 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T08:21:32.005 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T08:21:32.005 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-05-09T08:21:32.005 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T08:21:32.005 INFO:teuthology.orchestra.run.smithi107.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T08:21:32.005 INFO:teuthology.orchestra.run.smithi107.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T08:21:32.005 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T08:21:32.005 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.005 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.005 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.005 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.005 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:32.005 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.006 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.006 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.006 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:32.006 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:32.006 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:32.006 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:32.006 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:32.006 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.006 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.006 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:32.006 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.006 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.006 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.007 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:32.007 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:32.007 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.007 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.007 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:21:32.007 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T08:21:32.007 INFO:teuthology.orchestra.run.smithi107.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T08:21:32.007 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T08:21:32.007 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T08:21:32.007 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T08:21:32.007 INFO:teuthology.orchestra.run.smithi107.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T08:21:32.007 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T08:21:32.007 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T08:21:32.008 INFO:teuthology.orchestra.run.smithi107.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T08:21:32.008 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T08:21:32.008 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T08:21:32.008 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.008 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.008 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.008 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T08:21:32.008 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T08:21:32.008 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T08:21:32.008 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T08:21:32.008 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T08:21:32.008 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.008 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.009 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T08:21:32.009 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.009 INFO:teuthology.orchestra.run.smithi107.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T08:21:32.009 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T08:21:32.009 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T08:21:32.009 INFO:teuthology.orchestra.run.smithi107.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T08:21:32.009 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T08:21:32.009 INFO:teuthology.orchestra.run.smithi107.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T08:21:32.009 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T08:21:32.009 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T08:21:32.009 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T08:21:32.009 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T08:21:32.009 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T08:21:32.009 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T08:21:32.010 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T08:21:32.010 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T08:21:32.010 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T08:21:32.010 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T08:21:32.010 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T08:21:32.010 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T08:21:32.010 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T08:21:32.010 INFO:teuthology.orchestra.run.smithi107.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T08:21:32.010 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T08:21:32.010 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T08:21:32.010 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T08:21:32.010 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.010 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.011 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.011 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T08:21:32.011 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T08:21:32.011 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T08:21:32.011 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T08:21:32.011 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T08:21:32.011 INFO:teuthology.orchestra.run.smithi107.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T08:21:32.011 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T08:21:32.011 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T08:21:32.011 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T08:21:32.011 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T08:21:32.011 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T08:21:32.011 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T08:21:32.012 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T08:21:32.012 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T08:21:32.012 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T08:21:32.012 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T08:21:32.012 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T08:21:32.012 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T08:21:32.012 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T08:21:32.012 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T08:21:32.012 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T08:21:32.012 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T08:21:32.012 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T08:21:32.012 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T08:21:32.012 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T08:21:32.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T08:21:32.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T08:21:32.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T08:21:32.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T08:21:32.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T08:21:32.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T08:21:32.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T08:21:32.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T08:21:32.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T08:21:32.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T08:21:32.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T08:21:32.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T08:21:32.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T08:21:32.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T08:21:32.014 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T08:21:32.014 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.014 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.014 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T08:21:32.014 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T08:21:32.014 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T08:21:32.014 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.014 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T08:21:32.014 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T08:21:32.014 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T08:21:32.014 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T08:21:32.015 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T08:21:32.015 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T08:21:32.015 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T08:21:32.015 INFO:teuthology.orchestra.run.smithi107.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T08:21:32.015 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T08:21:32.015 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T08:21:32.015 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T08:21:32.015 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T08:21:32.015 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T08:21:32.015 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T08:21:32.015 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T08:21:32.015 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.015 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.016 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:21:32.016 INFO:teuthology.orchestra.run.smithi107.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T08:21:32.016 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T08:21:32.016 INFO:teuthology.orchestra.run.smithi107.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T08:21:32.016 INFO:teuthology.orchestra.run.smithi107.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T08:21:32.016 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T08:21:32.016 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T08:21:32.016 INFO:teuthology.orchestra.run.smithi107.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T08:21:32.017 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:21:32.017 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-05-09T08:21:32.442 DEBUG:teuthology.parallel:result is None 2024-05-09T08:21:41.194 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T08:21:41.194 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /sys 2024-05-09T08:21:41.194 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /proc 2024-05-09T08:21:41.195 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /mnt 2024-05-09T08:21:41.195 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /var/tmp 2024-05-09T08:21:41.195 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /home 2024-05-09T08:21:41.195 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /root 2024-05-09T08:21:41.195 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /tmp 2024-05-09T08:21:41.195 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:21:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:21:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T08:21:41.391 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:41.391 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T08:21:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:21:42.885 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:21:42.913 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T08:21:42.913 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:42.913 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T08:21:42.914 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:21:42.914 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T08:21:42.914 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:21:43.136 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:21:43.159 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T08:21:43.160 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:43.160 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T08:21:43.160 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:21:43.160 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T08:21:43.160 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:21:43.611 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:21:43.635 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T08:21:43.635 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:43.635 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T08:21:43.636 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:21:43.636 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T08:21:43.636 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:21:43.758 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T08:21:43.815 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T08:21:43.870 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T08:21:44.055 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T08:21:44.104 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T08:21:44.181 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T08:21:44.278 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T08:21:44.409 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T08:21:44.462 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T08:21:44.501 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T08:21:44.530 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T08:21:44.980 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:21:45.004 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T08:21:45.035 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 126/164 2024-05-09T08:21:45.088 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 127/164 2024-05-09T08:21:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:21:46.162 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T08:21:46.204 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:21:47.175 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T08:21:47.212 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:21:47.317 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T08:21:47.489 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 131/164 2024-05-09T08:21:47.514 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:21:47.539 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T08:21:47.539 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:47.539 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T08:21:47.539 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:21:47.539 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T08:21:47.539 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:21:47.590 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:21:47.601 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T08:21:47.698 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 134/164 2024-05-09T08:21:47.752 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 135/164 2024-05-09T08:21:48.653 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T08:21:48.684 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:21:48.708 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T08:21:48.708 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:48.708 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T08:21:48.708 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:21:48.708 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T08:21:48.708 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:21:48.751 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:21:48.775 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T08:21:48.775 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:48.775 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T08:21:48.775 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:21:48.827 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:21:48.851 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T08:21:48.851 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:48.851 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T08:21:48.851 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:21:48.851 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T08:21:48.851 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:21:49.153 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:21:49.176 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T08:21:49.177 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T08:21:49.177 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T08:21:49.177 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:21:49.177 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T08:21:49.177 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:21:53.748 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 141/164 2024-05-09T08:21:53.790 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 142/164 2024-05-09T08:21:53.838 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 143/164 2024-05-09T08:21:53.870 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T08:21:53.898 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T08:21:54.095 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 146/164 2024-05-09T08:21:54.136 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T08:21:54.166 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 148/164 2024-05-09T08:21:54.195 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T08:21:54.233 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T08:21:54.834 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:21:54.882 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:21:54.902 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T08:21:55.740 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 152/164 2024-05-09T08:21:55.828 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T08:21:55.881 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T08:21:55.977 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T08:21:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T08:21:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T08:21:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:21:56.108 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T08:21:56.133 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T08:21:56.133 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:21:56.217 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T08:21:56.242 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T08:21:56.248 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:21:56.248 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:21:56.427 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T08:21:56.428 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:21:56.504 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T08:21:56.569 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:22:02.028 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:22:02.028 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 1/164 2024-05-09T08:22:02.028 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2/164 2024-05-09T08:22:02.028 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 3/164 2024-05-09T08:22:02.028 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 4/164 2024-05-09T08:22:02.028 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2515.g96805 5/164 2024-05-09T08:22:02.028 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 6/164 2024-05-09T08:22:02.028 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 7/164 2024-05-09T08:22:02.028 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T08:22:02.028 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T08:22:02.028 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T08:22:02.029 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 11/164 2024-05-09T08:22:02.029 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 12/164 2024-05-09T08:22:02.029 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T08:22:02.029 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 14/164 2024-05-09T08:22:02.029 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 15/164 2024-05-09T08:22:02.029 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 16/164 2024-05-09T08:22:02.029 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 17/164 2024-05-09T08:22:02.029 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 18/164 2024-05-09T08:22:02.029 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 19/164 2024-05-09T08:22:02.029 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 20/164 2024-05-09T08:22:02.029 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 21/164 2024-05-09T08:22:02.029 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 22/164 2024-05-09T08:22:02.030 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 23/164 2024-05-09T08:22:02.030 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 24/164 2024-05-09T08:22:02.030 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 25/164 2024-05-09T08:22:02.030 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 26/164 2024-05-09T08:22:02.030 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 27/164 2024-05-09T08:22:02.030 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 28/164 2024-05-09T08:22:02.044 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 29/164 2024-05-09T08:22:02.044 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 30/164 2024-05-09T08:22:02.044 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 31/164 2024-05-09T08:22:02.044 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 32/164 2024-05-09T08:22:02.044 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 33/164 2024-05-09T08:22:02.044 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T08:22:02.044 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 35/164 2024-05-09T08:22:02.044 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 36/164 2024-05-09T08:22:02.044 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 37/164 2024-05-09T08:22:02.044 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 38/164 2024-05-09T08:22:02.045 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 39/164 2024-05-09T08:22:02.045 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T08:22:02.045 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T08:22:02.045 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T08:22:02.045 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T08:22:02.045 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T08:22:02.045 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T08:22:02.045 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T08:22:02.045 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T08:22:02.045 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T08:22:02.045 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T08:22:02.045 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T08:22:02.046 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T08:22:02.046 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T08:22:02.046 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T08:22:02.046 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T08:22:02.046 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T08:22:02.046 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T08:22:02.046 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T08:22:02.046 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T08:22:02.046 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T08:22:02.046 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T08:22:02.046 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T08:22:02.046 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T08:22:02.046 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T08:22:02.047 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T08:22:02.049 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T08:22:02.049 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T08:22:02.049 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T08:22:02.049 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T08:22:02.049 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T08:22:02.050 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T08:22:02.050 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T08:22:02.050 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T08:22:02.050 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T08:22:02.050 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T08:22:02.050 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T08:22:02.050 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T08:22:02.050 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T08:22:02.050 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T08:22:02.050 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T08:22:02.050 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T08:22:02.050 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T08:22:02.051 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T08:22:02.051 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T08:22:02.051 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T08:22:02.051 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T08:22:02.051 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T08:22:02.051 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T08:22:02.051 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T08:22:02.051 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T08:22:02.051 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T08:22:02.051 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T08:22:02.051 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T08:22:02.051 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T08:22:02.051 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T08:22:02.052 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T08:22:02.052 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T08:22:02.052 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T08:22:02.052 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T08:22:02.052 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T08:22:02.052 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T08:22:02.052 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T08:22:02.052 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T08:22:02.052 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T08:22:02.052 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T08:22:02.052 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T08:22:02.052 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T08:22:02.052 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T08:22:02.053 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T08:22:02.053 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T08:22:02.053 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T08:22:02.053 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T08:22:02.053 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T08:22:02.053 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T08:22:02.053 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T08:22:02.053 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T08:22:02.053 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T08:22:02.053 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T08:22:02.053 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T08:22:02.053 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T08:22:02.053 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T08:22:02.054 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T08:22:02.054 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T08:22:02.054 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T08:22:02.054 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T08:22:02.054 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T08:22:02.054 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T08:22:02.054 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T08:22:02.054 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T08:22:02.054 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T08:22:02.054 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T08:22:02.054 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T08:22:02.054 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T08:22:02.054 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T08:22:02.055 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T08:22:02.055 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T08:22:02.055 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T08:22:02.055 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T08:22:02.055 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T08:22:02.055 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T08:22:02.055 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T08:22:02.055 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T08:22:02.055 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T08:22:02.055 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T08:22:02.055 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T08:22:02.055 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T08:22:02.055 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T08:22:02.055 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T08:22:02.056 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T08:22:02.056 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 149/164 2024-05-09T08:22:02.056 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T08:22:02.056 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 151/164 2024-05-09T08:22:02.056 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T08:22:02.056 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T08:22:02.056 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T08:22:02.056 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T08:22:02.056 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T08:22:02.056 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T08:22:02.056 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T08:22:02.056 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T08:22:02.056 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T08:22:02.057 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T08:22:02.057 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T08:22:02.057 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T08:22:04.881 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T08:22:04.881 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:22:04.882 INFO:teuthology.orchestra.run.smithi157.stdout:Upgraded: 2024-05-09T08:22:04.882 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T08:22:04.882 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T08:22:04.882 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T08:22:04.882 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T08:22:04.882 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.882 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.882 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T08:22:04.882 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T08:22:04.882 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-05-09T08:22:04.882 INFO:teuthology.orchestra.run.smithi157.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T08:22:04.882 INFO:teuthology.orchestra.run.smithi157.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T08:22:04.883 INFO:teuthology.orchestra.run.smithi157.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T08:22:04.883 INFO:teuthology.orchestra.run.smithi157.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T08:22:04.883 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.883 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.883 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.883 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.883 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:22:04.883 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.883 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.883 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.883 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:22:04.883 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:22:04.883 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:22:04.884 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:22:04.884 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:22:04.884 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.884 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.884 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:22:04.884 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.884 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.884 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.884 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:22:04.884 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:22:04.884 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.884 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.884 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T08:22:04.885 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T08:22:04.885 INFO:teuthology.orchestra.run.smithi157.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T08:22:04.885 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T08:22:04.885 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T08:22:04.885 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T08:22:04.885 INFO:teuthology.orchestra.run.smithi157.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T08:22:04.885 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T08:22:04.885 INFO:teuthology.orchestra.run.smithi157.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T08:22:04.885 INFO:teuthology.orchestra.run.smithi157.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T08:22:04.885 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T08:22:04.885 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T08:22:04.885 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.885 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.886 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.886 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T08:22:04.886 INFO:teuthology.orchestra.run.smithi157.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T08:22:04.886 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T08:22:04.886 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T08:22:04.886 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T08:22:04.886 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.886 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.886 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T08:22:04.886 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.886 INFO:teuthology.orchestra.run.smithi157.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T08:22:04.886 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T08:22:04.887 INFO:teuthology.orchestra.run.smithi157.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T08:22:04.887 INFO:teuthology.orchestra.run.smithi157.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T08:22:04.887 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T08:22:04.887 INFO:teuthology.orchestra.run.smithi157.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T08:22:04.887 INFO:teuthology.orchestra.run.smithi157.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T08:22:04.887 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T08:22:04.887 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T08:22:04.887 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T08:22:04.887 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T08:22:04.887 INFO:teuthology.orchestra.run.smithi157.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T08:22:04.887 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T08:22:04.887 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T08:22:04.887 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T08:22:04.887 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T08:22:04.888 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T08:22:04.888 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T08:22:04.888 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T08:22:04.888 INFO:teuthology.orchestra.run.smithi157.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T08:22:04.888 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T08:22:04.888 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T08:22:04.888 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T08:22:04.888 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.888 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.888 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.888 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T08:22:04.888 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T08:22:04.888 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T08:22:04.889 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T08:22:04.889 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T08:22:04.889 INFO:teuthology.orchestra.run.smithi157.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T08:22:04.889 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T08:22:04.889 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T08:22:04.889 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T08:22:04.889 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T08:22:04.889 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T08:22:04.889 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T08:22:04.889 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T08:22:04.889 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T08:22:04.889 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T08:22:04.889 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T08:22:04.890 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T08:22:04.890 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T08:22:04.890 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T08:22:04.890 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T08:22:04.890 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T08:22:04.890 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T08:22:04.890 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T08:22:04.890 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T08:22:04.890 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T08:22:04.890 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T08:22:04.890 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T08:22:04.890 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T08:22:04.890 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T08:22:04.891 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T08:22:04.891 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T08:22:04.891 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T08:22:04.891 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T08:22:04.891 INFO:teuthology.orchestra.run.smithi157.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T08:22:04.891 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T08:22:04.891 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T08:22:04.891 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T08:22:04.891 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T08:22:04.891 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T08:22:04.891 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T08:22:04.891 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.891 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.891 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T08:22:04.892 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T08:22:04.892 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T08:22:04.892 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.892 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T08:22:04.892 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T08:22:04.892 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T08:22:04.892 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T08:22:04.892 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T08:22:04.892 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T08:22:04.892 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T08:22:04.892 INFO:teuthology.orchestra.run.smithi157.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T08:22:04.892 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T08:22:04.892 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T08:22:04.892 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T08:22:04.893 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T08:22:04.893 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T08:22:04.893 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T08:22:04.893 INFO:teuthology.orchestra.run.smithi157.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T08:22:04.893 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.893 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.893 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T08:22:04.893 INFO:teuthology.orchestra.run.smithi157.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T08:22:04.893 INFO:teuthology.orchestra.run.smithi157.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T08:22:04.893 INFO:teuthology.orchestra.run.smithi157.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T08:22:04.893 INFO:teuthology.orchestra.run.smithi157.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T08:22:04.893 INFO:teuthology.orchestra.run.smithi157.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T08:22:04.893 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T08:22:04.894 INFO:teuthology.orchestra.run.smithi157.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T08:22:04.894 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:22:04.894 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-05-09T08:22:05.346 DEBUG:teuthology.parallel:result is None 2024-05-09T08:22:05.346 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:22:05.636 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T08:22:05.663 INFO:teuthology.orchestra.run.smithi107.stdout:19.0.0-2515.g96805dcf.el9 2024-05-09T08:22:05.663 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2515.g96805dcf.el9 2024-05-09T08:22:05.663 INFO:teuthology.task.install:The correct ceph version 19.0.0-2515.g96805dcf is installed. 2024-05-09T08:22:05.665 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:22:05.971 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T08:22:05.997 INFO:teuthology.orchestra.run.smithi110.stdout:19.0.0-2515.g96805dcf.el9 2024-05-09T08:22:05.998 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2515.g96805dcf.el9 2024-05-09T08:22:05.998 INFO:teuthology.task.install:The correct ceph version 19.0.0-2515.g96805dcf is installed. 2024-05-09T08:22:05.999 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:22:06.143 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T08:22:06.170 INFO:teuthology.orchestra.run.smithi157.stdout:19.0.0-2515.g96805dcf.el9 2024-05-09T08:22:06.171 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2515.g96805dcf.el9 2024-05-09T08:22:06.171 INFO:teuthology.task.install:The correct ceph version 19.0.0-2515.g96805dcf is installed. 2024-05-09T08:22:06.173 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-09T08:22:06.173 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:22:06.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T08:22:06.206 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-09T08:22:06.206 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T08:22:06.238 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-05-09T08:22:06.239 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T08:22:06.272 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-09T08:22:06.272 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:22:06.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T08:22:06.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T08:22:06.371 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-09T08:22:06.371 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T08:22:06.401 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T08:22:06.470 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-05-09T08:22:06.470 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T08:22:06.500 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T08:22:06.570 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-09T08:22:06.570 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:22:06.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T08:22:06.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T08:22:06.664 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-09T08:22:06.664 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T08:22:06.694 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T08:22:06.761 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-05-09T08:22:06.761 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T08:22:06.795 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T08:22:06.864 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-09T08:22:06.864 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:22:06.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T08:22:06.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T08:22:06.957 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-09T08:22:06.957 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T08:22:06.983 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T08:22:07.049 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-05-09T08:22:07.049 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T08:22:07.079 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T08:22:07.150 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-09T08:22:07.255 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'subvols': {'create': 2}} 2024-05-09T08:22:07.256 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:22:07.256 INFO:tasks.cephadm:Cluster fsid is 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:22:07.256 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-09T08:22:07.257 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.107', 'mon.b': '172.21.15.110', 'mon.c': '172.21.15.157'} 2024-05-09T08:22:07.257 INFO:tasks.cephadm:First mon is mon.a on smithi107 2024-05-09T08:22:07.257 INFO:tasks.cephadm:First mgr is x 2024-05-09T08:22:07.257 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-09T08:22:07.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname $(hostname -s) 2024-05-09T08:22:07.283 DEBUG:teuthology.orchestra.run.smithi110:> sudo hostname $(hostname -s) 2024-05-09T08:22:07.309 DEBUG:teuthology.orchestra.run.smithi157:> sudo hostname $(hostname -s) 2024-05-09T08:22:07.340 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-09T08:22:07.340 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:22:07.637 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55097/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2515-g96805dcf', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2515.g96805dcf'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-07 19:00:25.721799', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-05-09T08:22:07.923 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:22:07.925 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/x86_64/flavors/default/cephadm 2024-05-09T08:22:07.925 DEBUG:teuthology.orchestra.run.smithi107:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-09T08:22:08.713 INFO:teuthology.orchestra.run.smithi107.stdout:-rw-r--r--. 1 ubuntu ubuntu 782039 May 9 08:22 /home/ubuntu/cephtest/cephadm 2024-05-09T08:22:08.713 DEBUG:teuthology.orchestra.run.smithi110:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-09T08:22:09.465 INFO:teuthology.orchestra.run.smithi110.stdout:-rw-r--r--. 1 ubuntu ubuntu 782039 May 9 08:22 /home/ubuntu/cephtest/cephadm 2024-05-09T08:22:09.465 DEBUG:teuthology.orchestra.run.smithi157:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-09T08:22:09.888 INFO:teuthology.orchestra.run.smithi157.stdout:-rw-r--r--. 1 ubuntu ubuntu 782039 May 9 08:22 /home/ubuntu/cephtest/cephadm 2024-05-09T08:22:09.888 DEBUG:teuthology.orchestra.run.smithi107:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-09T08:22:09.908 DEBUG:teuthology.orchestra.run.smithi110:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-09T08:22:09.928 DEBUG:teuthology.orchestra.run.smithi157:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-09T08:22:09.963 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc on all hosts... 2024-05-09T08:22:09.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc pull 2024-05-09T08:22:09.966 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc pull 2024-05-09T08:22:09.971 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc pull 2024-05-09T08:22:11.471 INFO:teuthology.orchestra.run.smithi107.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T08:22:11.647 INFO:teuthology.orchestra.run.smithi110.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T08:22:11.675 INFO:teuthology.orchestra.run.smithi157.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T08:22:40.102 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-05-09T08:22:40.102 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T08:22:40.102 INFO:teuthology.orchestra.run.smithi110.stdout: "image_id": "d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662", 2024-05-09T08:22:40.102 INFO:teuthology.orchestra.run.smithi110.stdout: "repo_digests": [ 2024-05-09T08:22:40.103 INFO:teuthology.orchestra.run.smithi110.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac" 2024-05-09T08:22:40.103 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-05-09T08:22:40.103 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-05-09T08:22:44.791 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:22:44.791 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T08:22:44.791 INFO:teuthology.orchestra.run.smithi107.stdout: "image_id": "d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662", 2024-05-09T08:22:44.791 INFO:teuthology.orchestra.run.smithi107.stdout: "repo_digests": [ 2024-05-09T08:22:44.791 INFO:teuthology.orchestra.run.smithi107.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac" 2024-05-09T08:22:44.791 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-05-09T08:22:44.791 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:22:47.868 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-05-09T08:22:47.869 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T08:22:47.869 INFO:teuthology.orchestra.run.smithi157.stdout: "image_id": "d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662", 2024-05-09T08:22:47.869 INFO:teuthology.orchestra.run.smithi157.stdout: "repo_digests": [ 2024-05-09T08:22:47.869 INFO:teuthology.orchestra.run.smithi157.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac" 2024-05-09T08:22:47.869 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-05-09T08:22:47.869 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-05-09T08:22:47.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph 2024-05-09T08:22:47.928 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph 2024-05-09T08:22:47.962 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /etc/ceph 2024-05-09T08:22:47.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /etc/ceph 2024-05-09T08:22:48.022 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /etc/ceph 2024-05-09T08:22:48.050 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 777 /etc/ceph 2024-05-09T08:22:48.076 INFO:tasks.cephadm:Writing seed config... 2024-05-09T08:22:48.077 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-09T08:22:48.077 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-09T08:22:48.077 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-09T08:22:48.077 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-09T08:22:48.077 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-09T08:22:48.077 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-09T08:22:48.077 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-09T08:22:48.077 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-09T08:22:48.077 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-05-09T08:22:48.077 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-05-09T08:22:48.077 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-05-09T08:22:48.078 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-05-09T08:22:48.078 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-09T08:22:48.078 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-09T08:22:48.078 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-09T08:22:48.078 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-09T08:22:48.078 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-05-09T08:22:48.078 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-05-09T08:22:48.078 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-09T08:22:48.078 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-09T08:22:48.078 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-09T08:22:48.078 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-05-09T08:22:48.078 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-05-09T08:22:48.078 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-09T08:22:48.078 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-09T08:22:48.079 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-09T08:22:48.079 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-09T08:22:48.079 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-09T08:22:48.079 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-09T08:22:48.079 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-09T08:22:48.079 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-09T08:22:48.079 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-09T08:22:48.079 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-09T08:22:48.079 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-05-09T08:22:48.079 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-05-09T08:22:48.079 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-09T08:22:48.079 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-09T08:22:48.079 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-05-09T08:22:48.079 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-05-09T08:22:48.079 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-09T08:22:48.080 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-09T08:22:48.080 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-09T08:22:48.080 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-09T08:22:48.080 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-05-09T08:22:48.080 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-09T08:22:48.080 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-09T08:22:48.080 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-09T08:22:48.080 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-05-09T08:22:48.081 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:22:48.081 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-09T08:22:48.100 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3954c88c-0ddd-11ef-bc98-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-09T08:22:48.100 DEBUG:teuthology.orchestra.run.smithi107:mon.a> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.a.service 2024-05-09T08:22:48.145 DEBUG:teuthology.orchestra.run.smithi107:mgr.x> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mgr.x.service 2024-05-09T08:22:48.189 INFO:tasks.cephadm:Bootstrapping... 2024-05-09T08:22:48.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc -v bootstrap --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.107 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:22:48.382 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:22:48.383 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', '-v', 'bootstrap', '--fsid', '3954c88c-0ddd-11ef-bc98-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.107', '--skip-admin-label'] 2024-05-09T08:22:48.383 INFO:teuthology.orchestra.run.smithi107.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-09T08:22:48.383 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying podman|docker is present... 2024-05-09T08:22:48.417 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-09T08:22:48.418 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying lvm2 is present... 2024-05-09T08:22:48.418 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying time synchronization is in place... 2024-05-09T08:22:48.425 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-09T08:22:48.426 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-09T08:22:48.432 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-09T08:22:48.432 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-05-09T08:22:48.438 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout enabled 2024-05-09T08:22:48.444 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout active 2024-05-09T08:22:48.444 INFO:teuthology.orchestra.run.smithi107.stdout:Unit chronyd.service is enabled and running 2024-05-09T08:22:48.445 INFO:teuthology.orchestra.run.smithi107.stdout:Repeating the final host check... 2024-05-09T08:22:48.471 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-09T08:22:48.471 INFO:teuthology.orchestra.run.smithi107.stdout:podman (/bin/podman) version 4.9.4 is present 2024-05-09T08:22:48.471 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl is present 2024-05-09T08:22:48.471 INFO:teuthology.orchestra.run.smithi107.stdout:lvcreate is present 2024-05-09T08:22:48.477 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-09T08:22:48.477 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-09T08:22:48.483 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-09T08:22:48.483 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-05-09T08:22:48.490 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout enabled 2024-05-09T08:22:48.497 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout active 2024-05-09T08:22:48.497 INFO:teuthology.orchestra.run.smithi107.stdout:Unit chronyd.service is enabled and running 2024-05-09T08:22:48.497 INFO:teuthology.orchestra.run.smithi107.stdout:Host looks OK 2024-05-09T08:22:48.497 INFO:teuthology.orchestra.run.smithi107.stdout:Cluster fsid: 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:22:48.497 INFO:teuthology.orchestra.run.smithi107.stdout:Acquiring lock 140598228283344 on /run/cephadm/3954c88c-0ddd-11ef-bc98-c7b262605968.lock 2024-05-09T08:22:48.497 INFO:teuthology.orchestra.run.smithi107.stdout:Lock 140598228283344 acquired on /run/cephadm/3954c88c-0ddd-11ef-bc98-c7b262605968.lock 2024-05-09T08:22:48.497 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying IP 172.21.15.107 port 3300 ... 2024-05-09T08:22:48.498 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying IP 172.21.15.107 port 6789 ... 2024-05-09T08:22:48.498 INFO:teuthology.orchestra.run.smithi107.stdout:Base mon IP(s) is [172.21.15.107:3300, 172.21.15.107:6789], mon addrv is [v2:172.21.15.107:3300,v1:172.21.15.107:6789] 2024-05-09T08:22:48.501 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.107 metric 100 2024-05-09T08:22:48.501 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.107 metric 100 2024-05-09T08:22:48.507 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-09T08:22:48.507 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-05-09T08:22:48.507 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1786sec hoplimit 64 pref medium 2024-05-09T08:22:48.507 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-09T08:22:48.507 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-09T08:22:48.507 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-09T08:22:48.507 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-09T08:22:48.507 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2024-05-09T08:22:48.507 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-09T08:22:48.507 INFO:teuthology.orchestra.run.smithi107.stdout:Mon IP `172.21.15.107` is in CIDR network `172.21.0.0/20` 2024-05-09T08:22:48.507 INFO:teuthology.orchestra.run.smithi107.stdout:Mon IP `172.21.15.107` is in CIDR network `172.21.0.0/20` 2024-05-09T08:22:48.508 INFO:teuthology.orchestra.run.smithi107.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-09T08:22:48.508 INFO:teuthology.orchestra.run.smithi107.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-09T08:22:48.508 INFO:teuthology.orchestra.run.smithi107.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T08:22:48.984 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 2024-05-09T08:22:48.984 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T08:22:48.984 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Getting image source signatures 2024-05-09T08:22:48.984 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Copying blob sha256:bf5b95bd68251da39c6cd2eb96c6ba4138bb37a62bdc39287908b25e92047d7e 2024-05-09T08:22:48.984 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Copying blob sha256:9852312f87792557815c3e860a5c4f8cbf2b59bbbde9b5f86ab07b438f905b2d 2024-05-09T08:22:48.984 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Copying config sha256:d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 2024-05-09T08:22:48.984 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-09T08:22:50.007 INFO:teuthology.orchestra.run.smithi107.stdout:ceph: stdout ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:22:50.008 INFO:teuthology.orchestra.run.smithi107.stdout:Ceph version: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:22:50.008 INFO:teuthology.orchestra.run.smithi107.stdout:Extracting ceph user uid/gid from container image... 2024-05-09T08:22:51.016 INFO:teuthology.orchestra.run.smithi107.stdout:stat: stdout 167 167 2024-05-09T08:22:51.016 INFO:teuthology.orchestra.run.smithi107.stdout:Creating initial keys... 2024-05-09T08:22:51.890 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph-authtool: stdout AQDbhzxmZ678ExAAqQ1eU0Ze1ENXYDDHvaKIng== 2024-05-09T08:22:52.815 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph-authtool: stdout AQDchzxmy55rCxAA6DkfDEn9bIEK8Vh3P+XwEQ== 2024-05-09T08:22:53.770 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph-authtool: stdout AQDdhzxmFz73CBAA2zJvfOnDDH4SK3XIEdUf5Q== 2024-05-09T08:22:53.772 INFO:teuthology.orchestra.run.smithi107.stdout:Creating initial monmap... 2024-05-09T08:22:54.694 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-09T08:22:54.694 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-09T08:22:54.694 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:22:54.695 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-09T08:22:54.695 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool for a [v2:172.21.15.107:3300,v1:172.21.15.107:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-09T08:22:54.695 INFO:teuthology.orchestra.run.smithi107.stdout:setting min_mon_release = quincy 2024-05-09T08:22:54.695 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: set fsid to 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:22:54.695 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-09T08:22:54.695 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:22:54.695 INFO:teuthology.orchestra.run.smithi107.stdout:Creating mon... 2024-05-09T08:22:55.626 INFO:teuthology.orchestra.run.smithi107.stdout:create mon.a on 2024-05-09T08:22:55.835 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-09T08:22:55.977 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-09T08:22:56.159 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3954c88c-0ddd-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-3954c88c-0ddd-11ef-bc98-c7b262605968.target. 2024-05-09T08:22:56.159 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-3954c88c-0ddd-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-3954c88c-0ddd-11ef-bc98-c7b262605968.target. 2024-05-09T08:22:56.435 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.a 2024-05-09T08:22:56.435 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.a.service: Unit ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.a.service not loaded. 2024-05-09T08:22:56.608 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3954c88c-0ddd-11ef-bc98-c7b262605968.target.wants/ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.a.service → /etc/systemd/system/ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@.service. 2024-05-09T08:22:57.074 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-09T08:22:57.074 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-05-09T08:22:57.081 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-09T08:22:57.081 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-05-09T08:22:57.081 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-05-09T08:22:57.082 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to enable service . firewalld.service is not available 2024-05-09T08:22:57.082 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mon to start... 2024-05-09T08:22:57.082 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mon... 2024-05-09T08:22:57.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:22:57 smithi107 systemd[1]: Started Ceph mon.a for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:22:58.126 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:22:58 smithi107 ceph-mon[29748]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-09T08:22:58.126 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:22:58 smithi107 ceph-mon[29748]: monmap epoch 1 2024-05-09T08:22:58.126 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:22:58 smithi107 ceph-mon[29748]: fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:22:58.126 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:22:58 smithi107 ceph-mon[29748]: last_changed 2024-05-09T08:22:54.137531+0000 2024-05-09T08:22:58.126 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:22:58 smithi107 ceph-mon[29748]: created 2024-05-09T08:22:54.137531+0000 2024-05-09T08:22:58.126 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:22:58 smithi107 ceph-mon[29748]: min_mon_release 19 (squid) 2024-05-09T08:22:58.126 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:22:58 smithi107 ceph-mon[29748]: election_strategy: 1 2024-05-09T08:22:58.126 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:22:58 smithi107 ceph-mon[29748]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-05-09T08:22:58.126 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:22:58 smithi107 ceph-mon[29748]: fsmap 2024-05-09T08:22:58.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:22:58 smithi107 ceph-mon[29748]: osdmap e1: 0 total, 0 up, 0 in 2024-05-09T08:22:58.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:22:58 smithi107 ceph-mon[29748]: mgrmap e1: no daemons active 2024-05-09T08:22:58.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:22:58 smithi107 ceph-mon[29748]: from='client.? 172.21.15.107:0/4218112112' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-09T08:22:58.129 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout cluster: 2024-05-09T08:22:58.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout id: 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:22:58.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-09T08:22:58.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-05-09T08:22:58.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout services: 2024-05-09T08:22:58.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.530405s) 2024-05-09T08:22:58.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-09T08:22:58.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-09T08:22:58.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-05-09T08:22:58.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout data: 2024-05-09T08:22:58.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-09T08:22:58.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-09T08:22:58.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-09T08:22:58.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout pgs: 2024-05-09T08:22:58.137 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-05-09T08:22:58.137 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.480+0000 7f5c36a12700 1 Processor -- start 2024-05-09T08:22:58.137 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.480+0000 7f5c36a12700 1 -- start start 2024-05-09T08:22:58.137 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.480+0000 7f5c36a12700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c30107e60 0x7f5c30108280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:22:58.137 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.480+0000 7f5c36a12700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5c301087c0 con 0x7f5c30107e60 2024-05-09T08:22:58.137 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.481+0000 7f5c2ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c30107e60 0x7f5c30108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:22:58.137 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.481+0000 7f5c2ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c30107e60 0x7f5c30108280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55418/0 (socket says 172.21.15.107:55418) 2024-05-09T08:22:58.137 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.481+0000 7f5c2ffff700 1 -- 172.21.15.107:0/391732476 learned_addr learned my addr 172.21.15.107:0/391732476 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:22:58.137 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.481+0000 7f5c2ffff700 1 -- 172.21.15.107:0/391732476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c30108fc0 con 0x7f5c30107e60 2024-05-09T08:22:58.138 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.481+0000 7f5c2ffff700 1 --2- 172.21.15.107:0/391732476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c30107e60 0x7f5c30108280 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5c200098a0 tx=0x7f5c20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a704d89bc3edee server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:22:58.138 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.481+0000 7f5c2effd700 1 -- 172.21.15.107:0/391732476 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c20004160 con 0x7f5c30107e60 2024-05-09T08:22:58.138 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.481+0000 7f5c2effd700 1 -- 172.21.15.107:0/391732476 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c20004300 con 0x7f5c30107e60 2024-05-09T08:22:58.138 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.481+0000 7f5c2effd700 1 -- 172.21.15.107:0/391732476 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c200036a0 con 0x7f5c30107e60 2024-05-09T08:22:58.138 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.482+0000 7f5c36a12700 1 -- 172.21.15.107:0/391732476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c30107e60 msgr2=0x7f5c30108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:22:58.138 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.482+0000 7f5c36a12700 1 --2- 172.21.15.107:0/391732476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c30107e60 0x7f5c30108280 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5c200098a0 tx=0x7f5c20009bb0 comp rx=0 tx=0).stop 2024-05-09T08:22:58.138 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.482+0000 7f5c36a12700 1 -- 172.21.15.107:0/391732476 shutdown_connections 2024-05-09T08:22:58.138 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.482+0000 7f5c36a12700 1 --2- 172.21.15.107:0/391732476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c30107e60 0x7f5c30108280 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:58.138 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.482+0000 7f5c36a12700 1 -- 172.21.15.107:0/391732476 >> 172.21.15.107:0/391732476 conn(0x7f5c30103440 msgr2=0x7f5c30105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:22:58.139 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.482+0000 7f5c36a12700 1 -- 172.21.15.107:0/391732476 shutdown_connections 2024-05-09T08:22:58.139 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.482+0000 7f5c36a12700 1 -- 172.21.15.107:0/391732476 wait complete. 2024-05-09T08:22:58.139 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.482+0000 7f5c36a12700 1 Processor -- start 2024-05-09T08:22:58.139 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.482+0000 7f5c36a12700 1 -- start start 2024-05-09T08:22:58.139 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.483+0000 7f5c36a12700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c30107e60 0x7f5c3007e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:22:58.139 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.483+0000 7f5c36a12700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5c30109490 con 0x7f5c30107e60 2024-05-09T08:22:58.139 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.483+0000 7f5c2ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c30107e60 0x7f5c3007e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:22:58.139 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.483+0000 7f5c2ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c30107e60 0x7f5c3007e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55422/0 (socket says 172.21.15.107:55422) 2024-05-09T08:22:58.140 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.483+0000 7f5c2ffff700 1 -- 172.21.15.107:0/4218112112 learned_addr learned my addr 172.21.15.107:0/4218112112 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:22:58.140 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.483+0000 7f5c2ffff700 1 -- 172.21.15.107:0/4218112112 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3007e920 con 0x7f5c30107e60 2024-05-09T08:22:58.140 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.483+0000 7f5c2ffff700 1 --2- 172.21.15.107:0/4218112112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c30107e60 0x7f5c3007e3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5c20000c00 tx=0x7f5c20004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:22:58.140 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.483+0000 7f5c2d7fa700 1 -- 172.21.15.107:0/4218112112 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c20003db0 con 0x7f5c30107e60 2024-05-09T08:22:58.140 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.483+0000 7f5c2d7fa700 1 -- 172.21.15.107:0/4218112112 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c20011630 con 0x7f5c30107e60 2024-05-09T08:22:58.140 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.483+0000 7f5c2d7fa700 1 -- 172.21.15.107:0/4218112112 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c20011950 con 0x7f5c30107e60 2024-05-09T08:22:58.140 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.483+0000 7f5c36a12700 1 -- 172.21.15.107:0/4218112112 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c3007cb10 con 0x7f5c30107e60 2024-05-09T08:22:58.140 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.483+0000 7f5c36a12700 1 -- 172.21.15.107:0/4218112112 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c3007d050 con 0x7f5c30107e60 2024-05-09T08:22:58.140 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.484+0000 7f5c2d7fa700 1 -- 172.21.15.107:0/4218112112 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5c20011af0 con 0x7f5c30107e60 2024-05-09T08:22:58.140 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.484+0000 7f5c2d7fa700 1 -- 172.21.15.107:0/4218112112 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5c2001c450 con 0x7f5c30107e60 2024-05-09T08:22:58.141 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.484+0000 7f5c36a12700 1 -- 172.21.15.107:0/4218112112 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c30063980 con 0x7f5c30107e60 2024-05-09T08:22:58.141 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.487+0000 7f5c2d7fa700 1 -- 172.21.15.107:0/4218112112 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f5c20032020 con 0x7f5c30107e60 2024-05-09T08:22:58.141 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.557+0000 7f5c36a12700 1 -- 172.21.15.107:0/4218112112 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f5c30194bc0 con 0x7f5c30107e60 2024-05-09T08:22:58.141 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.557+0000 7f5c2d7fa700 1 -- 172.21.15.107:0/4218112112 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f5c20004260 con 0x7f5c30107e60 2024-05-09T08:22:58.141 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.558+0000 7f5c36a12700 1 -- 172.21.15.107:0/4218112112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c30107e60 msgr2=0x7f5c3007e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:22:58.141 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.558+0000 7f5c36a12700 1 --2- 172.21.15.107:0/4218112112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c30107e60 0x7f5c3007e3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5c20000c00 tx=0x7f5c20004750 comp rx=0 tx=0).stop 2024-05-09T08:22:58.141 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.558+0000 7f5c36a12700 1 -- 172.21.15.107:0/4218112112 shutdown_connections 2024-05-09T08:22:58.141 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.558+0000 7f5c36a12700 1 --2- 172.21.15.107:0/4218112112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c30107e60 0x7f5c3007e3e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:58.141 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.558+0000 7f5c36a12700 1 -- 172.21.15.107:0/4218112112 >> 172.21.15.107:0/4218112112 conn(0x7f5c30103440 msgr2=0x7f5c30192000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:22:58.141 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.558+0000 7f5c36a12700 1 -- 172.21.15.107:0/4218112112 shutdown_connections 2024-05-09T08:22:58.141 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:57.558+0000 7f5c36a12700 1 -- 172.21.15.107:0/4218112112 wait complete. 2024-05-09T08:22:58.141 INFO:teuthology.orchestra.run.smithi107.stdout:mon is available 2024-05-09T08:22:58.141 INFO:teuthology.orchestra.run.smithi107.stdout:Assimilating anything we can from ceph.conf... 2024-05-09T08:22:59.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:22:59 smithi107 ceph-mon[29748]: from='client.? 172.21.15.107:0/1055065932' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-09T08:22:59.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:22:59 smithi107 ceph-mon[29748]: from='client.? 172.21.15.107:0/1055065932' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-09T08:22:59.150 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-05-09T08:22:59.150 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [global] 2024-05-09T08:22:59.150 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout fsid = 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:22:59.150 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.107:3300,v1:172.21.15.107:6789] 2024-05-09T08:22:59.150 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-09T08:22:59.150 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-09T08:22:59.150 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-09T08:22:59.150 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-09T08:22:59.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-05-09T08:22:59.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-09T08:22:59.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-09T08:22:59.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-05-09T08:22:59.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [osd] 2024-05-09T08:22:59.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-05-09T08:22:59.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-09T08:22:59.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-09T08:22:59.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-09T08:22:59.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-09T08:22:59.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-09T08:22:59.152 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.533+0000 7f5571bef700 1 Processor -- start 2024-05-09T08:22:59.152 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.533+0000 7f5571bef700 1 -- start start 2024-05-09T08:22:59.152 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.533+0000 7f5571bef700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f556c107e60 0x7f556c108280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:22:59.152 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.533+0000 7f5571bef700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f556c1087c0 con 0x7f556c107e60 2024-05-09T08:22:59.152 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.534+0000 7f556b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f556c107e60 0x7f556c108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:22:59.152 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.534+0000 7f556b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f556c107e60 0x7f556c108280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55428/0 (socket says 172.21.15.107:55428) 2024-05-09T08:22:59.152 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.534+0000 7f556b7fe700 1 -- 172.21.15.107:0/2220370783 learned_addr learned my addr 172.21.15.107:0/2220370783 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:22:59.152 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.534+0000 7f556b7fe700 1 -- 172.21.15.107:0/2220370783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556c108fc0 con 0x7f556c107e60 2024-05-09T08:22:59.152 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.534+0000 7f556b7fe700 1 --2- 172.21.15.107:0/2220370783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f556c107e60 0x7f556c108280 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f555c0098a0 tx=0x7f555c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4c79d464745c40 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:22:59.152 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.534+0000 7f556affd700 1 -- 172.21.15.107:0/2220370783 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f555c004160 con 0x7f556c107e60 2024-05-09T08:22:59.152 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.534+0000 7f556affd700 1 -- 172.21.15.107:0/2220370783 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f555c004300 con 0x7f556c107e60 2024-05-09T08:22:59.152 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.534+0000 7f556affd700 1 -- 172.21.15.107:0/2220370783 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f555c0036a0 con 0x7f556c107e60 2024-05-09T08:22:59.153 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.535+0000 7f5571bef700 1 -- 172.21.15.107:0/2220370783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f556c107e60 msgr2=0x7f556c108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:22:59.153 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.535+0000 7f5571bef700 1 --2- 172.21.15.107:0/2220370783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f556c107e60 0x7f556c108280 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f555c0098a0 tx=0x7f555c009bb0 comp rx=0 tx=0).stop 2024-05-09T08:22:59.153 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.535+0000 7f5571bef700 1 -- 172.21.15.107:0/2220370783 shutdown_connections 2024-05-09T08:22:59.153 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.535+0000 7f5571bef700 1 --2- 172.21.15.107:0/2220370783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f556c107e60 0x7f556c108280 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:59.153 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.535+0000 7f5571bef700 1 -- 172.21.15.107:0/2220370783 >> 172.21.15.107:0/2220370783 conn(0x7f556c103440 msgr2=0x7f556c105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:22:59.153 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.535+0000 7f5571bef700 1 -- 172.21.15.107:0/2220370783 shutdown_connections 2024-05-09T08:22:59.153 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.535+0000 7f5571bef700 1 -- 172.21.15.107:0/2220370783 wait complete. 2024-05-09T08:22:59.153 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.535+0000 7f5571bef700 1 Processor -- start 2024-05-09T08:22:59.153 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.535+0000 7f5571bef700 1 -- start start 2024-05-09T08:22:59.153 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.536+0000 7f5571bef700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f556c107e60 0x7f556c19b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:22:59.153 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.536+0000 7f5571bef700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f556c109490 con 0x7f556c107e60 2024-05-09T08:22:59.153 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.536+0000 7f556b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f556c107e60 0x7f556c19b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:22:59.154 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.536+0000 7f556b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f556c107e60 0x7f556c19b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55442/0 (socket says 172.21.15.107:55442) 2024-05-09T08:22:59.154 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.536+0000 7f556b7fe700 1 -- 172.21.15.107:0/1055065932 learned_addr learned my addr 172.21.15.107:0/1055065932 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:22:59.154 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.536+0000 7f556b7fe700 1 -- 172.21.15.107:0/1055065932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556c19b970 con 0x7f556c107e60 2024-05-09T08:22:59.154 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.536+0000 7f556b7fe700 1 --2- 172.21.15.107:0/1055065932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f556c107e60 0x7f556c19b430 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f555c000c00 tx=0x7f555c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:22:59.154 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.536+0000 7f55697fa700 1 -- 172.21.15.107:0/1055065932 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f555c003db0 con 0x7f556c107e60 2024-05-09T08:22:59.154 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.536+0000 7f55697fa700 1 -- 172.21.15.107:0/1055065932 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f555c011630 con 0x7f556c107e60 2024-05-09T08:22:59.154 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.537+0000 7f5571bef700 1 -- 172.21.15.107:0/1055065932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f556c19bc00 con 0x7f556c107e60 2024-05-09T08:22:59.154 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.537+0000 7f55697fa700 1 -- 172.21.15.107:0/1055065932 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f555c011950 con 0x7f556c107e60 2024-05-09T08:22:59.154 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.537+0000 7f5571bef700 1 -- 172.21.15.107:0/1055065932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f556c19e8f0 con 0x7f556c107e60 2024-05-09T08:22:59.154 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.537+0000 7f55697fa700 1 -- 172.21.15.107:0/1055065932 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f555c011af0 con 0x7f556c107e60 2024-05-09T08:22:59.154 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.537+0000 7f55697fa700 1 -- 172.21.15.107:0/1055065932 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f555c01c450 con 0x7f556c107e60 2024-05-09T08:22:59.155 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.538+0000 7f5571bef700 1 -- 172.21.15.107:0/1055065932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f556c063980 con 0x7f556c107e60 2024-05-09T08:22:59.155 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.540+0000 7f55697fa700 1 -- 172.21.15.107:0/1055065932 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f555c004260 con 0x7f556c107e60 2024-05-09T08:22:59.155 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.606+0000 7f5571bef700 1 -- 172.21.15.107:0/1055065932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f556c02dd00 con 0x7f556c107e60 2024-05-09T08:22:59.155 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.609+0000 7f55697fa700 1 -- 172.21.15.107:0/1055065932 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+456 (secure 0 0 0) 0x7f555c01b450 con 0x7f556c107e60 2024-05-09T08:22:59.155 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.609+0000 7f55697fa700 1 -- 172.21.15.107:0/1055065932 <== mon.0 v2:172.21.15.107:3300/0 8 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f555c01b630 con 0x7f556c107e60 2024-05-09T08:22:59.155 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.610+0000 7f5571bef700 1 -- 172.21.15.107:0/1055065932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f556c107e60 msgr2=0x7f556c19b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:22:59.155 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.610+0000 7f5571bef700 1 --2- 172.21.15.107:0/1055065932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f556c107e60 0x7f556c19b430 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f555c000c00 tx=0x7f555c004750 comp rx=0 tx=0).stop 2024-05-09T08:22:59.155 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.610+0000 7f5571bef700 1 -- 172.21.15.107:0/1055065932 shutdown_connections 2024-05-09T08:22:59.155 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.610+0000 7f5571bef700 1 --2- 172.21.15.107:0/1055065932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f556c107e60 0x7f556c19b430 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:22:59.155 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.610+0000 7f5571bef700 1 -- 172.21.15.107:0/1055065932 >> 172.21.15.107:0/1055065932 conn(0x7f556c103440 msgr2=0x7f556c192000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:22:59.155 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.610+0000 7f5571bef700 1 -- 172.21.15.107:0/1055065932 shutdown_connections 2024-05-09T08:22:59.155 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:58.610+0000 7f5571bef700 1 -- 172.21.15.107:0/1055065932 wait complete. 2024-05-09T08:22:59.156 INFO:teuthology.orchestra.run.smithi107.stdout:Generating new minimal ceph.conf... 2024-05-09T08:23:00.128 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.539+0000 7f70eb58c700 1 Processor -- start 2024-05-09T08:23:00.128 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.539+0000 7f70eb58c700 1 -- start start 2024-05-09T08:23:00.128 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.539+0000 7f70eb58c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e407e710 0x7f70e407cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:00.129 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.539+0000 7f70eb58c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f70e407d140 con 0x7f70e407e710 2024-05-09T08:23:00.129 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.539+0000 7f70e9328700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e407e710 0x7f70e407cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:00.129 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.539+0000 7f70e9328700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e407e710 0x7f70e407cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55450/0 (socket says 172.21.15.107:55450) 2024-05-09T08:23:00.129 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.539+0000 7f70e9328700 1 -- 172.21.15.107:0/3639280121 learned_addr learned my addr 172.21.15.107:0/3639280121 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:00.129 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.539+0000 7f70e9328700 1 -- 172.21.15.107:0/3639280121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e407d2c0 con 0x7f70e407e710 2024-05-09T08:23:00.129 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.540+0000 7f70e9328700 1 --2- 172.21.15.107:0/3639280121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e407e710 0x7f70e407cb70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f70e00098a0 tx=0x7f70e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310a063a4e167203 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:00.129 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.540+0000 7f70dbfff700 1 -- 172.21.15.107:0/3639280121 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70e0004160 con 0x7f70e407e710 2024-05-09T08:23:00.129 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.540+0000 7f70dbfff700 1 -- 172.21.15.107:0/3639280121 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f70e0004300 con 0x7f70e407e710 2024-05-09T08:23:00.129 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.540+0000 7f70dbfff700 1 -- 172.21.15.107:0/3639280121 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70e0011a80 con 0x7f70e407e710 2024-05-09T08:23:00.129 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.540+0000 7f70eb58c700 1 -- 172.21.15.107:0/3639280121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e407e710 msgr2=0x7f70e407cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:00.129 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.540+0000 7f70eb58c700 1 --2- 172.21.15.107:0/3639280121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e407e710 0x7f70e407cb70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f70e00098a0 tx=0x7f70e0009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:00.129 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.540+0000 7f70eb58c700 1 -- 172.21.15.107:0/3639280121 shutdown_connections 2024-05-09T08:23:00.129 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.540+0000 7f70eb58c700 1 --2- 172.21.15.107:0/3639280121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e407e710 0x7f70e407cb70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:00.129 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.540+0000 7f70eb58c700 1 -- 172.21.15.107:0/3639280121 >> 172.21.15.107:0/3639280121 conn(0x7f70e4103440 msgr2=0x7f70e4105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:00.130 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.541+0000 7f70eb58c700 1 -- 172.21.15.107:0/3639280121 shutdown_connections 2024-05-09T08:23:00.130 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.541+0000 7f70eb58c700 1 -- 172.21.15.107:0/3639280121 wait complete. 2024-05-09T08:23:00.130 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.541+0000 7f70eb58c700 1 Processor -- start 2024-05-09T08:23:00.130 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.541+0000 7f70eb58c700 1 -- start start 2024-05-09T08:23:00.130 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.541+0000 7f70eb58c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e407e710 0x7f70e419df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:00.130 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.541+0000 7f70eb58c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f70e410e340 con 0x7f70e407e710 2024-05-09T08:23:00.130 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.541+0000 7f70e9328700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e407e710 0x7f70e419df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:00.130 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.541+0000 7f70e9328700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e407e710 0x7f70e419df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55458/0 (socket says 172.21.15.107:55458) 2024-05-09T08:23:00.130 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.541+0000 7f70e9328700 1 -- 172.21.15.107:0/2302022701 learned_addr learned my addr 172.21.15.107:0/2302022701 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:00.130 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.542+0000 7f70e9328700 1 -- 172.21.15.107:0/2302022701 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e419e460 con 0x7f70e407e710 2024-05-09T08:23:00.130 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.542+0000 7f70e9328700 1 --2- 172.21.15.107:0/2302022701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e407e710 0x7f70e419df20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f70e0000c00 tx=0x7f70e0011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:00.130 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.542+0000 7f70da7fc700 1 -- 172.21.15.107:0/2302022701 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70e00039f0 con 0x7f70e407e710 2024-05-09T08:23:00.130 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.542+0000 7f70da7fc700 1 -- 172.21.15.107:0/2302022701 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f70e0003b90 con 0x7f70e407e710 2024-05-09T08:23:00.131 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.542+0000 7f70eb58c700 1 -- 172.21.15.107:0/2302022701 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70e419e6f0 con 0x7f70e407e710 2024-05-09T08:23:00.131 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.542+0000 7f70da7fc700 1 -- 172.21.15.107:0/2302022701 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70e001a740 con 0x7f70e407e710 2024-05-09T08:23:00.131 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.542+0000 7f70eb58c700 1 -- 172.21.15.107:0/2302022701 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70e41a13e0 con 0x7f70e407e710 2024-05-09T08:23:00.131 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.542+0000 7f70da7fc700 1 -- 172.21.15.107:0/2302022701 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f70e002d3f0 con 0x7f70e407e710 2024-05-09T08:23:00.131 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.542+0000 7f70da7fc700 1 -- 172.21.15.107:0/2302022701 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f70e0019700 con 0x7f70e407e710 2024-05-09T08:23:00.131 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.543+0000 7f70eb58c700 1 -- 172.21.15.107:0/2302022701 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70e4063980 con 0x7f70e407e710 2024-05-09T08:23:00.131 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.545+0000 7f70da7fc700 1 -- 172.21.15.107:0/2302022701 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f70e0024af0 con 0x7f70e407e710 2024-05-09T08:23:00.131 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.611+0000 7f70eb58c700 1 -- 172.21.15.107:0/2302022701 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f70e41a1920 con 0x7f70e407e710 2024-05-09T08:23:00.131 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.611+0000 7f70da7fc700 1 -- 172.21.15.107:0/2302022701 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7f70e0024440 con 0x7f70e407e710 2024-05-09T08:23:00.131 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.612+0000 7f70eb58c700 1 -- 172.21.15.107:0/2302022701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e407e710 msgr2=0x7f70e419df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:00.131 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.612+0000 7f70eb58c700 1 --2- 172.21.15.107:0/2302022701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e407e710 0x7f70e419df20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f70e0000c00 tx=0x7f70e0011b00 comp rx=0 tx=0).stop 2024-05-09T08:23:00.131 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.612+0000 7f70eb58c700 1 -- 172.21.15.107:0/2302022701 shutdown_connections 2024-05-09T08:23:00.131 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.612+0000 7f70eb58c700 1 --2- 172.21.15.107:0/2302022701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e407e710 0x7f70e419df20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:00.131 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.612+0000 7f70eb58c700 1 -- 172.21.15.107:0/2302022701 >> 172.21.15.107:0/2302022701 conn(0x7f70e4103440 msgr2=0x7f70e4103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:00.132 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.612+0000 7f70eb58c700 1 -- 172.21.15.107:0/2302022701 shutdown_connections 2024-05-09T08:23:00.132 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:22:59.612+0000 7f70eb58c700 1 -- 172.21.15.107:0/2302022701 wait complete. 2024-05-09T08:23:00.132 INFO:teuthology.orchestra.run.smithi107.stdout:Restarting the monitor... 2024-05-09T08:23:00.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:00 smithi107 ceph-mon[29748]: from='client.? 172.21.15.107:0/2302022701' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:23:00.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:00 smithi107 systemd[1]: Stopping Ceph mon.a for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:23:00.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:00 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-a[29744]: 2024-05-09T08:23:00.391+0000 7f1b418e5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:23:00.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:00 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-a[29744]: 2024-05-09T08:23:00.391+0000 7f1b418e5700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-09T08:23:00.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:00 smithi107 podman[29993]: 2024-05-09 08:23:00.575438058 +0000 UTC m=+0.295150206 container died 16bd8aeb26c15da35e8d808c78b7c9f36f4516983af719cb0beafd542ce158a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240507) 2024-05-09T08:23:01.100 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:00 smithi107 podman[29993]: 2024-05-09 08:23:00.907931426 +0000 UTC m=+0.627643571 container cleanup 16bd8aeb26c15da35e8d808c78b7c9f36f4516983af719cb0beafd542ce158a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-a, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc) 2024-05-09T08:23:01.101 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:00 smithi107 bash[29993]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-a 2024-05-09T08:23:01.526 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 podman[30007]: 2024-05-09 08:23:01.100483357 +0000 UTC m=+0.528748735 container remove 16bd8aeb26c15da35e8d808c78b7c9f36f4516983af719cb0beafd542ce158a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux ) 2024-05-09T08:23:01.526 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 systemd[1]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-09T08:23:01.526 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 systemd[1]: Stopped Ceph mon.a for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:23:01.526 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 systemd[1]: Starting Ceph mon.a for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:23:01.774 INFO:teuthology.orchestra.run.smithi107.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-09T08:23:01.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 podman[30103]: 2024-05-09 08:23:01.526266298 +0000 UTC m=+0.105372221 container create 754b572e3f5fb58e446791cb4e4edd20751a86603ed77774f7f0ca1378e22b32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240507) 2024-05-09T08:23:01.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 podman[30103]: 2024-05-09 08:23:01.438726543 +0000 UTC m=+0.017832475 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:23:01.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 podman[30103]: 2024-05-09 08:23:01.695414785 +0000 UTC m=+0.274520707 container init 754b572e3f5fb58e446791cb4e4edd20751a86603ed77774f7f0ca1378e22b32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-09T08:23:01.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 podman[30103]: 2024-05-09 08:23:01.698540267 +0000 UTC m=+0.277646190 container start 754b572e3f5fb58e446791cb4e4edd20751a86603ed77774f7f0ca1378e22b32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, io.buildah.version=1.33.5) 2024-05-09T08:23:01.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: set uid:gid to 167:167 (ceph:ceph) 2024-05-09T08:23:01.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev), process ceph-mon, pid 2 2024-05-09T08:23:01.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: pidfile_write: ignore empty --pid-file 2024-05-09T08:23:01.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: load: jerasure load: lrc 2024-05-09T08:23:01.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: RocksDB version: 7.9.2 2024-05-09T08:23:01.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Git sha 0 2024-05-09T08:23:01.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Compile date 2024-05-07 19:19:25 2024-05-09T08:23:01.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: DB SUMMARY 2024-05-09T08:23:01.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: DB Session ID: W87I5Z5T6DFRH6GVB4YR 2024-05-09T08:23:01.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: CURRENT file: CURRENT 2024-05-09T08:23:01.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: IDENTITY file: IDENTITY 2024-05-09T08:23:01.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-09T08:23:01.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-09T08:23:01.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105242 ; 2024-05-09T08:23:01.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.error_if_exists: 0 2024-05-09T08:23:01.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.create_if_missing: 0 2024-05-09T08:23:01.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.paranoid_checks: 1 2024-05-09T08:23:01.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-09T08:23:01.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-09T08:23:01.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-09T08:23:01.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.env: 0x5616f199ac60 2024-05-09T08:23:01.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.fs: PosixFileSystem 2024-05-09T08:23:01.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.info_log: 0x5616f2c23ac0 2024-05-09T08:23:01.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_file_opening_threads: 16 2024-05-09T08:23:01.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.statistics: (nil) 2024-05-09T08:23:01.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.use_fsync: 0 2024-05-09T08:23:01.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_log_file_size: 0 2024-05-09T08:23:01.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-09T08:23:01.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-09T08:23:01.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.keep_log_file_num: 1000 2024-05-09T08:23:01.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.recycle_log_file_num: 0 2024-05-09T08:23:01.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.allow_fallocate: 1 2024-05-09T08:23:01.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.allow_mmap_reads: 0 2024-05-09T08:23:01.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.allow_mmap_writes: 0 2024-05-09T08:23:01.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.use_direct_reads: 0 2024-05-09T08:23:01.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-09T08:23:01.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.create_missing_column_families: 0 2024-05-09T08:23:01.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.db_log_dir: 2024-05-09T08:23:01.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.wal_dir: 2024-05-09T08:23:01.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-09T08:23:01.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-09T08:23:01.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-09T08:23:01.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-09T08:23:01.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-09T08:23:01.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-09T08:23:01.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.advise_random_on_open: 1 2024-05-09T08:23:01.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.db_write_buffer_size: 0 2024-05-09T08:23:01.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.write_buffer_manager: 0x5616f2bf61e0 2024-05-09T08:23:01.785 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-09T08:23:01.785 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-09T08:23:01.785 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-09T08:23:01.785 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.rate_limiter: (nil) 2024-05-09T08:23:01.785 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-09T08:23:01.785 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.wal_recovery_mode: 2 2024-05-09T08:23:01.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.enable_thread_tracking: 0 2024-05-09T08:23:01.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.enable_pipelined_write: 0 2024-05-09T08:23:01.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.unordered_write: 0 2024-05-09T08:23:01.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-09T08:23:01.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-09T08:23:01.787 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-09T08:23:01.787 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-09T08:23:01.787 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.row_cache: None 2024-05-09T08:23:01.787 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.wal_filter: None 2024-05-09T08:23:01.787 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-09T08:23:01.787 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.allow_ingest_behind: 0 2024-05-09T08:23:01.787 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.two_write_queues: 0 2024-05-09T08:23:01.787 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.manual_wal_flush: 0 2024-05-09T08:23:01.787 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.wal_compression: 0 2024-05-09T08:23:01.787 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.atomic_flush: 0 2024-05-09T08:23:01.787 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-09T08:23:01.787 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-09T08:23:01.787 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-09T08:23:01.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.log_readahead_size: 0 2024-05-09T08:23:01.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-09T08:23:01.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.best_efforts_recovery: 0 2024-05-09T08:23:01.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-09T08:23:01.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-09T08:23:01.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.allow_data_in_errors: 0 2024-05-09T08:23:01.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.db_host_id: __hostname__ 2024-05-09T08:23:01.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-09T08:23:01.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_background_jobs: 2 2024-05-09T08:23:01.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_background_compactions: -1 2024-05-09T08:23:01.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_subcompactions: 1 2024-05-09T08:23:01.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-09T08:23:01.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-09T08:23:01.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-09T08:23:01.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_total_wal_size: 0 2024-05-09T08:23:01.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-09T08:23:01.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-09T08:23:01.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-09T08:23:01.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-09T08:23:01.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_open_files: -1 2024-05-09T08:23:01.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.bytes_per_sync: 0 2024-05-09T08:23:01.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-09T08:23:01.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-09T08:23:01.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compaction_readahead_size: 0 2024-05-09T08:23:01.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_background_flushes: -1 2024-05-09T08:23:01.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Compression algorithms supported: 2024-05-09T08:23:01.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-09T08:23:01.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: kZSTD supported: 0 2024-05-09T08:23:01.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: kXpressCompression supported: 0 2024-05-09T08:23:01.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: kLZ4HCCompression supported: 1 2024-05-09T08:23:01.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: kZlibCompression supported: 1 2024-05-09T08:23:01.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: kSnappyCompression supported: 1 2024-05-09T08:23:01.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: kLZ4Compression supported: 1 2024-05-09T08:23:01.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: kBZip2Compression supported: 0 2024-05-09T08:23:01.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-09T08:23:01.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-09T08:23:01.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-09T08:23:01.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-09T08:23:01.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-09T08:23:01.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.merge_operator: 2024-05-09T08:23:01.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compaction_filter: None 2024-05-09T08:23:01.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compaction_filter_factory: None 2024-05-09T08:23:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.sst_partitioner_factory: None 2024-05-09T08:23:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-09T08:23:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-09T08:23:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5616f2c23bc0) 2024-05-09T08:23:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout: cache_index_and_filter_blocks: 1 2024-05-09T08:23:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-09T08:23:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-09T08:23:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout: pin_top_level_index_and_filter: 1 2024-05-09T08:23:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_type: 0 2024-05-09T08:23:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout: data_block_index_type: 0 2024-05-09T08:23:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_shortening: 1 2024-05-09T08:23:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-09T08:23:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout: checksum: 4 2024-05-09T08:23:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout: no_block_cache: 0 2024-05-09T08:23:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache: 0x5616f2c27090 2024-05-09T08:23:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_name: BinnedLRUCache 2024-05-09T08:23:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_options: 2024-05-09T08:23:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout: capacity : 536870912 2024-05-09T08:23:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout: num_shard_bits : 4 2024-05-09T08:23:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout: strict_capacity_limit : 0 2024-05-09T08:23:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout: high_pri_pool_ratio: 0.000 2024-05-09T08:23:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_compressed: (nil) 2024-05-09T08:23:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout: persistent_cache: (nil) 2024-05-09T08:23:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_size: 4096 2024-05-09T08:23:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_size_deviation: 10 2024-05-09T08:23:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_restart_interval: 16 2024-05-09T08:23:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_block_restart_interval: 1 2024-05-09T08:23:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout: metadata_block_size: 4096 2024-05-09T08:23:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout: partition_filters: 0 2024-05-09T08:23:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout: use_delta_encoding: 1 2024-05-09T08:23:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout: filter_policy: bloomfilter 2024-05-09T08:23:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout: whole_key_filtering: 1 2024-05-09T08:23:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout: verify_compression: 0 2024-05-09T08:23:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout: read_amp_bytes_per_bit: 0 2024-05-09T08:23:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout: format_version: 5 2024-05-09T08:23:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout: enable_index_compression: 1 2024-05-09T08:23:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_align: 0 2024-05-09T08:23:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout: max_auto_readahead_size: 262144 2024-05-09T08:23:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout: prepopulate_block_cache: 0 2024-05-09T08:23:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout: initial_auto_readahead_size: 8192 2024-05-09T08:23:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout: num_file_reads_for_auto_readahead: 2 2024-05-09T08:23:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.write_buffer_size: 33554432 2024-05-09T08:23:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_write_buffer_number: 2 2024-05-09T08:23:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compression: NoCompression 2024-05-09T08:23:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.bottommost_compression: Disabled 2024-05-09T08:23:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.prefix_extractor: nullptr 2024-05-09T08:23:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-09T08:23:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.num_levels: 7 2024-05-09T08:23:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-09T08:23:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-09T08:23:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-09T08:23:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-09T08:23:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-09T08:23:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-09T08:23:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-09T08:23:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-09T08:23:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-09T08:23:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-09T08:23:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-09T08:23:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-09T08:23:01.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-09T08:23:01.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compression_opts.level: 32767 2024-05-09T08:23:01.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compression_opts.strategy: 0 2024-05-09T08:23:01.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-09T08:23:01.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-09T08:23:01.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-09T08:23:01.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-09T08:23:01.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compression_opts.enabled: false 2024-05-09T08:23:01.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-09T08:23:01.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-09T08:23:01.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-09T08:23:01.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-09T08:23:01.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.target_file_size_base: 67108864 2024-05-09T08:23:01.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-09T08:23:01.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-09T08:23:01.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-09T08:23:01.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-09T08:23:01.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-09T08:23:01.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-09T08:23:01.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-09T08:23:01.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-09T08:23:01.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-09T08:23:01.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-09T08:23:01.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-09T08:23:01.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-09T08:23:01.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-09T08:23:01.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-09T08:23:01.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.arena_block_size: 1048576 2024-05-09T08:23:01.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-09T08:23:01.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-09T08:23:01.797 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.disable_auto_compactions: 0 2024-05-09T08:23:01.797 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-09T08:23:01.797 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-09T08:23:01.797 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-09T08:23:01.797 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-09T08:23:01.797 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-09T08:23:01.797 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-09T08:23:01.797 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-09T08:23:01.797 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-09T08:23:01.797 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-09T08:23:01.797 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-09T08:23:01.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.table_properties_collectors: 2024-05-09T08:23:01.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.inplace_update_support: 0 2024-05-09T08:23:01.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-09T08:23:01.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-09T08:23:01.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-09T08:23:01.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-09T08:23:01.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.bloom_locality: 0 2024-05-09T08:23:01.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.max_successive_merges: 0 2024-05-09T08:23:01.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-09T08:23:01.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.paranoid_file_checks: 0 2024-05-09T08:23:01.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.force_consistency_checks: 1 2024-05-09T08:23:01.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.report_bg_io_stats: 0 2024-05-09T08:23:01.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.ttl: 2592000 2024-05-09T08:23:01.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-09T08:23:01.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-09T08:23:01.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-09T08:23:01.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.enable_blob_files: false 2024-05-09T08:23:01.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.min_blob_size: 0 2024-05-09T08:23:01.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.blob_file_size: 268435456 2024-05-09T08:23:01.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-09T08:23:01.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-09T08:23:01.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-09T08:23:01.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-09T08:23:01.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-09T08:23:01.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.blob_file_starting_level: 0 2024-05-09T08:23:01.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-09T08:23:01.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-09T08:23:01.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-09T08:23:01.800 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7384671b-113e-49ae-87a3-017fb8b51862 2024-05-09T08:23:01.800 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715242981728643, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-09T08:23:01.800 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-09T08:23:01.800 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715242981729598, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100331, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98241, "index_size": 308, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14290, "raw_average_key_size": 49, "raw_value_size": 90467, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715242981, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7384671b-113e-49ae-87a3-017fb8b51862", "db_session_id": "W87I5Z5T6DFRH6GVB4YR", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-09T08:23:01.800 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715242981729678, "job": 1, "event": "recovery_finished"} 2024-05-09T08:23:01.800 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-09T08:23:01.800 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-09T08:23:01.800 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5616f2d18000 2024-05-09T08:23:01.800 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: DB pointer 0x5616f2d02000 2024-05-09T08:23:01.800 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-09T08:23:01.800 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-09T08:23:01.800 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** DB Stats ** 2024-05-09T08:23:01.800 INFO:journalctl@ceph.mon.a.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-09T08:23:01.800 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-09T08:23:01.801 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-09T08:23:01.801 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-09T08:23:01.801 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-09T08:23:01.801 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-09T08:23:01.801 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-09T08:23:01.801 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-05-09T08:23:01.801 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** Compaction Stats [default] ** 2024-05-09T08:23:01.801 INFO:journalctl@ceph.mon.a.smithi107.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-09T08:23:01.801 INFO:journalctl@ceph.mon.a.smithi107.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-09T08:23:01.801 INFO:journalctl@ceph.mon.a.smithi107.stdout: L0 2/0 99.81 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-09T08:23:01.801 INFO:journalctl@ceph.mon.a.smithi107.stdout: Sum 2/0 99.81 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-09T08:23:01.801 INFO:journalctl@ceph.mon.a.smithi107.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-09T08:23:01.801 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-05-09T08:23:01.801 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** Compaction Stats [default] ** 2024-05-09T08:23:01.801 INFO:journalctl@ceph.mon.a.smithi107.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-09T08:23:01.802 INFO:journalctl@ceph.mon.a.smithi107.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-09T08:23:01.802 INFO:journalctl@ceph.mon.a.smithi107.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 167.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-09T08:23:01.802 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-05-09T08:23:01.802 INFO:journalctl@ceph.mon.a.smithi107.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-09T08:23:01.802 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-05-09T08:23:01.802 INFO:journalctl@ceph.mon.a.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-09T08:23:01.802 INFO:journalctl@ceph.mon.a.smithi107.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-09T08:23:01.802 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-09T08:23:01.802 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-09T08:23:01.802 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-09T08:23:01.802 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-09T08:23:01.802 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative compaction: 0.00 GB write, 31.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-09T08:23:01.802 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval compaction: 0.00 GB write, 31.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-09T08:23:01.802 INFO:journalctl@ceph.mon.a.smithi107.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-09T08:23:01.803 INFO:journalctl@ceph.mon.a.smithi107.stdout: Block cache BinnedLRUCache@0x5616f2c27090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-05-09T08:23:01.803 INFO:journalctl@ceph.mon.a.smithi107.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-05-09T08:23:01.803 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-05-09T08:23:01.803 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-09T08:23:01.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: starting mon.a rank 0 at public addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] at bind addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:23:01.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: mon.a@-1(???) e1 preinit fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:23:01.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: mon.a@-1(???).mds e1 new map 2024-05-09T08:23:01.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: mon.a@-1(???).mds e1 print_map 2024-05-09T08:23:01.803 INFO:journalctl@ceph.mon.a.smithi107.stdout: e1 2024-05-09T08:23:01.803 INFO:journalctl@ceph.mon.a.smithi107.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-09T08:23:01.803 INFO:journalctl@ceph.mon.a.smithi107.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-09T08:23:01.803 INFO:journalctl@ceph.mon.a.smithi107.stdout: legacy client fscid: -1 2024-05-09T08:23:01.803 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-05-09T08:23:01.803 INFO:journalctl@ceph.mon.a.smithi107.stdout: No filesystems configured 2024-05-09T08:23:01.804 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-09T08:23:01.804 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:23:01.804 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:23:01.804 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:23:01.804 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-09T08:23:01.804 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 bash[30103]: 754b572e3f5fb58e446791cb4e4edd20751a86603ed77774f7f0ca1378e22b32 2024-05-09T08:23:01.804 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 systemd[1]: Started Ceph mon.a for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:23:02.074 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-09T08:23:02.074 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: monmap epoch 1 2024-05-09T08:23:02.074 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:23:02.074 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: last_changed 2024-05-09T08:22:54.137531+0000 2024-05-09T08:23:02.074 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: created 2024-05-09T08:22:54.137531+0000 2024-05-09T08:23:02.074 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: min_mon_release 19 (squid) 2024-05-09T08:23:02.074 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: election_strategy: 1 2024-05-09T08:23:02.074 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-05-09T08:23:02.074 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: fsmap 2024-05-09T08:23:02.074 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: osdmap e1: 0 total, 0 up, 0 in 2024-05-09T08:23:02.074 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:01 smithi107 ceph-mon[30122]: mgrmap e1: no daemons active 2024-05-09T08:23:02.792 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.175+0000 7f05db72f700 1 Processor -- start 2024-05-09T08:23:02.792 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.175+0000 7f05db72f700 1 -- start start 2024-05-09T08:23:02.792 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.175+0000 7f05db72f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d4107e60 0x7f05d4108280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:02.792 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.175+0000 7f05db72f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f05d41087c0 con 0x7f05d4107e60 2024-05-09T08:23:02.792 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.176+0000 7f05d94cb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d4107e60 0x7f05d4108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:02.792 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.176+0000 7f05d94cb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d4107e60 0x7f05d4108280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58462/0 (socket says 172.21.15.107:58462) 2024-05-09T08:23:02.793 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.176+0000 7f05d94cb700 1 -- 172.21.15.107:0/2683142410 learned_addr learned my addr 172.21.15.107:0/2683142410 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:02.793 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.176+0000 7f05d94cb700 1 -- 172.21.15.107:0/2683142410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d4108fc0 con 0x7f05d4107e60 2024-05-09T08:23:02.793 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.176+0000 7f05d94cb700 1 --2- 172.21.15.107:0/2683142410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d4107e60 0x7f05d4108280 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009e80 tx=0x7f05d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de41a23f5502d953 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:02.793 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.176+0000 7f05cbfff700 1 -- 172.21.15.107:0/2683142410 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05d0017070 con 0x7f05d4107e60 2024-05-09T08:23:02.793 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.176+0000 7f05cbfff700 1 -- 172.21.15.107:0/2683142410 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f05d000b8c0 con 0x7f05d4107e60 2024-05-09T08:23:02.793 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.176+0000 7f05cbfff700 1 -- 172.21.15.107:0/2683142410 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05d0004520 con 0x7f05d4107e60 2024-05-09T08:23:02.793 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.177+0000 7f05db72f700 1 -- 172.21.15.107:0/2683142410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d4107e60 msgr2=0x7f05d4108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:02.793 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.177+0000 7f05db72f700 1 --2- 172.21.15.107:0/2683142410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d4107e60 0x7f05d4108280 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009e80 tx=0x7f05d000b330 comp rx=0 tx=0).stop 2024-05-09T08:23:02.793 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.177+0000 7f05db72f700 1 -- 172.21.15.107:0/2683142410 shutdown_connections 2024-05-09T08:23:02.793 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.177+0000 7f05db72f700 1 --2- 172.21.15.107:0/2683142410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d4107e60 0x7f05d4108280 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:02.793 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.177+0000 7f05db72f700 1 -- 172.21.15.107:0/2683142410 >> 172.21.15.107:0/2683142410 conn(0x7f05d4103440 msgr2=0x7f05d4105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:02.793 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.177+0000 7f05db72f700 1 -- 172.21.15.107:0/2683142410 shutdown_connections 2024-05-09T08:23:02.793 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.177+0000 7f05db72f700 1 -- 172.21.15.107:0/2683142410 wait complete. 2024-05-09T08:23:02.793 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.178+0000 7f05db72f700 1 Processor -- start 2024-05-09T08:23:02.794 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.178+0000 7f05db72f700 1 -- start start 2024-05-09T08:23:02.794 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.178+0000 7f05db72f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d4107e60 0x7f05d419bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:02.794 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.178+0000 7f05db72f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f05d410c280 con 0x7f05d4107e60 2024-05-09T08:23:02.794 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.178+0000 7f05d94cb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d4107e60 0x7f05d419bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:02.794 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.178+0000 7f05d94cb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d4107e60 0x7f05d419bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58468/0 (socket says 172.21.15.107:58468) 2024-05-09T08:23:02.794 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.178+0000 7f05d94cb700 1 -- 172.21.15.107:0/3218412859 learned_addr learned my addr 172.21.15.107:0/3218412859 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:02.794 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.178+0000 7f05d94cb700 1 -- 172.21.15.107:0/3218412859 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d419c190 con 0x7f05d4107e60 2024-05-09T08:23:02.794 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.178+0000 7f05d94cb700 1 --2- 172.21.15.107:0/3218412859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d4107e60 0x7f05d419bc50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009e50 tx=0x7f05d0004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:02.794 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.178+0000 7f05ca7fc700 1 -- 172.21.15.107:0/3218412859 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05d0020020 con 0x7f05d4107e60 2024-05-09T08:23:02.794 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.178+0000 7f05ca7fc700 1 -- 172.21.15.107:0/3218412859 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f05d000bba0 con 0x7f05d4107e60 2024-05-09T08:23:02.794 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.178+0000 7f05db72f700 1 -- 172.21.15.107:0/3218412859 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d419c420 con 0x7f05d4107e60 2024-05-09T08:23:02.794 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.178+0000 7f05ca7fc700 1 -- 172.21.15.107:0/3218412859 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05d0017040 con 0x7f05d4107e60 2024-05-09T08:23:02.794 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.178+0000 7f05db72f700 1 -- 172.21.15.107:0/3218412859 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d419f110 con 0x7f05d4107e60 2024-05-09T08:23:02.795 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.179+0000 7f05ca7fc700 1 -- 172.21.15.107:0/3218412859 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f05d0023020 con 0x7f05d4107e60 2024-05-09T08:23:02.795 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.179+0000 7f05ca7fc700 1 -- 172.21.15.107:0/3218412859 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f05d000f050 con 0x7f05d4107e60 2024-05-09T08:23:02.795 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.179+0000 7f05db72f700 1 -- 172.21.15.107:0/3218412859 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05d4063980 con 0x7f05d4107e60 2024-05-09T08:23:02.795 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.182+0000 7f05ca7fc700 1 -- 172.21.15.107:0/3218412859 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f05d002a020 con 0x7f05d4107e60 2024-05-09T08:23:02.795 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.248+0000 7f05db72f700 1 -- 172.21.15.107:0/3218412859 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f05d419f740 con 0x7f05d4107e60 2024-05-09T08:23:02.795 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.252+0000 7f05ca7fc700 1 -- 172.21.15.107:0/3218412859 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f05d000bd50 con 0x7f05d4107e60 2024-05-09T08:23:02.795 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.253+0000 7f05db72f700 1 -- 172.21.15.107:0/3218412859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d4107e60 msgr2=0x7f05d419bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:02.795 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.253+0000 7f05db72f700 1 --2- 172.21.15.107:0/3218412859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d4107e60 0x7f05d419bc50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009e50 tx=0x7f05d0004670 comp rx=0 tx=0).stop 2024-05-09T08:23:02.795 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.253+0000 7f05db72f700 1 -- 172.21.15.107:0/3218412859 shutdown_connections 2024-05-09T08:23:02.795 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.253+0000 7f05db72f700 1 --2- 172.21.15.107:0/3218412859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05d4107e60 0x7f05d419bc50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:02.795 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.253+0000 7f05db72f700 1 -- 172.21.15.107:0/3218412859 >> 172.21.15.107:0/3218412859 conn(0x7f05d4103440 msgr2=0x7f05d4103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:02.795 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.253+0000 7f05db72f700 1 -- 172.21.15.107:0/3218412859 shutdown_connections 2024-05-09T08:23:02.795 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:02.253+0000 7f05db72f700 1 -- 172.21.15.107:0/3218412859 wait complete. 2024-05-09T08:23:02.795 INFO:teuthology.orchestra.run.smithi107.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-09T08:23:02.818 INFO:teuthology.orchestra.run.smithi107.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:23:02.818 INFO:teuthology.orchestra.run.smithi107.stdout:Creating mgr... 2024-05-09T08:23:02.819 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-09T08:23:02.819 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-09T08:23:03.085 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mgr.x 2024-05-09T08:23:03.085 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mgr.x.service: Unit ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mgr.x.service not loaded. 2024-05-09T08:23:03.236 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3954c88c-0ddd-11ef-bc98-c7b262605968.target.wants/ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@.service. 2024-05-09T08:23:03.463 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:03 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3218412859' entity='client.admin' 2024-05-09T08:23:03.463 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:03 smithi107 systemd[1]: Starting Ceph mgr.x for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:23:03.732 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:03 smithi107 podman[30333]: 2024-05-09 08:23:03.462702748 +0000 UTC m=+0.097508963 container create a3dc9c9b852ef2d6a256f19726fde4535b1a760f7a4e6e4bda481804cbb87452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240507) 2024-05-09T08:23:03.732 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:03 smithi107 podman[30333]: 2024-05-09 08:23:03.382870392 +0000 UTC m=+0.017676565 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:23:03.732 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:03 smithi107 podman[30333]: 2024-05-09 08:23:03.673244549 +0000 UTC m=+0.308050728 container init a3dc9c9b852ef2d6a256f19726fde4535b1a760f7a4e6e4bda481804cbb87452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x, ceph=True, RELEASE=squid-96805dc, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-09T08:23:03.732 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:03 smithi107 podman[30333]: 2024-05-09 08:23:03.675835607 +0000 UTC m=+0.310641779 container start a3dc9c9b852ef2d6a256f19726fde4535b1a760f7a4e6e4bda481804cbb87452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, RELEASE=squid-96805dc, ceph=True, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-09T08:23:03.732 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:03 smithi107 bash[30333]: a3dc9c9b852ef2d6a256f19726fde4535b1a760f7a4e6e4bda481804cbb87452 2024-05-09T08:23:03.747 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-09T08:23:03.747 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-05-09T08:23:03.755 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-09T08:23:03.755 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-05-09T08:23:03.755 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-05-09T08:23:03.755 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to enable service . firewalld.service is not available 2024-05-09T08:23:03.764 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-09T08:23:03.765 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-05-09T08:23:03.773 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-09T08:23:03.773 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-05-09T08:23:03.773 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-05-09T08:23:03.773 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-09T08:23:03.773 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr to start... 2024-05-09T08:23:03.773 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr... 2024-05-09T08:23:03.991 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:03 smithi107 systemd[1]: Started Ceph mgr.x for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:23:03.991 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:03 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:03.841+0000 7f1295aa7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:23:04.250 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:04 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:04.029+0000 7f1295aa7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:23:04.250 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:04 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:04.135+0000 7f1295aa7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:23:04.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:04 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/4120623591' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-09T08:23:04.531 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:04 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:04.458+0000 7f1295aa7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:23:04.750 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-05-09T08:23:04.750 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:23:04.750 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "3954c88c-0ddd-11ef-bc98-c7b262605968", 2024-05-09T08:23:04.750 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-05-09T08:23:04.750 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-09T08:23:04.750 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-09T08:23:04.750 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-09T08:23:04.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:04.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-09T08:23:04.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-09T08:23:04.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-05-09T08:23:04.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:23:04.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-09T08:23:04.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-05-09T08:23:04.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:23:04.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-05-09T08:23:04.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-09T08:23:04.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:23:04.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-09T08:23:04.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-09T08:23:04.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:04.751 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-09T08:23:04.752 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:23:04.752 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-09T08:23:04.752 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-09T08:23:04.752 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-09T08:23:04.752 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-09T08:23:04.752 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-09T08:23:04.752 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-09T08:23:04.752 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:04.752 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-09T08:23:04.752 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-09T08:23:04.752 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-09T08:23:04.752 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-09T08:23:04.752 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-09T08:23:04.752 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-09T08:23:04.752 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-09T08:23:04.753 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-09T08:23:04.753 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-09T08:23:04.753 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:04.753 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-09T08:23:04.753 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:23:04.753 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-09T08:23:04.753 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-09T08:23:04.753 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:04.753 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-09T08:23:04.753 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-09T08:23:04.753 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-09T08:23:04.753 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-09T08:23:04.753 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-09T08:23:04.753 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-09T08:23:04.753 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-05-09T08:23:04.753 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:23:04.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:23:04.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:04.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-09T08:23:04.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:23:04.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-05-09T08:22:57.028819+0000", 2024-05-09T08:23:04.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:23:04.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:04.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-09T08:23:04.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:23:04.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.170+0000 7f2f43dfe700 1 Processor -- start 2024-05-09T08:23:04.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.170+0000 7f2f43dfe700 1 -- start start 2024-05-09T08:23:04.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.170+0000 7f2f43dfe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3c108920 0x7f2f3c10ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:04.754 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.170+0000 7f2f43dfe700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c07c6e0 con 0x7f2f3c108920 2024-05-09T08:23:04.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.171+0000 7f2f41b9a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3c108920 0x7f2f3c10ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:04.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.171+0000 7f2f41b9a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3c108920 0x7f2f3c10ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58498/0 (socket says 172.21.15.107:58498) 2024-05-09T08:23:04.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.171+0000 7f2f41b9a700 1 -- 172.21.15.107:0/1315136400 learned_addr learned my addr 172.21.15.107:0/1315136400 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:04.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.171+0000 7f2f41b9a700 1 -- 172.21.15.107:0/1315136400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f3c10b250 con 0x7f2f3c108920 2024-05-09T08:23:04.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.171+0000 7f2f41b9a700 1 --2- 172.21.15.107:0/1315136400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3c108920 0x7f2f3c10ad10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2f380098a0 tx=0x7f2f38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=126f8633e5e9bb98 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:04.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.171+0000 7f2f40b98700 1 -- 172.21.15.107:0/1315136400 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f38004160 con 0x7f2f3c108920 2024-05-09T08:23:04.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.171+0000 7f2f40b98700 1 -- 172.21.15.107:0/1315136400 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2f38004300 con 0x7f2f3c108920 2024-05-09T08:23:04.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.171+0000 7f2f40b98700 1 -- 172.21.15.107:0/1315136400 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f38011a80 con 0x7f2f3c108920 2024-05-09T08:23:04.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.172+0000 7f2f43dfe700 1 -- 172.21.15.107:0/1315136400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3c108920 msgr2=0x7f2f3c10ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:04.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.172+0000 7f2f43dfe700 1 --2- 172.21.15.107:0/1315136400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3c108920 0x7f2f3c10ad10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2f380098a0 tx=0x7f2f38009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:04.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.172+0000 7f2f43dfe700 1 -- 172.21.15.107:0/1315136400 shutdown_connections 2024-05-09T08:23:04.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.172+0000 7f2f43dfe700 1 --2- 172.21.15.107:0/1315136400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3c108920 0x7f2f3c10ad10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:04.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.172+0000 7f2f43dfe700 1 -- 172.21.15.107:0/1315136400 >> 172.21.15.107:0/1315136400 conn(0x7f2f3c102350 msgr2=0x7f2f3c104770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:04.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.172+0000 7f2f43dfe700 1 -- 172.21.15.107:0/1315136400 shutdown_connections 2024-05-09T08:23:04.755 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.172+0000 7f2f43dfe700 1 -- 172.21.15.107:0/1315136400 wait complete. 2024-05-09T08:23:04.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.173+0000 7f2f43dfe700 1 Processor -- start 2024-05-09T08:23:04.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.173+0000 7f2f43dfe700 1 -- start start 2024-05-09T08:23:04.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.173+0000 7f2f43dfe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3c108920 0x7f2f3c19bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:04.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.173+0000 7f2f43dfe700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c10c130 con 0x7f2f3c108920 2024-05-09T08:23:04.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.173+0000 7f2f41b9a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3c108920 0x7f2f3c19bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:04.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.173+0000 7f2f41b9a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3c108920 0x7f2f3c19bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58510/0 (socket says 172.21.15.107:58510) 2024-05-09T08:23:04.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.173+0000 7f2f41b9a700 1 -- 172.21.15.107:0/4120623591 learned_addr learned my addr 172.21.15.107:0/4120623591 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:04.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.173+0000 7f2f41b9a700 1 -- 172.21.15.107:0/4120623591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f3c19c1c0 con 0x7f2f3c108920 2024-05-09T08:23:04.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.174+0000 7f2f41b9a700 1 --2- 172.21.15.107:0/4120623591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3c108920 0x7f2f3c19bc80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2f38011a50 tx=0x7f2f38011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:04.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.174+0000 7f2f32ffd700 1 -- 172.21.15.107:0/4120623591 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f38003a80 con 0x7f2f3c108920 2024-05-09T08:23:04.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.174+0000 7f2f32ffd700 1 -- 172.21.15.107:0/4120623591 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2f38003c20 con 0x7f2f3c108920 2024-05-09T08:23:04.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.174+0000 7f2f43dfe700 1 -- 172.21.15.107:0/4120623591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f3c19c450 con 0x7f2f3c108920 2024-05-09T08:23:04.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.174+0000 7f2f32ffd700 1 -- 172.21.15.107:0/4120623591 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f3801a650 con 0x7f2f3c108920 2024-05-09T08:23:04.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.174+0000 7f2f43dfe700 1 -- 172.21.15.107:0/4120623591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f3c19f140 con 0x7f2f3c108920 2024-05-09T08:23:04.756 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.175+0000 7f2f32ffd700 1 -- 172.21.15.107:0/4120623591 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2f3801a970 con 0x7f2f3c108920 2024-05-09T08:23:04.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.175+0000 7f2f32ffd700 1 -- 172.21.15.107:0/4120623591 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2f380193c0 con 0x7f2f3c108920 2024-05-09T08:23:04.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.175+0000 7f2f43dfe700 1 -- 172.21.15.107:0/4120623591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f3c063980 con 0x7f2f3c108920 2024-05-09T08:23:04.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.177+0000 7f2f32ffd700 1 -- 172.21.15.107:0/4120623591 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2f380248d0 con 0x7f2f3c108920 2024-05-09T08:23:04.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.249+0000 7f2f43dfe700 1 -- 172.21.15.107:0/4120623591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f2f3c19f310 con 0x7f2f3c108920 2024-05-09T08:23:04.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.250+0000 7f2f32ffd700 1 -- 172.21.15.107:0/4120623591 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f2f38032030 con 0x7f2f3c108920 2024-05-09T08:23:04.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.251+0000 7f2f43dfe700 1 -- 172.21.15.107:0/4120623591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3c108920 msgr2=0x7f2f3c19bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:04.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.251+0000 7f2f43dfe700 1 --2- 172.21.15.107:0/4120623591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3c108920 0x7f2f3c19bc80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2f38011a50 tx=0x7f2f38011b30 comp rx=0 tx=0).stop 2024-05-09T08:23:04.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.251+0000 7f2f43dfe700 1 -- 172.21.15.107:0/4120623591 shutdown_connections 2024-05-09T08:23:04.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.251+0000 7f2f43dfe700 1 --2- 172.21.15.107:0/4120623591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f3c108920 0x7f2f3c19bc80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:04.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.251+0000 7f2f43dfe700 1 -- 172.21.15.107:0/4120623591 >> 172.21.15.107:0/4120623591 conn(0x7f2f3c102350 msgr2=0x7f2f3c102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:04.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.251+0000 7f2f43dfe700 1 -- 172.21.15.107:0/4120623591 shutdown_connections 2024-05-09T08:23:04.757 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:04.251+0000 7f2f43dfe700 1 -- 172.21.15.107:0/4120623591 wait complete. 2024-05-09T08:23:04.757 INFO:teuthology.orchestra.run.smithi107.stdout:mgr not available, waiting (1/15)... 2024-05-09T08:23:04.902 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:04 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:04.545+0000 7f1295aa7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:23:06.030 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:05 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:05.728+0000 7f1295aa7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:23:06.031 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:05 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:05.895+0000 7f1295aa7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:23:06.348 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:06 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:06.064+0000 7f1295aa7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:23:06.348 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:06 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:06.347+0000 7f1295aa7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:23:06.640 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:06 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:06.438+0000 7f1295aa7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:23:06.640 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:06 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:06.639+0000 7f1295aa7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:23:06.922 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:06 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:06.833+0000 7f1295aa7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:23:07.459 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:07 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2932576542' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-09T08:23:07.780 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:07 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:07.458+0000 7f1295aa7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:23:07.931 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-05-09T08:23:07.931 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:23:07.931 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "3954c88c-0ddd-11ef-bc98-c7b262605968", 2024-05-09T08:23:07.931 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-05-09T08:23:07.941 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-09T08:23:07.941 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-09T08:23:07.941 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-09T08:23:07.941 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:07.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-09T08:23:07.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-09T08:23:07.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-05-09T08:23:07.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:23:07.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-09T08:23:07.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-05-09T08:23:07.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:23:07.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-05-09T08:23:07.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-09T08:23:07.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:23:07.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-09T08:23:07.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-09T08:23:07.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:07.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-09T08:23:07.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:23:07.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-09T08:23:07.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-09T08:23:07.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-09T08:23:07.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-09T08:23:07.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-09T08:23:07.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-09T08:23:07.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:07.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-09T08:23:07.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-09T08:23:07.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-09T08:23:07.951 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-09T08:23:07.951 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-09T08:23:07.951 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-09T08:23:07.951 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-09T08:23:07.951 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-09T08:23:07.951 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-09T08:23:07.951 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:07.951 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-09T08:23:07.952 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:23:07.952 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-09T08:23:07.952 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-09T08:23:07.952 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:07.952 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-09T08:23:07.952 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-09T08:23:07.952 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-09T08:23:07.952 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-09T08:23:07.952 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-09T08:23:07.952 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-09T08:23:07.952 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-05-09T08:23:07.952 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:23:07.952 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:23:07.952 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:07.952 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-09T08:23:07.953 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:23:07.953 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-05-09T08:22:57.028819+0000", 2024-05-09T08:23:07.953 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:23:07.953 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:07.953 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-09T08:23:07.953 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:23:07.953 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.207+0000 7f25934cd700 1 Processor -- start 2024-05-09T08:23:07.953 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.207+0000 7f25934cd700 1 -- start start 2024-05-09T08:23:07.953 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.208+0000 7f25934cd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258c106170 0x7f258c106590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:07.953 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.208+0000 7f25934cd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f258c106b60 con 0x7f258c106170 2024-05-09T08:23:07.953 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.208+0000 7f2591269700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258c106170 0x7f258c106590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:07.953 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.208+0000 7f2591269700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258c106170 0x7f258c106590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58518/0 (socket says 172.21.15.107:58518) 2024-05-09T08:23:07.953 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.208+0000 7f2591269700 1 -- 172.21.15.107:0/1206871594 learned_addr learned my addr 172.21.15.107:0/1206871594 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:07.953 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.208+0000 7f2591269700 1 -- 172.21.15.107:0/1206871594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f258c109100 con 0x7f258c106170 2024-05-09T08:23:07.954 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.208+0000 7f2591269700 1 --2- 172.21.15.107:0/1206871594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258c106170 0x7f258c106590 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f25880098a0 tx=0x7f2588009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f282c30d03ba3d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:07.954 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.208+0000 7f2583fff700 1 -- 172.21.15.107:0/1206871594 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2588004160 con 0x7f258c106170 2024-05-09T08:23:07.954 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.208+0000 7f2583fff700 1 -- 172.21.15.107:0/1206871594 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2588004300 con 0x7f258c106170 2024-05-09T08:23:07.954 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.208+0000 7f2583fff700 1 -- 172.21.15.107:0/1206871594 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2588011a80 con 0x7f258c106170 2024-05-09T08:23:07.954 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.209+0000 7f25934cd700 1 -- 172.21.15.107:0/1206871594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258c106170 msgr2=0x7f258c106590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:07.954 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.209+0000 7f25934cd700 1 --2- 172.21.15.107:0/1206871594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258c106170 0x7f258c106590 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f25880098a0 tx=0x7f2588009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:07.954 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.209+0000 7f25934cd700 1 -- 172.21.15.107:0/1206871594 shutdown_connections 2024-05-09T08:23:07.954 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.209+0000 7f25934cd700 1 --2- 172.21.15.107:0/1206871594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258c106170 0x7f258c106590 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:07.954 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.209+0000 7f25934cd700 1 -- 172.21.15.107:0/1206871594 >> 172.21.15.107:0/1206871594 conn(0x7f258c101ac0 msgr2=0x7f258c103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:07.954 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.209+0000 7f25934cd700 1 -- 172.21.15.107:0/1206871594 shutdown_connections 2024-05-09T08:23:07.954 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.209+0000 7f25934cd700 1 -- 172.21.15.107:0/1206871594 wait complete. 2024-05-09T08:23:07.954 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.210+0000 7f25934cd700 1 Processor -- start 2024-05-09T08:23:07.954 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.210+0000 7f25934cd700 1 -- start start 2024-05-09T08:23:07.954 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.210+0000 7f25934cd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258c106170 0x7f258c19bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:07.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.210+0000 7f25934cd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f258c10c330 con 0x7f258c106170 2024-05-09T08:23:07.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.210+0000 7f2591269700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258c106170 0x7f258c19bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:07.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.210+0000 7f2591269700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258c106170 0x7f258c19bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58526/0 (socket says 172.21.15.107:58526) 2024-05-09T08:23:07.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.210+0000 7f2591269700 1 -- 172.21.15.107:0/2932576542 learned_addr learned my addr 172.21.15.107:0/2932576542 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:07.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.210+0000 7f2591269700 1 -- 172.21.15.107:0/2932576542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f258c19c1f0 con 0x7f258c106170 2024-05-09T08:23:07.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.211+0000 7f2591269700 1 --2- 172.21.15.107:0/2932576542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258c106170 0x7f258c19bcb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2588000c00 tx=0x7f2588011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:07.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.211+0000 7f25827fc700 1 -- 172.21.15.107:0/2932576542 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25880039f0 con 0x7f258c106170 2024-05-09T08:23:07.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.211+0000 7f25827fc700 1 -- 172.21.15.107:0/2932576542 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2588003b90 con 0x7f258c106170 2024-05-09T08:23:07.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.211+0000 7f25934cd700 1 -- 172.21.15.107:0/2932576542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f258c19c480 con 0x7f258c106170 2024-05-09T08:23:07.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.211+0000 7f25827fc700 1 -- 172.21.15.107:0/2932576542 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f258801a740 con 0x7f258c106170 2024-05-09T08:23:07.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.211+0000 7f25934cd700 1 -- 172.21.15.107:0/2932576542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f258c19f170 con 0x7f258c106170 2024-05-09T08:23:07.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.211+0000 7f25827fc700 1 -- 172.21.15.107:0/2932576542 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f258802d3f0 con 0x7f258c106170 2024-05-09T08:23:07.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.211+0000 7f25827fc700 1 -- 172.21.15.107:0/2932576542 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2588019700 con 0x7f258c106170 2024-05-09T08:23:07.955 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.212+0000 7f25934cd700 1 -- 172.21.15.107:0/2932576542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f258c10c4b0 con 0x7f258c106170 2024-05-09T08:23:07.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.214+0000 7f25827fc700 1 -- 172.21.15.107:0/2932576542 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2588024af0 con 0x7f258c106170 2024-05-09T08:23:07.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.285+0000 7f25934cd700 1 -- 172.21.15.107:0/2932576542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f258c19f340 con 0x7f258c106170 2024-05-09T08:23:07.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.286+0000 7f25827fc700 1 -- 172.21.15.107:0/2932576542 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f25880199a0 con 0x7f258c106170 2024-05-09T08:23:07.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.287+0000 7f25934cd700 1 -- 172.21.15.107:0/2932576542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258c106170 msgr2=0x7f258c19bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:07.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.287+0000 7f25934cd700 1 --2- 172.21.15.107:0/2932576542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258c106170 0x7f258c19bcb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2588000c00 tx=0x7f2588011b00 comp rx=0 tx=0).stop 2024-05-09T08:23:07.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.287+0000 7f25934cd700 1 -- 172.21.15.107:0/2932576542 shutdown_connections 2024-05-09T08:23:07.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.287+0000 7f25934cd700 1 --2- 172.21.15.107:0/2932576542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f258c106170 0x7f258c19bcb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:07.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.287+0000 7f25934cd700 1 -- 172.21.15.107:0/2932576542 >> 172.21.15.107:0/2932576542 conn(0x7f258c101ac0 msgr2=0x7f258c102510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:07.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.288+0000 7f25934cd700 1 -- 172.21.15.107:0/2932576542 shutdown_connections 2024-05-09T08:23:07.956 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:07.288+0000 7f25934cd700 1 -- 172.21.15.107:0/2932576542 wait complete. 2024-05-09T08:23:07.956 INFO:teuthology.orchestra.run.smithi107.stdout:mgr not available, waiting (2/15)... 2024-05-09T08:23:08.188 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:07 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:07.888+0000 7f1295aa7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:23:08.188 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:07 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:07.987+0000 7f1295aa7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:23:08.188 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:08 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:08.185+0000 7f1295aa7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:23:09.030 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:08 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:08.704+0000 7f1295aa7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:23:09.031 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:08 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:08.796+0000 7f1295aa7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:23:09.381 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:09 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:09.037+0000 7f1295aa7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:23:09.382 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:09 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:09.169+0000 7f1295aa7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:23:09.382 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:09 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:09.264+0000 7f1295aa7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:23:09.382 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:09 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:09.380+0000 7f1295aa7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:23:10.030 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:09 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:09.681+0000 7f1295aa7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:23:10.510 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:10 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:10.289+0000 7f1295aa7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:23:10.510 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:10 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:10.509+0000 7f1295aa7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:23:10.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:10 smithi107 ceph-mon[30122]: Activating manager daemon x 2024-05-09T08:23:10.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:10 smithi107 ceph-mon[30122]: mgrmap e2: x(active, starting, since 0.00292189s) 2024-05-09T08:23:10.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:10 smithi107 ceph-mon[30122]: from='mgr.14100 172.21.15.107:0/3516359523' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:23:10.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:10 smithi107 ceph-mon[30122]: from='mgr.14100 172.21.15.107:0/3516359523' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:23:10.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:10 smithi107 ceph-mon[30122]: from='mgr.14100 172.21.15.107:0/3516359523' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:23:10.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:10 smithi107 ceph-mon[30122]: from='mgr.14100 172.21.15.107:0/3516359523' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:23:10.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:10 smithi107 ceph-mon[30122]: from='mgr.14100 172.21.15.107:0/3516359523' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:23:10.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:10 smithi107 ceph-mon[30122]: Manager daemon x is now available 2024-05-09T08:23:10.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:10 smithi107 ceph-mon[30122]: from='mgr.14100 172.21.15.107:0/3516359523' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:23:10.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:10 smithi107 ceph-mon[30122]: from='mgr.14100 172.21.15.107:0/3516359523' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:23:10.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:10 smithi107 ceph-mon[30122]: from='mgr.14100 172.21.15.107:0/3516359523' entity='mgr.x' 2024-05-09T08:23:10.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:10 smithi107 ceph-mon[30122]: from='mgr.14100 172.21.15.107:0/3516359523' entity='mgr.x' 2024-05-09T08:23:10.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:10 smithi107 ceph-mon[30122]: from='mgr.14100 172.21.15.107:0/3516359523' entity='mgr.x' 2024-05-09T08:23:12.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:12 smithi107 ceph-mon[30122]: mgrmap e3: x(active, since 1.00729s) 2024-05-09T08:23:13.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:13 smithi107 ceph-mon[30122]: mgrmap e4: x(active, since 2s) 2024-05-09T08:23:13.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:13 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/408814081' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-09T08:23:18.790 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-05-09T08:23:18.790 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:23:18.790 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "3954c88c-0ddd-11ef-bc98-c7b262605968", 2024-05-09T08:23:18.790 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-05-09T08:23:18.790 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-09T08:23:18.791 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-09T08:23:18.791 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-09T08:23:18.791 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:18.791 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-09T08:23:18.791 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-09T08:23:18.791 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-05-09T08:23:18.791 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:23:18.791 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-09T08:23:18.791 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-05-09T08:23:18.791 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:23:18.791 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-05-09T08:23:18.791 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-09T08:23:18.791 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:23:18.791 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-09T08:23:18.792 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-09T08:23:18.792 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:18.792 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-09T08:23:18.792 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:23:18.792 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-09T08:23:18.792 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-09T08:23:18.792 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-09T08:23:18.801 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-09T08:23:18.801 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-09T08:23:18.801 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-09T08:23:18.801 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:18.801 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-09T08:23:18.801 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-09T08:23:18.801 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-09T08:23:18.801 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-09T08:23:18.801 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-09T08:23:18.802 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-09T08:23:18.802 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-09T08:23:18.802 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-09T08:23:18.802 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-09T08:23:18.802 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:18.802 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-09T08:23:18.802 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:23:18.802 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-09T08:23:18.802 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-09T08:23:18.802 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:18.802 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-09T08:23:18.802 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-09T08:23:18.802 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-09T08:23:18.803 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-09T08:23:18.803 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-09T08:23:18.803 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-09T08:23:18.803 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-05-09T08:23:18.803 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-05-09T08:23:18.803 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:23:18.803 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:18.803 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-09T08:23:18.803 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T08:23:18.803 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-05-09T08:22:57.028819+0000", 2024-05-09T08:23:18.803 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T08:23:18.803 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-05-09T08:23:18.803 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-09T08:23:18.803 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:23:18.804 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.630+0000 7f1a0f649700 1 Processor -- start 2024-05-09T08:23:18.804 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.630+0000 7f1a0f649700 1 -- start start 2024-05-09T08:23:18.804 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.631+0000 7f1a0f649700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a0807e710 0x7f1a0807cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:18.804 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.631+0000 7f1a0f649700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0807d140 con 0x7f1a0807e710 2024-05-09T08:23:18.804 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.631+0000 7f1a0d3e5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a0807e710 0x7f1a0807cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:18.804 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.631+0000 7f1a0d3e5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a0807e710 0x7f1a0807cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34944/0 (socket says 172.21.15.107:34944) 2024-05-09T08:23:18.804 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.631+0000 7f1a0d3e5700 1 -- 172.21.15.107:0/1102929253 learned_addr learned my addr 172.21.15.107:0/1102929253 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:18.804 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.631+0000 7f1a0d3e5700 1 -- 172.21.15.107:0/1102929253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a0807d2c0 con 0x7f1a0807e710 2024-05-09T08:23:18.804 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.631+0000 7f1a0d3e5700 1 --2- 172.21.15.107:0/1102929253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a0807e710 0x7f1a0807cb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1a040098a0 tx=0x7f1a04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d92d5c48fdd9b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:18.804 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.631+0000 7f19fffff700 1 -- 172.21.15.107:0/1102929253 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a04004160 con 0x7f1a0807e710 2024-05-09T08:23:18.804 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.631+0000 7f19fffff700 1 -- 172.21.15.107:0/1102929253 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1a04004300 con 0x7f1a0807e710 2024-05-09T08:23:18.804 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.631+0000 7f19fffff700 1 -- 172.21.15.107:0/1102929253 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a04011a80 con 0x7f1a0807e710 2024-05-09T08:23:18.804 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.632+0000 7f1a0f649700 1 -- 172.21.15.107:0/1102929253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a0807e710 msgr2=0x7f1a0807cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:18.805 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.632+0000 7f1a0f649700 1 --2- 172.21.15.107:0/1102929253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a0807e710 0x7f1a0807cb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1a040098a0 tx=0x7f1a04009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:18.805 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.632+0000 7f1a0f649700 1 -- 172.21.15.107:0/1102929253 shutdown_connections 2024-05-09T08:23:18.805 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.632+0000 7f1a0f649700 1 --2- 172.21.15.107:0/1102929253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a0807e710 0x7f1a0807cb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:18.805 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.632+0000 7f1a0f649700 1 -- 172.21.15.107:0/1102929253 >> 172.21.15.107:0/1102929253 conn(0x7f1a08103440 msgr2=0x7f1a08105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:18.805 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.632+0000 7f1a0f649700 1 -- 172.21.15.107:0/1102929253 shutdown_connections 2024-05-09T08:23:18.805 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.632+0000 7f1a0f649700 1 -- 172.21.15.107:0/1102929253 wait complete. 2024-05-09T08:23:18.805 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.633+0000 7f1a0f649700 1 Processor -- start 2024-05-09T08:23:18.805 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.633+0000 7f1a0f649700 1 -- start start 2024-05-09T08:23:18.805 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.633+0000 7f1a0f649700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a0807e710 0x7f1a0819df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:18.805 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.633+0000 7f1a0f649700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0810e340 con 0x7f1a0807e710 2024-05-09T08:23:18.805 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.633+0000 7f1a0d3e5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a0807e710 0x7f1a0819df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:18.805 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.633+0000 7f1a0d3e5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a0807e710 0x7f1a0819df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34960/0 (socket says 172.21.15.107:34960) 2024-05-09T08:23:18.805 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.633+0000 7f1a0d3e5700 1 -- 172.21.15.107:0/408814081 learned_addr learned my addr 172.21.15.107:0/408814081 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:18.805 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.633+0000 7f1a0d3e5700 1 -- 172.21.15.107:0/408814081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a0819e460 con 0x7f1a0807e710 2024-05-09T08:23:18.806 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.633+0000 7f1a0d3e5700 1 --2- 172.21.15.107:0/408814081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a0807e710 0x7f1a0819df20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1a04006af0 tx=0x7f1a04011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:18.806 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.634+0000 7f19fe7fc700 1 -- 172.21.15.107:0/408814081 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a040039e0 con 0x7f1a0807e710 2024-05-09T08:23:18.806 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.634+0000 7f19fe7fc700 1 -- 172.21.15.107:0/408814081 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1a04003b80 con 0x7f1a0807e710 2024-05-09T08:23:18.806 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.634+0000 7f1a0f649700 1 -- 172.21.15.107:0/408814081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a0819e6f0 con 0x7f1a0807e710 2024-05-09T08:23:18.806 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.634+0000 7f19fe7fc700 1 -- 172.21.15.107:0/408814081 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a040196a0 con 0x7f1a0807e710 2024-05-09T08:23:18.806 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.634+0000 7f1a0f649700 1 -- 172.21.15.107:0/408814081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a081a13e0 con 0x7f1a0807e710 2024-05-09T08:23:18.806 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.634+0000 7f19fe7fc700 1 -- 172.21.15.107:0/408814081 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f1a040199a0 con 0x7f1a0807e710 2024-05-09T08:23:18.806 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.634+0000 7f1a0f649700 1 -- 172.21.15.107:0/408814081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a08063980 con 0x7f1a0807e710 2024-05-09T08:23:18.806 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.635+0000 7f19fe7fc700 1 --2- 172.21.15.107:0/408814081 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f19f4038fb0 0x7f19f403b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:18.806 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.635+0000 7f19fe7fc700 1 -- 172.21.15.107:0/408814081 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1a0404d6b0 con 0x7f1a0807e710 2024-05-09T08:23:18.806 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.635+0000 7f1a0cbe4700 1 --2- 172.21.15.107:0/408814081 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f19f4038fb0 0x7f19f403b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:18.806 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.635+0000 7f1a0cbe4700 1 --2- 172.21.15.107:0/408814081 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f19f4038fb0 0x7f19f403b470 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f19f8006fd0 tx=0x7f19f8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:18.807 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.640+0000 7f19fe7fc700 1 -- 172.21.15.107:0/408814081 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1a0404d0d0 con 0x7f1a0807e710 2024-05-09T08:23:18.807 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.919+0000 7f1a0f649700 1 -- 172.21.15.107:0/408814081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f1a081a15b0 con 0x7f1a0807e710 2024-05-09T08:23:18.807 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.919+0000 7f19fe7fc700 1 -- 172.21.15.107:0/408814081 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f1a081a15b0 con 0x7f1a0807e710 2024-05-09T08:23:18.807 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.921+0000 7f1a0f649700 1 -- 172.21.15.107:0/408814081 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f19f4038fb0 msgr2=0x7f19f403b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:18.807 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.921+0000 7f1a0f649700 1 --2- 172.21.15.107:0/408814081 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f19f4038fb0 0x7f19f403b470 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f19f8006fd0 tx=0x7f19f8006e40 comp rx=0 tx=0).stop 2024-05-09T08:23:18.807 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.921+0000 7f1a0f649700 1 -- 172.21.15.107:0/408814081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a0807e710 msgr2=0x7f1a0819df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:18.807 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.921+0000 7f1a0f649700 1 --2- 172.21.15.107:0/408814081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a0807e710 0x7f1a0819df20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1a04006af0 tx=0x7f1a04011c40 comp rx=0 tx=0).stop 2024-05-09T08:23:18.807 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.921+0000 7f1a0f649700 1 -- 172.21.15.107:0/408814081 shutdown_connections 2024-05-09T08:23:18.807 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.921+0000 7f1a0f649700 1 --2- 172.21.15.107:0/408814081 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f19f4038fb0 0x7f19f403b470 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:18.807 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.921+0000 7f1a0f649700 1 --2- 172.21.15.107:0/408814081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1a0807e710 0x7f1a0819df20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:18.807 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.921+0000 7f1a0f649700 1 -- 172.21.15.107:0/408814081 >> 172.21.15.107:0/408814081 conn(0x7f1a08103440 msgr2=0x7f1a08103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:18.807 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.921+0000 7f1a0f649700 1 -- 172.21.15.107:0/408814081 shutdown_connections 2024-05-09T08:23:18.807 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:12.921+0000 7f1a0f649700 1 -- 172.21.15.107:0/408814081 wait complete. 2024-05-09T08:23:18.807 INFO:teuthology.orchestra.run.smithi107.stdout:mgr is available 2024-05-09T08:23:21.217 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:20 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2810445771' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-09T08:23:21.519 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-05-09T08:23:21.520 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [global] 2024-05-09T08:23:21.520 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout fsid = 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:23:21.520 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-09T08:23:21.520 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-09T08:23:21.520 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-09T08:23:21.520 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-09T08:23:21.520 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-05-09T08:23:21.520 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-09T08:23:21.520 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-09T08:23:21.521 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-05-09T08:23:21.521 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [osd] 2024-05-09T08:23:21.521 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-05-09T08:23:21.521 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-09T08:23:21.521 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-09T08:23:21.521 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-09T08:23:21.521 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-09T08:23:21.521 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-09T08:23:21.521 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.657+0000 7f1b2a598700 1 Processor -- start 2024-05-09T08:23:21.521 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.657+0000 7f1b2a598700 1 -- start start 2024-05-09T08:23:21.521 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.657+0000 7f1b2a598700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2407e710 0x7f1b2407cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:21.522 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.657+0000 7f1b2a598700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1b2407d140 con 0x7f1b2407e710 2024-05-09T08:23:21.522 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.657+0000 7f1b23fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2407e710 0x7f1b2407cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:21.522 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.657+0000 7f1b23fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2407e710 0x7f1b2407cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41180/0 (socket says 172.21.15.107:41180) 2024-05-09T08:23:21.522 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.657+0000 7f1b23fff700 1 -- 172.21.15.107:0/3944118207 learned_addr learned my addr 172.21.15.107:0/3944118207 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:21.522 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.658+0000 7f1b23fff700 1 -- 172.21.15.107:0/3944118207 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b2407d2c0 con 0x7f1b2407e710 2024-05-09T08:23:21.522 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.658+0000 7f1b23fff700 1 --2- 172.21.15.107:0/3944118207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2407e710 0x7f1b2407cb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1b140098a0 tx=0x7f1b14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca0f6fb4cea3076d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:21.522 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.658+0000 7f1b22ffd700 1 -- 172.21.15.107:0/3944118207 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b14004160 con 0x7f1b2407e710 2024-05-09T08:23:21.522 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.658+0000 7f1b22ffd700 1 -- 172.21.15.107:0/3944118207 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1b14004300 con 0x7f1b2407e710 2024-05-09T08:23:21.522 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.658+0000 7f1b22ffd700 1 -- 172.21.15.107:0/3944118207 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b14011a80 con 0x7f1b2407e710 2024-05-09T08:23:21.522 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.659+0000 7f1b2a598700 1 -- 172.21.15.107:0/3944118207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2407e710 msgr2=0x7f1b2407cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:21.522 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.659+0000 7f1b2a598700 1 --2- 172.21.15.107:0/3944118207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2407e710 0x7f1b2407cb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1b140098a0 tx=0x7f1b14009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:21.523 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.659+0000 7f1b2a598700 1 -- 172.21.15.107:0/3944118207 shutdown_connections 2024-05-09T08:23:21.523 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.659+0000 7f1b2a598700 1 --2- 172.21.15.107:0/3944118207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2407e710 0x7f1b2407cb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:21.523 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.659+0000 7f1b2a598700 1 -- 172.21.15.107:0/3944118207 >> 172.21.15.107:0/3944118207 conn(0x7f1b24103440 msgr2=0x7f1b24105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:21.523 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.659+0000 7f1b2a598700 1 -- 172.21.15.107:0/3944118207 shutdown_connections 2024-05-09T08:23:21.523 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.659+0000 7f1b2a598700 1 -- 172.21.15.107:0/3944118207 wait complete. 2024-05-09T08:23:21.523 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.659+0000 7f1b2a598700 1 Processor -- start 2024-05-09T08:23:21.523 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.659+0000 7f1b2a598700 1 -- start start 2024-05-09T08:23:21.523 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.660+0000 7f1b2a598700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2407e710 0x7f1b2419df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:21.523 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.660+0000 7f1b2a598700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1b2410e340 con 0x7f1b2407e710 2024-05-09T08:23:21.523 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.660+0000 7f1b23fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2407e710 0x7f1b2419df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:21.523 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.660+0000 7f1b23fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2407e710 0x7f1b2419df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41182/0 (socket says 172.21.15.107:41182) 2024-05-09T08:23:21.524 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.660+0000 7f1b23fff700 1 -- 172.21.15.107:0/2810445771 learned_addr learned my addr 172.21.15.107:0/2810445771 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:21.524 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.660+0000 7f1b23fff700 1 -- 172.21.15.107:0/2810445771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b2419e460 con 0x7f1b2407e710 2024-05-09T08:23:21.524 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.660+0000 7f1b23fff700 1 --2- 172.21.15.107:0/2810445771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2407e710 0x7f1b2419df20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1b14000c00 tx=0x7f1b14011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:21.524 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.660+0000 7f1b217fa700 1 -- 172.21.15.107:0/2810445771 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b140039f0 con 0x7f1b2407e710 2024-05-09T08:23:21.524 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.660+0000 7f1b217fa700 1 -- 172.21.15.107:0/2810445771 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1b14003b90 con 0x7f1b2407e710 2024-05-09T08:23:21.524 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.660+0000 7f1b2a598700 1 -- 172.21.15.107:0/2810445771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b2419e6f0 con 0x7f1b2407e710 2024-05-09T08:23:21.524 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.660+0000 7f1b217fa700 1 -- 172.21.15.107:0/2810445771 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b1401a740 con 0x7f1b2407e710 2024-05-09T08:23:21.524 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.660+0000 7f1b2a598700 1 -- 172.21.15.107:0/2810445771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b241a13e0 con 0x7f1b2407e710 2024-05-09T08:23:21.524 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.661+0000 7f1b217fa700 1 -- 172.21.15.107:0/2810445771 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f1b1401aa40 con 0x7f1b2407e710 2024-05-09T08:23:21.524 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.661+0000 7f1b217fa700 1 --2- 172.21.15.107:0/2810445771 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f1b0c039000 0x7f1b0c03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:21.524 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.662+0000 7f1b217fa700 1 -- 172.21.15.107:0/2810445771 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1b1404d3c0 con 0x7f1b2407e710 2024-05-09T08:23:21.525 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.662+0000 7f1b2a598700 1 -- 172.21.15.107:0/2810445771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b24063980 con 0x7f1b2407e710 2024-05-09T08:23:21.525 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.662+0000 7f1b237fe700 1 --2- 172.21.15.107:0/2810445771 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f1b0c039000 0x7f1b0c03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:21.525 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.662+0000 7f1b237fe700 1 --2- 172.21.15.107:0/2810445771 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f1b0c039000 0x7f1b0c03b4c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1b18006fd0 tx=0x7f1b18006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:21.525 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.667+0000 7f1b217fa700 1 -- 172.21.15.107:0/2810445771 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1b14020240 con 0x7f1b2407e710 2024-05-09T08:23:21.525 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.861+0000 7f1b2a598700 1 -- 172.21.15.107:0/2810445771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f1b241a2b30 con 0x7f1b2407e710 2024-05-09T08:23:21.525 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.862+0000 7f1b217fa700 1 -- 172.21.15.107:0/2810445771 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+398 (secure 0 0 0) 0x7f1b1403ce20 con 0x7f1b2407e710 2024-05-09T08:23:21.525 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.864+0000 7f1b2a598700 1 -- 172.21.15.107:0/2810445771 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f1b0c039000 msgr2=0x7f1b0c03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:21.525 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.864+0000 7f1b2a598700 1 --2- 172.21.15.107:0/2810445771 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f1b0c039000 0x7f1b0c03b4c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1b18006fd0 tx=0x7f1b18006e40 comp rx=0 tx=0).stop 2024-05-09T08:23:21.525 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.864+0000 7f1b2a598700 1 -- 172.21.15.107:0/2810445771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2407e710 msgr2=0x7f1b2419df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:21.525 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.864+0000 7f1b2a598700 1 --2- 172.21.15.107:0/2810445771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2407e710 0x7f1b2419df20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1b14000c00 tx=0x7f1b14011b00 comp rx=0 tx=0).stop 2024-05-09T08:23:21.525 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.864+0000 7f1b2a598700 1 -- 172.21.15.107:0/2810445771 shutdown_connections 2024-05-09T08:23:21.526 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.864+0000 7f1b2a598700 1 --2- 172.21.15.107:0/2810445771 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f1b0c039000 0x7f1b0c03b4c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:21.526 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.864+0000 7f1b2a598700 1 --2- 172.21.15.107:0/2810445771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b2407e710 0x7f1b2419df20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:21.526 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.864+0000 7f1b2a598700 1 -- 172.21.15.107:0/2810445771 >> 172.21.15.107:0/2810445771 conn(0x7f1b24103440 msgr2=0x7f1b24103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:21.526 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.864+0000 7f1b2a598700 1 -- 172.21.15.107:0/2810445771 shutdown_connections 2024-05-09T08:23:21.526 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:20.864+0000 7f1b2a598700 1 -- 172.21.15.107:0/2810445771 wait complete. 2024-05-09T08:23:21.526 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling cephadm module... 2024-05-09T08:23:22.524 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:22 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3228309463' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-09T08:23:22.524 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:22 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: ignoring --setuser ceph since I am not root 2024-05-09T08:23:22.524 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:22 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: ignoring --setgroup ceph since I am not root 2024-05-09T08:23:22.524 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:22 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:22.361+0000 7f5e9e48e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:23:22.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.877+0000 7f8a11d6f700 1 Processor -- start 2024-05-09T08:23:22.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.877+0000 7f8a11d6f700 1 -- start start 2024-05-09T08:23:22.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.877+0000 7f8a11d6f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a0c107e60 0x7f8a0c108280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:22.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.877+0000 7f8a11d6f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c1087c0 con 0x7f8a0c107e60 2024-05-09T08:23:22.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.877+0000 7f8a0b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a0c107e60 0x7f8a0c108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:22.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.877+0000 7f8a0b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a0c107e60 0x7f8a0c108280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41194/0 (socket says 172.21.15.107:41194) 2024-05-09T08:23:22.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.878+0000 7f8a0b7fe700 1 -- 172.21.15.107:0/2851893959 learned_addr learned my addr 172.21.15.107:0/2851893959 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:22.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.878+0000 7f8a0b7fe700 1 -- 172.21.15.107:0/2851893959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0c108fc0 con 0x7f8a0c107e60 2024-05-09T08:23:22.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.878+0000 7f8a0b7fe700 1 --2- 172.21.15.107:0/2851893959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a0c107e60 0x7f8a0c108280 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009e80 tx=0x7f89fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449de693a2a36e1e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:22.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.878+0000 7f8a0a7fc700 1 -- 172.21.15.107:0/2851893959 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89fc017070 con 0x7f8a0c107e60 2024-05-09T08:23:22.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.878+0000 7f8a0a7fc700 1 -- 172.21.15.107:0/2851893959 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f89fc00b8c0 con 0x7f8a0c107e60 2024-05-09T08:23:22.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.878+0000 7f8a0a7fc700 1 -- 172.21.15.107:0/2851893959 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89fc004520 con 0x7f8a0c107e60 2024-05-09T08:23:22.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.879+0000 7f8a11d6f700 1 -- 172.21.15.107:0/2851893959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a0c107e60 msgr2=0x7f8a0c108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:22.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.879+0000 7f8a11d6f700 1 --2- 172.21.15.107:0/2851893959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a0c107e60 0x7f8a0c108280 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009e80 tx=0x7f89fc00b330 comp rx=0 tx=0).stop 2024-05-09T08:23:22.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.879+0000 7f8a11d6f700 1 -- 172.21.15.107:0/2851893959 shutdown_connections 2024-05-09T08:23:22.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.879+0000 7f8a11d6f700 1 --2- 172.21.15.107:0/2851893959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a0c107e60 0x7f8a0c108280 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:22.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.879+0000 7f8a11d6f700 1 -- 172.21.15.107:0/2851893959 >> 172.21.15.107:0/2851893959 conn(0x7f8a0c103440 msgr2=0x7f8a0c105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:22.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.879+0000 7f8a11d6f700 1 -- 172.21.15.107:0/2851893959 shutdown_connections 2024-05-09T08:23:22.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.879+0000 7f8a11d6f700 1 -- 172.21.15.107:0/2851893959 wait complete. 2024-05-09T08:23:22.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.879+0000 7f8a11d6f700 1 Processor -- start 2024-05-09T08:23:22.776 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.880+0000 7f8a11d6f700 1 -- start start 2024-05-09T08:23:22.776 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.880+0000 7f8a11d6f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a0c107e60 0x7f8a0c19bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:22.776 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.880+0000 7f8a11d6f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c10c280 con 0x7f8a0c107e60 2024-05-09T08:23:22.776 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.880+0000 7f8a0b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a0c107e60 0x7f8a0c19bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:22.776 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.880+0000 7f8a0b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a0c107e60 0x7f8a0c19bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41202/0 (socket says 172.21.15.107:41202) 2024-05-09T08:23:22.776 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.880+0000 7f8a0b7fe700 1 -- 172.21.15.107:0/3228309463 learned_addr learned my addr 172.21.15.107:0/3228309463 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:22.776 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.880+0000 7f8a0b7fe700 1 -- 172.21.15.107:0/3228309463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0c19c180 con 0x7f8a0c107e60 2024-05-09T08:23:22.776 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.880+0000 7f8a0b7fe700 1 --2- 172.21.15.107:0/3228309463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a0c107e60 0x7f8a0c19bc40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009e50 tx=0x7f89fc004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:22.776 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.880+0000 7f8a08ff9700 1 -- 172.21.15.107:0/3228309463 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89fc020020 con 0x7f8a0c107e60 2024-05-09T08:23:22.776 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.880+0000 7f8a08ff9700 1 -- 172.21.15.107:0/3228309463 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f89fc00bba0 con 0x7f8a0c107e60 2024-05-09T08:23:22.776 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.880+0000 7f8a11d6f700 1 -- 172.21.15.107:0/3228309463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a0c19c410 con 0x7f8a0c107e60 2024-05-09T08:23:22.776 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.880+0000 7f8a08ff9700 1 -- 172.21.15.107:0/3228309463 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89fc017040 con 0x7f8a0c107e60 2024-05-09T08:23:22.776 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.880+0000 7f8a11d6f700 1 -- 172.21.15.107:0/3228309463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a0c19f100 con 0x7f8a0c107e60 2024-05-09T08:23:22.776 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.881+0000 7f8a08ff9700 1 -- 172.21.15.107:0/3228309463 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f89fc023020 con 0x7f8a0c107e60 2024-05-09T08:23:22.777 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.881+0000 7f8a11d6f700 1 -- 172.21.15.107:0/3228309463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a0c063980 con 0x7f8a0c107e60 2024-05-09T08:23:22.777 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.881+0000 7f8a08ff9700 1 --2- 172.21.15.107:0/3228309463 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f89f4038fb0 0x7f89f403b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:22.777 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.881+0000 7f8a08ff9700 1 -- 172.21.15.107:0/3228309463 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f89fc00f050 con 0x7f8a0c107e60 2024-05-09T08:23:22.777 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.882+0000 7f8a0affd700 1 --2- 172.21.15.107:0/3228309463 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f89f4038fb0 0x7f89f403b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:22.777 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.882+0000 7f8a0affd700 1 --2- 172.21.15.107:0/3228309463 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f89f4038fb0 0x7f89f403b470 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8a00006fd0 tx=0x7f8a00006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:22.777 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:21.886+0000 7f8a08ff9700 1 -- 172.21.15.107:0/3228309463 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f89fc02a020 con 0x7f8a0c107e60 2024-05-09T08:23:22.777 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:22.139+0000 7f8a11d6f700 1 -- 172.21.15.107:0/3228309463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f8a0c19f7f0 con 0x7f8a0c107e60 2024-05-09T08:23:22.777 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:22.190+0000 7f8a08ff9700 1 -- 172.21.15.107:0/3228309463 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f89fc04cab0 con 0x7f8a0c107e60 2024-05-09T08:23:22.777 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:22.192+0000 7f8a08ff9700 1 -- 172.21.15.107:0/3228309463 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f89fc02d350 con 0x7f8a0c107e60 2024-05-09T08:23:22.777 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:22.193+0000 7f8a11d6f700 1 -- 172.21.15.107:0/3228309463 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f89f4038fb0 msgr2=0x7f89f403b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:22.777 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:22.193+0000 7f8a11d6f700 1 --2- 172.21.15.107:0/3228309463 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f89f4038fb0 0x7f89f403b470 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8a00006fd0 tx=0x7f8a00006e40 comp rx=0 tx=0).stop 2024-05-09T08:23:22.777 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:22.193+0000 7f8a11d6f700 1 -- 172.21.15.107:0/3228309463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a0c107e60 msgr2=0x7f8a0c19bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:22.777 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:22.193+0000 7f8a11d6f700 1 --2- 172.21.15.107:0/3228309463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a0c107e60 0x7f8a0c19bc40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009e50 tx=0x7f89fc004670 comp rx=0 tx=0).stop 2024-05-09T08:23:22.777 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:22.193+0000 7f8a11d6f700 1 -- 172.21.15.107:0/3228309463 shutdown_connections 2024-05-09T08:23:22.778 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:22.193+0000 7f8a11d6f700 1 --2- 172.21.15.107:0/3228309463 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7f89f4038fb0 0x7f89f403b470 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:22.778 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:22.193+0000 7f8a11d6f700 1 --2- 172.21.15.107:0/3228309463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a0c107e60 0x7f8a0c19bc40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:22.778 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:22.193+0000 7f8a11d6f700 1 -- 172.21.15.107:0/3228309463 >> 172.21.15.107:0/3228309463 conn(0x7f8a0c103440 msgr2=0x7f8a0c103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:22.778 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:22.194+0000 7f8a11d6f700 1 -- 172.21.15.107:0/3228309463 shutdown_connections 2024-05-09T08:23:22.778 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:22.194+0000 7f8a11d6f700 1 -- 172.21.15.107:0/3228309463 wait complete. 2024-05-09T08:23:22.780 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:22 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:22.548+0000 7f5e9e48e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:23:22.780 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:22 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:22.649+0000 7f5e9e48e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:23:23.169 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:22 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:22.965+0000 7f5e9e48e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:23:23.169 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:23 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:23.054+0000 7f5e9e48e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:23:23.475 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:23 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3228309463' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-09T08:23:23.475 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:23 smithi107 ceph-mon[30122]: mgrmap e5: x(active, since 11s) 2024-05-09T08:23:24.040 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:23:24.040 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-09T08:23:24.040 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-09T08:23:24.040 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-09T08:23:24.040 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-09T08:23:24.040 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:23:24.040 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.185+0000 7fd7616a5700 1 Processor -- start 2024-05-09T08:23:24.041 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.185+0000 7fd7616a5700 1 -- start start 2024-05-09T08:23:24.041 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.185+0000 7fd7616a5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd75c106d80 0x7fd75c1071a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:24.041 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.185+0000 7fd7616a5700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd75c107770 con 0x7fd75c106d80 2024-05-09T08:23:24.041 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.185+0000 7fd75affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd75c106d80 0x7fd75c1071a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:24.041 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.185+0000 7fd75affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd75c106d80 0x7fd75c1071a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41224/0 (socket says 172.21.15.107:41224) 2024-05-09T08:23:24.041 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.185+0000 7fd75affd700 1 -- 172.21.15.107:0/2023291014 learned_addr learned my addr 172.21.15.107:0/2023291014 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:24.041 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.186+0000 7fd75affd700 1 -- 172.21.15.107:0/2023291014 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd75c107f80 con 0x7fd75c106d80 2024-05-09T08:23:24.041 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.186+0000 7fd75affd700 1 --2- 172.21.15.107:0/2023291014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd75c106d80 0x7fd75c1071a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd7500098a0 tx=0x7fd750009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37abe1f08c4e1d6c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:24.041 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.186+0000 7fd759ffb700 1 -- 172.21.15.107:0/2023291014 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd750004160 con 0x7fd75c106d80 2024-05-09T08:23:24.041 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.186+0000 7fd759ffb700 1 -- 172.21.15.107:0/2023291014 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fd750004300 con 0x7fd75c106d80 2024-05-09T08:23:24.041 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.186+0000 7fd759ffb700 1 -- 172.21.15.107:0/2023291014 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd750011a80 con 0x7fd75c106d80 2024-05-09T08:23:24.041 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.187+0000 7fd7616a5700 1 -- 172.21.15.107:0/2023291014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd75c106d80 msgr2=0x7fd75c1071a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:24.041 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.187+0000 7fd7616a5700 1 --2- 172.21.15.107:0/2023291014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd75c106d80 0x7fd75c1071a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd7500098a0 tx=0x7fd750009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:24.042 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.187+0000 7fd7616a5700 1 -- 172.21.15.107:0/2023291014 shutdown_connections 2024-05-09T08:23:24.042 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.187+0000 7fd7616a5700 1 --2- 172.21.15.107:0/2023291014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd75c106d80 0x7fd75c1071a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:24.042 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.187+0000 7fd7616a5700 1 -- 172.21.15.107:0/2023291014 >> 172.21.15.107:0/2023291014 conn(0x7fd75c102340 msgr2=0x7fd75c104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:24.042 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.187+0000 7fd7616a5700 1 -- 172.21.15.107:0/2023291014 shutdown_connections 2024-05-09T08:23:24.042 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.187+0000 7fd7616a5700 1 -- 172.21.15.107:0/2023291014 wait complete. 2024-05-09T08:23:24.042 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.188+0000 7fd7616a5700 1 Processor -- start 2024-05-09T08:23:24.042 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.188+0000 7fd7616a5700 1 -- start start 2024-05-09T08:23:24.042 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.188+0000 7fd7616a5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd75c106d80 0x7fd75c191e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:24.042 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.188+0000 7fd7616a5700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd75c10b1b0 con 0x7fd75c106d80 2024-05-09T08:23:24.042 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.188+0000 7fd75affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd75c106d80 0x7fd75c191e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:24.042 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.188+0000 7fd75affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd75c106d80 0x7fd75c191e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41240/0 (socket says 172.21.15.107:41240) 2024-05-09T08:23:24.042 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.188+0000 7fd75affd700 1 -- 172.21.15.107:0/2038629798 learned_addr learned my addr 172.21.15.107:0/2038629798 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:24.042 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.188+0000 7fd75affd700 1 -- 172.21.15.107:0/2038629798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd75c192340 con 0x7fd75c106d80 2024-05-09T08:23:24.043 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.189+0000 7fd75affd700 1 --2- 172.21.15.107:0/2038629798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd75c106d80 0x7fd75c191e00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd750006af0 tx=0x7fd750011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:24.043 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.189+0000 7fd743fff700 1 -- 172.21.15.107:0/2038629798 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7500039e0 con 0x7fd75c106d80 2024-05-09T08:23:24.043 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.189+0000 7fd743fff700 1 -- 172.21.15.107:0/2038629798 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fd750003b80 con 0x7fd75c106d80 2024-05-09T08:23:24.043 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.189+0000 7fd7616a5700 1 -- 172.21.15.107:0/2038629798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd75c190530 con 0x7fd75c106d80 2024-05-09T08:23:24.043 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.189+0000 7fd743fff700 1 -- 172.21.15.107:0/2038629798 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7500196a0 con 0x7fd75c106d80 2024-05-09T08:23:24.043 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.189+0000 7fd7616a5700 1 -- 172.21.15.107:0/2038629798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd75c190a70 con 0x7fd75c106d80 2024-05-09T08:23:24.043 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.190+0000 7fd743fff700 1 -- 172.21.15.107:0/2038629798 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fd750019950 con 0x7fd75c106d80 2024-05-09T08:23:24.043 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.190+0000 7fd7616a5700 1 -- 172.21.15.107:0/2038629798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd75c063980 con 0x7fd75c106d80 2024-05-09T08:23:24.043 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.190+0000 7fd743fff700 1 --2- 172.21.15.107:0/2038629798 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fd744039050 0x7fd74403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:24.043 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.190+0000 7fd743fff700 1 -- 172.21.15.107:0/2038629798 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd75004d6a0 con 0x7fd75c106d80 2024-05-09T08:23:24.043 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.190+0000 7fd75a7fc700 1 -- 172.21.15.107:0/2038629798 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fd744039050 msgr2=0x7fd74403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/2360407889 2024-05-09T08:23:24.043 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.190+0000 7fd75a7fc700 1 --2- 172.21.15.107:0/2038629798 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fd744039050 0x7fd74403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:23:24.043 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.195+0000 7fd743fff700 1 -- 172.21.15.107:0/2038629798 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd75004d0c0 con 0x7fd75c106d80 2024-05-09T08:23:24.044 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.391+0000 7fd75a7fc700 1 -- 172.21.15.107:0/2038629798 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fd744039050 msgr2=0x7fd74403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/2360407889 2024-05-09T08:23:24.044 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.391+0000 7fd75a7fc700 1 --2- 172.21.15.107:0/2038629798 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fd744039050 0x7fd74403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:23:24.044 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.473+0000 7fd7616a5700 1 -- 172.21.15.107:0/2038629798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fd75c195ba0 con 0x7fd75c106d80 2024-05-09T08:23:24.044 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.474+0000 7fd743fff700 1 -- 172.21.15.107:0/2038629798 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fd750019d40 con 0x7fd75c106d80 2024-05-09T08:23:24.044 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.479+0000 7fd7616a5700 1 -- 172.21.15.107:0/2038629798 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fd744039050 msgr2=0x7fd74403b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:23:24.044 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.479+0000 7fd7616a5700 1 --2- 172.21.15.107:0/2038629798 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fd744039050 0x7fd74403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:24.044 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.479+0000 7fd7616a5700 1 -- 172.21.15.107:0/2038629798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd75c106d80 msgr2=0x7fd75c191e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:24.044 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.479+0000 7fd7616a5700 1 --2- 172.21.15.107:0/2038629798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd75c106d80 0x7fd75c191e00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd750006af0 tx=0x7fd750011c40 comp rx=0 tx=0).stop 2024-05-09T08:23:24.044 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.483+0000 7fd7616a5700 1 -- 172.21.15.107:0/2038629798 shutdown_connections 2024-05-09T08:23:24.044 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.483+0000 7fd7616a5700 1 --2- 172.21.15.107:0/2038629798 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fd744039050 0x7fd74403b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:24.044 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.483+0000 7fd7616a5700 1 --2- 172.21.15.107:0/2038629798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd75c106d80 0x7fd75c191e00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:24.044 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.483+0000 7fd7616a5700 1 -- 172.21.15.107:0/2038629798 >> 172.21.15.107:0/2038629798 conn(0x7fd75c102340 msgr2=0x7fd75c1031c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:24.044 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.484+0000 7fd7616a5700 1 -- 172.21.15.107:0/2038629798 shutdown_connections 2024-05-09T08:23:24.044 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:23.485+0000 7fd7616a5700 1 -- 172.21.15.107:0/2038629798 wait complete. 2024-05-09T08:23:24.045 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for the mgr to restart... 2024-05-09T08:23:24.045 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr epoch 5... 2024-05-09T08:23:24.454 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:24 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2038629798' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-09T08:23:24.454 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:24 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:24.280+0000 7f5e9e48e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:23:24.454 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:24 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:24.452+0000 7f5e9e48e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:23:24.760 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:24 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:24.624+0000 7f5e9e48e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:23:25.030 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:24 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:24.917+0000 7f5e9e48e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:23:25.030 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:25 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:25.009+0000 7f5e9e48e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:23:25.403 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:25 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:25.211+0000 7f5e9e48e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:23:25.780 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:25 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:25.408+0000 7f5e9e48e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:23:26.280 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:26 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:26.008+0000 7f5e9e48e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:23:26.726 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:26 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:26.429+0000 7f5e9e48e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:23:26.727 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:26 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:26.525+0000 7f5e9e48e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:23:26.727 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:26 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:26.724+0000 7f5e9e48e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:23:27.495 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:27 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:27.243+0000 7f5e9e48e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:23:27.495 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:27 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:27.335+0000 7f5e9e48e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:23:27.780 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:27 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:27.576+0000 7f5e9e48e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:23:27.781 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:27 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:27.708+0000 7f5e9e48e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:23:28.247 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:27 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:27.804+0000 7f5e9e48e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:23:28.247 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:27 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:27.919+0000 7f5e9e48e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:23:28.247 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:28 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:28.245+0000 7f5e9e48e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:23:29.118 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:28 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:28.845+0000 7f5e9e48e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:23:29.118 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:29 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:29.063+0000 7f5e9e48e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:23:29.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:29 smithi107 ceph-mon[30122]: Active manager daemon x restarted 2024-05-09T08:23:29.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:29 smithi107 ceph-mon[30122]: Activating manager daemon x 2024-05-09T08:23:29.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:29 smithi107 ceph-mon[30122]: osdmap e2: 0 total, 0 up, 0 in 2024-05-09T08:23:29.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:29 smithi107 ceph-mon[30122]: mgrmap e6: x(active, starting, since 0.00393526s) 2024-05-09T08:23:29.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:29 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:23:29.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:29 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:23:29.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:29 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:23:29.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:29 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:23:29.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:29 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:23:29.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:29 smithi107 ceph-mon[30122]: Manager daemon x is now available 2024-05-09T08:23:29.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:29 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' 2024-05-09T08:23:29.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:29 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' 2024-05-09T08:23:29.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:29 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:23:29.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:29 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:23:29.532 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:29 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:23:30.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:30 smithi107 ceph-mon[30122]: Found migration_current of "None". Setting to last migration. 2024-05-09T08:23:30.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:30 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:23:30.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:30 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' 2024-05-09T08:23:30.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:30 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' 2024-05-09T08:23:30.353 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:30 smithi107 ceph-mon[30122]: mgrmap e7: x(active, since 1.00587s) 2024-05-09T08:23:30.631 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:23:30.631 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-09T08:23:30.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-09T08:23:30.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:23:30.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.484+0000 7fc154e52700 1 Processor -- start 2024-05-09T08:23:30.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.484+0000 7fc154e52700 1 -- start start 2024-05-09T08:23:30.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.484+0000 7fc154e52700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc150107e40 0x7fc150108260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:30.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.484+0000 7fc154e52700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc1501087a0 con 0x7fc150107e40 2024-05-09T08:23:30.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.484+0000 7fc14e59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc150107e40 0x7fc150108260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:30.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.484+0000 7fc14e59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc150107e40 0x7fc150108260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41242/0 (socket says 172.21.15.107:41242) 2024-05-09T08:23:30.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.484+0000 7fc14e59c700 1 -- 172.21.15.107:0/3533951826 learned_addr learned my addr 172.21.15.107:0/3533951826 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:30.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.485+0000 7fc14e59c700 1 -- 172.21.15.107:0/3533951826 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc150108fa0 con 0x7fc150107e40 2024-05-09T08:23:30.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.485+0000 7fc14e59c700 1 --2- 172.21.15.107:0/3533951826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc150107e40 0x7fc150108260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc1400098a0 tx=0x7fc140009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec3268c01269fed server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:30.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.485+0000 7fc14d59a700 1 -- 172.21.15.107:0/3533951826 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc140004160 con 0x7fc150107e40 2024-05-09T08:23:30.632 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.485+0000 7fc14d59a700 1 -- 172.21.15.107:0/3533951826 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fc140004300 con 0x7fc150107e40 2024-05-09T08:23:30.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.485+0000 7fc14d59a700 1 -- 172.21.15.107:0/3533951826 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc140011a80 con 0x7fc150107e40 2024-05-09T08:23:30.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.486+0000 7fc154e52700 1 -- 172.21.15.107:0/3533951826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc150107e40 msgr2=0x7fc150108260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:30.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.486+0000 7fc154e52700 1 --2- 172.21.15.107:0/3533951826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc150107e40 0x7fc150108260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc1400098a0 tx=0x7fc140009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:30.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.486+0000 7fc154e52700 1 -- 172.21.15.107:0/3533951826 shutdown_connections 2024-05-09T08:23:30.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.486+0000 7fc154e52700 1 --2- 172.21.15.107:0/3533951826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc150107e40 0x7fc150108260 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:30.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.486+0000 7fc154e52700 1 -- 172.21.15.107:0/3533951826 >> 172.21.15.107:0/3533951826 conn(0x7fc1501033c0 msgr2=0x7fc150105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:30.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.486+0000 7fc154e52700 1 -- 172.21.15.107:0/3533951826 shutdown_connections 2024-05-09T08:23:30.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.486+0000 7fc154e52700 1 -- 172.21.15.107:0/3533951826 wait complete. 2024-05-09T08:23:30.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.487+0000 7fc154e52700 1 Processor -- start 2024-05-09T08:23:30.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.487+0000 7fc154e52700 1 -- start start 2024-05-09T08:23:30.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.487+0000 7fc154e52700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc150107e40 0x7fc15019bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:30.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.487+0000 7fc154e52700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc15010c260 con 0x7fc150107e40 2024-05-09T08:23:30.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.487+0000 7fc14e59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc150107e40 0x7fc15019bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:30.633 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.487+0000 7fc14e59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc150107e40 0x7fc15019bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41250/0 (socket says 172.21.15.107:41250) 2024-05-09T08:23:30.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.487+0000 7fc14e59c700 1 -- 172.21.15.107:0/1658665197 learned_addr learned my addr 172.21.15.107:0/1658665197 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:30.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.487+0000 7fc14e59c700 1 -- 172.21.15.107:0/1658665197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc15019c1a0 con 0x7fc150107e40 2024-05-09T08:23:30.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.488+0000 7fc14e59c700 1 --2- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc150107e40 0x7fc15019bc60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc140006af0 tx=0x7fc140011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:30.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.488+0000 7fc13f7fe700 1 -- 172.21.15.107:0/1658665197 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1400039e0 con 0x7fc150107e40 2024-05-09T08:23:30.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.488+0000 7fc13f7fe700 1 -- 172.21.15.107:0/1658665197 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fc140003b80 con 0x7fc150107e40 2024-05-09T08:23:30.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.488+0000 7fc154e52700 1 -- 172.21.15.107:0/1658665197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc15019c430 con 0x7fc150107e40 2024-05-09T08:23:30.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.488+0000 7fc13f7fe700 1 -- 172.21.15.107:0/1658665197 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1400196a0 con 0x7fc150107e40 2024-05-09T08:23:30.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.488+0000 7fc154e52700 1 -- 172.21.15.107:0/1658665197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc15019f120 con 0x7fc150107e40 2024-05-09T08:23:30.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.489+0000 7fc13f7fe700 1 -- 172.21.15.107:0/1658665197 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fc1400199a0 con 0x7fc150107e40 2024-05-09T08:23:30.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.489+0000 7fc13f7fe700 1 --2- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fc138039000 0x7fc13803b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:30.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.489+0000 7fc13f7fe700 1 -- 172.21.15.107:0/1658665197 --> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc13803bbd0 con 0x7fc138039000 2024-05-09T08:23:30.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.489+0000 7fc13f7fe700 1 -- 172.21.15.107:0/1658665197 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc14004d630 con 0x7fc150107e40 2024-05-09T08:23:30.634 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.489+0000 7fc14dd9b700 1 -- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fc138039000 msgr2=0x7fc13803b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/2360407889 2024-05-09T08:23:30.635 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.489+0000 7fc14dd9b700 1 --2- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fc138039000 0x7fc13803b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:23:30.635 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.690+0000 7fc14dd9b700 1 -- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fc138039000 msgr2=0x7fc13803b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/2360407889 2024-05-09T08:23:30.635 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:24.690+0000 7fc14dd9b700 1 --2- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fc138039000 0x7fc13803b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:23:30.635 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:25.091+0000 7fc14dd9b700 1 -- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fc138039000 msgr2=0x7fc13803b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/2360407889 2024-05-09T08:23:30.635 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:25.091+0000 7fc14dd9b700 1 --2- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fc138039000 0x7fc13803b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T08:23:30.635 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:25.892+0000 7fc14dd9b700 1 -- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fc138039000 msgr2=0x7fc13803b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/2360407889 2024-05-09T08:23:30.635 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:25.892+0000 7fc14dd9b700 1 --2- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fc138039000 0x7fc13803b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T08:23:30.635 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:27.493+0000 7fc14dd9b700 1 -- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fc138039000 msgr2=0x7fc13803b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/2360407889 2024-05-09T08:23:30.635 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:27.493+0000 7fc14dd9b700 1 --2- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fc138039000 0x7fc13803b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-09T08:23:30.635 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:29.067+0000 7fc13f7fe700 1 -- 172.21.15.107:0/1658665197 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7fc140019ce0 con 0x7fc150107e40 2024-05-09T08:23:30.635 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:29.067+0000 7fc13f7fe700 1 -- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fc138039000 msgr2=0x7fc13803b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:23:30.635 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:29.067+0000 7fc13f7fe700 1 --2- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fc138039000 0x7fc13803b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:30.635 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.068+0000 7fc13f7fe700 1 -- 172.21.15.107:0/1658665197 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7fc14004eb30 con 0x7fc150107e40 2024-05-09T08:23:30.635 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.068+0000 7fc13f7fe700 1 --2- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7fc13803c930 0x7fc13803ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:30.636 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.068+0000 7fc13f7fe700 1 -- 172.21.15.107:0/1658665197 --> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc14004cd40 con 0x7fc13803c930 2024-05-09T08:23:30.636 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.068+0000 7fc14dd9b700 1 --2- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7fc13803c930 0x7fc13803ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:30.636 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.069+0000 7fc14dd9b700 1 --2- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7fc13803c930 0x7fc13803ed20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc144003a10 tx=0x7fc144009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:30.636 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.070+0000 7fc13f7fe700 1 -- 172.21.15.107:0/1658665197 <== mgr.14118 v2:172.21.15.107:6800/3922375261 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fc14004cd40 con 0x7fc13803c930 2024-05-09T08:23:30.636 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.077+0000 7fc154e52700 1 -- 172.21.15.107:0/1658665197 --> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fc150063980 con 0x7fc13803c930 2024-05-09T08:23:30.636 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.077+0000 7fc13f7fe700 1 -- 172.21.15.107:0/1658665197 <== mgr.14118 v2:172.21.15.107:6800/3922375261 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fc150063980 con 0x7fc13803c930 2024-05-09T08:23:30.636 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.077+0000 7fc154e52700 1 -- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7fc13803c930 msgr2=0x7fc13803ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:30.636 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.077+0000 7fc154e52700 1 --2- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7fc13803c930 0x7fc13803ed20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc144003a10 tx=0x7fc144009290 comp rx=0 tx=0).stop 2024-05-09T08:23:30.636 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.077+0000 7fc154e52700 1 -- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc150107e40 msgr2=0x7fc15019bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:30.636 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.077+0000 7fc154e52700 1 --2- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc150107e40 0x7fc15019bc60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc140006af0 tx=0x7fc140011c40 comp rx=0 tx=0).stop 2024-05-09T08:23:30.636 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.077+0000 7fc154e52700 1 -- 172.21.15.107:0/1658665197 shutdown_connections 2024-05-09T08:23:30.636 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.077+0000 7fc154e52700 1 --2- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7fc13803c930 0x7fc13803ed20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:30.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.077+0000 7fc154e52700 1 --2- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc150107e40 0x7fc15019bc60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:30.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.077+0000 7fc154e52700 1 --2- 172.21.15.107:0/1658665197 >> [v2:172.21.15.107:6800/2360407889,v1:172.21.15.107:6801/2360407889] conn(0x7fc138039000 0x7fc13803b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:30.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.077+0000 7fc154e52700 1 -- 172.21.15.107:0/1658665197 >> 172.21.15.107:0/1658665197 conn(0x7fc1501033c0 msgr2=0x7fc150103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:30.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.077+0000 7fc154e52700 1 -- 172.21.15.107:0/1658665197 shutdown_connections 2024-05-09T08:23:30.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:30.077+0000 7fc154e52700 1 -- 172.21.15.107:0/1658665197 wait complete. 2024-05-09T08:23:30.637 INFO:teuthology.orchestra.run.smithi107.stdout:mgr epoch 5 is available 2024-05-09T08:23:30.637 INFO:teuthology.orchestra.run.smithi107.stdout:Setting orchestrator backend to cephadm... 2024-05-09T08:23:31.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:31 smithi107 ceph-mon[30122]: [09/May/2024:08:23:30] ENGINE Bus STARTING 2024-05-09T08:23:31.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:31 smithi107 ceph-mon[30122]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-09T08:23:31.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:31 smithi107 ceph-mon[30122]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-09T08:23:31.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:31 smithi107 ceph-mon[30122]: [09/May/2024:08:23:30] ENGINE Serving on http://172.21.15.107:8765 2024-05-09T08:23:31.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:31 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:23:31.893 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.057+0000 7f339359e700 1 Processor -- start 2024-05-09T08:23:31.893 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.057+0000 7f339359e700 1 -- start start 2024-05-09T08:23:31.893 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.057+0000 7f339359e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3394107cb0 0x7f33941080d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:31.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.057+0000 7f339359e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f33941086a0 con 0x7f3394107cb0 2024-05-09T08:23:31.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.057+0000 7f339259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3394107cb0 0x7f33941080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:31.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.057+0000 7f339259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3394107cb0 0x7f33941080d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59664/0 (socket says 172.21.15.107:59664) 2024-05-09T08:23:31.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.057+0000 7f339259c700 1 -- 172.21.15.107:0/1606750261 learned_addr learned my addr 172.21.15.107:0/1606750261 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:31.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.057+0000 7f339259c700 1 -- 172.21.15.107:0/1606750261 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3394108eb0 con 0x7f3394107cb0 2024-05-09T08:23:31.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.058+0000 7f339259c700 1 --2- 172.21.15.107:0/1606750261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3394107cb0 0x7f33941080d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f33840098a0 tx=0x7f3384009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378efd12bad69cc8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:31.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.058+0000 7f339159a700 1 -- 172.21.15.107:0/1606750261 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3384004160 con 0x7f3394107cb0 2024-05-09T08:23:31.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.058+0000 7f339159a700 1 -- 172.21.15.107:0/1606750261 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f3384004300 con 0x7f3394107cb0 2024-05-09T08:23:31.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.058+0000 7f339159a700 1 -- 172.21.15.107:0/1606750261 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3384011a80 con 0x7f3394107cb0 2024-05-09T08:23:31.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.058+0000 7f339359e700 1 -- 172.21.15.107:0/1606750261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3394107cb0 msgr2=0x7f33941080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:31.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.058+0000 7f339359e700 1 --2- 172.21.15.107:0/1606750261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3394107cb0 0x7f33941080d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f33840098a0 tx=0x7f3384009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:31.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.058+0000 7f339359e700 1 -- 172.21.15.107:0/1606750261 shutdown_connections 2024-05-09T08:23:31.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.058+0000 7f339359e700 1 --2- 172.21.15.107:0/1606750261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3394107cb0 0x7f33941080d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:31.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.058+0000 7f339359e700 1 -- 172.21.15.107:0/1606750261 >> 172.21.15.107:0/1606750261 conn(0x7f3394103270 msgr2=0x7f3394105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:31.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.058+0000 7f339359e700 1 -- 172.21.15.107:0/1606750261 shutdown_connections 2024-05-09T08:23:31.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.058+0000 7f339359e700 1 -- 172.21.15.107:0/1606750261 wait complete. 2024-05-09T08:23:31.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.059+0000 7f339359e700 1 Processor -- start 2024-05-09T08:23:31.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.059+0000 7f339359e700 1 -- start start 2024-05-09T08:23:31.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.059+0000 7f339359e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3394107cb0 0x7f339419bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:31.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.059+0000 7f339359e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f339410c0e0 con 0x7f3394107cb0 2024-05-09T08:23:31.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.059+0000 7f339259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3394107cb0 0x7f339419bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:31.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.059+0000 7f339259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3394107cb0 0x7f339419bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59674/0 (socket says 172.21.15.107:59674) 2024-05-09T08:23:31.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.059+0000 7f339259c700 1 -- 172.21.15.107:0/822578086 learned_addr learned my addr 172.21.15.107:0/822578086 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:31.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.060+0000 7f339259c700 1 -- 172.21.15.107:0/822578086 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f339419c070 con 0x7f3394107cb0 2024-05-09T08:23:31.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.060+0000 7f339259c700 1 --2- 172.21.15.107:0/822578086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3394107cb0 0x7f339419bb30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3384006af0 tx=0x7f3384011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:31.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.060+0000 7f33837fe700 1 -- 172.21.15.107:0/822578086 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33840039e0 con 0x7f3394107cb0 2024-05-09T08:23:31.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.060+0000 7f33837fe700 1 -- 172.21.15.107:0/822578086 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f3384003b80 con 0x7f3394107cb0 2024-05-09T08:23:31.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.060+0000 7f339359e700 1 -- 172.21.15.107:0/822578086 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f339419c300 con 0x7f3394107cb0 2024-05-09T08:23:31.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.060+0000 7f33837fe700 1 -- 172.21.15.107:0/822578086 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33840196a0 con 0x7f3394107cb0 2024-05-09T08:23:31.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.060+0000 7f339359e700 1 -- 172.21.15.107:0/822578086 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f339419eff0 con 0x7f3394107cb0 2024-05-09T08:23:31.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.061+0000 7f33837fe700 1 -- 172.21.15.107:0/822578086 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f3384019950 con 0x7f3394107cb0 2024-05-09T08:23:31.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.061+0000 7f339359e700 1 -- 172.21.15.107:0/822578086 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3394063980 con 0x7f3394107cb0 2024-05-09T08:23:31.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.061+0000 7f33837fe700 1 --2- 172.21.15.107:0/822578086 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f337c038f20 0x7f337c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:31.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.061+0000 7f33837fe700 1 -- 172.21.15.107:0/822578086 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f338404d500 con 0x7f3394107cb0 2024-05-09T08:23:31.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.061+0000 7f3391d9b700 1 --2- 172.21.15.107:0/822578086 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f337c038f20 0x7f337c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:31.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.062+0000 7f3391d9b700 1 --2- 172.21.15.107:0/822578086 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f337c038f20 0x7f337c03b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3388006fd0 tx=0x7f3388006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:31.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.067+0000 7f33837fe700 1 -- 172.21.15.107:0/822578086 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3384019d40 con 0x7f3394107cb0 2024-05-09T08:23:31.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.120+0000 7f33837fe700 1 -- 172.21.15.107:0/822578086 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f338402d960 con 0x7f3394107cb0 2024-05-09T08:23:31.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.282+0000 7f339359e700 1 -- 172.21.15.107:0/822578086 --> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f33941044c0 con 0x7f337c038f20 2024-05-09T08:23:31.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.288+0000 7f33837fe700 1 -- 172.21.15.107:0/822578086 <== mgr.14118 v2:172.21.15.107:6800/3922375261 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f33941044c0 con 0x7f337c038f20 2024-05-09T08:23:31.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.290+0000 7f339359e700 1 -- 172.21.15.107:0/822578086 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f337c038f20 msgr2=0x7f337c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:31.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.290+0000 7f339359e700 1 --2- 172.21.15.107:0/822578086 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f337c038f20 0x7f337c03b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3388006fd0 tx=0x7f3388006e40 comp rx=0 tx=0).stop 2024-05-09T08:23:31.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.290+0000 7f339359e700 1 -- 172.21.15.107:0/822578086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3394107cb0 msgr2=0x7f339419bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:31.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.290+0000 7f339359e700 1 --2- 172.21.15.107:0/822578086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3394107cb0 0x7f339419bb30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3384006af0 tx=0x7f3384011c40 comp rx=0 tx=0).stop 2024-05-09T08:23:31.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.290+0000 7f339359e700 1 -- 172.21.15.107:0/822578086 shutdown_connections 2024-05-09T08:23:31.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.290+0000 7f339359e700 1 --2- 172.21.15.107:0/822578086 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f337c038f20 0x7f337c03b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:31.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.290+0000 7f339359e700 1 --2- 172.21.15.107:0/822578086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3394107cb0 0x7f339419bb30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:31.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.290+0000 7f339359e700 1 -- 172.21.15.107:0/822578086 >> 172.21.15.107:0/822578086 conn(0x7f3394103270 msgr2=0x7f3394103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:31.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.290+0000 7f339359e700 1 -- 172.21.15.107:0/822578086 shutdown_connections 2024-05-09T08:23:31.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:31.290+0000 7f339359e700 1 -- 172.21.15.107:0/822578086 wait complete. 2024-05-09T08:23:32.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:32 smithi107 ceph-mon[30122]: [09/May/2024:08:23:30] ENGINE Serving on https://172.21.15.107:7150 2024-05-09T08:23:32.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:32 smithi107 ceph-mon[30122]: [09/May/2024:08:23:30] ENGINE Bus STARTED 2024-05-09T08:23:32.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:32 smithi107 ceph-mon[30122]: mgrmap e8: x(active, since 2s) 2024-05-09T08:23:32.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:32 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' 2024-05-09T08:23:32.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:32 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:23:33.151 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:33 smithi107 ceph-mon[30122]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:23:33.160 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-09T08:23:33.160 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.329+0000 7f68b69b4700 1 Processor -- start 2024-05-09T08:23:33.160 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.329+0000 7f68b69b4700 1 -- start start 2024-05-09T08:23:33.160 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.329+0000 7f68b69b4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b0108330 0x7f68b0108750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:33.161 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.329+0000 7f68b69b4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f68b0108d20 con 0x7f68b0108330 2024-05-09T08:23:33.161 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.329+0000 7f68affff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b0108330 0x7f68b0108750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:33.161 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.329+0000 7f68affff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b0108330 0x7f68b0108750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59680/0 (socket says 172.21.15.107:59680) 2024-05-09T08:23:33.161 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.329+0000 7f68affff700 1 -- 172.21.15.107:0/2520748945 learned_addr learned my addr 172.21.15.107:0/2520748945 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:33.161 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.330+0000 7f68affff700 1 -- 172.21.15.107:0/2520748945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68b0106f90 con 0x7f68b0108330 2024-05-09T08:23:33.161 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.330+0000 7f68affff700 1 --2- 172.21.15.107:0/2520748945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b0108330 0x7f68b0108750 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f68a00098a0 tx=0x7f68a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=698b2e0e3ace2e8d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:33.161 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.330+0000 7f68aeffd700 1 -- 172.21.15.107:0/2520748945 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68a0004160 con 0x7f68b0108330 2024-05-09T08:23:33.161 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.330+0000 7f68aeffd700 1 -- 172.21.15.107:0/2520748945 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f68a0004300 con 0x7f68b0108330 2024-05-09T08:23:33.162 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.330+0000 7f68aeffd700 1 -- 172.21.15.107:0/2520748945 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68a0011a80 con 0x7f68b0108330 2024-05-09T08:23:33.163 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.330+0000 7f68b69b4700 1 -- 172.21.15.107:0/2520748945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b0108330 msgr2=0x7f68b0108750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:33.163 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.330+0000 7f68b69b4700 1 --2- 172.21.15.107:0/2520748945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b0108330 0x7f68b0108750 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f68a00098a0 tx=0x7f68a0009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:33.163 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.331+0000 7f68b69b4700 1 -- 172.21.15.107:0/2520748945 shutdown_connections 2024-05-09T08:23:33.163 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.331+0000 7f68b69b4700 1 --2- 172.21.15.107:0/2520748945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b0108330 0x7f68b0108750 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:33.163 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.331+0000 7f68b69b4700 1 -- 172.21.15.107:0/2520748945 >> 172.21.15.107:0/2520748945 conn(0x7f68b0101aa0 msgr2=0x7f68b0103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:33.163 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.331+0000 7f68b69b4700 1 -- 172.21.15.107:0/2520748945 shutdown_connections 2024-05-09T08:23:33.163 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.331+0000 7f68b69b4700 1 -- 172.21.15.107:0/2520748945 wait complete. 2024-05-09T08:23:33.163 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.331+0000 7f68b69b4700 1 Processor -- start 2024-05-09T08:23:33.164 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.331+0000 7f68b69b4700 1 -- start start 2024-05-09T08:23:33.164 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.332+0000 7f68b69b4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b0108330 0x7f68b0106900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:33.164 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.332+0000 7f68b69b4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f68b010b1d0 con 0x7f68b0108330 2024-05-09T08:23:33.164 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.332+0000 7f68affff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b0108330 0x7f68b0106900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:33.164 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.332+0000 7f68affff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b0108330 0x7f68b0106900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59682/0 (socket says 172.21.15.107:59682) 2024-05-09T08:23:33.164 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.332+0000 7f68affff700 1 -- 172.21.15.107:0/3061098922 learned_addr learned my addr 172.21.15.107:0/3061098922 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:33.164 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.332+0000 7f68affff700 1 -- 172.21.15.107:0/3061098922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68b019ea10 con 0x7f68b0108330 2024-05-09T08:23:33.164 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.332+0000 7f68affff700 1 --2- 172.21.15.107:0/3061098922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b0108330 0x7f68b0106900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f68a0000c00 tx=0x7f68a0011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:33.164 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.332+0000 7f68ad7fa700 1 -- 172.21.15.107:0/3061098922 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68a00039f0 con 0x7f68b0108330 2024-05-09T08:23:33.164 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.332+0000 7f68ad7fa700 1 -- 172.21.15.107:0/3061098922 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f68a0003b90 con 0x7f68b0108330 2024-05-09T08:23:33.164 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.332+0000 7f68b69b4700 1 -- 172.21.15.107:0/3061098922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68b019ebe0 con 0x7f68b0108330 2024-05-09T08:23:33.165 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.332+0000 7f68ad7fa700 1 -- 172.21.15.107:0/3061098922 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68a001a740 con 0x7f68b0108330 2024-05-09T08:23:33.165 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.332+0000 7f68b69b4700 1 -- 172.21.15.107:0/3061098922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68b019ef80 con 0x7f68b0108330 2024-05-09T08:23:33.165 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.333+0000 7f68ad7fa700 1 -- 172.21.15.107:0/3061098922 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f68a001a9f0 con 0x7f68b0108330 2024-05-09T08:23:33.165 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.333+0000 7f68b69b4700 1 -- 172.21.15.107:0/3061098922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68b0063980 con 0x7f68b0108330 2024-05-09T08:23:33.165 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.334+0000 7f68ad7fa700 1 --2- 172.21.15.107:0/3061098922 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f68980390a0 0x7f689803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:33.165 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.334+0000 7f68ad7fa700 1 -- 172.21.15.107:0/3061098922 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f68a004d520 con 0x7f68b0108330 2024-05-09T08:23:33.165 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.334+0000 7f68af7fe700 1 --2- 172.21.15.107:0/3061098922 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f68980390a0 0x7f689803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:33.165 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.334+0000 7f68af7fe700 1 --2- 172.21.15.107:0/3061098922 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f68980390a0 0x7f689803b560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f68a4006fd0 tx=0x7f68a4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:33.165 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.339+0000 7f68ad7fa700 1 -- 172.21.15.107:0/3061098922 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f68a0018e00 con 0x7f68b0108330 2024-05-09T08:23:33.165 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.535+0000 7f68b69b4700 1 -- 172.21.15.107:0/3061098922 --> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f68b0102c30 con 0x7f68980390a0 2024-05-09T08:23:33.165 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.535+0000 7f68ad7fa700 1 -- 172.21.15.107:0/3061098922 <== mgr.14118 v2:172.21.15.107:6800/3922375261 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f68b0102c30 con 0x7f68980390a0 2024-05-09T08:23:33.166 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.537+0000 7f68b69b4700 1 -- 172.21.15.107:0/3061098922 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f68980390a0 msgr2=0x7f689803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:33.166 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.537+0000 7f68b69b4700 1 --2- 172.21.15.107:0/3061098922 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f68980390a0 0x7f689803b560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f68a4006fd0 tx=0x7f68a4006e40 comp rx=0 tx=0).stop 2024-05-09T08:23:33.166 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.537+0000 7f68b69b4700 1 -- 172.21.15.107:0/3061098922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b0108330 msgr2=0x7f68b0106900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:33.166 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.537+0000 7f68b69b4700 1 --2- 172.21.15.107:0/3061098922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b0108330 0x7f68b0106900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f68a0000c00 tx=0x7f68a0011b00 comp rx=0 tx=0).stop 2024-05-09T08:23:33.166 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.538+0000 7f68b69b4700 1 -- 172.21.15.107:0/3061098922 shutdown_connections 2024-05-09T08:23:33.166 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.538+0000 7f68b69b4700 1 --2- 172.21.15.107:0/3061098922 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f68980390a0 0x7f689803b560 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:33.166 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.538+0000 7f68b69b4700 1 --2- 172.21.15.107:0/3061098922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68b0108330 0x7f68b0106900 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:33.166 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.538+0000 7f68b69b4700 1 -- 172.21.15.107:0/3061098922 >> 172.21.15.107:0/3061098922 conn(0x7f68b0101aa0 msgr2=0x7f68b0102520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:33.166 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.538+0000 7f68b69b4700 1 -- 172.21.15.107:0/3061098922 shutdown_connections 2024-05-09T08:23:33.166 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:32.538+0000 7f68b69b4700 1 -- 172.21.15.107:0/3061098922 wait complete. 2024-05-09T08:23:33.166 INFO:teuthology.orchestra.run.smithi107.stdout:Generating ssh key... 2024-05-09T08:23:34.031 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: Generating public/private rsa key pair. 2024-05-09T08:23:34.031 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: Your identification has been saved in /tmp/tmpgfmd0h72/key. 2024-05-09T08:23:34.031 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: Your public key has been saved in /tmp/tmpgfmd0h72/key.pub. 2024-05-09T08:23:34.031 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: The key fingerprint is: 2024-05-09T08:23:34.031 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: SHA256:HuOBSxHbOgEaUzgIigJ+r0jJyPmqWrQ0YT4jaAb8QqU ceph-3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:23:34.031 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: The key's randomart image is: 2024-05-09T08:23:34.031 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: +---[RSA 3072]----+ 2024-05-09T08:23:34.031 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: |+.o+o . | 2024-05-09T08:23:34.031 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: |B =+ . + | 2024-05-09T08:23:34.032 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: |=E+o + . | 2024-05-09T08:23:34.032 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: |Oo=.. = | 2024-05-09T08:23:34.032 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: |+XB. .= S | 2024-05-09T08:23:34.032 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: |+++=.. = + | 2024-05-09T08:23:34.032 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: | .oo . o | 2024-05-09T08:23:34.032 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: | .. | 2024-05-09T08:23:34.032 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: |=. | 2024-05-09T08:23:34.032 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:33 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: +----[SHA256]-----+ 2024-05-09T08:23:34.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:34 smithi107 ceph-mon[30122]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:23:34.400 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:34 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' 2024-05-09T08:23:34.400 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:34 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' 2024-05-09T08:23:34.408 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.543+0000 7f289e00d700 1 Processor -- start 2024-05-09T08:23:34.408 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.543+0000 7f289e00d700 1 -- start start 2024-05-09T08:23:34.409 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.543+0000 7f289e00d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289807e750 0x7f289807cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:34.409 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.543+0000 7f289e00d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f289807d180 con 0x7f289807e750 2024-05-09T08:23:34.409 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.543+0000 7f28977fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289807e750 0x7f289807cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:34.409 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.543+0000 7f28977fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289807e750 0x7f289807cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59690/0 (socket says 172.21.15.107:59690) 2024-05-09T08:23:34.409 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.543+0000 7f28977fe700 1 -- 172.21.15.107:0/1452436326 learned_addr learned my addr 172.21.15.107:0/1452436326 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:34.409 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.543+0000 7f28977fe700 1 -- 172.21.15.107:0/1452436326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f289807d300 con 0x7f289807e750 2024-05-09T08:23:34.409 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.544+0000 7f28977fe700 1 --2- 172.21.15.107:0/1452436326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289807e750 0x7f289807cbb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f28880098a0 tx=0x7f2888009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19a9d784546175c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:34.409 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.544+0000 7f28967fc700 1 -- 172.21.15.107:0/1452436326 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2888004160 con 0x7f289807e750 2024-05-09T08:23:34.409 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.544+0000 7f28967fc700 1 -- 172.21.15.107:0/1452436326 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2888004300 con 0x7f289807e750 2024-05-09T08:23:34.410 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.544+0000 7f28967fc700 1 -- 172.21.15.107:0/1452436326 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2888011a80 con 0x7f289807e750 2024-05-09T08:23:34.410 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.544+0000 7f289e00d700 1 -- 172.21.15.107:0/1452436326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289807e750 msgr2=0x7f289807cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:34.410 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.544+0000 7f289e00d700 1 --2- 172.21.15.107:0/1452436326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289807e750 0x7f289807cbb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f28880098a0 tx=0x7f2888009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:34.410 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.544+0000 7f289e00d700 1 -- 172.21.15.107:0/1452436326 shutdown_connections 2024-05-09T08:23:34.410 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.544+0000 7f289e00d700 1 --2- 172.21.15.107:0/1452436326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289807e750 0x7f289807cbb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:34.410 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.544+0000 7f289e00d700 1 -- 172.21.15.107:0/1452436326 >> 172.21.15.107:0/1452436326 conn(0x7f2898103440 msgr2=0x7f2898105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:34.410 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.545+0000 7f289e00d700 1 -- 172.21.15.107:0/1452436326 shutdown_connections 2024-05-09T08:23:34.410 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.545+0000 7f289e00d700 1 -- 172.21.15.107:0/1452436326 wait complete. 2024-05-09T08:23:34.410 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.545+0000 7f289e00d700 1 Processor -- start 2024-05-09T08:23:34.410 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.545+0000 7f289e00d700 1 -- start start 2024-05-09T08:23:34.410 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.545+0000 7f289e00d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289807e750 0x7f289819df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:34.410 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.545+0000 7f289e00d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f289810e350 con 0x7f289807e750 2024-05-09T08:23:34.410 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.546+0000 7f28977fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289807e750 0x7f289819df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:34.411 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.546+0000 7f28977fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289807e750 0x7f289819df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59706/0 (socket says 172.21.15.107:59706) 2024-05-09T08:23:34.411 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.546+0000 7f28977fe700 1 -- 172.21.15.107:0/1735562689 learned_addr learned my addr 172.21.15.107:0/1735562689 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:34.411 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.546+0000 7f28977fe700 1 -- 172.21.15.107:0/1735562689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f289819e440 con 0x7f289807e750 2024-05-09T08:23:34.411 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.546+0000 7f28977fe700 1 --2- 172.21.15.107:0/1735562689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289807e750 0x7f289819df00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2888011740 tx=0x7f2888011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:34.411 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.546+0000 7f2894ff9700 1 -- 172.21.15.107:0/1735562689 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2888003b70 con 0x7f289807e750 2024-05-09T08:23:34.411 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.546+0000 7f2894ff9700 1 -- 172.21.15.107:0/1735562689 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f288801a8c0 con 0x7f289807e750 2024-05-09T08:23:34.411 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.546+0000 7f289e00d700 1 -- 172.21.15.107:0/1735562689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f289819e6d0 con 0x7f289807e750 2024-05-09T08:23:34.411 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.546+0000 7f289e00d700 1 -- 172.21.15.107:0/1735562689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28981a13c0 con 0x7f289807e750 2024-05-09T08:23:34.411 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.546+0000 7f2894ff9700 1 -- 172.21.15.107:0/1735562689 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2888024800 con 0x7f289807e750 2024-05-09T08:23:34.411 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.547+0000 7f2894ff9700 1 -- 172.21.15.107:0/1735562689 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f2888022020 con 0x7f289807e750 2024-05-09T08:23:34.411 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.547+0000 7f289e00d700 1 -- 172.21.15.107:0/1735562689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2884005360 con 0x7f289807e750 2024-05-09T08:23:34.411 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.547+0000 7f2894ff9700 1 --2- 172.21.15.107:0/1735562689 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f2880039050 0x7f288003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:34.411 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.547+0000 7f2894ff9700 1 -- 172.21.15.107:0/1735562689 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f288804d3d0 con 0x7f289807e750 2024-05-09T08:23:34.411 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.547+0000 7f2896ffd700 1 --2- 172.21.15.107:0/1735562689 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f2880039050 0x7f288003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:34.412 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.548+0000 7f2896ffd700 1 --2- 172.21.15.107:0/1735562689 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f2880039050 0x7f288003b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f288c006fd0 tx=0x7f288c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:34.412 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.552+0000 7f2894ff9700 1 -- 172.21.15.107:0/1735562689 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f288800b3e0 con 0x7f289807e750 2024-05-09T08:23:34.412 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.748+0000 7f289e00d700 1 -- 172.21.15.107:0/1735562689 --> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f2884000bf0 con 0x7f2880039050 2024-05-09T08:23:34.412 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.872+0000 7f2894ff9700 1 -- 172.21.15.107:0/1735562689 <== mgr.14118 v2:172.21.15.107:6800/3922375261 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2884000bf0 con 0x7f2880039050 2024-05-09T08:23:34.412 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.874+0000 7f289e00d700 1 -- 172.21.15.107:0/1735562689 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f2880039050 msgr2=0x7f288003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:34.412 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.874+0000 7f289e00d700 1 --2- 172.21.15.107:0/1735562689 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f2880039050 0x7f288003b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f288c006fd0 tx=0x7f288c006e40 comp rx=0 tx=0).stop 2024-05-09T08:23:34.412 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.874+0000 7f289e00d700 1 -- 172.21.15.107:0/1735562689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289807e750 msgr2=0x7f289819df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:34.412 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.874+0000 7f289e00d700 1 --2- 172.21.15.107:0/1735562689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289807e750 0x7f289819df00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2888011740 tx=0x7f2888011b00 comp rx=0 tx=0).stop 2024-05-09T08:23:34.412 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.874+0000 7f289e00d700 1 -- 172.21.15.107:0/1735562689 shutdown_connections 2024-05-09T08:23:34.412 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.874+0000 7f289e00d700 1 --2- 172.21.15.107:0/1735562689 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f2880039050 0x7f288003b510 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:34.412 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.874+0000 7f289e00d700 1 --2- 172.21.15.107:0/1735562689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289807e750 0x7f289819df00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:34.412 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.874+0000 7f289e00d700 1 -- 172.21.15.107:0/1735562689 >> 172.21.15.107:0/1735562689 conn(0x7f2898103440 msgr2=0x7f2898103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:34.412 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.874+0000 7f289e00d700 1 -- 172.21.15.107:0/1735562689 shutdown_connections 2024-05-09T08:23:34.412 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:33.874+0000 7f289e00d700 1 -- 172.21.15.107:0/1735562689 wait complete. 2024-05-09T08:23:35.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:35 smithi107 ceph-mon[30122]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:23:35.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:35 smithi107 ceph-mon[30122]: Generating ssh key... 2024-05-09T08:23:35.623 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDvTcCDkTv5fUm8+6hFqg8pxKA6RI6aMD6TeJ/E+TBwDuZ4DjQZ7eKDaSdTx/xjw5Yd8Uf5MaG47X7yclkmoOFeEVb1qyrjXLM3+L7GMak2BrY/X9Ab641f3geXD1wcNbIra5XncAf1oZvisw5Wk0XNkDhErEIpw7LPHSZU1cMGBrqL4YCirmvwF+rMEBRAsd0gCqarwLgCOu3ls/13WMwLfIdUkzqfcpLYVMeMjRyFwAltslJGH53+w3RfYdYQOqQ6w+eIr2dkHOWewShY9MbHJ1VBslK6xqD3nb95+oDtICsdaCn7sNDq/38vx9wX56n2LffC9Lq3G4eqrSMsi3fksxbc71jNSOSphys/pvk1947Va5njIggp+H8il/5389ZRuYn/H187OMk3TBBDC8JOpSQoY8RI+e8ncFjQOp/AfUFRzgNTIdbPOZwHtPsyJWVjTIJTLj9NsrXFTjaGr8qFvltLhMd7xYRCaA2MjZkN7TlgJP3HMRSmDUDCCWqQDCU= ceph-3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:23:35.623 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.800+0000 7f0b383d8700 1 Processor -- start 2024-05-09T08:23:35.623 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.800+0000 7f0b383d8700 1 -- start start 2024-05-09T08:23:35.623 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.800+0000 7f0b383d8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b30106d70 0x7f0b30107190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:35.623 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.800+0000 7f0b383d8700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0b30107760 con 0x7f0b30106d70 2024-05-09T08:23:35.623 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.800+0000 7f0b36174700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b30106d70 0x7f0b30107190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:35.623 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.800+0000 7f0b36174700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b30106d70 0x7f0b30107190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59716/0 (socket says 172.21.15.107:59716) 2024-05-09T08:23:35.624 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.800+0000 7f0b36174700 1 -- 172.21.15.107:0/1424561686 learned_addr learned my addr 172.21.15.107:0/1424561686 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:35.624 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.801+0000 7f0b36174700 1 -- 172.21.15.107:0/1424561686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b30107f70 con 0x7f0b30106d70 2024-05-09T08:23:35.624 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.801+0000 7f0b36174700 1 --2- 172.21.15.107:0/1424561686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b30106d70 0x7f0b30107190 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00bb40 tx=0x7f0b2c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3493e62961887dba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:35.624 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.801+0000 7f0b35172700 1 -- 172.21.15.107:0/1424561686 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b2c00dbe0 con 0x7f0b30106d70 2024-05-09T08:23:35.624 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.801+0000 7f0b35172700 1 -- 172.21.15.107:0/1424561686 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0b2c0044c0 con 0x7f0b30106d70 2024-05-09T08:23:35.624 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.801+0000 7f0b35172700 1 -- 172.21.15.107:0/1424561686 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b2c010470 con 0x7f0b30106d70 2024-05-09T08:23:35.624 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.801+0000 7f0b383d8700 1 -- 172.21.15.107:0/1424561686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b30106d70 msgr2=0x7f0b30107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:35.624 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.801+0000 7f0b383d8700 1 --2- 172.21.15.107:0/1424561686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b30106d70 0x7f0b30107190 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00bb40 tx=0x7f0b2c00be50 comp rx=0 tx=0).stop 2024-05-09T08:23:35.624 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.801+0000 7f0b383d8700 1 -- 172.21.15.107:0/1424561686 shutdown_connections 2024-05-09T08:23:35.624 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.801+0000 7f0b383d8700 1 --2- 172.21.15.107:0/1424561686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b30106d70 0x7f0b30107190 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:35.624 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.801+0000 7f0b383d8700 1 -- 172.21.15.107:0/1424561686 >> 172.21.15.107:0/1424561686 conn(0x7f0b30102350 msgr2=0x7f0b30104770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:35.625 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.802+0000 7f0b383d8700 1 -- 172.21.15.107:0/1424561686 shutdown_connections 2024-05-09T08:23:35.625 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.802+0000 7f0b383d8700 1 -- 172.21.15.107:0/1424561686 wait complete. 2024-05-09T08:23:35.625 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.802+0000 7f0b383d8700 1 Processor -- start 2024-05-09T08:23:35.625 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.802+0000 7f0b383d8700 1 -- start start 2024-05-09T08:23:35.625 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.802+0000 7f0b383d8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b30106d70 0x7f0b30199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:35.625 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.802+0000 7f0b383d8700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0b30108e00 con 0x7f0b30106d70 2024-05-09T08:23:35.625 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.802+0000 7f0b36174700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b30106d70 0x7f0b30199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:35.625 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.802+0000 7f0b36174700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b30106d70 0x7f0b30199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59730/0 (socket says 172.21.15.107:59730) 2024-05-09T08:23:35.625 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.802+0000 7f0b36174700 1 -- 172.21.15.107:0/2812357204 learned_addr learned my addr 172.21.15.107:0/2812357204 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:35.625 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.803+0000 7f0b36174700 1 -- 172.21.15.107:0/2812357204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b3019a050 con 0x7f0b30106d70 2024-05-09T08:23:35.625 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.803+0000 7f0b36174700 1 --2- 172.21.15.107:0/2812357204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b30106d70 0x7f0b30199b10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00b340 tx=0x7f0b2c003fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:35.625 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.803+0000 7f0b277fe700 1 -- 172.21.15.107:0/2812357204 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b2c027070 con 0x7f0b30106d70 2024-05-09T08:23:35.626 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.803+0000 7f0b277fe700 1 -- 172.21.15.107:0/2812357204 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0b2c00de20 con 0x7f0b30106d70 2024-05-09T08:23:35.626 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.803+0000 7f0b383d8700 1 -- 172.21.15.107:0/2812357204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b3019a2e0 con 0x7f0b30106d70 2024-05-09T08:23:35.626 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.803+0000 7f0b277fe700 1 -- 172.21.15.107:0/2812357204 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b2c022a20 con 0x7f0b30106d70 2024-05-09T08:23:35.626 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.803+0000 7f0b383d8700 1 -- 172.21.15.107:0/2812357204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b3019cfd0 con 0x7f0b30106d70 2024-05-09T08:23:35.626 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.804+0000 7f0b277fe700 1 -- 172.21.15.107:0/2812357204 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f0b2c02a070 con 0x7f0b30106d70 2024-05-09T08:23:35.626 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.804+0000 7f0b383d8700 1 -- 172.21.15.107:0/2812357204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b30063980 con 0x7f0b30106d70 2024-05-09T08:23:35.626 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.804+0000 7f0b277fe700 1 --2- 172.21.15.107:0/2812357204 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f0b1c0390a0 0x7f0b1c03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:35.626 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.804+0000 7f0b277fe700 1 -- 172.21.15.107:0/2812357204 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0b2c01f070 con 0x7f0b30106d70 2024-05-09T08:23:35.626 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.804+0000 7f0b35973700 1 --2- 172.21.15.107:0/2812357204 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f0b1c0390a0 0x7f0b1c03b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:35.626 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.805+0000 7f0b35973700 1 --2- 172.21.15.107:0/2812357204 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f0b1c0390a0 0x7f0b1c03b560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0b20006fd0 tx=0x7f0b20006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:35.626 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:34.809+0000 7f0b277fe700 1 -- 172.21.15.107:0/2812357204 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0b2c015070 con 0x7f0b30106d70 2024-05-09T08:23:35.626 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:35.003+0000 7f0b383d8700 1 -- 172.21.15.107:0/2812357204 --> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f0b30062760 con 0x7f0b1c0390a0 2024-05-09T08:23:35.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:35.004+0000 7f0b277fe700 1 -- 172.21.15.107:0/2812357204 <== mgr.14118 v2:172.21.15.107:6800/3922375261 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f0b30062760 con 0x7f0b1c0390a0 2024-05-09T08:23:35.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:35.006+0000 7f0b383d8700 1 -- 172.21.15.107:0/2812357204 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f0b1c0390a0 msgr2=0x7f0b1c03b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:35.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:35.006+0000 7f0b383d8700 1 --2- 172.21.15.107:0/2812357204 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f0b1c0390a0 0x7f0b1c03b560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0b20006fd0 tx=0x7f0b20006e40 comp rx=0 tx=0).stop 2024-05-09T08:23:35.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:35.006+0000 7f0b383d8700 1 -- 172.21.15.107:0/2812357204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b30106d70 msgr2=0x7f0b30199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:35.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:35.006+0000 7f0b383d8700 1 --2- 172.21.15.107:0/2812357204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b30106d70 0x7f0b30199b10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00b340 tx=0x7f0b2c003fe0 comp rx=0 tx=0).stop 2024-05-09T08:23:35.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:35.006+0000 7f0b383d8700 1 -- 172.21.15.107:0/2812357204 shutdown_connections 2024-05-09T08:23:35.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:35.006+0000 7f0b383d8700 1 --2- 172.21.15.107:0/2812357204 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f0b1c0390a0 0x7f0b1c03b560 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:35.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:35.006+0000 7f0b383d8700 1 --2- 172.21.15.107:0/2812357204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b30106d70 0x7f0b30199b10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:35.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:35.006+0000 7f0b383d8700 1 -- 172.21.15.107:0/2812357204 >> 172.21.15.107:0/2812357204 conn(0x7f0b30102350 msgr2=0x7f0b30190130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:35.627 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:35.006+0000 7f0b383d8700 1 -- 172.21.15.107:0/2812357204 shutdown_connections 2024-05-09T08:23:35.628 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:35.006+0000 7f0b383d8700 1 -- 172.21.15.107:0/2812357204 wait complete. 2024-05-09T08:23:35.628 INFO:teuthology.orchestra.run.smithi107.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-09T08:23:35.628 INFO:teuthology.orchestra.run.smithi107.stdout:Adding key to root@localhost authorized_keys... 2024-05-09T08:23:35.628 INFO:teuthology.orchestra.run.smithi107.stdout:Adding host smithi107... 2024-05-09T08:23:36.233 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:36 smithi107 ceph-mon[30122]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:23:37.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:37 smithi107 ceph-mon[30122]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi107", "addr": "172.21.15.107", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:23:38.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:38 smithi107 ceph-mon[30122]: Deploying cephadm binary to smithi107 2024-05-09T08:23:40.881 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Added host 'smithi107' with addr '172.21.15.107' 2024-05-09T08:23:40.881 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.023+0000 7efdd5c9d700 1 Processor -- start 2024-05-09T08:23:40.881 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.023+0000 7efdd5c9d700 1 -- start start 2024-05-09T08:23:40.881 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.024+0000 7efdd5c9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0107e30 0x7efdd0108250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:40.881 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.024+0000 7efdd5c9d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efdd0108790 con 0x7efdd0107e30 2024-05-09T08:23:40.881 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.024+0000 7efdcf7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0107e30 0x7efdd0108250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:40.881 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.024+0000 7efdcf7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0107e30 0x7efdd0108250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59746/0 (socket says 172.21.15.107:59746) 2024-05-09T08:23:40.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.024+0000 7efdcf7fe700 1 -- 172.21.15.107:0/2916572174 learned_addr learned my addr 172.21.15.107:0/2916572174 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:40.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.024+0000 7efdcf7fe700 1 -- 172.21.15.107:0/2916572174 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd0108f90 con 0x7efdd0107e30 2024-05-09T08:23:40.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.024+0000 7efdcf7fe700 1 --2- 172.21.15.107:0/2916572174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0107e30 0x7efdd0108250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efdc00098a0 tx=0x7efdc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4613d116e5ba427b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:40.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.024+0000 7efdce7fc700 1 -- 172.21.15.107:0/2916572174 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdc0004160 con 0x7efdd0107e30 2024-05-09T08:23:40.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.024+0000 7efdce7fc700 1 -- 172.21.15.107:0/2916572174 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7efdc0004300 con 0x7efdd0107e30 2024-05-09T08:23:40.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.024+0000 7efdce7fc700 1 -- 172.21.15.107:0/2916572174 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdc0011a80 con 0x7efdd0107e30 2024-05-09T08:23:40.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.025+0000 7efdd5c9d700 1 -- 172.21.15.107:0/2916572174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0107e30 msgr2=0x7efdd0108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:40.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.025+0000 7efdd5c9d700 1 --2- 172.21.15.107:0/2916572174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0107e30 0x7efdd0108250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efdc00098a0 tx=0x7efdc0009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:40.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.025+0000 7efdd5c9d700 1 -- 172.21.15.107:0/2916572174 shutdown_connections 2024-05-09T08:23:40.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.025+0000 7efdd5c9d700 1 --2- 172.21.15.107:0/2916572174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0107e30 0x7efdd0108250 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:40.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.025+0000 7efdd5c9d700 1 -- 172.21.15.107:0/2916572174 >> 172.21.15.107:0/2916572174 conn(0x7efdd01033b0 msgr2=0x7efdd0105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:40.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.025+0000 7efdd5c9d700 1 -- 172.21.15.107:0/2916572174 shutdown_connections 2024-05-09T08:23:40.882 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.025+0000 7efdd5c9d700 1 -- 172.21.15.107:0/2916572174 wait complete. 2024-05-09T08:23:40.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.026+0000 7efdd5c9d700 1 Processor -- start 2024-05-09T08:23:40.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.026+0000 7efdd5c9d700 1 -- start start 2024-05-09T08:23:40.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.026+0000 7efdd5c9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0107e30 0x7efdd019bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:40.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.026+0000 7efdd5c9d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efdd010c250 con 0x7efdd0107e30 2024-05-09T08:23:40.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.026+0000 7efdcf7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0107e30 0x7efdd019bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:40.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.026+0000 7efdcf7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0107e30 0x7efdd019bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59762/0 (socket says 172.21.15.107:59762) 2024-05-09T08:23:40.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.026+0000 7efdcf7fe700 1 -- 172.21.15.107:0/2042474401 learned_addr learned my addr 172.21.15.107:0/2042474401 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:40.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.027+0000 7efdcf7fe700 1 -- 172.21.15.107:0/2042474401 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd019c1c0 con 0x7efdd0107e30 2024-05-09T08:23:40.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.027+0000 7efdcf7fe700 1 --2- 172.21.15.107:0/2042474401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0107e30 0x7efdd019bc80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efdc0011a50 tx=0x7efdc0011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:40.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.027+0000 7efdccff9700 1 -- 172.21.15.107:0/2042474401 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdc0003a80 con 0x7efdd0107e30 2024-05-09T08:23:40.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.027+0000 7efdccff9700 1 -- 172.21.15.107:0/2042474401 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7efdc0003c20 con 0x7efdd0107e30 2024-05-09T08:23:40.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.027+0000 7efdd5c9d700 1 -- 172.21.15.107:0/2042474401 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdd019c450 con 0x7efdd0107e30 2024-05-09T08:23:40.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.027+0000 7efdccff9700 1 -- 172.21.15.107:0/2042474401 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdc001a650 con 0x7efdd0107e30 2024-05-09T08:23:40.883 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.027+0000 7efdd5c9d700 1 -- 172.21.15.107:0/2042474401 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdd019f140 con 0x7efdd0107e30 2024-05-09T08:23:40.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.028+0000 7efdccff9700 1 -- 172.21.15.107:0/2042474401 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7efdc001a900 con 0x7efdd0107e30 2024-05-09T08:23:40.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.028+0000 7efdd5c9d700 1 -- 172.21.15.107:0/2042474401 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdd0063980 con 0x7efdd0107e30 2024-05-09T08:23:40.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.028+0000 7efdccff9700 1 --2- 172.21.15.107:0/2042474401 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7efdb80390a0 0x7efdb803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:40.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.028+0000 7efdccff9700 1 -- 172.21.15.107:0/2042474401 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7efdc004d4d0 con 0x7efdd0107e30 2024-05-09T08:23:40.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.028+0000 7efdceffd700 1 --2- 172.21.15.107:0/2042474401 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7efdb80390a0 0x7efdb803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:40.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.029+0000 7efdceffd700 1 --2- 172.21.15.107:0/2042474401 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7efdb80390a0 0x7efdb803b560 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efdc4006fd0 tx=0x7efdc4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:40.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.033+0000 7efdccff9700 1 -- 172.21.15.107:0/2042474401 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efdc0019c70 con 0x7efdd0107e30 2024-05-09T08:23:40.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:36.232+0000 7efdd5c9d700 1 -- 172.21.15.107:0/2042474401 --> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi107", "addr": "172.21.15.107", "target": ["mon-mgr", ""]}) -- 0x7efdd01045c0 con 0x7efdb80390a0 2024-05-09T08:23:40.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:40.318+0000 7efdccff9700 1 -- 172.21.15.107:0/2042474401 <== mgr.14118 v2:172.21.15.107:6800/3922375261 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7efdd01045c0 con 0x7efdb80390a0 2024-05-09T08:23:40.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:40.320+0000 7efdd5c9d700 1 -- 172.21.15.107:0/2042474401 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7efdb80390a0 msgr2=0x7efdb803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:40.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:40.320+0000 7efdd5c9d700 1 --2- 172.21.15.107:0/2042474401 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7efdb80390a0 0x7efdb803b560 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efdc4006fd0 tx=0x7efdc4006e40 comp rx=0 tx=0).stop 2024-05-09T08:23:40.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:40.320+0000 7efdd5c9d700 1 -- 172.21.15.107:0/2042474401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0107e30 msgr2=0x7efdd019bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:40.884 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:40.320+0000 7efdd5c9d700 1 --2- 172.21.15.107:0/2042474401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0107e30 0x7efdd019bc80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efdc0011a50 tx=0x7efdc0011b30 comp rx=0 tx=0).stop 2024-05-09T08:23:40.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:40.320+0000 7efdd5c9d700 1 -- 172.21.15.107:0/2042474401 shutdown_connections 2024-05-09T08:23:40.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:40.320+0000 7efdd5c9d700 1 --2- 172.21.15.107:0/2042474401 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7efdb80390a0 0x7efdb803b560 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:40.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:40.320+0000 7efdd5c9d700 1 --2- 172.21.15.107:0/2042474401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0107e30 0x7efdd019bc80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:40.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:40.320+0000 7efdd5c9d700 1 -- 172.21.15.107:0/2042474401 >> 172.21.15.107:0/2042474401 conn(0x7efdd01033b0 msgr2=0x7efdd0103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:40.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:40.321+0000 7efdd5c9d700 1 -- 172.21.15.107:0/2042474401 shutdown_connections 2024-05-09T08:23:40.885 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:40.321+0000 7efdd5c9d700 1 -- 172.21.15.107:0/2042474401 wait complete. 2024-05-09T08:23:40.885 INFO:teuthology.orchestra.run.smithi107.stdout:Deploying unmanaged mon service... 2024-05-09T08:23:41.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:41 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' 2024-05-09T08:23:41.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:41 smithi107 ceph-mon[30122]: Added host smithi107 2024-05-09T08:23:41.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:41 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:23:42.770 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-09T08:23:42.770 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.396+0000 7f3807258700 1 Processor -- start 2024-05-09T08:23:42.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.397+0000 7f3807258700 1 -- start start 2024-05-09T08:23:42.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.397+0000 7f3807258700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3800107280 0x7f38001076a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:42.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.397+0000 7f3807258700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3800107c70 con 0x7f3800107280 2024-05-09T08:23:42.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.397+0000 7f3804ff4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3800107280 0x7f38001076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:42.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.397+0000 7f3804ff4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3800107280 0x7f38001076a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36488/0 (socket says 172.21.15.107:36488) 2024-05-09T08:23:42.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.397+0000 7f3804ff4700 1 -- 172.21.15.107:0/3227271834 learned_addr learned my addr 172.21.15.107:0/3227271834 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:42.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.397+0000 7f3804ff4700 1 -- 172.21.15.107:0/3227271834 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3800107fa0 con 0x7f3800107280 2024-05-09T08:23:42.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.397+0000 7f3804ff4700 1 --2- 172.21.15.107:0/3227271834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3800107280 0x7f38001076a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f37f40098a0 tx=0x7f37f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9075d1960c540c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:42.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.398+0000 7f37ff7fe700 1 -- 172.21.15.107:0/3227271834 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37f4004160 con 0x7f3800107280 2024-05-09T08:23:42.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.398+0000 7f37ff7fe700 1 -- 172.21.15.107:0/3227271834 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f37f4004300 con 0x7f3800107280 2024-05-09T08:23:42.771 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.398+0000 7f37ff7fe700 1 -- 172.21.15.107:0/3227271834 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37f4011a80 con 0x7f3800107280 2024-05-09T08:23:42.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.398+0000 7f3807258700 1 -- 172.21.15.107:0/3227271834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3800107280 msgr2=0x7f38001076a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:42.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.398+0000 7f3807258700 1 --2- 172.21.15.107:0/3227271834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3800107280 0x7f38001076a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f37f40098a0 tx=0x7f37f4009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:42.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.398+0000 7f3807258700 1 -- 172.21.15.107:0/3227271834 shutdown_connections 2024-05-09T08:23:42.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.398+0000 7f3807258700 1 --2- 172.21.15.107:0/3227271834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3800107280 0x7f38001076a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:42.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.398+0000 7f3807258700 1 -- 172.21.15.107:0/3227271834 >> 172.21.15.107:0/3227271834 conn(0x7f3800101ac0 msgr2=0x7f3800103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:42.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.398+0000 7f3807258700 1 -- 172.21.15.107:0/3227271834 shutdown_connections 2024-05-09T08:23:42.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.398+0000 7f3807258700 1 -- 172.21.15.107:0/3227271834 wait complete. 2024-05-09T08:23:42.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.399+0000 7f3807258700 1 Processor -- start 2024-05-09T08:23:42.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.399+0000 7f3807258700 1 -- start start 2024-05-09T08:23:42.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.399+0000 7f3807258700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3800107280 0x7f380019bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:42.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.399+0000 7f3807258700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f380010b1d0 con 0x7f3800107280 2024-05-09T08:23:42.772 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.399+0000 7f3804ff4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3800107280 0x7f380019bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:42.773 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.399+0000 7f3804ff4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3800107280 0x7f380019bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36496/0 (socket says 172.21.15.107:36496) 2024-05-09T08:23:42.773 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.399+0000 7f3804ff4700 1 -- 172.21.15.107:0/1398631476 learned_addr learned my addr 172.21.15.107:0/1398631476 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:42.773 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.399+0000 7f3804ff4700 1 -- 172.21.15.107:0/1398631476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f380019c200 con 0x7f3800107280 2024-05-09T08:23:42.773 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.399+0000 7f3804ff4700 1 --2- 172.21.15.107:0/1398631476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3800107280 0x7f380019bcc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f37f4006af0 tx=0x7f37f4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:42.773 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.400+0000 7f37fdffb700 1 -- 172.21.15.107:0/1398631476 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37f40039e0 con 0x7f3800107280 2024-05-09T08:23:42.773 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.400+0000 7f37fdffb700 1 -- 172.21.15.107:0/1398631476 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f37f4003b80 con 0x7f3800107280 2024-05-09T08:23:42.773 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.400+0000 7f3807258700 1 -- 172.21.15.107:0/1398631476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f380019c490 con 0x7f3800107280 2024-05-09T08:23:42.773 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.400+0000 7f37fdffb700 1 -- 172.21.15.107:0/1398631476 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37f40196a0 con 0x7f3800107280 2024-05-09T08:23:42.773 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.400+0000 7f3807258700 1 -- 172.21.15.107:0/1398631476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f380019f180 con 0x7f3800107280 2024-05-09T08:23:42.773 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.401+0000 7f37fdffb700 1 -- 172.21.15.107:0/1398631476 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f37f401ab50 con 0x7f3800107280 2024-05-09T08:23:42.773 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.401+0000 7f3807258700 1 -- 172.21.15.107:0/1398631476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f380010b350 con 0x7f3800107280 2024-05-09T08:23:42.773 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.401+0000 7f37fdffb700 1 --2- 172.21.15.107:0/1398631476 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f37f0039000 0x7f37f003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:42.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.401+0000 7f37fdffb700 1 -- 172.21.15.107:0/1398631476 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f37f404d5b0 con 0x7f3800107280 2024-05-09T08:23:42.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.401+0000 7f37fffff700 1 --2- 172.21.15.107:0/1398631476 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f37f0039000 0x7f37f003b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:42.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.401+0000 7f37fffff700 1 --2- 172.21.15.107:0/1398631476 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f37f0039000 0x7f37f003b4c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f37ec006fd0 tx=0x7f37ec006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:42.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.406+0000 7f37fdffb700 1 -- 172.21.15.107:0/1398631476 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f37f400b3e0 con 0x7f3800107280 2024-05-09T08:23:42.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.635+0000 7f3807258700 1 -- 172.21.15.107:0/1398631476 --> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f380002df80 con 0x7f37f0039000 2024-05-09T08:23:42.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.641+0000 7f37fdffb700 1 -- 172.21.15.107:0/1398631476 <== mgr.14118 v2:172.21.15.107:6800/3922375261 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f380002df80 con 0x7f37f0039000 2024-05-09T08:23:42.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.643+0000 7f3807258700 1 -- 172.21.15.107:0/1398631476 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f37f0039000 msgr2=0x7f37f003b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:42.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.643+0000 7f3807258700 1 --2- 172.21.15.107:0/1398631476 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f37f0039000 0x7f37f003b4c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f37ec006fd0 tx=0x7f37ec006e40 comp rx=0 tx=0).stop 2024-05-09T08:23:42.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.643+0000 7f3807258700 1 -- 172.21.15.107:0/1398631476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3800107280 msgr2=0x7f380019bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:42.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.643+0000 7f3807258700 1 --2- 172.21.15.107:0/1398631476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3800107280 0x7f380019bcc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f37f4006af0 tx=0x7f37f4011c40 comp rx=0 tx=0).stop 2024-05-09T08:23:42.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.643+0000 7f3807258700 1 -- 172.21.15.107:0/1398631476 shutdown_connections 2024-05-09T08:23:42.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.643+0000 7f3807258700 1 --2- 172.21.15.107:0/1398631476 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f37f0039000 0x7f37f003b4c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:42.774 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.643+0000 7f3807258700 1 --2- 172.21.15.107:0/1398631476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3800107280 0x7f380019bcc0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:42.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.643+0000 7f3807258700 1 -- 172.21.15.107:0/1398631476 >> 172.21.15.107:0/1398631476 conn(0x7f3800101ac0 msgr2=0x7f38001028a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:42.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.643+0000 7f3807258700 1 -- 172.21.15.107:0/1398631476 shutdown_connections 2024-05-09T08:23:42.775 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:41.643+0000 7f3807258700 1 -- 172.21.15.107:0/1398631476 wait complete. 2024-05-09T08:23:42.775 INFO:teuthology.orchestra.run.smithi107.stdout:Deploying unmanaged mgr service... 2024-05-09T08:23:42.871 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:42 smithi107 ceph-mon[30122]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:23:42.871 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:42 smithi107 ceph-mon[30122]: Saving service mon spec with placement count:5 2024-05-09T08:23:42.871 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:42 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' 2024-05-09T08:23:44.078 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-09T08:23:44.078 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.126+0000 7faaff7eb700 1 Processor -- start 2024-05-09T08:23:44.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.126+0000 7faaff7eb700 1 -- start start 2024-05-09T08:23:44.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.127+0000 7faaff7eb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8108900 0x7faaf810acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:44.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.127+0000 7faaff7eb700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faaf807c6e0 con 0x7faaf8108900 2024-05-09T08:23:44.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.127+0000 7faafd587700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8108900 0x7faaf810acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:44.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.127+0000 7faafd587700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8108900 0x7faaf810acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36498/0 (socket says 172.21.15.107:36498) 2024-05-09T08:23:44.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.127+0000 7faafd587700 1 -- 172.21.15.107:0/2242449955 learned_addr learned my addr 172.21.15.107:0/2242449955 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:44.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.127+0000 7faafd587700 1 -- 172.21.15.107:0/2242449955 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf810b230 con 0x7faaf8108900 2024-05-09T08:23:44.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.127+0000 7faafd587700 1 --2- 172.21.15.107:0/2242449955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8108900 0x7faaf810acf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faaf40098a0 tx=0x7faaf4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=803109d7a6386dd4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:44.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.127+0000 7faaeffff700 1 -- 172.21.15.107:0/2242449955 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faaf4004160 con 0x7faaf8108900 2024-05-09T08:23:44.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.127+0000 7faaeffff700 1 -- 172.21.15.107:0/2242449955 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7faaf4004300 con 0x7faaf8108900 2024-05-09T08:23:44.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.128+0000 7faaeffff700 1 -- 172.21.15.107:0/2242449955 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faaf4011a80 con 0x7faaf8108900 2024-05-09T08:23:44.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.128+0000 7faaff7eb700 1 -- 172.21.15.107:0/2242449955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8108900 msgr2=0x7faaf810acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:44.079 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.128+0000 7faaff7eb700 1 --2- 172.21.15.107:0/2242449955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8108900 0x7faaf810acf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faaf40098a0 tx=0x7faaf4009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:44.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.128+0000 7faaff7eb700 1 -- 172.21.15.107:0/2242449955 shutdown_connections 2024-05-09T08:23:44.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.128+0000 7faaff7eb700 1 --2- 172.21.15.107:0/2242449955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8108900 0x7faaf810acf0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:44.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.128+0000 7faaff7eb700 1 -- 172.21.15.107:0/2242449955 >> 172.21.15.107:0/2242449955 conn(0x7faaf81022d0 msgr2=0x7faaf8104730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:44.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.128+0000 7faaff7eb700 1 -- 172.21.15.107:0/2242449955 shutdown_connections 2024-05-09T08:23:44.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.128+0000 7faaff7eb700 1 -- 172.21.15.107:0/2242449955 wait complete. 2024-05-09T08:23:44.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.129+0000 7faaff7eb700 1 Processor -- start 2024-05-09T08:23:44.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.129+0000 7faaff7eb700 1 -- start start 2024-05-09T08:23:44.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.129+0000 7faaff7eb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8108900 0x7faaf8108180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:44.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.129+0000 7faaff7eb700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faaf810c110 con 0x7faaf8108900 2024-05-09T08:23:44.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.129+0000 7faafd587700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8108900 0x7faaf8108180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:44.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.129+0000 7faafd587700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8108900 0x7faaf8108180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36510/0 (socket says 172.21.15.107:36510) 2024-05-09T08:23:44.080 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.129+0000 7faafd587700 1 -- 172.21.15.107:0/376965532 learned_addr learned my addr 172.21.15.107:0/376965532 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:44.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.129+0000 7faafd587700 1 -- 172.21.15.107:0/376965532 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf81086c0 con 0x7faaf8108900 2024-05-09T08:23:44.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.130+0000 7faafd587700 1 --2- 172.21.15.107:0/376965532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8108900 0x7faaf8108180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faaf4011a50 tx=0x7faaf4011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:44.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.130+0000 7faaee7fc700 1 -- 172.21.15.107:0/376965532 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faaf4003a80 con 0x7faaf8108900 2024-05-09T08:23:44.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.130+0000 7faaee7fc700 1 -- 172.21.15.107:0/376965532 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7faaf4003c20 con 0x7faaf8108900 2024-05-09T08:23:44.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.130+0000 7faaff7eb700 1 -- 172.21.15.107:0/376965532 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf81068b0 con 0x7faaf8108900 2024-05-09T08:23:44.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.130+0000 7faaee7fc700 1 -- 172.21.15.107:0/376965532 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faaf401a650 con 0x7faaf8108900 2024-05-09T08:23:44.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.130+0000 7faaff7eb700 1 -- 172.21.15.107:0/376965532 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf8106df0 con 0x7faaf8108900 2024-05-09T08:23:44.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.131+0000 7faaee7fc700 1 -- 172.21.15.107:0/376965532 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7faaf401a920 con 0x7faaf8108900 2024-05-09T08:23:44.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.131+0000 7faaff7eb700 1 -- 172.21.15.107:0/376965532 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaf8063980 con 0x7faaf8108900 2024-05-09T08:23:44.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.131+0000 7faaee7fc700 1 --2- 172.21.15.107:0/376965532 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7faae40390a0 0x7faae403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:44.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.131+0000 7faaee7fc700 1 -- 172.21.15.107:0/376965532 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7faaf404d4f0 con 0x7faaf8108900 2024-05-09T08:23:44.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.131+0000 7faafcd86700 1 --2- 172.21.15.107:0/376965532 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7faae40390a0 0x7faae403b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:44.081 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.131+0000 7faafcd86700 1 --2- 172.21.15.107:0/376965532 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7faae40390a0 0x7faae403b560 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faae8006fd0 tx=0x7faae8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:44.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.136+0000 7faaee7fc700 1 -- 172.21.15.107:0/376965532 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faaf4019c70 con 0x7faaf8108900 2024-05-09T08:23:44.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.350+0000 7faaff7eb700 1 -- 172.21.15.107:0/376965532 --> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7faaf8107650 con 0x7faae40390a0 2024-05-09T08:23:44.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.355+0000 7faaee7fc700 1 -- 172.21.15.107:0/376965532 <== mgr.14118 v2:172.21.15.107:6800/3922375261 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7faaf8107650 con 0x7faae40390a0 2024-05-09T08:23:44.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.357+0000 7faaff7eb700 1 -- 172.21.15.107:0/376965532 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7faae40390a0 msgr2=0x7faae403b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:44.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.357+0000 7faaff7eb700 1 --2- 172.21.15.107:0/376965532 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7faae40390a0 0x7faae403b560 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faae8006fd0 tx=0x7faae8006e40 comp rx=0 tx=0).stop 2024-05-09T08:23:44.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.357+0000 7faaff7eb700 1 -- 172.21.15.107:0/376965532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8108900 msgr2=0x7faaf8108180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:44.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.357+0000 7faaff7eb700 1 --2- 172.21.15.107:0/376965532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8108900 0x7faaf8108180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faaf4011a50 tx=0x7faaf4011b30 comp rx=0 tx=0).stop 2024-05-09T08:23:44.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.357+0000 7faaff7eb700 1 -- 172.21.15.107:0/376965532 shutdown_connections 2024-05-09T08:23:44.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.357+0000 7faaff7eb700 1 --2- 172.21.15.107:0/376965532 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7faae40390a0 0x7faae403b560 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:44.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.357+0000 7faaff7eb700 1 --2- 172.21.15.107:0/376965532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8108900 0x7faaf8108180 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:44.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.357+0000 7faaff7eb700 1 -- 172.21.15.107:0/376965532 >> 172.21.15.107:0/376965532 conn(0x7faaf81022d0 msgr2=0x7faaf8102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:44.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.357+0000 7faaff7eb700 1 -- 172.21.15.107:0/376965532 shutdown_connections 2024-05-09T08:23:44.082 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:43.357+0000 7faaff7eb700 1 -- 172.21.15.107:0/376965532 wait complete. 2024-05-09T08:23:44.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:44 smithi107 ceph-mon[30122]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:23:44.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:44 smithi107 ceph-mon[30122]: Saving service mgr spec with placement count:2 2024-05-09T08:23:44.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:44 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' 2024-05-09T08:23:44.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:44 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' 2024-05-09T08:23:44.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:44 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' 2024-05-09T08:23:45.655 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.558+0000 7f1dc14b3700 1 Processor -- start 2024-05-09T08:23:45.655 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.558+0000 7f1dc14b3700 1 -- start start 2024-05-09T08:23:45.655 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.558+0000 7f1dc14b3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dbc106170 0x7f1dbc106590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:45.656 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.558+0000 7f1dc14b3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc106b60 con 0x7f1dbc106170 2024-05-09T08:23:45.656 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.559+0000 7f1dbaffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dbc106170 0x7f1dbc106590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:45.656 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.559+0000 7f1dbaffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dbc106170 0x7f1dbc106590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36512/0 (socket says 172.21.15.107:36512) 2024-05-09T08:23:45.656 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.559+0000 7f1dbaffd700 1 -- 172.21.15.107:0/60294415 learned_addr learned my addr 172.21.15.107:0/60294415 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:45.656 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.559+0000 7f1dbaffd700 1 -- 172.21.15.107:0/60294415 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dbc1091a0 con 0x7f1dbc106170 2024-05-09T08:23:45.656 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.559+0000 7f1dbaffd700 1 --2- 172.21.15.107:0/60294415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dbc106170 0x7f1dbc106590 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0098a0 tx=0x7f1dac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52f511f3633aab6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:45.656 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.559+0000 7f1db9ffb700 1 -- 172.21.15.107:0/60294415 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dac004160 con 0x7f1dbc106170 2024-05-09T08:23:45.656 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.559+0000 7f1db9ffb700 1 -- 172.21.15.107:0/60294415 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1dac004300 con 0x7f1dbc106170 2024-05-09T08:23:45.656 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.560+0000 7f1db9ffb700 1 -- 172.21.15.107:0/60294415 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dac004160 con 0x7f1dbc106170 2024-05-09T08:23:45.656 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.560+0000 7f1dc14b3700 1 -- 172.21.15.107:0/60294415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dbc106170 msgr2=0x7f1dbc106590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:45.656 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.560+0000 7f1dc14b3700 1 --2- 172.21.15.107:0/60294415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dbc106170 0x7f1dbc106590 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0098a0 tx=0x7f1dac009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:45.656 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.561+0000 7f1dc14b3700 1 -- 172.21.15.107:0/60294415 shutdown_connections 2024-05-09T08:23:45.656 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.561+0000 7f1dc14b3700 1 --2- 172.21.15.107:0/60294415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dbc106170 0x7f1dbc106590 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:45.656 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.561+0000 7f1dc14b3700 1 -- 172.21.15.107:0/60294415 >> 172.21.15.107:0/60294415 conn(0x7f1dbc101ac0 msgr2=0x7f1dbc103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:45.657 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.561+0000 7f1dc14b3700 1 -- 172.21.15.107:0/60294415 shutdown_connections 2024-05-09T08:23:45.657 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.561+0000 7f1dc14b3700 1 -- 172.21.15.107:0/60294415 wait complete. 2024-05-09T08:23:45.657 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.561+0000 7f1dc14b3700 1 Processor -- start 2024-05-09T08:23:45.657 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.562+0000 7f1dc14b3700 1 -- start start 2024-05-09T08:23:45.657 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.562+0000 7f1dc14b3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dbc106170 0x7f1dbc19bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:45.657 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.562+0000 7f1dc14b3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc10c3d0 con 0x7f1dbc106170 2024-05-09T08:23:45.657 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.562+0000 7f1dbaffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dbc106170 0x7f1dbc19bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:45.657 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.562+0000 7f1dbaffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dbc106170 0x7f1dbc19bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36514/0 (socket says 172.21.15.107:36514) 2024-05-09T08:23:45.657 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.562+0000 7f1dbaffd700 1 -- 172.21.15.107:0/2827340084 learned_addr learned my addr 172.21.15.107:0/2827340084 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:45.657 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.562+0000 7f1dbaffd700 1 -- 172.21.15.107:0/2827340084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dbc19c1f0 con 0x7f1dbc106170 2024-05-09T08:23:45.657 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.563+0000 7f1dbaffd700 1 --2- 172.21.15.107:0/2827340084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dbc106170 0x7f1dbc19bcb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1dac011710 tx=0x7f1dac011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:45.657 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.563+0000 7f1da3fff700 1 -- 172.21.15.107:0/2827340084 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dac0044b0 con 0x7f1dbc106170 2024-05-09T08:23:45.657 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.563+0000 7f1da3fff700 1 -- 172.21.15.107:0/2827340084 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1dac0038b0 con 0x7f1dbc106170 2024-05-09T08:23:45.658 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.563+0000 7f1da3fff700 1 -- 172.21.15.107:0/2827340084 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dac0245a0 con 0x7f1dbc106170 2024-05-09T08:23:45.658 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.563+0000 7f1dc14b3700 1 -- 172.21.15.107:0/2827340084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dbc19c480 con 0x7f1dbc106170 2024-05-09T08:23:45.658 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.563+0000 7f1dc14b3700 1 -- 172.21.15.107:0/2827340084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dbc19f170 con 0x7f1dbc106170 2024-05-09T08:23:45.658 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.564+0000 7f1da3fff700 1 -- 172.21.15.107:0/2827340084 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f1dac022020 con 0x7f1dbc106170 2024-05-09T08:23:45.658 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.564+0000 7f1da3fff700 1 --2- 172.21.15.107:0/2827340084 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f1da40390c0 0x7f1da403b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:45.658 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.564+0000 7f1da3fff700 1 -- 172.21.15.107:0/2827340084 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1dac04d1b0 con 0x7f1dbc106170 2024-05-09T08:23:45.658 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.564+0000 7f1dc14b3700 1 -- 172.21.15.107:0/2827340084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dbc10c550 con 0x7f1dbc106170 2024-05-09T08:23:45.658 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.564+0000 7f1dba7fc700 1 --2- 172.21.15.107:0/2827340084 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f1da40390c0 0x7f1da403b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:45.658 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.565+0000 7f1dba7fc700 1 --2- 172.21.15.107:0/2827340084 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f1da40390c0 0x7f1da403b580 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1db0006fd0 tx=0x7f1db0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:45.658 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.569+0000 7f1da3fff700 1 -- 172.21.15.107:0/2827340084 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1dac02a2a0 con 0x7f1dbc106170 2024-05-09T08:23:45.658 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.806+0000 7f1dc14b3700 1 -- 172.21.15.107:0/2827340084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f1dbc063980 con 0x7f1dbc106170 2024-05-09T08:23:45.658 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.808+0000 7f1da3fff700 1 -- 172.21.15.107:0/2827340084 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f1dac003ad0 con 0x7f1dbc106170 2024-05-09T08:23:45.658 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.811+0000 7f1dc14b3700 1 -- 172.21.15.107:0/2827340084 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f1da40390c0 msgr2=0x7f1da403b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:45.659 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.811+0000 7f1dc14b3700 1 --2- 172.21.15.107:0/2827340084 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f1da40390c0 0x7f1da403b580 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1db0006fd0 tx=0x7f1db0006e40 comp rx=0 tx=0).stop 2024-05-09T08:23:45.659 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.811+0000 7f1dc14b3700 1 -- 172.21.15.107:0/2827340084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dbc106170 msgr2=0x7f1dbc19bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:45.659 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.811+0000 7f1dc14b3700 1 --2- 172.21.15.107:0/2827340084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dbc106170 0x7f1dbc19bcb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1dac011710 tx=0x7f1dac011cd0 comp rx=0 tx=0).stop 2024-05-09T08:23:45.659 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.811+0000 7f1dc14b3700 1 -- 172.21.15.107:0/2827340084 shutdown_connections 2024-05-09T08:23:45.659 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.811+0000 7f1dc14b3700 1 --2- 172.21.15.107:0/2827340084 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f1da40390c0 0x7f1da403b580 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:45.659 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.812+0000 7f1dc14b3700 1 --2- 172.21.15.107:0/2827340084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1dbc106170 0x7f1dbc19bcb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:45.659 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.812+0000 7f1dc14b3700 1 -- 172.21.15.107:0/2827340084 >> 172.21.15.107:0/2827340084 conn(0x7f1dbc101ac0 msgr2=0x7f1dbc1025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:45.659 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.812+0000 7f1dc14b3700 1 -- 172.21.15.107:0/2827340084 shutdown_connections 2024-05-09T08:23:45.659 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:44.812+0000 7f1dc14b3700 1 -- 172.21.15.107:0/2827340084 wait complete. 2024-05-09T08:23:46.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:45 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2827340084' entity='client.admin' 2024-05-09T08:23:46.939 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.077+0000 7ffb6a69c700 1 Processor -- start 2024-05-09T08:23:46.940 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.077+0000 7ffb6a69c700 1 -- start start 2024-05-09T08:23:46.940 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.077+0000 7ffb6a69c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb64106d70 0x7ffb64107190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:46.940 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.077+0000 7ffb6a69c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffb64107760 con 0x7ffb64106d70 2024-05-09T08:23:46.940 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.077+0000 7ffb63fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb64106d70 0x7ffb64107190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:46.940 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.077+0000 7ffb63fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb64106d70 0x7ffb64107190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36528/0 (socket says 172.21.15.107:36528) 2024-05-09T08:23:46.940 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.077+0000 7ffb63fff700 1 -- 172.21.15.107:0/3675066781 learned_addr learned my addr 172.21.15.107:0/3675066781 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:46.941 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.077+0000 7ffb63fff700 1 -- 172.21.15.107:0/3675066781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb64107f70 con 0x7ffb64106d70 2024-05-09T08:23:46.941 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.078+0000 7ffb63fff700 1 --2- 172.21.15.107:0/3675066781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb64106d70 0x7ffb64107190 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400bb40 tx=0x7ffb5400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=694e8c342274d74b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:46.941 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.078+0000 7ffb62ffd700 1 -- 172.21.15.107:0/3675066781 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb5400dbe0 con 0x7ffb64106d70 2024-05-09T08:23:46.941 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.078+0000 7ffb62ffd700 1 -- 172.21.15.107:0/3675066781 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffb540044e0 con 0x7ffb64106d70 2024-05-09T08:23:46.941 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.078+0000 7ffb62ffd700 1 -- 172.21.15.107:0/3675066781 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb54010470 con 0x7ffb64106d70 2024-05-09T08:23:46.941 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.078+0000 7ffb6a69c700 1 -- 172.21.15.107:0/3675066781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb64106d70 msgr2=0x7ffb64107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:46.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.078+0000 7ffb6a69c700 1 --2- 172.21.15.107:0/3675066781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb64106d70 0x7ffb64107190 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400bb40 tx=0x7ffb5400be50 comp rx=0 tx=0).stop 2024-05-09T08:23:46.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.078+0000 7ffb6a69c700 1 -- 172.21.15.107:0/3675066781 shutdown_connections 2024-05-09T08:23:46.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.079+0000 7ffb6a69c700 1 --2- 172.21.15.107:0/3675066781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb64106d70 0x7ffb64107190 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:46.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.079+0000 7ffb6a69c700 1 -- 172.21.15.107:0/3675066781 >> 172.21.15.107:0/3675066781 conn(0x7ffb64102350 msgr2=0x7ffb64104770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:46.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.079+0000 7ffb6a69c700 1 -- 172.21.15.107:0/3675066781 shutdown_connections 2024-05-09T08:23:46.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.079+0000 7ffb6a69c700 1 -- 172.21.15.107:0/3675066781 wait complete. 2024-05-09T08:23:46.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.079+0000 7ffb6a69c700 1 Processor -- start 2024-05-09T08:23:46.942 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.079+0000 7ffb6a69c700 1 -- start start 2024-05-09T08:23:46.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.080+0000 7ffb6a69c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb64106d70 0x7ffb64199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:46.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.080+0000 7ffb6a69c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6407b840 con 0x7ffb64106d70 2024-05-09T08:23:46.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.080+0000 7ffb63fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb64106d70 0x7ffb64199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:46.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.080+0000 7ffb63fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb64106d70 0x7ffb64199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36544/0 (socket says 172.21.15.107:36544) 2024-05-09T08:23:46.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.080+0000 7ffb63fff700 1 -- 172.21.15.107:0/1839387102 learned_addr learned my addr 172.21.15.107:0/1839387102 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:46.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.080+0000 7ffb63fff700 1 -- 172.21.15.107:0/1839387102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6419a010 con 0x7ffb64106d70 2024-05-09T08:23:46.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.080+0000 7ffb63fff700 1 --2- 172.21.15.107:0/1839387102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb64106d70 0x7ffb64199ad0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400d650 tx=0x7ffb54003fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:46.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.080+0000 7ffb617fa700 1 -- 172.21.15.107:0/1839387102 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb54027070 con 0x7ffb64106d70 2024-05-09T08:23:46.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.080+0000 7ffb6a69c700 1 -- 172.21.15.107:0/1839387102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb6419a2a0 con 0x7ffb64106d70 2024-05-09T08:23:46.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.080+0000 7ffb617fa700 1 -- 172.21.15.107:0/1839387102 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffb54010d50 con 0x7ffb64106d70 2024-05-09T08:23:46.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.080+0000 7ffb617fa700 1 -- 172.21.15.107:0/1839387102 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb54022bd0 con 0x7ffb64106d70 2024-05-09T08:23:46.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.080+0000 7ffb6a69c700 1 -- 172.21.15.107:0/1839387102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb6419a700 con 0x7ffb64106d70 2024-05-09T08:23:46.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.081+0000 7ffb617fa700 1 -- 172.21.15.107:0/1839387102 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7ffb5402a070 con 0x7ffb64106d70 2024-05-09T08:23:46.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.081+0000 7ffb617fa700 1 --2- 172.21.15.107:0/1839387102 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ffb4c039000 0x7ffb4c03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:46.944 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.081+0000 7ffb617fa700 1 -- 172.21.15.107:0/1839387102 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ffb5401f070 con 0x7ffb64106d70 2024-05-09T08:23:46.944 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.081+0000 7ffb6a69c700 1 -- 172.21.15.107:0/1839387102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb64063980 con 0x7ffb64106d70 2024-05-09T08:23:46.944 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.081+0000 7ffb637fe700 1 --2- 172.21.15.107:0/1839387102 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ffb4c039000 0x7ffb4c03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:46.944 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.082+0000 7ffb637fe700 1 --2- 172.21.15.107:0/1839387102 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ffb4c039000 0x7ffb4c03b4c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ffb58006fd0 tx=0x7ffb58006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:46.944 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.086+0000 7ffb617fa700 1 -- 172.21.15.107:0/1839387102 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ffb54015070 con 0x7ffb64106d70 2024-05-09T08:23:46.944 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.283+0000 7ffb6a69c700 1 -- 172.21.15.107:0/1839387102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7ffb64193030 con 0x7ffb64106d70 2024-05-09T08:23:46.944 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.286+0000 7ffb617fa700 1 -- 172.21.15.107:0/1839387102 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7ffb64193030 con 0x7ffb64106d70 2024-05-09T08:23:46.944 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.288+0000 7ffb6a69c700 1 -- 172.21.15.107:0/1839387102 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ffb4c039000 msgr2=0x7ffb4c03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:46.944 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.288+0000 7ffb6a69c700 1 --2- 172.21.15.107:0/1839387102 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ffb4c039000 0x7ffb4c03b4c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ffb58006fd0 tx=0x7ffb58006e40 comp rx=0 tx=0).stop 2024-05-09T08:23:46.944 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.288+0000 7ffb6a69c700 1 -- 172.21.15.107:0/1839387102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb64106d70 msgr2=0x7ffb64199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:46.944 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.288+0000 7ffb6a69c700 1 --2- 172.21.15.107:0/1839387102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb64106d70 0x7ffb64199ad0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400d650 tx=0x7ffb54003fb0 comp rx=0 tx=0).stop 2024-05-09T08:23:46.944 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.289+0000 7ffb6a69c700 1 -- 172.21.15.107:0/1839387102 shutdown_connections 2024-05-09T08:23:46.944 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.289+0000 7ffb6a69c700 1 --2- 172.21.15.107:0/1839387102 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ffb4c039000 0x7ffb4c03b4c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:46.945 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.289+0000 7ffb6a69c700 1 --2- 172.21.15.107:0/1839387102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb64106d70 0x7ffb64199ad0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:46.945 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.289+0000 7ffb6a69c700 1 -- 172.21.15.107:0/1839387102 >> 172.21.15.107:0/1839387102 conn(0x7ffb64102350 msgr2=0x7ffb641900f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:46.945 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.289+0000 7ffb6a69c700 1 -- 172.21.15.107:0/1839387102 shutdown_connections 2024-05-09T08:23:46.945 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:46.289+0000 7ffb6a69c700 1 -- 172.21.15.107:0/1839387102 wait complete. 2024-05-09T08:23:46.945 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling the dashboard module... 2024-05-09T08:23:47.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:47 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1839387102' entity='client.admin' 2024-05-09T08:23:48.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:48 smithi107 ceph-mon[30122]: from='mgr.14118 172.21.15.107:0/771156336' entity='mgr.x' 2024-05-09T08:23:48.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:48 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/271720083' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-09T08:23:48.686 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:48 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: ignoring --setuser ceph since I am not root 2024-05-09T08:23:48.686 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:48 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: ignoring --setgroup ceph since I am not root 2024-05-09T08:23:48.686 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:48 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:48.602+0000 7f5342004200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:23:49.022 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:48 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:48.796+0000 7f5342004200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:23:49.022 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:48 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:48.899+0000 7f5342004200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:23:49.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.365+0000 7f0a07931700 1 Processor -- start 2024-05-09T08:23:49.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.365+0000 7f0a07931700 1 -- start start 2024-05-09T08:23:49.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.366+0000 7f0a07931700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a00106d80 0x7f0a001071a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:49.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.366+0000 7f0a07931700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0a00107770 con 0x7f0a00106d80 2024-05-09T08:23:49.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.366+0000 7f0a056cd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a00106d80 0x7f0a001071a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:49.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.366+0000 7f0a056cd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a00106d80 0x7f0a001071a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36554/0 (socket says 172.21.15.107:36554) 2024-05-09T08:23:49.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.366+0000 7f0a056cd700 1 -- 172.21.15.107:0/2609958884 learned_addr learned my addr 172.21.15.107:0/2609958884 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:49.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.366+0000 7f0a056cd700 1 -- 172.21.15.107:0/2609958884 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a00107f80 con 0x7f0a00106d80 2024-05-09T08:23:49.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.366+0000 7f0a056cd700 1 --2- 172.21.15.107:0/2609958884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a00106d80 0x7f0a001071a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f09f00098a0 tx=0x7f09f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e08428d88783de server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:49.030 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.366+0000 7f09f7fff700 1 -- 172.21.15.107:0/2609958884 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09f0004160 con 0x7f0a00106d80 2024-05-09T08:23:49.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.366+0000 7f09f7fff700 1 -- 172.21.15.107:0/2609958884 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09f0004300 con 0x7f0a00106d80 2024-05-09T08:23:49.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.367+0000 7f09f7fff700 1 -- 172.21.15.107:0/2609958884 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09f0011a50 con 0x7f0a00106d80 2024-05-09T08:23:49.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.367+0000 7f0a07931700 1 -- 172.21.15.107:0/2609958884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a00106d80 msgr2=0x7f0a001071a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:49.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.367+0000 7f0a07931700 1 --2- 172.21.15.107:0/2609958884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a00106d80 0x7f0a001071a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f09f00098a0 tx=0x7f09f0009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:49.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.367+0000 7f0a07931700 1 -- 172.21.15.107:0/2609958884 shutdown_connections 2024-05-09T08:23:49.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.367+0000 7f0a07931700 1 --2- 172.21.15.107:0/2609958884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a00106d80 0x7f0a001071a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:49.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.367+0000 7f0a07931700 1 -- 172.21.15.107:0/2609958884 >> 172.21.15.107:0/2609958884 conn(0x7f0a00102340 msgr2=0x7f0a00104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:49.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.367+0000 7f0a07931700 1 -- 172.21.15.107:0/2609958884 shutdown_connections 2024-05-09T08:23:49.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.367+0000 7f0a07931700 1 -- 172.21.15.107:0/2609958884 wait complete. 2024-05-09T08:23:49.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.368+0000 7f0a07931700 1 Processor -- start 2024-05-09T08:23:49.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.368+0000 7f0a07931700 1 -- start start 2024-05-09T08:23:49.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.368+0000 7f0a07931700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a00106d80 0x7f0a0019bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:49.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.368+0000 7f0a07931700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0010b330 con 0x7f0a00106d80 2024-05-09T08:23:49.031 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.368+0000 7f0a056cd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a00106d80 0x7f0a0019bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:49.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.368+0000 7f0a056cd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a00106d80 0x7f0a0019bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36560/0 (socket says 172.21.15.107:36560) 2024-05-09T08:23:49.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.368+0000 7f0a056cd700 1 -- 172.21.15.107:0/271720083 learned_addr learned my addr 172.21.15.107:0/271720083 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:49.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.369+0000 7f0a056cd700 1 -- 172.21.15.107:0/271720083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a0019c1d0 con 0x7f0a00106d80 2024-05-09T08:23:49.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.369+0000 7f0a056cd700 1 --2- 172.21.15.107:0/271720083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a00106d80 0x7f0a0019bc90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f09f0006b20 tx=0x7f09f0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:49.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.369+0000 7f09f67fc700 1 -- 172.21.15.107:0/271720083 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09f00039e0 con 0x7f0a00106d80 2024-05-09T08:23:49.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.369+0000 7f09f67fc700 1 -- 172.21.15.107:0/271720083 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09f001a8e0 con 0x7f0a00106d80 2024-05-09T08:23:49.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.369+0000 7f0a07931700 1 -- 172.21.15.107:0/271720083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a0019c460 con 0x7f0a00106d80 2024-05-09T08:23:49.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.369+0000 7f0a07931700 1 -- 172.21.15.107:0/271720083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a0019f150 con 0x7f0a00106d80 2024-05-09T08:23:49.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.369+0000 7f09f67fc700 1 -- 172.21.15.107:0/271720083 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09f00247e0 con 0x7f0a00106d80 2024-05-09T08:23:49.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.370+0000 7f09f67fc700 1 -- 172.21.15.107:0/271720083 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f09f0022020 con 0x7f0a00106d80 2024-05-09T08:23:49.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.370+0000 7f0a07931700 1 -- 172.21.15.107:0/271720083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a00063980 con 0x7f0a00106d80 2024-05-09T08:23:49.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.370+0000 7f09f67fc700 1 --2- 172.21.15.107:0/271720083 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f09ec039050 0x7f09ec03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:49.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.370+0000 7f09f67fc700 1 -- 172.21.15.107:0/271720083 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f09f004d3f0 con 0x7f0a00106d80 2024-05-09T08:23:49.032 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.371+0000 7f0a04ecc700 1 --2- 172.21.15.107:0/271720083 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f09ec039050 0x7f09ec03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:49.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.371+0000 7f0a04ecc700 1 --2- 172.21.15.107:0/271720083 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f09ec039050 0x7f09ec03b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f09fc006fd0 tx=0x7f09fc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:49.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.376+0000 7f09f67fc700 1 -- 172.21.15.107:0/271720083 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f09f0020c10 con 0x7f0a00106d80 2024-05-09T08:23:49.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:47.638+0000 7f0a07931700 1 -- 172.21.15.107:0/271720083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f0a0019f700 con 0x7f0a00106d80 2024-05-09T08:23:49.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:48.415+0000 7f09f67fc700 1 -- 172.21.15.107:0/271720083 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f09f0021d80 con 0x7f0a00106d80 2024-05-09T08:23:49.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:48.417+0000 7f09f67fc700 1 -- 172.21.15.107:0/271720083 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f09f002dc90 con 0x7f0a00106d80 2024-05-09T08:23:49.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:48.418+0000 7f0a07931700 1 -- 172.21.15.107:0/271720083 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f09ec039050 msgr2=0x7f09ec03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:49.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:48.418+0000 7f0a07931700 1 --2- 172.21.15.107:0/271720083 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f09ec039050 0x7f09ec03b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f09fc006fd0 tx=0x7f09fc006e40 comp rx=0 tx=0).stop 2024-05-09T08:23:49.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:48.418+0000 7f0a07931700 1 -- 172.21.15.107:0/271720083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a00106d80 msgr2=0x7f0a0019bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:49.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:48.418+0000 7f0a07931700 1 --2- 172.21.15.107:0/271720083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a00106d80 0x7f0a0019bc90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f09f0006b20 tx=0x7f09f0011ad0 comp rx=0 tx=0).stop 2024-05-09T08:23:49.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:48.418+0000 7f0a07931700 1 -- 172.21.15.107:0/271720083 shutdown_connections 2024-05-09T08:23:49.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:48.418+0000 7f0a07931700 1 --2- 172.21.15.107:0/271720083 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f09ec039050 0x7f09ec03b510 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:49.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:48.418+0000 7f0a07931700 1 --2- 172.21.15.107:0/271720083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a00106d80 0x7f0a0019bc90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:49.033 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:48.418+0000 7f0a07931700 1 -- 172.21.15.107:0/271720083 >> 172.21.15.107:0/271720083 conn(0x7f0a00102340 msgr2=0x7f0a00102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:49.034 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:48.418+0000 7f0a07931700 1 -- 172.21.15.107:0/271720083 shutdown_connections 2024-05-09T08:23:49.034 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:48.418+0000 7f0a07931700 1 -- 172.21.15.107:0/271720083 wait complete. 2024-05-09T08:23:49.280 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:49 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:49.223+0000 7f5342004200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:23:49.743 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:49 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/271720083' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-09T08:23:49.743 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:49 smithi107 ceph-mon[30122]: mgrmap e9: x(active, since 19s) 2024-05-09T08:23:49.743 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:49 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:49.309+0000 7f5342004200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:23:50.428 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:50 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3772228171' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-09T08:23:50.437 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:23:50.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-09T08:23:50.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-09T08:23:50.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-09T08:23:50.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-09T08:23:50.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:23:50.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.536+0000 7f0ef5bbc700 1 Processor -- start 2024-05-09T08:23:50.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.536+0000 7f0ef5bbc700 1 -- start start 2024-05-09T08:23:50.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.536+0000 7f0ef5bbc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef0108920 0x7f0ef010ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:50.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.536+0000 7f0ef5bbc700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef007c6e0 con 0x7f0ef0108920 2024-05-09T08:23:50.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.536+0000 7f0eef7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef0108920 0x7f0ef010ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:50.438 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.536+0000 7f0eef7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef0108920 0x7f0ef010ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36580/0 (socket says 172.21.15.107:36580) 2024-05-09T08:23:50.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.536+0000 7f0eef7fe700 1 -- 172.21.15.107:0/2420460384 learned_addr learned my addr 172.21.15.107:0/2420460384 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:50.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.536+0000 7f0eef7fe700 1 -- 172.21.15.107:0/2420460384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef010b250 con 0x7f0ef0108920 2024-05-09T08:23:50.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.537+0000 7f0eef7fe700 1 --2- 172.21.15.107:0/2420460384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef0108920 0x7f0ef010ad10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0ee00098a0 tx=0x7f0ee0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53698c831f6b0306 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:50.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.537+0000 7f0eee7fc700 1 -- 172.21.15.107:0/2420460384 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ee0004160 con 0x7f0ef0108920 2024-05-09T08:23:50.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.537+0000 7f0eee7fc700 1 -- 172.21.15.107:0/2420460384 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ee0004300 con 0x7f0ef0108920 2024-05-09T08:23:50.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.537+0000 7f0eee7fc700 1 -- 172.21.15.107:0/2420460384 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ee0011a50 con 0x7f0ef0108920 2024-05-09T08:23:50.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.537+0000 7f0ef5bbc700 1 -- 172.21.15.107:0/2420460384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef0108920 msgr2=0x7f0ef010ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:50.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.537+0000 7f0ef5bbc700 1 --2- 172.21.15.107:0/2420460384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef0108920 0x7f0ef010ad10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0ee00098a0 tx=0x7f0ee0009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:50.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.537+0000 7f0ef5bbc700 1 -- 172.21.15.107:0/2420460384 shutdown_connections 2024-05-09T08:23:50.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.537+0000 7f0ef5bbc700 1 --2- 172.21.15.107:0/2420460384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef0108920 0x7f0ef010ad10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:50.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.537+0000 7f0ef5bbc700 1 -- 172.21.15.107:0/2420460384 >> 172.21.15.107:0/2420460384 conn(0x7f0ef0102350 msgr2=0x7f0ef0104770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:50.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.537+0000 7f0ef5bbc700 1 -- 172.21.15.107:0/2420460384 shutdown_connections 2024-05-09T08:23:50.439 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.537+0000 7f0ef5bbc700 1 -- 172.21.15.107:0/2420460384 wait complete. 2024-05-09T08:23:50.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.538+0000 7f0ef5bbc700 1 Processor -- start 2024-05-09T08:23:50.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.538+0000 7f0ef5bbc700 1 -- start start 2024-05-09T08:23:50.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.539+0000 7f0ef5bbc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef0108920 0x7f0ef019bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:50.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.539+0000 7f0ef5bbc700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef010c250 con 0x7f0ef0108920 2024-05-09T08:23:50.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.539+0000 7f0eef7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef0108920 0x7f0ef019bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:50.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.539+0000 7f0eef7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef0108920 0x7f0ef019bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36588/0 (socket says 172.21.15.107:36588) 2024-05-09T08:23:50.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.539+0000 7f0eef7fe700 1 -- 172.21.15.107:0/3772228171 learned_addr learned my addr 172.21.15.107:0/3772228171 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:50.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.539+0000 7f0eef7fe700 1 -- 172.21.15.107:0/3772228171 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef019c170 con 0x7f0ef0108920 2024-05-09T08:23:50.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.539+0000 7f0eef7fe700 1 --2- 172.21.15.107:0/3772228171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef0108920 0x7f0ef019bc30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0006b20 tx=0x7f0ee0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:50.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.539+0000 7f0eecff9700 1 -- 172.21.15.107:0/3772228171 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ee0003b70 con 0x7f0ef0108920 2024-05-09T08:23:50.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.539+0000 7f0eecff9700 1 -- 172.21.15.107:0/3772228171 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ee001a8e0 con 0x7f0ef0108920 2024-05-09T08:23:50.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.539+0000 7f0ef5bbc700 1 -- 172.21.15.107:0/3772228171 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef019c400 con 0x7f0ef0108920 2024-05-09T08:23:50.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.539+0000 7f0eecff9700 1 -- 172.21.15.107:0/3772228171 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ee00249b0 con 0x7f0ef0108920 2024-05-09T08:23:50.440 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.540+0000 7f0ef5bbc700 1 -- 172.21.15.107:0/3772228171 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef019f0f0 con 0x7f0ef0108920 2024-05-09T08:23:50.441 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.540+0000 7f0eecff9700 1 -- 172.21.15.107:0/3772228171 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f0ee0022020 con 0x7f0ef0108920 2024-05-09T08:23:50.441 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.540+0000 7f0ef5bbc700 1 -- 172.21.15.107:0/3772228171 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ef0063980 con 0x7f0ef0108920 2024-05-09T08:23:50.441 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.541+0000 7f0eecff9700 1 --2- 172.21.15.107:0/3772228171 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f0ed80390a0 0x7f0ed803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:50.441 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.541+0000 7f0eecff9700 1 -- 172.21.15.107:0/3772228171 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0ee004d870 con 0x7f0ef0108920 2024-05-09T08:23:50.441 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.541+0000 7f0eeeffd700 1 -- 172.21.15.107:0/3772228171 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f0ed80390a0 msgr2=0x7f0ed803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/3922375261 2024-05-09T08:23:50.441 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.541+0000 7f0eeeffd700 1 --2- 172.21.15.107:0/3772228171 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f0ed80390a0 0x7f0ed803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:23:50.441 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.549+0000 7f0eecff9700 1 -- 172.21.15.107:0/3772228171 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0ee000b3e0 con 0x7f0ef0108920 2024-05-09T08:23:50.441 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.742+0000 7f0eeeffd700 1 -- 172.21.15.107:0/3772228171 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f0ed80390a0 msgr2=0x7f0ed803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/3922375261 2024-05-09T08:23:50.441 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.742+0000 7f0eeeffd700 1 --2- 172.21.15.107:0/3772228171 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f0ed80390a0 0x7f0ed803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:23:50.441 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.844+0000 7f0ef5bbc700 1 -- 172.21.15.107:0/3772228171 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f0ef019f360 con 0x7f0ef0108920 2024-05-09T08:23:50.441 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.845+0000 7f0eecff9700 1 -- 172.21.15.107:0/3772228171 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f0ee002ce20 con 0x7f0ef0108920 2024-05-09T08:23:50.441 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.847+0000 7f0ef5bbc700 1 -- 172.21.15.107:0/3772228171 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f0ed80390a0 msgr2=0x7f0ed803b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:23:50.441 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.847+0000 7f0ef5bbc700 1 --2- 172.21.15.107:0/3772228171 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f0ed80390a0 0x7f0ed803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:50.441 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.847+0000 7f0ef5bbc700 1 -- 172.21.15.107:0/3772228171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef0108920 msgr2=0x7f0ef019bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:50.442 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.847+0000 7f0ef5bbc700 1 --2- 172.21.15.107:0/3772228171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef0108920 0x7f0ef019bc30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0006b20 tx=0x7f0ee0011ad0 comp rx=0 tx=0).stop 2024-05-09T08:23:50.442 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.847+0000 7f0ef5bbc700 1 -- 172.21.15.107:0/3772228171 shutdown_connections 2024-05-09T08:23:50.442 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.847+0000 7f0ef5bbc700 1 --2- 172.21.15.107:0/3772228171 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7f0ed80390a0 0x7f0ed803b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:50.442 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.847+0000 7f0ef5bbc700 1 --2- 172.21.15.107:0/3772228171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef0108920 0x7f0ef019bc30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:50.442 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.847+0000 7f0ef5bbc700 1 -- 172.21.15.107:0/3772228171 >> 172.21.15.107:0/3772228171 conn(0x7f0ef0102350 msgr2=0x7f0ef0102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:50.442 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.847+0000 7f0ef5bbc700 1 -- 172.21.15.107:0/3772228171 shutdown_connections 2024-05-09T08:23:50.442 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:49.847+0000 7f0ef5bbc700 1 -- 172.21.15.107:0/3772228171 wait complete. 2024-05-09T08:23:50.442 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for the mgr to restart... 2024-05-09T08:23:50.442 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr epoch 9... 2024-05-09T08:23:50.732 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:50 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:50.552+0000 7f5342004200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:23:50.732 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:50 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:50.731+0000 7f5342004200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:23:51.030 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:50 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:50.909+0000 7f5342004200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:23:51.487 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:51 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:51.202+0000 7f5342004200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:23:51.488 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:51 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:51.296+0000 7f5342004200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:23:51.780 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:51 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:51.499+0000 7f5342004200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:23:51.780 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:51 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:51.696+0000 7f5342004200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:23:52.723 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:52 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:52.298+0000 7f5342004200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:23:53.022 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:52 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:52.722+0000 7f5342004200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:23:53.023 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:52 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:52.819+0000 7f5342004200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:23:53.023 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:53 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:53.022+0000 7f5342004200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:23:53.887 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:53 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:53.546+0000 7f5342004200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:23:53.887 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:53 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:53.641+0000 7f5342004200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:23:54.236 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:53 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:53.886+0000 7f5342004200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:23:54.236 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:54 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:54.022+0000 7f5342004200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:23:54.236 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:54 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:54.118+0000 7f5342004200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:23:54.236 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:54 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:54.235+0000 7f5342004200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:23:54.893 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:54 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:54.546+0000 7f5342004200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:23:55.424 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:55 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:55.154+0000 7f5342004200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:23:55.424 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:23:55 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:23:55.370+0000 7f5342004200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:23:55.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:55 smithi107 ceph-mon[30122]: Active manager daemon x restarted 2024-05-09T08:23:55.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:55 smithi107 ceph-mon[30122]: Activating manager daemon x 2024-05-09T08:23:55.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:55 smithi107 ceph-mon[30122]: osdmap e3: 0 total, 0 up, 0 in 2024-05-09T08:23:55.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:55 smithi107 ceph-mon[30122]: mgrmap e10: x(active, starting, since 0.00315541s) 2024-05-09T08:23:55.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:23:55.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:23:55.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:23:55.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:23:55.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:23:55.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:55 smithi107 ceph-mon[30122]: Manager daemon x is now available 2024-05-09T08:23:55.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:23:55.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:23:56.727 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:23:56.727 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:56 smithi107 ceph-mon[30122]: mgrmap e11: x(active, since 1.00609s) 2024-05-09T08:23:57.321 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-05-09T08:23:57.321 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-09T08:23:57.321 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-09T08:23:57.321 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-05-09T08:23:57.321 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.880+0000 7ff946a7e700 1 Processor -- start 2024-05-09T08:23:57.321 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.880+0000 7ff946a7e700 1 -- start start 2024-05-09T08:23:57.321 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.880+0000 7ff946a7e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff940106d30 0x7ff940107150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:57.321 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.880+0000 7ff946a7e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff940107720 con 0x7ff940106d30 2024-05-09T08:23:57.321 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.880+0000 7ff94481a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff940106d30 0x7ff940107150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:57.322 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.880+0000 7ff94481a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff940106d30 0x7ff940107150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60626/0 (socket says 172.21.15.107:60626) 2024-05-09T08:23:57.322 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.880+0000 7ff94481a700 1 -- 172.21.15.107:0/792503732 learned_addr learned my addr 172.21.15.107:0/792503732 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:57.322 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.880+0000 7ff94481a700 1 -- 172.21.15.107:0/792503732 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff940107f30 con 0x7ff940106d30 2024-05-09T08:23:57.322 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.881+0000 7ff94481a700 1 --2- 172.21.15.107:0/792503732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff940106d30 0x7ff940107150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff9340098a0 tx=0x7ff934009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29d3aa57675ab1ff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:57.322 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.881+0000 7ff93f7fe700 1 -- 172.21.15.107:0/792503732 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff934004160 con 0x7ff940106d30 2024-05-09T08:23:57.322 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.881+0000 7ff93f7fe700 1 -- 172.21.15.107:0/792503732 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff934004300 con 0x7ff940106d30 2024-05-09T08:23:57.322 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.881+0000 7ff93f7fe700 1 -- 172.21.15.107:0/792503732 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff934011a50 con 0x7ff940106d30 2024-05-09T08:23:57.322 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.881+0000 7ff946a7e700 1 -- 172.21.15.107:0/792503732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff940106d30 msgr2=0x7ff940107150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.322 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.881+0000 7ff946a7e700 1 --2- 172.21.15.107:0/792503732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff940106d30 0x7ff940107150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff9340098a0 tx=0x7ff934009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.322 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.881+0000 7ff946a7e700 1 -- 172.21.15.107:0/792503732 shutdown_connections 2024-05-09T08:23:57.323 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.881+0000 7ff946a7e700 1 --2- 172.21.15.107:0/792503732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff940106d30 0x7ff940107150 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.323 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.881+0000 7ff946a7e700 1 -- 172.21.15.107:0/792503732 >> 172.21.15.107:0/792503732 conn(0x7ff940102310 msgr2=0x7ff940104730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:57.323 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.881+0000 7ff946a7e700 1 -- 172.21.15.107:0/792503732 shutdown_connections 2024-05-09T08:23:57.323 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.882+0000 7ff946a7e700 1 -- 172.21.15.107:0/792503732 wait complete. 2024-05-09T08:23:57.323 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.882+0000 7ff946a7e700 1 Processor -- start 2024-05-09T08:23:57.323 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.882+0000 7ff946a7e700 1 -- start start 2024-05-09T08:23:57.323 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.882+0000 7ff946a7e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff940106d30 0x7ff94019bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:57.323 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.882+0000 7ff946a7e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff94010b2e0 con 0x7ff940106d30 2024-05-09T08:23:57.323 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.882+0000 7ff94481a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff940106d30 0x7ff94019bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:57.323 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.882+0000 7ff94481a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff940106d30 0x7ff94019bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60632/0 (socket says 172.21.15.107:60632) 2024-05-09T08:23:57.323 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.882+0000 7ff94481a700 1 -- 172.21.15.107:0/2534356594 learned_addr learned my addr 172.21.15.107:0/2534356594 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:57.324 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.883+0000 7ff94481a700 1 -- 172.21.15.107:0/2534356594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff94019c1c0 con 0x7ff940106d30 2024-05-09T08:23:57.324 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.883+0000 7ff94481a700 1 --2- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff940106d30 0x7ff94019bc80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff934006b20 tx=0x7ff934011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:57.324 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.883+0000 7ff93dffb700 1 -- 172.21.15.107:0/2534356594 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff9340039e0 con 0x7ff940106d30 2024-05-09T08:23:57.324 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.883+0000 7ff946a7e700 1 -- 172.21.15.107:0/2534356594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff94019c450 con 0x7ff940106d30 2024-05-09T08:23:57.324 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.883+0000 7ff93dffb700 1 -- 172.21.15.107:0/2534356594 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff93401a8e0 con 0x7ff940106d30 2024-05-09T08:23:57.324 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.883+0000 7ff93dffb700 1 -- 172.21.15.107:0/2534356594 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff9340247e0 con 0x7ff940106d30 2024-05-09T08:23:57.324 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.883+0000 7ff946a7e700 1 -- 172.21.15.107:0/2534356594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff94019f140 con 0x7ff940106d30 2024-05-09T08:23:57.324 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.884+0000 7ff93dffb700 1 -- 172.21.15.107:0/2534356594 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7ff934022020 con 0x7ff940106d30 2024-05-09T08:23:57.324 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.885+0000 7ff93dffb700 1 --2- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ff9300390a0 0x7ff93003b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:57.324 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.885+0000 7ff93dffb700 1 -- 172.21.15.107:0/2534356594 --> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff93003bc70 con 0x7ff9300390a0 2024-05-09T08:23:57.324 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.885+0000 7ff93dffb700 1 -- 172.21.15.107:0/2534356594 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff93404d510 con 0x7ff940106d30 2024-05-09T08:23:57.324 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.885+0000 7ff93ffff700 1 -- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ff9300390a0 msgr2=0x7ff93003b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/3922375261 2024-05-09T08:23:57.325 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:50.885+0000 7ff93ffff700 1 --2- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ff9300390a0 0x7ff93003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:23:57.325 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:51.086+0000 7ff93ffff700 1 -- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ff9300390a0 msgr2=0x7ff93003b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/3922375261 2024-05-09T08:23:57.325 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:51.086+0000 7ff93ffff700 1 --2- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ff9300390a0 0x7ff93003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:23:57.325 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:51.487+0000 7ff93ffff700 1 -- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ff9300390a0 msgr2=0x7ff93003b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/3922375261 2024-05-09T08:23:57.325 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:51.487+0000 7ff93ffff700 1 --2- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ff9300390a0 0x7ff93003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T08:23:57.325 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:52.288+0000 7ff93ffff700 1 -- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ff9300390a0 msgr2=0x7ff93003b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/3922375261 2024-05-09T08:23:57.325 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:52.288+0000 7ff93ffff700 1 --2- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ff9300390a0 0x7ff93003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T08:23:57.325 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:53.889+0000 7ff93ffff700 1 -- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ff9300390a0 msgr2=0x7ff93003b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/3922375261 2024-05-09T08:23:57.325 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:53.889+0000 7ff93ffff700 1 --2- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ff9300390a0 0x7ff93003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-09T08:23:57.325 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:55.374+0000 7ff93dffb700 1 -- 172.21.15.107:0/2534356594 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7ff93402d3f0 con 0x7ff940106d30 2024-05-09T08:23:57.325 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:55.374+0000 7ff93dffb700 1 -- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ff9300390a0 msgr2=0x7ff93003b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:23:57.325 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:55.374+0000 7ff93dffb700 1 --2- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ff9300390a0 0x7ff93003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.325 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.376+0000 7ff93dffb700 1 -- 172.21.15.107:0/2534356594 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7ff93404f3f0 con 0x7ff940106d30 2024-05-09T08:23:57.325 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.376+0000 7ff93dffb700 1 --2- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff93003cbb0 0x7ff93003efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:57.326 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.376+0000 7ff93dffb700 1 -- 172.21.15.107:0/2534356594 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff934022e50 con 0x7ff93003cbb0 2024-05-09T08:23:57.326 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.377+0000 7ff93ffff700 1 --2- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff93003cbb0 0x7ff93003efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:57.326 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.378+0000 7ff93ffff700 1 --2- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff93003cbb0 0x7ff93003efa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff92c003a10 tx=0x7ff92c0092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:57.326 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.378+0000 7ff93dffb700 1 -- 172.21.15.107:0/2534356594 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7ff934022e50 con 0x7ff93003cbb0 2024-05-09T08:23:57.326 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.385+0000 7ff946a7e700 1 -- 172.21.15.107:0/2534356594 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7ff940063980 con 0x7ff93003cbb0 2024-05-09T08:23:57.326 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.386+0000 7ff93dffb700 1 -- 172.21.15.107:0/2534356594 <== mgr.14150 v2:172.21.15.107:6800/846948416 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7ff940063980 con 0x7ff93003cbb0 2024-05-09T08:23:57.326 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.386+0000 7ff946a7e700 1 -- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff93003cbb0 msgr2=0x7ff93003efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.326 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.386+0000 7ff946a7e700 1 --2- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff93003cbb0 0x7ff93003efa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff92c003a10 tx=0x7ff92c0092b0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.326 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.386+0000 7ff946a7e700 1 -- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff940106d30 msgr2=0x7ff94019bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:57.326 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.386+0000 7ff946a7e700 1 --2- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff940106d30 0x7ff94019bc80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff934006b20 tx=0x7ff934011ad0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.326 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.386+0000 7ff946a7e700 1 -- 172.21.15.107:0/2534356594 shutdown_connections 2024-05-09T08:23:57.326 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.386+0000 7ff946a7e700 1 --2- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff93003cbb0 0x7ff93003efa0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.326 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.386+0000 7ff946a7e700 1 --2- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff940106d30 0x7ff94019bc80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.386+0000 7ff946a7e700 1 --2- 172.21.15.107:0/2534356594 >> [v2:172.21.15.107:6800/3922375261,v1:172.21.15.107:6801/3922375261] conn(0x7ff9300390a0 0x7ff93003b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:57.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.386+0000 7ff946a7e700 1 -- 172.21.15.107:0/2534356594 >> 172.21.15.107:0/2534356594 conn(0x7ff940102310 msgr2=0x7ff940103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:57.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.386+0000 7ff946a7e700 1 -- 172.21.15.107:0/2534356594 shutdown_connections 2024-05-09T08:23:57.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:56.386+0000 7ff946a7e700 1 -- 172.21.15.107:0/2534356594 wait complete. 2024-05-09T08:23:57.327 INFO:teuthology.orchestra.run.smithi107.stdout:mgr epoch 9 is available 2024-05-09T08:23:57.327 INFO:teuthology.orchestra.run.smithi107.stdout:Generating a dashboard self-signed certificate... 2024-05-09T08:23:57.584 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:57 smithi107 ceph-mon[30122]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-09T08:23:57.585 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:57 smithi107 ceph-mon[30122]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-09T08:23:57.585 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:57 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:23:58.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-09T08:23:58.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:58 smithi107 ceph-mon[30122]: [09/May/2024:08:23:56] ENGINE Bus STARTING 2024-05-09T08:23:58.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:58 smithi107 ceph-mon[30122]: [09/May/2024:08:23:56] ENGINE Serving on https://172.21.15.107:7150 2024-05-09T08:23:58.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:58 smithi107 ceph-mon[30122]: [09/May/2024:08:23:56] ENGINE Serving on http://172.21.15.107:8765 2024-05-09T08:23:58.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:58 smithi107 ceph-mon[30122]: [09/May/2024:08:23:56] ENGINE Bus STARTED 2024-05-09T08:23:58.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:58 smithi107 ceph-mon[30122]: mgrmap e12: x(active, since 2s) 2024-05-09T08:23:58.679 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:23:58.679 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:23:58.679 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:23:58.684 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.801+0000 7fa017fb2700 1 Processor -- start 2024-05-09T08:23:58.684 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.802+0000 7fa017fb2700 1 -- start start 2024-05-09T08:23:58.684 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.802+0000 7fa017fb2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa01007e730 0x7fa01007cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.684 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.802+0000 7fa017fb2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa01007d160 con 0x7fa01007e730 2024-05-09T08:23:58.684 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.802+0000 7fa015d4e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa01007e730 0x7fa01007cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.684 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.802+0000 7fa015d4e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa01007e730 0x7fa01007cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60692/0 (socket says 172.21.15.107:60692) 2024-05-09T08:23:58.685 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.802+0000 7fa015d4e700 1 -- 172.21.15.107:0/1590894025 learned_addr learned my addr 172.21.15.107:0/1590894025 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:58.685 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.802+0000 7fa015d4e700 1 -- 172.21.15.107:0/1590894025 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa01007d2e0 con 0x7fa01007e730 2024-05-09T08:23:58.685 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.802+0000 7fa015d4e700 1 --2- 172.21.15.107:0/1590894025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa01007e730 0x7fa01007cb90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0098a0 tx=0x7fa00c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e6148109000ef53 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:58.685 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.803+0000 7fa014d4c700 1 -- 172.21.15.107:0/1590894025 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa00c004160 con 0x7fa01007e730 2024-05-09T08:23:58.685 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.803+0000 7fa014d4c700 1 -- 172.21.15.107:0/1590894025 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa00c004300 con 0x7fa01007e730 2024-05-09T08:23:58.685 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.803+0000 7fa014d4c700 1 -- 172.21.15.107:0/1590894025 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa00c011a50 con 0x7fa01007e730 2024-05-09T08:23:58.685 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.803+0000 7fa017fb2700 1 -- 172.21.15.107:0/1590894025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa01007e730 msgr2=0x7fa01007cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.685 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.803+0000 7fa017fb2700 1 --2- 172.21.15.107:0/1590894025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa01007e730 0x7fa01007cb90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0098a0 tx=0x7fa00c009bb0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.686 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.804+0000 7fa017fb2700 1 -- 172.21.15.107:0/1590894025 shutdown_connections 2024-05-09T08:23:58.686 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.804+0000 7fa017fb2700 1 --2- 172.21.15.107:0/1590894025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa01007e730 0x7fa01007cb90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.686 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.804+0000 7fa017fb2700 1 -- 172.21.15.107:0/1590894025 >> 172.21.15.107:0/1590894025 conn(0x7fa010103440 msgr2=0x7fa010105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:58.686 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.804+0000 7fa017fb2700 1 -- 172.21.15.107:0/1590894025 shutdown_connections 2024-05-09T08:23:58.686 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.804+0000 7fa017fb2700 1 -- 172.21.15.107:0/1590894025 wait complete. 2024-05-09T08:23:58.686 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.804+0000 7fa017fb2700 1 Processor -- start 2024-05-09T08:23:58.686 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.805+0000 7fa017fb2700 1 -- start start 2024-05-09T08:23:58.686 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.805+0000 7fa017fb2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa01007e730 0x7fa01019de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.686 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.805+0000 7fa017fb2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa01010e440 con 0x7fa01007e730 2024-05-09T08:23:58.687 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.805+0000 7fa015d4e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa01007e730 0x7fa01019de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.687 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.805+0000 7fa015d4e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa01007e730 0x7fa01019de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60704/0 (socket says 172.21.15.107:60704) 2024-05-09T08:23:58.687 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.805+0000 7fa015d4e700 1 -- 172.21.15.107:0/1679815461 learned_addr learned my addr 172.21.15.107:0/1679815461 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:23:58.687 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.805+0000 7fa015d4e700 1 -- 172.21.15.107:0/1679815461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa01019e370 con 0x7fa01007e730 2024-05-09T08:23:58.687 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.805+0000 7fa015d4e700 1 --2- 172.21.15.107:0/1679815461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa01007e730 0x7fa01019de30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa00c006ba0 tx=0x7fa00c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:58.687 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.805+0000 7fa006ffd700 1 -- 172.21.15.107:0/1679815461 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa00c003af0 con 0x7fa01007e730 2024-05-09T08:23:58.687 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.805+0000 7fa017fb2700 1 -- 172.21.15.107:0/1679815461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa01019e600 con 0x7fa01007e730 2024-05-09T08:23:58.687 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.805+0000 7fa006ffd700 1 -- 172.21.15.107:0/1679815461 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa00c01a8e0 con 0x7fa01007e730 2024-05-09T08:23:58.687 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.805+0000 7fa006ffd700 1 -- 172.21.15.107:0/1679815461 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa00c024900 con 0x7fa01007e730 2024-05-09T08:23:58.687 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.805+0000 7fa017fb2700 1 -- 172.21.15.107:0/1679815461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0101a12f0 con 0x7fa01007e730 2024-05-09T08:23:58.687 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.806+0000 7fa006ffd700 1 -- 172.21.15.107:0/1679815461 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7fa00c022020 con 0x7fa01007e730 2024-05-09T08:23:58.687 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.807+0000 7fa006ffd700 1 --2- 172.21.15.107:0/1679815461 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9ffc039050 0x7f9ffc03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:23:58.687 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.807+0000 7fa006ffd700 1 -- 172.21.15.107:0/1679815461 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa00c04d590 con 0x7fa01007e730 2024-05-09T08:23:58.688 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.807+0000 7fa017fb2700 1 -- 172.21.15.107:0/1679815461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa010063980 con 0x7fa01007e730 2024-05-09T08:23:58.688 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.807+0000 7fa01554d700 1 --2- 172.21.15.107:0/1679815461 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9ffc039050 0x7f9ffc03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:23:58.688 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.807+0000 7fa01554d700 1 --2- 172.21.15.107:0/1679815461 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9ffc039050 0x7f9ffc03b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa000006fd0 tx=0x7fa000006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:23:58.688 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:57.812+0000 7fa006ffd700 1 -- 172.21.15.107:0/1679815461 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa00c00b3e0 con 0x7fa01007e730 2024-05-09T08:23:58.688 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:58.017+0000 7fa017fb2700 1 -- 172.21.15.107:0/1679815461 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fa010104570 con 0x7f9ffc039050 2024-05-09T08:23:58.688 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:58.096+0000 7fa006ffd700 1 -- 172.21.15.107:0/1679815461 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fa010104570 con 0x7f9ffc039050 2024-05-09T08:23:58.688 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:58.098+0000 7fa017fb2700 1 -- 172.21.15.107:0/1679815461 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9ffc039050 msgr2=0x7f9ffc03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.688 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:58.098+0000 7fa017fb2700 1 --2- 172.21.15.107:0/1679815461 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9ffc039050 0x7f9ffc03b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa000006fd0 tx=0x7fa000006e40 comp rx=0 tx=0).stop 2024-05-09T08:23:58.688 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:58.098+0000 7fa017fb2700 1 -- 172.21.15.107:0/1679815461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa01007e730 msgr2=0x7fa01019de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:23:58.688 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:58.098+0000 7fa017fb2700 1 --2- 172.21.15.107:0/1679815461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa01007e730 0x7fa01019de30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa00c006ba0 tx=0x7fa00c011ad0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.688 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:58.098+0000 7fa017fb2700 1 -- 172.21.15.107:0/1679815461 shutdown_connections 2024-05-09T08:23:58.688 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:58.098+0000 7fa017fb2700 1 --2- 172.21.15.107:0/1679815461 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9ffc039050 0x7f9ffc03b510 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.688 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:58.098+0000 7fa017fb2700 1 --2- 172.21.15.107:0/1679815461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa01007e730 0x7fa01019de30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:23:58.689 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:58.098+0000 7fa017fb2700 1 -- 172.21.15.107:0/1679815461 >> 172.21.15.107:0/1679815461 conn(0x7fa010103440 msgr2=0x7fa010103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:23:58.689 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:58.098+0000 7fa017fb2700 1 -- 172.21.15.107:0/1679815461 shutdown_connections 2024-05-09T08:23:58.689 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:58.098+0000 7fa017fb2700 1 -- 172.21.15.107:0/1679815461 wait complete. 2024-05-09T08:23:58.689 INFO:teuthology.orchestra.run.smithi107.stdout:Creating initial admin user... 2024-05-09T08:23:59.606 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:23:59 smithi107 ceph-mon[30122]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:00.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:00 smithi107 ceph-mon[30122]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:00.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:00 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:00.904 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$jXRRFwQakMfE94MSY4y19OOWngve56oJ0.dlxTxGEMeTqL7GSD1Da", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1715243039, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-09T08:24:00.904 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.066+0000 7fb23e10a700 1 Processor -- start 2024-05-09T08:24:00.904 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.066+0000 7fb23e10a700 1 -- start start 2024-05-09T08:24:00.904 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.066+0000 7fb23e10a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb238107e60 0x7fb238108280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:00.904 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.066+0000 7fb23e10a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb2381087c0 con 0x7fb238107e60 2024-05-09T08:24:00.904 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.066+0000 7fb2377fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb238107e60 0x7fb238108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:00.904 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.067+0000 7fb2377fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb238107e60 0x7fb238108280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60720/0 (socket says 172.21.15.107:60720) 2024-05-09T08:24:00.904 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.067+0000 7fb2377fe700 1 -- 172.21.15.107:0/3120610170 learned_addr learned my addr 172.21.15.107:0/3120610170 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:24:00.905 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.067+0000 7fb2377fe700 1 -- 172.21.15.107:0/3120610170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb238108fc0 con 0x7fb238107e60 2024-05-09T08:24:00.905 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.067+0000 7fb2377fe700 1 --2- 172.21.15.107:0/3120610170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb238107e60 0x7fb238108280 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb2280098a0 tx=0x7fb228009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb054adf053992 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:00.905 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.067+0000 7fb236ffd700 1 -- 172.21.15.107:0/3120610170 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb228004160 con 0x7fb238107e60 2024-05-09T08:24:00.905 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.067+0000 7fb236ffd700 1 -- 172.21.15.107:0/3120610170 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb228004300 con 0x7fb238107e60 2024-05-09T08:24:00.905 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.067+0000 7fb236ffd700 1 -- 172.21.15.107:0/3120610170 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb228011a50 con 0x7fb238107e60 2024-05-09T08:24:00.905 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.068+0000 7fb23e10a700 1 -- 172.21.15.107:0/3120610170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb238107e60 msgr2=0x7fb238108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:00.905 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.068+0000 7fb23e10a700 1 --2- 172.21.15.107:0/3120610170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb238107e60 0x7fb238108280 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb2280098a0 tx=0x7fb228009bb0 comp rx=0 tx=0).stop 2024-05-09T08:24:00.905 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.068+0000 7fb23e10a700 1 -- 172.21.15.107:0/3120610170 shutdown_connections 2024-05-09T08:24:00.905 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.068+0000 7fb23e10a700 1 --2- 172.21.15.107:0/3120610170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb238107e60 0x7fb238108280 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:00.905 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.068+0000 7fb23e10a700 1 -- 172.21.15.107:0/3120610170 >> 172.21.15.107:0/3120610170 conn(0x7fb238103440 msgr2=0x7fb238105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:00.905 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.068+0000 7fb23e10a700 1 -- 172.21.15.107:0/3120610170 shutdown_connections 2024-05-09T08:24:00.905 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.068+0000 7fb23e10a700 1 -- 172.21.15.107:0/3120610170 wait complete. 2024-05-09T08:24:00.905 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.068+0000 7fb23e10a700 1 Processor -- start 2024-05-09T08:24:00.905 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.068+0000 7fb23e10a700 1 -- start start 2024-05-09T08:24:00.906 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.069+0000 7fb23e10a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb238107e60 0x7fb23819bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:00.906 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.069+0000 7fb23e10a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb23810c400 con 0x7fb238107e60 2024-05-09T08:24:00.906 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.069+0000 7fb2377fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb238107e60 0x7fb23819bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:00.906 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.069+0000 7fb2377fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb238107e60 0x7fb23819bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60724/0 (socket says 172.21.15.107:60724) 2024-05-09T08:24:00.906 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.069+0000 7fb2377fe700 1 -- 172.21.15.107:0/1366419940 learned_addr learned my addr 172.21.15.107:0/1366419940 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:24:00.906 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.069+0000 7fb2377fe700 1 -- 172.21.15.107:0/1366419940 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb23819c160 con 0x7fb238107e60 2024-05-09T08:24:00.906 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.069+0000 7fb2377fe700 1 --2- 172.21.15.107:0/1366419940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb238107e60 0x7fb23819bc20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb228011a20 tx=0x7fb228011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:00.906 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.069+0000 7fb2357fa700 1 -- 172.21.15.107:0/1366419940 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb228003b70 con 0x7fb238107e60 2024-05-09T08:24:00.906 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.069+0000 7fb2357fa700 1 -- 172.21.15.107:0/1366419940 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb22801a8e0 con 0x7fb238107e60 2024-05-09T08:24:00.906 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.069+0000 7fb23e10a700 1 -- 172.21.15.107:0/1366419940 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb23819c3f0 con 0x7fb238107e60 2024-05-09T08:24:00.906 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.069+0000 7fb2357fa700 1 -- 172.21.15.107:0/1366419940 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb228024990 con 0x7fb238107e60 2024-05-09T08:24:00.906 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.069+0000 7fb23e10a700 1 -- 172.21.15.107:0/1366419940 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb23819f0e0 con 0x7fb238107e60 2024-05-09T08:24:00.906 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.070+0000 7fb2357fa700 1 -- 172.21.15.107:0/1366419940 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7fb228022020 con 0x7fb238107e60 2024-05-09T08:24:00.906 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.070+0000 7fb23e10a700 1 -- 172.21.15.107:0/1366419940 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb238063980 con 0x7fb238107e60 2024-05-09T08:24:00.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.070+0000 7fb2357fa700 1 --2- 172.21.15.107:0/1366419940 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb2240390b0 0x7fb22403b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:00.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.070+0000 7fb2357fa700 1 -- 172.21.15.107:0/1366419940 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb22804d5f0 con 0x7fb238107e60 2024-05-09T08:24:00.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.070+0000 7fb22ffff700 1 --2- 172.21.15.107:0/1366419940 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb2240390b0 0x7fb22403b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:00.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.071+0000 7fb22ffff700 1 --2- 172.21.15.107:0/1366419940 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb2240390b0 0x7fb22403b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb220006fd0 tx=0x7fb220006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:00.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.075+0000 7fb2357fa700 1 -- 172.21.15.107:0/1366419940 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb228024cf0 con 0x7fb238107e60 2024-05-09T08:24:00.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.296+0000 7fb23e10a700 1 -- 172.21.15.107:0/1366419940 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fb23819c8d0 con 0x7fb2240390b0 2024-05-09T08:24:00.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.521+0000 7fb2357fa700 1 -- 172.21.15.107:0/1366419940 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fb23819c8d0 con 0x7fb2240390b0 2024-05-09T08:24:00.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.524+0000 7fb23e10a700 1 -- 172.21.15.107:0/1366419940 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb2240390b0 msgr2=0x7fb22403b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:00.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.524+0000 7fb23e10a700 1 --2- 172.21.15.107:0/1366419940 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb2240390b0 0x7fb22403b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb220006fd0 tx=0x7fb220006e40 comp rx=0 tx=0).stop 2024-05-09T08:24:00.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.524+0000 7fb23e10a700 1 -- 172.21.15.107:0/1366419940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb238107e60 msgr2=0x7fb23819bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:00.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.524+0000 7fb23e10a700 1 --2- 172.21.15.107:0/1366419940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb238107e60 0x7fb23819bc20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb228011a20 tx=0x7fb228011b00 comp rx=0 tx=0).stop 2024-05-09T08:24:00.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.524+0000 7fb23e10a700 1 -- 172.21.15.107:0/1366419940 shutdown_connections 2024-05-09T08:24:00.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.524+0000 7fb23e10a700 1 --2- 172.21.15.107:0/1366419940 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb2240390b0 0x7fb22403b570 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:00.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.524+0000 7fb23e10a700 1 --2- 172.21.15.107:0/1366419940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb238107e60 0x7fb23819bc20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:00.908 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.524+0000 7fb23e10a700 1 -- 172.21.15.107:0/1366419940 >> 172.21.15.107:0/1366419940 conn(0x7fb238103440 msgr2=0x7fb238103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:00.908 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.524+0000 7fb23e10a700 1 -- 172.21.15.107:0/1366419940 shutdown_connections 2024-05-09T08:24:00.908 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:23:59.524+0000 7fb23e10a700 1 -- 172.21.15.107:0/1366419940 wait complete. 2024-05-09T08:24:00.908 INFO:teuthology.orchestra.run.smithi107.stdout:Fetching dashboard port number... 2024-05-09T08:24:02.118 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 8443 2024-05-09T08:24:02.118 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.323+0000 7f23953a4700 1 Processor -- start 2024-05-09T08:24:02.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.323+0000 7f23953a4700 1 -- start start 2024-05-09T08:24:02.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.323+0000 7f23953a4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2390108970 0x7f239010ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:02.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.323+0000 7f23953a4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f239007be10 con 0x7f2390108970 2024-05-09T08:24:02.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.323+0000 7f238effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2390108970 0x7f239010ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:02.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.323+0000 7f238effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2390108970 0x7f239010ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43082/0 (socket says 172.21.15.107:43082) 2024-05-09T08:24:02.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.323+0000 7f238effd700 1 -- 172.21.15.107:0/4094745678 learned_addr learned my addr 172.21.15.107:0/4094745678 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:24:02.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.324+0000 7f238effd700 1 -- 172.21.15.107:0/4094745678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f239010b2a0 con 0x7f2390108970 2024-05-09T08:24:02.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.324+0000 7f238effd700 1 --2- 172.21.15.107:0/4094745678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2390108970 0x7f239010ad60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f23800098a0 tx=0x7f2380009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f736c5aa7d04538 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:02.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.324+0000 7f238dffb700 1 -- 172.21.15.107:0/4094745678 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2380004160 con 0x7f2390108970 2024-05-09T08:24:02.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.324+0000 7f238dffb700 1 -- 172.21.15.107:0/4094745678 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2380004300 con 0x7f2390108970 2024-05-09T08:24:02.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.324+0000 7f238dffb700 1 -- 172.21.15.107:0/4094745678 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2380011a50 con 0x7f2390108970 2024-05-09T08:24:02.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.324+0000 7f23953a4700 1 -- 172.21.15.107:0/4094745678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2390108970 msgr2=0x7f239010ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:02.119 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.324+0000 7f23953a4700 1 --2- 172.21.15.107:0/4094745678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2390108970 0x7f239010ad60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f23800098a0 tx=0x7f2380009bb0 comp rx=0 tx=0).stop 2024-05-09T08:24:02.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.325+0000 7f23953a4700 1 -- 172.21.15.107:0/4094745678 shutdown_connections 2024-05-09T08:24:02.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.325+0000 7f23953a4700 1 --2- 172.21.15.107:0/4094745678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2390108970 0x7f239010ad60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:02.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.325+0000 7f23953a4700 1 -- 172.21.15.107:0/4094745678 >> 172.21.15.107:0/4094745678 conn(0x7f2390102340 msgr2=0x7f2390104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:02.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.325+0000 7f23953a4700 1 -- 172.21.15.107:0/4094745678 shutdown_connections 2024-05-09T08:24:02.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.325+0000 7f23953a4700 1 -- 172.21.15.107:0/4094745678 wait complete. 2024-05-09T08:24:02.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.325+0000 7f23953a4700 1 Processor -- start 2024-05-09T08:24:02.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.325+0000 7f23953a4700 1 -- start start 2024-05-09T08:24:02.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.326+0000 7f23953a4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2390108970 0x7f239019bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:02.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.326+0000 7f23953a4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f239010c240 con 0x7f2390108970 2024-05-09T08:24:02.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.326+0000 7f238effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2390108970 0x7f239019bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:02.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.326+0000 7f238effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2390108970 0x7f239019bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43090/0 (socket says 172.21.15.107:43090) 2024-05-09T08:24:02.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.326+0000 7f238effd700 1 -- 172.21.15.107:0/568322782 learned_addr learned my addr 172.21.15.107:0/568322782 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:24:02.120 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.326+0000 7f238effd700 1 -- 172.21.15.107:0/568322782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f239019c1c0 con 0x7f2390108970 2024-05-09T08:24:02.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.326+0000 7f238effd700 1 --2- 172.21.15.107:0/568322782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2390108970 0x7f239019bc80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2380006b20 tx=0x7f2380011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:02.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.326+0000 7f2377fff700 1 -- 172.21.15.107:0/568322782 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2380003b70 con 0x7f2390108970 2024-05-09T08:24:02.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.326+0000 7f2377fff700 1 -- 172.21.15.107:0/568322782 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f238001a8e0 con 0x7f2390108970 2024-05-09T08:24:02.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.326+0000 7f23953a4700 1 -- 172.21.15.107:0/568322782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f239019c450 con 0x7f2390108970 2024-05-09T08:24:02.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.326+0000 7f2377fff700 1 -- 172.21.15.107:0/568322782 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f23800249b0 con 0x7f2390108970 2024-05-09T08:24:02.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.326+0000 7f23953a4700 1 -- 172.21.15.107:0/568322782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f239019f140 con 0x7f2390108970 2024-05-09T08:24:02.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.327+0000 7f2377fff700 1 -- 172.21.15.107:0/568322782 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f2380022020 con 0x7f2390108970 2024-05-09T08:24:02.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.327+0000 7f23953a4700 1 -- 172.21.15.107:0/568322782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2390063980 con 0x7f2390108970 2024-05-09T08:24:02.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.327+0000 7f2377fff700 1 --2- 172.21.15.107:0/568322782 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f23780390a0 0x7f237803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:02.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.327+0000 7f2377fff700 1 -- 172.21.15.107:0/568322782 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f238004daa0 con 0x7f2390108970 2024-05-09T08:24:02.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.327+0000 7f238e7fc700 1 --2- 172.21.15.107:0/568322782 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f23780390a0 0x7f237803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:02.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.328+0000 7f238e7fc700 1 --2- 172.21.15.107:0/568322782 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f23780390a0 0x7f237803b560 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2384006fd0 tx=0x7f2384006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:02.121 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.332+0000 7f2377fff700 1 -- 172.21.15.107:0/568322782 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2380019e10 con 0x7f2390108970 2024-05-09T08:24:02.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.429+0000 7f2377fff700 1 -- 172.21.15.107:0/568322782 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f238002da00 con 0x7f2390108970 2024-05-09T08:24:02.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.530+0000 7f23953a4700 1 -- 172.21.15.107:0/568322782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f239019f6f0 con 0x7f2390108970 2024-05-09T08:24:02.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.530+0000 7f2377fff700 1 -- 172.21.15.107:0/568322782 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f238002cdc0 con 0x7f2390108970 2024-05-09T08:24:02.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.533+0000 7f23953a4700 1 -- 172.21.15.107:0/568322782 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f23780390a0 msgr2=0x7f237803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:02.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.533+0000 7f23953a4700 1 --2- 172.21.15.107:0/568322782 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f23780390a0 0x7f237803b560 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2384006fd0 tx=0x7f2384006e40 comp rx=0 tx=0).stop 2024-05-09T08:24:02.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.533+0000 7f23953a4700 1 -- 172.21.15.107:0/568322782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2390108970 msgr2=0x7f239019bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:02.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.533+0000 7f23953a4700 1 --2- 172.21.15.107:0/568322782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2390108970 0x7f239019bc80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2380006b20 tx=0x7f2380011ad0 comp rx=0 tx=0).stop 2024-05-09T08:24:02.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.533+0000 7f23953a4700 1 -- 172.21.15.107:0/568322782 shutdown_connections 2024-05-09T08:24:02.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.533+0000 7f23953a4700 1 --2- 172.21.15.107:0/568322782 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f23780390a0 0x7f237803b560 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:02.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.533+0000 7f23953a4700 1 --2- 172.21.15.107:0/568322782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2390108970 0x7f239019bc80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:02.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.533+0000 7f23953a4700 1 -- 172.21.15.107:0/568322782 >> 172.21.15.107:0/568322782 conn(0x7f2390102340 msgr2=0x7f2390102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:02.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.533+0000 7f23953a4700 1 -- 172.21.15.107:0/568322782 shutdown_connections 2024-05-09T08:24:02.122 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:01.533+0000 7f23953a4700 1 -- 172.21.15.107:0/568322782 wait complete. 2024-05-09T08:24:02.124 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-09T08:24:02.124 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-05-09T08:24:02.131 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-09T08:24:02.131 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-05-09T08:24:02.131 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-05-09T08:24:02.131 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-09T08:24:02.133 INFO:teuthology.orchestra.run.smithi107.stdout:Ceph Dashboard is now available at: 2024-05-09T08:24:02.133 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:24:02.133 INFO:teuthology.orchestra.run.smithi107.stdout: URL: https://smithi107.front.sepia.ceph.com:8443/ 2024-05-09T08:24:02.133 INFO:teuthology.orchestra.run.smithi107.stdout: User: admin 2024-05-09T08:24:02.133 INFO:teuthology.orchestra.run.smithi107.stdout: Password: xjgl1oze22 2024-05-09T08:24:02.133 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:24:02.133 INFO:teuthology.orchestra.run.smithi107.stdout:Saving cluster configuration to /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config directory 2024-05-09T08:24:02.134 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling autotune for osd_memory_target 2024-05-09T08:24:02.754 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:02 smithi107 ceph-mon[30122]: mgrmap e13: x(active, since 6s) 2024-05-09T08:24:02.754 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:02 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/568322782' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-09T08:24:03.313 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.550+0000 7f4fac239700 1 Processor -- start 2024-05-09T08:24:03.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.550+0000 7f4fac239700 1 -- start start 2024-05-09T08:24:03.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.551+0000 7f4fac239700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa4108920 0x7f4fa410ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:03.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.551+0000 7f4fac239700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa407c6c0 con 0x7f4fa4108920 2024-05-09T08:24:03.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.551+0000 7f4fa9fd5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa4108920 0x7f4fa410ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:03.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.551+0000 7f4fa9fd5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa4108920 0x7f4fa410ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43100/0 (socket says 172.21.15.107:43100) 2024-05-09T08:24:03.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.551+0000 7f4fa9fd5700 1 -- 172.21.15.107:0/958993343 learned_addr learned my addr 172.21.15.107:0/958993343 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:24:03.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.551+0000 7f4fa9fd5700 1 -- 172.21.15.107:0/958993343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa410b250 con 0x7f4fa4108920 2024-05-09T08:24:03.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.551+0000 7f4fa9fd5700 1 --2- 172.21.15.107:0/958993343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa4108920 0x7f4fa410ad10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0009e80 tx=0x7f4fa000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02aaaede1d046c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:03.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.551+0000 7f4fa8fd3700 1 -- 172.21.15.107:0/958993343 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4fa0017070 con 0x7f4fa4108920 2024-05-09T08:24:03.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.551+0000 7f4fa8fd3700 1 -- 172.21.15.107:0/958993343 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4fa000b8c0 con 0x7f4fa4108920 2024-05-09T08:24:03.315 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.551+0000 7f4fa8fd3700 1 -- 172.21.15.107:0/958993343 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4fa0004490 con 0x7f4fa4108920 2024-05-09T08:24:03.315 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.552+0000 7f4fac239700 1 -- 172.21.15.107:0/958993343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa4108920 msgr2=0x7f4fa410ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:03.315 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.552+0000 7f4fac239700 1 --2- 172.21.15.107:0/958993343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa4108920 0x7f4fa410ad10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0009e80 tx=0x7f4fa000b330 comp rx=0 tx=0).stop 2024-05-09T08:24:03.315 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.552+0000 7f4fac239700 1 -- 172.21.15.107:0/958993343 shutdown_connections 2024-05-09T08:24:03.315 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.552+0000 7f4fac239700 1 --2- 172.21.15.107:0/958993343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa4108920 0x7f4fa410ad10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.315 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.552+0000 7f4fac239700 1 -- 172.21.15.107:0/958993343 >> 172.21.15.107:0/958993343 conn(0x7f4fa41022d0 msgr2=0x7f4fa4104730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:03.315 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.552+0000 7f4fac239700 1 -- 172.21.15.107:0/958993343 shutdown_connections 2024-05-09T08:24:03.315 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.552+0000 7f4fac239700 1 -- 172.21.15.107:0/958993343 wait complete. 2024-05-09T08:24:03.315 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.553+0000 7f4fac239700 1 Processor -- start 2024-05-09T08:24:03.315 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.553+0000 7f4fac239700 1 -- start start 2024-05-09T08:24:03.315 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.553+0000 7f4fac239700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa4108920 0x7f4fa419bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:03.315 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.553+0000 7f4fac239700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa410c200 con 0x7f4fa4108920 2024-05-09T08:24:03.315 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.553+0000 7f4fa9fd5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa4108920 0x7f4fa419bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:03.316 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.553+0000 7f4fa9fd5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa4108920 0x7f4fa419bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43116/0 (socket says 172.21.15.107:43116) 2024-05-09T08:24:03.316 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.553+0000 7f4fa9fd5700 1 -- 172.21.15.107:0/209460889 learned_addr learned my addr 172.21.15.107:0/209460889 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:24:03.316 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.553+0000 7f4fa9fd5700 1 -- 172.21.15.107:0/209460889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa419c1c0 con 0x7f4fa4108920 2024-05-09T08:24:03.316 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.553+0000 7f4fa9fd5700 1 --2- 172.21.15.107:0/209460889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa4108920 0x7f4fa419bc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0009e50 tx=0x7f4fa0003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:03.316 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.553+0000 7f4f9affd700 1 -- 172.21.15.107:0/209460889 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4fa0020020 con 0x7f4fa4108920 2024-05-09T08:24:03.316 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.554+0000 7f4f9affd700 1 -- 172.21.15.107:0/209460889 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4fa000bba0 con 0x7f4fa4108920 2024-05-09T08:24:03.316 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.554+0000 7f4fac239700 1 -- 172.21.15.107:0/209460889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fa419c450 con 0x7f4fa4108920 2024-05-09T08:24:03.316 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.554+0000 7f4f9affd700 1 -- 172.21.15.107:0/209460889 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4fa0017040 con 0x7f4fa4108920 2024-05-09T08:24:03.316 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.554+0000 7f4fac239700 1 -- 172.21.15.107:0/209460889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fa419f140 con 0x7f4fa4108920 2024-05-09T08:24:03.316 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.554+0000 7f4f9affd700 1 -- 172.21.15.107:0/209460889 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f4fa0023020 con 0x7f4fa4108920 2024-05-09T08:24:03.316 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.555+0000 7f4fac239700 1 -- 172.21.15.107:0/209460889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fa4063980 con 0x7f4fa4108920 2024-05-09T08:24:03.316 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.555+0000 7f4f9affd700 1 --2- 172.21.15.107:0/209460889 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4f90039150 0x7f4f9003b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:03.316 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.555+0000 7f4f9affd700 1 -- 172.21.15.107:0/209460889 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4fa000f050 con 0x7f4fa4108920 2024-05-09T08:24:03.316 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.555+0000 7f4fa97d4700 1 --2- 172.21.15.107:0/209460889 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4f90039150 0x7f4f9003b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.555+0000 7f4fa97d4700 1 --2- 172.21.15.107:0/209460889 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4f90039150 0x7f4f9003b610 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4f94006fd0 tx=0x7f4f94006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.560+0000 7f4f9affd700 1 -- 172.21.15.107:0/209460889 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4fa002a020 con 0x7f4fa4108920 2024-05-09T08:24:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.753+0000 7f4fac239700 1 -- 172.21.15.107:0/209460889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f4fa419f780 con 0x7f4fa4108920 2024-05-09T08:24:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.754+0000 7f4f9affd700 1 -- 172.21.15.107:0/209460889 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f4fa002c300 con 0x7f4fa4108920 2024-05-09T08:24:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.756+0000 7f4fac239700 1 -- 172.21.15.107:0/209460889 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4f90039150 msgr2=0x7f4f9003b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.756+0000 7f4fac239700 1 --2- 172.21.15.107:0/209460889 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4f90039150 0x7f4f9003b610 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4f94006fd0 tx=0x7f4f94006e40 comp rx=0 tx=0).stop 2024-05-09T08:24:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.756+0000 7f4fac239700 1 -- 172.21.15.107:0/209460889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa4108920 msgr2=0x7f4fa419bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.756+0000 7f4fac239700 1 --2- 172.21.15.107:0/209460889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa4108920 0x7f4fa419bc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0009e50 tx=0x7f4fa0003d00 comp rx=0 tx=0).stop 2024-05-09T08:24:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.756+0000 7f4fac239700 1 -- 172.21.15.107:0/209460889 shutdown_connections 2024-05-09T08:24:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.756+0000 7f4fac239700 1 --2- 172.21.15.107:0/209460889 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4f90039150 0x7f4f9003b610 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.756+0000 7f4fac239700 1 --2- 172.21.15.107:0/209460889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa4108920 0x7f4fa419bc80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.756+0000 7f4fac239700 1 -- 172.21.15.107:0/209460889 >> 172.21.15.107:0/209460889 conn(0x7f4fa41022d0 msgr2=0x7f4fa4102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.756+0000 7f4fac239700 1 -- 172.21.15.107:0/209460889 shutdown_connections 2024-05-09T08:24:03.317 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:02.756+0000 7f4fac239700 1 -- 172.21.15.107:0/209460889 wait complete. 2024-05-09T08:24:04.664 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.795+0000 7fbbd9e47700 1 Processor -- start 2024-05-09T08:24:04.664 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.795+0000 7fbbd9e47700 1 -- start start 2024-05-09T08:24:04.664 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.795+0000 7fbbd9e47700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbd4108920 0x7fbbd410ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:04.664 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.795+0000 7fbbd9e47700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd407c6e0 con 0x7fbbd4108920 2024-05-09T08:24:04.664 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.795+0000 7fbbd37fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbd4108920 0x7fbbd410ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:04.664 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.796+0000 7fbbd37fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbd4108920 0x7fbbd410ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43124/0 (socket says 172.21.15.107:43124) 2024-05-09T08:24:04.664 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.796+0000 7fbbd37fe700 1 -- 172.21.15.107:0/1880364594 learned_addr learned my addr 172.21.15.107:0/1880364594 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:24:04.665 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.796+0000 7fbbd37fe700 1 -- 172.21.15.107:0/1880364594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd410b250 con 0x7fbbd4108920 2024-05-09T08:24:04.665 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.796+0000 7fbbd37fe700 1 --2- 172.21.15.107:0/1880364594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbd4108920 0x7fbbd410ad10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4009e80 tx=0x7fbbc400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6bb27d4977e098d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:04.665 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.796+0000 7fbbd27fc700 1 -- 172.21.15.107:0/1880364594 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbbc4017070 con 0x7fbbd4108920 2024-05-09T08:24:04.665 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.796+0000 7fbbd27fc700 1 -- 172.21.15.107:0/1880364594 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbc400b8c0 con 0x7fbbd4108920 2024-05-09T08:24:04.665 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.796+0000 7fbbd27fc700 1 -- 172.21.15.107:0/1880364594 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbbc4004490 con 0x7fbbd4108920 2024-05-09T08:24:04.665 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.797+0000 7fbbd9e47700 1 -- 172.21.15.107:0/1880364594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbd4108920 msgr2=0x7fbbd410ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.665 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.797+0000 7fbbd9e47700 1 --2- 172.21.15.107:0/1880364594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbd4108920 0x7fbbd410ad10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4009e80 tx=0x7fbbc400b330 comp rx=0 tx=0).stop 2024-05-09T08:24:04.665 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.797+0000 7fbbd9e47700 1 -- 172.21.15.107:0/1880364594 shutdown_connections 2024-05-09T08:24:04.665 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.797+0000 7fbbd9e47700 1 --2- 172.21.15.107:0/1880364594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbd4108920 0x7fbbd410ad10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.665 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.797+0000 7fbbd9e47700 1 -- 172.21.15.107:0/1880364594 >> 172.21.15.107:0/1880364594 conn(0x7fbbd4102350 msgr2=0x7fbbd4104770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:04.665 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.797+0000 7fbbd9e47700 1 -- 172.21.15.107:0/1880364594 shutdown_connections 2024-05-09T08:24:04.665 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.797+0000 7fbbd9e47700 1 -- 172.21.15.107:0/1880364594 wait complete. 2024-05-09T08:24:04.665 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.797+0000 7fbbd9e47700 1 Processor -- start 2024-05-09T08:24:04.666 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.797+0000 7fbbd9e47700 1 -- start start 2024-05-09T08:24:04.666 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.798+0000 7fbbd9e47700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbd4108920 0x7fbbd4199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:04.666 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.798+0000 7fbbd9e47700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd4106e20 con 0x7fbbd4108920 2024-05-09T08:24:04.666 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.798+0000 7fbbd37fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbd4108920 0x7fbbd4199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:04.666 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.798+0000 7fbbd37fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbd4108920 0x7fbbd4199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43140/0 (socket says 172.21.15.107:43140) 2024-05-09T08:24:04.666 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.798+0000 7fbbd37fe700 1 -- 172.21.15.107:0/772816895 learned_addr learned my addr 172.21.15.107:0/772816895 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:24:04.666 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.798+0000 7fbbd37fe700 1 -- 172.21.15.107:0/772816895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd4199fc0 con 0x7fbbd4108920 2024-05-09T08:24:04.666 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.798+0000 7fbbd37fe700 1 --2- 172.21.15.107:0/772816895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbd4108920 0x7fbbd4199a80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4000f80 tx=0x7fbbc4003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:04.666 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.798+0000 7fbbd0ff9700 1 -- 172.21.15.107:0/772816895 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbbc4020020 con 0x7fbbd4108920 2024-05-09T08:24:04.666 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.798+0000 7fbbd0ff9700 1 -- 172.21.15.107:0/772816895 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbc400bba0 con 0x7fbbd4108920 2024-05-09T08:24:04.666 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.798+0000 7fbbd0ff9700 1 -- 172.21.15.107:0/772816895 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbbc4017040 con 0x7fbbd4108920 2024-05-09T08:24:04.666 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.798+0000 7fbbd9e47700 1 -- 172.21.15.107:0/772816895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbd419a250 con 0x7fbbd4108920 2024-05-09T08:24:04.666 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.798+0000 7fbbd9e47700 1 -- 172.21.15.107:0/772816895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbd419cf40 con 0x7fbbd4108920 2024-05-09T08:24:04.666 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.799+0000 7fbbd0ff9700 1 -- 172.21.15.107:0/772816895 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fbbc4023020 con 0x7fbbd4108920 2024-05-09T08:24:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.799+0000 7fbbd9e47700 1 -- 172.21.15.107:0/772816895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbd4063980 con 0x7fbbd4108920 2024-05-09T08:24:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.800+0000 7fbbd0ff9700 1 --2- 172.21.15.107:0/772816895 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbbbc03d5a0 0x7fbbbc03fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.800+0000 7fbbd0ff9700 1 -- 172.21.15.107:0/772816895 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbbc400f050 con 0x7fbbd4108920 2024-05-09T08:24:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.800+0000 7fbbd2ffd700 1 --2- 172.21.15.107:0/772816895 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbbbc03d5a0 0x7fbbbc03fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.800+0000 7fbbd2ffd700 1 --2- 172.21.15.107:0/772816895 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbbbc03d5a0 0x7fbbbc03fa60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8006fd0 tx=0x7fbbc8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:03.804+0000 7fbbd0ff9700 1 -- 172.21.15.107:0/772816895 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbbc402a020 con 0x7fbbd4108920 2024-05-09T08:24:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:04.107+0000 7fbbd9e47700 1 -- 172.21.15.107:0/772816895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fbbd4193000 con 0x7fbbd4108920 2024-05-09T08:24:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:04.109+0000 7fbbd0ff9700 1 -- 172.21.15.107:0/772816895 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7fbbd4193000 con 0x7fbbd4108920 2024-05-09T08:24:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:04.111+0000 7fbbd9e47700 1 -- 172.21.15.107:0/772816895 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbbbc03d5a0 msgr2=0x7fbbbc03fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:04.111+0000 7fbbd9e47700 1 --2- 172.21.15.107:0/772816895 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbbbc03d5a0 0x7fbbbc03fa60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8006fd0 tx=0x7fbbc8006e40 comp rx=0 tx=0).stop 2024-05-09T08:24:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:04.111+0000 7fbbd9e47700 1 -- 172.21.15.107:0/772816895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbd4108920 msgr2=0x7fbbd4199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:04.111+0000 7fbbd9e47700 1 --2- 172.21.15.107:0/772816895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbd4108920 0x7fbbd4199a80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4000f80 tx=0x7fbbc4003d70 comp rx=0 tx=0).stop 2024-05-09T08:24:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:04.111+0000 7fbbd9e47700 1 -- 172.21.15.107:0/772816895 shutdown_connections 2024-05-09T08:24:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:04.111+0000 7fbbd9e47700 1 --2- 172.21.15.107:0/772816895 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbbbc03d5a0 0x7fbbbc03fa60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.667 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:04.111+0000 7fbbd9e47700 1 --2- 172.21.15.107:0/772816895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbd4108920 0x7fbbd4199a80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:04.668 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:04.111+0000 7fbbd9e47700 1 -- 172.21.15.107:0/772816895 >> 172.21.15.107:0/772816895 conn(0x7fbbd4102350 msgr2=0x7fbbd4104740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:04.668 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:04.112+0000 7fbbd9e47700 1 -- 172.21.15.107:0/772816895 shutdown_connections 2024-05-09T08:24:04.668 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr 2024-05-09T08:24:04.112+0000 7fbbd9e47700 1 -- 172.21.15.107:0/772816895 wait complete. 2024-05-09T08:24:04.668 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-09T08:24:04.668 INFO:teuthology.orchestra.run.smithi107.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-09T08:24:04.668 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:24:04.668 INFO:teuthology.orchestra.run.smithi107.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:24:04.668 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:24:04.668 INFO:teuthology.orchestra.run.smithi107.stdout:Or, if you are only running a single cluster on this host: 2024-05-09T08:24:04.668 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:24:04.668 INFO:teuthology.orchestra.run.smithi107.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-09T08:24:04.668 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:24:04.668 INFO:teuthology.orchestra.run.smithi107.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-09T08:24:04.668 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:24:04.669 INFO:teuthology.orchestra.run.smithi107.stdout: ceph telemetry on 2024-05-09T08:24:04.669 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:24:04.669 INFO:teuthology.orchestra.run.smithi107.stdout:For more information see: 2024-05-09T08:24:04.669 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:24:04.669 INFO:teuthology.orchestra.run.smithi107.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-09T08:24:04.669 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:24:04.669 INFO:teuthology.orchestra.run.smithi107.stdout:Bootstrap complete. 2024-05-09T08:24:04.701 INFO:tasks.cephadm:Fetching config... 2024-05-09T08:24:04.701 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:24:04.701 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-09T08:24:04.718 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-09T08:24:04.718 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:24:04.718 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-09T08:24:04.774 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-09T08:24:04.774 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:24:04.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-09T08:24:04.840 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-09T08:24:04.840 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:24:04.840 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-09T08:24:04.896 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-09T08:24:04.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDvTcCDkTv5fUm8+6hFqg8pxKA6RI6aMD6TeJ/E+TBwDuZ4DjQZ7eKDaSdTx/xjw5Yd8Uf5MaG47X7yclkmoOFeEVb1qyrjXLM3+L7GMak2BrY/X9Ab641f3geXD1wcNbIra5XncAf1oZvisw5Wk0XNkDhErEIpw7LPHSZU1cMGBrqL4YCirmvwF+rMEBRAsd0gCqarwLgCOu3ls/13WMwLfIdUkzqfcpLYVMeMjRyFwAltslJGH53+w3RfYdYQOqQ6w+eIr2dkHOWewShY9MbHJ1VBslK6xqD3nb95+oDtICsdaCn7sNDq/38vx9wX56n2LffC9Lq3G4eqrSMsi3fksxbc71jNSOSphys/pvk1947Va5njIggp+H8il/5389ZRuYn/H187OMk3TBBDC8JOpSQoY8RI+e8ncFjQOp/AfUFRzgNTIdbPOZwHtPsyJWVjTIJTLj9NsrXFTjaGr8qFvltLhMd7xYRCaA2MjZkN7TlgJP3HMRSmDUDCCWqQDCU= ceph-3954c88c-0ddd-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-09T08:24:04.972 INFO:teuthology.orchestra.run.smithi107.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDvTcCDkTv5fUm8+6hFqg8pxKA6RI6aMD6TeJ/E+TBwDuZ4DjQZ7eKDaSdTx/xjw5Yd8Uf5MaG47X7yclkmoOFeEVb1qyrjXLM3+L7GMak2BrY/X9Ab641f3geXD1wcNbIra5XncAf1oZvisw5Wk0XNkDhErEIpw7LPHSZU1cMGBrqL4YCirmvwF+rMEBRAsd0gCqarwLgCOu3ls/13WMwLfIdUkzqfcpLYVMeMjRyFwAltslJGH53+w3RfYdYQOqQ6w+eIr2dkHOWewShY9MbHJ1VBslK6xqD3nb95+oDtICsdaCn7sNDq/38vx9wX56n2LffC9Lq3G4eqrSMsi3fksxbc71jNSOSphys/pvk1947Va5njIggp+H8il/5389ZRuYn/H187OMk3TBBDC8JOpSQoY8RI+e8ncFjQOp/AfUFRzgNTIdbPOZwHtPsyJWVjTIJTLj9NsrXFTjaGr8qFvltLhMd7xYRCaA2MjZkN7TlgJP3HMRSmDUDCCWqQDCU= ceph-3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:04.983 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDvTcCDkTv5fUm8+6hFqg8pxKA6RI6aMD6TeJ/E+TBwDuZ4DjQZ7eKDaSdTx/xjw5Yd8Uf5MaG47X7yclkmoOFeEVb1qyrjXLM3+L7GMak2BrY/X9Ab641f3geXD1wcNbIra5XncAf1oZvisw5Wk0XNkDhErEIpw7LPHSZU1cMGBrqL4YCirmvwF+rMEBRAsd0gCqarwLgCOu3ls/13WMwLfIdUkzqfcpLYVMeMjRyFwAltslJGH53+w3RfYdYQOqQ6w+eIr2dkHOWewShY9MbHJ1VBslK6xqD3nb95+oDtICsdaCn7sNDq/38vx9wX56n2LffC9Lq3G4eqrSMsi3fksxbc71jNSOSphys/pvk1947Va5njIggp+H8il/5389ZRuYn/H187OMk3TBBDC8JOpSQoY8RI+e8ncFjQOp/AfUFRzgNTIdbPOZwHtPsyJWVjTIJTLj9NsrXFTjaGr8qFvltLhMd7xYRCaA2MjZkN7TlgJP3HMRSmDUDCCWqQDCU= ceph-3954c88c-0ddd-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-09T08:24:05.025 INFO:teuthology.orchestra.run.smithi110.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDvTcCDkTv5fUm8+6hFqg8pxKA6RI6aMD6TeJ/E+TBwDuZ4DjQZ7eKDaSdTx/xjw5Yd8Uf5MaG47X7yclkmoOFeEVb1qyrjXLM3+L7GMak2BrY/X9Ab641f3geXD1wcNbIra5XncAf1oZvisw5Wk0XNkDhErEIpw7LPHSZU1cMGBrqL4YCirmvwF+rMEBRAsd0gCqarwLgCOu3ls/13WMwLfIdUkzqfcpLYVMeMjRyFwAltslJGH53+w3RfYdYQOqQ6w+eIr2dkHOWewShY9MbHJ1VBslK6xqD3nb95+oDtICsdaCn7sNDq/38vx9wX56n2LffC9Lq3G4eqrSMsi3fksxbc71jNSOSphys/pvk1947Va5njIggp+H8il/5389ZRuYn/H187OMk3TBBDC8JOpSQoY8RI+e8ncFjQOp/AfUFRzgNTIdbPOZwHtPsyJWVjTIJTLj9NsrXFTjaGr8qFvltLhMd7xYRCaA2MjZkN7TlgJP3HMRSmDUDCCWqQDCU= ceph-3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:05.038 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDvTcCDkTv5fUm8+6hFqg8pxKA6RI6aMD6TeJ/E+TBwDuZ4DjQZ7eKDaSdTx/xjw5Yd8Uf5MaG47X7yclkmoOFeEVb1qyrjXLM3+L7GMak2BrY/X9Ab641f3geXD1wcNbIra5XncAf1oZvisw5Wk0XNkDhErEIpw7LPHSZU1cMGBrqL4YCirmvwF+rMEBRAsd0gCqarwLgCOu3ls/13WMwLfIdUkzqfcpLYVMeMjRyFwAltslJGH53+w3RfYdYQOqQ6w+eIr2dkHOWewShY9MbHJ1VBslK6xqD3nb95+oDtICsdaCn7sNDq/38vx9wX56n2LffC9Lq3G4eqrSMsi3fksxbc71jNSOSphys/pvk1947Va5njIggp+H8il/5389ZRuYn/H187OMk3TBBDC8JOpSQoY8RI+e8ncFjQOp/AfUFRzgNTIdbPOZwHtPsyJWVjTIJTLj9NsrXFTjaGr8qFvltLhMd7xYRCaA2MjZkN7TlgJP3HMRSmDUDCCWqQDCU= ceph-3954c88c-0ddd-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-09T08:24:05.082 INFO:teuthology.orchestra.run.smithi157.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDvTcCDkTv5fUm8+6hFqg8pxKA6RI6aMD6TeJ/E+TBwDuZ4DjQZ7eKDaSdTx/xjw5Yd8Uf5MaG47X7yclkmoOFeEVb1qyrjXLM3+L7GMak2BrY/X9Ab641f3geXD1wcNbIra5XncAf1oZvisw5Wk0XNkDhErEIpw7LPHSZU1cMGBrqL4YCirmvwF+rMEBRAsd0gCqarwLgCOu3ls/13WMwLfIdUkzqfcpLYVMeMjRyFwAltslJGH53+w3RfYdYQOqQ6w+eIr2dkHOWewShY9MbHJ1VBslK6xqD3nb95+oDtICsdaCn7sNDq/38vx9wX56n2LffC9Lq3G4eqrSMsi3fksxbc71jNSOSphys/pvk1947Va5njIggp+H8il/5389ZRuYn/H187OMk3TBBDC8JOpSQoY8RI+e8ncFjQOp/AfUFRzgNTIdbPOZwHtPsyJWVjTIJTLj9NsrXFTjaGr8qFvltLhMd7xYRCaA2MjZkN7TlgJP3HMRSmDUDCCWqQDCU= ceph-3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:05.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-09T08:24:05.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:05 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/772816895' entity='client.admin' 2024-05-09T08:24:05.335 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:07.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.570+0000 7f8ee32e0700 1 -- 172.21.15.107:0/2252429383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8edc100c70 msgr2=0x7f8edc101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.570+0000 7f8ee32e0700 1 --2- 172.21.15.107:0/2252429383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8edc100c70 0x7f8edc101090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8009910 tx=0x7f8ed8009c20 comp rx=0 tx=0).stop 2024-05-09T08:24:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.570+0000 7f8ee32e0700 1 -- 172.21.15.107:0/2252429383 shutdown_connections 2024-05-09T08:24:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.570+0000 7f8ee32e0700 1 --2- 172.21.15.107:0/2252429383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8edc100c70 0x7f8edc101090 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.570+0000 7f8ee32e0700 1 -- 172.21.15.107:0/2252429383 >> 172.21.15.107:0/2252429383 conn(0x7f8edc0fc230 msgr2=0x7f8edc0fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.570+0000 7f8ee32e0700 1 -- 172.21.15.107:0/2252429383 shutdown_connections 2024-05-09T08:24:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.570+0000 7f8ee32e0700 1 -- 172.21.15.107:0/2252429383 wait complete. 2024-05-09T08:24:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.571+0000 7f8ee32e0700 1 Processor -- start 2024-05-09T08:24:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.571+0000 7f8ee32e0700 1 -- start start 2024-05-09T08:24:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.571+0000 7f8ee32e0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8edc100c70 0x7f8edc197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.571+0000 7f8ee32e0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc106bb0 con 0x7f8edc100c70 2024-05-09T08:24:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.571+0000 7f8ee107c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8edc100c70 0x7f8edc197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.571+0000 7f8ee107c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8edc100c70 0x7f8edc197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43152/0 (socket says 172.21.15.107:43152) 2024-05-09T08:24:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.571+0000 7f8ee107c700 1 -- 172.21.15.107:0/1115047513 learned_addr learned my addr 172.21.15.107:0/1115047513 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:24:07.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.571+0000 7f8ee107c700 1 -- 172.21.15.107:0/1115047513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8edc1978b0 con 0x7f8edc100c70 2024-05-09T08:24:07.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.572+0000 7f8ee107c700 1 --2- 172.21.15.107:0/1115047513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8edc100c70 0x7f8edc197370 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8005950 tx=0x7f8ed8004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:07.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.572+0000 7f8ed27fc700 1 -- 172.21.15.107:0/1115047513 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ed8026070 con 0x7f8edc100c70 2024-05-09T08:24:07.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.572+0000 7f8ed27fc700 1 -- 172.21.15.107:0/1115047513 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ed800bc00 con 0x7f8edc100c70 2024-05-09T08:24:07.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.572+0000 7f8ee32e0700 1 -- 172.21.15.107:0/1115047513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8edc197b40 con 0x7f8edc100c70 2024-05-09T08:24:07.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.572+0000 7f8ed27fc700 1 -- 172.21.15.107:0/1115047513 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ed80039a0 con 0x7f8edc100c70 2024-05-09T08:24:07.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.572+0000 7f8ee32e0700 1 -- 172.21.15.107:0/1115047513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8edc19a830 con 0x7f8edc100c70 2024-05-09T08:24:07.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.573+0000 7f8ed27fc700 1 -- 172.21.15.107:0/1115047513 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f8ed8029070 con 0x7f8edc100c70 2024-05-09T08:24:07.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.573+0000 7f8ee32e0700 1 -- 172.21.15.107:0/1115047513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8edc107330 con 0x7f8edc100c70 2024-05-09T08:24:07.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.573+0000 7f8ed27fc700 1 --2- 172.21.15.107:0/1115047513 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8ec80390f0 0x7f8ec803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:07.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.573+0000 7f8ed27fc700 1 -- 172.21.15.107:0/1115047513 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8ed801e070 con 0x7f8edc100c70 2024-05-09T08:24:07.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.573+0000 7f8ee087b700 1 --2- 172.21.15.107:0/1115047513 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8ec80390f0 0x7f8ec803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:07.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.573+0000 7f8ee087b700 1 --2- 172.21.15.107:0/1115047513 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8ec80390f0 0x7f8ec803b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc006fd0 tx=0x7f8ecc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:07.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.578+0000 7f8ed27fc700 1 -- 172.21.15.107:0/1115047513 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8ed8013070 con 0x7f8edc100c70 2024-05-09T08:24:07.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.777+0000 7f8ee32e0700 1 -- 172.21.15.107:0/1115047513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f8edc02dc90 con 0x7f8edc100c70 2024-05-09T08:24:07.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.781+0000 7f8ed27fc700 1 -- 172.21.15.107:0/1115047513 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f8ed80300c0 con 0x7f8edc100c70 2024-05-09T08:24:07.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.783+0000 7f8ee32e0700 1 -- 172.21.15.107:0/1115047513 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8ec80390f0 msgr2=0x7f8ec803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:07.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.783+0000 7f8ee32e0700 1 --2- 172.21.15.107:0/1115047513 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8ec80390f0 0x7f8ec803b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc006fd0 tx=0x7f8ecc006e40 comp rx=0 tx=0).stop 2024-05-09T08:24:07.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.783+0000 7f8ee32e0700 1 -- 172.21.15.107:0/1115047513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8edc100c70 msgr2=0x7f8edc197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:07.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.783+0000 7f8ee32e0700 1 --2- 172.21.15.107:0/1115047513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8edc100c70 0x7f8edc197370 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8005950 tx=0x7f8ed8004fa0 comp rx=0 tx=0).stop 2024-05-09T08:24:07.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.783+0000 7f8ee32e0700 1 -- 172.21.15.107:0/1115047513 shutdown_connections 2024-05-09T08:24:07.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.783+0000 7f8ee32e0700 1 --2- 172.21.15.107:0/1115047513 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8ec80390f0 0x7f8ec803b5b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:07.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.783+0000 7f8ee32e0700 1 --2- 172.21.15.107:0/1115047513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8edc100c70 0x7f8edc197370 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:07.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.783+0000 7f8ee32e0700 1 -- 172.21.15.107:0/1115047513 >> 172.21.15.107:0/1115047513 conn(0x7f8edc0fc230 msgr2=0x7f8edc06a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:07.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.783+0000 7f8ee32e0700 1 -- 172.21.15.107:0/1115047513 shutdown_connections 2024-05-09T08:24:07.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:07.783+0000 7f8ee32e0700 1 -- 172.21.15.107:0/1115047513 wait complete. 2024-05-09T08:24:08.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:08 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:08.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:08 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:08.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:08 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:24:08.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:08 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:08.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:08 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:24:08.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:08 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:08.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:08 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1115047513' entity='client.admin' 2024-05-09T08:24:08.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:08 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:24:08.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:08 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:08.377 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-09T08:24:08.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-09T08:24:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:10.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.063+0000 7f025495c700 1 -- 172.21.15.107:0/3459344409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0250100c90 msgr2=0x7f02501010b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:10.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.063+0000 7f025495c700 1 --2- 172.21.15.107:0/3459344409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0250100c90 0x7f02501010b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0240009910 tx=0x7f0240009c20 comp rx=0 tx=0).stop 2024-05-09T08:24:10.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.063+0000 7f025495c700 1 -- 172.21.15.107:0/3459344409 shutdown_connections 2024-05-09T08:24:10.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.063+0000 7f025495c700 1 --2- 172.21.15.107:0/3459344409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0250100c90 0x7f02501010b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:10.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.063+0000 7f025495c700 1 -- 172.21.15.107:0/3459344409 >> 172.21.15.107:0/3459344409 conn(0x7f02500fc230 msgr2=0x7f02500fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:10.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.064+0000 7f025495c700 1 -- 172.21.15.107:0/3459344409 shutdown_connections 2024-05-09T08:24:10.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.064+0000 7f025495c700 1 -- 172.21.15.107:0/3459344409 wait complete. 2024-05-09T08:24:10.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.064+0000 7f025495c700 1 Processor -- start 2024-05-09T08:24:10.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.064+0000 7f025495c700 1 -- start start 2024-05-09T08:24:10.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.064+0000 7f025495c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0250100c90 0x7f0250199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:10.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.064+0000 7f025495c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0250106bd0 con 0x7f0250100c90 2024-05-09T08:24:10.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.064+0000 7f024e59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0250100c90 0x7f0250199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:10.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.065+0000 7f024e59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0250100c90 0x7f0250199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43160/0 (socket says 172.21.15.107:43160) 2024-05-09T08:24:10.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.065+0000 7f024e59c700 1 -- 172.21.15.107:0/3176591823 learned_addr learned my addr 172.21.15.107:0/3176591823 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:24:10.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.065+0000 7f024e59c700 1 -- 172.21.15.107:0/3176591823 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0250199b70 con 0x7f0250100c90 2024-05-09T08:24:10.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.065+0000 7f024e59c700 1 --2- 172.21.15.107:0/3176591823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0250100c90 0x7f0250199630 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0240004fb0 tx=0x7f0240009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:10.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.065+0000 7f02477fe700 1 -- 172.21.15.107:0/3176591823 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0240027070 con 0x7f0250100c90 2024-05-09T08:24:10.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.065+0000 7f02477fe700 1 -- 172.21.15.107:0/3176591823 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f024000be10 con 0x7f0250100c90 2024-05-09T08:24:10.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.065+0000 7f025495c700 1 -- 172.21.15.107:0/3176591823 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0250199e00 con 0x7f0250100c90 2024-05-09T08:24:10.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.065+0000 7f02477fe700 1 -- 172.21.15.107:0/3176591823 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0240022de0 con 0x7f0250100c90 2024-05-09T08:24:10.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.065+0000 7f025495c700 1 -- 172.21.15.107:0/3176591823 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f025019caf0 con 0x7f0250100c90 2024-05-09T08:24:10.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.066+0000 7f02477fe700 1 -- 172.21.15.107:0/3176591823 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f024002a070 con 0x7f0250100c90 2024-05-09T08:24:10.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.066+0000 7f025495c700 1 -- 172.21.15.107:0/3176591823 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0250107350 con 0x7f0250100c90 2024-05-09T08:24:10.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.066+0000 7f02477fe700 1 --2- 172.21.15.107:0/3176591823 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f023c0390f0 0x7f023c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:10.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.066+0000 7f02477fe700 1 -- 172.21.15.107:0/3176591823 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f024001e030 con 0x7f0250100c90 2024-05-09T08:24:10.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.066+0000 7f024dd9b700 1 --2- 172.21.15.107:0/3176591823 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f023c0390f0 0x7f023c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:10.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.067+0000 7f024dd9b700 1 --2- 172.21.15.107:0/3176591823 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f023c0390f0 0x7f023c03b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0238006fd0 tx=0x7f0238006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:10.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.071+0000 7f02477fe700 1 -- 172.21.15.107:0/3176591823 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0240013070 con 0x7f0250100c90 2024-05-09T08:24:10.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.269+0000 7f025495c700 1 -- 172.21.15.107:0/3176591823 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f025002dc90 con 0x7f023c0390f0 2024-05-09T08:24:10.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.273+0000 7f02477fe700 1 -- 172.21.15.107:0/3176591823 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f025002dc90 con 0x7f023c0390f0 2024-05-09T08:24:10.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.275+0000 7f025495c700 1 -- 172.21.15.107:0/3176591823 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f023c0390f0 msgr2=0x7f023c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:10.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.275+0000 7f025495c700 1 --2- 172.21.15.107:0/3176591823 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f023c0390f0 0x7f023c03b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0238006fd0 tx=0x7f0238006e40 comp rx=0 tx=0).stop 2024-05-09T08:24:10.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.275+0000 7f025495c700 1 -- 172.21.15.107:0/3176591823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0250100c90 msgr2=0x7f0250199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:10.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.275+0000 7f025495c700 1 --2- 172.21.15.107:0/3176591823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0250100c90 0x7f0250199630 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0240004fb0 tx=0x7f0240009f60 comp rx=0 tx=0).stop 2024-05-09T08:24:10.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.275+0000 7f025495c700 1 -- 172.21.15.107:0/3176591823 shutdown_connections 2024-05-09T08:24:10.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.275+0000 7f025495c700 1 --2- 172.21.15.107:0/3176591823 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f023c0390f0 0x7f023c03b5b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:10.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.275+0000 7f025495c700 1 --2- 172.21.15.107:0/3176591823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0250100c90 0x7f0250199630 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:10.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.275+0000 7f025495c700 1 -- 172.21.15.107:0/3176591823 >> 172.21.15.107:0/3176591823 conn(0x7f02500fc230 msgr2=0x7f0250190680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:10.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.275+0000 7f025495c700 1 -- 172.21.15.107:0/3176591823 shutdown_connections 2024-05-09T08:24:10.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:10.275+0000 7f025495c700 1 -- 172.21.15.107:0/3176591823 wait complete. 2024-05-09T08:24:11.116 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi110 2024-05-09T08:24:11.117 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-09T08:24:11.117 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/etc/ceph/ceph.conf 2024-05-09T08:24:11.135 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-09T08:24:11.136 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:24:11.194 INFO:tasks.cephadm:Adding host smithi110 to orchestrator... 2024-05-09T08:24:11.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch host add smithi110 2024-05-09T08:24:11.427 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:11.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:11 smithi107 ceph-mon[30122]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:11.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:11.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:24:11.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:11.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:24:11.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:11 smithi107 ceph-mon[30122]: Updating smithi107:/etc/ceph/ceph.conf 2024-05-09T08:24:11.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:11.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:11.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:12.503 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:12 smithi107 ceph-mon[30122]: Updating smithi107:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:12.503 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:12 smithi107 ceph-mon[30122]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:24:12.503 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:12 smithi107 ceph-mon[30122]: Updating smithi107:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:24:13.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.011+0000 7fb71b17a700 1 -- 172.21.15.107:0/451756963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb714106d50 msgr2=0x7fb714107130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:13.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.011+0000 7fb71b17a700 1 --2- 172.21.15.107:0/451756963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb714106d50 0x7fb714107130 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb708009910 tx=0x7fb708009c20 comp rx=0 tx=0).stop 2024-05-09T08:24:13.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.011+0000 7fb71b17a700 1 -- 172.21.15.107:0/451756963 shutdown_connections 2024-05-09T08:24:13.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.011+0000 7fb71b17a700 1 --2- 172.21.15.107:0/451756963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb714106d50 0x7fb714107130 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:13.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.011+0000 7fb71b17a700 1 -- 172.21.15.107:0/451756963 >> 172.21.15.107:0/451756963 conn(0x7fb7140fbe70 msgr2=0x7fb7140fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:13.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.011+0000 7fb71b17a700 1 -- 172.21.15.107:0/451756963 shutdown_connections 2024-05-09T08:24:13.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.011+0000 7fb71b17a700 1 -- 172.21.15.107:0/451756963 wait complete. 2024-05-09T08:24:13.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.012+0000 7fb71b17a700 1 Processor -- start 2024-05-09T08:24:13.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.012+0000 7fb71b17a700 1 -- start start 2024-05-09T08:24:13.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.012+0000 7fb71b17a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb714106d50 0x7fb71419d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:13.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.012+0000 7fb71b17a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb71410c360 con 0x7fb714106d50 2024-05-09T08:24:13.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.012+0000 7fb718f16700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb714106d50 0x7fb71419d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:13.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.012+0000 7fb718f16700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb714106d50 0x7fb71419d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47920/0 (socket says 172.21.15.107:47920) 2024-05-09T08:24:13.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.012+0000 7fb718f16700 1 -- 172.21.15.107:0/2202054400 learned_addr learned my addr 172.21.15.107:0/2202054400 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:24:13.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.012+0000 7fb718f16700 1 -- 172.21.15.107:0/2202054400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71419d960 con 0x7fb714106d50 2024-05-09T08:24:13.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.012+0000 7fb718f16700 1 --2- 172.21.15.107:0/2202054400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb714106d50 0x7fb71419d420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb7080056d0 tx=0x7fb7080057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:13.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.013+0000 7fb711ffb700 1 -- 172.21.15.107:0/2202054400 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb708026070 con 0x7fb714106d50 2024-05-09T08:24:13.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.013+0000 7fb71b17a700 1 -- 172.21.15.107:0/2202054400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb71419dbf0 con 0x7fb714106d50 2024-05-09T08:24:13.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.013+0000 7fb711ffb700 1 -- 172.21.15.107:0/2202054400 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb70800bc00 con 0x7fb714106d50 2024-05-09T08:24:13.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.013+0000 7fb711ffb700 1 -- 172.21.15.107:0/2202054400 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7080039b0 con 0x7fb714106d50 2024-05-09T08:24:13.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.013+0000 7fb71b17a700 1 -- 172.21.15.107:0/2202054400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7141975e0 con 0x7fb714106d50 2024-05-09T08:24:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.014+0000 7fb711ffb700 1 -- 172.21.15.107:0/2202054400 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fb708029070 con 0x7fb714106d50 2024-05-09T08:24:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.014+0000 7fb71b17a700 1 -- 172.21.15.107:0/2202054400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb714103490 con 0x7fb714106d50 2024-05-09T08:24:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.014+0000 7fb711ffb700 1 --2- 172.21.15.107:0/2202054400 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb7040390f0 0x7fb70403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.014+0000 7fb711ffb700 1 -- 172.21.15.107:0/2202054400 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb70801e070 con 0x7fb714106d50 2024-05-09T08:24:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.014+0000 7fb713fff700 1 --2- 172.21.15.107:0/2202054400 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb7040390f0 0x7fb70403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.014+0000 7fb713fff700 1 --2- 172.21.15.107:0/2202054400 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb7040390f0 0x7fb70403b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb700006fd0 tx=0x7fb700006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:13.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.019+0000 7fb711ffb700 1 -- 172.21.15.107:0/2202054400 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb708013070 con 0x7fb714106d50 2024-05-09T08:24:13.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:13.218+0000 7fb71b17a700 1 -- 172.21.15.107:0/2202054400 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi110", "target": ["mon-mgr", ""]}) -- 0x7fb714063980 con 0x7fb7040390f0 2024-05-09T08:24:14.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:13 smithi107 ceph-mon[30122]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi110", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:15.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:14 smithi107 ceph-mon[30122]: Deploying cephadm binary to smithi110 2024-05-09T08:24:16.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:16 smithi107 ceph-mon[30122]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:17.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:17.417+0000 7fb711ffb700 1 -- 172.21.15.107:0/2202054400 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fb714063980 con 0x7fb7040390f0 2024-05-09T08:24:17.418 INFO:teuthology.orchestra.run.smithi107.stdout:Added host 'smithi110' with addr '172.21.15.110' 2024-05-09T08:24:17.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:17.420+0000 7fb71b17a700 1 -- 172.21.15.107:0/2202054400 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb7040390f0 msgr2=0x7fb70403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:17.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:17.420+0000 7fb71b17a700 1 --2- 172.21.15.107:0/2202054400 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb7040390f0 0x7fb70403b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb700006fd0 tx=0x7fb700006e40 comp rx=0 tx=0).stop 2024-05-09T08:24:17.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:17.420+0000 7fb71b17a700 1 -- 172.21.15.107:0/2202054400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb714106d50 msgr2=0x7fb71419d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:17.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:17.420+0000 7fb71b17a700 1 --2- 172.21.15.107:0/2202054400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb714106d50 0x7fb71419d420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb7080056d0 tx=0x7fb7080057b0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:17.420+0000 7fb71b17a700 1 -- 172.21.15.107:0/2202054400 shutdown_connections 2024-05-09T08:24:17.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:17.420+0000 7fb71b17a700 1 --2- 172.21.15.107:0/2202054400 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb7040390f0 0x7fb70403b5b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:17.420+0000 7fb71b17a700 1 --2- 172.21.15.107:0/2202054400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb714106d50 0x7fb71419d420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:17.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:17.420+0000 7fb71b17a700 1 -- 172.21.15.107:0/2202054400 >> 172.21.15.107:0/2202054400 conn(0x7fb7140fbe70 msgr2=0x7fb714109940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:17.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:17.420+0000 7fb71b17a700 1 -- 172.21.15.107:0/2202054400 shutdown_connections 2024-05-09T08:24:17.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:17.420+0000 7fb71b17a700 1 -- 172.21.15.107:0/2202054400 wait complete. 2024-05-09T08:24:18.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch host ls --format=json 2024-05-09T08:24:18.417 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:18 smithi107 ceph-mon[30122]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:18.417 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:18 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:18.417 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:18 smithi107 ceph-mon[30122]: Added host smithi110 2024-05-09T08:24:18.418 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:18 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:24:18.418 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:18 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:18.440 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:19.709 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:19 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:19.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:19 smithi107 ceph-mon[30122]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:19.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.917+0000 7fafb40b4700 1 -- 172.21.15.107:0/2382871508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafac102f60 msgr2=0x7fafac103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:19.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.917+0000 7fafb40b4700 1 --2- 172.21.15.107:0/2382871508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafac102f60 0x7fafac103340 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fafa400b0a0 tx=0x7fafa400b3b0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.917+0000 7fafb40b4700 1 -- 172.21.15.107:0/2382871508 shutdown_connections 2024-05-09T08:24:19.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.917+0000 7fafb40b4700 1 --2- 172.21.15.107:0/2382871508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafac102f60 0x7fafac103340 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:19.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.917+0000 7fafb40b4700 1 -- 172.21.15.107:0/2382871508 >> 172.21.15.107:0/2382871508 conn(0x7fafac0778f0 msgr2=0x7fafac0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:19.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.917+0000 7fafb40b4700 1 -- 172.21.15.107:0/2382871508 shutdown_connections 2024-05-09T08:24:19.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.917+0000 7fafb40b4700 1 -- 172.21.15.107:0/2382871508 wait complete. 2024-05-09T08:24:19.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.918+0000 7fafb40b4700 1 Processor -- start 2024-05-09T08:24:19.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.918+0000 7fafb40b4700 1 -- start start 2024-05-09T08:24:19.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.918+0000 7fafb40b4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafac19d640 0x7fafac19da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:19.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.918+0000 7fafb40b4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fafac10c260 con 0x7fafac19d640 2024-05-09T08:24:19.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.918+0000 7fafb1e50700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafac19d640 0x7fafac19da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:19.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.918+0000 7fafb1e50700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafac19d640 0x7fafac19da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47928/0 (socket says 172.21.15.107:47928) 2024-05-09T08:24:19.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.918+0000 7fafb1e50700 1 -- 172.21.15.107:0/1494560630 learned_addr learned my addr 172.21.15.107:0/1494560630 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:24:19.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.918+0000 7fafb1e50700 1 -- 172.21.15.107:0/1494560630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafac19df60 con 0x7fafac19d640 2024-05-09T08:24:19.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.918+0000 7fafb1e50700 1 --2- 172.21.15.107:0/1494560630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafac19d640 0x7fafac19da20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fafa40076a0 tx=0x7fafa4009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:19.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.918+0000 7fafaaffd700 1 -- 172.21.15.107:0/1494560630 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafa4021020 con 0x7fafac19d640 2024-05-09T08:24:19.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.919+0000 7fafaaffd700 1 -- 172.21.15.107:0/1494560630 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafa4007e10 con 0x7fafac19d640 2024-05-09T08:24:19.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.919+0000 7fafaaffd700 1 -- 172.21.15.107:0/1494560630 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafa4017040 con 0x7fafac19d640 2024-05-09T08:24:19.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.919+0000 7fafb40b4700 1 -- 172.21.15.107:0/1494560630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafac19e1f0 con 0x7fafac19d640 2024-05-09T08:24:19.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.919+0000 7fafb40b4700 1 -- 172.21.15.107:0/1494560630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafac1979f0 con 0x7fafac19d640 2024-05-09T08:24:19.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.919+0000 7fafaaffd700 1 -- 172.21.15.107:0/1494560630 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fafa4024020 con 0x7fafac19d640 2024-05-09T08:24:19.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.919+0000 7fafb40b4700 1 -- 172.21.15.107:0/1494560630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafac069f90 con 0x7fafac19d640 2024-05-09T08:24:19.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.920+0000 7fafaaffd700 1 --2- 172.21.15.107:0/1494560630 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7faf94039150 0x7faf9403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:19.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.920+0000 7fafaaffd700 1 -- 172.21.15.107:0/1494560630 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fafa400f050 con 0x7fafac19d640 2024-05-09T08:24:19.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.920+0000 7fafb164f700 1 --2- 172.21.15.107:0/1494560630 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7faf94039150 0x7faf9403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:19.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.920+0000 7fafb164f700 1 --2- 172.21.15.107:0/1494560630 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7faf94039150 0x7faf9403b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fafa0006fd0 tx=0x7fafa0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:19.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:19.925+0000 7fafaaffd700 1 -- 172.21.15.107:0/1494560630 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fafa402b020 con 0x7fafac19d640 2024-05-09T08:24:20.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:20.128+0000 7fafb40b4700 1 -- 172.21.15.107:0/1494560630 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fafac198e00 con 0x7faf94039150 2024-05-09T08:24:20.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:20.129+0000 7fafaaffd700 1 -- 172.21.15.107:0/1494560630 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7fafac198e00 con 0x7faf94039150 2024-05-09T08:24:20.130 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:24:20.130 INFO:teuthology.orchestra.run.smithi107.stdout:[{"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}, {"addr": "172.21.15.110", "hostname": "smithi110", "labels": [], "status": ""}] 2024-05-09T08:24:20.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:20.131+0000 7fafb40b4700 1 -- 172.21.15.107:0/1494560630 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7faf94039150 msgr2=0x7faf9403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:20.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:20.131+0000 7fafb40b4700 1 --2- 172.21.15.107:0/1494560630 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7faf94039150 0x7faf9403b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fafa0006fd0 tx=0x7fafa0006e40 comp rx=0 tx=0).stop 2024-05-09T08:24:20.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:20.131+0000 7fafb40b4700 1 -- 172.21.15.107:0/1494560630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafac19d640 msgr2=0x7fafac19da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:20.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:20.131+0000 7fafb40b4700 1 --2- 172.21.15.107:0/1494560630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafac19d640 0x7fafac19da20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fafa40076a0 tx=0x7fafa4009790 comp rx=0 tx=0).stop 2024-05-09T08:24:20.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:20.131+0000 7fafb40b4700 1 -- 172.21.15.107:0/1494560630 shutdown_connections 2024-05-09T08:24:20.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:20.131+0000 7fafb40b4700 1 --2- 172.21.15.107:0/1494560630 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7faf94039150 0x7faf9403b610 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:20.131+0000 7fafb40b4700 1 --2- 172.21.15.107:0/1494560630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafac19d640 0x7fafac19da20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:20.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:20.131+0000 7fafb40b4700 1 -- 172.21.15.107:0/1494560630 >> 172.21.15.107:0/1494560630 conn(0x7fafac0778f0 msgr2=0x7fafac105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:20.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:20.131+0000 7fafb40b4700 1 -- 172.21.15.107:0/1494560630 shutdown_connections 2024-05-09T08:24:20.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:20.131+0000 7fafb40b4700 1 -- 172.21.15.107:0/1494560630 wait complete. 2024-05-09T08:24:20.664 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi157 2024-05-09T08:24:20.664 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-05-09T08:24:20.664 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/etc/ceph/ceph.conf 2024-05-09T08:24:20.683 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-05-09T08:24:20.684 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:24:20.742 INFO:tasks.cephadm:Adding host smithi157 to orchestrator... 2024-05-09T08:24:20.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch host add smithi157 2024-05-09T08:24:20.977 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:21.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:21 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:21.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:21 smithi107 ceph-mon[30122]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:24:22.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:22 smithi107 ceph-mon[30122]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:22.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.538+0000 7fa128c21700 1 -- 172.21.15.107:0/3946544923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa124100e60 msgr2=0x7fa124101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:22.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.538+0000 7fa128c21700 1 --2- 172.21.15.107:0/3946544923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa124100e60 0x7fa124101240 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa118009910 tx=0x7fa118009c20 comp rx=0 tx=0).stop 2024-05-09T08:24:22.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.538+0000 7fa128c21700 1 -- 172.21.15.107:0/3946544923 shutdown_connections 2024-05-09T08:24:22.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.538+0000 7fa128c21700 1 --2- 172.21.15.107:0/3946544923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa124100e60 0x7fa124101240 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:22.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.538+0000 7fa128c21700 1 -- 172.21.15.107:0/3946544923 >> 172.21.15.107:0/3946544923 conn(0x7fa1240fc6f0 msgr2=0x7fa1240feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:22.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.538+0000 7fa128c21700 1 -- 172.21.15.107:0/3946544923 shutdown_connections 2024-05-09T08:24:22.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.538+0000 7fa128c21700 1 -- 172.21.15.107:0/3946544923 wait complete. 2024-05-09T08:24:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.539+0000 7fa128c21700 1 Processor -- start 2024-05-09T08:24:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.539+0000 7fa128c21700 1 -- start start 2024-05-09T08:24:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.539+0000 7fa128c21700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa124100e60 0x7fa124197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.539+0000 7fa128c21700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa12410b0e0 con 0x7fa124100e60 2024-05-09T08:24:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.540+0000 7fa12259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa124100e60 0x7fa124197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.540+0000 7fa12259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa124100e60 0x7fa124197750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54452/0 (socket says 172.21.15.107:54452) 2024-05-09T08:24:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.540+0000 7fa12259c700 1 -- 172.21.15.107:0/3889845244 learned_addr learned my addr 172.21.15.107:0/3889845244 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:24:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.540+0000 7fa12259c700 1 -- 172.21.15.107:0/3889845244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa124197c90 con 0x7fa124100e60 2024-05-09T08:24:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.540+0000 7fa12259c700 1 --2- 172.21.15.107:0/3889845244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa124100e60 0x7fa124197750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa1180056d0 tx=0x7fa1180057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.540+0000 7fa1137fe700 1 -- 172.21.15.107:0/3889845244 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa118026070 con 0x7fa124100e60 2024-05-09T08:24:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.540+0000 7fa1137fe700 1 -- 172.21.15.107:0/3889845244 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa11800bba0 con 0x7fa124100e60 2024-05-09T08:24:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.540+0000 7fa128c21700 1 -- 172.21.15.107:0/3889845244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa124197f80 con 0x7fa124100e60 2024-05-09T08:24:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.540+0000 7fa1137fe700 1 -- 172.21.15.107:0/3889845244 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa1180037f0 con 0x7fa124100e60 2024-05-09T08:24:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.540+0000 7fa128c21700 1 -- 172.21.15.107:0/3889845244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa12419c020 con 0x7fa124100e60 2024-05-09T08:24:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.541+0000 7fa1137fe700 1 -- 172.21.15.107:0/3889845244 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fa118029070 con 0x7fa124100e60 2024-05-09T08:24:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.541+0000 7fa128c21700 1 -- 172.21.15.107:0/3889845244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa124102340 con 0x7fa124100e60 2024-05-09T08:24:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.541+0000 7fa1137fe700 1 --2- 172.21.15.107:0/3889845244 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa10c0390f0 0x7fa10c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:22.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.541+0000 7fa1137fe700 1 -- 172.21.15.107:0/3889845244 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa11801e070 con 0x7fa124100e60 2024-05-09T08:24:22.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.541+0000 7fa121d9b700 1 --2- 172.21.15.107:0/3889845244 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa10c0390f0 0x7fa10c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:22.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.542+0000 7fa121d9b700 1 --2- 172.21.15.107:0/3889845244 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa10c0390f0 0x7fa10c03b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa114006fd0 tx=0x7fa114006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:22.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.546+0000 7fa1137fe700 1 -- 172.21.15.107:0/3889845244 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa118013070 con 0x7fa124100e60 2024-05-09T08:24:22.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:22.744+0000 7fa128c21700 1 -- 172.21.15.107:0/3889845244 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi157", "target": ["mon-mgr", ""]}) -- 0x7fa124198ab0 con 0x7fa10c0390f0 2024-05-09T08:24:24.534 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:24 smithi107 ceph-mon[30122]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi157", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:24.534 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:24 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:24.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:24 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:24.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:24 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:24.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:24 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:24.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:24 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:24:24.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:24 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:24.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:24 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:24:24.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:24 smithi107 ceph-mon[30122]: Updating smithi110:/etc/ceph/ceph.conf 2024-05-09T08:24:24.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:24 smithi107 ceph-mon[30122]: Updating smithi110:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:24.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:24 smithi107 ceph-mon[30122]: Deploying cephadm binary to smithi157 2024-05-09T08:24:24.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:24 smithi107 ceph-mon[30122]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:25.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:25 smithi107 ceph-mon[30122]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:24:25.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:25 smithi107 ceph-mon[30122]: Updating smithi110:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:24:26.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:26 smithi107 ceph-mon[30122]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:26.895 INFO:teuthology.orchestra.run.smithi107.stdout:Added host 'smithi157' with addr '172.21.15.157' 2024-05-09T08:24:26.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:26.892+0000 7fa1137fe700 1 -- 172.21.15.107:0/3889845244 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fa124198ab0 con 0x7fa10c0390f0 2024-05-09T08:24:26.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:26.895+0000 7fa128c21700 1 -- 172.21.15.107:0/3889845244 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa10c0390f0 msgr2=0x7fa10c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:26.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:26.895+0000 7fa128c21700 1 --2- 172.21.15.107:0/3889845244 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa10c0390f0 0x7fa10c03b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa114006fd0 tx=0x7fa114006e40 comp rx=0 tx=0).stop 2024-05-09T08:24:26.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:26.895+0000 7fa128c21700 1 -- 172.21.15.107:0/3889845244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa124100e60 msgr2=0x7fa124197750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:26.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:26.895+0000 7fa128c21700 1 --2- 172.21.15.107:0/3889845244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa124100e60 0x7fa124197750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa1180056d0 tx=0x7fa1180057b0 comp rx=0 tx=0).stop 2024-05-09T08:24:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:26.895+0000 7fa128c21700 1 -- 172.21.15.107:0/3889845244 shutdown_connections 2024-05-09T08:24:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:26.895+0000 7fa128c21700 1 --2- 172.21.15.107:0/3889845244 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa10c0390f0 0x7fa10c03b5b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:26.895+0000 7fa128c21700 1 --2- 172.21.15.107:0/3889845244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa124100e60 0x7fa124197750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:26.895+0000 7fa128c21700 1 -- 172.21.15.107:0/3889845244 >> 172.21.15.107:0/3889845244 conn(0x7fa1240fc6f0 msgr2=0x7fa12406a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:26.895+0000 7fa128c21700 1 -- 172.21.15.107:0/3889845244 shutdown_connections 2024-05-09T08:24:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:26.895+0000 7fa128c21700 1 -- 172.21.15.107:0/3889845244 wait complete. 2024-05-09T08:24:27.473 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:27 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:27.474 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:27 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:27.474 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:27 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:27.474 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:27 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:27.474 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:27 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:24:27.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch host ls --format=json 2024-05-09T08:24:27.738 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:28.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:28 smithi107 ceph-mon[30122]: Added host smithi157 2024-05-09T08:24:28.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:28 smithi107 ceph-mon[30122]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:28.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:28 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:28.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:28 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:29.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.290+0000 7f69229b0700 1 -- 172.21.15.107:0/4259887010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f691c100e80 msgr2=0x7f691c101260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:29.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.290+0000 7f69229b0700 1 --2- 172.21.15.107:0/4259887010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f691c100e80 0x7f691c101260 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f690c009910 tx=0x7f690c009c20 comp rx=0 tx=0).stop 2024-05-09T08:24:29.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.290+0000 7f69229b0700 1 -- 172.21.15.107:0/4259887010 shutdown_connections 2024-05-09T08:24:29.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.290+0000 7f69229b0700 1 --2- 172.21.15.107:0/4259887010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f691c100e80 0x7f691c101260 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:29.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.290+0000 7f69229b0700 1 -- 172.21.15.107:0/4259887010 >> 172.21.15.107:0/4259887010 conn(0x7f691c0fc770 msgr2=0x7f691c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:29.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.290+0000 7f69229b0700 1 -- 172.21.15.107:0/4259887010 shutdown_connections 2024-05-09T08:24:29.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.290+0000 7f69229b0700 1 -- 172.21.15.107:0/4259887010 wait complete. 2024-05-09T08:24:29.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.291+0000 7f69229b0700 1 Processor -- start 2024-05-09T08:24:29.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.291+0000 7f69229b0700 1 -- start start 2024-05-09T08:24:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.291+0000 7f69229b0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f691c100e80 0x7f691c19d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.291+0000 7f69229b0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f691c10b100 con 0x7f691c100e80 2024-05-09T08:24:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.291+0000 7f691bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f691c100e80 0x7f691c19d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.291+0000 7f691bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f691c100e80 0x7f691c19d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54464/0 (socket says 172.21.15.107:54464) 2024-05-09T08:24:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.291+0000 7f691bfff700 1 -- 172.21.15.107:0/1033819851 learned_addr learned my addr 172.21.15.107:0/1033819851 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:24:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.292+0000 7f691bfff700 1 -- 172.21.15.107:0/1033819851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f691c19d990 con 0x7f691c100e80 2024-05-09T08:24:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.292+0000 7f691bfff700 1 --2- 172.21.15.107:0/1033819851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f691c100e80 0x7f691c19d450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f690c0056d0 tx=0x7f690c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.292+0000 7f69197fa700 1 -- 172.21.15.107:0/1033819851 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f690c026070 con 0x7f691c100e80 2024-05-09T08:24:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.292+0000 7f69229b0700 1 -- 172.21.15.107:0/1033819851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f691c19dc80 con 0x7f691c100e80 2024-05-09T08:24:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.292+0000 7f69197fa700 1 -- 172.21.15.107:0/1033819851 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f690c00bba0 con 0x7f691c100e80 2024-05-09T08:24:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.292+0000 7f69197fa700 1 -- 172.21.15.107:0/1033819851 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f690c0037f0 con 0x7f691c100e80 2024-05-09T08:24:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.292+0000 7f69229b0700 1 -- 172.21.15.107:0/1033819851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f691c197610 con 0x7f691c100e80 2024-05-09T08:24:29.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.293+0000 7f69197fa700 1 -- 172.21.15.107:0/1033819851 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f690c029070 con 0x7f691c100e80 2024-05-09T08:24:29.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.293+0000 7f69197fa700 1 --2- 172.21.15.107:0/1033819851 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f69080390f0 0x7f690803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:29.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.293+0000 7f69197fa700 1 -- 172.21.15.107:0/1033819851 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f690c01e070 con 0x7f691c100e80 2024-05-09T08:24:29.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.293+0000 7f69229b0700 1 -- 172.21.15.107:0/1033819851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f691c102360 con 0x7f691c100e80 2024-05-09T08:24:29.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.293+0000 7f691b7fe700 1 --2- 172.21.15.107:0/1033819851 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f69080390f0 0x7f690803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:29.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.293+0000 7f691b7fe700 1 --2- 172.21.15.107:0/1033819851 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f69080390f0 0x7f690803b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6904006fd0 tx=0x7f6904006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:29.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.298+0000 7f69197fa700 1 -- 172.21.15.107:0/1033819851 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f690c030070 con 0x7f691c100e80 2024-05-09T08:24:29.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.500+0000 7f69229b0700 1 -- 172.21.15.107:0/1033819851 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f691c063980 con 0x7f69080390f0 2024-05-09T08:24:29.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.502+0000 7f69197fa700 1 -- 172.21.15.107:0/1033819851 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7f691c063980 con 0x7f69080390f0 2024-05-09T08:24:29.503 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:24:29.503 INFO:teuthology.orchestra.run.smithi107.stdout:[{"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}, {"addr": "172.21.15.110", "hostname": "smithi110", "labels": [], "status": ""}, {"addr": "172.21.15.157", "hostname": "smithi157", "labels": [], "status": ""}] 2024-05-09T08:24:29.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.504+0000 7f69229b0700 1 -- 172.21.15.107:0/1033819851 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f69080390f0 msgr2=0x7f690803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:29.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.504+0000 7f69229b0700 1 --2- 172.21.15.107:0/1033819851 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f69080390f0 0x7f690803b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6904006fd0 tx=0x7f6904006e40 comp rx=0 tx=0).stop 2024-05-09T08:24:29.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.504+0000 7f69229b0700 1 -- 172.21.15.107:0/1033819851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f691c100e80 msgr2=0x7f691c19d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:29.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.504+0000 7f69229b0700 1 --2- 172.21.15.107:0/1033819851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f691c100e80 0x7f691c19d450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f690c0056d0 tx=0x7f690c0057b0 comp rx=0 tx=0).stop 2024-05-09T08:24:29.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.504+0000 7f69229b0700 1 -- 172.21.15.107:0/1033819851 shutdown_connections 2024-05-09T08:24:29.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.504+0000 7f69229b0700 1 --2- 172.21.15.107:0/1033819851 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f69080390f0 0x7f690803b5b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:29.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.504+0000 7f69229b0700 1 --2- 172.21.15.107:0/1033819851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f691c100e80 0x7f691c19d450 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:29.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.504+0000 7f69229b0700 1 -- 172.21.15.107:0/1033819851 >> 172.21.15.107:0/1033819851 conn(0x7f691c0fc770 msgr2=0x7f691c06a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:29.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.504+0000 7f69229b0700 1 -- 172.21.15.107:0/1033819851 shutdown_connections 2024-05-09T08:24:29.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:29.504+0000 7f69229b0700 1 -- 172.21.15.107:0/1033819851 wait complete. 2024-05-09T08:24:30.138 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-09T08:24:30.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd crush tunables default 2024-05-09T08:24:30.372 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:30.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:30 smithi107 ceph-mon[30122]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:30.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:30 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:31.712 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:31 smithi107 ceph-mon[30122]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:24:31.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.945+0000 7f9e3f16e700 1 -- 172.21.15.107:0/364922755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e381004a0 msgr2=0x7f9e38100880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:31.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.945+0000 7f9e3f16e700 1 --2- 172.21.15.107:0/364922755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e381004a0 0x7f9e38100880 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c009910 tx=0x7f9e2c009c20 comp rx=0 tx=0).stop 2024-05-09T08:24:31.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.945+0000 7f9e3f16e700 1 -- 172.21.15.107:0/364922755 shutdown_connections 2024-05-09T08:24:31.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.945+0000 7f9e3f16e700 1 --2- 172.21.15.107:0/364922755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e381004a0 0x7f9e38100880 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:31.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.945+0000 7f9e3f16e700 1 -- 172.21.15.107:0/364922755 >> 172.21.15.107:0/364922755 conn(0x7f9e380fbea0 msgr2=0x7f9e380fe2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:31.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.945+0000 7f9e3f16e700 1 -- 172.21.15.107:0/364922755 shutdown_connections 2024-05-09T08:24:31.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.945+0000 7f9e3f16e700 1 -- 172.21.15.107:0/364922755 wait complete. 2024-05-09T08:24:31.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.946+0000 7f9e3f16e700 1 Processor -- start 2024-05-09T08:24:31.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.946+0000 7f9e3f16e700 1 -- start start 2024-05-09T08:24:31.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.946+0000 7f9e3f16e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e381004a0 0x7f9e3819d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:31.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.946+0000 7f9e3f16e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3810c210 con 0x7f9e381004a0 2024-05-09T08:24:31.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.946+0000 7f9e3cf0a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e381004a0 0x7f9e3819d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:31.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.946+0000 7f9e3cf0a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e381004a0 0x7f9e3819d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56114/0 (socket says 172.21.15.107:56114) 2024-05-09T08:24:31.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.946+0000 7f9e3cf0a700 1 -- 172.21.15.107:0/193375680 learned_addr learned my addr 172.21.15.107:0/193375680 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:24:31.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.947+0000 7f9e3cf0a700 1 -- 172.21.15.107:0/193375680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e3819d930 con 0x7f9e381004a0 2024-05-09T08:24:31.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.947+0000 7f9e3cf0a700 1 --2- 172.21.15.107:0/193375680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e381004a0 0x7f9e3819d3f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0056d0 tx=0x7f9e2c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:31.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.947+0000 7f9e35ffb700 1 -- 172.21.15.107:0/193375680 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e2c026070 con 0x7f9e381004a0 2024-05-09T08:24:31.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.947+0000 7f9e3f16e700 1 -- 172.21.15.107:0/193375680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e3819dc20 con 0x7f9e381004a0 2024-05-09T08:24:31.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.947+0000 7f9e35ffb700 1 -- 172.21.15.107:0/193375680 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e2c00bba0 con 0x7f9e381004a0 2024-05-09T08:24:31.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.947+0000 7f9e35ffb700 1 -- 172.21.15.107:0/193375680 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e2c0037f0 con 0x7f9e381004a0 2024-05-09T08:24:31.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.947+0000 7f9e3f16e700 1 -- 172.21.15.107:0/193375680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e381975b0 con 0x7f9e381004a0 2024-05-09T08:24:31.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.948+0000 7f9e35ffb700 1 -- 172.21.15.107:0/193375680 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f9e2c029070 con 0x7f9e381004a0 2024-05-09T08:24:31.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.948+0000 7f9e3f16e700 1 -- 172.21.15.107:0/193375680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e38103560 con 0x7f9e381004a0 2024-05-09T08:24:31.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.948+0000 7f9e35ffb700 1 --2- 172.21.15.107:0/193375680 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9e280390f0 0x7f9e2803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:31.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.948+0000 7f9e35ffb700 1 -- 172.21.15.107:0/193375680 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9e2c01e070 con 0x7f9e381004a0 2024-05-09T08:24:31.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.948+0000 7f9e37fff700 1 --2- 172.21.15.107:0/193375680 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9e280390f0 0x7f9e2803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:31.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.949+0000 7f9e37fff700 1 --2- 172.21.15.107:0/193375680 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9e280390f0 0x7f9e2803b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e24006fd0 tx=0x7f9e24006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:31.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:31.953+0000 7f9e35ffb700 1 -- 172.21.15.107:0/193375680 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9e2c013070 con 0x7f9e381004a0 2024-05-09T08:24:32.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:32.154+0000 7f9e3f16e700 1 -- 172.21.15.107:0/193375680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f9e38198ae0 con 0x7f9e381004a0 2024-05-09T08:24:32.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:32.460+0000 7f9e35ffb700 1 -- 172.21.15.107:0/193375680 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f9e2c030090 con 0x7f9e381004a0 2024-05-09T08:24:32.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:32.462+0000 7f9e3f16e700 1 -- 172.21.15.107:0/193375680 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9e280390f0 msgr2=0x7f9e2803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:32.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:32.462+0000 7f9e3f16e700 1 --2- 172.21.15.107:0/193375680 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9e280390f0 0x7f9e2803b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e24006fd0 tx=0x7f9e24006e40 comp rx=0 tx=0).stop 2024-05-09T08:24:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:32.462+0000 7f9e3f16e700 1 -- 172.21.15.107:0/193375680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e381004a0 msgr2=0x7f9e3819d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:32.462+0000 7f9e3f16e700 1 --2- 172.21.15.107:0/193375680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e381004a0 0x7f9e3819d3f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0056d0 tx=0x7f9e2c0057b0 comp rx=0 tx=0).stop 2024-05-09T08:24:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:32.462+0000 7f9e3f16e700 1 -- 172.21.15.107:0/193375680 shutdown_connections 2024-05-09T08:24:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:32.462+0000 7f9e3f16e700 1 --2- 172.21.15.107:0/193375680 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9e280390f0 0x7f9e2803b5b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:32.462+0000 7f9e3f16e700 1 --2- 172.21.15.107:0/193375680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e381004a0 0x7f9e3819d3f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:32.462+0000 7f9e3f16e700 1 -- 172.21.15.107:0/193375680 >> 172.21.15.107:0/193375680 conn(0x7f9e380fbea0 msgr2=0x7f9e38103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:32.462+0000 7f9e3f16e700 1 -- 172.21.15.107:0/193375680 shutdown_connections 2024-05-09T08:24:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:32.462+0000 7f9e3f16e700 1 -- 172.21.15.107:0/193375680 wait complete. 2024-05-09T08:24:32.464 INFO:teuthology.orchestra.run.smithi107.stderr:adjusted tunables profile to default 2024-05-09T08:24:32.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:32 smithi107 ceph-mon[30122]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:32.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:32 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/193375680' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-09T08:24:33.165 INFO:tasks.cephadm:Adding mon.a on smithi107 2024-05-09T08:24:33.165 INFO:tasks.cephadm:Adding mon.b on smithi110 2024-05-09T08:24:33.165 INFO:tasks.cephadm:Adding mon.c on smithi157 2024-05-09T08:24:33.165 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch apply mon '3;smithi107:172.21.15.107=a;smithi110:172.21.15.110=b;smithi157:172.21.15.157=c' 2024-05-09T08:24:33.404 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:33.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:33 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/193375680' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-09T08:24:33.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:33 smithi107 ceph-mon[30122]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:24:33.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:33.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:33.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:33.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:33.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi157", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:24:33.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:33.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:24:34.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:34 smithi107 ceph-mon[30122]: Updating smithi157:/etc/ceph/ceph.conf 2024-05-09T08:24:34.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:34 smithi107 ceph-mon[30122]: Updating smithi157:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:34.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:34 smithi107 ceph-mon[30122]: Updating smithi157:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:24:34.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:34 smithi107 ceph-mon[30122]: Updating smithi157:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:24:34.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:34 smithi107 ceph-mon[30122]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:34.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:34 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:34.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:34 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:34.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:34 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:34.937 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.935+0000 7f552f38b700 1 -- 172.21.15.157:0/2773850327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5528100e90 msgr2=0x7f5528101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:34.937 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.935+0000 7f552f38b700 1 --2- 172.21.15.157:0/2773850327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5528100e90 0x7f5528101270 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5524009910 tx=0x7f5524009c20 comp rx=0 tx=0).stop 2024-05-09T08:24:34.937 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.935+0000 7f552f38b700 1 -- 172.21.15.157:0/2773850327 shutdown_connections 2024-05-09T08:24:34.937 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.935+0000 7f552f38b700 1 --2- 172.21.15.157:0/2773850327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5528100e90 0x7f5528101270 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:34.937 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.935+0000 7f552f38b700 1 -- 172.21.15.157:0/2773850327 >> 172.21.15.157:0/2773850327 conn(0x7f55280fc760 msgr2=0x7f55280feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:34.937 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.935+0000 7f552f38b700 1 -- 172.21.15.157:0/2773850327 shutdown_connections 2024-05-09T08:24:34.937 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.935+0000 7f552f38b700 1 -- 172.21.15.157:0/2773850327 wait complete. 2024-05-09T08:24:34.937 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.936+0000 7f552f38b700 1 Processor -- start 2024-05-09T08:24:34.937 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.936+0000 7f552f38b700 1 -- start start 2024-05-09T08:24:34.938 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.936+0000 7f552f38b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5528100e90 0x7f552819d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:34.938 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.936+0000 7f552f38b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f552810b110 con 0x7f5528100e90 2024-05-09T08:24:34.938 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.936+0000 7f552d127700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5528100e90 0x7f552819d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:34.938 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.936+0000 7f552d127700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5528100e90 0x7f552819d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.157:35232/0 (socket says 172.21.15.157:35232) 2024-05-09T08:24:34.938 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.936+0000 7f552d127700 1 -- 172.21.15.157:0/3264178036 learned_addr learned my addr 172.21.15.157:0/3264178036 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-05-09T08:24:34.938 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.936+0000 7f552d127700 1 -- 172.21.15.157:0/3264178036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552819d930 con 0x7f5528100e90 2024-05-09T08:24:34.938 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.936+0000 7f552d127700 1 --2- 172.21.15.157:0/3264178036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5528100e90 0x7f552819d3f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f55240056d0 tx=0x7f55240057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:34.938 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.937+0000 7f551e7fc700 1 -- 172.21.15.157:0/3264178036 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5524026070 con 0x7f5528100e90 2024-05-09T08:24:34.938 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.937+0000 7f552f38b700 1 -- 172.21.15.157:0/3264178036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f552819dc20 con 0x7f5528100e90 2024-05-09T08:24:34.938 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.937+0000 7f551e7fc700 1 -- 172.21.15.157:0/3264178036 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f552400bba0 con 0x7f5528100e90 2024-05-09T08:24:34.938 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.937+0000 7f551e7fc700 1 -- 172.21.15.157:0/3264178036 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55240037f0 con 0x7f5528100e90 2024-05-09T08:24:34.938 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.937+0000 7f552f38b700 1 -- 172.21.15.157:0/3264178036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55281975b0 con 0x7f5528100e90 2024-05-09T08:24:34.939 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.938+0000 7f552f38b700 1 -- 172.21.15.157:0/3264178036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5528102370 con 0x7f5528100e90 2024-05-09T08:24:34.939 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.938+0000 7f551e7fc700 1 -- 172.21.15.157:0/3264178036 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f5524029070 con 0x7f5528100e90 2024-05-09T08:24:34.940 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.938+0000 7f551e7fc700 1 --2- 172.21.15.157:0/3264178036 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f55140390f0 0x7f551403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:34.940 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.938+0000 7f551e7fc700 1 -- 172.21.15.157:0/3264178036 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f552401e070 con 0x7f5528100e90 2024-05-09T08:24:34.940 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.938+0000 7f552c926700 1 --2- 172.21.15.157:0/3264178036 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f55140390f0 0x7f551403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:34.940 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.939+0000 7f552c926700 1 --2- 172.21.15.157:0/3264178036 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f55140390f0 0x7f551403b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5518006fd0 tx=0x7f5518006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:34.943 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:34.943+0000 7f551e7fc700 1 -- 172.21.15.157:0/3264178036 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5524013070 con 0x7f5528100e90 2024-05-09T08:24:35.152 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:35.152+0000 7f552f38b700 1 -- 172.21.15.157:0/3264178036 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:172.21.15.107=a;smithi110:172.21.15.110=b;smithi157:172.21.15.157=c", "target": ["mon-mgr", ""]}) -- 0x7f5528198ae0 con 0x7f55140390f0 2024-05-09T08:24:35.159 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:35.159+0000 7f551e7fc700 1 -- 172.21.15.157:0/3264178036 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5528198ae0 con 0x7f55140390f0 2024-05-09T08:24:35.159 INFO:teuthology.orchestra.run.smithi157.stdout:Scheduled mon update... 2024-05-09T08:24:35.161 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:35.161+0000 7f552f38b700 1 -- 172.21.15.157:0/3264178036 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f55140390f0 msgr2=0x7f551403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:35.161 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:35.161+0000 7f552f38b700 1 --2- 172.21.15.157:0/3264178036 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f55140390f0 0x7f551403b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5518006fd0 tx=0x7f5518006e40 comp rx=0 tx=0).stop 2024-05-09T08:24:35.162 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:35.161+0000 7f552f38b700 1 -- 172.21.15.157:0/3264178036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5528100e90 msgr2=0x7f552819d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:35.162 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:35.161+0000 7f552f38b700 1 --2- 172.21.15.157:0/3264178036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5528100e90 0x7f552819d3f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f55240056d0 tx=0x7f55240057b0 comp rx=0 tx=0).stop 2024-05-09T08:24:35.162 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:35.161+0000 7f552f38b700 1 -- 172.21.15.157:0/3264178036 shutdown_connections 2024-05-09T08:24:35.162 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:35.161+0000 7f552f38b700 1 --2- 172.21.15.157:0/3264178036 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f55140390f0 0x7f551403b5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:35.162 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:35.161+0000 7f552f38b700 1 --2- 172.21.15.157:0/3264178036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5528100e90 0x7f552819d3f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:35.162 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:35.161+0000 7f552f38b700 1 -- 172.21.15.157:0/3264178036 >> 172.21.15.157:0/3264178036 conn(0x7f55280fc760 msgr2=0x7f552806a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:35.162 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:35.161+0000 7f552f38b700 1 -- 172.21.15.157:0/3264178036 shutdown_connections 2024-05-09T08:24:35.162 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:35.161+0000 7f552f38b700 1 -- 172.21.15.157:0/3264178036 wait complete. 2024-05-09T08:24:35.734 DEBUG:teuthology.orchestra.run.smithi110:mon.b> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.b.service 2024-05-09T08:24:35.737 DEBUG:teuthology.orchestra.run.smithi157:mon.c> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.c.service 2024-05-09T08:24:35.739 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-09T08:24:35.739 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-09T08:24:36.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:36 smithi107 ceph-mon[30122]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:172.21.15.107=a;smithi110:172.21.15.110=b;smithi157:172.21.15.157=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:36.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:36 smithi107 ceph-mon[30122]: Saving service mon spec with placement smithi107:172.21.15.107=a;smithi110:172.21.15.110=b;smithi157:172.21.15.157=c;count:3 2024-05-09T08:24:36.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:36 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:36.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:36 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:24:36.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:36 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:36.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:36 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:24:36.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:36 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:36.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:36 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:24:36.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:36 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:36.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:36 smithi107 ceph-mon[30122]: Deploying daemon mon.c on smithi157 2024-05-09T08:24:36.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:36 smithi107 ceph-mon[30122]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:37.389 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:38.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:38 smithi107 ceph-mon[30122]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:39.047 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:38 smithi157 systemd[1]: Starting Ceph mon.c for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:24:39.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 podman[33569]: 2024-05-09 08:24:39.342098327 +0000 UTC m=+0.201248187 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:24:39.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 podman[33569]: 2024-05-09 08:24:39.4417726 +0000 UTC m=+0.300922482 container create 7cd67865c9ab2215318b780a03ba7562b86221ae5bd875a6535cfe82ec670815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-c, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=) 2024-05-09T08:24:39.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.747+0000 7ff3a6b61700 1 -- 172.21.15.157:0/41518105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3a0100670 msgr2=0x7ff3a0100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:39.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.747+0000 7ff3a6b61700 1 --2- 172.21.15.157:0/41518105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3a0100670 0x7ff3a0100a50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff39400b0a0 tx=0x7ff39400b3b0 comp rx=0 tx=0).stop 2024-05-09T08:24:39.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.747+0000 7ff3a6b61700 1 -- 172.21.15.157:0/41518105 shutdown_connections 2024-05-09T08:24:39.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.747+0000 7ff3a6b61700 1 --2- 172.21.15.157:0/41518105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3a0100670 0x7ff3a0100a50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:39.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.747+0000 7ff3a6b61700 1 -- 172.21.15.157:0/41518105 >> 172.21.15.157:0/41518105 conn(0x7ff3a0078280 msgr2=0x7ff3a0078690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:39.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.747+0000 7ff3a6b61700 1 -- 172.21.15.157:0/41518105 shutdown_connections 2024-05-09T08:24:39.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.747+0000 7ff3a6b61700 1 -- 172.21.15.157:0/41518105 wait complete. 2024-05-09T08:24:39.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.748+0000 7ff3a6b61700 1 Processor -- start 2024-05-09T08:24:39.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.748+0000 7ff3a6b61700 1 -- start start 2024-05-09T08:24:39.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.748+0000 7ff3a6b61700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3a0100670 0x7ff3a019f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:39.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.748+0000 7ff3a6b61700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a010c220 con 0x7ff3a0100670 2024-05-09T08:24:39.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.748+0000 7ff3a48fd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3a0100670 0x7ff3a019f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:39.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.748+0000 7ff3a48fd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3a0100670 0x7ff3a019f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.157:35256/0 (socket says 172.21.15.157:35256) 2024-05-09T08:24:39.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.748+0000 7ff3a48fd700 1 -- 172.21.15.157:0/2850624798 learned_addr learned my addr 172.21.15.157:0/2850624798 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-05-09T08:24:39.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.749+0000 7ff3a48fd700 1 -- 172.21.15.157:0/2850624798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a019fbe0 con 0x7ff3a0100670 2024-05-09T08:24:39.751 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.749+0000 7ff3a48fd700 1 --2- 172.21.15.157:0/2850624798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3a0100670 0x7ff3a019f6a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff3940076a0 tx=0x7ff394003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:39.751 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.749+0000 7ff39dffb700 1 -- 172.21.15.157:0/2850624798 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff394017040 con 0x7ff3a0100670 2024-05-09T08:24:39.751 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.749+0000 7ff3a6b61700 1 -- 172.21.15.157:0/2850624798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3a019fe70 con 0x7ff3a0100670 2024-05-09T08:24:39.751 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.749+0000 7ff39dffb700 1 -- 172.21.15.157:0/2850624798 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff394022070 con 0x7ff3a0100670 2024-05-09T08:24:39.751 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.749+0000 7ff39dffb700 1 -- 172.21.15.157:0/2850624798 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff39401d5e0 con 0x7ff3a0100670 2024-05-09T08:24:39.751 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.749+0000 7ff3a6b61700 1 -- 172.21.15.157:0/2850624798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3a0199770 con 0x7ff3a0100670 2024-05-09T08:24:39.752 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.750+0000 7ff39dffb700 1 -- 172.21.15.157:0/2850624798 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7ff39401d8a0 con 0x7ff3a0100670 2024-05-09T08:24:39.752 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.750+0000 7ff39dffb700 1 --2- 172.21.15.157:0/2850624798 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff3900390f0 0x7ff39003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:39.752 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.750+0000 7ff3a6b61700 1 -- 172.21.15.157:0/2850624798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3a0069e30 con 0x7ff3a0100670 2024-05-09T08:24:39.752 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.750+0000 7ff39dffb700 1 -- 172.21.15.157:0/2850624798 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff39400f050 con 0x7ff3a0100670 2024-05-09T08:24:39.752 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.751+0000 7ff39ffff700 1 --2- 172.21.15.157:0/2850624798 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff3900390f0 0x7ff39003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:39.752 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.751+0000 7ff39ffff700 1 --2- 172.21.15.157:0/2850624798 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff3900390f0 0x7ff39003b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff38c006fd0 tx=0x7ff38c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:39.756 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.756+0000 7ff39dffb700 1 -- 172.21.15.157:0/2850624798 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff39401c3b0 con 0x7ff3a0100670 2024-05-09T08:24:39.873 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:39.872+0000 7ff39dffb700 1 -- 172.21.15.157:0/2850624798 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7ff394025e90 con 0x7ff3a0100670 2024-05-09T08:24:40.041 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:40.041+0000 7ff3a6b61700 1 -- 172.21.15.157:0/2850624798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ff3a0063980 con 0x7ff3a0100670 2024-05-09T08:24:40.042 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 podman[33569]: 2024-05-09 08:24:39.812133635 +0000 UTC m=+0.671283484 container init 7cd67865c9ab2215318b780a03ba7562b86221ae5bd875a6535cfe82ec670815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:24:40.042 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 podman[33569]: 2024-05-09 08:24:39.815931461 +0000 UTC m=+0.675081307 container start 7cd67865c9ab2215318b780a03ba7562b86221ae5bd875a6535cfe82ec670815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-09T08:24:40.042 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: set uid:gid to 167:167 (ceph:ceph) 2024-05-09T08:24:40.042 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev), process ceph-mon, pid 2 2024-05-09T08:24:40.042 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: pidfile_write: ignore empty --pid-file 2024-05-09T08:24:40.042 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: load: jerasure load: lrc 2024-05-09T08:24:40.042 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: RocksDB version: 7.9.2 2024-05-09T08:24:40.042 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Git sha 0 2024-05-09T08:24:40.042 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Compile date 2024-05-07 19:19:25 2024-05-09T08:24:40.043 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: DB SUMMARY 2024-05-09T08:24:40.043 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: DB Session ID: 87QVHPR7E5H6933GXCK1 2024-05-09T08:24:40.043 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: CURRENT file: CURRENT 2024-05-09T08:24:40.043 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: IDENTITY file: IDENTITY 2024-05-09T08:24:40.043 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-09T08:24:40.043 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-05-09T08:24:40.043 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-05-09T08:24:40.043 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.error_if_exists: 0 2024-05-09T08:24:40.043 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.create_if_missing: 0 2024-05-09T08:24:40.043 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.paranoid_checks: 1 2024-05-09T08:24:40.043 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-09T08:24:40.043 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-09T08:24:40.043 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-09T08:24:40.044 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.env: 0x561f24591c60 2024-05-09T08:24:40.044 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.fs: PosixFileSystem 2024-05-09T08:24:40.044 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.info_log: 0x561f2631bac0 2024-05-09T08:24:40.044 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_file_opening_threads: 16 2024-05-09T08:24:40.045 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.statistics: (nil) 2024-05-09T08:24:40.045 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.use_fsync: 0 2024-05-09T08:24:40.045 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_log_file_size: 0 2024-05-09T08:24:40.045 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-09T08:24:40.045 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-09T08:24:40.045 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.keep_log_file_num: 1000 2024-05-09T08:24:40.045 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.recycle_log_file_num: 0 2024-05-09T08:24:40.045 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.allow_fallocate: 1 2024-05-09T08:24:40.045 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.allow_mmap_reads: 0 2024-05-09T08:24:40.045 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.allow_mmap_writes: 0 2024-05-09T08:24:40.045 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.use_direct_reads: 0 2024-05-09T08:24:40.045 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-09T08:24:40.045 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.create_missing_column_families: 0 2024-05-09T08:24:40.046 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.db_log_dir: 2024-05-09T08:24:40.046 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.wal_dir: 2024-05-09T08:24:40.046 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-09T08:24:40.046 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-09T08:24:40.046 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-09T08:24:40.046 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-09T08:24:40.046 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-09T08:24:40.046 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-09T08:24:40.046 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.advise_random_on_open: 1 2024-05-09T08:24:40.046 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.db_write_buffer_size: 0 2024-05-09T08:24:40.046 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.write_buffer_manager: 0x561f262ee1e0 2024-05-09T08:24:40.046 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-09T08:24:40.046 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-09T08:24:40.047 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-09T08:24:40.047 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.rate_limiter: (nil) 2024-05-09T08:24:40.047 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-09T08:24:40.047 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.wal_recovery_mode: 2 2024-05-09T08:24:40.047 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.enable_thread_tracking: 0 2024-05-09T08:24:40.047 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.enable_pipelined_write: 0 2024-05-09T08:24:40.047 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.unordered_write: 0 2024-05-09T08:24:40.047 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-09T08:24:40.047 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-09T08:24:40.047 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-09T08:24:40.047 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-09T08:24:40.047 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.row_cache: None 2024-05-09T08:24:40.047 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.wal_filter: None 2024-05-09T08:24:40.047 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-09T08:24:40.048 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.allow_ingest_behind: 0 2024-05-09T08:24:40.048 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.two_write_queues: 0 2024-05-09T08:24:40.048 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.manual_wal_flush: 0 2024-05-09T08:24:40.048 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.wal_compression: 0 2024-05-09T08:24:40.048 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.atomic_flush: 0 2024-05-09T08:24:40.048 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-09T08:24:40.048 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-09T08:24:40.048 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-09T08:24:40.048 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.log_readahead_size: 0 2024-05-09T08:24:40.048 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-09T08:24:40.048 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.best_efforts_recovery: 0 2024-05-09T08:24:40.048 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-09T08:24:40.048 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-09T08:24:40.048 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.allow_data_in_errors: 0 2024-05-09T08:24:40.049 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.db_host_id: __hostname__ 2024-05-09T08:24:40.049 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-09T08:24:40.049 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_background_jobs: 2 2024-05-09T08:24:40.049 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_background_compactions: -1 2024-05-09T08:24:40.049 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_subcompactions: 1 2024-05-09T08:24:40.049 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-09T08:24:40.049 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-09T08:24:40.049 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-09T08:24:40.049 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_total_wal_size: 0 2024-05-09T08:24:40.049 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-09T08:24:40.049 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-09T08:24:40.049 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-09T08:24:40.049 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-09T08:24:40.050 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_open_files: -1 2024-05-09T08:24:40.050 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.bytes_per_sync: 0 2024-05-09T08:24:40.050 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-09T08:24:40.050 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-09T08:24:40.050 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compaction_readahead_size: 0 2024-05-09T08:24:40.050 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_background_flushes: -1 2024-05-09T08:24:40.050 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Compression algorithms supported: 2024-05-09T08:24:40.050 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-09T08:24:40.050 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: kZSTD supported: 0 2024-05-09T08:24:40.050 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: kXpressCompression supported: 0 2024-05-09T08:24:40.050 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: kLZ4HCCompression supported: 1 2024-05-09T08:24:40.050 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: kZlibCompression supported: 1 2024-05-09T08:24:40.050 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: kSnappyCompression supported: 1 2024-05-09T08:24:40.050 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: kLZ4Compression supported: 1 2024-05-09T08:24:40.051 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: kBZip2Compression supported: 0 2024-05-09T08:24:40.051 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-09T08:24:40.051 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-09T08:24:40.051 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-05-09T08:24:40.051 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-09T08:24:40.051 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-09T08:24:40.051 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.merge_operator: 2024-05-09T08:24:40.051 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compaction_filter: None 2024-05-09T08:24:40.051 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compaction_filter_factory: None 2024-05-09T08:24:40.051 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.sst_partitioner_factory: None 2024-05-09T08:24:40.051 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-09T08:24:40.051 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-09T08:24:40.051 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561f2631bbc0) 2024-05-09T08:24:40.051 INFO:journalctl@ceph.mon.c.smithi157.stdout: cache_index_and_filter_blocks: 1 2024-05-09T08:24:40.052 INFO:journalctl@ceph.mon.c.smithi157.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-09T08:24:40.052 INFO:journalctl@ceph.mon.c.smithi157.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-09T08:24:40.052 INFO:journalctl@ceph.mon.c.smithi157.stdout: pin_top_level_index_and_filter: 1 2024-05-09T08:24:40.052 INFO:journalctl@ceph.mon.c.smithi157.stdout: index_type: 0 2024-05-09T08:24:40.052 INFO:journalctl@ceph.mon.c.smithi157.stdout: data_block_index_type: 0 2024-05-09T08:24:40.052 INFO:journalctl@ceph.mon.c.smithi157.stdout: index_shortening: 1 2024-05-09T08:24:40.052 INFO:journalctl@ceph.mon.c.smithi157.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-09T08:24:40.052 INFO:journalctl@ceph.mon.c.smithi157.stdout: checksum: 4 2024-05-09T08:24:40.052 INFO:journalctl@ceph.mon.c.smithi157.stdout: no_block_cache: 0 2024-05-09T08:24:40.052 INFO:journalctl@ceph.mon.c.smithi157.stdout: block_cache: 0x561f2631f090 2024-05-09T08:24:40.052 INFO:journalctl@ceph.mon.c.smithi157.stdout: block_cache_name: BinnedLRUCache 2024-05-09T08:24:40.052 INFO:journalctl@ceph.mon.c.smithi157.stdout: block_cache_options: 2024-05-09T08:24:40.052 INFO:journalctl@ceph.mon.c.smithi157.stdout: capacity : 536870912 2024-05-09T08:24:40.052 INFO:journalctl@ceph.mon.c.smithi157.stdout: num_shard_bits : 4 2024-05-09T08:24:40.053 INFO:journalctl@ceph.mon.c.smithi157.stdout: strict_capacity_limit : 0 2024-05-09T08:24:40.053 INFO:journalctl@ceph.mon.c.smithi157.stdout: high_pri_pool_ratio: 0.000 2024-05-09T08:24:40.053 INFO:journalctl@ceph.mon.c.smithi157.stdout: block_cache_compressed: (nil) 2024-05-09T08:24:40.053 INFO:journalctl@ceph.mon.c.smithi157.stdout: persistent_cache: (nil) 2024-05-09T08:24:40.053 INFO:journalctl@ceph.mon.c.smithi157.stdout: block_size: 4096 2024-05-09T08:24:40.053 INFO:journalctl@ceph.mon.c.smithi157.stdout: block_size_deviation: 10 2024-05-09T08:24:40.053 INFO:journalctl@ceph.mon.c.smithi157.stdout: block_restart_interval: 16 2024-05-09T08:24:40.053 INFO:journalctl@ceph.mon.c.smithi157.stdout: index_block_restart_interval: 1 2024-05-09T08:24:40.053 INFO:journalctl@ceph.mon.c.smithi157.stdout: metadata_block_size: 4096 2024-05-09T08:24:40.053 INFO:journalctl@ceph.mon.c.smithi157.stdout: partition_filters: 0 2024-05-09T08:24:40.053 INFO:journalctl@ceph.mon.c.smithi157.stdout: use_delta_encoding: 1 2024-05-09T08:24:40.053 INFO:journalctl@ceph.mon.c.smithi157.stdout: filter_policy: bloomfilter 2024-05-09T08:24:40.053 INFO:journalctl@ceph.mon.c.smithi157.stdout: whole_key_filtering: 1 2024-05-09T08:24:40.053 INFO:journalctl@ceph.mon.c.smithi157.stdout: verify_compression: 0 2024-05-09T08:24:40.054 INFO:journalctl@ceph.mon.c.smithi157.stdout: read_amp_bytes_per_bit: 0 2024-05-09T08:24:40.054 INFO:journalctl@ceph.mon.c.smithi157.stdout: format_version: 5 2024-05-09T08:24:40.054 INFO:journalctl@ceph.mon.c.smithi157.stdout: enable_index_compression: 1 2024-05-09T08:24:40.054 INFO:journalctl@ceph.mon.c.smithi157.stdout: block_align: 0 2024-05-09T08:24:40.054 INFO:journalctl@ceph.mon.c.smithi157.stdout: max_auto_readahead_size: 262144 2024-05-09T08:24:40.054 INFO:journalctl@ceph.mon.c.smithi157.stdout: prepopulate_block_cache: 0 2024-05-09T08:24:40.054 INFO:journalctl@ceph.mon.c.smithi157.stdout: initial_auto_readahead_size: 8192 2024-05-09T08:24:40.054 INFO:journalctl@ceph.mon.c.smithi157.stdout: num_file_reads_for_auto_readahead: 2 2024-05-09T08:24:40.054 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.write_buffer_size: 33554432 2024-05-09T08:24:40.054 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_write_buffer_number: 2 2024-05-09T08:24:40.054 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compression: NoCompression 2024-05-09T08:24:40.054 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.bottommost_compression: Disabled 2024-05-09T08:24:40.054 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.prefix_extractor: nullptr 2024-05-09T08:24:40.055 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-09T08:24:40.055 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.num_levels: 7 2024-05-09T08:24:40.055 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-09T08:24:40.055 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-09T08:24:40.055 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-09T08:24:40.055 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-09T08:24:40.055 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-09T08:24:40.055 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-09T08:24:40.055 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-09T08:24:40.055 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-09T08:24:40.055 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-09T08:24:40.055 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-09T08:24:40.056 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-09T08:24:40.056 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-09T08:24:40.056 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-09T08:24:40.056 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compression_opts.level: 32767 2024-05-09T08:24:40.056 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compression_opts.strategy: 0 2024-05-09T08:24:40.056 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-09T08:24:40.056 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-09T08:24:40.056 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-09T08:24:40.056 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-09T08:24:40.056 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compression_opts.enabled: false 2024-05-09T08:24:40.056 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-09T08:24:40.056 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-09T08:24:40.056 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-09T08:24:40.056 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-09T08:24:40.056 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.target_file_size_base: 67108864 2024-05-09T08:24:40.057 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-09T08:24:40.057 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-09T08:24:40.057 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-09T08:24:40.057 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-09T08:24:40.057 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-09T08:24:40.057 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-09T08:24:40.058 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-09T08:24:40.058 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-09T08:24:40.058 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-09T08:24:40.058 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-09T08:24:40.058 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-09T08:24:40.058 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-09T08:24:40.058 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-09T08:24:40.058 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-09T08:24:40.059 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.arena_block_size: 1048576 2024-05-09T08:24:40.059 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-09T08:24:40.059 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-09T08:24:40.059 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.disable_auto_compactions: 0 2024-05-09T08:24:40.059 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-09T08:24:40.059 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-09T08:24:40.059 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-09T08:24:40.059 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-09T08:24:40.059 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-09T08:24:40.059 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-09T08:24:40.059 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-09T08:24:40.059 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-09T08:24:40.059 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-09T08:24:40.059 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-09T08:24:40.060 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.table_properties_collectors: 2024-05-09T08:24:40.060 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.inplace_update_support: 0 2024-05-09T08:24:40.060 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-09T08:24:40.060 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-09T08:24:40.060 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-09T08:24:40.060 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-09T08:24:40.060 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.bloom_locality: 0 2024-05-09T08:24:40.060 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.max_successive_merges: 0 2024-05-09T08:24:40.060 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-09T08:24:40.060 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.paranoid_file_checks: 0 2024-05-09T08:24:40.060 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.force_consistency_checks: 1 2024-05-09T08:24:40.060 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.report_bg_io_stats: 0 2024-05-09T08:24:40.060 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.ttl: 2592000 2024-05-09T08:24:40.060 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-09T08:24:40.061 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-09T08:24:40.061 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-09T08:24:40.061 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.enable_blob_files: false 2024-05-09T08:24:40.061 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.min_blob_size: 0 2024-05-09T08:24:40.061 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.blob_file_size: 268435456 2024-05-09T08:24:40.061 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-09T08:24:40.061 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-09T08:24:40.061 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-09T08:24:40.061 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-09T08:24:40.061 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-09T08:24:40.061 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.blob_file_starting_level: 0 2024-05-09T08:24:40.061 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-09T08:24:40.061 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-09T08:24:40.061 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-09T08:24:40.062 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6930d50b-b7c4-4b3d-856d-c94ec8f9af11 2024-05-09T08:24:40.062 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715243079846797, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-09T08:24:40.062 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-09T08:24:40.062 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715243079847325, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715243079, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6930d50b-b7c4-4b3d-856d-c94ec8f9af11", "db_session_id": "87QVHPR7E5H6933GXCK1", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-09T08:24:40.062 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715243079847422, "job": 1, "event": "recovery_finished"} 2024-05-09T08:24:40.062 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-09T08:24:40.062 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-09T08:24:40.062 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561f26410000 2024-05-09T08:24:40.062 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: DB pointer 0x561f263fa000 2024-05-09T08:24:40.062 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-09T08:24:40.062 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-09T08:24:40.062 INFO:journalctl@ceph.mon.c.smithi157.stdout: ** DB Stats ** 2024-05-09T08:24:40.062 INFO:journalctl@ceph.mon.c.smithi157.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-09T08:24:40.063 INFO:journalctl@ceph.mon.c.smithi157.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-09T08:24:40.063 INFO:journalctl@ceph.mon.c.smithi157.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-09T08:24:40.063 INFO:journalctl@ceph.mon.c.smithi157.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-09T08:24:40.063 INFO:journalctl@ceph.mon.c.smithi157.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-09T08:24:40.063 INFO:journalctl@ceph.mon.c.smithi157.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-09T08:24:40.063 INFO:journalctl@ceph.mon.c.smithi157.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-09T08:24:40.063 INFO:journalctl@ceph.mon.c.smithi157.stdout: 2024-05-09T08:24:40.063 INFO:journalctl@ceph.mon.c.smithi157.stdout: ** Compaction Stats [default] ** 2024-05-09T08:24:40.063 INFO:journalctl@ceph.mon.c.smithi157.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-09T08:24:40.063 INFO:journalctl@ceph.mon.c.smithi157.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-09T08:24:40.063 INFO:journalctl@ceph.mon.c.smithi157.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-09T08:24:40.063 INFO:journalctl@ceph.mon.c.smithi157.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-09T08:24:40.063 INFO:journalctl@ceph.mon.c.smithi157.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-09T08:24:40.063 INFO:journalctl@ceph.mon.c.smithi157.stdout: 2024-05-09T08:24:40.063 INFO:journalctl@ceph.mon.c.smithi157.stdout: ** Compaction Stats [default] ** 2024-05-09T08:24:40.064 INFO:journalctl@ceph.mon.c.smithi157.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-09T08:24:40.064 INFO:journalctl@ceph.mon.c.smithi157.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-09T08:24:40.064 INFO:journalctl@ceph.mon.c.smithi157.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-09T08:24:40.064 INFO:journalctl@ceph.mon.c.smithi157.stdout: 2024-05-09T08:24:40.064 INFO:journalctl@ceph.mon.c.smithi157.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-09T08:24:40.064 INFO:journalctl@ceph.mon.c.smithi157.stdout: 2024-05-09T08:24:40.064 INFO:journalctl@ceph.mon.c.smithi157.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-09T08:24:40.064 INFO:journalctl@ceph.mon.c.smithi157.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-09T08:24:40.064 INFO:journalctl@ceph.mon.c.smithi157.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-09T08:24:40.064 INFO:journalctl@ceph.mon.c.smithi157.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-09T08:24:40.064 INFO:journalctl@ceph.mon.c.smithi157.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-09T08:24:40.064 INFO:journalctl@ceph.mon.c.smithi157.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-09T08:24:40.064 INFO:journalctl@ceph.mon.c.smithi157.stdout: Cumulative compaction: 0.00 GB write, 0.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-09T08:24:40.064 INFO:journalctl@ceph.mon.c.smithi157.stdout: Interval compaction: 0.00 GB write, 0.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-09T08:24:40.065 INFO:journalctl@ceph.mon.c.smithi157.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-09T08:24:40.065 INFO:journalctl@ceph.mon.c.smithi157.stdout: Block cache BinnedLRUCache@0x561f2631f090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-09T08:24:40.065 INFO:journalctl@ceph.mon.c.smithi157.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-09T08:24:40.065 INFO:journalctl@ceph.mon.c.smithi157.stdout: 2024-05-09T08:24:40.065 INFO:journalctl@ceph.mon.c.smithi157.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-09T08:24:40.065 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-05-09T08:24:40.065 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: using public_addr v2:172.21.15.157:0/0 -> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2024-05-09T08:24:40.065 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: starting mon.c rank -1 at public addrs [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] at bind addrs [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:40.065 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: mon.c@-1(???) e0 preinit fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:40.065 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: mon.c@-1(synchronizing).mds e1 new map 2024-05-09T08:24:40.065 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-09T08:24:40.065 INFO:journalctl@ceph.mon.c.smithi157.stdout: e1 2024-05-09T08:24:40.065 INFO:journalctl@ceph.mon.c.smithi157.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-09T08:24:40.065 INFO:journalctl@ceph.mon.c.smithi157.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-09T08:24:40.066 INFO:journalctl@ceph.mon.c.smithi157.stdout: legacy client fscid: -1 2024-05-09T08:24:40.066 INFO:journalctl@ceph.mon.c.smithi157.stdout: 2024-05-09T08:24:40.066 INFO:journalctl@ceph.mon.c.smithi157.stdout: No filesystems configured 2024-05-09T08:24:40.066 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-09T08:24:40.066 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-09T08:24:40.066 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-09T08:24:40.066 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-09T08:24:40.066 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-09T08:24:40.066 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-09T08:24:40.066 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-09T08:24:40.066 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:24:40.066 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:24:40.066 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:24:40.066 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:40.067 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi157", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:40.067 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.067 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.067 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.067 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.067 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:24:40.067 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:40.067 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:24:40.067 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: Updating smithi110:/etc/ceph/ceph.conf 2024-05-09T08:24:40.067 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: Updating smithi110:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:40.067 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: Deploying cephadm binary to smithi157 2024-05-09T08:24:40.067 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:40.067 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:24:40.067 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: Updating smithi110:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:24:40.068 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:40.068 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.068 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.068 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.068 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.068 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:24:40.068 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: Added host smithi157 2024-05-09T08:24:40.068 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:40.068 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.068 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.068 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:40.068 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.068 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:24:40.068 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:40.068 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/193375680' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-09T08:24:40.069 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/193375680' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-09T08:24:40.069 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:24:40.069 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.069 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.069 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.069 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.069 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi157", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:24:40.069 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:40.069 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:24:40.069 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: Updating smithi157:/etc/ceph/ceph.conf 2024-05-09T08:24:40.069 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: Updating smithi157:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:40.069 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: Updating smithi157:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:24:40.069 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: Updating smithi157:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:24:40.069 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:40.069 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.070 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.070 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.070 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:172.21.15.107=a;smithi110:172.21.15.110=b;smithi157:172.21.15.157=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:40.070 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: Saving service mon spec with placement smithi107:172.21.15.107=a;smithi110:172.21.15.110=b;smithi157:172.21.15.157=c;count:3 2024-05-09T08:24:40.070 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.070 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:24:40.070 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:40.070 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:24:40.070 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:40.070 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:24:40.070 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:40.070 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: Deploying daemon mon.c on smithi157 2024-05-09T08:24:40.070 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:40.070 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:40.070 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 ceph-mon[33614]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-09T08:24:40.071 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 bash[33569]: 7cd67865c9ab2215318b780a03ba7562b86221ae5bd875a6535cfe82ec670815 2024-05-09T08:24:40.071 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:39 smithi157 systemd[1]: Started Ceph mon.c for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:24:44.890 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:44.888+0000 7ff39dffb700 1 -- 172.21.15.157:0/2850624798 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7ff394025c00 con 0x7ff3a0100670 2024-05-09T08:24:44.890 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:24:44.890 INFO:teuthology.orchestra.run.smithi157.stdout:{"epoch":2,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","modified":"2024-05-09T08:24:39.870153Z","created":"2024-05-09T08:22:54.137531Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:3300","nonce":0},{"type":"v1","addr":"172.21.15.157:6789","nonce":0}]},"addr":"172.21.15.157:6789/0","public_addr":"172.21.15.157:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-09T08:24:44.893 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:44.891+0000 7ff3a6b61700 1 -- 172.21.15.157:0/2850624798 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff3900390f0 msgr2=0x7ff39003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:44.893 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:44.891+0000 7ff3a6b61700 1 --2- 172.21.15.157:0/2850624798 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff3900390f0 0x7ff39003b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff38c006fd0 tx=0x7ff38c006e40 comp rx=0 tx=0).stop 2024-05-09T08:24:44.893 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:44.891+0000 7ff3a6b61700 1 -- 172.21.15.157:0/2850624798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3a0100670 msgr2=0x7ff3a019f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:44.893 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:44.891+0000 7ff3a6b61700 1 --2- 172.21.15.157:0/2850624798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3a0100670 0x7ff3a019f6a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff3940076a0 tx=0x7ff394003ce0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.893 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:44.891+0000 7ff3a6b61700 1 -- 172.21.15.157:0/2850624798 shutdown_connections 2024-05-09T08:24:44.893 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:44.891+0000 7ff3a6b61700 1 --2- 172.21.15.157:0/2850624798 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff3900390f0 0x7ff39003b5b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.893 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:44.891+0000 7ff3a6b61700 1 --2- 172.21.15.157:0/2850624798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3a0100670 0x7ff3a019f6a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:44.893 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:44.891+0000 7ff3a6b61700 1 -- 172.21.15.157:0/2850624798 >> 172.21.15.157:0/2850624798 conn(0x7ff3a0078280 msgr2=0x7ff3a00fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:44.893 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:44.892+0000 7ff3a6b61700 1 -- 172.21.15.157:0/2850624798 shutdown_connections 2024-05-09T08:24:44.893 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:44.892+0000 7ff3a6b61700 1 -- 172.21.15.157:0/2850624798 wait complete. 2024-05-09T08:24:44.894 INFO:teuthology.orchestra.run.smithi157.stderr:dumped monmap epoch 2 2024-05-09T08:24:45.215 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:24:45.215 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: mon.a calling monitor election 2024-05-09T08:24:45.215 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:45.216 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: from='client.? 172.21.15.157:0/2850624798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T08:24:45.216 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:45.216 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:45.216 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:45.216 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: mon.c calling monitor election 2024-05-09T08:24:45.216 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:45.216 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:45.216 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:45.216 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:45.216 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-09T08:24:45.217 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: monmap epoch 2 2024-05-09T08:24:45.217 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:45.217 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: last_changed 2024-05-09T08:24:39.870153+0000 2024-05-09T08:24:45.217 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: created 2024-05-09T08:22:54.137531+0000 2024-05-09T08:24:45.217 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: min_mon_release 19 (squid) 2024-05-09T08:24:45.217 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: election_strategy: 1 2024-05-09T08:24:45.217 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-05-09T08:24:45.217 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: 1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.c 2024-05-09T08:24:45.217 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: fsmap 2024-05-09T08:24:45.217 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:24:45.217 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: mgrmap e13: x(active, since 49s) 2024-05-09T08:24:45.218 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: overall HEALTH_OK 2024-05-09T08:24:45.218 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:45.218 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:45.218 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:45.218 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:24:45.218 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:45.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:24:45.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: mon.a calling monitor election 2024-05-09T08:24:45.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:45.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: from='client.? 172.21.15.157:0/2850624798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T08:24:45.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:45.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:45.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:45.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: mon.c calling monitor election 2024-05-09T08:24:45.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:45.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:45.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:45.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:45.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-09T08:24:45.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: monmap epoch 2 2024-05-09T08:24:45.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:45.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: last_changed 2024-05-09T08:24:39.870153+0000 2024-05-09T08:24:45.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: created 2024-05-09T08:22:54.137531+0000 2024-05-09T08:24:45.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: min_mon_release 19 (squid) 2024-05-09T08:24:45.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: election_strategy: 1 2024-05-09T08:24:45.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-05-09T08:24:45.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: 1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.c 2024-05-09T08:24:45.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: fsmap 2024-05-09T08:24:45.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:24:45.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: mgrmap e13: x(active, since 49s) 2024-05-09T08:24:45.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: overall HEALTH_OK 2024-05-09T08:24:45.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:45.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:45.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:45.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:24:45.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:46.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:45 smithi107 ceph-mon[30122]: Deploying daemon mon.b on smithi110 2024-05-09T08:24:46.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:45 smithi107 ceph-mon[30122]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:46.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:45 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:46.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:45 smithi157 ceph-mon[33614]: Deploying daemon mon.b on smithi110 2024-05-09T08:24:46.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:45 smithi157 ceph-mon[33614]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:46.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:45 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:46.507 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-09T08:24:46.507 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-09T08:24:46.748 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:24:48.165 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.164+0000 7fad09017700 1 -- 172.21.15.157:0/2138073015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad04106600 msgr2=0x7fad041069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:48.165 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.164+0000 7fad09017700 1 --2- 172.21.15.157:0/2138073015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad04106600 0x7fad041069e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7facf4009910 tx=0x7facf4009c20 comp rx=0 tx=0).stop 2024-05-09T08:24:48.165 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.164+0000 7fad09017700 1 -- 172.21.15.157:0/2138073015 shutdown_connections 2024-05-09T08:24:48.166 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.164+0000 7fad09017700 1 --2- 172.21.15.157:0/2138073015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad04106600 0x7fad041069e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:48.166 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.164+0000 7fad09017700 1 -- 172.21.15.157:0/2138073015 >> 172.21.15.157:0/2138073015 conn(0x7fad04076080 msgr2=0x7fad04076490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:48.166 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.164+0000 7fad09017700 1 -- 172.21.15.157:0/2138073015 shutdown_connections 2024-05-09T08:24:48.166 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.164+0000 7fad09017700 1 -- 172.21.15.157:0/2138073015 wait complete. 2024-05-09T08:24:48.169 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.165+0000 7fad09017700 1 Processor -- start 2024-05-09T08:24:48.169 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.165+0000 7fad09017700 1 -- start start 2024-05-09T08:24:48.169 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.165+0000 7fad09017700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad04106600 0x7fad04102140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:48.170 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.165+0000 7fad09017700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad04102680 0x7fad04102b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:48.170 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.165+0000 7fad09017700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fad0410aa50 con 0x7fad04102680 2024-05-09T08:24:48.170 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.165+0000 7fad09017700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fad0410a8d0 con 0x7fad04106600 2024-05-09T08:24:48.170 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.165+0000 7fad0259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad04102680 0x7fad04102b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:48.170 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.165+0000 7fad0259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad04102680 0x7fad04102b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.157:58508/0 (socket says 172.21.15.157:58508) 2024-05-09T08:24:48.170 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.165+0000 7fad0259c700 1 -- 172.21.15.157:0/275556774 learned_addr learned my addr 172.21.15.157:0/275556774 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-05-09T08:24:48.170 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.165+0000 7fad02d9d700 1 --2- 172.21.15.157:0/275556774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad04106600 0x7fad04102140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:48.170 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.165+0000 7fad02d9d700 1 -- 172.21.15.157:0/275556774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad04106600 msgr2=0x7fad04102140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-05-09T08:24:48.170 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.165+0000 7fad02d9d700 1 -- 172.21.15.157:0/275556774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad04106600 msgr2=0x7fad04102140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T08:24:48.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.165+0000 7fad02d9d700 1 --2- 172.21.15.157:0/275556774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad04106600 0x7fad04102140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T08:24:48.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.165+0000 7fad02d9d700 1 --2- 172.21.15.157:0/275556774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad04106600 0x7fad04102140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:24:48.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.166+0000 7fad0259c700 1 -- 172.21.15.157:0/275556774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad04106600 msgr2=0x7fad04102140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:24:48.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.166+0000 7fad0259c700 1 --2- 172.21.15.157:0/275556774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad04106600 0x7fad04102140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:48.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.166+0000 7fad0259c700 1 -- 172.21.15.157:0/275556774 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad04104010 con 0x7fad04102680 2024-05-09T08:24:48.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.166+0000 7fad0259c700 1 --2- 172.21.15.157:0/275556774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad04102680 0x7fad04102b00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7facf800b700 tx=0x7facf800bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:48.172 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.166+0000 7facebfff700 1 -- 172.21.15.157:0/275556774 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7facf8011630 con 0x7fad04102680 2024-05-09T08:24:48.172 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.166+0000 7facebfff700 1 -- 172.21.15.157:0/275556774 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7facf8011c80 con 0x7fad04102680 2024-05-09T08:24:48.172 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.166+0000 7fad09017700 1 -- 172.21.15.157:0/275556774 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad041a3cb0 con 0x7fad04102680 2024-05-09T08:24:48.172 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.166+0000 7facebfff700 1 -- 172.21.15.157:0/275556774 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7facf8010da0 con 0x7fad04102680 2024-05-09T08:24:48.172 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.166+0000 7fad09017700 1 -- 172.21.15.157:0/275556774 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad041a4210 con 0x7fad04102680 2024-05-09T08:24:48.172 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.167+0000 7facebfff700 1 -- 172.21.15.157:0/275556774 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7facf8018020 con 0x7fad04102680 2024-05-09T08:24:48.172 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.167+0000 7fad09017700 1 -- 172.21.15.157:0/275556774 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad04101bf0 con 0x7fad04102680 2024-05-09T08:24:48.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.167+0000 7facebfff700 1 --2- 172.21.15.157:0/275556774 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7facec038e90 0x7facec03b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:48.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.167+0000 7facebfff700 1 -- 172.21.15.157:0/275556774 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7facf804be40 con 0x7fad04102680 2024-05-09T08:24:48.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.167+0000 7fad02d9d700 1 --2- 172.21.15.157:0/275556774 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7facec038e90 0x7facec03b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:48.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.168+0000 7fad02d9d700 1 --2- 172.21.15.157:0/275556774 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7facec038e90 0x7facec03b350 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7facf4006010 tx=0x7facf401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:48.174 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.172+0000 7facebfff700 1 -- 172.21.15.157:0/275556774 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7facf801f020 con 0x7fad04102680 2024-05-09T08:24:48.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.448+0000 7fad09017700 1 -- 172.21.15.157:0/275556774 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fad0402db80 con 0x7fad04102680 2024-05-09T08:24:48.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.449+0000 7facebfff700 1 -- 172.21.15.157:0/275556774 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7facf8018310 con 0x7fad04102680 2024-05-09T08:24:48.450 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:24:48.450 INFO:teuthology.orchestra.run.smithi157.stdout:{"epoch":2,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","modified":"2024-05-09T08:24:39.870153Z","created":"2024-05-09T08:22:54.137531Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:3300","nonce":0},{"type":"v1","addr":"172.21.15.157:6789","nonce":0}]},"addr":"172.21.15.157:6789/0","public_addr":"172.21.15.157:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-09T08:24:48.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.451+0000 7fad09017700 1 -- 172.21.15.157:0/275556774 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7facec038e90 msgr2=0x7facec03b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:48.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.451+0000 7fad09017700 1 --2- 172.21.15.157:0/275556774 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7facec038e90 0x7facec03b350 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7facf4006010 tx=0x7facf401a040 comp rx=0 tx=0).stop 2024-05-09T08:24:48.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.451+0000 7fad09017700 1 -- 172.21.15.157:0/275556774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad04102680 msgr2=0x7fad04102b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:48.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.451+0000 7fad09017700 1 --2- 172.21.15.157:0/275556774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad04102680 0x7fad04102b00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7facf800b700 tx=0x7facf800bac0 comp rx=0 tx=0).stop 2024-05-09T08:24:48.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.451+0000 7fad09017700 1 -- 172.21.15.157:0/275556774 shutdown_connections 2024-05-09T08:24:48.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.451+0000 7fad09017700 1 --2- 172.21.15.157:0/275556774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad04106600 0x7fad04102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:48.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.451+0000 7fad09017700 1 --2- 172.21.15.157:0/275556774 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7facec038e90 0x7facec03b350 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:48.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.451+0000 7fad09017700 1 --2- 172.21.15.157:0/275556774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad04102680 0x7fad04102b00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:48.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.451+0000 7fad09017700 1 -- 172.21.15.157:0/275556774 >> 172.21.15.157:0/275556774 conn(0x7fad04076080 msgr2=0x7fad04105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:48.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.451+0000 7fad09017700 1 -- 172.21.15.157:0/275556774 shutdown_connections 2024-05-09T08:24:48.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:48.451+0000 7fad09017700 1 -- 172.21.15.157:0/275556774 wait complete. 2024-05-09T08:24:48.453 INFO:teuthology.orchestra.run.smithi157.stderr:dumped monmap epoch 2 2024-05-09T08:24:48.534 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 podman[33602]: 2024-05-09 08:24:48.420407098 +0000 UTC m=+0.018184155 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:24:48.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:48 smithi107 ceph-mon[30122]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:48.786 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 podman[33602]: 2024-05-09 08:24:48.731945883 +0000 UTC m=+0.329722931 container init ca5207af62f5837d92f0d60f651018c8ca7d3ab294c81771dfc844555770d6d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507) 2024-05-09T08:24:48.786 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 podman[33602]: 2024-05-09 08:24:48.735022875 +0000 UTC m=+0.332799931 container start ca5207af62f5837d92f0d60f651018c8ca7d3ab294c81771dfc844555770d6d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507) 2024-05-09T08:24:48.786 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: set uid:gid to 167:167 (ceph:ceph) 2024-05-09T08:24:48.786 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev), process ceph-mon, pid 2 2024-05-09T08:24:48.786 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: pidfile_write: ignore empty --pid-file 2024-05-09T08:24:48.786 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: load: jerasure load: lrc 2024-05-09T08:24:48.786 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: RocksDB version: 7.9.2 2024-05-09T08:24:48.786 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Git sha 0 2024-05-09T08:24:48.787 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Compile date 2024-05-07 19:19:25 2024-05-09T08:24:48.787 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: DB SUMMARY 2024-05-09T08:24:48.787 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: DB Session ID: JPTX1JVBNTTBSMAYGJWH 2024-05-09T08:24:48.787 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: CURRENT file: CURRENT 2024-05-09T08:24:48.787 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: IDENTITY file: IDENTITY 2024-05-09T08:24:48.787 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-09T08:24:48.787 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-05-09T08:24:48.787 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-05-09T08:24:48.788 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.error_if_exists: 0 2024-05-09T08:24:48.788 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.create_if_missing: 0 2024-05-09T08:24:48.788 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.paranoid_checks: 1 2024-05-09T08:24:48.788 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-09T08:24:48.788 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-09T08:24:48.788 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-09T08:24:48.788 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.env: 0x559266874c60 2024-05-09T08:24:48.788 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.fs: PosixFileSystem 2024-05-09T08:24:48.788 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.info_log: 0x559268cc7ac0 2024-05-09T08:24:48.788 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_file_opening_threads: 16 2024-05-09T08:24:48.788 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.statistics: (nil) 2024-05-09T08:24:48.789 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.use_fsync: 0 2024-05-09T08:24:48.789 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_log_file_size: 0 2024-05-09T08:24:48.789 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-09T08:24:48.789 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-09T08:24:48.789 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.keep_log_file_num: 1000 2024-05-09T08:24:48.789 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.recycle_log_file_num: 0 2024-05-09T08:24:48.789 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.allow_fallocate: 1 2024-05-09T08:24:48.789 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.allow_mmap_reads: 0 2024-05-09T08:24:48.789 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.allow_mmap_writes: 0 2024-05-09T08:24:48.789 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.use_direct_reads: 0 2024-05-09T08:24:48.789 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-09T08:24:48.789 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.create_missing_column_families: 0 2024-05-09T08:24:48.790 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.db_log_dir: 2024-05-09T08:24:48.790 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.wal_dir: 2024-05-09T08:24:48.790 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-09T08:24:48.790 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-09T08:24:48.790 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-09T08:24:48.790 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-09T08:24:48.790 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-09T08:24:48.790 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-09T08:24:48.790 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.advise_random_on_open: 1 2024-05-09T08:24:48.790 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.db_write_buffer_size: 0 2024-05-09T08:24:48.790 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.write_buffer_manager: 0x559268c9a1e0 2024-05-09T08:24:48.790 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-09T08:24:48.790 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-09T08:24:48.791 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-09T08:24:48.791 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.rate_limiter: (nil) 2024-05-09T08:24:48.791 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-09T08:24:48.791 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.wal_recovery_mode: 2 2024-05-09T08:24:48.791 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.enable_thread_tracking: 0 2024-05-09T08:24:48.791 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.enable_pipelined_write: 0 2024-05-09T08:24:48.791 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.unordered_write: 0 2024-05-09T08:24:48.791 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-09T08:24:48.791 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-09T08:24:48.791 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-09T08:24:48.791 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-09T08:24:48.791 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.row_cache: None 2024-05-09T08:24:48.791 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.wal_filter: None 2024-05-09T08:24:48.793 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-09T08:24:48.793 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.allow_ingest_behind: 0 2024-05-09T08:24:48.793 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.two_write_queues: 0 2024-05-09T08:24:48.793 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.manual_wal_flush: 0 2024-05-09T08:24:48.794 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.wal_compression: 0 2024-05-09T08:24:48.794 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.atomic_flush: 0 2024-05-09T08:24:48.794 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-09T08:24:48.794 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-09T08:24:48.794 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-09T08:24:48.794 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.log_readahead_size: 0 2024-05-09T08:24:48.794 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-09T08:24:48.794 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.best_efforts_recovery: 0 2024-05-09T08:24:48.794 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-09T08:24:48.794 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-09T08:24:48.794 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.allow_data_in_errors: 0 2024-05-09T08:24:48.794 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.db_host_id: __hostname__ 2024-05-09T08:24:48.794 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-09T08:24:48.795 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_background_jobs: 2 2024-05-09T08:24:48.795 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_background_compactions: -1 2024-05-09T08:24:48.795 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_subcompactions: 1 2024-05-09T08:24:48.795 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-09T08:24:48.795 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-09T08:24:48.795 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-09T08:24:48.795 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_total_wal_size: 0 2024-05-09T08:24:48.795 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-09T08:24:48.795 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-09T08:24:48.795 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-09T08:24:48.795 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-09T08:24:48.795 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_open_files: -1 2024-05-09T08:24:48.796 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.bytes_per_sync: 0 2024-05-09T08:24:48.796 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-09T08:24:48.796 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-09T08:24:48.796 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compaction_readahead_size: 0 2024-05-09T08:24:48.796 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_background_flushes: -1 2024-05-09T08:24:48.796 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Compression algorithms supported: 2024-05-09T08:24:48.797 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-09T08:24:48.797 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: kZSTD supported: 0 2024-05-09T08:24:48.797 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: kXpressCompression supported: 0 2024-05-09T08:24:48.797 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: kLZ4HCCompression supported: 1 2024-05-09T08:24:48.797 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: kZlibCompression supported: 1 2024-05-09T08:24:48.797 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: kSnappyCompression supported: 1 2024-05-09T08:24:48.797 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: kLZ4Compression supported: 1 2024-05-09T08:24:48.798 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: kBZip2Compression supported: 0 2024-05-09T08:24:48.798 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-09T08:24:48.798 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-09T08:24:48.798 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-05-09T08:24:48.798 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-09T08:24:48.798 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-09T08:24:48.799 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.merge_operator: 2024-05-09T08:24:48.799 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compaction_filter: None 2024-05-09T08:24:48.799 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compaction_filter_factory: None 2024-05-09T08:24:48.799 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.sst_partitioner_factory: None 2024-05-09T08:24:48.799 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-09T08:24:48.799 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-09T08:24:48.800 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559268cc7bc0) 2024-05-09T08:24:48.800 INFO:journalctl@ceph.mon.b.smithi110.stdout: cache_index_and_filter_blocks: 1 2024-05-09T08:24:48.800 INFO:journalctl@ceph.mon.b.smithi110.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-09T08:24:48.800 INFO:journalctl@ceph.mon.b.smithi110.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-09T08:24:48.800 INFO:journalctl@ceph.mon.b.smithi110.stdout: pin_top_level_index_and_filter: 1 2024-05-09T08:24:48.801 INFO:journalctl@ceph.mon.b.smithi110.stdout: index_type: 0 2024-05-09T08:24:48.801 INFO:journalctl@ceph.mon.b.smithi110.stdout: data_block_index_type: 0 2024-05-09T08:24:48.801 INFO:journalctl@ceph.mon.b.smithi110.stdout: index_shortening: 1 2024-05-09T08:24:48.801 INFO:journalctl@ceph.mon.b.smithi110.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-09T08:24:48.801 INFO:journalctl@ceph.mon.b.smithi110.stdout: checksum: 4 2024-05-09T08:24:48.801 INFO:journalctl@ceph.mon.b.smithi110.stdout: no_block_cache: 0 2024-05-09T08:24:48.802 INFO:journalctl@ceph.mon.b.smithi110.stdout: block_cache: 0x559268ccb090 2024-05-09T08:24:48.802 INFO:journalctl@ceph.mon.b.smithi110.stdout: block_cache_name: BinnedLRUCache 2024-05-09T08:24:48.802 INFO:journalctl@ceph.mon.b.smithi110.stdout: block_cache_options: 2024-05-09T08:24:48.802 INFO:journalctl@ceph.mon.b.smithi110.stdout: capacity : 536870912 2024-05-09T08:24:48.802 INFO:journalctl@ceph.mon.b.smithi110.stdout: num_shard_bits : 4 2024-05-09T08:24:48.802 INFO:journalctl@ceph.mon.b.smithi110.stdout: strict_capacity_limit : 0 2024-05-09T08:24:48.803 INFO:journalctl@ceph.mon.b.smithi110.stdout: high_pri_pool_ratio: 0.000 2024-05-09T08:24:48.803 INFO:journalctl@ceph.mon.b.smithi110.stdout: block_cache_compressed: (nil) 2024-05-09T08:24:48.803 INFO:journalctl@ceph.mon.b.smithi110.stdout: persistent_cache: (nil) 2024-05-09T08:24:48.803 INFO:journalctl@ceph.mon.b.smithi110.stdout: block_size: 4096 2024-05-09T08:24:48.803 INFO:journalctl@ceph.mon.b.smithi110.stdout: block_size_deviation: 10 2024-05-09T08:24:48.804 INFO:journalctl@ceph.mon.b.smithi110.stdout: block_restart_interval: 16 2024-05-09T08:24:48.804 INFO:journalctl@ceph.mon.b.smithi110.stdout: index_block_restart_interval: 1 2024-05-09T08:24:48.804 INFO:journalctl@ceph.mon.b.smithi110.stdout: metadata_block_size: 4096 2024-05-09T08:24:48.804 INFO:journalctl@ceph.mon.b.smithi110.stdout: partition_filters: 0 2024-05-09T08:24:48.804 INFO:journalctl@ceph.mon.b.smithi110.stdout: use_delta_encoding: 1 2024-05-09T08:24:48.804 INFO:journalctl@ceph.mon.b.smithi110.stdout: filter_policy: bloomfilter 2024-05-09T08:24:48.804 INFO:journalctl@ceph.mon.b.smithi110.stdout: whole_key_filtering: 1 2024-05-09T08:24:48.805 INFO:journalctl@ceph.mon.b.smithi110.stdout: verify_compression: 0 2024-05-09T08:24:48.805 INFO:journalctl@ceph.mon.b.smithi110.stdout: read_amp_bytes_per_bit: 0 2024-05-09T08:24:48.805 INFO:journalctl@ceph.mon.b.smithi110.stdout: format_version: 5 2024-05-09T08:24:48.805 INFO:journalctl@ceph.mon.b.smithi110.stdout: enable_index_compression: 1 2024-05-09T08:24:48.805 INFO:journalctl@ceph.mon.b.smithi110.stdout: block_align: 0 2024-05-09T08:24:48.806 INFO:journalctl@ceph.mon.b.smithi110.stdout: max_auto_readahead_size: 262144 2024-05-09T08:24:48.806 INFO:journalctl@ceph.mon.b.smithi110.stdout: prepopulate_block_cache: 0 2024-05-09T08:24:48.806 INFO:journalctl@ceph.mon.b.smithi110.stdout: initial_auto_readahead_size: 8192 2024-05-09T08:24:48.806 INFO:journalctl@ceph.mon.b.smithi110.stdout: num_file_reads_for_auto_readahead: 2 2024-05-09T08:24:48.806 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.write_buffer_size: 33554432 2024-05-09T08:24:48.806 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_write_buffer_number: 2 2024-05-09T08:24:48.807 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compression: NoCompression 2024-05-09T08:24:48.807 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.bottommost_compression: Disabled 2024-05-09T08:24:48.807 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.prefix_extractor: nullptr 2024-05-09T08:24:48.807 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-09T08:24:48.807 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.num_levels: 7 2024-05-09T08:24:48.808 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-09T08:24:48.808 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-09T08:24:48.808 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-09T08:24:48.808 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-09T08:24:48.808 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-09T08:24:48.808 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-09T08:24:48.809 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-09T08:24:48.809 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-09T08:24:48.809 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-09T08:24:48.809 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-09T08:24:48.809 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-09T08:24:48.809 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-09T08:24:48.810 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-09T08:24:48.810 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compression_opts.level: 32767 2024-05-09T08:24:48.810 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compression_opts.strategy: 0 2024-05-09T08:24:48.810 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-09T08:24:48.810 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-09T08:24:48.810 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-09T08:24:48.811 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-09T08:24:48.811 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compression_opts.enabled: false 2024-05-09T08:24:48.811 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-09T08:24:48.811 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-09T08:24:48.811 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-09T08:24:48.811 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-09T08:24:48.812 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.target_file_size_base: 67108864 2024-05-09T08:24:48.812 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-09T08:24:48.812 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-09T08:24:48.812 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-09T08:24:48.812 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-09T08:24:48.813 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-09T08:24:48.813 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-09T08:24:48.813 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-09T08:24:48.813 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-09T08:24:48.813 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-09T08:24:48.813 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-09T08:24:48.814 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-09T08:24:48.814 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-09T08:24:48.814 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-09T08:24:48.814 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-09T08:24:48.814 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.arena_block_size: 1048576 2024-05-09T08:24:48.815 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-09T08:24:48.815 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-09T08:24:48.815 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.disable_auto_compactions: 0 2024-05-09T08:24:48.815 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-09T08:24:48.815 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-09T08:24:48.815 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-09T08:24:48.816 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-09T08:24:48.816 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-09T08:24:48.816 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-09T08:24:48.816 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-09T08:24:48.816 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-09T08:24:48.816 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-09T08:24:48.817 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-09T08:24:48.817 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.table_properties_collectors: 2024-05-09T08:24:48.817 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.inplace_update_support: 0 2024-05-09T08:24:48.817 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-09T08:24:48.817 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-09T08:24:48.817 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-09T08:24:48.818 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-09T08:24:48.818 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.bloom_locality: 0 2024-05-09T08:24:48.818 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.max_successive_merges: 0 2024-05-09T08:24:48.818 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-09T08:24:48.818 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.paranoid_file_checks: 0 2024-05-09T08:24:48.818 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.force_consistency_checks: 1 2024-05-09T08:24:48.819 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.report_bg_io_stats: 0 2024-05-09T08:24:48.819 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.ttl: 2592000 2024-05-09T08:24:48.819 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-09T08:24:48.819 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-09T08:24:48.819 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-09T08:24:48.819 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.enable_blob_files: false 2024-05-09T08:24:48.820 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.min_blob_size: 0 2024-05-09T08:24:48.820 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.blob_file_size: 268435456 2024-05-09T08:24:48.820 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-09T08:24:48.820 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-09T08:24:48.820 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-09T08:24:48.820 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-09T08:24:48.821 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-09T08:24:48.821 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.blob_file_starting_level: 0 2024-05-09T08:24:48.821 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-09T08:24:48.821 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-09T08:24:48.821 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-09T08:24:48.821 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 68345324-b235-41a1-87d0-6baee4dde9e4 2024-05-09T08:24:48.822 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715243088765145, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-09T08:24:48.822 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-09T08:24:48.822 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715243088765559, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715243088, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "68345324-b235-41a1-87d0-6baee4dde9e4", "db_session_id": "JPTX1JVBNTTBSMAYGJWH", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-09T08:24:48.822 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715243088765637, "job": 1, "event": "recovery_finished"} 2024-05-09T08:24:48.822 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-09T08:24:48.822 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-09T08:24:48.823 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559268dbc000 2024-05-09T08:24:48.823 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: DB pointer 0x559268da6000 2024-05-09T08:24:48.823 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-09T08:24:48.823 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-09T08:24:48.823 INFO:journalctl@ceph.mon.b.smithi110.stdout: ** DB Stats ** 2024-05-09T08:24:48.823 INFO:journalctl@ceph.mon.b.smithi110.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-09T08:24:48.824 INFO:journalctl@ceph.mon.b.smithi110.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-09T08:24:48.824 INFO:journalctl@ceph.mon.b.smithi110.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-09T08:24:48.824 INFO:journalctl@ceph.mon.b.smithi110.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-09T08:24:48.824 INFO:journalctl@ceph.mon.b.smithi110.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-09T08:24:48.824 INFO:journalctl@ceph.mon.b.smithi110.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-09T08:24:48.824 INFO:journalctl@ceph.mon.b.smithi110.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-09T08:24:48.825 INFO:journalctl@ceph.mon.b.smithi110.stdout: 2024-05-09T08:24:48.825 INFO:journalctl@ceph.mon.b.smithi110.stdout: ** Compaction Stats [default] ** 2024-05-09T08:24:48.825 INFO:journalctl@ceph.mon.b.smithi110.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-09T08:24:48.825 INFO:journalctl@ceph.mon.b.smithi110.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-09T08:24:48.825 INFO:journalctl@ceph.mon.b.smithi110.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-09T08:24:48.825 INFO:journalctl@ceph.mon.b.smithi110.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-09T08:24:48.826 INFO:journalctl@ceph.mon.b.smithi110.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-09T08:24:48.826 INFO:journalctl@ceph.mon.b.smithi110.stdout: 2024-05-09T08:24:48.826 INFO:journalctl@ceph.mon.b.smithi110.stdout: ** Compaction Stats [default] ** 2024-05-09T08:24:48.826 INFO:journalctl@ceph.mon.b.smithi110.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-09T08:24:48.826 INFO:journalctl@ceph.mon.b.smithi110.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-09T08:24:48.826 INFO:journalctl@ceph.mon.b.smithi110.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-09T08:24:48.827 INFO:journalctl@ceph.mon.b.smithi110.stdout: 2024-05-09T08:24:48.827 INFO:journalctl@ceph.mon.b.smithi110.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-09T08:24:48.827 INFO:journalctl@ceph.mon.b.smithi110.stdout: 2024-05-09T08:24:48.827 INFO:journalctl@ceph.mon.b.smithi110.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-09T08:24:48.827 INFO:journalctl@ceph.mon.b.smithi110.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-09T08:24:48.827 INFO:journalctl@ceph.mon.b.smithi110.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-09T08:24:48.828 INFO:journalctl@ceph.mon.b.smithi110.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-09T08:24:48.828 INFO:journalctl@ceph.mon.b.smithi110.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-09T08:24:48.828 INFO:journalctl@ceph.mon.b.smithi110.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-09T08:24:48.828 INFO:journalctl@ceph.mon.b.smithi110.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-09T08:24:48.828 INFO:journalctl@ceph.mon.b.smithi110.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-09T08:24:48.828 INFO:journalctl@ceph.mon.b.smithi110.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-09T08:24:48.829 INFO:journalctl@ceph.mon.b.smithi110.stdout: Block cache BinnedLRUCache@0x559268ccb090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-09T08:24:48.829 INFO:journalctl@ceph.mon.b.smithi110.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-09T08:24:48.829 INFO:journalctl@ceph.mon.b.smithi110.stdout: 2024-05-09T08:24:48.829 INFO:journalctl@ceph.mon.b.smithi110.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-09T08:24:48.829 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-05-09T08:24:48.829 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: using public_addr v2:172.21.15.110:0/0 -> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-05-09T08:24:48.830 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: starting mon.b rank -1 at public addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] at bind addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:48.830 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.b@-1(???) e0 preinit fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:48.830 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.b@-1(synchronizing).mds e1 new map 2024-05-09T08:24:48.830 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.b@-1(synchronizing).mds e1 print_map 2024-05-09T08:24:48.830 INFO:journalctl@ceph.mon.b.smithi110.stdout: e1 2024-05-09T08:24:48.831 INFO:journalctl@ceph.mon.b.smithi110.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-09T08:24:48.831 INFO:journalctl@ceph.mon.b.smithi110.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-09T08:24:48.831 INFO:journalctl@ceph.mon.b.smithi110.stdout: legacy client fscid: -1 2024-05-09T08:24:48.831 INFO:journalctl@ceph.mon.b.smithi110.stdout: 2024-05-09T08:24:48.831 INFO:journalctl@ceph.mon.b.smithi110.stdout: No filesystems configured 2024-05-09T08:24:48.831 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-09T08:24:48.832 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-09T08:24:48.832 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-09T08:24:48.832 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-09T08:24:48.832 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-09T08:24:48.832 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-09T08:24:48.832 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-09T08:24:48.833 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:24:48.833 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:24:48.833 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T08:24:48.833 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:48.833 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi157", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:48.833 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.834 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.834 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.834 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.834 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:24:48.834 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:48.834 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:24:48.835 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: Updating smithi110:/etc/ceph/ceph.conf 2024-05-09T08:24:48.835 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: Updating smithi110:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:48.835 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: Deploying cephadm binary to smithi157 2024-05-09T08:24:48.835 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:48.835 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:24:48.835 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: Updating smithi110:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:24:48.836 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:48.836 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.836 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.836 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.836 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.836 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:24:48.837 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: Added host smithi157 2024-05-09T08:24:48.837 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:48.837 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.837 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.837 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:48.837 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.838 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:24:48.838 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:48.838 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/193375680' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-09T08:24:48.838 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/193375680' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-09T08:24:48.838 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:24:48.839 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.839 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.839 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.839 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.839 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi157", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:24:48.839 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:48.840 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:24:48.840 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: Updating smithi157:/etc/ceph/ceph.conf 2024-05-09T08:24:48.840 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: Updating smithi157:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:48.840 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: Updating smithi157:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:24:48.840 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: Updating smithi157:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:24:48.840 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:48.841 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.841 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.841 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.841 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:172.21.15.107=a;smithi110:172.21.15.110=b;smithi157:172.21.15.157=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:24:48.841 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: Saving service mon spec with placement smithi107:172.21.15.107=a;smithi110:172.21.15.110=b;smithi157:172.21.15.157=c;count:3 2024-05-09T08:24:48.841 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.842 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:24:48.842 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:48.842 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:24:48.842 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.842 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:24:48.843 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:48.843 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: Deploying daemon mon.c on smithi157 2024-05-09T08:24:48.843 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:48.843 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:48.843 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:24:48.843 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.a calling monitor election 2024-05-09T08:24:48.844 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:48.844 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='client.? 172.21.15.157:0/2850624798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T08:24:48.844 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:48.844 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:48.844 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:48.844 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.c calling monitor election 2024-05-09T08:24:48.845 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:48.845 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:48.845 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:48.845 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:48.845 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-09T08:24:48.845 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: monmap epoch 2 2024-05-09T08:24:48.846 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:48.846 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: last_changed 2024-05-09T08:24:39.870153+0000 2024-05-09T08:24:48.846 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: created 2024-05-09T08:22:54.137531+0000 2024-05-09T08:24:48.846 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: min_mon_release 19 (squid) 2024-05-09T08:24:48.846 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: election_strategy: 1 2024-05-09T08:24:48.846 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-05-09T08:24:48.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: 1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.c 2024-05-09T08:24:48.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: fsmap 2024-05-09T08:24:48.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:24:48.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mgrmap e13: x(active, since 49s) 2024-05-09T08:24:48.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: overall HEALTH_OK 2024-05-09T08:24:48.847 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:48.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:24:48.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:48.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: Deploying daemon mon.b on smithi110 2024-05-09T08:24:48.848 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:48.849 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:48 smithi157 ceph-mon[33614]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:49.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:49.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:49.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 ceph-mon[33622]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-09T08:24:49.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 bash[33602]: ca5207af62f5837d92f0d60f651018c8ca7d3ab294c81771dfc844555770d6d9 2024-05-09T08:24:49.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:48 smithi110 systemd[1]: Started Ceph mon.b for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:24:50.046 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-09T08:24:50.046 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-09T08:24:50.304 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:24:54.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:24:54.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: mon.a calling monitor election 2024-05-09T08:24:54.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: mon.c calling monitor election 2024-05-09T08:24:54.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:54.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:54.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:54.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:54.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:54.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:54.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:54.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:54.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:54.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:54.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-09T08:24:54.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: monmap epoch 3 2024-05-09T08:24:54.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:54.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: last_changed 2024-05-09T08:24:48.792436+0000 2024-05-09T08:24:54.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: created 2024-05-09T08:22:54.137531+0000 2024-05-09T08:24:54.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: min_mon_release 19 (squid) 2024-05-09T08:24:54.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: election_strategy: 1 2024-05-09T08:24:54.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-05-09T08:24:54.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: 1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.c 2024-05-09T08:24:54.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-05-09T08:24:54.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: fsmap 2024-05-09T08:24:54.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:24:54.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: mgrmap e13: x(active, since 58s) 2024-05-09T08:24:54.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: overall HEALTH_OK 2024-05-09T08:24:54.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:54.283 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:53 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:54.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:24:54.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: mon.a calling monitor election 2024-05-09T08:24:54.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: mon.c calling monitor election 2024-05-09T08:24:54.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:54.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:54.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:54.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:54.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:54.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:54.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:54.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:54.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:54.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:54.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-09T08:24:54.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: monmap epoch 3 2024-05-09T08:24:54.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:54.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: last_changed 2024-05-09T08:24:48.792436+0000 2024-05-09T08:24:54.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: created 2024-05-09T08:22:54.137531+0000 2024-05-09T08:24:54.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: min_mon_release 19 (squid) 2024-05-09T08:24:54.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: election_strategy: 1 2024-05-09T08:24:54.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-05-09T08:24:54.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: 1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.c 2024-05-09T08:24:54.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-05-09T08:24:54.347 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: fsmap 2024-05-09T08:24:54.347 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:24:54.347 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: mgrmap e13: x(active, since 58s) 2024-05-09T08:24:54.347 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: overall HEALTH_OK 2024-05-09T08:24:54.347 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:54.347 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:53 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:54.739 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.737+0000 7fecb90dc700 1 -- 172.21.15.157:0/3879341915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec9c0035c0 msgr2=0x7fec9c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:54.739 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.737+0000 7fecb90dc700 1 --2- 172.21.15.157:0/3879341915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec9c0035c0 0x7fec9c005a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fecb4101d70 tx=0x7feca400feb0 comp rx=0 tx=0).stop 2024-05-09T08:24:54.739 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.737+0000 7fecb90dc700 1 -- 172.21.15.157:0/3879341915 shutdown_connections 2024-05-09T08:24:54.739 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.737+0000 7fecb90dc700 1 --2- 172.21.15.157:0/3879341915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec9c0035c0 0x7fec9c005a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:54.739 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.737+0000 7fecb90dc700 1 -- 172.21.15.157:0/3879341915 >> 172.21.15.157:0/3879341915 conn(0x7fecb40fd930 msgr2=0x7fecb40ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:54.739 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.737+0000 7fecb90dc700 1 -- 172.21.15.157:0/3879341915 shutdown_connections 2024-05-09T08:24:54.739 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.737+0000 7fecb90dc700 1 -- 172.21.15.157:0/3879341915 wait complete. 2024-05-09T08:24:54.741 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.738+0000 7fecb90dc700 1 Processor -- start 2024-05-09T08:24:54.741 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb90dc700 1 -- start start 2024-05-09T08:24:54.741 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb90dc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec9c0035c0 0x7fecb419d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:54.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb90dc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecb419dbf0 0x7fecb4197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:54.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb90dc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecb4197dc0 0x7fecb4198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:54.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb90dc700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fecb4109d30 con 0x7fecb4197dc0 2024-05-09T08:24:54.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb90dc700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fecb4109bb0 con 0x7fec9c0035c0 2024-05-09T08:24:54.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb90dc700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fecb4109eb0 con 0x7fecb419dbf0 2024-05-09T08:24:54.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb259c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecb419dbf0 0x7fecb4197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:54.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb259c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecb419dbf0 0x7fecb4197880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:36520/0 (socket says 172.21.15.157:36520) 2024-05-09T08:24:54.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb259c700 1 -- 172.21.15.157:0/3314489888 learned_addr learned my addr 172.21.15.157:0/3314489888 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-05-09T08:24:54.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb359e700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecb4197dc0 0x7fecb4198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:54.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb2d9d700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec9c0035c0 0x7fecb419d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:54.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb259c700 1 -- 172.21.15.157:0/3314489888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecb419dbf0 msgr2=0x7fecb4197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-09T08:24:54.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb259c700 1 -- 172.21.15.157:0/3314489888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecb419dbf0 msgr2=0x7fecb4197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T08:24:54.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb259c700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecb419dbf0 0x7fecb4197880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T08:24:54.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb259c700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecb419dbf0 0x7fecb4197880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:24:54.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb2d9d700 1 -- 172.21.15.157:0/3314489888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec9c0035c0 msgr2=0x7fecb419d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-05-09T08:24:54.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb2d9d700 1 -- 172.21.15.157:0/3314489888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec9c0035c0 msgr2=0x7fecb419d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T08:24:54.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb2d9d700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec9c0035c0 0x7fecb419d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T08:24:54.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb2d9d700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec9c0035c0 0x7fecb419d6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:24:54.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb359e700 1 -- 172.21.15.157:0/3314489888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecb419dbf0 msgr2=0x7fecb4197880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:24:54.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb359e700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecb419dbf0 0x7fecb4197880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:54.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb359e700 1 -- 172.21.15.157:0/3314489888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec9c0035c0 msgr2=0x7fecb419d6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:24:54.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.739+0000 7fecb359e700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec9c0035c0 0x7fecb419d6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:54.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.740+0000 7fecb359e700 1 -- 172.21.15.157:0/3314489888 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecb41a4510 con 0x7fecb4197dc0 2024-05-09T08:24:54.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.740+0000 7fecb359e700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecb4197dc0 0x7fecb4198270 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fecac00bd50 tx=0x7fecac00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:54.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.740+0000 7fec9bfff700 1 -- 172.21.15.157:0/3314489888 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecac019070 con 0x7fecb4197dc0 2024-05-09T08:24:54.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.740+0000 7fecb90dc700 1 -- 172.21.15.157:0/3314489888 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecb41a47a0 con 0x7fecb4197dc0 2024-05-09T08:24:54.744 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.740+0000 7fecb90dc700 1 -- 172.21.15.157:0/3314489888 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecb41a4cb0 con 0x7fecb4197dc0 2024-05-09T08:24:54.744 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.740+0000 7fec9bfff700 1 -- 172.21.15.157:0/3314489888 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fecac00cd10 con 0x7fecb4197dc0 2024-05-09T08:24:54.744 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.740+0000 7fec9bfff700 1 -- 172.21.15.157:0/3314489888 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecac01d6e0 con 0x7fecb4197dc0 2024-05-09T08:24:54.744 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.741+0000 7fec9bfff700 1 -- 172.21.15.157:0/3314489888 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fecac01c070 con 0x7fecb4197dc0 2024-05-09T08:24:54.744 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.741+0000 7fecb90dc700 1 -- 172.21.15.157:0/3314489888 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec9c002140 con 0x7fecb4197dc0 2024-05-09T08:24:54.745 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.741+0000 7fec9bfff700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7feca0039580 0x7feca003ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:54.745 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.741+0000 7fec9bfff700 1 -- 172.21.15.157:0/3314489888 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fecac04baf0 con 0x7fecb4197dc0 2024-05-09T08:24:54.745 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.741+0000 7fecb2d9d700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7feca0039580 0x7feca003ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:54.745 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.742+0000 7fecb2d9d700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7feca0039580 0x7feca003ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7feca4005b40 tx=0x7feca4005a90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:54.748 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:54.747+0000 7fec9bfff700 1 -- 172.21.15.157:0/3314489888 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fecac007ab0 con 0x7fecb4197dc0 2024-05-09T08:24:55.038 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:55.036+0000 7fecb90dc700 1 -- 172.21.15.157:0/3314489888 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fec9c002c30 con 0x7fecb4197dc0 2024-05-09T08:24:55.038 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:55.038 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:55.038 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:24:55.038 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:55.039 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:55.037+0000 7fec9bfff700 1 -- 172.21.15.157:0/3314489888 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1287 (secure 0 0 0) 0x7fecac01a260 con 0x7fecb4197dc0 2024-05-09T08:24:55.039 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-05-09T08:24:55.039 INFO:teuthology.orchestra.run.smithi157.stdout:{"epoch":3,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","modified":"2024-05-09T08:24:48.792436Z","created":"2024-05-09T08:22:54.137531Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:3300","nonce":0},{"type":"v1","addr":"172.21.15.157:6789","nonce":0}]},"addr":"172.21.15.157:6789/0","public_addr":"172.21.15.157:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-09T08:24:55.040 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:55.039+0000 7fecb90dc700 1 -- 172.21.15.157:0/3314489888 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7feca0039580 msgr2=0x7feca003ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:55.041 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:55.039+0000 7fecb90dc700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7feca0039580 0x7feca003ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7feca4005b40 tx=0x7feca4005a90 comp rx=0 tx=0).stop 2024-05-09T08:24:55.041 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:55.039+0000 7fecb90dc700 1 -- 172.21.15.157:0/3314489888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecb4197dc0 msgr2=0x7fecb4198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:55.041 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:55.039+0000 7fecb90dc700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecb4197dc0 0x7fecb4198270 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fecac00bd50 tx=0x7fecac00be30 comp rx=0 tx=0).stop 2024-05-09T08:24:55.041 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:55.039+0000 7fecb90dc700 1 -- 172.21.15.157:0/3314489888 shutdown_connections 2024-05-09T08:24:55.041 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:55.039+0000 7fecb90dc700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7feca0039580 0x7feca003ba40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:55.041 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:55.039+0000 7fecb90dc700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecb4197dc0 0x7fecb4198270 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:55.041 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:55.039+0000 7fecb90dc700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec9c0035c0 0x7fecb419d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:55.041 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:55.039+0000 7fecb90dc700 1 --2- 172.21.15.157:0/3314489888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecb419dbf0 0x7fecb4197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:55.041 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:55.039+0000 7fecb90dc700 1 -- 172.21.15.157:0/3314489888 >> 172.21.15.157:0/3314489888 conn(0x7fecb40fd930 msgr2=0x7fecb4105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:55.041 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:55.039+0000 7fecb90dc700 1 -- 172.21.15.157:0/3314489888 shutdown_connections 2024-05-09T08:24:55.041 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:24:55.039+0000 7fecb90dc700 1 -- 172.21.15.157:0/3314489888 wait complete. 2024-05-09T08:24:55.042 INFO:teuthology.orchestra.run.smithi157.stderr:dumped monmap epoch 3 2024-05-09T08:24:55.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:55.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:55.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:24:55.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:55.984 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-09T08:24:55.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph config generate-minimal-conf 2024-05-09T08:24:56.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:24:56.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: mon.a calling monitor election 2024-05-09T08:24:56.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: mon.c calling monitor election 2024-05-09T08:24:56.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:56.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:24:56.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:56.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:56.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:56.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:56.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:56.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:56.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:56.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:56.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-09T08:24:56.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: monmap epoch 3 2024-05-09T08:24:56.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:56.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: last_changed 2024-05-09T08:24:48.792436+0000 2024-05-09T08:24:56.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: created 2024-05-09T08:22:54.137531+0000 2024-05-09T08:24:56.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: min_mon_release 19 (squid) 2024-05-09T08:24:56.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: election_strategy: 1 2024-05-09T08:24:56.124 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-05-09T08:24:56.124 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: 1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.c 2024-05-09T08:24:56.124 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-05-09T08:24:56.124 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: fsmap 2024-05-09T08:24:56.124 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:24:56.124 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: mgrmap e13: x(active, since 58s) 2024-05-09T08:24:56.124 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: overall HEALTH_OK 2024-05-09T08:24:56.124 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.124 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.124 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.124 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.124 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:24:56.124 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:56.230 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:24:56.631 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: mon.b calling monitor election 2024-05-09T08:24:56.631 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: mon.b calling monitor election 2024-05-09T08:24:56.631 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: mon.c calling monitor election 2024-05-09T08:24:56.631 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: mon.a calling monitor election 2024-05-09T08:24:56.631 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-09T08:24:56.631 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: monmap epoch 3 2024-05-09T08:24:56.631 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:56.631 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: last_changed 2024-05-09T08:24:48.792436+0000 2024-05-09T08:24:56.631 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: created 2024-05-09T08:22:54.137531+0000 2024-05-09T08:24:56.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: min_mon_release 19 (squid) 2024-05-09T08:24:56.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: election_strategy: 1 2024-05-09T08:24:56.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-05-09T08:24:56.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: 1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.c 2024-05-09T08:24:56.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-05-09T08:24:56.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: fsmap 2024-05-09T08:24:56.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:24:56.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: mgrmap e13: x(active, since 60s) 2024-05-09T08:24:56.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: overall HEALTH_OK 2024-05-09T08:24:56.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:24:56.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:24:56.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:56.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: mon.b calling monitor election 2024-05-09T08:24:56.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: mon.b calling monitor election 2024-05-09T08:24:56.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: mon.c calling monitor election 2024-05-09T08:24:56.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: mon.a calling monitor election 2024-05-09T08:24:56.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-09T08:24:56.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: monmap epoch 3 2024-05-09T08:24:56.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:56.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: last_changed 2024-05-09T08:24:48.792436+0000 2024-05-09T08:24:56.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: created 2024-05-09T08:22:54.137531+0000 2024-05-09T08:24:56.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: min_mon_release 19 (squid) 2024-05-09T08:24:56.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: election_strategy: 1 2024-05-09T08:24:56.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-05-09T08:24:56.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: 1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.c 2024-05-09T08:24:56.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-05-09T08:24:56.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: fsmap 2024-05-09T08:24:56.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:24:56.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: mgrmap e13: x(active, since 60s) 2024-05-09T08:24:56.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: overall HEALTH_OK 2024-05-09T08:24:56.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:24:56.848 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:24:56.848 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: mon.b calling monitor election 2024-05-09T08:24:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: mon.b calling monitor election 2024-05-09T08:24:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: mon.c calling monitor election 2024-05-09T08:24:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: mon.a calling monitor election 2024-05-09T08:24:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-09T08:24:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: monmap epoch 3 2024-05-09T08:24:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: last_changed 2024-05-09T08:24:48.792436+0000 2024-05-09T08:24:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: created 2024-05-09T08:22:54.137531+0000 2024-05-09T08:24:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: min_mon_release 19 (squid) 2024-05-09T08:24:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: election_strategy: 1 2024-05-09T08:24:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-05-09T08:24:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: 1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.c 2024-05-09T08:24:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-05-09T08:24:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: fsmap 2024-05-09T08:24:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T08:24:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: mgrmap e13: x(active, since 60s) 2024-05-09T08:24:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: overall HEALTH_OK 2024-05-09T08:24:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:56.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:24:56.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:24:56.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:57.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:57 smithi107 ceph-mon[30122]: Updating smithi107:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:57.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:57 smithi107 ceph-mon[30122]: Updating smithi110:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:57.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:57 smithi107 ceph-mon[30122]: Updating smithi157:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:57.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:57 smithi107 ceph-mon[30122]: Reconfiguring mon.a (unknown last config time)... 2024-05-09T08:24:57.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:57 smithi107 ceph-mon[30122]: Reconfiguring daemon mon.a on smithi107 2024-05-09T08:24:57.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:57 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:57.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:57 smithi157 ceph-mon[33614]: Updating smithi107:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:57.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:57 smithi157 ceph-mon[33614]: Updating smithi110:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:57.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:57 smithi157 ceph-mon[33614]: Updating smithi157:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:57.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:57 smithi157 ceph-mon[33614]: Reconfiguring mon.a (unknown last config time)... 2024-05-09T08:24:57.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:57 smithi157 ceph-mon[33614]: Reconfiguring daemon mon.a on smithi107 2024-05-09T08:24:57.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:57 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:57.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:57 smithi110 ceph-mon[33622]: Updating smithi107:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:57.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:57 smithi110 ceph-mon[33622]: Updating smithi110:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:57.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:57 smithi110 ceph-mon[33622]: Updating smithi157:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:24:57.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:57 smithi110 ceph-mon[33622]: Reconfiguring mon.a (unknown last config time)... 2024-05-09T08:24:57.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:57 smithi110 ceph-mon[33622]: Reconfiguring daemon mon.a on smithi107 2024-05-09T08:24:57.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:57 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:24:58.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:58 smithi107 ceph-mon[30122]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:58.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:58.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:58.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:24:58.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:24:58.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:58.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.769+0000 7f2d29e4b700 1 -- 172.21.15.107:0/2181811421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d241083b0 msgr2=0x7f2d24108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:58.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.769+0000 7f2d29e4b700 1 --2- 172.21.15.107:0/2181811421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d241083b0 0x7f2d24108790 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2d140098f0 tx=0x7f2d14009c00 comp rx=0 tx=0).stop 2024-05-09T08:24:58.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.769+0000 7f2d29e4b700 1 -- 172.21.15.107:0/2181811421 shutdown_connections 2024-05-09T08:24:58.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.769+0000 7f2d29e4b700 1 --2- 172.21.15.107:0/2181811421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d241012b0 0x7f2d2410f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.769+0000 7f2d29e4b700 1 --2- 172.21.15.107:0/2181811421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d241083b0 0x7f2d24108790 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.769+0000 7f2d29e4b700 1 --2- 172.21.15.107:0/2181811421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d24100910 0x7f2d24100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.769+0000 7f2d29e4b700 1 -- 172.21.15.107:0/2181811421 >> 172.21.15.107:0/2181811421 conn(0x7f2d24077890 msgr2=0x7f2d240fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:58.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.769+0000 7f2d29e4b700 1 -- 172.21.15.107:0/2181811421 shutdown_connections 2024-05-09T08:24:58.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.769+0000 7f2d29e4b700 1 -- 172.21.15.107:0/2181811421 wait complete. 2024-05-09T08:24:58.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.770+0000 7f2d29e4b700 1 Processor -- start 2024-05-09T08:24:58.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.770+0000 7f2d29e4b700 1 -- start start 2024-05-09T08:24:58.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.770+0000 7f2d29e4b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d24100910 0x7f2d24101af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:58.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.770+0000 7f2d29e4b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d241012b0 0x7f2d24102030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:58.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.770+0000 7f2d29e4b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d241083b0 0x7f2d24102570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:58.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.770+0000 7f2d29e4b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2d241124a0 con 0x7f2d241012b0 2024-05-09T08:24:58.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.770+0000 7f2d29e4b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2d24112320 con 0x7f2d241083b0 2024-05-09T08:24:58.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.770+0000 7f2d29e4b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2d24112620 con 0x7f2d24100910 2024-05-09T08:24:58.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d1affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d241012b0 0x7f2d24102030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:58.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d1affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d241012b0 0x7f2d24102030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58144/0 (socket says 172.21.15.107:58144) 2024-05-09T08:24:58.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d1affd700 1 -- 172.21.15.107:0/2576714857 learned_addr learned my addr 172.21.15.107:0/2576714857 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:24:58.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d237fe700 1 --2- 172.21.15.107:0/2576714857 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d24100910 0x7f2d24101af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:58.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d23fff700 1 --2- 172.21.15.107:0/2576714857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d241083b0 0x7f2d24102570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:58.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d1affd700 1 -- 172.21.15.107:0/2576714857 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d24100910 msgr2=0x7f2d24101af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:58.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d1affd700 1 --2- 172.21.15.107:0/2576714857 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d24100910 0x7f2d24101af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d1affd700 1 -- 172.21.15.107:0/2576714857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d241083b0 msgr2=0x7f2d24102570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:58.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d1affd700 1 --2- 172.21.15.107:0/2576714857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d241083b0 0x7f2d24102570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d1affd700 1 -- 172.21.15.107:0/2576714857 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d241a6f40 con 0x7f2d241012b0 2024-05-09T08:24:58.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d23fff700 1 --2- 172.21.15.107:0/2576714857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d241083b0 0x7f2d24102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:24:58.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d1affd700 1 --2- 172.21.15.107:0/2576714857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d241012b0 0x7f2d24102030 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c00d950 tx=0x7f2d0c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:58.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d217fa700 1 -- 172.21.15.107:0/2576714857 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c016070 con 0x7f2d241012b0 2024-05-09T08:24:58.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d217fa700 1 -- 172.21.15.107:0/2576714857 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d0c009df0 con 0x7f2d241012b0 2024-05-09T08:24:58.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d29e4b700 1 -- 172.21.15.107:0/2576714857 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d241a7230 con 0x7f2d241012b0 2024-05-09T08:24:58.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d217fa700 1 -- 172.21.15.107:0/2576714857 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c00f3c0 con 0x7f2d241012b0 2024-05-09T08:24:58.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.771+0000 7f2d29e4b700 1 -- 172.21.15.107:0/2576714857 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d241a7770 con 0x7f2d241012b0 2024-05-09T08:24:58.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.772+0000 7f2d217fa700 1 -- 172.21.15.107:0/2576714857 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f2d0c019070 con 0x7f2d241012b0 2024-05-09T08:24:58.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.772+0000 7f2d29e4b700 1 -- 172.21.15.107:0/2576714857 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d24103c30 con 0x7f2d241012b0 2024-05-09T08:24:58.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.772+0000 7f2d217fa700 1 --2- 172.21.15.107:0/2576714857 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2d10039330 0x7f2d1003b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:24:58.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.772+0000 7f2d217fa700 1 -- 172.21.15.107:0/2576714857 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2d0c04c1e0 con 0x7f2d241012b0 2024-05-09T08:24:58.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.772+0000 7f2d237fe700 1 --2- 172.21.15.107:0/2576714857 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2d10039330 0x7f2d1003b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:24:58.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.773+0000 7f2d237fe700 1 --2- 172.21.15.107:0/2576714857 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2d10039330 0x7f2d1003b7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2d14009fd0 tx=0x7f2d14019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:24:58.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.777+0000 7f2d217fa700 1 -- 172.21.15.107:0/2576714857 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2d0c00fe20 con 0x7f2d241012b0 2024-05-09T08:24:58.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:58 smithi157 ceph-mon[33614]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:58.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:58.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:58.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:24:58.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:24:58.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:58.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:58 smithi110 ceph-mon[33622]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:24:58.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:58.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:24:58.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:24:58.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:24:58.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:58.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.975+0000 7f2d29e4b700 1 -- 172.21.15.107:0/2576714857 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f2d2402db80 con 0x7f2d241012b0 2024-05-09T08:24:58.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.975+0000 7f2d217fa700 1 -- 172.21.15.107:0/2576714857 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7f2d0c010460 con 0x7f2d241012b0 2024-05-09T08:24:58.976 INFO:teuthology.orchestra.run.smithi107.stdout:# minimal ceph.conf for 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:58.976 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2024-05-09T08:24:58.976 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:24:58.976 INFO:teuthology.orchestra.run.smithi107.stdout: mon_host = [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2024-05-09T08:24:58.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.978+0000 7f2d29e4b700 1 -- 172.21.15.107:0/2576714857 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2d10039330 msgr2=0x7f2d1003b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:58.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.978+0000 7f2d29e4b700 1 --2- 172.21.15.107:0/2576714857 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2d10039330 0x7f2d1003b7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2d14009fd0 tx=0x7f2d14019040 comp rx=0 tx=0).stop 2024-05-09T08:24:58.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.978+0000 7f2d29e4b700 1 -- 172.21.15.107:0/2576714857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d241012b0 msgr2=0x7f2d24102030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:24:58.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.978+0000 7f2d29e4b700 1 --2- 172.21.15.107:0/2576714857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d241012b0 0x7f2d24102030 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c00d950 tx=0x7f2d0c00dc60 comp rx=0 tx=0).stop 2024-05-09T08:24:58.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.979+0000 7f2d29e4b700 1 -- 172.21.15.107:0/2576714857 shutdown_connections 2024-05-09T08:24:58.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.979+0000 7f2d29e4b700 1 --2- 172.21.15.107:0/2576714857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d241083b0 0x7f2d24102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.979+0000 7f2d29e4b700 1 --2- 172.21.15.107:0/2576714857 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d24100910 0x7f2d24101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.979+0000 7f2d29e4b700 1 --2- 172.21.15.107:0/2576714857 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2d10039330 0x7f2d1003b7f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.979+0000 7f2d29e4b700 1 --2- 172.21.15.107:0/2576714857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d241012b0 0x7f2d24102030 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:24:58.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.979+0000 7f2d29e4b700 1 -- 172.21.15.107:0/2576714857 >> 172.21.15.107:0/2576714857 conn(0x7f2d24077890 msgr2=0x7f2d240ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:24:58.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.979+0000 7f2d29e4b700 1 -- 172.21.15.107:0/2576714857 shutdown_connections 2024-05-09T08:24:58.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:24:58.979+0000 7f2d29e4b700 1 -- 172.21.15.107:0/2576714857 wait complete. 2024-05-09T08:24:59.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:59 smithi107 ceph-mon[30122]: Reconfiguring mon.b (monmap changed)... 2024-05-09T08:24:59.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:59 smithi107 ceph-mon[30122]: Reconfiguring daemon mon.b on smithi110 2024-05-09T08:24:59.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:24:59 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2576714857' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:59.580 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-09T08:24:59.581 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:24:59.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.conf 2024-05-09T08:24:59.608 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:24:59.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:24:59.675 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-09T08:24:59.675 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.conf 2024-05-09T08:24:59.698 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:59 smithi110 ceph-mon[33622]: Reconfiguring mon.b (monmap changed)... 2024-05-09T08:24:59.698 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:59 smithi110 ceph-mon[33622]: Reconfiguring daemon mon.b on smithi110 2024-05-09T08:24:59.698 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:24:59 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2576714857' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:59.703 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-09T08:24:59.703 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:24:59.768 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-05-09T08:24:59.768 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.conf 2024-05-09T08:24:59.796 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:59 smithi157 ceph-mon[33614]: Reconfiguring mon.b (monmap changed)... 2024-05-09T08:24:59.796 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:59 smithi157 ceph-mon[33614]: Reconfiguring daemon mon.b on smithi110 2024-05-09T08:24:59.796 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:24:59 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2576714857' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:24:59.802 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-05-09T08:24:59.802 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:24:59.871 INFO:tasks.cephadm:Adding mgr.x on smithi107 2024-05-09T08:24:59.871 INFO:tasks.cephadm:Adding mgr.y on smithi110 2024-05-09T08:24:59.871 INFO:tasks.cephadm:Adding mgr.z on smithi157 2024-05-09T08:24:59.871 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch apply mgr '3;smithi107=x;smithi110=y;smithi157=z' 2024-05-09T08:25:00.157 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:25:00.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:00 smithi107 ceph-mon[30122]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:00.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:00 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:00.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:00 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:00.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:00 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:25:00.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:00 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:25:00.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:00 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:00.561 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:00 smithi157 ceph-mon[33614]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:00.562 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:00 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:00.562 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:00 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:00.562 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:00 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:25:00.562 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:00 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:25:00.562 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:00 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:00.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:00 smithi110 ceph-mon[33622]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:00.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:00 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:00.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:00 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:00.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:00 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T08:25:00.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:00 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T08:25:00.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:00 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:01.686 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:01 smithi157 ceph-mon[33614]: Reconfiguring mon.c (monmap changed)... 2024-05-09T08:25:01.687 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:01 smithi157 ceph-mon[33614]: Reconfiguring daemon mon.c on smithi157 2024-05-09T08:25:01.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:01 smithi107 ceph-mon[30122]: Reconfiguring mon.c (monmap changed)... 2024-05-09T08:25:01.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:01 smithi107 ceph-mon[30122]: Reconfiguring daemon mon.c on smithi157 2024-05-09T08:25:01.792 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.789+0000 7f409a174700 1 -- 172.21.15.157:0/4234168506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4094103490 msgr2=0x7f40941038b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:01.792 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.789+0000 7f409a174700 1 --2- 172.21.15.157:0/4234168506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4094103490 0x7f40941038b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4084009910 tx=0x7f4084009c20 comp rx=0 tx=0).stop 2024-05-09T08:25:01.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.789+0000 7f409a174700 1 -- 172.21.15.157:0/4234168506 shutdown_connections 2024-05-09T08:25:01.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.789+0000 7f409a174700 1 --2- 172.21.15.157:0/4234168506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4094103490 0x7f40941038b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:01.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.789+0000 7f409a174700 1 -- 172.21.15.157:0/4234168506 >> 172.21.15.157:0/4234168506 conn(0x7f40940fbad0 msgr2=0x7f40940fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:01.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.789+0000 7f409a174700 1 -- 172.21.15.157:0/4234168506 shutdown_connections 2024-05-09T08:25:01.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.789+0000 7f409a174700 1 -- 172.21.15.157:0/4234168506 wait complete. 2024-05-09T08:25:01.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.790+0000 7f409a174700 1 Processor -- start 2024-05-09T08:25:01.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.790+0000 7f409a174700 1 -- start start 2024-05-09T08:25:01.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.790+0000 7f409a174700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4094103490 0x7f4094197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:01.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.790+0000 7f409a174700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4094197b60 0x7f409419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:01.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.790+0000 7f409a174700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4094197fe0 0x7f409419f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:01.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.790+0000 7f409a174700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4094107d90 con 0x7f4094103490 2024-05-09T08:25:01.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.790+0000 7f409a174700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4094107c10 con 0x7f4094197b60 2024-05-09T08:25:01.794 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.790+0000 7f409a174700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4094107f10 con 0x7f4094197fe0 2024-05-09T08:25:01.794 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.790+0000 7f4093fff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4094197fe0 0x7f409419f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:01.794 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.790+0000 7f4093fff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4094197fe0 0x7f409419f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:36532/0 (socket says 172.21.15.157:36532) 2024-05-09T08:25:01.794 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.790+0000 7f4093fff700 1 -- 172.21.15.157:0/4160484392 learned_addr learned my addr 172.21.15.157:0/4160484392 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-05-09T08:25:01.794 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.790+0000 7f40937fe700 1 --2- 172.21.15.157:0/4160484392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4094103490 0x7f4094197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:01.794 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.790+0000 7f4092ffd700 1 --2- 172.21.15.157:0/4160484392 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4094197b60 0x7f409419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:01.794 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.791+0000 7f4093fff700 1 -- 172.21.15.157:0/4160484392 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4094197b60 msgr2=0x7f409419cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:01.794 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.791+0000 7f4093fff700 1 --2- 172.21.15.157:0/4160484392 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4094197b60 0x7f409419cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:01.794 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.791+0000 7f4093fff700 1 -- 172.21.15.157:0/4160484392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4094103490 msgr2=0x7f4094197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:01.794 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.791+0000 7f4093fff700 1 --2- 172.21.15.157:0/4160484392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4094103490 0x7f4094197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:01.794 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.791+0000 7f4093fff700 1 -- 172.21.15.157:0/4160484392 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409419fa30 con 0x7f4094197fe0 2024-05-09T08:25:01.794 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.791+0000 7f40937fe700 1 --2- 172.21.15.157:0/4160484392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4094103490 0x7f4094197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:25:01.795 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.791+0000 7f4093fff700 1 --2- 172.21.15.157:0/4160484392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4094197fe0 0x7f409419f160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f408c00ba50 tx=0x7f408c00be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:01.795 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.791+0000 7f4090ff9700 1 -- 172.21.15.157:0/4160484392 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c019070 con 0x7f4094197fe0 2024-05-09T08:25:01.795 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.791+0000 7f4090ff9700 1 -- 172.21.15.157:0/4160484392 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f408c014440 con 0x7f4094197fe0 2024-05-09T08:25:01.795 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.791+0000 7f409a174700 1 -- 172.21.15.157:0/4160484392 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f409419fd20 con 0x7f4094197fe0 2024-05-09T08:25:01.795 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.791+0000 7f4090ff9700 1 -- 172.21.15.157:0/4160484392 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c007590 con 0x7f4094197fe0 2024-05-09T08:25:01.795 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.791+0000 7f409a174700 1 -- 172.21.15.157:0/4160484392 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40941a02d0 con 0x7f4094197fe0 2024-05-09T08:25:01.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.792+0000 7f4090ff9700 1 -- 172.21.15.157:0/4160484392 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f408c01c070 con 0x7f4094197fe0 2024-05-09T08:25:01.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.792+0000 7f409a174700 1 -- 172.21.15.157:0/4160484392 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4094063980 con 0x7f4094197fe0 2024-05-09T08:25:01.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.792+0000 7f4090ff9700 1 --2- 172.21.15.157:0/4160484392 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f407c039580 0x7f407c03ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:01.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.792+0000 7f4090ff9700 1 -- 172.21.15.157:0/4160484392 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f408c04b7f0 con 0x7f4094197fe0 2024-05-09T08:25:01.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.792+0000 7f40937fe700 1 --2- 172.21.15.157:0/4160484392 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f407c039580 0x7f407c03ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:01.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.793+0000 7f40937fe700 1 --2- 172.21.15.157:0/4160484392 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f407c039580 0x7f407c03ba40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f408400b5c0 tx=0x7f408401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:01.798 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:01.797+0000 7f4090ff9700 1 -- 172.21.15.157:0/4160484392 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f408c014980 con 0x7f4094197fe0 2024-05-09T08:25:01.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:01 smithi110 ceph-mon[33622]: Reconfiguring mon.c (monmap changed)... 2024-05-09T08:25:01.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:01 smithi110 ceph-mon[33622]: Reconfiguring daemon mon.c on smithi157 2024-05-09T08:25:02.021 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:02.020+0000 7f409a174700 1 -- 172.21.15.157:0/4160484392 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi107=x;smithi110=y;smithi157=z", "target": ["mon-mgr", ""]}) -- 0x7f4094062760 con 0x7f407c039580 2024-05-09T08:25:02.027 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:02.026+0000 7f4090ff9700 1 -- 172.21.15.157:0/4160484392 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4094062760 con 0x7f407c039580 2024-05-09T08:25:02.027 INFO:teuthology.orchestra.run.smithi157.stdout:Scheduled mgr update... 2024-05-09T08:25:02.029 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:02.028+0000 7f409a174700 1 -- 172.21.15.157:0/4160484392 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f407c039580 msgr2=0x7f407c03ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:02.030 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:02.028+0000 7f409a174700 1 --2- 172.21.15.157:0/4160484392 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f407c039580 0x7f407c03ba40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f408400b5c0 tx=0x7f408401a040 comp rx=0 tx=0).stop 2024-05-09T08:25:02.030 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:02.028+0000 7f409a174700 1 -- 172.21.15.157:0/4160484392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4094197fe0 msgr2=0x7f409419f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:02.030 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:02.028+0000 7f409a174700 1 --2- 172.21.15.157:0/4160484392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4094197fe0 0x7f409419f160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f408c00ba50 tx=0x7f408c00be10 comp rx=0 tx=0).stop 2024-05-09T08:25:02.030 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:02.028+0000 7f409a174700 1 -- 172.21.15.157:0/4160484392 shutdown_connections 2024-05-09T08:25:02.030 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:02.028+0000 7f409a174700 1 --2- 172.21.15.157:0/4160484392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4094197fe0 0x7f409419f160 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:02.030 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:02.028+0000 7f409a174700 1 --2- 172.21.15.157:0/4160484392 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f407c039580 0x7f407c03ba40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:02.030 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:02.028+0000 7f409a174700 1 --2- 172.21.15.157:0/4160484392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4094103490 0x7f4094197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:02.030 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:02.028+0000 7f409a174700 1 --2- 172.21.15.157:0/4160484392 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4094197b60 0x7f409419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:02.030 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:02.028+0000 7f409a174700 1 -- 172.21.15.157:0/4160484392 >> 172.21.15.157:0/4160484392 conn(0x7f40940fbad0 msgr2=0x7f40940fc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:02.030 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:02.029+0000 7f409a174700 1 -- 172.21.15.157:0/4160484392 shutdown_connections 2024-05-09T08:25:02.030 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:02.029+0000 7f409a174700 1 -- 172.21.15.157:0/4160484392 wait complete. 2024-05-09T08:25:02.742 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:02 smithi157 ceph-mon[33614]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:02.742 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:02 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:02.773 DEBUG:teuthology.orchestra.run.smithi110:mgr.y> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mgr.y.service 2024-05-09T08:25:02.777 DEBUG:teuthology.orchestra.run.smithi157:mgr.z> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mgr.z.service 2024-05-09T08:25:02.780 INFO:tasks.cephadm:Deploying OSDs... 2024-05-09T08:25:02.781 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:25:02.781 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T08:25:02.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:02 smithi107 ceph-mon[30122]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:02.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:02 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:02.800 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:02 smithi110 ceph-mon[33622]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:02.800 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:02 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:02.803 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T08:25:02.803 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2024-05-09T08:25:02.862 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T08:25:02.862 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:25:02.862 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-05-09T08:25:02.863 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:25:02.863 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:25:02.863 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-05-09 08:24:05.604215343 +0000 2024-05-09T08:25:02.863 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-05-09 08:19:21.942773638 +0000 2024-05-09T08:25:02.863 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-05-09 08:19:21.942773638 +0000 2024-05-09T08:25:02.863 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-05-09 08:19:21.942773638 +0000 2024-05-09T08:25:02.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T08:25:02.930 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-05-09T08:25:02.931 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-05-09T08:25:02.931 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000207481 s, 2.5 MB/s 2024-05-09T08:25:02.932 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T08:25:02.993 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2024-05-09T08:25:03.052 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T08:25:03.052 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:25:03.052 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-05-09T08:25:03.052 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:25:03.052 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:25:03.052 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-05-09 08:24:05.630214927 +0000 2024-05-09T08:25:03.052 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-05-09 08:19:22.299767915 +0000 2024-05-09T08:25:03.052 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-05-09 08:19:22.299767915 +0000 2024-05-09T08:25:03.052 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-05-09 08:19:22.299767915 +0000 2024-05-09T08:25:03.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T08:25:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-05-09T08:25:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-05-09T08:25:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000219766 s, 2.3 MB/s 2024-05-09T08:25:03.121 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T08:25:03.179 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2024-05-09T08:25:03.238 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T08:25:03.238 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:25:03.238 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-05-09T08:25:03.238 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:25:03.238 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:25:03.238 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-05-09 08:24:05.654214543 +0000 2024-05-09T08:25:03.238 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-05-09 08:19:22.638762473 +0000 2024-05-09T08:25:03.238 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-05-09 08:19:22.638762473 +0000 2024-05-09T08:25:03.238 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-05-09 08:19:22.638762473 +0000 2024-05-09T08:25:03.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T08:25:03.304 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-05-09T08:25:03.305 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-05-09T08:25:03.305 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000188397 s, 2.7 MB/s 2024-05-09T08:25:03.306 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T08:25:03.363 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2024-05-09T08:25:03.420 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T08:25:03.420 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:25:03.420 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-09T08:25:03.420 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:25:03.420 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:25:03.420 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-05-09 08:24:05.678214159 +0000 2024-05-09T08:25:03.420 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-05-09 08:19:22.979756994 +0000 2024-05-09T08:25:03.420 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-05-09 08:19:22.979756994 +0000 2024-05-09T08:25:03.420 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-05-09 08:19:22.979756994 +0000 2024-05-09T08:25:03.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T08:25:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-05-09T08:25:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-05-09T08:25:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000248193 s, 2.1 MB/s 2024-05-09T08:25:03.485 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T08:25:03.541 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-05-09T08:25:03.541 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T08:25:03.561 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T08:25:03.561 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2024-05-09T08:25:03.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:03 smithi157 ceph-mon[33614]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi107=x;smithi110=y;smithi157=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:03.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:03 smithi157 ceph-mon[33614]: Saving service mgr spec with placement smithi107=x;smithi110=y;smithi157=z;count:3 2024-05-09T08:25:03.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:03 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:03.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:03 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:03.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:03 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:03.620 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T08:25:03.620 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:25:03.620 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-05-09T08:25:03.620 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:25:03.620 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:25:03.620 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-05-09 08:24:48.221117274 +0000 2024-05-09T08:25:03.620 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-05-09 08:19:34.037423497 +0000 2024-05-09T08:25:03.620 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-05-09 08:19:34.037423497 +0000 2024-05-09T08:25:03.621 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-05-09 08:19:34.037423497 +0000 2024-05-09T08:25:03.621 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T08:25:03.687 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-05-09T08:25:03.687 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-05-09T08:25:03.687 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.00012717 s, 4.0 MB/s 2024-05-09T08:25:03.688 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T08:25:03.746 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2024-05-09T08:25:03.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:03 smithi107 ceph-mon[30122]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi107=x;smithi110=y;smithi157=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:03.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:03 smithi107 ceph-mon[30122]: Saving service mgr spec with placement smithi107=x;smithi110=y;smithi157=z;count:3 2024-05-09T08:25:03.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:03 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:03.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:03 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:03.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:03 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:03.803 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T08:25:03.803 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:25:03.803 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-09T08:25:03.803 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:25:03.803 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:25:03.803 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-05-09 08:24:48.221117274 +0000 2024-05-09T08:25:03.803 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-05-09 08:19:34.373418457 +0000 2024-05-09T08:25:03.803 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-05-09 08:19:34.373418457 +0000 2024-05-09T08:25:03.804 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-05-09 08:19:34.373418457 +0000 2024-05-09T08:25:03.804 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T08:25:03.866 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:03 smithi110 ceph-mon[33622]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi107=x;smithi110=y;smithi157=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:03.866 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:03 smithi110 ceph-mon[33622]: Saving service mgr spec with placement smithi107=x;smithi110=y;smithi157=z;count:3 2024-05-09T08:25:03.866 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:03 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:03.866 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:03 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:03.866 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:03 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:03.868 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-05-09T08:25:03.868 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-05-09T08:25:03.868 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 8.9762e-05 s, 5.7 MB/s 2024-05-09T08:25:03.870 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T08:25:03.928 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2024-05-09T08:25:03.984 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T08:25:03.984 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:25:03.984 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-09T08:25:03.984 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:25:03.984 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:25:03.984 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-05-09 08:24:48.221117274 +0000 2024-05-09T08:25:03.984 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-05-09 08:19:34.708413432 +0000 2024-05-09T08:25:03.984 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-05-09 08:19:34.708413432 +0000 2024-05-09T08:25:03.984 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-05-09 08:19:34.708413432 +0000 2024-05-09T08:25:03.985 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T08:25:04.050 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-05-09T08:25:04.050 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-05-09T08:25:04.050 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000119869 s, 4.3 MB/s 2024-05-09T08:25:04.052 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T08:25:04.108 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2024-05-09T08:25:04.164 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T08:25:04.165 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:25:04.165 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-09T08:25:04.165 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:25:04.165 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:25:04.165 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-05-09 08:24:48.222117259 +0000 2024-05-09T08:25:04.165 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-05-09 08:19:35.048408332 +0000 2024-05-09T08:25:04.165 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-05-09 08:19:35.048408332 +0000 2024-05-09T08:25:04.165 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-05-09 08:19:35.048408332 +0000 2024-05-09T08:25:04.165 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T08:25:04.228 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-05-09T08:25:04.229 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-05-09T08:25:04.229 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 9.1598e-05 s, 5.6 MB/s 2024-05-09T08:25:04.230 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T08:25:04.287 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-05-09T08:25:04.288 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T08:25:04.304 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T08:25:04.304 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_1 2024-05-09T08:25:04.360 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T08:25:04.360 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:25:04.360 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-05-09T08:25:04.360 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:25:04.360 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:25:04.360 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-05-09 08:24:38.807081208 +0000 2024-05-09T08:25:04.361 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-05-09 08:19:22.724895480 +0000 2024-05-09T08:25:04.361 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-05-09 08:19:22.724895480 +0000 2024-05-09T08:25:04.361 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-05-09 08:19:22.724895480 +0000 2024-05-09T08:25:04.361 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T08:25:04.425 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-05-09T08:25:04.425 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-05-09T08:25:04.425 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000112963 s, 4.5 MB/s 2024-05-09T08:25:04.426 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T08:25:04.484 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_2 2024-05-09T08:25:04.540 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T08:25:04.540 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:25:04.541 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-05-09T08:25:04.541 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:25:04.541 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:25:04.541 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-05-09 08:24:38.808081189 +0000 2024-05-09T08:25:04.541 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-05-09 08:19:23.065889038 +0000 2024-05-09T08:25:04.541 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-05-09 08:19:23.065889038 +0000 2024-05-09T08:25:04.541 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-05-09 08:19:23.065889038 +0000 2024-05-09T08:25:04.541 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T08:25:04.605 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-05-09T08:25:04.605 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-05-09T08:25:04.605 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 9.0789e-05 s, 5.6 MB/s 2024-05-09T08:25:04.606 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T08:25:04.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:04 smithi110 ceph-mon[33622]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:04.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:04 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:04.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:04 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:04.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:04 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:04.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:04 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:25:04.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:04 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:25:04.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:04 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:25:04.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:04 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:04.662 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_3 2024-05-09T08:25:04.718 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T08:25:04.718 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:25:04.718 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-09T08:25:04.718 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:25:04.718 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:25:04.718 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-05-09 08:24:38.808081189 +0000 2024-05-09T08:25:04.719 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-05-09 08:19:23.399882729 +0000 2024-05-09T08:25:04.719 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-05-09 08:19:23.399882729 +0000 2024-05-09T08:25:04.719 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-05-09 08:19:23.399882729 +0000 2024-05-09T08:25:04.719 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T08:25:04.720 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:04 smithi157 ceph-mon[33614]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:04.720 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:04 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:04.720 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:04 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:04.720 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:04 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:04.720 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:04 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:25:04.720 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:04 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:25:04.720 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:04 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:25:04.720 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:04 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:04.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:04 smithi107 ceph-mon[30122]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:04.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:04 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:04.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:04 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:04.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:04 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:04.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:04 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:25:04.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:04 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:25:04.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:04 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:25:04.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:04 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:04.783 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-05-09T08:25:04.783 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-05-09T08:25:04.783 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000109199 s, 4.7 MB/s 2024-05-09T08:25:04.784 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T08:25:04.841 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_4 2024-05-09T08:25:04.899 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T08:25:04.899 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T08:25:04.899 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-09T08:25:04.899 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:25:04.899 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T08:25:04.899 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-05-09 08:24:38.808081189 +0000 2024-05-09T08:25:04.899 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-05-09 08:19:23.727876534 +0000 2024-05-09T08:25:04.899 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-05-09 08:19:23.727876534 +0000 2024-05-09T08:25:04.899 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-05-09 08:19:23.727876534 +0000 2024-05-09T08:25:04.899 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T08:25:04.963 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-05-09T08:25:04.963 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-05-09T08:25:04.963 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.00010944 s, 4.7 MB/s 2024-05-09T08:25:04.964 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T08:25:05.023 INFO:tasks.cephadm:Deploying osd.0 on smithi107 with /dev/vg_nvme/lv_4... 2024-05-09T08:25:05.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-09T08:25:05.268 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:05.706 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:05 smithi107 ceph-mon[30122]: Deploying daemon mgr.z on smithi157 2024-05-09T08:25:05.795 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:05 smithi110 ceph-mon[33622]: Deploying daemon mgr.z on smithi157 2024-05-09T08:25:05.821 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:05 smithi157 ceph-mon[33614]: Deploying daemon mgr.z on smithi157 2024-05-09T08:25:06.095 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:05 smithi157 systemd[1]: Starting Ceph mgr.z for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:25:06.490 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:06 smithi107 ceph-mon[30122]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:06.494 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:06 smithi157 ceph-mon[33614]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:06.494 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:06 smithi157 podman[35460]: 2024-05-09 08:25:06.138855983 +0000 UTC m=+0.018003983 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:25:06.494 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:06 smithi157 podman[35460]: 2024-05-09 08:25:06.236611714 +0000 UTC m=+0.115759715 container create e1a8727f5c6ddc6a0b3290e089328b8b100f8a60eb7420658a0360da7287aa6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:25:06.494 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:06 smithi157 podman[35460]: 2024-05-09 08:25:06.463783576 +0000 UTC m=+0.342931577 container init e1a8727f5c6ddc6a0b3290e089328b8b100f8a60eb7420658a0360da7287aa6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-09T08:25:06.494 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:06 smithi157 podman[35460]: 2024-05-09 08:25:06.467869927 +0000 UTC m=+0.347017928 container start e1a8727f5c6ddc6a0b3290e089328b8b100f8a60eb7420658a0360da7287aa6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:25:06.494 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:06 smithi157 ceph-mgr[35480]: -- 172.21.15.157:0/1323268502 <== mon.1 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c5bcf1c5a0 con 0x55c5bcef0800 2024-05-09T08:25:06.726 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:06 smithi110 ceph-mon[33622]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:06.790 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:06 smithi157 bash[35460]: e1a8727f5c6ddc6a0b3290e089328b8b100f8a60eb7420658a0360da7287aa6c 2024-05-09T08:25:06.790 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:06 smithi157 systemd[1]: Started Ceph mgr.z for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:25:06.790 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:06.624+0000 7f32d8f4a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:25:07.056 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-09T08:25:07.056 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-09T08:25:07.056 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-05-09T08:25:07.056 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-05-09T08:25:07.056 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113835 s, 921 MB/s 2024-05-09T08:25:07.056 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-05-09T08:25:07.094 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:06.819+0000 7f32d8f4a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:25:07.095 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:06.928+0000 7f32d8f4a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:25:07.574 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:07 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:07.247+0000 7f32d8f4a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:25:07.575 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:07 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:07.334+0000 7f32d8f4a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:25:07.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi107:vg_nvme/lv_4 2024-05-09T08:25:07.660 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:07 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:07.660 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:07 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:07.660 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:07 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:07.660 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:07 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:25:07.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:07 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:25:07.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:07 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:25:07.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:07 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:07.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:07 smithi107 ceph-mon[30122]: Deploying daemon mgr.y on smithi110 2024-05-09T08:25:07.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:07 smithi107 ceph-mon[30122]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:07.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:07 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:07.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:07 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:07.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:07 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:07.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:07 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:25:07.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:07 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:25:07.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:07 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:25:07.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:07 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:07.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:07 smithi157 ceph-mon[33614]: Deploying daemon mgr.y on smithi110 2024-05-09T08:25:07.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:07 smithi157 ceph-mon[33614]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:07.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:07 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:07.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:07 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:07.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:07 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:07.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:07 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:25:07.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:07 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T08:25:07.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:07 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:25:07.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:07 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:07.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:07 smithi110 ceph-mon[33622]: Deploying daemon mgr.y on smithi110 2024-05-09T08:25:07.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:07 smithi110 ceph-mon[33622]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:07.874 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:25:08.813 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:08 smithi110 systemd[1]: Starting Ceph mgr.y for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:25:08.844 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:08 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:08.543+0000 7f32d8f4a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:25:08.844 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:08 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:08.713+0000 7f32d8f4a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:25:09.071 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:08 smithi110 podman[34994]: 2024-05-09 08:25:08.716096502 +0000 UTC m=+0.018348455 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:25:09.072 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:08 smithi110 podman[34994]: 2024-05-09 08:25:08.833597556 +0000 UTC m=+0.135849502 container create 4edfd29d7064e1bb1579856ce1bf9deebe63d502e0c4a154ad42f96502afd0ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-09T08:25:09.072 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:09 smithi110 podman[34994]: 2024-05-09 08:25:09.044420285 +0000 UTC m=+0.346672232 container init 4edfd29d7064e1bb1579856ce1bf9deebe63d502e0c4a154ad42f96502afd0ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-09T08:25:09.072 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:09 smithi110 podman[34994]: 2024-05-09 08:25:09.047179932 +0000 UTC m=+0.349431880 container start 4edfd29d7064e1bb1579856ce1bf9deebe63d502e0c4a154ad42f96502afd0ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-09T08:25:09.072 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:09 smithi110 ceph-mgr[35013]: -- 172.21.15.110:0/3158490323 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5579dfa265a0 con 0x5579df9fac00 2024-05-09T08:25:09.178 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:08 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:08.886+0000 7f32d8f4a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:25:09.179 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:09 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:09.177+0000 7f32d8f4a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:25:09.372 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:09 smithi110 bash[34994]: 4edfd29d7064e1bb1579856ce1bf9deebe63d502e0c4a154ad42f96502afd0ef 2024-05-09T08:25:09.372 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:09 smithi110 systemd[1]: Started Ceph mgr.y for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:25:09.372 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:09 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:09.206+0000 7f4af736e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:25:09.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.420+0000 7f3c43508700 1 -- 172.21.15.107:0/2031836923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c3c102370 msgr2=0x7f3c3c102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:09.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.420+0000 7f3c43508700 1 --2- 172.21.15.107:0/2031836923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c3c102370 0x7f3c3c102790 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3c380099c0 tx=0x7f3c38009cd0 comp rx=0 tx=0).stop 2024-05-09T08:25:09.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.420+0000 7f3c43508700 1 -- 172.21.15.107:0/2031836923 shutdown_connections 2024-05-09T08:25:09.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.421+0000 7f3c43508700 1 --2- 172.21.15.107:0/2031836923 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c3c103f10 0x7f3c3c10a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:09.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.421+0000 7f3c43508700 1 --2- 172.21.15.107:0/2031836923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c3c102370 0x7f3c3c102790 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:09.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.421+0000 7f3c43508700 1 --2- 172.21.15.107:0/2031836923 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c3c103570 0x7f3c3c1039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:09.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.421+0000 7f3c43508700 1 -- 172.21.15.107:0/2031836923 >> 172.21.15.107:0/2031836923 conn(0x7f3c3c0fd910 msgr2=0x7f3c3c0ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:09.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.421+0000 7f3c43508700 1 -- 172.21.15.107:0/2031836923 shutdown_connections 2024-05-09T08:25:09.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.421+0000 7f3c43508700 1 -- 172.21.15.107:0/2031836923 wait complete. 2024-05-09T08:25:09.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.421+0000 7f3c43508700 1 Processor -- start 2024-05-09T08:25:09.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.421+0000 7f3c43508700 1 -- start start 2024-05-09T08:25:09.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.421+0000 7f3c43508700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c3c102370 0x7f3c3c1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.421+0000 7f3c43508700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c3c103570 0x7f3c3c198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.421+0000 7f3c43508700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c3c103f10 0x7f3c3c19faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.421+0000 7f3c43508700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3c10de10 con 0x7f3c3c103570 2024-05-09T08:25:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.421+0000 7f3c43508700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3c10dc90 con 0x7f3c3c103f10 2024-05-09T08:25:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.421+0000 7f3c43508700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3c10df90 con 0x7f3c3c102370 2024-05-09T08:25:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c40aa3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c3c103570 0x7f3c3c198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c40aa3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c3c103570 0x7f3c3c198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55132/0 (socket says 172.21.15.107:55132) 2024-05-09T08:25:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c412a4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c3c102370 0x7f3c3c1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c40aa3700 1 -- 172.21.15.107:0/2877562109 learned_addr learned my addr 172.21.15.107:0/2877562109 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:25:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c41aa5700 1 --2- 172.21.15.107:0/2877562109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c3c103f10 0x7f3c3c19faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c40aa3700 1 -- 172.21.15.107:0/2877562109 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c3c102370 msgr2=0x7f3c3c1984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c40aa3700 1 --2- 172.21.15.107:0/2877562109 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c3c102370 0x7f3c3c1984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c40aa3700 1 -- 172.21.15.107:0/2877562109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c3c103f10 msgr2=0x7f3c3c19faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:09.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c40aa3700 1 --2- 172.21.15.107:0/2877562109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c3c103f10 0x7f3c3c19faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c40aa3700 1 -- 172.21.15.107:0/2877562109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c3c1a0140 con 0x7f3c3c103570 2024-05-09T08:25:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c41aa5700 1 --2- 172.21.15.107:0/2877562109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c3c103f10 0x7f3c3c19faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:25:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c40aa3700 1 --2- 172.21.15.107:0/2877562109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c3c103570 0x7f3c3c198a20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00e9f0 tx=0x7f3c2c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c327fc700 1 -- 172.21.15.107:0/2877562109 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c00cbd0 con 0x7f3c3c103570 2024-05-09T08:25:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c327fc700 1 -- 172.21.15.107:0/2877562109 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3c2c00ee90 con 0x7f3c3c103570 2024-05-09T08:25:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c43508700 1 -- 172.21.15.107:0/2877562109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c3c1a0430 con 0x7f3c3c103570 2024-05-09T08:25:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c327fc700 1 -- 172.21.15.107:0/2877562109 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c005590 con 0x7f3c3c103570 2024-05-09T08:25:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.422+0000 7f3c43508700 1 -- 172.21.15.107:0/2877562109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c3c1a0920 con 0x7f3c3c103570 2024-05-09T08:25:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.423+0000 7f3c327fc700 1 -- 172.21.15.107:0/2877562109 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f3c2c004830 con 0x7f3c3c103570 2024-05-09T08:25:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.423+0000 7f3c43508700 1 -- 172.21.15.107:0/2877562109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c3c069010 con 0x7f3c3c103570 2024-05-09T08:25:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.423+0000 7f3c327fc700 1 --2- 172.21.15.107:0/2877562109 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3c28039330 0x7f3c2803b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.423+0000 7f3c327fc700 1 -- 172.21.15.107:0/2877562109 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f3c2c04c5b0 con 0x7f3c3c103570 2024-05-09T08:25:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.423+0000 7f3c412a4700 1 --2- 172.21.15.107:0/2877562109 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3c28039330 0x7f3c2803b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:09.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.424+0000 7f3c412a4700 1 --2- 172.21.15.107:0/2877562109 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3c28039330 0x7f3c2803b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3c3800b580 tx=0x7f3c38005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.428+0000 7f3c327fc700 1 -- 172.21.15.107:0/2877562109 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3c2c00b2c0 con 0x7f3c3c103570 2024-05-09T08:25:09.488 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:09 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:09.272+0000 7f32d8f4a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:25:09.488 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:09 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:09.488+0000 7f32d8f4a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:25:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:09.635+0000 7f3c43508700 1 -- 172.21.15.107:0/2877562109 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f3c3c062760 con 0x7f3c28039330 2024-05-09T08:25:09.833 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:09 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:09.395+0000 7f4af736e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:25:09.833 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:09 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:09.507+0000 7f4af736e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:25:09.844 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:09 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:09.688+0000 7f32d8f4a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:25:10.121 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:09 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:09.832+0000 7f4af736e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:25:10.122 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:09 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:09.926+0000 7f4af736e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:25:10.502 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:10 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:10.502 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:10 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:10.502 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:10 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:10.502 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:10 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:10.503 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:10 smithi107 ceph-mon[30122]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:10.503 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:10 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:10.503 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:10 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:25:10.503 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:10 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:25:10.503 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:10 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:10.595 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:10 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:10.335+0000 7f32d8f4a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:25:10.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:10 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:10.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:10 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:10.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:10 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:10.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:10 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:10.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:10 smithi157 ceph-mon[33614]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:10.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:10 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:10.596 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:10 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:25:10.596 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:10 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:25:10.596 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:10 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:10.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:10 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:10.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:10 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:10.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:10 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:10.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:10 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:10.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:10 smithi110 ceph-mon[33622]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:10.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:10 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:10.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:10 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:25:10.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:10 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:25:10.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:10 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:11.079 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:10 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:10.778+0000 7f32d8f4a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:25:11.079 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:10 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:10.876+0000 7f32d8f4a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:25:11.079 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:11 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:11.079+0000 7f32d8f4a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:25:11.432 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:11 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:11.170+0000 7f4af736e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:25:11.432 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:11 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:11.341+0000 7f4af736e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:25:11.504 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:11 smithi107 ceph-mon[30122]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:11.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:11.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:11.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:25:11.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:25:11.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:11.698 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:11 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:11.605+0000 7f32d8f4a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:25:11.698 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:11 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:11.698+0000 7f32d8f4a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:25:11.698 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:11 smithi157 ceph-mon[33614]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:11.698 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.698 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.698 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.699 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.699 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.699 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:11.699 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:11.699 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.699 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:25:11.699 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:25:11.699 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:11.810 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:11 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:11.522+0000 7f4af736e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:25:11.810 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:11 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:11.808+0000 7f4af736e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:25:11.810 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:11 smithi110 ceph-mon[33622]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:11.810 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.810 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.810 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.810 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.810 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.811 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:11.811 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:11.811 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:11.811 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T08:25:11.811 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T08:25:11.811 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:12.075 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:11 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:11.942+0000 7f32d8f4a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:25:12.075 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:12 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:12.074+0000 7f32d8f4a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:25:12.103 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:11 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:11.900+0000 7f4af736e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:25:12.103 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:12 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:12.102+0000 7f4af736e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:25:12.344 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:12 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:12.170+0000 7f32d8f4a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:25:12.344 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:12 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:12.287+0000 7f32d8f4a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:25:12.371 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:12 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:12.295+0000 7f4af736e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:25:12.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:12 smithi107 ceph-mon[30122]: Reconfiguring mgr.x (unknown last config time)... 2024-05-09T08:25:12.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:12 smithi107 ceph-mon[30122]: Reconfiguring daemon mgr.x on smithi107 2024-05-09T08:25:12.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:12 smithi107 ceph-mon[30122]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:12.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:12 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/94326238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ab0acea-9500-4d88-ab8e-2d148374937b"}]: dispatch 2024-05-09T08:25:12.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:12 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/94326238' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ab0acea-9500-4d88-ab8e-2d148374937b"}]': finished 2024-05-09T08:25:12.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:12 smithi107 ceph-mon[30122]: osdmap e5: 1 total, 0 up, 1 in 2024-05-09T08:25:12.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:12 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:12.845 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:12 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:12.593+0000 7f32d8f4a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:25:12.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:12 smithi157 ceph-mon[33614]: Reconfiguring mgr.x (unknown last config time)... 2024-05-09T08:25:12.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:12 smithi157 ceph-mon[33614]: Reconfiguring daemon mgr.x on smithi107 2024-05-09T08:25:12.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:12 smithi157 ceph-mon[33614]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:12.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:12 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/94326238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ab0acea-9500-4d88-ab8e-2d148374937b"}]: dispatch 2024-05-09T08:25:12.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:12 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/94326238' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ab0acea-9500-4d88-ab8e-2d148374937b"}]': finished 2024-05-09T08:25:12.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:12 smithi157 ceph-mon[33614]: osdmap e5: 1 total, 0 up, 1 in 2024-05-09T08:25:12.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:12 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:12.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:12 smithi110 ceph-mon[33622]: Reconfiguring mgr.x (unknown last config time)... 2024-05-09T08:25:12.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:12 smithi110 ceph-mon[33622]: Reconfiguring daemon mgr.x on smithi107 2024-05-09T08:25:12.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:12 smithi110 ceph-mon[33622]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:12.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:12 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/94326238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ab0acea-9500-4d88-ab8e-2d148374937b"}]: dispatch 2024-05-09T08:25:12.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:12 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/94326238' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ab0acea-9500-4d88-ab8e-2d148374937b"}]': finished 2024-05-09T08:25:12.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:12 smithi110 ceph-mon[33622]: osdmap e5: 1 total, 0 up, 1 in 2024-05-09T08:25:12.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:12 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:13.317 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:12 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:12.897+0000 7f4af736e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:25:13.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:13.441+0000 7f3c327fc700 1 -- 172.21.15.107:0/2877562109 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 14) ==== 92061+0+0 (secure 0 0 0) 0x7f3c2c0203a0 con 0x7f3c3c103570 2024-05-09T08:25:13.595 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:13 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:13.194+0000 7f32d8f4a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:25:13.595 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:25:13 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:25:13.409+0000 7f32d8f4a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:25:13.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:13 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/375965624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:25:13.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:13 smithi157 ceph-mon[33614]: Standby manager daemon z started 2024-05-09T08:25:13.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:13 smithi157 ceph-mon[33614]: from='mgr.? 172.21.15.157:0/1009680545' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:25:13.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:13 smithi157 ceph-mon[33614]: from='mgr.? 172.21.15.157:0/1009680545' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:25:13.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:13 smithi157 ceph-mon[33614]: from='mgr.? 172.21.15.157:0/1009680545' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:25:13.596 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:13 smithi157 ceph-mon[33614]: from='mgr.? 172.21.15.157:0/1009680545' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:25:13.615 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:13 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:13.316+0000 7f4af736e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:25:13.615 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:13 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:13.412+0000 7f4af736e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:25:13.615 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:13 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:13.614+0000 7f4af736e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:25:13.616 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:13 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/375965624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:25:13.616 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:13 smithi110 ceph-mon[33622]: Standby manager daemon z started 2024-05-09T08:25:13.616 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:13 smithi110 ceph-mon[33622]: from='mgr.? 172.21.15.157:0/1009680545' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:25:13.616 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:13 smithi110 ceph-mon[33622]: from='mgr.? 172.21.15.157:0/1009680545' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:25:13.616 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:13 smithi110 ceph-mon[33622]: from='mgr.? 172.21.15.157:0/1009680545' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:25:13.616 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:13 smithi110 ceph-mon[33622]: from='mgr.? 172.21.15.157:0/1009680545' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:25:13.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:13 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/375965624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:25:13.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:13 smithi107 ceph-mon[30122]: Standby manager daemon z started 2024-05-09T08:25:13.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:13 smithi107 ceph-mon[30122]: from='mgr.? 172.21.15.157:0/1009680545' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:25:13.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:13 smithi107 ceph-mon[30122]: from='mgr.? 172.21.15.157:0/1009680545' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:25:13.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:13 smithi107 ceph-mon[30122]: from='mgr.? 172.21.15.157:0/1009680545' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:25:13.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:13 smithi107 ceph-mon[30122]: from='mgr.? 172.21.15.157:0/1009680545' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:25:14.439 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:14 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:14.143+0000 7f4af736e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:25:14.439 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:14 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:14.234+0000 7f4af736e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:25:14.705 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:14 smithi110 ceph-mon[33622]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:14.705 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:14 smithi110 ceph-mon[33622]: mgrmap e14: x(active, since 78s), standbys: z 2024-05-09T08:25:14.705 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:14 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:25:14.705 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:14 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:14.477+0000 7f4af736e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:25:14.705 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:14 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:14.609+0000 7f4af736e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:25:14.705 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:14 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:14.703+0000 7f4af736e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:25:14.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:14 smithi107 ceph-mon[30122]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:14.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:14 smithi107 ceph-mon[30122]: mgrmap e14: x(active, since 78s), standbys: z 2024-05-09T08:25:14.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:14 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:25:14.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:14 smithi157 ceph-mon[33614]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:14.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:14 smithi157 ceph-mon[33614]: mgrmap e14: x(active, since 78s), standbys: z 2024-05-09T08:25:14.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:14 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:25:15.121 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:14 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:14.818+0000 7f4af736e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:25:15.621 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:15 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:15.136+0000 7f4af736e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:25:16.122 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:15 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:15.733+0000 7f4af736e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:25:16.122 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:25:15 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:25:15.949+0000 7f4af736e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:25:16.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:16.451+0000 7f3c327fc700 1 -- 172.21.15.107:0/2877562109 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f3c2c05f430 con 0x7f3c3c103570 2024-05-09T08:25:16.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:16 smithi110 ceph-mon[33622]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:16.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:16 smithi110 ceph-mon[33622]: Standby manager daemon y started 2024-05-09T08:25:16.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:16 smithi110 ceph-mon[33622]: from='mgr.? 172.21.15.110:0/1969973596' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:25:16.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:16 smithi110 ceph-mon[33622]: from='mgr.? 172.21.15.110:0/1969973596' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:25:16.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:16 smithi110 ceph-mon[33622]: from='mgr.? 172.21.15.110:0/1969973596' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:25:16.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:16 smithi110 ceph-mon[33622]: from='mgr.? 172.21.15.110:0/1969973596' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:25:16.663 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:16 smithi107 ceph-mon[30122]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:16.663 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:16 smithi107 ceph-mon[30122]: Standby manager daemon y started 2024-05-09T08:25:16.663 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:16 smithi107 ceph-mon[30122]: from='mgr.? 172.21.15.110:0/1969973596' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:25:16.663 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:16 smithi107 ceph-mon[30122]: from='mgr.? 172.21.15.110:0/1969973596' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:25:16.663 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:16 smithi107 ceph-mon[30122]: from='mgr.? 172.21.15.110:0/1969973596' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:25:16.663 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:16 smithi107 ceph-mon[30122]: from='mgr.? 172.21.15.110:0/1969973596' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:25:16.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:16 smithi157 ceph-mon[33614]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:16.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:16 smithi157 ceph-mon[33614]: Standby manager daemon y started 2024-05-09T08:25:16.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:16 smithi157 ceph-mon[33614]: from='mgr.? 172.21.15.110:0/1969973596' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:25:16.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:16 smithi157 ceph-mon[33614]: from='mgr.? 172.21.15.110:0/1969973596' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:25:16.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:16 smithi157 ceph-mon[33614]: from='mgr.? 172.21.15.110:0/1969973596' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:25:16.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:16 smithi157 ceph-mon[33614]: from='mgr.? 172.21.15.110:0/1969973596' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:25:17.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:17 smithi107 ceph-mon[30122]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-05-09T08:25:17.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:17 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:25:17.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:17 smithi157 ceph-mon[33614]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-05-09T08:25:17.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:17 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:25:17.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:17 smithi110 ceph-mon[33622]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-05-09T08:25:17.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:17 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:25:18.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:18 smithi107 ceph-mon[30122]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:18.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:18 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:18.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:18 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:18.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:18 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:18.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:18 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:18.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:18 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:18.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:18 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:18.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:18 smithi157 ceph-mon[33614]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:18.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:18 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:18.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:18 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:18.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:18 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:18.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:18 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:18.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:18 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:18.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:18 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:18.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:18 smithi110 ceph-mon[33622]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:18.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:18 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:18.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:18 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:18.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:18 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:18.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:18 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:18.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:18 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:18.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:18 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:20.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:20 smithi107 ceph-mon[30122]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:20.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-09T08:25:20.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:20.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:20 smithi157 ceph-mon[33614]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:20.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-09T08:25:20.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:20.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:20 smithi110 ceph-mon[33622]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:20.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-09T08:25:20.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:21.733 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:21 smithi107 ceph-mon[30122]: Deploying daemon osd.0 on smithi107 2024-05-09T08:25:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:21 smithi157 ceph-mon[33614]: Deploying daemon osd.0 on smithi107 2024-05-09T08:25:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:21 smithi110 ceph-mon[33622]: Deploying daemon osd.0 on smithi107 2024-05-09T08:25:22.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:22 smithi107 ceph-mon[30122]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:22.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:22 smithi157 ceph-mon[33614]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:22.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:22 smithi110 ceph-mon[33622]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:24.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:24 smithi107 ceph-mon[30122]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:24.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:24 smithi157 ceph-mon[33614]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:24.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:24 smithi110 ceph-mon[33622]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:26.570 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:26 smithi107 ceph-mon[30122]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:26.570 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:26.570 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:26.570 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:26 smithi157 ceph-mon[33614]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:26.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:26.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:26 smithi110 ceph-mon[33622]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:26.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:26.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:26.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:28.446+0000 7f3c327fc700 1 -- 172.21.15.107:0/2877562109 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3c3c062760 con 0x7f3c28039330 2024-05-09T08:25:28.447 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 0 on host 'smithi107' 2024-05-09T08:25:28.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:28.449+0000 7f3c43508700 1 -- 172.21.15.107:0/2877562109 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3c28039330 msgr2=0x7f3c2803b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:28.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:28.449+0000 7f3c43508700 1 --2- 172.21.15.107:0/2877562109 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3c28039330 0x7f3c2803b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3c3800b580 tx=0x7f3c38005fb0 comp rx=0 tx=0).stop 2024-05-09T08:25:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:28.449+0000 7f3c43508700 1 -- 172.21.15.107:0/2877562109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c3c103570 msgr2=0x7f3c3c198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:28.449+0000 7f3c43508700 1 --2- 172.21.15.107:0/2877562109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c3c103570 0x7f3c3c198a20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00e9f0 tx=0x7f3c2c00ed00 comp rx=0 tx=0).stop 2024-05-09T08:25:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:28.449+0000 7f3c43508700 1 -- 172.21.15.107:0/2877562109 shutdown_connections 2024-05-09T08:25:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:28.449+0000 7f3c43508700 1 --2- 172.21.15.107:0/2877562109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c3c103f10 0x7f3c3c19faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:28.449+0000 7f3c43508700 1 --2- 172.21.15.107:0/2877562109 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c3c102370 0x7f3c3c1984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:28.449+0000 7f3c43508700 1 --2- 172.21.15.107:0/2877562109 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3c28039330 0x7f3c2803b7f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:28.449+0000 7f3c43508700 1 --2- 172.21.15.107:0/2877562109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c3c103570 0x7f3c3c198a20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:28.449+0000 7f3c43508700 1 -- 172.21.15.107:0/2877562109 >> 172.21.15.107:0/2877562109 conn(0x7f3c3c0fd910 msgr2=0x7f3c3c1069b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:28.449+0000 7f3c43508700 1 -- 172.21.15.107:0/2877562109 shutdown_connections 2024-05-09T08:25:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:25:28.449+0000 7f3c43508700 1 -- 172.21.15.107:0/2877562109 wait complete. 2024-05-09T08:25:28.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:28 smithi107 ceph-mon[30122]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:28.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:28 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:28.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:28 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:28.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:28 smithi157 ceph-mon[33614]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:28.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:28 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:28.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:28 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:28.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:28 smithi110 ceph-mon[33622]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:28.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:28 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:28.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:28 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:29.433 DEBUG:teuthology.orchestra.run.smithi107:osd.0> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.0.service 2024-05-09T08:25:29.475 INFO:tasks.cephadm:Deploying osd.1 on smithi110 with /dev/vg_nvme/lv_4... 2024-05-09T08:25:29.476 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-09T08:25:29.722 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:25:29.780 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:25:29 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-0[42154]: 2024-05-09T08:25:29.511+0000 7f82ee2a97c0 -1 osd.0 0 log_to_monitors true 2024-05-09T08:25:30.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:30 smithi107 ceph-mon[30122]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:30.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:30 smithi107 ceph-mon[30122]: from='osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-09T08:25:30.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:30 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:30.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:30 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:30.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:30 smithi157 ceph-mon[33614]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:30.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:30 smithi157 ceph-mon[33614]: from='osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-09T08:25:30.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:30 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:30.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:30 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:30.867 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:30 smithi110 ceph-mon[33622]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:30.867 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:30 smithi110 ceph-mon[33622]: from='osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-09T08:25:30.868 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:30 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:30.868 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:30 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:31.577 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-09T08:25:31.577 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-09T08:25:31.577 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-05-09T08:25:31.577 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-05-09T08:25:31.578 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00851268 s, 1.2 GB/s 2024-05-09T08:25:31.578 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-05-09T08:25:31.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:31 smithi107 ceph-mon[30122]: from='osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-09T08:25:31.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:31 smithi107 ceph-mon[30122]: osdmap e6: 1 total, 0 up, 1 in 2024-05-09T08:25:31.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:31 smithi107 ceph-mon[30122]: from='osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-05-09T08:25:31.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:31 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:31.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:31 smithi107 ceph-mon[30122]: from='osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-05-09T08:25:31.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:31 smithi157 ceph-mon[33614]: from='osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-09T08:25:31.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:31 smithi157 ceph-mon[33614]: osdmap e6: 1 total, 0 up, 1 in 2024-05-09T08:25:31.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:31 smithi157 ceph-mon[33614]: from='osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-05-09T08:25:31.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:31 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:31.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:31 smithi157 ceph-mon[33614]: from='osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-05-09T08:25:31.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:31 smithi110 ceph-mon[33622]: from='osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-09T08:25:31.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:31 smithi110 ceph-mon[33622]: osdmap e6: 1 total, 0 up, 1 in 2024-05-09T08:25:31.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:31 smithi110 ceph-mon[33622]: from='osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-05-09T08:25:31.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:31 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:31.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:31 smithi110 ceph-mon[33622]: from='osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-05-09T08:25:32.246 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi110:vg_nvme/lv_4 2024-05-09T08:25:32.488 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:25:32.522 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:25:32 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-0[42154]: 2024-05-09T08:25:32.201+0000 7f82e390d700 -1 osd.0 0 waiting for initial osdmap 2024-05-09T08:25:32.522 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:25:32 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-0[42154]: 2024-05-09T08:25:32.206+0000 7f82db6f8700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:25:32.774 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:32 smithi110 ceph-mon[33622]: purged_snaps scrub starts 2024-05-09T08:25:32.775 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:32 smithi110 ceph-mon[33622]: purged_snaps scrub ok 2024-05-09T08:25:32.775 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:32 smithi110 ceph-mon[33622]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:32.775 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:32 smithi110 ceph-mon[33622]: osdmap e7: 1 total, 0 up, 1 in 2024-05-09T08:25:32.775 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:32.775 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:32.775 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:32 smithi110 ceph-mon[33622]: from='osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349]' entity='osd.0' 2024-05-09T08:25:32.775 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:32.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:32 smithi107 ceph-mon[30122]: purged_snaps scrub starts 2024-05-09T08:25:32.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:32 smithi107 ceph-mon[30122]: purged_snaps scrub ok 2024-05-09T08:25:32.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:32 smithi107 ceph-mon[30122]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:32.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:32 smithi107 ceph-mon[30122]: osdmap e7: 1 total, 0 up, 1 in 2024-05-09T08:25:32.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:32.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:32.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:32 smithi107 ceph-mon[30122]: from='osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349]' entity='osd.0' 2024-05-09T08:25:32.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:32.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:32 smithi157 ceph-mon[33614]: purged_snaps scrub starts 2024-05-09T08:25:32.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:32 smithi157 ceph-mon[33614]: purged_snaps scrub ok 2024-05-09T08:25:32.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:32 smithi157 ceph-mon[33614]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:32.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:32 smithi157 ceph-mon[33614]: osdmap e7: 1 total, 0 up, 1 in 2024-05-09T08:25:32.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:32.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:32.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:32 smithi157 ceph-mon[33614]: from='osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349]' entity='osd.0' 2024-05-09T08:25:32.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:33.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.985+0000 7f664bbb6700 1 -- 172.21.15.110:0/2784281880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6644102e10 msgr2=0x7f6644105200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:33.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.985+0000 7f664bbb6700 1 --2- 172.21.15.110:0/2784281880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6644102e10 0x7f6644105200 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6634009990 tx=0x7f6634009ca0 comp rx=0 tx=0).stop 2024-05-09T08:25:33.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.985+0000 7f664bbb6700 1 -- 172.21.15.110:0/2784281880 shutdown_connections 2024-05-09T08:25:33.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.985+0000 7f664bbb6700 1 --2- 172.21.15.110:0/2784281880 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6644108070 0x7f664410a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:33.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.985+0000 7f664bbb6700 1 --2- 172.21.15.110:0/2784281880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6644102e10 0x7f6644105200 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:33.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.985+0000 7f664bbb6700 1 --2- 172.21.15.110:0/2784281880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6644105740 0x7f6644107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:33.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.985+0000 7f664bbb6700 1 -- 172.21.15.110:0/2784281880 >> 172.21.15.110:0/2784281880 conn(0x7f66440fc820 msgr2=0x7f66440fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:33.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.985+0000 7f664bbb6700 1 -- 172.21.15.110:0/2784281880 shutdown_connections 2024-05-09T08:25:33.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.985+0000 7f664bbb6700 1 -- 172.21.15.110:0/2784281880 wait complete. 2024-05-09T08:25:33.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: Detected new or changed devices on smithi107 2024-05-09T08:25:33.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:33.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:33.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:25:33.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi107 to 17088M 2024-05-09T08:25:33.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:33.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:33.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:33.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:33.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:33.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349] boot 2024-05-09T08:25:33.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: osdmap e8: 1 total, 1 up, 1 in 2024-05-09T08:25:33.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:33.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:33.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:33.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:33.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.986+0000 7f664bbb6700 1 Processor -- start 2024-05-09T08:25:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.986+0000 7f664bbb6700 1 -- start start 2024-05-09T08:25:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.986+0000 7f664bbb6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6644102e10 0x7f66441984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.986+0000 7f664bbb6700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6644105740 0x7f66441989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.986+0000 7f664bbb6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6644108070 0x7f664419fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.986+0000 7f664bbb6700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f664410de90 con 0x7f6644102e10 2024-05-09T08:25:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.986+0000 7f664bbb6700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f664410dd10 con 0x7f6644108070 2024-05-09T08:25:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.986+0000 7f664bbb6700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f664410e010 con 0x7f6644105740 2024-05-09T08:25:33.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.986+0000 7f664a153700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6644108070 0x7f664419fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:33.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.986+0000 7f664a153700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6644108070 0x7f664419fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:49408/0 (socket says 172.21.15.110:49408) 2024-05-09T08:25:33.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.986+0000 7f664a153700 1 -- 172.21.15.110:0/2140550613 learned_addr learned my addr 172.21.15.110:0/2140550613 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-05-09T08:25:33.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.986+0000 7f6649952700 1 --2- 172.21.15.110:0/2140550613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6644102e10 0x7f66441984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:33.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.986+0000 7f6649151700 1 --2- 172.21.15.110:0/2140550613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6644105740 0x7f66441989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:33.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.987+0000 7f664a153700 1 -- 172.21.15.110:0/2140550613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6644105740 msgr2=0x7f66441989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:33.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.987+0000 7f664a153700 1 --2- 172.21.15.110:0/2140550613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6644105740 0x7f66441989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:33.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.987+0000 7f664a153700 1 -- 172.21.15.110:0/2140550613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6644102e10 msgr2=0x7f66441984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:33.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.987+0000 7f664a153700 1 --2- 172.21.15.110:0/2140550613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6644102e10 0x7f66441984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:33.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.987+0000 7f664a153700 1 -- 172.21.15.110:0/2140550613 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66441a0110 con 0x7f6644108070 2024-05-09T08:25:33.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.987+0000 7f6649952700 1 --2- 172.21.15.110:0/2140550613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6644102e10 0x7f66441984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:25:33.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.987+0000 7f664a153700 1 --2- 172.21.15.110:0/2140550613 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6644108070 0x7f664419fa70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f663c0094f0 tx=0x7f663c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:33.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.987+0000 7f663affd700 1 -- 172.21.15.110:0/2140550613 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c004db0 con 0x7f6644108070 2024-05-09T08:25:33.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.987+0000 7f664bbb6700 1 -- 172.21.15.110:0/2140550613 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66441a0400 con 0x7f6644108070 2024-05-09T08:25:33.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.987+0000 7f663affd700 1 -- 172.21.15.110:0/2140550613 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f663c009e20 con 0x7f6644108070 2024-05-09T08:25:33.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.987+0000 7f663affd700 1 -- 172.21.15.110:0/2140550613 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c015400 con 0x7f6644108070 2024-05-09T08:25:33.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.987+0000 7f664bbb6700 1 -- 172.21.15.110:0/2140550613 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66441a0940 con 0x7f6644108070 2024-05-09T08:25:33.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.988+0000 7f664bbb6700 1 -- 172.21.15.110:0/2140550613 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6644069010 con 0x7f6644108070 2024-05-09T08:25:33.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.989+0000 7f663affd700 1 -- 172.21.15.110:0/2140550613 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f663c00db20 con 0x7f6644108070 2024-05-09T08:25:33.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.990+0000 7f663affd700 1 --2- 172.21.15.110:0/2140550613 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f66300a2b90 0x7f66300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:33.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.990+0000 7f663affd700 1 -- 172.21.15.110:0/2140550613 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1767+0+0 (secure 0 0 0) 0x7f663c0cd6f0 con 0x7f6644108070 2024-05-09T08:25:33.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.990+0000 7f6649952700 1 --2- 172.21.15.110:0/2140550613 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f66300a2b90 0x7f66300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:33.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.991+0000 7f6649952700 1 --2- 172.21.15.110:0/2140550613 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f66300a2b90 0x7f66300a5050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f663400b5c0 tx=0x7f6634005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:33.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:33.994+0000 7f663affd700 1 -- 172.21.15.110:0/2140550613 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f663c010070 con 0x7f6644108070 2024-05-09T08:25:34.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:34.196+0000 7f664bbb6700 1 -- 172.21.15.110:0/2140550613 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6644062760 con 0x7f66300a2b90 2024-05-09T08:25:34.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: Detected new or changed devices on smithi107 2024-05-09T08:25:34.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:34.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:34.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:25:34.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi107 to 17088M 2024-05-09T08:25:34.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:34.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:34.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:34.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:34.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:34.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349] boot 2024-05-09T08:25:34.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: osdmap e8: 1 total, 1 up, 1 in 2024-05-09T08:25:34.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:34.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:34.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:34.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:34.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:34.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: Detected new or changed devices on smithi107 2024-05-09T08:25:34.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:34.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:34.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:25:34.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi107 to 17088M 2024-05-09T08:25:34.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:34.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:34.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:34.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:34.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:34.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: osd.0 [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349] boot 2024-05-09T08:25:34.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: osdmap e8: 1 total, 1 up, 1 in 2024-05-09T08:25:34.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:25:34.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T08:25:34.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:34.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:34.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:35.115 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:34 smithi110 ceph-mon[33622]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:35.115 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:34 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:25:35.115 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:34 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:25:35.115 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:34 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:35.115 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:34 smithi110 ceph-mon[33622]: osdmap e9: 1 total, 1 up, 1 in 2024-05-09T08:25:35.247 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:34 smithi157 ceph-mon[33614]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:35.247 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:34 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:25:35.247 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:34 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:25:35.247 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:34 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:35.247 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:34 smithi157 ceph-mon[33614]: osdmap e9: 1 total, 1 up, 1 in 2024-05-09T08:25:35.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:34 smithi107 ceph-mon[30122]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:35.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:34 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:25:35.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:34 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:25:35.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:34 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:35.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:34 smithi107 ceph-mon[30122]: osdmap e9: 1 total, 1 up, 1 in 2024-05-09T08:25:35.977 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:35 smithi110 ceph-mon[33622]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:36.262 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:35 smithi157 ceph-mon[33614]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:36.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:35 smithi107 ceph-mon[30122]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:37.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:36 smithi107 ceph-mon[30122]: from='client.? 172.21.15.110:0/939622174' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93653939-b195-46f0-a8a9-b25ad2af5464"}]: dispatch 2024-05-09T08:25:37.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:36 smithi107 ceph-mon[30122]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93653939-b195-46f0-a8a9-b25ad2af5464"}]: dispatch 2024-05-09T08:25:37.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:36 smithi107 ceph-mon[30122]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93653939-b195-46f0-a8a9-b25ad2af5464"}]': finished 2024-05-09T08:25:37.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:36 smithi107 ceph-mon[30122]: osdmap e10: 2 total, 1 up, 2 in 2024-05-09T08:25:37.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:36 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:37.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:36 smithi157 ceph-mon[33614]: from='client.? 172.21.15.110:0/939622174' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93653939-b195-46f0-a8a9-b25ad2af5464"}]: dispatch 2024-05-09T08:25:37.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:36 smithi157 ceph-mon[33614]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93653939-b195-46f0-a8a9-b25ad2af5464"}]: dispatch 2024-05-09T08:25:37.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:36 smithi157 ceph-mon[33614]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93653939-b195-46f0-a8a9-b25ad2af5464"}]': finished 2024-05-09T08:25:37.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:36 smithi157 ceph-mon[33614]: osdmap e10: 2 total, 1 up, 2 in 2024-05-09T08:25:37.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:36 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:37.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:36 smithi110 ceph-mon[33622]: from='client.? 172.21.15.110:0/939622174' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93653939-b195-46f0-a8a9-b25ad2af5464"}]: dispatch 2024-05-09T08:25:37.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:36 smithi110 ceph-mon[33622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93653939-b195-46f0-a8a9-b25ad2af5464"}]: dispatch 2024-05-09T08:25:37.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:36 smithi110 ceph-mon[33622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93653939-b195-46f0-a8a9-b25ad2af5464"}]': finished 2024-05-09T08:25:37.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:36 smithi110 ceph-mon[33622]: osdmap e10: 2 total, 1 up, 2 in 2024-05-09T08:25:37.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:36 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:38.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:37 smithi107 ceph-mon[30122]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:38.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:37 smithi107 ceph-mon[30122]: from='client.? 172.21.15.110:0/3878078637' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:25:38.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:37 smithi157 ceph-mon[33614]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:38.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:37 smithi157 ceph-mon[33614]: from='client.? 172.21.15.110:0/3878078637' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:25:38.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:37 smithi110 ceph-mon[33622]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:38.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:37 smithi110 ceph-mon[33622]: from='client.? 172.21.15.110:0/3878078637' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:25:40.669 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:40 smithi110 ceph-mon[33622]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:40.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:40 smithi107 ceph-mon[30122]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:40.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:40 smithi157 ceph-mon[33614]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:42.725 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:42 smithi110 ceph-mon[33622]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:42.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:42 smithi107 ceph-mon[30122]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:42.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:42 smithi157 ceph-mon[33614]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:44.651 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:44 smithi110 ceph-mon[33622]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:44.651 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:44 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-09T08:25:44.652 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:44 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:44.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:44 smithi107 ceph-mon[30122]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:44.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-09T08:25:44.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:44.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:44 smithi157 ceph-mon[33614]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:44.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-09T08:25:44.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:45.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:45 smithi107 ceph-mon[30122]: Deploying daemon osd.1 on smithi110 2024-05-09T08:25:45.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:45 smithi157 ceph-mon[33614]: Deploying daemon osd.1 on smithi110 2024-05-09T08:25:45.852 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:45 smithi110 ceph-mon[33622]: Deploying daemon osd.1 on smithi110 2024-05-09T08:25:46.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:46 smithi107 ceph-mon[30122]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:46.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:46 smithi157 ceph-mon[33614]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:46.860 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:46 smithi110 ceph-mon[33622]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:48.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:48 smithi110 ceph-mon[33622]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:48.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:48 smithi107 ceph-mon[30122]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:48.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:48 smithi157 ceph-mon[33614]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:50.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:49 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:50.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:49 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:50.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:49 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:50.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:49 smithi110 ceph-mon[33622]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:50.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:49 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:50.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:49 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:50.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:49 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:50.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:49 smithi107 ceph-mon[30122]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:49 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:49 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:50.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:49 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:50.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:49 smithi157 ceph-mon[33614]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:52.423 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:52 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:52.423 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:52 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:52.423 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:52 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:52.423 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:52 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:52.424 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:52 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:52.424 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:52 smithi110 ceph-mon[33622]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:52.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:52.508+0000 7f663affd700 1 -- 172.21.15.110:0/2140550613 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6644062760 con 0x7f66300a2b90 2024-05-09T08:25:52.508 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 1 on host 'smithi110' 2024-05-09T08:25:52.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:52.510+0000 7f664bbb6700 1 -- 172.21.15.110:0/2140550613 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f66300a2b90 msgr2=0x7f66300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:52.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:52.510+0000 7f664bbb6700 1 --2- 172.21.15.110:0/2140550613 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f66300a2b90 0x7f66300a5050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f663400b5c0 tx=0x7f6634005fd0 comp rx=0 tx=0).stop 2024-05-09T08:25:52.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:52.510+0000 7f664bbb6700 1 -- 172.21.15.110:0/2140550613 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6644108070 msgr2=0x7f664419fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:52.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:52.510+0000 7f664bbb6700 1 --2- 172.21.15.110:0/2140550613 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6644108070 0x7f664419fa70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f663c0094f0 tx=0x7f663c00d3b0 comp rx=0 tx=0).stop 2024-05-09T08:25:52.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:52.510+0000 7f664bbb6700 1 -- 172.21.15.110:0/2140550613 shutdown_connections 2024-05-09T08:25:52.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:52.510+0000 7f664bbb6700 1 --2- 172.21.15.110:0/2140550613 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6644108070 0x7f664419fa70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:52.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:52.510+0000 7f664bbb6700 1 --2- 172.21.15.110:0/2140550613 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f66300a2b90 0x7f66300a5050 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:52.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:52.510+0000 7f664bbb6700 1 --2- 172.21.15.110:0/2140550613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6644102e10 0x7f66441984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:52.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:52.510+0000 7f664bbb6700 1 --2- 172.21.15.110:0/2140550613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6644105740 0x7f66441989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:52.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:52.510+0000 7f664bbb6700 1 -- 172.21.15.110:0/2140550613 >> 172.21.15.110:0/2140550613 conn(0x7f66440fc820 msgr2=0x7f6644104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:52.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:52.510+0000 7f664bbb6700 1 -- 172.21.15.110:0/2140550613 shutdown_connections 2024-05-09T08:25:52.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:25:52.510+0000 7f664bbb6700 1 -- 172.21.15.110:0/2140550613 wait complete. 2024-05-09T08:25:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:52 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:52 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:52 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:52 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:52 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:52 smithi107 ceph-mon[30122]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:52.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:52 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:52.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:52 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:52.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:52 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:52.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:52 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:52.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:52 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:52.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:52 smithi157 ceph-mon[33614]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:53.098 DEBUG:teuthology.orchestra.run.smithi110:osd.1> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.1.service 2024-05-09T08:25:53.100 INFO:tasks.cephadm:Deploying osd.2 on smithi157 with /dev/vg_nvme/lv_4... 2024-05-09T08:25:53.101 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-09T08:25:53.352 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:25:53.774 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:53 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:53.774 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:53 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:53.774 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:53 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:53.774 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:53 smithi157 ceph-mon[33614]: from='osd.1 [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:25:53.774 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:53 smithi157 ceph-mon[33614]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:25:53.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:53 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:53.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:53 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:53.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:53 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:53.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:53 smithi107 ceph-mon[30122]: from='osd.1 [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:25:53.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:53 smithi107 ceph-mon[30122]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:25:53.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:53 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:53.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:53 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:25:53.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:53 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:53.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:53 smithi110 ceph-mon[33622]: from='osd.1 [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:25:53.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:53 smithi110 ceph-mon[33622]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T08:25:54.536 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:54 smithi110 ceph-mon[33622]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:54.536 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:54 smithi110 ceph-mon[33622]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-09T08:25:54.536 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:54 smithi110 ceph-mon[33622]: from='osd.1 [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:25:54.536 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:54 smithi110 ceph-mon[33622]: osdmap e11: 2 total, 1 up, 2 in 2024-05-09T08:25:54.536 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:54 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:54.536 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:54 smithi110 ceph-mon[33622]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:25:54.536 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:25:54 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-1[38212]: 2024-05-09T08:25:54.478+0000 7efd0e406700 -1 osd.1 0 waiting for initial osdmap 2024-05-09T08:25:54.536 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:25:54 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-1[38212]: 2024-05-09T08:25:54.481+0000 7efd0a1f9700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:25:54.740 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:54 smithi157 ceph-mon[33614]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:54.740 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:54 smithi157 ceph-mon[33614]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-09T08:25:54.740 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:54 smithi157 ceph-mon[33614]: from='osd.1 [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:25:54.740 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:54 smithi157 ceph-mon[33614]: osdmap e11: 2 total, 1 up, 2 in 2024-05-09T08:25:54.740 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:54.740 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:54 smithi157 ceph-mon[33614]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:25:54.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:54 smithi107 ceph-mon[30122]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:54.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:54 smithi107 ceph-mon[30122]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-09T08:25:54.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:54 smithi107 ceph-mon[30122]: from='osd.1 [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:25:54.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:54 smithi107 ceph-mon[30122]: osdmap e11: 2 total, 1 up, 2 in 2024-05-09T08:25:54.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:54.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:54 smithi107 ceph-mon[30122]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:25:55.313 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-09T08:25:55.314 INFO:teuthology.orchestra.run.smithi157.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-09T08:25:55.314 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10+0 records in 2024-05-09T08:25:55.314 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records out 2024-05-09T08:25:55.314 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00831259 s, 1.3 GB/s 2024-05-09T08:25:55.314 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping successful for: 2024-05-09T08:25:55.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:55 smithi110 ceph-mon[33622]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-05-09T08:25:55.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:55 smithi110 ceph-mon[33622]: osdmap e12: 2 total, 1 up, 2 in 2024-05-09T08:25:55.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:55.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:55.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:55.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:55 smithi107 ceph-mon[30122]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-05-09T08:25:55.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:55 smithi107 ceph-mon[30122]: osdmap e12: 2 total, 1 up, 2 in 2024-05-09T08:25:55.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:55.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:55.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:55.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:55 smithi157 ceph-mon[33614]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-05-09T08:25:55.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:55 smithi157 ceph-mon[33614]: osdmap e12: 2 total, 1 up, 2 in 2024-05-09T08:25:55.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:55.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:55.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:55.935 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi157:vg_nvme/lv_4 2024-05-09T08:25:56.195 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:25:56.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:56 smithi157 ceph-mon[33614]: purged_snaps scrub starts 2024-05-09T08:25:56.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:56 smithi157 ceph-mon[33614]: purged_snaps scrub ok 2024-05-09T08:25:56.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:56 smithi157 ceph-mon[33614]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:56.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:56 smithi157 ceph-mon[33614]: osd.1 [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671] boot 2024-05-09T08:25:56.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:56 smithi157 ceph-mon[33614]: osdmap e13: 2 total, 2 up, 2 in 2024-05-09T08:25:56.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:56.596 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:56.596 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:56.596 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:25:56.596 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:56.596 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:56.596 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:56.596 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:56.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:56 smithi107 ceph-mon[30122]: purged_snaps scrub starts 2024-05-09T08:25:56.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:56 smithi107 ceph-mon[30122]: purged_snaps scrub ok 2024-05-09T08:25:56.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:56 smithi107 ceph-mon[30122]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:56.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:56 smithi107 ceph-mon[30122]: osd.1 [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671] boot 2024-05-09T08:25:56.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:56 smithi107 ceph-mon[30122]: osdmap e13: 2 total, 2 up, 2 in 2024-05-09T08:25:56.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:56.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:56.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:56.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:25:56.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:56.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:56.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:56.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:56 smithi110 ceph-mon[33622]: purged_snaps scrub starts 2024-05-09T08:25:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:56 smithi110 ceph-mon[33622]: purged_snaps scrub ok 2024-05-09T08:25:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:56 smithi110 ceph-mon[33622]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-09T08:25:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:56 smithi110 ceph-mon[33622]: osd.1 [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671] boot 2024-05-09T08:25:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:56 smithi110 ceph-mon[33622]: osdmap e13: 2 total, 2 up, 2 in 2024-05-09T08:25:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:25:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:25:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:25:56.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:25:57.748 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.745+0000 7fa7a3c1d700 1 -- 172.21.15.157:0/2714052209 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa79c1004c0 msgr2=0x7fa79c100940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:57.748 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.745+0000 7fa7a3c1d700 1 --2- 172.21.15.157:0/2714052209 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa79c1004c0 0x7fa79c100940 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa78c009990 tx=0x7fa78c009ca0 comp rx=0 tx=0).stop 2024-05-09T08:25:57.748 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.746+0000 7fa7a3c1d700 1 -- 172.21.15.157:0/2714052209 shutdown_connections 2024-05-09T08:25:57.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.746+0000 7fa7a3c1d700 1 --2- 172.21.15.157:0/2714052209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa79c100e80 0x7fa79c10aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:57.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.746+0000 7fa7a3c1d700 1 --2- 172.21.15.157:0/2714052209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa79c103a40 0x7fa79c103e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:57.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.746+0000 7fa7a3c1d700 1 --2- 172.21.15.157:0/2714052209 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa79c1004c0 0x7fa79c100940 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:57.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.746+0000 7fa7a3c1d700 1 -- 172.21.15.157:0/2714052209 >> 172.21.15.157:0/2714052209 conn(0x7fa79c0fc060 msgr2=0x7fa79c0fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:25:57.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.746+0000 7fa7a3c1d700 1 -- 172.21.15.157:0/2714052209 shutdown_connections 2024-05-09T08:25:57.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.746+0000 7fa7a3c1d700 1 -- 172.21.15.157:0/2714052209 wait complete. 2024-05-09T08:25:57.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.746+0000 7fa7a3c1d700 1 Processor -- start 2024-05-09T08:25:57.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.746+0000 7fa7a3c1d700 1 -- start start 2024-05-09T08:25:57.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.746+0000 7fa7a3c1d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa79c1004c0 0x7fa79c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:57.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.746+0000 7fa7a3c1d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa79c100e80 0x7fa79c198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:57.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.746+0000 7fa7a3c1d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa79c103a40 0x7fa79c19fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:57.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.746+0000 7fa7a3c1d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c10dee0 con 0x7fa79c100e80 2024-05-09T08:25:57.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.746+0000 7fa7a3c1d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c10dd60 con 0x7fa79c103a40 2024-05-09T08:25:57.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.746+0000 7fa7a3c1d700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c10e060 con 0x7fa79c1004c0 2024-05-09T08:25:57.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.747+0000 7fa7a19b9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa79c1004c0 0x7fa79c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:57.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.747+0000 7fa7a19b9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa79c1004c0 0x7fa79c198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:34938/0 (socket says 172.21.15.157:34938) 2024-05-09T08:25:57.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.747+0000 7fa7a19b9700 1 -- 172.21.15.157:0/4079915414 learned_addr learned my addr 172.21.15.157:0/4079915414 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-05-09T08:25:57.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.747+0000 7fa7a21ba700 1 --2- 172.21.15.157:0/4079915414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa79c103a40 0x7fa79c19fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:57.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.747+0000 7fa7a11b8700 1 --2- 172.21.15.157:0/4079915414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa79c100e80 0x7fa79c198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:57.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.747+0000 7fa7a19b9700 1 -- 172.21.15.157:0/4079915414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa79c103a40 msgr2=0x7fa79c19fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:57.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.747+0000 7fa7a19b9700 1 --2- 172.21.15.157:0/4079915414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa79c103a40 0x7fa79c19fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:57.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.747+0000 7fa7a19b9700 1 -- 172.21.15.157:0/4079915414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa79c100e80 msgr2=0x7fa79c198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:25:57.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.747+0000 7fa7a19b9700 1 --2- 172.21.15.157:0/4079915414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa79c100e80 0x7fa79c198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:25:57.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.747+0000 7fa7a19b9700 1 -- 172.21.15.157:0/4079915414 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79c1a01c0 con 0x7fa79c1004c0 2024-05-09T08:25:57.751 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.747+0000 7fa7a19b9700 1 --2- 172.21.15.157:0/4079915414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa79c1004c0 0x7fa79c198560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa79800d9b0 tx=0x7fa79800dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:57.751 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.747+0000 7fa792ffd700 1 -- 172.21.15.157:0/4079915414 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa798017030 con 0x7fa79c1004c0 2024-05-09T08:25:57.751 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.747+0000 7fa7a3c1d700 1 -- 172.21.15.157:0/4079915414 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa79c1a04b0 con 0x7fa79c1004c0 2024-05-09T08:25:57.751 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.747+0000 7fa792ffd700 1 -- 172.21.15.157:0/4079915414 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7980044d0 con 0x7fa79c1004c0 2024-05-09T08:25:57.751 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.747+0000 7fa792ffd700 1 -- 172.21.15.157:0/4079915414 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7980039e0 con 0x7fa79c1004c0 2024-05-09T08:25:57.752 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.747+0000 7fa7a3c1d700 1 -- 172.21.15.157:0/4079915414 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa79c1a09f0 con 0x7fa79c1004c0 2024-05-09T08:25:57.752 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.748+0000 7fa7a3c1d700 1 -- 172.21.15.157:0/4079915414 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa79c069010 con 0x7fa79c1004c0 2024-05-09T08:25:57.752 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.750+0000 7fa792ffd700 1 -- 172.21.15.157:0/4079915414 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7fa79801a070 con 0x7fa79c1004c0 2024-05-09T08:25:57.752 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.751+0000 7fa792ffd700 1 --2- 172.21.15.157:0/4079915414 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa7880a2b90 0x7fa7880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:25:57.752 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.751+0000 7fa792ffd700 1 -- 172.21.15.157:0/4079915414 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fa7980cd490 con 0x7fa79c1004c0 2024-05-09T08:25:57.752 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:57 smithi157 ceph-mon[33614]: Detected new or changed devices on smithi110 2024-05-09T08:25:57.752 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:57 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi110 to 17088M 2024-05-09T08:25:57.753 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:57 smithi157 ceph-mon[33614]: osdmap e14: 2 total, 2 up, 2 in 2024-05-09T08:25:57.753 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.751+0000 7fa7a11b8700 1 --2- 172.21.15.157:0/4079915414 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa7880a2b90 0x7fa7880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:25:57.753 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.752+0000 7fa7a11b8700 1 --2- 172.21.15.157:0/4079915414 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa7880a2b90 0x7fa7880a5050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa78c003920 tx=0x7fa78c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:25:57.755 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.754+0000 7fa792ffd700 1 -- 172.21.15.157:0/4079915414 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa79809bc40 con 0x7fa79c1004c0 2024-05-09T08:25:57.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:57 smithi107 ceph-mon[30122]: Detected new or changed devices on smithi110 2024-05-09T08:25:57.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:57 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi110 to 17088M 2024-05-09T08:25:57.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:57 smithi107 ceph-mon[30122]: osdmap e14: 2 total, 2 up, 2 in 2024-05-09T08:25:57.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:57 smithi110 ceph-mon[33622]: Detected new or changed devices on smithi110 2024-05-09T08:25:57.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:57 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi110 to 17088M 2024-05-09T08:25:57.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:57 smithi110 ceph-mon[33622]: osdmap e14: 2 total, 2 up, 2 in 2024-05-09T08:25:57.958 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:25:57.957+0000 7fa7a3c1d700 1 -- 172.21.15.157:0/4079915414 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fa79c062760 con 0x7fa7880a2b90 2024-05-09T08:25:58.755 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:58 smithi157 ceph-mon[33614]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:25:58.756 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:58 smithi157 ceph-mon[33614]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:58.756 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:25:58.756 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:25:58.756 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:58.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:58 smithi107 ceph-mon[30122]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:25:58.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:58 smithi107 ceph-mon[30122]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:58.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:25:58.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:25:58.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:58.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:58 smithi110 ceph-mon[33622]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:25:58.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:58 smithi110 ceph-mon[33622]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:25:58.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:25:58.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:25:58.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:25:59.737 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:25:59 smithi157 ceph-mon[33614]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:25:59.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:25:59 smithi107 ceph-mon[30122]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:25:59.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:25:59 smithi110 ceph-mon[33622]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:00.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:00 smithi110 ceph-mon[33622]: from='client.? 172.21.15.157:0/1144853636' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc55ab61-bfc9-48ad-92b3-37ad560a0023"}]: dispatch 2024-05-09T08:26:00.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:00 smithi110 ceph-mon[33622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc55ab61-bfc9-48ad-92b3-37ad560a0023"}]: dispatch 2024-05-09T08:26:00.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:00 smithi110 ceph-mon[33622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc55ab61-bfc9-48ad-92b3-37ad560a0023"}]': finished 2024-05-09T08:26:00.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:00 smithi110 ceph-mon[33622]: osdmap e15: 3 total, 2 up, 3 in 2024-05-09T08:26:00.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:00 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:00.895 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:00 smithi157 ceph-mon[33614]: from='client.? 172.21.15.157:0/1144853636' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc55ab61-bfc9-48ad-92b3-37ad560a0023"}]: dispatch 2024-05-09T08:26:00.895 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:00 smithi157 ceph-mon[33614]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc55ab61-bfc9-48ad-92b3-37ad560a0023"}]: dispatch 2024-05-09T08:26:00.895 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:00 smithi157 ceph-mon[33614]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc55ab61-bfc9-48ad-92b3-37ad560a0023"}]': finished 2024-05-09T08:26:00.895 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:00 smithi157 ceph-mon[33614]: osdmap e15: 3 total, 2 up, 3 in 2024-05-09T08:26:00.896 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:00 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:01.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:00 smithi107 ceph-mon[30122]: from='client.? 172.21.15.157:0/1144853636' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc55ab61-bfc9-48ad-92b3-37ad560a0023"}]: dispatch 2024-05-09T08:26:01.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:00 smithi107 ceph-mon[30122]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc55ab61-bfc9-48ad-92b3-37ad560a0023"}]: dispatch 2024-05-09T08:26:01.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:00 smithi107 ceph-mon[30122]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc55ab61-bfc9-48ad-92b3-37ad560a0023"}]': finished 2024-05-09T08:26:01.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:00 smithi107 ceph-mon[30122]: osdmap e15: 3 total, 2 up, 3 in 2024-05-09T08:26:01.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:00 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:01.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:01 smithi110 ceph-mon[33622]: from='client.? 172.21.15.157:0/1592835387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:26:01.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:01 smithi110 ceph-mon[33622]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:02.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:01 smithi107 ceph-mon[30122]: from='client.? 172.21.15.157:0/1592835387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:26:02.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:01 smithi107 ceph-mon[30122]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:02.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:01 smithi157 ceph-mon[33614]: from='client.? 172.21.15.157:0/1592835387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:26:02.095 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:01 smithi157 ceph-mon[33614]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:04.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:04 smithi107 ceph-mon[30122]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:04.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:04 smithi157 ceph-mon[33614]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:04.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:04 smithi110 ceph-mon[33622]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:06.679 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:06 smithi157 ceph-mon[33614]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:06.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:06 smithi107 ceph-mon[30122]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:06.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:06 smithi110 ceph-mon[33622]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:08.708 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:08 smithi157 ceph-mon[33614]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:08.709 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:08 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-09T08:26:08.709 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:08 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:08.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:08 smithi107 ceph-mon[30122]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:08.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:08 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-09T08:26:08.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:08 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:08.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:08 smithi110 ceph-mon[33622]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:08.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:08 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-09T08:26:08.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:08 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:09.748 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:09 smithi157 ceph-mon[33614]: Deploying daemon osd.2 on smithi157 2024-05-09T08:26:09.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:09 smithi107 ceph-mon[30122]: Deploying daemon osd.2 on smithi157 2024-05-09T08:26:09.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:09 smithi110 ceph-mon[33622]: Deploying daemon osd.2 on smithi157 2024-05-09T08:26:10.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:10 smithi107 ceph-mon[30122]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:10.780 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:10 smithi157 ceph-mon[33614]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:10.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:10 smithi110 ceph-mon[33622]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:12.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:12 smithi107 ceph-mon[30122]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:12.783 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:12 smithi157 ceph-mon[33614]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:12.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:12 smithi110 ceph-mon[33622]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:14.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:14 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:14.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:14 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:14.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:14 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:14.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:14 smithi157 ceph-mon[33614]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:14.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:14 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:14.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:14 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:14.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:14 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:14.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:14 smithi110 ceph-mon[33622]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:14.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:14 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:14.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:14 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:14.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:14 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:14.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:14 smithi107 ceph-mon[30122]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:16.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:16 smithi107 ceph-mon[30122]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:16.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:16 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:16.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:16 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:16.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:16 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:16.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:16 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:16.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:16 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:16.800 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:16 smithi157 ceph-mon[33614]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:16.800 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:16 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:16.801 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:16 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:16.801 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:16 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:16.801 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:16 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:16.801 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:16 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:16.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:16 smithi110 ceph-mon[33622]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:16.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:16 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:16.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:16 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:16.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:16 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:16.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:16 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:16.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:16 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:17.127 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:26:17.128+0000 7fa792ffd700 1 -- 172.21.15.157:0/4079915414 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa79c062760 con 0x7fa7880a2b90 2024-05-09T08:26:17.128 INFO:teuthology.orchestra.run.smithi157.stdout:Created osd(s) 2 on host 'smithi157' 2024-05-09T08:26:17.130 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:26:17.130+0000 7fa7a3c1d700 1 -- 172.21.15.157:0/4079915414 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa7880a2b90 msgr2=0x7fa7880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:17.130 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:26:17.130+0000 7fa7a3c1d700 1 --2- 172.21.15.157:0/4079915414 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa7880a2b90 0x7fa7880a5050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa78c003920 tx=0x7fa78c005fb0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.130 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:26:17.130+0000 7fa7a3c1d700 1 -- 172.21.15.157:0/4079915414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa79c1004c0 msgr2=0x7fa79c198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:17.130 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:26:17.130+0000 7fa7a3c1d700 1 --2- 172.21.15.157:0/4079915414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa79c1004c0 0x7fa79c198560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa79800d9b0 tx=0x7fa79800dcc0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.131 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:26:17.130+0000 7fa7a3c1d700 1 -- 172.21.15.157:0/4079915414 shutdown_connections 2024-05-09T08:26:17.131 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:26:17.130+0000 7fa7a3c1d700 1 --2- 172.21.15.157:0/4079915414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa79c103a40 0x7fa79c19fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.131 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:26:17.130+0000 7fa7a3c1d700 1 --2- 172.21.15.157:0/4079915414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa79c1004c0 0x7fa79c198560 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.131 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:26:17.130+0000 7fa7a3c1d700 1 --2- 172.21.15.157:0/4079915414 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa7880a2b90 0x7fa7880a5050 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.131 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:26:17.130+0000 7fa7a3c1d700 1 --2- 172.21.15.157:0/4079915414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa79c100e80 0x7fa79c198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:17.131 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:26:17.130+0000 7fa7a3c1d700 1 -- 172.21.15.157:0/4079915414 >> 172.21.15.157:0/4079915414 conn(0x7fa79c0fc060 msgr2=0x7fa79c106ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:17.131 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:26:17.130+0000 7fa7a3c1d700 1 -- 172.21.15.157:0/4079915414 shutdown_connections 2024-05-09T08:26:17.131 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:26:17.130+0000 7fa7a3c1d700 1 -- 172.21.15.157:0/4079915414 wait complete. 2024-05-09T08:26:17.744 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:17 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:17.744 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:17 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:17.744 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:17 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:17.744 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:17 smithi157 ceph-mon[33614]: from='osd.2 [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:26:17.744 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:17 smithi157 ceph-mon[33614]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:26:17.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:17 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:17.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:17 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:17.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:17 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:17.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:17 smithi107 ceph-mon[30122]: from='osd.2 [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:26:17.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:17 smithi107 ceph-mon[30122]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:26:17.818 DEBUG:teuthology.orchestra.run.smithi157:osd.2> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.2.service 2024-05-09T08:26:17.820 INFO:tasks.cephadm:Deploying osd.3 on smithi107 with /dev/vg_nvme/lv_3... 2024-05-09T08:26:17.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-09T08:26:17.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:17 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:17.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:17 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:17.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:17 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:17.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:17 smithi110 ceph-mon[33622]: from='osd.2 [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:26:17.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:17 smithi110 ceph-mon[33622]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T08:26:18.077 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:26:18.771 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:18 smithi157 ceph-mon[33614]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:18.771 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:18 smithi157 ceph-mon[33614]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-09T08:26:18.771 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:18 smithi157 ceph-mon[33614]: osdmap e16: 3 total, 2 up, 3 in 2024-05-09T08:26:18.771 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:18 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:18.772 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:18 smithi157 ceph-mon[33614]: from='osd.2 [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:26:18.772 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:18 smithi157 ceph-mon[33614]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:26:18.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:18 smithi107 ceph-mon[30122]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:18.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:18 smithi107 ceph-mon[30122]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-09T08:26:18.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:18 smithi107 ceph-mon[30122]: osdmap e16: 3 total, 2 up, 3 in 2024-05-09T08:26:18.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:18 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:18.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:18 smithi107 ceph-mon[30122]: from='osd.2 [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:26:18.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:18 smithi107 ceph-mon[30122]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:26:18.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:18 smithi110 ceph-mon[33622]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:18.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:18 smithi110 ceph-mon[33622]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-09T08:26:18.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:18 smithi110 ceph-mon[33622]: osdmap e16: 3 total, 2 up, 3 in 2024-05-09T08:26:18.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:18 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:18.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:18 smithi110 ceph-mon[33622]: from='osd.2 [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:26:18.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:18 smithi110 ceph-mon[33622]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:26:20.095 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:26:19 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-2[38759]: 2024-05-09T08:26:19.710+0000 7f6e23392700 -1 osd.2 0 waiting for initial osdmap 2024-05-09T08:26:20.095 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:26:19 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-2[38759]: 2024-05-09T08:26:19.715+0000 7f6e1b17d700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:26:20.190 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-09T08:26:20.190 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-09T08:26:20.190 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-05-09T08:26:20.190 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-05-09T08:26:20.190 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113958 s, 920 MB/s 2024-05-09T08:26:20.190 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-05-09T08:26:20.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:20 smithi107 ceph-mon[30122]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-05-09T08:26:20.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:20 smithi107 ceph-mon[30122]: osdmap e17: 3 total, 2 up, 3 in 2024-05-09T08:26:20.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:20.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:20.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:20 smithi107 ceph-mon[30122]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:20.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:20 smithi107 ceph-mon[30122]: from='osd.2 ' entity='osd.2' 2024-05-09T08:26:20.544 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:20 smithi157 ceph-mon[33614]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-05-09T08:26:20.544 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:20 smithi157 ceph-mon[33614]: osdmap e17: 3 total, 2 up, 3 in 2024-05-09T08:26:20.544 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:20.544 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:20.544 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:20 smithi157 ceph-mon[33614]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:20.544 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:20 smithi157 ceph-mon[33614]: from='osd.2 ' entity='osd.2' 2024-05-09T08:26:20.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:20 smithi110 ceph-mon[33622]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-05-09T08:26:20.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:20 smithi110 ceph-mon[33622]: osdmap e17: 3 total, 2 up, 3 in 2024-05-09T08:26:20.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:20.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:20.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:20 smithi110 ceph-mon[33622]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-09T08:26:20.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:20 smithi110 ceph-mon[33622]: from='osd.2 ' entity='osd.2' 2024-05-09T08:26:20.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi107:vg_nvme/lv_3 2024-05-09T08:26:21.146 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:26:21.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:21 smithi157 ceph-mon[33614]: purged_snaps scrub starts 2024-05-09T08:26:21.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:21 smithi157 ceph-mon[33614]: purged_snaps scrub ok 2024-05-09T08:26:21.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:21 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:21.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:21 smithi157 ceph-mon[33614]: osd.2 [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647] boot 2024-05-09T08:26:21.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:21 smithi157 ceph-mon[33614]: osdmap e18: 3 total, 3 up, 3 in 2024-05-09T08:26:21.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:21 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:21.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:21 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:21.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:21 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:21.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:21 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:21.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:21 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:21.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:21 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:21.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:21 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:21.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:21 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:21.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:21 smithi107 ceph-mon[30122]: purged_snaps scrub starts 2024-05-09T08:26:21.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:21 smithi107 ceph-mon[30122]: purged_snaps scrub ok 2024-05-09T08:26:21.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:21 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:21.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:21 smithi107 ceph-mon[30122]: osd.2 [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647] boot 2024-05-09T08:26:21.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:21 smithi107 ceph-mon[30122]: osdmap e18: 3 total, 3 up, 3 in 2024-05-09T08:26:21.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:21 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:21.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:21 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:21.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:21 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:21.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:21 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:21.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:21 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:21.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:21 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:21.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:21 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:21.394 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:21 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:21.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:21 smithi110 ceph-mon[33622]: purged_snaps scrub starts 2024-05-09T08:26:21.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:21 smithi110 ceph-mon[33622]: purged_snaps scrub ok 2024-05-09T08:26:21.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:21 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:21.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:21 smithi110 ceph-mon[33622]: osd.2 [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647] boot 2024-05-09T08:26:21.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:21 smithi110 ceph-mon[33622]: osdmap e18: 3 total, 3 up, 3 in 2024-05-09T08:26:21.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:21 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:26:21.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:21 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:21.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:21 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:21.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:21 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:21.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:21 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:21.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:21 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:21.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:21 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:21.624 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:21 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:22.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:22 smithi107 ceph-mon[30122]: Detected new or changed devices on smithi157 2024-05-09T08:26:22.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:22 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi157 to 17088M 2024-05-09T08:26:22.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:22 smithi107 ceph-mon[30122]: pgmap v81: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:22.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:22 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:26:22.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:22 smithi157 ceph-mon[33614]: Detected new or changed devices on smithi157 2024-05-09T08:26:22.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:22 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi157 to 17088M 2024-05-09T08:26:22.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:22 smithi157 ceph-mon[33614]: pgmap v81: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:22.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:22 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:26:22.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:22 smithi110 ceph-mon[33622]: Detected new or changed devices on smithi157 2024-05-09T08:26:22.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:22 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi157 to 17088M 2024-05-09T08:26:22.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:22 smithi110 ceph-mon[33622]: pgmap v81: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:22.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:22 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:26:22.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.786+0000 7f6a43bb2700 1 -- 172.21.15.107:0/4130424694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a3c102480 msgr2=0x7f6a3c1028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.786+0000 7f6a43bb2700 1 --2- 172.21.15.107:0/4130424694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a3c102480 0x7f6a3c1028e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6a380099c0 tx=0x7f6a38009cd0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.786+0000 7f6a43bb2700 1 -- 172.21.15.107:0/4130424694 shutdown_connections 2024-05-09T08:26:22.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.786+0000 7f6a43bb2700 1 --2- 172.21.15.107:0/4130424694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a3c102e20 0x7f6a3c1096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.786+0000 7f6a43bb2700 1 --2- 172.21.15.107:0/4130424694 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a3c101280 0x7f6a3c1016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.786+0000 7f6a43bb2700 1 --2- 172.21.15.107:0/4130424694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a3c102480 0x7f6a3c1028e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.786+0000 7f6a43bb2700 1 -- 172.21.15.107:0/4130424694 >> 172.21.15.107:0/4130424694 conn(0x7f6a3c0fc820 msgr2=0x7f6a3c0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:22.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.786+0000 7f6a43bb2700 1 -- 172.21.15.107:0/4130424694 shutdown_connections 2024-05-09T08:26:22.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.786+0000 7f6a43bb2700 1 -- 172.21.15.107:0/4130424694 wait complete. 2024-05-09T08:26:22.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.787+0000 7f6a43bb2700 1 Processor -- start 2024-05-09T08:26:22.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.787+0000 7f6a43bb2700 1 -- start start 2024-05-09T08:26:22.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.787+0000 7f6a43bb2700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a3c101280 0x7f6a3c1973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:22.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.787+0000 7f6a43bb2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a3c102480 0x7f6a3c197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:22.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.787+0000 7f6a43bb2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a3c102e20 0x7f6a3c19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:22.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.787+0000 7f6a43bb2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c10cd10 con 0x7f6a3c102480 2024-05-09T08:26:22.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.787+0000 7f6a43bb2700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c10cb90 con 0x7f6a3c102e20 2024-05-09T08:26:22.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.787+0000 7f6a43bb2700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c10ce90 con 0x7f6a3c101280 2024-05-09T08:26:22.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.787+0000 7f6a4114d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a3c102480 0x7f6a3c197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.787+0000 7f6a4114d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a3c102480 0x7f6a3c197900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36914/0 (socket says 172.21.15.107:36914) 2024-05-09T08:26:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.787+0000 7f6a4114d700 1 -- 172.21.15.107:0/435410865 learned_addr learned my addr 172.21.15.107:0/435410865 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:26:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.787+0000 7f6a4194e700 1 --2- 172.21.15.107:0/435410865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a3c101280 0x7f6a3c1973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.788+0000 7f6a4214f700 1 --2- 172.21.15.107:0/435410865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a3c102e20 0x7f6a3c19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.788+0000 7f6a4114d700 1 -- 172.21.15.107:0/435410865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a3c101280 msgr2=0x7f6a3c1973c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.788+0000 7f6a4114d700 1 --2- 172.21.15.107:0/435410865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a3c101280 0x7f6a3c1973c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.788+0000 7f6a4114d700 1 -- 172.21.15.107:0/435410865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a3c102e20 msgr2=0x7f6a3c19e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.788+0000 7f6a4114d700 1 --2- 172.21.15.107:0/435410865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a3c102e20 0x7f6a3c19e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.788+0000 7f6a4114d700 1 -- 172.21.15.107:0/435410865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3c19f020 con 0x7f6a3c102480 2024-05-09T08:26:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.788+0000 7f6a4214f700 1 --2- 172.21.15.107:0/435410865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a3c102e20 0x7f6a3c19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:26:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.788+0000 7f6a4114d700 1 --2- 172.21.15.107:0/435410865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a3c102480 0x7f6a3c197900 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6a3800b580 tx=0x7f6a3800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.788+0000 7f6a32ffd700 1 -- 172.21.15.107:0/435410865 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a38027070 con 0x7f6a3c102480 2024-05-09T08:26:22.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.788+0000 7f6a43bb2700 1 -- 172.21.15.107:0/435410865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a3c19f2b0 con 0x7f6a3c102480 2024-05-09T08:26:22.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.788+0000 7f6a32ffd700 1 -- 172.21.15.107:0/435410865 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a3800fe30 con 0x7f6a3c102480 2024-05-09T08:26:22.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.788+0000 7f6a32ffd700 1 -- 172.21.15.107:0/435410865 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a38022e60 con 0x7f6a3c102480 2024-05-09T08:26:22.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.788+0000 7f6a43bb2700 1 -- 172.21.15.107:0/435410865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a3c19f760 con 0x7f6a3c102480 2024-05-09T08:26:22.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.789+0000 7f6a43bb2700 1 -- 172.21.15.107:0/435410865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a3c069010 con 0x7f6a3c102480 2024-05-09T08:26:22.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.790+0000 7f6a32ffd700 1 -- 172.21.15.107:0/435410865 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f6a3802a070 con 0x7f6a3c102480 2024-05-09T08:26:22.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.791+0000 7f6a32ffd700 1 --2- 172.21.15.107:0/435410865 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6a280a2b90 0x7f6a280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:22.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.791+0000 7f6a4194e700 1 --2- 172.21.15.107:0/435410865 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6a280a2b90 0x7f6a280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:22.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.791+0000 7f6a32ffd700 1 -- 172.21.15.107:0/435410865 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7f6a380ce7c0 con 0x7f6a3c102480 2024-05-09T08:26:22.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.791+0000 7f6a4194e700 1 --2- 172.21.15.107:0/435410865 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6a280a2b90 0x7f6a280a5050 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c1022e0 tx=0x7f6a2c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:22.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.795+0000 7f6a32ffd700 1 -- 172.21.15.107:0/435410865 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6a38013070 con 0x7f6a3c102480 2024-05-09T08:26:22.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:22.999+0000 7f6a43bb2700 1 -- 172.21.15.107:0/435410865 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f6a3c062760 con 0x7f6a280a2b90 2024-05-09T08:26:23.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:23 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-09T08:26:23.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:23 smithi107 ceph-mon[30122]: osdmap e19: 3 total, 3 up, 3 in 2024-05-09T08:26:23.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:23 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:26:23.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:23 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:26:23.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:23 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:26:23.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:23 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:23.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:23 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-09T08:26:23.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:23 smithi157 ceph-mon[33614]: osdmap e19: 3 total, 3 up, 3 in 2024-05-09T08:26:23.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:23 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:26:23.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:23 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:26:23.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:23 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:26:23.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:23 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:23.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:23 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-09T08:26:23.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:23 smithi110 ceph-mon[33622]: osdmap e19: 3 total, 3 up, 3 in 2024-05-09T08:26:23.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:23 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:26:23.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:23 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:26:23.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:23 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:26:23.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:23 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:23.758 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:26:23 smithi107 sudo[46806]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:26:23.759 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:26:23 smithi107 sudo[46806]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:26:23.759 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:26:23 smithi107 sudo[46806]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:26:23.759 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:26:23 smithi107 sudo[46806]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:26:24.030 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:26:23 smithi107 sudo[46830]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:26:24.030 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:26:23 smithi107 sudo[46830]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:26:24.030 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:26:23 smithi107 sudo[46830]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:26:24.030 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:26:23 smithi107 sudo[46830]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:26:24.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:24.159+0000 7f6a32ffd700 1 -- 172.21.15.107:0/435410865 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f6a38098d00 con 0x7f6a3c102480 2024-05-09T08:26:24.283 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:26:24 smithi110 sudo[42011]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:26:24.283 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:26:24 smithi110 sudo[42011]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:26:24.283 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:26:24 smithi110 sudo[42011]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:26:24.283 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:26:24 smithi110 sudo[42011]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:26:24.284 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:24 smithi110 ceph-mon[33622]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:24.284 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:24 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-09T08:26:24.284 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:24 smithi110 ceph-mon[33622]: osdmap e20: 3 total, 3 up, 3 in 2024-05-09T08:26:24.284 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:24 smithi110 ceph-mon[33622]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:24.284 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:24 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:24.284 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:24 smithi110 ceph-mon[33622]: osdmap e21: 3 total, 3 up, 3 in 2024-05-09T08:26:24.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:24 smithi107 ceph-mon[30122]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:24.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:24 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-09T08:26:24.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:24 smithi107 ceph-mon[30122]: osdmap e20: 3 total, 3 up, 3 in 2024-05-09T08:26:24.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:24 smithi107 ceph-mon[30122]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:24.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:24 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:24.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:24 smithi107 ceph-mon[30122]: osdmap e21: 3 total, 3 up, 3 in 2024-05-09T08:26:24.549 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:24 smithi157 ceph-mon[33614]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:24.550 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:24 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-09T08:26:24.550 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:24 smithi157 ceph-mon[33614]: osdmap e20: 3 total, 3 up, 3 in 2024-05-09T08:26:24.550 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:24 smithi157 ceph-mon[33614]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:24.550 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:24 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:24.550 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:24 smithi157 ceph-mon[33614]: osdmap e21: 3 total, 3 up, 3 in 2024-05-09T08:26:24.622 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:26:24 smithi110 sudo[42014]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:26:24.622 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:26:24 smithi110 sudo[42014]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:26:24.622 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:26:24 smithi110 sudo[42014]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:26:24.622 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:26:24 smithi110 sudo[42014]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:26:24.804 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:26:24 smithi157 sudo[42506]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:26:24.804 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:26:24 smithi157 sudo[42506]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:26:24.804 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:26:24 smithi157 sudo[42506]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:26:24.804 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:26:24 smithi157 sudo[42506]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:26:25.094 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:26:24 smithi157 sudo[42509]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:26:25.094 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:26:24 smithi157 sudo[42509]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:26:25.094 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:26:24 smithi157 sudo[42509]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:26:25.094 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:26:24 smithi157 sudo[42509]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:26:25.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:25 smithi107 sudo[46882]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:26:25.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:25 smithi107 sudo[46882]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:26:25.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:25 smithi107 sudo[46882]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:26:25.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:25 smithi107 sudo[46882]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:26:25.603 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:25 smithi157 ceph-mon[33614]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-09T08:26:25.603 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:25 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:25.603 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:25 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:25.603 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:25 smithi157 ceph-mon[33614]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:26:25.603 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:25 smithi157 sudo[42512]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:26:25.603 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:25 smithi157 sudo[42512]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:26:25.603 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:25 smithi157 sudo[42512]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:26:25.603 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:25 smithi157 sudo[42512]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:26:25.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:25 smithi110 ceph-mon[33622]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-09T08:26:25.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:25 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:25.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:25 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:25.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:25 smithi110 ceph-mon[33622]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:26:25.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:25 smithi107 sudo[46896]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:26:25.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:25 smithi107 sudo[46896]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:26:25.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:25 smithi107 sudo[46896]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:26:25.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:25 smithi107 sudo[46896]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:26:25.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:25 smithi107 ceph-mon[30122]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-09T08:26:25.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:25 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:25.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:25 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:25.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:25 smithi107 ceph-mon[30122]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:26:26.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:25 smithi157 sudo[42515]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:26:26.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:25 smithi157 sudo[42515]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:26:26.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:25 smithi157 sudo[42515]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:26:26.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:25 smithi157 sudo[42515]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:26:26.335 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 sudo[42017]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T08:26:26.335 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 sudo[42017]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:26:26.335 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 sudo[42017]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:26:26.335 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 sudo[42017]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:26:26.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:26:26.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:26:26.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:26:26.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:26:26.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:26:26.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:26:26.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:26:26.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:26:26.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:26.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3090919284' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b4ef7e0-8679-44c3-ac27-170d0ab391aa"}]: dispatch 2024-05-09T08:26:26.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:26:26.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3090919284' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b4ef7e0-8679-44c3-ac27-170d0ab391aa"}]': finished 2024-05-09T08:26:26.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: osdmap e22: 4 total, 3 up, 4 in 2024-05-09T08:26:26.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:26.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:26:26.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:26:26.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:26:26.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:26:26.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:26 smithi107 ceph-mon[30122]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:26:26.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 sudo[42020]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T08:26:26.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 sudo[42020]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T08:26:26.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 sudo[42020]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T08:26:26.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 sudo[42020]: pam_unix(sudo:session): session closed for user root 2024-05-09T08:26:26.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:26:26.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:26:26.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:26:26.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:26:26.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:26:26.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:26:26.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:26:26.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:26:26.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:26.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3090919284' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b4ef7e0-8679-44c3-ac27-170d0ab391aa"}]: dispatch 2024-05-09T08:26:26.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:26:26.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3090919284' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b4ef7e0-8679-44c3-ac27-170d0ab391aa"}]': finished 2024-05-09T08:26:26.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: osdmap e22: 4 total, 3 up, 4 in 2024-05-09T08:26:26.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:26.624 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:26:26.624 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:26:26.624 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:26:26.624 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:26:26.624 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:26 smithi110 ceph-mon[33622]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:26:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:26:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:26:26.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:26:26.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:26:26.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:26:26.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:26:26.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:26:26.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:26:26.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:26.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3090919284' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b4ef7e0-8679-44c3-ac27-170d0ab391aa"}]: dispatch 2024-05-09T08:26:26.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:26:26.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3090919284' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b4ef7e0-8679-44c3-ac27-170d0ab391aa"}]': finished 2024-05-09T08:26:26.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: osdmap e22: 4 total, 3 up, 4 in 2024-05-09T08:26:26.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:26.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T08:26:26.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:26:26.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:26:26.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:26:26.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:26 smithi157 ceph-mon[33614]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T08:26:27.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:27 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2611611684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:26:27.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:27 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2611611684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:26:27.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:27 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2611611684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:26:28.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:28 smithi110 ceph-mon[33622]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:28.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:28 smithi107 ceph-mon[30122]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:28.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:28 smithi157 ceph-mon[33614]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:30.739 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:30 smithi107 ceph-mon[30122]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:30.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:30 smithi157 ceph-mon[33614]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:30.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:30 smithi110 ceph-mon[33622]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:32.732 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:32 smithi107 ceph-mon[30122]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:32.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:32 smithi157 ceph-mon[33614]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:32.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:32 smithi110 ceph-mon[33622]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:33.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-09T08:26:33.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:33.804 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-09T08:26:33.804 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:33.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-09T08:26:33.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:34.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:34 smithi107 ceph-mon[30122]: Deploying daemon osd.3 on smithi107 2024-05-09T08:26:34.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:34 smithi107 ceph-mon[30122]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:34.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:34 smithi157 ceph-mon[33614]: Deploying daemon osd.3 on smithi107 2024-05-09T08:26:34.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:34 smithi157 ceph-mon[33614]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:34.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:34 smithi110 ceph-mon[33622]: Deploying daemon osd.3 on smithi107 2024-05-09T08:26:34.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:34 smithi110 ceph-mon[33622]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:36.740 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:36 smithi107 ceph-mon[30122]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:36.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:36 smithi157 ceph-mon[33614]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:36.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:36 smithi110 ceph-mon[33622]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:38.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:38 smithi107 ceph-mon[30122]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:38.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:38 smithi157 ceph-mon[33614]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:38.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:38 smithi110 ceph-mon[33622]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:40.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:40 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:40.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:40 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:40.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:40 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:40.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:40 smithi107 ceph-mon[30122]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:40.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:40 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:40.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:40 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:40.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:40 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:40.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:40 smithi157 ceph-mon[33614]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:40.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:40 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:40.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:40 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:40.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:40 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:40.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:40 smithi110 ceph-mon[33622]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:42.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:42 smithi107 ceph-mon[30122]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:42.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:42 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:42.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:42 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:42.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:42 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:42.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:42 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:42.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:42 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:42.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:42 smithi157 ceph-mon[33614]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:42.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:42 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:42.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:42 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:42.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:42 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:42.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:42 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:42.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:42 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:42.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:42 smithi110 ceph-mon[33622]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:42.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:42 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:42.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:42 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:42.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:42 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:42.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:42 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:42.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:42 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:42.907 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 3 on host 'smithi107' 2024-05-09T08:26:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:42.907+0000 7f6a32ffd700 1 -- 172.21.15.107:0/435410865 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6a3c062760 con 0x7f6a280a2b90 2024-05-09T08:26:42.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:42.909+0000 7f6a43bb2700 1 -- 172.21.15.107:0/435410865 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6a280a2b90 msgr2=0x7f6a280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:42.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:42.909+0000 7f6a43bb2700 1 --2- 172.21.15.107:0/435410865 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6a280a2b90 0x7f6a280a5050 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c1022e0 tx=0x7f6a2c009260 comp rx=0 tx=0).stop 2024-05-09T08:26:42.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:42.909+0000 7f6a43bb2700 1 -- 172.21.15.107:0/435410865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a3c102480 msgr2=0x7f6a3c197900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:42.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:42.909+0000 7f6a43bb2700 1 --2- 172.21.15.107:0/435410865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a3c102480 0x7f6a3c197900 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6a3800b580 tx=0x7f6a3800fa00 comp rx=0 tx=0).stop 2024-05-09T08:26:42.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:42.909+0000 7f6a43bb2700 1 -- 172.21.15.107:0/435410865 shutdown_connections 2024-05-09T08:26:42.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:42.909+0000 7f6a43bb2700 1 --2- 172.21.15.107:0/435410865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a3c102e20 0x7f6a3c19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:42.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:42.909+0000 7f6a43bb2700 1 --2- 172.21.15.107:0/435410865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a3c101280 0x7f6a3c1973c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:42.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:42.910+0000 7f6a43bb2700 1 --2- 172.21.15.107:0/435410865 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6a280a2b90 0x7f6a280a5050 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:42.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:42.910+0000 7f6a43bb2700 1 --2- 172.21.15.107:0/435410865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a3c102480 0x7f6a3c197900 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:42.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:42.910+0000 7f6a43bb2700 1 -- 172.21.15.107:0/435410865 >> 172.21.15.107:0/435410865 conn(0x7f6a3c0fc820 msgr2=0x7f6a3c1058a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:42.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:42.910+0000 7f6a43bb2700 1 -- 172.21.15.107:0/435410865 shutdown_connections 2024-05-09T08:26:42.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:26:42.910+0000 7f6a43bb2700 1 -- 172.21.15.107:0/435410865 wait complete. 2024-05-09T08:26:43.635 DEBUG:teuthology.orchestra.run.smithi107:osd.3> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.3.service 2024-05-09T08:26:43.637 INFO:tasks.cephadm:Deploying osd.4 on smithi110 with /dev/vg_nvme/lv_3... 2024-05-09T08:26:43.637 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-09T08:26:43.885 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:26:44.006 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:43 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:44.007 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:43 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:44.007 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:43 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:44.007 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:43 smithi110 ceph-mon[33622]: from='osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-09T08:26:44.007 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:43 smithi110 ceph-mon[33622]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:44.152 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:43 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:44.153 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:43 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:44.153 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:43 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:44.153 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:43 smithi107 ceph-mon[30122]: from='osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-09T08:26:44.153 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:43 smithi107 ceph-mon[30122]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:44.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:43 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:44.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:43 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:44.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:43 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:26:44.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:43 smithi157 ceph-mon[33614]: from='osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-09T08:26:44.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:43 smithi157 ceph-mon[33614]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:45.013 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:44 smithi107 ceph-mon[30122]: from='osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-09T08:26:45.013 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:44 smithi107 ceph-mon[30122]: osdmap e23: 4 total, 3 up, 4 in 2024-05-09T08:26:45.013 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:44 smithi107 ceph-mon[30122]: from='osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-05-09T08:26:45.013 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:45.215 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:44 smithi110 ceph-mon[33622]: from='osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-09T08:26:45.215 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:44 smithi110 ceph-mon[33622]: osdmap e23: 4 total, 3 up, 4 in 2024-05-09T08:26:45.215 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:44 smithi110 ceph-mon[33622]: from='osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-05-09T08:26:45.215 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:44 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:45.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:44 smithi157 ceph-mon[33614]: from='osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-09T08:26:45.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:44 smithi157 ceph-mon[33614]: osdmap e23: 4 total, 3 up, 4 in 2024-05-09T08:26:45.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:44 smithi157 ceph-mon[33614]: from='osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-05-09T08:26:45.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:45.780 INFO:journalctl@ceph.osd.3.smithi107.stdout:May 09 08:26:45 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-3[48973]: 2024-05-09T08:26:45.590+0000 7f7cefe50700 -1 osd.3 0 waiting for initial osdmap 2024-05-09T08:26:45.780 INFO:journalctl@ceph.osd.3.smithi107.stdout:May 09 08:26:45 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-3[48973]: 2024-05-09T08:26:45.596+0000 7f7ce743a700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:26:45.785 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-09T08:26:45.785 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-09T08:26:45.785 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-05-09T08:26:45.785 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-05-09T08:26:45.785 INFO:teuthology.orchestra.run.smithi110.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00819344 s, 1.3 GB/s 2024-05-09T08:26:45.785 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-05-09T08:26:46.153 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:45 smithi110 ceph-mon[33622]: from='osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-05-09T08:26:46.153 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:45 smithi110 ceph-mon[33622]: osdmap e24: 4 total, 3 up, 4 in 2024-05-09T08:26:46.153 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:45 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:46.153 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:45 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:46.153 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:45 smithi110 ceph-mon[33622]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:46.153 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:45 smithi110 ceph-mon[33622]: from='osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754]' entity='osd.3' 2024-05-09T08:26:46.153 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:45 smithi110 ceph-mon[33622]: osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754] boot 2024-05-09T08:26:46.153 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:45 smithi110 ceph-mon[33622]: osdmap e25: 4 total, 4 up, 4 in 2024-05-09T08:26:46.153 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:45 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:46.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:45 smithi107 ceph-mon[30122]: from='osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-05-09T08:26:46.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:45 smithi107 ceph-mon[30122]: osdmap e24: 4 total, 3 up, 4 in 2024-05-09T08:26:46.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:45 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:46.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:45 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:46.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:45 smithi107 ceph-mon[30122]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:46.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:45 smithi107 ceph-mon[30122]: from='osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754]' entity='osd.3' 2024-05-09T08:26:46.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:45 smithi107 ceph-mon[30122]: osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754] boot 2024-05-09T08:26:46.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:45 smithi107 ceph-mon[30122]: osdmap e25: 4 total, 4 up, 4 in 2024-05-09T08:26:46.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:45 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:46.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:45 smithi157 ceph-mon[33614]: from='osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-05-09T08:26:46.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:45 smithi157 ceph-mon[33614]: osdmap e24: 4 total, 3 up, 4 in 2024-05-09T08:26:46.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:45 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:46.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:45 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:46.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:45 smithi157 ceph-mon[33614]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-05-09T08:26:46.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:45 smithi157 ceph-mon[33614]: from='osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754]' entity='osd.3' 2024-05-09T08:26:46.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:45 smithi157 ceph-mon[33614]: osd.3 [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754] boot 2024-05-09T08:26:46.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:45 smithi157 ceph-mon[33614]: osdmap e25: 4 total, 4 up, 4 in 2024-05-09T08:26:46.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:45 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:26:46.438 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi110:vg_nvme/lv_3 2024-05-09T08:26:46.678 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:26:47.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:46 smithi107 ceph-mon[30122]: purged_snaps scrub starts 2024-05-09T08:26:47.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:46 smithi107 ceph-mon[30122]: purged_snaps scrub ok 2024-05-09T08:26:47.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:46 smithi107 ceph-mon[30122]: osdmap e26: 4 total, 4 up, 4 in 2024-05-09T08:26:47.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:47.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:47.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:47.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:47.031 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:47.031 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:47.031 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:47.031 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:47.049 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:46 smithi110 ceph-mon[33622]: purged_snaps scrub starts 2024-05-09T08:26:47.049 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:46 smithi110 ceph-mon[33622]: purged_snaps scrub ok 2024-05-09T08:26:47.049 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:46 smithi110 ceph-mon[33622]: osdmap e26: 4 total, 4 up, 4 in 2024-05-09T08:26:47.049 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:47.049 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:47.049 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:47.049 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:47.049 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:47.049 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:47.049 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:47.049 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:47.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:46 smithi157 ceph-mon[33614]: purged_snaps scrub starts 2024-05-09T08:26:47.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:46 smithi157 ceph-mon[33614]: purged_snaps scrub ok 2024-05-09T08:26:47.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:46 smithi157 ceph-mon[33614]: osdmap e26: 4 total, 4 up, 4 in 2024-05-09T08:26:47.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:47.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:47.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:47.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:26:47.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:47.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:47.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:26:47.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:26:48.045 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:47 smithi110 ceph-mon[33622]: Detected new or changed devices on smithi107 2024-05-09T08:26:48.045 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:47 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi107 to 8544M 2024-05-09T08:26:48.045 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:47 smithi110 ceph-mon[33622]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 196 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:26:48.045 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:47 smithi110 ceph-mon[33622]: osdmap e27: 4 total, 4 up, 4 in 2024-05-09T08:26:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.258+0000 7f9d8f70b700 1 -- 172.21.15.110:0/3374418497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d88102380 msgr2=0x7f9d881027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.258+0000 7f9d8f70b700 1 --2- 172.21.15.110:0/3374418497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d88102380 0x7f9d881027a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9d840099c0 tx=0x7f9d84009cd0 comp rx=0 tx=0).stop 2024-05-09T08:26:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.258+0000 7f9d8f70b700 1 -- 172.21.15.110:0/3374418497 shutdown_connections 2024-05-09T08:26:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.258+0000 7f9d8f70b700 1 --2- 172.21.15.110:0/3374418497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d88103f20 0x7f9d8810a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.258+0000 7f9d8f70b700 1 --2- 172.21.15.110:0/3374418497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d88102380 0x7f9d881027a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.258+0000 7f9d8f70b700 1 --2- 172.21.15.110:0/3374418497 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d88103580 0x7f9d881039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.258+0000 7f9d8f70b700 1 -- 172.21.15.110:0/3374418497 >> 172.21.15.110:0/3374418497 conn(0x7f9d880fd920 msgr2=0x7f9d880ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:26:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.258+0000 7f9d8f70b700 1 -- 172.21.15.110:0/3374418497 shutdown_connections 2024-05-09T08:26:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.258+0000 7f9d8f70b700 1 -- 172.21.15.110:0/3374418497 wait complete. 2024-05-09T08:26:48.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.258+0000 7f9d8f70b700 1 Processor -- start 2024-05-09T08:26:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8f70b700 1 -- start start 2024-05-09T08:26:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8f70b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d88102380 0x7f9d8819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8f70b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d88103580 0x7f9d8819ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8f70b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d88103f20 0x7f9d881a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8f70b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8810de10 con 0x7f9d88103f20 2024-05-09T08:26:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8f70b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8810dc90 con 0x7f9d88102380 2024-05-09T08:26:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8f70b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8810df90 con 0x7f9d88103580 2024-05-09T08:26:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8d4a7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d88102380 0x7f9d8819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8d4a7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d88102380 0x7f9d8819a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45292/0 (socket says 172.21.15.110:45292) 2024-05-09T08:26:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8d4a7700 1 -- 172.21.15.110:0/3343373514 learned_addr learned my addr 172.21.15.110:0/3343373514 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-05-09T08:26:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8cca6700 1 --2- 172.21.15.110:0/3343373514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d88103580 0x7f9d8819ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:48.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8d4a7700 1 -- 172.21.15.110:0/3343373514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d88103580 msgr2=0x7f9d8819ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8d4a7700 1 --2- 172.21.15.110:0/3343373514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d88103580 0x7f9d8819ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8d4a7700 1 -- 172.21.15.110:0/3343373514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d88103f20 msgr2=0x7f9d881a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:26:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8d4a7700 1 --2- 172.21.15.110:0/3343373514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d88103f20 0x7f9d881a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:26:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8d4a7700 1 -- 172.21.15.110:0/3343373514 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d881a2390 con 0x7f9d88102380 2024-05-09T08:26:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8d4a7700 1 --2- 172.21.15.110:0/3343373514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d88102380 0x7f9d8819a7d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9d8400fa00 tx=0x7f9d8400fae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d7e7fc700 1 -- 172.21.15.110:0/3343373514 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d84027070 con 0x7f9d88102380 2024-05-09T08:26:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d7e7fc700 1 -- 172.21.15.110:0/3343373514 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d84005940 con 0x7f9d88102380 2024-05-09T08:26:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d8f70b700 1 -- 172.21.15.110:0/3343373514 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d881a2620 con 0x7f9d88102380 2024-05-09T08:26:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.259+0000 7f9d7e7fc700 1 -- 172.21.15.110:0/3343373514 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d840173f0 con 0x7f9d88102380 2024-05-09T08:26:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.260+0000 7f9d8f70b700 1 -- 172.21.15.110:0/3343373514 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d881a2b50 con 0x7f9d88102380 2024-05-09T08:26:48.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.260+0000 7f9d8f70b700 1 -- 172.21.15.110:0/3343373514 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d88069010 con 0x7f9d88102380 2024-05-09T08:26:48.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.262+0000 7f9d7e7fc700 1 -- 172.21.15.110:0/3343373514 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9d8402a070 con 0x7f9d88102380 2024-05-09T08:26:48.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.262+0000 7f9d7e7fc700 1 --2- 172.21.15.110:0/3343373514 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9d740a2ca0 0x7f9d740a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:26:48.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.262+0000 7f9d7e7fc700 1 -- 172.21.15.110:0/3343373514 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f9d840ceae0 con 0x7f9d88102380 2024-05-09T08:26:48.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.263+0000 7f9d8cca6700 1 --2- 172.21.15.110:0/3343373514 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9d740a2ca0 0x7f9d740a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:26:48.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.263+0000 7f9d8cca6700 1 --2- 172.21.15.110:0/3343373514 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9d740a2ca0 0x7f9d740a5160 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9d78005950 tx=0x7f9d780058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:26:48.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.266+0000 7f9d7e7fc700 1 -- 172.21.15.110:0/3343373514 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9d84013070 con 0x7f9d88102380 2024-05-09T08:26:48.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:47 smithi107 ceph-mon[30122]: Detected new or changed devices on smithi107 2024-05-09T08:26:48.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:47 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi107 to 8544M 2024-05-09T08:26:48.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:47 smithi107 ceph-mon[30122]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 196 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:26:48.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:47 smithi107 ceph-mon[30122]: osdmap e27: 4 total, 4 up, 4 in 2024-05-09T08:26:48.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:47 smithi157 ceph-mon[33614]: Detected new or changed devices on smithi107 2024-05-09T08:26:48.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:47 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi107 to 8544M 2024-05-09T08:26:48.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:47 smithi157 ceph-mon[33614]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 196 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:26:48.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:47 smithi157 ceph-mon[33614]: osdmap e27: 4 total, 4 up, 4 in 2024-05-09T08:26:48.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:26:48.474+0000 7f9d8f70b700 1 -- 172.21.15.110:0/3343373514 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f9d8810a4c0 con 0x7f9d740a2ca0 2024-05-09T08:26:49.135 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:48 smithi107 ceph-mon[30122]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:49.135 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:48 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:26:49.135 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:48 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:26:49.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:48 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:49.190 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:48 smithi110 ceph-mon[33622]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:49.190 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:26:49.190 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:26:49.190 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:48 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:49.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:48 smithi157 ceph-mon[33614]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:26:49.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:48 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:26:49.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:48 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:26:49.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:48 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:50.201 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:49 smithi110 ceph-mon[33622]: osdmap e28: 4 total, 4 up, 4 in 2024-05-09T08:26:50.201 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:49 smithi110 ceph-mon[33622]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 196 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:26:50.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:49 smithi107 ceph-mon[30122]: osdmap e28: 4 total, 4 up, 4 in 2024-05-09T08:26:50.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:49 smithi107 ceph-mon[30122]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 196 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:26:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:49 smithi157 ceph-mon[33614]: osdmap e28: 4 total, 4 up, 4 in 2024-05-09T08:26:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:49 smithi157 ceph-mon[33614]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 196 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:26:51.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:50 smithi107 ceph-mon[30122]: osdmap e29: 4 total, 4 up, 4 in 2024-05-09T08:26:51.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:50 smithi157 ceph-mon[33614]: osdmap e29: 4 total, 4 up, 4 in 2024-05-09T08:26:51.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:50 smithi110 ceph-mon[33622]: osdmap e29: 4 total, 4 up, 4 in 2024-05-09T08:26:52.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:51 smithi107 ceph-mon[30122]: from='client.? 172.21.15.110:0/3762827165' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b67c4b46-7857-48cd-8737-d7cbb37aabaf"}]: dispatch 2024-05-09T08:26:52.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:51 smithi107 ceph-mon[30122]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b67c4b46-7857-48cd-8737-d7cbb37aabaf"}]: dispatch 2024-05-09T08:26:52.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:51 smithi107 ceph-mon[30122]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b67c4b46-7857-48cd-8737-d7cbb37aabaf"}]': finished 2024-05-09T08:26:52.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:51 smithi107 ceph-mon[30122]: osdmap e30: 5 total, 4 up, 5 in 2024-05-09T08:26:52.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:51 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:26:52.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:51 smithi107 ceph-mon[30122]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:26:52.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:51 smithi107 ceph-mon[30122]: from='client.? 172.21.15.110:0/3205405334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:26:52.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:51 smithi157 ceph-mon[33614]: from='client.? 172.21.15.110:0/3762827165' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b67c4b46-7857-48cd-8737-d7cbb37aabaf"}]: dispatch 2024-05-09T08:26:52.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:51 smithi157 ceph-mon[33614]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b67c4b46-7857-48cd-8737-d7cbb37aabaf"}]: dispatch 2024-05-09T08:26:52.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:51 smithi157 ceph-mon[33614]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b67c4b46-7857-48cd-8737-d7cbb37aabaf"}]': finished 2024-05-09T08:26:52.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:51 smithi157 ceph-mon[33614]: osdmap e30: 5 total, 4 up, 5 in 2024-05-09T08:26:52.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:51 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:26:52.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:51 smithi157 ceph-mon[33614]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:26:52.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:51 smithi157 ceph-mon[33614]: from='client.? 172.21.15.110:0/3205405334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:26:52.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:51 smithi110 ceph-mon[33622]: from='client.? 172.21.15.110:0/3762827165' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b67c4b46-7857-48cd-8737-d7cbb37aabaf"}]: dispatch 2024-05-09T08:26:52.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:51 smithi110 ceph-mon[33622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b67c4b46-7857-48cd-8737-d7cbb37aabaf"}]: dispatch 2024-05-09T08:26:52.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:51 smithi110 ceph-mon[33622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b67c4b46-7857-48cd-8737-d7cbb37aabaf"}]': finished 2024-05-09T08:26:52.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:51 smithi110 ceph-mon[33622]: osdmap e30: 5 total, 4 up, 5 in 2024-05-09T08:26:52.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:51 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:26:52.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:51 smithi110 ceph-mon[33622]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:26:52.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:51 smithi110 ceph-mon[33622]: from='client.? 172.21.15.110:0/3205405334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:26:54.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:54 smithi107 ceph-mon[30122]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-09T08:26:54.832 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:54 smithi110 ceph-mon[33622]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-09T08:26:54.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:54 smithi157 ceph-mon[33614]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-09T08:26:56.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:56 smithi110 ceph-mon[33622]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-09T08:26:56.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:56 smithi107 ceph-mon[30122]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-09T08:26:56.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:56 smithi157 ceph-mon[33614]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-09T08:26:58.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:58 smithi107 ceph-mon[30122]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:26:58.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-09T08:26:58.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:58.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:58 smithi157 ceph-mon[33614]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:26:58.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-09T08:26:58.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:58.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:58 smithi110 ceph-mon[33622]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:26:58.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-09T08:26:58.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:26:59.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:26:59 smithi110 ceph-mon[33622]: Deploying daemon osd.4 on smithi110 2024-05-09T08:26:59.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:26:59 smithi107 ceph-mon[30122]: Deploying daemon osd.4 on smithi110 2024-05-09T08:26:59.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:26:59 smithi157 ceph-mon[33614]: Deploying daemon osd.4 on smithi110 2024-05-09T08:27:00.550 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:00 smithi110 ceph-mon[33622]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-09T08:27:00.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:00 smithi107 ceph-mon[30122]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-09T08:27:00.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:00 smithi157 ceph-mon[33614]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-09T08:27:02.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:02 smithi110 ceph-mon[33622]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-05-09T08:27:02.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:02 smithi107 ceph-mon[30122]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-05-09T08:27:02.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:02 smithi157 ceph-mon[33614]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-05-09T08:27:04.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:04 smithi110 ceph-mon[33622]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-09T08:27:04.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:04 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:04.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:04 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:04.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:04 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:04.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:04 smithi107 ceph-mon[30122]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-09T08:27:04.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:04 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:04.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:04 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:04.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:04 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:04.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:04 smithi157 ceph-mon[33614]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-09T08:27:04.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:04 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:04.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:04 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:04.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:04 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:06.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:06 smithi107 ceph-mon[30122]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:27:06.831 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:06 smithi110 ceph-mon[33622]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:27:06.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:06 smithi157 ceph-mon[33614]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:27:06.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:27:06.886+0000 7f9d7e7fc700 1 -- 172.21.15.110:0/3343373514 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9d8810a4c0 con 0x7f9d740a2ca0 2024-05-09T08:27:06.886 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 4 on host 'smithi110' 2024-05-09T08:27:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:27:06.888+0000 7f9d8f70b700 1 -- 172.21.15.110:0/3343373514 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9d740a2ca0 msgr2=0x7f9d740a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:27:06.888+0000 7f9d8f70b700 1 --2- 172.21.15.110:0/3343373514 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9d740a2ca0 0x7f9d740a5160 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9d78005950 tx=0x7f9d780058e0 comp rx=0 tx=0).stop 2024-05-09T08:27:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:27:06.888+0000 7f9d8f70b700 1 -- 172.21.15.110:0/3343373514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d88102380 msgr2=0x7f9d8819a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:27:06.888+0000 7f9d8f70b700 1 --2- 172.21.15.110:0/3343373514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d88102380 0x7f9d8819a7d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9d8400fa00 tx=0x7f9d8400fae0 comp rx=0 tx=0).stop 2024-05-09T08:27:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:27:06.888+0000 7f9d8f70b700 1 -- 172.21.15.110:0/3343373514 shutdown_connections 2024-05-09T08:27:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:27:06.888+0000 7f9d8f70b700 1 --2- 172.21.15.110:0/3343373514 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9d740a2ca0 0x7f9d740a5160 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:27:06.888+0000 7f9d8f70b700 1 --2- 172.21.15.110:0/3343373514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d88103f20 0x7f9d881a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:27:06.888+0000 7f9d8f70b700 1 --2- 172.21.15.110:0/3343373514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d88102380 0x7f9d8819a7d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:27:06.888+0000 7f9d8f70b700 1 --2- 172.21.15.110:0/3343373514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d88103580 0x7f9d8819ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:27:06.888+0000 7f9d8f70b700 1 -- 172.21.15.110:0/3343373514 >> 172.21.15.110:0/3343373514 conn(0x7f9d880fd920 msgr2=0x7f9d88108da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:06.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:27:06.889+0000 7f9d8f70b700 1 -- 172.21.15.110:0/3343373514 shutdown_connections 2024-05-09T08:27:06.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:27:06.889+0000 7f9d8f70b700 1 -- 172.21.15.110:0/3343373514 wait complete. 2024-05-09T08:27:07.871 DEBUG:teuthology.orchestra.run.smithi110:osd.4> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.4.service 2024-05-09T08:27:07.873 INFO:tasks.cephadm:Deploying osd.5 on smithi157 with /dev/vg_nvme/lv_3... 2024-05-09T08:27:07.873 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-09T08:27:08.111 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:27:08.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:07 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:08.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:07 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:08.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:07 smithi110 ceph-mon[33622]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:27:08.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:07 smithi110 ceph-mon[33622]: from='osd.4 [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:27:08.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:07 smithi110 ceph-mon[33622]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:27:08.234 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:07 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:08.234 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:07 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:08.234 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:07 smithi157 ceph-mon[33614]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:27:08.234 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:07 smithi157 ceph-mon[33614]: from='osd.4 [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:27:08.234 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:07 smithi157 ceph-mon[33614]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:27:08.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:07 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:08.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:07 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:08.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:07 smithi107 ceph-mon[30122]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:27:08.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:07 smithi107 ceph-mon[30122]: from='osd.4 [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:27:08.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:07 smithi107 ceph-mon[30122]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T08:27:09.239 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:08 smithi157 ceph-mon[33614]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-09T08:27:09.239 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:08 smithi157 ceph-mon[33614]: osdmap e31: 5 total, 4 up, 5 in 2024-05-09T08:27:09.239 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:08 smithi157 ceph-mon[33614]: from='osd.4 [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:27:09.240 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:08 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:27:09.240 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:08 smithi157 ceph-mon[33614]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:27:09.240 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:08 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:09.240 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:08 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:09.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:08 smithi107 ceph-mon[30122]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-09T08:27:09.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:08 smithi107 ceph-mon[30122]: osdmap e31: 5 total, 4 up, 5 in 2024-05-09T08:27:09.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:08 smithi107 ceph-mon[30122]: from='osd.4 [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:27:09.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:08 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:27:09.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:08 smithi107 ceph-mon[30122]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:27:09.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:08 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:09.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:08 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:08 smithi110 ceph-mon[33622]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-09T08:27:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:08 smithi110 ceph-mon[33622]: osdmap e31: 5 total, 4 up, 5 in 2024-05-09T08:27:09.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:08 smithi110 ceph-mon[33622]: from='osd.4 [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:27:09.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:08 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:27:09.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:08 smithi110 ceph-mon[33622]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:27:09.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:08 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:09.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:08 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:09.725 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:27:09 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-4[44837]: 2024-05-09T08:27:09.423+0000 7f8222418700 -1 osd.4 0 waiting for initial osdmap 2024-05-09T08:27:09.725 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:27:09 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-4[44837]: 2024-05-09T08:27:09.429+0000 7f821a203700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:27:10.055 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:09 smithi110 ceph-mon[33622]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-05-09T08:27:10.056 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:09 smithi110 ceph-mon[33622]: osdmap e32: 5 total, 4 up, 5 in 2024-05-09T08:27:10.056 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:09 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:27:10.056 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:09 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:27:10.056 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:09 smithi110 ceph-mon[33622]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:27:10.056 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:09 smithi110 ceph-mon[33622]: from='osd.4 ' entity='osd.4' 2024-05-09T08:27:10.056 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:09 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:27:10.123 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-09T08:27:10.123 INFO:teuthology.orchestra.run.smithi157.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-09T08:27:10.123 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10+0 records in 2024-05-09T08:27:10.123 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records out 2024-05-09T08:27:10.123 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00838265 s, 1.3 GB/s 2024-05-09T08:27:10.123 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping successful for: 2024-05-09T08:27:10.123 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:09 smithi157 ceph-mon[33614]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-05-09T08:27:10.124 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:09 smithi157 ceph-mon[33614]: osdmap e32: 5 total, 4 up, 5 in 2024-05-09T08:27:10.124 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:09 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:27:10.124 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:09 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:27:10.124 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:09 smithi157 ceph-mon[33614]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:27:10.124 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:09 smithi157 ceph-mon[33614]: from='osd.4 ' entity='osd.4' 2024-05-09T08:27:10.124 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:09 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:27:10.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:09 smithi107 ceph-mon[30122]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-05-09T08:27:10.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:09 smithi107 ceph-mon[30122]: osdmap e32: 5 total, 4 up, 5 in 2024-05-09T08:27:10.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:09 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:27:10.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:09 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:27:10.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:09 smithi107 ceph-mon[30122]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-05-09T08:27:10.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:09 smithi107 ceph-mon[30122]: from='osd.4 ' entity='osd.4' 2024-05-09T08:27:10.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:09 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:27:10.768 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi157:vg_nvme/lv_3 2024-05-09T08:27:11.010 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:27:11.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:11 smithi110 ceph-mon[33622]: purged_snaps scrub starts 2024-05-09T08:27:11.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:11 smithi110 ceph-mon[33622]: purged_snaps scrub ok 2024-05-09T08:27:11.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:11 smithi110 ceph-mon[33622]: osd.4 [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505] boot 2024-05-09T08:27:11.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:11 smithi110 ceph-mon[33622]: osdmap e33: 5 total, 5 up, 5 in 2024-05-09T08:27:11.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:27:11.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:11.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:11.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:27:11.624 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:27:11.624 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:11.624 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:11.624 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:11.624 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:11.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:11 smithi107 ceph-mon[30122]: purged_snaps scrub starts 2024-05-09T08:27:11.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:11 smithi107 ceph-mon[30122]: purged_snaps scrub ok 2024-05-09T08:27:11.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:11 smithi107 ceph-mon[30122]: osd.4 [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505] boot 2024-05-09T08:27:11.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:11 smithi107 ceph-mon[30122]: osdmap e33: 5 total, 5 up, 5 in 2024-05-09T08:27:11.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:27:11.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:11.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:11.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:27:11.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:27:11.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:11.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:11.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:11.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:11.795 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:11 smithi157 ceph-mon[33614]: purged_snaps scrub starts 2024-05-09T08:27:11.795 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:11 smithi157 ceph-mon[33614]: purged_snaps scrub ok 2024-05-09T08:27:11.795 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:11 smithi157 ceph-mon[33614]: osd.4 [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505] boot 2024-05-09T08:27:11.795 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:11 smithi157 ceph-mon[33614]: osdmap e33: 5 total, 5 up, 5 in 2024-05-09T08:27:11.795 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:27:11.795 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:11.796 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:11.796 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:27:11.796 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:27:11.796 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:11.796 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:11.796 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:11.796 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:12.631 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.629+0000 7f02fb21b700 1 -- 172.21.15.157:0/473129957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f4103f20 msgr2=0x7f02f410a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:12.631 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.629+0000 7f02fb21b700 1 --2- 172.21.15.157:0/473129957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f4103f20 0x7f02f410a7d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00b0a0 tx=0x7f02ec00b3b0 comp rx=0 tx=0).stop 2024-05-09T08:27:12.631 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.629+0000 7f02fb21b700 1 -- 172.21.15.157:0/473129957 shutdown_connections 2024-05-09T08:27:12.631 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.629+0000 7f02fb21b700 1 --2- 172.21.15.157:0/473129957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f4103f20 0x7f02f410a7d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:12.631 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.629+0000 7f02fb21b700 1 --2- 172.21.15.157:0/473129957 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02f4102380 0x7f02f41027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:12.631 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.629+0000 7f02fb21b700 1 --2- 172.21.15.157:0/473129957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f4103580 0x7f02f41039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:12.631 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.629+0000 7f02fb21b700 1 -- 172.21.15.157:0/473129957 >> 172.21.15.157:0/473129957 conn(0x7f02f40fd920 msgr2=0x7f02f40ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:12.631 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.629+0000 7f02fb21b700 1 -- 172.21.15.157:0/473129957 shutdown_connections 2024-05-09T08:27:12.631 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.629+0000 7f02fb21b700 1 -- 172.21.15.157:0/473129957 wait complete. 2024-05-09T08:27:12.631 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.629+0000 7f02fb21b700 1 Processor -- start 2024-05-09T08:27:12.631 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.629+0000 7f02fb21b700 1 -- start start 2024-05-09T08:27:12.632 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02fb21b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f4102380 0x7f02f4198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:12.632 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02fb21b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02f4103580 0x7f02f41989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:12.632 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02fb21b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f4103f20 0x7f02f419fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:12.632 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02fb21b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f02f410de20 con 0x7f02f4103f20 2024-05-09T08:27:12.632 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02fb21b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f02f410dca0 con 0x7f02f4103580 2024-05-09T08:27:12.632 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02fb21b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f02f410dfa0 con 0x7f02f4102380 2024-05-09T08:27:12.632 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02f8fb7700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f4102380 0x7f02f4198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:12.632 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02f8fb7700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f4102380 0x7f02f4198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:57102/0 (socket says 172.21.15.157:57102) 2024-05-09T08:27:12.632 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02f8fb7700 1 -- 172.21.15.157:0/1666958486 learned_addr learned my addr 172.21.15.157:0/1666958486 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-05-09T08:27:12.632 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02f97b8700 1 --2- 172.21.15.157:0/1666958486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f4103f20 0x7f02f419fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:12.632 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02f3fff700 1 --2- 172.21.15.157:0/1666958486 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02f4103580 0x7f02f41989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:12.632 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02f8fb7700 1 -- 172.21.15.157:0/1666958486 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02f4103580 msgr2=0x7f02f41989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:12.633 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02f8fb7700 1 --2- 172.21.15.157:0/1666958486 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02f4103580 0x7f02f41989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:12.633 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02f8fb7700 1 -- 172.21.15.157:0/1666958486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f4103f20 msgr2=0x7f02f419fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:12.633 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02f8fb7700 1 --2- 172.21.15.157:0/1666958486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f4103f20 0x7f02f419fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:12.633 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02f8fb7700 1 -- 172.21.15.157:0/1666958486 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f41a00c0 con 0x7f02f4102380 2024-05-09T08:27:12.633 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02f3fff700 1 --2- 172.21.15.157:0/1666958486 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02f4103580 0x7f02f41989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:27:12.633 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02f8fb7700 1 --2- 172.21.15.157:0/1666958486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f4102380 0x7f02f4198460 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f02e800e980 tx=0x7f02e800ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:12.633 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02f1ffb700 1 -- 172.21.15.157:0/1666958486 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e800cbd0 con 0x7f02f4102380 2024-05-09T08:27:12.633 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02f1ffb700 1 -- 172.21.15.157:0/1666958486 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02e80044e0 con 0x7f02f4102380 2024-05-09T08:27:12.633 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02f1ffb700 1 -- 172.21.15.157:0/1666958486 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8010430 con 0x7f02f4102380 2024-05-09T08:27:12.633 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.630+0000 7f02fb21b700 1 -- 172.21.15.157:0/1666958486 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f41a03b0 con 0x7f02f4102380 2024-05-09T08:27:12.633 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.631+0000 7f02fb21b700 1 -- 172.21.15.157:0/1666958486 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f41a08f0 con 0x7f02f4102380 2024-05-09T08:27:12.634 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.631+0000 7f02fb21b700 1 -- 172.21.15.157:0/1666958486 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02f4069010 con 0x7f02f4102380 2024-05-09T08:27:12.634 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.632+0000 7f02f1ffb700 1 -- 172.21.15.157:0/1666958486 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f02e8003f30 con 0x7f02f4102380 2024-05-09T08:27:12.634 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.633+0000 7f02f1ffb700 1 --2- 172.21.15.157:0/1666958486 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f02e40a2ca0 0x7f02e40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:12.635 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.633+0000 7f02f1ffb700 1 -- 172.21.15.157:0/1666958486 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f02e80ce340 con 0x7f02f4102380 2024-05-09T08:27:12.635 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.633+0000 7f02f3fff700 1 --2- 172.21.15.157:0/1666958486 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f02e40a2ca0 0x7f02e40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:12.635 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.634+0000 7f02f3fff700 1 --2- 172.21.15.157:0/1666958486 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f02e40a2ca0 0x7f02e40a5160 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f02e0009cb0 tx=0x7f02e0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:12.637 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.638+0000 7f02f1ffb700 1 -- 172.21.15.157:0/1666958486 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f02e809c310 con 0x7f02f4102380 2024-05-09T08:27:12.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:12 smithi107 ceph-mon[30122]: Detected new or changed devices on smithi110 2024-05-09T08:27:12.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:12 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi110 to 8544M 2024-05-09T08:27:12.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:12 smithi107 ceph-mon[30122]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:12.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:12 smithi107 ceph-mon[30122]: osdmap e34: 5 total, 5 up, 5 in 2024-05-09T08:27:12.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:12 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:12.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:12 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:12.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:12 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:12.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:12 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:12.840 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:12.841+0000 7f02fb21b700 1 -- 172.21.15.157:0/1666958486 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f02f4062760 con 0x7f02e40a2ca0 2024-05-09T08:27:12.841 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:12 smithi157 ceph-mon[33614]: Detected new or changed devices on smithi110 2024-05-09T08:27:12.841 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:12 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi110 to 8544M 2024-05-09T08:27:12.841 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:12 smithi157 ceph-mon[33614]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:12.841 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:12 smithi157 ceph-mon[33614]: osdmap e34: 5 total, 5 up, 5 in 2024-05-09T08:27:12.841 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:12 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:12.841 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:12 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:12.841 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:12 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:12.841 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:12 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:12.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:12 smithi110 ceph-mon[33622]: Detected new or changed devices on smithi110 2024-05-09T08:27:12.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:12 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi110 to 8544M 2024-05-09T08:27:12.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:12 smithi110 ceph-mon[33622]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:12.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:12 smithi110 ceph-mon[33622]: osdmap e34: 5 total, 5 up, 5 in 2024-05-09T08:27:12.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:12 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:12.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:12 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:12.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:12 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:12.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:12 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:13.719 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:13 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:27:13.719 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:13 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:27:13.719 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:13 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:13.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:13 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:27:13.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:13 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:27:13.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:13 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:13.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:13 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:27:13.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:13 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:27:13.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:13 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:14.650 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:14 smithi157 ceph-mon[33614]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:27:14.650 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:14 smithi157 ceph-mon[33614]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:14.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:14 smithi107 ceph-mon[30122]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:27:14.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:14 smithi107 ceph-mon[30122]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:14.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:14 smithi110 ceph-mon[33622]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:27:14.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:14 smithi110 ceph-mon[33622]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:16.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:16 smithi107 ceph-mon[30122]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:16.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:16 smithi107 ceph-mon[30122]: from='client.? 172.21.15.157:0/1734578900' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f4cfd0f-4faa-4697-aa3b-ef48a801be37"}]: dispatch 2024-05-09T08:27:16.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:16 smithi107 ceph-mon[30122]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f4cfd0f-4faa-4697-aa3b-ef48a801be37"}]: dispatch 2024-05-09T08:27:16.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:16 smithi107 ceph-mon[30122]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f4cfd0f-4faa-4697-aa3b-ef48a801be37"}]': finished 2024-05-09T08:27:16.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:16 smithi107 ceph-mon[30122]: osdmap e35: 6 total, 5 up, 6 in 2024-05-09T08:27:16.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:16 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:27:16.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:16 smithi107 ceph-mon[30122]: from='client.? 172.21.15.157:0/1958838150' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:27:16.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:16 smithi157 ceph-mon[33614]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:16.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:16 smithi157 ceph-mon[33614]: from='client.? 172.21.15.157:0/1734578900' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f4cfd0f-4faa-4697-aa3b-ef48a801be37"}]: dispatch 2024-05-09T08:27:16.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:16 smithi157 ceph-mon[33614]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f4cfd0f-4faa-4697-aa3b-ef48a801be37"}]: dispatch 2024-05-09T08:27:16.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:16 smithi157 ceph-mon[33614]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f4cfd0f-4faa-4697-aa3b-ef48a801be37"}]': finished 2024-05-09T08:27:16.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:16 smithi157 ceph-mon[33614]: osdmap e35: 6 total, 5 up, 6 in 2024-05-09T08:27:16.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:16 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:27:16.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:16 smithi157 ceph-mon[33614]: from='client.? 172.21.15.157:0/1958838150' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:27:16.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:16 smithi110 ceph-mon[33622]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:16.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:16 smithi110 ceph-mon[33622]: from='client.? 172.21.15.157:0/1734578900' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f4cfd0f-4faa-4697-aa3b-ef48a801be37"}]: dispatch 2024-05-09T08:27:16.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:16 smithi110 ceph-mon[33622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f4cfd0f-4faa-4697-aa3b-ef48a801be37"}]: dispatch 2024-05-09T08:27:16.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:16 smithi110 ceph-mon[33622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f4cfd0f-4faa-4697-aa3b-ef48a801be37"}]': finished 2024-05-09T08:27:16.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:16 smithi110 ceph-mon[33622]: osdmap e35: 6 total, 5 up, 6 in 2024-05-09T08:27:16.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:16 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:27:16.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:16 smithi110 ceph-mon[33622]: from='client.? 172.21.15.157:0/1958838150' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:27:18.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:18 smithi107 ceph-mon[30122]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:18.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:18 smithi157 ceph-mon[33614]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:18.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:18 smithi110 ceph-mon[33622]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:20.527 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:20 smithi157 ceph-mon[33614]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:20.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:20 smithi107 ceph-mon[30122]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:20.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:20 smithi110 ceph-mon[33622]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:22.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:22 smithi107 ceph-mon[30122]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:22.812 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:22 smithi157 ceph-mon[33614]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:22.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:22 smithi110 ceph-mon[33622]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:23.710 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:23 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-09T08:27:23.710 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:23 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:23.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:23 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-09T08:27:23.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:23 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:23.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:23 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-09T08:27:23.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:23 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:24.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:24 smithi107 ceph-mon[30122]: Deploying daemon osd.5 on smithi157 2024-05-09T08:27:24.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:24 smithi107 ceph-mon[30122]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:24.802 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:24 smithi157 ceph-mon[33614]: Deploying daemon osd.5 on smithi157 2024-05-09T08:27:24.802 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:24 smithi157 ceph-mon[33614]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:24.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:24 smithi110 ceph-mon[33622]: Deploying daemon osd.5 on smithi157 2024-05-09T08:27:24.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:24 smithi110 ceph-mon[33622]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:26.772 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:26 smithi157 ceph-mon[33614]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:26 smithi107 ceph-mon[30122]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:26 smithi110 ceph-mon[33622]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:28.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:28 smithi157 ceph-mon[33614]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:28.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:28 smithi107 ceph-mon[30122]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:28.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:28 smithi110 ceph-mon[33622]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:29.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:29 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:29.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:29 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:29.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:29 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:29.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:29 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:29.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:29 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:29.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:29 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:29.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:29 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:29.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:29 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:29.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:29 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:30.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:30 smithi107 ceph-mon[30122]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:30.837 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:30 smithi157 ceph-mon[33614]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:30.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:30 smithi110 ceph-mon[33622]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:32.409 INFO:teuthology.orchestra.run.smithi157.stdout:Created osd(s) 5 on host 'smithi157' 2024-05-09T08:27:32.409 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:32.408+0000 7f02f1ffb700 1 -- 172.21.15.157:0/1666958486 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f02f4062760 con 0x7f02e40a2ca0 2024-05-09T08:27:32.411 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:32.410+0000 7f02fb21b700 1 -- 172.21.15.157:0/1666958486 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f02e40a2ca0 msgr2=0x7f02e40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:32.411 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:32.410+0000 7f02fb21b700 1 --2- 172.21.15.157:0/1666958486 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f02e40a2ca0 0x7f02e40a5160 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f02e0009cb0 tx=0x7f02e0009260 comp rx=0 tx=0).stop 2024-05-09T08:27:32.411 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:32.410+0000 7f02fb21b700 1 -- 172.21.15.157:0/1666958486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f4102380 msgr2=0x7f02f4198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:32.411 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:32.410+0000 7f02fb21b700 1 --2- 172.21.15.157:0/1666958486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f4102380 0x7f02f4198460 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f02e800e980 tx=0x7f02e800ed40 comp rx=0 tx=0).stop 2024-05-09T08:27:32.411 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:32.410+0000 7f02fb21b700 1 -- 172.21.15.157:0/1666958486 shutdown_connections 2024-05-09T08:27:32.411 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:32.410+0000 7f02fb21b700 1 --2- 172.21.15.157:0/1666958486 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f02e40a2ca0 0x7f02e40a5160 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:32.412 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:32.410+0000 7f02fb21b700 1 --2- 172.21.15.157:0/1666958486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f4103f20 0x7f02f419fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:32.412 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:32.410+0000 7f02fb21b700 1 --2- 172.21.15.157:0/1666958486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f4102380 0x7f02f4198460 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:32.412 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:32.410+0000 7f02fb21b700 1 --2- 172.21.15.157:0/1666958486 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02f4103580 0x7f02f41989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:32.412 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:32.410+0000 7f02fb21b700 1 -- 172.21.15.157:0/1666958486 >> 172.21.15.157:0/1666958486 conn(0x7f02f40fd920 msgr2=0x7f02f4106900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:32.412 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:32.410+0000 7f02fb21b700 1 -- 172.21.15.157:0/1666958486 shutdown_connections 2024-05-09T08:27:32.412 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:27:32.411+0000 7f02fb21b700 1 -- 172.21.15.157:0/1666958486 wait complete. 2024-05-09T08:27:32.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:32 smithi157 ceph-mon[33614]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:32.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:32.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:32.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:32.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:32.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:32.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:32.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:32.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:32.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:32 smithi157 ceph-mon[33614]: from='osd.5 [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:27:32.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:32 smithi157 ceph-mon[33614]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:27:32.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:32 smithi107 ceph-mon[30122]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:32.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:32.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:32.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:32.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:32.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:32.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:32.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:32.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:32.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:32 smithi107 ceph-mon[30122]: from='osd.5 [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:27:32.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:32 smithi107 ceph-mon[30122]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:27:32.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:32 smithi110 ceph-mon[33622]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:32.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:32.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:32.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:32.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:32.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:32.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:32.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:32.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:32.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:32 smithi110 ceph-mon[33622]: from='osd.5 [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:27:32.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:32 smithi110 ceph-mon[33622]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T08:27:33.136 DEBUG:teuthology.orchestra.run.smithi157:osd.5> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.5.service 2024-05-09T08:27:33.139 INFO:tasks.cephadm:Deploying osd.6 on smithi107 with /dev/vg_nvme/lv_2... 2024-05-09T08:27:33.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-09T08:27:33.388 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:27:34.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:34 smithi157 ceph-mon[33614]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-09T08:27:34.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:34 smithi157 ceph-mon[33614]: from='osd.5 [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:27:34.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:34 smithi157 ceph-mon[33614]: osdmap e36: 6 total, 5 up, 6 in 2024-05-09T08:27:34.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:34 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:27:34.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:34 smithi157 ceph-mon[33614]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:27:34.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:34 smithi157 ceph-mon[33614]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:34.696 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:34 smithi107 ceph-mon[30122]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-09T08:27:34.696 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:34 smithi107 ceph-mon[30122]: from='osd.5 [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:27:34.696 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:34 smithi107 ceph-mon[30122]: osdmap e36: 6 total, 5 up, 6 in 2024-05-09T08:27:34.696 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:34 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:27:34.696 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:34 smithi107 ceph-mon[30122]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:27:34.696 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:34 smithi107 ceph-mon[30122]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:34.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:34 smithi110 ceph-mon[33622]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-09T08:27:34.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:34 smithi110 ceph-mon[33622]: from='osd.5 [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:27:34.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:34 smithi110 ceph-mon[33622]: osdmap e36: 6 total, 5 up, 6 in 2024-05-09T08:27:34.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:34 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:27:34.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:34 smithi110 ceph-mon[33622]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:27:34.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:34 smithi110 ceph-mon[33622]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-05-09T08:27:35.094 INFO:journalctl@ceph.osd.5.smithi157.stdout:May 09 08:27:34 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-5[45403]: 2024-05-09T08:27:34.905+0000 7fa9f6c44700 -1 osd.5 0 waiting for initial osdmap 2024-05-09T08:27:35.095 INFO:journalctl@ceph.osd.5.smithi157.stdout:May 09 08:27:34 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-5[45403]: 2024-05-09T08:27:34.912+0000 7fa9eea2f700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:27:35.416 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-09T08:27:35.416 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-09T08:27:35.416 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-05-09T08:27:35.416 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-05-09T08:27:35.416 INFO:teuthology.orchestra.run.smithi107.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0133771 s, 784 MB/s 2024-05-09T08:27:35.417 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 2024-05-09T08:27:35.417 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-05-09T08:27:35.716 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:35 smithi107 ceph-mon[30122]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-05-09T08:27:35.716 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:35 smithi107 ceph-mon[30122]: osdmap e37: 6 total, 5 up, 6 in 2024-05-09T08:27:35.716 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:35 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:27:35.717 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:35 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:27:35.717 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:35 smithi107 ceph-mon[30122]: from='osd.5 ' entity='osd.5' 2024-05-09T08:27:35.717 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:35 smithi107 ceph-mon[30122]: osd.5 [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420] boot 2024-05-09T08:27:35.717 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:35 smithi107 ceph-mon[30122]: osdmap e38: 6 total, 6 up, 6 in 2024-05-09T08:27:35.717 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:35 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:27:35.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:35 smithi157 ceph-mon[33614]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-05-09T08:27:35.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:35 smithi157 ceph-mon[33614]: osdmap e37: 6 total, 5 up, 6 in 2024-05-09T08:27:35.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:35 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:27:35.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:35 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:27:35.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:35 smithi157 ceph-mon[33614]: from='osd.5 ' entity='osd.5' 2024-05-09T08:27:35.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:35 smithi157 ceph-mon[33614]: osd.5 [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420] boot 2024-05-09T08:27:35.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:35 smithi157 ceph-mon[33614]: osdmap e38: 6 total, 6 up, 6 in 2024-05-09T08:27:35.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:35 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:27:35.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:35 smithi110 ceph-mon[33622]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-05-09T08:27:35.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:35 smithi110 ceph-mon[33622]: osdmap e37: 6 total, 5 up, 6 in 2024-05-09T08:27:35.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:35 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:27:35.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:35 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:27:35.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:35 smithi110 ceph-mon[33622]: from='osd.5 ' entity='osd.5' 2024-05-09T08:27:35.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:35 smithi110 ceph-mon[33622]: osd.5 [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420] boot 2024-05-09T08:27:35.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:35 smithi110 ceph-mon[33622]: osdmap e38: 6 total, 6 up, 6 in 2024-05-09T08:27:35.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:35 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:27:36.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi107:vg_nvme/lv_2 2024-05-09T08:27:36.324 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:27:36.693 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:36 smithi107 ceph-mon[30122]: purged_snaps scrub starts 2024-05-09T08:27:36.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:36 smithi107 ceph-mon[30122]: purged_snaps scrub ok 2024-05-09T08:27:36.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:36 smithi107 ceph-mon[30122]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 486 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:36.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:36 smithi107 ceph-mon[30122]: osdmap e39: 6 total, 6 up, 6 in 2024-05-09T08:27:36.772 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:36 smithi110 ceph-mon[33622]: purged_snaps scrub starts 2024-05-09T08:27:36.773 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:36 smithi110 ceph-mon[33622]: purged_snaps scrub ok 2024-05-09T08:27:36.773 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:36 smithi110 ceph-mon[33622]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 486 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:36.773 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:36 smithi110 ceph-mon[33622]: osdmap e39: 6 total, 6 up, 6 in 2024-05-09T08:27:36.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:36 smithi157 ceph-mon[33614]: purged_snaps scrub starts 2024-05-09T08:27:36.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:36 smithi157 ceph-mon[33614]: purged_snaps scrub ok 2024-05-09T08:27:36.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:36 smithi157 ceph-mon[33614]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 486 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:36.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:36 smithi157 ceph-mon[33614]: osdmap e39: 6 total, 6 up, 6 in 2024-05-09T08:27:37.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:37 smithi107 ceph-mon[30122]: Detected new or changed devices on smithi157 2024-05-09T08:27:37.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:37 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:37.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:37 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:37.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:37 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:27:37.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:37 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:27:37.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:37 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi157 to 8544M 2024-05-09T08:27:37.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:37 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:37.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:37 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:37.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:37 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:37.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:37 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:37.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:37 smithi107 ceph-mon[30122]: osdmap e40: 6 total, 6 up, 6 in 2024-05-09T08:27:37.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:37 smithi107 ceph-mon[30122]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 486 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:37.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:37 smithi157 ceph-mon[33614]: Detected new or changed devices on smithi157 2024-05-09T08:27:37.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:37 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:37.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:37 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:37.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:37 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:27:37.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:37 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:27:37.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:37 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi157 to 8544M 2024-05-09T08:27:37.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:37 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:37.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:37 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:37.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:37 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:37.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:37 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:37.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:37 smithi157 ceph-mon[33614]: osdmap e40: 6 total, 6 up, 6 in 2024-05-09T08:27:37.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:37 smithi157 ceph-mon[33614]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 486 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:37.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:37 smithi110 ceph-mon[33622]: Detected new or changed devices on smithi157 2024-05-09T08:27:37.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:37 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:37.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:37 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:37.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:37 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:27:37.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:37 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:27:37.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:37 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi157 to 8544M 2024-05-09T08:27:37.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:37 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:37.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:37 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:37.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:37 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:37.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:37 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:37.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:37 smithi110 ceph-mon[33622]: osdmap e40: 6 total, 6 up, 6 in 2024-05-09T08:27:37.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:37 smithi110 ceph-mon[33622]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 486 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.047+0000 7fd77df29700 1 -- 172.21.15.107:0/3499280078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd778102380 msgr2=0x7fd7781027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.047+0000 7fd77df29700 1 --2- 172.21.15.107:0/3499280078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd778102380 0x7fd7781027a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd76800b0d0 tx=0x7fd76800b3e0 comp rx=0 tx=0).stop 2024-05-09T08:27:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.047+0000 7fd77df29700 1 -- 172.21.15.107:0/3499280078 shutdown_connections 2024-05-09T08:27:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.047+0000 7fd77df29700 1 --2- 172.21.15.107:0/3499280078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd778103f20 0x7fd77810a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.047+0000 7fd77df29700 1 --2- 172.21.15.107:0/3499280078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd778102380 0x7fd7781027a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.047+0000 7fd77df29700 1 --2- 172.21.15.107:0/3499280078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd778103580 0x7fd7781039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.047+0000 7fd77df29700 1 -- 172.21.15.107:0/3499280078 >> 172.21.15.107:0/3499280078 conn(0x7fd7780fd920 msgr2=0x7fd7780ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.047+0000 7fd77df29700 1 -- 172.21.15.107:0/3499280078 shutdown_connections 2024-05-09T08:27:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.047+0000 7fd77df29700 1 -- 172.21.15.107:0/3499280078 wait complete. 2024-05-09T08:27:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd77df29700 1 Processor -- start 2024-05-09T08:27:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd77df29700 1 -- start start 2024-05-09T08:27:38.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd77df29700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd778102380 0x7fd778198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:38.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd77df29700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd778103580 0x7fd7781989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:38.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd77df29700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd778103f20 0x7fd77819fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:38.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd77df29700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd77810de20 con 0x7fd778103f20 2024-05-09T08:27:38.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd77df29700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd77810dca0 con 0x7fd778102380 2024-05-09T08:27:38.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd77df29700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd77810dfa0 con 0x7fd778103580 2024-05-09T08:27:38.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd777fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd778103f20 0x7fd77819fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:38.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd7777fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd778102380 0x7fd778198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:38.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd777fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd778103f20 0x7fd77819fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57078/0 (socket says 172.21.15.107:57078) 2024-05-09T08:27:38.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd777fff700 1 -- 172.21.15.107:0/2693414952 learned_addr learned my addr 172.21.15.107:0/2693414952 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:27:38.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd776ffd700 1 --2- 172.21.15.107:0/2693414952 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd778103580 0x7fd7781989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:38.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd777fff700 1 -- 172.21.15.107:0/2693414952 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd778103580 msgr2=0x7fd7781989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:38.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd777fff700 1 --2- 172.21.15.107:0/2693414952 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd778103580 0x7fd7781989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:38.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd777fff700 1 -- 172.21.15.107:0/2693414952 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd778102380 msgr2=0x7fd778198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:38.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd777fff700 1 --2- 172.21.15.107:0/2693414952 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd778102380 0x7fd778198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:38.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd777fff700 1 -- 172.21.15.107:0/2693414952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7781a00f0 con 0x7fd778103f20 2024-05-09T08:27:38.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.048+0000 7fd7777fe700 1 --2- 172.21.15.107:0/2693414952 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd778102380 0x7fd778198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:27:38.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.049+0000 7fd777fff700 1 --2- 172.21.15.107:0/2693414952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd778103f20 0x7fd77819fa50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd77000d9b0 tx=0x7fd77000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:38.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.049+0000 7fd774ff9700 1 -- 172.21.15.107:0/2693414952 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd770017070 con 0x7fd778103f20 2024-05-09T08:27:38.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.049+0000 7fd774ff9700 1 -- 172.21.15.107:0/2693414952 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd770005020 con 0x7fd778103f20 2024-05-09T08:27:38.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.049+0000 7fd774ff9700 1 -- 172.21.15.107:0/2693414952 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd770009990 con 0x7fd778103f20 2024-05-09T08:27:38.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.049+0000 7fd77df29700 1 -- 172.21.15.107:0/2693414952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7781a03e0 con 0x7fd778103f20 2024-05-09T08:27:38.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.049+0000 7fd77df29700 1 -- 172.21.15.107:0/2693414952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7781a08d0 con 0x7fd778103f20 2024-05-09T08:27:38.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.050+0000 7fd77df29700 1 -- 172.21.15.107:0/2693414952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd778069010 con 0x7fd778103f20 2024-05-09T08:27:38.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.051+0000 7fd774ff9700 1 -- 172.21.15.107:0/2693414952 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd7700051d0 con 0x7fd778103f20 2024-05-09T08:27:38.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.052+0000 7fd774ff9700 1 --2- 172.21.15.107:0/2693414952 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd7600a2c20 0x7fd7600a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:27:38.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.052+0000 7fd7777fe700 1 --2- 172.21.15.107:0/2693414952 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd7600a2c20 0x7fd7600a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:27:38.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.052+0000 7fd774ff9700 1 -- 172.21.15.107:0/2693414952 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7fd7700cdfa0 con 0x7fd778103f20 2024-05-09T08:27:38.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.052+0000 7fd7777fe700 1 --2- 172.21.15.107:0/2693414952 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd7600a2c20 0x7fd7600a50e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd768009040 tx=0x7fd76800b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:27:38.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.057+0000 7fd774ff9700 1 -- 172.21.15.107:0/2693414952 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd770020020 con 0x7fd778103f20 2024-05-09T08:27:38.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:38.261+0000 7fd77df29700 1 -- 172.21.15.107:0/2693414952 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fd778062760 con 0x7fd7600a2c20 2024-05-09T08:27:38.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:38 smithi157 ceph-mon[33614]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:27:38.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:38 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:27:38.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:38 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:27:38.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:38 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:38.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:38 smithi157 ceph-mon[33614]: osdmap e41: 6 total, 6 up, 6 in 2024-05-09T08:27:38.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:38 smithi110 ceph-mon[33622]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:27:38.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:38 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:27:38.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:38 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:27:38.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:38 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:38.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:38 smithi110 ceph-mon[33622]: osdmap e41: 6 total, 6 up, 6 in 2024-05-09T08:27:38.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:38 smithi107 ceph-mon[30122]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:27:38.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:38 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:27:38.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:38 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:27:38.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:38 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:38.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:38 smithi107 ceph-mon[30122]: osdmap e41: 6 total, 6 up, 6 in 2024-05-09T08:27:40.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:40 smithi107 ceph-mon[30122]: osdmap e42: 6 total, 6 up, 6 in 2024-05-09T08:27:40.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:40 smithi107 ceph-mon[30122]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 486 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:40.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:40 smithi157 ceph-mon[33614]: osdmap e42: 6 total, 6 up, 6 in 2024-05-09T08:27:40.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:40 smithi157 ceph-mon[33614]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 486 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:40.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:40 smithi110 ceph-mon[33622]: osdmap e42: 6 total, 6 up, 6 in 2024-05-09T08:27:40.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:40 smithi110 ceph-mon[33622]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 486 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:41.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:41 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/4011934695' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0451766-461b-4d7e-8087-f2d066097c16"}]: dispatch 2024-05-09T08:27:41.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:41 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/4011934695' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0451766-461b-4d7e-8087-f2d066097c16"}]': finished 2024-05-09T08:27:41.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:41 smithi107 ceph-mon[30122]: osdmap e43: 7 total, 6 up, 7 in 2024-05-09T08:27:41.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:41 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:27:41.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:41 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/4011934695' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0451766-461b-4d7e-8087-f2d066097c16"}]: dispatch 2024-05-09T08:27:41.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:41 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/4011934695' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0451766-461b-4d7e-8087-f2d066097c16"}]': finished 2024-05-09T08:27:41.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:41 smithi157 ceph-mon[33614]: osdmap e43: 7 total, 6 up, 7 in 2024-05-09T08:27:41.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:41 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:27:41.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:41 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/4011934695' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0451766-461b-4d7e-8087-f2d066097c16"}]: dispatch 2024-05-09T08:27:41.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:41 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/4011934695' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0451766-461b-4d7e-8087-f2d066097c16"}]': finished 2024-05-09T08:27:41.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:41 smithi110 ceph-mon[33622]: osdmap e43: 7 total, 6 up, 7 in 2024-05-09T08:27:41.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:41 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:27:42.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:42 smithi107 ceph-mon[30122]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:42.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:42 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1588558805' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:27:42.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:42 smithi157 ceph-mon[33614]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:42.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:42 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1588558805' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:27:42.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:42 smithi110 ceph-mon[33622]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:42.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:42 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1588558805' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:27:44.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:44 smithi107 ceph-mon[30122]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-09T08:27:44.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:44 smithi157 ceph-mon[33614]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-09T08:27:44.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:44 smithi110 ceph-mon[33622]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-09T08:27:46.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:46 smithi107 ceph-mon[30122]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-05-09T08:27:46.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:46 smithi157 ceph-mon[33614]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-05-09T08:27:46.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:46 smithi110 ceph-mon[33622]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-05-09T08:27:48.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:48 smithi107 ceph-mon[30122]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:27:48.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:48 smithi157 ceph-mon[33614]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:27:48.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:48 smithi110 ceph-mon[33622]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:27:49.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:49 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-09T08:27:49.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:49 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:49.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:49 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-09T08:27:49.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:49 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:49.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:49 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-09T08:27:49.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:49 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:50.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:50 smithi107 ceph-mon[30122]: Deploying daemon osd.6 on smithi107 2024-05-09T08:27:50.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:50 smithi107 ceph-mon[30122]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-09T08:27:50.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:50 smithi157 ceph-mon[33614]: Deploying daemon osd.6 on smithi107 2024-05-09T08:27:50.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:50 smithi157 ceph-mon[33614]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-09T08:27:50.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:50 smithi110 ceph-mon[33622]: Deploying daemon osd.6 on smithi107 2024-05-09T08:27:50.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:50 smithi110 ceph-mon[33622]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-09T08:27:52.739 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:52 smithi107 ceph-mon[30122]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-05-09T08:27:52.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:52 smithi157 ceph-mon[33614]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-05-09T08:27:52.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:52 smithi110 ceph-mon[33622]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-05-09T08:27:54.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:54 smithi107 ceph-mon[30122]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-09T08:27:54.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:54.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:54.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:54 smithi157 ceph-mon[33614]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-09T08:27:54.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:54.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:54.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:54 smithi110 ceph-mon[33622]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-09T08:27:54.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:54 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:54.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:54 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:55.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:55.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:55.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:56.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:56 smithi107 ceph-mon[30122]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:56.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:56 smithi157 ceph-mon[33614]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:56.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:56 smithi110 ceph-mon[33622]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:58.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:58.525+0000 7fd774ff9700 1 -- 172.21.15.107:0/2693414952 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd778062760 con 0x7fd7600a2c20 2024-05-09T08:27:58.526 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 6 on host 'smithi107' 2024-05-09T08:27:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:58.527+0000 7fd77df29700 1 -- 172.21.15.107:0/2693414952 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd7600a2c20 msgr2=0x7fd7600a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:58.527+0000 7fd77df29700 1 --2- 172.21.15.107:0/2693414952 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd7600a2c20 0x7fd7600a50e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd768009040 tx=0x7fd76800b040 comp rx=0 tx=0).stop 2024-05-09T08:27:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:58.527+0000 7fd77df29700 1 -- 172.21.15.107:0/2693414952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd778103f20 msgr2=0x7fd77819fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:27:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:58.527+0000 7fd77df29700 1 --2- 172.21.15.107:0/2693414952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd778103f20 0x7fd77819fa50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd77000d9b0 tx=0x7fd77000dcc0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:58.527+0000 7fd77df29700 1 -- 172.21.15.107:0/2693414952 shutdown_connections 2024-05-09T08:27:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:58.527+0000 7fd77df29700 1 --2- 172.21.15.107:0/2693414952 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd7600a2c20 0x7fd7600a50e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:58.527+0000 7fd77df29700 1 --2- 172.21.15.107:0/2693414952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd778103f20 0x7fd77819fa50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:58.528+0000 7fd77df29700 1 --2- 172.21.15.107:0/2693414952 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd778102380 0x7fd778198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:58.528+0000 7fd77df29700 1 --2- 172.21.15.107:0/2693414952 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd778103580 0x7fd7781989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:27:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:58.528+0000 7fd77df29700 1 -- 172.21.15.107:0/2693414952 >> 172.21.15.107:0/2693414952 conn(0x7fd7780fd920 msgr2=0x7fd7781068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:27:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:58.528+0000 7fd77df29700 1 -- 172.21.15.107:0/2693414952 shutdown_connections 2024-05-09T08:27:58.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:27:58.528+0000 7fd77df29700 1 -- 172.21.15.107:0/2693414952 wait complete. 2024-05-09T08:27:58.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:58.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:58 smithi107 ceph-mon[30122]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:58.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:58.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:58.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:58.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:58.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:58 smithi107 ceph-mon[30122]: from='osd.6 [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-09T08:27:58.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:58.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:58 smithi157 ceph-mon[33614]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:58.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:58.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:58.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:58.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:58.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:58 smithi157 ceph-mon[33614]: from='osd.6 [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-09T08:27:58.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:58.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:58 smithi110 ceph-mon[33622]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:27:58.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:58.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:27:58.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:27:58.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:58.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:58 smithi110 ceph-mon[33622]: from='osd.6 [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-09T08:27:59.247 DEBUG:teuthology.orchestra.run.smithi107:osd.6> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.6.service 2024-05-09T08:27:59.250 INFO:tasks.cephadm:Deploying osd.7 on smithi110 with /dev/vg_nvme/lv_2... 2024-05-09T08:27:59.250 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-09T08:27:59.480 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:59 smithi107 ceph-mon[30122]: from='osd.6 [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-09T08:27:59.480 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:59 smithi107 ceph-mon[30122]: osdmap e44: 7 total, 6 up, 7 in 2024-05-09T08:27:59.481 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:59 smithi107 ceph-mon[30122]: from='osd.6 [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-05-09T08:27:59.481 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:59 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:27:59.481 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:59 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:59.481 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:59 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:59.481 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:27:59 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:59.481 INFO:journalctl@ceph.osd.6.smithi107.stdout:May 09 08:27:59 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-6[55944]: 2024-05-09T08:27:59.434+0000 7f75d4302700 -1 osd.6 0 waiting for initial osdmap 2024-05-09T08:27:59.481 INFO:journalctl@ceph.osd.6.smithi107.stdout:May 09 08:27:59 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-6[55944]: 2024-05-09T08:27:59.442+0000 7f75d00f5700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:27:59.498 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:27:59.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:59 smithi110 ceph-mon[33622]: from='osd.6 [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-09T08:27:59.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:59 smithi110 ceph-mon[33622]: osdmap e44: 7 total, 6 up, 7 in 2024-05-09T08:27:59.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:59 smithi110 ceph-mon[33622]: from='osd.6 [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-05-09T08:27:59.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:59 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:27:59.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:59 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:59.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:59 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:59.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:27:59 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:27:59.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:59 smithi157 ceph-mon[33614]: from='osd.6 [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-09T08:27:59.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:59 smithi157 ceph-mon[33614]: osdmap e44: 7 total, 6 up, 7 in 2024-05-09T08:27:59.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:59 smithi157 ceph-mon[33614]: from='osd.6 [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-05-09T08:27:59.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:59 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:27:59.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:59 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:59.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:59 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:27:59.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:27:59 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:28:00.707 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:00 smithi107 ceph-mon[30122]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:28:00.707 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:00 smithi107 ceph-mon[30122]: from='osd.6 [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-05-09T08:28:00.708 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:00 smithi107 ceph-mon[30122]: osdmap e45: 7 total, 6 up, 7 in 2024-05-09T08:28:00.708 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:00 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:28:00.708 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:00 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:28:00.776 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:00 smithi110 ceph-mon[33622]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:28:00.776 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:00 smithi110 ceph-mon[33622]: from='osd.6 [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-05-09T08:28:00.777 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:00 smithi110 ceph-mon[33622]: osdmap e45: 7 total, 6 up, 7 in 2024-05-09T08:28:00.777 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:00 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:28:00.777 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:00 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:28:00.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:00 smithi157 ceph-mon[33614]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 536 GiB / 536 GiB avail 2024-05-09T08:28:00.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:00 smithi157 ceph-mon[33614]: from='osd.6 [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-05-09T08:28:00.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:00 smithi157 ceph-mon[33614]: osdmap e45: 7 total, 6 up, 7 in 2024-05-09T08:28:00.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:00 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:28:00.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:00 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:28:01.487 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-09T08:28:01.487 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-09T08:28:01.487 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-05-09T08:28:01.488 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-05-09T08:28:01.488 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00894344 s, 1.2 GB/s 2024-05-09T08:28:01.488 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-05-09T08:28:01.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:01 smithi110 ceph-mon[33622]: purged_snaps scrub starts 2024-05-09T08:28:01.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:01 smithi110 ceph-mon[33622]: purged_snaps scrub ok 2024-05-09T08:28:01.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:01 smithi110 ceph-mon[33622]: osd.6 [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372] boot 2024-05-09T08:28:01.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:01 smithi110 ceph-mon[33622]: osdmap e46: 7 total, 7 up, 7 in 2024-05-09T08:28:01.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:01 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:28:01.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:01 smithi107 ceph-mon[30122]: purged_snaps scrub starts 2024-05-09T08:28:01.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:01 smithi107 ceph-mon[30122]: purged_snaps scrub ok 2024-05-09T08:28:01.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:01 smithi107 ceph-mon[30122]: osd.6 [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372] boot 2024-05-09T08:28:01.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:01 smithi107 ceph-mon[30122]: osdmap e46: 7 total, 7 up, 7 in 2024-05-09T08:28:01.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:01 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:28:01.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:01 smithi157 ceph-mon[33614]: purged_snaps scrub starts 2024-05-09T08:28:01.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:01 smithi157 ceph-mon[33614]: purged_snaps scrub ok 2024-05-09T08:28:01.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:01 smithi157 ceph-mon[33614]: osd.6 [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372] boot 2024-05-09T08:28:01.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:01 smithi157 ceph-mon[33614]: osdmap e46: 7 total, 7 up, 7 in 2024-05-09T08:28:01.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:01 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:28:02.156 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi110:vg_nvme/lv_2 2024-05-09T08:28:02.406 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:28:02.766 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:02 smithi110 ceph-mon[33622]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:02.767 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:02 smithi110 ceph-mon[33622]: osdmap e47: 7 total, 7 up, 7 in 2024-05-09T08:28:02.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:02 smithi107 ceph-mon[30122]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:02.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:02 smithi107 ceph-mon[30122]: osdmap e47: 7 total, 7 up, 7 in 2024-05-09T08:28:02.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:02 smithi157 ceph-mon[33614]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:02.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:02 smithi157 ceph-mon[33614]: osdmap e47: 7 total, 7 up, 7 in 2024-05-09T08:28:03.521 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:03 smithi110 ceph-mon[33622]: Detected new or changed devices on smithi107 2024-05-09T08:28:03.521 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:03 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:03.521 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:03 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:03.521 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:03 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:03.521 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:03 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:03.521 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:03 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:03.521 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:03 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi107 to 5696M 2024-05-09T08:28:03.521 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:03 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:03.521 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:03 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:03.521 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:03 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:03.521 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:03 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:03.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:03 smithi107 ceph-mon[30122]: Detected new or changed devices on smithi107 2024-05-09T08:28:03.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:03 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:03.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:03 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:03.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:03 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:03.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:03 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:03.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:03 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:03.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:03 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi107 to 5696M 2024-05-09T08:28:03.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:03 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:03.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:03 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:03.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:03 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:03.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:03 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:03.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:03 smithi157 ceph-mon[33614]: Detected new or changed devices on smithi107 2024-05-09T08:28:03.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:03 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:03.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:03 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:03.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:03 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:03.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:03 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:03.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:03 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:03.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:03 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi107 to 5696M 2024-05-09T08:28:03.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:03 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:03.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:03 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:03.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:03 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:03.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:03 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.072+0000 7f017d8da700 1 -- 172.21.15.110:0/3204475983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0178100690 msgr2=0x7f0178100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.072+0000 7f017d8da700 1 --2- 172.21.15.110:0/3204475983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0178100690 0x7f0178100ab0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0168009990 tx=0x7f0168009ca0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.072+0000 7f017d8da700 1 -- 172.21.15.110:0/3204475983 shutdown_connections 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.072+0000 7f017d8da700 1 --2- 172.21.15.110:0/3204475983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0178101dc0 0x7f0178106660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.072+0000 7f017d8da700 1 --2- 172.21.15.110:0/3204475983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0178100690 0x7f0178100ab0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.072+0000 7f017d8da700 1 --2- 172.21.15.110:0/3204475983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0178101400 0x7f0178101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.072+0000 7f017d8da700 1 -- 172.21.15.110:0/3204475983 >> 172.21.15.110:0/3204475983 conn(0x7f01780fbff0 msgr2=0x7f01780fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.072+0000 7f017d8da700 1 -- 172.21.15.110:0/3204475983 shutdown_connections 2024-05-09T08:28:04.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.072+0000 7f017d8da700 1 -- 172.21.15.110:0/3204475983 wait complete. 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.072+0000 7f017d8da700 1 Processor -- start 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f017d8da700 1 -- start start 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f017d8da700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0178100690 0x7f017819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f017d8da700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0178101400 0x7f017819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f017d8da700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0178101dc0 0x7f01781a6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f017d8da700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0178114480 con 0x7f0178101dc0 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f017d8da700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0178114300 con 0x7f0178100690 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f017d8da700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0178114600 con 0x7f0178101400 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f0176ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0178100690 0x7f017819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f0176ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0178100690 0x7f017819ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48426/0 (socket says 172.21.15.110:48426) 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f0176ffd700 1 -- 172.21.15.110:0/621271916 learned_addr learned my addr 172.21.15.110:0/621271916 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f016ffff700 1 --2- 172.21.15.110:0/621271916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0178101400 0x7f017819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f01777fe700 1 --2- 172.21.15.110:0/621271916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0178101dc0 0x7f01781a6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f0176ffd700 1 -- 172.21.15.110:0/621271916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0178101400 msgr2=0x7f017819f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:04.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f0176ffd700 1 --2- 172.21.15.110:0/621271916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0178101400 0x7f017819f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f0176ffd700 1 -- 172.21.15.110:0/621271916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0178101dc0 msgr2=0x7f01781a6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:04.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f0176ffd700 1 --2- 172.21.15.110:0/621271916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0178101dc0 0x7f01781a6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:04.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f0176ffd700 1 -- 172.21.15.110:0/621271916 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01781a66e0 con 0x7f0178100690 2024-05-09T08:28:04.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f0176ffd700 1 --2- 172.21.15.110:0/621271916 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0178100690 0x7f017819ead0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f016800b5c0 tx=0x7f01680047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:04.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.073+0000 7f017c8d8700 1 -- 172.21.15.110:0/621271916 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0168027070 con 0x7f0178100690 2024-05-09T08:28:04.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.074+0000 7f017c8d8700 1 -- 172.21.15.110:0/621271916 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01680040d0 con 0x7f0178100690 2024-05-09T08:28:04.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.074+0000 7f017d8da700 1 -- 172.21.15.110:0/621271916 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01781a6970 con 0x7f0178100690 2024-05-09T08:28:04.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.074+0000 7f017c8d8700 1 -- 172.21.15.110:0/621271916 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0168022dc0 con 0x7f0178100690 2024-05-09T08:28:04.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.074+0000 7f017d8da700 1 -- 172.21.15.110:0/621271916 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01781a6dc0 con 0x7f0178100690 2024-05-09T08:28:04.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.074+0000 7f017d8da700 1 -- 172.21.15.110:0/621271916 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0178069010 con 0x7f0178100690 2024-05-09T08:28:04.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.076+0000 7f017c8d8700 1 -- 172.21.15.110:0/621271916 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f016802a070 con 0x7f0178100690 2024-05-09T08:28:04.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.076+0000 7f017c8d8700 1 --2- 172.21.15.110:0/621271916 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f01640a2c20 0x7f01640a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:04.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.077+0000 7f017c8d8700 1 -- 172.21.15.110:0/621271916 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f01680cf850 con 0x7f0178100690 2024-05-09T08:28:04.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.077+0000 7f016ffff700 1 --2- 172.21.15.110:0/621271916 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f01640a2c20 0x7f01640a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:04.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.077+0000 7f016ffff700 1 --2- 172.21.15.110:0/621271916 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f01640a2c20 0x7f01640a50e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0160005950 tx=0x7f016000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:04.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.080+0000 7f017c8d8700 1 -- 172.21.15.110:0/621271916 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0168031070 con 0x7f0178100690 2024-05-09T08:28:04.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:04.282+0000 7f017d8da700 1 -- 172.21.15.110:0/621271916 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f0178062760 con 0x7f01640a2c20 2024-05-09T08:28:04.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:04 smithi107 ceph-mon[30122]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:04.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:04 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:28:04.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:04 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:28:04.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:04 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:04.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:04 smithi157 ceph-mon[33614]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:04.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:04 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:28:04.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:04 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:28:04.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:04 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:04.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:04 smithi110 ceph-mon[33622]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:04.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:04 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:28:04.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:04 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:28:04.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:04 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:05.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:05 smithi107 ceph-mon[30122]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:28:05.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:05 smithi157 ceph-mon[33614]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:28:05.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:05 smithi110 ceph-mon[33622]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:28:06.744 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:06 smithi157 ceph-mon[33614]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:06.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:06 smithi107 ceph-mon[30122]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:06.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:06 smithi110 ceph-mon[33622]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:07.764 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:07 smithi157 ceph-mon[33614]: from='client.? 172.21.15.110:0/1619684963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7258141-aef2-4d46-9825-8c51d817f7a3"}]: dispatch 2024-05-09T08:28:07.765 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:07 smithi157 ceph-mon[33614]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7258141-aef2-4d46-9825-8c51d817f7a3"}]: dispatch 2024-05-09T08:28:07.765 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:07 smithi157 ceph-mon[33614]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7258141-aef2-4d46-9825-8c51d817f7a3"}]': finished 2024-05-09T08:28:07.765 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:07 smithi157 ceph-mon[33614]: osdmap e48: 8 total, 7 up, 8 in 2024-05-09T08:28:07.765 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:07 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:28:07.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:07 smithi107 ceph-mon[30122]: from='client.? 172.21.15.110:0/1619684963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7258141-aef2-4d46-9825-8c51d817f7a3"}]: dispatch 2024-05-09T08:28:07.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:07 smithi107 ceph-mon[30122]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7258141-aef2-4d46-9825-8c51d817f7a3"}]: dispatch 2024-05-09T08:28:07.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:07 smithi107 ceph-mon[30122]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7258141-aef2-4d46-9825-8c51d817f7a3"}]': finished 2024-05-09T08:28:07.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:07 smithi107 ceph-mon[30122]: osdmap e48: 8 total, 7 up, 8 in 2024-05-09T08:28:07.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:07 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:28:07.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:07 smithi110 ceph-mon[33622]: from='client.? 172.21.15.110:0/1619684963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7258141-aef2-4d46-9825-8c51d817f7a3"}]: dispatch 2024-05-09T08:28:07.954 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:07 smithi110 ceph-mon[33622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7258141-aef2-4d46-9825-8c51d817f7a3"}]: dispatch 2024-05-09T08:28:07.954 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:07 smithi110 ceph-mon[33622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7258141-aef2-4d46-9825-8c51d817f7a3"}]': finished 2024-05-09T08:28:07.954 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:07 smithi110 ceph-mon[33622]: osdmap e48: 8 total, 7 up, 8 in 2024-05-09T08:28:07.955 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:07 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:28:08.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:08 smithi107 ceph-mon[30122]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:08.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:08 smithi107 ceph-mon[30122]: from='client.? 172.21.15.110:0/3708314887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:28:08.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:08 smithi157 ceph-mon[33614]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:08.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:08 smithi157 ceph-mon[33614]: from='client.? 172.21.15.110:0/3708314887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:28:08.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:08 smithi110 ceph-mon[33622]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:08.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:08 smithi110 ceph-mon[33622]: from='client.? 172.21.15.110:0/3708314887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:28:10.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:10 smithi107 ceph-mon[30122]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:10.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:10 smithi157 ceph-mon[33614]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:10.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:10 smithi110 ceph-mon[33622]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:12.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:12 smithi107 ceph-mon[30122]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:12.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:12 smithi157 ceph-mon[33614]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:12.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:12 smithi110 ceph-mon[33622]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:14.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:14 smithi107 ceph-mon[30122]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:14.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:14 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-09T08:28:14.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:14 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:14.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:14 smithi157 ceph-mon[33614]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:14.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:14 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-09T08:28:14.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:14 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:14.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:14 smithi110 ceph-mon[33622]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:14.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:14 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-09T08:28:14.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:14 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:15.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:15 smithi107 ceph-mon[30122]: Deploying daemon osd.7 on smithi110 2024-05-09T08:28:15.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:15 smithi157 ceph-mon[33614]: Deploying daemon osd.7 on smithi110 2024-05-09T08:28:15.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:15 smithi110 ceph-mon[33622]: Deploying daemon osd.7 on smithi110 2024-05-09T08:28:16.611 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:16 smithi110 ceph-mon[33622]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:16.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:16 smithi107 ceph-mon[30122]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:16.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:16 smithi157 ceph-mon[33614]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:18.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:18 smithi107 ceph-mon[30122]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:18.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:18 smithi157 ceph-mon[33614]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:18.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:18 smithi110 ceph-mon[33622]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:20.630 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:20 smithi110 ceph-mon[33622]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:20.630 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:20.630 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:20.630 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:28:20.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:20 smithi107 ceph-mon[30122]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:20.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:20.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:20.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:28:20.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:20 smithi157 ceph-mon[33614]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:20.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:20.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:20.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:28:22.579 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:22 smithi110 ceph-mon[33622]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:22.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:22 smithi107 ceph-mon[30122]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:22.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:22 smithi157 ceph-mon[33614]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:23.696 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 7 on host 'smithi110' 2024-05-09T08:28:23.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:23.696+0000 7f017c8d8700 1 -- 172.21.15.110:0/621271916 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0178062760 con 0x7f01640a2c20 2024-05-09T08:28:23.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:23.698+0000 7f017d8da700 1 -- 172.21.15.110:0/621271916 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f01640a2c20 msgr2=0x7f01640a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:23.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:23.698+0000 7f017d8da700 1 --2- 172.21.15.110:0/621271916 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f01640a2c20 0x7f01640a50e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0160005950 tx=0x7f016000b410 comp rx=0 tx=0).stop 2024-05-09T08:28:23.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:23.698+0000 7f017d8da700 1 -- 172.21.15.110:0/621271916 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0178100690 msgr2=0x7f017819ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:23.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:23.698+0000 7f017d8da700 1 --2- 172.21.15.110:0/621271916 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0178100690 0x7f017819ead0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f016800b5c0 tx=0x7f01680047b0 comp rx=0 tx=0).stop 2024-05-09T08:28:23.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:23.698+0000 7f017d8da700 1 -- 172.21.15.110:0/621271916 shutdown_connections 2024-05-09T08:28:23.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:23.698+0000 7f017d8da700 1 --2- 172.21.15.110:0/621271916 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f01640a2c20 0x7f01640a50e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:23.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:23.698+0000 7f017d8da700 1 --2- 172.21.15.110:0/621271916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0178101dc0 0x7f01781a6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:23.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:23.698+0000 7f017d8da700 1 --2- 172.21.15.110:0/621271916 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0178100690 0x7f017819ead0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:23.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:23.698+0000 7f017d8da700 1 --2- 172.21.15.110:0/621271916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0178101400 0x7f017819f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:23.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:23.698+0000 7f017d8da700 1 -- 172.21.15.110:0/621271916 >> 172.21.15.110:0/621271916 conn(0x7f01780fbff0 msgr2=0x7f0178102640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:23.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:23.698+0000 7f017d8da700 1 -- 172.21.15.110:0/621271916 shutdown_connections 2024-05-09T08:28:23.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:28:23.698+0000 7f017d8da700 1 -- 172.21.15.110:0/621271916 wait complete. 2024-05-09T08:28:24.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:23 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:24.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:23 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:24.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:23 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:24.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:23 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:24.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:23 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:24.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:23 smithi110 ceph-mon[33622]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:24.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:23 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:24.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:23 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:24.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:23 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:28:24.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:23 smithi110 ceph-mon[33622]: from='osd.7 [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:28:24.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:23 smithi110 ceph-mon[33622]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:28:24.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:23 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:24.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:23 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:24.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:23 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:24.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:23 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:24.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:23 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:24.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:23 smithi107 ceph-mon[30122]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:24.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:23 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:24.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:23 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:24.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:23 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:28:24.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:23 smithi107 ceph-mon[30122]: from='osd.7 [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:28:24.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:23 smithi107 ceph-mon[30122]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:28:24.315 DEBUG:teuthology.orchestra.run.smithi110:osd.7> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.7.service 2024-05-09T08:28:24.317 INFO:tasks.cephadm:Deploying osd.8 on smithi157 with /dev/vg_nvme/lv_2... 2024-05-09T08:28:24.317 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-09T08:28:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:23 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:23 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:23 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:23 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:23 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:23 smithi157 ceph-mon[33614]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:23 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:23 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:23 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:28:24.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:23 smithi157 ceph-mon[33614]: from='osd.7 [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:28:24.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:23 smithi157 ceph-mon[33614]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T08:28:24.569 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:28:25.776 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:25 smithi110 ceph-mon[33622]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-09T08:28:25.776 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:25 smithi110 ceph-mon[33622]: from='osd.7 [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:28:25.777 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:25 smithi110 ceph-mon[33622]: osdmap e49: 8 total, 7 up, 8 in 2024-05-09T08:28:25.777 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:25 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:28:25.777 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:25 smithi110 ceph-mon[33622]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:28:25.777 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:25 smithi110 ceph-mon[33622]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:25.953 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:25 smithi157 ceph-mon[33614]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-09T08:28:25.953 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:25 smithi157 ceph-mon[33614]: from='osd.7 [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:28:25.953 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:25 smithi157 ceph-mon[33614]: osdmap e49: 8 total, 7 up, 8 in 2024-05-09T08:28:25.953 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:25 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:28:25.953 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:25 smithi157 ceph-mon[33614]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:28:25.953 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:25 smithi157 ceph-mon[33614]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:26.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:25 smithi107 ceph-mon[30122]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-09T08:28:26.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:25 smithi107 ceph-mon[30122]: from='osd.7 [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:28:26.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:25 smithi107 ceph-mon[30122]: osdmap e49: 8 total, 7 up, 8 in 2024-05-09T08:28:26.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:25 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:28:26.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:25 smithi107 ceph-mon[30122]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:28:26.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:25 smithi107 ceph-mon[30122]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 626 GiB / 626 GiB avail 2024-05-09T08:28:26.604 INFO:journalctl@ceph.osd.7.smithi110.stdout:May 09 08:28:26 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-7[51809]: 2024-05-09T08:28:26.210+0000 7fa3282f2700 -1 osd.7 0 waiting for initial osdmap 2024-05-09T08:28:26.604 INFO:journalctl@ceph.osd.7.smithi110.stdout:May 09 08:28:26 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-7[51809]: 2024-05-09T08:28:26.220+0000 7fa3200dd700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:28:26.694 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-09T08:28:26.694 INFO:teuthology.orchestra.run.smithi157.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-09T08:28:26.694 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10+0 records in 2024-05-09T08:28:26.694 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records out 2024-05-09T08:28:26.695 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00844456 s, 1.2 GB/s 2024-05-09T08:28:26.695 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping successful for: 2024-05-09T08:28:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:26 smithi157 ceph-mon[33614]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-05-09T08:28:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:26 smithi157 ceph-mon[33614]: osdmap e50: 8 total, 7 up, 8 in 2024-05-09T08:28:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:28:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:28:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:26 smithi157 ceph-mon[33614]: from='osd.7 ' entity='osd.7' 2024-05-09T08:28:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:26 smithi110 ceph-mon[33622]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-05-09T08:28:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:26 smithi110 ceph-mon[33622]: osdmap e50: 8 total, 7 up, 8 in 2024-05-09T08:28:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:28:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:28:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:26 smithi110 ceph-mon[33622]: from='osd.7 ' entity='osd.7' 2024-05-09T08:28:27.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:26 smithi107 ceph-mon[30122]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-05-09T08:28:27.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:26 smithi107 ceph-mon[30122]: osdmap e50: 8 total, 7 up, 8 in 2024-05-09T08:28:27.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:28:27.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:28:27.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:26 smithi107 ceph-mon[30122]: from='osd.7 ' entity='osd.7' 2024-05-09T08:28:27.398 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi157:vg_nvme/lv_2 2024-05-09T08:28:27.678 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:28:27.951 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: purged_snaps scrub starts 2024-05-09T08:28:27.951 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: purged_snaps scrub ok 2024-05-09T08:28:27.951 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: osd.7 [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355] boot 2024-05-09T08:28:27.951 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: osdmap e51: 8 total, 8 up, 8 in 2024-05-09T08:28:27.951 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:28:27.951 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: osdmap e52: 8 total, 8 up, 8 in 2024-05-09T08:28:27.951 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: Detected new or changed devices on smithi110 2024-05-09T08:28:27.951 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:27.951 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:27.951 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:27.952 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:27.952 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:27.952 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi110 to 5696M 2024-05-09T08:28:27.952 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:27.952 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:27.952 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:27.952 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:27.952 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:27 smithi157 ceph-mon[33614]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:27.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: purged_snaps scrub starts 2024-05-09T08:28:27.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: purged_snaps scrub ok 2024-05-09T08:28:27.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: osd.7 [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355] boot 2024-05-09T08:28:27.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: osdmap e51: 8 total, 8 up, 8 in 2024-05-09T08:28:27.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:28:27.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: osdmap e52: 8 total, 8 up, 8 in 2024-05-09T08:28:27.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: Detected new or changed devices on smithi110 2024-05-09T08:28:27.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:27.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:27.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:27.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:27.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:27.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi110 to 5696M 2024-05-09T08:28:27.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:27.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:27.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:27.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:27.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:27 smithi107 ceph-mon[30122]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:28.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: purged_snaps scrub starts 2024-05-09T08:28:28.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: purged_snaps scrub ok 2024-05-09T08:28:28.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: osd.7 [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355] boot 2024-05-09T08:28:28.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: osdmap e51: 8 total, 8 up, 8 in 2024-05-09T08:28:28.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:28:28.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: osdmap e52: 8 total, 8 up, 8 in 2024-05-09T08:28:28.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: Detected new or changed devices on smithi110 2024-05-09T08:28:28.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:28.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:28.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:28.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:28.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:28.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi110 to 5696M 2024-05-09T08:28:28.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:28.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:28.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:28.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:28.123 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:27 smithi110 ceph-mon[33622]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:29.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:28 smithi157 ceph-mon[33614]: osdmap e53: 8 total, 8 up, 8 in 2024-05-09T08:28:29.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:28 smithi110 ceph-mon[33622]: osdmap e53: 8 total, 8 up, 8 in 2024-05-09T08:28:29.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:28 smithi107 ceph-mon[30122]: osdmap e53: 8 total, 8 up, 8 in 2024-05-09T08:28:29.382 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.378+0000 7fb0978c0700 1 -- 172.21.15.157:0/3634383333 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0901025d0 msgr2=0x7fb090108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:29.382 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.379+0000 7fb0978c0700 1 --2- 172.21.15.157:0/3634383333 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0901025d0 0x7fb090108e80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb08800b0a0 tx=0x7fb08800b3b0 comp rx=0 tx=0).stop 2024-05-09T08:28:29.382 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.379+0000 7fb0978c0700 1 -- 172.21.15.157:0/3634383333 shutdown_connections 2024-05-09T08:28:29.382 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.379+0000 7fb0978c0700 1 --2- 172.21.15.157:0/3634383333 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0901025d0 0x7fb090108e80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:29.383 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.379+0000 7fb0978c0700 1 --2- 172.21.15.157:0/3634383333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb090100a30 0x7fb090100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:29.383 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.379+0000 7fb0978c0700 1 --2- 172.21.15.157:0/3634383333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb090101c30 0x7fb090102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:29.383 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.379+0000 7fb0978c0700 1 -- 172.21.15.157:0/3634383333 >> 172.21.15.157:0/3634383333 conn(0x7fb0900fbfd0 msgr2=0x7fb0900fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:29.383 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.379+0000 7fb0978c0700 1 -- 172.21.15.157:0/3634383333 shutdown_connections 2024-05-09T08:28:29.383 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.379+0000 7fb0978c0700 1 -- 172.21.15.157:0/3634383333 wait complete. 2024-05-09T08:28:29.383 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.379+0000 7fb0978c0700 1 Processor -- start 2024-05-09T08:28:29.383 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.379+0000 7fb0978c0700 1 -- start start 2024-05-09T08:28:29.383 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.379+0000 7fb0978c0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb090100a30 0x7fb090197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:29.383 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.379+0000 7fb0978c0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb090101c30 0x7fb090197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:29.383 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb0978c0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0901025d0 0x7fb09019e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:29.383 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb0978c0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb09010c520 con 0x7fb0901025d0 2024-05-09T08:28:29.383 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb0978c0700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb09010c3a0 con 0x7fb090101c30 2024-05-09T08:28:29.384 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb0978c0700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb09010c6a0 con 0x7fb090100a30 2024-05-09T08:28:29.384 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb09565c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb090100a30 0x7fb090197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:29.384 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb09565c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb090100a30 0x7fb090197330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:45970/0 (socket says 172.21.15.157:45970) 2024-05-09T08:28:29.384 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb09565c700 1 -- 172.21.15.157:0/3770273733 learned_addr learned my addr 172.21.15.157:0/3770273733 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-05-09T08:28:29.384 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb094e5b700 1 --2- 172.21.15.157:0/3770273733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb090101c30 0x7fb090197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:29.384 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb09565c700 1 -- 172.21.15.157:0/3770273733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb090101c30 msgr2=0x7fb090197870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:29.384 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb09565c700 1 --2- 172.21.15.157:0/3770273733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb090101c30 0x7fb090197870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:29.384 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb09565c700 1 -- 172.21.15.157:0/3770273733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0901025d0 msgr2=0x7fb09019e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:29.384 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb09565c700 1 --2- 172.21.15.157:0/3770273733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0901025d0 0x7fb09019e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:29.384 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb09565c700 1 -- 172.21.15.157:0/3770273733 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09019ef90 con 0x7fb090100a30 2024-05-09T08:28:29.384 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb094e5b700 1 --2- 172.21.15.157:0/3770273733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb090101c30 0x7fb090197870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:28:29.384 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb09565c700 1 --2- 172.21.15.157:0/3770273733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb090100a30 0x7fb090197330 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb08c00c9c0 tx=0x7fb08c00cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:29.384 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb0867fc700 1 -- 172.21.15.157:0/3770273733 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c007ac0 con 0x7fb090100a30 2024-05-09T08:28:29.384 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb0867fc700 1 -- 172.21.15.157:0/3770273733 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb08c007c60 con 0x7fb090100a30 2024-05-09T08:28:29.384 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb0978c0700 1 -- 172.21.15.157:0/3770273733 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb09019f280 con 0x7fb090100a30 2024-05-09T08:28:29.385 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb0867fc700 1 -- 172.21.15.157:0/3770273733 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c018660 con 0x7fb090100a30 2024-05-09T08:28:29.385 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.380+0000 7fb0978c0700 1 -- 172.21.15.157:0/3770273733 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb09019f7c0 con 0x7fb090100a30 2024-05-09T08:28:29.385 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.381+0000 7fb0978c0700 1 -- 172.21.15.157:0/3770273733 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb090069010 con 0x7fb090100a30 2024-05-09T08:28:29.385 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.382+0000 7fb0867fc700 1 -- 172.21.15.157:0/3770273733 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fb08c00f450 con 0x7fb090100a30 2024-05-09T08:28:29.386 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.383+0000 7fb0867fc700 1 --2- 172.21.15.157:0/3770273733 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb07c0a2c20 0x7fb07c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:29.386 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.383+0000 7fb0867fc700 1 -- 172.21.15.157:0/3770273733 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5397+0+0 (secure 0 0 0) 0x7fb08c0ce550 con 0x7fb090100a30 2024-05-09T08:28:29.386 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.384+0000 7fb094e5b700 1 --2- 172.21.15.157:0/3770273733 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb07c0a2c20 0x7fb07c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:29.386 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.384+0000 7fb094e5b700 1 --2- 172.21.15.157:0/3770273733 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb07c0a2c20 0x7fb07c0a50e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb080005950 tx=0x7fb080005f30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:29.388 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.388+0000 7fb0867fc700 1 -- 172.21.15.157:0/3770273733 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb08c014030 con 0x7fb090100a30 2024-05-09T08:28:29.596 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:29.595+0000 7fb0978c0700 1 -- 172.21.15.157:0/3770273733 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb090062760 con 0x7fb07c0a2c20 2024-05-09T08:28:30.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:29 smithi157 ceph-mon[33614]: osdmap e54: 8 total, 8 up, 8 in 2024-05-09T08:28:30.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:29 smithi157 ceph-mon[33614]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:30.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:29 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:28:30.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:29 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:28:30.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:29 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:30.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:29 smithi157 ceph-mon[33614]: osdmap e55: 8 total, 8 up, 8 in 2024-05-09T08:28:30.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:29 smithi110 ceph-mon[33622]: osdmap e54: 8 total, 8 up, 8 in 2024-05-09T08:28:30.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:29 smithi110 ceph-mon[33622]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:30.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:29 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:28:30.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:29 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:28:30.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:29 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:30.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:29 smithi110 ceph-mon[33622]: osdmap e55: 8 total, 8 up, 8 in 2024-05-09T08:28:30.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:29 smithi107 ceph-mon[30122]: osdmap e54: 8 total, 8 up, 8 in 2024-05-09T08:28:30.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:29 smithi107 ceph-mon[30122]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:30.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:29 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:28:30.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:29 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:28:30.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:29 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:30.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:29 smithi107 ceph-mon[30122]: osdmap e55: 8 total, 8 up, 8 in 2024-05-09T08:28:31.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:30 smithi157 ceph-mon[33614]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:28:31.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:30 smithi110 ceph-mon[33622]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:28:31.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:30 smithi107 ceph-mon[30122]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:28:32.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:31 smithi157 ceph-mon[33614]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:32.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:31 smithi110 ceph-mon[33622]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:32.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:31 smithi107 ceph-mon[30122]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:33.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:32 smithi157 ceph-mon[33614]: from='client.? 172.21.15.157:0/4079538353' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bcc1105e-25be-4d89-bb7b-920287c3f6cd"}]: dispatch 2024-05-09T08:28:33.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:32 smithi157 ceph-mon[33614]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bcc1105e-25be-4d89-bb7b-920287c3f6cd"}]: dispatch 2024-05-09T08:28:33.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:32 smithi157 ceph-mon[33614]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bcc1105e-25be-4d89-bb7b-920287c3f6cd"}]': finished 2024-05-09T08:28:33.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:32 smithi157 ceph-mon[33614]: osdmap e56: 9 total, 8 up, 9 in 2024-05-09T08:28:33.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:28:33.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:32 smithi110 ceph-mon[33622]: from='client.? 172.21.15.157:0/4079538353' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bcc1105e-25be-4d89-bb7b-920287c3f6cd"}]: dispatch 2024-05-09T08:28:33.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:32 smithi110 ceph-mon[33622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bcc1105e-25be-4d89-bb7b-920287c3f6cd"}]: dispatch 2024-05-09T08:28:33.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:32 smithi110 ceph-mon[33622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bcc1105e-25be-4d89-bb7b-920287c3f6cd"}]': finished 2024-05-09T08:28:33.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:32 smithi110 ceph-mon[33622]: osdmap e56: 9 total, 8 up, 9 in 2024-05-09T08:28:33.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:28:33.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:32 smithi107 ceph-mon[30122]: from='client.? 172.21.15.157:0/4079538353' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bcc1105e-25be-4d89-bb7b-920287c3f6cd"}]: dispatch 2024-05-09T08:28:33.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:32 smithi107 ceph-mon[30122]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bcc1105e-25be-4d89-bb7b-920287c3f6cd"}]: dispatch 2024-05-09T08:28:33.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:32 smithi107 ceph-mon[30122]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bcc1105e-25be-4d89-bb7b-920287c3f6cd"}]': finished 2024-05-09T08:28:33.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:32 smithi107 ceph-mon[30122]: osdmap e56: 9 total, 8 up, 9 in 2024-05-09T08:28:33.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:28:34.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:33 smithi157 ceph-mon[33614]: from='client.? 172.21.15.157:0/2586153333' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:28:34.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:33 smithi157 ceph-mon[33614]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:34.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:33 smithi110 ceph-mon[33622]: from='client.? 172.21.15.157:0/2586153333' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:28:34.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:33 smithi110 ceph-mon[33622]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:34.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:33 smithi107 ceph-mon[30122]: from='client.? 172.21.15.157:0/2586153333' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:28:34.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:33 smithi107 ceph-mon[30122]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:36.740 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:36 smithi157 ceph-mon[33614]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:36.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:36 smithi107 ceph-mon[30122]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:36.824 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:36 smithi110 ceph-mon[33622]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:38.676 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:38 smithi157 ceph-mon[33614]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:38.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:38 smithi107 ceph-mon[30122]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:38.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:38 smithi110 ceph-mon[33622]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:40.681 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:40 smithi157 ceph-mon[33614]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:40.682 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:40 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-09T08:28:40.682 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:40 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:40.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:40 smithi107 ceph-mon[30122]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:40.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:40 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-09T08:28:40.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:40 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:40.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:40 smithi110 ceph-mon[33622]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:40.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:40 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-09T08:28:40.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:40 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:41.732 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:41 smithi157 ceph-mon[33614]: Deploying daemon osd.8 on smithi157 2024-05-09T08:28:41.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:41 smithi107 ceph-mon[30122]: Deploying daemon osd.8 on smithi157 2024-05-09T08:28:41.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:41 smithi110 ceph-mon[33622]: Deploying daemon osd.8 on smithi157 2024-05-09T08:28:42.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:42 smithi107 ceph-mon[30122]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:42.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:42 smithi157 ceph-mon[33614]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:42.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:42 smithi110 ceph-mon[33622]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:44.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:44 smithi107 ceph-mon[30122]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:44.822 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:44 smithi157 ceph-mon[33614]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:44.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:44 smithi110 ceph-mon[33622]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:46.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:46.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:46.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:46 smithi107 ceph-mon[30122]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:46.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:28:46.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:46.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:46.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:46 smithi157 ceph-mon[33614]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:46.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:28:46.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:46.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:46.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:46 smithi110 ceph-mon[33622]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:46.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:28:48.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:48 smithi157 ceph-mon[33614]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:48.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:48 smithi107 ceph-mon[30122]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:48.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:48 smithi110 ceph-mon[33622]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:49.107 INFO:teuthology.orchestra.run.smithi157.stdout:Created osd(s) 8 on host 'smithi157' 2024-05-09T08:28:49.108 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:49.108+0000 7fb0867fc700 1 -- 172.21.15.157:0/3770273733 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb090062760 con 0x7fb07c0a2c20 2024-05-09T08:28:49.110 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:49.110+0000 7fb0978c0700 1 -- 172.21.15.157:0/3770273733 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb07c0a2c20 msgr2=0x7fb07c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:49.110 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:49.110+0000 7fb0978c0700 1 --2- 172.21.15.157:0/3770273733 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb07c0a2c20 0x7fb07c0a50e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb080005950 tx=0x7fb080005f30 comp rx=0 tx=0).stop 2024-05-09T08:28:49.110 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:49.110+0000 7fb0978c0700 1 -- 172.21.15.157:0/3770273733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb090100a30 msgr2=0x7fb090197330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:49.110 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:49.110+0000 7fb0978c0700 1 --2- 172.21.15.157:0/3770273733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb090100a30 0x7fb090197330 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb08c00c9c0 tx=0x7fb08c00cd80 comp rx=0 tx=0).stop 2024-05-09T08:28:49.110 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:49.110+0000 7fb0978c0700 1 -- 172.21.15.157:0/3770273733 shutdown_connections 2024-05-09T08:28:49.110 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:49.110+0000 7fb0978c0700 1 --2- 172.21.15.157:0/3770273733 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb07c0a2c20 0x7fb07c0a50e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:49.110 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:49.110+0000 7fb0978c0700 1 --2- 172.21.15.157:0/3770273733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0901025d0 0x7fb09019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:49.111 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:49.111+0000 7fb0978c0700 1 --2- 172.21.15.157:0/3770273733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb090100a30 0x7fb090197330 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:49.111 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:49.111+0000 7fb0978c0700 1 --2- 172.21.15.157:0/3770273733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb090101c30 0x7fb090197870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:49.111 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:49.111+0000 7fb0978c0700 1 -- 172.21.15.157:0/3770273733 >> 172.21.15.157:0/3770273733 conn(0x7fb0900fbfd0 msgr2=0x7fb0901052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:49.111 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:49.111+0000 7fb0978c0700 1 -- 172.21.15.157:0/3770273733 shutdown_connections 2024-05-09T08:28:49.111 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:28:49.111+0000 7fb0978c0700 1 -- 172.21.15.157:0/3770273733 wait complete. 2024-05-09T08:28:49.769 DEBUG:teuthology.orchestra.run.smithi157:osd.8> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.8.service 2024-05-09T08:28:49.771 INFO:tasks.cephadm:Deploying osd.9 on smithi107 with /dev/vg_nvme/lv_1... 2024-05-09T08:28:49.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-09T08:28:50.023 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:28:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:50 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:50 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:50 smithi157 ceph-mon[33614]: from='osd.8 [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:28:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:50 smithi157 ceph-mon[33614]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:28:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:50 smithi157 ceph-mon[33614]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:50 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:50 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:50 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:50 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:50 smithi110 ceph-mon[33622]: from='osd.8 [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:28:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:50 smithi110 ceph-mon[33622]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:28:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:50 smithi110 ceph-mon[33622]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:50 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:50.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:50 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:50.401 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:50 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:50.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:50 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:50.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:50 smithi107 ceph-mon[30122]: from='osd.8 [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:28:50.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:50 smithi107 ceph-mon[30122]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T08:28:50.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:50 smithi107 ceph-mon[30122]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:50.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:50 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:50.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:50 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:51.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:51 smithi157 ceph-mon[33614]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-09T08:28:51.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:51 smithi157 ceph-mon[33614]: osdmap e57: 9 total, 8 up, 9 in 2024-05-09T08:28:51.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:51 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:28:51.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:51 smithi157 ceph-mon[33614]: from='osd.8 [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:28:51.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:51 smithi157 ceph-mon[33614]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:28:51.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:51 smithi110 ceph-mon[33622]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-09T08:28:51.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:51 smithi110 ceph-mon[33622]: osdmap e57: 9 total, 8 up, 9 in 2024-05-09T08:28:51.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:51 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:28:51.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:51 smithi110 ceph-mon[33622]: from='osd.8 [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:28:51.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:51 smithi110 ceph-mon[33622]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:28:51.420 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:51 smithi107 ceph-mon[30122]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-09T08:28:51.420 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:51 smithi107 ceph-mon[30122]: osdmap e57: 9 total, 8 up, 9 in 2024-05-09T08:28:51.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:51 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:28:51.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:51 smithi107 ceph-mon[30122]: from='osd.8 [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:28:51.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:51 smithi107 ceph-mon[30122]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:28:51.823 INFO:journalctl@ceph.osd.8.smithi157.stdout:May 09 08:28:51 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-8[52376]: 2024-05-09T08:28:51.616+0000 7f50c9d09700 -1 osd.8 0 waiting for initial osdmap 2024-05-09T08:28:51.824 INFO:journalctl@ceph.osd.8.smithi157.stdout:May 09 08:28:51 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-8[52376]: 2024-05-09T08:28:51.624+0000 7f50c1af4700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:28:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-09T08:28:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-09T08:28:52.156 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-05-09T08:28:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-05-09T08:28:52.157 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113372 s, 925 MB/s 2024-05-09T08:28:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-05-09T08:28:52.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:52 smithi157 ceph-mon[33614]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-05-09T08:28:52.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:52 smithi157 ceph-mon[33614]: osdmap e58: 9 total, 8 up, 9 in 2024-05-09T08:28:52.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:52 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:28:52.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:52 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:28:52.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:52 smithi157 ceph-mon[33614]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:52.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:52 smithi157 ceph-mon[33614]: from='osd.8 ' entity='osd.8' 2024-05-09T08:28:52.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:52 smithi157 ceph-mon[33614]: osd.8 [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287] boot 2024-05-09T08:28:52.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:52 smithi157 ceph-mon[33614]: osdmap e59: 9 total, 9 up, 9 in 2024-05-09T08:28:52.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:52 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:28:52.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:52 smithi110 ceph-mon[33622]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-05-09T08:28:52.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:52 smithi110 ceph-mon[33622]: osdmap e58: 9 total, 8 up, 9 in 2024-05-09T08:28:52.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:52 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:28:52.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:52 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:28:52.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:52 smithi110 ceph-mon[33622]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:52.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:52 smithi110 ceph-mon[33622]: from='osd.8 ' entity='osd.8' 2024-05-09T08:28:52.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:52 smithi110 ceph-mon[33622]: osd.8 [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287] boot 2024-05-09T08:28:52.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:52 smithi110 ceph-mon[33622]: osdmap e59: 9 total, 9 up, 9 in 2024-05-09T08:28:52.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:52 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:28:52.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:52 smithi107 ceph-mon[30122]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-05-09T08:28:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:52 smithi107 ceph-mon[30122]: osdmap e58: 9 total, 8 up, 9 in 2024-05-09T08:28:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:52 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:28:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:52 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:28:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:52 smithi107 ceph-mon[30122]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-05-09T08:28:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:52 smithi107 ceph-mon[30122]: from='osd.8 ' entity='osd.8' 2024-05-09T08:28:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:52 smithi107 ceph-mon[30122]: osd.8 [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287] boot 2024-05-09T08:28:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:52 smithi107 ceph-mon[30122]: osdmap e59: 9 total, 9 up, 9 in 2024-05-09T08:28:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:52 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:28:52.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi107:vg_nvme/lv_1 2024-05-09T08:28:53.108 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:28:53.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:53 smithi107 ceph-mon[30122]: purged_snaps scrub starts 2024-05-09T08:28:53.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:53 smithi107 ceph-mon[30122]: purged_snaps scrub ok 2024-05-09T08:28:53.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:53 smithi107 ceph-mon[30122]: osdmap e60: 9 total, 9 up, 9 in 2024-05-09T08:28:53.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:53 smithi157 ceph-mon[33614]: purged_snaps scrub starts 2024-05-09T08:28:53.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:53 smithi157 ceph-mon[33614]: purged_snaps scrub ok 2024-05-09T08:28:53.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:53 smithi157 ceph-mon[33614]: osdmap e60: 9 total, 9 up, 9 in 2024-05-09T08:28:53.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:53 smithi110 ceph-mon[33622]: purged_snaps scrub starts 2024-05-09T08:28:53.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:53 smithi110 ceph-mon[33622]: purged_snaps scrub ok 2024-05-09T08:28:53.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:53 smithi110 ceph-mon[33622]: osdmap e60: 9 total, 9 up, 9 in 2024-05-09T08:28:54.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:54 smithi107 ceph-mon[30122]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 684 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:28:54.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:54.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:54.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:54.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:54.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:54.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:54.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:54.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:54.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:54.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:54 smithi107 ceph-mon[30122]: osdmap e61: 9 total, 9 up, 9 in 2024-05-09T08:28:54.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:28:54.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:54.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:54.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:54 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:54.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:54 smithi157 ceph-mon[33614]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 684 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:28:54.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:54.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:54.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:54.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:54.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:54.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:54.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:54.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:54.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:54.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:54 smithi157 ceph-mon[33614]: osdmap e61: 9 total, 9 up, 9 in 2024-05-09T08:28:54.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:28:54.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:54.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:54.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:54 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:54.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:54 smithi110 ceph-mon[33622]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 684 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:28:54.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:54 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:54.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:54 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:54.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:54 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:54.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:54 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:54.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:54 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:28:54.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:54 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:54.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:54 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:54.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:54 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:54.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:54 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:54.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:54 smithi110 ceph-mon[33622]: osdmap e61: 9 total, 9 up, 9 in 2024-05-09T08:28:54.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:54 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:28:54.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:54 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:54.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:54 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:28:54.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:54 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:28:54.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.767+0000 7f5cf6d31700 1 -- 172.21.15.107:0/404876178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cf0101290 msgr2=0x7f5cf01016b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:54.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.767+0000 7f5cf6d31700 1 --2- 172.21.15.107:0/404876178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cf0101290 0x7f5cf01016b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400b0d0 tx=0x7f5ce400b3e0 comp rx=0 tx=0).stop 2024-05-09T08:28:54.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.767+0000 7f5cf6d31700 1 -- 172.21.15.107:0/404876178 shutdown_connections 2024-05-09T08:28:54.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.767+0000 7f5cf6d31700 1 --2- 172.21.15.107:0/404876178 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5cf0102e30 0x7f5cf01096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:54.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.767+0000 7f5cf6d31700 1 --2- 172.21.15.107:0/404876178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cf0101290 0x7f5cf01016b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:54.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.767+0000 7f5cf6d31700 1 --2- 172.21.15.107:0/404876178 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5cf0102490 0x7f5cf01028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:54.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.767+0000 7f5cf6d31700 1 -- 172.21.15.107:0/404876178 >> 172.21.15.107:0/404876178 conn(0x7f5cf00fc850 msgr2=0x7f5cf00fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:28:54.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.767+0000 7f5cf6d31700 1 -- 172.21.15.107:0/404876178 shutdown_connections 2024-05-09T08:28:54.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.767+0000 7f5cf6d31700 1 -- 172.21.15.107:0/404876178 wait complete. 2024-05-09T08:28:54.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.767+0000 7f5cf6d31700 1 Processor -- start 2024-05-09T08:28:54.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.767+0000 7f5cf6d31700 1 -- start start 2024-05-09T08:28:54.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf6d31700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5cf0101290 0x7f5cf006b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:54.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf6d31700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5cf0102490 0x7f5cf019d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:54.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf6d31700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cf0102e30 0x7f5cf019f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:54.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf6d31700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf010cd20 con 0x7f5cf0102e30 2024-05-09T08:28:54.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf6d31700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf010cba0 con 0x7f5cf0101290 2024-05-09T08:28:54.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf6d31700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf010cea0 con 0x7f5cf0102490 2024-05-09T08:28:54.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf52ce700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cf0102e30 0x7f5cf019f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:54.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf52ce700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cf0102e30 0x7f5cf019f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38160/0 (socket says 172.21.15.107:38160) 2024-05-09T08:28:54.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf52ce700 1 -- 172.21.15.107:0/1678308323 learned_addr learned my addr 172.21.15.107:0/1678308323 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:28:54.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5ceffff700 1 --2- 172.21.15.107:0/1678308323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5cf0102490 0x7f5cf019d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:54.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf4acd700 1 --2- 172.21.15.107:0/1678308323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5cf0101290 0x7f5cf006b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:54.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf52ce700 1 -- 172.21.15.107:0/1678308323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5cf0102490 msgr2=0x7f5cf019d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:54.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf52ce700 1 --2- 172.21.15.107:0/1678308323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5cf0102490 0x7f5cf019d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:54.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf52ce700 1 -- 172.21.15.107:0/1678308323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5cf0101290 msgr2=0x7f5cf006b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:28:54.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf52ce700 1 --2- 172.21.15.107:0/1678308323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5cf0101290 0x7f5cf006b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:28:54.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf52ce700 1 -- 172.21.15.107:0/1678308323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf01a01c0 con 0x7f5cf0102e30 2024-05-09T08:28:54.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf52ce700 1 --2- 172.21.15.107:0/1678308323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cf0102e30 0x7f5cf019f8f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800c970 tx=0x7f5ce800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:54.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cedffb700 1 -- 172.21.15.107:0/1678308323 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8007b00 con 0x7f5cf0102e30 2024-05-09T08:28:54.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cedffb700 1 -- 172.21.15.107:0/1678308323 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ce8004ce0 con 0x7f5cf0102e30 2024-05-09T08:28:54.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cf6d31700 1 -- 172.21.15.107:0/1678308323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf01a04b0 con 0x7f5cf0102e30 2024-05-09T08:28:54.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.768+0000 7f5cedffb700 1 -- 172.21.15.107:0/1678308323 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8005590 con 0x7f5cf0102e30 2024-05-09T08:28:54.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.769+0000 7f5cf6d31700 1 -- 172.21.15.107:0/1678308323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf01a09f0 con 0x7f5cf0102e30 2024-05-09T08:28:54.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.769+0000 7f5cf6d31700 1 -- 172.21.15.107:0/1678308323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cf0069010 con 0x7f5cf0102e30 2024-05-09T08:28:54.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.770+0000 7f5cedffb700 1 -- 172.21.15.107:0/1678308323 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f5ce8004750 con 0x7f5cf0102e30 2024-05-09T08:28:54.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.771+0000 7f5cedffb700 1 --2- 172.21.15.107:0/1678308323 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5ce00a2c20 0x7f5ce00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:28:54.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.771+0000 7f5cf4acd700 1 --2- 172.21.15.107:0/1678308323 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5ce00a2c20 0x7f5ce00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:28:54.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.771+0000 7f5cedffb700 1 -- 172.21.15.107:0/1678308323 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5858+0+0 (secure 0 0 0) 0x7f5ce80ce360 con 0x7f5cf0102e30 2024-05-09T08:28:54.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.772+0000 7f5cf4acd700 1 --2- 172.21.15.107:0/1678308323 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5ce00a2c20 0x7f5ce00a50e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4009fd0 tx=0x7f5ce400b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:28:54.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.776+0000 7f5cedffb700 1 -- 172.21.15.107:0/1678308323 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5ce8014030 con 0x7f5cf0102e30 2024-05-09T08:28:54.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:28:54.982+0000 7f5cf6d31700 1 -- 172.21.15.107:0/1678308323 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f5cf0062760 con 0x7f5ce00a2c20 2024-05-09T08:28:55.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:55 smithi107 ceph-mon[30122]: Detected new or changed devices on smithi157 2024-05-09T08:28:55.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:55 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi157 to 5696M 2024-05-09T08:28:55.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:28:55.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:28:55.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:55.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:55 smithi157 ceph-mon[33614]: Detected new or changed devices on smithi157 2024-05-09T08:28:55.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:55 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi157 to 5696M 2024-05-09T08:28:55.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:28:55.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:28:55.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:55.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:55 smithi110 ceph-mon[33622]: Detected new or changed devices on smithi157 2024-05-09T08:28:55.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:55 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi157 to 5696M 2024-05-09T08:28:55.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:28:55.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:28:55.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:28:56.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:56 smithi107 ceph-mon[30122]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:28:56.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:56 smithi107 ceph-mon[30122]: osdmap e62: 9 total, 9 up, 9 in 2024-05-09T08:28:56.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:56 smithi107 ceph-mon[30122]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 684 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:28:56.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:56 smithi110 ceph-mon[33622]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:28:56.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:56 smithi110 ceph-mon[33622]: osdmap e62: 9 total, 9 up, 9 in 2024-05-09T08:28:56.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:56 smithi110 ceph-mon[33622]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 684 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:28:56.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:56 smithi157 ceph-mon[33614]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:28:56.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:56 smithi157 ceph-mon[33614]: osdmap e62: 9 total, 9 up, 9 in 2024-05-09T08:28:56.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:56 smithi157 ceph-mon[33614]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 684 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:28:57.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:57 smithi107 ceph-mon[30122]: osdmap e63: 9 total, 9 up, 9 in 2024-05-09T08:28:57.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:57 smithi110 ceph-mon[33622]: osdmap e63: 9 total, 9 up, 9 in 2024-05-09T08:28:57.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:57 smithi157 ceph-mon[33614]: osdmap e63: 9 total, 9 up, 9 in 2024-05-09T08:28:58.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:58 smithi110 ceph-mon[33622]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-05-09T08:28:58.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:58 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1542248806' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56600856-ce7c-47d1-aff0-a57cf7ce75e8"}]: dispatch 2024-05-09T08:28:58.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:58 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1542248806' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56600856-ce7c-47d1-aff0-a57cf7ce75e8"}]': finished 2024-05-09T08:28:58.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:58 smithi110 ceph-mon[33622]: osdmap e64: 10 total, 9 up, 10 in 2024-05-09T08:28:58.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:28:58.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:58 smithi107 ceph-mon[30122]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-05-09T08:28:58.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:58 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1542248806' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56600856-ce7c-47d1-aff0-a57cf7ce75e8"}]: dispatch 2024-05-09T08:28:58.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:58 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1542248806' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56600856-ce7c-47d1-aff0-a57cf7ce75e8"}]': finished 2024-05-09T08:28:58.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:58 smithi107 ceph-mon[30122]: osdmap e64: 10 total, 9 up, 10 in 2024-05-09T08:28:58.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:28:58.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:58 smithi157 ceph-mon[33614]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-05-09T08:28:58.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:58 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1542248806' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56600856-ce7c-47d1-aff0-a57cf7ce75e8"}]: dispatch 2024-05-09T08:28:58.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:58 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1542248806' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56600856-ce7c-47d1-aff0-a57cf7ce75e8"}]': finished 2024-05-09T08:28:58.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:58 smithi157 ceph-mon[33614]: osdmap e64: 10 total, 9 up, 10 in 2024-05-09T08:28:58.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:28:59.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:28:59 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1794467884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:28:59.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:28:59 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1794467884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:28:59.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:28:59 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1794467884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:29:00.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:00 smithi110 ceph-mon[33622]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-09T08:29:00.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:00 smithi107 ceph-mon[30122]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-09T08:29:00.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:00 smithi157 ceph-mon[33614]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-09T08:29:02.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:02 smithi107 ceph-mon[30122]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-05-09T08:29:02.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:02 smithi157 ceph-mon[33614]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-05-09T08:29:02.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:02 smithi110 ceph-mon[33622]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-05-09T08:29:04.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:04 smithi107 ceph-mon[30122]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:29:04.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:04 smithi157 ceph-mon[33614]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:29:04.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:04 smithi110 ceph-mon[33622]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T08:29:06.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:06 smithi107 ceph-mon[30122]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-05-09T08:29:06.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:06 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-09T08:29:06.635 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:06 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:06.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:06 smithi157 ceph-mon[33614]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-05-09T08:29:06.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:06 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-09T08:29:06.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:06 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:06.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:06 smithi110 ceph-mon[33622]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-05-09T08:29:06.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:06 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-09T08:29:06.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:06 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:07.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:07 smithi107 ceph-mon[30122]: Deploying daemon osd.9 on smithi107 2024-05-09T08:29:07.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:07 smithi157 ceph-mon[33614]: Deploying daemon osd.9 on smithi107 2024-05-09T08:29:07.869 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:07 smithi110 ceph-mon[33622]: Deploying daemon osd.9 on smithi107 2024-05-09T08:29:08.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:08 smithi107 ceph-mon[30122]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:08.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:08 smithi157 ceph-mon[33614]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:08.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:08 smithi110 ceph-mon[33622]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:10.631 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:10 smithi107 ceph-mon[30122]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:10.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:10 smithi157 ceph-mon[33614]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:10.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:10 smithi110 ceph-mon[33622]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:12.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:12 smithi107 ceph-mon[30122]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:12.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:12 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:12.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:12 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:12.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:12 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:29:12.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:12 smithi157 ceph-mon[33614]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:12.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:12 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:12.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:12 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:12.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:12 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:29:12.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:12 smithi110 ceph-mon[33622]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:12.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:12 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:12.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:12 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:12.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:12 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:29:14.728 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:14 smithi107 ceph-mon[30122]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:14.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:14 smithi157 ceph-mon[33614]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:14.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:14 smithi110 ceph-mon[33622]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:15.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:29:15.908+0000 7f5cedffb700 1 -- 172.21.15.107:0/1678308323 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5cf0062760 con 0x7f5ce00a2c20 2024-05-09T08:29:15.908 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 9 on host 'smithi107' 2024-05-09T08:29:15.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:29:15.911+0000 7f5cf6d31700 1 -- 172.21.15.107:0/1678308323 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5ce00a2c20 msgr2=0x7f5ce00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:15.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:29:15.911+0000 7f5cf6d31700 1 --2- 172.21.15.107:0/1678308323 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5ce00a2c20 0x7f5ce00a50e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4009fd0 tx=0x7f5ce400b040 comp rx=0 tx=0).stop 2024-05-09T08:29:15.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:29:15.911+0000 7f5cf6d31700 1 -- 172.21.15.107:0/1678308323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cf0102e30 msgr2=0x7f5cf019f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:15.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:29:15.911+0000 7f5cf6d31700 1 --2- 172.21.15.107:0/1678308323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cf0102e30 0x7f5cf019f8f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800c970 tx=0x7f5ce800cd30 comp rx=0 tx=0).stop 2024-05-09T08:29:15.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:29:15.911+0000 7f5cf6d31700 1 -- 172.21.15.107:0/1678308323 shutdown_connections 2024-05-09T08:29:15.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:29:15.911+0000 7f5cf6d31700 1 --2- 172.21.15.107:0/1678308323 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5ce00a2c20 0x7f5ce00a50e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:15.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:29:15.911+0000 7f5cf6d31700 1 --2- 172.21.15.107:0/1678308323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cf0102e30 0x7f5cf019f8f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:15.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:29:15.911+0000 7f5cf6d31700 1 --2- 172.21.15.107:0/1678308323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5cf0101290 0x7f5cf006b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:15.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:29:15.911+0000 7f5cf6d31700 1 --2- 172.21.15.107:0/1678308323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5cf0102490 0x7f5cf019d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:15.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:29:15.911+0000 7f5cf6d31700 1 -- 172.21.15.107:0/1678308323 >> 172.21.15.107:0/1678308323 conn(0x7f5cf00fc850 msgr2=0x7f5cf0105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:15.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:29:15.911+0000 7f5cf6d31700 1 -- 172.21.15.107:0/1678308323 shutdown_connections 2024-05-09T08:29:15.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:29:15.911+0000 7f5cf6d31700 1 -- 172.21.15.107:0/1678308323 wait complete. 2024-05-09T08:29:16.221 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:15 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:16.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:15 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:16.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:15 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:16.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:15 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:29:16.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:15 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:16.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:15 smithi107 ceph-mon[30122]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:16.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:15 smithi107 ceph-mon[30122]: from='osd.9 [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-09T08:29:16.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:15 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:16.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:15 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:16.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:15 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:29:16.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:15 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:16.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:15 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:16.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:15 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:16.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:15 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:29:16.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:15 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:16.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:16 smithi157 ceph-mon[33614]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:16.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:16 smithi157 ceph-mon[33614]: from='osd.9 [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-09T08:29:16.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:16 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:16.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:16 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:16.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:16 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:29:16.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:15 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:16.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:15 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:16.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:15 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:16.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:15 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:29:16.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:15 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:16.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:15 smithi110 ceph-mon[33622]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:16.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:16 smithi110 ceph-mon[33622]: from='osd.9 [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-09T08:29:16.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:16 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:16.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:16 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:16.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:16 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:29:16.575 DEBUG:teuthology.orchestra.run.smithi107:osd.9> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.9.service 2024-05-09T08:29:16.577 INFO:tasks.cephadm:Deploying osd.10 on smithi110 with /dev/vg_nvme/lv_1... 2024-05-09T08:29:16.577 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-09T08:29:16.824 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:29:17.265 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:17 smithi110 ceph-mon[33622]: from='osd.9 [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-09T08:29:17.265 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:17 smithi110 ceph-mon[33622]: osdmap e65: 10 total, 9 up, 10 in 2024-05-09T08:29:17.265 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:17 smithi110 ceph-mon[33622]: from='osd.9 [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-05-09T08:29:17.265 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:17 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:29:17.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:17 smithi107 ceph-mon[30122]: from='osd.9 [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-09T08:29:17.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:17 smithi107 ceph-mon[30122]: osdmap e65: 10 total, 9 up, 10 in 2024-05-09T08:29:17.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:17 smithi107 ceph-mon[30122]: from='osd.9 [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-05-09T08:29:17.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:17 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:29:17.280 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:29:17 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:29:17.021+0000 7f7bddd50700 -1 osd.9 0 waiting for initial osdmap 2024-05-09T08:29:17.280 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:29:17 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:29:17.033+0000 7f7bd9b43700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:29:17.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:17 smithi157 ceph-mon[33614]: from='osd.9 [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-09T08:29:17.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:17 smithi157 ceph-mon[33614]: osdmap e65: 10 total, 9 up, 10 in 2024-05-09T08:29:17.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:17 smithi157 ceph-mon[33614]: from='osd.9 [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-05-09T08:29:17.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:17 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:29:18.066 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:18 smithi107 ceph-mon[30122]: from='osd.9 [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-05-09T08:29:18.066 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:18 smithi107 ceph-mon[30122]: osdmap e66: 10 total, 9 up, 10 in 2024-05-09T08:29:18.066 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:18 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:29:18.067 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:18 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:29:18.067 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:18 smithi107 ceph-mon[30122]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:18.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:18 smithi110 ceph-mon[33622]: from='osd.9 [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-05-09T08:29:18.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:18 smithi110 ceph-mon[33622]: osdmap e66: 10 total, 9 up, 10 in 2024-05-09T08:29:18.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:18 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:29:18.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:18 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:29:18.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:18 smithi110 ceph-mon[33622]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:18.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:18 smithi157 ceph-mon[33614]: from='osd.9 [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-05-09T08:29:18.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:18 smithi157 ceph-mon[33614]: osdmap e66: 10 total, 9 up, 10 in 2024-05-09T08:29:18.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:18 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:29:18.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:18 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:29:18.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:18 smithi157 ceph-mon[33614]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 804 GiB / 805 GiB avail 2024-05-09T08:29:18.792 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-09T08:29:18.792 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-09T08:29:18.792 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-05-09T08:29:18.792 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-05-09T08:29:18.792 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0083759 s, 1.3 GB/s 2024-05-09T08:29:18.792 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-05-09T08:29:19.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:19 smithi107 ceph-mon[30122]: purged_snaps scrub starts 2024-05-09T08:29:19.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:19 smithi107 ceph-mon[30122]: purged_snaps scrub ok 2024-05-09T08:29:19.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:19 smithi107 ceph-mon[30122]: osd.9 [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916] boot 2024-05-09T08:29:19.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:19 smithi107 ceph-mon[30122]: osdmap e67: 10 total, 10 up, 10 in 2024-05-09T08:29:19.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:19 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:29:19.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:19 smithi157 ceph-mon[33614]: purged_snaps scrub starts 2024-05-09T08:29:19.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:19 smithi157 ceph-mon[33614]: purged_snaps scrub ok 2024-05-09T08:29:19.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:19 smithi157 ceph-mon[33614]: osd.9 [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916] boot 2024-05-09T08:29:19.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:19 smithi157 ceph-mon[33614]: osdmap e67: 10 total, 10 up, 10 in 2024-05-09T08:29:19.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:19 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:29:19.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:19 smithi110 ceph-mon[33622]: purged_snaps scrub starts 2024-05-09T08:29:19.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:19 smithi110 ceph-mon[33622]: purged_snaps scrub ok 2024-05-09T08:29:19.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:19 smithi110 ceph-mon[33622]: osd.9 [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916] boot 2024-05-09T08:29:19.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:19 smithi110 ceph-mon[33622]: osdmap e67: 10 total, 10 up, 10 in 2024-05-09T08:29:19.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:19 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:29:19.471 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi110:vg_nvme/lv_1 2024-05-09T08:29:19.715 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:29:20.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:20 smithi107 ceph-mon[30122]: osdmap e68: 10 total, 10 up, 10 in 2024-05-09T08:29:20.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:20 smithi107 ceph-mon[30122]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:20.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:20.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:20.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:20.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:20.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:20.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:20.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:20.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:20.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:29:20.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:20.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:20 smithi157 ceph-mon[33614]: osdmap e68: 10 total, 10 up, 10 in 2024-05-09T08:29:20.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:20 smithi157 ceph-mon[33614]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:20.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:20.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:20.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:20.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:20.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:20.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:20.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:20.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:20.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:29:20.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:20.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:20 smithi110 ceph-mon[33622]: osdmap e68: 10 total, 10 up, 10 in 2024-05-09T08:29:20.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:20 smithi110 ceph-mon[33622]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:20.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:20.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:20.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:20.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:20.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:20.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:20.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:20.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:20.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:29:20.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:21.297 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:21 smithi110 ceph-mon[33622]: Detected new or changed devices on smithi107 2024-05-09T08:29:21.297 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:21 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi107 to 4272M 2024-05-09T08:29:21.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.342+0000 7fb87e792700 1 -- 172.21.15.110:0/3987697586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb878101280 msgr2=0x7fb8781016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.342+0000 7fb87e792700 1 --2- 172.21.15.110:0/3987697586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb878101280 0x7fb8781016a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb8680099c0 tx=0x7fb868009cd0 comp rx=0 tx=0).stop 2024-05-09T08:29:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.342+0000 7fb87e792700 1 -- 172.21.15.110:0/3987697586 shutdown_connections 2024-05-09T08:29:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.342+0000 7fb87e792700 1 --2- 172.21.15.110:0/3987697586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb878102e20 0x7fb8781096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.342+0000 7fb87e792700 1 --2- 172.21.15.110:0/3987697586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb878101280 0x7fb8781016a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.342+0000 7fb87e792700 1 --2- 172.21.15.110:0/3987697586 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb878102480 0x7fb8781028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.342+0000 7fb87e792700 1 -- 172.21.15.110:0/3987697586 >> 172.21.15.110:0/3987697586 conn(0x7fb8780fc820 msgr2=0x7fb8780fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.342+0000 7fb87e792700 1 -- 172.21.15.110:0/3987697586 shutdown_connections 2024-05-09T08:29:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.342+0000 7fb87e792700 1 -- 172.21.15.110:0/3987697586 wait complete. 2024-05-09T08:29:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.343+0000 7fb87e792700 1 Processor -- start 2024-05-09T08:29:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.343+0000 7fb87e792700 1 -- start start 2024-05-09T08:29:21.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.343+0000 7fb87e792700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb878101280 0x7fb87806b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:21.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.343+0000 7fb87e792700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb878102480 0x7fb87819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:21.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.343+0000 7fb87e792700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb878102e20 0x7fb8781a1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:21.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.343+0000 7fb87e792700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb87810cd10 con 0x7fb878102e20 2024-05-09T08:29:21.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.343+0000 7fb87e792700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb87810cb90 con 0x7fb878101280 2024-05-09T08:29:21.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.343+0000 7fb87e792700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb87810ce90 con 0x7fb878102480 2024-05-09T08:29:21.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.343+0000 7fb877fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb878101280 0x7fb87806b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:21.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.343+0000 7fb877fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb878101280 0x7fb87806b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:58114/0 (socket says 172.21.15.110:58114) 2024-05-09T08:29:21.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.343+0000 7fb877fff700 1 -- 172.21.15.110:0/421338590 learned_addr learned my addr 172.21.15.110:0/421338590 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-05-09T08:29:21.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.343+0000 7fb8777fe700 1 --2- 172.21.15.110:0/421338590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb878102480 0x7fb87819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:21.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.343+0000 7fb87cd2f700 1 --2- 172.21.15.110:0/421338590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb878102e20 0x7fb8781a1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:21.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.344+0000 7fb877fff700 1 -- 172.21.15.110:0/421338590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb878102480 msgr2=0x7fb87819f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:21.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.344+0000 7fb877fff700 1 --2- 172.21.15.110:0/421338590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb878102480 0x7fb87819f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:21.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.344+0000 7fb877fff700 1 -- 172.21.15.110:0/421338590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb878102e20 msgr2=0x7fb8781a1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:21.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.344+0000 7fb877fff700 1 --2- 172.21.15.110:0/421338590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb878102e20 0x7fb8781a1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:21.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.344+0000 7fb877fff700 1 -- 172.21.15.110:0/421338590 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8781a2360 con 0x7fb878101280 2024-05-09T08:29:21.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.344+0000 7fb8777fe700 1 --2- 172.21.15.110:0/421338590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb878102480 0x7fb87819f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:29:21.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.344+0000 7fb877fff700 1 --2- 172.21.15.110:0/421338590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb878101280 0x7fb87806b560 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb86800fa00 tx=0x7fb86800fae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:21.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.344+0000 7fb8757fa700 1 -- 172.21.15.110:0/421338590 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb868027070 con 0x7fb878101280 2024-05-09T08:29:21.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.344+0000 7fb87e792700 1 -- 172.21.15.110:0/421338590 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8781a25f0 con 0x7fb878101280 2024-05-09T08:29:21.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.344+0000 7fb8757fa700 1 -- 172.21.15.110:0/421338590 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb868005940 con 0x7fb878101280 2024-05-09T08:29:21.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.344+0000 7fb8757fa700 1 -- 172.21.15.110:0/421338590 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb868017650 con 0x7fb878101280 2024-05-09T08:29:21.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.344+0000 7fb87e792700 1 -- 172.21.15.110:0/421338590 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8781a2b20 con 0x7fb878101280 2024-05-09T08:29:21.348 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:21 smithi157 ceph-mon[33614]: Detected new or changed devices on smithi107 2024-05-09T08:29:21.348 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:21 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi107 to 4272M 2024-05-09T08:29:21.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.345+0000 7fb87e792700 1 -- 172.21.15.110:0/421338590 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb878069010 con 0x7fb878101280 2024-05-09T08:29:21.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.347+0000 7fb8757fa700 1 -- 172.21.15.110:0/421338590 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fb86802a070 con 0x7fb878101280 2024-05-09T08:29:21.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.348+0000 7fb8757fa700 1 --2- 172.21.15.110:0/421338590 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb8600a2c20 0x7fb8600a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:21.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.348+0000 7fb8757fa700 1 -- 172.21.15.110:0/421338590 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7fb8680cfac0 con 0x7fb878101280 2024-05-09T08:29:21.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.348+0000 7fb8777fe700 1 --2- 172.21.15.110:0/421338590 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb8600a2c20 0x7fb8600a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:21.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.348+0000 7fb8777fe700 1 --2- 172.21.15.110:0/421338590 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb8600a2c20 0x7fb8600a50e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb86c00bcf0 tx=0x7fb86c00b2c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:21.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.351+0000 7fb8757fa700 1 -- 172.21.15.110:0/421338590 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb868031080 con 0x7fb878101280 2024-05-09T08:29:21.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:21 smithi107 ceph-mon[30122]: Detected new or changed devices on smithi107 2024-05-09T08:29:21.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:21 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi107 to 4272M 2024-05-09T08:29:21.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:21.554+0000 7fb87e792700 1 -- 172.21.15.110:0/421338590 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fb878062760 con 0x7fb8600a2c20 2024-05-09T08:29:22.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:22 smithi157 ceph-mon[33614]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:22.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:22 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:29:22.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:22 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:29:22.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:22 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:22.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:22 smithi110 ceph-mon[33622]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:22.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:22 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:29:22.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:22 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:29:22.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:22 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:22.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:22 smithi107 ceph-mon[30122]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:22.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:22 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:29:22.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:22 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:29:22.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:22 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:23.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:23 smithi157 ceph-mon[33614]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:29:23.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:23 smithi110 ceph-mon[33622]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:29:23.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:23 smithi107 ceph-mon[30122]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:29:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:24 smithi157 ceph-mon[33614]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:24.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:24 smithi110 ceph-mon[33622]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:24.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:24 smithi107 ceph-mon[30122]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:25.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:25 smithi157 ceph-mon[33614]: from='client.? 172.21.15.110:0/296627999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4022636-ba24-4685-9678-d261e04df620"}]: dispatch 2024-05-09T08:29:25.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:25 smithi157 ceph-mon[33614]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4022636-ba24-4685-9678-d261e04df620"}]: dispatch 2024-05-09T08:29:25.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:25 smithi157 ceph-mon[33614]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4022636-ba24-4685-9678-d261e04df620"}]': finished 2024-05-09T08:29:25.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:25 smithi157 ceph-mon[33614]: osdmap e69: 11 total, 10 up, 11 in 2024-05-09T08:29:25.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:25 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:29:25.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:25 smithi110 ceph-mon[33622]: from='client.? 172.21.15.110:0/296627999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4022636-ba24-4685-9678-d261e04df620"}]: dispatch 2024-05-09T08:29:25.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:25 smithi110 ceph-mon[33622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4022636-ba24-4685-9678-d261e04df620"}]: dispatch 2024-05-09T08:29:25.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:25 smithi110 ceph-mon[33622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4022636-ba24-4685-9678-d261e04df620"}]': finished 2024-05-09T08:29:25.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:25 smithi110 ceph-mon[33622]: osdmap e69: 11 total, 10 up, 11 in 2024-05-09T08:29:25.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:25 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:29:25.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:25 smithi107 ceph-mon[30122]: from='client.? 172.21.15.110:0/296627999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4022636-ba24-4685-9678-d261e04df620"}]: dispatch 2024-05-09T08:29:25.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:25 smithi107 ceph-mon[30122]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4022636-ba24-4685-9678-d261e04df620"}]: dispatch 2024-05-09T08:29:25.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:25 smithi107 ceph-mon[30122]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4022636-ba24-4685-9678-d261e04df620"}]': finished 2024-05-09T08:29:25.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:25 smithi107 ceph-mon[30122]: osdmap e69: 11 total, 10 up, 11 in 2024-05-09T08:29:25.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:25 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:29:26.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:26 smithi157 ceph-mon[33614]: from='client.? 172.21.15.110:0/1267337643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:29:26.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:26 smithi157 ceph-mon[33614]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:26.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:26 smithi110 ceph-mon[33622]: from='client.? 172.21.15.110:0/1267337643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:29:26.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:26 smithi110 ceph-mon[33622]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:26.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:26 smithi107 ceph-mon[30122]: from='client.? 172.21.15.110:0/1267337643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:29:26.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:26 smithi107 ceph-mon[30122]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:28.699 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:28 smithi110 ceph-mon[33622]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:28.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:28 smithi107 ceph-mon[30122]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:28.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:28 smithi157 ceph-mon[33614]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:30.762 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:30 smithi110 ceph-mon[33622]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:30.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:30 smithi107 ceph-mon[30122]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:30.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:30 smithi157 ceph-mon[33614]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:32.693 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:32 smithi110 ceph-mon[33622]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:32.694 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-09T08:29:32.694 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:32.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:32 smithi107 ceph-mon[30122]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:32.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-09T08:29:32.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:32.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:32 smithi157 ceph-mon[33614]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:32.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-09T08:29:32.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:33.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:33 smithi110 ceph-mon[33622]: Deploying daemon osd.10 on smithi110 2024-05-09T08:29:33.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:33 smithi107 ceph-mon[30122]: Deploying daemon osd.10 on smithi110 2024-05-09T08:29:33.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:33 smithi157 ceph-mon[33614]: Deploying daemon osd.10 on smithi110 2024-05-09T08:29:34.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:34 smithi110 ceph-mon[33622]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:34.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:34 smithi107 ceph-mon[30122]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:34.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:34 smithi157 ceph-mon[33614]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:36.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:36 smithi107 ceph-mon[30122]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:36.814 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:36 smithi110 ceph-mon[33622]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:36.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:36 smithi157 ceph-mon[33614]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:38.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:38 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:38.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:38 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:29:38.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:38 smithi110 ceph-mon[33622]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:38.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:38 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:38.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:38 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:38.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:38 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:29:38.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:38 smithi107 ceph-mon[30122]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:38.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:38 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:38.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:38 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:38.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:38 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:29:38.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:38 smithi157 ceph-mon[33614]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:38.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:38 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:40.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:40 smithi110 ceph-mon[33622]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:40.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:40 smithi107 ceph-mon[30122]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:40.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:40 smithi157 ceph-mon[33614]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:41.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:41.018+0000 7fb8757fa700 1 -- 172.21.15.110:0/421338590 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fb878062760 con 0x7fb8600a2c20 2024-05-09T08:29:41.018 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 10 on host 'smithi110' 2024-05-09T08:29:41.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:41.020+0000 7fb87e792700 1 -- 172.21.15.110:0/421338590 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb8600a2c20 msgr2=0x7fb8600a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:41.020+0000 7fb87e792700 1 --2- 172.21.15.110:0/421338590 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb8600a2c20 0x7fb8600a50e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb86c00bcf0 tx=0x7fb86c00b2c0 comp rx=0 tx=0).stop 2024-05-09T08:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:41.020+0000 7fb87e792700 1 -- 172.21.15.110:0/421338590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb878101280 msgr2=0x7fb87806b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:41.020+0000 7fb87e792700 1 --2- 172.21.15.110:0/421338590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb878101280 0x7fb87806b560 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb86800fa00 tx=0x7fb86800fae0 comp rx=0 tx=0).stop 2024-05-09T08:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:41.020+0000 7fb87e792700 1 -- 172.21.15.110:0/421338590 shutdown_connections 2024-05-09T08:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:41.020+0000 7fb87e792700 1 --2- 172.21.15.110:0/421338590 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb8600a2c20 0x7fb8600a50e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:41.020+0000 7fb87e792700 1 --2- 172.21.15.110:0/421338590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb878102e20 0x7fb8781a1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:41.020+0000 7fb87e792700 1 --2- 172.21.15.110:0/421338590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb878101280 0x7fb87806b560 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:41.020+0000 7fb87e792700 1 --2- 172.21.15.110:0/421338590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb878102480 0x7fb87819f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:41.020+0000 7fb87e792700 1 -- 172.21.15.110:0/421338590 >> 172.21.15.110:0/421338590 conn(0x7fb8780fc820 msgr2=0x7fb8781057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:41.021+0000 7fb87e792700 1 -- 172.21.15.110:0/421338590 shutdown_connections 2024-05-09T08:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-05-09T08:29:41.021+0000 7fb87e792700 1 -- 172.21.15.110:0/421338590 wait complete. 2024-05-09T08:29:41.633 DEBUG:teuthology.orchestra.run.smithi110:osd.10> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.10.service 2024-05-09T08:29:41.636 INFO:tasks.cephadm:Deploying osd.11 on smithi157 with /dev/vg_nvme/lv_1... 2024-05-09T08:29:41.636 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-09T08:29:41.887 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:29:42.259 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:41 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:42.259 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:41 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:42.260 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:41 smithi157 ceph-mon[33614]: from='osd.10 [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:29:42.260 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:41 smithi157 ceph-mon[33614]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:29:42.260 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:41 smithi157 ceph-mon[33614]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:42.260 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:41 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:42.260 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:41 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:42.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:41 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:42.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:41 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:42.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:41 smithi107 ceph-mon[30122]: from='osd.10 [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:29:42.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:41 smithi107 ceph-mon[30122]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:29:42.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:41 smithi107 ceph-mon[30122]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:42.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:41 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:42.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:41 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:42.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:41 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:42.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:41 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:42.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:41 smithi110 ceph-mon[33622]: from='osd.10 [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:29:42.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:41 smithi110 ceph-mon[33622]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T08:29:42.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:41 smithi110 ceph-mon[33622]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:42.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:41 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:42.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:41 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:43.257 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:43 smithi157 ceph-mon[33614]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-09T08:29:43.257 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:43 smithi157 ceph-mon[33614]: osdmap e70: 11 total, 10 up, 11 in 2024-05-09T08:29:43.257 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:43 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:29:43.257 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:43 smithi157 ceph-mon[33614]: from='osd.10 [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:29:43.257 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:43 smithi157 ceph-mon[33614]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:29:43.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:43 smithi107 ceph-mon[30122]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-09T08:29:43.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:43 smithi107 ceph-mon[30122]: osdmap e70: 11 total, 10 up, 11 in 2024-05-09T08:29:43.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:43 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:29:43.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:43 smithi107 ceph-mon[30122]: from='osd.10 [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:29:43.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:43 smithi107 ceph-mon[30122]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:29:43.334 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:43 smithi110 ceph-mon[33622]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-09T08:29:43.334 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:43 smithi110 ceph-mon[33622]: osdmap e70: 11 total, 10 up, 11 in 2024-05-09T08:29:43.334 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:43 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:29:43.334 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:43 smithi110 ceph-mon[33622]: from='osd.10 [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:29:43.334 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:43 smithi110 ceph-mon[33622]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-05-09T08:29:43.334 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:29:43 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:29:43.003+0000 7f75b8cbd700 -1 osd.10 0 waiting for initial osdmap 2024-05-09T08:29:43.334 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:29:43 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:29:43.017+0000 7f75b4ab0700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:29:43.974 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-09T08:29:43.974 INFO:teuthology.orchestra.run.smithi157.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-09T08:29:43.974 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10+0 records in 2024-05-09T08:29:43.974 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records out 2024-05-09T08:29:43.974 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0082907 s, 1.3 GB/s 2024-05-09T08:29:43.975 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping successful for: 2024-05-09T08:29:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:44 smithi107 ceph-mon[30122]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-05-09T08:29:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:44 smithi107 ceph-mon[30122]: osdmap e71: 11 total, 10 up, 11 in 2024-05-09T08:29:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:29:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:29:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:44 smithi107 ceph-mon[30122]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:44.281 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:44 smithi157 ceph-mon[33614]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-05-09T08:29:44.281 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:44 smithi157 ceph-mon[33614]: osdmap e71: 11 total, 10 up, 11 in 2024-05-09T08:29:44.281 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:29:44.281 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:29:44.281 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:44 smithi157 ceph-mon[33614]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:44 smithi110 ceph-mon[33622]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-05-09T08:29:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:44 smithi110 ceph-mon[33622]: osdmap e71: 11 total, 10 up, 11 in 2024-05-09T08:29:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:44 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:29:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:44 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:29:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:44 smithi110 ceph-mon[33622]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 894 GiB / 894 GiB avail 2024-05-09T08:29:44.801 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi157:vg_nvme/lv_1 2024-05-09T08:29:45.046 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:29:45.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:45 smithi157 ceph-mon[33614]: purged_snaps scrub starts 2024-05-09T08:29:45.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:45 smithi157 ceph-mon[33614]: purged_snaps scrub ok 2024-05-09T08:29:45.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:45 smithi157 ceph-mon[33614]: osd.10 [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764] boot 2024-05-09T08:29:45.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:45 smithi157 ceph-mon[33614]: osdmap e72: 11 total, 11 up, 11 in 2024-05-09T08:29:45.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:45 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:29:45.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:45 smithi157 ceph-mon[33614]: osdmap e73: 11 total, 11 up, 11 in 2024-05-09T08:29:45.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:45 smithi107 ceph-mon[30122]: purged_snaps scrub starts 2024-05-09T08:29:45.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:45 smithi107 ceph-mon[30122]: purged_snaps scrub ok 2024-05-09T08:29:45.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:45 smithi107 ceph-mon[30122]: osd.10 [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764] boot 2024-05-09T08:29:45.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:45 smithi107 ceph-mon[30122]: osdmap e72: 11 total, 11 up, 11 in 2024-05-09T08:29:45.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:45 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:29:45.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:45 smithi107 ceph-mon[30122]: osdmap e73: 11 total, 11 up, 11 in 2024-05-09T08:29:45.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:45 smithi110 ceph-mon[33622]: purged_snaps scrub starts 2024-05-09T08:29:45.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:45 smithi110 ceph-mon[33622]: purged_snaps scrub ok 2024-05-09T08:29:45.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:45 smithi110 ceph-mon[33622]: osd.10 [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764] boot 2024-05-09T08:29:45.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:45 smithi110 ceph-mon[33622]: osdmap e72: 11 total, 11 up, 11 in 2024-05-09T08:29:45.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:45 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:29:45.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:45 smithi110 ceph-mon[33622]: osdmap e73: 11 total, 11 up, 11 in 2024-05-09T08:29:46.148 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: Detected new or changed devices on smithi110 2024-05-09T08:29:46.149 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:46.149 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:46.149 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:46.149 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:46.149 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:46.149 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:46.149 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi110 to 4272M 2024-05-09T08:29:46.149 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:46.149 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:46.149 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:29:46.149 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:46.150 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:29:46.150 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:46.150 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:29:46.150 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:46.150 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:29:46.150 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:46 smithi157 ceph-mon[33614]: osdmap e74: 11 total, 11 up, 11 in 2024-05-09T08:29:46.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: Detected new or changed devices on smithi110 2024-05-09T08:29:46.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:46.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:46.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:46.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:46.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:46.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:46.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi110 to 4272M 2024-05-09T08:29:46.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:46.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:46.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:29:46.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:46.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:29:46.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:46.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:29:46.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:46.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:29:46.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:46 smithi110 ceph-mon[33622]: osdmap e74: 11 total, 11 up, 11 in 2024-05-09T08:29:46.468 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: Detected new or changed devices on smithi110 2024-05-09T08:29:46.469 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:46.469 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:46.469 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:46.469 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:46.469 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:46.469 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:29:46.469 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi110 to 4272M 2024-05-09T08:29:46.469 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:46.469 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:46.469 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:29:46.469 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:46.469 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:29:46.470 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:46.470 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:29:46.470 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:29:46.470 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:29:46.470 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:46 smithi107 ceph-mon[30122]: osdmap e74: 11 total, 11 up, 11 in 2024-05-09T08:29:46.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.693+0000 7f3abba48700 1 -- 172.21.15.157:0/15455503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ab4105f50 msgr2=0x7f3ab41063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:46.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.693+0000 7f3abba48700 1 --2- 172.21.15.157:0/15455503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ab4105f50 0x7f3ab41063d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3aac00b0a0 tx=0x7f3aac00b3b0 comp rx=0 tx=0).stop 2024-05-09T08:29:46.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.693+0000 7f3abba48700 1 -- 172.21.15.157:0/15455503 shutdown_connections 2024-05-09T08:29:46.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.693+0000 7f3abba48700 1 --2- 172.21.15.157:0/15455503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ab4105f50 0x7f3ab41063d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:46.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.693+0000 7f3abba48700 1 --2- 172.21.15.157:0/15455503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ab4100c10 0x7f3ab4103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:46.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.693+0000 7f3abba48700 1 --2- 172.21.15.157:0/15455503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ab4103580 0x7f3ab4105a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:46.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.693+0000 7f3abba48700 1 -- 172.21.15.157:0/15455503 >> 172.21.15.157:0/15455503 conn(0x7f3ab40fc850 msgr2=0x7f3ab40fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:29:46.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.693+0000 7f3abba48700 1 -- 172.21.15.157:0/15455503 shutdown_connections 2024-05-09T08:29:46.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.693+0000 7f3abba48700 1 -- 172.21.15.157:0/15455503 wait complete. 2024-05-09T08:29:46.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3abba48700 1 Processor -- start 2024-05-09T08:29:46.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3abba48700 1 -- start start 2024-05-09T08:29:46.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3abba48700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ab4100c10 0x7f3ab419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:46.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3abba48700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ab4103580 0x7f3ab419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:46.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3abba48700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ab4105f50 0x7f3ab41a6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:46.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3abba48700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab4114400 con 0x7f3ab4105f50 2024-05-09T08:29:46.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3abba48700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab4114280 con 0x7f3ab4100c10 2024-05-09T08:29:46.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3abba48700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab4114580 con 0x7f3ab4103580 2024-05-09T08:29:46.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3ab8fe3700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ab4103580 0x7f3ab419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:46.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3ab8fe3700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ab4103580 0x7f3ab419f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:34572/0 (socket says 172.21.15.157:34572) 2024-05-09T08:29:46.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3ab8fe3700 1 -- 172.21.15.157:0/1601006607 learned_addr learned my addr 172.21.15.157:0/1601006607 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-05-09T08:29:46.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3ab9fe5700 1 --2- 172.21.15.157:0/1601006607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ab4105f50 0x7f3ab41a6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:46.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3ab97e4700 1 --2- 172.21.15.157:0/1601006607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ab4100c10 0x7f3ab419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:46.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3ab8fe3700 1 -- 172.21.15.157:0/1601006607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ab4100c10 msgr2=0x7f3ab419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:46.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3ab8fe3700 1 --2- 172.21.15.157:0/1601006607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ab4100c10 0x7f3ab419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:46.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3ab8fe3700 1 -- 172.21.15.157:0/1601006607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ab4105f50 msgr2=0x7f3ab41a6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:29:46.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3ab8fe3700 1 --2- 172.21.15.157:0/1601006607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ab4105f50 0x7f3ab41a6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:29:46.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.694+0000 7f3ab8fe3700 1 -- 172.21.15.157:0/1601006607 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab41a6730 con 0x7f3ab4103580 2024-05-09T08:29:46.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.695+0000 7f3ab8fe3700 1 --2- 172.21.15.157:0/1601006607 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ab4103580 0x7f3ab419f0b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3aa400d9b0 tx=0x7f3aa400dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:46.697 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.695+0000 7f3aaa7fc700 1 -- 172.21.15.157:0/1601006607 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4016070 con 0x7f3ab4103580 2024-05-09T08:29:46.697 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.695+0000 7f3abba48700 1 -- 172.21.15.157:0/1601006607 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ab41a6a20 con 0x7f3ab4103580 2024-05-09T08:29:46.697 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.695+0000 7f3aaa7fc700 1 -- 172.21.15.157:0/1601006607 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3aa4004830 con 0x7f3ab4103580 2024-05-09T08:29:46.697 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.695+0000 7f3aaa7fc700 1 -- 172.21.15.157:0/1601006607 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa40054f0 con 0x7f3ab4103580 2024-05-09T08:29:46.697 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.695+0000 7f3abba48700 1 -- 172.21.15.157:0/1601006607 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ab41a6f60 con 0x7f3ab4103580 2024-05-09T08:29:46.697 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.696+0000 7f3abba48700 1 -- 172.21.15.157:0/1601006607 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ab4069010 con 0x7f3ab4103580 2024-05-09T08:29:46.698 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.697+0000 7f3aaa7fc700 1 -- 172.21.15.157:0/1601006607 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3aa40049e0 con 0x7f3ab4103580 2024-05-09T08:29:46.698 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.698+0000 7f3aaa7fc700 1 --2- 172.21.15.157:0/1601006607 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3aa00a2c20 0x7f3aa00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:29:46.698 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.698+0000 7f3aaa7fc700 1 -- 172.21.15.157:0/1601006607 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7f3aa40ced50 con 0x7f3ab4103580 2024-05-09T08:29:46.698 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.698+0000 7f3ab97e4700 1 --2- 172.21.15.157:0/1601006607 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3aa00a2c20 0x7f3aa00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:29:46.698 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.698+0000 7f3ab97e4700 1 --2- 172.21.15.157:0/1601006607 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3aa00a2c20 0x7f3aa00a50e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0006fd0 tx=0x7f3ab0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:29:46.700 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.701+0000 7f3aaa7fc700 1 -- 172.21.15.157:0/1601006607 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3aa401f020 con 0x7f3ab4103580 2024-05-09T08:29:46.907 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:29:46.908+0000 7f3abba48700 1 -- 172.21.15.157:0/1601006607 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3ab4062760 con 0x7f3aa00a2c20 2024-05-09T08:29:48.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:47 smithi157 ceph-mon[33614]: osdmap e75: 11 total, 11 up, 11 in 2024-05-09T08:29:48.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:47 smithi157 ceph-mon[33614]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:29:48.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:47 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:29:48.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:47 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:29:48.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:47 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:48.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:47 smithi157 ceph-mon[33614]: pgmap v241: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T08:29:48.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:47 smithi110 ceph-mon[33622]: osdmap e75: 11 total, 11 up, 11 in 2024-05-09T08:29:48.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:47 smithi110 ceph-mon[33622]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:29:48.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:47 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:29:48.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:47 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:29:48.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:47 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:48.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:47 smithi110 ceph-mon[33622]: pgmap v241: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T08:29:48.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:47 smithi107 ceph-mon[30122]: osdmap e75: 11 total, 11 up, 11 in 2024-05-09T08:29:48.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:47 smithi107 ceph-mon[30122]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:29:48.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:47 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T08:29:48.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:47 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T08:29:48.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:47 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:48.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:47 smithi107 ceph-mon[30122]: pgmap v241: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T08:29:49.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:48 smithi157 ceph-mon[33614]: osdmap e76: 11 total, 11 up, 11 in 2024-05-09T08:29:49.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:48 smithi110 ceph-mon[33622]: osdmap e76: 11 total, 11 up, 11 in 2024-05-09T08:29:49.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:48 smithi107 ceph-mon[30122]: osdmap e76: 11 total, 11 up, 11 in 2024-05-09T08:29:50.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:49 smithi157 ceph-mon[33614]: pgmap v243: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-05-09T08:29:50.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:49 smithi110 ceph-mon[33622]: pgmap v243: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-05-09T08:29:50.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:49 smithi107 ceph-mon[30122]: pgmap v243: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-05-09T08:29:51.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:50 smithi157 ceph-mon[33614]: from='client.? 172.21.15.157:0/4080159298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fef75204-3f64-4365-8ece-384eb4598b1b"}]: dispatch 2024-05-09T08:29:51.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:50 smithi157 ceph-mon[33614]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fef75204-3f64-4365-8ece-384eb4598b1b"}]: dispatch 2024-05-09T08:29:51.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:50 smithi157 ceph-mon[33614]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fef75204-3f64-4365-8ece-384eb4598b1b"}]': finished 2024-05-09T08:29:51.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:50 smithi157 ceph-mon[33614]: osdmap e77: 12 total, 11 up, 12 in 2024-05-09T08:29:51.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:50 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:29:51.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:50 smithi157 ceph-mon[33614]: from='client.? 172.21.15.157:0/3008405094' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:29:51.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:50 smithi110 ceph-mon[33622]: from='client.? 172.21.15.157:0/4080159298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fef75204-3f64-4365-8ece-384eb4598b1b"}]: dispatch 2024-05-09T08:29:51.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:50 smithi110 ceph-mon[33622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fef75204-3f64-4365-8ece-384eb4598b1b"}]: dispatch 2024-05-09T08:29:51.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:50 smithi110 ceph-mon[33622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fef75204-3f64-4365-8ece-384eb4598b1b"}]': finished 2024-05-09T08:29:51.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:50 smithi110 ceph-mon[33622]: osdmap e77: 12 total, 11 up, 12 in 2024-05-09T08:29:51.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:50 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:29:51.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:50 smithi110 ceph-mon[33622]: from='client.? 172.21.15.157:0/3008405094' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:29:51.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:50 smithi107 ceph-mon[30122]: from='client.? 172.21.15.157:0/4080159298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fef75204-3f64-4365-8ece-384eb4598b1b"}]: dispatch 2024-05-09T08:29:51.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:50 smithi107 ceph-mon[30122]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fef75204-3f64-4365-8ece-384eb4598b1b"}]: dispatch 2024-05-09T08:29:51.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:50 smithi107 ceph-mon[30122]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fef75204-3f64-4365-8ece-384eb4598b1b"}]': finished 2024-05-09T08:29:51.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:50 smithi107 ceph-mon[30122]: osdmap e77: 12 total, 11 up, 12 in 2024-05-09T08:29:51.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:50 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:29:51.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:50 smithi107 ceph-mon[30122]: from='client.? 172.21.15.157:0/3008405094' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T08:29:52.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:51 smithi157 ceph-mon[33614]: pgmap v245: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-05-09T08:29:52.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:51 smithi110 ceph-mon[33622]: pgmap v245: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-05-09T08:29:52.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:51 smithi107 ceph-mon[30122]: pgmap v245: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-05-09T08:29:54.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:54 smithi157 ceph-mon[33614]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:29:54.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:54 smithi107 ceph-mon[30122]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:29:54.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:54 smithi110 ceph-mon[33622]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:29:56.549 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:56 smithi157 ceph-mon[33614]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:29:56.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:56 smithi107 ceph-mon[30122]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:29:56.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:56 smithi110 ceph-mon[33622]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:29:58.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:58 smithi107 ceph-mon[30122]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:29:58.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-09T08:29:58.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:58 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:58.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:58 smithi157 ceph-mon[33614]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:29:58.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-09T08:29:58.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:58 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:58.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:58 smithi110 ceph-mon[33622]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:29:58.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-09T08:29:58.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:58 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:29:59.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:29:59 smithi107 ceph-mon[30122]: Deploying daemon osd.11 on smithi157 2024-05-09T08:29:59.841 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:29:59 smithi157 ceph-mon[33614]: Deploying daemon osd.11 on smithi157 2024-05-09T08:29:59.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:29:59 smithi110 ceph-mon[33622]: Deploying daemon osd.11 on smithi157 2024-05-09T08:30:00.716 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:00 smithi157 ceph-mon[33614]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:00.716 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:00 smithi157 ceph-mon[33614]: overall HEALTH_OK 2024-05-09T08:30:00.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:00 smithi107 ceph-mon[30122]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:00.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:00 smithi107 ceph-mon[30122]: overall HEALTH_OK 2024-05-09T08:30:00.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:00 smithi110 ceph-mon[33622]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:00.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:00 smithi110 ceph-mon[33622]: overall HEALTH_OK 2024-05-09T08:30:02.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:02 smithi107 ceph-mon[30122]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:02.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:02 smithi157 ceph-mon[33614]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:02.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:02 smithi110 ceph-mon[33622]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:04.529 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:04 smithi157 ceph-mon[33614]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:04.529 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:04 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:04.529 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:04 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:04.529 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:04 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:04.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:04 smithi107 ceph-mon[30122]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:04.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:04 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:04.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:04 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:04.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:04 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:04.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:04 smithi110 ceph-mon[33622]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:04.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:04 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:04.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:04 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:04.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:04 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:06.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:06 smithi107 ceph-mon[30122]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:06.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:06 smithi157 ceph-mon[33614]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:06.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:06 smithi110 ceph-mon[33622]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:06.973 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:06.973+0000 7f3aaa7fc700 1 -- 172.21.15.157:0/1601006607 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f3ab4062760 con 0x7f3aa00a2c20 2024-05-09T08:30:06.973 INFO:teuthology.orchestra.run.smithi157.stdout:Created osd(s) 11 on host 'smithi157' 2024-05-09T08:30:06.975 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:06.975+0000 7f3abba48700 1 -- 172.21.15.157:0/1601006607 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3aa00a2c20 msgr2=0x7f3aa00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:06.975 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:06.975+0000 7f3abba48700 1 --2- 172.21.15.157:0/1601006607 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3aa00a2c20 0x7f3aa00a50e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0006fd0 tx=0x7f3ab0008040 comp rx=0 tx=0).stop 2024-05-09T08:30:06.975 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:06.975+0000 7f3abba48700 1 -- 172.21.15.157:0/1601006607 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ab4103580 msgr2=0x7f3ab419f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:06.975 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:06.975+0000 7f3abba48700 1 --2- 172.21.15.157:0/1601006607 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ab4103580 0x7f3ab419f0b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3aa400d9b0 tx=0x7f3aa400dcc0 comp rx=0 tx=0).stop 2024-05-09T08:30:06.975 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:06.976+0000 7f3abba48700 1 -- 172.21.15.157:0/1601006607 shutdown_connections 2024-05-09T08:30:06.975 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:06.976+0000 7f3abba48700 1 --2- 172.21.15.157:0/1601006607 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3aa00a2c20 0x7f3aa00a50e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:06.975 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:06.976+0000 7f3abba48700 1 --2- 172.21.15.157:0/1601006607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ab4105f50 0x7f3ab41a6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:06.975 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:06.976+0000 7f3abba48700 1 --2- 172.21.15.157:0/1601006607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ab4100c10 0x7f3ab419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:06.975 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:06.976+0000 7f3abba48700 1 --2- 172.21.15.157:0/1601006607 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ab4103580 0x7f3ab419f0b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:06.976 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:06.976+0000 7f3abba48700 1 -- 172.21.15.157:0/1601006607 >> 172.21.15.157:0/1601006607 conn(0x7f3ab40fc850 msgr2=0x7f3ab4101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:06.976 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:06.976+0000 7f3abba48700 1 -- 172.21.15.157:0/1601006607 shutdown_connections 2024-05-09T08:30:06.976 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:06.976+0000 7f3abba48700 1 -- 172.21.15.157:0/1601006607 wait complete. 2024-05-09T08:30:07.983 DEBUG:teuthology.orchestra.run.smithi157:osd.11> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.11.service 2024-05-09T08:30:07.986 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-09T08:30:07.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd stat -f json 2024-05-09T08:30:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:30:08.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:07 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:08.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:07 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:08.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:07 smithi107 ceph-mon[30122]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:08.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:07 smithi107 ceph-mon[30122]: from='osd.11 [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:30:08.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:07 smithi107 ceph-mon[30122]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:30:08.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:07 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:08.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:07 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:08.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:07 smithi157 ceph-mon[33614]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:08.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:07 smithi157 ceph-mon[33614]: from='osd.11 [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:30:08.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:07 smithi157 ceph-mon[33614]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:30:08.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:07 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:08.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:07 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:08.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:07 smithi110 ceph-mon[33622]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:08.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:07 smithi110 ceph-mon[33622]: from='osd.11 [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:30:08.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:07 smithi110 ceph-mon[33622]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T08:30:09.063 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:08 smithi157 ceph-mon[33614]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-09T08:30:09.064 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:08 smithi157 ceph-mon[33614]: from='osd.11 [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:30:09.064 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:08 smithi157 ceph-mon[33614]: osdmap e78: 12 total, 11 up, 12 in 2024-05-09T08:30:09.064 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:08 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:30:09.064 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:08 smithi157 ceph-mon[33614]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:30:09.064 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:08 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:09.064 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:08 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:09.064 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:30:08 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:30:08.956+0000 7fa844b95700 -1 osd.11 0 waiting for initial osdmap 2024-05-09T08:30:09.065 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:30:08 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:30:08.970+0000 7fa840988700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T08:30:09.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:08 smithi107 ceph-mon[30122]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-09T08:30:09.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:08 smithi107 ceph-mon[30122]: from='osd.11 [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:30:09.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:08 smithi107 ceph-mon[30122]: osdmap e78: 12 total, 11 up, 12 in 2024-05-09T08:30:09.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:08 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:30:09.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:08 smithi107 ceph-mon[30122]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:30:09.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:08 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:09.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:08 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:08 smithi110 ceph-mon[33622]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-09T08:30:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:08 smithi110 ceph-mon[33622]: from='osd.11 [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:30:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:08 smithi110 ceph-mon[33622]: osdmap e78: 12 total, 11 up, 12 in 2024-05-09T08:30:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:08 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:30:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:08 smithi110 ceph-mon[33622]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-05-09T08:30:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:08 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:08 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:10.052 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:09 smithi107 ceph-mon[30122]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-05-09T08:30:10.052 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:09 smithi107 ceph-mon[30122]: osdmap e79: 12 total, 11 up, 12 in 2024-05-09T08:30:10.052 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:09 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:30:10.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:09 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:30:10.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:09 smithi107 ceph-mon[30122]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:10.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:09 smithi157 ceph-mon[33614]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-05-09T08:30:10.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:09 smithi157 ceph-mon[33614]: osdmap e79: 12 total, 11 up, 12 in 2024-05-09T08:30:10.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:09 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:30:10.095 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:09 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:30:10.095 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:09 smithi157 ceph-mon[33614]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:10.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.158+0000 7f8bcb27b700 1 -- 172.21.15.107:0/312329843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bc4105740 msgr2=0x7f8bc4107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:10.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.158+0000 7f8bcb27b700 1 --2- 172.21.15.107:0/312329843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bc4105740 0x7f8bc4107b30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009990 tx=0x7f8bb8009ca0 comp rx=0 tx=0).stop 2024-05-09T08:30:10.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.159+0000 7f8bcb27b700 1 -- 172.21.15.107:0/312329843 shutdown_connections 2024-05-09T08:30:10.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.159+0000 7f8bcb27b700 1 --2- 172.21.15.107:0/312329843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8bc4108070 0x7f8bc410a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:10.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.159+0000 7f8bcb27b700 1 --2- 172.21.15.107:0/312329843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bc4102e10 0x7f8bc4105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.159+0000 7f8bcb27b700 1 --2- 172.21.15.107:0/312329843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bc4105740 0x7f8bc4107b30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.159+0000 7f8bcb27b700 1 -- 172.21.15.107:0/312329843 >> 172.21.15.107:0/312329843 conn(0x7f8bc40fc840 msgr2=0x7f8bc40fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.159+0000 7f8bcb27b700 1 -- 172.21.15.107:0/312329843 shutdown_connections 2024-05-09T08:30:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.159+0000 7f8bcb27b700 1 -- 172.21.15.107:0/312329843 wait complete. 2024-05-09T08:30:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.159+0000 7f8bcb27b700 1 Processor -- start 2024-05-09T08:30:10.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.159+0000 7f8bcb27b700 1 -- start start 2024-05-09T08:30:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.159+0000 7f8bcb27b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bc4102e10 0x7f8bc41984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.159+0000 7f8bcb27b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8bc4105740 0x7f8bc4198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.159+0000 7f8bcb27b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bc4108070 0x7f8bc419fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.159+0000 7f8bcb27b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc410de40 con 0x7f8bc4108070 2024-05-09T08:30:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.159+0000 7f8bcb27b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc410dcc0 con 0x7f8bc4102e10 2024-05-09T08:30:10.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.159+0000 7f8bcb27b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc410dfc0 con 0x7f8bc4105740 2024-05-09T08:30:10.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.159+0000 7f8bc9818700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bc4108070 0x7f8bc419fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:10.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.160+0000 7f8bc8816700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8bc4105740 0x7f8bc4198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:10.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.160+0000 7f8bc9818700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bc4108070 0x7f8bc419fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45292/0 (socket says 172.21.15.107:45292) 2024-05-09T08:30:10.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.160+0000 7f8bc9818700 1 -- 172.21.15.107:0/2972811399 learned_addr learned my addr 172.21.15.107:0/2972811399 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:10.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.160+0000 7f8bc9017700 1 --2- 172.21.15.107:0/2972811399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bc4102e10 0x7f8bc41984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:10.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.160+0000 7f8bc9818700 1 -- 172.21.15.107:0/2972811399 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8bc4105740 msgr2=0x7f8bc4198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:10.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.160+0000 7f8bc9818700 1 --2- 172.21.15.107:0/2972811399 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8bc4105740 0x7f8bc4198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:10.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.160+0000 7f8bc9818700 1 -- 172.21.15.107:0/2972811399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bc4102e10 msgr2=0x7f8bc41984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:10.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.160+0000 7f8bc9818700 1 --2- 172.21.15.107:0/2972811399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bc4102e10 0x7f8bc41984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:10.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.160+0000 7f8bc9818700 1 -- 172.21.15.107:0/2972811399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc41a0130 con 0x7f8bc4108070 2024-05-09T08:30:10.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.160+0000 7f8bc9818700 1 --2- 172.21.15.107:0/2972811399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bc4108070 0x7f8bc419fa90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc00d770 tx=0x7f8bbc00da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:10.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.160+0000 7f8bc27fc700 1 -- 172.21.15.107:0/2972811399 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc014030 con 0x7f8bc4108070 2024-05-09T08:30:10.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.160+0000 7f8bcb27b700 1 -- 172.21.15.107:0/2972811399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bc41a0420 con 0x7f8bc4108070 2024-05-09T08:30:10.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.160+0000 7f8bc27fc700 1 -- 172.21.15.107:0/2972811399 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8bbc0044d0 con 0x7f8bc4108070 2024-05-09T08:30:10.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.160+0000 7f8bc27fc700 1 -- 172.21.15.107:0/2972811399 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc0038e0 con 0x7f8bc4108070 2024-05-09T08:30:10.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.160+0000 7f8bcb27b700 1 -- 172.21.15.107:0/2972811399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bc41a0960 con 0x7f8bc4108070 2024-05-09T08:30:10.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.161+0000 7f8bcb27b700 1 -- 172.21.15.107:0/2972811399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bc4069010 con 0x7f8bc4108070 2024-05-09T08:30:10.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.162+0000 7f8bc27fc700 1 -- 172.21.15.107:0/2972811399 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f8bbc003a80 con 0x7f8bc4108070 2024-05-09T08:30:10.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.163+0000 7f8bc27fc700 1 --2- 172.21.15.107:0/2972811399 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8bac0a2c20 0x7f8bac0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:10.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.163+0000 7f8bc9017700 1 --2- 172.21.15.107:0/2972811399 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8bac0a2c20 0x7f8bac0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:10.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.163+0000 7f8bc27fc700 1 -- 172.21.15.107:0/2972811399 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7127+0+0 (secure 0 0 0) 0x7f8bbc0ced00 con 0x7f8bc4108070 2024-05-09T08:30:10.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.163+0000 7f8bc9017700 1 --2- 172.21.15.107:0/2972811399 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8bac0a2c20 0x7f8bac0a50e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4009550 tx=0x7f8bb40091b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:10.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.167+0000 7f8bc27fc700 1 -- 172.21.15.107:0/2972811399 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8bbc09c100 con 0x7f8bc4108070 2024-05-09T08:30:10.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.363+0000 7f8bcb27b700 1 -- 172.21.15.107:0/2972811399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f8bc402dc20 con 0x7f8bc4108070 2024-05-09T08:30:10.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.363+0000 7f8bc27fc700 1 -- 172.21.15.107:0/2972811399 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f8bc402dc20 con 0x7f8bc4108070 2024-05-09T08:30:10.363 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:10.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.366+0000 7f8bcb27b700 1 -- 172.21.15.107:0/2972811399 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8bac0a2c20 msgr2=0x7f8bac0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:10.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.366+0000 7f8bcb27b700 1 --2- 172.21.15.107:0/2972811399 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8bac0a2c20 0x7f8bac0a50e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4009550 tx=0x7f8bb40091b0 comp rx=0 tx=0).stop 2024-05-09T08:30:10.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.366+0000 7f8bcb27b700 1 -- 172.21.15.107:0/2972811399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bc4108070 msgr2=0x7f8bc419fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:10.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.366+0000 7f8bcb27b700 1 --2- 172.21.15.107:0/2972811399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bc4108070 0x7f8bc419fa90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc00d770 tx=0x7f8bbc00da80 comp rx=0 tx=0).stop 2024-05-09T08:30:10.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.366+0000 7f8bcb27b700 1 -- 172.21.15.107:0/2972811399 shutdown_connections 2024-05-09T08:30:10.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.366+0000 7f8bcb27b700 1 --2- 172.21.15.107:0/2972811399 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8bac0a2c20 0x7f8bac0a50e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:10.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.366+0000 7f8bcb27b700 1 --2- 172.21.15.107:0/2972811399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bc4108070 0x7f8bc419fa90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:10.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.366+0000 7f8bcb27b700 1 --2- 172.21.15.107:0/2972811399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bc4102e10 0x7f8bc41984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:10.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.366+0000 7f8bcb27b700 1 --2- 172.21.15.107:0/2972811399 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8bc4105740 0x7f8bc4198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:10.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.366+0000 7f8bcb27b700 1 -- 172.21.15.107:0/2972811399 >> 172.21.15.107:0/2972811399 conn(0x7f8bc40fc840 msgr2=0x7f8bc4103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:10.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.366+0000 7f8bcb27b700 1 -- 172.21.15.107:0/2972811399 shutdown_connections 2024-05-09T08:30:10.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:10.366+0000 7f8bcb27b700 1 -- 172.21.15.107:0/2972811399 wait complete. 2024-05-09T08:30:10.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:09 smithi110 ceph-mon[33622]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-05-09T08:30:10.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:09 smithi110 ceph-mon[33622]: osdmap e79: 12 total, 11 up, 12 in 2024-05-09T08:30:10.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:09 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:30:10.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:09 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:30:10.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:09 smithi110 ceph-mon[33622]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-05-09T08:30:11.216 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:10 smithi107 ceph-mon[30122]: purged_snaps scrub starts 2024-05-09T08:30:11.216 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:10 smithi107 ceph-mon[30122]: purged_snaps scrub ok 2024-05-09T08:30:11.216 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:10 smithi107 ceph-mon[30122]: osd.11 [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474] boot 2024-05-09T08:30:11.216 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:10 smithi107 ceph-mon[30122]: osdmap e80: 12 total, 12 up, 12 in 2024-05-09T08:30:11.217 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:10 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:30:11.217 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:10 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2972811399' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-09T08:30:11.243 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1715243409,"num_in_osds":12,"osd_in_since":1715243389,"num_remapped_pgs":0} 2024-05-09T08:30:11.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-09T08:30:11.311 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:10 smithi157 ceph-mon[33614]: purged_snaps scrub starts 2024-05-09T08:30:11.311 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:10 smithi157 ceph-mon[33614]: purged_snaps scrub ok 2024-05-09T08:30:11.311 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:10 smithi157 ceph-mon[33614]: osd.11 [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474] boot 2024-05-09T08:30:11.311 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:10 smithi157 ceph-mon[33614]: osdmap e80: 12 total, 12 up, 12 in 2024-05-09T08:30:11.311 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:10 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:30:11.311 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:10 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2972811399' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-09T08:30:11.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:10 smithi110 ceph-mon[33622]: purged_snaps scrub starts 2024-05-09T08:30:11.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:10 smithi110 ceph-mon[33622]: purged_snaps scrub ok 2024-05-09T08:30:11.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:10 smithi110 ceph-mon[33622]: osd.11 [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474] boot 2024-05-09T08:30:11.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:10 smithi110 ceph-mon[33622]: osdmap e80: 12 total, 12 up, 12 in 2024-05-09T08:30:11.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:10 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:30:11.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:10 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2972811399' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-09T08:30:11.497 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:30:12.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:11 smithi107 ceph-mon[30122]: osdmap e81: 12 total, 12 up, 12 in 2024-05-09T08:30:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:11 smithi107 ceph-mon[30122]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:30:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:30:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:30:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:30:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:30:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:12.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:12.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:30:12.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:11 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:11 smithi157 ceph-mon[33614]: osdmap e81: 12 total, 12 up, 12 in 2024-05-09T08:30:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:11 smithi157 ceph-mon[33614]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:30:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:30:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:30:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:30:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:12.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:30:12.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:12.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:12.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:12.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:30:12.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:11 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:11 smithi110 ceph-mon[33622]: osdmap e81: 12 total, 12 up, 12 in 2024-05-09T08:30:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:11 smithi110 ceph-mon[33622]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:30:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:30:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:30:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:30:12.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:12.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:12.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:30:12.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:12.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:12.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:12.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:30:12.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:11 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.233+0000 7f2f6e4f4700 1 -- 172.21.15.107:0/3103544033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f68101510 msgr2=0x7f2f681018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.233+0000 7f2f6e4f4700 1 --2- 172.21.15.107:0/3103544033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f68101510 0x7f2f681018f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2f580099c0 tx=0x7f2f58009cd0 comp rx=0 tx=0).stop 2024-05-09T08:30:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.233+0000 7f2f6e4f4700 1 -- 172.21.15.107:0/3103544033 shutdown_connections 2024-05-09T08:30:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.233+0000 7f2f6e4f4700 1 --2- 172.21.15.107:0/3103544033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f68106560 0x7f2f6810e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.233+0000 7f2f6e4f4700 1 --2- 172.21.15.107:0/3103544033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f68101510 0x7f2f681018f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.233+0000 7f2f6e4f4700 1 --2- 172.21.15.107:0/3103544033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f68101ec0 0x7f2f68105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.233+0000 7f2f6e4f4700 1 -- 172.21.15.107:0/3103544033 >> 172.21.15.107:0/3103544033 conn(0x7f2f68077270 msgr2=0x7f2f68077680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:13.234 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:12 smithi107 ceph-mon[30122]: Detected new or changed devices on smithi157 2024-05-09T08:30:13.234 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:12 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi157 to 4272M 2024-05-09T08:30:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.233+0000 7f2f6e4f4700 1 -- 172.21.15.107:0/3103544033 shutdown_connections 2024-05-09T08:30:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.234+0000 7f2f6e4f4700 1 -- 172.21.15.107:0/3103544033 wait complete. 2024-05-09T08:30:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.234+0000 7f2f6e4f4700 1 Processor -- start 2024-05-09T08:30:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.234+0000 7f2f6e4f4700 1 -- start start 2024-05-09T08:30:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f6e4f4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f68101510 0x7f2f681977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f6e4f4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f68101ec0 0x7f2f68197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f6e4f4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f68106560 0x7f2f68198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f6e4f4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2f681114f0 con 0x7f2f68106560 2024-05-09T08:30:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f6e4f4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2f68111370 con 0x7f2f68101510 2024-05-09T08:30:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f6e4f4700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2f68111670 con 0x7f2f68101ec0 2024-05-09T08:30:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f6ca91700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f68106560 0x7f2f68198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f6ca91700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f68106560 0x7f2f68198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48402/0 (socket says 172.21.15.107:48402) 2024-05-09T08:30:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f6ca91700 1 -- 172.21.15.107:0/1529439112 learned_addr learned my addr 172.21.15.107:0/1529439112 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f677fe700 1 --2- 172.21.15.107:0/1529439112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f68101ec0 0x7f2f68197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f67fff700 1 --2- 172.21.15.107:0/1529439112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f68101510 0x7f2f681977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f6ca91700 1 -- 172.21.15.107:0/1529439112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f68101ec0 msgr2=0x7f2f68197d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f6ca91700 1 --2- 172.21.15.107:0/1529439112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f68101ec0 0x7f2f68197d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f6ca91700 1 -- 172.21.15.107:0/1529439112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f68101510 msgr2=0x7f2f681977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f6ca91700 1 --2- 172.21.15.107:0/1529439112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f68101510 0x7f2f681977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f6ca91700 1 -- 172.21.15.107:0/1529439112 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f681a7050 con 0x7f2f68106560 2024-05-09T08:30:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f6ca91700 1 --2- 172.21.15.107:0/1529439112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f68106560 0x7f2f68198260 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000b970 tx=0x7f2f6000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f657fa700 1 -- 172.21.15.107:0/1529439112 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6000d860 con 0x7f2f68106560 2024-05-09T08:30:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f6e4f4700 1 -- 172.21.15.107:0/1529439112 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f681a7340 con 0x7f2f68106560 2024-05-09T08:30:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.235+0000 7f2f657fa700 1 -- 172.21.15.107:0/1529439112 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2f6000e430 con 0x7f2f68106560 2024-05-09T08:30:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.236+0000 7f2f657fa700 1 -- 172.21.15.107:0/1529439112 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f60017660 con 0x7f2f68106560 2024-05-09T08:30:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.236+0000 7f2f6e4f4700 1 -- 172.21.15.107:0/1529439112 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f681a7880 con 0x7f2f68106560 2024-05-09T08:30:13.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.237+0000 7f2f6e4f4700 1 -- 172.21.15.107:0/1529439112 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f68069010 con 0x7f2f68106560 2024-05-09T08:30:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.238+0000 7f2f657fa700 1 -- 172.21.15.107:0/1529439112 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f2f6000e5e0 con 0x7f2f68106560 2024-05-09T08:30:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.239+0000 7f2f657fa700 1 --2- 172.21.15.107:0/1529439112 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2f540a2ca0 0x7f2f540a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.239+0000 7f2f67fff700 1 --2- 172.21.15.107:0/1529439112 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2f540a2ca0 0x7f2f540a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.239+0000 7f2f657fa700 1 -- 172.21.15.107:0/1529439112 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2f600ce750 con 0x7f2f68106560 2024-05-09T08:30:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.239+0000 7f2f67fff700 1 --2- 172.21.15.107:0/1529439112 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2f540a2ca0 0x7f2f540a5160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2f58000c00 tx=0x7f2f58005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.243+0000 7f2f657fa700 1 -- 172.21.15.107:0/1529439112 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2f60013030 con 0x7f2f68106560 2024-05-09T08:30:13.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:12 smithi157 ceph-mon[33614]: Detected new or changed devices on smithi157 2024-05-09T08:30:13.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:12 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi157 to 4272M 2024-05-09T08:30:13.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:12 smithi110 ceph-mon[33622]: Detected new or changed devices on smithi157 2024-05-09T08:30:13.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:12 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi157 to 4272M 2024-05-09T08:30:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.438+0000 7f2f6e4f4700 1 -- 172.21.15.107:0/1529439112 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f6802db80 con 0x7f2f68106560 2024-05-09T08:30:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.439+0000 7f2f657fa700 1 -- 172.21.15.107:0/1529439112 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18515 (secure 0 0 0) 0x7f2f6009bb50 con 0x7f2f68106560 2024-05-09T08:30:13.439 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:13.439 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":81,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:30:10.960298+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:30:13.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.441+0000 7f2f6e4f4700 1 -- 172.21.15.107:0/1529439112 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2f540a2ca0 msgr2=0x7f2f540a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:13.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.441+0000 7f2f6e4f4700 1 --2- 172.21.15.107:0/1529439112 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2f540a2ca0 0x7f2f540a5160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2f58000c00 tx=0x7f2f58005fb0 comp rx=0 tx=0).stop 2024-05-09T08:30:13.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.441+0000 7f2f6e4f4700 1 -- 172.21.15.107:0/1529439112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f68106560 msgr2=0x7f2f68198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:13.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.441+0000 7f2f6e4f4700 1 --2- 172.21.15.107:0/1529439112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f68106560 0x7f2f68198260 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000b970 tx=0x7f2f6000bc80 comp rx=0 tx=0).stop 2024-05-09T08:30:13.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.441+0000 7f2f6e4f4700 1 -- 172.21.15.107:0/1529439112 shutdown_connections 2024-05-09T08:30:13.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.441+0000 7f2f6e4f4700 1 --2- 172.21.15.107:0/1529439112 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2f540a2ca0 0x7f2f540a5160 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:13.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.441+0000 7f2f6e4f4700 1 --2- 172.21.15.107:0/1529439112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f68106560 0x7f2f68198260 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:13.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.441+0000 7f2f6e4f4700 1 --2- 172.21.15.107:0/1529439112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f68101510 0x7f2f681977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:13.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.441+0000 7f2f6e4f4700 1 --2- 172.21.15.107:0/1529439112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f68101ec0 0x7f2f68197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:13.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.441+0000 7f2f6e4f4700 1 -- 172.21.15.107:0/1529439112 >> 172.21.15.107:0/1529439112 conn(0x7f2f68077270 msgr2=0x7f2f680ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:13.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.441+0000 7f2f6e4f4700 1 -- 172.21.15.107:0/1529439112 shutdown_connections 2024-05-09T08:30:13.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:13.441+0000 7f2f6e4f4700 1 -- 172.21.15.107:0/1529439112 wait complete. 2024-05-09T08:30:14.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:13 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1529439112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:30:14.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:13 smithi107 ceph-mon[30122]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:14.139 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:26:21.698338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T08:30:14.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-09T08:30:14.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:13 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1529439112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:30:14.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:13 smithi157 ceph-mon[33614]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:14.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:13 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1529439112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:30:14.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:13 smithi110 ceph-mon[33622]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:14.380 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:30:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.121+0000 7f7e4cbc6700 1 -- 172.21.15.107:0/1310878497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e4806b410 msgr2=0x7f7e4810f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.121+0000 7f7e4cbc6700 1 --2- 172.21.15.107:0/1310878497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e4806b410 0x7f7e4810f1e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7e4000b0a0 tx=0x7f7e4000b3b0 comp rx=0 tx=0).stop 2024-05-09T08:30:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.121+0000 7f7e4cbc6700 1 -- 172.21.15.107:0/1310878497 shutdown_connections 2024-05-09T08:30:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.121+0000 7f7e4cbc6700 1 --2- 172.21.15.107:0/1310878497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e4806b410 0x7f7e4810f1e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.121+0000 7f7e4cbc6700 1 --2- 172.21.15.107:0/1310878497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e481083a0 0x7f7e48108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.121+0000 7f7e4cbc6700 1 --2- 172.21.15.107:0/1310878497 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e4806aa70 0x7f7e4806aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.121+0000 7f7e4cbc6700 1 -- 172.21.15.107:0/1310878497 >> 172.21.15.107:0/1310878497 conn(0x7f7e480781f0 msgr2=0x7f7e48078600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.121+0000 7f7e4cbc6700 1 -- 172.21.15.107:0/1310878497 shutdown_connections 2024-05-09T08:30:16.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.121+0000 7f7e4cbc6700 1 -- 172.21.15.107:0/1310878497 wait complete. 2024-05-09T08:30:16.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.122+0000 7f7e4cbc6700 1 Processor -- start 2024-05-09T08:30:16.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.122+0000 7f7e4cbc6700 1 -- start start 2024-05-09T08:30:16.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.122+0000 7f7e4cbc6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e4806aa70 0x7f7e48101320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.122+0000 7f7e4cbc6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e4806b410 0x7f7e48101860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.122+0000 7f7e4cbc6700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e481083a0 0x7f7e48101da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.122+0000 7f7e4cbc6700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7e48112550 con 0x7f7e4806b410 2024-05-09T08:30:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.122+0000 7f7e4cbc6700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7e481123d0 con 0x7f7e4806aa70 2024-05-09T08:30:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.122+0000 7f7e4cbc6700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7e481126d0 con 0x7f7e481083a0 2024-05-09T08:30:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.122+0000 7f7e45d9b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e4806b410 0x7f7e48101860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.122+0000 7f7e45d9b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e4806b410 0x7f7e48101860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48412/0 (socket says 172.21.15.107:48412) 2024-05-09T08:30:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.122+0000 7f7e45d9b700 1 -- 172.21.15.107:0/1755374470 learned_addr learned my addr 172.21.15.107:0/1755374470 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.122+0000 7f7e46d9d700 1 --2- 172.21.15.107:0/1755374470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e481083a0 0x7f7e48101da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:16.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.122+0000 7f7e4659c700 1 --2- 172.21.15.107:0/1755374470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e4806aa70 0x7f7e48101320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.123+0000 7f7e45d9b700 1 -- 172.21.15.107:0/1755374470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e481083a0 msgr2=0x7f7e48101da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.123+0000 7f7e45d9b700 1 --2- 172.21.15.107:0/1755374470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e481083a0 0x7f7e48101da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.123+0000 7f7e45d9b700 1 -- 172.21.15.107:0/1755374470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e4806aa70 msgr2=0x7f7e48101320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.123+0000 7f7e45d9b700 1 --2- 172.21.15.107:0/1755374470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e4806aa70 0x7f7e48101320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.123+0000 7f7e45d9b700 1 -- 172.21.15.107:0/1755374470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e481a6fb0 con 0x7f7e4806b410 2024-05-09T08:30:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.123+0000 7f7e45d9b700 1 --2- 172.21.15.107:0/1755374470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e4806b410 0x7f7e48101860 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c00b530 tx=0x7f7e3c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.123+0000 7f7e377fe700 1 -- 172.21.15.107:0/1755374470 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c015020 con 0x7f7e4806b410 2024-05-09T08:30:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.123+0000 7f7e4cbc6700 1 -- 172.21.15.107:0/1755374470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e481a72a0 con 0x7f7e4806b410 2024-05-09T08:30:16.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.123+0000 7f7e377fe700 1 -- 172.21.15.107:0/1755374470 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e3c011ce0 con 0x7f7e4806b410 2024-05-09T08:30:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.123+0000 7f7e377fe700 1 -- 172.21.15.107:0/1755374470 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c010d60 con 0x7f7e4806b410 2024-05-09T08:30:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.123+0000 7f7e4cbc6700 1 -- 172.21.15.107:0/1755374470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e481a77e0 con 0x7f7e4806b410 2024-05-09T08:30:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.124+0000 7f7e4cbc6700 1 -- 172.21.15.107:0/1755374470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e48069010 con 0x7f7e4806b410 2024-05-09T08:30:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.126+0000 7f7e377fe700 1 -- 172.21.15.107:0/1755374470 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f7e3c018020 con 0x7f7e4806b410 2024-05-09T08:30:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.126+0000 7f7e377fe700 1 --2- 172.21.15.107:0/1755374470 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7e300a2c20 0x7f7e300a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.127+0000 7f7e4659c700 1 --2- 172.21.15.107:0/1755374470 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7e300a2c20 0x7f7e300a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.127+0000 7f7e377fe700 1 -- 172.21.15.107:0/1755374470 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7e3c0ce9d0 con 0x7f7e4806b410 2024-05-09T08:30:16.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.127+0000 7f7e4659c700 1 --2- 172.21.15.107:0/1755374470 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7e300a2c20 0x7f7e300a50e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7e38005ea0 tx=0x7f7e38005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.131+0000 7f7e377fe700 1 -- 172.21.15.107:0/1755374470 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7e3c01f020 con 0x7f7e4806b410 2024-05-09T08:30:16.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.327+0000 7f7e4cbc6700 1 -- 172.21.15.107:0/1755374470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7e4802db80 con 0x7f7e4806b410 2024-05-09T08:30:16.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.328+0000 7f7e377fe700 1 -- 172.21.15.107:0/1755374470 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f7e3c09bdd0 con 0x7f7e4806b410 2024-05-09T08:30:16.327 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-05-09T08:30:16.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.330+0000 7f7e4cbc6700 1 -- 172.21.15.107:0/1755374470 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7e300a2c20 msgr2=0x7f7e300a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:16.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.330+0000 7f7e4cbc6700 1 --2- 172.21.15.107:0/1755374470 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7e300a2c20 0x7f7e300a50e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7e38005ea0 tx=0x7f7e38005e30 comp rx=0 tx=0).stop 2024-05-09T08:30:16.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.330+0000 7f7e4cbc6700 1 -- 172.21.15.107:0/1755374470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e4806b410 msgr2=0x7f7e48101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:16.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.330+0000 7f7e4cbc6700 1 --2- 172.21.15.107:0/1755374470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e4806b410 0x7f7e48101860 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c00b530 tx=0x7f7e3c00b8f0 comp rx=0 tx=0).stop 2024-05-09T08:30:16.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.330+0000 7f7e4cbc6700 1 -- 172.21.15.107:0/1755374470 shutdown_connections 2024-05-09T08:30:16.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.330+0000 7f7e4cbc6700 1 --2- 172.21.15.107:0/1755374470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e481083a0 0x7f7e48101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:16.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.330+0000 7f7e4cbc6700 1 --2- 172.21.15.107:0/1755374470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e4806aa70 0x7f7e48101320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:16.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.330+0000 7f7e4cbc6700 1 --2- 172.21.15.107:0/1755374470 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7e300a2c20 0x7f7e300a50e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:16.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.330+0000 7f7e4cbc6700 1 --2- 172.21.15.107:0/1755374470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e4806b410 0x7f7e48101860 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:16.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.330+0000 7f7e4cbc6700 1 -- 172.21.15.107:0/1755374470 >> 172.21.15.107:0/1755374470 conn(0x7f7e480781f0 msgr2=0x7f7e480ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:16.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.330+0000 7f7e4cbc6700 1 -- 172.21.15.107:0/1755374470 shutdown_connections 2024-05-09T08:30:16.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:16.330+0000 7f7e4cbc6700 1 -- 172.21.15.107:0/1755374470 wait complete. 2024-05-09T08:30:16.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:16 smithi107 ceph-mon[30122]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:16.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:16 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1755374470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:30:16.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:16 smithi157 ceph-mon[33614]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:16.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:16 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1755374470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:30:16.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:16 smithi110 ceph-mon[33622]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:16.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:16 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1755374470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:30:17.129 INFO:tasks.cephadm:Adding mds.a on smithi107 2024-05-09T08:30:17.129 INFO:tasks.cephadm:Adding mds.d on smithi107 2024-05-09T08:30:17.129 INFO:tasks.cephadm:Adding mds.g on smithi107 2024-05-09T08:30:17.130 INFO:tasks.cephadm:Adding mds.j on smithi107 2024-05-09T08:30:17.130 INFO:tasks.cephadm:Adding mds.b on smithi110 2024-05-09T08:30:17.130 INFO:tasks.cephadm:Adding mds.e on smithi110 2024-05-09T08:30:17.130 INFO:tasks.cephadm:Adding mds.h on smithi110 2024-05-09T08:30:17.130 INFO:tasks.cephadm:Adding mds.k on smithi110 2024-05-09T08:30:17.130 INFO:tasks.cephadm:Adding mds.c on smithi157 2024-05-09T08:30:17.130 INFO:tasks.cephadm:Adding mds.f on smithi157 2024-05-09T08:30:17.130 INFO:tasks.cephadm:Adding mds.i on smithi157 2024-05-09T08:30:17.130 INFO:tasks.cephadm:Adding mds.l on smithi157 2024-05-09T08:30:17.130 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph orch apply mds all '12;smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi157=c;smithi157=f;smithi157=i;smithi157=l' 2024-05-09T08:30:17.384 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:30:18.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:18 smithi107 ceph-mon[30122]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:18.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:18 smithi157 ceph-mon[33614]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:18.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:18 smithi110 ceph-mon[33622]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:19.113 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.110+0000 7ff79bedd700 1 -- 172.21.15.157:0/2298222019 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff794102480 msgr2=0x7ff7941028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:19.113 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.110+0000 7ff79bedd700 1 --2- 172.21.15.157:0/2298222019 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff794102480 0x7ff7941028e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff784009990 tx=0x7ff784009ca0 comp rx=0 tx=0).stop 2024-05-09T08:30:19.113 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.110+0000 7ff79bedd700 1 -- 172.21.15.157:0/2298222019 shutdown_connections 2024-05-09T08:30:19.113 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.110+0000 7ff79bedd700 1 --2- 172.21.15.157:0/2298222019 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff794102e20 0x7ff7941096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:19.113 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.110+0000 7ff79bedd700 1 --2- 172.21.15.157:0/2298222019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff794101280 0x7ff7941016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:19.113 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.110+0000 7ff79bedd700 1 --2- 172.21.15.157:0/2298222019 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff794102480 0x7ff7941028e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:19.113 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.110+0000 7ff79bedd700 1 -- 172.21.15.157:0/2298222019 >> 172.21.15.157:0/2298222019 conn(0x7ff7940fc820 msgr2=0x7ff7940fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:19.113 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.110+0000 7ff79bedd700 1 -- 172.21.15.157:0/2298222019 shutdown_connections 2024-05-09T08:30:19.113 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.110+0000 7ff79bedd700 1 -- 172.21.15.157:0/2298222019 wait complete. 2024-05-09T08:30:19.113 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.110+0000 7ff79bedd700 1 Processor -- start 2024-05-09T08:30:19.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.110+0000 7ff79bedd700 1 -- start start 2024-05-09T08:30:19.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff79bedd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff794101280 0x7ff7941973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:19.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff79bedd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff794102480 0x7ff7941978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:19.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff79bedd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff794102e20 0x7ff79419e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:19.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff79bedd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff79410cd10 con 0x7ff794102e20 2024-05-09T08:30:19.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff79bedd700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff79410cb90 con 0x7ff794101280 2024-05-09T08:30:19.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff79bedd700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff79410ce90 con 0x7ff794102480 2024-05-09T08:30:19.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff799478700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff794102480 0x7ff7941978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:19.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff799478700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff794102480 0x7ff7941978f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:60750/0 (socket says 172.21.15.157:60750) 2024-05-09T08:30:19.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff799478700 1 -- 172.21.15.157:0/1210623001 learned_addr learned my addr 172.21.15.157:0/1210623001 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-05-09T08:30:19.114 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff799c79700 1 --2- 172.21.15.157:0/1210623001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff794101280 0x7ff7941973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:19.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff79a47a700 1 --2- 172.21.15.157:0/1210623001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff794102e20 0x7ff79419e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:19.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff799478700 1 -- 172.21.15.157:0/1210623001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff794101280 msgr2=0x7ff7941973b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:19.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff799478700 1 --2- 172.21.15.157:0/1210623001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff794101280 0x7ff7941973b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:19.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff799478700 1 -- 172.21.15.157:0/1210623001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff794102e20 msgr2=0x7ff79419e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:19.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff799478700 1 --2- 172.21.15.157:0/1210623001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff794102e20 0x7ff79419e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:19.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff799478700 1 -- 172.21.15.157:0/1210623001 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff79406aad0 con 0x7ff794102480 2024-05-09T08:30:19.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff799478700 1 --2- 172.21.15.157:0/1210623001 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff794102480 0x7ff7941978f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff784004790 tx=0x7ff784009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:19.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff78affd700 1 -- 172.21.15.157:0/1210623001 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff784027070 con 0x7ff794102480 2024-05-09T08:30:19.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff79bedd700 1 -- 172.21.15.157:0/1210623001 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff79406ad60 con 0x7ff794102480 2024-05-09T08:30:19.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff78affd700 1 -- 172.21.15.157:0/1210623001 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff784004140 con 0x7ff794102480 2024-05-09T08:30:19.116 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.111+0000 7ff78affd700 1 -- 172.21.15.157:0/1210623001 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff784022d70 con 0x7ff794102480 2024-05-09T08:30:19.116 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.112+0000 7ff79bedd700 1 -- 172.21.15.157:0/1210623001 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff79406b240 con 0x7ff794102480 2024-05-09T08:30:19.116 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.112+0000 7ff79bedd700 1 -- 172.21.15.157:0/1210623001 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff794069010 con 0x7ff794102480 2024-05-09T08:30:19.117 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.114+0000 7ff78affd700 1 -- 172.21.15.157:0/1210623001 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7ff78402a070 con 0x7ff794102480 2024-05-09T08:30:19.117 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.114+0000 7ff78affd700 1 --2- 172.21.15.157:0/1210623001 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff7800a2c20 0x7ff7800a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:19.117 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.115+0000 7ff78affd700 1 -- 172.21.15.157:0/1210623001 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff7840d01f0 con 0x7ff794102480 2024-05-09T08:30:19.117 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.115+0000 7ff799c79700 1 --2- 172.21.15.157:0/1210623001 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff7800a2c20 0x7ff7800a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:19.117 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.115+0000 7ff799c79700 1 --2- 172.21.15.157:0/1210623001 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff7800a2c20 0x7ff7800a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff7941022e0 tx=0x7ff79000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:19.120 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.120+0000 7ff78affd700 1 -- 172.21.15.157:0/1210623001 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff784031070 con 0x7ff794102480 2024-05-09T08:30:19.334 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.334+0000 7ff79bedd700 1 -- 172.21.15.157:0/1210623001 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi157=c;smithi157=f;smithi157=i;smithi157=l", "target": ["mon-mgr", ""]}) -- 0x7ff79406b660 con 0x7ff7800a2c20 2024-05-09T08:30:19.342 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.342+0000 7ff78affd700 1 -- 172.21.15.157:0/1210623001 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7ff79406b660 con 0x7ff7800a2c20 2024-05-09T08:30:19.342 INFO:teuthology.orchestra.run.smithi157.stdout:Scheduled mds.all update... 2024-05-09T08:30:19.344 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.344+0000 7ff79bedd700 1 -- 172.21.15.157:0/1210623001 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff7800a2c20 msgr2=0x7ff7800a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:19.344 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.344+0000 7ff79bedd700 1 --2- 172.21.15.157:0/1210623001 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff7800a2c20 0x7ff7800a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff7941022e0 tx=0x7ff79000a380 comp rx=0 tx=0).stop 2024-05-09T08:30:19.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.344+0000 7ff79bedd700 1 -- 172.21.15.157:0/1210623001 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff794102480 msgr2=0x7ff7941978f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:19.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.344+0000 7ff79bedd700 1 --2- 172.21.15.157:0/1210623001 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff794102480 0x7ff7941978f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff784004790 tx=0x7ff784009580 comp rx=0 tx=0).stop 2024-05-09T08:30:19.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.344+0000 7ff79bedd700 1 -- 172.21.15.157:0/1210623001 shutdown_connections 2024-05-09T08:30:19.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.344+0000 7ff79bedd700 1 --2- 172.21.15.157:0/1210623001 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff7800a2c20 0x7ff7800a50e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:19.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.344+0000 7ff79bedd700 1 --2- 172.21.15.157:0/1210623001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff794102e20 0x7ff79419e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:19.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.344+0000 7ff79bedd700 1 --2- 172.21.15.157:0/1210623001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff794101280 0x7ff7941973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:19.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.344+0000 7ff79bedd700 1 --2- 172.21.15.157:0/1210623001 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff794102480 0x7ff7941978f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:19.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.344+0000 7ff79bedd700 1 -- 172.21.15.157:0/1210623001 >> 172.21.15.157:0/1210623001 conn(0x7ff7940fc820 msgr2=0x7ff794105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:19.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.344+0000 7ff79bedd700 1 -- 172.21.15.157:0/1210623001 shutdown_connections 2024-05-09T08:30:19.345 INFO:teuthology.orchestra.run.smithi157.stderr:2024-05-09T08:30:19.344+0000 7ff79bedd700 1 -- 172.21.15.157:0/1210623001 wait complete. 2024-05-09T08:30:20.045 DEBUG:teuthology.orchestra.run.smithi107:mds.a> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.a.service 2024-05-09T08:30:20.048 DEBUG:teuthology.orchestra.run.smithi107:mds.d> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.d.service 2024-05-09T08:30:20.091 DEBUG:teuthology.orchestra.run.smithi107:mds.g> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.g.service 2024-05-09T08:30:20.134 DEBUG:teuthology.orchestra.run.smithi107:mds.j> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.j.service 2024-05-09T08:30:20.177 DEBUG:teuthology.orchestra.run.smithi110:mds.b> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.b.service 2024-05-09T08:30:20.180 DEBUG:teuthology.orchestra.run.smithi110:mds.e> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.e.service 2024-05-09T08:30:20.223 DEBUG:teuthology.orchestra.run.smithi110:mds.h> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.h.service 2024-05-09T08:30:20.267 DEBUG:teuthology.orchestra.run.smithi110:mds.k> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.k.service 2024-05-09T08:30:20.311 DEBUG:teuthology.orchestra.run.smithi157:mds.c> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.c.service 2024-05-09T08:30:20.313 DEBUG:teuthology.orchestra.run.smithi157:mds.f> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.f.service 2024-05-09T08:30:20.356 DEBUG:teuthology.orchestra.run.smithi157:mds.i> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.i.service 2024-05-09T08:30:20.399 DEBUG:teuthology.orchestra.run.smithi157:mds.l> sudo journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.l.service 2024-05-09T08:30:20.442 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-05-09T08:30:20.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:30:20.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.562+0000 7fe7712df640 1 -- 172.21.15.107:0/3574300649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76c101960 msgr2=0x7fe76c101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:20.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.562+0000 7fe7712df640 1 --2- 172.21.15.107:0/3574300649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76c101960 0x7fe76c101de0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe760009900 tx=0x7fe76002a410 comp rx=0 tx=0).stop 2024-05-09T08:30:20.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.562+0000 7fe7712df640 1 -- 172.21.15.107:0/3574300649 shutdown_connections 2024-05-09T08:30:20.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.562+0000 7fe7712df640 1 --2- 172.21.15.107:0/3574300649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe76c102320 0x7fe76c108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.562+0000 7fe7712df640 1 --2- 172.21.15.107:0/3574300649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76c101960 0x7fe76c101de0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.562+0000 7fe7712df640 1 --2- 172.21.15.107:0/3574300649 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe76c100770 0x7fe76c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.562+0000 7fe7712df640 1 -- 172.21.15.107:0/3574300649 >> 172.21.15.107:0/3574300649 conn(0x7fe76c0fbce0 msgr2=0x7fe76c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.562+0000 7fe7712df640 1 -- 172.21.15.107:0/3574300649 shutdown_connections 2024-05-09T08:30:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.562+0000 7fe7712df640 1 -- 172.21.15.107:0/3574300649 wait complete. 2024-05-09T08:30:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.562+0000 7fe7712df640 1 Processor -- start 2024-05-09T08:30:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe7712df640 1 -- start start 2024-05-09T08:30:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe7712df640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe76c100770 0x7fe76c069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe7712df640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76c101960 0x7fe76c19c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe7712df640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe76c102320 0x7fe76c19ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe7712df640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe76c10c280 con 0x7fe76c101960 2024-05-09T08:30:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe7712df640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe76c10c100 con 0x7fe76c102320 2024-05-09T08:30:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe7712df640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe76c10c400 con 0x7fe76c100770 2024-05-09T08:30:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe76a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76c101960 0x7fe76c19c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe76a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76c101960 0x7fe76c19c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48420/0 (socket says 172.21.15.107:48420) 2024-05-09T08:30:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe76a7fc640 1 -- 172.21.15.107:0/1046125357 learned_addr learned my addr 172.21.15.107:0/1046125357 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe76affd640 1 --2- 172.21.15.107:0/1046125357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe76c100770 0x7fe76c069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe76b7fe640 1 --2- 172.21.15.107:0/1046125357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe76c102320 0x7fe76c19ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe76a7fc640 1 -- 172.21.15.107:0/1046125357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe76c100770 msgr2=0x7fe76c069530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe76a7fc640 1 --2- 172.21.15.107:0/1046125357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe76c100770 0x7fe76c069530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe76a7fc640 1 -- 172.21.15.107:0/1046125357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe76c102320 msgr2=0x7fe76c19ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe76a7fc640 1 --2- 172.21.15.107:0/1046125357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe76c102320 0x7fe76c19ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe76a7fc640 1 -- 172.21.15.107:0/1046125357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe76c19f740 con 0x7fe76c101960 2024-05-09T08:30:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe76b7fe640 1 --2- 172.21.15.107:0/1046125357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe76c102320 0x7fe76c19ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:30:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe76a7fc640 1 --2- 172.21.15.107:0/1046125357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76c101960 0x7fe76c19c810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe760004770 tx=0x7fe7600042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe753fff640 1 -- 172.21.15.107:0/1046125357 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76003f030 con 0x7fe76c101960 2024-05-09T08:30:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.563+0000 7fe7712df640 1 -- 172.21.15.107:0/1046125357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe76c19f970 con 0x7fe76c101960 2024-05-09T08:30:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.564+0000 7fe753fff640 1 -- 172.21.15.107:0/1046125357 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe760003cb0 con 0x7fe76c101960 2024-05-09T08:30:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.564+0000 7fe753fff640 1 -- 172.21.15.107:0/1046125357 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe760035070 con 0x7fe76c101960 2024-05-09T08:30:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.564+0000 7fe7712df640 1 -- 172.21.15.107:0/1046125357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe76c19feb0 con 0x7fe76c101960 2024-05-09T08:30:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.564+0000 7fe7712df640 1 -- 172.21.15.107:0/1046125357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe76c107bb0 con 0x7fe76c101960 2024-05-09T08:30:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.566+0000 7fe753fff640 1 -- 172.21.15.107:0/1046125357 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fe760042030 con 0x7fe76c101960 2024-05-09T08:30:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.566+0000 7fe753fff640 1 --2- 172.21.15.107:0/1046125357 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe7540a2b90 0x7fe7540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.566+0000 7fe76affd640 1 --2- 172.21.15.107:0/1046125357 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe7540a2b90 0x7fe7540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.566+0000 7fe753fff640 1 -- 172.21.15.107:0/1046125357 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe7600ef4e0 con 0x7fe76c101960 2024-05-09T08:30:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.567+0000 7fe76affd640 1 --2- 172.21.15.107:0/1046125357 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe7540a2b90 0x7fe7540a5050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe76c1017c0 tx=0x7fe75c009670 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.571+0000 7fe753fff640 1 -- 172.21.15.107:0/1046125357 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe7600bc8e0 con 0x7fe76c101960 2024-05-09T08:30:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:20 smithi157 ceph-mon[33614]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi157=c;smithi157=f;smithi157=i;smithi157=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:30:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:20 smithi157 ceph-mon[33614]: Saving service mds.all spec with placement smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi157=c;smithi157=f;smithi157=i;smithi157=l;count:12 2024-05-09T08:30:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:20 smithi157 ceph-mon[33614]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:30:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:20 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:20 smithi110 ceph-mon[33622]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi157=c;smithi157=f;smithi157=i;smithi157=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:30:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:20 smithi110 ceph-mon[33622]: Saving service mds.all spec with placement smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi157=c;smithi157=f;smithi157=i;smithi157=l;count:12 2024-05-09T08:30:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:20 smithi110 ceph-mon[33622]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:30:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:20.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:20 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:20.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.746+0000 7fe7712df640 1 -- 172.21.15.107:0/1046125357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe76c02d3c0 con 0x7fe76c101960 2024-05-09T08:30:20.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.747+0000 7fe753fff640 1 -- 172.21.15.107:0/1046125357 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18515 (secure 0 0 0) 0x7fe7600dae20 con 0x7fe76c101960 2024-05-09T08:30:20.746 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:20.747 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":81,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:30:10.960298+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:30:20.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.749+0000 7fe7712df640 1 -- 172.21.15.107:0/1046125357 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe7540a2b90 msgr2=0x7fe7540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:20.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.749+0000 7fe7712df640 1 --2- 172.21.15.107:0/1046125357 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe7540a2b90 0x7fe7540a5050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe76c1017c0 tx=0x7fe75c009670 comp rx=0 tx=0).stop 2024-05-09T08:30:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.749+0000 7fe7712df640 1 -- 172.21.15.107:0/1046125357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76c101960 msgr2=0x7fe76c19c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.749+0000 7fe7712df640 1 --2- 172.21.15.107:0/1046125357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76c101960 0x7fe76c19c810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe760004770 tx=0x7fe7600042f0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.749+0000 7fe7712df640 1 -- 172.21.15.107:0/1046125357 shutdown_connections 2024-05-09T08:30:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.749+0000 7fe7712df640 1 --2- 172.21.15.107:0/1046125357 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe7540a2b90 0x7fe7540a5050 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.749+0000 7fe7712df640 1 --2- 172.21.15.107:0/1046125357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe76c102320 0x7fe76c19ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.749+0000 7fe7712df640 1 --2- 172.21.15.107:0/1046125357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76c101960 0x7fe76c19c810 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.749+0000 7fe7712df640 1 --2- 172.21.15.107:0/1046125357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe76c100770 0x7fe76c069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.749+0000 7fe7712df640 1 -- 172.21.15.107:0/1046125357 >> 172.21.15.107:0/1046125357 conn(0x7fe76c0fbce0 msgr2=0x7fe76c104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.749+0000 7fe7712df640 1 -- 172.21.15.107:0/1046125357 shutdown_connections 2024-05-09T08:30:20.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.749+0000 7fe7712df640 1 -- 172.21.15.107:0/1046125357 wait complete. 2024-05-09T08:30:20.758 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:20 smithi107 ceph-mon[30122]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi157=c;smithi157=f;smithi157=i;smithi157=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:30:20.758 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:20 smithi107 ceph-mon[30122]: Saving service mds.all spec with placement smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi157=c;smithi157=f;smithi157=i;smithi157=l;count:12 2024-05-09T08:30:20.758 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:20.759 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:20.759 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:20 smithi107 ceph-mon[30122]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:20.759 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:20.759 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:30:20.759 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:20.759 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:20.759 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:20.759 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:20 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:20.760 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:26:21.698338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T08:30:20.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:30:20.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.866+0000 7f4bd6790640 1 -- 172.21.15.107:0/1886624044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0100100 msgr2=0x7f4bd0100520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:20.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.866+0000 7f4bd6790640 1 --2- 172.21.15.107:0/1886624044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0100100 0x7f4bd0100520 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009900 tx=0x7f4bc002a410 comp rx=0 tx=0).stop 2024-05-09T08:30:20.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.866+0000 7f4bd6790640 1 -- 172.21.15.107:0/1886624044 shutdown_connections 2024-05-09T08:30:20.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.866+0000 7f4bd6790640 1 --2- 172.21.15.107:0/1886624044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bd0101400 0x7f4bd0101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.866+0000 7f4bd6790640 1 --2- 172.21.15.107:0/1886624044 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4bd0100a60 0x7f4bd0100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.866+0000 7f4bd6790640 1 --2- 172.21.15.107:0/1886624044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0100100 0x7f4bd0100520 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.866+0000 7f4bd6790640 1 -- 172.21.15.107:0/1886624044 >> 172.21.15.107:0/1886624044 conn(0x7f4bd00fbd00 msgr2=0x7f4bd00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:20.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.866+0000 7f4bd6790640 1 -- 172.21.15.107:0/1886624044 shutdown_connections 2024-05-09T08:30:20.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.866+0000 7f4bd6790640 1 -- 172.21.15.107:0/1886624044 wait complete. 2024-05-09T08:30:20.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bd6790640 1 Processor -- start 2024-05-09T08:30:20.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bd6790640 1 -- start start 2024-05-09T08:30:20.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bd6790640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0100100 0x7f4bd0199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bd6790640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bd0100a60 0x7f4bd019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bd6790640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4bd0101400 0x7f4bd01a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bd6790640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd010f580 con 0x7f4bd0100100 2024-05-09T08:30:20.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bd6790640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd010f400 con 0x7f4bd0100a60 2024-05-09T08:30:20.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bd6790640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd010f700 con 0x7f4bd0101400 2024-05-09T08:30:20.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bcffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0100100 0x7f4bd0199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bcffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0100100 0x7f4bd0199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51322/0 (socket says 172.21.15.107:51322) 2024-05-09T08:30:20.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bcffff640 1 -- 172.21.15.107:0/2300949977 learned_addr learned my addr 172.21.15.107:0/2300949977 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:20.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bd4d2d640 1 --2- 172.21.15.107:0/2300949977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4bd0101400 0x7f4bd01a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bc7fff640 1 --2- 172.21.15.107:0/2300949977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bd0100a60 0x7f4bd019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bcffff640 1 -- 172.21.15.107:0/2300949977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4bd0101400 msgr2=0x7f4bd01a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bcffff640 1 --2- 172.21.15.107:0/2300949977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4bd0101400 0x7f4bd01a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bcffff640 1 -- 172.21.15.107:0/2300949977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bd0100a60 msgr2=0x7f4bd019a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bcffff640 1 --2- 172.21.15.107:0/2300949977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bd0100a60 0x7f4bd019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bcffff640 1 -- 172.21.15.107:0/2300949977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd01a1910 con 0x7f4bd0100100 2024-05-09T08:30:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bd4d2d640 1 --2- 172.21.15.107:0/2300949977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4bd0101400 0x7f4bd01a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:30:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.867+0000 7f4bcffff640 1 --2- 172.21.15.107:0/2300949977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0100100 0x7f4bd0199d00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0004770 tx=0x7f4bc00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.868+0000 7f4bcdffb640 1 -- 172.21.15.107:0/2300949977 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc003f030 con 0x7f4bd0100100 2024-05-09T08:30:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.868+0000 7f4bcdffb640 1 -- 172.21.15.107:0/2300949977 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bc0003cb0 con 0x7f4bd0100100 2024-05-09T08:30:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.868+0000 7f4bd6790640 1 -- 172.21.15.107:0/2300949977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd01a1ba0 con 0x7f4bd0100100 2024-05-09T08:30:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.868+0000 7f4bcdffb640 1 -- 172.21.15.107:0/2300949977 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0035070 con 0x7f4bd0100100 2024-05-09T08:30:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.868+0000 7f4bd6790640 1 -- 172.21.15.107:0/2300949977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd01a20e0 con 0x7f4bd0100100 2024-05-09T08:30:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.869+0000 7f4bd6790640 1 -- 172.21.15.107:0/2300949977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bd010b0e0 con 0x7f4bd0100100 2024-05-09T08:30:20.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.870+0000 7f4bcdffb640 1 -- 172.21.15.107:0/2300949977 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f4bc0042030 con 0x7f4bd0100100 2024-05-09T08:30:20.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.870+0000 7f4bcdffb640 1 --2- 172.21.15.107:0/2300949977 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4bbc0a2c10 0x7f4bbc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:20.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.870+0000 7f4bc7fff640 1 --2- 172.21.15.107:0/2300949977 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4bbc0a2c10 0x7f4bbc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:20.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.870+0000 7f4bcdffb640 1 -- 172.21.15.107:0/2300949977 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4bc00ef830 con 0x7f4bd0100100 2024-05-09T08:30:20.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.871+0000 7f4bc7fff640 1 --2- 172.21.15.107:0/2300949977 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4bbc0a2c10 0x7f4bbc0a50d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4bb80058c0 tx=0x7f4bb8005c60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:20.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:20.874+0000 7f4bcdffb640 1 -- 172.21.15.107:0/2300949977 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4bc00bcc30 con 0x7f4bd0100100 2024-05-09T08:30:21.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.051+0000 7f4bd6790640 1 -- 172.21.15.107:0/2300949977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4bd002d350 con 0x7f4bd0100100 2024-05-09T08:30:21.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.052+0000 7f4bcdffb640 1 -- 172.21.15.107:0/2300949977 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f4bc00bc530 con 0x7f4bd0100100 2024-05-09T08:30:21.052 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-05-09T08:30:21.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.054+0000 7f4bd6790640 1 -- 172.21.15.107:0/2300949977 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4bbc0a2c10 msgr2=0x7f4bbc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.054+0000 7f4bd6790640 1 --2- 172.21.15.107:0/2300949977 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4bbc0a2c10 0x7f4bbc0a50d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4bb80058c0 tx=0x7f4bb8005c60 comp rx=0 tx=0).stop 2024-05-09T08:30:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.054+0000 7f4bd6790640 1 -- 172.21.15.107:0/2300949977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0100100 msgr2=0x7f4bd0199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.054+0000 7f4bd6790640 1 --2- 172.21.15.107:0/2300949977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0100100 0x7f4bd0199d00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0004770 tx=0x7f4bc00042f0 comp rx=0 tx=0).stop 2024-05-09T08:30:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.054+0000 7f4bd6790640 1 -- 172.21.15.107:0/2300949977 shutdown_connections 2024-05-09T08:30:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.054+0000 7f4bd6790640 1 --2- 172.21.15.107:0/2300949977 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4bbc0a2c10 0x7f4bbc0a50d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.054+0000 7f4bd6790640 1 --2- 172.21.15.107:0/2300949977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4bd0101400 0x7f4bd01a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.054+0000 7f4bd6790640 1 --2- 172.21.15.107:0/2300949977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bd0100a60 0x7f4bd019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.054+0000 7f4bd6790640 1 --2- 172.21.15.107:0/2300949977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0100100 0x7f4bd0199d00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.054+0000 7f4bd6790640 1 -- 172.21.15.107:0/2300949977 >> 172.21.15.107:0/2300949977 conn(0x7f4bd00fbd00 msgr2=0x7f4bd00fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.054+0000 7f4bd6790640 1 -- 172.21.15.107:0/2300949977 shutdown_connections 2024-05-09T08:30:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.054+0000 7f4bd6790640 1 -- 172.21.15.107:0/2300949977 wait complete. 2024-05-09T08:30:21.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:30:21.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.166+0000 7f0ec3942640 1 -- 172.21.15.107:0/3727021699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ebc100790 msgr2=0x7f0ebc100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:21.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.166+0000 7f0ec3942640 1 --2- 172.21.15.107:0/3727021699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ebc100790 0x7f0ebc100bb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009900 tx=0x7f0eb002a410 comp rx=0 tx=0).stop 2024-05-09T08:30:21.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.166+0000 7f0ec3942640 1 -- 172.21.15.107:0/3727021699 shutdown_connections 2024-05-09T08:30:21.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.166+0000 7f0ec3942640 1 --2- 172.21.15.107:0/3727021699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ebc102340 0x7f0ebc108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:21.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.166+0000 7f0ec3942640 1 --2- 172.21.15.107:0/3727021699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ebc101980 0x7f0ebc101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:21.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.166+0000 7f0ec3942640 1 --2- 172.21.15.107:0/3727021699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ebc100790 0x7f0ebc100bb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:21.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.166+0000 7f0ec3942640 1 -- 172.21.15.107:0/3727021699 >> 172.21.15.107:0/3727021699 conn(0x7f0ebc0fbd20 msgr2=0x7f0ebc0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:21.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.166+0000 7f0ec3942640 1 -- 172.21.15.107:0/3727021699 shutdown_connections 2024-05-09T08:30:21.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.166+0000 7f0ec3942640 1 -- 172.21.15.107:0/3727021699 wait complete. 2024-05-09T08:30:21.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.166+0000 7f0ec3942640 1 Processor -- start 2024-05-09T08:30:21.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec3942640 1 -- start start 2024-05-09T08:30:21.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec3942640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ebc100790 0x7f0ebc197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:21.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec3942640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ebc101980 0x7f0ebc198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:21.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec3942640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ebc102340 0x7f0ebc19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:21.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec3942640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc10d3e0 con 0x7f0ebc101980 2024-05-09T08:30:21.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec3942640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc10d260 con 0x7f0ebc102340 2024-05-09T08:30:21.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec3942640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc10d560 con 0x7f0ebc100790 2024-05-09T08:30:21.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec0edd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ebc101980 0x7f0ebc198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:21.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec0edd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ebc101980 0x7f0ebc198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51342/0 (socket says 172.21.15.107:51342) 2024-05-09T08:30:21.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec0edd640 1 -- 172.21.15.107:0/3459789419 learned_addr learned my addr 172.21.15.107:0/3459789419 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:21.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec1edf640 1 --2- 172.21.15.107:0/3459789419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ebc102340 0x7f0ebc19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:21.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec16de640 1 --2- 172.21.15.107:0/3459789419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ebc100790 0x7f0ebc197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:21.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec0edd640 1 -- 172.21.15.107:0/3459789419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ebc100790 msgr2=0x7f0ebc197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:21.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec0edd640 1 --2- 172.21.15.107:0/3459789419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ebc100790 0x7f0ebc197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:21.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec0edd640 1 -- 172.21.15.107:0/3459789419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ebc102340 msgr2=0x7f0ebc19f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:21.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec0edd640 1 --2- 172.21.15.107:0/3459789419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ebc102340 0x7f0ebc19f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:21.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec0edd640 1 -- 172.21.15.107:0/3459789419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ebc19f760 con 0x7f0ebc101980 2024-05-09T08:30:21.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec16de640 1 --2- 172.21.15.107:0/3459789419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ebc100790 0x7f0ebc197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:30:21.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec0edd640 1 --2- 172.21.15.107:0/3459789419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ebc101980 0x7f0ebc198080 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800c970 tx=0x7f0eb800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:21.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0eae7fc640 1 -- 172.21.15.107:0/3459789419 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8007bb0 con 0x7f0ebc101980 2024-05-09T08:30:21.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0eae7fc640 1 -- 172.21.15.107:0/3459789419 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0eb8007d50 con 0x7f0ebc101980 2024-05-09T08:30:21.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0ec3942640 1 -- 172.21.15.107:0/3459789419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ebc19fa50 con 0x7f0ebc101980 2024-05-09T08:30:21.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.167+0000 7f0eae7fc640 1 -- 172.21.15.107:0/3459789419 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb800f450 con 0x7f0ebc101980 2024-05-09T08:30:21.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.168+0000 7f0ec3942640 1 -- 172.21.15.107:0/3459789419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ebc19ff50 con 0x7f0ebc101980 2024-05-09T08:30:21.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.168+0000 7f0ec3942640 1 -- 172.21.15.107:0/3459789419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ebc107b80 con 0x7f0ebc101980 2024-05-09T08:30:21.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.169+0000 7f0eae7fc640 1 -- 172.21.15.107:0/3459789419 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0eb8004f20 con 0x7f0ebc101980 2024-05-09T08:30:21.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.170+0000 7f0eae7fc640 1 --2- 172.21.15.107:0/3459789419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0ea80a2b90 0x7f0ea80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:21.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.170+0000 7f0ec16de640 1 --2- 172.21.15.107:0/3459789419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0ea80a2b90 0x7f0ea80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:21.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.170+0000 7f0eae7fc640 1 -- 172.21.15.107:0/3459789419 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0eb80ce9d0 con 0x7f0ebc101980 2024-05-09T08:30:21.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.170+0000 7f0ec16de640 1 --2- 172.21.15.107:0/3459789419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0ea80a2b90 0x7f0ea80a5050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0006010 tx=0x7f0eb0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:21.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.174+0000 7f0eae7fc640 1 -- 172.21.15.107:0/3459789419 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0eb8014030 con 0x7f0ebc101980 2024-05-09T08:30:21.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.388+0000 7f0ec3942640 1 -- 172.21.15.107:0/3459789419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ebc02d3a0 con 0x7f0ebc101980 2024-05-09T08:30:21.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.389+0000 7f0eae7fc640 1 -- 172.21.15.107:0/3459789419 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f0eb809bdd0 con 0x7f0ebc101980 2024-05-09T08:30:21.388 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:21.389 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-05-09T08:30:21.389 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 1 2024-05-09T08:30:21.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.391+0000 7f0ec3942640 1 -- 172.21.15.107:0/3459789419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0ea80a2b90 msgr2=0x7f0ea80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:21.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.391+0000 7f0ec3942640 1 --2- 172.21.15.107:0/3459789419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0ea80a2b90 0x7f0ea80a5050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0006010 tx=0x7f0eb0000bc0 comp rx=0 tx=0).stop 2024-05-09T08:30:21.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.391+0000 7f0ec3942640 1 -- 172.21.15.107:0/3459789419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ebc101980 msgr2=0x7f0ebc198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:21.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.391+0000 7f0ec3942640 1 --2- 172.21.15.107:0/3459789419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ebc101980 0x7f0ebc198080 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800c970 tx=0x7f0eb800ce30 comp rx=0 tx=0).stop 2024-05-09T08:30:21.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.391+0000 7f0ec3942640 1 -- 172.21.15.107:0/3459789419 shutdown_connections 2024-05-09T08:30:21.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.391+0000 7f0ec3942640 1 --2- 172.21.15.107:0/3459789419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0ea80a2b90 0x7f0ea80a5050 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:21.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.391+0000 7f0ec3942640 1 --2- 172.21.15.107:0/3459789419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ebc102340 0x7f0ebc19f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:21.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.391+0000 7f0ec3942640 1 --2- 172.21.15.107:0/3459789419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ebc101980 0x7f0ebc198080 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:21.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.391+0000 7f0ec3942640 1 --2- 172.21.15.107:0/3459789419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ebc100790 0x7f0ebc197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:21.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.391+0000 7f0ec3942640 1 -- 172.21.15.107:0/3459789419 >> 172.21.15.107:0/3459789419 conn(0x7f0ebc0fbd20 msgr2=0x7f0ebc104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:21.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.391+0000 7f0ec3942640 1 -- 172.21.15.107:0/3459789419 shutdown_connections 2024-05-09T08:30:21.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:21.391+0000 7f0ec3942640 1 -- 172.21.15.107:0/3459789419 wait complete. 2024-05-09T08:30:21.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:21 smithi157 ceph-mon[33614]: Deploying daemon mds.l on smithi157 2024-05-09T08:30:21.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:21 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1046125357' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:30:21.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:21 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2300949977' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:30:21.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:21 smithi110 ceph-mon[33622]: Deploying daemon mds.l on smithi157 2024-05-09T08:30:21.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:21 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1046125357' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:30:21.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:21 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2300949977' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:30:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:21 smithi107 ceph-mon[30122]: Deploying daemon mds.l on smithi157 2024-05-09T08:30:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:21 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1046125357' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:30:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:21 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2300949977' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:30:22.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:22 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3459789419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:22.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:22 smithi157 ceph-mon[33614]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:22.594 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:30:22 smithi157 systemd[1]: Starting Ceph mds.l for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:30:22.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:22 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3459789419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:22.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:22 smithi110 ceph-mon[33622]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:22.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:22 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3459789419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:22.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:22 smithi107 ceph-mon[30122]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:22.968 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:30:22 smithi157 podman[63400]: 2024-05-09 08:30:22.603754902 +0000 UTC m=+0.018202231 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:30:22.968 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:30:22 smithi157 podman[63400]: 2024-05-09 08:30:22.70740881 +0000 UTC m=+0.121856172 container create c7aa358e7a80e8fd0b4dca39033facfa2c77813bffd1aacc9f54605f7eb8e587 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-l, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, RELEASE=squid-96805dc) 2024-05-09T08:30:22.968 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:30:22 smithi157 podman[63400]: 2024-05-09 08:30:22.969210033 +0000 UTC m=+0.383657356 container init c7aa358e7a80e8fd0b4dca39033facfa2c77813bffd1aacc9f54605f7eb8e587 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-l, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-09T08:30:23.344 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:30:22 smithi157 podman[63400]: 2024-05-09 08:30:22.972028125 +0000 UTC m=+0.386475444 container start c7aa358e7a80e8fd0b4dca39033facfa2c77813bffd1aacc9f54605f7eb8e587 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-l, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-09T08:30:23.344 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:30:22 smithi157 ceph-mds[63419]: -- 172.21.15.157:0/4074258563 <== mon.1 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55927ce3da40 con 0x55927ce11000 2024-05-09T08:30:23.344 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:30:22 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-l[63415]: starting mds.l at 2024-05-09T08:30:23.344 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:30:23 smithi157 bash[63400]: c7aa358e7a80e8fd0b4dca39033facfa2c77813bffd1aacc9f54605f7eb8e587 2024-05-09T08:30:23.344 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:30:23 smithi157 systemd[1]: Started Ceph mds.l for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:30:23.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:30:23.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.518+0000 7fb9aad90640 1 -- 172.21.15.107:0/4088905555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9a4100770 msgr2=0x7fb9a4100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:23.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.518+0000 7fb9aad90640 1 --2- 172.21.15.107:0/4088905555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9a4100770 0x7fb9a4100b90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb998009900 tx=0x7fb99802a410 comp rx=0 tx=0).stop 2024-05-09T08:30:23.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.518+0000 7fb9aad90640 1 -- 172.21.15.107:0/4088905555 shutdown_connections 2024-05-09T08:30:23.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.518+0000 7fb9aad90640 1 --2- 172.21.15.107:0/4088905555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9a4102320 0x7fb9a4108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.518+0000 7fb9aad90640 1 --2- 172.21.15.107:0/4088905555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9a4101960 0x7fb9a4101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.518+0000 7fb9aad90640 1 --2- 172.21.15.107:0/4088905555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9a4100770 0x7fb9a4100b90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.518+0000 7fb9aad90640 1 -- 172.21.15.107:0/4088905555 >> 172.21.15.107:0/4088905555 conn(0x7fb9a40fbce0 msgr2=0x7fb9a40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:23.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.518+0000 7fb9aad90640 1 -- 172.21.15.107:0/4088905555 shutdown_connections 2024-05-09T08:30:23.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.518+0000 7fb9aad90640 1 -- 172.21.15.107:0/4088905555 wait complete. 2024-05-09T08:30:23.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.518+0000 7fb9aad90640 1 Processor -- start 2024-05-09T08:30:23.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9aad90640 1 -- start start 2024-05-09T08:30:23.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9aad90640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9a4100770 0x7fb9a4104700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:23.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9aad90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9a4101960 0x7fb9a4102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:23.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9aad90640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9a4102320 0x7fb9a4103280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:23.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9aad90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a410c270 con 0x7fb9a4101960 2024-05-09T08:30:23.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9aad90640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a410c0f0 con 0x7fb9a4102320 2024-05-09T08:30:23.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9aad90640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a410c3f0 con 0x7fb9a4100770 2024-05-09T08:30:23.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9a3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9a4101960 0x7fb9a4102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:23.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9a3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9a4101960 0x7fb9a4102d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51358/0 (socket says 172.21.15.107:51358) 2024-05-09T08:30:23.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9a3fff640 1 -- 172.21.15.107:0/4248411652 learned_addr learned my addr 172.21.15.107:0/4248411652 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:23.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9a8b2c640 1 --2- 172.21.15.107:0/4248411652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9a4100770 0x7fb9a4104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:23.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9a932d640 1 --2- 172.21.15.107:0/4248411652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9a4102320 0x7fb9a4103280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:23.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9a3fff640 1 -- 172.21.15.107:0/4248411652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9a4100770 msgr2=0x7fb9a4104700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:23.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9a3fff640 1 --2- 172.21.15.107:0/4248411652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9a4100770 0x7fb9a4104700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9a3fff640 1 -- 172.21.15.107:0/4248411652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9a4102320 msgr2=0x7fb9a4103280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:23.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9a3fff640 1 --2- 172.21.15.107:0/4248411652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9a4102320 0x7fb9a4103280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9a3fff640 1 -- 172.21.15.107:0/4248411652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a4103b10 con 0x7fb9a4101960 2024-05-09T08:30:23.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9a3fff640 1 --2- 172.21.15.107:0/4248411652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9a4101960 0x7fb9a4102d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb99000d950 tx=0x7fb99000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:23.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9a1ffb640 1 -- 172.21.15.107:0/4248411652 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb990017070 con 0x7fb9a4101960 2024-05-09T08:30:23.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9a1ffb640 1 -- 172.21.15.107:0/4248411652 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb990009c30 con 0x7fb9a4101960 2024-05-09T08:30:23.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.519+0000 7fb9aad90640 1 -- 172.21.15.107:0/4248411652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a41a1c40 con 0x7fb9a4101960 2024-05-09T08:30:23.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.520+0000 7fb9a1ffb640 1 -- 172.21.15.107:0/4248411652 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99000f620 con 0x7fb9a4101960 2024-05-09T08:30:23.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.520+0000 7fb9aad90640 1 -- 172.21.15.107:0/4248411652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a41a2190 con 0x7fb9a4101960 2024-05-09T08:30:23.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.521+0000 7fb9aad90640 1 -- 172.21.15.107:0/4248411652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9a41a2360 con 0x7fb9a4101960 2024-05-09T08:30:23.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.522+0000 7fb9a1ffb640 1 -- 172.21.15.107:0/4248411652 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fb9900109a0 con 0x7fb9a4101960 2024-05-09T08:30:23.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.523+0000 7fb9a1ffb640 1 --2- 172.21.15.107:0/4248411652 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb9940a2b90 0x7fb9940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:23.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.523+0000 7fb9a8b2c640 1 --2- 172.21.15.107:0/4248411652 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb9940a2b90 0x7fb9940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:23.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.523+0000 7fb9a1ffb640 1 -- 172.21.15.107:0/4248411652 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb9900ce6c0 con 0x7fb9a4101960 2024-05-09T08:30:23.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.523+0000 7fb9a8b2c640 1 --2- 172.21.15.107:0/4248411652 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb9940a2b90 0x7fb9940a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb998004770 tx=0x7fb998000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:23.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.526+0000 7fb9a1ffb640 1 -- 172.21.15.107:0/4248411652 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb990009de0 con 0x7fb9a4101960 2024-05-09T08:30:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.745+0000 7fb9aad90640 1 -- 172.21.15.107:0/4248411652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9a41a2650 con 0x7fb9a4101960 2024-05-09T08:30:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.745+0000 7fb9a1ffb640 1 -- 172.21.15.107:0/4248411652 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fb99009b470 con 0x7fb9a4101960 2024-05-09T08:30:23.745 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:23.745 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-09T08:30:23.745 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 2 2024-05-09T08:30:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.747+0000 7fb9aad90640 1 -- 172.21.15.107:0/4248411652 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb9940a2b90 msgr2=0x7fb9940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.747+0000 7fb9aad90640 1 --2- 172.21.15.107:0/4248411652 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb9940a2b90 0x7fb9940a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb998004770 tx=0x7fb998000bc0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.747+0000 7fb9aad90640 1 -- 172.21.15.107:0/4248411652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9a4101960 msgr2=0x7fb9a4102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.747+0000 7fb9aad90640 1 --2- 172.21.15.107:0/4248411652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9a4101960 0x7fb9a4102d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb99000d950 tx=0x7fb99000de10 comp rx=0 tx=0).stop 2024-05-09T08:30:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.747+0000 7fb9aad90640 1 -- 172.21.15.107:0/4248411652 shutdown_connections 2024-05-09T08:30:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.747+0000 7fb9aad90640 1 --2- 172.21.15.107:0/4248411652 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb9940a2b90 0x7fb9940a5050 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.747+0000 7fb9aad90640 1 --2- 172.21.15.107:0/4248411652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9a4102320 0x7fb9a4103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.747+0000 7fb9aad90640 1 --2- 172.21.15.107:0/4248411652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9a4101960 0x7fb9a4102d40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.747+0000 7fb9aad90640 1 --2- 172.21.15.107:0/4248411652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9a4100770 0x7fb9a4104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.747+0000 7fb9aad90640 1 -- 172.21.15.107:0/4248411652 >> 172.21.15.107:0/4248411652 conn(0x7fb9a40fbce0 msgr2=0x7fb9a4068a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.747+0000 7fb9aad90640 1 -- 172.21.15.107:0/4248411652 shutdown_connections 2024-05-09T08:30:23.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:23.747+0000 7fb9aad90640 1 -- 172.21.15.107:0/4248411652 wait complete. 2024-05-09T08:30:24.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:24 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.157:6832/2435727962,v1:172.21.15.157:6833/2435727962] up:boot 2024-05-09T08:30:24.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:24 smithi107 ceph-mon[30122]: fsmap 1 up:standby 2024-05-09T08:30:24.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:24 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:30:24.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:24 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:24.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:24 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:24.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:24 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:24.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:24 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:24.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:24 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:24.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:24 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:24.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:24 smithi107 ceph-mon[30122]: Deploying daemon mds.k on smithi110 2024-05-09T08:30:24.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:24 smithi107 ceph-mon[30122]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:24.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:24 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/4248411652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:24.078 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:24 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.157:6832/2435727962,v1:172.21.15.157:6833/2435727962] up:boot 2024-05-09T08:30:24.079 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:24 smithi110 ceph-mon[33622]: fsmap 1 up:standby 2024-05-09T08:30:24.079 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:24 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:30:24.079 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:24 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:24.079 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:24 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:24.079 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:24 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:24.079 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:24 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:24.079 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:24 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:24.079 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:24 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:24.079 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:24 smithi110 ceph-mon[33622]: Deploying daemon mds.k on smithi110 2024-05-09T08:30:24.079 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:24 smithi110 ceph-mon[33622]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:24.079 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:24 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/4248411652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:24 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.157:6832/2435727962,v1:172.21.15.157:6833/2435727962] up:boot 2024-05-09T08:30:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:24 smithi157 ceph-mon[33614]: fsmap 1 up:standby 2024-05-09T08:30:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:24 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:30:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:24 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:24 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:24 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:24 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:24 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:24 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:24.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:24 smithi157 ceph-mon[33614]: Deploying daemon mds.k on smithi110 2024-05-09T08:30:24.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:24 smithi157 ceph-mon[33614]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:24.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:24 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/4248411652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:25.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:30:25.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.872+0000 7ff0d85e7640 1 -- 172.21.15.107:0/1576854353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0d0100770 msgr2=0x7ff0d0100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:25.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.872+0000 7ff0d85e7640 1 --2- 172.21.15.107:0/1576854353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0d0100770 0x7ff0d0100b90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009900 tx=0x7ff0cc02a410 comp rx=0 tx=0).stop 2024-05-09T08:30:25.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.872+0000 7ff0d85e7640 1 -- 172.21.15.107:0/1576854353 shutdown_connections 2024-05-09T08:30:25.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.872+0000 7ff0d85e7640 1 --2- 172.21.15.107:0/1576854353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0d0102320 0x7ff0d0108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:25.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.872+0000 7ff0d85e7640 1 --2- 172.21.15.107:0/1576854353 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff0d0101960 0x7ff0d0101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:25.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.872+0000 7ff0d85e7640 1 --2- 172.21.15.107:0/1576854353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0d0100770 0x7ff0d0100b90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:25.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.872+0000 7ff0d85e7640 1 -- 172.21.15.107:0/1576854353 >> 172.21.15.107:0/1576854353 conn(0x7ff0d00fbce0 msgr2=0x7ff0d00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:25.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.872+0000 7ff0d85e7640 1 -- 172.21.15.107:0/1576854353 shutdown_connections 2024-05-09T08:30:25.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.872+0000 7ff0d85e7640 1 -- 172.21.15.107:0/1576854353 wait complete. 2024-05-09T08:30:25.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d85e7640 1 Processor -- start 2024-05-09T08:30:25.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d85e7640 1 -- start start 2024-05-09T08:30:25.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d85e7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff0d0100770 0x7ff0d0199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:25.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d85e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0d0101960 0x7ff0d019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:25.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d85e7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0d0102320 0x7ff0d01a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:25.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d85e7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d010c270 con 0x7ff0d0101960 2024-05-09T08:30:25.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d85e7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d010c0f0 con 0x7ff0d0102320 2024-05-09T08:30:25.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d85e7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d010c3f0 con 0x7ff0d0100770 2024-05-09T08:30:25.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d5b82640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0d0101960 0x7ff0d019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:25.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d5b82640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0d0101960 0x7ff0d019a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51376/0 (socket says 172.21.15.107:51376) 2024-05-09T08:30:25.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d5b82640 1 -- 172.21.15.107:0/2022918905 learned_addr learned my addr 172.21.15.107:0/2022918905 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:25.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d6b84640 1 --2- 172.21.15.107:0/2022918905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0d0102320 0x7ff0d01a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:25.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d6383640 1 --2- 172.21.15.107:0/2022918905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff0d0100770 0x7ff0d0199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:25.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d5b82640 1 -- 172.21.15.107:0/2022918905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff0d0100770 msgr2=0x7ff0d0199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:25.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d5b82640 1 --2- 172.21.15.107:0/2022918905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff0d0100770 0x7ff0d0199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:25.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d5b82640 1 -- 172.21.15.107:0/2022918905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0d0102320 msgr2=0x7ff0d01a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:25.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d5b82640 1 --2- 172.21.15.107:0/2022918905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0d0102320 0x7ff0d01a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:25.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d5b82640 1 -- 172.21.15.107:0/2022918905 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d01a1870 con 0x7ff0d0101960 2024-05-09T08:30:25.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d6383640 1 --2- 172.21.15.107:0/2022918905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff0d0100770 0x7ff0d0199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:30:25.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.873+0000 7ff0d6b84640 1 --2- 172.21.15.107:0/2022918905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0d0102320 0x7ff0d01a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:30:25.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.874+0000 7ff0d5b82640 1 --2- 172.21.15.107:0/2022918905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0d0101960 0x7ff0d019a1a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff0c000c920 tx=0x7ff0c000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:25.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.874+0000 7ff0c77fe640 1 -- 172.21.15.107:0/2022918905 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0007c70 con 0x7ff0d0101960 2024-05-09T08:30:25.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.874+0000 7ff0c77fe640 1 -- 172.21.15.107:0/2022918905 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0c0007e10 con 0x7ff0d0101960 2024-05-09T08:30:25.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.874+0000 7ff0d85e7640 1 -- 172.21.15.107:0/2022918905 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d01a1b60 con 0x7ff0d0101960 2024-05-09T08:30:25.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.874+0000 7ff0c77fe640 1 -- 172.21.15.107:0/2022918905 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0018820 con 0x7ff0d0101960 2024-05-09T08:30:25.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.874+0000 7ff0d85e7640 1 -- 172.21.15.107:0/2022918905 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d01a20b0 con 0x7ff0d0101960 2024-05-09T08:30:25.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.875+0000 7ff0d85e7640 1 -- 172.21.15.107:0/2022918905 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0d01a2280 con 0x7ff0d0101960 2024-05-09T08:30:25.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.876+0000 7ff0c77fe640 1 -- 172.21.15.107:0/2022918905 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7ff0c00189c0 con 0x7ff0d0101960 2024-05-09T08:30:25.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.877+0000 7ff0c77fe640 1 --2- 172.21.15.107:0/2022918905 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff0bc0a2b90 0x7ff0bc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:25.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.877+0000 7ff0d6383640 1 --2- 172.21.15.107:0/2022918905 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff0bc0a2b90 0x7ff0bc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:25.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.877+0000 7ff0c77fe640 1 -- 172.21.15.107:0/2022918905 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff0c00ced60 con 0x7ff0d0101960 2024-05-09T08:30:25.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.878+0000 7ff0d6383640 1 --2- 172.21.15.107:0/2022918905 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff0bc0a2b90 0x7ff0bc0a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc000c00 tx=0x7ff0cc0314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:25.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:25.882+0000 7ff0c77fe640 1 -- 172.21.15.107:0/2022918905 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff0c0014030 con 0x7ff0d0101960 2024-05-09T08:30:26.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:26.095+0000 7ff0d85e7640 1 -- 172.21.15.107:0/2022918905 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0d01a2450 con 0x7ff0d0101960 2024-05-09T08:30:26.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:26.096+0000 7ff0c77fe640 1 -- 172.21.15.107:0/2022918905 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7ff0c009c160 con 0x7ff0d0101960 2024-05-09T08:30:26.096 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:26.096 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-09T08:30:26.096 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 2 2024-05-09T08:30:26.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:26.098+0000 7ff0d85e7640 1 -- 172.21.15.107:0/2022918905 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff0bc0a2b90 msgr2=0x7ff0bc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:26.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:26.098+0000 7ff0d85e7640 1 --2- 172.21.15.107:0/2022918905 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff0bc0a2b90 0x7ff0bc0a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc000c00 tx=0x7ff0cc0314a0 comp rx=0 tx=0).stop 2024-05-09T08:30:26.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:26.098+0000 7ff0d85e7640 1 -- 172.21.15.107:0/2022918905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0d0101960 msgr2=0x7ff0d019a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:26.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:26.098+0000 7ff0d85e7640 1 --2- 172.21.15.107:0/2022918905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0d0101960 0x7ff0d019a1a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff0c000c920 tx=0x7ff0c000cde0 comp rx=0 tx=0).stop 2024-05-09T08:30:26.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:26.098+0000 7ff0d85e7640 1 -- 172.21.15.107:0/2022918905 shutdown_connections 2024-05-09T08:30:26.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:26.098+0000 7ff0d85e7640 1 --2- 172.21.15.107:0/2022918905 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff0bc0a2b90 0x7ff0bc0a5050 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:26.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:26.098+0000 7ff0d85e7640 1 --2- 172.21.15.107:0/2022918905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0d0102320 0x7ff0d01a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:26.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:26.098+0000 7ff0d85e7640 1 --2- 172.21.15.107:0/2022918905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0d0101960 0x7ff0d019a1a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:26.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:26.098+0000 7ff0d85e7640 1 --2- 172.21.15.107:0/2022918905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff0d0100770 0x7ff0d0199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:26.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:26.098+0000 7ff0d85e7640 1 -- 172.21.15.107:0/2022918905 >> 172.21.15.107:0/2022918905 conn(0x7ff0d00fbce0 msgr2=0x7ff0d00fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:26.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:26.098+0000 7ff0d85e7640 1 -- 172.21.15.107:0/2022918905 shutdown_connections 2024-05-09T08:30:26.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:26.098+0000 7ff0d85e7640 1 -- 172.21.15.107:0/2022918905 wait complete. 2024-05-09T08:30:26.121 INFO:journalctl@ceph.mds.k.smithi110.stdout:May 09 08:30:26 smithi110 systemd[1]: Started Ceph mds.k for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:30:26.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:26 smithi107 ceph-mon[30122]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:26.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:26 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2022918905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:26.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:26.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:26.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:26.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:26.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:26.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:26 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:26.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:26 smithi157 ceph-mon[33614]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:26.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:26 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2022918905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:26.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:26.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:26.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:26.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:26.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:26.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:26 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:26 smithi110 ceph-mon[33622]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:26 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2022918905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:26 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:27.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:27 smithi107 ceph-mon[30122]: Deploying daemon mds.j on smithi107 2024-05-09T08:30:27.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:27 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.110:6832/844626352,v1:172.21.15.110:6833/844626352] up:boot 2024-05-09T08:30:27.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:27 smithi107 ceph-mon[30122]: fsmap 2 up:standby 2024-05-09T08:30:27.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:27 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:30:27.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:27 smithi157 ceph-mon[33614]: Deploying daemon mds.j on smithi107 2024-05-09T08:30:27.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:27 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.110:6832/844626352,v1:172.21.15.110:6833/844626352] up:boot 2024-05-09T08:30:27.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:27 smithi157 ceph-mon[33614]: fsmap 2 up:standby 2024-05-09T08:30:27.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:27 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:30:27.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:27 smithi110 ceph-mon[33622]: Deploying daemon mds.j on smithi107 2024-05-09T08:30:27.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:27 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.110:6832/844626352,v1:172.21.15.110:6833/844626352] up:boot 2024-05-09T08:30:27.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:27 smithi110 ceph-mon[33622]: fsmap 2 up:standby 2024-05-09T08:30:27.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:27 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:30:28.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:30:28.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.224+0000 7f326c7dc640 1 -- 172.21.15.107:0/1647423320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3264102bc0 msgr2=0x7f3264103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:28.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.224+0000 7f326c7dc640 1 --2- 172.21.15.107:0/1647423320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3264102bc0 0x7f3264103040 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3260009930 tx=0x7f326002a420 comp rx=0 tx=0).stop 2024-05-09T08:30:28.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.224+0000 7f326c7dc640 1 -- 172.21.15.107:0/1647423320 shutdown_connections 2024-05-09T08:30:28.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.224+0000 7f326c7dc640 1 --2- 172.21.15.107:0/1647423320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3264103580 0x7f3264109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:28.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.224+0000 7f326c7dc640 1 --2- 172.21.15.107:0/1647423320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3264102bc0 0x7f3264103040 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:28.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.224+0000 7f326c7dc640 1 --2- 172.21.15.107:0/1647423320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32640ffb50 0x7f32640fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:28.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.224+0000 7f326c7dc640 1 -- 172.21.15.107:0/1647423320 >> 172.21.15.107:0/1647423320 conn(0x7f32640fb460 msgr2=0x7f32640fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:28.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.224+0000 7f326c7dc640 1 -- 172.21.15.107:0/1647423320 shutdown_connections 2024-05-09T08:30:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.224+0000 7f326c7dc640 1 -- 172.21.15.107:0/1647423320 wait complete. 2024-05-09T08:30:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.224+0000 7f326c7dc640 1 Processor -- start 2024-05-09T08:30:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f326c7dc640 1 -- start start 2024-05-09T08:30:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f326c7dc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32640ffb50 0x7f3264102530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f326c7dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3264102bc0 0x7f3264100b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f326c7dc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3264103580 0x7f32641010b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f326c7dc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f326410d3b0 con 0x7f3264102bc0 2024-05-09T08:30:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f326c7dc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f326410d230 con 0x7f32640ffb50 2024-05-09T08:30:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f326c7dc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f326410d530 con 0x7f3264103580 2024-05-09T08:30:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f3269d77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3264102bc0 0x7f3264100b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f3269d77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3264102bc0 0x7f3264100b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51404/0 (socket says 172.21.15.107:51404) 2024-05-09T08:30:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f3269d77640 1 -- 172.21.15.107:0/4271346455 learned_addr learned my addr 172.21.15.107:0/4271346455 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f326a578640 1 --2- 172.21.15.107:0/4271346455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32640ffb50 0x7f3264102530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:28.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f326ad79640 1 --2- 172.21.15.107:0/4271346455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3264103580 0x7f32641010b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:28.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f3269d77640 1 -- 172.21.15.107:0/4271346455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3264103580 msgr2=0x7f32641010b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:28.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f3269d77640 1 --2- 172.21.15.107:0/4271346455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3264103580 0x7f32641010b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:28.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f3269d77640 1 -- 172.21.15.107:0/4271346455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32640ffb50 msgr2=0x7f3264102530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:28.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f3269d77640 1 --2- 172.21.15.107:0/4271346455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32640ffb50 0x7f3264102530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:28.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f3269d77640 1 -- 172.21.15.107:0/4271346455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3264101940 con 0x7f3264102bc0 2024-05-09T08:30:28.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f326a578640 1 --2- 172.21.15.107:0/4271346455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32640ffb50 0x7f3264102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:30:28.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f3269d77640 1 --2- 172.21.15.107:0/4271346455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3264102bc0 0x7f3264100b70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f32600314e0 tx=0x7f3260031a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:28.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f325b7fe640 1 -- 172.21.15.107:0/4271346455 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326003f030 con 0x7f3264102bc0 2024-05-09T08:30:28.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.225+0000 7f326c7dc640 1 -- 172.21.15.107:0/4271346455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32641a1ba0 con 0x7f3264102bc0 2024-05-09T08:30:28.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.226+0000 7f325b7fe640 1 -- 172.21.15.107:0/4271346455 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32600044b0 con 0x7f3264102bc0 2024-05-09T08:30:28.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.226+0000 7f325b7fe640 1 -- 172.21.15.107:0/4271346455 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260035070 con 0x7f3264102bc0 2024-05-09T08:30:28.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.226+0000 7f326c7dc640 1 -- 172.21.15.107:0/4271346455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32641a2140 con 0x7f3264102bc0 2024-05-09T08:30:28.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.227+0000 7f326c7dc640 1 -- 172.21.15.107:0/4271346455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3264108d10 con 0x7f3264102bc0 2024-05-09T08:30:28.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.228+0000 7f325b7fe640 1 -- 172.21.15.107:0/4271346455 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3260042030 con 0x7f3264102bc0 2024-05-09T08:30:28.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.228+0000 7f325b7fe640 1 --2- 172.21.15.107:0/4271346455 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f32500a2b90 0x7f32500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:28.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.228+0000 7f325b7fe640 1 -- 172.21.15.107:0/4271346455 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f32600ef0e0 con 0x7f3264102bc0 2024-05-09T08:30:28.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.228+0000 7f326a578640 1 --2- 172.21.15.107:0/4271346455 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f32500a2b90 0x7f32500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:28.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.229+0000 7f326a578640 1 --2- 172.21.15.107:0/4271346455 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f32500a2b90 0x7f32500a5050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3264068700 tx=0x7f3254005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:28.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.233+0000 7f325b7fe640 1 -- 172.21.15.107:0/4271346455 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f32600bc4e0 con 0x7f3264102bc0 2024-05-09T08:30:28.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.460+0000 7f326c7dc640 1 -- 172.21.15.107:0/4271346455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f326402d400 con 0x7f3264102bc0 2024-05-09T08:30:28.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.461+0000 7f325b7fe640 1 -- 172.21.15.107:0/4271346455 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f32600bbde0 con 0x7f3264102bc0 2024-05-09T08:30:28.461 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:28.461 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-09T08:30:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 3 2024-05-09T08:30:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.463+0000 7f326c7dc640 1 -- 172.21.15.107:0/4271346455 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f32500a2b90 msgr2=0x7f32500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.463+0000 7f326c7dc640 1 --2- 172.21.15.107:0/4271346455 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f32500a2b90 0x7f32500a5050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3264068700 tx=0x7f3254005f90 comp rx=0 tx=0).stop 2024-05-09T08:30:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.463+0000 7f326c7dc640 1 -- 172.21.15.107:0/4271346455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3264102bc0 msgr2=0x7f3264100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.463+0000 7f326c7dc640 1 --2- 172.21.15.107:0/4271346455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3264102bc0 0x7f3264100b70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f32600314e0 tx=0x7f3260031a30 comp rx=0 tx=0).stop 2024-05-09T08:30:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.463+0000 7f326c7dc640 1 -- 172.21.15.107:0/4271346455 shutdown_connections 2024-05-09T08:30:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.463+0000 7f326c7dc640 1 --2- 172.21.15.107:0/4271346455 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f32500a2b90 0x7f32500a5050 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.463+0000 7f326c7dc640 1 --2- 172.21.15.107:0/4271346455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3264103580 0x7f32641010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.463+0000 7f326c7dc640 1 --2- 172.21.15.107:0/4271346455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3264102bc0 0x7f3264100b70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.463+0000 7f326c7dc640 1 --2- 172.21.15.107:0/4271346455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32640ffb50 0x7f3264102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:28.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.463+0000 7f326c7dc640 1 -- 172.21.15.107:0/4271346455 >> 172.21.15.107:0/4271346455 conn(0x7f32640fb460 msgr2=0x7f3264105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:28.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.463+0000 7f326c7dc640 1 -- 172.21.15.107:0/4271346455 shutdown_connections 2024-05-09T08:30:28.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:28.463+0000 7f326c7dc640 1 -- 172.21.15.107:0/4271346455 wait complete. 2024-05-09T08:30:28.473 INFO:journalctl@ceph.mds.j.smithi107.stdout:May 09 08:30:28 smithi107 systemd[1]: Starting Ceph mds.j for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:30:28.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:28 smithi110 ceph-mon[33622]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:28.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:28 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/4271346455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:28.725 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:28 smithi107 ceph-mon[30122]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:28.725 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:28 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/4271346455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:28.726 INFO:journalctl@ceph.mds.j.smithi107.stdout:May 09 08:30:28 smithi107 podman[68106]: 2024-05-09 08:30:28.461421552 +0000 UTC m=+0.018096053 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:30:28.726 INFO:journalctl@ceph.mds.j.smithi107.stdout:May 09 08:30:28 smithi107 podman[68106]: 2024-05-09 08:30:28.584194681 +0000 UTC m=+0.140869168 container create 908a53818100522105bc2473dbec051e3bd15b13bf736889db8cd3680c99db13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-j, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=) 2024-05-09T08:30:28.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:28 smithi157 ceph-mon[33614]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:28.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:28 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/4271346455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:29.279 INFO:journalctl@ceph.mds.j.smithi107.stdout:May 09 08:30:28 smithi107 podman[68106]: 2024-05-09 08:30:28.829551591 +0000 UTC m=+0.386226089 container init 908a53818100522105bc2473dbec051e3bd15b13bf736889db8cd3680c99db13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, io.buildah.version=1.33.5) 2024-05-09T08:30:29.279 INFO:journalctl@ceph.mds.j.smithi107.stdout:May 09 08:30:28 smithi107 podman[68106]: 2024-05-09 08:30:28.832362543 +0000 UTC m=+0.389037036 container start 908a53818100522105bc2473dbec051e3bd15b13bf736889db8cd3680c99db13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-j, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-09T08:30:29.279 INFO:journalctl@ceph.mds.j.smithi107.stdout:May 09 08:30:28 smithi107 ceph-mds[68127]: -- 172.21.15.107:0/1885826 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d77166fa40 con 0x55d771642c00 2024-05-09T08:30:29.279 INFO:journalctl@ceph.mds.j.smithi107.stdout:May 09 08:30:28 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-j[68123]: starting mds.j at 2024-05-09T08:30:29.280 INFO:journalctl@ceph.mds.j.smithi107.stdout:May 09 08:30:28 smithi107 bash[68106]: 908a53818100522105bc2473dbec051e3bd15b13bf736889db8cd3680c99db13 2024-05-09T08:30:29.280 INFO:journalctl@ceph.mds.j.smithi107.stdout:May 09 08:30:28 smithi107 systemd[1]: Started Ceph mds.j for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:30:30.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:29 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:30.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:29 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:30.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:29 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:30.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:29 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:30.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:29 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:30.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:29 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:30.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:29 smithi107 ceph-mon[30122]: Deploying daemon mds.d on smithi107 2024-05-09T08:30:30.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:29 smithi107 ceph-mon[30122]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:30.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:29 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.107:6834/3401311866,v1:172.21.15.107:6835/3401311866] up:boot 2024-05-09T08:30:30.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:29 smithi107 ceph-mon[30122]: fsmap 3 up:standby 2024-05-09T08:30:30.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:29 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:30:30.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:29 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:30.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:29 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:30.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:29 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:30.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:29 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:30.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:29 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:30.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:29 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:30.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:29 smithi157 ceph-mon[33614]: Deploying daemon mds.d on smithi107 2024-05-09T08:30:30.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:29 smithi157 ceph-mon[33614]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:30.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:29 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.107:6834/3401311866,v1:172.21.15.107:6835/3401311866] up:boot 2024-05-09T08:30:30.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:29 smithi157 ceph-mon[33614]: fsmap 3 up:standby 2024-05-09T08:30:30.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:29 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:30:30.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:29 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:30.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:29 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:30.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:29 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:30.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:29 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:30.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:29 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:30.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:29 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:30.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:29 smithi110 ceph-mon[33622]: Deploying daemon mds.d on smithi107 2024-05-09T08:30:30.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:29 smithi110 ceph-mon[33622]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:30.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:29 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.107:6834/3401311866,v1:172.21.15.107:6835/3401311866] up:boot 2024-05-09T08:30:30.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:29 smithi110 ceph-mon[33622]: fsmap 3 up:standby 2024-05-09T08:30:30.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:29 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:30:30.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:30:30.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.583+0000 7f5f2f2ba640 1 -- 172.21.15.107:0/2706192891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f28103430 msgr2=0x7f5f28109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:30.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.583+0000 7f5f2f2ba640 1 --2- 172.21.15.107:0/2706192891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f28103430 0x7f5f28109cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009ca0 tx=0x7f5f2002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:30:30.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.583+0000 7f5f2f2ba640 1 -- 172.21.15.107:0/2706192891 shutdown_connections 2024-05-09T08:30:30.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.583+0000 7f5f2f2ba640 1 --2- 172.21.15.107:0/2706192891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f28103430 0x7f5f28109cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:30.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.583+0000 7f5f2f2ba640 1 --2- 172.21.15.107:0/2706192891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f28102a70 0x7f5f28102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:30.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.583+0000 7f5f2f2ba640 1 --2- 172.21.15.107:0/2706192891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5f28101880 0x7f5f28101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:30.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.584+0000 7f5f2f2ba640 1 -- 172.21.15.107:0/2706192891 >> 172.21.15.107:0/2706192891 conn(0x7f5f280fce10 msgr2=0x7f5f280ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:30.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.584+0000 7f5f2f2ba640 1 -- 172.21.15.107:0/2706192891 shutdown_connections 2024-05-09T08:30:30.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.584+0000 7f5f2f2ba640 1 -- 172.21.15.107:0/2706192891 wait complete. 2024-05-09T08:30:30.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.584+0000 7f5f2f2ba640 1 Processor -- start 2024-05-09T08:30:30.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.584+0000 7f5f2f2ba640 1 -- start start 2024-05-09T08:30:30.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.584+0000 7f5f2f2ba640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5f28101880 0x7f5f280755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:30.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.584+0000 7f5f2f2ba640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f28102a70 0x7f5f28073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:30.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.584+0000 7f5f2f2ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f28103430 0x7f5f28074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:30.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.584+0000 7f5f2f2ba640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2810d430 con 0x7f5f28103430 2024-05-09T08:30:30.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.584+0000 7f5f2f2ba640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2810d2b0 con 0x7f5f28102a70 2024-05-09T08:30:30.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.584+0000 7f5f2f2ba640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2810d5b0 con 0x7f5f28101880 2024-05-09T08:30:30.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.584+0000 7f5f2d857640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f28103430 0x7f5f28074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:30.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.584+0000 7f5f2d857640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f28103430 0x7f5f28074160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51442/0 (socket says 172.21.15.107:51442) 2024-05-09T08:30:30.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.584+0000 7f5f2d857640 1 -- 172.21.15.107:0/2178203172 learned_addr learned my addr 172.21.15.107:0/2178203172 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:30.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.585+0000 7f5f2d056640 1 --2- 172.21.15.107:0/2178203172 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5f28101880 0x7f5f280755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:30.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.585+0000 7f5f2d857640 1 -- 172.21.15.107:0/2178203172 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5f28101880 msgr2=0x7f5f280755e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:30.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.585+0000 7f5f2d857640 1 --2- 172.21.15.107:0/2178203172 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5f28101880 0x7f5f280755e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:30.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.585+0000 7f5f2c855640 1 --2- 172.21.15.107:0/2178203172 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f28102a70 0x7f5f28073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:30.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.585+0000 7f5f2d857640 1 -- 172.21.15.107:0/2178203172 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f28102a70 msgr2=0x7f5f28073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:30.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.585+0000 7f5f2d857640 1 --2- 172.21.15.107:0/2178203172 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f28102a70 0x7f5f28073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:30.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.585+0000 7f5f2d857640 1 -- 172.21.15.107:0/2178203172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f280746a0 con 0x7f5f28103430 2024-05-09T08:30:30.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.585+0000 7f5f2d056640 1 --2- 172.21.15.107:0/2178203172 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5f28101880 0x7f5f280755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:30:30.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.585+0000 7f5f2c855640 1 --2- 172.21.15.107:0/2178203172 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f28102a70 0x7f5f28073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:30:30.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.585+0000 7f5f2d857640 1 --2- 172.21.15.107:0/2178203172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f28103430 0x7f5f28074160 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009dc0 tx=0x7f5f20003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:30.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.585+0000 7f5f1e7fc640 1 -- 172.21.15.107:0/2178203172 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f20032ec0 con 0x7f5f28103430 2024-05-09T08:30:30.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.585+0000 7f5f1e7fc640 1 -- 172.21.15.107:0/2178203172 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f200044f0 con 0x7f5f28103430 2024-05-09T08:30:30.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.585+0000 7f5f2f2ba640 1 -- 172.21.15.107:0/2178203172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f28074930 con 0x7f5f28103430 2024-05-09T08:30:30.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.585+0000 7f5f1e7fc640 1 -- 172.21.15.107:0/2178203172 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f20030d40 con 0x7f5f28103430 2024-05-09T08:30:30.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.585+0000 7f5f2f2ba640 1 -- 172.21.15.107:0/2178203172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f281a6580 con 0x7f5f28103430 2024-05-09T08:30:30.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.586+0000 7f5f2f2ba640 1 -- 172.21.15.107:0/2178203172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f28108bf0 con 0x7f5f28103430 2024-05-09T08:30:30.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.587+0000 7f5f1e7fc640 1 -- 172.21.15.107:0/2178203172 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f5f20004040 con 0x7f5f28103430 2024-05-09T08:30:30.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.588+0000 7f5f1e7fc640 1 --2- 172.21.15.107:0/2178203172 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5f140a2db0 0x7f5f140a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:30.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.588+0000 7f5f2d056640 1 --2- 172.21.15.107:0/2178203172 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5f140a2db0 0x7f5f140a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:30.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.588+0000 7f5f1e7fc640 1 -- 172.21.15.107:0/2178203172 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5f200f04c0 con 0x7f5f28103430 2024-05-09T08:30:30.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.588+0000 7f5f2d056640 1 --2- 172.21.15.107:0/2178203172 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5f140a2db0 0x7f5f140a5270 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5f240058c0 tx=0x7f5f24005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:30.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.592+0000 7f5f1e7fc640 1 -- 172.21.15.107:0/2178203172 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5f200bd820 con 0x7f5f28103430 2024-05-09T08:30:30.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.808+0000 7f5f2f2ba640 1 -- 172.21.15.107:0/2178203172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f2802d1c0 con 0x7f5f28103430 2024-05-09T08:30:30.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.808+0000 7f5f1e7fc640 1 -- 172.21.15.107:0/2178203172 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7f5f200bd120 con 0x7f5f28103430 2024-05-09T08:30:30.808 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:30.808 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-09T08:30:30.808 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 4 2024-05-09T08:30:30.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.810+0000 7f5f2f2ba640 1 -- 172.21.15.107:0/2178203172 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5f140a2db0 msgr2=0x7f5f140a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:30.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.810+0000 7f5f2f2ba640 1 --2- 172.21.15.107:0/2178203172 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5f140a2db0 0x7f5f140a5270 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5f240058c0 tx=0x7f5f24005850 comp rx=0 tx=0).stop 2024-05-09T08:30:30.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.810+0000 7f5f2f2ba640 1 -- 172.21.15.107:0/2178203172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f28103430 msgr2=0x7f5f28074160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:30.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.810+0000 7f5f2f2ba640 1 --2- 172.21.15.107:0/2178203172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f28103430 0x7f5f28074160 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009dc0 tx=0x7f5f20003ba0 comp rx=0 tx=0).stop 2024-05-09T08:30:30.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.811+0000 7f5f2f2ba640 1 -- 172.21.15.107:0/2178203172 shutdown_connections 2024-05-09T08:30:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.811+0000 7f5f2f2ba640 1 --2- 172.21.15.107:0/2178203172 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5f140a2db0 0x7f5f140a5270 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.811+0000 7f5f2f2ba640 1 --2- 172.21.15.107:0/2178203172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f28103430 0x7f5f28074160 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.811+0000 7f5f2f2ba640 1 --2- 172.21.15.107:0/2178203172 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f28102a70 0x7f5f28073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.811+0000 7f5f2f2ba640 1 --2- 172.21.15.107:0/2178203172 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5f28101880 0x7f5f280755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.811+0000 7f5f2f2ba640 1 -- 172.21.15.107:0/2178203172 >> 172.21.15.107:0/2178203172 conn(0x7f5f280fce10 msgr2=0x7f5f28105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.811+0000 7f5f2f2ba640 1 -- 172.21.15.107:0/2178203172 shutdown_connections 2024-05-09T08:30:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:30.811+0000 7f5f2f2ba640 1 -- 172.21.15.107:0/2178203172 wait complete. 2024-05-09T08:30:31.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:31 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2178203172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:31.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:31 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2178203172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:31.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:31 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2178203172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:31.436 INFO:journalctl@ceph.mds.d.smithi107.stdout:May 09 08:30:31 smithi107 systemd[1]: Starting Ceph mds.d for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:30:31.752 INFO:journalctl@ceph.mds.d.smithi107.stdout:May 09 08:30:31 smithi107 podman[68458]: 2024-05-09 08:30:31.340200084 +0000 UTC m=+0.017543714 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:30:31.752 INFO:journalctl@ceph.mds.d.smithi107.stdout:May 09 08:30:31 smithi107 podman[68458]: 2024-05-09 08:30:31.517036877 +0000 UTC m=+0.194380492 container create 8dc3fa32b084a7c674d0136484916ea60cb3223abd39f9325eb947cfe0fa2422 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, GIT_BRANCH=HEAD) 2024-05-09T08:30:31.752 INFO:journalctl@ceph.mds.d.smithi107.stdout:May 09 08:30:31 smithi107 podman[68458]: 2024-05-09 08:30:31.75355895 +0000 UTC m=+0.430902571 container init 8dc3fa32b084a7c674d0136484916ea60cb3223abd39f9325eb947cfe0fa2422 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-d, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-09T08:30:32.014 INFO:journalctl@ceph.mds.d.smithi107.stdout:May 09 08:30:31 smithi107 podman[68458]: 2024-05-09 08:30:31.75641521 +0000 UTC m=+0.433758835 container start 8dc3fa32b084a7c674d0136484916ea60cb3223abd39f9325eb947cfe0fa2422 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, ceph=True, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux ) 2024-05-09T08:30:32.014 INFO:journalctl@ceph.mds.d.smithi107.stdout:May 09 08:30:31 smithi107 ceph-mds[68477]: -- 172.21.15.107:0/413544984 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562ae8bd1a40 con 0x562ae8ba5000 2024-05-09T08:30:32.014 INFO:journalctl@ceph.mds.d.smithi107.stdout:May 09 08:30:31 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-d[68473]: starting mds.d at 2024-05-09T08:30:32.014 INFO:journalctl@ceph.mds.d.smithi107.stdout:May 09 08:30:31 smithi107 bash[68458]: 8dc3fa32b084a7c674d0136484916ea60cb3223abd39f9325eb947cfe0fa2422 2024-05-09T08:30:32.014 INFO:journalctl@ceph.mds.d.smithi107.stdout:May 09 08:30:31 smithi107 systemd[1]: Started Ceph mds.d for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:30:32.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:32 smithi107 ceph-mon[30122]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:32.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:32.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:32.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:32.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:32.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:32.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:32 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:32.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:32 smithi157 ceph-mon[33614]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:32.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:32.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:32.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:32.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:32.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:32.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:32 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:32.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:32 smithi110 ceph-mon[33622]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:32.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:32.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:32.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:32.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:32.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:32.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:32 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:32.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:30:32.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.939+0000 7f0b6e521640 1 -- 172.21.15.107:0/209067492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b68101960 msgr2=0x7f0b68101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:32.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.939+0000 7f0b6e521640 1 --2- 172.21.15.107:0/209067492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b68101960 0x7f0b68101de0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009900 tx=0x7f0b5002a410 comp rx=0 tx=0).stop 2024-05-09T08:30:32.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.939+0000 7f0b6e521640 1 -- 172.21.15.107:0/209067492 shutdown_connections 2024-05-09T08:30:32.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.939+0000 7f0b6e521640 1 --2- 172.21.15.107:0/209067492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b68102320 0x7f0b68108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:32.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.939+0000 7f0b6e521640 1 --2- 172.21.15.107:0/209067492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b68101960 0x7f0b68101de0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:32.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.939+0000 7f0b6e521640 1 --2- 172.21.15.107:0/209067492 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b68100770 0x7f0b68100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:32.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.939+0000 7f0b6e521640 1 -- 172.21.15.107:0/209067492 >> 172.21.15.107:0/209067492 conn(0x7f0b680fbce0 msgr2=0x7f0b680fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:32.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.939+0000 7f0b6e521640 1 -- 172.21.15.107:0/209067492 shutdown_connections 2024-05-09T08:30:32.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.939+0000 7f0b6e521640 1 -- 172.21.15.107:0/209067492 wait complete. 2024-05-09T08:30:32.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b6e521640 1 Processor -- start 2024-05-09T08:30:32.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b6e521640 1 -- start start 2024-05-09T08:30:32.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b6e521640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b68100770 0x7f0b68069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:32.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b6e521640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b68101960 0x7f0b6819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:32.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b6e521640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b68102320 0x7f0b6819ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:32.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b6e521640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6810c2d0 con 0x7f0b68100770 2024-05-09T08:30:32.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b6e521640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6810c150 con 0x7f0b68102320 2024-05-09T08:30:32.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b6e521640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6810c450 con 0x7f0b68101960 2024-05-09T08:30:32.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b67fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b68100770 0x7f0b68069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:32.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b67fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b68100770 0x7f0b68069530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48480/0 (socket says 172.21.15.107:48480) 2024-05-09T08:30:32.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b67fff640 1 -- 172.21.15.107:0/1783052125 learned_addr learned my addr 172.21.15.107:0/1783052125 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:32.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b6cabe640 1 --2- 172.21.15.107:0/1783052125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b68102320 0x7f0b6819ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:32.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b67fff640 1 -- 172.21.15.107:0/1783052125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b68101960 msgr2=0x7f0b6819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:32.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b5ffff640 1 --2- 172.21.15.107:0/1783052125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b68101960 0x7f0b6819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:32.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b67fff640 1 --2- 172.21.15.107:0/1783052125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b68101960 0x7f0b6819c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:32.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b67fff640 1 -- 172.21.15.107:0/1783052125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b68102320 msgr2=0x7f0b6819ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:32.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b67fff640 1 --2- 172.21.15.107:0/1783052125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b68102320 0x7f0b6819ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:32.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b67fff640 1 -- 172.21.15.107:0/1783052125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6819f780 con 0x7f0b68100770 2024-05-09T08:30:32.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b67fff640 1 --2- 172.21.15.107:0/1783052125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b68100770 0x7f0b68069530 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800d950 tx=0x7f0b5800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:32.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b65ffb640 1 -- 172.21.15.107:0/1783052125 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b58017020 con 0x7f0b68100770 2024-05-09T08:30:32.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b65ffb640 1 -- 172.21.15.107:0/1783052125 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b58009c30 con 0x7f0b68100770 2024-05-09T08:30:32.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b6e521640 1 -- 172.21.15.107:0/1783052125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b6819fa10 con 0x7f0b68100770 2024-05-09T08:30:32.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.940+0000 7f0b65ffb640 1 -- 172.21.15.107:0/1783052125 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5800f6b0 con 0x7f0b68100770 2024-05-09T08:30:32.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.941+0000 7f0b6e521640 1 -- 172.21.15.107:0/1783052125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b6819ffb0 con 0x7f0b68100770 2024-05-09T08:30:32.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.941+0000 7f0b6e521640 1 -- 172.21.15.107:0/1783052125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b68107ac0 con 0x7f0b68100770 2024-05-09T08:30:32.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.943+0000 7f0b65ffb640 1 -- 172.21.15.107:0/1783052125 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0b5801a020 con 0x7f0b68100770 2024-05-09T08:30:32.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.943+0000 7f0b65ffb640 1 --2- 172.21.15.107:0/1783052125 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0b540a2c10 0x7f0b540a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:32.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.943+0000 7f0b5ffff640 1 --2- 172.21.15.107:0/1783052125 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0b540a2c10 0x7f0b540a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:32.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.943+0000 7f0b65ffb640 1 -- 172.21.15.107:0/1783052125 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0b580ce800 con 0x7f0b68100770 2024-05-09T08:30:32.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.944+0000 7f0b5ffff640 1 --2- 172.21.15.107:0/1783052125 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0b540a2c10 0x7f0b540a50d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0b5002a930 tx=0x7f0b50005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:32.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:32.948+0000 7f0b65ffb640 1 -- 172.21.15.107:0/1783052125 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0b58009de0 con 0x7f0b68100770 2024-05-09T08:30:33.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:33 smithi110 ceph-mon[33622]: Deploying daemon mds.e on smithi110 2024-05-09T08:30:33.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:33 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.107:6836/4170346807,v1:172.21.15.107:6837/4170346807] up:boot 2024-05-09T08:30:33.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:33 smithi110 ceph-mon[33622]: fsmap 4 up:standby 2024-05-09T08:30:33.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:30:33.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:33.161+0000 7f0b6e521640 1 -- 172.21.15.107:0/1783052125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b6802d1c0 con 0x7f0b68100770 2024-05-09T08:30:33.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:33.162+0000 7f0b65ffb640 1 -- 172.21.15.107:0/1783052125 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3877 (secure 0 0 0) 0x7f0b5809bbe0 con 0x7f0b68100770 2024-05-09T08:30:33.163 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:33.163 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-09T08:30:33.163 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 5 2024-05-09T08:30:33.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:33.164+0000 7f0b6e521640 1 -- 172.21.15.107:0/1783052125 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0b540a2c10 msgr2=0x7f0b540a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:33.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:33.164+0000 7f0b6e521640 1 --2- 172.21.15.107:0/1783052125 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0b540a2c10 0x7f0b540a50d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0b5002a930 tx=0x7f0b50005fd0 comp rx=0 tx=0).stop 2024-05-09T08:30:33.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:33.164+0000 7f0b6e521640 1 -- 172.21.15.107:0/1783052125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b68100770 msgr2=0x7f0b68069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:33.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:33.164+0000 7f0b6e521640 1 --2- 172.21.15.107:0/1783052125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b68100770 0x7f0b68069530 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800d950 tx=0x7f0b5800de10 comp rx=0 tx=0).stop 2024-05-09T08:30:33.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:33.164+0000 7f0b6e521640 1 -- 172.21.15.107:0/1783052125 shutdown_connections 2024-05-09T08:30:33.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:33.164+0000 7f0b6e521640 1 --2- 172.21.15.107:0/1783052125 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0b540a2c10 0x7f0b540a50d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:33.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:33.164+0000 7f0b6e521640 1 --2- 172.21.15.107:0/1783052125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b68102320 0x7f0b6819ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:33.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:33.164+0000 7f0b6e521640 1 --2- 172.21.15.107:0/1783052125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b68101960 0x7f0b6819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:33.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:33.164+0000 7f0b6e521640 1 --2- 172.21.15.107:0/1783052125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b68100770 0x7f0b68069530 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:33.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:33.164+0000 7f0b6e521640 1 -- 172.21.15.107:0/1783052125 >> 172.21.15.107:0/1783052125 conn(0x7f0b680fbce0 msgr2=0x7f0b68104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:33.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:33.164+0000 7f0b6e521640 1 -- 172.21.15.107:0/1783052125 shutdown_connections 2024-05-09T08:30:33.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:33.164+0000 7f0b6e521640 1 -- 172.21.15.107:0/1783052125 wait complete. 2024-05-09T08:30:33.174 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:33 smithi107 ceph-mon[30122]: Deploying daemon mds.e on smithi110 2024-05-09T08:30:33.174 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:33 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.107:6836/4170346807,v1:172.21.15.107:6837/4170346807] up:boot 2024-05-09T08:30:33.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:33 smithi107 ceph-mon[30122]: fsmap 4 up:standby 2024-05-09T08:30:33.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:30:33.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:33 smithi157 ceph-mon[33614]: Deploying daemon mds.e on smithi110 2024-05-09T08:30:33.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:33 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.107:6836/4170346807,v1:172.21.15.107:6837/4170346807] up:boot 2024-05-09T08:30:33.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:33 smithi157 ceph-mon[33614]: fsmap 4 up:standby 2024-05-09T08:30:33.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:30:34.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:34 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1783052125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:34.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:34 smithi110 ceph-mon[33622]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:34.122 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:30:34 smithi110 systemd[1]: Starting Ceph mds.e for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:30:34.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:34 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1783052125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:34.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:34 smithi107 ceph-mon[30122]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:34.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:34 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1783052125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:34.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:34 smithi157 ceph-mon[33614]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:34.522 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:30:34 smithi110 podman[63149]: 2024-05-09 08:30:34.174058247 +0000 UTC m=+0.017642154 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:30:34.523 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:30:34 smithi110 podman[63149]: 2024-05-09 08:30:34.302014398 +0000 UTC m=+0.145598292 container create 388cf819a5548bc3cfdb6162ff11453b5e2a9a1efddd7344dd3988aa86c3897a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-09T08:30:34.523 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:30:34 smithi110 podman[63149]: 2024-05-09 08:30:34.521200232 +0000 UTC m=+0.364784120 container init 388cf819a5548bc3cfdb6162ff11453b5e2a9a1efddd7344dd3988aa86c3897a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-09T08:30:34.871 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:30:34 smithi110 podman[63149]: 2024-05-09 08:30:34.523964138 +0000 UTC m=+0.367548026 container start 388cf819a5548bc3cfdb6162ff11453b5e2a9a1efddd7344dd3988aa86c3897a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=) 2024-05-09T08:30:34.871 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:30:34 smithi110 ceph-mds[63167]: -- 172.21.15.110:0/1645067131 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56444c069a40 con 0x56444c03d000 2024-05-09T08:30:34.871 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:30:34 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-e[63163]: starting mds.e at 2024-05-09T08:30:34.871 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:30:34 smithi110 bash[63149]: 388cf819a5548bc3cfdb6162ff11453b5e2a9a1efddd7344dd3988aa86c3897a 2024-05-09T08:30:34.871 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:30:34 smithi110 systemd[1]: Started Ceph mds.e for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:30:35.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:30:35.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.289+0000 7f4c5e666640 1 -- 172.21.15.107:0/1228072335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c58102a70 msgr2=0x7f4c58102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:35.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.289+0000 7f4c5e666640 1 --2- 172.21.15.107:0/1228072335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c58102a70 0x7f4c58102ef0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009900 tx=0x7f4c4c02a410 comp rx=0 tx=0).stop 2024-05-09T08:30:35.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.289+0000 7f4c5e666640 1 -- 172.21.15.107:0/1228072335 shutdown_connections 2024-05-09T08:30:35.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.289+0000 7f4c5e666640 1 --2- 172.21.15.107:0/1228072335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c58103430 0x7f4c58109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:35.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.289+0000 7f4c5e666640 1 --2- 172.21.15.107:0/1228072335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c58102a70 0x7f4c58102ef0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:35.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.289+0000 7f4c5e666640 1 --2- 172.21.15.107:0/1228072335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c58101880 0x7f4c58101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:35.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.289+0000 7f4c5e666640 1 -- 172.21.15.107:0/1228072335 >> 172.21.15.107:0/1228072335 conn(0x7f4c580fce10 msgr2=0x7f4c580ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:35.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.289+0000 7f4c5e666640 1 -- 172.21.15.107:0/1228072335 shutdown_connections 2024-05-09T08:30:35.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.289+0000 7f4c5e666640 1 -- 172.21.15.107:0/1228072335 wait complete. 2024-05-09T08:30:35.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.290+0000 7f4c5e666640 1 Processor -- start 2024-05-09T08:30:35.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.290+0000 7f4c5e666640 1 -- start start 2024-05-09T08:30:35.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.290+0000 7f4c5e666640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c58101880 0x7f4c581979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:35.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.290+0000 7f4c5e666640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c58102a70 0x7f4c58197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:35.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.290+0000 7f4c5e666640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c58103430 0x7f4c5819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.290+0000 7f4c5e666640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5810d390 con 0x7f4c58103430 2024-05-09T08:30:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.290+0000 7f4c5e666640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5810d210 con 0x7f4c58101880 2024-05-09T08:30:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.290+0000 7f4c5e666640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5810d510 con 0x7f4c58102a70 2024-05-09T08:30:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.290+0000 7f4c5cc03640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c58103430 0x7f4c5819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.290+0000 7f4c577fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c58102a70 0x7f4c58197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.291+0000 7f4c5cc03640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c58103430 0x7f4c5819ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48504/0 (socket says 172.21.15.107:48504) 2024-05-09T08:30:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.291+0000 7f4c5cc03640 1 -- 172.21.15.107:0/2313365008 learned_addr learned my addr 172.21.15.107:0/2313365008 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.291+0000 7f4c57fff640 1 --2- 172.21.15.107:0/2313365008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c58101880 0x7f4c581979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.291+0000 7f4c5cc03640 1 -- 172.21.15.107:0/2313365008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c58102a70 msgr2=0x7f4c58197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.291+0000 7f4c5cc03640 1 --2- 172.21.15.107:0/2313365008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c58102a70 0x7f4c58197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.291+0000 7f4c5cc03640 1 -- 172.21.15.107:0/2313365008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c58101880 msgr2=0x7f4c581979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.291+0000 7f4c5cc03640 1 --2- 172.21.15.107:0/2313365008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c58101880 0x7f4c581979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.291+0000 7f4c5cc03640 1 -- 172.21.15.107:0/2313365008 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c5819f5c0 con 0x7f4c58103430 2024-05-09T08:30:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.291+0000 7f4c57fff640 1 --2- 172.21.15.107:0/2313365008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c58101880 0x7f4c581979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:30:35.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.291+0000 7f4c5cc03640 1 --2- 172.21.15.107:0/2313365008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c58103430 0x7f4c5819ef50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000cce0 tx=0x7f4c5000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:35.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.291+0000 7f4c557fa640 1 -- 172.21.15.107:0/2313365008 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5000f450 con 0x7f4c58103430 2024-05-09T08:30:35.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.291+0000 7f4c557fa640 1 -- 172.21.15.107:0/2313365008 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c5000faa0 con 0x7f4c58103430 2024-05-09T08:30:35.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.291+0000 7f4c5e666640 1 -- 172.21.15.107:0/2313365008 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c5819f8b0 con 0x7f4c58103430 2024-05-09T08:30:35.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.291+0000 7f4c557fa640 1 -- 172.21.15.107:0/2313365008 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c50018b70 con 0x7f4c58103430 2024-05-09T08:30:35.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.291+0000 7f4c5e666640 1 -- 172.21.15.107:0/2313365008 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c5819fd80 con 0x7f4c58103430 2024-05-09T08:30:35.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.292+0000 7f4c5e666640 1 -- 172.21.15.107:0/2313365008 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c58108ba0 con 0x7f4c58103430 2024-05-09T08:30:35.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.293+0000 7f4c557fa640 1 -- 172.21.15.107:0/2313365008 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f4c50018d10 con 0x7f4c58103430 2024-05-09T08:30:35.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.294+0000 7f4c557fa640 1 --2- 172.21.15.107:0/2313365008 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4c400a2b90 0x7f4c400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:35.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.295+0000 7f4c57fff640 1 --2- 172.21.15.107:0/2313365008 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4c400a2b90 0x7f4c400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:35.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.295+0000 7f4c557fa640 1 -- 172.21.15.107:0/2313365008 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4c500cf030 con 0x7f4c58103430 2024-05-09T08:30:35.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.295+0000 7f4c57fff640 1 --2- 172.21.15.107:0/2313365008 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4c400a2b90 0x7f4c400a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4c581028d0 tx=0x7f4c4800a5e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:35.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.299+0000 7f4c557fa640 1 -- 172.21.15.107:0/2313365008 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4c50014030 con 0x7f4c58103430 2024-05-09T08:30:35.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.513+0000 7f4c5e666640 1 -- 172.21.15.107:0/2313365008 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c5802d3a0 con 0x7f4c58103430 2024-05-09T08:30:35.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.514+0000 7f4c557fa640 1 -- 172.21.15.107:0/2313365008 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4705 (secure 0 0 0) 0x7f4c5009c430 con 0x7f4c58103430 2024-05-09T08:30:35.515 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:35.515 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-09T08:30:35.515 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 6 2024-05-09T08:30:35.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.516+0000 7f4c5e666640 1 -- 172.21.15.107:0/2313365008 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4c400a2b90 msgr2=0x7f4c400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:35.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.516+0000 7f4c5e666640 1 --2- 172.21.15.107:0/2313365008 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4c400a2b90 0x7f4c400a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4c581028d0 tx=0x7f4c4800a5e0 comp rx=0 tx=0).stop 2024-05-09T08:30:35.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.516+0000 7f4c5e666640 1 -- 172.21.15.107:0/2313365008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c58103430 msgr2=0x7f4c5819ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:35.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.516+0000 7f4c5e666640 1 --2- 172.21.15.107:0/2313365008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c58103430 0x7f4c5819ef50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000cce0 tx=0x7f4c5000a620 comp rx=0 tx=0).stop 2024-05-09T08:30:35.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.516+0000 7f4c5e666640 1 -- 172.21.15.107:0/2313365008 shutdown_connections 2024-05-09T08:30:35.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.516+0000 7f4c5e666640 1 --2- 172.21.15.107:0/2313365008 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4c400a2b90 0x7f4c400a5050 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:35.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.516+0000 7f4c5e666640 1 --2- 172.21.15.107:0/2313365008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c58103430 0x7f4c5819ef50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:35.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.516+0000 7f4c5e666640 1 --2- 172.21.15.107:0/2313365008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c58102a70 0x7f4c58197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:35.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.516+0000 7f4c5e666640 1 --2- 172.21.15.107:0/2313365008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c58101880 0x7f4c581979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:35.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.516+0000 7f4c5e666640 1 -- 172.21.15.107:0/2313365008 >> 172.21.15.107:0/2313365008 conn(0x7f4c580fce10 msgr2=0x7f4c58105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:35.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.516+0000 7f4c5e666640 1 -- 172.21.15.107:0/2313365008 shutdown_connections 2024-05-09T08:30:35.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:35.516+0000 7f4c5e666640 1 -- 172.21.15.107:0/2313365008 wait complete. 2024-05-09T08:30:35.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:35 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:35.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:35 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:35.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:35 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:35.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:35 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:35.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:35 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:35.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:35 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:35.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:35 smithi107 ceph-mon[30122]: Deploying daemon mds.f on smithi157 2024-05-09T08:30:35.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:35 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.110:6834/2138165106,v1:172.21.15.110:6835/2138165106] up:boot 2024-05-09T08:30:35.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:35 smithi107 ceph-mon[30122]: fsmap 5 up:standby 2024-05-09T08:30:35.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:35 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:30:35.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:35 smithi107 ceph-mon[30122]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:35.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:35 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2313365008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:36.071 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:35 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:36.071 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:35 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:36.071 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:35 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:36.071 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:35 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:36.071 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:35 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:36.071 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:35 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:36.071 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:35 smithi157 ceph-mon[33614]: Deploying daemon mds.f on smithi157 2024-05-09T08:30:36.071 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:35 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.110:6834/2138165106,v1:172.21.15.110:6835/2138165106] up:boot 2024-05-09T08:30:36.071 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:35 smithi157 ceph-mon[33614]: fsmap 5 up:standby 2024-05-09T08:30:36.071 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:35 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:30:36.071 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:35 smithi157 ceph-mon[33614]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:36.071 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:35 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2313365008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:36.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:35 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:36.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:35 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:36.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:35 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:36.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:35 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:36.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:35 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:36.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:35 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:36.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:35 smithi110 ceph-mon[33622]: Deploying daemon mds.f on smithi157 2024-05-09T08:30:36.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:35 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.110:6834/2138165106,v1:172.21.15.110:6835/2138165106] up:boot 2024-05-09T08:30:36.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:35 smithi110 ceph-mon[33622]: fsmap 5 up:standby 2024-05-09T08:30:36.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:35 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:30:36.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:35 smithi110 ceph-mon[33622]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:36.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:35 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2313365008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:37.063 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:30:36 smithi157 systemd[1]: Starting Ceph mds.f for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:30:37.336 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:30:37 smithi157 podman[63713]: 2024-05-09 08:30:36.966776846 +0000 UTC m=+0.018151976 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:30:37.336 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:30:37 smithi157 podman[63713]: 2024-05-09 08:30:37.083284725 +0000 UTC m=+0.134659838 container create 9a5a8cc830f29a1e50651c8d87fcaee1ef81375587d670cd36e2c3fc2756c002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:30:37.336 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:30:37 smithi157 podman[63713]: 2024-05-09 08:30:37.336945486 +0000 UTC m=+0.388320607 container init 9a5a8cc830f29a1e50651c8d87fcaee1ef81375587d670cd36e2c3fc2756c002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-96805dc) 2024-05-09T08:30:37.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:30:37.594 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:30:37 smithi157 podman[63713]: 2024-05-09 08:30:37.340362668 +0000 UTC m=+0.391737778 container start 9a5a8cc830f29a1e50651c8d87fcaee1ef81375587d670cd36e2c3fc2756c002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-05-09T08:30:37.594 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:30:37 smithi157 ceph-mds[63733]: -- 172.21.15.157:0/840816892 <== mon.1 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561b2c389a40 con 0x561b2c35d000 2024-05-09T08:30:37.594 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:30:37 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: starting mds.f at 2024-05-09T08:30:37.594 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:30:37 smithi157 bash[63713]: 9a5a8cc830f29a1e50651c8d87fcaee1ef81375587d670cd36e2c3fc2756c002 2024-05-09T08:30:37.594 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:30:37 smithi157 systemd[1]: Started Ceph mds.f for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:30:37.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.642+0000 7fa28fa4d640 1 -- 172.21.15.107:0/1884365624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288102320 msgr2=0x7fa288108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:37.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.642+0000 7fa28fa4d640 1 --2- 172.21.15.107:0/1884365624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288102320 0x7fa288108bb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa280009900 tx=0x7fa28002a410 comp rx=0 tx=0).stop 2024-05-09T08:30:37.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.642+0000 7fa28fa4d640 1 -- 172.21.15.107:0/1884365624 shutdown_connections 2024-05-09T08:30:37.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.642+0000 7fa28fa4d640 1 --2- 172.21.15.107:0/1884365624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288102320 0x7fa288108bb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:37.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.642+0000 7fa28fa4d640 1 --2- 172.21.15.107:0/1884365624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa288101960 0x7fa288101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:37.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.642+0000 7fa28fa4d640 1 --2- 172.21.15.107:0/1884365624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa288100770 0x7fa288100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:37.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.642+0000 7fa28fa4d640 1 -- 172.21.15.107:0/1884365624 >> 172.21.15.107:0/1884365624 conn(0x7fa2880fbce0 msgr2=0x7fa2880fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:37.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.642+0000 7fa28fa4d640 1 -- 172.21.15.107:0/1884365624 shutdown_connections 2024-05-09T08:30:37.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.642+0000 7fa28fa4d640 1 -- 172.21.15.107:0/1884365624 wait complete. 2024-05-09T08:30:37.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.642+0000 7fa28fa4d640 1 Processor -- start 2024-05-09T08:30:37.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.642+0000 7fa28fa4d640 1 -- start start 2024-05-09T08:30:37.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28fa4d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288100770 0x7fa288196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:37.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28fa4d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa288101960 0x7fa288196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:37.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28fa4d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa288102320 0x7fa28819df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:37.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28fa4d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa28810c2a0 con 0x7fa288100770 2024-05-09T08:30:37.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28fa4d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa28810c120 con 0x7fa288102320 2024-05-09T08:30:37.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28fa4d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa28810c420 con 0x7fa288101960 2024-05-09T08:30:37.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28d7e9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288100770 0x7fa288196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:37.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28d7e9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288100770 0x7fa288196960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48520/0 (socket says 172.21.15.107:48520) 2024-05-09T08:30:37.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28d7e9640 1 -- 172.21.15.107:0/2553248267 learned_addr learned my addr 172.21.15.107:0/2553248267 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:37.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28cfe8640 1 --2- 172.21.15.107:0/2553248267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa288101960 0x7fa288196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28dfea640 1 --2- 172.21.15.107:0/2553248267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa288102320 0x7fa28819df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28d7e9640 1 -- 172.21.15.107:0/2553248267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa288101960 msgr2=0x7fa288196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28d7e9640 1 --2- 172.21.15.107:0/2553248267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa288101960 0x7fa288196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28d7e9640 1 -- 172.21.15.107:0/2553248267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa288102320 msgr2=0x7fa28819df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28d7e9640 1 --2- 172.21.15.107:0/2553248267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa288102320 0x7fa28819df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28d7e9640 1 -- 172.21.15.107:0/2553248267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa28819e580 con 0x7fa288100770 2024-05-09T08:30:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28cfe8640 1 --2- 172.21.15.107:0/2553248267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa288101960 0x7fa288196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:30:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28dfea640 1 --2- 172.21.15.107:0/2553248267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa288102320 0x7fa28819df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:30:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28d7e9640 1 --2- 172.21.15.107:0/2553248267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288100770 0x7fa288196960 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa27800ea80 tx=0x7fa27800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa2867fc640 1 -- 172.21.15.107:0/2553248267 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27800ce40 con 0x7fa288100770 2024-05-09T08:30:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa2867fc640 1 -- 172.21.15.107:0/2553248267 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa278011920 con 0x7fa288100770 2024-05-09T08:30:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa2867fc640 1 -- 172.21.15.107:0/2553248267 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27801aaa0 con 0x7fa288100770 2024-05-09T08:30:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28fa4d640 1 -- 172.21.15.107:0/2553248267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa28819e870 con 0x7fa288100770 2024-05-09T08:30:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.643+0000 7fa28fa4d640 1 -- 172.21.15.107:0/2553248267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa28819edc0 con 0x7fa288100770 2024-05-09T08:30:37.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.644+0000 7fa28fa4d640 1 -- 172.21.15.107:0/2553248267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa288107a90 con 0x7fa288100770 2024-05-09T08:30:37.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.645+0000 7fa2867fc640 1 -- 172.21.15.107:0/2553248267 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fa278011ad0 con 0x7fa288100770 2024-05-09T08:30:37.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.646+0000 7fa2867fc640 1 --2- 172.21.15.107:0/2553248267 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa2700a2b90 0x7fa2700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:37.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.646+0000 7fa28cfe8640 1 --2- 172.21.15.107:0/2553248267 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa2700a2b90 0x7fa2700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:37.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.646+0000 7fa2867fc640 1 -- 172.21.15.107:0/2553248267 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa2780cf230 con 0x7fa288100770 2024-05-09T08:30:37.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.646+0000 7fa28cfe8640 1 --2- 172.21.15.107:0/2553248267 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa2700a2b90 0x7fa2700a5050 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa288197e70 tx=0x7fa27c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:37.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.650+0000 7fa2867fc640 1 -- 172.21.15.107:0/2553248267 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa278016030 con 0x7fa288100770 2024-05-09T08:30:37.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.863+0000 7fa28fa4d640 1 -- 172.21.15.107:0/2553248267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa28802d1c0 con 0x7fa288100770 2024-05-09T08:30:37.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.864+0000 7fa2867fc640 1 -- 172.21.15.107:0/2553248267 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5530 (secure 0 0 0) 0x7fa27809c630 con 0x7fa288100770 2024-05-09T08:30:37.865 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:37.865 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-05-09T08:30:37.867 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 7 2024-05-09T08:30:37.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.866+0000 7fa28fa4d640 1 -- 172.21.15.107:0/2553248267 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa2700a2b90 msgr2=0x7fa2700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:37.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.866+0000 7fa28fa4d640 1 --2- 172.21.15.107:0/2553248267 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa2700a2b90 0x7fa2700a5050 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa288197e70 tx=0x7fa27c005f90 comp rx=0 tx=0).stop 2024-05-09T08:30:37.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.866+0000 7fa28fa4d640 1 -- 172.21.15.107:0/2553248267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288100770 msgr2=0x7fa288196960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:37.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.866+0000 7fa28fa4d640 1 --2- 172.21.15.107:0/2553248267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288100770 0x7fa288196960 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa27800ea80 tx=0x7fa27800ef40 comp rx=0 tx=0).stop 2024-05-09T08:30:37.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.866+0000 7fa28fa4d640 1 -- 172.21.15.107:0/2553248267 shutdown_connections 2024-05-09T08:30:37.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.866+0000 7fa28fa4d640 1 --2- 172.21.15.107:0/2553248267 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa2700a2b90 0x7fa2700a5050 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:37.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.866+0000 7fa28fa4d640 1 --2- 172.21.15.107:0/2553248267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa288102320 0x7fa28819df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:37.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.866+0000 7fa28fa4d640 1 --2- 172.21.15.107:0/2553248267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa288101960 0x7fa288196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:37.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.866+0000 7fa28fa4d640 1 --2- 172.21.15.107:0/2553248267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288100770 0x7fa288196960 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:37.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.866+0000 7fa28fa4d640 1 -- 172.21.15.107:0/2553248267 >> 172.21.15.107:0/2553248267 conn(0x7fa2880fbce0 msgr2=0x7fa288104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:37.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.866+0000 7fa28fa4d640 1 -- 172.21.15.107:0/2553248267 shutdown_connections 2024-05-09T08:30:37.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:37.866+0000 7fa28fa4d640 1 -- 172.21.15.107:0/2553248267 wait complete. 2024-05-09T08:30:38.457 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:38 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.157:6834/843217781,v1:172.21.15.157:6835/843217781] up:boot 2024-05-09T08:30:38.457 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:38 smithi157 ceph-mon[33614]: fsmap 6 up:standby 2024-05-09T08:30:38.457 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:38 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:30:38.457 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:38 smithi157 ceph-mon[33614]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:38.457 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:38 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:38.457 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:38 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:38.457 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:38 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:38.457 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:38 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:38.457 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:38 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:38.457 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:38 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:38.457 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:38 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2553248267' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:38.641 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:38 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.157:6834/843217781,v1:172.21.15.157:6835/843217781] up:boot 2024-05-09T08:30:38.641 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:38 smithi110 ceph-mon[33622]: fsmap 6 up:standby 2024-05-09T08:30:38.642 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:38 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:30:38.642 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:38 smithi110 ceph-mon[33622]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:38.642 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:38 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:38.642 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:38 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:38.642 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:38 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:38.642 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:38 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:38.642 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:38 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:38.642 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:38 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:38.643 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:38 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2553248267' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:38.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:38 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.157:6834/843217781,v1:172.21.15.157:6835/843217781] up:boot 2024-05-09T08:30:38.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:38 smithi107 ceph-mon[30122]: fsmap 6 up:standby 2024-05-09T08:30:38.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:38 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:30:38.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:38 smithi107 ceph-mon[30122]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:38.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:38 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:38.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:38 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:38.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:38 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:38.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:38 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:38.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:38 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:38.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:38 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:38.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:38 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2553248267' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:39.574 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:39 smithi157 ceph-mon[33614]: Deploying daemon mds.c on smithi157 2024-05-09T08:30:39.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:39 smithi107 ceph-mon[30122]: Deploying daemon mds.c on smithi157 2024-05-09T08:30:39.843 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:30:39 smithi157 systemd[1]: Starting Ceph mds.c for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:30:39.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:39 smithi110 ceph-mon[33622]: Deploying daemon mds.c on smithi157 2024-05-09T08:30:39.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:30:39.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.992+0000 7fdaaf53b640 1 -- 172.21.15.107:0/2055988374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa8101460 msgr2=0x7fdaa8108d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:39.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.992+0000 7fdaaf53b640 1 --2- 172.21.15.107:0/2055988374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa8101460 0x7fdaa8108d00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0009cd0 tx=0x7fdaa002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.992+0000 7fdaaf53b640 1 -- 172.21.15.107:0/2055988374 shutdown_connections 2024-05-09T08:30:39.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.992+0000 7fdaaf53b640 1 --2- 172.21.15.107:0/2055988374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa8101460 0x7fdaa8108d00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.992+0000 7fdaaf53b640 1 --2- 172.21.15.107:0/2055988374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdaa8100aa0 0x7fdaa8100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.992+0000 7fdaaf53b640 1 --2- 172.21.15.107:0/2055988374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdaa8101e00 0x7fdaa8102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.992+0000 7fdaaf53b640 1 -- 172.21.15.107:0/2055988374 >> 172.21.15.107:0/2055988374 conn(0x7fdaa80fb510 msgr2=0x7fdaa80fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:39.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.992+0000 7fdaaf53b640 1 -- 172.21.15.107:0/2055988374 shutdown_connections 2024-05-09T08:30:39.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.992+0000 7fdaaf53b640 1 -- 172.21.15.107:0/2055988374 wait complete. 2024-05-09T08:30:39.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaaf53b640 1 Processor -- start 2024-05-09T08:30:39.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaaf53b640 1 -- start start 2024-05-09T08:30:39.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaaf53b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa8100aa0 0x7fdaa8100500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:39.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaaf53b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdaa8101460 0x7fdaa819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:39.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaaf53b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdaa8101e00 0x7fdaa819eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:39.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaaf53b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa810c370 con 0x7fdaa8100aa0 2024-05-09T08:30:39.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaaf53b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa810c1f0 con 0x7fdaa8101460 2024-05-09T08:30:39.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaaf53b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa810c4f0 con 0x7fdaa8101e00 2024-05-09T08:30:39.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaad2d7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa8100aa0 0x7fdaa8100500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:39.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaad2d7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa8100aa0 0x7fdaa8100500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48548/0 (socket says 172.21.15.107:48548) 2024-05-09T08:30:39.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaad2d7640 1 -- 172.21.15.107:0/3122127550 learned_addr learned my addr 172.21.15.107:0/3122127550 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:39.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaadad8640 1 --2- 172.21.15.107:0/3122127550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdaa8101e00 0x7fdaa819eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:39.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaacad6640 1 --2- 172.21.15.107:0/3122127550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdaa8101460 0x7fdaa819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:39.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaad2d7640 1 -- 172.21.15.107:0/3122127550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdaa8101e00 msgr2=0x7fdaa819eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:39.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaad2d7640 1 --2- 172.21.15.107:0/3122127550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdaa8101e00 0x7fdaa819eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaad2d7640 1 -- 172.21.15.107:0/3122127550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdaa8101460 msgr2=0x7fdaa819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:39.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaad2d7640 1 --2- 172.21.15.107:0/3122127550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdaa8101460 0x7fdaa819c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:39.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaad2d7640 1 -- 172.21.15.107:0/3122127550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa819f7c0 con 0x7fdaa8100aa0 2024-05-09T08:30:39.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.993+0000 7fdaadad8640 1 --2- 172.21.15.107:0/3122127550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdaa8101e00 0x7fdaa819eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:30:39.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.994+0000 7fdaad2d7640 1 --2- 172.21.15.107:0/3122127550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa8100aa0 0x7fdaa8100500 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400e9c0 tx=0x7fdaa400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:39.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.994+0000 7fda9a7fc640 1 -- 172.21.15.107:0/3122127550 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa400cd80 con 0x7fdaa8100aa0 2024-05-09T08:30:39.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.994+0000 7fda9a7fc640 1 -- 172.21.15.107:0/3122127550 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdaa40035f0 con 0x7fdaa8100aa0 2024-05-09T08:30:39.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.994+0000 7fdaaf53b640 1 -- 172.21.15.107:0/3122127550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa819fa50 con 0x7fdaa8100aa0 2024-05-09T08:30:39.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.994+0000 7fda9a7fc640 1 -- 172.21.15.107:0/3122127550 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4010690 con 0x7fdaa8100aa0 2024-05-09T08:30:39.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.994+0000 7fdaaf53b640 1 -- 172.21.15.107:0/3122127550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa819fff0 con 0x7fdaa8100aa0 2024-05-09T08:30:39.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.995+0000 7fdaaf53b640 1 -- 172.21.15.107:0/3122127550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaa8107c80 con 0x7fdaa8100aa0 2024-05-09T08:30:39.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.996+0000 7fda9a7fc640 1 -- 172.21.15.107:0/3122127550 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fdaa40037a0 con 0x7fdaa8100aa0 2024-05-09T08:30:39.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.997+0000 7fda9a7fc640 1 --2- 172.21.15.107:0/3122127550 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fda940a2c10 0x7fda940a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:39.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.997+0000 7fdaacad6640 1 --2- 172.21.15.107:0/3122127550 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fda940a2c10 0x7fda940a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:39.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.997+0000 7fda9a7fc640 1 -- 172.21.15.107:0/3122127550 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fdaa40cf0c0 con 0x7fdaa8100aa0 2024-05-09T08:30:39.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:39.997+0000 7fdaacad6640 1 --2- 172.21.15.107:0/3122127550 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fda940a2c10 0x7fda940a50d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8100270 tx=0x7fda9c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:40.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:40.000+0000 7fda9a7fc640 1 -- 172.21.15.107:0/3122127550 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdaa401f340 con 0x7fdaa8100aa0 2024-05-09T08:30:40.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:40.218+0000 7fdaaf53b640 1 -- 172.21.15.107:0/3122127550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdaa802d1c0 con 0x7fdaa8100aa0 2024-05-09T08:30:40.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:40.218+0000 7fda9a7fc640 1 -- 172.21.15.107:0/3122127550 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5530 (secure 0 0 0) 0x7fdaa409c4c0 con 0x7fdaa8100aa0 2024-05-09T08:30:40.219 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:40.219 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-05-09T08:30:40.219 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 7 2024-05-09T08:30:40.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:40.220+0000 7fdaaf53b640 1 -- 172.21.15.107:0/3122127550 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fda940a2c10 msgr2=0x7fda940a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:40.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:40.220+0000 7fdaaf53b640 1 --2- 172.21.15.107:0/3122127550 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fda940a2c10 0x7fda940a50d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8100270 tx=0x7fda9c005f90 comp rx=0 tx=0).stop 2024-05-09T08:30:40.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:40.220+0000 7fdaaf53b640 1 -- 172.21.15.107:0/3122127550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa8100aa0 msgr2=0x7fdaa8100500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:40.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:40.220+0000 7fdaaf53b640 1 --2- 172.21.15.107:0/3122127550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa8100aa0 0x7fdaa8100500 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400e9c0 tx=0x7fdaa400ee80 comp rx=0 tx=0).stop 2024-05-09T08:30:40.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:40.221+0000 7fdaaf53b640 1 -- 172.21.15.107:0/3122127550 shutdown_connections 2024-05-09T08:30:40.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:40.221+0000 7fdaaf53b640 1 --2- 172.21.15.107:0/3122127550 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fda940a2c10 0x7fda940a50d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:40.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:40.221+0000 7fdaaf53b640 1 --2- 172.21.15.107:0/3122127550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdaa8101e00 0x7fdaa819eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:40.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:40.221+0000 7fdaaf53b640 1 --2- 172.21.15.107:0/3122127550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdaa8101460 0x7fdaa819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:40.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:40.221+0000 7fdaaf53b640 1 --2- 172.21.15.107:0/3122127550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa8100aa0 0x7fdaa8100500 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:40.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:40.221+0000 7fdaaf53b640 1 -- 172.21.15.107:0/3122127550 >> 172.21.15.107:0/3122127550 conn(0x7fdaa80fb510 msgr2=0x7fdaa8104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:40.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:40.221+0000 7fdaaf53b640 1 -- 172.21.15.107:0/3122127550 shutdown_connections 2024-05-09T08:30:40.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:40.221+0000 7fdaaf53b640 1 -- 172.21.15.107:0/3122127550 wait complete. 2024-05-09T08:30:40.258 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:30:39 smithi157 podman[64039]: 2024-05-09 08:30:39.887280333 +0000 UTC m=+0.018310959 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:30:40.258 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:30:39 smithi157 podman[64039]: 2024-05-09 08:30:39.997502315 +0000 UTC m=+0.128532935 container create 02e19a76ad1ccd4b735985d83fb6d375bcf51cb4254fcb50ad9ee47186401402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-c, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-09T08:30:40.258 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:30:40 smithi157 podman[64039]: 2024-05-09 08:30:40.259103485 +0000 UTC m=+0.390134105 container init 02e19a76ad1ccd4b735985d83fb6d375bcf51cb4254fcb50ad9ee47186401402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-c, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-09T08:30:40.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:40 smithi107 ceph-mon[30122]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:40.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:40 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3122127550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:40.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:40 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:40.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:40 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:40.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:40 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:40.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:40 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:40.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:40 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:40.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:40 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:40.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:40 smithi157 ceph-mon[33614]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:40.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:40 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3122127550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:40.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:40 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:40.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:40 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:40.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:40 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:40.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:40 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:40.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:40 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:40.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:40 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:40.595 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:30:40 smithi157 podman[64039]: 2024-05-09 08:30:40.261911454 +0000 UTC m=+0.392942075 container start 02e19a76ad1ccd4b735985d83fb6d375bcf51cb4254fcb50ad9ee47186401402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, RELEASE=squid-96805dc) 2024-05-09T08:30:40.595 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:30:40 smithi157 ceph-mds[64058]: -- 172.21.15.157:0/1959393957 <== mon.1 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562dcd1fba40 con 0x562dcd1cf000 2024-05-09T08:30:40.595 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:30:40 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-c[64054]: starting mds.c at 2024-05-09T08:30:40.595 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:30:40 smithi157 bash[64039]: 02e19a76ad1ccd4b735985d83fb6d375bcf51cb4254fcb50ad9ee47186401402 2024-05-09T08:30:40.595 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:30:40 smithi157 systemd[1]: Started Ceph mds.c for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:30:40.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:40 smithi110 ceph-mon[33622]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:40.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:40 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3122127550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:40.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:40 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:40.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:40 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:40.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:40 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:40.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:40 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:40.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:40 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:40.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:40 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:41.752 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:41 smithi110 ceph-mon[33622]: Deploying daemon mds.h on smithi110 2024-05-09T08:30:41.753 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:41 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] up:boot 2024-05-09T08:30:41.753 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:41 smithi110 ceph-mon[33622]: fsmap 7 up:standby 2024-05-09T08:30:41.753 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:41 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:30:41.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:41 smithi107 ceph-mon[30122]: Deploying daemon mds.h on smithi110 2024-05-09T08:30:41.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:41 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] up:boot 2024-05-09T08:30:41.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:41 smithi107 ceph-mon[30122]: fsmap 7 up:standby 2024-05-09T08:30:41.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:41 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:30:41.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:41 smithi157 ceph-mon[33614]: Deploying daemon mds.h on smithi110 2024-05-09T08:30:41.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:41 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] up:boot 2024-05-09T08:30:41.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:41 smithi157 ceph-mon[33614]: fsmap 7 up:standby 2024-05-09T08:30:41.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:41 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:30:42.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:30:42.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.347+0000 7f2225d53640 1 -- 172.21.15.107:0/2044078415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2220104c40 msgr2=0x7f2220107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:42.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.347+0000 7f2225d53640 1 --2- 172.21.15.107:0/2044078415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2220104c40 0x7f2220107030 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2214009900 tx=0x7f221402a410 comp rx=0 tx=0).stop 2024-05-09T08:30:42.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.347+0000 7f2225d53640 1 -- 172.21.15.107:0/2044078415 shutdown_connections 2024-05-09T08:30:42.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.347+0000 7f2225d53640 1 --2- 172.21.15.107:0/2044078415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2220107570 0x7f22201099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:42.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.347+0000 7f2225d53640 1 --2- 172.21.15.107:0/2044078415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2220104c40 0x7f2220107030 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:42.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.347+0000 7f2225d53640 1 --2- 172.21.15.107:0/2044078415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2220102310 0x7f2220104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:42.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.347+0000 7f2225d53640 1 -- 172.21.15.107:0/2044078415 >> 172.21.15.107:0/2044078415 conn(0x7f22200fbd20 msgr2=0x7f22200fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.347+0000 7f2225d53640 1 -- 172.21.15.107:0/2044078415 shutdown_connections 2024-05-09T08:30:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.347+0000 7f2225d53640 1 -- 172.21.15.107:0/2044078415 wait complete. 2024-05-09T08:30:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.348+0000 7f2225d53640 1 Processor -- start 2024-05-09T08:30:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.348+0000 7f2225d53640 1 -- start start 2024-05-09T08:30:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.348+0000 7f2225d53640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2220102310 0x7f2220197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.348+0000 7f2225d53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2220104c40 0x7f2220198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.348+0000 7f2225d53640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2220107570 0x7f222019f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.348+0000 7f2225d53640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f222010d420 con 0x7f2220104c40 2024-05-09T08:30:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.348+0000 7f2225d53640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f222010d2a0 con 0x7f2220107570 2024-05-09T08:30:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.348+0000 7f2225d53640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f222010d5a0 con 0x7f2220102310 2024-05-09T08:30:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.348+0000 7f221effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2220104c40 0x7f2220198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.348+0000 7f221effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2220104c40 0x7f2220198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54854/0 (socket says 172.21.15.107:54854) 2024-05-09T08:30:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.348+0000 7f221effd640 1 -- 172.21.15.107:0/195790108 learned_addr learned my addr 172.21.15.107:0/195790108 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.348+0000 7f221ffff640 1 --2- 172.21.15.107:0/195790108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2220107570 0x7f222019f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.348+0000 7f221f7fe640 1 --2- 172.21.15.107:0/195790108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2220102310 0x7f2220197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.349+0000 7f221effd640 1 -- 172.21.15.107:0/195790108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2220102310 msgr2=0x7f2220197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.349+0000 7f221effd640 1 --2- 172.21.15.107:0/195790108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2220102310 0x7f2220197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.349+0000 7f221effd640 1 -- 172.21.15.107:0/195790108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2220107570 msgr2=0x7f222019f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.349+0000 7f221effd640 1 --2- 172.21.15.107:0/195790108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2220107570 0x7f222019f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.349+0000 7f221effd640 1 -- 172.21.15.107:0/195790108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f222019f770 con 0x7f2220104c40 2024-05-09T08:30:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.349+0000 7f221effd640 1 --2- 172.21.15.107:0/195790108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2220104c40 0x7f2220198090 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f22140314e0 tx=0x7f2214003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.349+0000 7f221cff9640 1 -- 172.21.15.107:0/195790108 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221403f070 con 0x7f2220104c40 2024-05-09T08:30:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.349+0000 7f221cff9640 1 -- 172.21.15.107:0/195790108 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2214031c40 con 0x7f2220104c40 2024-05-09T08:30:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.349+0000 7f2225d53640 1 -- 172.21.15.107:0/195790108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f222019fa00 con 0x7f2220104c40 2024-05-09T08:30:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.349+0000 7f221cff9640 1 -- 172.21.15.107:0/195790108 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2214035030 con 0x7f2220104c40 2024-05-09T08:30:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.349+0000 7f2225d53640 1 -- 172.21.15.107:0/195790108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f222019fef0 con 0x7f2220104c40 2024-05-09T08:30:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.350+0000 7f2225d53640 1 -- 172.21.15.107:0/195790108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22200fd810 con 0x7f2220104c40 2024-05-09T08:30:42.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.351+0000 7f221cff9640 1 -- 172.21.15.107:0/195790108 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f2214042070 con 0x7f2220104c40 2024-05-09T08:30:42.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.352+0000 7f221cff9640 1 --2- 172.21.15.107:0/195790108 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f22080a2b90 0x7f22080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:42.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.352+0000 7f221f7fe640 1 --2- 172.21.15.107:0/195790108 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f22080a2b90 0x7f22080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:42.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.352+0000 7f221cff9640 1 -- 172.21.15.107:0/195790108 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f22140ef310 con 0x7f2220104c40 2024-05-09T08:30:42.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.352+0000 7f221f7fe640 1 --2- 172.21.15.107:0/195790108 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f22080a2b90 0x7f22080a5050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2210005fd0 tx=0x7f221000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:42.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.357+0000 7f221cff9640 1 -- 172.21.15.107:0/195790108 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f22140bc710 con 0x7f2220104c40 2024-05-09T08:30:42.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:42 smithi107 ceph-mon[30122]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:42.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.575+0000 7f2225d53640 1 -- 172.21.15.107:0/195790108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f222002d410 con 0x7f2220104c40 2024-05-09T08:30:42.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.575+0000 7f221cff9640 1 -- 172.21.15.107:0/195790108 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6358 (secure 0 0 0) 0x7f22140bc010 con 0x7f2220104c40 2024-05-09T08:30:42.576 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:42.576 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-05-09T08:30:42.576 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 8 2024-05-09T08:30:42.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.578+0000 7f2225d53640 1 -- 172.21.15.107:0/195790108 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f22080a2b90 msgr2=0x7f22080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:42.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.578+0000 7f2225d53640 1 --2- 172.21.15.107:0/195790108 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f22080a2b90 0x7f22080a5050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2210005fd0 tx=0x7f221000b390 comp rx=0 tx=0).stop 2024-05-09T08:30:42.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.578+0000 7f2225d53640 1 -- 172.21.15.107:0/195790108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2220104c40 msgr2=0x7f2220198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:42.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.578+0000 7f2225d53640 1 --2- 172.21.15.107:0/195790108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2220104c40 0x7f2220198090 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f22140314e0 tx=0x7f2214003c10 comp rx=0 tx=0).stop 2024-05-09T08:30:42.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.578+0000 7f2225d53640 1 -- 172.21.15.107:0/195790108 shutdown_connections 2024-05-09T08:30:42.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.578+0000 7f2225d53640 1 --2- 172.21.15.107:0/195790108 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f22080a2b90 0x7f22080a5050 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:42.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.578+0000 7f2225d53640 1 --2- 172.21.15.107:0/195790108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2220107570 0x7f222019f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:42.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.578+0000 7f2225d53640 1 --2- 172.21.15.107:0/195790108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2220104c40 0x7f2220198090 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:42.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.578+0000 7f2225d53640 1 --2- 172.21.15.107:0/195790108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2220102310 0x7f2220197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:42.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.578+0000 7f2225d53640 1 -- 172.21.15.107:0/195790108 >> 172.21.15.107:0/195790108 conn(0x7f22200fbd20 msgr2=0x7f2220102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:42.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.578+0000 7f2225d53640 1 -- 172.21.15.107:0/195790108 shutdown_connections 2024-05-09T08:30:42.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:42.578+0000 7f2225d53640 1 -- 172.21.15.107:0/195790108 wait complete. 2024-05-09T08:30:42.781 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:42 smithi110 ceph-mon[33622]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:42.781 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:30:42 smithi110 systemd[1]: Starting Ceph mds.h for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:30:42.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:42 smithi157 ceph-mon[33614]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:43.059 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:30:42 smithi110 podman[63465]: 2024-05-09 08:30:42.684901989 +0000 UTC m=+0.017836446 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:30:43.059 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:30:42 smithi110 podman[63465]: 2024-05-09 08:30:42.790661705 +0000 UTC m=+0.123596147 container create a262e31f23219fc51cba0d90f5df685df74b314db73ff1a24744a0959b11540f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-96805dc, ceph=True, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-09T08:30:43.060 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:30:43 smithi110 podman[63465]: 2024-05-09 08:30:43.060720985 +0000 UTC m=+0.393655436 container init a262e31f23219fc51cba0d90f5df685df74b314db73ff1a24744a0959b11540f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-09T08:30:43.371 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:30:43 smithi110 podman[63465]: 2024-05-09 08:30:43.063697687 +0000 UTC m=+0.396632141 container start a262e31f23219fc51cba0d90f5df685df74b314db73ff1a24744a0959b11540f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-h, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-09T08:30:43.371 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:30:43 smithi110 ceph-mds[63484]: -- 172.21.15.110:0/933818818 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ec5d7afa40 con 0x55ec5d782c00 2024-05-09T08:30:43.371 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:30:43 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-h[63480]: starting mds.h at 2024-05-09T08:30:43.371 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:30:43 smithi110 bash[63465]: a262e31f23219fc51cba0d90f5df685df74b314db73ff1a24744a0959b11540f 2024-05-09T08:30:43.371 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:30:43 smithi110 systemd[1]: Started Ceph mds.h for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:30:43.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:43 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/195790108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:43.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:43 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:43.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:43 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:43.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:43 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:43.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:43 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:43.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:43 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:43.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:43 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:43.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:43 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/195790108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:43.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:43 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:43.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:43 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:43.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:43 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:43.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:43 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:43.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:43 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:43.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:43 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:43.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:43 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/195790108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:43.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:43 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:43.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:43 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:43.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:43 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:43.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:43 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:43.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:43 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:43.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:43 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:44.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:30:44.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.704+0000 7f22d89a9640 1 -- 172.21.15.107:0/1964398956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22d4102340 msgr2=0x7f22d4108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:44.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.704+0000 7f22d89a9640 1 --2- 172.21.15.107:0/1964398956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22d4102340 0x7f22d4108bd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009ca0 tx=0x7f22cc02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:30:44.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.704+0000 7f22d89a9640 1 -- 172.21.15.107:0/1964398956 shutdown_connections 2024-05-09T08:30:44.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.704+0000 7f22d89a9640 1 --2- 172.21.15.107:0/1964398956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22d4102340 0x7f22d4108bd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:44.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.704+0000 7f22d89a9640 1 --2- 172.21.15.107:0/1964398956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22d4101980 0x7f22d4101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:44.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.704+0000 7f22d89a9640 1 --2- 172.21.15.107:0/1964398956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22d4100790 0x7f22d4100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:44.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.704+0000 7f22d89a9640 1 -- 172.21.15.107:0/1964398956 >> 172.21.15.107:0/1964398956 conn(0x7f22d40fbd20 msgr2=0x7f22d40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:44.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.704+0000 7f22d89a9640 1 -- 172.21.15.107:0/1964398956 shutdown_connections 2024-05-09T08:30:44.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.704+0000 7f22d89a9640 1 -- 172.21.15.107:0/1964398956 wait complete. 2024-05-09T08:30:44.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d89a9640 1 Processor -- start 2024-05-09T08:30:44.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d89a9640 1 -- start start 2024-05-09T08:30:44.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d89a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22d4100790 0x7f22d4104890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:44.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d89a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22d4101980 0x7f22d4102f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:44.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d89a9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22d4102340 0x7f22d4103460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:44.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d89a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f22d410c2a0 con 0x7f22d4100790 2024-05-09T08:30:44.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d89a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f22d410c120 con 0x7f22d4101980 2024-05-09T08:30:44.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d89a9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f22d410c420 con 0x7f22d4102340 2024-05-09T08:30:44.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22d4100790 0x7f22d4104890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:44.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22d4100790 0x7f22d4104890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54864/0 (socket says 172.21.15.107:54864) 2024-05-09T08:30:44.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d259c640 1 -- 172.21.15.107:0/3819792839 learned_addr learned my addr 172.21.15.107:0/3819792839 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:44.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d1d9b640 1 --2- 172.21.15.107:0/3819792839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22d4101980 0x7f22d4102f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:44.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d2d9d640 1 --2- 172.21.15.107:0/3819792839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22d4102340 0x7f22d4103460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:44.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d259c640 1 -- 172.21.15.107:0/3819792839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22d4102340 msgr2=0x7f22d4103460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:44.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d259c640 1 --2- 172.21.15.107:0/3819792839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22d4102340 0x7f22d4103460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:44.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d259c640 1 -- 172.21.15.107:0/3819792839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22d4101980 msgr2=0x7f22d4102f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:44.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d259c640 1 --2- 172.21.15.107:0/3819792839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22d4101980 0x7f22d4102f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:44.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.705+0000 7f22d259c640 1 -- 172.21.15.107:0/3819792839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d4103d20 con 0x7f22d4100790 2024-05-09T08:30:44.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.706+0000 7f22d259c640 1 --2- 172.21.15.107:0/3819792839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22d4100790 0x7f22d4104890 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f22c400d770 tx=0x7f22c400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:44.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.706+0000 7f22cb7fe640 1 -- 172.21.15.107:0/3819792839 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c4017070 con 0x7f22d4100790 2024-05-09T08:30:44.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.706+0000 7f22cb7fe640 1 -- 172.21.15.107:0/3819792839 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22c4009c30 con 0x7f22d4100790 2024-05-09T08:30:44.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.706+0000 7f22d89a9640 1 -- 172.21.15.107:0/3819792839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d41a1ba0 con 0x7f22d4100790 2024-05-09T08:30:44.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.706+0000 7f22cb7fe640 1 -- 172.21.15.107:0/3819792839 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c4003d00 con 0x7f22d4100790 2024-05-09T08:30:44.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.706+0000 7f22d89a9640 1 -- 172.21.15.107:0/3819792839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d41a20c0 con 0x7f22d4100790 2024-05-09T08:30:44.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.707+0000 7f22d89a9640 1 -- 172.21.15.107:0/3819792839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22d41a2290 con 0x7f22d4100790 2024-05-09T08:30:44.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.708+0000 7f22cb7fe640 1 -- 172.21.15.107:0/3819792839 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f22c4003ea0 con 0x7f22d4100790 2024-05-09T08:30:44.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.709+0000 7f22cb7fe640 1 --2- 172.21.15.107:0/3819792839 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f22c00a2b90 0x7f22c00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:44.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.709+0000 7f22d1d9b640 1 --2- 172.21.15.107:0/3819792839 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f22c00a2b90 0x7f22c00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:44.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.709+0000 7f22cb7fe640 1 -- 172.21.15.107:0/3819792839 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f22c40ce7f0 con 0x7f22d4100790 2024-05-09T08:30:44.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.709+0000 7f22d1d9b640 1 --2- 172.21.15.107:0/3819792839 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f22c00a2b90 0x7f22c00a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f22d4104600 tx=0x7f22bc00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:44.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.712+0000 7f22cb7fe640 1 -- 172.21.15.107:0/3819792839 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f22c4009de0 con 0x7f22d4100790 2024-05-09T08:30:44.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:44 smithi107 ceph-mon[30122]: Deploying daemon mds.i on smithi157 2024-05-09T08:30:44.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:44 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.110:6836/624105719,v1:172.21.15.110:6837/624105719] up:boot 2024-05-09T08:30:44.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:44 smithi107 ceph-mon[30122]: fsmap 8 up:standby 2024-05-09T08:30:44.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:44 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:30:44.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:44 smithi107 ceph-mon[30122]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:44.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:44 smithi157 ceph-mon[33614]: Deploying daemon mds.i on smithi157 2024-05-09T08:30:44.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:44 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.110:6836/624105719,v1:172.21.15.110:6837/624105719] up:boot 2024-05-09T08:30:44.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:44 smithi157 ceph-mon[33614]: fsmap 8 up:standby 2024-05-09T08:30:44.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:44 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:30:44.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:44 smithi157 ceph-mon[33614]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:44.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:44 smithi110 ceph-mon[33622]: Deploying daemon mds.i on smithi157 2024-05-09T08:30:44.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:44 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.110:6836/624105719,v1:172.21.15.110:6837/624105719] up:boot 2024-05-09T08:30:44.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:44 smithi110 ceph-mon[33622]: fsmap 8 up:standby 2024-05-09T08:30:44.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:44 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:30:44.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:44 smithi110 ceph-mon[33622]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:44.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.925+0000 7f22d89a9640 1 -- 172.21.15.107:0/3819792839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22d41a2620 con 0x7f22d4100790 2024-05-09T08:30:44.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.926+0000 7f22cb7fe640 1 -- 172.21.15.107:0/3819792839 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7183 (secure 0 0 0) 0x7f22c409b5a0 con 0x7f22d4100790 2024-05-09T08:30:44.926 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:44.927 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-05-09T08:30:44.927 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 9 2024-05-09T08:30:44.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.928+0000 7f22d89a9640 1 -- 172.21.15.107:0/3819792839 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f22c00a2b90 msgr2=0x7f22c00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:44.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.928+0000 7f22d89a9640 1 --2- 172.21.15.107:0/3819792839 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f22c00a2b90 0x7f22c00a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f22d4104600 tx=0x7f22bc00b390 comp rx=0 tx=0).stop 2024-05-09T08:30:44.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.928+0000 7f22d89a9640 1 -- 172.21.15.107:0/3819792839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22d4100790 msgr2=0x7f22d4104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.928+0000 7f22d89a9640 1 --2- 172.21.15.107:0/3819792839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22d4100790 0x7f22d4104890 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f22c400d770 tx=0x7f22c400dc30 comp rx=0 tx=0).stop 2024-05-09T08:30:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.928+0000 7f22d89a9640 1 -- 172.21.15.107:0/3819792839 shutdown_connections 2024-05-09T08:30:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.928+0000 7f22d89a9640 1 --2- 172.21.15.107:0/3819792839 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f22c00a2b90 0x7f22c00a5050 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.928+0000 7f22d89a9640 1 --2- 172.21.15.107:0/3819792839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22d4102340 0x7f22d4103460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.928+0000 7f22d89a9640 1 --2- 172.21.15.107:0/3819792839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22d4101980 0x7f22d4102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.928+0000 7f22d89a9640 1 --2- 172.21.15.107:0/3819792839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22d4100790 0x7f22d4104890 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.928+0000 7f22d89a9640 1 -- 172.21.15.107:0/3819792839 >> 172.21.15.107:0/3819792839 conn(0x7f22d40fbd20 msgr2=0x7f22d4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.928+0000 7f22d89a9640 1 -- 172.21.15.107:0/3819792839 shutdown_connections 2024-05-09T08:30:44.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:44.928+0000 7f22d89a9640 1 -- 172.21.15.107:0/3819792839 wait complete. 2024-05-09T08:30:45.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:45 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3819792839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:45.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:45 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3819792839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:45.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:45 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3819792839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:45.913 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:30:45 smithi157 podman[64345]: 2024-05-09 08:30:45.891493297 +0000 UTC m=+0.360009759 container start 60aa0a0934a421f4e376727412f162b148d2c9645c1cb16ef92fea95bff8acbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-i, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-09T08:30:45.914 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:30:45 smithi157 ceph-mds[64364]: -- 172.21.15.157:0/969933981 <== mon.1 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5601dab79a40 con 0x5601dab4d000 2024-05-09T08:30:46.343 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:30:45 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-i[64360]: starting mds.i at 2024-05-09T08:30:46.344 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:30:45 smithi157 bash[64345]: 60aa0a0934a421f4e376727412f162b148d2c9645c1cb16ef92fea95bff8acbb 2024-05-09T08:30:46.344 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:30:45 smithi157 systemd[1]: Started Ceph mds.i for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:30:46.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:46 smithi107 ceph-mon[30122]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:46.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:46.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:46.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:46.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:46.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:46.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:46 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:46.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:46 smithi157 ceph-mon[33614]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:46.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:46.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:46.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:46.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:46.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:46.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:46 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:46.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:46 smithi110 ceph-mon[33622]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:46.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:46.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:46.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:46.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:46.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:46.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:46 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:46.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:30:47.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.050+0000 7fd555563640 1 -- 172.21.15.107:0/1073491181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5501034c0 msgr2=0x7fd550109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:47.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.050+0000 7fd555563640 1 --2- 172.21.15.107:0/1073491181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5501034c0 0x7fd550109d50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd548009cd0 tx=0x7fd54802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:30:47.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.050+0000 7fd555563640 1 -- 172.21.15.107:0/1073491181 shutdown_connections 2024-05-09T08:30:47.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.050+0000 7fd555563640 1 --2- 172.21.15.107:0/1073491181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5501034c0 0x7fd550109d50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:47.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.050+0000 7fd555563640 1 --2- 172.21.15.107:0/1073491181 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd550102b00 0x7fd550102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:47.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.050+0000 7fd555563640 1 --2- 172.21.15.107:0/1073491181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5500ffb30 0x7fd5500fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:47.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.050+0000 7fd555563640 1 -- 172.21.15.107:0/1073491181 >> 172.21.15.107:0/1073491181 conn(0x7fd5500fb460 msgr2=0x7fd5500fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:47.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.050+0000 7fd555563640 1 -- 172.21.15.107:0/1073491181 shutdown_connections 2024-05-09T08:30:47.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.050+0000 7fd555563640 1 -- 172.21.15.107:0/1073491181 wait complete. 2024-05-09T08:30:47.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.051+0000 7fd555563640 1 Processor -- start 2024-05-09T08:30:47.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.051+0000 7fd555563640 1 -- start start 2024-05-09T08:30:47.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.051+0000 7fd555563640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd5500ffb30 0x7fd550102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:47.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.051+0000 7fd555563640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd550102b00 0x7fd550100ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:47.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.051+0000 7fd555563640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5501034c0 0x7fd550100ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:47.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.051+0000 7fd555563640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd55010d3c0 con 0x7fd550102b00 2024-05-09T08:30:47.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.051+0000 7fd555563640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd55010d240 con 0x7fd5501034c0 2024-05-09T08:30:47.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.051+0000 7fd555563640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd55010d540 con 0x7fd5500ffb30 2024-05-09T08:30:47.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.051+0000 7fd54e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd550102b00 0x7fd550100ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:47.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.051+0000 7fd54e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd550102b00 0x7fd550100ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54872/0 (socket says 172.21.15.107:54872) 2024-05-09T08:30:47.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.051+0000 7fd54e7fc640 1 -- 172.21.15.107:0/4160591547 learned_addr learned my addr 172.21.15.107:0/4160591547 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:47.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.051+0000 7fd54f7fe640 1 --2- 172.21.15.107:0/4160591547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5501034c0 0x7fd550100ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:47.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.051+0000 7fd54effd640 1 --2- 172.21.15.107:0/4160591547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd5500ffb30 0x7fd550102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:47.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.052+0000 7fd54e7fc640 1 -- 172.21.15.107:0/4160591547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd5500ffb30 msgr2=0x7fd550102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:47.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.052+0000 7fd54e7fc640 1 --2- 172.21.15.107:0/4160591547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd5500ffb30 0x7fd550102470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:47.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.052+0000 7fd54e7fc640 1 -- 172.21.15.107:0/4160591547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5501034c0 msgr2=0x7fd550100ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:47.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.052+0000 7fd54e7fc640 1 --2- 172.21.15.107:0/4160591547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5501034c0 0x7fd550100ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:47.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.052+0000 7fd54e7fc640 1 -- 172.21.15.107:0/4160591547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd550101880 con 0x7fd550102b00 2024-05-09T08:30:47.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.052+0000 7fd54e7fc640 1 --2- 172.21.15.107:0/4160591547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd550102b00 0x7fd550100ab0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd54400c970 tx=0x7fd54400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:47.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.052+0000 7fd537fff640 1 -- 172.21.15.107:0/4160591547 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd544007bb0 con 0x7fd550102b00 2024-05-09T08:30:47.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.052+0000 7fd555563640 1 -- 172.21.15.107:0/4160591547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5501a1bc0 con 0x7fd550102b00 2024-05-09T08:30:47.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.052+0000 7fd537fff640 1 -- 172.21.15.107:0/4160591547 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd544007d50 con 0x7fd550102b00 2024-05-09T08:30:47.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.052+0000 7fd537fff640 1 -- 172.21.15.107:0/4160591547 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd54400f450 con 0x7fd550102b00 2024-05-09T08:30:47.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.052+0000 7fd555563640 1 -- 172.21.15.107:0/4160591547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5501a2110 con 0x7fd550102b00 2024-05-09T08:30:47.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.053+0000 7fd555563640 1 -- 172.21.15.107:0/4160591547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd550108cb0 con 0x7fd550102b00 2024-05-09T08:30:47.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.054+0000 7fd537fff640 1 -- 172.21.15.107:0/4160591547 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd544004f20 con 0x7fd550102b00 2024-05-09T08:30:47.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.055+0000 7fd537fff640 1 --2- 172.21.15.107:0/4160591547 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd5380a2b90 0x7fd5380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:47.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.056+0000 7fd54effd640 1 --2- 172.21.15.107:0/4160591547 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd5380a2b90 0x7fd5380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:47.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.056+0000 7fd537fff640 1 -- 172.21.15.107:0/4160591547 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd5440ce8d0 con 0x7fd550102b00 2024-05-09T08:30:47.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.056+0000 7fd54effd640 1 --2- 172.21.15.107:0/4160591547 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd5380a2b90 0x7fd5380a5050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd5500687c0 tx=0x7fd54000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:47.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.060+0000 7fd537fff640 1 -- 172.21.15.107:0/4160591547 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd544014030 con 0x7fd550102b00 2024-05-09T08:30:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.284+0000 7fd555563640 1 -- 172.21.15.107:0/4160591547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd550062430 con 0x7fd550102b00 2024-05-09T08:30:47.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.284+0000 7fd537fff640 1 -- 172.21.15.107:0/4160591547 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8013 (secure 0 0 0) 0x7fd54409b680 con 0x7fd550102b00 2024-05-09T08:30:47.284 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:47.284 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-09T08:30:47.284 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 10 2024-05-09T08:30:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.286+0000 7fd555563640 1 -- 172.21.15.107:0/4160591547 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd5380a2b90 msgr2=0x7fd5380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.286+0000 7fd555563640 1 --2- 172.21.15.107:0/4160591547 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd5380a2b90 0x7fd5380a5050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd5500687c0 tx=0x7fd54000a2d0 comp rx=0 tx=0).stop 2024-05-09T08:30:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.286+0000 7fd555563640 1 -- 172.21.15.107:0/4160591547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd550102b00 msgr2=0x7fd550100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.286+0000 7fd555563640 1 --2- 172.21.15.107:0/4160591547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd550102b00 0x7fd550100ab0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd54400c970 tx=0x7fd54400ce30 comp rx=0 tx=0).stop 2024-05-09T08:30:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.287+0000 7fd555563640 1 -- 172.21.15.107:0/4160591547 shutdown_connections 2024-05-09T08:30:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.287+0000 7fd555563640 1 --2- 172.21.15.107:0/4160591547 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd5380a2b90 0x7fd5380a5050 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.287+0000 7fd555563640 1 --2- 172.21.15.107:0/4160591547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5501034c0 0x7fd550100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.287+0000 7fd555563640 1 --2- 172.21.15.107:0/4160591547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd550102b00 0x7fd550100ab0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:47.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.287+0000 7fd555563640 1 --2- 172.21.15.107:0/4160591547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd5500ffb30 0x7fd550102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:47.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.287+0000 7fd555563640 1 -- 172.21.15.107:0/4160591547 >> 172.21.15.107:0/4160591547 conn(0x7fd5500fb460 msgr2=0x7fd550105e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:47.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.287+0000 7fd555563640 1 -- 172.21.15.107:0/4160591547 shutdown_connections 2024-05-09T08:30:47.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:47.287+0000 7fd555563640 1 -- 172.21.15.107:0/4160591547 wait complete. 2024-05-09T08:30:47.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:47 smithi107 ceph-mon[30122]: Deploying daemon mds.g on smithi107 2024-05-09T08:30:47.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:47 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] up:boot 2024-05-09T08:30:47.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:47 smithi107 ceph-mon[30122]: fsmap 9 up:standby 2024-05-09T08:30:47.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:47 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:30:47.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:47 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/4160591547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:47.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:47 smithi157 ceph-mon[33614]: Deploying daemon mds.g on smithi107 2024-05-09T08:30:47.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:47 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] up:boot 2024-05-09T08:30:47.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:47 smithi157 ceph-mon[33614]: fsmap 9 up:standby 2024-05-09T08:30:47.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:47 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:30:47.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:47 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/4160591547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:47.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:47 smithi110 ceph-mon[33622]: Deploying daemon mds.g on smithi107 2024-05-09T08:30:47.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:47 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] up:boot 2024-05-09T08:30:47.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:47 smithi110 ceph-mon[33622]: fsmap 9 up:standby 2024-05-09T08:30:47.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:47 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:30:47.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:47 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/4160591547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:48.753 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:48 smithi107 ceph-mon[30122]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:48.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:48 smithi157 ceph-mon[33614]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:48.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:48 smithi110 ceph-mon[33622]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:49.029 INFO:journalctl@ceph.mds.g.smithi107.stdout:May 09 08:30:48 smithi107 systemd[1]: Started Ceph mds.g for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:30:49.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:30:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.405+0000 7fbbd2da9640 1 -- 172.21.15.107:0/3143142354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbcc100770 msgr2=0x7fbbcc100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.405+0000 7fbbd2da9640 1 --2- 172.21.15.107:0/3143142354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbcc100770 0x7fbbcc100b90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0009900 tx=0x7fbbc002a410 comp rx=0 tx=0).stop 2024-05-09T08:30:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.405+0000 7fbbd2da9640 1 -- 172.21.15.107:0/3143142354 shutdown_connections 2024-05-09T08:30:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.405+0000 7fbbd2da9640 1 --2- 172.21.15.107:0/3143142354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbcc102320 0x7fbbcc108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.405+0000 7fbbd2da9640 1 --2- 172.21.15.107:0/3143142354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbcc101960 0x7fbbcc101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.405+0000 7fbbd2da9640 1 --2- 172.21.15.107:0/3143142354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbcc100770 0x7fbbcc100b90 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.405+0000 7fbbd2da9640 1 -- 172.21.15.107:0/3143142354 >> 172.21.15.107:0/3143142354 conn(0x7fbbcc0fbce0 msgr2=0x7fbbcc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.405+0000 7fbbd2da9640 1 -- 172.21.15.107:0/3143142354 shutdown_connections 2024-05-09T08:30:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.405+0000 7fbbd2da9640 1 -- 172.21.15.107:0/3143142354 wait complete. 2024-05-09T08:30:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd2da9640 1 Processor -- start 2024-05-09T08:30:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd2da9640 1 -- start start 2024-05-09T08:30:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd2da9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbcc100770 0x7fbbcc197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd2da9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbcc101960 0x7fbbcc1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd2da9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbcc102320 0x7fbbcc19f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd2da9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc10d480 con 0x7fbbcc102320 2024-05-09T08:30:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd2da9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc10d300 con 0x7fbbcc100770 2024-05-09T08:30:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd2da9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc10d600 con 0x7fbbcc101960 2024-05-09T08:30:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd1346640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbcc102320 0x7fbbcc19f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd1346640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbcc102320 0x7fbbcc19f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54906/0 (socket says 172.21.15.107:54906) 2024-05-09T08:30:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd1346640 1 -- 172.21.15.107:0/3313507073 learned_addr learned my addr 172.21.15.107:0/3313507073 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbcbfff640 1 --2- 172.21.15.107:0/3313507073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbcc101960 0x7fbbcc1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd0b45640 1 --2- 172.21.15.107:0/3313507073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbcc100770 0x7fbbcc197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd1346640 1 -- 172.21.15.107:0/3313507073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbcc101960 msgr2=0x7fbbcc1981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd1346640 1 --2- 172.21.15.107:0/3313507073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbcc101960 0x7fbbcc1981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd1346640 1 -- 172.21.15.107:0/3313507073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbcc100770 msgr2=0x7fbbcc197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd1346640 1 --2- 172.21.15.107:0/3313507073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbcc100770 0x7fbbcc197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd1346640 1 -- 172.21.15.107:0/3313507073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbcc19f8b0 con 0x7fbbcc102320 2024-05-09T08:30:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd0b45640 1 --2- 172.21.15.107:0/3313507073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbcc100770 0x7fbbcc197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:30:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.406+0000 7fbbd1346640 1 --2- 172.21.15.107:0/3313507073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbcc102320 0x7fbbcc19f210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbbc400b970 tx=0x7fbbc400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.407+0000 7fbbc9ffb640 1 -- 172.21.15.107:0/3313507073 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc4019070 con 0x7fbbcc102320 2024-05-09T08:30:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.407+0000 7fbbd2da9640 1 -- 172.21.15.107:0/3313507073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbcc19fba0 con 0x7fbbcc102320 2024-05-09T08:30:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.407+0000 7fbbc9ffb640 1 -- 172.21.15.107:0/3313507073 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbc4014440 con 0x7fbbcc102320 2024-05-09T08:30:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.407+0000 7fbbc9ffb640 1 -- 172.21.15.107:0/3313507073 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc40075d0 con 0x7fbbcc102320 2024-05-09T08:30:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.407+0000 7fbbd2da9640 1 -- 172.21.15.107:0/3313507073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbcc1a01c0 con 0x7fbbcc102320 2024-05-09T08:30:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.408+0000 7fbbd2da9640 1 -- 172.21.15.107:0/3313507073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbcc107bd0 con 0x7fbbcc102320 2024-05-09T08:30:49.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.409+0000 7fbbc9ffb640 1 -- 172.21.15.107:0/3313507073 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fbbc401c070 con 0x7fbbcc102320 2024-05-09T08:30:49.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.409+0000 7fbbc9ffb640 1 --2- 172.21.15.107:0/3313507073 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbbbc0a2b90 0x7fbbbc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:49.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.409+0000 7fbbc9ffb640 1 -- 172.21.15.107:0/3313507073 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbbc40ce280 con 0x7fbbcc102320 2024-05-09T08:30:49.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.409+0000 7fbbd0b45640 1 --2- 172.21.15.107:0/3313507073 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbbbc0a2b90 0x7fbbbc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.410+0000 7fbbd0b45640 1 --2- 172.21.15.107:0/3313507073 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbbbc0a2b90 0x7fbbbc0a5050 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0000c00 tx=0x7fbbc00314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:49.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.414+0000 7fbbc9ffb640 1 -- 172.21.15.107:0/3313507073 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbbc409b5e0 con 0x7fbbcc102320 2024-05-09T08:30:49.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.653+0000 7fbbd2da9640 1 -- 172.21.15.107:0/3313507073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbbcc02d1c0 con 0x7fbbcc102320 2024-05-09T08:30:49.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.653+0000 7fbbc9ffb640 1 -- 172.21.15.107:0/3313507073 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8842 (secure 0 0 0) 0x7fbbc409aee0 con 0x7fbbcc102320 2024-05-09T08:30:49.653 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:49.653 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-09T08:30:49.653 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 11 2024-05-09T08:30:49.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.655+0000 7fbbd2da9640 1 -- 172.21.15.107:0/3313507073 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbbbc0a2b90 msgr2=0x7fbbbc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:49.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.655+0000 7fbbd2da9640 1 --2- 172.21.15.107:0/3313507073 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbbbc0a2b90 0x7fbbbc0a5050 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0000c00 tx=0x7fbbc00314a0 comp rx=0 tx=0).stop 2024-05-09T08:30:49.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.655+0000 7fbbd2da9640 1 -- 172.21.15.107:0/3313507073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbcc102320 msgr2=0x7fbbcc19f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:49.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.655+0000 7fbbd2da9640 1 --2- 172.21.15.107:0/3313507073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbcc102320 0x7fbbcc19f210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbbc400b970 tx=0x7fbbc400be30 comp rx=0 tx=0).stop 2024-05-09T08:30:49.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.655+0000 7fbbd2da9640 1 -- 172.21.15.107:0/3313507073 shutdown_connections 2024-05-09T08:30:49.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.655+0000 7fbbd2da9640 1 --2- 172.21.15.107:0/3313507073 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbbbc0a2b90 0x7fbbbc0a5050 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:49.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.655+0000 7fbbd2da9640 1 --2- 172.21.15.107:0/3313507073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbcc102320 0x7fbbcc19f210 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:49.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.655+0000 7fbbd2da9640 1 --2- 172.21.15.107:0/3313507073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbcc101960 0x7fbbcc1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:49.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.655+0000 7fbbd2da9640 1 --2- 172.21.15.107:0/3313507073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbcc100770 0x7fbbcc197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:49.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.655+0000 7fbbd2da9640 1 -- 172.21.15.107:0/3313507073 >> 172.21.15.107:0/3313507073 conn(0x7fbbcc0fbce0 msgr2=0x7fbbcc104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:49.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.655+0000 7fbbd2da9640 1 -- 172.21.15.107:0/3313507073 shutdown_connections 2024-05-09T08:30:49.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:49.656+0000 7fbbd2da9640 1 -- 172.21.15.107:0/3313507073 wait complete. 2024-05-09T08:30:50.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:50 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:50.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:50 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:50.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:50 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:50.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:50 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:50.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:50 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:50.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:50 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:50.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:50 smithi107 ceph-mon[30122]: Deploying daemon mds.a on smithi107 2024-05-09T08:30:50.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:50 smithi107 ceph-mon[30122]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:50.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:50 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.107:6838/2088211251,v1:172.21.15.107:6839/2088211251] up:boot 2024-05-09T08:30:50.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:50 smithi107 ceph-mon[30122]: fsmap 10 up:standby 2024-05-09T08:30:50.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:50 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:30:50.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:50 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3313507073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:50 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:50 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:50 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:50 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:50 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:50 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:50 smithi157 ceph-mon[33614]: Deploying daemon mds.a on smithi107 2024-05-09T08:30:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:50 smithi157 ceph-mon[33614]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:50 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.107:6838/2088211251,v1:172.21.15.107:6839/2088211251] up:boot 2024-05-09T08:30:50.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:50 smithi157 ceph-mon[33614]: fsmap 10 up:standby 2024-05-09T08:30:50.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:50 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:30:50.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:50 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3313507073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:50 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:50 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:50 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:50 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:50 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:50 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:50 smithi110 ceph-mon[33622]: Deploying daemon mds.a on smithi107 2024-05-09T08:30:50.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:50 smithi110 ceph-mon[33622]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:50.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:50 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.107:6838/2088211251,v1:172.21.15.107:6839/2088211251] up:boot 2024-05-09T08:30:50.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:50 smithi110 ceph-mon[33622]: fsmap 10 up:standby 2024-05-09T08:30:50.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:50 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:30:50.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:50 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3313507073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:51.279 INFO:journalctl@ceph.mds.a.smithi107.stdout:May 09 08:30:51 smithi107 systemd[1]: Starting Ceph mds.a for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:30:51.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:30:51.691 INFO:journalctl@ceph.mds.a.smithi107.stdout:May 09 08:30:51 smithi107 podman[69413]: 2024-05-09 08:30:51.339644164 +0000 UTC m=+0.018673930 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:30:51.691 INFO:journalctl@ceph.mds.a.smithi107.stdout:May 09 08:30:51 smithi107 podman[69413]: 2024-05-09 08:30:51.438501038 +0000 UTC m=+0.117530788 container create de4ae376929bbf0b76f1a05a35d3935a0f3b362c1ab78bf64353332a50c1415b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507) 2024-05-09T08:30:51.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.776+0000 7fc8abc2c640 1 -- 172.21.15.107:0/32394175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8a4100770 msgr2=0x7fc8a4100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:51.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.776+0000 7fc8abc2c640 1 --2- 172.21.15.107:0/32394175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8a4100770 0x7fc8a4100b90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc894009900 tx=0x7fc89402a410 comp rx=0 tx=0).stop 2024-05-09T08:30:51.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.776+0000 7fc8abc2c640 1 -- 172.21.15.107:0/32394175 shutdown_connections 2024-05-09T08:30:51.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.776+0000 7fc8abc2c640 1 --2- 172.21.15.107:0/32394175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8a4102320 0x7fc8a4108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:51.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.776+0000 7fc8abc2c640 1 --2- 172.21.15.107:0/32394175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8a4101960 0x7fc8a4101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:51.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.776+0000 7fc8abc2c640 1 --2- 172.21.15.107:0/32394175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8a4100770 0x7fc8a4100b90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:51.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.776+0000 7fc8abc2c640 1 -- 172.21.15.107:0/32394175 >> 172.21.15.107:0/32394175 conn(0x7fc8a40fbce0 msgr2=0x7fc8a40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:51.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.776+0000 7fc8abc2c640 1 -- 172.21.15.107:0/32394175 shutdown_connections 2024-05-09T08:30:51.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.776+0000 7fc8abc2c640 1 -- 172.21.15.107:0/32394175 wait complete. 2024-05-09T08:30:51.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8abc2c640 1 Processor -- start 2024-05-09T08:30:51.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8abc2c640 1 -- start start 2024-05-09T08:30:51.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8abc2c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8a4100770 0x7fc8a4104890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:51.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8abc2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8a4101960 0x7fc8a4102ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:51.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8abc2c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8a4102320 0x7fc8a4103410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:51.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8abc2c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a410c290 con 0x7fc8a4101960 2024-05-09T08:30:51.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8abc2c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a410c110 con 0x7fc8a4102320 2024-05-09T08:30:51.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8abc2c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a410c410 con 0x7fc8a4100770 2024-05-09T08:30:51.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8a91c7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8a4101960 0x7fc8a4102ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:51.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8a91c7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8a4101960 0x7fc8a4102ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40402/0 (socket says 172.21.15.107:40402) 2024-05-09T08:30:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8a91c7640 1 -- 172.21.15.107:0/2706024131 learned_addr learned my addr 172.21.15.107:0/2706024131 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8a99c8640 1 --2- 172.21.15.107:0/2706024131 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8a4100770 0x7fc8a4104890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8aa1c9640 1 --2- 172.21.15.107:0/2706024131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8a4102320 0x7fc8a4103410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8a91c7640 1 -- 172.21.15.107:0/2706024131 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8a4100770 msgr2=0x7fc8a4104890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8a91c7640 1 --2- 172.21.15.107:0/2706024131 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8a4100770 0x7fc8a4104890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8a91c7640 1 -- 172.21.15.107:0/2706024131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8a4102320 msgr2=0x7fc8a4103410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.777+0000 7fc8a91c7640 1 --2- 172.21.15.107:0/2706024131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8a4102320 0x7fc8a4103410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.778+0000 7fc8a91c7640 1 -- 172.21.15.107:0/2706024131 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8a4103ca0 con 0x7fc8a4101960 2024-05-09T08:30:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.778+0000 7fc8aa1c9640 1 --2- 172.21.15.107:0/2706024131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8a4102320 0x7fc8a4103410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:30:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.778+0000 7fc8a91c7640 1 --2- 172.21.15.107:0/2706024131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8a4101960 0x7fc8a4102ed0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc8a000c970 tx=0x7fc8a000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.778+0000 7fc89affd640 1 -- 172.21.15.107:0/2706024131 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a0007bb0 con 0x7fc8a4101960 2024-05-09T08:30:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.778+0000 7fc89affd640 1 -- 172.21.15.107:0/2706024131 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc8a0007d50 con 0x7fc8a4101960 2024-05-09T08:30:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.778+0000 7fc8abc2c640 1 -- 172.21.15.107:0/2706024131 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8a41a1c30 con 0x7fc8a4101960 2024-05-09T08:30:51.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.778+0000 7fc89affd640 1 -- 172.21.15.107:0/2706024131 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a000f450 con 0x7fc8a4101960 2024-05-09T08:30:51.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.778+0000 7fc8abc2c640 1 -- 172.21.15.107:0/2706024131 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8a41a2130 con 0x7fc8a4101960 2024-05-09T08:30:51.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.779+0000 7fc8abc2c640 1 -- 172.21.15.107:0/2706024131 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8a41a2360 con 0x7fc8a4101960 2024-05-09T08:30:51.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.780+0000 7fc89affd640 1 -- 172.21.15.107:0/2706024131 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc8a0004f20 con 0x7fc8a4101960 2024-05-09T08:30:51.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.781+0000 7fc89affd640 1 --2- 172.21.15.107:0/2706024131 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc8900a2b90 0x7fc8900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:51.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.781+0000 7fc8a99c8640 1 --2- 172.21.15.107:0/2706024131 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc8900a2b90 0x7fc8900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:51.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.781+0000 7fc89affd640 1 -- 172.21.15.107:0/2706024131 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc8a00cea30 con 0x7fc8a4101960 2024-05-09T08:30:51.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.781+0000 7fc8a99c8640 1 --2- 172.21.15.107:0/2706024131 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc8900a2b90 0x7fc8900a5050 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc894000c00 tx=0x7fc8940314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:51.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:51.785+0000 7fc89affd640 1 -- 172.21.15.107:0/2706024131 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc8a0014030 con 0x7fc8a4101960 2024-05-09T08:30:52.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:52.011+0000 7fc8abc2c640 1 -- 172.21.15.107:0/2706024131 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8a41a26f0 con 0x7fc8a4101960 2024-05-09T08:30:52.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:52.011+0000 7fc89affd640 1 -- 172.21.15.107:0/2706024131 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8842 (secure 0 0 0) 0x7fc8a009be30 con 0x7fc8a4101960 2024-05-09T08:30:52.011 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:52.011 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-09T08:30:52.011 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 11 2024-05-09T08:30:52.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:52.014+0000 7fc8abc2c640 1 -- 172.21.15.107:0/2706024131 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc8900a2b90 msgr2=0x7fc8900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:52.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:52.014+0000 7fc8abc2c640 1 --2- 172.21.15.107:0/2706024131 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc8900a2b90 0x7fc8900a5050 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc894000c00 tx=0x7fc8940314a0 comp rx=0 tx=0).stop 2024-05-09T08:30:52.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:52.014+0000 7fc8abc2c640 1 -- 172.21.15.107:0/2706024131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8a4101960 msgr2=0x7fc8a4102ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:52.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:52.014+0000 7fc8abc2c640 1 --2- 172.21.15.107:0/2706024131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8a4101960 0x7fc8a4102ed0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc8a000c970 tx=0x7fc8a000ce30 comp rx=0 tx=0).stop 2024-05-09T08:30:52.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:52.014+0000 7fc8abc2c640 1 -- 172.21.15.107:0/2706024131 shutdown_connections 2024-05-09T08:30:52.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:52.014+0000 7fc8abc2c640 1 --2- 172.21.15.107:0/2706024131 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc8900a2b90 0x7fc8900a5050 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:52.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:52.014+0000 7fc8abc2c640 1 --2- 172.21.15.107:0/2706024131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8a4102320 0x7fc8a4103410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:52.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:52.014+0000 7fc8abc2c640 1 --2- 172.21.15.107:0/2706024131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8a4101960 0x7fc8a4102ed0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:52.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:52.014+0000 7fc8abc2c640 1 --2- 172.21.15.107:0/2706024131 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8a4100770 0x7fc8a4104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:52.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:52.014+0000 7fc8abc2c640 1 -- 172.21.15.107:0/2706024131 >> 172.21.15.107:0/2706024131 conn(0x7fc8a40fbce0 msgr2=0x7fc8a40689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:52.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:52.014+0000 7fc8abc2c640 1 -- 172.21.15.107:0/2706024131 shutdown_connections 2024-05-09T08:30:52.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:52.014+0000 7fc8abc2c640 1 -- 172.21.15.107:0/2706024131 wait complete. 2024-05-09T08:30:52.023 INFO:journalctl@ceph.mds.a.smithi107.stdout:May 09 08:30:51 smithi107 podman[69413]: 2024-05-09 08:30:51.700067442 +0000 UTC m=+0.379097203 container init de4ae376929bbf0b76f1a05a35d3935a0f3b362c1ab78bf64353332a50c1415b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-a, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD) 2024-05-09T08:30:52.023 INFO:journalctl@ceph.mds.a.smithi107.stdout:May 09 08:30:51 smithi107 podman[69413]: 2024-05-09 08:30:51.702888469 +0000 UTC m=+0.381918220 container start de4ae376929bbf0b76f1a05a35d3935a0f3b362c1ab78bf64353332a50c1415b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0) 2024-05-09T08:30:52.024 INFO:journalctl@ceph.mds.a.smithi107.stdout:May 09 08:30:51 smithi107 ceph-mds[69452]: -- 172.21.15.107:0/1365743376 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55636e49fa40 con 0x55636e472c00 2024-05-09T08:30:52.024 INFO:journalctl@ceph.mds.a.smithi107.stdout:May 09 08:30:51 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-a[69428]: starting mds.a at 2024-05-09T08:30:52.024 INFO:journalctl@ceph.mds.a.smithi107.stdout:May 09 08:30:51 smithi107 bash[69413]: de4ae376929bbf0b76f1a05a35d3935a0f3b362c1ab78bf64353332a50c1415b 2024-05-09T08:30:52.024 INFO:journalctl@ceph.mds.a.smithi107.stdout:May 09 08:30:51 smithi107 systemd[1]: Started Ceph mds.a for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:30:52.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:52 smithi107 ceph-mon[30122]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:52.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:52 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:52.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:52 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:52.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:52 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:52.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:52 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:52.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:52 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:52.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:52 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:52.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:52 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2706024131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:52.821 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:52 smithi110 ceph-mon[33622]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:52.821 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:52 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:52.822 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:52 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:52.822 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:52 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:52.822 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:52 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:52.822 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:52 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:52.822 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:52 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:52.822 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:52 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2706024131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:52.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:52 smithi157 ceph-mon[33614]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:52.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:52 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:52.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:52 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:52.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:52 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:52.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:52 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T08:30:52.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:52 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T08:30:52.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:52 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:52.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:52 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2706024131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:53.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:53 smithi107 ceph-mon[30122]: Deploying daemon mds.b on smithi110 2024-05-09T08:30:53.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:53 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.107:6840/2463167549,v1:172.21.15.107:6841/2463167549] up:boot 2024-05-09T08:30:53.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:53 smithi107 ceph-mon[30122]: fsmap 11 up:standby 2024-05-09T08:30:53.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:53 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:30:53.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:53 smithi157 ceph-mon[33614]: Deploying daemon mds.b on smithi110 2024-05-09T08:30:53.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:53 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.107:6840/2463167549,v1:172.21.15.107:6841/2463167549] up:boot 2024-05-09T08:30:53.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:53 smithi157 ceph-mon[33614]: fsmap 11 up:standby 2024-05-09T08:30:53.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:53 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:30:53.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:53 smithi110 ceph-mon[33622]: Deploying daemon mds.b on smithi110 2024-05-09T08:30:53.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:53 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.107:6840/2463167549,v1:172.21.15.107:6841/2463167549] up:boot 2024-05-09T08:30:53.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:53 smithi110 ceph-mon[33622]: fsmap 11 up:standby 2024-05-09T08:30:53.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:53 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:30:54.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:30:54.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.141+0000 7f49c167d640 1 -- 172.21.15.107:0/4050508439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49bc102340 msgr2=0x7f49bc108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:54.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.141+0000 7f49c167d640 1 --2- 172.21.15.107:0/4050508439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49bc102340 0x7f49bc108bd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f49b4009ca0 tx=0x7f49b402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:30:54.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.141+0000 7f49c167d640 1 -- 172.21.15.107:0/4050508439 shutdown_connections 2024-05-09T08:30:54.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.141+0000 7f49c167d640 1 --2- 172.21.15.107:0/4050508439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49bc102340 0x7f49bc108bd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:54.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.141+0000 7f49c167d640 1 --2- 172.21.15.107:0/4050508439 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49bc101980 0x7f49bc101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:54.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.141+0000 7f49c167d640 1 --2- 172.21.15.107:0/4050508439 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49bc100790 0x7f49bc100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:54.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.141+0000 7f49c167d640 1 -- 172.21.15.107:0/4050508439 >> 172.21.15.107:0/4050508439 conn(0x7f49bc0fbd20 msgr2=0x7f49bc0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:54.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.141+0000 7f49c167d640 1 -- 172.21.15.107:0/4050508439 shutdown_connections 2024-05-09T08:30:54.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.141+0000 7f49c167d640 1 -- 172.21.15.107:0/4050508439 wait complete. 2024-05-09T08:30:54.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.142+0000 7f49c167d640 1 Processor -- start 2024-05-09T08:30:54.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.142+0000 7f49c167d640 1 -- start start 2024-05-09T08:30:54.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.142+0000 7f49c167d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49bc100790 0x7f49bc197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:54.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.142+0000 7f49c167d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49bc101980 0x7f49bc197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:54.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.142+0000 7f49c167d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49bc102340 0x7f49bc19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:54.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.142+0000 7f49c167d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc10d3a0 con 0x7f49bc102340 2024-05-09T08:30:54.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.142+0000 7f49c167d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc10d220 con 0x7f49bc101980 2024-05-09T08:30:54.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.142+0000 7f49c167d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc10d520 con 0x7f49bc100790 2024-05-09T08:30:54.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.142+0000 7f49bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49bc102340 0x7f49bc19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:54.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.142+0000 7f49bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49bc102340 0x7f49bc19f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40412/0 (socket says 172.21.15.107:40412) 2024-05-09T08:30:54.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.142+0000 7f49bb7fe640 1 -- 172.21.15.107:0/2323308448 learned_addr learned my addr 172.21.15.107:0/2323308448 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:54.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.142+0000 7f49b3fff640 1 --2- 172.21.15.107:0/2323308448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49bc101980 0x7f49bc197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:54.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.143+0000 7f49baffd640 1 --2- 172.21.15.107:0/2323308448 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49bc100790 0x7f49bc197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:54.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.143+0000 7f49bb7fe640 1 -- 172.21.15.107:0/2323308448 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49bc100790 msgr2=0x7f49bc197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:54.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.143+0000 7f49bb7fe640 1 --2- 172.21.15.107:0/2323308448 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49bc100790 0x7f49bc197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:54.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.143+0000 7f49bb7fe640 1 -- 172.21.15.107:0/2323308448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49bc101980 msgr2=0x7f49bc197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:54.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.143+0000 7f49bb7fe640 1 --2- 172.21.15.107:0/2323308448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49bc101980 0x7f49bc197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:54.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.143+0000 7f49bb7fe640 1 -- 172.21.15.107:0/2323308448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49bc19f6a0 con 0x7f49bc102340 2024-05-09T08:30:54.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.143+0000 7f49bb7fe640 1 --2- 172.21.15.107:0/2323308448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49bc102340 0x7f49bc19f030 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f49b4009dc0 tx=0x7f49b4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:54.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.143+0000 7f49b8ff9640 1 -- 172.21.15.107:0/2323308448 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b4032ec0 con 0x7f49bc102340 2024-05-09T08:30:54.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.143+0000 7f49b8ff9640 1 -- 172.21.15.107:0/2323308448 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49b40044f0 con 0x7f49bc102340 2024-05-09T08:30:54.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.143+0000 7f49c167d640 1 -- 172.21.15.107:0/2323308448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49bc19f930 con 0x7f49bc102340 2024-05-09T08:30:54.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.143+0000 7f49b8ff9640 1 -- 172.21.15.107:0/2323308448 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b4030d40 con 0x7f49bc102340 2024-05-09T08:30:54.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.143+0000 7f49c167d640 1 -- 172.21.15.107:0/2323308448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49bc19fe80 con 0x7f49bc102340 2024-05-09T08:30:54.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.144+0000 7f49c167d640 1 -- 172.21.15.107:0/2323308448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49bc107b20 con 0x7f49bc102340 2024-05-09T08:30:54.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.145+0000 7f49b8ff9640 1 -- 172.21.15.107:0/2323308448 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f49b4004040 con 0x7f49bc102340 2024-05-09T08:30:54.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.146+0000 7f49b8ff9640 1 --2- 172.21.15.107:0/2323308448 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f49a80a2d30 0x7f49a80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:54.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.146+0000 7f49baffd640 1 --2- 172.21.15.107:0/2323308448 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f49a80a2d30 0x7f49a80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:54.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.146+0000 7f49b8ff9640 1 -- 172.21.15.107:0/2323308448 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f49b40f0180 con 0x7f49bc102340 2024-05-09T08:30:54.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.146+0000 7f49baffd640 1 --2- 172.21.15.107:0/2323308448 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f49a80a2d30 0x7f49a80a51f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f49ac005d00 tx=0x7f49ac005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:54.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.150+0000 7f49b8ff9640 1 -- 172.21.15.107:0/2323308448 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f49b40bd580 con 0x7f49bc102340 2024-05-09T08:30:54.289 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:30:54 smithi110 systemd[1]: Starting Ceph mds.b for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:30:54.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.361+0000 7f49c167d640 1 -- 172.21.15.107:0/2323308448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49bc02d1c0 con 0x7f49bc102340 2024-05-09T08:30:54.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.362+0000 7f49b8ff9640 1 -- 172.21.15.107:0/2323308448 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9671 (secure 0 0 0) 0x7f49b40bce80 con 0x7f49bc102340 2024-05-09T08:30:54.361 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:54.361 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-09T08:30:54.362 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2024-05-09T08:30:54.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.363+0000 7f49c167d640 1 -- 172.21.15.107:0/2323308448 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f49a80a2d30 msgr2=0x7f49a80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:54.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.363+0000 7f49c167d640 1 --2- 172.21.15.107:0/2323308448 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f49a80a2d30 0x7f49a80a51f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f49ac005d00 tx=0x7f49ac005c50 comp rx=0 tx=0).stop 2024-05-09T08:30:54.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.364+0000 7f49c167d640 1 -- 172.21.15.107:0/2323308448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49bc102340 msgr2=0x7f49bc19f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:54.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.364+0000 7f49c167d640 1 --2- 172.21.15.107:0/2323308448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49bc102340 0x7f49bc19f030 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f49b4009dc0 tx=0x7f49b4003ba0 comp rx=0 tx=0).stop 2024-05-09T08:30:54.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.364+0000 7f49c167d640 1 -- 172.21.15.107:0/2323308448 shutdown_connections 2024-05-09T08:30:54.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.364+0000 7f49c167d640 1 --2- 172.21.15.107:0/2323308448 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f49a80a2d30 0x7f49a80a51f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:54.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.364+0000 7f49c167d640 1 --2- 172.21.15.107:0/2323308448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49bc102340 0x7f49bc19f030 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:54.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.364+0000 7f49c167d640 1 --2- 172.21.15.107:0/2323308448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49bc101980 0x7f49bc197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:54.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.364+0000 7f49c167d640 1 --2- 172.21.15.107:0/2323308448 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49bc100790 0x7f49bc197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:54.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.364+0000 7f49c167d640 1 -- 172.21.15.107:0/2323308448 >> 172.21.15.107:0/2323308448 conn(0x7f49bc0fbd20 msgr2=0x7f49bc104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:54.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.364+0000 7f49c167d640 1 -- 172.21.15.107:0/2323308448 shutdown_connections 2024-05-09T08:30:54.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:54.364+0000 7f49c167d640 1 -- 172.21.15.107:0/2323308448 wait complete. 2024-05-09T08:30:54.556 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:54 smithi110 ceph-mon[33622]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:54.556 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:54 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2323308448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:54.556 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:30:54 smithi110 podman[63768]: 2024-05-09 08:30:54.19267195 +0000 UTC m=+0.017496117 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:30:54.557 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:30:54 smithi110 podman[63768]: 2024-05-09 08:30:54.304721703 +0000 UTC m=+0.129545857 container create 100c0de814566a37ab7fa9734ef7b111ef30fecf568430216c225f2189cbc0dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-09T08:30:54.557 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:30:54 smithi110 podman[63768]: 2024-05-09 08:30:54.557662969 +0000 UTC m=+0.382487123 container init 100c0de814566a37ab7fa9734ef7b111ef30fecf568430216c225f2189cbc0dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-b, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240507, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-09T08:30:54.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:54 smithi107 ceph-mon[30122]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:54.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:54 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2323308448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:54.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:54 smithi157 ceph-mon[33614]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:54.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:54 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2323308448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:54.870 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:30:54 smithi110 podman[63768]: 2024-05-09 08:30:54.560774856 +0000 UTC m=+0.385599023 container start 100c0de814566a37ab7fa9734ef7b111ef30fecf568430216c225f2189cbc0dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507) 2024-05-09T08:30:54.871 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:30:54 smithi110 ceph-mds[63787]: -- 172.21.15.110:0/783726904 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eb03403a40 con 0x55eb033d6c00 2024-05-09T08:30:54.871 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:30:54 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-b[63783]: starting mds.b at 2024-05-09T08:30:54.871 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:30:54 smithi110 bash[63768]: 100c0de814566a37ab7fa9734ef7b111ef30fecf568430216c225f2189cbc0dc 2024-05-09T08:30:54.871 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:30:54 smithi110 systemd[1]: Started Ceph mds.b for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:30:55.981 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:55.981 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:55.981 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:55.981 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:55.982 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:55.982 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:55.982 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:55 smithi157 ceph-mon[33614]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:55.982 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:55.982 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:55 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] up:boot 2024-05-09T08:30:55.982 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:55 smithi157 ceph-mon[33614]: fsmap 12 up:standby 2024-05-09T08:30:55.982 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:30:56.018 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:56.019 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:56.019 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:56.019 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:56.019 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:56.019 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:56.019 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:55 smithi110 ceph-mon[33622]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:56.019 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:56.019 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:55 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] up:boot 2024-05-09T08:30:56.019 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:55 smithi110 ceph-mon[33622]: fsmap 12 up:standby 2024-05-09T08:30:56.019 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:30:56.027 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:56.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:56.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:56.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:56.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:56.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:56.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:55 smithi107 ceph-mon[30122]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:56.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:56.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:55 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] up:boot 2024-05-09T08:30:56.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:55 smithi107 ceph-mon[30122]: fsmap 12 up:standby 2024-05-09T08:30:56.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:30:56.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:30:56.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.488+0000 7fe52211d640 1 -- 172.21.15.107:0/1950549387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c101960 msgr2=0x7fe51c101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:56.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.488+0000 7fe52211d640 1 --2- 172.21.15.107:0/1950549387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c101960 0x7fe51c101de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe510009900 tx=0x7fe51002a410 comp rx=0 tx=0).stop 2024-05-09T08:30:56.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.488+0000 7fe52211d640 1 -- 172.21.15.107:0/1950549387 shutdown_connections 2024-05-09T08:30:56.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.488+0000 7fe52211d640 1 --2- 172.21.15.107:0/1950549387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe51c102320 0x7fe51c108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:56.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.488+0000 7fe52211d640 1 --2- 172.21.15.107:0/1950549387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c101960 0x7fe51c101de0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:56.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.488+0000 7fe52211d640 1 --2- 172.21.15.107:0/1950549387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe51c100770 0x7fe51c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:56.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.488+0000 7fe52211d640 1 -- 172.21.15.107:0/1950549387 >> 172.21.15.107:0/1950549387 conn(0x7fe51c0fbce0 msgr2=0x7fe51c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:56.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.488+0000 7fe52211d640 1 -- 172.21.15.107:0/1950549387 shutdown_connections 2024-05-09T08:30:56.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.488+0000 7fe52211d640 1 -- 172.21.15.107:0/1950549387 wait complete. 2024-05-09T08:30:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.489+0000 7fe52211d640 1 Processor -- start 2024-05-09T08:30:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe52211d640 1 -- start start 2024-05-09T08:30:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe52211d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe51c100770 0x7fe51c1969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe52211d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe51c101960 0x7fe51c196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe52211d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c102320 0x7fe51c19df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe52211d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c10c2a0 con 0x7fe51c102320 2024-05-09T08:30:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe52211d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c10c120 con 0x7fe51c100770 2024-05-09T08:30:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe52211d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c10c420 con 0x7fe51c101960 2024-05-09T08:30:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe51bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c102320 0x7fe51c19df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe51bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c102320 0x7fe51c19df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40426/0 (socket says 172.21.15.107:40426) 2024-05-09T08:30:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe51bfff640 1 -- 172.21.15.107:0/999452494 learned_addr learned my addr 172.21.15.107:0/999452494 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe51affd640 1 --2- 172.21.15.107:0/999452494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe51c101960 0x7fe51c196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe51b7fe640 1 --2- 172.21.15.107:0/999452494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe51c100770 0x7fe51c1969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe51bfff640 1 -- 172.21.15.107:0/999452494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe51c101960 msgr2=0x7fe51c196f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe51bfff640 1 --2- 172.21.15.107:0/999452494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe51c101960 0x7fe51c196f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe51bfff640 1 -- 172.21.15.107:0/999452494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe51c100770 msgr2=0x7fe51c1969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe51bfff640 1 --2- 172.21.15.107:0/999452494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe51c100770 0x7fe51c1969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe51bfff640 1 -- 172.21.15.107:0/999452494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51c19e600 con 0x7fe51c102320 2024-05-09T08:30:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.490+0000 7fe51bfff640 1 --2- 172.21.15.107:0/999452494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c102320 0x7fe51c19df90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe514009fd0 tx=0x7fe514007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.491+0000 7fe518ff9640 1 -- 172.21.15.107:0/999452494 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51400de10 con 0x7fe51c102320 2024-05-09T08:30:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.491+0000 7fe518ff9640 1 -- 172.21.15.107:0/999452494 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe514003e20 con 0x7fe51c102320 2024-05-09T08:30:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.491+0000 7fe52211d640 1 -- 172.21.15.107:0/999452494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51c068a20 con 0x7fe51c102320 2024-05-09T08:30:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.491+0000 7fe518ff9640 1 -- 172.21.15.107:0/999452494 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe514004810 con 0x7fe51c102320 2024-05-09T08:30:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.491+0000 7fe52211d640 1 -- 172.21.15.107:0/999452494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe51c068ef0 con 0x7fe51c102320 2024-05-09T08:30:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.492+0000 7fe52211d640 1 -- 172.21.15.107:0/999452494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe51c068650 con 0x7fe51c102320 2024-05-09T08:30:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.493+0000 7fe518ff9640 1 -- 172.21.15.107:0/999452494 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fe514003970 con 0x7fe51c102320 2024-05-09T08:30:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.494+0000 7fe518ff9640 1 --2- 172.21.15.107:0/999452494 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe5040a2d30 0x7fe5040a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.494+0000 7fe51b7fe640 1 --2- 172.21.15.107:0/999452494 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe5040a2d30 0x7fe5040a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.494+0000 7fe518ff9640 1 -- 172.21.15.107:0/999452494 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe5140d0710 con 0x7fe51c102320 2024-05-09T08:30:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.494+0000 7fe51b7fe640 1 --2- 172.21.15.107:0/999452494 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe5040a2d30 0x7fe5040a51f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe50c005fd0 tx=0x7fe50c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:56.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.500+0000 7fe518ff9640 1 -- 172.21.15.107:0/999452494 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe51409d4c0 con 0x7fe51c102320 2024-05-09T08:30:56.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.736+0000 7fe52211d640 1 -- 172.21.15.107:0/999452494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe51c107800 con 0x7fe51c102320 2024-05-09T08:30:56.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.737+0000 7fe518ff9640 1 -- 172.21.15.107:0/999452494 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10497 (secure 0 0 0) 0x7fe5140a17c0 con 0x7fe51c102320 2024-05-09T08:30:56.736 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:56.737 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24491,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-09T08:30:56.737 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 13 2024-05-09T08:30:56.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.739+0000 7fe52211d640 1 -- 172.21.15.107:0/999452494 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe5040a2d30 msgr2=0x7fe5040a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:56.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.739+0000 7fe52211d640 1 --2- 172.21.15.107:0/999452494 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe5040a2d30 0x7fe5040a51f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe50c005fd0 tx=0x7fe50c00b390 comp rx=0 tx=0).stop 2024-05-09T08:30:56.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.739+0000 7fe52211d640 1 -- 172.21.15.107:0/999452494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c102320 msgr2=0x7fe51c19df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:56.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.739+0000 7fe52211d640 1 --2- 172.21.15.107:0/999452494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c102320 0x7fe51c19df90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe514009fd0 tx=0x7fe514007590 comp rx=0 tx=0).stop 2024-05-09T08:30:56.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.739+0000 7fe52211d640 1 -- 172.21.15.107:0/999452494 shutdown_connections 2024-05-09T08:30:56.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.739+0000 7fe52211d640 1 --2- 172.21.15.107:0/999452494 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe5040a2d30 0x7fe5040a51f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:56.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.739+0000 7fe52211d640 1 --2- 172.21.15.107:0/999452494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51c102320 0x7fe51c19df90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:56.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.739+0000 7fe52211d640 1 --2- 172.21.15.107:0/999452494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe51c101960 0x7fe51c196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:56.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.739+0000 7fe52211d640 1 --2- 172.21.15.107:0/999452494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe51c100770 0x7fe51c1969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:56.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.739+0000 7fe52211d640 1 -- 172.21.15.107:0/999452494 >> 172.21.15.107:0/999452494 conn(0x7fe51c0fbce0 msgr2=0x7fe51c104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:56.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.739+0000 7fe52211d640 1 -- 172.21.15.107:0/999452494 shutdown_connections 2024-05-09T08:30:56.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.739+0000 7fe52211d640 1 -- 172.21.15.107:0/999452494 wait complete. 2024-05-09T08:30:56.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:30:56.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.855+0000 7f0193ac3640 1 -- 172.21.15.107:0/811149359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f018c100770 msgr2=0x7f018c100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:56.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.855+0000 7f0193ac3640 1 --2- 172.21.15.107:0/811149359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f018c100770 0x7f018c100b90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0184009ca0 tx=0x7f018402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:30:56.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.855+0000 7f0193ac3640 1 -- 172.21.15.107:0/811149359 shutdown_connections 2024-05-09T08:30:56.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.855+0000 7f0193ac3640 1 --2- 172.21.15.107:0/811149359 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f018c102320 0x7f018c108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:56.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.855+0000 7f0193ac3640 1 --2- 172.21.15.107:0/811149359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f018c101960 0x7f018c101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:56.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.855+0000 7f0193ac3640 1 --2- 172.21.15.107:0/811149359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f018c100770 0x7f018c100b90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:56.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.855+0000 7f0193ac3640 1 -- 172.21.15.107:0/811149359 >> 172.21.15.107:0/811149359 conn(0x7f018c0fbce0 msgr2=0x7f018c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:56.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.855+0000 7f0193ac3640 1 -- 172.21.15.107:0/811149359 shutdown_connections 2024-05-09T08:30:56.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.855+0000 7f0193ac3640 1 -- 172.21.15.107:0/811149359 wait complete. 2024-05-09T08:30:56.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f0193ac3640 1 Processor -- start 2024-05-09T08:30:56.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f0193ac3640 1 -- start start 2024-05-09T08:30:56.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f0193ac3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f018c100770 0x7f018c069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:56.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f0193ac3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f018c101960 0x7f018c19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:56.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f0193ac3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f018c102320 0x7f018c19ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:56.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f0193ac3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f018c10c2a0 con 0x7f018c100770 2024-05-09T08:30:56.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f0193ac3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f018c10c120 con 0x7f018c102320 2024-05-09T08:30:56.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f0193ac3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f018c10c420 con 0x7f018c101960 2024-05-09T08:30:56.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f019185f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f018c100770 0x7f018c069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:56.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f019185f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f018c100770 0x7f018c069550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40440/0 (socket says 172.21.15.107:40440) 2024-05-09T08:30:56.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f019185f640 1 -- 172.21.15.107:0/2050936598 learned_addr learned my addr 172.21.15.107:0/2050936598 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:56.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f019105e640 1 --2- 172.21.15.107:0/2050936598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f018c101960 0x7f018c19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:56.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f0192060640 1 --2- 172.21.15.107:0/2050936598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f018c102320 0x7f018c19ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:56.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f019185f640 1 -- 172.21.15.107:0/2050936598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f018c101960 msgr2=0x7f018c19c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:56.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f019185f640 1 --2- 172.21.15.107:0/2050936598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f018c101960 0x7f018c19c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:56.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f019185f640 1 -- 172.21.15.107:0/2050936598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f018c102320 msgr2=0x7f018c19ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:56.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f019185f640 1 --2- 172.21.15.107:0/2050936598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f018c102320 0x7f018c19ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:56.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.856+0000 7f019185f640 1 -- 172.21.15.107:0/2050936598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018c19f700 con 0x7f018c100770 2024-05-09T08:30:56.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.857+0000 7f019185f640 1 --2- 172.21.15.107:0/2050936598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f018c100770 0x7f018c069550 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0184003ba0 tx=0x7f0184003bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:56.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.857+0000 7f018affd640 1 -- 172.21.15.107:0/2050936598 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184004040 con 0x7f018c100770 2024-05-09T08:30:56.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.857+0000 7f018affd640 1 -- 172.21.15.107:0/2050936598 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0184004690 con 0x7f018c100770 2024-05-09T08:30:56.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.857+0000 7f0193ac3640 1 -- 172.21.15.107:0/2050936598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f018c19f990 con 0x7f018c100770 2024-05-09T08:30:56.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.857+0000 7f018affd640 1 -- 172.21.15.107:0/2050936598 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184030cd0 con 0x7f018c100770 2024-05-09T08:30:56.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.857+0000 7f0193ac3640 1 -- 172.21.15.107:0/2050936598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f018c19ff30 con 0x7f018c100770 2024-05-09T08:30:56.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.858+0000 7f0193ac3640 1 -- 172.21.15.107:0/2050936598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f018c107b10 con 0x7f018c100770 2024-05-09T08:30:56.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.859+0000 7f018affd640 1 -- 172.21.15.107:0/2050936598 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f01840041e0 con 0x7f018c100770 2024-05-09T08:30:56.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.860+0000 7f018affd640 1 --2- 172.21.15.107:0/2050936598 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f01740a2c10 0x7f01740a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:56.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.860+0000 7f019105e640 1 --2- 172.21.15.107:0/2050936598 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f01740a2c10 0x7f01740a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:56.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.860+0000 7f018affd640 1 -- 172.21.15.107:0/2050936598 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f01840f02f0 con 0x7f018c100770 2024-05-09T08:30:56.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.860+0000 7f019105e640 1 --2- 172.21.15.107:0/2050936598 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f01740a2c10 0x7f01740a50d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f018c0692c0 tx=0x7f0180005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:56.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:56.863+0000 7f018affd640 1 -- 172.21.15.107:0/2050936598 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f01840bd6f0 con 0x7f018c100770 2024-05-09T08:30:57.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.046+0000 7f0193ac3640 1 -- 172.21.15.107:0/2050936598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f018c02d3c0 con 0x7f018c100770 2024-05-09T08:30:57.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.047+0000 7f018affd640 1 -- 172.21.15.107:0/2050936598 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18515 (secure 0 0 0) 0x7f01840bcff0 con 0x7f018c100770 2024-05-09T08:30:57.046 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:57.046 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":81,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:30:10.960298+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:30:57.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.049+0000 7f0193ac3640 1 -- 172.21.15.107:0/2050936598 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f01740a2c10 msgr2=0x7f01740a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:57.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.049+0000 7f0193ac3640 1 --2- 172.21.15.107:0/2050936598 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f01740a2c10 0x7f01740a50d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f018c0692c0 tx=0x7f0180005f90 comp rx=0 tx=0).stop 2024-05-09T08:30:57.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.049+0000 7f0193ac3640 1 -- 172.21.15.107:0/2050936598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f018c100770 msgr2=0x7f018c069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:57.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.049+0000 7f0193ac3640 1 --2- 172.21.15.107:0/2050936598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f018c100770 0x7f018c069550 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0184003ba0 tx=0x7f0184003bd0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.049+0000 7f0193ac3640 1 -- 172.21.15.107:0/2050936598 shutdown_connections 2024-05-09T08:30:57.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.049+0000 7f0193ac3640 1 --2- 172.21.15.107:0/2050936598 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f01740a2c10 0x7f01740a50d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.049+0000 7f0193ac3640 1 --2- 172.21.15.107:0/2050936598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f018c102320 0x7f018c19ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.049+0000 7f0193ac3640 1 --2- 172.21.15.107:0/2050936598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f018c101960 0x7f018c19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.049+0000 7f0193ac3640 1 --2- 172.21.15.107:0/2050936598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f018c100770 0x7f018c069550 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.049+0000 7f0193ac3640 1 -- 172.21.15.107:0/2050936598 >> 172.21.15.107:0/2050936598 conn(0x7f018c0fbce0 msgr2=0x7f018c104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:57.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.049+0000 7f0193ac3640 1 -- 172.21.15.107:0/2050936598 shutdown_connections 2024-05-09T08:30:57.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.049+0000 7f0193ac3640 1 -- 172.21.15.107:0/2050936598 wait complete. 2024-05-09T08:30:57.060 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:26:21.698338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T08:30:57.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:30:57.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.168+0000 7f3b94822640 1 -- 172.21.15.107:0/2987294899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b90102350 msgr2=0x7f3b90104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:57.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.168+0000 7f3b94822640 1 --2- 172.21.15.107:0/2987294899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b90102350 0x7f3b90104740 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3b80009900 tx=0x7f3b8002a410 comp rx=0 tx=0).stop 2024-05-09T08:30:57.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.168+0000 7f3b94822640 1 -- 172.21.15.107:0/2987294899 shutdown_connections 2024-05-09T08:30:57.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.168+0000 7f3b94822640 1 --2- 172.21.15.107:0/2987294899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b901075b0 0x7f3b901099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.168+0000 7f3b94822640 1 --2- 172.21.15.107:0/2987294899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b90104c80 0x7f3b90107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.168+0000 7f3b94822640 1 --2- 172.21.15.107:0/2987294899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b90102350 0x7f3b90104740 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.168+0000 7f3b94822640 1 -- 172.21.15.107:0/2987294899 >> 172.21.15.107:0/2987294899 conn(0x7f3b900fbce0 msgr2=0x7f3b900fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:57.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.168+0000 7f3b94822640 1 -- 172.21.15.107:0/2987294899 shutdown_connections 2024-05-09T08:30:57.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.168+0000 7f3b94822640 1 -- 172.21.15.107:0/2987294899 wait complete. 2024-05-09T08:30:57.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b94822640 1 Processor -- start 2024-05-09T08:30:57.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b94822640 1 -- start start 2024-05-09T08:30:57.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b94822640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b90102350 0x7f3b90197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:57.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b94822640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b90104c80 0x7f3b901980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:57.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b94822640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b901075b0 0x7f3b9019f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:57.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b94822640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9010d3c0 con 0x7f3b90102350 2024-05-09T08:30:57.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b94822640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9010d240 con 0x7f3b90104c80 2024-05-09T08:30:57.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b94822640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9010d540 con 0x7f3b901075b0 2024-05-09T08:30:57.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b8e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b90102350 0x7f3b90197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:57.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b8e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b90102350 0x7f3b90197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40458/0 (socket says 172.21.15.107:40458) 2024-05-09T08:30:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b8e59c640 1 -- 172.21.15.107:0/679451831 learned_addr learned my addr 172.21.15.107:0/679451831 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b87fff640 1 --2- 172.21.15.107:0/679451831 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b90104c80 0x7f3b901980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b8ed9d640 1 --2- 172.21.15.107:0/679451831 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b901075b0 0x7f3b9019f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b8e59c640 1 -- 172.21.15.107:0/679451831 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b901075b0 msgr2=0x7f3b9019f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b8e59c640 1 --2- 172.21.15.107:0/679451831 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b901075b0 0x7f3b9019f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b8e59c640 1 -- 172.21.15.107:0/679451831 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b90104c80 msgr2=0x7f3b901980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b8e59c640 1 --2- 172.21.15.107:0/679451831 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b90104c80 0x7f3b901980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.169+0000 7f3b8e59c640 1 -- 172.21.15.107:0/679451831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b9019f790 con 0x7f3b90102350 2024-05-09T08:30:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.170+0000 7f3b8e59c640 1 --2- 172.21.15.107:0/679451831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b90102350 0x7f3b90197b70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3b80004770 tx=0x7f3b800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:57.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.170+0000 7f3b877fe640 1 -- 172.21.15.107:0/679451831 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8003f030 con 0x7f3b90102350 2024-05-09T08:30:57.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.170+0000 7f3b94822640 1 -- 172.21.15.107:0/679451831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b9019fa20 con 0x7f3b90102350 2024-05-09T08:30:57.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.170+0000 7f3b877fe640 1 -- 172.21.15.107:0/679451831 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b80003cb0 con 0x7f3b90102350 2024-05-09T08:30:57.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.170+0000 7f3b94822640 1 -- 172.21.15.107:0/679451831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b9019ff60 con 0x7f3b90102350 2024-05-09T08:30:57.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.170+0000 7f3b877fe640 1 -- 172.21.15.107:0/679451831 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b80049060 con 0x7f3b90102350 2024-05-09T08:30:57.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.171+0000 7f3b94822640 1 -- 172.21.15.107:0/679451831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b900fd7d0 con 0x7f3b90102350 2024-05-09T08:30:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.172+0000 7f3b877fe640 1 -- 172.21.15.107:0/679451831 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3b80042030 con 0x7f3b90102350 2024-05-09T08:30:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.173+0000 7f3b877fe640 1 --2- 172.21.15.107:0/679451831 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3b7c0a2b90 0x7f3b7c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.173+0000 7f3b877fe640 1 -- 172.21.15.107:0/679451831 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3b800ef4c0 con 0x7f3b90102350 2024-05-09T08:30:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.173+0000 7f3b87fff640 1 --2- 172.21.15.107:0/679451831 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3b7c0a2b90 0x7f3b7c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.173+0000 7f3b87fff640 1 --2- 172.21.15.107:0/679451831 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3b7c0a2b90 0x7f3b7c0a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3b90199080 tx=0x7f3b7800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:57.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.177+0000 7f3b877fe640 1 -- 172.21.15.107:0/679451831 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3b800bc270 con 0x7f3b90102350 2024-05-09T08:30:57.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.375+0000 7f3b94822640 1 -- 172.21.15.107:0/679451831 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3b9002d1c0 con 0x7f3b90102350 2024-05-09T08:30:57.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.376+0000 7f3b877fe640 1 -- 172.21.15.107:0/679451831 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f3b800c0570 con 0x7f3b90102350 2024-05-09T08:30:57.375 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-05-09T08:30:57.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.377+0000 7f3b94822640 1 -- 172.21.15.107:0/679451831 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3b7c0a2b90 msgr2=0x7f3b7c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:57.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.377+0000 7f3b94822640 1 --2- 172.21.15.107:0/679451831 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3b7c0a2b90 0x7f3b7c0a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3b90199080 tx=0x7f3b7800b390 comp rx=0 tx=0).stop 2024-05-09T08:30:57.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.377+0000 7f3b94822640 1 -- 172.21.15.107:0/679451831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b90102350 msgr2=0x7f3b90197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:57.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.377+0000 7f3b94822640 1 --2- 172.21.15.107:0/679451831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b90102350 0x7f3b90197b70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3b80004770 tx=0x7f3b800042f0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.377+0000 7f3b94822640 1 -- 172.21.15.107:0/679451831 shutdown_connections 2024-05-09T08:30:57.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.377+0000 7f3b94822640 1 --2- 172.21.15.107:0/679451831 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3b7c0a2b90 0x7f3b7c0a5050 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.377+0000 7f3b94822640 1 --2- 172.21.15.107:0/679451831 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b901075b0 0x7f3b9019f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.377+0000 7f3b94822640 1 --2- 172.21.15.107:0/679451831 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b90104c80 0x7f3b901980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.377+0000 7f3b94822640 1 --2- 172.21.15.107:0/679451831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b90102350 0x7f3b90197b70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.378+0000 7f3b94822640 1 -- 172.21.15.107:0/679451831 >> 172.21.15.107:0/679451831 conn(0x7f3b900fbce0 msgr2=0x7f3b90102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:57.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.378+0000 7f3b94822640 1 -- 172.21.15.107:0/679451831 shutdown_connections 2024-05-09T08:30:57.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.378+0000 7f3b94822640 1 -- 172.21.15.107:0/679451831 wait complete. 2024-05-09T08:30:57.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T08:30:57.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.498+0000 7fbf2855a640 1 -- 172.21.15.107:0/743675638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf20102a90 msgr2=0x7fbf20102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:57.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.498+0000 7fbf2855a640 1 --2- 172.21.15.107:0/743675638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf20102a90 0x7fbf20102f10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbf10009900 tx=0x7fbf1002a410 comp rx=0 tx=0).stop 2024-05-09T08:30:57.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.499+0000 7fbf2855a640 1 -- 172.21.15.107:0/743675638 shutdown_connections 2024-05-09T08:30:57.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.499+0000 7fbf2855a640 1 --2- 172.21.15.107:0/743675638 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf20103450 0x7fbf20109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.499+0000 7fbf2855a640 1 --2- 172.21.15.107:0/743675638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf20102a90 0x7fbf20102f10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.499+0000 7fbf2855a640 1 --2- 172.21.15.107:0/743675638 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf201018a0 0x7fbf20101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.499+0000 7fbf2855a640 1 -- 172.21.15.107:0/743675638 >> 172.21.15.107:0/743675638 conn(0x7fbf200fce30 msgr2=0x7fbf200ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:57.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.499+0000 7fbf2855a640 1 -- 172.21.15.107:0/743675638 shutdown_connections 2024-05-09T08:30:57.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.499+0000 7fbf2855a640 1 -- 172.21.15.107:0/743675638 wait complete. 2024-05-09T08:30:57.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.499+0000 7fbf2855a640 1 Processor -- start 2024-05-09T08:30:57.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.499+0000 7fbf2855a640 1 -- start start 2024-05-09T08:30:57.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.499+0000 7fbf2855a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf201018a0 0x7fbf20197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:57.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.499+0000 7fbf2855a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf20102a90 0x7fbf20197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:57.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.499+0000 7fbf2855a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf20103450 0x7fbf2019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:57.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.499+0000 7fbf2855a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2010d400 con 0x7fbf20102a90 2024-05-09T08:30:57.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.499+0000 7fbf2855a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2010d280 con 0x7fbf201018a0 2024-05-09T08:30:57.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.499+0000 7fbf2855a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2010d580 con 0x7fbf20103450 2024-05-09T08:30:57.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.500+0000 7fbf25af5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf20102a90 0x7fbf20197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:57.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.500+0000 7fbf25af5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf20102a90 0x7fbf20197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40468/0 (socket says 172.21.15.107:40468) 2024-05-09T08:30:57.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.500+0000 7fbf25af5640 1 -- 172.21.15.107:0/3325112687 learned_addr learned my addr 172.21.15.107:0/3325112687 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:57.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.500+0000 7fbf26af7640 1 --2- 172.21.15.107:0/3325112687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf20103450 0x7fbf2019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:57.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.500+0000 7fbf262f6640 1 --2- 172.21.15.107:0/3325112687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf201018a0 0x7fbf20197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:57.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.500+0000 7fbf25af5640 1 -- 172.21.15.107:0/3325112687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf20103450 msgr2=0x7fbf2019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:57.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.500+0000 7fbf25af5640 1 --2- 172.21.15.107:0/3325112687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf20103450 0x7fbf2019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.500+0000 7fbf25af5640 1 -- 172.21.15.107:0/3325112687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf201018a0 msgr2=0x7fbf20197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:57.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.500+0000 7fbf25af5640 1 --2- 172.21.15.107:0/3325112687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf201018a0 0x7fbf20197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.500+0000 7fbf25af5640 1 -- 172.21.15.107:0/3325112687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf2019f6c0 con 0x7fbf20102a90 2024-05-09T08:30:57.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.500+0000 7fbf25af5640 1 --2- 172.21.15.107:0/3325112687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf20102a90 0x7fbf20197fe0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbf100314e0 tx=0x7fbf10003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:57.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.501+0000 7fbf177fe640 1 -- 172.21.15.107:0/3325112687 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1003f070 con 0x7fbf20102a90 2024-05-09T08:30:57.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.501+0000 7fbf2855a640 1 -- 172.21.15.107:0/3325112687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf2019f950 con 0x7fbf20102a90 2024-05-09T08:30:57.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.501+0000 7fbf177fe640 1 -- 172.21.15.107:0/3325112687 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf10031c40 con 0x7fbf20102a90 2024-05-09T08:30:57.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.501+0000 7fbf177fe640 1 -- 172.21.15.107:0/3325112687 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf10035030 con 0x7fbf20102a90 2024-05-09T08:30:57.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.501+0000 7fbf2855a640 1 -- 172.21.15.107:0/3325112687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf2019fe90 con 0x7fbf20102a90 2024-05-09T08:30:57.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.502+0000 7fbf2855a640 1 -- 172.21.15.107:0/3325112687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf20108bc0 con 0x7fbf20102a90 2024-05-09T08:30:57.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.503+0000 7fbf177fe640 1 -- 172.21.15.107:0/3325112687 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fbf10042070 con 0x7fbf20102a90 2024-05-09T08:30:57.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.503+0000 7fbf177fe640 1 --2- 172.21.15.107:0/3325112687 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbf0c0a2b90 0x7fbf0c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:57.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.504+0000 7fbf262f6640 1 --2- 172.21.15.107:0/3325112687 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbf0c0a2b90 0x7fbf0c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:57.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.504+0000 7fbf177fe640 1 -- 172.21.15.107:0/3325112687 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbf100ef360 con 0x7fbf20102a90 2024-05-09T08:30:57.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.504+0000 7fbf262f6640 1 --2- 172.21.15.107:0/3325112687 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbf0c0a2b90 0x7fbf0c0a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbf201028f0 tx=0x7fbf1c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:57.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.508+0000 7fbf177fe640 1 -- 172.21.15.107:0/3325112687 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbf100bc110 con 0x7fbf20102a90 2024-05-09T08:30:57.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.715+0000 7fbf2855a640 1 -- 172.21.15.107:0/3325112687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fbf2002d390 con 0x7fbf20102a90 2024-05-09T08:30:57.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.715+0000 7fbf177fe640 1 -- 172.21.15.107:0/3325112687 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7fbf100c0410 con 0x7fbf20102a90 2024-05-09T08:30:57.714 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:30:57.714 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2024-05-09T08:30:57.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:30:57.715 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2024-05-09T08:30:57.715 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2024-05-09T08:30:57.715 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:30:57.715 INFO:teuthology.orchestra.run.smithi107.stdout:] 2024-05-09T08:30:57.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.717+0000 7fbf2855a640 1 -- 172.21.15.107:0/3325112687 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbf0c0a2b90 msgr2=0x7fbf0c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:57.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.717+0000 7fbf2855a640 1 --2- 172.21.15.107:0/3325112687 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbf0c0a2b90 0x7fbf0c0a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbf201028f0 tx=0x7fbf1c005f90 comp rx=0 tx=0).stop 2024-05-09T08:30:57.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.717+0000 7fbf2855a640 1 -- 172.21.15.107:0/3325112687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf20102a90 msgr2=0x7fbf20197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:57.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.717+0000 7fbf2855a640 1 --2- 172.21.15.107:0/3325112687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf20102a90 0x7fbf20197fe0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbf100314e0 tx=0x7fbf10003c10 comp rx=0 tx=0).stop 2024-05-09T08:30:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.717+0000 7fbf2855a640 1 -- 172.21.15.107:0/3325112687 shutdown_connections 2024-05-09T08:30:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.717+0000 7fbf2855a640 1 --2- 172.21.15.107:0/3325112687 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbf0c0a2b90 0x7fbf0c0a5050 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.717+0000 7fbf2855a640 1 --2- 172.21.15.107:0/3325112687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf20103450 0x7fbf2019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.717+0000 7fbf2855a640 1 --2- 172.21.15.107:0/3325112687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf20102a90 0x7fbf20197fe0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.717+0000 7fbf2855a640 1 --2- 172.21.15.107:0/3325112687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf201018a0 0x7fbf20197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.717+0000 7fbf2855a640 1 -- 172.21.15.107:0/3325112687 >> 172.21.15.107:0/3325112687 conn(0x7fbf200fce30 msgr2=0x7fbf20105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.717+0000 7fbf2855a640 1 -- 172.21.15.107:0/3325112687 shutdown_connections 2024-05-09T08:30:57.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.717+0000 7fbf2855a640 1 -- 172.21.15.107:0/3325112687 wait complete. 2024-05-09T08:30:57.727 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T08:30:57.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T08:30:57.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.836+0000 7f99fdabd640 1 -- 172.21.15.107:0/1096413919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99f8102b10 msgr2=0x7f99f8102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:57.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.836+0000 7f99fdabd640 1 --2- 172.21.15.107:0/1096413919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99f8102b10 0x7f99f8102f90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009930 tx=0x7f99e002a420 comp rx=0 tx=0).stop 2024-05-09T08:30:57.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.837+0000 7f99fdabd640 1 -- 172.21.15.107:0/1096413919 shutdown_connections 2024-05-09T08:30:57.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.837+0000 7f99fdabd640 1 --2- 172.21.15.107:0/1096413919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99f81034d0 0x7f99f8109d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.837+0000 7f99fdabd640 1 --2- 172.21.15.107:0/1096413919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99f8102b10 0x7f99f8102f90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.837+0000 7f99fdabd640 1 --2- 172.21.15.107:0/1096413919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99f80ffb40 0x7f99f80fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.837+0000 7f99fdabd640 1 -- 172.21.15.107:0/1096413919 >> 172.21.15.107:0/1096413919 conn(0x7f99f80fb470 msgr2=0x7f99f80fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.837+0000 7f99fdabd640 1 -- 172.21.15.107:0/1096413919 shutdown_connections 2024-05-09T08:30:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.837+0000 7f99fdabd640 1 -- 172.21.15.107:0/1096413919 wait complete. 2024-05-09T08:30:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.837+0000 7f99fdabd640 1 Processor -- start 2024-05-09T08:30:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.837+0000 7f99fdabd640 1 -- start start 2024-05-09T08:30:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.837+0000 7f99fdabd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99f80ffb40 0x7f99f8197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.837+0000 7f99fdabd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99f8102b10 0x7f99f81980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99fdabd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99f81034d0 0x7f99f819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99fdabd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f99f810d3d0 con 0x7f99f8102b10 2024-05-09T08:30:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99fdabd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f99f810d250 con 0x7f99f80ffb40 2024-05-09T08:30:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99fdabd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f99f810d550 con 0x7f99f81034d0 2024-05-09T08:30:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99f8102b10 0x7f99f81980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99f8102b10 0x7f99f81980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40484/0 (socket says 172.21.15.107:40484) 2024-05-09T08:30:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99effff640 1 -- 172.21.15.107:0/1508059390 learned_addr learned my addr 172.21.15.107:0/1508059390 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99f77fe640 1 --2- 172.21.15.107:0/1508059390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99f80ffb40 0x7f99f8197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99f7fff640 1 --2- 172.21.15.107:0/1508059390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99f81034d0 0x7f99f819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99effff640 1 -- 172.21.15.107:0/1508059390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99f81034d0 msgr2=0x7f99f819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99effff640 1 --2- 172.21.15.107:0/1508059390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99f81034d0 0x7f99f819f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99effff640 1 -- 172.21.15.107:0/1508059390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99f80ffb40 msgr2=0x7f99f8197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99effff640 1 --2- 172.21.15.107:0/1508059390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99f80ffb40 0x7f99f8197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99effff640 1 -- 172.21.15.107:0/1508059390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f819f740 con 0x7f99f8102b10 2024-05-09T08:30:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99f77fe640 1 --2- 172.21.15.107:0/1508059390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99f80ffb40 0x7f99f8197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:30:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99effff640 1 --2- 172.21.15.107:0/1508059390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99f8102b10 0x7f99f81980b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f99e0004770 tx=0x7f99e0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99f57fa640 1 -- 172.21.15.107:0/1508059390 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e003f030 con 0x7f99f8102b10 2024-05-09T08:30:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99f57fa640 1 -- 172.21.15.107:0/1508059390 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99e0003aa0 con 0x7f99f8102b10 2024-05-09T08:30:57.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99fdabd640 1 -- 172.21.15.107:0/1508059390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99f819f9d0 con 0x7f99f8102b10 2024-05-09T08:30:57.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99f57fa640 1 -- 172.21.15.107:0/1508059390 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0035070 con 0x7f99f8102b10 2024-05-09T08:30:57.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.838+0000 7f99fdabd640 1 -- 172.21.15.107:0/1508059390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99f819fec0 con 0x7f99f8102b10 2024-05-09T08:30:57.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.839+0000 7f99fdabd640 1 -- 172.21.15.107:0/1508059390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99f8108db0 con 0x7f99f8102b10 2024-05-09T08:30:57.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.841+0000 7f99f57fa640 1 -- 172.21.15.107:0/1508059390 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f99e0042030 con 0x7f99f8102b10 2024-05-09T08:30:57.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.842+0000 7f99f57fa640 1 --2- 172.21.15.107:0/1508059390 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f99e40a2b90 0x7f99e40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:57.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.842+0000 7f99f57fa640 1 -- 172.21.15.107:0/1508059390 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f99e00ef640 con 0x7f99f8102b10 2024-05-09T08:30:57.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.842+0000 7f99f77fe640 1 --2- 172.21.15.107:0/1508059390 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f99e40a2b90 0x7f99e40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:57.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.842+0000 7f99f77fe640 1 --2- 172.21.15.107:0/1508059390 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f99e40a2b90 0x7f99e40a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f99f8069cd0 tx=0x7f99e8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:57.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:57.847+0000 7f99f57fa640 1 -- 172.21.15.107:0/1508059390 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f99e00bc350 con 0x7f99f8102b10 2024-05-09T08:30:57.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:57 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:57 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:57 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:57 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:57 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/999452494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:57.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:57 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:57 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:57 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:57.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:57 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:30:57.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:57 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:57 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:57.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:57 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2050936598' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:30:57.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:57 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/679451831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:30:57.943 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:57 smithi110 ceph-mon[33622]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:57.973 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:57 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.973 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:57 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.973 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:57 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.973 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:57 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.974 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:57 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/999452494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:57.974 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:57 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.974 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:57 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.974 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:57 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:57.974 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:57 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:30:57.974 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:57 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.975 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:57 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:57.975 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:57 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2050936598' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:30:57.975 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:57 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/679451831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:30:57.975 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:57 smithi157 ceph-mon[33614]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:57.994 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:57 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.994 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:57 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.994 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:57 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.995 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:57 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.995 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:57 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/999452494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:30:57.995 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:57 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.995 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:57 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.995 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:57 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:30:57.995 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:57 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:30:57.996 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:57 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:30:57.996 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:57 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:30:57.996 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:57 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2050936598' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:30:57.996 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:57 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/679451831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:30:57.996 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:57 smithi107 ceph-mon[30122]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:30:58.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.032+0000 7f99fdabd640 1 -- 172.21.15.107:0/1508059390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f99f802d340 con 0x7f99f8102b10 2024-05-09T08:30:58.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.801+0000 7f99f57fa640 1 -- 172.21.15.107:0/1508059390 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f99e00c0650 con 0x7f99f8102b10 2024-05-09T08:30:58.800 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2024-05-09T08:30:58.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.803+0000 7f99fdabd640 1 -- 172.21.15.107:0/1508059390 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f99e40a2b90 msgr2=0x7f99e40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:58.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.803+0000 7f99fdabd640 1 --2- 172.21.15.107:0/1508059390 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f99e40a2b90 0x7f99e40a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f99f8069cd0 tx=0x7f99e8005f90 comp rx=0 tx=0).stop 2024-05-09T08:30:58.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.803+0000 7f99fdabd640 1 -- 172.21.15.107:0/1508059390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99f8102b10 msgr2=0x7f99f81980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:58.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.803+0000 7f99fdabd640 1 --2- 172.21.15.107:0/1508059390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99f8102b10 0x7f99f81980b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f99e0004770 tx=0x7f99e0004270 comp rx=0 tx=0).stop 2024-05-09T08:30:58.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.803+0000 7f99fdabd640 1 -- 172.21.15.107:0/1508059390 shutdown_connections 2024-05-09T08:30:58.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.803+0000 7f99fdabd640 1 --2- 172.21.15.107:0/1508059390 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f99e40a2b90 0x7f99e40a5050 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:58.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.803+0000 7f99fdabd640 1 --2- 172.21.15.107:0/1508059390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99f81034d0 0x7f99f819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:58.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.803+0000 7f99fdabd640 1 --2- 172.21.15.107:0/1508059390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99f8102b10 0x7f99f81980b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:58.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.803+0000 7f99fdabd640 1 --2- 172.21.15.107:0/1508059390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99f80ffb40 0x7f99f8197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:58.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.803+0000 7f99fdabd640 1 -- 172.21.15.107:0/1508059390 >> 172.21.15.107:0/1508059390 conn(0x7f99f80fb470 msgr2=0x7f99f8105f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:58.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.804+0000 7f99fdabd640 1 -- 172.21.15.107:0/1508059390 shutdown_connections 2024-05-09T08:30:58.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.804+0000 7f99fdabd640 1 -- 172.21.15.107:0/1508059390 wait complete. 2024-05-09T08:30:58.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-05-09T08:30:58.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.927+0000 7fd1c6876640 1 -- 172.21.15.107:0/1165104947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1c0100750 msgr2=0x7fd1c0100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:58.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.927+0000 7fd1c6876640 1 --2- 172.21.15.107:0/1165104947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1c0100750 0x7fd1c0100b70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0009900 tx=0x7fd1b002a410 comp rx=0 tx=0).stop 2024-05-09T08:30:58.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.927+0000 7fd1c6876640 1 -- 172.21.15.107:0/1165104947 shutdown_connections 2024-05-09T08:30:58.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.927+0000 7fd1c6876640 1 --2- 172.21.15.107:0/1165104947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1c0102300 0x7fd1c0108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:58.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.927+0000 7fd1c6876640 1 --2- 172.21.15.107:0/1165104947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1c0101940 0x7fd1c0101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:58.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.927+0000 7fd1c6876640 1 --2- 172.21.15.107:0/1165104947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1c0100750 0x7fd1c0100b70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:58.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.927+0000 7fd1c6876640 1 -- 172.21.15.107:0/1165104947 >> 172.21.15.107:0/1165104947 conn(0x7fd1c00fbcc0 msgr2=0x7fd1c00fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:58.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.927+0000 7fd1c6876640 1 -- 172.21.15.107:0/1165104947 shutdown_connections 2024-05-09T08:30:58.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.927+0000 7fd1c6876640 1 -- 172.21.15.107:0/1165104947 wait complete. 2024-05-09T08:30:58.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.927+0000 7fd1c6876640 1 Processor -- start 2024-05-09T08:30:58.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.927+0000 7fd1c6876640 1 -- start start 2024-05-09T08:30:58.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.927+0000 7fd1c6876640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1c0100750 0x7fd1c0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:58.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1c6876640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1c0101940 0x7fd1c0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:58.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1c6876640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1c0102300 0x7fd1c019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:58.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1c6876640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c010c270 con 0x7fd1c0101940 2024-05-09T08:30:58.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1c6876640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c010c0f0 con 0x7fd1c0100750 2024-05-09T08:30:58.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1c6876640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c010c3f0 con 0x7fd1c0102300 2024-05-09T08:30:58.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1b7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1c0101940 0x7fd1c0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:58.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1b7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1c0101940 0x7fd1c0198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40506/0 (socket says 172.21.15.107:40506) 2024-05-09T08:30:58.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1b7fff640 1 -- 172.21.15.107:0/671966734 learned_addr learned my addr 172.21.15.107:0/671966734 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:58.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1bffff640 1 --2- 172.21.15.107:0/671966734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1c0100750 0x7fd1c0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:58.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1b7fff640 1 -- 172.21.15.107:0/671966734 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1c0102300 msgr2=0x7fd1c019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:58.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1b7fff640 1 --2- 172.21.15.107:0/671966734 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1c0102300 0x7fd1c019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:58.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1b7fff640 1 -- 172.21.15.107:0/671966734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1c0100750 msgr2=0x7fd1c0197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:58.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1b7fff640 1 --2- 172.21.15.107:0/671966734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1c0100750 0x7fd1c0197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:58.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1b7fff640 1 -- 172.21.15.107:0/671966734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c019f6e0 con 0x7fd1c0101940 2024-05-09T08:30:58.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1b7fff640 1 --2- 172.21.15.107:0/671966734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1c0101940 0x7fd1c0198000 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800e9f0 tx=0x7fd1a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:58.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1bdffb640 1 -- 172.21.15.107:0/671966734 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a800cd80 con 0x7fd1c0101940 2024-05-09T08:30:58.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1c6876640 1 -- 172.21.15.107:0/671966734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1c019f9d0 con 0x7fd1c0101940 2024-05-09T08:30:58.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1bdffb640 1 -- 172.21.15.107:0/671966734 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1a80054d0 con 0x7fd1c0101940 2024-05-09T08:30:58.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1bdffb640 1 -- 172.21.15.107:0/671966734 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8010490 con 0x7fd1c0101940 2024-05-09T08:30:58.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.928+0000 7fd1c6876640 1 -- 172.21.15.107:0/671966734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1c019ff20 con 0x7fd1c0101940 2024-05-09T08:30:58.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.929+0000 7fd1c6876640 1 -- 172.21.15.107:0/671966734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1c01a0210 con 0x7fd1c0101940 2024-05-09T08:30:58.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.930+0000 7fd1bdffb640 1 -- 172.21.15.107:0/671966734 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd1a8005000 con 0x7fd1c0101940 2024-05-09T08:30:58.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.931+0000 7fd1bdffb640 1 --2- 172.21.15.107:0/671966734 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd1ac0a2b90 0x7fd1ac0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:58.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.931+0000 7fd1bdffb640 1 -- 172.21.15.107:0/671966734 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7fd1a80cf360 con 0x7fd1c0101940 2024-05-09T08:30:58.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.931+0000 7fd1bffff640 1 --2- 172.21.15.107:0/671966734 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd1ac0a2b90 0x7fd1ac0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:58.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.932+0000 7fd1bffff640 1 --2- 172.21.15.107:0/671966734 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd1ac0a2b90 0x7fd1ac0a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0004770 tx=0x7fd1b0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:58.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:58.935+0000 7fd1bdffb640 1 -- 172.21.15.107:0/671966734 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd1a8053740 con 0x7fd1c0101940 2024-05-09T08:30:59.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:58 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3325112687' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-09T08:30:59.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:58 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1508059390' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-09T08:30:59.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:58 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3325112687' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-09T08:30:59.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:58 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1508059390' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-09T08:30:59.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.112+0000 7fd1c6876640 1 -- 172.21.15.107:0/671966734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fd1c01a03e0 con 0x7fd1c0101940 2024-05-09T08:30:59.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:58 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3325112687' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-09T08:30:59.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:58 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1508059390' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-09T08:30:59.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.805+0000 7fd1bdffb640 1 -- 172.21.15.107:0/671966734 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7fd1a80052f0 con 0x7fd1c0101940 2024-05-09T08:30:59.805 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2024-05-09T08:30:59.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.808+0000 7fd1c6876640 1 -- 172.21.15.107:0/671966734 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd1ac0a2b90 msgr2=0x7fd1ac0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:59.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.808+0000 7fd1c6876640 1 --2- 172.21.15.107:0/671966734 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd1ac0a2b90 0x7fd1ac0a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0004770 tx=0x7fd1b0000bc0 comp rx=0 tx=0).stop 2024-05-09T08:30:59.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.808+0000 7fd1c6876640 1 -- 172.21.15.107:0/671966734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1c0101940 msgr2=0x7fd1c0198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:59.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.808+0000 7fd1c6876640 1 --2- 172.21.15.107:0/671966734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1c0101940 0x7fd1c0198000 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800e9f0 tx=0x7fd1a800edf0 comp rx=0 tx=0).stop 2024-05-09T08:30:59.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.808+0000 7fd1c6876640 1 -- 172.21.15.107:0/671966734 shutdown_connections 2024-05-09T08:30:59.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.808+0000 7fd1c6876640 1 --2- 172.21.15.107:0/671966734 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd1ac0a2b90 0x7fd1ac0a5050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:59.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.808+0000 7fd1c6876640 1 --2- 172.21.15.107:0/671966734 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1c0102300 0x7fd1c019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:59.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.808+0000 7fd1c6876640 1 --2- 172.21.15.107:0/671966734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1c0101940 0x7fd1c0198000 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:59.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.808+0000 7fd1c6876640 1 --2- 172.21.15.107:0/671966734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1c0100750 0x7fd1c0197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:59.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.808+0000 7fd1c6876640 1 -- 172.21.15.107:0/671966734 >> 172.21.15.107:0/671966734 conn(0x7fd1c00fbcc0 msgr2=0x7fd1c00fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:59.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.808+0000 7fd1c6876640 1 -- 172.21.15.107:0/671966734 shutdown_connections 2024-05-09T08:30:59.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.808+0000 7fd1c6876640 1 -- 172.21.15.107:0/671966734 wait complete. 2024-05-09T08:30:59.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T08:30:59.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.933+0000 7f3fd03a9640 1 -- 172.21.15.107:0/4142193799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fc8104ca0 msgr2=0x7f3fc8107090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:59.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.933+0000 7f3fd03a9640 1 --2- 172.21.15.107:0/4142193799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fc8104ca0 0x7f3fc8107090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009900 tx=0x7f3fb802a410 comp rx=0 tx=0).stop 2024-05-09T08:30:59.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fd03a9640 1 -- 172.21.15.107:0/4142193799 shutdown_connections 2024-05-09T08:30:59.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fd03a9640 1 --2- 172.21.15.107:0/4142193799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fc81075d0 0x7f3fc81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:59.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fd03a9640 1 --2- 172.21.15.107:0/4142193799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fc8104ca0 0x7f3fc8107090 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:59.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fd03a9640 1 --2- 172.21.15.107:0/4142193799 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3fc8102370 0x7f3fc8104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:59.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fd03a9640 1 -- 172.21.15.107:0/4142193799 >> 172.21.15.107:0/4142193799 conn(0x7f3fc80fbce0 msgr2=0x7f3fc80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:30:59.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fd03a9640 1 -- 172.21.15.107:0/4142193799 shutdown_connections 2024-05-09T08:30:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fd03a9640 1 -- 172.21.15.107:0/4142193799 wait complete. 2024-05-09T08:30:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fd03a9640 1 Processor -- start 2024-05-09T08:30:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fd03a9640 1 -- start start 2024-05-09T08:30:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fd03a9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3fc8102370 0x7f3fc8199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fd03a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fc8104ca0 0x7f3fc819a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fd03a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fc81075d0 0x7f3fc81a1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fd03a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc810d470 con 0x7f3fc81075d0 2024-05-09T08:30:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fd03a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc810d2f0 con 0x7f3fc8104ca0 2024-05-09T08:30:59.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fd03a9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc810d5f0 con 0x7f3fc8102370 2024-05-09T08:30:59.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fce946640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fc81075d0 0x7f3fc81a1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:59.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fce946640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fc81075d0 0x7f3fc81a1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40526/0 (socket says 172.21.15.107:40526) 2024-05-09T08:30:59.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.934+0000 7f3fce946640 1 -- 172.21.15.107:0/555357614 learned_addr learned my addr 172.21.15.107:0/555357614 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:30:59.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.935+0000 7f3fce145640 1 --2- 172.21.15.107:0/555357614 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3fc8102370 0x7f3fc8199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:59.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.935+0000 7f3fce946640 1 -- 172.21.15.107:0/555357614 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3fc8102370 msgr2=0x7f3fc8199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:59.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.935+0000 7f3fce946640 1 --2- 172.21.15.107:0/555357614 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3fc8102370 0x7f3fc8199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:59.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.935+0000 7f3fce946640 1 -- 172.21.15.107:0/555357614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fc8104ca0 msgr2=0x7f3fc819a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:30:59.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.935+0000 7f3fce946640 1 --2- 172.21.15.107:0/555357614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fc8104ca0 0x7f3fc819a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:30:59.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.935+0000 7f3fce946640 1 -- 172.21.15.107:0/555357614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fc81a1aa0 con 0x7f3fc81075d0 2024-05-09T08:30:59.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.935+0000 7f3fce946640 1 --2- 172.21.15.107:0/555357614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fc81075d0 0x7f3fc81a1430 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3fc000b970 tx=0x7f3fc000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:59.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.935+0000 7f3fbf7fe640 1 -- 172.21.15.107:0/555357614 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc0019070 con 0x7f3fc81075d0 2024-05-09T08:30:59.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.935+0000 7f3fbf7fe640 1 -- 172.21.15.107:0/555357614 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fc0014440 con 0x7f3fc81075d0 2024-05-09T08:30:59.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.935+0000 7f3fd03a9640 1 -- 172.21.15.107:0/555357614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fc81a1d90 con 0x7f3fc81075d0 2024-05-09T08:30:59.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.935+0000 7f3fbf7fe640 1 -- 172.21.15.107:0/555357614 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc00075d0 con 0x7f3fc81075d0 2024-05-09T08:30:59.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.935+0000 7f3fd03a9640 1 -- 172.21.15.107:0/555357614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fc81a2310 con 0x7f3fc81075d0 2024-05-09T08:30:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.936+0000 7f3fd03a9640 1 -- 172.21.15.107:0/555357614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fc81a25a0 con 0x7f3fc81075d0 2024-05-09T08:30:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.937+0000 7f3fbf7fe640 1 -- 172.21.15.107:0/555357614 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3fc001c070 con 0x7f3fc81075d0 2024-05-09T08:30:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.938+0000 7f3fbf7fe640 1 --2- 172.21.15.107:0/555357614 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3fb40a2b90 0x7f3fb40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:30:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.938+0000 7f3fce145640 1 --2- 172.21.15.107:0/555357614 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3fb40a2b90 0x7f3fb40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:30:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.938+0000 7f3fbf7fe640 1 -- 172.21.15.107:0/555357614 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f3fc00ce970 con 0x7f3fc81075d0 2024-05-09T08:30:59.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.938+0000 7f3fce145640 1 --2- 172.21.15.107:0/555357614 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3fb40a2b90 0x7f3fb40a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4009890 tx=0x7f3fc4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:30:59.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:30:59.941+0000 7f3fbf7fe640 1 -- 172.21.15.107:0/555357614 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3fc009ba80 con 0x7f3fc81075d0 2024-05-09T08:31:00.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:59 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:59 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:59 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:59 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:59 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:59 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:59 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:31:00.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:59 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:31:00.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:59 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:59 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1508059390' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-09T08:31:00.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:59 smithi107 ceph-mon[30122]: osdmap e82: 12 total, 12 up, 12 in 2024-05-09T08:31:00.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:59 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/671966734' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-09T08:31:00.031 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:30:59 smithi107 ceph-mon[30122]: pgmap v284: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:31:00.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:59 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:59 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:59 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:59 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:59 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.095 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:59 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.095 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:59 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:31:00.095 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:59 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:31:00.095 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:59 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.095 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:59 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1508059390' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-09T08:31:00.095 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:59 smithi157 ceph-mon[33614]: osdmap e82: 12 total, 12 up, 12 in 2024-05-09T08:31:00.096 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:59 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/671966734' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-09T08:31:00.096 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:30:59 smithi157 ceph-mon[33614]: pgmap v284: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:31:00.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:59 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:59 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:59 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:59 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:59 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:59 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:59 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:31:00.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:59 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:31:00.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:59 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:31:00.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:59 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1508059390' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-09T08:31:00.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:59 smithi110 ceph-mon[33622]: osdmap e82: 12 total, 12 up, 12 in 2024-05-09T08:31:00.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:59 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/671966734' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-09T08:31:00.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:30:59 smithi110 ceph-mon[33622]: pgmap v284: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:31:00.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.155+0000 7f3fd03a9640 1 -- 172.21.15.107:0/555357614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f3fc81a2770 con 0x7f3fc81075d0 2024-05-09T08:31:00.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.807+0000 7f3fbf7fe640 1 -- 172.21.15.107:0/555357614 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f3fc009b380 con 0x7f3fc81075d0 2024-05-09T08:31:00.806 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 2 and data pool 3 2024-05-09T08:31:00.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.810+0000 7f3fd03a9640 1 -- 172.21.15.107:0/555357614 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3fb40a2b90 msgr2=0x7f3fb40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:00.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.810+0000 7f3fd03a9640 1 --2- 172.21.15.107:0/555357614 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3fb40a2b90 0x7f3fb40a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4009890 tx=0x7f3fc4008040 comp rx=0 tx=0).stop 2024-05-09T08:31:00.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.810+0000 7f3fd03a9640 1 -- 172.21.15.107:0/555357614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fc81075d0 msgr2=0x7f3fc81a1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:00.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.810+0000 7f3fd03a9640 1 --2- 172.21.15.107:0/555357614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fc81075d0 0x7f3fc81a1430 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3fc000b970 tx=0x7f3fc000be30 comp rx=0 tx=0).stop 2024-05-09T08:31:00.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.810+0000 7f3fd03a9640 1 -- 172.21.15.107:0/555357614 shutdown_connections 2024-05-09T08:31:00.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.810+0000 7f3fd03a9640 1 --2- 172.21.15.107:0/555357614 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3fb40a2b90 0x7f3fb40a5050 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.810+0000 7f3fd03a9640 1 --2- 172.21.15.107:0/555357614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fc81075d0 0x7f3fc81a1430 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.810+0000 7f3fd03a9640 1 --2- 172.21.15.107:0/555357614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fc8104ca0 0x7f3fc819a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.810+0000 7f3fd03a9640 1 --2- 172.21.15.107:0/555357614 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3fc8102370 0x7f3fc8199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.810+0000 7f3fd03a9640 1 -- 172.21.15.107:0/555357614 >> 172.21.15.107:0/555357614 conn(0x7f3fc80fbce0 msgr2=0x7f3fc8100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:00.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.810+0000 7f3fd03a9640 1 -- 172.21.15.107:0/555357614 shutdown_connections 2024-05-09T08:31:00.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.810+0000 7f3fd03a9640 1 -- 172.21.15.107:0/555357614 wait complete. 2024-05-09T08:31:00.825 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-05-09T08:31:00.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-05-09T08:31:00.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 -- 172.21.15.107:0/761735104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9cc101a50 msgr2=0x7fc9cc108370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:00.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 --2- 172.21.15.107:0/761735104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9cc101a50 0x7fc9cc108370 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4009900 tx=0x7fc9c402a410 comp rx=0 tx=0).stop 2024-05-09T08:31:00.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 -- 172.21.15.107:0/761735104 shutdown_connections 2024-05-09T08:31:00.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 --2- 172.21.15.107:0/761735104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9cc101a50 0x7fc9cc108370 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 --2- 172.21.15.107:0/761735104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9cc101090 0x7fc9cc101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 --2- 172.21.15.107:0/761735104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9cc0ffeb0 0x7fc9cc1002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 -- 172.21.15.107:0/761735104 >> 172.21.15.107:0/761735104 conn(0x7fc9cc0fb470 msgr2=0x7fc9cc0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:00.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 -- 172.21.15.107:0/761735104 shutdown_connections 2024-05-09T08:31:00.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 -- 172.21.15.107:0/761735104 wait complete. 2024-05-09T08:31:00.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 Processor -- start 2024-05-09T08:31:00.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 -- start start 2024-05-09T08:31:00.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9cc0ffeb0 0x7fc9cc0fd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:00.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9cc101090 0x7fc9cc19ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:00.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9cc101a50 0x7fc9cc1a12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:00.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc9cc10bb40 con 0x7fc9cc101090 2024-05-09T08:31:00.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc9cc10b9c0 con 0x7fc9cc0ffeb0 2024-05-09T08:31:00.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.935+0000 7fc9d3514640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc9cc10bcc0 con 0x7fc9cc101a50 2024-05-09T08:31:00.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9d0aaf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9cc101090 0x7fc9cc19ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:00.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9d0aaf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9cc101090 0x7fc9cc19ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40124/0 (socket says 172.21.15.107:40124) 2024-05-09T08:31:00.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9d0aaf640 1 -- 172.21.15.107:0/2424796648 learned_addr learned my addr 172.21.15.107:0/2424796648 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:00.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9d12b0640 1 --2- 172.21.15.107:0/2424796648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9cc0ffeb0 0x7fc9cc0fd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:00.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9d1ab1640 1 --2- 172.21.15.107:0/2424796648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9cc101a50 0x7fc9cc1a12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:00.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9d0aaf640 1 -- 172.21.15.107:0/2424796648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9cc101a50 msgr2=0x7fc9cc1a12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:00.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9d0aaf640 1 --2- 172.21.15.107:0/2424796648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9cc101a50 0x7fc9cc1a12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9d0aaf640 1 -- 172.21.15.107:0/2424796648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9cc0ffeb0 msgr2=0x7fc9cc0fd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:00.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9d0aaf640 1 --2- 172.21.15.107:0/2424796648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9cc0ffeb0 0x7fc9cc0fd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:00.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9d0aaf640 1 -- 172.21.15.107:0/2424796648 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9cc1a1a30 con 0x7fc9cc101090 2024-05-09T08:31:00.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9d1ab1640 1 --2- 172.21.15.107:0/2424796648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9cc101a50 0x7fc9cc1a12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:00.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9d0aaf640 1 --2- 172.21.15.107:0/2424796648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9cc101090 0x7fc9cc19ed90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc9c000ba50 tx=0x7fc9c000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:00.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9ca7fc640 1 -- 172.21.15.107:0/2424796648 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c0019070 con 0x7fc9cc101090 2024-05-09T08:31:00.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9ca7fc640 1 -- 172.21.15.107:0/2424796648 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc9c000cbe0 con 0x7fc9cc101090 2024-05-09T08:31:00.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9ca7fc640 1 -- 172.21.15.107:0/2424796648 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c00075d0 con 0x7fc9cc101090 2024-05-09T08:31:00.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9d3514640 1 -- 172.21.15.107:0/2424796648 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9cc1a1d20 con 0x7fc9cc101090 2024-05-09T08:31:00.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.936+0000 7fc9d3514640 1 -- 172.21.15.107:0/2424796648 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9cc1a2290 con 0x7fc9cc101090 2024-05-09T08:31:00.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.937+0000 7fc9d3514640 1 -- 172.21.15.107:0/2424796648 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9cc107af0 con 0x7fc9cc101090 2024-05-09T08:31:00.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.938+0000 7fc9ca7fc640 1 -- 172.21.15.107:0/2424796648 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc9c001c070 con 0x7fc9cc101090 2024-05-09T08:31:00.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.939+0000 7fc9ca7fc640 1 --2- 172.21.15.107:0/2424796648 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc9b40a2bc0 0x7fc9b40a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:00.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.939+0000 7fc9d12b0640 1 --2- 172.21.15.107:0/2424796648 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc9b40a2bc0 0x7fc9b40a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:00.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.940+0000 7fc9ca7fc640 1 -- 172.21.15.107:0/2424796648 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7fc9c00ce530 con 0x7fc9cc101090 2024-05-09T08:31:00.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.940+0000 7fc9d12b0640 1 --2- 172.21.15.107:0/2424796648 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc9b40a2bc0 0x7fc9b40a5080 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc100c00 tx=0x7fc9bc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:00.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:00.944+0000 7fc9ca7fc640 1 -- 172.21.15.107:0/2424796648 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc9c009b5e0 con 0x7fc9cc101090 2024-05-09T08:31:01.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:00 smithi107 ceph-mon[30122]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:31:01.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:00 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/671966734' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-09T08:31:01.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:00 smithi107 ceph-mon[30122]: osdmap e83: 12 total, 12 up, 12 in 2024-05-09T08:31:01.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:00 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/555357614' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-09T08:31:01.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:00 smithi107 ceph-mon[30122]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-05-09T08:31:01.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:00 smithi157 ceph-mon[33614]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:31:01.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:00 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/671966734' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-09T08:31:01.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:00 smithi157 ceph-mon[33614]: osdmap e83: 12 total, 12 up, 12 in 2024-05-09T08:31:01.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:00 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/555357614' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-09T08:31:01.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:00 smithi157 ceph-mon[33614]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-05-09T08:31:01.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:00 smithi110 ceph-mon[33622]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:31:01.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:00 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/671966734' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-09T08:31:01.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:00 smithi110 ceph-mon[33622]: osdmap e83: 12 total, 12 up, 12 in 2024-05-09T08:31:01.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:00 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/555357614' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-09T08:31:01.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:00 smithi110 ceph-mon[33622]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-05-09T08:31:01.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.136+0000 7fc9d3514640 1 -- 172.21.15.107:0/2424796648 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fc9cc062430 con 0x7fc9cc101090 2024-05-09T08:31:01.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.673+0000 7fc9ca7fc640 1 -- 172.21.15.107:0/2424796648 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7fc9c009aee0 con 0x7fc9cc101090 2024-05-09T08:31:01.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.676+0000 7fc9d3514640 1 -- 172.21.15.107:0/2424796648 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc9b40a2bc0 msgr2=0x7fc9b40a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:01.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.676+0000 7fc9d3514640 1 --2- 172.21.15.107:0/2424796648 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc9b40a2bc0 0x7fc9b40a5080 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc100c00 tx=0x7fc9bc008040 comp rx=0 tx=0).stop 2024-05-09T08:31:01.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.676+0000 7fc9d3514640 1 -- 172.21.15.107:0/2424796648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9cc101090 msgr2=0x7fc9cc19ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.676+0000 7fc9d3514640 1 --2- 172.21.15.107:0/2424796648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9cc101090 0x7fc9cc19ed90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc9c000ba50 tx=0x7fc9c000bf10 comp rx=0 tx=0).stop 2024-05-09T08:31:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.677+0000 7fc9d3514640 1 -- 172.21.15.107:0/2424796648 shutdown_connections 2024-05-09T08:31:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.677+0000 7fc9d3514640 1 --2- 172.21.15.107:0/2424796648 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc9b40a2bc0 0x7fc9b40a5080 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.677+0000 7fc9d3514640 1 --2- 172.21.15.107:0/2424796648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9cc101a50 0x7fc9cc1a12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.677+0000 7fc9d3514640 1 --2- 172.21.15.107:0/2424796648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9cc101090 0x7fc9cc19ed90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.677+0000 7fc9d3514640 1 --2- 172.21.15.107:0/2424796648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9cc0ffeb0 0x7fc9cc0fd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.677+0000 7fc9d3514640 1 -- 172.21.15.107:0/2424796648 >> 172.21.15.107:0/2424796648 conn(0x7fc9cc0fb470 msgr2=0x7fc9cc102840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.677+0000 7fc9d3514640 1 -- 172.21.15.107:0/2424796648 shutdown_connections 2024-05-09T08:31:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.677+0000 7fc9d3514640 1 -- 172.21.15.107:0/2424796648 wait complete. 2024-05-09T08:31:01.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T08:31:01.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.806+0000 7f948dcd2640 1 -- 172.21.15.107:0/3881050203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9488104ce0 msgr2=0x7f94881070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:01.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.806+0000 7f948dcd2640 1 --2- 172.21.15.107:0/3881050203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9488104ce0 0x7f94881070d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9470009900 tx=0x7f947002a410 comp rx=0 tx=0).stop 2024-05-09T08:31:01.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.806+0000 7f948dcd2640 1 -- 172.21.15.107:0/3881050203 shutdown_connections 2024-05-09T08:31:01.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.806+0000 7f948dcd2640 1 --2- 172.21.15.107:0/3881050203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9488107610 0x7f9488109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:01.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.806+0000 7f948dcd2640 1 --2- 172.21.15.107:0/3881050203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9488104ce0 0x7f94881070d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:01.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.806+0000 7f948dcd2640 1 --2- 172.21.15.107:0/3881050203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94881023b0 0x7f94881047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:01.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.806+0000 7f948dcd2640 1 -- 172.21.15.107:0/3881050203 >> 172.21.15.107:0/3881050203 conn(0x7f94880fbce0 msgr2=0x7f94880fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:01.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.806+0000 7f948dcd2640 1 -- 172.21.15.107:0/3881050203 shutdown_connections 2024-05-09T08:31:01.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.806+0000 7f948dcd2640 1 -- 172.21.15.107:0/3881050203 wait complete. 2024-05-09T08:31:01.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.807+0000 7f948dcd2640 1 Processor -- start 2024-05-09T08:31:01.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.807+0000 7f948dcd2640 1 -- start start 2024-05-09T08:31:01.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.807+0000 7f948dcd2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94881023b0 0x7f9488197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:01.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.807+0000 7f948dcd2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9488104ce0 0x7f94881981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:01.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.807+0000 7f948dcd2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9488107610 0x7f948819f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:01.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.807+0000 7f948dcd2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f948810d480 con 0x7f9488107610 2024-05-09T08:31:01.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.807+0000 7f948dcd2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f948810d300 con 0x7f94881023b0 2024-05-09T08:31:01.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.807+0000 7f948dcd2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f948810d600 con 0x7f9488104ce0 2024-05-09T08:31:01.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.808+0000 7f9487fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9488107610 0x7f948819f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:01.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.808+0000 7f9487fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9488107610 0x7f948819f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40140/0 (socket says 172.21.15.107:40140) 2024-05-09T08:31:01.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.808+0000 7f947effd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9488104ce0 0x7f94881981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:01.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.808+0000 7f9487fff640 1 -- 172.21.15.107:0/55083485 learned_addr learned my addr 172.21.15.107:0/55083485 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:01.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.808+0000 7f94877fe640 1 --2- 172.21.15.107:0/55083485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94881023b0 0x7f9488197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:01.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.808+0000 7f9487fff640 1 -- 172.21.15.107:0/55083485 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9488104ce0 msgr2=0x7f94881981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:01.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.808+0000 7f9487fff640 1 --2- 172.21.15.107:0/55083485 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9488104ce0 0x7f94881981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:01.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.808+0000 7f9487fff640 1 -- 172.21.15.107:0/55083485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94881023b0 msgr2=0x7f9488197c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:01.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.808+0000 7f9487fff640 1 --2- 172.21.15.107:0/55083485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94881023b0 0x7f9488197c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:01.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.808+0000 7f9487fff640 1 -- 172.21.15.107:0/55083485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f948819f870 con 0x7f9488107610 2024-05-09T08:31:01.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.808+0000 7f9487fff640 1 --2- 172.21.15.107:0/55083485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9488107610 0x7f948819f1d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f948000b970 tx=0x7f948000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:01.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.811+0000 7f94857fa640 1 -- 172.21.15.107:0/55083485 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480019070 con 0x7f9488107610 2024-05-09T08:31:01.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.811+0000 7f94857fa640 1 -- 172.21.15.107:0/55083485 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9480014440 con 0x7f9488107610 2024-05-09T08:31:01.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.811+0000 7f94857fa640 1 -- 172.21.15.107:0/55083485 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94800075d0 con 0x7f9488107610 2024-05-09T08:31:01.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.811+0000 7f948dcd2640 1 -- 172.21.15.107:0/55083485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f948819fb60 con 0x7f9488107610 2024-05-09T08:31:01.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.811+0000 7f948dcd2640 1 -- 172.21.15.107:0/55083485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94881a00b0 con 0x7f9488107610 2024-05-09T08:31:01.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.812+0000 7f948dcd2640 1 -- 172.21.15.107:0/55083485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94880fd7d0 con 0x7f9488107610 2024-05-09T08:31:01.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.813+0000 7f94857fa640 1 -- 172.21.15.107:0/55083485 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f948001c070 con 0x7f9488107610 2024-05-09T08:31:01.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.813+0000 7f94857fa640 1 --2- 172.21.15.107:0/55083485 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f94740a2c10 0x7f94740a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:01.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.814+0000 7f94877fe640 1 --2- 172.21.15.107:0/55083485 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f94740a2c10 0x7f94740a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:01.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.814+0000 7f94857fa640 1 -- 172.21.15.107:0/55083485 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7f94800ce5c0 con 0x7f9488107610 2024-05-09T08:31:01.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.814+0000 7f94877fe640 1 --2- 172.21.15.107:0/55083485 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f94740a2c10 0x7f94740a50d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9488199130 tx=0x7f9478009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:01.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.819+0000 7f94857fa640 1 -- 172.21.15.107:0/55083485 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f948009aea0 con 0x7f9488107610 2024-05-09T08:31:01.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:01.998+0000 7f948dcd2640 1 -- 172.21.15.107:0/55083485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f9488062430 con 0x7f9488107610 2024-05-09T08:31:02.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:01 smithi107 ceph-mon[30122]: osdmap e84: 12 total, 12 up, 12 in 2024-05-09T08:31:02.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:01 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/555357614' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-09T08:31:02.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:01 smithi107 ceph-mon[30122]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-05-09T08:31:02.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:01 smithi107 ceph-mon[30122]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-05-09T08:31:02.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:01 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2424796648' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-09T08:31:02.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:01 smithi107 ceph-mon[30122]: pgmap v287: 97 pgs: 17 creating+peering, 38 active+clean, 42 unknown; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 0 op/s 2024-05-09T08:31:02.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:01 smithi157 ceph-mon[33614]: osdmap e84: 12 total, 12 up, 12 in 2024-05-09T08:31:02.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:01 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/555357614' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-09T08:31:02.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:01 smithi157 ceph-mon[33614]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-05-09T08:31:02.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:01 smithi157 ceph-mon[33614]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-05-09T08:31:02.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:01 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2424796648' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-09T08:31:02.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:01 smithi157 ceph-mon[33614]: pgmap v287: 97 pgs: 17 creating+peering, 38 active+clean, 42 unknown; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 0 op/s 2024-05-09T08:31:02.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:01 smithi110 ceph-mon[33622]: osdmap e84: 12 total, 12 up, 12 in 2024-05-09T08:31:02.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:01 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/555357614' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-09T08:31:02.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:01 smithi110 ceph-mon[33622]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-05-09T08:31:02.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:01 smithi110 ceph-mon[33622]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-05-09T08:31:02.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:01 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2424796648' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-09T08:31:02.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:01 smithi110 ceph-mon[33622]: pgmap v287: 97 pgs: 17 creating+peering, 38 active+clean, 42 unknown; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 0 op/s 2024-05-09T08:31:03.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:02 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2424796648' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-09T08:31:03.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:02 smithi107 ceph-mon[30122]: osdmap e85: 12 total, 12 up, 12 in 2024-05-09T08:31:03.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:02 smithi107 ceph-mon[30122]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:31:03.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:02 smithi107 ceph-mon[30122]: Cluster is now healthy 2024-05-09T08:31:03.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:02 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] up:active 2024-05-09T08:31:03.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:02 smithi107 ceph-mon[30122]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-09T08:31:03.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:02 smithi107 ceph-mon[30122]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-09T08:31:03.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:02 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/55083485' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-09T08:31:03.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:02 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2424796648' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-09T08:31:03.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:02 smithi157 ceph-mon[33614]: osdmap e85: 12 total, 12 up, 12 in 2024-05-09T08:31:03.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:02 smithi157 ceph-mon[33614]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:31:03.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:02 smithi157 ceph-mon[33614]: Cluster is now healthy 2024-05-09T08:31:03.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:02 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] up:active 2024-05-09T08:31:03.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:02 smithi157 ceph-mon[33614]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-09T08:31:03.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:02 smithi157 ceph-mon[33614]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-09T08:31:03.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:02 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/55083485' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-09T08:31:03.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:02 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2424796648' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-09T08:31:03.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:02 smithi110 ceph-mon[33622]: osdmap e85: 12 total, 12 up, 12 in 2024-05-09T08:31:03.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:02 smithi110 ceph-mon[33622]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:31:03.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:02 smithi110 ceph-mon[33622]: Cluster is now healthy 2024-05-09T08:31:03.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:02 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] up:active 2024-05-09T08:31:03.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:02 smithi110 ceph-mon[33622]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-09T08:31:03.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:02 smithi110 ceph-mon[33622]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-09T08:31:03.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:02 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/55083485' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-09T08:31:03.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.833+0000 7f94857fa640 1 -- 172.21.15.107:0/55083485 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7f948009f1a0 con 0x7f9488107610 2024-05-09T08:31:03.833 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data_ec' created 2024-05-09T08:31:03.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.837+0000 7f948dcd2640 1 -- 172.21.15.107:0/55083485 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f94740a2c10 msgr2=0x7f94740a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:03.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.837+0000 7f948dcd2640 1 --2- 172.21.15.107:0/55083485 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f94740a2c10 0x7f94740a50d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9488199130 tx=0x7f9478009110 comp rx=0 tx=0).stop 2024-05-09T08:31:03.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.837+0000 7f948dcd2640 1 -- 172.21.15.107:0/55083485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9488107610 msgr2=0x7f948819f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:03.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.837+0000 7f948dcd2640 1 --2- 172.21.15.107:0/55083485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9488107610 0x7f948819f1d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f948000b970 tx=0x7f948000bd70 comp rx=0 tx=0).stop 2024-05-09T08:31:03.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.837+0000 7f948dcd2640 1 -- 172.21.15.107:0/55083485 shutdown_connections 2024-05-09T08:31:03.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.837+0000 7f948dcd2640 1 --2- 172.21.15.107:0/55083485 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f94740a2c10 0x7f94740a50d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:03.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.837+0000 7f948dcd2640 1 --2- 172.21.15.107:0/55083485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9488107610 0x7f948819f1d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:03.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.837+0000 7f948dcd2640 1 --2- 172.21.15.107:0/55083485 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9488104ce0 0x7f94881981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:03.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.837+0000 7f948dcd2640 1 --2- 172.21.15.107:0/55083485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94881023b0 0x7f9488197c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:03.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.837+0000 7f948dcd2640 1 -- 172.21.15.107:0/55083485 >> 172.21.15.107:0/55083485 conn(0x7f94880fbce0 msgr2=0x7f9488104240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:03.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.837+0000 7f948dcd2640 1 -- 172.21.15.107:0/55083485 shutdown_connections 2024-05-09T08:31:03.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.837+0000 7f948dcd2640 1 -- 172.21.15.107:0/55083485 wait complete. 2024-05-09T08:31:03.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-05-09T08:31:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.966+0000 7fad7705d640 1 -- 172.21.15.107:0/3014493125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad70103430 msgr2=0x7fad70109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.966+0000 7fad7705d640 1 --2- 172.21.15.107:0/3014493125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad70103430 0x7fad70109cc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fad68009ca0 tx=0x7fad6802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:31:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.966+0000 7fad7705d640 1 -- 172.21.15.107:0/3014493125 shutdown_connections 2024-05-09T08:31:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.966+0000 7fad7705d640 1 --2- 172.21.15.107:0/3014493125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad70103430 0x7fad70109cc0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.966+0000 7fad7705d640 1 --2- 172.21.15.107:0/3014493125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad70102a70 0x7fad70102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.966+0000 7fad7705d640 1 --2- 172.21.15.107:0/3014493125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad70101880 0x7fad70101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.966+0000 7fad7705d640 1 -- 172.21.15.107:0/3014493125 >> 172.21.15.107:0/3014493125 conn(0x7fad700fce10 msgr2=0x7fad700ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.966+0000 7fad7705d640 1 -- 172.21.15.107:0/3014493125 shutdown_connections 2024-05-09T08:31:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.966+0000 7fad7705d640 1 -- 172.21.15.107:0/3014493125 wait complete. 2024-05-09T08:31:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.966+0000 7fad7705d640 1 Processor -- start 2024-05-09T08:31:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad7705d640 1 -- start start 2024-05-09T08:31:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad7705d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad70101880 0x7fad70197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad7705d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad70102a70 0x7fad70197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad7705d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad70103430 0x7fad7019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad7705d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fad7010d3b0 con 0x7fad70103430 2024-05-09T08:31:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad7705d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fad7010d230 con 0x7fad70101880 2024-05-09T08:31:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad7705d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fad7010d530 con 0x7fad70102a70 2024-05-09T08:31:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad755fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad70103430 0x7fad7019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad755fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad70103430 0x7fad7019f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40150/0 (socket says 172.21.15.107:40150) 2024-05-09T08:31:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad755fa640 1 -- 172.21.15.107:0/1891186675 learned_addr learned my addr 172.21.15.107:0/1891186675 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad6ffff640 1 --2- 172.21.15.107:0/1891186675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad70102a70 0x7fad70197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad74df9640 1 --2- 172.21.15.107:0/1891186675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad70101880 0x7fad70197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad755fa640 1 -- 172.21.15.107:0/1891186675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad70102a70 msgr2=0x7fad70197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad755fa640 1 --2- 172.21.15.107:0/1891186675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad70102a70 0x7fad70197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad755fa640 1 -- 172.21.15.107:0/1891186675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad70101880 msgr2=0x7fad70197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad755fa640 1 --2- 172.21.15.107:0/1891186675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad70101880 0x7fad70197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad755fa640 1 -- 172.21.15.107:0/1891186675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad7019f6c0 con 0x7fad70103430 2024-05-09T08:31:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad74df9640 1 --2- 172.21.15.107:0/1891186675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad70101880 0x7fad70197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad755fa640 1 --2- 172.21.15.107:0/1891186675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad70103430 0x7fad7019f050 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fad68009dc0 tx=0x7fad68003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad6dffb640 1 -- 172.21.15.107:0/1891186675 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad68032ec0 con 0x7fad70103430 2024-05-09T08:31:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad6dffb640 1 -- 172.21.15.107:0/1891186675 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad680044f0 con 0x7fad70103430 2024-05-09T08:31:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad7705d640 1 -- 172.21.15.107:0/1891186675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad7019f950 con 0x7fad70103430 2024-05-09T08:31:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.967+0000 7fad6dffb640 1 -- 172.21.15.107:0/1891186675 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad68030d40 con 0x7fad70103430 2024-05-09T08:31:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.968+0000 7fad7705d640 1 -- 172.21.15.107:0/1891186675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad7019fea0 con 0x7fad70103430 2024-05-09T08:31:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.969+0000 7fad7705d640 1 -- 172.21.15.107:0/1891186675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad70108cc0 con 0x7fad70103430 2024-05-09T08:31:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.970+0000 7fad6dffb640 1 -- 172.21.15.107:0/1891186675 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fad68004040 con 0x7fad70103430 2024-05-09T08:31:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.971+0000 7fad6dffb640 1 --2- 172.21.15.107:0/1891186675 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fad600a2d30 0x7fad600a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.971+0000 7fad74df9640 1 --2- 172.21.15.107:0/1891186675 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fad600a2d30 0x7fad600a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.971+0000 7fad6dffb640 1 -- 172.21.15.107:0/1891186675 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7fad680f0840 con 0x7fad70103430 2024-05-09T08:31:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.971+0000 7fad74df9640 1 --2- 172.21.15.107:0/1891186675 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fad600a2d30 0x7fad600a51f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fad640058c0 tx=0x7fad64005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:03.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:03.975+0000 7fad6dffb640 1 -- 172.21.15.107:0/1891186675 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fad680bd580 con 0x7fad70103430 2024-05-09T08:31:04.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:03 smithi157 ceph-mon[33614]: osdmap e86: 12 total, 12 up, 12 in 2024-05-09T08:31:04.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:03 smithi157 ceph-mon[33614]: pgmap v290: 97 pgs: 17 creating+peering, 66 active+clean, 14 unknown; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 6 op/s 2024-05-09T08:31:04.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:03 smithi110 ceph-mon[33622]: osdmap e86: 12 total, 12 up, 12 in 2024-05-09T08:31:04.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:03 smithi110 ceph-mon[33622]: pgmap v290: 97 pgs: 17 creating+peering, 66 active+clean, 14 unknown; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 6 op/s 2024-05-09T08:31:04.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.153+0000 7fad7705d640 1 -- 172.21.15.107:0/1891186675 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fad700624d0 con 0x7fad70103430 2024-05-09T08:31:04.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:03 smithi107 ceph-mon[30122]: osdmap e86: 12 total, 12 up, 12 in 2024-05-09T08:31:04.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:03 smithi107 ceph-mon[30122]: pgmap v290: 97 pgs: 17 creating+peering, 66 active+clean, 14 unknown; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 6 op/s 2024-05-09T08:31:04.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.835+0000 7fad6dffb640 1 -- 172.21.15.107:0/1891186675 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7fad680bce80 con 0x7fad70103430 2024-05-09T08:31:04.835 INFO:teuthology.orchestra.run.smithi107.stderr:set pool 4 allow_ec_overwrites to true 2024-05-09T08:31:04.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.839+0000 7fad7705d640 1 -- 172.21.15.107:0/1891186675 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fad600a2d30 msgr2=0x7fad600a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:04.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.839+0000 7fad7705d640 1 --2- 172.21.15.107:0/1891186675 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fad600a2d30 0x7fad600a51f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fad640058c0 tx=0x7fad64005850 comp rx=0 tx=0).stop 2024-05-09T08:31:04.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.839+0000 7fad7705d640 1 -- 172.21.15.107:0/1891186675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad70103430 msgr2=0x7fad7019f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:04.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.839+0000 7fad7705d640 1 --2- 172.21.15.107:0/1891186675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad70103430 0x7fad7019f050 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fad68009dc0 tx=0x7fad68003ba0 comp rx=0 tx=0).stop 2024-05-09T08:31:04.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.839+0000 7fad7705d640 1 -- 172.21.15.107:0/1891186675 shutdown_connections 2024-05-09T08:31:04.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.839+0000 7fad7705d640 1 --2- 172.21.15.107:0/1891186675 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fad600a2d30 0x7fad600a51f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:04.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.839+0000 7fad7705d640 1 --2- 172.21.15.107:0/1891186675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad70103430 0x7fad7019f050 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:04.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.839+0000 7fad7705d640 1 --2- 172.21.15.107:0/1891186675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad70102a70 0x7fad70197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:04.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.839+0000 7fad7705d640 1 --2- 172.21.15.107:0/1891186675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad70101880 0x7fad70197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:04.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.839+0000 7fad7705d640 1 -- 172.21.15.107:0/1891186675 >> 172.21.15.107:0/1891186675 conn(0x7fad700fce10 msgr2=0x7fad70105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:04.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.840+0000 7fad7705d640 1 -- 172.21.15.107:0/1891186675 shutdown_connections 2024-05-09T08:31:04.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.840+0000 7fad7705d640 1 -- 172.21.15.107:0/1891186675 wait complete. 2024-05-09T08:31:04.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-05-09T08:31:04.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.975+0000 7f1b61a4d640 1 -- 172.21.15.107:0/71584505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5c100780 msgr2=0x7f1b5c100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:04.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.975+0000 7f1b61a4d640 1 --2- 172.21.15.107:0/71584505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5c100780 0x7f1b5c100ba0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009900 tx=0x7f1b4c02a410 comp rx=0 tx=0).stop 2024-05-09T08:31:04.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.975+0000 7f1b61a4d640 1 -- 172.21.15.107:0/71584505 shutdown_connections 2024-05-09T08:31:04.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.975+0000 7f1b61a4d640 1 --2- 172.21.15.107:0/71584505 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b5c102330 0x7f1b5c108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:04.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.975+0000 7f1b61a4d640 1 --2- 172.21.15.107:0/71584505 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b5c101970 0x7f1b5c101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:04.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.975+0000 7f1b61a4d640 1 --2- 172.21.15.107:0/71584505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5c100780 0x7f1b5c100ba0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:04.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.975+0000 7f1b61a4d640 1 -- 172.21.15.107:0/71584505 >> 172.21.15.107:0/71584505 conn(0x7f1b5c0fbcf0 msgr2=0x7f1b5c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:04.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.975+0000 7f1b61a4d640 1 -- 172.21.15.107:0/71584505 shutdown_connections 2024-05-09T08:31:04.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.975+0000 7f1b61a4d640 1 -- 172.21.15.107:0/71584505 wait complete. 2024-05-09T08:31:04.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b61a4d640 1 Processor -- start 2024-05-09T08:31:04.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b61a4d640 1 -- start start 2024-05-09T08:31:04.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b61a4d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5c100780 0x7f1b5c197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:04.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b61a4d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b5c101970 0x7f1b5c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:04.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b61a4d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b5c102330 0x7f1b5c19f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:04.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b61a4d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c10c300 con 0x7f1b5c100780 2024-05-09T08:31:04.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b61a4d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c10c180 con 0x7f1b5c102330 2024-05-09T08:31:04.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b61a4d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c10c480 con 0x7f1b5c101970 2024-05-09T08:31:04.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b5affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5c100780 0x7f1b5c197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:04.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b5affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5c100780 0x7f1b5c197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40166/0 (socket says 172.21.15.107:40166) 2024-05-09T08:31:04.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b5affd640 1 -- 172.21.15.107:0/3073188770 learned_addr learned my addr 172.21.15.107:0/3073188770 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:04.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b5b7fe640 1 --2- 172.21.15.107:0/3073188770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b5c102330 0x7f1b5c19f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:04.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b5affd640 1 -- 172.21.15.107:0/3073188770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b5c101970 msgr2=0x7f1b5c1980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:04.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b5affd640 1 --2- 172.21.15.107:0/3073188770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b5c101970 0x7f1b5c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:04.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b5affd640 1 -- 172.21.15.107:0/3073188770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b5c102330 msgr2=0x7f1b5c19f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:04.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b5affd640 1 --2- 172.21.15.107:0/3073188770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b5c102330 0x7f1b5c19f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:04.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b5affd640 1 -- 172.21.15.107:0/3073188770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5c19f750 con 0x7f1b5c100780 2024-05-09T08:31:04.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b5b7fe640 1 --2- 172.21.15.107:0/3073188770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b5c102330 0x7f1b5c19f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:04.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b5affd640 1 --2- 172.21.15.107:0/3073188770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5c100780 0x7f1b5c197b80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009230 tx=0x7f1b4c004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:04.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b60a4b640 1 -- 172.21.15.107:0/3073188770 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c03f030 con 0x7f1b5c100780 2024-05-09T08:31:04.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.976+0000 7f1b60a4b640 1 -- 172.21.15.107:0/3073188770 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b4c003cb0 con 0x7f1b5c100780 2024-05-09T08:31:04.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.977+0000 7f1b61a4d640 1 -- 172.21.15.107:0/3073188770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5c19f9e0 con 0x7f1b5c100780 2024-05-09T08:31:04.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.977+0000 7f1b60a4b640 1 -- 172.21.15.107:0/3073188770 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c035070 con 0x7f1b5c100780 2024-05-09T08:31:04.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.977+0000 7f1b61a4d640 1 -- 172.21.15.107:0/3073188770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b5c19ffa0 con 0x7f1b5c100780 2024-05-09T08:31:04.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.977+0000 7f1b61a4d640 1 -- 172.21.15.107:0/3073188770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b5c1a01d0 con 0x7f1b5c100780 2024-05-09T08:31:04.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.979+0000 7f1b60a4b640 1 -- 172.21.15.107:0/3073188770 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f1b4c042030 con 0x7f1b5c100780 2024-05-09T08:31:04.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.979+0000 7f1b60a4b640 1 --2- 172.21.15.107:0/3073188770 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1b480a2c10 0x7f1b480a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:04.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.979+0000 7f1b527fc640 1 --2- 172.21.15.107:0/3073188770 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1b480a2c10 0x7f1b480a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:04.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.979+0000 7f1b60a4b640 1 -- 172.21.15.107:0/3073188770 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f1b4c0ef960 con 0x7f1b5c100780 2024-05-09T08:31:04.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.980+0000 7f1b527fc640 1 --2- 172.21.15.107:0/3073188770 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1b480a2c10 0x7f1b480a50d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c199040 tx=0x7f1b44005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:04.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:04.984+0000 7f1b60a4b640 1 -- 172.21.15.107:0/3073188770 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1b4c0bc6a0 con 0x7f1b5c100780 2024-05-09T08:31:05.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:04 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/55083485' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-09T08:31:05.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:04 smithi110 ceph-mon[33622]: osdmap e87: 12 total, 12 up, 12 in 2024-05-09T08:31:05.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:04 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1891186675' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-09T08:31:05.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.200+0000 7f1b61a4d640 1 -- 172.21.15.107:0/3073188770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f1b5c1a0560 con 0x7f1b5c100780 2024-05-09T08:31:05.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:04 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/55083485' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-09T08:31:05.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:04 smithi107 ceph-mon[30122]: osdmap e87: 12 total, 12 up, 12 in 2024-05-09T08:31:05.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:04 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1891186675' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-09T08:31:05.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:04 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/55083485' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-09T08:31:05.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:04 smithi157 ceph-mon[33614]: osdmap e87: 12 total, 12 up, 12 in 2024-05-09T08:31:05.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:04 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1891186675' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-09T08:31:05.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.854+0000 7f1b60a4b640 1 -- 172.21.15.107:0/3073188770 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f1b4c0bbfa0 con 0x7f1b5c100780 2024-05-09T08:31:05.854 INFO:teuthology.orchestra.run.smithi107.stderr:added data pool 4 to fsmap 2024-05-09T08:31:05.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.856+0000 7f1b61a4d640 1 -- 172.21.15.107:0/3073188770 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1b480a2c10 msgr2=0x7f1b480a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:05.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.856+0000 7f1b61a4d640 1 --2- 172.21.15.107:0/3073188770 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1b480a2c10 0x7f1b480a50d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c199040 tx=0x7f1b44005c70 comp rx=0 tx=0).stop 2024-05-09T08:31:05.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.856+0000 7f1b61a4d640 1 -- 172.21.15.107:0/3073188770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5c100780 msgr2=0x7f1b5c197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:05.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.856+0000 7f1b61a4d640 1 --2- 172.21.15.107:0/3073188770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5c100780 0x7f1b5c197b80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009230 tx=0x7f1b4c004340 comp rx=0 tx=0).stop 2024-05-09T08:31:05.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.856+0000 7f1b61a4d640 1 -- 172.21.15.107:0/3073188770 shutdown_connections 2024-05-09T08:31:05.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.856+0000 7f1b61a4d640 1 --2- 172.21.15.107:0/3073188770 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1b480a2c10 0x7f1b480a50d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:05.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.856+0000 7f1b61a4d640 1 --2- 172.21.15.107:0/3073188770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b5c102330 0x7f1b5c19f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:05.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.856+0000 7f1b61a4d640 1 --2- 172.21.15.107:0/3073188770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b5c101970 0x7f1b5c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:05.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.856+0000 7f1b61a4d640 1 --2- 172.21.15.107:0/3073188770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b5c100780 0x7f1b5c197b80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:05.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.856+0000 7f1b61a4d640 1 -- 172.21.15.107:0/3073188770 >> 172.21.15.107:0/3073188770 conn(0x7f1b5c0fbcf0 msgr2=0x7f1b5c0fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:05.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.857+0000 7f1b61a4d640 1 -- 172.21.15.107:0/3073188770 shutdown_connections 2024-05-09T08:31:05.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.857+0000 7f1b61a4d640 1 -- 172.21.15.107:0/3073188770 wait complete. 2024-05-09T08:31:05.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:31:05.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.977+0000 7f1be8215640 1 -- 172.21.15.107:0/4122167501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1be0101d90 msgr2=0x7f1be01021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:05.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.977+0000 7f1be8215640 1 --2- 172.21.15.107:0/4122167501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1be0101d90 0x7f1be01021b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc009930 tx=0x7f1bdc02a420 comp rx=0 tx=0).stop 2024-05-09T08:31:05.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be8215640 1 -- 172.21.15.107:0/4122167501 shutdown_connections 2024-05-09T08:31:05.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be8215640 1 --2- 172.21.15.107:0/4122167501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1be01013f0 0x7f1be0108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:05.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be8215640 1 --2- 172.21.15.107:0/4122167501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1be0100a30 0x7f1be0100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be8215640 1 --2- 172.21.15.107:0/4122167501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1be0101d90 0x7f1be01021b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be8215640 1 -- 172.21.15.107:0/4122167501 >> 172.21.15.107:0/4122167501 conn(0x7f1be00fb4a0 msgr2=0x7f1be00fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be8215640 1 -- 172.21.15.107:0/4122167501 shutdown_connections 2024-05-09T08:31:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be8215640 1 -- 172.21.15.107:0/4122167501 wait complete. 2024-05-09T08:31:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be8215640 1 Processor -- start 2024-05-09T08:31:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be8215640 1 -- start start 2024-05-09T08:31:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be8215640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1be0100a30 0x7f1be0100490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be8215640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1be01013f0 0x7f1be019ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be8215640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1be0101d90 0x7f1be01a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be8215640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1be010c330 con 0x7f1be0100a30 2024-05-09T08:31:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be8215640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1be010c1b0 con 0x7f1be01013f0 2024-05-09T08:31:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be8215640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1be010c4b0 con 0x7f1be0101d90 2024-05-09T08:31:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be5fb1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1be0100a30 0x7f1be0100490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:05.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be5fb1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1be0100a30 0x7f1be0100490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40192/0 (socket says 172.21.15.107:40192) 2024-05-09T08:31:05.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.978+0000 7f1be5fb1640 1 -- 172.21.15.107:0/2997580959 learned_addr learned my addr 172.21.15.107:0/2997580959 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:05.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.979+0000 7f1be67b2640 1 --2- 172.21.15.107:0/2997580959 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1be0101d90 0x7f1be01a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:05.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.979+0000 7f1be57b0640 1 --2- 172.21.15.107:0/2997580959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1be01013f0 0x7f1be019ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:05.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.979+0000 7f1be5fb1640 1 -- 172.21.15.107:0/2997580959 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1be0101d90 msgr2=0x7f1be01a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:05.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.979+0000 7f1be5fb1640 1 --2- 172.21.15.107:0/2997580959 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1be0101d90 0x7f1be01a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:05.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.979+0000 7f1be5fb1640 1 -- 172.21.15.107:0/2997580959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1be01013f0 msgr2=0x7f1be019ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:05.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.979+0000 7f1be5fb1640 1 --2- 172.21.15.107:0/2997580959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1be01013f0 0x7f1be019ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:05.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.979+0000 7f1be5fb1640 1 -- 172.21.15.107:0/2997580959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be01a1980 con 0x7f1be0100a30 2024-05-09T08:31:05.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.979+0000 7f1be5fb1640 1 --2- 172.21.15.107:0/2997580959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1be0100a30 0x7f1be0100490 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc009900 tx=0x7f1bdc004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:05.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.979+0000 7f1bd6ffd640 1 -- 172.21.15.107:0/2997580959 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc03f030 con 0x7f1be0100a30 2024-05-09T08:31:05.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.979+0000 7f1bd6ffd640 1 -- 172.21.15.107:0/2997580959 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1bdc003800 con 0x7f1be0100a30 2024-05-09T08:31:05.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.979+0000 7f1be8215640 1 -- 172.21.15.107:0/2997580959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1be01a1bb0 con 0x7f1be0100a30 2024-05-09T08:31:05.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.979+0000 7f1bd6ffd640 1 -- 172.21.15.107:0/2997580959 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc035070 con 0x7f1be0100a30 2024-05-09T08:31:05.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.979+0000 7f1be8215640 1 -- 172.21.15.107:0/2997580959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1be01a20a0 con 0x7f1be0100a30 2024-05-09T08:31:05.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.980+0000 7f1be8215640 1 -- 172.21.15.107:0/2997580959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1be0107b70 con 0x7f1be0100a30 2024-05-09T08:31:05.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.981+0000 7f1bd6ffd640 1 -- 172.21.15.107:0/2997580959 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f1bdc0039b0 con 0x7f1be0100a30 2024-05-09T08:31:05.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.982+0000 7f1bd6ffd640 1 --2- 172.21.15.107:0/2997580959 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1bcc0a2bc0 0x7f1bcc0a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:05.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.982+0000 7f1be57b0640 1 --2- 172.21.15.107:0/2997580959 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1bcc0a2bc0 0x7f1bcc0a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:05.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.982+0000 7f1bd6ffd640 1 -- 172.21.15.107:0/2997580959 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1bdc0f01a0 con 0x7f1be0100a30 2024-05-09T08:31:05.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.982+0000 7f1be57b0640 1 --2- 172.21.15.107:0/2997580959 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1bcc0a2bc0 0x7f1bcc0a5080 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0005e10 tx=0x7f1bd0005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:05.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:05.985+0000 7f1bd6ffd640 1 -- 172.21.15.107:0/2997580959 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1bdc0bcec0 con 0x7f1be0100a30 2024-05-09T08:31:06.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:05 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1891186675' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-09T08:31:06.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:05 smithi110 ceph-mon[33622]: osdmap e88: 12 total, 12 up, 12 in 2024-05-09T08:31:06.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:05 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3073188770' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-09T08:31:06.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:05 smithi110 ceph-mon[33622]: pgmap v293: 129 pgs: 10 creating+peering, 95 active+clean, 24 unknown; 579 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 8 op/s 2024-05-09T08:31:06.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.198+0000 7f1be8215640 1 -- 172.21.15.107:0/2997580959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1be002d1c0 con 0x7f1be0100a30 2024-05-09T08:31:06.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.199+0000 7f1bd6ffd640 1 -- 172.21.15.107:0/2997580959 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11815 (secure 0 0 0) 0x7f1bdc0bc7c0 con 0x7f1be0100a30 2024-05-09T08:31:06.198 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:06.198 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24491,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:31:05.201970+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24499,"qdb_cluster":[24499]},"id":1}]} 2024-05-09T08:31:06.199 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2024-05-09T08:31:06.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.201+0000 7f1be8215640 1 -- 172.21.15.107:0/2997580959 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1bcc0a2bc0 msgr2=0x7f1bcc0a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.201+0000 7f1be8215640 1 --2- 172.21.15.107:0/2997580959 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1bcc0a2bc0 0x7f1bcc0a5080 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0005e10 tx=0x7f1bd0005da0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.201+0000 7f1be8215640 1 -- 172.21.15.107:0/2997580959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1be0100a30 msgr2=0x7f1be0100490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.201+0000 7f1be8215640 1 --2- 172.21.15.107:0/2997580959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1be0100a30 0x7f1be0100490 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc009900 tx=0x7f1bdc004340 comp rx=0 tx=0).stop 2024-05-09T08:31:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.201+0000 7f1be8215640 1 -- 172.21.15.107:0/2997580959 shutdown_connections 2024-05-09T08:31:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.201+0000 7f1be8215640 1 --2- 172.21.15.107:0/2997580959 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1bcc0a2bc0 0x7f1bcc0a5080 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.201+0000 7f1be8215640 1 --2- 172.21.15.107:0/2997580959 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1be0101d90 0x7f1be01a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.201+0000 7f1be8215640 1 --2- 172.21.15.107:0/2997580959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1be01013f0 0x7f1be019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.201+0000 7f1be8215640 1 --2- 172.21.15.107:0/2997580959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1be0100a30 0x7f1be0100490 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.201+0000 7f1be8215640 1 -- 172.21.15.107:0/2997580959 >> 172.21.15.107:0/2997580959 conn(0x7f1be00fb4a0 msgr2=0x7f1be0104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.201+0000 7f1be8215640 1 -- 172.21.15.107:0/2997580959 shutdown_connections 2024-05-09T08:31:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.201+0000 7f1be8215640 1 -- 172.21.15.107:0/2997580959 wait complete. 2024-05-09T08:31:06.210 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:05 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1891186675' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-09T08:31:06.210 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:05 smithi107 ceph-mon[30122]: osdmap e88: 12 total, 12 up, 12 in 2024-05-09T08:31:06.210 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:05 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3073188770' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-09T08:31:06.211 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:05 smithi107 ceph-mon[30122]: pgmap v293: 129 pgs: 10 creating+peering, 95 active+clean, 24 unknown; 579 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 8 op/s 2024-05-09T08:31:06.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:31:06.245 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:05 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1891186675' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-09T08:31:06.245 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:05 smithi157 ceph-mon[33614]: osdmap e88: 12 total, 12 up, 12 in 2024-05-09T08:31:06.245 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:05 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3073188770' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-09T08:31:06.245 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:05 smithi157 ceph-mon[33614]: pgmap v293: 129 pgs: 10 creating+peering, 95 active+clean, 24 unknown; 579 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 8 op/s 2024-05-09T08:31:06.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.313+0000 7f8e1e4ca640 1 -- 172.21.15.107:0/4091571692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e18101970 msgr2=0x7f8e18101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:06.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.313+0000 7f8e1e4ca640 1 --2- 172.21.15.107:0/4091571692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e18101970 0x7f8e18101df0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009900 tx=0x7f8e0002a410 comp rx=0 tx=0).stop 2024-05-09T08:31:06.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.313+0000 7f8e1e4ca640 1 -- 172.21.15.107:0/4091571692 shutdown_connections 2024-05-09T08:31:06.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.313+0000 7f8e1e4ca640 1 --2- 172.21.15.107:0/4091571692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e18102330 0x7f8e18108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.313+0000 7f8e1e4ca640 1 --2- 172.21.15.107:0/4091571692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e18101970 0x7f8e18101df0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.313+0000 7f8e1e4ca640 1 --2- 172.21.15.107:0/4091571692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e18100780 0x7f8e18100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.313+0000 7f8e1e4ca640 1 -- 172.21.15.107:0/4091571692 >> 172.21.15.107:0/4091571692 conn(0x7f8e180fbd10 msgr2=0x7f8e180fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:06.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.313+0000 7f8e1e4ca640 1 -- 172.21.15.107:0/4091571692 shutdown_connections 2024-05-09T08:31:06.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.313+0000 7f8e1e4ca640 1 -- 172.21.15.107:0/4091571692 wait complete. 2024-05-09T08:31:06.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e1e4ca640 1 Processor -- start 2024-05-09T08:31:06.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e1e4ca640 1 -- start start 2024-05-09T08:31:06.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e1e4ca640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e18100780 0x7f8e1806aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:06.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e1e4ca640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e18101970 0x7f8e1819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:06.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e1e4ca640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e18102330 0x7f8e1819ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:06.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e1e4ca640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1810c2b0 con 0x7f8e18102330 2024-05-09T08:31:06.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e1e4ca640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1810c130 con 0x7f8e18101970 2024-05-09T08:31:06.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e1e4ca640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1810c430 con 0x7f8e18100780 2024-05-09T08:31:06.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e1ca67640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e18102330 0x7f8e1819ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:06.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e1ca67640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e18102330 0x7f8e1819ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40218/0 (socket says 172.21.15.107:40218) 2024-05-09T08:31:06.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e1ca67640 1 -- 172.21.15.107:0/1133983252 learned_addr learned my addr 172.21.15.107:0/1133983252 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:06.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e17fff640 1 --2- 172.21.15.107:0/1133983252 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e18100780 0x7f8e1806aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:06.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e0ffff640 1 --2- 172.21.15.107:0/1133983252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e18101970 0x7f8e1819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:06.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e1ca67640 1 -- 172.21.15.107:0/1133983252 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e18100780 msgr2=0x7f8e1806aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:06.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e1ca67640 1 --2- 172.21.15.107:0/1133983252 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e18100780 0x7f8e1806aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e1ca67640 1 -- 172.21.15.107:0/1133983252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e18101970 msgr2=0x7f8e1819c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:06.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e1ca67640 1 --2- 172.21.15.107:0/1133983252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e18101970 0x7f8e1819c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e1ca67640 1 -- 172.21.15.107:0/1133983252 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1819f6e0 con 0x7f8e18102330 2024-05-09T08:31:06.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.314+0000 7f8e0ffff640 1 --2- 172.21.15.107:0/1133983252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e18101970 0x7f8e1819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:06.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.315+0000 7f8e1ca67640 1 --2- 172.21.15.107:0/1133983252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e18102330 0x7f8e1819ee20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8e1000eaa0 tx=0x7f8e1000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:06.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.315+0000 7f8e15ffb640 1 -- 172.21.15.107:0/1133983252 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1000cdb0 con 0x7f8e18102330 2024-05-09T08:31:06.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.315+0000 7f8e1e4ca640 1 -- 172.21.15.107:0/1133983252 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e1819f9d0 con 0x7f8e18102330 2024-05-09T08:31:06.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.315+0000 7f8e15ffb640 1 -- 172.21.15.107:0/1133983252 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e10005020 con 0x7f8e18102330 2024-05-09T08:31:06.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.315+0000 7f8e15ffb640 1 -- 172.21.15.107:0/1133983252 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e100116d0 con 0x7f8e18102330 2024-05-09T08:31:06.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.315+0000 7f8e1e4ca640 1 -- 172.21.15.107:0/1133983252 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e1819ff70 con 0x7f8e18102330 2024-05-09T08:31:06.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.316+0000 7f8e1e4ca640 1 -- 172.21.15.107:0/1133983252 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e18107b20 con 0x7f8e18102330 2024-05-09T08:31:06.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.317+0000 7f8e15ffb640 1 -- 172.21.15.107:0/1133983252 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f8e100051d0 con 0x7f8e18102330 2024-05-09T08:31:06.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.317+0000 7f8e15ffb640 1 --2- 172.21.15.107:0/1133983252 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8e040a2b90 0x7f8e040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:06.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.317+0000 7f8e17fff640 1 --2- 172.21.15.107:0/1133983252 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8e040a2b90 0x7f8e040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:06.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.318+0000 7f8e15ffb640 1 -- 172.21.15.107:0/1133983252 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8e100cf410 con 0x7f8e18102330 2024-05-09T08:31:06.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.318+0000 7f8e17fff640 1 --2- 172.21.15.107:0/1133983252 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8e040a2b90 0x7f8e040a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8e181017d0 tx=0x7f8e0800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:06.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.322+0000 7f8e15ffb640 1 -- 172.21.15.107:0/1133983252 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8e10016030 con 0x7f8e18102330 2024-05-09T08:31:06.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.501+0000 7f8e1e4ca640 1 -- 172.21.15.107:0/1133983252 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8e18062430 con 0x7f8e18102330 2024-05-09T08:31:06.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.503+0000 7f8e15ffb640 1 -- 172.21.15.107:0/1133983252 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f8e1009c130 con 0x7f8e18102330 2024-05-09T08:31:06.502 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:06.502 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:31:05.201799+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:31:06.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.505+0000 7f8e1e4ca640 1 -- 172.21.15.107:0/1133983252 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8e040a2b90 msgr2=0x7f8e040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:06.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.505+0000 7f8e1e4ca640 1 --2- 172.21.15.107:0/1133983252 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8e040a2b90 0x7f8e040a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8e181017d0 tx=0x7f8e0800a2d0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.505+0000 7f8e1e4ca640 1 -- 172.21.15.107:0/1133983252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e18102330 msgr2=0x7f8e1819ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:06.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.505+0000 7f8e1e4ca640 1 --2- 172.21.15.107:0/1133983252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e18102330 0x7f8e1819ee20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8e1000eaa0 tx=0x7f8e1000ef60 comp rx=0 tx=0).stop 2024-05-09T08:31:06.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.505+0000 7f8e1e4ca640 1 -- 172.21.15.107:0/1133983252 shutdown_connections 2024-05-09T08:31:06.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.505+0000 7f8e1e4ca640 1 --2- 172.21.15.107:0/1133983252 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8e040a2b90 0x7f8e040a5050 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.505+0000 7f8e1e4ca640 1 --2- 172.21.15.107:0/1133983252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e18102330 0x7f8e1819ee20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.505+0000 7f8e1e4ca640 1 --2- 172.21.15.107:0/1133983252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e18101970 0x7f8e1819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.505+0000 7f8e1e4ca640 1 --2- 172.21.15.107:0/1133983252 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e18100780 0x7f8e1806aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.505+0000 7f8e1e4ca640 1 -- 172.21.15.107:0/1133983252 >> 172.21.15.107:0/1133983252 conn(0x7f8e180fbd10 msgr2=0x7f8e18104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:06.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.505+0000 7f8e1e4ca640 1 -- 172.21.15.107:0/1133983252 shutdown_connections 2024-05-09T08:31:06.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.505+0000 7f8e1e4ca640 1 -- 172.21.15.107:0/1133983252 wait complete. 2024-05-09T08:31:06.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:31:06.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.618+0000 7f64ebc6b640 1 -- 172.21.15.107:0/1717405630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64e4109750 msgr2=0x7f64e410bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.618+0000 7f64ebc6b640 1 --2- 172.21.15.107:0/1717405630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64e4109750 0x7f64e410bbe0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f64dc009ca0 tx=0x7f64dc02a950 comp rx=0 tx=0).stop 2024-05-09T08:31:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.618+0000 7f64ebc6b640 1 -- 172.21.15.107:0/1717405630 shutdown_connections 2024-05-09T08:31:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.618+0000 7f64ebc6b640 1 --2- 172.21.15.107:0/1717405630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64e4109750 0x7f64e410bbe0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.618+0000 7f64ebc6b640 1 --2- 172.21.15.107:0/1717405630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f64e4074210 0x7f64e4074670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.618+0000 7f64ebc6b640 1 --2- 172.21.15.107:0/1717405630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64e40757f0 0x7f64e4073c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.618+0000 7f64ebc6b640 1 -- 172.21.15.107:0/1717405630 >> 172.21.15.107:0/1717405630 conn(0x7f64e40fce30 msgr2=0x7f64e40ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.618+0000 7f64ebc6b640 1 -- 172.21.15.107:0/1717405630 shutdown_connections 2024-05-09T08:31:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.618+0000 7f64ebc6b640 1 -- 172.21.15.107:0/1717405630 wait complete. 2024-05-09T08:31:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.618+0000 7f64ebc6b640 1 Processor -- start 2024-05-09T08:31:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.618+0000 7f64ebc6b640 1 -- start start 2024-05-09T08:31:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.618+0000 7f64ebc6b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64e4074210 0x7f64e4199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.618+0000 7f64ebc6b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64e40757f0 0x7f64e419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64ebc6b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f64e4109750 0x7f64e41a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64ebc6b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f64e410f650 con 0x7f64e40757f0 2024-05-09T08:31:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64ebc6b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f64e410f4d0 con 0x7f64e4074210 2024-05-09T08:31:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64ebc6b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f64e410f7d0 con 0x7f64e4109750 2024-05-09T08:31:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64e9206640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64e40757f0 0x7f64e419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64e9206640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64e40757f0 0x7f64e419a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40240/0 (socket says 172.21.15.107:40240) 2024-05-09T08:31:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64e9206640 1 -- 172.21.15.107:0/1851307196 learned_addr learned my addr 172.21.15.107:0/1851307196 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64ea208640 1 --2- 172.21.15.107:0/1851307196 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f64e4109750 0x7f64e41a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64e9206640 1 -- 172.21.15.107:0/1851307196 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f64e4109750 msgr2=0x7f64e41a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64e9206640 1 --2- 172.21.15.107:0/1851307196 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f64e4109750 0x7f64e41a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64e9206640 1 -- 172.21.15.107:0/1851307196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64e4074210 msgr2=0x7f64e4199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64e9206640 1 --2- 172.21.15.107:0/1851307196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64e4074210 0x7f64e4199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64e9206640 1 -- 172.21.15.107:0/1851307196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e41a1940 con 0x7f64e40757f0 2024-05-09T08:31:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64ea208640 1 --2- 172.21.15.107:0/1851307196 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f64e4109750 0x7f64e41a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64e9206640 1 --2- 172.21.15.107:0/1851307196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64e40757f0 0x7f64e419a270 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f64e000d770 tx=0x7f64e000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:06.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64d6ffd640 1 -- 172.21.15.107:0/1851307196 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0017070 con 0x7f64e40757f0 2024-05-09T08:31:06.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64d6ffd640 1 -- 172.21.15.107:0/1851307196 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64e00054d0 con 0x7f64e40757f0 2024-05-09T08:31:06.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64ebc6b640 1 -- 172.21.15.107:0/1851307196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64e41a1c30 con 0x7f64e40757f0 2024-05-09T08:31:06.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64d6ffd640 1 -- 172.21.15.107:0/1851307196 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0004cf0 con 0x7f64e40757f0 2024-05-09T08:31:06.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.619+0000 7f64ebc6b640 1 -- 172.21.15.107:0/1851307196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64e41a2130 con 0x7f64e40757f0 2024-05-09T08:31:06.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.620+0000 7f64ebc6b640 1 -- 172.21.15.107:0/1851307196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64e4108510 con 0x7f64e40757f0 2024-05-09T08:31:06.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.621+0000 7f64d6ffd640 1 -- 172.21.15.107:0/1851307196 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f64e0004e90 con 0x7f64e40757f0 2024-05-09T08:31:06.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.622+0000 7f64d6ffd640 1 --2- 172.21.15.107:0/1851307196 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f64d00a2b90 0x7f64d00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:06.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.622+0000 7f64e9a07640 1 --2- 172.21.15.107:0/1851307196 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f64d00a2b90 0x7f64d00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:06.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.622+0000 7f64d6ffd640 1 -- 172.21.15.107:0/1851307196 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f64e00cee30 con 0x7f64e40757f0 2024-05-09T08:31:06.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.622+0000 7f64e9a07640 1 --2- 172.21.15.107:0/1851307196 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f64d00a2b90 0x7f64d00a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f64e419b200 tx=0x7f64d8006c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:06.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.626+0000 7f64d6ffd640 1 -- 172.21.15.107:0/1851307196 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f64e009bb50 con 0x7f64e40757f0 2024-05-09T08:31:06.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.804+0000 7f64ebc6b640 1 -- 172.21.15.107:0/1851307196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f64e4062530 con 0x7f64e40757f0 2024-05-09T08:31:06.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.806+0000 7f64d6ffd640 1 -- 172.21.15.107:0/1851307196 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f64e009b450 con 0x7f64e40757f0 2024-05-09T08:31:06.805 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:06.806 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:31:05.201799+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:31:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.808+0000 7f64ebc6b640 1 -- 172.21.15.107:0/1851307196 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f64d00a2b90 msgr2=0x7f64d00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.808+0000 7f64ebc6b640 1 --2- 172.21.15.107:0/1851307196 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f64d00a2b90 0x7f64d00a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f64e419b200 tx=0x7f64d8006c90 comp rx=0 tx=0).stop 2024-05-09T08:31:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.808+0000 7f64ebc6b640 1 -- 172.21.15.107:0/1851307196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64e40757f0 msgr2=0x7f64e419a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.808+0000 7f64ebc6b640 1 --2- 172.21.15.107:0/1851307196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64e40757f0 0x7f64e419a270 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f64e000d770 tx=0x7f64e000dc30 comp rx=0 tx=0).stop 2024-05-09T08:31:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.808+0000 7f64ebc6b640 1 -- 172.21.15.107:0/1851307196 shutdown_connections 2024-05-09T08:31:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.808+0000 7f64ebc6b640 1 --2- 172.21.15.107:0/1851307196 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f64d00a2b90 0x7f64d00a5050 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.808+0000 7f64ebc6b640 1 --2- 172.21.15.107:0/1851307196 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f64e4109750 0x7f64e41a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.808+0000 7f64ebc6b640 1 --2- 172.21.15.107:0/1851307196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64e40757f0 0x7f64e419a270 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.808+0000 7f64ebc6b640 1 --2- 172.21.15.107:0/1851307196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64e4074210 0x7f64e4199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.808+0000 7f64ebc6b640 1 -- 172.21.15.107:0/1851307196 >> 172.21.15.107:0/1851307196 conn(0x7f64e40fce30 msgr2=0x7f64e41056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.808+0000 7f64ebc6b640 1 -- 172.21.15.107:0/1851307196 shutdown_connections 2024-05-09T08:31:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.808+0000 7f64ebc6b640 1 -- 172.21.15.107:0/1851307196 wait complete. 2024-05-09T08:31:06.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:31:06.843 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:31:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-i[64360]: 2024-05-09T08:31:06.514+0000 7f98e1dec700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T08:31:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.936+0000 7f203e8fc640 1 -- 172.21.15.107:0/3713377536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2038100790 msgr2=0x7f2038100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.936+0000 7f203e8fc640 1 --2- 172.21.15.107:0/3713377536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2038100790 0x7f2038100bb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2028009900 tx=0x7f202802a410 comp rx=0 tx=0).stop 2024-05-09T08:31:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.936+0000 7f203e8fc640 1 -- 172.21.15.107:0/3713377536 shutdown_connections 2024-05-09T08:31:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.936+0000 7f203e8fc640 1 --2- 172.21.15.107:0/3713377536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2038102340 0x7f2038108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.936+0000 7f203e8fc640 1 --2- 172.21.15.107:0/3713377536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2038101980 0x7f2038101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.936+0000 7f203e8fc640 1 --2- 172.21.15.107:0/3713377536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2038100790 0x7f2038100bb0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.936+0000 7f203e8fc640 1 -- 172.21.15.107:0/3713377536 >> 172.21.15.107:0/3713377536 conn(0x7f20380fbd20 msgr2=0x7f20380fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.936+0000 7f203e8fc640 1 -- 172.21.15.107:0/3713377536 shutdown_connections 2024-05-09T08:31:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.936+0000 7f203e8fc640 1 -- 172.21.15.107:0/3713377536 wait complete. 2024-05-09T08:31:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f203e8fc640 1 Processor -- start 2024-05-09T08:31:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f203e8fc640 1 -- start start 2024-05-09T08:31:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f203e8fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2038100790 0x7f2038197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f203e8fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2038101980 0x7f2038198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f203e8fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2038102340 0x7f203819f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f203e8fc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f203810d460 con 0x7f2038101980 2024-05-09T08:31:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f203e8fc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f203810d2e0 con 0x7f2038102340 2024-05-09T08:31:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f203e8fc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f203810d5e0 con 0x7f2038100790 2024-05-09T08:31:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f20377fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2038101980 0x7f2038198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f20377fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2038101980 0x7f2038198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40264/0 (socket says 172.21.15.107:40264) 2024-05-09T08:31:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f20377fe640 1 -- 172.21.15.107:0/449726152 learned_addr learned my addr 172.21.15.107:0/449726152 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f2037fff640 1 --2- 172.21.15.107:0/449726152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2038100790 0x7f2038197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f203ce99640 1 --2- 172.21.15.107:0/449726152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2038102340 0x7f203819f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f20377fe640 1 -- 172.21.15.107:0/449726152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2038100790 msgr2=0x7f2038197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f20377fe640 1 --2- 172.21.15.107:0/449726152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2038100790 0x7f2038197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f20377fe640 1 -- 172.21.15.107:0/449726152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2038102340 msgr2=0x7f203819f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:06.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f20377fe640 1 --2- 172.21.15.107:0/449726152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2038102340 0x7f203819f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:06.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f20377fe640 1 -- 172.21.15.107:0/449726152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f203819f7c0 con 0x7f2038101980 2024-05-09T08:31:06.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.937+0000 7f203ce99640 1 --2- 172.21.15.107:0/449726152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2038102340 0x7f203819f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:06.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.938+0000 7f20377fe640 1 --2- 172.21.15.107:0/449726152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2038101980 0x7f2038198130 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f202c00b4a0 tx=0x7f202c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:06.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.938+0000 7f20357fa640 1 -- 172.21.15.107:0/449726152 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c014070 con 0x7f2038101980 2024-05-09T08:31:06.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.938+0000 7f20357fa640 1 -- 172.21.15.107:0/449726152 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f202c010ac0 con 0x7f2038101980 2024-05-09T08:31:06.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.938+0000 7f20357fa640 1 -- 172.21.15.107:0/449726152 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c017c30 con 0x7f2038101980 2024-05-09T08:31:06.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.938+0000 7f203e8fc640 1 -- 172.21.15.107:0/449726152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f203819fab0 con 0x7f2038101980 2024-05-09T08:31:06.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.938+0000 7f203e8fc640 1 -- 172.21.15.107:0/449726152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20381a0000 con 0x7f2038101980 2024-05-09T08:31:06.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.939+0000 7f203e8fc640 1 -- 172.21.15.107:0/449726152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2038107bd0 con 0x7f2038101980 2024-05-09T08:31:06.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.940+0000 7f20357fa640 1 -- 172.21.15.107:0/449726152 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f202c017070 con 0x7f2038101980 2024-05-09T08:31:06.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.941+0000 7f20357fa640 1 --2- 172.21.15.107:0/449726152 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f20200a2b90 0x7f20200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:06.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.941+0000 7f2037fff640 1 --2- 172.21.15.107:0/449726152 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f20200a2b90 0x7f20200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:06.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.941+0000 7f20357fa640 1 -- 172.21.15.107:0/449726152 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f202c0ce950 con 0x7f2038101980 2024-05-09T08:31:06.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.941+0000 7f2037fff640 1 --2- 172.21.15.107:0/449726152 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f20200a2b90 0x7f20200a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2028005ab0 tx=0x7f2028005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:06.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:06.945+0000 7f20357fa640 1 -- 172.21.15.107:0/449726152 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f202c09b670 con 0x7f2038101980 2024-05-09T08:31:07.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:06 smithi110 ceph-mon[33622]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:31:07.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:06 smithi110 ceph-mon[33622]: osdmap e89: 12 total, 12 up, 12 in 2024-05-09T08:31:07.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:06 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3073188770' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-09T08:31:07.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:06 smithi110 ceph-mon[33622]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-09T08:31:07.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:06 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2997580959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:31:07.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:06 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1133983252' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:07.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:06 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1851307196' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:07.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.159+0000 7f203e8fc640 1 -- 172.21.15.107:0/449726152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2038062460 con 0x7f2038101980 2024-05-09T08:31:07.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.159+0000 7f20357fa640 1 -- 172.21.15.107:0/449726152 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11815 (secure 0 0 0) 0x7f202c09af70 con 0x7f2038101980 2024-05-09T08:31:07.159 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:07.159 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24491,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:31:05.201970+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24499,"qdb_cluster":[24499]},"id":1}]} 2024-05-09T08:31:07.159 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2024-05-09T08:31:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.161+0000 7f203e8fc640 1 -- 172.21.15.107:0/449726152 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f20200a2b90 msgr2=0x7f20200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.161+0000 7f203e8fc640 1 --2- 172.21.15.107:0/449726152 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f20200a2b90 0x7f20200a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2028005ab0 tx=0x7f2028005a20 comp rx=0 tx=0).stop 2024-05-09T08:31:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.161+0000 7f203e8fc640 1 -- 172.21.15.107:0/449726152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2038101980 msgr2=0x7f2038198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.161+0000 7f203e8fc640 1 --2- 172.21.15.107:0/449726152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2038101980 0x7f2038198130 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f202c00b4a0 tx=0x7f202c00b960 comp rx=0 tx=0).stop 2024-05-09T08:31:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.161+0000 7f203e8fc640 1 -- 172.21.15.107:0/449726152 shutdown_connections 2024-05-09T08:31:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.161+0000 7f203e8fc640 1 --2- 172.21.15.107:0/449726152 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f20200a2b90 0x7f20200a5050 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.161+0000 7f203e8fc640 1 --2- 172.21.15.107:0/449726152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2038102340 0x7f203819f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.161+0000 7f203e8fc640 1 --2- 172.21.15.107:0/449726152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2038101980 0x7f2038198130 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.161+0000 7f203e8fc640 1 --2- 172.21.15.107:0/449726152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2038100790 0x7f2038197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.161+0000 7f203e8fc640 1 -- 172.21.15.107:0/449726152 >> 172.21.15.107:0/449726152 conn(0x7f20380fbd20 msgr2=0x7f2038104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.161+0000 7f203e8fc640 1 -- 172.21.15.107:0/449726152 shutdown_connections 2024-05-09T08:31:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.161+0000 7f203e8fc640 1 -- 172.21.15.107:0/449726152 wait complete. 2024-05-09T08:31:07.171 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:06 smithi107 ceph-mon[30122]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:31:07.171 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:06 smithi107 ceph-mon[30122]: osdmap e89: 12 total, 12 up, 12 in 2024-05-09T08:31:07.171 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:06 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3073188770' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-09T08:31:07.171 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:06 smithi107 ceph-mon[30122]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-09T08:31:07.171 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:06 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2997580959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:31:07.171 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:06 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1133983252' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:07.171 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:06 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1851307196' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:07.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:31:07.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 -- 172.21.15.107:0/3281933831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4660102310 msgr2=0x7f4660104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:07.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 --2- 172.21.15.107:0/3281933831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4660102310 0x7f4660104700 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4654009900 tx=0x7f465402a410 comp rx=0 tx=0).stop 2024-05-09T08:31:07.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 -- 172.21.15.107:0/3281933831 shutdown_connections 2024-05-09T08:31:07.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 --2- 172.21.15.107:0/3281933831 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4660107570 0x7f46601099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 --2- 172.21.15.107:0/3281933831 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4660104c40 0x7f4660107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 --2- 172.21.15.107:0/3281933831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4660102310 0x7f4660104700 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 -- 172.21.15.107:0/3281933831 >> 172.21.15.107:0/3281933831 conn(0x7f46600fbce0 msgr2=0x7f46600fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:07.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 -- 172.21.15.107:0/3281933831 shutdown_connections 2024-05-09T08:31:07.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 -- 172.21.15.107:0/3281933831 wait complete. 2024-05-09T08:31:07.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 Processor -- start 2024-05-09T08:31:07.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 -- start start 2024-05-09T08:31:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4660102310 0x7f4660197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4660104c40 0x7f4660198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4660107570 0x7f466019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f466010d3c0 con 0x7f4660107570 2024-05-09T08:31:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f466010d240 con 0x7f4660102310 2024-05-09T08:31:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.274+0000 7f466844d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f466010d540 con 0x7f4660104c40 2024-05-09T08:31:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f46669ea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4660107570 0x7f466019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f46669ea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4660107570 0x7f466019f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40292/0 (socket says 172.21.15.107:40292) 2024-05-09T08:31:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f46669ea640 1 -- 172.21.15.107:0/3186599856 learned_addr learned my addr 172.21.15.107:0/3186599856 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f46659e8640 1 --2- 172.21.15.107:0/3186599856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4660104c40 0x7f4660198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f46661e9640 1 --2- 172.21.15.107:0/3186599856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4660102310 0x7f4660197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f46669ea640 1 -- 172.21.15.107:0/3186599856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4660104c40 msgr2=0x7f4660198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f46669ea640 1 --2- 172.21.15.107:0/3186599856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4660104c40 0x7f4660198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f46669ea640 1 -- 172.21.15.107:0/3186599856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4660102310 msgr2=0x7f4660197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f46669ea640 1 --2- 172.21.15.107:0/3186599856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4660102310 0x7f4660197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f46669ea640 1 -- 172.21.15.107:0/3186599856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f466019f6f0 con 0x7f4660107570 2024-05-09T08:31:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f46659e8640 1 --2- 172.21.15.107:0/3186599856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4660104c40 0x7f4660198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f46661e9640 1 --2- 172.21.15.107:0/3186599856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4660102310 0x7f4660197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f46669ea640 1 --2- 172.21.15.107:0/3186599856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4660107570 0x7f466019f080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f465800eb10 tx=0x7f465800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f46537fe640 1 -- 172.21.15.107:0/3186599856 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4658004b20 con 0x7f4660107570 2024-05-09T08:31:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f46537fe640 1 -- 172.21.15.107:0/3186599856 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4658004cc0 con 0x7f4660107570 2024-05-09T08:31:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f466844d640 1 -- 172.21.15.107:0/3186599856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f466019f9e0 con 0x7f4660107570 2024-05-09T08:31:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f46537fe640 1 -- 172.21.15.107:0/3186599856 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4658010690 con 0x7f4660107570 2024-05-09T08:31:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.275+0000 7f466844d640 1 -- 172.21.15.107:0/3186599856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f466019ff80 con 0x7f4660107570 2024-05-09T08:31:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.276+0000 7f466844d640 1 -- 172.21.15.107:0/3186599856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46600fd7d0 con 0x7f4660107570 2024-05-09T08:31:07.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.277+0000 7f46537fe640 1 -- 172.21.15.107:0/3186599856 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f4658005020 con 0x7f4660107570 2024-05-09T08:31:07.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.278+0000 7f46537fe640 1 --2- 172.21.15.107:0/3186599856 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f464c0a2c10 0x7f464c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.278+0000 7f46661e9640 1 --2- 172.21.15.107:0/3186599856 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f464c0a2c10 0x7f464c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.278+0000 7f46537fe640 1 -- 172.21.15.107:0/3186599856 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f46580cfa30 con 0x7f4660107570 2024-05-09T08:31:07.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.279+0000 7f46661e9640 1 --2- 172.21.15.107:0/3186599856 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f464c0a2c10 0x7f464c0a50d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4654006010 tx=0x7f4654000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:07.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.282+0000 7f46537fe640 1 -- 172.21.15.107:0/3186599856 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f465809c750 con 0x7f4660107570 2024-05-09T08:31:07.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:06 smithi157 ceph-mon[33614]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T08:31:07.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:06 smithi157 ceph-mon[33614]: osdmap e89: 12 total, 12 up, 12 in 2024-05-09T08:31:07.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:06 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3073188770' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-09T08:31:07.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:06 smithi157 ceph-mon[33614]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-09T08:31:07.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:06 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2997580959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:31:07.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:06 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1133983252' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:07.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:06 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1851307196' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.459+0000 7f466844d640 1 -- 172.21.15.107:0/3186599856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4660062430 con 0x7f4660107570 2024-05-09T08:31:07.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.461+0000 7f46537fe640 1 -- 172.21.15.107:0/3186599856 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f465809c050 con 0x7f4660107570 2024-05-09T08:31:07.460 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:07.461 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:31:05.201799+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:31:07.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.463+0000 7f466844d640 1 -- 172.21.15.107:0/3186599856 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f464c0a2c10 msgr2=0x7f464c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:07.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.463+0000 7f466844d640 1 --2- 172.21.15.107:0/3186599856 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f464c0a2c10 0x7f464c0a50d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4654006010 tx=0x7f4654000bc0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.463+0000 7f466844d640 1 -- 172.21.15.107:0/3186599856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4660107570 msgr2=0x7f466019f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.463+0000 7f466844d640 1 --2- 172.21.15.107:0/3186599856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4660107570 0x7f466019f080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f465800eb10 tx=0x7f465800c2d0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.463+0000 7f466844d640 1 -- 172.21.15.107:0/3186599856 shutdown_connections 2024-05-09T08:31:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.463+0000 7f466844d640 1 --2- 172.21.15.107:0/3186599856 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f464c0a2c10 0x7f464c0a50d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.463+0000 7f466844d640 1 --2- 172.21.15.107:0/3186599856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4660107570 0x7f466019f080 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.463+0000 7f466844d640 1 --2- 172.21.15.107:0/3186599856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4660104c40 0x7f4660198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.463+0000 7f466844d640 1 --2- 172.21.15.107:0/3186599856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4660102310 0x7f4660197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.463+0000 7f466844d640 1 -- 172.21.15.107:0/3186599856 >> 172.21.15.107:0/3186599856 conn(0x7f46600fbce0 msgr2=0x7f4660102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.463+0000 7f466844d640 1 -- 172.21.15.107:0/3186599856 shutdown_connections 2024-05-09T08:31:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.463+0000 7f466844d640 1 -- 172.21.15.107:0/3186599856 wait complete. 2024-05-09T08:31:07.474 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T08:31:00.157448+0000', 'modified': '2024-05-09T08:31:05.201970+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24499}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.157:6839/3003356919', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3003356919}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3003356919}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24499, 'qdb_cluster': [24499]} 2024-05-09T08:31:07.474 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T08:31:07.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-05-09T08:31:07.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 -- 172.21.15.107:0/862926129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9398100780 msgr2=0x7f9398100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:07.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 --2- 172.21.15.107:0/862926129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9398100780 0x7f9398100ba0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9388009900 tx=0x7f938802a410 comp rx=0 tx=0).stop 2024-05-09T08:31:07.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 -- 172.21.15.107:0/862926129 shutdown_connections 2024-05-09T08:31:07.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 --2- 172.21.15.107:0/862926129 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9398102330 0x7f9398108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 --2- 172.21.15.107:0/862926129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9398101970 0x7f9398101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 --2- 172.21.15.107:0/862926129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9398100780 0x7f9398100ba0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 -- 172.21.15.107:0/862926129 >> 172.21.15.107:0/862926129 conn(0x7f93980fbcf0 msgr2=0x7f93980fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:07.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 -- 172.21.15.107:0/862926129 shutdown_connections 2024-05-09T08:31:07.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 -- 172.21.15.107:0/862926129 wait complete. 2024-05-09T08:31:07.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 Processor -- start 2024-05-09T08:31:07.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 -- start start 2024-05-09T08:31:07.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9398100780 0x7f939806a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9398101970 0x7f939819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9398102330 0x7f939819eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f939810c300 con 0x7f9398101970 2024-05-09T08:31:07.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f939810c180 con 0x7f9398102330 2024-05-09T08:31:07.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.578+0000 7f939e4a5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f939810c480 con 0x7f9398100780 2024-05-09T08:31:07.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.579+0000 7f938ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9398101970 0x7f939819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.579+0000 7f9397fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9398100780 0x7f939806a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.579+0000 7f938ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9398101970 0x7f939819c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40308/0 (socket says 172.21.15.107:40308) 2024-05-09T08:31:07.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.579+0000 7f938ffff640 1 -- 172.21.15.107:0/1096150364 learned_addr learned my addr 172.21.15.107:0/1096150364 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:07.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.579+0000 7f939ca42640 1 --2- 172.21.15.107:0/1096150364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9398102330 0x7f939819eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.579+0000 7f938ffff640 1 -- 172.21.15.107:0/1096150364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9398100780 msgr2=0x7f939806a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:07.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.579+0000 7f938ffff640 1 --2- 172.21.15.107:0/1096150364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9398100780 0x7f939806a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.579+0000 7f938ffff640 1 -- 172.21.15.107:0/1096150364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9398102330 msgr2=0x7f939819eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:07.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.579+0000 7f938ffff640 1 --2- 172.21.15.107:0/1096150364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9398102330 0x7f939819eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.579+0000 7f938ffff640 1 -- 172.21.15.107:0/1096150364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f939819f7b0 con 0x7f9398101970 2024-05-09T08:31:07.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.579+0000 7f938ffff640 1 --2- 172.21.15.107:0/1096150364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9398101970 0x7f939819c880 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f938000d950 tx=0x7f938000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:07.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.579+0000 7f9395ffb640 1 -- 172.21.15.107:0/1096150364 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9380017070 con 0x7f9398101970 2024-05-09T08:31:07.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.579+0000 7f939e4a5640 1 -- 172.21.15.107:0/1096150364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f939819fa40 con 0x7f9398101970 2024-05-09T08:31:07.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.579+0000 7f9395ffb640 1 -- 172.21.15.107:0/1096150364 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9380009c30 con 0x7f9398101970 2024-05-09T08:31:07.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.579+0000 7f9395ffb640 1 -- 172.21.15.107:0/1096150364 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938000f620 con 0x7f9398101970 2024-05-09T08:31:07.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.579+0000 7f939e4a5640 1 -- 172.21.15.107:0/1096150364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f939819ff90 con 0x7f9398101970 2024-05-09T08:31:07.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.581+0000 7f9395ffb640 1 -- 172.21.15.107:0/1096150364 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f93800109a0 con 0x7f9398101970 2024-05-09T08:31:07.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f93a11ba740 1 Processor -- start 2024-05-09T08:31:07.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f93a11ba740 1 -- start start 2024-05-09T08:31:07.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f93a11ba740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d608c0 0x55d266d60ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f93a11ba740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d612b0 0x55d266d65db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f93a11ba740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d66420 0x55d266d688b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f93a11ba740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d266cc44d0 con 0x55d266d66420 2024-05-09T08:31:07.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f93a11ba740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d266c6c3e0 con 0x55d266d612b0 2024-05-09T08:31:07.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f93a11ba740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d266cc0490 con 0x55d266d608c0 2024-05-09T08:31:07.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f938ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d66420 0x55d266d688b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f938ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d66420 0x55d266d688b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40320/0 (socket says 172.21.15.107:40320) 2024-05-09T08:31:07.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f938ffff640 1 -- 172.21.15.107:0/3483463595 learned_addr learned my addr 172.21.15.107:0/3483463595 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:07.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f9395ffb640 1 --2- 172.21.15.107:0/1096150364 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f93840a2b90 0x7f93840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f9397fff640 1 --2- 172.21.15.107:0/3483463595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d612b0 0x55d266d65db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f9395ffb640 1 -- 172.21.15.107:0/1096150364 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f93800cee00 con 0x7f9398101970 2024-05-09T08:31:07.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f939ca42640 1 --2- 172.21.15.107:0/3483463595 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d608c0 0x55d266d60ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f938ffff640 1 -- 172.21.15.107:0/3483463595 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d608c0 msgr2=0x55d266d60ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:07.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.582+0000 7f938ffff640 1 --2- 172.21.15.107:0/3483463595 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d608c0 0x55d266d60ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.583+0000 7f938ffff640 1 -- 172.21.15.107:0/3483463595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d612b0 msgr2=0x55d266d65db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:07.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.583+0000 7f938ffff640 1 --2- 172.21.15.107:0/3483463595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d612b0 0x55d266d65db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.583+0000 7f938ffff640 1 -- 172.21.15.107:0/3483463595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2669d82d0 con 0x55d266d66420 2024-05-09T08:31:07.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.583+0000 7f939ca42640 1 --2- 172.21.15.107:0/3483463595 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d608c0 0x55d266d60ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:07.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.583+0000 7f939ca42640 1 --2- 172.21.15.107:0/1096150364 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f93840a2b90 0x7f93840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.583+0000 7f938ffff640 1 --2- 172.21.15.107:0/3483463595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d66420 0x55d266d688b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9380097930 tx=0x7f9380064b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd357e7f915e29b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:07.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.583+0000 7f938e7fc640 1 -- 172.21.15.107:0/3483463595 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9380020030 con 0x55d266d66420 2024-05-09T08:31:07.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.583+0000 7f939ca42640 1 --2- 172.21.15.107:0/1096150364 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f93840a2b90 0x7f93840a5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x55d266d61b10 tx=0x7f9390009600 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:07.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.583+0000 7f938e7fc640 1 -- 172.21.15.107:0/3483463595 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9380009de0 con 0x55d266d66420 2024-05-09T08:31:07.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.583+0000 7f938e7fc640 1 -- 172.21.15.107:0/3483463595 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9380097b40 con 0x55d266d66420 2024-05-09T08:31:07.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.583+0000 7f93a11ba740 1 -- 172.21.15.107:0/3483463595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d66420 msgr2=0x55d266d688b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:07.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.583+0000 7f93a11ba740 1 --2- 172.21.15.107:0/3483463595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d66420 0x55d266d688b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9380097930 tx=0x7f9380064b60 comp rx=0 tx=0).stop 2024-05-09T08:31:07.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f93a11ba740 1 -- 172.21.15.107:0/3483463595 shutdown_connections 2024-05-09T08:31:07.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f93a11ba740 1 --2- 172.21.15.107:0/3483463595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d66420 0x55d266d688b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f93a11ba740 1 --2- 172.21.15.107:0/3483463595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d612b0 0x55d266d65db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f93a11ba740 1 --2- 172.21.15.107:0/3483463595 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d608c0 0x55d266d60ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f93a11ba740 1 -- 172.21.15.107:0/3483463595 >> 172.21.15.107:0/3483463595 conn(0x55d266d5bf30 msgr2=0x55d266d5c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:07.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f93a11ba740 1 -- 172.21.15.107:0/3483463595 shutdown_connections 2024-05-09T08:31:07.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f93a11ba740 1 -- 172.21.15.107:0/3483463595 wait complete. 2024-05-09T08:31:07.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f93a11ba740 1 -- start start 2024-05-09T08:31:07.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f93a11ba740 1 Processor -- start 2024-05-09T08:31:07.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f93a11ba740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d608c0 0x55d266d6ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f93a11ba740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d612b0 0x55d266d6b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f93a11ba740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d66420 0x55d266d721b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f93a11ba740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d266d69370 con 0x55d266d608c0 2024-05-09T08:31:07.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f93a11ba740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d266c7f8e0 con 0x55d266d612b0 2024-05-09T08:31:07.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f93a11ba740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d266d694f0 con 0x55d266d66420 2024-05-09T08:31:07.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f93a11ba740 10 client.0 _ll_register_callbacks cb 0x55d266c4c9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:31:07.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.584+0000 7f9397fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d608c0 0x55d266d6ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f9397fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d608c0 0x55d266d6ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40328/0 (socket says 172.21.15.107:40328) 2024-05-09T08:31:07.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f9397fff640 1 -- 172.21.15.107:0/1022565636 learned_addr learned my addr 172.21.15.107:0/1022565636 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:07.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f939ca42640 1 --2- 172.21.15.107:0/1022565636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d66420 0x55d266d721b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f938ffff640 1 --2- 172.21.15.107:0/1022565636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d612b0 0x55d266d6b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f9397fff640 1 -- 172.21.15.107:0/1022565636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d66420 msgr2=0x55d266d721b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:07.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f9397fff640 1 --2- 172.21.15.107:0/1022565636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d66420 0x55d266d721b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f9397fff640 1 -- 172.21.15.107:0/1022565636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d612b0 msgr2=0x55d266d6b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:07.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f9397fff640 1 --2- 172.21.15.107:0/1022565636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d612b0 0x55d266d6b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f9397fff640 1 -- 172.21.15.107:0/1022565636 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d266aa26b0 con 0x55d266d608c0 2024-05-09T08:31:07.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f938ffff640 1 --2- 172.21.15.107:0/1022565636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d612b0 0x55d266d6b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:07.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f9397fff640 1 --2- 172.21.15.107:0/1022565636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d608c0 0x55d266d6ac70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f93981017d0 tx=0x7f93880044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5eef44f3e19c33b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:07.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f93777fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:31:07.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f93777fe640 1 -- 172.21.15.107:0/1022565636 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9388035040 con 0x55d266d608c0 2024-05-09T08:31:07.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f93777fe640 1 -- 172.21.15.107:0/1022565636 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9388005590 con 0x55d266d608c0 2024-05-09T08:31:07.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f93a11ba740 1 -- 172.21.15.107:0/1022565636 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9380009de0 con 0x55d266d608c0 2024-05-09T08:31:07.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f93777fe640 1 -- 172.21.15.107:0/1022565636 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f9388004750 con 0x55d266d608c0 2024-05-09T08:31:07.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f93a11ba740 10 client.14775 fetch_fsmap learned FSMap version 17 2024-05-09T08:31:07.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f93a11ba740 1 -- 172.21.15.107:0/1022565636 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55d266d75190 con 0x55d266d608c0 2024-05-09T08:31:07.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.585+0000 7f93777fe640 1 -- 172.21.15.107:0/1022565636 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9380009de0 con 0x55d266d608c0 2024-05-09T08:31:07.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.586+0000 7f93777fe640 1 -- 172.21.15.107:0/1022565636 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f9388003b90 con 0x55d266d608c0 2024-05-09T08:31:07.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.586+0000 7f93a11ba740 10 client.14775 fetch_fsmap finished waiting for FSMap version 17 2024-05-09T08:31:07.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.586+0000 7f93a11ba740 10 client.14775 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T08:31:07.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.586+0000 7f93a11ba740 20 client.14775 populate_metadata read hostname 'smithi107' 2024-05-09T08:31:07.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.586+0000 7f93a11ba740 1 --2- 172.21.15.107:0/1022565636 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d266d777f0 0x55d266d78c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.586+0000 7f93a11ba740 4 client.14775 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:31:07.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.586+0000 7f93a11ba740 1 -- 172.21.15.107:0/1022565636 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d266cd1c00 con 0x55d266d777f0 2024-05-09T08:31:07.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.586+0000 7f938ffff640 1 --2- 172.21.15.107:0/1022565636 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d266d777f0 0x55d266d78c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.586+0000 7f93777fe640 10 client.14775 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:31:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.586+0000 7f938ffff640 1 --2- 172.21.15.107:0/1022565636 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d266d777f0 0x55d266d78c60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cf8a1b29a9fc34d server_cookie=be8aab5589c55980 in_seq=0 out_seq=0 2024-05-09T08:31:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.587+0000 7f93777fe640 1 -- 172.21.15.107:0/1022565636 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d266cd1c00 con 0x55d266d777f0 2024-05-09T08:31:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.587+0000 7f93777fe640 10 client.14775 handle_command_reply: tid=0 2024-05-09T08:31:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.587+0000 7f93a11ba740 1 client.14775 shutdown 2024-05-09T08:31:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.587+0000 7f93a11ba740 1 -- 172.21.15.107:0/1022565636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d608c0 msgr2=0x55d266d6ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.587+0000 7f93a11ba740 1 --2- 172.21.15.107:0/1022565636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d608c0 0x55d266d6ac70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f93981017d0 tx=0x7f93880044f0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.587+0000 7f93a11ba740 1 -- 172.21.15.107:0/1022565636 shutdown_connections 2024-05-09T08:31:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.587+0000 7f93a11ba740 1 --2- 172.21.15.107:0/1022565636 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d266d777f0 0x55d266d78c60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.587+0000 7f93a11ba740 1 --2- 172.21.15.107:0/1022565636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d66420 0x55d266d721b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.588+0000 7f93a11ba740 1 --2- 172.21.15.107:0/1022565636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d612b0 0x55d266d6b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.588+0000 7f93777fe640 0 client.14775 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:31:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.588+0000 7f93a11ba740 1 --2- 172.21.15.107:0/1022565636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d608c0 0x55d266d6ac70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.588+0000 7f93a11ba740 1 -- 172.21.15.107:0/1022565636 >> 172.21.15.107:0/1022565636 conn(0x55d266d5bf30 msgr2=0x55d266d5b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:07.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.588+0000 7f93a11ba740 1 -- 172.21.15.107:0/1022565636 shutdown_connections 2024-05-09T08:31:07.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.588+0000 7f93a11ba740 1 -- 172.21.15.107:0/1022565636 wait complete. 2024-05-09T08:31:07.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.588+0000 7f93a11ba740 20 client.14775 trim_cache size 0 max 16384 2024-05-09T08:31:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.605+0000 7f93a11ba740 1 Processor -- start 2024-05-09T08:31:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.605+0000 7f93a11ba740 1 -- start start 2024-05-09T08:31:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f93a11ba740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d608c0 0x55d266d62680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f93a11ba740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d612b0 0x55d266d72170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f93a11ba740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d66420 0x55d266d62bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f93a11ba740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d266d76fa0 con 0x55d266d612b0 2024-05-09T08:31:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f93a11ba740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d266d77670 con 0x55d266d66420 2024-05-09T08:31:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f93a11ba740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d266d77d40 con 0x55d266d608c0 2024-05-09T08:31:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f938ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d612b0 0x55d266d72170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f938ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d612b0 0x55d266d72170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40338/0 (socket says 172.21.15.107:40338) 2024-05-09T08:31:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f938ffff640 1 -- 172.21.15.107:0/2790216589 learned_addr learned my addr 172.21.15.107:0/2790216589 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f9397fff640 1 --2- 172.21.15.107:0/2790216589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d608c0 0x55d266d62680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f938ffff640 1 -- 172.21.15.107:0/2790216589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d608c0 msgr2=0x55d266d62680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f938ffff640 1 --2- 172.21.15.107:0/2790216589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d608c0 0x55d266d62680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f939ca42640 1 --2- 172.21.15.107:0/2790216589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d66420 0x55d266d62bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f938ffff640 1 -- 172.21.15.107:0/2790216589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d66420 msgr2=0x55d266d62bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f938ffff640 1 --2- 172.21.15.107:0/2790216589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d66420 0x55d266d62bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f938ffff640 1 -- 172.21.15.107:0/2790216589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d266d5ba70 con 0x55d266d612b0 2024-05-09T08:31:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f939ca42640 1 --2- 172.21.15.107:0/2790216589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d66420 0x55d266d62bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f938ffff640 1 --2- 172.21.15.107:0/2790216589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d612b0 0x55d266d72170 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f93800cca30 tx=0x7f93800976d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e304b49f07e089a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f938dffb640 1 -- 172.21.15.107:0/2790216589 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9380020040 con 0x55d266d612b0 2024-05-09T08:31:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f938dffb640 1 -- 172.21.15.107:0/2790216589 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9380080e50 con 0x55d266d612b0 2024-05-09T08:31:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.606+0000 7f938dffb640 1 -- 172.21.15.107:0/2790216589 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9380097480 con 0x55d266d612b0 2024-05-09T08:31:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.607+0000 7f93a11ba740 1 -- 172.21.15.107:0/2790216589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d612b0 msgr2=0x55d266d72170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.607+0000 7f93a11ba740 1 --2- 172.21.15.107:0/2790216589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d612b0 0x55d266d72170 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f93800cca30 tx=0x7f93800976d0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.607+0000 7f93a11ba740 1 -- 172.21.15.107:0/2790216589 shutdown_connections 2024-05-09T08:31:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.607+0000 7f93a11ba740 1 --2- 172.21.15.107:0/2790216589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d66420 0x55d266d62bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.607+0000 7f93a11ba740 1 --2- 172.21.15.107:0/2790216589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d612b0 0x55d266d72170 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.607+0000 7f93a11ba740 1 --2- 172.21.15.107:0/2790216589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d608c0 0x55d266d62680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.607+0000 7f93a11ba740 1 -- 172.21.15.107:0/2790216589 >> 172.21.15.107:0/2790216589 conn(0x55d266d5bf30 msgr2=0x55d266d5c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.607+0000 7f93a11ba740 1 -- 172.21.15.107:0/2790216589 shutdown_connections 2024-05-09T08:31:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.607+0000 7f93a11ba740 1 -- 172.21.15.107:0/2790216589 wait complete. 2024-05-09T08:31:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.607+0000 7f93a11ba740 1 -- start start 2024-05-09T08:31:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.607+0000 7f93a11ba740 1 Processor -- start 2024-05-09T08:31:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.607+0000 7f93a11ba740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d608c0 0x55d266d6d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.607+0000 7f93a11ba740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d612b0 0x55d266d6da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f93a11ba740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d66420 0x55d266d70aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f93a11ba740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d266d78ae0 con 0x55d266d612b0 2024-05-09T08:31:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f93a11ba740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d266d78410 con 0x55d266d608c0 2024-05-09T08:31:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f93a11ba740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d266d791b0 con 0x55d266d66420 2024-05-09T08:31:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f93a11ba740 10 client.0 _ll_register_callbacks cb 0x7f9388009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:31:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f938ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d612b0 0x55d266d6da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f938ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d612b0 0x55d266d6da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40350/0 (socket says 172.21.15.107:40350) 2024-05-09T08:31:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f938ffff640 1 -- 172.21.15.107:0/936376643 learned_addr learned my addr 172.21.15.107:0/936376643 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f939ca42640 1 --2- 172.21.15.107:0/936376643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d66420 0x55d266d70aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f9397fff640 1 --2- 172.21.15.107:0/936376643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d608c0 0x55d266d6d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f938ffff640 1 -- 172.21.15.107:0/936376643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d66420 msgr2=0x55d266d70aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f938ffff640 1 --2- 172.21.15.107:0/936376643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d66420 0x55d266d70aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f938ffff640 1 -- 172.21.15.107:0/936376643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d608c0 msgr2=0x55d266d6d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:31:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f938ffff640 1 --2- 172.21.15.107:0/936376643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d608c0 0x55d266d6d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f938ffff640 1 -- 172.21.15.107:0/936376643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d266d71090 con 0x55d266d612b0 2024-05-09T08:31:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f9397fff640 1 --2- 172.21.15.107:0/936376643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d608c0 0x55d266d6d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f938ffff640 1 --2- 172.21.15.107:0/936376643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d612b0 0x55d266d6da80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f938009b570 tx=0x7f9380097170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e41271495d778f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f93777fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:31:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f93777fe640 1 -- 172.21.15.107:0/936376643 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9380020040 con 0x55d266d612b0 2024-05-09T08:31:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f93777fe640 1 -- 172.21.15.107:0/936376643 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9380097430 con 0x55d266d612b0 2024-05-09T08:31:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f93a11ba740 1 -- 172.21.15.107:0/936376643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9380080e50 con 0x55d266d612b0 2024-05-09T08:31:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f93777fe640 1 -- 172.21.15.107:0/936376643 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f9380009de0 con 0x55d266d612b0 2024-05-09T08:31:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f93a11ba740 10 client.14781 fetch_fsmap learned FSMap version 17 2024-05-09T08:31:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f93a11ba740 1 -- 172.21.15.107:0/936376643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55d266d72300 con 0x55d266d612b0 2024-05-09T08:31:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.608+0000 7f93777fe640 1 -- 172.21.15.107:0/936376643 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9380080e50 con 0x55d266d612b0 2024-05-09T08:31:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.609+0000 7f93777fe640 1 -- 172.21.15.107:0/936376643 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f93800ccb80 con 0x55d266d612b0 2024-05-09T08:31:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.609+0000 7f93a11ba740 10 client.14781 fetch_fsmap finished waiting for FSMap version 17 2024-05-09T08:31:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.609+0000 7f93a11ba740 10 client.14781 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T08:31:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.609+0000 7f93a11ba740 20 client.14781 populate_metadata read hostname 'smithi107' 2024-05-09T08:31:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.609+0000 7f93a11ba740 1 --2- 172.21.15.107:0/936376643 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d266de1890 0x55d266de05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.609+0000 7f93a11ba740 4 client.14781 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:31:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.609+0000 7f93a11ba740 1 -- 172.21.15.107:0/936376643 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "status"}) -- 0x55d266ccbf50 con 0x55d266de1890 2024-05-09T08:31:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.609+0000 7f9397fff640 1 --2- 172.21.15.107:0/936376643 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d266de1890 0x55d266de05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.609+0000 7f9397fff640 1 --2- 172.21.15.107:0/936376643 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d266de1890 0x55d266de05a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b86b3148293722d1 server_cookie=3bfb235b59b0cbc6 in_seq=0 out_seq=0 2024-05-09T08:31:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.609+0000 7f93777fe640 10 client.14781 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:31:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.609+0000 7f93777fe640 1 -- 172.21.15.107:0/936376643 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d266ccbf50 con 0x55d266de1890 2024-05-09T08:31:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.609+0000 7f93777fe640 10 client.14781 handle_command_reply: tid=0 2024-05-09T08:31:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.609+0000 7f93a11ba740 1 client.14781 shutdown 2024-05-09T08:31:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f93a11ba740 1 -- 172.21.15.107:0/936376643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d612b0 msgr2=0x55d266d6da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f93a11ba740 1 --2- 172.21.15.107:0/936376643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d612b0 0x55d266d6da80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f938009b570 tx=0x7f9380097170 comp rx=0 tx=0).stop 2024-05-09T08:31:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f93a11ba740 1 -- 172.21.15.107:0/936376643 shutdown_connections 2024-05-09T08:31:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f93a11ba740 1 --2- 172.21.15.107:0/936376643 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d266de1890 0x55d266de05a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f93a11ba740 1 --2- 172.21.15.107:0/936376643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d266d66420 0x55d266d70aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f93777fe640 0 client.14781 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:31:07.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f93a11ba740 1 --2- 172.21.15.107:0/936376643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d266d612b0 0x55d266d6da80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f93a11ba740 1 --2- 172.21.15.107:0/936376643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d266d608c0 0x55d266d6d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f93a11ba740 1 -- 172.21.15.107:0/936376643 >> 172.21.15.107:0/936376643 conn(0x55d266d5bf30 msgr2=0x55d266d62680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:07.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f93a11ba740 1 -- 172.21.15.107:0/936376643 shutdown_connections 2024-05-09T08:31:07.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f93a11ba740 1 -- 172.21.15.107:0/936376643 wait complete. 2024-05-09T08:31:07.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f93a11ba740 20 client.14781 trim_cache size 0 max 16384 2024-05-09T08:31:07.617 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:31:07.617 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "3954c88c-0ddd-11ef-bc98-c7b262605968", 2024-05-09T08:31:07.617 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2024-05-09T08:31:07.617 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24499, 2024-05-09T08:31:07.617 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-05-09T08:31:07.617 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-05-09T08:31:07.618 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-05-09T08:31:07.618 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 6.8009987680000004, 2024-05-09T08:31:07.618 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 17, 2024-05-09T08:31:07.618 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 89, 2024-05-09T08:31:07.618 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:31:07.618 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 21.693173507000001 2024-05-09T08:31:07.618 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:31:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f939e4a5640 1 -- 172.21.15.107:0/1096150364 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f93840a2b90 msgr2=0x7f93840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f939e4a5640 1 --2- 172.21.15.107:0/1096150364 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f93840a2b90 0x7f93840a5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x55d266d61b10 tx=0x7f9390009600 comp rx=0 tx=0).stop 2024-05-09T08:31:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f939e4a5640 1 -- 172.21.15.107:0/1096150364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9398101970 msgr2=0x7f939819c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f939e4a5640 1 --2- 172.21.15.107:0/1096150364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9398101970 0x7f939819c880 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f938000d950 tx=0x7f938000de10 comp rx=0 tx=0).stop 2024-05-09T08:31:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f939e4a5640 1 -- 172.21.15.107:0/1096150364 shutdown_connections 2024-05-09T08:31:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f939e4a5640 1 --2- 172.21.15.107:0/1096150364 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f93840a2b90 0x7f93840a5050 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f939e4a5640 1 --2- 172.21.15.107:0/1096150364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9398102330 0x7f939819eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f939e4a5640 1 --2- 172.21.15.107:0/1096150364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9398101970 0x7f939819c880 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f939e4a5640 1 --2- 172.21.15.107:0/1096150364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9398100780 0x7f939806a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f939e4a5640 1 -- 172.21.15.107:0/1096150364 >> 172.21.15.107:0/1096150364 conn(0x7f93980fbcf0 msgr2=0x7f9398104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f939e4a5640 1 -- 172.21.15.107:0/1096150364 shutdown_connections 2024-05-09T08:31:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:07.610+0000 7f939e4a5640 1 -- 172.21.15.107:0/1096150364 wait complete. 2024-05-09T08:31:07.621 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T08:31:07.621 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:31:07.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph fs ls 2024-05-09T08:31:07.859 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:08.117 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:07 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/449726152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:31:08.117 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:07 smithi107 ceph-mon[30122]: pgmap v295: 129 pgs: 4 creating+peering, 123 active+clean, 2 unknown; 579 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-05-09T08:31:08.117 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:07 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3186599856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:08.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:07 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/449726152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:31:08.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:07 smithi110 ceph-mon[33622]: pgmap v295: 129 pgs: 4 creating+peering, 123 active+clean, 2 unknown; 579 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-05-09T08:31:08.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:07 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3186599856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:08.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:07 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/449726152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:31:08.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:07 smithi157 ceph-mon[33614]: pgmap v295: 129 pgs: 4 creating+peering, 123 active+clean, 2 unknown; 579 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-05-09T08:31:08.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:07 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3186599856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:09.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:08 smithi110 ceph-mon[33622]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:31:09.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:08 smithi110 ceph-mon[33622]: Cluster is now healthy 2024-05-09T08:31:09.221 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:08 smithi107 ceph-mon[30122]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:31:09.221 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:08 smithi107 ceph-mon[30122]: Cluster is now healthy 2024-05-09T08:31:09.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:08 smithi157 ceph-mon[33614]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T08:31:09.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:08 smithi157 ceph-mon[33614]: Cluster is now healthy 2024-05-09T08:31:09.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.650+0000 7f26f5e5f700 1 -- 172.21.15.107:0/1011091488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f01083e0 msgr2=0x7f26f01087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:09.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.650+0000 7f26f5e5f700 1 --2- 172.21.15.107:0/1011091488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f01083e0 0x7f26f01087c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f26e800b0d0 tx=0x7f26e800b3e0 comp rx=0 tx=0).stop 2024-05-09T08:31:09.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.650+0000 7f26f5e5f700 1 -- 172.21.15.107:0/1011091488 shutdown_connections 2024-05-09T08:31:09.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.650+0000 7f26f5e5f700 1 --2- 172.21.15.107:0/1011091488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26f0102d90 0x7f26f010f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:09.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.650+0000 7f26f5e5f700 1 --2- 172.21.15.107:0/1011091488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f01083e0 0x7f26f01087c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:09.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.650+0000 7f26f5e5f700 1 --2- 172.21.15.107:0/1011091488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26f01023d0 0x7f26f0102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:09.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.650+0000 7f26f5e5f700 1 -- 172.21.15.107:0/1011091488 >> 172.21.15.107:0/1011091488 conn(0x7f26f0079480 msgr2=0x7f26f0079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:09.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.650+0000 7f26f5e5f700 1 -- 172.21.15.107:0/1011091488 shutdown_connections 2024-05-09T08:31:09.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.650+0000 7f26f5e5f700 1 -- 172.21.15.107:0/1011091488 wait complete. 2024-05-09T08:31:09.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.651+0000 7f26f5e5f700 1 Processor -- start 2024-05-09T08:31:09.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.651+0000 7f26f5e5f700 1 -- start start 2024-05-09T08:31:09.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26f5e5f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f01023d0 0x7f26f019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:09.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26f5e5f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26f0102d90 0x7f26f019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:09.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26f5e5f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26f01083e0 0x7f26f01987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:09.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26f5e5f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f26f0112590 con 0x7f26f01023d0 2024-05-09T08:31:09.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26f5e5f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f26f0112410 con 0x7f26f0102d90 2024-05-09T08:31:09.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26f5e5f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f26f0112710 con 0x7f26f01083e0 2024-05-09T08:31:09.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26ef7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f01023d0 0x7f26f019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:09.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26ef7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f01023d0 0x7f26f019e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40374/0 (socket says 172.21.15.107:40374) 2024-05-09T08:31:09.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26ef7fe700 1 -- 172.21.15.107:0/1628753976 learned_addr learned my addr 172.21.15.107:0/1628753976 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:09.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26eeffd700 1 --2- 172.21.15.107:0/1628753976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26f0102d90 0x7f26f019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:09.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26ef7fe700 1 -- 172.21.15.107:0/1628753976 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26f01083e0 msgr2=0x7f26f01987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:09.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26ef7fe700 1 --2- 172.21.15.107:0/1628753976 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26f01083e0 0x7f26f01987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:09.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26ef7fe700 1 -- 172.21.15.107:0/1628753976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26f0102d90 msgr2=0x7f26f019eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:09.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26ef7fe700 1 --2- 172.21.15.107:0/1628753976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26f0102d90 0x7f26f019eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:09.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26ef7fe700 1 -- 172.21.15.107:0/1628753976 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f0198f90 con 0x7f26f01023d0 2024-05-09T08:31:09.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26ef7fe700 1 --2- 172.21.15.107:0/1628753976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f01023d0 0x7f26f019e630 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f26e800d010 tx=0x7f26e8007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:09.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26ecff9700 1 -- 172.21.15.107:0/1628753976 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e801f070 con 0x7f26f01023d0 2024-05-09T08:31:09.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26f5e5f700 1 -- 172.21.15.107:0/1628753976 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26f0199220 con 0x7f26f01023d0 2024-05-09T08:31:09.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26ecff9700 1 -- 172.21.15.107:0/1628753976 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26e8003c00 con 0x7f26f01023d0 2024-05-09T08:31:09.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26ecff9700 1 -- 172.21.15.107:0/1628753976 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e801b6a0 con 0x7f26f01023d0 2024-05-09T08:31:09.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.652+0000 7f26f5e5f700 1 -- 172.21.15.107:0/1628753976 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26f01a5490 con 0x7f26f01023d0 2024-05-09T08:31:09.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.653+0000 7f26f5e5f700 1 -- 172.21.15.107:0/1628753976 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26f0103a90 con 0x7f26f01023d0 2024-05-09T08:31:09.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.655+0000 7f26ecff9700 1 -- 172.21.15.107:0/1628753976 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f26e8022070 con 0x7f26f01023d0 2024-05-09T08:31:09.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.655+0000 7f26ecff9700 1 --2- 172.21.15.107:0/1628753976 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f26dc0a2c20 0x7f26dc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:09.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.656+0000 7f26eeffd700 1 --2- 172.21.15.107:0/1628753976 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f26dc0a2c20 0x7f26dc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:09.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.656+0000 7f26ecff9700 1 -- 172.21.15.107:0/1628753976 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f26e800f020 con 0x7f26f01023d0 2024-05-09T08:31:09.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.656+0000 7f26eeffd700 1 --2- 172.21.15.107:0/1628753976 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f26dc0a2c20 0x7f26dc0a50e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f26f0079a80 tx=0x7f26d800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:09.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.659+0000 7f26ecff9700 1 -- 172.21.15.107:0/1628753976 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f26e8029020 con 0x7f26f01023d0 2024-05-09T08:31:09.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.899+0000 7f26f5e5f700 1 -- 172.21.15.107:0/1628753976 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f26f0199ce0 con 0x7f26f01023d0 2024-05-09T08:31:09.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.899+0000 7f26ecff9700 1 -- 172.21.15.107:0/1628753976 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f26e8011390 con 0x7f26f01023d0 2024-05-09T08:31:09.899 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-09T08:31:09.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.901+0000 7f26f5e5f700 1 -- 172.21.15.107:0/1628753976 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f26dc0a2c20 msgr2=0x7f26dc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:09.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.901+0000 7f26f5e5f700 1 --2- 172.21.15.107:0/1628753976 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f26dc0a2c20 0x7f26dc0a50e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f26f0079a80 tx=0x7f26d800b410 comp rx=0 tx=0).stop 2024-05-09T08:31:09.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.901+0000 7f26f5e5f700 1 -- 172.21.15.107:0/1628753976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f01023d0 msgr2=0x7f26f019e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:09.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.901+0000 7f26f5e5f700 1 --2- 172.21.15.107:0/1628753976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f01023d0 0x7f26f019e630 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f26e800d010 tx=0x7f26e8007a20 comp rx=0 tx=0).stop 2024-05-09T08:31:09.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.901+0000 7f26f5e5f700 1 -- 172.21.15.107:0/1628753976 shutdown_connections 2024-05-09T08:31:09.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.901+0000 7f26f5e5f700 1 --2- 172.21.15.107:0/1628753976 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26f01083e0 0x7f26f01987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:09.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.901+0000 7f26f5e5f700 1 --2- 172.21.15.107:0/1628753976 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f26dc0a2c20 0x7f26dc0a50e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:09.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.901+0000 7f26f5e5f700 1 --2- 172.21.15.107:0/1628753976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26f01023d0 0x7f26f019e630 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:09.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.901+0000 7f26f5e5f700 1 --2- 172.21.15.107:0/1628753976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26f0102d90 0x7f26f019eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:09.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.901+0000 7f26f5e5f700 1 -- 172.21.15.107:0/1628753976 >> 172.21.15.107:0/1628753976 conn(0x7f26f0079480 msgr2=0x7f26f0101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:09.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.901+0000 7f26f5e5f700 1 -- 172.21.15.107:0/1628753976 shutdown_connections 2024-05-09T08:31:09.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:09.901+0000 7f26f5e5f700 1 -- 172.21.15.107:0/1628753976 wait complete. 2024-05-09T08:31:10.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:09 smithi110 ceph-mon[33622]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2024-05-09T08:31:10.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:09 smithi107 ceph-mon[30122]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2024-05-09T08:31:10.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:09 smithi157 ceph-mon[33614]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2024-05-09T08:31:10.698 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-09T08:31:10.698 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T08:31:10.698 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2024-05-09T08:31:10.698 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-05-09T08:31:10.698 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:31:10.699 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-09T08:31:10.699 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T08:31:10.699 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T08:31:10.699 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-05-09T08:31:10.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:31:10.699 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2024-05-09T08:31:10.716 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T08:31:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T08:31:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T08:31:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:31:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2024-05-09T08:31:10.716 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:31:10.718 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:31:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2024-05-09T08:31:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2024-05-09T08:31:10.718 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:31:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2024-05-09T08:31:10.718 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2024-05-09T08:31:10.718 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:31:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2024-05-09T08:31:10.719 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T08:31:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2024-05-09T08:31:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-09T08:31:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 42229sec preferred_lft 42229sec 2024-05-09T08:31:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2024-05-09T08:31:10.719 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:31:10.720 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-09T08:31:10.720 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:31:10.720 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-05-09T08:31:10.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add name ceph-brx type bridge 2024-05-09T08:31:10.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr flush dev ceph-brx 2024-05-09T08:31:10.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set ceph-brx up 2024-05-09T08:31:10.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-09T08:31:10.720 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-05-09T08:31:10.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:31:10.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:31:10.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:31:10.911 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T08:31:10.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:10 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1628753976' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:31:10.936 INFO:teuthology.orchestra.run.smithi107.stdout:1 2024-05-09T08:31:10.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:31:10.937 DEBUG:teuthology.orchestra.run.smithi107:> route 2024-05-09T08:31:11.014 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2024-05-09T08:31:11.015 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T08:31:11.015 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T08:31:11.015 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T08:31:11.015 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T08:31:11.015 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:31:11.015 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-05-09T08:31:11.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T08:31:11.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T08:31:11.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T08:31:11.016 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-05-09T08:31:11.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:31:11.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:10 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1628753976' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:31:11.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:31:11.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:31:11.317 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2024-05-09T08:31:11.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:31:11.334 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list-id 2024-05-09T08:31:11.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:10 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1628753976' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:31:11.390 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:31:11.390 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-05-09T08:31:11.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:31:11.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-05-09T08:31:11.390 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-05-09T08:31:11.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:31:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:31:11.510 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-05-09T08:31:11.510 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:31:11.511 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-05-09T08:31:11.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-05-09T08:31:11.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-09T08:31:11.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-05-09T08:31:11.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-05-09T08:31:11.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-05-09T08:31:11.511 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-05-09T08:31:11.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:31:11.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:31:11.639 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:31:11.639 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-05-09T08:31:11.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2024-05-09T08:31:11.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set dev brx.0 master ceph-brx 2024-05-09T08:31:11.640 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-05-09T08:31:11.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:31:11.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:31:11.723 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-05-09T08:31:11.723 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T08:31:11.723 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:31:11.738 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-05-09T08:31:11.739 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T08:31:11.739 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:31:11.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe fuse 2024-05-09T08:31:11.859 DEBUG:teuthology.orchestra.run.smithi107:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T08:31:11.916 INFO:teuthology.orchestra.run.smithi107.stdout:/proc 2024-05-09T08:31:11.916 INFO:teuthology.orchestra.run.smithi107.stdout:/sys 2024-05-09T08:31:11.916 INFO:teuthology.orchestra.run.smithi107.stdout:/dev 2024-05-09T08:31:11.916 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/security 2024-05-09T08:31:11.916 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/shm 2024-05-09T08:31:11.917 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/pts 2024-05-09T08:31:11.917 INFO:teuthology.orchestra.run.smithi107.stdout:/run 2024-05-09T08:31:11.917 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/cgroup 2024-05-09T08:31:11.917 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/pstore 2024-05-09T08:31:11.917 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/bpf 2024-05-09T08:31:11.917 INFO:teuthology.orchestra.run.smithi107.stdout:/ 2024-05-09T08:31:11.917 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/selinux 2024-05-09T08:31:11.917 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T08:31:11.917 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/hugepages 2024-05-09T08:31:11.917 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/mqueue 2024-05-09T08:31:11.917 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/debug 2024-05-09T08:31:11.917 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/tracing 2024-05-09T08:31:11.917 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/fuse/connections 2024-05-09T08:31:11.918 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/config 2024-05-09T08:31:11.918 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T08:31:11.918 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T08:31:11.918 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T08:31:11.918 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T08:31:11.918 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/1000 2024-05-09T08:31:11.918 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph 2024-05-09T08:31:11.918 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay 2024-05-09T08:31:11.918 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/704d9ece21f21936ee943a1df4a489c227907ae684576c39a8326b3dffe1a938/merged 2024-05-09T08:31:11.918 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/1e139d9b7f0b4f3d0acd9e514aed736ee8b756b059cc812befde73c4c74283e1/merged 2024-05-09T08:31:11.918 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/0 2024-05-09T08:31:11.918 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T08:31:11.918 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/a889fd515b5c82d4e0853e5640c5b24fcb34bff091c8962c29ae0a604310eb0b/merged 2024-05-09T08:31:11.919 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/3d41909142bc28d4fd2e879ba0a88e8c06c4840feaa0e281755495824be69163/merged 2024-05-09T08:31:11.919 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/05b85167b2c4ce29a61409c04100fde4c31847c93e288fb610ba6e1602ab8244/merged 2024-05-09T08:31:11.919 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/b7725f5648670ca7bd7869e201bca45c3f7149327f817da5c38cf7746e9f9351/merged 2024-05-09T08:31:11.919 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/f13a66b4af406ede639af2c293c228fc4964dcfda360e9e3f2801e94971040de/merged 2024-05-09T08:31:11.919 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/608847f69c5463ec5a21eb5968f202c44dc6fc079b4197023aeb147c241fa57a/merged 2024-05-09T08:31:11.919 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/5dd9f0ea09fa33d9913a931882dcca1e45f9f3d58177ac85ca98f0378aa9ed6c/merged 2024-05-09T08:31:11.919 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/bf4c3ee07438db841c1cc72e7775f915ef64c3d8c7af00ee701c8214a7dfcec9/merged 2024-05-09T08:31:11.919 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns 2024-05-09T08:31:11.919 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:31:11.919 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:31:11.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:31:11.920 DEBUG:teuthology.orchestra.run.smithi107:> ls /sys/fs/fuse/connections 2024-05-09T08:31:11.973 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T08:31:11.973 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-05-09T08:31:12.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe fuse 2024-05-09T08:31:12.040 DEBUG:teuthology.orchestra.run.smithi107:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T08:31:12.097 INFO:teuthology.orchestra.run.smithi107.stdout:/proc 2024-05-09T08:31:12.097 INFO:teuthology.orchestra.run.smithi107.stdout:/sys 2024-05-09T08:31:12.097 INFO:teuthology.orchestra.run.smithi107.stdout:/dev 2024-05-09T08:31:12.097 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/security 2024-05-09T08:31:12.097 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/shm 2024-05-09T08:31:12.097 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/pts 2024-05-09T08:31:12.097 INFO:teuthology.orchestra.run.smithi107.stdout:/run 2024-05-09T08:31:12.097 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/cgroup 2024-05-09T08:31:12.097 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/pstore 2024-05-09T08:31:12.097 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/bpf 2024-05-09T08:31:12.097 INFO:teuthology.orchestra.run.smithi107.stdout:/ 2024-05-09T08:31:12.097 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/selinux 2024-05-09T08:31:12.098 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T08:31:12.098 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/hugepages 2024-05-09T08:31:12.098 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/mqueue 2024-05-09T08:31:12.098 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/debug 2024-05-09T08:31:12.098 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/tracing 2024-05-09T08:31:12.098 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/fuse/connections 2024-05-09T08:31:12.098 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/config 2024-05-09T08:31:12.098 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T08:31:12.098 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T08:31:12.098 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T08:31:12.098 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T08:31:12.098 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/1000 2024-05-09T08:31:12.098 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph 2024-05-09T08:31:12.098 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay 2024-05-09T08:31:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/704d9ece21f21936ee943a1df4a489c227907ae684576c39a8326b3dffe1a938/merged 2024-05-09T08:31:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/1e139d9b7f0b4f3d0acd9e514aed736ee8b756b059cc812befde73c4c74283e1/merged 2024-05-09T08:31:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/0 2024-05-09T08:31:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T08:31:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/a889fd515b5c82d4e0853e5640c5b24fcb34bff091c8962c29ae0a604310eb0b/merged 2024-05-09T08:31:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/3d41909142bc28d4fd2e879ba0a88e8c06c4840feaa0e281755495824be69163/merged 2024-05-09T08:31:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/05b85167b2c4ce29a61409c04100fde4c31847c93e288fb610ba6e1602ab8244/merged 2024-05-09T08:31:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/b7725f5648670ca7bd7869e201bca45c3f7149327f817da5c38cf7746e9f9351/merged 2024-05-09T08:31:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/f13a66b4af406ede639af2c293c228fc4964dcfda360e9e3f2801e94971040de/merged 2024-05-09T08:31:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/608847f69c5463ec5a21eb5968f202c44dc6fc079b4197023aeb147c241fa57a/merged 2024-05-09T08:31:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/5dd9f0ea09fa33d9913a931882dcca1e45f9f3d58177ac85ca98f0378aa9ed6c/merged 2024-05-09T08:31:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/bf4c3ee07438db841c1cc72e7775f915ef64c3d8c7af00ee701c8214a7dfcec9/merged 2024-05-09T08:31:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns 2024-05-09T08:31:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:31:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:31:12.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:31:12.100 DEBUG:teuthology.orchestra.run.smithi107:> ls /sys/fs/fuse/connections 2024-05-09T08:31:12.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi107.stderr:2024-05-09T08:31:12.118+0000 7fdd5d42d480 -1 init, newargv = 0x560695ace340 newargc=15 2024-05-09T08:31:12.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi107.stderr:ceph-fuse[71586]: starting ceph client 2024-05-09T08:31:12.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:11 smithi110 ceph-mon[33622]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 806 B/s wr, 2 op/s 2024-05-09T08:31:12.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi107.stderr:ceph-fuse[71586]: starting fuse 2024-05-09T08:31:12.134 INFO:teuthology.orchestra.run.smithi107.stdout:114 2024-05-09T08:31:12.134 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [114] 2024-05-09T08:31:12.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo stdin-killer -- python3 -c ' 2024-05-09T08:31:12.134 DEBUG:teuthology.orchestra.run.smithi107:> import glob 2024-05-09T08:31:12.134 DEBUG:teuthology.orchestra.run.smithi107:> import re 2024-05-09T08:31:12.134 DEBUG:teuthology.orchestra.run.smithi107:> import os 2024-05-09T08:31:12.134 DEBUG:teuthology.orchestra.run.smithi107:> import subprocess 2024-05-09T08:31:12.134 DEBUG:teuthology.orchestra.run.smithi107:> 2024-05-09T08:31:12.135 DEBUG:teuthology.orchestra.run.smithi107:> def _find_admin_socket(client_name): 2024-05-09T08:31:12.135 DEBUG:teuthology.orchestra.run.smithi107:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-05-09T08:31:12.135 DEBUG:teuthology.orchestra.run.smithi107:> files = glob.glob(asok_path) 2024-05-09T08:31:12.135 DEBUG:teuthology.orchestra.run.smithi107:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-05-09T08:31:12.135 DEBUG:teuthology.orchestra.run.smithi107:> 2024-05-09T08:31:12.135 DEBUG:teuthology.orchestra.run.smithi107:> # Given a non-glob path, it better be there 2024-05-09T08:31:12.135 DEBUG:teuthology.orchestra.run.smithi107:> if "*" not in asok_path: 2024-05-09T08:31:12.135 DEBUG:teuthology.orchestra.run.smithi107:> assert(len(files) == 1) 2024-05-09T08:31:12.135 DEBUG:teuthology.orchestra.run.smithi107:> return files[0] 2024-05-09T08:31:12.135 DEBUG:teuthology.orchestra.run.smithi107:> 2024-05-09T08:31:12.135 DEBUG:teuthology.orchestra.run.smithi107:> for f in files: 2024-05-09T08:31:12.135 DEBUG:teuthology.orchestra.run.smithi107:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T08:31:12.135 DEBUG:teuthology.orchestra.run.smithi107:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T08:31:12.135 DEBUG:teuthology.orchestra.run.smithi107:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T08:31:12.135 DEBUG:teuthology.orchestra.run.smithi107:> contents = proc_f.read() 2024-05-09T08:31:12.136 DEBUG:teuthology.orchestra.run.smithi107:> if mountpoint in contents: 2024-05-09T08:31:12.136 DEBUG:teuthology.orchestra.run.smithi107:> return f 2024-05-09T08:31:12.136 DEBUG:teuthology.orchestra.run.smithi107:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T08:31:12.136 DEBUG:teuthology.orchestra.run.smithi107:> 2024-05-09T08:31:12.136 DEBUG:teuthology.orchestra.run.smithi107:> print(_find_admin_socket("client.admin")) 2024-05-09T08:31:12.136 DEBUG:teuthology.orchestra.run.smithi107:> ' 2024-05-09T08:31:12.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:11 smithi107 ceph-mon[30122]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 806 B/s wr, 2 op/s 2024-05-09T08:31:12.253 INFO:teuthology.orchestra.run.smithi107.stdout:/var/run/ceph/ceph-client.admin.71586.asok 2024-05-09T08:31:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:31:12.263 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.71586.asok 2024-05-09T08:31:12.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:31:12.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.71586.asok status 2024-05-09T08:31:12.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:11 smithi157 ceph-mon[33614]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 806 B/s wr, 2 op/s 2024-05-09T08:31:12.399 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:31:12.399 INFO:teuthology.orchestra.run.smithi107.stdout: "metadata": { 2024-05-09T08:31:12.399 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_sha1": "96805dcf909541c4ca9a1e90258de8ae5b6fa1bc", 2024-05-09T08:31:12.399 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T08:31:12.399 INFO:teuthology.orchestra.run.smithi107.stdout: "entity_id": "admin", 2024-05-09T08:31:12.399 INFO:teuthology.orchestra.run.smithi107.stdout: "hostname": "smithi107", 2024-05-09T08:31:12.399 INFO:teuthology.orchestra.run.smithi107.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-05-09T08:31:12.399 INFO:teuthology.orchestra.run.smithi107.stdout: "pid": "71586", 2024-05-09T08:31:12.399 INFO:teuthology.orchestra.run.smithi107.stdout: "root": "/" 2024-05-09T08:31:12.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:31:12.399 INFO:teuthology.orchestra.run.smithi107.stdout: "dentry_count": 0, 2024-05-09T08:31:12.400 INFO:teuthology.orchestra.run.smithi107.stdout: "dentry_pinned_count": 0, 2024-05-09T08:31:12.400 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 14793, 2024-05-09T08:31:12.400 INFO:teuthology.orchestra.run.smithi107.stdout: "inst": { 2024-05-09T08:31:12.400 INFO:teuthology.orchestra.run.smithi107.stdout: "name": { 2024-05-09T08:31:12.400 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client", 2024-05-09T08:31:12.400 INFO:teuthology.orchestra.run.smithi107.stdout: "num": 14793 2024-05-09T08:31:12.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:31:12.400 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": { 2024-05-09T08:31:12.400 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-05-09T08:31:12.400 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "192.168.0.1:0", 2024-05-09T08:31:12.400 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3103176492 2024-05-09T08:31:12.400 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:31:12.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:31:12.400 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": { 2024-05-09T08:31:12.400 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-05-09T08:31:12.401 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "192.168.0.1:0", 2024-05-09T08:31:12.401 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 3103176492 2024-05-09T08:31:12.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:31:12.401 INFO:teuthology.orchestra.run.smithi107.stdout: "inst_str": "client.14793 192.168.0.1:0/3103176492", 2024-05-09T08:31:12.401 INFO:teuthology.orchestra.run.smithi107.stdout: "addr_str": "192.168.0.1:0/3103176492", 2024-05-09T08:31:12.401 INFO:teuthology.orchestra.run.smithi107.stdout: "inode_count": 1, 2024-05-09T08:31:12.401 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_epoch": 17, 2024-05-09T08:31:12.401 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_epoch": 89, 2024-05-09T08:31:12.401 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_epoch_barrier": 0, 2024-05-09T08:31:12.401 INFO:teuthology.orchestra.run.smithi107.stdout: "blocklisted": false, 2024-05-09T08:31:12.401 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs" 2024-05-09T08:31:12.401 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:31:12.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:31:12.407 DEBUG:teuthology.orchestra.run.smithi107:> stat --file-system '--printf=%T 2024-05-09T08:31:12.408 DEBUG:teuthology.orchestra.run.smithi107:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:31:12.424 INFO:teuthology.orchestra.run.smithi107.stdout:fuseblk 2024-05-09T08:31:12.424 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:31:12.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:31:12.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:31:12.490 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-05-09T08:31:12.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T08:31:12.612 INFO:teuthology.orchestra.run.smithi107.stdout:# file: . 2024-05-09T08:31:12.612 INFO:teuthology.orchestra.run.smithi107.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-05-09T08:31:12.612 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:12.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:31:12.617 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:31:12.618 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T08:31:12.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:31:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.774+0000 7f4ba3110640 1 -- 172.21.15.107:0/480537304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b9c102a90 msgr2=0x7f4b9c102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.774+0000 7f4ba3110640 1 --2- 172.21.15.107:0/480537304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b9c102a90 0x7f4b9c102f10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009900 tx=0x7f4b8802a410 comp rx=0 tx=0).stop 2024-05-09T08:31:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba3110640 1 -- 172.21.15.107:0/480537304 shutdown_connections 2024-05-09T08:31:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba3110640 1 --2- 172.21.15.107:0/480537304 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b9c103450 0x7f4b9c109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba3110640 1 --2- 172.21.15.107:0/480537304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b9c102a90 0x7f4b9c102f10 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba3110640 1 --2- 172.21.15.107:0/480537304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b9c1018a0 0x7f4b9c101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba3110640 1 -- 172.21.15.107:0/480537304 >> 172.21.15.107:0/480537304 conn(0x7f4b9c0fce30 msgr2=0x7f4b9c0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba3110640 1 -- 172.21.15.107:0/480537304 shutdown_connections 2024-05-09T08:31:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba3110640 1 -- 172.21.15.107:0/480537304 wait complete. 2024-05-09T08:31:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba3110640 1 Processor -- start 2024-05-09T08:31:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba3110640 1 -- start start 2024-05-09T08:31:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba3110640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b9c1018a0 0x7f4b9c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba3110640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b9c102a90 0x7f4b9c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba3110640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b9c103450 0x7f4b9c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba3110640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9c10d3d0 con 0x7f4b9c103450 2024-05-09T08:31:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba3110640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9c10d250 con 0x7f4b9c1018a0 2024-05-09T08:31:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba3110640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9c10d550 con 0x7f4b9c102a90 2024-05-09T08:31:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba16ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b9c103450 0x7f4b9c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba16ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b9c103450 0x7f4b9c19f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37512/0 (socket says 172.21.15.107:37512) 2024-05-09T08:31:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.775+0000 7f4ba16ad640 1 -- 172.21.15.107:0/319624301 learned_addr learned my addr 172.21.15.107:0/319624301 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.776+0000 7f4ba0eac640 1 --2- 172.21.15.107:0/319624301 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b9c1018a0 0x7f4b9c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.776+0000 7f4b9bfff640 1 --2- 172.21.15.107:0/319624301 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b9c102a90 0x7f4b9c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.776+0000 7f4ba16ad640 1 -- 172.21.15.107:0/319624301 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b9c102a90 msgr2=0x7f4b9c198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.776+0000 7f4ba16ad640 1 --2- 172.21.15.107:0/319624301 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b9c102a90 0x7f4b9c198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.776+0000 7f4ba16ad640 1 -- 172.21.15.107:0/319624301 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b9c1018a0 msgr2=0x7f4b9c197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.776+0000 7f4ba16ad640 1 --2- 172.21.15.107:0/319624301 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b9c1018a0 0x7f4b9c197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.776+0000 7f4ba16ad640 1 -- 172.21.15.107:0/319624301 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9c19f720 con 0x7f4b9c103450 2024-05-09T08:31:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.776+0000 7f4ba16ad640 1 --2- 172.21.15.107:0/319624301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b9c103450 0x7f4b9c19f0b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4b9400eaa0 tx=0x7f4b9400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.776+0000 7f4b99ffb640 1 -- 172.21.15.107:0/319624301 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9400cdb0 con 0x7f4b9c103450 2024-05-09T08:31:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.776+0000 7f4b99ffb640 1 -- 172.21.15.107:0/319624301 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b94005020 con 0x7f4b9c103450 2024-05-09T08:31:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.776+0000 7f4b99ffb640 1 -- 172.21.15.107:0/319624301 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b940116d0 con 0x7f4b9c103450 2024-05-09T08:31:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.776+0000 7f4ba3110640 1 -- 172.21.15.107:0/319624301 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b9c19fa10 con 0x7f4b9c103450 2024-05-09T08:31:12.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.776+0000 7f4ba3110640 1 -- 172.21.15.107:0/319624301 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b9c19ff10 con 0x7f4b9c103450 2024-05-09T08:31:12.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.777+0000 7f4ba3110640 1 -- 172.21.15.107:0/319624301 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b9c108ce0 con 0x7f4b9c103450 2024-05-09T08:31:12.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.779+0000 7f4b99ffb640 1 -- 172.21.15.107:0/319624301 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f4b940051d0 con 0x7f4b9c103450 2024-05-09T08:31:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.779+0000 7f4b99ffb640 1 --2- 172.21.15.107:0/319624301 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4b8c0a2b90 0x7f4b8c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.779+0000 7f4ba0eac640 1 --2- 172.21.15.107:0/319624301 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4b8c0a2b90 0x7f4b8c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.779+0000 7f4b99ffb640 1 -- 172.21.15.107:0/319624301 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4b940cf530 con 0x7f4b9c103450 2024-05-09T08:31:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.780+0000 7f4ba0eac640 1 --2- 172.21.15.107:0/319624301 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4b8c0a2b90 0x7f4b8c0a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4b900058c0 tx=0x7f4b9000a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:12.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.784+0000 7f4b99ffb640 1 -- 172.21.15.107:0/319624301 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4b94016030 con 0x7f4b9c103450 2024-05-09T08:31:12.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.962+0000 7f4ba3110640 1 -- 172.21.15.107:0/319624301 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4b9c02d400 con 0x7f4b9c103450 2024-05-09T08:31:12.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.964+0000 7f4b99ffb640 1 -- 172.21.15.107:0/319624301 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f4b9409c250 con 0x7f4b9c103450 2024-05-09T08:31:12.963 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:12.964 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:31:05.201799+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:31:12.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.966+0000 7f4ba3110640 1 -- 172.21.15.107:0/319624301 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4b8c0a2b90 msgr2=0x7f4b8c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:12.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.966+0000 7f4ba3110640 1 --2- 172.21.15.107:0/319624301 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4b8c0a2b90 0x7f4b8c0a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4b900058c0 tx=0x7f4b9000a4c0 comp rx=0 tx=0).stop 2024-05-09T08:31:12.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.966+0000 7f4ba3110640 1 -- 172.21.15.107:0/319624301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b9c103450 msgr2=0x7f4b9c19f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:12.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.966+0000 7f4ba3110640 1 --2- 172.21.15.107:0/319624301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b9c103450 0x7f4b9c19f0b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4b9400eaa0 tx=0x7f4b9400ef60 comp rx=0 tx=0).stop 2024-05-09T08:31:12.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.966+0000 7f4ba3110640 1 -- 172.21.15.107:0/319624301 shutdown_connections 2024-05-09T08:31:12.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.966+0000 7f4ba3110640 1 --2- 172.21.15.107:0/319624301 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f4b8c0a2b90 0x7f4b8c0a5050 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:12.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.966+0000 7f4ba3110640 1 --2- 172.21.15.107:0/319624301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b9c103450 0x7f4b9c19f0b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:12.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.966+0000 7f4ba3110640 1 --2- 172.21.15.107:0/319624301 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b9c102a90 0x7f4b9c198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:12.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.966+0000 7f4ba3110640 1 --2- 172.21.15.107:0/319624301 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b9c1018a0 0x7f4b9c197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:12.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.966+0000 7f4ba3110640 1 -- 172.21.15.107:0/319624301 >> 172.21.15.107:0/319624301 conn(0x7f4b9c0fce30 msgr2=0x7f4b9c105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:12.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.966+0000 7f4ba3110640 1 -- 172.21.15.107:0/319624301 shutdown_connections 2024-05-09T08:31:12.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:12.966+0000 7f4ba3110640 1 -- 172.21.15.107:0/319624301 wait complete. 2024-05-09T08:31:12.977 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:26:21.698338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:30:58.268633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:30:59.345579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:31:03.065991+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:31:12.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:31:13.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.083+0000 7fb49e196640 1 -- 172.21.15.107:0/1365390520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4980757f0 msgr2=0x7fb498073c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.083+0000 7fb49e196640 1 --2- 172.21.15.107:0/1365390520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4980757f0 0x7fb498073c40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb488009900 tx=0x7fb48802a480 comp rx=0 tx=0).stop 2024-05-09T08:31:13.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.083+0000 7fb49e196640 1 -- 172.21.15.107:0/1365390520 shutdown_connections 2024-05-09T08:31:13.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.083+0000 7fb49e196640 1 --2- 172.21.15.107:0/1365390520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb498109740 0x7fb49810bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.083+0000 7fb49e196640 1 --2- 172.21.15.107:0/1365390520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb498074210 0x7fb498074670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.083+0000 7fb49e196640 1 --2- 172.21.15.107:0/1365390520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4980757f0 0x7fb498073c40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.083+0000 7fb49e196640 1 -- 172.21.15.107:0/1365390520 >> 172.21.15.107:0/1365390520 conn(0x7fb4980fcde0 msgr2=0x7fb4980ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:13.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.083+0000 7fb49e196640 1 -- 172.21.15.107:0/1365390520 shutdown_connections 2024-05-09T08:31:13.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.083+0000 7fb49e196640 1 -- 172.21.15.107:0/1365390520 wait complete. 2024-05-09T08:31:13.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.083+0000 7fb49e196640 1 Processor -- start 2024-05-09T08:31:13.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb49e196640 1 -- start start 2024-05-09T08:31:13.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb49e196640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb498074210 0x7fb498199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb49e196640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4980757f0 0x7fb49819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb49e196640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb498109740 0x7fb4981a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb49e196640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb49810f620 con 0x7fb498109740 2024-05-09T08:31:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb49e196640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb49810f4a0 con 0x7fb4980757f0 2024-05-09T08:31:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb49e196640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb49810f7a0 con 0x7fb498074210 2024-05-09T08:31:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb497fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb498109740 0x7fb4981a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb497fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb498109740 0x7fb4981a1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37536/0 (socket says 172.21.15.107:37536) 2024-05-09T08:31:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb497fff640 1 -- 172.21.15.107:0/2642673488 learned_addr learned my addr 172.21.15.107:0/2642673488 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb497fff640 1 -- 172.21.15.107:0/2642673488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb498074210 msgr2=0x7fb498199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb497fff640 1 --2- 172.21.15.107:0/2642673488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb498074210 0x7fb498199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb497fff640 1 -- 172.21.15.107:0/2642673488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4980757f0 msgr2=0x7fb49819a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb496ffd640 1 --2- 172.21.15.107:0/2642673488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4980757f0 0x7fb49819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:13.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb497fff640 1 --2- 172.21.15.107:0/2642673488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4980757f0 0x7fb49819a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb497fff640 1 -- 172.21.15.107:0/2642673488 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4981a1970 con 0x7fb498109740 2024-05-09T08:31:13.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.084+0000 7fb497fff640 1 --2- 172.21.15.107:0/2642673488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb498109740 0x7fb4981a1300 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb49000b970 tx=0x7fb49000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:13.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.085+0000 7fb494ff9640 1 -- 172.21.15.107:0/2642673488 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb490019070 con 0x7fb498109740 2024-05-09T08:31:13.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.085+0000 7fb494ff9640 1 -- 172.21.15.107:0/2642673488 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb490014440 con 0x7fb498109740 2024-05-09T08:31:13.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.085+0000 7fb49e196640 1 -- 172.21.15.107:0/2642673488 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4981a1c60 con 0x7fb498109740 2024-05-09T08:31:13.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.085+0000 7fb494ff9640 1 -- 172.21.15.107:0/2642673488 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4900075d0 con 0x7fb498109740 2024-05-09T08:31:13.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.085+0000 7fb49e196640 1 -- 172.21.15.107:0/2642673488 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4981a2160 con 0x7fb498109740 2024-05-09T08:31:13.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.086+0000 7fb49e196640 1 -- 172.21.15.107:0/2642673488 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb498108500 con 0x7fb498109740 2024-05-09T08:31:13.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.087+0000 7fb494ff9640 1 -- 172.21.15.107:0/2642673488 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fb49001c070 con 0x7fb498109740 2024-05-09T08:31:13.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.087+0000 7fb494ff9640 1 --2- 172.21.15.107:0/2642673488 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb4800a2b90 0x7fb4800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:13.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.087+0000 7fb4977fe640 1 --2- 172.21.15.107:0/2642673488 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb4800a2b90 0x7fb4800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:13.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.087+0000 7fb494ff9640 1 -- 172.21.15.107:0/2642673488 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb4900ce900 con 0x7fb498109740 2024-05-09T08:31:13.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.088+0000 7fb4977fe640 1 --2- 172.21.15.107:0/2642673488 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb4800a2b90 0x7fb4800a5050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb48802a9a0 tx=0x7fb4880314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:13.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.092+0000 7fb494ff9640 1 -- 172.21.15.107:0/2642673488 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb49009b620 con 0x7fb498109740 2024-05-09T08:31:13.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.266+0000 7fb49e196640 1 -- 172.21.15.107:0/2642673488 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb4980624d0 con 0x7fb498109740 2024-05-09T08:31:13.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.266+0000 7fb494ff9640 1 -- 172.21.15.107:0/2642673488 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fb49009af20 con 0x7fb498109740 2024-05-09T08:31:13.266 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-05-09T08:31:13.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.268+0000 7fb49e196640 1 -- 172.21.15.107:0/2642673488 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb4800a2b90 msgr2=0x7fb4800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.268+0000 7fb49e196640 1 --2- 172.21.15.107:0/2642673488 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb4800a2b90 0x7fb4800a5050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb48802a9a0 tx=0x7fb4880314a0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.268+0000 7fb49e196640 1 -- 172.21.15.107:0/2642673488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb498109740 msgr2=0x7fb4981a1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.268+0000 7fb49e196640 1 --2- 172.21.15.107:0/2642673488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb498109740 0x7fb4981a1300 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb49000b970 tx=0x7fb49000be30 comp rx=0 tx=0).stop 2024-05-09T08:31:13.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.268+0000 7fb49e196640 1 -- 172.21.15.107:0/2642673488 shutdown_connections 2024-05-09T08:31:13.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.268+0000 7fb49e196640 1 --2- 172.21.15.107:0/2642673488 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb4800a2b90 0x7fb4800a5050 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.268+0000 7fb49e196640 1 --2- 172.21.15.107:0/2642673488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb498109740 0x7fb4981a1300 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.268+0000 7fb49e196640 1 --2- 172.21.15.107:0/2642673488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4980757f0 0x7fb49819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.268+0000 7fb49e196640 1 --2- 172.21.15.107:0/2642673488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb498074210 0x7fb498199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.268+0000 7fb49e196640 1 -- 172.21.15.107:0/2642673488 >> 172.21.15.107:0/2642673488 conn(0x7fb4980fcde0 msgr2=0x7fb4981056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:13.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.268+0000 7fb49e196640 1 -- 172.21.15.107:0/2642673488 shutdown_connections 2024-05-09T08:31:13.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.268+0000 7fb49e196640 1 -- 172.21.15.107:0/2642673488 wait complete. 2024-05-09T08:31:13.277 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:13 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/319624301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:13.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:31:13.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:13 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/319624301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:13.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:13 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/319624301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:13.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.383+0000 7f9cc5293640 1 -- 172.21.15.107:0/1985420105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc0100780 msgr2=0x7f9cc0100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.383+0000 7f9cc5293640 1 --2- 172.21.15.107:0/1985420105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc0100780 0x7f9cc0100ba0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0009900 tx=0x7f9cb002a410 comp rx=0 tx=0).stop 2024-05-09T08:31:13.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.383+0000 7f9cc5293640 1 -- 172.21.15.107:0/1985420105 shutdown_connections 2024-05-09T08:31:13.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.383+0000 7f9cc5293640 1 --2- 172.21.15.107:0/1985420105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cc0102330 0x7f9cc0108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.383+0000 7f9cc5293640 1 --2- 172.21.15.107:0/1985420105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc0101970 0x7f9cc0101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.383+0000 7f9cc5293640 1 --2- 172.21.15.107:0/1985420105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc0100780 0x7f9cc0100ba0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.383+0000 7f9cc5293640 1 -- 172.21.15.107:0/1985420105 >> 172.21.15.107:0/1985420105 conn(0x7f9cc00fbcf0 msgr2=0x7f9cc00fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:13.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.383+0000 7f9cc5293640 1 -- 172.21.15.107:0/1985420105 shutdown_connections 2024-05-09T08:31:13.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.383+0000 7f9cc5293640 1 -- 172.21.15.107:0/1985420105 wait complete. 2024-05-09T08:31:13.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.383+0000 7f9cc5293640 1 Processor -- start 2024-05-09T08:31:13.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cc5293640 1 -- start start 2024-05-09T08:31:13.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cc5293640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc0100780 0x7f9cc0197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:13.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cc5293640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc0101970 0x7f9cc0198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:13.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cc5293640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cc0102330 0x7f9cc019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:13.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cc5293640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc010d3d0 con 0x7f9cc0101970 2024-05-09T08:31:13.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cc5293640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc010d250 con 0x7f9cc0102330 2024-05-09T08:31:13.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cc5293640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc010d550 con 0x7f9cc0100780 2024-05-09T08:31:13.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cb67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc0101970 0x7f9cc0198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:13.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cb67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc0101970 0x7f9cc0198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37544/0 (socket says 172.21.15.107:37544) 2024-05-09T08:31:13.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cb67fc640 1 -- 172.21.15.107:0/3430368419 learned_addr learned my addr 172.21.15.107:0/3430368419 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:13.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cbf7fe640 1 --2- 172.21.15.107:0/3430368419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cc0102330 0x7f9cc019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:13.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cbeffd640 1 --2- 172.21.15.107:0/3430368419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc0100780 0x7f9cc0197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:13.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cb67fc640 1 -- 172.21.15.107:0/3430368419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc0100780 msgr2=0x7f9cc0197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cb67fc640 1 --2- 172.21.15.107:0/3430368419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc0100780 0x7f9cc0197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cb67fc640 1 -- 172.21.15.107:0/3430368419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cc0102330 msgr2=0x7f9cc019f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cb67fc640 1 --2- 172.21.15.107:0/3430368419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cc0102330 0x7f9cc019f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cb67fc640 1 -- 172.21.15.107:0/3430368419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc019f750 con 0x7f9cc0101970 2024-05-09T08:31:13.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cbeffd640 1 --2- 172.21.15.107:0/3430368419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc0100780 0x7f9cc0197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:13.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cbf7fe640 1 --2- 172.21.15.107:0/3430368419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cc0102330 0x7f9cc019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:13.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.384+0000 7f9cb67fc640 1 --2- 172.21.15.107:0/3430368419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc0101970 0x7f9cc0198070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9ca800e9f0 tx=0x7f9ca800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:13.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.385+0000 7f9cbcff9640 1 -- 172.21.15.107:0/3430368419 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca800cd80 con 0x7f9cc0101970 2024-05-09T08:31:13.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.385+0000 7f9cbcff9640 1 -- 172.21.15.107:0/3430368419 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ca80054d0 con 0x7f9cc0101970 2024-05-09T08:31:13.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.385+0000 7f9cbcff9640 1 -- 172.21.15.107:0/3430368419 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8010490 con 0x7f9cc0101970 2024-05-09T08:31:13.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.385+0000 7f9cc5293640 1 -- 172.21.15.107:0/3430368419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc019fa40 con 0x7f9cc0101970 2024-05-09T08:31:13.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.385+0000 7f9cc5293640 1 -- 172.21.15.107:0/3430368419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc019ff40 con 0x7f9cc0101970 2024-05-09T08:31:13.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.386+0000 7f9cc5293640 1 -- 172.21.15.107:0/3430368419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cc0068aa0 con 0x7f9cc0101970 2024-05-09T08:31:13.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.387+0000 7f9cbcff9640 1 -- 172.21.15.107:0/3430368419 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9ca8005020 con 0x7f9cc0101970 2024-05-09T08:31:13.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.388+0000 7f9cbcff9640 1 --2- 172.21.15.107:0/3430368419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9cac0a2b90 0x7f9cac0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:13.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.388+0000 7f9cbeffd640 1 --2- 172.21.15.107:0/3430368419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9cac0a2b90 0x7f9cac0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:13.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.388+0000 7f9cbcff9640 1 -- 172.21.15.107:0/3430368419 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9ca80cf760 con 0x7f9cc0101970 2024-05-09T08:31:13.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.388+0000 7f9cbeffd640 1 --2- 172.21.15.107:0/3430368419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9cac0a2b90 0x7f9cac0a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0006010 tx=0x7f9cb0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:13.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.392+0000 7f9cbcff9640 1 -- 172.21.15.107:0/3430368419 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9ca809c480 con 0x7f9cc0101970 2024-05-09T08:31:13.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.571+0000 7f9cc5293640 1 -- 172.21.15.107:0/3430368419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9cc00624f0 con 0x7f9cc0101970 2024-05-09T08:31:13.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.572+0000 7f9cbcff9640 1 -- 172.21.15.107:0/3430368419 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f9ca809bd80 con 0x7f9cc0101970 2024-05-09T08:31:13.571 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:31:13.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.574+0000 7f9cc5293640 1 -- 172.21.15.107:0/3430368419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9cac0a2b90 msgr2=0x7f9cac0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.574+0000 7f9cc5293640 1 --2- 172.21.15.107:0/3430368419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9cac0a2b90 0x7f9cac0a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0006010 tx=0x7f9cb0000bc0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.574+0000 7f9cc5293640 1 -- 172.21.15.107:0/3430368419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc0101970 msgr2=0x7f9cc0198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.574+0000 7f9cc5293640 1 --2- 172.21.15.107:0/3430368419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc0101970 0x7f9cc0198070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9ca800e9f0 tx=0x7f9ca800edf0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.574+0000 7f9cc5293640 1 -- 172.21.15.107:0/3430368419 shutdown_connections 2024-05-09T08:31:13.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.574+0000 7f9cc5293640 1 --2- 172.21.15.107:0/3430368419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9cac0a2b90 0x7f9cac0a5050 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.574+0000 7f9cc5293640 1 --2- 172.21.15.107:0/3430368419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cc0102330 0x7f9cc019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.574+0000 7f9cc5293640 1 --2- 172.21.15.107:0/3430368419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc0101970 0x7f9cc0198070 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.574+0000 7f9cc5293640 1 --2- 172.21.15.107:0/3430368419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc0100780 0x7f9cc0197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.574+0000 7f9cc5293640 1 -- 172.21.15.107:0/3430368419 >> 172.21.15.107:0/3430368419 conn(0x7f9cc00fbcf0 msgr2=0x7f9cc0104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:13.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.574+0000 7f9cc5293640 1 -- 172.21.15.107:0/3430368419 shutdown_connections 2024-05-09T08:31:13.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.574+0000 7f9cc5293640 1 -- 172.21.15.107:0/3430368419 wait complete. 2024-05-09T08:31:13.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:31:13.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.688+0000 7f705f838640 1 -- 172.21.15.107:0/669252236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7058102340 msgr2=0x7f7058108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.688+0000 7f705f838640 1 --2- 172.21.15.107:0/669252236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7058102340 0x7f7058108bd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7050009ca0 tx=0x7f705002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.688+0000 7f705f838640 1 -- 172.21.15.107:0/669252236 shutdown_connections 2024-05-09T08:31:13.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.688+0000 7f705f838640 1 --2- 172.21.15.107:0/669252236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7058102340 0x7f7058108bd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.688+0000 7f705f838640 1 --2- 172.21.15.107:0/669252236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7058101980 0x7f7058101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.688+0000 7f705f838640 1 --2- 172.21.15.107:0/669252236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7058100790 0x7f7058100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.688+0000 7f705f838640 1 -- 172.21.15.107:0/669252236 >> 172.21.15.107:0/669252236 conn(0x7f70580fbd20 msgr2=0x7f70580fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:13.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.688+0000 7f705f838640 1 -- 172.21.15.107:0/669252236 shutdown_connections 2024-05-09T08:31:13.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.688+0000 7f705f838640 1 -- 172.21.15.107:0/669252236 wait complete. 2024-05-09T08:31:13.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.688+0000 7f705f838640 1 Processor -- start 2024-05-09T08:31:13.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.688+0000 7f705f838640 1 -- start start 2024-05-09T08:31:13.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.688+0000 7f705f838640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7058100790 0x7f7058196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:13.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.688+0000 7f705f838640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7058101980 0x7f7058196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:13.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.688+0000 7f705f838640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7058102340 0x7f705819df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:13.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.688+0000 7f705f838640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f705810c2a0 con 0x7f7058102340 2024-05-09T08:31:13.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.688+0000 7f705f838640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f705810c120 con 0x7f7058101980 2024-05-09T08:31:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f705f838640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f705810c420 con 0x7f7058100790 2024-05-09T08:31:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f705ddd5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7058102340 0x7f705819df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f705ddd5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7058102340 0x7f705819df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37566/0 (socket says 172.21.15.107:37566) 2024-05-09T08:31:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f705ddd5640 1 -- 172.21.15.107:0/3938596476 learned_addr learned my addr 172.21.15.107:0/3938596476 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f705d5d4640 1 --2- 172.21.15.107:0/3938596476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7058100790 0x7f7058196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f705cdd3640 1 --2- 172.21.15.107:0/3938596476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7058101980 0x7f7058196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f705ddd5640 1 -- 172.21.15.107:0/3938596476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7058100790 msgr2=0x7f7058196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f705ddd5640 1 --2- 172.21.15.107:0/3938596476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7058100790 0x7f7058196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f705ddd5640 1 -- 172.21.15.107:0/3938596476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7058101980 msgr2=0x7f7058196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f705ddd5640 1 --2- 172.21.15.107:0/3938596476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7058101980 0x7f7058196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f705ddd5640 1 -- 172.21.15.107:0/3938596476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f705819e580 con 0x7f7058102340 2024-05-09T08:31:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f705ddd5640 1 --2- 172.21.15.107:0/3938596476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7058102340 0x7f705819df10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7050009dc0 tx=0x7f7050003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f704a7fc640 1 -- 172.21.15.107:0/3938596476 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7050004040 con 0x7f7058102340 2024-05-09T08:31:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f704a7fc640 1 -- 172.21.15.107:0/3938596476 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7050004690 con 0x7f7058102340 2024-05-09T08:31:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f704a7fc640 1 -- 172.21.15.107:0/3938596476 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705002f4c0 con 0x7f7058102340 2024-05-09T08:31:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f705f838640 1 -- 172.21.15.107:0/3938596476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f705819e810 con 0x7f7058102340 2024-05-09T08:31:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.689+0000 7f705f838640 1 -- 172.21.15.107:0/3938596476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f705819ed60 con 0x7f7058102340 2024-05-09T08:31:13.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.690+0000 7f705f838640 1 -- 172.21.15.107:0/3938596476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7058107b80 con 0x7f7058102340 2024-05-09T08:31:13.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.691+0000 7f704a7fc640 1 -- 172.21.15.107:0/3938596476 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f7050030650 con 0x7f7058102340 2024-05-09T08:31:13.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.692+0000 7f704a7fc640 1 --2- 172.21.15.107:0/3938596476 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f70440a2b90 0x7f70440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:13.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.692+0000 7f705d5d4640 1 --2- 172.21.15.107:0/3938596476 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f70440a2b90 0x7f70440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:13.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.692+0000 7f704a7fc640 1 -- 172.21.15.107:0/3938596476 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f70500f0970 con 0x7f7058102340 2024-05-09T08:31:13.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.692+0000 7f705d5d4640 1 --2- 172.21.15.107:0/3938596476 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f70440a2b90 0x7f70440a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f70581017e0 tx=0x7f705400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.696+0000 7f704a7fc640 1 -- 172.21.15.107:0/3938596476 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f70500bd690 con 0x7f7058102340 2024-05-09T08:31:13.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.871+0000 7f705f838640 1 -- 172.21.15.107:0/3938596476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7058062430 con 0x7f7058102340 2024-05-09T08:31:13.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.871+0000 7f704a7fc640 1 -- 172.21.15.107:0/3938596476 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f70500bcf90 con 0x7f7058102340 2024-05-09T08:31:13.871 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-05-09T08:31:13.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.873+0000 7f705f838640 1 -- 172.21.15.107:0/3938596476 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f70440a2b90 msgr2=0x7f70440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.873+0000 7f705f838640 1 --2- 172.21.15.107:0/3938596476 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f70440a2b90 0x7f70440a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f70581017e0 tx=0x7f705400a2d0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.873+0000 7f705f838640 1 -- 172.21.15.107:0/3938596476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7058102340 msgr2=0x7f705819df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.873+0000 7f705f838640 1 --2- 172.21.15.107:0/3938596476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7058102340 0x7f705819df10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7050009dc0 tx=0x7f7050003ba0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.873+0000 7f705f838640 1 -- 172.21.15.107:0/3938596476 shutdown_connections 2024-05-09T08:31:13.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.873+0000 7f705f838640 1 --2- 172.21.15.107:0/3938596476 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f70440a2b90 0x7f70440a5050 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.873+0000 7f705f838640 1 --2- 172.21.15.107:0/3938596476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7058102340 0x7f705819df10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.873+0000 7f705f838640 1 --2- 172.21.15.107:0/3938596476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7058101980 0x7f7058196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.873+0000 7f705f838640 1 --2- 172.21.15.107:0/3938596476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7058100790 0x7f7058196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.873+0000 7f705f838640 1 -- 172.21.15.107:0/3938596476 >> 172.21.15.107:0/3938596476 conn(0x7f70580fbd20 msgr2=0x7f7058104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:13.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.873+0000 7f705f838640 1 -- 172.21.15.107:0/3938596476 shutdown_connections 2024-05-09T08:31:13.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.873+0000 7f705f838640 1 -- 172.21.15.107:0/3938596476 wait complete. 2024-05-09T08:31:13.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:31:13.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.988+0000 7eff9fe12640 1 -- 172.21.15.107:0/1102119005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff980ffb40 msgr2=0x7eff980fff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.988+0000 7eff9fe12640 1 --2- 172.21.15.107:0/1102119005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff980ffb40 0x7eff980fff60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7eff94009930 tx=0x7eff9402a420 comp rx=0 tx=0).stop 2024-05-09T08:31:13.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.988+0000 7eff9fe12640 1 -- 172.21.15.107:0/1102119005 shutdown_connections 2024-05-09T08:31:13.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.988+0000 7eff9fe12640 1 --2- 172.21.15.107:0/1102119005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff981034d0 0x7eff98109d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.988+0000 7eff9fe12640 1 --2- 172.21.15.107:0/1102119005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff98102b10 0x7eff98102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.988+0000 7eff9fe12640 1 --2- 172.21.15.107:0/1102119005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff980ffb40 0x7eff980fff60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.988+0000 7eff9fe12640 1 -- 172.21.15.107:0/1102119005 >> 172.21.15.107:0/1102119005 conn(0x7eff980fb470 msgr2=0x7eff980fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:13.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.988+0000 7eff9fe12640 1 -- 172.21.15.107:0/1102119005 shutdown_connections 2024-05-09T08:31:13.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.988+0000 7eff9fe12640 1 -- 172.21.15.107:0/1102119005 wait complete. 2024-05-09T08:31:13.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.988+0000 7eff9fe12640 1 Processor -- start 2024-05-09T08:31:13.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9fe12640 1 -- start start 2024-05-09T08:31:13.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9fe12640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff980ffb40 0x7eff98197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:13.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9fe12640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff98102b10 0x7eff981980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:13.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9fe12640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff981034d0 0x7eff9819f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:13.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9fe12640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7eff9810d470 con 0x7eff980ffb40 2024-05-09T08:31:13.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9fe12640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7eff9810d2f0 con 0x7eff98102b10 2024-05-09T08:31:13.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9fe12640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7eff9810d5f0 con 0x7eff981034d0 2024-05-09T08:31:13.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9dbae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff980ffb40 0x7eff98197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:13.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9dbae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff980ffb40 0x7eff98197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37596/0 (socket says 172.21.15.107:37596) 2024-05-09T08:31:13.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9dbae640 1 -- 172.21.15.107:0/2297477949 learned_addr learned my addr 172.21.15.107:0/2297477949 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:13.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9d3ad640 1 --2- 172.21.15.107:0/2297477949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff98102b10 0x7eff981980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:13.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9e3af640 1 --2- 172.21.15.107:0/2297477949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff981034d0 0x7eff9819f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:13.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9dbae640 1 -- 172.21.15.107:0/2297477949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff981034d0 msgr2=0x7eff9819f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9dbae640 1 --2- 172.21.15.107:0/2297477949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff981034d0 0x7eff9819f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9dbae640 1 -- 172.21.15.107:0/2297477949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff98102b10 msgr2=0x7eff981980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:13.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9dbae640 1 --2- 172.21.15.107:0/2297477949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff98102b10 0x7eff981980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:13.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9dbae640 1 -- 172.21.15.107:0/2297477949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9819f790 con 0x7eff980ffb40 2024-05-09T08:31:13.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9e3af640 1 --2- 172.21.15.107:0/2297477949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff981034d0 0x7eff9819f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:13.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff9dbae640 1 --2- 172.21.15.107:0/2297477949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff980ffb40 0x7eff98197b70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7eff940314e0 tx=0x7eff94003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:13.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff8effd640 1 -- 172.21.15.107:0/2297477949 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff9403f070 con 0x7eff980ffb40 2024-05-09T08:31:13.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.989+0000 7eff8effd640 1 -- 172.21.15.107:0/2297477949 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff94031e40 con 0x7eff980ffb40 2024-05-09T08:31:13.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.990+0000 7eff9fe12640 1 -- 172.21.15.107:0/2297477949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff9819fa20 con 0x7eff980ffb40 2024-05-09T08:31:13.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.990+0000 7eff8effd640 1 -- 172.21.15.107:0/2297477949 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff94035030 con 0x7eff980ffb40 2024-05-09T08:31:13.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.990+0000 7eff9fe12640 1 -- 172.21.15.107:0/2297477949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff9819ff90 con 0x7eff980ffb40 2024-05-09T08:31:13.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.991+0000 7eff9fe12640 1 -- 172.21.15.107:0/2297477949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff98108d10 con 0x7eff980ffb40 2024-05-09T08:31:13.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.992+0000 7eff8effd640 1 -- 172.21.15.107:0/2297477949 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7eff94042070 con 0x7eff980ffb40 2024-05-09T08:31:13.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.993+0000 7eff8effd640 1 --2- 172.21.15.107:0/2297477949 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7eff840a2b90 0x7eff840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:13.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.993+0000 7eff9d3ad640 1 --2- 172.21.15.107:0/2297477949 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7eff840a2b90 0x7eff840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:13.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.993+0000 7eff8effd640 1 -- 172.21.15.107:0/2297477949 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7eff940efb40 con 0x7eff980ffb40 2024-05-09T08:31:13.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.993+0000 7eff9d3ad640 1 --2- 172.21.15.107:0/2297477949 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7eff840a2b90 0x7eff840a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7eff98199080 tx=0x7eff8800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:13.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:13.997+0000 7eff8effd640 1 -- 172.21.15.107:0/2297477949 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7eff940bc860 con 0x7eff980ffb40 2024-05-09T08:31:14.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.175+0000 7eff9fe12640 1 -- 172.21.15.107:0/2297477949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7eff98062460 con 0x7eff980ffb40 2024-05-09T08:31:14.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.176+0000 7eff8effd640 1 -- 172.21.15.107:0/2297477949 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7eff940bc160 con 0x7eff980ffb40 2024-05-09T08:31:14.175 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:31:14.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.178+0000 7eff9fe12640 1 -- 172.21.15.107:0/2297477949 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7eff840a2b90 msgr2=0x7eff840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:14.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.178+0000 7eff9fe12640 1 --2- 172.21.15.107:0/2297477949 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7eff840a2b90 0x7eff840a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7eff98199080 tx=0x7eff8800b390 comp rx=0 tx=0).stop 2024-05-09T08:31:14.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.178+0000 7eff9fe12640 1 -- 172.21.15.107:0/2297477949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff980ffb40 msgr2=0x7eff98197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:14.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.178+0000 7eff9fe12640 1 --2- 172.21.15.107:0/2297477949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff980ffb40 0x7eff98197b70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7eff940314e0 tx=0x7eff94003c80 comp rx=0 tx=0).stop 2024-05-09T08:31:14.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.178+0000 7eff9fe12640 1 -- 172.21.15.107:0/2297477949 shutdown_connections 2024-05-09T08:31:14.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.178+0000 7eff9fe12640 1 --2- 172.21.15.107:0/2297477949 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7eff840a2b90 0x7eff840a5050 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.178+0000 7eff9fe12640 1 --2- 172.21.15.107:0/2297477949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff981034d0 0x7eff9819f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.178+0000 7eff9fe12640 1 --2- 172.21.15.107:0/2297477949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff98102b10 0x7eff981980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.178+0000 7eff9fe12640 1 --2- 172.21.15.107:0/2297477949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff980ffb40 0x7eff98197b70 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.178+0000 7eff9fe12640 1 -- 172.21.15.107:0/2297477949 >> 172.21.15.107:0/2297477949 conn(0x7eff980fb470 msgr2=0x7eff98105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:14.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.178+0000 7eff9fe12640 1 -- 172.21.15.107:0/2297477949 shutdown_connections 2024-05-09T08:31:14.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.178+0000 7eff9fe12640 1 -- 172.21.15.107:0/2297477949 wait complete. 2024-05-09T08:31:14.187 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:14 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2642673488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:31:14.187 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:14 smithi107 ceph-mon[30122]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 237 B/s wr, 0 op/s 2024-05-09T08:31:14.187 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:14 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3430368419' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:31:14.187 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:14 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3938596476' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:31:14.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T08:31:14.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.290+0000 7fefcb69a640 1 -- 172.21.15.107:0/3423922076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefc4101960 msgr2=0x7fefc4101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:14.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.290+0000 7fefcb69a640 1 --2- 172.21.15.107:0/3423922076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefc4101960 0x7fefc4101de0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009900 tx=0x7fefb402a410 comp rx=0 tx=0).stop 2024-05-09T08:31:14.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.290+0000 7fefcb69a640 1 -- 172.21.15.107:0/3423922076 shutdown_connections 2024-05-09T08:31:14.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.290+0000 7fefcb69a640 1 --2- 172.21.15.107:0/3423922076 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefc4102320 0x7fefc4108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.290+0000 7fefcb69a640 1 --2- 172.21.15.107:0/3423922076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefc4101960 0x7fefc4101de0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.290+0000 7fefcb69a640 1 --2- 172.21.15.107:0/3423922076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefc4100770 0x7fefc4100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.290+0000 7fefcb69a640 1 -- 172.21.15.107:0/3423922076 >> 172.21.15.107:0/3423922076 conn(0x7fefc40fbce0 msgr2=0x7fefc40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:14.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.291+0000 7fefcb69a640 1 -- 172.21.15.107:0/3423922076 shutdown_connections 2024-05-09T08:31:14.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.291+0000 7fefcb69a640 1 -- 172.21.15.107:0/3423922076 wait complete. 2024-05-09T08:31:14.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.291+0000 7fefcb69a640 1 Processor -- start 2024-05-09T08:31:14.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.291+0000 7fefcb69a640 1 -- start start 2024-05-09T08:31:14.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.291+0000 7fefcb69a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefc4100770 0x7fefc41969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:14.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.291+0000 7fefcb69a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefc4101960 0x7fefc4196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:14.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.291+0000 7fefcb69a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefc4102320 0x7fefc419df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:14.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.291+0000 7fefcb69a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fefc410c300 con 0x7fefc4100770 2024-05-09T08:31:14.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.291+0000 7fefcb69a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fefc410c180 con 0x7fefc4102320 2024-05-09T08:31:14.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.291+0000 7fefcb69a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fefc410c480 con 0x7fefc4101960 2024-05-09T08:31:14.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.291+0000 7fefc9436640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefc4100770 0x7fefc41969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:14.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.291+0000 7fefc9436640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefc4100770 0x7fefc41969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37604/0 (socket says 172.21.15.107:37604) 2024-05-09T08:31:14.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.291+0000 7fefc9436640 1 -- 172.21.15.107:0/1873878666 learned_addr learned my addr 172.21.15.107:0/1873878666 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:14.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.291+0000 7fefc8c35640 1 --2- 172.21.15.107:0/1873878666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefc4101960 0x7fefc4196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:14.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.292+0000 7fefc9436640 1 -- 172.21.15.107:0/1873878666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefc4101960 msgr2=0x7fefc4196f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:14.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.292+0000 7fefc9436640 1 --2- 172.21.15.107:0/1873878666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefc4101960 0x7fefc4196f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.292+0000 7fefc9436640 1 -- 172.21.15.107:0/1873878666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefc4102320 msgr2=0x7fefc419df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:14.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.292+0000 7fefc9436640 1 --2- 172.21.15.107:0/1873878666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefc4102320 0x7fefc419df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.292+0000 7fefc9436640 1 -- 172.21.15.107:0/1873878666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc419e5f0 con 0x7fefc4100770 2024-05-09T08:31:14.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.292+0000 7fefc9436640 1 --2- 172.21.15.107:0/1873878666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefc4100770 0x7fefc41969d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fefc0007b50 tx=0x7fefc000e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:14.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.292+0000 7fefba7fc640 1 -- 172.21.15.107:0/1873878666 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc0007c40 con 0x7fefc4100770 2024-05-09T08:31:14.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.292+0000 7fefba7fc640 1 -- 172.21.15.107:0/1873878666 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fefc0007de0 con 0x7fefc4100770 2024-05-09T08:31:14.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.292+0000 7fefcb69a640 1 -- 172.21.15.107:0/1873878666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefc4068a20 con 0x7fefc4100770 2024-05-09T08:31:14.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.292+0000 7fefba7fc640 1 -- 172.21.15.107:0/1873878666 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc000acd0 con 0x7fefc4100770 2024-05-09T08:31:14.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.292+0000 7fefcb69a640 1 -- 172.21.15.107:0/1873878666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefc4068ff0 con 0x7fefc4100770 2024-05-09T08:31:14.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.293+0000 7fefcb69a640 1 -- 172.21.15.107:0/1873878666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefc4107580 con 0x7fefc4100770 2024-05-09T08:31:14.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.294+0000 7fefba7fc640 1 -- 172.21.15.107:0/1873878666 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fefc000eb30 con 0x7fefc4100770 2024-05-09T08:31:14.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.295+0000 7fefba7fc640 1 --2- 172.21.15.107:0/1873878666 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fefb00a2b90 0x7fefb00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:14.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.295+0000 7fefc8c35640 1 --2- 172.21.15.107:0/1873878666 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fefb00a2b90 0x7fefb00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:14.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.295+0000 7fefba7fc640 1 -- 172.21.15.107:0/1873878666 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fefc00d0a20 con 0x7fefc4100770 2024-05-09T08:31:14.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.295+0000 7fefc8c35640 1 --2- 172.21.15.107:0/1873878666 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fefb00a2b90 0x7fefb00a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fefc4197ee0 tx=0x7fefb40314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:14.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.299+0000 7fefba7fc640 1 -- 172.21.15.107:0/1873878666 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fefc0023020 con 0x7fefc4100770 2024-05-09T08:31:14.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:14 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2642673488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:31:14.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:14 smithi157 ceph-mon[33614]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 237 B/s wr, 0 op/s 2024-05-09T08:31:14.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:14 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3430368419' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:31:14.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:14 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3938596476' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:31:14.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:14 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2642673488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:31:14.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:14 smithi110 ceph-mon[33622]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 237 B/s wr, 0 op/s 2024-05-09T08:31:14.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:14 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3430368419' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:31:14.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:14 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3938596476' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:31:14.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.487+0000 7fefcb69a640 1 -- 172.21.15.107:0/1873878666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fefc419e880 con 0x7fefc4100770 2024-05-09T08:31:14.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.488+0000 7fefba7fc640 1 -- 172.21.15.107:0/1873878666 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+601 (secure 0 0 0) 0x7fefc000ee20 con 0x7fefc4100770 2024-05-09T08:31:14.487 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/1064417772 2024-05-10T08:23:55.372959+0000 2024-05-09T08:31:14.487 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/3922375261 2024-05-10T08:23:55.372959+0000 2024-05-09T08:31:14.487 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/3922375261 2024-05-10T08:23:55.372959+0000 2024-05-09T08:31:14.487 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/3934964045 2024-05-10T08:23:55.372959+0000 2024-05-09T08:31:14.487 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/1620469781 2024-05-10T08:23:29.066232+0000 2024-05-09T08:31:14.488 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4101303556 2024-05-10T08:23:29.066232+0000 2024-05-09T08:31:14.488 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/2360407889 2024-05-10T08:23:29.066232+0000 2024-05-09T08:31:14.488 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/164842293 2024-05-10T08:23:29.066232+0000 2024-05-09T08:31:14.488 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/2360407889 2024-05-10T08:23:29.066232+0000 2024-05-09T08:31:14.488 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4237993479 2024-05-10T08:23:55.372959+0000 2024-05-09T08:31:14.488 INFO:teuthology.orchestra.run.smithi107.stderr:listed 10 entries 2024-05-09T08:31:14.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.490+0000 7fefcb69a640 1 -- 172.21.15.107:0/1873878666 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fefb00a2b90 msgr2=0x7fefb00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:14.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.490+0000 7fefcb69a640 1 --2- 172.21.15.107:0/1873878666 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fefb00a2b90 0x7fefb00a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fefc4197ee0 tx=0x7fefb40314a0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.490+0000 7fefcb69a640 1 -- 172.21.15.107:0/1873878666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefc4100770 msgr2=0x7fefc41969d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:14.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.490+0000 7fefcb69a640 1 --2- 172.21.15.107:0/1873878666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefc4100770 0x7fefc41969d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fefc0007b50 tx=0x7fefc000e420 comp rx=0 tx=0).stop 2024-05-09T08:31:14.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.490+0000 7fefcb69a640 1 -- 172.21.15.107:0/1873878666 shutdown_connections 2024-05-09T08:31:14.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.490+0000 7fefcb69a640 1 --2- 172.21.15.107:0/1873878666 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fefb00a2b90 0x7fefb00a5050 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.490+0000 7fefcb69a640 1 --2- 172.21.15.107:0/1873878666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefc4102320 0x7fefc419df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.490+0000 7fefcb69a640 1 --2- 172.21.15.107:0/1873878666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefc4101960 0x7fefc4196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.490+0000 7fefcb69a640 1 --2- 172.21.15.107:0/1873878666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefc4100770 0x7fefc41969d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.490+0000 7fefcb69a640 1 -- 172.21.15.107:0/1873878666 >> 172.21.15.107:0/1873878666 conn(0x7fefc40fbce0 msgr2=0x7fefc4104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:14.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.490+0000 7fefcb69a640 1 -- 172.21.15.107:0/1873878666 shutdown_connections 2024-05-09T08:31:14.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.490+0000 7fefcb69a640 1 -- 172.21.15.107:0/1873878666 wait complete. 2024-05-09T08:31:14.500 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:31:14.500 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T08:31:14.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:31:14.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.659+0000 7f96df31b640 1 -- 172.21.15.107:0/76985161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96d8102a70 msgr2=0x7f96d8102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:14.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.659+0000 7f96df31b640 1 --2- 172.21.15.107:0/76985161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96d8102a70 0x7f96d8102ef0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f96d4009900 tx=0x7f96d402a410 comp rx=0 tx=0).stop 2024-05-09T08:31:14.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.659+0000 7f96df31b640 1 -- 172.21.15.107:0/76985161 shutdown_connections 2024-05-09T08:31:14.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.659+0000 7f96df31b640 1 --2- 172.21.15.107:0/76985161 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96d8103430 0x7f96d8109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.659+0000 7f96df31b640 1 --2- 172.21.15.107:0/76985161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96d8102a70 0x7f96d8102ef0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.659+0000 7f96df31b640 1 --2- 172.21.15.107:0/76985161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96d8101880 0x7f96d8101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.659+0000 7f96df31b640 1 -- 172.21.15.107:0/76985161 >> 172.21.15.107:0/76985161 conn(0x7f96d80fce10 msgr2=0x7f96d80ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:14.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.659+0000 7f96df31b640 1 -- 172.21.15.107:0/76985161 shutdown_connections 2024-05-09T08:31:14.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.659+0000 7f96df31b640 1 -- 172.21.15.107:0/76985161 wait complete. 2024-05-09T08:31:14.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.659+0000 7f96df31b640 1 Processor -- start 2024-05-09T08:31:14.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.659+0000 7f96df31b640 1 -- start start 2024-05-09T08:31:14.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.659+0000 7f96df31b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96d8101880 0x7f96d8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:14.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96df31b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96d8102a70 0x7f96d8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:14.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96df31b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96d8103430 0x7f96d819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:14.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96df31b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f96d810d390 con 0x7f96d8102a70 2024-05-09T08:31:14.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96df31b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f96d810d210 con 0x7f96d8101880 2024-05-09T08:31:14.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96df31b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f96d810d510 con 0x7f96d8103430 2024-05-09T08:31:14.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96dc8b6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96d8102a70 0x7f96d8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:14.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96dc8b6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96d8102a70 0x7f96d8198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37614/0 (socket says 172.21.15.107:37614) 2024-05-09T08:31:14.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96dc8b6640 1 -- 172.21.15.107:0/4016719693 learned_addr learned my addr 172.21.15.107:0/4016719693 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:14.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96dd8b8640 1 --2- 172.21.15.107:0/4016719693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96d8103430 0x7f96d819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:14.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96dc8b6640 1 -- 172.21.15.107:0/4016719693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96d8103430 msgr2=0x7f96d819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:14.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96dc8b6640 1 --2- 172.21.15.107:0/4016719693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96d8103430 0x7f96d819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96dc8b6640 1 -- 172.21.15.107:0/4016719693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96d8101880 msgr2=0x7f96d8197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:14.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96dc8b6640 1 --2- 172.21.15.107:0/4016719693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96d8101880 0x7f96d8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96dc8b6640 1 -- 172.21.15.107:0/4016719693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d819f6e0 con 0x7f96d8102a70 2024-05-09T08:31:14.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96dd8b8640 1 --2- 172.21.15.107:0/4016719693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96d8103430 0x7f96d819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:14.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96dc8b6640 1 --2- 172.21.15.107:0/4016719693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96d8102a70 0x7f96d8198000 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f96d4004770 tx=0x7f96d40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:14.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96ca7fc640 1 -- 172.21.15.107:0/4016719693 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d403f030 con 0x7f96d8102a70 2024-05-09T08:31:14.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96ca7fc640 1 -- 172.21.15.107:0/4016719693 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96d4003cb0 con 0x7f96d8102a70 2024-05-09T08:31:14.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96ca7fc640 1 -- 172.21.15.107:0/4016719693 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d4035070 con 0x7f96d8102a70 2024-05-09T08:31:14.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96df31b640 1 -- 172.21.15.107:0/4016719693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96d819f970 con 0x7f96d8102a70 2024-05-09T08:31:14.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.660+0000 7f96df31b640 1 -- 172.21.15.107:0/4016719693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96d819feb0 con 0x7f96d8102a70 2024-05-09T08:31:14.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.661+0000 7f96df31b640 1 -- 172.21.15.107:0/4016719693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96d8108bc0 con 0x7f96d8102a70 2024-05-09T08:31:14.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.662+0000 7f96ca7fc640 1 -- 172.21.15.107:0/4016719693 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f96d4042030 con 0x7f96d8102a70 2024-05-09T08:31:14.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.663+0000 7f96ca7fc640 1 --2- 172.21.15.107:0/4016719693 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f96c40a2c10 0x7f96c40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:14.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.663+0000 7f96dd0b7640 1 --2- 172.21.15.107:0/4016719693 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f96c40a2c10 0x7f96c40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:14.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.663+0000 7f96ca7fc640 1 -- 172.21.15.107:0/4016719693 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f96d40efc00 con 0x7f96d8102a70 2024-05-09T08:31:14.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.663+0000 7f96dd0b7640 1 --2- 172.21.15.107:0/4016719693 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f96c40a2c10 0x7f96c40a50d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f96d81028d0 tx=0x7f96cc005c10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:14.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.667+0000 7f96ca7fc640 1 -- 172.21.15.107:0/4016719693 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f96d40f6080 con 0x7f96d8102a70 2024-05-09T08:31:14.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.846+0000 7f96df31b640 1 -- 172.21.15.107:0/4016719693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f96d8062460 con 0x7f96d8102a70 2024-05-09T08:31:14.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.847+0000 7f96ca7fc640 1 -- 172.21.15.107:0/4016719693 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f96d40bc920 con 0x7f96d8102a70 2024-05-09T08:31:14.847 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:14.847 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:31:05.201799+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:31:14.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.849+0000 7f96df31b640 1 -- 172.21.15.107:0/4016719693 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f96c40a2c10 msgr2=0x7f96c40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:14.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.849+0000 7f96df31b640 1 --2- 172.21.15.107:0/4016719693 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f96c40a2c10 0x7f96c40a50d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f96d81028d0 tx=0x7f96cc005c10 comp rx=0 tx=0).stop 2024-05-09T08:31:14.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.849+0000 7f96df31b640 1 -- 172.21.15.107:0/4016719693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96d8102a70 msgr2=0x7f96d8198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:14.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.849+0000 7f96df31b640 1 --2- 172.21.15.107:0/4016719693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96d8102a70 0x7f96d8198000 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f96d4004770 tx=0x7f96d40042f0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.849+0000 7f96df31b640 1 -- 172.21.15.107:0/4016719693 shutdown_connections 2024-05-09T08:31:14.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.849+0000 7f96df31b640 1 --2- 172.21.15.107:0/4016719693 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f96c40a2c10 0x7f96c40a50d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.849+0000 7f96df31b640 1 --2- 172.21.15.107:0/4016719693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96d8103430 0x7f96d819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.849+0000 7f96df31b640 1 --2- 172.21.15.107:0/4016719693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96d8102a70 0x7f96d8198000 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.849+0000 7f96df31b640 1 --2- 172.21.15.107:0/4016719693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96d8101880 0x7f96d8197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.849+0000 7f96df31b640 1 -- 172.21.15.107:0/4016719693 >> 172.21.15.107:0/4016719693 conn(0x7f96d80fce10 msgr2=0x7f96d8105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:14.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.849+0000 7f96df31b640 1 -- 172.21.15.107:0/4016719693 shutdown_connections 2024-05-09T08:31:14.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.849+0000 7f96df31b640 1 -- 172.21.15.107:0/4016719693 wait complete. 2024-05-09T08:31:14.860 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:26:21.698338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:30:58.268633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:30:59.345579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:31:03.065991+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:31:14.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:31:14.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.965+0000 7f9d4639e640 1 -- 172.21.15.107:0/4250415531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d40103430 msgr2=0x7f9d40109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:14.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.965+0000 7f9d4639e640 1 --2- 172.21.15.107:0/4250415531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d40103430 0x7f9d40109cc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9d38009ca0 tx=0x7f9d3802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.965+0000 7f9d4639e640 1 -- 172.21.15.107:0/4250415531 shutdown_connections 2024-05-09T08:31:14.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.965+0000 7f9d4639e640 1 --2- 172.21.15.107:0/4250415531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d40103430 0x7f9d40109cc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.965+0000 7f9d4639e640 1 --2- 172.21.15.107:0/4250415531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d40102a70 0x7f9d40102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.965+0000 7f9d4639e640 1 --2- 172.21.15.107:0/4250415531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d40101880 0x7f9d40101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.965+0000 7f9d4639e640 1 -- 172.21.15.107:0/4250415531 >> 172.21.15.107:0/4250415531 conn(0x7f9d400fce10 msgr2=0x7f9d400ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.965+0000 7f9d4639e640 1 -- 172.21.15.107:0/4250415531 shutdown_connections 2024-05-09T08:31:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.965+0000 7f9d4639e640 1 -- 172.21.15.107:0/4250415531 wait complete. 2024-05-09T08:31:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.966+0000 7f9d4639e640 1 Processor -- start 2024-05-09T08:31:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.966+0000 7f9d4639e640 1 -- start start 2024-05-09T08:31:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.966+0000 7f9d4639e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d40101880 0x7f9d40197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.966+0000 7f9d4639e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d40102a70 0x7f9d40197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.966+0000 7f9d4639e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d40103430 0x7f9d4019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.966+0000 7f9d4639e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4010d3b0 con 0x7f9d40102a70 2024-05-09T08:31:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.966+0000 7f9d4639e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4010d230 con 0x7f9d40101880 2024-05-09T08:31:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.966+0000 7f9d4639e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4010d530 con 0x7f9d40103430 2024-05-09T08:31:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.966+0000 7f9d37fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d40102a70 0x7f9d40197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.966+0000 7f9d37fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d40102a70 0x7f9d40197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37622/0 (socket says 172.21.15.107:37622) 2024-05-09T08:31:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.966+0000 7f9d37fff640 1 -- 172.21.15.107:0/1592588815 learned_addr learned my addr 172.21.15.107:0/1592588815 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.966+0000 7f9d3ffff640 1 --2- 172.21.15.107:0/1592588815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d40101880 0x7f9d40197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.966+0000 7f9d37fff640 1 -- 172.21.15.107:0/1592588815 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d40103430 msgr2=0x7f9d4019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.966+0000 7f9d37fff640 1 --2- 172.21.15.107:0/1592588815 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d40103430 0x7f9d4019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.967+0000 7f9d37fff640 1 -- 172.21.15.107:0/1592588815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d40101880 msgr2=0x7f9d40197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.967+0000 7f9d37fff640 1 --2- 172.21.15.107:0/1592588815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d40101880 0x7f9d40197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.967+0000 7f9d37fff640 1 -- 172.21.15.107:0/1592588815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4019f6a0 con 0x7f9d40102a70 2024-05-09T08:31:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.967+0000 7f9d37fff640 1 --2- 172.21.15.107:0/1592588815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d40102a70 0x7f9d40197fc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800e9f0 tx=0x7f9d2800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.967+0000 7f9d3dffb640 1 -- 172.21.15.107:0/1592588815 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2800cd80 con 0x7f9d40102a70 2024-05-09T08:31:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.967+0000 7f9d4639e640 1 -- 172.21.15.107:0/1592588815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4019f990 con 0x7f9d40102a70 2024-05-09T08:31:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.967+0000 7f9d3dffb640 1 -- 172.21.15.107:0/1592588815 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d280054d0 con 0x7f9d40102a70 2024-05-09T08:31:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.967+0000 7f9d3dffb640 1 -- 172.21.15.107:0/1592588815 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d28010490 con 0x7f9d40102a70 2024-05-09T08:31:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.967+0000 7f9d4639e640 1 -- 172.21.15.107:0/1592588815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d4019fe90 con 0x7f9d40102a70 2024-05-09T08:31:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.968+0000 7f9d4639e640 1 -- 172.21.15.107:0/1592588815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d40108c20 con 0x7f9d40102a70 2024-05-09T08:31:14.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.969+0000 7f9d3dffb640 1 -- 172.21.15.107:0/1592588815 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9d28005020 con 0x7f9d40102a70 2024-05-09T08:31:14.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.970+0000 7f9d3dffb640 1 --2- 172.21.15.107:0/1592588815 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9d2c0a2b90 0x7f9d2c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:14.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.970+0000 7f9d3dffb640 1 -- 172.21.15.107:0/1592588815 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9d280cf6e0 con 0x7f9d40102a70 2024-05-09T08:31:14.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.970+0000 7f9d3ffff640 1 --2- 172.21.15.107:0/1592588815 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9d2c0a2b90 0x7f9d2c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:14.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.970+0000 7f9d3ffff640 1 --2- 172.21.15.107:0/1592588815 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9d2c0a2b90 0x7f9d2c0a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9d30005fd0 tx=0x7f9d30005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:14.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:14.974+0000 7f9d3dffb640 1 -- 172.21.15.107:0/1592588815 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9d2809c400 con 0x7f9d40102a70 2024-05-09T08:31:15.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.152+0000 7f9d4639e640 1 -- 172.21.15.107:0/1592588815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9d40062430 con 0x7f9d40102a70 2024-05-09T08:31:15.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.152+0000 7f9d3dffb640 1 -- 172.21.15.107:0/1592588815 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f9d2809bd00 con 0x7f9d40102a70 2024-05-09T08:31:15.152 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-05-09T08:31:15.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.154+0000 7f9d4639e640 1 -- 172.21.15.107:0/1592588815 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9d2c0a2b90 msgr2=0x7f9d2c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.154+0000 7f9d4639e640 1 --2- 172.21.15.107:0/1592588815 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9d2c0a2b90 0x7f9d2c0a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9d30005fd0 tx=0x7f9d30005c70 comp rx=0 tx=0).stop 2024-05-09T08:31:15.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.154+0000 7f9d4639e640 1 -- 172.21.15.107:0/1592588815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d40102a70 msgr2=0x7f9d40197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.154+0000 7f9d4639e640 1 --2- 172.21.15.107:0/1592588815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d40102a70 0x7f9d40197fc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800e9f0 tx=0x7f9d2800edf0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.154+0000 7f9d4639e640 1 -- 172.21.15.107:0/1592588815 shutdown_connections 2024-05-09T08:31:15.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.154+0000 7f9d4639e640 1 --2- 172.21.15.107:0/1592588815 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9d2c0a2b90 0x7f9d2c0a5050 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.154+0000 7f9d4639e640 1 --2- 172.21.15.107:0/1592588815 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d40103430 0x7f9d4019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.154+0000 7f9d4639e640 1 --2- 172.21.15.107:0/1592588815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d40102a70 0x7f9d40197fc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.154+0000 7f9d4639e640 1 --2- 172.21.15.107:0/1592588815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d40101880 0x7f9d40197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.154+0000 7f9d4639e640 1 -- 172.21.15.107:0/1592588815 >> 172.21.15.107:0/1592588815 conn(0x7f9d400fce10 msgr2=0x7f9d40105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:15.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.154+0000 7f9d4639e640 1 -- 172.21.15.107:0/1592588815 shutdown_connections 2024-05-09T08:31:15.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.154+0000 7f9d4639e640 1 -- 172.21.15.107:0/1592588815 wait complete. 2024-05-09T08:31:15.163 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:15 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2297477949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:31:15.163 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:15 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1873878666' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:31:15.164 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:15 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/4016719693' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:15.165 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:31:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.269+0000 7f845429a640 1 -- 172.21.15.107:0/2303330810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f844c101880 msgr2=0x7f844c101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.269+0000 7f845429a640 1 --2- 172.21.15.107:0/2303330810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f844c101880 0x7f844c101ca0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8448009900 tx=0x7f844802a410 comp rx=0 tx=0).stop 2024-05-09T08:31:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.269+0000 7f845429a640 1 -- 172.21.15.107:0/2303330810 shutdown_connections 2024-05-09T08:31:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.269+0000 7f845429a640 1 --2- 172.21.15.107:0/2303330810 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f844c103430 0x7f844c109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.269+0000 7f845429a640 1 --2- 172.21.15.107:0/2303330810 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f844c102a70 0x7f844c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.269+0000 7f845429a640 1 --2- 172.21.15.107:0/2303330810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f844c101880 0x7f844c101ca0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.269+0000 7f845429a640 1 -- 172.21.15.107:0/2303330810 >> 172.21.15.107:0/2303330810 conn(0x7f844c0fce10 msgr2=0x7f844c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.269+0000 7f845429a640 1 -- 172.21.15.107:0/2303330810 shutdown_connections 2024-05-09T08:31:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.269+0000 7f845429a640 1 -- 172.21.15.107:0/2303330810 wait complete. 2024-05-09T08:31:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.269+0000 7f845429a640 1 Processor -- start 2024-05-09T08:31:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.269+0000 7f845429a640 1 -- start start 2024-05-09T08:31:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f845429a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f844c101880 0x7f844c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f845429a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f844c102a70 0x7f844c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f845429a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f844c103430 0x7f844c19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f845429a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f844c10d3a0 con 0x7f844c103430 2024-05-09T08:31:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f845429a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f844c10d220 con 0x7f844c102a70 2024-05-09T08:31:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f845429a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f844c10d520 con 0x7f844c101880 2024-05-09T08:31:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f8452837640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f844c103430 0x7f844c19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f8452837640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f844c103430 0x7f844c19f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37652/0 (socket says 172.21.15.107:37652) 2024-05-09T08:31:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f8452837640 1 -- 172.21.15.107:0/3221522510 learned_addr learned my addr 172.21.15.107:0/3221522510 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f8451835640 1 --2- 172.21.15.107:0/3221522510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f844c102a70 0x7f844c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f8452036640 1 --2- 172.21.15.107:0/3221522510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f844c101880 0x7f844c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f8452837640 1 -- 172.21.15.107:0/3221522510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f844c101880 msgr2=0x7f844c197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f8452837640 1 --2- 172.21.15.107:0/3221522510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f844c101880 0x7f844c197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f8452837640 1 -- 172.21.15.107:0/3221522510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f844c102a70 msgr2=0x7f844c198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f8452837640 1 --2- 172.21.15.107:0/3221522510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f844c102a70 0x7f844c198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f8452837640 1 -- 172.21.15.107:0/3221522510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f844c19f6f0 con 0x7f844c103430 2024-05-09T08:31:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f8451835640 1 --2- 172.21.15.107:0/3221522510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f844c102a70 0x7f844c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f8452837640 1 --2- 172.21.15.107:0/3221522510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f844c103430 0x7f844c19f080 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f844400eaa0 tx=0x7f844400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f843f7fe640 1 -- 172.21.15.107:0/3221522510 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844400cdb0 con 0x7f844c103430 2024-05-09T08:31:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f843f7fe640 1 -- 172.21.15.107:0/3221522510 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8444005020 con 0x7f844c103430 2024-05-09T08:31:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f845429a640 1 -- 172.21.15.107:0/3221522510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f844c19f9e0 con 0x7f844c103430 2024-05-09T08:31:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f843f7fe640 1 -- 172.21.15.107:0/3221522510 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84440116d0 con 0x7f844c103430 2024-05-09T08:31:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.270+0000 7f845429a640 1 -- 172.21.15.107:0/3221522510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f844c19fee0 con 0x7f844c103430 2024-05-09T08:31:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.271+0000 7f845429a640 1 -- 172.21.15.107:0/3221522510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f844c108d10 con 0x7f844c103430 2024-05-09T08:31:15.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.273+0000 7f843f7fe640 1 -- 172.21.15.107:0/3221522510 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f84440051d0 con 0x7f844c103430 2024-05-09T08:31:15.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.273+0000 7f843f7fe640 1 --2- 172.21.15.107:0/3221522510 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f84380a2b90 0x7f84380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.273+0000 7f8452036640 1 --2- 172.21.15.107:0/3221522510 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f84380a2b90 0x7f84380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.273+0000 7f843f7fe640 1 -- 172.21.15.107:0/3221522510 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f84440cf530 con 0x7f844c103430 2024-05-09T08:31:15.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.274+0000 7f8452036640 1 --2- 172.21.15.107:0/3221522510 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f84380a2b90 0x7f84380a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f844802a930 tx=0x7f8448000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:15.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.278+0000 7f843f7fe640 1 -- 172.21.15.107:0/3221522510 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8444016030 con 0x7f844c103430 2024-05-09T08:31:15.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:15 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2297477949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:31:15.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:15 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1873878666' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:31:15.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:15 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/4016719693' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:15.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:15 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2297477949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:31:15.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:15 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1873878666' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:31:15.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:15 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/4016719693' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:15.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.451+0000 7f845429a640 1 -- 172.21.15.107:0/3221522510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f844c062430 con 0x7f844c103430 2024-05-09T08:31:15.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.451+0000 7f843f7fe640 1 -- 172.21.15.107:0/3221522510 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f844409c250 con 0x7f844c103430 2024-05-09T08:31:15.451 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:31:15.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.453+0000 7f845429a640 1 -- 172.21.15.107:0/3221522510 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f84380a2b90 msgr2=0x7f84380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.453+0000 7f845429a640 1 --2- 172.21.15.107:0/3221522510 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f84380a2b90 0x7f84380a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f844802a930 tx=0x7f8448000bc0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.453+0000 7f845429a640 1 -- 172.21.15.107:0/3221522510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f844c103430 msgr2=0x7f844c19f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.453+0000 7f845429a640 1 --2- 172.21.15.107:0/3221522510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f844c103430 0x7f844c19f080 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f844400eaa0 tx=0x7f844400ef60 comp rx=0 tx=0).stop 2024-05-09T08:31:15.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.453+0000 7f845429a640 1 -- 172.21.15.107:0/3221522510 shutdown_connections 2024-05-09T08:31:15.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.453+0000 7f845429a640 1 --2- 172.21.15.107:0/3221522510 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f84380a2b90 0x7f84380a5050 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.453+0000 7f845429a640 1 --2- 172.21.15.107:0/3221522510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f844c103430 0x7f844c19f080 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.453+0000 7f845429a640 1 --2- 172.21.15.107:0/3221522510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f844c102a70 0x7f844c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.453+0000 7f845429a640 1 --2- 172.21.15.107:0/3221522510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f844c101880 0x7f844c197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.453+0000 7f845429a640 1 -- 172.21.15.107:0/3221522510 >> 172.21.15.107:0/3221522510 conn(0x7f844c0fce10 msgr2=0x7f844c105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:15.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.453+0000 7f845429a640 1 -- 172.21.15.107:0/3221522510 shutdown_connections 2024-05-09T08:31:15.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.453+0000 7f845429a640 1 -- 172.21.15.107:0/3221522510 wait complete. 2024-05-09T08:31:15.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:31:15.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.570+0000 7f319cdf9640 1 -- 172.21.15.107:0/3495980173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3198103430 msgr2=0x7f3198109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.570+0000 7f319cdf9640 1 --2- 172.21.15.107:0/3495980173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3198103430 0x7f3198109cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3188009900 tx=0x7f318802a410 comp rx=0 tx=0).stop 2024-05-09T08:31:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.570+0000 7f319cdf9640 1 -- 172.21.15.107:0/3495980173 shutdown_connections 2024-05-09T08:31:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.570+0000 7f319cdf9640 1 --2- 172.21.15.107:0/3495980173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3198103430 0x7f3198109cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.570+0000 7f319cdf9640 1 --2- 172.21.15.107:0/3495980173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3198102a70 0x7f3198102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.570+0000 7f319cdf9640 1 --2- 172.21.15.107:0/3495980173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3198101880 0x7f3198101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.570+0000 7f319cdf9640 1 -- 172.21.15.107:0/3495980173 >> 172.21.15.107:0/3495980173 conn(0x7f31980fce10 msgr2=0x7f31980ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.570+0000 7f319cdf9640 1 -- 172.21.15.107:0/3495980173 shutdown_connections 2024-05-09T08:31:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.570+0000 7f319cdf9640 1 -- 172.21.15.107:0/3495980173 wait complete. 2024-05-09T08:31:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f319cdf9640 1 Processor -- start 2024-05-09T08:31:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f319cdf9640 1 -- start start 2024-05-09T08:31:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f319cdf9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3198101880 0x7f3198197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f319cdf9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3198102a70 0x7f3198198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f319cdf9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3198103430 0x7f319819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f319cdf9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f319810d410 con 0x7f3198102a70 2024-05-09T08:31:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f319cdf9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f319810d290 con 0x7f3198103430 2024-05-09T08:31:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f319cdf9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f319810d590 con 0x7f3198101880 2024-05-09T08:31:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f3195d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3198102a70 0x7f3198198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f3195d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3198102a70 0x7f3198198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37662/0 (socket says 172.21.15.107:37662) 2024-05-09T08:31:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f3195d9b640 1 -- 172.21.15.107:0/2335510377 learned_addr learned my addr 172.21.15.107:0/2335510377 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f319659c640 1 --2- 172.21.15.107:0/2335510377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3198101880 0x7f3198197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f3196d9d640 1 --2- 172.21.15.107:0/2335510377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3198103430 0x7f319819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f3195d9b640 1 -- 172.21.15.107:0/2335510377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3198101880 msgr2=0x7f3198197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f3195d9b640 1 --2- 172.21.15.107:0/2335510377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3198101880 0x7f3198197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f3195d9b640 1 -- 172.21.15.107:0/2335510377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3198103430 msgr2=0x7f319819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f3195d9b640 1 --2- 172.21.15.107:0/2335510377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3198103430 0x7f319819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f3195d9b640 1 -- 172.21.15.107:0/2335510377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319819f6e0 con 0x7f3198102a70 2024-05-09T08:31:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f319659c640 1 --2- 172.21.15.107:0/2335510377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3198101880 0x7f3198197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.571+0000 7f3196d9d640 1 --2- 172.21.15.107:0/2335510377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3198103430 0x7f319819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.572+0000 7f3195d9b640 1 --2- 172.21.15.107:0/2335510377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3198102a70 0x7f3198198000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f318c00ba50 tx=0x7f318c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.572+0000 7f31877fe640 1 -- 172.21.15.107:0/2335510377 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c019070 con 0x7f3198102a70 2024-05-09T08:31:15.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.572+0000 7f31877fe640 1 -- 172.21.15.107:0/2335510377 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f318c00cbe0 con 0x7f3198102a70 2024-05-09T08:31:15.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.572+0000 7f31877fe640 1 -- 172.21.15.107:0/2335510377 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c0075d0 con 0x7f3198102a70 2024-05-09T08:31:15.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.572+0000 7f319cdf9640 1 -- 172.21.15.107:0/2335510377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f319819f9d0 con 0x7f3198102a70 2024-05-09T08:31:15.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.572+0000 7f319cdf9640 1 -- 172.21.15.107:0/2335510377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f319819ff50 con 0x7f3198102a70 2024-05-09T08:31:15.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.573+0000 7f319cdf9640 1 -- 172.21.15.107:0/2335510377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3198108ba0 con 0x7f3198102a70 2024-05-09T08:31:15.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.574+0000 7f31877fe640 1 -- 172.21.15.107:0/2335510377 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f318c01c070 con 0x7f3198102a70 2024-05-09T08:31:15.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.575+0000 7f31877fe640 1 --2- 172.21.15.107:0/2335510377 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f31800a2b90 0x7f31800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.575+0000 7f319659c640 1 --2- 172.21.15.107:0/2335510377 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f31800a2b90 0x7f31800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.575+0000 7f31877fe640 1 -- 172.21.15.107:0/2335510377 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f318c0ce8e0 con 0x7f3198102a70 2024-05-09T08:31:15.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.575+0000 7f319659c640 1 --2- 172.21.15.107:0/2335510377 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f31800a2b90 0x7f31800a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f31981028d0 tx=0x7f319000a560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.579+0000 7f31877fe640 1 -- 172.21.15.107:0/2335510377 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f318c09b600 con 0x7f3198102a70 2024-05-09T08:31:15.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.756+0000 7f319cdf9640 1 -- 172.21.15.107:0/2335510377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f319802d3c0 con 0x7f3198102a70 2024-05-09T08:31:15.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.756+0000 7f31877fe640 1 -- 172.21.15.107:0/2335510377 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f318c09af00 con 0x7f3198102a70 2024-05-09T08:31:15.755 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-05-09T08:31:15.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.758+0000 7f319cdf9640 1 -- 172.21.15.107:0/2335510377 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f31800a2b90 msgr2=0x7f31800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.758+0000 7f319cdf9640 1 --2- 172.21.15.107:0/2335510377 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f31800a2b90 0x7f31800a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f31981028d0 tx=0x7f319000a560 comp rx=0 tx=0).stop 2024-05-09T08:31:15.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.758+0000 7f319cdf9640 1 -- 172.21.15.107:0/2335510377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3198102a70 msgr2=0x7f3198198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.758+0000 7f319cdf9640 1 --2- 172.21.15.107:0/2335510377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3198102a70 0x7f3198198000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f318c00ba50 tx=0x7f318c00bf10 comp rx=0 tx=0).stop 2024-05-09T08:31:15.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.758+0000 7f319cdf9640 1 -- 172.21.15.107:0/2335510377 shutdown_connections 2024-05-09T08:31:15.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.758+0000 7f319cdf9640 1 --2- 172.21.15.107:0/2335510377 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f31800a2b90 0x7f31800a5050 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.758+0000 7f319cdf9640 1 --2- 172.21.15.107:0/2335510377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3198103430 0x7f319819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.758+0000 7f319cdf9640 1 --2- 172.21.15.107:0/2335510377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3198102a70 0x7f3198198000 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.758+0000 7f319cdf9640 1 --2- 172.21.15.107:0/2335510377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3198101880 0x7f3198197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.758+0000 7f319cdf9640 1 -- 172.21.15.107:0/2335510377 >> 172.21.15.107:0/2335510377 conn(0x7f31980fce10 msgr2=0x7f3198105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:15.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.758+0000 7f319cdf9640 1 -- 172.21.15.107:0/2335510377 shutdown_connections 2024-05-09T08:31:15.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.758+0000 7f319cdf9640 1 -- 172.21.15.107:0/2335510377 wait complete. 2024-05-09T08:31:15.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:31:15.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.873+0000 7f1950b63640 1 -- 172.21.15.107:0/1593462486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f194c102340 msgr2=0x7f194c108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.873+0000 7f1950b63640 1 --2- 172.21.15.107:0/1593462486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f194c102340 0x7f194c108bd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1944009900 tx=0x7f194402a410 comp rx=0 tx=0).stop 2024-05-09T08:31:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.873+0000 7f1950b63640 1 -- 172.21.15.107:0/1593462486 shutdown_connections 2024-05-09T08:31:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.873+0000 7f1950b63640 1 --2- 172.21.15.107:0/1593462486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f194c102340 0x7f194c108bd0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.873+0000 7f1950b63640 1 --2- 172.21.15.107:0/1593462486 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f194c101980 0x7f194c101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.873+0000 7f1950b63640 1 --2- 172.21.15.107:0/1593462486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f194c100790 0x7f194c100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.873+0000 7f1950b63640 1 -- 172.21.15.107:0/1593462486 >> 172.21.15.107:0/1593462486 conn(0x7f194c0fbd00 msgr2=0x7f194c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.873+0000 7f1950b63640 1 -- 172.21.15.107:0/1593462486 shutdown_connections 2024-05-09T08:31:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.873+0000 7f1950b63640 1 -- 172.21.15.107:0/1593462486 wait complete. 2024-05-09T08:31:15.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f1950b63640 1 Processor -- start 2024-05-09T08:31:15.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f1950b63640 1 -- start start 2024-05-09T08:31:15.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f1950b63640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f194c100790 0x7f194c10b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f1950b63640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f194c101980 0x7f194c109750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f1950b63640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f194c102340 0x7f194c109c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f1950b63640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f194c10d440 con 0x7f194c102340 2024-05-09T08:31:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f1950b63640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f194c10d2c0 con 0x7f194c101980 2024-05-09T08:31:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f1950b63640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f194c10d5c0 con 0x7f194c100790 2024-05-09T08:31:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f194ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f194c102340 0x7f194c109c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f194ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f194c102340 0x7f194c109c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37670/0 (socket says 172.21.15.107:37670) 2024-05-09T08:31:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f194ad9d640 1 -- 172.21.15.107:0/2876083466 learned_addr learned my addr 172.21.15.107:0/2876083466 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f194a59c640 1 --2- 172.21.15.107:0/2876083466 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f194c100790 0x7f194c10b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f1949d9b640 1 --2- 172.21.15.107:0/2876083466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f194c101980 0x7f194c109750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f194ad9d640 1 -- 172.21.15.107:0/2876083466 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f194c100790 msgr2=0x7f194c10b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f194ad9d640 1 --2- 172.21.15.107:0/2876083466 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f194c100790 0x7f194c10b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f194ad9d640 1 -- 172.21.15.107:0/2876083466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f194c101980 msgr2=0x7f194c109750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f194ad9d640 1 --2- 172.21.15.107:0/2876083466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f194c101980 0x7f194c109750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f194ad9d640 1 -- 172.21.15.107:0/2876083466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194c10a520 con 0x7f194c102340 2024-05-09T08:31:15.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f1949d9b640 1 --2- 172.21.15.107:0/2876083466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f194c101980 0x7f194c109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:15.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.874+0000 7f194ad9d640 1 --2- 172.21.15.107:0/2876083466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f194c102340 0x7f194c109c90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1944005940 tx=0x7f19440043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:15.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.875+0000 7f193b7fe640 1 -- 172.21.15.107:0/2876083466 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194403f030 con 0x7f194c102340 2024-05-09T08:31:15.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.875+0000 7f193b7fe640 1 -- 172.21.15.107:0/2876083466 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1944003cb0 con 0x7f194c102340 2024-05-09T08:31:15.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.875+0000 7f1950b63640 1 -- 172.21.15.107:0/2876083466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f194c1a1b70 con 0x7f194c102340 2024-05-09T08:31:15.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.875+0000 7f193b7fe640 1 -- 172.21.15.107:0/2876083466 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1944035070 con 0x7f194c102340 2024-05-09T08:31:15.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.875+0000 7f1950b63640 1 -- 172.21.15.107:0/2876083466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f194c1a20c0 con 0x7f194c102340 2024-05-09T08:31:15.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.876+0000 7f1950b63640 1 -- 172.21.15.107:0/2876083466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f194c107ab0 con 0x7f194c102340 2024-05-09T08:31:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.877+0000 7f193b7fe640 1 -- 172.21.15.107:0/2876083466 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f1944042030 con 0x7f194c102340 2024-05-09T08:31:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.877+0000 7f193b7fe640 1 --2- 172.21.15.107:0/2876083466 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f19340a2b90 0x7f19340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.877+0000 7f193b7fe640 1 -- 172.21.15.107:0/2876083466 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f19440efa50 con 0x7f194c102340 2024-05-09T08:31:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.877+0000 7f194a59c640 1 --2- 172.21.15.107:0/2876083466 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f19340a2b90 0x7f19340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:15.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.878+0000 7f194a59c640 1 --2- 172.21.15.107:0/2876083466 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f19340a2b90 0x7f19340a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f194c1017e0 tx=0x7f1940008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:15.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:15.882+0000 7f193b7fe640 1 -- 172.21.15.107:0/2876083466 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f19440bc770 con 0x7f194c102340 2024-05-09T08:31:16.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.062+0000 7f1950b63640 1 -- 172.21.15.107:0/2876083466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f194c0624d0 con 0x7f194c102340 2024-05-09T08:31:16.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.062+0000 7f193b7fe640 1 -- 172.21.15.107:0/2876083466 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f19440bc070 con 0x7f194c102340 2024-05-09T08:31:16.062 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:31:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.065+0000 7f1950b63640 1 -- 172.21.15.107:0/2876083466 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f19340a2b90 msgr2=0x7f19340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.065+0000 7f1950b63640 1 --2- 172.21.15.107:0/2876083466 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f19340a2b90 0x7f19340a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f194c1017e0 tx=0x7f1940008040 comp rx=0 tx=0).stop 2024-05-09T08:31:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.065+0000 7f1950b63640 1 -- 172.21.15.107:0/2876083466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f194c102340 msgr2=0x7f194c109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:16.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.065+0000 7f1950b63640 1 --2- 172.21.15.107:0/2876083466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f194c102340 0x7f194c109c90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1944005940 tx=0x7f19440043c0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.065+0000 7f1950b63640 1 -- 172.21.15.107:0/2876083466 shutdown_connections 2024-05-09T08:31:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.065+0000 7f1950b63640 1 --2- 172.21.15.107:0/2876083466 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f19340a2b90 0x7f19340a5050 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.065+0000 7f1950b63640 1 --2- 172.21.15.107:0/2876083466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f194c102340 0x7f194c109c90 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.065+0000 7f1950b63640 1 --2- 172.21.15.107:0/2876083466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f194c101980 0x7f194c109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.065+0000 7f1950b63640 1 --2- 172.21.15.107:0/2876083466 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f194c100790 0x7f194c10b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.065+0000 7f1950b63640 1 -- 172.21.15.107:0/2876083466 >> 172.21.15.107:0/2876083466 conn(0x7f194c0fbd00 msgr2=0x7f194c104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.065+0000 7f1950b63640 1 -- 172.21.15.107:0/2876083466 shutdown_connections 2024-05-09T08:31:16.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.065+0000 7f1950b63640 1 -- 172.21.15.107:0/2876083466 wait complete. 2024-05-09T08:31:16.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T08:31:16.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.181+0000 7fde3c8c3640 1 -- 172.21.15.107:0/3204038269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde38102320 msgr2=0x7fde38108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:16.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.181+0000 7fde3c8c3640 1 --2- 172.21.15.107:0/3204038269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde38102320 0x7fde38108bb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fde30009ca0 tx=0x7fde3002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.182+0000 7fde3c8c3640 1 -- 172.21.15.107:0/3204038269 shutdown_connections 2024-05-09T08:31:16.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.182+0000 7fde3c8c3640 1 --2- 172.21.15.107:0/3204038269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde38102320 0x7fde38108bb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.182+0000 7fde3c8c3640 1 --2- 172.21.15.107:0/3204038269 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde38101960 0x7fde38101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.182+0000 7fde3c8c3640 1 --2- 172.21.15.107:0/3204038269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde38100770 0x7fde38100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.182+0000 7fde3c8c3640 1 -- 172.21.15.107:0/3204038269 >> 172.21.15.107:0/3204038269 conn(0x7fde380fbce0 msgr2=0x7fde380fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:16.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.182+0000 7fde3c8c3640 1 -- 172.21.15.107:0/3204038269 shutdown_connections 2024-05-09T08:31:16.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.182+0000 7fde3c8c3640 1 -- 172.21.15.107:0/3204038269 wait complete. 2024-05-09T08:31:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.182+0000 7fde3c8c3640 1 Processor -- start 2024-05-09T08:31:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.182+0000 7fde3c8c3640 1 -- start start 2024-05-09T08:31:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.182+0000 7fde3c8c3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde38100770 0x7fde38197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.182+0000 7fde3c8c3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde38101960 0x7fde38197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde3c8c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde38102320 0x7fde3819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde3c8c3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fde3810d3c0 con 0x7fde38102320 2024-05-09T08:31:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde3c8c3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fde3810d240 con 0x7fde38100770 2024-05-09T08:31:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde3c8c3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fde3810d540 con 0x7fde38101960 2024-05-09T08:31:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde36d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde38102320 0x7fde3819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde36d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde38102320 0x7fde3819efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37686/0 (socket says 172.21.15.107:37686) 2024-05-09T08:31:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde36d9d640 1 -- 172.21.15.107:0/561143917 learned_addr learned my addr 172.21.15.107:0/561143917 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde3659c640 1 --2- 172.21.15.107:0/561143917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde38100770 0x7fde38197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde35d9b640 1 --2- 172.21.15.107:0/561143917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde38101960 0x7fde38197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde36d9d640 1 -- 172.21.15.107:0/561143917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde38101960 msgr2=0x7fde38197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde36d9d640 1 --2- 172.21.15.107:0/561143917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde38101960 0x7fde38197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde36d9d640 1 -- 172.21.15.107:0/561143917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde38100770 msgr2=0x7fde38197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde36d9d640 1 --2- 172.21.15.107:0/561143917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde38100770 0x7fde38197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde36d9d640 1 -- 172.21.15.107:0/561143917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3819f630 con 0x7fde38102320 2024-05-09T08:31:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde36d9d640 1 --2- 172.21.15.107:0/561143917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde38102320 0x7fde3819efc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fde30009dc0 tx=0x7fde30003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde2f7fe640 1 -- 172.21.15.107:0/561143917 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde30004040 con 0x7fde38102320 2024-05-09T08:31:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde3c8c3640 1 -- 172.21.15.107:0/561143917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3819f8c0 con 0x7fde38102320 2024-05-09T08:31:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde2f7fe640 1 -- 172.21.15.107:0/561143917 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde30004690 con 0x7fde38102320 2024-05-09T08:31:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde2f7fe640 1 -- 172.21.15.107:0/561143917 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3002f4c0 con 0x7fde38102320 2024-05-09T08:31:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.183+0000 7fde3c8c3640 1 -- 172.21.15.107:0/561143917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde3819fe10 con 0x7fde38102320 2024-05-09T08:31:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.184+0000 7fde3c8c3640 1 -- 172.21.15.107:0/561143917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde38107a90 con 0x7fde38102320 2024-05-09T08:31:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.186+0000 7fde2f7fe640 1 -- 172.21.15.107:0/561143917 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fde30030650 con 0x7fde38102320 2024-05-09T08:31:16.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.187+0000 7fde2f7fe640 1 --2- 172.21.15.107:0/561143917 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fde240a2b90 0x7fde240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:16.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.187+0000 7fde3659c640 1 --2- 172.21.15.107:0/561143917 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fde240a2b90 0x7fde240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:16.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.187+0000 7fde2f7fe640 1 -- 172.21.15.107:0/561143917 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fde300f0b90 con 0x7fde38102320 2024-05-09T08:31:16.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.187+0000 7fde3659c640 1 --2- 172.21.15.107:0/561143917 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fde240a2b90 0x7fde240a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fde381017c0 tx=0x7fde28005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.190+0000 7fde2f7fe640 1 -- 172.21.15.107:0/561143917 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fde300bd8b0 con 0x7fde38102320 2024-05-09T08:31:16.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:16 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1592588815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:31:16.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:16 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3221522510' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:31:16.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:16 smithi107 ceph-mon[30122]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-05-09T08:31:16.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:16 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2335510377' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:31:16.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:16 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1592588815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:31:16.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:16 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3221522510' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:31:16.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:16 smithi157 ceph-mon[33614]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-05-09T08:31:16.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:16 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2335510377' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:31:16.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:16 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1592588815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:31:16.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:16 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3221522510' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:31:16.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:16 smithi110 ceph-mon[33622]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-05-09T08:31:16.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:16 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2335510377' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:31:16.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.387+0000 7fde3c8c3640 1 -- 172.21.15.107:0/561143917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fde3802d2e0 con 0x7fde38102320 2024-05-09T08:31:16.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.387+0000 7fde2f7fe640 1 -- 172.21.15.107:0/561143917 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+601 (secure 0 0 0) 0x7fde300bd1b0 con 0x7fde38102320 2024-05-09T08:31:16.386 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/1064417772 2024-05-10T08:23:55.372959+0000 2024-05-09T08:31:16.387 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/3922375261 2024-05-10T08:23:55.372959+0000 2024-05-09T08:31:16.387 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/3922375261 2024-05-10T08:23:55.372959+0000 2024-05-09T08:31:16.387 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/3934964045 2024-05-10T08:23:55.372959+0000 2024-05-09T08:31:16.387 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/1620469781 2024-05-10T08:23:29.066232+0000 2024-05-09T08:31:16.387 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4101303556 2024-05-10T08:23:29.066232+0000 2024-05-09T08:31:16.387 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/2360407889 2024-05-10T08:23:29.066232+0000 2024-05-09T08:31:16.387 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/164842293 2024-05-10T08:23:29.066232+0000 2024-05-09T08:31:16.387 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/2360407889 2024-05-10T08:23:29.066232+0000 2024-05-09T08:31:16.387 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4237993479 2024-05-10T08:23:55.372959+0000 2024-05-09T08:31:16.387 INFO:teuthology.orchestra.run.smithi107.stderr:listed 10 entries 2024-05-09T08:31:16.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.389+0000 7fde3c8c3640 1 -- 172.21.15.107:0/561143917 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fde240a2b90 msgr2=0x7fde240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:16.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.389+0000 7fde3c8c3640 1 --2- 172.21.15.107:0/561143917 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fde240a2b90 0x7fde240a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fde381017c0 tx=0x7fde28005f90 comp rx=0 tx=0).stop 2024-05-09T08:31:16.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.389+0000 7fde3c8c3640 1 -- 172.21.15.107:0/561143917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde38102320 msgr2=0x7fde3819efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:16.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.389+0000 7fde3c8c3640 1 --2- 172.21.15.107:0/561143917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde38102320 0x7fde3819efc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fde30009dc0 tx=0x7fde30003ba0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.389+0000 7fde3c8c3640 1 -- 172.21.15.107:0/561143917 shutdown_connections 2024-05-09T08:31:16.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.389+0000 7fde3c8c3640 1 --2- 172.21.15.107:0/561143917 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fde240a2b90 0x7fde240a5050 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.389+0000 7fde3c8c3640 1 --2- 172.21.15.107:0/561143917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde38102320 0x7fde3819efc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.389+0000 7fde3c8c3640 1 --2- 172.21.15.107:0/561143917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde38101960 0x7fde38197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.389+0000 7fde3c8c3640 1 --2- 172.21.15.107:0/561143917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde38100770 0x7fde38197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.389+0000 7fde3c8c3640 1 -- 172.21.15.107:0/561143917 >> 172.21.15.107:0/561143917 conn(0x7fde380fbce0 msgr2=0x7fde38104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:16.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.389+0000 7fde3c8c3640 1 -- 172.21.15.107:0/561143917 shutdown_connections 2024-05-09T08:31:16.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.389+0000 7fde3c8c3640 1 -- 172.21.15.107:0/561143917 wait complete. 2024-05-09T08:31:16.400 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi107.front.sepia.ceph.com... 2024-05-09T08:31:16.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:31:16.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-05-09T08:31:16.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi107.stderr:ceph-fuse[71586]: fuse finished with error 0 and tester_r 0 2024-05-09T08:31:16.427 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T08:31:16.505 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2024-05-09T08:31:16.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:31:16.505 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-05-09T08:31:16.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:31:16.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.671+0000 7f11d79ce640 1 -- 172.21.15.107:0/4180150809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11d0101960 msgr2=0x7f11d0101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:16.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.671+0000 7f11d79ce640 1 --2- 172.21.15.107:0/4180150809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11d0101960 0x7f11d0101de0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f11c4009900 tx=0x7f11c402a410 comp rx=0 tx=0).stop 2024-05-09T08:31:16.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.671+0000 7f11d79ce640 1 -- 172.21.15.107:0/4180150809 shutdown_connections 2024-05-09T08:31:16.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.671+0000 7f11d79ce640 1 --2- 172.21.15.107:0/4180150809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11d0102320 0x7f11d0108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.671+0000 7f11d79ce640 1 --2- 172.21.15.107:0/4180150809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11d0101960 0x7f11d0101de0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.671+0000 7f11d79ce640 1 --2- 172.21.15.107:0/4180150809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11d0100770 0x7f11d0100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.671+0000 7f11d79ce640 1 -- 172.21.15.107:0/4180150809 >> 172.21.15.107:0/4180150809 conn(0x7f11d00fbce0 msgr2=0x7f11d00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.672+0000 7f11d79ce640 1 -- 172.21.15.107:0/4180150809 shutdown_connections 2024-05-09T08:31:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.672+0000 7f11d79ce640 1 -- 172.21.15.107:0/4180150809 wait complete. 2024-05-09T08:31:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.672+0000 7f11d79ce640 1 Processor -- start 2024-05-09T08:31:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.672+0000 7f11d79ce640 1 -- start start 2024-05-09T08:31:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.672+0000 7f11d79ce640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11d0100770 0x7f11d0197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.672+0000 7f11d79ce640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11d0101960 0x7f11d0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.672+0000 7f11d79ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11d0102320 0x7f11d019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.672+0000 7f11d79ce640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f11d010d420 con 0x7f11d0102320 2024-05-09T08:31:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.672+0000 7f11d79ce640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f11d010d2a0 con 0x7f11d0100770 2024-05-09T08:31:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.672+0000 7f11d79ce640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f11d010d5a0 con 0x7f11d0101960 2024-05-09T08:31:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.672+0000 7f11d5f6b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11d0102320 0x7f11d019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.672+0000 7f11d5f6b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11d0102320 0x7f11d019f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37714/0 (socket says 172.21.15.107:37714) 2024-05-09T08:31:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.672+0000 7f11d5f6b640 1 -- 172.21.15.107:0/1125697983 learned_addr learned my addr 172.21.15.107:0/1125697983 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.673+0000 7f11d4f69640 1 --2- 172.21.15.107:0/1125697983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11d0101960 0x7f11d0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.673+0000 7f11d576a640 1 --2- 172.21.15.107:0/1125697983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11d0100770 0x7f11d0197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.673+0000 7f11d5f6b640 1 -- 172.21.15.107:0/1125697983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11d0101960 msgr2=0x7f11d0197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.673+0000 7f11d5f6b640 1 --2- 172.21.15.107:0/1125697983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11d0101960 0x7f11d0197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.673+0000 7f11d5f6b640 1 -- 172.21.15.107:0/1125697983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11d0100770 msgr2=0x7f11d0197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.673+0000 7f11d5f6b640 1 --2- 172.21.15.107:0/1125697983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11d0100770 0x7f11d0197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.673+0000 7f11d5f6b640 1 -- 172.21.15.107:0/1125697983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d019f6a0 con 0x7f11d0102320 2024-05-09T08:31:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.673+0000 7f11d5f6b640 1 --2- 172.21.15.107:0/1125697983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11d0102320 0x7f11d019f030 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f11c800eaa0 tx=0x7f11c800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.673+0000 7f11c27fc640 1 -- 172.21.15.107:0/1125697983 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c800cdb0 con 0x7f11d0102320 2024-05-09T08:31:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.673+0000 7f11d79ce640 1 -- 172.21.15.107:0/1125697983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d019f990 con 0x7f11d0102320 2024-05-09T08:31:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.673+0000 7f11c27fc640 1 -- 172.21.15.107:0/1125697983 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f11c8005020 con 0x7f11d0102320 2024-05-09T08:31:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.673+0000 7f11c27fc640 1 -- 172.21.15.107:0/1125697983 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c80116d0 con 0x7f11d0102320 2024-05-09T08:31:16.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.673+0000 7f11d79ce640 1 -- 172.21.15.107:0/1125697983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d019feb0 con 0x7f11d0102320 2024-05-09T08:31:16.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.674+0000 7f11d79ce640 1 -- 172.21.15.107:0/1125697983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d0107a90 con 0x7f11d0102320 2024-05-09T08:31:16.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.675+0000 7f11c27fc640 1 -- 172.21.15.107:0/1125697983 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f11c80051d0 con 0x7f11d0102320 2024-05-09T08:31:16.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.676+0000 7f11c27fc640 1 --2- 172.21.15.107:0/1125697983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f11bc0a2b90 0x7f11bc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:16.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.676+0000 7f11d576a640 1 --2- 172.21.15.107:0/1125697983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f11bc0a2b90 0x7f11bc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:16.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.676+0000 7f11c27fc640 1 -- 172.21.15.107:0/1125697983 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f11c80cf310 con 0x7f11d0102320 2024-05-09T08:31:16.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.676+0000 7f11d576a640 1 --2- 172.21.15.107:0/1125697983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f11bc0a2b90 0x7f11bc0a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f11cc005fd0 tx=0x7f11cc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:16.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.680+0000 7f11c27fc640 1 -- 172.21.15.107:0/1125697983 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f11c8016030 con 0x7f11d0102320 2024-05-09T08:31:16.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.864+0000 7f11d79ce640 1 -- 172.21.15.107:0/1125697983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f11d002d290 con 0x7f11d0102320 2024-05-09T08:31:16.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.866+0000 7f11c27fc640 1 -- 172.21.15.107:0/1125697983 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f11c809b9e0 con 0x7f11d0102320 2024-05-09T08:31:16.866 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:16.866 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:31:05.201799+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:31:16.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.868+0000 7f11d79ce640 1 -- 172.21.15.107:0/1125697983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f11bc0a2b90 msgr2=0x7f11bc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:16.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.868+0000 7f11d79ce640 1 --2- 172.21.15.107:0/1125697983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f11bc0a2b90 0x7f11bc0a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f11cc005fd0 tx=0x7f11cc00b410 comp rx=0 tx=0).stop 2024-05-09T08:31:16.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.868+0000 7f11d79ce640 1 -- 172.21.15.107:0/1125697983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11d0102320 msgr2=0x7f11d019f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:16.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.868+0000 7f11d79ce640 1 --2- 172.21.15.107:0/1125697983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11d0102320 0x7f11d019f030 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f11c800eaa0 tx=0x7f11c800ef60 comp rx=0 tx=0).stop 2024-05-09T08:31:16.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.868+0000 7f11d79ce640 1 -- 172.21.15.107:0/1125697983 shutdown_connections 2024-05-09T08:31:16.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.868+0000 7f11d79ce640 1 --2- 172.21.15.107:0/1125697983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f11bc0a2b90 0x7f11bc0a5050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.868+0000 7f11d79ce640 1 --2- 172.21.15.107:0/1125697983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11d0102320 0x7f11d019f030 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.868+0000 7f11d79ce640 1 --2- 172.21.15.107:0/1125697983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11d0101960 0x7f11d0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.868+0000 7f11d79ce640 1 --2- 172.21.15.107:0/1125697983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11d0100770 0x7f11d0197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.868+0000 7f11d79ce640 1 -- 172.21.15.107:0/1125697983 >> 172.21.15.107:0/1125697983 conn(0x7f11d00fbce0 msgr2=0x7f11d0104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:16.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.868+0000 7f11d79ce640 1 -- 172.21.15.107:0/1125697983 shutdown_connections 2024-05-09T08:31:16.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.868+0000 7f11d79ce640 1 -- 172.21.15.107:0/1125697983 wait complete. 2024-05-09T08:31:16.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:31:16.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.982+0000 7f5d5d21c640 1 -- 172.21.15.107:0/28050927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d58101960 msgr2=0x7f5d58101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.982+0000 7f5d5d21c640 1 --2- 172.21.15.107:0/28050927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d58101960 0x7f5d58101de0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c009900 tx=0x7f5d4c02a410 comp rx=0 tx=0).stop 2024-05-09T08:31:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.982+0000 7f5d5d21c640 1 -- 172.21.15.107:0/28050927 shutdown_connections 2024-05-09T08:31:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.982+0000 7f5d5d21c640 1 --2- 172.21.15.107:0/28050927 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d58102320 0x7f5d58108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.982+0000 7f5d5d21c640 1 --2- 172.21.15.107:0/28050927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d58101960 0x7f5d58101de0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.982+0000 7f5d5d21c640 1 --2- 172.21.15.107:0/28050927 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d58100770 0x7f5d58100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.982+0000 7f5d5d21c640 1 -- 172.21.15.107:0/28050927 >> 172.21.15.107:0/28050927 conn(0x7f5d580fbce0 msgr2=0x7f5d580fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.982+0000 7f5d5d21c640 1 -- 172.21.15.107:0/28050927 shutdown_connections 2024-05-09T08:31:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.982+0000 7f5d5d21c640 1 -- 172.21.15.107:0/28050927 wait complete. 2024-05-09T08:31:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.982+0000 7f5d5d21c640 1 Processor -- start 2024-05-09T08:31:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5d21c640 1 -- start start 2024-05-09T08:31:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5d21c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d58100770 0x7f5d58199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5d21c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d58101960 0x7f5d5819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5d21c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d58102320 0x7f5d581a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:16.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5d21c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5810c2a0 con 0x7f5d58102320 2024-05-09T08:31:16.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5d21c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5810c120 con 0x7f5d58100770 2024-05-09T08:31:16.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5d21c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5810c420 con 0x7f5d58101960 2024-05-09T08:31:16.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d58102320 0x7f5d581a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:16.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d58102320 0x7f5d581a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37738/0 (socket says 172.21.15.107:37738) 2024-05-09T08:31:16.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5759e640 1 -- 172.21.15.107:0/2004152983 learned_addr learned my addr 172.21.15.107:0/2004152983 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:16.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d56d9d640 1 --2- 172.21.15.107:0/2004152983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d58100770 0x7f5d58199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:16.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5659c640 1 --2- 172.21.15.107:0/2004152983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d58101960 0x7f5d5819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:16.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5759e640 1 -- 172.21.15.107:0/2004152983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d58101960 msgr2=0x7f5d5819a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:16.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5759e640 1 --2- 172.21.15.107:0/2004152983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d58101960 0x7f5d5819a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5759e640 1 -- 172.21.15.107:0/2004152983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d58100770 msgr2=0x7f5d58199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5759e640 1 --2- 172.21.15.107:0/2004152983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d58100770 0x7f5d58199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5759e640 1 -- 172.21.15.107:0/2004152983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d581a1840 con 0x7f5d58102320 2024-05-09T08:31:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5659c640 1 --2- 172.21.15.107:0/2004152983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d58101960 0x7f5d5819a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.983+0000 7f5d5759e640 1 --2- 172.21.15.107:0/2004152983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d58102320 0x7f5d581a11a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000eaa0 tx=0x7f5d5000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.984+0000 7f5d3ffff640 1 -- 172.21.15.107:0/2004152983 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5000cdb0 con 0x7f5d58102320 2024-05-09T08:31:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.984+0000 7f5d5d21c640 1 -- 172.21.15.107:0/2004152983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d581a1b30 con 0x7f5d58102320 2024-05-09T08:31:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.984+0000 7f5d3ffff640 1 -- 172.21.15.107:0/2004152983 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5d50005020 con 0x7f5d58102320 2024-05-09T08:31:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.984+0000 7f5d3ffff640 1 -- 172.21.15.107:0/2004152983 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d500116d0 con 0x7f5d58102320 2024-05-09T08:31:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.984+0000 7f5d5d21c640 1 -- 172.21.15.107:0/2004152983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d581a20d0 con 0x7f5d58102320 2024-05-09T08:31:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.985+0000 7f5d5d21c640 1 -- 172.21.15.107:0/2004152983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d581a22a0 con 0x7f5d58102320 2024-05-09T08:31:16.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.986+0000 7f5d3ffff640 1 -- 172.21.15.107:0/2004152983 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f5d500051d0 con 0x7f5d58102320 2024-05-09T08:31:16.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.986+0000 7f5d3ffff640 1 --2- 172.21.15.107:0/2004152983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5d400a2b90 0x7f5d400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:16.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.986+0000 7f5d3ffff640 1 -- 172.21.15.107:0/2004152983 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5d500cf4c0 con 0x7f5d58102320 2024-05-09T08:31:16.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.987+0000 7f5d56d9d640 1 --2- 172.21.15.107:0/2004152983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5d400a2b90 0x7f5d400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:16.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.987+0000 7f5d56d9d640 1 --2- 172.21.15.107:0/2004152983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5d400a2b90 0x7f5d400a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5d5819b100 tx=0x7f5d48005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:16.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:16.991+0000 7f5d3ffff640 1 -- 172.21.15.107:0/2004152983 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5d50016030 con 0x7f5d58102320 2024-05-09T08:31:17.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.175+0000 7f5d5d21c640 1 -- 172.21.15.107:0/2004152983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d581a2630 con 0x7f5d58102320 2024-05-09T08:31:17.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.177+0000 7f5d3ffff640 1 -- 172.21.15.107:0/2004152983 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f5d5009c1e0 con 0x7f5d58102320 2024-05-09T08:31:17.176 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:17.176 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:31:05.201799+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:31:17.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.179+0000 7f5d5d21c640 1 -- 172.21.15.107:0/2004152983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5d400a2b90 msgr2=0x7f5d400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:17.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.179+0000 7f5d5d21c640 1 --2- 172.21.15.107:0/2004152983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5d400a2b90 0x7f5d400a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5d5819b100 tx=0x7f5d48005f90 comp rx=0 tx=0).stop 2024-05-09T08:31:17.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.179+0000 7f5d5d21c640 1 -- 172.21.15.107:0/2004152983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d58102320 msgr2=0x7f5d581a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:17.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.179+0000 7f5d5d21c640 1 --2- 172.21.15.107:0/2004152983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d58102320 0x7f5d581a11a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000eaa0 tx=0x7f5d5000ef60 comp rx=0 tx=0).stop 2024-05-09T08:31:17.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.179+0000 7f5d5d21c640 1 -- 172.21.15.107:0/2004152983 shutdown_connections 2024-05-09T08:31:17.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.179+0000 7f5d5d21c640 1 --2- 172.21.15.107:0/2004152983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5d400a2b90 0x7f5d400a5050 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:17.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.179+0000 7f5d5d21c640 1 --2- 172.21.15.107:0/2004152983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d58102320 0x7f5d581a11a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:17.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.179+0000 7f5d5d21c640 1 --2- 172.21.15.107:0/2004152983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d58101960 0x7f5d5819a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:17.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.179+0000 7f5d5d21c640 1 --2- 172.21.15.107:0/2004152983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d58100770 0x7f5d58199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:17.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.179+0000 7f5d5d21c640 1 -- 172.21.15.107:0/2004152983 >> 172.21.15.107:0/2004152983 conn(0x7f5d580fbce0 msgr2=0x7f5d580fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:17.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.179+0000 7f5d5d21c640 1 -- 172.21.15.107:0/2004152983 shutdown_connections 2024-05-09T08:31:17.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.179+0000 7f5d5d21c640 1 -- 172.21.15.107:0/2004152983 wait complete. 2024-05-09T08:31:17.188 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:17 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2876083466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:31:17.188 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:17 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/561143917' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:31:17.188 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:17 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1125697983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:17.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T08:31:17.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.297+0000 7fb74c114640 1 -- 172.21.15.107:0/855834765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb744104cc0 msgr2=0x7fb7441070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:17.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.297+0000 7fb74c114640 1 --2- 172.21.15.107:0/855834765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb744104cc0 0x7fb7441070b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb734009900 tx=0x7fb73402a410 comp rx=0 tx=0).stop 2024-05-09T08:31:17.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.297+0000 7fb74c114640 1 -- 172.21.15.107:0/855834765 shutdown_connections 2024-05-09T08:31:17.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.297+0000 7fb74c114640 1 --2- 172.21.15.107:0/855834765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7441075f0 0x7fb7441099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:17.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.297+0000 7fb74c114640 1 --2- 172.21.15.107:0/855834765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb744104cc0 0x7fb7441070b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:17.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.297+0000 7fb74c114640 1 --2- 172.21.15.107:0/855834765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb744102390 0x7fb744104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:17.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.297+0000 7fb74c114640 1 -- 172.21.15.107:0/855834765 >> 172.21.15.107:0/855834765 conn(0x7fb7440fbce0 msgr2=0x7fb7440fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:17.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.297+0000 7fb74c114640 1 -- 172.21.15.107:0/855834765 shutdown_connections 2024-05-09T08:31:17.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.297+0000 7fb74c114640 1 -- 172.21.15.107:0/855834765 wait complete. 2024-05-09T08:31:17.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.297+0000 7fb74c114640 1 Processor -- start 2024-05-09T08:31:17.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.297+0000 7fb74c114640 1 -- start start 2024-05-09T08:31:17.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb74c114640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb744102390 0x7fb744197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:17.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb74c114640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb744104cc0 0x7fb744198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:17.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb74c114640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7441075f0 0x7fb74419f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:17.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb74c114640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb74410d470 con 0x7fb744104cc0 2024-05-09T08:31:17.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb74c114640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb74410d2f0 con 0x7fb7441075f0 2024-05-09T08:31:17.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb74c114640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb74410d5f0 con 0x7fb744102390 2024-05-09T08:31:17.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb7496af640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb744104cc0 0x7fb744198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:17.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb7496af640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb744104cc0 0x7fb744198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37754/0 (socket says 172.21.15.107:37754) 2024-05-09T08:31:17.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb7496af640 1 -- 172.21.15.107:0/3335230327 learned_addr learned my addr 172.21.15.107:0/3335230327 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:17.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb749eb0640 1 --2- 172.21.15.107:0/3335230327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb744102390 0x7fb744197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:17.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb74a6b1640 1 --2- 172.21.15.107:0/3335230327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7441075f0 0x7fb74419f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:17.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb7496af640 1 -- 172.21.15.107:0/3335230327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb744102390 msgr2=0x7fb744197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:17.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb7496af640 1 --2- 172.21.15.107:0/3335230327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb744102390 0x7fb744197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:17.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb7496af640 1 -- 172.21.15.107:0/3335230327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7441075f0 msgr2=0x7fb74419f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:17.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb7496af640 1 --2- 172.21.15.107:0/3335230327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7441075f0 0x7fb74419f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:17.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb7496af640 1 -- 172.21.15.107:0/3335230327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74419f940 con 0x7fb744104cc0 2024-05-09T08:31:17.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb74a6b1640 1 --2- 172.21.15.107:0/3335230327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7441075f0 0x7fb74419f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:17.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb7496af640 1 --2- 172.21.15.107:0/3335230327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb744104cc0 0x7fb744198270 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb734009230 tx=0x7fb734004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:17.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb73affd640 1 -- 172.21.15.107:0/3335230327 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73403f030 con 0x7fb744104cc0 2024-05-09T08:31:17.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb73affd640 1 -- 172.21.15.107:0/3335230327 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb734003cb0 con 0x7fb744104cc0 2024-05-09T08:31:17.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb74c114640 1 -- 172.21.15.107:0/3335230327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb74419fbd0 con 0x7fb744104cc0 2024-05-09T08:31:17.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb73affd640 1 -- 172.21.15.107:0/3335230327 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb734035070 con 0x7fb744104cc0 2024-05-09T08:31:17.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.298+0000 7fb74c114640 1 -- 172.21.15.107:0/3335230327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7441a00c0 con 0x7fb744104cc0 2024-05-09T08:31:17.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.299+0000 7fb74c114640 1 -- 172.21.15.107:0/3335230327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7440fd7d0 con 0x7fb744104cc0 2024-05-09T08:31:17.360 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:17 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2876083466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:31:17.360 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:17 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/561143917' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:31:17.360 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:17 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1125697983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:17.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.301+0000 7fb73affd640 1 -- 172.21.15.107:0/3335230327 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fb734042030 con 0x7fb744104cc0 2024-05-09T08:31:17.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.301+0000 7fb73affd640 1 --2- 172.21.15.107:0/3335230327 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb7300a2b90 0x7fb7300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:17.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.302+0000 7fb749eb0640 1 --2- 172.21.15.107:0/3335230327 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb7300a2b90 0x7fb7300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:17.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.302+0000 7fb73affd640 1 -- 172.21.15.107:0/3335230327 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb7340efa90 con 0x7fb744104cc0 2024-05-09T08:31:17.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.302+0000 7fb749eb0640 1 --2- 172.21.15.107:0/3335230327 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb7300a2b90 0x7fb7300a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb744199200 tx=0x7fb740008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:17.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.306+0000 7fb73affd640 1 -- 172.21.15.107:0/3335230327 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb7340bc7b0 con 0x7fb744104cc0 2024-05-09T08:31:17.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:17 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2876083466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:31:17.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:17 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/561143917' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:31:17.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:17 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1125697983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:17.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:17.519+0000 7fb74c114640 1 -- 172.21.15.107:0/3335230327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb744062460 con 0x7fb744104cc0 2024-05-09T08:31:18.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.033+0000 7fb73affd640 1 -- 172.21.15.107:0/3335230327 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fb7340bc0b0 con 0x7fb744104cc0 2024-05-09T08:31:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.036+0000 7fb74c114640 1 -- 172.21.15.107:0/3335230327 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb7300a2b90 msgr2=0x7fb7300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.036+0000 7fb74c114640 1 --2- 172.21.15.107:0/3335230327 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb7300a2b90 0x7fb7300a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb744199200 tx=0x7fb740008040 comp rx=0 tx=0).stop 2024-05-09T08:31:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.036+0000 7fb74c114640 1 -- 172.21.15.107:0/3335230327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb744104cc0 msgr2=0x7fb744198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:18.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.036+0000 7fb74c114640 1 --2- 172.21.15.107:0/3335230327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb744104cc0 0x7fb744198270 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb734009230 tx=0x7fb734004340 comp rx=0 tx=0).stop 2024-05-09T08:31:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.036+0000 7fb74c114640 1 -- 172.21.15.107:0/3335230327 shutdown_connections 2024-05-09T08:31:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.036+0000 7fb74c114640 1 --2- 172.21.15.107:0/3335230327 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb7300a2b90 0x7fb7300a5050 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.036+0000 7fb74c114640 1 --2- 172.21.15.107:0/3335230327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7441075f0 0x7fb74419f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.036+0000 7fb74c114640 1 --2- 172.21.15.107:0/3335230327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb744104cc0 0x7fb744198270 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.036+0000 7fb74c114640 1 --2- 172.21.15.107:0/3335230327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb744102390 0x7fb744197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.036+0000 7fb74c114640 1 -- 172.21.15.107:0/3335230327 >> 172.21.15.107:0/3335230327 conn(0x7fb7440fbce0 msgr2=0x7fb744104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.036+0000 7fb74c114640 1 -- 172.21.15.107:0/3335230327 shutdown_connections 2024-05-09T08:31:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.036+0000 7fb74c114640 1 -- 172.21.15.107:0/3335230327 wait complete. 2024-05-09T08:31:18.051 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-05-09T08:31:18.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-05-09T08:31:18.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.159+0000 7fbab3948640 1 -- 172.21.15.107:0/1795955690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaac100790 msgr2=0x7fbaac100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:18.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.159+0000 7fbab3948640 1 --2- 172.21.15.107:0/1795955690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaac100790 0x7fbaac100bb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4009900 tx=0x7fbaa402a410 comp rx=0 tx=0).stop 2024-05-09T08:31:18.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.159+0000 7fbab3948640 1 -- 172.21.15.107:0/1795955690 shutdown_connections 2024-05-09T08:31:18.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.159+0000 7fbab3948640 1 --2- 172.21.15.107:0/1795955690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaac102340 0x7fbaac108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:18.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.159+0000 7fbab3948640 1 --2- 172.21.15.107:0/1795955690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbaac101980 0x7fbaac101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:18.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.159+0000 7fbab3948640 1 --2- 172.21.15.107:0/1795955690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaac100790 0x7fbaac100bb0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:18.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.159+0000 7fbab3948640 1 -- 172.21.15.107:0/1795955690 >> 172.21.15.107:0/1795955690 conn(0x7fbaac0fbd20 msgr2=0x7fbaac0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:18.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.159+0000 7fbab3948640 1 -- 172.21.15.107:0/1795955690 shutdown_connections 2024-05-09T08:31:18.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.159+0000 7fbab3948640 1 -- 172.21.15.107:0/1795955690 wait complete. 2024-05-09T08:31:18.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab3948640 1 Processor -- start 2024-05-09T08:31:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab3948640 1 -- start start 2024-05-09T08:31:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab3948640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbaac100790 0x7fbaac06a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab3948640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaac101980 0x7fbaac19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab3948640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaac102340 0x7fbaac19ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab3948640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbaac10c290 con 0x7fbaac101980 2024-05-09T08:31:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab3948640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbaac10c110 con 0x7fbaac102340 2024-05-09T08:31:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab3948640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbaac10c410 con 0x7fbaac100790 2024-05-09T08:31:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab0ee3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaac101980 0x7fbaac19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab0ee3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaac101980 0x7fbaac19c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37770/0 (socket says 172.21.15.107:37770) 2024-05-09T08:31:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab0ee3640 1 -- 172.21.15.107:0/599668853 learned_addr learned my addr 172.21.15.107:0/599668853 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab1ee5640 1 --2- 172.21.15.107:0/599668853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaac102340 0x7fbaac19ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab0ee3640 1 -- 172.21.15.107:0/599668853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbaac100790 msgr2=0x7fbaac06a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab16e4640 1 --2- 172.21.15.107:0/599668853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbaac100790 0x7fbaac06a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:18.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab0ee3640 1 --2- 172.21.15.107:0/599668853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbaac100790 0x7fbaac06a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:18.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab0ee3640 1 -- 172.21.15.107:0/599668853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaac102340 msgr2=0x7fbaac19ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:18.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab0ee3640 1 --2- 172.21.15.107:0/599668853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaac102340 0x7fbaac19ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:18.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab0ee3640 1 -- 172.21.15.107:0/599668853 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaac19f710 con 0x7fbaac101980 2024-05-09T08:31:18.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab1ee5640 1 --2- 172.21.15.107:0/599668853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaac102340 0x7fbaac19ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:18.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab0ee3640 1 --2- 172.21.15.107:0/599668853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaac101980 0x7fbaac19c7e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fba9c00c920 tx=0x7fba9c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:18.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbaa27fc640 1 -- 172.21.15.107:0/599668853 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c007c70 con 0x7fbaac101980 2024-05-09T08:31:18.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbaa27fc640 1 -- 172.21.15.107:0/599668853 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba9c007e10 con 0x7fbaac101980 2024-05-09T08:31:18.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbab3948640 1 -- 172.21.15.107:0/599668853 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaac19f9a0 con 0x7fbaac101980 2024-05-09T08:31:18.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.160+0000 7fbaa27fc640 1 -- 172.21.15.107:0/599668853 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c018820 con 0x7fbaac101980 2024-05-09T08:31:18.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.161+0000 7fbab3948640 1 -- 172.21.15.107:0/599668853 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaac19ff40 con 0x7fbaac101980 2024-05-09T08:31:18.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.161+0000 7fbab3948640 1 -- 172.21.15.107:0/599668853 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaac107bd0 con 0x7fbaac101980 2024-05-09T08:31:18.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.163+0000 7fbaa27fc640 1 -- 172.21.15.107:0/599668853 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fba9c0189c0 con 0x7fbaac101980 2024-05-09T08:31:18.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.163+0000 7fbaa27fc640 1 --2- 172.21.15.107:0/599668853 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fba980a2c10 0x7fba980a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:18.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.163+0000 7fbab16e4640 1 --2- 172.21.15.107:0/599668853 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fba980a2c10 0x7fba980a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:18.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.163+0000 7fbaa27fc640 1 -- 172.21.15.107:0/599668853 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fba9c0cf400 con 0x7fbaac101980 2024-05-09T08:31:18.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.164+0000 7fbab16e4640 1 --2- 172.21.15.107:0/599668853 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fba980a2c10 0x7fba980a50d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbaa402a930 tx=0x7fbaa4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:18.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.168+0000 7fbaa27fc640 1 -- 172.21.15.107:0/599668853 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fba9c014030 con 0x7fbaac101980 2024-05-09T08:31:18.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:18 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2004152983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:18.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:18 smithi157 ceph-mon[33614]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 264 B/s wr, 0 op/s 2024-05-09T08:31:18.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:18 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3335230327' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-09T08:31:18.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:18 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2004152983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:18.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:18 smithi110 ceph-mon[33622]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 264 B/s wr, 0 op/s 2024-05-09T08:31:18.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:18 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3335230327' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-09T08:31:18.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:18.382+0000 7fbab3948640 1 -- 172.21.15.107:0/599668853 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fbaac062460 con 0x7fbaac101980 2024-05-09T08:31:18.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:18 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2004152983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:18.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:18 smithi107 ceph-mon[30122]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 264 B/s wr, 0 op/s 2024-05-09T08:31:18.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:18 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3335230327' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-09T08:31:19.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.035+0000 7fbaa27fc640 1 -- 172.21.15.107:0/599668853 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7fba9c00de20 con 0x7fbaac101980 2024-05-09T08:31:19.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.038+0000 7fbab3948640 1 -- 172.21.15.107:0/599668853 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fba980a2c10 msgr2=0x7fba980a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:19.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.038+0000 7fbab3948640 1 --2- 172.21.15.107:0/599668853 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fba980a2c10 0x7fba980a50d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbaa402a930 tx=0x7fbaa4000bc0 comp rx=0 tx=0).stop 2024-05-09T08:31:19.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.038+0000 7fbab3948640 1 -- 172.21.15.107:0/599668853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaac101980 msgr2=0x7fbaac19c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:19.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.038+0000 7fbab3948640 1 --2- 172.21.15.107:0/599668853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaac101980 0x7fbaac19c7e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fba9c00c920 tx=0x7fba9c00cde0 comp rx=0 tx=0).stop 2024-05-09T08:31:19.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.038+0000 7fbab3948640 1 -- 172.21.15.107:0/599668853 shutdown_connections 2024-05-09T08:31:19.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.038+0000 7fbab3948640 1 --2- 172.21.15.107:0/599668853 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fba980a2c10 0x7fba980a50d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:19.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.038+0000 7fbab3948640 1 --2- 172.21.15.107:0/599668853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaac102340 0x7fbaac19ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:19.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.038+0000 7fbab3948640 1 --2- 172.21.15.107:0/599668853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaac101980 0x7fbaac19c7e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:19.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.038+0000 7fbab3948640 1 --2- 172.21.15.107:0/599668853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbaac100790 0x7fbaac06a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:19.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.038+0000 7fbab3948640 1 -- 172.21.15.107:0/599668853 >> 172.21.15.107:0/599668853 conn(0x7fbaac0fbd20 msgr2=0x7fbaac104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:19.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.038+0000 7fbab3948640 1 -- 172.21.15.107:0/599668853 shutdown_connections 2024-05-09T08:31:19.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.038+0000 7fbab3948640 1 -- 172.21.15.107:0/599668853 wait complete. 2024-05-09T08:31:19.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-05-09T08:31:19.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.166+0000 7fa8967ee640 1 -- 172.21.15.107:0/3899228268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8901009c0 msgr2=0x7fa890100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:19.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.166+0000 7fa8967ee640 1 --2- 172.21.15.107:0/3899228268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8901009c0 0x7fa890100e40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa878009900 tx=0x7fa87802a400 comp rx=0 tx=0).stop 2024-05-09T08:31:19.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.166+0000 7fa8967ee640 1 -- 172.21.15.107:0/3899228268 shutdown_connections 2024-05-09T08:31:19.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.166+0000 7fa8967ee640 1 --2- 172.21.15.107:0/3899228268 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa890101380 0x7fa890108c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:19.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.166+0000 7fa8967ee640 1 --2- 172.21.15.107:0/3899228268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8901009c0 0x7fa890100e40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:19.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.166+0000 7fa8967ee640 1 --2- 172.21.15.107:0/3899228268 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa890101e00 0x7fa890102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:19.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.166+0000 7fa8967ee640 1 -- 172.21.15.107:0/3899228268 >> 172.21.15.107:0/3899228268 conn(0x7fa8900fb470 msgr2=0x7fa8900fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:19.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.166+0000 7fa8967ee640 1 -- 172.21.15.107:0/3899228268 shutdown_connections 2024-05-09T08:31:19.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.166+0000 7fa8967ee640 1 -- 172.21.15.107:0/3899228268 wait complete. 2024-05-09T08:31:19.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.166+0000 7fa8967ee640 1 Processor -- start 2024-05-09T08:31:19.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.166+0000 7fa8967ee640 1 -- start start 2024-05-09T08:31:19.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa8967ee640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8901009c0 0x7fa890199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa8967ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa890101380 0x7fa89019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa8967ee640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa890101e00 0x7fa8901a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa8967ee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa89010c2a0 con 0x7fa890101380 2024-05-09T08:31:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa8967ee640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa89010c120 con 0x7fa890101e00 2024-05-09T08:31:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa8967ee640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa89010c420 con 0x7fa8901009c0 2024-05-09T08:31:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa887fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa890101380 0x7fa89019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa887fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa890101380 0x7fa89019a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37782/0 (socket says 172.21.15.107:37782) 2024-05-09T08:31:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa887fff640 1 -- 172.21.15.107:0/3097665343 learned_addr learned my addr 172.21.15.107:0/3097665343 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa894d8b640 1 --2- 172.21.15.107:0/3097665343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa890101e00 0x7fa8901a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa88ffff640 1 --2- 172.21.15.107:0/3097665343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8901009c0 0x7fa890199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa887fff640 1 -- 172.21.15.107:0/3097665343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8901009c0 msgr2=0x7fa890199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa887fff640 1 --2- 172.21.15.107:0/3097665343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8901009c0 0x7fa890199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa887fff640 1 -- 172.21.15.107:0/3097665343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa890101e00 msgr2=0x7fa8901a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa887fff640 1 --2- 172.21.15.107:0/3097665343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa890101e00 0x7fa8901a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:19.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa887fff640 1 -- 172.21.15.107:0/3097665343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8901a1800 con 0x7fa890101380 2024-05-09T08:31:19.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa894d8b640 1 --2- 172.21.15.107:0/3097665343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa890101e00 0x7fa8901a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:19.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa887fff640 1 --2- 172.21.15.107:0/3097665343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa890101380 0x7fa89019a120 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa878009230 tx=0x7fa878004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:19.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa88dffb640 1 -- 172.21.15.107:0/3097665343 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87803f030 con 0x7fa890101380 2024-05-09T08:31:19.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa88dffb640 1 -- 172.21.15.107:0/3097665343 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa878003cb0 con 0x7fa890101380 2024-05-09T08:31:19.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa8967ee640 1 -- 172.21.15.107:0/3097665343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8901a1a90 con 0x7fa890101380 2024-05-09T08:31:19.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa88dffb640 1 -- 172.21.15.107:0/3097665343 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa878035070 con 0x7fa890101380 2024-05-09T08:31:19.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.167+0000 7fa8967ee640 1 -- 172.21.15.107:0/3097665343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8901a1f80 con 0x7fa890101380 2024-05-09T08:31:19.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.169+0000 7fa8967ee640 1 -- 172.21.15.107:0/3097665343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8901a21b0 con 0x7fa890101380 2024-05-09T08:31:19.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.170+0000 7fa88dffb640 1 -- 172.21.15.107:0/3097665343 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fa878042030 con 0x7fa890101380 2024-05-09T08:31:19.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.170+0000 7fa88dffb640 1 --2- 172.21.15.107:0/3097665343 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa87c0a2b90 0x7fa87c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:19.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.170+0000 7fa88ffff640 1 --2- 172.21.15.107:0/3097665343 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa87c0a2b90 0x7fa87c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:19.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.170+0000 7fa88dffb640 1 -- 172.21.15.107:0/3097665343 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa8780ef970 con 0x7fa890101380 2024-05-09T08:31:19.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.171+0000 7fa88ffff640 1 --2- 172.21.15.107:0/3097665343 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa87c0a2b90 0x7fa87c0a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa88000ba40 tx=0x7fa880005cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:19.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.175+0000 7fa88dffb640 1 -- 172.21.15.107:0/3097665343 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa8780bc690 con 0x7fa890101380 2024-05-09T08:31:19.304 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:19 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3335230327' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-09T08:31:19.304 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:19 smithi110 ceph-mon[33622]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-09T08:31:19.304 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:19 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/599668853' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-09T08:31:19.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:19 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3335230327' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-09T08:31:19.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:19 smithi157 ceph-mon[33614]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-09T08:31:19.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:19 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/599668853' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-09T08:31:19.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:19.387+0000 7fa8967ee640 1 -- 172.21.15.107:0/3097665343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fa8901a29e0 con 0x7fa890101380 2024-05-09T08:31:19.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:19 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3335230327' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-09T08:31:19.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:19 smithi107 ceph-mon[30122]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-09T08:31:19.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:19 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/599668853' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-09T08:31:20.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.046+0000 7fa88dffb640 1 -- 172.21.15.107:0/3097665343 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7fa8780bbf90 con 0x7fa890101380 2024-05-09T08:31:20.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.048+0000 7fa8967ee640 1 -- 172.21.15.107:0/3097665343 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa87c0a2b90 msgr2=0x7fa87c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:20.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.048+0000 7fa8967ee640 1 --2- 172.21.15.107:0/3097665343 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa87c0a2b90 0x7fa87c0a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa88000ba40 tx=0x7fa880005cc0 comp rx=0 tx=0).stop 2024-05-09T08:31:20.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.048+0000 7fa8967ee640 1 -- 172.21.15.107:0/3097665343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa890101380 msgr2=0x7fa89019a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:20.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.048+0000 7fa8967ee640 1 --2- 172.21.15.107:0/3097665343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa890101380 0x7fa89019a120 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa878009230 tx=0x7fa878004340 comp rx=0 tx=0).stop 2024-05-09T08:31:20.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.048+0000 7fa8967ee640 1 -- 172.21.15.107:0/3097665343 shutdown_connections 2024-05-09T08:31:20.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.048+0000 7fa8967ee640 1 --2- 172.21.15.107:0/3097665343 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa87c0a2b90 0x7fa87c0a5050 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:20.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.048+0000 7fa8967ee640 1 --2- 172.21.15.107:0/3097665343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa890101e00 0x7fa8901a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:20.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.048+0000 7fa8967ee640 1 --2- 172.21.15.107:0/3097665343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa890101380 0x7fa89019a120 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:20.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.048+0000 7fa8967ee640 1 --2- 172.21.15.107:0/3097665343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8901009c0 0x7fa890199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:20.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.048+0000 7fa8967ee640 1 -- 172.21.15.107:0/3097665343 >> 172.21.15.107:0/3097665343 conn(0x7fa8900fb470 msgr2=0x7fa8900fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:20.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.049+0000 7fa8967ee640 1 -- 172.21.15.107:0/3097665343 shutdown_connections 2024-05-09T08:31:20.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.049+0000 7fa8967ee640 1 -- 172.21.15.107:0/3097665343 wait complete. 2024-05-09T08:31:20.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-05-09T08:31:20.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.172+0000 7f2165542640 1 -- 172.21.15.107:0/1047107835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21600ff8f0 msgr2=0x7f21600ffd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:20.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.172+0000 7f2165542640 1 --2- 172.21.15.107:0/1047107835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21600ff8f0 0x7f21600ffd70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2148009900 tx=0x7f214802a410 comp rx=0 tx=0).stop 2024-05-09T08:31:20.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.172+0000 7f2165542640 1 -- 172.21.15.107:0/1047107835 shutdown_connections 2024-05-09T08:31:20.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.172+0000 7f2165542640 1 --2- 172.21.15.107:0/1047107835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21601002b0 0x7f2160109e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:20.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.172+0000 7f2165542640 1 --2- 172.21.15.107:0/1047107835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21600ff8f0 0x7f21600ffd70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:20.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.172+0000 7f2165542640 1 --2- 172.21.15.107:0/1047107835 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2160102e80 0x7f21601032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:20.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.172+0000 7f2165542640 1 -- 172.21.15.107:0/1047107835 >> 172.21.15.107:0/1047107835 conn(0x7f21600fb470 msgr2=0x7f21600fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:20.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.172+0000 7f2165542640 1 -- 172.21.15.107:0/1047107835 shutdown_connections 2024-05-09T08:31:20.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.172+0000 7f2165542640 1 -- 172.21.15.107:0/1047107835 wait complete. 2024-05-09T08:31:20.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f2165542640 1 Processor -- start 2024-05-09T08:31:20.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f2165542640 1 -- start start 2024-05-09T08:31:20.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f2165542640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21600ff8f0 0x7f2160197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:20.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f2165542640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21601002b0 0x7f2160198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:20.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f2165542640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2160102e80 0x7f216019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:20.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f2165542640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f216010d3b0 con 0x7f2160102e80 2024-05-09T08:31:20.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f2165542640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f216010d230 con 0x7f21601002b0 2024-05-09T08:31:20.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f2165542640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f216010d530 con 0x7f21600ff8f0 2024-05-09T08:31:20.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2160102e80 0x7f216019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:20.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2160102e80 0x7f216019f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37802/0 (socket says 172.21.15.107:37802) 2024-05-09T08:31:20.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f215f7fe640 1 -- 172.21.15.107:0/1511911944 learned_addr learned my addr 172.21.15.107:0/1511911944 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:20.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f215effd640 1 --2- 172.21.15.107:0/1511911944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21600ff8f0 0x7f2160197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:20.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f2157fff640 1 --2- 172.21.15.107:0/1511911944 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21601002b0 0x7f2160198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:20.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f215f7fe640 1 -- 172.21.15.107:0/1511911944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21600ff8f0 msgr2=0x7f2160197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:20.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f215f7fe640 1 --2- 172.21.15.107:0/1511911944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21600ff8f0 0x7f2160197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:20.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f215f7fe640 1 -- 172.21.15.107:0/1511911944 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21601002b0 msgr2=0x7f2160198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:20.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f215f7fe640 1 --2- 172.21.15.107:0/1511911944 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21601002b0 0x7f2160198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:20.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f215f7fe640 1 -- 172.21.15.107:0/1511911944 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f216019f750 con 0x7f2160102e80 2024-05-09T08:31:20.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.173+0000 7f215f7fe640 1 --2- 172.21.15.107:0/1511911944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2160102e80 0x7f216019f0e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f215800cad0 tx=0x7f215800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:20.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.174+0000 7f215cff9640 1 -- 172.21.15.107:0/1511911944 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215800ae20 con 0x7f2160102e80 2024-05-09T08:31:20.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.174+0000 7f2165542640 1 -- 172.21.15.107:0/1511911944 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f216019fa40 con 0x7f2160102e80 2024-05-09T08:31:20.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.174+0000 7f215cff9640 1 -- 172.21.15.107:0/1511911944 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2158005020 con 0x7f2160102e80 2024-05-09T08:31:20.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.174+0000 7f215cff9640 1 -- 172.21.15.107:0/1511911944 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215800f6b0 con 0x7f2160102e80 2024-05-09T08:31:20.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.174+0000 7f2165542640 1 -- 172.21.15.107:0/1511911944 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f216019ff40 con 0x7f2160102e80 2024-05-09T08:31:20.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.175+0000 7f2165542640 1 -- 172.21.15.107:0/1511911944 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2160108e30 con 0x7f2160102e80 2024-05-09T08:31:20.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.176+0000 7f215cff9640 1 -- 172.21.15.107:0/1511911944 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f21580051d0 con 0x7f2160102e80 2024-05-09T08:31:20.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.176+0000 7f215cff9640 1 --2- 172.21.15.107:0/1511911944 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f214c0a2b90 0x7f214c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:20.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.176+0000 7f215cff9640 1 -- 172.21.15.107:0/1511911944 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f21580cf360 con 0x7f2160102e80 2024-05-09T08:31:20.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.176+0000 7f215effd640 1 --2- 172.21.15.107:0/1511911944 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f214c0a2b90 0x7f214c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:20.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.177+0000 7f215effd640 1 --2- 172.21.15.107:0/1511911944 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f214c0a2b90 0x7f214c0a5050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2160069b10 tx=0x7f2150005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:20.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.181+0000 7f215cff9640 1 -- 172.21.15.107:0/1511911944 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2158014030 con 0x7f2160102e80 2024-05-09T08:31:20.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:20 smithi157 ceph-mon[33614]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-09T08:31:20.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:20 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/599668853' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-09T08:31:20.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:20 smithi157 ceph-mon[33614]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-09T08:31:20.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:20 smithi157 ceph-mon[33614]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-09T08:31:20.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:20 smithi157 ceph-mon[33614]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-05-09T08:31:20.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:20 smithi157 ceph-mon[33614]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-05-09T08:31:20.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:20 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3097665343' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-09T08:31:20.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:20 smithi157 ceph-mon[33614]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-09T08:31:20.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:20 smithi110 ceph-mon[33622]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-09T08:31:20.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:20 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/599668853' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-09T08:31:20.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:20 smithi110 ceph-mon[33622]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-09T08:31:20.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:20 smithi110 ceph-mon[33622]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-09T08:31:20.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:20 smithi110 ceph-mon[33622]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-05-09T08:31:20.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:20 smithi110 ceph-mon[33622]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-05-09T08:31:20.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:20 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3097665343' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-09T08:31:20.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:20 smithi110 ceph-mon[33622]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-09T08:31:20.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:20.392+0000 7f2165542640 1 -- 172.21.15.107:0/1511911944 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f2160062430 con 0x7f2160102e80 2024-05-09T08:31:20.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:20 smithi107 ceph-mon[30122]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-09T08:31:20.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:20 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/599668853' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-09T08:31:20.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:20 smithi107 ceph-mon[30122]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-09T08:31:20.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:20 smithi107 ceph-mon[30122]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-05-09T08:31:20.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:20 smithi107 ceph-mon[30122]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-05-09T08:31:20.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:20 smithi107 ceph-mon[30122]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-05-09T08:31:20.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:20 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3097665343' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-09T08:31:20.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:20 smithi107 ceph-mon[30122]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-09T08:31:21.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.052+0000 7f215cff9640 1 -- 172.21.15.107:0/1511911944 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f215809c080 con 0x7f2160102e80 2024-05-09T08:31:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.054+0000 7f2165542640 1 -- 172.21.15.107:0/1511911944 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f214c0a2b90 msgr2=0x7f214c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.054+0000 7f2165542640 1 --2- 172.21.15.107:0/1511911944 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f214c0a2b90 0x7f214c0a5050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2160069b10 tx=0x7f2150005f90 comp rx=0 tx=0).stop 2024-05-09T08:31:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.054+0000 7f2165542640 1 -- 172.21.15.107:0/1511911944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2160102e80 msgr2=0x7f216019f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.054+0000 7f2165542640 1 --2- 172.21.15.107:0/1511911944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2160102e80 0x7f216019f0e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f215800cad0 tx=0x7f215800cf90 comp rx=0 tx=0).stop 2024-05-09T08:31:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.054+0000 7f2165542640 1 -- 172.21.15.107:0/1511911944 shutdown_connections 2024-05-09T08:31:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.054+0000 7f2165542640 1 --2- 172.21.15.107:0/1511911944 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f214c0a2b90 0x7f214c0a5050 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.054+0000 7f2165542640 1 --2- 172.21.15.107:0/1511911944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2160102e80 0x7f216019f0e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.054+0000 7f2165542640 1 --2- 172.21.15.107:0/1511911944 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21601002b0 0x7f2160198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.054+0000 7f2165542640 1 --2- 172.21.15.107:0/1511911944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21600ff8f0 0x7f2160197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.054+0000 7f2165542640 1 -- 172.21.15.107:0/1511911944 >> 172.21.15.107:0/1511911944 conn(0x7f21600fb470 msgr2=0x7f2160106000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.054+0000 7f2165542640 1 -- 172.21.15.107:0/1511911944 shutdown_connections 2024-05-09T08:31:21.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.054+0000 7f2165542640 1 -- 172.21.15.107:0/1511911944 wait complete. 2024-05-09T08:31:21.066 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-05-09T08:31:21.066 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-05-09T08:31:21.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.172+0000 7f5b293d5640 1 -- 172.21.15.107:0/1602659258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b24104c40 msgr2=0x7f5b24107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:21.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.172+0000 7f5b293d5640 1 --2- 172.21.15.107:0/1602659258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b24104c40 0x7f5b24107030 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5b18009900 tx=0x7f5b1802a410 comp rx=0 tx=0).stop 2024-05-09T08:31:21.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.172+0000 7f5b293d5640 1 -- 172.21.15.107:0/1602659258 shutdown_connections 2024-05-09T08:31:21.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.172+0000 7f5b293d5640 1 --2- 172.21.15.107:0/1602659258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b24107570 0x7f5b241099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.172+0000 7f5b293d5640 1 --2- 172.21.15.107:0/1602659258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b24104c40 0x7f5b24107030 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.172+0000 7f5b293d5640 1 --2- 172.21.15.107:0/1602659258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b24102310 0x7f5b24104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.172+0000 7f5b293d5640 1 -- 172.21.15.107:0/1602659258 >> 172.21.15.107:0/1602659258 conn(0x7f5b240fbd20 msgr2=0x7f5b240fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:21.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.172+0000 7f5b293d5640 1 -- 172.21.15.107:0/1602659258 shutdown_connections 2024-05-09T08:31:21.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.172+0000 7f5b293d5640 1 -- 172.21.15.107:0/1602659258 wait complete. 2024-05-09T08:31:21.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.173+0000 7f5b293d5640 1 Processor -- start 2024-05-09T08:31:21.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.173+0000 7f5b293d5640 1 -- start start 2024-05-09T08:31:21.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.173+0000 7f5b293d5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b24102310 0x7f5b24197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:21.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.173+0000 7f5b293d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b24104c40 0x7f5b241980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:21.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.173+0000 7f5b293d5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b24107570 0x7f5b2419f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:21.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.173+0000 7f5b293d5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2410d3c0 con 0x7f5b24104c40 2024-05-09T08:31:21.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.173+0000 7f5b293d5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2410d240 con 0x7f5b24107570 2024-05-09T08:31:21.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.173+0000 7f5b293d5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2410d540 con 0x7f5b24102310 2024-05-09T08:31:21.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.173+0000 7f5b227fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b24104c40 0x7f5b241980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:21.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.173+0000 7f5b227fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b24104c40 0x7f5b241980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53150/0 (socket says 172.21.15.107:53150) 2024-05-09T08:31:21.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.173+0000 7f5b227fc640 1 -- 172.21.15.107:0/3598888502 learned_addr learned my addr 172.21.15.107:0/3598888502 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:21.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.173+0000 7f5b22ffd640 1 --2- 172.21.15.107:0/3598888502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b24102310 0x7f5b24197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:21.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.174+0000 7f5b237fe640 1 --2- 172.21.15.107:0/3598888502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b24107570 0x7f5b2419f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:21.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.174+0000 7f5b227fc640 1 -- 172.21.15.107:0/3598888502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b24102310 msgr2=0x7f5b24197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:21.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.174+0000 7f5b227fc640 1 --2- 172.21.15.107:0/3598888502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b24102310 0x7f5b24197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.174+0000 7f5b227fc640 1 -- 172.21.15.107:0/3598888502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b24107570 msgr2=0x7f5b2419f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:21.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.174+0000 7f5b227fc640 1 --2- 172.21.15.107:0/3598888502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b24107570 0x7f5b2419f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.174+0000 7f5b227fc640 1 -- 172.21.15.107:0/3598888502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b2419f780 con 0x7f5b24104c40 2024-05-09T08:31:21.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.174+0000 7f5b237fe640 1 --2- 172.21.15.107:0/3598888502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b24107570 0x7f5b2419f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:21.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.174+0000 7f5b227fc640 1 --2- 172.21.15.107:0/3598888502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b24104c40 0x7f5b241980a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5b18032fd0 tx=0x7f5b18004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:21.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.174+0000 7f5b0bfff640 1 -- 172.21.15.107:0/3598888502 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1803f030 con 0x7f5b24104c40 2024-05-09T08:31:21.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.174+0000 7f5b293d5640 1 -- 172.21.15.107:0/3598888502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b2419fa10 con 0x7f5b24104c40 2024-05-09T08:31:21.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.174+0000 7f5b0bfff640 1 -- 172.21.15.107:0/3598888502 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b18004490 con 0x7f5b24104c40 2024-05-09T08:31:21.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.174+0000 7f5b0bfff640 1 -- 172.21.15.107:0/3598888502 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b18035070 con 0x7f5b24104c40 2024-05-09T08:31:21.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.174+0000 7f5b293d5640 1 -- 172.21.15.107:0/3598888502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b2419ff00 con 0x7f5b24104c40 2024-05-09T08:31:21.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.175+0000 7f5b293d5640 1 -- 172.21.15.107:0/3598888502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b240fd810 con 0x7f5b24104c40 2024-05-09T08:31:21.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.176+0000 7f5b0bfff640 1 -- 172.21.15.107:0/3598888502 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f5b18042030 con 0x7f5b24104c40 2024-05-09T08:31:21.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.177+0000 7f5b0bfff640 1 --2- 172.21.15.107:0/3598888502 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5b0c0a2b90 0x7f5b0c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:21.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.177+0000 7f5b22ffd640 1 --2- 172.21.15.107:0/3598888502 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5b0c0a2b90 0x7f5b0c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:21.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.177+0000 7f5b0bfff640 1 -- 172.21.15.107:0/3598888502 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5b180ef820 con 0x7f5b24104c40 2024-05-09T08:31:21.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.178+0000 7f5b22ffd640 1 --2- 172.21.15.107:0/3598888502 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5b0c0a2b90 0x7f5b0c0a5050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5b14005c40 tx=0x7f5b1400a590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:21.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.182+0000 7f5b0bfff640 1 -- 172.21.15.107:0/3598888502 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5b180bc540 con 0x7f5b24104c40 2024-05-09T08:31:21.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:21 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3097665343' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-09T08:31:21.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:21 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] up:active 2024-05-09T08:31:21.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:21 smithi107 ceph-mon[30122]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-09T08:31:21.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:21 smithi107 ceph-mon[30122]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-09T08:31:21.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:21 smithi107 ceph-mon[30122]: Cluster is now healthy 2024-05-09T08:31:21.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:21 smithi107 ceph-mon[30122]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-05-09T08:31:21.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:21 smithi107 ceph-mon[30122]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-05-09T08:31:21.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:21 smithi107 ceph-mon[30122]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-05-09T08:31:21.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:21 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1511911944' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-09T08:31:21.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:21 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3097665343' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-09T08:31:21.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:21 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] up:active 2024-05-09T08:31:21.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:21 smithi157 ceph-mon[33614]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-09T08:31:21.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:21 smithi157 ceph-mon[33614]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-09T08:31:21.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:21 smithi157 ceph-mon[33614]: Cluster is now healthy 2024-05-09T08:31:21.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:21 smithi157 ceph-mon[33614]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-05-09T08:31:21.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:21 smithi157 ceph-mon[33614]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-05-09T08:31:21.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:21 smithi157 ceph-mon[33614]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-05-09T08:31:21.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:21 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1511911944' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-09T08:31:21.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:21 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3097665343' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-09T08:31:21.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:21 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] up:active 2024-05-09T08:31:21.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:21 smithi110 ceph-mon[33622]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-09T08:31:21.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:21 smithi110 ceph-mon[33622]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-09T08:31:21.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:21 smithi110 ceph-mon[33622]: Cluster is now healthy 2024-05-09T08:31:21.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:21 smithi110 ceph-mon[33622]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-05-09T08:31:21.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:21 smithi110 ceph-mon[33622]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-05-09T08:31:21.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:21 smithi110 ceph-mon[33622]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-05-09T08:31:21.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:21 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1511911944' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-09T08:31:21.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.380+0000 7f5b293d5640 1 -- 172.21.15.107:0/3598888502 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f5b24061c70 con 0x7f5b0c0a2b90 2024-05-09T08:31:21.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.406+0000 7f5b0bfff640 1 -- 172.21.15.107:0/3598888502 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5b24061c70 con 0x7f5b0c0a2b90 2024-05-09T08:31:21.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.408+0000 7f5b293d5640 1 -- 172.21.15.107:0/3598888502 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5b0c0a2b90 msgr2=0x7f5b0c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:21.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.408+0000 7f5b293d5640 1 --2- 172.21.15.107:0/3598888502 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5b0c0a2b90 0x7f5b0c0a5050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5b14005c40 tx=0x7f5b1400a590 comp rx=0 tx=0).stop 2024-05-09T08:31:21.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.408+0000 7f5b293d5640 1 -- 172.21.15.107:0/3598888502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b24104c40 msgr2=0x7f5b241980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:21.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.408+0000 7f5b293d5640 1 --2- 172.21.15.107:0/3598888502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b24104c40 0x7f5b241980a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5b18032fd0 tx=0x7f5b18004270 comp rx=0 tx=0).stop 2024-05-09T08:31:21.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.408+0000 7f5b293d5640 1 -- 172.21.15.107:0/3598888502 shutdown_connections 2024-05-09T08:31:21.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.408+0000 7f5b293d5640 1 --2- 172.21.15.107:0/3598888502 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5b0c0a2b90 0x7f5b0c0a5050 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.408+0000 7f5b293d5640 1 --2- 172.21.15.107:0/3598888502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b24107570 0x7f5b2419f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.408+0000 7f5b293d5640 1 --2- 172.21.15.107:0/3598888502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b24104c40 0x7f5b241980a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.408+0000 7f5b293d5640 1 --2- 172.21.15.107:0/3598888502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b24102310 0x7f5b24197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.408+0000 7f5b293d5640 1 -- 172.21.15.107:0/3598888502 >> 172.21.15.107:0/3598888502 conn(0x7f5b240fbd20 msgr2=0x7f5b241041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:21.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.408+0000 7f5b293d5640 1 -- 172.21.15.107:0/3598888502 shutdown_connections 2024-05-09T08:31:21.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.408+0000 7f5b293d5640 1 -- 172.21.15.107:0/3598888502 wait complete. 2024-05-09T08:31:21.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-05-09T08:31:21.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.525+0000 7f615424c640 1 -- 172.21.15.107:0/1468608566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c102a70 msgr2=0x7f614c102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:21.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.525+0000 7f615424c640 1 --2- 172.21.15.107:0/1468608566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c102a70 0x7f614c102ef0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f613c009900 tx=0x7f613c02a410 comp rx=0 tx=0).stop 2024-05-09T08:31:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.525+0000 7f615424c640 1 -- 172.21.15.107:0/1468608566 shutdown_connections 2024-05-09T08:31:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.525+0000 7f615424c640 1 --2- 172.21.15.107:0/1468608566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f614c103430 0x7f614c109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.525+0000 7f615424c640 1 --2- 172.21.15.107:0/1468608566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c102a70 0x7f614c102ef0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.525+0000 7f615424c640 1 --2- 172.21.15.107:0/1468608566 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f614c101880 0x7f614c101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.526+0000 7f615424c640 1 -- 172.21.15.107:0/1468608566 >> 172.21.15.107:0/1468608566 conn(0x7f614c0fce10 msgr2=0x7f614c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.526+0000 7f615424c640 1 -- 172.21.15.107:0/1468608566 shutdown_connections 2024-05-09T08:31:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.526+0000 7f615424c640 1 -- 172.21.15.107:0/1468608566 wait complete. 2024-05-09T08:31:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.526+0000 7f615424c640 1 Processor -- start 2024-05-09T08:31:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.526+0000 7f615424c640 1 -- start start 2024-05-09T08:31:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.526+0000 7f615424c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f614c101880 0x7f614c197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.526+0000 7f615424c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f614c102a70 0x7f614c198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.526+0000 7f615424c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c103430 0x7f614c19f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:21.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.526+0000 7f615424c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f614c10d410 con 0x7f614c103430 2024-05-09T08:31:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.526+0000 7f615424c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f614c10d290 con 0x7f614c101880 2024-05-09T08:31:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.526+0000 7f615424c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f614c10d590 con 0x7f614c102a70 2024-05-09T08:31:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.526+0000 7f61527e9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c103430 0x7f614c19f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.526+0000 7f61527e9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c103430 0x7f614c19f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53188/0 (socket says 172.21.15.107:53188) 2024-05-09T08:31:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.526+0000 7f61527e9640 1 -- 172.21.15.107:0/862383188 learned_addr learned my addr 172.21.15.107:0/862383188 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.526+0000 7f61517e7640 1 --2- 172.21.15.107:0/862383188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f614c102a70 0x7f614c198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.527+0000 7f6151fe8640 1 --2- 172.21.15.107:0/862383188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f614c101880 0x7f614c197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.527+0000 7f61527e9640 1 -- 172.21.15.107:0/862383188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f614c102a70 msgr2=0x7f614c198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.527+0000 7f61527e9640 1 --2- 172.21.15.107:0/862383188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f614c102a70 0x7f614c198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.527+0000 7f61527e9640 1 -- 172.21.15.107:0/862383188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f614c101880 msgr2=0x7f614c197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.527+0000 7f61527e9640 1 --2- 172.21.15.107:0/862383188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f614c101880 0x7f614c197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.527+0000 7f61527e9640 1 -- 172.21.15.107:0/862383188 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f614c19f750 con 0x7f614c103430 2024-05-09T08:31:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.527+0000 7f61527e9640 1 --2- 172.21.15.107:0/862383188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c103430 0x7f614c19f0e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f614400b970 tx=0x7f614400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.527+0000 7f6142ffd640 1 -- 172.21.15.107:0/862383188 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6144019070 con 0x7f614c103430 2024-05-09T08:31:21.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.527+0000 7f6142ffd640 1 -- 172.21.15.107:0/862383188 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6144014440 con 0x7f614c103430 2024-05-09T08:31:21.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.527+0000 7f6142ffd640 1 -- 172.21.15.107:0/862383188 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61440075d0 con 0x7f614c103430 2024-05-09T08:31:21.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.527+0000 7f615424c640 1 -- 172.21.15.107:0/862383188 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f614c19fa40 con 0x7f614c103430 2024-05-09T08:31:21.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.527+0000 7f615424c640 1 -- 172.21.15.107:0/862383188 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f614c19ff90 con 0x7f614c103430 2024-05-09T08:31:21.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.528+0000 7f615424c640 1 -- 172.21.15.107:0/862383188 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f614c108c70 con 0x7f614c103430 2024-05-09T08:31:21.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.529+0000 7f6142ffd640 1 -- 172.21.15.107:0/862383188 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f614401c070 con 0x7f614c103430 2024-05-09T08:31:21.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.530+0000 7f6142ffd640 1 --2- 172.21.15.107:0/862383188 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f61380a2b90 0x7f61380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:21.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.530+0000 7f6151fe8640 1 --2- 172.21.15.107:0/862383188 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f61380a2b90 0x7f61380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:21.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.530+0000 7f6142ffd640 1 -- 172.21.15.107:0/862383188 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f61440ce850 con 0x7f614c103430 2024-05-09T08:31:21.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.530+0000 7f6151fe8640 1 --2- 172.21.15.107:0/862383188 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f61380a2b90 0x7f61380a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6148005fd0 tx=0x7f6148005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:21.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.534+0000 7f6142ffd640 1 -- 172.21.15.107:0/862383188 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f614409af20 con 0x7f614c103430 2024-05-09T08:31:21.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.738+0000 7f615424c640 1 -- 172.21.15.107:0/862383188 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f614c061c70 con 0x7f61380a2b90 2024-05-09T08:31:21.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.763+0000 7f6142ffd640 1 -- 172.21.15.107:0/862383188 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f614c061c70 con 0x7f61380a2b90 2024-05-09T08:31:21.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.765+0000 7f615424c640 1 -- 172.21.15.107:0/862383188 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f61380a2b90 msgr2=0x7f61380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:21.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.765+0000 7f615424c640 1 --2- 172.21.15.107:0/862383188 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f61380a2b90 0x7f61380a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6148005fd0 tx=0x7f6148005850 comp rx=0 tx=0).stop 2024-05-09T08:31:21.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.765+0000 7f615424c640 1 -- 172.21.15.107:0/862383188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c103430 msgr2=0x7f614c19f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:21.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.765+0000 7f615424c640 1 --2- 172.21.15.107:0/862383188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c103430 0x7f614c19f0e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f614400b970 tx=0x7f614400be30 comp rx=0 tx=0).stop 2024-05-09T08:31:21.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.765+0000 7f615424c640 1 -- 172.21.15.107:0/862383188 shutdown_connections 2024-05-09T08:31:21.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.765+0000 7f615424c640 1 --2- 172.21.15.107:0/862383188 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f61380a2b90 0x7f61380a5050 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.766+0000 7f615424c640 1 --2- 172.21.15.107:0/862383188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c103430 0x7f614c19f0e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.766+0000 7f615424c640 1 --2- 172.21.15.107:0/862383188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f614c102a70 0x7f614c198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.766+0000 7f615424c640 1 --2- 172.21.15.107:0/862383188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f614c101880 0x7f614c197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.766+0000 7f615424c640 1 -- 172.21.15.107:0/862383188 >> 172.21.15.107:0/862383188 conn(0x7f614c0fce10 msgr2=0x7f614c105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:21.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.766+0000 7f615424c640 1 -- 172.21.15.107:0/862383188 shutdown_connections 2024-05-09T08:31:21.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.766+0000 7f615424c640 1 -- 172.21.15.107:0/862383188 wait complete. 2024-05-09T08:31:21.775 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:31:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:31:21.385+0000 7f52a43ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:31:21.775 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:31:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:31:21.385+0000 7f52a43ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:31:21.775 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:31:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:31:21.385+0000 7f52a43ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:31:21.775 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:31:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:31:21.385+0000 7f52a43ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:31:21.775 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:31:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:31:21.385+0000 7f52a43ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:31:21.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-05-09T08:31:21.844 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:31:21 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-i[64360]: 2024-05-09T08:31:21.515+0000 7f98e1dec700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-05-09T08:31:21.844 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:31:21 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-i[64360]: 2024-05-09T08:31:21.515+0000 7f98e1dec700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-05-09T08:31:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.883+0000 7f95708f5640 1 -- 172.21.15.107:0/3972529122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f956c1034d0 msgr2=0x7f956c109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.883+0000 7f95708f5640 1 --2- 172.21.15.107:0/3972529122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f956c1034d0 0x7f956c109d60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9564009cd0 tx=0x7f956402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.883+0000 7f95708f5640 1 -- 172.21.15.107:0/3972529122 shutdown_connections 2024-05-09T08:31:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.883+0000 7f95708f5640 1 --2- 172.21.15.107:0/3972529122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f956c1034d0 0x7f956c109d60 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.883+0000 7f95708f5640 1 --2- 172.21.15.107:0/3972529122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f956c102b10 0x7f956c102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.883+0000 7f95708f5640 1 --2- 172.21.15.107:0/3972529122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f956c0ffb40 0x7f956c0fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.883+0000 7f95708f5640 1 -- 172.21.15.107:0/3972529122 >> 172.21.15.107:0/3972529122 conn(0x7f956c0fb470 msgr2=0x7f956c0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.883+0000 7f95708f5640 1 -- 172.21.15.107:0/3972529122 shutdown_connections 2024-05-09T08:31:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.883+0000 7f95708f5640 1 -- 172.21.15.107:0/3972529122 wait complete. 2024-05-09T08:31:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.884+0000 7f95708f5640 1 Processor -- start 2024-05-09T08:31:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.884+0000 7f95708f5640 1 -- start start 2024-05-09T08:31:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.884+0000 7f95708f5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f956c0ffb40 0x7f956c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.884+0000 7f95708f5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f956c102b10 0x7f956c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.884+0000 7f95708f5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f956c1034d0 0x7f956c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.884+0000 7f95708f5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f956c10d3a0 con 0x7f956c102b10 2024-05-09T08:31:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.884+0000 7f95708f5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f956c10d220 con 0x7f956c1034d0 2024-05-09T08:31:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.884+0000 7f95708f5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f956c10d520 con 0x7f956c0ffb40 2024-05-09T08:31:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.884+0000 7f9569d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f956c102b10 0x7f956c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.885+0000 7f9569d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f956c102b10 0x7f956c197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53210/0 (socket says 172.21.15.107:53210) 2024-05-09T08:31:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.885+0000 7f956a59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f956c0ffb40 0x7f956c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.885+0000 7f9569d9b640 1 -- 172.21.15.107:0/3440073901 learned_addr learned my addr 172.21.15.107:0/3440073901 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.885+0000 7f956ad9d640 1 --2- 172.21.15.107:0/3440073901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f956c1034d0 0x7f956c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.885+0000 7f9569d9b640 1 -- 172.21.15.107:0/3440073901 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f956c0ffb40 msgr2=0x7f956c197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.885+0000 7f9569d9b640 1 --2- 172.21.15.107:0/3440073901 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f956c0ffb40 0x7f956c197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.885+0000 7f9569d9b640 1 -- 172.21.15.107:0/3440073901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f956c1034d0 msgr2=0x7f956c19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.885+0000 7f9569d9b640 1 --2- 172.21.15.107:0/3440073901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f956c1034d0 0x7f956c19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.885+0000 7f9569d9b640 1 -- 172.21.15.107:0/3440073901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956c19f6a0 con 0x7f956c102b10 2024-05-09T08:31:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.885+0000 7f956ad9d640 1 --2- 172.21.15.107:0/3440073901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f956c1034d0 0x7f956c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.885+0000 7f9569d9b640 1 --2- 172.21.15.107:0/3440073901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f956c102b10 0x7f956c197fc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f956000e9f0 tx=0x7f956000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.885+0000 7f955b7fe640 1 -- 172.21.15.107:0/3440073901 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956000cd80 con 0x7f956c102b10 2024-05-09T08:31:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.885+0000 7f955b7fe640 1 -- 172.21.15.107:0/3440073901 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f95600108e0 con 0x7f956c102b10 2024-05-09T08:31:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.885+0000 7f95708f5640 1 -- 172.21.15.107:0/3440073901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f956c19f990 con 0x7f956c102b10 2024-05-09T08:31:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.885+0000 7f955b7fe640 1 -- 172.21.15.107:0/3440073901 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9560018920 con 0x7f956c102b10 2024-05-09T08:31:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.885+0000 7f95708f5640 1 -- 172.21.15.107:0/3440073901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f956c19ff30 con 0x7f956c102b10 2024-05-09T08:31:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.886+0000 7f95708f5640 1 -- 172.21.15.107:0/3440073901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f956c108c60 con 0x7f956c102b10 2024-05-09T08:31:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.887+0000 7f955b7fe640 1 -- 172.21.15.107:0/3440073901 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9560010a90 con 0x7f956c102b10 2024-05-09T08:31:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.888+0000 7f955b7fe640 1 --2- 172.21.15.107:0/3440073901 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f95540a2c10 0x7f95540a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.888+0000 7f956a59c640 1 --2- 172.21.15.107:0/3440073901 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f95540a2c10 0x7f95540a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:21.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.888+0000 7f955b7fe640 1 -- 172.21.15.107:0/3440073901 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f95600cf510 con 0x7f956c102b10 2024-05-09T08:31:21.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.888+0000 7f956a59c640 1 --2- 172.21.15.107:0/3440073901 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f95540a2c10 0x7f95540a50d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f955c005fd0 tx=0x7f955c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:21.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:21.893+0000 7f955b7fe640 1 -- 172.21.15.107:0/3440073901 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f956009c230 con 0x7f956c102b10 2024-05-09T08:31:22.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.069+0000 7f955b7fe640 1 -- 172.21.15.107:0/3440073901 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f95600981a0 con 0x7f956c102b10 2024-05-09T08:31:22.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.103+0000 7f95708f5640 1 -- 172.21.15.107:0/3440073901 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f956c061d00 con 0x7f95540a2c10 2024-05-09T08:31:22.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.110+0000 7f955b7fe640 1 -- 172.21.15.107:0/3440073901 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f956c061d00 con 0x7f95540a2c10 2024-05-09T08:31:22.109 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/qa/sv_0/53dbbdc5-d77e-40a8-9491-15f95d558fdc 2024-05-09T08:31:22.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.112+0000 7f95708f5640 1 -- 172.21.15.107:0/3440073901 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f95540a2c10 msgr2=0x7f95540a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:22.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.112+0000 7f95708f5640 1 --2- 172.21.15.107:0/3440073901 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f95540a2c10 0x7f95540a50d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f955c005fd0 tx=0x7f955c00b410 comp rx=0 tx=0).stop 2024-05-09T08:31:22.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.112+0000 7f95708f5640 1 -- 172.21.15.107:0/3440073901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f956c102b10 msgr2=0x7f956c197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:22.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.112+0000 7f95708f5640 1 --2- 172.21.15.107:0/3440073901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f956c102b10 0x7f956c197fc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f956000e9f0 tx=0x7f956000eeb0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.112+0000 7f95708f5640 1 -- 172.21.15.107:0/3440073901 shutdown_connections 2024-05-09T08:31:22.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.112+0000 7f95708f5640 1 --2- 172.21.15.107:0/3440073901 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f95540a2c10 0x7f95540a50d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.112+0000 7f95708f5640 1 --2- 172.21.15.107:0/3440073901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f956c1034d0 0x7f956c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.112+0000 7f95708f5640 1 --2- 172.21.15.107:0/3440073901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f956c102b10 0x7f956c197fc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.112+0000 7f95708f5640 1 --2- 172.21.15.107:0/3440073901 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f956c0ffb40 0x7f956c197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.112+0000 7f95708f5640 1 -- 172.21.15.107:0/3440073901 >> 172.21.15.107:0/3440073901 conn(0x7f956c0fb470 msgr2=0x7f956c105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.112+0000 7f95708f5640 1 -- 172.21.15.107:0/3440073901 shutdown_connections 2024-05-09T08:31:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.112+0000 7f95708f5640 1 -- 172.21.15.107:0/3440073901 wait complete. 2024-05-09T08:31:22.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-05-09T08:31:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.228+0000 7f444755c640 1 -- 172.21.15.107:0/2431998382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4440100790 msgr2=0x7f4440100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.228+0000 7f444755c640 1 --2- 172.21.15.107:0/2431998382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4440100790 0x7f4440100bb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4430009900 tx=0x7f443002a410 comp rx=0 tx=0).stop 2024-05-09T08:31:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.228+0000 7f444755c640 1 -- 172.21.15.107:0/2431998382 shutdown_connections 2024-05-09T08:31:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.228+0000 7f444755c640 1 --2- 172.21.15.107:0/2431998382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4440102340 0x7f4440108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.228+0000 7f444755c640 1 --2- 172.21.15.107:0/2431998382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4440101980 0x7f4440101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.228+0000 7f444755c640 1 --2- 172.21.15.107:0/2431998382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4440100790 0x7f4440100bb0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.228+0000 7f444755c640 1 -- 172.21.15.107:0/2431998382 >> 172.21.15.107:0/2431998382 conn(0x7f44400fbd00 msgr2=0x7f44400fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.228+0000 7f444755c640 1 -- 172.21.15.107:0/2431998382 shutdown_connections 2024-05-09T08:31:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.228+0000 7f444755c640 1 -- 172.21.15.107:0/2431998382 wait complete. 2024-05-09T08:31:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f444755c640 1 Processor -- start 2024-05-09T08:31:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f444755c640 1 -- start start 2024-05-09T08:31:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f444755c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4440100790 0x7f4440197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f444755c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4440101980 0x7f4440198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f444755c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4440102340 0x7f444019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f444755c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f444010c2c0 con 0x7f4440101980 2024-05-09T08:31:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f444755c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f444010c140 con 0x7f4440102340 2024-05-09T08:31:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f444755c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f444010c440 con 0x7f4440100790 2024-05-09T08:31:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f4444af7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4440101980 0x7f4440198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:22.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f4444af7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4440101980 0x7f4440198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53240/0 (socket says 172.21.15.107:53240) 2024-05-09T08:31:22.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f4444af7640 1 -- 172.21.15.107:0/1068800153 learned_addr learned my addr 172.21.15.107:0/1068800153 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:22.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f4445af9640 1 --2- 172.21.15.107:0/1068800153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4440102340 0x7f444019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:22.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f44452f8640 1 --2- 172.21.15.107:0/1068800153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4440100790 0x7f4440197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:22.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f4444af7640 1 -- 172.21.15.107:0/1068800153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4440100790 msgr2=0x7f4440197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:22.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f4444af7640 1 --2- 172.21.15.107:0/1068800153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4440100790 0x7f4440197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f4444af7640 1 -- 172.21.15.107:0/1068800153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4440102340 msgr2=0x7f444019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:22.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f4444af7640 1 --2- 172.21.15.107:0/1068800153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4440102340 0x7f444019f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.229+0000 7f4444af7640 1 -- 172.21.15.107:0/1068800153 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f444019f710 con 0x7f4440101980 2024-05-09T08:31:22.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.230+0000 7f4444af7640 1 --2- 172.21.15.107:0/1068800153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4440101980 0x7f4440198030 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f443c00d950 tx=0x7f443c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:22.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.230+0000 7f44367fc640 1 -- 172.21.15.107:0/1068800153 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443c017070 con 0x7f4440101980 2024-05-09T08:31:22.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.230+0000 7f44367fc640 1 -- 172.21.15.107:0/1068800153 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f443c009c30 con 0x7f4440101980 2024-05-09T08:31:22.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.230+0000 7f444755c640 1 -- 172.21.15.107:0/1068800153 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f444019fa00 con 0x7f4440101980 2024-05-09T08:31:22.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.230+0000 7f44367fc640 1 -- 172.21.15.107:0/1068800153 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443c00f620 con 0x7f4440101980 2024-05-09T08:31:22.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.230+0000 7f444755c640 1 -- 172.21.15.107:0/1068800153 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f444019ffa0 con 0x7f4440101980 2024-05-09T08:31:22.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.231+0000 7f444755c640 1 -- 172.21.15.107:0/1068800153 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44401a01d0 con 0x7f4440101980 2024-05-09T08:31:22.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.232+0000 7f44367fc640 1 -- 172.21.15.107:0/1068800153 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f443c0109a0 con 0x7f4440101980 2024-05-09T08:31:22.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.233+0000 7f44367fc640 1 --2- 172.21.15.107:0/1068800153 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f442c0a2ca0 0x7f442c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:22.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.233+0000 7f44452f8640 1 --2- 172.21.15.107:0/1068800153 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f442c0a2ca0 0x7f442c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:22.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.233+0000 7f44367fc640 1 -- 172.21.15.107:0/1068800153 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f443c0cef20 con 0x7f4440101980 2024-05-09T08:31:22.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.233+0000 7f44452f8640 1 --2- 172.21.15.107:0/1068800153 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f442c0a2ca0 0x7f442c0a5160 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4430004770 tx=0x7f4430000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:22.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.237+0000 7f44367fc640 1 -- 172.21.15.107:0/1068800153 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f443c009de0 con 0x7f4440101980 2024-05-09T08:31:22.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:22 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1511911944' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-09T08:31:22.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:22 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] up:active 2024-05-09T08:31:22.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:22 smithi157 ceph-mon[33614]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-05-09T08:31:22.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:22 smithi157 ceph-mon[33614]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:31:22.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:22 smithi157 ceph-mon[33614]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:31:22.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:22 smithi157 ceph-mon[33614]: pgmap v302: 129 pgs: 129 active+clean; 584 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.6 KiB/s wr, 3 op/s 2024-05-09T08:31:22.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:22 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1511911944' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-09T08:31:22.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:22 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] up:active 2024-05-09T08:31:22.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:22 smithi110 ceph-mon[33622]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-05-09T08:31:22.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:22 smithi110 ceph-mon[33622]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:31:22.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:22 smithi110 ceph-mon[33622]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:31:22.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:22 smithi110 ceph-mon[33622]: pgmap v302: 129 pgs: 129 active+clean; 584 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.6 KiB/s wr, 3 op/s 2024-05-09T08:31:22.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.440+0000 7f444755c640 1 -- 172.21.15.107:0/1068800153 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f444006a570 con 0x7f442c0a2ca0 2024-05-09T08:31:22.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.465+0000 7f44367fc640 1 -- 172.21.15.107:0/1068800153 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f444006a570 con 0x7f442c0a2ca0 2024-05-09T08:31:22.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.467+0000 7f444755c640 1 -- 172.21.15.107:0/1068800153 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f442c0a2ca0 msgr2=0x7f442c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:22.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.467+0000 7f444755c640 1 --2- 172.21.15.107:0/1068800153 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f442c0a2ca0 0x7f442c0a5160 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4430004770 tx=0x7f4430000bc0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.467+0000 7f444755c640 1 -- 172.21.15.107:0/1068800153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4440101980 msgr2=0x7f4440198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:22.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.467+0000 7f444755c640 1 --2- 172.21.15.107:0/1068800153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4440101980 0x7f4440198030 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f443c00d950 tx=0x7f443c00de10 comp rx=0 tx=0).stop 2024-05-09T08:31:22.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.467+0000 7f444755c640 1 -- 172.21.15.107:0/1068800153 shutdown_connections 2024-05-09T08:31:22.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.467+0000 7f444755c640 1 --2- 172.21.15.107:0/1068800153 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f442c0a2ca0 0x7f442c0a5160 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.467+0000 7f444755c640 1 --2- 172.21.15.107:0/1068800153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4440102340 0x7f444019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.467+0000 7f444755c640 1 --2- 172.21.15.107:0/1068800153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4440101980 0x7f4440198030 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.467+0000 7f444755c640 1 --2- 172.21.15.107:0/1068800153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4440100790 0x7f4440197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.467+0000 7f444755c640 1 -- 172.21.15.107:0/1068800153 >> 172.21.15.107:0/1068800153 conn(0x7f44400fbd00 msgr2=0x7f4440069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:22.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.468+0000 7f444755c640 1 -- 172.21.15.107:0/1068800153 shutdown_connections 2024-05-09T08:31:22.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.468+0000 7f444755c640 1 -- 172.21.15.107:0/1068800153 wait complete. 2024-05-09T08:31:22.476 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:22 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1511911944' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-09T08:31:22.477 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:22 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] up:active 2024-05-09T08:31:22.477 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:22 smithi107 ceph-mon[30122]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-05-09T08:31:22.477 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:22 smithi107 ceph-mon[30122]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:31:22.477 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:22 smithi107 ceph-mon[30122]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:31:22.477 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:22 smithi107 ceph-mon[30122]: pgmap v302: 129 pgs: 129 active+clean; 584 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.6 KiB/s wr, 3 op/s 2024-05-09T08:31:22.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-05-09T08:31:22.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.584+0000 7fb559a41640 1 -- 172.21.15.107:0/505106685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb554102e80 msgr2=0x7fb5541032a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:22.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.584+0000 7fb559a41640 1 --2- 172.21.15.107:0/505106685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb554102e80 0x7fb5541032a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb544009900 tx=0x7fb54402a410 comp rx=0 tx=0).stop 2024-05-09T08:31:22.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.584+0000 7fb559a41640 1 -- 172.21.15.107:0/505106685 shutdown_connections 2024-05-09T08:31:22.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.584+0000 7fb559a41640 1 --2- 172.21.15.107:0/505106685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5541002b0 0x7fb554109e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.584+0000 7fb559a41640 1 --2- 172.21.15.107:0/505106685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5540ff8f0 0x7fb5540ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.584+0000 7fb559a41640 1 --2- 172.21.15.107:0/505106685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb554102e80 0x7fb5541032a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.584+0000 7fb559a41640 1 -- 172.21.15.107:0/505106685 >> 172.21.15.107:0/505106685 conn(0x7fb5540fb470 msgr2=0x7fb5540fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:22.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.584+0000 7fb559a41640 1 -- 172.21.15.107:0/505106685 shutdown_connections 2024-05-09T08:31:22.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.584+0000 7fb559a41640 1 -- 172.21.15.107:0/505106685 wait complete. 2024-05-09T08:31:22.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb559a41640 1 Processor -- start 2024-05-09T08:31:22.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb559a41640 1 -- start start 2024-05-09T08:31:22.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb559a41640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5540ff8f0 0x7fb554197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:22.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb559a41640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5541002b0 0x7fb554198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:22.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb559a41640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb554102e80 0x7fb55419f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:22.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb559a41640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb55410d3b0 con 0x7fb554102e80 2024-05-09T08:31:22.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb559a41640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb55410d230 con 0x7fb5540ff8f0 2024-05-09T08:31:22.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb559a41640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb55410d530 con 0x7fb5541002b0 2024-05-09T08:31:22.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb5537fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb554102e80 0x7fb55419f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:22.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb5537fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb554102e80 0x7fb55419f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53256/0 (socket says 172.21.15.107:53256) 2024-05-09T08:31:22.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb5537fe640 1 -- 172.21.15.107:0/1469444788 learned_addr learned my addr 172.21.15.107:0/1469444788 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:22.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb552ffd640 1 --2- 172.21.15.107:0/1469444788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5540ff8f0 0x7fb554197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:22.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb5527fc640 1 --2- 172.21.15.107:0/1469444788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5541002b0 0x7fb554198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:22.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb5537fe640 1 -- 172.21.15.107:0/1469444788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5541002b0 msgr2=0x7fb554198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:22.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb5537fe640 1 --2- 172.21.15.107:0/1469444788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5541002b0 0x7fb554198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb5537fe640 1 -- 172.21.15.107:0/1469444788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5540ff8f0 msgr2=0x7fb554197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:22.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb5537fe640 1 --2- 172.21.15.107:0/1469444788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5540ff8f0 0x7fb554197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb5537fe640 1 -- 172.21.15.107:0/1469444788 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb55419f750 con 0x7fb554102e80 2024-05-09T08:31:22.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.585+0000 7fb5537fe640 1 --2- 172.21.15.107:0/1469444788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb554102e80 0x7fb55419f0e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00cad0 tx=0x7fb54c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:22.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.586+0000 7fb558a3f640 1 -- 172.21.15.107:0/1469444788 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c00ae20 con 0x7fb554102e80 2024-05-09T08:31:22.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.586+0000 7fb558a3f640 1 -- 172.21.15.107:0/1469444788 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb54c005020 con 0x7fb554102e80 2024-05-09T08:31:22.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.586+0000 7fb558a3f640 1 -- 172.21.15.107:0/1469444788 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c00f6b0 con 0x7fb554102e80 2024-05-09T08:31:22.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.586+0000 7fb559a41640 1 -- 172.21.15.107:0/1469444788 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb55419fa40 con 0x7fb554102e80 2024-05-09T08:31:22.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.586+0000 7fb559a41640 1 -- 172.21.15.107:0/1469444788 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb55419ff40 con 0x7fb554102e80 2024-05-09T08:31:22.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.587+0000 7fb559a41640 1 -- 172.21.15.107:0/1469444788 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb554108e30 con 0x7fb554102e80 2024-05-09T08:31:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.588+0000 7fb558a3f640 1 -- 172.21.15.107:0/1469444788 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fb54c0051d0 con 0x7fb554102e80 2024-05-09T08:31:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.588+0000 7fb558a3f640 1 --2- 172.21.15.107:0/1469444788 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb53c0a2c20 0x7fb53c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.588+0000 7fb552ffd640 1 --2- 172.21.15.107:0/1469444788 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb53c0a2c20 0x7fb53c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.588+0000 7fb558a3f640 1 -- 172.21.15.107:0/1469444788 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb54c0cf4b0 con 0x7fb554102e80 2024-05-09T08:31:22.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.589+0000 7fb552ffd640 1 --2- 172.21.15.107:0/1469444788 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb53c0a2c20 0x7fb53c0a50e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb544004770 tx=0x7fb544000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:22.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.593+0000 7fb558a3f640 1 -- 172.21.15.107:0/1469444788 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb54c014030 con 0x7fb554102e80 2024-05-09T08:31:22.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.795+0000 7fb559a41640 1 -- 172.21.15.107:0/1469444788 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fb554061c70 con 0x7fb53c0a2c20 2024-05-09T08:31:22.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.800+0000 7fb558a3f640 1 -- 172.21.15.107:0/1469444788 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fb554061c70 con 0x7fb53c0a2c20 2024-05-09T08:31:22.799 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/qa/sv_1/66dd6095-6618-47bb-a521-e1dceb88764f 2024-05-09T08:31:22.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.802+0000 7fb559a41640 1 -- 172.21.15.107:0/1469444788 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb53c0a2c20 msgr2=0x7fb53c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.802+0000 7fb559a41640 1 --2- 172.21.15.107:0/1469444788 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb53c0a2c20 0x7fb53c0a50e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb544004770 tx=0x7fb544000bc0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.802+0000 7fb559a41640 1 -- 172.21.15.107:0/1469444788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb554102e80 msgr2=0x7fb55419f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.802+0000 7fb559a41640 1 --2- 172.21.15.107:0/1469444788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb554102e80 0x7fb55419f0e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00cad0 tx=0x7fb54c00cf90 comp rx=0 tx=0).stop 2024-05-09T08:31:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.802+0000 7fb559a41640 1 -- 172.21.15.107:0/1469444788 shutdown_connections 2024-05-09T08:31:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.802+0000 7fb559a41640 1 --2- 172.21.15.107:0/1469444788 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb53c0a2c20 0x7fb53c0a50e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.802+0000 7fb559a41640 1 --2- 172.21.15.107:0/1469444788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb554102e80 0x7fb55419f0e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.802+0000 7fb559a41640 1 --2- 172.21.15.107:0/1469444788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5541002b0 0x7fb554198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.802+0000 7fb559a41640 1 --2- 172.21.15.107:0/1469444788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5540ff8f0 0x7fb554197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.802+0000 7fb559a41640 1 -- 172.21.15.107:0/1469444788 >> 172.21.15.107:0/1469444788 conn(0x7fb5540fb470 msgr2=0x7fb554106000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.802+0000 7fb559a41640 1 -- 172.21.15.107:0/1469444788 shutdown_connections 2024-05-09T08:31:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.802+0000 7fb559a41640 1 -- 172.21.15.107:0/1469444788 wait complete. 2024-05-09T08:31:22.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:31:22.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.914+0000 7fde48c38640 1 -- 172.21.15.107:0/1742578590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde44100770 msgr2=0x7fde44100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:22.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.914+0000 7fde48c38640 1 --2- 172.21.15.107:0/1742578590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde44100770 0x7fde44100b90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fde34009ca0 tx=0x7fde3402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.914+0000 7fde48c38640 1 -- 172.21.15.107:0/1742578590 shutdown_connections 2024-05-09T08:31:22.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.914+0000 7fde48c38640 1 --2- 172.21.15.107:0/1742578590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde44102320 0x7fde44108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.914+0000 7fde48c38640 1 --2- 172.21.15.107:0/1742578590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde44101960 0x7fde44101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.914+0000 7fde48c38640 1 --2- 172.21.15.107:0/1742578590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde44100770 0x7fde44100b90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.914+0000 7fde48c38640 1 -- 172.21.15.107:0/1742578590 >> 172.21.15.107:0/1742578590 conn(0x7fde440fbce0 msgr2=0x7fde440fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:22.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.914+0000 7fde48c38640 1 -- 172.21.15.107:0/1742578590 shutdown_connections 2024-05-09T08:31:22.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.914+0000 7fde48c38640 1 -- 172.21.15.107:0/1742578590 wait complete. 2024-05-09T08:31:22.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.914+0000 7fde48c38640 1 Processor -- start 2024-05-09T08:31:22.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.914+0000 7fde48c38640 1 -- start start 2024-05-09T08:31:22.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde48c38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde44100770 0x7fde44196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:22.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde48c38640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde44101960 0x7fde44196f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:22.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde48c38640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde44102320 0x7fde4419dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:22.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde48c38640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fde4410c2e0 con 0x7fde44100770 2024-05-09T08:31:22.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde48c38640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fde4410c160 con 0x7fde44102320 2024-05-09T08:31:22.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde48c38640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fde4410c460 con 0x7fde44101960 2024-05-09T08:31:22.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde4259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde44100770 0x7fde44196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:22.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde4259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde44100770 0x7fde44196a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53280/0 (socket says 172.21.15.107:53280) 2024-05-09T08:31:22.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde4259c640 1 -- 172.21.15.107:0/1217040889 learned_addr learned my addr 172.21.15.107:0/1217040889 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:22.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde41d9b640 1 --2- 172.21.15.107:0/1217040889 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde44101960 0x7fde44196f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:22.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde4259c640 1 -- 172.21.15.107:0/1217040889 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde44101960 msgr2=0x7fde44196f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:22.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde4259c640 1 --2- 172.21.15.107:0/1217040889 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde44101960 0x7fde44196f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde4259c640 1 -- 172.21.15.107:0/1217040889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde44102320 msgr2=0x7fde4419dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:22.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde4259c640 1 --2- 172.21.15.107:0/1217040889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde44102320 0x7fde4419dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:22.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde4259c640 1 -- 172.21.15.107:0/1217040889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde4419e660 con 0x7fde44100770 2024-05-09T08:31:22.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde4259c640 1 --2- 172.21.15.107:0/1217040889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde44100770 0x7fde44196a40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fde34000f80 tx=0x7fde34003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:22.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde3b7fe640 1 -- 172.21.15.107:0/1217040889 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde34004240 con 0x7fde44100770 2024-05-09T08:31:22.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde3b7fe640 1 -- 172.21.15.107:0/1217040889 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde34003e10 con 0x7fde44100770 2024-05-09T08:31:22.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde48c38640 1 -- 172.21.15.107:0/1217040889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde4419e8f0 con 0x7fde44100770 2024-05-09T08:31:22.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde3b7fe640 1 -- 172.21.15.107:0/1217040889 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde34030cd0 con 0x7fde44100770 2024-05-09T08:31:22.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.915+0000 7fde48c38640 1 -- 172.21.15.107:0/1217040889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde4419ee40 con 0x7fde44100770 2024-05-09T08:31:22.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.916+0000 7fde48c38640 1 -- 172.21.15.107:0/1217040889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde44107bb0 con 0x7fde44100770 2024-05-09T08:31:22.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.917+0000 7fde3b7fe640 1 -- 172.21.15.107:0/1217040889 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fde340043e0 con 0x7fde44100770 2024-05-09T08:31:22.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.918+0000 7fde3b7fe640 1 --2- 172.21.15.107:0/1217040889 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fde300a2c20 0x7fde300a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:22.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.918+0000 7fde41d9b640 1 --2- 172.21.15.107:0/1217040889 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fde300a2c20 0x7fde300a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:22.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.918+0000 7fde3b7fe640 1 -- 172.21.15.107:0/1217040889 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fde340f0870 con 0x7fde44100770 2024-05-09T08:31:22.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.918+0000 7fde41d9b640 1 --2- 172.21.15.107:0/1217040889 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fde300a2c20 0x7fde300a50e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fde44197f50 tx=0x7fde2c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:22.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:22.922+0000 7fde3b7fe640 1 -- 172.21.15.107:0/1217040889 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fde340bd540 con 0x7fde44100770 2024-05-09T08:31:23.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.140+0000 7fde48c38640 1 -- 172.21.15.107:0/1217040889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde4402d1c0 con 0x7fde44100770 2024-05-09T08:31:23.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.141+0000 7fde3b7fe640 1 -- 172.21.15.107:0/1217040889 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11887 (secure 0 0 0) 0x7fde340bce40 con 0x7fde44100770 2024-05-09T08:31:23.140 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:23.140 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:31:21.056050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24478,24499]},"id":1}]} 2024-05-09T08:31:23.141 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 24 2024-05-09T08:31:23.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.143+0000 7fde48c38640 1 -- 172.21.15.107:0/1217040889 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fde300a2c20 msgr2=0x7fde300a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:23.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.143+0000 7fde48c38640 1 --2- 172.21.15.107:0/1217040889 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fde300a2c20 0x7fde300a50e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fde44197f50 tx=0x7fde2c005f90 comp rx=0 tx=0).stop 2024-05-09T08:31:23.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.143+0000 7fde48c38640 1 -- 172.21.15.107:0/1217040889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde44100770 msgr2=0x7fde44196a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:23.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.143+0000 7fde48c38640 1 --2- 172.21.15.107:0/1217040889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde44100770 0x7fde44196a40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fde34000f80 tx=0x7fde34003c20 comp rx=0 tx=0).stop 2024-05-09T08:31:23.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.143+0000 7fde48c38640 1 -- 172.21.15.107:0/1217040889 shutdown_connections 2024-05-09T08:31:23.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.143+0000 7fde48c38640 1 --2- 172.21.15.107:0/1217040889 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fde300a2c20 0x7fde300a50e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.143+0000 7fde48c38640 1 --2- 172.21.15.107:0/1217040889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde44102320 0x7fde4419dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.143+0000 7fde48c38640 1 --2- 172.21.15.107:0/1217040889 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde44101960 0x7fde44196f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.143+0000 7fde48c38640 1 --2- 172.21.15.107:0/1217040889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde44100770 0x7fde44196a40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.143+0000 7fde48c38640 1 -- 172.21.15.107:0/1217040889 >> 172.21.15.107:0/1217040889 conn(0x7fde440fbce0 msgr2=0x7fde44104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:23.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.143+0000 7fde48c38640 1 -- 172.21.15.107:0/1217040889 shutdown_connections 2024-05-09T08:31:23.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.143+0000 7fde48c38640 1 -- 172.21.15.107:0/1217040889 wait complete. 2024-05-09T08:31:23.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:31:23.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.254+0000 7f411f6cd640 1 -- 172.21.15.107:0/4227716746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4118102a70 msgr2=0x7f4118102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:23.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.254+0000 7f411f6cd640 1 --2- 172.21.15.107:0/4227716746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4118102a70 0x7f4118102ef0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4108009900 tx=0x7f410802a410 comp rx=0 tx=0).stop 2024-05-09T08:31:23.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.254+0000 7f411f6cd640 1 -- 172.21.15.107:0/4227716746 shutdown_connections 2024-05-09T08:31:23.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.254+0000 7f411f6cd640 1 --2- 172.21.15.107:0/4227716746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4118103430 0x7f4118109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.254+0000 7f411f6cd640 1 --2- 172.21.15.107:0/4227716746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4118102a70 0x7f4118102ef0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.254+0000 7f411f6cd640 1 --2- 172.21.15.107:0/4227716746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4118101880 0x7f4118101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.254+0000 7f411f6cd640 1 -- 172.21.15.107:0/4227716746 >> 172.21.15.107:0/4227716746 conn(0x7f41180fce10 msgr2=0x7f41180ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:23.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.254+0000 7f411f6cd640 1 -- 172.21.15.107:0/4227716746 shutdown_connections 2024-05-09T08:31:23.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.254+0000 7f411f6cd640 1 -- 172.21.15.107:0/4227716746 wait complete. 2024-05-09T08:31:23.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411f6cd640 1 Processor -- start 2024-05-09T08:31:23.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411f6cd640 1 -- start start 2024-05-09T08:31:23.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411f6cd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4118101880 0x7f4118197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:23.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411f6cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4118102a70 0x7f4118197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:23.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411f6cd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4118103430 0x7f411819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:23.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411f6cd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f411810d3b0 con 0x7f4118102a70 2024-05-09T08:31:23.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411f6cd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f411810d230 con 0x7f4118101880 2024-05-09T08:31:23.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411f6cd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f411810d530 con 0x7f4118103430 2024-05-09T08:31:23.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411cc68640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4118102a70 0x7f4118197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:23.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411cc68640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4118102a70 0x7f4118197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53288/0 (socket says 172.21.15.107:53288) 2024-05-09T08:31:23.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411cc68640 1 -- 172.21.15.107:0/1452960864 learned_addr learned my addr 172.21.15.107:0/1452960864 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:23.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411d469640 1 --2- 172.21.15.107:0/1452960864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4118101880 0x7f4118197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:23.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411dc6a640 1 --2- 172.21.15.107:0/1452960864 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4118103430 0x7f411819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:23.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411cc68640 1 -- 172.21.15.107:0/1452960864 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4118103430 msgr2=0x7f411819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:23.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411cc68640 1 --2- 172.21.15.107:0/1452960864 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4118103430 0x7f411819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411cc68640 1 -- 172.21.15.107:0/1452960864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4118101880 msgr2=0x7f4118197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:23.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411cc68640 1 --2- 172.21.15.107:0/1452960864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4118101880 0x7f4118197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411cc68640 1 -- 172.21.15.107:0/1452960864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f411819f6d0 con 0x7f4118102a70 2024-05-09T08:31:23.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.255+0000 7f411dc6a640 1 --2- 172.21.15.107:0/1452960864 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4118103430 0x7f411819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:23.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.256+0000 7f411cc68640 1 --2- 172.21.15.107:0/1452960864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4118102a70 0x7f4118197ff0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4108004770 tx=0x7f41080042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:23.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.256+0000 7f410e7fc640 1 -- 172.21.15.107:0/1452960864 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410803f030 con 0x7f4118102a70 2024-05-09T08:31:23.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.256+0000 7f410e7fc640 1 -- 172.21.15.107:0/1452960864 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4108003cb0 con 0x7f4118102a70 2024-05-09T08:31:23.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.256+0000 7f411f6cd640 1 -- 172.21.15.107:0/1452960864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f411819f960 con 0x7f4118102a70 2024-05-09T08:31:23.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.256+0000 7f410e7fc640 1 -- 172.21.15.107:0/1452960864 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4108035070 con 0x7f4118102a70 2024-05-09T08:31:23.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.256+0000 7f411f6cd640 1 -- 172.21.15.107:0/1452960864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f411819fe50 con 0x7f4118102a70 2024-05-09T08:31:23.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.257+0000 7f411f6cd640 1 -- 172.21.15.107:0/1452960864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4118108bf0 con 0x7f4118102a70 2024-05-09T08:31:23.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.258+0000 7f410e7fc640 1 -- 172.21.15.107:0/1452960864 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f4108042030 con 0x7f4118102a70 2024-05-09T08:31:23.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.259+0000 7f410e7fc640 1 --2- 172.21.15.107:0/1452960864 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f41040a2c20 0x7f41040a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:23.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.259+0000 7f411d469640 1 --2- 172.21.15.107:0/1452960864 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f41040a2c20 0x7f41040a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:23.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.259+0000 7f410e7fc640 1 -- 172.21.15.107:0/1452960864 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f41080effd0 con 0x7f4118102a70 2024-05-09T08:31:23.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.260+0000 7f411d469640 1 --2- 172.21.15.107:0/1452960864 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f41040a2c20 0x7f41040a50e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f41181028d0 tx=0x7f411400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:23.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.263+0000 7f410e7fc640 1 -- 172.21.15.107:0/1452960864 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f41080bcb90 con 0x7f4118102a70 2024-05-09T08:31:23.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:23 smithi157 ceph-mon[33614]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:31:23.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:23 smithi157 ceph-mon[33614]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-09T08:31:23.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:23 smithi157 ceph-mon[33614]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:31:23.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:23 smithi157 ceph-mon[33614]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:31:23.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:23 smithi110 ceph-mon[33622]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:31:23.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:23 smithi110 ceph-mon[33622]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-09T08:31:23.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:23 smithi110 ceph-mon[33622]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:31:23.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:23 smithi110 ceph-mon[33622]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:31:23.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.438+0000 7f411f6cd640 1 -- 172.21.15.107:0/1452960864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f411802d410 con 0x7f4118102a70 2024-05-09T08:31:23.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.439+0000 7f410e7fc640 1 -- 172.21.15.107:0/1452960864 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f41080bc490 con 0x7f4118102a70 2024-05-09T08:31:23.439 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:23.439 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:31:05.201799+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:31:23.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.441+0000 7f411f6cd640 1 -- 172.21.15.107:0/1452960864 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f41040a2c20 msgr2=0x7f41040a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:23.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.441+0000 7f411f6cd640 1 --2- 172.21.15.107:0/1452960864 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f41040a2c20 0x7f41040a50e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f41181028d0 tx=0x7f411400a2d0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.441+0000 7f411f6cd640 1 -- 172.21.15.107:0/1452960864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4118102a70 msgr2=0x7f4118197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:23.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.441+0000 7f411f6cd640 1 --2- 172.21.15.107:0/1452960864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4118102a70 0x7f4118197ff0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4108004770 tx=0x7f41080042f0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.441+0000 7f411f6cd640 1 -- 172.21.15.107:0/1452960864 shutdown_connections 2024-05-09T08:31:23.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.441+0000 7f411f6cd640 1 --2- 172.21.15.107:0/1452960864 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f41040a2c20 0x7f41040a50e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.441+0000 7f411f6cd640 1 --2- 172.21.15.107:0/1452960864 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4118103430 0x7f411819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.441+0000 7f411f6cd640 1 --2- 172.21.15.107:0/1452960864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4118102a70 0x7f4118197ff0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.441+0000 7f411f6cd640 1 --2- 172.21.15.107:0/1452960864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4118101880 0x7f4118197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.441+0000 7f411f6cd640 1 -- 172.21.15.107:0/1452960864 >> 172.21.15.107:0/1452960864 conn(0x7f41180fce10 msgr2=0x7f4118105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:23.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.442+0000 7f411f6cd640 1 -- 172.21.15.107:0/1452960864 shutdown_connections 2024-05-09T08:31:23.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.442+0000 7f411f6cd640 1 -- 172.21.15.107:0/1452960864 wait complete. 2024-05-09T08:31:23.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:23 smithi107 ceph-mon[30122]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:31:23.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:23 smithi107 ceph-mon[30122]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-09T08:31:23.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:23 smithi107 ceph-mon[30122]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:31:23.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:23 smithi107 ceph-mon[30122]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:31:23.453 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-09T08:31:23.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T08:31:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.556+0000 7f2887f5a640 1 -- 172.21.15.107:0/697434604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2880102320 msgr2=0x7f2880108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.556+0000 7f2887f5a640 1 --2- 172.21.15.107:0/697434604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2880102320 0x7f2880108bb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2870009ca0 tx=0x7f287002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.556+0000 7f2887f5a640 1 -- 172.21.15.107:0/697434604 shutdown_connections 2024-05-09T08:31:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.556+0000 7f2887f5a640 1 --2- 172.21.15.107:0/697434604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2880102320 0x7f2880108bb0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.556+0000 7f2887f5a640 1 --2- 172.21.15.107:0/697434604 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2880101960 0x7f2880101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.556+0000 7f2887f5a640 1 --2- 172.21.15.107:0/697434604 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2880100770 0x7f2880100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.557+0000 7f2887f5a640 1 -- 172.21.15.107:0/697434604 >> 172.21.15.107:0/697434604 conn(0x7f28800fbce0 msgr2=0x7f28800fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.557+0000 7f2887f5a640 1 -- 172.21.15.107:0/697434604 shutdown_connections 2024-05-09T08:31:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.557+0000 7f2887f5a640 1 -- 172.21.15.107:0/697434604 wait complete. 2024-05-09T08:31:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.557+0000 7f2887f5a640 1 Processor -- start 2024-05-09T08:31:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.557+0000 7f2887f5a640 1 -- start start 2024-05-09T08:31:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.557+0000 7f2887f5a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2880100770 0x7f2880197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.557+0000 7f2887f5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2880101960 0x7f2880198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.557+0000 7f2887f5a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2880102320 0x7f288019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.557+0000 7f2887f5a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f288010c320 con 0x7f2880101960 2024-05-09T08:31:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.557+0000 7f2887f5a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f288010c1a0 con 0x7f2880102320 2024-05-09T08:31:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.557+0000 7f2887f5a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f288010c4a0 con 0x7f2880100770 2024-05-09T08:31:23.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.557+0000 7f28854f5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2880101960 0x7f2880198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:23.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.558+0000 7f28854f5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2880101960 0x7f2880198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53314/0 (socket says 172.21.15.107:53314) 2024-05-09T08:31:23.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.558+0000 7f28854f5640 1 -- 172.21.15.107:0/234789088 learned_addr learned my addr 172.21.15.107:0/234789088 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:23.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.558+0000 7f2885cf6640 1 --2- 172.21.15.107:0/234789088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2880100770 0x7f2880197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:23.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.558+0000 7f28864f7640 1 --2- 172.21.15.107:0/234789088 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2880102320 0x7f288019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:23.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.558+0000 7f28854f5640 1 -- 172.21.15.107:0/234789088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2880100770 msgr2=0x7f2880197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:23.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.558+0000 7f28854f5640 1 --2- 172.21.15.107:0/234789088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2880100770 0x7f2880197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.558+0000 7f28854f5640 1 -- 172.21.15.107:0/234789088 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2880102320 msgr2=0x7f288019f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:23.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.558+0000 7f28854f5640 1 --2- 172.21.15.107:0/234789088 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2880102320 0x7f288019f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.558+0000 7f28854f5640 1 -- 172.21.15.107:0/234789088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f288019f720 con 0x7f2880101960 2024-05-09T08:31:23.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.558+0000 7f2885cf6640 1 --2- 172.21.15.107:0/234789088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2880100770 0x7f2880197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:23.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.558+0000 7f28854f5640 1 --2- 172.21.15.107:0/234789088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2880101960 0x7f2880198040 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f287400b4a0 tx=0x7f287400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:23.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.558+0000 7f287effd640 1 -- 172.21.15.107:0/234789088 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2874014070 con 0x7f2880101960 2024-05-09T08:31:23.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.558+0000 7f287effd640 1 -- 172.21.15.107:0/234789088 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2874010ac0 con 0x7f2880101960 2024-05-09T08:31:23.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.558+0000 7f2887f5a640 1 -- 172.21.15.107:0/234789088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f288019fa10 con 0x7f2880101960 2024-05-09T08:31:23.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.558+0000 7f287effd640 1 -- 172.21.15.107:0/234789088 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2874017c30 con 0x7f2880101960 2024-05-09T08:31:23.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.558+0000 7f2887f5a640 1 -- 172.21.15.107:0/234789088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f288019ffb0 con 0x7f2880101960 2024-05-09T08:31:23.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.559+0000 7f2887f5a640 1 -- 172.21.15.107:0/234789088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28801a0240 con 0x7f2880101960 2024-05-09T08:31:23.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.560+0000 7f287effd640 1 -- 172.21.15.107:0/234789088 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f2874017070 con 0x7f2880101960 2024-05-09T08:31:23.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.561+0000 7f287effd640 1 --2- 172.21.15.107:0/234789088 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f28680a2ca0 0x7f28680a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:23.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.561+0000 7f2885cf6640 1 --2- 172.21.15.107:0/234789088 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f28680a2ca0 0x7f28680a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:23.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.561+0000 7f287effd640 1 -- 172.21.15.107:0/234789088 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f28740ce9f0 con 0x7f2880101960 2024-05-09T08:31:23.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.561+0000 7f2885cf6640 1 --2- 172.21.15.107:0/234789088 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f28680a2ca0 0x7f28680a5160 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f28780058c0 tx=0x7f2878005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:23.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.566+0000 7f287effd640 1 -- 172.21.15.107:0/234789088 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f287409b6c0 con 0x7f2880101960 2024-05-09T08:31:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.739+0000 7f2887f5a640 1 -- 172.21.15.107:0/234789088 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f28801a0410 con 0x7f28680a2ca0 2024-05-09T08:31:23.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.749+0000 7f287effd640 1 -- 172.21.15.107:0/234789088 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340300 (secure 0 0 0) 0x7f28801a0410 con 0x7f28680a2ca0 2024-05-09T08:31:23.753 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-05-09T08:31:23.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.752+0000 7f2887f5a640 1 -- 172.21.15.107:0/234789088 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f28680a2ca0 msgr2=0x7f28680a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:23.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.752+0000 7f2887f5a640 1 --2- 172.21.15.107:0/234789088 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f28680a2ca0 0x7f28680a5160 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f28780058c0 tx=0x7f2878005850 comp rx=0 tx=0).stop 2024-05-09T08:31:23.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.752+0000 7f2887f5a640 1 -- 172.21.15.107:0/234789088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2880101960 msgr2=0x7f2880198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:23.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.752+0000 7f2887f5a640 1 --2- 172.21.15.107:0/234789088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2880101960 0x7f2880198040 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f287400b4a0 tx=0x7f287400b960 comp rx=0 tx=0).stop 2024-05-09T08:31:23.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.752+0000 7f2887f5a640 1 -- 172.21.15.107:0/234789088 shutdown_connections 2024-05-09T08:31:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.752+0000 7f2887f5a640 1 --2- 172.21.15.107:0/234789088 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f28680a2ca0 0x7f28680a5160 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.752+0000 7f2887f5a640 1 --2- 172.21.15.107:0/234789088 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2880102320 0x7f288019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.752+0000 7f2887f5a640 1 --2- 172.21.15.107:0/234789088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2880101960 0x7f2880198040 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.752+0000 7f2887f5a640 1 --2- 172.21.15.107:0/234789088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2880100770 0x7f2880197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.752+0000 7f2887f5a640 1 -- 172.21.15.107:0/234789088 >> 172.21.15.107:0/234789088 conn(0x7f28800fbce0 msgr2=0x7f28800fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.752+0000 7f2887f5a640 1 -- 172.21.15.107:0/234789088 shutdown_connections 2024-05-09T08:31:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.752+0000 7f2887f5a640 1 -- 172.21.15.107:0/234789088 wait complete. 2024-05-09T08:31:23.763 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-05-09T08:31:23.457834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597730,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":143,"num_read_kb":126,"num_write":291,"num_write_kb":2980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":391732,"kb_used_data":13436,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124550092,"statfs":{"total":1151940427776,"available":1151539294208,"internally_reserved":0,"allocated":13758464,"data_stored":7517333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":387233081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4952,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":65,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002987"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903504+0000","last_change":"2024-05-09T08:31:00.695034+0000","last_active":"2024-05-09T08:31:05.903504+0000","last_peered":"2024-05-09T08:31:05.903504+0000","last_clean":"2024-05-09T08:31:05.903504+0000","last_became_active":"2024-05-09T08:31:00.694229+0000","last_became_peered":"2024-05-09T08:31:00.694229+0000","last_unstale":"2024-05-09T08:31:05.903504+0000","last_undegraded":"2024-05-09T08:31:05.903504+0000","last_fullsized":"2024-05-09T08:31:05.903504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:57:22.964848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924693+0000","last_change":"2024-05-09T08:31:00.686219+0000","last_active":"2024-05-09T08:31:05.924693+0000","last_peered":"2024-05-09T08:31:05.924693+0000","last_clean":"2024-05-09T08:31:05.924693+0000","last_became_active":"2024-05-09T08:31:00.685911+0000","last_became_peered":"2024-05-09T08:31:00.685911+0000","last_unstale":"2024-05-09T08:31:05.924693+0000","last_undegraded":"2024-05-09T08:31:05.924693+0000","last_fullsized":"2024-05-09T08:31:05.924693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:28.246199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924448+0000","last_change":"2024-05-09T08:31:00.694228+0000","last_active":"2024-05-09T08:31:05.924448+0000","last_peered":"2024-05-09T08:31:05.924448+0000","last_clean":"2024-05-09T08:31:05.924448+0000","last_became_active":"2024-05-09T08:31:00.693838+0000","last_became_peered":"2024-05-09T08:31:00.693838+0000","last_unstale":"2024-05-09T08:31:05.924448+0000","last_undegraded":"2024-05-09T08:31:05.924448+0000","last_fullsized":"2024-05-09T08:31:05.924448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:44.847966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903457+0000","last_change":"2024-05-09T08:31:00.702770+0000","last_active":"2024-05-09T08:31:05.903457+0000","last_peered":"2024-05-09T08:31:05.903457+0000","last_clean":"2024-05-09T08:31:05.903457+0000","last_became_active":"2024-05-09T08:31:00.702424+0000","last_became_peered":"2024-05-09T08:31:00.702424+0000","last_unstale":"2024-05-09T08:31:05.903457+0000","last_undegraded":"2024-05-09T08:31:05.903457+0000","last_fullsized":"2024-05-09T08:31:05.903457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:24:05.228942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902921+0000","last_change":"2024-05-09T08:31:00.688308+0000","last_active":"2024-05-09T08:31:05.902921+0000","last_peered":"2024-05-09T08:31:05.902921+0000","last_clean":"2024-05-09T08:31:05.902921+0000","last_became_active":"2024-05-09T08:31:00.687891+0000","last_became_peered":"2024-05-09T08:31:00.687891+0000","last_unstale":"2024-05-09T08:31:05.902921+0000","last_undegraded":"2024-05-09T08:31:05.902921+0000","last_fullsized":"2024-05-09T08:31:05.902921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:48:42.648940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903780+0000","last_change":"2024-05-09T08:31:00.694705+0000","last_active":"2024-05-09T08:31:05.903780+0000","last_peered":"2024-05-09T08:31:05.903780+0000","last_clean":"2024-05-09T08:31:05.903780+0000","last_became_active":"2024-05-09T08:31:00.694361+0000","last_became_peered":"2024-05-09T08:31:00.694361+0000","last_unstale":"2024-05-09T08:31:05.903780+0000","last_undegraded":"2024-05-09T08:31:05.903780+0000","last_fullsized":"2024-05-09T08:31:05.903780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:07:23.137410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.851966+0000","last_change":"2024-05-09T08:31:00.689824+0000","last_active":"2024-05-09T08:31:05.851966+0000","last_peered":"2024-05-09T08:31:05.851966+0000","last_clean":"2024-05-09T08:31:05.851966+0000","last_became_active":"2024-05-09T08:31:00.689491+0000","last_became_peered":"2024-05-09T08:31:00.689491+0000","last_unstale":"2024-05-09T08:31:05.851966+0000","last_undegraded":"2024-05-09T08:31:05.851966+0000","last_fullsized":"2024-05-09T08:31:05.851966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:18.791225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902743+0000","last_change":"2024-05-09T08:31:00.695014+0000","last_active":"2024-05-09T08:31:05.902743+0000","last_peered":"2024-05-09T08:31:05.902743+0000","last_clean":"2024-05-09T08:31:05.902743+0000","last_became_active":"2024-05-09T08:31:00.694683+0000","last_became_peered":"2024-05-09T08:31:00.694683+0000","last_unstale":"2024-05-09T08:31:05.902743+0000","last_undegraded":"2024-05-09T08:31:05.902743+0000","last_fullsized":"2024-05-09T08:31:05.902743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:08.570258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904494+0000","last_change":"2024-05-09T08:31:00.701538+0000","last_active":"2024-05-09T08:31:05.904494+0000","last_peered":"2024-05-09T08:31:05.904494+0000","last_clean":"2024-05-09T08:31:05.904494+0000","last_became_active":"2024-05-09T08:31:00.701226+0000","last_became_peered":"2024-05-09T08:31:00.701226+0000","last_unstale":"2024-05-09T08:31:05.904494+0000","last_undegraded":"2024-05-09T08:31:05.904494+0000","last_fullsized":"2024-05-09T08:31:05.904494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:14:50.853063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904210+0000","last_change":"2024-05-09T08:31:00.698145+0000","last_active":"2024-05-09T08:31:05.904210+0000","last_peered":"2024-05-09T08:31:05.904210+0000","last_clean":"2024-05-09T08:31:05.904210+0000","last_became_active":"2024-05-09T08:31:00.697761+0000","last_became_peered":"2024-05-09T08:31:00.697761+0000","last_unstale":"2024-05-09T08:31:05.904210+0000","last_undegraded":"2024-05-09T08:31:05.904210+0000","last_fullsized":"2024-05-09T08:31:05.904210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:26.094013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852595+0000","last_change":"2024-05-09T08:31:00.699520+0000","last_active":"2024-05-09T08:31:05.852595+0000","last_peered":"2024-05-09T08:31:05.852595+0000","last_clean":"2024-05-09T08:31:05.852595+0000","last_became_active":"2024-05-09T08:31:00.697579+0000","last_became_peered":"2024-05-09T08:31:00.697579+0000","last_unstale":"2024-05-09T08:31:05.852595+0000","last_undegraded":"2024-05-09T08:31:05.852595+0000","last_fullsized":"2024-05-09T08:31:05.852595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:26:34.324474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903344+0000","last_change":"2024-05-09T08:31:00.693356+0000","last_active":"2024-05-09T08:31:05.903344+0000","last_peered":"2024-05-09T08:31:05.903344+0000","last_clean":"2024-05-09T08:31:05.903344+0000","last_became_active":"2024-05-09T08:31:00.692985+0000","last_became_peered":"2024-05-09T08:31:00.692985+0000","last_unstale":"2024-05-09T08:31:05.903344+0000","last_undegraded":"2024-05-09T08:31:05.903344+0000","last_fullsized":"2024-05-09T08:31:05.903344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:49:01.115497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902630+0000","last_change":"2024-05-09T08:31:00.677768+0000","last_active":"2024-05-09T08:31:05.902630+0000","last_peered":"2024-05-09T08:31:05.902630+0000","last_clean":"2024-05-09T08:31:05.902630+0000","last_became_active":"2024-05-09T08:31:00.677386+0000","last_became_peered":"2024-05-09T08:31:00.677386+0000","last_unstale":"2024-05-09T08:31:05.902630+0000","last_undegraded":"2024-05-09T08:31:05.902630+0000","last_fullsized":"2024-05-09T08:31:05.902630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:55.363324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924381+0000","last_change":"2024-05-09T08:31:00.689609+0000","last_active":"2024-05-09T08:31:05.924381+0000","last_peered":"2024-05-09T08:31:05.924381+0000","last_clean":"2024-05-09T08:31:05.924381+0000","last_became_active":"2024-05-09T08:31:00.689303+0000","last_became_peered":"2024-05-09T08:31:00.689303+0000","last_unstale":"2024-05-09T08:31:05.924381+0000","last_undegraded":"2024-05-09T08:31:05.924381+0000","last_fullsized":"2024-05-09T08:31:05.924381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:15:58.672460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924356+0000","last_change":"2024-05-09T08:31:00.704708+0000","last_active":"2024-05-09T08:31:05.924356+0000","last_peered":"2024-05-09T08:31:05.924356+0000","last_clean":"2024-05-09T08:31:05.924356+0000","last_became_active":"2024-05-09T08:31:00.704269+0000","last_became_peered":"2024-05-09T08:31:00.704269+0000","last_unstale":"2024-05-09T08:31:05.924356+0000","last_undegraded":"2024-05-09T08:31:05.924356+0000","last_fullsized":"2024-05-09T08:31:05.924356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:17:44.580420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902413+0000","last_change":"2024-05-09T08:31:00.703089+0000","last_active":"2024-05-09T08:31:05.902413+0000","last_peered":"2024-05-09T08:31:05.902413+0000","last_clean":"2024-05-09T08:31:05.902413+0000","last_became_active":"2024-05-09T08:31:00.702531+0000","last_became_peered":"2024-05-09T08:31:00.702531+0000","last_unstale":"2024-05-09T08:31:05.902413+0000","last_undegraded":"2024-05-09T08:31:05.902413+0000","last_fullsized":"2024-05-09T08:31:05.902413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:07:39.887748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903343+0000","last_change":"2024-05-09T08:31:00.695332+0000","last_active":"2024-05-09T08:31:05.903343+0000","last_peered":"2024-05-09T08:31:05.903343+0000","last_clean":"2024-05-09T08:31:05.903343+0000","last_became_active":"2024-05-09T08:31:00.694703+0000","last_became_peered":"2024-05-09T08:31:00.694703+0000","last_unstale":"2024-05-09T08:31:05.903343+0000","last_undegraded":"2024-05-09T08:31:05.903343+0000","last_fullsized":"2024-05-09T08:31:05.903343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:53.727922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852069+0000","last_change":"2024-05-09T08:31:00.689871+0000","last_active":"2024-05-09T08:31:05.852069+0000","last_peered":"2024-05-09T08:31:05.852069+0000","last_clean":"2024-05-09T08:31:05.852069+0000","last_became_active":"2024-05-09T08:31:00.689498+0000","last_became_peered":"2024-05-09T08:31:00.689498+0000","last_unstale":"2024-05-09T08:31:05.852069+0000","last_undegraded":"2024-05-09T08:31:05.852069+0000","last_fullsized":"2024-05-09T08:31:05.852069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:18:14.123442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852452+0000","last_change":"2024-05-09T08:31:00.697175+0000","last_active":"2024-05-09T08:31:05.852452+0000","last_peered":"2024-05-09T08:31:05.852452+0000","last_clean":"2024-05-09T08:31:05.852452+0000","last_became_active":"2024-05-09T08:31:00.694103+0000","last_became_peered":"2024-05-09T08:31:00.694103+0000","last_unstale":"2024-05-09T08:31:05.852452+0000","last_undegraded":"2024-05-09T08:31:05.852452+0000","last_fullsized":"2024-05-09T08:31:05.852452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:41:12.549854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852261+0000","last_change":"2024-05-09T08:31:04.859128+0000","last_active":"2024-05-09T08:31:05.852261+0000","last_peered":"2024-05-09T08:31:05.852261+0000","last_clean":"2024-05-09T08:31:05.852261+0000","last_became_active":"2024-05-09T08:31:04.858713+0000","last_became_peered":"2024-05-09T08:31:04.858713+0000","last_unstale":"2024-05-09T08:31:05.852261+0000","last_undegraded":"2024-05-09T08:31:05.852261+0000","last_fullsized":"2024-05-09T08:31:05.852261+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:35:22.620160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852195+0000","last_change":"2024-05-09T08:30:59.828030+0000","last_active":"2024-05-09T08:31:05.852195+0000","last_peered":"2024-05-09T08:31:05.852195+0000","last_clean":"2024-05-09T08:31:05.852195+0000","last_became_active":"2024-05-09T08:30:59.827687+0000","last_became_peered":"2024-05-09T08:30:59.827687+0000","last_unstale":"2024-05-09T08:31:05.852195+0000","last_undegraded":"2024-05-09T08:31:05.852195+0000","last_fullsized":"2024-05-09T08:31:05.852195+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:52:57.321232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902914+0000","last_change":"2024-05-09T08:31:00.696023+0000","last_active":"2024-05-09T08:31:05.902914+0000","last_peered":"2024-05-09T08:31:05.902914+0000","last_clean":"2024-05-09T08:31:05.902914+0000","last_became_active":"2024-05-09T08:31:00.695677+0000","last_became_peered":"2024-05-09T08:31:00.695677+0000","last_unstale":"2024-05-09T08:31:05.902914+0000","last_undegraded":"2024-05-09T08:31:05.902914+0000","last_fullsized":"2024-05-09T08:31:05.902914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:55.989133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902977+0000","last_change":"2024-05-09T08:31:04.859302+0000","last_active":"2024-05-09T08:31:05.902977+0000","last_peered":"2024-05-09T08:31:05.902977+0000","last_clean":"2024-05-09T08:31:05.902977+0000","last_became_active":"2024-05-09T08:31:04.858897+0000","last_became_peered":"2024-05-09T08:31:04.858897+0000","last_unstale":"2024-05-09T08:31:05.902977+0000","last_undegraded":"2024-05-09T08:31:05.902977+0000","last_fullsized":"2024-05-09T08:31:05.902977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:25:32.970833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903397+0000","last_change":"2024-05-09T08:30:59.824451+0000","last_active":"2024-05-09T08:31:05.903397+0000","last_peered":"2024-05-09T08:31:05.903397+0000","last_clean":"2024-05-09T08:31:05.903397+0000","last_became_active":"2024-05-09T08:30:59.823964+0000","last_became_peered":"2024-05-09T08:30:59.823964+0000","last_unstale":"2024-05-09T08:31:05.903397+0000","last_undegraded":"2024-05-09T08:31:05.903397+0000","last_fullsized":"2024-05-09T08:31:05.903397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:50:13.562262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924544+0000","last_change":"2024-05-09T08:31:00.701015+0000","last_active":"2024-05-09T08:31:05.924544+0000","last_peered":"2024-05-09T08:31:05.924544+0000","last_clean":"2024-05-09T08:31:05.924544+0000","last_became_active":"2024-05-09T08:31:00.700654+0000","last_became_peered":"2024-05-09T08:31:00.700654+0000","last_unstale":"2024-05-09T08:31:05.924544+0000","last_undegraded":"2024-05-09T08:31:05.924544+0000","last_fullsized":"2024-05-09T08:31:05.924544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:03.710790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902772+0000","last_change":"2024-05-09T08:31:04.858453+0000","last_active":"2024-05-09T08:31:05.902772+0000","last_peered":"2024-05-09T08:31:05.902772+0000","last_clean":"2024-05-09T08:31:05.902772+0000","last_became_active":"2024-05-09T08:31:04.857974+0000","last_became_peered":"2024-05-09T08:31:04.857974+0000","last_unstale":"2024-05-09T08:31:05.902772+0000","last_undegraded":"2024-05-09T08:31:05.902772+0000","last_fullsized":"2024-05-09T08:31:05.902772+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:42:52.868625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:20.064683+0000","last_change":"2024-05-09T08:30:59.825974+0000","last_active":"2024-05-09T08:31:20.064683+0000","last_peered":"2024-05-09T08:31:20.064683+0000","last_clean":"2024-05-09T08:31:20.064683+0000","last_became_active":"2024-05-09T08:30:59.825671+0000","last_became_peered":"2024-05-09T08:30:59.825671+0000","last_unstale":"2024-05-09T08:31:20.064683+0000","last_undegraded":"2024-05-09T08:31:20.064683+0000","last_fullsized":"2024-05-09T08:31:20.064683+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:38:20.179466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903259+0000","last_change":"2024-05-09T08:31:00.698324+0000","last_active":"2024-05-09T08:31:05.903259+0000","last_peered":"2024-05-09T08:31:05.903259+0000","last_clean":"2024-05-09T08:31:05.903259+0000","last_became_active":"2024-05-09T08:31:00.697992+0000","last_became_peered":"2024-05-09T08:31:00.697992+0000","last_unstale":"2024-05-09T08:31:05.903259+0000","last_undegraded":"2024-05-09T08:31:05.903259+0000","last_fullsized":"2024-05-09T08:31:05.903259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:25:20.786423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902683+0000","last_change":"2024-05-09T08:31:04.861946+0000","last_active":"2024-05-09T08:31:05.902683+0000","last_peered":"2024-05-09T08:31:05.902683+0000","last_clean":"2024-05-09T08:31:05.902683+0000","last_became_active":"2024-05-09T08:31:04.861511+0000","last_became_peered":"2024-05-09T08:31:04.861511+0000","last_unstale":"2024-05-09T08:31:05.902683+0000","last_undegraded":"2024-05-09T08:31:05.902683+0000","last_fullsized":"2024-05-09T08:31:05.902683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:36:44.580499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903998+0000","last_change":"2024-05-09T08:30:59.829961+0000","last_active":"2024-05-09T08:31:05.903998+0000","last_peered":"2024-05-09T08:31:05.903998+0000","last_clean":"2024-05-09T08:31:05.903998+0000","last_became_active":"2024-05-09T08:30:59.828057+0000","last_became_peered":"2024-05-09T08:30:59.828057+0000","last_unstale":"2024-05-09T08:31:05.903998+0000","last_undegraded":"2024-05-09T08:31:05.903998+0000","last_fullsized":"2024-05-09T08:31:05.903998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:39:18.229438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904977+0000","last_change":"2024-05-09T08:31:00.700285+0000","last_active":"2024-05-09T08:31:05.904977+0000","last_peered":"2024-05-09T08:31:05.904977+0000","last_clean":"2024-05-09T08:31:05.904977+0000","last_became_active":"2024-05-09T08:31:00.699958+0000","last_became_peered":"2024-05-09T08:31:00.699958+0000","last_unstale":"2024-05-09T08:31:05.904977+0000","last_undegraded":"2024-05-09T08:31:05.904977+0000","last_fullsized":"2024-05-09T08:31:05.904977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:48.934827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852447+0000","last_change":"2024-05-09T08:31:04.857060+0000","last_active":"2024-05-09T08:31:05.852447+0000","last_peered":"2024-05-09T08:31:05.852447+0000","last_clean":"2024-05-09T08:31:05.852447+0000","last_became_active":"2024-05-09T08:31:04.855261+0000","last_became_peered":"2024-05-09T08:31:04.855261+0000","last_unstale":"2024-05-09T08:31:05.852447+0000","last_undegraded":"2024-05-09T08:31:05.852447+0000","last_fullsized":"2024-05-09T08:31:05.852447+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:08:08.230524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.052984+0000","last_change":"2024-05-09T08:30:59.831089+0000","last_active":"2024-05-09T08:31:19.052984+0000","last_peered":"2024-05-09T08:31:19.052984+0000","last_clean":"2024-05-09T08:31:19.052984+0000","last_became_active":"2024-05-09T08:30:59.830572+0000","last_became_peered":"2024-05-09T08:30:59.830572+0000","last_unstale":"2024-05-09T08:31:19.052984+0000","last_undegraded":"2024-05-09T08:31:19.052984+0000","last_fullsized":"2024-05-09T08:31:19.052984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:39:10.495104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903462+0000","last_change":"2024-05-09T08:31:00.688792+0000","last_active":"2024-05-09T08:31:05.903462+0000","last_peered":"2024-05-09T08:31:05.903462+0000","last_clean":"2024-05-09T08:31:05.903462+0000","last_became_active":"2024-05-09T08:31:00.688386+0000","last_became_peered":"2024-05-09T08:31:00.688386+0000","last_unstale":"2024-05-09T08:31:05.903462+0000","last_undegraded":"2024-05-09T08:31:05.903462+0000","last_fullsized":"2024-05-09T08:31:05.903462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:24.287333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852532+0000","last_change":"2024-05-09T08:31:04.862287+0000","last_active":"2024-05-09T08:31:05.852532+0000","last_peered":"2024-05-09T08:31:05.852532+0000","last_clean":"2024-05-09T08:31:05.852532+0000","last_became_active":"2024-05-09T08:31:04.861866+0000","last_became_peered":"2024-05-09T08:31:04.861866+0000","last_unstale":"2024-05-09T08:31:05.852532+0000","last_undegraded":"2024-05-09T08:31:05.852532+0000","last_fullsized":"2024-05-09T08:31:05.852532+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:14:21.552787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903660+0000","last_change":"2024-05-09T08:30:59.824857+0000","last_active":"2024-05-09T08:31:05.903660+0000","last_peered":"2024-05-09T08:31:05.903660+0000","last_clean":"2024-05-09T08:31:05.903660+0000","last_became_active":"2024-05-09T08:30:59.824363+0000","last_became_peered":"2024-05-09T08:30:59.824363+0000","last_unstale":"2024-05-09T08:31:05.903660+0000","last_undegraded":"2024-05-09T08:31:05.903660+0000","last_fullsized":"2024-05-09T08:31:05.903660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:20:32.267237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904922+0000","last_change":"2024-05-09T08:31:00.694440+0000","last_active":"2024-05-09T08:31:05.904922+0000","last_peered":"2024-05-09T08:31:05.904922+0000","last_clean":"2024-05-09T08:31:05.904922+0000","last_became_active":"2024-05-09T08:31:00.694085+0000","last_became_peered":"2024-05-09T08:31:00.694085+0000","last_unstale":"2024-05-09T08:31:05.904922+0000","last_undegraded":"2024-05-09T08:31:05.904922+0000","last_fullsized":"2024-05-09T08:31:05.904922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:03:24.697479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903268+0000","last_change":"2024-05-09T08:31:04.850142+0000","last_active":"2024-05-09T08:31:05.903268+0000","last_peered":"2024-05-09T08:31:05.903268+0000","last_clean":"2024-05-09T08:31:05.903268+0000","last_became_active":"2024-05-09T08:31:04.849689+0000","last_became_peered":"2024-05-09T08:31:04.849689+0000","last_unstale":"2024-05-09T08:31:05.903268+0000","last_undegraded":"2024-05-09T08:31:05.903268+0000","last_fullsized":"2024-05-09T08:31:05.903268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:39:46.811808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:20.062955+0000","last_change":"2024-05-09T08:30:59.818373+0000","last_active":"2024-05-09T08:31:20.062955+0000","last_peered":"2024-05-09T08:31:20.062955+0000","last_clean":"2024-05-09T08:31:20.062955+0000","last_became_active":"2024-05-09T08:30:59.818023+0000","last_became_peered":"2024-05-09T08:30:59.818023+0000","last_unstale":"2024-05-09T08:31:20.062955+0000","last_undegraded":"2024-05-09T08:31:20.062955+0000","last_fullsized":"2024-05-09T08:31:20.062955+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:23:01.994257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903204+0000","last_change":"2024-05-09T08:31:00.689506+0000","last_active":"2024-05-09T08:31:05.903204+0000","last_peered":"2024-05-09T08:31:05.903204+0000","last_clean":"2024-05-09T08:31:05.903204+0000","last_became_active":"2024-05-09T08:31:00.689151+0000","last_became_peered":"2024-05-09T08:31:00.689151+0000","last_unstale":"2024-05-09T08:31:05.903204+0000","last_undegraded":"2024-05-09T08:31:05.903204+0000","last_fullsized":"2024-05-09T08:31:05.903204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:48:13.539509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924721+0000","last_change":"2024-05-09T08:31:04.859215+0000","last_active":"2024-05-09T08:31:05.924721+0000","last_peered":"2024-05-09T08:31:05.924721+0000","last_clean":"2024-05-09T08:31:05.924721+0000","last_became_active":"2024-05-09T08:31:04.858759+0000","last_became_peered":"2024-05-09T08:31:04.858759+0000","last_unstale":"2024-05-09T08:31:05.924721+0000","last_undegraded":"2024-05-09T08:31:05.924721+0000","last_fullsized":"2024-05-09T08:31:05.924721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:41:42.124234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904511+0000","last_change":"2024-05-09T08:30:59.824260+0000","last_active":"2024-05-09T08:31:05.904511+0000","last_peered":"2024-05-09T08:31:05.904511+0000","last_clean":"2024-05-09T08:31:05.904511+0000","last_became_active":"2024-05-09T08:30:59.823959+0000","last_became_peered":"2024-05-09T08:30:59.823959+0000","last_unstale":"2024-05-09T08:31:05.904511+0000","last_undegraded":"2024-05-09T08:31:05.904511+0000","last_fullsized":"2024-05-09T08:31:05.904511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:48:42.354279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.851454+0000","last_change":"2024-05-09T08:31:00.687810+0000","last_active":"2024-05-09T08:31:05.851454+0000","last_peered":"2024-05-09T08:31:05.851454+0000","last_clean":"2024-05-09T08:31:05.851454+0000","last_became_active":"2024-05-09T08:31:00.687486+0000","last_became_peered":"2024-05-09T08:31:00.687486+0000","last_unstale":"2024-05-09T08:31:05.851454+0000","last_undegraded":"2024-05-09T08:31:05.851454+0000","last_fullsized":"2024-05-09T08:31:05.851454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:51.428208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903674+0000","last_change":"2024-05-09T08:31:04.853222+0000","last_active":"2024-05-09T08:31:05.903674+0000","last_peered":"2024-05-09T08:31:05.903674+0000","last_clean":"2024-05-09T08:31:05.903674+0000","last_became_active":"2024-05-09T08:31:04.852676+0000","last_became_peered":"2024-05-09T08:31:04.852676+0000","last_unstale":"2024-05-09T08:31:05.903674+0000","last_undegraded":"2024-05-09T08:31:05.903674+0000","last_fullsized":"2024-05-09T08:31:05.903674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:58:31.949938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903634+0000","last_change":"2024-05-09T08:30:59.824653+0000","last_active":"2024-05-09T08:31:05.903634+0000","last_peered":"2024-05-09T08:31:05.903634+0000","last_clean":"2024-05-09T08:31:05.903634+0000","last_became_active":"2024-05-09T08:30:59.824277+0000","last_became_peered":"2024-05-09T08:30:59.824277+0000","last_unstale":"2024-05-09T08:31:05.903634+0000","last_undegraded":"2024-05-09T08:31:05.903634+0000","last_fullsized":"2024-05-09T08:31:05.903634+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:45:20.340629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904459+0000","last_change":"2024-05-09T08:31:00.697484+0000","last_active":"2024-05-09T08:31:05.904459+0000","last_peered":"2024-05-09T08:31:05.904459+0000","last_clean":"2024-05-09T08:31:05.904459+0000","last_became_active":"2024-05-09T08:31:00.697095+0000","last_became_peered":"2024-05-09T08:31:00.697095+0000","last_unstale":"2024-05-09T08:31:05.904459+0000","last_undegraded":"2024-05-09T08:31:05.904459+0000","last_fullsized":"2024-05-09T08:31:05.904459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:29:40.702057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.851529+0000","last_change":"2024-05-09T08:31:04.860048+0000","last_active":"2024-05-09T08:31:05.851529+0000","last_peered":"2024-05-09T08:31:05.851529+0000","last_clean":"2024-05-09T08:31:05.851529+0000","last_became_active":"2024-05-09T08:31:04.857677+0000","last_became_peered":"2024-05-09T08:31:04.857677+0000","last_unstale":"2024-05-09T08:31:05.851529+0000","last_undegraded":"2024-05-09T08:31:05.851529+0000","last_fullsized":"2024-05-09T08:31:05.851529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:40.850305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.052341+0000","last_change":"2024-05-09T08:30:59.822683+0000","last_active":"2024-05-09T08:31:19.052341+0000","last_peered":"2024-05-09T08:31:19.052341+0000","last_clean":"2024-05-09T08:31:19.052341+0000","last_became_active":"2024-05-09T08:30:59.822150+0000","last_became_peered":"2024-05-09T08:30:59.822150+0000","last_unstale":"2024-05-09T08:31:19.052341+0000","last_undegraded":"2024-05-09T08:31:19.052341+0000","last_fullsized":"2024-05-09T08:31:19.052341+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:51:32.782076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924495+0000","last_change":"2024-05-09T08:31:00.694165+0000","last_active":"2024-05-09T08:31:05.924495+0000","last_peered":"2024-05-09T08:31:05.924495+0000","last_clean":"2024-05-09T08:31:05.924495+0000","last_became_active":"2024-05-09T08:31:00.693150+0000","last_became_peered":"2024-05-09T08:31:00.693150+0000","last_unstale":"2024-05-09T08:31:05.924495+0000","last_undegraded":"2024-05-09T08:31:05.924495+0000","last_fullsized":"2024-05-09T08:31:05.924495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:29.278078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904155+0000","last_change":"2024-05-09T08:31:04.858785+0000","last_active":"2024-05-09T08:31:05.904155+0000","last_peered":"2024-05-09T08:31:05.904155+0000","last_clean":"2024-05-09T08:31:05.904155+0000","last_became_active":"2024-05-09T08:31:04.857947+0000","last_became_peered":"2024-05-09T08:31:04.857947+0000","last_unstale":"2024-05-09T08:31:05.904155+0000","last_undegraded":"2024-05-09T08:31:05.904155+0000","last_fullsized":"2024-05-09T08:31:05.904155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:55:48.595607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.905027+0000","last_change":"2024-05-09T08:30:59.822946+0000","last_active":"2024-05-09T08:31:05.905027+0000","last_peered":"2024-05-09T08:31:05.905027+0000","last_clean":"2024-05-09T08:31:05.905027+0000","last_became_active":"2024-05-09T08:30:59.822174+0000","last_became_peered":"2024-05-09T08:30:59.822174+0000","last_unstale":"2024-05-09T08:31:05.905027+0000","last_undegraded":"2024-05-09T08:31:05.905027+0000","last_fullsized":"2024-05-09T08:31:05.905027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:14.224033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903439+0000","last_change":"2024-05-09T08:31:00.704781+0000","last_active":"2024-05-09T08:31:05.903439+0000","last_peered":"2024-05-09T08:31:05.903439+0000","last_clean":"2024-05-09T08:31:05.903439+0000","last_became_active":"2024-05-09T08:31:00.704170+0000","last_became_peered":"2024-05-09T08:31:00.704170+0000","last_unstale":"2024-05-09T08:31:05.903439+0000","last_undegraded":"2024-05-09T08:31:05.903439+0000","last_fullsized":"2024-05-09T08:31:05.903439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:27:47.402290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.851510+0000","last_change":"2024-05-09T08:31:04.855648+0000","last_active":"2024-05-09T08:31:05.851510+0000","last_peered":"2024-05-09T08:31:05.851510+0000","last_clean":"2024-05-09T08:31:05.851510+0000","last_became_active":"2024-05-09T08:31:04.854678+0000","last_became_peered":"2024-05-09T08:31:04.854678+0000","last_unstale":"2024-05-09T08:31:05.851510+0000","last_undegraded":"2024-05-09T08:31:05.851510+0000","last_fullsized":"2024-05-09T08:31:05.851510+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:53:46.039172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'10","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:16.429196+0000","last_change":"2024-05-09T08:30:59.826498+0000","last_active":"2024-05-09T08:31:16.429196+0000","last_peered":"2024-05-09T08:31:16.429196+0000","last_clean":"2024-05-09T08:31:16.429196+0000","last_became_active":"2024-05-09T08:30:59.826050+0000","last_became_peered":"2024-05-09T08:30:59.826050+0000","last_unstale":"2024-05-09T08:31:16.429196+0000","last_undegraded":"2024-05-09T08:31:16.429196+0000","last_fullsized":"2024-05-09T08:31:16.429196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:19.920901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903767+0000","last_change":"2024-05-09T08:31:00.688379+0000","last_active":"2024-05-09T08:31:05.903767+0000","last_peered":"2024-05-09T08:31:05.903767+0000","last_clean":"2024-05-09T08:31:05.903767+0000","last_became_active":"2024-05-09T08:31:00.688030+0000","last_became_peered":"2024-05-09T08:31:00.688030+0000","last_unstale":"2024-05-09T08:31:05.903767+0000","last_undegraded":"2024-05-09T08:31:05.903767+0000","last_fullsized":"2024-05-09T08:31:05.903767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:17.248154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.905313+0000","last_change":"2024-05-09T08:31:04.858513+0000","last_active":"2024-05-09T08:31:05.905313+0000","last_peered":"2024-05-09T08:31:05.905313+0000","last_clean":"2024-05-09T08:31:05.905313+0000","last_became_active":"2024-05-09T08:31:04.857988+0000","last_became_peered":"2024-05-09T08:31:04.857988+0000","last_unstale":"2024-05-09T08:31:05.905313+0000","last_undegraded":"2024-05-09T08:31:05.905313+0000","last_fullsized":"2024-05-09T08:31:05.905313+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:59:24.264503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:20.061824+0000","last_change":"2024-05-09T08:30:59.823403+0000","last_active":"2024-05-09T08:31:20.061824+0000","last_peered":"2024-05-09T08:31:20.061824+0000","last_clean":"2024-05-09T08:31:20.061824+0000","last_became_active":"2024-05-09T08:30:59.823066+0000","last_became_peered":"2024-05-09T08:30:59.823066+0000","last_unstale":"2024-05-09T08:31:20.061824+0000","last_undegraded":"2024-05-09T08:31:20.061824+0000","last_fullsized":"2024-05-09T08:31:20.061824+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:16:39.157895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902459+0000","last_change":"2024-05-09T08:31:00.688009+0000","last_active":"2024-05-09T08:31:05.902459+0000","last_peered":"2024-05-09T08:31:05.902459+0000","last_clean":"2024-05-09T08:31:05.902459+0000","last_became_active":"2024-05-09T08:31:00.687679+0000","last_became_peered":"2024-05-09T08:31:00.687679+0000","last_unstale":"2024-05-09T08:31:05.902459+0000","last_undegraded":"2024-05-09T08:31:05.902459+0000","last_fullsized":"2024-05-09T08:31:05.902459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:44.403639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903957+0000","last_change":"2024-05-09T08:31:04.851880+0000","last_active":"2024-05-09T08:31:05.903957+0000","last_peered":"2024-05-09T08:31:05.903957+0000","last_clean":"2024-05-09T08:31:05.903957+0000","last_became_active":"2024-05-09T08:31:04.851316+0000","last_became_peered":"2024-05-09T08:31:04.851316+0000","last_unstale":"2024-05-09T08:31:05.903957+0000","last_undegraded":"2024-05-09T08:31:05.903957+0000","last_fullsized":"2024-05-09T08:31:05.903957+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:53.144996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.051820+0000","last_change":"2024-05-09T08:30:59.823411+0000","last_active":"2024-05-09T08:31:19.051820+0000","last_peered":"2024-05-09T08:31:19.051820+0000","last_clean":"2024-05-09T08:31:19.051820+0000","last_became_active":"2024-05-09T08:30:59.822854+0000","last_became_peered":"2024-05-09T08:30:59.822854+0000","last_unstale":"2024-05-09T08:31:19.051820+0000","last_undegraded":"2024-05-09T08:31:19.051820+0000","last_fullsized":"2024-05-09T08:31:19.051820+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:36:03.666937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852369+0000","last_change":"2024-05-09T08:31:00.698508+0000","last_active":"2024-05-09T08:31:05.852369+0000","last_peered":"2024-05-09T08:31:05.852369+0000","last_clean":"2024-05-09T08:31:05.852369+0000","last_became_active":"2024-05-09T08:31:00.696947+0000","last_became_peered":"2024-05-09T08:31:00.696947+0000","last_unstale":"2024-05-09T08:31:05.852369+0000","last_undegraded":"2024-05-09T08:31:05.852369+0000","last_fullsized":"2024-05-09T08:31:05.852369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:47:27.484554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852070+0000","last_change":"2024-05-09T08:31:04.859338+0000","last_active":"2024-05-09T08:31:05.852070+0000","last_peered":"2024-05-09T08:31:05.852070+0000","last_clean":"2024-05-09T08:31:05.852070+0000","last_became_active":"2024-05-09T08:31:04.858951+0000","last_became_peered":"2024-05-09T08:31:04.858951+0000","last_unstale":"2024-05-09T08:31:05.852070+0000","last_undegraded":"2024-05-09T08:31:05.852070+0000","last_fullsized":"2024-05-09T08:31:05.852070+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:26:46.738359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:20.061213+0000","last_change":"2024-05-09T08:30:59.823710+0000","last_active":"2024-05-09T08:31:20.061213+0000","last_peered":"2024-05-09T08:31:20.061213+0000","last_clean":"2024-05-09T08:31:20.061213+0000","last_became_active":"2024-05-09T08:30:59.823213+0000","last_became_peered":"2024-05-09T08:30:59.823213+0000","last_unstale":"2024-05-09T08:31:20.061213+0000","last_undegraded":"2024-05-09T08:31:20.061213+0000","last_fullsized":"2024-05-09T08:31:20.061213+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:09.134942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852079+0000","last_change":"2024-05-09T08:31:00.693735+0000","last_active":"2024-05-09T08:31:05.852079+0000","last_peered":"2024-05-09T08:31:05.852079+0000","last_clean":"2024-05-09T08:31:05.852079+0000","last_became_active":"2024-05-09T08:31:00.693401+0000","last_became_peered":"2024-05-09T08:31:00.693401+0000","last_unstale":"2024-05-09T08:31:05.852079+0000","last_undegraded":"2024-05-09T08:31:05.852079+0000","last_fullsized":"2024-05-09T08:31:05.852079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:58:41.429854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903130+0000","last_change":"2024-05-09T08:31:04.861289+0000","last_active":"2024-05-09T08:31:05.903130+0000","last_peered":"2024-05-09T08:31:05.903130+0000","last_clean":"2024-05-09T08:31:05.903130+0000","last_became_active":"2024-05-09T08:31:04.860915+0000","last_became_peered":"2024-05-09T08:31:04.860915+0000","last_unstale":"2024-05-09T08:31:05.903130+0000","last_undegraded":"2024-05-09T08:31:05.903130+0000","last_fullsized":"2024-05-09T08:31:05.903130+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:01:45.724961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:20.065310+0000","last_change":"2024-05-09T08:30:59.827631+0000","last_active":"2024-05-09T08:31:20.065310+0000","last_peered":"2024-05-09T08:31:20.065310+0000","last_clean":"2024-05-09T08:31:20.065310+0000","last_became_active":"2024-05-09T08:30:59.827295+0000","last_became_peered":"2024-05-09T08:30:59.827295+0000","last_unstale":"2024-05-09T08:31:20.065310+0000","last_undegraded":"2024-05-09T08:31:20.065310+0000","last_fullsized":"2024-05-09T08:31:20.065310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:35.282564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904558+0000","last_change":"2024-05-09T08:31:00.697712+0000","last_active":"2024-05-09T08:31:05.904558+0000","last_peered":"2024-05-09T08:31:05.904558+0000","last_clean":"2024-05-09T08:31:05.904558+0000","last_became_active":"2024-05-09T08:31:00.697385+0000","last_became_peered":"2024-05-09T08:31:00.697385+0000","last_unstale":"2024-05-09T08:31:05.904558+0000","last_undegraded":"2024-05-09T08:31:05.904558+0000","last_fullsized":"2024-05-09T08:31:05.904558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:09.087279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904633+0000","last_change":"2024-05-09T08:31:04.860691+0000","last_active":"2024-05-09T08:31:05.904633+0000","last_peered":"2024-05-09T08:31:05.904633+0000","last_clean":"2024-05-09T08:31:05.904633+0000","last_became_active":"2024-05-09T08:31:04.858458+0000","last_became_peered":"2024-05-09T08:31:04.858458+0000","last_unstale":"2024-05-09T08:31:05.904633+0000","last_undegraded":"2024-05-09T08:31:05.904633+0000","last_fullsized":"2024-05-09T08:31:05.904633+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:16.274359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:21.057390+0000","last_change":"2024-05-09T08:30:59.825188+0000","last_active":"2024-05-09T08:31:21.057390+0000","last_peered":"2024-05-09T08:31:21.057390+0000","last_clean":"2024-05-09T08:31:21.057390+0000","last_became_active":"2024-05-09T08:30:59.824855+0000","last_became_peered":"2024-05-09T08:30:59.824855+0000","last_unstale":"2024-05-09T08:31:21.057390+0000","last_undegraded":"2024-05-09T08:31:21.057390+0000","last_fullsized":"2024-05-09T08:31:21.057390+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:17:21.753652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903360+0000","last_change":"2024-05-09T08:31:00.695862+0000","last_active":"2024-05-09T08:31:05.903360+0000","last_peered":"2024-05-09T08:31:05.903360+0000","last_clean":"2024-05-09T08:31:05.903360+0000","last_became_active":"2024-05-09T08:31:00.695404+0000","last_became_peered":"2024-05-09T08:31:00.695404+0000","last_unstale":"2024-05-09T08:31:05.903360+0000","last_undegraded":"2024-05-09T08:31:05.903360+0000","last_fullsized":"2024-05-09T08:31:05.903360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:53:07.567583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903393+0000","last_change":"2024-05-09T08:31:04.856322+0000","last_active":"2024-05-09T08:31:05.903393+0000","last_peered":"2024-05-09T08:31:05.903393+0000","last_clean":"2024-05-09T08:31:05.903393+0000","last_became_active":"2024-05-09T08:31:04.855821+0000","last_became_peered":"2024-05-09T08:31:04.855821+0000","last_unstale":"2024-05-09T08:31:05.903393+0000","last_undegraded":"2024-05-09T08:31:05.903393+0000","last_fullsized":"2024-05-09T08:31:05.903393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:29:18.594928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.851831+0000","last_change":"2024-05-09T08:30:59.825924+0000","last_active":"2024-05-09T08:31:05.851831+0000","last_peered":"2024-05-09T08:31:05.851831+0000","last_clean":"2024-05-09T08:31:05.851831+0000","last_became_active":"2024-05-09T08:30:59.825505+0000","last_became_peered":"2024-05-09T08:30:59.825505+0000","last_unstale":"2024-05-09T08:31:05.851831+0000","last_undegraded":"2024-05-09T08:31:05.851831+0000","last_fullsized":"2024-05-09T08:31:05.851831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:31:07.416783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903863+0000","last_change":"2024-05-09T08:31:00.684523+0000","last_active":"2024-05-09T08:31:05.903863+0000","last_peered":"2024-05-09T08:31:05.903863+0000","last_clean":"2024-05-09T08:31:05.903863+0000","last_became_active":"2024-05-09T08:31:00.684153+0000","last_became_peered":"2024-05-09T08:31:00.684153+0000","last_unstale":"2024-05-09T08:31:05.903863+0000","last_undegraded":"2024-05-09T08:31:05.903863+0000","last_fullsized":"2024-05-09T08:31:05.903863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:02.423401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903520+0000","last_change":"2024-05-09T08:31:04.857961+0000","last_active":"2024-05-09T08:31:05.903520+0000","last_peered":"2024-05-09T08:31:05.903520+0000","last_clean":"2024-05-09T08:31:05.903520+0000","last_became_active":"2024-05-09T08:31:04.857543+0000","last_became_peered":"2024-05-09T08:31:04.857543+0000","last_unstale":"2024-05-09T08:31:05.903520+0000","last_undegraded":"2024-05-09T08:31:05.903520+0000","last_fullsized":"2024-05-09T08:31:05.903520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:22:30.820297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852189+0000","last_change":"2024-05-09T08:30:59.827878+0000","last_active":"2024-05-09T08:31:05.852189+0000","last_peered":"2024-05-09T08:31:05.852189+0000","last_clean":"2024-05-09T08:31:05.852189+0000","last_became_active":"2024-05-09T08:30:59.827487+0000","last_became_peered":"2024-05-09T08:30:59.827487+0000","last_unstale":"2024-05-09T08:31:05.852189+0000","last_undegraded":"2024-05-09T08:31:05.852189+0000","last_fullsized":"2024-05-09T08:31:05.852189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:34.465457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903788+0000","last_change":"2024-05-09T08:31:00.695617+0000","last_active":"2024-05-09T08:31:05.903788+0000","last_peered":"2024-05-09T08:31:05.903788+0000","last_clean":"2024-05-09T08:31:05.903788+0000","last_became_active":"2024-05-09T08:31:00.695293+0000","last_became_peered":"2024-05-09T08:31:00.695293+0000","last_unstale":"2024-05-09T08:31:05.903788+0000","last_undegraded":"2024-05-09T08:31:05.903788+0000","last_fullsized":"2024-05-09T08:31:05.903788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:44:03.958559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903306+0000","last_change":"2024-05-09T08:31:04.860184+0000","last_active":"2024-05-09T08:31:05.903306+0000","last_peered":"2024-05-09T08:31:05.903306+0000","last_clean":"2024-05-09T08:31:05.903306+0000","last_became_active":"2024-05-09T08:31:04.859730+0000","last_became_peered":"2024-05-09T08:31:04.859730+0000","last_unstale":"2024-05-09T08:31:05.903306+0000","last_undegraded":"2024-05-09T08:31:05.903306+0000","last_fullsized":"2024-05-09T08:31:05.903306+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:13:17.292678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904221+0000","last_change":"2024-05-09T08:31:00.694183+0000","last_active":"2024-05-09T08:31:05.904221+0000","last_peered":"2024-05-09T08:31:05.904221+0000","last_clean":"2024-05-09T08:31:05.904221+0000","last_became_active":"2024-05-09T08:31:00.693819+0000","last_became_peered":"2024-05-09T08:31:00.693819+0000","last_unstale":"2024-05-09T08:31:05.904221+0000","last_undegraded":"2024-05-09T08:31:05.904221+0000","last_fullsized":"2024-05-09T08:31:05.904221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:09:26.255663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:20.062975+0000","last_change":"2024-05-09T08:30:59.830758+0000","last_active":"2024-05-09T08:31:20.062975+0000","last_peered":"2024-05-09T08:31:20.062975+0000","last_clean":"2024-05-09T08:31:20.062975+0000","last_became_active":"2024-05-09T08:30:59.830430+0000","last_became_peered":"2024-05-09T08:30:59.830430+0000","last_unstale":"2024-05-09T08:31:20.062975+0000","last_undegraded":"2024-05-09T08:31:20.062975+0000","last_fullsized":"2024-05-09T08:31:20.062975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:24:05.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852582+0000","last_change":"2024-05-09T08:31:04.860599+0000","last_active":"2024-05-09T08:31:05.852582+0000","last_peered":"2024-05-09T08:31:05.852582+0000","last_clean":"2024-05-09T08:31:05.852582+0000","last_became_active":"2024-05-09T08:31:04.860077+0000","last_became_peered":"2024-05-09T08:31:04.860077+0000","last_unstale":"2024-05-09T08:31:05.852582+0000","last_undegraded":"2024-05-09T08:31:05.852582+0000","last_fullsized":"2024-05-09T08:31:05.852582+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:34.594241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903082+0000","last_change":"2024-05-09T08:31:00.692197+0000","last_active":"2024-05-09T08:31:05.903082+0000","last_peered":"2024-05-09T08:31:05.903082+0000","last_clean":"2024-05-09T08:31:05.903082+0000","last_became_active":"2024-05-09T08:31:00.691348+0000","last_became_peered":"2024-05-09T08:31:00.691348+0000","last_unstale":"2024-05-09T08:31:05.903082+0000","last_undegraded":"2024-05-09T08:31:05.903082+0000","last_fullsized":"2024-05-09T08:31:05.903082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:00:45.826433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.052780+0000","last_change":"2024-05-09T08:30:59.825341+0000","last_active":"2024-05-09T08:31:19.052780+0000","last_peered":"2024-05-09T08:31:19.052780+0000","last_clean":"2024-05-09T08:31:19.052780+0000","last_became_active":"2024-05-09T08:30:59.824860+0000","last_became_peered":"2024-05-09T08:30:59.824860+0000","last_unstale":"2024-05-09T08:31:19.052780+0000","last_undegraded":"2024-05-09T08:31:19.052780+0000","last_fullsized":"2024-05-09T08:31:19.052780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:13.868040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.851912+0000","last_change":"2024-05-09T08:29:47.816504+0000","last_active":"2024-05-09T08:31:05.851912+0000","last_peered":"2024-05-09T08:31:05.851912+0000","last_clean":"2024-05-09T08:31:05.851912+0000","last_became_active":"2024-05-09T08:29:47.816050+0000","last_became_peered":"2024-05-09T08:29:47.816050+0000","last_unstale":"2024-05-09T08:31:05.851912+0000","last_undegraded":"2024-05-09T08:31:05.851912+0000","last_fullsized":"2024-05-09T08:31:05.851912+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:26:22.135491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:26:22.135491+0000","last_clean_scrub_stamp":"2024-05-09T08:26:22.135491+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:28.543694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.905286+0000","last_change":"2024-05-09T08:31:04.860417+0000","last_active":"2024-05-09T08:31:05.905286+0000","last_peered":"2024-05-09T08:31:05.905286+0000","last_clean":"2024-05-09T08:31:05.905286+0000","last_became_active":"2024-05-09T08:31:04.859040+0000","last_became_peered":"2024-05-09T08:31:04.859040+0000","last_unstale":"2024-05-09T08:31:05.905286+0000","last_undegraded":"2024-05-09T08:31:05.905286+0000","last_fullsized":"2024-05-09T08:31:05.905286+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:12:04.157354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902189+0000","last_change":"2024-05-09T08:31:00.703131+0000","last_active":"2024-05-09T08:31:05.902189+0000","last_peered":"2024-05-09T08:31:05.902189+0000","last_clean":"2024-05-09T08:31:05.902189+0000","last_became_active":"2024-05-09T08:31:00.702817+0000","last_became_peered":"2024-05-09T08:31:00.702817+0000","last_unstale":"2024-05-09T08:31:05.902189+0000","last_undegraded":"2024-05-09T08:31:05.902189+0000","last_fullsized":"2024-05-09T08:31:05.902189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:31.059334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:20.063532+0000","last_change":"2024-05-09T08:30:59.829636+0000","last_active":"2024-05-09T08:31:20.063532+0000","last_peered":"2024-05-09T08:31:20.063532+0000","last_clean":"2024-05-09T08:31:20.063532+0000","last_became_active":"2024-05-09T08:30:59.829276+0000","last_became_peered":"2024-05-09T08:30:59.829276+0000","last_unstale":"2024-05-09T08:31:20.063532+0000","last_undegraded":"2024-05-09T08:31:20.063532+0000","last_fullsized":"2024-05-09T08:31:20.063532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:17.847258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904168+0000","last_change":"2024-05-09T08:31:04.853945+0000","last_active":"2024-05-09T08:31:05.904168+0000","last_peered":"2024-05-09T08:31:05.904168+0000","last_clean":"2024-05-09T08:31:05.904168+0000","last_became_active":"2024-05-09T08:31:04.853417+0000","last_became_peered":"2024-05-09T08:31:04.853417+0000","last_unstale":"2024-05-09T08:31:05.904168+0000","last_undegraded":"2024-05-09T08:31:05.904168+0000","last_fullsized":"2024-05-09T08:31:05.904168+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:21:31.126562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.051050+0000","last_change":"2024-05-09T08:30:59.828613+0000","last_active":"2024-05-09T08:31:19.051050+0000","last_peered":"2024-05-09T08:31:19.051050+0000","last_clean":"2024-05-09T08:31:19.051050+0000","last_became_active":"2024-05-09T08:30:59.828256+0000","last_became_peered":"2024-05-09T08:30:59.828256+0000","last_unstale":"2024-05-09T08:31:19.051050+0000","last_undegraded":"2024-05-09T08:31:19.051050+0000","last_fullsized":"2024-05-09T08:31:19.051050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:17:04.488271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903709+0000","last_change":"2024-05-09T08:31:00.703478+0000","last_active":"2024-05-09T08:31:05.903709+0000","last_peered":"2024-05-09T08:31:05.903709+0000","last_clean":"2024-05-09T08:31:05.903709+0000","last_became_active":"2024-05-09T08:31:00.702852+0000","last_became_peered":"2024-05-09T08:31:00.702852+0000","last_unstale":"2024-05-09T08:31:05.903709+0000","last_undegraded":"2024-05-09T08:31:05.903709+0000","last_fullsized":"2024-05-09T08:31:05.903709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:12.635341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903152+0000","last_change":"2024-05-09T08:31:04.860801+0000","last_active":"2024-05-09T08:31:05.903152+0000","last_peered":"2024-05-09T08:31:05.903152+0000","last_clean":"2024-05-09T08:31:05.903152+0000","last_became_active":"2024-05-09T08:31:04.857387+0000","last_became_peered":"2024-05-09T08:31:04.857387+0000","last_unstale":"2024-05-09T08:31:05.903152+0000","last_undegraded":"2024-05-09T08:31:05.903152+0000","last_fullsized":"2024-05-09T08:31:05.903152+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:50:16.555987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852097+0000","last_change":"2024-05-09T08:30:59.827082+0000","last_active":"2024-05-09T08:31:05.852097+0000","last_peered":"2024-05-09T08:31:05.852097+0000","last_clean":"2024-05-09T08:31:05.852097+0000","last_became_active":"2024-05-09T08:30:59.826746+0000","last_became_peered":"2024-05-09T08:30:59.826746+0000","last_unstale":"2024-05-09T08:31:05.852097+0000","last_undegraded":"2024-05-09T08:31:05.852097+0000","last_fullsized":"2024-05-09T08:31:05.852097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:49.176889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903291+0000","last_change":"2024-05-09T08:31:00.691902+0000","last_active":"2024-05-09T08:31:05.903291+0000","last_peered":"2024-05-09T08:31:05.903291+0000","last_clean":"2024-05-09T08:31:05.903291+0000","last_became_active":"2024-05-09T08:31:00.689491+0000","last_became_peered":"2024-05-09T08:31:00.689491+0000","last_unstale":"2024-05-09T08:31:05.903291+0000","last_undegraded":"2024-05-09T08:31:05.903291+0000","last_fullsized":"2024-05-09T08:31:05.903291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:48:07.335144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852014+0000","last_change":"2024-05-09T08:31:04.857788+0000","last_active":"2024-05-09T08:31:05.852014+0000","last_peered":"2024-05-09T08:31:05.852014+0000","last_clean":"2024-05-09T08:31:05.852014+0000","last_became_active":"2024-05-09T08:31:04.853831+0000","last_became_peered":"2024-05-09T08:31:04.853831+0000","last_unstale":"2024-05-09T08:31:05.852014+0000","last_undegraded":"2024-05-09T08:31:05.852014+0000","last_fullsized":"2024-05-09T08:31:05.852014+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:08:15.451606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.051381+0000","last_change":"2024-05-09T08:30:59.826482+0000","last_active":"2024-05-09T08:31:19.051381+0000","last_peered":"2024-05-09T08:31:19.051381+0000","last_clean":"2024-05-09T08:31:19.051381+0000","last_became_active":"2024-05-09T08:30:59.826123+0000","last_became_peered":"2024-05-09T08:30:59.826123+0000","last_unstale":"2024-05-09T08:31:19.051381+0000","last_undegraded":"2024-05-09T08:31:19.051381+0000","last_fullsized":"2024-05-09T08:31:19.051381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:38.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924786+0000","last_change":"2024-05-09T08:31:00.693440+0000","last_active":"2024-05-09T08:31:05.924786+0000","last_peered":"2024-05-09T08:31:05.924786+0000","last_clean":"2024-05-09T08:31:05.924786+0000","last_became_active":"2024-05-09T08:31:00.693117+0000","last_became_peered":"2024-05-09T08:31:00.693117+0000","last_unstale":"2024-05-09T08:31:05.924786+0000","last_undegraded":"2024-05-09T08:31:05.924786+0000","last_fullsized":"2024-05-09T08:31:05.924786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:10:01.205605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903764+0000","last_change":"2024-05-09T08:31:04.868120+0000","last_active":"2024-05-09T08:31:05.903764+0000","last_peered":"2024-05-09T08:31:05.903764+0000","last_clean":"2024-05-09T08:31:05.903764+0000","last_became_active":"2024-05-09T08:31:04.867571+0000","last_became_peered":"2024-05-09T08:31:04.867571+0000","last_unstale":"2024-05-09T08:31:05.903764+0000","last_undegraded":"2024-05-09T08:31:05.903764+0000","last_fullsized":"2024-05-09T08:31:05.903764+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:12:36.503972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.052891+0000","last_change":"2024-05-09T08:30:59.826280+0000","last_active":"2024-05-09T08:31:19.052891+0000","last_peered":"2024-05-09T08:31:19.052891+0000","last_clean":"2024-05-09T08:31:19.052891+0000","last_became_active":"2024-05-09T08:30:59.825805+0000","last_became_peered":"2024-05-09T08:30:59.825805+0000","last_unstale":"2024-05-09T08:31:19.052891+0000","last_undegraded":"2024-05-09T08:31:19.052891+0000","last_fullsized":"2024-05-09T08:31:19.052891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:46:03.833159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902843+0000","last_change":"2024-05-09T08:31:00.687581+0000","last_active":"2024-05-09T08:31:05.902843+0000","last_peered":"2024-05-09T08:31:05.902843+0000","last_clean":"2024-05-09T08:31:05.902843+0000","last_became_active":"2024-05-09T08:31:00.684608+0000","last_became_peered":"2024-05-09T08:31:00.684608+0000","last_unstale":"2024-05-09T08:31:05.902843+0000","last_undegraded":"2024-05-09T08:31:05.902843+0000","last_fullsized":"2024-05-09T08:31:05.902843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:06:06.436840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852178+0000","last_change":"2024-05-09T08:31:04.859399+0000","last_active":"2024-05-09T08:31:05.852178+0000","last_peered":"2024-05-09T08:31:05.852178+0000","last_clean":"2024-05-09T08:31:05.852178+0000","last_became_active":"2024-05-09T08:31:04.859027+0000","last_became_peered":"2024-05-09T08:31:04.859027+0000","last_unstale":"2024-05-09T08:31:05.852178+0000","last_undegraded":"2024-05-09T08:31:05.852178+0000","last_fullsized":"2024-05-09T08:31:05.852178+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:18:00.829342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.052609+0000","last_change":"2024-05-09T08:30:59.824171+0000","last_active":"2024-05-09T08:31:19.052609+0000","last_peered":"2024-05-09T08:31:19.052609+0000","last_clean":"2024-05-09T08:31:19.052609+0000","last_became_active":"2024-05-09T08:30:59.823859+0000","last_became_peered":"2024-05-09T08:30:59.823859+0000","last_unstale":"2024-05-09T08:31:19.052609+0000","last_undegraded":"2024-05-09T08:31:19.052609+0000","last_fullsized":"2024-05-09T08:31:19.052609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:30.199504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903514+0000","last_change":"2024-05-09T08:31:00.700056+0000","last_active":"2024-05-09T08:31:05.903514+0000","last_peered":"2024-05-09T08:31:05.903514+0000","last_clean":"2024-05-09T08:31:05.903514+0000","last_became_active":"2024-05-09T08:31:00.699673+0000","last_became_peered":"2024-05-09T08:31:00.699673+0000","last_unstale":"2024-05-09T08:31:05.903514+0000","last_undegraded":"2024-05-09T08:31:05.903514+0000","last_fullsized":"2024-05-09T08:31:05.903514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:23.873645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903599+0000","last_change":"2024-05-09T08:31:04.858940+0000","last_active":"2024-05-09T08:31:05.903599+0000","last_peered":"2024-05-09T08:31:05.903599+0000","last_clean":"2024-05-09T08:31:05.903599+0000","last_became_active":"2024-05-09T08:31:04.858532+0000","last_became_peered":"2024-05-09T08:31:04.858532+0000","last_unstale":"2024-05-09T08:31:05.903599+0000","last_undegraded":"2024-05-09T08:31:05.903599+0000","last_fullsized":"2024-05-09T08:31:05.903599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:57:42.128068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852056+0000","last_change":"2024-05-09T08:30:59.824182+0000","last_active":"2024-05-09T08:31:05.852056+0000","last_peered":"2024-05-09T08:31:05.852056+0000","last_clean":"2024-05-09T08:31:05.852056+0000","last_became_active":"2024-05-09T08:30:59.823691+0000","last_became_peered":"2024-05-09T08:30:59.823691+0000","last_unstale":"2024-05-09T08:31:05.852056+0000","last_undegraded":"2024-05-09T08:31:05.852056+0000","last_fullsized":"2024-05-09T08:31:05.852056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:28:07.810733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903637+0000","last_change":"2024-05-09T08:31:00.692614+0000","last_active":"2024-05-09T08:31:05.903637+0000","last_peered":"2024-05-09T08:31:05.903637+0000","last_clean":"2024-05-09T08:31:05.903637+0000","last_became_active":"2024-05-09T08:31:00.692280+0000","last_became_peered":"2024-05-09T08:31:00.692280+0000","last_unstale":"2024-05-09T08:31:05.903637+0000","last_undegraded":"2024-05-09T08:31:05.903637+0000","last_fullsized":"2024-05-09T08:31:05.903637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:23:15.398196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903133+0000","last_change":"2024-05-09T08:31:04.854029+0000","last_active":"2024-05-09T08:31:05.903133+0000","last_peered":"2024-05-09T08:31:05.903133+0000","last_clean":"2024-05-09T08:31:05.903133+0000","last_became_active":"2024-05-09T08:31:04.853554+0000","last_became_peered":"2024-05-09T08:31:04.853554+0000","last_unstale":"2024-05-09T08:31:05.903133+0000","last_undegraded":"2024-05-09T08:31:05.903133+0000","last_fullsized":"2024-05-09T08:31:05.903133+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:34:41.059614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.051300+0000","last_change":"2024-05-09T08:30:59.824512+0000","last_active":"2024-05-09T08:31:19.051300+0000","last_peered":"2024-05-09T08:31:19.051300+0000","last_clean":"2024-05-09T08:31:19.051300+0000","last_became_active":"2024-05-09T08:30:59.824146+0000","last_became_peered":"2024-05-09T08:30:59.824146+0000","last_unstale":"2024-05-09T08:31:19.051300+0000","last_undegraded":"2024-05-09T08:31:19.051300+0000","last_fullsized":"2024-05-09T08:31:19.051300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:27.734292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852224+0000","last_change":"2024-05-09T08:31:00.697272+0000","last_active":"2024-05-09T08:31:05.852224+0000","last_peered":"2024-05-09T08:31:05.852224+0000","last_clean":"2024-05-09T08:31:05.852224+0000","last_became_active":"2024-05-09T08:31:00.696026+0000","last_became_peered":"2024-05-09T08:31:00.696026+0000","last_unstale":"2024-05-09T08:31:05.852224+0000","last_undegraded":"2024-05-09T08:31:05.852224+0000","last_fullsized":"2024-05-09T08:31:05.852224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:04.376142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902852+0000","last_change":"2024-05-09T08:31:04.856641+0000","last_active":"2024-05-09T08:31:05.902852+0000","last_peered":"2024-05-09T08:31:05.902852+0000","last_clean":"2024-05-09T08:31:05.902852+0000","last_became_active":"2024-05-09T08:31:04.856102+0000","last_became_peered":"2024-05-09T08:31:04.856102+0000","last_unstale":"2024-05-09T08:31:05.902852+0000","last_undegraded":"2024-05-09T08:31:05.902852+0000","last_fullsized":"2024-05-09T08:31:05.902852+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:35:49.831076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:20.064491+0000","last_change":"2024-05-09T08:30:59.826808+0000","last_active":"2024-05-09T08:31:20.064491+0000","last_peered":"2024-05-09T08:31:20.064491+0000","last_clean":"2024-05-09T08:31:20.064491+0000","last_became_active":"2024-05-09T08:30:59.826481+0000","last_became_peered":"2024-05-09T08:30:59.826481+0000","last_unstale":"2024-05-09T08:31:20.064491+0000","last_undegraded":"2024-05-09T08:31:20.064491+0000","last_fullsized":"2024-05-09T08:31:20.064491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:27:37.820758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852657+0000","last_change":"2024-05-09T08:31:00.697876+0000","last_active":"2024-05-09T08:31:05.852657+0000","last_peered":"2024-05-09T08:31:05.852657+0000","last_clean":"2024-05-09T08:31:05.852657+0000","last_became_active":"2024-05-09T08:31:00.696299+0000","last_became_peered":"2024-05-09T08:31:00.696299+0000","last_unstale":"2024-05-09T08:31:05.852657+0000","last_undegraded":"2024-05-09T08:31:05.852657+0000","last_fullsized":"2024-05-09T08:31:05.852657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:48:57.373734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924502+0000","last_change":"2024-05-09T08:31:04.858588+0000","last_active":"2024-05-09T08:31:05.924502+0000","last_peered":"2024-05-09T08:31:05.924502+0000","last_clean":"2024-05-09T08:31:05.924502+0000","last_became_active":"2024-05-09T08:31:04.858154+0000","last_became_peered":"2024-05-09T08:31:04.858154+0000","last_unstale":"2024-05-09T08:31:05.924502+0000","last_undegraded":"2024-05-09T08:31:05.924502+0000","last_fullsized":"2024-05-09T08:31:05.924502+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:06:01.768743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.905078+0000","last_change":"2024-05-09T08:30:59.826066+0000","last_active":"2024-05-09T08:31:05.905078+0000","last_peered":"2024-05-09T08:31:05.905078+0000","last_clean":"2024-05-09T08:31:05.905078+0000","last_became_active":"2024-05-09T08:30:59.825586+0000","last_became_peered":"2024-05-09T08:30:59.825586+0000","last_unstale":"2024-05-09T08:31:05.905078+0000","last_undegraded":"2024-05-09T08:31:05.905078+0000","last_fullsized":"2024-05-09T08:31:05.905078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:19:04.629132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852335+0000","last_change":"2024-05-09T08:31:00.693788+0000","last_active":"2024-05-09T08:31:05.852335+0000","last_peered":"2024-05-09T08:31:05.852335+0000","last_clean":"2024-05-09T08:31:05.852335+0000","last_became_active":"2024-05-09T08:31:00.693448+0000","last_became_peered":"2024-05-09T08:31:00.693448+0000","last_unstale":"2024-05-09T08:31:05.852335+0000","last_undegraded":"2024-05-09T08:31:05.852335+0000","last_fullsized":"2024-05-09T08:31:05.852335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:06.516534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903801+0000","last_change":"2024-05-09T08:31:04.856185+0000","last_active":"2024-05-09T08:31:05.903801+0000","last_peered":"2024-05-09T08:31:05.903801+0000","last_clean":"2024-05-09T08:31:05.903801+0000","last_became_active":"2024-05-09T08:31:04.855779+0000","last_became_peered":"2024-05-09T08:31:04.855779+0000","last_unstale":"2024-05-09T08:31:05.903801+0000","last_undegraded":"2024-05-09T08:31:05.903801+0000","last_fullsized":"2024-05-09T08:31:05.903801+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:45:25.034077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902968+0000","last_change":"2024-05-09T08:30:59.822938+0000","last_active":"2024-05-09T08:31:05.902968+0000","last_peered":"2024-05-09T08:31:05.902968+0000","last_clean":"2024-05-09T08:31:05.902968+0000","last_became_active":"2024-05-09T08:30:59.822629+0000","last_became_peered":"2024-05-09T08:30:59.822629+0000","last_unstale":"2024-05-09T08:31:05.902968+0000","last_undegraded":"2024-05-09T08:31:05.902968+0000","last_fullsized":"2024-05-09T08:31:05.902968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:51.321350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852629+0000","last_change":"2024-05-09T08:31:00.691971+0000","last_active":"2024-05-09T08:31:05.852629+0000","last_peered":"2024-05-09T08:31:05.852629+0000","last_clean":"2024-05-09T08:31:05.852629+0000","last_became_active":"2024-05-09T08:31:00.691641+0000","last_became_peered":"2024-05-09T08:31:00.691641+0000","last_unstale":"2024-05-09T08:31:05.852629+0000","last_undegraded":"2024-05-09T08:31:05.852629+0000","last_fullsized":"2024-05-09T08:31:05.852629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:28.413886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903668+0000","last_change":"2024-05-09T08:31:00.701026+0000","last_active":"2024-05-09T08:31:05.903668+0000","last_peered":"2024-05-09T08:31:05.903668+0000","last_clean":"2024-05-09T08:31:05.903668+0000","last_became_active":"2024-05-09T08:31:00.700457+0000","last_became_peered":"2024-05-09T08:31:00.700457+0000","last_unstale":"2024-05-09T08:31:05.903668+0000","last_undegraded":"2024-05-09T08:31:05.903668+0000","last_fullsized":"2024-05-09T08:31:05.903668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:40.065173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852179+0000","last_change":"2024-05-09T08:31:00.699212+0000","last_active":"2024-05-09T08:31:05.852179+0000","last_peered":"2024-05-09T08:31:05.852179+0000","last_clean":"2024-05-09T08:31:05.852179+0000","last_became_active":"2024-05-09T08:31:00.698794+0000","last_became_peered":"2024-05-09T08:31:00.698794+0000","last_unstale":"2024-05-09T08:31:05.852179+0000","last_undegraded":"2024-05-09T08:31:05.852179+0000","last_fullsized":"2024-05-09T08:31:05.852179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:13:21.834839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.851906+0000","last_change":"2024-05-09T08:31:00.703201+0000","last_active":"2024-05-09T08:31:05.851906+0000","last_peered":"2024-05-09T08:31:05.851906+0000","last_clean":"2024-05-09T08:31:05.851906+0000","last_became_active":"2024-05-09T08:31:00.702801+0000","last_became_peered":"2024-05-09T08:31:00.702801+0000","last_unstale":"2024-05-09T08:31:05.851906+0000","last_undegraded":"2024-05-09T08:31:05.851906+0000","last_fullsized":"2024-05-09T08:31:05.851906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:58:35.106013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924791+0000","last_change":"2024-05-09T08:31:00.693696+0000","last_active":"2024-05-09T08:31:05.924791+0000","last_peered":"2024-05-09T08:31:05.924791+0000","last_clean":"2024-05-09T08:31:05.924791+0000","last_became_active":"2024-05-09T08:31:00.693394+0000","last_became_peered":"2024-05-09T08:31:00.693394+0000","last_unstale":"2024-05-09T08:31:05.924791+0000","last_undegraded":"2024-05-09T08:31:05.924791+0000","last_fullsized":"2024-05-09T08:31:05.924791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:01.700968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903684+0000","last_change":"2024-05-09T08:31:00.694363+0000","last_active":"2024-05-09T08:31:05.903684+0000","last_peered":"2024-05-09T08:31:05.903684+0000","last_clean":"2024-05-09T08:31:05.903684+0000","last_became_active":"2024-05-09T08:31:00.693970+0000","last_became_peered":"2024-05-09T08:31:00.693970+0000","last_unstale":"2024-05-09T08:31:05.903684+0000","last_undegraded":"2024-05-09T08:31:05.903684+0000","last_fullsized":"2024-05-09T08:31:05.903684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:56.185801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903904+0000","last_change":"2024-05-09T08:31:00.691970+0000","last_active":"2024-05-09T08:31:05.903904+0000","last_peered":"2024-05-09T08:31:05.903904+0000","last_clean":"2024-05-09T08:31:05.903904+0000","last_became_active":"2024-05-09T08:31:00.691632+0000","last_became_peered":"2024-05-09T08:31:00.691632+0000","last_unstale":"2024-05-09T08:31:05.903904+0000","last_undegraded":"2024-05-09T08:31:05.903904+0000","last_fullsized":"2024-05-09T08:31:05.903904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:04:49.316808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852658+0000","last_change":"2024-05-09T08:31:00.692042+0000","last_active":"2024-05-09T08:31:05.852658+0000","last_peered":"2024-05-09T08:31:05.852658+0000","last_clean":"2024-05-09T08:31:05.852658+0000","last_became_active":"2024-05-09T08:31:00.691700+0000","last_became_peered":"2024-05-09T08:31:00.691700+0000","last_unstale":"2024-05-09T08:31:05.852658+0000","last_undegraded":"2024-05-09T08:31:05.852658+0000","last_fullsized":"2024-05-09T08:31:05.852658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:54.236319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903520+0000","last_change":"2024-05-09T08:31:00.697261+0000","last_active":"2024-05-09T08:31:05.903520+0000","last_peered":"2024-05-09T08:31:05.903520+0000","last_clean":"2024-05-09T08:31:05.903520+0000","last_became_active":"2024-05-09T08:31:00.696299+0000","last_became_peered":"2024-05-09T08:31:00.696299+0000","last_unstale":"2024-05-09T08:31:05.903520+0000","last_undegraded":"2024-05-09T08:31:05.903520+0000","last_fullsized":"2024-05-09T08:31:05.903520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:06:52.061740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904115+0000","last_change":"2024-05-09T08:31:00.691097+0000","last_active":"2024-05-09T08:31:05.904115+0000","last_peered":"2024-05-09T08:31:05.904115+0000","last_clean":"2024-05-09T08:31:05.904115+0000","last_became_active":"2024-05-09T08:31:00.690487+0000","last_became_peered":"2024-05-09T08:31:00.690487+0000","last_unstale":"2024-05-09T08:31:05.904115+0000","last_undegraded":"2024-05-09T08:31:05.904115+0000","last_fullsized":"2024-05-09T08:31:05.904115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:47:26.468296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903783+0000","last_change":"2024-05-09T08:31:00.703201+0000","last_active":"2024-05-09T08:31:05.903783+0000","last_peered":"2024-05-09T08:31:05.903783+0000","last_clean":"2024-05-09T08:31:05.903783+0000","last_became_active":"2024-05-09T08:31:00.702853+0000","last_became_peered":"2024-05-09T08:31:00.702853+0000","last_unstale":"2024-05-09T08:31:05.903783+0000","last_undegraded":"2024-05-09T08:31:05.903783+0000","last_fullsized":"2024-05-09T08:31:05.903783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:18.510321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902931+0000","last_change":"2024-05-09T08:31:00.703958+0000","last_active":"2024-05-09T08:31:05.902931+0000","last_peered":"2024-05-09T08:31:05.902931+0000","last_clean":"2024-05-09T08:31:05.902931+0000","last_became_active":"2024-05-09T08:31:00.703110+0000","last_became_peered":"2024-05-09T08:31:00.703110+0000","last_unstale":"2024-05-09T08:31:05.902931+0000","last_undegraded":"2024-05-09T08:31:05.902931+0000","last_fullsized":"2024-05-09T08:31:05.902931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:37:32.371869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902749+0000","last_change":"2024-05-09T08:31:00.686763+0000","last_active":"2024-05-09T08:31:05.902749+0000","last_peered":"2024-05-09T08:31:05.902749+0000","last_clean":"2024-05-09T08:31:05.902749+0000","last_became_active":"2024-05-09T08:31:00.685642+0000","last_became_peered":"2024-05-09T08:31:00.685642+0000","last_unstale":"2024-05-09T08:31:05.902749+0000","last_undegraded":"2024-05-09T08:31:05.902749+0000","last_fullsized":"2024-05-09T08:31:05.902749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:40:18.201247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904038+0000","last_change":"2024-05-09T08:31:00.697160+0000","last_active":"2024-05-09T08:31:05.904038+0000","last_peered":"2024-05-09T08:31:05.904038+0000","last_clean":"2024-05-09T08:31:05.904038+0000","last_became_active":"2024-05-09T08:31:00.696374+0000","last_became_peered":"2024-05-09T08:31:00.696374+0000","last_unstale":"2024-05-09T08:31:05.904038+0000","last_undegraded":"2024-05-09T08:31:05.904038+0000","last_fullsized":"2024-05-09T08:31:05.904038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:49.082049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7362,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":110,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":26339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383697,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":477797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645333,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28580,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716572,"statfs":{"total":95995035648,"available":95965769728,"internally_reserved":0,"allocated":1601536,"data_stored":1071619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808858,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":987136,"data_stored":477797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070496,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":477953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36892,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708260,"statfs":{"total":95995035648,"available":95957258240,"internally_reserved":0,"allocated":1593344,"data_stored":1069766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":481057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":482407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":477499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":477859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36904,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708248,"statfs":{"total":95995035648,"available":95957245952,"internally_reserved":0,"allocated":1605632,"data_stored":1069059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":10442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:31:23.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T08:31:23.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.905+0000 7ff914c5d640 1 -- 172.21.15.107:0/3428861809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff910101460 msgr2=0x7ff910108d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:23.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.905+0000 7ff914c5d640 1 --2- 172.21.15.107:0/3428861809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff910101460 0x7ff910108d00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff908009cd0 tx=0x7ff90802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.905+0000 7ff914c5d640 1 -- 172.21.15.107:0/3428861809 shutdown_connections 2024-05-09T08:31:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.905+0000 7ff914c5d640 1 --2- 172.21.15.107:0/3428861809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff910101460 0x7ff910108d00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.905+0000 7ff914c5d640 1 --2- 172.21.15.107:0/3428861809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff910100aa0 0x7ff910100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.905+0000 7ff914c5d640 1 --2- 172.21.15.107:0/3428861809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff910101e00 0x7ff910102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.905+0000 7ff914c5d640 1 -- 172.21.15.107:0/3428861809 >> 172.21.15.107:0/3428861809 conn(0x7ff9100fb510 msgr2=0x7ff9100fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.905+0000 7ff914c5d640 1 -- 172.21.15.107:0/3428861809 shutdown_connections 2024-05-09T08:31:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.905+0000 7ff914c5d640 1 -- 172.21.15.107:0/3428861809 wait complete. 2024-05-09T08:31:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.906+0000 7ff914c5d640 1 Processor -- start 2024-05-09T08:31:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.906+0000 7ff914c5d640 1 -- start start 2024-05-09T08:31:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.906+0000 7ff914c5d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff910100aa0 0x7ff910100500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.906+0000 7ff914c5d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff910101460 0x7ff91019ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.906+0000 7ff914c5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff910101e00 0x7ff9101a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.906+0000 7ff914c5d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff91010c370 con 0x7ff910101e00 2024-05-09T08:31:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.906+0000 7ff914c5d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff91010c1f0 con 0x7ff910101460 2024-05-09T08:31:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.906+0000 7ff914c5d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff91010c4f0 con 0x7ff910100aa0 2024-05-09T08:31:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.906+0000 7ff90ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff910101e00 0x7ff9101a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.906+0000 7ff90ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff910101e00 0x7ff9101a11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53330/0 (socket says 172.21.15.107:53330) 2024-05-09T08:31:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.906+0000 7ff90e59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff910100aa0 0x7ff910100500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.906+0000 7ff90ed9d640 1 -- 172.21.15.107:0/3521939624 learned_addr learned my addr 172.21.15.107:0/3521939624 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.907+0000 7ff905d9b640 1 --2- 172.21.15.107:0/3521939624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff910101460 0x7ff91019ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.907+0000 7ff90ed9d640 1 -- 172.21.15.107:0/3521939624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff910100aa0 msgr2=0x7ff910100500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.907+0000 7ff90ed9d640 1 --2- 172.21.15.107:0/3521939624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff910100aa0 0x7ff910100500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.907+0000 7ff90ed9d640 1 -- 172.21.15.107:0/3521939624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff910101460 msgr2=0x7ff91019ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.907+0000 7ff90ed9d640 1 --2- 172.21.15.107:0/3521939624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff910101460 0x7ff91019ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:23.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.907+0000 7ff90ed9d640 1 -- 172.21.15.107:0/3521939624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9101a19d0 con 0x7ff910101e00 2024-05-09T08:31:23.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.907+0000 7ff90ed9d640 1 --2- 172.21.15.107:0/3521939624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff910101e00 0x7ff9101a11e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff9080318b0 tx=0x7ff908003b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:23.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.907+0000 7ff907fff640 1 -- 172.21.15.107:0/3521939624 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff908004040 con 0x7ff910101e00 2024-05-09T08:31:23.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.907+0000 7ff907fff640 1 -- 172.21.15.107:0/3521939624 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff908004690 con 0x7ff910101e00 2024-05-09T08:31:23.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.907+0000 7ff907fff640 1 -- 172.21.15.107:0/3521939624 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff90803b400 con 0x7ff910101e00 2024-05-09T08:31:23.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.907+0000 7ff914c5d640 1 -- 172.21.15.107:0/3521939624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9101a1c00 con 0x7ff910101e00 2024-05-09T08:31:23.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.907+0000 7ff914c5d640 1 -- 172.21.15.107:0/3521939624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9101a2150 con 0x7ff910101e00 2024-05-09T08:31:23.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.908+0000 7ff914c5d640 1 -- 172.21.15.107:0/3521939624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff910107c80 con 0x7ff910101e00 2024-05-09T08:31:23.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.909+0000 7ff907fff640 1 -- 172.21.15.107:0/3521939624 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff9080306a0 con 0x7ff910101e00 2024-05-09T08:31:23.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.910+0000 7ff907fff640 1 --2- 172.21.15.107:0/3521939624 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff8fc0a2c50 0x7ff8fc0a5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:23.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.910+0000 7ff90e59c640 1 --2- 172.21.15.107:0/3521939624 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff8fc0a2c50 0x7ff8fc0a5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:23.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.910+0000 7ff907fff640 1 -- 172.21.15.107:0/3521939624 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff90802e030 con 0x7ff910101e00 2024-05-09T08:31:23.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.910+0000 7ff90e59c640 1 --2- 172.21.15.107:0/3521939624 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff8fc0a2c50 0x7ff8fc0a5110 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff910069cd0 tx=0x7ff900005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:23.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:23.914+0000 7ff907fff640 1 -- 172.21.15.107:0/3521939624 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff9080bd190 con 0x7ff910101e00 2024-05-09T08:31:24.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:24.089+0000 7ff914c5d640 1 -- 172.21.15.107:0/3521939624 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff910061c70 con 0x7ff8fc0a2c50 2024-05-09T08:31:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:24.098+0000 7ff907fff640 1 -- 172.21.15.107:0/3521939624 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340300 (secure 0 0 0) 0x7ff910061c70 con 0x7ff8fc0a2c50 2024-05-09T08:31:24.102 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:24.104 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-05-09T08:31:24.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:24.101+0000 7ff914c5d640 1 -- 172.21.15.107:0/3521939624 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff8fc0a2c50 msgr2=0x7ff8fc0a5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:24.101+0000 7ff914c5d640 1 --2- 172.21.15.107:0/3521939624 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff8fc0a2c50 0x7ff8fc0a5110 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff910069cd0 tx=0x7ff900005f90 comp rx=0 tx=0).stop 2024-05-09T08:31:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:24.101+0000 7ff914c5d640 1 -- 172.21.15.107:0/3521939624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff910101e00 msgr2=0x7ff9101a11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:24.101+0000 7ff914c5d640 1 --2- 172.21.15.107:0/3521939624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff910101e00 0x7ff9101a11e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff9080318b0 tx=0x7ff908003b80 comp rx=0 tx=0).stop 2024-05-09T08:31:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:24.101+0000 7ff914c5d640 1 -- 172.21.15.107:0/3521939624 shutdown_connections 2024-05-09T08:31:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:24.101+0000 7ff914c5d640 1 --2- 172.21.15.107:0/3521939624 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff8fc0a2c50 0x7ff8fc0a5110 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:24.101+0000 7ff914c5d640 1 --2- 172.21.15.107:0/3521939624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff910101e00 0x7ff9101a11e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:24.101+0000 7ff914c5d640 1 --2- 172.21.15.107:0/3521939624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff910101460 0x7ff91019ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:24.101+0000 7ff914c5d640 1 --2- 172.21.15.107:0/3521939624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff910100aa0 0x7ff910100500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:24.101+0000 7ff914c5d640 1 -- 172.21.15.107:0/3521939624 >> 172.21.15.107:0/3521939624 conn(0x7ff9100fb510 msgr2=0x7ff910104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:24.102+0000 7ff914c5d640 1 -- 172.21.15.107:0/3521939624 shutdown_connections 2024-05-09T08:31:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:24.102+0000 7ff914c5d640 1 -- 172.21.15.107:0/3521939624 wait complete. 2024-05-09T08:31:24.112 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-05-09T08:31:23.457834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597730,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":143,"num_read_kb":126,"num_write":291,"num_write_kb":2980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":391732,"kb_used_data":13436,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124550092,"statfs":{"total":1151940427776,"available":1151539294208,"internally_reserved":0,"allocated":13758464,"data_stored":7517333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":387233081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4952,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":65,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002987"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903504+0000","last_change":"2024-05-09T08:31:00.695034+0000","last_active":"2024-05-09T08:31:05.903504+0000","last_peered":"2024-05-09T08:31:05.903504+0000","last_clean":"2024-05-09T08:31:05.903504+0000","last_became_active":"2024-05-09T08:31:00.694229+0000","last_became_peered":"2024-05-09T08:31:00.694229+0000","last_unstale":"2024-05-09T08:31:05.903504+0000","last_undegraded":"2024-05-09T08:31:05.903504+0000","last_fullsized":"2024-05-09T08:31:05.903504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:57:22.964848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924693+0000","last_change":"2024-05-09T08:31:00.686219+0000","last_active":"2024-05-09T08:31:05.924693+0000","last_peered":"2024-05-09T08:31:05.924693+0000","last_clean":"2024-05-09T08:31:05.924693+0000","last_became_active":"2024-05-09T08:31:00.685911+0000","last_became_peered":"2024-05-09T08:31:00.685911+0000","last_unstale":"2024-05-09T08:31:05.924693+0000","last_undegraded":"2024-05-09T08:31:05.924693+0000","last_fullsized":"2024-05-09T08:31:05.924693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:28.246199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924448+0000","last_change":"2024-05-09T08:31:00.694228+0000","last_active":"2024-05-09T08:31:05.924448+0000","last_peered":"2024-05-09T08:31:05.924448+0000","last_clean":"2024-05-09T08:31:05.924448+0000","last_became_active":"2024-05-09T08:31:00.693838+0000","last_became_peered":"2024-05-09T08:31:00.693838+0000","last_unstale":"2024-05-09T08:31:05.924448+0000","last_undegraded":"2024-05-09T08:31:05.924448+0000","last_fullsized":"2024-05-09T08:31:05.924448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:44.847966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903457+0000","last_change":"2024-05-09T08:31:00.702770+0000","last_active":"2024-05-09T08:31:05.903457+0000","last_peered":"2024-05-09T08:31:05.903457+0000","last_clean":"2024-05-09T08:31:05.903457+0000","last_became_active":"2024-05-09T08:31:00.702424+0000","last_became_peered":"2024-05-09T08:31:00.702424+0000","last_unstale":"2024-05-09T08:31:05.903457+0000","last_undegraded":"2024-05-09T08:31:05.903457+0000","last_fullsized":"2024-05-09T08:31:05.903457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:24:05.228942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902921+0000","last_change":"2024-05-09T08:31:00.688308+0000","last_active":"2024-05-09T08:31:05.902921+0000","last_peered":"2024-05-09T08:31:05.902921+0000","last_clean":"2024-05-09T08:31:05.902921+0000","last_became_active":"2024-05-09T08:31:00.687891+0000","last_became_peered":"2024-05-09T08:31:00.687891+0000","last_unstale":"2024-05-09T08:31:05.902921+0000","last_undegraded":"2024-05-09T08:31:05.902921+0000","last_fullsized":"2024-05-09T08:31:05.902921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:48:42.648940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903780+0000","last_change":"2024-05-09T08:31:00.694705+0000","last_active":"2024-05-09T08:31:05.903780+0000","last_peered":"2024-05-09T08:31:05.903780+0000","last_clean":"2024-05-09T08:31:05.903780+0000","last_became_active":"2024-05-09T08:31:00.694361+0000","last_became_peered":"2024-05-09T08:31:00.694361+0000","last_unstale":"2024-05-09T08:31:05.903780+0000","last_undegraded":"2024-05-09T08:31:05.903780+0000","last_fullsized":"2024-05-09T08:31:05.903780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:07:23.137410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.851966+0000","last_change":"2024-05-09T08:31:00.689824+0000","last_active":"2024-05-09T08:31:05.851966+0000","last_peered":"2024-05-09T08:31:05.851966+0000","last_clean":"2024-05-09T08:31:05.851966+0000","last_became_active":"2024-05-09T08:31:00.689491+0000","last_became_peered":"2024-05-09T08:31:00.689491+0000","last_unstale":"2024-05-09T08:31:05.851966+0000","last_undegraded":"2024-05-09T08:31:05.851966+0000","last_fullsized":"2024-05-09T08:31:05.851966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:18.791225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902743+0000","last_change":"2024-05-09T08:31:00.695014+0000","last_active":"2024-05-09T08:31:05.902743+0000","last_peered":"2024-05-09T08:31:05.902743+0000","last_clean":"2024-05-09T08:31:05.902743+0000","last_became_active":"2024-05-09T08:31:00.694683+0000","last_became_peered":"2024-05-09T08:31:00.694683+0000","last_unstale":"2024-05-09T08:31:05.902743+0000","last_undegraded":"2024-05-09T08:31:05.902743+0000","last_fullsized":"2024-05-09T08:31:05.902743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:08.570258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904494+0000","last_change":"2024-05-09T08:31:00.701538+0000","last_active":"2024-05-09T08:31:05.904494+0000","last_peered":"2024-05-09T08:31:05.904494+0000","last_clean":"2024-05-09T08:31:05.904494+0000","last_became_active":"2024-05-09T08:31:00.701226+0000","last_became_peered":"2024-05-09T08:31:00.701226+0000","last_unstale":"2024-05-09T08:31:05.904494+0000","last_undegraded":"2024-05-09T08:31:05.904494+0000","last_fullsized":"2024-05-09T08:31:05.904494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:14:50.853063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904210+0000","last_change":"2024-05-09T08:31:00.698145+0000","last_active":"2024-05-09T08:31:05.904210+0000","last_peered":"2024-05-09T08:31:05.904210+0000","last_clean":"2024-05-09T08:31:05.904210+0000","last_became_active":"2024-05-09T08:31:00.697761+0000","last_became_peered":"2024-05-09T08:31:00.697761+0000","last_unstale":"2024-05-09T08:31:05.904210+0000","last_undegraded":"2024-05-09T08:31:05.904210+0000","last_fullsized":"2024-05-09T08:31:05.904210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:26.094013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852595+0000","last_change":"2024-05-09T08:31:00.699520+0000","last_active":"2024-05-09T08:31:05.852595+0000","last_peered":"2024-05-09T08:31:05.852595+0000","last_clean":"2024-05-09T08:31:05.852595+0000","last_became_active":"2024-05-09T08:31:00.697579+0000","last_became_peered":"2024-05-09T08:31:00.697579+0000","last_unstale":"2024-05-09T08:31:05.852595+0000","last_undegraded":"2024-05-09T08:31:05.852595+0000","last_fullsized":"2024-05-09T08:31:05.852595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:26:34.324474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903344+0000","last_change":"2024-05-09T08:31:00.693356+0000","last_active":"2024-05-09T08:31:05.903344+0000","last_peered":"2024-05-09T08:31:05.903344+0000","last_clean":"2024-05-09T08:31:05.903344+0000","last_became_active":"2024-05-09T08:31:00.692985+0000","last_became_peered":"2024-05-09T08:31:00.692985+0000","last_unstale":"2024-05-09T08:31:05.903344+0000","last_undegraded":"2024-05-09T08:31:05.903344+0000","last_fullsized":"2024-05-09T08:31:05.903344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:49:01.115497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902630+0000","last_change":"2024-05-09T08:31:00.677768+0000","last_active":"2024-05-09T08:31:05.902630+0000","last_peered":"2024-05-09T08:31:05.902630+0000","last_clean":"2024-05-09T08:31:05.902630+0000","last_became_active":"2024-05-09T08:31:00.677386+0000","last_became_peered":"2024-05-09T08:31:00.677386+0000","last_unstale":"2024-05-09T08:31:05.902630+0000","last_undegraded":"2024-05-09T08:31:05.902630+0000","last_fullsized":"2024-05-09T08:31:05.902630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:55.363324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924381+0000","last_change":"2024-05-09T08:31:00.689609+0000","last_active":"2024-05-09T08:31:05.924381+0000","last_peered":"2024-05-09T08:31:05.924381+0000","last_clean":"2024-05-09T08:31:05.924381+0000","last_became_active":"2024-05-09T08:31:00.689303+0000","last_became_peered":"2024-05-09T08:31:00.689303+0000","last_unstale":"2024-05-09T08:31:05.924381+0000","last_undegraded":"2024-05-09T08:31:05.924381+0000","last_fullsized":"2024-05-09T08:31:05.924381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:15:58.672460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924356+0000","last_change":"2024-05-09T08:31:00.704708+0000","last_active":"2024-05-09T08:31:05.924356+0000","last_peered":"2024-05-09T08:31:05.924356+0000","last_clean":"2024-05-09T08:31:05.924356+0000","last_became_active":"2024-05-09T08:31:00.704269+0000","last_became_peered":"2024-05-09T08:31:00.704269+0000","last_unstale":"2024-05-09T08:31:05.924356+0000","last_undegraded":"2024-05-09T08:31:05.924356+0000","last_fullsized":"2024-05-09T08:31:05.924356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:17:44.580420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902413+0000","last_change":"2024-05-09T08:31:00.703089+0000","last_active":"2024-05-09T08:31:05.902413+0000","last_peered":"2024-05-09T08:31:05.902413+0000","last_clean":"2024-05-09T08:31:05.902413+0000","last_became_active":"2024-05-09T08:31:00.702531+0000","last_became_peered":"2024-05-09T08:31:00.702531+0000","last_unstale":"2024-05-09T08:31:05.902413+0000","last_undegraded":"2024-05-09T08:31:05.902413+0000","last_fullsized":"2024-05-09T08:31:05.902413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:07:39.887748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903343+0000","last_change":"2024-05-09T08:31:00.695332+0000","last_active":"2024-05-09T08:31:05.903343+0000","last_peered":"2024-05-09T08:31:05.903343+0000","last_clean":"2024-05-09T08:31:05.903343+0000","last_became_active":"2024-05-09T08:31:00.694703+0000","last_became_peered":"2024-05-09T08:31:00.694703+0000","last_unstale":"2024-05-09T08:31:05.903343+0000","last_undegraded":"2024-05-09T08:31:05.903343+0000","last_fullsized":"2024-05-09T08:31:05.903343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:53.727922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852069+0000","last_change":"2024-05-09T08:31:00.689871+0000","last_active":"2024-05-09T08:31:05.852069+0000","last_peered":"2024-05-09T08:31:05.852069+0000","last_clean":"2024-05-09T08:31:05.852069+0000","last_became_active":"2024-05-09T08:31:00.689498+0000","last_became_peered":"2024-05-09T08:31:00.689498+0000","last_unstale":"2024-05-09T08:31:05.852069+0000","last_undegraded":"2024-05-09T08:31:05.852069+0000","last_fullsized":"2024-05-09T08:31:05.852069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:18:14.123442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852452+0000","last_change":"2024-05-09T08:31:00.697175+0000","last_active":"2024-05-09T08:31:05.852452+0000","last_peered":"2024-05-09T08:31:05.852452+0000","last_clean":"2024-05-09T08:31:05.852452+0000","last_became_active":"2024-05-09T08:31:00.694103+0000","last_became_peered":"2024-05-09T08:31:00.694103+0000","last_unstale":"2024-05-09T08:31:05.852452+0000","last_undegraded":"2024-05-09T08:31:05.852452+0000","last_fullsized":"2024-05-09T08:31:05.852452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:41:12.549854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852261+0000","last_change":"2024-05-09T08:31:04.859128+0000","last_active":"2024-05-09T08:31:05.852261+0000","last_peered":"2024-05-09T08:31:05.852261+0000","last_clean":"2024-05-09T08:31:05.852261+0000","last_became_active":"2024-05-09T08:31:04.858713+0000","last_became_peered":"2024-05-09T08:31:04.858713+0000","last_unstale":"2024-05-09T08:31:05.852261+0000","last_undegraded":"2024-05-09T08:31:05.852261+0000","last_fullsized":"2024-05-09T08:31:05.852261+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:35:22.620160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852195+0000","last_change":"2024-05-09T08:30:59.828030+0000","last_active":"2024-05-09T08:31:05.852195+0000","last_peered":"2024-05-09T08:31:05.852195+0000","last_clean":"2024-05-09T08:31:05.852195+0000","last_became_active":"2024-05-09T08:30:59.827687+0000","last_became_peered":"2024-05-09T08:30:59.827687+0000","last_unstale":"2024-05-09T08:31:05.852195+0000","last_undegraded":"2024-05-09T08:31:05.852195+0000","last_fullsized":"2024-05-09T08:31:05.852195+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:52:57.321232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902914+0000","last_change":"2024-05-09T08:31:00.696023+0000","last_active":"2024-05-09T08:31:05.902914+0000","last_peered":"2024-05-09T08:31:05.902914+0000","last_clean":"2024-05-09T08:31:05.902914+0000","last_became_active":"2024-05-09T08:31:00.695677+0000","last_became_peered":"2024-05-09T08:31:00.695677+0000","last_unstale":"2024-05-09T08:31:05.902914+0000","last_undegraded":"2024-05-09T08:31:05.902914+0000","last_fullsized":"2024-05-09T08:31:05.902914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:55.989133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902977+0000","last_change":"2024-05-09T08:31:04.859302+0000","last_active":"2024-05-09T08:31:05.902977+0000","last_peered":"2024-05-09T08:31:05.902977+0000","last_clean":"2024-05-09T08:31:05.902977+0000","last_became_active":"2024-05-09T08:31:04.858897+0000","last_became_peered":"2024-05-09T08:31:04.858897+0000","last_unstale":"2024-05-09T08:31:05.902977+0000","last_undegraded":"2024-05-09T08:31:05.902977+0000","last_fullsized":"2024-05-09T08:31:05.902977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:25:32.970833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903397+0000","last_change":"2024-05-09T08:30:59.824451+0000","last_active":"2024-05-09T08:31:05.903397+0000","last_peered":"2024-05-09T08:31:05.903397+0000","last_clean":"2024-05-09T08:31:05.903397+0000","last_became_active":"2024-05-09T08:30:59.823964+0000","last_became_peered":"2024-05-09T08:30:59.823964+0000","last_unstale":"2024-05-09T08:31:05.903397+0000","last_undegraded":"2024-05-09T08:31:05.903397+0000","last_fullsized":"2024-05-09T08:31:05.903397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:50:13.562262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924544+0000","last_change":"2024-05-09T08:31:00.701015+0000","last_active":"2024-05-09T08:31:05.924544+0000","last_peered":"2024-05-09T08:31:05.924544+0000","last_clean":"2024-05-09T08:31:05.924544+0000","last_became_active":"2024-05-09T08:31:00.700654+0000","last_became_peered":"2024-05-09T08:31:00.700654+0000","last_unstale":"2024-05-09T08:31:05.924544+0000","last_undegraded":"2024-05-09T08:31:05.924544+0000","last_fullsized":"2024-05-09T08:31:05.924544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:03.710790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902772+0000","last_change":"2024-05-09T08:31:04.858453+0000","last_active":"2024-05-09T08:31:05.902772+0000","last_peered":"2024-05-09T08:31:05.902772+0000","last_clean":"2024-05-09T08:31:05.902772+0000","last_became_active":"2024-05-09T08:31:04.857974+0000","last_became_peered":"2024-05-09T08:31:04.857974+0000","last_unstale":"2024-05-09T08:31:05.902772+0000","last_undegraded":"2024-05-09T08:31:05.902772+0000","last_fullsized":"2024-05-09T08:31:05.902772+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:42:52.868625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:20.064683+0000","last_change":"2024-05-09T08:30:59.825974+0000","last_active":"2024-05-09T08:31:20.064683+0000","last_peered":"2024-05-09T08:31:20.064683+0000","last_clean":"2024-05-09T08:31:20.064683+0000","last_became_active":"2024-05-09T08:30:59.825671+0000","last_became_peered":"2024-05-09T08:30:59.825671+0000","last_unstale":"2024-05-09T08:31:20.064683+0000","last_undegraded":"2024-05-09T08:31:20.064683+0000","last_fullsized":"2024-05-09T08:31:20.064683+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:38:20.179466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903259+0000","last_change":"2024-05-09T08:31:00.698324+0000","last_active":"2024-05-09T08:31:05.903259+0000","last_peered":"2024-05-09T08:31:05.903259+0000","last_clean":"2024-05-09T08:31:05.903259+0000","last_became_active":"2024-05-09T08:31:00.697992+0000","last_became_peered":"2024-05-09T08:31:00.697992+0000","last_unstale":"2024-05-09T08:31:05.903259+0000","last_undegraded":"2024-05-09T08:31:05.903259+0000","last_fullsized":"2024-05-09T08:31:05.903259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:25:20.786423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902683+0000","last_change":"2024-05-09T08:31:04.861946+0000","last_active":"2024-05-09T08:31:05.902683+0000","last_peered":"2024-05-09T08:31:05.902683+0000","last_clean":"2024-05-09T08:31:05.902683+0000","last_became_active":"2024-05-09T08:31:04.861511+0000","last_became_peered":"2024-05-09T08:31:04.861511+0000","last_unstale":"2024-05-09T08:31:05.902683+0000","last_undegraded":"2024-05-09T08:31:05.902683+0000","last_fullsized":"2024-05-09T08:31:05.902683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:36:44.580499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903998+0000","last_change":"2024-05-09T08:30:59.829961+0000","last_active":"2024-05-09T08:31:05.903998+0000","last_peered":"2024-05-09T08:31:05.903998+0000","last_clean":"2024-05-09T08:31:05.903998+0000","last_became_active":"2024-05-09T08:30:59.828057+0000","last_became_peered":"2024-05-09T08:30:59.828057+0000","last_unstale":"2024-05-09T08:31:05.903998+0000","last_undegraded":"2024-05-09T08:31:05.903998+0000","last_fullsized":"2024-05-09T08:31:05.903998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:39:18.229438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904977+0000","last_change":"2024-05-09T08:31:00.700285+0000","last_active":"2024-05-09T08:31:05.904977+0000","last_peered":"2024-05-09T08:31:05.904977+0000","last_clean":"2024-05-09T08:31:05.904977+0000","last_became_active":"2024-05-09T08:31:00.699958+0000","last_became_peered":"2024-05-09T08:31:00.699958+0000","last_unstale":"2024-05-09T08:31:05.904977+0000","last_undegraded":"2024-05-09T08:31:05.904977+0000","last_fullsized":"2024-05-09T08:31:05.904977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:48.934827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852447+0000","last_change":"2024-05-09T08:31:04.857060+0000","last_active":"2024-05-09T08:31:05.852447+0000","last_peered":"2024-05-09T08:31:05.852447+0000","last_clean":"2024-05-09T08:31:05.852447+0000","last_became_active":"2024-05-09T08:31:04.855261+0000","last_became_peered":"2024-05-09T08:31:04.855261+0000","last_unstale":"2024-05-09T08:31:05.852447+0000","last_undegraded":"2024-05-09T08:31:05.852447+0000","last_fullsized":"2024-05-09T08:31:05.852447+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:08:08.230524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.052984+0000","last_change":"2024-05-09T08:30:59.831089+0000","last_active":"2024-05-09T08:31:19.052984+0000","last_peered":"2024-05-09T08:31:19.052984+0000","last_clean":"2024-05-09T08:31:19.052984+0000","last_became_active":"2024-05-09T08:30:59.830572+0000","last_became_peered":"2024-05-09T08:30:59.830572+0000","last_unstale":"2024-05-09T08:31:19.052984+0000","last_undegraded":"2024-05-09T08:31:19.052984+0000","last_fullsized":"2024-05-09T08:31:19.052984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:39:10.495104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903462+0000","last_change":"2024-05-09T08:31:00.688792+0000","last_active":"2024-05-09T08:31:05.903462+0000","last_peered":"2024-05-09T08:31:05.903462+0000","last_clean":"2024-05-09T08:31:05.903462+0000","last_became_active":"2024-05-09T08:31:00.688386+0000","last_became_peered":"2024-05-09T08:31:00.688386+0000","last_unstale":"2024-05-09T08:31:05.903462+0000","last_undegraded":"2024-05-09T08:31:05.903462+0000","last_fullsized":"2024-05-09T08:31:05.903462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:24.287333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852532+0000","last_change":"2024-05-09T08:31:04.862287+0000","last_active":"2024-05-09T08:31:05.852532+0000","last_peered":"2024-05-09T08:31:05.852532+0000","last_clean":"2024-05-09T08:31:05.852532+0000","last_became_active":"2024-05-09T08:31:04.861866+0000","last_became_peered":"2024-05-09T08:31:04.861866+0000","last_unstale":"2024-05-09T08:31:05.852532+0000","last_undegraded":"2024-05-09T08:31:05.852532+0000","last_fullsized":"2024-05-09T08:31:05.852532+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:14:21.552787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903660+0000","last_change":"2024-05-09T08:30:59.824857+0000","last_active":"2024-05-09T08:31:05.903660+0000","last_peered":"2024-05-09T08:31:05.903660+0000","last_clean":"2024-05-09T08:31:05.903660+0000","last_became_active":"2024-05-09T08:30:59.824363+0000","last_became_peered":"2024-05-09T08:30:59.824363+0000","last_unstale":"2024-05-09T08:31:05.903660+0000","last_undegraded":"2024-05-09T08:31:05.903660+0000","last_fullsized":"2024-05-09T08:31:05.903660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:20:32.267237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904922+0000","last_change":"2024-05-09T08:31:00.694440+0000","last_active":"2024-05-09T08:31:05.904922+0000","last_peered":"2024-05-09T08:31:05.904922+0000","last_clean":"2024-05-09T08:31:05.904922+0000","last_became_active":"2024-05-09T08:31:00.694085+0000","last_became_peered":"2024-05-09T08:31:00.694085+0000","last_unstale":"2024-05-09T08:31:05.904922+0000","last_undegraded":"2024-05-09T08:31:05.904922+0000","last_fullsized":"2024-05-09T08:31:05.904922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:03:24.697479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903268+0000","last_change":"2024-05-09T08:31:04.850142+0000","last_active":"2024-05-09T08:31:05.903268+0000","last_peered":"2024-05-09T08:31:05.903268+0000","last_clean":"2024-05-09T08:31:05.903268+0000","last_became_active":"2024-05-09T08:31:04.849689+0000","last_became_peered":"2024-05-09T08:31:04.849689+0000","last_unstale":"2024-05-09T08:31:05.903268+0000","last_undegraded":"2024-05-09T08:31:05.903268+0000","last_fullsized":"2024-05-09T08:31:05.903268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:39:46.811808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:20.062955+0000","last_change":"2024-05-09T08:30:59.818373+0000","last_active":"2024-05-09T08:31:20.062955+0000","last_peered":"2024-05-09T08:31:20.062955+0000","last_clean":"2024-05-09T08:31:20.062955+0000","last_became_active":"2024-05-09T08:30:59.818023+0000","last_became_peered":"2024-05-09T08:30:59.818023+0000","last_unstale":"2024-05-09T08:31:20.062955+0000","last_undegraded":"2024-05-09T08:31:20.062955+0000","last_fullsized":"2024-05-09T08:31:20.062955+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:23:01.994257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903204+0000","last_change":"2024-05-09T08:31:00.689506+0000","last_active":"2024-05-09T08:31:05.903204+0000","last_peered":"2024-05-09T08:31:05.903204+0000","last_clean":"2024-05-09T08:31:05.903204+0000","last_became_active":"2024-05-09T08:31:00.689151+0000","last_became_peered":"2024-05-09T08:31:00.689151+0000","last_unstale":"2024-05-09T08:31:05.903204+0000","last_undegraded":"2024-05-09T08:31:05.903204+0000","last_fullsized":"2024-05-09T08:31:05.903204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:48:13.539509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924721+0000","last_change":"2024-05-09T08:31:04.859215+0000","last_active":"2024-05-09T08:31:05.924721+0000","last_peered":"2024-05-09T08:31:05.924721+0000","last_clean":"2024-05-09T08:31:05.924721+0000","last_became_active":"2024-05-09T08:31:04.858759+0000","last_became_peered":"2024-05-09T08:31:04.858759+0000","last_unstale":"2024-05-09T08:31:05.924721+0000","last_undegraded":"2024-05-09T08:31:05.924721+0000","last_fullsized":"2024-05-09T08:31:05.924721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:41:42.124234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904511+0000","last_change":"2024-05-09T08:30:59.824260+0000","last_active":"2024-05-09T08:31:05.904511+0000","last_peered":"2024-05-09T08:31:05.904511+0000","last_clean":"2024-05-09T08:31:05.904511+0000","last_became_active":"2024-05-09T08:30:59.823959+0000","last_became_peered":"2024-05-09T08:30:59.823959+0000","last_unstale":"2024-05-09T08:31:05.904511+0000","last_undegraded":"2024-05-09T08:31:05.904511+0000","last_fullsized":"2024-05-09T08:31:05.904511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:48:42.354279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.851454+0000","last_change":"2024-05-09T08:31:00.687810+0000","last_active":"2024-05-09T08:31:05.851454+0000","last_peered":"2024-05-09T08:31:05.851454+0000","last_clean":"2024-05-09T08:31:05.851454+0000","last_became_active":"2024-05-09T08:31:00.687486+0000","last_became_peered":"2024-05-09T08:31:00.687486+0000","last_unstale":"2024-05-09T08:31:05.851454+0000","last_undegraded":"2024-05-09T08:31:05.851454+0000","last_fullsized":"2024-05-09T08:31:05.851454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:51.428208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903674+0000","last_change":"2024-05-09T08:31:04.853222+0000","last_active":"2024-05-09T08:31:05.903674+0000","last_peered":"2024-05-09T08:31:05.903674+0000","last_clean":"2024-05-09T08:31:05.903674+0000","last_became_active":"2024-05-09T08:31:04.852676+0000","last_became_peered":"2024-05-09T08:31:04.852676+0000","last_unstale":"2024-05-09T08:31:05.903674+0000","last_undegraded":"2024-05-09T08:31:05.903674+0000","last_fullsized":"2024-05-09T08:31:05.903674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:58:31.949938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903634+0000","last_change":"2024-05-09T08:30:59.824653+0000","last_active":"2024-05-09T08:31:05.903634+0000","last_peered":"2024-05-09T08:31:05.903634+0000","last_clean":"2024-05-09T08:31:05.903634+0000","last_became_active":"2024-05-09T08:30:59.824277+0000","last_became_peered":"2024-05-09T08:30:59.824277+0000","last_unstale":"2024-05-09T08:31:05.903634+0000","last_undegraded":"2024-05-09T08:31:05.903634+0000","last_fullsized":"2024-05-09T08:31:05.903634+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:45:20.340629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904459+0000","last_change":"2024-05-09T08:31:00.697484+0000","last_active":"2024-05-09T08:31:05.904459+0000","last_peered":"2024-05-09T08:31:05.904459+0000","last_clean":"2024-05-09T08:31:05.904459+0000","last_became_active":"2024-05-09T08:31:00.697095+0000","last_became_peered":"2024-05-09T08:31:00.697095+0000","last_unstale":"2024-05-09T08:31:05.904459+0000","last_undegraded":"2024-05-09T08:31:05.904459+0000","last_fullsized":"2024-05-09T08:31:05.904459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:29:40.702057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.851529+0000","last_change":"2024-05-09T08:31:04.860048+0000","last_active":"2024-05-09T08:31:05.851529+0000","last_peered":"2024-05-09T08:31:05.851529+0000","last_clean":"2024-05-09T08:31:05.851529+0000","last_became_active":"2024-05-09T08:31:04.857677+0000","last_became_peered":"2024-05-09T08:31:04.857677+0000","last_unstale":"2024-05-09T08:31:05.851529+0000","last_undegraded":"2024-05-09T08:31:05.851529+0000","last_fullsized":"2024-05-09T08:31:05.851529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:40.850305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.052341+0000","last_change":"2024-05-09T08:30:59.822683+0000","last_active":"2024-05-09T08:31:19.052341+0000","last_peered":"2024-05-09T08:31:19.052341+0000","last_clean":"2024-05-09T08:31:19.052341+0000","last_became_active":"2024-05-09T08:30:59.822150+0000","last_became_peered":"2024-05-09T08:30:59.822150+0000","last_unstale":"2024-05-09T08:31:19.052341+0000","last_undegraded":"2024-05-09T08:31:19.052341+0000","last_fullsized":"2024-05-09T08:31:19.052341+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:51:32.782076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924495+0000","last_change":"2024-05-09T08:31:00.694165+0000","last_active":"2024-05-09T08:31:05.924495+0000","last_peered":"2024-05-09T08:31:05.924495+0000","last_clean":"2024-05-09T08:31:05.924495+0000","last_became_active":"2024-05-09T08:31:00.693150+0000","last_became_peered":"2024-05-09T08:31:00.693150+0000","last_unstale":"2024-05-09T08:31:05.924495+0000","last_undegraded":"2024-05-09T08:31:05.924495+0000","last_fullsized":"2024-05-09T08:31:05.924495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:29.278078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904155+0000","last_change":"2024-05-09T08:31:04.858785+0000","last_active":"2024-05-09T08:31:05.904155+0000","last_peered":"2024-05-09T08:31:05.904155+0000","last_clean":"2024-05-09T08:31:05.904155+0000","last_became_active":"2024-05-09T08:31:04.857947+0000","last_became_peered":"2024-05-09T08:31:04.857947+0000","last_unstale":"2024-05-09T08:31:05.904155+0000","last_undegraded":"2024-05-09T08:31:05.904155+0000","last_fullsized":"2024-05-09T08:31:05.904155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:55:48.595607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.905027+0000","last_change":"2024-05-09T08:30:59.822946+0000","last_active":"2024-05-09T08:31:05.905027+0000","last_peered":"2024-05-09T08:31:05.905027+0000","last_clean":"2024-05-09T08:31:05.905027+0000","last_became_active":"2024-05-09T08:30:59.822174+0000","last_became_peered":"2024-05-09T08:30:59.822174+0000","last_unstale":"2024-05-09T08:31:05.905027+0000","last_undegraded":"2024-05-09T08:31:05.905027+0000","last_fullsized":"2024-05-09T08:31:05.905027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:14.224033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903439+0000","last_change":"2024-05-09T08:31:00.704781+0000","last_active":"2024-05-09T08:31:05.903439+0000","last_peered":"2024-05-09T08:31:05.903439+0000","last_clean":"2024-05-09T08:31:05.903439+0000","last_became_active":"2024-05-09T08:31:00.704170+0000","last_became_peered":"2024-05-09T08:31:00.704170+0000","last_unstale":"2024-05-09T08:31:05.903439+0000","last_undegraded":"2024-05-09T08:31:05.903439+0000","last_fullsized":"2024-05-09T08:31:05.903439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:27:47.402290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.851510+0000","last_change":"2024-05-09T08:31:04.855648+0000","last_active":"2024-05-09T08:31:05.851510+0000","last_peered":"2024-05-09T08:31:05.851510+0000","last_clean":"2024-05-09T08:31:05.851510+0000","last_became_active":"2024-05-09T08:31:04.854678+0000","last_became_peered":"2024-05-09T08:31:04.854678+0000","last_unstale":"2024-05-09T08:31:05.851510+0000","last_undegraded":"2024-05-09T08:31:05.851510+0000","last_fullsized":"2024-05-09T08:31:05.851510+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:53:46.039172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'10","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:16.429196+0000","last_change":"2024-05-09T08:30:59.826498+0000","last_active":"2024-05-09T08:31:16.429196+0000","last_peered":"2024-05-09T08:31:16.429196+0000","last_clean":"2024-05-09T08:31:16.429196+0000","last_became_active":"2024-05-09T08:30:59.826050+0000","last_became_peered":"2024-05-09T08:30:59.826050+0000","last_unstale":"2024-05-09T08:31:16.429196+0000","last_undegraded":"2024-05-09T08:31:16.429196+0000","last_fullsized":"2024-05-09T08:31:16.429196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:19.920901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903767+0000","last_change":"2024-05-09T08:31:00.688379+0000","last_active":"2024-05-09T08:31:05.903767+0000","last_peered":"2024-05-09T08:31:05.903767+0000","last_clean":"2024-05-09T08:31:05.903767+0000","last_became_active":"2024-05-09T08:31:00.688030+0000","last_became_peered":"2024-05-09T08:31:00.688030+0000","last_unstale":"2024-05-09T08:31:05.903767+0000","last_undegraded":"2024-05-09T08:31:05.903767+0000","last_fullsized":"2024-05-09T08:31:05.903767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:17.248154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.905313+0000","last_change":"2024-05-09T08:31:04.858513+0000","last_active":"2024-05-09T08:31:05.905313+0000","last_peered":"2024-05-09T08:31:05.905313+0000","last_clean":"2024-05-09T08:31:05.905313+0000","last_became_active":"2024-05-09T08:31:04.857988+0000","last_became_peered":"2024-05-09T08:31:04.857988+0000","last_unstale":"2024-05-09T08:31:05.905313+0000","last_undegraded":"2024-05-09T08:31:05.905313+0000","last_fullsized":"2024-05-09T08:31:05.905313+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:59:24.264503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:20.061824+0000","last_change":"2024-05-09T08:30:59.823403+0000","last_active":"2024-05-09T08:31:20.061824+0000","last_peered":"2024-05-09T08:31:20.061824+0000","last_clean":"2024-05-09T08:31:20.061824+0000","last_became_active":"2024-05-09T08:30:59.823066+0000","last_became_peered":"2024-05-09T08:30:59.823066+0000","last_unstale":"2024-05-09T08:31:20.061824+0000","last_undegraded":"2024-05-09T08:31:20.061824+0000","last_fullsized":"2024-05-09T08:31:20.061824+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:16:39.157895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902459+0000","last_change":"2024-05-09T08:31:00.688009+0000","last_active":"2024-05-09T08:31:05.902459+0000","last_peered":"2024-05-09T08:31:05.902459+0000","last_clean":"2024-05-09T08:31:05.902459+0000","last_became_active":"2024-05-09T08:31:00.687679+0000","last_became_peered":"2024-05-09T08:31:00.687679+0000","last_unstale":"2024-05-09T08:31:05.902459+0000","last_undegraded":"2024-05-09T08:31:05.902459+0000","last_fullsized":"2024-05-09T08:31:05.902459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:44.403639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903957+0000","last_change":"2024-05-09T08:31:04.851880+0000","last_active":"2024-05-09T08:31:05.903957+0000","last_peered":"2024-05-09T08:31:05.903957+0000","last_clean":"2024-05-09T08:31:05.903957+0000","last_became_active":"2024-05-09T08:31:04.851316+0000","last_became_peered":"2024-05-09T08:31:04.851316+0000","last_unstale":"2024-05-09T08:31:05.903957+0000","last_undegraded":"2024-05-09T08:31:05.903957+0000","last_fullsized":"2024-05-09T08:31:05.903957+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:53.144996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.051820+0000","last_change":"2024-05-09T08:30:59.823411+0000","last_active":"2024-05-09T08:31:19.051820+0000","last_peered":"2024-05-09T08:31:19.051820+0000","last_clean":"2024-05-09T08:31:19.051820+0000","last_became_active":"2024-05-09T08:30:59.822854+0000","last_became_peered":"2024-05-09T08:30:59.822854+0000","last_unstale":"2024-05-09T08:31:19.051820+0000","last_undegraded":"2024-05-09T08:31:19.051820+0000","last_fullsized":"2024-05-09T08:31:19.051820+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:36:03.666937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852369+0000","last_change":"2024-05-09T08:31:00.698508+0000","last_active":"2024-05-09T08:31:05.852369+0000","last_peered":"2024-05-09T08:31:05.852369+0000","last_clean":"2024-05-09T08:31:05.852369+0000","last_became_active":"2024-05-09T08:31:00.696947+0000","last_became_peered":"2024-05-09T08:31:00.696947+0000","last_unstale":"2024-05-09T08:31:05.852369+0000","last_undegraded":"2024-05-09T08:31:05.852369+0000","last_fullsized":"2024-05-09T08:31:05.852369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:47:27.484554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852070+0000","last_change":"2024-05-09T08:31:04.859338+0000","last_active":"2024-05-09T08:31:05.852070+0000","last_peered":"2024-05-09T08:31:05.852070+0000","last_clean":"2024-05-09T08:31:05.852070+0000","last_became_active":"2024-05-09T08:31:04.858951+0000","last_became_peered":"2024-05-09T08:31:04.858951+0000","last_unstale":"2024-05-09T08:31:05.852070+0000","last_undegraded":"2024-05-09T08:31:05.852070+0000","last_fullsized":"2024-05-09T08:31:05.852070+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:26:46.738359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:20.061213+0000","last_change":"2024-05-09T08:30:59.823710+0000","last_active":"2024-05-09T08:31:20.061213+0000","last_peered":"2024-05-09T08:31:20.061213+0000","last_clean":"2024-05-09T08:31:20.061213+0000","last_became_active":"2024-05-09T08:30:59.823213+0000","last_became_peered":"2024-05-09T08:30:59.823213+0000","last_unstale":"2024-05-09T08:31:20.061213+0000","last_undegraded":"2024-05-09T08:31:20.061213+0000","last_fullsized":"2024-05-09T08:31:20.061213+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:09.134942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852079+0000","last_change":"2024-05-09T08:31:00.693735+0000","last_active":"2024-05-09T08:31:05.852079+0000","last_peered":"2024-05-09T08:31:05.852079+0000","last_clean":"2024-05-09T08:31:05.852079+0000","last_became_active":"2024-05-09T08:31:00.693401+0000","last_became_peered":"2024-05-09T08:31:00.693401+0000","last_unstale":"2024-05-09T08:31:05.852079+0000","last_undegraded":"2024-05-09T08:31:05.852079+0000","last_fullsized":"2024-05-09T08:31:05.852079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:58:41.429854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903130+0000","last_change":"2024-05-09T08:31:04.861289+0000","last_active":"2024-05-09T08:31:05.903130+0000","last_peered":"2024-05-09T08:31:05.903130+0000","last_clean":"2024-05-09T08:31:05.903130+0000","last_became_active":"2024-05-09T08:31:04.860915+0000","last_became_peered":"2024-05-09T08:31:04.860915+0000","last_unstale":"2024-05-09T08:31:05.903130+0000","last_undegraded":"2024-05-09T08:31:05.903130+0000","last_fullsized":"2024-05-09T08:31:05.903130+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:01:45.724961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:20.065310+0000","last_change":"2024-05-09T08:30:59.827631+0000","last_active":"2024-05-09T08:31:20.065310+0000","last_peered":"2024-05-09T08:31:20.065310+0000","last_clean":"2024-05-09T08:31:20.065310+0000","last_became_active":"2024-05-09T08:30:59.827295+0000","last_became_peered":"2024-05-09T08:30:59.827295+0000","last_unstale":"2024-05-09T08:31:20.065310+0000","last_undegraded":"2024-05-09T08:31:20.065310+0000","last_fullsized":"2024-05-09T08:31:20.065310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:35.282564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904558+0000","last_change":"2024-05-09T08:31:00.697712+0000","last_active":"2024-05-09T08:31:05.904558+0000","last_peered":"2024-05-09T08:31:05.904558+0000","last_clean":"2024-05-09T08:31:05.904558+0000","last_became_active":"2024-05-09T08:31:00.697385+0000","last_became_peered":"2024-05-09T08:31:00.697385+0000","last_unstale":"2024-05-09T08:31:05.904558+0000","last_undegraded":"2024-05-09T08:31:05.904558+0000","last_fullsized":"2024-05-09T08:31:05.904558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:09.087279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904633+0000","last_change":"2024-05-09T08:31:04.860691+0000","last_active":"2024-05-09T08:31:05.904633+0000","last_peered":"2024-05-09T08:31:05.904633+0000","last_clean":"2024-05-09T08:31:05.904633+0000","last_became_active":"2024-05-09T08:31:04.858458+0000","last_became_peered":"2024-05-09T08:31:04.858458+0000","last_unstale":"2024-05-09T08:31:05.904633+0000","last_undegraded":"2024-05-09T08:31:05.904633+0000","last_fullsized":"2024-05-09T08:31:05.904633+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:16.274359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:21.057390+0000","last_change":"2024-05-09T08:30:59.825188+0000","last_active":"2024-05-09T08:31:21.057390+0000","last_peered":"2024-05-09T08:31:21.057390+0000","last_clean":"2024-05-09T08:31:21.057390+0000","last_became_active":"2024-05-09T08:30:59.824855+0000","last_became_peered":"2024-05-09T08:30:59.824855+0000","last_unstale":"2024-05-09T08:31:21.057390+0000","last_undegraded":"2024-05-09T08:31:21.057390+0000","last_fullsized":"2024-05-09T08:31:21.057390+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:17:21.753652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903360+0000","last_change":"2024-05-09T08:31:00.695862+0000","last_active":"2024-05-09T08:31:05.903360+0000","last_peered":"2024-05-09T08:31:05.903360+0000","last_clean":"2024-05-09T08:31:05.903360+0000","last_became_active":"2024-05-09T08:31:00.695404+0000","last_became_peered":"2024-05-09T08:31:00.695404+0000","last_unstale":"2024-05-09T08:31:05.903360+0000","last_undegraded":"2024-05-09T08:31:05.903360+0000","last_fullsized":"2024-05-09T08:31:05.903360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:53:07.567583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903393+0000","last_change":"2024-05-09T08:31:04.856322+0000","last_active":"2024-05-09T08:31:05.903393+0000","last_peered":"2024-05-09T08:31:05.903393+0000","last_clean":"2024-05-09T08:31:05.903393+0000","last_became_active":"2024-05-09T08:31:04.855821+0000","last_became_peered":"2024-05-09T08:31:04.855821+0000","last_unstale":"2024-05-09T08:31:05.903393+0000","last_undegraded":"2024-05-09T08:31:05.903393+0000","last_fullsized":"2024-05-09T08:31:05.903393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:29:18.594928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.851831+0000","last_change":"2024-05-09T08:30:59.825924+0000","last_active":"2024-05-09T08:31:05.851831+0000","last_peered":"2024-05-09T08:31:05.851831+0000","last_clean":"2024-05-09T08:31:05.851831+0000","last_became_active":"2024-05-09T08:30:59.825505+0000","last_became_peered":"2024-05-09T08:30:59.825505+0000","last_unstale":"2024-05-09T08:31:05.851831+0000","last_undegraded":"2024-05-09T08:31:05.851831+0000","last_fullsized":"2024-05-09T08:31:05.851831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:31:07.416783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903863+0000","last_change":"2024-05-09T08:31:00.684523+0000","last_active":"2024-05-09T08:31:05.903863+0000","last_peered":"2024-05-09T08:31:05.903863+0000","last_clean":"2024-05-09T08:31:05.903863+0000","last_became_active":"2024-05-09T08:31:00.684153+0000","last_became_peered":"2024-05-09T08:31:00.684153+0000","last_unstale":"2024-05-09T08:31:05.903863+0000","last_undegraded":"2024-05-09T08:31:05.903863+0000","last_fullsized":"2024-05-09T08:31:05.903863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:02.423401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903520+0000","last_change":"2024-05-09T08:31:04.857961+0000","last_active":"2024-05-09T08:31:05.903520+0000","last_peered":"2024-05-09T08:31:05.903520+0000","last_clean":"2024-05-09T08:31:05.903520+0000","last_became_active":"2024-05-09T08:31:04.857543+0000","last_became_peered":"2024-05-09T08:31:04.857543+0000","last_unstale":"2024-05-09T08:31:05.903520+0000","last_undegraded":"2024-05-09T08:31:05.903520+0000","last_fullsized":"2024-05-09T08:31:05.903520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:22:30.820297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852189+0000","last_change":"2024-05-09T08:30:59.827878+0000","last_active":"2024-05-09T08:31:05.852189+0000","last_peered":"2024-05-09T08:31:05.852189+0000","last_clean":"2024-05-09T08:31:05.852189+0000","last_became_active":"2024-05-09T08:30:59.827487+0000","last_became_peered":"2024-05-09T08:30:59.827487+0000","last_unstale":"2024-05-09T08:31:05.852189+0000","last_undegraded":"2024-05-09T08:31:05.852189+0000","last_fullsized":"2024-05-09T08:31:05.852189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:34.465457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903788+0000","last_change":"2024-05-09T08:31:00.695617+0000","last_active":"2024-05-09T08:31:05.903788+0000","last_peered":"2024-05-09T08:31:05.903788+0000","last_clean":"2024-05-09T08:31:05.903788+0000","last_became_active":"2024-05-09T08:31:00.695293+0000","last_became_peered":"2024-05-09T08:31:00.695293+0000","last_unstale":"2024-05-09T08:31:05.903788+0000","last_undegraded":"2024-05-09T08:31:05.903788+0000","last_fullsized":"2024-05-09T08:31:05.903788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:44:03.958559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903306+0000","last_change":"2024-05-09T08:31:04.860184+0000","last_active":"2024-05-09T08:31:05.903306+0000","last_peered":"2024-05-09T08:31:05.903306+0000","last_clean":"2024-05-09T08:31:05.903306+0000","last_became_active":"2024-05-09T08:31:04.859730+0000","last_became_peered":"2024-05-09T08:31:04.859730+0000","last_unstale":"2024-05-09T08:31:05.903306+0000","last_undegraded":"2024-05-09T08:31:05.903306+0000","last_fullsized":"2024-05-09T08:31:05.903306+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:13:17.292678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904221+0000","last_change":"2024-05-09T08:31:00.694183+0000","last_active":"2024-05-09T08:31:05.904221+0000","last_peered":"2024-05-09T08:31:05.904221+0000","last_clean":"2024-05-09T08:31:05.904221+0000","last_became_active":"2024-05-09T08:31:00.693819+0000","last_became_peered":"2024-05-09T08:31:00.693819+0000","last_unstale":"2024-05-09T08:31:05.904221+0000","last_undegraded":"2024-05-09T08:31:05.904221+0000","last_fullsized":"2024-05-09T08:31:05.904221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:09:26.255663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:20.062975+0000","last_change":"2024-05-09T08:30:59.830758+0000","last_active":"2024-05-09T08:31:20.062975+0000","last_peered":"2024-05-09T08:31:20.062975+0000","last_clean":"2024-05-09T08:31:20.062975+0000","last_became_active":"2024-05-09T08:30:59.830430+0000","last_became_peered":"2024-05-09T08:30:59.830430+0000","last_unstale":"2024-05-09T08:31:20.062975+0000","last_undegraded":"2024-05-09T08:31:20.062975+0000","last_fullsized":"2024-05-09T08:31:20.062975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:24:05.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852582+0000","last_change":"2024-05-09T08:31:04.860599+0000","last_active":"2024-05-09T08:31:05.852582+0000","last_peered":"2024-05-09T08:31:05.852582+0000","last_clean":"2024-05-09T08:31:05.852582+0000","last_became_active":"2024-05-09T08:31:04.860077+0000","last_became_peered":"2024-05-09T08:31:04.860077+0000","last_unstale":"2024-05-09T08:31:05.852582+0000","last_undegraded":"2024-05-09T08:31:05.852582+0000","last_fullsized":"2024-05-09T08:31:05.852582+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:34.594241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903082+0000","last_change":"2024-05-09T08:31:00.692197+0000","last_active":"2024-05-09T08:31:05.903082+0000","last_peered":"2024-05-09T08:31:05.903082+0000","last_clean":"2024-05-09T08:31:05.903082+0000","last_became_active":"2024-05-09T08:31:00.691348+0000","last_became_peered":"2024-05-09T08:31:00.691348+0000","last_unstale":"2024-05-09T08:31:05.903082+0000","last_undegraded":"2024-05-09T08:31:05.903082+0000","last_fullsized":"2024-05-09T08:31:05.903082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:00:45.826433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.052780+0000","last_change":"2024-05-09T08:30:59.825341+0000","last_active":"2024-05-09T08:31:19.052780+0000","last_peered":"2024-05-09T08:31:19.052780+0000","last_clean":"2024-05-09T08:31:19.052780+0000","last_became_active":"2024-05-09T08:30:59.824860+0000","last_became_peered":"2024-05-09T08:30:59.824860+0000","last_unstale":"2024-05-09T08:31:19.052780+0000","last_undegraded":"2024-05-09T08:31:19.052780+0000","last_fullsized":"2024-05-09T08:31:19.052780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:13.868040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.851912+0000","last_change":"2024-05-09T08:29:47.816504+0000","last_active":"2024-05-09T08:31:05.851912+0000","last_peered":"2024-05-09T08:31:05.851912+0000","last_clean":"2024-05-09T08:31:05.851912+0000","last_became_active":"2024-05-09T08:29:47.816050+0000","last_became_peered":"2024-05-09T08:29:47.816050+0000","last_unstale":"2024-05-09T08:31:05.851912+0000","last_undegraded":"2024-05-09T08:31:05.851912+0000","last_fullsized":"2024-05-09T08:31:05.851912+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:26:22.135491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:26:22.135491+0000","last_clean_scrub_stamp":"2024-05-09T08:26:22.135491+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:28.543694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.905286+0000","last_change":"2024-05-09T08:31:04.860417+0000","last_active":"2024-05-09T08:31:05.905286+0000","last_peered":"2024-05-09T08:31:05.905286+0000","last_clean":"2024-05-09T08:31:05.905286+0000","last_became_active":"2024-05-09T08:31:04.859040+0000","last_became_peered":"2024-05-09T08:31:04.859040+0000","last_unstale":"2024-05-09T08:31:05.905286+0000","last_undegraded":"2024-05-09T08:31:05.905286+0000","last_fullsized":"2024-05-09T08:31:05.905286+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:12:04.157354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902189+0000","last_change":"2024-05-09T08:31:00.703131+0000","last_active":"2024-05-09T08:31:05.902189+0000","last_peered":"2024-05-09T08:31:05.902189+0000","last_clean":"2024-05-09T08:31:05.902189+0000","last_became_active":"2024-05-09T08:31:00.702817+0000","last_became_peered":"2024-05-09T08:31:00.702817+0000","last_unstale":"2024-05-09T08:31:05.902189+0000","last_undegraded":"2024-05-09T08:31:05.902189+0000","last_fullsized":"2024-05-09T08:31:05.902189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:31.059334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:20.063532+0000","last_change":"2024-05-09T08:30:59.829636+0000","last_active":"2024-05-09T08:31:20.063532+0000","last_peered":"2024-05-09T08:31:20.063532+0000","last_clean":"2024-05-09T08:31:20.063532+0000","last_became_active":"2024-05-09T08:30:59.829276+0000","last_became_peered":"2024-05-09T08:30:59.829276+0000","last_unstale":"2024-05-09T08:31:20.063532+0000","last_undegraded":"2024-05-09T08:31:20.063532+0000","last_fullsized":"2024-05-09T08:31:20.063532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:17.847258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904168+0000","last_change":"2024-05-09T08:31:04.853945+0000","last_active":"2024-05-09T08:31:05.904168+0000","last_peered":"2024-05-09T08:31:05.904168+0000","last_clean":"2024-05-09T08:31:05.904168+0000","last_became_active":"2024-05-09T08:31:04.853417+0000","last_became_peered":"2024-05-09T08:31:04.853417+0000","last_unstale":"2024-05-09T08:31:05.904168+0000","last_undegraded":"2024-05-09T08:31:05.904168+0000","last_fullsized":"2024-05-09T08:31:05.904168+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:21:31.126562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.051050+0000","last_change":"2024-05-09T08:30:59.828613+0000","last_active":"2024-05-09T08:31:19.051050+0000","last_peered":"2024-05-09T08:31:19.051050+0000","last_clean":"2024-05-09T08:31:19.051050+0000","last_became_active":"2024-05-09T08:30:59.828256+0000","last_became_peered":"2024-05-09T08:30:59.828256+0000","last_unstale":"2024-05-09T08:31:19.051050+0000","last_undegraded":"2024-05-09T08:31:19.051050+0000","last_fullsized":"2024-05-09T08:31:19.051050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:17:04.488271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903709+0000","last_change":"2024-05-09T08:31:00.703478+0000","last_active":"2024-05-09T08:31:05.903709+0000","last_peered":"2024-05-09T08:31:05.903709+0000","last_clean":"2024-05-09T08:31:05.903709+0000","last_became_active":"2024-05-09T08:31:00.702852+0000","last_became_peered":"2024-05-09T08:31:00.702852+0000","last_unstale":"2024-05-09T08:31:05.903709+0000","last_undegraded":"2024-05-09T08:31:05.903709+0000","last_fullsized":"2024-05-09T08:31:05.903709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:12.635341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903152+0000","last_change":"2024-05-09T08:31:04.860801+0000","last_active":"2024-05-09T08:31:05.903152+0000","last_peered":"2024-05-09T08:31:05.903152+0000","last_clean":"2024-05-09T08:31:05.903152+0000","last_became_active":"2024-05-09T08:31:04.857387+0000","last_became_peered":"2024-05-09T08:31:04.857387+0000","last_unstale":"2024-05-09T08:31:05.903152+0000","last_undegraded":"2024-05-09T08:31:05.903152+0000","last_fullsized":"2024-05-09T08:31:05.903152+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:50:16.555987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852097+0000","last_change":"2024-05-09T08:30:59.827082+0000","last_active":"2024-05-09T08:31:05.852097+0000","last_peered":"2024-05-09T08:31:05.852097+0000","last_clean":"2024-05-09T08:31:05.852097+0000","last_became_active":"2024-05-09T08:30:59.826746+0000","last_became_peered":"2024-05-09T08:30:59.826746+0000","last_unstale":"2024-05-09T08:31:05.852097+0000","last_undegraded":"2024-05-09T08:31:05.852097+0000","last_fullsized":"2024-05-09T08:31:05.852097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:49.176889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903291+0000","last_change":"2024-05-09T08:31:00.691902+0000","last_active":"2024-05-09T08:31:05.903291+0000","last_peered":"2024-05-09T08:31:05.903291+0000","last_clean":"2024-05-09T08:31:05.903291+0000","last_became_active":"2024-05-09T08:31:00.689491+0000","last_became_peered":"2024-05-09T08:31:00.689491+0000","last_unstale":"2024-05-09T08:31:05.903291+0000","last_undegraded":"2024-05-09T08:31:05.903291+0000","last_fullsized":"2024-05-09T08:31:05.903291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:48:07.335144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852014+0000","last_change":"2024-05-09T08:31:04.857788+0000","last_active":"2024-05-09T08:31:05.852014+0000","last_peered":"2024-05-09T08:31:05.852014+0000","last_clean":"2024-05-09T08:31:05.852014+0000","last_became_active":"2024-05-09T08:31:04.853831+0000","last_became_peered":"2024-05-09T08:31:04.853831+0000","last_unstale":"2024-05-09T08:31:05.852014+0000","last_undegraded":"2024-05-09T08:31:05.852014+0000","last_fullsized":"2024-05-09T08:31:05.852014+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:08:15.451606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.051381+0000","last_change":"2024-05-09T08:30:59.826482+0000","last_active":"2024-05-09T08:31:19.051381+0000","last_peered":"2024-05-09T08:31:19.051381+0000","last_clean":"2024-05-09T08:31:19.051381+0000","last_became_active":"2024-05-09T08:30:59.826123+0000","last_became_peered":"2024-05-09T08:30:59.826123+0000","last_unstale":"2024-05-09T08:31:19.051381+0000","last_undegraded":"2024-05-09T08:31:19.051381+0000","last_fullsized":"2024-05-09T08:31:19.051381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:38.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924786+0000","last_change":"2024-05-09T08:31:00.693440+0000","last_active":"2024-05-09T08:31:05.924786+0000","last_peered":"2024-05-09T08:31:05.924786+0000","last_clean":"2024-05-09T08:31:05.924786+0000","last_became_active":"2024-05-09T08:31:00.693117+0000","last_became_peered":"2024-05-09T08:31:00.693117+0000","last_unstale":"2024-05-09T08:31:05.924786+0000","last_undegraded":"2024-05-09T08:31:05.924786+0000","last_fullsized":"2024-05-09T08:31:05.924786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:10:01.205605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903764+0000","last_change":"2024-05-09T08:31:04.868120+0000","last_active":"2024-05-09T08:31:05.903764+0000","last_peered":"2024-05-09T08:31:05.903764+0000","last_clean":"2024-05-09T08:31:05.903764+0000","last_became_active":"2024-05-09T08:31:04.867571+0000","last_became_peered":"2024-05-09T08:31:04.867571+0000","last_unstale":"2024-05-09T08:31:05.903764+0000","last_undegraded":"2024-05-09T08:31:05.903764+0000","last_fullsized":"2024-05-09T08:31:05.903764+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:12:36.503972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.052891+0000","last_change":"2024-05-09T08:30:59.826280+0000","last_active":"2024-05-09T08:31:19.052891+0000","last_peered":"2024-05-09T08:31:19.052891+0000","last_clean":"2024-05-09T08:31:19.052891+0000","last_became_active":"2024-05-09T08:30:59.825805+0000","last_became_peered":"2024-05-09T08:30:59.825805+0000","last_unstale":"2024-05-09T08:31:19.052891+0000","last_undegraded":"2024-05-09T08:31:19.052891+0000","last_fullsized":"2024-05-09T08:31:19.052891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:46:03.833159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902843+0000","last_change":"2024-05-09T08:31:00.687581+0000","last_active":"2024-05-09T08:31:05.902843+0000","last_peered":"2024-05-09T08:31:05.902843+0000","last_clean":"2024-05-09T08:31:05.902843+0000","last_became_active":"2024-05-09T08:31:00.684608+0000","last_became_peered":"2024-05-09T08:31:00.684608+0000","last_unstale":"2024-05-09T08:31:05.902843+0000","last_undegraded":"2024-05-09T08:31:05.902843+0000","last_fullsized":"2024-05-09T08:31:05.902843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:06:06.436840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852178+0000","last_change":"2024-05-09T08:31:04.859399+0000","last_active":"2024-05-09T08:31:05.852178+0000","last_peered":"2024-05-09T08:31:05.852178+0000","last_clean":"2024-05-09T08:31:05.852178+0000","last_became_active":"2024-05-09T08:31:04.859027+0000","last_became_peered":"2024-05-09T08:31:04.859027+0000","last_unstale":"2024-05-09T08:31:05.852178+0000","last_undegraded":"2024-05-09T08:31:05.852178+0000","last_fullsized":"2024-05-09T08:31:05.852178+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:18:00.829342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.052609+0000","last_change":"2024-05-09T08:30:59.824171+0000","last_active":"2024-05-09T08:31:19.052609+0000","last_peered":"2024-05-09T08:31:19.052609+0000","last_clean":"2024-05-09T08:31:19.052609+0000","last_became_active":"2024-05-09T08:30:59.823859+0000","last_became_peered":"2024-05-09T08:30:59.823859+0000","last_unstale":"2024-05-09T08:31:19.052609+0000","last_undegraded":"2024-05-09T08:31:19.052609+0000","last_fullsized":"2024-05-09T08:31:19.052609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:30.199504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903514+0000","last_change":"2024-05-09T08:31:00.700056+0000","last_active":"2024-05-09T08:31:05.903514+0000","last_peered":"2024-05-09T08:31:05.903514+0000","last_clean":"2024-05-09T08:31:05.903514+0000","last_became_active":"2024-05-09T08:31:00.699673+0000","last_became_peered":"2024-05-09T08:31:00.699673+0000","last_unstale":"2024-05-09T08:31:05.903514+0000","last_undegraded":"2024-05-09T08:31:05.903514+0000","last_fullsized":"2024-05-09T08:31:05.903514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:23.873645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903599+0000","last_change":"2024-05-09T08:31:04.858940+0000","last_active":"2024-05-09T08:31:05.903599+0000","last_peered":"2024-05-09T08:31:05.903599+0000","last_clean":"2024-05-09T08:31:05.903599+0000","last_became_active":"2024-05-09T08:31:04.858532+0000","last_became_peered":"2024-05-09T08:31:04.858532+0000","last_unstale":"2024-05-09T08:31:05.903599+0000","last_undegraded":"2024-05-09T08:31:05.903599+0000","last_fullsized":"2024-05-09T08:31:05.903599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:57:42.128068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852056+0000","last_change":"2024-05-09T08:30:59.824182+0000","last_active":"2024-05-09T08:31:05.852056+0000","last_peered":"2024-05-09T08:31:05.852056+0000","last_clean":"2024-05-09T08:31:05.852056+0000","last_became_active":"2024-05-09T08:30:59.823691+0000","last_became_peered":"2024-05-09T08:30:59.823691+0000","last_unstale":"2024-05-09T08:31:05.852056+0000","last_undegraded":"2024-05-09T08:31:05.852056+0000","last_fullsized":"2024-05-09T08:31:05.852056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:28:07.810733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903637+0000","last_change":"2024-05-09T08:31:00.692614+0000","last_active":"2024-05-09T08:31:05.903637+0000","last_peered":"2024-05-09T08:31:05.903637+0000","last_clean":"2024-05-09T08:31:05.903637+0000","last_became_active":"2024-05-09T08:31:00.692280+0000","last_became_peered":"2024-05-09T08:31:00.692280+0000","last_unstale":"2024-05-09T08:31:05.903637+0000","last_undegraded":"2024-05-09T08:31:05.903637+0000","last_fullsized":"2024-05-09T08:31:05.903637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:23:15.398196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903133+0000","last_change":"2024-05-09T08:31:04.854029+0000","last_active":"2024-05-09T08:31:05.903133+0000","last_peered":"2024-05-09T08:31:05.903133+0000","last_clean":"2024-05-09T08:31:05.903133+0000","last_became_active":"2024-05-09T08:31:04.853554+0000","last_became_peered":"2024-05-09T08:31:04.853554+0000","last_unstale":"2024-05-09T08:31:05.903133+0000","last_undegraded":"2024-05-09T08:31:05.903133+0000","last_fullsized":"2024-05-09T08:31:05.903133+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:34:41.059614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:19.051300+0000","last_change":"2024-05-09T08:30:59.824512+0000","last_active":"2024-05-09T08:31:19.051300+0000","last_peered":"2024-05-09T08:31:19.051300+0000","last_clean":"2024-05-09T08:31:19.051300+0000","last_became_active":"2024-05-09T08:30:59.824146+0000","last_became_peered":"2024-05-09T08:30:59.824146+0000","last_unstale":"2024-05-09T08:31:19.051300+0000","last_undegraded":"2024-05-09T08:31:19.051300+0000","last_fullsized":"2024-05-09T08:31:19.051300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:27.734292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852224+0000","last_change":"2024-05-09T08:31:00.697272+0000","last_active":"2024-05-09T08:31:05.852224+0000","last_peered":"2024-05-09T08:31:05.852224+0000","last_clean":"2024-05-09T08:31:05.852224+0000","last_became_active":"2024-05-09T08:31:00.696026+0000","last_became_peered":"2024-05-09T08:31:00.696026+0000","last_unstale":"2024-05-09T08:31:05.852224+0000","last_undegraded":"2024-05-09T08:31:05.852224+0000","last_fullsized":"2024-05-09T08:31:05.852224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:04.376142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902852+0000","last_change":"2024-05-09T08:31:04.856641+0000","last_active":"2024-05-09T08:31:05.902852+0000","last_peered":"2024-05-09T08:31:05.902852+0000","last_clean":"2024-05-09T08:31:05.902852+0000","last_became_active":"2024-05-09T08:31:04.856102+0000","last_became_peered":"2024-05-09T08:31:04.856102+0000","last_unstale":"2024-05-09T08:31:05.902852+0000","last_undegraded":"2024-05-09T08:31:05.902852+0000","last_fullsized":"2024-05-09T08:31:05.902852+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:35:49.831076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:20.064491+0000","last_change":"2024-05-09T08:30:59.826808+0000","last_active":"2024-05-09T08:31:20.064491+0000","last_peered":"2024-05-09T08:31:20.064491+0000","last_clean":"2024-05-09T08:31:20.064491+0000","last_became_active":"2024-05-09T08:30:59.826481+0000","last_became_peered":"2024-05-09T08:30:59.826481+0000","last_unstale":"2024-05-09T08:31:20.064491+0000","last_undegraded":"2024-05-09T08:31:20.064491+0000","last_fullsized":"2024-05-09T08:31:20.064491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:27:37.820758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852657+0000","last_change":"2024-05-09T08:31:00.697876+0000","last_active":"2024-05-09T08:31:05.852657+0000","last_peered":"2024-05-09T08:31:05.852657+0000","last_clean":"2024-05-09T08:31:05.852657+0000","last_became_active":"2024-05-09T08:31:00.696299+0000","last_became_peered":"2024-05-09T08:31:00.696299+0000","last_unstale":"2024-05-09T08:31:05.852657+0000","last_undegraded":"2024-05-09T08:31:05.852657+0000","last_fullsized":"2024-05-09T08:31:05.852657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:48:57.373734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924502+0000","last_change":"2024-05-09T08:31:04.858588+0000","last_active":"2024-05-09T08:31:05.924502+0000","last_peered":"2024-05-09T08:31:05.924502+0000","last_clean":"2024-05-09T08:31:05.924502+0000","last_became_active":"2024-05-09T08:31:04.858154+0000","last_became_peered":"2024-05-09T08:31:04.858154+0000","last_unstale":"2024-05-09T08:31:05.924502+0000","last_undegraded":"2024-05-09T08:31:05.924502+0000","last_fullsized":"2024-05-09T08:31:05.924502+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:06:01.768743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.905078+0000","last_change":"2024-05-09T08:30:59.826066+0000","last_active":"2024-05-09T08:31:05.905078+0000","last_peered":"2024-05-09T08:31:05.905078+0000","last_clean":"2024-05-09T08:31:05.905078+0000","last_became_active":"2024-05-09T08:30:59.825586+0000","last_became_peered":"2024-05-09T08:30:59.825586+0000","last_unstale":"2024-05-09T08:31:05.905078+0000","last_undegraded":"2024-05-09T08:31:05.905078+0000","last_fullsized":"2024-05-09T08:31:05.905078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:19:04.629132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852335+0000","last_change":"2024-05-09T08:31:00.693788+0000","last_active":"2024-05-09T08:31:05.852335+0000","last_peered":"2024-05-09T08:31:05.852335+0000","last_clean":"2024-05-09T08:31:05.852335+0000","last_became_active":"2024-05-09T08:31:00.693448+0000","last_became_peered":"2024-05-09T08:31:00.693448+0000","last_unstale":"2024-05-09T08:31:05.852335+0000","last_undegraded":"2024-05-09T08:31:05.852335+0000","last_fullsized":"2024-05-09T08:31:05.852335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:06.516534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903801+0000","last_change":"2024-05-09T08:31:04.856185+0000","last_active":"2024-05-09T08:31:05.903801+0000","last_peered":"2024-05-09T08:31:05.903801+0000","last_clean":"2024-05-09T08:31:05.903801+0000","last_became_active":"2024-05-09T08:31:04.855779+0000","last_became_peered":"2024-05-09T08:31:04.855779+0000","last_unstale":"2024-05-09T08:31:05.903801+0000","last_undegraded":"2024-05-09T08:31:05.903801+0000","last_fullsized":"2024-05-09T08:31:05.903801+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:45:25.034077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902968+0000","last_change":"2024-05-09T08:30:59.822938+0000","last_active":"2024-05-09T08:31:05.902968+0000","last_peered":"2024-05-09T08:31:05.902968+0000","last_clean":"2024-05-09T08:31:05.902968+0000","last_became_active":"2024-05-09T08:30:59.822629+0000","last_became_peered":"2024-05-09T08:30:59.822629+0000","last_unstale":"2024-05-09T08:31:05.902968+0000","last_undegraded":"2024-05-09T08:31:05.902968+0000","last_fullsized":"2024-05-09T08:31:05.902968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:51.321350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852629+0000","last_change":"2024-05-09T08:31:00.691971+0000","last_active":"2024-05-09T08:31:05.852629+0000","last_peered":"2024-05-09T08:31:05.852629+0000","last_clean":"2024-05-09T08:31:05.852629+0000","last_became_active":"2024-05-09T08:31:00.691641+0000","last_became_peered":"2024-05-09T08:31:00.691641+0000","last_unstale":"2024-05-09T08:31:05.852629+0000","last_undegraded":"2024-05-09T08:31:05.852629+0000","last_fullsized":"2024-05-09T08:31:05.852629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:28.413886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903668+0000","last_change":"2024-05-09T08:31:00.701026+0000","last_active":"2024-05-09T08:31:05.903668+0000","last_peered":"2024-05-09T08:31:05.903668+0000","last_clean":"2024-05-09T08:31:05.903668+0000","last_became_active":"2024-05-09T08:31:00.700457+0000","last_became_peered":"2024-05-09T08:31:00.700457+0000","last_unstale":"2024-05-09T08:31:05.903668+0000","last_undegraded":"2024-05-09T08:31:05.903668+0000","last_fullsized":"2024-05-09T08:31:05.903668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:40.065173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852179+0000","last_change":"2024-05-09T08:31:00.699212+0000","last_active":"2024-05-09T08:31:05.852179+0000","last_peered":"2024-05-09T08:31:05.852179+0000","last_clean":"2024-05-09T08:31:05.852179+0000","last_became_active":"2024-05-09T08:31:00.698794+0000","last_became_peered":"2024-05-09T08:31:00.698794+0000","last_unstale":"2024-05-09T08:31:05.852179+0000","last_undegraded":"2024-05-09T08:31:05.852179+0000","last_fullsized":"2024-05-09T08:31:05.852179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:13:21.834839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.851906+0000","last_change":"2024-05-09T08:31:00.703201+0000","last_active":"2024-05-09T08:31:05.851906+0000","last_peered":"2024-05-09T08:31:05.851906+0000","last_clean":"2024-05-09T08:31:05.851906+0000","last_became_active":"2024-05-09T08:31:00.702801+0000","last_became_peered":"2024-05-09T08:31:00.702801+0000","last_unstale":"2024-05-09T08:31:05.851906+0000","last_undegraded":"2024-05-09T08:31:05.851906+0000","last_fullsized":"2024-05-09T08:31:05.851906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:58:35.106013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.924791+0000","last_change":"2024-05-09T08:31:00.693696+0000","last_active":"2024-05-09T08:31:05.924791+0000","last_peered":"2024-05-09T08:31:05.924791+0000","last_clean":"2024-05-09T08:31:05.924791+0000","last_became_active":"2024-05-09T08:31:00.693394+0000","last_became_peered":"2024-05-09T08:31:00.693394+0000","last_unstale":"2024-05-09T08:31:05.924791+0000","last_undegraded":"2024-05-09T08:31:05.924791+0000","last_fullsized":"2024-05-09T08:31:05.924791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:01.700968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903684+0000","last_change":"2024-05-09T08:31:00.694363+0000","last_active":"2024-05-09T08:31:05.903684+0000","last_peered":"2024-05-09T08:31:05.903684+0000","last_clean":"2024-05-09T08:31:05.903684+0000","last_became_active":"2024-05-09T08:31:00.693970+0000","last_became_peered":"2024-05-09T08:31:00.693970+0000","last_unstale":"2024-05-09T08:31:05.903684+0000","last_undegraded":"2024-05-09T08:31:05.903684+0000","last_fullsized":"2024-05-09T08:31:05.903684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:56.185801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903904+0000","last_change":"2024-05-09T08:31:00.691970+0000","last_active":"2024-05-09T08:31:05.903904+0000","last_peered":"2024-05-09T08:31:05.903904+0000","last_clean":"2024-05-09T08:31:05.903904+0000","last_became_active":"2024-05-09T08:31:00.691632+0000","last_became_peered":"2024-05-09T08:31:00.691632+0000","last_unstale":"2024-05-09T08:31:05.903904+0000","last_undegraded":"2024-05-09T08:31:05.903904+0000","last_fullsized":"2024-05-09T08:31:05.903904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:04:49.316808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.852658+0000","last_change":"2024-05-09T08:31:00.692042+0000","last_active":"2024-05-09T08:31:05.852658+0000","last_peered":"2024-05-09T08:31:05.852658+0000","last_clean":"2024-05-09T08:31:05.852658+0000","last_became_active":"2024-05-09T08:31:00.691700+0000","last_became_peered":"2024-05-09T08:31:00.691700+0000","last_unstale":"2024-05-09T08:31:05.852658+0000","last_undegraded":"2024-05-09T08:31:05.852658+0000","last_fullsized":"2024-05-09T08:31:05.852658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:54.236319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903520+0000","last_change":"2024-05-09T08:31:00.697261+0000","last_active":"2024-05-09T08:31:05.903520+0000","last_peered":"2024-05-09T08:31:05.903520+0000","last_clean":"2024-05-09T08:31:05.903520+0000","last_became_active":"2024-05-09T08:31:00.696299+0000","last_became_peered":"2024-05-09T08:31:00.696299+0000","last_unstale":"2024-05-09T08:31:05.903520+0000","last_undegraded":"2024-05-09T08:31:05.903520+0000","last_fullsized":"2024-05-09T08:31:05.903520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:06:52.061740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904115+0000","last_change":"2024-05-09T08:31:00.691097+0000","last_active":"2024-05-09T08:31:05.904115+0000","last_peered":"2024-05-09T08:31:05.904115+0000","last_clean":"2024-05-09T08:31:05.904115+0000","last_became_active":"2024-05-09T08:31:00.690487+0000","last_became_peered":"2024-05-09T08:31:00.690487+0000","last_unstale":"2024-05-09T08:31:05.904115+0000","last_undegraded":"2024-05-09T08:31:05.904115+0000","last_fullsized":"2024-05-09T08:31:05.904115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:47:26.468296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.903783+0000","last_change":"2024-05-09T08:31:00.703201+0000","last_active":"2024-05-09T08:31:05.903783+0000","last_peered":"2024-05-09T08:31:05.903783+0000","last_clean":"2024-05-09T08:31:05.903783+0000","last_became_active":"2024-05-09T08:31:00.702853+0000","last_became_peered":"2024-05-09T08:31:00.702853+0000","last_unstale":"2024-05-09T08:31:05.903783+0000","last_undegraded":"2024-05-09T08:31:05.903783+0000","last_fullsized":"2024-05-09T08:31:05.903783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:18.510321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902931+0000","last_change":"2024-05-09T08:31:00.703958+0000","last_active":"2024-05-09T08:31:05.902931+0000","last_peered":"2024-05-09T08:31:05.902931+0000","last_clean":"2024-05-09T08:31:05.902931+0000","last_became_active":"2024-05-09T08:31:00.703110+0000","last_became_peered":"2024-05-09T08:31:00.703110+0000","last_unstale":"2024-05-09T08:31:05.902931+0000","last_undegraded":"2024-05-09T08:31:05.902931+0000","last_fullsized":"2024-05-09T08:31:05.902931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:37:32.371869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.902749+0000","last_change":"2024-05-09T08:31:00.686763+0000","last_active":"2024-05-09T08:31:05.902749+0000","last_peered":"2024-05-09T08:31:05.902749+0000","last_clean":"2024-05-09T08:31:05.902749+0000","last_became_active":"2024-05-09T08:31:00.685642+0000","last_became_peered":"2024-05-09T08:31:00.685642+0000","last_unstale":"2024-05-09T08:31:05.902749+0000","last_undegraded":"2024-05-09T08:31:05.902749+0000","last_fullsized":"2024-05-09T08:31:05.902749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:40:18.201247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T08:31:05.904038+0000","last_change":"2024-05-09T08:31:00.697160+0000","last_active":"2024-05-09T08:31:05.904038+0000","last_peered":"2024-05-09T08:31:05.904038+0000","last_clean":"2024-05-09T08:31:05.904038+0000","last_became_active":"2024-05-09T08:31:00.696374+0000","last_became_peered":"2024-05-09T08:31:00.696374+0000","last_unstale":"2024-05-09T08:31:05.904038+0000","last_undegraded":"2024-05-09T08:31:05.904038+0000","last_fullsized":"2024-05-09T08:31:05.904038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:41:49.082049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7362,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":110,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":26339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383697,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":477797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645333,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28580,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716572,"statfs":{"total":95995035648,"available":95965769728,"internally_reserved":0,"allocated":1601536,"data_stored":1071619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808858,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":987136,"data_stored":477797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070496,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":477953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36892,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708260,"statfs":{"total":95995035648,"available":95957258240,"internally_reserved":0,"allocated":1593344,"data_stored":1069766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":481057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":482407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":477499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":477859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36904,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708248,"statfs":{"total":95995035648,"available":95957245952,"internally_reserved":0,"allocated":1605632,"data_stored":1069059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":10442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:31:24.117 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-09T08:31:24.118 INFO:tasks.cephadm:Setting up client nodes... 2024-05-09T08:31:24.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-09T08:31:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:24 smithi157 ceph-mon[33614]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:31:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:24 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1217040889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:31:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:24 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1452960864' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:24.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:24 smithi157 ceph-mon[33614]: pgmap v303: 129 pgs: 129 active+clean; 584 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 2.3 KiB/s wr, 5 op/s 2024-05-09T08:31:24.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:24 smithi110 ceph-mon[33622]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:31:24.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:24 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1217040889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:31:24.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:24 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1452960864' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:24.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:24 smithi110 ceph-mon[33622]: pgmap v303: 129 pgs: 129 active+clean; 584 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 2.3 KiB/s wr, 5 op/s 2024-05-09T08:31:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:24.513 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:24 smithi107 ceph-mon[30122]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:31:24.513 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:24 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1217040889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:31:24.513 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:24 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1452960864' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:24.514 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:24 smithi107 ceph-mon[30122]: pgmap v303: 129 pgs: 129 active+clean; 584 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 2.3 KiB/s wr, 5 op/s 2024-05-09T08:31:25.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:25 smithi157 ceph-mon[33614]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:31:25.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:25 smithi157 ceph-mon[33614]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:31:25.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:25 smithi110 ceph-mon[33622]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:31:25.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:25 smithi110 ceph-mon[33622]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:31:25.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:25 smithi107 ceph-mon[30122]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:31:25.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:25 smithi107 ceph-mon[30122]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:31:26.133 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:26 smithi107 ceph-mon[30122]: pgmap v304: 129 pgs: 129 active+clean; 620 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 5.9 KiB/s wr, 10 op/s 2024-05-09T08:31:26.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.246+0000 7f9f8e6aa700 1 -- 172.21.15.107:0/452337245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f881024a0 msgr2=0x7f9f88102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:26.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.246+0000 7f9f8e6aa700 1 --2- 172.21.15.107:0/452337245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f881024a0 0x7f9f88102900 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0099c0 tx=0x7f9f7c009cd0 comp rx=0 tx=0).stop 2024-05-09T08:31:26.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.246+0000 7f9f8e6aa700 1 -- 172.21.15.107:0/452337245 shutdown_connections 2024-05-09T08:31:26.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.246+0000 7f9f8e6aa700 1 --2- 172.21.15.107:0/452337245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f88102e40 0x7f9f881096f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:26.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.246+0000 7f9f8e6aa700 1 --2- 172.21.15.107:0/452337245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f881012a0 0x7f9f881016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:26.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.246+0000 7f9f8e6aa700 1 --2- 172.21.15.107:0/452337245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f881024a0 0x7f9f88102900 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:26.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.246+0000 7f9f8e6aa700 1 -- 172.21.15.107:0/452337245 >> 172.21.15.107:0/452337245 conn(0x7f9f880fc840 msgr2=0x7f9f880fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:26.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.246+0000 7f9f8e6aa700 1 -- 172.21.15.107:0/452337245 shutdown_connections 2024-05-09T08:31:26.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.246+0000 7f9f8e6aa700 1 -- 172.21.15.107:0/452337245 wait complete. 2024-05-09T08:31:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.247+0000 7f9f8e6aa700 1 Processor -- start 2024-05-09T08:31:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.247+0000 7f9f8e6aa700 1 -- start start 2024-05-09T08:31:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.247+0000 7f9f8e6aa700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f881012a0 0x7f9f8806b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.247+0000 7f9f8e6aa700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f881024a0 0x7f9f8819d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.247+0000 7f9f8e6aa700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f88102e40 0x7f9f8819f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.247+0000 7f9f8e6aa700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8810cd90 con 0x7f9f881024a0 2024-05-09T08:31:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.247+0000 7f9f8e6aa700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8810cc10 con 0x7f9f881012a0 2024-05-09T08:31:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.247+0000 7f9f8e6aa700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8810cf10 con 0x7f9f88102e40 2024-05-09T08:31:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.247+0000 7f9f877fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f881024a0 0x7f9f8819d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.248+0000 7f9f877fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f881024a0 0x7f9f8819d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53354/0 (socket says 172.21.15.107:53354) 2024-05-09T08:31:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.248+0000 7f9f877fe700 1 -- 172.21.15.107:0/4280865550 learned_addr learned my addr 172.21.15.107:0/4280865550 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.248+0000 7f9f87fff700 1 --2- 172.21.15.107:0/4280865550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f881012a0 0x7f9f8806b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.248+0000 7f9f8cc47700 1 --2- 172.21.15.107:0/4280865550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f88102e40 0x7f9f8819f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.248+0000 7f9f877fe700 1 -- 172.21.15.107:0/4280865550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f88102e40 msgr2=0x7f9f8819f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.248+0000 7f9f877fe700 1 --2- 172.21.15.107:0/4280865550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f88102e40 0x7f9f8819f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.248+0000 7f9f877fe700 1 -- 172.21.15.107:0/4280865550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f881012a0 msgr2=0x7f9f8806b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.248+0000 7f9f877fe700 1 --2- 172.21.15.107:0/4280865550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f881012a0 0x7f9f8806b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.248+0000 7f9f877fe700 1 -- 172.21.15.107:0/4280865550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f881a00c0 con 0x7f9f881024a0 2024-05-09T08:31:26.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.248+0000 7f9f877fe700 1 --2- 172.21.15.107:0/4280865550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f881024a0 0x7f9f8819d1b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c006010 tx=0x7f9f7c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:26.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.248+0000 7f9f857fa700 1 -- 172.21.15.107:0/4280865550 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c027070 con 0x7f9f881024a0 2024-05-09T08:31:26.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.248+0000 7f9f857fa700 1 -- 172.21.15.107:0/4280865550 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f7c004050 con 0x7f9f881024a0 2024-05-09T08:31:26.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.248+0000 7f9f857fa700 1 -- 172.21.15.107:0/4280865550 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c022cf0 con 0x7f9f881024a0 2024-05-09T08:31:26.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.248+0000 7f9f8e6aa700 1 -- 172.21.15.107:0/4280865550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f881a0350 con 0x7f9f881024a0 2024-05-09T08:31:26.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.248+0000 7f9f8e6aa700 1 -- 172.21.15.107:0/4280865550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f881a0820 con 0x7f9f881024a0 2024-05-09T08:31:26.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.249+0000 7f9f8e6aa700 1 -- 172.21.15.107:0/4280865550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f88050890 con 0x7f9f881024a0 2024-05-09T08:31:26.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.250+0000 7f9f857fa700 1 -- 172.21.15.107:0/4280865550 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f9f7c02a070 con 0x7f9f881024a0 2024-05-09T08:31:26.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.251+0000 7f9f857fa700 1 --2- 172.21.15.107:0/4280865550 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9f700a2d60 0x7f9f700a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:26.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.251+0000 7f9f87fff700 1 --2- 172.21.15.107:0/4280865550 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9f700a2d60 0x7f9f700a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:26.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.251+0000 7f9f857fa700 1 -- 172.21.15.107:0/4280865550 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9f7c0d0220 con 0x7f9f881024a0 2024-05-09T08:31:26.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.251+0000 7f9f87fff700 1 --2- 172.21.15.107:0/4280865550 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9f700a2d60 0x7f9f700a5220 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9f88102300 tx=0x7f9f7800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:26.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.255+0000 7f9f857fa700 1 -- 172.21.15.107:0/4280865550 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9f7c013070 con 0x7f9f881024a0 2024-05-09T08:31:26.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:26 smithi157 ceph-mon[33614]: pgmap v304: 129 pgs: 129 active+clean; 620 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 5.9 KiB/s wr, 10 op/s 2024-05-09T08:31:26.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:26 smithi110 ceph-mon[33622]: pgmap v304: 129 pgs: 129 active+clean; 620 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 5.9 KiB/s wr, 10 op/s 2024-05-09T08:31:26.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.528+0000 7f9f8e6aa700 1 -- 172.21.15.107:0/4280865550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f9f8802dcf0 con 0x7f9f881024a0 2024-05-09T08:31:26.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.531+0000 7f9f857fa700 1 -- 172.21.15.107:0/4280865550 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f9f8802dcf0 con 0x7f9f881024a0 2024-05-09T08:31:26.530 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2024-05-09T08:31:26.531 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDeiTxmOIyZHxAA63r4J1D0GUL4PFioCtopeA== 2024-05-09T08:31:26.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.533+0000 7f9f8e6aa700 1 -- 172.21.15.107:0/4280865550 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9f700a2d60 msgr2=0x7f9f700a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.533+0000 7f9f8e6aa700 1 --2- 172.21.15.107:0/4280865550 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9f700a2d60 0x7f9f700a5220 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9f88102300 tx=0x7f9f7800b410 comp rx=0 tx=0).stop 2024-05-09T08:31:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.533+0000 7f9f8e6aa700 1 -- 172.21.15.107:0/4280865550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f881024a0 msgr2=0x7f9f8819d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.533+0000 7f9f8e6aa700 1 --2- 172.21.15.107:0/4280865550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f881024a0 0x7f9f8819d1b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c006010 tx=0x7f9f7c0047b0 comp rx=0 tx=0).stop 2024-05-09T08:31:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.533+0000 7f9f8e6aa700 1 -- 172.21.15.107:0/4280865550 shutdown_connections 2024-05-09T08:31:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.533+0000 7f9f8e6aa700 1 --2- 172.21.15.107:0/4280865550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f88102e40 0x7f9f8819f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.533+0000 7f9f8e6aa700 1 --2- 172.21.15.107:0/4280865550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f881012a0 0x7f9f8806b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.533+0000 7f9f8e6aa700 1 --2- 172.21.15.107:0/4280865550 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9f700a2d60 0x7f9f700a5220 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.533+0000 7f9f8e6aa700 1 --2- 172.21.15.107:0/4280865550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f881024a0 0x7f9f8819d1b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.533+0000 7f9f8e6aa700 1 -- 172.21.15.107:0/4280865550 >> 172.21.15.107:0/4280865550 conn(0x7f9f880fc840 msgr2=0x7f9f881057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.533+0000 7f9f8e6aa700 1 -- 172.21.15.107:0/4280865550 shutdown_connections 2024-05-09T08:31:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:26.533+0000 7f9f8e6aa700 1 -- 172.21.15.107:0/4280865550 wait complete. 2024-05-09T08:31:27.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:27 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/4280865550' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-09T08:31:27.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:27 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/4280865550' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-09T08:31:27.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:27 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/4280865550' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-09T08:31:27.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:27 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/4280865550' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-09T08:31:27.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:27 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/4280865550' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-09T08:31:27.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:27 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/4280865550' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-09T08:31:27.383 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:31:27.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-09T08:31:27.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-09T08:31:27.422 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-09T08:31:27.422 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-09T08:31:27.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph mgr dump --format=json 2024-05-09T08:31:27.703 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:28.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:28 smithi157 ceph-mon[33614]: pgmap v305: 129 pgs: 129 active+clean; 620 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.9 KiB/s wr, 12 op/s 2024-05-09T08:31:28.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:28 smithi110 ceph-mon[33622]: pgmap v305: 129 pgs: 129 active+clean; 620 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.9 KiB/s wr, 12 op/s 2024-05-09T08:31:28.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:28 smithi107 ceph-mon[30122]: pgmap v305: 129 pgs: 129 active+clean; 620 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.9 KiB/s wr, 12 op/s 2024-05-09T08:31:29.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.542+0000 7ffaf44a1700 1 -- 172.21.15.107:0/1700373005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec101ec0 msgr2=0x7ffaec105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:29.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.542+0000 7ffaf44a1700 1 --2- 172.21.15.107:0/1700373005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec101ec0 0x7ffaec105e80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0099c0 tx=0x7ffadc009cd0 comp rx=0 tx=0).stop 2024-05-09T08:31:29.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.542+0000 7ffaf44a1700 1 -- 172.21.15.107:0/1700373005 shutdown_connections 2024-05-09T08:31:29.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.542+0000 7ffaf44a1700 1 --2- 172.21.15.107:0/1700373005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaec106560 0x7ffaec10e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:29.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.542+0000 7ffaf44a1700 1 --2- 172.21.15.107:0/1700373005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffaec101510 0x7ffaec1018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:29.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.542+0000 7ffaf44a1700 1 --2- 172.21.15.107:0/1700373005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec101ec0 0x7ffaec105e80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:29.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.542+0000 7ffaf44a1700 1 -- 172.21.15.107:0/1700373005 >> 172.21.15.107:0/1700373005 conn(0x7ffaec077270 msgr2=0x7ffaec077680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:29.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.542+0000 7ffaf44a1700 1 -- 172.21.15.107:0/1700373005 shutdown_connections 2024-05-09T08:31:29.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.542+0000 7ffaf44a1700 1 -- 172.21.15.107:0/1700373005 wait complete. 2024-05-09T08:31:29.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.542+0000 7ffaf44a1700 1 Processor -- start 2024-05-09T08:31:29.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf44a1700 1 -- start start 2024-05-09T08:31:29.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf44a1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffaec101510 0x7ffaec1998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:29.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf44a1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaec101ec0 0x7ffaec199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:29.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf44a1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec106560 0x7ffaec1a4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:29.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf44a1700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec111450 con 0x7ffaec106560 2024-05-09T08:31:29.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf44a1700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec1112d0 con 0x7ffaec101ec0 2024-05-09T08:31:29.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf44a1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec1115d0 con 0x7ffaec101510 2024-05-09T08:31:29.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf2a3e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec106560 0x7ffaec1a4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:29.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf2a3e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec106560 0x7ffaec1a4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53368/0 (socket says 172.21.15.107:53368) 2024-05-09T08:31:29.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf2a3e700 1 -- 172.21.15.107:0/1538215343 learned_addr learned my addr 172.21.15.107:0/1538215343 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:29.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf1a3c700 1 --2- 172.21.15.107:0/1538215343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaec101ec0 0x7ffaec199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:29.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf223d700 1 --2- 172.21.15.107:0/1538215343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffaec101510 0x7ffaec1998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:29.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf2a3e700 1 -- 172.21.15.107:0/1538215343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffaec101510 msgr2=0x7ffaec1998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:29.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf2a3e700 1 --2- 172.21.15.107:0/1538215343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffaec101510 0x7ffaec1998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:29.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf2a3e700 1 -- 172.21.15.107:0/1538215343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaec101ec0 msgr2=0x7ffaec199e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:29.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf2a3e700 1 --2- 172.21.15.107:0/1538215343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaec101ec0 0x7ffaec199e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:29.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.543+0000 7ffaf2a3e700 1 -- 172.21.15.107:0/1538215343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaec1a4eb0 con 0x7ffaec106560 2024-05-09T08:31:29.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.544+0000 7ffaf2a3e700 1 --2- 172.21.15.107:0/1538215343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec106560 0x7ffaec1a4970 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ffae4007ee0 tx=0x7ffae400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:29.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.544+0000 7ffae37fe700 1 -- 172.21.15.107:0/1538215343 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae400dbd0 con 0x7ffaec106560 2024-05-09T08:31:29.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.544+0000 7ffaf44a1700 1 -- 172.21.15.107:0/1538215343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaec1a50e0 con 0x7ffaec106560 2024-05-09T08:31:29.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.544+0000 7ffae37fe700 1 -- 172.21.15.107:0/1538215343 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffae400dd70 con 0x7ffaec106560 2024-05-09T08:31:29.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.544+0000 7ffae37fe700 1 -- 172.21.15.107:0/1538215343 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae4005990 con 0x7ffaec106560 2024-05-09T08:31:29.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.544+0000 7ffaf44a1700 1 -- 172.21.15.107:0/1538215343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaec1a5640 con 0x7ffaec106560 2024-05-09T08:31:29.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.545+0000 7ffaf44a1700 1 -- 172.21.15.107:0/1538215343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaec069010 con 0x7ffaec106560 2024-05-09T08:31:29.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.546+0000 7ffae37fe700 1 -- 172.21.15.107:0/1538215343 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ffae401a020 con 0x7ffaec106560 2024-05-09T08:31:29.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.547+0000 7ffae37fe700 1 --2- 172.21.15.107:0/1538215343 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ffad80a2d30 0x7ffad80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:29.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.547+0000 7ffaf223d700 1 --2- 172.21.15.107:0/1538215343 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ffad80a2d30 0x7ffad80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:29.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.547+0000 7ffae37fe700 1 -- 172.21.15.107:0/1538215343 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ffae40ceb00 con 0x7ffaec106560 2024-05-09T08:31:29.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.547+0000 7ffaf223d700 1 --2- 172.21.15.107:0/1538215343 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ffad80a2d30 0x7ffad80a51f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ffae8006fd0 tx=0x7ffae8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:29.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.551+0000 7ffae37fe700 1 -- 172.21.15.107:0/1538215343 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ffae4010070 con 0x7ffaec106560 2024-05-09T08:31:29.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.804+0000 7ffaf44a1700 1 -- 172.21.15.107:0/1538215343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7ffaec02dd40 con 0x7ffaec106560 2024-05-09T08:31:29.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.809+0000 7ffae37fe700 1 -- 172.21.15.107:0/1538215343 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263207 (secure 0 0 0) 0x7ffae409b6c0 con 0x7ffaec106560 2024-05-09T08:31:29.811 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:29.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.811+0000 7ffaf44a1700 1 -- 172.21.15.107:0/1538215343 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ffad80a2d30 msgr2=0x7ffad80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:29.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.811+0000 7ffaf44a1700 1 --2- 172.21.15.107:0/1538215343 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ffad80a2d30 0x7ffad80a51f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ffae8006fd0 tx=0x7ffae8008040 comp rx=0 tx=0).stop 2024-05-09T08:31:29.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.811+0000 7ffaf44a1700 1 -- 172.21.15.107:0/1538215343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec106560 msgr2=0x7ffaec1a4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:29.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.811+0000 7ffaf44a1700 1 --2- 172.21.15.107:0/1538215343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec106560 0x7ffaec1a4970 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ffae4007ee0 tx=0x7ffae400d3b0 comp rx=0 tx=0).stop 2024-05-09T08:31:29.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.811+0000 7ffaf44a1700 1 -- 172.21.15.107:0/1538215343 shutdown_connections 2024-05-09T08:31:29.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.811+0000 7ffaf44a1700 1 --2- 172.21.15.107:0/1538215343 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ffad80a2d30 0x7ffad80a51f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:29.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.811+0000 7ffaf44a1700 1 --2- 172.21.15.107:0/1538215343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec106560 0x7ffaec1a4970 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:29.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.811+0000 7ffaf44a1700 1 --2- 172.21.15.107:0/1538215343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffaec101510 0x7ffaec1998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:29.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.812+0000 7ffaf44a1700 1 --2- 172.21.15.107:0/1538215343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaec101ec0 0x7ffaec199e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:29.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.812+0000 7ffaf44a1700 1 -- 172.21.15.107:0/1538215343 >> 172.21.15.107:0/1538215343 conn(0x7ffaec077270 msgr2=0x7ffaec100a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:29.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.812+0000 7ffaf44a1700 1 -- 172.21.15.107:0/1538215343 shutdown_connections 2024-05-09T08:31:29.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:29.812+0000 7ffaf44a1700 1 -- 172.21.15.107:0/1538215343 wait complete. 2024-05-09T08:31:30.611 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":846948416},{"type":"v1","addr":"172.21.15.107:6801","nonce":846948416}]},"active_addr":"172.21.15.107:6801/846948416","active_change":"2024-05-09T08:23:55.373069+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.107:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2923766500}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2417648229}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":1642826815}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2689615560}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2773893805}]}]} 2024-05-09T08:31:30.617 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-09T08:31:30.617 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-09T08:31:30.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-09T08:31:30.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:30 smithi107 ceph-mon[30122]: pgmap v306: 129 pgs: 129 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-05-09T08:31:30.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:30 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1538215343' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-09T08:31:30.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:30 smithi157 ceph-mon[33614]: pgmap v306: 129 pgs: 129 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-05-09T08:31:30.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:30 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1538215343' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-09T08:31:30.861 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:30.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:30 smithi110 ceph-mon[33622]: pgmap v306: 129 pgs: 129 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-05-09T08:31:30.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:30 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1538215343' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-09T08:31:32.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.619+0000 7efeea715700 1 -- 172.21.15.107:0/3464282595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee4102790 msgr2=0x7efee410e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:32.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.619+0000 7efeea715700 1 --2- 172.21.15.107:0/3464282595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee4102790 0x7efee410e1a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efedc00b0d0 tx=0x7efedc00b3e0 comp rx=0 tx=0).stop 2024-05-09T08:31:32.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.619+0000 7efeea715700 1 -- 172.21.15.107:0/3464282595 shutdown_connections 2024-05-09T08:31:32.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.619+0000 7efeea715700 1 --2- 172.21.15.107:0/3464282595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee4102790 0x7efee410e1a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:32.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.619+0000 7efeea715700 1 --2- 172.21.15.107:0/3464282595 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efee41074f0 0x7efee41078d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:32.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.619+0000 7efeea715700 1 --2- 172.21.15.107:0/3464282595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efee4101dd0 0x7efee4102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:32.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.619+0000 7efeea715700 1 -- 172.21.15.107:0/3464282595 >> 172.21.15.107:0/3464282595 conn(0x7efee4076540 msgr2=0x7efee4076950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:32.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.619+0000 7efeea715700 1 -- 172.21.15.107:0/3464282595 shutdown_connections 2024-05-09T08:31:32.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.619+0000 7efeea715700 1 -- 172.21.15.107:0/3464282595 wait complete. 2024-05-09T08:31:32.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.620+0000 7efeea715700 1 Processor -- start 2024-05-09T08:31:32.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.620+0000 7efeea715700 1 -- start start 2024-05-09T08:31:32.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.620+0000 7efeea715700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efee4101dd0 0x7efee419e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:32.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.620+0000 7efeea715700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efee4102790 0x7efee419ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:32.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.620+0000 7efeea715700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee41074f0 0x7efee41988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:32.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.620+0000 7efeea715700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efee4111510 con 0x7efee41074f0 2024-05-09T08:31:32.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.620+0000 7efeea715700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7efee4111390 con 0x7efee4101dd0 2024-05-09T08:31:32.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.620+0000 7efeea715700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efee4111690 con 0x7efee4102790 2024-05-09T08:31:32.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.620+0000 7efee8cb2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee41074f0 0x7efee41988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:32.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.620+0000 7efee8cb2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee41074f0 0x7efee41988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49676/0 (socket says 172.21.15.107:49676) 2024-05-09T08:31:32.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.620+0000 7efee3fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efee4101dd0 0x7efee419e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:32.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.620+0000 7efee8cb2700 1 -- 172.21.15.107:0/3942756088 learned_addr learned my addr 172.21.15.107:0/3942756088 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:32.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.620+0000 7efedbfff700 1 --2- 172.21.15.107:0/3942756088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efee4102790 0x7efee419ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:32.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.621+0000 7efee8cb2700 1 -- 172.21.15.107:0/3942756088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efee4102790 msgr2=0x7efee419ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:32.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.621+0000 7efee8cb2700 1 --2- 172.21.15.107:0/3942756088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efee4102790 0x7efee419ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:32.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.621+0000 7efee8cb2700 1 -- 172.21.15.107:0/3942756088 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efee4101dd0 msgr2=0x7efee419e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:32.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.621+0000 7efee8cb2700 1 --2- 172.21.15.107:0/3942756088 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efee4101dd0 0x7efee419e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:32.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.621+0000 7efee8cb2700 1 -- 172.21.15.107:0/3942756088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee4199050 con 0x7efee41074f0 2024-05-09T08:31:32.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.621+0000 7efee8cb2700 1 --2- 172.21.15.107:0/3942756088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee41074f0 0x7efee41988f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efedc009fd0 tx=0x7efedc007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:32.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.621+0000 7efee1ffb700 1 -- 172.21.15.107:0/3942756088 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc01f070 con 0x7efee41074f0 2024-05-09T08:31:32.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.621+0000 7efee1ffb700 1 -- 172.21.15.107:0/3942756088 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efedc007e40 con 0x7efee41074f0 2024-05-09T08:31:32.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.621+0000 7efeea715700 1 -- 172.21.15.107:0/3942756088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efee41992e0 con 0x7efee41074f0 2024-05-09T08:31:32.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.621+0000 7efee1ffb700 1 -- 172.21.15.107:0/3942756088 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc01b6a0 con 0x7efee41074f0 2024-05-09T08:31:32.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.621+0000 7efeea715700 1 -- 172.21.15.107:0/3942756088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efee41a5600 con 0x7efee41074f0 2024-05-09T08:31:32.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.622+0000 7efeea715700 1 -- 172.21.15.107:0/3942756088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efee406a670 con 0x7efee41074f0 2024-05-09T08:31:32.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.623+0000 7efee1ffb700 1 -- 172.21.15.107:0/3942756088 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7efedc022070 con 0x7efee41074f0 2024-05-09T08:31:32.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.624+0000 7efee1ffb700 1 --2- 172.21.15.107:0/3942756088 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efed00a2d30 0x7efed00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:32.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.624+0000 7efee3fff700 1 --2- 172.21.15.107:0/3942756088 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efed00a2d30 0x7efed00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:32.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.624+0000 7efee1ffb700 1 -- 172.21.15.107:0/3942756088 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7efedc0cffe0 con 0x7efee41074f0 2024-05-09T08:31:32.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.624+0000 7efee3fff700 1 --2- 172.21.15.107:0/3942756088 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efed00a2d30 0x7efed00a51f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efee4199a30 tx=0x7efed400b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:32.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.628+0000 7efee1ffb700 1 -- 172.21.15.107:0/3942756088 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efedc09cc30 con 0x7efee41074f0 2024-05-09T08:31:32.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:32 smithi107 ceph-mon[30122]: pgmap v307: 129 pgs: 129 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.3 KiB/s wr, 15 op/s 2024-05-09T08:31:32.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.824+0000 7efeea715700 1 -- 172.21.15.107:0/3942756088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efee4069010 con 0x7efee41074f0 2024-05-09T08:31:32.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.826+0000 7efee1ffb700 1 -- 172.21.15.107:0/3942756088 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7efedc003c00 con 0x7efee41074f0 2024-05-09T08:31:32.826 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:32.826 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:31:05.201799+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:31:32.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.828+0000 7efeea715700 1 -- 172.21.15.107:0/3942756088 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efed00a2d30 msgr2=0x7efed00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:32.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.828+0000 7efeea715700 1 --2- 172.21.15.107:0/3942756088 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efed00a2d30 0x7efed00a51f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efee4199a30 tx=0x7efed400b3c0 comp rx=0 tx=0).stop 2024-05-09T08:31:32.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.828+0000 7efeea715700 1 -- 172.21.15.107:0/3942756088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee41074f0 msgr2=0x7efee41988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:32.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.828+0000 7efeea715700 1 --2- 172.21.15.107:0/3942756088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee41074f0 0x7efee41988f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efedc009fd0 tx=0x7efedc007a00 comp rx=0 tx=0).stop 2024-05-09T08:31:32.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.828+0000 7efeea715700 1 -- 172.21.15.107:0/3942756088 shutdown_connections 2024-05-09T08:31:32.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.828+0000 7efeea715700 1 --2- 172.21.15.107:0/3942756088 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efed00a2d30 0x7efed00a51f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:32.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.828+0000 7efeea715700 1 --2- 172.21.15.107:0/3942756088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee41074f0 0x7efee41988f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:32.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.828+0000 7efeea715700 1 --2- 172.21.15.107:0/3942756088 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efee4101dd0 0x7efee419e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:32.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.828+0000 7efeea715700 1 --2- 172.21.15.107:0/3942756088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efee4102790 0x7efee419ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:32.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.828+0000 7efeea715700 1 -- 172.21.15.107:0/3942756088 >> 172.21.15.107:0/3942756088 conn(0x7efee4076540 msgr2=0x7efee4100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:32.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.828+0000 7efeea715700 1 -- 172.21.15.107:0/3942756088 shutdown_connections 2024-05-09T08:31:32.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:32.828+0000 7efeea715700 1 -- 172.21.15.107:0/3942756088 wait complete. 2024-05-09T08:31:32.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:32 smithi157 ceph-mon[33614]: pgmap v307: 129 pgs: 129 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.3 KiB/s wr, 15 op/s 2024-05-09T08:31:32.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:32 smithi110 ceph-mon[33622]: pgmap v307: 129 pgs: 129 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.3 KiB/s wr, 15 op/s 2024-05-09T08:31:33.665 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-09T08:31:33.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-09T08:31:33.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:33 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3942756088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:33.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:33 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3942756088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:33.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:33 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3942756088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:34.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:34 smithi107 ceph-mon[30122]: pgmap v308: 129 pgs: 129 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.2 KiB/s wr, 11 op/s 2024-05-09T08:31:34.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:34 smithi157 ceph-mon[33614]: pgmap v308: 129 pgs: 129 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.2 KiB/s wr, 11 op/s 2024-05-09T08:31:34.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:34 smithi110 ceph-mon[33622]: pgmap v308: 129 pgs: 129 active+clean; 625 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.2 KiB/s wr, 11 op/s 2024-05-09T08:31:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.721+0000 7f677c84a700 1 -- 172.21.15.107:0/2733860880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6778102fe0 msgr2=0x7f6778106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.721+0000 7f677c84a700 1 --2- 172.21.15.107:0/2733860880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6778102fe0 0x7f6778106fb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6768009990 tx=0x7f6768009ca0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.721+0000 7f677c84a700 1 -- 172.21.15.107:0/2733860880 shutdown_connections 2024-05-09T08:31:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.721+0000 7f677c84a700 1 --2- 172.21.15.107:0/2733860880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6778107690 0x7f677810f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.721+0000 7f677c84a700 1 --2- 172.21.15.107:0/2733860880 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6778102630 0x7f6778102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.721+0000 7f677c84a700 1 --2- 172.21.15.107:0/2733860880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6778102fe0 0x7f6778106fb0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.721+0000 7f677c84a700 1 -- 172.21.15.107:0/2733860880 >> 172.21.15.107:0/2733860880 conn(0x7f6778079400 msgr2=0x7f6778079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.721+0000 7f677c84a700 1 -- 172.21.15.107:0/2733860880 shutdown_connections 2024-05-09T08:31:35.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.721+0000 7f677c84a700 1 -- 172.21.15.107:0/2733860880 wait complete. 2024-05-09T08:31:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.722+0000 7f677c84a700 1 Processor -- start 2024-05-09T08:31:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.722+0000 7f677c84a700 1 -- start start 2024-05-09T08:31:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f677c84a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6778102630 0x7f677819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f677c84a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6778102fe0 0x7f677819ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f677c84a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6778107690 0x7f67781988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f677c84a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6778112560 con 0x7f6778102fe0 2024-05-09T08:31:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f677c84a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f67781123e0 con 0x7f6778107690 2024-05-09T08:31:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f677c84a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f67781126e0 con 0x7f6778102630 2024-05-09T08:31:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f6775d9b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6778102fe0 0x7f677819ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f6775d9b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6778102fe0 0x7f677819ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49702/0 (socket says 172.21.15.107:49702) 2024-05-09T08:31:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f6775d9b700 1 -- 172.21.15.107:0/3344891382 learned_addr learned my addr 172.21.15.107:0/3344891382 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f677659c700 1 --2- 172.21.15.107:0/3344891382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6778102630 0x7f677819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f6776d9d700 1 --2- 172.21.15.107:0/3344891382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6778107690 0x7f67781988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f6775d9b700 1 -- 172.21.15.107:0/3344891382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6778102630 msgr2=0x7f677819e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f6775d9b700 1 --2- 172.21.15.107:0/3344891382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6778102630 0x7f677819e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f6775d9b700 1 -- 172.21.15.107:0/3344891382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6778107690 msgr2=0x7f67781988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f6775d9b700 1 --2- 172.21.15.107:0/3344891382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6778107690 0x7f67781988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f6775d9b700 1 -- 172.21.15.107:0/3344891382 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6778199020 con 0x7f6778102fe0 2024-05-09T08:31:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.723+0000 7f6775d9b700 1 --2- 172.21.15.107:0/3344891382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6778102fe0 0x7f677819ec40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6768000c00 tx=0x7f6768003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.724+0000 7f67677fe700 1 -- 172.21.15.107:0/3344891382 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6768027070 con 0x7f6778102fe0 2024-05-09T08:31:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.724+0000 7f67677fe700 1 -- 172.21.15.107:0/3344891382 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f676800bdc0 con 0x7f6778102fe0 2024-05-09T08:31:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.724+0000 7f677c84a700 1 -- 172.21.15.107:0/3344891382 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67781992b0 con 0x7f6778102fe0 2024-05-09T08:31:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.724+0000 7f67677fe700 1 -- 172.21.15.107:0/3344891382 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676800f460 con 0x7f6778102fe0 2024-05-09T08:31:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.724+0000 7f677c84a700 1 -- 172.21.15.107:0/3344891382 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67781a55d0 con 0x7f6778102fe0 2024-05-09T08:31:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.725+0000 7f677c84a700 1 -- 172.21.15.107:0/3344891382 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6778069010 con 0x7f6778102fe0 2024-05-09T08:31:35.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.726+0000 7f67677fe700 1 -- 172.21.15.107:0/3344891382 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f676802a070 con 0x7f6778102fe0 2024-05-09T08:31:35.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.727+0000 7f67677fe700 1 --2- 172.21.15.107:0/3344891382 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f67600a2d30 0x7f67600a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:35.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.727+0000 7f677659c700 1 --2- 172.21.15.107:0/3344891382 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f67600a2d30 0x7f67600a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:35.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.727+0000 7f67677fe700 1 -- 172.21.15.107:0/3344891382 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f67680d0350 con 0x7f6778102fe0 2024-05-09T08:31:35.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.727+0000 7f677659c700 1 --2- 172.21.15.107:0/3344891382 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f67600a2d30 0x7f67600a51f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6778199a00 tx=0x7f676c009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.731+0000 7f67677fe700 1 -- 172.21.15.107:0/3344891382 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f676805e740 con 0x7f6778102fe0 2024-05-09T08:31:35.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.926+0000 7f677c84a700 1 -- 172.21.15.107:0/3344891382 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f677802db80 con 0x7f6778102fe0 2024-05-09T08:31:35.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.927+0000 7f67677fe700 1 -- 172.21.15.107:0/3344891382 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24187 (secure 0 0 0) 0x7f6768013070 con 0x7f6778102fe0 2024-05-09T08:31:35.928 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:31:35.928 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:31:05.201799+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:31:35.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.930+0000 7f677c84a700 1 -- 172.21.15.107:0/3344891382 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f67600a2d30 msgr2=0x7f67600a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:35.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.930+0000 7f677c84a700 1 --2- 172.21.15.107:0/3344891382 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f67600a2d30 0x7f67600a51f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6778199a00 tx=0x7f676c009190 comp rx=0 tx=0).stop 2024-05-09T08:31:35.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.930+0000 7f677c84a700 1 -- 172.21.15.107:0/3344891382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6778102fe0 msgr2=0x7f677819ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:35.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.930+0000 7f677c84a700 1 --2- 172.21.15.107:0/3344891382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6778102fe0 0x7f677819ec40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6768000c00 tx=0x7f6768003a80 comp rx=0 tx=0).stop 2024-05-09T08:31:35.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.930+0000 7f677c84a700 1 -- 172.21.15.107:0/3344891382 shutdown_connections 2024-05-09T08:31:35.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.930+0000 7f677c84a700 1 --2- 172.21.15.107:0/3344891382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6778107690 0x7f67781988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.930+0000 7f677c84a700 1 --2- 172.21.15.107:0/3344891382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6778102630 0x7f677819e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.930+0000 7f677c84a700 1 --2- 172.21.15.107:0/3344891382 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f67600a2d30 0x7f67600a51f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.930+0000 7f677c84a700 1 --2- 172.21.15.107:0/3344891382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6778102fe0 0x7f677819ec40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:35.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.930+0000 7f677c84a700 1 -- 172.21.15.107:0/3344891382 >> 172.21.15.107:0/3344891382 conn(0x7f6778079400 msgr2=0x7f6778108810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:35.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.930+0000 7f677c84a700 1 -- 172.21.15.107:0/3344891382 shutdown_connections 2024-05-09T08:31:35.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:35.930+0000 7f677c84a700 1 -- 172.21.15.107:0/3344891382 wait complete. 2024-05-09T08:31:36.594 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:36 smithi107 ceph-mon[30122]: pgmap v309: 129 pgs: 129 active+clean; 637 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-05-09T08:31:36.594 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:36 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3344891382' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:36.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-09T08:31:36.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-09T08:31:36.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-09T08:31:36.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-09T08:31:36.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-09T08:31:36.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-09T08:31:36.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-09T08:31:36.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-09T08:31:36.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-09T08:31:36.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-09T08:31:36.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-09T08:31:36.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-09T08:31:36.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:36 smithi157 ceph-mon[33614]: pgmap v309: 129 pgs: 129 active+clean; 637 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-05-09T08:31:36.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:36 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3344891382' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:36.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:36 smithi110 ceph-mon[33622]: pgmap v309: 129 pgs: 129 active+clean; 637 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-05-09T08:31:36.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:36 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3344891382' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:31:37.445 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:37.446 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:37.464 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:37.465 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:37.469 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:37.551 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:37.787 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:37.788 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:37.788 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:38.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:38 smithi107 ceph-mon[30122]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-05-09T08:31:38.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:38 smithi157 ceph-mon[33614]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-05-09T08:31:38.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:38 smithi110 ceph-mon[33622]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-05-09T08:31:40.580 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:40 smithi107 ceph-mon[30122]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-05-09T08:31:40.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:40 smithi157 ceph-mon[33614]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-05-09T08:31:40.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:40 smithi110 ceph-mon[33622]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-05-09T08:31:42.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:42 smithi107 ceph-mon[30122]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-09T08:31:42.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:42 smithi157 ceph-mon[33614]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-09T08:31:42.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:42 smithi110 ceph-mon[33622]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-09T08:31:44.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:44 smithi157 ceph-mon[33614]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-05-09T08:31:44.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:44 smithi110 ceph-mon[33622]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-05-09T08:31:45.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:44 smithi107 ceph-mon[30122]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-05-09T08:31:45.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.943+0000 7f0802111700 1 -- 172.21.15.107:0/3027305248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07fc101c80 msgr2=0x7f07fc102060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:45.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.943+0000 7f0802111700 1 --2- 172.21.15.107:0/3027305248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07fc101c80 0x7f07fc102060 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f07e40099c0 tx=0x7f07e4009cd0 comp rx=0 tx=0).stop 2024-05-09T08:31:45.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.943+0000 7f0802111700 1 -- 172.21.15.107:0/3027305248 shutdown_connections 2024-05-09T08:31:45.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.943+0000 7f0802111700 1 --2- 172.21.15.107:0/3027305248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07fc1065a0 0x7f07fc10e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:45.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.943+0000 7f0802111700 1 --2- 172.21.15.107:0/3027305248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07fc101c80 0x7f07fc102060 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:45.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.943+0000 7f0802111700 1 --2- 172.21.15.107:0/3027305248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07fc102630 0x7f07fc105ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:45.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.943+0000 7f0802111700 1 -- 172.21.15.107:0/3027305248 >> 172.21.15.107:0/3027305248 conn(0x7f07fc077890 msgr2=0x7f07fc0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:45.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.943+0000 7f0802111700 1 -- 172.21.15.107:0/3027305248 shutdown_connections 2024-05-09T08:31:45.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.943+0000 7f0802111700 1 -- 172.21.15.107:0/3027305248 wait complete. 2024-05-09T08:31:45.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.944+0000 7f0802111700 1 Processor -- start 2024-05-09T08:31:45.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.944+0000 7f0802111700 1 -- start start 2024-05-09T08:31:45.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.944+0000 7f0802111700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07fc101c80 0x7f07fc1998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:45.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.944+0000 7f0802111700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07fc102630 0x7f07fc199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:45.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.944+0000 7f0802111700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07fc1065a0 0x7f07fc1a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:45.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.944+0000 7f0802111700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f07fc1114f0 con 0x7f07fc102630 2024-05-09T08:31:45.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.944+0000 7f0802111700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f07fc111370 con 0x7f07fc101c80 2024-05-09T08:31:45.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.944+0000 7f0802111700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f07fc111670 con 0x7f07fc1065a0 2024-05-09T08:31:45.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f07faffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07fc102630 0x7f07fc199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:45.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f07faffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07fc102630 0x7f07fc199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41872/0 (socket says 172.21.15.107:41872) 2024-05-09T08:31:45.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f07faffd700 1 -- 172.21.15.107:0/2117625947 learned_addr learned my addr 172.21.15.107:0/2117625947 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:45.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f07fb7fe700 1 --2- 172.21.15.107:0/2117625947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07fc101c80 0x7f07fc1998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:45.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f07fbfff700 1 --2- 172.21.15.107:0/2117625947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07fc1065a0 0x7f07fc1a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:45.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f07faffd700 1 -- 172.21.15.107:0/2117625947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07fc1065a0 msgr2=0x7f07fc1a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:45.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f07faffd700 1 --2- 172.21.15.107:0/2117625947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07fc1065a0 0x7f07fc1a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:45.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f07faffd700 1 -- 172.21.15.107:0/2117625947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07fc101c80 msgr2=0x7f07fc1998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:45.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f07faffd700 1 --2- 172.21.15.107:0/2117625947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07fc101c80 0x7f07fc1998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:45.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f07faffd700 1 -- 172.21.15.107:0/2117625947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07fc1a4e90 con 0x7f07fc102630 2024-05-09T08:31:45.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f07fbfff700 1 --2- 172.21.15.107:0/2117625947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07fc1065a0 0x7f07fc1a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:45.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f07faffd700 1 --2- 172.21.15.107:0/2117625947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07fc102630 0x7f07fc199e10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f07ec00b500 tx=0x7f07ec00b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:45.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f07f8ff9700 1 -- 172.21.15.107:0/2117625947 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec015020 con 0x7f07fc102630 2024-05-09T08:31:45.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f07f8ff9700 1 -- 172.21.15.107:0/2117625947 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07ec011ce0 con 0x7f07fc102630 2024-05-09T08:31:45.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f0802111700 1 -- 172.21.15.107:0/2117625947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07fc1a50c0 con 0x7f07fc102630 2024-05-09T08:31:45.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f07f8ff9700 1 -- 172.21.15.107:0/2117625947 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec010d40 con 0x7f07fc102630 2024-05-09T08:31:45.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.945+0000 7f0802111700 1 -- 172.21.15.107:0/2117625947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07fc1a55b0 con 0x7f07fc102630 2024-05-09T08:31:45.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.946+0000 7f0802111700 1 -- 172.21.15.107:0/2117625947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f07fc062a30 con 0x7f07fc102630 2024-05-09T08:31:45.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.948+0000 7f07f8ff9700 1 -- 172.21.15.107:0/2117625947 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f07ec018020 con 0x7f07fc102630 2024-05-09T08:31:45.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.948+0000 7f07f8ff9700 1 --2- 172.21.15.107:0/2117625947 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f07e80a2cb0 0x7f07e80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.948+0000 7f07fb7fe700 1 --2- 172.21.15.107:0/2117625947 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f07e80a2cb0 0x7f07e80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.948+0000 7f07f8ff9700 1 -- 172.21.15.107:0/2117625947 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f07ec0cf1e0 con 0x7f07fc102630 2024-05-09T08:31:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.949+0000 7f07f8ff9700 1 --2- 172.21.15.107:0/2117625947 >> [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420] conn(0x7f07e80ad970 0x7f07e80afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.949+0000 7f07f8ff9700 1 -- 172.21.15.107:0/2117625947 --> [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07ec018ec0 con 0x7f07e80ad970 2024-05-09T08:31:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.949+0000 7f07f8ff9700 1 -- 172.21.15.107:0/2117625947 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f07ec0cf630 con 0x7f07fc102630 2024-05-09T08:31:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.949+0000 7f07fb7fe700 1 --2- 172.21.15.107:0/2117625947 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f07e80a2cb0 0x7f07e80a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f07e4000c00 tx=0x7f07e4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.949+0000 7f07fbfff700 1 --2- 172.21.15.107:0/2117625947 >> [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420] conn(0x7f07e80ad970 0x7f07e80afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.950+0000 7f07fbfff700 1 --2- 172.21.15.107:0/2117625947 >> [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420] conn(0x7f07e80ad970 0x7f07e80afdb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.950+0000 7f07f8ff9700 1 -- 172.21.15.107:0/2117625947 <== osd.5 v2:172.21.15.157:6808/423586420 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f07ec018ec0 con 0x7f07e80ad970 2024-05-09T08:31:45.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.967+0000 7f0802111700 1 -- 172.21.15.107:0/2117625947 --> [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f07fc062e30 con 0x7f07e80ad970 2024-05-09T08:31:45.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.967+0000 7f07f8ff9700 1 -- 172.21.15.107:0/2117625947 <== osd.5 v2:172.21.15.157:6808/423586420 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f07fc062e30 con 0x7f07e80ad970 2024-05-09T08:31:45.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.968+0000 7f0802111700 1 -- 172.21.15.107:0/2117625947 >> [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420] conn(0x7f07e80ad970 msgr2=0x7f07e80afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:45.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.968+0000 7f0802111700 1 --2- 172.21.15.107:0/2117625947 >> [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420] conn(0x7f07e80ad970 0x7f07e80afdb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:45.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.968+0000 7f0802111700 1 -- 172.21.15.107:0/2117625947 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f07e80a2cb0 msgr2=0x7f07e80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:45.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.968+0000 7f0802111700 1 --2- 172.21.15.107:0/2117625947 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f07e80a2cb0 0x7f07e80a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f07e4000c00 tx=0x7f07e4005fb0 comp rx=0 tx=0).stop 2024-05-09T08:31:45.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.968+0000 7f0802111700 1 -- 172.21.15.107:0/2117625947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07fc102630 msgr2=0x7f07fc199e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:45.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.968+0000 7f0802111700 1 --2- 172.21.15.107:0/2117625947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07fc102630 0x7f07fc199e10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f07ec00b500 tx=0x7f07ec00b810 comp rx=0 tx=0).stop 2024-05-09T08:31:45.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.968+0000 7f07fbfff700 1 -- 172.21.15.107:0/2117625947 reap_dead start 2024-05-09T08:31:45.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.968+0000 7f0802111700 1 -- 172.21.15.107:0/2117625947 shutdown_connections 2024-05-09T08:31:45.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.968+0000 7f0802111700 1 -- 172.21.15.107:0/2117625947 >> 172.21.15.107:0/2117625947 conn(0x7f07fc077890 msgr2=0x7f07fc0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:45.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.968+0000 7f0802111700 1 -- 172.21.15.107:0/2117625947 shutdown_connections 2024-05-09T08:31:45.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:45.968+0000 7f0802111700 1 -- 172.21.15.107:0/2117625947 wait complete. 2024-05-09T08:31:46.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:46 smithi107 ceph-mon[30122]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-05-09T08:31:46.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:46 smithi157 ceph-mon[33614]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-05-09T08:31:46.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:46 smithi110 ceph-mon[33622]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-05-09T08:31:47.178 INFO:teuthology.orchestra.run.smithi107.stdout:163208757301 2024-05-09T08:31:47.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-09T08:31:48.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.208+0000 7fb82408b700 1 -- 172.21.15.107:0/2255442752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb81c101250 msgr2=0x7fb81c1016d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:48.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.208+0000 7fb82408b700 1 --2- 172.21.15.107:0/2255442752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb81c101250 0x7fb81c1016d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009990 tx=0x7fb80c009ca0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.208+0000 7fb82408b700 1 -- 172.21.15.107:0/2255442752 shutdown_connections 2024-05-09T08:31:48.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.208+0000 7fb82408b700 1 --2- 172.21.15.107:0/2255442752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb81c101c10 0x7fb81c10e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.208+0000 7fb82408b700 1 --2- 172.21.15.107:0/2255442752 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb81c1072c0 0x7fb81c1076a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.208+0000 7fb82408b700 1 --2- 172.21.15.107:0/2255442752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb81c101250 0x7fb81c1016d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.208+0000 7fb82408b700 1 -- 172.21.15.107:0/2255442752 >> 172.21.15.107:0/2255442752 conn(0x7fb81c078260 msgr2=0x7fb81c078670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:48.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.208+0000 7fb82408b700 1 -- 172.21.15.107:0/2255442752 shutdown_connections 2024-05-09T08:31:48.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.208+0000 7fb82408b700 1 -- 172.21.15.107:0/2255442752 wait complete. 2024-05-09T08:31:48.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.208+0000 7fb82408b700 1 Processor -- start 2024-05-09T08:31:48.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb82408b700 1 -- start start 2024-05-09T08:31:48.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb82408b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb81c101250 0x7fb81c19d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb82408b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb81c101c10 0x7fb81c19dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb82408b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb81c1072c0 0x7fb81c197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb82408b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c111400 con 0x7fb81c101c10 2024-05-09T08:31:48.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb82408b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c111280 con 0x7fb81c1072c0 2024-05-09T08:31:48.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb82408b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c111580 con 0x7fb81c101250 2024-05-09T08:31:48.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb821626700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb81c101c10 0x7fb81c19dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb821626700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb81c101c10 0x7fb81c19dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41886/0 (socket says 172.21.15.107:41886) 2024-05-09T08:31:48.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb821626700 1 -- 172.21.15.107:0/2256138715 learned_addr learned my addr 172.21.15.107:0/2256138715 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:48.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb821e27700 1 --2- 172.21.15.107:0/2256138715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb81c101250 0x7fb81c19d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb822628700 1 --2- 172.21.15.107:0/2256138715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb81c1072c0 0x7fb81c197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb821626700 1 -- 172.21.15.107:0/2256138715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb81c101250 msgr2=0x7fb81c19d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:48.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb821626700 1 --2- 172.21.15.107:0/2256138715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb81c101250 0x7fb81c19d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb821626700 1 -- 172.21.15.107:0/2256138715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb81c1072c0 msgr2=0x7fb81c197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:48.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb821626700 1 --2- 172.21.15.107:0/2256138715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb81c1072c0 0x7fb81c197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb821626700 1 -- 172.21.15.107:0/2256138715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb81c198020 con 0x7fb81c101c10 2024-05-09T08:31:48.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.209+0000 7fb822628700 1 --2- 172.21.15.107:0/2256138715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb81c1072c0 0x7fb81c197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:48.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.210+0000 7fb821626700 1 --2- 172.21.15.107:0/2256138715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb81c101c10 0x7fb81c19dbb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb80c000c00 tx=0x7fb80c01a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:48.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.210+0000 7fb812ffd700 1 -- 172.21.15.107:0/2256138715 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c02a070 con 0x7fb81c101c10 2024-05-09T08:31:48.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.210+0000 7fb812ffd700 1 -- 172.21.15.107:0/2256138715 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb80c00bdc0 con 0x7fb81c101c10 2024-05-09T08:31:48.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.210+0000 7fb812ffd700 1 -- 172.21.15.107:0/2256138715 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c0040b0 con 0x7fb81c101c10 2024-05-09T08:31:48.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.210+0000 7fb82408b700 1 -- 172.21.15.107:0/2256138715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb81c1982b0 con 0x7fb81c101c10 2024-05-09T08:31:48.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.210+0000 7fb82408b700 1 -- 172.21.15.107:0/2256138715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb81c06a9f0 con 0x7fb81c101c10 2024-05-09T08:31:48.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.211+0000 7fb82408b700 1 -- 172.21.15.107:0/2256138715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb81c06afa0 con 0x7fb81c101c10 2024-05-09T08:31:48.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.212+0000 7fb812ffd700 1 -- 172.21.15.107:0/2256138715 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fb80c02d070 con 0x7fb81c101c10 2024-05-09T08:31:48.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.213+0000 7fb812ffd700 1 --2- 172.21.15.107:0/2256138715 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb8080a70c0 0x7fb8080a9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.213+0000 7fb821e27700 1 --2- 172.21.15.107:0/2256138715 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb8080a70c0 0x7fb8080a9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.213+0000 7fb812ffd700 1 -- 172.21.15.107:0/2256138715 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb80c0d0160 con 0x7fb81c101c10 2024-05-09T08:31:48.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.213+0000 7fb812ffd700 1 --2- 172.21.15.107:0/2256138715 >> [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505] conn(0x7fb8080b1d00 0x7fb8080b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:48.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.213+0000 7fb812ffd700 1 -- 172.21.15.107:0/2256138715 --> [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb80c027ec0 con 0x7fb8080b1d00 2024-05-09T08:31:48.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.213+0000 7fb812ffd700 1 -- 172.21.15.107:0/2256138715 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fb80c0d05b0 con 0x7fb81c101c10 2024-05-09T08:31:48.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.213+0000 7fb821e27700 1 --2- 172.21.15.107:0/2256138715 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb8080a70c0 0x7fb8080a9580 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb8180045b0 tx=0x7fb81800b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:48.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.214+0000 7fb822628700 1 --2- 172.21.15.107:0/2256138715 >> [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505] conn(0x7fb8080b1d00 0x7fb8080b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:48.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.214+0000 7fb822628700 1 --2- 172.21.15.107:0/2256138715 >> [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505] conn(0x7fb8080b1d00 0x7fb8080b4140 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:48.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.215+0000 7fb812ffd700 1 -- 172.21.15.107:0/2256138715 <== osd.4 v2:172.21.15.110:6808/2317455505 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb80c027ec0 con 0x7fb8080b1d00 2024-05-09T08:31:48.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.231+0000 7fb82408b700 1 -- 172.21.15.107:0/2256138715 --> [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb81c06b420 con 0x7fb8080b1d00 2024-05-09T08:31:48.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.232+0000 7fb812ffd700 1 -- 172.21.15.107:0/2256138715 <== osd.4 v2:172.21.15.110:6808/2317455505 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb81c06b420 con 0x7fb8080b1d00 2024-05-09T08:31:48.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.232+0000 7fb82408b700 1 -- 172.21.15.107:0/2256138715 >> [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505] conn(0x7fb8080b1d00 msgr2=0x7fb8080b4140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:48.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.232+0000 7fb82408b700 1 --2- 172.21.15.107:0/2256138715 >> [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505] conn(0x7fb8080b1d00 0x7fb8080b4140 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.232+0000 7fb82408b700 1 -- 172.21.15.107:0/2256138715 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb8080a70c0 msgr2=0x7fb8080a9580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:48.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.232+0000 7fb82408b700 1 --2- 172.21.15.107:0/2256138715 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb8080a70c0 0x7fb8080a9580 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb8180045b0 tx=0x7fb81800b3c0 comp rx=0 tx=0).stop 2024-05-09T08:31:48.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.232+0000 7fb82408b700 1 -- 172.21.15.107:0/2256138715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb81c101c10 msgr2=0x7fb81c19dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:48.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.232+0000 7fb82408b700 1 --2- 172.21.15.107:0/2256138715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb81c101c10 0x7fb81c19dbb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb80c000c00 tx=0x7fb80c01a840 comp rx=0 tx=0).stop 2024-05-09T08:31:48.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.232+0000 7fb822628700 1 -- 172.21.15.107:0/2256138715 reap_dead start 2024-05-09T08:31:48.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.232+0000 7fb82408b700 1 -- 172.21.15.107:0/2256138715 shutdown_connections 2024-05-09T08:31:48.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.232+0000 7fb82408b700 1 -- 172.21.15.107:0/2256138715 >> 172.21.15.107:0/2256138715 conn(0x7fb81c078260 msgr2=0x7fb81c1000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:48.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.232+0000 7fb82408b700 1 -- 172.21.15.107:0/2256138715 shutdown_connections 2024-05-09T08:31:48.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:48.232+0000 7fb82408b700 1 -- 172.21.15.107:0/2256138715 wait complete. 2024-05-09T08:31:48.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:48 smithi107 ceph-mon[30122]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:31:48.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:48 smithi157 ceph-mon[33614]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:31:48.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:48 smithi110 ceph-mon[33622]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:31:48.925 INFO:teuthology.orchestra.run.smithi107.stdout:141733920827 2024-05-09T08:31:48.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-09T08:31:49.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.561+0000 7f9f765d8700 1 -- 172.21.15.107:0/1286988847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f70101220 msgr2=0x7f9f701016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:49.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.561+0000 7f9f765d8700 1 --2- 172.21.15.107:0/1286988847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f70101220 0x7f9f701016a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9f58009990 tx=0x7f9f58009ca0 comp rx=0 tx=0).stop 2024-05-09T08:31:49.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.561+0000 7f9f765d8700 1 -- 172.21.15.107:0/1286988847 shutdown_connections 2024-05-09T08:31:49.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.561+0000 7f9f765d8700 1 --2- 172.21.15.107:0/1286988847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f70101be0 0x7f9f7010e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:49.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.561+0000 7f9f765d8700 1 --2- 172.21.15.107:0/1286988847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f70107290 0x7f9f70107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:49.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.561+0000 7f9f765d8700 1 --2- 172.21.15.107:0/1286988847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f70101220 0x7f9f701016a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:49.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.561+0000 7f9f765d8700 1 -- 172.21.15.107:0/1286988847 >> 172.21.15.107:0/1286988847 conn(0x7f9f700781d0 msgr2=0x7f9f700785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:49.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.561+0000 7f9f765d8700 1 -- 172.21.15.107:0/1286988847 shutdown_connections 2024-05-09T08:31:49.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.561+0000 7f9f765d8700 1 -- 172.21.15.107:0/1286988847 wait complete. 2024-05-09T08:31:49.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.561+0000 7f9f765d8700 1 Processor -- start 2024-05-09T08:31:49.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f765d8700 1 -- start start 2024-05-09T08:31:49.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f765d8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f70101220 0x7f9f70199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:49.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f765d8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f70101be0 0x7f9f70199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:49.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f765d8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f70107290 0x7f9f701a4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:49.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f765d8700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9f701113d0 con 0x7f9f70101be0 2024-05-09T08:31:49.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f765d8700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9f70111250 con 0x7f9f70107290 2024-05-09T08:31:49.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f765d8700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9f70111550 con 0x7f9f70101220 2024-05-09T08:31:49.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f6f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f70101be0 0x7f9f70199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:49.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f6f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f70101be0 0x7f9f70199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41906/0 (socket says 172.21.15.107:41906) 2024-05-09T08:31:49.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f6f7fe700 1 -- 172.21.15.107:0/3957374498 learned_addr learned my addr 172.21.15.107:0/3957374498 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:49.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f6ffff700 1 --2- 172.21.15.107:0/3957374498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f70101220 0x7f9f70199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:49.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f74b75700 1 --2- 172.21.15.107:0/3957374498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f70107290 0x7f9f701a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:49.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f6f7fe700 1 -- 172.21.15.107:0/3957374498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f70101220 msgr2=0x7f9f70199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:49.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f6f7fe700 1 --2- 172.21.15.107:0/3957374498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f70101220 0x7f9f70199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:49.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f6f7fe700 1 -- 172.21.15.107:0/3957374498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f70107290 msgr2=0x7f9f701a4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:49.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f6f7fe700 1 --2- 172.21.15.107:0/3957374498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f70107290 0x7f9f701a4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:49.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f6f7fe700 1 -- 172.21.15.107:0/3957374498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f701a4dd0 con 0x7f9f70101be0 2024-05-09T08:31:49.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.562+0000 7f9f74b75700 1 --2- 172.21.15.107:0/3957374498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f70107290 0x7f9f701a4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:31:49.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.563+0000 7f9f6f7fe700 1 --2- 172.21.15.107:0/3957374498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f70101be0 0x7f9f70199d50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9f58000c00 tx=0x7f9f58003b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:49.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.563+0000 7f9f6d7fa700 1 -- 172.21.15.107:0/3957374498 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58027070 con 0x7f9f70101be0 2024-05-09T08:31:49.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.563+0000 7f9f6d7fa700 1 -- 172.21.15.107:0/3957374498 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f5800bdc0 con 0x7f9f70101be0 2024-05-09T08:31:49.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.563+0000 7f9f765d8700 1 -- 172.21.15.107:0/3957374498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f701a4fa0 con 0x7f9f70101be0 2024-05-09T08:31:49.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.563+0000 7f9f6d7fa700 1 -- 172.21.15.107:0/3957374498 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5800f460 con 0x7f9f70101be0 2024-05-09T08:31:49.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.563+0000 7f9f765d8700 1 -- 172.21.15.107:0/3957374498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f701a5390 con 0x7f9f70101be0 2024-05-09T08:31:49.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.563+0000 7f9f765d8700 1 -- 172.21.15.107:0/3957374498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9f700629b0 con 0x7f9f70101be0 2024-05-09T08:31:49.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.565+0000 7f9f6d7fa700 1 -- 172.21.15.107:0/3957374498 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f9f5802a070 con 0x7f9f70101be0 2024-05-09T08:31:49.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.566+0000 7f9f6d7fa700 1 --2- 172.21.15.107:0/3957374498 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9f5c0a2cb0 0x7f9f5c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:49.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.566+0000 7f9f6ffff700 1 --2- 172.21.15.107:0/3957374498 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9f5c0a2cb0 0x7f9f5c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:49.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.566+0000 7f9f6d7fa700 1 -- 172.21.15.107:0/3957374498 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9f580d08f0 con 0x7f9f70101be0 2024-05-09T08:31:49.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.566+0000 7f9f6d7fa700 1 --2- 172.21.15.107:0/3957374498 >> [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647] conn(0x7f9f5c0ad970 0x7f9f5c0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:49.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.566+0000 7f9f6d7fa700 1 -- 172.21.15.107:0/3957374498 --> [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f580095d0 con 0x7f9f5c0ad970 2024-05-09T08:31:49.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.566+0000 7f9f6d7fa700 1 -- 172.21.15.107:0/3957374498 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f9f580d0d40 con 0x7f9f70101be0 2024-05-09T08:31:49.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.566+0000 7f9f6ffff700 1 --2- 172.21.15.107:0/3957374498 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9f5c0a2cb0 0x7f9f5c0a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9f6000bbb0 tx=0x7f9f6000b590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:49.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.566+0000 7f9f74b75700 1 --2- 172.21.15.107:0/3957374498 >> [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647] conn(0x7f9f5c0ad970 0x7f9f5c0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:49.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.567+0000 7f9f74b75700 1 --2- 172.21.15.107:0/3957374498 >> [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647] conn(0x7f9f5c0ad970 0x7f9f5c0afdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:49.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.568+0000 7f9f6d7fa700 1 -- 172.21.15.107:0/3957374498 <== osd.2 v2:172.21.15.157:6800/3005990647 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9f580095d0 con 0x7f9f5c0ad970 2024-05-09T08:31:49.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.584+0000 7f9f765d8700 1 -- 172.21.15.107:0/3957374498 --> [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9f70062e30 con 0x7f9f5c0ad970 2024-05-09T08:31:49.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.585+0000 7f9f6d7fa700 1 -- 172.21.15.107:0/3957374498 <== osd.2 v2:172.21.15.157:6800/3005990647 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9f70062e30 con 0x7f9f5c0ad970 2024-05-09T08:31:49.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.585+0000 7f9f765d8700 1 -- 172.21.15.107:0/3957374498 >> [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647] conn(0x7f9f5c0ad970 msgr2=0x7f9f5c0afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:49.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.585+0000 7f9f765d8700 1 --2- 172.21.15.107:0/3957374498 >> [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647] conn(0x7f9f5c0ad970 0x7f9f5c0afdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:49.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.585+0000 7f9f765d8700 1 -- 172.21.15.107:0/3957374498 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9f5c0a2cb0 msgr2=0x7f9f5c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:49.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.585+0000 7f9f765d8700 1 --2- 172.21.15.107:0/3957374498 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9f5c0a2cb0 0x7f9f5c0a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9f6000bbb0 tx=0x7f9f6000b590 comp rx=0 tx=0).stop 2024-05-09T08:31:49.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.585+0000 7f9f765d8700 1 -- 172.21.15.107:0/3957374498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f70101be0 msgr2=0x7f9f70199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:49.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.585+0000 7f9f765d8700 1 --2- 172.21.15.107:0/3957374498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f70101be0 0x7f9f70199d50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9f58000c00 tx=0x7f9f58003b30 comp rx=0 tx=0).stop 2024-05-09T08:31:49.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.585+0000 7f9f74b75700 1 -- 172.21.15.107:0/3957374498 reap_dead start 2024-05-09T08:31:49.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.585+0000 7f9f765d8700 1 -- 172.21.15.107:0/3957374498 shutdown_connections 2024-05-09T08:31:49.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.585+0000 7f9f765d8700 1 -- 172.21.15.107:0/3957374498 >> 172.21.15.107:0/3957374498 conn(0x7f9f700781d0 msgr2=0x7f9f700ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:49.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.585+0000 7f9f765d8700 1 -- 172.21.15.107:0/3957374498 shutdown_connections 2024-05-09T08:31:49.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.585+0000 7f9f765d8700 1 -- 172.21.15.107:0/3957374498 wait complete. 2024-05-09T08:31:49.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.786+0000 7f7ed38e4700 1 -- 172.21.15.107:0/350505096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ecc101220 msgr2=0x7f7ecc1016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:49.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.786+0000 7f7ed38e4700 1 --2- 172.21.15.107:0/350505096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ecc101220 0x7f7ecc1016a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0009990 tx=0x7f7ec0009ca0 comp rx=0 tx=0).stop 2024-05-09T08:31:49.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.786+0000 7f7ed38e4700 1 -- 172.21.15.107:0/350505096 shutdown_connections 2024-05-09T08:31:49.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.786+0000 7f7ed38e4700 1 --2- 172.21.15.107:0/350505096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ecc101be0 0x7f7ecc10e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:49.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.786+0000 7f7ed38e4700 1 --2- 172.21.15.107:0/350505096 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ecc107290 0x7f7ecc107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:49.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.786+0000 7f7ed38e4700 1 --2- 172.21.15.107:0/350505096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ecc101220 0x7f7ecc1016a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:49.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.786+0000 7f7ed38e4700 1 -- 172.21.15.107:0/350505096 >> 172.21.15.107:0/350505096 conn(0x7f7ecc0781d0 msgr2=0x7f7ecc0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:49.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.786+0000 7f7ed38e4700 1 -- 172.21.15.107:0/350505096 shutdown_connections 2024-05-09T08:31:49.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.786+0000 7f7ed38e4700 1 -- 172.21.15.107:0/350505096 wait complete. 2024-05-09T08:31:49.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.787+0000 7f7ed38e4700 1 Processor -- start 2024-05-09T08:31:49.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.787+0000 7f7ed38e4700 1 -- start start 2024-05-09T08:31:49.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.787+0000 7f7ed38e4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ecc101220 0x7f7ecc102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:49.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.787+0000 7f7ed38e4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ecc101be0 0x7f7ecc103440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:49.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.787+0000 7f7ed38e4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ecc107290 0x7f7ecc103980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:49.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.787+0000 7f7ed38e4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc1113b0 con 0x7f7ecc101be0 2024-05-09T08:31:49.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.787+0000 7f7ed38e4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc111230 con 0x7f7ecc101220 2024-05-09T08:31:49.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.787+0000 7f7ed38e4700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc111530 con 0x7f7ecc107290 2024-05-09T08:31:49.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.787+0000 7f7ed0e7f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ecc101be0 0x7f7ecc103440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:49.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.787+0000 7f7ed0e7f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ecc101be0 0x7f7ecc103440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41932/0 (socket says 172.21.15.107:41932) 2024-05-09T08:31:49.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.787+0000 7f7ed0e7f700 1 -- 172.21.15.107:0/988806955 learned_addr learned my addr 172.21.15.107:0/988806955 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:49.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.787+0000 7f7ed1e81700 1 --2- 172.21.15.107:0/988806955 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ecc107290 0x7f7ecc103980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:49.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.787+0000 7f7ed1680700 1 --2- 172.21.15.107:0/988806955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ecc101220 0x7f7ecc102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:49.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.788+0000 7f7ed0e7f700 1 -- 172.21.15.107:0/988806955 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ecc107290 msgr2=0x7f7ecc103980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:49.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.788+0000 7f7ed0e7f700 1 --2- 172.21.15.107:0/988806955 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ecc107290 0x7f7ecc103980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:49.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.788+0000 7f7ed0e7f700 1 -- 172.21.15.107:0/988806955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ecc101220 msgr2=0x7f7ecc102f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:49.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.788+0000 7f7ed0e7f700 1 --2- 172.21.15.107:0/988806955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ecc101220 0x7f7ecc102f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:49.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.788+0000 7f7ed0e7f700 1 -- 172.21.15.107:0/988806955 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ecc1a6e90 con 0x7f7ecc101be0 2024-05-09T08:31:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.788+0000 7f7ed0e7f700 1 --2- 172.21.15.107:0/988806955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ecc101be0 0x7f7ecc103440 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0000c00 tx=0x7f7ec0003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.788+0000 7f7eca7fc700 1 -- 172.21.15.107:0/988806955 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0027070 con 0x7f7ecc101be0 2024-05-09T08:31:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.788+0000 7f7eca7fc700 1 -- 172.21.15.107:0/988806955 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ec000bdc0 con 0x7f7ecc101be0 2024-05-09T08:31:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.788+0000 7f7eca7fc700 1 -- 172.21.15.107:0/988806955 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec000f460 con 0x7f7ecc101be0 2024-05-09T08:31:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.788+0000 7f7ed38e4700 1 -- 172.21.15.107:0/988806955 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ecc1a7120 con 0x7f7ecc101be0 2024-05-09T08:31:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.788+0000 7f7ed38e4700 1 -- 172.21.15.107:0/988806955 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ecc1a7570 con 0x7f7ecc101be0 2024-05-09T08:31:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.789+0000 7f7ed38e4700 1 -- 172.21.15.107:0/988806955 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7ecc105310 con 0x7f7ecc101be0 2024-05-09T08:31:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.790+0000 7f7eca7fc700 1 -- 172.21.15.107:0/988806955 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f7ec002a070 con 0x7f7ecc101be0 2024-05-09T08:31:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.791+0000 7f7eca7fc700 1 --2- 172.21.15.107:0/988806955 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7eb40a2cb0 0x7f7eb40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.791+0000 7f7ed1680700 1 --2- 172.21.15.107:0/988806955 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7eb40a2cb0 0x7f7eb40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.791+0000 7f7eca7fc700 1 -- 172.21.15.107:0/988806955 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7ec00d0b90 con 0x7f7ecc101be0 2024-05-09T08:31:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.791+0000 7f7eca7fc700 1 --2- 172.21.15.107:0/988806955 >> [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372] conn(0x7f7eb40ad970 0x7f7eb40afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.791+0000 7f7eca7fc700 1 -- 172.21.15.107:0/988806955 --> [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7eb40b0480 con 0x7f7eb40ad970 2024-05-09T08:31:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.791+0000 7f7eca7fc700 1 -- 172.21.15.107:0/988806955 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f7ec00d0fe0 con 0x7f7ecc101be0 2024-05-09T08:31:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.791+0000 7f7ed1e81700 1 --2- 172.21.15.107:0/988806955 >> [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372] conn(0x7f7eb40ad970 0x7f7eb40afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.791+0000 7f7ed1680700 1 --2- 172.21.15.107:0/988806955 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7eb40a2cb0 0x7f7eb40a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc007fd0 tx=0x7f7ebc00c040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.791+0000 7f7ed1e81700 1 --2- 172.21.15.107:0/988806955 >> [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372] conn(0x7f7eb40ad970 0x7f7eb40afdb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.792+0000 7f7eca7fc700 1 -- 172.21.15.107:0/988806955 <== osd.6 v2:172.21.15.107:6818/3011505372 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7eb40b0480 con 0x7f7eb40ad970 2024-05-09T08:31:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.808+0000 7f7ed38e4700 1 -- 172.21.15.107:0/988806955 --> [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7ecc102940 con 0x7f7eb40ad970 2024-05-09T08:31:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.808+0000 7f7eca7fc700 1 -- 172.21.15.107:0/988806955 <== osd.6 v2:172.21.15.107:6818/3011505372 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7ecc102940 con 0x7f7eb40ad970 2024-05-09T08:31:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.809+0000 7f7ed38e4700 1 -- 172.21.15.107:0/988806955 >> [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372] conn(0x7f7eb40ad970 msgr2=0x7f7eb40afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.809+0000 7f7ed38e4700 1 --2- 172.21.15.107:0/988806955 >> [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372] conn(0x7f7eb40ad970 0x7f7eb40afdb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.809+0000 7f7ed38e4700 1 -- 172.21.15.107:0/988806955 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7eb40a2cb0 msgr2=0x7f7eb40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.809+0000 7f7ed38e4700 1 --2- 172.21.15.107:0/988806955 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7eb40a2cb0 0x7f7eb40a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc007fd0 tx=0x7f7ebc00c040 comp rx=0 tx=0).stop 2024-05-09T08:31:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.809+0000 7f7ed38e4700 1 -- 172.21.15.107:0/988806955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ecc101be0 msgr2=0x7f7ecc103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:49.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.809+0000 7f7ed38e4700 1 --2- 172.21.15.107:0/988806955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ecc101be0 0x7f7ecc103440 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0000c00 tx=0x7f7ec0003a80 comp rx=0 tx=0).stop 2024-05-09T08:31:49.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.809+0000 7f7ed1e81700 1 -- 172.21.15.107:0/988806955 reap_dead start 2024-05-09T08:31:49.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.809+0000 7f7ed38e4700 1 -- 172.21.15.107:0/988806955 shutdown_connections 2024-05-09T08:31:49.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.809+0000 7f7ed38e4700 1 -- 172.21.15.107:0/988806955 >> 172.21.15.107:0/988806955 conn(0x7f7ecc0781d0 msgr2=0x7f7ecc1004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:49.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.809+0000 7f7ed38e4700 1 -- 172.21.15.107:0/988806955 shutdown_connections 2024-05-09T08:31:49.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:49.809+0000 7f7ed38e4700 1 -- 172.21.15.107:0/988806955 wait complete. 2024-05-09T08:31:50.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:50 smithi107 ceph-mon[30122]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:31:50.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:50 smithi157 ceph-mon[33614]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:31:50.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:50 smithi110 ceph-mon[33622]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:31:51.736 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:51.737 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:51.737 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:31:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.360+0000 7fc9f4a07700 1 -- 172.21.15.107:0/3168294663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f0102f80 msgr2=0x7fc9f0106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.360+0000 7fc9f4a07700 1 --2- 172.21.15.107:0/3168294663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f0102f80 0x7fc9f0106f80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8009990 tx=0x7fc9d8009ca0 comp rx=0 tx=0).stop 2024-05-09T08:31:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.360+0000 7fc9f4a07700 1 -- 172.21.15.107:0/3168294663 shutdown_connections 2024-05-09T08:31:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.360+0000 7fc9f4a07700 1 --2- 172.21.15.107:0/3168294663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9f0107660 0x7fc9f010f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.360+0000 7fc9f4a07700 1 --2- 172.21.15.107:0/3168294663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9f01025d0 0x7fc9f01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.360+0000 7fc9f4a07700 1 --2- 172.21.15.107:0/3168294663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f0102f80 0x7fc9f0106f80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.360+0000 7fc9f4a07700 1 -- 172.21.15.107:0/3168294663 >> 172.21.15.107:0/3168294663 conn(0x7fc9f0079480 msgr2=0x7fc9f0079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.360+0000 7fc9f4a07700 1 -- 172.21.15.107:0/3168294663 shutdown_connections 2024-05-09T08:31:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.360+0000 7fc9f4a07700 1 -- 172.21.15.107:0/3168294663 wait complete. 2024-05-09T08:31:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.361+0000 7fc9f4a07700 1 Processor -- start 2024-05-09T08:31:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.361+0000 7fc9f4a07700 1 -- start start 2024-05-09T08:31:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.361+0000 7fc9f4a07700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9f01025d0 0x7fc9f019e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.361+0000 7fc9f4a07700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9f0102f80 0x7fc9f019ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.361+0000 7fc9f4a07700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f0107660 0x7fc9f01988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.361+0000 7fc9f4a07700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f0112530 con 0x7fc9f0107660 2024-05-09T08:31:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.361+0000 7fc9f4a07700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f01123b0 con 0x7fc9f01025d0 2024-05-09T08:31:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.361+0000 7fc9f4a07700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f01126b0 con 0x7fc9f0102f80 2024-05-09T08:31:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.362+0000 7fc9eed9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f0107660 0x7fc9f01988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.362+0000 7fc9eed9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f0107660 0x7fc9f01988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56872/0 (socket says 172.21.15.107:56872) 2024-05-09T08:31:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.362+0000 7fc9eed9d700 1 -- 172.21.15.107:0/2863155510 learned_addr learned my addr 172.21.15.107:0/2863155510 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.362+0000 7fc9ee59c700 1 --2- 172.21.15.107:0/2863155510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9f01025d0 0x7fc9f019e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.362+0000 7fc9edd9b700 1 --2- 172.21.15.107:0/2863155510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9f0102f80 0x7fc9f019ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.362+0000 7fc9eed9d700 1 -- 172.21.15.107:0/2863155510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9f0102f80 msgr2=0x7fc9f019ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.362+0000 7fc9eed9d700 1 --2- 172.21.15.107:0/2863155510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9f0102f80 0x7fc9f019ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.362+0000 7fc9eed9d700 1 -- 172.21.15.107:0/2863155510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9f01025d0 msgr2=0x7fc9f019e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.362+0000 7fc9eed9d700 1 --2- 172.21.15.107:0/2863155510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9f01025d0 0x7fc9f019e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.362+0000 7fc9eed9d700 1 -- 172.21.15.107:0/2863155510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f0199010 con 0x7fc9f0107660 2024-05-09T08:31:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.362+0000 7fc9eed9d700 1 --2- 172.21.15.107:0/2863155510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f0107660 0x7fc9f01988b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000b550 tx=0x7fc9e000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.362+0000 7fc9e77fe700 1 -- 172.21.15.107:0/2863155510 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0015020 con 0x7fc9f0107660 2024-05-09T08:31:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.362+0000 7fc9e77fe700 1 -- 172.21.15.107:0/2863155510 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc9e0011ce0 con 0x7fc9f0107660 2024-05-09T08:31:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.362+0000 7fc9e77fe700 1 -- 172.21.15.107:0/2863155510 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0010cd0 con 0x7fc9f0107660 2024-05-09T08:31:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.362+0000 7fc9f4a07700 1 -- 172.21.15.107:0/2863155510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f0199300 con 0x7fc9f0107660 2024-05-09T08:31:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.362+0000 7fc9f4a07700 1 -- 172.21.15.107:0/2863155510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f01a55c0 con 0x7fc9f0107660 2024-05-09T08:31:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.363+0000 7fc9f4a07700 1 -- 172.21.15.107:0/2863155510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc9f0062ab0 con 0x7fc9f0107660 2024-05-09T08:31:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.364+0000 7fc9e77fe700 1 -- 172.21.15.107:0/2863155510 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fc9e0018020 con 0x7fc9f0107660 2024-05-09T08:31:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.365+0000 7fc9e77fe700 1 --2- 172.21.15.107:0/2863155510 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc9dc0a2d30 0x7fc9dc0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.365+0000 7fc9ee59c700 1 --2- 172.21.15.107:0/2863155510 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc9dc0a2d30 0x7fc9dc0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.365+0000 7fc9e77fe700 1 -- 172.21.15.107:0/2863155510 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc9e00cf000 con 0x7fc9f0107660 2024-05-09T08:31:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.365+0000 7fc9e77fe700 1 --2- 172.21.15.107:0/2863155510 >> [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287] conn(0x7fc9dc0ad9f0 0x7fc9dc0afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.365+0000 7fc9e77fe700 1 -- 172.21.15.107:0/2863155510 --> [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9e0018ec0 con 0x7fc9dc0ad9f0 2024-05-09T08:31:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.365+0000 7fc9e77fe700 1 -- 172.21.15.107:0/2863155510 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fc9e00cf450 con 0x7fc9f0107660 2024-05-09T08:31:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.366+0000 7fc9ee59c700 1 --2- 172.21.15.107:0/2863155510 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc9dc0a2d30 0x7fc9dc0a51f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009550 tx=0x7fc9e80091b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.366+0000 7fc9edd9b700 1 --2- 172.21.15.107:0/2863155510 >> [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287] conn(0x7fc9dc0ad9f0 0x7fc9dc0afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.366+0000 7fc9edd9b700 1 --2- 172.21.15.107:0/2863155510 >> [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287] conn(0x7fc9dc0ad9f0 0x7fc9dc0afe30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.367+0000 7fc9e77fe700 1 -- 172.21.15.107:0/2863155510 <== osd.8 v2:172.21.15.157:6816/4115873287 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9e0018ec0 con 0x7fc9dc0ad9f0 2024-05-09T08:31:52.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.383+0000 7fc9f4a07700 1 -- 172.21.15.107:0/2863155510 --> [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc9f0062f30 con 0x7fc9dc0ad9f0 2024-05-09T08:31:52.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.383+0000 7fc9e77fe700 1 -- 172.21.15.107:0/2863155510 <== osd.8 v2:172.21.15.157:6816/4115873287 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc9f0062f30 con 0x7fc9dc0ad9f0 2024-05-09T08:31:52.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.384+0000 7fc9f4a07700 1 -- 172.21.15.107:0/2863155510 >> [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287] conn(0x7fc9dc0ad9f0 msgr2=0x7fc9dc0afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:52.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.384+0000 7fc9f4a07700 1 --2- 172.21.15.107:0/2863155510 >> [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287] conn(0x7fc9dc0ad9f0 0x7fc9dc0afe30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:52.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.384+0000 7fc9f4a07700 1 -- 172.21.15.107:0/2863155510 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc9dc0a2d30 msgr2=0x7fc9dc0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:52.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.384+0000 7fc9f4a07700 1 --2- 172.21.15.107:0/2863155510 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fc9dc0a2d30 0x7fc9dc0a51f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009550 tx=0x7fc9e80091b0 comp rx=0 tx=0).stop 2024-05-09T08:31:52.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.384+0000 7fc9f4a07700 1 -- 172.21.15.107:0/2863155510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f0107660 msgr2=0x7fc9f01988b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:52.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.384+0000 7fc9f4a07700 1 --2- 172.21.15.107:0/2863155510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9f0107660 0x7fc9f01988b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000b550 tx=0x7fc9e000b860 comp rx=0 tx=0).stop 2024-05-09T08:31:52.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.384+0000 7fc9eed9d700 1 -- 172.21.15.107:0/2863155510 reap_dead start 2024-05-09T08:31:52.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.384+0000 7fc9f4a07700 1 -- 172.21.15.107:0/2863155510 shutdown_connections 2024-05-09T08:31:52.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.384+0000 7fc9f4a07700 1 -- 172.21.15.107:0/2863155510 >> 172.21.15.107:0/2863155510 conn(0x7fc9f0079480 msgr2=0x7fc9f0100700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:52.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.384+0000 7fc9f4a07700 1 -- 172.21.15.107:0/2863155510 shutdown_connections 2024-05-09T08:31:52.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.384+0000 7fc9f4a07700 1 -- 172.21.15.107:0/2863155510 wait complete. 2024-05-09T08:31:52.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:52 smithi107 ceph-mon[30122]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:31:52.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:52 smithi157 ceph-mon[33614]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:31:52.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.855+0000 7f0487f39700 1 -- 172.21.15.107:0/3763379418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f048006b260 msgr2=0x7f048010f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:52.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.855+0000 7f0487f39700 1 --2- 172.21.15.107:0/3763379418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f048006b260 0x7f048010f1d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f047800b0a0 tx=0x7f047800b3b0 comp rx=0 tx=0).stop 2024-05-09T08:31:52.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.855+0000 7f0487f39700 1 -- 172.21.15.107:0/3763379418 shutdown_connections 2024-05-09T08:31:52.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.855+0000 7f0487f39700 1 --2- 172.21.15.107:0/3763379418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f048006b260 0x7f048010f1d0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:52.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.855+0000 7f0487f39700 1 --2- 172.21.15.107:0/3763379418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0480108390 0x7f0480108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:52.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.855+0000 7f0487f39700 1 --2- 172.21.15.107:0/3763379418 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f048006a8c0 0x7f048006ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:52.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.855+0000 7f0487f39700 1 -- 172.21.15.107:0/3763379418 >> 172.21.15.107:0/3763379418 conn(0x7f0480078190 msgr2=0x7f04800785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:52.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.855+0000 7f0487f39700 1 -- 172.21.15.107:0/3763379418 shutdown_connections 2024-05-09T08:31:52.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.855+0000 7f0487f39700 1 -- 172.21.15.107:0/3763379418 wait complete. 2024-05-09T08:31:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.856+0000 7f0487f39700 1 Processor -- start 2024-05-09T08:31:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.856+0000 7f0487f39700 1 -- start start 2024-05-09T08:31:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.856+0000 7f0487f39700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f048006a8c0 0x7f048019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.856+0000 7f0487f39700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f048006b260 0x7f048019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:52.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.856+0000 7f0487f39700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0480108390 0x7f0480198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.856+0000 7f0487f39700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0480112490 con 0x7f0480108390 2024-05-09T08:31:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.856+0000 7f0487f39700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0480112310 con 0x7f048006a8c0 2024-05-09T08:31:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.856+0000 7f0487f39700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0480112610 con 0x7f048006b260 2024-05-09T08:31:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.856+0000 7f04864d6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0480108390 0x7f0480198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.856+0000 7f04864d6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0480108390 0x7f0480198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56890/0 (socket says 172.21.15.107:56890) 2024-05-09T08:31:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.856+0000 7f04864d6700 1 -- 172.21.15.107:0/2150141410 learned_addr learned my addr 172.21.15.107:0/2150141410 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.856+0000 7f0485cd5700 1 --2- 172.21.15.107:0/2150141410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f048006a8c0 0x7f048019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.856+0000 7f04854d4700 1 --2- 172.21.15.107:0/2150141410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f048006b260 0x7f048019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.857+0000 7f04864d6700 1 -- 172.21.15.107:0/2150141410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f048006b260 msgr2=0x7f048019ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.857+0000 7f04864d6700 1 --2- 172.21.15.107:0/2150141410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f048006b260 0x7f048019ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.857+0000 7f04864d6700 1 -- 172.21.15.107:0/2150141410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f048006a8c0 msgr2=0x7f048019e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.857+0000 7f04864d6700 1 --2- 172.21.15.107:0/2150141410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f048006a8c0 0x7f048019e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:52.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.857+0000 7f04864d6700 1 -- 172.21.15.107:0/2150141410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0480198ff0 con 0x7f0480108390 2024-05-09T08:31:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.857+0000 7f04854d4700 1 --2- 172.21.15.107:0/2150141410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f048006b260 0x7f048019ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.857+0000 7f0485cd5700 1 --2- 172.21.15.107:0/2150141410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f048006a8c0 0x7f048019e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:31:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.857+0000 7f04864d6700 1 --2- 172.21.15.107:0/2150141410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0480108390 0x7f0480198810 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f047800b780 tx=0x7f0478004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.857+0000 7f0476ffd700 1 -- 172.21.15.107:0/2150141410 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478021070 con 0x7f0480108390 2024-05-09T08:31:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.857+0000 7f0476ffd700 1 -- 172.21.15.107:0/2150141410 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0478007e30 con 0x7f0480108390 2024-05-09T08:31:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.857+0000 7f0476ffd700 1 -- 172.21.15.107:0/2150141410 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04780047b0 con 0x7f0480108390 2024-05-09T08:31:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.859+0000 7f0487f39700 1 -- 172.21.15.107:0/2150141410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0480199280 con 0x7f0480108390 2024-05-09T08:31:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.859+0000 7f0487f39700 1 -- 172.21.15.107:0/2150141410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04801a54f0 con 0x7f0480108390 2024-05-09T08:31:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.860+0000 7f0487f39700 1 -- 172.21.15.107:0/2150141410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0480062a70 con 0x7f0480108390 2024-05-09T08:31:52.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.861+0000 7f0476ffd700 1 -- 172.21.15.107:0/2150141410 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f0478007980 con 0x7f0480108390 2024-05-09T08:31:52.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.862+0000 7f0476ffd700 1 --2- 172.21.15.107:0/2150141410 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f046c0a2d30 0x7f046c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:52.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.862+0000 7f0485cd5700 1 --2- 172.21.15.107:0/2150141410 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f046c0a2d30 0x7f046c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:52.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.862+0000 7f0476ffd700 1 -- 172.21.15.107:0/2150141410 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f047800f070 con 0x7f0480108390 2024-05-09T08:31:52.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.862+0000 7f0476ffd700 1 --2- 172.21.15.107:0/2150141410 >> [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349] conn(0x7f046c0ad9f0 0x7f046c0afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:52.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.862+0000 7f0476ffd700 1 -- 172.21.15.107:0/2150141410 --> [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f04780020b0 con 0x7f046c0ad9f0 2024-05-09T08:31:52.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.862+0000 7f0476ffd700 1 -- 172.21.15.107:0/2150141410 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f04780d0a60 con 0x7f0480108390 2024-05-09T08:31:52.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.862+0000 7f04854d4700 1 --2- 172.21.15.107:0/2150141410 >> [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349] conn(0x7f046c0ad9f0 0x7f046c0afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:52.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.863+0000 7f0485cd5700 1 --2- 172.21.15.107:0/2150141410 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f046c0a2d30 0x7f046c0a51f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f047c009fd0 tx=0x7f047c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:52.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.863+0000 7f04854d4700 1 --2- 172.21.15.107:0/2150141410 >> [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349] conn(0x7f046c0ad9f0 0x7f046c0afe30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:52.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.864+0000 7f0476ffd700 1 -- 172.21.15.107:0/2150141410 <== osd.0 v2:172.21.15.107:6802/1215261349 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04780020b0 con 0x7f046c0ad9f0 2024-05-09T08:31:52.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:52 smithi110 ceph-mon[33622]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:31:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.881+0000 7f0487f39700 1 -- 172.21.15.107:0/2150141410 --> [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0480062ef0 con 0x7f046c0ad9f0 2024-05-09T08:31:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.881+0000 7f0476ffd700 1 -- 172.21.15.107:0/2150141410 <== osd.0 v2:172.21.15.107:6802/1215261349 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0480062ef0 con 0x7f046c0ad9f0 2024-05-09T08:31:52.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.881+0000 7f0487f39700 1 -- 172.21.15.107:0/2150141410 >> [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349] conn(0x7f046c0ad9f0 msgr2=0x7f046c0afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:52.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.881+0000 7f0487f39700 1 --2- 172.21.15.107:0/2150141410 >> [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349] conn(0x7f046c0ad9f0 0x7f046c0afe30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:52.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.881+0000 7f0487f39700 1 -- 172.21.15.107:0/2150141410 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f046c0a2d30 msgr2=0x7f046c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.881+0000 7f0487f39700 1 --2- 172.21.15.107:0/2150141410 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f046c0a2d30 0x7f046c0a51f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f047c009fd0 tx=0x7f047c009260 comp rx=0 tx=0).stop 2024-05-09T08:31:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.882+0000 7f0487f39700 1 -- 172.21.15.107:0/2150141410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0480108390 msgr2=0x7f0480198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.882+0000 7f0487f39700 1 --2- 172.21.15.107:0/2150141410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0480108390 0x7f0480198810 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f047800b780 tx=0x7f0478004d00 comp rx=0 tx=0).stop 2024-05-09T08:31:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.882+0000 7f04864d6700 1 -- 172.21.15.107:0/2150141410 reap_dead start 2024-05-09T08:31:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.882+0000 7f0487f39700 1 -- 172.21.15.107:0/2150141410 shutdown_connections 2024-05-09T08:31:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.882+0000 7f0487f39700 1 -- 172.21.15.107:0/2150141410 >> 172.21.15.107:0/2150141410 conn(0x7f0480078190 msgr2=0x7f04800ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.882+0000 7f0487f39700 1 -- 172.21.15.107:0/2150141410 shutdown_connections 2024-05-09T08:31:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:52.882+0000 7f0487f39700 1 -- 172.21.15.107:0/2150141410 wait complete. 2024-05-09T08:31:54.487 INFO:teuthology.orchestra.run.smithi107.stdout:77309411397 2024-05-09T08:31:54.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-09T08:31:54.709 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:54 smithi107 ceph-mon[30122]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T08:31:54.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:54 smithi157 ceph-mon[33614]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T08:31:54.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:54 smithi110 ceph-mon[33622]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T08:31:55.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.259+0000 7f065f2b7700 1 -- 172.21.15.107:0/3432622168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06581072d0 msgr2=0x7f06581076b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:55.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.259+0000 7f065f2b7700 1 --2- 172.21.15.107:0/3432622168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06581072d0 0x7f06581076b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0648009990 tx=0x7f0648009ca0 comp rx=0 tx=0).stop 2024-05-09T08:31:55.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.260+0000 7f065f2b7700 1 -- 172.21.15.107:0/3432622168 shutdown_connections 2024-05-09T08:31:55.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.260+0000 7f065f2b7700 1 --2- 172.21.15.107:0/3432622168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0658101c10 0x7f065810e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:55.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.260+0000 7f065f2b7700 1 --2- 172.21.15.107:0/3432622168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06581072d0 0x7f06581076b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:55.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.260+0000 7f065f2b7700 1 --2- 172.21.15.107:0/3432622168 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0658101250 0x7f06581016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:55.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.260+0000 7f065f2b7700 1 -- 172.21.15.107:0/3432622168 >> 172.21.15.107:0/3432622168 conn(0x7f0658077290 msgr2=0x7f06580776a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:55.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.260+0000 7f065f2b7700 1 -- 172.21.15.107:0/3432622168 shutdown_connections 2024-05-09T08:31:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.260+0000 7f065f2b7700 1 -- 172.21.15.107:0/3432622168 wait complete. 2024-05-09T08:31:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.260+0000 7f065f2b7700 1 Processor -- start 2024-05-09T08:31:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.260+0000 7f065f2b7700 1 -- start start 2024-05-09T08:31:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.260+0000 7f065f2b7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0658101250 0x7f06581a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.260+0000 7f065f2b7700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0658101c10 0x7f06581a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.260+0000 7f065f2b7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06581072d0 0x7f065819ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.260+0000 7f065f2b7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f06581102f0 con 0x7f0658101250 2024-05-09T08:31:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.260+0000 7f065f2b7700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0658110170 con 0x7f06581072d0 2024-05-09T08:31:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.260+0000 7f065f2b7700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0658110470 con 0x7f0658101c10 2024-05-09T08:31:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f065d053700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0658101250 0x7f06581a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f065d053700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0658101250 0x7f06581a0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56920/0 (socket says 172.21.15.107:56920) 2024-05-09T08:31:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f065d053700 1 -- 172.21.15.107:0/1437526112 learned_addr learned my addr 172.21.15.107:0/1437526112 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f065c852700 1 --2- 172.21.15.107:0/1437526112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0658101c10 0x7f06581a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f065d854700 1 --2- 172.21.15.107:0/1437526112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06581072d0 0x7f065819ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f065d053700 1 -- 172.21.15.107:0/1437526112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0658101c10 msgr2=0x7f06581a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f065d053700 1 --2- 172.21.15.107:0/1437526112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0658101c10 0x7f06581a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f065d053700 1 -- 172.21.15.107:0/1437526112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06581072d0 msgr2=0x7f065819ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f065d053700 1 --2- 172.21.15.107:0/1437526112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06581072d0 0x7f065819ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f065d053700 1 -- 172.21.15.107:0/1437526112 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f065819b260 con 0x7f0658101250 2024-05-09T08:31:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f065d854700 1 --2- 172.21.15.107:0/1437526112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06581072d0 0x7f065819ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f065d053700 1 --2- 172.21.15.107:0/1437526112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0658101250 0x7f06581a0940 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f06480057f0 tx=0x7f0648005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f064e7fc700 1 -- 172.21.15.107:0/1437526112 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0648027070 con 0x7f0658101250 2024-05-09T08:31:55.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f065f2b7700 1 -- 172.21.15.107:0/1437526112 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f065819b4f0 con 0x7f0658101250 2024-05-09T08:31:55.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f064e7fc700 1 -- 172.21.15.107:0/1437526112 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0648003f80 con 0x7f0658101250 2024-05-09T08:31:55.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f064e7fc700 1 -- 172.21.15.107:0/1437526112 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0648022b10 con 0x7f0658101250 2024-05-09T08:31:55.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.261+0000 7f065f2b7700 1 -- 172.21.15.107:0/1437526112 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06581a7810 con 0x7f0658101250 2024-05-09T08:31:55.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.262+0000 7f065f2b7700 1 -- 172.21.15.107:0/1437526112 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0658062a30 con 0x7f0658101250 2024-05-09T08:31:55.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.263+0000 7f064e7fc700 1 -- 172.21.15.107:0/1437526112 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f064802a070 con 0x7f0658101250 2024-05-09T08:31:55.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.264+0000 7f064e7fc700 1 --2- 172.21.15.107:0/1437526112 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f06440a2c90 0x7f06440a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:55.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.264+0000 7f065c852700 1 --2- 172.21.15.107:0/1437526112 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f06440a2c90 0x7f06440a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:55.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.264+0000 7f064e7fc700 1 -- 172.21.15.107:0/1437526112 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f06480d0c00 con 0x7f0658101250 2024-05-09T08:31:55.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.264+0000 7f064e7fc700 1 --2- 172.21.15.107:0/1437526112 >> [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754] conn(0x7f06440ad950 0x7f06440afd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:55.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.264+0000 7f064e7fc700 1 -- 172.21.15.107:0/1437526112 --> [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0648010ec0 con 0x7f06440ad950 2024-05-09T08:31:55.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.264+0000 7f064e7fc700 1 -- 172.21.15.107:0/1437526112 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f06480d1050 con 0x7f0658101250 2024-05-09T08:31:55.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.265+0000 7f065d854700 1 --2- 172.21.15.107:0/1437526112 >> [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754] conn(0x7f06440ad950 0x7f06440afd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:55.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.265+0000 7f065c852700 1 --2- 172.21.15.107:0/1437526112 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f06440a2c90 0x7f06440a5150 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f06580779c0 tx=0x7f065400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:55.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.265+0000 7f065d854700 1 --2- 172.21.15.107:0/1437526112 >> [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754] conn(0x7f06440ad950 0x7f06440afd90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:55.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.266+0000 7f064e7fc700 1 -- 172.21.15.107:0/1437526112 <== osd.3 v2:172.21.15.107:6810/1056009754 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0648010ec0 con 0x7f06440ad950 2024-05-09T08:31:55.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.282+0000 7f065f2b7700 1 -- 172.21.15.107:0/1437526112 --> [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0658062e30 con 0x7f06440ad950 2024-05-09T08:31:55.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.282+0000 7f064e7fc700 1 -- 172.21.15.107:0/1437526112 <== osd.3 v2:172.21.15.107:6810/1056009754 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0658062e30 con 0x7f06440ad950 2024-05-09T08:31:55.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.282+0000 7f065f2b7700 1 -- 172.21.15.107:0/1437526112 >> [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754] conn(0x7f06440ad950 msgr2=0x7f06440afd90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:55.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.282+0000 7f065f2b7700 1 --2- 172.21.15.107:0/1437526112 >> [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754] conn(0x7f06440ad950 0x7f06440afd90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:55.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.282+0000 7f065f2b7700 1 -- 172.21.15.107:0/1437526112 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f06440a2c90 msgr2=0x7f06440a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:55.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.282+0000 7f065f2b7700 1 --2- 172.21.15.107:0/1437526112 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f06440a2c90 0x7f06440a5150 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f06580779c0 tx=0x7f065400b410 comp rx=0 tx=0).stop 2024-05-09T08:31:55.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.282+0000 7f065f2b7700 1 -- 172.21.15.107:0/1437526112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0658101250 msgr2=0x7f06581a0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:55.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.282+0000 7f065f2b7700 1 --2- 172.21.15.107:0/1437526112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0658101250 0x7f06581a0940 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f06480057f0 tx=0x7f0648005e80 comp rx=0 tx=0).stop 2024-05-09T08:31:55.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.282+0000 7f065d854700 1 -- 172.21.15.107:0/1437526112 reap_dead start 2024-05-09T08:31:55.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.282+0000 7f065f2b7700 1 -- 172.21.15.107:0/1437526112 shutdown_connections 2024-05-09T08:31:55.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.282+0000 7f065f2b7700 1 -- 172.21.15.107:0/1437526112 >> 172.21.15.107:0/1437526112 conn(0x7f0658077290 msgr2=0x7f06581000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:55.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.282+0000 7f065f2b7700 1 -- 172.21.15.107:0/1437526112 shutdown_connections 2024-05-09T08:31:55.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:55.283+0000 7f065f2b7700 1 -- 172.21.15.107:0/1437526112 wait complete. 2024-05-09T08:31:55.687 INFO:teuthology.orchestra.run.smithi107.stdout:34359738447 2024-05-09T08:31:55.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-09T08:31:55.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-05-09T08:31:55.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-05-09T08:31:55.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-05-09T08:31:55.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-05-09T08:31:55.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-05-09T08:31:55.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-05-09T08:31:55.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]: dispatch 2024-05-09T08:31:55.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-05-09T08:31:55.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-05-09T08:31:55.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:55 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-09T08:31:55.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-05-09T08:31:55.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-05-09T08:31:55.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-05-09T08:31:55.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-05-09T08:31:55.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-05-09T08:31:55.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-05-09T08:31:55.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]: dispatch 2024-05-09T08:31:55.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-05-09T08:31:55.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-05-09T08:31:55.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:55 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-09T08:31:55.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-05-09T08:31:55.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-05-09T08:31:55.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-05-09T08:31:55.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-05-09T08:31:55.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-05-09T08:31:55.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-05-09T08:31:55.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]: dispatch 2024-05-09T08:31:55.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-05-09T08:31:55.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-05-09T08:31:55.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:55 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-09T08:31:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.319+0000 7fba11884700 1 -- 172.21.15.107:0/2835817341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba0c06aa70 msgr2=0x7fba0c06aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.319+0000 7fba11884700 1 --2- 172.21.15.107:0/2835817341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba0c06aa70 0x7fba0c06aed0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fba00009990 tx=0x7fba00009ca0 comp rx=0 tx=0).stop 2024-05-09T08:31:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.319+0000 7fba11884700 1 -- 172.21.15.107:0/2835817341 shutdown_connections 2024-05-09T08:31:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.319+0000 7fba11884700 1 --2- 172.21.15.107:0/2835817341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba0c06b410 0x7fba0c10f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.319+0000 7fba11884700 1 --2- 172.21.15.107:0/2835817341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0c1083a0 0x7fba0c108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.319+0000 7fba11884700 1 --2- 172.21.15.107:0/2835817341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba0c06aa70 0x7fba0c06aed0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.319+0000 7fba11884700 1 -- 172.21.15.107:0/2835817341 >> 172.21.15.107:0/2835817341 conn(0x7fba0c0781f0 msgr2=0x7fba0c078600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.319+0000 7fba11884700 1 -- 172.21.15.107:0/2835817341 shutdown_connections 2024-05-09T08:31:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.319+0000 7fba11884700 1 -- 172.21.15.107:0/2835817341 wait complete. 2024-05-09T08:31:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.319+0000 7fba11884700 1 Processor -- start 2024-05-09T08:31:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.319+0000 7fba11884700 1 -- start start 2024-05-09T08:31:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba11884700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0c06aa70 0x7fba0c19e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba11884700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba0c06b410 0x7fba0c19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba11884700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba0c1083a0 0x7fba0c198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba11884700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c1124d0 con 0x7fba0c1083a0 2024-05-09T08:31:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba11884700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c112350 con 0x7fba0c06aa70 2024-05-09T08:31:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba11884700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c112650 con 0x7fba0c06b410 2024-05-09T08:31:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba0b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba0c1083a0 0x7fba0c198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba0b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba0c1083a0 0x7fba0c198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56940/0 (socket says 172.21.15.107:56940) 2024-05-09T08:31:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba0b7fe700 1 -- 172.21.15.107:0/2359945972 learned_addr learned my addr 172.21.15.107:0/2359945972 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba0a7fc700 1 --2- 172.21.15.107:0/2359945972 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba0c06b410 0x7fba0c19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba0affd700 1 --2- 172.21.15.107:0/2359945972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0c06aa70 0x7fba0c19e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba0b7fe700 1 -- 172.21.15.107:0/2359945972 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba0c06b410 msgr2=0x7fba0c19ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba0b7fe700 1 --2- 172.21.15.107:0/2359945972 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba0c06b410 0x7fba0c19ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba0b7fe700 1 -- 172.21.15.107:0/2359945972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0c06aa70 msgr2=0x7fba0c19e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba0b7fe700 1 --2- 172.21.15.107:0/2359945972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0c06aa70 0x7fba0c19e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba0b7fe700 1 -- 172.21.15.107:0/2359945972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0c199010 con 0x7fba0c1083a0 2024-05-09T08:31:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba0b7fe700 1 --2- 172.21.15.107:0/2359945972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba0c1083a0 0x7fba0c198830 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fba04007ee0 tx=0x7fba0400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba10882700 1 -- 172.21.15.107:0/2359945972 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0400dbd0 con 0x7fba0c1083a0 2024-05-09T08:31:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba11884700 1 -- 172.21.15.107:0/2359945972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba0c199300 con 0x7fba0c1083a0 2024-05-09T08:31:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba10882700 1 -- 172.21.15.107:0/2359945972 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba0400dd70 con 0x7fba0c1083a0 2024-05-09T08:31:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba10882700 1 -- 172.21.15.107:0/2359945972 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba04005990 con 0x7fba0c1083a0 2024-05-09T08:31:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.320+0000 7fba11884700 1 -- 172.21.15.107:0/2359945972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba0c1a5510 con 0x7fba0c1083a0 2024-05-09T08:31:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.321+0000 7fba11884700 1 -- 172.21.15.107:0/2359945972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fba0c062ab0 con 0x7fba0c1083a0 2024-05-09T08:31:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.323+0000 7fba10882700 1 -- 172.21.15.107:0/2359945972 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fba0401a020 con 0x7fba0c1083a0 2024-05-09T08:31:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.323+0000 7fba10882700 1 --2- 172.21.15.107:0/2359945972 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb9f40a2cb0 0x7fb9f40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.323+0000 7fba0affd700 1 --2- 172.21.15.107:0/2359945972 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb9f40a2cb0 0x7fb9f40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.323+0000 7fba10882700 1 -- 172.21.15.107:0/2359945972 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fba040cf6b0 con 0x7fba0c1083a0 2024-05-09T08:31:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.324+0000 7fba10882700 1 --2- 172.21.15.107:0/2359945972 >> [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764] conn(0x7fb9f40ade10 0x7fb9f40b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.324+0000 7fba10882700 1 -- 172.21.15.107:0/2359945972 --> [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fba04021e70 con 0x7fb9f40ade10 2024-05-09T08:31:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.324+0000 7fba10882700 1 -- 172.21.15.107:0/2359945972 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fba040cfb00 con 0x7fba0c1083a0 2024-05-09T08:31:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.324+0000 7fba0affd700 1 --2- 172.21.15.107:0/2359945972 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb9f40a2cb0 0x7fb9f40a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc005950 tx=0x7fb9fc0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.324+0000 7fba0a7fc700 1 --2- 172.21.15.107:0/2359945972 >> [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764] conn(0x7fb9f40ade10 0x7fb9f40b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.324+0000 7fba0a7fc700 1 --2- 172.21.15.107:0/2359945972 >> [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764] conn(0x7fb9f40ade10 0x7fb9f40b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:56.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.325+0000 7fba10882700 1 -- 172.21.15.107:0/2359945972 <== osd.10 v2:172.21.15.110:6824/1566970764 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba04021e70 con 0x7fb9f40ade10 2024-05-09T08:31:56.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.346+0000 7fba11884700 1 -- 172.21.15.107:0/2359945972 --> [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fba0c062ee0 con 0x7fb9f40ade10 2024-05-09T08:31:56.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.347+0000 7fba10882700 1 -- 172.21.15.107:0/2359945972 <== osd.10 v2:172.21.15.110:6824/1566970764 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fba0c062ee0 con 0x7fb9f40ade10 2024-05-09T08:31:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.347+0000 7fba11884700 1 -- 172.21.15.107:0/2359945972 >> [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764] conn(0x7fb9f40ade10 msgr2=0x7fb9f40b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.347+0000 7fba11884700 1 --2- 172.21.15.107:0/2359945972 >> [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764] conn(0x7fb9f40ade10 0x7fb9f40b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.347+0000 7fba11884700 1 -- 172.21.15.107:0/2359945972 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb9f40a2cb0 msgr2=0x7fb9f40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.347+0000 7fba11884700 1 --2- 172.21.15.107:0/2359945972 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb9f40a2cb0 0x7fb9f40a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc005950 tx=0x7fb9fc0058e0 comp rx=0 tx=0).stop 2024-05-09T08:31:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.347+0000 7fba11884700 1 -- 172.21.15.107:0/2359945972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba0c1083a0 msgr2=0x7fba0c198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.347+0000 7fba11884700 1 --2- 172.21.15.107:0/2359945972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba0c1083a0 0x7fba0c198830 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fba04007ee0 tx=0x7fba0400d3b0 comp rx=0 tx=0).stop 2024-05-09T08:31:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.347+0000 7fba0b7fe700 1 -- 172.21.15.107:0/2359945972 reap_dead start 2024-05-09T08:31:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.348+0000 7fba11884700 1 -- 172.21.15.107:0/2359945972 shutdown_connections 2024-05-09T08:31:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.348+0000 7fba11884700 1 -- 172.21.15.107:0/2359945972 >> 172.21.15.107:0/2359945972 conn(0x7fba0c0781f0 msgr2=0x7fba0c100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.348+0000 7fba11884700 1 -- 172.21.15.107:0/2359945972 shutdown_connections 2024-05-09T08:31:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:56.348+0000 7fba11884700 1 -- 172.21.15.107:0/2359945972 wait complete. 2024-05-09T08:31:56.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:56 smithi157 ceph-mon[33614]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-09T08:31:56.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-05-09T08:31:56.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-05-09T08:31:56.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-05-09T08:31:56.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-05-09T08:31:56.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-05-09T08:31:56.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-05-09T08:31:56.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-05-09T08:31:56.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-05-09T08:31:56.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-05-09T08:31:56.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-09T08:31:56.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:56 smithi157 ceph-mon[33614]: osdmap e90: 12 total, 12 up, 12 in 2024-05-09T08:31:56.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:56 smithi110 ceph-mon[33622]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-09T08:31:56.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-05-09T08:31:56.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-05-09T08:31:56.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-05-09T08:31:56.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-05-09T08:31:56.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-05-09T08:31:56.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-05-09T08:31:56.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-05-09T08:31:56.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-05-09T08:31:56.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-05-09T08:31:56.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-09T08:31:56.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:56 smithi110 ceph-mon[33622]: osdmap e90: 12 total, 12 up, 12 in 2024-05-09T08:31:56.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:56 smithi107 ceph-mon[30122]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-09T08:31:56.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-05-09T08:31:56.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-05-09T08:31:56.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-05-09T08:31:56.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-05-09T08:31:56.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-05-09T08:31:56.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-05-09T08:31:56.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-05-09T08:31:56.975 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-05-09T08:31:56.976 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-05-09T08:31:56.976 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-09T08:31:56.976 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:56 smithi107 ceph-mon[30122]: osdmap e90: 12 total, 12 up, 12 in 2024-05-09T08:31:57.647 INFO:teuthology.orchestra.run.smithi107.stdout:197568495664 2024-05-09T08:31:57.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-09T08:31:57.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:57 smithi157 ceph-mon[33614]: osdmap e91: 12 total, 12 up, 12 in 2024-05-09T08:31:57.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:57 smithi157 ceph-mon[33614]: pgmap v322: 129 pgs: 1 activating, 5 peering, 123 active+clean; 643 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s; 14 B/s, 0 objects/s recovering 2024-05-09T08:31:57.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:57 smithi110 ceph-mon[33622]: osdmap e91: 12 total, 12 up, 12 in 2024-05-09T08:31:57.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:57 smithi110 ceph-mon[33622]: pgmap v322: 129 pgs: 1 activating, 5 peering, 123 active+clean; 643 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s; 14 B/s, 0 objects/s recovering 2024-05-09T08:31:57.901 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:57 smithi107 ceph-mon[30122]: osdmap e91: 12 total, 12 up, 12 in 2024-05-09T08:31:57.901 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:57 smithi107 ceph-mon[30122]: pgmap v322: 129 pgs: 1 activating, 5 peering, 123 active+clean; 643 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s; 14 B/s, 0 objects/s recovering 2024-05-09T08:31:58.294 INFO:teuthology.orchestra.run.smithi107.stdout:253403070503 2024-05-09T08:31:58.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-09T08:31:58.808 INFO:teuthology.orchestra.run.smithi107.stdout:107374182464 2024-05-09T08:31:58.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-09T08:31:59.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:31:59 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:31:59.356 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:31:59 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:31:59.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:31:59 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:31:59.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.926+0000 7fe8c1dd2700 1 -- 172.21.15.107:0/1943922098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8bc101220 msgr2=0x7fe8bc1016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.926+0000 7fe8c1dd2700 1 --2- 172.21.15.107:0/1943922098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8bc101220 0x7fe8bc1016a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0009990 tx=0x7fe8b0009ca0 comp rx=0 tx=0).stop 2024-05-09T08:31:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.926+0000 7fe8c1dd2700 1 -- 172.21.15.107:0/1943922098 shutdown_connections 2024-05-09T08:31:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.926+0000 7fe8c1dd2700 1 --2- 172.21.15.107:0/1943922098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8bc101be0 0x7fe8bc10e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.926+0000 7fe8c1dd2700 1 --2- 172.21.15.107:0/1943922098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8bc107290 0x7fe8bc107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.926+0000 7fe8c1dd2700 1 --2- 172.21.15.107:0/1943922098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8bc101220 0x7fe8bc1016a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.926+0000 7fe8c1dd2700 1 -- 172.21.15.107:0/1943922098 >> 172.21.15.107:0/1943922098 conn(0x7fe8bc0781d0 msgr2=0x7fe8bc0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.926+0000 7fe8c1dd2700 1 -- 172.21.15.107:0/1943922098 shutdown_connections 2024-05-09T08:31:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.926+0000 7fe8c1dd2700 1 -- 172.21.15.107:0/1943922098 wait complete. 2024-05-09T08:31:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.927+0000 7fe8c1dd2700 1 Processor -- start 2024-05-09T08:31:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.927+0000 7fe8c1dd2700 1 -- start start 2024-05-09T08:31:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.927+0000 7fe8c1dd2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8bc101220 0x7fe8bc19d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.927+0000 7fe8c1dd2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8bc101be0 0x7fe8bc19db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.927+0000 7fe8c1dd2700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8bc107290 0x7fe8bc1977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.927+0000 7fe8c1dd2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc1113b0 con 0x7fe8bc101220 2024-05-09T08:31:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.927+0000 7fe8c1dd2700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc111230 con 0x7fe8bc101be0 2024-05-09T08:31:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.927+0000 7fe8c1dd2700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc111530 con 0x7fe8bc107290 2024-05-09T08:31:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.927+0000 7fe8bb7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8bc101220 0x7fe8bc19d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.927+0000 7fe8bb7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8bc101220 0x7fe8bc19d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56898/0 (socket says 172.21.15.107:56898) 2024-05-09T08:31:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.927+0000 7fe8bb7fe700 1 -- 172.21.15.107:0/1390931652 learned_addr learned my addr 172.21.15.107:0/1390931652 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:31:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.927+0000 7fe8baffd700 1 --2- 172.21.15.107:0/1390931652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8bc101be0 0x7fe8bc19db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.927+0000 7fe8bbfff700 1 --2- 172.21.15.107:0/1390931652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8bc107290 0x7fe8bc1977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.928+0000 7fe8bb7fe700 1 -- 172.21.15.107:0/1390931652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8bc107290 msgr2=0x7fe8bc1977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.928+0000 7fe8bb7fe700 1 --2- 172.21.15.107:0/1390931652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8bc107290 0x7fe8bc1977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.928+0000 7fe8bb7fe700 1 -- 172.21.15.107:0/1390931652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8bc101be0 msgr2=0x7fe8bc19db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.928+0000 7fe8bb7fe700 1 --2- 172.21.15.107:0/1390931652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8bc101be0 0x7fe8bc19db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.928+0000 7fe8bb7fe700 1 -- 172.21.15.107:0/1390931652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8bc197f90 con 0x7fe8bc101220 2024-05-09T08:31:59.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.928+0000 7fe8baffd700 1 --2- 172.21.15.107:0/1390931652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8bc101be0 0x7fe8bc19db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:31:59.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.928+0000 7fe8bb7fe700 1 --2- 172.21.15.107:0/1390931652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8bc101220 0x7fe8bc19d640 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00e9a0 tx=0x7fe8ac00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:59.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.928+0000 7fe8b8ff9700 1 -- 172.21.15.107:0/1390931652 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac00cbd0 con 0x7fe8bc101220 2024-05-09T08:31:59.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.928+0000 7fe8c1dd2700 1 -- 172.21.15.107:0/1390931652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8bc198280 con 0x7fe8bc101220 2024-05-09T08:31:59.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.928+0000 7fe8b8ff9700 1 -- 172.21.15.107:0/1390931652 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8ac00cd70 con 0x7fe8bc101220 2024-05-09T08:31:59.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.928+0000 7fe8b8ff9700 1 -- 172.21.15.107:0/1390931652 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac018750 con 0x7fe8bc101220 2024-05-09T08:31:59.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.928+0000 7fe8c1dd2700 1 -- 172.21.15.107:0/1390931652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8bc1a4630 con 0x7fe8bc101220 2024-05-09T08:31:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.929+0000 7fe8c1dd2700 1 -- 172.21.15.107:0/1390931652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe8bc06a2d0 con 0x7fe8bc101220 2024-05-09T08:31:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.930+0000 7fe8b8ff9700 1 -- 172.21.15.107:0/1390931652 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fe8ac010430 con 0x7fe8bc101220 2024-05-09T08:31:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.931+0000 7fe8b8ff9700 1 --2- 172.21.15.107:0/1390931652 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe8a40a2cb0 0x7fe8a40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.931+0000 7fe8baffd700 1 --2- 172.21.15.107:0/1390931652 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe8a40a2cb0 0x7fe8a40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.931+0000 7fe8b8ff9700 1 -- 172.21.15.107:0/1390931652 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe8ac0cfb20 con 0x7fe8bc101220 2024-05-09T08:31:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.931+0000 7fe8b8ff9700 1 --2- 172.21.15.107:0/1390931652 >> [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916] conn(0x7fe8a40ad970 0x7fe8a40afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:31:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.931+0000 7fe8b8ff9700 1 -- 172.21.15.107:0/1390931652 --> [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe8a40b0480 con 0x7fe8a40ad970 2024-05-09T08:31:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.931+0000 7fe8b8ff9700 1 -- 172.21.15.107:0/1390931652 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fe8ac0cff70 con 0x7fe8bc101220 2024-05-09T08:31:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.931+0000 7fe8bbfff700 1 --2- 172.21.15.107:0/1390931652 >> [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916] conn(0x7fe8a40ad970 0x7fe8a40afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:31:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.932+0000 7fe8baffd700 1 --2- 172.21.15.107:0/1390931652 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe8a40a2cb0 0x7fe8a40a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0006010 tx=0x7fe8b001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.932+0000 7fe8bbfff700 1 --2- 172.21.15.107:0/1390931652 >> [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916] conn(0x7fe8a40ad970 0x7fe8a40afdb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:31:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.932+0000 7fe8b8ff9700 1 -- 172.21.15.107:0/1390931652 <== osd.9 v2:172.21.15.107:6826/2273353916 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8a40b0480 con 0x7fe8a40ad970 2024-05-09T08:31:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.948+0000 7fe8c1dd2700 1 -- 172.21.15.107:0/1390931652 --> [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe8bc06a6d0 con 0x7fe8a40ad970 2024-05-09T08:31:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.949+0000 7fe8b8ff9700 1 -- 172.21.15.107:0/1390931652 <== osd.9 v2:172.21.15.107:6826/2273353916 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe8bc06a6d0 con 0x7fe8a40ad970 2024-05-09T08:31:59.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.949+0000 7fe8c1dd2700 1 -- 172.21.15.107:0/1390931652 >> [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916] conn(0x7fe8a40ad970 msgr2=0x7fe8a40afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:59.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.949+0000 7fe8c1dd2700 1 --2- 172.21.15.107:0/1390931652 >> [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916] conn(0x7fe8a40ad970 0x7fe8a40afdb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:31:59.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.949+0000 7fe8c1dd2700 1 -- 172.21.15.107:0/1390931652 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe8a40a2cb0 msgr2=0x7fe8a40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:59.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.949+0000 7fe8c1dd2700 1 --2- 172.21.15.107:0/1390931652 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe8a40a2cb0 0x7fe8a40a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0006010 tx=0x7fe8b001a040 comp rx=0 tx=0).stop 2024-05-09T08:31:59.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.949+0000 7fe8c1dd2700 1 -- 172.21.15.107:0/1390931652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8bc101220 msgr2=0x7fe8bc19d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:31:59.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.949+0000 7fe8c1dd2700 1 --2- 172.21.15.107:0/1390931652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8bc101220 0x7fe8bc19d640 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00e9a0 tx=0x7fe8ac00ed60 comp rx=0 tx=0).stop 2024-05-09T08:31:59.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.949+0000 7fe8bbfff700 1 -- 172.21.15.107:0/1390931652 reap_dead start 2024-05-09T08:31:59.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.949+0000 7fe8c1dd2700 1 -- 172.21.15.107:0/1390931652 shutdown_connections 2024-05-09T08:31:59.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.949+0000 7fe8c1dd2700 1 -- 172.21.15.107:0/1390931652 >> 172.21.15.107:0/1390931652 conn(0x7fe8bc0781d0 msgr2=0x7fe8bc0ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:31:59.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.949+0000 7fe8c1dd2700 1 -- 172.21.15.107:0/1390931652 shutdown_connections 2024-05-09T08:31:59.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:31:54.949+0000 7fe8c1dd2700 1 -- 172.21.15.107:0/1390931652 wait complete. 2024-05-09T08:32:00.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:00 smithi110 ceph-mon[33622]: pgmap v323: 129 pgs: 1 activating, 5 peering, 123 active+clean; 643 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s; 87 B/s, 0 objects/s recovering 2024-05-09T08:32:00.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:00 smithi107 ceph-mon[30122]: pgmap v323: 129 pgs: 1 activating, 5 peering, 123 active+clean; 643 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s; 87 B/s, 0 objects/s recovering 2024-05-09T08:32:00.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:00 smithi157 ceph-mon[33614]: pgmap v323: 129 pgs: 1 activating, 5 peering, 123 active+clean; 643 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s; 87 B/s, 0 objects/s recovering 2024-05-09T08:32:00.814 INFO:teuthology.orchestra.run.smithi107.stdout:309237645341 2024-05-09T08:32:00.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-09T08:32:01.610 INFO:teuthology.orchestra.run.smithi107.stdout:287762808866 2024-05-09T08:32:01.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-09T08:32:01.958 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:01 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:01.959 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:01 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:01.959 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:01 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:01.959 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:01 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:01.959 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:01 smithi107 ceph-mon[30122]: pgmap v324: 129 pgs: 1 activating, 1 peering, 127 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s; 87 B/s, 0 objects/s recovering 2024-05-09T08:32:02.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:01 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:02.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:01 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:02.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:01 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:02.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:01 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:02.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:01 smithi157 ceph-mon[33614]: pgmap v324: 129 pgs: 1 activating, 1 peering, 127 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s; 87 B/s, 0 objects/s recovering 2024-05-09T08:32:02.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:01 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:02.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:01 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:02.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:01 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:02.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:01 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:02.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:01 smithi110 ceph-mon[33622]: pgmap v324: 129 pgs: 1 activating, 1 peering, 127 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s; 87 B/s, 0 objects/s recovering 2024-05-09T08:32:02.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.126+0000 7f7fb1392700 1 -- 172.21.15.107:0/2005202078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fac06b260 msgr2=0x7f7fac10f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:02.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.126+0000 7f7fb1392700 1 --2- 172.21.15.107:0/2005202078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fac06b260 0x7f7fac10f1d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400b0a0 tx=0x7f7fa400b3b0 comp rx=0 tx=0).stop 2024-05-09T08:32:02.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.126+0000 7f7fb1392700 1 -- 172.21.15.107:0/2005202078 shutdown_connections 2024-05-09T08:32:02.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.126+0000 7f7fb1392700 1 --2- 172.21.15.107:0/2005202078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fac06b260 0x7f7fac10f1d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:02.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.126+0000 7f7fb1392700 1 --2- 172.21.15.107:0/2005202078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7fac108390 0x7f7fac108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:02.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.126+0000 7f7fb1392700 1 --2- 172.21.15.107:0/2005202078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7fac06a8c0 0x7f7fac06ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:02.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.126+0000 7f7fb1392700 1 -- 172.21.15.107:0/2005202078 >> 172.21.15.107:0/2005202078 conn(0x7f7fac078190 msgr2=0x7f7fac0785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:02.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.126+0000 7f7fb1392700 1 -- 172.21.15.107:0/2005202078 shutdown_connections 2024-05-09T08:32:02.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.126+0000 7f7fb1392700 1 -- 172.21.15.107:0/2005202078 wait complete. 2024-05-09T08:32:02.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.127+0000 7f7fb1392700 1 Processor -- start 2024-05-09T08:32:02.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.127+0000 7f7fb1392700 1 -- start start 2024-05-09T08:32:02.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.127+0000 7f7fb1392700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7fac06a8c0 0x7f7fac1012e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:02.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.127+0000 7f7fb1392700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fac06b260 0x7f7fac101820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:02.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.127+0000 7f7fb1392700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7fac108390 0x7f7fac101d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:02.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.127+0000 7f7fb1392700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7fac1124c0 con 0x7f7fac06b260 2024-05-09T08:32:02.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.127+0000 7f7fb1392700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7fac112340 con 0x7f7fac108390 2024-05-09T08:32:02.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.127+0000 7f7fb1392700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7fac112640 con 0x7f7fac06a8c0 2024-05-09T08:32:02.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.127+0000 7f7faa7fc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fac06b260 0x7f7fac101820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:02.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.127+0000 7f7faa7fc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fac06b260 0x7f7fac101820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47094/0 (socket says 172.21.15.107:47094) 2024-05-09T08:32:02.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.127+0000 7f7faa7fc700 1 -- 172.21.15.107:0/3510847230 learned_addr learned my addr 172.21.15.107:0/3510847230 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:02.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.127+0000 7f7faaffd700 1 --2- 172.21.15.107:0/3510847230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7fac06a8c0 0x7f7fac1012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:02.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.127+0000 7f7fab7fe700 1 --2- 172.21.15.107:0/3510847230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7fac108390 0x7f7fac101d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:02.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.128+0000 7f7faa7fc700 1 -- 172.21.15.107:0/3510847230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7fac06a8c0 msgr2=0x7f7fac1012e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:02.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.128+0000 7f7faa7fc700 1 --2- 172.21.15.107:0/3510847230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7fac06a8c0 0x7f7fac1012e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:02.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.128+0000 7f7faa7fc700 1 -- 172.21.15.107:0/3510847230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7fac108390 msgr2=0x7f7fac101d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:02.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.128+0000 7f7faa7fc700 1 --2- 172.21.15.107:0/3510847230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7fac108390 0x7f7fac101d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:02.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.128+0000 7f7faa7fc700 1 -- 172.21.15.107:0/3510847230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fac1a6f20 con 0x7f7fac06b260 2024-05-09T08:32:02.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.128+0000 7f7faaffd700 1 --2- 172.21.15.107:0/3510847230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7fac06a8c0 0x7f7fac1012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:32:02.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.128+0000 7f7faa7fc700 1 --2- 172.21.15.107:0/3510847230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fac06b260 0x7f7fac101820 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7fa000e9a0 tx=0x7f7fa000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:02.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.128+0000 7f7f93fff700 1 -- 172.21.15.107:0/3510847230 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa000ca10 con 0x7f7fac06b260 2024-05-09T08:32:02.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.128+0000 7f7fb1392700 1 -- 172.21.15.107:0/3510847230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fac1a7210 con 0x7f7fac06b260 2024-05-09T08:32:02.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.128+0000 7f7f93fff700 1 -- 172.21.15.107:0/3510847230 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7fa000cbb0 con 0x7f7fac06b260 2024-05-09T08:32:02.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.128+0000 7f7f93fff700 1 -- 172.21.15.107:0/3510847230 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa0018680 con 0x7f7fac06b260 2024-05-09T08:32:02.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.128+0000 7f7fb1392700 1 -- 172.21.15.107:0/3510847230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fac1a77a0 con 0x7f7fac06b260 2024-05-09T08:32:02.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.129+0000 7f7fb1392700 1 -- 172.21.15.107:0/3510847230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7fac062a30 con 0x7f7fac06b260 2024-05-09T08:32:02.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.130+0000 7f7f93fff700 1 -- 172.21.15.107:0/3510847230 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f7fa0010430 con 0x7f7fac06b260 2024-05-09T08:32:02.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.131+0000 7f7f93fff700 1 --2- 172.21.15.107:0/3510847230 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7f940a2cb0 0x7f7f940a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:02.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.131+0000 7f7faaffd700 1 --2- 172.21.15.107:0/3510847230 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7f940a2cb0 0x7f7f940a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:02.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.131+0000 7f7f93fff700 1 -- 172.21.15.107:0/3510847230 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7fa00cfb30 con 0x7f7fac06b260 2024-05-09T08:32:02.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.131+0000 7f7f93fff700 1 --2- 172.21.15.107:0/3510847230 >> [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474] conn(0x7f7f940ade10 0x7f7f940b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:02.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.131+0000 7f7f93fff700 1 -- 172.21.15.107:0/3510847230 --> [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f940b0980 con 0x7f7f940ade10 2024-05-09T08:32:02.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.131+0000 7f7f93fff700 1 -- 172.21.15.107:0/3510847230 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f7fa00cff80 con 0x7f7fac06b260 2024-05-09T08:32:02.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.132+0000 7f7fab7fe700 1 --2- 172.21.15.107:0/3510847230 >> [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474] conn(0x7f7f940ade10 0x7f7f940b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:02.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.132+0000 7f7faaffd700 1 --2- 172.21.15.107:0/3510847230 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7f940a2cb0 0x7f7f940a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c009fd0 tx=0x7f7f9c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:02.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.132+0000 7f7fab7fe700 1 --2- 172.21.15.107:0/3510847230 >> [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474] conn(0x7f7f940ade10 0x7f7f940b02b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:02.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.133+0000 7f7f93fff700 1 -- 172.21.15.107:0/3510847230 <== osd.11 v2:172.21.15.157:6824/3347498474 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7f940b0980 con 0x7f7f940ade10 2024-05-09T08:32:02.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.149+0000 7f7fb1392700 1 -- 172.21.15.107:0/3510847230 --> [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7fac062e30 con 0x7f7f940ade10 2024-05-09T08:32:02.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.149+0000 7f7f93fff700 1 -- 172.21.15.107:0/3510847230 <== osd.11 v2:172.21.15.157:6824/3347498474 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7fac062e30 con 0x7f7f940ade10 2024-05-09T08:32:02.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.150+0000 7f7fb1392700 1 -- 172.21.15.107:0/3510847230 >> [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474] conn(0x7f7f940ade10 msgr2=0x7f7f940b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:02.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.150+0000 7f7fb1392700 1 --2- 172.21.15.107:0/3510847230 >> [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474] conn(0x7f7f940ade10 0x7f7f940b02b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:02.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.150+0000 7f7fb1392700 1 -- 172.21.15.107:0/3510847230 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7f940a2cb0 msgr2=0x7f7f940a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:02.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.150+0000 7f7fb1392700 1 --2- 172.21.15.107:0/3510847230 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7f940a2cb0 0x7f7f940a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c009fd0 tx=0x7f7f9c009260 comp rx=0 tx=0).stop 2024-05-09T08:32:02.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.150+0000 7f7fb1392700 1 -- 172.21.15.107:0/3510847230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fac06b260 msgr2=0x7f7fac101820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:02.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.150+0000 7f7fb1392700 1 --2- 172.21.15.107:0/3510847230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fac06b260 0x7f7fac101820 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7fa000e9a0 tx=0x7f7fa000ecb0 comp rx=0 tx=0).stop 2024-05-09T08:32:02.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.150+0000 7f7fab7fe700 1 -- 172.21.15.107:0/3510847230 reap_dead start 2024-05-09T08:32:02.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.150+0000 7f7fb1392700 1 -- 172.21.15.107:0/3510847230 shutdown_connections 2024-05-09T08:32:02.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.150+0000 7f7fb1392700 1 -- 172.21.15.107:0/3510847230 >> 172.21.15.107:0/3510847230 conn(0x7f7fac078190 msgr2=0x7f7fac0ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:02.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.150+0000 7f7fb1392700 1 -- 172.21.15.107:0/3510847230 shutdown_connections 2024-05-09T08:32:02.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:02.150+0000 7f7fb1392700 1 -- 172.21.15.107:0/3510847230 wait complete. 2024-05-09T08:32:02.845 INFO:teuthology.orchestra.run.smithi107.stdout:343597383706 2024-05-09T08:32:02.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-09T08:32:02.890 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:02.892 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:03.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.411+0000 7f269ed66700 1 -- 172.21.15.107:0/282908055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26981083e0 msgr2=0x7f26981087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:03.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.411+0000 7f269ed66700 1 --2- 172.21.15.107:0/282908055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26981083e0 0x7f26981087c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f268c009a10 tx=0x7f268c009d20 comp rx=0 tx=0).stop 2024-05-09T08:32:03.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.411+0000 7f269ed66700 1 -- 172.21.15.107:0/282908055 shutdown_connections 2024-05-09T08:32:03.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.411+0000 7f269ed66700 1 --2- 172.21.15.107:0/282908055 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2698102d40 0x7f269810f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:03.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.411+0000 7f269ed66700 1 --2- 172.21.15.107:0/282908055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26981083e0 0x7f26981087c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:03.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.411+0000 7f269ed66700 1 --2- 172.21.15.107:0/282908055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26981023a0 0x7f2698102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:03.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.411+0000 7f269ed66700 1 -- 172.21.15.107:0/282908055 >> 172.21.15.107:0/282908055 conn(0x7f26980793f0 msgr2=0x7f2698079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:03.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.411+0000 7f269ed66700 1 -- 172.21.15.107:0/282908055 shutdown_connections 2024-05-09T08:32:03.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.411+0000 7f269ed66700 1 -- 172.21.15.107:0/282908055 wait complete. 2024-05-09T08:32:03.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.412+0000 7f269ed66700 1 Processor -- start 2024-05-09T08:32:03.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.412+0000 7f269ed66700 1 -- start start 2024-05-09T08:32:03.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.412+0000 7f269ed66700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26981023a0 0x7f269819d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:03.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.412+0000 7f269ed66700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2698102d40 0x7f26981966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:03.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.412+0000 7f269ed66700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26981083e0 0x7f2698196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:03.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.412+0000 7f269ed66700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2698110380 con 0x7f2698102d40 2024-05-09T08:32:03.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.412+0000 7f269ed66700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2698110200 con 0x7f26981023a0 2024-05-09T08:32:03.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.412+0000 7f269ed66700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2698110500 con 0x7f26981083e0 2024-05-09T08:32:03.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.413+0000 7f2697fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2698102d40 0x7f26981966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:03.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.413+0000 7f2697fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2698102d40 0x7f26981966a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47104/0 (socket says 172.21.15.107:47104) 2024-05-09T08:32:03.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.413+0000 7f2697fff700 1 -- 172.21.15.107:0/631875295 learned_addr learned my addr 172.21.15.107:0/631875295 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:03.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.413+0000 7f269d303700 1 --2- 172.21.15.107:0/631875295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26981083e0 0x7f2698196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:03.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.413+0000 7f2697fff700 1 -- 172.21.15.107:0/631875295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26981083e0 msgr2=0x7f2698196be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:03.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.413+0000 7f2697fff700 1 --2- 172.21.15.107:0/631875295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26981083e0 0x7f2698196be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:03.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.413+0000 7f2697fff700 1 -- 172.21.15.107:0/631875295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26981023a0 msgr2=0x7f269819d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:03.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.413+0000 7f2697fff700 1 --2- 172.21.15.107:0/631875295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26981023a0 0x7f269819d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:03.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.413+0000 7f2697fff700 1 -- 172.21.15.107:0/631875295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2698197220 con 0x7f2698102d40 2024-05-09T08:32:03.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.413+0000 7f2697fff700 1 --2- 172.21.15.107:0/631875295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2698102d40 0x7f26981966a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f268400d950 tx=0x7f268400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:03.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.414+0000 7f2695ffb700 1 -- 172.21.15.107:0/631875295 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684016070 con 0x7f2698102d40 2024-05-09T08:32:03.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.414+0000 7f269ed66700 1 -- 172.21.15.107:0/631875295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f269819b0f0 con 0x7f2698102d40 2024-05-09T08:32:03.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.414+0000 7f269ed66700 1 -- 172.21.15.107:0/631875295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f269819b680 con 0x7f2698102d40 2024-05-09T08:32:03.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.414+0000 7f2695ffb700 1 -- 172.21.15.107:0/631875295 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2684010460 con 0x7f2698102d40 2024-05-09T08:32:03.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.414+0000 7f2695ffb700 1 -- 172.21.15.107:0/631875295 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268400f620 con 0x7f2698102d40 2024-05-09T08:32:03.415 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:03.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.415+0000 7f269ed66700 1 -- 172.21.15.107:0/631875295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f269819bbc0 con 0x7f2698102d40 2024-05-09T08:32:03.415 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:03.416 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:03.416 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:03.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.417+0000 7f2695ffb700 1 -- 172.21.15.107:0/631875295 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f2684019070 con 0x7f2698102d40 2024-05-09T08:32:03.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.418+0000 7f2695ffb700 1 --2- 172.21.15.107:0/631875295 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f26880a2d30 0x7f26880a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:03.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.418+0000 7f2695ffb700 1 -- 172.21.15.107:0/631875295 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2684009b40 con 0x7f2698102d40 2024-05-09T08:32:03.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.418+0000 7f269cb02700 1 --2- 172.21.15.107:0/631875295 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f26880a2d30 0x7f26880a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:03.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.418+0000 7f2695ffb700 1 --2- 172.21.15.107:0/631875295 >> [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671] conn(0x7f26880ade90 0x7f26880b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:03.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.418+0000 7f2695ffb700 1 -- 172.21.15.107:0/631875295 --> [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f26880b0a00 con 0x7f26880ade90 2024-05-09T08:32:03.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.418+0000 7f2695ffb700 1 -- 172.21.15.107:0/631875295 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f26840d06a0 con 0x7f2698102d40 2024-05-09T08:32:03.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.419+0000 7f269d303700 1 --2- 172.21.15.107:0/631875295 >> [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671] conn(0x7f26880ade90 0x7f26880b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:03.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.419+0000 7f269cb02700 1 --2- 172.21.15.107:0/631875295 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f26880a2d30 0x7f26880a51f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f268c009750 tx=0x7f268c009920 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:03.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.420+0000 7f269d303700 1 --2- 172.21.15.107:0/631875295 >> [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671] conn(0x7f26880ade90 0x7f26880b0330 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:03.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.422+0000 7f2695ffb700 1 -- 172.21.15.107:0/631875295 <== osd.1 v2:172.21.15.110:6800/370054671 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f26880b0a00 con 0x7f26880ade90 2024-05-09T08:32:03.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.444+0000 7f269ed66700 1 -- 172.21.15.107:0/631875295 --> [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2698103a90 con 0x7f26880ade90 2024-05-09T08:32:03.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.445+0000 7f2695ffb700 1 -- 172.21.15.107:0/631875295 <== osd.1 v2:172.21.15.110:6800/370054671 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2698103a90 con 0x7f26880ade90 2024-05-09T08:32:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.446+0000 7f269ed66700 1 -- 172.21.15.107:0/631875295 >> [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671] conn(0x7f26880ade90 msgr2=0x7f26880b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.446+0000 7f269ed66700 1 --2- 172.21.15.107:0/631875295 >> [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671] conn(0x7f26880ade90 0x7f26880b0330 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.446+0000 7f269ed66700 1 -- 172.21.15.107:0/631875295 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f26880a2d30 msgr2=0x7f26880a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.446+0000 7f269ed66700 1 --2- 172.21.15.107:0/631875295 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f26880a2d30 0x7f26880a51f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f268c009750 tx=0x7f268c009920 comp rx=0 tx=0).stop 2024-05-09T08:32:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.446+0000 7f269ed66700 1 -- 172.21.15.107:0/631875295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2698102d40 msgr2=0x7f26981966a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.446+0000 7f269ed66700 1 --2- 172.21.15.107:0/631875295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2698102d40 0x7f26981966a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f268400d950 tx=0x7f268400dd10 comp rx=0 tx=0).stop 2024-05-09T08:32:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.446+0000 7f269d303700 1 -- 172.21.15.107:0/631875295 reap_dead start 2024-05-09T08:32:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.446+0000 7f269ed66700 1 -- 172.21.15.107:0/631875295 shutdown_connections 2024-05-09T08:32:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.446+0000 7f269ed66700 1 -- 172.21.15.107:0/631875295 >> 172.21.15.107:0/631875295 conn(0x7f26980793f0 msgr2=0x7f26981013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.447+0000 7f269ed66700 1 -- 172.21.15.107:0/631875295 shutdown_connections 2024-05-09T08:32:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:03.447+0000 7f269ed66700 1 -- 172.21.15.107:0/631875295 wait complete. 2024-05-09T08:32:04.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:04 smithi107 ceph-mon[30122]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 160 B/s, 0 objects/s recovering 2024-05-09T08:32:04.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.838+0000 7fa00c22c700 1 -- 172.21.15.107:0/1206704184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa004102f80 msgr2=0x7fa004106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:04.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.838+0000 7fa00c22c700 1 --2- 172.21.15.107:0/1206704184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa004102f80 0x7fa004106f80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4009990 tx=0x7f9ff4009ca0 comp rx=0 tx=0).stop 2024-05-09T08:32:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.838+0000 7fa00c22c700 1 -- 172.21.15.107:0/1206704184 shutdown_connections 2024-05-09T08:32:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.838+0000 7fa00c22c700 1 --2- 172.21.15.107:0/1206704184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa004107660 0x7fa00410f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.838+0000 7fa00c22c700 1 --2- 172.21.15.107:0/1206704184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0041025d0 0x7fa0041029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.838+0000 7fa00c22c700 1 --2- 172.21.15.107:0/1206704184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa004102f80 0x7fa004106f80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.838+0000 7fa00c22c700 1 -- 172.21.15.107:0/1206704184 >> 172.21.15.107:0/1206704184 conn(0x7fa004079480 msgr2=0x7fa004079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.838+0000 7fa00c22c700 1 -- 172.21.15.107:0/1206704184 shutdown_connections 2024-05-09T08:32:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.838+0000 7fa00c22c700 1 -- 172.21.15.107:0/1206704184 wait complete. 2024-05-09T08:32:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.838+0000 7fa00c22c700 1 Processor -- start 2024-05-09T08:32:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.838+0000 7fa00c22c700 1 -- start start 2024-05-09T08:32:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.838+0000 7fa00c22c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0041025d0 0x7fa00419e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.838+0000 7fa00c22c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa004102f80 0x7fa00419ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa00c22c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa004107660 0x7fa004198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa00c22c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa0041125b0 con 0x7fa0041025d0 2024-05-09T08:32:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa00c22c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa004112430 con 0x7fa004102f80 2024-05-09T08:32:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa00c22c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa004112730 con 0x7fa004107660 2024-05-09T08:32:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa009fc8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0041025d0 0x7fa00419e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa009fc8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0041025d0 0x7fa00419e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47136/0 (socket says 172.21.15.107:47136) 2024-05-09T08:32:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa009fc8700 1 -- 172.21.15.107:0/1197032828 learned_addr learned my addr 172.21.15.107:0/1197032828 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa0097c7700 1 --2- 172.21.15.107:0/1197032828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa004102f80 0x7fa00419ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa00a7c9700 1 --2- 172.21.15.107:0/1197032828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa004107660 0x7fa004198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa009fc8700 1 -- 172.21.15.107:0/1197032828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa004107660 msgr2=0x7fa004198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa009fc8700 1 --2- 172.21.15.107:0/1197032828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa004107660 0x7fa004198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa009fc8700 1 -- 172.21.15.107:0/1197032828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa004102f80 msgr2=0x7fa00419ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa009fc8700 1 --2- 172.21.15.107:0/1197032828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa004102f80 0x7fa00419ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa009fc8700 1 -- 172.21.15.107:0/1197032828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa004199070 con 0x7fa0041025d0 2024-05-09T08:32:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa009fc8700 1 --2- 172.21.15.107:0/1197032828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0041025d0 0x7fa00419e750 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa00000b530 tx=0x7fa00000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7f9ffaffd700 1 -- 172.21.15.107:0/1197032828 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa000015020 con 0x7fa0041025d0 2024-05-09T08:32:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa00c22c700 1 -- 172.21.15.107:0/1197032828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa004199360 con 0x7fa0041025d0 2024-05-09T08:32:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7f9ffaffd700 1 -- 172.21.15.107:0/1197032828 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa000011ce0 con 0x7fa0041025d0 2024-05-09T08:32:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7f9ffaffd700 1 -- 172.21.15.107:0/1197032828 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa000010e40 con 0x7fa0041025d0 2024-05-09T08:32:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.839+0000 7fa00c22c700 1 -- 172.21.15.107:0/1197032828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0041a5620 con 0x7fa0041025d0 2024-05-09T08:32:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.840+0000 7fa00c22c700 1 -- 172.21.15.107:0/1197032828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa004062ab0 con 0x7fa0041025d0 2024-05-09T08:32:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.841+0000 7f9ffaffd700 1 -- 172.21.15.107:0/1197032828 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fa000018020 con 0x7fa0041025d0 2024-05-09T08:32:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.842+0000 7f9ffaffd700 1 --2- 172.21.15.107:0/1197032828 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9ff00a2cb0 0x7f9ff00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.842+0000 7fa0097c7700 1 --2- 172.21.15.107:0/1197032828 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9ff00a2cb0 0x7f9ff00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.842+0000 7f9ffaffd700 1 -- 172.21.15.107:0/1197032828 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa0000cf330 con 0x7fa0041025d0 2024-05-09T08:32:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.842+0000 7f9ffaffd700 1 --2- 172.21.15.107:0/1197032828 >> [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355] conn(0x7f9ff00ade10 0x7f9ff00b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.842+0000 7f9ffaffd700 1 -- 172.21.15.107:0/1197032828 --> [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9ff00b0980 con 0x7f9ff00ade10 2024-05-09T08:32:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.842+0000 7f9ffaffd700 1 -- 172.21.15.107:0/1197032828 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fa0000cf780 con 0x7fa0041025d0 2024-05-09T08:32:04.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.842+0000 7fa0097c7700 1 --2- 172.21.15.107:0/1197032828 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9ff00a2cb0 0x7f9ff00a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4006010 tx=0x7f9ff4005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:04.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.843+0000 7fa00a7c9700 1 --2- 172.21.15.107:0/1197032828 >> [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355] conn(0x7f9ff00ade10 0x7f9ff00b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:04.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.843+0000 7fa00a7c9700 1 --2- 172.21.15.107:0/1197032828 >> [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355] conn(0x7f9ff00ade10 0x7f9ff00b02b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:04.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:04 smithi157 ceph-mon[33614]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 160 B/s, 0 objects/s recovering 2024-05-09T08:32:04.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.844+0000 7f9ffaffd700 1 -- 172.21.15.107:0/1197032828 <== osd.7 v2:172.21.15.110:6816/204549355 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9ff00b0980 con 0x7f9ff00ade10 2024-05-09T08:32:04.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.860+0000 7fa00c22c700 1 -- 172.21.15.107:0/1197032828 --> [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa004062f30 con 0x7f9ff00ade10 2024-05-09T08:32:04.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.861+0000 7f9ffaffd700 1 -- 172.21.15.107:0/1197032828 <== osd.7 v2:172.21.15.110:6816/204549355 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa004062f30 con 0x7f9ff00ade10 2024-05-09T08:32:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.861+0000 7fa00c22c700 1 -- 172.21.15.107:0/1197032828 >> [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355] conn(0x7f9ff00ade10 msgr2=0x7f9ff00b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.861+0000 7fa00c22c700 1 --2- 172.21.15.107:0/1197032828 >> [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355] conn(0x7f9ff00ade10 0x7f9ff00b02b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.861+0000 7fa00c22c700 1 -- 172.21.15.107:0/1197032828 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9ff00a2cb0 msgr2=0x7f9ff00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.861+0000 7fa00c22c700 1 --2- 172.21.15.107:0/1197032828 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9ff00a2cb0 0x7f9ff00a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4006010 tx=0x7f9ff4005960 comp rx=0 tx=0).stop 2024-05-09T08:32:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.861+0000 7fa00c22c700 1 -- 172.21.15.107:0/1197032828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0041025d0 msgr2=0x7fa00419e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.861+0000 7fa00c22c700 1 --2- 172.21.15.107:0/1197032828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0041025d0 0x7fa00419e750 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa00000b530 tx=0x7fa00000b840 comp rx=0 tx=0).stop 2024-05-09T08:32:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.861+0000 7fa00a7c9700 1 -- 172.21.15.107:0/1197032828 reap_dead start 2024-05-09T08:32:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.861+0000 7fa00c22c700 1 -- 172.21.15.107:0/1197032828 shutdown_connections 2024-05-09T08:32:04.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.861+0000 7fa00c22c700 1 -- 172.21.15.107:0/1197032828 >> 172.21.15.107:0/1197032828 conn(0x7fa004079480 msgr2=0x7fa0041006a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:04.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.861+0000 7fa00c22c700 1 -- 172.21.15.107:0/1197032828 shutdown_connections 2024-05-09T08:32:04.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:04.861+0000 7fa00c22c700 1 -- 172.21.15.107:0/1197032828 wait complete. 2024-05-09T08:32:04.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:04 smithi110 ceph-mon[33622]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 160 B/s, 0 objects/s recovering 2024-05-09T08:32:06.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:06 smithi107 ceph-mon[30122]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 129 B/s, 0 objects/s recovering 2024-05-09T08:32:06.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:06 smithi157 ceph-mon[33614]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 129 B/s, 0 objects/s recovering 2024-05-09T08:32:06.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:06 smithi110 ceph-mon[33622]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 129 B/s, 0 objects/s recovering 2024-05-09T08:32:07.998 INFO:teuthology.orchestra.run.smithi107.stdout:219043332142 2024-05-09T08:32:07.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-09T08:32:08.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:08 smithi107 ceph-mon[30122]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 117 B/s, 0 objects/s recovering 2024-05-09T08:32:08.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:08 smithi157 ceph-mon[33614]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 117 B/s, 0 objects/s recovering 2024-05-09T08:32:08.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:08 smithi110 ceph-mon[33622]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 117 B/s, 0 objects/s recovering 2024-05-09T08:32:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:10.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:10 smithi107 ceph-mon[30122]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 96 B/s, 0 objects/s recovering 2024-05-09T08:32:10.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:10 smithi157 ceph-mon[33614]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 96 B/s, 0 objects/s recovering 2024-05-09T08:32:10.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:10 smithi110 ceph-mon[33622]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 96 B/s, 0 objects/s recovering 2024-05-09T08:32:12.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:12 smithi157 ceph-mon[33614]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 48 B/s, 0 objects/s recovering 2024-05-09T08:32:12.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:12 smithi110 ceph-mon[33622]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 48 B/s, 0 objects/s recovering 2024-05-09T08:32:12.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:12 smithi107 ceph-mon[30122]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 48 B/s, 0 objects/s recovering 2024-05-09T08:32:12.903 INFO:teuthology.orchestra.run.smithi107.stdout:55834574925 2024-05-09T08:32:12.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-09T08:32:13.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.339+0000 7f7b3cef7700 1 -- 172.21.15.107:0/526927633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38106540 msgr2=0x7f7b3810e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:13.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.339+0000 7f7b3cef7700 1 --2- 172.21.15.107:0/526927633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38106540 0x7f7b3810e110 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000b0d0 tx=0x7f7b3000b3e0 comp rx=0 tx=0).stop 2024-05-09T08:32:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.339+0000 7f7b3cef7700 1 -- 172.21.15.107:0/526927633 shutdown_connections 2024-05-09T08:32:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.339+0000 7f7b3cef7700 1 --2- 172.21.15.107:0/526927633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38106540 0x7f7b3810e110 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.339+0000 7f7b3cef7700 1 --2- 172.21.15.107:0/526927633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b381014b0 0x7f7b38101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.339+0000 7f7b3cef7700 1 --2- 172.21.15.107:0/526927633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b38101e60 0x7f7b38105e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.339+0000 7f7b3cef7700 1 -- 172.21.15.107:0/526927633 >> 172.21.15.107:0/526927633 conn(0x7f7b380781f0 msgr2=0x7f7b38078600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.339+0000 7f7b3cef7700 1 -- 172.21.15.107:0/526927633 shutdown_connections 2024-05-09T08:32:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.339+0000 7f7b3cef7700 1 -- 172.21.15.107:0/526927633 wait complete. 2024-05-09T08:32:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.340+0000 7f7b3cef7700 1 Processor -- start 2024-05-09T08:32:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.340+0000 7f7b3cef7700 1 -- start start 2024-05-09T08:32:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.340+0000 7f7b3cef7700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b381014b0 0x7f7b3819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.340+0000 7f7b3cef7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b38101e60 0x7f7b3819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b3cef7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38106540 0x7f7b38198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b3cef7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7b381113e0 con 0x7f7b38106540 2024-05-09T08:32:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b3cef7700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7b38111260 con 0x7f7b38101e60 2024-05-09T08:32:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b3cef7700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7b38111560 con 0x7f7b381014b0 2024-05-09T08:32:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b36d9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38106540 0x7f7b38198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b36d9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38106540 0x7f7b38198850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43384/0 (socket says 172.21.15.107:43384) 2024-05-09T08:32:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b36d9d700 1 -- 172.21.15.107:0/2948064031 learned_addr learned my addr 172.21.15.107:0/2948064031 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b35d9b700 1 --2- 172.21.15.107:0/2948064031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b38101e60 0x7f7b3819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b3659c700 1 --2- 172.21.15.107:0/2948064031 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b381014b0 0x7f7b3819e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b36d9d700 1 -- 172.21.15.107:0/2948064031 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b381014b0 msgr2=0x7f7b3819e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b36d9d700 1 --2- 172.21.15.107:0/2948064031 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b381014b0 0x7f7b3819e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b36d9d700 1 -- 172.21.15.107:0/2948064031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b38101e60 msgr2=0x7f7b3819ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b36d9d700 1 --2- 172.21.15.107:0/2948064031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b38101e60 0x7f7b3819ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b36d9d700 1 -- 172.21.15.107:0/2948064031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b38199030 con 0x7f7b38106540 2024-05-09T08:32:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b3659c700 1 --2- 172.21.15.107:0/2948064031 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b381014b0 0x7f7b3819e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:32:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b36d9d700 1 --2- 172.21.15.107:0/2948064031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38106540 0x7f7b38198850 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7b300095c0 tx=0x7f7b30007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b277fe700 1 -- 172.21.15.107:0/2948064031 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3001f070 con 0x7f7b38106540 2024-05-09T08:32:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b3cef7700 1 -- 172.21.15.107:0/2948064031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b381992c0 con 0x7f7b38106540 2024-05-09T08:32:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b277fe700 1 -- 172.21.15.107:0/2948064031 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b30003c00 con 0x7f7b38106540 2024-05-09T08:32:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b277fe700 1 -- 172.21.15.107:0/2948064031 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3001b6a0 con 0x7f7b38106540 2024-05-09T08:32:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.341+0000 7f7b3cef7700 1 -- 172.21.15.107:0/2948064031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b381a5530 con 0x7f7b38106540 2024-05-09T08:32:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.342+0000 7f7b3cef7700 1 -- 172.21.15.107:0/2948064031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b38102940 con 0x7f7b38106540 2024-05-09T08:32:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.344+0000 7f7b277fe700 1 -- 172.21.15.107:0/2948064031 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f7b30022070 con 0x7f7b38106540 2024-05-09T08:32:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.344+0000 7f7b277fe700 1 --2- 172.21.15.107:0/2948064031 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7b200a2cb0 0x7f7b200a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.344+0000 7f7b3659c700 1 --2- 172.21.15.107:0/2948064031 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7b200a2cb0 0x7f7b200a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.344+0000 7f7b277fe700 1 -- 172.21.15.107:0/2948064031 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7b3000f020 con 0x7f7b38106540 2024-05-09T08:32:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.345+0000 7f7b3659c700 1 --2- 172.21.15.107:0/2948064031 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7b200a2cb0 0x7f7b200a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7b28005950 tx=0x7f7b280058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.348+0000 7f7b277fe700 1 -- 172.21.15.107:0/2948064031 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7b30029020 con 0x7f7b38106540 2024-05-09T08:32:13.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.549+0000 7f7b3cef7700 1 -- 172.21.15.107:0/2948064031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f7b3802db80 con 0x7f7b38106540 2024-05-09T08:32:13.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.549+0000 7f7b277fe700 1 -- 172.21.15.107:0/2948064031 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7b3009d630 con 0x7f7b38106540 2024-05-09T08:32:13.549 INFO:teuthology.orchestra.run.smithi107.stdout:141733920832 2024-05-09T08:32:13.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.552+0000 7f7b3cef7700 1 -- 172.21.15.107:0/2948064031 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7b200a2cb0 msgr2=0x7f7b200a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:13.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.552+0000 7f7b3cef7700 1 --2- 172.21.15.107:0/2948064031 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7b200a2cb0 0x7f7b200a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7b28005950 tx=0x7f7b280058e0 comp rx=0 tx=0).stop 2024-05-09T08:32:13.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.552+0000 7f7b3cef7700 1 -- 172.21.15.107:0/2948064031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38106540 msgr2=0x7f7b38198850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:13.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.552+0000 7f7b3cef7700 1 --2- 172.21.15.107:0/2948064031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38106540 0x7f7b38198850 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7b300095c0 tx=0x7f7b30007a20 comp rx=0 tx=0).stop 2024-05-09T08:32:13.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.552+0000 7f7b3cef7700 1 -- 172.21.15.107:0/2948064031 shutdown_connections 2024-05-09T08:32:13.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.552+0000 7f7b3cef7700 1 --2- 172.21.15.107:0/2948064031 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7b200a2cb0 0x7f7b200a5170 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:13.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.552+0000 7f7b3cef7700 1 --2- 172.21.15.107:0/2948064031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b38106540 0x7f7b38198850 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:13.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.552+0000 7f7b3cef7700 1 --2- 172.21.15.107:0/2948064031 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b381014b0 0x7f7b3819e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:13.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.552+0000 7f7b3cef7700 1 --2- 172.21.15.107:0/2948064031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b38101e60 0x7f7b3819ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:13.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.552+0000 7f7b3cef7700 1 -- 172.21.15.107:0/2948064031 >> 172.21.15.107:0/2948064031 conn(0x7f7b380781f0 msgr2=0x7f7b3806adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:13.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.552+0000 7f7b3cef7700 1 -- 172.21.15.107:0/2948064031 shutdown_connections 2024-05-09T08:32:13.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:13.552+0000 7f7b3cef7700 1 -- 172.21.15.107:0/2948064031 wait complete. 2024-05-09T08:32:14.367 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920827 got 141733920832 for osd.4 2024-05-09T08:32:14.367 DEBUG:teuthology.parallel:result is None 2024-05-09T08:32:14.632 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:14 smithi107 ceph-mon[30122]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 48 B/s, 0 objects/s recovering 2024-05-09T08:32:14.633 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:14 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2948064031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-09T08:32:14.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.633+0000 7fea1fa2a700 1 -- 172.21.15.107:0/3938879839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea18101570 msgr2=0x7fea181019f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:14.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.633+0000 7fea1fa2a700 1 --2- 172.21.15.107:0/3938879839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea18101570 0x7fea181019f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fea080099c0 tx=0x7fea08009cd0 comp rx=0 tx=0).stop 2024-05-09T08:32:14.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.633+0000 7fea1fa2a700 1 -- 172.21.15.107:0/3938879839 shutdown_connections 2024-05-09T08:32:14.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.633+0000 7fea1fa2a700 1 --2- 172.21.15.107:0/3938879839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea18101f30 0x7fea1810e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:14.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.633+0000 7fea1fa2a700 1 --2- 172.21.15.107:0/3938879839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea181062a0 0x7fea18106680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:14.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.633+0000 7fea1fa2a700 1 --2- 172.21.15.107:0/3938879839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea18101570 0x7fea181019f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:14.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.633+0000 7fea1fa2a700 1 -- 172.21.15.107:0/3938879839 >> 172.21.15.107:0/3938879839 conn(0x7fea18077090 msgr2=0x7fea180774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:14.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.633+0000 7fea1fa2a700 1 -- 172.21.15.107:0/3938879839 shutdown_connections 2024-05-09T08:32:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.633+0000 7fea1fa2a700 1 -- 172.21.15.107:0/3938879839 wait complete. 2024-05-09T08:32:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.634+0000 7fea1fa2a700 1 Processor -- start 2024-05-09T08:32:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.634+0000 7fea1fa2a700 1 -- start start 2024-05-09T08:32:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1fa2a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea18101570 0x7fea181975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1fa2a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea18101f30 0x7fea18197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1fa2a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea181062a0 0x7fea1819c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1fa2a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fea18111470 con 0x7fea18101570 2024-05-09T08:32:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1fa2a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fea181112f0 con 0x7fea181062a0 2024-05-09T08:32:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1fa2a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fea181115f0 con 0x7fea18101f30 2024-05-09T08:32:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1d7c6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea18101570 0x7fea181975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1d7c6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea18101570 0x7fea181975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43398/0 (socket says 172.21.15.107:43398) 2024-05-09T08:32:14.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1d7c6700 1 -- 172.21.15.107:0/1697098405 learned_addr learned my addr 172.21.15.107:0/1697098405 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1dfc7700 1 --2- 172.21.15.107:0/1697098405 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea181062a0 0x7fea1819c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1cfc5700 1 --2- 172.21.15.107:0/1697098405 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea18101f30 0x7fea18197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1d7c6700 1 -- 172.21.15.107:0/1697098405 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea18101f30 msgr2=0x7fea18197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1d7c6700 1 --2- 172.21.15.107:0/1697098405 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea18101f30 0x7fea18197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1d7c6700 1 -- 172.21.15.107:0/1697098405 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea181062a0 msgr2=0x7fea1819c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1d7c6700 1 --2- 172.21.15.107:0/1697098405 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea181062a0 0x7fea1819c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1d7c6700 1 -- 172.21.15.107:0/1697098405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1819cb50 con 0x7fea18101570 2024-05-09T08:32:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1cfc5700 1 --2- 172.21.15.107:0/1697098405 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea18101f30 0x7fea18197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:32:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1dfc7700 1 --2- 172.21.15.107:0/1697098405 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea181062a0 0x7fea1819c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:32:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1d7c6700 1 --2- 172.21.15.107:0/1697098405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea18101570 0x7fea181975c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fea1400e9a0 tx=0x7fea1400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:14.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea0e7fc700 1 -- 172.21.15.107:0/1697098405 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1400cbd0 con 0x7fea18101570 2024-05-09T08:32:14.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea0e7fc700 1 -- 172.21.15.107:0/1697098405 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea1400cd70 con 0x7fea18101570 2024-05-09T08:32:14.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1fa2a700 1 -- 172.21.15.107:0/1697098405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea1819ce40 con 0x7fea18101570 2024-05-09T08:32:14.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea0e7fc700 1 -- 172.21.15.107:0/1697098405 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea14018750 con 0x7fea18101570 2024-05-09T08:32:14.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.635+0000 7fea1fa2a700 1 -- 172.21.15.107:0/1697098405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea1819d350 con 0x7fea18101570 2024-05-09T08:32:14.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.636+0000 7fea1fa2a700 1 -- 172.21.15.107:0/1697098405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea18102990 con 0x7fea18101570 2024-05-09T08:32:14.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.638+0000 7fea0e7fc700 1 -- 172.21.15.107:0/1697098405 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fea14010430 con 0x7fea18101570 2024-05-09T08:32:14.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.639+0000 7fea0e7fc700 1 --2- 172.21.15.107:0/1697098405 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fea040a2cb0 0x7fea040a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:14.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.639+0000 7fea1cfc5700 1 --2- 172.21.15.107:0/1697098405 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fea040a2cb0 0x7fea040a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:14.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.639+0000 7fea0e7fc700 1 -- 172.21.15.107:0/1697098405 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fea140cf9a0 con 0x7fea18101570 2024-05-09T08:32:14.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.640+0000 7fea1cfc5700 1 --2- 172.21.15.107:0/1697098405 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fea040a2cb0 0x7fea040a5170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fea1806b600 tx=0x7fea08005940 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:14.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.643+0000 7fea0e7fc700 1 -- 172.21.15.107:0/1697098405 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fea1409c440 con 0x7fea18101570 2024-05-09T08:32:14.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.842+0000 7fea1fa2a700 1 -- 172.21.15.107:0/1697098405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fea1802db80 con 0x7fea18101570 2024-05-09T08:32:14.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.842+0000 7fea0e7fc700 1 -- 172.21.15.107:0/1697098405 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fea1409bd40 con 0x7fea18101570 2024-05-09T08:32:14.842 INFO:teuthology.orchestra.run.smithi107.stdout:309237645344 2024-05-09T08:32:14.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:14 smithi157 ceph-mon[33614]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 48 B/s, 0 objects/s recovering 2024-05-09T08:32:14.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:14 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2948064031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-09T08:32:14.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.844+0000 7fea1fa2a700 1 -- 172.21.15.107:0/1697098405 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fea040a2cb0 msgr2=0x7fea040a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:14.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.844+0000 7fea1fa2a700 1 --2- 172.21.15.107:0/1697098405 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fea040a2cb0 0x7fea040a5170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fea1806b600 tx=0x7fea08005940 comp rx=0 tx=0).stop 2024-05-09T08:32:14.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.844+0000 7fea1fa2a700 1 -- 172.21.15.107:0/1697098405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea18101570 msgr2=0x7fea181975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:14.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.844+0000 7fea1fa2a700 1 --2- 172.21.15.107:0/1697098405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea18101570 0x7fea181975c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fea1400e9a0 tx=0x7fea1400ed60 comp rx=0 tx=0).stop 2024-05-09T08:32:14.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.844+0000 7fea1fa2a700 1 -- 172.21.15.107:0/1697098405 shutdown_connections 2024-05-09T08:32:14.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.844+0000 7fea1fa2a700 1 --2- 172.21.15.107:0/1697098405 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea181062a0 0x7fea1819c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:14.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.844+0000 7fea1fa2a700 1 --2- 172.21.15.107:0/1697098405 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fea040a2cb0 0x7fea040a5170 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:14.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.844+0000 7fea1fa2a700 1 --2- 172.21.15.107:0/1697098405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea18101570 0x7fea181975c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:14.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.844+0000 7fea1fa2a700 1 --2- 172.21.15.107:0/1697098405 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea18101f30 0x7fea18197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:14.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.844+0000 7fea1fa2a700 1 -- 172.21.15.107:0/1697098405 >> 172.21.15.107:0/1697098405 conn(0x7fea18077090 msgr2=0x7fea180ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:14.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.845+0000 7fea1fa2a700 1 -- 172.21.15.107:0/1697098405 shutdown_connections 2024-05-09T08:32:14.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:14.845+0000 7fea1fa2a700 1 -- 172.21.15.107:0/1697098405 wait complete. 2024-05-09T08:32:14.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:14 smithi110 ceph-mon[33622]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 48 B/s, 0 objects/s recovering 2024-05-09T08:32:14.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:14 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2948064031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-09T08:32:15.625 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645341 got 309237645344 for osd.10 2024-05-09T08:32:15.626 DEBUG:teuthology.parallel:result is None 2024-05-09T08:32:15.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.800+0000 7fcd597c0700 1 -- 172.21.15.107:0/1414006374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd54102640 msgr2=0x7fcd54102a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:15.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.800+0000 7fcd597c0700 1 --2- 172.21.15.107:0/1414006374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd54102640 0x7fcd54102a20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcd440099c0 tx=0x7fcd44009cd0 comp rx=0 tx=0).stop 2024-05-09T08:32:15.800 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:15 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1697098405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-09T08:32:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.800+0000 7fcd597c0700 1 -- 172.21.15.107:0/1414006374 shutdown_connections 2024-05-09T08:32:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.800+0000 7fcd597c0700 1 --2- 172.21.15.107:0/1414006374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd54107630 0x7fcd5410f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.800+0000 7fcd597c0700 1 --2- 172.21.15.107:0/1414006374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd54102640 0x7fcd54102a20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.800+0000 7fcd597c0700 1 --2- 172.21.15.107:0/1414006374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd54102ff0 0x7fcd54106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.800+0000 7fcd597c0700 1 -- 172.21.15.107:0/1414006374 >> 172.21.15.107:0/1414006374 conn(0x7fcd540794e0 msgr2=0x7fcd540798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.801+0000 7fcd597c0700 1 -- 172.21.15.107:0/1414006374 shutdown_connections 2024-05-09T08:32:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.801+0000 7fcd597c0700 1 -- 172.21.15.107:0/1414006374 wait complete. 2024-05-09T08:32:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.801+0000 7fcd597c0700 1 Processor -- start 2024-05-09T08:32:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.801+0000 7fcd597c0700 1 -- start start 2024-05-09T08:32:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd597c0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd54102640 0x7fcd5419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd597c0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd54102ff0 0x7fcd5419ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd597c0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd54107630 0x7fcd541988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd597c0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcd54112560 con 0x7fcd54102ff0 2024-05-09T08:32:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd597c0700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcd541123e0 con 0x7fcd54102640 2024-05-09T08:32:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd597c0700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcd541126e0 con 0x7fcd54107630 2024-05-09T08:32:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd527fc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd54102ff0 0x7fcd5419ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd527fc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd54102ff0 0x7fcd5419ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43420/0 (socket says 172.21.15.107:43420) 2024-05-09T08:32:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd527fc700 1 -- 172.21.15.107:0/2220657105 learned_addr learned my addr 172.21.15.107:0/2220657105 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd52ffd700 1 --2- 172.21.15.107:0/2220657105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd54102640 0x7fcd5419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:15.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd537fe700 1 --2- 172.21.15.107:0/2220657105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd54107630 0x7fcd541988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd527fc700 1 -- 172.21.15.107:0/2220657105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd54107630 msgr2=0x7fcd541988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd527fc700 1 --2- 172.21.15.107:0/2220657105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd54107630 0x7fcd541988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd527fc700 1 -- 172.21.15.107:0/2220657105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd54102640 msgr2=0x7fcd5419e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd527fc700 1 --2- 172.21.15.107:0/2220657105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd54102640 0x7fcd5419e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd527fc700 1 -- 172.21.15.107:0/2220657105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd54199000 con 0x7fcd54102ff0 2024-05-09T08:32:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd52ffd700 1 --2- 172.21.15.107:0/2220657105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd54102640 0x7fcd5419e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:32:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd527fc700 1 --2- 172.21.15.107:0/2220657105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd54102ff0 0x7fcd5419ec20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800d950 tx=0x7fcd4800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd3bfff700 1 -- 172.21.15.107:0/2220657105 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48017070 con 0x7fcd54102ff0 2024-05-09T08:32:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd597c0700 1 -- 172.21.15.107:0/2220657105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd541992f0 con 0x7fcd54102ff0 2024-05-09T08:32:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd3bfff700 1 -- 172.21.15.107:0/2220657105 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd4800de30 con 0x7fcd54102ff0 2024-05-09T08:32:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.802+0000 7fcd3bfff700 1 -- 172.21.15.107:0/2220657105 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48005440 con 0x7fcd54102ff0 2024-05-09T08:32:15.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.803+0000 7fcd597c0700 1 -- 172.21.15.107:0/2220657105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd541a55b0 con 0x7fcd54102ff0 2024-05-09T08:32:15.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.804+0000 7fcd597c0700 1 -- 172.21.15.107:0/2220657105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd54069010 con 0x7fcd54102ff0 2024-05-09T08:32:15.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.805+0000 7fcd3bfff700 1 -- 172.21.15.107:0/2220657105 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fcd48004750 con 0x7fcd54102ff0 2024-05-09T08:32:15.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.805+0000 7fcd3bfff700 1 --2- 172.21.15.107:0/2220657105 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fcd3c0a2cb0 0x7fcd3c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:15.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.806+0000 7fcd52ffd700 1 --2- 172.21.15.107:0/2220657105 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fcd3c0a2cb0 0x7fcd3c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:15.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.806+0000 7fcd3bfff700 1 -- 172.21.15.107:0/2220657105 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcd480cf250 con 0x7fcd54102ff0 2024-05-09T08:32:15.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.806+0000 7fcd52ffd700 1 --2- 172.21.15.107:0/2220657105 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fcd3c0a2cb0 0x7fcd3c0a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009990 tx=0x7fcd44005a90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:15.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:15.809+0000 7fcd3bfff700 1 -- 172.21.15.107:0/2220657105 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcd4809bcf0 con 0x7fcd54102ff0 2024-05-09T08:32:15.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:15 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1697098405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-09T08:32:15.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:15 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1697098405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-09T08:32:16.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.013+0000 7fcd597c0700 1 -- 172.21.15.107:0/2220657105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fcd54199dd0 con 0x7fcd54102ff0 2024-05-09T08:32:16.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.014+0000 7fcd3bfff700 1 -- 172.21.15.107:0/2220657105 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcd4809b5f0 con 0x7fcd54102ff0 2024-05-09T08:32:16.013 INFO:teuthology.orchestra.run.smithi107.stdout:77309411402 2024-05-09T08:32:16.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.016+0000 7fcd597c0700 1 -- 172.21.15.107:0/2220657105 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fcd3c0a2cb0 msgr2=0x7fcd3c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:16.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.016+0000 7fcd597c0700 1 --2- 172.21.15.107:0/2220657105 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fcd3c0a2cb0 0x7fcd3c0a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009990 tx=0x7fcd44005a90 comp rx=0 tx=0).stop 2024-05-09T08:32:16.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.016+0000 7fcd597c0700 1 -- 172.21.15.107:0/2220657105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd54102ff0 msgr2=0x7fcd5419ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:16.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.016+0000 7fcd597c0700 1 --2- 172.21.15.107:0/2220657105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd54102ff0 0x7fcd5419ec20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800d950 tx=0x7fcd4800dd10 comp rx=0 tx=0).stop 2024-05-09T08:32:16.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.016+0000 7fcd597c0700 1 -- 172.21.15.107:0/2220657105 shutdown_connections 2024-05-09T08:32:16.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.016+0000 7fcd597c0700 1 --2- 172.21.15.107:0/2220657105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd54107630 0x7fcd541988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:16.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.016+0000 7fcd597c0700 1 --2- 172.21.15.107:0/2220657105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd54102640 0x7fcd5419e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:16.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.016+0000 7fcd597c0700 1 --2- 172.21.15.107:0/2220657105 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fcd3c0a2cb0 0x7fcd3c0a5170 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:16.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.016+0000 7fcd597c0700 1 --2- 172.21.15.107:0/2220657105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd54102ff0 0x7fcd5419ec20 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:16.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.016+0000 7fcd597c0700 1 -- 172.21.15.107:0/2220657105 >> 172.21.15.107:0/2220657105 conn(0x7fcd540794e0 msgr2=0x7fcd541067c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:16.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.016+0000 7fcd597c0700 1 -- 172.21.15.107:0/2220657105 shutdown_connections 2024-05-09T08:32:16.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.016+0000 7fcd597c0700 1 -- 172.21.15.107:0/2220657105 wait complete. 2024-05-09T08:32:16.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.028+0000 7f6fb32a9700 1 -- 172.21.15.107:0/4199870751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac1065a0 msgr2=0x7f6fac10e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:16.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.028+0000 7f6fb32a9700 1 --2- 172.21.15.107:0/4199870751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac1065a0 0x7f6fac10e1b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6fa400b0d0 tx=0x7f6fa400b3e0 comp rx=0 tx=0).stop 2024-05-09T08:32:16.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.028+0000 7f6fb32a9700 1 -- 172.21.15.107:0/4199870751 shutdown_connections 2024-05-09T08:32:16.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.028+0000 7f6fb32a9700 1 --2- 172.21.15.107:0/4199870751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac1065a0 0x7f6fac10e1b0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:16.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.028+0000 7f6fb32a9700 1 --2- 172.21.15.107:0/4199870751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fac101c80 0x7f6fac102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:16.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.028+0000 7f6fb32a9700 1 --2- 172.21.15.107:0/4199870751 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fac102630 0x7f6fac105ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:16.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.028+0000 7f6fb32a9700 1 -- 172.21.15.107:0/4199870751 >> 172.21.15.107:0/4199870751 conn(0x7f6fac077890 msgr2=0x7f6fac0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:16.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.028+0000 7f6fb32a9700 1 -- 172.21.15.107:0/4199870751 shutdown_connections 2024-05-09T08:32:16.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.028+0000 7f6fb32a9700 1 -- 172.21.15.107:0/4199870751 wait complete. 2024-05-09T08:32:16.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.029+0000 7f6fb32a9700 1 Processor -- start 2024-05-09T08:32:16.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.029+0000 7f6fb32a9700 1 -- start start 2024-05-09T08:32:16.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.029+0000 7f6fb32a9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac101c80 0x7f6fac19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:16.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.029+0000 7f6fb32a9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fac102630 0x7f6fac19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:16.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.029+0000 7f6fb32a9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fac1065a0 0x7f6fac1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:16.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.029+0000 7f6fb32a9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac111490 con 0x7f6fac101c80 2024-05-09T08:32:16.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.029+0000 7f6fb32a9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac111310 con 0x7f6fac1065a0 2024-05-09T08:32:16.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.029+0000 7f6fb32a9700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac111610 con 0x7f6fac102630 2024-05-09T08:32:16.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.029+0000 7f6fb1045700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac101c80 0x7f6fac19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:16.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.029+0000 7f6fb1045700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac101c80 0x7f6fac19e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43436/0 (socket says 172.21.15.107:43436) 2024-05-09T08:32:16.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.029+0000 7f6fb1045700 1 -- 172.21.15.107:0/4218581054 learned_addr learned my addr 172.21.15.107:0/4218581054 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:16.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.030+0000 7f6fb0844700 1 --2- 172.21.15.107:0/4218581054 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fac102630 0x7f6fac19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:16.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.030+0000 7f6fb1846700 1 --2- 172.21.15.107:0/4218581054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fac1065a0 0x7f6fac1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:16.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.030+0000 7f6fb1045700 1 -- 172.21.15.107:0/4218581054 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fac102630 msgr2=0x7f6fac19eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:16.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.030+0000 7f6fb1045700 1 --2- 172.21.15.107:0/4218581054 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fac102630 0x7f6fac19eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:16.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.030+0000 7f6fb1045700 1 -- 172.21.15.107:0/4218581054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fac1065a0 msgr2=0x7f6fac1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:16.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.030+0000 7f6fb1045700 1 --2- 172.21.15.107:0/4218581054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fac1065a0 0x7f6fac1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:16.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.030+0000 7f6fb1045700 1 -- 172.21.15.107:0/4218581054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fac198fc0 con 0x7f6fac101c80 2024-05-09T08:32:16.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.030+0000 7f6fb1045700 1 --2- 172.21.15.107:0/4218581054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac101c80 0x7f6fac19e660 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800d980 tx=0x7f6fa800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:16.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.030+0000 7f6fa27fc700 1 -- 172.21.15.107:0/4218581054 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8016070 con 0x7f6fac101c80 2024-05-09T08:32:16.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.030+0000 7f6fb32a9700 1 -- 172.21.15.107:0/4218581054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fac1992b0 con 0x7f6fac101c80 2024-05-09T08:32:16.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.030+0000 7f6fa27fc700 1 -- 172.21.15.107:0/4218581054 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6fa8004830 con 0x7f6fac101c80 2024-05-09T08:32:16.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.030+0000 7f6fa27fc700 1 -- 172.21.15.107:0/4218581054 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8005530 con 0x7f6fac101c80 2024-05-09T08:32:16.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.030+0000 7f6fb32a9700 1 -- 172.21.15.107:0/4218581054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fac0fcc30 con 0x7f6fac101c80 2024-05-09T08:32:16.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.031+0000 7f6fb32a9700 1 -- 172.21.15.107:0/4218581054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fac102a30 con 0x7f6fac101c80 2024-05-09T08:32:16.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.033+0000 7f6fa27fc700 1 -- 172.21.15.107:0/4218581054 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f6fa80049e0 con 0x7f6fac101c80 2024-05-09T08:32:16.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.034+0000 7f6fa27fc700 1 --2- 172.21.15.107:0/4218581054 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6f980a7140 0x7f6f980a9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:16.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.034+0000 7f6fb0844700 1 --2- 172.21.15.107:0/4218581054 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6f980a7140 0x7f6f980a9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:16.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.034+0000 7f6fa27fc700 1 -- 172.21.15.107:0/4218581054 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6fa80cf730 con 0x7f6fac101c80 2024-05-09T08:32:16.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.034+0000 7f6fb0844700 1 --2- 172.21.15.107:0/4218581054 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6f980a7140 0x7f6f980a9600 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6fac107f20 tx=0x7f6f9c00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:16.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.037+0000 7f6fa27fc700 1 -- 172.21.15.107:0/4218581054 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6fa801f020 con 0x7f6fac101c80 2024-05-09T08:32:16.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.238+0000 7f6fb32a9700 1 -- 172.21.15.107:0/4218581054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f6fac062f30 con 0x7f6fac101c80 2024-05-09T08:32:16.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.238+0000 7f6fa27fc700 1 -- 172.21.15.107:0/4218581054 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6fa809c170 con 0x7f6fac101c80 2024-05-09T08:32:16.238 INFO:teuthology.orchestra.run.smithi107.stdout:253403070507 2024-05-09T08:32:16.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.241+0000 7f6fb32a9700 1 -- 172.21.15.107:0/4218581054 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6f980a7140 msgr2=0x7f6f980a9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:16.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.241+0000 7f6fb32a9700 1 --2- 172.21.15.107:0/4218581054 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6f980a7140 0x7f6f980a9600 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6fac107f20 tx=0x7f6f9c00a380 comp rx=0 tx=0).stop 2024-05-09T08:32:16.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.241+0000 7f6fb32a9700 1 -- 172.21.15.107:0/4218581054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac101c80 msgr2=0x7f6fac19e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:16.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.241+0000 7f6fb32a9700 1 --2- 172.21.15.107:0/4218581054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac101c80 0x7f6fac19e660 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800d980 tx=0x7f6fa800dc90 comp rx=0 tx=0).stop 2024-05-09T08:32:16.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.241+0000 7f6fb32a9700 1 -- 172.21.15.107:0/4218581054 shutdown_connections 2024-05-09T08:32:16.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.241+0000 7f6fb32a9700 1 --2- 172.21.15.107:0/4218581054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6fac1065a0 0x7f6fac1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:16.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.241+0000 7f6fb32a9700 1 --2- 172.21.15.107:0/4218581054 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6f980a7140 0x7f6f980a9600 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:16.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.241+0000 7f6fb32a9700 1 --2- 172.21.15.107:0/4218581054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6fac101c80 0x7f6fac19e660 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:16.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.241+0000 7f6fb32a9700 1 --2- 172.21.15.107:0/4218581054 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fac102630 0x7f6fac19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:16.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.241+0000 7f6fb32a9700 1 -- 172.21.15.107:0/4218581054 >> 172.21.15.107:0/4218581054 conn(0x7f6fac077890 msgr2=0x7f6fac104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:16.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.241+0000 7f6fb32a9700 1 -- 172.21.15.107:0/4218581054 shutdown_connections 2024-05-09T08:32:16.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:16.241+0000 7f6fb32a9700 1 -- 172.21.15.107:0/4218581054 wait complete. 2024-05-09T08:32:16.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:16 smithi157 ceph-mon[33614]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-09T08:32:16.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:16 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2220657105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-09T08:32:16.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:16 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/4218581054' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-09T08:32:16.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:16 smithi110 ceph-mon[33622]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-09T08:32:16.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:16 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2220657105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-09T08:32:16.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:16 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/4218581054' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-09T08:32:16.912 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:16 smithi107 ceph-mon[30122]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-09T08:32:16.913 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:16 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2220657105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-09T08:32:16.913 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:16 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/4218581054' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-09T08:32:16.921 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:16.922 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:17.726 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070503 got 253403070507 for osd.8 2024-05-09T08:32:17.726 DEBUG:teuthology.parallel:result is None 2024-05-09T08:32:18.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:18 smithi157 ceph-mon[33614]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-09T08:32:18.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:18 smithi110 ceph-mon[33622]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-09T08:32:18.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:18 smithi107 ceph-mon[30122]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-09T08:32:19.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.470+0000 7fe8f2c21700 1 -- 172.21.15.107:0/1399585031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8ec101f00 msgr2=0x7fe8ec10e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:19.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.470+0000 7fe8f2c21700 1 --2- 172.21.15.107:0/1399585031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8ec101f00 0x7fe8ec10e080 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe8e400b0d0 tx=0x7fe8e400b3e0 comp rx=0 tx=0).stop 2024-05-09T08:32:19.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.470+0000 7fe8f2c21700 1 -- 172.21.15.107:0/1399585031 shutdown_connections 2024-05-09T08:32:19.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.470+0000 7fe8f2c21700 1 --2- 172.21.15.107:0/1399585031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8ec101f00 0x7fe8ec10e080 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:19.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.470+0000 7fe8f2c21700 1 --2- 172.21.15.107:0/1399585031 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8ec107430 0x7fe8ec107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:19.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.470+0000 7fe8f2c21700 1 --2- 172.21.15.107:0/1399585031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8ec101540 0x7fe8ec1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:19.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.470+0000 7fe8f2c21700 1 -- 172.21.15.107:0/1399585031 >> 172.21.15.107:0/1399585031 conn(0x7fe8ec077f80 msgr2=0x7fe8ec078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:19.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.470+0000 7fe8f2c21700 1 -- 172.21.15.107:0/1399585031 shutdown_connections 2024-05-09T08:32:19.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.470+0000 7fe8f2c21700 1 -- 172.21.15.107:0/1399585031 wait complete. 2024-05-09T08:32:19.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.471+0000 7fe8f2c21700 1 Processor -- start 2024-05-09T08:32:19.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.471+0000 7fe8f2c21700 1 -- start start 2024-05-09T08:32:19.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.471+0000 7fe8f2c21700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8ec101540 0x7fe8ec19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:19.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.471+0000 7fe8f2c21700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8ec101f00 0x7fe8ec19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:19.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.471+0000 7fe8f2c21700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8ec107430 0x7fe8ec1988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:19.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.471+0000 7fe8f2c21700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ec1114f0 con 0x7fe8ec101f00 2024-05-09T08:32:19.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.471+0000 7fe8f2c21700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ec111370 con 0x7fe8ec101540 2024-05-09T08:32:19.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.471+0000 7fe8f2c21700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ec111670 con 0x7fe8ec107430 2024-05-09T08:32:19.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.471+0000 7fe8ebfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8ec101f00 0x7fe8ec19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:19.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.471+0000 7fe8ebfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8ec101f00 0x7fe8ec19ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43448/0 (socket says 172.21.15.107:43448) 2024-05-09T08:32:19.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.471+0000 7fe8ebfff700 1 -- 172.21.15.107:0/1143524838 learned_addr learned my addr 172.21.15.107:0/1143524838 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:19.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.472+0000 7fe8f11be700 1 --2- 172.21.15.107:0/1143524838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8ec107430 0x7fe8ec1988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:19.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.472+0000 7fe8f09bd700 1 --2- 172.21.15.107:0/1143524838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8ec101540 0x7fe8ec19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:19.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.472+0000 7fe8ebfff700 1 -- 172.21.15.107:0/1143524838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8ec107430 msgr2=0x7fe8ec1988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:19.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.472+0000 7fe8ebfff700 1 --2- 172.21.15.107:0/1143524838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8ec107430 0x7fe8ec1988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:19.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.472+0000 7fe8ebfff700 1 -- 172.21.15.107:0/1143524838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8ec101540 msgr2=0x7fe8ec19e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:19.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.472+0000 7fe8ebfff700 1 --2- 172.21.15.107:0/1143524838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8ec101540 0x7fe8ec19e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:19.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.472+0000 7fe8ebfff700 1 -- 172.21.15.107:0/1143524838 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8ec199030 con 0x7fe8ec101f00 2024-05-09T08:32:19.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.472+0000 7fe8f11be700 1 --2- 172.21.15.107:0/1143524838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8ec107430 0x7fe8ec1988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:32:19.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.472+0000 7fe8ebfff700 1 --2- 172.21.15.107:0/1143524838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8ec101f00 0x7fe8ec19ec50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800b530 tx=0x7fe8d800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:19.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.472+0000 7fe8e9ffb700 1 -- 172.21.15.107:0/1143524838 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d8015020 con 0x7fe8ec101f00 2024-05-09T08:32:19.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.472+0000 7fe8f2c21700 1 -- 172.21.15.107:0/1143524838 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8ec199320 con 0x7fe8ec101f00 2024-05-09T08:32:19.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.472+0000 7fe8e9ffb700 1 -- 172.21.15.107:0/1143524838 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8d8011ce0 con 0x7fe8ec101f00 2024-05-09T08:32:19.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.472+0000 7fe8e9ffb700 1 -- 172.21.15.107:0/1143524838 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d8010d60 con 0x7fe8ec101f00 2024-05-09T08:32:19.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.472+0000 7fe8f2c21700 1 -- 172.21.15.107:0/1143524838 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8ec1a55e0 con 0x7fe8ec101f00 2024-05-09T08:32:19.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.473+0000 7fe8f2c21700 1 -- 172.21.15.107:0/1143524838 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8ec069010 con 0x7fe8ec101f00 2024-05-09T08:32:19.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.475+0000 7fe8e9ffb700 1 -- 172.21.15.107:0/1143524838 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fe8d8018020 con 0x7fe8ec101f00 2024-05-09T08:32:19.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.475+0000 7fe8e9ffb700 1 --2- 172.21.15.107:0/1143524838 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe8dc0a2cb0 0x7fe8dc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:19.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.475+0000 7fe8f09bd700 1 --2- 172.21.15.107:0/1143524838 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe8dc0a2cb0 0x7fe8dc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:19.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.475+0000 7fe8e9ffb700 1 -- 172.21.15.107:0/1143524838 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe8d80ceca0 con 0x7fe8ec101f00 2024-05-09T08:32:19.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.476+0000 7fe8f09bd700 1 --2- 172.21.15.107:0/1143524838 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe8dc0a2cb0 0x7fe8dc0a5170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0005950 tx=0x7fe8e00058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:19.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.480+0000 7fe8e9ffb700 1 -- 172.21.15.107:0/1143524838 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe8d801f020 con 0x7fe8ec101f00 2024-05-09T08:32:19.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.678+0000 7fe8f2c21700 1 -- 172.21.15.107:0/1143524838 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fe8ec199e00 con 0x7fe8ec101f00 2024-05-09T08:32:19.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.679+0000 7fe8e9ffb700 1 -- 172.21.15.107:0/1143524838 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe8d809b850 con 0x7fe8ec101f00 2024-05-09T08:32:19.678 INFO:teuthology.orchestra.run.smithi107.stdout:197568495670 2024-05-09T08:32:19.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.681+0000 7fe8f2c21700 1 -- 172.21.15.107:0/1143524838 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe8dc0a2cb0 msgr2=0x7fe8dc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:19.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.681+0000 7fe8f2c21700 1 --2- 172.21.15.107:0/1143524838 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe8dc0a2cb0 0x7fe8dc0a5170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0005950 tx=0x7fe8e00058e0 comp rx=0 tx=0).stop 2024-05-09T08:32:19.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.681+0000 7fe8f2c21700 1 -- 172.21.15.107:0/1143524838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8ec101f00 msgr2=0x7fe8ec19ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:19.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.681+0000 7fe8f2c21700 1 --2- 172.21.15.107:0/1143524838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8ec101f00 0x7fe8ec19ec50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800b530 tx=0x7fe8d800b8f0 comp rx=0 tx=0).stop 2024-05-09T08:32:19.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.681+0000 7fe8f2c21700 1 -- 172.21.15.107:0/1143524838 shutdown_connections 2024-05-09T08:32:19.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.681+0000 7fe8f2c21700 1 --2- 172.21.15.107:0/1143524838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8ec107430 0x7fe8ec1988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:19.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.681+0000 7fe8f2c21700 1 --2- 172.21.15.107:0/1143524838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8ec101540 0x7fe8ec19e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:19.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.681+0000 7fe8f2c21700 1 --2- 172.21.15.107:0/1143524838 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe8dc0a2cb0 0x7fe8dc0a5170 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:19.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.681+0000 7fe8f2c21700 1 --2- 172.21.15.107:0/1143524838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8ec101f00 0x7fe8ec19ec50 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:19.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.681+0000 7fe8f2c21700 1 -- 172.21.15.107:0/1143524838 >> 172.21.15.107:0/1143524838 conn(0x7fe8ec077f80 msgr2=0x7fe8ec06ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:19.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.681+0000 7fe8f2c21700 1 -- 172.21.15.107:0/1143524838 shutdown_connections 2024-05-09T08:32:19.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:19.681+0000 7fe8f2c21700 1 -- 172.21.15.107:0/1143524838 wait complete. 2024-05-09T08:32:20.800 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:20 smithi107 ceph-mon[30122]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T08:32:20.800 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:20 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1143524838' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-09T08:32:20.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.803+0000 7f3b71e88700 1 -- 172.21.15.107:0/3730972697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b6c102fb0 msgr2=0x7f3b6c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.803+0000 7f3b71e88700 1 --2- 172.21.15.107:0/3730972697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b6c102fb0 0x7f3b6c106ff0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3b600099c0 tx=0x7f3b60009cd0 comp rx=0 tx=0).stop 2024-05-09T08:32:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.803+0000 7f3b71e88700 1 -- 172.21.15.107:0/3730972697 shutdown_connections 2024-05-09T08:32:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.803+0000 7f3b71e88700 1 --2- 172.21.15.107:0/3730972697 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b6c1076d0 0x7f3b6c10f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.803+0000 7f3b71e88700 1 --2- 172.21.15.107:0/3730972697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b6c102600 0x7f3b6c1029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.803+0000 7f3b71e88700 1 --2- 172.21.15.107:0/3730972697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b6c102fb0 0x7f3b6c106ff0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.803+0000 7f3b71e88700 1 -- 172.21.15.107:0/3730972697 >> 172.21.15.107:0/3730972697 conn(0x7f3b6c0794d0 msgr2=0x7f3b6c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.803+0000 7f3b71e88700 1 -- 172.21.15.107:0/3730972697 shutdown_connections 2024-05-09T08:32:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.803+0000 7f3b71e88700 1 -- 172.21.15.107:0/3730972697 wait complete. 2024-05-09T08:32:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b71e88700 1 Processor -- start 2024-05-09T08:32:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b71e88700 1 -- start start 2024-05-09T08:32:20.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b71e88700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b6c102600 0x7f3b6c19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:20.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b71e88700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b6c102fb0 0x7f3b6c19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:20.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b71e88700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b6c1076d0 0x7f3b6c1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:20.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b71e88700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6c112540 con 0x7f3b6c102600 2024-05-09T08:32:20.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b71e88700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6c1123c0 con 0x7f3b6c1076d0 2024-05-09T08:32:20.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b71e88700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6c1126c0 con 0x7f3b6c102fb0 2024-05-09T08:32:20.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b6b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b6c102600 0x7f3b6c19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:20.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b6b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b6c102600 0x7f3b6c19e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36270/0 (socket says 172.21.15.107:36270) 2024-05-09T08:32:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b6b7fe700 1 -- 172.21.15.107:0/1611242419 learned_addr learned my addr 172.21.15.107:0/1611242419 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b6bfff700 1 --2- 172.21.15.107:0/1611242419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b6c1076d0 0x7f3b6c1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b6affd700 1 --2- 172.21.15.107:0/1611242419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b6c102fb0 0x7f3b6c19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b6b7fe700 1 -- 172.21.15.107:0/1611242419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b6c102fb0 msgr2=0x7f3b6c19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b6b7fe700 1 --2- 172.21.15.107:0/1611242419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b6c102fb0 0x7f3b6c19ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b6b7fe700 1 -- 172.21.15.107:0/1611242419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b6c1076d0 msgr2=0x7f3b6c1988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.804+0000 7f3b6b7fe700 1 --2- 172.21.15.107:0/1611242419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b6c1076d0 0x7f3b6c1988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.805+0000 7f3b6b7fe700 1 -- 172.21.15.107:0/1611242419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b6c199020 con 0x7f3b6c102600 2024-05-09T08:32:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.805+0000 7f3b6b7fe700 1 --2- 172.21.15.107:0/1611242419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b6c102600 0x7f3b6c19e700 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00d980 tx=0x7f3b5c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.805+0000 7f3b68ff9700 1 -- 172.21.15.107:0/1611242419 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c016070 con 0x7f3b6c102600 2024-05-09T08:32:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.805+0000 7f3b68ff9700 1 -- 172.21.15.107:0/1611242419 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b5c004830 con 0x7f3b6c102600 2024-05-09T08:32:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.805+0000 7f3b68ff9700 1 -- 172.21.15.107:0/1611242419 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c005530 con 0x7f3b6c102600 2024-05-09T08:32:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.805+0000 7f3b71e88700 1 -- 172.21.15.107:0/1611242419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b6c199310 con 0x7f3b6c102600 2024-05-09T08:32:20.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.805+0000 7f3b71e88700 1 -- 172.21.15.107:0/1611242419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b6c1a55d0 con 0x7f3b6c102600 2024-05-09T08:32:20.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.806+0000 7f3b71e88700 1 -- 172.21.15.107:0/1611242419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b6c103ad0 con 0x7f3b6c102600 2024-05-09T08:32:20.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.807+0000 7f3b68ff9700 1 -- 172.21.15.107:0/1611242419 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f3b5c0049e0 con 0x7f3b6c102600 2024-05-09T08:32:20.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.808+0000 7f3b68ff9700 1 --2- 172.21.15.107:0/1611242419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3b540a2cb0 0x7f3b540a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:20.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.808+0000 7f3b6affd700 1 --2- 172.21.15.107:0/1611242419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3b540a2cb0 0x7f3b540a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:20.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.808+0000 7f3b68ff9700 1 -- 172.21.15.107:0/1611242419 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3b5c0cf250 con 0x7f3b6c102600 2024-05-09T08:32:20.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.808+0000 7f3b6affd700 1 --2- 172.21.15.107:0/1611242419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3b540a2cb0 0x7f3b540a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3b60000c00 tx=0x7f3b60005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:20.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:20.812+0000 7f3b68ff9700 1 -- 172.21.15.107:0/1611242419 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3b5c01f020 con 0x7f3b6c102600 2024-05-09T08:32:20.815 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:20.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:20 smithi157 ceph-mon[33614]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T08:32:20.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:20 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1143524838' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-09T08:32:20.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:20 smithi110 ceph-mon[33622]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T08:32:20.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:20 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1143524838' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-09T08:32:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:21.017+0000 7f3b71e88700 1 -- 172.21.15.107:0/1611242419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f3b6c02db80 con 0x7f3b6c102600 2024-05-09T08:32:21.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:21.017+0000 7f3b68ff9700 1 -- 172.21.15.107:0/1611242419 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3b5c022350 con 0x7f3b6c102600 2024-05-09T08:32:21.017 INFO:teuthology.orchestra.run.smithi107.stdout:287762808871 2024-05-09T08:32:21.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:21.020+0000 7f3b71e88700 1 -- 172.21.15.107:0/1611242419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3b540a2cb0 msgr2=0x7f3b540a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:21.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:21.020+0000 7f3b71e88700 1 --2- 172.21.15.107:0/1611242419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3b540a2cb0 0x7f3b540a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3b60000c00 tx=0x7f3b60005fb0 comp rx=0 tx=0).stop 2024-05-09T08:32:21.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:21.020+0000 7f3b71e88700 1 -- 172.21.15.107:0/1611242419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b6c102600 msgr2=0x7f3b6c19e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:21.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:21.020+0000 7f3b71e88700 1 --2- 172.21.15.107:0/1611242419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b6c102600 0x7f3b6c19e700 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00d980 tx=0x7f3b5c00dc90 comp rx=0 tx=0).stop 2024-05-09T08:32:21.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:21.020+0000 7f3b71e88700 1 -- 172.21.15.107:0/1611242419 shutdown_connections 2024-05-09T08:32:21.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:21.020+0000 7f3b71e88700 1 --2- 172.21.15.107:0/1611242419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b6c1076d0 0x7f3b6c1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:21.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:21.020+0000 7f3b71e88700 1 --2- 172.21.15.107:0/1611242419 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3b540a2cb0 0x7f3b540a5170 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:21.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:21.020+0000 7f3b71e88700 1 --2- 172.21.15.107:0/1611242419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b6c102600 0x7f3b6c19e700 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:21.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:21.020+0000 7f3b71e88700 1 --2- 172.21.15.107:0/1611242419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b6c102fb0 0x7f3b6c19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:21.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:21.020+0000 7f3b71e88700 1 -- 172.21.15.107:0/1611242419 >> 172.21.15.107:0/1611242419 conn(0x7f3b6c0794d0 msgr2=0x7f3b6c100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:21.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:21.020+0000 7f3b71e88700 1 -- 172.21.15.107:0/1611242419 shutdown_connections 2024-05-09T08:32:21.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:21.020+0000 7f3b71e88700 1 -- 172.21.15.107:0/1611242419 wait complete. 2024-05-09T08:32:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:21 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1611242419' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-09T08:32:21.827 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495664 got 197568495670 for osd.6 2024-05-09T08:32:21.828 DEBUG:teuthology.parallel:result is None 2024-05-09T08:32:21.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:21 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1611242419' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-09T08:32:21.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:21 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1611242419' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-09T08:32:22.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:22 smithi157 ceph-mon[33614]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-09T08:32:22.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:22 smithi110 ceph-mon[33622]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-09T08:32:22.992 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:22 smithi107 ceph-mon[30122]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-09T08:32:23.357 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411397 got 77309411402 for osd.2 2024-05-09T08:32:23.358 DEBUG:teuthology.parallel:result is None 2024-05-09T08:32:23.833 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:23 smithi107 ceph-mon[30122]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:32:23.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:23 smithi110 ceph-mon[33622]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:32:24.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:23 smithi157 ceph-mon[33614]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:32:24.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:24 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1983880703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-09T08:32:24.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.791+0000 7fe5d2de8700 1 -- 172.21.15.107:0/1033159556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5cc1014b0 msgr2=0x7fe5cc101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:24.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.791+0000 7fe5d2de8700 1 --2- 172.21.15.107:0/1033159556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5cc1014b0 0x7fe5cc101890 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe5c00099c0 tx=0x7fe5c0009cd0 comp rx=0 tx=0).stop 2024-05-09T08:32:24.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.791+0000 7fe5d2de8700 1 -- 172.21.15.107:0/1033159556 shutdown_connections 2024-05-09T08:32:24.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.791+0000 7fe5d2de8700 1 --2- 172.21.15.107:0/1033159556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe5cc106540 0x7fe5cc10e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:24.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.791+0000 7fe5d2de8700 1 --2- 172.21.15.107:0/1033159556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5cc1014b0 0x7fe5cc101890 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:24.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.791+0000 7fe5d2de8700 1 --2- 172.21.15.107:0/1033159556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5cc101e60 0x7fe5cc105e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:24.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.791+0000 7fe5d2de8700 1 -- 172.21.15.107:0/1033159556 >> 172.21.15.107:0/1033159556 conn(0x7fe5cc0781d0 msgr2=0x7fe5cc0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:24.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.791+0000 7fe5d2de8700 1 -- 172.21.15.107:0/1033159556 shutdown_connections 2024-05-09T08:32:24.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.791+0000 7fe5d2de8700 1 -- 172.21.15.107:0/1033159556 wait complete. 2024-05-09T08:32:24.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d2de8700 1 Processor -- start 2024-05-09T08:32:24.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d2de8700 1 -- start start 2024-05-09T08:32:24.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d2de8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5cc1014b0 0x7fe5cc19d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:24.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d2de8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe5cc101e60 0x7fe5cc19dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:24.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d2de8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5cc106540 0x7fe5cc197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:24.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d2de8700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc111400 con 0x7fe5cc106540 2024-05-09T08:32:24.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d2de8700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc111280 con 0x7fe5cc1014b0 2024-05-09T08:32:24.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d2de8700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc111580 con 0x7fe5cc101e60 2024-05-09T08:32:24.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d1385700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5cc106540 0x7fe5cc197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:24.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d1385700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5cc106540 0x7fe5cc197800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36326/0 (socket says 172.21.15.107:36326) 2024-05-09T08:32:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d1385700 1 -- 172.21.15.107:0/1361943264 learned_addr learned my addr 172.21.15.107:0/1361943264 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d0b84700 1 --2- 172.21.15.107:0/1361943264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5cc1014b0 0x7fe5cc19d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5cbfff700 1 --2- 172.21.15.107:0/1361943264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe5cc101e60 0x7fe5cc19dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d1385700 1 -- 172.21.15.107:0/1361943264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe5cc101e60 msgr2=0x7fe5cc19dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d1385700 1 --2- 172.21.15.107:0/1361943264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe5cc101e60 0x7fe5cc19dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d1385700 1 -- 172.21.15.107:0/1361943264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5cc1014b0 msgr2=0x7fe5cc19d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d1385700 1 --2- 172.21.15.107:0/1361943264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5cc1014b0 0x7fe5cc19d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d1385700 1 -- 172.21.15.107:0/1361943264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5cc197fe0 con 0x7fe5cc106540 2024-05-09T08:32:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.792+0000 7fe5d0b84700 1 --2- 172.21.15.107:0/1361943264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5cc1014b0 0x7fe5cc19d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:32:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.793+0000 7fe5d1385700 1 --2- 172.21.15.107:0/1361943264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5cc106540 0x7fe5cc197800 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4007ee0 tx=0x7fe5c400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.793+0000 7fe5c9ffb700 1 -- 172.21.15.107:0/1361943264 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c4004bb0 con 0x7fe5cc106540 2024-05-09T08:32:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.793+0000 7fe5c9ffb700 1 -- 172.21.15.107:0/1361943264 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5c4004d50 con 0x7fe5cc106540 2024-05-09T08:32:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.793+0000 7fe5d2de8700 1 -- 172.21.15.107:0/1361943264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5cc1982d0 con 0x7fe5cc106540 2024-05-09T08:32:24.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.793+0000 7fe5c9ffb700 1 -- 172.21.15.107:0/1361943264 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c40058e0 con 0x7fe5cc106540 2024-05-09T08:32:24.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.793+0000 7fe5d2de8700 1 -- 172.21.15.107:0/1361943264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5cc1a44e0 con 0x7fe5cc106540 2024-05-09T08:32:24.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.794+0000 7fe5d2de8700 1 -- 172.21.15.107:0/1361943264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5cc102940 con 0x7fe5cc106540 2024-05-09T08:32:24.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.795+0000 7fe5c9ffb700 1 -- 172.21.15.107:0/1361943264 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fe5c401a020 con 0x7fe5cc106540 2024-05-09T08:32:24.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.796+0000 7fe5c9ffb700 1 --2- 172.21.15.107:0/1361943264 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe5bc0a2cb0 0x7fe5bc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:24.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.796+0000 7fe5d0b84700 1 --2- 172.21.15.107:0/1361943264 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe5bc0a2cb0 0x7fe5bc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:24.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.796+0000 7fe5c9ffb700 1 -- 172.21.15.107:0/1361943264 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe5c40ceb80 con 0x7fe5cc106540 2024-05-09T08:32:24.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.796+0000 7fe5d0b84700 1 --2- 172.21.15.107:0/1361943264 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe5bc0a2cb0 0x7fe5bc0a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0000c00 tx=0x7fe5c0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:24.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.800+0000 7fe5c9ffb700 1 -- 172.21.15.107:0/1361943264 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe5c4010070 con 0x7fe5cc106540 2024-05-09T08:32:24.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:24 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1983880703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-09T08:32:24.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.267+0000 7f1e339f9700 1 -- 172.21.15.107:0/3598339362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e2c101bd0 msgr2=0x7f1e2c10e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:24.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.267+0000 7f1e339f9700 1 --2- 172.21.15.107:0/3598339362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e2c101bd0 0x7f1e2c10e0c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1e2400b0a0 tx=0x7f1e2400b3b0 comp rx=0 tx=0).stop 2024-05-09T08:32:24.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.267+0000 7f1e339f9700 1 -- 172.21.15.107:0/3598339362 shutdown_connections 2024-05-09T08:32:24.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.267+0000 7f1e339f9700 1 --2- 172.21.15.107:0/3598339362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e2c101bd0 0x7f1e2c10e0c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:24.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.267+0000 7f1e339f9700 1 --2- 172.21.15.107:0/3598339362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e2c107280 0x7f1e2c107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:24.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.267+0000 7f1e339f9700 1 --2- 172.21.15.107:0/3598339362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e2c101210 0x7f1e2c101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:24.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.267+0000 7f1e339f9700 1 -- 172.21.15.107:0/3598339362 >> 172.21.15.107:0/3598339362 conn(0x7f1e2c078190 msgr2=0x7f1e2c0785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:24.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.268+0000 7f1e339f9700 1 -- 172.21.15.107:0/3598339362 shutdown_connections 2024-05-09T08:32:24.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.268+0000 7f1e339f9700 1 -- 172.21.15.107:0/3598339362 wait complete. 2024-05-09T08:32:24.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.268+0000 7f1e339f9700 1 Processor -- start 2024-05-09T08:32:24.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.268+0000 7f1e339f9700 1 -- start start 2024-05-09T08:32:24.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.268+0000 7f1e339f9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e2c101210 0x7f1e2c1997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:24.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.268+0000 7f1e339f9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e2c101bd0 0x7f1e2c199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:24.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.268+0000 7f1e339f9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e2c107280 0x7f1e2c1a4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:24.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.268+0000 7f1e339f9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c1113a0 con 0x7f1e2c101bd0 2024-05-09T08:32:24.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.268+0000 7f1e339f9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c111220 con 0x7f1e2c101210 2024-05-09T08:32:24.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.268+0000 7f1e339f9700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c111520 con 0x7f1e2c107280 2024-05-09T08:32:24.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.269+0000 7f1e30f94700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e2c101bd0 0x7f1e2c199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:24.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.269+0000 7f1e30f94700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e2c101bd0 0x7f1e2c199ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36300/0 (socket says 172.21.15.107:36300) 2024-05-09T08:32:24.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.269+0000 7f1e30f94700 1 -- 172.21.15.107:0/1983880703 learned_addr learned my addr 172.21.15.107:0/1983880703 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:24.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.269+0000 7f1e31795700 1 --2- 172.21.15.107:0/1983880703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e2c101210 0x7f1e2c1997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:24.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.269+0000 7f1e31f96700 1 --2- 172.21.15.107:0/1983880703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e2c107280 0x7f1e2c1a4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:24.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.269+0000 7f1e30f94700 1 -- 172.21.15.107:0/1983880703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e2c107280 msgr2=0x7f1e2c1a4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:24.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.269+0000 7f1e30f94700 1 --2- 172.21.15.107:0/1983880703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e2c107280 0x7f1e2c1a4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:24.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.269+0000 7f1e30f94700 1 -- 172.21.15.107:0/1983880703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e2c101210 msgr2=0x7f1e2c1997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:24.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.269+0000 7f1e30f94700 1 --2- 172.21.15.107:0/1983880703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e2c101210 0x7f1e2c1997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:24.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.269+0000 7f1e30f94700 1 -- 172.21.15.107:0/1983880703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e2c1a4d60 con 0x7f1e2c101bd0 2024-05-09T08:32:24.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.269+0000 7f1e30f94700 1 --2- 172.21.15.107:0/1983880703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e2c101bd0 0x7f1e2c199ce0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00d950 tx=0x7f1e1c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:24.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.269+0000 7f1e227fc700 1 -- 172.21.15.107:0/1983880703 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c016070 con 0x7f1e2c101bd0 2024-05-09T08:32:24.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.269+0000 7f1e227fc700 1 -- 172.21.15.107:0/1983880703 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e1c00fd40 con 0x7f1e2c101bd0 2024-05-09T08:32:24.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.269+0000 7f1e339f9700 1 -- 172.21.15.107:0/1983880703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e2c1a4f90 con 0x7f1e2c101bd0 2024-05-09T08:32:24.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.269+0000 7f1e227fc700 1 -- 172.21.15.107:0/1983880703 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c00e3b0 con 0x7f1e2c101bd0 2024-05-09T08:32:24.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.269+0000 7f1e339f9700 1 -- 172.21.15.107:0/1983880703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e2c1a5440 con 0x7f1e2c101bd0 2024-05-09T08:32:24.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.270+0000 7f1e339f9700 1 -- 172.21.15.107:0/1983880703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e2c069010 con 0x7f1e2c101bd0 2024-05-09T08:32:24.892 INFO:teuthology.orchestra.run.smithi107.stdout:343597383710 2024-05-09T08:32:24.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.272+0000 7f1e227fc700 1 -- 172.21.15.107:0/1983880703 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f1e1c019070 con 0x7f1e2c101bd0 2024-05-09T08:32:24.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.272+0000 7f1e227fc700 1 --2- 172.21.15.107:0/1983880703 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1e180a2cb0 0x7f1e180a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:24.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.272+0000 7f1e227fc700 1 -- 172.21.15.107:0/1983880703 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1e1c0cefa0 con 0x7f1e2c101bd0 2024-05-09T08:32:24.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.272+0000 7f1e31795700 1 --2- 172.21.15.107:0/1983880703 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1e180a2cb0 0x7f1e180a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:24.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.273+0000 7f1e31795700 1 --2- 172.21.15.107:0/1983880703 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1e180a2cb0 0x7f1e180a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1e28005950 tx=0x7f1e2800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:24.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.276+0000 7f1e227fc700 1 -- 172.21.15.107:0/1983880703 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1e1c09ba40 con 0x7f1e2c101bd0 2024-05-09T08:32:24.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.478+0000 7f1e339f9700 1 -- 172.21.15.107:0/1983880703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f1e2c02dde0 con 0x7f1e2c101bd0 2024-05-09T08:32:24.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.478+0000 7f1e227fc700 1 -- 172.21.15.107:0/1983880703 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f1e1c09b340 con 0x7f1e2c101bd0 2024-05-09T08:32:24.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.480+0000 7f1e339f9700 1 -- 172.21.15.107:0/1983880703 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1e180a2cb0 msgr2=0x7f1e180a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:24.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.480+0000 7f1e339f9700 1 --2- 172.21.15.107:0/1983880703 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1e180a2cb0 0x7f1e180a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1e28005950 tx=0x7f1e2800b410 comp rx=0 tx=0).stop 2024-05-09T08:32:24.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.480+0000 7f1e339f9700 1 -- 172.21.15.107:0/1983880703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e2c101bd0 msgr2=0x7f1e2c199ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:24.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.480+0000 7f1e339f9700 1 --2- 172.21.15.107:0/1983880703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e2c101bd0 0x7f1e2c199ce0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00d950 tx=0x7f1e1c00dc60 comp rx=0 tx=0).stop 2024-05-09T08:32:24.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.480+0000 7f1e339f9700 1 -- 172.21.15.107:0/1983880703 shutdown_connections 2024-05-09T08:32:24.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.480+0000 7f1e339f9700 1 --2- 172.21.15.107:0/1983880703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e2c107280 0x7f1e2c1a4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:24.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.480+0000 7f1e339f9700 1 --2- 172.21.15.107:0/1983880703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e2c101210 0x7f1e2c1997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:24.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.480+0000 7f1e339f9700 1 --2- 172.21.15.107:0/1983880703 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f1e180a2cb0 0x7f1e180a5170 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:24.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.480+0000 7f1e339f9700 1 --2- 172.21.15.107:0/1983880703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e2c101bd0 0x7f1e2c199ce0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:24.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.480+0000 7f1e339f9700 1 -- 172.21.15.107:0/1983880703 >> 172.21.15.107:0/1983880703 conn(0x7f1e2c078190 msgr2=0x7f1e2c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:24.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.480+0000 7f1e339f9700 1 -- 172.21.15.107:0/1983880703 shutdown_connections 2024-05-09T08:32:24.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:24.480+0000 7f1e339f9700 1 -- 172.21.15.107:0/1983880703 wait complete. 2024-05-09T08:32:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.000+0000 7fe5d2de8700 1 -- 172.21.15.107:0/1361943264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fe5cc02db80 con 0x7fe5cc106540 2024-05-09T08:32:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.000+0000 7fe5c9ffb700 1 -- 172.21.15.107:0/1361943264 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe5c409b620 con 0x7fe5cc106540 2024-05-09T08:32:24.999 INFO:teuthology.orchestra.run.smithi107.stdout:34359738453 2024-05-09T08:32:25.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.002+0000 7fe5d2de8700 1 -- 172.21.15.107:0/1361943264 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe5bc0a2cb0 msgr2=0x7fe5bc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:25.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.002+0000 7fe5d2de8700 1 --2- 172.21.15.107:0/1361943264 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe5bc0a2cb0 0x7fe5bc0a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0000c00 tx=0x7fe5c0005fb0 comp rx=0 tx=0).stop 2024-05-09T08:32:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.002+0000 7fe5d2de8700 1 -- 172.21.15.107:0/1361943264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5cc106540 msgr2=0x7fe5cc197800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.002+0000 7fe5d2de8700 1 --2- 172.21.15.107:0/1361943264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5cc106540 0x7fe5cc197800 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4007ee0 tx=0x7fe5c400d3b0 comp rx=0 tx=0).stop 2024-05-09T08:32:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.002+0000 7fe5d2de8700 1 -- 172.21.15.107:0/1361943264 shutdown_connections 2024-05-09T08:32:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.002+0000 7fe5d2de8700 1 --2- 172.21.15.107:0/1361943264 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe5bc0a2cb0 0x7fe5bc0a5170 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.002+0000 7fe5d2de8700 1 --2- 172.21.15.107:0/1361943264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5cc106540 0x7fe5cc197800 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.002+0000 7fe5d2de8700 1 --2- 172.21.15.107:0/1361943264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5cc1014b0 0x7fe5cc19d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.002+0000 7fe5d2de8700 1 --2- 172.21.15.107:0/1361943264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe5cc101e60 0x7fe5cc19dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.002+0000 7fe5d2de8700 1 -- 172.21.15.107:0/1361943264 >> 172.21.15.107:0/1361943264 conn(0x7fe5cc0781d0 msgr2=0x7fe5cc0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.002+0000 7fe5d2de8700 1 -- 172.21.15.107:0/1361943264 shutdown_connections 2024-05-09T08:32:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.002+0000 7fe5d2de8700 1 -- 172.21.15.107:0/1361943264 wait complete. 2024-05-09T08:32:25.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:24 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1983880703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-09T08:32:25.695 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:25 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1361943264' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-09T08:32:25.696 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:25 smithi107 ceph-mon[30122]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T08:32:25.701 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:25.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:25 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1361943264' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-09T08:32:25.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:25 smithi110 ceph-mon[33622]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T08:32:25.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.900+0000 7efca09b2700 1 -- 172.21.15.107:0/36164777 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc9c1025d0 msgr2=0x7efc9c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:25.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.900+0000 7efca09b2700 1 --2- 172.21.15.107:0/36164777 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc9c1025d0 0x7efc9c1029b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7efc90009990 tx=0x7efc90009ca0 comp rx=0 tx=0).stop 2024-05-09T08:32:25.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.901+0000 7efca09b2700 1 -- 172.21.15.107:0/36164777 shutdown_connections 2024-05-09T08:32:25.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.901+0000 7efca09b2700 1 --2- 172.21.15.107:0/36164777 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc9c107660 0x7efc9c10f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:25.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.901+0000 7efca09b2700 1 --2- 172.21.15.107:0/36164777 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc9c1025d0 0x7efc9c1029b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:25.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.901+0000 7efca09b2700 1 --2- 172.21.15.107:0/36164777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc9c102f80 0x7efc9c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:25.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.901+0000 7efca09b2700 1 -- 172.21.15.107:0/36164777 >> 172.21.15.107:0/36164777 conn(0x7efc9c079480 msgr2=0x7efc9c079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:25.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.901+0000 7efca09b2700 1 -- 172.21.15.107:0/36164777 shutdown_connections 2024-05-09T08:32:25.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.901+0000 7efca09b2700 1 -- 172.21.15.107:0/36164777 wait complete. 2024-05-09T08:32:25.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.901+0000 7efca09b2700 1 Processor -- start 2024-05-09T08:32:25.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.901+0000 7efca09b2700 1 -- start start 2024-05-09T08:32:25.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efca09b2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc9c1025d0 0x7efc9c074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:25.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efca09b2700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc9c102f80 0x7efc9c074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:25.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efca09b2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc9c107660 0x7efc9c0750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:25.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efca09b2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c112500 con 0x7efc9c1025d0 2024-05-09T08:32:25.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efca09b2700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c112380 con 0x7efc9c107660 2024-05-09T08:32:25.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efca09b2700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c112680 con 0x7efc9c102f80 2024-05-09T08:32:25.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efc9a59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc9c1025d0 0x7efc9c074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:25.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efc9a59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc9c1025d0 0x7efc9c074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36344/0 (socket says 172.21.15.107:36344) 2024-05-09T08:32:25.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efc9a59c700 1 -- 172.21.15.107:0/3100031366 learned_addr learned my addr 172.21.15.107:0/3100031366 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:25.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efc9ad9d700 1 --2- 172.21.15.107:0/3100031366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc9c107660 0x7efc9c0750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:25.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efc99d9b700 1 --2- 172.21.15.107:0/3100031366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc9c102f80 0x7efc9c074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:25.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efc9a59c700 1 -- 172.21.15.107:0/3100031366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc9c102f80 msgr2=0x7efc9c074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:25.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efc9a59c700 1 --2- 172.21.15.107:0/3100031366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc9c102f80 0x7efc9c074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:25.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efc9a59c700 1 -- 172.21.15.107:0/3100031366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc9c107660 msgr2=0x7efc9c0750a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:25.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efc9a59c700 1 --2- 172.21.15.107:0/3100031366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc9c107660 0x7efc9c0750a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:25.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efc9a59c700 1 -- 172.21.15.107:0/3100031366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc9c1a6ee0 con 0x7efc9c1025d0 2024-05-09T08:32:25.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efc9a59c700 1 --2- 172.21.15.107:0/3100031366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc9c1025d0 0x7efc9c074620 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efc90000c00 tx=0x7efc900047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:25.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.902+0000 7efc8b7fe700 1 -- 172.21.15.107:0/3100031366 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90027070 con 0x7efc9c1025d0 2024-05-09T08:32:25.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.903+0000 7efca09b2700 1 -- 172.21.15.107:0/3100031366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc9c1a7170 con 0x7efc9c1025d0 2024-05-09T08:32:25.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.903+0000 7efca09b2700 1 -- 172.21.15.107:0/3100031366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc9c1a7620 con 0x7efc9c1025d0 2024-05-09T08:32:25.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.903+0000 7efc8b7fe700 1 -- 172.21.15.107:0/3100031366 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc90022470 con 0x7efc9c1025d0 2024-05-09T08:32:25.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.903+0000 7efc8b7fe700 1 -- 172.21.15.107:0/3100031366 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90017650 con 0x7efc9c1025d0 2024-05-09T08:32:25.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.904+0000 7efca09b2700 1 -- 172.21.15.107:0/3100031366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc7c005330 con 0x7efc9c1025d0 2024-05-09T08:32:25.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.906+0000 7efc8b7fe700 1 -- 172.21.15.107:0/3100031366 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7efc9002a070 con 0x7efc9c1025d0 2024-05-09T08:32:25.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.906+0000 7efc8b7fe700 1 --2- 172.21.15.107:0/3100031366 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efc840a2ca0 0x7efc840a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:25.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.907+0000 7efc8b7fe700 1 -- 172.21.15.107:0/3100031366 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7efc900cff00 con 0x7efc9c1025d0 2024-05-09T08:32:25.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.907+0000 7efc99d9b700 1 --2- 172.21.15.107:0/3100031366 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efc840a2ca0 0x7efc840a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:25.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.907+0000 7efc99d9b700 1 --2- 172.21.15.107:0/3100031366 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efc840a2ca0 0x7efc840a5160 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efc9c100e60 tx=0x7efc8c00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:25.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:25.911+0000 7efc8b7fe700 1 -- 172.21.15.107:0/3100031366 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efc90013070 con 0x7efc9c1025d0 2024-05-09T08:32:26.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:25 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1361943264' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-09T08:32:26.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:25 smithi157 ceph-mon[33614]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T08:32:26.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:26.116+0000 7efca09b2700 1 -- 172.21.15.107:0/3100031366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7efc7c005620 con 0x7efc9c1025d0 2024-05-09T08:32:26.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:26.117+0000 7efc8b7fe700 1 -- 172.21.15.107:0/3100031366 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7efc9009cb90 con 0x7efc9c1025d0 2024-05-09T08:32:26.116 INFO:teuthology.orchestra.run.smithi107.stdout:163208757309 2024-05-09T08:32:26.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:26.119+0000 7efca09b2700 1 -- 172.21.15.107:0/3100031366 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efc840a2ca0 msgr2=0x7efc840a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:26.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:26.119+0000 7efca09b2700 1 --2- 172.21.15.107:0/3100031366 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efc840a2ca0 0x7efc840a5160 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efc9c100e60 tx=0x7efc8c00a380 comp rx=0 tx=0).stop 2024-05-09T08:32:26.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:26.119+0000 7efca09b2700 1 -- 172.21.15.107:0/3100031366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc9c1025d0 msgr2=0x7efc9c074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:26.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:26.119+0000 7efca09b2700 1 --2- 172.21.15.107:0/3100031366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc9c1025d0 0x7efc9c074620 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efc90000c00 tx=0x7efc900047b0 comp rx=0 tx=0).stop 2024-05-09T08:32:26.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:26.119+0000 7efca09b2700 1 -- 172.21.15.107:0/3100031366 shutdown_connections 2024-05-09T08:32:26.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:26.119+0000 7efca09b2700 1 --2- 172.21.15.107:0/3100031366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc9c107660 0x7efc9c0750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:26.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:26.119+0000 7efca09b2700 1 --2- 172.21.15.107:0/3100031366 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efc840a2ca0 0x7efc840a5160 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:26.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:26.119+0000 7efca09b2700 1 --2- 172.21.15.107:0/3100031366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc9c1025d0 0x7efc9c074620 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:26.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:26.119+0000 7efca09b2700 1 --2- 172.21.15.107:0/3100031366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc9c102f80 0x7efc9c074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:26.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:26.119+0000 7efca09b2700 1 -- 172.21.15.107:0/3100031366 >> 172.21.15.107:0/3100031366 conn(0x7efc9c079480 msgr2=0x7efc9c101b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:26.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:26.119+0000 7efca09b2700 1 -- 172.21.15.107:0/3100031366 shutdown_connections 2024-05-09T08:32:26.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:26.119+0000 7efca09b2700 1 -- 172.21.15.107:0/3100031366 wait complete. 2024-05-09T08:32:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:26 smithi107 ceph-mon[30122]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-05-09T08:32:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:26 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3100031366' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-09T08:32:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:26 smithi110 ceph-mon[33622]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-05-09T08:32:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:26 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3100031366' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-09T08:32:27.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:26 smithi157 ceph-mon[33614]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-05-09T08:32:27.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:26 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3100031366' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-09T08:32:27.564 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808866 got 287762808871 for osd.9 2024-05-09T08:32:27.564 DEBUG:teuthology.parallel:result is None 2024-05-09T08:32:27.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:27 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1978356394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-09T08:32:27.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:27 smithi107 ceph-mon[30122]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-09T08:32:28.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:27 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1978356394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-09T08:32:28.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:27 smithi157 ceph-mon[33614]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-09T08:32:28.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:27 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1978356394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-09T08:32:28.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:27 smithi110 ceph-mon[33622]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-09T08:32:28.435 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383706 got 343597383710 for osd.11 2024-05-09T08:32:28.435 DEBUG:teuthology.parallel:result is None 2024-05-09T08:32:29.316 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738447 got 34359738453 for osd.0 2024-05-09T08:32:29.317 DEBUG:teuthology.parallel:result is None 2024-05-09T08:32:29.440 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757301 got 163208757309 for osd.5 2024-05-09T08:32:29.440 DEBUG:teuthology.parallel:result is None 2024-05-09T08:32:29.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.170+0000 7f3c4e69c700 1 -- 172.21.15.107:0/2631272030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c481012b0 msgr2=0x7f3c4810f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:29.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.170+0000 7f3c4e69c700 1 --2- 172.21.15.107:0/2631272030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c481012b0 0x7f3c4810f1f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000b0a0 tx=0x7f3c4000b3b0 comp rx=0 tx=0).stop 2024-05-09T08:32:29.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.170+0000 7f3c4e69c700 1 -- 172.21.15.107:0/2631272030 shutdown_connections 2024-05-09T08:32:29.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.170+0000 7f3c4e69c700 1 --2- 172.21.15.107:0/2631272030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c481012b0 0x7f3c4810f1f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:29.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.170+0000 7f3c4e69c700 1 --2- 172.21.15.107:0/2631272030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c481083b0 0x7f3c48108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:29.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.170+0000 7f3c4e69c700 1 --2- 172.21.15.107:0/2631272030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c48100910 0x7f3c48100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:29.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.170+0000 7f3c4e69c700 1 -- 172.21.15.107:0/2631272030 >> 172.21.15.107:0/2631272030 conn(0x7f3c48077890 msgr2=0x7f3c480fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:29.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.170+0000 7f3c4e69c700 1 -- 172.21.15.107:0/2631272030 shutdown_connections 2024-05-09T08:32:29.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.170+0000 7f3c4e69c700 1 -- 172.21.15.107:0/2631272030 wait complete. 2024-05-09T08:32:29.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c4e69c700 1 Processor -- start 2024-05-09T08:32:29.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c4e69c700 1 -- start start 2024-05-09T08:32:29.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c4e69c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c48100910 0x7f3c4819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:29.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c4e69c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c481012b0 0x7f3c4819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:29.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c4e69c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c481083b0 0x7f3c481987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:29.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c4e69c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3c48112530 con 0x7f3c481083b0 2024-05-09T08:32:29.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c4e69c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3c481123b0 con 0x7f3c48100910 2024-05-09T08:32:29.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c4e69c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3c481126b0 con 0x7f3c481012b0 2024-05-09T08:32:29.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c4cc39700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c481083b0 0x7f3c481987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:29.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c47fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c48100910 0x7f3c4819e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:29.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c4cc39700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c481083b0 0x7f3c481987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36366/0 (socket says 172.21.15.107:36366) 2024-05-09T08:32:29.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c4cc39700 1 -- 172.21.15.107:0/1978356394 learned_addr learned my addr 172.21.15.107:0/1978356394 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:29.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c477fe700 1 --2- 172.21.15.107:0/1978356394 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c481012b0 0x7f3c4819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:29.743 INFO:teuthology.orchestra.run.smithi107.stdout:107374182470 2024-05-09T08:32:29.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c4cc39700 1 -- 172.21.15.107:0/1978356394 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c481012b0 msgr2=0x7f3c4819ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:29.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c4cc39700 1 --2- 172.21.15.107:0/1978356394 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c481012b0 0x7f3c4819ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:29.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c4cc39700 1 -- 172.21.15.107:0/1978356394 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c48100910 msgr2=0x7f3c4819e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:29.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c4cc39700 1 --2- 172.21.15.107:0/1978356394 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c48100910 0x7f3c4819e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:29.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.171+0000 7f3c4cc39700 1 -- 172.21.15.107:0/1978356394 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c48198fd0 con 0x7f3c481083b0 2024-05-09T08:32:29.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.172+0000 7f3c4cc39700 1 --2- 172.21.15.107:0/1978356394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c481083b0 0x7f3c481987f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000b070 tx=0x7f3c40004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:29.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.172+0000 7f3c457fa700 1 -- 172.21.15.107:0/1978356394 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c40021070 con 0x7f3c481083b0 2024-05-09T08:32:29.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.172+0000 7f3c4e69c700 1 -- 172.21.15.107:0/1978356394 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c48199260 con 0x7f3c481083b0 2024-05-09T08:32:29.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.172+0000 7f3c457fa700 1 -- 172.21.15.107:0/1978356394 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3c40007e30 con 0x7f3c481083b0 2024-05-09T08:32:29.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.172+0000 7f3c457fa700 1 -- 172.21.15.107:0/1978356394 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c400046d0 con 0x7f3c481083b0 2024-05-09T08:32:29.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.172+0000 7f3c4e69c700 1 -- 172.21.15.107:0/1978356394 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c481a54d0 con 0x7f3c481083b0 2024-05-09T08:32:29.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.173+0000 7f3c4e69c700 1 -- 172.21.15.107:0/1978356394 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c48103c30 con 0x7f3c481083b0 2024-05-09T08:32:29.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.174+0000 7f3c457fa700 1 -- 172.21.15.107:0/1978356394 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f3c40007980 con 0x7f3c481083b0 2024-05-09T08:32:29.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.175+0000 7f3c457fa700 1 --2- 172.21.15.107:0/1978356394 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3c300a2c20 0x7f3c300a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:29.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.175+0000 7f3c47fff700 1 --2- 172.21.15.107:0/1978356394 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3c300a2c20 0x7f3c300a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:29.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.175+0000 7f3c457fa700 1 -- 172.21.15.107:0/1978356394 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3c4000f020 con 0x7f3c481083b0 2024-05-09T08:32:29.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.175+0000 7f3c47fff700 1 --2- 172.21.15.107:0/1978356394 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3c300a2c20 0x7f3c300a50e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3c38009be0 tx=0x7f3c38009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:29.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.178+0000 7f3c457fa700 1 -- 172.21.15.107:0/1978356394 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3c4002a020 con 0x7f3c481083b0 2024-05-09T08:32:29.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.378+0000 7f3c4e69c700 1 -- 172.21.15.107:0/1978356394 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f3c48199d20 con 0x7f3c481083b0 2024-05-09T08:32:29.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.379+0000 7f3c457fa700 1 -- 172.21.15.107:0/1978356394 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3c40003c00 con 0x7f3c481083b0 2024-05-09T08:32:29.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.381+0000 7f3c4e69c700 1 -- 172.21.15.107:0/1978356394 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3c300a2c20 msgr2=0x7f3c300a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:29.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.381+0000 7f3c4e69c700 1 --2- 172.21.15.107:0/1978356394 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3c300a2c20 0x7f3c300a50e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3c38009be0 tx=0x7f3c38009260 comp rx=0 tx=0).stop 2024-05-09T08:32:29.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.381+0000 7f3c4e69c700 1 -- 172.21.15.107:0/1978356394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c481083b0 msgr2=0x7f3c481987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:29.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.381+0000 7f3c4e69c700 1 --2- 172.21.15.107:0/1978356394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c481083b0 0x7f3c481987f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000b070 tx=0x7f3c40004cf0 comp rx=0 tx=0).stop 2024-05-09T08:32:29.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.381+0000 7f3c4e69c700 1 -- 172.21.15.107:0/1978356394 shutdown_connections 2024-05-09T08:32:29.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.381+0000 7f3c4e69c700 1 --2- 172.21.15.107:0/1978356394 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3c300a2c20 0x7f3c300a50e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:29.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.381+0000 7f3c4e69c700 1 --2- 172.21.15.107:0/1978356394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c481083b0 0x7f3c481987f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:29.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.381+0000 7f3c4e69c700 1 --2- 172.21.15.107:0/1978356394 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c48100910 0x7f3c4819e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:29.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.381+0000 7f3c4e69c700 1 --2- 172.21.15.107:0/1978356394 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c481012b0 0x7f3c4819ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:29.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.381+0000 7f3c4e69c700 1 -- 172.21.15.107:0/1978356394 >> 172.21.15.107:0/1978356394 conn(0x7f3c48077890 msgr2=0x7f3c480ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:29.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.381+0000 7f3c4e69c700 1 -- 172.21.15.107:0/1978356394 shutdown_connections 2024-05-09T08:32:29.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:27.381+0000 7f3c4e69c700 1 -- 172.21.15.107:0/1978356394 wait complete. 2024-05-09T08:32:30.394 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182464 got 107374182470 for osd.3 2024-05-09T08:32:30.394 DEBUG:teuthology.parallel:result is None 2024-05-09T08:32:30.649 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:30 smithi107 ceph-mon[30122]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:32:30.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:30 smithi157 ceph-mon[33614]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:32:30.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:30 smithi110 ceph-mon[33622]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:32:31.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.957+0000 7f1453482700 1 -- 172.21.15.107:0/934994408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f144c100cc0 msgr2=0x7f144c101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:31.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.957+0000 7f1453482700 1 --2- 172.21.15.107:0/934994408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f144c100cc0 0x7f144c101140 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f143c0099c0 tx=0x7f143c009cd0 comp rx=0 tx=0).stop 2024-05-09T08:32:31.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.957+0000 7f1453482700 1 -- 172.21.15.107:0/934994408 shutdown_connections 2024-05-09T08:32:31.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.957+0000 7f1453482700 1 --2- 172.21.15.107:0/934994408 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f144c101680 0x7f144c10f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:31.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.957+0000 7f1453482700 1 --2- 172.21.15.107:0/934994408 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f144c107390 0x7f144c107770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:31.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.957+0000 7f1453482700 1 --2- 172.21.15.107:0/934994408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f144c100cc0 0x7f144c101140 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:31.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.957+0000 7f1453482700 1 -- 172.21.15.107:0/934994408 >> 172.21.15.107:0/934994408 conn(0x7f144c0765c0 msgr2=0x7f144c0769d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:31.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.957+0000 7f1453482700 1 -- 172.21.15.107:0/934994408 shutdown_connections 2024-05-09T08:32:31.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.957+0000 7f1453482700 1 -- 172.21.15.107:0/934994408 wait complete. 2024-05-09T08:32:31.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.957+0000 7f1453482700 1 Processor -- start 2024-05-09T08:32:31.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f1453482700 1 -- start start 2024-05-09T08:32:31.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f1453482700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f144c100cc0 0x7f144c19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:31.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f1453482700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f144c101680 0x7f144c19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:31.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f1453482700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f144c107390 0x7f144c198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:31.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f1453482700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f144c112550 con 0x7f144c100cc0 2024-05-09T08:32:31.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f1453482700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f144c1123d0 con 0x7f144c101680 2024-05-09T08:32:31.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f1453482700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f144c1126d0 con 0x7f144c107390 2024-05-09T08:32:31.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f145121e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f144c100cc0 0x7f144c19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:31.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f145121e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f144c100cc0 0x7f144c19e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50202/0 (socket says 172.21.15.107:50202) 2024-05-09T08:32:31.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f145121e700 1 -- 172.21.15.107:0/3426816569 learned_addr learned my addr 172.21.15.107:0/3426816569 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:31.279 INFO:teuthology.orchestra.run.smithi107.stdout:219043332147 2024-05-09T08:32:31.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f1450a1d700 1 --2- 172.21.15.107:0/3426816569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f144c101680 0x7f144c19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:31.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f1451a1f700 1 --2- 172.21.15.107:0/3426816569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f144c107390 0x7f144c198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:31.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f145121e700 1 -- 172.21.15.107:0/3426816569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f144c107390 msgr2=0x7f144c198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:31.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f145121e700 1 --2- 172.21.15.107:0/3426816569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f144c107390 0x7f144c198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:31.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f145121e700 1 -- 172.21.15.107:0/3426816569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f144c101680 msgr2=0x7f144c19eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:31.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f145121e700 1 --2- 172.21.15.107:0/3426816569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f144c101680 0x7f144c19eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:31.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f145121e700 1 -- 172.21.15.107:0/3426816569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144c198f70 con 0x7f144c100cc0 2024-05-09T08:32:31.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f145121e700 1 --2- 172.21.15.107:0/3426816569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f144c100cc0 0x7f144c19e610 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f144800b530 tx=0x7f144800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:31.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.958+0000 7f14427fc700 1 -- 172.21.15.107:0/3426816569 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1448015020 con 0x7f144c100cc0 2024-05-09T08:32:31.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.959+0000 7f14427fc700 1 -- 172.21.15.107:0/3426816569 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1448011ce0 con 0x7f144c100cc0 2024-05-09T08:32:31.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.959+0000 7f1453482700 1 -- 172.21.15.107:0/3426816569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f144c199260 con 0x7f144c100cc0 2024-05-09T08:32:31.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.959+0000 7f14427fc700 1 -- 172.21.15.107:0/3426816569 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1448010e40 con 0x7f144c100cc0 2024-05-09T08:32:31.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.959+0000 7f1453482700 1 -- 172.21.15.107:0/3426816569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f144c1a5470 con 0x7f144c100cc0 2024-05-09T08:32:31.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.959+0000 7f1453482700 1 -- 172.21.15.107:0/3426816569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f144c10a940 con 0x7f144c100cc0 2024-05-09T08:32:31.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.961+0000 7f14427fc700 1 -- 172.21.15.107:0/3426816569 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f1448018020 con 0x7f144c100cc0 2024-05-09T08:32:31.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.961+0000 7f14427fc700 1 --2- 172.21.15.107:0/3426816569 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f14380a2c20 0x7f14380a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:31.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.961+0000 7f1450a1d700 1 --2- 172.21.15.107:0/3426816569 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f14380a2c20 0x7f14380a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:31.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.961+0000 7f14427fc700 1 -- 172.21.15.107:0/3426816569 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f14480cf1d0 con 0x7f144c100cc0 2024-05-09T08:32:31.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.962+0000 7f1450a1d700 1 --2- 172.21.15.107:0/3426816569 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f14380a2c20 0x7f14380a50e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f144c076bb0 tx=0x7f143c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:31.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:30.965+0000 7f14427fc700 1 -- 172.21.15.107:0/3426816569 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f144801f020 con 0x7f144c100cc0 2024-05-09T08:32:31.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:31.164+0000 7f1453482700 1 -- 172.21.15.107:0/3426816569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f144c06a490 con 0x7f144c100cc0 2024-05-09T08:32:31.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:31.164+0000 7f14427fc700 1 -- 172.21.15.107:0/3426816569 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1448010380 con 0x7f144c100cc0 2024-05-09T08:32:31.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:31.167+0000 7f1453482700 1 -- 172.21.15.107:0/3426816569 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f14380a2c20 msgr2=0x7f14380a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:31.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:31.167+0000 7f1453482700 1 --2- 172.21.15.107:0/3426816569 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f14380a2c20 0x7f14380a50e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f144c076bb0 tx=0x7f143c005fb0 comp rx=0 tx=0).stop 2024-05-09T08:32:31.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:31.167+0000 7f1453482700 1 -- 172.21.15.107:0/3426816569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f144c100cc0 msgr2=0x7f144c19e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:31.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:31.167+0000 7f1453482700 1 --2- 172.21.15.107:0/3426816569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f144c100cc0 0x7f144c19e610 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f144800b530 tx=0x7f144800b840 comp rx=0 tx=0).stop 2024-05-09T08:32:31.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:31.167+0000 7f1453482700 1 -- 172.21.15.107:0/3426816569 shutdown_connections 2024-05-09T08:32:31.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:31.167+0000 7f1453482700 1 --2- 172.21.15.107:0/3426816569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f144c107390 0x7f144c198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:31.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:31.167+0000 7f1453482700 1 --2- 172.21.15.107:0/3426816569 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f14380a2c20 0x7f14380a50e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:31.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:31.167+0000 7f1453482700 1 --2- 172.21.15.107:0/3426816569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f144c100cc0 0x7f144c19e610 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:31.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:31.167+0000 7f1453482700 1 --2- 172.21.15.107:0/3426816569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f144c101680 0x7f144c19eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:31.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:31.167+0000 7f1453482700 1 -- 172.21.15.107:0/3426816569 >> 172.21.15.107:0/3426816569 conn(0x7f144c0765c0 msgr2=0x7f144c0fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:31.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:31.167+0000 7f1453482700 1 -- 172.21.15.107:0/3426816569 shutdown_connections 2024-05-09T08:32:31.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:31.167+0000 7f1453482700 1 -- 172.21.15.107:0/3426816569 wait complete. 2024-05-09T08:32:31.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:31 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3426816569' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-09T08:32:31.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:31 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3426816569' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-09T08:32:31.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:31 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3426816569' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-09T08:32:31.972 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332142 got 219043332147 for osd.7 2024-05-09T08:32:31.972 DEBUG:teuthology.parallel:result is None 2024-05-09T08:32:32.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.414+0000 7f0b17494700 1 -- 172.21.15.107:0/4283787656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b10103460 msgr2=0x7f0b10100870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:32.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.414+0000 7f0b17494700 1 --2- 172.21.15.107:0/4283787656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b10103460 0x7f0b10100870 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009990 tx=0x7f0b0c009ca0 comp rx=0 tx=0).stop 2024-05-09T08:32:32.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.414+0000 7f0b17494700 1 -- 172.21.15.107:0/4283787656 shutdown_connections 2024-05-09T08:32:32.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.414+0000 7f0b17494700 1 --2- 172.21.15.107:0/4283787656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b1010eff0 0x7f0b101113e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:32.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.414+0000 7f0b17494700 1 --2- 172.21.15.107:0/4283787656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b10103460 0x7f0b10100870 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:32.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.414+0000 7f0b17494700 1 --2- 172.21.15.107:0/4283787656 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b10100db0 0x7f0b10101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:32.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.414+0000 7f0b17494700 1 -- 172.21.15.107:0/4283787656 >> 172.21.15.107:0/4283787656 conn(0x7f0b10077820 msgr2=0x7f0b100feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:32.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.415+0000 7f0b17494700 1 -- 172.21.15.107:0/4283787656 shutdown_connections 2024-05-09T08:32:32.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.415+0000 7f0b17494700 1 -- 172.21.15.107:0/4283787656 wait complete. 2024-05-09T08:32:32.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.415+0000 7f0b17494700 1 Processor -- start 2024-05-09T08:32:32.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.415+0000 7f0b17494700 1 -- start start 2024-05-09T08:32:32.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.415+0000 7f0b17494700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b10100db0 0x7f0b1019ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:32.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.415+0000 7f0b17494700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b10103460 0x7f0b1019b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:32.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.415+0000 7f0b17494700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b1010eff0 0x7f0b1019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:32.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.415+0000 7f0b17494700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0b10114970 con 0x7f0b1010eff0 2024-05-09T08:32:32.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.415+0000 7f0b17494700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0b101147f0 con 0x7f0b10100db0 2024-05-09T08:32:32.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.415+0000 7f0b17494700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0b10114af0 con 0x7f0b10103460 2024-05-09T08:32:32.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.416+0000 7f0b15a31700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b1010eff0 0x7f0b1019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:32.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.416+0000 7f0b15a31700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b1010eff0 0x7f0b1019f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50232/0 (socket says 172.21.15.107:50232) 2024-05-09T08:32:32.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.416+0000 7f0b15a31700 1 -- 172.21.15.107:0/3767255499 learned_addr learned my addr 172.21.15.107:0/3767255499 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:32.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.416+0000 7f0b15230700 1 --2- 172.21.15.107:0/3767255499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b10100db0 0x7f0b1019ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:32.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.416+0000 7f0b14a2f700 1 --2- 172.21.15.107:0/3767255499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b10103460 0x7f0b1019b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.416+0000 7f0b15a31700 1 -- 172.21.15.107:0/3767255499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b10103460 msgr2=0x7f0b1019b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.416+0000 7f0b15a31700 1 --2- 172.21.15.107:0/3767255499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b10103460 0x7f0b1019b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.416+0000 7f0b15a31700 1 -- 172.21.15.107:0/3767255499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b10100db0 msgr2=0x7f0b1019ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:32.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.416+0000 7f0b15a31700 1 --2- 172.21.15.107:0/3767255499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b10100db0 0x7f0b1019ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:32.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.416+0000 7f0b15a31700 1 -- 172.21.15.107:0/3767255499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1019fe30 con 0x7f0b1010eff0 2024-05-09T08:32:32.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.416+0000 7f0b15a31700 1 --2- 172.21.15.107:0/3767255499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b1010eff0 0x7f0b1019f6f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0b08007ee0 tx=0x7f0b0800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:32.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.416+0000 7f0b067fc700 1 -- 172.21.15.107:0/3767255499 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b08004db0 con 0x7f0b1010eff0 2024-05-09T08:32:32.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.416+0000 7f0b17494700 1 -- 172.21.15.107:0/3767255499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b101a0120 con 0x7f0b1010eff0 2024-05-09T08:32:32.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.416+0000 7f0b067fc700 1 -- 172.21.15.107:0/3767255499 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b08009e20 con 0x7f0b1010eff0 2024-05-09T08:32:32.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.416+0000 7f0b067fc700 1 -- 172.21.15.107:0/3767255499 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b08015400 con 0x7f0b1010eff0 2024-05-09T08:32:32.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.416+0000 7f0b17494700 1 -- 172.21.15.107:0/3767255499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b101a79a0 con 0x7f0b1010eff0 2024-05-09T08:32:32.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.417+0000 7f0b17494700 1 -- 172.21.15.107:0/3767255499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b10069010 con 0x7f0b1010eff0 2024-05-09T08:32:32.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.418+0000 7f0b067fc700 1 -- 172.21.15.107:0/3767255499 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f0b08005240 con 0x7f0b1010eff0 2024-05-09T08:32:32.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.419+0000 7f0b067fc700 1 --2- 172.21.15.107:0/3767255499 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0afc0a2c20 0x7f0afc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:32.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.419+0000 7f0b15230700 1 --2- 172.21.15.107:0/3767255499 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0afc0a2c20 0x7f0afc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:32.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.419+0000 7f0b067fc700 1 -- 172.21.15.107:0/3767255499 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0b080cec30 con 0x7f0b1010eff0 2024-05-09T08:32:32.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.420+0000 7f0b15230700 1 --2- 172.21.15.107:0/3767255499 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0afc0a2c20 0x7f0afc0a50e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00b5c0 tx=0x7f0b0c005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:32.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.424+0000 7f0b067fc700 1 -- 172.21.15.107:0/3767255499 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0b08010070 con 0x7f0b1010eff0 2024-05-09T08:32:32.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.639+0000 7f0b17494700 1 -- 172.21.15.107:0/3767255499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f0b1006a870 con 0x7f0b1010eff0 2024-05-09T08:32:32.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.639+0000 7f0b067fc700 1 -- 172.21.15.107:0/3767255499 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0b0809b830 con 0x7f0b1010eff0 2024-05-09T08:32:32.639 INFO:teuthology.orchestra.run.smithi107.stdout:55834574930 2024-05-09T08:32:32.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.642+0000 7f0b17494700 1 -- 172.21.15.107:0/3767255499 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0afc0a2c20 msgr2=0x7f0afc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:32.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.642+0000 7f0b17494700 1 --2- 172.21.15.107:0/3767255499 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0afc0a2c20 0x7f0afc0a50e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00b5c0 tx=0x7f0b0c005960 comp rx=0 tx=0).stop 2024-05-09T08:32:32.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.642+0000 7f0b17494700 1 -- 172.21.15.107:0/3767255499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b1010eff0 msgr2=0x7f0b1019f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:32.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.642+0000 7f0b17494700 1 --2- 172.21.15.107:0/3767255499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b1010eff0 0x7f0b1019f6f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0b08007ee0 tx=0x7f0b0800d3b0 comp rx=0 tx=0).stop 2024-05-09T08:32:32.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.642+0000 7f0b17494700 1 -- 172.21.15.107:0/3767255499 shutdown_connections 2024-05-09T08:32:32.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.642+0000 7f0b17494700 1 --2- 172.21.15.107:0/3767255499 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f0afc0a2c20 0x7f0afc0a50e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:32.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.642+0000 7f0b17494700 1 --2- 172.21.15.107:0/3767255499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b1010eff0 0x7f0b1019f6f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:32.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.642+0000 7f0b17494700 1 --2- 172.21.15.107:0/3767255499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b10100db0 0x7f0b1019ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:32.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.642+0000 7f0b17494700 1 --2- 172.21.15.107:0/3767255499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b10103460 0x7f0b1019b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:32.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.642+0000 7f0b17494700 1 -- 172.21.15.107:0/3767255499 >> 172.21.15.107:0/3767255499 conn(0x7f0b10077820 msgr2=0x7f0b100feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:32.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.642+0000 7f0b17494700 1 -- 172.21.15.107:0/3767255499 shutdown_connections 2024-05-09T08:32:32.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:32.642+0000 7f0b17494700 1 -- 172.21.15.107:0/3767255499 wait complete. 2024-05-09T08:32:32.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:32 smithi157 ceph-mon[33614]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:32:32.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:32 smithi110 ceph-mon[33622]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:32:33.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:32 smithi107 ceph-mon[30122]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:32:33.409 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574925 got 55834574930 for osd.1 2024-05-09T08:32:33.409 DEBUG:teuthology.parallel:result is None 2024-05-09T08:32:33.409 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-09T08:32:33.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-09T08:32:33.681 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:33.719 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:33.719 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:33 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:33.719 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:33 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3767255499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-09T08:32:33.720 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:33 smithi107 ceph-mon[30122]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T08:32:33.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:33.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:33 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:33.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:33 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3767255499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-09T08:32:33.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:33 smithi157 ceph-mon[33614]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T08:32:33.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:33.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:33 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:33.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:33 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3767255499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-09T08:32:33.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:33 smithi110 ceph-mon[33622]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T08:32:34.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:34 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:32:34.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:34 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:32:34.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:34 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:34.874 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:34 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:32:34.874 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:34 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:32:34.874 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:34 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:35.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:34 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:32:35.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:34 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:32:35.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:34 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' 2024-05-09T08:32:35.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.388+0000 7f5b4530f700 1 -- 172.21.15.107:0/2778964534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b40101c10 msgr2=0x7f5b4010e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:35.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.388+0000 7f5b4530f700 1 --2- 172.21.15.107:0/2778964534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b40101c10 0x7f5b4010e100 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800b0a0 tx=0x7f5b3800b3b0 comp rx=0 tx=0).stop 2024-05-09T08:32:35.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.388+0000 7f5b4530f700 1 -- 172.21.15.107:0/2778964534 shutdown_connections 2024-05-09T08:32:35.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.388+0000 7f5b4530f700 1 --2- 172.21.15.107:0/2778964534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b40101c10 0x7f5b4010e100 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:35.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.388+0000 7f5b4530f700 1 --2- 172.21.15.107:0/2778964534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b401072c0 0x7f5b401076a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.388+0000 7f5b4530f700 1 --2- 172.21.15.107:0/2778964534 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b40101250 0x7f5b401016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.388+0000 7f5b4530f700 1 -- 172.21.15.107:0/2778964534 >> 172.21.15.107:0/2778964534 conn(0x7f5b40078260 msgr2=0x7f5b40078670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.388+0000 7f5b4530f700 1 -- 172.21.15.107:0/2778964534 shutdown_connections 2024-05-09T08:32:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.388+0000 7f5b4530f700 1 -- 172.21.15.107:0/2778964534 wait complete. 2024-05-09T08:32:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.388+0000 7f5b4530f700 1 Processor -- start 2024-05-09T08:32:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b4530f700 1 -- start start 2024-05-09T08:32:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b4530f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b40101250 0x7f5b401997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b4530f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b40101c10 0x7f5b40199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b4530f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b401072c0 0x7f5b401a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b4530f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5b401113e0 con 0x7f5b401072c0 2024-05-09T08:32:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b4530f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5b40111260 con 0x7f5b40101c10 2024-05-09T08:32:35.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b4530f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5b40111560 con 0x7f5b40101250 2024-05-09T08:32:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b3f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b401072c0 0x7f5b401a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b3f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b401072c0 0x7f5b401a4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50246/0 (socket says 172.21.15.107:50246) 2024-05-09T08:32:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b3f7fe700 1 -- 172.21.15.107:0/1176839165 learned_addr learned my addr 172.21.15.107:0/1176839165 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b3e7fc700 1 --2- 172.21.15.107:0/1176839165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b40101c10 0x7f5b40199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b3effd700 1 --2- 172.21.15.107:0/1176839165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b40101250 0x7f5b401997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b3f7fe700 1 -- 172.21.15.107:0/1176839165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b40101250 msgr2=0x7f5b401997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b3f7fe700 1 --2- 172.21.15.107:0/1176839165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b40101250 0x7f5b401997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b3f7fe700 1 -- 172.21.15.107:0/1176839165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b40101c10 msgr2=0x7f5b40199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b3f7fe700 1 --2- 172.21.15.107:0/1176839165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b40101c10 0x7f5b40199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b3f7fe700 1 -- 172.21.15.107:0/1176839165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b401a4d80 con 0x7f5b401072c0 2024-05-09T08:32:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b3effd700 1 --2- 172.21.15.107:0/1176839165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b40101250 0x7f5b401997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:32:35.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.389+0000 7f5b3f7fe700 1 --2- 172.21.15.107:0/1176839165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b401072c0 0x7f5b401a4840 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5b380062a0 tx=0x7f5b38004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.390+0000 7f5b27fff700 1 -- 172.21.15.107:0/1176839165 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b38021070 con 0x7f5b401072c0 2024-05-09T08:32:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.390+0000 7f5b27fff700 1 -- 172.21.15.107:0/1176839165 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b38007d00 con 0x7f5b401072c0 2024-05-09T08:32:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.390+0000 7f5b4530f700 1 -- 172.21.15.107:0/1176839165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b401a4f50 con 0x7f5b401072c0 2024-05-09T08:32:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.390+0000 7f5b27fff700 1 -- 172.21.15.107:0/1176839165 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3801d450 con 0x7f5b401072c0 2024-05-09T08:32:35.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.390+0000 7f5b4530f700 1 -- 172.21.15.107:0/1176839165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b401a5400 con 0x7f5b401072c0 2024-05-09T08:32:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.391+0000 7f5b4530f700 1 -- 172.21.15.107:0/1176839165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b40102970 con 0x7f5b401072c0 2024-05-09T08:32:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.392+0000 7f5b27fff700 1 -- 172.21.15.107:0/1176839165 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f5b38003a00 con 0x7f5b401072c0 2024-05-09T08:32:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.392+0000 7f5b27fff700 1 --2- 172.21.15.107:0/1176839165 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5b280a2c20 0x7f5b280a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:35.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.392+0000 7f5b3effd700 1 --2- 172.21.15.107:0/1176839165 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5b280a2c20 0x7f5b280a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:35.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.392+0000 7f5b27fff700 1 -- 172.21.15.107:0/1176839165 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5b3800f020 con 0x7f5b401072c0 2024-05-09T08:32:35.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.393+0000 7f5b3effd700 1 --2- 172.21.15.107:0/1176839165 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5b280a2c20 0x7f5b280a50e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5b3000baa0 tx=0x7f5b30005db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:35.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.396+0000 7f5b27fff700 1 -- 172.21.15.107:0/1176839165 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5b3802a020 con 0x7f5b401072c0 2024-05-09T08:32:35.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.588+0000 7f5b4530f700 1 -- 172.21.15.107:0/1176839165 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5b40062760 con 0x7f5b280a2c20 2024-05-09T08:32:35.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.597+0000 7f5b27fff700 1 -- 172.21.15.107:0/1176839165 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340477 (secure 0 0 0) 0x7f5b40062760 con 0x7f5b280a2c20 2024-05-09T08:32:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:32:35.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.601+0000 7f5b4530f700 1 -- 172.21.15.107:0/1176839165 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5b280a2c20 msgr2=0x7f5b280a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:35.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.601+0000 7f5b4530f700 1 --2- 172.21.15.107:0/1176839165 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5b280a2c20 0x7f5b280a50e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5b3000baa0 tx=0x7f5b30005db0 comp rx=0 tx=0).stop 2024-05-09T08:32:35.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.601+0000 7f5b4530f700 1 -- 172.21.15.107:0/1176839165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b401072c0 msgr2=0x7f5b401a4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:35.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.601+0000 7f5b4530f700 1 --2- 172.21.15.107:0/1176839165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b401072c0 0x7f5b401a4840 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5b380062a0 tx=0x7f5b38004ed0 comp rx=0 tx=0).stop 2024-05-09T08:32:35.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.601+0000 7f5b4530f700 1 -- 172.21.15.107:0/1176839165 shutdown_connections 2024-05-09T08:32:35.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.601+0000 7f5b4530f700 1 --2- 172.21.15.107:0/1176839165 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f5b280a2c20 0x7f5b280a50e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:35.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.601+0000 7f5b4530f700 1 --2- 172.21.15.107:0/1176839165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b401072c0 0x7f5b401a4840 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:35.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.601+0000 7f5b4530f700 1 --2- 172.21.15.107:0/1176839165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b40101250 0x7f5b401997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:35.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.601+0000 7f5b4530f700 1 --2- 172.21.15.107:0/1176839165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b40101c10 0x7f5b40199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:35.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.601+0000 7f5b4530f700 1 -- 172.21.15.107:0/1176839165 >> 172.21.15.107:0/1176839165 conn(0x7f5b40078260 msgr2=0x7f5b400ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:35.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.601+0000 7f5b4530f700 1 -- 172.21.15.107:0/1176839165 shutdown_connections 2024-05-09T08:32:35.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:35.601+0000 7f5b4530f700 1 -- 172.21.15.107:0/1176839165 wait complete. 2024-05-09T08:32:35.605 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-05-09T08:32:35.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:35 smithi110 ceph-mon[33622]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T08:32:35.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:35 smithi107 ceph-mon[30122]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T08:32:36.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:35 smithi157 ceph-mon[33614]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T08:32:36.445 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-05-09T08:32:35.480606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658349,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":603,"num_read_kb":426,"num_write":350,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1182017,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":784,"ondisk_log_size":784,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":396528,"kb_used_data":14096,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124545296,"statfs":{"total":1151940427776,"available":1151534383104,"internally_reserved":0,"allocated":14434304,"data_stored":7960757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":87,"num_read_kb":47,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005040"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.493327+0000","last_change":"2024-05-09T08:31:00.695034+0000","last_active":"2024-05-09T08:32:25.493327+0000","last_peered":"2024-05-09T08:32:25.493327+0000","last_clean":"2024-05-09T08:32:25.493327+0000","last_became_active":"2024-05-09T08:31:00.694229+0000","last_became_peered":"2024-05-09T08:31:00.694229+0000","last_unstale":"2024-05-09T08:32:25.493327+0000","last_undegraded":"2024-05-09T08:32:25.493327+0000","last_fullsized":"2024-05-09T08:32:25.493327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:57:22.964848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.483449+0000","last_change":"2024-05-09T08:31:00.686219+0000","last_active":"2024-05-09T08:32:25.483449+0000","last_peered":"2024-05-09T08:32:25.483449+0000","last_clean":"2024-05-09T08:32:25.483449+0000","last_became_active":"2024-05-09T08:31:00.685911+0000","last_became_peered":"2024-05-09T08:31:00.685911+0000","last_unstale":"2024-05-09T08:32:25.483449+0000","last_undegraded":"2024-05-09T08:32:25.483449+0000","last_fullsized":"2024-05-09T08:32:25.483449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:28.246199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.473632+0000","last_change":"2024-05-09T08:31:00.694228+0000","last_active":"2024-05-09T08:32:25.473632+0000","last_peered":"2024-05-09T08:32:25.473632+0000","last_clean":"2024-05-09T08:32:25.473632+0000","last_became_active":"2024-05-09T08:31:00.693838+0000","last_became_peered":"2024-05-09T08:31:00.693838+0000","last_unstale":"2024-05-09T08:32:25.473632+0000","last_undegraded":"2024-05-09T08:32:25.473632+0000","last_fullsized":"2024-05-09T08:32:25.473632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:44.847966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.480390+0000","last_change":"2024-05-09T08:31:00.702770+0000","last_active":"2024-05-09T08:32:25.480390+0000","last_peered":"2024-05-09T08:32:25.480390+0000","last_clean":"2024-05-09T08:32:25.480390+0000","last_became_active":"2024-05-09T08:31:00.702424+0000","last_became_peered":"2024-05-09T08:31:00.702424+0000","last_unstale":"2024-05-09T08:32:25.480390+0000","last_undegraded":"2024-05-09T08:32:25.480390+0000","last_fullsized":"2024-05-09T08:32:25.480390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:24:05.228942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.482645+0000","last_change":"2024-05-09T08:31:00.688308+0000","last_active":"2024-05-09T08:32:25.482645+0000","last_peered":"2024-05-09T08:32:25.482645+0000","last_clean":"2024-05-09T08:32:25.482645+0000","last_became_active":"2024-05-09T08:31:00.687891+0000","last_became_peered":"2024-05-09T08:31:00.687891+0000","last_unstale":"2024-05-09T08:32:25.482645+0000","last_undegraded":"2024-05-09T08:32:25.482645+0000","last_fullsized":"2024-05-09T08:32:25.482645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:48:42.648940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.482462+0000","last_change":"2024-05-09T08:31:00.694705+0000","last_active":"2024-05-09T08:32:25.482462+0000","last_peered":"2024-05-09T08:32:25.482462+0000","last_clean":"2024-05-09T08:32:25.482462+0000","last_became_active":"2024-05-09T08:31:00.694361+0000","last_became_peered":"2024-05-09T08:31:00.694361+0000","last_unstale":"2024-05-09T08:32:25.482462+0000","last_undegraded":"2024-05-09T08:32:25.482462+0000","last_fullsized":"2024-05-09T08:32:25.482462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:07:23.137410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.481999+0000","last_change":"2024-05-09T08:31:00.689824+0000","last_active":"2024-05-09T08:32:25.481999+0000","last_peered":"2024-05-09T08:32:25.481999+0000","last_clean":"2024-05-09T08:32:25.481999+0000","last_became_active":"2024-05-09T08:31:00.689491+0000","last_became_peered":"2024-05-09T08:31:00.689491+0000","last_unstale":"2024-05-09T08:32:25.481999+0000","last_undegraded":"2024-05-09T08:32:25.481999+0000","last_fullsized":"2024-05-09T08:32:25.481999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:18.791225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.483621+0000","last_change":"2024-05-09T08:31:00.695014+0000","last_active":"2024-05-09T08:32:25.483621+0000","last_peered":"2024-05-09T08:32:25.483621+0000","last_clean":"2024-05-09T08:32:25.483621+0000","last_became_active":"2024-05-09T08:31:00.694683+0000","last_became_peered":"2024-05-09T08:31:00.694683+0000","last_unstale":"2024-05-09T08:32:25.483621+0000","last_undegraded":"2024-05-09T08:32:25.483621+0000","last_fullsized":"2024-05-09T08:32:25.483621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:08.570258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.494960+0000","last_change":"2024-05-09T08:31:00.701538+0000","last_active":"2024-05-09T08:32:25.494960+0000","last_peered":"2024-05-09T08:32:25.494960+0000","last_clean":"2024-05-09T08:32:25.494960+0000","last_became_active":"2024-05-09T08:31:00.701226+0000","last_became_peered":"2024-05-09T08:31:00.701226+0000","last_unstale":"2024-05-09T08:32:25.494960+0000","last_undegraded":"2024-05-09T08:32:25.494960+0000","last_fullsized":"2024-05-09T08:32:25.494960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:14:50.853063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484188+0000","last_change":"2024-05-09T08:31:00.698145+0000","last_active":"2024-05-09T08:32:25.484188+0000","last_peered":"2024-05-09T08:32:25.484188+0000","last_clean":"2024-05-09T08:32:25.484188+0000","last_became_active":"2024-05-09T08:31:00.697761+0000","last_became_peered":"2024-05-09T08:31:00.697761+0000","last_unstale":"2024-05-09T08:32:25.484188+0000","last_undegraded":"2024-05-09T08:32:25.484188+0000","last_fullsized":"2024-05-09T08:32:25.484188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:26.094013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.490789+0000","last_change":"2024-05-09T08:31:00.699520+0000","last_active":"2024-05-09T08:32:25.490789+0000","last_peered":"2024-05-09T08:32:25.490789+0000","last_clean":"2024-05-09T08:32:25.490789+0000","last_became_active":"2024-05-09T08:31:00.697579+0000","last_became_peered":"2024-05-09T08:31:00.697579+0000","last_unstale":"2024-05-09T08:32:25.490789+0000","last_undegraded":"2024-05-09T08:32:25.490789+0000","last_fullsized":"2024-05-09T08:32:25.490789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:26:34.324474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.478295+0000","last_change":"2024-05-09T08:31:00.693356+0000","last_active":"2024-05-09T08:32:25.478295+0000","last_peered":"2024-05-09T08:32:25.478295+0000","last_clean":"2024-05-09T08:32:25.478295+0000","last_became_active":"2024-05-09T08:31:00.692985+0000","last_became_peered":"2024-05-09T08:31:00.692985+0000","last_unstale":"2024-05-09T08:32:25.478295+0000","last_undegraded":"2024-05-09T08:32:25.478295+0000","last_fullsized":"2024-05-09T08:32:25.478295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:49:01.115497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484938+0000","last_change":"2024-05-09T08:31:00.677768+0000","last_active":"2024-05-09T08:32:25.484938+0000","last_peered":"2024-05-09T08:32:25.484938+0000","last_clean":"2024-05-09T08:32:25.484938+0000","last_became_active":"2024-05-09T08:31:00.677386+0000","last_became_peered":"2024-05-09T08:31:00.677386+0000","last_unstale":"2024-05-09T08:32:25.484938+0000","last_undegraded":"2024-05-09T08:32:25.484938+0000","last_fullsized":"2024-05-09T08:32:25.484938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:55.363324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484172+0000","last_change":"2024-05-09T08:31:00.689609+0000","last_active":"2024-05-09T08:32:25.484172+0000","last_peered":"2024-05-09T08:32:25.484172+0000","last_clean":"2024-05-09T08:32:25.484172+0000","last_became_active":"2024-05-09T08:31:00.689303+0000","last_became_peered":"2024-05-09T08:31:00.689303+0000","last_unstale":"2024-05-09T08:32:25.484172+0000","last_undegraded":"2024-05-09T08:32:25.484172+0000","last_fullsized":"2024-05-09T08:32:25.484172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:15:58.672460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.482196+0000","last_change":"2024-05-09T08:31:00.704708+0000","last_active":"2024-05-09T08:32:25.482196+0000","last_peered":"2024-05-09T08:32:25.482196+0000","last_clean":"2024-05-09T08:32:25.482196+0000","last_became_active":"2024-05-09T08:31:00.704269+0000","last_became_peered":"2024-05-09T08:31:00.704269+0000","last_unstale":"2024-05-09T08:32:25.482196+0000","last_undegraded":"2024-05-09T08:32:25.482196+0000","last_fullsized":"2024-05-09T08:32:25.482196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:17:44.580420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.493676+0000","last_change":"2024-05-09T08:31:00.703089+0000","last_active":"2024-05-09T08:32:25.493676+0000","last_peered":"2024-05-09T08:32:25.493676+0000","last_clean":"2024-05-09T08:32:25.493676+0000","last_became_active":"2024-05-09T08:31:00.702531+0000","last_became_peered":"2024-05-09T08:31:00.702531+0000","last_unstale":"2024-05-09T08:32:25.493676+0000","last_undegraded":"2024-05-09T08:32:25.493676+0000","last_fullsized":"2024-05-09T08:32:25.493676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:07:39.887748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.493600+0000","last_change":"2024-05-09T08:31:00.695332+0000","last_active":"2024-05-09T08:32:25.493600+0000","last_peered":"2024-05-09T08:32:25.493600+0000","last_clean":"2024-05-09T08:32:25.493600+0000","last_became_active":"2024-05-09T08:31:00.694703+0000","last_became_peered":"2024-05-09T08:31:00.694703+0000","last_unstale":"2024-05-09T08:32:25.493600+0000","last_undegraded":"2024-05-09T08:32:25.493600+0000","last_fullsized":"2024-05-09T08:32:25.493600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:53.727922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.488840+0000","last_change":"2024-05-09T08:31:00.689871+0000","last_active":"2024-05-09T08:32:25.488840+0000","last_peered":"2024-05-09T08:32:25.488840+0000","last_clean":"2024-05-09T08:32:25.488840+0000","last_became_active":"2024-05-09T08:31:00.689498+0000","last_became_peered":"2024-05-09T08:31:00.689498+0000","last_unstale":"2024-05-09T08:32:25.488840+0000","last_undegraded":"2024-05-09T08:32:25.488840+0000","last_fullsized":"2024-05-09T08:32:25.488840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:18:14.123442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.483055+0000","last_change":"2024-05-09T08:31:00.697175+0000","last_active":"2024-05-09T08:32:25.483055+0000","last_peered":"2024-05-09T08:32:25.483055+0000","last_clean":"2024-05-09T08:32:25.483055+0000","last_became_active":"2024-05-09T08:31:00.694103+0000","last_became_peered":"2024-05-09T08:31:00.694103+0000","last_unstale":"2024-05-09T08:32:25.483055+0000","last_undegraded":"2024-05-09T08:32:25.483055+0000","last_fullsized":"2024-05-09T08:32:25.483055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:41:12.549854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.797212+0000","last_change":"2024-05-09T08:31:56.797212+0000","last_active":"2024-05-09T08:31:56.797212+0000","last_peered":"2024-05-09T08:31:56.797212+0000","last_clean":"2024-05-09T08:31:56.797212+0000","last_became_active":"2024-05-09T08:31:56.796786+0000","last_became_peered":"2024-05-09T08:31:56.796786+0000","last_unstale":"2024-05-09T08:31:56.797212+0000","last_undegraded":"2024-05-09T08:31:56.797212+0000","last_fullsized":"2024-05-09T08:31:56.797212+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:55.270522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.578564+0000","last_change":"2024-05-09T08:31:56.578564+0000","last_active":"2024-05-09T08:31:56.578564+0000","last_peered":"2024-05-09T08:31:56.578564+0000","last_clean":"2024-05-09T08:31:56.578564+0000","last_became_active":"2024-05-09T08:31:56.571246+0000","last_became_peered":"2024-05-09T08:31:56.571246+0000","last_unstale":"2024-05-09T08:31:56.578564+0000","last_undegraded":"2024-05-09T08:31:56.578564+0000","last_fullsized":"2024-05-09T08:31:56.578564+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:15:10.322529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.479738+0000","last_change":"2024-05-09T08:31:00.696023+0000","last_active":"2024-05-09T08:32:25.479738+0000","last_peered":"2024-05-09T08:32:25.479738+0000","last_clean":"2024-05-09T08:32:25.479738+0000","last_became_active":"2024-05-09T08:31:00.695677+0000","last_became_peered":"2024-05-09T08:31:00.695677+0000","last_unstale":"2024-05-09T08:32:25.479738+0000","last_undegraded":"2024-05-09T08:32:25.479738+0000","last_fullsized":"2024-05-09T08:32:25.479738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:55.989133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566480+0000","last_change":"2024-05-09T08:31:04.859302+0000","last_active":"2024-05-09T08:31:56.566480+0000","last_peered":"2024-05-09T08:31:56.566480+0000","last_clean":"2024-05-09T08:31:56.566480+0000","last_became_active":"2024-05-09T08:31:04.858897+0000","last_became_peered":"2024-05-09T08:31:04.858897+0000","last_unstale":"2024-05-09T08:31:56.566480+0000","last_undegraded":"2024-05-09T08:31:56.566480+0000","last_fullsized":"2024-05-09T08:31:56.566480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:25:32.970833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.796130+0000","last_change":"2024-05-09T08:30:59.824451+0000","last_active":"2024-05-09T08:31:56.796130+0000","last_peered":"2024-05-09T08:31:56.796130+0000","last_clean":"2024-05-09T08:31:56.796130+0000","last_became_active":"2024-05-09T08:30:59.823964+0000","last_became_peered":"2024-05-09T08:30:59.823964+0000","last_unstale":"2024-05-09T08:31:56.796130+0000","last_undegraded":"2024-05-09T08:31:56.796130+0000","last_fullsized":"2024-05-09T08:31:56.796130+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:50:13.562262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.489318+0000","last_change":"2024-05-09T08:31:00.701015+0000","last_active":"2024-05-09T08:32:25.489318+0000","last_peered":"2024-05-09T08:32:25.489318+0000","last_clean":"2024-05-09T08:32:25.489318+0000","last_became_active":"2024-05-09T08:31:00.700654+0000","last_became_peered":"2024-05-09T08:31:00.700654+0000","last_unstale":"2024-05-09T08:32:25.489318+0000","last_undegraded":"2024-05-09T08:32:25.489318+0000","last_fullsized":"2024-05-09T08:32:25.489318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:03.710790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566700+0000","last_change":"2024-05-09T08:31:04.858453+0000","last_active":"2024-05-09T08:31:56.566700+0000","last_peered":"2024-05-09T08:31:56.566700+0000","last_clean":"2024-05-09T08:31:56.566700+0000","last_became_active":"2024-05-09T08:31:04.857974+0000","last_became_peered":"2024-05-09T08:31:04.857974+0000","last_unstale":"2024-05-09T08:31:56.566700+0000","last_undegraded":"2024-05-09T08:31:56.566700+0000","last_fullsized":"2024-05-09T08:31:56.566700+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:42:52.868625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.569135+0000","last_change":"2024-05-09T08:30:59.825974+0000","last_active":"2024-05-09T08:31:56.569135+0000","last_peered":"2024-05-09T08:31:56.569135+0000","last_clean":"2024-05-09T08:31:56.569135+0000","last_became_active":"2024-05-09T08:30:59.825671+0000","last_became_peered":"2024-05-09T08:30:59.825671+0000","last_unstale":"2024-05-09T08:31:56.569135+0000","last_undegraded":"2024-05-09T08:31:56.569135+0000","last_fullsized":"2024-05-09T08:31:56.569135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:38:20.179466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.494048+0000","last_change":"2024-05-09T08:31:00.698324+0000","last_active":"2024-05-09T08:32:25.494048+0000","last_peered":"2024-05-09T08:32:25.494048+0000","last_clean":"2024-05-09T08:32:25.494048+0000","last_became_active":"2024-05-09T08:31:00.697992+0000","last_became_peered":"2024-05-09T08:31:00.697992+0000","last_unstale":"2024-05-09T08:32:25.494048+0000","last_undegraded":"2024-05-09T08:32:25.494048+0000","last_fullsized":"2024-05-09T08:32:25.494048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:25:20.786423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566561+0000","last_change":"2024-05-09T08:31:04.861946+0000","last_active":"2024-05-09T08:31:56.566561+0000","last_peered":"2024-05-09T08:31:56.566561+0000","last_clean":"2024-05-09T08:31:56.566561+0000","last_became_active":"2024-05-09T08:31:04.861511+0000","last_became_peered":"2024-05-09T08:31:04.861511+0000","last_unstale":"2024-05-09T08:31:56.566561+0000","last_undegraded":"2024-05-09T08:31:56.566561+0000","last_fullsized":"2024-05-09T08:31:56.566561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:36:44.580499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.568935+0000","last_change":"2024-05-09T08:30:59.829961+0000","last_active":"2024-05-09T08:31:56.568935+0000","last_peered":"2024-05-09T08:31:56.568935+0000","last_clean":"2024-05-09T08:31:56.568935+0000","last_became_active":"2024-05-09T08:30:59.828057+0000","last_became_peered":"2024-05-09T08:30:59.828057+0000","last_unstale":"2024-05-09T08:31:56.568935+0000","last_undegraded":"2024-05-09T08:31:56.568935+0000","last_fullsized":"2024-05-09T08:31:56.568935+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:39:18.229438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.493731+0000","last_change":"2024-05-09T08:31:00.700285+0000","last_active":"2024-05-09T08:32:25.493731+0000","last_peered":"2024-05-09T08:32:25.493731+0000","last_clean":"2024-05-09T08:32:25.493731+0000","last_became_active":"2024-05-09T08:31:00.699958+0000","last_became_peered":"2024-05-09T08:31:00.699958+0000","last_unstale":"2024-05-09T08:32:25.493731+0000","last_undegraded":"2024-05-09T08:32:25.493731+0000","last_fullsized":"2024-05-09T08:32:25.493731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:48.934827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566653+0000","last_change":"2024-05-09T08:31:04.857060+0000","last_active":"2024-05-09T08:31:56.566653+0000","last_peered":"2024-05-09T08:31:56.566653+0000","last_clean":"2024-05-09T08:31:56.566653+0000","last_became_active":"2024-05-09T08:31:04.855261+0000","last_became_peered":"2024-05-09T08:31:04.855261+0000","last_unstale":"2024-05-09T08:31:56.566653+0000","last_undegraded":"2024-05-09T08:31:56.566653+0000","last_fullsized":"2024-05-09T08:31:56.566653+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:08:08.230524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":109,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:34.170208+0000","last_change":"2024-05-09T08:30:59.831089+0000","last_active":"2024-05-09T08:32:34.170208+0000","last_peered":"2024-05-09T08:32:34.170208+0000","last_clean":"2024-05-09T08:32:34.170208+0000","last_became_active":"2024-05-09T08:30:59.830572+0000","last_became_peered":"2024-05-09T08:30:59.830572+0000","last_unstale":"2024-05-09T08:32:34.170208+0000","last_undegraded":"2024-05-09T08:32:34.170208+0000","last_fullsized":"2024-05-09T08:32:34.170208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:39:10.495104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.478511+0000","last_change":"2024-05-09T08:31:00.688792+0000","last_active":"2024-05-09T08:32:25.478511+0000","last_peered":"2024-05-09T08:32:25.478511+0000","last_clean":"2024-05-09T08:32:25.478511+0000","last_became_active":"2024-05-09T08:31:00.688386+0000","last_became_peered":"2024-05-09T08:31:00.688386+0000","last_unstale":"2024-05-09T08:32:25.478511+0000","last_undegraded":"2024-05-09T08:32:25.478511+0000","last_fullsized":"2024-05-09T08:32:25.478511+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:24.287333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566566+0000","last_change":"2024-05-09T08:31:04.862287+0000","last_active":"2024-05-09T08:31:56.566566+0000","last_peered":"2024-05-09T08:31:56.566566+0000","last_clean":"2024-05-09T08:31:56.566566+0000","last_became_active":"2024-05-09T08:31:04.861866+0000","last_became_peered":"2024-05-09T08:31:04.861866+0000","last_unstale":"2024-05-09T08:31:56.566566+0000","last_undegraded":"2024-05-09T08:31:56.566566+0000","last_fullsized":"2024-05-09T08:31:56.566566+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:14:21.552787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.669987+0000","last_change":"2024-05-09T08:30:59.824857+0000","last_active":"2024-05-09T08:31:56.669987+0000","last_peered":"2024-05-09T08:31:56.669987+0000","last_clean":"2024-05-09T08:31:56.669987+0000","last_became_active":"2024-05-09T08:30:59.824363+0000","last_became_peered":"2024-05-09T08:30:59.824363+0000","last_unstale":"2024-05-09T08:31:56.669987+0000","last_undegraded":"2024-05-09T08:31:56.669987+0000","last_fullsized":"2024-05-09T08:31:56.669987+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:20:32.267237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.483663+0000","last_change":"2024-05-09T08:31:00.694440+0000","last_active":"2024-05-09T08:32:25.483663+0000","last_peered":"2024-05-09T08:32:25.483663+0000","last_clean":"2024-05-09T08:32:25.483663+0000","last_became_active":"2024-05-09T08:31:00.694085+0000","last_became_peered":"2024-05-09T08:31:00.694085+0000","last_unstale":"2024-05-09T08:32:25.483663+0000","last_undegraded":"2024-05-09T08:32:25.483663+0000","last_fullsized":"2024-05-09T08:32:25.483663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:03:24.697479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.574514+0000","last_change":"2024-05-09T08:31:56.574514+0000","last_active":"2024-05-09T08:31:56.574514+0000","last_peered":"2024-05-09T08:31:56.574514+0000","last_clean":"2024-05-09T08:31:56.574514+0000","last_became_active":"2024-05-09T08:31:56.573786+0000","last_became_peered":"2024-05-09T08:31:56.573786+0000","last_unstale":"2024-05-09T08:31:56.574514+0000","last_undegraded":"2024-05-09T08:31:56.574514+0000","last_fullsized":"2024-05-09T08:31:56.574514+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:17.626431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.776789+0000","last_change":"2024-05-09T08:30:59.818373+0000","last_active":"2024-05-09T08:31:56.776789+0000","last_peered":"2024-05-09T08:31:56.776789+0000","last_clean":"2024-05-09T08:31:56.776789+0000","last_became_active":"2024-05-09T08:30:59.818023+0000","last_became_peered":"2024-05-09T08:30:59.818023+0000","last_unstale":"2024-05-09T08:31:56.776789+0000","last_undegraded":"2024-05-09T08:31:56.776789+0000","last_fullsized":"2024-05-09T08:31:56.776789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:23:01.994257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.492336+0000","last_change":"2024-05-09T08:31:00.689506+0000","last_active":"2024-05-09T08:32:25.492336+0000","last_peered":"2024-05-09T08:32:25.492336+0000","last_clean":"2024-05-09T08:32:25.492336+0000","last_became_active":"2024-05-09T08:31:00.689151+0000","last_became_peered":"2024-05-09T08:31:00.689151+0000","last_unstale":"2024-05-09T08:32:25.492336+0000","last_undegraded":"2024-05-09T08:32:25.492336+0000","last_fullsized":"2024-05-09T08:32:25.492336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:48:13.539509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.572238+0000","last_change":"2024-05-09T08:31:04.859215+0000","last_active":"2024-05-09T08:31:56.572238+0000","last_peered":"2024-05-09T08:31:56.572238+0000","last_clean":"2024-05-09T08:31:56.572238+0000","last_became_active":"2024-05-09T08:31:04.858759+0000","last_became_peered":"2024-05-09T08:31:04.858759+0000","last_unstale":"2024-05-09T08:31:56.572238+0000","last_undegraded":"2024-05-09T08:31:56.572238+0000","last_fullsized":"2024-05-09T08:31:56.572238+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:41:42.124234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.574342+0000","last_change":"2024-05-09T08:30:59.824260+0000","last_active":"2024-05-09T08:31:56.574342+0000","last_peered":"2024-05-09T08:31:56.574342+0000","last_clean":"2024-05-09T08:31:56.574342+0000","last_became_active":"2024-05-09T08:30:59.823959+0000","last_became_peered":"2024-05-09T08:30:59.823959+0000","last_unstale":"2024-05-09T08:31:56.574342+0000","last_undegraded":"2024-05-09T08:31:56.574342+0000","last_fullsized":"2024-05-09T08:31:56.574342+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:48:42.354279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.480127+0000","last_change":"2024-05-09T08:31:00.687810+0000","last_active":"2024-05-09T08:32:25.480127+0000","last_peered":"2024-05-09T08:32:25.480127+0000","last_clean":"2024-05-09T08:32:25.480127+0000","last_became_active":"2024-05-09T08:31:00.687486+0000","last_became_peered":"2024-05-09T08:31:00.687486+0000","last_unstale":"2024-05-09T08:32:25.480127+0000","last_undegraded":"2024-05-09T08:32:25.480127+0000","last_fullsized":"2024-05-09T08:32:25.480127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:51.428208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.796119+0000","last_change":"2024-05-09T08:31:04.853222+0000","last_active":"2024-05-09T08:31:56.796119+0000","last_peered":"2024-05-09T08:31:56.796119+0000","last_clean":"2024-05-09T08:31:56.796119+0000","last_became_active":"2024-05-09T08:31:04.852676+0000","last_became_peered":"2024-05-09T08:31:04.852676+0000","last_unstale":"2024-05-09T08:31:56.796119+0000","last_undegraded":"2024-05-09T08:31:56.796119+0000","last_fullsized":"2024-05-09T08:31:56.796119+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:58:31.949938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:32.166257+0000","last_change":"2024-05-09T08:30:59.824653+0000","last_active":"2024-05-09T08:32:32.166257+0000","last_peered":"2024-05-09T08:32:32.166257+0000","last_clean":"2024-05-09T08:32:32.166257+0000","last_became_active":"2024-05-09T08:30:59.824277+0000","last_became_peered":"2024-05-09T08:30:59.824277+0000","last_unstale":"2024-05-09T08:32:32.166257+0000","last_undegraded":"2024-05-09T08:32:32.166257+0000","last_fullsized":"2024-05-09T08:32:32.166257+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:45:20.340629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.493911+0000","last_change":"2024-05-09T08:31:00.697484+0000","last_active":"2024-05-09T08:32:25.493911+0000","last_peered":"2024-05-09T08:32:25.493911+0000","last_clean":"2024-05-09T08:32:25.493911+0000","last_became_active":"2024-05-09T08:31:00.697095+0000","last_became_peered":"2024-05-09T08:31:00.697095+0000","last_unstale":"2024-05-09T08:32:25.493911+0000","last_undegraded":"2024-05-09T08:32:25.493911+0000","last_fullsized":"2024-05-09T08:32:25.493911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:29:40.702057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.576003+0000","last_change":"2024-05-09T08:31:56.574176+0000","last_active":"2024-05-09T08:31:56.576003+0000","last_peered":"2024-05-09T08:31:56.576003+0000","last_clean":"2024-05-09T08:31:56.576003+0000","last_became_active":"2024-05-09T08:31:56.573556+0000","last_became_peered":"2024-05-09T08:31:56.573556+0000","last_unstale":"2024-05-09T08:31:56.576003+0000","last_undegraded":"2024-05-09T08:31:56.576003+0000","last_fullsized":"2024-05-09T08:31:56.576003+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:50.685712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.571430+0000","last_change":"2024-05-09T08:30:59.822683+0000","last_active":"2024-05-09T08:31:56.571430+0000","last_peered":"2024-05-09T08:31:56.571430+0000","last_clean":"2024-05-09T08:31:56.571430+0000","last_became_active":"2024-05-09T08:30:59.822150+0000","last_became_peered":"2024-05-09T08:30:59.822150+0000","last_unstale":"2024-05-09T08:31:56.571430+0000","last_undegraded":"2024-05-09T08:31:56.571430+0000","last_fullsized":"2024-05-09T08:31:56.571430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:51:32.782076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.481801+0000","last_change":"2024-05-09T08:31:00.694165+0000","last_active":"2024-05-09T08:32:25.481801+0000","last_peered":"2024-05-09T08:32:25.481801+0000","last_clean":"2024-05-09T08:32:25.481801+0000","last_became_active":"2024-05-09T08:31:00.693150+0000","last_became_peered":"2024-05-09T08:31:00.693150+0000","last_unstale":"2024-05-09T08:32:25.481801+0000","last_undegraded":"2024-05-09T08:32:25.481801+0000","last_fullsized":"2024-05-09T08:32:25.481801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:29.278078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566842+0000","last_change":"2024-05-09T08:31:04.858785+0000","last_active":"2024-05-09T08:31:56.566842+0000","last_peered":"2024-05-09T08:31:56.566842+0000","last_clean":"2024-05-09T08:31:56.566842+0000","last_became_active":"2024-05-09T08:31:04.857947+0000","last_became_peered":"2024-05-09T08:31:04.857947+0000","last_unstale":"2024-05-09T08:31:56.566842+0000","last_undegraded":"2024-05-09T08:31:56.566842+0000","last_fullsized":"2024-05-09T08:31:56.566842+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:55:48.595607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.576695+0000","last_change":"2024-05-09T08:30:59.822946+0000","last_active":"2024-05-09T08:31:56.576695+0000","last_peered":"2024-05-09T08:31:56.576695+0000","last_clean":"2024-05-09T08:31:56.576695+0000","last_became_active":"2024-05-09T08:30:59.822174+0000","last_became_peered":"2024-05-09T08:30:59.822174+0000","last_unstale":"2024-05-09T08:31:56.576695+0000","last_undegraded":"2024-05-09T08:31:56.576695+0000","last_fullsized":"2024-05-09T08:31:56.576695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:14.224033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.493668+0000","last_change":"2024-05-09T08:31:00.704781+0000","last_active":"2024-05-09T08:32:25.493668+0000","last_peered":"2024-05-09T08:32:25.493668+0000","last_clean":"2024-05-09T08:32:25.493668+0000","last_became_active":"2024-05-09T08:31:00.704170+0000","last_became_peered":"2024-05-09T08:31:00.704170+0000","last_unstale":"2024-05-09T08:32:25.493668+0000","last_undegraded":"2024-05-09T08:32:25.493668+0000","last_fullsized":"2024-05-09T08:32:25.493668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:27:47.402290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.567049+0000","last_change":"2024-05-09T08:31:04.855648+0000","last_active":"2024-05-09T08:31:56.567049+0000","last_peered":"2024-05-09T08:31:56.567049+0000","last_clean":"2024-05-09T08:31:56.567049+0000","last_became_active":"2024-05-09T08:31:04.854678+0000","last_became_peered":"2024-05-09T08:31:04.854678+0000","last_unstale":"2024-05-09T08:31:56.567049+0000","last_undegraded":"2024-05-09T08:31:56.567049+0000","last_fullsized":"2024-05-09T08:31:56.567049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:53:46.039172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'35","reported_seq":217,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:33.180049+0000","last_change":"2024-05-09T08:30:59.826498+0000","last_active":"2024-05-09T08:32:33.180049+0000","last_peered":"2024-05-09T08:32:33.180049+0000","last_clean":"2024-05-09T08:32:33.180049+0000","last_became_active":"2024-05-09T08:30:59.826050+0000","last_became_peered":"2024-05-09T08:30:59.826050+0000","last_unstale":"2024-05-09T08:32:33.180049+0000","last_undegraded":"2024-05-09T08:32:33.180049+0000","last_fullsized":"2024-05-09T08:32:33.180049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:19.920901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63605,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":155,"num_read_kb":140,"num_write":36,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.485055+0000","last_change":"2024-05-09T08:31:00.688379+0000","last_active":"2024-05-09T08:32:25.485055+0000","last_peered":"2024-05-09T08:32:25.485055+0000","last_clean":"2024-05-09T08:32:25.485055+0000","last_became_active":"2024-05-09T08:31:00.688030+0000","last_became_peered":"2024-05-09T08:31:00.688030+0000","last_unstale":"2024-05-09T08:32:25.485055+0000","last_undegraded":"2024-05-09T08:32:25.485055+0000","last_fullsized":"2024-05-09T08:32:25.485055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:17.248154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.576929+0000","last_change":"2024-05-09T08:31:04.858513+0000","last_active":"2024-05-09T08:31:56.576929+0000","last_peered":"2024-05-09T08:31:56.576929+0000","last_clean":"2024-05-09T08:31:56.576929+0000","last_became_active":"2024-05-09T08:31:04.857988+0000","last_became_peered":"2024-05-09T08:31:04.857988+0000","last_unstale":"2024-05-09T08:31:56.576929+0000","last_undegraded":"2024-05-09T08:31:56.576929+0000","last_fullsized":"2024-05-09T08:31:56.576929+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:59:24.264503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566645+0000","last_change":"2024-05-09T08:30:59.823403+0000","last_active":"2024-05-09T08:31:56.566645+0000","last_peered":"2024-05-09T08:31:56.566645+0000","last_clean":"2024-05-09T08:31:56.566645+0000","last_became_active":"2024-05-09T08:30:59.823066+0000","last_became_peered":"2024-05-09T08:30:59.823066+0000","last_unstale":"2024-05-09T08:31:56.566645+0000","last_undegraded":"2024-05-09T08:31:56.566645+0000","last_fullsized":"2024-05-09T08:31:56.566645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:16:39.157895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484172+0000","last_change":"2024-05-09T08:31:00.688009+0000","last_active":"2024-05-09T08:32:25.484172+0000","last_peered":"2024-05-09T08:32:25.484172+0000","last_clean":"2024-05-09T08:32:25.484172+0000","last_became_active":"2024-05-09T08:31:00.687679+0000","last_became_peered":"2024-05-09T08:31:00.687679+0000","last_unstale":"2024-05-09T08:32:25.484172+0000","last_undegraded":"2024-05-09T08:32:25.484172+0000","last_fullsized":"2024-05-09T08:32:25.484172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:44.403639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.570884+0000","last_change":"2024-05-09T08:31:04.851880+0000","last_active":"2024-05-09T08:31:56.570884+0000","last_peered":"2024-05-09T08:31:56.570884+0000","last_clean":"2024-05-09T08:31:56.570884+0000","last_became_active":"2024-05-09T08:31:04.851316+0000","last_became_peered":"2024-05-09T08:31:04.851316+0000","last_unstale":"2024-05-09T08:31:56.570884+0000","last_undegraded":"2024-05-09T08:31:56.570884+0000","last_fullsized":"2024-05-09T08:31:56.570884+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:53.144996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.575140+0000","last_change":"2024-05-09T08:30:59.823411+0000","last_active":"2024-05-09T08:31:56.575140+0000","last_peered":"2024-05-09T08:31:56.575140+0000","last_clean":"2024-05-09T08:31:56.575140+0000","last_became_active":"2024-05-09T08:30:59.822854+0000","last_became_peered":"2024-05-09T08:30:59.822854+0000","last_unstale":"2024-05-09T08:31:56.575140+0000","last_undegraded":"2024-05-09T08:31:56.575140+0000","last_fullsized":"2024-05-09T08:31:56.575140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:36:03.666937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.486496+0000","last_change":"2024-05-09T08:31:56.777818+0000","last_active":"2024-05-09T08:32:25.486496+0000","last_peered":"2024-05-09T08:32:25.486496+0000","last_clean":"2024-05-09T08:32:25.486496+0000","last_became_active":"2024-05-09T08:31:56.777555+0000","last_became_peered":"2024-05-09T08:31:56.777555+0000","last_unstale":"2024-05-09T08:32:25.486496+0000","last_undegraded":"2024-05-09T08:32:25.486496+0000","last_fullsized":"2024-05-09T08:32:25.486496+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:16:17.695073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.567497+0000","last_change":"2024-05-09T08:31:04.859338+0000","last_active":"2024-05-09T08:31:56.567497+0000","last_peered":"2024-05-09T08:31:56.567497+0000","last_clean":"2024-05-09T08:31:56.567497+0000","last_became_active":"2024-05-09T08:31:04.858951+0000","last_became_peered":"2024-05-09T08:31:04.858951+0000","last_unstale":"2024-05-09T08:31:56.567497+0000","last_undegraded":"2024-05-09T08:31:56.567497+0000","last_fullsized":"2024-05-09T08:31:56.567497+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:26:46.738359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:29.162658+0000","last_change":"2024-05-09T08:30:59.823710+0000","last_active":"2024-05-09T08:32:29.162658+0000","last_peered":"2024-05-09T08:32:29.162658+0000","last_clean":"2024-05-09T08:32:29.162658+0000","last_became_active":"2024-05-09T08:30:59.823213+0000","last_became_peered":"2024-05-09T08:30:59.823213+0000","last_unstale":"2024-05-09T08:32:29.162658+0000","last_undegraded":"2024-05-09T08:32:29.162658+0000","last_fullsized":"2024-05-09T08:32:29.162658+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:09.134942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.483614+0000","last_change":"2024-05-09T08:31:00.693735+0000","last_active":"2024-05-09T08:32:25.483614+0000","last_peered":"2024-05-09T08:32:25.483614+0000","last_clean":"2024-05-09T08:32:25.483614+0000","last_became_active":"2024-05-09T08:31:00.693401+0000","last_became_peered":"2024-05-09T08:31:00.693401+0000","last_unstale":"2024-05-09T08:32:25.483614+0000","last_undegraded":"2024-05-09T08:32:25.483614+0000","last_fullsized":"2024-05-09T08:32:25.483614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:58:41.429854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.777034+0000","last_change":"2024-05-09T08:31:04.861289+0000","last_active":"2024-05-09T08:31:56.777034+0000","last_peered":"2024-05-09T08:31:56.777034+0000","last_clean":"2024-05-09T08:31:56.777034+0000","last_became_active":"2024-05-09T08:31:04.860915+0000","last_became_peered":"2024-05-09T08:31:04.860915+0000","last_unstale":"2024-05-09T08:31:56.777034+0000","last_undegraded":"2024-05-09T08:31:56.777034+0000","last_fullsized":"2024-05-09T08:31:56.777034+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:01:45.724961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.570589+0000","last_change":"2024-05-09T08:30:59.827631+0000","last_active":"2024-05-09T08:31:56.570589+0000","last_peered":"2024-05-09T08:31:56.570589+0000","last_clean":"2024-05-09T08:31:56.570589+0000","last_became_active":"2024-05-09T08:30:59.827295+0000","last_became_peered":"2024-05-09T08:30:59.827295+0000","last_unstale":"2024-05-09T08:31:56.570589+0000","last_undegraded":"2024-05-09T08:31:56.570589+0000","last_fullsized":"2024-05-09T08:31:56.570589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:35.282564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.483419+0000","last_change":"2024-05-09T08:31:00.697712+0000","last_active":"2024-05-09T08:32:25.483419+0000","last_peered":"2024-05-09T08:32:25.483419+0000","last_clean":"2024-05-09T08:32:25.483419+0000","last_became_active":"2024-05-09T08:31:00.697385+0000","last_became_peered":"2024-05-09T08:31:00.697385+0000","last_unstale":"2024-05-09T08:32:25.483419+0000","last_undegraded":"2024-05-09T08:32:25.483419+0000","last_fullsized":"2024-05-09T08:32:25.483419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:09.087279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.575131+0000","last_change":"2024-05-09T08:31:04.860691+0000","last_active":"2024-05-09T08:31:56.575131+0000","last_peered":"2024-05-09T08:31:56.575131+0000","last_clean":"2024-05-09T08:31:56.575131+0000","last_became_active":"2024-05-09T08:31:04.858458+0000","last_became_peered":"2024-05-09T08:31:04.858458+0000","last_unstale":"2024-05-09T08:31:56.575131+0000","last_undegraded":"2024-05-09T08:31:56.575131+0000","last_fullsized":"2024-05-09T08:31:56.575131+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:16.274359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.569480+0000","last_change":"2024-05-09T08:30:59.825188+0000","last_active":"2024-05-09T08:31:56.569480+0000","last_peered":"2024-05-09T08:31:56.569480+0000","last_clean":"2024-05-09T08:31:56.569480+0000","last_became_active":"2024-05-09T08:30:59.824855+0000","last_became_peered":"2024-05-09T08:30:59.824855+0000","last_unstale":"2024-05-09T08:31:56.569480+0000","last_undegraded":"2024-05-09T08:31:56.569480+0000","last_fullsized":"2024-05-09T08:31:56.569480+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:17:21.753652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.492648+0000","last_change":"2024-05-09T08:31:00.695862+0000","last_active":"2024-05-09T08:32:25.492648+0000","last_peered":"2024-05-09T08:32:25.492648+0000","last_clean":"2024-05-09T08:32:25.492648+0000","last_became_active":"2024-05-09T08:31:00.695404+0000","last_became_peered":"2024-05-09T08:31:00.695404+0000","last_unstale":"2024-05-09T08:32:25.492648+0000","last_undegraded":"2024-05-09T08:32:25.492648+0000","last_fullsized":"2024-05-09T08:32:25.492648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:53:07.567583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.568737+0000","last_change":"2024-05-09T08:31:04.856322+0000","last_active":"2024-05-09T08:31:56.568737+0000","last_peered":"2024-05-09T08:31:56.568737+0000","last_clean":"2024-05-09T08:31:56.568737+0000","last_became_active":"2024-05-09T08:31:04.855821+0000","last_became_peered":"2024-05-09T08:31:04.855821+0000","last_unstale":"2024-05-09T08:31:56.568737+0000","last_undegraded":"2024-05-09T08:31:56.568737+0000","last_fullsized":"2024-05-09T08:31:56.568737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:29:18.594928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.574061+0000","last_change":"2024-05-09T08:31:56.574061+0000","last_active":"2024-05-09T08:31:56.574061+0000","last_peered":"2024-05-09T08:31:56.574061+0000","last_clean":"2024-05-09T08:31:56.574061+0000","last_became_active":"2024-05-09T08:31:56.573673+0000","last_became_peered":"2024-05-09T08:31:56.573673+0000","last_unstale":"2024-05-09T08:31:56.574061+0000","last_undegraded":"2024-05-09T08:31:56.574061+0000","last_fullsized":"2024-05-09T08:31:56.574061+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:12:13.699915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484283+0000","last_change":"2024-05-09T08:31:00.684523+0000","last_active":"2024-05-09T08:32:25.484283+0000","last_peered":"2024-05-09T08:32:25.484283+0000","last_clean":"2024-05-09T08:32:25.484283+0000","last_became_active":"2024-05-09T08:31:00.684153+0000","last_became_peered":"2024-05-09T08:31:00.684153+0000","last_unstale":"2024-05-09T08:32:25.484283+0000","last_undegraded":"2024-05-09T08:32:25.484283+0000","last_fullsized":"2024-05-09T08:32:25.484283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:02.423401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.776940+0000","last_change":"2024-05-09T08:31:04.857961+0000","last_active":"2024-05-09T08:31:56.776940+0000","last_peered":"2024-05-09T08:31:56.776940+0000","last_clean":"2024-05-09T08:31:56.776940+0000","last_became_active":"2024-05-09T08:31:04.857543+0000","last_became_peered":"2024-05-09T08:31:04.857543+0000","last_unstale":"2024-05-09T08:31:56.776940+0000","last_undegraded":"2024-05-09T08:31:56.776940+0000","last_fullsized":"2024-05-09T08:31:56.776940+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:22:30.820297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.570636+0000","last_change":"2024-05-09T08:30:59.827878+0000","last_active":"2024-05-09T08:31:56.570636+0000","last_peered":"2024-05-09T08:31:56.570636+0000","last_clean":"2024-05-09T08:31:56.570636+0000","last_became_active":"2024-05-09T08:30:59.827487+0000","last_became_peered":"2024-05-09T08:30:59.827487+0000","last_unstale":"2024-05-09T08:31:56.570636+0000","last_undegraded":"2024-05-09T08:31:56.570636+0000","last_fullsized":"2024-05-09T08:31:56.570636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:34.465457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.478753+0000","last_change":"2024-05-09T08:31:00.695617+0000","last_active":"2024-05-09T08:32:25.478753+0000","last_peered":"2024-05-09T08:32:25.478753+0000","last_clean":"2024-05-09T08:32:25.478753+0000","last_became_active":"2024-05-09T08:31:00.695293+0000","last_became_peered":"2024-05-09T08:31:00.695293+0000","last_unstale":"2024-05-09T08:32:25.478753+0000","last_undegraded":"2024-05-09T08:32:25.478753+0000","last_fullsized":"2024-05-09T08:32:25.478753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:44:03.958559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.777135+0000","last_change":"2024-05-09T08:31:04.860184+0000","last_active":"2024-05-09T08:31:56.777135+0000","last_peered":"2024-05-09T08:31:56.777135+0000","last_clean":"2024-05-09T08:31:56.777135+0000","last_became_active":"2024-05-09T08:31:04.859730+0000","last_became_peered":"2024-05-09T08:31:04.859730+0000","last_unstale":"2024-05-09T08:31:56.777135+0000","last_undegraded":"2024-05-09T08:31:56.777135+0000","last_fullsized":"2024-05-09T08:31:56.777135+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:13:17.292678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.483826+0000","last_change":"2024-05-09T08:31:00.694183+0000","last_active":"2024-05-09T08:32:25.483826+0000","last_peered":"2024-05-09T08:32:25.483826+0000","last_clean":"2024-05-09T08:32:25.483826+0000","last_became_active":"2024-05-09T08:31:00.693819+0000","last_became_peered":"2024-05-09T08:31:00.693819+0000","last_unstale":"2024-05-09T08:32:25.483826+0000","last_undegraded":"2024-05-09T08:32:25.483826+0000","last_fullsized":"2024-05-09T08:32:25.483826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:09:26.255663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:34.169060+0000","last_change":"2024-05-09T08:30:59.830758+0000","last_active":"2024-05-09T08:32:34.169060+0000","last_peered":"2024-05-09T08:32:34.169060+0000","last_clean":"2024-05-09T08:32:34.169060+0000","last_became_active":"2024-05-09T08:30:59.830430+0000","last_became_peered":"2024-05-09T08:30:59.830430+0000","last_unstale":"2024-05-09T08:32:34.169060+0000","last_undegraded":"2024-05-09T08:32:34.169060+0000","last_fullsized":"2024-05-09T08:32:34.169060+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:24:05.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.675063+0000","last_change":"2024-05-09T08:31:56.675063+0000","last_active":"2024-05-09T08:31:56.675063+0000","last_peered":"2024-05-09T08:31:56.675063+0000","last_clean":"2024-05-09T08:31:56.675063+0000","last_became_active":"2024-05-09T08:31:56.670995+0000","last_became_peered":"2024-05-09T08:31:56.670995+0000","last_unstale":"2024-05-09T08:31:56.675063+0000","last_undegraded":"2024-05-09T08:31:56.675063+0000","last_fullsized":"2024-05-09T08:31:56.675063+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:57:52.985420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.476532+0000","last_change":"2024-05-09T08:31:00.692197+0000","last_active":"2024-05-09T08:32:25.476532+0000","last_peered":"2024-05-09T08:32:25.476532+0000","last_clean":"2024-05-09T08:32:25.476532+0000","last_became_active":"2024-05-09T08:31:00.691348+0000","last_became_peered":"2024-05-09T08:31:00.691348+0000","last_unstale":"2024-05-09T08:32:25.476532+0000","last_undegraded":"2024-05-09T08:32:25.476532+0000","last_fullsized":"2024-05-09T08:32:25.476532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:00:45.826433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566752+0000","last_change":"2024-05-09T08:30:59.825341+0000","last_active":"2024-05-09T08:31:56.566752+0000","last_peered":"2024-05-09T08:31:56.566752+0000","last_clean":"2024-05-09T08:31:56.566752+0000","last_became_active":"2024-05-09T08:30:59.824860+0000","last_became_peered":"2024-05-09T08:30:59.824860+0000","last_unstale":"2024-05-09T08:31:56.566752+0000","last_undegraded":"2024-05-09T08:31:56.566752+0000","last_fullsized":"2024-05-09T08:31:56.566752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:13.868040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":359,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.570198+0000","last_change":"2024-05-09T08:29:47.816504+0000","last_active":"2024-05-09T08:31:56.570198+0000","last_peered":"2024-05-09T08:31:56.570198+0000","last_clean":"2024-05-09T08:31:56.570198+0000","last_became_active":"2024-05-09T08:29:47.816050+0000","last_became_peered":"2024-05-09T08:29:47.816050+0000","last_unstale":"2024-05-09T08:31:56.570198+0000","last_undegraded":"2024-05-09T08:31:56.570198+0000","last_fullsized":"2024-05-09T08:31:56.570198+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:26:22.135491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:26:22.135491+0000","last_clean_scrub_stamp":"2024-05-09T08:26:22.135491+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:28.543694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.575846+0000","last_change":"2024-05-09T08:31:04.860417+0000","last_active":"2024-05-09T08:31:56.575846+0000","last_peered":"2024-05-09T08:31:56.575846+0000","last_clean":"2024-05-09T08:31:56.575846+0000","last_became_active":"2024-05-09T08:31:04.859040+0000","last_became_peered":"2024-05-09T08:31:04.859040+0000","last_unstale":"2024-05-09T08:31:56.575846+0000","last_undegraded":"2024-05-09T08:31:56.575846+0000","last_fullsized":"2024-05-09T08:31:56.575846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:12:04.157354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.480855+0000","last_change":"2024-05-09T08:31:00.703131+0000","last_active":"2024-05-09T08:32:25.480855+0000","last_peered":"2024-05-09T08:32:25.480855+0000","last_clean":"2024-05-09T08:32:25.480855+0000","last_became_active":"2024-05-09T08:31:00.702817+0000","last_became_peered":"2024-05-09T08:31:00.702817+0000","last_unstale":"2024-05-09T08:32:25.480855+0000","last_undegraded":"2024-05-09T08:32:25.480855+0000","last_fullsized":"2024-05-09T08:32:25.480855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:31.059334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.570265+0000","last_change":"2024-05-09T08:30:59.829636+0000","last_active":"2024-05-09T08:31:56.570265+0000","last_peered":"2024-05-09T08:31:56.570265+0000","last_clean":"2024-05-09T08:31:56.570265+0000","last_became_active":"2024-05-09T08:30:59.829276+0000","last_became_peered":"2024-05-09T08:30:59.829276+0000","last_unstale":"2024-05-09T08:31:56.570265+0000","last_undegraded":"2024-05-09T08:31:56.570265+0000","last_fullsized":"2024-05-09T08:31:56.570265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:17.847258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.571002+0000","last_change":"2024-05-09T08:31:04.853945+0000","last_active":"2024-05-09T08:31:56.571002+0000","last_peered":"2024-05-09T08:31:56.571002+0000","last_clean":"2024-05-09T08:31:56.571002+0000","last_became_active":"2024-05-09T08:31:04.853417+0000","last_became_peered":"2024-05-09T08:31:04.853417+0000","last_unstale":"2024-05-09T08:31:56.571002+0000","last_undegraded":"2024-05-09T08:31:56.571002+0000","last_fullsized":"2024-05-09T08:31:56.571002+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:21:31.126562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.568918+0000","last_change":"2024-05-09T08:30:59.828613+0000","last_active":"2024-05-09T08:31:56.568918+0000","last_peered":"2024-05-09T08:31:56.568918+0000","last_clean":"2024-05-09T08:31:56.568918+0000","last_became_active":"2024-05-09T08:30:59.828256+0000","last_became_peered":"2024-05-09T08:30:59.828256+0000","last_unstale":"2024-05-09T08:31:56.568918+0000","last_undegraded":"2024-05-09T08:31:56.568918+0000","last_fullsized":"2024-05-09T08:31:56.568918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:17:04.488271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.482238+0000","last_change":"2024-05-09T08:31:00.703478+0000","last_active":"2024-05-09T08:32:25.482238+0000","last_peered":"2024-05-09T08:32:25.482238+0000","last_clean":"2024-05-09T08:32:25.482238+0000","last_became_active":"2024-05-09T08:31:00.702852+0000","last_became_peered":"2024-05-09T08:31:00.702852+0000","last_unstale":"2024-05-09T08:32:25.482238+0000","last_undegraded":"2024-05-09T08:32:25.482238+0000","last_fullsized":"2024-05-09T08:32:25.482238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:12.635341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.795827+0000","last_change":"2024-05-09T08:31:04.860801+0000","last_active":"2024-05-09T08:31:56.795827+0000","last_peered":"2024-05-09T08:31:56.795827+0000","last_clean":"2024-05-09T08:31:56.795827+0000","last_became_active":"2024-05-09T08:31:04.857387+0000","last_became_peered":"2024-05-09T08:31:04.857387+0000","last_unstale":"2024-05-09T08:31:56.795827+0000","last_undegraded":"2024-05-09T08:31:56.795827+0000","last_fullsized":"2024-05-09T08:31:56.795827+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:50:16.555987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.567897+0000","last_change":"2024-05-09T08:30:59.827082+0000","last_active":"2024-05-09T08:31:56.567897+0000","last_peered":"2024-05-09T08:31:56.567897+0000","last_clean":"2024-05-09T08:31:56.567897+0000","last_became_active":"2024-05-09T08:30:59.826746+0000","last_became_peered":"2024-05-09T08:30:59.826746+0000","last_unstale":"2024-05-09T08:31:56.567897+0000","last_undegraded":"2024-05-09T08:31:56.567897+0000","last_fullsized":"2024-05-09T08:31:56.567897+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:49.176889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.491742+0000","last_change":"2024-05-09T08:31:00.691902+0000","last_active":"2024-05-09T08:32:25.491742+0000","last_peered":"2024-05-09T08:32:25.491742+0000","last_clean":"2024-05-09T08:32:25.491742+0000","last_became_active":"2024-05-09T08:31:00.689491+0000","last_became_peered":"2024-05-09T08:31:00.689491+0000","last_unstale":"2024-05-09T08:32:25.491742+0000","last_undegraded":"2024-05-09T08:32:25.491742+0000","last_fullsized":"2024-05-09T08:32:25.491742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:48:07.335144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566278+0000","last_change":"2024-05-09T08:31:04.857788+0000","last_active":"2024-05-09T08:31:56.566278+0000","last_peered":"2024-05-09T08:31:56.566278+0000","last_clean":"2024-05-09T08:31:56.566278+0000","last_became_active":"2024-05-09T08:31:04.853831+0000","last_became_peered":"2024-05-09T08:31:04.853831+0000","last_unstale":"2024-05-09T08:31:56.566278+0000","last_undegraded":"2024-05-09T08:31:56.566278+0000","last_fullsized":"2024-05-09T08:31:56.566278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:08:15.451606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.567361+0000","last_change":"2024-05-09T08:30:59.826482+0000","last_active":"2024-05-09T08:31:56.567361+0000","last_peered":"2024-05-09T08:31:56.567361+0000","last_clean":"2024-05-09T08:31:56.567361+0000","last_became_active":"2024-05-09T08:30:59.826123+0000","last_became_peered":"2024-05-09T08:30:59.826123+0000","last_unstale":"2024-05-09T08:31:56.567361+0000","last_undegraded":"2024-05-09T08:31:56.567361+0000","last_fullsized":"2024-05-09T08:31:56.567361+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:38.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.482173+0000","last_change":"2024-05-09T08:31:00.693440+0000","last_active":"2024-05-09T08:32:25.482173+0000","last_peered":"2024-05-09T08:32:25.482173+0000","last_clean":"2024-05-09T08:32:25.482173+0000","last_became_active":"2024-05-09T08:31:00.693117+0000","last_became_peered":"2024-05-09T08:31:00.693117+0000","last_unstale":"2024-05-09T08:32:25.482173+0000","last_undegraded":"2024-05-09T08:32:25.482173+0000","last_fullsized":"2024-05-09T08:32:25.482173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:10:01.205605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.567085+0000","last_change":"2024-05-09T08:31:04.868120+0000","last_active":"2024-05-09T08:31:56.567085+0000","last_peered":"2024-05-09T08:31:56.567085+0000","last_clean":"2024-05-09T08:31:56.567085+0000","last_became_active":"2024-05-09T08:31:04.867571+0000","last_became_peered":"2024-05-09T08:31:04.867571+0000","last_unstale":"2024-05-09T08:31:56.567085+0000","last_undegraded":"2024-05-09T08:31:56.567085+0000","last_fullsized":"2024-05-09T08:31:56.567085+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:12:36.503972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.569407+0000","last_change":"2024-05-09T08:30:59.826280+0000","last_active":"2024-05-09T08:31:56.569407+0000","last_peered":"2024-05-09T08:31:56.569407+0000","last_clean":"2024-05-09T08:31:56.569407+0000","last_became_active":"2024-05-09T08:30:59.825805+0000","last_became_peered":"2024-05-09T08:30:59.825805+0000","last_unstale":"2024-05-09T08:31:56.569407+0000","last_undegraded":"2024-05-09T08:31:56.569407+0000","last_fullsized":"2024-05-09T08:31:56.569407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:46:03.833159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.454159+0000","last_change":"2024-05-09T08:31:00.687581+0000","last_active":"2024-05-09T08:32:25.454159+0000","last_peered":"2024-05-09T08:32:25.454159+0000","last_clean":"2024-05-09T08:32:25.454159+0000","last_became_active":"2024-05-09T08:31:00.684608+0000","last_became_peered":"2024-05-09T08:31:00.684608+0000","last_unstale":"2024-05-09T08:32:25.454159+0000","last_undegraded":"2024-05-09T08:32:25.454159+0000","last_fullsized":"2024-05-09T08:32:25.454159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:06:06.436840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.567738+0000","last_change":"2024-05-09T08:31:04.859399+0000","last_active":"2024-05-09T08:31:56.567738+0000","last_peered":"2024-05-09T08:31:56.567738+0000","last_clean":"2024-05-09T08:31:56.567738+0000","last_became_active":"2024-05-09T08:31:04.859027+0000","last_became_peered":"2024-05-09T08:31:04.859027+0000","last_unstale":"2024-05-09T08:31:56.567738+0000","last_undegraded":"2024-05-09T08:31:56.567738+0000","last_fullsized":"2024-05-09T08:31:56.567738+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:18:00.829342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.796124+0000","last_change":"2024-05-09T08:30:59.824171+0000","last_active":"2024-05-09T08:31:56.796124+0000","last_peered":"2024-05-09T08:31:56.796124+0000","last_clean":"2024-05-09T08:31:56.796124+0000","last_became_active":"2024-05-09T08:30:59.823859+0000","last_became_peered":"2024-05-09T08:30:59.823859+0000","last_unstale":"2024-05-09T08:31:56.796124+0000","last_undegraded":"2024-05-09T08:31:56.796124+0000","last_fullsized":"2024-05-09T08:31:56.796124+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:30.199504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.492260+0000","last_change":"2024-05-09T08:31:00.700056+0000","last_active":"2024-05-09T08:32:25.492260+0000","last_peered":"2024-05-09T08:32:25.492260+0000","last_clean":"2024-05-09T08:32:25.492260+0000","last_became_active":"2024-05-09T08:31:00.699673+0000","last_became_peered":"2024-05-09T08:31:00.699673+0000","last_unstale":"2024-05-09T08:32:25.492260+0000","last_undegraded":"2024-05-09T08:32:25.492260+0000","last_fullsized":"2024-05-09T08:32:25.492260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:23.873645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566824+0000","last_change":"2024-05-09T08:31:04.858940+0000","last_active":"2024-05-09T08:31:56.566824+0000","last_peered":"2024-05-09T08:31:56.566824+0000","last_clean":"2024-05-09T08:31:56.566824+0000","last_became_active":"2024-05-09T08:31:04.858532+0000","last_became_peered":"2024-05-09T08:31:04.858532+0000","last_unstale":"2024-05-09T08:31:56.566824+0000","last_undegraded":"2024-05-09T08:31:56.566824+0000","last_fullsized":"2024-05-09T08:31:56.566824+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:57:42.128068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566477+0000","last_change":"2024-05-09T08:30:59.824182+0000","last_active":"2024-05-09T08:31:56.566477+0000","last_peered":"2024-05-09T08:31:56.566477+0000","last_clean":"2024-05-09T08:31:56.566477+0000","last_became_active":"2024-05-09T08:30:59.823691+0000","last_became_peered":"2024-05-09T08:30:59.823691+0000","last_unstale":"2024-05-09T08:31:56.566477+0000","last_undegraded":"2024-05-09T08:31:56.566477+0000","last_fullsized":"2024-05-09T08:31:56.566477+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:28:07.810733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484483+0000","last_change":"2024-05-09T08:31:00.692614+0000","last_active":"2024-05-09T08:32:25.484483+0000","last_peered":"2024-05-09T08:32:25.484483+0000","last_clean":"2024-05-09T08:32:25.484483+0000","last_became_active":"2024-05-09T08:31:00.692280+0000","last_became_peered":"2024-05-09T08:31:00.692280+0000","last_unstale":"2024-05-09T08:32:25.484483+0000","last_undegraded":"2024-05-09T08:32:25.484483+0000","last_fullsized":"2024-05-09T08:32:25.484483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:23:15.398196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566897+0000","last_change":"2024-05-09T08:31:04.854029+0000","last_active":"2024-05-09T08:31:56.566897+0000","last_peered":"2024-05-09T08:31:56.566897+0000","last_clean":"2024-05-09T08:31:56.566897+0000","last_became_active":"2024-05-09T08:31:04.853554+0000","last_became_peered":"2024-05-09T08:31:04.853554+0000","last_unstale":"2024-05-09T08:31:56.566897+0000","last_undegraded":"2024-05-09T08:31:56.566897+0000","last_fullsized":"2024-05-09T08:31:56.566897+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:34:41.059614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.569879+0000","last_change":"2024-05-09T08:30:59.824512+0000","last_active":"2024-05-09T08:31:56.569879+0000","last_peered":"2024-05-09T08:31:56.569879+0000","last_clean":"2024-05-09T08:31:56.569879+0000","last_became_active":"2024-05-09T08:30:59.824146+0000","last_became_peered":"2024-05-09T08:30:59.824146+0000","last_unstale":"2024-05-09T08:31:56.569879+0000","last_undegraded":"2024-05-09T08:31:56.569879+0000","last_fullsized":"2024-05-09T08:31:56.569879+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:27.734292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.490636+0000","last_change":"2024-05-09T08:31:00.697272+0000","last_active":"2024-05-09T08:32:25.490636+0000","last_peered":"2024-05-09T08:32:25.490636+0000","last_clean":"2024-05-09T08:32:25.490636+0000","last_became_active":"2024-05-09T08:31:00.696026+0000","last_became_peered":"2024-05-09T08:31:00.696026+0000","last_unstale":"2024-05-09T08:32:25.490636+0000","last_undegraded":"2024-05-09T08:32:25.490636+0000","last_fullsized":"2024-05-09T08:32:25.490636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:04.376142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.569574+0000","last_change":"2024-05-09T08:31:04.856641+0000","last_active":"2024-05-09T08:31:56.569574+0000","last_peered":"2024-05-09T08:31:56.569574+0000","last_clean":"2024-05-09T08:31:56.569574+0000","last_became_active":"2024-05-09T08:31:04.856102+0000","last_became_peered":"2024-05-09T08:31:04.856102+0000","last_unstale":"2024-05-09T08:31:56.569574+0000","last_undegraded":"2024-05-09T08:31:56.569574+0000","last_fullsized":"2024-05-09T08:31:56.569574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:35:49.831076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.579569+0000","last_change":"2024-05-09T08:31:56.579569+0000","last_active":"2024-05-09T08:31:56.579569+0000","last_peered":"2024-05-09T08:31:56.579569+0000","last_clean":"2024-05-09T08:31:56.579569+0000","last_became_active":"2024-05-09T08:31:56.573496+0000","last_became_peered":"2024-05-09T08:31:56.573496+0000","last_unstale":"2024-05-09T08:31:56.579569+0000","last_undegraded":"2024-05-09T08:31:56.579569+0000","last_fullsized":"2024-05-09T08:31:56.579569+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:47:47.944200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.490646+0000","last_change":"2024-05-09T08:31:00.697876+0000","last_active":"2024-05-09T08:32:25.490646+0000","last_peered":"2024-05-09T08:32:25.490646+0000","last_clean":"2024-05-09T08:32:25.490646+0000","last_became_active":"2024-05-09T08:31:00.696299+0000","last_became_peered":"2024-05-09T08:31:00.696299+0000","last_unstale":"2024-05-09T08:32:25.490646+0000","last_undegraded":"2024-05-09T08:32:25.490646+0000","last_fullsized":"2024-05-09T08:32:25.490646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:48:57.373734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.572156+0000","last_change":"2024-05-09T08:31:04.858588+0000","last_active":"2024-05-09T08:31:56.572156+0000","last_peered":"2024-05-09T08:31:56.572156+0000","last_clean":"2024-05-09T08:31:56.572156+0000","last_became_active":"2024-05-09T08:31:04.858154+0000","last_became_peered":"2024-05-09T08:31:04.858154+0000","last_unstale":"2024-05-09T08:31:56.572156+0000","last_undegraded":"2024-05-09T08:31:56.572156+0000","last_fullsized":"2024-05-09T08:31:56.572156+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:06:01.768743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.576803+0000","last_change":"2024-05-09T08:30:59.826066+0000","last_active":"2024-05-09T08:31:56.576803+0000","last_peered":"2024-05-09T08:31:56.576803+0000","last_clean":"2024-05-09T08:31:56.576803+0000","last_became_active":"2024-05-09T08:30:59.825586+0000","last_became_peered":"2024-05-09T08:30:59.825586+0000","last_unstale":"2024-05-09T08:31:56.576803+0000","last_undegraded":"2024-05-09T08:31:56.576803+0000","last_fullsized":"2024-05-09T08:31:56.576803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:19:04.629132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484950+0000","last_change":"2024-05-09T08:31:00.693788+0000","last_active":"2024-05-09T08:32:25.484950+0000","last_peered":"2024-05-09T08:32:25.484950+0000","last_clean":"2024-05-09T08:32:25.484950+0000","last_became_active":"2024-05-09T08:31:00.693448+0000","last_became_peered":"2024-05-09T08:31:00.693448+0000","last_unstale":"2024-05-09T08:32:25.484950+0000","last_undegraded":"2024-05-09T08:32:25.484950+0000","last_fullsized":"2024-05-09T08:32:25.484950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:06.516534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.574037+0000","last_change":"2024-05-09T08:31:04.856185+0000","last_active":"2024-05-09T08:31:56.574037+0000","last_peered":"2024-05-09T08:31:56.574037+0000","last_clean":"2024-05-09T08:31:56.574037+0000","last_became_active":"2024-05-09T08:31:04.855779+0000","last_became_peered":"2024-05-09T08:31:04.855779+0000","last_unstale":"2024-05-09T08:31:56.574037+0000","last_undegraded":"2024-05-09T08:31:56.574037+0000","last_fullsized":"2024-05-09T08:31:56.574037+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:45:25.034077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.777025+0000","last_change":"2024-05-09T08:30:59.822938+0000","last_active":"2024-05-09T08:31:56.777025+0000","last_peered":"2024-05-09T08:31:56.777025+0000","last_clean":"2024-05-09T08:31:56.777025+0000","last_became_active":"2024-05-09T08:30:59.822629+0000","last_became_peered":"2024-05-09T08:30:59.822629+0000","last_unstale":"2024-05-09T08:31:56.777025+0000","last_undegraded":"2024-05-09T08:31:56.777025+0000","last_fullsized":"2024-05-09T08:31:56.777025+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:51.321350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484173+0000","last_change":"2024-05-09T08:31:00.691971+0000","last_active":"2024-05-09T08:32:25.484173+0000","last_peered":"2024-05-09T08:32:25.484173+0000","last_clean":"2024-05-09T08:32:25.484173+0000","last_became_active":"2024-05-09T08:31:00.691641+0000","last_became_peered":"2024-05-09T08:31:00.691641+0000","last_unstale":"2024-05-09T08:32:25.484173+0000","last_undegraded":"2024-05-09T08:32:25.484173+0000","last_fullsized":"2024-05-09T08:32:25.484173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:28.413886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.493734+0000","last_change":"2024-05-09T08:31:00.701026+0000","last_active":"2024-05-09T08:32:25.493734+0000","last_peered":"2024-05-09T08:32:25.493734+0000","last_clean":"2024-05-09T08:32:25.493734+0000","last_became_active":"2024-05-09T08:31:00.700457+0000","last_became_peered":"2024-05-09T08:31:00.700457+0000","last_unstale":"2024-05-09T08:32:25.493734+0000","last_undegraded":"2024-05-09T08:32:25.493734+0000","last_fullsized":"2024-05-09T08:32:25.493734+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:40.065173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.481582+0000","last_change":"2024-05-09T08:31:56.570220+0000","last_active":"2024-05-09T08:32:25.481582+0000","last_peered":"2024-05-09T08:32:25.481582+0000","last_clean":"2024-05-09T08:32:25.481582+0000","last_became_active":"2024-05-09T08:31:56.569682+0000","last_became_peered":"2024-05-09T08:31:56.569682+0000","last_unstale":"2024-05-09T08:32:25.481582+0000","last_undegraded":"2024-05-09T08:32:25.481582+0000","last_fullsized":"2024-05-09T08:32:25.481582+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:47:07.986064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.480090+0000","last_change":"2024-05-09T08:31:00.703201+0000","last_active":"2024-05-09T08:32:25.480090+0000","last_peered":"2024-05-09T08:32:25.480090+0000","last_clean":"2024-05-09T08:32:25.480090+0000","last_became_active":"2024-05-09T08:31:00.702801+0000","last_became_peered":"2024-05-09T08:31:00.702801+0000","last_unstale":"2024-05-09T08:32:25.480090+0000","last_undegraded":"2024-05-09T08:32:25.480090+0000","last_fullsized":"2024-05-09T08:32:25.480090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:58:35.106013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.482324+0000","last_change":"2024-05-09T08:31:00.693696+0000","last_active":"2024-05-09T08:32:25.482324+0000","last_peered":"2024-05-09T08:32:25.482324+0000","last_clean":"2024-05-09T08:32:25.482324+0000","last_became_active":"2024-05-09T08:31:00.693394+0000","last_became_peered":"2024-05-09T08:31:00.693394+0000","last_unstale":"2024-05-09T08:32:25.482324+0000","last_undegraded":"2024-05-09T08:32:25.482324+0000","last_fullsized":"2024-05-09T08:32:25.482324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:01.700968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484061+0000","last_change":"2024-05-09T08:31:00.694363+0000","last_active":"2024-05-09T08:32:25.484061+0000","last_peered":"2024-05-09T08:32:25.484061+0000","last_clean":"2024-05-09T08:32:25.484061+0000","last_became_active":"2024-05-09T08:31:00.693970+0000","last_became_peered":"2024-05-09T08:31:00.693970+0000","last_unstale":"2024-05-09T08:32:25.484061+0000","last_undegraded":"2024-05-09T08:32:25.484061+0000","last_fullsized":"2024-05-09T08:32:25.484061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:56.185801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.492564+0000","last_change":"2024-05-09T08:31:00.691970+0000","last_active":"2024-05-09T08:32:25.492564+0000","last_peered":"2024-05-09T08:32:25.492564+0000","last_clean":"2024-05-09T08:32:25.492564+0000","last_became_active":"2024-05-09T08:31:00.691632+0000","last_became_peered":"2024-05-09T08:31:00.691632+0000","last_unstale":"2024-05-09T08:32:25.492564+0000","last_undegraded":"2024-05-09T08:32:25.492564+0000","last_fullsized":"2024-05-09T08:32:25.492564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:04:49.316808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.477847+0000","last_change":"2024-05-09T08:31:00.692042+0000","last_active":"2024-05-09T08:32:25.477847+0000","last_peered":"2024-05-09T08:32:25.477847+0000","last_clean":"2024-05-09T08:32:25.477847+0000","last_became_active":"2024-05-09T08:31:00.691700+0000","last_became_peered":"2024-05-09T08:31:00.691700+0000","last_unstale":"2024-05-09T08:32:25.477847+0000","last_undegraded":"2024-05-09T08:32:25.477847+0000","last_fullsized":"2024-05-09T08:32:25.477847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:54.236319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.479387+0000","last_change":"2024-05-09T08:31:00.697261+0000","last_active":"2024-05-09T08:32:25.479387+0000","last_peered":"2024-05-09T08:32:25.479387+0000","last_clean":"2024-05-09T08:32:25.479387+0000","last_became_active":"2024-05-09T08:31:00.696299+0000","last_became_peered":"2024-05-09T08:31:00.696299+0000","last_unstale":"2024-05-09T08:32:25.479387+0000","last_undegraded":"2024-05-09T08:32:25.479387+0000","last_fullsized":"2024-05-09T08:32:25.479387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:06:52.061740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.480323+0000","last_change":"2024-05-09T08:31:00.691097+0000","last_active":"2024-05-09T08:32:25.480323+0000","last_peered":"2024-05-09T08:32:25.480323+0000","last_clean":"2024-05-09T08:32:25.480323+0000","last_became_active":"2024-05-09T08:31:00.690487+0000","last_became_peered":"2024-05-09T08:31:00.690487+0000","last_unstale":"2024-05-09T08:32:25.480323+0000","last_undegraded":"2024-05-09T08:32:25.480323+0000","last_fullsized":"2024-05-09T08:32:25.480323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:47:26.468296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.491778+0000","last_change":"2024-05-09T08:31:00.703201+0000","last_active":"2024-05-09T08:32:25.491778+0000","last_peered":"2024-05-09T08:32:25.491778+0000","last_clean":"2024-05-09T08:32:25.491778+0000","last_became_active":"2024-05-09T08:31:00.702853+0000","last_became_peered":"2024-05-09T08:31:00.702853+0000","last_unstale":"2024-05-09T08:32:25.491778+0000","last_undegraded":"2024-05-09T08:32:25.491778+0000","last_fullsized":"2024-05-09T08:32:25.491778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:18.510321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.493816+0000","last_change":"2024-05-09T08:31:00.703958+0000","last_active":"2024-05-09T08:32:25.493816+0000","last_peered":"2024-05-09T08:32:25.493816+0000","last_clean":"2024-05-09T08:32:25.493816+0000","last_became_active":"2024-05-09T08:31:00.703110+0000","last_became_peered":"2024-05-09T08:31:00.703110+0000","last_unstale":"2024-05-09T08:32:25.493816+0000","last_undegraded":"2024-05-09T08:32:25.493816+0000","last_fullsized":"2024-05-09T08:32:25.493816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:37:32.371869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484826+0000","last_change":"2024-05-09T08:31:00.686763+0000","last_active":"2024-05-09T08:32:25.484826+0000","last_peered":"2024-05-09T08:32:25.484826+0000","last_clean":"2024-05-09T08:32:25.484826+0000","last_became_active":"2024-05-09T08:31:00.685642+0000","last_became_peered":"2024-05-09T08:31:00.685642+0000","last_unstale":"2024-05-09T08:32:25.484826+0000","last_undegraded":"2024-05-09T08:32:25.484826+0000","last_fullsized":"2024-05-09T08:32:25.484826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:40:18.201247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.492627+0000","last_change":"2024-05-09T08:31:56.573837+0000","last_active":"2024-05-09T08:32:25.492627+0000","last_peered":"2024-05-09T08:32:25.492627+0000","last_clean":"2024-05-09T08:32:25.492627+0000","last_became_active":"2024-05-09T08:31:56.573408+0000","last_became_peered":"2024-05-09T08:31:56.573408+0000","last_unstale":"2024-05-09T08:32:25.492627+0000","last_undegraded":"2024-05-09T08:32:25.492627+0000","last_fullsized":"2024-05-09T08:32:25.492627+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:12:46.397877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67747,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":467,"num_read_kb":307,"num_write":167,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":203241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383712,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":500815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645349,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28696,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716456,"statfs":{"total":95995035648,"available":95965650944,"internally_reserved":0,"allocated":1720320,"data_stored":1156316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808874,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":505147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070511,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":500389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332148,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36932,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708220,"statfs":{"total":95995035648,"available":95957217280,"internally_reserved":0,"allocated":1634304,"data_stored":1097021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":496227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":560324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":497615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1110016,"data_stored":560510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1032192,"data_stored":496421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":501991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757311,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36936,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708216,"statfs":{"total":95995035648,"available":95957213184,"internally_reserved":0,"allocated":1638400,"data_stored":1087981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":64187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:32:36.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-09T08:32:36.699 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:36.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:36 smithi107 ceph-mon[30122]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:32:36.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:36 smithi110 ceph-mon[33622]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:32:36.995 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:36 smithi157 ceph-mon[33614]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:32:37.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:37 smithi110 ceph-mon[33622]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:32:38.000 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:37 smithi107 ceph-mon[30122]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:32:38.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:37 smithi157 ceph-mon[33614]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T08:32:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.395+0000 7f287ccb1700 1 -- 172.21.15.107:0/758038657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2878101380 msgr2=0x7f287810d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.395+0000 7f287ccb1700 1 --2- 172.21.15.107:0/758038657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2878101380 0x7f287810d890 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f287000c210 tx=0x7f287000c520 comp rx=0 tx=0).stop 2024-05-09T08:32:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.395+0000 7f287ccb1700 1 -- 172.21.15.107:0/758038657 shutdown_connections 2024-05-09T08:32:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.395+0000 7f287ccb1700 1 --2- 172.21.15.107:0/758038657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2878101380 0x7f287810d890 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.395+0000 7f287ccb1700 1 --2- 172.21.15.107:0/758038657 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28781069c0 0x7f2878106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.395+0000 7f287ccb1700 1 --2- 172.21.15.107:0/758038657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28781009c0 0x7f2878100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.395+0000 7f287ccb1700 1 -- 172.21.15.107:0/758038657 >> 172.21.15.107:0/758038657 conn(0x7f2878077820 msgr2=0x7f28780feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.395+0000 7f287ccb1700 1 -- 172.21.15.107:0/758038657 shutdown_connections 2024-05-09T08:32:38.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.396+0000 7f287ccb1700 1 -- 172.21.15.107:0/758038657 wait complete. 2024-05-09T08:32:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.396+0000 7f287ccb1700 1 Processor -- start 2024-05-09T08:32:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.396+0000 7f287ccb1700 1 -- start start 2024-05-09T08:32:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f287ccb1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28781009c0 0x7f287819c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f287ccb1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2878101380 0x7f287819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f287ccb1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28781069c0 0x7f2878196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f287ccb1700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2878110370 con 0x7f28781009c0 2024-05-09T08:32:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f287ccb1700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f28781101f0 con 0x7f2878101380 2024-05-09T08:32:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f287ccb1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f28781104f0 con 0x7f28781069c0 2024-05-09T08:32:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f287659c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28781009c0 0x7f287819c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f287659c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28781009c0 0x7f287819c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50270/0 (socket says 172.21.15.107:50270) 2024-05-09T08:32:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f287659c700 1 -- 172.21.15.107:0/322149080 learned_addr learned my addr 172.21.15.107:0/322149080 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f2875d9b700 1 --2- 172.21.15.107:0/322149080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2878101380 0x7f287819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:38.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f2876d9d700 1 --2- 172.21.15.107:0/322149080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28781069c0 0x7f2878196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:38.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f287659c700 1 -- 172.21.15.107:0/322149080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28781069c0 msgr2=0x7f2878196630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:38.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f287659c700 1 --2- 172.21.15.107:0/322149080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28781069c0 0x7f2878196630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:38.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f287659c700 1 -- 172.21.15.107:0/322149080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2878101380 msgr2=0x7f287819caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:38.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f287659c700 1 --2- 172.21.15.107:0/322149080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2878101380 0x7f287819caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:38.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f287659c700 1 -- 172.21.15.107:0/322149080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2878196f10 con 0x7f28781009c0 2024-05-09T08:32:38.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f287659c700 1 --2- 172.21.15.107:0/322149080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28781009c0 0x7f287819c560 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f286800b530 tx=0x7f286800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:38.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.397+0000 7f28677fe700 1 -- 172.21.15.107:0/322149080 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2868015020 con 0x7f28781009c0 2024-05-09T08:32:38.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.398+0000 7f287ccb1700 1 -- 172.21.15.107:0/322149080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2878197200 con 0x7f28781009c0 2024-05-09T08:32:38.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.398+0000 7f28677fe700 1 -- 172.21.15.107:0/322149080 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2868011ce0 con 0x7f28781009c0 2024-05-09T08:32:38.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.398+0000 7f28677fe700 1 -- 172.21.15.107:0/322149080 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2868010d60 con 0x7f28781009c0 2024-05-09T08:32:38.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.398+0000 7f287ccb1700 1 -- 172.21.15.107:0/322149080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28781a34e0 con 0x7f28781009c0 2024-05-09T08:32:38.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.399+0000 7f287ccb1700 1 -- 172.21.15.107:0/322149080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2878102100 con 0x7f28781009c0 2024-05-09T08:32:38.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.400+0000 7f28677fe700 1 -- 172.21.15.107:0/322149080 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f2868018020 con 0x7f28781009c0 2024-05-09T08:32:38.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.400+0000 7f28677fe700 1 --2- 172.21.15.107:0/322149080 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f28600a2c20 0x7f28600a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:38.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.401+0000 7f2875d9b700 1 --2- 172.21.15.107:0/322149080 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f28600a2c20 0x7f28600a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:38.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.401+0000 7f28677fe700 1 -- 172.21.15.107:0/322149080 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f28680cec40 con 0x7f28781009c0 2024-05-09T08:32:38.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.401+0000 7f2875d9b700 1 --2- 172.21.15.107:0/322149080 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f28600a2c20 0x7f28600a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f286c005fd0 tx=0x7f286c00a560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:38.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.405+0000 7f28677fe700 1 -- 172.21.15.107:0/322149080 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f286801f020 con 0x7f28781009c0 2024-05-09T08:32:38.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.596+0000 7f287ccb1700 1 -- 172.21.15.107:0/322149080 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2878197b60 con 0x7f28600a2c20 2024-05-09T08:32:38.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.605+0000 7f28677fe700 1 -- 172.21.15.107:0/322149080 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340477 (secure 0 0 0) 0x7f2878197b60 con 0x7f28600a2c20 2024-05-09T08:32:38.610 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:32:38.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.608+0000 7f287ccb1700 1 -- 172.21.15.107:0/322149080 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f28600a2c20 msgr2=0x7f28600a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.608+0000 7f287ccb1700 1 --2- 172.21.15.107:0/322149080 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f28600a2c20 0x7f28600a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f286c005fd0 tx=0x7f286c00a560 comp rx=0 tx=0).stop 2024-05-09T08:32:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.608+0000 7f287ccb1700 1 -- 172.21.15.107:0/322149080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28781009c0 msgr2=0x7f287819c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.608+0000 7f287ccb1700 1 --2- 172.21.15.107:0/322149080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28781009c0 0x7f287819c560 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f286800b530 tx=0x7f286800b8f0 comp rx=0 tx=0).stop 2024-05-09T08:32:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.608+0000 7f287ccb1700 1 -- 172.21.15.107:0/322149080 shutdown_connections 2024-05-09T08:32:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.608+0000 7f287ccb1700 1 --2- 172.21.15.107:0/322149080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28781069c0 0x7f2878196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.608+0000 7f287ccb1700 1 --2- 172.21.15.107:0/322149080 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f28600a2c20 0x7f28600a50e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.608+0000 7f287ccb1700 1 --2- 172.21.15.107:0/322149080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28781009c0 0x7f287819c560 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.608+0000 7f287ccb1700 1 --2- 172.21.15.107:0/322149080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2878101380 0x7f287819caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.608+0000 7f287ccb1700 1 -- 172.21.15.107:0/322149080 >> 172.21.15.107:0/322149080 conn(0x7f2878077820 msgr2=0x7f28781037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.608+0000 7f287ccb1700 1 -- 172.21.15.107:0/322149080 shutdown_connections 2024-05-09T08:32:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:38.608+0000 7f287ccb1700 1 -- 172.21.15.107:0/322149080 wait complete. 2024-05-09T08:32:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-05-09T08:32:39.487 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-05-09T08:32:37.481120+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658349,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":608,"num_read_kb":426,"num_write":350,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1182017,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":784,"ondisk_log_size":784,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":396528,"kb_used_data":14096,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124545296,"statfs":{"total":1151940427776,"available":1151534383104,"internally_reserved":0,"allocated":14434304,"data_stored":7960757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":37,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003639"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.493327+0000","last_change":"2024-05-09T08:31:00.695034+0000","last_active":"2024-05-09T08:32:25.493327+0000","last_peered":"2024-05-09T08:32:25.493327+0000","last_clean":"2024-05-09T08:32:25.493327+0000","last_became_active":"2024-05-09T08:31:00.694229+0000","last_became_peered":"2024-05-09T08:31:00.694229+0000","last_unstale":"2024-05-09T08:32:25.493327+0000","last_undegraded":"2024-05-09T08:32:25.493327+0000","last_fullsized":"2024-05-09T08:32:25.493327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:57:22.964848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.483449+0000","last_change":"2024-05-09T08:31:00.686219+0000","last_active":"2024-05-09T08:32:25.483449+0000","last_peered":"2024-05-09T08:32:25.483449+0000","last_clean":"2024-05-09T08:32:25.483449+0000","last_became_active":"2024-05-09T08:31:00.685911+0000","last_became_peered":"2024-05-09T08:31:00.685911+0000","last_unstale":"2024-05-09T08:32:25.483449+0000","last_undegraded":"2024-05-09T08:32:25.483449+0000","last_fullsized":"2024-05-09T08:32:25.483449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:28.246199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.473632+0000","last_change":"2024-05-09T08:31:00.694228+0000","last_active":"2024-05-09T08:32:25.473632+0000","last_peered":"2024-05-09T08:32:25.473632+0000","last_clean":"2024-05-09T08:32:25.473632+0000","last_became_active":"2024-05-09T08:31:00.693838+0000","last_became_peered":"2024-05-09T08:31:00.693838+0000","last_unstale":"2024-05-09T08:32:25.473632+0000","last_undegraded":"2024-05-09T08:32:25.473632+0000","last_fullsized":"2024-05-09T08:32:25.473632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:44.847966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.480390+0000","last_change":"2024-05-09T08:31:00.702770+0000","last_active":"2024-05-09T08:32:25.480390+0000","last_peered":"2024-05-09T08:32:25.480390+0000","last_clean":"2024-05-09T08:32:25.480390+0000","last_became_active":"2024-05-09T08:31:00.702424+0000","last_became_peered":"2024-05-09T08:31:00.702424+0000","last_unstale":"2024-05-09T08:32:25.480390+0000","last_undegraded":"2024-05-09T08:32:25.480390+0000","last_fullsized":"2024-05-09T08:32:25.480390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:24:05.228942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.482645+0000","last_change":"2024-05-09T08:31:00.688308+0000","last_active":"2024-05-09T08:32:25.482645+0000","last_peered":"2024-05-09T08:32:25.482645+0000","last_clean":"2024-05-09T08:32:25.482645+0000","last_became_active":"2024-05-09T08:31:00.687891+0000","last_became_peered":"2024-05-09T08:31:00.687891+0000","last_unstale":"2024-05-09T08:32:25.482645+0000","last_undegraded":"2024-05-09T08:32:25.482645+0000","last_fullsized":"2024-05-09T08:32:25.482645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:48:42.648940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.482462+0000","last_change":"2024-05-09T08:31:00.694705+0000","last_active":"2024-05-09T08:32:25.482462+0000","last_peered":"2024-05-09T08:32:25.482462+0000","last_clean":"2024-05-09T08:32:25.482462+0000","last_became_active":"2024-05-09T08:31:00.694361+0000","last_became_peered":"2024-05-09T08:31:00.694361+0000","last_unstale":"2024-05-09T08:32:25.482462+0000","last_undegraded":"2024-05-09T08:32:25.482462+0000","last_fullsized":"2024-05-09T08:32:25.482462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:07:23.137410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.481999+0000","last_change":"2024-05-09T08:31:00.689824+0000","last_active":"2024-05-09T08:32:25.481999+0000","last_peered":"2024-05-09T08:32:25.481999+0000","last_clean":"2024-05-09T08:32:25.481999+0000","last_became_active":"2024-05-09T08:31:00.689491+0000","last_became_peered":"2024-05-09T08:31:00.689491+0000","last_unstale":"2024-05-09T08:32:25.481999+0000","last_undegraded":"2024-05-09T08:32:25.481999+0000","last_fullsized":"2024-05-09T08:32:25.481999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:18.791225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.483621+0000","last_change":"2024-05-09T08:31:00.695014+0000","last_active":"2024-05-09T08:32:25.483621+0000","last_peered":"2024-05-09T08:32:25.483621+0000","last_clean":"2024-05-09T08:32:25.483621+0000","last_became_active":"2024-05-09T08:31:00.694683+0000","last_became_peered":"2024-05-09T08:31:00.694683+0000","last_unstale":"2024-05-09T08:32:25.483621+0000","last_undegraded":"2024-05-09T08:32:25.483621+0000","last_fullsized":"2024-05-09T08:32:25.483621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:08.570258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.494960+0000","last_change":"2024-05-09T08:31:00.701538+0000","last_active":"2024-05-09T08:32:25.494960+0000","last_peered":"2024-05-09T08:32:25.494960+0000","last_clean":"2024-05-09T08:32:25.494960+0000","last_became_active":"2024-05-09T08:31:00.701226+0000","last_became_peered":"2024-05-09T08:31:00.701226+0000","last_unstale":"2024-05-09T08:32:25.494960+0000","last_undegraded":"2024-05-09T08:32:25.494960+0000","last_fullsized":"2024-05-09T08:32:25.494960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:14:50.853063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484188+0000","last_change":"2024-05-09T08:31:00.698145+0000","last_active":"2024-05-09T08:32:25.484188+0000","last_peered":"2024-05-09T08:32:25.484188+0000","last_clean":"2024-05-09T08:32:25.484188+0000","last_became_active":"2024-05-09T08:31:00.697761+0000","last_became_peered":"2024-05-09T08:31:00.697761+0000","last_unstale":"2024-05-09T08:32:25.484188+0000","last_undegraded":"2024-05-09T08:32:25.484188+0000","last_fullsized":"2024-05-09T08:32:25.484188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:26.094013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.490789+0000","last_change":"2024-05-09T08:31:00.699520+0000","last_active":"2024-05-09T08:32:25.490789+0000","last_peered":"2024-05-09T08:32:25.490789+0000","last_clean":"2024-05-09T08:32:25.490789+0000","last_became_active":"2024-05-09T08:31:00.697579+0000","last_became_peered":"2024-05-09T08:31:00.697579+0000","last_unstale":"2024-05-09T08:32:25.490789+0000","last_undegraded":"2024-05-09T08:32:25.490789+0000","last_fullsized":"2024-05-09T08:32:25.490789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:26:34.324474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.478295+0000","last_change":"2024-05-09T08:31:00.693356+0000","last_active":"2024-05-09T08:32:25.478295+0000","last_peered":"2024-05-09T08:32:25.478295+0000","last_clean":"2024-05-09T08:32:25.478295+0000","last_became_active":"2024-05-09T08:31:00.692985+0000","last_became_peered":"2024-05-09T08:31:00.692985+0000","last_unstale":"2024-05-09T08:32:25.478295+0000","last_undegraded":"2024-05-09T08:32:25.478295+0000","last_fullsized":"2024-05-09T08:32:25.478295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:49:01.115497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484938+0000","last_change":"2024-05-09T08:31:00.677768+0000","last_active":"2024-05-09T08:32:25.484938+0000","last_peered":"2024-05-09T08:32:25.484938+0000","last_clean":"2024-05-09T08:32:25.484938+0000","last_became_active":"2024-05-09T08:31:00.677386+0000","last_became_peered":"2024-05-09T08:31:00.677386+0000","last_unstale":"2024-05-09T08:32:25.484938+0000","last_undegraded":"2024-05-09T08:32:25.484938+0000","last_fullsized":"2024-05-09T08:32:25.484938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:55.363324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484172+0000","last_change":"2024-05-09T08:31:00.689609+0000","last_active":"2024-05-09T08:32:25.484172+0000","last_peered":"2024-05-09T08:32:25.484172+0000","last_clean":"2024-05-09T08:32:25.484172+0000","last_became_active":"2024-05-09T08:31:00.689303+0000","last_became_peered":"2024-05-09T08:31:00.689303+0000","last_unstale":"2024-05-09T08:32:25.484172+0000","last_undegraded":"2024-05-09T08:32:25.484172+0000","last_fullsized":"2024-05-09T08:32:25.484172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:15:58.672460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.482196+0000","last_change":"2024-05-09T08:31:00.704708+0000","last_active":"2024-05-09T08:32:25.482196+0000","last_peered":"2024-05-09T08:32:25.482196+0000","last_clean":"2024-05-09T08:32:25.482196+0000","last_became_active":"2024-05-09T08:31:00.704269+0000","last_became_peered":"2024-05-09T08:31:00.704269+0000","last_unstale":"2024-05-09T08:32:25.482196+0000","last_undegraded":"2024-05-09T08:32:25.482196+0000","last_fullsized":"2024-05-09T08:32:25.482196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:17:44.580420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.493676+0000","last_change":"2024-05-09T08:31:00.703089+0000","last_active":"2024-05-09T08:32:25.493676+0000","last_peered":"2024-05-09T08:32:25.493676+0000","last_clean":"2024-05-09T08:32:25.493676+0000","last_became_active":"2024-05-09T08:31:00.702531+0000","last_became_peered":"2024-05-09T08:31:00.702531+0000","last_unstale":"2024-05-09T08:32:25.493676+0000","last_undegraded":"2024-05-09T08:32:25.493676+0000","last_fullsized":"2024-05-09T08:32:25.493676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:07:39.887748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.493600+0000","last_change":"2024-05-09T08:31:00.695332+0000","last_active":"2024-05-09T08:32:25.493600+0000","last_peered":"2024-05-09T08:32:25.493600+0000","last_clean":"2024-05-09T08:32:25.493600+0000","last_became_active":"2024-05-09T08:31:00.694703+0000","last_became_peered":"2024-05-09T08:31:00.694703+0000","last_unstale":"2024-05-09T08:32:25.493600+0000","last_undegraded":"2024-05-09T08:32:25.493600+0000","last_fullsized":"2024-05-09T08:32:25.493600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:53.727922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.488840+0000","last_change":"2024-05-09T08:31:00.689871+0000","last_active":"2024-05-09T08:32:25.488840+0000","last_peered":"2024-05-09T08:32:25.488840+0000","last_clean":"2024-05-09T08:32:25.488840+0000","last_became_active":"2024-05-09T08:31:00.689498+0000","last_became_peered":"2024-05-09T08:31:00.689498+0000","last_unstale":"2024-05-09T08:32:25.488840+0000","last_undegraded":"2024-05-09T08:32:25.488840+0000","last_fullsized":"2024-05-09T08:32:25.488840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:18:14.123442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.483055+0000","last_change":"2024-05-09T08:31:00.697175+0000","last_active":"2024-05-09T08:32:25.483055+0000","last_peered":"2024-05-09T08:32:25.483055+0000","last_clean":"2024-05-09T08:32:25.483055+0000","last_became_active":"2024-05-09T08:31:00.694103+0000","last_became_peered":"2024-05-09T08:31:00.694103+0000","last_unstale":"2024-05-09T08:32:25.483055+0000","last_undegraded":"2024-05-09T08:32:25.483055+0000","last_fullsized":"2024-05-09T08:32:25.483055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:41:12.549854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.797212+0000","last_change":"2024-05-09T08:31:56.797212+0000","last_active":"2024-05-09T08:31:56.797212+0000","last_peered":"2024-05-09T08:31:56.797212+0000","last_clean":"2024-05-09T08:31:56.797212+0000","last_became_active":"2024-05-09T08:31:56.796786+0000","last_became_peered":"2024-05-09T08:31:56.796786+0000","last_unstale":"2024-05-09T08:31:56.797212+0000","last_undegraded":"2024-05-09T08:31:56.797212+0000","last_fullsized":"2024-05-09T08:31:56.797212+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:55.270522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.578564+0000","last_change":"2024-05-09T08:31:56.578564+0000","last_active":"2024-05-09T08:31:56.578564+0000","last_peered":"2024-05-09T08:31:56.578564+0000","last_clean":"2024-05-09T08:31:56.578564+0000","last_became_active":"2024-05-09T08:31:56.571246+0000","last_became_peered":"2024-05-09T08:31:56.571246+0000","last_unstale":"2024-05-09T08:31:56.578564+0000","last_undegraded":"2024-05-09T08:31:56.578564+0000","last_fullsized":"2024-05-09T08:31:56.578564+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:15:10.322529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.479738+0000","last_change":"2024-05-09T08:31:00.696023+0000","last_active":"2024-05-09T08:32:25.479738+0000","last_peered":"2024-05-09T08:32:25.479738+0000","last_clean":"2024-05-09T08:32:25.479738+0000","last_became_active":"2024-05-09T08:31:00.695677+0000","last_became_peered":"2024-05-09T08:31:00.695677+0000","last_unstale":"2024-05-09T08:32:25.479738+0000","last_undegraded":"2024-05-09T08:32:25.479738+0000","last_fullsized":"2024-05-09T08:32:25.479738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:55.989133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566480+0000","last_change":"2024-05-09T08:31:04.859302+0000","last_active":"2024-05-09T08:31:56.566480+0000","last_peered":"2024-05-09T08:31:56.566480+0000","last_clean":"2024-05-09T08:31:56.566480+0000","last_became_active":"2024-05-09T08:31:04.858897+0000","last_became_peered":"2024-05-09T08:31:04.858897+0000","last_unstale":"2024-05-09T08:31:56.566480+0000","last_undegraded":"2024-05-09T08:31:56.566480+0000","last_fullsized":"2024-05-09T08:31:56.566480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:25:32.970833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.796130+0000","last_change":"2024-05-09T08:30:59.824451+0000","last_active":"2024-05-09T08:31:56.796130+0000","last_peered":"2024-05-09T08:31:56.796130+0000","last_clean":"2024-05-09T08:31:56.796130+0000","last_became_active":"2024-05-09T08:30:59.823964+0000","last_became_peered":"2024-05-09T08:30:59.823964+0000","last_unstale":"2024-05-09T08:31:56.796130+0000","last_undegraded":"2024-05-09T08:31:56.796130+0000","last_fullsized":"2024-05-09T08:31:56.796130+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:50:13.562262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.489318+0000","last_change":"2024-05-09T08:31:00.701015+0000","last_active":"2024-05-09T08:32:25.489318+0000","last_peered":"2024-05-09T08:32:25.489318+0000","last_clean":"2024-05-09T08:32:25.489318+0000","last_became_active":"2024-05-09T08:31:00.700654+0000","last_became_peered":"2024-05-09T08:31:00.700654+0000","last_unstale":"2024-05-09T08:32:25.489318+0000","last_undegraded":"2024-05-09T08:32:25.489318+0000","last_fullsized":"2024-05-09T08:32:25.489318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:03.710790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566700+0000","last_change":"2024-05-09T08:31:04.858453+0000","last_active":"2024-05-09T08:31:56.566700+0000","last_peered":"2024-05-09T08:31:56.566700+0000","last_clean":"2024-05-09T08:31:56.566700+0000","last_became_active":"2024-05-09T08:31:04.857974+0000","last_became_peered":"2024-05-09T08:31:04.857974+0000","last_unstale":"2024-05-09T08:31:56.566700+0000","last_undegraded":"2024-05-09T08:31:56.566700+0000","last_fullsized":"2024-05-09T08:31:56.566700+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:42:52.868625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.569135+0000","last_change":"2024-05-09T08:30:59.825974+0000","last_active":"2024-05-09T08:31:56.569135+0000","last_peered":"2024-05-09T08:31:56.569135+0000","last_clean":"2024-05-09T08:31:56.569135+0000","last_became_active":"2024-05-09T08:30:59.825671+0000","last_became_peered":"2024-05-09T08:30:59.825671+0000","last_unstale":"2024-05-09T08:31:56.569135+0000","last_undegraded":"2024-05-09T08:31:56.569135+0000","last_fullsized":"2024-05-09T08:31:56.569135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:38:20.179466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.494048+0000","last_change":"2024-05-09T08:31:00.698324+0000","last_active":"2024-05-09T08:32:25.494048+0000","last_peered":"2024-05-09T08:32:25.494048+0000","last_clean":"2024-05-09T08:32:25.494048+0000","last_became_active":"2024-05-09T08:31:00.697992+0000","last_became_peered":"2024-05-09T08:31:00.697992+0000","last_unstale":"2024-05-09T08:32:25.494048+0000","last_undegraded":"2024-05-09T08:32:25.494048+0000","last_fullsized":"2024-05-09T08:32:25.494048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:25:20.786423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566561+0000","last_change":"2024-05-09T08:31:04.861946+0000","last_active":"2024-05-09T08:31:56.566561+0000","last_peered":"2024-05-09T08:31:56.566561+0000","last_clean":"2024-05-09T08:31:56.566561+0000","last_became_active":"2024-05-09T08:31:04.861511+0000","last_became_peered":"2024-05-09T08:31:04.861511+0000","last_unstale":"2024-05-09T08:31:56.566561+0000","last_undegraded":"2024-05-09T08:31:56.566561+0000","last_fullsized":"2024-05-09T08:31:56.566561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:36:44.580499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.568935+0000","last_change":"2024-05-09T08:30:59.829961+0000","last_active":"2024-05-09T08:31:56.568935+0000","last_peered":"2024-05-09T08:31:56.568935+0000","last_clean":"2024-05-09T08:31:56.568935+0000","last_became_active":"2024-05-09T08:30:59.828057+0000","last_became_peered":"2024-05-09T08:30:59.828057+0000","last_unstale":"2024-05-09T08:31:56.568935+0000","last_undegraded":"2024-05-09T08:31:56.568935+0000","last_fullsized":"2024-05-09T08:31:56.568935+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:39:18.229438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.493731+0000","last_change":"2024-05-09T08:31:00.700285+0000","last_active":"2024-05-09T08:32:25.493731+0000","last_peered":"2024-05-09T08:32:25.493731+0000","last_clean":"2024-05-09T08:32:25.493731+0000","last_became_active":"2024-05-09T08:31:00.699958+0000","last_became_peered":"2024-05-09T08:31:00.699958+0000","last_unstale":"2024-05-09T08:32:25.493731+0000","last_undegraded":"2024-05-09T08:32:25.493731+0000","last_fullsized":"2024-05-09T08:32:25.493731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:48.934827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566653+0000","last_change":"2024-05-09T08:31:04.857060+0000","last_active":"2024-05-09T08:31:56.566653+0000","last_peered":"2024-05-09T08:31:56.566653+0000","last_clean":"2024-05-09T08:31:56.566653+0000","last_became_active":"2024-05-09T08:31:04.855261+0000","last_became_peered":"2024-05-09T08:31:04.855261+0000","last_unstale":"2024-05-09T08:31:56.566653+0000","last_undegraded":"2024-05-09T08:31:56.566653+0000","last_fullsized":"2024-05-09T08:31:56.566653+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:08:08.230524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":109,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:34.170208+0000","last_change":"2024-05-09T08:30:59.831089+0000","last_active":"2024-05-09T08:32:34.170208+0000","last_peered":"2024-05-09T08:32:34.170208+0000","last_clean":"2024-05-09T08:32:34.170208+0000","last_became_active":"2024-05-09T08:30:59.830572+0000","last_became_peered":"2024-05-09T08:30:59.830572+0000","last_unstale":"2024-05-09T08:32:34.170208+0000","last_undegraded":"2024-05-09T08:32:34.170208+0000","last_fullsized":"2024-05-09T08:32:34.170208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:39:10.495104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.478511+0000","last_change":"2024-05-09T08:31:00.688792+0000","last_active":"2024-05-09T08:32:25.478511+0000","last_peered":"2024-05-09T08:32:25.478511+0000","last_clean":"2024-05-09T08:32:25.478511+0000","last_became_active":"2024-05-09T08:31:00.688386+0000","last_became_peered":"2024-05-09T08:31:00.688386+0000","last_unstale":"2024-05-09T08:32:25.478511+0000","last_undegraded":"2024-05-09T08:32:25.478511+0000","last_fullsized":"2024-05-09T08:32:25.478511+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:24.287333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566566+0000","last_change":"2024-05-09T08:31:04.862287+0000","last_active":"2024-05-09T08:31:56.566566+0000","last_peered":"2024-05-09T08:31:56.566566+0000","last_clean":"2024-05-09T08:31:56.566566+0000","last_became_active":"2024-05-09T08:31:04.861866+0000","last_became_peered":"2024-05-09T08:31:04.861866+0000","last_unstale":"2024-05-09T08:31:56.566566+0000","last_undegraded":"2024-05-09T08:31:56.566566+0000","last_fullsized":"2024-05-09T08:31:56.566566+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:14:21.552787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.669987+0000","last_change":"2024-05-09T08:30:59.824857+0000","last_active":"2024-05-09T08:31:56.669987+0000","last_peered":"2024-05-09T08:31:56.669987+0000","last_clean":"2024-05-09T08:31:56.669987+0000","last_became_active":"2024-05-09T08:30:59.824363+0000","last_became_peered":"2024-05-09T08:30:59.824363+0000","last_unstale":"2024-05-09T08:31:56.669987+0000","last_undegraded":"2024-05-09T08:31:56.669987+0000","last_fullsized":"2024-05-09T08:31:56.669987+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:20:32.267237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.483663+0000","last_change":"2024-05-09T08:31:00.694440+0000","last_active":"2024-05-09T08:32:25.483663+0000","last_peered":"2024-05-09T08:32:25.483663+0000","last_clean":"2024-05-09T08:32:25.483663+0000","last_became_active":"2024-05-09T08:31:00.694085+0000","last_became_peered":"2024-05-09T08:31:00.694085+0000","last_unstale":"2024-05-09T08:32:25.483663+0000","last_undegraded":"2024-05-09T08:32:25.483663+0000","last_fullsized":"2024-05-09T08:32:25.483663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:03:24.697479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.574514+0000","last_change":"2024-05-09T08:31:56.574514+0000","last_active":"2024-05-09T08:31:56.574514+0000","last_peered":"2024-05-09T08:31:56.574514+0000","last_clean":"2024-05-09T08:31:56.574514+0000","last_became_active":"2024-05-09T08:31:56.573786+0000","last_became_peered":"2024-05-09T08:31:56.573786+0000","last_unstale":"2024-05-09T08:31:56.574514+0000","last_undegraded":"2024-05-09T08:31:56.574514+0000","last_fullsized":"2024-05-09T08:31:56.574514+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:17.626431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.776789+0000","last_change":"2024-05-09T08:30:59.818373+0000","last_active":"2024-05-09T08:31:56.776789+0000","last_peered":"2024-05-09T08:31:56.776789+0000","last_clean":"2024-05-09T08:31:56.776789+0000","last_became_active":"2024-05-09T08:30:59.818023+0000","last_became_peered":"2024-05-09T08:30:59.818023+0000","last_unstale":"2024-05-09T08:31:56.776789+0000","last_undegraded":"2024-05-09T08:31:56.776789+0000","last_fullsized":"2024-05-09T08:31:56.776789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:23:01.994257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.492336+0000","last_change":"2024-05-09T08:31:00.689506+0000","last_active":"2024-05-09T08:32:25.492336+0000","last_peered":"2024-05-09T08:32:25.492336+0000","last_clean":"2024-05-09T08:32:25.492336+0000","last_became_active":"2024-05-09T08:31:00.689151+0000","last_became_peered":"2024-05-09T08:31:00.689151+0000","last_unstale":"2024-05-09T08:32:25.492336+0000","last_undegraded":"2024-05-09T08:32:25.492336+0000","last_fullsized":"2024-05-09T08:32:25.492336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:48:13.539509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.572238+0000","last_change":"2024-05-09T08:31:04.859215+0000","last_active":"2024-05-09T08:31:56.572238+0000","last_peered":"2024-05-09T08:31:56.572238+0000","last_clean":"2024-05-09T08:31:56.572238+0000","last_became_active":"2024-05-09T08:31:04.858759+0000","last_became_peered":"2024-05-09T08:31:04.858759+0000","last_unstale":"2024-05-09T08:31:56.572238+0000","last_undegraded":"2024-05-09T08:31:56.572238+0000","last_fullsized":"2024-05-09T08:31:56.572238+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:41:42.124234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.574342+0000","last_change":"2024-05-09T08:30:59.824260+0000","last_active":"2024-05-09T08:31:56.574342+0000","last_peered":"2024-05-09T08:31:56.574342+0000","last_clean":"2024-05-09T08:31:56.574342+0000","last_became_active":"2024-05-09T08:30:59.823959+0000","last_became_peered":"2024-05-09T08:30:59.823959+0000","last_unstale":"2024-05-09T08:31:56.574342+0000","last_undegraded":"2024-05-09T08:31:56.574342+0000","last_fullsized":"2024-05-09T08:31:56.574342+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:48:42.354279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.480127+0000","last_change":"2024-05-09T08:31:00.687810+0000","last_active":"2024-05-09T08:32:25.480127+0000","last_peered":"2024-05-09T08:32:25.480127+0000","last_clean":"2024-05-09T08:32:25.480127+0000","last_became_active":"2024-05-09T08:31:00.687486+0000","last_became_peered":"2024-05-09T08:31:00.687486+0000","last_unstale":"2024-05-09T08:32:25.480127+0000","last_undegraded":"2024-05-09T08:32:25.480127+0000","last_fullsized":"2024-05-09T08:32:25.480127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:51.428208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.796119+0000","last_change":"2024-05-09T08:31:04.853222+0000","last_active":"2024-05-09T08:31:56.796119+0000","last_peered":"2024-05-09T08:31:56.796119+0000","last_clean":"2024-05-09T08:31:56.796119+0000","last_became_active":"2024-05-09T08:31:04.852676+0000","last_became_peered":"2024-05-09T08:31:04.852676+0000","last_unstale":"2024-05-09T08:31:56.796119+0000","last_undegraded":"2024-05-09T08:31:56.796119+0000","last_fullsized":"2024-05-09T08:31:56.796119+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:58:31.949938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:32.166257+0000","last_change":"2024-05-09T08:30:59.824653+0000","last_active":"2024-05-09T08:32:32.166257+0000","last_peered":"2024-05-09T08:32:32.166257+0000","last_clean":"2024-05-09T08:32:32.166257+0000","last_became_active":"2024-05-09T08:30:59.824277+0000","last_became_peered":"2024-05-09T08:30:59.824277+0000","last_unstale":"2024-05-09T08:32:32.166257+0000","last_undegraded":"2024-05-09T08:32:32.166257+0000","last_fullsized":"2024-05-09T08:32:32.166257+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:45:20.340629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.493911+0000","last_change":"2024-05-09T08:31:00.697484+0000","last_active":"2024-05-09T08:32:25.493911+0000","last_peered":"2024-05-09T08:32:25.493911+0000","last_clean":"2024-05-09T08:32:25.493911+0000","last_became_active":"2024-05-09T08:31:00.697095+0000","last_became_peered":"2024-05-09T08:31:00.697095+0000","last_unstale":"2024-05-09T08:32:25.493911+0000","last_undegraded":"2024-05-09T08:32:25.493911+0000","last_fullsized":"2024-05-09T08:32:25.493911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:29:40.702057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.576003+0000","last_change":"2024-05-09T08:31:56.574176+0000","last_active":"2024-05-09T08:31:56.576003+0000","last_peered":"2024-05-09T08:31:56.576003+0000","last_clean":"2024-05-09T08:31:56.576003+0000","last_became_active":"2024-05-09T08:31:56.573556+0000","last_became_peered":"2024-05-09T08:31:56.573556+0000","last_unstale":"2024-05-09T08:31:56.576003+0000","last_undegraded":"2024-05-09T08:31:56.576003+0000","last_fullsized":"2024-05-09T08:31:56.576003+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:50.685712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.571430+0000","last_change":"2024-05-09T08:30:59.822683+0000","last_active":"2024-05-09T08:31:56.571430+0000","last_peered":"2024-05-09T08:31:56.571430+0000","last_clean":"2024-05-09T08:31:56.571430+0000","last_became_active":"2024-05-09T08:30:59.822150+0000","last_became_peered":"2024-05-09T08:30:59.822150+0000","last_unstale":"2024-05-09T08:31:56.571430+0000","last_undegraded":"2024-05-09T08:31:56.571430+0000","last_fullsized":"2024-05-09T08:31:56.571430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:51:32.782076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.481801+0000","last_change":"2024-05-09T08:31:00.694165+0000","last_active":"2024-05-09T08:32:25.481801+0000","last_peered":"2024-05-09T08:32:25.481801+0000","last_clean":"2024-05-09T08:32:25.481801+0000","last_became_active":"2024-05-09T08:31:00.693150+0000","last_became_peered":"2024-05-09T08:31:00.693150+0000","last_unstale":"2024-05-09T08:32:25.481801+0000","last_undegraded":"2024-05-09T08:32:25.481801+0000","last_fullsized":"2024-05-09T08:32:25.481801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:29.278078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566842+0000","last_change":"2024-05-09T08:31:04.858785+0000","last_active":"2024-05-09T08:31:56.566842+0000","last_peered":"2024-05-09T08:31:56.566842+0000","last_clean":"2024-05-09T08:31:56.566842+0000","last_became_active":"2024-05-09T08:31:04.857947+0000","last_became_peered":"2024-05-09T08:31:04.857947+0000","last_unstale":"2024-05-09T08:31:56.566842+0000","last_undegraded":"2024-05-09T08:31:56.566842+0000","last_fullsized":"2024-05-09T08:31:56.566842+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:55:48.595607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.576695+0000","last_change":"2024-05-09T08:30:59.822946+0000","last_active":"2024-05-09T08:31:56.576695+0000","last_peered":"2024-05-09T08:31:56.576695+0000","last_clean":"2024-05-09T08:31:56.576695+0000","last_became_active":"2024-05-09T08:30:59.822174+0000","last_became_peered":"2024-05-09T08:30:59.822174+0000","last_unstale":"2024-05-09T08:31:56.576695+0000","last_undegraded":"2024-05-09T08:31:56.576695+0000","last_fullsized":"2024-05-09T08:31:56.576695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:14.224033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.493668+0000","last_change":"2024-05-09T08:31:00.704781+0000","last_active":"2024-05-09T08:32:25.493668+0000","last_peered":"2024-05-09T08:32:25.493668+0000","last_clean":"2024-05-09T08:32:25.493668+0000","last_became_active":"2024-05-09T08:31:00.704170+0000","last_became_peered":"2024-05-09T08:31:00.704170+0000","last_unstale":"2024-05-09T08:32:25.493668+0000","last_undegraded":"2024-05-09T08:32:25.493668+0000","last_fullsized":"2024-05-09T08:32:25.493668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:27:47.402290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.567049+0000","last_change":"2024-05-09T08:31:04.855648+0000","last_active":"2024-05-09T08:31:56.567049+0000","last_peered":"2024-05-09T08:31:56.567049+0000","last_clean":"2024-05-09T08:31:56.567049+0000","last_became_active":"2024-05-09T08:31:04.854678+0000","last_became_peered":"2024-05-09T08:31:04.854678+0000","last_unstale":"2024-05-09T08:31:56.567049+0000","last_undegraded":"2024-05-09T08:31:56.567049+0000","last_fullsized":"2024-05-09T08:31:56.567049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:53:46.039172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'35","reported_seq":217,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:33.180049+0000","last_change":"2024-05-09T08:30:59.826498+0000","last_active":"2024-05-09T08:32:33.180049+0000","last_peered":"2024-05-09T08:32:33.180049+0000","last_clean":"2024-05-09T08:32:33.180049+0000","last_became_active":"2024-05-09T08:30:59.826050+0000","last_became_peered":"2024-05-09T08:30:59.826050+0000","last_unstale":"2024-05-09T08:32:33.180049+0000","last_undegraded":"2024-05-09T08:32:33.180049+0000","last_fullsized":"2024-05-09T08:32:33.180049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:19.920901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63605,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":155,"num_read_kb":140,"num_write":36,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.485055+0000","last_change":"2024-05-09T08:31:00.688379+0000","last_active":"2024-05-09T08:32:25.485055+0000","last_peered":"2024-05-09T08:32:25.485055+0000","last_clean":"2024-05-09T08:32:25.485055+0000","last_became_active":"2024-05-09T08:31:00.688030+0000","last_became_peered":"2024-05-09T08:31:00.688030+0000","last_unstale":"2024-05-09T08:32:25.485055+0000","last_undegraded":"2024-05-09T08:32:25.485055+0000","last_fullsized":"2024-05-09T08:32:25.485055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:17.248154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.576929+0000","last_change":"2024-05-09T08:31:04.858513+0000","last_active":"2024-05-09T08:31:56.576929+0000","last_peered":"2024-05-09T08:31:56.576929+0000","last_clean":"2024-05-09T08:31:56.576929+0000","last_became_active":"2024-05-09T08:31:04.857988+0000","last_became_peered":"2024-05-09T08:31:04.857988+0000","last_unstale":"2024-05-09T08:31:56.576929+0000","last_undegraded":"2024-05-09T08:31:56.576929+0000","last_fullsized":"2024-05-09T08:31:56.576929+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:59:24.264503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566645+0000","last_change":"2024-05-09T08:30:59.823403+0000","last_active":"2024-05-09T08:31:56.566645+0000","last_peered":"2024-05-09T08:31:56.566645+0000","last_clean":"2024-05-09T08:31:56.566645+0000","last_became_active":"2024-05-09T08:30:59.823066+0000","last_became_peered":"2024-05-09T08:30:59.823066+0000","last_unstale":"2024-05-09T08:31:56.566645+0000","last_undegraded":"2024-05-09T08:31:56.566645+0000","last_fullsized":"2024-05-09T08:31:56.566645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:16:39.157895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484172+0000","last_change":"2024-05-09T08:31:00.688009+0000","last_active":"2024-05-09T08:32:25.484172+0000","last_peered":"2024-05-09T08:32:25.484172+0000","last_clean":"2024-05-09T08:32:25.484172+0000","last_became_active":"2024-05-09T08:31:00.687679+0000","last_became_peered":"2024-05-09T08:31:00.687679+0000","last_unstale":"2024-05-09T08:32:25.484172+0000","last_undegraded":"2024-05-09T08:32:25.484172+0000","last_fullsized":"2024-05-09T08:32:25.484172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:44.403639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.570884+0000","last_change":"2024-05-09T08:31:04.851880+0000","last_active":"2024-05-09T08:31:56.570884+0000","last_peered":"2024-05-09T08:31:56.570884+0000","last_clean":"2024-05-09T08:31:56.570884+0000","last_became_active":"2024-05-09T08:31:04.851316+0000","last_became_peered":"2024-05-09T08:31:04.851316+0000","last_unstale":"2024-05-09T08:31:56.570884+0000","last_undegraded":"2024-05-09T08:31:56.570884+0000","last_fullsized":"2024-05-09T08:31:56.570884+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:53.144996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.575140+0000","last_change":"2024-05-09T08:30:59.823411+0000","last_active":"2024-05-09T08:31:56.575140+0000","last_peered":"2024-05-09T08:31:56.575140+0000","last_clean":"2024-05-09T08:31:56.575140+0000","last_became_active":"2024-05-09T08:30:59.822854+0000","last_became_peered":"2024-05-09T08:30:59.822854+0000","last_unstale":"2024-05-09T08:31:56.575140+0000","last_undegraded":"2024-05-09T08:31:56.575140+0000","last_fullsized":"2024-05-09T08:31:56.575140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:36:03.666937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.486496+0000","last_change":"2024-05-09T08:31:56.777818+0000","last_active":"2024-05-09T08:32:25.486496+0000","last_peered":"2024-05-09T08:32:25.486496+0000","last_clean":"2024-05-09T08:32:25.486496+0000","last_became_active":"2024-05-09T08:31:56.777555+0000","last_became_peered":"2024-05-09T08:31:56.777555+0000","last_unstale":"2024-05-09T08:32:25.486496+0000","last_undegraded":"2024-05-09T08:32:25.486496+0000","last_fullsized":"2024-05-09T08:32:25.486496+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:16:17.695073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.567497+0000","last_change":"2024-05-09T08:31:04.859338+0000","last_active":"2024-05-09T08:31:56.567497+0000","last_peered":"2024-05-09T08:31:56.567497+0000","last_clean":"2024-05-09T08:31:56.567497+0000","last_became_active":"2024-05-09T08:31:04.858951+0000","last_became_peered":"2024-05-09T08:31:04.858951+0000","last_unstale":"2024-05-09T08:31:56.567497+0000","last_undegraded":"2024-05-09T08:31:56.567497+0000","last_fullsized":"2024-05-09T08:31:56.567497+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:26:46.738359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":111,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:34.169548+0000","last_change":"2024-05-09T08:30:59.823710+0000","last_active":"2024-05-09T08:32:34.169548+0000","last_peered":"2024-05-09T08:32:34.169548+0000","last_clean":"2024-05-09T08:32:34.169548+0000","last_became_active":"2024-05-09T08:30:59.823213+0000","last_became_peered":"2024-05-09T08:30:59.823213+0000","last_unstale":"2024-05-09T08:32:34.169548+0000","last_undegraded":"2024-05-09T08:32:34.169548+0000","last_fullsized":"2024-05-09T08:32:34.169548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:09.134942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.483614+0000","last_change":"2024-05-09T08:31:00.693735+0000","last_active":"2024-05-09T08:32:25.483614+0000","last_peered":"2024-05-09T08:32:25.483614+0000","last_clean":"2024-05-09T08:32:25.483614+0000","last_became_active":"2024-05-09T08:31:00.693401+0000","last_became_peered":"2024-05-09T08:31:00.693401+0000","last_unstale":"2024-05-09T08:32:25.483614+0000","last_undegraded":"2024-05-09T08:32:25.483614+0000","last_fullsized":"2024-05-09T08:32:25.483614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:58:41.429854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.777034+0000","last_change":"2024-05-09T08:31:04.861289+0000","last_active":"2024-05-09T08:31:56.777034+0000","last_peered":"2024-05-09T08:31:56.777034+0000","last_clean":"2024-05-09T08:31:56.777034+0000","last_became_active":"2024-05-09T08:31:04.860915+0000","last_became_peered":"2024-05-09T08:31:04.860915+0000","last_unstale":"2024-05-09T08:31:56.777034+0000","last_undegraded":"2024-05-09T08:31:56.777034+0000","last_fullsized":"2024-05-09T08:31:56.777034+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:01:45.724961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.570589+0000","last_change":"2024-05-09T08:30:59.827631+0000","last_active":"2024-05-09T08:31:56.570589+0000","last_peered":"2024-05-09T08:31:56.570589+0000","last_clean":"2024-05-09T08:31:56.570589+0000","last_became_active":"2024-05-09T08:30:59.827295+0000","last_became_peered":"2024-05-09T08:30:59.827295+0000","last_unstale":"2024-05-09T08:31:56.570589+0000","last_undegraded":"2024-05-09T08:31:56.570589+0000","last_fullsized":"2024-05-09T08:31:56.570589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:35.282564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.483419+0000","last_change":"2024-05-09T08:31:00.697712+0000","last_active":"2024-05-09T08:32:25.483419+0000","last_peered":"2024-05-09T08:32:25.483419+0000","last_clean":"2024-05-09T08:32:25.483419+0000","last_became_active":"2024-05-09T08:31:00.697385+0000","last_became_peered":"2024-05-09T08:31:00.697385+0000","last_unstale":"2024-05-09T08:32:25.483419+0000","last_undegraded":"2024-05-09T08:32:25.483419+0000","last_fullsized":"2024-05-09T08:32:25.483419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:09.087279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.575131+0000","last_change":"2024-05-09T08:31:04.860691+0000","last_active":"2024-05-09T08:31:56.575131+0000","last_peered":"2024-05-09T08:31:56.575131+0000","last_clean":"2024-05-09T08:31:56.575131+0000","last_became_active":"2024-05-09T08:31:04.858458+0000","last_became_peered":"2024-05-09T08:31:04.858458+0000","last_unstale":"2024-05-09T08:31:56.575131+0000","last_undegraded":"2024-05-09T08:31:56.575131+0000","last_fullsized":"2024-05-09T08:31:56.575131+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:16.274359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.569480+0000","last_change":"2024-05-09T08:30:59.825188+0000","last_active":"2024-05-09T08:31:56.569480+0000","last_peered":"2024-05-09T08:31:56.569480+0000","last_clean":"2024-05-09T08:31:56.569480+0000","last_became_active":"2024-05-09T08:30:59.824855+0000","last_became_peered":"2024-05-09T08:30:59.824855+0000","last_unstale":"2024-05-09T08:31:56.569480+0000","last_undegraded":"2024-05-09T08:31:56.569480+0000","last_fullsized":"2024-05-09T08:31:56.569480+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:17:21.753652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.492648+0000","last_change":"2024-05-09T08:31:00.695862+0000","last_active":"2024-05-09T08:32:25.492648+0000","last_peered":"2024-05-09T08:32:25.492648+0000","last_clean":"2024-05-09T08:32:25.492648+0000","last_became_active":"2024-05-09T08:31:00.695404+0000","last_became_peered":"2024-05-09T08:31:00.695404+0000","last_unstale":"2024-05-09T08:32:25.492648+0000","last_undegraded":"2024-05-09T08:32:25.492648+0000","last_fullsized":"2024-05-09T08:32:25.492648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:53:07.567583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.568737+0000","last_change":"2024-05-09T08:31:04.856322+0000","last_active":"2024-05-09T08:31:56.568737+0000","last_peered":"2024-05-09T08:31:56.568737+0000","last_clean":"2024-05-09T08:31:56.568737+0000","last_became_active":"2024-05-09T08:31:04.855821+0000","last_became_peered":"2024-05-09T08:31:04.855821+0000","last_unstale":"2024-05-09T08:31:56.568737+0000","last_undegraded":"2024-05-09T08:31:56.568737+0000","last_fullsized":"2024-05-09T08:31:56.568737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:29:18.594928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.574061+0000","last_change":"2024-05-09T08:31:56.574061+0000","last_active":"2024-05-09T08:31:56.574061+0000","last_peered":"2024-05-09T08:31:56.574061+0000","last_clean":"2024-05-09T08:31:56.574061+0000","last_became_active":"2024-05-09T08:31:56.573673+0000","last_became_peered":"2024-05-09T08:31:56.573673+0000","last_unstale":"2024-05-09T08:31:56.574061+0000","last_undegraded":"2024-05-09T08:31:56.574061+0000","last_fullsized":"2024-05-09T08:31:56.574061+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:12:13.699915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484283+0000","last_change":"2024-05-09T08:31:00.684523+0000","last_active":"2024-05-09T08:32:25.484283+0000","last_peered":"2024-05-09T08:32:25.484283+0000","last_clean":"2024-05-09T08:32:25.484283+0000","last_became_active":"2024-05-09T08:31:00.684153+0000","last_became_peered":"2024-05-09T08:31:00.684153+0000","last_unstale":"2024-05-09T08:32:25.484283+0000","last_undegraded":"2024-05-09T08:32:25.484283+0000","last_fullsized":"2024-05-09T08:32:25.484283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:02.423401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.776940+0000","last_change":"2024-05-09T08:31:04.857961+0000","last_active":"2024-05-09T08:31:56.776940+0000","last_peered":"2024-05-09T08:31:56.776940+0000","last_clean":"2024-05-09T08:31:56.776940+0000","last_became_active":"2024-05-09T08:31:04.857543+0000","last_became_peered":"2024-05-09T08:31:04.857543+0000","last_unstale":"2024-05-09T08:31:56.776940+0000","last_undegraded":"2024-05-09T08:31:56.776940+0000","last_fullsized":"2024-05-09T08:31:56.776940+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:22:30.820297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.570636+0000","last_change":"2024-05-09T08:30:59.827878+0000","last_active":"2024-05-09T08:31:56.570636+0000","last_peered":"2024-05-09T08:31:56.570636+0000","last_clean":"2024-05-09T08:31:56.570636+0000","last_became_active":"2024-05-09T08:30:59.827487+0000","last_became_peered":"2024-05-09T08:30:59.827487+0000","last_unstale":"2024-05-09T08:31:56.570636+0000","last_undegraded":"2024-05-09T08:31:56.570636+0000","last_fullsized":"2024-05-09T08:31:56.570636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:34.465457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.478753+0000","last_change":"2024-05-09T08:31:00.695617+0000","last_active":"2024-05-09T08:32:25.478753+0000","last_peered":"2024-05-09T08:32:25.478753+0000","last_clean":"2024-05-09T08:32:25.478753+0000","last_became_active":"2024-05-09T08:31:00.695293+0000","last_became_peered":"2024-05-09T08:31:00.695293+0000","last_unstale":"2024-05-09T08:32:25.478753+0000","last_undegraded":"2024-05-09T08:32:25.478753+0000","last_fullsized":"2024-05-09T08:32:25.478753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:44:03.958559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.777135+0000","last_change":"2024-05-09T08:31:04.860184+0000","last_active":"2024-05-09T08:31:56.777135+0000","last_peered":"2024-05-09T08:31:56.777135+0000","last_clean":"2024-05-09T08:31:56.777135+0000","last_became_active":"2024-05-09T08:31:04.859730+0000","last_became_peered":"2024-05-09T08:31:04.859730+0000","last_unstale":"2024-05-09T08:31:56.777135+0000","last_undegraded":"2024-05-09T08:31:56.777135+0000","last_fullsized":"2024-05-09T08:31:56.777135+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:13:17.292678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.483826+0000","last_change":"2024-05-09T08:31:00.694183+0000","last_active":"2024-05-09T08:32:25.483826+0000","last_peered":"2024-05-09T08:32:25.483826+0000","last_clean":"2024-05-09T08:32:25.483826+0000","last_became_active":"2024-05-09T08:31:00.693819+0000","last_became_peered":"2024-05-09T08:31:00.693819+0000","last_unstale":"2024-05-09T08:32:25.483826+0000","last_undegraded":"2024-05-09T08:32:25.483826+0000","last_fullsized":"2024-05-09T08:32:25.483826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:09:26.255663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:34.169060+0000","last_change":"2024-05-09T08:30:59.830758+0000","last_active":"2024-05-09T08:32:34.169060+0000","last_peered":"2024-05-09T08:32:34.169060+0000","last_clean":"2024-05-09T08:32:34.169060+0000","last_became_active":"2024-05-09T08:30:59.830430+0000","last_became_peered":"2024-05-09T08:30:59.830430+0000","last_unstale":"2024-05-09T08:32:34.169060+0000","last_undegraded":"2024-05-09T08:32:34.169060+0000","last_fullsized":"2024-05-09T08:32:34.169060+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:24:05.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.675063+0000","last_change":"2024-05-09T08:31:56.675063+0000","last_active":"2024-05-09T08:31:56.675063+0000","last_peered":"2024-05-09T08:31:56.675063+0000","last_clean":"2024-05-09T08:31:56.675063+0000","last_became_active":"2024-05-09T08:31:56.670995+0000","last_became_peered":"2024-05-09T08:31:56.670995+0000","last_unstale":"2024-05-09T08:31:56.675063+0000","last_undegraded":"2024-05-09T08:31:56.675063+0000","last_fullsized":"2024-05-09T08:31:56.675063+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:57:52.985420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.476532+0000","last_change":"2024-05-09T08:31:00.692197+0000","last_active":"2024-05-09T08:32:25.476532+0000","last_peered":"2024-05-09T08:32:25.476532+0000","last_clean":"2024-05-09T08:32:25.476532+0000","last_became_active":"2024-05-09T08:31:00.691348+0000","last_became_peered":"2024-05-09T08:31:00.691348+0000","last_unstale":"2024-05-09T08:32:25.476532+0000","last_undegraded":"2024-05-09T08:32:25.476532+0000","last_fullsized":"2024-05-09T08:32:25.476532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:00:45.826433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566752+0000","last_change":"2024-05-09T08:30:59.825341+0000","last_active":"2024-05-09T08:31:56.566752+0000","last_peered":"2024-05-09T08:31:56.566752+0000","last_clean":"2024-05-09T08:31:56.566752+0000","last_became_active":"2024-05-09T08:30:59.824860+0000","last_became_peered":"2024-05-09T08:30:59.824860+0000","last_unstale":"2024-05-09T08:31:56.566752+0000","last_undegraded":"2024-05-09T08:31:56.566752+0000","last_fullsized":"2024-05-09T08:31:56.566752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:13.868040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":359,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.570198+0000","last_change":"2024-05-09T08:29:47.816504+0000","last_active":"2024-05-09T08:31:56.570198+0000","last_peered":"2024-05-09T08:31:56.570198+0000","last_clean":"2024-05-09T08:31:56.570198+0000","last_became_active":"2024-05-09T08:29:47.816050+0000","last_became_peered":"2024-05-09T08:29:47.816050+0000","last_unstale":"2024-05-09T08:31:56.570198+0000","last_undegraded":"2024-05-09T08:31:56.570198+0000","last_fullsized":"2024-05-09T08:31:56.570198+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:26:22.135491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:26:22.135491+0000","last_clean_scrub_stamp":"2024-05-09T08:26:22.135491+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:28.543694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.575846+0000","last_change":"2024-05-09T08:31:04.860417+0000","last_active":"2024-05-09T08:31:56.575846+0000","last_peered":"2024-05-09T08:31:56.575846+0000","last_clean":"2024-05-09T08:31:56.575846+0000","last_became_active":"2024-05-09T08:31:04.859040+0000","last_became_peered":"2024-05-09T08:31:04.859040+0000","last_unstale":"2024-05-09T08:31:56.575846+0000","last_undegraded":"2024-05-09T08:31:56.575846+0000","last_fullsized":"2024-05-09T08:31:56.575846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:12:04.157354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.480855+0000","last_change":"2024-05-09T08:31:00.703131+0000","last_active":"2024-05-09T08:32:25.480855+0000","last_peered":"2024-05-09T08:32:25.480855+0000","last_clean":"2024-05-09T08:32:25.480855+0000","last_became_active":"2024-05-09T08:31:00.702817+0000","last_became_peered":"2024-05-09T08:31:00.702817+0000","last_unstale":"2024-05-09T08:32:25.480855+0000","last_undegraded":"2024-05-09T08:32:25.480855+0000","last_fullsized":"2024-05-09T08:32:25.480855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:31.059334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.570265+0000","last_change":"2024-05-09T08:30:59.829636+0000","last_active":"2024-05-09T08:31:56.570265+0000","last_peered":"2024-05-09T08:31:56.570265+0000","last_clean":"2024-05-09T08:31:56.570265+0000","last_became_active":"2024-05-09T08:30:59.829276+0000","last_became_peered":"2024-05-09T08:30:59.829276+0000","last_unstale":"2024-05-09T08:31:56.570265+0000","last_undegraded":"2024-05-09T08:31:56.570265+0000","last_fullsized":"2024-05-09T08:31:56.570265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:17.847258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.571002+0000","last_change":"2024-05-09T08:31:04.853945+0000","last_active":"2024-05-09T08:31:56.571002+0000","last_peered":"2024-05-09T08:31:56.571002+0000","last_clean":"2024-05-09T08:31:56.571002+0000","last_became_active":"2024-05-09T08:31:04.853417+0000","last_became_peered":"2024-05-09T08:31:04.853417+0000","last_unstale":"2024-05-09T08:31:56.571002+0000","last_undegraded":"2024-05-09T08:31:56.571002+0000","last_fullsized":"2024-05-09T08:31:56.571002+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:21:31.126562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.568918+0000","last_change":"2024-05-09T08:30:59.828613+0000","last_active":"2024-05-09T08:31:56.568918+0000","last_peered":"2024-05-09T08:31:56.568918+0000","last_clean":"2024-05-09T08:31:56.568918+0000","last_became_active":"2024-05-09T08:30:59.828256+0000","last_became_peered":"2024-05-09T08:30:59.828256+0000","last_unstale":"2024-05-09T08:31:56.568918+0000","last_undegraded":"2024-05-09T08:31:56.568918+0000","last_fullsized":"2024-05-09T08:31:56.568918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:17:04.488271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.482238+0000","last_change":"2024-05-09T08:31:00.703478+0000","last_active":"2024-05-09T08:32:25.482238+0000","last_peered":"2024-05-09T08:32:25.482238+0000","last_clean":"2024-05-09T08:32:25.482238+0000","last_became_active":"2024-05-09T08:31:00.702852+0000","last_became_peered":"2024-05-09T08:31:00.702852+0000","last_unstale":"2024-05-09T08:32:25.482238+0000","last_undegraded":"2024-05-09T08:32:25.482238+0000","last_fullsized":"2024-05-09T08:32:25.482238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:12.635341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.795827+0000","last_change":"2024-05-09T08:31:04.860801+0000","last_active":"2024-05-09T08:31:56.795827+0000","last_peered":"2024-05-09T08:31:56.795827+0000","last_clean":"2024-05-09T08:31:56.795827+0000","last_became_active":"2024-05-09T08:31:04.857387+0000","last_became_peered":"2024-05-09T08:31:04.857387+0000","last_unstale":"2024-05-09T08:31:56.795827+0000","last_undegraded":"2024-05-09T08:31:56.795827+0000","last_fullsized":"2024-05-09T08:31:56.795827+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:50:16.555987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.567897+0000","last_change":"2024-05-09T08:30:59.827082+0000","last_active":"2024-05-09T08:31:56.567897+0000","last_peered":"2024-05-09T08:31:56.567897+0000","last_clean":"2024-05-09T08:31:56.567897+0000","last_became_active":"2024-05-09T08:30:59.826746+0000","last_became_peered":"2024-05-09T08:30:59.826746+0000","last_unstale":"2024-05-09T08:31:56.567897+0000","last_undegraded":"2024-05-09T08:31:56.567897+0000","last_fullsized":"2024-05-09T08:31:56.567897+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:49.176889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.491742+0000","last_change":"2024-05-09T08:31:00.691902+0000","last_active":"2024-05-09T08:32:25.491742+0000","last_peered":"2024-05-09T08:32:25.491742+0000","last_clean":"2024-05-09T08:32:25.491742+0000","last_became_active":"2024-05-09T08:31:00.689491+0000","last_became_peered":"2024-05-09T08:31:00.689491+0000","last_unstale":"2024-05-09T08:32:25.491742+0000","last_undegraded":"2024-05-09T08:32:25.491742+0000","last_fullsized":"2024-05-09T08:32:25.491742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:48:07.335144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566278+0000","last_change":"2024-05-09T08:31:04.857788+0000","last_active":"2024-05-09T08:31:56.566278+0000","last_peered":"2024-05-09T08:31:56.566278+0000","last_clean":"2024-05-09T08:31:56.566278+0000","last_became_active":"2024-05-09T08:31:04.853831+0000","last_became_peered":"2024-05-09T08:31:04.853831+0000","last_unstale":"2024-05-09T08:31:56.566278+0000","last_undegraded":"2024-05-09T08:31:56.566278+0000","last_fullsized":"2024-05-09T08:31:56.566278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:08:15.451606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.567361+0000","last_change":"2024-05-09T08:30:59.826482+0000","last_active":"2024-05-09T08:31:56.567361+0000","last_peered":"2024-05-09T08:31:56.567361+0000","last_clean":"2024-05-09T08:31:56.567361+0000","last_became_active":"2024-05-09T08:30:59.826123+0000","last_became_peered":"2024-05-09T08:30:59.826123+0000","last_unstale":"2024-05-09T08:31:56.567361+0000","last_undegraded":"2024-05-09T08:31:56.567361+0000","last_fullsized":"2024-05-09T08:31:56.567361+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:38.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.482173+0000","last_change":"2024-05-09T08:31:00.693440+0000","last_active":"2024-05-09T08:32:25.482173+0000","last_peered":"2024-05-09T08:32:25.482173+0000","last_clean":"2024-05-09T08:32:25.482173+0000","last_became_active":"2024-05-09T08:31:00.693117+0000","last_became_peered":"2024-05-09T08:31:00.693117+0000","last_unstale":"2024-05-09T08:32:25.482173+0000","last_undegraded":"2024-05-09T08:32:25.482173+0000","last_fullsized":"2024-05-09T08:32:25.482173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:10:01.205605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.567085+0000","last_change":"2024-05-09T08:31:04.868120+0000","last_active":"2024-05-09T08:31:56.567085+0000","last_peered":"2024-05-09T08:31:56.567085+0000","last_clean":"2024-05-09T08:31:56.567085+0000","last_became_active":"2024-05-09T08:31:04.867571+0000","last_became_peered":"2024-05-09T08:31:04.867571+0000","last_unstale":"2024-05-09T08:31:56.567085+0000","last_undegraded":"2024-05-09T08:31:56.567085+0000","last_fullsized":"2024-05-09T08:31:56.567085+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:12:36.503972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.569407+0000","last_change":"2024-05-09T08:30:59.826280+0000","last_active":"2024-05-09T08:31:56.569407+0000","last_peered":"2024-05-09T08:31:56.569407+0000","last_clean":"2024-05-09T08:31:56.569407+0000","last_became_active":"2024-05-09T08:30:59.825805+0000","last_became_peered":"2024-05-09T08:30:59.825805+0000","last_unstale":"2024-05-09T08:31:56.569407+0000","last_undegraded":"2024-05-09T08:31:56.569407+0000","last_fullsized":"2024-05-09T08:31:56.569407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:46:03.833159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.454159+0000","last_change":"2024-05-09T08:31:00.687581+0000","last_active":"2024-05-09T08:32:25.454159+0000","last_peered":"2024-05-09T08:32:25.454159+0000","last_clean":"2024-05-09T08:32:25.454159+0000","last_became_active":"2024-05-09T08:31:00.684608+0000","last_became_peered":"2024-05-09T08:31:00.684608+0000","last_unstale":"2024-05-09T08:32:25.454159+0000","last_undegraded":"2024-05-09T08:32:25.454159+0000","last_fullsized":"2024-05-09T08:32:25.454159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:06:06.436840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.567738+0000","last_change":"2024-05-09T08:31:04.859399+0000","last_active":"2024-05-09T08:31:56.567738+0000","last_peered":"2024-05-09T08:31:56.567738+0000","last_clean":"2024-05-09T08:31:56.567738+0000","last_became_active":"2024-05-09T08:31:04.859027+0000","last_became_peered":"2024-05-09T08:31:04.859027+0000","last_unstale":"2024-05-09T08:31:56.567738+0000","last_undegraded":"2024-05-09T08:31:56.567738+0000","last_fullsized":"2024-05-09T08:31:56.567738+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:18:00.829342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.796124+0000","last_change":"2024-05-09T08:30:59.824171+0000","last_active":"2024-05-09T08:31:56.796124+0000","last_peered":"2024-05-09T08:31:56.796124+0000","last_clean":"2024-05-09T08:31:56.796124+0000","last_became_active":"2024-05-09T08:30:59.823859+0000","last_became_peered":"2024-05-09T08:30:59.823859+0000","last_unstale":"2024-05-09T08:31:56.796124+0000","last_undegraded":"2024-05-09T08:31:56.796124+0000","last_fullsized":"2024-05-09T08:31:56.796124+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:30.199504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.492260+0000","last_change":"2024-05-09T08:31:00.700056+0000","last_active":"2024-05-09T08:32:25.492260+0000","last_peered":"2024-05-09T08:32:25.492260+0000","last_clean":"2024-05-09T08:32:25.492260+0000","last_became_active":"2024-05-09T08:31:00.699673+0000","last_became_peered":"2024-05-09T08:31:00.699673+0000","last_unstale":"2024-05-09T08:32:25.492260+0000","last_undegraded":"2024-05-09T08:32:25.492260+0000","last_fullsized":"2024-05-09T08:32:25.492260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:23.873645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566824+0000","last_change":"2024-05-09T08:31:04.858940+0000","last_active":"2024-05-09T08:31:56.566824+0000","last_peered":"2024-05-09T08:31:56.566824+0000","last_clean":"2024-05-09T08:31:56.566824+0000","last_became_active":"2024-05-09T08:31:04.858532+0000","last_became_peered":"2024-05-09T08:31:04.858532+0000","last_unstale":"2024-05-09T08:31:56.566824+0000","last_undegraded":"2024-05-09T08:31:56.566824+0000","last_fullsized":"2024-05-09T08:31:56.566824+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:57:42.128068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566477+0000","last_change":"2024-05-09T08:30:59.824182+0000","last_active":"2024-05-09T08:31:56.566477+0000","last_peered":"2024-05-09T08:31:56.566477+0000","last_clean":"2024-05-09T08:31:56.566477+0000","last_became_active":"2024-05-09T08:30:59.823691+0000","last_became_peered":"2024-05-09T08:30:59.823691+0000","last_unstale":"2024-05-09T08:31:56.566477+0000","last_undegraded":"2024-05-09T08:31:56.566477+0000","last_fullsized":"2024-05-09T08:31:56.566477+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:28:07.810733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484483+0000","last_change":"2024-05-09T08:31:00.692614+0000","last_active":"2024-05-09T08:32:25.484483+0000","last_peered":"2024-05-09T08:32:25.484483+0000","last_clean":"2024-05-09T08:32:25.484483+0000","last_became_active":"2024-05-09T08:31:00.692280+0000","last_became_peered":"2024-05-09T08:31:00.692280+0000","last_unstale":"2024-05-09T08:32:25.484483+0000","last_undegraded":"2024-05-09T08:32:25.484483+0000","last_fullsized":"2024-05-09T08:32:25.484483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:23:15.398196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.566897+0000","last_change":"2024-05-09T08:31:04.854029+0000","last_active":"2024-05-09T08:31:56.566897+0000","last_peered":"2024-05-09T08:31:56.566897+0000","last_clean":"2024-05-09T08:31:56.566897+0000","last_became_active":"2024-05-09T08:31:04.853554+0000","last_became_peered":"2024-05-09T08:31:04.853554+0000","last_unstale":"2024-05-09T08:31:56.566897+0000","last_undegraded":"2024-05-09T08:31:56.566897+0000","last_fullsized":"2024-05-09T08:31:56.566897+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:34:41.059614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.569879+0000","last_change":"2024-05-09T08:30:59.824512+0000","last_active":"2024-05-09T08:31:56.569879+0000","last_peered":"2024-05-09T08:31:56.569879+0000","last_clean":"2024-05-09T08:31:56.569879+0000","last_became_active":"2024-05-09T08:30:59.824146+0000","last_became_peered":"2024-05-09T08:30:59.824146+0000","last_unstale":"2024-05-09T08:31:56.569879+0000","last_undegraded":"2024-05-09T08:31:56.569879+0000","last_fullsized":"2024-05-09T08:31:56.569879+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:27.734292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.490636+0000","last_change":"2024-05-09T08:31:00.697272+0000","last_active":"2024-05-09T08:32:25.490636+0000","last_peered":"2024-05-09T08:32:25.490636+0000","last_clean":"2024-05-09T08:32:25.490636+0000","last_became_active":"2024-05-09T08:31:00.696026+0000","last_became_peered":"2024-05-09T08:31:00.696026+0000","last_unstale":"2024-05-09T08:32:25.490636+0000","last_undegraded":"2024-05-09T08:32:25.490636+0000","last_fullsized":"2024-05-09T08:32:25.490636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:04.376142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.569574+0000","last_change":"2024-05-09T08:31:04.856641+0000","last_active":"2024-05-09T08:31:56.569574+0000","last_peered":"2024-05-09T08:31:56.569574+0000","last_clean":"2024-05-09T08:31:56.569574+0000","last_became_active":"2024-05-09T08:31:04.856102+0000","last_became_peered":"2024-05-09T08:31:04.856102+0000","last_unstale":"2024-05-09T08:31:56.569574+0000","last_undegraded":"2024-05-09T08:31:56.569574+0000","last_fullsized":"2024-05-09T08:31:56.569574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:35:49.831076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.579569+0000","last_change":"2024-05-09T08:31:56.579569+0000","last_active":"2024-05-09T08:31:56.579569+0000","last_peered":"2024-05-09T08:31:56.579569+0000","last_clean":"2024-05-09T08:31:56.579569+0000","last_became_active":"2024-05-09T08:31:56.573496+0000","last_became_peered":"2024-05-09T08:31:56.573496+0000","last_unstale":"2024-05-09T08:31:56.579569+0000","last_undegraded":"2024-05-09T08:31:56.579569+0000","last_fullsized":"2024-05-09T08:31:56.579569+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:47:47.944200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.490646+0000","last_change":"2024-05-09T08:31:00.697876+0000","last_active":"2024-05-09T08:32:25.490646+0000","last_peered":"2024-05-09T08:32:25.490646+0000","last_clean":"2024-05-09T08:32:25.490646+0000","last_became_active":"2024-05-09T08:31:00.696299+0000","last_became_peered":"2024-05-09T08:31:00.696299+0000","last_unstale":"2024-05-09T08:32:25.490646+0000","last_undegraded":"2024-05-09T08:32:25.490646+0000","last_fullsized":"2024-05-09T08:32:25.490646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:48:57.373734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.572156+0000","last_change":"2024-05-09T08:31:04.858588+0000","last_active":"2024-05-09T08:31:56.572156+0000","last_peered":"2024-05-09T08:31:56.572156+0000","last_clean":"2024-05-09T08:31:56.572156+0000","last_became_active":"2024-05-09T08:31:04.858154+0000","last_became_peered":"2024-05-09T08:31:04.858154+0000","last_unstale":"2024-05-09T08:31:56.572156+0000","last_undegraded":"2024-05-09T08:31:56.572156+0000","last_fullsized":"2024-05-09T08:31:56.572156+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:06:01.768743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.576803+0000","last_change":"2024-05-09T08:30:59.826066+0000","last_active":"2024-05-09T08:31:56.576803+0000","last_peered":"2024-05-09T08:31:56.576803+0000","last_clean":"2024-05-09T08:31:56.576803+0000","last_became_active":"2024-05-09T08:30:59.825586+0000","last_became_peered":"2024-05-09T08:30:59.825586+0000","last_unstale":"2024-05-09T08:31:56.576803+0000","last_undegraded":"2024-05-09T08:31:56.576803+0000","last_fullsized":"2024-05-09T08:31:56.576803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:19:04.629132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484950+0000","last_change":"2024-05-09T08:31:00.693788+0000","last_active":"2024-05-09T08:32:25.484950+0000","last_peered":"2024-05-09T08:32:25.484950+0000","last_clean":"2024-05-09T08:32:25.484950+0000","last_became_active":"2024-05-09T08:31:00.693448+0000","last_became_peered":"2024-05-09T08:31:00.693448+0000","last_unstale":"2024-05-09T08:32:25.484950+0000","last_undegraded":"2024-05-09T08:32:25.484950+0000","last_fullsized":"2024-05-09T08:32:25.484950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:06.516534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.574037+0000","last_change":"2024-05-09T08:31:04.856185+0000","last_active":"2024-05-09T08:31:56.574037+0000","last_peered":"2024-05-09T08:31:56.574037+0000","last_clean":"2024-05-09T08:31:56.574037+0000","last_became_active":"2024-05-09T08:31:04.855779+0000","last_became_peered":"2024-05-09T08:31:04.855779+0000","last_unstale":"2024-05-09T08:31:56.574037+0000","last_undegraded":"2024-05-09T08:31:56.574037+0000","last_fullsized":"2024-05-09T08:31:56.574037+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:45:25.034077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:31:56.777025+0000","last_change":"2024-05-09T08:30:59.822938+0000","last_active":"2024-05-09T08:31:56.777025+0000","last_peered":"2024-05-09T08:31:56.777025+0000","last_clean":"2024-05-09T08:31:56.777025+0000","last_became_active":"2024-05-09T08:30:59.822629+0000","last_became_peered":"2024-05-09T08:30:59.822629+0000","last_unstale":"2024-05-09T08:31:56.777025+0000","last_undegraded":"2024-05-09T08:31:56.777025+0000","last_fullsized":"2024-05-09T08:31:56.777025+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:51.321350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484173+0000","last_change":"2024-05-09T08:31:00.691971+0000","last_active":"2024-05-09T08:32:25.484173+0000","last_peered":"2024-05-09T08:32:25.484173+0000","last_clean":"2024-05-09T08:32:25.484173+0000","last_became_active":"2024-05-09T08:31:00.691641+0000","last_became_peered":"2024-05-09T08:31:00.691641+0000","last_unstale":"2024-05-09T08:32:25.484173+0000","last_undegraded":"2024-05-09T08:32:25.484173+0000","last_fullsized":"2024-05-09T08:32:25.484173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:28.413886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.493734+0000","last_change":"2024-05-09T08:31:00.701026+0000","last_active":"2024-05-09T08:32:25.493734+0000","last_peered":"2024-05-09T08:32:25.493734+0000","last_clean":"2024-05-09T08:32:25.493734+0000","last_became_active":"2024-05-09T08:31:00.700457+0000","last_became_peered":"2024-05-09T08:31:00.700457+0000","last_unstale":"2024-05-09T08:32:25.493734+0000","last_undegraded":"2024-05-09T08:32:25.493734+0000","last_fullsized":"2024-05-09T08:32:25.493734+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:40.065173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.481582+0000","last_change":"2024-05-09T08:31:56.570220+0000","last_active":"2024-05-09T08:32:25.481582+0000","last_peered":"2024-05-09T08:32:25.481582+0000","last_clean":"2024-05-09T08:32:25.481582+0000","last_became_active":"2024-05-09T08:31:56.569682+0000","last_became_peered":"2024-05-09T08:31:56.569682+0000","last_unstale":"2024-05-09T08:32:25.481582+0000","last_undegraded":"2024-05-09T08:32:25.481582+0000","last_fullsized":"2024-05-09T08:32:25.481582+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:47:07.986064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.480090+0000","last_change":"2024-05-09T08:31:00.703201+0000","last_active":"2024-05-09T08:32:25.480090+0000","last_peered":"2024-05-09T08:32:25.480090+0000","last_clean":"2024-05-09T08:32:25.480090+0000","last_became_active":"2024-05-09T08:31:00.702801+0000","last_became_peered":"2024-05-09T08:31:00.702801+0000","last_unstale":"2024-05-09T08:32:25.480090+0000","last_undegraded":"2024-05-09T08:32:25.480090+0000","last_fullsized":"2024-05-09T08:32:25.480090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:58:35.106013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.482324+0000","last_change":"2024-05-09T08:31:00.693696+0000","last_active":"2024-05-09T08:32:25.482324+0000","last_peered":"2024-05-09T08:32:25.482324+0000","last_clean":"2024-05-09T08:32:25.482324+0000","last_became_active":"2024-05-09T08:31:00.693394+0000","last_became_peered":"2024-05-09T08:31:00.693394+0000","last_unstale":"2024-05-09T08:32:25.482324+0000","last_undegraded":"2024-05-09T08:32:25.482324+0000","last_fullsized":"2024-05-09T08:32:25.482324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:01.700968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484061+0000","last_change":"2024-05-09T08:31:00.694363+0000","last_active":"2024-05-09T08:32:25.484061+0000","last_peered":"2024-05-09T08:32:25.484061+0000","last_clean":"2024-05-09T08:32:25.484061+0000","last_became_active":"2024-05-09T08:31:00.693970+0000","last_became_peered":"2024-05-09T08:31:00.693970+0000","last_unstale":"2024-05-09T08:32:25.484061+0000","last_undegraded":"2024-05-09T08:32:25.484061+0000","last_fullsized":"2024-05-09T08:32:25.484061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:56.185801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.492564+0000","last_change":"2024-05-09T08:31:00.691970+0000","last_active":"2024-05-09T08:32:25.492564+0000","last_peered":"2024-05-09T08:32:25.492564+0000","last_clean":"2024-05-09T08:32:25.492564+0000","last_became_active":"2024-05-09T08:31:00.691632+0000","last_became_peered":"2024-05-09T08:31:00.691632+0000","last_unstale":"2024-05-09T08:32:25.492564+0000","last_undegraded":"2024-05-09T08:32:25.492564+0000","last_fullsized":"2024-05-09T08:32:25.492564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:04:49.316808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.477847+0000","last_change":"2024-05-09T08:31:00.692042+0000","last_active":"2024-05-09T08:32:25.477847+0000","last_peered":"2024-05-09T08:32:25.477847+0000","last_clean":"2024-05-09T08:32:25.477847+0000","last_became_active":"2024-05-09T08:31:00.691700+0000","last_became_peered":"2024-05-09T08:31:00.691700+0000","last_unstale":"2024-05-09T08:32:25.477847+0000","last_undegraded":"2024-05-09T08:32:25.477847+0000","last_fullsized":"2024-05-09T08:32:25.477847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:54.236319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.479387+0000","last_change":"2024-05-09T08:31:00.697261+0000","last_active":"2024-05-09T08:32:25.479387+0000","last_peered":"2024-05-09T08:32:25.479387+0000","last_clean":"2024-05-09T08:32:25.479387+0000","last_became_active":"2024-05-09T08:31:00.696299+0000","last_became_peered":"2024-05-09T08:31:00.696299+0000","last_unstale":"2024-05-09T08:32:25.479387+0000","last_undegraded":"2024-05-09T08:32:25.479387+0000","last_fullsized":"2024-05-09T08:32:25.479387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:06:52.061740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.480323+0000","last_change":"2024-05-09T08:31:00.691097+0000","last_active":"2024-05-09T08:32:25.480323+0000","last_peered":"2024-05-09T08:32:25.480323+0000","last_clean":"2024-05-09T08:32:25.480323+0000","last_became_active":"2024-05-09T08:31:00.690487+0000","last_became_peered":"2024-05-09T08:31:00.690487+0000","last_unstale":"2024-05-09T08:32:25.480323+0000","last_undegraded":"2024-05-09T08:32:25.480323+0000","last_fullsized":"2024-05-09T08:32:25.480323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:47:26.468296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.491778+0000","last_change":"2024-05-09T08:31:00.703201+0000","last_active":"2024-05-09T08:32:25.491778+0000","last_peered":"2024-05-09T08:32:25.491778+0000","last_clean":"2024-05-09T08:32:25.491778+0000","last_became_active":"2024-05-09T08:31:00.702853+0000","last_became_peered":"2024-05-09T08:31:00.702853+0000","last_unstale":"2024-05-09T08:32:25.491778+0000","last_undegraded":"2024-05-09T08:32:25.491778+0000","last_fullsized":"2024-05-09T08:32:25.491778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:18.510321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.493816+0000","last_change":"2024-05-09T08:31:00.703958+0000","last_active":"2024-05-09T08:32:25.493816+0000","last_peered":"2024-05-09T08:32:25.493816+0000","last_clean":"2024-05-09T08:32:25.493816+0000","last_became_active":"2024-05-09T08:31:00.703110+0000","last_became_peered":"2024-05-09T08:31:00.703110+0000","last_unstale":"2024-05-09T08:32:25.493816+0000","last_undegraded":"2024-05-09T08:32:25.493816+0000","last_fullsized":"2024-05-09T08:32:25.493816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:37:32.371869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.484826+0000","last_change":"2024-05-09T08:31:00.686763+0000","last_active":"2024-05-09T08:32:25.484826+0000","last_peered":"2024-05-09T08:32:25.484826+0000","last_clean":"2024-05-09T08:32:25.484826+0000","last_became_active":"2024-05-09T08:31:00.685642+0000","last_became_peered":"2024-05-09T08:31:00.685642+0000","last_unstale":"2024-05-09T08:32:25.484826+0000","last_undegraded":"2024-05-09T08:32:25.484826+0000","last_fullsized":"2024-05-09T08:32:25.484826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:40:18.201247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T08:32:25.492627+0000","last_change":"2024-05-09T08:31:56.573837+0000","last_active":"2024-05-09T08:32:25.492627+0000","last_peered":"2024-05-09T08:32:25.492627+0000","last_clean":"2024-05-09T08:32:25.492627+0000","last_became_active":"2024-05-09T08:31:56.573408+0000","last_became_peered":"2024-05-09T08:31:56.573408+0000","last_unstale":"2024-05-09T08:32:25.492627+0000","last_undegraded":"2024-05-09T08:32:25.492627+0000","last_fullsized":"2024-05-09T08:32:25.492627+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:12:46.397877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67747,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":472,"num_read_kb":307,"num_write":167,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":203241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383713,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":500815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645349,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28696,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716456,"statfs":{"total":95995035648,"available":95965650944,"internally_reserved":0,"allocated":1720320,"data_stored":1156316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808874,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":505147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070512,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":500389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332149,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36932,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708220,"statfs":{"total":95995035648,"available":95957217280,"internally_reserved":0,"allocated":1634304,"data_stored":1097021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":496227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":560324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":497615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1110016,"data_stored":560510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1032192,"data_stored":496421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920837,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":501991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757311,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36936,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708216,"statfs":{"total":95995035648,"available":95957213184,"internally_reserved":0,"allocated":1638400,"data_stored":1087981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":64187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:32:39.492 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-09T08:32:39.492 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-09T08:32:39.492 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-09T08:32:39.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph health --format=json 2024-05-09T08:32:39.735 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:40.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:40 smithi157 ceph-mon[33614]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:32:40.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:40 smithi157 ceph-mon[33614]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T08:32:40.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:40 smithi110 ceph-mon[33622]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:32:40.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:40 smithi110 ceph-mon[33622]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T08:32:40.889 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:40 smithi107 ceph-mon[30122]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:32:40.889 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:40 smithi107 ceph-mon[30122]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T08:32:41.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.505+0000 7fd20226f700 1 -- 172.21.15.107:0/2510704357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1fc102f80 msgr2=0x7fd1fc106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:41.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.505+0000 7fd20226f700 1 --2- 172.21.15.107:0/2510704357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1fc102f80 0x7fd1fc106fb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0009990 tx=0x7fd1f0009ca0 comp rx=0 tx=0).stop 2024-05-09T08:32:41.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.505+0000 7fd20226f700 1 -- 172.21.15.107:0/2510704357 shutdown_connections 2024-05-09T08:32:41.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.505+0000 7fd20226f700 1 --2- 172.21.15.107:0/2510704357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1fc107690 0x7fd1fc10f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:41.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.505+0000 7fd20226f700 1 --2- 172.21.15.107:0/2510704357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1fc1025d0 0x7fd1fc1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:41.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.505+0000 7fd20226f700 1 --2- 172.21.15.107:0/2510704357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1fc102f80 0x7fd1fc106fb0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:41.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.505+0000 7fd20226f700 1 -- 172.21.15.107:0/2510704357 >> 172.21.15.107:0/2510704357 conn(0x7fd1fc0794d0 msgr2=0x7fd1fc0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:41.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.505+0000 7fd20226f700 1 -- 172.21.15.107:0/2510704357 shutdown_connections 2024-05-09T08:32:41.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.506+0000 7fd20226f700 1 -- 172.21.15.107:0/2510704357 wait complete. 2024-05-09T08:32:41.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.506+0000 7fd20226f700 1 Processor -- start 2024-05-09T08:32:41.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.506+0000 7fd20226f700 1 -- start start 2024-05-09T08:32:41.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd20226f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1fc1025d0 0x7fd1fc19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd20226f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1fc102f80 0x7fd1fc19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd20226f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1fc107690 0x7fd1fc198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd20226f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd1fc112500 con 0x7fd1fc107690 2024-05-09T08:32:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd20226f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd1fc112380 con 0x7fd1fc1025d0 2024-05-09T08:32:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd20226f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd1fc112680 con 0x7fd1fc102f80 2024-05-09T08:32:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd20080c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1fc107690 0x7fd1fc198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd20080c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1fc107690 0x7fd1fc198860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55430/0 (socket says 172.21.15.107:55430) 2024-05-09T08:32:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd20080c700 1 -- 172.21.15.107:0/2209726785 learned_addr learned my addr 172.21.15.107:0/2209726785 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd1fbfff700 1 --2- 172.21.15.107:0/2209726785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1fc1025d0 0x7fd1fc19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd1fb7fe700 1 --2- 172.21.15.107:0/2209726785 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1fc102f80 0x7fd1fc19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd20080c700 1 -- 172.21.15.107:0/2209726785 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1fc102f80 msgr2=0x7fd1fc19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd20080c700 1 --2- 172.21.15.107:0/2209726785 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1fc102f80 0x7fd1fc19ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd20080c700 1 -- 172.21.15.107:0/2209726785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1fc1025d0 msgr2=0x7fd1fc19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd20080c700 1 --2- 172.21.15.107:0/2209726785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1fc1025d0 0x7fd1fc19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd20080c700 1 -- 172.21.15.107:0/2209726785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1fc198fc0 con 0x7fd1fc107690 2024-05-09T08:32:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.507+0000 7fd20080c700 1 --2- 172.21.15.107:0/2209726785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1fc107690 0x7fd1fc198860 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd1f400ab50 tx=0x7fd1f400ae60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.508+0000 7fd1f97fa700 1 -- 172.21.15.107:0/2209726785 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4017020 con 0x7fd1fc107690 2024-05-09T08:32:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.508+0000 7fd20226f700 1 -- 172.21.15.107:0/2209726785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1fc1992b0 con 0x7fd1fc107690 2024-05-09T08:32:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.508+0000 7fd1f97fa700 1 -- 172.21.15.107:0/2209726785 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1f4004bb0 con 0x7fd1fc107690 2024-05-09T08:32:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.508+0000 7fd1f97fa700 1 -- 172.21.15.107:0/2209726785 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4005280 con 0x7fd1fc107690 2024-05-09T08:32:41.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.508+0000 7fd20226f700 1 -- 172.21.15.107:0/2209726785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1fc1a5570 con 0x7fd1fc107690 2024-05-09T08:32:41.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.509+0000 7fd20226f700 1 -- 172.21.15.107:0/2209726785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1fc069010 con 0x7fd1fc107690 2024-05-09T08:32:41.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.510+0000 7fd1f97fa700 1 -- 172.21.15.107:0/2209726785 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd1f400c620 con 0x7fd1fc107690 2024-05-09T08:32:41.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.511+0000 7fd1f97fa700 1 --2- 172.21.15.107:0/2209726785 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd1e40a2ca0 0x7fd1e40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:41.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.511+0000 7fd1fbfff700 1 --2- 172.21.15.107:0/2209726785 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd1e40a2ca0 0x7fd1e40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:41.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.511+0000 7fd1f97fa700 1 -- 172.21.15.107:0/2209726785 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd1f40ce880 con 0x7fd1fc107690 2024-05-09T08:32:41.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.511+0000 7fd1fbfff700 1 --2- 172.21.15.107:0/2209726785 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd1e40a2ca0 0x7fd1e40a5160 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc1999a0 tx=0x7fd1ec00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:41.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.515+0000 7fd1f97fa700 1 -- 172.21.15.107:0/2209726785 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd1f4021020 con 0x7fd1fc107690 2024-05-09T08:32:41.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.756+0000 7fd20226f700 1 -- 172.21.15.107:0/2209726785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd1fc02db80 con 0x7fd1fc107690 2024-05-09T08:32:41.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.756+0000 7fd1f97fa700 1 -- 172.21.15.107:0/2209726785 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fd1f409b510 con 0x7fd1fc107690 2024-05-09T08:32:41.756 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:32:41.757 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-09T08:32:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.758+0000 7fd20226f700 1 -- 172.21.15.107:0/2209726785 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd1e40a2ca0 msgr2=0x7fd1e40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.758+0000 7fd20226f700 1 --2- 172.21.15.107:0/2209726785 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd1e40a2ca0 0x7fd1e40a5160 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc1999a0 tx=0x7fd1ec00b410 comp rx=0 tx=0).stop 2024-05-09T08:32:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.758+0000 7fd20226f700 1 -- 172.21.15.107:0/2209726785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1fc107690 msgr2=0x7fd1fc198860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.758+0000 7fd20226f700 1 --2- 172.21.15.107:0/2209726785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1fc107690 0x7fd1fc198860 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd1f400ab50 tx=0x7fd1f400ae60 comp rx=0 tx=0).stop 2024-05-09T08:32:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.758+0000 7fd20226f700 1 -- 172.21.15.107:0/2209726785 shutdown_connections 2024-05-09T08:32:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.758+0000 7fd20226f700 1 --2- 172.21.15.107:0/2209726785 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd1e40a2ca0 0x7fd1e40a5160 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.758+0000 7fd20226f700 1 --2- 172.21.15.107:0/2209726785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1fc107690 0x7fd1fc198860 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.758+0000 7fd20226f700 1 --2- 172.21.15.107:0/2209726785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1fc1025d0 0x7fd1fc19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.758+0000 7fd20226f700 1 --2- 172.21.15.107:0/2209726785 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1fc102f80 0x7fd1fc19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.758+0000 7fd20226f700 1 -- 172.21.15.107:0/2209726785 >> 172.21.15.107:0/2209726785 conn(0x7fd1fc0794d0 msgr2=0x7fd1fc108810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.759+0000 7fd20226f700 1 -- 172.21.15.107:0/2209726785 shutdown_connections 2024-05-09T08:32:41.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:41.759+0000 7fd20226f700 1 -- 172.21.15.107:0/2209726785 wait complete. 2024-05-09T08:32:42.421 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-09T08:32:42.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:32:42.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.526+0000 7f848dad2640 1 -- 172.21.15.107:0/1507032230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8488102350 msgr2=0x7f8488104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:42.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.526+0000 7f848dad2640 1 --2- 172.21.15.107:0/1507032230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8488102350 0x7f8488104740 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8478009900 tx=0x7f847802a410 comp rx=0 tx=0).stop 2024-05-09T08:32:42.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.526+0000 7f848dad2640 1 -- 172.21.15.107:0/1507032230 shutdown_connections 2024-05-09T08:32:42.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.526+0000 7f848dad2640 1 --2- 172.21.15.107:0/1507032230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84881075b0 0x7f84881099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:42.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.526+0000 7f848dad2640 1 --2- 172.21.15.107:0/1507032230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8488104c80 0x7f8488107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:42.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.526+0000 7f848dad2640 1 --2- 172.21.15.107:0/1507032230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8488102350 0x7f8488104740 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:42.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.526+0000 7f848dad2640 1 -- 172.21.15.107:0/1507032230 >> 172.21.15.107:0/1507032230 conn(0x7f84880fbce0 msgr2=0x7f84880fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:42.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.526+0000 7f848dad2640 1 -- 172.21.15.107:0/1507032230 shutdown_connections 2024-05-09T08:32:42.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.526+0000 7f848dad2640 1 -- 172.21.15.107:0/1507032230 wait complete. 2024-05-09T08:32:42.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.526+0000 7f848dad2640 1 Processor -- start 2024-05-09T08:32:42.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f848dad2640 1 -- start start 2024-05-09T08:32:42.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f848dad2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8488102350 0x7f8488197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:42.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f848dad2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8488104c80 0x7f84881980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:42.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f848dad2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84881075b0 0x7f848819f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:42.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f848dad2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f848810d490 con 0x7f8488102350 2024-05-09T08:32:42.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f848dad2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f848810d310 con 0x7f84881075b0 2024-05-09T08:32:42.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f848dad2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f848810d610 con 0x7f8488104c80 2024-05-09T08:32:42.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f84877fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8488102350 0x7f8488197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:42.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f84877fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8488102350 0x7f8488197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55452/0 (socket says 172.21.15.107:55452) 2024-05-09T08:32:42.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f84877fe640 1 -- 172.21.15.107:0/2997438095 learned_addr learned my addr 172.21.15.107:0/2997438095 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:42.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f84877fe640 1 -- 172.21.15.107:0/2997438095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8488104c80 msgr2=0x7f84881980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:42.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f84877fe640 1 --2- 172.21.15.107:0/2997438095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8488104c80 0x7f84881980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:42.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f84877fe640 1 -- 172.21.15.107:0/2997438095 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84881075b0 msgr2=0x7f848819f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:42.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f84877fe640 1 --2- 172.21.15.107:0/2997438095 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84881075b0 0x7f848819f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:42.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f84877fe640 1 -- 172.21.15.107:0/2997438095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f848819f7b0 con 0x7f8488102350 2024-05-09T08:32:42.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f84877fe640 1 --2- 172.21.15.107:0/2997438095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8488102350 0x7f8488197b90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f84780314e0 tx=0x7f8478003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:42.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.527+0000 7f8484ff9640 1 -- 172.21.15.107:0/2997438095 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f847803f070 con 0x7f8488102350 2024-05-09T08:32:42.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.528+0000 7f848dad2640 1 -- 172.21.15.107:0/2997438095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f848819fa40 con 0x7f8488102350 2024-05-09T08:32:42.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.528+0000 7f8484ff9640 1 -- 172.21.15.107:0/2997438095 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8478031e40 con 0x7f8488102350 2024-05-09T08:32:42.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.528+0000 7f8484ff9640 1 -- 172.21.15.107:0/2997438095 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8478035030 con 0x7f8488102350 2024-05-09T08:32:42.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.528+0000 7f848dad2640 1 -- 172.21.15.107:0/2997438095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84881a0000 con 0x7f8488102350 2024-05-09T08:32:42.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.529+0000 7f848dad2640 1 -- 172.21.15.107:0/2997438095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84880fd7d0 con 0x7f8488102350 2024-05-09T08:32:42.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.530+0000 7f8484ff9640 1 -- 172.21.15.107:0/2997438095 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f8478042070 con 0x7f8488102350 2024-05-09T08:32:42.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.530+0000 7f8484ff9640 1 --2- 172.21.15.107:0/2997438095 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f84740a2b90 0x7f84740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:42.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.530+0000 7f8486ffd640 1 --2- 172.21.15.107:0/2997438095 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f84740a2b90 0x7f84740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:42.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.530+0000 7f8484ff9640 1 -- 172.21.15.107:0/2997438095 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f84780f0220 con 0x7f8488102350 2024-05-09T08:32:42.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.531+0000 7f8486ffd640 1 --2- 172.21.15.107:0/2997438095 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f84740a2b90 0x7f84740a5050 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f84881990a0 tx=0x7f8470005bf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:42.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.534+0000 7f8484ff9640 1 -- 172.21.15.107:0/2997438095 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f84780bce20 con 0x7f8488102350 2024-05-09T08:32:42.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.710+0000 7f848dad2640 1 -- 172.21.15.107:0/2997438095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f848802d3e0 con 0x7f8488102350 2024-05-09T08:32:42.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.712+0000 7f8484ff9640 1 -- 172.21.15.107:0/2997438095 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24658 (secure 0 0 0) 0x7f84780bc720 con 0x7f8488102350 2024-05-09T08:32:42.712 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:32:42.712 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":91,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:31:56.556402+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:32:42.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.714+0000 7f848dad2640 1 -- 172.21.15.107:0/2997438095 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f84740a2b90 msgr2=0x7f84740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:42.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.714+0000 7f848dad2640 1 --2- 172.21.15.107:0/2997438095 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f84740a2b90 0x7f84740a5050 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f84881990a0 tx=0x7f8470005bf0 comp rx=0 tx=0).stop 2024-05-09T08:32:42.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.714+0000 7f848dad2640 1 -- 172.21.15.107:0/2997438095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8488102350 msgr2=0x7f8488197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:42.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.714+0000 7f848dad2640 1 --2- 172.21.15.107:0/2997438095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8488102350 0x7f8488197b90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f84780314e0 tx=0x7f8478003d50 comp rx=0 tx=0).stop 2024-05-09T08:32:42.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.714+0000 7f848dad2640 1 -- 172.21.15.107:0/2997438095 shutdown_connections 2024-05-09T08:32:42.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.714+0000 7f848dad2640 1 --2- 172.21.15.107:0/2997438095 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f84740a2b90 0x7f84740a5050 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:42.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.714+0000 7f848dad2640 1 --2- 172.21.15.107:0/2997438095 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84881075b0 0x7f848819f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:42.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.714+0000 7f848dad2640 1 --2- 172.21.15.107:0/2997438095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8488104c80 0x7f84881980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:42.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.714+0000 7f848dad2640 1 --2- 172.21.15.107:0/2997438095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8488102350 0x7f8488197b90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:42.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.714+0000 7f848dad2640 1 -- 172.21.15.107:0/2997438095 >> 172.21.15.107:0/2997438095 conn(0x7f84880fbce0 msgr2=0x7f8488102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:42.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.714+0000 7f848dad2640 1 -- 172.21.15.107:0/2997438095 shutdown_connections 2024-05-09T08:32:42.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.714+0000 7f848dad2640 1 -- 172.21.15.107:0/2997438095 wait complete. 2024-05-09T08:32:42.724 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:42 smithi107 ceph-mon[30122]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-05-09T08:32:42.724 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:42 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2209726785' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-09T08:32:42.725 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:26:21.698338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:30:58.268633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:30:59.345579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:31:03.065991+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:32:42.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:32:42.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.830+0000 7f8c15a88640 1 -- 172.21.15.107:0/194257054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c101034d0 msgr2=0x7f8c10109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:42.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.830+0000 7f8c15a88640 1 --2- 172.21.15.107:0/194257054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c101034d0 0x7f8c10109d60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8c08009cd0 tx=0x7f8c0802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:32:42.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.830+0000 7f8c15a88640 1 -- 172.21.15.107:0/194257054 shutdown_connections 2024-05-09T08:32:42.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.830+0000 7f8c15a88640 1 --2- 172.21.15.107:0/194257054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c101034d0 0x7f8c10109d60 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:42.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.830+0000 7f8c15a88640 1 --2- 172.21.15.107:0/194257054 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c10102b10 0x7f8c10102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:42.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.830+0000 7f8c15a88640 1 --2- 172.21.15.107:0/194257054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c100ffb40 0x7f8c100fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:42.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.830+0000 7f8c15a88640 1 -- 172.21.15.107:0/194257054 >> 172.21.15.107:0/194257054 conn(0x7f8c100fb470 msgr2=0x7f8c100fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:42.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.830+0000 7f8c15a88640 1 -- 172.21.15.107:0/194257054 shutdown_connections 2024-05-09T08:32:42.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.830+0000 7f8c15a88640 1 -- 172.21.15.107:0/194257054 wait complete. 2024-05-09T08:32:42.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.831+0000 7f8c15a88640 1 Processor -- start 2024-05-09T08:32:42.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.831+0000 7f8c15a88640 1 -- start start 2024-05-09T08:32:42.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.831+0000 7f8c15a88640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c100ffb40 0x7f8c10102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:42.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.831+0000 7f8c15a88640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c10102b10 0x7f8c10100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:42.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.831+0000 7f8c15a88640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c101034d0 0x7f8c10101000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:42.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.831+0000 7f8c15a88640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1010d420 con 0x7f8c100ffb40 2024-05-09T08:32:42.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.831+0000 7f8c15a88640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1010d2a0 con 0x7f8c10102b10 2024-05-09T08:32:42.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.831+0000 7f8c15a88640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1010d5a0 con 0x7f8c101034d0 2024-05-09T08:32:42.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.831+0000 7f8c0f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c100ffb40 0x7f8c10102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:42.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.831+0000 7f8c0f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c100ffb40 0x7f8c10102480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55462/0 (socket says 172.21.15.107:55462) 2024-05-09T08:32:42.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.831+0000 7f8c0f7fe640 1 -- 172.21.15.107:0/259249917 learned_addr learned my addr 172.21.15.107:0/259249917 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:42.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.832+0000 7f8c0ffff640 1 --2- 172.21.15.107:0/259249917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c101034d0 0x7f8c10101000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:42.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.832+0000 7f8c0effd640 1 --2- 172.21.15.107:0/259249917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c10102b10 0x7f8c10100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:42.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.832+0000 7f8c0f7fe640 1 -- 172.21.15.107:0/259249917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c101034d0 msgr2=0x7f8c10101000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:42.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.832+0000 7f8c0f7fe640 1 --2- 172.21.15.107:0/259249917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c101034d0 0x7f8c10101000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:42.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.832+0000 7f8c0f7fe640 1 -- 172.21.15.107:0/259249917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c10102b10 msgr2=0x7f8c10100ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:42.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.832+0000 7f8c0f7fe640 1 --2- 172.21.15.107:0/259249917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c10102b10 0x7f8c10100ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:42.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.832+0000 7f8c0f7fe640 1 -- 172.21.15.107:0/259249917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c10101540 con 0x7f8c100ffb40 2024-05-09T08:32:42.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.832+0000 7f8c0ffff640 1 --2- 172.21.15.107:0/259249917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c101034d0 0x7f8c10101000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:32:42.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.832+0000 7f8c0f7fe640 1 --2- 172.21.15.107:0/259249917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c100ffb40 0x7f8c10102480 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000c970 tx=0x7f8c0000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:42.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.832+0000 7f8c0cff9640 1 -- 172.21.15.107:0/259249917 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c00004220 con 0x7f8c100ffb40 2024-05-09T08:32:42.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.832+0000 7f8c0cff9640 1 -- 172.21.15.107:0/259249917 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c000043c0 con 0x7f8c100ffb40 2024-05-09T08:32:42.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.832+0000 7f8c15a88640 1 -- 172.21.15.107:0/259249917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c10101830 con 0x7f8c100ffb40 2024-05-09T08:32:42.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.832+0000 7f8c0cff9640 1 -- 172.21.15.107:0/259249917 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0000f450 con 0x7f8c100ffb40 2024-05-09T08:32:42.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.832+0000 7f8c15a88640 1 -- 172.21.15.107:0/259249917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c101a65d0 con 0x7f8c100ffb40 2024-05-09T08:32:42.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.833+0000 7f8c15a88640 1 -- 172.21.15.107:0/259249917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c10108c60 con 0x7f8c100ffb40 2024-05-09T08:32:42.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.834+0000 7f8c0cff9640 1 -- 172.21.15.107:0/259249917 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f8c000079b0 con 0x7f8c100ffb40 2024-05-09T08:32:42.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.835+0000 7f8c0cff9640 1 --2- 172.21.15.107:0/259249917 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8bf80a2b90 0x7f8bf80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:42.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.835+0000 7f8c0effd640 1 --2- 172.21.15.107:0/259249917 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8bf80a2b90 0x7f8bf80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:42.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.835+0000 7f8c0cff9640 1 -- 172.21.15.107:0/259249917 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8c000cf210 con 0x7f8c100ffb40 2024-05-09T08:32:42.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.835+0000 7f8c0effd640 1 --2- 172.21.15.107:0/259249917 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8bf80a2b90 0x7f8bf80a5050 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8c04005fd0 tx=0x7f8c04005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:42.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:42.839+0000 7f8c0cff9640 1 -- 172.21.15.107:0/259249917 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8c00014030 con 0x7f8c100ffb40 2024-05-09T08:32:42.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:42 smithi157 ceph-mon[33614]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-05-09T08:32:42.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:42 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2209726785' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-09T08:32:42.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:42 smithi110 ceph-mon[33622]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-05-09T08:32:42.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:42 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2209726785' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-09T08:32:43.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.014+0000 7f8c15a88640 1 -- 172.21.15.107:0/259249917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8c100624d0 con 0x7f8c100ffb40 2024-05-09T08:32:43.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.014+0000 7f8c0cff9640 1 -- 172.21.15.107:0/259249917 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f8c0009be10 con 0x7f8c100ffb40 2024-05-09T08:32:43.014 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-05-09T08:32:43.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.016+0000 7f8c15a88640 1 -- 172.21.15.107:0/259249917 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8bf80a2b90 msgr2=0x7f8bf80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.016+0000 7f8c15a88640 1 --2- 172.21.15.107:0/259249917 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8bf80a2b90 0x7f8bf80a5050 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8c04005fd0 tx=0x7f8c04005850 comp rx=0 tx=0).stop 2024-05-09T08:32:43.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.016+0000 7f8c15a88640 1 -- 172.21.15.107:0/259249917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c100ffb40 msgr2=0x7f8c10102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.016+0000 7f8c15a88640 1 --2- 172.21.15.107:0/259249917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c100ffb40 0x7f8c10102480 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000c970 tx=0x7f8c0000ce30 comp rx=0 tx=0).stop 2024-05-09T08:32:43.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.016+0000 7f8c15a88640 1 -- 172.21.15.107:0/259249917 shutdown_connections 2024-05-09T08:32:43.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.016+0000 7f8c15a88640 1 --2- 172.21.15.107:0/259249917 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8bf80a2b90 0x7f8bf80a5050 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.016+0000 7f8c15a88640 1 --2- 172.21.15.107:0/259249917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c101034d0 0x7f8c10101000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.016+0000 7f8c15a88640 1 --2- 172.21.15.107:0/259249917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c10102b10 0x7f8c10100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.016+0000 7f8c15a88640 1 --2- 172.21.15.107:0/259249917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c100ffb40 0x7f8c10102480 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.016+0000 7f8c15a88640 1 -- 172.21.15.107:0/259249917 >> 172.21.15.107:0/259249917 conn(0x7f8c100fb470 msgr2=0x7f8c10105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:43.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.016+0000 7f8c15a88640 1 -- 172.21.15.107:0/259249917 shutdown_connections 2024-05-09T08:32:43.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.016+0000 7f8c15a88640 1 -- 172.21.15.107:0/259249917 wait complete. 2024-05-09T08:32:43.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:32:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.133+0000 7fbb68f29640 1 -- 172.21.15.107:0/2280517210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb641019b0 msgr2=0x7fbb64101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.133+0000 7fbb68f29640 1 --2- 172.21.15.107:0/2280517210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb641019b0 0x7fbb64101e30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbb54009900 tx=0x7fbb5402a410 comp rx=0 tx=0).stop 2024-05-09T08:32:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.133+0000 7fbb68f29640 1 -- 172.21.15.107:0/2280517210 shutdown_connections 2024-05-09T08:32:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.133+0000 7fbb68f29640 1 --2- 172.21.15.107:0/2280517210 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb64102370 0x7fbb64108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.133+0000 7fbb68f29640 1 --2- 172.21.15.107:0/2280517210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb641019b0 0x7fbb64101e30 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.133+0000 7fbb68f29640 1 --2- 172.21.15.107:0/2280517210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb64100ca0 0x7fbb641010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.133+0000 7fbb68f29640 1 -- 172.21.15.107:0/2280517210 >> 172.21.15.107:0/2280517210 conn(0x7fbb640fb470 msgr2=0x7fbb640fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.133+0000 7fbb68f29640 1 -- 172.21.15.107:0/2280517210 shutdown_connections 2024-05-09T08:32:43.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.133+0000 7fbb68f29640 1 -- 172.21.15.107:0/2280517210 wait complete. 2024-05-09T08:32:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.133+0000 7fbb68f29640 1 Processor -- start 2024-05-09T08:32:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.133+0000 7fbb68f29640 1 -- start start 2024-05-09T08:32:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb68f29640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb64100ca0 0x7fbb64100360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb68f29640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb641019b0 0x7fbb6419c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb68f29640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb64102370 0x7fbb6419edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb68f29640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6410c2c0 con 0x7fbb64102370 2024-05-09T08:32:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb68f29640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6410c140 con 0x7fbb64100ca0 2024-05-09T08:32:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb68f29640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6410c440 con 0x7fbb641019b0 2024-05-09T08:32:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb62d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb64102370 0x7fbb6419edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb62d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb64102370 0x7fbb6419edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55480/0 (socket says 172.21.15.107:55480) 2024-05-09T08:32:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb62d9d640 1 -- 172.21.15.107:0/3698305314 learned_addr learned my addr 172.21.15.107:0/3698305314 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb6259c640 1 --2- 172.21.15.107:0/3698305314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb64100ca0 0x7fbb64100360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb61d9b640 1 --2- 172.21.15.107:0/3698305314 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb641019b0 0x7fbb6419c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:43.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb62d9d640 1 -- 172.21.15.107:0/3698305314 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb641019b0 msgr2=0x7fbb6419c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb62d9d640 1 --2- 172.21.15.107:0/3698305314 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb641019b0 0x7fbb6419c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb62d9d640 1 -- 172.21.15.107:0/3698305314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb64100ca0 msgr2=0x7fbb64100360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb62d9d640 1 --2- 172.21.15.107:0/3698305314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb64100ca0 0x7fbb64100360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb62d9d640 1 -- 172.21.15.107:0/3698305314 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6419f700 con 0x7fbb64102370 2024-05-09T08:32:43.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb62d9d640 1 --2- 172.21.15.107:0/3698305314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb64102370 0x7fbb6419edf0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c00cad0 tx=0x7fbb5c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:43.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb537fe640 1 -- 172.21.15.107:0/3698305314 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c00ae20 con 0x7fbb64102370 2024-05-09T08:32:43.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb68f29640 1 -- 172.21.15.107:0/3698305314 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb6419f990 con 0x7fbb64102370 2024-05-09T08:32:43.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb537fe640 1 -- 172.21.15.107:0/3698305314 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb5c005020 con 0x7fbb64102370 2024-05-09T08:32:43.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb537fe640 1 -- 172.21.15.107:0/3698305314 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c00f6b0 con 0x7fbb64102370 2024-05-09T08:32:43.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.134+0000 7fbb68f29640 1 -- 172.21.15.107:0/3698305314 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb6419fee0 con 0x7fbb64102370 2024-05-09T08:32:43.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.135+0000 7fbb68f29640 1 -- 172.21.15.107:0/3698305314 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb64107bb0 con 0x7fbb64102370 2024-05-09T08:32:43.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.136+0000 7fbb537fe640 1 -- 172.21.15.107:0/3698305314 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fbb5c0051d0 con 0x7fbb64102370 2024-05-09T08:32:43.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.137+0000 7fbb537fe640 1 --2- 172.21.15.107:0/3698305314 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbb4c0a2b90 0x7fbb4c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:43.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.137+0000 7fbb6259c640 1 --2- 172.21.15.107:0/3698305314 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbb4c0a2b90 0x7fbb4c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:43.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.137+0000 7fbb537fe640 1 -- 172.21.15.107:0/3698305314 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbb5c0cf490 con 0x7fbb64102370 2024-05-09T08:32:43.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.138+0000 7fbb6259c640 1 --2- 172.21.15.107:0/3698305314 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbb4c0a2b90 0x7fbb4c0a5050 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbb580058c0 tx=0x7fbb58005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:43.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.141+0000 7fbb537fe640 1 -- 172.21.15.107:0/3698305314 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbb5c014030 con 0x7fbb64102370 2024-05-09T08:32:43.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.319+0000 7fbb68f29640 1 -- 172.21.15.107:0/3698305314 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbb6402d3c0 con 0x7fbb64102370 2024-05-09T08:32:43.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.319+0000 7fbb537fe640 1 -- 172.21.15.107:0/3698305314 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fbb5c09c090 con 0x7fbb64102370 2024-05-09T08:32:43.320 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:32:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.321+0000 7fbb68f29640 1 -- 172.21.15.107:0/3698305314 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbb4c0a2b90 msgr2=0x7fbb4c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.321+0000 7fbb68f29640 1 --2- 172.21.15.107:0/3698305314 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbb4c0a2b90 0x7fbb4c0a5050 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbb580058c0 tx=0x7fbb58005850 comp rx=0 tx=0).stop 2024-05-09T08:32:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.321+0000 7fbb68f29640 1 -- 172.21.15.107:0/3698305314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb64102370 msgr2=0x7fbb6419edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.321+0000 7fbb68f29640 1 --2- 172.21.15.107:0/3698305314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb64102370 0x7fbb6419edf0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c00cad0 tx=0x7fbb5c00cf90 comp rx=0 tx=0).stop 2024-05-09T08:32:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.322+0000 7fbb68f29640 1 -- 172.21.15.107:0/3698305314 shutdown_connections 2024-05-09T08:32:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.322+0000 7fbb68f29640 1 --2- 172.21.15.107:0/3698305314 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbb4c0a2b90 0x7fbb4c0a5050 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.322+0000 7fbb68f29640 1 --2- 172.21.15.107:0/3698305314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb64102370 0x7fbb6419edf0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.322+0000 7fbb68f29640 1 --2- 172.21.15.107:0/3698305314 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb641019b0 0x7fbb6419c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.322+0000 7fbb68f29640 1 --2- 172.21.15.107:0/3698305314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb64100ca0 0x7fbb64100360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.322+0000 7fbb68f29640 1 -- 172.21.15.107:0/3698305314 >> 172.21.15.107:0/3698305314 conn(0x7fbb640fb470 msgr2=0x7fbb64104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.322+0000 7fbb68f29640 1 -- 172.21.15.107:0/3698305314 shutdown_connections 2024-05-09T08:32:43.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.322+0000 7fbb68f29640 1 -- 172.21.15.107:0/3698305314 wait complete. 2024-05-09T08:32:43.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:32:43.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.437+0000 7f2b83db7640 1 -- 172.21.15.107:0/250493716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b7c102370 msgr2=0x7f2b7c104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.437+0000 7f2b83db7640 1 --2- 172.21.15.107:0/250493716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b7c102370 0x7f2b7c104760 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009900 tx=0x7f2b7802a410 comp rx=0 tx=0).stop 2024-05-09T08:32:43.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.437+0000 7f2b83db7640 1 -- 172.21.15.107:0/250493716 shutdown_connections 2024-05-09T08:32:43.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.437+0000 7f2b83db7640 1 --2- 172.21.15.107:0/250493716 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2b7c1075d0 0x7f2b7c1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.437+0000 7f2b83db7640 1 --2- 172.21.15.107:0/250493716 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b7c104ca0 0x7f2b7c107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.437+0000 7f2b83db7640 1 --2- 172.21.15.107:0/250493716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b7c102370 0x7f2b7c104760 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.437+0000 7f2b83db7640 1 -- 172.21.15.107:0/250493716 >> 172.21.15.107:0/250493716 conn(0x7f2b7c0fbd00 msgr2=0x7f2b7c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:43.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.437+0000 7f2b83db7640 1 -- 172.21.15.107:0/250493716 shutdown_connections 2024-05-09T08:32:43.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.437+0000 7f2b83db7640 1 -- 172.21.15.107:0/250493716 wait complete. 2024-05-09T08:32:43.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.437+0000 7f2b83db7640 1 Processor -- start 2024-05-09T08:32:43.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.437+0000 7f2b83db7640 1 -- start start 2024-05-09T08:32:43.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b83db7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2b7c102370 0x7f2b7c197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b83db7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b7c104ca0 0x7f2b7c198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b83db7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b7c1075d0 0x7f2b7c19f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b83db7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c10d3b0 con 0x7f2b7c104ca0 2024-05-09T08:32:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b83db7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c10d230 con 0x7f2b7c1075d0 2024-05-09T08:32:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b83db7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c10d530 con 0x7f2b7c102370 2024-05-09T08:32:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b81352640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b7c104ca0 0x7f2b7c198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b81352640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b7c104ca0 0x7f2b7c198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55496/0 (socket says 172.21.15.107:55496) 2024-05-09T08:32:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b81352640 1 -- 172.21.15.107:0/2440517253 learned_addr learned my addr 172.21.15.107:0/2440517253 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b82354640 1 --2- 172.21.15.107:0/2440517253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b7c1075d0 0x7f2b7c19f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b81b53640 1 --2- 172.21.15.107:0/2440517253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2b7c102370 0x7f2b7c197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b81352640 1 -- 172.21.15.107:0/2440517253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2b7c102370 msgr2=0x7f2b7c197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b81352640 1 --2- 172.21.15.107:0/2440517253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2b7c102370 0x7f2b7c197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b81352640 1 -- 172.21.15.107:0/2440517253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b7c1075d0 msgr2=0x7f2b7c19f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b81352640 1 --2- 172.21.15.107:0/2440517253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b7c1075d0 0x7f2b7c19f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b81352640 1 -- 172.21.15.107:0/2440517253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b7c19f750 con 0x7f2b7c104ca0 2024-05-09T08:32:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b81b53640 1 --2- 172.21.15.107:0/2440517253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2b7c102370 0x7f2b7c197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:32:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b81352640 1 --2- 172.21.15.107:0/2440517253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b7c104ca0 0x7f2b7c198070 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c00c970 tx=0x7f2b6c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b72ffd640 1 -- 172.21.15.107:0/2440517253 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c007bb0 con 0x7f2b7c104ca0 2024-05-09T08:32:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b83db7640 1 -- 172.21.15.107:0/2440517253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b7c19fa40 con 0x7f2b7c104ca0 2024-05-09T08:32:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b72ffd640 1 -- 172.21.15.107:0/2440517253 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b6c007d50 con 0x7f2b7c104ca0 2024-05-09T08:32:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b72ffd640 1 -- 172.21.15.107:0/2440517253 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c00f450 con 0x7f2b7c104ca0 2024-05-09T08:32:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.438+0000 7f2b83db7640 1 -- 172.21.15.107:0/2440517253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b7c19ff90 con 0x7f2b7c104ca0 2024-05-09T08:32:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.439+0000 7f2b83db7640 1 -- 172.21.15.107:0/2440517253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b7c0fd7f0 con 0x7f2b7c104ca0 2024-05-09T08:32:43.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.441+0000 7f2b72ffd640 1 -- 172.21.15.107:0/2440517253 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f2b6c004eb0 con 0x7f2b7c104ca0 2024-05-09T08:32:43.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.441+0000 7f2b72ffd640 1 --2- 172.21.15.107:0/2440517253 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2b680a2b90 0x7f2b680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:43.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.441+0000 7f2b81b53640 1 --2- 172.21.15.107:0/2440517253 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2b680a2b90 0x7f2b680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.441+0000 7f2b72ffd640 1 -- 172.21.15.107:0/2440517253 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2b6c0cf0a0 con 0x7f2b7c104ca0 2024-05-09T08:32:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.442+0000 7f2b81b53640 1 --2- 172.21.15.107:0/2440517253 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2b680a2b90 0x7f2b680a5050 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2b78006010 tx=0x7f2b78000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.446+0000 7f2b72ffd640 1 -- 172.21.15.107:0/2440517253 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2b6c014030 con 0x7f2b7c104ca0 2024-05-09T08:32:43.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.625+0000 7f2b83db7640 1 -- 172.21.15.107:0/2440517253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2b7c062430 con 0x7f2b7c104ca0 2024-05-09T08:32:43.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.625+0000 7f2b72ffd640 1 -- 172.21.15.107:0/2440517253 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f2b6c09bca0 con 0x7f2b7c104ca0 2024-05-09T08:32:43.625 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-05-09T08:32:43.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.627+0000 7f2b83db7640 1 -- 172.21.15.107:0/2440517253 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2b680a2b90 msgr2=0x7f2b680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.627+0000 7f2b83db7640 1 --2- 172.21.15.107:0/2440517253 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2b680a2b90 0x7f2b680a5050 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2b78006010 tx=0x7f2b78000bc0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.627+0000 7f2b83db7640 1 -- 172.21.15.107:0/2440517253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b7c104ca0 msgr2=0x7f2b7c198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.627+0000 7f2b83db7640 1 --2- 172.21.15.107:0/2440517253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b7c104ca0 0x7f2b7c198070 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c00c970 tx=0x7f2b6c00ce30 comp rx=0 tx=0).stop 2024-05-09T08:32:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.627+0000 7f2b83db7640 1 -- 172.21.15.107:0/2440517253 shutdown_connections 2024-05-09T08:32:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.627+0000 7f2b83db7640 1 --2- 172.21.15.107:0/2440517253 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f2b680a2b90 0x7f2b680a5050 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.627+0000 7f2b83db7640 1 --2- 172.21.15.107:0/2440517253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b7c1075d0 0x7f2b7c19f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.627+0000 7f2b83db7640 1 --2- 172.21.15.107:0/2440517253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b7c104ca0 0x7f2b7c198070 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.627+0000 7f2b83db7640 1 --2- 172.21.15.107:0/2440517253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2b7c102370 0x7f2b7c197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.627+0000 7f2b83db7640 1 -- 172.21.15.107:0/2440517253 >> 172.21.15.107:0/2440517253 conn(0x7f2b7c0fbd00 msgr2=0x7f2b7c102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.627+0000 7f2b83db7640 1 -- 172.21.15.107:0/2440517253 shutdown_connections 2024-05-09T08:32:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.627+0000 7f2b83db7640 1 -- 172.21.15.107:0/2440517253 wait complete. 2024-05-09T08:32:43.637 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:43 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2997438095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:32:43.637 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:43 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/259249917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:32:43.637 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:43 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3698305314' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:32:43.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:32:43.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.742+0000 7fb85a429640 1 -- 172.21.15.107:0/2322444913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb854104c40 msgr2=0x7fb854107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.742+0000 7fb85a429640 1 --2- 172.21.15.107:0/2322444913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb854104c40 0x7fb854107030 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb848009900 tx=0x7fb84802a410 comp rx=0 tx=0).stop 2024-05-09T08:32:43.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.742+0000 7fb85a429640 1 -- 172.21.15.107:0/2322444913 shutdown_connections 2024-05-09T08:32:43.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.742+0000 7fb85a429640 1 --2- 172.21.15.107:0/2322444913 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb854107570 0x7fb8541099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.742+0000 7fb85a429640 1 --2- 172.21.15.107:0/2322444913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb854104c40 0x7fb854107030 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.742+0000 7fb85a429640 1 --2- 172.21.15.107:0/2322444913 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb854102310 0x7fb854104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.742+0000 7fb85a429640 1 -- 172.21.15.107:0/2322444913 >> 172.21.15.107:0/2322444913 conn(0x7fb8540fbd20 msgr2=0x7fb8540fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:43.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.742+0000 7fb85a429640 1 -- 172.21.15.107:0/2322444913 shutdown_connections 2024-05-09T08:32:43.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.742+0000 7fb85a429640 1 -- 172.21.15.107:0/2322444913 wait complete. 2024-05-09T08:32:43.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.742+0000 7fb85a429640 1 Processor -- start 2024-05-09T08:32:43.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.742+0000 7fb85a429640 1 -- start start 2024-05-09T08:32:43.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb85a429640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb854102310 0x7fb854101c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:43.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb85a429640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb854104c40 0x7fb8541002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb85a429640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb854107570 0x7fb854100800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb85a429640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb85410d3c0 con 0x7fb854107570 2024-05-09T08:32:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb85a429640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb85410d240 con 0x7fb854104c40 2024-05-09T08:32:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb85a429640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb85410d540 con 0x7fb854102310 2024-05-09T08:32:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb8589c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb854107570 0x7fb854100800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb8589c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb854107570 0x7fb854100800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55508/0 (socket says 172.21.15.107:55508) 2024-05-09T08:32:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb8589c6640 1 -- 172.21.15.107:0/2698034621 learned_addr learned my addr 172.21.15.107:0/2698034621 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb8589c6640 1 -- 172.21.15.107:0/2698034621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb854102310 msgr2=0x7fb854101c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb8589c6640 1 --2- 172.21.15.107:0/2698034621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb854102310 0x7fb854101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb8589c6640 1 -- 172.21.15.107:0/2698034621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb854104c40 msgr2=0x7fb8541002c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb8589c6640 1 --2- 172.21.15.107:0/2698034621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb854104c40 0x7fb8541002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb8589c6640 1 -- 172.21.15.107:0/2698034621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8541010c0 con 0x7fb854107570 2024-05-09T08:32:43.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb8589c6640 1 --2- 172.21.15.107:0/2698034621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb854107570 0x7fb854100800 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb84c00eaa0 tx=0x7fb84c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:43.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb8517fa640 1 -- 172.21.15.107:0/2698034621 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84c00cdb0 con 0x7fb854107570 2024-05-09T08:32:43.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.743+0000 7fb85a429640 1 -- 172.21.15.107:0/2698034621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8541a1c80 con 0x7fb854107570 2024-05-09T08:32:43.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.744+0000 7fb8517fa640 1 -- 172.21.15.107:0/2698034621 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb84c005020 con 0x7fb854107570 2024-05-09T08:32:43.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.744+0000 7fb8517fa640 1 -- 172.21.15.107:0/2698034621 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84c0116d0 con 0x7fb854107570 2024-05-09T08:32:43.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.744+0000 7fb85a429640 1 -- 172.21.15.107:0/2698034621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8541a21f0 con 0x7fb854107570 2024-05-09T08:32:43.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.745+0000 7fb85a429640 1 -- 172.21.15.107:0/2698034621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8540fd810 con 0x7fb854107570 2024-05-09T08:32:43.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.746+0000 7fb8517fa640 1 -- 172.21.15.107:0/2698034621 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fb84c0051d0 con 0x7fb854107570 2024-05-09T08:32:43.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.746+0000 7fb8517fa640 1 --2- 172.21.15.107:0/2698034621 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb83c0a2b90 0x7fb83c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:43.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.746+0000 7fb853fff640 1 --2- 172.21.15.107:0/2698034621 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb83c0a2b90 0x7fb83c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:43.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.747+0000 7fb8517fa640 1 -- 172.21.15.107:0/2698034621 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb84c0cf600 con 0x7fb854107570 2024-05-09T08:32:43.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.747+0000 7fb853fff640 1 --2- 172.21.15.107:0/2698034621 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb83c0a2b90 0x7fb83c0a5050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb84400a8f0 tx=0x7fb844005bf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:43.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.751+0000 7fb8517fa640 1 -- 172.21.15.107:0/2698034621 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb84c016030 con 0x7fb854107570 2024-05-09T08:32:43.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:43 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2997438095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:32:43.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:43 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/259249917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:32:43.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:43 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3698305314' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:32:43.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:43 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2997438095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:32:43.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:43 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/259249917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:32:43.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:43 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3698305314' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:32:43.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.926+0000 7fb85a429640 1 -- 172.21.15.107:0/2698034621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb8540624d0 con 0x7fb854107570 2024-05-09T08:32:43.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.926+0000 7fb8517fa640 1 -- 172.21.15.107:0/2698034621 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7fb84c09c200 con 0x7fb854107570 2024-05-09T08:32:43.926 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:32:43.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.928+0000 7fb85a429640 1 -- 172.21.15.107:0/2698034621 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb83c0a2b90 msgr2=0x7fb83c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.928+0000 7fb85a429640 1 --2- 172.21.15.107:0/2698034621 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb83c0a2b90 0x7fb83c0a5050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb84400a8f0 tx=0x7fb844005bf0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.928+0000 7fb85a429640 1 -- 172.21.15.107:0/2698034621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb854107570 msgr2=0x7fb854100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:43.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.928+0000 7fb85a429640 1 --2- 172.21.15.107:0/2698034621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb854107570 0x7fb854100800 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb84c00eaa0 tx=0x7fb84c00ef60 comp rx=0 tx=0).stop 2024-05-09T08:32:43.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.928+0000 7fb85a429640 1 -- 172.21.15.107:0/2698034621 shutdown_connections 2024-05-09T08:32:43.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.928+0000 7fb85a429640 1 --2- 172.21.15.107:0/2698034621 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb83c0a2b90 0x7fb83c0a5050 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.928+0000 7fb85a429640 1 --2- 172.21.15.107:0/2698034621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb854107570 0x7fb854100800 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.928+0000 7fb85a429640 1 --2- 172.21.15.107:0/2698034621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb854104c40 0x7fb8541002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.928+0000 7fb85a429640 1 --2- 172.21.15.107:0/2698034621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb854102310 0x7fb854101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:43.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.928+0000 7fb85a429640 1 -- 172.21.15.107:0/2698034621 >> 172.21.15.107:0/2698034621 conn(0x7fb8540fbd20 msgr2=0x7fb854102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:43.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.928+0000 7fb85a429640 1 -- 172.21.15.107:0/2698034621 shutdown_connections 2024-05-09T08:32:43.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:43.928+0000 7fb85a429640 1 -- 172.21.15.107:0/2698034621 wait complete. 2024-05-09T08:32:43.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:32:44.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 -- 172.21.15.107:0/2571921080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e3c1019a0 msgr2=0x7f8e3c101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:44.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 --2- 172.21.15.107:0/2571921080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e3c1019a0 0x7f8e3c101e20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8e24009900 tx=0x7f8e2402a410 comp rx=0 tx=0).stop 2024-05-09T08:32:44.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 -- 172.21.15.107:0/2571921080 shutdown_connections 2024-05-09T08:32:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 --2- 172.21.15.107:0/2571921080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e3c102360 0x7f8e3c108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 --2- 172.21.15.107:0/2571921080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e3c1019a0 0x7f8e3c101e20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 --2- 172.21.15.107:0/2571921080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e3c1007b0 0x7f8e3c100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 -- 172.21.15.107:0/2571921080 >> 172.21.15.107:0/2571921080 conn(0x7f8e3c0fbd20 msgr2=0x7f8e3c0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 -- 172.21.15.107:0/2571921080 shutdown_connections 2024-05-09T08:32:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 -- 172.21.15.107:0/2571921080 wait complete. 2024-05-09T08:32:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 Processor -- start 2024-05-09T08:32:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 -- start start 2024-05-09T08:32:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e3c1007b0 0x7f8e3c06a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e3c1019a0 0x7f8e3c19c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e3c102360 0x7f8e3c19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c10c340 con 0x7f8e3c102360 2024-05-09T08:32:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c10c1c0 con 0x7f8e3c1019a0 2024-05-09T08:32:44.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.041+0000 7f8e4184d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c10c4c0 con 0x7f8e3c1007b0 2024-05-09T08:32:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.042+0000 7f8e3b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e3c102360 0x7f8e3c19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.042+0000 7f8e3b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e3c102360 0x7f8e3c19ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55524/0 (socket says 172.21.15.107:55524) 2024-05-09T08:32:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.042+0000 7f8e3b7fe640 1 -- 172.21.15.107:0/2151648230 learned_addr learned my addr 172.21.15.107:0/2151648230 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.042+0000 7f8e3a7fc640 1 --2- 172.21.15.107:0/2151648230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e3c1019a0 0x7f8e3c19c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.042+0000 7f8e3affd640 1 --2- 172.21.15.107:0/2151648230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e3c1007b0 0x7f8e3c06a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.042+0000 7f8e3b7fe640 1 -- 172.21.15.107:0/2151648230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e3c1007b0 msgr2=0x7f8e3c06a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.042+0000 7f8e3b7fe640 1 --2- 172.21.15.107:0/2151648230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e3c1007b0 0x7f8e3c06a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.042+0000 7f8e3b7fe640 1 -- 172.21.15.107:0/2151648230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e3c1019a0 msgr2=0x7f8e3c19c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.042+0000 7f8e3b7fe640 1 --2- 172.21.15.107:0/2151648230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e3c1019a0 0x7f8e3c19c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.042+0000 7f8e3b7fe640 1 -- 172.21.15.107:0/2151648230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e3c19f7d0 con 0x7f8e3c102360 2024-05-09T08:32:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.042+0000 7f8e3b7fe640 1 --2- 172.21.15.107:0/2151648230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e3c102360 0x7f8e3c19ef10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c009fd0 tx=0x7f8e2c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.042+0000 7f8e4084b640 1 -- 172.21.15.107:0/2151648230 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c011830 con 0x7f8e3c102360 2024-05-09T08:32:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.042+0000 7f8e4184d640 1 -- 172.21.15.107:0/2151648230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e3c19fac0 con 0x7f8e3c102360 2024-05-09T08:32:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.042+0000 7f8e4084b640 1 -- 172.21.15.107:0/2151648230 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e2c016070 con 0x7f8e3c102360 2024-05-09T08:32:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.042+0000 7f8e4084b640 1 -- 172.21.15.107:0/2151648230 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c010de0 con 0x7f8e3c102360 2024-05-09T08:32:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.042+0000 7f8e4184d640 1 -- 172.21.15.107:0/2151648230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e3c1a0060 con 0x7f8e3c102360 2024-05-09T08:32:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.043+0000 7f8e4184d640 1 -- 172.21.15.107:0/2151648230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e3c107bf0 con 0x7f8e3c102360 2024-05-09T08:32:44.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.044+0000 7f8e4084b640 1 -- 172.21.15.107:0/2151648230 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f8e2c0119d0 con 0x7f8e3c102360 2024-05-09T08:32:44.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.045+0000 7f8e4084b640 1 --2- 172.21.15.107:0/2151648230 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8e280a2b90 0x7f8e280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:44.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.045+0000 7f8e3affd640 1 --2- 172.21.15.107:0/2151648230 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8e280a2b90 0x7f8e280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:44.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.045+0000 7f8e4084b640 1 -- 172.21.15.107:0/2151648230 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8e2c0cea40 con 0x7f8e3c102360 2024-05-09T08:32:44.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.045+0000 7f8e3affd640 1 --2- 172.21.15.107:0/2151648230 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8e280a2b90 0x7f8e280a5050 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c101800 tx=0x7f8e34009870 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:44.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.049+0000 7f8e4084b640 1 -- 172.21.15.107:0/2151648230 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8e2c09b640 con 0x7f8e3c102360 2024-05-09T08:32:44.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.268+0000 7f8e4184d640 1 -- 172.21.15.107:0/2151648230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e3c062430 con 0x7f8e3c102360 2024-05-09T08:32:44.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.269+0000 7f8e4084b640 1 -- 172.21.15.107:0/2151648230 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11887 (secure 0 0 0) 0x7f8e2c09af40 con 0x7f8e3c102360 2024-05-09T08:32:44.269 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:32:44.269 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:31:21.056050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24478,24499]},"id":1}]} 2024-05-09T08:32:44.269 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 24 2024-05-09T08:32:44.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.271+0000 7f8e4184d640 1 -- 172.21.15.107:0/2151648230 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8e280a2b90 msgr2=0x7f8e280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:44.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.271+0000 7f8e4184d640 1 --2- 172.21.15.107:0/2151648230 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8e280a2b90 0x7f8e280a5050 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c101800 tx=0x7f8e34009870 comp rx=0 tx=0).stop 2024-05-09T08:32:44.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.271+0000 7f8e4184d640 1 -- 172.21.15.107:0/2151648230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e3c102360 msgr2=0x7f8e3c19ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:44.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.271+0000 7f8e4184d640 1 --2- 172.21.15.107:0/2151648230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e3c102360 0x7f8e3c19ef10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c009fd0 tx=0x7f8e2c00b8e0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.271+0000 7f8e4184d640 1 -- 172.21.15.107:0/2151648230 shutdown_connections 2024-05-09T08:32:44.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.271+0000 7f8e4184d640 1 --2- 172.21.15.107:0/2151648230 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8e280a2b90 0x7f8e280a5050 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.271+0000 7f8e4184d640 1 --2- 172.21.15.107:0/2151648230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e3c102360 0x7f8e3c19ef10 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.271+0000 7f8e4184d640 1 --2- 172.21.15.107:0/2151648230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e3c1019a0 0x7f8e3c19c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.271+0000 7f8e4184d640 1 --2- 172.21.15.107:0/2151648230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e3c1007b0 0x7f8e3c06a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.271+0000 7f8e4184d640 1 -- 172.21.15.107:0/2151648230 >> 172.21.15.107:0/2151648230 conn(0x7f8e3c0fbd20 msgr2=0x7f8e3c104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:44.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.271+0000 7f8e4184d640 1 -- 172.21.15.107:0/2151648230 shutdown_connections 2024-05-09T08:32:44.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.271+0000 7f8e4184d640 1 -- 172.21.15.107:0/2151648230 wait complete. 2024-05-09T08:32:44.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:32:44.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.384+0000 7fcd032f5640 1 -- 172.21.15.107:0/4119318296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc102320 msgr2=0x7fccfc108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:44.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.384+0000 7fcd032f5640 1 --2- 172.21.15.107:0/4119318296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc102320 0x7fccfc108bb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fccf4009ca0 tx=0x7fccf402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.384+0000 7fcd032f5640 1 -- 172.21.15.107:0/4119318296 shutdown_connections 2024-05-09T08:32:44.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.384+0000 7fcd032f5640 1 --2- 172.21.15.107:0/4119318296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc102320 0x7fccfc108bb0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.384+0000 7fcd032f5640 1 --2- 172.21.15.107:0/4119318296 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccfc101960 0x7fccfc101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.384+0000 7fcd032f5640 1 --2- 172.21.15.107:0/4119318296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc100770 0x7fccfc100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.384+0000 7fcd032f5640 1 -- 172.21.15.107:0/4119318296 >> 172.21.15.107:0/4119318296 conn(0x7fccfc0fbce0 msgr2=0x7fccfc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:44.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.385+0000 7fcd032f5640 1 -- 172.21.15.107:0/4119318296 shutdown_connections 2024-05-09T08:32:44.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.385+0000 7fcd032f5640 1 -- 172.21.15.107:0/4119318296 wait complete. 2024-05-09T08:32:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.385+0000 7fcd032f5640 1 Processor -- start 2024-05-09T08:32:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.385+0000 7fcd032f5640 1 -- start start 2024-05-09T08:32:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.385+0000 7fcd032f5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc100770 0x7fccfc197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.385+0000 7fcd032f5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc101960 0x7fccfc197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.385+0000 7fcd032f5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccfc102320 0x7fccfc19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.385+0000 7fcd032f5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc10c280 con 0x7fccfc101960 2024-05-09T08:32:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.385+0000 7fcd032f5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc10c100 con 0x7fccfc100770 2024-05-09T08:32:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.385+0000 7fcd032f5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc10c400 con 0x7fccfc102320 2024-05-09T08:32:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.385+0000 7fcd00890640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc101960 0x7fccfc197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.385+0000 7fcd00890640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc101960 0x7fccfc197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55532/0 (socket says 172.21.15.107:55532) 2024-05-09T08:32:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.385+0000 7fcd00890640 1 -- 172.21.15.107:0/393612489 learned_addr learned my addr 172.21.15.107:0/393612489 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.385+0000 7fcd01091640 1 --2- 172.21.15.107:0/393612489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc100770 0x7fccfc197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.386+0000 7fcd00890640 1 -- 172.21.15.107:0/393612489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccfc102320 msgr2=0x7fccfc19ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.386+0000 7fcd01892640 1 --2- 172.21.15.107:0/393612489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccfc102320 0x7fccfc19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.386+0000 7fcd00890640 1 --2- 172.21.15.107:0/393612489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccfc102320 0x7fccfc19ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.386+0000 7fcd00890640 1 -- 172.21.15.107:0/393612489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc100770 msgr2=0x7fccfc197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.386+0000 7fcd00890640 1 --2- 172.21.15.107:0/393612489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc100770 0x7fccfc197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.386+0000 7fcd00890640 1 -- 172.21.15.107:0/393612489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccfc19f5b0 con 0x7fccfc101960 2024-05-09T08:32:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.386+0000 7fcd01091640 1 --2- 172.21.15.107:0/393612489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc100770 0x7fccfc197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:32:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.386+0000 7fcd00890640 1 --2- 172.21.15.107:0/393612489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc101960 0x7fccfc197ed0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fccec00c920 tx=0x7fccec00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.386+0000 7fccf27fc640 1 -- 172.21.15.107:0/393612489 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec007c70 con 0x7fccfc101960 2024-05-09T08:32:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.386+0000 7fccf27fc640 1 -- 172.21.15.107:0/393612489 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fccec007e10 con 0x7fccfc101960 2024-05-09T08:32:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.386+0000 7fcd032f5640 1 -- 172.21.15.107:0/393612489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccfc19f8a0 con 0x7fccfc101960 2024-05-09T08:32:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.386+0000 7fccf27fc640 1 -- 172.21.15.107:0/393612489 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec018820 con 0x7fccfc101960 2024-05-09T08:32:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.386+0000 7fcd032f5640 1 -- 172.21.15.107:0/393612489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccfc19fdf0 con 0x7fccfc101960 2024-05-09T08:32:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.387+0000 7fcd032f5640 1 -- 172.21.15.107:0/393612489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccfc1a0050 con 0x7fccfc101960 2024-05-09T08:32:44.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.388+0000 7fccf27fc640 1 -- 172.21.15.107:0/393612489 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fccec0189c0 con 0x7fccfc101960 2024-05-09T08:32:44.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.389+0000 7fccf27fc640 1 --2- 172.21.15.107:0/393612489 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fcce80a2c10 0x7fcce80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:44.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.389+0000 7fcd01091640 1 --2- 172.21.15.107:0/393612489 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fcce80a2c10 0x7fcce80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:44.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.389+0000 7fccf27fc640 1 -- 172.21.15.107:0/393612489 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fccec0cf820 con 0x7fccfc101960 2024-05-09T08:32:44.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.389+0000 7fcd01091640 1 --2- 172.21.15.107:0/393612489 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fcce80a2c10 0x7fcce80a50d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fccf8005d00 tx=0x7fccf8005c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:44.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.392+0000 7fccf27fc640 1 -- 172.21.15.107:0/393612489 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fccec014030 con 0x7fccfc101960 2024-05-09T08:32:44.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.571+0000 7fcd032f5640 1 -- 172.21.15.107:0/393612489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fccfc1a0220 con 0x7fccfc101960 2024-05-09T08:32:44.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.573+0000 7fccf27fc640 1 -- 172.21.15.107:0/393612489 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24658 (secure 0 0 0) 0x7fccec09c450 con 0x7fccfc101960 2024-05-09T08:32:44.573 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:32:44.573 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":91,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:31:56.556402+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:32:44.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.575+0000 7fcd032f5640 1 -- 172.21.15.107:0/393612489 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fcce80a2c10 msgr2=0x7fcce80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:44.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.575+0000 7fcd032f5640 1 --2- 172.21.15.107:0/393612489 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fcce80a2c10 0x7fcce80a50d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fccf8005d00 tx=0x7fccf8005c90 comp rx=0 tx=0).stop 2024-05-09T08:32:44.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.575+0000 7fcd032f5640 1 -- 172.21.15.107:0/393612489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc101960 msgr2=0x7fccfc197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:44.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.575+0000 7fcd032f5640 1 --2- 172.21.15.107:0/393612489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc101960 0x7fccfc197ed0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fccec00c920 tx=0x7fccec00cde0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.575+0000 7fcd032f5640 1 -- 172.21.15.107:0/393612489 shutdown_connections 2024-05-09T08:32:44.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.575+0000 7fcd032f5640 1 --2- 172.21.15.107:0/393612489 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fcce80a2c10 0x7fcce80a50d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.575+0000 7fcd032f5640 1 --2- 172.21.15.107:0/393612489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccfc102320 0x7fccfc19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.575+0000 7fcd032f5640 1 --2- 172.21.15.107:0/393612489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc101960 0x7fccfc197ed0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.575+0000 7fcd032f5640 1 --2- 172.21.15.107:0/393612489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc100770 0x7fccfc197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.575+0000 7fcd032f5640 1 -- 172.21.15.107:0/393612489 >> 172.21.15.107:0/393612489 conn(0x7fccfc0fbce0 msgr2=0x7fccfc0689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:44.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.575+0000 7fcd032f5640 1 -- 172.21.15.107:0/393612489 shutdown_connections 2024-05-09T08:32:44.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.575+0000 7fcd032f5640 1 -- 172.21.15.107:0/393612489 wait complete. 2024-05-09T08:32:44.586 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:26:21.698338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:30:58.268633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:30:59.345579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:31:03.065991+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:32:44.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:32:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.691+0000 7f94df709640 1 -- 172.21.15.107:0/2300015670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d81000e0 msgr2=0x7f94d8102510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.691+0000 7f94df709640 1 --2- 172.21.15.107:0/2300015670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d81000e0 0x7f94d8102510 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f94d4009900 tx=0x7f94d402a410 comp rx=0 tx=0).stop 2024-05-09T08:32:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.691+0000 7f94df709640 1 -- 172.21.15.107:0/2300015670 shutdown_connections 2024-05-09T08:32:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.691+0000 7f94df709640 1 --2- 172.21.15.107:0/2300015670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94d8105400 0x7f94d8105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.691+0000 7f94df709640 1 --2- 172.21.15.107:0/2300015670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94d8102a50 0x7f94d8104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.691+0000 7f94df709640 1 --2- 172.21.15.107:0/2300015670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d81000e0 0x7f94d8102510 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.691+0000 7f94df709640 1 -- 172.21.15.107:0/2300015670 >> 172.21.15.107:0/2300015670 conn(0x7f94d80fbce0 msgr2=0x7f94d80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.692+0000 7f94df709640 1 -- 172.21.15.107:0/2300015670 shutdown_connections 2024-05-09T08:32:44.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.692+0000 7f94df709640 1 -- 172.21.15.107:0/2300015670 wait complete. 2024-05-09T08:32:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.692+0000 7f94df709640 1 Processor -- start 2024-05-09T08:32:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.692+0000 7f94df709640 1 -- start start 2024-05-09T08:32:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.692+0000 7f94df709640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94d81000e0 0x7f94d819e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.692+0000 7f94df709640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d8102a50 0x7f94d819e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.692+0000 7f94df709640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94d8105400 0x7f94d81a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.692+0000 7f94df709640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f94d8113970 con 0x7f94d8102a50 2024-05-09T08:32:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.692+0000 7f94df709640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f94d81137f0 con 0x7f94d8105400 2024-05-09T08:32:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.692+0000 7f94df709640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f94d8113af0 con 0x7f94d81000e0 2024-05-09T08:32:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.692+0000 7f94dcca4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d8102a50 0x7f94d819e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.692+0000 7f94dcca4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d8102a50 0x7f94d819e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55550/0 (socket says 172.21.15.107:55550) 2024-05-09T08:32:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.692+0000 7f94dcca4640 1 -- 172.21.15.107:0/2350931886 learned_addr learned my addr 172.21.15.107:0/2350931886 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:44.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.692+0000 7f94ddca6640 1 --2- 172.21.15.107:0/2350931886 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94d8105400 0x7f94d81a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.693+0000 7f94dd4a5640 1 --2- 172.21.15.107:0/2350931886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94d81000e0 0x7f94d819e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.693+0000 7f94dcca4640 1 -- 172.21.15.107:0/2350931886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94d81000e0 msgr2=0x7f94d819e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.693+0000 7f94dcca4640 1 --2- 172.21.15.107:0/2350931886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94d81000e0 0x7f94d819e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.693+0000 7f94dcca4640 1 -- 172.21.15.107:0/2350931886 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94d8105400 msgr2=0x7f94d81a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.693+0000 7f94dcca4640 1 --2- 172.21.15.107:0/2350931886 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94d8105400 0x7f94d81a5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.693+0000 7f94dcca4640 1 -- 172.21.15.107:0/2350931886 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d81a5c80 con 0x7f94d8102a50 2024-05-09T08:32:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.693+0000 7f94dd4a5640 1 --2- 172.21.15.107:0/2350931886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94d81000e0 0x7f94d819e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:32:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.693+0000 7f94dcca4640 1 --2- 172.21.15.107:0/2350931886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d8102a50 0x7f94d819e5f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f94c800c920 tx=0x7f94c800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.693+0000 7f94ce7fc640 1 -- 172.21.15.107:0/2350931886 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c8007c70 con 0x7f94d8102a50 2024-05-09T08:32:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.693+0000 7f94ce7fc640 1 -- 172.21.15.107:0/2350931886 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94c8007e10 con 0x7f94d8102a50 2024-05-09T08:32:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.693+0000 7f94df709640 1 -- 172.21.15.107:0/2350931886 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94d81a5f70 con 0x7f94d8102a50 2024-05-09T08:32:44.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.693+0000 7f94ce7fc640 1 -- 172.21.15.107:0/2350931886 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c8018820 con 0x7f94d8102a50 2024-05-09T08:32:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.693+0000 7f94df709640 1 -- 172.21.15.107:0/2350931886 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94d81a6430 con 0x7f94d8102a50 2024-05-09T08:32:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.694+0000 7f94df709640 1 -- 172.21.15.107:0/2350931886 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94d80fd7d0 con 0x7f94d8102a50 2024-05-09T08:32:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.695+0000 7f94ce7fc640 1 -- 172.21.15.107:0/2350931886 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f94c80189c0 con 0x7f94d8102a50 2024-05-09T08:32:44.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.696+0000 7f94ce7fc640 1 --2- 172.21.15.107:0/2350931886 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f94c40a2b90 0x7f94c40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:44.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.696+0000 7f94dd4a5640 1 --2- 172.21.15.107:0/2350931886 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f94c40a2b90 0x7f94c40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:44.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.696+0000 7f94ce7fc640 1 -- 172.21.15.107:0/2350931886 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f94c80cf770 con 0x7f94d8102a50 2024-05-09T08:32:44.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.696+0000 7f94dd4a5640 1 --2- 172.21.15.107:0/2350931886 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f94c40a2b90 0x7f94c40a5050 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f94d4006010 tx=0x7f94d4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:44.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.699+0000 7f94ce7fc640 1 -- 172.21.15.107:0/2350931886 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f94c8014030 con 0x7f94d8102a50 2024-05-09T08:32:44.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:44 smithi157 ceph-mon[33614]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:32:44.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:44 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2440517253' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:32:44.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:44 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2698034621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:32:44.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:44 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2151648230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:32:44.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:44 smithi110 ceph-mon[33622]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:32:44.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:44 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2440517253' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:32:44.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:44 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2698034621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:32:44.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:44 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2151648230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:32:44.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.894+0000 7f94df709640 1 -- 172.21.15.107:0/2350931886 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f94d802d1c0 con 0x7f94d8102a50 2024-05-09T08:32:44.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.895+0000 7f94ce7fc640 1 -- 172.21.15.107:0/2350931886 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f94c809c400 con 0x7f94d8102a50 2024-05-09T08:32:44.895 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-05-09T08:32:44.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.897+0000 7f94df709640 1 -- 172.21.15.107:0/2350931886 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f94c40a2b90 msgr2=0x7f94c40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:44.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.897+0000 7f94df709640 1 --2- 172.21.15.107:0/2350931886 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f94c40a2b90 0x7f94c40a5050 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f94d4006010 tx=0x7f94d4000bc0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.897+0000 7f94df709640 1 -- 172.21.15.107:0/2350931886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d8102a50 msgr2=0x7f94d819e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:44.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.897+0000 7f94df709640 1 --2- 172.21.15.107:0/2350931886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d8102a50 0x7f94d819e5f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f94c800c920 tx=0x7f94c800cde0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.897+0000 7f94df709640 1 -- 172.21.15.107:0/2350931886 shutdown_connections 2024-05-09T08:32:44.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.897+0000 7f94df709640 1 --2- 172.21.15.107:0/2350931886 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f94c40a2b90 0x7f94c40a5050 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.897+0000 7f94df709640 1 --2- 172.21.15.107:0/2350931886 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94d8105400 0x7f94d81a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.897+0000 7f94df709640 1 --2- 172.21.15.107:0/2350931886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94d8102a50 0x7f94d819e5f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.897+0000 7f94df709640 1 --2- 172.21.15.107:0/2350931886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94d81000e0 0x7f94d819e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:44.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.897+0000 7f94df709640 1 -- 172.21.15.107:0/2350931886 >> 172.21.15.107:0/2350931886 conn(0x7f94d80fbce0 msgr2=0x7f94d8103400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:44.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.897+0000 7f94df709640 1 -- 172.21.15.107:0/2350931886 shutdown_connections 2024-05-09T08:32:44.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:44.897+0000 7f94df709640 1 -- 172.21.15.107:0/2350931886 wait complete. 2024-05-09T08:32:44.907 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:44 smithi107 ceph-mon[30122]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:32:44.907 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:44 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2440517253' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:32:44.907 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:44 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2698034621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:32:44.907 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:44 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2151648230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:32:44.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:32:45.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.011+0000 7f29bbec4640 1 -- 172.21.15.107:0/1422577950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4102340 msgr2=0x7f29b4108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.011+0000 7f29bbec4640 1 --2- 172.21.15.107:0/1422577950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4102340 0x7f29b4108bd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f29ac009ca0 tx=0x7f29ac02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.011+0000 7f29bbec4640 1 -- 172.21.15.107:0/1422577950 shutdown_connections 2024-05-09T08:32:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.011+0000 7f29bbec4640 1 --2- 172.21.15.107:0/1422577950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4102340 0x7f29b4108bd0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.011+0000 7f29bbec4640 1 --2- 172.21.15.107:0/1422577950 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29b4101980 0x7f29b4101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.011+0000 7f29bbec4640 1 --2- 172.21.15.107:0/1422577950 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29b4100790 0x7f29b4100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.011+0000 7f29bbec4640 1 -- 172.21.15.107:0/1422577950 >> 172.21.15.107:0/1422577950 conn(0x7f29b40fbd00 msgr2=0x7f29b40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.011+0000 7f29bbec4640 1 -- 172.21.15.107:0/1422577950 shutdown_connections 2024-05-09T08:32:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.011+0000 7f29bbec4640 1 -- 172.21.15.107:0/1422577950 wait complete. 2024-05-09T08:32:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29bbec4640 1 Processor -- start 2024-05-09T08:32:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29bbec4640 1 -- start start 2024-05-09T08:32:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29bbec4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29b4100790 0x7f29b406a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29bbec4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29b4101980 0x7f29b419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29bbec4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4102340 0x7f29b419edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29bbec4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f29b410c2c0 con 0x7f29b4102340 2024-05-09T08:32:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29bbec4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f29b410c140 con 0x7f29b4100790 2024-05-09T08:32:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29bbec4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f29b410c440 con 0x7f29b4101980 2024-05-09T08:32:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29ba461640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4102340 0x7f29b419edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29ba461640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4102340 0x7f29b419edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55568/0 (socket says 172.21.15.107:55568) 2024-05-09T08:32:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29ba461640 1 -- 172.21.15.107:0/2127047985 learned_addr learned my addr 172.21.15.107:0/2127047985 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29b9c60640 1 --2- 172.21.15.107:0/2127047985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29b4100790 0x7f29b406a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29b945f640 1 --2- 172.21.15.107:0/2127047985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29b4101980 0x7f29b419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29ba461640 1 -- 172.21.15.107:0/2127047985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29b4101980 msgr2=0x7f29b419c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29ba461640 1 --2- 172.21.15.107:0/2127047985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29b4101980 0x7f29b419c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29ba461640 1 -- 172.21.15.107:0/2127047985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29b4100790 msgr2=0x7f29b406a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29ba461640 1 --2- 172.21.15.107:0/2127047985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29b4100790 0x7f29b406a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29ba461640 1 -- 172.21.15.107:0/2127047985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b419f670 con 0x7f29b4102340 2024-05-09T08:32:45.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29b945f640 1 --2- 172.21.15.107:0/2127047985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29b4101980 0x7f29b419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:32:45.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29b9c60640 1 --2- 172.21.15.107:0/2127047985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29b4100790 0x7f29b406a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:32:45.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29ba461640 1 --2- 172.21.15.107:0/2127047985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4102340 0x7f29b419edb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f29ac009dc0 tx=0x7f29ac003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:45.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.012+0000 7f29aaffd640 1 -- 172.21.15.107:0/2127047985 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac032ec0 con 0x7f29b4102340 2024-05-09T08:32:45.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.013+0000 7f29aaffd640 1 -- 172.21.15.107:0/2127047985 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29ac0044f0 con 0x7f29b4102340 2024-05-09T08:32:45.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.013+0000 7f29bbec4640 1 -- 172.21.15.107:0/2127047985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b419f900 con 0x7f29b4102340 2024-05-09T08:32:45.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.013+0000 7f29aaffd640 1 -- 172.21.15.107:0/2127047985 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac030d40 con 0x7f29b4102340 2024-05-09T08:32:45.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.013+0000 7f29bbec4640 1 -- 172.21.15.107:0/2127047985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b419fea0 con 0x7f29b4102340 2024-05-09T08:32:45.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.013+0000 7f29bbec4640 1 -- 172.21.15.107:0/2127047985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29b4107ab0 con 0x7f29b4102340 2024-05-09T08:32:45.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.015+0000 7f29aaffd640 1 -- 172.21.15.107:0/2127047985 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f29ac004040 con 0x7f29b4102340 2024-05-09T08:32:45.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.015+0000 7f29aaffd640 1 --2- 172.21.15.107:0/2127047985 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f29a00a2db0 0x7f29a00a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.015+0000 7f29b9c60640 1 --2- 172.21.15.107:0/2127047985 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f29a00a2db0 0x7f29a00a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.015+0000 7f29aaffd640 1 -- 172.21.15.107:0/2127047985 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f29ac0f0b00 con 0x7f29b4102340 2024-05-09T08:32:45.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.016+0000 7f29b9c60640 1 --2- 172.21.15.107:0/2127047985 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f29a00a2db0 0x7f29a00a5270 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f29b00058c0 tx=0x7f29b0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:45.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.019+0000 7f29aaffd640 1 -- 172.21.15.107:0/2127047985 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f29ac0bd700 con 0x7f29b4102340 2024-05-09T08:32:45.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.193+0000 7f29bbec4640 1 -- 172.21.15.107:0/2127047985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f29b402d350 con 0x7f29b4102340 2024-05-09T08:32:45.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.194+0000 7f29aaffd640 1 -- 172.21.15.107:0/2127047985 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f29ac0bd000 con 0x7f29b4102340 2024-05-09T08:32:45.194 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:32:45.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.195+0000 7f29bbec4640 1 -- 172.21.15.107:0/2127047985 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f29a00a2db0 msgr2=0x7f29a00a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.195+0000 7f29bbec4640 1 --2- 172.21.15.107:0/2127047985 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f29a00a2db0 0x7f29a00a5270 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f29b00058c0 tx=0x7f29b0005850 comp rx=0 tx=0).stop 2024-05-09T08:32:45.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.195+0000 7f29bbec4640 1 -- 172.21.15.107:0/2127047985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4102340 msgr2=0x7f29b419edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.195+0000 7f29bbec4640 1 --2- 172.21.15.107:0/2127047985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4102340 0x7f29b419edb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f29ac009dc0 tx=0x7f29ac003ba0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.195+0000 7f29bbec4640 1 -- 172.21.15.107:0/2127047985 shutdown_connections 2024-05-09T08:32:45.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.195+0000 7f29bbec4640 1 --2- 172.21.15.107:0/2127047985 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f29a00a2db0 0x7f29a00a5270 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.195+0000 7f29bbec4640 1 --2- 172.21.15.107:0/2127047985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4102340 0x7f29b419edb0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.195+0000 7f29bbec4640 1 --2- 172.21.15.107:0/2127047985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29b4101980 0x7f29b419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.195+0000 7f29bbec4640 1 --2- 172.21.15.107:0/2127047985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29b4100790 0x7f29b406a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.196+0000 7f29bbec4640 1 -- 172.21.15.107:0/2127047985 >> 172.21.15.107:0/2127047985 conn(0x7f29b40fbd00 msgr2=0x7f29b4104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:45.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.196+0000 7f29bbec4640 1 -- 172.21.15.107:0/2127047985 shutdown_connections 2024-05-09T08:32:45.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.196+0000 7f29bbec4640 1 -- 172.21.15.107:0/2127047985 wait complete. 2024-05-09T08:32:45.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:32:45.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.310+0000 7f593e114640 1 -- 172.21.15.107:0/1413961651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5938102320 msgr2=0x7f5938108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.310+0000 7f593e114640 1 --2- 172.21.15.107:0/1413961651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5938102320 0x7f5938108bb0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5930009ca0 tx=0x7f593002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.310+0000 7f593e114640 1 -- 172.21.15.107:0/1413961651 shutdown_connections 2024-05-09T08:32:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.310+0000 7f593e114640 1 --2- 172.21.15.107:0/1413961651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5938102320 0x7f5938108bb0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.310+0000 7f593e114640 1 --2- 172.21.15.107:0/1413961651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5938101960 0x7f5938101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.310+0000 7f593e114640 1 --2- 172.21.15.107:0/1413961651 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5938100770 0x7f5938100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.310+0000 7f593e114640 1 -- 172.21.15.107:0/1413961651 >> 172.21.15.107:0/1413961651 conn(0x7f59380fbce0 msgr2=0x7f59380fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.310+0000 7f593e114640 1 -- 172.21.15.107:0/1413961651 shutdown_connections 2024-05-09T08:32:45.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.310+0000 7f593e114640 1 -- 172.21.15.107:0/1413961651 wait complete. 2024-05-09T08:32:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.310+0000 7f593e114640 1 Processor -- start 2024-05-09T08:32:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.310+0000 7f593e114640 1 -- start start 2024-05-09T08:32:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f593e114640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5938100770 0x7f5938197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f593e114640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5938101960 0x7f5938197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f593e114640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5938102320 0x7f593819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f593e114640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f593810c280 con 0x7f5938101960 2024-05-09T08:32:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f593e114640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f593810c100 con 0x7f5938102320 2024-05-09T08:32:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f593e114640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f593810c400 con 0x7f5938100770 2024-05-09T08:32:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f5936ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5938101960 0x7f5938197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f5936ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5938101960 0x7f5938197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55572/0 (socket says 172.21.15.107:55572) 2024-05-09T08:32:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f5936ffd640 1 -- 172.21.15.107:0/685525979 learned_addr learned my addr 172.21.15.107:0/685525979 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f5936ffd640 1 -- 172.21.15.107:0/685525979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5938100770 msgr2=0x7f5938197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f59377fe640 1 --2- 172.21.15.107:0/685525979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5938100770 0x7f5938197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f5936ffd640 1 --2- 172.21.15.107:0/685525979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5938100770 0x7f5938197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f5937fff640 1 --2- 172.21.15.107:0/685525979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5938102320 0x7f593819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f5936ffd640 1 -- 172.21.15.107:0/685525979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5938102320 msgr2=0x7f593819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f5936ffd640 1 --2- 172.21.15.107:0/685525979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5938102320 0x7f593819f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f5936ffd640 1 -- 172.21.15.107:0/685525979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f593819f670 con 0x7f5938101960 2024-05-09T08:32:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f5936ffd640 1 --2- 172.21.15.107:0/685525979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5938101960 0x7f5938197fe0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f592c00c920 tx=0x7f592c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f5934ff9640 1 -- 172.21.15.107:0/685525979 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f592c007c70 con 0x7f5938101960 2024-05-09T08:32:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f5934ff9640 1 -- 172.21.15.107:0/685525979 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f592c007e10 con 0x7f5938101960 2024-05-09T08:32:45.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f593e114640 1 -- 172.21.15.107:0/685525979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f593819f960 con 0x7f5938101960 2024-05-09T08:32:45.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f5934ff9640 1 -- 172.21.15.107:0/685525979 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f592c018820 con 0x7f5938101960 2024-05-09T08:32:45.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.311+0000 7f593e114640 1 -- 172.21.15.107:0/685525979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f593819feb0 con 0x7f5938101960 2024-05-09T08:32:45.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.313+0000 7f593e114640 1 -- 172.21.15.107:0/685525979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59381a0140 con 0x7f5938101960 2024-05-09T08:32:45.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.314+0000 7f5934ff9640 1 -- 172.21.15.107:0/685525979 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f592c0189c0 con 0x7f5938101960 2024-05-09T08:32:45.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.314+0000 7f5934ff9640 1 --2- 172.21.15.107:0/685525979 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f59200a2b90 0x7f59200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.314+0000 7f5934ff9640 1 -- 172.21.15.107:0/685525979 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f592c0cf500 con 0x7f5938101960 2024-05-09T08:32:45.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.314+0000 7f59377fe640 1 --2- 172.21.15.107:0/685525979 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f59200a2b90 0x7f59200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.315+0000 7f59377fe640 1 --2- 172.21.15.107:0/685525979 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f59200a2b90 0x7f59200a5050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5928005fd0 tx=0x7f5928005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:45.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.319+0000 7f5934ff9640 1 -- 172.21.15.107:0/685525979 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f592c014030 con 0x7f5938101960 2024-05-09T08:32:45.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.497+0000 7f593e114640 1 -- 172.21.15.107:0/685525979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f59381a0810 con 0x7f5938101960 2024-05-09T08:32:45.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.497+0000 7f5934ff9640 1 -- 172.21.15.107:0/685525979 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f592c00de20 con 0x7f5938101960 2024-05-09T08:32:45.497 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-05-09T08:32:45.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.499+0000 7f593e114640 1 -- 172.21.15.107:0/685525979 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f59200a2b90 msgr2=0x7f59200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.499+0000 7f593e114640 1 --2- 172.21.15.107:0/685525979 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f59200a2b90 0x7f59200a5050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5928005fd0 tx=0x7f5928005c70 comp rx=0 tx=0).stop 2024-05-09T08:32:45.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.499+0000 7f593e114640 1 -- 172.21.15.107:0/685525979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5938101960 msgr2=0x7f5938197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.499+0000 7f593e114640 1 --2- 172.21.15.107:0/685525979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5938101960 0x7f5938197fe0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f592c00c920 tx=0x7f592c00cde0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.499+0000 7f593e114640 1 -- 172.21.15.107:0/685525979 shutdown_connections 2024-05-09T08:32:45.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.499+0000 7f593e114640 1 --2- 172.21.15.107:0/685525979 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f59200a2b90 0x7f59200a5050 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.499+0000 7f593e114640 1 --2- 172.21.15.107:0/685525979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5938102320 0x7f593819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.499+0000 7f593e114640 1 --2- 172.21.15.107:0/685525979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5938101960 0x7f5938197fe0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.499+0000 7f593e114640 1 --2- 172.21.15.107:0/685525979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5938100770 0x7f5938197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.499+0000 7f593e114640 1 -- 172.21.15.107:0/685525979 >> 172.21.15.107:0/685525979 conn(0x7f59380fbce0 msgr2=0x7f59380fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:45.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.499+0000 7f593e114640 1 -- 172.21.15.107:0/685525979 shutdown_connections 2024-05-09T08:32:45.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.499+0000 7f593e114640 1 -- 172.21.15.107:0/685525979 wait complete. 2024-05-09T08:32:45.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:32:45.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.616+0000 7f81a5189640 1 -- 172.21.15.107:0/1665075067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a0101980 msgr2=0x7f81a0101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.616+0000 7f81a5189640 1 --2- 172.21.15.107:0/1665075067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a0101980 0x7f81a0101e00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8188009900 tx=0x7f818802a410 comp rx=0 tx=0).stop 2024-05-09T08:32:45.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.616+0000 7f81a5189640 1 -- 172.21.15.107:0/1665075067 shutdown_connections 2024-05-09T08:32:45.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.616+0000 7f81a5189640 1 --2- 172.21.15.107:0/1665075067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81a0102340 0x7f81a0108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.616+0000 7f81a5189640 1 --2- 172.21.15.107:0/1665075067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a0101980 0x7f81a0101e00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.616+0000 7f81a5189640 1 --2- 172.21.15.107:0/1665075067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81a0100790 0x7f81a0100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.616+0000 7f81a5189640 1 -- 172.21.15.107:0/1665075067 >> 172.21.15.107:0/1665075067 conn(0x7f81a00fbd20 msgr2=0x7f81a00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:45.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.616+0000 7f81a5189640 1 -- 172.21.15.107:0/1665075067 shutdown_connections 2024-05-09T08:32:45.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.616+0000 7f81a5189640 1 -- 172.21.15.107:0/1665075067 wait complete. 2024-05-09T08:32:45.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.616+0000 7f81a5189640 1 Processor -- start 2024-05-09T08:32:45.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.616+0000 7f81a5189640 1 -- start start 2024-05-09T08:32:45.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.616+0000 7f81a5189640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81a0100790 0x7f81a0197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f81a5189640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81a0101980 0x7f81a0198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f81a5189640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a0102340 0x7f81a019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f81a5189640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f81a010d3e0 con 0x7f81a0102340 2024-05-09T08:32:45.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f81a5189640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f81a010d260 con 0x7f81a0101980 2024-05-09T08:32:45.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f81a5189640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f81a010d560 con 0x7f81a0100790 2024-05-09T08:32:45.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f819f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a0102340 0x7f81a019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f819f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a0102340 0x7f81a019f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55596/0 (socket says 172.21.15.107:55596) 2024-05-09T08:32:45.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f819f59e640 1 -- 172.21.15.107:0/3653782218 learned_addr learned my addr 172.21.15.107:0/3653782218 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:45.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f8197fff640 1 --2- 172.21.15.107:0/3653782218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81a0101980 0x7f81a0198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f819ed9d640 1 --2- 172.21.15.107:0/3653782218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81a0100790 0x7f81a0197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f819f59e640 1 -- 172.21.15.107:0/3653782218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81a0100790 msgr2=0x7f81a0197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f819f59e640 1 --2- 172.21.15.107:0/3653782218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81a0100790 0x7f81a0197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f819f59e640 1 -- 172.21.15.107:0/3653782218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81a0101980 msgr2=0x7f81a0198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f819f59e640 1 --2- 172.21.15.107:0/3653782218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81a0101980 0x7f81a0198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f819f59e640 1 -- 172.21.15.107:0/3653782218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a019f700 con 0x7f81a0102340 2024-05-09T08:32:45.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f819f59e640 1 --2- 172.21.15.107:0/3653782218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a0102340 0x7f81a019f090 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f819800eb10 tx=0x7f819800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:45.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f819cd99640 1 -- 172.21.15.107:0/3653782218 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8198004b20 con 0x7f81a0102340 2024-05-09T08:32:45.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f81a5189640 1 -- 172.21.15.107:0/3653782218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81a019f9f0 con 0x7f81a0102340 2024-05-09T08:32:45.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f819cd99640 1 -- 172.21.15.107:0/3653782218 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8198004cc0 con 0x7f81a0102340 2024-05-09T08:32:45.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f819cd99640 1 -- 172.21.15.107:0/3653782218 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8198010690 con 0x7f81a0102340 2024-05-09T08:32:45.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.617+0000 7f81a5189640 1 -- 172.21.15.107:0/3653782218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81a019ff40 con 0x7f81a0102340 2024-05-09T08:32:45.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.618+0000 7f81a5189640 1 -- 172.21.15.107:0/3653782218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81a0107b70 con 0x7f81a0102340 2024-05-09T08:32:45.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.619+0000 7f819cd99640 1 -- 172.21.15.107:0/3653782218 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f8198005020 con 0x7f81a0102340 2024-05-09T08:32:45.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.620+0000 7f819cd99640 1 --2- 172.21.15.107:0/3653782218 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f818c0a2b90 0x7f818c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.620+0000 7f819ed9d640 1 --2- 172.21.15.107:0/3653782218 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f818c0a2b90 0x7f818c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.620+0000 7f819cd99640 1 -- 172.21.15.107:0/3653782218 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f81980cfb70 con 0x7f81a0102340 2024-05-09T08:32:45.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.621+0000 7f819ed9d640 1 --2- 172.21.15.107:0/3653782218 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f818c0a2b90 0x7f818c0a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8190005d00 tx=0x7f8190005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:45.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.625+0000 7f819cd99640 1 -- 172.21.15.107:0/3653782218 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f819809c770 con 0x7f81a0102340 2024-05-09T08:32:45.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:45 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/393612489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:32:45.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:45 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2350931886' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:32:45.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:45 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2127047985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:32:45.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:45 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/685525979' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:32:45.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.800+0000 7f81a5189640 1 -- 172.21.15.107:0/3653782218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f81a0062430 con 0x7f81a0102340 2024-05-09T08:32:45.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.800+0000 7f819cd99640 1 -- 172.21.15.107:0/3653782218 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f819809c070 con 0x7f81a0102340 2024-05-09T08:32:45.801 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:32:45.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.802+0000 7f81a5189640 1 -- 172.21.15.107:0/3653782218 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f818c0a2b90 msgr2=0x7f818c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.802+0000 7f81a5189640 1 --2- 172.21.15.107:0/3653782218 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f818c0a2b90 0x7f818c0a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8190005d00 tx=0x7f8190005c50 comp rx=0 tx=0).stop 2024-05-09T08:32:45.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.802+0000 7f81a5189640 1 -- 172.21.15.107:0/3653782218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a0102340 msgr2=0x7f81a019f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.802+0000 7f81a5189640 1 --2- 172.21.15.107:0/3653782218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a0102340 0x7f81a019f090 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f819800eb10 tx=0x7f819800c2d0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.802+0000 7f81a5189640 1 -- 172.21.15.107:0/3653782218 shutdown_connections 2024-05-09T08:32:45.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.802+0000 7f81a5189640 1 --2- 172.21.15.107:0/3653782218 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f818c0a2b90 0x7f818c0a5050 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.802+0000 7f81a5189640 1 --2- 172.21.15.107:0/3653782218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81a0102340 0x7f81a019f090 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.802+0000 7f81a5189640 1 --2- 172.21.15.107:0/3653782218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81a0101980 0x7f81a0198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.802+0000 7f81a5189640 1 --2- 172.21.15.107:0/3653782218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81a0100790 0x7f81a0197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.802+0000 7f81a5189640 1 -- 172.21.15.107:0/3653782218 >> 172.21.15.107:0/3653782218 conn(0x7f81a00fbd20 msgr2=0x7f81a0104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:45.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.803+0000 7f81a5189640 1 -- 172.21.15.107:0/3653782218 shutdown_connections 2024-05-09T08:32:45.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.803+0000 7f81a5189640 1 -- 172.21.15.107:0/3653782218 wait complete. 2024-05-09T08:32:45.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:32:45.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:45 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/393612489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:32:45.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:45 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2350931886' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:32:45.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:45 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2127047985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:32:45.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:45 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/685525979' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:32:45.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:45 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/393612489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:32:45.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:45 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2350931886' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:32:45.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:45 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2127047985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:32:45.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:45 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/685525979' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:32:45.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.915+0000 7facf909e640 1 -- 172.21.15.107:0/2932569033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf41023a0 msgr2=0x7facf4108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.915+0000 7facf909e640 1 --2- 172.21.15.107:0/2932569033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf41023a0 0x7facf4108c30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7facec009ca0 tx=0x7facec02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.915+0000 7facf909e640 1 -- 172.21.15.107:0/2932569033 shutdown_connections 2024-05-09T08:32:45.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.915+0000 7facf909e640 1 --2- 172.21.15.107:0/2932569033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf41023a0 0x7facf4108c30 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.915+0000 7facf909e640 1 --2- 172.21.15.107:0/2932569033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf41019e0 0x7facf4101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.915+0000 7facf909e640 1 --2- 172.21.15.107:0/2932569033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facf41007f0 0x7facf4100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.915+0000 7facf909e640 1 -- 172.21.15.107:0/2932569033 >> 172.21.15.107:0/2932569033 conn(0x7facf40fbd80 msgr2=0x7facf40fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:45.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.915+0000 7facf909e640 1 -- 172.21.15.107:0/2932569033 shutdown_connections 2024-05-09T08:32:45.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.915+0000 7facf909e640 1 -- 172.21.15.107:0/2932569033 wait complete. 2024-05-09T08:32:45.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf909e640 1 Processor -- start 2024-05-09T08:32:45.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf909e640 1 -- start start 2024-05-09T08:32:45.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf909e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf41007f0 0x7facf41979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf909e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf41019e0 0x7facf4197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf909e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facf41023a0 0x7facf419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf909e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7facf410c290 con 0x7facf41007f0 2024-05-09T08:32:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf909e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7facf410c110 con 0x7facf41023a0 2024-05-09T08:32:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf909e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7facf410c410 con 0x7facf41019e0 2024-05-09T08:32:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf41007f0 0x7facf41979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf41007f0 0x7facf41979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55620/0 (socket says 172.21.15.107:55620) 2024-05-09T08:32:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf2d9d640 1 -- 172.21.15.107:0/1562258897 learned_addr learned my addr 172.21.15.107:0/1562258897 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf359e640 1 --2- 172.21.15.107:0/1562258897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facf41023a0 0x7facf419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf259c640 1 --2- 172.21.15.107:0/1562258897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf41019e0 0x7facf4197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf2d9d640 1 -- 172.21.15.107:0/1562258897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf41019e0 msgr2=0x7facf4197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf2d9d640 1 --2- 172.21.15.107:0/1562258897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf41019e0 0x7facf4197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf2d9d640 1 -- 172.21.15.107:0/1562258897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facf41023a0 msgr2=0x7facf419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf2d9d640 1 --2- 172.21.15.107:0/1562258897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facf41023a0 0x7facf419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf2d9d640 1 -- 172.21.15.107:0/1562258897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf419f610 con 0x7facf41007f0 2024-05-09T08:32:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.916+0000 7facf2d9d640 1 --2- 172.21.15.107:0/1562258897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf41007f0 0x7facf41979f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7face400c970 tx=0x7face400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.917+0000 7facdbfff640 1 -- 172.21.15.107:0/1562258897 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face4004220 con 0x7facf41007f0 2024-05-09T08:32:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.917+0000 7facf909e640 1 -- 172.21.15.107:0/1562258897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf419f900 con 0x7facf41007f0 2024-05-09T08:32:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.917+0000 7facdbfff640 1 -- 172.21.15.107:0/1562258897 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7face40043c0 con 0x7facf41007f0 2024-05-09T08:32:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.917+0000 7facdbfff640 1 -- 172.21.15.107:0/1562258897 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face400f450 con 0x7facf41007f0 2024-05-09T08:32:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.917+0000 7facf909e640 1 -- 172.21.15.107:0/1562258897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf419fea0 con 0x7facf41007f0 2024-05-09T08:32:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.918+0000 7facf909e640 1 -- 172.21.15.107:0/1562258897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf4068aa0 con 0x7facf41007f0 2024-05-09T08:32:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.919+0000 7facdbfff640 1 -- 172.21.15.107:0/1562258897 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7face40079b0 con 0x7facf41007f0 2024-05-09T08:32:45.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.919+0000 7facdbfff640 1 --2- 172.21.15.107:0/1562258897 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7facdc0a2b90 0x7facdc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:45.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.919+0000 7facdbfff640 1 -- 172.21.15.107:0/1562258897 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7face40cf160 con 0x7facf41007f0 2024-05-09T08:32:45.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.919+0000 7facf259c640 1 --2- 172.21.15.107:0/1562258897 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7facdc0a2b90 0x7facdc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:45.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.920+0000 7facf259c640 1 --2- 172.21.15.107:0/1562258897 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7facdc0a2b90 0x7facdc0a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7facf4198f00 tx=0x7face8005cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:45.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:45.924+0000 7facdbfff640 1 -- 172.21.15.107:0/1562258897 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7face4014030 con 0x7facf41007f0 2024-05-09T08:32:46.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.138+0000 7facf909e640 1 -- 172.21.15.107:0/1562258897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7facf41a0130 con 0x7facf41007f0 2024-05-09T08:32:46.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.139+0000 7facdbfff640 1 -- 172.21.15.107:0/1562258897 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11887 (secure 0 0 0) 0x7face409bd60 con 0x7facf41007f0 2024-05-09T08:32:46.139 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:32:46.139 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:31:21.056050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24478,24499]},"id":1}]} 2024-05-09T08:32:46.139 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 24 2024-05-09T08:32:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.141+0000 7facf909e640 1 -- 172.21.15.107:0/1562258897 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7facdc0a2b90 msgr2=0x7facdc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.141+0000 7facf909e640 1 --2- 172.21.15.107:0/1562258897 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7facdc0a2b90 0x7facdc0a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7facf4198f00 tx=0x7face8005cc0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.141+0000 7facf909e640 1 -- 172.21.15.107:0/1562258897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf41007f0 msgr2=0x7facf41979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.141+0000 7facf909e640 1 --2- 172.21.15.107:0/1562258897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf41007f0 0x7facf41979f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7face400c970 tx=0x7face400ce30 comp rx=0 tx=0).stop 2024-05-09T08:32:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.141+0000 7facf909e640 1 -- 172.21.15.107:0/1562258897 shutdown_connections 2024-05-09T08:32:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.141+0000 7facf909e640 1 --2- 172.21.15.107:0/1562258897 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7facdc0a2b90 0x7facdc0a5050 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.141+0000 7facf909e640 1 --2- 172.21.15.107:0/1562258897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facf41023a0 0x7facf419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.141+0000 7facf909e640 1 --2- 172.21.15.107:0/1562258897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf41019e0 0x7facf4197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.141+0000 7facf909e640 1 --2- 172.21.15.107:0/1562258897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf41007f0 0x7facf41979f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.141+0000 7facf909e640 1 -- 172.21.15.107:0/1562258897 >> 172.21.15.107:0/1562258897 conn(0x7facf40fbd80 msgr2=0x7facf40fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.141+0000 7facf909e640 1 -- 172.21.15.107:0/1562258897 shutdown_connections 2024-05-09T08:32:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.141+0000 7facf909e640 1 -- 172.21.15.107:0/1562258897 wait complete. 2024-05-09T08:32:46.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:32:46.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.256+0000 7fdc53876640 1 -- 172.21.15.107:0/3920017360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc4c104ca0 msgr2=0x7fdc4c107090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.256+0000 7fdc53876640 1 --2- 172.21.15.107:0/3920017360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc4c104ca0 0x7fdc4c107090 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009900 tx=0x7fdc3c02a410 comp rx=0 tx=0).stop 2024-05-09T08:32:46.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.256+0000 7fdc53876640 1 -- 172.21.15.107:0/3920017360 shutdown_connections 2024-05-09T08:32:46.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.256+0000 7fdc53876640 1 --2- 172.21.15.107:0/3920017360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc4c1075d0 0x7fdc4c1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.256+0000 7fdc53876640 1 --2- 172.21.15.107:0/3920017360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc4c104ca0 0x7fdc4c107090 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.256+0000 7fdc53876640 1 --2- 172.21.15.107:0/3920017360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc4c102370 0x7fdc4c104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.256+0000 7fdc53876640 1 -- 172.21.15.107:0/3920017360 >> 172.21.15.107:0/3920017360 conn(0x7fdc4c0fbce0 msgr2=0x7fdc4c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:46.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.256+0000 7fdc53876640 1 -- 172.21.15.107:0/3920017360 shutdown_connections 2024-05-09T08:32:46.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.256+0000 7fdc53876640 1 -- 172.21.15.107:0/3920017360 wait complete. 2024-05-09T08:32:46.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.256+0000 7fdc53876640 1 Processor -- start 2024-05-09T08:32:46.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.256+0000 7fdc53876640 1 -- start start 2024-05-09T08:32:46.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc53876640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc4c102370 0x7fdc4c197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc53876640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc4c104ca0 0x7fdc4c1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc53876640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc4c1075d0 0x7fdc4c19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc53876640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4c10d3f0 con 0x7fdc4c1075d0 2024-05-09T08:32:46.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc53876640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4c10d270 con 0x7fdc4c102370 2024-05-09T08:32:46.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc53876640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4c10d570 con 0x7fdc4c104ca0 2024-05-09T08:32:46.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc51e13640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc4c1075d0 0x7fdc4c19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc51e13640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc4c1075d0 0x7fdc4c19f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55634/0 (socket says 172.21.15.107:55634) 2024-05-09T08:32:46.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc51e13640 1 -- 172.21.15.107:0/1625366672 learned_addr learned my addr 172.21.15.107:0/1625366672 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:46.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc51612640 1 --2- 172.21.15.107:0/1625366672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc4c102370 0x7fdc4c197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc50e11640 1 --2- 172.21.15.107:0/1625366672 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc4c104ca0 0x7fdc4c1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc51e13640 1 -- 172.21.15.107:0/1625366672 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc4c104ca0 msgr2=0x7fdc4c1980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc51e13640 1 --2- 172.21.15.107:0/1625366672 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc4c104ca0 0x7fdc4c1980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc51e13640 1 -- 172.21.15.107:0/1625366672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc4c102370 msgr2=0x7fdc4c197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc51e13640 1 --2- 172.21.15.107:0/1625366672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc4c102370 0x7fdc4c197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc51e13640 1 -- 172.21.15.107:0/1625366672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc4c19f760 con 0x7fdc4c1075d0 2024-05-09T08:32:46.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc51e13640 1 --2- 172.21.15.107:0/1625366672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc4c1075d0 0x7fdc4c19f0f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fdc4400b970 tx=0x7fdc4400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:46.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc427fc640 1 -- 172.21.15.107:0/1625366672 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc44019070 con 0x7fdc4c1075d0 2024-05-09T08:32:46.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc427fc640 1 -- 172.21.15.107:0/1625366672 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc44014440 con 0x7fdc4c1075d0 2024-05-09T08:32:46.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc53876640 1 -- 172.21.15.107:0/1625366672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc4c19fa50 con 0x7fdc4c1075d0 2024-05-09T08:32:46.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.257+0000 7fdc427fc640 1 -- 172.21.15.107:0/1625366672 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc440075d0 con 0x7fdc4c1075d0 2024-05-09T08:32:46.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.258+0000 7fdc53876640 1 -- 172.21.15.107:0/1625366672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc4c19ffd0 con 0x7fdc4c1075d0 2024-05-09T08:32:46.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.258+0000 7fdc53876640 1 -- 172.21.15.107:0/1625366672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc4c0fd7d0 con 0x7fdc4c1075d0 2024-05-09T08:32:46.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.260+0000 7fdc427fc640 1 -- 172.21.15.107:0/1625366672 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fdc4401c070 con 0x7fdc4c1075d0 2024-05-09T08:32:46.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.260+0000 7fdc427fc640 1 --2- 172.21.15.107:0/1625366672 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fdc380a2b90 0x7fdc380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.260+0000 7fdc427fc640 1 -- 172.21.15.107:0/1625366672 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdc440ceec0 con 0x7fdc4c1075d0 2024-05-09T08:32:46.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.260+0000 7fdc51612640 1 --2- 172.21.15.107:0/1625366672 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fdc380a2b90 0x7fdc380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.261+0000 7fdc51612640 1 --2- 172.21.15.107:0/1625366672 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fdc380a2b90 0x7fdc380a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdc48005fd0 tx=0x7fdc48005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:46.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.264+0000 7fdc427fc640 1 -- 172.21.15.107:0/1625366672 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdc4409bac0 con 0x7fdc4c1075d0 2024-05-09T08:32:46.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.449+0000 7fdc53876640 1 -- 172.21.15.107:0/1625366672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdc4c062430 con 0x7fdc4c1075d0 2024-05-09T08:32:46.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.451+0000 7fdc427fc640 1 -- 172.21.15.107:0/1625366672 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24658 (secure 0 0 0) 0x7fdc4409b3c0 con 0x7fdc4c1075d0 2024-05-09T08:32:46.451 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:32:46.451 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":91,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:31:56.556402+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:32:46.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.453+0000 7fdc53876640 1 -- 172.21.15.107:0/1625366672 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fdc380a2b90 msgr2=0x7fdc380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.453+0000 7fdc53876640 1 --2- 172.21.15.107:0/1625366672 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fdc380a2b90 0x7fdc380a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdc48005fd0 tx=0x7fdc48005850 comp rx=0 tx=0).stop 2024-05-09T08:32:46.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.453+0000 7fdc53876640 1 -- 172.21.15.107:0/1625366672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc4c1075d0 msgr2=0x7fdc4c19f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.453+0000 7fdc53876640 1 --2- 172.21.15.107:0/1625366672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc4c1075d0 0x7fdc4c19f0f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fdc4400b970 tx=0x7fdc4400be30 comp rx=0 tx=0).stop 2024-05-09T08:32:46.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.453+0000 7fdc53876640 1 -- 172.21.15.107:0/1625366672 shutdown_connections 2024-05-09T08:32:46.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.453+0000 7fdc53876640 1 --2- 172.21.15.107:0/1625366672 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fdc380a2b90 0x7fdc380a5050 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.453+0000 7fdc53876640 1 --2- 172.21.15.107:0/1625366672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc4c1075d0 0x7fdc4c19f0f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.453+0000 7fdc53876640 1 --2- 172.21.15.107:0/1625366672 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc4c104ca0 0x7fdc4c1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.453+0000 7fdc53876640 1 --2- 172.21.15.107:0/1625366672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc4c102370 0x7fdc4c197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.453+0000 7fdc53876640 1 -- 172.21.15.107:0/1625366672 >> 172.21.15.107:0/1625366672 conn(0x7fdc4c0fbce0 msgr2=0x7fdc4c102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.453+0000 7fdc53876640 1 -- 172.21.15.107:0/1625366672 shutdown_connections 2024-05-09T08:32:46.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.453+0000 7fdc53876640 1 -- 172.21.15.107:0/1625366672 wait complete. 2024-05-09T08:32:46.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:32:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.571+0000 7f04a74e1640 1 -- 172.21.15.107:0/3543353088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a0075810 msgr2=0x7f04a0073c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.571+0000 7f04a74e1640 1 --2- 172.21.15.107:0/3543353088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a0075810 0x7f04a0073c60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f049c009900 tx=0x7f049c02a480 comp rx=0 tx=0).stop 2024-05-09T08:32:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.571+0000 7f04a74e1640 1 -- 172.21.15.107:0/3543353088 shutdown_connections 2024-05-09T08:32:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.571+0000 7f04a74e1640 1 --2- 172.21.15.107:0/3543353088 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04a0109720 0x7f04a010bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.571+0000 7f04a74e1640 1 --2- 172.21.15.107:0/3543353088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04a0074230 0x7f04a0074690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.571+0000 7f04a74e1640 1 --2- 172.21.15.107:0/3543353088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a0075810 0x7f04a0073c60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.571+0000 7f04a74e1640 1 -- 172.21.15.107:0/3543353088 >> 172.21.15.107:0/3543353088 conn(0x7f04a00fcde0 msgr2=0x7f04a00ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.571+0000 7f04a74e1640 1 -- 172.21.15.107:0/3543353088 shutdown_connections 2024-05-09T08:32:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.571+0000 7f04a74e1640 1 -- 172.21.15.107:0/3543353088 wait complete. 2024-05-09T08:32:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.572+0000 7f04a74e1640 1 Processor -- start 2024-05-09T08:32:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.572+0000 7f04a74e1640 1 -- start start 2024-05-09T08:32:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.572+0000 7f04a74e1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04a0074230 0x7f04a0199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.572+0000 7f04a74e1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04a0075810 0x7f04a019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.572+0000 7f04a74e1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a0109720 0x7f04a01a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.572+0000 7f04a74e1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f04a010f5b0 con 0x7f04a0109720 2024-05-09T08:32:46.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.572+0000 7f04a74e1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f04a010f430 con 0x7f04a0074230 2024-05-09T08:32:46.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.572+0000 7f04a74e1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f04a010f730 con 0x7f04a0075810 2024-05-09T08:32:46.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.572+0000 7f04a5a7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a0109720 0x7f04a01a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.572+0000 7f04a5a7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a0109720 0x7f04a01a1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55662/0 (socket says 172.21.15.107:55662) 2024-05-09T08:32:46.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.572+0000 7f04a5a7e640 1 -- 172.21.15.107:0/3525614236 learned_addr learned my addr 172.21.15.107:0/3525614236 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:46.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.572+0000 7f04a4a7c640 1 --2- 172.21.15.107:0/3525614236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04a0075810 0x7f04a019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.572+0000 7f04a527d640 1 --2- 172.21.15.107:0/3525614236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04a0074230 0x7f04a0199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.572+0000 7f04a5a7e640 1 -- 172.21.15.107:0/3525614236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04a0075810 msgr2=0x7f04a019a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.573+0000 7f04a5a7e640 1 --2- 172.21.15.107:0/3525614236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04a0075810 0x7f04a019a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.573+0000 7f04a5a7e640 1 -- 172.21.15.107:0/3525614236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04a0074230 msgr2=0x7f04a0199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.573+0000 7f04a5a7e640 1 --2- 172.21.15.107:0/3525614236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04a0074230 0x7f04a0199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.573+0000 7f04a5a7e640 1 -- 172.21.15.107:0/3525614236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a01a1900 con 0x7f04a0109720 2024-05-09T08:32:46.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.573+0000 7f04a4a7c640 1 --2- 172.21.15.107:0/3525614236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04a0075810 0x7f04a019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:32:46.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.573+0000 7f04a527d640 1 --2- 172.21.15.107:0/3525614236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04a0074230 0x7f04a0199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:32:46.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.573+0000 7f04a5a7e640 1 --2- 172.21.15.107:0/3525614236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a0109720 0x7f04a01a1260 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f049800cad0 tx=0x7f049800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:46.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.573+0000 7f04967fc640 1 -- 172.21.15.107:0/3525614236 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049800ae20 con 0x7f04a0109720 2024-05-09T08:32:46.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.573+0000 7f04967fc640 1 -- 172.21.15.107:0/3525614236 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0498005020 con 0x7f04a0109720 2024-05-09T08:32:46.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.573+0000 7f04a74e1640 1 -- 172.21.15.107:0/3525614236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a01a1bf0 con 0x7f04a0109720 2024-05-09T08:32:46.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.573+0000 7f04967fc640 1 -- 172.21.15.107:0/3525614236 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049800f6b0 con 0x7f04a0109720 2024-05-09T08:32:46.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.573+0000 7f04a74e1640 1 -- 172.21.15.107:0/3525614236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a01a20f0 con 0x7f04a0109720 2024-05-09T08:32:46.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.574+0000 7f04a74e1640 1 -- 172.21.15.107:0/3525614236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04a01084e0 con 0x7f04a0109720 2024-05-09T08:32:46.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.575+0000 7f04967fc640 1 -- 172.21.15.107:0/3525614236 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f04980051d0 con 0x7f04a0109720 2024-05-09T08:32:46.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.576+0000 7f04967fc640 1 --2- 172.21.15.107:0/3525614236 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f048c0a2b90 0x7f048c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.576+0000 7f04967fc640 1 -- 172.21.15.107:0/3525614236 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f04980cf580 con 0x7f04a0109720 2024-05-09T08:32:46.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.576+0000 7f04a527d640 1 --2- 172.21.15.107:0/3525614236 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f048c0a2b90 0x7f048c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.576+0000 7f04a527d640 1 --2- 172.21.15.107:0/3525614236 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f048c0a2b90 0x7f048c0a5050 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f049c000c00 tx=0x7f049c009610 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:46.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.580+0000 7f04967fc640 1 -- 172.21.15.107:0/3525614236 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0498014030 con 0x7f04a0109720 2024-05-09T08:32:46.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:46 smithi107 ceph-mon[30122]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-09T08:32:46.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:46 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3653782218' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:32:46.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:46 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1562258897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:32:46.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:46 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1625366672' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:32:46.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.801+0000 7f04a74e1640 1 -- 172.21.15.107:0/3525614236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04a002d1c0 con 0x7f04a0109720 2024-05-09T08:32:46.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.802+0000 7f04967fc640 1 -- 172.21.15.107:0/3525614236 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11887 (secure 0 0 0) 0x7f049809c180 con 0x7f04a0109720 2024-05-09T08:32:46.802 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:32:46.802 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:31:21.056050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24478,24499]},"id":1}]} 2024-05-09T08:32:46.802 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 24 2024-05-09T08:32:46.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.804+0000 7f04a74e1640 1 -- 172.21.15.107:0/3525614236 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f048c0a2b90 msgr2=0x7f048c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.804+0000 7f04a74e1640 1 --2- 172.21.15.107:0/3525614236 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f048c0a2b90 0x7f048c0a5050 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f049c000c00 tx=0x7f049c009610 comp rx=0 tx=0).stop 2024-05-09T08:32:46.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.804+0000 7f04a74e1640 1 -- 172.21.15.107:0/3525614236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a0109720 msgr2=0x7f04a01a1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.804+0000 7f04a74e1640 1 --2- 172.21.15.107:0/3525614236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a0109720 0x7f04a01a1260 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f049800cad0 tx=0x7f049800cf90 comp rx=0 tx=0).stop 2024-05-09T08:32:46.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.804+0000 7f04a74e1640 1 -- 172.21.15.107:0/3525614236 shutdown_connections 2024-05-09T08:32:46.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.804+0000 7f04a74e1640 1 --2- 172.21.15.107:0/3525614236 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f048c0a2b90 0x7f048c0a5050 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.804+0000 7f04a74e1640 1 --2- 172.21.15.107:0/3525614236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a0109720 0x7f04a01a1260 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.804+0000 7f04a74e1640 1 --2- 172.21.15.107:0/3525614236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04a0075810 0x7f04a019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.804+0000 7f04a74e1640 1 --2- 172.21.15.107:0/3525614236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04a0074230 0x7f04a0199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.804+0000 7f04a74e1640 1 -- 172.21.15.107:0/3525614236 >> 172.21.15.107:0/3525614236 conn(0x7f04a00fcde0 msgr2=0x7f04a01056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:46.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.804+0000 7f04a74e1640 1 -- 172.21.15.107:0/3525614236 shutdown_connections 2024-05-09T08:32:46.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.804+0000 7f04a74e1640 1 -- 172.21.15.107:0/3525614236 wait complete. 2024-05-09T08:32:46.816 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-09T08:31:00.157448+0000', 'modified': '2024-05-09T08:31:21.056050+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24499, 'mds_1': 24491, 'mds_2': 24478}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24431': {'gid': 24431, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.110:6835/2138165106', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6834', 'nonce': 2138165106}, {'type': 'v1', 'addr': '172.21.15.110:6835', 'nonce': 2138165106}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24458': {'gid': 24458, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.110:6837/624105719', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6836', 'nonce': 624105719}, {'type': 'v1', 'addr': '172.21.15.110:6837', 'nonce': 624105719}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.157:6835/843217781', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 843217781}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 843217781}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.157:6837/1973902800', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6836', 'nonce': 1973902800}, {'type': 'v1', 'addr': '172.21.15.157:6837', 'nonce': 1973902800}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24491': {'gid': 24491, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.110:6839/593143450', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6838', 'nonce': 593143450}, {'type': 'v1', 'addr': '172.21.15.110:6839', 'nonce': 593143450}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.157:6839/3003356919', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3003356919}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3003356919}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24499, 'qdb_cluster': [24491, 24478, 24499]} 2024-05-09T08:32:46.816 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-09T08:32:46.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T08:32:46.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:46 smithi157 ceph-mon[33614]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-09T08:32:46.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:46 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3653782218' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:32:46.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:46 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1562258897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:32:46.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:46 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1625366672' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:32:46.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:46 smithi110 ceph-mon[33622]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-09T08:32:46.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:46 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3653782218' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:32:46.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:46 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1562258897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:32:46.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:46 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1625366672' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:32:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 -- 172.21.15.107:0/2689413118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae68101960 msgr2=0x7fae68101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 --2- 172.21.15.107:0/2689413118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae68101960 0x7fae68101de0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fae64009900 tx=0x7fae6402a410 comp rx=0 tx=0).stop 2024-05-09T08:32:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 -- 172.21.15.107:0/2689413118 shutdown_connections 2024-05-09T08:32:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 --2- 172.21.15.107:0/2689413118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae68102320 0x7fae68108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 --2- 172.21.15.107:0/2689413118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae68101960 0x7fae68101de0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 --2- 172.21.15.107:0/2689413118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae68100770 0x7fae68100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 -- 172.21.15.107:0/2689413118 >> 172.21.15.107:0/2689413118 conn(0x7fae680fbce0 msgr2=0x7fae680fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 -- 172.21.15.107:0/2689413118 shutdown_connections 2024-05-09T08:32:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 -- 172.21.15.107:0/2689413118 wait complete. 2024-05-09T08:32:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 Processor -- start 2024-05-09T08:32:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 -- start start 2024-05-09T08:32:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae68100770 0x7fae68197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae68101960 0x7fae68197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae68102320 0x7fae6819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fae6810d3c0 con 0x7fae68101960 2024-05-09T08:32:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fae6810d240 con 0x7fae68100770 2024-05-09T08:32:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.922+0000 7fae70134640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fae6810d540 con 0x7fae68102320 2024-05-09T08:32:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae6d6cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae68101960 0x7fae68197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae6d6cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae68101960 0x7fae68197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55684/0 (socket says 172.21.15.107:55684) 2024-05-09T08:32:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae6d6cf640 1 -- 172.21.15.107:0/1899288781 learned_addr learned my addr 172.21.15.107:0/1899288781 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae6e6d1640 1 --2- 172.21.15.107:0/1899288781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae68102320 0x7fae6819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae6ded0640 1 --2- 172.21.15.107:0/1899288781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae68100770 0x7fae68197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae6d6cf640 1 -- 172.21.15.107:0/1899288781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae68102320 msgr2=0x7fae6819f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae6d6cf640 1 --2- 172.21.15.107:0/1899288781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae68102320 0x7fae6819f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae6d6cf640 1 -- 172.21.15.107:0/1899288781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae68100770 msgr2=0x7fae68197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae6d6cf640 1 --2- 172.21.15.107:0/1899288781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae68100770 0x7fae68197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae6d6cf640 1 -- 172.21.15.107:0/1899288781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae6819f680 con 0x7fae68101960 2024-05-09T08:32:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae6ded0640 1 --2- 172.21.15.107:0/1899288781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae68100770 0x7fae68197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:32:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae6d6cf640 1 --2- 172.21.15.107:0/1899288781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae68101960 0x7fae68197fa0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fae64005940 tx=0x7fae64004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae5affd640 1 -- 172.21.15.107:0/1899288781 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6403f030 con 0x7fae68101960 2024-05-09T08:32:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae5affd640 1 -- 172.21.15.107:0/1899288781 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae64003cb0 con 0x7fae68101960 2024-05-09T08:32:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae70134640 1 -- 172.21.15.107:0/1899288781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae6819f910 con 0x7fae68101960 2024-05-09T08:32:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae5affd640 1 -- 172.21.15.107:0/1899288781 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae64035070 con 0x7fae68101960 2024-05-09T08:32:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.923+0000 7fae70134640 1 -- 172.21.15.107:0/1899288781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae6819fe50 con 0x7fae68101960 2024-05-09T08:32:46.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.925+0000 7fae5affd640 1 -- 172.21.15.107:0/1899288781 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fae64042030 con 0x7fae68101960 2024-05-09T08:32:46.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.926+0000 7fae5affd640 1 --2- 172.21.15.107:0/1899288781 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fae540a2c10 0x7fae540a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.926+0000 7fae6ded0640 1 --2- 172.21.15.107:0/1899288781 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fae540a2c10 0x7fae540a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.926+0000 7fae5affd640 1 -- 172.21.15.107:0/1899288781 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fae640efe90 con 0x7fae68101960 2024-05-09T08:32:46.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.926+0000 7fae6ded0640 1 --2- 172.21.15.107:0/1899288781 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fae540a2c10 0x7fae540a50d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fae681017c0 tx=0x7fae5c005db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:46.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae72e43740 1 Processor -- start 2024-05-09T08:32:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae72e43740 1 -- start start 2024-05-09T08:32:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae72e43740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c1058c0 0x55593c105ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae72e43740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1062b0 0x55593c10adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae72e43740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c10b420 0x55593c10d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae72e43740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55593c0694d0 con 0x55593c1058c0 2024-05-09T08:32:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae72e43740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55593c0113e0 con 0x55593c1062b0 2024-05-09T08:32:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae72e43740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55593c065490 con 0x55593c10b420 2024-05-09T08:32:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae6ded0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c1058c0 0x55593c105ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae6ded0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c1058c0 0x55593c105ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55692/0 (socket says 172.21.15.107:55692) 2024-05-09T08:32:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae6ded0640 1 -- 172.21.15.107:0/2533913843 learned_addr learned my addr 172.21.15.107:0/2533913843 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae6d6cf640 1 --2- 172.21.15.107:0/2533913843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1062b0 0x55593c10adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae6e6d1640 1 --2- 172.21.15.107:0/2533913843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c10b420 0x55593c10d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae6ded0640 1 -- 172.21.15.107:0/2533913843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c10b420 msgr2=0x55593c10d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae6ded0640 1 --2- 172.21.15.107:0/2533913843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c10b420 0x55593c10d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae6ded0640 1 -- 172.21.15.107:0/2533913843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1062b0 msgr2=0x55593c10adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae6ded0640 1 --2- 172.21.15.107:0/2533913843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1062b0 0x55593c10adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.927+0000 7fae6ded0640 1 -- 172.21.15.107:0/2533913843 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55593bd7d2d0 con 0x55593c1058c0 2024-05-09T08:32:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.928+0000 7fae6ded0640 1 --2- 172.21.15.107:0/2533913843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c1058c0 0x55593c105ce0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00ddc0 tx=0x7fae5c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702fb5ca57d2e75b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.928+0000 7fae53fff640 1 -- 172.21.15.107:0/2533913843 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c020070 con 0x55593c1058c0 2024-05-09T08:32:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.928+0000 7fae53fff640 1 -- 172.21.15.107:0/2533913843 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae5c00fe30 con 0x55593c1058c0 2024-05-09T08:32:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.928+0000 7fae53fff640 1 -- 172.21.15.107:0/2533913843 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c01bb60 con 0x55593c1058c0 2024-05-09T08:32:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.929+0000 7fae72e43740 1 -- 172.21.15.107:0/2533913843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c1058c0 msgr2=0x55593c105ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.929+0000 7fae72e43740 1 --2- 172.21.15.107:0/2533913843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c1058c0 0x55593c105ce0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00ddc0 tx=0x7fae5c010000 comp rx=0 tx=0).stop 2024-05-09T08:32:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.929+0000 7fae72e43740 1 -- 172.21.15.107:0/2533913843 shutdown_connections 2024-05-09T08:32:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.929+0000 7fae72e43740 1 --2- 172.21.15.107:0/2533913843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c10b420 0x55593c10d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.929+0000 7fae72e43740 1 --2- 172.21.15.107:0/2533913843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1062b0 0x55593c10adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.929+0000 7fae72e43740 1 --2- 172.21.15.107:0/2533913843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c1058c0 0x55593c105ce0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.929+0000 7fae72e43740 1 -- 172.21.15.107:0/2533913843 >> 172.21.15.107:0/2533913843 conn(0x55593c100f30 msgr2=0x55593c101330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.929+0000 7fae72e43740 1 -- 172.21.15.107:0/2533913843 shutdown_connections 2024-05-09T08:32:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.929+0000 7fae72e43740 1 -- 172.21.15.107:0/2533913843 wait complete. 2024-05-09T08:32:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.929+0000 7fae72e43740 1 -- start start 2024-05-09T08:32:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.929+0000 7fae72e43740 1 Processor -- start 2024-05-09T08:32:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae72e43740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1058c0 0x55593c10fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae72e43740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c1062b0 0x55593c1101f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae72e43740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c117220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae72e43740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55593c10e3d0 con 0x55593c10b420 2024-05-09T08:32:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae72e43740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55593c0248e0 con 0x55593c1058c0 2024-05-09T08:32:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae72e43740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55593c10e550 con 0x55593c1062b0 2024-05-09T08:32:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae72e43740 10 client.0 _ll_register_callbacks cb 0x55593bff19c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:32:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae6e6d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c117220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae6e6d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c117220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55694/0 (socket says 172.21.15.107:55694) 2024-05-09T08:32:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae6e6d1640 1 -- 172.21.15.107:0/2525430745 learned_addr learned my addr 172.21.15.107:0/2525430745 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae6d6cf640 1 --2- 172.21.15.107:0/2525430745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c1062b0 0x55593c1101f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae6ded0640 1 --2- 172.21.15.107:0/2525430745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1058c0 0x55593c10fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae6e6d1640 1 -- 172.21.15.107:0/2525430745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c1062b0 msgr2=0x55593c1101f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae6e6d1640 1 --2- 172.21.15.107:0/2525430745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c1062b0 0x55593c1101f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae6e6d1640 1 -- 172.21.15.107:0/2525430745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1058c0 msgr2=0x55593c10fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae6e6d1640 1 --2- 172.21.15.107:0/2525430745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1058c0 0x55593c10fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae6e6d1640 1 -- 172.21.15.107:0/2525430745 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55593be476b0 con 0x55593c10b420 2024-05-09T08:32:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae6e6d1640 1 --2- 172.21.15.107:0/2525430745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c117220 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fae60010a40 tx=0x7fae60010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e778423c72abda8e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae517fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:32:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae517fa640 1 -- 172.21.15.107:0/2525430745 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6000ee00 con 0x55593c10b420 2024-05-09T08:32:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.930+0000 7fae517fa640 1 -- 172.21.15.107:0/2525430745 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fae60004b20 con 0x55593c10b420 2024-05-09T08:32:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.931+0000 7fae72e43740 1 -- 172.21.15.107:0/2525430745 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae5c00fe30 con 0x55593c10b420 2024-05-09T08:32:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.931+0000 7fae517fa640 1 -- 172.21.15.107:0/2525430745 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fae60024050 con 0x55593c10b420 2024-05-09T08:32:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.931+0000 7fae72e43740 10 client.15243 fetch_fsmap learned FSMap version 24 2024-05-09T08:32:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.931+0000 7fae72e43740 1 -- 172.21.15.107:0/2525430745 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55593c11a2d0 con 0x55593c10b420 2024-05-09T08:32:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.931+0000 7fae517fa640 1 -- 172.21.15.107:0/2525430745 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae5c00fe30 con 0x55593c10b420 2024-05-09T08:32:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.931+0000 7fae517fa640 1 -- 172.21.15.107:0/2525430745 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fae60011440 con 0x55593c10b420 2024-05-09T08:32:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.931+0000 7fae72e43740 10 client.15243 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T08:32:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.931+0000 7fae72e43740 10 client.15243 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T08:32:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.931+0000 7fae72e43740 20 client.15243 populate_metadata read hostname 'smithi107' 2024-05-09T08:32:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.931+0000 7fae72e43740 1 --2- 172.21.15.107:0/2525430745 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55593c11c960 0x55593c11ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.931+0000 7fae72e43740 4 client.15243 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:32:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.931+0000 7fae72e43740 1 -- 172.21.15.107:0/2525430745 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55593c076c00 con 0x55593c11c960 2024-05-09T08:32:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.931+0000 7fae6ded0640 1 --2- 172.21.15.107:0/2525430745 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55593c11c960 0x55593c11ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.932+0000 7fae517fa640 10 client.15243 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:32:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.932+0000 7fae6ded0640 1 --2- 172.21.15.107:0/2525430745 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55593c11c960 0x55593c11ddd0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4d750be7c8ee3d77 server_cookie=ed92057a65070abc in_seq=0 out_seq=0 2024-05-09T08:32:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.933+0000 7fae517fa640 1 -- 172.21.15.107:0/2525430745 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55593c076c00 con 0x55593c11c960 2024-05-09T08:32:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.933+0000 7fae517fa640 10 client.15243 handle_command_reply: tid=0 2024-05-09T08:32:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.933+0000 7fae72e43740 1 client.15243 shutdown 2024-05-09T08:32:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.933+0000 7fae72e43740 1 -- 172.21.15.107:0/2525430745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 msgr2=0x55593c117220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.933+0000 7fae72e43740 1 --2- 172.21.15.107:0/2525430745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c117220 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fae60010a40 tx=0x7fae60010f00 comp rx=0 tx=0).stop 2024-05-09T08:32:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.933+0000 7fae72e43740 1 -- 172.21.15.107:0/2525430745 shutdown_connections 2024-05-09T08:32:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.933+0000 7fae72e43740 1 --2- 172.21.15.107:0/2525430745 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55593c11c960 0x55593c11ddd0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.933+0000 7fae72e43740 1 --2- 172.21.15.107:0/2525430745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c117220 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.933+0000 7fae517fa640 0 client.15243 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:32:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.933+0000 7fae72e43740 1 --2- 172.21.15.107:0/2525430745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c1062b0 0x55593c1101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.933+0000 7fae72e43740 1 --2- 172.21.15.107:0/2525430745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1058c0 0x55593c10fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.933+0000 7fae72e43740 1 -- 172.21.15.107:0/2525430745 >> 172.21.15.107:0/2525430745 conn(0x55593c100f30 msgr2=0x55593c100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.933+0000 7fae72e43740 1 -- 172.21.15.107:0/2525430745 shutdown_connections 2024-05-09T08:32:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.933+0000 7fae72e43740 1 -- 172.21.15.107:0/2525430745 wait complete. 2024-05-09T08:32:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.933+0000 7fae72e43740 20 client.15243 trim_cache size 0 max 16384 2024-05-09T08:32:46.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae72e43740 1 Processor -- start 2024-05-09T08:32:46.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae72e43740 1 -- start start 2024-05-09T08:32:46.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae72e43740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c1058c0 0x55593c10ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae72e43740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1062b0 0x55593c10e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae72e43740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c10ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae72e43740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55593c11c110 con 0x55593c10b420 2024-05-09T08:32:46.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae72e43740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55593c11c7e0 con 0x55593c1062b0 2024-05-09T08:32:46.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae72e43740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55593c11ceb0 con 0x55593c1058c0 2024-05-09T08:32:46.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae6e6d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c10ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae6e6d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c10ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55698/0 (socket says 172.21.15.107:55698) 2024-05-09T08:32:46.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae6e6d1640 1 -- 172.21.15.107:0/2627093776 learned_addr learned my addr 172.21.15.107:0/2627093776 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:46.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae6ded0640 1 --2- 172.21.15.107:0/2627093776 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c1058c0 0x55593c10ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae6d6cf640 1 --2- 172.21.15.107:0/2627093776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1062b0 0x55593c10e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae6e6d1640 1 -- 172.21.15.107:0/2627093776 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c1058c0 msgr2=0x55593c10ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:46.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae6e6d1640 1 --2- 172.21.15.107:0/2627093776 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c1058c0 0x55593c10ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae6e6d1640 1 -- 172.21.15.107:0/2627093776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1062b0 msgr2=0x55593c10e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:46.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae6e6d1640 1 --2- 172.21.15.107:0/2627093776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1062b0 0x55593c10e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae6e6d1640 1 -- 172.21.15.107:0/2627093776 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55593c106690 con 0x55593c10b420 2024-05-09T08:32:46.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.951+0000 7fae6d6cf640 1 --2- 172.21.15.107:0/2627093776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1062b0 0x55593c10e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:32:46.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.952+0000 7fae6e6d1640 1 --2- 172.21.15.107:0/2627093776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c10ee30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fae60000f80 tx=0x7fae6000bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d30b3465bfde45 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:46.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.952+0000 7fae537fe640 1 -- 172.21.15.107:0/2627093776 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6002c070 con 0x55593c10b420 2024-05-09T08:32:46.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.952+0000 7fae537fe640 1 -- 172.21.15.107:0/2627093776 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae6000e720 con 0x55593c10b420 2024-05-09T08:32:46.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.952+0000 7fae537fe640 1 -- 172.21.15.107:0/2627093776 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae60005030 con 0x55593c10b420 2024-05-09T08:32:46.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.952+0000 7fae72e43740 1 -- 172.21.15.107:0/2627093776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 msgr2=0x55593c10ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.952+0000 7fae72e43740 1 --2- 172.21.15.107:0/2627093776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c10ee30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fae60000f80 tx=0x7fae6000bf60 comp rx=0 tx=0).stop 2024-05-09T08:32:46.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.952+0000 7fae72e43740 1 -- 172.21.15.107:0/2627093776 shutdown_connections 2024-05-09T08:32:46.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.952+0000 7fae72e43740 1 --2- 172.21.15.107:0/2627093776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c10ee30 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.952+0000 7fae72e43740 1 --2- 172.21.15.107:0/2627093776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1062b0 0x55593c10e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.952+0000 7fae72e43740 1 --2- 172.21.15.107:0/2627093776 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c1058c0 0x55593c10ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.952+0000 7fae72e43740 1 -- 172.21.15.107:0/2627093776 >> 172.21.15.107:0/2627093776 conn(0x55593c100f30 msgr2=0x55593c10f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:46.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.953+0000 7fae72e43740 1 -- 172.21.15.107:0/2627093776 shutdown_connections 2024-05-09T08:32:46.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.953+0000 7fae72e43740 1 -- 172.21.15.107:0/2627093776 wait complete. 2024-05-09T08:32:46.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.953+0000 7fae72e43740 1 -- start start 2024-05-09T08:32:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.953+0000 7fae72e43740 1 Processor -- start 2024-05-09T08:32:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.953+0000 7fae72e43740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1058c0 0x55593c180af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.953+0000 7fae72e43740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c1062b0 0x55593c183080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.953+0000 7fae72e43740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c185730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.953+0000 7fae72e43740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55593c11dc50 con 0x55593c10b420 2024-05-09T08:32:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.953+0000 7fae72e43740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55593c11d580 con 0x55593c1058c0 2024-05-09T08:32:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.953+0000 7fae72e43740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55593c11e320 con 0x55593c1062b0 2024-05-09T08:32:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.953+0000 7fae72e43740 10 client.0 _ll_register_callbacks cb 0x7fae6000b520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:32:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.953+0000 7fae6e6d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c185730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.953+0000 7fae6e6d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c185730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55712/0 (socket says 172.21.15.107:55712) 2024-05-09T08:32:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.953+0000 7fae6e6d1640 1 -- 172.21.15.107:0/3735307097 learned_addr learned my addr 172.21.15.107:0/3735307097 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.953+0000 7fae6d6cf640 1 --2- 172.21.15.107:0/3735307097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c1062b0 0x55593c183080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.953+0000 7fae6ded0640 1 --2- 172.21.15.107:0/3735307097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1058c0 0x55593c180af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.954+0000 7fae6e6d1640 1 -- 172.21.15.107:0/3735307097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c1062b0 msgr2=0x55593c183080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.954+0000 7fae6e6d1640 1 --2- 172.21.15.107:0/3735307097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c1062b0 0x55593c183080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.954+0000 7fae6e6d1640 1 -- 172.21.15.107:0/3735307097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1058c0 msgr2=0x55593c180af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.954+0000 7fae6e6d1640 1 --2- 172.21.15.107:0/3735307097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1058c0 0x55593c180af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.954+0000 7fae6e6d1640 1 -- 172.21.15.107:0/3735307097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55593c185f60 con 0x55593c10b420 2024-05-09T08:32:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.954+0000 7fae6ded0640 1 --2- 172.21.15.107:0/3735307097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1058c0 0x55593c180af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:32:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.954+0000 7fae6e6d1640 1 --2- 172.21.15.107:0/3735307097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c185730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fae60010a40 tx=0x7fae60011650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d80c9b86e73e7820 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.954+0000 7fae517fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:32:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.954+0000 7fae517fa640 1 -- 172.21.15.107:0/3735307097 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6002c040 con 0x55593c10b420 2024-05-09T08:32:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.954+0000 7fae517fa640 1 -- 172.21.15.107:0/3735307097 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fae6000ec70 con 0x55593c10b420 2024-05-09T08:32:46.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.954+0000 7fae72e43740 1 -- 172.21.15.107:0/3735307097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae6000e720 con 0x55593c10b420 2024-05-09T08:32:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.954+0000 7fae517fa640 1 -- 172.21.15.107:0/3735307097 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fae6000ee70 con 0x55593c10b420 2024-05-09T08:32:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.954+0000 7fae72e43740 10 client.15249 fetch_fsmap learned FSMap version 24 2024-05-09T08:32:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.954+0000 7fae72e43740 1 -- 172.21.15.107:0/3735307097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55593c1871d0 con 0x55593c10b420 2024-05-09T08:32:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.954+0000 7fae517fa640 1 -- 172.21.15.107:0/3735307097 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae6000e720 con 0x55593c10b420 2024-05-09T08:32:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.955+0000 7fae517fa640 1 -- 172.21.15.107:0/3735307097 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fae6003abf0 con 0x55593c10b420 2024-05-09T08:32:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.955+0000 7fae72e43740 10 client.15249 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T08:32:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.955+0000 7fae72e43740 10 client.15249 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T08:32:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.955+0000 7fae72e43740 20 client.15249 populate_metadata read hostname 'smithi107' 2024-05-09T08:32:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.955+0000 7fae72e43740 1 --2- 172.21.15.107:0/3735307097 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55593c18bba0 0x55593c18a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.955+0000 7fae72e43740 4 client.15249 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:32:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.955+0000 7fae72e43740 1 -- 172.21.15.107:0/3735307097 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "status"}) -- 0x55593c070f50 con 0x55593c18bba0 2024-05-09T08:32:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.955+0000 7fae6ded0640 1 --2- 172.21.15.107:0/3735307097 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55593c18bba0 0x55593c18a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:46.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.955+0000 7fae6ded0640 1 --2- 172.21.15.107:0/3735307097 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55593c18bba0 0x55593c18a8b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1aa969cbe61c55f8 server_cookie=9227246646f35a97 in_seq=0 out_seq=0 2024-05-09T08:32:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.955+0000 7fae517fa640 10 client.15249 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:32:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.955+0000 7fae517fa640 1 -- 172.21.15.107:0/3735307097 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55593c070f50 con 0x55593c18bba0 2024-05-09T08:32:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.955+0000 7fae517fa640 10 client.15249 handle_command_reply: tid=0 2024-05-09T08:32:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.955+0000 7fae72e43740 1 client.15249 shutdown 2024-05-09T08:32:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae72e43740 1 -- 172.21.15.107:0/3735307097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 msgr2=0x55593c185730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae72e43740 1 --2- 172.21.15.107:0/3735307097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c185730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fae60010a40 tx=0x7fae60011650 comp rx=0 tx=0).stop 2024-05-09T08:32:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae72e43740 1 -- 172.21.15.107:0/3735307097 shutdown_connections 2024-05-09T08:32:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae72e43740 1 --2- 172.21.15.107:0/3735307097 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55593c18bba0 0x55593c18a8b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae72e43740 1 --2- 172.21.15.107:0/3735307097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55593c10b420 0x55593c185730 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae72e43740 1 --2- 172.21.15.107:0/3735307097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55593c1062b0 0x55593c183080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae72e43740 1 --2- 172.21.15.107:0/3735307097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55593c1058c0 0x55593c180af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae517fa640 0 client.15249 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:32:46.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae72e43740 1 -- 172.21.15.107:0/3735307097 >> 172.21.15.107:0/3735307097 conn(0x55593c100f30 msgr2=0x55593c10e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:46.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae72e43740 1 -- 172.21.15.107:0/3735307097 shutdown_connections 2024-05-09T08:32:46.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae72e43740 1 -- 172.21.15.107:0/3735307097 wait complete. 2024-05-09T08:32:46.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae72e43740 20 client.15249 trim_cache size 0 max 16384 2024-05-09T08:32:46.964 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:32:46.964 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "3954c88c-0ddd-11ef-bc98-c7b262605968", 2024-05-09T08:32:46.964 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 2, 2024-05-09T08:32:46.964 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24478, 2024-05-09T08:32:46.965 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-05-09T08:32:46.965 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-05-09T08:32:46.965 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-05-09T08:32:46.965 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 86.904561279999996, 2024-05-09T08:32:46.965 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 24, 2024-05-09T08:32:46.965 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 91, 2024-05-09T08:32:46.965 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:32:46.965 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 126.669422649 2024-05-09T08:32:46.965 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:32:46.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae70134640 1 -- 172.21.15.107:0/1899288781 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fae540a2c10 msgr2=0x7fae540a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae70134640 1 --2- 172.21.15.107:0/1899288781 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fae540a2c10 0x7fae540a50d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fae681017c0 tx=0x7fae5c005db0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae70134640 1 -- 172.21.15.107:0/1899288781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae68101960 msgr2=0x7fae68197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:46.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae70134640 1 --2- 172.21.15.107:0/1899288781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae68101960 0x7fae68197fa0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fae64005940 tx=0x7fae64004430 comp rx=0 tx=0).stop 2024-05-09T08:32:46.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae70134640 1 -- 172.21.15.107:0/1899288781 shutdown_connections 2024-05-09T08:32:46.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae70134640 1 --2- 172.21.15.107:0/1899288781 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fae540a2c10 0x7fae540a50d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae70134640 1 --2- 172.21.15.107:0/1899288781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae68102320 0x7fae6819f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae70134640 1 --2- 172.21.15.107:0/1899288781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae68101960 0x7fae68197fa0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae70134640 1 --2- 172.21.15.107:0/1899288781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae68100770 0x7fae68197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:46.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.956+0000 7fae70134640 1 -- 172.21.15.107:0/1899288781 >> 172.21.15.107:0/1899288781 conn(0x7fae680fbce0 msgr2=0x7fae68104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:46.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.957+0000 7fae70134640 1 -- 172.21.15.107:0/1899288781 shutdown_connections 2024-05-09T08:32:46.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:46.957+0000 7fae70134640 1 -- 172.21.15.107:0/1899288781 wait complete. 2024-05-09T08:32:46.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-09T08:32:47.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.072+0000 7fd823472640 1 -- 172.21.15.107:0/3333433606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd81c101960 msgr2=0x7fd81c101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.072+0000 7fd823472640 1 --2- 172.21.15.107:0/3333433606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd81c101960 0x7fd81c101de0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd80c009900 tx=0x7fd80c02a410 comp rx=0 tx=0).stop 2024-05-09T08:32:47.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.072+0000 7fd823472640 1 -- 172.21.15.107:0/3333433606 shutdown_connections 2024-05-09T08:32:47.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.072+0000 7fd823472640 1 --2- 172.21.15.107:0/3333433606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd81c102320 0x7fd81c108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.072+0000 7fd823472640 1 --2- 172.21.15.107:0/3333433606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd81c101960 0x7fd81c101de0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.072+0000 7fd823472640 1 --2- 172.21.15.107:0/3333433606 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd81c100770 0x7fd81c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.072+0000 7fd823472640 1 -- 172.21.15.107:0/3333433606 >> 172.21.15.107:0/3333433606 conn(0x7fd81c0fbce0 msgr2=0x7fd81c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:47.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.072+0000 7fd823472640 1 -- 172.21.15.107:0/3333433606 shutdown_connections 2024-05-09T08:32:47.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.072+0000 7fd823472640 1 -- 172.21.15.107:0/3333433606 wait complete. 2024-05-09T08:32:47.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.072+0000 7fd823472640 1 Processor -- start 2024-05-09T08:32:47.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.072+0000 7fd823472640 1 -- start start 2024-05-09T08:32:47.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd823472640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd81c100770 0x7fd81c199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd823472640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd81c101960 0x7fd81c19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd823472640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd81c102320 0x7fd81c1a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd823472640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c10c300 con 0x7fd81c102320 2024-05-09T08:32:47.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd823472640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c10c180 con 0x7fd81c101960 2024-05-09T08:32:47.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd823472640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c10c480 con 0x7fd81c100770 2024-05-09T08:32:47.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd821a0f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd81c102320 0x7fd81c1a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd821a0f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd81c102320 0x7fd81c1a1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55726/0 (socket says 172.21.15.107:55726) 2024-05-09T08:32:47.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd821a0f640 1 -- 172.21.15.107:0/3410513694 learned_addr learned my addr 172.21.15.107:0/3410513694 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:47.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd82120e640 1 --2- 172.21.15.107:0/3410513694 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd81c100770 0x7fd81c199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd820a0d640 1 --2- 172.21.15.107:0/3410513694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd81c101960 0x7fd81c19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd821a0f640 1 -- 172.21.15.107:0/3410513694 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd81c100770 msgr2=0x7fd81c199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd821a0f640 1 --2- 172.21.15.107:0/3410513694 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd81c100770 0x7fd81c199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd821a0f640 1 -- 172.21.15.107:0/3410513694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd81c101960 msgr2=0x7fd81c19a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd821a0f640 1 --2- 172.21.15.107:0/3410513694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd81c101960 0x7fd81c19a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd821a0f640 1 -- 172.21.15.107:0/3410513694 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd81c1a1990 con 0x7fd81c102320 2024-05-09T08:32:47.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd821a0f640 1 --2- 172.21.15.107:0/3410513694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd81c102320 0x7fd81c1a1320 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd81400b970 tx=0x7fd81400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:47.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd8127fc640 1 -- 172.21.15.107:0/3410513694 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd814019070 con 0x7fd81c102320 2024-05-09T08:32:47.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd8127fc640 1 -- 172.21.15.107:0/3410513694 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd814014440 con 0x7fd81c102320 2024-05-09T08:32:47.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd823472640 1 -- 172.21.15.107:0/3410513694 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd81c1a1c80 con 0x7fd81c102320 2024-05-09T08:32:47.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd8127fc640 1 -- 172.21.15.107:0/3410513694 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8140075d0 con 0x7fd81c102320 2024-05-09T08:32:47.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.073+0000 7fd823472640 1 -- 172.21.15.107:0/3410513694 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd81c1a2180 con 0x7fd81c102320 2024-05-09T08:32:47.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.076+0000 7fd8127fc640 1 -- 172.21.15.107:0/3410513694 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd81401c070 con 0x7fd81c102320 2024-05-09T08:32:47.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.076+0000 7fd82617d740 1 Processor -- start 2024-05-09T08:32:47.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.076+0000 7fd82617d740 1 -- start start 2024-05-09T08:32:47.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.076+0000 7fd82617d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c55898c0 0x5646c5589ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.076+0000 7fd82617d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c558a2b0 0x5646c558edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.076+0000 7fd82617d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558f420 0x5646c55918b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.076+0000 7fd82617d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5646c54ed4d0 con 0x5646c55898c0 2024-05-09T08:32:47.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.076+0000 7fd82617d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5646c54953e0 con 0x5646c558a2b0 2024-05-09T08:32:47.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.076+0000 7fd82617d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5646c54e9490 con 0x5646c558f420 2024-05-09T08:32:47.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.076+0000 7fd820a0d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c55898c0 0x5646c5589ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.076+0000 7fd820a0d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c55898c0 0x5646c5589ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55734/0 (socket says 172.21.15.107:55734) 2024-05-09T08:32:47.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.076+0000 7fd820a0d640 1 -- 172.21.15.107:0/3054902004 learned_addr learned my addr 172.21.15.107:0/3054902004 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:47.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.076+0000 7fd8127fc640 1 --2- 172.21.15.107:0/3410513694 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd8080a2b90 0x7fd8080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.076+0000 7fd82120e640 1 --2- 172.21.15.107:0/3054902004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c558a2b0 0x5646c558edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.077+0000 7fd821a0f640 1 --2- 172.21.15.107:0/3410513694 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd8080a2b90 0x7fd8080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.077+0000 7fd820a0d640 1 --2- 172.21.15.107:0/3054902004 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558f420 0x5646c55918b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.077+0000 7fd820a0d640 1 -- 172.21.15.107:0/3054902004 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558f420 msgr2=0x5646c55918b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:47.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.077+0000 7fd820a0d640 1 --2- 172.21.15.107:0/3054902004 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558f420 0x5646c55918b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.077+0000 7fd820a0d640 1 -- 172.21.15.107:0/3054902004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c558a2b0 msgr2=0x5646c558edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:47.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.077+0000 7fd820a0d640 1 --2- 172.21.15.107:0/3054902004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c558a2b0 0x5646c558edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.077+0000 7fd820a0d640 1 -- 172.21.15.107:0/3054902004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646c52012d0 con 0x5646c55898c0 2024-05-09T08:32:47.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.077+0000 7fd8127fc640 1 -- 172.21.15.107:0/3410513694 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd8140cea20 con 0x7fd81c102320 2024-05-09T08:32:47.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.077+0000 7fd820a0d640 1 --2- 172.21.15.107:0/3054902004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c55898c0 0x5646c5589ce0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd80c006010 tx=0x7fd80c005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89930d9d047b9f0b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:47.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.077+0000 7fd821a0f640 1 --2- 172.21.15.107:0/3410513694 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd8080a2b90 0x7fd8080a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd814016000 tx=0x7fd81400e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:47.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.077+0000 7fd8077fe640 1 -- 172.21.15.107:0/3054902004 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c035050 con 0x5646c55898c0 2024-05-09T08:32:47.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.077+0000 7fd8077fe640 1 -- 172.21.15.107:0/3054902004 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd80c005040 con 0x5646c55898c0 2024-05-09T08:32:47.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.077+0000 7fd8077fe640 1 -- 172.21.15.107:0/3054902004 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c004a00 con 0x5646c55898c0 2024-05-09T08:32:47.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.078+0000 7fd82617d740 1 -- 172.21.15.107:0/3054902004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c55898c0 msgr2=0x5646c5589ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.078+0000 7fd82617d740 1 --2- 172.21.15.107:0/3054902004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c55898c0 0x5646c5589ce0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd80c006010 tx=0x7fd80c005d90 comp rx=0 tx=0).stop 2024-05-09T08:32:47.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.078+0000 7fd82617d740 1 -- 172.21.15.107:0/3054902004 shutdown_connections 2024-05-09T08:32:47.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.078+0000 7fd82617d740 1 --2- 172.21.15.107:0/3054902004 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558f420 0x5646c55918b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.078+0000 7fd82617d740 1 --2- 172.21.15.107:0/3054902004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c558a2b0 0x5646c558edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.078+0000 7fd82617d740 1 --2- 172.21.15.107:0/3054902004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c55898c0 0x5646c5589ce0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.078+0000 7fd82617d740 1 -- 172.21.15.107:0/3054902004 >> 172.21.15.107:0/3054902004 conn(0x5646c5584f30 msgr2=0x5646c5585330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:47.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.078+0000 7fd82617d740 1 -- 172.21.15.107:0/3054902004 shutdown_connections 2024-05-09T08:32:47.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.078+0000 7fd82617d740 1 -- 172.21.15.107:0/3054902004 wait complete. 2024-05-09T08:32:47.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.078+0000 7fd82617d740 1 -- start start 2024-05-09T08:32:47.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.078+0000 7fd82617d740 1 Processor -- start 2024-05-09T08:32:47.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd82617d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c55898c0 0x5646c5593ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd82617d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558a2b0 0x5646c5594220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd82617d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c558f420 0x5646c559b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd82617d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5646c5592370 con 0x5646c55898c0 2024-05-09T08:32:47.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd82617d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5646c54a88e0 con 0x5646c558f420 2024-05-09T08:32:47.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd82617d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5646c55924f0 con 0x5646c558a2b0 2024-05-09T08:32:47.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd82617d740 10 client.0 _ll_register_callbacks cb 0x5646c54759c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:32:47.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd820a0d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c55898c0 0x5646c5593ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd820a0d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c55898c0 0x5646c5593ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55742/0 (socket says 172.21.15.107:55742) 2024-05-09T08:32:47.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd820a0d640 1 -- 172.21.15.107:0/919548590 learned_addr learned my addr 172.21.15.107:0/919548590 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:47.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd82120e640 1 --2- 172.21.15.107:0/919548590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558a2b0 0x5646c5594220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd820a0d640 1 -- 172.21.15.107:0/919548590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558a2b0 msgr2=0x5646c5594220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd820a0d640 1 --2- 172.21.15.107:0/919548590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558a2b0 0x5646c5594220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd820a0d640 1 -- 172.21.15.107:0/919548590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c558f420 msgr2=0x5646c559b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd820a0d640 1 --2- 172.21.15.107:0/919548590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c558f420 0x5646c559b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd820a0d640 1 -- 172.21.15.107:0/919548590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646c52cb6b0 con 0x5646c55898c0 2024-05-09T08:32:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd820a0d640 1 --2- 172.21.15.107:0/919548590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c55898c0 0x5646c5593ce0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd80c02a930 tx=0x7fd80c02aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=325fdb681cfc5b08 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.079+0000 7fd804ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:32:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.080+0000 7fd804ff9640 1 -- 172.21.15.107:0/919548590 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c04d020 con 0x5646c55898c0 2024-05-09T08:32:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.080+0000 7fd804ff9640 1 -- 172.21.15.107:0/919548590 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd80c004a80 con 0x5646c55898c0 2024-05-09T08:32:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.080+0000 7fd82617d740 1 -- 172.21.15.107:0/919548590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd80c005040 con 0x5646c55898c0 2024-05-09T08:32:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.080+0000 7fd804ff9640 1 -- 172.21.15.107:0/919548590 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd80c035040 con 0x5646c55898c0 2024-05-09T08:32:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.080+0000 7fd82617d740 10 client.15261 fetch_fsmap learned FSMap version 24 2024-05-09T08:32:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.080+0000 7fd82617d740 1 -- 172.21.15.107:0/919548590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5646c559e230 con 0x5646c55898c0 2024-05-09T08:32:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.080+0000 7fd804ff9640 1 -- 172.21.15.107:0/919548590 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd80c005040 con 0x5646c55898c0 2024-05-09T08:32:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.080+0000 7fd804ff9640 1 -- 172.21.15.107:0/919548590 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd80c049c60 con 0x5646c55898c0 2024-05-09T08:32:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.080+0000 7fd82617d740 10 client.15261 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T08:32:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.080+0000 7fd82617d740 10 client.15261 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T08:32:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.080+0000 7fd82617d740 20 client.15261 populate_metadata read hostname 'smithi107' 2024-05-09T08:32:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.080+0000 7fd82617d740 1 --2- 172.21.15.107:0/919548590 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5646c55a0420 0x5646c55a1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.080+0000 7fd82617d740 4 client.15261 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:32:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.080+0000 7fd82617d740 1 -- 172.21.15.107:0/919548590 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646c54fac00 con 0x5646c55a0420 2024-05-09T08:32:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.081+0000 7fd82120e640 1 --2- 172.21.15.107:0/919548590 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5646c55a0420 0x5646c55a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.081+0000 7fd804ff9640 10 client.15261 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:32:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.081+0000 7fd82120e640 1 --2- 172.21.15.107:0/919548590 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5646c55a0420 0x5646c55a1890 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=833561ad3e6c1523 server_cookie=b6b54239114b1814 in_seq=0 out_seq=0 2024-05-09T08:32:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.082+0000 7fd804ff9640 1 -- 172.21.15.107:0/919548590 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5646c54fac00 con 0x5646c55a0420 2024-05-09T08:32:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.082+0000 7fd804ff9640 10 client.15261 handle_command_reply: tid=0 2024-05-09T08:32:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.082+0000 7fd82617d740 1 client.15261 shutdown 2024-05-09T08:32:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.082+0000 7fd82617d740 1 -- 172.21.15.107:0/919548590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c55898c0 msgr2=0x5646c5593ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.082+0000 7fd82617d740 1 --2- 172.21.15.107:0/919548590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c55898c0 0x5646c5593ce0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd80c02a930 tx=0x7fd80c02aa90 comp rx=0 tx=0).stop 2024-05-09T08:32:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.082+0000 7fd82617d740 1 -- 172.21.15.107:0/919548590 shutdown_connections 2024-05-09T08:32:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.082+0000 7fd82617d740 1 --2- 172.21.15.107:0/919548590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c558f420 0x5646c559b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.082+0000 7fd82617d740 1 --2- 172.21.15.107:0/919548590 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5646c55a0420 0x5646c55a1890 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.082+0000 7fd804ff9640 0 client.15261 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:32:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.082+0000 7fd82617d740 1 --2- 172.21.15.107:0/919548590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558a2b0 0x5646c5594220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.082+0000 7fd82617d740 1 --2- 172.21.15.107:0/919548590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c55898c0 0x5646c5593ce0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.082+0000 7fd82617d740 1 -- 172.21.15.107:0/919548590 >> 172.21.15.107:0/919548590 conn(0x5646c5584f30 msgr2=0x5646c5584000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.082+0000 7fd82617d740 1 -- 172.21.15.107:0/919548590 shutdown_connections 2024-05-09T08:32:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.082+0000 7fd82617d740 1 -- 172.21.15.107:0/919548590 wait complete. 2024-05-09T08:32:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.082+0000 7fd82617d740 20 client.15261 trim_cache size 0 max 16384 2024-05-09T08:32:47.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.100+0000 7fd82617d740 1 Processor -- start 2024-05-09T08:32:47.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.100+0000 7fd82617d740 1 -- start start 2024-05-09T08:32:47.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.100+0000 7fd82617d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c55898c0 0x5646c5588820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.100+0000 7fd82617d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c558a2b0 0x5646c559b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.100+0000 7fd82617d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558f420 0x5646c5594000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.100+0000 7fd82617d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5646c559fbd0 con 0x5646c558a2b0 2024-05-09T08:32:47.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.100+0000 7fd82617d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5646c55a02a0 con 0x5646c55898c0 2024-05-09T08:32:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.100+0000 7fd82617d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5646c55a0970 con 0x5646c558f420 2024-05-09T08:32:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.100+0000 7fd82120e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c558a2b0 0x5646c559b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.100+0000 7fd82120e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c558a2b0 0x5646c559b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55746/0 (socket says 172.21.15.107:55746) 2024-05-09T08:32:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.100+0000 7fd82120e640 1 -- 172.21.15.107:0/91190755 learned_addr learned my addr 172.21.15.107:0/91190755 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.101+0000 7fd820a0d640 1 --2- 172.21.15.107:0/91190755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558f420 0x5646c5594000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.101+0000 7fd820a0d640 1 --2- 172.21.15.107:0/91190755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c55898c0 0x5646c5588820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.101+0000 7fd82120e640 1 -- 172.21.15.107:0/91190755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558f420 msgr2=0x5646c5594000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.101+0000 7fd82120e640 1 --2- 172.21.15.107:0/91190755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558f420 0x5646c5594000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.101+0000 7fd82120e640 1 -- 172.21.15.107:0/91190755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c55898c0 msgr2=0x5646c5588820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.101+0000 7fd82120e640 1 --2- 172.21.15.107:0/91190755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c55898c0 0x5646c5588820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.101+0000 7fd82120e640 1 -- 172.21.15.107:0/91190755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646c558e980 con 0x5646c558a2b0 2024-05-09T08:32:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.101+0000 7fd820a0d640 1 --2- 172.21.15.107:0/91190755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c55898c0 0x5646c5588820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:32:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.101+0000 7fd82120e640 1 --2- 172.21.15.107:0/91190755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c558a2b0 0x5646c559b8e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd818000c00 tx=0x7fd81800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc2530216a8c1f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.101+0000 7fd806ffd640 1 -- 172.21.15.107:0/91190755 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818003cc0 con 0x5646c558a2b0 2024-05-09T08:32:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.101+0000 7fd806ffd640 1 -- 172.21.15.107:0/91190755 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd81801c920 con 0x5646c558a2b0 2024-05-09T08:32:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.101+0000 7fd806ffd640 1 -- 172.21.15.107:0/91190755 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818010cf0 con 0x5646c558a2b0 2024-05-09T08:32:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 -- 172.21.15.107:0/91190755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c558a2b0 msgr2=0x5646c559b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 --2- 172.21.15.107:0/91190755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c558a2b0 0x5646c559b8e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd818000c00 tx=0x7fd81800cd20 comp rx=0 tx=0).stop 2024-05-09T08:32:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 -- 172.21.15.107:0/91190755 shutdown_connections 2024-05-09T08:32:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 --2- 172.21.15.107:0/91190755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558f420 0x5646c5594000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 --2- 172.21.15.107:0/91190755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c558a2b0 0x5646c559b8e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 --2- 172.21.15.107:0/91190755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c55898c0 0x5646c5588820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 -- 172.21.15.107:0/91190755 >> 172.21.15.107:0/91190755 conn(0x5646c5584f30 msgr2=0x5646c55938f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 -- 172.21.15.107:0/91190755 shutdown_connections 2024-05-09T08:32:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 -- 172.21.15.107:0/91190755 wait complete. 2024-05-09T08:32:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 -- start start 2024-05-09T08:32:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 Processor -- start 2024-05-09T08:32:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c55898c0 0x5646c5596bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c558a2b0 0x5646c5597130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558f420 0x5646c559a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5646c55a1710 con 0x5646c558a2b0 2024-05-09T08:32:47.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5646c55a1040 con 0x5646c55898c0 2024-05-09T08:32:47.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.102+0000 7fd82617d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5646c55a1de0 con 0x5646c558f420 2024-05-09T08:32:47.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd82617d740 10 client.0 _ll_register_callbacks cb 0x7fd81800c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:32:47.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd82120e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c558a2b0 0x5646c5597130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd82120e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c558a2b0 0x5646c5597130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55750/0 (socket says 172.21.15.107:55750) 2024-05-09T08:32:47.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd82120e640 1 -- 172.21.15.107:0/3635487689 learned_addr learned my addr 172.21.15.107:0/3635487689 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:47.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd820a0d640 1 --2- 172.21.15.107:0/3635487689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558f420 0x5646c559a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd820a0d640 1 --2- 172.21.15.107:0/3635487689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c55898c0 0x5646c5596bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd82120e640 1 -- 172.21.15.107:0/3635487689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558f420 msgr2=0x5646c559a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:47.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd82120e640 1 --2- 172.21.15.107:0/3635487689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558f420 0x5646c559a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd82120e640 1 -- 172.21.15.107:0/3635487689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c55898c0 msgr2=0x5646c5596bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:47.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd82120e640 1 --2- 172.21.15.107:0/3635487689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c55898c0 0x5646c5596bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd82120e640 1 -- 172.21.15.107:0/3635487689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646c559a740 con 0x5646c558a2b0 2024-05-09T08:32:47.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd820a0d640 1 --2- 172.21.15.107:0/3635487689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c55898c0 0x5646c5596bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:32:47.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd82120e640 1 --2- 172.21.15.107:0/3635487689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c558a2b0 0x5646c5597130 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd81801f040 tx=0x7fd818010820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cb17ccc5e2b7a2b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:47.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd804ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:32:47.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd804ff9640 1 -- 172.21.15.107:0/3635487689 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818010c70 con 0x5646c558a2b0 2024-05-09T08:32:47.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd804ff9640 1 -- 172.21.15.107:0/3635487689 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd818003ac0 con 0x5646c558a2b0 2024-05-09T08:32:47.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd82617d740 1 -- 172.21.15.107:0/3635487689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5646c54f4f50 con 0x5646c558a2b0 2024-05-09T08:32:47.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd804ff9640 1 -- 172.21.15.107:0/3635487689 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd818036050 con 0x5646c558a2b0 2024-05-09T08:32:47.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd82617d740 10 client.15267 fetch_fsmap learned FSMap version 24 2024-05-09T08:32:47.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.103+0000 7fd82617d740 1 -- 172.21.15.107:0/3635487689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5646c5606900 con 0x5646c558a2b0 2024-05-09T08:32:47.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.104+0000 7fd804ff9640 1 -- 172.21.15.107:0/3635487689 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5646c54f4f50 con 0x5646c558a2b0 2024-05-09T08:32:47.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.104+0000 7fd804ff9640 1 -- 172.21.15.107:0/3635487689 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd8180048c0 con 0x5646c558a2b0 2024-05-09T08:32:47.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.104+0000 7fd82617d740 10 client.15267 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T08:32:47.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.104+0000 7fd82617d740 10 client.15267 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T08:32:47.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.104+0000 7fd82617d740 20 client.15267 populate_metadata read hostname 'smithi107' 2024-05-09T08:32:47.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.104+0000 7fd82617d740 1 --2- 172.21.15.107:0/3635487689 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5646c55a0420 0x5646c560a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.104+0000 7fd82617d740 4 client.15267 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:32:47.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.104+0000 7fd82617d740 1 -- 172.21.15.107:0/3635487689 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "status"}) -- 0x5646c5472b80 con 0x5646c55a0420 2024-05-09T08:32:47.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.104+0000 7fd820a0d640 1 --2- 172.21.15.107:0/3635487689 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5646c55a0420 0x5646c560a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.104+0000 7fd820a0d640 1 --2- 172.21.15.107:0/3635487689 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5646c55a0420 0x5646c560a030 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=67387e9c3f201f37 server_cookie=6bcf413eb84d8e65 in_seq=0 out_seq=0 2024-05-09T08:32:47.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.104+0000 7fd804ff9640 10 client.15267 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:32:47.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.105+0000 7fd804ff9640 1 -- 172.21.15.107:0/3635487689 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5646c5472b80 con 0x5646c55a0420 2024-05-09T08:32:47.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.105+0000 7fd804ff9640 10 client.15267 handle_command_reply: tid=0 2024-05-09T08:32:47.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.105+0000 7fd82617d740 1 client.15267 shutdown 2024-05-09T08:32:47.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.105+0000 7fd82617d740 1 -- 172.21.15.107:0/3635487689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c558a2b0 msgr2=0x5646c5597130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.105+0000 7fd82617d740 1 --2- 172.21.15.107:0/3635487689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c558a2b0 0x5646c5597130 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd81801f040 tx=0x7fd818010820 comp rx=0 tx=0).stop 2024-05-09T08:32:47.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.105+0000 7fd82617d740 1 -- 172.21.15.107:0/3635487689 shutdown_connections 2024-05-09T08:32:47.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.105+0000 7fd82617d740 1 --2- 172.21.15.107:0/3635487689 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5646c55a0420 0x5646c560a030 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.105+0000 7fd82617d740 1 --2- 172.21.15.107:0/3635487689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646c558f420 0x5646c559a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.105+0000 7fd82617d740 1 --2- 172.21.15.107:0/3635487689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646c558a2b0 0x5646c5597130 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.105+0000 7fd804ff9640 0 client.15267 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:32:47.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.105+0000 7fd82617d740 1 --2- 172.21.15.107:0/3635487689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646c55898c0 0x5646c5596bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.105+0000 7fd82617d740 1 -- 172.21.15.107:0/3635487689 >> 172.21.15.107:0/3635487689 conn(0x5646c5584f30 msgr2=0x5646c55886a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:47.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.105+0000 7fd82617d740 1 -- 172.21.15.107:0/3635487689 shutdown_connections 2024-05-09T08:32:47.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.105+0000 7fd82617d740 1 -- 172.21.15.107:0/3635487689 wait complete. 2024-05-09T08:32:47.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.105+0000 7fd82617d740 20 client.15267 trim_cache size 0 max 16384 2024-05-09T08:32:47.114 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:32:47.114 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "3954c88c-0ddd-11ef-bc98-c7b262605968", 2024-05-09T08:32:47.114 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 1, 2024-05-09T08:32:47.114 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24491, 2024-05-09T08:32:47.114 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-05-09T08:32:47.115 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-05-09T08:32:47.115 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-05-09T08:32:47.115 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 88.063955215999997, 2024-05-09T08:32:47.115 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 24, 2024-05-09T08:32:47.115 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 91, 2024-05-09T08:32:47.115 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:32:47.115 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 112.52013106299999 2024-05-09T08:32:47.115 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:32:47.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.106+0000 7fd823472640 1 -- 172.21.15.107:0/3410513694 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd8080a2b90 msgr2=0x7fd8080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.106+0000 7fd823472640 1 --2- 172.21.15.107:0/3410513694 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd8080a2b90 0x7fd8080a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd814016000 tx=0x7fd81400e040 comp rx=0 tx=0).stop 2024-05-09T08:32:47.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.106+0000 7fd823472640 1 -- 172.21.15.107:0/3410513694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd81c102320 msgr2=0x7fd81c1a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.106+0000 7fd823472640 1 --2- 172.21.15.107:0/3410513694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd81c102320 0x7fd81c1a1320 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd81400b970 tx=0x7fd81400be30 comp rx=0 tx=0).stop 2024-05-09T08:32:47.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.106+0000 7fd823472640 1 -- 172.21.15.107:0/3410513694 shutdown_connections 2024-05-09T08:32:47.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.106+0000 7fd823472640 1 --2- 172.21.15.107:0/3410513694 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd8080a2b90 0x7fd8080a5050 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.106+0000 7fd823472640 1 --2- 172.21.15.107:0/3410513694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd81c102320 0x7fd81c1a1320 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.106+0000 7fd823472640 1 --2- 172.21.15.107:0/3410513694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd81c101960 0x7fd81c19a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.106+0000 7fd823472640 1 --2- 172.21.15.107:0/3410513694 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd81c100770 0x7fd81c199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.106+0000 7fd823472640 1 -- 172.21.15.107:0/3410513694 >> 172.21.15.107:0/3410513694 conn(0x7fd81c0fbce0 msgr2=0x7fd81c068a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:47.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.106+0000 7fd823472640 1 -- 172.21.15.107:0/3410513694 shutdown_connections 2024-05-09T08:32:47.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.106+0000 7fd823472640 1 -- 172.21.15.107:0/3410513694 wait complete. 2024-05-09T08:32:47.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-05-09T08:32:47.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.221+0000 7fbfb45dc640 1 -- 172.21.15.107:0/2915122574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfac100770 msgr2=0x7fbfac100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.221+0000 7fbfb45dc640 1 --2- 172.21.15.107:0/2915122574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfac100770 0x7fbfac100b90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009900 tx=0x7fbfa002a410 comp rx=0 tx=0).stop 2024-05-09T08:32:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.221+0000 7fbfb45dc640 1 -- 172.21.15.107:0/2915122574 shutdown_connections 2024-05-09T08:32:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.221+0000 7fbfb45dc640 1 --2- 172.21.15.107:0/2915122574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfac102320 0x7fbfac108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.221+0000 7fbfb45dc640 1 --2- 172.21.15.107:0/2915122574 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfac101960 0x7fbfac101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.221+0000 7fbfb45dc640 1 --2- 172.21.15.107:0/2915122574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfac100770 0x7fbfac100b90 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.221+0000 7fbfb45dc640 1 -- 172.21.15.107:0/2915122574 >> 172.21.15.107:0/2915122574 conn(0x7fbfac0fbce0 msgr2=0x7fbfac0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.221+0000 7fbfb45dc640 1 -- 172.21.15.107:0/2915122574 shutdown_connections 2024-05-09T08:32:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.221+0000 7fbfb45dc640 1 -- 172.21.15.107:0/2915122574 wait complete. 2024-05-09T08:32:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.222+0000 7fbfb45dc640 1 Processor -- start 2024-05-09T08:32:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.222+0000 7fbfb45dc640 1 -- start start 2024-05-09T08:32:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.222+0000 7fbfb45dc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfac100770 0x7fbfac069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.222+0000 7fbfb45dc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfac101960 0x7fbfac19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.222+0000 7fbfb45dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfac102320 0x7fbfac19ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.222+0000 7fbfb45dc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbfac10c290 con 0x7fbfac102320 2024-05-09T08:32:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.222+0000 7fbfb45dc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbfac10c110 con 0x7fbfac100770 2024-05-09T08:32:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.222+0000 7fbfb45dc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbfac10c410 con 0x7fbfac101960 2024-05-09T08:32:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.222+0000 7fbfb2b79640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfac102320 0x7fbfac19ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.222+0000 7fbfb2b79640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfac102320 0x7fbfac19ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55762/0 (socket says 172.21.15.107:55762) 2024-05-09T08:32:47.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.222+0000 7fbfb2b79640 1 -- 172.21.15.107:0/401171606 learned_addr learned my addr 172.21.15.107:0/401171606 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:47.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.222+0000 7fbfb2378640 1 --2- 172.21.15.107:0/401171606 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfac100770 0x7fbfac069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.222+0000 7fbfb1b77640 1 --2- 172.21.15.107:0/401171606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfac101960 0x7fbfac19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.222+0000 7fbfb2b79640 1 -- 172.21.15.107:0/401171606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfac101960 msgr2=0x7fbfac19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.223+0000 7fbfb2b79640 1 --2- 172.21.15.107:0/401171606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfac101960 0x7fbfac19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.223+0000 7fbfb2b79640 1 -- 172.21.15.107:0/401171606 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfac100770 msgr2=0x7fbfac069530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.223+0000 7fbfb2b79640 1 --2- 172.21.15.107:0/401171606 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfac100770 0x7fbfac069530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.223+0000 7fbfb2b79640 1 -- 172.21.15.107:0/401171606 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfac19f7a0 con 0x7fbfac102320 2024-05-09T08:32:47.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.223+0000 7fbfb2b79640 1 --2- 172.21.15.107:0/401171606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfac102320 0x7fbfac19ee90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fbfa400eaa0 tx=0x7fbfa400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:47.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.223+0000 7fbf9f7fe640 1 -- 172.21.15.107:0/401171606 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa400cdb0 con 0x7fbfac102320 2024-05-09T08:32:47.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.223+0000 7fbf9f7fe640 1 -- 172.21.15.107:0/401171606 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfa4005020 con 0x7fbfac102320 2024-05-09T08:32:47.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.223+0000 7fbf9f7fe640 1 -- 172.21.15.107:0/401171606 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa40116d0 con 0x7fbfac102320 2024-05-09T08:32:47.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.223+0000 7fbfb45dc640 1 -- 172.21.15.107:0/401171606 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfac19fa30 con 0x7fbfac102320 2024-05-09T08:32:47.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.223+0000 7fbfb45dc640 1 -- 172.21.15.107:0/401171606 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfac19ff80 con 0x7fbfac102320 2024-05-09T08:32:47.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.225+0000 7fbf9f7fe640 1 -- 172.21.15.107:0/401171606 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fbfa40051d0 con 0x7fbfac102320 2024-05-09T08:32:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.226+0000 7fbfb73c9740 1 Processor -- start 2024-05-09T08:32:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.226+0000 7fbfb73c9740 1 -- start start 2024-05-09T08:32:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.226+0000 7fbfb73c9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df695c8c0 0x560df695cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.226+0000 7fbfb73c9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df695d2b0 0x560df6961db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.226+0000 7fbfb73c9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df6962420 0x560df69648b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.226+0000 7fbfb73c9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560df68c04d0 con 0x560df6962420 2024-05-09T08:32:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.226+0000 7fbfb73c9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560df68683e0 con 0x560df695d2b0 2024-05-09T08:32:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.226+0000 7fbfb73c9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560df68bc490 con 0x560df695c8c0 2024-05-09T08:32:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.226+0000 7fbf9f7fe640 1 --2- 172.21.15.107:0/401171606 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbf980a2c10 0x7fbf980a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.226+0000 7fbfb1b77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df6962420 0x560df69648b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.226+0000 7fbfb1b77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df6962420 0x560df69648b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55772/0 (socket says 172.21.15.107:55772) 2024-05-09T08:32:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.226+0000 7fbfb1b77640 1 -- 172.21.15.107:0/1126192817 learned_addr learned my addr 172.21.15.107:0/1126192817 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.226+0000 7fbfb2378640 1 --2- 172.21.15.107:0/1126192817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df695d2b0 0x560df6961db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.226+0000 7fbfb1b77640 1 --2- 172.21.15.107:0/1126192817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df695c8c0 0x560df695cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.226+0000 7fbf9f7fe640 1 -- 172.21.15.107:0/401171606 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbfa40cf540 con 0x7fbfac102320 2024-05-09T08:32:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.226+0000 7fbfb2b79640 1 --2- 172.21.15.107:0/401171606 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbf980a2c10 0x7fbf980a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.227+0000 7fbfb1b77640 1 -- 172.21.15.107:0/1126192817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df695c8c0 msgr2=0x560df695cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.227+0000 7fbfb1b77640 1 --2- 172.21.15.107:0/1126192817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df695c8c0 0x560df695cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.227+0000 7fbfb1b77640 1 -- 172.21.15.107:0/1126192817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df695d2b0 msgr2=0x560df6961db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.227+0000 7fbfb1b77640 1 --2- 172.21.15.107:0/1126192817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df695d2b0 0x560df6961db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.227+0000 7fbfb1b77640 1 -- 172.21.15.107:0/1126192817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560df65d42d0 con 0x560df6962420 2024-05-09T08:32:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.227+0000 7fbfb2b79640 1 --2- 172.21.15.107:0/401171606 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbf980a2c10 0x7fbf980a50d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4097ea0 tx=0x7fbfa400cf50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.227+0000 7fbfb1b77640 1 --2- 172.21.15.107:0/1126192817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df6962420 0x560df69648b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbfa80058c0 tx=0x7fbfa800c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96efb3304a314da server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:47.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.227+0000 7fbf9cbfe640 1 -- 172.21.15.107:0/1126192817 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa80142d0 con 0x560df6962420 2024-05-09T08:32:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.227+0000 7fbf9cbfe640 1 -- 172.21.15.107:0/1126192817 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfa80054d0 con 0x560df6962420 2024-05-09T08:32:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.227+0000 7fbf9cbfe640 1 -- 172.21.15.107:0/1126192817 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8004e50 con 0x560df6962420 2024-05-09T08:32:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 -- 172.21.15.107:0/1126192817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df6962420 msgr2=0x560df69648b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/1126192817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df6962420 0x560df69648b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbfa80058c0 tx=0x7fbfa800c4d0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 -- 172.21.15.107:0/1126192817 shutdown_connections 2024-05-09T08:32:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/1126192817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df6962420 0x560df69648b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/1126192817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df695d2b0 0x560df6961db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/1126192817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df695c8c0 0x560df695cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 -- 172.21.15.107:0/1126192817 >> 172.21.15.107:0/1126192817 conn(0x560df6957f30 msgr2=0x560df6958330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 -- 172.21.15.107:0/1126192817 shutdown_connections 2024-05-09T08:32:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 -- 172.21.15.107:0/1126192817 wait complete. 2024-05-09T08:32:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 -- start start 2024-05-09T08:32:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 Processor -- start 2024-05-09T08:32:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df695c8c0 0x560df6966cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695d2b0 0x560df6967200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df6962420 0x560df696e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560df69653d0 con 0x560df695d2b0 2024-05-09T08:32:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560df687b8e0 con 0x560df695c8c0 2024-05-09T08:32:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.228+0000 7fbfb73c9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560df6965550 con 0x560df6962420 2024-05-09T08:32:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbfb73c9740 10 client.0 _ll_register_callbacks cb 0x560df68489c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:32:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbfb2378640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695d2b0 0x560df6967200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbfb2378640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695d2b0 0x560df6967200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55786/0 (socket says 172.21.15.107:55786) 2024-05-09T08:32:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbfb2378640 1 -- 172.21.15.107:0/3386599567 learned_addr learned my addr 172.21.15.107:0/3386599567 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbfb1b77640 1 --2- 172.21.15.107:0/3386599567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df6962420 0x560df696e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbfb2378640 1 -- 172.21.15.107:0/3386599567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df6962420 msgr2=0x560df696e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbfb2378640 1 --2- 172.21.15.107:0/3386599567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df6962420 0x560df696e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbfb2378640 1 -- 172.21.15.107:0/3386599567 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df695c8c0 msgr2=0x560df6966cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbfb2378640 1 --2- 172.21.15.107:0/3386599567 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df695c8c0 0x560df6966cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbfb2378640 1 -- 172.21.15.107:0/3386599567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560df669e6b0 con 0x560df695d2b0 2024-05-09T08:32:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbfb1b77640 1 --2- 172.21.15.107:0/3386599567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df6962420 0x560df696e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:32:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbfb2378640 1 --2- 172.21.15.107:0/3386599567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695d2b0 0x560df6967200 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009900 tx=0x7fbfa0005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=262bca1971142253 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbf95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:32:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbf95ffb640 1 -- 172.21.15.107:0/3386599567 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0035040 con 0x560df695d2b0 2024-05-09T08:32:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbf95ffb640 1 -- 172.21.15.107:0/3386599567 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbfa0004240 con 0x560df695d2b0 2024-05-09T08:32:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbfb73c9740 1 -- 172.21.15.107:0/3386599567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfa80054d0 con 0x560df695d2b0 2024-05-09T08:32:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbf95ffb640 1 -- 172.21.15.107:0/3386599567 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbfa0004440 con 0x560df695d2b0 2024-05-09T08:32:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbfb73c9740 10 client.15279 fetch_fsmap learned FSMap version 24 2024-05-09T08:32:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbfb73c9740 1 -- 172.21.15.107:0/3386599567 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560df6971280 con 0x560df695d2b0 2024-05-09T08:32:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.229+0000 7fbf95ffb640 1 -- 172.21.15.107:0/3386599567 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfa80054d0 con 0x560df695d2b0 2024-05-09T08:32:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.230+0000 7fbf95ffb640 1 -- 172.21.15.107:0/3386599567 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fbfa0044020 con 0x560df695d2b0 2024-05-09T08:32:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.230+0000 7fbfb73c9740 10 client.15279 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T08:32:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.230+0000 7fbfb73c9740 10 client.15279 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T08:32:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.230+0000 7fbfb73c9740 20 client.15279 populate_metadata read hostname 'smithi107' 2024-05-09T08:32:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.230+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/3386599567 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560df69738a0 0x560df6974d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.230+0000 7fbfb73c9740 4 client.15279 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:32:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.230+0000 7fbfb73c9740 1 -- 172.21.15.107:0/3386599567 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560df68cdc00 con 0x560df69738a0 2024-05-09T08:32:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.230+0000 7fbfb1b77640 1 --2- 172.21.15.107:0/3386599567 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560df69738a0 0x560df6974d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.230+0000 7fbf95ffb640 10 client.15279 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:32:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.230+0000 7fbfb1b77640 1 --2- 172.21.15.107:0/3386599567 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560df69738a0 0x560df6974d10 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd18db8b618aa7fb server_cookie=54695ae1cff74883 in_seq=0 out_seq=0 2024-05-09T08:32:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.231+0000 7fbf95ffb640 1 -- 172.21.15.107:0/3386599567 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560df68cdc00 con 0x560df69738a0 2024-05-09T08:32:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.231+0000 7fbf95ffb640 10 client.15279 handle_command_reply: tid=0 2024-05-09T08:32:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.231+0000 7fbfb73c9740 1 client.15279 shutdown 2024-05-09T08:32:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.231+0000 7fbfb73c9740 1 -- 172.21.15.107:0/3386599567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695d2b0 msgr2=0x560df6967200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.231+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/3386599567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695d2b0 0x560df6967200 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009900 tx=0x7fbfa0005d90 comp rx=0 tx=0).stop 2024-05-09T08:32:47.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.231+0000 7fbfb73c9740 1 -- 172.21.15.107:0/3386599567 shutdown_connections 2024-05-09T08:32:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.231+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/3386599567 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560df69738a0 0x560df6974d10 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.231+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/3386599567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df6962420 0x560df696e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.231+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/3386599567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695d2b0 0x560df6967200 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.231+0000 7fbf95ffb640 0 client.15279 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:32:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.231+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/3386599567 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df695c8c0 0x560df6966cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.231+0000 7fbfb73c9740 1 -- 172.21.15.107:0/3386599567 >> 172.21.15.107:0/3386599567 conn(0x560df6957f30 msgr2=0x560df6957000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.232+0000 7fbfb73c9740 1 -- 172.21.15.107:0/3386599567 shutdown_connections 2024-05-09T08:32:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.232+0000 7fbfb73c9740 1 -- 172.21.15.107:0/3386599567 wait complete. 2024-05-09T08:32:47.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.232+0000 7fbfb73c9740 20 client.15279 trim_cache size 0 max 16384 2024-05-09T08:32:47.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.249+0000 7fbfb73c9740 1 Processor -- start 2024-05-09T08:32:47.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.249+0000 7fbfb73c9740 1 -- start start 2024-05-09T08:32:47.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb73c9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695c8c0 0x560df6964810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb73c9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df695d2b0 0x560df696e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb73c9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df6962420 0x560df6952d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb73c9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560df6973050 con 0x560df695c8c0 2024-05-09T08:32:47.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb73c9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560df6973720 con 0x560df695d2b0 2024-05-09T08:32:47.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb73c9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560df6973df0 con 0x560df6962420 2024-05-09T08:32:47.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb1b77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695c8c0 0x560df6964810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb1b77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695c8c0 0x560df6964810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55788/0 (socket says 172.21.15.107:55788) 2024-05-09T08:32:47.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb1b77640 1 -- 172.21.15.107:0/297709169 learned_addr learned my addr 172.21.15.107:0/297709169 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:47.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb2378640 1 --2- 172.21.15.107:0/297709169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df695d2b0 0x560df696e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb1b77640 1 -- 172.21.15.107:0/297709169 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df6962420 msgr2=0x560df6952d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:47.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb1b77640 1 --2- 172.21.15.107:0/297709169 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df6962420 0x560df6952d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb1b77640 1 -- 172.21.15.107:0/297709169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df695d2b0 msgr2=0x560df696e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:47.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb1b77640 1 --2- 172.21.15.107:0/297709169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df695d2b0 0x560df696e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb1b77640 1 -- 172.21.15.107:0/297709169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560df6957a70 con 0x560df695c8c0 2024-05-09T08:32:47.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb2378640 1 --2- 172.21.15.107:0/297709169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df695d2b0 0x560df696e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:32:47.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbfb1b77640 1 --2- 172.21.15.107:0/297709169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695c8c0 0x560df6964810 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbfa80058c0 tx=0x7fbfa8014f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67284b80704a46b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:47.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbf97fff640 1 -- 172.21.15.107:0/297709169 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa802f070 con 0x560df695c8c0 2024-05-09T08:32:47.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbf97fff640 1 -- 172.21.15.107:0/297709169 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfa802a770 con 0x560df695c8c0 2024-05-09T08:32:47.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.250+0000 7fbf97fff640 1 -- 172.21.15.107:0/297709169 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa800f990 con 0x560df695c8c0 2024-05-09T08:32:47.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.251+0000 7fbfb73c9740 1 -- 172.21.15.107:0/297709169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695c8c0 msgr2=0x560df6964810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.251+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/297709169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695c8c0 0x560df6964810 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbfa80058c0 tx=0x7fbfa8014f10 comp rx=0 tx=0).stop 2024-05-09T08:32:47.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.251+0000 7fbfb73c9740 1 -- 172.21.15.107:0/297709169 shutdown_connections 2024-05-09T08:32:47.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.251+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/297709169 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df6962420 0x560df6952d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.251+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/297709169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df695d2b0 0x560df696e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.251+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/297709169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695c8c0 0x560df6964810 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.251+0000 7fbfb73c9740 1 -- 172.21.15.107:0/297709169 >> 172.21.15.107:0/297709169 conn(0x560df6957f30 msgr2=0x560df6958310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:47.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.251+0000 7fbfb73c9740 1 -- 172.21.15.107:0/297709169 shutdown_connections 2024-05-09T08:32:47.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.251+0000 7fbfb73c9740 1 -- 172.21.15.107:0/297709169 wait complete. 2024-05-09T08:32:47.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.251+0000 7fbfb73c9740 1 -- start start 2024-05-09T08:32:47.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.251+0000 7fbfb73c9740 1 Processor -- start 2024-05-09T08:32:47.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb73c9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695c8c0 0x560df6969f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb73c9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df695d2b0 0x560df696a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb73c9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df6962420 0x560df696d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb73c9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560df6974b90 con 0x560df695c8c0 2024-05-09T08:32:47.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb73c9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560df69744c0 con 0x560df6962420 2024-05-09T08:32:47.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb73c9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560df6975260 con 0x560df695d2b0 2024-05-09T08:32:47.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb73c9740 10 client.0 _ll_register_callbacks cb 0x7fbfa0009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:32:47.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb1b77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695c8c0 0x560df6969f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb1b77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695c8c0 0x560df6969f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55796/0 (socket says 172.21.15.107:55796) 2024-05-09T08:32:47.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb1b77640 1 -- 172.21.15.107:0/110487943 learned_addr learned my addr 172.21.15.107:0/110487943 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:47.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb2378640 1 --2- 172.21.15.107:0/110487943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df695d2b0 0x560df696a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb1b77640 1 -- 172.21.15.107:0/110487943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df695d2b0 msgr2=0x560df696a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:47.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb1b77640 1 --2- 172.21.15.107:0/110487943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df695d2b0 0x560df696a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb1b77640 1 -- 172.21.15.107:0/110487943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df6962420 msgr2=0x560df696d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:32:47.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb1b77640 1 --2- 172.21.15.107:0/110487943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df6962420 0x560df696d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb1b77640 1 -- 172.21.15.107:0/110487943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560df696da00 con 0x560df695c8c0 2024-05-09T08:32:47.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb1b77640 1 --2- 172.21.15.107:0/110487943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695c8c0 0x560df6969f00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbfa802d040 tx=0x7fbfa800ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d770e30b8ad3f14d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:47.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbf95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:32:47.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbf95ffb640 1 -- 172.21.15.107:0/110487943 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa802f030 con 0x560df695c8c0 2024-05-09T08:32:47.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbf95ffb640 1 -- 172.21.15.107:0/110487943 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbfa802a920 con 0x560df695c8c0 2024-05-09T08:32:47.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.252+0000 7fbfb73c9740 1 -- 172.21.15.107:0/110487943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfa802a770 con 0x560df695c8c0 2024-05-09T08:32:47.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.253+0000 7fbf95ffb640 1 -- 172.21.15.107:0/110487943 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbfa802ab20 con 0x560df695c8c0 2024-05-09T08:32:47.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.253+0000 7fbfb73c9740 10 client.15285 fetch_fsmap learned FSMap version 24 2024-05-09T08:32:47.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.253+0000 7fbfb73c9740 1 -- 172.21.15.107:0/110487943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560df69d9d90 con 0x560df695c8c0 2024-05-09T08:32:47.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.253+0000 7fbf95ffb640 1 -- 172.21.15.107:0/110487943 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfa802a770 con 0x560df695c8c0 2024-05-09T08:32:47.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.253+0000 7fbf95ffb640 1 -- 172.21.15.107:0/110487943 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fbfa8033070 con 0x560df695c8c0 2024-05-09T08:32:47.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.253+0000 7fbfb73c9740 10 client.15285 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T08:32:47.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.253+0000 7fbfb73c9740 10 client.15285 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T08:32:47.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.253+0000 7fbfb73c9740 20 client.15285 populate_metadata read hostname 'smithi107' 2024-05-09T08:32:47.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.253+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/110487943 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560df69de230 0x560df69dcf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:47.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.253+0000 7fbfb73c9740 4 client.15285 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:32:47.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.253+0000 7fbfb73c9740 1 -- 172.21.15.107:0/110487943 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "status"}) -- 0x560df68c7f50 con 0x560df69de230 2024-05-09T08:32:47.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.253+0000 7fbfb2378640 1 --2- 172.21.15.107:0/110487943 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560df69de230 0x560df69dcf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:47.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbf95ffb640 10 client.15285 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:32:47.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbfb2378640 1 --2- 172.21.15.107:0/110487943 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560df69de230 0x560df69dcf40 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=742bade3ee9daf50 server_cookie=8f2360077a5ff709 in_seq=0 out_seq=0 2024-05-09T08:32:47.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbf95ffb640 1 -- 172.21.15.107:0/110487943 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x560df68c7f50 con 0x560df69de230 2024-05-09T08:32:47.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbf95ffb640 10 client.15285 handle_command_reply: tid=0 2024-05-09T08:32:47.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbfb73c9740 1 client.15285 shutdown 2024-05-09T08:32:47.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbfb73c9740 1 -- 172.21.15.107:0/110487943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695c8c0 msgr2=0x560df6969f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/110487943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695c8c0 0x560df6969f00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbfa802d040 tx=0x7fbfa800ca00 comp rx=0 tx=0).stop 2024-05-09T08:32:47.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbfb73c9740 1 -- 172.21.15.107:0/110487943 shutdown_connections 2024-05-09T08:32:47.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/110487943 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560df69de230 0x560df69dcf40 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/110487943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560df6962420 0x560df696d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/110487943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560df695d2b0 0x560df696a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbf95ffb640 0 client.15285 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:32:47.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbfb73c9740 1 --2- 172.21.15.107:0/110487943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560df695c8c0 0x560df6969f00 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbfb73c9740 1 -- 172.21.15.107:0/110487943 >> 172.21.15.107:0/110487943 conn(0x560df6957f30 msgr2=0x560df69668d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:47.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbfb73c9740 1 -- 172.21.15.107:0/110487943 shutdown_connections 2024-05-09T08:32:47.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbfb73c9740 1 -- 172.21.15.107:0/110487943 wait complete. 2024-05-09T08:32:47.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.254+0000 7fbfb73c9740 20 client.15285 trim_cache size 0 max 16384 2024-05-09T08:32:47.263 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:32:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "3954c88c-0ddd-11ef-bc98-c7b262605968", 2024-05-09T08:32:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2024-05-09T08:32:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24499, 2024-05-09T08:32:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-05-09T08:32:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-05-09T08:32:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-05-09T08:32:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 106.446437037, 2024-05-09T08:32:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 24, 2024-05-09T08:32:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 91, 2024-05-09T08:32:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:32:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 121.338615007 2024-05-09T08:32:47.264 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:32:47.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.255+0000 7fbfb45dc640 1 -- 172.21.15.107:0/401171606 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbf980a2c10 msgr2=0x7fbf980a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.255+0000 7fbfb45dc640 1 --2- 172.21.15.107:0/401171606 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbf980a2c10 0x7fbf980a50d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4097ea0 tx=0x7fbfa400cf50 comp rx=0 tx=0).stop 2024-05-09T08:32:47.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.255+0000 7fbfb45dc640 1 -- 172.21.15.107:0/401171606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfac102320 msgr2=0x7fbfac19ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:47.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.255+0000 7fbfb45dc640 1 --2- 172.21.15.107:0/401171606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfac102320 0x7fbfac19ee90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fbfa400eaa0 tx=0x7fbfa400ef60 comp rx=0 tx=0).stop 2024-05-09T08:32:47.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.255+0000 7fbfb45dc640 1 -- 172.21.15.107:0/401171606 shutdown_connections 2024-05-09T08:32:47.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.255+0000 7fbfb45dc640 1 --2- 172.21.15.107:0/401171606 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbf980a2c10 0x7fbf980a50d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.255+0000 7fbfb45dc640 1 --2- 172.21.15.107:0/401171606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfac102320 0x7fbfac19ee90 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.255+0000 7fbfb45dc640 1 --2- 172.21.15.107:0/401171606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfac101960 0x7fbfac19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.255+0000 7fbfb45dc640 1 --2- 172.21.15.107:0/401171606 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfac100770 0x7fbfac069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:47.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.255+0000 7fbfb45dc640 1 -- 172.21.15.107:0/401171606 >> 172.21.15.107:0/401171606 conn(0x7fbfac0fbce0 msgr2=0x7fbfac104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:47.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.255+0000 7fbfb45dc640 1 -- 172.21.15.107:0/401171606 shutdown_connections 2024-05-09T08:32:47.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:47.255+0000 7fbfb45dc640 1 -- 172.21.15.107:0/401171606 wait complete. 2024-05-09T08:32:47.268 INFO:tasks.cephadm:Setup complete, yielding 2024-05-09T08:32:47.268 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-09T08:32:47.279 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-05-09T08:32:47.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- bash -c 'ceph orch status' 2024-05-09T08:32:47.526 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:47.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:47 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3525614236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:32:47.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:47 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3525614236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:32:47.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:47 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3525614236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:32:48.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:48 smithi157 ceph-mon[33614]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-09T08:32:48.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:48 smithi110 ceph-mon[33622]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-09T08:32:48.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:48 smithi107 ceph-mon[30122]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-09T08:32:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.410+0000 7f51fa03f700 1 -- 172.21.15.107:0/621719638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51f4101270 msgr2=0x7f51f4101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.410+0000 7f51fa03f700 1 --2- 172.21.15.107:0/621719638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51f4101270 0x7f51f4101690 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00b0d0 tx=0x7f51ec00b3e0 comp rx=0 tx=0).stop 2024-05-09T08:32:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.410+0000 7f51fa03f700 1 -- 172.21.15.107:0/621719638 shutdown_connections 2024-05-09T08:32:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.410+0000 7f51fa03f700 1 --2- 172.21.15.107:0/621719638 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51f4102e10 0x7f51f41096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.410+0000 7f51fa03f700 1 --2- 172.21.15.107:0/621719638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51f4101270 0x7f51f4101690 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.410+0000 7f51fa03f700 1 --2- 172.21.15.107:0/621719638 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51f4102470 0x7f51f41028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.410+0000 7f51fa03f700 1 -- 172.21.15.107:0/621719638 >> 172.21.15.107:0/621719638 conn(0x7f51f40fc810 msgr2=0x7f51f40fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.410+0000 7f51fa03f700 1 -- 172.21.15.107:0/621719638 shutdown_connections 2024-05-09T08:32:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.410+0000 7f51fa03f700 1 -- 172.21.15.107:0/621719638 wait complete. 2024-05-09T08:32:49.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.411+0000 7f51fa03f700 1 Processor -- start 2024-05-09T08:32:49.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.411+0000 7f51fa03f700 1 -- start start 2024-05-09T08:32:49.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.411+0000 7f51fa03f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51f4101270 0x7f51f4197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.411+0000 7f51fa03f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51f4102470 0x7f51f4197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.411+0000 7f51fa03f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51f4102e10 0x7f51f419e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.411+0000 7f51fa03f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f51f410cd60 con 0x7f51f4101270 2024-05-09T08:32:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.411+0000 7f51fa03f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f51f410cbe0 con 0x7f51f4102e10 2024-05-09T08:32:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.411+0000 7f51fa03f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f51f410cee0 con 0x7f51f4102470 2024-05-09T08:32:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.411+0000 7f51f37fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51f4101270 0x7f51f4197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.411+0000 7f51f37fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51f4101270 0x7f51f4197430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55810/0 (socket says 172.21.15.107:55810) 2024-05-09T08:32:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.411+0000 7f51f37fe700 1 -- 172.21.15.107:0/1257397903 learned_addr learned my addr 172.21.15.107:0/1257397903 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.412+0000 7f51f2ffd700 1 --2- 172.21.15.107:0/1257397903 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51f4102470 0x7f51f4197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.412+0000 7f51f3fff700 1 --2- 172.21.15.107:0/1257397903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51f4102e10 0x7f51f419e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.412+0000 7f51f37fe700 1 -- 172.21.15.107:0/1257397903 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51f4102470 msgr2=0x7f51f4197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.412+0000 7f51f37fe700 1 --2- 172.21.15.107:0/1257397903 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51f4102470 0x7f51f4197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.412+0000 7f51f37fe700 1 -- 172.21.15.107:0/1257397903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51f4102e10 msgr2=0x7f51f419e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.412+0000 7f51f37fe700 1 --2- 172.21.15.107:0/1257397903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51f4102e10 0x7f51f419e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.412+0000 7f51f37fe700 1 -- 172.21.15.107:0/1257397903 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f419f090 con 0x7f51f4101270 2024-05-09T08:32:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.412+0000 7f51f37fe700 1 --2- 172.21.15.107:0/1257397903 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51f4101270 0x7f51f4197430 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f51ec000f80 tx=0x7f51ec004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.412+0000 7f51f0ff9700 1 -- 172.21.15.107:0/1257397903 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec01f030 con 0x7f51f4101270 2024-05-09T08:32:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.412+0000 7f51fa03f700 1 -- 172.21.15.107:0/1257397903 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f419f320 con 0x7f51f4101270 2024-05-09T08:32:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.412+0000 7f51f0ff9700 1 -- 172.21.15.107:0/1257397903 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51ec007bc0 con 0x7f51f4101270 2024-05-09T08:32:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.412+0000 7f51f0ff9700 1 -- 172.21.15.107:0/1257397903 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0047a0 con 0x7f51f4101270 2024-05-09T08:32:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.412+0000 7f51fa03f700 1 -- 172.21.15.107:0/1257397903 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f419f7f0 con 0x7f51f4101270 2024-05-09T08:32:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.413+0000 7f51fa03f700 1 -- 172.21.15.107:0/1257397903 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51f4069010 con 0x7f51f4101270 2024-05-09T08:32:49.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.414+0000 7f51f0ff9700 1 -- 172.21.15.107:0/1257397903 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f51ec003a00 con 0x7f51f4101270 2024-05-09T08:32:49.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.415+0000 7f51f0ff9700 1 --2- 172.21.15.107:0/1257397903 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f51e00a2c20 0x7f51e00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.415+0000 7f51f2ffd700 1 --2- 172.21.15.107:0/1257397903 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f51e00a2c20 0x7f51e00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.415+0000 7f51f0ff9700 1 -- 172.21.15.107:0/1257397903 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f51ec00f020 con 0x7f51f4101270 2024-05-09T08:32:49.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.415+0000 7f51f2ffd700 1 --2- 172.21.15.107:0/1257397903 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f51e00a2c20 0x7f51e00a50e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009a10 tx=0x7f51dc009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:49.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.418+0000 7f51f0ff9700 1 -- 172.21.15.107:0/1257397903 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f51ec029020 con 0x7f51f4101270 2024-05-09T08:32:49.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.633+0000 7f51fa03f700 1 -- 172.21.15.107:0/1257397903 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f51f4062760 con 0x7f51e00a2c20 2024-05-09T08:32:49.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.634+0000 7f51f0ff9700 1 -- 172.21.15.107:0/1257397903 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f51f4062760 con 0x7f51e00a2c20 2024-05-09T08:32:49.634 INFO:teuthology.orchestra.run.smithi107.stdout:Backend: cephadm 2024-05-09T08:32:49.634 INFO:teuthology.orchestra.run.smithi107.stdout:Available: Yes 2024-05-09T08:32:49.634 INFO:teuthology.orchestra.run.smithi107.stdout:Paused: No 2024-05-09T08:32:49.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.636+0000 7f51fa03f700 1 -- 172.21.15.107:0/1257397903 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f51e00a2c20 msgr2=0x7f51e00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:49.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.636+0000 7f51fa03f700 1 --2- 172.21.15.107:0/1257397903 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f51e00a2c20 0x7f51e00a50e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009a10 tx=0x7f51dc009190 comp rx=0 tx=0).stop 2024-05-09T08:32:49.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.636+0000 7f51fa03f700 1 -- 172.21.15.107:0/1257397903 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51f4101270 msgr2=0x7f51f4197430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:49.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.636+0000 7f51fa03f700 1 --2- 172.21.15.107:0/1257397903 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51f4101270 0x7f51f4197430 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f51ec000f80 tx=0x7f51ec004e40 comp rx=0 tx=0).stop 2024-05-09T08:32:49.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.636+0000 7f51fa03f700 1 -- 172.21.15.107:0/1257397903 shutdown_connections 2024-05-09T08:32:49.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.636+0000 7f51fa03f700 1 --2- 172.21.15.107:0/1257397903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51f4102e10 0x7f51f419e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:49.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.636+0000 7f51fa03f700 1 --2- 172.21.15.107:0/1257397903 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f51e00a2c20 0x7f51e00a50e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:49.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.636+0000 7f51fa03f700 1 --2- 172.21.15.107:0/1257397903 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51f4101270 0x7f51f4197430 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:49.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.636+0000 7f51fa03f700 1 --2- 172.21.15.107:0/1257397903 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51f4102470 0x7f51f4197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:49.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.636+0000 7f51fa03f700 1 -- 172.21.15.107:0/1257397903 >> 172.21.15.107:0/1257397903 conn(0x7f51f40fc810 msgr2=0x7f51f41057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:49.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.636+0000 7f51fa03f700 1 -- 172.21.15.107:0/1257397903 shutdown_connections 2024-05-09T08:32:49.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:49.636+0000 7f51fa03f700 1 -- 172.21.15.107:0/1257397903 wait complete. 2024-05-09T08:32:50.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-09T08:32:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:50.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:50 smithi107 ceph-mon[30122]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T08:32:50.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:50 smithi157 ceph-mon[33614]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T08:32:50.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:50 smithi110 ceph-mon[33622]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T08:32:51.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:51 smithi157 ceph-mon[33614]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:32:51.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:51 smithi110 ceph-mon[33622]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:32:51.935 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:51 smithi107 ceph-mon[30122]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:32:52.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.385+0000 7efc63cd1700 1 -- 172.21.15.107:0/2335577211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc5c102480 msgr2=0x7efc5c1028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:52.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.385+0000 7efc63cd1700 1 --2- 172.21.15.107:0/2335577211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc5c102480 0x7efc5c1028e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7efc4c0099c0 tx=0x7efc4c009cd0 comp rx=0 tx=0).stop 2024-05-09T08:32:52.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.385+0000 7efc63cd1700 1 -- 172.21.15.107:0/2335577211 shutdown_connections 2024-05-09T08:32:52.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.385+0000 7efc63cd1700 1 --2- 172.21.15.107:0/2335577211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc5c102e20 0x7efc5c1096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:52.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.385+0000 7efc63cd1700 1 --2- 172.21.15.107:0/2335577211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc5c101280 0x7efc5c1016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:52.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.385+0000 7efc63cd1700 1 --2- 172.21.15.107:0/2335577211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc5c102480 0x7efc5c1028e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:52.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.385+0000 7efc63cd1700 1 -- 172.21.15.107:0/2335577211 >> 172.21.15.107:0/2335577211 conn(0x7efc5c0fc820 msgr2=0x7efc5c0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:52.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.385+0000 7efc63cd1700 1 -- 172.21.15.107:0/2335577211 shutdown_connections 2024-05-09T08:32:52.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.385+0000 7efc63cd1700 1 -- 172.21.15.107:0/2335577211 wait complete. 2024-05-09T08:32:52.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.386+0000 7efc63cd1700 1 Processor -- start 2024-05-09T08:32:52.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.386+0000 7efc63cd1700 1 -- start start 2024-05-09T08:32:52.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.386+0000 7efc63cd1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc5c101280 0x7efc5c198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:52.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.386+0000 7efc63cd1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc5c102480 0x7efc5c198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:52.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.386+0000 7efc63cd1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc5c102e20 0x7efc5c19fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:52.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.386+0000 7efc63cd1700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c10cd10 con 0x7efc5c102480 2024-05-09T08:32:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.386+0000 7efc63cd1700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c10cb90 con 0x7efc5c101280 2024-05-09T08:32:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.386+0000 7efc63cd1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c10ce90 con 0x7efc5c102e20 2024-05-09T08:32:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.386+0000 7efc6126c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc5c102480 0x7efc5c198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.386+0000 7efc6126c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc5c102480 0x7efc5c198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54580/0 (socket says 172.21.15.107:54580) 2024-05-09T08:32:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.386+0000 7efc6126c700 1 -- 172.21.15.107:0/3251157799 learned_addr learned my addr 172.21.15.107:0/3251157799 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.387+0000 7efc61a6d700 1 --2- 172.21.15.107:0/3251157799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc5c101280 0x7efc5c198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.387+0000 7efc6226e700 1 --2- 172.21.15.107:0/3251157799 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc5c102e20 0x7efc5c19fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.387+0000 7efc6126c700 1 -- 172.21.15.107:0/3251157799 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc5c102e20 msgr2=0x7efc5c19fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.387+0000 7efc6126c700 1 --2- 172.21.15.107:0/3251157799 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc5c102e20 0x7efc5c19fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.387+0000 7efc6126c700 1 -- 172.21.15.107:0/3251157799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc5c101280 msgr2=0x7efc5c198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.387+0000 7efc6126c700 1 --2- 172.21.15.107:0/3251157799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc5c101280 0x7efc5c198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.387+0000 7efc6126c700 1 -- 172.21.15.107:0/3251157799 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc5c1a01e0 con 0x7efc5c102480 2024-05-09T08:32:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.387+0000 7efc6126c700 1 --2- 172.21.15.107:0/3251157799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc5c102480 0x7efc5c198ac0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efc4c000c00 tx=0x7efc4c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.387+0000 7efc52ffd700 1 -- 172.21.15.107:0/3251157799 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4c027070 con 0x7efc5c102480 2024-05-09T08:32:52.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.387+0000 7efc52ffd700 1 -- 172.21.15.107:0/3251157799 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc4c003680 con 0x7efc5c102480 2024-05-09T08:32:52.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.387+0000 7efc63cd1700 1 -- 172.21.15.107:0/3251157799 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc5c1a0470 con 0x7efc5c102480 2024-05-09T08:32:52.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.387+0000 7efc52ffd700 1 -- 172.21.15.107:0/3251157799 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4c022c30 con 0x7efc5c102480 2024-05-09T08:32:52.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.387+0000 7efc63cd1700 1 -- 172.21.15.107:0/3251157799 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc5c1a08c0 con 0x7efc5c102480 2024-05-09T08:32:52.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.388+0000 7efc63cd1700 1 -- 172.21.15.107:0/3251157799 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc5c069010 con 0x7efc5c102480 2024-05-09T08:32:52.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.389+0000 7efc52ffd700 1 -- 172.21.15.107:0/3251157799 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7efc4c02a070 con 0x7efc5c102480 2024-05-09T08:32:52.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.390+0000 7efc52ffd700 1 --2- 172.21.15.107:0/3251157799 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efc480a2c20 0x7efc480a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:52.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.390+0000 7efc61a6d700 1 --2- 172.21.15.107:0/3251157799 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efc480a2c20 0x7efc480a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:52.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.390+0000 7efc52ffd700 1 -- 172.21.15.107:0/3251157799 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7efc4c0d06b0 con 0x7efc5c102480 2024-05-09T08:32:52.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.390+0000 7efc61a6d700 1 --2- 172.21.15.107:0/3251157799 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efc480a2c20 0x7efc480a50e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efc5c1022e0 tx=0x7efc5800a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:52.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.393+0000 7efc52ffd700 1 -- 172.21.15.107:0/3251157799 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efc4c031070 con 0x7efc5c102480 2024-05-09T08:32:52.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.598+0000 7efc63cd1700 1 -- 172.21.15.107:0/3251157799 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7efc5c0fea70 con 0x7efc480a2c20 2024-05-09T08:32:52.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.611+0000 7efc52ffd700 1 -- 172.21.15.107:0/3251157799 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7efc5c0fea70 con 0x7efc480a2c20 2024-05-09T08:32:52.611 INFO:teuthology.orchestra.run.smithi107.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-09T08:32:52.611 INFO:teuthology.orchestra.run.smithi107.stdout:mds.a smithi107 running (2m) 20s ago 2m 15.2M - 19.0.0-2515-g96805dcf d7acac7d5657 de4ae376929b 2024-05-09T08:32:52.611 INFO:teuthology.orchestra.run.smithi107.stdout:mds.b smithi110 running (118s) 51s ago 117s 19.8M - 19.0.0-2515-g96805dcf d7acac7d5657 100c0de81456 2024-05-09T08:32:52.611 INFO:teuthology.orchestra.run.smithi107.stdout:mds.c smithi157 running (2m) 51s ago 2m 19.9M - 19.0.0-2515-g96805dcf d7acac7d5657 02e19a76ad1c 2024-05-09T08:32:52.611 INFO:teuthology.orchestra.run.smithi107.stdout:mds.d smithi107 running (2m) 20s ago 2m 16.2M - 19.0.0-2515-g96805dcf d7acac7d5657 8dc3fa32b084 2024-05-09T08:32:52.611 INFO:teuthology.orchestra.run.smithi107.stdout:mds.e smithi110 running (2m) 51s ago 2m 16.0M - 19.0.0-2515-g96805dcf d7acac7d5657 388cf819a554 2024-05-09T08:32:52.611 INFO:teuthology.orchestra.run.smithi107.stdout:mds.f smithi157 running (2m) 51s ago 2m 17.5M - 19.0.0-2515-g96805dcf d7acac7d5657 9a5a8cc830f2 2024-05-09T08:32:52.611 INFO:teuthology.orchestra.run.smithi107.stdout:mds.g smithi107 running (2m) 20s ago 2m 13.9M - 19.0.0-2515-g96805dcf d7acac7d5657 a3f4774b0a1b 2024-05-09T08:32:52.611 INFO:teuthology.orchestra.run.smithi107.stdout:mds.h smithi110 running (2m) 51s ago 2m 16.2M - 19.0.0-2515-g96805dcf d7acac7d5657 a262e31f2321 2024-05-09T08:32:52.611 INFO:teuthology.orchestra.run.smithi107.stdout:mds.i smithi157 running (2m) 51s ago 2m 26.2M - 19.0.0-2515-g96805dcf d7acac7d5657 60aa0a0934a4 2024-05-09T08:32:52.611 INFO:teuthology.orchestra.run.smithi107.stdout:mds.j smithi107 running (2m) 20s ago 2m 13.7M - 19.0.0-2515-g96805dcf d7acac7d5657 908a53818100 2024-05-09T08:32:52.611 INFO:teuthology.orchestra.run.smithi107.stdout:mds.k smithi110 running (2m) 51s ago 2m 14.1M - 19.0.0-2515-g96805dcf d7acac7d5657 d83b79ffa267 2024-05-09T08:32:52.611 INFO:teuthology.orchestra.run.smithi107.stdout:mds.l smithi157 running (2m) 51s ago 2m 14.0M - 19.0.0-2515-g96805dcf d7acac7d5657 c7aa358e7a80 2024-05-09T08:32:52.612 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.x smithi107 *:9283,8765 running (9m) 20s ago 9m 515M - 19.0.0-2515-g96805dcf d7acac7d5657 a3dc9c9b852e 2024-05-09T08:32:52.612 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.y smithi110 *:8443,8765 running (7m) 51s ago 7m 455M - 19.0.0-2515-g96805dcf d7acac7d5657 4edfd29d7064 2024-05-09T08:32:52.612 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.z smithi157 *:8443,8765 running (7m) 51s ago 7m 452M - 19.0.0-2515-g96805dcf d7acac7d5657 e1a8727f5c6d 2024-05-09T08:32:52.612 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a smithi107 running (9m) 20s ago 9m 52.5M 2048M 19.0.0-2515-g96805dcf d7acac7d5657 754b572e3f5f 2024-05-09T08:32:52.612 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b smithi110 running (8m) 51s ago 8m 39.2M 2048M 19.0.0-2515-g96805dcf d7acac7d5657 ca5207af62f5 2024-05-09T08:32:52.612 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c smithi157 running (8m) 51s ago 8m 38.5M 2048M 19.0.0-2515-g96805dcf d7acac7d5657 7cd67865c9ab 2024-05-09T08:32:52.612 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 smithi107 running (7m) 20s ago 7m 55.4M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 79add31993e6 2024-05-09T08:32:52.612 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 smithi110 running (7m) 51s ago 7m 45.9M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 051155b4c6ea 2024-05-09T08:32:52.612 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 smithi157 running (6m) 51s ago 6m 54.7M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 d91f6b187209 2024-05-09T08:32:52.612 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 smithi107 running (6m) 20s ago 6m 52.5M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 7b72a0dfa6d2 2024-05-09T08:32:52.612 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 smithi110 running (5m) 51s ago 5m 50.0M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 5fa7ede6cd98 2024-05-09T08:32:52.612 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 smithi157 running (5m) 51s ago 5m 52.3M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 9ba4a45ed654 2024-05-09T08:32:52.612 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 smithi107 running (4m) 20s ago 4m 42.8M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 966083d185f1 2024-05-09T08:32:52.612 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 smithi110 running (4m) 51s ago 4m 57.5M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 225cf89c2732 2024-05-09T08:32:52.613 INFO:teuthology.orchestra.run.smithi107.stdout:osd.8 smithi157 running (4m) 51s ago 4m 52.6M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 6de81f2e9117 2024-05-09T08:32:52.613 INFO:teuthology.orchestra.run.smithi107.stdout:osd.9 smithi107 running (3m) 20s ago 3m 42.5M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 ae8968d530d6 2024-05-09T08:32:52.613 INFO:teuthology.orchestra.run.smithi107.stdout:osd.10 smithi110 running (3m) 51s ago 3m 43.9M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 8f3dfac6caed 2024-05-09T08:32:52.613 INFO:teuthology.orchestra.run.smithi107.stdout:osd.11 smithi157 running (2m) 51s ago 2m 40.7M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 afafdccd53bb 2024-05-09T08:32:52.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.614+0000 7efc63cd1700 1 -- 172.21.15.107:0/3251157799 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efc480a2c20 msgr2=0x7efc480a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:52.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.614+0000 7efc63cd1700 1 --2- 172.21.15.107:0/3251157799 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efc480a2c20 0x7efc480a50e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efc5c1022e0 tx=0x7efc5800a380 comp rx=0 tx=0).stop 2024-05-09T08:32:52.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.614+0000 7efc63cd1700 1 -- 172.21.15.107:0/3251157799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc5c102480 msgr2=0x7efc5c198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:52.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.614+0000 7efc63cd1700 1 --2- 172.21.15.107:0/3251157799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc5c102480 0x7efc5c198ac0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efc4c000c00 tx=0x7efc4c0047b0 comp rx=0 tx=0).stop 2024-05-09T08:32:52.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.614+0000 7efc63cd1700 1 -- 172.21.15.107:0/3251157799 shutdown_connections 2024-05-09T08:32:52.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.614+0000 7efc63cd1700 1 --2- 172.21.15.107:0/3251157799 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc5c102e20 0x7efc5c19fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:52.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.614+0000 7efc63cd1700 1 --2- 172.21.15.107:0/3251157799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc5c101280 0x7efc5c198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:52.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.614+0000 7efc63cd1700 1 --2- 172.21.15.107:0/3251157799 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7efc480a2c20 0x7efc480a50e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:52.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.614+0000 7efc63cd1700 1 --2- 172.21.15.107:0/3251157799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc5c102480 0x7efc5c198ac0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:52.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.614+0000 7efc63cd1700 1 -- 172.21.15.107:0/3251157799 >> 172.21.15.107:0/3251157799 conn(0x7efc5c0fc820 msgr2=0x7efc5c0fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:52.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.614+0000 7efc63cd1700 1 -- 172.21.15.107:0/3251157799 shutdown_connections 2024-05-09T08:32:52.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:52.614+0000 7efc63cd1700 1 -- 172.21.15.107:0/3251157799 wait complete. 2024-05-09T08:32:52.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:52 smithi157 ceph-mon[33614]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-09T08:32:52.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:52 smithi110 ceph-mon[33622]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-09T08:32:52.935 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:52 smithi107 ceph-mon[30122]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-09T08:32:53.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-09T08:32:53.613 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:53.735 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:53 smithi107 ceph-mon[30122]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:32:53.735 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:53 smithi107 ceph-mon[30122]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:32:53.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:53 smithi157 ceph-mon[33614]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:32:53.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:53 smithi157 ceph-mon[33614]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:32:53.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:53 smithi110 ceph-mon[33622]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:32:53.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:53 smithi110 ceph-mon[33622]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:32:55.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.311+0000 7fb5b4101700 1 -- 172.21.15.107:0/2349908665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5ac102e20 msgr2=0x7fb5ac1096d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:55.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.311+0000 7fb5b4101700 1 --2- 172.21.15.107:0/2349908665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5ac102e20 0x7fb5ac1096d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400b0d0 tx=0x7fb5a400b3e0 comp rx=0 tx=0).stop 2024-05-09T08:32:55.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.311+0000 7fb5b4101700 1 -- 172.21.15.107:0/2349908665 shutdown_connections 2024-05-09T08:32:55.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.311+0000 7fb5b4101700 1 --2- 172.21.15.107:0/2349908665 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5ac102e20 0x7fb5ac1096d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:55.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.311+0000 7fb5b4101700 1 --2- 172.21.15.107:0/2349908665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5ac101280 0x7fb5ac1016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:55.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.311+0000 7fb5b4101700 1 --2- 172.21.15.107:0/2349908665 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5ac102480 0x7fb5ac1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:55.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.311+0000 7fb5b4101700 1 -- 172.21.15.107:0/2349908665 >> 172.21.15.107:0/2349908665 conn(0x7fb5ac0fc820 msgr2=0x7fb5ac0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:55.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.311+0000 7fb5b4101700 1 -- 172.21.15.107:0/2349908665 shutdown_connections 2024-05-09T08:32:55.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.311+0000 7fb5b4101700 1 -- 172.21.15.107:0/2349908665 wait complete. 2024-05-09T08:32:55.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.311+0000 7fb5b4101700 1 Processor -- start 2024-05-09T08:32:55.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b4101700 1 -- start start 2024-05-09T08:32:55.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b4101700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5ac101280 0x7fb5ac198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:55.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b4101700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5ac102480 0x7fb5ac198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:55.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b4101700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5ac102e20 0x7fb5ac19faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:55.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b4101700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac10cd70 con 0x7fb5ac101280 2024-05-09T08:32:55.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b4101700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac10cbf0 con 0x7fb5ac102480 2024-05-09T08:32:55.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b4101700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac10cef0 con 0x7fb5ac102e20 2024-05-09T08:32:55.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b1e9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5ac101280 0x7fb5ac198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:55.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b1e9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5ac101280 0x7fb5ac198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54608/0 (socket says 172.21.15.107:54608) 2024-05-09T08:32:55.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b1e9d700 1 -- 172.21.15.107:0/4197703761 learned_addr learned my addr 172.21.15.107:0/4197703761 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:55.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b269e700 1 --2- 172.21.15.107:0/4197703761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5ac102e20 0x7fb5ac19faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:55.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b169c700 1 --2- 172.21.15.107:0/4197703761 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5ac102480 0x7fb5ac198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:55.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b1e9d700 1 -- 172.21.15.107:0/4197703761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5ac102e20 msgr2=0x7fb5ac19faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:55.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b1e9d700 1 --2- 172.21.15.107:0/4197703761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5ac102e20 0x7fb5ac19faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:55.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b1e9d700 1 -- 172.21.15.107:0/4197703761 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5ac102480 msgr2=0x7fb5ac198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:55.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b1e9d700 1 --2- 172.21.15.107:0/4197703761 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5ac102480 0x7fb5ac198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:55.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b1e9d700 1 -- 172.21.15.107:0/4197703761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac1a0190 con 0x7fb5ac101280 2024-05-09T08:32:55.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b269e700 1 --2- 172.21.15.107:0/4197703761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5ac102e20 0x7fb5ac19faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:32:55.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b169c700 1 --2- 172.21.15.107:0/4197703761 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5ac102480 0x7fb5ac198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:32:55.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.312+0000 7fb5b1e9d700 1 --2- 172.21.15.107:0/4197703761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5ac101280 0x7fb5ac198530 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800c9a0 tx=0x7fb5a800cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:55.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.313+0000 7fb5a2ffd700 1 -- 172.21.15.107:0/4197703761 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8004220 con 0x7fb5ac101280 2024-05-09T08:32:55.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.313+0000 7fb5a2ffd700 1 -- 172.21.15.107:0/4197703761 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb5a8007d10 con 0x7fb5ac101280 2024-05-09T08:32:55.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.313+0000 7fb5a2ffd700 1 -- 172.21.15.107:0/4197703761 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8003e10 con 0x7fb5ac101280 2024-05-09T08:32:55.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.313+0000 7fb5b4101700 1 -- 172.21.15.107:0/4197703761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5ac1a0480 con 0x7fb5ac101280 2024-05-09T08:32:55.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.313+0000 7fb5b4101700 1 -- 172.21.15.107:0/4197703761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5ac1a09f0 con 0x7fb5ac101280 2024-05-09T08:32:55.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.313+0000 7fb5b4101700 1 -- 172.21.15.107:0/4197703761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5ac069010 con 0x7fb5ac101280 2024-05-09T08:32:55.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.315+0000 7fb5a2ffd700 1 -- 172.21.15.107:0/4197703761 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fb5a8007860 con 0x7fb5ac101280 2024-05-09T08:32:55.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.315+0000 7fb5a2ffd700 1 --2- 172.21.15.107:0/4197703761 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb5980a2ca0 0x7fb5980a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:55.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.316+0000 7fb5b169c700 1 --2- 172.21.15.107:0/4197703761 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb5980a2ca0 0x7fb5980a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:55.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.316+0000 7fb5a2ffd700 1 -- 172.21.15.107:0/4197703761 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb5a80cf2f0 con 0x7fb5ac101280 2024-05-09T08:32:55.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.316+0000 7fb5b169c700 1 --2- 172.21.15.107:0/4197703761 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb5980a2ca0 0x7fb5980a5160 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb59c007900 tx=0x7fb59c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:55.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.319+0000 7fb5a2ffd700 1 -- 172.21.15.107:0/4197703761 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb5a8014030 con 0x7fb5ac101280 2024-05-09T08:32:55.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.531+0000 7fb5b4101700 1 -- 172.21.15.107:0/4197703761 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fb5ac0fea40 con 0x7fb5980a2ca0 2024-05-09T08:32:55.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.537+0000 7fb5a2ffd700 1 -- 172.21.15.107:0/4197703761 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fb5ac0fea40 con 0x7fb5980a2ca0 2024-05-09T08:32:55.536 INFO:teuthology.orchestra.run.smithi107.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-09T08:32:55.536 INFO:teuthology.orchestra.run.smithi107.stdout:mds.all 12/12 54s ago 2m smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi157=c;smithi157=f;smithi157=i;smithi157=l;count:12 2024-05-09T08:32:55.536 INFO:teuthology.orchestra.run.smithi107.stdout:mgr 3/3 54s ago 7m smithi107=x;smithi110=y;smithi157=z;count:3 2024-05-09T08:32:55.536 INFO:teuthology.orchestra.run.smithi107.stdout:mon 3/3 54s ago 8m smithi107:172.21.15.107=a;smithi110:172.21.15.110=b;smithi157:172.21.15.157=c;count:3 2024-05-09T08:32:55.536 INFO:teuthology.orchestra.run.smithi107.stdout:osd 12 54s ago - 2024-05-09T08:32:55.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.539+0000 7fb5b4101700 1 -- 172.21.15.107:0/4197703761 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb5980a2ca0 msgr2=0x7fb5980a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:55.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.539+0000 7fb5b4101700 1 --2- 172.21.15.107:0/4197703761 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb5980a2ca0 0x7fb5980a5160 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb59c007900 tx=0x7fb59c008040 comp rx=0 tx=0).stop 2024-05-09T08:32:55.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.539+0000 7fb5b4101700 1 -- 172.21.15.107:0/4197703761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5ac101280 msgr2=0x7fb5ac198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:55.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.539+0000 7fb5b4101700 1 --2- 172.21.15.107:0/4197703761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5ac101280 0x7fb5ac198530 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800c9a0 tx=0x7fb5a800cd60 comp rx=0 tx=0).stop 2024-05-09T08:32:55.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.539+0000 7fb5b4101700 1 -- 172.21.15.107:0/4197703761 shutdown_connections 2024-05-09T08:32:55.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.539+0000 7fb5b4101700 1 --2- 172.21.15.107:0/4197703761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5ac102e20 0x7fb5ac19faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:55.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.539+0000 7fb5b4101700 1 --2- 172.21.15.107:0/4197703761 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fb5980a2ca0 0x7fb5980a5160 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:55.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.539+0000 7fb5b4101700 1 --2- 172.21.15.107:0/4197703761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5ac101280 0x7fb5ac198530 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:55.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.539+0000 7fb5b4101700 1 --2- 172.21.15.107:0/4197703761 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5ac102480 0x7fb5ac198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:55.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.539+0000 7fb5b4101700 1 -- 172.21.15.107:0/4197703761 >> 172.21.15.107:0/4197703761 conn(0x7fb5ac0fc820 msgr2=0x7fb5ac0fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:55.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.539+0000 7fb5b4101700 1 -- 172.21.15.107:0/4197703761 shutdown_connections 2024-05-09T08:32:55.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:55.539+0000 7fb5b4101700 1 -- 172.21.15.107:0/4197703761 wait complete. 2024-05-09T08:32:56.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-09T08:32:56.595 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:32:56.718 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:56 smithi107 ceph-mon[30122]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-09T08:32:56.718 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:56 smithi107 ceph-mon[30122]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:32:56.718 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:56 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-05-09T08:32:56.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:56 smithi157 ceph-mon[33614]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-09T08:32:56.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:56 smithi157 ceph-mon[33614]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:32:56.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:56 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-05-09T08:32:56.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:56 smithi110 ceph-mon[33622]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-09T08:32:56.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:56 smithi110 ceph-mon[33622]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:32:56.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:56 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-05-09T08:32:57.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:57 smithi107 ceph-mon[30122]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-05-09T08:32:57.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:57 smithi107 ceph-mon[30122]: osdmap e92: 12 total, 12 up, 12 in 2024-05-09T08:32:57.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:57 smithi157 ceph-mon[33614]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-05-09T08:32:57.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:57 smithi157 ceph-mon[33614]: osdmap e92: 12 total, 12 up, 12 in 2024-05-09T08:32:57.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:57 smithi110 ceph-mon[33622]: from='mgr.14150 172.21.15.107:0/1604318167' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-05-09T08:32:57.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:57 smithi110 ceph-mon[33622]: osdmap e92: 12 total, 12 up, 12 in 2024-05-09T08:32:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.474+0000 7f9fd6286700 1 -- 172.21.15.107:0/2428787607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fd0102380 msgr2=0x7f9fd01027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.474+0000 7f9fd6286700 1 --2- 172.21.15.107:0/2428787607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fd0102380 0x7f9fd01027a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00099c0 tx=0x7f9fc0009cd0 comp rx=0 tx=0).stop 2024-05-09T08:32:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.474+0000 7f9fd6286700 1 -- 172.21.15.107:0/2428787607 shutdown_connections 2024-05-09T08:32:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.474+0000 7f9fd6286700 1 --2- 172.21.15.107:0/2428787607 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fd0103f20 0x7f9fd010a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.474+0000 7f9fd6286700 1 --2- 172.21.15.107:0/2428787607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fd0102380 0x7f9fd01027a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.474+0000 7f9fd6286700 1 --2- 172.21.15.107:0/2428787607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9fd0103580 0x7f9fd01039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.474+0000 7f9fd6286700 1 -- 172.21.15.107:0/2428787607 >> 172.21.15.107:0/2428787607 conn(0x7f9fd00fd920 msgr2=0x7f9fd00ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.474+0000 7f9fd6286700 1 -- 172.21.15.107:0/2428787607 shutdown_connections 2024-05-09T08:32:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.474+0000 7f9fd6286700 1 -- 172.21.15.107:0/2428787607 wait complete. 2024-05-09T08:32:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.474+0000 7f9fd6286700 1 Processor -- start 2024-05-09T08:32:58.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.474+0000 7f9fd6286700 1 -- start start 2024-05-09T08:32:58.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fd6286700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fd0102380 0x7f9fd01984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:58.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fd6286700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9fd0103580 0x7f9fd01989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:58.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fd6286700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fd0103f20 0x7f9fd019fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:58.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fd6286700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd010ddf0 con 0x7f9fd0102380 2024-05-09T08:32:58.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fd6286700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd010dc70 con 0x7f9fd0103580 2024-05-09T08:32:58.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fd6286700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd010df70 con 0x7f9fd0103f20 2024-05-09T08:32:58.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fcffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fd0102380 0x7f9fd01984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:58.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fcffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fd0102380 0x7f9fd01984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54630/0 (socket says 172.21.15.107:54630) 2024-05-09T08:32:58.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fcffff700 1 -- 172.21.15.107:0/2139879212 learned_addr learned my addr 172.21.15.107:0/2139879212 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:32:58.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fcffff700 1 -- 172.21.15.107:0/2139879212 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fd0103f20 msgr2=0x7f9fd019fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:58.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fcffff700 1 --2- 172.21.15.107:0/2139879212 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fd0103f20 0x7f9fd019fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:58.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fcffff700 1 -- 172.21.15.107:0/2139879212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9fd0103580 msgr2=0x7f9fd01989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:58.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fcffff700 1 --2- 172.21.15.107:0/2139879212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9fd0103580 0x7f9fd01989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:58.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fcffff700 1 -- 172.21.15.107:0/2139879212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd01a0110 con 0x7f9fd0102380 2024-05-09T08:32:58.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fcffff700 1 --2- 172.21.15.107:0/2139879212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fd0102380 0x7f9fd01984b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0000c00 tx=0x7f9fc0009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:58.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fcdffb700 1 -- 172.21.15.107:0/2139879212 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0027070 con 0x7f9fd0102380 2024-05-09T08:32:58.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fd6286700 1 -- 172.21.15.107:0/2139879212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fd01a03a0 con 0x7f9fd0102380 2024-05-09T08:32:58.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fcdffb700 1 -- 172.21.15.107:0/2139879212 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9fc0004050 con 0x7f9fd0102380 2024-05-09T08:32:58.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fcdffb700 1 -- 172.21.15.107:0/2139879212 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0022e40 con 0x7f9fd0102380 2024-05-09T08:32:58.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.475+0000 7f9fd6286700 1 -- 172.21.15.107:0/2139879212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fd01a07f0 con 0x7f9fd0102380 2024-05-09T08:32:58.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.476+0000 7f9fd6286700 1 -- 172.21.15.107:0/2139879212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fd002db80 con 0x7f9fd0102380 2024-05-09T08:32:58.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.477+0000 7f9fcdffb700 1 -- 172.21.15.107:0/2139879212 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f9fc002a070 con 0x7f9fd0102380 2024-05-09T08:32:58.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.478+0000 7f9fcdffb700 1 --2- 172.21.15.107:0/2139879212 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9fbc0a2c20 0x7f9fbc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:32:58.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.478+0000 7f9fc7fff700 1 --2- 172.21.15.107:0/2139879212 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9fbc0a2c20 0x7f9fbc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:32:58.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.478+0000 7f9fcdffb700 1 -- 172.21.15.107:0/2139879212 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f9fc00d01c0 con 0x7f9fd0102380 2024-05-09T08:32:58.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.479+0000 7f9fc7fff700 1 --2- 172.21.15.107:0/2139879212 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9fbc0a2c20 0x7f9fbc0a50e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8005950 tx=0x7f9fb80058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:32:58.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.483+0000 7f9fcdffb700 1 -- 172.21.15.107:0/2139879212 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9fc0013070 con 0x7f9fd0102380 2024-05-09T08:32:58.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.684+0000 7f9fd6286700 1 -- 172.21.15.107:0/2139879212 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f9fd0062760 con 0x7f9fbc0a2c20 2024-05-09T08:32:58.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:58 smithi107 ceph-mon[30122]: pgmap v353: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-09T08:32:58.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:32:58 smithi107 ceph-mon[30122]: osdmap e93: 12 total, 12 up, 12 in 2024-05-09T08:32:58.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.685+0000 7f9fcdffb700 1 -- 172.21.15.107:0/2139879212 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f9fd0062760 con 0x7f9fbc0a2c20 2024-05-09T08:32:58.684 INFO:teuthology.orchestra.run.smithi107.stdout:HOST ADDR LABELS STATUS 2024-05-09T08:32:58.684 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 172.21.15.107 2024-05-09T08:32:58.684 INFO:teuthology.orchestra.run.smithi107.stdout:smithi110 172.21.15.110 2024-05-09T08:32:58.684 INFO:teuthology.orchestra.run.smithi107.stdout:smithi157 172.21.15.157 2024-05-09T08:32:58.685 INFO:teuthology.orchestra.run.smithi107.stdout:3 hosts in cluster 2024-05-09T08:32:58.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.687+0000 7f9fd6286700 1 -- 172.21.15.107:0/2139879212 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9fbc0a2c20 msgr2=0x7f9fbc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:58.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.687+0000 7f9fd6286700 1 --2- 172.21.15.107:0/2139879212 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9fbc0a2c20 0x7f9fbc0a50e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8005950 tx=0x7f9fb80058e0 comp rx=0 tx=0).stop 2024-05-09T08:32:58.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.687+0000 7f9fd6286700 1 -- 172.21.15.107:0/2139879212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fd0102380 msgr2=0x7f9fd01984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:32:58.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.687+0000 7f9fd6286700 1 --2- 172.21.15.107:0/2139879212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fd0102380 0x7f9fd01984b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0000c00 tx=0x7f9fc0009f10 comp rx=0 tx=0).stop 2024-05-09T08:32:58.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.687+0000 7f9fd6286700 1 -- 172.21.15.107:0/2139879212 shutdown_connections 2024-05-09T08:32:58.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.687+0000 7f9fd6286700 1 --2- 172.21.15.107:0/2139879212 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fd0103f20 0x7f9fd019fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:58.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.687+0000 7f9fd6286700 1 --2- 172.21.15.107:0/2139879212 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f9fbc0a2c20 0x7f9fbc0a50e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:58.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.687+0000 7f9fd6286700 1 --2- 172.21.15.107:0/2139879212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fd0102380 0x7f9fd01984b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:58.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.687+0000 7f9fd6286700 1 --2- 172.21.15.107:0/2139879212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9fd0103580 0x7f9fd01989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:32:58.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.687+0000 7f9fd6286700 1 -- 172.21.15.107:0/2139879212 >> 172.21.15.107:0/2139879212 conn(0x7f9fd00fd920 msgr2=0x7f9fd0106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:32:58.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.687+0000 7f9fd6286700 1 -- 172.21.15.107:0/2139879212 shutdown_connections 2024-05-09T08:32:58.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:32:58.687+0000 7f9fd6286700 1 -- 172.21.15.107:0/2139879212 wait complete. 2024-05-09T08:32:58.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:58 smithi157 ceph-mon[33614]: pgmap v353: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-09T08:32:58.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:32:58 smithi157 ceph-mon[33614]: osdmap e93: 12 total, 12 up, 12 in 2024-05-09T08:32:58.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:58 smithi110 ceph-mon[33622]: pgmap v353: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-09T08:32:58.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:32:58 smithi110 ceph-mon[33622]: osdmap e93: 12 total, 12 up, 12 in 2024-05-09T08:32:59.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-09T08:32:59.771 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:33:00.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:00 smithi157 ceph-mon[33614]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:00.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:00 smithi157 ceph-mon[33614]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-05-09T08:33:00.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:00 smithi110 ceph-mon[33622]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:00.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:00 smithi110 ceph-mon[33622]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-05-09T08:33:00.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:00 smithi107 ceph-mon[30122]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:00.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:00 smithi107 ceph-mon[30122]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-05-09T08:33:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.642+0000 7fe19278d700 1 -- 172.21.15.107:0/4163029689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c1023b0 msgr2=0x7fe18c1027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.642+0000 7fe19278d700 1 --2- 172.21.15.107:0/4163029689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c1023b0 0x7fe18c1027d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe17c0099c0 tx=0x7fe17c009cd0 comp rx=0 tx=0).stop 2024-05-09T08:33:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.642+0000 7fe19278d700 1 -- 172.21.15.107:0/4163029689 shutdown_connections 2024-05-09T08:33:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.642+0000 7fe19278d700 1 --2- 172.21.15.107:0/4163029689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe18c103f50 0x7fe18c10a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.642+0000 7fe19278d700 1 --2- 172.21.15.107:0/4163029689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c1023b0 0x7fe18c1027d0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.642+0000 7fe19278d700 1 --2- 172.21.15.107:0/4163029689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe18c1035b0 0x7fe18c103a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.642+0000 7fe19278d700 1 -- 172.21.15.107:0/4163029689 >> 172.21.15.107:0/4163029689 conn(0x7fe18c0fd930 msgr2=0x7fe18c0ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.642+0000 7fe19278d700 1 -- 172.21.15.107:0/4163029689 shutdown_connections 2024-05-09T08:33:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.642+0000 7fe19278d700 1 -- 172.21.15.107:0/4163029689 wait complete. 2024-05-09T08:33:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.642+0000 7fe19278d700 1 Processor -- start 2024-05-09T08:33:01.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.642+0000 7fe19278d700 1 -- start start 2024-05-09T08:33:01.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe19278d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe18c1023b0 0x7fe18c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:01.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe19278d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c1035b0 0x7fe18c198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:01.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe19278d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe18c103f50 0x7fe18c19fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:01.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe19278d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c10dea0 con 0x7fe18c1035b0 2024-05-09T08:33:01.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe19278d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c10dd20 con 0x7fe18c103f50 2024-05-09T08:33:01.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe19278d700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c10e020 con 0x7fe18c1023b0 2024-05-09T08:33:01.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe18b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c1035b0 0x7fe18c198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:01.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe18b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c1035b0 0x7fe18c198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38266/0 (socket says 172.21.15.107:38266) 2024-05-09T08:33:01.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe18b7fe700 1 -- 172.21.15.107:0/327868000 learned_addr learned my addr 172.21.15.107:0/327868000 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:01.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe18bfff700 1 --2- 172.21.15.107:0/327868000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe18c1023b0 0x7fe18c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:01.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe18b7fe700 1 -- 172.21.15.107:0/327868000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe18c1023b0 msgr2=0x7fe18c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:01.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe18b7fe700 1 --2- 172.21.15.107:0/327868000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe18c1023b0 0x7fe18c198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:01.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe18b7fe700 1 -- 172.21.15.107:0/327868000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe18c103f50 msgr2=0x7fe18c19fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe190d2a700 1 --2- 172.21.15.107:0/327868000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe18c103f50 0x7fe18c19fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe18b7fe700 1 --2- 172.21.15.107:0/327868000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe18c103f50 0x7fe18c19fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe18b7fe700 1 -- 172.21.15.107:0/327868000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18c1a0180 con 0x7fe18c1035b0 2024-05-09T08:33:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe18b7fe700 1 --2- 172.21.15.107:0/327868000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c1035b0 0x7fe18c198ab0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe18000e9d0 tx=0x7fe18000ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe1897fa700 1 -- 172.21.15.107:0/327868000 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18000cbd0 con 0x7fe18c1035b0 2024-05-09T08:33:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe19278d700 1 -- 172.21.15.107:0/327868000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe18c1a0470 con 0x7fe18c1035b0 2024-05-09T08:33:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe1897fa700 1 -- 172.21.15.107:0/327868000 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe18000cd70 con 0x7fe18c1035b0 2024-05-09T08:33:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe1897fa700 1 -- 172.21.15.107:0/327868000 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe180018710 con 0x7fe18c1035b0 2024-05-09T08:33:01.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.643+0000 7fe19278d700 1 -- 172.21.15.107:0/327868000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe18c1a0a80 con 0x7fe18c1035b0 2024-05-09T08:33:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.644+0000 7fe19278d700 1 -- 172.21.15.107:0/327868000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe18c069010 con 0x7fe18c1035b0 2024-05-09T08:33:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.646+0000 7fe1897fa700 1 -- 172.21.15.107:0/327868000 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fe180010430 con 0x7fe18c1035b0 2024-05-09T08:33:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.646+0000 7fe1897fa700 1 --2- 172.21.15.107:0/327868000 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe1740a2c20 0x7fe1740a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.646+0000 7fe18bfff700 1 --2- 172.21.15.107:0/327868000 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe1740a2c20 0x7fe1740a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.646+0000 7fe1897fa700 1 -- 172.21.15.107:0/327868000 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fe1800cfb90 con 0x7fe18c1035b0 2024-05-09T08:33:01.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.647+0000 7fe18bfff700 1 --2- 172.21.15.107:0/327868000 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe1740a2c20 0x7fe1740a50e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00b580 tx=0x7fe17c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:01.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.650+0000 7fe1897fa700 1 -- 172.21.15.107:0/327868000 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe18009c770 con 0x7fe18c1035b0 2024-05-09T08:33:01.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.852+0000 7fe19278d700 1 -- 172.21.15.107:0/327868000 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fe18c062760 con 0x7fe1740a2c20 2024-05-09T08:33:01.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.854+0000 7fe1897fa700 1 -- 172.21.15.107:0/327868000 <== mgr.14150 v2:172.21.15.107:6800/846948416 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fe18c062760 con 0x7fe1740a2c20 2024-05-09T08:33:01.853 INFO:teuthology.orchestra.run.smithi107.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-09T08:33:01.853 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NK400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-05-09T08:33:01.853 INFO:teuthology.orchestra.run.smithi107.stdout:smithi110 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476002T480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-05-09T08:33:01.853 INFO:teuthology.orchestra.run.smithi107.stdout:smithi157 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476002V480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-05-09T08:33:01.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.856+0000 7fe19278d700 1 -- 172.21.15.107:0/327868000 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe1740a2c20 msgr2=0x7fe1740a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:01.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.856+0000 7fe19278d700 1 --2- 172.21.15.107:0/327868000 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe1740a2c20 0x7fe1740a50e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00b580 tx=0x7fe17c005fb0 comp rx=0 tx=0).stop 2024-05-09T08:33:01.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.856+0000 7fe19278d700 1 -- 172.21.15.107:0/327868000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c1035b0 msgr2=0x7fe18c198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:01.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.856+0000 7fe19278d700 1 --2- 172.21.15.107:0/327868000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c1035b0 0x7fe18c198ab0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe18000e9d0 tx=0x7fe18000ed90 comp rx=0 tx=0).stop 2024-05-09T08:33:01.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.856+0000 7fe19278d700 1 -- 172.21.15.107:0/327868000 shutdown_connections 2024-05-09T08:33:01.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.856+0000 7fe19278d700 1 --2- 172.21.15.107:0/327868000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe18c103f50 0x7fe18c19fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:01.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.856+0000 7fe19278d700 1 --2- 172.21.15.107:0/327868000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe18c1023b0 0x7fe18c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:01.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.856+0000 7fe19278d700 1 --2- 172.21.15.107:0/327868000 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fe1740a2c20 0x7fe1740a50e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:01.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.856+0000 7fe19278d700 1 --2- 172.21.15.107:0/327868000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe18c1035b0 0x7fe18c198ab0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:01.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.856+0000 7fe19278d700 1 -- 172.21.15.107:0/327868000 >> 172.21.15.107:0/327868000 conn(0x7fe18c0fd930 msgr2=0x7fe18c106980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:01.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.857+0000 7fe19278d700 1 -- 172.21.15.107:0/327868000 shutdown_connections 2024-05-09T08:33:01.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:01.857+0000 7fe19278d700 1 -- 172.21.15.107:0/327868000 wait complete. 2024-05-09T08:33:02.620 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-09T08:33:02.640 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-05-09T08:33:02.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-09T08:33:02.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:02 smithi157 ceph-mon[33614]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 2 B/s, 0 objects/s recovering 2024-05-09T08:33:02.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:02 smithi110 ceph-mon[33622]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 2 B/s, 0 objects/s recovering 2024-05-09T08:33:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:33:03.004 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:02 smithi107 ceph-mon[30122]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 2 B/s, 0 objects/s recovering 2024-05-09T08:33:03.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:03 smithi157 ceph-mon[33614]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:03.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:03 smithi110 ceph-mon[33622]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:04.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:03 smithi107 ceph-mon[30122]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:04.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.601+0000 7ff5462b6700 1 -- 172.21.15.107:0/1746300022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5401036e0 msgr2=0x7ff540103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:04.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.601+0000 7ff5462b6700 1 --2- 172.21.15.107:0/1746300022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5401036e0 0x7ff540103b40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff528009990 tx=0x7ff528009ca0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.601+0000 7ff5462b6700 1 -- 172.21.15.107:0/1746300022 shutdown_connections 2024-05-09T08:33:04.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.601+0000 7ff5462b6700 1 --2- 172.21.15.107:0/1746300022 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff540104080 0x7ff54010a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.601+0000 7ff5462b6700 1 --2- 172.21.15.107:0/1746300022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff540100700 0x7ff540100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.601+0000 7ff5462b6700 1 --2- 172.21.15.107:0/1746300022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5401036e0 0x7ff540103b40 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.601+0000 7ff5462b6700 1 -- 172.21.15.107:0/1746300022 >> 172.21.15.107:0/1746300022 conn(0x7ff5400fc060 msgr2=0x7ff5400fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:04.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.601+0000 7ff5462b6700 1 -- 172.21.15.107:0/1746300022 shutdown_connections 2024-05-09T08:33:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.601+0000 7ff5462b6700 1 -- 172.21.15.107:0/1746300022 wait complete. 2024-05-09T08:33:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.601+0000 7ff5462b6700 1 Processor -- start 2024-05-09T08:33:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.601+0000 7ff5462b6700 1 -- start start 2024-05-09T08:33:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff5462b6700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff540100700 0x7ff5401984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff5462b6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5401036e0 0x7ff540198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff5462b6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff540104080 0x7ff54019fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff5462b6700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff54010deb0 con 0x7ff5401036e0 2024-05-09T08:33:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff5462b6700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff54010dd30 con 0x7ff540104080 2024-05-09T08:33:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff5462b6700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff54010e030 con 0x7ff540100700 2024-05-09T08:33:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff537fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5401036e0 0x7ff540198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff537fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5401036e0 0x7ff540198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38296/0 (socket says 172.21.15.107:38296) 2024-05-09T08:33:04.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff537fff700 1 -- 172.21.15.107:0/561122379 learned_addr learned my addr 172.21.15.107:0/561122379 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:04.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff53ffff700 1 --2- 172.21.15.107:0/561122379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff540100700 0x7ff5401984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:04.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff544853700 1 --2- 172.21.15.107:0/561122379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff540104080 0x7ff54019fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:04.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff537fff700 1 -- 172.21.15.107:0/561122379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff540100700 msgr2=0x7ff5401984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:04.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff537fff700 1 --2- 172.21.15.107:0/561122379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff540100700 0x7ff5401984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff537fff700 1 -- 172.21.15.107:0/561122379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff540104080 msgr2=0x7ff54019fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:04.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff537fff700 1 --2- 172.21.15.107:0/561122379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff540104080 0x7ff54019fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff537fff700 1 -- 172.21.15.107:0/561122379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5401a0150 con 0x7ff5401036e0 2024-05-09T08:33:04.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff537fff700 1 --2- 172.21.15.107:0/561122379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5401036e0 0x7ff540198a30 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff5280047b0 tx=0x7ff528003710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:04.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff53dffb700 1 -- 172.21.15.107:0/561122379 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff528027070 con 0x7ff5401036e0 2024-05-09T08:33:04.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff53dffb700 1 -- 172.21.15.107:0/561122379 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff528004100 con 0x7ff5401036e0 2024-05-09T08:33:04.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff5462b6700 1 -- 172.21.15.107:0/561122379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5401a03e0 con 0x7ff5401036e0 2024-05-09T08:33:04.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff53dffb700 1 -- 172.21.15.107:0/561122379 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52800f460 con 0x7ff5401036e0 2024-05-09T08:33:04.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.602+0000 7ff5462b6700 1 -- 172.21.15.107:0/561122379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5401a0890 con 0x7ff5401036e0 2024-05-09T08:33:04.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.603+0000 7ff5462b6700 1 -- 172.21.15.107:0/561122379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff540069010 con 0x7ff5401036e0 2024-05-09T08:33:04.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.605+0000 7ff53dffb700 1 -- 172.21.15.107:0/561122379 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ff52802a070 con 0x7ff5401036e0 2024-05-09T08:33:04.607 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:04 smithi107 ceph-mon[30122]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2 B/s, 0 objects/s recovering 2024-05-09T08:33:04.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.606+0000 7ff53dffb700 1 --2- 172.21.15.107:0/561122379 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff52c0a2ca0 0x7ff52c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:04.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.606+0000 7ff53ffff700 1 --2- 172.21.15.107:0/561122379 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff52c0a2ca0 0x7ff52c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:04.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.606+0000 7ff53dffb700 1 -- 172.21.15.107:0/561122379 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7ff5280d0950 con 0x7ff5401036e0 2024-05-09T08:33:04.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.606+0000 7ff53ffff700 1 --2- 172.21.15.107:0/561122379 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff52c0a2ca0 0x7ff52c0a5160 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff530005fd0 tx=0x7ff530005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:04.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.609+0000 7ff53dffb700 1 -- 172.21.15.107:0/561122379 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff528031070 con 0x7ff5401036e0 2024-05-09T08:33:04.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:04 smithi157 ceph-mon[33614]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2 B/s, 0 objects/s recovering 2024-05-09T08:33:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.846+0000 7ff5462b6700 1 -- 172.21.15.107:0/561122379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7ff54002dc20 con 0x7ff5401036e0 2024-05-09T08:33:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.846+0000 7ff53dffb700 1 -- 172.21.15.107:0/561122379 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3061 (secure 0 0 0) 0x7ff54002dc20 con 0x7ff5401036e0 2024-05-09T08:33:04.845 INFO:teuthology.orchestra.run.smithi107.stdout:e24 2024-05-09T08:33:04.845 INFO:teuthology.orchestra.run.smithi107.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-09T08:33:04.846 INFO:teuthology.orchestra.run.smithi107.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-09T08:33:04.846 INFO:teuthology.orchestra.run.smithi107.stdout:legacy client fscid: 1 2024-05-09T08:33:04.846 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:04.846 INFO:teuthology.orchestra.run.smithi107.stdout:Filesystem 'cephfs' (1) 2024-05-09T08:33:04.846 INFO:teuthology.orchestra.run.smithi107.stdout:fs_name cephfs 2024-05-09T08:33:04.846 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 24 2024-05-09T08:33:04.846 INFO:teuthology.orchestra.run.smithi107.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-05-09T08:33:04.846 INFO:teuthology.orchestra.run.smithi107.stdout:created 2024-05-09T08:31:00.157448+0000 2024-05-09T08:33:04.846 INFO:teuthology.orchestra.run.smithi107.stdout:modified 2024-05-09T08:31:21.056050+0000 2024-05-09T08:33:04.846 INFO:teuthology.orchestra.run.smithi107.stdout:tableserver 0 2024-05-09T08:33:04.846 INFO:teuthology.orchestra.run.smithi107.stdout:root 0 2024-05-09T08:33:04.846 INFO:teuthology.orchestra.run.smithi107.stdout:session_timeout 300 2024-05-09T08:33:04.846 INFO:teuthology.orchestra.run.smithi107.stdout:session_autoclose 300 2024-05-09T08:33:04.847 INFO:teuthology.orchestra.run.smithi107.stdout:max_file_size 1099511627776 2024-05-09T08:33:04.847 INFO:teuthology.orchestra.run.smithi107.stdout:max_xattr_size 65536 2024-05-09T08:33:04.847 INFO:teuthology.orchestra.run.smithi107.stdout:required_client_features {} 2024-05-09T08:33:04.847 INFO:teuthology.orchestra.run.smithi107.stdout:last_failure 0 2024-05-09T08:33:04.847 INFO:teuthology.orchestra.run.smithi107.stdout:last_failure_osd_epoch 0 2024-05-09T08:33:04.847 INFO:teuthology.orchestra.run.smithi107.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-09T08:33:04.847 INFO:teuthology.orchestra.run.smithi107.stdout:max_mds 3 2024-05-09T08:33:04.847 INFO:teuthology.orchestra.run.smithi107.stdout:in 0,1,2 2024-05-09T08:33:04.847 INFO:teuthology.orchestra.run.smithi107.stdout:up {0=24499,1=24491,2=24478} 2024-05-09T08:33:04.847 INFO:teuthology.orchestra.run.smithi107.stdout:failed 2024-05-09T08:33:04.847 INFO:teuthology.orchestra.run.smithi107.stdout:damaged 2024-05-09T08:33:04.848 INFO:teuthology.orchestra.run.smithi107.stdout:stopped 2024-05-09T08:33:04.848 INFO:teuthology.orchestra.run.smithi107.stdout:data_pools [3,4] 2024-05-09T08:33:04.848 INFO:teuthology.orchestra.run.smithi107.stdout:metadata_pool 2 2024-05-09T08:33:04.848 INFO:teuthology.orchestra.run.smithi107.stdout:inline_data disabled 2024-05-09T08:33:04.848 INFO:teuthology.orchestra.run.smithi107.stdout:balancer 2024-05-09T08:33:04.848 INFO:teuthology.orchestra.run.smithi107.stdout:bal_rank_mask -1 2024-05-09T08:33:04.848 INFO:teuthology.orchestra.run.smithi107.stdout:standby_count_wanted 0 2024-05-09T08:33:04.848 INFO:teuthology.orchestra.run.smithi107.stdout:qdb_cluster leader: 24499 members: 24491,24478,24499 2024-05-09T08:33:04.848 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.i{0:24499} state up:active seq 5 addr [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:33:04.848 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.f{0:24460} state up:standby-replay seq 1 addr [v2:172.21.15.157:6834/843217781,v1:172.21.15.157:6835/843217781] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:33:04.848 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.b{1:24491} state up:active seq 8 addr [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:33:04.848 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.h{1:24458} state up:standby-replay seq 1 addr [v2:172.21.15.110:6836/624105719,v1:172.21.15.110:6837/624105719] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:33:04.848 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.c{2:24478} state up:active seq 11 addr [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:33:04.848 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.e{2:24431} state up:standby-replay seq 1 addr [v2:172.21.15.110:6834/2138165106,v1:172.21.15.110:6835/2138165106] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:33:04.849 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:04.849 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:04.849 INFO:teuthology.orchestra.run.smithi107.stdout:Standby daemons: 2024-05-09T08:33:04.849 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:04.849 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.j{-1:14562} state up:standby seq 1 addr [v2:172.21.15.107:6834/3401311866,v1:172.21.15.107:6835/3401311866] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:33:04.849 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.d{-1:14574} state up:standby seq 1 addr [v2:172.21.15.107:6836/4170346807,v1:172.21.15.107:6837/4170346807] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:33:04.849 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.g{-1:14637} state up:standby seq 1 addr [v2:172.21.15.107:6838/2088211251,v1:172.21.15.107:6839/2088211251] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:33:04.849 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.a{-1:14649} state up:standby seq 1 addr [v2:172.21.15.107:6840/2463167549,v1:172.21.15.107:6841/2463167549] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:33:04.849 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.k{-1:24407} state up:standby seq 1 addr [v2:172.21.15.110:6832/844626352,v1:172.21.15.110:6833/844626352] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:33:04.849 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.l{-1:24418} state up:standby seq 1 addr [v2:172.21.15.157:6832/2435727962,v1:172.21.15.157:6833/2435727962] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T08:33:04.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.848+0000 7ff5462b6700 1 -- 172.21.15.107:0/561122379 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff52c0a2ca0 msgr2=0x7ff52c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:04.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.848+0000 7ff5462b6700 1 --2- 172.21.15.107:0/561122379 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff52c0a2ca0 0x7ff52c0a5160 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff530005fd0 tx=0x7ff530005f60 comp rx=0 tx=0).stop 2024-05-09T08:33:04.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.848+0000 7ff5462b6700 1 -- 172.21.15.107:0/561122379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5401036e0 msgr2=0x7ff540198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.848+0000 7ff5462b6700 1 --2- 172.21.15.107:0/561122379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5401036e0 0x7ff540198a30 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff5280047b0 tx=0x7ff528003710 comp rx=0 tx=0).stop 2024-05-09T08:33:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.849+0000 7ff5462b6700 1 -- 172.21.15.107:0/561122379 shutdown_connections 2024-05-09T08:33:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.849+0000 7ff5462b6700 1 --2- 172.21.15.107:0/561122379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff540104080 0x7ff54019fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.849+0000 7ff5462b6700 1 --2- 172.21.15.107:0/561122379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff540100700 0x7ff5401984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.849+0000 7ff5462b6700 1 --2- 172.21.15.107:0/561122379 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff52c0a2ca0 0x7ff52c0a5160 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.849+0000 7ff5462b6700 1 --2- 172.21.15.107:0/561122379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5401036e0 0x7ff540198a30 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.849+0000 7ff5462b6700 1 -- 172.21.15.107:0/561122379 >> 172.21.15.107:0/561122379 conn(0x7ff5400fc060 msgr2=0x7ff5401069e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.849+0000 7ff5462b6700 1 -- 172.21.15.107:0/561122379 shutdown_connections 2024-05-09T08:33:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:04.849+0000 7ff5462b6700 1 -- 172.21.15.107:0/561122379 wait complete. 2024-05-09T08:33:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 24 2024-05-09T08:33:04.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:04 smithi110 ceph-mon[33622]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2 B/s, 0 objects/s recovering 2024-05-09T08:33:05.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- bash -c 'ceph osd dump' 2024-05-09T08:33:05.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:05 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/561122379' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-09T08:33:05.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:05 smithi107 ceph-mon[30122]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 2 B/s, 0 objects/s recovering 2024-05-09T08:33:05.807 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:33:05.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:05 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/561122379' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-09T08:33:05.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:05 smithi157 ceph-mon[33614]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 2 B/s, 0 objects/s recovering 2024-05-09T08:33:05.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:05 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/561122379' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-09T08:33:05.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:05 smithi110 ceph-mon[33622]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 2 B/s, 0 objects/s recovering 2024-05-09T08:33:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.601+0000 7ff273e98700 1 -- 172.21.15.107:0/3450840788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff26c102e20 msgr2=0x7ff26c1096d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.601+0000 7ff273e98700 1 --2- 172.21.15.107:0/3450840788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff26c102e20 0x7ff26c1096d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff26400b0d0 tx=0x7ff26400b3e0 comp rx=0 tx=0).stop 2024-05-09T08:33:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.601+0000 7ff273e98700 1 -- 172.21.15.107:0/3450840788 shutdown_connections 2024-05-09T08:33:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.601+0000 7ff273e98700 1 --2- 172.21.15.107:0/3450840788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff26c102e20 0x7ff26c1096d0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.601+0000 7ff273e98700 1 --2- 172.21.15.107:0/3450840788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff26c101280 0x7ff26c1016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.601+0000 7ff273e98700 1 --2- 172.21.15.107:0/3450840788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff26c102480 0x7ff26c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.601+0000 7ff273e98700 1 -- 172.21.15.107:0/3450840788 >> 172.21.15.107:0/3450840788 conn(0x7ff26c0fc820 msgr2=0x7ff26c0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.601+0000 7ff273e98700 1 -- 172.21.15.107:0/3450840788 shutdown_connections 2024-05-09T08:33:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.601+0000 7ff273e98700 1 -- 172.21.15.107:0/3450840788 wait complete. 2024-05-09T08:33:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.602+0000 7ff273e98700 1 Processor -- start 2024-05-09T08:33:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.602+0000 7ff273e98700 1 -- start start 2024-05-09T08:33:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.602+0000 7ff273e98700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff26c101280 0x7ff26c06ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.602+0000 7ff273e98700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff26c102480 0x7ff26c19d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.602+0000 7ff273e98700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff26c102e20 0x7ff26c19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.602+0000 7ff273e98700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c10cd10 con 0x7ff26c102e20 2024-05-09T08:33:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.602+0000 7ff273e98700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c10cb90 con 0x7ff26c102480 2024-05-09T08:33:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.602+0000 7ff273e98700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c10ce90 con 0x7ff26c101280 2024-05-09T08:33:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff272435700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff26c102e20 0x7ff26c19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff272435700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff26c102e20 0x7ff26c19f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38304/0 (socket says 172.21.15.107:38304) 2024-05-09T08:33:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff272435700 1 -- 172.21.15.107:0/2943807527 learned_addr learned my addr 172.21.15.107:0/2943807527 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff271c34700 1 --2- 172.21.15.107:0/2943807527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff26c101280 0x7ff26c06ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff271433700 1 --2- 172.21.15.107:0/2943807527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff26c102480 0x7ff26c19d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff272435700 1 -- 172.21.15.107:0/2943807527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff26c101280 msgr2=0x7ff26c06ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff272435700 1 --2- 172.21.15.107:0/2943807527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff26c101280 0x7ff26c06ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff272435700 1 -- 172.21.15.107:0/2943807527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff26c102480 msgr2=0x7ff26c19d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff272435700 1 --2- 172.21.15.107:0/2943807527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff26c102480 0x7ff26c19d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff272435700 1 -- 172.21.15.107:0/2943807527 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff26c1a00f0 con 0x7ff26c102e20 2024-05-09T08:33:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff271433700 1 --2- 172.21.15.107:0/2943807527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff26c102480 0x7ff26c19d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff272435700 1 --2- 172.21.15.107:0/2943807527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff26c102e20 0x7ff26c19f820 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff26400d010 tx=0x7ff264007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff262ffd700 1 -- 172.21.15.107:0/2943807527 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff264021070 con 0x7ff26c102e20 2024-05-09T08:33:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff273e98700 1 -- 172.21.15.107:0/2943807527 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff26c1a0380 con 0x7ff26c102e20 2024-05-09T08:33:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff262ffd700 1 -- 172.21.15.107:0/2943807527 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff26401db00 con 0x7ff26c102e20 2024-05-09T08:33:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff262ffd700 1 -- 172.21.15.107:0/2943807527 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26401bb50 con 0x7ff26c102e20 2024-05-09T08:33:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.603+0000 7ff273e98700 1 -- 172.21.15.107:0/2943807527 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff26c1a0880 con 0x7ff26c102e20 2024-05-09T08:33:07.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.604+0000 7ff273e98700 1 -- 172.21.15.107:0/2943807527 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff26c069010 con 0x7ff26c102e20 2024-05-09T08:33:07.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.605+0000 7ff262ffd700 1 -- 172.21.15.107:0/2943807527 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ff264024070 con 0x7ff26c102e20 2024-05-09T08:33:07.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.606+0000 7ff262ffd700 1 --2- 172.21.15.107:0/2943807527 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff2580a2c20 0x7ff2580a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:07.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.606+0000 7ff271c34700 1 --2- 172.21.15.107:0/2943807527 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff2580a2c20 0x7ff2580a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:07.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.606+0000 7ff262ffd700 1 -- 172.21.15.107:0/2943807527 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7ff26400f020 con 0x7ff26c102e20 2024-05-09T08:33:07.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.606+0000 7ff271c34700 1 --2- 172.21.15.107:0/2943807527 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff2580a2c20 0x7ff2580a50e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff26c1022e0 tx=0x7ff26800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.609+0000 7ff262ffd700 1 -- 172.21.15.107:0/2943807527 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff26402b020 con 0x7ff26c102e20 2024-05-09T08:33:07.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.808+0000 7ff273e98700 1 -- 172.21.15.107:0/2943807527 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7ff26c02dd10 con 0x7ff26c102e20 2024-05-09T08:33:07.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.809+0000 7ff262ffd700 1 -- 172.21.15.107:0/2943807527 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5655 (secure 0 0 0) 0x7ff26409e170 con 0x7ff26c102e20 2024-05-09T08:33:07.808 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 93 2024-05-09T08:33:07.808 INFO:teuthology.orchestra.run.smithi107.stdout:fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:33:07.808 INFO:teuthology.orchestra.run.smithi107.stdout:created 2024-05-09T08:22:57.028352+0000 2024-05-09T08:33:07.809 INFO:teuthology.orchestra.run.smithi107.stdout:modified 2024-05-09T08:32:57.545099+0000 2024-05-09T08:33:07.809 INFO:teuthology.orchestra.run.smithi107.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-09T08:33:07.809 INFO:teuthology.orchestra.run.smithi107.stdout:crush_version 27 2024-05-09T08:33:07.809 INFO:teuthology.orchestra.run.smithi107.stdout:full_ratio 0.95 2024-05-09T08:33:07.809 INFO:teuthology.orchestra.run.smithi107.stdout:backfillfull_ratio 0.9 2024-05-09T08:33:07.809 INFO:teuthology.orchestra.run.smithi107.stdout:nearfull_ratio 0.85 2024-05-09T08:33:07.809 INFO:teuthology.orchestra.run.smithi107.stdout:require_min_compat_client luminous 2024-05-09T08:33:07.809 INFO:teuthology.orchestra.run.smithi107.stdout:min_compat_client luminous 2024-05-09T08:33:07.809 INFO:teuthology.orchestra.run.smithi107.stdout:require_osd_release squid 2024-05-09T08:33:07.809 INFO:teuthology.orchestra.run.smithi107.stdout:stretch_mode_enabled false 2024-05-09T08:33:07.809 INFO:teuthology.orchestra.run.smithi107.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-09T08:33:07.809 INFO:teuthology.orchestra.run.smithi107.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-05-09T08:33:07.809 INFO:teuthology.orchestra.run.smithi107.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-05-09T08:33:07.810 INFO:teuthology.orchestra.run.smithi107.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-05-09T08:33:07.810 INFO:teuthology.orchestra.run.smithi107.stdout:max_osd 12 2024-05-09T08:33:07.810 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6802/1215261349,v1:172.21.15.107:6803/1215261349] [v2:172.21.15.107:6804/1215261349,v1:172.21.15.107:6805/1215261349] exists,up 8ab0acea-9500-4d88-ab8e-2d148374937b 2024-05-09T08:33:07.810 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 up in weight 1 up_from 13 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6800/370054671,v1:172.21.15.110:6801/370054671] [v2:172.21.15.110:6802/370054671,v1:172.21.15.110:6803/370054671] exists,up 93653939-b195-46f0-a8a9-b25ad2af5464 2024-05-09T08:33:07.810 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.157:6800/3005990647,v1:172.21.15.157:6801/3005990647] [v2:172.21.15.157:6802/3005990647,v1:172.21.15.157:6803/3005990647] exists,up dc55ab61-bfc9-48ad-92b3-37ad560a0023 2024-05-09T08:33:07.810 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6810/1056009754,v1:172.21.15.107:6811/1056009754] [v2:172.21.15.107:6812/1056009754,v1:172.21.15.107:6813/1056009754] exists,up 8b4ef7e0-8679-44c3-ac27-170d0ab391aa 2024-05-09T08:33:07.810 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 up in weight 1 up_from 33 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6808/2317455505,v1:172.21.15.110:6809/2317455505] [v2:172.21.15.110:6810/2317455505,v1:172.21.15.110:6811/2317455505] exists,up b67c4b46-7857-48cd-8737-d7cbb37aabaf 2024-05-09T08:33:07.810 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 up in weight 1 up_from 38 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.157:6808/423586420,v1:172.21.15.157:6809/423586420] [v2:172.21.15.157:6810/423586420,v1:172.21.15.157:6811/423586420] exists,up 2f4cfd0f-4faa-4697-aa3b-ef48a801be37 2024-05-09T08:33:07.810 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 up in weight 1 up_from 46 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6818/3011505372,v1:172.21.15.107:6819/3011505372] [v2:172.21.15.107:6820/3011505372,v1:172.21.15.107:6821/3011505372] exists,up c0451766-461b-4d7e-8087-f2d066097c16 2024-05-09T08:33:07.810 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6816/204549355,v1:172.21.15.110:6817/204549355] [v2:172.21.15.110:6818/204549355,v1:172.21.15.110:6819/204549355] exists,up b7258141-aef2-4d46-9825-8c51d817f7a3 2024-05-09T08:33:07.810 INFO:teuthology.orchestra.run.smithi107.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.157:6816/4115873287,v1:172.21.15.157:6817/4115873287] [v2:172.21.15.157:6818/4115873287,v1:172.21.15.157:6819/4115873287] exists,up bcc1105e-25be-4d89-bb7b-920287c3f6cd 2024-05-09T08:33:07.810 INFO:teuthology.orchestra.run.smithi107.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6826/2273353916,v1:172.21.15.107:6827/2273353916] [v2:172.21.15.107:6828/2273353916,v1:172.21.15.107:6829/2273353916] exists,up 56600856-ce7c-47d1-aff0-a57cf7ce75e8 2024-05-09T08:33:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6824/1566970764,v1:172.21.15.110:6825/1566970764] [v2:172.21.15.110:6826/1566970764,v1:172.21.15.110:6827/1566970764] exists,up f4022636-ba24-4685-9678-d261e04df620 2024-05-09T08:33:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:osd.11 up in weight 1 up_from 80 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.157:6824/3347498474,v1:172.21.15.157:6825/3347498474] [v2:172.21.15.157:6826/3347498474,v1:172.21.15.157:6827/3347498474] exists,up fef75204-3f64-4365-8ece-384eb4598b1b 2024-05-09T08:33:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 2.4 [2,11] 2024-05-09T08:33:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 2.5 [8,11] 2024-05-09T08:33:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 2.c [2,1] 2024-05-09T08:33:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 2.1f [8,4] 2024-05-09T08:33:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.13 [4,9] 2024-05-09T08:33:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.21 [5,9] 2024-05-09T08:33:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.2c [4,9] 2024-05-09T08:33:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.5 [5,10] 2024-05-09T08:33:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.10 [9,8] 2024-05-09T08:33:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.19 [9,8] 2024-05-09T08:33:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.1f [9,8] 2024-05-09T08:33:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/1064417772 expires 2024-05-10T08:23:55.372959+0000 2024-05-09T08:33:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6801/3922375261 expires 2024-05-10T08:23:55.372959+0000 2024-05-09T08:33:07.811 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6800/3922375261 expires 2024-05-10T08:23:55.372959+0000 2024-05-09T08:33:07.812 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/3934964045 expires 2024-05-10T08:23:55.372959+0000 2024-05-09T08:33:07.812 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/1620469781 expires 2024-05-10T08:23:29.066232+0000 2024-05-09T08:33:07.812 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/4101303556 expires 2024-05-10T08:23:29.066232+0000 2024-05-09T08:33:07.812 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6800/2360407889 expires 2024-05-10T08:23:29.066232+0000 2024-05-09T08:33:07.812 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/164842293 expires 2024-05-10T08:23:29.066232+0000 2024-05-09T08:33:07.812 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6801/2360407889 expires 2024-05-10T08:23:29.066232+0000 2024-05-09T08:33:07.812 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/4237993479 expires 2024-05-10T08:23:55.372959+0000 2024-05-09T08:33:07.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.811+0000 7ff273e98700 1 -- 172.21.15.107:0/2943807527 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff2580a2c20 msgr2=0x7ff2580a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:07.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.811+0000 7ff273e98700 1 --2- 172.21.15.107:0/2943807527 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff2580a2c20 0x7ff2580a50e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff26c1022e0 tx=0x7ff26800b410 comp rx=0 tx=0).stop 2024-05-09T08:33:07.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.811+0000 7ff273e98700 1 -- 172.21.15.107:0/2943807527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff26c102e20 msgr2=0x7ff26c19f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:07.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.811+0000 7ff273e98700 1 --2- 172.21.15.107:0/2943807527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff26c102e20 0x7ff26c19f820 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff26400d010 tx=0x7ff264007a20 comp rx=0 tx=0).stop 2024-05-09T08:33:07.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.812+0000 7ff273e98700 1 -- 172.21.15.107:0/2943807527 shutdown_connections 2024-05-09T08:33:07.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.812+0000 7ff273e98700 1 --2- 172.21.15.107:0/2943807527 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff2580a2c20 0x7ff2580a50e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:07.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.812+0000 7ff273e98700 1 --2- 172.21.15.107:0/2943807527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff26c102e20 0x7ff26c19f820 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:07.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.812+0000 7ff273e98700 1 --2- 172.21.15.107:0/2943807527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff26c101280 0x7ff26c06ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:07.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.812+0000 7ff273e98700 1 --2- 172.21.15.107:0/2943807527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff26c102480 0x7ff26c19d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:07.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.812+0000 7ff273e98700 1 -- 172.21.15.107:0/2943807527 >> 172.21.15.107:0/2943807527 conn(0x7ff26c0fc820 msgr2=0x7ff26c105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:07.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.812+0000 7ff273e98700 1 -- 172.21.15.107:0/2943807527 shutdown_connections 2024-05-09T08:33:07.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:07.812+0000 7ff273e98700 1 -- 172.21.15.107:0/2943807527 wait complete. 2024-05-09T08:33:08.487 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-09T08:33:08.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:33:08.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.606+0000 7f38e225d640 1 -- 172.21.15.107:0/3852166386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38dc1018a0 msgr2=0x7f38dc101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:08.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.606+0000 7f38e225d640 1 --2- 172.21.15.107:0/3852166386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38dc1018a0 0x7f38dc101cc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f38cc009900 tx=0x7f38cc02a410 comp rx=0 tx=0).stop 2024-05-09T08:33:08.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.606+0000 7f38e225d640 1 -- 172.21.15.107:0/3852166386 shutdown_connections 2024-05-09T08:33:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.606+0000 7f38e225d640 1 --2- 172.21.15.107:0/3852166386 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38dc103450 0x7f38dc109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.606+0000 7f38e225d640 1 --2- 172.21.15.107:0/3852166386 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38dc102a90 0x7f38dc102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.606+0000 7f38e225d640 1 --2- 172.21.15.107:0/3852166386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38dc1018a0 0x7f38dc101cc0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.606+0000 7f38e225d640 1 -- 172.21.15.107:0/3852166386 >> 172.21.15.107:0/3852166386 conn(0x7f38dc0fce10 msgr2=0x7f38dc0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.606+0000 7f38e225d640 1 -- 172.21.15.107:0/3852166386 shutdown_connections 2024-05-09T08:33:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.606+0000 7f38e225d640 1 -- 172.21.15.107:0/3852166386 wait complete. 2024-05-09T08:33:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.607+0000 7f38e225d640 1 Processor -- start 2024-05-09T08:33:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.607+0000 7f38e225d640 1 -- start start 2024-05-09T08:33:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.607+0000 7f38e225d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38dc1018a0 0x7f38dc197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.607+0000 7f38e225d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38dc102a90 0x7f38dc198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.607+0000 7f38e225d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38dc103450 0x7f38dc19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.607+0000 7f38e225d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f38dc10d3c0 con 0x7f38dc103450 2024-05-09T08:33:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.607+0000 7f38e225d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f38dc10d240 con 0x7f38dc102a90 2024-05-09T08:33:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.607+0000 7f38e225d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f38dc10d540 con 0x7f38dc1018a0 2024-05-09T08:33:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.607+0000 7f38dbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38dc103450 0x7f38dc19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.607+0000 7f38dbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38dc103450 0x7f38dc19f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38324/0 (socket says 172.21.15.107:38324) 2024-05-09T08:33:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.607+0000 7f38dbfff640 1 -- 172.21.15.107:0/1950617173 learned_addr learned my addr 172.21.15.107:0/1950617173 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.607+0000 7f38db7fe640 1 --2- 172.21.15.107:0/1950617173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38dc1018a0 0x7f38dc197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.608+0000 7f38daffd640 1 --2- 172.21.15.107:0/1950617173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38dc102a90 0x7f38dc198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.608+0000 7f38dbfff640 1 -- 172.21.15.107:0/1950617173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38dc1018a0 msgr2=0x7f38dc197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.608+0000 7f38dbfff640 1 --2- 172.21.15.107:0/1950617173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38dc1018a0 0x7f38dc197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.608+0000 7f38dbfff640 1 -- 172.21.15.107:0/1950617173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38dc102a90 msgr2=0x7f38dc198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.608+0000 7f38dbfff640 1 --2- 172.21.15.107:0/1950617173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38dc102a90 0x7f38dc198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.608+0000 7f38dbfff640 1 -- 172.21.15.107:0/1950617173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38dc19f760 con 0x7f38dc103450 2024-05-09T08:33:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.608+0000 7f38daffd640 1 --2- 172.21.15.107:0/1950617173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38dc102a90 0x7f38dc198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.608+0000 7f38dbfff640 1 --2- 172.21.15.107:0/1950617173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38dc103450 0x7f38dc19f0f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f38d400d750 tx=0x7f38d400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.608+0000 7f38d8ff9640 1 -- 172.21.15.107:0/1950617173 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d4017070 con 0x7f38dc103450 2024-05-09T08:33:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.608+0000 7f38d8ff9640 1 -- 172.21.15.107:0/1950617173 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f38d40054d0 con 0x7f38dc103450 2024-05-09T08:33:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.608+0000 7f38e225d640 1 -- 172.21.15.107:0/1950617173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38dc19fa50 con 0x7f38dc103450 2024-05-09T08:33:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.608+0000 7f38d8ff9640 1 -- 172.21.15.107:0/1950617173 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d4004c60 con 0x7f38dc103450 2024-05-09T08:33:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.608+0000 7f38e225d640 1 -- 172.21.15.107:0/1950617173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38dc19ff50 con 0x7f38dc103450 2024-05-09T08:33:08.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.609+0000 7f38e225d640 1 -- 172.21.15.107:0/1950617173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38dc108ce0 con 0x7f38dc103450 2024-05-09T08:33:08.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.610+0000 7f38d8ff9640 1 -- 172.21.15.107:0/1950617173 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f38d4004e00 con 0x7f38dc103450 2024-05-09T08:33:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.611+0000 7f38d8ff9640 1 --2- 172.21.15.107:0/1950617173 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f38c40a2b90 0x7f38c40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.611+0000 7f38db7fe640 1 --2- 172.21.15.107:0/1950617173 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f38c40a2b90 0x7f38c40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.611+0000 7f38d8ff9640 1 -- 172.21.15.107:0/1950617173 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f38d40ced80 con 0x7f38dc103450 2024-05-09T08:33:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.611+0000 7f38db7fe640 1 --2- 172.21.15.107:0/1950617173 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f38c40a2b90 0x7f38c40a5050 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f38cc004770 tx=0x7f38cc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:08.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.615+0000 7f38d8ff9640 1 -- 172.21.15.107:0/1950617173 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f38d409b960 con 0x7f38dc103450 2024-05-09T08:33:08.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:08 smithi107 ceph-mon[30122]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-05-09T08:33:08.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:08 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2943807527' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-09T08:33:08.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.792+0000 7f38e225d640 1 -- 172.21.15.107:0/1950617173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f38dc062430 con 0x7f38dc103450 2024-05-09T08:33:08.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.793+0000 7f38d8ff9640 1 -- 172.21.15.107:0/1950617173 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24761 (secure 0 0 0) 0x7f38d409b260 con 0x7f38dc103450 2024-05-09T08:33:08.793 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:08.793 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":93,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:32:57.545099+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:33:08.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.795+0000 7f38e225d640 1 -- 172.21.15.107:0/1950617173 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f38c40a2b90 msgr2=0x7f38c40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:08.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.795+0000 7f38e225d640 1 --2- 172.21.15.107:0/1950617173 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f38c40a2b90 0x7f38c40a5050 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f38cc004770 tx=0x7f38cc000bc0 comp rx=0 tx=0).stop 2024-05-09T08:33:08.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.795+0000 7f38e225d640 1 -- 172.21.15.107:0/1950617173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38dc103450 msgr2=0x7f38dc19f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:08.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.795+0000 7f38e225d640 1 --2- 172.21.15.107:0/1950617173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38dc103450 0x7f38dc19f0f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f38d400d750 tx=0x7f38d400dc10 comp rx=0 tx=0).stop 2024-05-09T08:33:08.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.795+0000 7f38e225d640 1 -- 172.21.15.107:0/1950617173 shutdown_connections 2024-05-09T08:33:08.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.795+0000 7f38e225d640 1 --2- 172.21.15.107:0/1950617173 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f38c40a2b90 0x7f38c40a5050 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:08.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.795+0000 7f38e225d640 1 --2- 172.21.15.107:0/1950617173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38dc103450 0x7f38dc19f0f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:08.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.795+0000 7f38e225d640 1 --2- 172.21.15.107:0/1950617173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38dc102a90 0x7f38dc198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:08.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.795+0000 7f38e225d640 1 --2- 172.21.15.107:0/1950617173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38dc1018a0 0x7f38dc197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:08.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.796+0000 7f38e225d640 1 -- 172.21.15.107:0/1950617173 >> 172.21.15.107:0/1950617173 conn(0x7f38dc0fce10 msgr2=0x7f38dc105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:08.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.796+0000 7f38e225d640 1 -- 172.21.15.107:0/1950617173 shutdown_connections 2024-05-09T08:33:08.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.796+0000 7f38e225d640 1 -- 172.21.15.107:0/1950617173 wait complete. 2024-05-09T08:33:08.806 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:26:21.698338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:30:58.268633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:30:59.345579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:31:03.065991+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:33:08.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:33:08.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:08 smithi157 ceph-mon[33614]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-05-09T08:33:08.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:08 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2943807527' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-09T08:33:08.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:08 smithi110 ceph-mon[33622]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 2 B/s, 0 objects/s recovering 2024-05-09T08:33:08.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:08 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2943807527' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-09T08:33:08.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.913+0000 7f6b8043f640 1 -- 172.21.15.107:0/3191611560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b781034d0 msgr2=0x7f6b78109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.913+0000 7f6b8043f640 1 --2- 172.21.15.107:0/3191611560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b781034d0 0x7f6b78109d60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009930 tx=0x7f6b7002a420 comp rx=0 tx=0).stop 2024-05-09T08:33:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.913+0000 7f6b8043f640 1 -- 172.21.15.107:0/3191611560 shutdown_connections 2024-05-09T08:33:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.913+0000 7f6b8043f640 1 --2- 172.21.15.107:0/3191611560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b781034d0 0x7f6b78109d60 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.913+0000 7f6b8043f640 1 --2- 172.21.15.107:0/3191611560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b78102b10 0x7f6b78102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.913+0000 7f6b8043f640 1 --2- 172.21.15.107:0/3191611560 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b780ffb40 0x7f6b780fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.913+0000 7f6b8043f640 1 -- 172.21.15.107:0/3191611560 >> 172.21.15.107:0/3191611560 conn(0x7f6b780fb470 msgr2=0x7f6b780fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.913+0000 7f6b8043f640 1 -- 172.21.15.107:0/3191611560 shutdown_connections 2024-05-09T08:33:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.913+0000 7f6b8043f640 1 -- 172.21.15.107:0/3191611560 wait complete. 2024-05-09T08:33:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.913+0000 7f6b8043f640 1 Processor -- start 2024-05-09T08:33:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b8043f640 1 -- start start 2024-05-09T08:33:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b8043f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b780ffb40 0x7f6b78197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:08.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b8043f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b78102b10 0x7f6b78197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:08.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b8043f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b781034d0 0x7f6b7819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:08.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b8043f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7810d3c0 con 0x7f6b78102b10 2024-05-09T08:33:08.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b8043f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7810d240 con 0x7f6b780ffb40 2024-05-09T08:33:08.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b8043f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7810d540 con 0x7f6b781034d0 2024-05-09T08:33:08.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b7d9da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b78102b10 0x7f6b78197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:08.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b7d9da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b78102b10 0x7f6b78197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38336/0 (socket says 172.21.15.107:38336) 2024-05-09T08:33:08.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b7d9da640 1 -- 172.21.15.107:0/3921209297 learned_addr learned my addr 172.21.15.107:0/3921209297 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:08.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b7d9da640 1 -- 172.21.15.107:0/3921209297 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b781034d0 msgr2=0x7f6b7819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:08.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b7d9da640 1 --2- 172.21.15.107:0/3921209297 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b781034d0 0x7f6b7819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:08.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b7d9da640 1 -- 172.21.15.107:0/3921209297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b780ffb40 msgr2=0x7f6b78197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:08.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b7d9da640 1 --2- 172.21.15.107:0/3921209297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b780ffb40 0x7f6b78197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:08.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b7d9da640 1 -- 172.21.15.107:0/3921209297 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b7819f6b0 con 0x7f6b78102b10 2024-05-09T08:33:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b7d9da640 1 --2- 172.21.15.107:0/3921209297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b78102b10 0x7f6b78197fd0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800d950 tx=0x7f6b6800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b6f7fe640 1 -- 172.21.15.107:0/3921209297 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b68017030 con 0x7f6b78102b10 2024-05-09T08:33:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b6f7fe640 1 -- 172.21.15.107:0/3921209297 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b68004020 con 0x7f6b78102b10 2024-05-09T08:33:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b8043f640 1 -- 172.21.15.107:0/3921209297 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b7819f9a0 con 0x7f6b78102b10 2024-05-09T08:33:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b6f7fe640 1 -- 172.21.15.107:0/3921209297 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b680047a0 con 0x7f6b78102b10 2024-05-09T08:33:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.914+0000 7f6b8043f640 1 -- 172.21.15.107:0/3921209297 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b7819ff40 con 0x7f6b78102b10 2024-05-09T08:33:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.916+0000 7f6b8043f640 1 -- 172.21.15.107:0/3921209297 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b78108c90 con 0x7f6b78102b10 2024-05-09T08:33:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.917+0000 7f6b6f7fe640 1 -- 172.21.15.107:0/3921209297 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f6b680041d0 con 0x7f6b78102b10 2024-05-09T08:33:08.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.917+0000 7f6b6f7fe640 1 --2- 172.21.15.107:0/3921209297 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6b640a2c10 0x7f6b640a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:08.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.917+0000 7f6b7e1db640 1 --2- 172.21.15.107:0/3921209297 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6b640a2c10 0x7f6b640a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:08.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.917+0000 7f6b6f7fe640 1 -- 172.21.15.107:0/3921209297 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f6b680cee60 con 0x7f6b78102b10 2024-05-09T08:33:08.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.918+0000 7f6b7e1db640 1 --2- 172.21.15.107:0/3921209297 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6b640a2c10 0x7f6b640a50d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6b78069c50 tx=0x7f6b740093c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:08.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:08.922+0000 7f6b6f7fe640 1 -- 172.21.15.107:0/3921209297 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6b6809ba40 con 0x7f6b78102b10 2024-05-09T08:33:09.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.099+0000 7f6b8043f640 1 -- 172.21.15.107:0/3921209297 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6b780624d0 con 0x7f6b78102b10 2024-05-09T08:33:09.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.099+0000 7f6b6f7fe640 1 -- 172.21.15.107:0/3921209297 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f6b6809b340 con 0x7f6b78102b10 2024-05-09T08:33:09.099 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-05-09T08:33:09.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.101+0000 7f6b8043f640 1 -- 172.21.15.107:0/3921209297 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6b640a2c10 msgr2=0x7f6b640a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:09.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.101+0000 7f6b8043f640 1 --2- 172.21.15.107:0/3921209297 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6b640a2c10 0x7f6b640a50d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6b78069c50 tx=0x7f6b740093c0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.101+0000 7f6b8043f640 1 -- 172.21.15.107:0/3921209297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b78102b10 msgr2=0x7f6b78197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.101+0000 7f6b8043f640 1 --2- 172.21.15.107:0/3921209297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b78102b10 0x7f6b78197fd0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800d950 tx=0x7f6b6800de10 comp rx=0 tx=0).stop 2024-05-09T08:33:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.101+0000 7f6b8043f640 1 -- 172.21.15.107:0/3921209297 shutdown_connections 2024-05-09T08:33:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.101+0000 7f6b8043f640 1 --2- 172.21.15.107:0/3921209297 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6b640a2c10 0x7f6b640a50d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.101+0000 7f6b8043f640 1 --2- 172.21.15.107:0/3921209297 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b781034d0 0x7f6b7819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.101+0000 7f6b8043f640 1 --2- 172.21.15.107:0/3921209297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b78102b10 0x7f6b78197fd0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.101+0000 7f6b8043f640 1 --2- 172.21.15.107:0/3921209297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b780ffb40 0x7f6b78197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.101+0000 7f6b8043f640 1 -- 172.21.15.107:0/3921209297 >> 172.21.15.107:0/3921209297 conn(0x7f6b780fb470 msgr2=0x7f6b78105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:09.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.101+0000 7f6b8043f640 1 -- 172.21.15.107:0/3921209297 shutdown_connections 2024-05-09T08:33:09.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.101+0000 7f6b8043f640 1 -- 172.21.15.107:0/3921209297 wait complete. 2024-05-09T08:33:09.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:33:09.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.218+0000 7f6de7e5d640 1 -- 172.21.15.107:0/3954125007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de0101880 msgr2=0x7f6de0101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:09.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.218+0000 7f6de7e5d640 1 --2- 172.21.15.107:0/3954125007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de0101880 0x7f6de0101ca0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009900 tx=0x7f6ddc02a410 comp rx=0 tx=0).stop 2024-05-09T08:33:09.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.218+0000 7f6de7e5d640 1 -- 172.21.15.107:0/3954125007 shutdown_connections 2024-05-09T08:33:09.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.218+0000 7f6de7e5d640 1 --2- 172.21.15.107:0/3954125007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de0103430 0x7f6de0109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.218+0000 7f6de7e5d640 1 --2- 172.21.15.107:0/3954125007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6de0102a70 0x7f6de0102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.218+0000 7f6de7e5d640 1 --2- 172.21.15.107:0/3954125007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de0101880 0x7f6de0101ca0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.218+0000 7f6de7e5d640 1 -- 172.21.15.107:0/3954125007 >> 172.21.15.107:0/3954125007 conn(0x7f6de00fce10 msgr2=0x7f6de00ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:09.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.218+0000 7f6de7e5d640 1 -- 172.21.15.107:0/3954125007 shutdown_connections 2024-05-09T08:33:09.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.218+0000 7f6de7e5d640 1 -- 172.21.15.107:0/3954125007 wait complete. 2024-05-09T08:33:09.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.218+0000 7f6de7e5d640 1 Processor -- start 2024-05-09T08:33:09.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.218+0000 7f6de7e5d640 1 -- start start 2024-05-09T08:33:09.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.218+0000 7f6de7e5d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6de0101880 0x7f6de0197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:09.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de7e5d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de0102a70 0x7f6de0198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:09.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de7e5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de0103430 0x7f6de019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:09.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de7e5d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6de010d3a0 con 0x7f6de0103430 2024-05-09T08:33:09.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de7e5d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6de010d220 con 0x7f6de0102a70 2024-05-09T08:33:09.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de7e5d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6de010d520 con 0x7f6de0101880 2024-05-09T08:33:09.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de63fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de0103430 0x7f6de019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:09.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de63fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de0103430 0x7f6de019f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38348/0 (socket says 172.21.15.107:38348) 2024-05-09T08:33:09.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de63fa640 1 -- 172.21.15.107:0/1292816237 learned_addr learned my addr 172.21.15.107:0/1292816237 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:09.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de5bf9640 1 --2- 172.21.15.107:0/1292816237 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6de0101880 0x7f6de0197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:09.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de53f8640 1 --2- 172.21.15.107:0/1292816237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de0102a70 0x7f6de0198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:09.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de63fa640 1 -- 172.21.15.107:0/1292816237 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6de0101880 msgr2=0x7f6de0197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:09.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de63fa640 1 --2- 172.21.15.107:0/1292816237 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6de0101880 0x7f6de0197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de63fa640 1 -- 172.21.15.107:0/1292816237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de0102a70 msgr2=0x7f6de0198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:09.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de63fa640 1 --2- 172.21.15.107:0/1292816237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de0102a70 0x7f6de0198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de63fa640 1 -- 172.21.15.107:0/1292816237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6de019f6f0 con 0x7f6de0103430 2024-05-09T08:33:09.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de63fa640 1 --2- 172.21.15.107:0/1292816237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de0103430 0x7f6de019f080 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6dd800cce0 tx=0x7f6dd800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:09.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6dd6ffd640 1 -- 172.21.15.107:0/1292816237 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd800f450 con 0x7f6de0103430 2024-05-09T08:33:09.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6dd6ffd640 1 -- 172.21.15.107:0/1292816237 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dd800faa0 con 0x7f6de0103430 2024-05-09T08:33:09.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de7e5d640 1 -- 172.21.15.107:0/1292816237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6de019f9e0 con 0x7f6de0103430 2024-05-09T08:33:09.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6dd6ffd640 1 -- 172.21.15.107:0/1292816237 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd8018b70 con 0x7f6de0103430 2024-05-09T08:33:09.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.219+0000 7f6de7e5d640 1 -- 172.21.15.107:0/1292816237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6de019fee0 con 0x7f6de0103430 2024-05-09T08:33:09.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.220+0000 7f6de7e5d640 1 -- 172.21.15.107:0/1292816237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6de0108ce0 con 0x7f6de0103430 2024-05-09T08:33:09.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.221+0000 7f6dd6ffd640 1 -- 172.21.15.107:0/1292816237 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f6dd8018d10 con 0x7f6de0103430 2024-05-09T08:33:09.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.222+0000 7f6dd6ffd640 1 --2- 172.21.15.107:0/1292816237 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6dcc0a2b90 0x7f6dcc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:09.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.222+0000 7f6de5bf9640 1 --2- 172.21.15.107:0/1292816237 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6dcc0a2b90 0x7f6dcc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:09.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.222+0000 7f6dd6ffd640 1 -- 172.21.15.107:0/1292816237 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f6dd80cf760 con 0x7f6de0103430 2024-05-09T08:33:09.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.223+0000 7f6de5bf9640 1 --2- 172.21.15.107:0/1292816237 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6dcc0a2b90 0x7f6dcc0a5050 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc004770 tx=0x7f6ddc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:09.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.226+0000 7f6dd6ffd640 1 -- 172.21.15.107:0/1292816237 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6dd8014030 con 0x7f6de0103430 2024-05-09T08:33:09.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.402+0000 7f6de7e5d640 1 -- 172.21.15.107:0/1292816237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6de0062430 con 0x7f6de0103430 2024-05-09T08:33:09.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.402+0000 7f6dd6ffd640 1 -- 172.21.15.107:0/1292816237 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f6dd809c340 con 0x7f6de0103430 2024-05-09T08:33:09.401 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:33:09.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.404+0000 7f6de7e5d640 1 -- 172.21.15.107:0/1292816237 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6dcc0a2b90 msgr2=0x7f6dcc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:09.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.404+0000 7f6de7e5d640 1 --2- 172.21.15.107:0/1292816237 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6dcc0a2b90 0x7f6dcc0a5050 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc004770 tx=0x7f6ddc000bc0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.404+0000 7f6de7e5d640 1 -- 172.21.15.107:0/1292816237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de0103430 msgr2=0x7f6de019f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:09.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.404+0000 7f6de7e5d640 1 --2- 172.21.15.107:0/1292816237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de0103430 0x7f6de019f080 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6dd800cce0 tx=0x7f6dd800a620 comp rx=0 tx=0).stop 2024-05-09T08:33:09.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.404+0000 7f6de7e5d640 1 -- 172.21.15.107:0/1292816237 shutdown_connections 2024-05-09T08:33:09.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.404+0000 7f6de7e5d640 1 --2- 172.21.15.107:0/1292816237 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f6dcc0a2b90 0x7f6dcc0a5050 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.404+0000 7f6de7e5d640 1 --2- 172.21.15.107:0/1292816237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de0103430 0x7f6de019f080 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.404+0000 7f6de7e5d640 1 --2- 172.21.15.107:0/1292816237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de0102a70 0x7f6de0198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.404+0000 7f6de7e5d640 1 --2- 172.21.15.107:0/1292816237 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6de0101880 0x7f6de0197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.404+0000 7f6de7e5d640 1 -- 172.21.15.107:0/1292816237 >> 172.21.15.107:0/1292816237 conn(0x7f6de00fce10 msgr2=0x7f6de0105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:09.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.404+0000 7f6de7e5d640 1 -- 172.21.15.107:0/1292816237 shutdown_connections 2024-05-09T08:33:09.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.404+0000 7f6de7e5d640 1 -- 172.21.15.107:0/1292816237 wait complete. 2024-05-09T08:33:09.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:33:09.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.519+0000 7f967915c640 1 -- 172.21.15.107:0/3821314429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9674109730 msgr2=0x7f967410bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:09.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.519+0000 7f967915c640 1 --2- 172.21.15.107:0/3821314429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9674109730 0x7f967410bbc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f966c009ca0 tx=0x7f966c02a950 comp rx=0 tx=0).stop 2024-05-09T08:33:09.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.519+0000 7f967915c640 1 -- 172.21.15.107:0/3821314429 shutdown_connections 2024-05-09T08:33:09.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.519+0000 7f967915c640 1 --2- 172.21.15.107:0/3821314429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9674109730 0x7f967410bbc0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.519+0000 7f967915c640 1 --2- 172.21.15.107:0/3821314429 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9674074210 0x7f9674074670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.519+0000 7f967915c640 1 --2- 172.21.15.107:0/3821314429 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96740757f0 0x7f9674073c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.519+0000 7f967915c640 1 -- 172.21.15.107:0/3821314429 >> 172.21.15.107:0/3821314429 conn(0x7f96740fce10 msgr2=0x7f96740ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:09.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.519+0000 7f967915c640 1 -- 172.21.15.107:0/3821314429 shutdown_connections 2024-05-09T08:33:09.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.519+0000 7f967915c640 1 -- 172.21.15.107:0/3821314429 wait complete. 2024-05-09T08:33:09.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.520+0000 7f967915c640 1 Processor -- start 2024-05-09T08:33:09.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.520+0000 7f967915c640 1 -- start start 2024-05-09T08:33:09.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.520+0000 7f967915c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9674074210 0x7f9674199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:09.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.520+0000 7f967915c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96740757f0 0x7f967419a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:09.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.520+0000 7f967915c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9674109730 0x7f96741a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:09.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.520+0000 7f967915c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f967410f5c0 con 0x7f96740757f0 2024-05-09T08:33:09.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.520+0000 7f967915c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f967410f440 con 0x7f9674074210 2024-05-09T08:33:09.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.520+0000 7f967915c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f967410f740 con 0x7f9674109730 2024-05-09T08:33:09.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.520+0000 7f966bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96740757f0 0x7f967419a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:09.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.520+0000 7f966bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96740757f0 0x7f967419a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38368/0 (socket says 172.21.15.107:38368) 2024-05-09T08:33:09.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.520+0000 7f966bfff640 1 -- 172.21.15.107:0/2289772602 learned_addr learned my addr 172.21.15.107:0/2289772602 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:09.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.520+0000 7f9672d9d640 1 --2- 172.21.15.107:0/2289772602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9674074210 0x7f9674199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:09.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.520+0000 7f967359e640 1 --2- 172.21.15.107:0/2289772602 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9674109730 0x7f96741a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:09.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.520+0000 7f966bfff640 1 -- 172.21.15.107:0/2289772602 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9674109730 msgr2=0x7f96741a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:09.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.520+0000 7f966bfff640 1 --2- 172.21.15.107:0/2289772602 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9674109730 0x7f96741a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.521+0000 7f966bfff640 1 -- 172.21.15.107:0/2289772602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9674074210 msgr2=0x7f9674199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:09.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.521+0000 7f966bfff640 1 --2- 172.21.15.107:0/2289772602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9674074210 0x7f9674199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.521+0000 7f966bfff640 1 -- 172.21.15.107:0/2289772602 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96741a1840 con 0x7f96740757f0 2024-05-09T08:33:09.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.521+0000 7f966bfff640 1 --2- 172.21.15.107:0/2289772602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96740757f0 0x7f967419a170 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f965c00c920 tx=0x7f965c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:09.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.521+0000 7f9670d99640 1 -- 172.21.15.107:0/2289772602 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965c007c70 con 0x7f96740757f0 2024-05-09T08:33:09.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.521+0000 7f967915c640 1 -- 172.21.15.107:0/2289772602 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96741a1b30 con 0x7f96740757f0 2024-05-09T08:33:09.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.521+0000 7f9670d99640 1 -- 172.21.15.107:0/2289772602 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f965c007e10 con 0x7f96740757f0 2024-05-09T08:33:09.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.521+0000 7f9670d99640 1 -- 172.21.15.107:0/2289772602 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965c018820 con 0x7f96740757f0 2024-05-09T08:33:09.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.521+0000 7f967915c640 1 -- 172.21.15.107:0/2289772602 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96741a20d0 con 0x7f96740757f0 2024-05-09T08:33:09.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.522+0000 7f967915c640 1 -- 172.21.15.107:0/2289772602 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96741084f0 con 0x7f96740757f0 2024-05-09T08:33:09.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.523+0000 7f9670d99640 1 -- 172.21.15.107:0/2289772602 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f965c0189c0 con 0x7f96740757f0 2024-05-09T08:33:09.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.524+0000 7f9670d99640 1 --2- 172.21.15.107:0/2289772602 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f96600a2c10 0x7f96600a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:09.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.524+0000 7f9672d9d640 1 --2- 172.21.15.107:0/2289772602 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f96600a2c10 0x7f96600a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:09.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.524+0000 7f9670d99640 1 -- 172.21.15.107:0/2289772602 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f965c0cf470 con 0x7f96740757f0 2024-05-09T08:33:09.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.524+0000 7f9672d9d640 1 --2- 172.21.15.107:0/2289772602 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f96600a2c10 0x7f96600a50d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f96640058c0 tx=0x7f966400a250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:09.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.528+0000 7f9670d99640 1 -- 172.21.15.107:0/2289772602 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f965c014030 con 0x7f96740757f0 2024-05-09T08:33:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.706+0000 7f967915c640 1 -- 172.21.15.107:0/2289772602 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9674062430 con 0x7f96740757f0 2024-05-09T08:33:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.707+0000 7f9670d99640 1 -- 172.21.15.107:0/2289772602 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f965c09c050 con 0x7f96740757f0 2024-05-09T08:33:09.706 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-05-09T08:33:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.709+0000 7f967915c640 1 -- 172.21.15.107:0/2289772602 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f96600a2c10 msgr2=0x7f96600a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.709+0000 7f967915c640 1 --2- 172.21.15.107:0/2289772602 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f96600a2c10 0x7f96600a50d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f96640058c0 tx=0x7f966400a250 comp rx=0 tx=0).stop 2024-05-09T08:33:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.709+0000 7f967915c640 1 -- 172.21.15.107:0/2289772602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96740757f0 msgr2=0x7f967419a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.709+0000 7f967915c640 1 --2- 172.21.15.107:0/2289772602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96740757f0 0x7f967419a170 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f965c00c920 tx=0x7f965c00cde0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.709+0000 7f967915c640 1 -- 172.21.15.107:0/2289772602 shutdown_connections 2024-05-09T08:33:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.709+0000 7f967915c640 1 --2- 172.21.15.107:0/2289772602 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f96600a2c10 0x7f96600a50d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.709+0000 7f967915c640 1 --2- 172.21.15.107:0/2289772602 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9674109730 0x7f96741a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.709+0000 7f967915c640 1 --2- 172.21.15.107:0/2289772602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96740757f0 0x7f967419a170 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.709+0000 7f967915c640 1 --2- 172.21.15.107:0/2289772602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9674074210 0x7f9674199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.709+0000 7f967915c640 1 -- 172.21.15.107:0/2289772602 >> 172.21.15.107:0/2289772602 conn(0x7f96740fce10 msgr2=0x7f96741056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.709+0000 7f967915c640 1 -- 172.21.15.107:0/2289772602 shutdown_connections 2024-05-09T08:33:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.709+0000 7f967915c640 1 -- 172.21.15.107:0/2289772602 wait complete. 2024-05-09T08:33:09.718 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:09 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1950617173' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:09.718 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:09 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3921209297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:33:09.718 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:09 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1292816237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:33:09.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:33:09.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.824+0000 7f392f59e640 1 -- 172.21.15.107:0/4215612938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3930102220 msgr2=0x7f3930108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:09.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.824+0000 7f392f59e640 1 --2- 172.21.15.107:0/4215612938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3930102220 0x7f3930108ab0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3920009900 tx=0x7f392002a410 comp rx=0 tx=0).stop 2024-05-09T08:33:09.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.824+0000 7f392f59e640 1 -- 172.21.15.107:0/4215612938 shutdown_connections 2024-05-09T08:33:09.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.824+0000 7f392f59e640 1 --2- 172.21.15.107:0/4215612938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3930102220 0x7f3930108ab0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.824+0000 7f392f59e640 1 --2- 172.21.15.107:0/4215612938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3930101860 0x7f3930101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.825+0000 7f392f59e640 1 --2- 172.21.15.107:0/4215612938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3930100670 0x7f3930100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.825+0000 7f392f59e640 1 -- 172.21.15.107:0/4215612938 >> 172.21.15.107:0/4215612938 conn(0x7f39300fbbe0 msgr2=0x7f39300fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:09.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.825+0000 7f392f59e640 1 -- 172.21.15.107:0/4215612938 shutdown_connections 2024-05-09T08:33:09.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.825+0000 7f392f59e640 1 -- 172.21.15.107:0/4215612938 wait complete. 2024-05-09T08:33:09.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.825+0000 7f392f59e640 1 Processor -- start 2024-05-09T08:33:09.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.825+0000 7f392f59e640 1 -- start start 2024-05-09T08:33:09.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.825+0000 7f392f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3930100670 0x7f393006a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:09.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.825+0000 7f392f59e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3930101860 0x7f393019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:09.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.825+0000 7f392f59e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3930102220 0x7f393019ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:09.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.825+0000 7f392f59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f393010c1a0 con 0x7f3930100670 2024-05-09T08:33:09.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f392f59e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f393010c020 con 0x7f3930102220 2024-05-09T08:33:09.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f392f59e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f393010c320 con 0x7f3930101860 2024-05-09T08:33:09.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f392e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3930100670 0x7f393006a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:09.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f392e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3930100670 0x7f393006a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38384/0 (socket says 172.21.15.107:38384) 2024-05-09T08:33:09.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f392e59c640 1 -- 172.21.15.107:0/482478 learned_addr learned my addr 172.21.15.107:0/482478 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:09.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f392dd9b640 1 --2- 172.21.15.107:0/482478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3930101860 0x7f393019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:09.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f392ed9d640 1 --2- 172.21.15.107:0/482478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3930102220 0x7f393019ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:09.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f392e59c640 1 -- 172.21.15.107:0/482478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3930101860 msgr2=0x7f393019c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:09.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f392e59c640 1 --2- 172.21.15.107:0/482478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3930101860 0x7f393019c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f392e59c640 1 -- 172.21.15.107:0/482478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3930102220 msgr2=0x7f393019ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f392e59c640 1 --2- 172.21.15.107:0/482478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3930102220 0x7f393019ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f392e59c640 1 -- 172.21.15.107:0/482478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f393019f590 con 0x7f3930100670 2024-05-09T08:33:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f392ed9d640 1 --2- 172.21.15.107:0/482478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3930102220 0x7f393019ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f392e59c640 1 --2- 172.21.15.107:0/482478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3930100670 0x7f393006a920 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f392800eaa0 tx=0x7f392800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f391f7fe640 1 -- 172.21.15.107:0/482478 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3928016070 con 0x7f3930100670 2024-05-09T08:33:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f391f7fe640 1 -- 172.21.15.107:0/482478 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3928003e20 con 0x7f3930100670 2024-05-09T08:33:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f392f59e640 1 -- 172.21.15.107:0/482478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f393019f820 con 0x7f3930100670 2024-05-09T08:33:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f391f7fe640 1 -- 172.21.15.107:0/482478 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39280116d0 con 0x7f3930100670 2024-05-09T08:33:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.826+0000 7f392f59e640 1 -- 172.21.15.107:0/482478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f393019fd20 con 0x7f3930100670 2024-05-09T08:33:09.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.827+0000 7f392f59e640 1 -- 172.21.15.107:0/482478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3930107990 con 0x7f3930100670 2024-05-09T08:33:09.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.828+0000 7f391f7fe640 1 -- 172.21.15.107:0/482478 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f3928019070 con 0x7f3930100670 2024-05-09T08:33:09.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.829+0000 7f391f7fe640 1 --2- 172.21.15.107:0/482478 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f39180a2b90 0x7f39180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:09.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.829+0000 7f392dd9b640 1 --2- 172.21.15.107:0/482478 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f39180a2b90 0x7f39180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:09.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.829+0000 7f391f7fe640 1 -- 172.21.15.107:0/482478 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f39280cf580 con 0x7f3930100670 2024-05-09T08:33:09.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.830+0000 7f392dd9b640 1 --2- 172.21.15.107:0/482478 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f39180a2b90 0x7f39180a5050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3924005fd0 tx=0x7f3924005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:09.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:09.834+0000 7f391f7fe640 1 -- 172.21.15.107:0/482478 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f392809c160 con 0x7f3930100670 2024-05-09T08:33:09.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:09 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1950617173' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:09.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:09 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3921209297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:33:09.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:09 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1292816237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:33:09.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:09 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1950617173' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:09.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:09 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3921209297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:33:09.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:09 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1292816237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:33:10.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.009+0000 7f392f59e640 1 -- 172.21.15.107:0/482478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3930062430 con 0x7f3930100670 2024-05-09T08:33:10.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.009+0000 7f391f7fe640 1 -- 172.21.15.107:0/482478 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f392809ba60 con 0x7f3930100670 2024-05-09T08:33:10.009 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:33:10.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.011+0000 7f392f59e640 1 -- 172.21.15.107:0/482478 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f39180a2b90 msgr2=0x7f39180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.011+0000 7f392f59e640 1 --2- 172.21.15.107:0/482478 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f39180a2b90 0x7f39180a5050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3924005fd0 tx=0x7f3924005850 comp rx=0 tx=0).stop 2024-05-09T08:33:10.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.011+0000 7f392f59e640 1 -- 172.21.15.107:0/482478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3930100670 msgr2=0x7f393006a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.011+0000 7f392f59e640 1 --2- 172.21.15.107:0/482478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3930100670 0x7f393006a920 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f392800eaa0 tx=0x7f392800ef60 comp rx=0 tx=0).stop 2024-05-09T08:33:10.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.011+0000 7f392f59e640 1 -- 172.21.15.107:0/482478 shutdown_connections 2024-05-09T08:33:10.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.011+0000 7f392f59e640 1 --2- 172.21.15.107:0/482478 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f39180a2b90 0x7f39180a5050 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.011+0000 7f392f59e640 1 --2- 172.21.15.107:0/482478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3930102220 0x7f393019ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.011+0000 7f392f59e640 1 --2- 172.21.15.107:0/482478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3930101860 0x7f393019c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.011+0000 7f392f59e640 1 --2- 172.21.15.107:0/482478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3930100670 0x7f393006a920 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.011+0000 7f392f59e640 1 -- 172.21.15.107:0/482478 >> 172.21.15.107:0/482478 conn(0x7f39300fbbe0 msgr2=0x7f3930104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:10.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.012+0000 7f392f59e640 1 -- 172.21.15.107:0/482478 shutdown_connections 2024-05-09T08:33:10.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.012+0000 7f392f59e640 1 -- 172.21.15.107:0/482478 wait complete. 2024-05-09T08:33:10.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:33:10.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.124+0000 7fbc81fc8640 1 -- 172.21.15.107:0/2343922691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc7c102320 msgr2=0x7fbc7c108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.124+0000 7fbc81fc8640 1 --2- 172.21.15.107:0/2343922691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc7c102320 0x7fbc7c108bb0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fbc74009ca0 tx=0x7fbc7402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.124+0000 7fbc81fc8640 1 -- 172.21.15.107:0/2343922691 shutdown_connections 2024-05-09T08:33:10.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.124+0000 7fbc81fc8640 1 --2- 172.21.15.107:0/2343922691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc7c102320 0x7fbc7c108bb0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.124+0000 7fbc81fc8640 1 --2- 172.21.15.107:0/2343922691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc7c101960 0x7fbc7c101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.124+0000 7fbc81fc8640 1 --2- 172.21.15.107:0/2343922691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc7c100770 0x7fbc7c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.124+0000 7fbc81fc8640 1 -- 172.21.15.107:0/2343922691 >> 172.21.15.107:0/2343922691 conn(0x7fbc7c0fbce0 msgr2=0x7fbc7c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:10.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.124+0000 7fbc81fc8640 1 -- 172.21.15.107:0/2343922691 shutdown_connections 2024-05-09T08:33:10.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.124+0000 7fbc81fc8640 1 -- 172.21.15.107:0/2343922691 wait complete. 2024-05-09T08:33:10.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.124+0000 7fbc81fc8640 1 Processor -- start 2024-05-09T08:33:10.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.124+0000 7fbc81fc8640 1 -- start start 2024-05-09T08:33:10.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc81fc8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc7c100770 0x7fbc7c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:10.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc81fc8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc7c101960 0x7fbc7c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:10.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc81fc8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc7c102320 0x7fbc7c19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:10.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc81fc8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c10c310 con 0x7fbc7c100770 2024-05-09T08:33:10.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc81fc8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c10c190 con 0x7fbc7c102320 2024-05-09T08:33:10.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc81fc8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c10c490 con 0x7fbc7c101960 2024-05-09T08:33:10.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc7b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc7c100770 0x7fbc7c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:10.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc7b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc7c100770 0x7fbc7c197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38408/0 (socket says 172.21.15.107:38408) 2024-05-09T08:33:10.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc7b7fe640 1 -- 172.21.15.107:0/3387120931 learned_addr learned my addr 172.21.15.107:0/3387120931 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:10.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc7affd640 1 --2- 172.21.15.107:0/3387120931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc7c101960 0x7fbc7c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:10.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc7bfff640 1 --2- 172.21.15.107:0/3387120931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc7c102320 0x7fbc7c19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:10.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc7b7fe640 1 -- 172.21.15.107:0/3387120931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc7c101960 msgr2=0x7fbc7c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc7b7fe640 1 --2- 172.21.15.107:0/3387120931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc7c101960 0x7fbc7c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc7b7fe640 1 -- 172.21.15.107:0/3387120931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc7c102320 msgr2=0x7fbc7c19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc7b7fe640 1 --2- 172.21.15.107:0/3387120931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc7c102320 0x7fbc7c19f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc7b7fe640 1 -- 172.21.15.107:0/3387120931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7c19f700 con 0x7fbc7c100770 2024-05-09T08:33:10.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc7bfff640 1 --2- 172.21.15.107:0/3387120931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc7c102320 0x7fbc7c19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:10.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc7b7fe640 1 --2- 172.21.15.107:0/3387120931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc7c100770 0x7fbc7c197ae0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00b450 tx=0x7fbc6c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:10.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc78ff9640 1 -- 172.21.15.107:0/3387120931 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c015020 con 0x7fbc7c100770 2024-05-09T08:33:10.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc81fc8640 1 -- 172.21.15.107:0/3387120931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc7c19f9f0 con 0x7fbc7c100770 2024-05-09T08:33:10.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc78ff9640 1 -- 172.21.15.107:0/3387120931 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc6c011d40 con 0x7fbc7c100770 2024-05-09T08:33:10.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc78ff9640 1 -- 172.21.15.107:0/3387120931 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c00f340 con 0x7fbc7c100770 2024-05-09T08:33:10.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.125+0000 7fbc81fc8640 1 -- 172.21.15.107:0/3387120931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc7c19ff90 con 0x7fbc7c100770 2024-05-09T08:33:10.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.126+0000 7fbc81fc8640 1 -- 172.21.15.107:0/3387120931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc7c068680 con 0x7fbc7c100770 2024-05-09T08:33:10.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.127+0000 7fbc78ff9640 1 -- 172.21.15.107:0/3387120931 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fbc6c018020 con 0x7fbc7c100770 2024-05-09T08:33:10.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.128+0000 7fbc78ff9640 1 --2- 172.21.15.107:0/3387120931 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbc640a2b90 0x7fbc640a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:10.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.128+0000 7fbc7affd640 1 --2- 172.21.15.107:0/3387120931 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbc640a2b90 0x7fbc640a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:10.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.128+0000 7fbc78ff9640 1 -- 172.21.15.107:0/3387120931 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fbc6c0cf4a0 con 0x7fbc7c100770 2024-05-09T08:33:10.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.129+0000 7fbc7affd640 1 --2- 172.21.15.107:0/3387120931 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbc640a2b90 0x7fbc640a5050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbc70005fd0 tx=0x7fbc70005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:10.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.131+0000 7fbc78ff9640 1 -- 172.21.15.107:0/3387120931 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbc7c068680 con 0x7fbc7c100770 2024-05-09T08:33:10.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.346+0000 7fbc81fc8640 1 -- 172.21.15.107:0/3387120931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc7c1a01b0 con 0x7fbc7c100770 2024-05-09T08:33:10.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.346+0000 7fbc78ff9640 1 -- 172.21.15.107:0/3387120931 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11887 (secure 0 0 0) 0x7fbc6c01f020 con 0x7fbc7c100770 2024-05-09T08:33:10.346 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:10.346 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:31:21.056050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24478,24499]},"id":1}]} 2024-05-09T08:33:10.346 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 24 2024-05-09T08:33:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.348+0000 7fbc81fc8640 1 -- 172.21.15.107:0/3387120931 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbc640a2b90 msgr2=0x7fbc640a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.348+0000 7fbc81fc8640 1 --2- 172.21.15.107:0/3387120931 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbc640a2b90 0x7fbc640a5050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbc70005fd0 tx=0x7fbc70005850 comp rx=0 tx=0).stop 2024-05-09T08:33:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.348+0000 7fbc81fc8640 1 -- 172.21.15.107:0/3387120931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc7c100770 msgr2=0x7fbc7c197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.348+0000 7fbc81fc8640 1 --2- 172.21.15.107:0/3387120931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc7c100770 0x7fbc7c197ae0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00b450 tx=0x7fbc6c00b910 comp rx=0 tx=0).stop 2024-05-09T08:33:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.348+0000 7fbc81fc8640 1 -- 172.21.15.107:0/3387120931 shutdown_connections 2024-05-09T08:33:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.348+0000 7fbc81fc8640 1 --2- 172.21.15.107:0/3387120931 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fbc640a2b90 0x7fbc640a5050 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.348+0000 7fbc81fc8640 1 --2- 172.21.15.107:0/3387120931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc7c102320 0x7fbc7c19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.348+0000 7fbc81fc8640 1 --2- 172.21.15.107:0/3387120931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc7c101960 0x7fbc7c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.348+0000 7fbc81fc8640 1 --2- 172.21.15.107:0/3387120931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc7c100770 0x7fbc7c197ae0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.349+0000 7fbc81fc8640 1 -- 172.21.15.107:0/3387120931 >> 172.21.15.107:0/3387120931 conn(0x7fbc7c0fbce0 msgr2=0x7fbc7c0fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:10.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.349+0000 7fbc81fc8640 1 -- 172.21.15.107:0/3387120931 shutdown_connections 2024-05-09T08:33:10.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.349+0000 7fbc81fc8640 1 -- 172.21.15.107:0/3387120931 wait complete. 2024-05-09T08:33:10.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:33:10.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.462+0000 7f52ea5c8640 1 -- 172.21.15.107:0/606441399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52e4102a90 msgr2=0x7f52e4102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.462+0000 7f52ea5c8640 1 --2- 172.21.15.107:0/606441399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52e4102a90 0x7f52e4102f10 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009900 tx=0x7f52d802a410 comp rx=0 tx=0).stop 2024-05-09T08:33:10.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.462+0000 7f52ea5c8640 1 -- 172.21.15.107:0/606441399 shutdown_connections 2024-05-09T08:33:10.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.462+0000 7f52ea5c8640 1 --2- 172.21.15.107:0/606441399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52e4103450 0x7f52e4109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.462+0000 7f52ea5c8640 1 --2- 172.21.15.107:0/606441399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52e4102a90 0x7f52e4102f10 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.462+0000 7f52ea5c8640 1 --2- 172.21.15.107:0/606441399 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52e41018a0 0x7f52e4101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.462+0000 7f52ea5c8640 1 -- 172.21.15.107:0/606441399 >> 172.21.15.107:0/606441399 conn(0x7f52e40fce10 msgr2=0x7f52e40ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:10.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.462+0000 7f52ea5c8640 1 -- 172.21.15.107:0/606441399 shutdown_connections 2024-05-09T08:33:10.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.462+0000 7f52ea5c8640 1 -- 172.21.15.107:0/606441399 wait complete. 2024-05-09T08:33:10.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.462+0000 7f52ea5c8640 1 Processor -- start 2024-05-09T08:33:10.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.462+0000 7f52ea5c8640 1 -- start start 2024-05-09T08:33:10.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52ea5c8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52e41018a0 0x7f52e4197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:10.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52ea5c8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52e4102a90 0x7f52e4198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:10.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52ea5c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52e4103450 0x7f52e419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:10.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52ea5c8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f52e410d3d0 con 0x7f52e4103450 2024-05-09T08:33:10.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52ea5c8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f52e410d250 con 0x7f52e41018a0 2024-05-09T08:33:10.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52ea5c8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f52e410d550 con 0x7f52e4102a90 2024-05-09T08:33:10.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52e8b65640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52e4103450 0x7f52e419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:10.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52e8b65640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52e4103450 0x7f52e419f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38430/0 (socket says 172.21.15.107:38430) 2024-05-09T08:33:10.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52e8b65640 1 -- 172.21.15.107:0/1896987853 learned_addr learned my addr 172.21.15.107:0/1896987853 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:10.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52e3fff640 1 --2- 172.21.15.107:0/1896987853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52e41018a0 0x7f52e4197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:10.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52e37fe640 1 --2- 172.21.15.107:0/1896987853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52e4102a90 0x7f52e4198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:10.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52e8b65640 1 -- 172.21.15.107:0/1896987853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52e4102a90 msgr2=0x7f52e4198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52e8b65640 1 --2- 172.21.15.107:0/1896987853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52e4102a90 0x7f52e4198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52e8b65640 1 -- 172.21.15.107:0/1896987853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52e41018a0 msgr2=0x7f52e4197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52e8b65640 1 --2- 172.21.15.107:0/1896987853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52e41018a0 0x7f52e4197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52e8b65640 1 -- 172.21.15.107:0/1896987853 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e419f740 con 0x7f52e4103450 2024-05-09T08:33:10.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52e3fff640 1 --2- 172.21.15.107:0/1896987853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52e41018a0 0x7f52e4197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:10.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52e8b65640 1 --2- 172.21.15.107:0/1896987853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52e4103450 0x7f52e419f0d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00eaa0 tx=0x7f52dc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:10.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52e17fa640 1 -- 172.21.15.107:0/1896987853 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc00cdb0 con 0x7f52e4103450 2024-05-09T08:33:10.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52e17fa640 1 -- 172.21.15.107:0/1896987853 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f52dc005020 con 0x7f52e4103450 2024-05-09T08:33:10.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52ea5c8640 1 -- 172.21.15.107:0/1896987853 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52e419fa30 con 0x7f52e4103450 2024-05-09T08:33:10.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52e17fa640 1 -- 172.21.15.107:0/1896987853 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc0116d0 con 0x7f52e4103450 2024-05-09T08:33:10.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.463+0000 7f52ea5c8640 1 -- 172.21.15.107:0/1896987853 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52e419ff30 con 0x7f52e4103450 2024-05-09T08:33:10.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.464+0000 7f52ea5c8640 1 -- 172.21.15.107:0/1896987853 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52e4108c60 con 0x7f52e4103450 2024-05-09T08:33:10.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.465+0000 7f52e17fa640 1 -- 172.21.15.107:0/1896987853 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f52dc0051d0 con 0x7f52e4103450 2024-05-09T08:33:10.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.466+0000 7f52e17fa640 1 --2- 172.21.15.107:0/1896987853 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f52cc0a2b90 0x7f52cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:10.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.466+0000 7f52e3fff640 1 --2- 172.21.15.107:0/1896987853 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f52cc0a2b90 0x7f52cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:10.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.466+0000 7f52e17fa640 1 -- 172.21.15.107:0/1896987853 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f52dc0cf670 con 0x7f52e4103450 2024-05-09T08:33:10.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.467+0000 7f52e3fff640 1 --2- 172.21.15.107:0/1896987853 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f52cc0a2b90 0x7f52cc0a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f52d40058c0 tx=0x7f52d4005ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:10.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.471+0000 7f52e17fa640 1 -- 172.21.15.107:0/1896987853 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f52dc016030 con 0x7f52e4103450 2024-05-09T08:33:10.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.648+0000 7f52ea5c8640 1 -- 172.21.15.107:0/1896987853 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f52e402d400 con 0x7f52e4103450 2024-05-09T08:33:10.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.649+0000 7f52e17fa640 1 -- 172.21.15.107:0/1896987853 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24761 (secure 0 0 0) 0x7f52dc09c250 con 0x7f52e4103450 2024-05-09T08:33:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:10.649 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":93,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:32:57.545099+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:33:10.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.651+0000 7f52ea5c8640 1 -- 172.21.15.107:0/1896987853 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f52cc0a2b90 msgr2=0x7f52cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.651+0000 7f52ea5c8640 1 --2- 172.21.15.107:0/1896987853 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f52cc0a2b90 0x7f52cc0a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f52d40058c0 tx=0x7f52d4005ec0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.651+0000 7f52ea5c8640 1 -- 172.21.15.107:0/1896987853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52e4103450 msgr2=0x7f52e419f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.651+0000 7f52ea5c8640 1 --2- 172.21.15.107:0/1896987853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52e4103450 0x7f52e419f0d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00eaa0 tx=0x7f52dc00ef60 comp rx=0 tx=0).stop 2024-05-09T08:33:10.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.651+0000 7f52ea5c8640 1 -- 172.21.15.107:0/1896987853 shutdown_connections 2024-05-09T08:33:10.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.651+0000 7f52ea5c8640 1 --2- 172.21.15.107:0/1896987853 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f52cc0a2b90 0x7f52cc0a5050 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.651+0000 7f52ea5c8640 1 --2- 172.21.15.107:0/1896987853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52e4103450 0x7f52e419f0d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.651+0000 7f52ea5c8640 1 --2- 172.21.15.107:0/1896987853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52e4102a90 0x7f52e4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.651+0000 7f52ea5c8640 1 --2- 172.21.15.107:0/1896987853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52e41018a0 0x7f52e4197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.651+0000 7f52ea5c8640 1 -- 172.21.15.107:0/1896987853 >> 172.21.15.107:0/1896987853 conn(0x7f52e40fce10 msgr2=0x7f52e4105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:10.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.652+0000 7f52ea5c8640 1 -- 172.21.15.107:0/1896987853 shutdown_connections 2024-05-09T08:33:10.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.652+0000 7f52ea5c8640 1 -- 172.21.15.107:0/1896987853 wait complete. 2024-05-09T08:33:10.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:10 smithi107 ceph-mon[30122]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 1 B/s, 0 objects/s recovering 2024-05-09T08:33:10.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:10 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2289772602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:33:10.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:10 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/482478' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:33:10.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:10 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3387120931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:33:10.662 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:26:21.698338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:30:58.268633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:30:59.345579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:31:03.065991+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:33:10.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:33:10.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.768+0000 7f42a9b09640 1 -- 172.21.15.107:0/1577512025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a4100790 msgr2=0x7f42a4100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.768+0000 7f42a9b09640 1 --2- 172.21.15.107:0/1577512025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a4100790 0x7f42a4100bb0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4294009900 tx=0x7f429402a410 comp rx=0 tx=0).stop 2024-05-09T08:33:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.768+0000 7f42a9b09640 1 -- 172.21.15.107:0/1577512025 shutdown_connections 2024-05-09T08:33:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.768+0000 7f42a9b09640 1 --2- 172.21.15.107:0/1577512025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42a4102340 0x7f42a4108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.768+0000 7f42a9b09640 1 --2- 172.21.15.107:0/1577512025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42a4101980 0x7f42a4101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.768+0000 7f42a9b09640 1 --2- 172.21.15.107:0/1577512025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a4100790 0x7f42a4100bb0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.768+0000 7f42a9b09640 1 -- 172.21.15.107:0/1577512025 >> 172.21.15.107:0/1577512025 conn(0x7f42a40fbd20 msgr2=0x7f42a40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.768+0000 7f42a9b09640 1 -- 172.21.15.107:0/1577512025 shutdown_connections 2024-05-09T08:33:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.768+0000 7f42a9b09640 1 -- 172.21.15.107:0/1577512025 wait complete. 2024-05-09T08:33:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.768+0000 7f42a9b09640 1 Processor -- start 2024-05-09T08:33:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a9b09640 1 -- start start 2024-05-09T08:33:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a9b09640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42a4100790 0x7f42a4197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a9b09640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a4101980 0x7f42a41980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a9b09640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42a4102340 0x7f42a419f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a9b09640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f42a410d420 con 0x7f42a4101980 2024-05-09T08:33:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a9b09640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f42a410d2a0 con 0x7f42a4102340 2024-05-09T08:33:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a9b09640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f42a410d5a0 con 0x7f42a4100790 2024-05-09T08:33:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a4101980 0x7f42a41980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a4101980 0x7f42a41980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34120/0 (socket says 172.21.15.107:34120) 2024-05-09T08:33:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a2ffd640 1 -- 172.21.15.107:0/3018188769 learned_addr learned my addr 172.21.15.107:0/3018188769 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a3fff640 1 --2- 172.21.15.107:0/3018188769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42a4102340 0x7f42a419f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a2ffd640 1 -- 172.21.15.107:0/3018188769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42a4100790 msgr2=0x7f42a4197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a2ffd640 1 --2- 172.21.15.107:0/3018188769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42a4100790 0x7f42a4197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a2ffd640 1 -- 172.21.15.107:0/3018188769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42a4102340 msgr2=0x7f42a419f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a2ffd640 1 --2- 172.21.15.107:0/3018188769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42a4102340 0x7f42a419f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a2ffd640 1 -- 172.21.15.107:0/3018188769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a419f780 con 0x7f42a4101980 2024-05-09T08:33:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a3fff640 1 --2- 172.21.15.107:0/3018188769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42a4102340 0x7f42a419f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a2ffd640 1 --2- 172.21.15.107:0/3018188769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a4101980 0x7f42a41980a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f429800c890 tx=0x7f429800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a0ff9640 1 -- 172.21.15.107:0/3018188769 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4298004220 con 0x7f42a4101980 2024-05-09T08:33:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a9b09640 1 -- 172.21.15.107:0/3018188769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42a419fa70 con 0x7f42a4101980 2024-05-09T08:33:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.769+0000 7f42a0ff9640 1 -- 172.21.15.107:0/3018188769 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f42980043c0 con 0x7f42a4101980 2024-05-09T08:33:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.770+0000 7f42a0ff9640 1 -- 172.21.15.107:0/3018188769 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4298004a00 con 0x7f42a4101980 2024-05-09T08:33:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.770+0000 7f42a9b09640 1 -- 172.21.15.107:0/3018188769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42a41a0040 con 0x7f42a4101980 2024-05-09T08:33:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.770+0000 7f42a9b09640 1 -- 172.21.15.107:0/3018188769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42a4107bd0 con 0x7f42a4101980 2024-05-09T08:33:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.772+0000 7f42a0ff9640 1 -- 172.21.15.107:0/3018188769 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f4298004ba0 con 0x7f42a4101980 2024-05-09T08:33:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.772+0000 7f42a0ff9640 1 --2- 172.21.15.107:0/3018188769 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f428c0a2b90 0x7f428c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.773+0000 7f42a0ff9640 1 -- 172.21.15.107:0/3018188769 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f42980cf560 con 0x7f42a4101980 2024-05-09T08:33:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.773+0000 7f42a37fe640 1 --2- 172.21.15.107:0/3018188769 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f428c0a2b90 0x7f428c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.773+0000 7f42a37fe640 1 --2- 172.21.15.107:0/3018188769 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f428c0a2b90 0x7f428c0a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f429402a930 tx=0x7f42940314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:10.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.776+0000 7f42a0ff9640 1 -- 172.21.15.107:0/3018188769 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4298014030 con 0x7f42a4101980 2024-05-09T08:33:10.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:10 smithi157 ceph-mon[33614]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 1 B/s, 0 objects/s recovering 2024-05-09T08:33:10.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:10 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2289772602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:33:10.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:10 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/482478' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:33:10.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:10 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3387120931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:33:10.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:10 smithi110 ceph-mon[33622]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 1 B/s, 0 objects/s recovering 2024-05-09T08:33:10.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:10 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2289772602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:33:10.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:10 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/482478' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:33:10.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:10 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3387120931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:33:10.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.952+0000 7f42a9b09640 1 -- 172.21.15.107:0/3018188769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f42a402d370 con 0x7f42a4101980 2024-05-09T08:33:10.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.952+0000 7f42a0ff9640 1 -- 172.21.15.107:0/3018188769 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f429809c140 con 0x7f42a4101980 2024-05-09T08:33:10.951 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-05-09T08:33:10.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.954+0000 7f42a9b09640 1 -- 172.21.15.107:0/3018188769 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f428c0a2b90 msgr2=0x7f428c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.954+0000 7f42a9b09640 1 --2- 172.21.15.107:0/3018188769 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f428c0a2b90 0x7f428c0a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f429402a930 tx=0x7f42940314a0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.954+0000 7f42a9b09640 1 -- 172.21.15.107:0/3018188769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a4101980 msgr2=0x7f42a41980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:10.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.954+0000 7f42a9b09640 1 --2- 172.21.15.107:0/3018188769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a4101980 0x7f42a41980a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f429800c890 tx=0x7f429800cd50 comp rx=0 tx=0).stop 2024-05-09T08:33:10.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.954+0000 7f42a9b09640 1 -- 172.21.15.107:0/3018188769 shutdown_connections 2024-05-09T08:33:10.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.954+0000 7f42a9b09640 1 --2- 172.21.15.107:0/3018188769 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f428c0a2b90 0x7f428c0a5050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.954+0000 7f42a9b09640 1 --2- 172.21.15.107:0/3018188769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42a4102340 0x7f42a419f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.954+0000 7f42a9b09640 1 --2- 172.21.15.107:0/3018188769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a4101980 0x7f42a41980a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.954+0000 7f42a9b09640 1 --2- 172.21.15.107:0/3018188769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42a4100790 0x7f42a4197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:10.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.954+0000 7f42a9b09640 1 -- 172.21.15.107:0/3018188769 >> 172.21.15.107:0/3018188769 conn(0x7f42a40fbd20 msgr2=0x7f42a4104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:10.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.954+0000 7f42a9b09640 1 -- 172.21.15.107:0/3018188769 shutdown_connections 2024-05-09T08:33:10.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:10.954+0000 7f42a9b09640 1 -- 172.21.15.107:0/3018188769 wait complete. 2024-05-09T08:33:10.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:33:11.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.071+0000 7ff55c231640 1 -- 172.21.15.107:0/3491589603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff554069a00 msgr2=0x7ff5541003f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.071+0000 7ff55c231640 1 --2- 172.21.15.107:0/3491589603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff554069a00 0x7ff5541003f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff544009900 tx=0x7ff54402a410 comp rx=0 tx=0).stop 2024-05-09T08:33:11.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.071+0000 7ff55c231640 1 -- 172.21.15.107:0/3491589603 shutdown_connections 2024-05-09T08:33:11.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.071+0000 7ff55c231640 1 --2- 172.21.15.107:0/3491589603 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff554101090 0x7ff554101540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.071+0000 7ff55c231640 1 --2- 172.21.15.107:0/3491589603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff554069a00 0x7ff5541003f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.071+0000 7ff55c231640 1 --2- 172.21.15.107:0/3491589603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5540695e0 0x7ff5540ffeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.071+0000 7ff55c231640 1 -- 172.21.15.107:0/3491589603 >> 172.21.15.107:0/3491589603 conn(0x7ff5540fb470 msgr2=0x7ff5540fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:11.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.071+0000 7ff55c231640 1 -- 172.21.15.107:0/3491589603 shutdown_connections 2024-05-09T08:33:11.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.071+0000 7ff55c231640 1 -- 172.21.15.107:0/3491589603 wait complete. 2024-05-09T08:33:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff55c231640 1 Processor -- start 2024-05-09T08:33:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff55c231640 1 -- start start 2024-05-09T08:33:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff55c231640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5540695e0 0x7ff554102eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff55c231640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff554069a00 0x7ff55419ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff55c231640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff554101090 0x7ff55419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff55c231640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff55410c3d0 con 0x7ff554069a00 2024-05-09T08:33:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff55c231640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff55410c250 con 0x7ff554101090 2024-05-09T08:33:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff55c231640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff55410c550 con 0x7ff5540695e0 2024-05-09T08:33:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff5597cc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff554069a00 0x7ff55419ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff5597cc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff554069a00 0x7ff55419ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34128/0 (socket says 172.21.15.107:34128) 2024-05-09T08:33:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff5597cc640 1 -- 172.21.15.107:0/3239739690 learned_addr learned my addr 172.21.15.107:0/3239739690 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff55a7ce640 1 --2- 172.21.15.107:0/3239739690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff554101090 0x7ff55419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff559fcd640 1 --2- 172.21.15.107:0/3239739690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5540695e0 0x7ff554102eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:11.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff5597cc640 1 -- 172.21.15.107:0/3239739690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5540695e0 msgr2=0x7ff554102eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff5597cc640 1 --2- 172.21.15.107:0/3239739690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5540695e0 0x7ff554102eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff5597cc640 1 -- 172.21.15.107:0/3239739690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff554101090 msgr2=0x7ff55419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff5597cc640 1 --2- 172.21.15.107:0/3239739690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff554101090 0x7ff55419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff5597cc640 1 -- 172.21.15.107:0/3239739690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff55419f960 con 0x7ff554069a00 2024-05-09T08:33:11.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff5597cc640 1 --2- 172.21.15.107:0/3239739690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff554069a00 0x7ff55419ca30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff544009230 tx=0x7ff544004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:11.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.072+0000 7ff54affd640 1 -- 172.21.15.107:0/3239739690 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54403f030 con 0x7ff554069a00 2024-05-09T08:33:11.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.073+0000 7ff54affd640 1 -- 172.21.15.107:0/3239739690 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff544003800 con 0x7ff554069a00 2024-05-09T08:33:11.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.073+0000 7ff54affd640 1 -- 172.21.15.107:0/3239739690 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff544035070 con 0x7ff554069a00 2024-05-09T08:33:11.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.073+0000 7ff55c231640 1 -- 172.21.15.107:0/3239739690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff55419fb90 con 0x7ff554069a00 2024-05-09T08:33:11.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.073+0000 7ff55c231640 1 -- 172.21.15.107:0/3239739690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5541a00d0 con 0x7ff554069a00 2024-05-09T08:33:11.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.073+0000 7ff55c231640 1 -- 172.21.15.107:0/3239739690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff554107eb0 con 0x7ff554069a00 2024-05-09T08:33:11.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.075+0000 7ff54affd640 1 -- 172.21.15.107:0/3239739690 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ff5440039b0 con 0x7ff554069a00 2024-05-09T08:33:11.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.075+0000 7ff54affd640 1 --2- 172.21.15.107:0/3239739690 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff5400a2c10 0x7ff5400a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:11.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.075+0000 7ff559fcd640 1 --2- 172.21.15.107:0/3239739690 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff5400a2c10 0x7ff5400a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:11.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.075+0000 7ff54affd640 1 -- 172.21.15.107:0/3239739690 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7ff5440efe70 con 0x7ff554069a00 2024-05-09T08:33:11.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.076+0000 7ff559fcd640 1 --2- 172.21.15.107:0/3239739690 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff5400a2c10 0x7ff5400a50d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff554100c00 tx=0x7ff550008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:11.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.080+0000 7ff54affd640 1 -- 172.21.15.107:0/3239739690 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff5440bca50 con 0x7ff554069a00 2024-05-09T08:33:11.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.255+0000 7ff55c231640 1 -- 172.21.15.107:0/3239739690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff5540624d0 con 0x7ff554069a00 2024-05-09T08:33:11.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.255+0000 7ff54affd640 1 -- 172.21.15.107:0/3239739690 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7ff5440bc350 con 0x7ff554069a00 2024-05-09T08:33:11.254 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:33:11.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.257+0000 7ff55c231640 1 -- 172.21.15.107:0/3239739690 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff5400a2c10 msgr2=0x7ff5400a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.257+0000 7ff55c231640 1 --2- 172.21.15.107:0/3239739690 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff5400a2c10 0x7ff5400a50d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff554100c00 tx=0x7ff550008040 comp rx=0 tx=0).stop 2024-05-09T08:33:11.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.257+0000 7ff55c231640 1 -- 172.21.15.107:0/3239739690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff554069a00 msgr2=0x7ff55419ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.257+0000 7ff55c231640 1 --2- 172.21.15.107:0/3239739690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff554069a00 0x7ff55419ca30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff544009230 tx=0x7ff544004340 comp rx=0 tx=0).stop 2024-05-09T08:33:11.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.257+0000 7ff55c231640 1 -- 172.21.15.107:0/3239739690 shutdown_connections 2024-05-09T08:33:11.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.257+0000 7ff55c231640 1 --2- 172.21.15.107:0/3239739690 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7ff5400a2c10 0x7ff5400a50d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.257+0000 7ff55c231640 1 --2- 172.21.15.107:0/3239739690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff554101090 0x7ff55419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.257+0000 7ff55c231640 1 --2- 172.21.15.107:0/3239739690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff554069a00 0x7ff55419ca30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.257+0000 7ff55c231640 1 --2- 172.21.15.107:0/3239739690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5540695e0 0x7ff554102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.257+0000 7ff55c231640 1 -- 172.21.15.107:0/3239739690 >> 172.21.15.107:0/3239739690 conn(0x7ff5540fb470 msgr2=0x7ff5540fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:11.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.257+0000 7ff55c231640 1 -- 172.21.15.107:0/3239739690 shutdown_connections 2024-05-09T08:33:11.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.257+0000 7ff55c231640 1 -- 172.21.15.107:0/3239739690 wait complete. 2024-05-09T08:33:11.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:33:11.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.372+0000 7f8c574a5640 1 -- 172.21.15.107:0/787237604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c50103430 msgr2=0x7f8c50109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.372+0000 7f8c574a5640 1 --2- 172.21.15.107:0/787237604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c50103430 0x7f8c50109cc0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009ca0 tx=0x7f8c4802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.372+0000 7f8c574a5640 1 -- 172.21.15.107:0/787237604 shutdown_connections 2024-05-09T08:33:11.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.372+0000 7f8c574a5640 1 --2- 172.21.15.107:0/787237604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c50103430 0x7f8c50109cc0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.372+0000 7f8c574a5640 1 --2- 172.21.15.107:0/787237604 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c50102a70 0x7f8c50102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.372+0000 7f8c574a5640 1 --2- 172.21.15.107:0/787237604 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c50101880 0x7f8c50101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.372+0000 7f8c574a5640 1 -- 172.21.15.107:0/787237604 >> 172.21.15.107:0/787237604 conn(0x7f8c500fce10 msgr2=0x7f8c500ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.372+0000 7f8c574a5640 1 -- 172.21.15.107:0/787237604 shutdown_connections 2024-05-09T08:33:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.372+0000 7f8c574a5640 1 -- 172.21.15.107:0/787237604 wait complete. 2024-05-09T08:33:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c574a5640 1 Processor -- start 2024-05-09T08:33:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c574a5640 1 -- start start 2024-05-09T08:33:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c574a5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c50101880 0x7f8c50197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c574a5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c50102a70 0x7f8c50198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c574a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c50103430 0x7f8c5019f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c574a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5010d420 con 0x7f8c50103430 2024-05-09T08:33:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c574a5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5010d2a0 con 0x7f8c50101880 2024-05-09T08:33:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c574a5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5010d5a0 con 0x7f8c50102a70 2024-05-09T08:33:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c55a42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c50103430 0x7f8c5019f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c55a42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c50103430 0x7f8c5019f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34150/0 (socket says 172.21.15.107:34150) 2024-05-09T08:33:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c55a42640 1 -- 172.21.15.107:0/24296493 learned_addr learned my addr 172.21.15.107:0/24296493 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c55241640 1 --2- 172.21.15.107:0/24296493 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c50101880 0x7f8c50197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c55a42640 1 -- 172.21.15.107:0/24296493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c50102a70 msgr2=0x7f8c50198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c54a40640 1 --2- 172.21.15.107:0/24296493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c50102a70 0x7f8c50198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c55a42640 1 --2- 172.21.15.107:0/24296493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c50102a70 0x7f8c50198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c55a42640 1 -- 172.21.15.107:0/24296493 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c50101880 msgr2=0x7f8c50197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c55a42640 1 --2- 172.21.15.107:0/24296493 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c50101880 0x7f8c50197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c55a42640 1 -- 172.21.15.107:0/24296493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c5019f740 con 0x7f8c50103430 2024-05-09T08:33:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c54a40640 1 --2- 172.21.15.107:0/24296493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c50102a70 0x7f8c50198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:11.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.373+0000 7f8c55241640 1 --2- 172.21.15.107:0/24296493 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c50101880 0x7f8c50197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:11.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.374+0000 7f8c55a42640 1 --2- 172.21.15.107:0/24296493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c50103430 0x7f8c5019f0d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009dc0 tx=0x7f8c48003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:11.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.374+0000 7f8c467fc640 1 -- 172.21.15.107:0/24296493 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c48032ec0 con 0x7f8c50103430 2024-05-09T08:33:11.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.374+0000 7f8c467fc640 1 -- 172.21.15.107:0/24296493 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c480044f0 con 0x7f8c50103430 2024-05-09T08:33:11.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.374+0000 7f8c574a5640 1 -- 172.21.15.107:0/24296493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c5019f9d0 con 0x7f8c50103430 2024-05-09T08:33:11.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.374+0000 7f8c467fc640 1 -- 172.21.15.107:0/24296493 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c48030d40 con 0x7f8c50103430 2024-05-09T08:33:11.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.374+0000 7f8c574a5640 1 -- 172.21.15.107:0/24296493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c5019ff20 con 0x7f8c50103430 2024-05-09T08:33:11.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.375+0000 7f8c574a5640 1 -- 172.21.15.107:0/24296493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c50108c20 con 0x7f8c50103430 2024-05-09T08:33:11.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.376+0000 7f8c467fc640 1 -- 172.21.15.107:0/24296493 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f8c48004040 con 0x7f8c50103430 2024-05-09T08:33:11.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.376+0000 7f8c467fc640 1 --2- 172.21.15.107:0/24296493 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8c3c0a2d30 0x7f8c3c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:11.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.377+0000 7f8c55241640 1 --2- 172.21.15.107:0/24296493 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8c3c0a2d30 0x7f8c3c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:11.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.377+0000 7f8c467fc640 1 -- 172.21.15.107:0/24296493 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f8c480f0c40 con 0x7f8c50103430 2024-05-09T08:33:11.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.377+0000 7f8c55241640 1 --2- 172.21.15.107:0/24296493 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8c3c0a2d30 0x7f8c3c0a51f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8c501028d0 tx=0x7f8c4c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:11.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.381+0000 7f8c467fc640 1 -- 172.21.15.107:0/24296493 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8c480bd820 con 0x7f8c50103430 2024-05-09T08:33:11.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.559+0000 7f8c574a5640 1 -- 172.21.15.107:0/24296493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8c50062430 con 0x7f8c50103430 2024-05-09T08:33:11.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.560+0000 7f8c467fc640 1 -- 172.21.15.107:0/24296493 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f8c480bd120 con 0x7f8c50103430 2024-05-09T08:33:11.559 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-05-09T08:33:11.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.562+0000 7f8c574a5640 1 -- 172.21.15.107:0/24296493 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8c3c0a2d30 msgr2=0x7f8c3c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.562+0000 7f8c574a5640 1 --2- 172.21.15.107:0/24296493 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8c3c0a2d30 0x7f8c3c0a51f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8c501028d0 tx=0x7f8c4c008040 comp rx=0 tx=0).stop 2024-05-09T08:33:11.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.562+0000 7f8c574a5640 1 -- 172.21.15.107:0/24296493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c50103430 msgr2=0x7f8c5019f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.562+0000 7f8c574a5640 1 --2- 172.21.15.107:0/24296493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c50103430 0x7f8c5019f0d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009dc0 tx=0x7f8c48003ba0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.562+0000 7f8c574a5640 1 -- 172.21.15.107:0/24296493 shutdown_connections 2024-05-09T08:33:11.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.562+0000 7f8c574a5640 1 --2- 172.21.15.107:0/24296493 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f8c3c0a2d30 0x7f8c3c0a51f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.562+0000 7f8c574a5640 1 --2- 172.21.15.107:0/24296493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c50103430 0x7f8c5019f0d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.562+0000 7f8c574a5640 1 --2- 172.21.15.107:0/24296493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c50102a70 0x7f8c50198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.562+0000 7f8c574a5640 1 --2- 172.21.15.107:0/24296493 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c50101880 0x7f8c50197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.562+0000 7f8c574a5640 1 -- 172.21.15.107:0/24296493 >> 172.21.15.107:0/24296493 conn(0x7f8c500fce10 msgr2=0x7f8c50105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:11.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.562+0000 7f8c574a5640 1 -- 172.21.15.107:0/24296493 shutdown_connections 2024-05-09T08:33:11.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.562+0000 7f8c574a5640 1 -- 172.21.15.107:0/24296493 wait complete. 2024-05-09T08:33:11.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:33:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 -- 172.21.15.107:0/3794108164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838100790 msgr2=0x7f7838100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 --2- 172.21.15.107:0/3794108164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838100790 0x7f7838100bb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7834009900 tx=0x7f783402a410 comp rx=0 tx=0).stop 2024-05-09T08:33:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 -- 172.21.15.107:0/3794108164 shutdown_connections 2024-05-09T08:33:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 --2- 172.21.15.107:0/3794108164 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7838102340 0x7f7838108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 --2- 172.21.15.107:0/3794108164 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7838101980 0x7f7838101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 --2- 172.21.15.107:0/3794108164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838100790 0x7f7838100bb0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 -- 172.21.15.107:0/3794108164 >> 172.21.15.107:0/3794108164 conn(0x7f78380fbd20 msgr2=0x7f78380fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 -- 172.21.15.107:0/3794108164 shutdown_connections 2024-05-09T08:33:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 -- 172.21.15.107:0/3794108164 wait complete. 2024-05-09T08:33:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 Processor -- start 2024-05-09T08:33:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 -- start start 2024-05-09T08:33:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7838100790 0x7f7838197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7838101980 0x7f7838197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838102340 0x7f783819f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f783810c2c0 con 0x7f7838102340 2024-05-09T08:33:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f783810c140 con 0x7f7838101980 2024-05-09T08:33:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.678+0000 7f783f93b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f783810c440 con 0x7f7838100790 2024-05-09T08:33:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f783ded8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838102340 0x7f783819f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f783ded8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838102340 0x7f783819f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34156/0 (socket says 172.21.15.107:34156) 2024-05-09T08:33:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f783ded8640 1 -- 172.21.15.107:0/1363141234 learned_addr learned my addr 172.21.15.107:0/1363141234 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f783d6d7640 1 --2- 172.21.15.107:0/1363141234 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7838100790 0x7f7838197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f783ced6640 1 --2- 172.21.15.107:0/1363141234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7838101980 0x7f7838197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f783ded8640 1 -- 172.21.15.107:0/1363141234 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7838100790 msgr2=0x7f7838197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f783ded8640 1 --2- 172.21.15.107:0/1363141234 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7838100790 0x7f7838197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f783ded8640 1 -- 172.21.15.107:0/1363141234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7838101980 msgr2=0x7f7838197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f783ded8640 1 --2- 172.21.15.107:0/1363141234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7838101980 0x7f7838197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f783ded8640 1 -- 172.21.15.107:0/1363141234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783819f6a0 con 0x7f7838102340 2024-05-09T08:33:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f783ced6640 1 --2- 172.21.15.107:0/1363141234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7838101980 0x7f7838197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f783ded8640 1 --2- 172.21.15.107:0/1363141234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838102340 0x7f783819f030 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f783000eaa0 tx=0x7f783000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f782a7fc640 1 -- 172.21.15.107:0/1363141234 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783000cdb0 con 0x7f7838102340 2024-05-09T08:33:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f782a7fc640 1 -- 172.21.15.107:0/1363141234 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7830005020 con 0x7f7838102340 2024-05-09T08:33:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f783f93b640 1 -- 172.21.15.107:0/1363141234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f783819f990 con 0x7f7838102340 2024-05-09T08:33:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f782a7fc640 1 -- 172.21.15.107:0/1363141234 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78300116d0 con 0x7f7838102340 2024-05-09T08:33:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.679+0000 7f783f93b640 1 -- 172.21.15.107:0/1363141234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f783819fee0 con 0x7f7838102340 2024-05-09T08:33:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.680+0000 7f783f93b640 1 -- 172.21.15.107:0/1363141234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78381a01a0 con 0x7f7838102340 2024-05-09T08:33:11.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.682+0000 7f782a7fc640 1 -- 172.21.15.107:0/1363141234 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f78300051d0 con 0x7f7838102340 2024-05-09T08:33:11.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.683+0000 7f782a7fc640 1 --2- 172.21.15.107:0/1363141234 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f78240a2b90 0x7f78240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:11.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.683+0000 7f783d6d7640 1 --2- 172.21.15.107:0/1363141234 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f78240a2b90 0x7f78240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:11.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.683+0000 7f782a7fc640 1 -- 172.21.15.107:0/1363141234 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f78300cf940 con 0x7f7838102340 2024-05-09T08:33:11.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.683+0000 7f783d6d7640 1 --2- 172.21.15.107:0/1363141234 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f78240a2b90 0x7f78240a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7834004770 tx=0x7f7834000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:11.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.686+0000 7f782a7fc640 1 -- 172.21.15.107:0/1363141234 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7830016030 con 0x7f7838102340 2024-05-09T08:33:11.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:11 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1896987853' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:11.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:11 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3018188769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:33:11.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:11 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3239739690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:33:11.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.862+0000 7f783f93b640 1 -- 172.21.15.107:0/1363141234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f78381a0370 con 0x7f7838102340 2024-05-09T08:33:11.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.862+0000 7f782a7fc640 1 -- 172.21.15.107:0/1363141234 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f783009c520 con 0x7f7838102340 2024-05-09T08:33:11.861 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:33:11.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.864+0000 7f783f93b640 1 -- 172.21.15.107:0/1363141234 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f78240a2b90 msgr2=0x7f78240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.864+0000 7f783f93b640 1 --2- 172.21.15.107:0/1363141234 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f78240a2b90 0x7f78240a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7834004770 tx=0x7f7834000bc0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.864+0000 7f783f93b640 1 -- 172.21.15.107:0/1363141234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838102340 msgr2=0x7f783819f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.864+0000 7f783f93b640 1 --2- 172.21.15.107:0/1363141234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838102340 0x7f783819f030 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f783000eaa0 tx=0x7f783000ef60 comp rx=0 tx=0).stop 2024-05-09T08:33:11.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.864+0000 7f783f93b640 1 -- 172.21.15.107:0/1363141234 shutdown_connections 2024-05-09T08:33:11.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.864+0000 7f783f93b640 1 --2- 172.21.15.107:0/1363141234 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f78240a2b90 0x7f78240a5050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.864+0000 7f783f93b640 1 --2- 172.21.15.107:0/1363141234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838102340 0x7f783819f030 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.864+0000 7f783f93b640 1 --2- 172.21.15.107:0/1363141234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7838101980 0x7f7838197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.864+0000 7f783f93b640 1 --2- 172.21.15.107:0/1363141234 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7838100790 0x7f7838197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.864+0000 7f783f93b640 1 -- 172.21.15.107:0/1363141234 >> 172.21.15.107:0/1363141234 conn(0x7f78380fbd20 msgr2=0x7f7838069e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:11.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.864+0000 7f783f93b640 1 -- 172.21.15.107:0/1363141234 shutdown_connections 2024-05-09T08:33:11.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.864+0000 7f783f93b640 1 -- 172.21.15.107:0/1363141234 wait complete. 2024-05-09T08:33:11.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:11 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1896987853' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:11.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:11 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3018188769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:33:11.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:11 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3239739690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:33:11.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:11 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1896987853' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:11.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:11 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3018188769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:33:11.873 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:11 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3239739690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:33:11.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:33:11.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.977+0000 7f3a848bd640 1 -- 172.21.15.107:0/2586741611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a80101880 msgr2=0x7f3a80101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.977+0000 7f3a848bd640 1 --2- 172.21.15.107:0/2586741611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a80101880 0x7f3a80101ca0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009900 tx=0x7f3a7002a410 comp rx=0 tx=0).stop 2024-05-09T08:33:11.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.977+0000 7f3a848bd640 1 -- 172.21.15.107:0/2586741611 shutdown_connections 2024-05-09T08:33:11.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.977+0000 7f3a848bd640 1 --2- 172.21.15.107:0/2586741611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a80103430 0x7f3a80109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.977+0000 7f3a848bd640 1 --2- 172.21.15.107:0/2586741611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a80102a70 0x7f3a80102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.977+0000 7f3a848bd640 1 --2- 172.21.15.107:0/2586741611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a80101880 0x7f3a80101ca0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.977+0000 7f3a848bd640 1 -- 172.21.15.107:0/2586741611 >> 172.21.15.107:0/2586741611 conn(0x7f3a800fce10 msgr2=0x7f3a800ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:11.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.977+0000 7f3a848bd640 1 -- 172.21.15.107:0/2586741611 shutdown_connections 2024-05-09T08:33:11.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.977+0000 7f3a848bd640 1 -- 172.21.15.107:0/2586741611 wait complete. 2024-05-09T08:33:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a848bd640 1 Processor -- start 2024-05-09T08:33:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a848bd640 1 -- start start 2024-05-09T08:33:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a848bd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a80101880 0x7f3a80197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a848bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a80102a70 0x7f3a80198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a848bd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a80103430 0x7f3a8019f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a848bd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8010d420 con 0x7f3a80102a70 2024-05-09T08:33:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a848bd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8010d2a0 con 0x7f3a80103430 2024-05-09T08:33:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a848bd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8010d5a0 con 0x7f3a80101880 2024-05-09T08:33:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a7dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a80102a70 0x7f3a80198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a7dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a80102a70 0x7f3a80198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34172/0 (socket says 172.21.15.107:34172) 2024-05-09T08:33:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a7dd9b640 1 -- 172.21.15.107:0/173676618 learned_addr learned my addr 172.21.15.107:0/173676618 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:11.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a7e59c640 1 --2- 172.21.15.107:0/173676618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a80101880 0x7f3a80197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:11.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a7ed9d640 1 --2- 172.21.15.107:0/173676618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a80103430 0x7f3a8019f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:11.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a7dd9b640 1 -- 172.21.15.107:0/173676618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a80101880 msgr2=0x7f3a80197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a7dd9b640 1 --2- 172.21.15.107:0/173676618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a80101880 0x7f3a80197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a7dd9b640 1 -- 172.21.15.107:0/173676618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a80103430 msgr2=0x7f3a8019f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:11.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a7dd9b640 1 --2- 172.21.15.107:0/173676618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a80103430 0x7f3a8019f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:11.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a7dd9b640 1 -- 172.21.15.107:0/173676618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a8019f770 con 0x7f3a80102a70 2024-05-09T08:33:11.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a7ed9d640 1 --2- 172.21.15.107:0/173676618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a80103430 0x7f3a8019f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:11.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a7dd9b640 1 --2- 172.21.15.107:0/173676618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a80102a70 0x7f3a80198090 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400b4a0 tx=0x7f3a7400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:11.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.978+0000 7f3a6f7fe640 1 -- 172.21.15.107:0/173676618 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74014070 con 0x7f3a80102a70 2024-05-09T08:33:11.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.979+0000 7f3a6f7fe640 1 -- 172.21.15.107:0/173676618 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a74010ac0 con 0x7f3a80102a70 2024-05-09T08:33:11.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.979+0000 7f3a848bd640 1 -- 172.21.15.107:0/173676618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a8019fa60 con 0x7f3a80102a70 2024-05-09T08:33:11.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.979+0000 7f3a6f7fe640 1 -- 172.21.15.107:0/173676618 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74017c30 con 0x7f3a80102a70 2024-05-09T08:33:11.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.979+0000 7f3a848bd640 1 -- 172.21.15.107:0/173676618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a801a0030 con 0x7f3a80102a70 2024-05-09T08:33:11.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.979+0000 7f3a848bd640 1 -- 172.21.15.107:0/173676618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a80108bc0 con 0x7f3a80102a70 2024-05-09T08:33:11.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.981+0000 7f3a6f7fe640 1 -- 172.21.15.107:0/173676618 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f3a74017070 con 0x7f3a80102a70 2024-05-09T08:33:11.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.981+0000 7f3a6f7fe640 1 --2- 172.21.15.107:0/173676618 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3a680a2b90 0x7f3a680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:11.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.981+0000 7f3a7e59c640 1 --2- 172.21.15.107:0/173676618 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3a680a2b90 0x7f3a680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:11.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.981+0000 7f3a6f7fe640 1 -- 172.21.15.107:0/173676618 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f3a740cea90 con 0x7f3a80102a70 2024-05-09T08:33:11.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.982+0000 7f3a7e59c640 1 --2- 172.21.15.107:0/173676618 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3a680a2b90 0x7f3a680a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3a70004770 tx=0x7f3a70000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:11.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:11.986+0000 7f3a6f7fe640 1 -- 172.21.15.107:0/173676618 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3a7409b670 con 0x7f3a80102a70 2024-05-09T08:33:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.197+0000 7f3a848bd640 1 -- 172.21.15.107:0/173676618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a8002d1c0 con 0x7f3a80102a70 2024-05-09T08:33:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.197+0000 7f3a6f7fe640 1 -- 172.21.15.107:0/173676618 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11887 (secure 0 0 0) 0x7f3a7409af70 con 0x7f3a80102a70 2024-05-09T08:33:12.197 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:12.197 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:31:21.056050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24478,24499]},"id":1}]} 2024-05-09T08:33:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 24 2024-05-09T08:33:12.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.199+0000 7f3a848bd640 1 -- 172.21.15.107:0/173676618 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3a680a2b90 msgr2=0x7f3a680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.199+0000 7f3a848bd640 1 --2- 172.21.15.107:0/173676618 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3a680a2b90 0x7f3a680a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3a70004770 tx=0x7f3a70000bc0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.199+0000 7f3a848bd640 1 -- 172.21.15.107:0/173676618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a80102a70 msgr2=0x7f3a80198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.199+0000 7f3a848bd640 1 --2- 172.21.15.107:0/173676618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a80102a70 0x7f3a80198090 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400b4a0 tx=0x7f3a7400b960 comp rx=0 tx=0).stop 2024-05-09T08:33:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.199+0000 7f3a848bd640 1 -- 172.21.15.107:0/173676618 shutdown_connections 2024-05-09T08:33:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.199+0000 7f3a848bd640 1 --2- 172.21.15.107:0/173676618 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f3a680a2b90 0x7f3a680a5050 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.199+0000 7f3a848bd640 1 --2- 172.21.15.107:0/173676618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a80103430 0x7f3a8019f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.199+0000 7f3a848bd640 1 --2- 172.21.15.107:0/173676618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a80102a70 0x7f3a80198090 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.199+0000 7f3a848bd640 1 --2- 172.21.15.107:0/173676618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a80101880 0x7f3a80197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.199+0000 7f3a848bd640 1 -- 172.21.15.107:0/173676618 >> 172.21.15.107:0/173676618 conn(0x7f3a800fce10 msgr2=0x7f3a80105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.199+0000 7f3a848bd640 1 -- 172.21.15.107:0/173676618 shutdown_connections 2024-05-09T08:33:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.199+0000 7f3a848bd640 1 -- 172.21.15.107:0/173676618 wait complete. 2024-05-09T08:33:12.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:33:12.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.312+0000 7f06c6db7640 1 -- 172.21.15.107:0/1524018589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c0102f40 msgr2=0x7f06c0103360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.312+0000 7f06c6db7640 1 --2- 172.21.15.107:0/1524018589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c0102f40 0x7f06c0103360 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009ca0 tx=0x7f06a802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.312+0000 7f06c6db7640 1 -- 172.21.15.107:0/1524018589 shutdown_connections 2024-05-09T08:33:12.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.312+0000 7f06c6db7640 1 --2- 172.21.15.107:0/1524018589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06c0100370 0x7f06c0109f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.312+0000 7f06c6db7640 1 --2- 172.21.15.107:0/1524018589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06c00ff9b0 0x7f06c00ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.312+0000 7f06c6db7640 1 --2- 172.21.15.107:0/1524018589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c0102f40 0x7f06c0103360 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.312+0000 7f06c6db7640 1 -- 172.21.15.107:0/1524018589 >> 172.21.15.107:0/1524018589 conn(0x7f06c00fb510 msgr2=0x7f06c00fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.312+0000 7f06c6db7640 1 -- 172.21.15.107:0/1524018589 shutdown_connections 2024-05-09T08:33:12.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.312+0000 7f06c6db7640 1 -- 172.21.15.107:0/1524018589 wait complete. 2024-05-09T08:33:12.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c6db7640 1 Processor -- start 2024-05-09T08:33:12.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c6db7640 1 -- start start 2024-05-09T08:33:12.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c6db7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c00ff9b0 0x7f06c0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c6db7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06c0100370 0x7f06c0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c6db7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06c0102f40 0x7f06c019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c6db7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f06c010d420 con 0x7f06c00ff9b0 2024-05-09T08:33:12.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c6db7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f06c010d2a0 con 0x7f06c0100370 2024-05-09T08:33:12.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c6db7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f06c010d5a0 con 0x7f06c0102f40 2024-05-09T08:33:12.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c4b53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c00ff9b0 0x7f06c0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c4b53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c00ff9b0 0x7f06c0197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34192/0 (socket says 172.21.15.107:34192) 2024-05-09T08:33:12.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c4b53640 1 -- 172.21.15.107:0/894452070 learned_addr learned my addr 172.21.15.107:0/894452070 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:12.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c5354640 1 --2- 172.21.15.107:0/894452070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06c0102f40 0x7f06c019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06bffff640 1 --2- 172.21.15.107:0/894452070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06c0100370 0x7f06c0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c4b53640 1 -- 172.21.15.107:0/894452070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06c0102f40 msgr2=0x7f06c019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c4b53640 1 --2- 172.21.15.107:0/894452070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06c0102f40 0x7f06c019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c4b53640 1 -- 172.21.15.107:0/894452070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06c0100370 msgr2=0x7f06c0198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c4b53640 1 --2- 172.21.15.107:0/894452070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06c0100370 0x7f06c0198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c4b53640 1 -- 172.21.15.107:0/894452070 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c019f6e0 con 0x7f06c00ff9b0 2024-05-09T08:33:12.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06bffff640 1 --2- 172.21.15.107:0/894452070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06c0100370 0x7f06c0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:12.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c5354640 1 --2- 172.21.15.107:0/894452070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06c0102f40 0x7f06c019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:12.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06c4b53640 1 --2- 172.21.15.107:0/894452070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c00ff9b0 0x7f06c0197ac0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f06a8005e30 tx=0x7f06a8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.313+0000 7f06bdffb640 1 -- 172.21.15.107:0/894452070 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a8032ec0 con 0x7f06c00ff9b0 2024-05-09T08:33:12.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.314+0000 7f06bdffb640 1 -- 172.21.15.107:0/894452070 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06a80044f0 con 0x7f06c00ff9b0 2024-05-09T08:33:12.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.314+0000 7f06bdffb640 1 -- 172.21.15.107:0/894452070 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a8030d90 con 0x7f06c00ff9b0 2024-05-09T08:33:12.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.314+0000 7f06c6db7640 1 -- 172.21.15.107:0/894452070 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06c019f970 con 0x7f06c00ff9b0 2024-05-09T08:33:12.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.314+0000 7f06c6db7640 1 -- 172.21.15.107:0/894452070 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06c019fe70 con 0x7f06c00ff9b0 2024-05-09T08:33:12.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.315+0000 7f06c6db7640 1 -- 172.21.15.107:0/894452070 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06c0108e20 con 0x7f06c00ff9b0 2024-05-09T08:33:12.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.316+0000 7f06bdffb640 1 -- 172.21.15.107:0/894452070 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f06a80305c0 con 0x7f06c00ff9b0 2024-05-09T08:33:12.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.316+0000 7f06bdffb640 1 --2- 172.21.15.107:0/894452070 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f06ac0a2d30 0x7f06ac0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.317+0000 7f06bffff640 1 --2- 172.21.15.107:0/894452070 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f06ac0a2d30 0x7f06ac0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.317+0000 7f06bdffb640 1 -- 172.21.15.107:0/894452070 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f06a80f0be0 con 0x7f06c00ff9b0 2024-05-09T08:33:12.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.317+0000 7f06bffff640 1 --2- 172.21.15.107:0/894452070 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f06ac0a2d30 0x7f06ac0a51f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f06c0198fd0 tx=0x7f06b0005d90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.321+0000 7f06bdffb640 1 -- 172.21.15.107:0/894452070 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f06a80bd7c0 con 0x7f06c00ff9b0 2024-05-09T08:33:12.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.498+0000 7f06c6db7640 1 -- 172.21.15.107:0/894452070 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f06c0062430 con 0x7f06c00ff9b0 2024-05-09T08:33:12.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.500+0000 7f06bdffb640 1 -- 172.21.15.107:0/894452070 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24761 (secure 0 0 0) 0x7f06a80bd0c0 con 0x7f06c00ff9b0 2024-05-09T08:33:12.500 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:12.500 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":93,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:32:57.545099+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:33:12.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.502+0000 7f06c6db7640 1 -- 172.21.15.107:0/894452070 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f06ac0a2d30 msgr2=0x7f06ac0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.502+0000 7f06c6db7640 1 --2- 172.21.15.107:0/894452070 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f06ac0a2d30 0x7f06ac0a51f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f06c0198fd0 tx=0x7f06b0005d90 comp rx=0 tx=0).stop 2024-05-09T08:33:12.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.502+0000 7f06c6db7640 1 -- 172.21.15.107:0/894452070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c00ff9b0 msgr2=0x7f06c0197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.502+0000 7f06c6db7640 1 --2- 172.21.15.107:0/894452070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c00ff9b0 0x7f06c0197ac0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f06a8005e30 tx=0x7f06a8003ba0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.502+0000 7f06c6db7640 1 -- 172.21.15.107:0/894452070 shutdown_connections 2024-05-09T08:33:12.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.502+0000 7f06c6db7640 1 --2- 172.21.15.107:0/894452070 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f06ac0a2d30 0x7f06ac0a51f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.502+0000 7f06c6db7640 1 --2- 172.21.15.107:0/894452070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06c0102f40 0x7f06c019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.502+0000 7f06c6db7640 1 --2- 172.21.15.107:0/894452070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06c0100370 0x7f06c0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.502+0000 7f06c6db7640 1 --2- 172.21.15.107:0/894452070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06c00ff9b0 0x7f06c0197ac0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.502+0000 7f06c6db7640 1 -- 172.21.15.107:0/894452070 >> 172.21.15.107:0/894452070 conn(0x7f06c00fb510 msgr2=0x7f06c0105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.502+0000 7f06c6db7640 1 -- 172.21.15.107:0/894452070 shutdown_connections 2024-05-09T08:33:12.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.502+0000 7f06c6db7640 1 -- 172.21.15.107:0/894452070 wait complete. 2024-05-09T08:33:12.513 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-09T08:31:00.157448+0000', 'modified': '2024-05-09T08:31:21.056050+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24499, 'mds_1': 24491, 'mds_2': 24478}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24431': {'gid': 24431, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.110:6835/2138165106', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6834', 'nonce': 2138165106}, {'type': 'v1', 'addr': '172.21.15.110:6835', 'nonce': 2138165106}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24458': {'gid': 24458, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.110:6837/624105719', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6836', 'nonce': 624105719}, {'type': 'v1', 'addr': '172.21.15.110:6837', 'nonce': 624105719}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.157:6835/843217781', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 843217781}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 843217781}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.157:6837/1973902800', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6836', 'nonce': 1973902800}, {'type': 'v1', 'addr': '172.21.15.157:6837', 'nonce': 1973902800}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24491': {'gid': 24491, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.110:6839/593143450', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6838', 'nonce': 593143450}, {'type': 'v1', 'addr': '172.21.15.110:6839', 'nonce': 593143450}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.157:6839/3003356919', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3003356919}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3003356919}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24499, 'qdb_cluster': [24491, 24478, 24499]} 2024-05-09T08:33:12.513 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-09T08:33:12.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T08:33:12.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.616+0000 7eff7a8d6640 1 -- 172.21.15.107:0/3644235850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff74101960 msgr2=0x7eff74101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.616+0000 7eff7a8d6640 1 --2- 172.21.15.107:0/3644235850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff74101960 0x7eff74101de0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7eff68009900 tx=0x7eff6802a410 comp rx=0 tx=0).stop 2024-05-09T08:33:12.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.616+0000 7eff7a8d6640 1 -- 172.21.15.107:0/3644235850 shutdown_connections 2024-05-09T08:33:12.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.616+0000 7eff7a8d6640 1 --2- 172.21.15.107:0/3644235850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff74102320 0x7eff74108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.616+0000 7eff7a8d6640 1 --2- 172.21.15.107:0/3644235850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff74101960 0x7eff74101de0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.616+0000 7eff7a8d6640 1 --2- 172.21.15.107:0/3644235850 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff74100770 0x7eff74100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.616+0000 7eff7a8d6640 1 -- 172.21.15.107:0/3644235850 >> 172.21.15.107:0/3644235850 conn(0x7eff740fbd20 msgr2=0x7eff740fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.616+0000 7eff7a8d6640 1 -- 172.21.15.107:0/3644235850 shutdown_connections 2024-05-09T08:33:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.616+0000 7eff7a8d6640 1 -- 172.21.15.107:0/3644235850 wait complete. 2024-05-09T08:33:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.616+0000 7eff7a8d6640 1 Processor -- start 2024-05-09T08:33:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.616+0000 7eff7a8d6640 1 -- start start 2024-05-09T08:33:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff7a8d6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff74100770 0x7eff7410b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff7a8d6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff74101960 0x7eff74109730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff7a8d6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff74102320 0x7eff74109c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff7a8d6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7eff7410d3c0 con 0x7eff74102320 2024-05-09T08:33:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff7a8d6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7eff7410d240 con 0x7eff74100770 2024-05-09T08:33:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff7a8d6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7eff7410d540 con 0x7eff74101960 2024-05-09T08:33:12.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff78e73640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff74102320 0x7eff74109c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff78e73640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff74102320 0x7eff74109c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34210/0 (socket says 172.21.15.107:34210) 2024-05-09T08:33:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff78e73640 1 -- 172.21.15.107:0/3617622161 learned_addr learned my addr 172.21.15.107:0/3617622161 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff73fff640 1 --2- 172.21.15.107:0/3617622161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff74100770 0x7eff7410b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff737fe640 1 --2- 172.21.15.107:0/3617622161 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff74101960 0x7eff74109730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff78e73640 1 -- 172.21.15.107:0/3617622161 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff74101960 msgr2=0x7eff74109730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff78e73640 1 --2- 172.21.15.107:0/3617622161 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff74101960 0x7eff74109730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff78e73640 1 -- 172.21.15.107:0/3617622161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff74100770 msgr2=0x7eff7410b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff78e73640 1 --2- 172.21.15.107:0/3617622161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff74100770 0x7eff7410b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff78e73640 1 -- 172.21.15.107:0/3617622161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7410a500 con 0x7eff74102320 2024-05-09T08:33:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff78e73640 1 --2- 172.21.15.107:0/3617622161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff74102320 0x7eff74109c70 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00cce0 tx=0x7eff6c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.617+0000 7eff717fa640 1 -- 172.21.15.107:0/3617622161 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c00f450 con 0x7eff74102320 2024-05-09T08:33:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.618+0000 7eff7a8d6640 1 -- 172.21.15.107:0/3617622161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff741a1b70 con 0x7eff74102320 2024-05-09T08:33:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.618+0000 7eff717fa640 1 -- 172.21.15.107:0/3617622161 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff6c00faa0 con 0x7eff74102320 2024-05-09T08:33:12.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.618+0000 7eff717fa640 1 -- 172.21.15.107:0/3617622161 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c018b70 con 0x7eff74102320 2024-05-09T08:33:12.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.618+0000 7eff7a8d6640 1 -- 172.21.15.107:0/3617622161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff741a2040 con 0x7eff74102320 2024-05-09T08:33:12.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.620+0000 7eff717fa640 1 -- 172.21.15.107:0/3617622161 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7eff6c018d10 con 0x7eff74102320 2024-05-09T08:33:12.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.620+0000 7eff7d5e1740 1 Processor -- start 2024-05-09T08:33:12.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.620+0000 7eff7d5e1740 1 -- start start 2024-05-09T08:33:12.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.620+0000 7eff7d5e1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820ac8c0 0x55cf820acce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.620+0000 7eff7d5e1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ad2b0 0x55cf820b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.620+0000 7eff7d5e1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820b2420 0x55cf820b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.620+0000 7eff7d5e1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cf820104d0 con 0x55cf820ad2b0 2024-05-09T08:33:12.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.620+0000 7eff7d5e1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cf81fb83e0 con 0x55cf820ac8c0 2024-05-09T08:33:12.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.620+0000 7eff7d5e1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55cf8200c490 con 0x55cf820b2420 2024-05-09T08:33:12.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.620+0000 7eff73fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ad2b0 0x55cf820b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.620+0000 7eff73fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ad2b0 0x55cf820b1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34220/0 (socket says 172.21.15.107:34220) 2024-05-09T08:33:12.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.620+0000 7eff73fff640 1 -- 172.21.15.107:0/2131506547 learned_addr learned my addr 172.21.15.107:0/2131506547 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:12.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.620+0000 7eff717fa640 1 --2- 172.21.15.107:0/3617622161 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7eff5c0a2b90 0x7eff5c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.621+0000 7eff737fe640 1 --2- 172.21.15.107:0/2131506547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820b2420 0x55cf820b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.621+0000 7eff78e73640 1 --2- 172.21.15.107:0/3617622161 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7eff5c0a2b90 0x7eff5c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.621+0000 7eff737fe640 1 --2- 172.21.15.107:0/2131506547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820ac8c0 0x55cf820acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.621+0000 7eff73fff640 1 -- 172.21.15.107:0/2131506547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820b2420 msgr2=0x55cf820b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.621+0000 7eff73fff640 1 --2- 172.21.15.107:0/2131506547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820b2420 0x55cf820b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.621+0000 7eff73fff640 1 -- 172.21.15.107:0/2131506547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820ac8c0 msgr2=0x55cf820acce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.621+0000 7eff73fff640 1 --2- 172.21.15.107:0/2131506547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820ac8c0 0x55cf820acce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.621+0000 7eff73fff640 1 -- 172.21.15.107:0/2131506547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf81d242d0 con 0x55cf820ad2b0 2024-05-09T08:33:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.621+0000 7eff737fe640 1 --2- 172.21.15.107:0/2131506547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820ac8c0 0x55cf820acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.621+0000 7eff717fa640 1 -- 172.21.15.107:0/3617622161 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7eff6c0cf810 con 0x7eff74102320 2024-05-09T08:33:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.621+0000 7eff73fff640 1 --2- 172.21.15.107:0/2131506547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ad2b0 0x55cf820b1db0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7eff6400bcb0 tx=0x7eff640107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6b21883d230969 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.621+0000 7eff78e73640 1 --2- 172.21.15.107:0/3617622161 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7eff5c0a2b90 0x7eff5c0a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7eff6c09b9f0 tx=0x7eff6c0911d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.621+0000 7eff5a7fc640 1 -- 172.21.15.107:0/2131506547 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff64005020 con 0x55cf820ad2b0 2024-05-09T08:33:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.621+0000 7eff5a7fc640 1 -- 172.21.15.107:0/2131506547 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff640047a0 con 0x55cf820ad2b0 2024-05-09T08:33:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.621+0000 7eff5a7fc640 1 -- 172.21.15.107:0/2131506547 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff640116a0 con 0x55cf820ad2b0 2024-05-09T08:33:12.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.622+0000 7eff7d5e1740 1 -- 172.21.15.107:0/2131506547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ad2b0 msgr2=0x55cf820b1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.622+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/2131506547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ad2b0 0x55cf820b1db0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7eff6400bcb0 tx=0x7eff640107a0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.622+0000 7eff7d5e1740 1 -- 172.21.15.107:0/2131506547 shutdown_connections 2024-05-09T08:33:12.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.622+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/2131506547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820b2420 0x55cf820b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.622+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/2131506547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ad2b0 0x55cf820b1db0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.622+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/2131506547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820ac8c0 0x55cf820acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.622+0000 7eff7d5e1740 1 -- 172.21.15.107:0/2131506547 >> 172.21.15.107:0/2131506547 conn(0x55cf820a7f30 msgr2=0x55cf820a8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.622+0000 7eff7d5e1740 1 -- 172.21.15.107:0/2131506547 shutdown_connections 2024-05-09T08:33:12.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.622+0000 7eff7d5e1740 1 -- 172.21.15.107:0/2131506547 wait complete. 2024-05-09T08:33:12.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.622+0000 7eff7d5e1740 1 -- start start 2024-05-09T08:33:12.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.622+0000 7eff7d5e1740 1 Processor -- start 2024-05-09T08:33:12.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff7d5e1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ac8c0 0x55cf820b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff7d5e1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820ad2b0 0x55cf820b7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff7d5e1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820b2420 0x55cf820be260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff7d5e1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cf820b53d0 con 0x55cf820ac8c0 2024-05-09T08:33:12.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff7d5e1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cf81fcb8e0 con 0x55cf820ad2b0 2024-05-09T08:33:12.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff7d5e1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55cf820b5550 con 0x55cf820b2420 2024-05-09T08:33:12.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff7d5e1740 10 client.0 _ll_register_callbacks cb 0x55cf81f989c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:12.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff737fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ac8c0 0x55cf820b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff737fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ac8c0 0x55cf820b6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34228/0 (socket says 172.21.15.107:34228) 2024-05-09T08:33:12.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff737fe640 1 -- 172.21.15.107:0/1874061538 learned_addr learned my addr 172.21.15.107:0/1874061538 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:12.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff737fe640 1 --2- 172.21.15.107:0/1874061538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820b2420 0x55cf820be260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff737fe640 1 -- 172.21.15.107:0/1874061538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820b2420 msgr2=0x55cf820be260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff737fe640 1 --2- 172.21.15.107:0/1874061538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820b2420 0x55cf820be260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff737fe640 1 -- 172.21.15.107:0/1874061538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820ad2b0 msgr2=0x55cf820b7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff737fe640 1 --2- 172.21.15.107:0/1874061538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820ad2b0 0x55cf820b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff737fe640 1 -- 172.21.15.107:0/1874061538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf81dee6b0 con 0x55cf820ac8c0 2024-05-09T08:33:12.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff737fe640 1 --2- 172.21.15.107:0/1874061538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ac8c0 0x55cf820b6cf0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7eff6802a930 tx=0x7eff68009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d66700b32f257acd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff537fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:12.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff537fe640 1 -- 172.21.15.107:0/1874061538 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff68035040 con 0x55cf820ac8c0 2024-05-09T08:33:12.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff537fe640 1 -- 172.21.15.107:0/1874061538 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7eff68004140 con 0x55cf820ac8c0 2024-05-09T08:33:12.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff7d5e1740 1 -- 172.21.15.107:0/1874061538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff640047a0 con 0x55cf820ac8c0 2024-05-09T08:33:12.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff537fe640 1 -- 172.21.15.107:0/1874061538 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7eff68050050 con 0x55cf820ac8c0 2024-05-09T08:33:12.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff7d5e1740 10 client.15417 fetch_fsmap learned FSMap version 24 2024-05-09T08:33:12.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.623+0000 7eff7d5e1740 1 -- 172.21.15.107:0/1874061538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55cf820c1250 con 0x55cf820ac8c0 2024-05-09T08:33:12.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.624+0000 7eff537fe640 1 -- 172.21.15.107:0/1874061538 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff640047a0 con 0x55cf820ac8c0 2024-05-09T08:33:12.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.624+0000 7eff537fe640 1 -- 172.21.15.107:0/1874061538 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7eff68049060 con 0x55cf820ac8c0 2024-05-09T08:33:12.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.624+0000 7eff7d5e1740 10 client.15417 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T08:33:12.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.624+0000 7eff7d5e1740 10 client.15417 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T08:33:12.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.624+0000 7eff7d5e1740 20 client.15417 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:12.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.624+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/1874061538 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55cf820c3950 0x55cf820c4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.624+0000 7eff7d5e1740 4 client.15417 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:33:12.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.624+0000 7eff7d5e1740 1 -- 172.21.15.107:0/1874061538 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf8201dc00 con 0x55cf820c3950 2024-05-09T08:33:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.624+0000 7eff73fff640 1 --2- 172.21.15.107:0/1874061538 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55cf820c3950 0x55cf820c4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.625+0000 7eff537fe640 10 client.15417 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:33:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.625+0000 7eff73fff640 1 --2- 172.21.15.107:0/1874061538 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55cf820c3950 0x55cf820c4dc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=92382182fd621a65 server_cookie=65da2a7994ce2457 in_seq=0 out_seq=0 2024-05-09T08:33:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.625+0000 7eff537fe640 1 -- 172.21.15.107:0/1874061538 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf8201dc00 con 0x55cf820c3950 2024-05-09T08:33:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.625+0000 7eff537fe640 10 client.15417 handle_command_reply: tid=0 2024-05-09T08:33:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.625+0000 7eff7d5e1740 1 client.15417 shutdown 2024-05-09T08:33:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.626+0000 7eff7d5e1740 1 -- 172.21.15.107:0/1874061538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ac8c0 msgr2=0x55cf820b6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.626+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/1874061538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ac8c0 0x55cf820b6cf0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7eff6802a930 tx=0x7eff68009260 comp rx=0 tx=0).stop 2024-05-09T08:33:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.626+0000 7eff7d5e1740 1 -- 172.21.15.107:0/1874061538 shutdown_connections 2024-05-09T08:33:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.626+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/1874061538 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55cf820c3950 0x55cf820c4dc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.626+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/1874061538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820b2420 0x55cf820be260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.626+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/1874061538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820ad2b0 0x55cf820b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.626+0000 7eff537fe640 0 client.15417 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:33:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.626+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/1874061538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ac8c0 0x55cf820b6cf0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.626+0000 7eff7d5e1740 1 -- 172.21.15.107:0/1874061538 >> 172.21.15.107:0/1874061538 conn(0x55cf820a7f30 msgr2=0x55cf820a7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.626+0000 7eff7d5e1740 1 -- 172.21.15.107:0/1874061538 shutdown_connections 2024-05-09T08:33:12.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.626+0000 7eff7d5e1740 1 -- 172.21.15.107:0/1874061538 wait complete. 2024-05-09T08:33:12.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.626+0000 7eff7d5e1740 20 client.15417 trim_cache size 0 max 16384 2024-05-09T08:33:12.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.643+0000 7eff7d5e1740 1 Processor -- start 2024-05-09T08:33:12.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.643+0000 7eff7d5e1740 1 -- start start 2024-05-09T08:33:12.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.643+0000 7eff7d5e1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820ac8c0 0x55cf820a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.643+0000 7eff7d5e1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ad2b0 0x55cf820be220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.643+0000 7eff7d5e1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820b2420 0x55cf820aaee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.643+0000 7eff7d5e1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cf820c3100 con 0x55cf820ad2b0 2024-05-09T08:33:12.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.643+0000 7eff7d5e1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cf820c37d0 con 0x55cf820ac8c0 2024-05-09T08:33:12.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.643+0000 7eff7d5e1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55cf820c3ea0 con 0x55cf820b2420 2024-05-09T08:33:12.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.644+0000 7eff73fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ad2b0 0x55cf820be220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.644+0000 7eff73fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ad2b0 0x55cf820be220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34236/0 (socket says 172.21.15.107:34236) 2024-05-09T08:33:12.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.644+0000 7eff73fff640 1 -- 172.21.15.107:0/1932154774 learned_addr learned my addr 172.21.15.107:0/1932154774 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:12.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.644+0000 7eff737fe640 1 --2- 172.21.15.107:0/1932154774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820ac8c0 0x55cf820a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.644+0000 7eff737fe640 1 --2- 172.21.15.107:0/1932154774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820b2420 0x55cf820aaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.644+0000 7eff73fff640 1 -- 172.21.15.107:0/1932154774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820b2420 msgr2=0x55cf820aaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.644+0000 7eff73fff640 1 --2- 172.21.15.107:0/1932154774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820b2420 0x55cf820aaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.644+0000 7eff73fff640 1 -- 172.21.15.107:0/1932154774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820ac8c0 msgr2=0x55cf820a2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.644+0000 7eff73fff640 1 --2- 172.21.15.107:0/1932154774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820ac8c0 0x55cf820a2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.644+0000 7eff73fff640 1 -- 172.21.15.107:0/1932154774 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf820b1940 con 0x55cf820ad2b0 2024-05-09T08:33:12.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.644+0000 7eff73fff640 1 --2- 172.21.15.107:0/1932154774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ad2b0 0x55cf820be220 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7eff6400bcb0 tx=0x7eff64013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6791d5f410a56703 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.644+0000 7eff59ffb640 1 -- 172.21.15.107:0/1932154774 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff64027630 con 0x55cf820ad2b0 2024-05-09T08:33:12.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.644+0000 7eff59ffb640 1 -- 172.21.15.107:0/1932154774 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff6402c070 con 0x55cf820ad2b0 2024-05-09T08:33:12.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.644+0000 7eff59ffb640 1 -- 172.21.15.107:0/1932154774 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff64004940 con 0x55cf820ad2b0 2024-05-09T08:33:12.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 -- 172.21.15.107:0/1932154774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ad2b0 msgr2=0x55cf820be220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/1932154774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ad2b0 0x55cf820be220 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7eff6400bcb0 tx=0x7eff64013040 comp rx=0 tx=0).stop 2024-05-09T08:33:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 -- 172.21.15.107:0/1932154774 shutdown_connections 2024-05-09T08:33:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/1932154774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820b2420 0x55cf820aaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/1932154774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ad2b0 0x55cf820be220 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/1932154774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820ac8c0 0x55cf820a2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 -- 172.21.15.107:0/1932154774 >> 172.21.15.107:0/1932154774 conn(0x55cf820a7f30 msgr2=0x55cf820a8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 -- 172.21.15.107:0/1932154774 shutdown_connections 2024-05-09T08:33:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 -- 172.21.15.107:0/1932154774 wait complete. 2024-05-09T08:33:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 -- start start 2024-05-09T08:33:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 Processor -- start 2024-05-09T08:33:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ac8c0 0x55cf820b9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820ad2b0 0x55cf820ba500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820b2420 0x55cf820bd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cf820c4c40 con 0x55cf820ac8c0 2024-05-09T08:33:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cf820c4570 con 0x55cf820b2420 2024-05-09T08:33:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55cf820c5310 con 0x55cf820ad2b0 2024-05-09T08:33:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff7d5e1740 10 client.0 _ll_register_callbacks cb 0x7eff68009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.645+0000 7eff737fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ac8c0 0x55cf820b9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff737fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ac8c0 0x55cf820b9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34250/0 (socket says 172.21.15.107:34250) 2024-05-09T08:33:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff737fe640 1 -- 172.21.15.107:0/439937320 learned_addr learned my addr 172.21.15.107:0/439937320 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff73fff640 1 --2- 172.21.15.107:0/439937320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820ad2b0 0x55cf820ba500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff737fe640 1 --2- 172.21.15.107:0/439937320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820b2420 0x55cf820bd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff737fe640 1 -- 172.21.15.107:0/439937320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820ad2b0 msgr2=0x55cf820ba500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff737fe640 1 --2- 172.21.15.107:0/439937320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820ad2b0 0x55cf820ba500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff737fe640 1 -- 172.21.15.107:0/439937320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820b2420 msgr2=0x55cf820bd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff737fe640 1 --2- 172.21.15.107:0/439937320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820b2420 0x55cf820bd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff737fe640 1 -- 172.21.15.107:0/439937320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf820bdb10 con 0x55cf820ac8c0 2024-05-09T08:33:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff737fe640 1 --2- 172.21.15.107:0/439937320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ac8c0 0x55cf820b9fc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7eff68009230 tx=0x7eff68032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee19b07278bb5ef7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff537fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff537fe640 1 -- 172.21.15.107:0/439937320 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6805b020 con 0x55cf820ac8c0 2024-05-09T08:33:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff7d5e1740 1 -- 172.21.15.107:0/439937320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff6402c070 con 0x55cf820ac8c0 2024-05-09T08:33:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff537fe640 1 -- 172.21.15.107:0/439937320 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7eff68004240 con 0x55cf820ac8c0 2024-05-09T08:33:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff537fe640 1 -- 172.21.15.107:0/439937320 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7eff68066050 con 0x55cf820ac8c0 2024-05-09T08:33:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff7d5e1740 10 client.15423 fetch_fsmap learned FSMap version 24 2024-05-09T08:33:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff7d5e1740 1 -- 172.21.15.107:0/439937320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55cf82129fc0 con 0x55cf820ac8c0 2024-05-09T08:33:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.646+0000 7eff537fe640 1 -- 172.21.15.107:0/439937320 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff6402c070 con 0x55cf820ac8c0 2024-05-09T08:33:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.647+0000 7eff537fe640 1 -- 172.21.15.107:0/439937320 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7eff68049040 con 0x55cf820ac8c0 2024-05-09T08:33:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.647+0000 7eff7d5e1740 10 client.15423 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T08:33:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.647+0000 7eff7d5e1740 10 client.15423 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T08:33:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.647+0000 7eff7d5e1740 20 client.15423 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.647+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/439937320 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55cf8212eb60 0x55cf8212d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.647+0000 7eff7d5e1740 4 client.15423 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:33:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.647+0000 7eff7d5e1740 1 -- 172.21.15.107:0/439937320 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "status"}) -- 0x55cf82017f50 con 0x55cf8212eb60 2024-05-09T08:33:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.647+0000 7eff73fff640 1 --2- 172.21.15.107:0/439937320 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55cf8212eb60 0x55cf8212d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.647+0000 7eff537fe640 10 client.15423 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:33:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.647+0000 7eff73fff640 1 --2- 172.21.15.107:0/439937320 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55cf8212eb60 0x55cf8212d870 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1e07c3d2a773f473 server_cookie=96131a67280afde7 in_seq=0 out_seq=0 2024-05-09T08:33:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.647+0000 7eff537fe640 1 -- 172.21.15.107:0/439937320 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55cf82017f50 con 0x55cf8212eb60 2024-05-09T08:33:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.647+0000 7eff537fe640 10 client.15423 handle_command_reply: tid=0 2024-05-09T08:33:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.647+0000 7eff7d5e1740 1 client.15423 shutdown 2024-05-09T08:33:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7d5e1740 1 -- 172.21.15.107:0/439937320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ac8c0 msgr2=0x55cf820b9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/439937320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ac8c0 0x55cf820b9fc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7eff68009230 tx=0x7eff68032f20 comp rx=0 tx=0).stop 2024-05-09T08:33:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7d5e1740 1 -- 172.21.15.107:0/439937320 shutdown_connections 2024-05-09T08:33:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/439937320 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55cf8212eb60 0x55cf8212d870 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/439937320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf820b2420 0x55cf820bd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/439937320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf820ad2b0 0x55cf820ba500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff537fe640 0 client.15423 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:33:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7d5e1740 1 --2- 172.21.15.107:0/439937320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf820ac8c0 0x55cf820b9fc0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7d5e1740 1 -- 172.21.15.107:0/439937320 >> 172.21.15.107:0/439937320 conn(0x55cf820a7f30 msgr2=0x55cf820aaee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7d5e1740 1 -- 172.21.15.107:0/439937320 shutdown_connections 2024-05-09T08:33:12.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7d5e1740 1 -- 172.21.15.107:0/439937320 wait complete. 2024-05-09T08:33:12.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7d5e1740 20 client.15423 trim_cache size 0 max 16384 2024-05-09T08:33:12.654 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:33:12.654 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "3954c88c-0ddd-11ef-bc98-c7b262605968", 2024-05-09T08:33:12.654 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 2, 2024-05-09T08:33:12.654 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24478, 2024-05-09T08:33:12.654 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-05-09T08:33:12.654 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-05-09T08:33:12.655 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-05-09T08:33:12.655 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 112.595525516, 2024-05-09T08:33:12.655 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 24, 2024-05-09T08:33:12.655 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 93, 2024-05-09T08:33:12.655 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:33:12.655 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 152.360386675 2024-05-09T08:33:12.655 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:33:12.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7a8d6640 1 -- 172.21.15.107:0/3617622161 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7eff5c0a2b90 msgr2=0x7eff5c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7a8d6640 1 --2- 172.21.15.107:0/3617622161 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7eff5c0a2b90 0x7eff5c0a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7eff6c09b9f0 tx=0x7eff6c0911d0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7a8d6640 1 -- 172.21.15.107:0/3617622161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff74102320 msgr2=0x7eff74109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7a8d6640 1 --2- 172.21.15.107:0/3617622161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff74102320 0x7eff74109c70 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00cce0 tx=0x7eff6c00a620 comp rx=0 tx=0).stop 2024-05-09T08:33:12.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7a8d6640 1 -- 172.21.15.107:0/3617622161 shutdown_connections 2024-05-09T08:33:12.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7a8d6640 1 --2- 172.21.15.107:0/3617622161 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7eff5c0a2b90 0x7eff5c0a5050 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7a8d6640 1 --2- 172.21.15.107:0/3617622161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff74102320 0x7eff74109c70 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7a8d6640 1 --2- 172.21.15.107:0/3617622161 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff74101960 0x7eff74109730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7a8d6640 1 --2- 172.21.15.107:0/3617622161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff74100770 0x7eff7410b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7a8d6640 1 -- 172.21.15.107:0/3617622161 >> 172.21.15.107:0/3617622161 conn(0x7eff740fbd20 msgr2=0x7eff74104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7a8d6640 1 -- 172.21.15.107:0/3617622161 shutdown_connections 2024-05-09T08:33:12.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.648+0000 7eff7a8d6640 1 -- 172.21.15.107:0/3617622161 wait complete. 2024-05-09T08:33:12.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-09T08:33:12.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.761+0000 7f60611a0640 1 -- 172.21.15.107:0/2560596143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f605c0ffa30 msgr2=0x7f605c0ffe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.761+0000 7f60611a0640 1 --2- 172.21.15.107:0/2560596143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f605c0ffa30 0x7f605c0ffe50 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f604c009930 tx=0x7f604c02a420 comp rx=0 tx=0).stop 2024-05-09T08:33:12.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.761+0000 7f60611a0640 1 -- 172.21.15.107:0/2560596143 shutdown_connections 2024-05-09T08:33:12.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.761+0000 7f60611a0640 1 --2- 172.21.15.107:0/2560596143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f605c1033c0 0x7f605c109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.761+0000 7f60611a0640 1 --2- 172.21.15.107:0/2560596143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f605c102a00 0x7f605c102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.761+0000 7f60611a0640 1 --2- 172.21.15.107:0/2560596143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f605c0ffa30 0x7f605c0ffe50 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.761+0000 7f60611a0640 1 -- 172.21.15.107:0/2560596143 >> 172.21.15.107:0/2560596143 conn(0x7f605c0fb360 msgr2=0x7f605c0fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.761+0000 7f60611a0640 1 -- 172.21.15.107:0/2560596143 shutdown_connections 2024-05-09T08:33:12.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.761+0000 7f60611a0640 1 -- 172.21.15.107:0/2560596143 wait complete. 2024-05-09T08:33:12.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.761+0000 7f60611a0640 1 Processor -- start 2024-05-09T08:33:12.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.761+0000 7f60611a0640 1 -- start start 2024-05-09T08:33:12.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f60611a0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f605c0ffa30 0x7f605c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f60611a0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f605c102a00 0x7f605c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f60611a0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f605c1033c0 0x7f605c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f60611a0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f605c10d2b0 con 0x7f605c102a00 2024-05-09T08:33:12.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f60611a0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f605c10d130 con 0x7f605c0ffa30 2024-05-09T08:33:12.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f60611a0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f605c10d430 con 0x7f605c1033c0 2024-05-09T08:33:12.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f605b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f605c102a00 0x7f605c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f605b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f605c102a00 0x7f605c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34266/0 (socket says 172.21.15.107:34266) 2024-05-09T08:33:12.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f605b7fe640 1 -- 172.21.15.107:0/2858764244 learned_addr learned my addr 172.21.15.107:0/2858764244 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:12.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f606099f640 1 --2- 172.21.15.107:0/2858764244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f605c1033c0 0x7f605c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f605bfff640 1 --2- 172.21.15.107:0/2858764244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f605c0ffa30 0x7f605c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f605b7fe640 1 -- 172.21.15.107:0/2858764244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f605c1033c0 msgr2=0x7f605c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f605b7fe640 1 --2- 172.21.15.107:0/2858764244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f605c1033c0 0x7f605c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f605b7fe640 1 -- 172.21.15.107:0/2858764244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f605c0ffa30 msgr2=0x7f605c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f605b7fe640 1 --2- 172.21.15.107:0/2858764244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f605c0ffa30 0x7f605c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f605b7fe640 1 -- 172.21.15.107:0/2858764244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c19f650 con 0x7f605c102a00 2024-05-09T08:33:12.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f605b7fe640 1 --2- 172.21.15.107:0/2858764244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f605c102a00 0x7f605c197f70 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f605000d770 tx=0x7f605000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f60597fa640 1 -- 172.21.15.107:0/2858764244 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6050017070 con 0x7f605c102a00 2024-05-09T08:33:12.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f60597fa640 1 -- 172.21.15.107:0/2858764244 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60500054d0 con 0x7f605c102a00 2024-05-09T08:33:12.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f60611a0640 1 -- 172.21.15.107:0/2858764244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605c19f940 con 0x7f605c102a00 2024-05-09T08:33:12.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f60597fa640 1 -- 172.21.15.107:0/2858764244 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6050004cf0 con 0x7f605c102a00 2024-05-09T08:33:12.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.762+0000 7f60611a0640 1 -- 172.21.15.107:0/2858764244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f605c19fe40 con 0x7f605c102a00 2024-05-09T08:33:12.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f60597fa640 1 -- 172.21.15.107:0/2858764244 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f6050004e90 con 0x7f605c102a00 2024-05-09T08:33:12.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f60651ea740 1 Processor -- start 2024-05-09T08:33:12.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f60651ea740 1 -- start start 2024-05-09T08:33:12.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f60651ea740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772ddf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f60651ea740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772ddf8b0 0x55a772de43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f60651ea740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772de4a20 0x55a772de6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f60651ea740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a772d3e4d0 con 0x55a772ddeec0 2024-05-09T08:33:12.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f60651ea740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a772ce63e0 con 0x55a772de4a20 2024-05-09T08:33:12.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f60651ea740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a772d3a490 con 0x55a772ddf8b0 2024-05-09T08:33:12.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f60597fa640 1 --2- 172.21.15.107:0/2858764244 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f60440a2b90 0x7f60440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f606099f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772ddf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f606099f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772ddf2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34268/0 (socket says 172.21.15.107:34268) 2024-05-09T08:33:12.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f606099f640 1 -- 172.21.15.107:0/2060108445 learned_addr learned my addr 172.21.15.107:0/2060108445 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:12.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f606099f640 1 --2- 172.21.15.107:0/2858764244 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f60440a2b90 0x7f60440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f605bfff640 1 --2- 172.21.15.107:0/2060108445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772ddf8b0 0x55a772de43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f60597fa640 1 -- 172.21.15.107:0/2858764244 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f60500ceee0 con 0x7f605c102a00 2024-05-09T08:33:12.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.765+0000 7f605b7fe640 1 --2- 172.21.15.107:0/2060108445 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772de4a20 0x55a772de6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.766+0000 7f606099f640 1 -- 172.21.15.107:0/2060108445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772ddf8b0 msgr2=0x55a772de43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.766+0000 7f606099f640 1 --2- 172.21.15.107:0/2060108445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772ddf8b0 0x55a772de43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.766+0000 7f606099f640 1 -- 172.21.15.107:0/2060108445 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772de4a20 msgr2=0x55a772de6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.766+0000 7f606099f640 1 --2- 172.21.15.107:0/2060108445 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772de4a20 0x55a772de6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.766+0000 7f606099f640 1 -- 172.21.15.107:0/2060108445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a772a522d0 con 0x55a772ddeec0 2024-05-09T08:33:12.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.766+0000 7f606099f640 1 --2- 172.21.15.107:0/2060108445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772ddf2e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6054013df0 tx=0x7f60540107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da94ffdb6e07980 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.766+0000 7f606099f640 1 --2- 172.21.15.107:0/2858764244 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f60440a2b90 0x7f60440a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6054005c40 tx=0x7f6054015000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.766+0000 7f60427fc640 1 -- 172.21.15.107:0/2060108445 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605400ade0 con 0x55a772ddeec0 2024-05-09T08:33:12.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.766+0000 7f60427fc640 1 -- 172.21.15.107:0/2060108445 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f605400bb70 con 0x55a772ddeec0 2024-05-09T08:33:12.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.766+0000 7f60427fc640 1 -- 172.21.15.107:0/2060108445 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6054004660 con 0x55a772ddeec0 2024-05-09T08:33:12.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.767+0000 7f60651ea740 1 -- 172.21.15.107:0/2060108445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 msgr2=0x55a772ddf2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.767+0000 7f60651ea740 1 --2- 172.21.15.107:0/2060108445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772ddf2e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6054013df0 tx=0x7f60540107a0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.767+0000 7f60651ea740 1 -- 172.21.15.107:0/2060108445 shutdown_connections 2024-05-09T08:33:12.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.767+0000 7f60651ea740 1 --2- 172.21.15.107:0/2060108445 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772de4a20 0x55a772de6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.767+0000 7f60651ea740 1 --2- 172.21.15.107:0/2060108445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772ddf8b0 0x55a772de43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.767+0000 7f60651ea740 1 --2- 172.21.15.107:0/2060108445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772ddf2e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.767+0000 7f60651ea740 1 -- 172.21.15.107:0/2060108445 >> 172.21.15.107:0/2060108445 conn(0x55a772dd6120 msgr2=0x55a772dd6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.767+0000 7f60651ea740 1 -- 172.21.15.107:0/2060108445 shutdown_connections 2024-05-09T08:33:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.767+0000 7f60651ea740 1 -- 172.21.15.107:0/2060108445 wait complete. 2024-05-09T08:33:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.767+0000 7f60651ea740 1 -- start start 2024-05-09T08:33:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.767+0000 7f60651ea740 1 Processor -- start 2024-05-09T08:33:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.767+0000 7f60651ea740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772de9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f60651ea740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772ddf8b0 0x55a772de9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f60651ea740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772de4a20 0x55a772df0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f60651ea740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a772de7970 con 0x55a772ddeec0 2024-05-09T08:33:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f60651ea740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a772cf98e0 con 0x55a772ddf8b0 2024-05-09T08:33:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f60651ea740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a772de7af0 con 0x55a772de4a20 2024-05-09T08:33:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f60651ea740 10 client.0 _ll_register_callbacks cb 0x55a772cc69c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:12.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f605bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772de9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f605bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772de9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34274/0 (socket says 172.21.15.107:34274) 2024-05-09T08:33:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f605bfff640 1 -- 172.21.15.107:0/159167035 learned_addr learned my addr 172.21.15.107:0/159167035 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f606099f640 1 --2- 172.21.15.107:0/159167035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772de4a20 0x55a772df0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f605bfff640 1 -- 172.21.15.107:0/159167035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772de4a20 msgr2=0x55a772df0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f605bfff640 1 --2- 172.21.15.107:0/159167035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772de4a20 0x55a772df0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f605bfff640 1 -- 172.21.15.107:0/159167035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772ddf8b0 msgr2=0x55a772de9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f605b7fe640 1 --2- 172.21.15.107:0/159167035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772ddf8b0 0x55a772de9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f605bfff640 1 --2- 172.21.15.107:0/159167035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772ddf8b0 0x55a772de9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f605bfff640 1 -- 172.21.15.107:0/159167035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a772b1c6b0 con 0x55a772ddeec0 2024-05-09T08:33:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f606099f640 1 --2- 172.21.15.107:0/159167035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772de4a20 0x55a772df0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f605bfff640 1 --2- 172.21.15.107:0/159167035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772de9310 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f604c005ab0 tx=0x7f604c032e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17603f2702238fe7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f603b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f603b7fe640 1 -- 172.21.15.107:0/159167035 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c035040 con 0x55a772ddeec0 2024-05-09T08:33:12.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f603b7fe640 1 -- 172.21.15.107:0/159167035 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f604c004300 con 0x55a772ddeec0 2024-05-09T08:33:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f60651ea740 1 -- 172.21.15.107:0/159167035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f605400bb70 con 0x55a772ddeec0 2024-05-09T08:33:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f603b7fe640 1 -- 172.21.15.107:0/159167035 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f604c005030 con 0x55a772ddeec0 2024-05-09T08:33:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f60651ea740 10 client.15435 fetch_fsmap learned FSMap version 24 2024-05-09T08:33:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.768+0000 7f60651ea740 1 -- 172.21.15.107:0/159167035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a772df38e0 con 0x55a772ddeec0 2024-05-09T08:33:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.769+0000 7f603b7fe640 1 -- 172.21.15.107:0/159167035 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f605400bb70 con 0x55a772ddeec0 2024-05-09T08:33:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.769+0000 7f603b7fe640 1 -- 172.21.15.107:0/159167035 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f604c004cc0 con 0x55a772ddeec0 2024-05-09T08:33:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.769+0000 7f60651ea740 10 client.15435 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T08:33:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.769+0000 7f60651ea740 10 client.15435 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T08:33:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.769+0000 7f60651ea740 20 client.15435 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.769+0000 7f60651ea740 1 --2- 172.21.15.107:0/159167035 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a772df5f40 0x55a772df73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.769+0000 7f60651ea740 4 client.15435 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:33:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.769+0000 7f60651ea740 1 -- 172.21.15.107:0/159167035 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a772d4bc00 con 0x55a772df5f40 2024-05-09T08:33:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.769+0000 7f605b7fe640 1 --2- 172.21.15.107:0/159167035 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a772df5f40 0x55a772df73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.769+0000 7f603b7fe640 10 client.15435 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:33:12.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.769+0000 7f605b7fe640 1 --2- 172.21.15.107:0/159167035 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a772df5f40 0x55a772df73b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6bc4e574e856e99c server_cookie=b97320e98521cb69 in_seq=0 out_seq=0 2024-05-09T08:33:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.770+0000 7f603b7fe640 1 -- 172.21.15.107:0/159167035 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a772d4bc00 con 0x55a772df5f40 2024-05-09T08:33:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.770+0000 7f603b7fe640 10 client.15435 handle_command_reply: tid=0 2024-05-09T08:33:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.770+0000 7f60651ea740 1 client.15435 shutdown 2024-05-09T08:33:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.770+0000 7f60651ea740 1 -- 172.21.15.107:0/159167035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 msgr2=0x55a772de9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.770+0000 7f60651ea740 1 --2- 172.21.15.107:0/159167035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772de9310 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f604c005ab0 tx=0x7f604c032e30 comp rx=0 tx=0).stop 2024-05-09T08:33:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.770+0000 7f60651ea740 1 -- 172.21.15.107:0/159167035 shutdown_connections 2024-05-09T08:33:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.770+0000 7f60651ea740 1 --2- 172.21.15.107:0/159167035 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a772df5f40 0x55a772df73b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.770+0000 7f60651ea740 1 --2- 172.21.15.107:0/159167035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772de4a20 0x55a772df0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.770+0000 7f60651ea740 1 --2- 172.21.15.107:0/159167035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772ddf8b0 0x55a772de9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.770+0000 7f60651ea740 1 --2- 172.21.15.107:0/159167035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772de9310 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.770+0000 7f603b7fe640 0 client.15435 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:33:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.770+0000 7f60651ea740 1 -- 172.21.15.107:0/159167035 >> 172.21.15.107:0/159167035 conn(0x55a772dd6120 msgr2=0x55a772dd5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.771+0000 7f60651ea740 1 -- 172.21.15.107:0/159167035 shutdown_connections 2024-05-09T08:33:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.771+0000 7f60651ea740 1 -- 172.21.15.107:0/159167035 wait complete. 2024-05-09T08:33:12.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.771+0000 7f60651ea740 20 client.15435 trim_cache size 0 max 16384 2024-05-09T08:33:12.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:12 smithi107 ceph-mon[30122]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s; 1 B/s, 0 objects/s recovering 2024-05-09T08:33:12.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:12 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/24296493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:33:12.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:12 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1363141234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:33:12.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:12 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/173676618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:33:12.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:12 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/894452070' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:12.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.788+0000 7f60651ea740 1 Processor -- start 2024-05-09T08:33:12.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f60651ea740 1 -- start start 2024-05-09T08:33:12.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f60651ea740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772ddeec0 0x55a772de6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f60651ea740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddf8b0 0x55a772df0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f60651ea740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772de4a20 0x55a772dd4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f60651ea740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a772df56f0 con 0x55a772ddf8b0 2024-05-09T08:33:12.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f60651ea740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a772df5dc0 con 0x55a772ddeec0 2024-05-09T08:33:12.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f60651ea740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a772df6490 con 0x55a772de4a20 2024-05-09T08:33:12.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f605b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddf8b0 0x55a772df0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f605b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddf8b0 0x55a772df0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34288/0 (socket says 172.21.15.107:34288) 2024-05-09T08:33:12.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f605b7fe640 1 -- 172.21.15.107:0/2594052678 learned_addr learned my addr 172.21.15.107:0/2594052678 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f606099f640 1 --2- 172.21.15.107:0/2594052678 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772de4a20 0x55a772dd4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f605b7fe640 1 -- 172.21.15.107:0/2594052678 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772de4a20 msgr2=0x55a772dd4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f605b7fe640 1 --2- 172.21.15.107:0/2594052678 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772de4a20 0x55a772dd4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f605b7fe640 1 -- 172.21.15.107:0/2594052678 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772ddeec0 msgr2=0x55a772de6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f605bfff640 1 --2- 172.21.15.107:0/2594052678 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772ddeec0 0x55a772de6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f605b7fe640 1 --2- 172.21.15.107:0/2594052678 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772ddeec0 0x55a772de6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f605b7fe640 1 -- 172.21.15.107:0/2594052678 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a772de4020 con 0x55a772ddf8b0 2024-05-09T08:33:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f605b7fe640 1 --2- 172.21.15.107:0/2594052678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddf8b0 0x55a772df0850 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f60500cc6b0 tx=0x7f60500cc530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a0dff7c71f1d2d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f6041ffb640 1 -- 172.21.15.107:0/2594052678 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6050020040 con 0x55a772ddf8b0 2024-05-09T08:33:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f6041ffb640 1 -- 172.21.15.107:0/2594052678 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6050009c30 con 0x55a772ddf8b0 2024-05-09T08:33:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.789+0000 7f6041ffb640 1 -- 172.21.15.107:0/2594052678 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60500996c0 con 0x55a772ddf8b0 2024-05-09T08:33:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.790+0000 7f60651ea740 1 -- 172.21.15.107:0/2594052678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddf8b0 msgr2=0x55a772df0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.790+0000 7f60651ea740 1 --2- 172.21.15.107:0/2594052678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddf8b0 0x55a772df0850 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f60500cc6b0 tx=0x7f60500cc530 comp rx=0 tx=0).stop 2024-05-09T08:33:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.790+0000 7f60651ea740 1 -- 172.21.15.107:0/2594052678 shutdown_connections 2024-05-09T08:33:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.790+0000 7f60651ea740 1 --2- 172.21.15.107:0/2594052678 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772de4a20 0x55a772dd4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.790+0000 7f60651ea740 1 --2- 172.21.15.107:0/2594052678 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddf8b0 0x55a772df0850 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.790+0000 7f60651ea740 1 --2- 172.21.15.107:0/2594052678 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772ddeec0 0x55a772de6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.790+0000 7f60651ea740 1 -- 172.21.15.107:0/2594052678 >> 172.21.15.107:0/2594052678 conn(0x55a772dd6120 msgr2=0x55a772de0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.790+0000 7f60651ea740 1 -- 172.21.15.107:0/2594052678 shutdown_connections 2024-05-09T08:33:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.790+0000 7f60651ea740 1 -- 172.21.15.107:0/2594052678 wait complete. 2024-05-09T08:33:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.790+0000 7f60651ea740 1 -- start start 2024-05-09T08:33:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.790+0000 7f60651ea740 1 Processor -- start 2024-05-09T08:33:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f60651ea740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772dec060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f60651ea740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772ddf8b0 0x55a772dec5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f60651ea740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772de4a20 0x55a772def5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f60651ea740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a772df7230 con 0x55a772ddeec0 2024-05-09T08:33:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f60651ea740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a772df6b60 con 0x55a772de4a20 2024-05-09T08:33:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f60651ea740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a772df7900 con 0x55a772ddf8b0 2024-05-09T08:33:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f60651ea740 10 client.0 _ll_register_callbacks cb 0x7f604c032ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:12.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f605bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772dec060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f605bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772dec060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34290/0 (socket says 172.21.15.107:34290) 2024-05-09T08:33:12.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f605bfff640 1 -- 172.21.15.107:0/610449748 learned_addr learned my addr 172.21.15.107:0/610449748 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:12.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f605b7fe640 1 --2- 172.21.15.107:0/610449748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772ddf8b0 0x55a772dec5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f606099f640 1 --2- 172.21.15.107:0/610449748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772de4a20 0x55a772def5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f605bfff640 1 -- 172.21.15.107:0/610449748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772ddf8b0 msgr2=0x55a772dec5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f605bfff640 1 --2- 172.21.15.107:0/610449748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772ddf8b0 0x55a772dec5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f605bfff640 1 -- 172.21.15.107:0/610449748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772de4a20 msgr2=0x55a772def5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f605bfff640 1 --2- 172.21.15.107:0/610449748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772de4a20 0x55a772def5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f605bfff640 1 -- 172.21.15.107:0/610449748 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a772defbb0 con 0x55a772ddeec0 2024-05-09T08:33:12.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f606099f640 1 --2- 172.21.15.107:0/610449748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772de4a20 0x55a772def5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:12.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f605bfff640 1 --2- 172.21.15.107:0/610449748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772dec060 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f604c003860 tx=0x7f604c009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b044821602751bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f603b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:12.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f603b7fe640 1 -- 172.21.15.107:0/610449748 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c035070 con 0x55a772ddeec0 2024-05-09T08:33:12.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f60651ea740 1 -- 172.21.15.107:0/610449748 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a772d45f50 con 0x55a772ddeec0 2024-05-09T08:33:12.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f603b7fe640 1 -- 172.21.15.107:0/610449748 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f604c0051d0 con 0x55a772ddeec0 2024-05-09T08:33:12.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f603b7fe640 1 -- 172.21.15.107:0/610449748 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f604c0053d0 con 0x55a772ddeec0 2024-05-09T08:33:12.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f60651ea740 10 client.15441 fetch_fsmap learned FSMap version 24 2024-05-09T08:33:12.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.791+0000 7f60651ea740 1 -- 172.21.15.107:0/610449748 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a772e5c090 con 0x55a772ddeec0 2024-05-09T08:33:12.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.792+0000 7f603b7fe640 1 -- 172.21.15.107:0/610449748 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a772d45f50 con 0x55a772ddeec0 2024-05-09T08:33:12.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.792+0000 7f603b7fe640 1 -- 172.21.15.107:0/610449748 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f604c004950 con 0x55a772ddeec0 2024-05-09T08:33:12.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.792+0000 7f60651ea740 10 client.15441 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T08:33:12.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.792+0000 7f60651ea740 10 client.15441 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T08:33:12.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.792+0000 7f60651ea740 20 client.15441 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:12.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.792+0000 7f60651ea740 1 --2- 172.21.15.107:0/610449748 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a772e60ca0 0x55a772e5f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.792+0000 7f60651ea740 4 client.15441 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:33:12.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.792+0000 7f60651ea740 1 -- 172.21.15.107:0/610449748 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "status"}) -- 0x55a772cc3b80 con 0x55a772e60ca0 2024-05-09T08:33:12.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.792+0000 7f605b7fe640 1 --2- 172.21.15.107:0/610449748 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a772e60ca0 0x55a772e5f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.792+0000 7f603b7fe640 10 client.15441 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:33:12.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.792+0000 7f605b7fe640 1 --2- 172.21.15.107:0/610449748 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a772e60ca0 0x55a772e5f9b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=194c8888f9fdee34 server_cookie=b29d7105b25b407e in_seq=0 out_seq=0 2024-05-09T08:33:12.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f603b7fe640 1 -- 172.21.15.107:0/610449748 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a772cc3b80 con 0x55a772e60ca0 2024-05-09T08:33:12.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f603b7fe640 10 client.15441 handle_command_reply: tid=0 2024-05-09T08:33:12.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f60651ea740 1 client.15441 shutdown 2024-05-09T08:33:12.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f60651ea740 1 -- 172.21.15.107:0/610449748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 msgr2=0x55a772dec060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f60651ea740 1 --2- 172.21.15.107:0/610449748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772dec060 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f604c003860 tx=0x7f604c009670 comp rx=0 tx=0).stop 2024-05-09T08:33:12.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f60651ea740 1 -- 172.21.15.107:0/610449748 shutdown_connections 2024-05-09T08:33:12.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f60651ea740 1 --2- 172.21.15.107:0/610449748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a772de4a20 0x55a772def5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f60651ea740 1 --2- 172.21.15.107:0/610449748 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a772e60ca0 0x55a772e5f9b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f60651ea740 1 --2- 172.21.15.107:0/610449748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a772ddf8b0 0x55a772dec5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f603b7fe640 0 client.15441 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:33:12.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f60651ea740 1 --2- 172.21.15.107:0/610449748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a772ddeec0 0x55a772dec060 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f60651ea740 1 -- 172.21.15.107:0/610449748 >> 172.21.15.107:0/610449748 conn(0x55a772dd6120 msgr2=0x55a772dd8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f60651ea740 1 -- 172.21.15.107:0/610449748 shutdown_connections 2024-05-09T08:33:12.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f60651ea740 1 -- 172.21.15.107:0/610449748 wait complete. 2024-05-09T08:33:12.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f60651ea740 20 client.15441 trim_cache size 0 max 16384 2024-05-09T08:33:12.800 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:33:12.801 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "3954c88c-0ddd-11ef-bc98-c7b262605968", 2024-05-09T08:33:12.801 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 1, 2024-05-09T08:33:12.801 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24491, 2024-05-09T08:33:12.801 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-05-09T08:33:12.801 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-05-09T08:33:12.801 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-05-09T08:33:12.801 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 113.750868436, 2024-05-09T08:33:12.801 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 24, 2024-05-09T08:33:12.801 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 93, 2024-05-09T08:33:12.801 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:33:12.801 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 138.20704491399999 2024-05-09T08:33:12.801 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:33:12.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f60611a0640 1 -- 172.21.15.107:0/2858764244 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f60440a2b90 msgr2=0x7f60440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f60611a0640 1 --2- 172.21.15.107:0/2858764244 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f60440a2b90 0x7f60440a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6054005c40 tx=0x7f6054015000 comp rx=0 tx=0).stop 2024-05-09T08:33:12.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f60611a0640 1 -- 172.21.15.107:0/2858764244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f605c102a00 msgr2=0x7f605c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.793+0000 7f60611a0640 1 --2- 172.21.15.107:0/2858764244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f605c102a00 0x7f605c197f70 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f605000d770 tx=0x7f605000dc30 comp rx=0 tx=0).stop 2024-05-09T08:33:12.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.794+0000 7f60611a0640 1 -- 172.21.15.107:0/2858764244 shutdown_connections 2024-05-09T08:33:12.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.794+0000 7f60611a0640 1 --2- 172.21.15.107:0/2858764244 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f60440a2b90 0x7f60440a5050 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.794+0000 7f60611a0640 1 --2- 172.21.15.107:0/2858764244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f605c1033c0 0x7f605c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.794+0000 7f60611a0640 1 --2- 172.21.15.107:0/2858764244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f605c102a00 0x7f605c197f70 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.794+0000 7f60611a0640 1 --2- 172.21.15.107:0/2858764244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f605c0ffa30 0x7f605c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.794+0000 7f60611a0640 1 -- 172.21.15.107:0/2858764244 >> 172.21.15.107:0/2858764244 conn(0x7f605c0fb360 msgr2=0x7f605c105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.794+0000 7f60611a0640 1 -- 172.21.15.107:0/2858764244 shutdown_connections 2024-05-09T08:33:12.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.794+0000 7f60611a0640 1 -- 172.21.15.107:0/2858764244 wait complete. 2024-05-09T08:33:12.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-05-09T08:33:12.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:12 smithi157 ceph-mon[33614]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s; 1 B/s, 0 objects/s recovering 2024-05-09T08:33:12.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:12 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/24296493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:33:12.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:12 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1363141234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:33:12.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:12 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/173676618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:33:12.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:12 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/894452070' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:12.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:12 smithi110 ceph-mon[33622]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s; 1 B/s, 0 objects/s recovering 2024-05-09T08:33:12.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:12 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/24296493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:33:12.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:12 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1363141234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:33:12.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:12 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/173676618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:33:12.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:12 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/894452070' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:12.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.907+0000 7fa40d3ce640 1 -- 172.21.15.107:0/1390446973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa408102bd0 msgr2=0x7fa408103050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.907+0000 7fa40d3ce640 1 --2- 172.21.15.107:0/1390446973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa408102bd0 0x7fa408103050 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009930 tx=0x7fa3fc02a420 comp rx=0 tx=0).stop 2024-05-09T08:33:12.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.907+0000 7fa40d3ce640 1 -- 172.21.15.107:0/1390446973 shutdown_connections 2024-05-09T08:33:12.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.907+0000 7fa40d3ce640 1 --2- 172.21.15.107:0/1390446973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa408103590 0x7fa408109e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.907+0000 7fa40d3ce640 1 --2- 172.21.15.107:0/1390446973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa408102bd0 0x7fa408103050 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.907+0000 7fa40d3ce640 1 --2- 172.21.15.107:0/1390446973 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4080ffb60 0x7fa4080fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.907+0000 7fa40d3ce640 1 -- 172.21.15.107:0/1390446973 >> 172.21.15.107:0/1390446973 conn(0x7fa4080fb470 msgr2=0x7fa4080fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.907+0000 7fa40d3ce640 1 -- 172.21.15.107:0/1390446973 shutdown_connections 2024-05-09T08:33:12.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.907+0000 7fa40d3ce640 1 -- 172.21.15.107:0/1390446973 wait complete. 2024-05-09T08:33:12.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.907+0000 7fa40d3ce640 1 Processor -- start 2024-05-09T08:33:12.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa40d3ce640 1 -- start start 2024-05-09T08:33:12.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa40d3ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4080ffb60 0x7fa408197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa40d3ce640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa408102bd0 0x7fa408198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa40d3ce640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa408103590 0x7fa40819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa40d3ce640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa40810d3c0 con 0x7fa4080ffb60 2024-05-09T08:33:12.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa40d3ce640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa40810d240 con 0x7fa408103590 2024-05-09T08:33:12.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa40d3ce640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa40810d540 con 0x7fa408102bd0 2024-05-09T08:33:12.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa406ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4080ffb60 0x7fa408197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa406ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4080ffb60 0x7fa408197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34298/0 (socket says 172.21.15.107:34298) 2024-05-09T08:33:12.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa406ffd640 1 -- 172.21.15.107:0/1539993000 learned_addr learned my addr 172.21.15.107:0/1539993000 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:12.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa4067fc640 1 --2- 172.21.15.107:0/1539993000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa408102bd0 0x7fa408198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa4077fe640 1 --2- 172.21.15.107:0/1539993000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa408103590 0x7fa40819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa406ffd640 1 -- 172.21.15.107:0/1539993000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa408102bd0 msgr2=0x7fa408198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa406ffd640 1 --2- 172.21.15.107:0/1539993000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa408102bd0 0x7fa408198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa406ffd640 1 -- 172.21.15.107:0/1539993000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa408103590 msgr2=0x7fa40819f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa406ffd640 1 --2- 172.21.15.107:0/1539993000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa408103590 0x7fa40819f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa406ffd640 1 -- 172.21.15.107:0/1539993000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa40819f770 con 0x7fa4080ffb60 2024-05-09T08:33:12.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa4067fc640 1 --2- 172.21.15.107:0/1539993000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa408102bd0 0x7fa408198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:12.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa406ffd640 1 --2- 172.21.15.107:0/1539993000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4080ffb60 0x7fa408197b50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800d950 tx=0x7fa3f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.908+0000 7fa3effff640 1 -- 172.21.15.107:0/1539993000 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8017070 con 0x7fa4080ffb60 2024-05-09T08:33:12.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.909+0000 7fa40d3ce640 1 -- 172.21.15.107:0/1539993000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa40819fa60 con 0x7fa4080ffb60 2024-05-09T08:33:12.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.909+0000 7fa3effff640 1 -- 172.21.15.107:0/1539993000 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3f8009c30 con 0x7fa4080ffb60 2024-05-09T08:33:12.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.909+0000 7fa3effff640 1 -- 172.21.15.107:0/1539993000 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f800f620 con 0x7fa4080ffb60 2024-05-09T08:33:12.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.909+0000 7fa40d3ce640 1 -- 172.21.15.107:0/1539993000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa40819ff80 con 0x7fa4080ffb60 2024-05-09T08:33:12.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.911+0000 7fa3effff640 1 -- 172.21.15.107:0/1539993000 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fa3f801a070 con 0x7fa4080ffb60 2024-05-09T08:33:12.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.911+0000 7fa4101b9740 1 Processor -- start 2024-05-09T08:33:12.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.911+0000 7fa4101b9740 1 -- start start 2024-05-09T08:33:12.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.911+0000 7fa4101b9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971ca48c0 0x564971ca4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.911+0000 7fa4101b9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971ca52b0 0x564971ca9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.911+0000 7fa4101b9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971caa420 0x564971cac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.911+0000 7fa4101b9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564971c084d0 con 0x564971caa420 2024-05-09T08:33:12.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.911+0000 7fa4101b9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564971bb03e0 con 0x564971ca52b0 2024-05-09T08:33:12.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.911+0000 7fa4101b9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564971c04490 con 0x564971ca48c0 2024-05-09T08:33:12.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.911+0000 7fa3effff640 1 --2- 172.21.15.107:0/1539993000 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa3f00a2b90 0x7fa3f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.911+0000 7fa4067fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971caa420 0x564971cac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.911+0000 7fa4067fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971caa420 0x564971cac8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34312/0 (socket says 172.21.15.107:34312) 2024-05-09T08:33:12.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.911+0000 7fa4067fc640 1 -- 172.21.15.107:0/1556065893 learned_addr learned my addr 172.21.15.107:0/1556065893 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:12.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.911+0000 7fa4077fe640 1 --2- 172.21.15.107:0/1539993000 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa3f00a2b90 0x7fa3f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.912+0000 7fa3effff640 1 -- 172.21.15.107:0/1539993000 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fa3f80ceed0 con 0x7fa4080ffb60 2024-05-09T08:33:12.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.912+0000 7fa406ffd640 1 --2- 172.21.15.107:0/1556065893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971ca52b0 0x564971ca9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.912+0000 7fa4077fe640 1 --2- 172.21.15.107:0/1556065893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971ca48c0 0x564971ca4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.912+0000 7fa4067fc640 1 -- 172.21.15.107:0/1556065893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971ca48c0 msgr2=0x564971ca4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.912+0000 7fa4067fc640 1 --2- 172.21.15.107:0/1556065893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971ca48c0 0x564971ca4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.912+0000 7fa4067fc640 1 -- 172.21.15.107:0/1556065893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971ca52b0 msgr2=0x564971ca9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.912+0000 7fa4067fc640 1 --2- 172.21.15.107:0/1556065893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971ca52b0 0x564971ca9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.912+0000 7fa4067fc640 1 -- 172.21.15.107:0/1556065893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56497191c2d0 con 0x564971caa420 2024-05-09T08:33:12.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.912+0000 7fa4077fe640 1 --2- 172.21.15.107:0/1556065893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971ca48c0 0x564971ca4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:12.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.912+0000 7fa4067fc640 1 --2- 172.21.15.107:0/1556065893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971caa420 0x564971cac8b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0314e0 tx=0x7fa3fc032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61119d9192baac5f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.912+0000 7fa4077fe640 1 --2- 172.21.15.107:0/1539993000 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa3f00a2b90 0x7fa3f00a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x564971ca5b10 tx=0x7fa400009590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.912+0000 7fa3ed7fa640 1 -- 172.21.15.107:0/1556065893 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc03f070 con 0x564971caa420 2024-05-09T08:33:12.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.912+0000 7fa3ed7fa640 1 -- 172.21.15.107:0/1556065893 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3fc031780 con 0x564971caa420 2024-05-09T08:33:12.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.912+0000 7fa3ed7fa640 1 -- 172.21.15.107:0/1556065893 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc035070 con 0x564971caa420 2024-05-09T08:33:12.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 -- 172.21.15.107:0/1556065893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971caa420 msgr2=0x564971cac8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 --2- 172.21.15.107:0/1556065893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971caa420 0x564971cac8b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0314e0 tx=0x7fa3fc032c60 comp rx=0 tx=0).stop 2024-05-09T08:33:12.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 -- 172.21.15.107:0/1556065893 shutdown_connections 2024-05-09T08:33:12.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 --2- 172.21.15.107:0/1556065893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971caa420 0x564971cac8b0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 --2- 172.21.15.107:0/1556065893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971ca52b0 0x564971ca9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 --2- 172.21.15.107:0/1556065893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971ca48c0 0x564971ca4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 -- 172.21.15.107:0/1556065893 >> 172.21.15.107:0/1556065893 conn(0x564971c9ff30 msgr2=0x564971ca0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 -- 172.21.15.107:0/1556065893 shutdown_connections 2024-05-09T08:33:12.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 -- 172.21.15.107:0/1556065893 wait complete. 2024-05-09T08:33:12.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 -- start start 2024-05-09T08:33:12.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 Processor -- start 2024-05-09T08:33:12.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971ca48c0 0x564971caebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971ca52b0 0x564971caf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971caa420 0x564971cb6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564971cad370 con 0x564971ca52b0 2024-05-09T08:33:12.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564971bc38e0 con 0x564971ca48c0 2024-05-09T08:33:12.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564971cad4f0 con 0x564971caa420 2024-05-09T08:33:12.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4101b9740 10 client.0 _ll_register_callbacks cb 0x564971b909c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:12.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4067fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971ca52b0 0x564971caf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4067fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971ca52b0 0x564971caf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34328/0 (socket says 172.21.15.107:34328) 2024-05-09T08:33:12.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.913+0000 7fa4067fc640 1 -- 172.21.15.107:0/4135530501 learned_addr learned my addr 172.21.15.107:0/4135530501 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:12.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa4077fe640 1 --2- 172.21.15.107:0/4135530501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971caa420 0x564971cb6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa406ffd640 1 --2- 172.21.15.107:0/4135530501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971ca48c0 0x564971caebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa4067fc640 1 -- 172.21.15.107:0/4135530501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971caa420 msgr2=0x564971cb6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa4067fc640 1 --2- 172.21.15.107:0/4135530501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971caa420 0x564971cb6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa4067fc640 1 -- 172.21.15.107:0/4135530501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971ca48c0 msgr2=0x564971caebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa4067fc640 1 --2- 172.21.15.107:0/4135530501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971ca48c0 0x564971caebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa4067fc640 1 -- 172.21.15.107:0/4135530501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649719e66b0 con 0x564971ca52b0 2024-05-09T08:33:12.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa406ffd640 1 --2- 172.21.15.107:0/4135530501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971ca48c0 0x564971caebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:12.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa4067fc640 1 --2- 172.21.15.107:0/4135530501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971ca52b0 0x564971caf130 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc000c00 tx=0x7fa3fc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cac30d346c36b3c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa3eaff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:12.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa3eaff5640 1 -- 172.21.15.107:0/4135530501 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc04a070 con 0x564971ca52b0 2024-05-09T08:33:12.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa3eaff5640 1 -- 172.21.15.107:0/4135530501 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fa3fc0037f0 con 0x564971ca52b0 2024-05-09T08:33:12.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa4101b9740 1 -- 172.21.15.107:0/4135530501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3fc031780 con 0x564971ca52b0 2024-05-09T08:33:12.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa3eaff5640 1 -- 172.21.15.107:0/4135530501 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa3fc056050 con 0x564971ca52b0 2024-05-09T08:33:12.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa4101b9740 10 client.15453 fetch_fsmap learned FSMap version 24 2024-05-09T08:33:12.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa4101b9740 1 -- 172.21.15.107:0/4135530501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564971cb90c0 con 0x564971ca52b0 2024-05-09T08:33:12.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.914+0000 7fa3eaff5640 1 -- 172.21.15.107:0/4135530501 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3fc031780 con 0x564971ca52b0 2024-05-09T08:33:12.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.915+0000 7fa3eaff5640 1 -- 172.21.15.107:0/4135530501 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa3fc03b430 con 0x564971ca52b0 2024-05-09T08:33:12.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.915+0000 7fa4101b9740 10 client.15453 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T08:33:12.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.915+0000 7fa4101b9740 10 client.15453 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T08:33:12.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.915+0000 7fa4101b9740 20 client.15453 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:12.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.915+0000 7fa4101b9740 1 --2- 172.21.15.107:0/4135530501 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564971cbb350 0x564971cbc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.915+0000 7fa4101b9740 4 client.15453 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:33:12.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.915+0000 7fa4101b9740 1 -- 172.21.15.107:0/4135530501 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564971c15c00 con 0x564971cbb350 2024-05-09T08:33:12.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.915+0000 7fa406ffd640 1 --2- 172.21.15.107:0/4135530501 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564971cbb350 0x564971cbc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.915+0000 7fa406ffd640 1 --2- 172.21.15.107:0/4135530501 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564971cbb350 0x564971cbc7c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54a0f1ded24ffc0 server_cookie=4d88bb36e1549907 in_seq=0 out_seq=0 2024-05-09T08:33:12.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.915+0000 7fa3eaff5640 10 client.15453 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:12.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.916+0000 7fa3eaff5640 1 -- 172.21.15.107:0/4135530501 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564971c15c00 con 0x564971cbb350 2024-05-09T08:33:12.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.916+0000 7fa3eaff5640 10 client.15453 handle_command_reply: tid=0 2024-05-09T08:33:12.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.916+0000 7fa4101b9740 1 client.15453 shutdown 2024-05-09T08:33:12.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.916+0000 7fa4101b9740 1 -- 172.21.15.107:0/4135530501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971ca52b0 msgr2=0x564971caf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.916+0000 7fa4101b9740 1 --2- 172.21.15.107:0/4135530501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971ca52b0 0x564971caf130 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc000c00 tx=0x7fa3fc004270 comp rx=0 tx=0).stop 2024-05-09T08:33:12.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.916+0000 7fa4101b9740 1 -- 172.21.15.107:0/4135530501 shutdown_connections 2024-05-09T08:33:12.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.916+0000 7fa4101b9740 1 --2- 172.21.15.107:0/4135530501 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564971cbb350 0x564971cbc7c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.916+0000 7fa4101b9740 1 --2- 172.21.15.107:0/4135530501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971caa420 0x564971cb6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.916+0000 7fa4101b9740 1 --2- 172.21.15.107:0/4135530501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971ca52b0 0x564971caf130 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.916+0000 7fa4101b9740 1 --2- 172.21.15.107:0/4135530501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971ca48c0 0x564971caebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.916+0000 7fa3eaff5640 0 client.15453 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:12.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.916+0000 7fa4101b9740 1 -- 172.21.15.107:0/4135530501 >> 172.21.15.107:0/4135530501 conn(0x564971c9ff30 msgr2=0x564971c9ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.916+0000 7fa4101b9740 1 -- 172.21.15.107:0/4135530501 shutdown_connections 2024-05-09T08:33:12.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.917+0000 7fa4101b9740 1 -- 172.21.15.107:0/4135530501 wait complete. 2024-05-09T08:33:12.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.917+0000 7fa4101b9740 20 client.15453 trim_cache size 0 max 16384 2024-05-09T08:33:12.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.934+0000 7fa4101b9740 1 Processor -- start 2024-05-09T08:33:12.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.934+0000 7fa4101b9740 1 -- start start 2024-05-09T08:33:12.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.934+0000 7fa4101b9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971ca48c0 0x564971cac810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.934+0000 7fa4101b9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971ca52b0 0x564971cb6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.934+0000 7fa4101b9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971caa420 0x564971cae800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.934+0000 7fa4101b9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564971cbab00 con 0x564971caa420 2024-05-09T08:33:12.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.934+0000 7fa4101b9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564971cbb1d0 con 0x564971ca48c0 2024-05-09T08:33:12.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.934+0000 7fa4101b9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564971cbb8a0 con 0x564971ca52b0 2024-05-09T08:33:12.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.935+0000 7fa4077fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971caa420 0x564971cae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.935+0000 7fa4077fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971caa420 0x564971cae800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34342/0 (socket says 172.21.15.107:34342) 2024-05-09T08:33:12.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.935+0000 7fa4077fe640 1 -- 172.21.15.107:0/2934010124 learned_addr learned my addr 172.21.15.107:0/2934010124 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:12.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.935+0000 7fa406ffd640 1 --2- 172.21.15.107:0/2934010124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971ca48c0 0x564971cac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.935+0000 7fa4077fe640 1 -- 172.21.15.107:0/2934010124 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971ca52b0 msgr2=0x564971cb6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.935+0000 7fa4077fe640 1 --2- 172.21.15.107:0/2934010124 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971ca52b0 0x564971cb6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.935+0000 7fa4077fe640 1 -- 172.21.15.107:0/2934010124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971ca48c0 msgr2=0x564971cac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.935+0000 7fa4077fe640 1 --2- 172.21.15.107:0/2934010124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971ca48c0 0x564971cac810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.935+0000 7fa4077fe640 1 -- 172.21.15.107:0/2934010124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564971ca5690 con 0x564971caa420 2024-05-09T08:33:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.935+0000 7fa4077fe640 1 --2- 172.21.15.107:0/2934010124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971caa420 0x564971cae800 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa400003cb0 tx=0x7fa400015070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1873de765d1ccf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.935+0000 7fa3ecff9640 1 -- 172.21.15.107:0/2934010124 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40000f5c0 con 0x564971caa420 2024-05-09T08:33:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.935+0000 7fa3ecff9640 1 -- 172.21.15.107:0/2934010124 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa40000fc10 con 0x564971caa420 2024-05-09T08:33:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.935+0000 7fa3ecff9640 1 -- 172.21.15.107:0/2934010124 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa400020cb0 con 0x564971caa420 2024-05-09T08:33:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 -- 172.21.15.107:0/2934010124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971caa420 msgr2=0x564971cae800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 --2- 172.21.15.107:0/2934010124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971caa420 0x564971cae800 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa400003cb0 tx=0x7fa400015070 comp rx=0 tx=0).stop 2024-05-09T08:33:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 -- 172.21.15.107:0/2934010124 shutdown_connections 2024-05-09T08:33:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 --2- 172.21.15.107:0/2934010124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971caa420 0x564971cae800 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 --2- 172.21.15.107:0/2934010124 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971ca52b0 0x564971cb6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 --2- 172.21.15.107:0/2934010124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971ca48c0 0x564971cac810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 -- 172.21.15.107:0/2934010124 >> 172.21.15.107:0/2934010124 conn(0x564971c9ff30 msgr2=0x564971ca0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 -- 172.21.15.107:0/2934010124 shutdown_connections 2024-05-09T08:33:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 -- 172.21.15.107:0/2934010124 wait complete. 2024-05-09T08:33:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 -- start start 2024-05-09T08:33:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 Processor -- start 2024-05-09T08:33:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971ca48c0 0x564971cb2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971ca52b0 0x564971cb2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971caa420 0x564971cb5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564971cbc640 con 0x564971ca48c0 2024-05-09T08:33:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564971cbbf70 con 0x564971caa420 2024-05-09T08:33:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564971cbcd10 con 0x564971ca52b0 2024-05-09T08:33:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa4101b9740 10 client.0 _ll_register_callbacks cb 0x7fa3fc032b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa406ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971ca48c0 0x564971cb2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa406ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971ca48c0 0x564971cb2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34352/0 (socket says 172.21.15.107:34352) 2024-05-09T08:33:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.936+0000 7fa406ffd640 1 -- 172.21.15.107:0/2674749 learned_addr learned my addr 172.21.15.107:0/2674749 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa4067fc640 1 --2- 172.21.15.107:0/2674749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971ca52b0 0x564971cb2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa4077fe640 1 --2- 172.21.15.107:0/2674749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971caa420 0x564971cb5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa406ffd640 1 -- 172.21.15.107:0/2674749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971ca52b0 msgr2=0x564971cb2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa406ffd640 1 --2- 172.21.15.107:0/2674749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971ca52b0 0x564971cb2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa406ffd640 1 -- 172.21.15.107:0/2674749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971caa420 msgr2=0x564971cb5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa406ffd640 1 --2- 172.21.15.107:0/2674749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971caa420 0x564971cb5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa406ffd640 1 -- 172.21.15.107:0/2674749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564971cb5c50 con 0x564971ca48c0 2024-05-09T08:33:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa406ffd640 1 --2- 172.21.15.107:0/2674749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971ca48c0 0x564971cb2360 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa3f809ad80 tx=0x7fa3f8097740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b3dd2be42dd9df server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa3eaff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa3eaff5640 1 -- 172.21.15.107:0/2674749 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8021040 con 0x564971ca48c0 2024-05-09T08:33:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa3eaff5640 1 -- 172.21.15.107:0/2674749 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fa3f8097a20 con 0x564971ca48c0 2024-05-09T08:33:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa4101b9740 1 -- 172.21.15.107:0/2674749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa40000fc10 con 0x564971ca48c0 2024-05-09T08:33:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa3eaff5640 1 -- 172.21.15.107:0/2674749 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa3f809b3b0 con 0x564971ca48c0 2024-05-09T08:33:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa4101b9740 10 client.15459 fetch_fsmap learned FSMap version 24 2024-05-09T08:33:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa4101b9740 1 -- 172.21.15.107:0/2674749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564971d21c70 con 0x564971ca48c0 2024-05-09T08:33:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa3eaff5640 1 -- 172.21.15.107:0/2674749 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa40000fc10 con 0x564971ca48c0 2024-05-09T08:33:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa3eaff5640 1 -- 172.21.15.107:0/2674749 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa3f80ccba0 con 0x564971ca48c0 2024-05-09T08:33:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa4101b9740 10 client.15459 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T08:33:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa4101b9740 10 client.15459 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T08:33:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa4101b9740 20 client.15459 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa4101b9740 1 --2- 172.21.15.107:0/2674749 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564971d26810 0x564971d22c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa4101b9740 4 client.15459 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T08:33:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.937+0000 7fa4101b9740 1 -- 172.21.15.107:0/2674749 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "status"}) -- 0x564971c0ff50 con 0x564971d26810 2024-05-09T08:33:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.938+0000 7fa4067fc640 1 --2- 172.21.15.107:0/2674749 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564971d26810 0x564971d22c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.938+0000 7fa3eaff5640 10 client.15459 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.938+0000 7fa4067fc640 1 --2- 172.21.15.107:0/2674749 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564971d26810 0x564971d22c50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0f6aadee0c0442c server_cookie=4705cde8800cce28 in_seq=0 out_seq=0 2024-05-09T08:33:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.938+0000 7fa3eaff5640 1 -- 172.21.15.107:0/2674749 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x564971c0ff50 con 0x564971d26810 2024-05-09T08:33:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.938+0000 7fa3eaff5640 10 client.15459 handle_command_reply: tid=0 2024-05-09T08:33:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.938+0000 7fa4101b9740 1 client.15459 shutdown 2024-05-09T08:33:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.938+0000 7fa4101b9740 1 -- 172.21.15.107:0/2674749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971ca48c0 msgr2=0x564971cb2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.938+0000 7fa4101b9740 1 --2- 172.21.15.107:0/2674749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971ca48c0 0x564971cb2360 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa3f809ad80 tx=0x7fa3f8097740 comp rx=0 tx=0).stop 2024-05-09T08:33:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.938+0000 7fa4101b9740 1 -- 172.21.15.107:0/2674749 shutdown_connections 2024-05-09T08:33:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.938+0000 7fa4101b9740 1 --2- 172.21.15.107:0/2674749 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564971d26810 0x564971d22c50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.938+0000 7fa4101b9740 1 --2- 172.21.15.107:0/2674749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564971caa420 0x564971cb5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.938+0000 7fa4101b9740 1 --2- 172.21.15.107:0/2674749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564971ca52b0 0x564971cb2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.938+0000 7fa4101b9740 1 --2- 172.21.15.107:0/2674749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564971ca48c0 0x564971cb2360 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.938+0000 7fa3eaff5640 0 client.15459 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.938+0000 7fa4101b9740 1 -- 172.21.15.107:0/2674749 >> 172.21.15.107:0/2674749 conn(0x564971c9ff30 msgr2=0x564971cae800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.938+0000 7fa4101b9740 1 -- 172.21.15.107:0/2674749 shutdown_connections 2024-05-09T08:33:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.939+0000 7fa4101b9740 1 -- 172.21.15.107:0/2674749 wait complete. 2024-05-09T08:33:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.939+0000 7fa4101b9740 20 client.15459 trim_cache size 0 max 16384 2024-05-09T08:33:12.946 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:33:12.946 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "3954c88c-0ddd-11ef-bc98-c7b262605968", 2024-05-09T08:33:12.946 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2024-05-09T08:33:12.946 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24499, 2024-05-09T08:33:12.946 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-05-09T08:33:12.946 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-05-09T08:33:12.947 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-05-09T08:33:12.947 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 132.12985384999999, 2024-05-09T08:33:12.947 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 24, 2024-05-09T08:33:12.947 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 93, 2024-05-09T08:33:12.947 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T08:33:12.947 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 147.02203007700001 2024-05-09T08:33:12.947 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:33:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.939+0000 7fa40d3ce640 1 -- 172.21.15.107:0/1539993000 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa3f00a2b90 msgr2=0x7fa3f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.939+0000 7fa40d3ce640 1 --2- 172.21.15.107:0/1539993000 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa3f00a2b90 0x7fa3f00a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x564971ca5b10 tx=0x7fa400009590 comp rx=0 tx=0).stop 2024-05-09T08:33:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.939+0000 7fa40d3ce640 1 -- 172.21.15.107:0/1539993000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4080ffb60 msgr2=0x7fa408197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.939+0000 7fa40d3ce640 1 --2- 172.21.15.107:0/1539993000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4080ffb60 0x7fa408197b50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800d950 tx=0x7fa3f800de10 comp rx=0 tx=0).stop 2024-05-09T08:33:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.939+0000 7fa40d3ce640 1 -- 172.21.15.107:0/1539993000 shutdown_connections 2024-05-09T08:33:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.939+0000 7fa40d3ce640 1 --2- 172.21.15.107:0/1539993000 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fa3f00a2b90 0x7fa3f00a5050 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.939+0000 7fa40d3ce640 1 --2- 172.21.15.107:0/1539993000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa408103590 0x7fa40819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.939+0000 7fa40d3ce640 1 --2- 172.21.15.107:0/1539993000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa408102bd0 0x7fa408198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.939+0000 7fa40d3ce640 1 --2- 172.21.15.107:0/1539993000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4080ffb60 0x7fa408197b50 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.939+0000 7fa40d3ce640 1 -- 172.21.15.107:0/1539993000 >> 172.21.15.107:0/1539993000 conn(0x7fa4080fb470 msgr2=0x7fa408105fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.939+0000 7fa40d3ce640 1 -- 172.21.15.107:0/1539993000 shutdown_connections 2024-05-09T08:33:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:12.939+0000 7fa40d3ce640 1 -- 172.21.15.107:0/1539993000 wait complete. 2024-05-09T08:33:12.950 INFO:teuthology.run_tasks:Running task sequential... 2024-05-09T08:33:12.960 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-09T08:33:12.960 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-09T08:33:12.961 INFO:teuthology.task.print:Enabling mgr modules 2024-05-09T08:33:12.961 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-05-09T08:33:12.961 INFO:teuthology.task.exec:Executing custom commands... 2024-05-09T08:33:12.962 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-05-09T08:33:12.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-05-09T08:33:13.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 -- 172.21.15.107:0/836427695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45e4102340 msgr2=0x7f45e4108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:13.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 --2- 172.21.15.107:0/836427695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45e4102340 0x7f45e4108bd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f45dc009ca0 tx=0x7f45dc02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 -- 172.21.15.107:0/836427695 shutdown_connections 2024-05-09T08:33:13.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 --2- 172.21.15.107:0/836427695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45e4102340 0x7f45e4108bd0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 --2- 172.21.15.107:0/836427695 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45e4101980 0x7f45e4101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 --2- 172.21.15.107:0/836427695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45e4100790 0x7f45e4100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 -- 172.21.15.107:0/836427695 >> 172.21.15.107:0/836427695 conn(0x7f45e40fbd00 msgr2=0x7f45e40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:13.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 -- 172.21.15.107:0/836427695 shutdown_connections 2024-05-09T08:33:13.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 -- 172.21.15.107:0/836427695 wait complete. 2024-05-09T08:33:13.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 Processor -- start 2024-05-09T08:33:13.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 -- start start 2024-05-09T08:33:13.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45e4100790 0x7f45e41979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:13.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45e4101980 0x7f45e4197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:13.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45e4102340 0x7f45e419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:13.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f45e410d3e0 con 0x7f45e4100790 2024-05-09T08:33:13.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f45e410d260 con 0x7f45e4102340 2024-05-09T08:33:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45eaf22640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f45e410d560 con 0x7f45e4101980 2024-05-09T08:33:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45e8cbe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45e4100790 0x7f45e41979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45e8cbe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45e4100790 0x7f45e41979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34362/0 (socket says 172.21.15.107:34362) 2024-05-09T08:33:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.064+0000 7f45e8cbe640 1 -- 172.21.15.107:0/624621920 learned_addr learned my addr 172.21.15.107:0/624621920 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.065+0000 7f45e3fff640 1 --2- 172.21.15.107:0/624621920 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45e4101980 0x7f45e4197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.065+0000 7f45e8cbe640 1 -- 172.21.15.107:0/624621920 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45e4101980 msgr2=0x7f45e4197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.065+0000 7f45e8cbe640 1 --2- 172.21.15.107:0/624621920 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45e4101980 0x7f45e4197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.065+0000 7f45e8cbe640 1 -- 172.21.15.107:0/624621920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45e4102340 msgr2=0x7f45e419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.065+0000 7f45e8cbe640 1 --2- 172.21.15.107:0/624621920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45e4102340 0x7f45e419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.065+0000 7f45e8cbe640 1 -- 172.21.15.107:0/624621920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e419f5f0 con 0x7f45e4100790 2024-05-09T08:33:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.065+0000 7f45e3fff640 1 --2- 172.21.15.107:0/624621920 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45e4101980 0x7f45e4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.065+0000 7f45e8cbe640 1 --2- 172.21.15.107:0/624621920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45e4100790 0x7f45e41979d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f45d800d770 tx=0x7f45d800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.065+0000 7f45e1ffb640 1 -- 172.21.15.107:0/624621920 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8017070 con 0x7f45e4100790 2024-05-09T08:33:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.065+0000 7f45e1ffb640 1 -- 172.21.15.107:0/624621920 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f45d8009c30 con 0x7f45e4100790 2024-05-09T08:33:13.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.065+0000 7f45eaf22640 1 -- 172.21.15.107:0/624621920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45e419f8e0 con 0x7f45e4100790 2024-05-09T08:33:13.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.065+0000 7f45e1ffb640 1 -- 172.21.15.107:0/624621920 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8003d00 con 0x7f45e4100790 2024-05-09T08:33:13.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.065+0000 7f45eaf22640 1 -- 172.21.15.107:0/624621920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45e419fe30 con 0x7f45e4100790 2024-05-09T08:33:13.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.066+0000 7f45eaf22640 1 -- 172.21.15.107:0/624621920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45e4107b00 con 0x7f45e4100790 2024-05-09T08:33:13.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.067+0000 7f45e1ffb640 1 -- 172.21.15.107:0/624621920 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f45d8003ea0 con 0x7f45e4100790 2024-05-09T08:33:13.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.068+0000 7f45e1ffb640 1 --2- 172.21.15.107:0/624621920 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f45d40a2c30 0x7f45d40a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:13.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.068+0000 7f45e3fff640 1 --2- 172.21.15.107:0/624621920 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f45d40a2c30 0x7f45d40a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:13.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.068+0000 7f45e1ffb640 1 -- 172.21.15.107:0/624621920 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f45d80cf1c0 con 0x7f45e4100790 2024-05-09T08:33:13.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.068+0000 7f45e3fff640 1 --2- 172.21.15.107:0/624621920 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f45d40a2c30 0x7f45d40a50f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f45d00058c0 tx=0x7f45d0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:13.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.072+0000 7f45e1ffb640 1 -- 172.21.15.107:0/624621920 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f45d809bda0 con 0x7f45e4100790 2024-05-09T08:33:13.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.300+0000 7f45eaf22640 1 -- 172.21.15.107:0/624621920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f45e402d1c0 con 0x7f45e4100790 2024-05-09T08:33:13.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.561+0000 7f45e1ffb640 1 -- 172.21.15.107:0/624621920 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f45d8009de0 con 0x7f45e4100790 2024-05-09T08:33:13.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.563+0000 7f45eaf22640 1 -- 172.21.15.107:0/624621920 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f45d40a2c30 msgr2=0x7f45d40a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:13.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.563+0000 7f45eaf22640 1 --2- 172.21.15.107:0/624621920 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f45d40a2c30 0x7f45d40a50f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f45d00058c0 tx=0x7f45d0005850 comp rx=0 tx=0).stop 2024-05-09T08:33:13.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.563+0000 7f45eaf22640 1 -- 172.21.15.107:0/624621920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45e4100790 msgr2=0x7f45e41979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:13.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.563+0000 7f45eaf22640 1 --2- 172.21.15.107:0/624621920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45e4100790 0x7f45e41979d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f45d800d770 tx=0x7f45d800dc30 comp rx=0 tx=0).stop 2024-05-09T08:33:13.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.563+0000 7f45eaf22640 1 -- 172.21.15.107:0/624621920 shutdown_connections 2024-05-09T08:33:13.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.563+0000 7f45eaf22640 1 --2- 172.21.15.107:0/624621920 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f45d40a2c30 0x7f45d40a50f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.563+0000 7f45eaf22640 1 --2- 172.21.15.107:0/624621920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45e4102340 0x7f45e419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.563+0000 7f45eaf22640 1 --2- 172.21.15.107:0/624621920 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45e4101980 0x7f45e4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.563+0000 7f45eaf22640 1 --2- 172.21.15.107:0/624621920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45e4100790 0x7f45e41979d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.563+0000 7f45eaf22640 1 -- 172.21.15.107:0/624621920 >> 172.21.15.107:0/624621920 conn(0x7f45e40fbd00 msgr2=0x7f45e4104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:13.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.565+0000 7f45eaf22640 1 -- 172.21.15.107:0/624621920 shutdown_connections 2024-05-09T08:33:13.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.566+0000 7f45eaf22640 1 -- 172.21.15.107:0/624621920 wait complete. 2024-05-09T08:33:13.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-05-09T08:33:13.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.689+0000 7feffcae1640 1 -- 172.21.15.107:0/3671287484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff8102a80 msgr2=0x7feff8102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:13.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.689+0000 7feffcae1640 1 --2- 172.21.15.107:0/3671287484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff8102a80 0x7feff8102f00 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fefe0009900 tx=0x7fefe002a410 comp rx=0 tx=0).stop 2024-05-09T08:33:13.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.689+0000 7feffcae1640 1 -- 172.21.15.107:0/3671287484 shutdown_connections 2024-05-09T08:33:13.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.689+0000 7feffcae1640 1 --2- 172.21.15.107:0/3671287484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff8103440 0x7feff8109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.689+0000 7feffcae1640 1 --2- 172.21.15.107:0/3671287484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff8102a80 0x7feff8102f00 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.689+0000 7feffcae1640 1 --2- 172.21.15.107:0/3671287484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feff8101890 0x7feff8101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.689+0000 7feffcae1640 1 -- 172.21.15.107:0/3671287484 >> 172.21.15.107:0/3671287484 conn(0x7feff80fce20 msgr2=0x7feff80ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.689+0000 7feffcae1640 1 -- 172.21.15.107:0/3671287484 shutdown_connections 2024-05-09T08:33:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.689+0000 7feffcae1640 1 -- 172.21.15.107:0/3671287484 wait complete. 2024-05-09T08:33:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.689+0000 7feffcae1640 1 Processor -- start 2024-05-09T08:33:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feffcae1640 1 -- start start 2024-05-09T08:33:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feffcae1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feff8101890 0x7feff8197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feffcae1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff8102a80 0x7feff8197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feffcae1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff8103440 0x7feff819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feffcae1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feff810d3c0 con 0x7feff8103440 2024-05-09T08:33:13.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feffcae1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feff810d240 con 0x7feff8102a80 2024-05-09T08:33:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feffcae1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feff810d540 con 0x7feff8101890 2024-05-09T08:33:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feff6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff8103440 0x7feff819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feff5d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff8102a80 0x7feff8197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feff6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff8103440 0x7feff819efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34388/0 (socket says 172.21.15.107:34388) 2024-05-09T08:33:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feff6d9d640 1 -- 172.21.15.107:0/387267755 learned_addr learned my addr 172.21.15.107:0/387267755 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feff659c640 1 --2- 172.21.15.107:0/387267755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feff8101890 0x7feff8197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feff6d9d640 1 -- 172.21.15.107:0/387267755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feff8101890 msgr2=0x7feff8197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feff6d9d640 1 --2- 172.21.15.107:0/387267755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feff8101890 0x7feff8197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feff6d9d640 1 -- 172.21.15.107:0/387267755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff8102a80 msgr2=0x7feff8197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feff6d9d640 1 --2- 172.21.15.107:0/387267755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff8102a80 0x7feff8197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feff6d9d640 1 -- 172.21.15.107:0/387267755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feff819f620 con 0x7feff8103440 2024-05-09T08:33:13.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feff5d9b640 1 --2- 172.21.15.107:0/387267755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff8102a80 0x7feff8197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:33:13.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feff6d9d640 1 --2- 172.21.15.107:0/387267755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff8103440 0x7feff819efb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fefe800e970 tx=0x7fefe800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:13.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7fefef7fe640 1 -- 172.21.15.107:0/387267755 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe800ccc0 con 0x7feff8103440 2024-05-09T08:33:13.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feffcae1640 1 -- 172.21.15.107:0/387267755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feff819f910 con 0x7feff8103440 2024-05-09T08:33:13.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7fefef7fe640 1 -- 172.21.15.107:0/387267755 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fefe800ce60 con 0x7feff8103440 2024-05-09T08:33:13.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7fefef7fe640 1 -- 172.21.15.107:0/387267755 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe8010430 con 0x7feff8103440 2024-05-09T08:33:13.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.690+0000 7feffcae1640 1 -- 172.21.15.107:0/387267755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feff819fe30 con 0x7feff8103440 2024-05-09T08:33:13.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.691+0000 7feffcae1640 1 -- 172.21.15.107:0/387267755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feff8108bb0 con 0x7feff8103440 2024-05-09T08:33:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.693+0000 7fefef7fe640 1 -- 172.21.15.107:0/387267755 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7fefe8004ff0 con 0x7feff8103440 2024-05-09T08:33:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.693+0000 7fefef7fe640 1 --2- 172.21.15.107:0/387267755 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fefe40a2c80 0x7fefe40a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.693+0000 7feff659c640 1 -- 172.21.15.107:0/387267755 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fefe40a2c80 msgr2=0x7fefe40a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/846948416 2024-05-09T08:33:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.693+0000 7fefef7fe640 1 -- 172.21.15.107:0/387267755 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fefe80cf830 con 0x7feff8103440 2024-05-09T08:33:13.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.693+0000 7feff659c640 1 --2- 172.21.15.107:0/387267755 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fefe40a2c80 0x7fefe40a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:33:13.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.698+0000 7fefef7fe640 1 -- 172.21.15.107:0/387267755 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fefe809c410 con 0x7feff8103440 2024-05-09T08:33:13.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:13 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/624621920' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-09T08:33:13.844 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:13 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: ignoring --setuser ceph since I am not root 2024-05-09T08:33:13.844 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:13 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: ignoring --setgroup ceph since I am not root 2024-05-09T08:33:13.844 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:13 smithi157 ceph-mgr[35480]: -- 172.21.15.157:0/133683065 <== mon.1 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563979b345a0 con 0x563979b09000 2024-05-09T08:33:13.844 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:13 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:13.742+0000 7fb30586e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:33:13.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:13 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/624621920' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-09T08:33:13.871 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:13 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: ignoring --setuser ceph since I am not root 2024-05-09T08:33:13.871 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:13 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: ignoring --setgroup ceph since I am not root 2024-05-09T08:33:13.871 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:13 smithi110 ceph-mgr[35013]: -- 172.21.15.110:0/2813789132 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f26dc265a0 con 0x55f26dbfac00 2024-05-09T08:33:13.871 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:13 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:13.742+0000 7f175fa23200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:33:13.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.881+0000 7feffcae1640 1 -- 172.21.15.107:0/387267755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7feff8062430 con 0x7feff8103440 2024-05-09T08:33:13.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.885+0000 7fefef7fe640 1 -- 172.21.15.107:0/387267755 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v30)=0 v30) ==== 138+0+0 (secure 0 0 0) 0x7fefe809bd10 con 0x7feff8103440 2024-05-09T08:33:13.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.887+0000 7feffcae1640 1 -- 172.21.15.107:0/387267755 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fefe40a2c80 msgr2=0x7fefe40a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:33:13.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.887+0000 7feffcae1640 1 --2- 172.21.15.107:0/387267755 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fefe40a2c80 0x7fefe40a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.887+0000 7feffcae1640 1 -- 172.21.15.107:0/387267755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff8103440 msgr2=0x7feff819efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:13.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.887+0000 7feffcae1640 1 --2- 172.21.15.107:0/387267755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff8103440 0x7feff819efb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fefe800e970 tx=0x7fefe800ed70 comp rx=0 tx=0).stop 2024-05-09T08:33:13.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.888+0000 7feffcae1640 1 -- 172.21.15.107:0/387267755 shutdown_connections 2024-05-09T08:33:13.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.888+0000 7feffcae1640 1 --2- 172.21.15.107:0/387267755 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fefe40a2c80 0x7fefe40a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.888+0000 7feffcae1640 1 --2- 172.21.15.107:0/387267755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff8103440 0x7feff819efb0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.888+0000 7feffcae1640 1 --2- 172.21.15.107:0/387267755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff8102a80 0x7feff8197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.888+0000 7feffcae1640 1 --2- 172.21.15.107:0/387267755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feff8101890 0x7feff8197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:13.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.888+0000 7feffcae1640 1 -- 172.21.15.107:0/387267755 >> 172.21.15.107:0/387267755 conn(0x7feff80fce20 msgr2=0x7feff8105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:13.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.888+0000 7feffcae1640 1 -- 172.21.15.107:0/387267755 shutdown_connections 2024-05-09T08:33:13.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:13.888+0000 7feffcae1640 1 -- 172.21.15.107:0/387267755 wait complete. 2024-05-09T08:33:13.897 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:13 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/624621920' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-09T08:33:13.897 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:13 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: ignoring --setuser ceph since I am not root 2024-05-09T08:33:13.897 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:13 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: ignoring --setgroup ceph since I am not root 2024-05-09T08:33:13.897 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:13 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:13.750+0000 7fe2e5350200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T08:33:13.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-05-09T08:33:14.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.002+0000 7f8103561640 1 -- 172.21.15.107:0/2264437990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc102a80 msgr2=0x7f80fc102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:14.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.002+0000 7f8103561640 1 --2- 172.21.15.107:0/2264437990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc102a80 0x7f80fc102f00 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009900 tx=0x7f80f802a410 comp rx=0 tx=0).stop 2024-05-09T08:33:14.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.002+0000 7f8103561640 1 -- 172.21.15.107:0/2264437990 shutdown_connections 2024-05-09T08:33:14.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.002+0000 7f8103561640 1 --2- 172.21.15.107:0/2264437990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80fc103440 0x7f80fc109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:14.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.002+0000 7f8103561640 1 --2- 172.21.15.107:0/2264437990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc102a80 0x7f80fc102f00 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:14.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.002+0000 7f8103561640 1 --2- 172.21.15.107:0/2264437990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80fc101890 0x7f80fc101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:14.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.002+0000 7f8103561640 1 -- 172.21.15.107:0/2264437990 >> 172.21.15.107:0/2264437990 conn(0x7f80fc0fce20 msgr2=0x7f80fc0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:14.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.002+0000 7f8103561640 1 -- 172.21.15.107:0/2264437990 shutdown_connections 2024-05-09T08:33:14.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.002+0000 7f8103561640 1 -- 172.21.15.107:0/2264437990 wait complete. 2024-05-09T08:33:14.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.003+0000 7f8103561640 1 Processor -- start 2024-05-09T08:33:14.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.003+0000 7f8103561640 1 -- start start 2024-05-09T08:33:14.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.003+0000 7f8103561640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc101890 0x7f80fc1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:14.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.003+0000 7f8103561640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80fc102a80 0x7f80fc197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:14.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.003+0000 7f8103561640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80fc103440 0x7f80fc19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:14.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.003+0000 7f8103561640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc10d3a0 con 0x7f80fc101890 2024-05-09T08:33:14.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.003+0000 7f8103561640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc10d220 con 0x7f80fc102a80 2024-05-09T08:33:14.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.003+0000 7f8103561640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc10d520 con 0x7f80fc103440 2024-05-09T08:33:14.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.003+0000 7f81012fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc101890 0x7f80fc1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:14.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.003+0000 7f8100afc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80fc102a80 0x7f80fc197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:14.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.003+0000 7f81012fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc101890 0x7f80fc1979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34406/0 (socket says 172.21.15.107:34406) 2024-05-09T08:33:14.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.003+0000 7f81012fd640 1 -- 172.21.15.107:0/1531080440 learned_addr learned my addr 172.21.15.107:0/1531080440 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:14.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.003+0000 7f8100afc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80fc102a80 0x7f80fc197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:50884/0 (socket says 172.21.15.107:50884) 2024-05-09T08:33:14.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.003+0000 7f8101afe640 1 --2- 172.21.15.107:0/1531080440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80fc103440 0x7f80fc19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:14.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.004+0000 7f81012fd640 1 -- 172.21.15.107:0/1531080440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80fc103440 msgr2=0x7f80fc19ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:14.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.004+0000 7f81012fd640 1 --2- 172.21.15.107:0/1531080440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80fc103440 0x7f80fc19ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:14.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.004+0000 7f81012fd640 1 -- 172.21.15.107:0/1531080440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80fc102a80 msgr2=0x7f80fc197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:14.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.004+0000 7f81012fd640 1 --2- 172.21.15.107:0/1531080440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80fc102a80 0x7f80fc197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:14.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.004+0000 7f81012fd640 1 -- 172.21.15.107:0/1531080440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80fc19f600 con 0x7f80fc101890 2024-05-09T08:33:14.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.004+0000 7f8100afc640 1 --2- 172.21.15.107:0/1531080440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80fc102a80 0x7f80fc197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:33:14.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.004+0000 7f81012fd640 1 --2- 172.21.15.107:0/1531080440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc101890 0x7f80fc1979e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00e9c0 tx=0x7f80ec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:14.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.004+0000 7f80f27fc640 1 -- 172.21.15.107:0/1531080440 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec00cd80 con 0x7f80fc101890 2024-05-09T08:33:14.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.004+0000 7f8103561640 1 -- 172.21.15.107:0/1531080440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80fc19f8f0 con 0x7f80fc101890 2024-05-09T08:33:14.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.004+0000 7f80f27fc640 1 -- 172.21.15.107:0/1531080440 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80ec0035f0 con 0x7f80fc101890 2024-05-09T08:33:14.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.004+0000 7f80f27fc640 1 -- 172.21.15.107:0/1531080440 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec010690 con 0x7f80fc101890 2024-05-09T08:33:14.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.004+0000 7f8103561640 1 -- 172.21.15.107:0/1531080440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80fc19fe10 con 0x7f80fc101890 2024-05-09T08:33:14.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.005+0000 7f8103561640 1 -- 172.21.15.107:0/1531080440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80fc108bb0 con 0x7f80fc101890 2024-05-09T08:33:14.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.006+0000 7f80f27fc640 1 -- 172.21.15.107:0/1531080440 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f80ec0037a0 con 0x7f80fc101890 2024-05-09T08:33:14.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.007+0000 7f80f27fc640 1 --2- 172.21.15.107:0/1531080440 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f80e80a2c80 0x7f80e80a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:14.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.007+0000 7f8100afc640 1 -- 172.21.15.107:0/1531080440 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f80e80a2c80 msgr2=0x7f80e80a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/846948416 2024-05-09T08:33:14.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.007+0000 7f80f27fc640 1 -- 172.21.15.107:0/1531080440 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f80ec0cf7c0 con 0x7f80fc101890 2024-05-09T08:33:14.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.007+0000 7f8100afc640 1 --2- 172.21.15.107:0/1531080440 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f80e80a2c80 0x7f80e80a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:33:14.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.012+0000 7f80f27fc640 1 -- 172.21.15.107:0/1531080440 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f80ec09c3a0 con 0x7f80fc101890 2024-05-09T08:33:14.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.193+0000 7f8103561640 1 -- 172.21.15.107:0/1531080440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f80fc0624f0 con 0x7f80fc101890 2024-05-09T08:33:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.197+0000 7f80f27fc640 1 -- 172.21.15.107:0/1531080440 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v31)=0 v31) ==== 133+0+0 (secure 0 0 0) 0x7f80ec09bca0 con 0x7f80fc101890 2024-05-09T08:33:14.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.199+0000 7f8103561640 1 -- 172.21.15.107:0/1531080440 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f80e80a2c80 msgr2=0x7f80e80a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:33:14.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.199+0000 7f8103561640 1 --2- 172.21.15.107:0/1531080440 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f80e80a2c80 0x7f80e80a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:14.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.199+0000 7f8103561640 1 -- 172.21.15.107:0/1531080440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc101890 msgr2=0x7f80fc1979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:14.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.199+0000 7f8103561640 1 --2- 172.21.15.107:0/1531080440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc101890 0x7f80fc1979e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00e9c0 tx=0x7f80ec00ee80 comp rx=0 tx=0).stop 2024-05-09T08:33:14.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.199+0000 7f8103561640 1 -- 172.21.15.107:0/1531080440 shutdown_connections 2024-05-09T08:33:14.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.199+0000 7f8103561640 1 --2- 172.21.15.107:0/1531080440 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f80e80a2c80 0x7f80e80a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:14.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.199+0000 7f8103561640 1 --2- 172.21.15.107:0/1531080440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80fc103440 0x7f80fc19ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:14.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.199+0000 7f8103561640 1 --2- 172.21.15.107:0/1531080440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80fc102a80 0x7f80fc197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:14.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.199+0000 7f8103561640 1 --2- 172.21.15.107:0/1531080440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc101890 0x7f80fc1979e0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:14.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.199+0000 7f8103561640 1 -- 172.21.15.107:0/1531080440 >> 172.21.15.107:0/1531080440 conn(0x7f80fc0fce20 msgr2=0x7f80fc105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:14.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.199+0000 7f8103561640 1 -- 172.21.15.107:0/1531080440 shutdown_connections 2024-05-09T08:33:14.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:14.199+0000 7f8103561640 1 -- 172.21.15.107:0/1531080440 wait complete. 2024-05-09T08:33:14.208 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:13 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:13.943+0000 7fe2e5350200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:33:14.209 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:14 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:14.052+0000 7fe2e5350200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:33:14.210 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-05-09T08:33:14.221 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-05-09T08:33:14.221 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-05-09T08:33:14.222 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-05-09T08:33:14.222 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T08:33:14.222 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T08:33:14.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:33:14.222 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2024-05-09T08:33:14.238 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-05-09T08:33:14.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:33:14.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T08:33:14.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:33:14.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link delete ceph-brx 2024-05-09T08:33:14.335 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:13 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:13.927+0000 7fb30586e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:33:14.335 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:14 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:14.029+0000 7fb30586e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:33:14.336 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:14 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:14.335+0000 7fb30586e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:33:14.336 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:13 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:13.926+0000 7f175fa23200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:33:14.336 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:14 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:14.028+0000 7f175fa23200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T08:33:14.336 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:14 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:14.336+0000 7f175fa23200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:33:14.407 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-05-09T08:33:14.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph fs ls 2024-05-09T08:33:14.473 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:14 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:14.381+0000 7fe2e5350200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T08:33:14.474 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:14 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:14.474+0000 7fe2e5350200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:33:14.593 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:14 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:14.420+0000 7fb30586e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:33:14.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:14 smithi157 ceph-mon[33614]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:33:14.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:14 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/624621920' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-09T08:33:14.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:14 smithi157 ceph-mon[33614]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-05-09T08:33:14.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:14 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/387267755' entity='client.admin' 2024-05-09T08:33:14.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:14 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1531080440' entity='client.admin' 2024-05-09T08:33:14.621 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:14 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:14.421+0000 7f175fa23200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T08:33:14.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:14 smithi110 ceph-mon[33622]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:33:14.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:14 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/624621920' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-09T08:33:14.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:14 smithi110 ceph-mon[33622]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-05-09T08:33:14.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:14 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/387267755' entity='client.admin' 2024-05-09T08:33:14.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:14 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1531080440' entity='client.admin' 2024-05-09T08:33:14.661 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:33:14.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:14 smithi107 ceph-mon[30122]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:33:14.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:14 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/624621920' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-09T08:33:14.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:14 smithi107 ceph-mon[30122]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-05-09T08:33:14.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:14 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/387267755' entity='client.admin' 2024-05-09T08:33:14.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:14 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1531080440' entity='client.admin' 2024-05-09T08:33:15.779 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:15 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:15.706+0000 7fe2e5350200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:33:15.870 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:15 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:15.589+0000 7f175fa23200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:33:15.871 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:15 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:15.757+0000 7f175fa23200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:33:15.936 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:15 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:15.598+0000 7fb30586e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T08:33:15.937 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:15 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:15.766+0000 7fb30586e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:33:15.937 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:15 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:15.937+0000 7fb30586e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:33:16.155 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:15 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:15.887+0000 7fe2e5350200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T08:33:16.155 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:16 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:16.066+0000 7fe2e5350200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:33:16.216 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:15 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:15.927+0000 7f175fa23200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T08:33:16.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.396+0000 7fd487612700 1 -- 172.21.15.107:0/1050907819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd480101210 msgr2=0x7fd480101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:16.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.396+0000 7fd487612700 1 --2- 172.21.15.107:0/1050907819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd480101210 0x7fd480101690 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd470009990 tx=0x7fd470009ca0 comp rx=0 tx=0).stop 2024-05-09T08:33:16.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.396+0000 7fd487612700 1 -- 172.21.15.107:0/1050907819 shutdown_connections 2024-05-09T08:33:16.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.396+0000 7fd487612700 1 --2- 172.21.15.107:0/1050907819 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd480101bd0 0x7fd48010e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:16.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.396+0000 7fd487612700 1 --2- 172.21.15.107:0/1050907819 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd480107280 0x7fd480107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:16.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.396+0000 7fd487612700 1 --2- 172.21.15.107:0/1050907819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd480101210 0x7fd480101690 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:16.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.396+0000 7fd487612700 1 -- 172.21.15.107:0/1050907819 >> 172.21.15.107:0/1050907819 conn(0x7fd480078190 msgr2=0x7fd4800785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:16.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.396+0000 7fd487612700 1 -- 172.21.15.107:0/1050907819 shutdown_connections 2024-05-09T08:33:16.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.396+0000 7fd487612700 1 -- 172.21.15.107:0/1050907819 wait complete. 2024-05-09T08:33:16.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd487612700 1 Processor -- start 2024-05-09T08:33:16.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd487612700 1 -- start start 2024-05-09T08:33:16.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd487612700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd480101210 0x7fd4801975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd487612700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd480101bd0 0x7fd480197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd487612700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd480107280 0x7fd480198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd487612700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd4801113a0 con 0x7fd480101210 2024-05-09T08:33:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd487612700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd480111220 con 0x7fd480101bd0 2024-05-09T08:33:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd487612700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd480111520 con 0x7fd480107280 2024-05-09T08:33:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd4853ae700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd480101210 0x7fd4801975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd484bad700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd480101bd0 0x7fd480197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd4853ae700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd480101210 0x7fd4801975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34412/0 (socket says 172.21.15.107:34412) 2024-05-09T08:33:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd4853ae700 1 -- 172.21.15.107:0/3423733512 learned_addr learned my addr 172.21.15.107:0/3423733512 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd485baf700 1 --2- 172.21.15.107:0/3423733512 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd480107280 0x7fd480198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd484bad700 1 -- 172.21.15.107:0/3423733512 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd480107280 msgr2=0x7fd480198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd484bad700 1 --2- 172.21.15.107:0/3423733512 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd480107280 0x7fd480198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd484bad700 1 -- 172.21.15.107:0/3423733512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd480101210 msgr2=0x7fd4801975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd484bad700 1 --2- 172.21.15.107:0/3423733512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd480101210 0x7fd4801975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd484bad700 1 -- 172.21.15.107:0/3423733512 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4801a6e40 con 0x7fd480101bd0 2024-05-09T08:33:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.397+0000 7fd4853ae700 1 --2- 172.21.15.107:0/3423733512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd480101210 0x7fd4801975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:33:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.398+0000 7fd484bad700 1 --2- 172.21.15.107:0/3423733512 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd480101bd0 0x7fd480197b10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd470006010 tx=0x7fd4700057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.398+0000 7fd4767fc700 1 -- 172.21.15.107:0/3423733512 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd470027070 con 0x7fd480101bd0 2024-05-09T08:33:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.398+0000 7fd487612700 1 -- 172.21.15.107:0/3423733512 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4801a70d0 con 0x7fd480101bd0 2024-05-09T08:33:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.398+0000 7fd4767fc700 1 -- 172.21.15.107:0/3423733512 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd470003f10 con 0x7fd480101bd0 2024-05-09T08:33:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.398+0000 7fd4767fc700 1 -- 172.21.15.107:0/3423733512 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd470022ba0 con 0x7fd480101bd0 2024-05-09T08:33:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.398+0000 7fd487612700 1 -- 172.21.15.107:0/3423733512 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4801a7580 con 0x7fd480101bd0 2024-05-09T08:33:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.399+0000 7fd487612700 1 -- 172.21.15.107:0/3423733512 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd480069010 con 0x7fd480101bd0 2024-05-09T08:33:16.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.401+0000 7fd4767fc700 1 -- 172.21.15.107:0/3423733512 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7fd47002a070 con 0x7fd480101bd0 2024-05-09T08:33:16.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.401+0000 7fd4767fc700 1 --2- 172.21.15.107:0/3423733512 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd46c0a2c70 0x7fd46c0a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:16.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.401+0000 7fd4853ae700 1 -- 172.21.15.107:0/3423733512 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd46c0a2c70 msgr2=0x7fd46c0a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/846948416 2024-05-09T08:33:16.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.402+0000 7fd4767fc700 1 -- 172.21.15.107:0/3423733512 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fd4700d0320 con 0x7fd480101bd0 2024-05-09T08:33:16.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.402+0000 7fd4853ae700 1 --2- 172.21.15.107:0/3423733512 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd46c0a2c70 0x7fd46c0a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:33:16.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.406+0000 7fd4767fc700 1 -- 172.21.15.107:0/3423733512 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd47005e740 con 0x7fd480101bd0 2024-05-09T08:33:16.405 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:16 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:16.385+0000 7fe2e5350200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:33:16.509 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:16 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:16.216+0000 7f175fa23200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:33:16.509 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:16 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:16.308+0000 7f175fa23200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:33:16.509 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:16 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:16.509+0000 7f175fa23200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:33:16.528 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:16 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:16.231+0000 7fb30586e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T08:33:16.528 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:16 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:16.324+0000 7fb30586e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:33:16.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.603+0000 7fd4853ae700 1 -- 172.21.15.107:0/3423733512 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd46c0a2c70 msgr2=0x7fd46c0a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/846948416 2024-05-09T08:33:16.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.603+0000 7fd4853ae700 1 --2- 172.21.15.107:0/3423733512 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd46c0a2c70 0x7fd46c0a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:33:16.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.655+0000 7fd487612700 1 -- 172.21.15.107:0/3423733512 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd48002db80 con 0x7fd480101bd0 2024-05-09T08:33:16.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.656+0000 7fd4767fc700 1 -- 172.21.15.107:0/3423733512 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+88 (secure 0 0 0) 0x7fd470013070 con 0x7fd480101bd0 2024-05-09T08:33:16.655 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-09T08:33:16.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.658+0000 7fd487612700 1 -- 172.21.15.107:0/3423733512 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd46c0a2c70 msgr2=0x7fd46c0a5130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:33:16.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.658+0000 7fd487612700 1 --2- 172.21.15.107:0/3423733512 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd46c0a2c70 0x7fd46c0a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:16.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.658+0000 7fd487612700 1 -- 172.21.15.107:0/3423733512 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd480101bd0 msgr2=0x7fd480197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:16.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.658+0000 7fd487612700 1 --2- 172.21.15.107:0/3423733512 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd480101bd0 0x7fd480197b10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd470006010 tx=0x7fd4700057c0 comp rx=0 tx=0).stop 2024-05-09T08:33:16.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.658+0000 7fd487612700 1 -- 172.21.15.107:0/3423733512 shutdown_connections 2024-05-09T08:33:16.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.658+0000 7fd487612700 1 --2- 172.21.15.107:0/3423733512 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd480107280 0x7fd480198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:16.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.658+0000 7fd487612700 1 --2- 172.21.15.107:0/3423733512 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7fd46c0a2c70 0x7fd46c0a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:16.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.658+0000 7fd487612700 1 --2- 172.21.15.107:0/3423733512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd480101210 0x7fd4801975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:16.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.658+0000 7fd487612700 1 --2- 172.21.15.107:0/3423733512 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd480101bd0 0x7fd480197b10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:16.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.658+0000 7fd487612700 1 -- 172.21.15.107:0/3423733512 >> 172.21.15.107:0/3423733512 conn(0x7fd480078190 msgr2=0x7fd4800ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:16.658 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:16 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:16.485+0000 7fe2e5350200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T08:33:16.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.659+0000 7fd487612700 1 -- 172.21.15.107:0/3423733512 shutdown_connections 2024-05-09T08:33:16.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:16.659+0000 7fd487612700 1 -- 172.21.15.107:0/3423733512 wait complete. 2024-05-09T08:33:16.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:16 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3423733512' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:33:16.843 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:16 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:16.529+0000 7fb30586e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:33:16.844 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:16 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:16.724+0000 7fb30586e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:33:16.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:16 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3423733512' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:33:16.870 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:16 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:16.704+0000 7f175fa23200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:33:17.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:16 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3423733512' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T08:33:17.029 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:16 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:16.704+0000 7fe2e5350200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T08:33:17.029 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:16 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:16.907+0000 7fe2e5350200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T08:33:17.366 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T08:33:17.366 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T08:33:17.366 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2024-05-09T08:33:17.366 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T08:33:17.366 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T08:33:17.366 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T08:33:17.366 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T08:33:17.367 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T08:33:17.367 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T08:33:17.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:33:17.367 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2024-05-09T08:33:17.384 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T08:33:17.384 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T08:33:17.384 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T08:33:17.384 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:33:17.385 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2024-05-09T08:33:17.385 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:33:17.385 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:33:17.385 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2024-05-09T08:33:17.385 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2024-05-09T08:33:17.385 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:33:17.385 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2024-05-09T08:33:17.385 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2024-05-09T08:33:17.385 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T08:33:17.385 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2024-05-09T08:33:17.385 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T08:33:17.385 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2024-05-09T08:33:17.385 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-09T08:33:17.385 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 42102sec preferred_lft 42102sec 2024-05-09T08:33:17.386 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2024-05-09T08:33:17.386 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-05-09T08:33:17.386 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-09T08:33:17.386 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:33:17.386 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-05-09T08:33:17.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add name ceph-brx type bridge 2024-05-09T08:33:17.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr flush dev ceph-brx 2024-05-09T08:33:17.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set ceph-brx up 2024-05-09T08:33:17.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-09T08:33:17.386 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-05-09T08:33:17.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:33:17.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:33:17.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:33:17.533 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T08:33:17.558 INFO:teuthology.orchestra.run.smithi107.stdout:1 2024-05-09T08:33:17.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:33:17.560 DEBUG:teuthology.orchestra.run.smithi107:> route 2024-05-09T08:33:17.593 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:17 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:17.326+0000 7fb30586e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:33:17.617 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2024-05-09T08:33:17.617 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T08:33:17.617 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T08:33:17.617 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T08:33:17.617 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T08:33:17.618 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:33:17.618 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-05-09T08:33:17.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T08:33:17.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T08:33:17.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T08:33:17.618 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-05-09T08:33:17.620 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:17 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:17.303+0000 7f175fa23200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:33:17.664 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:17 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:17.538+0000 7fe2e5350200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T08:33:17.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:33:17.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:33:17.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:33:17.749 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2024-05-09T08:33:17.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:33:17.805 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list-id 2024-05-09T08:33:17.861 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:33:17.862 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-05-09T08:33:17.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T08:33:17.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-09T08:33:17.862 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-05-09T08:33:17.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:33:17.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:33:18.003 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-09T08:33:18.003 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:33:18.003 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-05-09T08:33:18.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-09T08:33:18.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-09T08:33:18.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-09T08:33:18.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-09T08:33:18.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-09T08:33:18.004 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-05-09T08:33:18.017 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:17 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:17.721+0000 7f175fa23200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:33:18.017 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:17 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:17.817+0000 7f175fa23200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:33:18.017 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:18 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:18.018+0000 7f175fa23200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:33:18.038 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:17 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:17.742+0000 7fb30586e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:33:18.038 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:17 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:17.838+0000 7fb30586e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:33:18.038 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:18 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:18.039+0000 7fb30586e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:33:18.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:33:18.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:33:18.125 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T08:33:18.125 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-05-09T08:33:18.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2024-05-09T08:33:18.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set dev brx.0 master ceph-brx 2024-05-09T08:33:18.125 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-05-09T08:33:18.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T08:33:18.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:33:18.204 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-05-09T08:33:18.204 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T08:33:18.204 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T08:33:18.219 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T08:33:18.219 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T08:33:18.219 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T08:33:18.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe fuse 2024-05-09T08:33:18.279 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:18 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:18.041+0000 7fe2e5350200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T08:33:18.279 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:18 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:18.154+0000 7fe2e5350200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T08:33:18.304 DEBUG:teuthology.orchestra.run.smithi107:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T08:33:18.361 INFO:teuthology.orchestra.run.smithi107.stdout:/proc 2024-05-09T08:33:18.361 INFO:teuthology.orchestra.run.smithi107.stdout:/sys 2024-05-09T08:33:18.361 INFO:teuthology.orchestra.run.smithi107.stdout:/dev 2024-05-09T08:33:18.361 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/security 2024-05-09T08:33:18.361 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/shm 2024-05-09T08:33:18.361 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/pts 2024-05-09T08:33:18.361 INFO:teuthology.orchestra.run.smithi107.stdout:/run 2024-05-09T08:33:18.361 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/cgroup 2024-05-09T08:33:18.362 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/pstore 2024-05-09T08:33:18.362 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/bpf 2024-05-09T08:33:18.362 INFO:teuthology.orchestra.run.smithi107.stdout:/ 2024-05-09T08:33:18.362 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/selinux 2024-05-09T08:33:18.362 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T08:33:18.362 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/hugepages 2024-05-09T08:33:18.362 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/mqueue 2024-05-09T08:33:18.362 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/debug 2024-05-09T08:33:18.362 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/tracing 2024-05-09T08:33:18.362 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/fuse/connections 2024-05-09T08:33:18.362 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/config 2024-05-09T08:33:18.362 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T08:33:18.362 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T08:33:18.363 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T08:33:18.363 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T08:33:18.363 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/1000 2024-05-09T08:33:18.363 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph 2024-05-09T08:33:18.363 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay 2024-05-09T08:33:18.363 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/704d9ece21f21936ee943a1df4a489c227907ae684576c39a8326b3dffe1a938/merged 2024-05-09T08:33:18.363 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/1e139d9b7f0b4f3d0acd9e514aed736ee8b756b059cc812befde73c4c74283e1/merged 2024-05-09T08:33:18.363 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/0 2024-05-09T08:33:18.363 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T08:33:18.363 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/a889fd515b5c82d4e0853e5640c5b24fcb34bff091c8962c29ae0a604310eb0b/merged 2024-05-09T08:33:18.363 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/3d41909142bc28d4fd2e879ba0a88e8c06c4840feaa0e281755495824be69163/merged 2024-05-09T08:33:18.363 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/05b85167b2c4ce29a61409c04100fde4c31847c93e288fb610ba6e1602ab8244/merged 2024-05-09T08:33:18.363 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/b7725f5648670ca7bd7869e201bca45c3f7149327f817da5c38cf7746e9f9351/merged 2024-05-09T08:33:18.364 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/f13a66b4af406ede639af2c293c228fc4964dcfda360e9e3f2801e94971040de/merged 2024-05-09T08:33:18.364 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/608847f69c5463ec5a21eb5968f202c44dc6fc079b4197023aeb147c241fa57a/merged 2024-05-09T08:33:18.364 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/5dd9f0ea09fa33d9913a931882dcca1e45f9f3d58177ac85ca98f0378aa9ed6c/merged 2024-05-09T08:33:18.364 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/bf4c3ee07438db841c1cc72e7775f915ef64c3d8c7af00ee701c8214a7dfcec9/merged 2024-05-09T08:33:18.364 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns 2024-05-09T08:33:18.364 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T08:33:18.364 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T08:33:18.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:33:18.364 DEBUG:teuthology.orchestra.run.smithi107:> ls /sys/fs/fuse/connections 2024-05-09T08:33:18.417 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T08:33:18.417 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/53dbbdc5-d77e-40a8-9491-15f95d558fdc --client_fs=cephfs) 2024-05-09T08:33:18.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe fuse 2024-05-09T08:33:18.487 DEBUG:teuthology.orchestra.run.smithi107:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T08:33:18.545 INFO:teuthology.orchestra.run.smithi107.stdout:/proc 2024-05-09T08:33:18.545 INFO:teuthology.orchestra.run.smithi107.stdout:/sys 2024-05-09T08:33:18.545 INFO:teuthology.orchestra.run.smithi107.stdout:/dev 2024-05-09T08:33:18.545 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/security 2024-05-09T08:33:18.545 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/shm 2024-05-09T08:33:18.546 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/pts 2024-05-09T08:33:18.546 INFO:teuthology.orchestra.run.smithi107.stdout:/run 2024-05-09T08:33:18.546 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/cgroup 2024-05-09T08:33:18.546 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/pstore 2024-05-09T08:33:18.546 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/bpf 2024-05-09T08:33:18.546 INFO:teuthology.orchestra.run.smithi107.stdout:/ 2024-05-09T08:33:18.546 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/selinux 2024-05-09T08:33:18.546 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T08:33:18.546 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/hugepages 2024-05-09T08:33:18.546 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/mqueue 2024-05-09T08:33:18.546 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/debug 2024-05-09T08:33:18.547 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/tracing 2024-05-09T08:33:18.547 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/fuse/connections 2024-05-09T08:33:18.547 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/config 2024-05-09T08:33:18.547 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T08:33:18.547 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T08:33:18.547 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T08:33:18.547 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T08:33:18.547 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/1000 2024-05-09T08:33:18.547 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph 2024-05-09T08:33:18.547 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay 2024-05-09T08:33:18.547 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/704d9ece21f21936ee943a1df4a489c227907ae684576c39a8326b3dffe1a938/merged 2024-05-09T08:33:18.547 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/1e139d9b7f0b4f3d0acd9e514aed736ee8b756b059cc812befde73c4c74283e1/merged 2024-05-09T08:33:18.547 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/0 2024-05-09T08:33:18.547 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T08:33:18.548 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/a889fd515b5c82d4e0853e5640c5b24fcb34bff091c8962c29ae0a604310eb0b/merged 2024-05-09T08:33:18.548 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/3d41909142bc28d4fd2e879ba0a88e8c06c4840feaa0e281755495824be69163/merged 2024-05-09T08:33:18.548 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/05b85167b2c4ce29a61409c04100fde4c31847c93e288fb610ba6e1602ab8244/merged 2024-05-09T08:33:18.548 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/b7725f5648670ca7bd7869e201bca45c3f7149327f817da5c38cf7746e9f9351/merged 2024-05-09T08:33:18.548 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/f13a66b4af406ede639af2c293c228fc4964dcfda360e9e3f2801e94971040de/merged 2024-05-09T08:33:18.548 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/608847f69c5463ec5a21eb5968f202c44dc6fc079b4197023aeb147c241fa57a/merged 2024-05-09T08:33:18.548 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/5dd9f0ea09fa33d9913a931882dcca1e45f9f3d58177ac85ca98f0378aa9ed6c/merged 2024-05-09T08:33:18.548 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/bf4c3ee07438db841c1cc72e7775f915ef64c3d8c7af00ee701c8214a7dfcec9/merged 2024-05-09T08:33:18.548 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns 2024-05-09T08:33:18.548 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T08:33:18.548 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T08:33:18.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:33:18.549 DEBUG:teuthology.orchestra.run.smithi107:> ls /sys/fs/fuse/connections 2024-05-09T08:33:18.549 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi107.stderr:2024-05-09T08:33:18.550+0000 7f12422f5480 -1 init, newargv = 0x560a5fe38300 newargc=15 2024-05-09T08:33:18.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi107.stderr:ceph-fuse[81701]: starting ceph client 2024-05-09T08:33:18.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi107.stderr:ceph-fuse[81701]: starting fuse 2024-05-09T08:33:18.567 INFO:teuthology.orchestra.run.smithi107.stdout:114 2024-05-09T08:33:18.567 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [114] 2024-05-09T08:33:18.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo stdin-killer -- python3 -c ' 2024-05-09T08:33:18.567 DEBUG:teuthology.orchestra.run.smithi107:> import glob 2024-05-09T08:33:18.567 DEBUG:teuthology.orchestra.run.smithi107:> import re 2024-05-09T08:33:18.567 DEBUG:teuthology.orchestra.run.smithi107:> import os 2024-05-09T08:33:18.567 DEBUG:teuthology.orchestra.run.smithi107:> import subprocess 2024-05-09T08:33:18.567 DEBUG:teuthology.orchestra.run.smithi107:> 2024-05-09T08:33:18.568 DEBUG:teuthology.orchestra.run.smithi107:> def _find_admin_socket(client_name): 2024-05-09T08:33:18.568 DEBUG:teuthology.orchestra.run.smithi107:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T08:33:18.568 DEBUG:teuthology.orchestra.run.smithi107:> files = glob.glob(asok_path) 2024-05-09T08:33:18.568 DEBUG:teuthology.orchestra.run.smithi107:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T08:33:18.568 DEBUG:teuthology.orchestra.run.smithi107:> 2024-05-09T08:33:18.568 DEBUG:teuthology.orchestra.run.smithi107:> # Given a non-glob path, it better be there 2024-05-09T08:33:18.568 DEBUG:teuthology.orchestra.run.smithi107:> if "*" not in asok_path: 2024-05-09T08:33:18.568 DEBUG:teuthology.orchestra.run.smithi107:> assert(len(files) == 1) 2024-05-09T08:33:18.568 DEBUG:teuthology.orchestra.run.smithi107:> return files[0] 2024-05-09T08:33:18.568 DEBUG:teuthology.orchestra.run.smithi107:> 2024-05-09T08:33:18.568 DEBUG:teuthology.orchestra.run.smithi107:> for f in files: 2024-05-09T08:33:18.568 DEBUG:teuthology.orchestra.run.smithi107:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T08:33:18.568 DEBUG:teuthology.orchestra.run.smithi107:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T08:33:18.568 DEBUG:teuthology.orchestra.run.smithi107:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T08:33:18.569 DEBUG:teuthology.orchestra.run.smithi107:> contents = proc_f.read() 2024-05-09T08:33:18.569 DEBUG:teuthology.orchestra.run.smithi107:> if mountpoint in contents: 2024-05-09T08:33:18.569 DEBUG:teuthology.orchestra.run.smithi107:> return f 2024-05-09T08:33:18.569 DEBUG:teuthology.orchestra.run.smithi107:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T08:33:18.569 DEBUG:teuthology.orchestra.run.smithi107:> 2024-05-09T08:33:18.569 DEBUG:teuthology.orchestra.run.smithi107:> print(_find_admin_socket("client.0")) 2024-05-09T08:33:18.569 DEBUG:teuthology.orchestra.run.smithi107:> ' 2024-05-09T08:33:18.630 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:18 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:18.383+0000 7fe2e5350200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T08:33:18.689 INFO:teuthology.orchestra.run.smithi107.stdout:/var/run/ceph/ceph-client.0.81701.asok 2024-05-09T08:33:18.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T08:33:18.700 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.81701.asok 2024-05-09T08:33:18.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:33:18.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.81701.asok status 2024-05-09T08:33:18.840 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:33:18.840 INFO:teuthology.orchestra.run.smithi107.stdout: "metadata": { 2024-05-09T08:33:18.840 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_sha1": "96805dcf909541c4ca9a1e90258de8ae5b6fa1bc", 2024-05-09T08:33:18.840 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T08:33:18.840 INFO:teuthology.orchestra.run.smithi107.stdout: "entity_id": "0", 2024-05-09T08:33:18.840 INFO:teuthology.orchestra.run.smithi107.stdout: "hostname": "smithi107", 2024-05-09T08:33:18.841 INFO:teuthology.orchestra.run.smithi107.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T08:33:18.841 INFO:teuthology.orchestra.run.smithi107.stdout: "pid": "81701", 2024-05-09T08:33:18.841 INFO:teuthology.orchestra.run.smithi107.stdout: "root": "/volumes/qa/sv_0/53dbbdc5-d77e-40a8-9491-15f95d558fdc" 2024-05-09T08:33:18.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:33:18.841 INFO:teuthology.orchestra.run.smithi107.stdout: "dentry_count": 0, 2024-05-09T08:33:18.841 INFO:teuthology.orchestra.run.smithi107.stdout: "dentry_pinned_count": 0, 2024-05-09T08:33:18.841 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 15507, 2024-05-09T08:33:18.841 INFO:teuthology.orchestra.run.smithi107.stdout: "inst": { 2024-05-09T08:33:18.841 INFO:teuthology.orchestra.run.smithi107.stdout: "name": { 2024-05-09T08:33:18.841 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client", 2024-05-09T08:33:18.841 INFO:teuthology.orchestra.run.smithi107.stdout: "num": 15507 2024-05-09T08:33:18.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:33:18.841 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": { 2024-05-09T08:33:18.841 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-05-09T08:33:18.842 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "192.168.0.1:0", 2024-05-09T08:33:18.842 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2968783461 2024-05-09T08:33:18.842 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:33:18.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:33:18.842 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": { 2024-05-09T08:33:18.842 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-05-09T08:33:18.842 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "192.168.0.1:0", 2024-05-09T08:33:18.842 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 2968783461 2024-05-09T08:33:18.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:33:18.842 INFO:teuthology.orchestra.run.smithi107.stdout: "inst_str": "client.15507 192.168.0.1:0/2968783461", 2024-05-09T08:33:18.842 INFO:teuthology.orchestra.run.smithi107.stdout: "addr_str": "192.168.0.1:0/2968783461", 2024-05-09T08:33:18.842 INFO:teuthology.orchestra.run.smithi107.stdout: "inode_count": 5, 2024-05-09T08:33:18.842 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_epoch": 24, 2024-05-09T08:33:18.842 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_epoch": 93, 2024-05-09T08:33:18.843 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_epoch_barrier": 0, 2024-05-09T08:33:18.843 INFO:teuthology.orchestra.run.smithi107.stdout: "blocklisted": false, 2024-05-09T08:33:18.843 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs" 2024-05-09T08:33:18.843 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:33:18.843 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:18 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:18.561+0000 7fb30586e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:33:18.843 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:18 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:18.653+0000 7fb30586e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:33:18.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:33:18.849 DEBUG:teuthology.orchestra.run.smithi107:> stat --file-system '--printf=%T 2024-05-09T08:33:18.849 DEBUG:teuthology.orchestra.run.smithi107:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T08:33:18.865 INFO:teuthology.orchestra.run.smithi107.stdout:fuseblk 2024-05-09T08:33:18.865 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T08:33:18.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:33:18.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T08:33:18.867 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:18 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:18.534+0000 7f175fa23200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:33:18.867 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:18 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:18.629+0000 7f175fa23200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:33:18.933 INFO:teuthology.run_tasks:Running task exec... 2024-05-09T08:33:18.943 INFO:teuthology.task.exec:Executing custom commands... 2024-05-09T08:33:18.943 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-05-09T08:33:18.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-05-09T08:33:19.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.079+0000 7f7d7bf42640 1 -- 172.21.15.107:0/3129158112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d74101980 msgr2=0x7f7d74101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:19.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.079+0000 7f7d7bf42640 1 --2- 172.21.15.107:0/3129158112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d74101980 0x7f7d74101e00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009860 tx=0x7f7d7002a340 comp rx=0 tx=0).stop 2024-05-09T08:33:19.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.079+0000 7f7d7bf42640 1 -- 172.21.15.107:0/3129158112 shutdown_connections 2024-05-09T08:33:19.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.079+0000 7f7d7bf42640 1 --2- 172.21.15.107:0/3129158112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d74102340 0x7f7d74108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:19.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.079+0000 7f7d7bf42640 1 --2- 172.21.15.107:0/3129158112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d74101980 0x7f7d74101e00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.079+0000 7f7d7bf42640 1 --2- 172.21.15.107:0/3129158112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d74100790 0x7f7d74100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.079+0000 7f7d7bf42640 1 -- 172.21.15.107:0/3129158112 >> 172.21.15.107:0/3129158112 conn(0x7f7d740fbd00 msgr2=0x7f7d740fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.079+0000 7f7d7bf42640 1 -- 172.21.15.107:0/3129158112 shutdown_connections 2024-05-09T08:33:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.079+0000 7f7d7bf42640 1 -- 172.21.15.107:0/3129158112 wait complete. 2024-05-09T08:33:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d7bf42640 1 Processor -- start 2024-05-09T08:33:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d7bf42640 1 -- start start 2024-05-09T08:33:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d7bf42640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d74100790 0x7f7d74197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d7bf42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d74101980 0x7f7d74197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d7bf42640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d74102340 0x7f7d7419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d7bf42640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7410c2a0 con 0x7f7d74101980 2024-05-09T08:33:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d7bf42640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7410c120 con 0x7f7d74100790 2024-05-09T08:33:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d7bf42640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7410c420 con 0x7f7d74102340 2024-05-09T08:33:19.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d794dd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d74101980 0x7f7d74197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d79cde640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d74100790 0x7f7d74197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d794dd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d74101980 0x7f7d74197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34422/0 (socket says 172.21.15.107:34422) 2024-05-09T08:33:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d794dd640 1 -- 172.21.15.107:0/2946603752 learned_addr learned my addr 172.21.15.107:0/2946603752 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d7a4df640 1 --2- 172.21.15.107:0/2946603752 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d74102340 0x7f7d7419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d794dd640 1 -- 172.21.15.107:0/2946603752 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d74102340 msgr2=0x7f7d7419eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d794dd640 1 --2- 172.21.15.107:0/2946603752 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d74102340 0x7f7d7419eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d794dd640 1 -- 172.21.15.107:0/2946603752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d74100790 msgr2=0x7f7d74197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d794dd640 1 --2- 172.21.15.107:0/2946603752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d74100790 0x7f7d74197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d794dd640 1 -- 172.21.15.107:0/2946603752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d7419f660 con 0x7f7d74101980 2024-05-09T08:33:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.080+0000 7f7d79cde640 1 --2- 172.21.15.107:0/2946603752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d74100790 0x7f7d74197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:33:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.081+0000 7f7d794dd640 1 --2- 172.21.15.107:0/2946603752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d74101980 0x7f7d74197f80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7d70005940 tx=0x7f7d70030e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.081+0000 7f7d6affd640 1 -- 172.21.15.107:0/2946603752 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d70037050 con 0x7f7d74101980 2024-05-09T08:33:19.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.081+0000 7f7d7bf42640 1 -- 172.21.15.107:0/2946603752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d7419f8f0 con 0x7f7d74101980 2024-05-09T08:33:19.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.081+0000 7f7d6affd640 1 -- 172.21.15.107:0/2946603752 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d70043070 con 0x7f7d74101980 2024-05-09T08:33:19.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.081+0000 7f7d6affd640 1 -- 172.21.15.107:0/2946603752 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d70003cd0 con 0x7f7d74101980 2024-05-09T08:33:19.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.081+0000 7f7d7bf42640 1 -- 172.21.15.107:0/2946603752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d7419fde0 con 0x7f7d74101980 2024-05-09T08:33:19.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.082+0000 7f7d7bf42640 1 -- 172.21.15.107:0/2946603752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d7419ffe0 con 0x7f7d74101980 2024-05-09T08:33:19.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.083+0000 7f7d6affd640 1 -- 172.21.15.107:0/2946603752 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f7d70004040 con 0x7f7d74101980 2024-05-09T08:33:19.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.084+0000 7f7d6affd640 1 --2- 172.21.15.107:0/2946603752 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7d600a2c80 0x7f7d600a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:19.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.084+0000 7f7d79cde640 1 -- 172.21.15.107:0/2946603752 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7d600a2c80 msgr2=0x7f7d600a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/846948416 2024-05-09T08:33:19.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.084+0000 7f7d6affd640 1 -- 172.21.15.107:0/2946603752 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f7d700efe40 con 0x7f7d74101980 2024-05-09T08:33:19.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.084+0000 7f7d79cde640 1 --2- 172.21.15.107:0/2946603752 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7d600a2c80 0x7f7d600a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:33:19.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.088+0000 7f7d6affd640 1 -- 172.21.15.107:0/2946603752 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7d7004c020 con 0x7f7d74101980 2024-05-09T08:33:19.120 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:18 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:18.868+0000 7f175fa23200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:33:19.121 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:19 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:18.999+0000 7f175fa23200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:33:19.121 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:19 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:19.094+0000 7f175fa23200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:33:19.236 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:18 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:18.894+0000 7fb30586e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:33:19.236 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:19 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:19.027+0000 7fb30586e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:33:19.236 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:19 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:19.123+0000 7fb30586e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:33:19.237 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:19 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:19.237+0000 7fb30586e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:33:19.279 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:18 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:18.957+0000 7fe2e5350200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T08:33:19.279 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:19 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:19.059+0000 7fe2e5350200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T08:33:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.285+0000 7f7d79cde640 1 -- 172.21.15.107:0/2946603752 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7d600a2c80 msgr2=0x7f7d600a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/846948416 2024-05-09T08:33:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.285+0000 7f7d79cde640 1 --2- 172.21.15.107:0/2946603752 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7d600a2c80 0x7f7d600a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:33:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.307+0000 7f7d7bf42640 1 -- 172.21.15.107:0/2946603752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f7d741a03d0 con 0x7f7d74101980 2024-05-09T08:33:19.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.359+0000 7f7d6affd640 1 -- 172.21.15.107:0/2946603752 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v25) ==== 118+0+0 (secure 0 0 0) 0x7f7d700bca20 con 0x7f7d74101980 2024-05-09T08:33:19.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.361+0000 7f7d7bf42640 1 -- 172.21.15.107:0/2946603752 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7d600a2c80 msgr2=0x7f7d600a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:33:19.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.361+0000 7f7d7bf42640 1 --2- 172.21.15.107:0/2946603752 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7d600a2c80 0x7f7d600a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:19.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.361+0000 7f7d7bf42640 1 -- 172.21.15.107:0/2946603752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d74101980 msgr2=0x7f7d74197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:19.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.361+0000 7f7d7bf42640 1 --2- 172.21.15.107:0/2946603752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d74101980 0x7f7d74197f80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7d70005940 tx=0x7f7d70030e00 comp rx=0 tx=0).stop 2024-05-09T08:33:19.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.361+0000 7f7d7bf42640 1 -- 172.21.15.107:0/2946603752 shutdown_connections 2024-05-09T08:33:19.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.361+0000 7f7d7bf42640 1 --2- 172.21.15.107:0/2946603752 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f7d600a2c80 0x7f7d600a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:19.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.361+0000 7f7d7bf42640 1 --2- 172.21.15.107:0/2946603752 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d74102340 0x7f7d7419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:19.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.361+0000 7f7d7bf42640 1 --2- 172.21.15.107:0/2946603752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d74101980 0x7f7d74197f80 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:19.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.361+0000 7f7d7bf42640 1 --2- 172.21.15.107:0/2946603752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d74100790 0x7f7d74197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:19.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.361+0000 7f7d7bf42640 1 -- 172.21.15.107:0/2946603752 >> 172.21.15.107:0/2946603752 conn(0x7f7d740fbd00 msgr2=0x7f7d740fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:19.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.361+0000 7f7d7bf42640 1 -- 172.21.15.107:0/2946603752 shutdown_connections 2024-05-09T08:33:19.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.361+0000 7f7d7bf42640 1 -- 172.21.15.107:0/2946603752 wait complete. 2024-05-09T08:33:19.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-05-09T08:33:19.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.477+0000 7f90c3161640 1 -- 172.21.15.107:0/2343605715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90bc100790 msgr2=0x7f90bc100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:19.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.477+0000 7f90c3161640 1 --2- 172.21.15.107:0/2343605715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90bc100790 0x7f90bc100bb0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f90b0009900 tx=0x7f90b002a410 comp rx=0 tx=0).stop 2024-05-09T08:33:19.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.477+0000 7f90c3161640 1 -- 172.21.15.107:0/2343605715 shutdown_connections 2024-05-09T08:33:19.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.477+0000 7f90c3161640 1 --2- 172.21.15.107:0/2343605715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90bc102340 0x7f90bc108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:19.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.477+0000 7f90c3161640 1 --2- 172.21.15.107:0/2343605715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90bc101980 0x7f90bc101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:19.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.477+0000 7f90c3161640 1 --2- 172.21.15.107:0/2343605715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90bc100790 0x7f90bc100bb0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:19.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.477+0000 7f90c3161640 1 -- 172.21.15.107:0/2343605715 >> 172.21.15.107:0/2343605715 conn(0x7f90bc0fbd00 msgr2=0x7f90bc0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:19.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.477+0000 7f90c3161640 1 -- 172.21.15.107:0/2343605715 shutdown_connections 2024-05-09T08:33:19.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.477+0000 7f90c3161640 1 -- 172.21.15.107:0/2343605715 wait complete. 2024-05-09T08:33:19.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.478+0000 7f90c3161640 1 Processor -- start 2024-05-09T08:33:19.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.478+0000 7f90c3161640 1 -- start start 2024-05-09T08:33:19.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.478+0000 7f90c3161640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90bc100790 0x7f90bc104720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:19.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.478+0000 7f90c3161640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90bc101980 0x7f90bc102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:19.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.478+0000 7f90c3161640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90bc102340 0x7f90bc1032a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:19.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.478+0000 7f90c3161640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f90bc10c300 con 0x7f90bc100790 2024-05-09T08:33:19.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.478+0000 7f90c3161640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f90bc10c180 con 0x7f90bc102340 2024-05-09T08:33:19.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.478+0000 7f90c3161640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f90bc10c480 con 0x7f90bc101980 2024-05-09T08:33:19.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.478+0000 7f90c0efd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90bc100790 0x7f90bc104720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:19.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.478+0000 7f90c0efd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90bc100790 0x7f90bc104720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34440/0 (socket says 172.21.15.107:34440) 2024-05-09T08:33:19.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.478+0000 7f90c0efd640 1 -- 172.21.15.107:0/244555983 learned_addr learned my addr 172.21.15.107:0/244555983 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:19.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.478+0000 7f90c16fe640 1 --2- 172.21.15.107:0/244555983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90bc102340 0x7f90bc1032a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:19.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.478+0000 7f90bbfff640 1 --2- 172.21.15.107:0/244555983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90bc101980 0x7f90bc102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:19.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.479+0000 7f90c0efd640 1 -- 172.21.15.107:0/244555983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90bc101980 msgr2=0x7f90bc102d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:19.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.479+0000 7f90c0efd640 1 --2- 172.21.15.107:0/244555983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90bc101980 0x7f90bc102d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:19.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.479+0000 7f90c0efd640 1 -- 172.21.15.107:0/244555983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90bc102340 msgr2=0x7f90bc1032a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:19.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.479+0000 7f90c0efd640 1 --2- 172.21.15.107:0/244555983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90bc102340 0x7f90bc1032a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:19.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.479+0000 7f90c0efd640 1 -- 172.21.15.107:0/244555983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90bc103b30 con 0x7f90bc100790 2024-05-09T08:33:19.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.479+0000 7f90c0efd640 1 --2- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90bc100790 0x7f90bc104720 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f90b00314e0 tx=0x7f90b0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:19.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.479+0000 7f90b9ffb640 1 -- 172.21.15.107:0/244555983 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b003f070 con 0x7f90bc100790 2024-05-09T08:33:19.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.479+0000 7f90b9ffb640 1 -- 172.21.15.107:0/244555983 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90b0031c40 con 0x7f90bc100790 2024-05-09T08:33:19.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.479+0000 7f90c3161640 1 -- 172.21.15.107:0/244555983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90bc1a1b50 con 0x7f90bc100790 2024-05-09T08:33:19.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.479+0000 7f90b9ffb640 1 -- 172.21.15.107:0/244555983 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b0035030 con 0x7f90bc100790 2024-05-09T08:33:19.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.479+0000 7f90c3161640 1 -- 172.21.15.107:0/244555983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90bc1a20f0 con 0x7f90bc100790 2024-05-09T08:33:19.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.480+0000 7f90c3161640 1 -- 172.21.15.107:0/244555983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90bc1a22c0 con 0x7f90bc100790 2024-05-09T08:33:19.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.482+0000 7f90b9ffb640 1 -- 172.21.15.107:0/244555983 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f90b0042070 con 0x7f90bc100790 2024-05-09T08:33:19.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.483+0000 7f90b9ffb640 1 --2- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f90ac0a2d00 0x7f90ac0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:19.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.483+0000 7f90bbfff640 1 -- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f90ac0a2d00 msgr2=0x7f90ac0a51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/846948416 2024-05-09T08:33:19.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.483+0000 7f90b9ffb640 1 -- 172.21.15.107:0/244555983 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f90b00f01b0 con 0x7f90bc100790 2024-05-09T08:33:19.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.483+0000 7f90bbfff640 1 --2- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f90ac0a2d00 0x7f90ac0a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T08:33:19.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.486+0000 7f90b9ffb640 1 -- 172.21.15.107:0/244555983 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f90b0031df0 con 0x7f90bc100790 2024-05-09T08:33:19.509 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:19 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2946603752' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-09T08:33:19.509 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:19 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:19.210+0000 7f175fa23200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:33:19.541 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:19 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2946603752' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-09T08:33:19.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.684+0000 7f90bbfff640 1 -- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f90ac0a2d00 msgr2=0x7f90ac0a51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/846948416 2024-05-09T08:33:19.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.684+0000 7f90bbfff640 1 --2- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f90ac0a2d00 0x7f90ac0a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T08:33:19.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:19 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2946603752' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-09T08:33:19.683 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:19 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:19.319+0000 7fe2e5350200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T08:33:19.684 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:19 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:19.461+0000 7fe2e5350200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T08:33:19.684 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:19 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:19.564+0000 7fe2e5350200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T08:33:19.684 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:19 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:19.684+0000 7fe2e5350200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T08:33:19.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:19.695+0000 7f90c3161640 1 -- 172.21.15.107:0/244555983 --> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f90bc0fde90 con 0x7f90ac0a2d00 2024-05-09T08:33:19.843 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:19 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:19.542+0000 7fb30586e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:33:19.870 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:19 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:19.509+0000 7f175fa23200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:33:20.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:20.085+0000 7f90bbfff640 1 -- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f90ac0a2d00 msgr2=0x7f90ac0a51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/846948416 2024-05-09T08:33:20.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:20.085+0000 7f90bbfff640 1 --2- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f90ac0a2d00 0x7f90ac0a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T08:33:20.278 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:19 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:19.988+0000 7fe2e5350200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T08:33:20.363 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:20 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:20.106+0000 7f175fa23200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:33:20.363 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:33:20 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y[35009]: 2024-05-09T08:33:20.320+0000 7f175fa23200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:33:20.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:20.378+0000 7f90b9ffb640 1 -- 172.21.15.107:0/244555983 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 20) ==== 137851+0+0 (secure 0 0 0) 0x7f90b00b8d90 con 0x7f90bc100790 2024-05-09T08:33:20.593 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:20 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:20.139+0000 7fb30586e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:33:20.593 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:33:20 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z[35475]: 2024-05-09T08:33:20.354+0000 7fb30586e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:33:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:20 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2946603752' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-09T08:33:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:20 smithi157 ceph-mon[33614]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:33:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:20 smithi157 ceph-mon[33614]: Standby manager daemon y restarted 2024-05-09T08:33:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:20 smithi157 ceph-mon[33614]: Standby manager daemon y started 2024-05-09T08:33:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:20 smithi157 ceph-mon[33614]: from='mgr.? 172.21.15.110:0/1689328530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:33:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:20 smithi157 ceph-mon[33614]: from='mgr.? 172.21.15.110:0/1689328530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:33:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:20 smithi157 ceph-mon[33614]: from='mgr.? 172.21.15.110:0/1689328530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:33:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:20 smithi157 ceph-mon[33614]: from='mgr.? 172.21.15.110:0/1689328530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:33:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:20 smithi157 ceph-mon[33614]: Standby manager daemon z restarted 2024-05-09T08:33:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:20 smithi157 ceph-mon[33614]: Standby manager daemon z started 2024-05-09T08:33:20.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:20 smithi157 ceph-mon[33614]: from='mgr.? 172.21.15.157:0/2288299582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:33:20.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:20 smithi157 ceph-mon[33614]: from='mgr.? 172.21.15.157:0/2288299582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:33:20.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:20 smithi157 ceph-mon[33614]: from='mgr.? 172.21.15.157:0/2288299582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:33:20.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:20 smithi157 ceph-mon[33614]: from='mgr.? 172.21.15.157:0/2288299582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:33:20.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:20 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2946603752' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-09T08:33:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:20 smithi110 ceph-mon[33622]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:33:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:20 smithi110 ceph-mon[33622]: Standby manager daemon y restarted 2024-05-09T08:33:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:20 smithi110 ceph-mon[33622]: Standby manager daemon y started 2024-05-09T08:33:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:20 smithi110 ceph-mon[33622]: from='mgr.? 172.21.15.110:0/1689328530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:33:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:20 smithi110 ceph-mon[33622]: from='mgr.? 172.21.15.110:0/1689328530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:33:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:20 smithi110 ceph-mon[33622]: from='mgr.? 172.21.15.110:0/1689328530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:33:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:20 smithi110 ceph-mon[33622]: from='mgr.? 172.21.15.110:0/1689328530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:33:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:20 smithi110 ceph-mon[33622]: Standby manager daemon z restarted 2024-05-09T08:33:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:20 smithi110 ceph-mon[33622]: Standby manager daemon z started 2024-05-09T08:33:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:20 smithi110 ceph-mon[33622]: from='mgr.? 172.21.15.157:0/2288299582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:33:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:20 smithi110 ceph-mon[33622]: from='mgr.? 172.21.15.157:0/2288299582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:33:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:20 smithi110 ceph-mon[33622]: from='mgr.? 172.21.15.157:0/2288299582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:33:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:20 smithi110 ceph-mon[33622]: from='mgr.? 172.21.15.157:0/2288299582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:33:20.779 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:20 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:20.605+0000 7fe2e5350200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T08:33:20.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:20 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2946603752' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-09T08:33:20.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:20 smithi107 ceph-mon[30122]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:33:20.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:20 smithi107 ceph-mon[30122]: Standby manager daemon y restarted 2024-05-09T08:33:20.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:20 smithi107 ceph-mon[30122]: Standby manager daemon y started 2024-05-09T08:33:20.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:20 smithi107 ceph-mon[30122]: from='mgr.? 172.21.15.110:0/1689328530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T08:33:20.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:20 smithi107 ceph-mon[30122]: from='mgr.? 172.21.15.110:0/1689328530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:33:20.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:20 smithi107 ceph-mon[30122]: from='mgr.? 172.21.15.110:0/1689328530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T08:33:20.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:20 smithi107 ceph-mon[30122]: from='mgr.? 172.21.15.110:0/1689328530' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:33:20.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:20 smithi107 ceph-mon[30122]: Standby manager daemon z restarted 2024-05-09T08:33:20.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:20 smithi107 ceph-mon[30122]: Standby manager daemon z started 2024-05-09T08:33:20.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:20 smithi107 ceph-mon[30122]: from='mgr.? 172.21.15.157:0/2288299582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T08:33:20.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:20 smithi107 ceph-mon[30122]: from='mgr.? 172.21.15.157:0/2288299582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T08:33:20.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:20 smithi107 ceph-mon[30122]: from='mgr.? 172.21.15.157:0/2288299582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T08:33:20.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:20 smithi107 ceph-mon[30122]: from='mgr.? 172.21.15.157:0/2288299582' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T08:33:20.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:20.842+0000 7f90b9ffb640 1 -- 172.21.15.107:0/244555983 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mgrmap(e 21) ==== 137518+0+0 (secure 0 0 0) 0x7f90b00ed990 con 0x7f90bc100790 2024-05-09T08:33:20.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:20.843+0000 7f90b9ffb640 1 -- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f90ac0a2d00 msgr2=0x7f90ac0a51c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:33:20.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:20.843+0000 7f90b9ffb640 1 --2- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:6800/846948416,v1:172.21.15.107:6801/846948416] conn(0x7f90ac0a2d00 0x7f90ac0a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:21.080 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:20 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:20.826+0000 7fe2e5350200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T08:33:21.081 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:21.068+0000 7fe243432700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:33:21.081 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:21.068+0000 7fe243432700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:33:21.081 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:21.068+0000 7fe243432700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:33:21.081 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:21.068+0000 7fe243432700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:33:21.081 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:21.068+0000 7fe243432700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:33:21.081 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:21.068+0000 7fe246438700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:33:21.081 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:21.068+0000 7fe246438700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:33:21.081 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:21.068+0000 7fe246438700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:33:21.081 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:21.068+0000 7fe246438700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:33:21.081 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:21.068+0000 7fe246438700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:33:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-05-09T08:33:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: Active manager daemon x restarted 2024-05-09T08:33:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: Activating manager daemon x 2024-05-09T08:33:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: osdmap e94: 12 total, 12 up, 12 in 2024-05-09T08:33:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: mgrmap e21: x(active, starting, since 0.0141184s), standbys: y, z 2024-05-09T08:33:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:33:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:33:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:33:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:33:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:33:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:33:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:33:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:33:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:33:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:33:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:33:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:33:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:33:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:33:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:33:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:33:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:33:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:33:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:33:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:33:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:33:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:33:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:33:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:33:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:33:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:33:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:33:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:33:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:33:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:33:21.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:33:21.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:33:21.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:33:21.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: Manager daemon x is now available 2024-05-09T08:33:21.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:33:21.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:33:21.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:21 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:33:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-05-09T08:33:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: Active manager daemon x restarted 2024-05-09T08:33:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: Activating manager daemon x 2024-05-09T08:33:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: osdmap e94: 12 total, 12 up, 12 in 2024-05-09T08:33:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: mgrmap e21: x(active, starting, since 0.0141184s), standbys: y, z 2024-05-09T08:33:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:33:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:33:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:33:21.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:33:21.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:33:21.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:33:21.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:33:21.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:33:21.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:33:21.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:33:21.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:33:21.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:33:21.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:33:21.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:33:21.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:33:21.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:33:21.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:33:21.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:33:21.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:33:21.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:33:21.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:33:21.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:33:21.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:33:21.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:33:21.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:33:21.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:33:21.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:33:21.846 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:33:21.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:33:21.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:33:21.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:33:21.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:33:21.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:33:21.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: Manager daemon x is now available 2024-05-09T08:33:21.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:33:21.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:33:21.847 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:21 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:33:21.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:21.846+0000 7f90b9ffb640 1 -- 172.21.15.107:0/244555983 <== mon.0 v2:172.21.15.107:3300/0 9 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f90b003b4f0 con 0x7f90bc100790 2024-05-09T08:33:21.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:21.846+0000 7f90b9ffb640 1 --2- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f90ac0356e0 0x7f90ac035ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:21.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:21.846+0000 7f90b9ffb640 1 -- 172.21.15.107:0/244555983 --> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f90bc0fde90 con 0x7f90ac0356e0 2024-05-09T08:33:21.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:21.848+0000 7f90bbfff640 1 --2- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f90ac0356e0 0x7f90ac035ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:21.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:21.851+0000 7f90bbfff640 1 --2- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f90ac0356e0 0x7f90ac035ac0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f90bc104490 tx=0x7f90a8005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:21.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:21.870+0000 7f90b9ffb640 1 -- 172.21.15.107:0/244555983 <== mgr.15483 v2:172.21.15.107:6800/165281804 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f90bc0fde90 con 0x7f90ac0356e0 2024-05-09T08:33:21.870 INFO:teuthology.orchestra.run.smithi107.stdout:{} 2024-05-09T08:33:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-05-09T08:33:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: Active manager daemon x restarted 2024-05-09T08:33:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: Activating manager daemon x 2024-05-09T08:33:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: osdmap e94: 12 total, 12 up, 12 in 2024-05-09T08:33:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: mgrmap e21: x(active, starting, since 0.0141184s), standbys: y, z 2024-05-09T08:33:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T08:33:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T08:33:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T08:33:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T08:33:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T08:33:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T08:33:21.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T08:33:21.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T08:33:21.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:33:21.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:33:21.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:33:21.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:33:21.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:33:21.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:33:21.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:33:21.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T08:33:21.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T08:33:21.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T08:33:21.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T08:33:21.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T08:33:21.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T08:33:21.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T08:33:21.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T08:33:21.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T08:33:21.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T08:33:21.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T08:33:21.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T08:33:21.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T08:33:21.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T08:33:21.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T08:33:21.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T08:33:21.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T08:33:21.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T08:33:21.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: Manager daemon x is now available 2024-05-09T08:33:21.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:33:21.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T08:33:21.874 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:21 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T08:33:21.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:21.873+0000 7f90c3161640 1 -- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f90ac0356e0 msgr2=0x7f90ac035ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:21.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:21.873+0000 7f90c3161640 1 --2- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f90ac0356e0 0x7f90ac035ac0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f90bc104490 tx=0x7f90a8005f90 comp rx=0 tx=0).stop 2024-05-09T08:33:21.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:21.873+0000 7f90c3161640 1 -- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90bc100790 msgr2=0x7f90bc104720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:21.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:21.873+0000 7f90c3161640 1 --2- 172.21.15.107:0/244555983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90bc100790 0x7f90bc104720 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f90b00314e0 tx=0x7f90b0003c10 comp rx=0 tx=0).stop 2024-05-09T08:33:21.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:21.873+0000 7f90c16fe640 1 -- 172.21.15.107:0/244555983 reap_dead start 2024-05-09T08:33:21.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:21.873+0000 7f90c3161640 1 -- 172.21.15.107:0/244555983 shutdown_connections 2024-05-09T08:33:21.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:21.873+0000 7f90c3161640 1 -- 172.21.15.107:0/244555983 >> 172.21.15.107:0/244555983 conn(0x7f90bc0fbd00 msgr2=0x7f90bc0fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:21.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:21.873+0000 7f90c3161640 1 -- 172.21.15.107:0/244555983 shutdown_connections 2024-05-09T08:33:21.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:21.873+0000 7f90c3161640 1 -- 172.21.15.107:0/244555983 wait complete. 2024-05-09T08:33:21.883 INFO:teuthology.run_tasks:Running task check-counter... 2024-05-09T08:33:21.895 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-05-09T08:33:21.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:33:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.026+0000 7f84c5ebd640 1 -- 172.21.15.107:0/4255030289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c0100a60 msgr2=0x7f84c0100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.026+0000 7f84c5ebd640 1 --2- 172.21.15.107:0/4255030289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c0100a60 0x7f84c0100ec0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f84a8009900 tx=0x7f84a802a410 comp rx=0 tx=0).stop 2024-05-09T08:33:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.026+0000 7f84c5ebd640 1 -- 172.21.15.107:0/4255030289 shutdown_connections 2024-05-09T08:33:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.026+0000 7f84c5ebd640 1 --2- 172.21.15.107:0/4255030289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84c0101400 0x7f84c0101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.026+0000 7f84c5ebd640 1 --2- 172.21.15.107:0/4255030289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c0100a60 0x7f84c0100ec0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.026+0000 7f84c5ebd640 1 --2- 172.21.15.107:0/4255030289 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84c0100100 0x7f84c0100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.026+0000 7f84c5ebd640 1 -- 172.21.15.107:0/4255030289 >> 172.21.15.107:0/4255030289 conn(0x7f84c00fbd20 msgr2=0x7f84c00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.027+0000 7f84c5ebd640 1 -- 172.21.15.107:0/4255030289 shutdown_connections 2024-05-09T08:33:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.027+0000 7f84c5ebd640 1 -- 172.21.15.107:0/4255030289 wait complete. 2024-05-09T08:33:22.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.027+0000 7f84c5ebd640 1 Processor -- start 2024-05-09T08:33:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.027+0000 7f84c5ebd640 1 -- start start 2024-05-09T08:33:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.027+0000 7f84c5ebd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c0100100 0x7f84c0199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.027+0000 7f84c5ebd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84c0100a60 0x7f84c019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.027+0000 7f84c5ebd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84c0101400 0x7f84c01a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.027+0000 7f84c5ebd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f84c010f580 con 0x7f84c0100100 2024-05-09T08:33:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.027+0000 7f84c5ebd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f84c010f400 con 0x7f84c0100a60 2024-05-09T08:33:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.027+0000 7f84c5ebd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f84c010f700 con 0x7f84c0101400 2024-05-09T08:33:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.028+0000 7f84bf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c0100100 0x7f84c0199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.028+0000 7f84bf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c0100100 0x7f84c0199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50230/0 (socket says 172.21.15.107:50230) 2024-05-09T08:33:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.028+0000 7f84bf7fe640 1 -- 172.21.15.107:0/815582495 learned_addr learned my addr 172.21.15.107:0/815582495 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.028+0000 7f84beffd640 1 --2- 172.21.15.107:0/815582495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84c0100a60 0x7f84c019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.028+0000 7f84bffff640 1 --2- 172.21.15.107:0/815582495 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84c0101400 0x7f84c01a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:22.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.028+0000 7f84bf7fe640 1 -- 172.21.15.107:0/815582495 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84c0101400 msgr2=0x7f84c01a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:22.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.028+0000 7f84bf7fe640 1 --2- 172.21.15.107:0/815582495 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84c0101400 0x7f84c01a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.028+0000 7f84bf7fe640 1 -- 172.21.15.107:0/815582495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84c0100a60 msgr2=0x7f84c019a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:22.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.028+0000 7f84bf7fe640 1 --2- 172.21.15.107:0/815582495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84c0100a60 0x7f84c019a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.028+0000 7f84bf7fe640 1 -- 172.21.15.107:0/815582495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c01a1870 con 0x7f84c0100100 2024-05-09T08:33:22.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.028+0000 7f84bf7fe640 1 --2- 172.21.15.107:0/815582495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c0100100 0x7f84c0199c60 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f84b000d950 tx=0x7f84b000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:22.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.028+0000 7f84bcff9640 1 -- 172.21.15.107:0/815582495 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b0017070 con 0x7f84c0100100 2024-05-09T08:33:22.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.028+0000 7f84bcff9640 1 -- 172.21.15.107:0/815582495 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84b0009c30 con 0x7f84c0100100 2024-05-09T08:33:22.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.028+0000 7f84c5ebd640 1 -- 172.21.15.107:0/815582495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84c01a1b60 con 0x7f84c0100100 2024-05-09T08:33:22.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.028+0000 7f84bcff9640 1 -- 172.21.15.107:0/815582495 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b000f620 con 0x7f84c0100100 2024-05-09T08:33:22.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.028+0000 7f84c5ebd640 1 -- 172.21.15.107:0/815582495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84c01a20b0 con 0x7f84c0100100 2024-05-09T08:33:22.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.029+0000 7f84c5ebd640 1 -- 172.21.15.107:0/815582495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84c010ae50 con 0x7f84c0100100 2024-05-09T08:33:22.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.030+0000 7f84bcff9640 1 -- 172.21.15.107:0/815582495 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f84b00109a0 con 0x7f84c0100100 2024-05-09T08:33:22.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.031+0000 7f84bcff9640 1 --2- 172.21.15.107:0/815582495 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f84ac0a2990 0x7f84ac0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:22.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.031+0000 7f84bcff9640 1 -- 172.21.15.107:0/815582495 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f84b00cec70 con 0x7f84c0100100 2024-05-09T08:33:22.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.031+0000 7f84beffd640 1 --2- 172.21.15.107:0/815582495 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f84ac0a2990 0x7f84ac0a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:22.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.031+0000 7f84beffd640 1 --2- 172.21.15.107:0/815582495 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f84ac0a2990 0x7f84ac0a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f84a8006010 tx=0x7f84a8000bc0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:22.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.035+0000 7f84bcff9640 1 -- 172.21.15.107:0/815582495 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f84b0009de0 con 0x7f84c0100100 2024-05-09T08:33:22.176 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:21.857+0000 7fe241c2f700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:33:22.176 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:21.857+0000 7fe241c2f700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:33:22.176 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:21.857+0000 7fe241c2f700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:33:22.176 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:21.857+0000 7fe241c2f700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:33:22.176 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:33:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:33:21.857+0000 7fe241c2f700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:33:22.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.227+0000 7f84c5ebd640 1 -- 172.21.15.107:0/815582495 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84c002d1c0 con 0x7f84c0100100 2024-05-09T08:33:22.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.228+0000 7f84bcff9640 1 -- 172.21.15.107:0/815582495 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25143 (secure 0 0 0) 0x7f84b009b3f0 con 0x7f84c0100100 2024-05-09T08:33:22.228 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:22.228 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":94,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:33:20.828605+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/2923766500":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/2417648229":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1642826815":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/2689615560":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/846948416":"2024-05-10T08:33:20.828572+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/846948416":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:33:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.231+0000 7f84c5ebd640 1 -- 172.21.15.107:0/815582495 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f84ac0a2990 msgr2=0x7f84ac0a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.231+0000 7f84c5ebd640 1 --2- 172.21.15.107:0/815582495 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f84ac0a2990 0x7f84ac0a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f84a8006010 tx=0x7f84a8000bc0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.231+0000 7f84c5ebd640 1 -- 172.21.15.107:0/815582495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c0100100 msgr2=0x7f84c0199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.231+0000 7f84c5ebd640 1 --2- 172.21.15.107:0/815582495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c0100100 0x7f84c0199c60 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f84b000d950 tx=0x7f84b000de10 comp rx=0 tx=0).stop 2024-05-09T08:33:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.231+0000 7f84c5ebd640 1 -- 172.21.15.107:0/815582495 shutdown_connections 2024-05-09T08:33:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.231+0000 7f84c5ebd640 1 --2- 172.21.15.107:0/815582495 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f84ac0a2990 0x7f84ac0a4e50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.231+0000 7f84c5ebd640 1 --2- 172.21.15.107:0/815582495 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84c0101400 0x7f84c01a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.231+0000 7f84c5ebd640 1 --2- 172.21.15.107:0/815582495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84c0100a60 0x7f84c019a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.231+0000 7f84c5ebd640 1 --2- 172.21.15.107:0/815582495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c0100100 0x7f84c0199c60 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.231+0000 7f84c5ebd640 1 -- 172.21.15.107:0/815582495 >> 172.21.15.107:0/815582495 conn(0x7f84c00fbd20 msgr2=0x7f84c00fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.231+0000 7f84c5ebd640 1 -- 172.21.15.107:0/815582495 shutdown_connections 2024-05-09T08:33:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.231+0000 7f84c5ebd640 1 -- 172.21.15.107:0/815582495 wait complete. 2024-05-09T08:33:22.241 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:26:21.698338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:30:58.268633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:30:59.345579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:31:03.065991+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:33:22.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:33:22.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.348+0000 7fdfb137d640 1 -- 172.21.15.107:0/2969932557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfac100780 msgr2=0x7fdfac100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:22.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.348+0000 7fdfb137d640 1 --2- 172.21.15.107:0/2969932557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfac100780 0x7fdfac100ba0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c009900 tx=0x7fdf9c02a410 comp rx=0 tx=0).stop 2024-05-09T08:33:22.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.348+0000 7fdfb137d640 1 -- 172.21.15.107:0/2969932557 shutdown_connections 2024-05-09T08:33:22.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.348+0000 7fdfb137d640 1 --2- 172.21.15.107:0/2969932557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfac102330 0x7fdfac108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.348+0000 7fdfb137d640 1 --2- 172.21.15.107:0/2969932557 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfac101970 0x7fdfac101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.348+0000 7fdfb137d640 1 --2- 172.21.15.107:0/2969932557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfac100780 0x7fdfac100ba0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.348+0000 7fdfb137d640 1 -- 172.21.15.107:0/2969932557 >> 172.21.15.107:0/2969932557 conn(0x7fdfac0fbcf0 msgr2=0x7fdfac0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:22.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.348+0000 7fdfb137d640 1 -- 172.21.15.107:0/2969932557 shutdown_connections 2024-05-09T08:33:22.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.348+0000 7fdfb137d640 1 -- 172.21.15.107:0/2969932557 wait complete. 2024-05-09T08:33:22.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfb137d640 1 Processor -- start 2024-05-09T08:33:22.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfb137d640 1 -- start start 2024-05-09T08:33:22.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfb137d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfac100780 0x7fdfac06a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:22.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfb137d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfac101970 0x7fdfac19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:22.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfb137d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfac102330 0x7fdfac19ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:22.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfb137d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdfac10c2a0 con 0x7fdfac101970 2024-05-09T08:33:22.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfb137d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdfac10c120 con 0x7fdfac102330 2024-05-09T08:33:22.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfb137d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdfac10c420 con 0x7fdfac100780 2024-05-09T08:33:22.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfa3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfac101970 0x7fdfac19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:22.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfa3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfac101970 0x7fdfac19c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50248/0 (socket says 172.21.15.107:50248) 2024-05-09T08:33:22.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfa3fff640 1 -- 172.21.15.107:0/730717028 learned_addr learned my addr 172.21.15.107:0/730717028 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:22.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfaaffd640 1 --2- 172.21.15.107:0/730717028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfac100780 0x7fdfac06a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:22.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfab7fe640 1 --2- 172.21.15.107:0/730717028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfac102330 0x7fdfac19ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfa3fff640 1 -- 172.21.15.107:0/730717028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfac100780 msgr2=0x7fdfac06a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfa3fff640 1 --2- 172.21.15.107:0/730717028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfac100780 0x7fdfac06a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfa3fff640 1 -- 172.21.15.107:0/730717028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfac102330 msgr2=0x7fdfac19ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfa3fff640 1 --2- 172.21.15.107:0/730717028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfac102330 0x7fdfac19ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.349+0000 7fdfa3fff640 1 -- 172.21.15.107:0/730717028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfac19f700 con 0x7fdfac101970 2024-05-09T08:33:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.350+0000 7fdfaaffd640 1 --2- 172.21.15.107:0/730717028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfac100780 0x7fdfac06a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:33:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.350+0000 7fdfa3fff640 1 --2- 172.21.15.107:0/730717028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfac101970 0x7fdfac19c800 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400e9f0 tx=0x7fdf9400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.350+0000 7fdfa8ff9640 1 -- 172.21.15.107:0/730717028 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9400cd80 con 0x7fdfac101970 2024-05-09T08:33:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.350+0000 7fdfa8ff9640 1 -- 172.21.15.107:0/730717028 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf940054d0 con 0x7fdfac101970 2024-05-09T08:33:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.350+0000 7fdfb137d640 1 -- 172.21.15.107:0/730717028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfac19f9f0 con 0x7fdfac101970 2024-05-09T08:33:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.350+0000 7fdfa8ff9640 1 -- 172.21.15.107:0/730717028 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf94010490 con 0x7fdfac101970 2024-05-09T08:33:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.350+0000 7fdfb137d640 1 -- 172.21.15.107:0/730717028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfac19fef0 con 0x7fdfac101970 2024-05-09T08:33:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.351+0000 7fdfb137d640 1 -- 172.21.15.107:0/730717028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfac107b70 con 0x7fdfac101970 2024-05-09T08:33:22.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.352+0000 7fdfa8ff9640 1 -- 172.21.15.107:0/730717028 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7fdf94005020 con 0x7fdfac101970 2024-05-09T08:33:22.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.353+0000 7fdfa8ff9640 1 --2- 172.21.15.107:0/730717028 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf980a2990 0x7fdf980a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:22.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.353+0000 7fdfaaffd640 1 --2- 172.21.15.107:0/730717028 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf980a2990 0x7fdf980a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:22.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.353+0000 7fdfa8ff9640 1 -- 172.21.15.107:0/730717028 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fdf940cf420 con 0x7fdfac101970 2024-05-09T08:33:22.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.353+0000 7fdfaaffd640 1 --2- 172.21.15.107:0/730717028 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf980a2990 0x7fdf980a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c004770 tx=0x7fdf9c000bc0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.357+0000 7fdfa8ff9640 1 -- 172.21.15.107:0/730717028 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdf9409c1f0 con 0x7fdfac101970 2024-05-09T08:33:22.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:22 smithi107 ceph-mon[30122]: [09/May/2024:08:33:21] ENGINE Bus STARTING 2024-05-09T08:33:22.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:22 smithi107 ceph-mon[30122]: [09/May/2024:08:33:21] ENGINE Serving on https://172.21.15.107:7150 2024-05-09T08:33:22.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:22 smithi107 ceph-mon[30122]: [09/May/2024:08:33:21] ENGINE Serving on http://172.21.15.107:8765 2024-05-09T08:33:22.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:22 smithi107 ceph-mon[30122]: [09/May/2024:08:33:21] ENGINE Bus STARTED 2024-05-09T08:33:22.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:22 smithi107 ceph-mon[30122]: mgrmap e22: x(active, since 1.01761s), standbys: y, z 2024-05-09T08:33:22.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:22 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/815582495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:22.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.539+0000 7fdfb137d640 1 -- 172.21.15.107:0/730717028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdfac062430 con 0x7fdfac101970 2024-05-09T08:33:22.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.540+0000 7fdfa8ff9640 1 -- 172.21.15.107:0/730717028 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fdf9409baf0 con 0x7fdfac101970 2024-05-09T08:33:22.539 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-05-09T08:33:22.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.542+0000 7fdfb137d640 1 -- 172.21.15.107:0/730717028 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf980a2990 msgr2=0x7fdf980a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:22.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.542+0000 7fdfb137d640 1 --2- 172.21.15.107:0/730717028 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf980a2990 0x7fdf980a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c004770 tx=0x7fdf9c000bc0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.542+0000 7fdfb137d640 1 -- 172.21.15.107:0/730717028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfac101970 msgr2=0x7fdfac19c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.542+0000 7fdfb137d640 1 --2- 172.21.15.107:0/730717028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfac101970 0x7fdfac19c800 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400e9f0 tx=0x7fdf9400edf0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.542+0000 7fdfb137d640 1 -- 172.21.15.107:0/730717028 shutdown_connections 2024-05-09T08:33:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.542+0000 7fdfb137d640 1 --2- 172.21.15.107:0/730717028 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf980a2990 0x7fdf980a4e50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.542+0000 7fdfb137d640 1 --2- 172.21.15.107:0/730717028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfac102330 0x7fdfac19ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.542+0000 7fdfb137d640 1 --2- 172.21.15.107:0/730717028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfac101970 0x7fdfac19c800 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.542+0000 7fdfb137d640 1 --2- 172.21.15.107:0/730717028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfac100780 0x7fdfac06a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.542+0000 7fdfb137d640 1 -- 172.21.15.107:0/730717028 >> 172.21.15.107:0/730717028 conn(0x7fdfac0fbcf0 msgr2=0x7fdfac104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.542+0000 7fdfb137d640 1 -- 172.21.15.107:0/730717028 shutdown_connections 2024-05-09T08:33:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.542+0000 7fdfb137d640 1 -- 172.21.15.107:0/730717028 wait complete. 2024-05-09T08:33:22.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:33:22.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.664+0000 7f18b5f1b640 1 -- 172.21.15.107:0/3650158078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b0103580 msgr2=0x7f18b0109e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:22.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.664+0000 7f18b5f1b640 1 --2- 172.21.15.107:0/3650158078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b0103580 0x7f18b0109e10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009930 tx=0x7f18a802a420 comp rx=0 tx=0).stop 2024-05-09T08:33:22.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.664+0000 7f18b5f1b640 1 -- 172.21.15.107:0/3650158078 shutdown_connections 2024-05-09T08:33:22.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.664+0000 7f18b5f1b640 1 --2- 172.21.15.107:0/3650158078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b0103580 0x7f18b0109e10 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.664+0000 7f18b5f1b640 1 --2- 172.21.15.107:0/3650158078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b0102bc0 0x7f18b0103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.664+0000 7f18b5f1b640 1 --2- 172.21.15.107:0/3650158078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18b00ffb50 0x7f18b00fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.664+0000 7f18b5f1b640 1 -- 172.21.15.107:0/3650158078 >> 172.21.15.107:0/3650158078 conn(0x7f18b00fb460 msgr2=0x7f18b00fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:22.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.664+0000 7f18b5f1b640 1 -- 172.21.15.107:0/3650158078 shutdown_connections 2024-05-09T08:33:22.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.664+0000 7f18b5f1b640 1 -- 172.21.15.107:0/3650158078 wait complete. 2024-05-09T08:33:22.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.665+0000 7f18b5f1b640 1 Processor -- start 2024-05-09T08:33:22.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.665+0000 7f18b5f1b640 1 -- start start 2024-05-09T08:33:22.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.665+0000 7f18b5f1b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b00ffb50 0x7f18b0197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:22.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.665+0000 7f18b5f1b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18b0102bc0 0x7f18b0198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:22.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.665+0000 7f18b5f1b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b0103580 0x7f18b019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:22.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.665+0000 7f18b5f1b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f18b010d3b0 con 0x7f18b0103580 2024-05-09T08:33:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.665+0000 7f18b5f1b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f18b010d230 con 0x7f18b0102bc0 2024-05-09T08:33:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.665+0000 7f18b5f1b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f18b010d530 con 0x7f18b00ffb50 2024-05-09T08:33:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.665+0000 7f18affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b0103580 0x7f18b019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.665+0000 7f18af7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b00ffb50 0x7f18b0197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.665+0000 7f18affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b0103580 0x7f18b019f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50266/0 (socket says 172.21.15.107:50266) 2024-05-09T08:33:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.665+0000 7f18affff640 1 -- 172.21.15.107:0/2019851079 learned_addr learned my addr 172.21.15.107:0/2019851079 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.666+0000 7f18aeffd640 1 --2- 172.21.15.107:0/2019851079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18b0102bc0 0x7f18b0198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.666+0000 7f18af7fe640 1 -- 172.21.15.107:0/2019851079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18b0102bc0 msgr2=0x7f18b0198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.666+0000 7f18af7fe640 1 --2- 172.21.15.107:0/2019851079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18b0102bc0 0x7f18b0198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.666+0000 7f18af7fe640 1 -- 172.21.15.107:0/2019851079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b0103580 msgr2=0x7f18b019f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.666+0000 7f18af7fe640 1 --2- 172.21.15.107:0/2019851079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b0103580 0x7f18b019f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.666+0000 7f18af7fe640 1 -- 172.21.15.107:0/2019851079 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b019f750 con 0x7f18b00ffb50 2024-05-09T08:33:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.666+0000 7f18affff640 1 --2- 172.21.15.107:0/2019851079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b0103580 0x7f18b019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:33:22.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.666+0000 7f18af7fe640 1 --2- 172.21.15.107:0/2019851079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b00ffb50 0x7f18b0197b30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f18a0005fd0 tx=0x7f18a000ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:22.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.666+0000 7f18acff9640 1 -- 172.21.15.107:0/2019851079 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a0003970 con 0x7f18b00ffb50 2024-05-09T08:33:22.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.666+0000 7f18b5f1b640 1 -- 172.21.15.107:0/2019851079 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b019fa40 con 0x7f18b00ffb50 2024-05-09T08:33:22.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.666+0000 7f18acff9640 1 -- 172.21.15.107:0/2019851079 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18a0004030 con 0x7f18b00ffb50 2024-05-09T08:33:22.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.666+0000 7f18acff9640 1 -- 172.21.15.107:0/2019851079 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a0010720 con 0x7f18b00ffb50 2024-05-09T08:33:22.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.666+0000 7f18b5f1b640 1 -- 172.21.15.107:0/2019851079 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b019ffe0 con 0x7f18b00ffb50 2024-05-09T08:33:22.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.667+0000 7f18b5f1b640 1 -- 172.21.15.107:0/2019851079 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18b0108de0 con 0x7f18b00ffb50 2024-05-09T08:33:22.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.669+0000 7f18acff9640 1 -- 172.21.15.107:0/2019851079 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f18a00041e0 con 0x7f18b00ffb50 2024-05-09T08:33:22.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.670+0000 7f18acff9640 1 --2- 172.21.15.107:0/2019851079 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f18980a2990 0x7f18980a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:22.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.670+0000 7f18acff9640 1 -- 172.21.15.107:0/2019851079 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f18a00cf520 con 0x7f18b00ffb50 2024-05-09T08:33:22.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.670+0000 7f18aeffd640 1 --2- 172.21.15.107:0/2019851079 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f18980a2990 0x7f18980a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:22.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.670+0000 7f18aeffd640 1 --2- 172.21.15.107:0/2019851079 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f18980a2990 0x7f18980a4e50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18b0199040 tx=0x7f18a4005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:22.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.675+0000 7f18acff9640 1 -- 172.21.15.107:0/2019851079 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f18a009c2f0 con 0x7f18b00ffb50 2024-05-09T08:33:22.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:22 smithi157 ceph-mon[33614]: [09/May/2024:08:33:21] ENGINE Bus STARTING 2024-05-09T08:33:22.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:22 smithi157 ceph-mon[33614]: [09/May/2024:08:33:21] ENGINE Serving on https://172.21.15.107:7150 2024-05-09T08:33:22.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:22 smithi157 ceph-mon[33614]: [09/May/2024:08:33:21] ENGINE Serving on http://172.21.15.107:8765 2024-05-09T08:33:22.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:22 smithi157 ceph-mon[33614]: [09/May/2024:08:33:21] ENGINE Bus STARTED 2024-05-09T08:33:22.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:22 smithi157 ceph-mon[33614]: mgrmap e22: x(active, since 1.01761s), standbys: y, z 2024-05-09T08:33:22.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:22 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/815582495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:22.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:22 smithi110 ceph-mon[33622]: [09/May/2024:08:33:21] ENGINE Bus STARTING 2024-05-09T08:33:22.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:22 smithi110 ceph-mon[33622]: [09/May/2024:08:33:21] ENGINE Serving on https://172.21.15.107:7150 2024-05-09T08:33:22.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:22 smithi110 ceph-mon[33622]: [09/May/2024:08:33:21] ENGINE Serving on http://172.21.15.107:8765 2024-05-09T08:33:22.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:22 smithi110 ceph-mon[33622]: [09/May/2024:08:33:21] ENGINE Bus STARTED 2024-05-09T08:33:22.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:22 smithi110 ceph-mon[33622]: mgrmap e22: x(active, since 1.01761s), standbys: y, z 2024-05-09T08:33:22.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:22 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/815582495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:22.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.872+0000 7f18b5f1b640 1 -- 172.21.15.107:0/2019851079 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f18b0062430 con 0x7f18b00ffb50 2024-05-09T08:33:22.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.873+0000 7f18acff9640 1 -- 172.21.15.107:0/2019851079 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f18a009bbf0 con 0x7f18b00ffb50 2024-05-09T08:33:22.872 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:33:22.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.875+0000 7f18b5f1b640 1 -- 172.21.15.107:0/2019851079 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f18980a2990 msgr2=0x7f18980a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:22.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.875+0000 7f18b5f1b640 1 --2- 172.21.15.107:0/2019851079 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f18980a2990 0x7f18980a4e50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18b0199040 tx=0x7f18a4005f90 comp rx=0 tx=0).stop 2024-05-09T08:33:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.875+0000 7f18b5f1b640 1 -- 172.21.15.107:0/2019851079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b00ffb50 msgr2=0x7f18b0197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.875+0000 7f18b5f1b640 1 --2- 172.21.15.107:0/2019851079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b00ffb50 0x7f18b0197b30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f18a0005fd0 tx=0x7f18a000ef90 comp rx=0 tx=0).stop 2024-05-09T08:33:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.875+0000 7f18b5f1b640 1 -- 172.21.15.107:0/2019851079 shutdown_connections 2024-05-09T08:33:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.875+0000 7f18b5f1b640 1 --2- 172.21.15.107:0/2019851079 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f18980a2990 0x7f18980a4e50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.875+0000 7f18b5f1b640 1 --2- 172.21.15.107:0/2019851079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b0103580 0x7f18b019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.875+0000 7f18b5f1b640 1 --2- 172.21.15.107:0/2019851079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18b0102bc0 0x7f18b0198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.875+0000 7f18b5f1b640 1 --2- 172.21.15.107:0/2019851079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b00ffb50 0x7f18b0197b30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.875+0000 7f18b5f1b640 1 -- 172.21.15.107:0/2019851079 >> 172.21.15.107:0/2019851079 conn(0x7f18b00fb460 msgr2=0x7f18b0105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.875+0000 7f18b5f1b640 1 -- 172.21.15.107:0/2019851079 shutdown_connections 2024-05-09T08:33:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:22.875+0000 7f18b5f1b640 1 -- 172.21.15.107:0/2019851079 wait complete. 2024-05-09T08:33:22.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:33:23.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.005+0000 7f4f68de7640 1 -- 172.21.15.107:0/3963209501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f64101d80 msgr2=0x7f4f641021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.005+0000 7f4f68de7640 1 --2- 172.21.15.107:0/3963209501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f64101d80 0x7f4f641021a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4f54009900 tx=0x7f4f5402a410 comp rx=0 tx=0).stop 2024-05-09T08:33:23.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.005+0000 7f4f68de7640 1 -- 172.21.15.107:0/3963209501 shutdown_connections 2024-05-09T08:33:23.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.005+0000 7f4f68de7640 1 --2- 172.21.15.107:0/3963209501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f64101390 0x7f4f64108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.005+0000 7f4f68de7640 1 --2- 172.21.15.107:0/3963209501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f641009d0 0x7f4f64100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.005+0000 7f4f68de7640 1 --2- 172.21.15.107:0/3963209501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f64101d80 0x7f4f641021a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.005+0000 7f4f68de7640 1 -- 172.21.15.107:0/3963209501 >> 172.21.15.107:0/3963209501 conn(0x7f4f640fb460 msgr2=0x7f4f640fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:23.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.005+0000 7f4f68de7640 1 -- 172.21.15.107:0/3963209501 shutdown_connections 2024-05-09T08:33:23.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.005+0000 7f4f68de7640 1 -- 172.21.15.107:0/3963209501 wait complete. 2024-05-09T08:33:23.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.005+0000 7f4f68de7640 1 Processor -- start 2024-05-09T08:33:23.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.005+0000 7f4f68de7640 1 -- start start 2024-05-09T08:33:23.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f68de7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f641009d0 0x7f4f64196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:23.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f68de7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f64101390 0x7f4f64196f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:23.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f68de7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f64101d80 0x7f4f6419dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:23.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f68de7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6410c2b0 con 0x7f4f64101390 2024-05-09T08:33:23.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f68de7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6410c130 con 0x7f4f64101d80 2024-05-09T08:33:23.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f68de7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6410c430 con 0x7f4f641009d0 2024-05-09T08:33:23.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f61d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f64101390 0x7f4f64196f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:23.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f61d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f64101390 0x7f4f64196f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50294/0 (socket says 172.21.15.107:50294) 2024-05-09T08:33:23.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f61d9b640 1 -- 172.21.15.107:0/3426185680 learned_addr learned my addr 172.21.15.107:0/3426185680 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:23.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f62d9d640 1 --2- 172.21.15.107:0/3426185680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f64101d80 0x7f4f6419dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:23.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f6259c640 1 --2- 172.21.15.107:0/3426185680 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f641009d0 0x7f4f64196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:23.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f61d9b640 1 -- 172.21.15.107:0/3426185680 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f641009d0 msgr2=0x7f4f64196a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f61d9b640 1 --2- 172.21.15.107:0/3426185680 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f641009d0 0x7f4f64196a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f61d9b640 1 -- 172.21.15.107:0/3426185680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f64101d80 msgr2=0x7f4f6419dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f61d9b640 1 --2- 172.21.15.107:0/3426185680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f64101d80 0x7f4f6419dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f61d9b640 1 -- 172.21.15.107:0/3426185680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f6419e660 con 0x7f4f64101390 2024-05-09T08:33:23.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f62d9d640 1 --2- 172.21.15.107:0/3426185680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f64101d80 0x7f4f6419dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:33:23.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f61d9b640 1 --2- 172.21.15.107:0/3426185680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f64101390 0x7f4f64196f80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4f5800e9c0 tx=0x7f4f5800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:23.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f537fe640 1 -- 172.21.15.107:0/3426185680 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5800cd80 con 0x7f4f64101390 2024-05-09T08:33:23.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f68de7640 1 -- 172.21.15.107:0/3426185680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f6419e950 con 0x7f4f64101390 2024-05-09T08:33:23.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.006+0000 7f4f537fe640 1 -- 172.21.15.107:0/3426185680 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f580035f0 con 0x7f4f64101390 2024-05-09T08:33:23.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.007+0000 7f4f537fe640 1 -- 172.21.15.107:0/3426185680 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f58010690 con 0x7f4f64101390 2024-05-09T08:33:23.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.007+0000 7f4f68de7640 1 -- 172.21.15.107:0/3426185680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f6419ee50 con 0x7f4f64101390 2024-05-09T08:33:23.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.008+0000 7f4f68de7640 1 -- 172.21.15.107:0/3426185680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f64107cd0 con 0x7f4f64101390 2024-05-09T08:33:23.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.009+0000 7f4f537fe640 1 -- 172.21.15.107:0/3426185680 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f4f580037a0 con 0x7f4f64101390 2024-05-09T08:33:23.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.009+0000 7f4f537fe640 1 --2- 172.21.15.107:0/3426185680 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4f4c0a2990 0x7f4f4c0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:23.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.010+0000 7f4f6259c640 1 --2- 172.21.15.107:0/3426185680 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4f4c0a2990 0x7f4f4c0a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:23.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.010+0000 7f4f537fe640 1 -- 172.21.15.107:0/3426185680 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4f580cf5f0 con 0x7f4f64101390 2024-05-09T08:33:23.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.010+0000 7f4f6259c640 1 --2- 172.21.15.107:0/3426185680 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4f4c0a2990 0x7f4f4c0a4e50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4f54004770 tx=0x7f4f54000bc0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:23.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.014+0000 7f4f537fe640 1 -- 172.21.15.107:0/3426185680 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4f5809c3c0 con 0x7f4f64101390 2024-05-09T08:33:23.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.206+0000 7f4f68de7640 1 -- 172.21.15.107:0/3426185680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4f64062430 con 0x7f4f64101390 2024-05-09T08:33:23.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.206+0000 7f4f537fe640 1 -- 172.21.15.107:0/3426185680 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f4f5809bcc0 con 0x7f4f64101390 2024-05-09T08:33:23.206 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-05-09T08:33:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.208+0000 7f4f68de7640 1 -- 172.21.15.107:0/3426185680 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4f4c0a2990 msgr2=0x7f4f4c0a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.208+0000 7f4f68de7640 1 --2- 172.21.15.107:0/3426185680 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4f4c0a2990 0x7f4f4c0a4e50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4f54004770 tx=0x7f4f54000bc0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.208+0000 7f4f68de7640 1 -- 172.21.15.107:0/3426185680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f64101390 msgr2=0x7f4f64196f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.208+0000 7f4f68de7640 1 --2- 172.21.15.107:0/3426185680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f64101390 0x7f4f64196f80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4f5800e9c0 tx=0x7f4f5800ee80 comp rx=0 tx=0).stop 2024-05-09T08:33:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.209+0000 7f4f68de7640 1 -- 172.21.15.107:0/3426185680 shutdown_connections 2024-05-09T08:33:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.209+0000 7f4f68de7640 1 --2- 172.21.15.107:0/3426185680 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4f4c0a2990 0x7f4f4c0a4e50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.209+0000 7f4f68de7640 1 --2- 172.21.15.107:0/3426185680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f64101d80 0x7f4f6419dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.209+0000 7f4f68de7640 1 --2- 172.21.15.107:0/3426185680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f64101390 0x7f4f64196f80 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.209+0000 7f4f68de7640 1 --2- 172.21.15.107:0/3426185680 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f641009d0 0x7f4f64196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.209+0000 7f4f68de7640 1 -- 172.21.15.107:0/3426185680 >> 172.21.15.107:0/3426185680 conn(0x7f4f640fb460 msgr2=0x7f4f64104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.209+0000 7f4f68de7640 1 -- 172.21.15.107:0/3426185680 shutdown_connections 2024-05-09T08:33:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.209+0000 7f4f68de7640 1 -- 172.21.15.107:0/3426185680 wait complete. 2024-05-09T08:33:23.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:33:23.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.325+0000 7fe6bf1b7640 1 -- 172.21.15.107:0/777409929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6b8100770 msgr2=0x7fe6b8100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.325+0000 7fe6bf1b7640 1 --2- 172.21.15.107:0/777409929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6b8100770 0x7fe6b8100b90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009900 tx=0x7fe6ac02a410 comp rx=0 tx=0).stop 2024-05-09T08:33:23.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.325+0000 7fe6bf1b7640 1 -- 172.21.15.107:0/777409929 shutdown_connections 2024-05-09T08:33:23.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.325+0000 7fe6bf1b7640 1 --2- 172.21.15.107:0/777409929 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6b8102320 0x7fe6b8108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.325+0000 7fe6bf1b7640 1 --2- 172.21.15.107:0/777409929 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6b8101960 0x7fe6b8101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.325+0000 7fe6bf1b7640 1 --2- 172.21.15.107:0/777409929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6b8100770 0x7fe6b8100b90 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.325+0000 7fe6bf1b7640 1 -- 172.21.15.107:0/777409929 >> 172.21.15.107:0/777409929 conn(0x7fe6b80fbce0 msgr2=0x7fe6b80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:23.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.325+0000 7fe6bf1b7640 1 -- 172.21.15.107:0/777409929 shutdown_connections 2024-05-09T08:33:23.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.325+0000 7fe6bf1b7640 1 -- 172.21.15.107:0/777409929 wait complete. 2024-05-09T08:33:23.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bf1b7640 1 Processor -- start 2024-05-09T08:33:23.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bf1b7640 1 -- start start 2024-05-09T08:33:23.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bf1b7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6b8100770 0x7fe6b8069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:23.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bf1b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6b8101960 0x7fe6b819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:23.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bf1b7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6b8102320 0x7fe6b819eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:23.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bf1b7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b810c2e0 con 0x7fe6b8101960 2024-05-09T08:33:23.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bf1b7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b810c160 con 0x7fe6b8100770 2024-05-09T08:33:23.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bf1b7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b810c460 con 0x7fe6b8102320 2024-05-09T08:33:23.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6b7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6b8101960 0x7fe6b819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:23.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bcf53640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6b8100770 0x7fe6b8069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:23.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bcf53640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6b8100770 0x7fe6b8069530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:47966/0 (socket says 172.21.15.107:47966) 2024-05-09T08:33:23.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bcf53640 1 -- 172.21.15.107:0/646797162 learned_addr learned my addr 172.21.15.107:0/646797162 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:23.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6b7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6b8101960 0x7fe6b819c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50308/0 (socket says 172.21.15.107:50308) 2024-05-09T08:33:23.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bd754640 1 --2- 172.21.15.107:0/646797162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6b8102320 0x7fe6b819eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:23.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bcf53640 1 -- 172.21.15.107:0/646797162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6b8102320 msgr2=0x7fe6b819eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bcf53640 1 --2- 172.21.15.107:0/646797162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6b8102320 0x7fe6b819eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bcf53640 1 -- 172.21.15.107:0/646797162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6b8101960 msgr2=0x7fe6b819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bcf53640 1 --2- 172.21.15.107:0/646797162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6b8101960 0x7fe6b819c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6bcf53640 1 -- 172.21.15.107:0/646797162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b819f7c0 con 0x7fe6b8100770 2024-05-09T08:33:23.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.326+0000 7fe6b7fff640 1 --2- 172.21.15.107:0/646797162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6b8101960 0x7fe6b819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:33:23.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.327+0000 7fe6bcf53640 1 --2- 172.21.15.107:0/646797162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6b8100770 0x7fe6b8069530 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac0314e0 tx=0x7fe6ac003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:23.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.327+0000 7fe6b5ffb640 1 -- 172.21.15.107:0/646797162 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac03f070 con 0x7fe6b8100770 2024-05-09T08:33:23.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.327+0000 7fe6b5ffb640 1 -- 172.21.15.107:0/646797162 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6ac031c40 con 0x7fe6b8100770 2024-05-09T08:33:23.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.327+0000 7fe6bf1b7640 1 -- 172.21.15.107:0/646797162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b819f9f0 con 0x7fe6b8100770 2024-05-09T08:33:23.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.327+0000 7fe6b5ffb640 1 -- 172.21.15.107:0/646797162 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac035030 con 0x7fe6b8100770 2024-05-09T08:33:23.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.327+0000 7fe6bf1b7640 1 -- 172.21.15.107:0/646797162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6b819fee0 con 0x7fe6b8100770 2024-05-09T08:33:23.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.328+0000 7fe6bf1b7640 1 -- 172.21.15.107:0/646797162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6b8107c00 con 0x7fe6b8100770 2024-05-09T08:33:23.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.329+0000 7fe6b5ffb640 1 -- 172.21.15.107:0/646797162 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7fe6ac042070 con 0x7fe6b8100770 2024-05-09T08:33:23.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.330+0000 7fe6b5ffb640 1 --2- 172.21.15.107:0/646797162 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6a80a2990 0x7fe6a80a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:23.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.330+0000 7fe6b7fff640 1 --2- 172.21.15.107:0/646797162 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6a80a2990 0x7fe6a80a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:23.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.330+0000 7fe6b5ffb640 1 -- 172.21.15.107:0/646797162 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe6ac0efd70 con 0x7fe6b8100770 2024-05-09T08:33:23.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.330+0000 7fe6b7fff640 1 --2- 172.21.15.107:0/646797162 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6a80a2990 0x7fe6a80a4e50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80692a0 tx=0x7fe6a4005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:23.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.333+0000 7fe6b5ffb640 1 -- 172.21.15.107:0/646797162 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe6ac031df0 con 0x7fe6b8100770 2024-05-09T08:33:23.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.389+0000 7fe6b5ffb640 1 -- 172.21.15.107:0/646797162 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mgrmap(e 23) ==== 138005+0+0 (secure 0 0 0) 0x7fe6ac0b8b20 con 0x7fe6b8100770 2024-05-09T08:33:23.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.522+0000 7fe6bf1b7640 1 -- 172.21.15.107:0/646797162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe6b802d410 con 0x7fe6b8100770 2024-05-09T08:33:23.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.522+0000 7fe6b5ffb640 1 -- 172.21.15.107:0/646797162 <== mon.2 v2:172.21.15.110:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fe6ac0aac30 con 0x7fe6b8100770 2024-05-09T08:33:23.522 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:33:23.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.525+0000 7fe6bf1b7640 1 -- 172.21.15.107:0/646797162 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6a80a2990 msgr2=0x7fe6a80a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.525+0000 7fe6bf1b7640 1 --2- 172.21.15.107:0/646797162 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6a80a2990 0x7fe6a80a4e50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80692a0 tx=0x7fe6a4005f90 comp rx=0 tx=0).stop 2024-05-09T08:33:23.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.525+0000 7fe6bf1b7640 1 -- 172.21.15.107:0/646797162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6b8100770 msgr2=0x7fe6b8069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.525+0000 7fe6bf1b7640 1 --2- 172.21.15.107:0/646797162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6b8100770 0x7fe6b8069530 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac0314e0 tx=0x7fe6ac003c10 comp rx=0 tx=0).stop 2024-05-09T08:33:23.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.525+0000 7fe6bf1b7640 1 -- 172.21.15.107:0/646797162 shutdown_connections 2024-05-09T08:33:23.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.525+0000 7fe6bf1b7640 1 --2- 172.21.15.107:0/646797162 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6a80a2990 0x7fe6a80a4e50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.525+0000 7fe6bf1b7640 1 --2- 172.21.15.107:0/646797162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6b8102320 0x7fe6b819eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.525+0000 7fe6bf1b7640 1 --2- 172.21.15.107:0/646797162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6b8101960 0x7fe6b819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.525+0000 7fe6bf1b7640 1 --2- 172.21.15.107:0/646797162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6b8100770 0x7fe6b8069530 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.525+0000 7fe6bf1b7640 1 -- 172.21.15.107:0/646797162 >> 172.21.15.107:0/646797162 conn(0x7fe6b80fbce0 msgr2=0x7fe6b8104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:23.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.525+0000 7fe6bf1b7640 1 -- 172.21.15.107:0/646797162 shutdown_connections 2024-05-09T08:33:23.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.525+0000 7fe6bf1b7640 1 -- 172.21.15.107:0/646797162 wait complete. 2024-05-09T08:33:23.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:23 smithi107 ceph-mon[30122]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:23.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:23 smithi107 ceph-mon[30122]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:33:23.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:23 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/730717028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:33:23.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:23 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2019851079' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:33:23.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:23 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:23 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:23 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:23 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:23 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:23 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:23 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3426185680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:33:23.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:33:23.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.643+0000 7fec04142640 1 -- 172.21.15.107:0/4258299899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febfc102320 msgr2=0x7febfc108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.643+0000 7fec04142640 1 --2- 172.21.15.107:0/4258299899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febfc102320 0x7febfc108bb0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7febf4009ca0 tx=0x7febf402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.643+0000 7fec04142640 1 -- 172.21.15.107:0/4258299899 shutdown_connections 2024-05-09T08:33:23.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.643+0000 7fec04142640 1 --2- 172.21.15.107:0/4258299899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febfc102320 0x7febfc108bb0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.643+0000 7fec04142640 1 --2- 172.21.15.107:0/4258299899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7febfc101960 0x7febfc101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.643+0000 7fec04142640 1 --2- 172.21.15.107:0/4258299899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febfc100770 0x7febfc100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.643+0000 7fec04142640 1 -- 172.21.15.107:0/4258299899 >> 172.21.15.107:0/4258299899 conn(0x7febfc0fbce0 msgr2=0x7febfc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:23.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.643+0000 7fec04142640 1 -- 172.21.15.107:0/4258299899 shutdown_connections 2024-05-09T08:33:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.643+0000 7fec04142640 1 -- 172.21.15.107:0/4258299899 wait complete. 2024-05-09T08:33:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.643+0000 7fec04142640 1 Processor -- start 2024-05-09T08:33:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec04142640 1 -- start start 2024-05-09T08:33:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec04142640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febfc100770 0x7febfc196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec04142640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7febfc101960 0x7febfc196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec04142640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febfc102320 0x7febfc19df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec04142640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7febfc10c2a0 con 0x7febfc100770 2024-05-09T08:33:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec04142640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7febfc10c120 con 0x7febfc102320 2024-05-09T08:33:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec04142640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7febfc10c420 con 0x7febfc101960 2024-05-09T08:33:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec01ede640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febfc100770 0x7febfc196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec01ede640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febfc100770 0x7febfc196960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50322/0 (socket says 172.21.15.107:50322) 2024-05-09T08:33:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec01ede640 1 -- 172.21.15.107:0/1867328022 learned_addr learned my addr 172.21.15.107:0/1867328022 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec016dd640 1 --2- 172.21.15.107:0/1867328022 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7febfc101960 0x7febfc196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec026df640 1 --2- 172.21.15.107:0/1867328022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febfc102320 0x7febfc19df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec01ede640 1 -- 172.21.15.107:0/1867328022 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7febfc101960 msgr2=0x7febfc196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec01ede640 1 --2- 172.21.15.107:0/1867328022 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7febfc101960 0x7febfc196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec01ede640 1 -- 172.21.15.107:0/1867328022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febfc102320 msgr2=0x7febfc19df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec01ede640 1 --2- 172.21.15.107:0/1867328022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febfc102320 0x7febfc19df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec01ede640 1 -- 172.21.15.107:0/1867328022 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febfc19e580 con 0x7febfc100770 2024-05-09T08:33:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7fec01ede640 1 --2- 172.21.15.107:0/1867328022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febfc100770 0x7febfc196960 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7febf800e9c0 tx=0x7febf800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.644+0000 7febf2ffd640 1 -- 172.21.15.107:0/1867328022 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf800cd80 con 0x7febfc100770 2024-05-09T08:33:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.645+0000 7febf2ffd640 1 -- 172.21.15.107:0/1867328022 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febf80035f0 con 0x7febfc100770 2024-05-09T08:33:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.645+0000 7fec04142640 1 -- 172.21.15.107:0/1867328022 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febfc19e870 con 0x7febfc100770 2024-05-09T08:33:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.645+0000 7febf2ffd640 1 -- 172.21.15.107:0/1867328022 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8010690 con 0x7febfc100770 2024-05-09T08:33:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.645+0000 7fec04142640 1 -- 172.21.15.107:0/1867328022 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febfc19edc0 con 0x7febfc100770 2024-05-09T08:33:23.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.646+0000 7fec04142640 1 -- 172.21.15.107:0/1867328022 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febfc107a90 con 0x7febfc100770 2024-05-09T08:33:23.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.647+0000 7febf2ffd640 1 -- 172.21.15.107:0/1867328022 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) ==== 138005+0+0 (secure 0 0 0) 0x7febf80037a0 con 0x7febfc100770 2024-05-09T08:33:23.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.648+0000 7febf2ffd640 1 --2- 172.21.15.107:0/1867328022 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7febe80a2d90 0x7febe80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:23.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.648+0000 7fec016dd640 1 --2- 172.21.15.107:0/1867328022 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7febe80a2d90 0x7febe80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:23.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.648+0000 7febf2ffd640 1 -- 172.21.15.107:0/1867328022 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7febf80cfcf0 con 0x7febfc100770 2024-05-09T08:33:23.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.648+0000 7fec016dd640 1 --2- 172.21.15.107:0/1867328022 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7febe80a2d90 0x7febe80a5250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7febfc197e70 tx=0x7febec005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:23.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.652+0000 7febf2ffd640 1 -- 172.21.15.107:0/1867328022 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7febf80d4070 con 0x7febfc100770 2024-05-09T08:33:23.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.840+0000 7fec04142640 1 -- 172.21.15.107:0/1867328022 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7febfc02d420 con 0x7febfc100770 2024-05-09T08:33:23.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.841+0000 7febf2ffd640 1 -- 172.21.15.107:0/1867328022 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25143 (secure 0 0 0) 0x7febf809c440 con 0x7febfc100770 2024-05-09T08:33:23.841 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:23.841 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":94,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:33:20.828605+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/2923766500":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/2417648229":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1642826815":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/2689615560":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/846948416":"2024-05-10T08:33:20.828572+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/846948416":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:33:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.844+0000 7fec04142640 1 -- 172.21.15.107:0/1867328022 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7febe80a2d90 msgr2=0x7febe80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.844+0000 7fec04142640 1 --2- 172.21.15.107:0/1867328022 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7febe80a2d90 0x7febe80a5250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7febfc197e70 tx=0x7febec005f90 comp rx=0 tx=0).stop 2024-05-09T08:33:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.844+0000 7fec04142640 1 -- 172.21.15.107:0/1867328022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febfc100770 msgr2=0x7febfc196960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.844+0000 7fec04142640 1 --2- 172.21.15.107:0/1867328022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febfc100770 0x7febfc196960 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7febf800e9c0 tx=0x7febf800ee80 comp rx=0 tx=0).stop 2024-05-09T08:33:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.844+0000 7fec04142640 1 -- 172.21.15.107:0/1867328022 shutdown_connections 2024-05-09T08:33:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.844+0000 7fec04142640 1 --2- 172.21.15.107:0/1867328022 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7febe80a2d90 0x7febe80a5250 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.844+0000 7fec04142640 1 --2- 172.21.15.107:0/1867328022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febfc102320 0x7febfc19df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.844+0000 7fec04142640 1 --2- 172.21.15.107:0/1867328022 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7febfc101960 0x7febfc196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.844+0000 7fec04142640 1 --2- 172.21.15.107:0/1867328022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febfc100770 0x7febfc196960 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.844+0000 7fec04142640 1 -- 172.21.15.107:0/1867328022 >> 172.21.15.107:0/1867328022 conn(0x7febfc0fbce0 msgr2=0x7febfc104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:23.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.844+0000 7fec04142640 1 -- 172.21.15.107:0/1867328022 shutdown_connections 2024-05-09T08:33:23.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.844+0000 7fec04142640 1 -- 172.21.15.107:0/1867328022 wait complete. 2024-05-09T08:33:23.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:23 smithi157 ceph-mon[33614]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:23.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:23 smithi157 ceph-mon[33614]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:33:23.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:23 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/730717028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:33:23.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:23 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2019851079' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:33:23.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:23 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:23 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:23 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:23 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:23 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:23 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:23 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3426185680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:33:23.855 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:26:21.698338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:30:58.268633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:30:59.345579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:31:03.065991+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:33:23.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:33:23.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:23 smithi110 ceph-mon[33622]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:23.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:23 smithi110 ceph-mon[33622]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:33:23.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:23 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/730717028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:33:23.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:23 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2019851079' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:33:23.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:23 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:23 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:23 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:23 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:23 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:23 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:23.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:23 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3426185680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:33:23.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.962+0000 7fed356f5640 1 -- 172.21.15.107:0/2194397202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed30102a70 msgr2=0x7fed30102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.962+0000 7fed356f5640 1 --2- 172.21.15.107:0/2194397202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed30102a70 0x7fed30102ef0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fed18009900 tx=0x7fed1802a410 comp rx=0 tx=0).stop 2024-05-09T08:33:23.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.962+0000 7fed356f5640 1 -- 172.21.15.107:0/2194397202 shutdown_connections 2024-05-09T08:33:23.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.962+0000 7fed356f5640 1 --2- 172.21.15.107:0/2194397202 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed30103430 0x7fed30109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.962+0000 7fed356f5640 1 --2- 172.21.15.107:0/2194397202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed30102a70 0x7fed30102ef0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.962+0000 7fed356f5640 1 --2- 172.21.15.107:0/2194397202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed30101880 0x7fed30101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.962+0000 7fed356f5640 1 -- 172.21.15.107:0/2194397202 >> 172.21.15.107:0/2194397202 conn(0x7fed300fce10 msgr2=0x7fed300ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:23.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.962+0000 7fed356f5640 1 -- 172.21.15.107:0/2194397202 shutdown_connections 2024-05-09T08:33:23.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.962+0000 7fed356f5640 1 -- 172.21.15.107:0/2194397202 wait complete. 2024-05-09T08:33:23.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.963+0000 7fed356f5640 1 Processor -- start 2024-05-09T08:33:23.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.963+0000 7fed356f5640 1 -- start start 2024-05-09T08:33:23.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.963+0000 7fed356f5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed30101880 0x7fed300755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:23.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.963+0000 7fed356f5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed30102a70 0x7fed30073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:23.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.963+0000 7fed356f5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed30103430 0x7fed30074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:23.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.963+0000 7fed356f5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fed3010d400 con 0x7fed30101880 2024-05-09T08:33:23.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.963+0000 7fed356f5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fed3010d280 con 0x7fed30102a70 2024-05-09T08:33:23.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.963+0000 7fed356f5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fed3010d580 con 0x7fed30103430 2024-05-09T08:33:23.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.963+0000 7fed2effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed30101880 0x7fed300755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:23.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.963+0000 7fed2effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed30101880 0x7fed300755e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50344/0 (socket says 172.21.15.107:50344) 2024-05-09T08:33:23.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.963+0000 7fed2effd640 1 -- 172.21.15.107:0/708409265 learned_addr learned my addr 172.21.15.107:0/708409265 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:23.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.964+0000 7fed2f7fe640 1 --2- 172.21.15.107:0/708409265 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed30103430 0x7fed30074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:23.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.964+0000 7fed2e7fc640 1 --2- 172.21.15.107:0/708409265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed30102a70 0x7fed30073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:23.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.964+0000 7fed2effd640 1 -- 172.21.15.107:0/708409265 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed30103430 msgr2=0x7fed30074160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.964+0000 7fed2effd640 1 --2- 172.21.15.107:0/708409265 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed30103430 0x7fed30074160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.964+0000 7fed2effd640 1 -- 172.21.15.107:0/708409265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed30102a70 msgr2=0x7fed30073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:23.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.964+0000 7fed2effd640 1 --2- 172.21.15.107:0/708409265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed30102a70 0x7fed30073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:23.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.964+0000 7fed2effd640 1 -- 172.21.15.107:0/708409265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed300749f0 con 0x7fed30101880 2024-05-09T08:33:23.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.964+0000 7fed2effd640 1 --2- 172.21.15.107:0/708409265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed30101880 0x7fed300755e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fed2000e9c0 tx=0x7fed2000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:23.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.964+0000 7fed27fff640 1 -- 172.21.15.107:0/708409265 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed2000cd80 con 0x7fed30101880 2024-05-09T08:33:23.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.964+0000 7fed356f5640 1 -- 172.21.15.107:0/708409265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed301a1bf0 con 0x7fed30101880 2024-05-09T08:33:23.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.964+0000 7fed27fff640 1 -- 172.21.15.107:0/708409265 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed200035f0 con 0x7fed30101880 2024-05-09T08:33:23.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.964+0000 7fed27fff640 1 -- 172.21.15.107:0/708409265 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed20010690 con 0x7fed30101880 2024-05-09T08:33:23.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.964+0000 7fed356f5640 1 -- 172.21.15.107:0/708409265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed301a20c0 con 0x7fed30101880 2024-05-09T08:33:23.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.965+0000 7fed356f5640 1 -- 172.21.15.107:0/708409265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed30067dc0 con 0x7fed30101880 2024-05-09T08:33:23.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.966+0000 7fed27fff640 1 -- 172.21.15.107:0/708409265 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) ==== 138005+0+0 (secure 0 0 0) 0x7fed200037a0 con 0x7fed30101880 2024-05-09T08:33:23.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.967+0000 7fed27fff640 1 --2- 172.21.15.107:0/708409265 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fed1c0a2d90 0x7fed1c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:23.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.967+0000 7fed27fff640 1 -- 172.21.15.107:0/708409265 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fed200cfec0 con 0x7fed30101880 2024-05-09T08:33:23.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.967+0000 7fed2e7fc640 1 --2- 172.21.15.107:0/708409265 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fed1c0a2d90 0x7fed1c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:23.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.968+0000 7fed2e7fc640 1 --2- 172.21.15.107:0/708409265 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fed1c0a2d90 0x7fed1c0a5250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fed180314e0 tx=0x7fed18005a20 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:23.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:23.972+0000 7fed27fff640 1 -- 172.21.15.107:0/708409265 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fed2009c610 con 0x7fed30101880 2024-05-09T08:33:24.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.166+0000 7fed356f5640 1 -- 172.21.15.107:0/708409265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fed3002d240 con 0x7fed30101880 2024-05-09T08:33:24.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.166+0000 7fed27fff640 1 -- 172.21.15.107:0/708409265 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fed2009bf10 con 0x7fed30101880 2024-05-09T08:33:24.166 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-05-09T08:33:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.168+0000 7fed356f5640 1 -- 172.21.15.107:0/708409265 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fed1c0a2d90 msgr2=0x7fed1c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.168+0000 7fed356f5640 1 --2- 172.21.15.107:0/708409265 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fed1c0a2d90 0x7fed1c0a5250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fed180314e0 tx=0x7fed18005a20 comp rx=0 tx=0).stop 2024-05-09T08:33:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.168+0000 7fed356f5640 1 -- 172.21.15.107:0/708409265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed30101880 msgr2=0x7fed300755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.168+0000 7fed356f5640 1 --2- 172.21.15.107:0/708409265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed30101880 0x7fed300755e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fed2000e9c0 tx=0x7fed2000ee80 comp rx=0 tx=0).stop 2024-05-09T08:33:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.168+0000 7fed356f5640 1 -- 172.21.15.107:0/708409265 shutdown_connections 2024-05-09T08:33:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.168+0000 7fed356f5640 1 --2- 172.21.15.107:0/708409265 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fed1c0a2d90 0x7fed1c0a5250 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.168+0000 7fed356f5640 1 --2- 172.21.15.107:0/708409265 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed30103430 0x7fed30074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.168+0000 7fed356f5640 1 --2- 172.21.15.107:0/708409265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed30102a70 0x7fed30073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.168+0000 7fed356f5640 1 --2- 172.21.15.107:0/708409265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed30101880 0x7fed300755e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.168+0000 7fed356f5640 1 -- 172.21.15.107:0/708409265 >> 172.21.15.107:0/708409265 conn(0x7fed300fce10 msgr2=0x7fed30105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.169+0000 7fed356f5640 1 -- 172.21.15.107:0/708409265 shutdown_connections 2024-05-09T08:33:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.169+0000 7fed356f5640 1 -- 172.21.15.107:0/708409265 wait complete. 2024-05-09T08:33:24.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:33:24.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.286+0000 7fba3239d640 1 -- 172.21.15.107:0/2540328351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba2c102320 msgr2=0x7fba2c108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:24.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.286+0000 7fba3239d640 1 --2- 172.21.15.107:0/2540328351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba2c102320 0x7fba2c108bb0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fba24009ca0 tx=0x7fba2402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.286+0000 7fba3239d640 1 -- 172.21.15.107:0/2540328351 shutdown_connections 2024-05-09T08:33:24.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.286+0000 7fba3239d640 1 --2- 172.21.15.107:0/2540328351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba2c102320 0x7fba2c108bb0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.286+0000 7fba3239d640 1 --2- 172.21.15.107:0/2540328351 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba2c101960 0x7fba2c101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.286+0000 7fba3239d640 1 --2- 172.21.15.107:0/2540328351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba2c100770 0x7fba2c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.286+0000 7fba3239d640 1 -- 172.21.15.107:0/2540328351 >> 172.21.15.107:0/2540328351 conn(0x7fba2c0fbce0 msgr2=0x7fba2c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:24.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.286+0000 7fba3239d640 1 -- 172.21.15.107:0/2540328351 shutdown_connections 2024-05-09T08:33:24.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.287+0000 7fba3239d640 1 -- 172.21.15.107:0/2540328351 wait complete. 2024-05-09T08:33:24.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.287+0000 7fba3239d640 1 Processor -- start 2024-05-09T08:33:24.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.287+0000 7fba3239d640 1 -- start start 2024-05-09T08:33:24.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.287+0000 7fba3239d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba2c100770 0x7fba2c069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:24.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.287+0000 7fba3239d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba2c101960 0x7fba2c19c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:24.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.287+0000 7fba3239d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba2c102320 0x7fba2c19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.287+0000 7fba3239d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fba2c10c2a0 con 0x7fba2c100770 2024-05-09T08:33:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.287+0000 7fba3239d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fba2c10c120 con 0x7fba2c102320 2024-05-09T08:33:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.287+0000 7fba3239d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fba2c10c420 con 0x7fba2c101960 2024-05-09T08:33:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.288+0000 7fba2bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba2c100770 0x7fba2c069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.288+0000 7fba2b7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba2c101960 0x7fba2c19c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.288+0000 7fba2bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba2c100770 0x7fba2c069550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50370/0 (socket says 172.21.15.107:50370) 2024-05-09T08:33:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.288+0000 7fba2bfff640 1 -- 172.21.15.107:0/3923181866 learned_addr learned my addr 172.21.15.107:0/3923181866 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.288+0000 7fba2bfff640 1 -- 172.21.15.107:0/3923181866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba2c101960 msgr2=0x7fba2c19c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.288+0000 7fba2bfff640 1 --2- 172.21.15.107:0/3923181866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba2c101960 0x7fba2c19c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.288+0000 7fba2bfff640 1 -- 172.21.15.107:0/3923181866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba2c102320 msgr2=0x7fba2c19ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.288+0000 7fba2bfff640 1 --2- 172.21.15.107:0/3923181866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba2c102320 0x7fba2c19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.288+0000 7fba2bfff640 1 -- 172.21.15.107:0/3923181866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba2c19f6d0 con 0x7fba2c100770 2024-05-09T08:33:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.288+0000 7fba2b7fe640 1 --2- 172.21.15.107:0/3923181866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba2c101960 0x7fba2c19c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:33:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.288+0000 7fba2bfff640 1 --2- 172.21.15.107:0/3923181866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba2c100770 0x7fba2c069550 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fba1c00e9c0 tx=0x7fba1c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:24.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.288+0000 7fba297fa640 1 -- 172.21.15.107:0/3923181866 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1c00cde0 con 0x7fba2c100770 2024-05-09T08:33:24.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.288+0000 7fba3239d640 1 -- 172.21.15.107:0/3923181866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba2c19f9c0 con 0x7fba2c100770 2024-05-09T08:33:24.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.288+0000 7fba297fa640 1 -- 172.21.15.107:0/3923181866 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba1c0108e0 con 0x7fba2c100770 2024-05-09T08:33:24.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.288+0000 7fba297fa640 1 -- 172.21.15.107:0/3923181866 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1c0188e0 con 0x7fba2c100770 2024-05-09T08:33:24.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.288+0000 7fba3239d640 1 -- 172.21.15.107:0/3923181866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba2c19ff60 con 0x7fba2c100770 2024-05-09T08:33:24.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.289+0000 7fba3239d640 1 -- 172.21.15.107:0/3923181866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba2c107b60 con 0x7fba2c100770 2024-05-09T08:33:24.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.291+0000 7fba297fa640 1 -- 172.21.15.107:0/3923181866 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) ==== 138005+0+0 (secure 0 0 0) 0x7fba1c01f020 con 0x7fba2c100770 2024-05-09T08:33:24.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.292+0000 7fba297fa640 1 --2- 172.21.15.107:0/3923181866 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fba140a2d90 0x7fba140a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:24.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.292+0000 7fba2b7fe640 1 --2- 172.21.15.107:0/3923181866 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fba140a2d90 0x7fba140a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:24.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.292+0000 7fba297fa640 1 -- 172.21.15.107:0/3923181866 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fba1c0d0190 con 0x7fba2c100770 2024-05-09T08:33:24.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.292+0000 7fba2b7fe640 1 --2- 172.21.15.107:0/3923181866 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fba140a2d90 0x7fba140a5250 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0692c0 tx=0x7fba20005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:24.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.296+0000 7fba297fa640 1 -- 172.21.15.107:0/3923181866 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fba1c09c8c0 con 0x7fba2c100770 2024-05-09T08:33:24.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.484+0000 7fba3239d640 1 -- 172.21.15.107:0/3923181866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fba2c02d380 con 0x7fba2c100770 2024-05-09T08:33:24.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.485+0000 7fba297fa640 1 -- 172.21.15.107:0/3923181866 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fba1c09c1c0 con 0x7fba2c100770 2024-05-09T08:33:24.484 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:33:24.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.487+0000 7fba3239d640 1 -- 172.21.15.107:0/3923181866 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fba140a2d90 msgr2=0x7fba140a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:24.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.487+0000 7fba3239d640 1 --2- 172.21.15.107:0/3923181866 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fba140a2d90 0x7fba140a5250 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0692c0 tx=0x7fba20005f90 comp rx=0 tx=0).stop 2024-05-09T08:33:24.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.487+0000 7fba3239d640 1 -- 172.21.15.107:0/3923181866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba2c100770 msgr2=0x7fba2c069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:24.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.487+0000 7fba3239d640 1 --2- 172.21.15.107:0/3923181866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba2c100770 0x7fba2c069550 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fba1c00e9c0 tx=0x7fba1c00ee80 comp rx=0 tx=0).stop 2024-05-09T08:33:24.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.487+0000 7fba3239d640 1 -- 172.21.15.107:0/3923181866 shutdown_connections 2024-05-09T08:33:24.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.487+0000 7fba3239d640 1 --2- 172.21.15.107:0/3923181866 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fba140a2d90 0x7fba140a5250 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.487+0000 7fba3239d640 1 --2- 172.21.15.107:0/3923181866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba2c102320 0x7fba2c19ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.487+0000 7fba3239d640 1 --2- 172.21.15.107:0/3923181866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba2c101960 0x7fba2c19c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.487+0000 7fba3239d640 1 --2- 172.21.15.107:0/3923181866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba2c100770 0x7fba2c069550 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.487+0000 7fba3239d640 1 -- 172.21.15.107:0/3923181866 >> 172.21.15.107:0/3923181866 conn(0x7fba2c0fbce0 msgr2=0x7fba2c104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:24.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.487+0000 7fba3239d640 1 -- 172.21.15.107:0/3923181866 shutdown_connections 2024-05-09T08:33:24.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.487+0000 7fba3239d640 1 -- 172.21.15.107:0/3923181866 wait complete. 2024-05-09T08:33:24.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:33:24.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-05-09T08:33:24.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/646797162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:33:24.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1867328022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:24.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.499 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.499 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.499 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:33:24.499 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:33:24.499 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:24 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/708409265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:33:24.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:33:24.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.617+0000 7f0770631640 1 -- 172.21.15.107:0/2268757586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0768103430 msgr2=0x7f0768109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:24.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.617+0000 7f0770631640 1 --2- 172.21.15.107:0/2268757586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0768103430 0x7f0768109cc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0760009ca0 tx=0x7f076002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.617+0000 7f0770631640 1 -- 172.21.15.107:0/2268757586 shutdown_connections 2024-05-09T08:33:24.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.617+0000 7f0770631640 1 --2- 172.21.15.107:0/2268757586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0768103430 0x7f0768109cc0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.617+0000 7f0770631640 1 --2- 172.21.15.107:0/2268757586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0768102a70 0x7f0768102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.617+0000 7f0770631640 1 --2- 172.21.15.107:0/2268757586 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0768101880 0x7f0768101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.617+0000 7f0770631640 1 -- 172.21.15.107:0/2268757586 >> 172.21.15.107:0/2268757586 conn(0x7f07680fce10 msgr2=0x7f07680ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:24.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.617+0000 7f0770631640 1 -- 172.21.15.107:0/2268757586 shutdown_connections 2024-05-09T08:33:24.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.617+0000 7f0770631640 1 -- 172.21.15.107:0/2268757586 wait complete. 2024-05-09T08:33:24.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.618+0000 7f0770631640 1 Processor -- start 2024-05-09T08:33:24.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.618+0000 7f0770631640 1 -- start start 2024-05-09T08:33:24.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.618+0000 7f0770631640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0768101880 0x7f07680755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:24.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.618+0000 7f0770631640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0768102a70 0x7f0768073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.618+0000 7f0770631640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0768103430 0x7f0768074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.618+0000 7f0770631640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f076810d3b0 con 0x7f0768101880 2024-05-09T08:33:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.618+0000 7f0770631640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f076810d230 con 0x7f0768103430 2024-05-09T08:33:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.618+0000 7f0770631640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f076810d530 con 0x7f0768102a70 2024-05-09T08:33:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.618+0000 7f076e3cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0768101880 0x7f07680755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.618+0000 7f076e3cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0768101880 0x7f07680755e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50394/0 (socket says 172.21.15.107:50394) 2024-05-09T08:33:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.618+0000 7f076e3cd640 1 -- 172.21.15.107:0/3141164181 learned_addr learned my addr 172.21.15.107:0/3141164181 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.618+0000 7f076dbcc640 1 --2- 172.21.15.107:0/3141164181 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0768102a70 0x7f0768073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.619+0000 7f076ebce640 1 --2- 172.21.15.107:0/3141164181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0768103430 0x7f0768074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.619+0000 7f076e3cd640 1 -- 172.21.15.107:0/3141164181 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0768102a70 msgr2=0x7f0768073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.619+0000 7f076e3cd640 1 --2- 172.21.15.107:0/3141164181 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0768102a70 0x7f0768073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.619+0000 7f076e3cd640 1 -- 172.21.15.107:0/3141164181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0768103430 msgr2=0x7f0768074160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.619+0000 7f076e3cd640 1 --2- 172.21.15.107:0/3141164181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0768103430 0x7f0768074160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.619+0000 7f076e3cd640 1 -- 172.21.15.107:0/3141164181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07680749f0 con 0x7f0768101880 2024-05-09T08:33:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.619+0000 7f076e3cd640 1 --2- 172.21.15.107:0/3141164181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0768101880 0x7f07680755e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f076400e9c0 tx=0x7f076400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.619+0000 7f075f7fe640 1 -- 172.21.15.107:0/3141164181 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f076400cd80 con 0x7f0768101880 2024-05-09T08:33:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.619+0000 7f0770631640 1 -- 172.21.15.107:0/3141164181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07681a1b80 con 0x7f0768101880 2024-05-09T08:33:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.619+0000 7f075f7fe640 1 -- 172.21.15.107:0/3141164181 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07640035f0 con 0x7f0768101880 2024-05-09T08:33:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.619+0000 7f075f7fe640 1 -- 172.21.15.107:0/3141164181 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0764010690 con 0x7f0768101880 2024-05-09T08:33:24.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.619+0000 7f0770631640 1 -- 172.21.15.107:0/3141164181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07681a2080 con 0x7f0768101880 2024-05-09T08:33:24.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.620+0000 7f0770631640 1 -- 172.21.15.107:0/3141164181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0768108ba0 con 0x7f0768101880 2024-05-09T08:33:24.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.621+0000 7f075f7fe640 1 -- 172.21.15.107:0/3141164181 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) ==== 138005+0+0 (secure 0 0 0) 0x7f07640037a0 con 0x7f0768101880 2024-05-09T08:33:24.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.622+0000 7f075f7fe640 1 --2- 172.21.15.107:0/3141164181 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07540a2d90 0x7f07540a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:24.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.623+0000 7f076dbcc640 1 --2- 172.21.15.107:0/3141164181 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07540a2d90 0x7f07540a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:24.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.623+0000 7f075f7fe640 1 -- 172.21.15.107:0/3141164181 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f07640cffe0 con 0x7f0768101880 2024-05-09T08:33:24.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.623+0000 7f076dbcc640 1 --2- 172.21.15.107:0/3141164181 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07540a2d90 0x7f07540a5250 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0768075350 tx=0x7f0758005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:24.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.627+0000 7f075f7fe640 1 -- 172.21.15.107:0/3141164181 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f076409c6d0 con 0x7f0768101880 2024-05-09T08:33:24.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.817+0000 7f0770631640 1 -- 172.21.15.107:0/3141164181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f076802d400 con 0x7f0768101880 2024-05-09T08:33:24.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.817+0000 7f075f7fe640 1 -- 172.21.15.107:0/3141164181 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f076409bfd0 con 0x7f0768101880 2024-05-09T08:33:24.816 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-05-09T08:33:24.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.819+0000 7f0770631640 1 -- 172.21.15.107:0/3141164181 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07540a2d90 msgr2=0x7f07540a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:24.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.819+0000 7f0770631640 1 --2- 172.21.15.107:0/3141164181 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07540a2d90 0x7f07540a5250 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0768075350 tx=0x7f0758005f90 comp rx=0 tx=0).stop 2024-05-09T08:33:24.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.819+0000 7f0770631640 1 -- 172.21.15.107:0/3141164181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0768101880 msgr2=0x7f07680755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:24.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.819+0000 7f0770631640 1 --2- 172.21.15.107:0/3141164181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0768101880 0x7f07680755e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f076400e9c0 tx=0x7f076400ee80 comp rx=0 tx=0).stop 2024-05-09T08:33:24.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.820+0000 7f0770631640 1 -- 172.21.15.107:0/3141164181 shutdown_connections 2024-05-09T08:33:24.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.820+0000 7f0770631640 1 --2- 172.21.15.107:0/3141164181 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07540a2d90 0x7f07540a5250 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.820+0000 7f0770631640 1 --2- 172.21.15.107:0/3141164181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0768103430 0x7f0768074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.820+0000 7f0770631640 1 --2- 172.21.15.107:0/3141164181 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0768102a70 0x7f0768073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.820+0000 7f0770631640 1 --2- 172.21.15.107:0/3141164181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0768101880 0x7f07680755e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.820+0000 7f0770631640 1 -- 172.21.15.107:0/3141164181 >> 172.21.15.107:0/3141164181 conn(0x7f07680fce10 msgr2=0x7f0768105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:24.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.820+0000 7f0770631640 1 -- 172.21.15.107:0/3141164181 shutdown_connections 2024-05-09T08:33:24.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.820+0000 7f0770631640 1 -- 172.21.15.107:0/3141164181 wait complete. 2024-05-09T08:33:24.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:33:24.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:33:24.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-05-09T08:33:24.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/646797162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:33:24.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1867328022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:24.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:33:24.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:33:24.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:24 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/708409265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:33:24.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:33:24.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-05-09T08:33:24.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/646797162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:33:24.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1867328022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:24.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:24.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:33:24.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:33:24.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:33:24.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:24 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/708409265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:33:24.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.942+0000 7f6df2c3f640 1 -- 172.21.15.107:0/1614660695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dec10bdf0 msgr2=0x7f6dec10e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:24.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.942+0000 7f6df2c3f640 1 --2- 172.21.15.107:0/1614660695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dec10bdf0 0x7f6dec10e290 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4009900 tx=0x7f6dd402a510 comp rx=0 tx=0).stop 2024-05-09T08:33:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.942+0000 7f6df2c3f640 1 -- 172.21.15.107:0/1614660695 shutdown_connections 2024-05-09T08:33:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.942+0000 7f6df2c3f640 1 --2- 172.21.15.107:0/1614660695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dec10bdf0 0x7f6dec10e290 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.942+0000 7f6df2c3f640 1 --2- 172.21.15.107:0/1614660695 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6dec1033d0 0x7f6dec10b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.942+0000 7f6df2c3f640 1 --2- 172.21.15.107:0/1614660695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dec102ab0 0x7f6dec102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.942+0000 7f6df2c3f640 1 -- 172.21.15.107:0/1614660695 >> 172.21.15.107:0/1614660695 conn(0x7f6dec077830 msgr2=0x7f6dec077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.942+0000 7f6df2c3f640 1 -- 172.21.15.107:0/1614660695 shutdown_connections 2024-05-09T08:33:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.942+0000 7f6df2c3f640 1 -- 172.21.15.107:0/1614660695 wait complete. 2024-05-09T08:33:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.942+0000 7f6df2c3f640 1 Processor -- start 2024-05-09T08:33:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.942+0000 7f6df2c3f640 1 -- start start 2024-05-09T08:33:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.942+0000 7f6df2c3f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dec102ab0 0x7f6dec197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df2c3f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6dec1033d0 0x7f6dec1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df2c3f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dec10bdf0 0x7f6dec19c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df2c3f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec111aa0 con 0x7f6dec10bdf0 2024-05-09T08:33:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df2c3f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec111920 con 0x7f6dec102ab0 2024-05-09T08:33:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df2c3f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec111c20 con 0x7f6dec1033d0 2024-05-09T08:33:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df11dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dec10bdf0 0x7f6dec19c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df11dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dec10bdf0 0x7f6dec19c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50408/0 (socket says 172.21.15.107:50408) 2024-05-09T08:33:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df11dc640 1 -- 172.21.15.107:0/2892770957 learned_addr learned my addr 172.21.15.107:0/2892770957 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df09db640 1 --2- 172.21.15.107:0/2892770957 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dec102ab0 0x7f6dec197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6debfff640 1 --2- 172.21.15.107:0/2892770957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6dec1033d0 0x7f6dec1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df11dc640 1 -- 172.21.15.107:0/2892770957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6dec1033d0 msgr2=0x7f6dec1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df11dc640 1 --2- 172.21.15.107:0/2892770957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6dec1033d0 0x7f6dec1984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df11dc640 1 -- 172.21.15.107:0/2892770957 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dec102ab0 msgr2=0x7f6dec197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df11dc640 1 --2- 172.21.15.107:0/2892770957 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dec102ab0 0x7f6dec197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:24.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df11dc640 1 -- 172.21.15.107:0/2892770957 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dec19cf20 con 0x7f6dec10bdf0 2024-05-09T08:33:24.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df09db640 1 --2- 172.21.15.107:0/2892770957 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dec102ab0 0x7f6dec197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:33:24.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df11dc640 1 --2- 172.21.15.107:0/2892770957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dec10bdf0 0x7f6dec19c7f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4004770 tx=0x7f6dd4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:24.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6de9ffb640 1 -- 172.21.15.107:0/2892770957 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd403f030 con 0x7f6dec10bdf0 2024-05-09T08:33:24.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6df2c3f640 1 -- 172.21.15.107:0/2892770957 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dec19d1b0 con 0x7f6dec10bdf0 2024-05-09T08:33:24.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6de9ffb640 1 -- 172.21.15.107:0/2892770957 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dd40318e0 con 0x7f6dec10bdf0 2024-05-09T08:33:24.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.943+0000 7f6de9ffb640 1 -- 172.21.15.107:0/2892770957 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4035070 con 0x7f6dec10bdf0 2024-05-09T08:33:24.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.944+0000 7f6df2c3f640 1 -- 172.21.15.107:0/2892770957 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dec1a4aa0 con 0x7f6dec10bdf0 2024-05-09T08:33:24.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.944+0000 7f6df2c3f640 1 -- 172.21.15.107:0/2892770957 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dec068650 con 0x7f6dec10bdf0 2024-05-09T08:33:24.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.946+0000 7f6de9ffb640 1 -- 172.21.15.107:0/2892770957 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) ==== 138005+0+0 (secure 0 0 0) 0x7f6dd4042030 con 0x7f6dec10bdf0 2024-05-09T08:33:24.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.946+0000 7f6de9ffb640 1 --2- 172.21.15.107:0/2892770957 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6dd80a2d90 0x7f6dd80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:24.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.946+0000 7f6de9ffb640 1 -- 172.21.15.107:0/2892770957 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6dd40f01d0 con 0x7f6dec10bdf0 2024-05-09T08:33:24.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.946+0000 7f6df09db640 1 --2- 172.21.15.107:0/2892770957 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6dd80a2d90 0x7f6dd80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:24.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.947+0000 7f6df09db640 1 --2- 172.21.15.107:0/2892770957 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6dd80a2d90 0x7f6dd80a5250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009fc0 tx=0x7f6de4005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:24.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:24.950+0000 7f6de9ffb640 1 -- 172.21.15.107:0/2892770957 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6dd40bc920 con 0x7f6dec10bdf0 2024-05-09T08:33:25.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.159+0000 7f6df2c3f640 1 -- 172.21.15.107:0/2892770957 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6dec106f70 con 0x7f6dec10bdf0 2024-05-09T08:33:25.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.159+0000 7f6de9ffb640 1 -- 172.21.15.107:0/2892770957 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f6dd40bc220 con 0x7f6dec10bdf0 2024-05-09T08:33:25.159 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:33:25.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.161+0000 7f6df2c3f640 1 -- 172.21.15.107:0/2892770957 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6dd80a2d90 msgr2=0x7f6dd80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:25.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.161+0000 7f6df2c3f640 1 --2- 172.21.15.107:0/2892770957 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6dd80a2d90 0x7f6dd80a5250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009fc0 tx=0x7f6de4005f90 comp rx=0 tx=0).stop 2024-05-09T08:33:25.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.161+0000 7f6df2c3f640 1 -- 172.21.15.107:0/2892770957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dec10bdf0 msgr2=0x7f6dec19c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:25.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.161+0000 7f6df2c3f640 1 --2- 172.21.15.107:0/2892770957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dec10bdf0 0x7f6dec19c7f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4004770 tx=0x7f6dd4004270 comp rx=0 tx=0).stop 2024-05-09T08:33:25.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.161+0000 7f6df2c3f640 1 -- 172.21.15.107:0/2892770957 shutdown_connections 2024-05-09T08:33:25.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.161+0000 7f6df2c3f640 1 --2- 172.21.15.107:0/2892770957 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6dd80a2d90 0x7f6dd80a5250 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.161+0000 7f6df2c3f640 1 --2- 172.21.15.107:0/2892770957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dec10bdf0 0x7f6dec19c7f0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.161+0000 7f6df2c3f640 1 --2- 172.21.15.107:0/2892770957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6dec1033d0 0x7f6dec1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.161+0000 7f6df2c3f640 1 --2- 172.21.15.107:0/2892770957 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dec102ab0 0x7f6dec197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.161+0000 7f6df2c3f640 1 -- 172.21.15.107:0/2892770957 >> 172.21.15.107:0/2892770957 conn(0x7f6dec077830 msgr2=0x7f6dec0fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:25.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.162+0000 7f6df2c3f640 1 -- 172.21.15.107:0/2892770957 shutdown_connections 2024-05-09T08:33:25.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.162+0000 7f6df2c3f640 1 -- 172.21.15.107:0/2892770957 wait complete. 2024-05-09T08:33:25.172 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-05-09T08:33:25.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:33:25.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.288+0000 7f4e9e69e640 1 -- 172.21.15.107:0/3105467287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e98102ac0 msgr2=0x7f4e98102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.288+0000 7f4e9e69e640 1 --2- 172.21.15.107:0/3105467287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e98102ac0 0x7f4e98102ea0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f4e88009900 tx=0x7f4e8802a510 comp rx=0 tx=0).stop 2024-05-09T08:33:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.288+0000 7f4e9e69e640 1 -- 172.21.15.107:0/3105467287 shutdown_connections 2024-05-09T08:33:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.288+0000 7f4e9e69e640 1 --2- 172.21.15.107:0/3105467287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e9810be00 0x7f4e9810e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.288+0000 7f4e9e69e640 1 --2- 172.21.15.107:0/3105467287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e981033e0 0x7f4e9810b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.288+0000 7f4e9e69e640 1 --2- 172.21.15.107:0/3105467287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e98102ac0 0x7f4e98102ea0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.288+0000 7f4e9e69e640 1 -- 172.21.15.107:0/3105467287 >> 172.21.15.107:0/3105467287 conn(0x7f4e98077830 msgr2=0x7f4e98077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.288+0000 7f4e9e69e640 1 -- 172.21.15.107:0/3105467287 shutdown_connections 2024-05-09T08:33:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.288+0000 7f4e9e69e640 1 -- 172.21.15.107:0/3105467287 wait complete. 2024-05-09T08:33:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.289+0000 7f4e9e69e640 1 Processor -- start 2024-05-09T08:33:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.289+0000 7f4e9e69e640 1 -- start start 2024-05-09T08:33:25.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.289+0000 7f4e9e69e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e98102ac0 0x7f4e98197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:25.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.289+0000 7f4e9e69e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e981033e0 0x7f4e981984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.289+0000 7f4e9e69e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e9810be00 0x7f4e9819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.289+0000 7f4e9e69e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4e98111ab0 con 0x7f4e98102ac0 2024-05-09T08:33:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.289+0000 7f4e9e69e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4e98111930 con 0x7f4e9810be00 2024-05-09T08:33:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.289+0000 7f4e9e69e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4e98111c30 con 0x7f4e981033e0 2024-05-09T08:33:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.289+0000 7f4e97fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e98102ac0 0x7f4e98197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.289+0000 7f4e97fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e98102ac0 0x7f4e98197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50430/0 (socket says 172.21.15.107:50430) 2024-05-09T08:33:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.289+0000 7f4e97fff640 1 -- 172.21.15.107:0/1439959666 learned_addr learned my addr 172.21.15.107:0/1439959666 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.289+0000 7f4e977fe640 1 --2- 172.21.15.107:0/1439959666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e981033e0 0x7f4e981984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.290+0000 7f4e9cc3b640 1 --2- 172.21.15.107:0/1439959666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e9810be00 0x7f4e9819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.290+0000 7f4e97fff640 1 -- 172.21.15.107:0/1439959666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e981033e0 msgr2=0x7f4e981984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.290+0000 7f4e97fff640 1 --2- 172.21.15.107:0/1439959666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e981033e0 0x7f4e981984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.290+0000 7f4e97fff640 1 -- 172.21.15.107:0/1439959666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e9810be00 msgr2=0x7f4e9819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.290+0000 7f4e97fff640 1 --2- 172.21.15.107:0/1439959666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e9810be00 0x7f4e9819c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.290+0000 7f4e97fff640 1 -- 172.21.15.107:0/1439959666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9819cf80 con 0x7f4e98102ac0 2024-05-09T08:33:25.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.290+0000 7f4e97fff640 1 --2- 172.21.15.107:0/1439959666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e98102ac0 0x7f4e98197f90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f4e88005940 tx=0x7f4e880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:25.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.290+0000 7f4e957fa640 1 -- 172.21.15.107:0/1439959666 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8803f030 con 0x7f4e98102ac0 2024-05-09T08:33:25.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.290+0000 7f4e9e69e640 1 -- 172.21.15.107:0/1439959666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e9819d210 con 0x7f4e98102ac0 2024-05-09T08:33:25.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.290+0000 7f4e957fa640 1 -- 172.21.15.107:0/1439959666 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e88003cb0 con 0x7f4e98102ac0 2024-05-09T08:33:25.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.290+0000 7f4e957fa640 1 -- 172.21.15.107:0/1439959666 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e88035070 con 0x7f4e98102ac0 2024-05-09T08:33:25.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.290+0000 7f4e9e69e640 1 -- 172.21.15.107:0/1439959666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e981a4b00 con 0x7f4e98102ac0 2024-05-09T08:33:25.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.291+0000 7f4e9e69e640 1 -- 172.21.15.107:0/1439959666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e98068650 con 0x7f4e98102ac0 2024-05-09T08:33:25.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.293+0000 7f4e957fa640 1 -- 172.21.15.107:0/1439959666 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) ==== 138005+0+0 (secure 0 0 0) 0x7f4e88042030 con 0x7f4e98102ac0 2024-05-09T08:33:25.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.293+0000 7f4e957fa640 1 --2- 172.21.15.107:0/1439959666 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4e800a2d90 0x7f4e800a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:25.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.293+0000 7f4e977fe640 1 --2- 172.21.15.107:0/1439959666 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4e800a2d90 0x7f4e800a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:25.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.293+0000 7f4e957fa640 1 -- 172.21.15.107:0/1439959666 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4e880f02c0 con 0x7f4e98102ac0 2024-05-09T08:33:25.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.294+0000 7f4e977fe640 1 --2- 172.21.15.107:0/1439959666 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4e800a2d90 0x7f4e800a5250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4e98100110 tx=0x7f4e8c00a2d0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:25.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.297+0000 7f4e957fa640 1 -- 172.21.15.107:0/1439959666 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4e880bcb30 con 0x7f4e98102ac0 2024-05-09T08:33:25.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.398+0000 7f4e957fa640 1 -- 172.21.15.107:0/1439959666 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7f4e880b8aa0 con 0x7f4e98102ac0 2024-05-09T08:33:25.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.520+0000 7f4e9e69e640 1 -- 172.21.15.107:0/1439959666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e98106f80 con 0x7f4e98102ac0 2024-05-09T08:33:25.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.521+0000 7f4e957fa640 1 -- 172.21.15.107:0/1439959666 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11887 (secure 0 0 0) 0x7f4e880db700 con 0x7f4e98102ac0 2024-05-09T08:33:25.520 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:25.520 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:33:19.308598+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:33:25.521 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 25 2024-05-09T08:33:25.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.523+0000 7f4e9e69e640 1 -- 172.21.15.107:0/1439959666 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4e800a2d90 msgr2=0x7f4e800a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:25.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.523+0000 7f4e9e69e640 1 --2- 172.21.15.107:0/1439959666 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4e800a2d90 0x7f4e800a5250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4e98100110 tx=0x7f4e8c00a2d0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.523+0000 7f4e9e69e640 1 -- 172.21.15.107:0/1439959666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e98102ac0 msgr2=0x7f4e98197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:25.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.523+0000 7f4e9e69e640 1 --2- 172.21.15.107:0/1439959666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e98102ac0 0x7f4e98197f90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f4e88005940 tx=0x7f4e880042f0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.523+0000 7f4e9e69e640 1 -- 172.21.15.107:0/1439959666 shutdown_connections 2024-05-09T08:33:25.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.523+0000 7f4e9e69e640 1 --2- 172.21.15.107:0/1439959666 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4e800a2d90 0x7f4e800a5250 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.523+0000 7f4e9e69e640 1 --2- 172.21.15.107:0/1439959666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e9810be00 0x7f4e9819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.523+0000 7f4e9e69e640 1 --2- 172.21.15.107:0/1439959666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e981033e0 0x7f4e981984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.523+0000 7f4e9e69e640 1 --2- 172.21.15.107:0/1439959666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e98102ac0 0x7f4e98197f90 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.523+0000 7f4e9e69e640 1 -- 172.21.15.107:0/1439959666 >> 172.21.15.107:0/1439959666 conn(0x7f4e98077830 msgr2=0x7f4e980fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:25.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.523+0000 7f4e9e69e640 1 -- 172.21.15.107:0/1439959666 shutdown_connections 2024-05-09T08:33:25.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.523+0000 7f4e9e69e640 1 -- 172.21.15.107:0/1439959666 wait complete. 2024-05-09T08:33:25.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi157 to 176.2M 2024-05-09T08:33:25.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: Unable to set osd_memory_target on smithi157 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-05-09T08:33:25.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi107 to 176.1M 2024-05-09T08:33:25.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: Unable to set osd_memory_target on smithi107 to 184708608: error parsing value: Value '184708608' is below minimum 939524096 2024-05-09T08:33:25.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi110 to 176.1M 2024-05-09T08:33:25.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: Unable to set osd_memory_target on smithi110 to 184745881: error parsing value: Value '184745881' is below minimum 939524096 2024-05-09T08:33:25.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: Updating smithi107:/etc/ceph/ceph.conf 2024-05-09T08:33:25.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: Updating smithi110:/etc/ceph/ceph.conf 2024-05-09T08:33:25.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: Updating smithi157:/etc/ceph/ceph.conf 2024-05-09T08:33:25.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: Updating smithi107:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:33:25.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: Updating smithi110:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:33:25.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: Updating smithi157:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:33:25.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3923181866' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:33:25.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3141164181' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:33:25.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2892770957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:33:25.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:25 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.534 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-05-09T08:33:25.534 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-05-09T08:33:25.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T08:33:25.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.637+0000 7f6983015640 1 -- 172.21.15.107:0/3428327435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f697c100710 msgr2=0x7f697c100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:25.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.637+0000 7f6983015640 1 --2- 172.21.15.107:0/3428327435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f697c100710 0x7f697c100b90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6964009930 tx=0x7f696402a420 comp rx=0 tx=0).stop 2024-05-09T08:33:25.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.637+0000 7f6983015640 1 -- 172.21.15.107:0/3428327435 shutdown_connections 2024-05-09T08:33:25.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.637+0000 7f6983015640 1 --2- 172.21.15.107:0/3428327435 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f697c1010d0 0x7f697c10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.637+0000 7f6983015640 1 --2- 172.21.15.107:0/3428327435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f697c100710 0x7f697c100b90 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.637+0000 7f6983015640 1 --2- 172.21.15.107:0/3428327435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f697c106720 0x7f697c106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.637+0000 7f6983015640 1 -- 172.21.15.107:0/3428327435 >> 172.21.15.107:0/3428327435 conn(0x7f697c077800 msgr2=0x7f697c077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:25.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.637+0000 7f6983015640 1 -- 172.21.15.107:0/3428327435 shutdown_connections 2024-05-09T08:33:25.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.637+0000 7f6983015640 1 -- 172.21.15.107:0/3428327435 wait complete. 2024-05-09T08:33:25.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f6983015640 1 Processor -- start 2024-05-09T08:33:25.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f6983015640 1 -- start start 2024-05-09T08:33:25.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f6983015640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f697c100710 0x7f697c196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:25.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f6983015640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f697c1010d0 0x7f697c197490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:25.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f6983015640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f697c106720 0x7f697c19b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:25.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f6983015640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f697c1109e0 con 0x7f697c106720 2024-05-09T08:33:25.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f6983015640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f697c110860 con 0x7f697c100710 2024-05-09T08:33:25.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f6983015640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f697c110b60 con 0x7f697c1010d0 2024-05-09T08:33:25.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f69815b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f697c106720 0x7f697c19b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:25.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f69815b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f697c106720 0x7f697c19b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50438/0 (socket says 172.21.15.107:50438) 2024-05-09T08:33:25.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f69815b2640 1 -- 172.21.15.107:0/1817696455 learned_addr learned my addr 172.21.15.107:0/1817696455 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:25.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f6980db1640 1 --2- 172.21.15.107:0/1817696455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f697c100710 0x7f697c196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:25.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f697bfff640 1 --2- 172.21.15.107:0/1817696455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f697c1010d0 0x7f697c197490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:25.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f69815b2640 1 -- 172.21.15.107:0/1817696455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f697c1010d0 msgr2=0x7f697c197490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:25.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f69815b2640 1 --2- 172.21.15.107:0/1817696455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f697c1010d0 0x7f697c197490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f69815b2640 1 -- 172.21.15.107:0/1817696455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f697c100710 msgr2=0x7f697c196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:25.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f69815b2640 1 --2- 172.21.15.107:0/1817696455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f697c100710 0x7f697c196f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f69815b2640 1 -- 172.21.15.107:0/1817696455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697c19bf40 con 0x7f697c106720 2024-05-09T08:33:25.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f69815b2640 1 --2- 172.21.15.107:0/1817696455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f697c106720 0x7f697c19b810 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f697400d950 tx=0x7f697400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:25.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.638+0000 7f6979ffb640 1 -- 172.21.15.107:0/1817696455 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974017030 con 0x7f697c106720 2024-05-09T08:33:25.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.639+0000 7f6983015640 1 -- 172.21.15.107:0/1817696455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f697c19c230 con 0x7f697c106720 2024-05-09T08:33:25.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.639+0000 7f6979ffb640 1 -- 172.21.15.107:0/1817696455 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6974004020 con 0x7f697c106720 2024-05-09T08:33:25.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.639+0000 7f6979ffb640 1 -- 172.21.15.107:0/1817696455 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974003b20 con 0x7f697c106720 2024-05-09T08:33:25.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.639+0000 7f6983015640 1 -- 172.21.15.107:0/1817696455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f697c1a3ac0 con 0x7f697c106720 2024-05-09T08:33:25.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.640+0000 7f6983015640 1 -- 172.21.15.107:0/1817696455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f697c068aa0 con 0x7f697c106720 2024-05-09T08:33:25.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.641+0000 7f6979ffb640 1 -- 172.21.15.107:0/1817696455 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7f697401a070 con 0x7f697c106720 2024-05-09T08:33:25.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.641+0000 7f6979ffb640 1 --2- 172.21.15.107:0/1817696455 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f69680a2e20 0x7f69680a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:25.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.641+0000 7f6980db1640 1 --2- 172.21.15.107:0/1817696455 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f69680a2e20 0x7f69680a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:25.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.641+0000 7f6979ffb640 1 -- 172.21.15.107:0/1817696455 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f69740cf200 con 0x7f697c106720 2024-05-09T08:33:25.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.642+0000 7f6980db1640 1 --2- 172.21.15.107:0/1817696455 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f69680a2e20 0x7f69680a52e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f696c005fd0 tx=0x7f696c009680 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:25.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.646+0000 7f6979ffb640 1 -- 172.21.15.107:0/1817696455 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6974021070 con 0x7f697c106720 2024-05-09T08:33:25.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.819+0000 7f6983015640 1 -- 172.21.15.107:0/1817696455 --> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f697c108ce0 con 0x7f69680a2e20 2024-05-09T08:33:25.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.829+0000 7f6979ffb640 1 -- 172.21.15.107:0/1817696455 <== mgr.15483 v2:172.21.15.107:6800/165281804 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339384 (secure 0 0 0) 0x7f697c108ce0 con 0x7f69680a2e20 2024-05-09T08:33:25.832 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:25.834 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-05-09T08:33:25.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.832+0000 7f6983015640 1 -- 172.21.15.107:0/1817696455 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f69680a2e20 msgr2=0x7f69680a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:25.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.832+0000 7f6983015640 1 --2- 172.21.15.107:0/1817696455 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f69680a2e20 0x7f69680a52e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f696c005fd0 tx=0x7f696c009680 comp rx=0 tx=0).stop 2024-05-09T08:33:25.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.832+0000 7f6983015640 1 -- 172.21.15.107:0/1817696455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f697c106720 msgr2=0x7f697c19b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:25.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.832+0000 7f6983015640 1 --2- 172.21.15.107:0/1817696455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f697c106720 0x7f697c19b810 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f697400d950 tx=0x7f697400de10 comp rx=0 tx=0).stop 2024-05-09T08:33:25.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.832+0000 7f6983015640 1 -- 172.21.15.107:0/1817696455 shutdown_connections 2024-05-09T08:33:25.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.832+0000 7f6983015640 1 --2- 172.21.15.107:0/1817696455 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f69680a2e20 0x7f69680a52e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.832+0000 7f6983015640 1 --2- 172.21.15.107:0/1817696455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f697c106720 0x7f697c19b810 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.832+0000 7f6983015640 1 --2- 172.21.15.107:0/1817696455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f697c1010d0 0x7f697c197490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.832+0000 7f6983015640 1 --2- 172.21.15.107:0/1817696455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f697c100710 0x7f697c196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.832+0000 7f6983015640 1 -- 172.21.15.107:0/1817696455 >> 172.21.15.107:0/1817696455 conn(0x7f697c077800 msgr2=0x7f697c103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:25.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.832+0000 7f6983015640 1 -- 172.21.15.107:0/1817696455 shutdown_connections 2024-05-09T08:33:25.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.833+0000 7f6983015640 1 -- 172.21.15.107:0/1817696455 wait complete. 2024-05-09T08:33:25.843 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-05-09T08:33:24.848804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191304,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":987,"num_read_kb":656,"num_write":477,"num_write_kb":4386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1182039,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":872,"ondisk_log_size":872,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":398688,"kb_used_data":16152,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124543136,"statfs":{"total":1151940427776,"available":1151532171264,"internally_reserved":0,"allocated":16539648,"data_stored":9826433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.992086"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224506+0000","last_change":"2024-05-09T08:31:00.695034+0000","last_active":"2024-05-09T08:33:21.224506+0000","last_peered":"2024-05-09T08:33:21.224506+0000","last_clean":"2024-05-09T08:33:21.224506+0000","last_became_active":"2024-05-09T08:31:00.694229+0000","last_became_peered":"2024-05-09T08:31:00.694229+0000","last_unstale":"2024-05-09T08:33:21.224506+0000","last_undegraded":"2024-05-09T08:33:21.224506+0000","last_fullsized":"2024-05-09T08:33:21.224506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:57:22.964848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224706+0000","last_change":"2024-05-09T08:31:00.686219+0000","last_active":"2024-05-09T08:33:21.224706+0000","last_peered":"2024-05-09T08:33:21.224706+0000","last_clean":"2024-05-09T08:33:21.224706+0000","last_became_active":"2024-05-09T08:31:00.685911+0000","last_became_peered":"2024-05-09T08:31:00.685911+0000","last_unstale":"2024-05-09T08:33:21.224706+0000","last_undegraded":"2024-05-09T08:33:21.224706+0000","last_fullsized":"2024-05-09T08:33:21.224706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:28.246199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224414+0000","last_change":"2024-05-09T08:31:00.694228+0000","last_active":"2024-05-09T08:33:21.224414+0000","last_peered":"2024-05-09T08:33:21.224414+0000","last_clean":"2024-05-09T08:33:21.224414+0000","last_became_active":"2024-05-09T08:31:00.693838+0000","last_became_peered":"2024-05-09T08:31:00.693838+0000","last_unstale":"2024-05-09T08:33:21.224414+0000","last_undegraded":"2024-05-09T08:33:21.224414+0000","last_fullsized":"2024-05-09T08:33:21.224414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:44.847966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.078034+0000","last_change":"2024-05-09T08:31:00.702770+0000","last_active":"2024-05-09T08:33:21.078034+0000","last_peered":"2024-05-09T08:33:21.078034+0000","last_clean":"2024-05-09T08:33:21.078034+0000","last_became_active":"2024-05-09T08:31:00.702424+0000","last_became_peered":"2024-05-09T08:31:00.702424+0000","last_unstale":"2024-05-09T08:33:21.078034+0000","last_undegraded":"2024-05-09T08:33:21.078034+0000","last_fullsized":"2024-05-09T08:33:21.078034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:24:05.228942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933801+0000","last_change":"2024-05-09T08:31:00.688308+0000","last_active":"2024-05-09T08:33:20.933801+0000","last_peered":"2024-05-09T08:33:20.933801+0000","last_clean":"2024-05-09T08:33:20.933801+0000","last_became_active":"2024-05-09T08:31:00.687891+0000","last_became_peered":"2024-05-09T08:31:00.687891+0000","last_unstale":"2024-05-09T08:33:20.933801+0000","last_undegraded":"2024-05-09T08:33:20.933801+0000","last_fullsized":"2024-05-09T08:33:20.933801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:48:42.648940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835147+0000","last_change":"2024-05-09T08:31:00.694705+0000","last_active":"2024-05-09T08:33:20.835147+0000","last_peered":"2024-05-09T08:33:20.835147+0000","last_clean":"2024-05-09T08:33:20.835147+0000","last_became_active":"2024-05-09T08:31:00.694361+0000","last_became_peered":"2024-05-09T08:31:00.694361+0000","last_unstale":"2024-05-09T08:33:20.835147+0000","last_undegraded":"2024-05-09T08:33:20.835147+0000","last_fullsized":"2024-05-09T08:33:20.835147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:07:23.137410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074662+0000","last_change":"2024-05-09T08:31:00.689824+0000","last_active":"2024-05-09T08:33:21.074662+0000","last_peered":"2024-05-09T08:33:21.074662+0000","last_clean":"2024-05-09T08:33:21.074662+0000","last_became_active":"2024-05-09T08:31:00.689491+0000","last_became_peered":"2024-05-09T08:31:00.689491+0000","last_unstale":"2024-05-09T08:33:21.074662+0000","last_undegraded":"2024-05-09T08:33:21.074662+0000","last_fullsized":"2024-05-09T08:33:21.074662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:18.791225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835733+0000","last_change":"2024-05-09T08:31:00.695014+0000","last_active":"2024-05-09T08:33:20.835733+0000","last_peered":"2024-05-09T08:33:20.835733+0000","last_clean":"2024-05-09T08:33:20.835733+0000","last_became_active":"2024-05-09T08:31:00.694683+0000","last_became_peered":"2024-05-09T08:31:00.694683+0000","last_unstale":"2024-05-09T08:33:20.835733+0000","last_undegraded":"2024-05-09T08:33:20.835733+0000","last_fullsized":"2024-05-09T08:33:20.835733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:08.570258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.924473+0000","last_change":"2024-05-09T08:31:00.701538+0000","last_active":"2024-05-09T08:33:20.924473+0000","last_peered":"2024-05-09T08:33:20.924473+0000","last_clean":"2024-05-09T08:33:20.924473+0000","last_became_active":"2024-05-09T08:31:00.701226+0000","last_became_peered":"2024-05-09T08:31:00.701226+0000","last_unstale":"2024-05-09T08:33:20.924473+0000","last_undegraded":"2024-05-09T08:33:20.924473+0000","last_fullsized":"2024-05-09T08:33:20.924473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:14:50.853063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835676+0000","last_change":"2024-05-09T08:31:00.698145+0000","last_active":"2024-05-09T08:33:20.835676+0000","last_peered":"2024-05-09T08:33:20.835676+0000","last_clean":"2024-05-09T08:33:20.835676+0000","last_became_active":"2024-05-09T08:31:00.697761+0000","last_became_peered":"2024-05-09T08:31:00.697761+0000","last_unstale":"2024-05-09T08:33:20.835676+0000","last_undegraded":"2024-05-09T08:33:20.835676+0000","last_fullsized":"2024-05-09T08:33:20.835676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:26.094013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075195+0000","last_change":"2024-05-09T08:31:00.699520+0000","last_active":"2024-05-09T08:33:21.075195+0000","last_peered":"2024-05-09T08:33:21.075195+0000","last_clean":"2024-05-09T08:33:21.075195+0000","last_became_active":"2024-05-09T08:31:00.697579+0000","last_became_peered":"2024-05-09T08:31:00.697579+0000","last_unstale":"2024-05-09T08:33:21.075195+0000","last_undegraded":"2024-05-09T08:33:21.075195+0000","last_fullsized":"2024-05-09T08:33:21.075195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:26:34.324474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077854+0000","last_change":"2024-05-09T08:31:00.693356+0000","last_active":"2024-05-09T08:33:21.077854+0000","last_peered":"2024-05-09T08:33:21.077854+0000","last_clean":"2024-05-09T08:33:21.077854+0000","last_became_active":"2024-05-09T08:31:00.692985+0000","last_became_peered":"2024-05-09T08:31:00.692985+0000","last_unstale":"2024-05-09T08:33:21.077854+0000","last_undegraded":"2024-05-09T08:33:21.077854+0000","last_fullsized":"2024-05-09T08:33:21.077854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:49:01.115497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225089+0000","last_change":"2024-05-09T08:31:00.677768+0000","last_active":"2024-05-09T08:33:21.225089+0000","last_peered":"2024-05-09T08:33:21.225089+0000","last_clean":"2024-05-09T08:33:21.225089+0000","last_became_active":"2024-05-09T08:31:00.677386+0000","last_became_peered":"2024-05-09T08:31:00.677386+0000","last_unstale":"2024-05-09T08:33:21.225089+0000","last_undegraded":"2024-05-09T08:33:21.225089+0000","last_fullsized":"2024-05-09T08:33:21.225089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:55.363324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224484+0000","last_change":"2024-05-09T08:31:00.689609+0000","last_active":"2024-05-09T08:33:21.224484+0000","last_peered":"2024-05-09T08:33:21.224484+0000","last_clean":"2024-05-09T08:33:21.224484+0000","last_became_active":"2024-05-09T08:31:00.689303+0000","last_became_peered":"2024-05-09T08:31:00.689303+0000","last_unstale":"2024-05-09T08:33:21.224484+0000","last_undegraded":"2024-05-09T08:33:21.224484+0000","last_fullsized":"2024-05-09T08:33:21.224484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:15:58.672460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224405+0000","last_change":"2024-05-09T08:31:00.704708+0000","last_active":"2024-05-09T08:33:21.224405+0000","last_peered":"2024-05-09T08:33:21.224405+0000","last_clean":"2024-05-09T08:33:21.224405+0000","last_became_active":"2024-05-09T08:31:00.704269+0000","last_became_peered":"2024-05-09T08:31:00.704269+0000","last_unstale":"2024-05-09T08:33:21.224405+0000","last_undegraded":"2024-05-09T08:33:21.224405+0000","last_fullsized":"2024-05-09T08:33:21.224405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:17:44.580420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.923876+0000","last_change":"2024-05-09T08:31:00.703089+0000","last_active":"2024-05-09T08:33:20.923876+0000","last_peered":"2024-05-09T08:33:20.923876+0000","last_clean":"2024-05-09T08:33:20.923876+0000","last_became_active":"2024-05-09T08:31:00.702531+0000","last_became_peered":"2024-05-09T08:31:00.702531+0000","last_unstale":"2024-05-09T08:33:20.923876+0000","last_undegraded":"2024-05-09T08:33:20.923876+0000","last_fullsized":"2024-05-09T08:33:20.923876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:07:39.887748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224486+0000","last_change":"2024-05-09T08:31:00.695332+0000","last_active":"2024-05-09T08:33:21.224486+0000","last_peered":"2024-05-09T08:33:21.224486+0000","last_clean":"2024-05-09T08:33:21.224486+0000","last_became_active":"2024-05-09T08:31:00.694703+0000","last_became_peered":"2024-05-09T08:31:00.694703+0000","last_unstale":"2024-05-09T08:33:21.224486+0000","last_undegraded":"2024-05-09T08:33:21.224486+0000","last_fullsized":"2024-05-09T08:33:21.224486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:53.727922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931989+0000","last_change":"2024-05-09T08:31:00.689871+0000","last_active":"2024-05-09T08:33:20.931989+0000","last_peered":"2024-05-09T08:33:20.931989+0000","last_clean":"2024-05-09T08:33:20.931989+0000","last_became_active":"2024-05-09T08:31:00.689498+0000","last_became_peered":"2024-05-09T08:31:00.689498+0000","last_unstale":"2024-05-09T08:33:20.931989+0000","last_undegraded":"2024-05-09T08:33:20.931989+0000","last_fullsized":"2024-05-09T08:33:20.931989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:18:14.123442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835195+0000","last_change":"2024-05-09T08:31:00.697175+0000","last_active":"2024-05-09T08:33:20.835195+0000","last_peered":"2024-05-09T08:33:20.835195+0000","last_clean":"2024-05-09T08:33:20.835195+0000","last_became_active":"2024-05-09T08:31:00.694103+0000","last_became_peered":"2024-05-09T08:31:00.694103+0000","last_unstale":"2024-05-09T08:33:20.835195+0000","last_undegraded":"2024-05-09T08:33:20.835195+0000","last_fullsized":"2024-05-09T08:33:20.835195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:41:12.549854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834497+0000","last_change":"2024-05-09T08:31:56.797212+0000","last_active":"2024-05-09T08:33:20.834497+0000","last_peered":"2024-05-09T08:33:20.834497+0000","last_clean":"2024-05-09T08:33:20.834497+0000","last_became_active":"2024-05-09T08:31:56.796786+0000","last_became_peered":"2024-05-09T08:31:56.796786+0000","last_unstale":"2024-05-09T08:33:20.834497+0000","last_undegraded":"2024-05-09T08:33:20.834497+0000","last_fullsized":"2024-05-09T08:33:20.834497+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:55.270522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931844+0000","last_change":"2024-05-09T08:31:56.578564+0000","last_active":"2024-05-09T08:33:20.931844+0000","last_peered":"2024-05-09T08:33:20.931844+0000","last_clean":"2024-05-09T08:33:20.931844+0000","last_became_active":"2024-05-09T08:31:56.571246+0000","last_became_peered":"2024-05-09T08:31:56.571246+0000","last_unstale":"2024-05-09T08:33:20.931844+0000","last_undegraded":"2024-05-09T08:33:20.931844+0000","last_fullsized":"2024-05-09T08:33:20.931844+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:15:10.322529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836117+0000","last_change":"2024-05-09T08:31:00.696023+0000","last_active":"2024-05-09T08:33:20.836117+0000","last_peered":"2024-05-09T08:33:20.836117+0000","last_clean":"2024-05-09T08:33:20.836117+0000","last_became_active":"2024-05-09T08:31:00.695677+0000","last_became_peered":"2024-05-09T08:31:00.695677+0000","last_unstale":"2024-05-09T08:33:20.836117+0000","last_undegraded":"2024-05-09T08:33:20.836117+0000","last_fullsized":"2024-05-09T08:33:20.836117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:55.989133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077506+0000","last_change":"2024-05-09T08:31:04.859302+0000","last_active":"2024-05-09T08:33:21.077506+0000","last_peered":"2024-05-09T08:33:21.077506+0000","last_clean":"2024-05-09T08:33:21.077506+0000","last_became_active":"2024-05-09T08:31:04.858897+0000","last_became_peered":"2024-05-09T08:31:04.858897+0000","last_unstale":"2024-05-09T08:33:21.077506+0000","last_undegraded":"2024-05-09T08:33:21.077506+0000","last_fullsized":"2024-05-09T08:33:21.077506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:25:32.970833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836421+0000","last_change":"2024-05-09T08:30:59.824451+0000","last_active":"2024-05-09T08:33:20.836421+0000","last_peered":"2024-05-09T08:33:20.836421+0000","last_clean":"2024-05-09T08:33:20.836421+0000","last_became_active":"2024-05-09T08:30:59.823964+0000","last_became_peered":"2024-05-09T08:30:59.823964+0000","last_unstale":"2024-05-09T08:33:20.836421+0000","last_undegraded":"2024-05-09T08:33:20.836421+0000","last_fullsized":"2024-05-09T08:33:20.836421+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:50:13.562262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225113+0000","last_change":"2024-05-09T08:31:00.701015+0000","last_active":"2024-05-09T08:33:21.225113+0000","last_peered":"2024-05-09T08:33:21.225113+0000","last_clean":"2024-05-09T08:33:21.225113+0000","last_became_active":"2024-05-09T08:31:00.700654+0000","last_became_peered":"2024-05-09T08:31:00.700654+0000","last_unstale":"2024-05-09T08:33:21.225113+0000","last_undegraded":"2024-05-09T08:33:21.225113+0000","last_fullsized":"2024-05-09T08:33:21.225113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:03.710790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077636+0000","last_change":"2024-05-09T08:31:04.858453+0000","last_active":"2024-05-09T08:33:21.077636+0000","last_peered":"2024-05-09T08:33:21.077636+0000","last_clean":"2024-05-09T08:33:21.077636+0000","last_became_active":"2024-05-09T08:31:04.857974+0000","last_became_peered":"2024-05-09T08:31:04.857974+0000","last_unstale":"2024-05-09T08:33:21.077636+0000","last_undegraded":"2024-05-09T08:33:21.077636+0000","last_fullsized":"2024-05-09T08:33:21.077636+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:42:52.868625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.934212+0000","last_change":"2024-05-09T08:30:59.825974+0000","last_active":"2024-05-09T08:33:20.934212+0000","last_peered":"2024-05-09T08:33:20.934212+0000","last_clean":"2024-05-09T08:33:20.934212+0000","last_became_active":"2024-05-09T08:30:59.825671+0000","last_became_peered":"2024-05-09T08:30:59.825671+0000","last_unstale":"2024-05-09T08:33:20.934212+0000","last_undegraded":"2024-05-09T08:33:20.934212+0000","last_fullsized":"2024-05-09T08:33:20.934212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:38:20.179466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.924124+0000","last_change":"2024-05-09T08:31:00.698324+0000","last_active":"2024-05-09T08:33:20.924124+0000","last_peered":"2024-05-09T08:33:20.924124+0000","last_clean":"2024-05-09T08:33:20.924124+0000","last_became_active":"2024-05-09T08:31:00.697992+0000","last_became_peered":"2024-05-09T08:31:00.697992+0000","last_unstale":"2024-05-09T08:33:20.924124+0000","last_undegraded":"2024-05-09T08:33:20.924124+0000","last_fullsized":"2024-05-09T08:33:20.924124+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:25:20.786423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077543+0000","last_change":"2024-05-09T08:31:04.861946+0000","last_active":"2024-05-09T08:33:21.077543+0000","last_peered":"2024-05-09T08:33:21.077543+0000","last_clean":"2024-05-09T08:33:21.077543+0000","last_became_active":"2024-05-09T08:31:04.861511+0000","last_became_peered":"2024-05-09T08:31:04.861511+0000","last_unstale":"2024-05-09T08:33:21.077543+0000","last_undegraded":"2024-05-09T08:33:21.077543+0000","last_fullsized":"2024-05-09T08:33:21.077543+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:36:44.580499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933976+0000","last_change":"2024-05-09T08:30:59.829961+0000","last_active":"2024-05-09T08:33:20.933976+0000","last_peered":"2024-05-09T08:33:20.933976+0000","last_clean":"2024-05-09T08:33:20.933976+0000","last_became_active":"2024-05-09T08:30:59.828057+0000","last_became_peered":"2024-05-09T08:30:59.828057+0000","last_unstale":"2024-05-09T08:33:20.933976+0000","last_undegraded":"2024-05-09T08:33:20.933976+0000","last_fullsized":"2024-05-09T08:33:20.933976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:39:18.229438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940848+0000","last_change":"2024-05-09T08:31:00.700285+0000","last_active":"2024-05-09T08:33:20.940848+0000","last_peered":"2024-05-09T08:33:20.940848+0000","last_clean":"2024-05-09T08:33:20.940848+0000","last_became_active":"2024-05-09T08:31:00.699958+0000","last_became_peered":"2024-05-09T08:31:00.699958+0000","last_unstale":"2024-05-09T08:33:20.940848+0000","last_undegraded":"2024-05-09T08:33:20.940848+0000","last_fullsized":"2024-05-09T08:33:20.940848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:48.934827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835038+0000","last_change":"2024-05-09T08:31:04.857060+0000","last_active":"2024-05-09T08:33:20.835038+0000","last_peered":"2024-05-09T08:33:20.835038+0000","last_clean":"2024-05-09T08:33:20.835038+0000","last_became_active":"2024-05-09T08:31:04.855261+0000","last_became_peered":"2024-05-09T08:31:04.855261+0000","last_unstale":"2024-05-09T08:33:20.835038+0000","last_undegraded":"2024-05-09T08:33:20.835038+0000","last_fullsized":"2024-05-09T08:33:20.835038+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:08:08.230524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'8","reported_seq":166,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.237104+0000","last_change":"2024-05-09T08:30:59.831089+0000","last_active":"2024-05-09T08:33:21.237104+0000","last_peered":"2024-05-09T08:33:21.237104+0000","last_clean":"2024-05-09T08:33:21.237104+0000","last_became_active":"2024-05-09T08:30:59.830572+0000","last_became_peered":"2024-05-09T08:30:59.830572+0000","last_unstale":"2024-05-09T08:33:21.237104+0000","last_undegraded":"2024-05-09T08:33:21.237104+0000","last_fullsized":"2024-05-09T08:33:21.237104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:39:10.495104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1897,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933830+0000","last_change":"2024-05-09T08:31:00.688792+0000","last_active":"2024-05-09T08:33:20.933830+0000","last_peered":"2024-05-09T08:33:20.933830+0000","last_clean":"2024-05-09T08:33:20.933830+0000","last_became_active":"2024-05-09T08:31:00.688386+0000","last_became_peered":"2024-05-09T08:31:00.688386+0000","last_unstale":"2024-05-09T08:33:20.933830+0000","last_undegraded":"2024-05-09T08:33:20.933830+0000","last_fullsized":"2024-05-09T08:33:20.933830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:24.287333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834903+0000","last_change":"2024-05-09T08:31:04.862287+0000","last_active":"2024-05-09T08:33:20.834903+0000","last_peered":"2024-05-09T08:33:20.834903+0000","last_clean":"2024-05-09T08:33:20.834903+0000","last_became_active":"2024-05-09T08:31:04.861866+0000","last_became_peered":"2024-05-09T08:31:04.861866+0000","last_unstale":"2024-05-09T08:33:20.834903+0000","last_undegraded":"2024-05-09T08:33:20.834903+0000","last_fullsized":"2024-05-09T08:33:20.834903+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:14:21.552787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835100+0000","last_change":"2024-05-09T08:30:59.824857+0000","last_active":"2024-05-09T08:33:20.835100+0000","last_peered":"2024-05-09T08:33:20.835100+0000","last_clean":"2024-05-09T08:33:20.835100+0000","last_became_active":"2024-05-09T08:30:59.824363+0000","last_became_peered":"2024-05-09T08:30:59.824363+0000","last_unstale":"2024-05-09T08:33:20.835100+0000","last_undegraded":"2024-05-09T08:33:20.835100+0000","last_fullsized":"2024-05-09T08:33:20.835100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:20:32.267237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940931+0000","last_change":"2024-05-09T08:31:00.694440+0000","last_active":"2024-05-09T08:33:20.940931+0000","last_peered":"2024-05-09T08:33:20.940931+0000","last_clean":"2024-05-09T08:33:20.940931+0000","last_became_active":"2024-05-09T08:31:00.694085+0000","last_became_peered":"2024-05-09T08:31:00.694085+0000","last_unstale":"2024-05-09T08:33:20.940931+0000","last_undegraded":"2024-05-09T08:33:20.940931+0000","last_fullsized":"2024-05-09T08:33:20.940931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:03:24.697479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224410+0000","last_change":"2024-05-09T08:31:56.574514+0000","last_active":"2024-05-09T08:33:21.224410+0000","last_peered":"2024-05-09T08:33:21.224410+0000","last_clean":"2024-05-09T08:33:21.224410+0000","last_became_active":"2024-05-09T08:31:56.573786+0000","last_became_peered":"2024-05-09T08:31:56.573786+0000","last_unstale":"2024-05-09T08:33:21.224410+0000","last_undegraded":"2024-05-09T08:33:21.224410+0000","last_fullsized":"2024-05-09T08:33:21.224410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:17.626431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224705+0000","last_change":"2024-05-09T08:30:59.818373+0000","last_active":"2024-05-09T08:33:21.224705+0000","last_peered":"2024-05-09T08:33:21.224705+0000","last_clean":"2024-05-09T08:33:21.224705+0000","last_became_active":"2024-05-09T08:30:59.818023+0000","last_became_peered":"2024-05-09T08:30:59.818023+0000","last_unstale":"2024-05-09T08:33:21.224705+0000","last_undegraded":"2024-05-09T08:33:21.224705+0000","last_fullsized":"2024-05-09T08:33:21.224705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:23:01.994257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835918+0000","last_change":"2024-05-09T08:31:00.689506+0000","last_active":"2024-05-09T08:33:20.835918+0000","last_peered":"2024-05-09T08:33:20.835918+0000","last_clean":"2024-05-09T08:33:20.835918+0000","last_became_active":"2024-05-09T08:31:00.689151+0000","last_became_peered":"2024-05-09T08:31:00.689151+0000","last_unstale":"2024-05-09T08:33:20.835918+0000","last_undegraded":"2024-05-09T08:33:20.835918+0000","last_fullsized":"2024-05-09T08:33:20.835918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:48:13.539509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224607+0000","last_change":"2024-05-09T08:31:04.859215+0000","last_active":"2024-05-09T08:33:21.224607+0000","last_peered":"2024-05-09T08:33:21.224607+0000","last_clean":"2024-05-09T08:33:21.224607+0000","last_became_active":"2024-05-09T08:31:04.858759+0000","last_became_peered":"2024-05-09T08:31:04.858759+0000","last_unstale":"2024-05-09T08:33:21.224607+0000","last_undegraded":"2024-05-09T08:33:21.224607+0000","last_fullsized":"2024-05-09T08:33:21.224607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:41:42.124234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940549+0000","last_change":"2024-05-09T08:30:59.824260+0000","last_active":"2024-05-09T08:33:20.940549+0000","last_peered":"2024-05-09T08:33:20.940549+0000","last_clean":"2024-05-09T08:33:20.940549+0000","last_became_active":"2024-05-09T08:30:59.823959+0000","last_became_peered":"2024-05-09T08:30:59.823959+0000","last_unstale":"2024-05-09T08:33:20.940549+0000","last_undegraded":"2024-05-09T08:33:20.940549+0000","last_fullsized":"2024-05-09T08:33:20.940549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:48:42.354279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931182+0000","last_change":"2024-05-09T08:31:00.687810+0000","last_active":"2024-05-09T08:33:20.931182+0000","last_peered":"2024-05-09T08:33:20.931182+0000","last_clean":"2024-05-09T08:33:20.931182+0000","last_became_active":"2024-05-09T08:31:00.687486+0000","last_became_peered":"2024-05-09T08:31:00.687486+0000","last_unstale":"2024-05-09T08:33:20.931182+0000","last_undegraded":"2024-05-09T08:33:20.931182+0000","last_fullsized":"2024-05-09T08:33:20.931182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:51.428208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835883+0000","last_change":"2024-05-09T08:31:04.853222+0000","last_active":"2024-05-09T08:33:20.835883+0000","last_peered":"2024-05-09T08:33:20.835883+0000","last_clean":"2024-05-09T08:33:20.835883+0000","last_became_active":"2024-05-09T08:31:04.852676+0000","last_became_peered":"2024-05-09T08:31:04.852676+0000","last_unstale":"2024-05-09T08:33:20.835883+0000","last_undegraded":"2024-05-09T08:33:20.835883+0000","last_fullsized":"2024-05-09T08:33:20.835883+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:58:31.949938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":158,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835698+0000","last_change":"2024-05-09T08:30:59.824653+0000","last_active":"2024-05-09T08:33:20.835698+0000","last_peered":"2024-05-09T08:33:20.835698+0000","last_clean":"2024-05-09T08:33:20.835698+0000","last_became_active":"2024-05-09T08:30:59.824277+0000","last_became_peered":"2024-05-09T08:30:59.824277+0000","last_unstale":"2024-05-09T08:33:20.835698+0000","last_undegraded":"2024-05-09T08:33:20.835698+0000","last_fullsized":"2024-05-09T08:33:20.835698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:45:20.340629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.924392+0000","last_change":"2024-05-09T08:31:00.697484+0000","last_active":"2024-05-09T08:33:20.924392+0000","last_peered":"2024-05-09T08:33:20.924392+0000","last_clean":"2024-05-09T08:33:20.924392+0000","last_became_active":"2024-05-09T08:31:00.697095+0000","last_became_peered":"2024-05-09T08:31:00.697095+0000","last_unstale":"2024-05-09T08:33:20.924392+0000","last_undegraded":"2024-05-09T08:33:20.924392+0000","last_fullsized":"2024-05-09T08:33:20.924392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:29:40.702057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933647+0000","last_change":"2024-05-09T08:31:56.574176+0000","last_active":"2024-05-09T08:33:20.933647+0000","last_peered":"2024-05-09T08:33:20.933647+0000","last_clean":"2024-05-09T08:33:20.933647+0000","last_became_active":"2024-05-09T08:31:56.573556+0000","last_became_peered":"2024-05-09T08:31:56.573556+0000","last_unstale":"2024-05-09T08:33:20.933647+0000","last_undegraded":"2024-05-09T08:33:20.933647+0000","last_fullsized":"2024-05-09T08:33:20.933647+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:50.685712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075051+0000","last_change":"2024-05-09T08:30:59.822683+0000","last_active":"2024-05-09T08:33:21.075051+0000","last_peered":"2024-05-09T08:33:21.075051+0000","last_clean":"2024-05-09T08:33:21.075051+0000","last_became_active":"2024-05-09T08:30:59.822150+0000","last_became_peered":"2024-05-09T08:30:59.822150+0000","last_unstale":"2024-05-09T08:33:21.075051+0000","last_undegraded":"2024-05-09T08:33:21.075051+0000","last_fullsized":"2024-05-09T08:33:21.075051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:51:32.782076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224974+0000","last_change":"2024-05-09T08:31:00.694165+0000","last_active":"2024-05-09T08:33:21.224974+0000","last_peered":"2024-05-09T08:33:21.224974+0000","last_clean":"2024-05-09T08:33:21.224974+0000","last_became_active":"2024-05-09T08:31:00.693150+0000","last_became_peered":"2024-05-09T08:31:00.693150+0000","last_unstale":"2024-05-09T08:33:21.224974+0000","last_undegraded":"2024-05-09T08:33:21.224974+0000","last_fullsized":"2024-05-09T08:33:21.224974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:29.278078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077747+0000","last_change":"2024-05-09T08:31:04.858785+0000","last_active":"2024-05-09T08:33:21.077747+0000","last_peered":"2024-05-09T08:33:21.077747+0000","last_clean":"2024-05-09T08:33:21.077747+0000","last_became_active":"2024-05-09T08:31:04.857947+0000","last_became_peered":"2024-05-09T08:31:04.857947+0000","last_unstale":"2024-05-09T08:33:21.077747+0000","last_undegraded":"2024-05-09T08:33:21.077747+0000","last_fullsized":"2024-05-09T08:33:21.077747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:55:48.595607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940992+0000","last_change":"2024-05-09T08:30:59.822946+0000","last_active":"2024-05-09T08:33:20.940992+0000","last_peered":"2024-05-09T08:33:20.940992+0000","last_clean":"2024-05-09T08:33:20.940992+0000","last_became_active":"2024-05-09T08:30:59.822174+0000","last_became_peered":"2024-05-09T08:30:59.822174+0000","last_unstale":"2024-05-09T08:33:20.940992+0000","last_undegraded":"2024-05-09T08:33:20.940992+0000","last_fullsized":"2024-05-09T08:33:20.940992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:14.224033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.924269+0000","last_change":"2024-05-09T08:31:00.704781+0000","last_active":"2024-05-09T08:33:20.924269+0000","last_peered":"2024-05-09T08:33:20.924269+0000","last_clean":"2024-05-09T08:33:20.924269+0000","last_became_active":"2024-05-09T08:31:00.704170+0000","last_became_peered":"2024-05-09T08:31:00.704170+0000","last_unstale":"2024-05-09T08:33:20.924269+0000","last_undegraded":"2024-05-09T08:33:20.924269+0000","last_fullsized":"2024-05-09T08:33:20.924269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:27:47.402290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931199+0000","last_change":"2024-05-09T08:31:04.855648+0000","last_active":"2024-05-09T08:33:20.931199+0000","last_peered":"2024-05-09T08:33:20.931199+0000","last_clean":"2024-05-09T08:33:20.931199+0000","last_became_active":"2024-05-09T08:31:04.854678+0000","last_became_peered":"2024-05-09T08:31:04.854678+0000","last_unstale":"2024-05-09T08:33:20.931199+0000","last_undegraded":"2024-05-09T08:33:20.931199+0000","last_fullsized":"2024-05-09T08:33:20.931199+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:53:46.039172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":327,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.255521+0000","last_change":"2024-05-09T08:30:59.826498+0000","last_active":"2024-05-09T08:33:21.255521+0000","last_peered":"2024-05-09T08:33:21.255521+0000","last_clean":"2024-05-09T08:33:21.255521+0000","last_became_active":"2024-05-09T08:30:59.826050+0000","last_became_peered":"2024-05-09T08:30:59.826050+0000","last_unstale":"2024-05-09T08:33:21.255521+0000","last_undegraded":"2024-05-09T08:33:21.255521+0000","last_fullsized":"2024-05-09T08:33:21.255521+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:19.920901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69194,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":194,"num_write":41,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.934091+0000","last_change":"2024-05-09T08:31:00.688379+0000","last_active":"2024-05-09T08:33:20.934091+0000","last_peered":"2024-05-09T08:33:20.934091+0000","last_clean":"2024-05-09T08:33:20.934091+0000","last_became_active":"2024-05-09T08:31:00.688030+0000","last_became_peered":"2024-05-09T08:31:00.688030+0000","last_unstale":"2024-05-09T08:33:20.934091+0000","last_undegraded":"2024-05-09T08:33:20.934091+0000","last_fullsized":"2024-05-09T08:33:20.934091+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:17.248154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.941177+0000","last_change":"2024-05-09T08:31:04.858513+0000","last_active":"2024-05-09T08:33:20.941177+0000","last_peered":"2024-05-09T08:33:20.941177+0000","last_clean":"2024-05-09T08:33:20.941177+0000","last_became_active":"2024-05-09T08:31:04.857988+0000","last_became_peered":"2024-05-09T08:31:04.857988+0000","last_unstale":"2024-05-09T08:33:20.941177+0000","last_undegraded":"2024-05-09T08:33:20.941177+0000","last_fullsized":"2024-05-09T08:33:20.941177+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:59:24.264503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834912+0000","last_change":"2024-05-09T08:30:59.823403+0000","last_active":"2024-05-09T08:33:20.834912+0000","last_peered":"2024-05-09T08:33:20.834912+0000","last_clean":"2024-05-09T08:33:20.834912+0000","last_became_active":"2024-05-09T08:30:59.823066+0000","last_became_peered":"2024-05-09T08:30:59.823066+0000","last_unstale":"2024-05-09T08:33:20.834912+0000","last_undegraded":"2024-05-09T08:33:20.834912+0000","last_fullsized":"2024-05-09T08:33:20.834912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:16:39.157895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224735+0000","last_change":"2024-05-09T08:31:00.688009+0000","last_active":"2024-05-09T08:33:21.224735+0000","last_peered":"2024-05-09T08:33:21.224735+0000","last_clean":"2024-05-09T08:33:21.224735+0000","last_became_active":"2024-05-09T08:31:00.687679+0000","last_became_peered":"2024-05-09T08:31:00.687679+0000","last_unstale":"2024-05-09T08:33:21.224735+0000","last_undegraded":"2024-05-09T08:33:21.224735+0000","last_fullsized":"2024-05-09T08:33:21.224735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:44.403639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224273+0000","last_change":"2024-05-09T08:31:04.851880+0000","last_active":"2024-05-09T08:33:21.224273+0000","last_peered":"2024-05-09T08:33:21.224273+0000","last_clean":"2024-05-09T08:33:21.224273+0000","last_became_active":"2024-05-09T08:31:04.851316+0000","last_became_peered":"2024-05-09T08:31:04.851316+0000","last_unstale":"2024-05-09T08:33:21.224273+0000","last_undegraded":"2024-05-09T08:33:21.224273+0000","last_fullsized":"2024-05-09T08:33:21.224273+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:53.144996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940735+0000","last_change":"2024-05-09T08:30:59.823411+0000","last_active":"2024-05-09T08:33:20.940735+0000","last_peered":"2024-05-09T08:33:20.940735+0000","last_clean":"2024-05-09T08:33:20.940735+0000","last_became_active":"2024-05-09T08:30:59.822854+0000","last_became_peered":"2024-05-09T08:30:59.822854+0000","last_unstale":"2024-05-09T08:33:20.940735+0000","last_undegraded":"2024-05-09T08:33:20.940735+0000","last_fullsized":"2024-05-09T08:33:20.940735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:36:03.666937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074998+0000","last_change":"2024-05-09T08:31:56.777818+0000","last_active":"2024-05-09T08:33:21.074998+0000","last_peered":"2024-05-09T08:33:21.074998+0000","last_clean":"2024-05-09T08:33:21.074998+0000","last_became_active":"2024-05-09T08:31:56.777555+0000","last_became_peered":"2024-05-09T08:31:56.777555+0000","last_unstale":"2024-05-09T08:33:21.074998+0000","last_undegraded":"2024-05-09T08:33:21.074998+0000","last_fullsized":"2024-05-09T08:33:21.074998+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:16:17.695073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931723+0000","last_change":"2024-05-09T08:31:04.859338+0000","last_active":"2024-05-09T08:33:20.931723+0000","last_peered":"2024-05-09T08:33:20.931723+0000","last_clean":"2024-05-09T08:33:20.931723+0000","last_became_active":"2024-05-09T08:31:04.858951+0000","last_became_peered":"2024-05-09T08:31:04.858951+0000","last_unstale":"2024-05-09T08:33:20.931723+0000","last_undegraded":"2024-05-09T08:33:20.931723+0000","last_fullsized":"2024-05-09T08:33:20.931723+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:26:46.738359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'8","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.236886+0000","last_change":"2024-05-09T08:30:59.823710+0000","last_active":"2024-05-09T08:33:21.236886+0000","last_peered":"2024-05-09T08:33:21.236886+0000","last_clean":"2024-05-09T08:33:21.236886+0000","last_became_active":"2024-05-09T08:30:59.823213+0000","last_became_peered":"2024-05-09T08:30:59.823213+0000","last_unstale":"2024-05-09T08:33:21.236886+0000","last_undegraded":"2024-05-09T08:33:21.236886+0000","last_fullsized":"2024-05-09T08:33:21.236886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:09.134942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":128,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074753+0000","last_change":"2024-05-09T08:31:00.693735+0000","last_active":"2024-05-09T08:33:21.074753+0000","last_peered":"2024-05-09T08:33:21.074753+0000","last_clean":"2024-05-09T08:33:21.074753+0000","last_became_active":"2024-05-09T08:31:00.693401+0000","last_became_peered":"2024-05-09T08:31:00.693401+0000","last_unstale":"2024-05-09T08:33:21.074753+0000","last_undegraded":"2024-05-09T08:33:21.074753+0000","last_fullsized":"2024-05-09T08:33:21.074753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:58:41.429854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225625+0000","last_change":"2024-05-09T08:31:04.861289+0000","last_active":"2024-05-09T08:33:21.225625+0000","last_peered":"2024-05-09T08:33:21.225625+0000","last_clean":"2024-05-09T08:33:21.225625+0000","last_became_active":"2024-05-09T08:31:04.860915+0000","last_became_peered":"2024-05-09T08:31:04.860915+0000","last_unstale":"2024-05-09T08:33:21.225625+0000","last_undegraded":"2024-05-09T08:33:21.225625+0000","last_fullsized":"2024-05-09T08:33:21.225625+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:01:45.724961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933721+0000","last_change":"2024-05-09T08:30:59.827631+0000","last_active":"2024-05-09T08:33:20.933721+0000","last_peered":"2024-05-09T08:33:20.933721+0000","last_clean":"2024-05-09T08:33:20.933721+0000","last_became_active":"2024-05-09T08:30:59.827295+0000","last_became_peered":"2024-05-09T08:30:59.827295+0000","last_unstale":"2024-05-09T08:33:20.933721+0000","last_undegraded":"2024-05-09T08:33:20.933721+0000","last_fullsized":"2024-05-09T08:33:20.933721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:35.282564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940612+0000","last_change":"2024-05-09T08:31:00.697712+0000","last_active":"2024-05-09T08:33:20.940612+0000","last_peered":"2024-05-09T08:33:20.940612+0000","last_clean":"2024-05-09T08:33:20.940612+0000","last_became_active":"2024-05-09T08:31:00.697385+0000","last_became_peered":"2024-05-09T08:31:00.697385+0000","last_unstale":"2024-05-09T08:33:20.940612+0000","last_undegraded":"2024-05-09T08:33:20.940612+0000","last_fullsized":"2024-05-09T08:33:20.940612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:09.087279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940816+0000","last_change":"2024-05-09T08:31:04.860691+0000","last_active":"2024-05-09T08:33:20.940816+0000","last_peered":"2024-05-09T08:33:20.940816+0000","last_clean":"2024-05-09T08:33:20.940816+0000","last_became_active":"2024-05-09T08:31:04.858458+0000","last_became_peered":"2024-05-09T08:31:04.858458+0000","last_unstale":"2024-05-09T08:33:20.940816+0000","last_undegraded":"2024-05-09T08:33:20.940816+0000","last_fullsized":"2024-05-09T08:33:20.940816+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:16.274359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.934269+0000","last_change":"2024-05-09T08:30:59.825188+0000","last_active":"2024-05-09T08:33:20.934269+0000","last_peered":"2024-05-09T08:33:20.934269+0000","last_clean":"2024-05-09T08:33:20.934269+0000","last_became_active":"2024-05-09T08:30:59.824855+0000","last_became_peered":"2024-05-09T08:30:59.824855+0000","last_unstale":"2024-05-09T08:33:20.934269+0000","last_undegraded":"2024-05-09T08:33:20.934269+0000","last_fullsized":"2024-05-09T08:33:20.934269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:17:21.753652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224274+0000","last_change":"2024-05-09T08:31:00.695862+0000","last_active":"2024-05-09T08:33:21.224274+0000","last_peered":"2024-05-09T08:33:21.224274+0000","last_clean":"2024-05-09T08:33:21.224274+0000","last_became_active":"2024-05-09T08:31:00.695404+0000","last_became_peered":"2024-05-09T08:31:00.695404+0000","last_unstale":"2024-05-09T08:33:21.224274+0000","last_undegraded":"2024-05-09T08:33:21.224274+0000","last_fullsized":"2024-05-09T08:33:21.224274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:53:07.567583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933822+0000","last_change":"2024-05-09T08:31:04.856322+0000","last_active":"2024-05-09T08:33:20.933822+0000","last_peered":"2024-05-09T08:33:20.933822+0000","last_clean":"2024-05-09T08:33:20.933822+0000","last_became_active":"2024-05-09T08:31:04.855821+0000","last_became_peered":"2024-05-09T08:31:04.855821+0000","last_unstale":"2024-05-09T08:33:20.933822+0000","last_undegraded":"2024-05-09T08:33:20.933822+0000","last_fullsized":"2024-05-09T08:33:20.933822+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:29:18.594928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931676+0000","last_change":"2024-05-09T08:31:56.574061+0000","last_active":"2024-05-09T08:33:20.931676+0000","last_peered":"2024-05-09T08:33:20.931676+0000","last_clean":"2024-05-09T08:33:20.931676+0000","last_became_active":"2024-05-09T08:31:56.573673+0000","last_became_peered":"2024-05-09T08:31:56.573673+0000","last_unstale":"2024-05-09T08:33:20.931676+0000","last_undegraded":"2024-05-09T08:33:20.931676+0000","last_fullsized":"2024-05-09T08:33:20.931676+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:12:13.699915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835535+0000","last_change":"2024-05-09T08:31:00.684523+0000","last_active":"2024-05-09T08:33:20.835535+0000","last_peered":"2024-05-09T08:33:20.835535+0000","last_clean":"2024-05-09T08:33:20.835535+0000","last_became_active":"2024-05-09T08:31:00.684153+0000","last_became_peered":"2024-05-09T08:31:00.684153+0000","last_unstale":"2024-05-09T08:33:20.835535+0000","last_undegraded":"2024-05-09T08:33:20.835535+0000","last_fullsized":"2024-05-09T08:33:20.835535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:02.423401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225194+0000","last_change":"2024-05-09T08:31:04.857961+0000","last_active":"2024-05-09T08:33:21.225194+0000","last_peered":"2024-05-09T08:33:21.225194+0000","last_clean":"2024-05-09T08:33:21.225194+0000","last_became_active":"2024-05-09T08:31:04.857543+0000","last_became_peered":"2024-05-09T08:31:04.857543+0000","last_unstale":"2024-05-09T08:33:21.225194+0000","last_undegraded":"2024-05-09T08:33:21.225194+0000","last_fullsized":"2024-05-09T08:33:21.225194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:22:30.820297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074717+0000","last_change":"2024-05-09T08:30:59.827878+0000","last_active":"2024-05-09T08:33:21.074717+0000","last_peered":"2024-05-09T08:33:21.074717+0000","last_clean":"2024-05-09T08:33:21.074717+0000","last_became_active":"2024-05-09T08:30:59.827487+0000","last_became_peered":"2024-05-09T08:30:59.827487+0000","last_unstale":"2024-05-09T08:33:21.074717+0000","last_undegraded":"2024-05-09T08:33:21.074717+0000","last_fullsized":"2024-05-09T08:33:21.074717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:34.465457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835244+0000","last_change":"2024-05-09T08:31:00.695617+0000","last_active":"2024-05-09T08:33:20.835244+0000","last_peered":"2024-05-09T08:33:20.835244+0000","last_clean":"2024-05-09T08:33:20.835244+0000","last_became_active":"2024-05-09T08:31:00.695293+0000","last_became_peered":"2024-05-09T08:31:00.695293+0000","last_unstale":"2024-05-09T08:33:20.835244+0000","last_undegraded":"2024-05-09T08:33:20.835244+0000","last_fullsized":"2024-05-09T08:33:20.835244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:44:03.958559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225556+0000","last_change":"2024-05-09T08:31:04.860184+0000","last_active":"2024-05-09T08:33:21.225556+0000","last_peered":"2024-05-09T08:33:21.225556+0000","last_clean":"2024-05-09T08:33:21.225556+0000","last_became_active":"2024-05-09T08:31:04.859730+0000","last_became_peered":"2024-05-09T08:31:04.859730+0000","last_unstale":"2024-05-09T08:33:21.225556+0000","last_undegraded":"2024-05-09T08:33:21.225556+0000","last_fullsized":"2024-05-09T08:33:21.225556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:13:17.292678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940687+0000","last_change":"2024-05-09T08:31:00.694183+0000","last_active":"2024-05-09T08:33:20.940687+0000","last_peered":"2024-05-09T08:33:20.940687+0000","last_clean":"2024-05-09T08:33:20.940687+0000","last_became_active":"2024-05-09T08:31:00.693819+0000","last_became_peered":"2024-05-09T08:31:00.693819+0000","last_unstale":"2024-05-09T08:33:20.940687+0000","last_undegraded":"2024-05-09T08:33:20.940687+0000","last_fullsized":"2024-05-09T08:33:20.940687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:09:26.255663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224626+0000","last_change":"2024-05-09T08:30:59.830758+0000","last_active":"2024-05-09T08:33:21.224626+0000","last_peered":"2024-05-09T08:33:21.224626+0000","last_clean":"2024-05-09T08:33:21.224626+0000","last_became_active":"2024-05-09T08:30:59.830430+0000","last_became_peered":"2024-05-09T08:30:59.830430+0000","last_unstale":"2024-05-09T08:33:21.224626+0000","last_undegraded":"2024-05-09T08:33:21.224626+0000","last_fullsized":"2024-05-09T08:33:21.224626+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:24:05.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075024+0000","last_change":"2024-05-09T08:31:56.675063+0000","last_active":"2024-05-09T08:33:21.075024+0000","last_peered":"2024-05-09T08:33:21.075024+0000","last_clean":"2024-05-09T08:33:21.075024+0000","last_became_active":"2024-05-09T08:31:56.670995+0000","last_became_peered":"2024-05-09T08:31:56.670995+0000","last_unstale":"2024-05-09T08:33:21.075024+0000","last_undegraded":"2024-05-09T08:33:21.075024+0000","last_fullsized":"2024-05-09T08:33:21.075024+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:57:52.985420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835992+0000","last_change":"2024-05-09T08:31:00.692197+0000","last_active":"2024-05-09T08:33:20.835992+0000","last_peered":"2024-05-09T08:33:20.835992+0000","last_clean":"2024-05-09T08:33:20.835992+0000","last_became_active":"2024-05-09T08:31:00.691348+0000","last_became_peered":"2024-05-09T08:31:00.691348+0000","last_unstale":"2024-05-09T08:33:20.835992+0000","last_undegraded":"2024-05-09T08:33:20.835992+0000","last_fullsized":"2024-05-09T08:33:20.835992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:00:45.826433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077650+0000","last_change":"2024-05-09T08:30:59.825341+0000","last_active":"2024-05-09T08:33:21.077650+0000","last_peered":"2024-05-09T08:33:21.077650+0000","last_clean":"2024-05-09T08:33:21.077650+0000","last_became_active":"2024-05-09T08:30:59.824860+0000","last_became_peered":"2024-05-09T08:30:59.824860+0000","last_unstale":"2024-05-09T08:33:21.077650+0000","last_undegraded":"2024-05-09T08:33:21.077650+0000","last_fullsized":"2024-05-09T08:33:21.077650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:13.868040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":365,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074596+0000","last_change":"2024-05-09T08:29:47.816504+0000","last_active":"2024-05-09T08:33:21.074596+0000","last_peered":"2024-05-09T08:33:21.074596+0000","last_clean":"2024-05-09T08:33:21.074596+0000","last_became_active":"2024-05-09T08:29:47.816050+0000","last_became_peered":"2024-05-09T08:29:47.816050+0000","last_unstale":"2024-05-09T08:33:21.074596+0000","last_undegraded":"2024-05-09T08:33:21.074596+0000","last_fullsized":"2024-05-09T08:33:21.074596+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:26:22.135491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:26:22.135491+0000","last_clean_scrub_stamp":"2024-05-09T08:26:22.135491+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:28.543694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940955+0000","last_change":"2024-05-09T08:31:04.860417+0000","last_active":"2024-05-09T08:33:20.940955+0000","last_peered":"2024-05-09T08:33:20.940955+0000","last_clean":"2024-05-09T08:33:20.940955+0000","last_became_active":"2024-05-09T08:31:04.859040+0000","last_became_peered":"2024-05-09T08:31:04.859040+0000","last_unstale":"2024-05-09T08:33:20.940955+0000","last_undegraded":"2024-05-09T08:33:20.940955+0000","last_fullsized":"2024-05-09T08:33:20.940955+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:12:04.157354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224420+0000","last_change":"2024-05-09T08:31:00.703131+0000","last_active":"2024-05-09T08:33:21.224420+0000","last_peered":"2024-05-09T08:33:21.224420+0000","last_clean":"2024-05-09T08:33:21.224420+0000","last_became_active":"2024-05-09T08:31:00.702817+0000","last_became_peered":"2024-05-09T08:31:00.702817+0000","last_unstale":"2024-05-09T08:33:21.224420+0000","last_undegraded":"2024-05-09T08:33:21.224420+0000","last_fullsized":"2024-05-09T08:33:21.224420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:31.059334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074669+0000","last_change":"2024-05-09T08:30:59.829636+0000","last_active":"2024-05-09T08:33:21.074669+0000","last_peered":"2024-05-09T08:33:21.074669+0000","last_clean":"2024-05-09T08:33:21.074669+0000","last_became_active":"2024-05-09T08:30:59.829276+0000","last_became_peered":"2024-05-09T08:30:59.829276+0000","last_unstale":"2024-05-09T08:33:21.074669+0000","last_undegraded":"2024-05-09T08:33:21.074669+0000","last_fullsized":"2024-05-09T08:33:21.074669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:17.847258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224095+0000","last_change":"2024-05-09T08:31:04.853945+0000","last_active":"2024-05-09T08:33:21.224095+0000","last_peered":"2024-05-09T08:33:21.224095+0000","last_clean":"2024-05-09T08:33:21.224095+0000","last_became_active":"2024-05-09T08:31:04.853417+0000","last_became_peered":"2024-05-09T08:31:04.853417+0000","last_unstale":"2024-05-09T08:33:21.224095+0000","last_undegraded":"2024-05-09T08:33:21.224095+0000","last_fullsized":"2024-05-09T08:33:21.224095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:21:31.126562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224172+0000","last_change":"2024-05-09T08:32:57.561844+0000","last_active":"2024-05-09T08:33:21.224172+0000","last_peered":"2024-05-09T08:33:21.224172+0000","last_clean":"2024-05-09T08:33:21.224172+0000","last_became_active":"2024-05-09T08:32:57.559144+0000","last_became_peered":"2024-05-09T08:32:57.559144+0000","last_unstale":"2024-05-09T08:33:21.224172+0000","last_undegraded":"2024-05-09T08:33:21.224172+0000","last_fullsized":"2024-05-09T08:33:21.224172+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:01.772256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225310+0000","last_change":"2024-05-09T08:31:00.703478+0000","last_active":"2024-05-09T08:33:21.225310+0000","last_peered":"2024-05-09T08:33:21.225310+0000","last_clean":"2024-05-09T08:33:21.225310+0000","last_became_active":"2024-05-09T08:31:00.702852+0000","last_became_peered":"2024-05-09T08:31:00.702852+0000","last_unstale":"2024-05-09T08:33:21.225310+0000","last_undegraded":"2024-05-09T08:33:21.225310+0000","last_fullsized":"2024-05-09T08:33:21.225310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:12.635341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835899+0000","last_change":"2024-05-09T08:31:04.860801+0000","last_active":"2024-05-09T08:33:20.835899+0000","last_peered":"2024-05-09T08:33:20.835899+0000","last_clean":"2024-05-09T08:33:20.835899+0000","last_became_active":"2024-05-09T08:31:04.857387+0000","last_became_peered":"2024-05-09T08:31:04.857387+0000","last_unstale":"2024-05-09T08:33:20.835899+0000","last_undegraded":"2024-05-09T08:33:20.835899+0000","last_fullsized":"2024-05-09T08:33:20.835899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:50:16.555987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'45","reported_seq":102,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.044579+0000","last_change":"2024-05-09T08:30:59.827082+0000","last_active":"2024-05-09T08:33:21.044579+0000","last_peered":"2024-05-09T08:33:21.044579+0000","last_clean":"2024-05-09T08:33:21.044579+0000","last_became_active":"2024-05-09T08:30:59.826746+0000","last_became_peered":"2024-05-09T08:30:59.826746+0000","last_unstale":"2024-05-09T08:33:21.044579+0000","last_undegraded":"2024-05-09T08:33:21.044579+0000","last_fullsized":"2024-05-09T08:33:21.044579+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:49.176889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836291+0000","last_change":"2024-05-09T08:31:00.691902+0000","last_active":"2024-05-09T08:33:20.836291+0000","last_peered":"2024-05-09T08:33:20.836291+0000","last_clean":"2024-05-09T08:33:20.836291+0000","last_became_active":"2024-05-09T08:31:00.689491+0000","last_became_peered":"2024-05-09T08:31:00.689491+0000","last_unstale":"2024-05-09T08:33:20.836291+0000","last_undegraded":"2024-05-09T08:33:20.836291+0000","last_fullsized":"2024-05-09T08:33:20.836291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:48:07.335144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834367+0000","last_change":"2024-05-09T08:31:04.857788+0000","last_active":"2024-05-09T08:33:20.834367+0000","last_peered":"2024-05-09T08:33:20.834367+0000","last_clean":"2024-05-09T08:33:20.834367+0000","last_became_active":"2024-05-09T08:31:04.853831+0000","last_became_peered":"2024-05-09T08:31:04.853831+0000","last_unstale":"2024-05-09T08:33:20.834367+0000","last_undegraded":"2024-05-09T08:33:20.834367+0000","last_fullsized":"2024-05-09T08:33:20.834367+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:08:15.451606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.078037+0000","last_change":"2024-05-09T08:30:59.826482+0000","last_active":"2024-05-09T08:33:21.078037+0000","last_peered":"2024-05-09T08:33:21.078037+0000","last_clean":"2024-05-09T08:33:21.078037+0000","last_became_active":"2024-05-09T08:30:59.826123+0000","last_became_peered":"2024-05-09T08:30:59.826123+0000","last_unstale":"2024-05-09T08:33:21.078037+0000","last_undegraded":"2024-05-09T08:33:21.078037+0000","last_fullsized":"2024-05-09T08:33:21.078037+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:38.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224830+0000","last_change":"2024-05-09T08:31:00.693440+0000","last_active":"2024-05-09T08:33:21.224830+0000","last_peered":"2024-05-09T08:33:21.224830+0000","last_clean":"2024-05-09T08:33:21.224830+0000","last_became_active":"2024-05-09T08:31:00.693117+0000","last_became_peered":"2024-05-09T08:31:00.693117+0000","last_unstale":"2024-05-09T08:33:21.224830+0000","last_undegraded":"2024-05-09T08:33:21.224830+0000","last_fullsized":"2024-05-09T08:33:21.224830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:10:01.205605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077861+0000","last_change":"2024-05-09T08:31:04.868120+0000","last_active":"2024-05-09T08:33:21.077861+0000","last_peered":"2024-05-09T08:33:21.077861+0000","last_clean":"2024-05-09T08:33:21.077861+0000","last_became_active":"2024-05-09T08:31:04.867571+0000","last_became_peered":"2024-05-09T08:31:04.867571+0000","last_unstale":"2024-05-09T08:33:21.077861+0000","last_undegraded":"2024-05-09T08:33:21.077861+0000","last_fullsized":"2024-05-09T08:33:21.077861+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:12:36.503972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.923833+0000","last_change":"2024-05-09T08:30:59.826280+0000","last_active":"2024-05-09T08:33:20.923833+0000","last_peered":"2024-05-09T08:33:20.923833+0000","last_clean":"2024-05-09T08:33:20.923833+0000","last_became_active":"2024-05-09T08:30:59.825805+0000","last_became_peered":"2024-05-09T08:30:59.825805+0000","last_unstale":"2024-05-09T08:33:20.923833+0000","last_undegraded":"2024-05-09T08:33:20.923833+0000","last_fullsized":"2024-05-09T08:33:20.923833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:46:03.833159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933725+0000","last_change":"2024-05-09T08:31:00.687581+0000","last_active":"2024-05-09T08:33:20.933725+0000","last_peered":"2024-05-09T08:33:20.933725+0000","last_clean":"2024-05-09T08:33:20.933725+0000","last_became_active":"2024-05-09T08:31:00.684608+0000","last_became_peered":"2024-05-09T08:31:00.684608+0000","last_unstale":"2024-05-09T08:33:20.933725+0000","last_undegraded":"2024-05-09T08:33:20.933725+0000","last_fullsized":"2024-05-09T08:33:20.933725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:06:06.436840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931830+0000","last_change":"2024-05-09T08:31:04.859399+0000","last_active":"2024-05-09T08:33:20.931830+0000","last_peered":"2024-05-09T08:33:20.931830+0000","last_clean":"2024-05-09T08:33:20.931830+0000","last_became_active":"2024-05-09T08:31:04.859027+0000","last_became_peered":"2024-05-09T08:31:04.859027+0000","last_unstale":"2024-05-09T08:33:20.931830+0000","last_undegraded":"2024-05-09T08:33:20.931830+0000","last_fullsized":"2024-05-09T08:33:20.931830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:18:00.829342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835967+0000","last_change":"2024-05-09T08:30:59.824171+0000","last_active":"2024-05-09T08:33:20.835967+0000","last_peered":"2024-05-09T08:33:20.835967+0000","last_clean":"2024-05-09T08:33:20.835967+0000","last_became_active":"2024-05-09T08:30:59.823859+0000","last_became_peered":"2024-05-09T08:30:59.823859+0000","last_unstale":"2024-05-09T08:33:20.835967+0000","last_undegraded":"2024-05-09T08:33:20.835967+0000","last_fullsized":"2024-05-09T08:33:20.835967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:30.199504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224030+0000","last_change":"2024-05-09T08:31:00.700056+0000","last_active":"2024-05-09T08:33:21.224030+0000","last_peered":"2024-05-09T08:33:21.224030+0000","last_clean":"2024-05-09T08:33:21.224030+0000","last_became_active":"2024-05-09T08:31:00.699673+0000","last_became_peered":"2024-05-09T08:31:00.699673+0000","last_unstale":"2024-05-09T08:33:21.224030+0000","last_undegraded":"2024-05-09T08:33:21.224030+0000","last_fullsized":"2024-05-09T08:33:21.224030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:23.873645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077627+0000","last_change":"2024-05-09T08:31:04.858940+0000","last_active":"2024-05-09T08:33:21.077627+0000","last_peered":"2024-05-09T08:33:21.077627+0000","last_clean":"2024-05-09T08:33:21.077627+0000","last_became_active":"2024-05-09T08:31:04.858532+0000","last_became_peered":"2024-05-09T08:31:04.858532+0000","last_unstale":"2024-05-09T08:33:21.077627+0000","last_undegraded":"2024-05-09T08:33:21.077627+0000","last_fullsized":"2024-05-09T08:33:21.077627+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:57:42.128068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834600+0000","last_change":"2024-05-09T08:30:59.824182+0000","last_active":"2024-05-09T08:33:20.834600+0000","last_peered":"2024-05-09T08:33:20.834600+0000","last_clean":"2024-05-09T08:33:20.834600+0000","last_became_active":"2024-05-09T08:30:59.823691+0000","last_became_peered":"2024-05-09T08:30:59.823691+0000","last_unstale":"2024-05-09T08:33:20.834600+0000","last_undegraded":"2024-05-09T08:33:20.834600+0000","last_fullsized":"2024-05-09T08:33:20.834600+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:28:07.810733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836255+0000","last_change":"2024-05-09T08:31:00.692614+0000","last_active":"2024-05-09T08:33:20.836255+0000","last_peered":"2024-05-09T08:33:20.836255+0000","last_clean":"2024-05-09T08:33:20.836255+0000","last_became_active":"2024-05-09T08:31:00.692280+0000","last_became_peered":"2024-05-09T08:31:00.692280+0000","last_unstale":"2024-05-09T08:33:20.836255+0000","last_undegraded":"2024-05-09T08:33:20.836255+0000","last_fullsized":"2024-05-09T08:33:20.836255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:23:15.398196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077712+0000","last_change":"2024-05-09T08:31:04.854029+0000","last_active":"2024-05-09T08:33:21.077712+0000","last_peered":"2024-05-09T08:33:21.077712+0000","last_clean":"2024-05-09T08:33:21.077712+0000","last_became_active":"2024-05-09T08:31:04.853554+0000","last_became_peered":"2024-05-09T08:31:04.853554+0000","last_unstale":"2024-05-09T08:33:21.077712+0000","last_undegraded":"2024-05-09T08:33:21.077712+0000","last_fullsized":"2024-05-09T08:33:21.077712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:34:41.059614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933921+0000","last_change":"2024-05-09T08:30:59.824512+0000","last_active":"2024-05-09T08:33:20.933921+0000","last_peered":"2024-05-09T08:33:20.933921+0000","last_clean":"2024-05-09T08:33:20.933921+0000","last_became_active":"2024-05-09T08:30:59.824146+0000","last_became_peered":"2024-05-09T08:30:59.824146+0000","last_unstale":"2024-05-09T08:33:20.933921+0000","last_undegraded":"2024-05-09T08:33:20.933921+0000","last_fullsized":"2024-05-09T08:33:20.933921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:27.734292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074854+0000","last_change":"2024-05-09T08:31:00.697272+0000","last_active":"2024-05-09T08:33:21.074854+0000","last_peered":"2024-05-09T08:33:21.074854+0000","last_clean":"2024-05-09T08:33:21.074854+0000","last_became_active":"2024-05-09T08:31:00.696026+0000","last_became_peered":"2024-05-09T08:31:00.696026+0000","last_unstale":"2024-05-09T08:33:21.074854+0000","last_undegraded":"2024-05-09T08:33:21.074854+0000","last_fullsized":"2024-05-09T08:33:21.074854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:04.376142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.923977+0000","last_change":"2024-05-09T08:31:04.856641+0000","last_active":"2024-05-09T08:33:20.923977+0000","last_peered":"2024-05-09T08:33:20.923977+0000","last_clean":"2024-05-09T08:33:20.923977+0000","last_became_active":"2024-05-09T08:31:04.856102+0000","last_became_peered":"2024-05-09T08:31:04.856102+0000","last_unstale":"2024-05-09T08:33:20.923977+0000","last_undegraded":"2024-05-09T08:33:20.923977+0000","last_fullsized":"2024-05-09T08:33:20.923977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:35:49.831076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.923979+0000","last_change":"2024-05-09T08:31:56.579569+0000","last_active":"2024-05-09T08:33:20.923979+0000","last_peered":"2024-05-09T08:33:20.923979+0000","last_clean":"2024-05-09T08:33:20.923979+0000","last_became_active":"2024-05-09T08:31:56.573496+0000","last_became_peered":"2024-05-09T08:31:56.573496+0000","last_unstale":"2024-05-09T08:33:20.923979+0000","last_undegraded":"2024-05-09T08:33:20.923979+0000","last_fullsized":"2024-05-09T08:33:20.923979+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:47:47.944200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075274+0000","last_change":"2024-05-09T08:31:00.697876+0000","last_active":"2024-05-09T08:33:21.075274+0000","last_peered":"2024-05-09T08:33:21.075274+0000","last_clean":"2024-05-09T08:33:21.075274+0000","last_became_active":"2024-05-09T08:31:00.696299+0000","last_became_peered":"2024-05-09T08:31:00.696299+0000","last_unstale":"2024-05-09T08:33:21.075274+0000","last_undegraded":"2024-05-09T08:33:21.075274+0000","last_fullsized":"2024-05-09T08:33:21.075274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:48:57.373734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224675+0000","last_change":"2024-05-09T08:31:04.858588+0000","last_active":"2024-05-09T08:33:21.224675+0000","last_peered":"2024-05-09T08:33:21.224675+0000","last_clean":"2024-05-09T08:33:21.224675+0000","last_became_active":"2024-05-09T08:31:04.858154+0000","last_became_peered":"2024-05-09T08:31:04.858154+0000","last_unstale":"2024-05-09T08:33:21.224675+0000","last_undegraded":"2024-05-09T08:33:21.224675+0000","last_fullsized":"2024-05-09T08:33:21.224675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:06:01.768743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.043305+0000","last_change":"2024-05-09T08:30:59.826066+0000","last_active":"2024-05-09T08:33:21.043305+0000","last_peered":"2024-05-09T08:33:21.043305+0000","last_clean":"2024-05-09T08:33:21.043305+0000","last_became_active":"2024-05-09T08:30:59.825586+0000","last_became_peered":"2024-05-09T08:30:59.825586+0000","last_unstale":"2024-05-09T08:33:21.043305+0000","last_undegraded":"2024-05-09T08:33:21.043305+0000","last_fullsized":"2024-05-09T08:33:21.043305+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:19:04.629132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075338+0000","last_change":"2024-05-09T08:31:00.693788+0000","last_active":"2024-05-09T08:33:21.075338+0000","last_peered":"2024-05-09T08:33:21.075338+0000","last_clean":"2024-05-09T08:33:21.075338+0000","last_became_active":"2024-05-09T08:31:00.693448+0000","last_became_peered":"2024-05-09T08:31:00.693448+0000","last_unstale":"2024-05-09T08:33:21.075338+0000","last_undegraded":"2024-05-09T08:33:21.075338+0000","last_fullsized":"2024-05-09T08:33:21.075338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:06.516534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940422+0000","last_change":"2024-05-09T08:31:04.856185+0000","last_active":"2024-05-09T08:33:20.940422+0000","last_peered":"2024-05-09T08:33:20.940422+0000","last_clean":"2024-05-09T08:33:20.940422+0000","last_became_active":"2024-05-09T08:31:04.855779+0000","last_became_peered":"2024-05-09T08:31:04.855779+0000","last_unstale":"2024-05-09T08:33:20.940422+0000","last_undegraded":"2024-05-09T08:33:20.940422+0000","last_fullsized":"2024-05-09T08:33:20.940422+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:45:25.034077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225735+0000","last_change":"2024-05-09T08:30:59.822938+0000","last_active":"2024-05-09T08:33:21.225735+0000","last_peered":"2024-05-09T08:33:21.225735+0000","last_clean":"2024-05-09T08:33:21.225735+0000","last_became_active":"2024-05-09T08:30:59.822629+0000","last_became_peered":"2024-05-09T08:30:59.822629+0000","last_unstale":"2024-05-09T08:33:21.225735+0000","last_undegraded":"2024-05-09T08:33:21.225735+0000","last_fullsized":"2024-05-09T08:33:21.225735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:51.321350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075213+0000","last_change":"2024-05-09T08:31:00.691971+0000","last_active":"2024-05-09T08:33:21.075213+0000","last_peered":"2024-05-09T08:33:21.075213+0000","last_clean":"2024-05-09T08:33:21.075213+0000","last_became_active":"2024-05-09T08:31:00.691641+0000","last_became_peered":"2024-05-09T08:31:00.691641+0000","last_unstale":"2024-05-09T08:33:21.075213+0000","last_undegraded":"2024-05-09T08:33:21.075213+0000","last_fullsized":"2024-05-09T08:33:21.075213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:28.413886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224253+0000","last_change":"2024-05-09T08:31:00.701026+0000","last_active":"2024-05-09T08:33:21.224253+0000","last_peered":"2024-05-09T08:33:21.224253+0000","last_clean":"2024-05-09T08:33:21.224253+0000","last_became_active":"2024-05-09T08:31:00.700457+0000","last_became_peered":"2024-05-09T08:31:00.700457+0000","last_unstale":"2024-05-09T08:33:21.224253+0000","last_undegraded":"2024-05-09T08:33:21.224253+0000","last_fullsized":"2024-05-09T08:33:21.224253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:40.065173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834622+0000","last_change":"2024-05-09T08:31:56.570220+0000","last_active":"2024-05-09T08:33:20.834622+0000","last_peered":"2024-05-09T08:33:20.834622+0000","last_clean":"2024-05-09T08:33:20.834622+0000","last_became_active":"2024-05-09T08:31:56.569682+0000","last_became_peered":"2024-05-09T08:31:56.569682+0000","last_unstale":"2024-05-09T08:33:20.834622+0000","last_undegraded":"2024-05-09T08:33:20.834622+0000","last_fullsized":"2024-05-09T08:33:20.834622+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:47:07.986064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834439+0000","last_change":"2024-05-09T08:31:00.703201+0000","last_active":"2024-05-09T08:33:20.834439+0000","last_peered":"2024-05-09T08:33:20.834439+0000","last_clean":"2024-05-09T08:33:20.834439+0000","last_became_active":"2024-05-09T08:31:00.702801+0000","last_became_peered":"2024-05-09T08:31:00.702801+0000","last_unstale":"2024-05-09T08:33:20.834439+0000","last_undegraded":"2024-05-09T08:33:20.834439+0000","last_fullsized":"2024-05-09T08:33:20.834439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:58:35.106013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224905+0000","last_change":"2024-05-09T08:31:00.693696+0000","last_active":"2024-05-09T08:33:21.224905+0000","last_peered":"2024-05-09T08:33:21.224905+0000","last_clean":"2024-05-09T08:33:21.224905+0000","last_became_active":"2024-05-09T08:31:00.693394+0000","last_became_peered":"2024-05-09T08:31:00.693394+0000","last_unstale":"2024-05-09T08:33:21.224905+0000","last_undegraded":"2024-05-09T08:33:21.224905+0000","last_fullsized":"2024-05-09T08:33:21.224905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:01.700968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225308+0000","last_change":"2024-05-09T08:31:00.694363+0000","last_active":"2024-05-09T08:33:21.225308+0000","last_peered":"2024-05-09T08:33:21.225308+0000","last_clean":"2024-05-09T08:33:21.225308+0000","last_became_active":"2024-05-09T08:31:00.693970+0000","last_became_peered":"2024-05-09T08:31:00.693970+0000","last_unstale":"2024-05-09T08:33:21.225308+0000","last_undegraded":"2024-05-09T08:33:21.225308+0000","last_fullsized":"2024-05-09T08:33:21.225308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:56.185801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224406+0000","last_change":"2024-05-09T08:31:00.691970+0000","last_active":"2024-05-09T08:33:21.224406+0000","last_peered":"2024-05-09T08:33:21.224406+0000","last_clean":"2024-05-09T08:33:21.224406+0000","last_became_active":"2024-05-09T08:31:00.691632+0000","last_became_peered":"2024-05-09T08:31:00.691632+0000","last_unstale":"2024-05-09T08:33:21.224406+0000","last_undegraded":"2024-05-09T08:33:21.224406+0000","last_fullsized":"2024-05-09T08:33:21.224406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:04:49.316808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835398+0000","last_change":"2024-05-09T08:31:00.692042+0000","last_active":"2024-05-09T08:33:20.835398+0000","last_peered":"2024-05-09T08:33:20.835398+0000","last_clean":"2024-05-09T08:33:20.835398+0000","last_became_active":"2024-05-09T08:31:00.691700+0000","last_became_peered":"2024-05-09T08:31:00.691700+0000","last_unstale":"2024-05-09T08:33:20.835398+0000","last_undegraded":"2024-05-09T08:33:20.835398+0000","last_fullsized":"2024-05-09T08:33:20.835398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:54.236319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836506+0000","last_change":"2024-05-09T08:31:00.697261+0000","last_active":"2024-05-09T08:33:20.836506+0000","last_peered":"2024-05-09T08:33:20.836506+0000","last_clean":"2024-05-09T08:33:20.836506+0000","last_became_active":"2024-05-09T08:31:00.696299+0000","last_became_peered":"2024-05-09T08:31:00.696299+0000","last_unstale":"2024-05-09T08:33:20.836506+0000","last_undegraded":"2024-05-09T08:33:20.836506+0000","last_fullsized":"2024-05-09T08:33:20.836506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:06:52.061740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940597+0000","last_change":"2024-05-09T08:31:00.691097+0000","last_active":"2024-05-09T08:33:20.940597+0000","last_peered":"2024-05-09T08:33:20.940597+0000","last_clean":"2024-05-09T08:33:20.940597+0000","last_became_active":"2024-05-09T08:31:00.690487+0000","last_became_peered":"2024-05-09T08:31:00.690487+0000","last_unstale":"2024-05-09T08:33:20.940597+0000","last_undegraded":"2024-05-09T08:33:20.940597+0000","last_fullsized":"2024-05-09T08:33:20.940597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:47:26.468296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836151+0000","last_change":"2024-05-09T08:31:00.703201+0000","last_active":"2024-05-09T08:33:20.836151+0000","last_peered":"2024-05-09T08:33:20.836151+0000","last_clean":"2024-05-09T08:33:20.836151+0000","last_became_active":"2024-05-09T08:31:00.702853+0000","last_became_peered":"2024-05-09T08:31:00.702853+0000","last_unstale":"2024-05-09T08:33:20.836151+0000","last_undegraded":"2024-05-09T08:33:20.836151+0000","last_fullsized":"2024-05-09T08:33:20.836151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:18.510321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.924123+0000","last_change":"2024-05-09T08:31:00.703958+0000","last_active":"2024-05-09T08:33:20.924123+0000","last_peered":"2024-05-09T08:33:20.924123+0000","last_clean":"2024-05-09T08:33:20.924123+0000","last_became_active":"2024-05-09T08:31:00.703110+0000","last_became_peered":"2024-05-09T08:31:00.703110+0000","last_unstale":"2024-05-09T08:33:20.924123+0000","last_undegraded":"2024-05-09T08:33:20.924123+0000","last_fullsized":"2024-05-09T08:33:20.924123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:37:32.371869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225122+0000","last_change":"2024-05-09T08:31:00.686763+0000","last_active":"2024-05-09T08:33:21.225122+0000","last_peered":"2024-05-09T08:33:21.225122+0000","last_clean":"2024-05-09T08:33:21.225122+0000","last_became_active":"2024-05-09T08:31:00.685642+0000","last_became_peered":"2024-05-09T08:31:00.685642+0000","last_unstale":"2024-05-09T08:33:21.225122+0000","last_undegraded":"2024-05-09T08:33:21.225122+0000","last_fullsized":"2024-05-09T08:33:21.225122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:40:18.201247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224401+0000","last_change":"2024-05-09T08:31:56.573837+0000","last_active":"2024-05-09T08:33:21.224401+0000","last_peered":"2024-05-09T08:33:21.224401+0000","last_clean":"2024-05-09T08:33:21.224401+0000","last_became_active":"2024-05-09T08:31:56.573408+0000","last_became_peered":"2024-05-09T08:31:56.573408+0000","last_unstale":"2024-05-09T08:33:21.224401+0000","last_undegraded":"2024-05-09T08:33:21.224401+0000","last_fullsized":"2024-05-09T08:33:21.224401+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:12:46.397877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600702,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":851,"num_read_kb":537,"num_write":294,"num_write_kb":1446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1186,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1802106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383721,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":529353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645357,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28880,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716272,"statfs":{"total":95995035648,"available":95965462528,"internally_reserved":0,"allocated":1908736,"data_stored":1323296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28472,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716680,"statfs":{"total":95995035648,"available":95965880320,"internally_reserved":0,"allocated":1486848,"data_stored":926879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36780,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708372,"statfs":{"total":95995035648,"available":95957372928,"internally_reserved":0,"allocated":1474560,"data_stored":922099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36984,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708168,"statfs":{"total":95995035648,"available":95957164032,"internally_reserved":0,"allocated":1683456,"data_stored":1128087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":514906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28140,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717012,"statfs":{"total":95995035648,"available":95966220288,"internally_reserved":0,"allocated":1146880,"data_stored":594429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1093632,"data_stored":526131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36476,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708676,"statfs":{"total":95995035648,"available":95957684224,"internally_reserved":0,"allocated":1163264,"data_stored":595890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36908,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708244,"statfs":{"total":95995035648,"available":95957241856,"internally_reserved":0,"allocated":1544192,"data_stored":984209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1085440,"data_stored":531782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37116,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708036,"statfs":{"total":95995035648,"available":95957028864,"internally_reserved":0,"allocated":1822720,"data_stored":1249372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":69776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":134351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":393350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":204179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:33:25.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T08:33:25.849 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi157 to 176.2M 2024-05-09T08:33:25.849 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: Unable to set osd_memory_target on smithi157 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-05-09T08:33:25.849 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi107 to 176.1M 2024-05-09T08:33:25.849 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: Unable to set osd_memory_target on smithi107 to 184708608: error parsing value: Value '184708608' is below minimum 939524096 2024-05-09T08:33:25.850 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi110 to 176.1M 2024-05-09T08:33:25.850 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: Unable to set osd_memory_target on smithi110 to 184745881: error parsing value: Value '184745881' is below minimum 939524096 2024-05-09T08:33:25.850 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: Updating smithi107:/etc/ceph/ceph.conf 2024-05-09T08:33:25.850 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: Updating smithi110:/etc/ceph/ceph.conf 2024-05-09T08:33:25.850 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: Updating smithi157:/etc/ceph/ceph.conf 2024-05-09T08:33:25.850 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: Updating smithi107:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:33:25.850 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: Updating smithi110:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:33:25.850 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: Updating smithi157:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:33:25.850 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3923181866' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:33:25.850 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3141164181' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:33:25.850 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2892770957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:33:25.850 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.851 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.851 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.851 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.851 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.851 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.851 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:25 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi157 to 176.2M 2024-05-09T08:33:25.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: Unable to set osd_memory_target on smithi157 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-05-09T08:33:25.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi107 to 176.1M 2024-05-09T08:33:25.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: Unable to set osd_memory_target on smithi107 to 184708608: error parsing value: Value '184708608' is below minimum 939524096 2024-05-09T08:33:25.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi110 to 176.1M 2024-05-09T08:33:25.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: Unable to set osd_memory_target on smithi110 to 184745881: error parsing value: Value '184745881' is below minimum 939524096 2024-05-09T08:33:25.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: Updating smithi107:/etc/ceph/ceph.conf 2024-05-09T08:33:25.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: Updating smithi110:/etc/ceph/ceph.conf 2024-05-09T08:33:25.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: Updating smithi157:/etc/ceph/ceph.conf 2024-05-09T08:33:25.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: Updating smithi107:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:33:25.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: Updating smithi110:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:33:25.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: Updating smithi157:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T08:33:25.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3923181866' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:33:25.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3141164181' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:33:25.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2892770957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:33:25.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.873 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:25 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:25.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.986+0000 7fdf9581c640 1 -- 172.21.15.107:0/536261769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf90107810 msgr2=0x7fdf90107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:25.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.986+0000 7fdf9581c640 1 --2- 172.21.15.107:0/536261769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf90107810 0x7fdf90107bf0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fdf80009930 tx=0x7fdf8002a420 comp rx=0 tx=0).stop 2024-05-09T08:33:25.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.986+0000 7fdf9581c640 1 -- 172.21.15.107:0/536261769 shutdown_connections 2024-05-09T08:33:25.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.986+0000 7fdf9581c640 1 --2- 172.21.15.107:0/536261769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf901021c0 0x7fdf9010e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.986+0000 7fdf9581c640 1 --2- 172.21.15.107:0/536261769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf90101800 0x7fdf90101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.986+0000 7fdf9581c640 1 --2- 172.21.15.107:0/536261769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf90107810 0x7fdf90107bf0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.986+0000 7fdf9581c640 1 -- 172.21.15.107:0/536261769 >> 172.21.15.107:0/536261769 conn(0x7fdf90078bf0 msgr2=0x7fdf90079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.986+0000 7fdf9581c640 1 -- 172.21.15.107:0/536261769 shutdown_connections 2024-05-09T08:33:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.986+0000 7fdf9581c640 1 -- 172.21.15.107:0/536261769 wait complete. 2024-05-09T08:33:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.986+0000 7fdf9581c640 1 Processor -- start 2024-05-09T08:33:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf9581c640 1 -- start start 2024-05-09T08:33:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf9581c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf90101800 0x7fdf90198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf9581c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf901021c0 0x7fdf901985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf9581c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf90107810 0x7fdf9019c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf9581c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdf90111ae0 con 0x7fdf90107810 2024-05-09T08:33:25.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf9581c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdf90111960 con 0x7fdf90101800 2024-05-09T08:33:25.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf9581c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdf90111c60 con 0x7fdf901021c0 2024-05-09T08:33:25.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf8f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf90107810 0x7fdf9019c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:25.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf8f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf90107810 0x7fdf9019c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50466/0 (socket says 172.21.15.107:50466) 2024-05-09T08:33:25.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf8f7fe640 1 -- 172.21.15.107:0/3032340651 learned_addr learned my addr 172.21.15.107:0/3032340651 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:25.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf8effd640 1 --2- 172.21.15.107:0/3032340651 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf90101800 0x7fdf90198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:25.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf8e7fc640 1 --2- 172.21.15.107:0/3032340651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf901021c0 0x7fdf901985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:25.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf8f7fe640 1 -- 172.21.15.107:0/3032340651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf901021c0 msgr2=0x7fdf901985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:25.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf8f7fe640 1 --2- 172.21.15.107:0/3032340651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf901021c0 0x7fdf901985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf8f7fe640 1 -- 172.21.15.107:0/3032340651 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf90101800 msgr2=0x7fdf90198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:25.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf8f7fe640 1 --2- 172.21.15.107:0/3032340651 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf90101800 0x7fdf90198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:25.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf8f7fe640 1 -- 172.21.15.107:0/3032340651 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9019d050 con 0x7fdf90107810 2024-05-09T08:33:25.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf8effd640 1 --2- 172.21.15.107:0/3032340651 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf90101800 0x7fdf90198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:25.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf8f7fe640 1 --2- 172.21.15.107:0/3032340651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf90107810 0x7fdf9019c920 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800eaa0 tx=0x7fdf8800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:25.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf9481a640 1 -- 172.21.15.107:0/3032340651 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8800cdb0 con 0x7fdf90107810 2024-05-09T08:33:25.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf9481a640 1 -- 172.21.15.107:0/3032340651 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf88005020 con 0x7fdf90107810 2024-05-09T08:33:25.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf9481a640 1 -- 172.21.15.107:0/3032340651 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf880116d0 con 0x7fdf90107810 2024-05-09T08:33:25.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.987+0000 7fdf9581c640 1 -- 172.21.15.107:0/3032340651 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf9019d340 con 0x7fdf90107810 2024-05-09T08:33:25.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.988+0000 7fdf9581c640 1 -- 172.21.15.107:0/3032340651 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf901a4bd0 con 0x7fdf90107810 2024-05-09T08:33:25.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.989+0000 7fdf9581c640 1 -- 172.21.15.107:0/3032340651 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf90068aa0 con 0x7fdf90107810 2024-05-09T08:33:25.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.989+0000 7fdf9481a640 1 -- 172.21.15.107:0/3032340651 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7fdf880051d0 con 0x7fdf90107810 2024-05-09T08:33:25.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.990+0000 7fdf9481a640 1 --2- 172.21.15.107:0/3032340651 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf780a2e20 0x7fdf780a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:25.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.990+0000 7fdf8effd640 1 --2- 172.21.15.107:0/3032340651 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf780a2e20 0x7fdf780a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:25.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.990+0000 7fdf9481a640 1 -- 172.21.15.107:0/3032340651 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fdf880cfca0 con 0x7fdf90107810 2024-05-09T08:33:25.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.991+0000 7fdf8effd640 1 --2- 172.21.15.107:0/3032340651 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf780a2e20 0x7fdf780a52e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdf80006010 tx=0x7fdf80000bc0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:25.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:25.995+0000 7fdf9481a640 1 -- 172.21.15.107:0/3032340651 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdf88016030 con 0x7fdf90107810 2024-05-09T08:33:26.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.171+0000 7fdf9581c640 1 -- 172.21.15.107:0/3032340651 --> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdf90061c70 con 0x7fdf780a2e20 2024-05-09T08:33:26.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.180+0000 7fdf9481a640 1 -- 172.21.15.107:0/3032340651 <== mgr.15483 v2:172.21.15.107:6800/165281804 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339384 (secure 0 0 0) 0x7fdf90061c70 con 0x7fdf780a2e20 2024-05-09T08:33:26.182 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:26.184 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-05-09T08:33:26.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.183+0000 7fdf9581c640 1 -- 172.21.15.107:0/3032340651 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf780a2e20 msgr2=0x7fdf780a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:26.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.183+0000 7fdf9581c640 1 --2- 172.21.15.107:0/3032340651 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf780a2e20 0x7fdf780a52e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdf80006010 tx=0x7fdf80000bc0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.183+0000 7fdf9581c640 1 -- 172.21.15.107:0/3032340651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf90107810 msgr2=0x7fdf9019c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:26.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.183+0000 7fdf9581c640 1 --2- 172.21.15.107:0/3032340651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf90107810 0x7fdf9019c920 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800eaa0 tx=0x7fdf8800ef60 comp rx=0 tx=0).stop 2024-05-09T08:33:26.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.183+0000 7fdf9581c640 1 -- 172.21.15.107:0/3032340651 shutdown_connections 2024-05-09T08:33:26.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.183+0000 7fdf9581c640 1 --2- 172.21.15.107:0/3032340651 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf780a2e20 0x7fdf780a52e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.183+0000 7fdf9581c640 1 --2- 172.21.15.107:0/3032340651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf90107810 0x7fdf9019c920 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.183+0000 7fdf9581c640 1 --2- 172.21.15.107:0/3032340651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf901021c0 0x7fdf901985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.183+0000 7fdf9581c640 1 --2- 172.21.15.107:0/3032340651 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf90101800 0x7fdf90198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.183+0000 7fdf9581c640 1 -- 172.21.15.107:0/3032340651 >> 172.21.15.107:0/3032340651 conn(0x7fdf90078bf0 msgr2=0x7fdf90104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:26.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.183+0000 7fdf9581c640 1 -- 172.21.15.107:0/3032340651 shutdown_connections 2024-05-09T08:33:26.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.183+0000 7fdf9581c640 1 -- 172.21.15.107:0/3032340651 wait complete. 2024-05-09T08:33:26.193 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-05-09T08:33:24.848804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191304,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":987,"num_read_kb":656,"num_write":477,"num_write_kb":4386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1182039,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":872,"ondisk_log_size":872,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":398688,"kb_used_data":16152,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124543136,"statfs":{"total":1151940427776,"available":1151532171264,"internally_reserved":0,"allocated":16539648,"data_stored":9826433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.992086"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224506+0000","last_change":"2024-05-09T08:31:00.695034+0000","last_active":"2024-05-09T08:33:21.224506+0000","last_peered":"2024-05-09T08:33:21.224506+0000","last_clean":"2024-05-09T08:33:21.224506+0000","last_became_active":"2024-05-09T08:31:00.694229+0000","last_became_peered":"2024-05-09T08:31:00.694229+0000","last_unstale":"2024-05-09T08:33:21.224506+0000","last_undegraded":"2024-05-09T08:33:21.224506+0000","last_fullsized":"2024-05-09T08:33:21.224506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:57:22.964848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224706+0000","last_change":"2024-05-09T08:31:00.686219+0000","last_active":"2024-05-09T08:33:21.224706+0000","last_peered":"2024-05-09T08:33:21.224706+0000","last_clean":"2024-05-09T08:33:21.224706+0000","last_became_active":"2024-05-09T08:31:00.685911+0000","last_became_peered":"2024-05-09T08:31:00.685911+0000","last_unstale":"2024-05-09T08:33:21.224706+0000","last_undegraded":"2024-05-09T08:33:21.224706+0000","last_fullsized":"2024-05-09T08:33:21.224706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:28.246199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224414+0000","last_change":"2024-05-09T08:31:00.694228+0000","last_active":"2024-05-09T08:33:21.224414+0000","last_peered":"2024-05-09T08:33:21.224414+0000","last_clean":"2024-05-09T08:33:21.224414+0000","last_became_active":"2024-05-09T08:31:00.693838+0000","last_became_peered":"2024-05-09T08:31:00.693838+0000","last_unstale":"2024-05-09T08:33:21.224414+0000","last_undegraded":"2024-05-09T08:33:21.224414+0000","last_fullsized":"2024-05-09T08:33:21.224414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:44.847966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.078034+0000","last_change":"2024-05-09T08:31:00.702770+0000","last_active":"2024-05-09T08:33:21.078034+0000","last_peered":"2024-05-09T08:33:21.078034+0000","last_clean":"2024-05-09T08:33:21.078034+0000","last_became_active":"2024-05-09T08:31:00.702424+0000","last_became_peered":"2024-05-09T08:31:00.702424+0000","last_unstale":"2024-05-09T08:33:21.078034+0000","last_undegraded":"2024-05-09T08:33:21.078034+0000","last_fullsized":"2024-05-09T08:33:21.078034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:24:05.228942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933801+0000","last_change":"2024-05-09T08:31:00.688308+0000","last_active":"2024-05-09T08:33:20.933801+0000","last_peered":"2024-05-09T08:33:20.933801+0000","last_clean":"2024-05-09T08:33:20.933801+0000","last_became_active":"2024-05-09T08:31:00.687891+0000","last_became_peered":"2024-05-09T08:31:00.687891+0000","last_unstale":"2024-05-09T08:33:20.933801+0000","last_undegraded":"2024-05-09T08:33:20.933801+0000","last_fullsized":"2024-05-09T08:33:20.933801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:48:42.648940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835147+0000","last_change":"2024-05-09T08:31:00.694705+0000","last_active":"2024-05-09T08:33:20.835147+0000","last_peered":"2024-05-09T08:33:20.835147+0000","last_clean":"2024-05-09T08:33:20.835147+0000","last_became_active":"2024-05-09T08:31:00.694361+0000","last_became_peered":"2024-05-09T08:31:00.694361+0000","last_unstale":"2024-05-09T08:33:20.835147+0000","last_undegraded":"2024-05-09T08:33:20.835147+0000","last_fullsized":"2024-05-09T08:33:20.835147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:07:23.137410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074662+0000","last_change":"2024-05-09T08:31:00.689824+0000","last_active":"2024-05-09T08:33:21.074662+0000","last_peered":"2024-05-09T08:33:21.074662+0000","last_clean":"2024-05-09T08:33:21.074662+0000","last_became_active":"2024-05-09T08:31:00.689491+0000","last_became_peered":"2024-05-09T08:31:00.689491+0000","last_unstale":"2024-05-09T08:33:21.074662+0000","last_undegraded":"2024-05-09T08:33:21.074662+0000","last_fullsized":"2024-05-09T08:33:21.074662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:18.791225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835733+0000","last_change":"2024-05-09T08:31:00.695014+0000","last_active":"2024-05-09T08:33:20.835733+0000","last_peered":"2024-05-09T08:33:20.835733+0000","last_clean":"2024-05-09T08:33:20.835733+0000","last_became_active":"2024-05-09T08:31:00.694683+0000","last_became_peered":"2024-05-09T08:31:00.694683+0000","last_unstale":"2024-05-09T08:33:20.835733+0000","last_undegraded":"2024-05-09T08:33:20.835733+0000","last_fullsized":"2024-05-09T08:33:20.835733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:08.570258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.924473+0000","last_change":"2024-05-09T08:31:00.701538+0000","last_active":"2024-05-09T08:33:20.924473+0000","last_peered":"2024-05-09T08:33:20.924473+0000","last_clean":"2024-05-09T08:33:20.924473+0000","last_became_active":"2024-05-09T08:31:00.701226+0000","last_became_peered":"2024-05-09T08:31:00.701226+0000","last_unstale":"2024-05-09T08:33:20.924473+0000","last_undegraded":"2024-05-09T08:33:20.924473+0000","last_fullsized":"2024-05-09T08:33:20.924473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:14:50.853063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835676+0000","last_change":"2024-05-09T08:31:00.698145+0000","last_active":"2024-05-09T08:33:20.835676+0000","last_peered":"2024-05-09T08:33:20.835676+0000","last_clean":"2024-05-09T08:33:20.835676+0000","last_became_active":"2024-05-09T08:31:00.697761+0000","last_became_peered":"2024-05-09T08:31:00.697761+0000","last_unstale":"2024-05-09T08:33:20.835676+0000","last_undegraded":"2024-05-09T08:33:20.835676+0000","last_fullsized":"2024-05-09T08:33:20.835676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:26.094013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075195+0000","last_change":"2024-05-09T08:31:00.699520+0000","last_active":"2024-05-09T08:33:21.075195+0000","last_peered":"2024-05-09T08:33:21.075195+0000","last_clean":"2024-05-09T08:33:21.075195+0000","last_became_active":"2024-05-09T08:31:00.697579+0000","last_became_peered":"2024-05-09T08:31:00.697579+0000","last_unstale":"2024-05-09T08:33:21.075195+0000","last_undegraded":"2024-05-09T08:33:21.075195+0000","last_fullsized":"2024-05-09T08:33:21.075195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:26:34.324474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077854+0000","last_change":"2024-05-09T08:31:00.693356+0000","last_active":"2024-05-09T08:33:21.077854+0000","last_peered":"2024-05-09T08:33:21.077854+0000","last_clean":"2024-05-09T08:33:21.077854+0000","last_became_active":"2024-05-09T08:31:00.692985+0000","last_became_peered":"2024-05-09T08:31:00.692985+0000","last_unstale":"2024-05-09T08:33:21.077854+0000","last_undegraded":"2024-05-09T08:33:21.077854+0000","last_fullsized":"2024-05-09T08:33:21.077854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:49:01.115497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225089+0000","last_change":"2024-05-09T08:31:00.677768+0000","last_active":"2024-05-09T08:33:21.225089+0000","last_peered":"2024-05-09T08:33:21.225089+0000","last_clean":"2024-05-09T08:33:21.225089+0000","last_became_active":"2024-05-09T08:31:00.677386+0000","last_became_peered":"2024-05-09T08:31:00.677386+0000","last_unstale":"2024-05-09T08:33:21.225089+0000","last_undegraded":"2024-05-09T08:33:21.225089+0000","last_fullsized":"2024-05-09T08:33:21.225089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:55.363324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224484+0000","last_change":"2024-05-09T08:31:00.689609+0000","last_active":"2024-05-09T08:33:21.224484+0000","last_peered":"2024-05-09T08:33:21.224484+0000","last_clean":"2024-05-09T08:33:21.224484+0000","last_became_active":"2024-05-09T08:31:00.689303+0000","last_became_peered":"2024-05-09T08:31:00.689303+0000","last_unstale":"2024-05-09T08:33:21.224484+0000","last_undegraded":"2024-05-09T08:33:21.224484+0000","last_fullsized":"2024-05-09T08:33:21.224484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:15:58.672460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224405+0000","last_change":"2024-05-09T08:31:00.704708+0000","last_active":"2024-05-09T08:33:21.224405+0000","last_peered":"2024-05-09T08:33:21.224405+0000","last_clean":"2024-05-09T08:33:21.224405+0000","last_became_active":"2024-05-09T08:31:00.704269+0000","last_became_peered":"2024-05-09T08:31:00.704269+0000","last_unstale":"2024-05-09T08:33:21.224405+0000","last_undegraded":"2024-05-09T08:33:21.224405+0000","last_fullsized":"2024-05-09T08:33:21.224405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:17:44.580420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.923876+0000","last_change":"2024-05-09T08:31:00.703089+0000","last_active":"2024-05-09T08:33:20.923876+0000","last_peered":"2024-05-09T08:33:20.923876+0000","last_clean":"2024-05-09T08:33:20.923876+0000","last_became_active":"2024-05-09T08:31:00.702531+0000","last_became_peered":"2024-05-09T08:31:00.702531+0000","last_unstale":"2024-05-09T08:33:20.923876+0000","last_undegraded":"2024-05-09T08:33:20.923876+0000","last_fullsized":"2024-05-09T08:33:20.923876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:07:39.887748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224486+0000","last_change":"2024-05-09T08:31:00.695332+0000","last_active":"2024-05-09T08:33:21.224486+0000","last_peered":"2024-05-09T08:33:21.224486+0000","last_clean":"2024-05-09T08:33:21.224486+0000","last_became_active":"2024-05-09T08:31:00.694703+0000","last_became_peered":"2024-05-09T08:31:00.694703+0000","last_unstale":"2024-05-09T08:33:21.224486+0000","last_undegraded":"2024-05-09T08:33:21.224486+0000","last_fullsized":"2024-05-09T08:33:21.224486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:53.727922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931989+0000","last_change":"2024-05-09T08:31:00.689871+0000","last_active":"2024-05-09T08:33:20.931989+0000","last_peered":"2024-05-09T08:33:20.931989+0000","last_clean":"2024-05-09T08:33:20.931989+0000","last_became_active":"2024-05-09T08:31:00.689498+0000","last_became_peered":"2024-05-09T08:31:00.689498+0000","last_unstale":"2024-05-09T08:33:20.931989+0000","last_undegraded":"2024-05-09T08:33:20.931989+0000","last_fullsized":"2024-05-09T08:33:20.931989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:18:14.123442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835195+0000","last_change":"2024-05-09T08:31:00.697175+0000","last_active":"2024-05-09T08:33:20.835195+0000","last_peered":"2024-05-09T08:33:20.835195+0000","last_clean":"2024-05-09T08:33:20.835195+0000","last_became_active":"2024-05-09T08:31:00.694103+0000","last_became_peered":"2024-05-09T08:31:00.694103+0000","last_unstale":"2024-05-09T08:33:20.835195+0000","last_undegraded":"2024-05-09T08:33:20.835195+0000","last_fullsized":"2024-05-09T08:33:20.835195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:41:12.549854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834497+0000","last_change":"2024-05-09T08:31:56.797212+0000","last_active":"2024-05-09T08:33:20.834497+0000","last_peered":"2024-05-09T08:33:20.834497+0000","last_clean":"2024-05-09T08:33:20.834497+0000","last_became_active":"2024-05-09T08:31:56.796786+0000","last_became_peered":"2024-05-09T08:31:56.796786+0000","last_unstale":"2024-05-09T08:33:20.834497+0000","last_undegraded":"2024-05-09T08:33:20.834497+0000","last_fullsized":"2024-05-09T08:33:20.834497+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:55.270522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931844+0000","last_change":"2024-05-09T08:31:56.578564+0000","last_active":"2024-05-09T08:33:20.931844+0000","last_peered":"2024-05-09T08:33:20.931844+0000","last_clean":"2024-05-09T08:33:20.931844+0000","last_became_active":"2024-05-09T08:31:56.571246+0000","last_became_peered":"2024-05-09T08:31:56.571246+0000","last_unstale":"2024-05-09T08:33:20.931844+0000","last_undegraded":"2024-05-09T08:33:20.931844+0000","last_fullsized":"2024-05-09T08:33:20.931844+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:15:10.322529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836117+0000","last_change":"2024-05-09T08:31:00.696023+0000","last_active":"2024-05-09T08:33:20.836117+0000","last_peered":"2024-05-09T08:33:20.836117+0000","last_clean":"2024-05-09T08:33:20.836117+0000","last_became_active":"2024-05-09T08:31:00.695677+0000","last_became_peered":"2024-05-09T08:31:00.695677+0000","last_unstale":"2024-05-09T08:33:20.836117+0000","last_undegraded":"2024-05-09T08:33:20.836117+0000","last_fullsized":"2024-05-09T08:33:20.836117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:55.989133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077506+0000","last_change":"2024-05-09T08:31:04.859302+0000","last_active":"2024-05-09T08:33:21.077506+0000","last_peered":"2024-05-09T08:33:21.077506+0000","last_clean":"2024-05-09T08:33:21.077506+0000","last_became_active":"2024-05-09T08:31:04.858897+0000","last_became_peered":"2024-05-09T08:31:04.858897+0000","last_unstale":"2024-05-09T08:33:21.077506+0000","last_undegraded":"2024-05-09T08:33:21.077506+0000","last_fullsized":"2024-05-09T08:33:21.077506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:25:32.970833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836421+0000","last_change":"2024-05-09T08:30:59.824451+0000","last_active":"2024-05-09T08:33:20.836421+0000","last_peered":"2024-05-09T08:33:20.836421+0000","last_clean":"2024-05-09T08:33:20.836421+0000","last_became_active":"2024-05-09T08:30:59.823964+0000","last_became_peered":"2024-05-09T08:30:59.823964+0000","last_unstale":"2024-05-09T08:33:20.836421+0000","last_undegraded":"2024-05-09T08:33:20.836421+0000","last_fullsized":"2024-05-09T08:33:20.836421+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:50:13.562262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225113+0000","last_change":"2024-05-09T08:31:00.701015+0000","last_active":"2024-05-09T08:33:21.225113+0000","last_peered":"2024-05-09T08:33:21.225113+0000","last_clean":"2024-05-09T08:33:21.225113+0000","last_became_active":"2024-05-09T08:31:00.700654+0000","last_became_peered":"2024-05-09T08:31:00.700654+0000","last_unstale":"2024-05-09T08:33:21.225113+0000","last_undegraded":"2024-05-09T08:33:21.225113+0000","last_fullsized":"2024-05-09T08:33:21.225113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:03.710790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077636+0000","last_change":"2024-05-09T08:31:04.858453+0000","last_active":"2024-05-09T08:33:21.077636+0000","last_peered":"2024-05-09T08:33:21.077636+0000","last_clean":"2024-05-09T08:33:21.077636+0000","last_became_active":"2024-05-09T08:31:04.857974+0000","last_became_peered":"2024-05-09T08:31:04.857974+0000","last_unstale":"2024-05-09T08:33:21.077636+0000","last_undegraded":"2024-05-09T08:33:21.077636+0000","last_fullsized":"2024-05-09T08:33:21.077636+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:42:52.868625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.934212+0000","last_change":"2024-05-09T08:30:59.825974+0000","last_active":"2024-05-09T08:33:20.934212+0000","last_peered":"2024-05-09T08:33:20.934212+0000","last_clean":"2024-05-09T08:33:20.934212+0000","last_became_active":"2024-05-09T08:30:59.825671+0000","last_became_peered":"2024-05-09T08:30:59.825671+0000","last_unstale":"2024-05-09T08:33:20.934212+0000","last_undegraded":"2024-05-09T08:33:20.934212+0000","last_fullsized":"2024-05-09T08:33:20.934212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:38:20.179466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.924124+0000","last_change":"2024-05-09T08:31:00.698324+0000","last_active":"2024-05-09T08:33:20.924124+0000","last_peered":"2024-05-09T08:33:20.924124+0000","last_clean":"2024-05-09T08:33:20.924124+0000","last_became_active":"2024-05-09T08:31:00.697992+0000","last_became_peered":"2024-05-09T08:31:00.697992+0000","last_unstale":"2024-05-09T08:33:20.924124+0000","last_undegraded":"2024-05-09T08:33:20.924124+0000","last_fullsized":"2024-05-09T08:33:20.924124+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:25:20.786423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077543+0000","last_change":"2024-05-09T08:31:04.861946+0000","last_active":"2024-05-09T08:33:21.077543+0000","last_peered":"2024-05-09T08:33:21.077543+0000","last_clean":"2024-05-09T08:33:21.077543+0000","last_became_active":"2024-05-09T08:31:04.861511+0000","last_became_peered":"2024-05-09T08:31:04.861511+0000","last_unstale":"2024-05-09T08:33:21.077543+0000","last_undegraded":"2024-05-09T08:33:21.077543+0000","last_fullsized":"2024-05-09T08:33:21.077543+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:36:44.580499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933976+0000","last_change":"2024-05-09T08:30:59.829961+0000","last_active":"2024-05-09T08:33:20.933976+0000","last_peered":"2024-05-09T08:33:20.933976+0000","last_clean":"2024-05-09T08:33:20.933976+0000","last_became_active":"2024-05-09T08:30:59.828057+0000","last_became_peered":"2024-05-09T08:30:59.828057+0000","last_unstale":"2024-05-09T08:33:20.933976+0000","last_undegraded":"2024-05-09T08:33:20.933976+0000","last_fullsized":"2024-05-09T08:33:20.933976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:39:18.229438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940848+0000","last_change":"2024-05-09T08:31:00.700285+0000","last_active":"2024-05-09T08:33:20.940848+0000","last_peered":"2024-05-09T08:33:20.940848+0000","last_clean":"2024-05-09T08:33:20.940848+0000","last_became_active":"2024-05-09T08:31:00.699958+0000","last_became_peered":"2024-05-09T08:31:00.699958+0000","last_unstale":"2024-05-09T08:33:20.940848+0000","last_undegraded":"2024-05-09T08:33:20.940848+0000","last_fullsized":"2024-05-09T08:33:20.940848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:48.934827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835038+0000","last_change":"2024-05-09T08:31:04.857060+0000","last_active":"2024-05-09T08:33:20.835038+0000","last_peered":"2024-05-09T08:33:20.835038+0000","last_clean":"2024-05-09T08:33:20.835038+0000","last_became_active":"2024-05-09T08:31:04.855261+0000","last_became_peered":"2024-05-09T08:31:04.855261+0000","last_unstale":"2024-05-09T08:33:20.835038+0000","last_undegraded":"2024-05-09T08:33:20.835038+0000","last_fullsized":"2024-05-09T08:33:20.835038+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:08:08.230524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'8","reported_seq":166,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.237104+0000","last_change":"2024-05-09T08:30:59.831089+0000","last_active":"2024-05-09T08:33:21.237104+0000","last_peered":"2024-05-09T08:33:21.237104+0000","last_clean":"2024-05-09T08:33:21.237104+0000","last_became_active":"2024-05-09T08:30:59.830572+0000","last_became_peered":"2024-05-09T08:30:59.830572+0000","last_unstale":"2024-05-09T08:33:21.237104+0000","last_undegraded":"2024-05-09T08:33:21.237104+0000","last_fullsized":"2024-05-09T08:33:21.237104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:39:10.495104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1897,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933830+0000","last_change":"2024-05-09T08:31:00.688792+0000","last_active":"2024-05-09T08:33:20.933830+0000","last_peered":"2024-05-09T08:33:20.933830+0000","last_clean":"2024-05-09T08:33:20.933830+0000","last_became_active":"2024-05-09T08:31:00.688386+0000","last_became_peered":"2024-05-09T08:31:00.688386+0000","last_unstale":"2024-05-09T08:33:20.933830+0000","last_undegraded":"2024-05-09T08:33:20.933830+0000","last_fullsized":"2024-05-09T08:33:20.933830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:24.287333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834903+0000","last_change":"2024-05-09T08:31:04.862287+0000","last_active":"2024-05-09T08:33:20.834903+0000","last_peered":"2024-05-09T08:33:20.834903+0000","last_clean":"2024-05-09T08:33:20.834903+0000","last_became_active":"2024-05-09T08:31:04.861866+0000","last_became_peered":"2024-05-09T08:31:04.861866+0000","last_unstale":"2024-05-09T08:33:20.834903+0000","last_undegraded":"2024-05-09T08:33:20.834903+0000","last_fullsized":"2024-05-09T08:33:20.834903+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:14:21.552787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835100+0000","last_change":"2024-05-09T08:30:59.824857+0000","last_active":"2024-05-09T08:33:20.835100+0000","last_peered":"2024-05-09T08:33:20.835100+0000","last_clean":"2024-05-09T08:33:20.835100+0000","last_became_active":"2024-05-09T08:30:59.824363+0000","last_became_peered":"2024-05-09T08:30:59.824363+0000","last_unstale":"2024-05-09T08:33:20.835100+0000","last_undegraded":"2024-05-09T08:33:20.835100+0000","last_fullsized":"2024-05-09T08:33:20.835100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:20:32.267237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940931+0000","last_change":"2024-05-09T08:31:00.694440+0000","last_active":"2024-05-09T08:33:20.940931+0000","last_peered":"2024-05-09T08:33:20.940931+0000","last_clean":"2024-05-09T08:33:20.940931+0000","last_became_active":"2024-05-09T08:31:00.694085+0000","last_became_peered":"2024-05-09T08:31:00.694085+0000","last_unstale":"2024-05-09T08:33:20.940931+0000","last_undegraded":"2024-05-09T08:33:20.940931+0000","last_fullsized":"2024-05-09T08:33:20.940931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:03:24.697479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224410+0000","last_change":"2024-05-09T08:31:56.574514+0000","last_active":"2024-05-09T08:33:21.224410+0000","last_peered":"2024-05-09T08:33:21.224410+0000","last_clean":"2024-05-09T08:33:21.224410+0000","last_became_active":"2024-05-09T08:31:56.573786+0000","last_became_peered":"2024-05-09T08:31:56.573786+0000","last_unstale":"2024-05-09T08:33:21.224410+0000","last_undegraded":"2024-05-09T08:33:21.224410+0000","last_fullsized":"2024-05-09T08:33:21.224410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:17.626431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224705+0000","last_change":"2024-05-09T08:30:59.818373+0000","last_active":"2024-05-09T08:33:21.224705+0000","last_peered":"2024-05-09T08:33:21.224705+0000","last_clean":"2024-05-09T08:33:21.224705+0000","last_became_active":"2024-05-09T08:30:59.818023+0000","last_became_peered":"2024-05-09T08:30:59.818023+0000","last_unstale":"2024-05-09T08:33:21.224705+0000","last_undegraded":"2024-05-09T08:33:21.224705+0000","last_fullsized":"2024-05-09T08:33:21.224705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:23:01.994257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835918+0000","last_change":"2024-05-09T08:31:00.689506+0000","last_active":"2024-05-09T08:33:20.835918+0000","last_peered":"2024-05-09T08:33:20.835918+0000","last_clean":"2024-05-09T08:33:20.835918+0000","last_became_active":"2024-05-09T08:31:00.689151+0000","last_became_peered":"2024-05-09T08:31:00.689151+0000","last_unstale":"2024-05-09T08:33:20.835918+0000","last_undegraded":"2024-05-09T08:33:20.835918+0000","last_fullsized":"2024-05-09T08:33:20.835918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:48:13.539509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224607+0000","last_change":"2024-05-09T08:31:04.859215+0000","last_active":"2024-05-09T08:33:21.224607+0000","last_peered":"2024-05-09T08:33:21.224607+0000","last_clean":"2024-05-09T08:33:21.224607+0000","last_became_active":"2024-05-09T08:31:04.858759+0000","last_became_peered":"2024-05-09T08:31:04.858759+0000","last_unstale":"2024-05-09T08:33:21.224607+0000","last_undegraded":"2024-05-09T08:33:21.224607+0000","last_fullsized":"2024-05-09T08:33:21.224607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:41:42.124234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940549+0000","last_change":"2024-05-09T08:30:59.824260+0000","last_active":"2024-05-09T08:33:20.940549+0000","last_peered":"2024-05-09T08:33:20.940549+0000","last_clean":"2024-05-09T08:33:20.940549+0000","last_became_active":"2024-05-09T08:30:59.823959+0000","last_became_peered":"2024-05-09T08:30:59.823959+0000","last_unstale":"2024-05-09T08:33:20.940549+0000","last_undegraded":"2024-05-09T08:33:20.940549+0000","last_fullsized":"2024-05-09T08:33:20.940549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:48:42.354279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931182+0000","last_change":"2024-05-09T08:31:00.687810+0000","last_active":"2024-05-09T08:33:20.931182+0000","last_peered":"2024-05-09T08:33:20.931182+0000","last_clean":"2024-05-09T08:33:20.931182+0000","last_became_active":"2024-05-09T08:31:00.687486+0000","last_became_peered":"2024-05-09T08:31:00.687486+0000","last_unstale":"2024-05-09T08:33:20.931182+0000","last_undegraded":"2024-05-09T08:33:20.931182+0000","last_fullsized":"2024-05-09T08:33:20.931182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:51.428208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835883+0000","last_change":"2024-05-09T08:31:04.853222+0000","last_active":"2024-05-09T08:33:20.835883+0000","last_peered":"2024-05-09T08:33:20.835883+0000","last_clean":"2024-05-09T08:33:20.835883+0000","last_became_active":"2024-05-09T08:31:04.852676+0000","last_became_peered":"2024-05-09T08:31:04.852676+0000","last_unstale":"2024-05-09T08:33:20.835883+0000","last_undegraded":"2024-05-09T08:33:20.835883+0000","last_fullsized":"2024-05-09T08:33:20.835883+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:58:31.949938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":158,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835698+0000","last_change":"2024-05-09T08:30:59.824653+0000","last_active":"2024-05-09T08:33:20.835698+0000","last_peered":"2024-05-09T08:33:20.835698+0000","last_clean":"2024-05-09T08:33:20.835698+0000","last_became_active":"2024-05-09T08:30:59.824277+0000","last_became_peered":"2024-05-09T08:30:59.824277+0000","last_unstale":"2024-05-09T08:33:20.835698+0000","last_undegraded":"2024-05-09T08:33:20.835698+0000","last_fullsized":"2024-05-09T08:33:20.835698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:45:20.340629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.924392+0000","last_change":"2024-05-09T08:31:00.697484+0000","last_active":"2024-05-09T08:33:20.924392+0000","last_peered":"2024-05-09T08:33:20.924392+0000","last_clean":"2024-05-09T08:33:20.924392+0000","last_became_active":"2024-05-09T08:31:00.697095+0000","last_became_peered":"2024-05-09T08:31:00.697095+0000","last_unstale":"2024-05-09T08:33:20.924392+0000","last_undegraded":"2024-05-09T08:33:20.924392+0000","last_fullsized":"2024-05-09T08:33:20.924392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:29:40.702057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933647+0000","last_change":"2024-05-09T08:31:56.574176+0000","last_active":"2024-05-09T08:33:20.933647+0000","last_peered":"2024-05-09T08:33:20.933647+0000","last_clean":"2024-05-09T08:33:20.933647+0000","last_became_active":"2024-05-09T08:31:56.573556+0000","last_became_peered":"2024-05-09T08:31:56.573556+0000","last_unstale":"2024-05-09T08:33:20.933647+0000","last_undegraded":"2024-05-09T08:33:20.933647+0000","last_fullsized":"2024-05-09T08:33:20.933647+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:50.685712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075051+0000","last_change":"2024-05-09T08:30:59.822683+0000","last_active":"2024-05-09T08:33:21.075051+0000","last_peered":"2024-05-09T08:33:21.075051+0000","last_clean":"2024-05-09T08:33:21.075051+0000","last_became_active":"2024-05-09T08:30:59.822150+0000","last_became_peered":"2024-05-09T08:30:59.822150+0000","last_unstale":"2024-05-09T08:33:21.075051+0000","last_undegraded":"2024-05-09T08:33:21.075051+0000","last_fullsized":"2024-05-09T08:33:21.075051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:51:32.782076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224974+0000","last_change":"2024-05-09T08:31:00.694165+0000","last_active":"2024-05-09T08:33:21.224974+0000","last_peered":"2024-05-09T08:33:21.224974+0000","last_clean":"2024-05-09T08:33:21.224974+0000","last_became_active":"2024-05-09T08:31:00.693150+0000","last_became_peered":"2024-05-09T08:31:00.693150+0000","last_unstale":"2024-05-09T08:33:21.224974+0000","last_undegraded":"2024-05-09T08:33:21.224974+0000","last_fullsized":"2024-05-09T08:33:21.224974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:29.278078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077747+0000","last_change":"2024-05-09T08:31:04.858785+0000","last_active":"2024-05-09T08:33:21.077747+0000","last_peered":"2024-05-09T08:33:21.077747+0000","last_clean":"2024-05-09T08:33:21.077747+0000","last_became_active":"2024-05-09T08:31:04.857947+0000","last_became_peered":"2024-05-09T08:31:04.857947+0000","last_unstale":"2024-05-09T08:33:21.077747+0000","last_undegraded":"2024-05-09T08:33:21.077747+0000","last_fullsized":"2024-05-09T08:33:21.077747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:55:48.595607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940992+0000","last_change":"2024-05-09T08:30:59.822946+0000","last_active":"2024-05-09T08:33:20.940992+0000","last_peered":"2024-05-09T08:33:20.940992+0000","last_clean":"2024-05-09T08:33:20.940992+0000","last_became_active":"2024-05-09T08:30:59.822174+0000","last_became_peered":"2024-05-09T08:30:59.822174+0000","last_unstale":"2024-05-09T08:33:20.940992+0000","last_undegraded":"2024-05-09T08:33:20.940992+0000","last_fullsized":"2024-05-09T08:33:20.940992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:14.224033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.924269+0000","last_change":"2024-05-09T08:31:00.704781+0000","last_active":"2024-05-09T08:33:20.924269+0000","last_peered":"2024-05-09T08:33:20.924269+0000","last_clean":"2024-05-09T08:33:20.924269+0000","last_became_active":"2024-05-09T08:31:00.704170+0000","last_became_peered":"2024-05-09T08:31:00.704170+0000","last_unstale":"2024-05-09T08:33:20.924269+0000","last_undegraded":"2024-05-09T08:33:20.924269+0000","last_fullsized":"2024-05-09T08:33:20.924269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:27:47.402290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931199+0000","last_change":"2024-05-09T08:31:04.855648+0000","last_active":"2024-05-09T08:33:20.931199+0000","last_peered":"2024-05-09T08:33:20.931199+0000","last_clean":"2024-05-09T08:33:20.931199+0000","last_became_active":"2024-05-09T08:31:04.854678+0000","last_became_peered":"2024-05-09T08:31:04.854678+0000","last_unstale":"2024-05-09T08:33:20.931199+0000","last_undegraded":"2024-05-09T08:33:20.931199+0000","last_fullsized":"2024-05-09T08:33:20.931199+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:53:46.039172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":327,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.255521+0000","last_change":"2024-05-09T08:30:59.826498+0000","last_active":"2024-05-09T08:33:21.255521+0000","last_peered":"2024-05-09T08:33:21.255521+0000","last_clean":"2024-05-09T08:33:21.255521+0000","last_became_active":"2024-05-09T08:30:59.826050+0000","last_became_peered":"2024-05-09T08:30:59.826050+0000","last_unstale":"2024-05-09T08:33:21.255521+0000","last_undegraded":"2024-05-09T08:33:21.255521+0000","last_fullsized":"2024-05-09T08:33:21.255521+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:19.920901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69194,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":194,"num_write":41,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.934091+0000","last_change":"2024-05-09T08:31:00.688379+0000","last_active":"2024-05-09T08:33:20.934091+0000","last_peered":"2024-05-09T08:33:20.934091+0000","last_clean":"2024-05-09T08:33:20.934091+0000","last_became_active":"2024-05-09T08:31:00.688030+0000","last_became_peered":"2024-05-09T08:31:00.688030+0000","last_unstale":"2024-05-09T08:33:20.934091+0000","last_undegraded":"2024-05-09T08:33:20.934091+0000","last_fullsized":"2024-05-09T08:33:20.934091+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:17.248154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.941177+0000","last_change":"2024-05-09T08:31:04.858513+0000","last_active":"2024-05-09T08:33:20.941177+0000","last_peered":"2024-05-09T08:33:20.941177+0000","last_clean":"2024-05-09T08:33:20.941177+0000","last_became_active":"2024-05-09T08:31:04.857988+0000","last_became_peered":"2024-05-09T08:31:04.857988+0000","last_unstale":"2024-05-09T08:33:20.941177+0000","last_undegraded":"2024-05-09T08:33:20.941177+0000","last_fullsized":"2024-05-09T08:33:20.941177+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:59:24.264503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834912+0000","last_change":"2024-05-09T08:30:59.823403+0000","last_active":"2024-05-09T08:33:20.834912+0000","last_peered":"2024-05-09T08:33:20.834912+0000","last_clean":"2024-05-09T08:33:20.834912+0000","last_became_active":"2024-05-09T08:30:59.823066+0000","last_became_peered":"2024-05-09T08:30:59.823066+0000","last_unstale":"2024-05-09T08:33:20.834912+0000","last_undegraded":"2024-05-09T08:33:20.834912+0000","last_fullsized":"2024-05-09T08:33:20.834912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:16:39.157895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224735+0000","last_change":"2024-05-09T08:31:00.688009+0000","last_active":"2024-05-09T08:33:21.224735+0000","last_peered":"2024-05-09T08:33:21.224735+0000","last_clean":"2024-05-09T08:33:21.224735+0000","last_became_active":"2024-05-09T08:31:00.687679+0000","last_became_peered":"2024-05-09T08:31:00.687679+0000","last_unstale":"2024-05-09T08:33:21.224735+0000","last_undegraded":"2024-05-09T08:33:21.224735+0000","last_fullsized":"2024-05-09T08:33:21.224735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:44.403639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224273+0000","last_change":"2024-05-09T08:31:04.851880+0000","last_active":"2024-05-09T08:33:21.224273+0000","last_peered":"2024-05-09T08:33:21.224273+0000","last_clean":"2024-05-09T08:33:21.224273+0000","last_became_active":"2024-05-09T08:31:04.851316+0000","last_became_peered":"2024-05-09T08:31:04.851316+0000","last_unstale":"2024-05-09T08:33:21.224273+0000","last_undegraded":"2024-05-09T08:33:21.224273+0000","last_fullsized":"2024-05-09T08:33:21.224273+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:53.144996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940735+0000","last_change":"2024-05-09T08:30:59.823411+0000","last_active":"2024-05-09T08:33:20.940735+0000","last_peered":"2024-05-09T08:33:20.940735+0000","last_clean":"2024-05-09T08:33:20.940735+0000","last_became_active":"2024-05-09T08:30:59.822854+0000","last_became_peered":"2024-05-09T08:30:59.822854+0000","last_unstale":"2024-05-09T08:33:20.940735+0000","last_undegraded":"2024-05-09T08:33:20.940735+0000","last_fullsized":"2024-05-09T08:33:20.940735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:36:03.666937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074998+0000","last_change":"2024-05-09T08:31:56.777818+0000","last_active":"2024-05-09T08:33:21.074998+0000","last_peered":"2024-05-09T08:33:21.074998+0000","last_clean":"2024-05-09T08:33:21.074998+0000","last_became_active":"2024-05-09T08:31:56.777555+0000","last_became_peered":"2024-05-09T08:31:56.777555+0000","last_unstale":"2024-05-09T08:33:21.074998+0000","last_undegraded":"2024-05-09T08:33:21.074998+0000","last_fullsized":"2024-05-09T08:33:21.074998+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:16:17.695073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931723+0000","last_change":"2024-05-09T08:31:04.859338+0000","last_active":"2024-05-09T08:33:20.931723+0000","last_peered":"2024-05-09T08:33:20.931723+0000","last_clean":"2024-05-09T08:33:20.931723+0000","last_became_active":"2024-05-09T08:31:04.858951+0000","last_became_peered":"2024-05-09T08:31:04.858951+0000","last_unstale":"2024-05-09T08:33:20.931723+0000","last_undegraded":"2024-05-09T08:33:20.931723+0000","last_fullsized":"2024-05-09T08:33:20.931723+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:26:46.738359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'8","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.236886+0000","last_change":"2024-05-09T08:30:59.823710+0000","last_active":"2024-05-09T08:33:21.236886+0000","last_peered":"2024-05-09T08:33:21.236886+0000","last_clean":"2024-05-09T08:33:21.236886+0000","last_became_active":"2024-05-09T08:30:59.823213+0000","last_became_peered":"2024-05-09T08:30:59.823213+0000","last_unstale":"2024-05-09T08:33:21.236886+0000","last_undegraded":"2024-05-09T08:33:21.236886+0000","last_fullsized":"2024-05-09T08:33:21.236886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:09.134942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":128,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074753+0000","last_change":"2024-05-09T08:31:00.693735+0000","last_active":"2024-05-09T08:33:21.074753+0000","last_peered":"2024-05-09T08:33:21.074753+0000","last_clean":"2024-05-09T08:33:21.074753+0000","last_became_active":"2024-05-09T08:31:00.693401+0000","last_became_peered":"2024-05-09T08:31:00.693401+0000","last_unstale":"2024-05-09T08:33:21.074753+0000","last_undegraded":"2024-05-09T08:33:21.074753+0000","last_fullsized":"2024-05-09T08:33:21.074753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:58:41.429854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225625+0000","last_change":"2024-05-09T08:31:04.861289+0000","last_active":"2024-05-09T08:33:21.225625+0000","last_peered":"2024-05-09T08:33:21.225625+0000","last_clean":"2024-05-09T08:33:21.225625+0000","last_became_active":"2024-05-09T08:31:04.860915+0000","last_became_peered":"2024-05-09T08:31:04.860915+0000","last_unstale":"2024-05-09T08:33:21.225625+0000","last_undegraded":"2024-05-09T08:33:21.225625+0000","last_fullsized":"2024-05-09T08:33:21.225625+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:01:45.724961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933721+0000","last_change":"2024-05-09T08:30:59.827631+0000","last_active":"2024-05-09T08:33:20.933721+0000","last_peered":"2024-05-09T08:33:20.933721+0000","last_clean":"2024-05-09T08:33:20.933721+0000","last_became_active":"2024-05-09T08:30:59.827295+0000","last_became_peered":"2024-05-09T08:30:59.827295+0000","last_unstale":"2024-05-09T08:33:20.933721+0000","last_undegraded":"2024-05-09T08:33:20.933721+0000","last_fullsized":"2024-05-09T08:33:20.933721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:35.282564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940612+0000","last_change":"2024-05-09T08:31:00.697712+0000","last_active":"2024-05-09T08:33:20.940612+0000","last_peered":"2024-05-09T08:33:20.940612+0000","last_clean":"2024-05-09T08:33:20.940612+0000","last_became_active":"2024-05-09T08:31:00.697385+0000","last_became_peered":"2024-05-09T08:31:00.697385+0000","last_unstale":"2024-05-09T08:33:20.940612+0000","last_undegraded":"2024-05-09T08:33:20.940612+0000","last_fullsized":"2024-05-09T08:33:20.940612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:09.087279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940816+0000","last_change":"2024-05-09T08:31:04.860691+0000","last_active":"2024-05-09T08:33:20.940816+0000","last_peered":"2024-05-09T08:33:20.940816+0000","last_clean":"2024-05-09T08:33:20.940816+0000","last_became_active":"2024-05-09T08:31:04.858458+0000","last_became_peered":"2024-05-09T08:31:04.858458+0000","last_unstale":"2024-05-09T08:33:20.940816+0000","last_undegraded":"2024-05-09T08:33:20.940816+0000","last_fullsized":"2024-05-09T08:33:20.940816+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:16.274359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.934269+0000","last_change":"2024-05-09T08:30:59.825188+0000","last_active":"2024-05-09T08:33:20.934269+0000","last_peered":"2024-05-09T08:33:20.934269+0000","last_clean":"2024-05-09T08:33:20.934269+0000","last_became_active":"2024-05-09T08:30:59.824855+0000","last_became_peered":"2024-05-09T08:30:59.824855+0000","last_unstale":"2024-05-09T08:33:20.934269+0000","last_undegraded":"2024-05-09T08:33:20.934269+0000","last_fullsized":"2024-05-09T08:33:20.934269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:17:21.753652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224274+0000","last_change":"2024-05-09T08:31:00.695862+0000","last_active":"2024-05-09T08:33:21.224274+0000","last_peered":"2024-05-09T08:33:21.224274+0000","last_clean":"2024-05-09T08:33:21.224274+0000","last_became_active":"2024-05-09T08:31:00.695404+0000","last_became_peered":"2024-05-09T08:31:00.695404+0000","last_unstale":"2024-05-09T08:33:21.224274+0000","last_undegraded":"2024-05-09T08:33:21.224274+0000","last_fullsized":"2024-05-09T08:33:21.224274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:53:07.567583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933822+0000","last_change":"2024-05-09T08:31:04.856322+0000","last_active":"2024-05-09T08:33:20.933822+0000","last_peered":"2024-05-09T08:33:20.933822+0000","last_clean":"2024-05-09T08:33:20.933822+0000","last_became_active":"2024-05-09T08:31:04.855821+0000","last_became_peered":"2024-05-09T08:31:04.855821+0000","last_unstale":"2024-05-09T08:33:20.933822+0000","last_undegraded":"2024-05-09T08:33:20.933822+0000","last_fullsized":"2024-05-09T08:33:20.933822+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:29:18.594928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931676+0000","last_change":"2024-05-09T08:31:56.574061+0000","last_active":"2024-05-09T08:33:20.931676+0000","last_peered":"2024-05-09T08:33:20.931676+0000","last_clean":"2024-05-09T08:33:20.931676+0000","last_became_active":"2024-05-09T08:31:56.573673+0000","last_became_peered":"2024-05-09T08:31:56.573673+0000","last_unstale":"2024-05-09T08:33:20.931676+0000","last_undegraded":"2024-05-09T08:33:20.931676+0000","last_fullsized":"2024-05-09T08:33:20.931676+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:12:13.699915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835535+0000","last_change":"2024-05-09T08:31:00.684523+0000","last_active":"2024-05-09T08:33:20.835535+0000","last_peered":"2024-05-09T08:33:20.835535+0000","last_clean":"2024-05-09T08:33:20.835535+0000","last_became_active":"2024-05-09T08:31:00.684153+0000","last_became_peered":"2024-05-09T08:31:00.684153+0000","last_unstale":"2024-05-09T08:33:20.835535+0000","last_undegraded":"2024-05-09T08:33:20.835535+0000","last_fullsized":"2024-05-09T08:33:20.835535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:02.423401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225194+0000","last_change":"2024-05-09T08:31:04.857961+0000","last_active":"2024-05-09T08:33:21.225194+0000","last_peered":"2024-05-09T08:33:21.225194+0000","last_clean":"2024-05-09T08:33:21.225194+0000","last_became_active":"2024-05-09T08:31:04.857543+0000","last_became_peered":"2024-05-09T08:31:04.857543+0000","last_unstale":"2024-05-09T08:33:21.225194+0000","last_undegraded":"2024-05-09T08:33:21.225194+0000","last_fullsized":"2024-05-09T08:33:21.225194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:22:30.820297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074717+0000","last_change":"2024-05-09T08:30:59.827878+0000","last_active":"2024-05-09T08:33:21.074717+0000","last_peered":"2024-05-09T08:33:21.074717+0000","last_clean":"2024-05-09T08:33:21.074717+0000","last_became_active":"2024-05-09T08:30:59.827487+0000","last_became_peered":"2024-05-09T08:30:59.827487+0000","last_unstale":"2024-05-09T08:33:21.074717+0000","last_undegraded":"2024-05-09T08:33:21.074717+0000","last_fullsized":"2024-05-09T08:33:21.074717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:34.465457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835244+0000","last_change":"2024-05-09T08:31:00.695617+0000","last_active":"2024-05-09T08:33:20.835244+0000","last_peered":"2024-05-09T08:33:20.835244+0000","last_clean":"2024-05-09T08:33:20.835244+0000","last_became_active":"2024-05-09T08:31:00.695293+0000","last_became_peered":"2024-05-09T08:31:00.695293+0000","last_unstale":"2024-05-09T08:33:20.835244+0000","last_undegraded":"2024-05-09T08:33:20.835244+0000","last_fullsized":"2024-05-09T08:33:20.835244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:44:03.958559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225556+0000","last_change":"2024-05-09T08:31:04.860184+0000","last_active":"2024-05-09T08:33:21.225556+0000","last_peered":"2024-05-09T08:33:21.225556+0000","last_clean":"2024-05-09T08:33:21.225556+0000","last_became_active":"2024-05-09T08:31:04.859730+0000","last_became_peered":"2024-05-09T08:31:04.859730+0000","last_unstale":"2024-05-09T08:33:21.225556+0000","last_undegraded":"2024-05-09T08:33:21.225556+0000","last_fullsized":"2024-05-09T08:33:21.225556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:13:17.292678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940687+0000","last_change":"2024-05-09T08:31:00.694183+0000","last_active":"2024-05-09T08:33:20.940687+0000","last_peered":"2024-05-09T08:33:20.940687+0000","last_clean":"2024-05-09T08:33:20.940687+0000","last_became_active":"2024-05-09T08:31:00.693819+0000","last_became_peered":"2024-05-09T08:31:00.693819+0000","last_unstale":"2024-05-09T08:33:20.940687+0000","last_undegraded":"2024-05-09T08:33:20.940687+0000","last_fullsized":"2024-05-09T08:33:20.940687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:09:26.255663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224626+0000","last_change":"2024-05-09T08:30:59.830758+0000","last_active":"2024-05-09T08:33:21.224626+0000","last_peered":"2024-05-09T08:33:21.224626+0000","last_clean":"2024-05-09T08:33:21.224626+0000","last_became_active":"2024-05-09T08:30:59.830430+0000","last_became_peered":"2024-05-09T08:30:59.830430+0000","last_unstale":"2024-05-09T08:33:21.224626+0000","last_undegraded":"2024-05-09T08:33:21.224626+0000","last_fullsized":"2024-05-09T08:33:21.224626+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:24:05.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075024+0000","last_change":"2024-05-09T08:31:56.675063+0000","last_active":"2024-05-09T08:33:21.075024+0000","last_peered":"2024-05-09T08:33:21.075024+0000","last_clean":"2024-05-09T08:33:21.075024+0000","last_became_active":"2024-05-09T08:31:56.670995+0000","last_became_peered":"2024-05-09T08:31:56.670995+0000","last_unstale":"2024-05-09T08:33:21.075024+0000","last_undegraded":"2024-05-09T08:33:21.075024+0000","last_fullsized":"2024-05-09T08:33:21.075024+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:57:52.985420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835992+0000","last_change":"2024-05-09T08:31:00.692197+0000","last_active":"2024-05-09T08:33:20.835992+0000","last_peered":"2024-05-09T08:33:20.835992+0000","last_clean":"2024-05-09T08:33:20.835992+0000","last_became_active":"2024-05-09T08:31:00.691348+0000","last_became_peered":"2024-05-09T08:31:00.691348+0000","last_unstale":"2024-05-09T08:33:20.835992+0000","last_undegraded":"2024-05-09T08:33:20.835992+0000","last_fullsized":"2024-05-09T08:33:20.835992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:00:45.826433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077650+0000","last_change":"2024-05-09T08:30:59.825341+0000","last_active":"2024-05-09T08:33:21.077650+0000","last_peered":"2024-05-09T08:33:21.077650+0000","last_clean":"2024-05-09T08:33:21.077650+0000","last_became_active":"2024-05-09T08:30:59.824860+0000","last_became_peered":"2024-05-09T08:30:59.824860+0000","last_unstale":"2024-05-09T08:33:21.077650+0000","last_undegraded":"2024-05-09T08:33:21.077650+0000","last_fullsized":"2024-05-09T08:33:21.077650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:13.868040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":365,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074596+0000","last_change":"2024-05-09T08:29:47.816504+0000","last_active":"2024-05-09T08:33:21.074596+0000","last_peered":"2024-05-09T08:33:21.074596+0000","last_clean":"2024-05-09T08:33:21.074596+0000","last_became_active":"2024-05-09T08:29:47.816050+0000","last_became_peered":"2024-05-09T08:29:47.816050+0000","last_unstale":"2024-05-09T08:33:21.074596+0000","last_undegraded":"2024-05-09T08:33:21.074596+0000","last_fullsized":"2024-05-09T08:33:21.074596+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:26:22.135491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:26:22.135491+0000","last_clean_scrub_stamp":"2024-05-09T08:26:22.135491+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:28.543694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940955+0000","last_change":"2024-05-09T08:31:04.860417+0000","last_active":"2024-05-09T08:33:20.940955+0000","last_peered":"2024-05-09T08:33:20.940955+0000","last_clean":"2024-05-09T08:33:20.940955+0000","last_became_active":"2024-05-09T08:31:04.859040+0000","last_became_peered":"2024-05-09T08:31:04.859040+0000","last_unstale":"2024-05-09T08:33:20.940955+0000","last_undegraded":"2024-05-09T08:33:20.940955+0000","last_fullsized":"2024-05-09T08:33:20.940955+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:12:04.157354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224420+0000","last_change":"2024-05-09T08:31:00.703131+0000","last_active":"2024-05-09T08:33:21.224420+0000","last_peered":"2024-05-09T08:33:21.224420+0000","last_clean":"2024-05-09T08:33:21.224420+0000","last_became_active":"2024-05-09T08:31:00.702817+0000","last_became_peered":"2024-05-09T08:31:00.702817+0000","last_unstale":"2024-05-09T08:33:21.224420+0000","last_undegraded":"2024-05-09T08:33:21.224420+0000","last_fullsized":"2024-05-09T08:33:21.224420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:31.059334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074669+0000","last_change":"2024-05-09T08:30:59.829636+0000","last_active":"2024-05-09T08:33:21.074669+0000","last_peered":"2024-05-09T08:33:21.074669+0000","last_clean":"2024-05-09T08:33:21.074669+0000","last_became_active":"2024-05-09T08:30:59.829276+0000","last_became_peered":"2024-05-09T08:30:59.829276+0000","last_unstale":"2024-05-09T08:33:21.074669+0000","last_undegraded":"2024-05-09T08:33:21.074669+0000","last_fullsized":"2024-05-09T08:33:21.074669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:17.847258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224095+0000","last_change":"2024-05-09T08:31:04.853945+0000","last_active":"2024-05-09T08:33:21.224095+0000","last_peered":"2024-05-09T08:33:21.224095+0000","last_clean":"2024-05-09T08:33:21.224095+0000","last_became_active":"2024-05-09T08:31:04.853417+0000","last_became_peered":"2024-05-09T08:31:04.853417+0000","last_unstale":"2024-05-09T08:33:21.224095+0000","last_undegraded":"2024-05-09T08:33:21.224095+0000","last_fullsized":"2024-05-09T08:33:21.224095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:21:31.126562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224172+0000","last_change":"2024-05-09T08:32:57.561844+0000","last_active":"2024-05-09T08:33:21.224172+0000","last_peered":"2024-05-09T08:33:21.224172+0000","last_clean":"2024-05-09T08:33:21.224172+0000","last_became_active":"2024-05-09T08:32:57.559144+0000","last_became_peered":"2024-05-09T08:32:57.559144+0000","last_unstale":"2024-05-09T08:33:21.224172+0000","last_undegraded":"2024-05-09T08:33:21.224172+0000","last_fullsized":"2024-05-09T08:33:21.224172+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:01.772256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225310+0000","last_change":"2024-05-09T08:31:00.703478+0000","last_active":"2024-05-09T08:33:21.225310+0000","last_peered":"2024-05-09T08:33:21.225310+0000","last_clean":"2024-05-09T08:33:21.225310+0000","last_became_active":"2024-05-09T08:31:00.702852+0000","last_became_peered":"2024-05-09T08:31:00.702852+0000","last_unstale":"2024-05-09T08:33:21.225310+0000","last_undegraded":"2024-05-09T08:33:21.225310+0000","last_fullsized":"2024-05-09T08:33:21.225310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:12.635341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835899+0000","last_change":"2024-05-09T08:31:04.860801+0000","last_active":"2024-05-09T08:33:20.835899+0000","last_peered":"2024-05-09T08:33:20.835899+0000","last_clean":"2024-05-09T08:33:20.835899+0000","last_became_active":"2024-05-09T08:31:04.857387+0000","last_became_peered":"2024-05-09T08:31:04.857387+0000","last_unstale":"2024-05-09T08:33:20.835899+0000","last_undegraded":"2024-05-09T08:33:20.835899+0000","last_fullsized":"2024-05-09T08:33:20.835899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:50:16.555987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'45","reported_seq":102,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.044579+0000","last_change":"2024-05-09T08:30:59.827082+0000","last_active":"2024-05-09T08:33:21.044579+0000","last_peered":"2024-05-09T08:33:21.044579+0000","last_clean":"2024-05-09T08:33:21.044579+0000","last_became_active":"2024-05-09T08:30:59.826746+0000","last_became_peered":"2024-05-09T08:30:59.826746+0000","last_unstale":"2024-05-09T08:33:21.044579+0000","last_undegraded":"2024-05-09T08:33:21.044579+0000","last_fullsized":"2024-05-09T08:33:21.044579+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:49.176889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836291+0000","last_change":"2024-05-09T08:31:00.691902+0000","last_active":"2024-05-09T08:33:20.836291+0000","last_peered":"2024-05-09T08:33:20.836291+0000","last_clean":"2024-05-09T08:33:20.836291+0000","last_became_active":"2024-05-09T08:31:00.689491+0000","last_became_peered":"2024-05-09T08:31:00.689491+0000","last_unstale":"2024-05-09T08:33:20.836291+0000","last_undegraded":"2024-05-09T08:33:20.836291+0000","last_fullsized":"2024-05-09T08:33:20.836291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:48:07.335144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834367+0000","last_change":"2024-05-09T08:31:04.857788+0000","last_active":"2024-05-09T08:33:20.834367+0000","last_peered":"2024-05-09T08:33:20.834367+0000","last_clean":"2024-05-09T08:33:20.834367+0000","last_became_active":"2024-05-09T08:31:04.853831+0000","last_became_peered":"2024-05-09T08:31:04.853831+0000","last_unstale":"2024-05-09T08:33:20.834367+0000","last_undegraded":"2024-05-09T08:33:20.834367+0000","last_fullsized":"2024-05-09T08:33:20.834367+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:08:15.451606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.078037+0000","last_change":"2024-05-09T08:30:59.826482+0000","last_active":"2024-05-09T08:33:21.078037+0000","last_peered":"2024-05-09T08:33:21.078037+0000","last_clean":"2024-05-09T08:33:21.078037+0000","last_became_active":"2024-05-09T08:30:59.826123+0000","last_became_peered":"2024-05-09T08:30:59.826123+0000","last_unstale":"2024-05-09T08:33:21.078037+0000","last_undegraded":"2024-05-09T08:33:21.078037+0000","last_fullsized":"2024-05-09T08:33:21.078037+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:38.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224830+0000","last_change":"2024-05-09T08:31:00.693440+0000","last_active":"2024-05-09T08:33:21.224830+0000","last_peered":"2024-05-09T08:33:21.224830+0000","last_clean":"2024-05-09T08:33:21.224830+0000","last_became_active":"2024-05-09T08:31:00.693117+0000","last_became_peered":"2024-05-09T08:31:00.693117+0000","last_unstale":"2024-05-09T08:33:21.224830+0000","last_undegraded":"2024-05-09T08:33:21.224830+0000","last_fullsized":"2024-05-09T08:33:21.224830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:10:01.205605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077861+0000","last_change":"2024-05-09T08:31:04.868120+0000","last_active":"2024-05-09T08:33:21.077861+0000","last_peered":"2024-05-09T08:33:21.077861+0000","last_clean":"2024-05-09T08:33:21.077861+0000","last_became_active":"2024-05-09T08:31:04.867571+0000","last_became_peered":"2024-05-09T08:31:04.867571+0000","last_unstale":"2024-05-09T08:33:21.077861+0000","last_undegraded":"2024-05-09T08:33:21.077861+0000","last_fullsized":"2024-05-09T08:33:21.077861+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:12:36.503972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.923833+0000","last_change":"2024-05-09T08:30:59.826280+0000","last_active":"2024-05-09T08:33:20.923833+0000","last_peered":"2024-05-09T08:33:20.923833+0000","last_clean":"2024-05-09T08:33:20.923833+0000","last_became_active":"2024-05-09T08:30:59.825805+0000","last_became_peered":"2024-05-09T08:30:59.825805+0000","last_unstale":"2024-05-09T08:33:20.923833+0000","last_undegraded":"2024-05-09T08:33:20.923833+0000","last_fullsized":"2024-05-09T08:33:20.923833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:46:03.833159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933725+0000","last_change":"2024-05-09T08:31:00.687581+0000","last_active":"2024-05-09T08:33:20.933725+0000","last_peered":"2024-05-09T08:33:20.933725+0000","last_clean":"2024-05-09T08:33:20.933725+0000","last_became_active":"2024-05-09T08:31:00.684608+0000","last_became_peered":"2024-05-09T08:31:00.684608+0000","last_unstale":"2024-05-09T08:33:20.933725+0000","last_undegraded":"2024-05-09T08:33:20.933725+0000","last_fullsized":"2024-05-09T08:33:20.933725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:06:06.436840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931830+0000","last_change":"2024-05-09T08:31:04.859399+0000","last_active":"2024-05-09T08:33:20.931830+0000","last_peered":"2024-05-09T08:33:20.931830+0000","last_clean":"2024-05-09T08:33:20.931830+0000","last_became_active":"2024-05-09T08:31:04.859027+0000","last_became_peered":"2024-05-09T08:31:04.859027+0000","last_unstale":"2024-05-09T08:33:20.931830+0000","last_undegraded":"2024-05-09T08:33:20.931830+0000","last_fullsized":"2024-05-09T08:33:20.931830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:18:00.829342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835967+0000","last_change":"2024-05-09T08:30:59.824171+0000","last_active":"2024-05-09T08:33:20.835967+0000","last_peered":"2024-05-09T08:33:20.835967+0000","last_clean":"2024-05-09T08:33:20.835967+0000","last_became_active":"2024-05-09T08:30:59.823859+0000","last_became_peered":"2024-05-09T08:30:59.823859+0000","last_unstale":"2024-05-09T08:33:20.835967+0000","last_undegraded":"2024-05-09T08:33:20.835967+0000","last_fullsized":"2024-05-09T08:33:20.835967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:30.199504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224030+0000","last_change":"2024-05-09T08:31:00.700056+0000","last_active":"2024-05-09T08:33:21.224030+0000","last_peered":"2024-05-09T08:33:21.224030+0000","last_clean":"2024-05-09T08:33:21.224030+0000","last_became_active":"2024-05-09T08:31:00.699673+0000","last_became_peered":"2024-05-09T08:31:00.699673+0000","last_unstale":"2024-05-09T08:33:21.224030+0000","last_undegraded":"2024-05-09T08:33:21.224030+0000","last_fullsized":"2024-05-09T08:33:21.224030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:23.873645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077627+0000","last_change":"2024-05-09T08:31:04.858940+0000","last_active":"2024-05-09T08:33:21.077627+0000","last_peered":"2024-05-09T08:33:21.077627+0000","last_clean":"2024-05-09T08:33:21.077627+0000","last_became_active":"2024-05-09T08:31:04.858532+0000","last_became_peered":"2024-05-09T08:31:04.858532+0000","last_unstale":"2024-05-09T08:33:21.077627+0000","last_undegraded":"2024-05-09T08:33:21.077627+0000","last_fullsized":"2024-05-09T08:33:21.077627+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:57:42.128068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834600+0000","last_change":"2024-05-09T08:30:59.824182+0000","last_active":"2024-05-09T08:33:20.834600+0000","last_peered":"2024-05-09T08:33:20.834600+0000","last_clean":"2024-05-09T08:33:20.834600+0000","last_became_active":"2024-05-09T08:30:59.823691+0000","last_became_peered":"2024-05-09T08:30:59.823691+0000","last_unstale":"2024-05-09T08:33:20.834600+0000","last_undegraded":"2024-05-09T08:33:20.834600+0000","last_fullsized":"2024-05-09T08:33:20.834600+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:28:07.810733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836255+0000","last_change":"2024-05-09T08:31:00.692614+0000","last_active":"2024-05-09T08:33:20.836255+0000","last_peered":"2024-05-09T08:33:20.836255+0000","last_clean":"2024-05-09T08:33:20.836255+0000","last_became_active":"2024-05-09T08:31:00.692280+0000","last_became_peered":"2024-05-09T08:31:00.692280+0000","last_unstale":"2024-05-09T08:33:20.836255+0000","last_undegraded":"2024-05-09T08:33:20.836255+0000","last_fullsized":"2024-05-09T08:33:20.836255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:23:15.398196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077712+0000","last_change":"2024-05-09T08:31:04.854029+0000","last_active":"2024-05-09T08:33:21.077712+0000","last_peered":"2024-05-09T08:33:21.077712+0000","last_clean":"2024-05-09T08:33:21.077712+0000","last_became_active":"2024-05-09T08:31:04.853554+0000","last_became_peered":"2024-05-09T08:31:04.853554+0000","last_unstale":"2024-05-09T08:33:21.077712+0000","last_undegraded":"2024-05-09T08:33:21.077712+0000","last_fullsized":"2024-05-09T08:33:21.077712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:34:41.059614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933921+0000","last_change":"2024-05-09T08:30:59.824512+0000","last_active":"2024-05-09T08:33:20.933921+0000","last_peered":"2024-05-09T08:33:20.933921+0000","last_clean":"2024-05-09T08:33:20.933921+0000","last_became_active":"2024-05-09T08:30:59.824146+0000","last_became_peered":"2024-05-09T08:30:59.824146+0000","last_unstale":"2024-05-09T08:33:20.933921+0000","last_undegraded":"2024-05-09T08:33:20.933921+0000","last_fullsized":"2024-05-09T08:33:20.933921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:27.734292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074854+0000","last_change":"2024-05-09T08:31:00.697272+0000","last_active":"2024-05-09T08:33:21.074854+0000","last_peered":"2024-05-09T08:33:21.074854+0000","last_clean":"2024-05-09T08:33:21.074854+0000","last_became_active":"2024-05-09T08:31:00.696026+0000","last_became_peered":"2024-05-09T08:31:00.696026+0000","last_unstale":"2024-05-09T08:33:21.074854+0000","last_undegraded":"2024-05-09T08:33:21.074854+0000","last_fullsized":"2024-05-09T08:33:21.074854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:04.376142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.923977+0000","last_change":"2024-05-09T08:31:04.856641+0000","last_active":"2024-05-09T08:33:20.923977+0000","last_peered":"2024-05-09T08:33:20.923977+0000","last_clean":"2024-05-09T08:33:20.923977+0000","last_became_active":"2024-05-09T08:31:04.856102+0000","last_became_peered":"2024-05-09T08:31:04.856102+0000","last_unstale":"2024-05-09T08:33:20.923977+0000","last_undegraded":"2024-05-09T08:33:20.923977+0000","last_fullsized":"2024-05-09T08:33:20.923977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:35:49.831076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.923979+0000","last_change":"2024-05-09T08:31:56.579569+0000","last_active":"2024-05-09T08:33:20.923979+0000","last_peered":"2024-05-09T08:33:20.923979+0000","last_clean":"2024-05-09T08:33:20.923979+0000","last_became_active":"2024-05-09T08:31:56.573496+0000","last_became_peered":"2024-05-09T08:31:56.573496+0000","last_unstale":"2024-05-09T08:33:20.923979+0000","last_undegraded":"2024-05-09T08:33:20.923979+0000","last_fullsized":"2024-05-09T08:33:20.923979+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:47:47.944200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075274+0000","last_change":"2024-05-09T08:31:00.697876+0000","last_active":"2024-05-09T08:33:21.075274+0000","last_peered":"2024-05-09T08:33:21.075274+0000","last_clean":"2024-05-09T08:33:21.075274+0000","last_became_active":"2024-05-09T08:31:00.696299+0000","last_became_peered":"2024-05-09T08:31:00.696299+0000","last_unstale":"2024-05-09T08:33:21.075274+0000","last_undegraded":"2024-05-09T08:33:21.075274+0000","last_fullsized":"2024-05-09T08:33:21.075274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:48:57.373734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224675+0000","last_change":"2024-05-09T08:31:04.858588+0000","last_active":"2024-05-09T08:33:21.224675+0000","last_peered":"2024-05-09T08:33:21.224675+0000","last_clean":"2024-05-09T08:33:21.224675+0000","last_became_active":"2024-05-09T08:31:04.858154+0000","last_became_peered":"2024-05-09T08:31:04.858154+0000","last_unstale":"2024-05-09T08:33:21.224675+0000","last_undegraded":"2024-05-09T08:33:21.224675+0000","last_fullsized":"2024-05-09T08:33:21.224675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:06:01.768743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.043305+0000","last_change":"2024-05-09T08:30:59.826066+0000","last_active":"2024-05-09T08:33:21.043305+0000","last_peered":"2024-05-09T08:33:21.043305+0000","last_clean":"2024-05-09T08:33:21.043305+0000","last_became_active":"2024-05-09T08:30:59.825586+0000","last_became_peered":"2024-05-09T08:30:59.825586+0000","last_unstale":"2024-05-09T08:33:21.043305+0000","last_undegraded":"2024-05-09T08:33:21.043305+0000","last_fullsized":"2024-05-09T08:33:21.043305+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:19:04.629132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075338+0000","last_change":"2024-05-09T08:31:00.693788+0000","last_active":"2024-05-09T08:33:21.075338+0000","last_peered":"2024-05-09T08:33:21.075338+0000","last_clean":"2024-05-09T08:33:21.075338+0000","last_became_active":"2024-05-09T08:31:00.693448+0000","last_became_peered":"2024-05-09T08:31:00.693448+0000","last_unstale":"2024-05-09T08:33:21.075338+0000","last_undegraded":"2024-05-09T08:33:21.075338+0000","last_fullsized":"2024-05-09T08:33:21.075338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:06.516534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940422+0000","last_change":"2024-05-09T08:31:04.856185+0000","last_active":"2024-05-09T08:33:20.940422+0000","last_peered":"2024-05-09T08:33:20.940422+0000","last_clean":"2024-05-09T08:33:20.940422+0000","last_became_active":"2024-05-09T08:31:04.855779+0000","last_became_peered":"2024-05-09T08:31:04.855779+0000","last_unstale":"2024-05-09T08:33:20.940422+0000","last_undegraded":"2024-05-09T08:33:20.940422+0000","last_fullsized":"2024-05-09T08:33:20.940422+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:45:25.034077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225735+0000","last_change":"2024-05-09T08:30:59.822938+0000","last_active":"2024-05-09T08:33:21.225735+0000","last_peered":"2024-05-09T08:33:21.225735+0000","last_clean":"2024-05-09T08:33:21.225735+0000","last_became_active":"2024-05-09T08:30:59.822629+0000","last_became_peered":"2024-05-09T08:30:59.822629+0000","last_unstale":"2024-05-09T08:33:21.225735+0000","last_undegraded":"2024-05-09T08:33:21.225735+0000","last_fullsized":"2024-05-09T08:33:21.225735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:51.321350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075213+0000","last_change":"2024-05-09T08:31:00.691971+0000","last_active":"2024-05-09T08:33:21.075213+0000","last_peered":"2024-05-09T08:33:21.075213+0000","last_clean":"2024-05-09T08:33:21.075213+0000","last_became_active":"2024-05-09T08:31:00.691641+0000","last_became_peered":"2024-05-09T08:31:00.691641+0000","last_unstale":"2024-05-09T08:33:21.075213+0000","last_undegraded":"2024-05-09T08:33:21.075213+0000","last_fullsized":"2024-05-09T08:33:21.075213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:28.413886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224253+0000","last_change":"2024-05-09T08:31:00.701026+0000","last_active":"2024-05-09T08:33:21.224253+0000","last_peered":"2024-05-09T08:33:21.224253+0000","last_clean":"2024-05-09T08:33:21.224253+0000","last_became_active":"2024-05-09T08:31:00.700457+0000","last_became_peered":"2024-05-09T08:31:00.700457+0000","last_unstale":"2024-05-09T08:33:21.224253+0000","last_undegraded":"2024-05-09T08:33:21.224253+0000","last_fullsized":"2024-05-09T08:33:21.224253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:40.065173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834622+0000","last_change":"2024-05-09T08:31:56.570220+0000","last_active":"2024-05-09T08:33:20.834622+0000","last_peered":"2024-05-09T08:33:20.834622+0000","last_clean":"2024-05-09T08:33:20.834622+0000","last_became_active":"2024-05-09T08:31:56.569682+0000","last_became_peered":"2024-05-09T08:31:56.569682+0000","last_unstale":"2024-05-09T08:33:20.834622+0000","last_undegraded":"2024-05-09T08:33:20.834622+0000","last_fullsized":"2024-05-09T08:33:20.834622+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:47:07.986064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834439+0000","last_change":"2024-05-09T08:31:00.703201+0000","last_active":"2024-05-09T08:33:20.834439+0000","last_peered":"2024-05-09T08:33:20.834439+0000","last_clean":"2024-05-09T08:33:20.834439+0000","last_became_active":"2024-05-09T08:31:00.702801+0000","last_became_peered":"2024-05-09T08:31:00.702801+0000","last_unstale":"2024-05-09T08:33:20.834439+0000","last_undegraded":"2024-05-09T08:33:20.834439+0000","last_fullsized":"2024-05-09T08:33:20.834439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:58:35.106013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224905+0000","last_change":"2024-05-09T08:31:00.693696+0000","last_active":"2024-05-09T08:33:21.224905+0000","last_peered":"2024-05-09T08:33:21.224905+0000","last_clean":"2024-05-09T08:33:21.224905+0000","last_became_active":"2024-05-09T08:31:00.693394+0000","last_became_peered":"2024-05-09T08:31:00.693394+0000","last_unstale":"2024-05-09T08:33:21.224905+0000","last_undegraded":"2024-05-09T08:33:21.224905+0000","last_fullsized":"2024-05-09T08:33:21.224905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:01.700968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225308+0000","last_change":"2024-05-09T08:31:00.694363+0000","last_active":"2024-05-09T08:33:21.225308+0000","last_peered":"2024-05-09T08:33:21.225308+0000","last_clean":"2024-05-09T08:33:21.225308+0000","last_became_active":"2024-05-09T08:31:00.693970+0000","last_became_peered":"2024-05-09T08:31:00.693970+0000","last_unstale":"2024-05-09T08:33:21.225308+0000","last_undegraded":"2024-05-09T08:33:21.225308+0000","last_fullsized":"2024-05-09T08:33:21.225308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:56.185801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224406+0000","last_change":"2024-05-09T08:31:00.691970+0000","last_active":"2024-05-09T08:33:21.224406+0000","last_peered":"2024-05-09T08:33:21.224406+0000","last_clean":"2024-05-09T08:33:21.224406+0000","last_became_active":"2024-05-09T08:31:00.691632+0000","last_became_peered":"2024-05-09T08:31:00.691632+0000","last_unstale":"2024-05-09T08:33:21.224406+0000","last_undegraded":"2024-05-09T08:33:21.224406+0000","last_fullsized":"2024-05-09T08:33:21.224406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:04:49.316808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835398+0000","last_change":"2024-05-09T08:31:00.692042+0000","last_active":"2024-05-09T08:33:20.835398+0000","last_peered":"2024-05-09T08:33:20.835398+0000","last_clean":"2024-05-09T08:33:20.835398+0000","last_became_active":"2024-05-09T08:31:00.691700+0000","last_became_peered":"2024-05-09T08:31:00.691700+0000","last_unstale":"2024-05-09T08:33:20.835398+0000","last_undegraded":"2024-05-09T08:33:20.835398+0000","last_fullsized":"2024-05-09T08:33:20.835398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:54.236319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836506+0000","last_change":"2024-05-09T08:31:00.697261+0000","last_active":"2024-05-09T08:33:20.836506+0000","last_peered":"2024-05-09T08:33:20.836506+0000","last_clean":"2024-05-09T08:33:20.836506+0000","last_became_active":"2024-05-09T08:31:00.696299+0000","last_became_peered":"2024-05-09T08:31:00.696299+0000","last_unstale":"2024-05-09T08:33:20.836506+0000","last_undegraded":"2024-05-09T08:33:20.836506+0000","last_fullsized":"2024-05-09T08:33:20.836506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:06:52.061740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940597+0000","last_change":"2024-05-09T08:31:00.691097+0000","last_active":"2024-05-09T08:33:20.940597+0000","last_peered":"2024-05-09T08:33:20.940597+0000","last_clean":"2024-05-09T08:33:20.940597+0000","last_became_active":"2024-05-09T08:31:00.690487+0000","last_became_peered":"2024-05-09T08:31:00.690487+0000","last_unstale":"2024-05-09T08:33:20.940597+0000","last_undegraded":"2024-05-09T08:33:20.940597+0000","last_fullsized":"2024-05-09T08:33:20.940597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:47:26.468296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836151+0000","last_change":"2024-05-09T08:31:00.703201+0000","last_active":"2024-05-09T08:33:20.836151+0000","last_peered":"2024-05-09T08:33:20.836151+0000","last_clean":"2024-05-09T08:33:20.836151+0000","last_became_active":"2024-05-09T08:31:00.702853+0000","last_became_peered":"2024-05-09T08:31:00.702853+0000","last_unstale":"2024-05-09T08:33:20.836151+0000","last_undegraded":"2024-05-09T08:33:20.836151+0000","last_fullsized":"2024-05-09T08:33:20.836151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:18.510321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.924123+0000","last_change":"2024-05-09T08:31:00.703958+0000","last_active":"2024-05-09T08:33:20.924123+0000","last_peered":"2024-05-09T08:33:20.924123+0000","last_clean":"2024-05-09T08:33:20.924123+0000","last_became_active":"2024-05-09T08:31:00.703110+0000","last_became_peered":"2024-05-09T08:31:00.703110+0000","last_unstale":"2024-05-09T08:33:20.924123+0000","last_undegraded":"2024-05-09T08:33:20.924123+0000","last_fullsized":"2024-05-09T08:33:20.924123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:37:32.371869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225122+0000","last_change":"2024-05-09T08:31:00.686763+0000","last_active":"2024-05-09T08:33:21.225122+0000","last_peered":"2024-05-09T08:33:21.225122+0000","last_clean":"2024-05-09T08:33:21.225122+0000","last_became_active":"2024-05-09T08:31:00.685642+0000","last_became_peered":"2024-05-09T08:31:00.685642+0000","last_unstale":"2024-05-09T08:33:21.225122+0000","last_undegraded":"2024-05-09T08:33:21.225122+0000","last_fullsized":"2024-05-09T08:33:21.225122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:40:18.201247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224401+0000","last_change":"2024-05-09T08:31:56.573837+0000","last_active":"2024-05-09T08:33:21.224401+0000","last_peered":"2024-05-09T08:33:21.224401+0000","last_clean":"2024-05-09T08:33:21.224401+0000","last_became_active":"2024-05-09T08:31:56.573408+0000","last_became_peered":"2024-05-09T08:31:56.573408+0000","last_unstale":"2024-05-09T08:33:21.224401+0000","last_undegraded":"2024-05-09T08:33:21.224401+0000","last_fullsized":"2024-05-09T08:33:21.224401+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:12:46.397877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600702,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":851,"num_read_kb":537,"num_write":294,"num_write_kb":1446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1186,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1802106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383721,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":529353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645357,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28880,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716272,"statfs":{"total":95995035648,"available":95965462528,"internally_reserved":0,"allocated":1908736,"data_stored":1323296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28472,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716680,"statfs":{"total":95995035648,"available":95965880320,"internally_reserved":0,"allocated":1486848,"data_stored":926879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36780,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708372,"statfs":{"total":95995035648,"available":95957372928,"internally_reserved":0,"allocated":1474560,"data_stored":922099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36984,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708168,"statfs":{"total":95995035648,"available":95957164032,"internally_reserved":0,"allocated":1683456,"data_stored":1128087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":514906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28140,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717012,"statfs":{"total":95995035648,"available":95966220288,"internally_reserved":0,"allocated":1146880,"data_stored":594429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1093632,"data_stored":526131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36476,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708676,"statfs":{"total":95995035648,"available":95957684224,"internally_reserved":0,"allocated":1163264,"data_stored":595890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36908,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708244,"statfs":{"total":95995035648,"available":95957241856,"internally_reserved":0,"allocated":1544192,"data_stored":984209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1085440,"data_stored":531782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37116,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708036,"statfs":{"total":95995035648,"available":95957028864,"internally_reserved":0,"allocated":1822720,"data_stored":1249372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":69776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":134351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":393350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":204179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:33:26.199 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-05-09T08:33:26.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T08:33:26.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.337+0000 7f3c5ea4d640 1 -- 172.21.15.107:0/3334310176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c58101140 msgr2=0x7f3c5810d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:26.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.337+0000 7f3c5ea4d640 1 --2- 172.21.15.107:0/3334310176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c58101140 0x7f3c5810d600 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009cd0 tx=0x7f3c5002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.337+0000 7f3c5ea4d640 1 -- 172.21.15.107:0/3334310176 shutdown_connections 2024-05-09T08:33:26.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.337+0000 7f3c5ea4d640 1 --2- 172.21.15.107:0/3334310176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c58101140 0x7f3c5810d600 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.337+0000 7f3c5ea4d640 1 --2- 172.21.15.107:0/3334310176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c58100780 0x7f3c58100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.337+0000 7f3c5ea4d640 1 --2- 172.21.15.107:0/3334310176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c58106740 0x7f3c58106b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.337+0000 7f3c5ea4d640 1 -- 172.21.15.107:0/3334310176 >> 172.21.15.107:0/3334310176 conn(0x7f3c58077890 msgr2=0x7f3c58077ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:26.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.337+0000 7f3c5ea4d640 1 -- 172.21.15.107:0/3334310176 shutdown_connections 2024-05-09T08:33:26.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.337+0000 7f3c5ea4d640 1 -- 172.21.15.107:0/3334310176 wait complete. 2024-05-09T08:33:26.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.337+0000 7f3c5ea4d640 1 Processor -- start 2024-05-09T08:33:26.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.337+0000 7f3c5ea4d640 1 -- start start 2024-05-09T08:33:26.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c5ea4d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c58100780 0x7f3c58198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:26.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c5ea4d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c58101140 0x7f3c581985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:26.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c5ea4d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c58106740 0x7f3c5819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:26.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c5ea4d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3c58110a00 con 0x7f3c58100780 2024-05-09T08:33:26.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c5ea4d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3c58110880 con 0x7f3c58101140 2024-05-09T08:33:26.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c5ea4d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3c58110b80 con 0x7f3c58106740 2024-05-09T08:33:26.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c58100780 0x7f3c58198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:26.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c58100780 0x7f3c58198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50470/0 (socket says 172.21.15.107:50470) 2024-05-09T08:33:26.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c57fff640 1 -- 172.21.15.107:0/1123132670 learned_addr learned my addr 172.21.15.107:0/1123132670 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:26.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c5cfea640 1 --2- 172.21.15.107:0/1123132670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c58106740 0x7f3c5819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:26.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c4ffff640 1 --2- 172.21.15.107:0/1123132670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c58101140 0x7f3c581985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:26.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c57fff640 1 -- 172.21.15.107:0/1123132670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c58106740 msgr2=0x7f3c5819c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:26.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c57fff640 1 --2- 172.21.15.107:0/1123132670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c58106740 0x7f3c5819c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c57fff640 1 -- 172.21.15.107:0/1123132670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c58101140 msgr2=0x7f3c581985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:26.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c57fff640 1 --2- 172.21.15.107:0/1123132670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c58101140 0x7f3c581985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c57fff640 1 -- 172.21.15.107:0/1123132670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5819d060 con 0x7f3c58100780 2024-05-09T08:33:26.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c57fff640 1 --2- 172.21.15.107:0/1123132670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c58100780 0x7f3c58198070 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800d770 tx=0x7f3c4800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:26.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c55ffb640 1 -- 172.21.15.107:0/1123132670 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c48017070 con 0x7f3c58100780 2024-05-09T08:33:26.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c55ffb640 1 -- 172.21.15.107:0/1123132670 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3c480054d0 con 0x7f3c58100780 2024-05-09T08:33:26.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c55ffb640 1 -- 172.21.15.107:0/1123132670 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c48004cf0 con 0x7f3c58100780 2024-05-09T08:33:26.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c5ea4d640 1 -- 172.21.15.107:0/1123132670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5819d350 con 0x7f3c58100780 2024-05-09T08:33:26.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.338+0000 7f3c5ea4d640 1 -- 172.21.15.107:0/1123132670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c581a4be0 con 0x7f3c58100780 2024-05-09T08:33:26.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.339+0000 7f3c5ea4d640 1 -- 172.21.15.107:0/1123132670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c58068aa0 con 0x7f3c58100780 2024-05-09T08:33:26.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.340+0000 7f3c55ffb640 1 -- 172.21.15.107:0/1123132670 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7f3c48004e90 con 0x7f3c58100780 2024-05-09T08:33:26.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.341+0000 7f3c55ffb640 1 --2- 172.21.15.107:0/1123132670 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3c440a2ea0 0x7f3c440a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:26.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.341+0000 7f3c4ffff640 1 --2- 172.21.15.107:0/1123132670 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3c440a2ea0 0x7f3c440a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:26.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.341+0000 7f3c55ffb640 1 -- 172.21.15.107:0/1123132670 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3c480cf490 con 0x7f3c58100780 2024-05-09T08:33:26.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.342+0000 7f3c4ffff640 1 --2- 172.21.15.107:0/1123132670 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3c440a2ea0 0x7f3c440a5360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3c40005fd0 tx=0x7f3c40009470 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:26.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.345+0000 7f3c55ffb640 1 -- 172.21.15.107:0/1123132670 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3c4809baa0 con 0x7f3c58100780 2024-05-09T08:33:26.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.517+0000 7f3c5ea4d640 1 -- 172.21.15.107:0/1123132670 --> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3c58109d20 con 0x7f3c440a2ea0 2024-05-09T08:33:26.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.526+0000 7f3c55ffb640 1 -- 172.21.15.107:0/1123132670 <== mgr.15483 v2:172.21.15.107:6800/165281804 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339384 (secure 0 0 0) 0x7f3c4000e940 con 0x7f3c440a2ea0 2024-05-09T08:33:26.529 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:26.531 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-05-09T08:33:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.530+0000 7f3c5ea4d640 1 -- 172.21.15.107:0/1123132670 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3c440a2ea0 msgr2=0x7f3c440a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.530+0000 7f3c5ea4d640 1 --2- 172.21.15.107:0/1123132670 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3c440a2ea0 0x7f3c440a5360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3c40005fd0 tx=0x7f3c40009470 comp rx=0 tx=0).stop 2024-05-09T08:33:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.530+0000 7f3c5ea4d640 1 -- 172.21.15.107:0/1123132670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c58100780 msgr2=0x7f3c58198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.530+0000 7f3c5ea4d640 1 --2- 172.21.15.107:0/1123132670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c58100780 0x7f3c58198070 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800d770 tx=0x7f3c4800dc30 comp rx=0 tx=0).stop 2024-05-09T08:33:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.530+0000 7f3c5ea4d640 1 -- 172.21.15.107:0/1123132670 shutdown_connections 2024-05-09T08:33:26.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.530+0000 7f3c5ea4d640 1 --2- 172.21.15.107:0/1123132670 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3c440a2ea0 0x7f3c440a5360 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.530+0000 7f3c5ea4d640 1 --2- 172.21.15.107:0/1123132670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c58106740 0x7f3c5819c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.530+0000 7f3c5ea4d640 1 --2- 172.21.15.107:0/1123132670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c58101140 0x7f3c581985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.530+0000 7f3c5ea4d640 1 --2- 172.21.15.107:0/1123132670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c58100780 0x7f3c58198070 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.530+0000 7f3c5ea4d640 1 -- 172.21.15.107:0/1123132670 >> 172.21.15.107:0/1123132670 conn(0x7f3c58077890 msgr2=0x7f3c5806a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:26.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.530+0000 7f3c5ea4d640 1 -- 172.21.15.107:0/1123132670 shutdown_connections 2024-05-09T08:33:26.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.530+0000 7f3c5ea4d640 1 -- 172.21.15.107:0/1123132670 wait complete. 2024-05-09T08:33:26.534 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:26 smithi107 ceph-mon[30122]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:33:26.534 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:26 smithi107 ceph-mon[30122]: Updating smithi157:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:33:26.534 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:26 smithi107 ceph-mon[30122]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:33:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:26 smithi107 ceph-mon[30122]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:33:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:26 smithi107 ceph-mon[30122]: Updating smithi110:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:33:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:26 smithi107 ceph-mon[30122]: Updating smithi157:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:33:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:26 smithi107 ceph-mon[30122]: Updating smithi107:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:33:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:26 smithi107 ceph-mon[30122]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-05-09T08:33:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:26 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:33:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:26 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:33:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:26 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:33:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:26 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:26.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:26 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1439959666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:33:26.540 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-05-09T08:33:24.848804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191304,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":987,"num_read_kb":656,"num_write":477,"num_write_kb":4386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1182039,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":872,"ondisk_log_size":872,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":398688,"kb_used_data":16152,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124543136,"statfs":{"total":1151940427776,"available":1151532171264,"internally_reserved":0,"allocated":16539648,"data_stored":9826433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.992086"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224506+0000","last_change":"2024-05-09T08:31:00.695034+0000","last_active":"2024-05-09T08:33:21.224506+0000","last_peered":"2024-05-09T08:33:21.224506+0000","last_clean":"2024-05-09T08:33:21.224506+0000","last_became_active":"2024-05-09T08:31:00.694229+0000","last_became_peered":"2024-05-09T08:31:00.694229+0000","last_unstale":"2024-05-09T08:33:21.224506+0000","last_undegraded":"2024-05-09T08:33:21.224506+0000","last_fullsized":"2024-05-09T08:33:21.224506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:57:22.964848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224706+0000","last_change":"2024-05-09T08:31:00.686219+0000","last_active":"2024-05-09T08:33:21.224706+0000","last_peered":"2024-05-09T08:33:21.224706+0000","last_clean":"2024-05-09T08:33:21.224706+0000","last_became_active":"2024-05-09T08:31:00.685911+0000","last_became_peered":"2024-05-09T08:31:00.685911+0000","last_unstale":"2024-05-09T08:33:21.224706+0000","last_undegraded":"2024-05-09T08:33:21.224706+0000","last_fullsized":"2024-05-09T08:33:21.224706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:28.246199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224414+0000","last_change":"2024-05-09T08:31:00.694228+0000","last_active":"2024-05-09T08:33:21.224414+0000","last_peered":"2024-05-09T08:33:21.224414+0000","last_clean":"2024-05-09T08:33:21.224414+0000","last_became_active":"2024-05-09T08:31:00.693838+0000","last_became_peered":"2024-05-09T08:31:00.693838+0000","last_unstale":"2024-05-09T08:33:21.224414+0000","last_undegraded":"2024-05-09T08:33:21.224414+0000","last_fullsized":"2024-05-09T08:33:21.224414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:44.847966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.078034+0000","last_change":"2024-05-09T08:31:00.702770+0000","last_active":"2024-05-09T08:33:21.078034+0000","last_peered":"2024-05-09T08:33:21.078034+0000","last_clean":"2024-05-09T08:33:21.078034+0000","last_became_active":"2024-05-09T08:31:00.702424+0000","last_became_peered":"2024-05-09T08:31:00.702424+0000","last_unstale":"2024-05-09T08:33:21.078034+0000","last_undegraded":"2024-05-09T08:33:21.078034+0000","last_fullsized":"2024-05-09T08:33:21.078034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:24:05.228942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933801+0000","last_change":"2024-05-09T08:31:00.688308+0000","last_active":"2024-05-09T08:33:20.933801+0000","last_peered":"2024-05-09T08:33:20.933801+0000","last_clean":"2024-05-09T08:33:20.933801+0000","last_became_active":"2024-05-09T08:31:00.687891+0000","last_became_peered":"2024-05-09T08:31:00.687891+0000","last_unstale":"2024-05-09T08:33:20.933801+0000","last_undegraded":"2024-05-09T08:33:20.933801+0000","last_fullsized":"2024-05-09T08:33:20.933801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:48:42.648940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835147+0000","last_change":"2024-05-09T08:31:00.694705+0000","last_active":"2024-05-09T08:33:20.835147+0000","last_peered":"2024-05-09T08:33:20.835147+0000","last_clean":"2024-05-09T08:33:20.835147+0000","last_became_active":"2024-05-09T08:31:00.694361+0000","last_became_peered":"2024-05-09T08:31:00.694361+0000","last_unstale":"2024-05-09T08:33:20.835147+0000","last_undegraded":"2024-05-09T08:33:20.835147+0000","last_fullsized":"2024-05-09T08:33:20.835147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:07:23.137410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074662+0000","last_change":"2024-05-09T08:31:00.689824+0000","last_active":"2024-05-09T08:33:21.074662+0000","last_peered":"2024-05-09T08:33:21.074662+0000","last_clean":"2024-05-09T08:33:21.074662+0000","last_became_active":"2024-05-09T08:31:00.689491+0000","last_became_peered":"2024-05-09T08:31:00.689491+0000","last_unstale":"2024-05-09T08:33:21.074662+0000","last_undegraded":"2024-05-09T08:33:21.074662+0000","last_fullsized":"2024-05-09T08:33:21.074662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:18.791225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835733+0000","last_change":"2024-05-09T08:31:00.695014+0000","last_active":"2024-05-09T08:33:20.835733+0000","last_peered":"2024-05-09T08:33:20.835733+0000","last_clean":"2024-05-09T08:33:20.835733+0000","last_became_active":"2024-05-09T08:31:00.694683+0000","last_became_peered":"2024-05-09T08:31:00.694683+0000","last_unstale":"2024-05-09T08:33:20.835733+0000","last_undegraded":"2024-05-09T08:33:20.835733+0000","last_fullsized":"2024-05-09T08:33:20.835733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:55:08.570258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.924473+0000","last_change":"2024-05-09T08:31:00.701538+0000","last_active":"2024-05-09T08:33:20.924473+0000","last_peered":"2024-05-09T08:33:20.924473+0000","last_clean":"2024-05-09T08:33:20.924473+0000","last_became_active":"2024-05-09T08:31:00.701226+0000","last_became_peered":"2024-05-09T08:31:00.701226+0000","last_unstale":"2024-05-09T08:33:20.924473+0000","last_undegraded":"2024-05-09T08:33:20.924473+0000","last_fullsized":"2024-05-09T08:33:20.924473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:14:50.853063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835676+0000","last_change":"2024-05-09T08:31:00.698145+0000","last_active":"2024-05-09T08:33:20.835676+0000","last_peered":"2024-05-09T08:33:20.835676+0000","last_clean":"2024-05-09T08:33:20.835676+0000","last_became_active":"2024-05-09T08:31:00.697761+0000","last_became_peered":"2024-05-09T08:31:00.697761+0000","last_unstale":"2024-05-09T08:33:20.835676+0000","last_undegraded":"2024-05-09T08:33:20.835676+0000","last_fullsized":"2024-05-09T08:33:20.835676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:26.094013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075195+0000","last_change":"2024-05-09T08:31:00.699520+0000","last_active":"2024-05-09T08:33:21.075195+0000","last_peered":"2024-05-09T08:33:21.075195+0000","last_clean":"2024-05-09T08:33:21.075195+0000","last_became_active":"2024-05-09T08:31:00.697579+0000","last_became_peered":"2024-05-09T08:31:00.697579+0000","last_unstale":"2024-05-09T08:33:21.075195+0000","last_undegraded":"2024-05-09T08:33:21.075195+0000","last_fullsized":"2024-05-09T08:33:21.075195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:26:34.324474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077854+0000","last_change":"2024-05-09T08:31:00.693356+0000","last_active":"2024-05-09T08:33:21.077854+0000","last_peered":"2024-05-09T08:33:21.077854+0000","last_clean":"2024-05-09T08:33:21.077854+0000","last_became_active":"2024-05-09T08:31:00.692985+0000","last_became_peered":"2024-05-09T08:31:00.692985+0000","last_unstale":"2024-05-09T08:33:21.077854+0000","last_undegraded":"2024-05-09T08:33:21.077854+0000","last_fullsized":"2024-05-09T08:33:21.077854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:49:01.115497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225089+0000","last_change":"2024-05-09T08:31:00.677768+0000","last_active":"2024-05-09T08:33:21.225089+0000","last_peered":"2024-05-09T08:33:21.225089+0000","last_clean":"2024-05-09T08:33:21.225089+0000","last_became_active":"2024-05-09T08:31:00.677386+0000","last_became_peered":"2024-05-09T08:31:00.677386+0000","last_unstale":"2024-05-09T08:33:21.225089+0000","last_undegraded":"2024-05-09T08:33:21.225089+0000","last_fullsized":"2024-05-09T08:33:21.225089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:55.363324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224484+0000","last_change":"2024-05-09T08:31:00.689609+0000","last_active":"2024-05-09T08:33:21.224484+0000","last_peered":"2024-05-09T08:33:21.224484+0000","last_clean":"2024-05-09T08:33:21.224484+0000","last_became_active":"2024-05-09T08:31:00.689303+0000","last_became_peered":"2024-05-09T08:31:00.689303+0000","last_unstale":"2024-05-09T08:33:21.224484+0000","last_undegraded":"2024-05-09T08:33:21.224484+0000","last_fullsized":"2024-05-09T08:33:21.224484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:15:58.672460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224405+0000","last_change":"2024-05-09T08:31:00.704708+0000","last_active":"2024-05-09T08:33:21.224405+0000","last_peered":"2024-05-09T08:33:21.224405+0000","last_clean":"2024-05-09T08:33:21.224405+0000","last_became_active":"2024-05-09T08:31:00.704269+0000","last_became_peered":"2024-05-09T08:31:00.704269+0000","last_unstale":"2024-05-09T08:33:21.224405+0000","last_undegraded":"2024-05-09T08:33:21.224405+0000","last_fullsized":"2024-05-09T08:33:21.224405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:17:44.580420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.923876+0000","last_change":"2024-05-09T08:31:00.703089+0000","last_active":"2024-05-09T08:33:20.923876+0000","last_peered":"2024-05-09T08:33:20.923876+0000","last_clean":"2024-05-09T08:33:20.923876+0000","last_became_active":"2024-05-09T08:31:00.702531+0000","last_became_peered":"2024-05-09T08:31:00.702531+0000","last_unstale":"2024-05-09T08:33:20.923876+0000","last_undegraded":"2024-05-09T08:33:20.923876+0000","last_fullsized":"2024-05-09T08:33:20.923876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:07:39.887748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224486+0000","last_change":"2024-05-09T08:31:00.695332+0000","last_active":"2024-05-09T08:33:21.224486+0000","last_peered":"2024-05-09T08:33:21.224486+0000","last_clean":"2024-05-09T08:33:21.224486+0000","last_became_active":"2024-05-09T08:31:00.694703+0000","last_became_peered":"2024-05-09T08:31:00.694703+0000","last_unstale":"2024-05-09T08:33:21.224486+0000","last_undegraded":"2024-05-09T08:33:21.224486+0000","last_fullsized":"2024-05-09T08:33:21.224486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:53.727922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931989+0000","last_change":"2024-05-09T08:31:00.689871+0000","last_active":"2024-05-09T08:33:20.931989+0000","last_peered":"2024-05-09T08:33:20.931989+0000","last_clean":"2024-05-09T08:33:20.931989+0000","last_became_active":"2024-05-09T08:31:00.689498+0000","last_became_peered":"2024-05-09T08:31:00.689498+0000","last_unstale":"2024-05-09T08:33:20.931989+0000","last_undegraded":"2024-05-09T08:33:20.931989+0000","last_fullsized":"2024-05-09T08:33:20.931989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:18:14.123442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835195+0000","last_change":"2024-05-09T08:31:00.697175+0000","last_active":"2024-05-09T08:33:20.835195+0000","last_peered":"2024-05-09T08:33:20.835195+0000","last_clean":"2024-05-09T08:33:20.835195+0000","last_became_active":"2024-05-09T08:31:00.694103+0000","last_became_peered":"2024-05-09T08:31:00.694103+0000","last_unstale":"2024-05-09T08:33:20.835195+0000","last_undegraded":"2024-05-09T08:33:20.835195+0000","last_fullsized":"2024-05-09T08:33:20.835195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:41:12.549854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834497+0000","last_change":"2024-05-09T08:31:56.797212+0000","last_active":"2024-05-09T08:33:20.834497+0000","last_peered":"2024-05-09T08:33:20.834497+0000","last_clean":"2024-05-09T08:33:20.834497+0000","last_became_active":"2024-05-09T08:31:56.796786+0000","last_became_peered":"2024-05-09T08:31:56.796786+0000","last_unstale":"2024-05-09T08:33:20.834497+0000","last_undegraded":"2024-05-09T08:33:20.834497+0000","last_fullsized":"2024-05-09T08:33:20.834497+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:55.270522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931844+0000","last_change":"2024-05-09T08:31:56.578564+0000","last_active":"2024-05-09T08:33:20.931844+0000","last_peered":"2024-05-09T08:33:20.931844+0000","last_clean":"2024-05-09T08:33:20.931844+0000","last_became_active":"2024-05-09T08:31:56.571246+0000","last_became_peered":"2024-05-09T08:31:56.571246+0000","last_unstale":"2024-05-09T08:33:20.931844+0000","last_undegraded":"2024-05-09T08:33:20.931844+0000","last_fullsized":"2024-05-09T08:33:20.931844+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:15:10.322529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836117+0000","last_change":"2024-05-09T08:31:00.696023+0000","last_active":"2024-05-09T08:33:20.836117+0000","last_peered":"2024-05-09T08:33:20.836117+0000","last_clean":"2024-05-09T08:33:20.836117+0000","last_became_active":"2024-05-09T08:31:00.695677+0000","last_became_peered":"2024-05-09T08:31:00.695677+0000","last_unstale":"2024-05-09T08:33:20.836117+0000","last_undegraded":"2024-05-09T08:33:20.836117+0000","last_fullsized":"2024-05-09T08:33:20.836117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:55.989133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077506+0000","last_change":"2024-05-09T08:31:04.859302+0000","last_active":"2024-05-09T08:33:21.077506+0000","last_peered":"2024-05-09T08:33:21.077506+0000","last_clean":"2024-05-09T08:33:21.077506+0000","last_became_active":"2024-05-09T08:31:04.858897+0000","last_became_peered":"2024-05-09T08:31:04.858897+0000","last_unstale":"2024-05-09T08:33:21.077506+0000","last_undegraded":"2024-05-09T08:33:21.077506+0000","last_fullsized":"2024-05-09T08:33:21.077506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:25:32.970833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836421+0000","last_change":"2024-05-09T08:30:59.824451+0000","last_active":"2024-05-09T08:33:20.836421+0000","last_peered":"2024-05-09T08:33:20.836421+0000","last_clean":"2024-05-09T08:33:20.836421+0000","last_became_active":"2024-05-09T08:30:59.823964+0000","last_became_peered":"2024-05-09T08:30:59.823964+0000","last_unstale":"2024-05-09T08:33:20.836421+0000","last_undegraded":"2024-05-09T08:33:20.836421+0000","last_fullsized":"2024-05-09T08:33:20.836421+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:50:13.562262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225113+0000","last_change":"2024-05-09T08:31:00.701015+0000","last_active":"2024-05-09T08:33:21.225113+0000","last_peered":"2024-05-09T08:33:21.225113+0000","last_clean":"2024-05-09T08:33:21.225113+0000","last_became_active":"2024-05-09T08:31:00.700654+0000","last_became_peered":"2024-05-09T08:31:00.700654+0000","last_unstale":"2024-05-09T08:33:21.225113+0000","last_undegraded":"2024-05-09T08:33:21.225113+0000","last_fullsized":"2024-05-09T08:33:21.225113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:03.710790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077636+0000","last_change":"2024-05-09T08:31:04.858453+0000","last_active":"2024-05-09T08:33:21.077636+0000","last_peered":"2024-05-09T08:33:21.077636+0000","last_clean":"2024-05-09T08:33:21.077636+0000","last_became_active":"2024-05-09T08:31:04.857974+0000","last_became_peered":"2024-05-09T08:31:04.857974+0000","last_unstale":"2024-05-09T08:33:21.077636+0000","last_undegraded":"2024-05-09T08:33:21.077636+0000","last_fullsized":"2024-05-09T08:33:21.077636+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:42:52.868625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.934212+0000","last_change":"2024-05-09T08:30:59.825974+0000","last_active":"2024-05-09T08:33:20.934212+0000","last_peered":"2024-05-09T08:33:20.934212+0000","last_clean":"2024-05-09T08:33:20.934212+0000","last_became_active":"2024-05-09T08:30:59.825671+0000","last_became_peered":"2024-05-09T08:30:59.825671+0000","last_unstale":"2024-05-09T08:33:20.934212+0000","last_undegraded":"2024-05-09T08:33:20.934212+0000","last_fullsized":"2024-05-09T08:33:20.934212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:38:20.179466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.924124+0000","last_change":"2024-05-09T08:31:00.698324+0000","last_active":"2024-05-09T08:33:20.924124+0000","last_peered":"2024-05-09T08:33:20.924124+0000","last_clean":"2024-05-09T08:33:20.924124+0000","last_became_active":"2024-05-09T08:31:00.697992+0000","last_became_peered":"2024-05-09T08:31:00.697992+0000","last_unstale":"2024-05-09T08:33:20.924124+0000","last_undegraded":"2024-05-09T08:33:20.924124+0000","last_fullsized":"2024-05-09T08:33:20.924124+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:25:20.786423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077543+0000","last_change":"2024-05-09T08:31:04.861946+0000","last_active":"2024-05-09T08:33:21.077543+0000","last_peered":"2024-05-09T08:33:21.077543+0000","last_clean":"2024-05-09T08:33:21.077543+0000","last_became_active":"2024-05-09T08:31:04.861511+0000","last_became_peered":"2024-05-09T08:31:04.861511+0000","last_unstale":"2024-05-09T08:33:21.077543+0000","last_undegraded":"2024-05-09T08:33:21.077543+0000","last_fullsized":"2024-05-09T08:33:21.077543+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:36:44.580499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933976+0000","last_change":"2024-05-09T08:30:59.829961+0000","last_active":"2024-05-09T08:33:20.933976+0000","last_peered":"2024-05-09T08:33:20.933976+0000","last_clean":"2024-05-09T08:33:20.933976+0000","last_became_active":"2024-05-09T08:30:59.828057+0000","last_became_peered":"2024-05-09T08:30:59.828057+0000","last_unstale":"2024-05-09T08:33:20.933976+0000","last_undegraded":"2024-05-09T08:33:20.933976+0000","last_fullsized":"2024-05-09T08:33:20.933976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:39:18.229438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940848+0000","last_change":"2024-05-09T08:31:00.700285+0000","last_active":"2024-05-09T08:33:20.940848+0000","last_peered":"2024-05-09T08:33:20.940848+0000","last_clean":"2024-05-09T08:33:20.940848+0000","last_became_active":"2024-05-09T08:31:00.699958+0000","last_became_peered":"2024-05-09T08:31:00.699958+0000","last_unstale":"2024-05-09T08:33:20.940848+0000","last_undegraded":"2024-05-09T08:33:20.940848+0000","last_fullsized":"2024-05-09T08:33:20.940848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:48.934827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835038+0000","last_change":"2024-05-09T08:31:04.857060+0000","last_active":"2024-05-09T08:33:20.835038+0000","last_peered":"2024-05-09T08:33:20.835038+0000","last_clean":"2024-05-09T08:33:20.835038+0000","last_became_active":"2024-05-09T08:31:04.855261+0000","last_became_peered":"2024-05-09T08:31:04.855261+0000","last_unstale":"2024-05-09T08:33:20.835038+0000","last_undegraded":"2024-05-09T08:33:20.835038+0000","last_fullsized":"2024-05-09T08:33:20.835038+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:08:08.230524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'8","reported_seq":166,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.237104+0000","last_change":"2024-05-09T08:30:59.831089+0000","last_active":"2024-05-09T08:33:21.237104+0000","last_peered":"2024-05-09T08:33:21.237104+0000","last_clean":"2024-05-09T08:33:21.237104+0000","last_became_active":"2024-05-09T08:30:59.830572+0000","last_became_peered":"2024-05-09T08:30:59.830572+0000","last_unstale":"2024-05-09T08:33:21.237104+0000","last_undegraded":"2024-05-09T08:33:21.237104+0000","last_fullsized":"2024-05-09T08:33:21.237104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:39:10.495104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1897,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933830+0000","last_change":"2024-05-09T08:31:00.688792+0000","last_active":"2024-05-09T08:33:20.933830+0000","last_peered":"2024-05-09T08:33:20.933830+0000","last_clean":"2024-05-09T08:33:20.933830+0000","last_became_active":"2024-05-09T08:31:00.688386+0000","last_became_peered":"2024-05-09T08:31:00.688386+0000","last_unstale":"2024-05-09T08:33:20.933830+0000","last_undegraded":"2024-05-09T08:33:20.933830+0000","last_fullsized":"2024-05-09T08:33:20.933830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:25:24.287333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834903+0000","last_change":"2024-05-09T08:31:04.862287+0000","last_active":"2024-05-09T08:33:20.834903+0000","last_peered":"2024-05-09T08:33:20.834903+0000","last_clean":"2024-05-09T08:33:20.834903+0000","last_became_active":"2024-05-09T08:31:04.861866+0000","last_became_peered":"2024-05-09T08:31:04.861866+0000","last_unstale":"2024-05-09T08:33:20.834903+0000","last_undegraded":"2024-05-09T08:33:20.834903+0000","last_fullsized":"2024-05-09T08:33:20.834903+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:14:21.552787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835100+0000","last_change":"2024-05-09T08:30:59.824857+0000","last_active":"2024-05-09T08:33:20.835100+0000","last_peered":"2024-05-09T08:33:20.835100+0000","last_clean":"2024-05-09T08:33:20.835100+0000","last_became_active":"2024-05-09T08:30:59.824363+0000","last_became_peered":"2024-05-09T08:30:59.824363+0000","last_unstale":"2024-05-09T08:33:20.835100+0000","last_undegraded":"2024-05-09T08:33:20.835100+0000","last_fullsized":"2024-05-09T08:33:20.835100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:20:32.267237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940931+0000","last_change":"2024-05-09T08:31:00.694440+0000","last_active":"2024-05-09T08:33:20.940931+0000","last_peered":"2024-05-09T08:33:20.940931+0000","last_clean":"2024-05-09T08:33:20.940931+0000","last_became_active":"2024-05-09T08:31:00.694085+0000","last_became_peered":"2024-05-09T08:31:00.694085+0000","last_unstale":"2024-05-09T08:33:20.940931+0000","last_undegraded":"2024-05-09T08:33:20.940931+0000","last_fullsized":"2024-05-09T08:33:20.940931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:03:24.697479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224410+0000","last_change":"2024-05-09T08:31:56.574514+0000","last_active":"2024-05-09T08:33:21.224410+0000","last_peered":"2024-05-09T08:33:21.224410+0000","last_clean":"2024-05-09T08:33:21.224410+0000","last_became_active":"2024-05-09T08:31:56.573786+0000","last_became_peered":"2024-05-09T08:31:56.573786+0000","last_unstale":"2024-05-09T08:33:21.224410+0000","last_undegraded":"2024-05-09T08:33:21.224410+0000","last_fullsized":"2024-05-09T08:33:21.224410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:17.626431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224705+0000","last_change":"2024-05-09T08:30:59.818373+0000","last_active":"2024-05-09T08:33:21.224705+0000","last_peered":"2024-05-09T08:33:21.224705+0000","last_clean":"2024-05-09T08:33:21.224705+0000","last_became_active":"2024-05-09T08:30:59.818023+0000","last_became_peered":"2024-05-09T08:30:59.818023+0000","last_unstale":"2024-05-09T08:33:21.224705+0000","last_undegraded":"2024-05-09T08:33:21.224705+0000","last_fullsized":"2024-05-09T08:33:21.224705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:23:01.994257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835918+0000","last_change":"2024-05-09T08:31:00.689506+0000","last_active":"2024-05-09T08:33:20.835918+0000","last_peered":"2024-05-09T08:33:20.835918+0000","last_clean":"2024-05-09T08:33:20.835918+0000","last_became_active":"2024-05-09T08:31:00.689151+0000","last_became_peered":"2024-05-09T08:31:00.689151+0000","last_unstale":"2024-05-09T08:33:20.835918+0000","last_undegraded":"2024-05-09T08:33:20.835918+0000","last_fullsized":"2024-05-09T08:33:20.835918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:48:13.539509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224607+0000","last_change":"2024-05-09T08:31:04.859215+0000","last_active":"2024-05-09T08:33:21.224607+0000","last_peered":"2024-05-09T08:33:21.224607+0000","last_clean":"2024-05-09T08:33:21.224607+0000","last_became_active":"2024-05-09T08:31:04.858759+0000","last_became_peered":"2024-05-09T08:31:04.858759+0000","last_unstale":"2024-05-09T08:33:21.224607+0000","last_undegraded":"2024-05-09T08:33:21.224607+0000","last_fullsized":"2024-05-09T08:33:21.224607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:41:42.124234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940549+0000","last_change":"2024-05-09T08:30:59.824260+0000","last_active":"2024-05-09T08:33:20.940549+0000","last_peered":"2024-05-09T08:33:20.940549+0000","last_clean":"2024-05-09T08:33:20.940549+0000","last_became_active":"2024-05-09T08:30:59.823959+0000","last_became_peered":"2024-05-09T08:30:59.823959+0000","last_unstale":"2024-05-09T08:33:20.940549+0000","last_undegraded":"2024-05-09T08:33:20.940549+0000","last_fullsized":"2024-05-09T08:33:20.940549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:48:42.354279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931182+0000","last_change":"2024-05-09T08:31:00.687810+0000","last_active":"2024-05-09T08:33:20.931182+0000","last_peered":"2024-05-09T08:33:20.931182+0000","last_clean":"2024-05-09T08:33:20.931182+0000","last_became_active":"2024-05-09T08:31:00.687486+0000","last_became_peered":"2024-05-09T08:31:00.687486+0000","last_unstale":"2024-05-09T08:33:20.931182+0000","last_undegraded":"2024-05-09T08:33:20.931182+0000","last_fullsized":"2024-05-09T08:33:20.931182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:51.428208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835883+0000","last_change":"2024-05-09T08:31:04.853222+0000","last_active":"2024-05-09T08:33:20.835883+0000","last_peered":"2024-05-09T08:33:20.835883+0000","last_clean":"2024-05-09T08:33:20.835883+0000","last_became_active":"2024-05-09T08:31:04.852676+0000","last_became_peered":"2024-05-09T08:31:04.852676+0000","last_unstale":"2024-05-09T08:33:20.835883+0000","last_undegraded":"2024-05-09T08:33:20.835883+0000","last_fullsized":"2024-05-09T08:33:20.835883+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:58:31.949938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":158,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835698+0000","last_change":"2024-05-09T08:30:59.824653+0000","last_active":"2024-05-09T08:33:20.835698+0000","last_peered":"2024-05-09T08:33:20.835698+0000","last_clean":"2024-05-09T08:33:20.835698+0000","last_became_active":"2024-05-09T08:30:59.824277+0000","last_became_peered":"2024-05-09T08:30:59.824277+0000","last_unstale":"2024-05-09T08:33:20.835698+0000","last_undegraded":"2024-05-09T08:33:20.835698+0000","last_fullsized":"2024-05-09T08:33:20.835698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:45:20.340629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.924392+0000","last_change":"2024-05-09T08:31:00.697484+0000","last_active":"2024-05-09T08:33:20.924392+0000","last_peered":"2024-05-09T08:33:20.924392+0000","last_clean":"2024-05-09T08:33:20.924392+0000","last_became_active":"2024-05-09T08:31:00.697095+0000","last_became_peered":"2024-05-09T08:31:00.697095+0000","last_unstale":"2024-05-09T08:33:20.924392+0000","last_undegraded":"2024-05-09T08:33:20.924392+0000","last_fullsized":"2024-05-09T08:33:20.924392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:29:40.702057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933647+0000","last_change":"2024-05-09T08:31:56.574176+0000","last_active":"2024-05-09T08:33:20.933647+0000","last_peered":"2024-05-09T08:33:20.933647+0000","last_clean":"2024-05-09T08:33:20.933647+0000","last_became_active":"2024-05-09T08:31:56.573556+0000","last_became_peered":"2024-05-09T08:31:56.573556+0000","last_unstale":"2024-05-09T08:33:20.933647+0000","last_undegraded":"2024-05-09T08:33:20.933647+0000","last_fullsized":"2024-05-09T08:33:20.933647+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:50.685712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075051+0000","last_change":"2024-05-09T08:30:59.822683+0000","last_active":"2024-05-09T08:33:21.075051+0000","last_peered":"2024-05-09T08:33:21.075051+0000","last_clean":"2024-05-09T08:33:21.075051+0000","last_became_active":"2024-05-09T08:30:59.822150+0000","last_became_peered":"2024-05-09T08:30:59.822150+0000","last_unstale":"2024-05-09T08:33:21.075051+0000","last_undegraded":"2024-05-09T08:33:21.075051+0000","last_fullsized":"2024-05-09T08:33:21.075051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:51:32.782076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224974+0000","last_change":"2024-05-09T08:31:00.694165+0000","last_active":"2024-05-09T08:33:21.224974+0000","last_peered":"2024-05-09T08:33:21.224974+0000","last_clean":"2024-05-09T08:33:21.224974+0000","last_became_active":"2024-05-09T08:31:00.693150+0000","last_became_peered":"2024-05-09T08:31:00.693150+0000","last_unstale":"2024-05-09T08:33:21.224974+0000","last_undegraded":"2024-05-09T08:33:21.224974+0000","last_fullsized":"2024-05-09T08:33:21.224974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:29.278078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077747+0000","last_change":"2024-05-09T08:31:04.858785+0000","last_active":"2024-05-09T08:33:21.077747+0000","last_peered":"2024-05-09T08:33:21.077747+0000","last_clean":"2024-05-09T08:33:21.077747+0000","last_became_active":"2024-05-09T08:31:04.857947+0000","last_became_peered":"2024-05-09T08:31:04.857947+0000","last_unstale":"2024-05-09T08:33:21.077747+0000","last_undegraded":"2024-05-09T08:33:21.077747+0000","last_fullsized":"2024-05-09T08:33:21.077747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:55:48.595607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940992+0000","last_change":"2024-05-09T08:30:59.822946+0000","last_active":"2024-05-09T08:33:20.940992+0000","last_peered":"2024-05-09T08:33:20.940992+0000","last_clean":"2024-05-09T08:33:20.940992+0000","last_became_active":"2024-05-09T08:30:59.822174+0000","last_became_peered":"2024-05-09T08:30:59.822174+0000","last_unstale":"2024-05-09T08:33:20.940992+0000","last_undegraded":"2024-05-09T08:33:20.940992+0000","last_fullsized":"2024-05-09T08:33:20.940992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:14.224033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.924269+0000","last_change":"2024-05-09T08:31:00.704781+0000","last_active":"2024-05-09T08:33:20.924269+0000","last_peered":"2024-05-09T08:33:20.924269+0000","last_clean":"2024-05-09T08:33:20.924269+0000","last_became_active":"2024-05-09T08:31:00.704170+0000","last_became_peered":"2024-05-09T08:31:00.704170+0000","last_unstale":"2024-05-09T08:33:20.924269+0000","last_undegraded":"2024-05-09T08:33:20.924269+0000","last_fullsized":"2024-05-09T08:33:20.924269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:27:47.402290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931199+0000","last_change":"2024-05-09T08:31:04.855648+0000","last_active":"2024-05-09T08:33:20.931199+0000","last_peered":"2024-05-09T08:33:20.931199+0000","last_clean":"2024-05-09T08:33:20.931199+0000","last_became_active":"2024-05-09T08:31:04.854678+0000","last_became_peered":"2024-05-09T08:31:04.854678+0000","last_unstale":"2024-05-09T08:33:20.931199+0000","last_undegraded":"2024-05-09T08:33:20.931199+0000","last_fullsized":"2024-05-09T08:33:20.931199+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:53:46.039172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":327,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.255521+0000","last_change":"2024-05-09T08:30:59.826498+0000","last_active":"2024-05-09T08:33:21.255521+0000","last_peered":"2024-05-09T08:33:21.255521+0000","last_clean":"2024-05-09T08:33:21.255521+0000","last_became_active":"2024-05-09T08:30:59.826050+0000","last_became_peered":"2024-05-09T08:30:59.826050+0000","last_unstale":"2024-05-09T08:33:21.255521+0000","last_undegraded":"2024-05-09T08:33:21.255521+0000","last_fullsized":"2024-05-09T08:33:21.255521+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:19.920901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69194,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":253,"num_read_kb":194,"num_write":41,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.934091+0000","last_change":"2024-05-09T08:31:00.688379+0000","last_active":"2024-05-09T08:33:20.934091+0000","last_peered":"2024-05-09T08:33:20.934091+0000","last_clean":"2024-05-09T08:33:20.934091+0000","last_became_active":"2024-05-09T08:31:00.688030+0000","last_became_peered":"2024-05-09T08:31:00.688030+0000","last_unstale":"2024-05-09T08:33:20.934091+0000","last_undegraded":"2024-05-09T08:33:20.934091+0000","last_fullsized":"2024-05-09T08:33:20.934091+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:17.248154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.941177+0000","last_change":"2024-05-09T08:31:04.858513+0000","last_active":"2024-05-09T08:33:20.941177+0000","last_peered":"2024-05-09T08:33:20.941177+0000","last_clean":"2024-05-09T08:33:20.941177+0000","last_became_active":"2024-05-09T08:31:04.857988+0000","last_became_peered":"2024-05-09T08:31:04.857988+0000","last_unstale":"2024-05-09T08:33:20.941177+0000","last_undegraded":"2024-05-09T08:33:20.941177+0000","last_fullsized":"2024-05-09T08:33:20.941177+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:59:24.264503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834912+0000","last_change":"2024-05-09T08:30:59.823403+0000","last_active":"2024-05-09T08:33:20.834912+0000","last_peered":"2024-05-09T08:33:20.834912+0000","last_clean":"2024-05-09T08:33:20.834912+0000","last_became_active":"2024-05-09T08:30:59.823066+0000","last_became_peered":"2024-05-09T08:30:59.823066+0000","last_unstale":"2024-05-09T08:33:20.834912+0000","last_undegraded":"2024-05-09T08:33:20.834912+0000","last_fullsized":"2024-05-09T08:33:20.834912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:16:39.157895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224735+0000","last_change":"2024-05-09T08:31:00.688009+0000","last_active":"2024-05-09T08:33:21.224735+0000","last_peered":"2024-05-09T08:33:21.224735+0000","last_clean":"2024-05-09T08:33:21.224735+0000","last_became_active":"2024-05-09T08:31:00.687679+0000","last_became_peered":"2024-05-09T08:31:00.687679+0000","last_unstale":"2024-05-09T08:33:21.224735+0000","last_undegraded":"2024-05-09T08:33:21.224735+0000","last_fullsized":"2024-05-09T08:33:21.224735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:44.403639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224273+0000","last_change":"2024-05-09T08:31:04.851880+0000","last_active":"2024-05-09T08:33:21.224273+0000","last_peered":"2024-05-09T08:33:21.224273+0000","last_clean":"2024-05-09T08:33:21.224273+0000","last_became_active":"2024-05-09T08:31:04.851316+0000","last_became_peered":"2024-05-09T08:31:04.851316+0000","last_unstale":"2024-05-09T08:33:21.224273+0000","last_undegraded":"2024-05-09T08:33:21.224273+0000","last_fullsized":"2024-05-09T08:33:21.224273+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:53.144996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940735+0000","last_change":"2024-05-09T08:30:59.823411+0000","last_active":"2024-05-09T08:33:20.940735+0000","last_peered":"2024-05-09T08:33:20.940735+0000","last_clean":"2024-05-09T08:33:20.940735+0000","last_became_active":"2024-05-09T08:30:59.822854+0000","last_became_peered":"2024-05-09T08:30:59.822854+0000","last_unstale":"2024-05-09T08:33:20.940735+0000","last_undegraded":"2024-05-09T08:33:20.940735+0000","last_fullsized":"2024-05-09T08:33:20.940735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:36:03.666937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074998+0000","last_change":"2024-05-09T08:31:56.777818+0000","last_active":"2024-05-09T08:33:21.074998+0000","last_peered":"2024-05-09T08:33:21.074998+0000","last_clean":"2024-05-09T08:33:21.074998+0000","last_became_active":"2024-05-09T08:31:56.777555+0000","last_became_peered":"2024-05-09T08:31:56.777555+0000","last_unstale":"2024-05-09T08:33:21.074998+0000","last_undegraded":"2024-05-09T08:33:21.074998+0000","last_fullsized":"2024-05-09T08:33:21.074998+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:16:17.695073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931723+0000","last_change":"2024-05-09T08:31:04.859338+0000","last_active":"2024-05-09T08:33:20.931723+0000","last_peered":"2024-05-09T08:33:20.931723+0000","last_clean":"2024-05-09T08:33:20.931723+0000","last_became_active":"2024-05-09T08:31:04.858951+0000","last_became_peered":"2024-05-09T08:31:04.858951+0000","last_unstale":"2024-05-09T08:33:20.931723+0000","last_undegraded":"2024-05-09T08:33:20.931723+0000","last_fullsized":"2024-05-09T08:33:20.931723+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:26:46.738359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'8","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.236886+0000","last_change":"2024-05-09T08:30:59.823710+0000","last_active":"2024-05-09T08:33:21.236886+0000","last_peered":"2024-05-09T08:33:21.236886+0000","last_clean":"2024-05-09T08:33:21.236886+0000","last_became_active":"2024-05-09T08:30:59.823213+0000","last_became_peered":"2024-05-09T08:30:59.823213+0000","last_unstale":"2024-05-09T08:33:21.236886+0000","last_undegraded":"2024-05-09T08:33:21.236886+0000","last_fullsized":"2024-05-09T08:33:21.236886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:09.134942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":128,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074753+0000","last_change":"2024-05-09T08:31:00.693735+0000","last_active":"2024-05-09T08:33:21.074753+0000","last_peered":"2024-05-09T08:33:21.074753+0000","last_clean":"2024-05-09T08:33:21.074753+0000","last_became_active":"2024-05-09T08:31:00.693401+0000","last_became_peered":"2024-05-09T08:31:00.693401+0000","last_unstale":"2024-05-09T08:33:21.074753+0000","last_undegraded":"2024-05-09T08:33:21.074753+0000","last_fullsized":"2024-05-09T08:33:21.074753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:58:41.429854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225625+0000","last_change":"2024-05-09T08:31:04.861289+0000","last_active":"2024-05-09T08:33:21.225625+0000","last_peered":"2024-05-09T08:33:21.225625+0000","last_clean":"2024-05-09T08:33:21.225625+0000","last_became_active":"2024-05-09T08:31:04.860915+0000","last_became_peered":"2024-05-09T08:31:04.860915+0000","last_unstale":"2024-05-09T08:33:21.225625+0000","last_undegraded":"2024-05-09T08:33:21.225625+0000","last_fullsized":"2024-05-09T08:33:21.225625+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:01:45.724961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933721+0000","last_change":"2024-05-09T08:30:59.827631+0000","last_active":"2024-05-09T08:33:20.933721+0000","last_peered":"2024-05-09T08:33:20.933721+0000","last_clean":"2024-05-09T08:33:20.933721+0000","last_became_active":"2024-05-09T08:30:59.827295+0000","last_became_peered":"2024-05-09T08:30:59.827295+0000","last_unstale":"2024-05-09T08:33:20.933721+0000","last_undegraded":"2024-05-09T08:33:20.933721+0000","last_fullsized":"2024-05-09T08:33:20.933721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:35.282564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940612+0000","last_change":"2024-05-09T08:31:00.697712+0000","last_active":"2024-05-09T08:33:20.940612+0000","last_peered":"2024-05-09T08:33:20.940612+0000","last_clean":"2024-05-09T08:33:20.940612+0000","last_became_active":"2024-05-09T08:31:00.697385+0000","last_became_peered":"2024-05-09T08:31:00.697385+0000","last_unstale":"2024-05-09T08:33:20.940612+0000","last_undegraded":"2024-05-09T08:33:20.940612+0000","last_fullsized":"2024-05-09T08:33:20.940612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:09.087279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940816+0000","last_change":"2024-05-09T08:31:04.860691+0000","last_active":"2024-05-09T08:33:20.940816+0000","last_peered":"2024-05-09T08:33:20.940816+0000","last_clean":"2024-05-09T08:33:20.940816+0000","last_became_active":"2024-05-09T08:31:04.858458+0000","last_became_peered":"2024-05-09T08:31:04.858458+0000","last_unstale":"2024-05-09T08:33:20.940816+0000","last_undegraded":"2024-05-09T08:33:20.940816+0000","last_fullsized":"2024-05-09T08:33:20.940816+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:16.274359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.934269+0000","last_change":"2024-05-09T08:30:59.825188+0000","last_active":"2024-05-09T08:33:20.934269+0000","last_peered":"2024-05-09T08:33:20.934269+0000","last_clean":"2024-05-09T08:33:20.934269+0000","last_became_active":"2024-05-09T08:30:59.824855+0000","last_became_peered":"2024-05-09T08:30:59.824855+0000","last_unstale":"2024-05-09T08:33:20.934269+0000","last_undegraded":"2024-05-09T08:33:20.934269+0000","last_fullsized":"2024-05-09T08:33:20.934269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:17:21.753652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224274+0000","last_change":"2024-05-09T08:31:00.695862+0000","last_active":"2024-05-09T08:33:21.224274+0000","last_peered":"2024-05-09T08:33:21.224274+0000","last_clean":"2024-05-09T08:33:21.224274+0000","last_became_active":"2024-05-09T08:31:00.695404+0000","last_became_peered":"2024-05-09T08:31:00.695404+0000","last_unstale":"2024-05-09T08:33:21.224274+0000","last_undegraded":"2024-05-09T08:33:21.224274+0000","last_fullsized":"2024-05-09T08:33:21.224274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:53:07.567583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933822+0000","last_change":"2024-05-09T08:31:04.856322+0000","last_active":"2024-05-09T08:33:20.933822+0000","last_peered":"2024-05-09T08:33:20.933822+0000","last_clean":"2024-05-09T08:33:20.933822+0000","last_became_active":"2024-05-09T08:31:04.855821+0000","last_became_peered":"2024-05-09T08:31:04.855821+0000","last_unstale":"2024-05-09T08:33:20.933822+0000","last_undegraded":"2024-05-09T08:33:20.933822+0000","last_fullsized":"2024-05-09T08:33:20.933822+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:29:18.594928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931676+0000","last_change":"2024-05-09T08:31:56.574061+0000","last_active":"2024-05-09T08:33:20.931676+0000","last_peered":"2024-05-09T08:33:20.931676+0000","last_clean":"2024-05-09T08:33:20.931676+0000","last_became_active":"2024-05-09T08:31:56.573673+0000","last_became_peered":"2024-05-09T08:31:56.573673+0000","last_unstale":"2024-05-09T08:33:20.931676+0000","last_undegraded":"2024-05-09T08:33:20.931676+0000","last_fullsized":"2024-05-09T08:33:20.931676+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:12:13.699915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835535+0000","last_change":"2024-05-09T08:31:00.684523+0000","last_active":"2024-05-09T08:33:20.835535+0000","last_peered":"2024-05-09T08:33:20.835535+0000","last_clean":"2024-05-09T08:33:20.835535+0000","last_became_active":"2024-05-09T08:31:00.684153+0000","last_became_peered":"2024-05-09T08:31:00.684153+0000","last_unstale":"2024-05-09T08:33:20.835535+0000","last_undegraded":"2024-05-09T08:33:20.835535+0000","last_fullsized":"2024-05-09T08:33:20.835535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:02.423401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225194+0000","last_change":"2024-05-09T08:31:04.857961+0000","last_active":"2024-05-09T08:33:21.225194+0000","last_peered":"2024-05-09T08:33:21.225194+0000","last_clean":"2024-05-09T08:33:21.225194+0000","last_became_active":"2024-05-09T08:31:04.857543+0000","last_became_peered":"2024-05-09T08:31:04.857543+0000","last_unstale":"2024-05-09T08:33:21.225194+0000","last_undegraded":"2024-05-09T08:33:21.225194+0000","last_fullsized":"2024-05-09T08:33:21.225194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:22:30.820297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074717+0000","last_change":"2024-05-09T08:30:59.827878+0000","last_active":"2024-05-09T08:33:21.074717+0000","last_peered":"2024-05-09T08:33:21.074717+0000","last_clean":"2024-05-09T08:33:21.074717+0000","last_became_active":"2024-05-09T08:30:59.827487+0000","last_became_peered":"2024-05-09T08:30:59.827487+0000","last_unstale":"2024-05-09T08:33:21.074717+0000","last_undegraded":"2024-05-09T08:33:21.074717+0000","last_fullsized":"2024-05-09T08:33:21.074717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:34.465457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835244+0000","last_change":"2024-05-09T08:31:00.695617+0000","last_active":"2024-05-09T08:33:20.835244+0000","last_peered":"2024-05-09T08:33:20.835244+0000","last_clean":"2024-05-09T08:33:20.835244+0000","last_became_active":"2024-05-09T08:31:00.695293+0000","last_became_peered":"2024-05-09T08:31:00.695293+0000","last_unstale":"2024-05-09T08:33:20.835244+0000","last_undegraded":"2024-05-09T08:33:20.835244+0000","last_fullsized":"2024-05-09T08:33:20.835244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:44:03.958559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225556+0000","last_change":"2024-05-09T08:31:04.860184+0000","last_active":"2024-05-09T08:33:21.225556+0000","last_peered":"2024-05-09T08:33:21.225556+0000","last_clean":"2024-05-09T08:33:21.225556+0000","last_became_active":"2024-05-09T08:31:04.859730+0000","last_became_peered":"2024-05-09T08:31:04.859730+0000","last_unstale":"2024-05-09T08:33:21.225556+0000","last_undegraded":"2024-05-09T08:33:21.225556+0000","last_fullsized":"2024-05-09T08:33:21.225556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:13:17.292678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940687+0000","last_change":"2024-05-09T08:31:00.694183+0000","last_active":"2024-05-09T08:33:20.940687+0000","last_peered":"2024-05-09T08:33:20.940687+0000","last_clean":"2024-05-09T08:33:20.940687+0000","last_became_active":"2024-05-09T08:31:00.693819+0000","last_became_peered":"2024-05-09T08:31:00.693819+0000","last_unstale":"2024-05-09T08:33:20.940687+0000","last_undegraded":"2024-05-09T08:33:20.940687+0000","last_fullsized":"2024-05-09T08:33:20.940687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:09:26.255663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224626+0000","last_change":"2024-05-09T08:30:59.830758+0000","last_active":"2024-05-09T08:33:21.224626+0000","last_peered":"2024-05-09T08:33:21.224626+0000","last_clean":"2024-05-09T08:33:21.224626+0000","last_became_active":"2024-05-09T08:30:59.830430+0000","last_became_peered":"2024-05-09T08:30:59.830430+0000","last_unstale":"2024-05-09T08:33:21.224626+0000","last_undegraded":"2024-05-09T08:33:21.224626+0000","last_fullsized":"2024-05-09T08:33:21.224626+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:24:05.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075024+0000","last_change":"2024-05-09T08:31:56.675063+0000","last_active":"2024-05-09T08:33:21.075024+0000","last_peered":"2024-05-09T08:33:21.075024+0000","last_clean":"2024-05-09T08:33:21.075024+0000","last_became_active":"2024-05-09T08:31:56.670995+0000","last_became_peered":"2024-05-09T08:31:56.670995+0000","last_unstale":"2024-05-09T08:33:21.075024+0000","last_undegraded":"2024-05-09T08:33:21.075024+0000","last_fullsized":"2024-05-09T08:33:21.075024+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:57:52.985420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835992+0000","last_change":"2024-05-09T08:31:00.692197+0000","last_active":"2024-05-09T08:33:20.835992+0000","last_peered":"2024-05-09T08:33:20.835992+0000","last_clean":"2024-05-09T08:33:20.835992+0000","last_became_active":"2024-05-09T08:31:00.691348+0000","last_became_peered":"2024-05-09T08:31:00.691348+0000","last_unstale":"2024-05-09T08:33:20.835992+0000","last_undegraded":"2024-05-09T08:33:20.835992+0000","last_fullsized":"2024-05-09T08:33:20.835992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:00:45.826433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077650+0000","last_change":"2024-05-09T08:30:59.825341+0000","last_active":"2024-05-09T08:33:21.077650+0000","last_peered":"2024-05-09T08:33:21.077650+0000","last_clean":"2024-05-09T08:33:21.077650+0000","last_became_active":"2024-05-09T08:30:59.824860+0000","last_became_peered":"2024-05-09T08:30:59.824860+0000","last_unstale":"2024-05-09T08:33:21.077650+0000","last_undegraded":"2024-05-09T08:33:21.077650+0000","last_fullsized":"2024-05-09T08:33:21.077650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:13.868040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":365,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074596+0000","last_change":"2024-05-09T08:29:47.816504+0000","last_active":"2024-05-09T08:33:21.074596+0000","last_peered":"2024-05-09T08:33:21.074596+0000","last_clean":"2024-05-09T08:33:21.074596+0000","last_became_active":"2024-05-09T08:29:47.816050+0000","last_became_peered":"2024-05-09T08:29:47.816050+0000","last_unstale":"2024-05-09T08:33:21.074596+0000","last_undegraded":"2024-05-09T08:33:21.074596+0000","last_fullsized":"2024-05-09T08:33:21.074596+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:26:22.135491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:26:22.135491+0000","last_clean_scrub_stamp":"2024-05-09T08:26:22.135491+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:28.543694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940955+0000","last_change":"2024-05-09T08:31:04.860417+0000","last_active":"2024-05-09T08:33:20.940955+0000","last_peered":"2024-05-09T08:33:20.940955+0000","last_clean":"2024-05-09T08:33:20.940955+0000","last_became_active":"2024-05-09T08:31:04.859040+0000","last_became_peered":"2024-05-09T08:31:04.859040+0000","last_unstale":"2024-05-09T08:33:20.940955+0000","last_undegraded":"2024-05-09T08:33:20.940955+0000","last_fullsized":"2024-05-09T08:33:20.940955+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:12:04.157354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224420+0000","last_change":"2024-05-09T08:31:00.703131+0000","last_active":"2024-05-09T08:33:21.224420+0000","last_peered":"2024-05-09T08:33:21.224420+0000","last_clean":"2024-05-09T08:33:21.224420+0000","last_became_active":"2024-05-09T08:31:00.702817+0000","last_became_peered":"2024-05-09T08:31:00.702817+0000","last_unstale":"2024-05-09T08:33:21.224420+0000","last_undegraded":"2024-05-09T08:33:21.224420+0000","last_fullsized":"2024-05-09T08:33:21.224420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:31.059334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074669+0000","last_change":"2024-05-09T08:30:59.829636+0000","last_active":"2024-05-09T08:33:21.074669+0000","last_peered":"2024-05-09T08:33:21.074669+0000","last_clean":"2024-05-09T08:33:21.074669+0000","last_became_active":"2024-05-09T08:30:59.829276+0000","last_became_peered":"2024-05-09T08:30:59.829276+0000","last_unstale":"2024-05-09T08:33:21.074669+0000","last_undegraded":"2024-05-09T08:33:21.074669+0000","last_fullsized":"2024-05-09T08:33:21.074669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:17.847258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224095+0000","last_change":"2024-05-09T08:31:04.853945+0000","last_active":"2024-05-09T08:33:21.224095+0000","last_peered":"2024-05-09T08:33:21.224095+0000","last_clean":"2024-05-09T08:33:21.224095+0000","last_became_active":"2024-05-09T08:31:04.853417+0000","last_became_peered":"2024-05-09T08:31:04.853417+0000","last_unstale":"2024-05-09T08:33:21.224095+0000","last_undegraded":"2024-05-09T08:33:21.224095+0000","last_fullsized":"2024-05-09T08:33:21.224095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:21:31.126562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224172+0000","last_change":"2024-05-09T08:32:57.561844+0000","last_active":"2024-05-09T08:33:21.224172+0000","last_peered":"2024-05-09T08:33:21.224172+0000","last_clean":"2024-05-09T08:33:21.224172+0000","last_became_active":"2024-05-09T08:32:57.559144+0000","last_became_peered":"2024-05-09T08:32:57.559144+0000","last_unstale":"2024-05-09T08:33:21.224172+0000","last_undegraded":"2024-05-09T08:33:21.224172+0000","last_fullsized":"2024-05-09T08:33:21.224172+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:01.772256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225310+0000","last_change":"2024-05-09T08:31:00.703478+0000","last_active":"2024-05-09T08:33:21.225310+0000","last_peered":"2024-05-09T08:33:21.225310+0000","last_clean":"2024-05-09T08:33:21.225310+0000","last_became_active":"2024-05-09T08:31:00.702852+0000","last_became_peered":"2024-05-09T08:31:00.702852+0000","last_unstale":"2024-05-09T08:33:21.225310+0000","last_undegraded":"2024-05-09T08:33:21.225310+0000","last_fullsized":"2024-05-09T08:33:21.225310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:12.635341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835899+0000","last_change":"2024-05-09T08:31:04.860801+0000","last_active":"2024-05-09T08:33:20.835899+0000","last_peered":"2024-05-09T08:33:20.835899+0000","last_clean":"2024-05-09T08:33:20.835899+0000","last_became_active":"2024-05-09T08:31:04.857387+0000","last_became_peered":"2024-05-09T08:31:04.857387+0000","last_unstale":"2024-05-09T08:33:20.835899+0000","last_undegraded":"2024-05-09T08:33:20.835899+0000","last_fullsized":"2024-05-09T08:33:20.835899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:50:16.555987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'45","reported_seq":102,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.044579+0000","last_change":"2024-05-09T08:30:59.827082+0000","last_active":"2024-05-09T08:33:21.044579+0000","last_peered":"2024-05-09T08:33:21.044579+0000","last_clean":"2024-05-09T08:33:21.044579+0000","last_became_active":"2024-05-09T08:30:59.826746+0000","last_became_peered":"2024-05-09T08:30:59.826746+0000","last_unstale":"2024-05-09T08:33:21.044579+0000","last_undegraded":"2024-05-09T08:33:21.044579+0000","last_fullsized":"2024-05-09T08:33:21.044579+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:49.176889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836291+0000","last_change":"2024-05-09T08:31:00.691902+0000","last_active":"2024-05-09T08:33:20.836291+0000","last_peered":"2024-05-09T08:33:20.836291+0000","last_clean":"2024-05-09T08:33:20.836291+0000","last_became_active":"2024-05-09T08:31:00.689491+0000","last_became_peered":"2024-05-09T08:31:00.689491+0000","last_unstale":"2024-05-09T08:33:20.836291+0000","last_undegraded":"2024-05-09T08:33:20.836291+0000","last_fullsized":"2024-05-09T08:33:20.836291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:48:07.335144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834367+0000","last_change":"2024-05-09T08:31:04.857788+0000","last_active":"2024-05-09T08:33:20.834367+0000","last_peered":"2024-05-09T08:33:20.834367+0000","last_clean":"2024-05-09T08:33:20.834367+0000","last_became_active":"2024-05-09T08:31:04.853831+0000","last_became_peered":"2024-05-09T08:31:04.853831+0000","last_unstale":"2024-05-09T08:33:20.834367+0000","last_undegraded":"2024-05-09T08:33:20.834367+0000","last_fullsized":"2024-05-09T08:33:20.834367+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:08:15.451606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.078037+0000","last_change":"2024-05-09T08:30:59.826482+0000","last_active":"2024-05-09T08:33:21.078037+0000","last_peered":"2024-05-09T08:33:21.078037+0000","last_clean":"2024-05-09T08:33:21.078037+0000","last_became_active":"2024-05-09T08:30:59.826123+0000","last_became_peered":"2024-05-09T08:30:59.826123+0000","last_unstale":"2024-05-09T08:33:21.078037+0000","last_undegraded":"2024-05-09T08:33:21.078037+0000","last_fullsized":"2024-05-09T08:33:21.078037+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:38.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224830+0000","last_change":"2024-05-09T08:31:00.693440+0000","last_active":"2024-05-09T08:33:21.224830+0000","last_peered":"2024-05-09T08:33:21.224830+0000","last_clean":"2024-05-09T08:33:21.224830+0000","last_became_active":"2024-05-09T08:31:00.693117+0000","last_became_peered":"2024-05-09T08:31:00.693117+0000","last_unstale":"2024-05-09T08:33:21.224830+0000","last_undegraded":"2024-05-09T08:33:21.224830+0000","last_fullsized":"2024-05-09T08:33:21.224830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:10:01.205605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077861+0000","last_change":"2024-05-09T08:31:04.868120+0000","last_active":"2024-05-09T08:33:21.077861+0000","last_peered":"2024-05-09T08:33:21.077861+0000","last_clean":"2024-05-09T08:33:21.077861+0000","last_became_active":"2024-05-09T08:31:04.867571+0000","last_became_peered":"2024-05-09T08:31:04.867571+0000","last_unstale":"2024-05-09T08:33:21.077861+0000","last_undegraded":"2024-05-09T08:33:21.077861+0000","last_fullsized":"2024-05-09T08:33:21.077861+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:12:36.503972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.923833+0000","last_change":"2024-05-09T08:30:59.826280+0000","last_active":"2024-05-09T08:33:20.923833+0000","last_peered":"2024-05-09T08:33:20.923833+0000","last_clean":"2024-05-09T08:33:20.923833+0000","last_became_active":"2024-05-09T08:30:59.825805+0000","last_became_peered":"2024-05-09T08:30:59.825805+0000","last_unstale":"2024-05-09T08:33:20.923833+0000","last_undegraded":"2024-05-09T08:33:20.923833+0000","last_fullsized":"2024-05-09T08:33:20.923833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:46:03.833159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933725+0000","last_change":"2024-05-09T08:31:00.687581+0000","last_active":"2024-05-09T08:33:20.933725+0000","last_peered":"2024-05-09T08:33:20.933725+0000","last_clean":"2024-05-09T08:33:20.933725+0000","last_became_active":"2024-05-09T08:31:00.684608+0000","last_became_peered":"2024-05-09T08:31:00.684608+0000","last_unstale":"2024-05-09T08:33:20.933725+0000","last_undegraded":"2024-05-09T08:33:20.933725+0000","last_fullsized":"2024-05-09T08:33:20.933725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:06:06.436840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.931830+0000","last_change":"2024-05-09T08:31:04.859399+0000","last_active":"2024-05-09T08:33:20.931830+0000","last_peered":"2024-05-09T08:33:20.931830+0000","last_clean":"2024-05-09T08:33:20.931830+0000","last_became_active":"2024-05-09T08:31:04.859027+0000","last_became_peered":"2024-05-09T08:31:04.859027+0000","last_unstale":"2024-05-09T08:33:20.931830+0000","last_undegraded":"2024-05-09T08:33:20.931830+0000","last_fullsized":"2024-05-09T08:33:20.931830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:18:00.829342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835967+0000","last_change":"2024-05-09T08:30:59.824171+0000","last_active":"2024-05-09T08:33:20.835967+0000","last_peered":"2024-05-09T08:33:20.835967+0000","last_clean":"2024-05-09T08:33:20.835967+0000","last_became_active":"2024-05-09T08:30:59.823859+0000","last_became_peered":"2024-05-09T08:30:59.823859+0000","last_unstale":"2024-05-09T08:33:20.835967+0000","last_undegraded":"2024-05-09T08:33:20.835967+0000","last_fullsized":"2024-05-09T08:33:20.835967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:30.199504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224030+0000","last_change":"2024-05-09T08:31:00.700056+0000","last_active":"2024-05-09T08:33:21.224030+0000","last_peered":"2024-05-09T08:33:21.224030+0000","last_clean":"2024-05-09T08:33:21.224030+0000","last_became_active":"2024-05-09T08:31:00.699673+0000","last_became_peered":"2024-05-09T08:31:00.699673+0000","last_unstale":"2024-05-09T08:33:21.224030+0000","last_undegraded":"2024-05-09T08:33:21.224030+0000","last_fullsized":"2024-05-09T08:33:21.224030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:23.873645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077627+0000","last_change":"2024-05-09T08:31:04.858940+0000","last_active":"2024-05-09T08:33:21.077627+0000","last_peered":"2024-05-09T08:33:21.077627+0000","last_clean":"2024-05-09T08:33:21.077627+0000","last_became_active":"2024-05-09T08:31:04.858532+0000","last_became_peered":"2024-05-09T08:31:04.858532+0000","last_unstale":"2024-05-09T08:33:21.077627+0000","last_undegraded":"2024-05-09T08:33:21.077627+0000","last_fullsized":"2024-05-09T08:33:21.077627+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:57:42.128068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834600+0000","last_change":"2024-05-09T08:30:59.824182+0000","last_active":"2024-05-09T08:33:20.834600+0000","last_peered":"2024-05-09T08:33:20.834600+0000","last_clean":"2024-05-09T08:33:20.834600+0000","last_became_active":"2024-05-09T08:30:59.823691+0000","last_became_peered":"2024-05-09T08:30:59.823691+0000","last_unstale":"2024-05-09T08:33:20.834600+0000","last_undegraded":"2024-05-09T08:33:20.834600+0000","last_fullsized":"2024-05-09T08:33:20.834600+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:28:07.810733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836255+0000","last_change":"2024-05-09T08:31:00.692614+0000","last_active":"2024-05-09T08:33:20.836255+0000","last_peered":"2024-05-09T08:33:20.836255+0000","last_clean":"2024-05-09T08:33:20.836255+0000","last_became_active":"2024-05-09T08:31:00.692280+0000","last_became_peered":"2024-05-09T08:31:00.692280+0000","last_unstale":"2024-05-09T08:33:20.836255+0000","last_undegraded":"2024-05-09T08:33:20.836255+0000","last_fullsized":"2024-05-09T08:33:20.836255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:23:15.398196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.077712+0000","last_change":"2024-05-09T08:31:04.854029+0000","last_active":"2024-05-09T08:33:21.077712+0000","last_peered":"2024-05-09T08:33:21.077712+0000","last_clean":"2024-05-09T08:33:21.077712+0000","last_became_active":"2024-05-09T08:31:04.853554+0000","last_became_peered":"2024-05-09T08:31:04.853554+0000","last_unstale":"2024-05-09T08:33:21.077712+0000","last_undegraded":"2024-05-09T08:33:21.077712+0000","last_fullsized":"2024-05-09T08:33:21.077712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:34:41.059614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.933921+0000","last_change":"2024-05-09T08:30:59.824512+0000","last_active":"2024-05-09T08:33:20.933921+0000","last_peered":"2024-05-09T08:33:20.933921+0000","last_clean":"2024-05-09T08:33:20.933921+0000","last_became_active":"2024-05-09T08:30:59.824146+0000","last_became_peered":"2024-05-09T08:30:59.824146+0000","last_unstale":"2024-05-09T08:33:20.933921+0000","last_undegraded":"2024-05-09T08:33:20.933921+0000","last_fullsized":"2024-05-09T08:33:20.933921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:27.734292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.074854+0000","last_change":"2024-05-09T08:31:00.697272+0000","last_active":"2024-05-09T08:33:21.074854+0000","last_peered":"2024-05-09T08:33:21.074854+0000","last_clean":"2024-05-09T08:33:21.074854+0000","last_became_active":"2024-05-09T08:31:00.696026+0000","last_became_peered":"2024-05-09T08:31:00.696026+0000","last_unstale":"2024-05-09T08:33:21.074854+0000","last_undegraded":"2024-05-09T08:33:21.074854+0000","last_fullsized":"2024-05-09T08:33:21.074854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:04.376142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.923977+0000","last_change":"2024-05-09T08:31:04.856641+0000","last_active":"2024-05-09T08:33:20.923977+0000","last_peered":"2024-05-09T08:33:20.923977+0000","last_clean":"2024-05-09T08:33:20.923977+0000","last_became_active":"2024-05-09T08:31:04.856102+0000","last_became_peered":"2024-05-09T08:31:04.856102+0000","last_unstale":"2024-05-09T08:33:20.923977+0000","last_undegraded":"2024-05-09T08:33:20.923977+0000","last_fullsized":"2024-05-09T08:33:20.923977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:35:49.831076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.923979+0000","last_change":"2024-05-09T08:31:56.579569+0000","last_active":"2024-05-09T08:33:20.923979+0000","last_peered":"2024-05-09T08:33:20.923979+0000","last_clean":"2024-05-09T08:33:20.923979+0000","last_became_active":"2024-05-09T08:31:56.573496+0000","last_became_peered":"2024-05-09T08:31:56.573496+0000","last_unstale":"2024-05-09T08:33:20.923979+0000","last_undegraded":"2024-05-09T08:33:20.923979+0000","last_fullsized":"2024-05-09T08:33:20.923979+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:47:47.944200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075274+0000","last_change":"2024-05-09T08:31:00.697876+0000","last_active":"2024-05-09T08:33:21.075274+0000","last_peered":"2024-05-09T08:33:21.075274+0000","last_clean":"2024-05-09T08:33:21.075274+0000","last_became_active":"2024-05-09T08:31:00.696299+0000","last_became_peered":"2024-05-09T08:31:00.696299+0000","last_unstale":"2024-05-09T08:33:21.075274+0000","last_undegraded":"2024-05-09T08:33:21.075274+0000","last_fullsized":"2024-05-09T08:33:21.075274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:48:57.373734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224675+0000","last_change":"2024-05-09T08:31:04.858588+0000","last_active":"2024-05-09T08:33:21.224675+0000","last_peered":"2024-05-09T08:33:21.224675+0000","last_clean":"2024-05-09T08:33:21.224675+0000","last_became_active":"2024-05-09T08:31:04.858154+0000","last_became_peered":"2024-05-09T08:31:04.858154+0000","last_unstale":"2024-05-09T08:33:21.224675+0000","last_undegraded":"2024-05-09T08:33:21.224675+0000","last_fullsized":"2024-05-09T08:33:21.224675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:06:01.768743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.043305+0000","last_change":"2024-05-09T08:30:59.826066+0000","last_active":"2024-05-09T08:33:21.043305+0000","last_peered":"2024-05-09T08:33:21.043305+0000","last_clean":"2024-05-09T08:33:21.043305+0000","last_became_active":"2024-05-09T08:30:59.825586+0000","last_became_peered":"2024-05-09T08:30:59.825586+0000","last_unstale":"2024-05-09T08:33:21.043305+0000","last_undegraded":"2024-05-09T08:33:21.043305+0000","last_fullsized":"2024-05-09T08:33:21.043305+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:19:04.629132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075338+0000","last_change":"2024-05-09T08:31:00.693788+0000","last_active":"2024-05-09T08:33:21.075338+0000","last_peered":"2024-05-09T08:33:21.075338+0000","last_clean":"2024-05-09T08:33:21.075338+0000","last_became_active":"2024-05-09T08:31:00.693448+0000","last_became_peered":"2024-05-09T08:31:00.693448+0000","last_unstale":"2024-05-09T08:33:21.075338+0000","last_undegraded":"2024-05-09T08:33:21.075338+0000","last_fullsized":"2024-05-09T08:33:21.075338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:06.516534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940422+0000","last_change":"2024-05-09T08:31:04.856185+0000","last_active":"2024-05-09T08:33:20.940422+0000","last_peered":"2024-05-09T08:33:20.940422+0000","last_clean":"2024-05-09T08:33:20.940422+0000","last_became_active":"2024-05-09T08:31:04.855779+0000","last_became_peered":"2024-05-09T08:31:04.855779+0000","last_unstale":"2024-05-09T08:33:20.940422+0000","last_undegraded":"2024-05-09T08:33:20.940422+0000","last_fullsized":"2024-05-09T08:33:20.940422+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:31:03.828736+0000","last_clean_scrub_stamp":"2024-05-09T08:31:03.828736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:45:25.034077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225735+0000","last_change":"2024-05-09T08:30:59.822938+0000","last_active":"2024-05-09T08:33:21.225735+0000","last_peered":"2024-05-09T08:33:21.225735+0000","last_clean":"2024-05-09T08:33:21.225735+0000","last_became_active":"2024-05-09T08:30:59.822629+0000","last_became_peered":"2024-05-09T08:30:59.822629+0000","last_unstale":"2024-05-09T08:33:21.225735+0000","last_undegraded":"2024-05-09T08:33:21.225735+0000","last_fullsized":"2024-05-09T08:33:21.225735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:58.799995+0000","last_clean_scrub_stamp":"2024-05-09T08:30:58.799995+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:51.321350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.075213+0000","last_change":"2024-05-09T08:31:00.691971+0000","last_active":"2024-05-09T08:33:21.075213+0000","last_peered":"2024-05-09T08:33:21.075213+0000","last_clean":"2024-05-09T08:33:21.075213+0000","last_became_active":"2024-05-09T08:31:00.691641+0000","last_became_peered":"2024-05-09T08:31:00.691641+0000","last_unstale":"2024-05-09T08:33:21.075213+0000","last_undegraded":"2024-05-09T08:33:21.075213+0000","last_fullsized":"2024-05-09T08:33:21.075213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:28.413886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224253+0000","last_change":"2024-05-09T08:31:00.701026+0000","last_active":"2024-05-09T08:33:21.224253+0000","last_peered":"2024-05-09T08:33:21.224253+0000","last_clean":"2024-05-09T08:33:21.224253+0000","last_became_active":"2024-05-09T08:31:00.700457+0000","last_became_peered":"2024-05-09T08:31:00.700457+0000","last_unstale":"2024-05-09T08:33:21.224253+0000","last_undegraded":"2024-05-09T08:33:21.224253+0000","last_fullsized":"2024-05-09T08:33:21.224253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:40.065173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834622+0000","last_change":"2024-05-09T08:31:56.570220+0000","last_active":"2024-05-09T08:33:20.834622+0000","last_peered":"2024-05-09T08:33:20.834622+0000","last_clean":"2024-05-09T08:33:20.834622+0000","last_became_active":"2024-05-09T08:31:56.569682+0000","last_became_peered":"2024-05-09T08:31:56.569682+0000","last_unstale":"2024-05-09T08:33:20.834622+0000","last_undegraded":"2024-05-09T08:33:20.834622+0000","last_fullsized":"2024-05-09T08:33:20.834622+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:47:07.986064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.834439+0000","last_change":"2024-05-09T08:31:00.703201+0000","last_active":"2024-05-09T08:33:20.834439+0000","last_peered":"2024-05-09T08:33:20.834439+0000","last_clean":"2024-05-09T08:33:20.834439+0000","last_became_active":"2024-05-09T08:31:00.702801+0000","last_became_peered":"2024-05-09T08:31:00.702801+0000","last_unstale":"2024-05-09T08:33:20.834439+0000","last_undegraded":"2024-05-09T08:33:20.834439+0000","last_fullsized":"2024-05-09T08:33:20.834439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:58:35.106013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224905+0000","last_change":"2024-05-09T08:31:00.693696+0000","last_active":"2024-05-09T08:33:21.224905+0000","last_peered":"2024-05-09T08:33:21.224905+0000","last_clean":"2024-05-09T08:33:21.224905+0000","last_became_active":"2024-05-09T08:31:00.693394+0000","last_became_peered":"2024-05-09T08:31:00.693394+0000","last_unstale":"2024-05-09T08:33:21.224905+0000","last_undegraded":"2024-05-09T08:33:21.224905+0000","last_fullsized":"2024-05-09T08:33:21.224905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:01.700968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225308+0000","last_change":"2024-05-09T08:31:00.694363+0000","last_active":"2024-05-09T08:33:21.225308+0000","last_peered":"2024-05-09T08:33:21.225308+0000","last_clean":"2024-05-09T08:33:21.225308+0000","last_became_active":"2024-05-09T08:31:00.693970+0000","last_became_peered":"2024-05-09T08:31:00.693970+0000","last_unstale":"2024-05-09T08:33:21.225308+0000","last_undegraded":"2024-05-09T08:33:21.225308+0000","last_fullsized":"2024-05-09T08:33:21.225308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:56.185801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224406+0000","last_change":"2024-05-09T08:31:00.691970+0000","last_active":"2024-05-09T08:33:21.224406+0000","last_peered":"2024-05-09T08:33:21.224406+0000","last_clean":"2024-05-09T08:33:21.224406+0000","last_became_active":"2024-05-09T08:31:00.691632+0000","last_became_peered":"2024-05-09T08:31:00.691632+0000","last_unstale":"2024-05-09T08:33:21.224406+0000","last_undegraded":"2024-05-09T08:33:21.224406+0000","last_fullsized":"2024-05-09T08:33:21.224406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:04:49.316808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.835398+0000","last_change":"2024-05-09T08:31:00.692042+0000","last_active":"2024-05-09T08:33:20.835398+0000","last_peered":"2024-05-09T08:33:20.835398+0000","last_clean":"2024-05-09T08:33:20.835398+0000","last_became_active":"2024-05-09T08:31:00.691700+0000","last_became_peered":"2024-05-09T08:31:00.691700+0000","last_unstale":"2024-05-09T08:33:20.835398+0000","last_undegraded":"2024-05-09T08:33:20.835398+0000","last_fullsized":"2024-05-09T08:33:20.835398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:54.236319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836506+0000","last_change":"2024-05-09T08:31:00.697261+0000","last_active":"2024-05-09T08:33:20.836506+0000","last_peered":"2024-05-09T08:33:20.836506+0000","last_clean":"2024-05-09T08:33:20.836506+0000","last_became_active":"2024-05-09T08:31:00.696299+0000","last_became_peered":"2024-05-09T08:31:00.696299+0000","last_unstale":"2024-05-09T08:33:20.836506+0000","last_undegraded":"2024-05-09T08:33:20.836506+0000","last_fullsized":"2024-05-09T08:33:20.836506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:06:52.061740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.940597+0000","last_change":"2024-05-09T08:31:00.691097+0000","last_active":"2024-05-09T08:33:20.940597+0000","last_peered":"2024-05-09T08:33:20.940597+0000","last_clean":"2024-05-09T08:33:20.940597+0000","last_became_active":"2024-05-09T08:31:00.690487+0000","last_became_peered":"2024-05-09T08:31:00.690487+0000","last_unstale":"2024-05-09T08:33:20.940597+0000","last_undegraded":"2024-05-09T08:33:20.940597+0000","last_fullsized":"2024-05-09T08:33:20.940597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:47:26.468296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.836151+0000","last_change":"2024-05-09T08:31:00.703201+0000","last_active":"2024-05-09T08:33:20.836151+0000","last_peered":"2024-05-09T08:33:20.836151+0000","last_clean":"2024-05-09T08:33:20.836151+0000","last_became_active":"2024-05-09T08:31:00.702853+0000","last_became_peered":"2024-05-09T08:31:00.702853+0000","last_unstale":"2024-05-09T08:33:20.836151+0000","last_undegraded":"2024-05-09T08:33:20.836151+0000","last_fullsized":"2024-05-09T08:33:20.836151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:18.510321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:20.924123+0000","last_change":"2024-05-09T08:31:00.703958+0000","last_active":"2024-05-09T08:33:20.924123+0000","last_peered":"2024-05-09T08:33:20.924123+0000","last_clean":"2024-05-09T08:33:20.924123+0000","last_became_active":"2024-05-09T08:31:00.703110+0000","last_became_peered":"2024-05-09T08:31:00.703110+0000","last_unstale":"2024-05-09T08:33:20.924123+0000","last_undegraded":"2024-05-09T08:33:20.924123+0000","last_fullsized":"2024-05-09T08:33:20.924123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:37:32.371869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.225122+0000","last_change":"2024-05-09T08:31:00.686763+0000","last_active":"2024-05-09T08:33:21.225122+0000","last_peered":"2024-05-09T08:33:21.225122+0000","last_clean":"2024-05-09T08:33:21.225122+0000","last_became_active":"2024-05-09T08:31:00.685642+0000","last_became_peered":"2024-05-09T08:31:00.685642+0000","last_unstale":"2024-05-09T08:33:21.225122+0000","last_undegraded":"2024-05-09T08:33:21.225122+0000","last_fullsized":"2024-05-09T08:33:21.225122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:40:18.201247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-09T08:33:21.224401+0000","last_change":"2024-05-09T08:31:56.573837+0000","last_active":"2024-05-09T08:33:21.224401+0000","last_peered":"2024-05-09T08:33:21.224401+0000","last_clean":"2024-05-09T08:33:21.224401+0000","last_became_active":"2024-05-09T08:31:56.573408+0000","last_became_peered":"2024-05-09T08:31:56.573408+0000","last_unstale":"2024-05-09T08:33:21.224401+0000","last_undegraded":"2024-05-09T08:33:21.224401+0000","last_fullsized":"2024-05-09T08:33:21.224401+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T08:30:59.802114+0000","last_clean_scrub_stamp":"2024-05-09T08:30:59.802114+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:12:46.397877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600702,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":851,"num_read_kb":537,"num_write":294,"num_write_kb":1446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1186,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1802106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383721,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":529353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645357,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28880,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716272,"statfs":{"total":95995035648,"available":95965462528,"internally_reserved":0,"allocated":1908736,"data_stored":1323296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28472,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716680,"statfs":{"total":95995035648,"available":95965880320,"internally_reserved":0,"allocated":1486848,"data_stored":926879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36780,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708372,"statfs":{"total":95995035648,"available":95957372928,"internally_reserved":0,"allocated":1474560,"data_stored":922099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36984,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708168,"statfs":{"total":95995035648,"available":95957164032,"internally_reserved":0,"allocated":1683456,"data_stored":1128087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":514906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28140,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717012,"statfs":{"total":95995035648,"available":95966220288,"internally_reserved":0,"allocated":1146880,"data_stored":594429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1093632,"data_stored":526131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36476,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708676,"statfs":{"total":95995035648,"available":95957684224,"internally_reserved":0,"allocated":1163264,"data_stored":595890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36908,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708244,"statfs":{"total":95995035648,"available":95957241856,"internally_reserved":0,"allocated":1544192,"data_stored":984209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1085440,"data_stored":531782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37116,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708036,"statfs":{"total":95995035648,"available":95957028864,"internally_reserved":0,"allocated":1822720,"data_stored":1249372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":69776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":134351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":393350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":204179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T08:33:26.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:33:26.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.682+0000 7f0022baf640 1 -- 172.21.15.107:0/825921552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001c102a90 msgr2=0x7f001c102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.682+0000 7f0022baf640 1 --2- 172.21.15.107:0/825921552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001c102a90 0x7f001c102e70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0018009900 tx=0x7f001802a480 comp rx=0 tx=0).stop 2024-05-09T08:33:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.682+0000 7f0022baf640 1 -- 172.21.15.107:0/825921552 shutdown_connections 2024-05-09T08:33:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.682+0000 7f0022baf640 1 --2- 172.21.15.107:0/825921552 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f001c10be60 0x7f001c10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.682+0000 7f0022baf640 1 --2- 172.21.15.107:0/825921552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f001c1033b0 0x7f001c10b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.682+0000 7f0022baf640 1 --2- 172.21.15.107:0/825921552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001c102a90 0x7f001c102e70 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.682+0000 7f0022baf640 1 -- 172.21.15.107:0/825921552 >> 172.21.15.107:0/825921552 conn(0x7f001c076ab0 msgr2=0x7f001c076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.682+0000 7f0022baf640 1 -- 172.21.15.107:0/825921552 shutdown_connections 2024-05-09T08:33:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.682+0000 7f0022baf640 1 -- 172.21.15.107:0/825921552 wait complete. 2024-05-09T08:33:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f0022baf640 1 Processor -- start 2024-05-09T08:33:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f0022baf640 1 -- start start 2024-05-09T08:33:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f0022baf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f001c102a90 0x7f001c1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f0022baf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f001c1033b0 0x7f001c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:26.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f0022baf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001c10be60 0x7f001c19c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f0022baf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f001c111b20 con 0x7f001c10be60 2024-05-09T08:33:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f0022baf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f001c1119a0 con 0x7f001c1033b0 2024-05-09T08:33:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f0022baf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f001c111ca0 con 0x7f001c102a90 2024-05-09T08:33:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f002114c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001c10be60 0x7f001c19c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f002114c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001c10be60 0x7f001c19c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50476/0 (socket says 172.21.15.107:50476) 2024-05-09T08:33:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f002114c640 1 -- 172.21.15.107:0/1778776556 learned_addr learned my addr 172.21.15.107:0/1778776556 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f002094b640 1 --2- 172.21.15.107:0/1778776556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f001c102a90 0x7f001c1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f0013fff640 1 --2- 172.21.15.107:0/1778776556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f001c1033b0 0x7f001c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f002114c640 1 -- 172.21.15.107:0/1778776556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f001c102a90 msgr2=0x7f001c1980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f002114c640 1 --2- 172.21.15.107:0/1778776556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f001c102a90 0x7f001c1980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f002114c640 1 -- 172.21.15.107:0/1778776556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f001c1033b0 msgr2=0x7f001c1985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f002114c640 1 --2- 172.21.15.107:0/1778776556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f001c1033b0 0x7f001c1985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f002114c640 1 -- 172.21.15.107:0/1778776556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f001c19d090 con 0x7f001c10be60 2024-05-09T08:33:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f0013fff640 1 --2- 172.21.15.107:0/1778776556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f001c1033b0 0x7f001c1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:26.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.683+0000 7f002114c640 1 --2- 172.21.15.107:0/1778776556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001c10be60 0x7f001c19c960 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f001400e9d0 tx=0x7f001400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.684+0000 7f0011ffb640 1 -- 172.21.15.107:0/1778776556 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001400cd80 con 0x7f001c10be60 2024-05-09T08:33:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.684+0000 7f0011ffb640 1 -- 172.21.15.107:0/1778776556 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0014005020 con 0x7f001c10be60 2024-05-09T08:33:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.684+0000 7f0022baf640 1 -- 172.21.15.107:0/1778776556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f001c19d380 con 0x7f001c10be60 2024-05-09T08:33:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.684+0000 7f0011ffb640 1 -- 172.21.15.107:0/1778776556 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00140116d0 con 0x7f001c10be60 2024-05-09T08:33:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.684+0000 7f0022baf640 1 -- 172.21.15.107:0/1778776556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f001c1a4c10 con 0x7f001c10be60 2024-05-09T08:33:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.685+0000 7f0022baf640 1 -- 172.21.15.107:0/1778776556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f001c068aa0 con 0x7f001c10be60 2024-05-09T08:33:26.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.686+0000 7f0011ffb640 1 -- 172.21.15.107:0/1778776556 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7f00140051d0 con 0x7f001c10be60 2024-05-09T08:33:26.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.686+0000 7f0011ffb640 1 --2- 172.21.15.107:0/1778776556 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f00040a2e20 0x7f00040a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:26.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.686+0000 7f002094b640 1 --2- 172.21.15.107:0/1778776556 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f00040a2e20 0x7f00040a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:26.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.687+0000 7f0011ffb640 1 -- 172.21.15.107:0/1778776556 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f00140cffb0 con 0x7f001c10be60 2024-05-09T08:33:26.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.687+0000 7f002094b640 1 --2- 172.21.15.107:0/1778776556 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f00040a2e20 0x7f00040a52e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0018031440 tx=0x7f0018005b50 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:26.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.691+0000 7f0011ffb640 1 -- 172.21.15.107:0/1778776556 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0014016030 con 0x7f001c10be60 2024-05-09T08:33:26.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:26 smithi157 ceph-mon[33614]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:33:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:26 smithi157 ceph-mon[33614]: Updating smithi157:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:33:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:26 smithi157 ceph-mon[33614]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:33:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:26 smithi157 ceph-mon[33614]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:33:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:26 smithi157 ceph-mon[33614]: Updating smithi110:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:33:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:26 smithi157 ceph-mon[33614]: Updating smithi157:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:33:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:26 smithi157 ceph-mon[33614]: Updating smithi107:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:33:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:26 smithi157 ceph-mon[33614]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-05-09T08:33:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:26 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:33:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:26 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:33:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:26 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:33:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:26 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:26 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1439959666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:33:26.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.870+0000 7f0022baf640 1 -- 172.21.15.107:0/1778776556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f001c1069c0 con 0x7f001c10be60 2024-05-09T08:33:26.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.872+0000 7f0011ffb640 1 -- 172.21.15.107:0/1778776556 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25143 (secure 0 0 0) 0x7f001409c620 con 0x7f001c10be60 2024-05-09T08:33:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:26 smithi110 ceph-mon[33622]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:33:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:26 smithi110 ceph-mon[33622]: Updating smithi157:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:33:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:26 smithi110 ceph-mon[33622]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-05-09T08:33:26.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:26 smithi110 ceph-mon[33622]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T08:33:26.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:26 smithi110 ceph-mon[33622]: Updating smithi110:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:33:26.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:26 smithi110 ceph-mon[33622]: Updating smithi157:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:33:26.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:26 smithi110 ceph-mon[33622]: Updating smithi107:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T08:33:26.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:26 smithi110 ceph-mon[33622]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-05-09T08:33:26.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:26 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:33:26.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:26 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:33:26.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:26 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:33:26.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:26 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:33:26.872 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:26 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1439959666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:33:26.872 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:26.873 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":94,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:33:20.828605+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/2923766500":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/2417648229":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1642826815":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/2689615560":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/846948416":"2024-05-10T08:33:20.828572+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/846948416":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:33:26.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.874+0000 7f0022baf640 1 -- 172.21.15.107:0/1778776556 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f00040a2e20 msgr2=0x7f00040a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:26.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.874+0000 7f0022baf640 1 --2- 172.21.15.107:0/1778776556 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f00040a2e20 0x7f00040a52e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0018031440 tx=0x7f0018005b50 comp rx=0 tx=0).stop 2024-05-09T08:33:26.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.874+0000 7f0022baf640 1 -- 172.21.15.107:0/1778776556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001c10be60 msgr2=0x7f001c19c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:26.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.874+0000 7f0022baf640 1 --2- 172.21.15.107:0/1778776556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001c10be60 0x7f001c19c960 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f001400e9d0 tx=0x7f001400ee90 comp rx=0 tx=0).stop 2024-05-09T08:33:26.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.874+0000 7f0022baf640 1 -- 172.21.15.107:0/1778776556 shutdown_connections 2024-05-09T08:33:26.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.874+0000 7f0022baf640 1 --2- 172.21.15.107:0/1778776556 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f00040a2e20 0x7f00040a52e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.874+0000 7f0022baf640 1 --2- 172.21.15.107:0/1778776556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001c10be60 0x7f001c19c960 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.874+0000 7f0022baf640 1 --2- 172.21.15.107:0/1778776556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f001c1033b0 0x7f001c1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.874+0000 7f0022baf640 1 --2- 172.21.15.107:0/1778776556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f001c102a90 0x7f001c1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.874+0000 7f0022baf640 1 -- 172.21.15.107:0/1778776556 >> 172.21.15.107:0/1778776556 conn(0x7f001c076ab0 msgr2=0x7f001c0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:26.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.874+0000 7f0022baf640 1 -- 172.21.15.107:0/1778776556 shutdown_connections 2024-05-09T08:33:26.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.874+0000 7f0022baf640 1 -- 172.21.15.107:0/1778776556 wait complete. 2024-05-09T08:33:26.885 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:26:21.698338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:30:58.268633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:30:59.345579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:31:03.065991+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:33:26.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:33:26.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.990+0000 7fb5fb4df640 1 -- 172.21.15.107:0/4232356234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5f4100710 msgr2=0x7fb5f4100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:26.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.990+0000 7fb5fb4df640 1 --2- 172.21.15.107:0/4232356234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5f4100710 0x7fb5f4100b90 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8009930 tx=0x7fb5e802a420 comp rx=0 tx=0).stop 2024-05-09T08:33:26.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.990+0000 7fb5fb4df640 1 -- 172.21.15.107:0/4232356234 shutdown_connections 2024-05-09T08:33:26.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.990+0000 7fb5fb4df640 1 --2- 172.21.15.107:0/4232356234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5f41010d0 0x7fb5f410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.990+0000 7fb5fb4df640 1 --2- 172.21.15.107:0/4232356234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5f4100710 0x7fb5f4100b90 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.990+0000 7fb5fb4df640 1 --2- 172.21.15.107:0/4232356234 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5f4106720 0x7fb5f4106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.990+0000 7fb5fb4df640 1 -- 172.21.15.107:0/4232356234 >> 172.21.15.107:0/4232356234 conn(0x7fb5f4077790 msgr2=0x7fb5f4077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.991+0000 7fb5fb4df640 1 -- 172.21.15.107:0/4232356234 shutdown_connections 2024-05-09T08:33:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.991+0000 7fb5fb4df640 1 -- 172.21.15.107:0/4232356234 wait complete. 2024-05-09T08:33:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.991+0000 7fb5fb4df640 1 Processor -- start 2024-05-09T08:33:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.991+0000 7fb5fb4df640 1 -- start start 2024-05-09T08:33:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.991+0000 7fb5fb4df640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5f4100710 0x7fb5f4198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.991+0000 7fb5fb4df640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5f41010d0 0x7fb5f41985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.991+0000 7fb5fb4df640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5f4106720 0x7fb5f419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.991+0000 7fb5fb4df640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f4111ae0 con 0x7fb5f4106720 2024-05-09T08:33:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.991+0000 7fb5fb4df640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f4111960 con 0x7fb5f4100710 2024-05-09T08:33:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.991+0000 7fb5fb4df640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f4111c60 con 0x7fb5f41010d0 2024-05-09T08:33:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.991+0000 7fb5f9a7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5f4106720 0x7fb5f419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.991+0000 7fb5f9a7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5f4106720 0x7fb5f419c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50488/0 (socket says 172.21.15.107:50488) 2024-05-09T08:33:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.991+0000 7fb5f9a7c640 1 -- 172.21.15.107:0/1442921795 learned_addr learned my addr 172.21.15.107:0/1442921795 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.992+0000 7fb5f927b640 1 --2- 172.21.15.107:0/1442921795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5f4100710 0x7fb5f4198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.992+0000 7fb5f8a7a640 1 --2- 172.21.15.107:0/1442921795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5f41010d0 0x7fb5f41985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.992+0000 7fb5f9a7c640 1 -- 172.21.15.107:0/1442921795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5f41010d0 msgr2=0x7fb5f41985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.992+0000 7fb5f9a7c640 1 --2- 172.21.15.107:0/1442921795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5f41010d0 0x7fb5f41985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.992+0000 7fb5f9a7c640 1 -- 172.21.15.107:0/1442921795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5f4100710 msgr2=0x7fb5f4198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.992+0000 7fb5f9a7c640 1 --2- 172.21.15.107:0/1442921795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5f4100710 0x7fb5f4198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.992+0000 7fb5f9a7c640 1 -- 172.21.15.107:0/1442921795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f419d000 con 0x7fb5f4106720 2024-05-09T08:33:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.992+0000 7fb5f8a7a640 1 --2- 172.21.15.107:0/1442921795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5f41010d0 0x7fb5f41985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.992+0000 7fb5f9a7c640 1 --2- 172.21.15.107:0/1442921795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5f4106720 0x7fb5f419c8d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec009fd0 tx=0x7fb5ec00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.992+0000 7fb5f27fc640 1 -- 172.21.15.107:0/1442921795 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec017070 con 0x7fb5f4106720 2024-05-09T08:33:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.992+0000 7fb5fb4df640 1 -- 172.21.15.107:0/1442921795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5f419d2f0 con 0x7fb5f4106720 2024-05-09T08:33:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.992+0000 7fb5f27fc640 1 -- 172.21.15.107:0/1442921795 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb5ec003e20 con 0x7fb5f4106720 2024-05-09T08:33:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.992+0000 7fb5f27fc640 1 -- 172.21.15.107:0/1442921795 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec010460 con 0x7fb5f4106720 2024-05-09T08:33:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.992+0000 7fb5fb4df640 1 -- 172.21.15.107:0/1442921795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5f41a4b80 con 0x7fb5f4106720 2024-05-09T08:33:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.993+0000 7fb5fb4df640 1 -- 172.21.15.107:0/1442921795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5f4068aa0 con 0x7fb5f4106720 2024-05-09T08:33:26.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.994+0000 7fb5f27fc640 1 -- 172.21.15.107:0/1442921795 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7fb5ec003970 con 0x7fb5f4106720 2024-05-09T08:33:26.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.995+0000 7fb5f27fc640 1 --2- 172.21.15.107:0/1442921795 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb5dc0a2e20 0x7fb5dc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:26.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.995+0000 7fb5f927b640 1 --2- 172.21.15.107:0/1442921795 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb5dc0a2e20 0x7fb5dc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:26.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.995+0000 7fb5f27fc640 1 -- 172.21.15.107:0/1442921795 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb5ec0cf890 con 0x7fb5f4106720 2024-05-09T08:33:26.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:26.996+0000 7fb5f927b640 1 --2- 172.21.15.107:0/1442921795 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb5dc0a2e20 0x7fb5dc0a52e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4005d00 tx=0x7fb5e400b410 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:26.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.000+0000 7fb5f27fc640 1 -- 172.21.15.107:0/1442921795 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb5ec09bf00 con 0x7fb5f4106720 2024-05-09T08:33:27.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.176+0000 7fb5fb4df640 1 -- 172.21.15.107:0/1442921795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb5f4108ce0 con 0x7fb5f4106720 2024-05-09T08:33:27.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.176+0000 7fb5f27fc640 1 -- 172.21.15.107:0/1442921795 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fb5ec09b800 con 0x7fb5f4106720 2024-05-09T08:33:27.176 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-05-09T08:33:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.178+0000 7fb5fb4df640 1 -- 172.21.15.107:0/1442921795 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb5dc0a2e20 msgr2=0x7fb5dc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.178+0000 7fb5fb4df640 1 --2- 172.21.15.107:0/1442921795 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb5dc0a2e20 0x7fb5dc0a52e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4005d00 tx=0x7fb5e400b410 comp rx=0 tx=0).stop 2024-05-09T08:33:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.178+0000 7fb5fb4df640 1 -- 172.21.15.107:0/1442921795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5f4106720 msgr2=0x7fb5f419c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.178+0000 7fb5fb4df640 1 --2- 172.21.15.107:0/1442921795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5f4106720 0x7fb5f419c8d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec009fd0 tx=0x7fb5ec00b260 comp rx=0 tx=0).stop 2024-05-09T08:33:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.178+0000 7fb5fb4df640 1 -- 172.21.15.107:0/1442921795 shutdown_connections 2024-05-09T08:33:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.178+0000 7fb5fb4df640 1 --2- 172.21.15.107:0/1442921795 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb5dc0a2e20 0x7fb5dc0a52e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.178+0000 7fb5fb4df640 1 --2- 172.21.15.107:0/1442921795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5f4106720 0x7fb5f419c8d0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.178+0000 7fb5fb4df640 1 --2- 172.21.15.107:0/1442921795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5f41010d0 0x7fb5f41985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.178+0000 7fb5fb4df640 1 --2- 172.21.15.107:0/1442921795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5f4100710 0x7fb5f4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.178+0000 7fb5fb4df640 1 -- 172.21.15.107:0/1442921795 >> 172.21.15.107:0/1442921795 conn(0x7fb5f4077790 msgr2=0x7fb5f4103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:27.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.178+0000 7fb5fb4df640 1 -- 172.21.15.107:0/1442921795 shutdown_connections 2024-05-09T08:33:27.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.178+0000 7fb5fb4df640 1 -- 172.21.15.107:0/1442921795 wait complete. 2024-05-09T08:33:27.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:33:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.293+0000 7f2ccd6f7640 1 -- 172.21.15.107:0/3965401289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc8106b90 msgr2=0x7f2cc810e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.293+0000 7f2ccd6f7640 1 --2- 172.21.15.107:0/3965401289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc8106b90 0x7f2cc810e7a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0009ca0 tx=0x7f2cc002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.293+0000 7f2ccd6f7640 1 -- 172.21.15.107:0/3965401289 shutdown_connections 2024-05-09T08:33:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.293+0000 7f2ccd6f7640 1 --2- 172.21.15.107:0/3965401289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc8106b90 0x7f2cc810e7a0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.293+0000 7f2ccd6f7640 1 --2- 172.21.15.107:0/3965401289 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2cc8100d80 0x7f2cc81064b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.293+0000 7f2ccd6f7640 1 --2- 172.21.15.107:0/3965401289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc81003d0 0x7f2cc81007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.293+0000 7f2ccd6f7640 1 -- 172.21.15.107:0/3965401289 >> 172.21.15.107:0/3965401289 conn(0x7f2cc80759c0 msgr2=0x7f2cc8075dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.293+0000 7f2ccd6f7640 1 -- 172.21.15.107:0/3965401289 shutdown_connections 2024-05-09T08:33:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.293+0000 7f2ccd6f7640 1 -- 172.21.15.107:0/3965401289 wait complete. 2024-05-09T08:33:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.294+0000 7f2ccd6f7640 1 Processor -- start 2024-05-09T08:33:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.294+0000 7f2ccd6f7640 1 -- start start 2024-05-09T08:33:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.294+0000 7f2ccd6f7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2cc81003d0 0x7f2cc8198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.294+0000 7f2ccd6f7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8106b90 0x7f2cc8198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.294+0000 7f2ccd6f7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc819c910 0x7f2cc819cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.294+0000 7f2ccd6f7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc8111b30 con 0x7f2cc819c910 2024-05-09T08:33:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.294+0000 7f2ccd6f7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc81119b0 con 0x7f2cc8106b90 2024-05-09T08:33:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.294+0000 7f2ccd6f7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc8111cb0 con 0x7f2cc81003d0 2024-05-09T08:33:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.294+0000 7f2cc77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc819c910 0x7f2cc819cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.295+0000 7f2cc77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc819c910 0x7f2cc819cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50494/0 (socket says 172.21.15.107:50494) 2024-05-09T08:33:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.295+0000 7f2cc77fe640 1 -- 172.21.15.107:0/3259364600 learned_addr learned my addr 172.21.15.107:0/3259364600 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.295+0000 7f2cc6ffd640 1 --2- 172.21.15.107:0/3259364600 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2cc81003d0 0x7f2cc8198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.295+0000 7f2cbe7fc640 1 --2- 172.21.15.107:0/3259364600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8106b90 0x7f2cc8198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.295+0000 7f2cc77fe640 1 -- 172.21.15.107:0/3259364600 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2cc81003d0 msgr2=0x7f2cc8198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.295+0000 7f2cc77fe640 1 --2- 172.21.15.107:0/3259364600 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2cc81003d0 0x7f2cc8198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.295+0000 7f2cc77fe640 1 -- 172.21.15.107:0/3259364600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8106b90 msgr2=0x7f2cc8198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.295+0000 7f2cc77fe640 1 --2- 172.21.15.107:0/3259364600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8106b90 0x7f2cc8198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.295+0000 7f2cc77fe640 1 -- 172.21.15.107:0/3259364600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc819d400 con 0x7f2cc819c910 2024-05-09T08:33:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.295+0000 7f2cc77fe640 1 --2- 172.21.15.107:0/3259364600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc819c910 0x7f2cc819cdc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0009dc0 tx=0x7f2cc0032ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.295+0000 7f2cc4ff9640 1 -- 172.21.15.107:0/3259364600 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0003bf0 con 0x7f2cc819c910 2024-05-09T08:33:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.295+0000 7f2ccd6f7640 1 -- 172.21.15.107:0/3259364600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc819d690 con 0x7f2cc819c910 2024-05-09T08:33:27.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.295+0000 7f2cc4ff9640 1 -- 172.21.15.107:0/3259364600 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2cc0003d90 con 0x7f2cc819c910 2024-05-09T08:33:27.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.295+0000 7f2cc4ff9640 1 -- 172.21.15.107:0/3259364600 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0030b80 con 0x7f2cc819c910 2024-05-09T08:33:27.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.295+0000 7f2ccd6f7640 1 -- 172.21.15.107:0/3259364600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc81a4eb0 con 0x7f2cc819c910 2024-05-09T08:33:27.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.296+0000 7f2ccd6f7640 1 -- 172.21.15.107:0/3259364600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cc8068aa0 con 0x7f2cc819c910 2024-05-09T08:33:27.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.298+0000 7f2cc4ff9640 1 -- 172.21.15.107:0/3259364600 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7f2cc0004040 con 0x7f2cc819c910 2024-05-09T08:33:27.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.298+0000 7f2cc4ff9640 1 --2- 172.21.15.107:0/3259364600 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2cb40a2e20 0x7f2cb40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:27.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.298+0000 7f2cc6ffd640 1 --2- 172.21.15.107:0/3259364600 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2cb40a2e20 0x7f2cb40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:27.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.299+0000 7f2cc4ff9640 1 -- 172.21.15.107:0/3259364600 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f2cc00f1150 con 0x7f2cc819c910 2024-05-09T08:33:27.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.299+0000 7f2cc6ffd640 1 --2- 172.21.15.107:0/3259364600 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2cb40a2e20 0x7f2cb40a52e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009bd0 tx=0x7f2cb8005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:27.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.302+0000 7f2cc4ff9640 1 -- 172.21.15.107:0/3259364600 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2cc00bd8d0 con 0x7f2cc819c910 2024-05-09T08:33:27.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.477+0000 7f2ccd6f7640 1 -- 172.21.15.107:0/3259364600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2cc8100d80 con 0x7f2cc819c910 2024-05-09T08:33:27.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.477+0000 7f2cc4ff9640 1 -- 172.21.15.107:0/3259364600 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f2cc00bd1d0 con 0x7f2cc819c910 2024-05-09T08:33:27.476 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:33:27.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.479+0000 7f2ccd6f7640 1 -- 172.21.15.107:0/3259364600 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2cb40a2e20 msgr2=0x7f2cb40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:27.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.479+0000 7f2ccd6f7640 1 --2- 172.21.15.107:0/3259364600 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2cb40a2e20 0x7f2cb40a52e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009bd0 tx=0x7f2cb8005f90 comp rx=0 tx=0).stop 2024-05-09T08:33:27.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.479+0000 7f2ccd6f7640 1 -- 172.21.15.107:0/3259364600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc819c910 msgr2=0x7f2cc819cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:27.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.479+0000 7f2ccd6f7640 1 --2- 172.21.15.107:0/3259364600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc819c910 0x7f2cc819cdc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0009dc0 tx=0x7f2cc0032ff0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.479+0000 7f2ccd6f7640 1 -- 172.21.15.107:0/3259364600 shutdown_connections 2024-05-09T08:33:27.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.479+0000 7f2ccd6f7640 1 --2- 172.21.15.107:0/3259364600 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2cb40a2e20 0x7f2cb40a52e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.479+0000 7f2ccd6f7640 1 --2- 172.21.15.107:0/3259364600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cc819c910 0x7f2cc819cdc0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.479+0000 7f2ccd6f7640 1 --2- 172.21.15.107:0/3259364600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8106b90 0x7f2cc8198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.479+0000 7f2ccd6f7640 1 --2- 172.21.15.107:0/3259364600 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2cc81003d0 0x7f2cc8198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.479+0000 7f2ccd6f7640 1 -- 172.21.15.107:0/3259364600 >> 172.21.15.107:0/3259364600 conn(0x7f2cc80759c0 msgr2=0x7f2cc80ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:27.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.479+0000 7f2ccd6f7640 1 -- 172.21.15.107:0/3259364600 shutdown_connections 2024-05-09T08:33:27.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.479+0000 7f2ccd6f7640 1 -- 172.21.15.107:0/3259364600 wait complete. 2024-05-09T08:33:27.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:27 smithi107 ceph-mon[30122]: from='client.15639 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:33:27.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:27 smithi107 ceph-mon[30122]: from='client.15645 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:33:27.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:27 smithi107 ceph-mon[30122]: from='client.15651 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:33:27.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:27 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1778776556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:27.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:27 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1442921795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:33:27.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:33:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.594+0000 7f5ea02cd640 1 -- 172.21.15.107:0/4290014573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e981010c0 msgr2=0x7f5e9810d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.594+0000 7f5ea02cd640 1 --2- 172.21.15.107:0/4290014573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e981010c0 0x7f5e9810d5d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5e90009cd0 tx=0x7f5e9002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.595+0000 7f5ea02cd640 1 -- 172.21.15.107:0/4290014573 shutdown_connections 2024-05-09T08:33:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.595+0000 7f5ea02cd640 1 --2- 172.21.15.107:0/4290014573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e981010c0 0x7f5e9810d5d0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.595+0000 7f5ea02cd640 1 --2- 172.21.15.107:0/4290014573 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e98100700 0x7f5e98100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.595+0000 7f5ea02cd640 1 --2- 172.21.15.107:0/4290014573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e98106710 0x7f5e98106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.595+0000 7f5ea02cd640 1 -- 172.21.15.107:0/4290014573 >> 172.21.15.107:0/4290014573 conn(0x7f5e98077830 msgr2=0x7f5e98077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.595+0000 7f5ea02cd640 1 -- 172.21.15.107:0/4290014573 shutdown_connections 2024-05-09T08:33:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.595+0000 7f5ea02cd640 1 -- 172.21.15.107:0/4290014573 wait complete. 2024-05-09T08:33:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.595+0000 7f5ea02cd640 1 Processor -- start 2024-05-09T08:33:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.595+0000 7f5ea02cd640 1 -- start start 2024-05-09T08:33:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.595+0000 7f5ea02cd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e98100700 0x7f5e98197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.595+0000 7f5ea02cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e981010c0 0x7f5e981984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.595+0000 7f5ea02cd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e98106710 0x7f5e9819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.595+0000 7f5ea02cd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5e98111ad0 con 0x7f5e981010c0 2024-05-09T08:33:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.595+0000 7f5ea02cd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5e98111950 con 0x7f5e98100700 2024-05-09T08:33:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.595+0000 7f5ea02cd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5e98111c50 con 0x7f5e98106710 2024-05-09T08:33:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.595+0000 7f5e9d868640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e981010c0 0x7f5e981984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.596+0000 7f5e9d868640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e981010c0 0x7f5e981984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50520/0 (socket says 172.21.15.107:50520) 2024-05-09T08:33:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.596+0000 7f5e9d868640 1 -- 172.21.15.107:0/935590617 learned_addr learned my addr 172.21.15.107:0/935590617 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:27.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.596+0000 7f5e9e86a640 1 --2- 172.21.15.107:0/935590617 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e98106710 0x7f5e9819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.596+0000 7f5e9e069640 1 --2- 172.21.15.107:0/935590617 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e98100700 0x7f5e98197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.596+0000 7f5e9d868640 1 -- 172.21.15.107:0/935590617 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e98106710 msgr2=0x7f5e9819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.596+0000 7f5e9d868640 1 --2- 172.21.15.107:0/935590617 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e98106710 0x7f5e9819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.596+0000 7f5e9d868640 1 -- 172.21.15.107:0/935590617 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e98100700 msgr2=0x7f5e98197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.596+0000 7f5e9d868640 1 --2- 172.21.15.107:0/935590617 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e98100700 0x7f5e98197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.596+0000 7f5e9d868640 1 -- 172.21.15.107:0/935590617 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9819cf50 con 0x7f5e981010c0 2024-05-09T08:33:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.596+0000 7f5e9e86a640 1 --2- 172.21.15.107:0/935590617 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e98106710 0x7f5e9819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.596+0000 7f5e9d868640 1 --2- 172.21.15.107:0/935590617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e981010c0 0x7f5e981984f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800c890 tx=0x7f5e8800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.596+0000 7f5e8f7fe640 1 -- 172.21.15.107:0/935590617 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e88004220 con 0x7f5e981010c0 2024-05-09T08:33:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.596+0000 7f5e8f7fe640 1 -- 172.21.15.107:0/935590617 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e880043c0 con 0x7f5e981010c0 2024-05-09T08:33:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.596+0000 7f5ea02cd640 1 -- 172.21.15.107:0/935590617 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e9819d240 con 0x7f5e981010c0 2024-05-09T08:33:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.596+0000 7f5e8f7fe640 1 -- 172.21.15.107:0/935590617 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e88004a00 con 0x7f5e981010c0 2024-05-09T08:33:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.596+0000 7f5ea02cd640 1 -- 172.21.15.107:0/935590617 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e981a4ad0 con 0x7f5e981010c0 2024-05-09T08:33:27.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.597+0000 7f5ea02cd640 1 -- 172.21.15.107:0/935590617 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e98068650 con 0x7f5e981010c0 2024-05-09T08:33:27.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.598+0000 7f5e8f7fe640 1 -- 172.21.15.107:0/935590617 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7f5e88004ba0 con 0x7f5e981010c0 2024-05-09T08:33:27.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.599+0000 7f5e8f7fe640 1 --2- 172.21.15.107:0/935590617 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5e840a2e20 0x7f5e840a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:27.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.599+0000 7f5e9e069640 1 --2- 172.21.15.107:0/935590617 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5e840a2e20 0x7f5e840a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:27.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.599+0000 7f5e8f7fe640 1 -- 172.21.15.107:0/935590617 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f5e880cf7d0 con 0x7f5e981010c0 2024-05-09T08:33:27.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.599+0000 7f5e9e069640 1 --2- 172.21.15.107:0/935590617 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5e840a2e20 0x7f5e840a52e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400cac0 tx=0x7f5e9400c4a0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:27.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.603+0000 7f5e8f7fe640 1 -- 172.21.15.107:0/935590617 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5e88014030 con 0x7f5e981010c0 2024-05-09T08:33:27.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.776+0000 7f5ea02cd640 1 -- 172.21.15.107:0/935590617 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5e9802d1c0 con 0x7f5e981010c0 2024-05-09T08:33:27.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.777+0000 7f5e8f7fe640 1 -- 172.21.15.107:0/935590617 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f5e8809be40 con 0x7f5e981010c0 2024-05-09T08:33:27.776 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-05-09T08:33:27.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.778+0000 7f5ea02cd640 1 -- 172.21.15.107:0/935590617 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5e840a2e20 msgr2=0x7f5e840a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:27.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.778+0000 7f5ea02cd640 1 --2- 172.21.15.107:0/935590617 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5e840a2e20 0x7f5e840a52e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400cac0 tx=0x7f5e9400c4a0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.778+0000 7f5ea02cd640 1 -- 172.21.15.107:0/935590617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e981010c0 msgr2=0x7f5e981984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:27.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.778+0000 7f5ea02cd640 1 --2- 172.21.15.107:0/935590617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e981010c0 0x7f5e981984f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800c890 tx=0x7f5e8800cd50 comp rx=0 tx=0).stop 2024-05-09T08:33:27.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.779+0000 7f5ea02cd640 1 -- 172.21.15.107:0/935590617 shutdown_connections 2024-05-09T08:33:27.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.779+0000 7f5ea02cd640 1 --2- 172.21.15.107:0/935590617 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5e840a2e20 0x7f5e840a52e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.779+0000 7f5ea02cd640 1 --2- 172.21.15.107:0/935590617 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e98106710 0x7f5e9819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.779+0000 7f5ea02cd640 1 --2- 172.21.15.107:0/935590617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e981010c0 0x7f5e981984f0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.779+0000 7f5ea02cd640 1 --2- 172.21.15.107:0/935590617 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e98100700 0x7f5e98197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.779+0000 7f5ea02cd640 1 -- 172.21.15.107:0/935590617 >> 172.21.15.107:0/935590617 conn(0x7f5e98077830 msgr2=0x7f5e98102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:27.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.779+0000 7f5ea02cd640 1 -- 172.21.15.107:0/935590617 shutdown_connections 2024-05-09T08:33:27.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.779+0000 7f5ea02cd640 1 -- 172.21.15.107:0/935590617 wait complete. 2024-05-09T08:33:27.789 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:33:27.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:27 smithi157 ceph-mon[33614]: from='client.15639 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:33:27.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:27 smithi157 ceph-mon[33614]: from='client.15645 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:33:27.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:27 smithi157 ceph-mon[33614]: from='client.15651 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:33:27.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:27 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1778776556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:27.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:27 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1442921795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:33:27.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:27 smithi110 ceph-mon[33622]: from='client.15639 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:33:27.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:27 smithi110 ceph-mon[33622]: from='client.15645 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:33:27.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:27 smithi110 ceph-mon[33622]: from='client.15651 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T08:33:27.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:27 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1778776556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:27.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:27 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1442921795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:33:27.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.895+0000 7f02e8b14640 1 -- 172.21.15.107:0/1417674648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02e4100a10 msgr2=0x7f02e4100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:27.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.895+0000 7f02e8b14640 1 --2- 172.21.15.107:0/1417674648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02e4100a10 0x7f02e4100e90 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009930 tx=0x7f02cc02a420 comp rx=0 tx=0).stop 2024-05-09T08:33:27.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.895+0000 7f02e8b14640 1 -- 172.21.15.107:0/1417674648 shutdown_connections 2024-05-09T08:33:27.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.895+0000 7f02e8b14640 1 --2- 172.21.15.107:0/1417674648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02e41013d0 0x7f02e410d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.895+0000 7f02e8b14640 1 --2- 172.21.15.107:0/1417674648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02e4100a10 0x7f02e4100e90 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.895+0000 7f02e8b14640 1 --2- 172.21.15.107:0/1417674648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02e4106900 0x7f02e4106ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.895+0000 7f02e8b14640 1 -- 172.21.15.107:0/1417674648 >> 172.21.15.107:0/1417674648 conn(0x7f02e4077710 msgr2=0x7f02e4077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:27.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.895+0000 7f02e8b14640 1 -- 172.21.15.107:0/1417674648 shutdown_connections 2024-05-09T08:33:27.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.896+0000 7f02e8b14640 1 -- 172.21.15.107:0/1417674648 wait complete. 2024-05-09T08:33:27.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.896+0000 7f02e8b14640 1 Processor -- start 2024-05-09T08:33:27.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.896+0000 7f02e8b14640 1 -- start start 2024-05-09T08:33:27.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.896+0000 7f02e8b14640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02e4100a10 0x7f02e4196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:27.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.896+0000 7f02e8b14640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02e41013d0 0x7f02e4197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:27.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.896+0000 7f02e8b14640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02e4106900 0x7f02e419b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:27.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.896+0000 7f02e8b14640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f02e4110a80 con 0x7f02e41013d0 2024-05-09T08:33:27.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.896+0000 7f02e8b14640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f02e4110900 con 0x7f02e4106900 2024-05-09T08:33:27.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.896+0000 7f02e8b14640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f02e4110c00 con 0x7f02e4100a10 2024-05-09T08:33:27.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.897+0000 7f02e1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02e41013d0 0x7f02e4197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:27.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.897+0000 7f02e1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02e41013d0 0x7f02e4197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50540/0 (socket says 172.21.15.107:50540) 2024-05-09T08:33:27.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.897+0000 7f02e1d9b640 1 -- 172.21.15.107:0/1939079705 learned_addr learned my addr 172.21.15.107:0/1939079705 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:27.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.897+0000 7f02e259c640 1 --2- 172.21.15.107:0/1939079705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02e4100a10 0x7f02e4196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:27.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.897+0000 7f02e1d9b640 1 -- 172.21.15.107:0/1939079705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02e4100a10 msgr2=0x7f02e4196fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.897+0000 7f02e1d9b640 1 --2- 172.21.15.107:0/1939079705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02e4100a10 0x7f02e4196fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.897+0000 7f02e1d9b640 1 -- 172.21.15.107:0/1939079705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02e4106900 msgr2=0x7f02e419b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.897+0000 7f02e1d9b640 1 --2- 172.21.15.107:0/1939079705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02e4106900 0x7f02e419b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.897+0000 7f02e1d9b640 1 -- 172.21.15.107:0/1939079705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e419bfd0 con 0x7f02e41013d0 2024-05-09T08:33:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.897+0000 7f02e1d9b640 1 --2- 172.21.15.107:0/1939079705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02e41013d0 0x7f02e4197520 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f02cc004770 tx=0x7f02cc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.897+0000 7f02db7fe640 1 -- 172.21.15.107:0/1939079705 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02cc03f070 con 0x7f02e41013d0 2024-05-09T08:33:27.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.897+0000 7f02db7fe640 1 -- 172.21.15.107:0/1939079705 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02cc03b460 con 0x7f02e41013d0 2024-05-09T08:33:27.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.897+0000 7f02db7fe640 1 -- 172.21.15.107:0/1939079705 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02cc035070 con 0x7f02e41013d0 2024-05-09T08:33:27.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.897+0000 7f02e8b14640 1 -- 172.21.15.107:0/1939079705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02e419c260 con 0x7f02e41013d0 2024-05-09T08:33:27.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.897+0000 7f02e8b14640 1 -- 172.21.15.107:0/1939079705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02e41a3b50 con 0x7f02e41013d0 2024-05-09T08:33:27.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.898+0000 7f02e8b14640 1 -- 172.21.15.107:0/1939079705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02e4068aa0 con 0x7f02e41013d0 2024-05-09T08:33:27.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.899+0000 7f02db7fe640 1 -- 172.21.15.107:0/1939079705 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7f02cc03b610 con 0x7f02e41013d0 2024-05-09T08:33:27.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.900+0000 7f02db7fe640 1 --2- 172.21.15.107:0/1939079705 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f02d00a2e20 0x7f02d00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:27.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.900+0000 7f02e259c640 1 --2- 172.21.15.107:0/1939079705 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f02d00a2e20 0x7f02d00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:27.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.900+0000 7f02db7fe640 1 -- 172.21.15.107:0/1939079705 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f02cc0f0090 con 0x7f02e41013d0 2024-05-09T08:33:27.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.900+0000 7f02e259c640 1 --2- 172.21.15.107:0/1939079705 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f02d00a2e20 0x7f02d00a52e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f02dc005fd0 tx=0x7f02dc005850 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:27.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:27.904+0000 7f02db7fe640 1 -- 172.21.15.107:0/1939079705 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f02cc0bc7d0 con 0x7f02e41013d0 2024-05-09T08:33:28.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.082+0000 7f02e8b14640 1 -- 172.21.15.107:0/1939079705 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f02e4108c20 con 0x7f02e41013d0 2024-05-09T08:33:28.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.082+0000 7f02db7fe640 1 -- 172.21.15.107:0/1939079705 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f02cc0bc0d0 con 0x7f02e41013d0 2024-05-09T08:33:28.081 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:33:28.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.084+0000 7f02e8b14640 1 -- 172.21.15.107:0/1939079705 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f02d00a2e20 msgr2=0x7f02d00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.084+0000 7f02e8b14640 1 --2- 172.21.15.107:0/1939079705 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f02d00a2e20 0x7f02d00a52e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f02dc005fd0 tx=0x7f02dc005850 comp rx=0 tx=0).stop 2024-05-09T08:33:28.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.084+0000 7f02e8b14640 1 -- 172.21.15.107:0/1939079705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02e41013d0 msgr2=0x7f02e4197520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.084+0000 7f02e8b14640 1 --2- 172.21.15.107:0/1939079705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02e41013d0 0x7f02e4197520 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f02cc004770 tx=0x7f02cc0042f0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.084+0000 7f02e8b14640 1 -- 172.21.15.107:0/1939079705 shutdown_connections 2024-05-09T08:33:28.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.084+0000 7f02e8b14640 1 --2- 172.21.15.107:0/1939079705 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f02d00a2e20 0x7f02d00a52e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.084+0000 7f02e8b14640 1 --2- 172.21.15.107:0/1939079705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02e4106900 0x7f02e419b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.084+0000 7f02e8b14640 1 --2- 172.21.15.107:0/1939079705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02e41013d0 0x7f02e4197520 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.084+0000 7f02e8b14640 1 --2- 172.21.15.107:0/1939079705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02e4100a10 0x7f02e4196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.084+0000 7f02e8b14640 1 -- 172.21.15.107:0/1939079705 >> 172.21.15.107:0/1939079705 conn(0x7f02e4077710 msgr2=0x7f02e40ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:28.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.084+0000 7f02e8b14640 1 -- 172.21.15.107:0/1939079705 shutdown_connections 2024-05-09T08:33:28.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.084+0000 7f02e8b14640 1 -- 172.21.15.107:0/1939079705 wait complete. 2024-05-09T08:33:28.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:33:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.199+0000 7f575295a640 1 -- 172.21.15.107:0/3840295228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c10be00 msgr2=0x7f574c10e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.199+0000 7f575295a640 1 --2- 172.21.15.107:0/3840295228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c10be00 0x7f574c10e2a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5744009ca0 tx=0x7f574402a9e0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.199+0000 7f575295a640 1 -- 172.21.15.107:0/3840295228 shutdown_connections 2024-05-09T08:33:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.199+0000 7f575295a640 1 --2- 172.21.15.107:0/3840295228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c10be00 0x7f574c10e2a0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.199+0000 7f575295a640 1 --2- 172.21.15.107:0/3840295228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f574c1033e0 0x7f574c10b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.200+0000 7f575295a640 1 --2- 172.21.15.107:0/3840295228 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f574c102ac0 0x7f574c102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.200+0000 7f575295a640 1 -- 172.21.15.107:0/3840295228 >> 172.21.15.107:0/3840295228 conn(0x7f574c077830 msgr2=0x7f574c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.200+0000 7f575295a640 1 -- 172.21.15.107:0/3840295228 shutdown_connections 2024-05-09T08:33:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.200+0000 7f575295a640 1 -- 172.21.15.107:0/3840295228 wait complete. 2024-05-09T08:33:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.200+0000 7f575295a640 1 Processor -- start 2024-05-09T08:33:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.200+0000 7f575295a640 1 -- start start 2024-05-09T08:33:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.200+0000 7f575295a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c102ac0 0x7f574c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.200+0000 7f575295a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f574c1033e0 0x7f574c1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.200+0000 7f575295a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f574c10be00 0x7f574c19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.200+0000 7f575295a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f574c111ab0 con 0x7f574c102ac0 2024-05-09T08:33:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.200+0000 7f575295a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f574c111930 con 0x7f574c1033e0 2024-05-09T08:33:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.200+0000 7f575295a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f574c111c30 con 0x7f574c10be00 2024-05-09T08:33:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.201+0000 7f574bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c102ac0 0x7f574c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.201+0000 7f574bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c102ac0 0x7f574c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50562/0 (socket says 172.21.15.107:50562) 2024-05-09T08:33:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.201+0000 7f574bfff640 1 -- 172.21.15.107:0/1643533759 learned_addr learned my addr 172.21.15.107:0/1643533759 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.201+0000 7f5750ef7640 1 --2- 172.21.15.107:0/1643533759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f574c10be00 0x7f574c19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.201+0000 7f574bfff640 1 -- 172.21.15.107:0/1643533759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f574c10be00 msgr2=0x7f574c19c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.201+0000 7f574bfff640 1 --2- 172.21.15.107:0/1643533759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f574c10be00 0x7f574c19c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.201+0000 7f574bfff640 1 -- 172.21.15.107:0/1643533759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f574c1033e0 msgr2=0x7f574c1984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.201+0000 7f574bfff640 1 --2- 172.21.15.107:0/1643533759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f574c1033e0 0x7f574c1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.201+0000 7f574bfff640 1 -- 172.21.15.107:0/1643533759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f574c19cf10 con 0x7f574c102ac0 2024-05-09T08:33:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.201+0000 7f5750ef7640 1 --2- 172.21.15.107:0/1643533759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f574c10be00 0x7f574c19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.201+0000 7f574bfff640 1 --2- 172.21.15.107:0/1643533759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c102ac0 0x7f574c197f70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f573c00d770 tx=0x7f573c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.201+0000 7f57497fa640 1 -- 172.21.15.107:0/1643533759 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c017070 con 0x7f574c102ac0 2024-05-09T08:33:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.201+0000 7f57497fa640 1 -- 172.21.15.107:0/1643533759 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f573c0054d0 con 0x7f574c102ac0 2024-05-09T08:33:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.201+0000 7f575295a640 1 -- 172.21.15.107:0/1643533759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f574c19d200 con 0x7f574c102ac0 2024-05-09T08:33:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.201+0000 7f57497fa640 1 -- 172.21.15.107:0/1643533759 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c004cf0 con 0x7f574c102ac0 2024-05-09T08:33:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.201+0000 7f575295a640 1 -- 172.21.15.107:0/1643533759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f574c1a4a90 con 0x7f574c102ac0 2024-05-09T08:33:28.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.202+0000 7f575295a640 1 -- 172.21.15.107:0/1643533759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f574c068650 con 0x7f574c102ac0 2024-05-09T08:33:28.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.203+0000 7f57497fa640 1 -- 172.21.15.107:0/1643533759 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7f573c004e90 con 0x7f574c102ac0 2024-05-09T08:33:28.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.204+0000 7f57497fa640 1 --2- 172.21.15.107:0/1643533759 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f57340a2e20 0x7f57340a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.204+0000 7f574b7fe640 1 --2- 172.21.15.107:0/1643533759 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f57340a2e20 0x7f57340a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.204+0000 7f57497fa640 1 -- 172.21.15.107:0/1643533759 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f573c0cf580 con 0x7f574c102ac0 2024-05-09T08:33:28.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.204+0000 7f574b7fe640 1 --2- 172.21.15.107:0/1643533759 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f57340a2e20 0x7f57340a52e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f574c10dc60 tx=0x7f5740005c00 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:28.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.208+0000 7f57497fa640 1 -- 172.21.15.107:0/1643533759 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f573c09bbf0 con 0x7f574c102ac0 2024-05-09T08:33:28.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.423+0000 7f575295a640 1 -- 172.21.15.107:0/1643533759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f574c1069f0 con 0x7f574c102ac0 2024-05-09T08:33:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.424+0000 7f57497fa640 1 -- 172.21.15.107:0/1643533759 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11887 (secure 0 0 0) 0x7f573c09b4f0 con 0x7f574c102ac0 2024-05-09T08:33:28.424 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:28.425 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:33:19.308598+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:33:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 25 2024-05-09T08:33:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.426+0000 7f575295a640 1 -- 172.21.15.107:0/1643533759 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f57340a2e20 msgr2=0x7f57340a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.426+0000 7f575295a640 1 --2- 172.21.15.107:0/1643533759 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f57340a2e20 0x7f57340a52e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f574c10dc60 tx=0x7f5740005c00 comp rx=0 tx=0).stop 2024-05-09T08:33:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.426+0000 7f575295a640 1 -- 172.21.15.107:0/1643533759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c102ac0 msgr2=0x7f574c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.426+0000 7f575295a640 1 --2- 172.21.15.107:0/1643533759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c102ac0 0x7f574c197f70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f573c00d770 tx=0x7f573c00dc30 comp rx=0 tx=0).stop 2024-05-09T08:33:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.427+0000 7f575295a640 1 -- 172.21.15.107:0/1643533759 shutdown_connections 2024-05-09T08:33:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.427+0000 7f575295a640 1 --2- 172.21.15.107:0/1643533759 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f57340a2e20 0x7f57340a52e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.427+0000 7f575295a640 1 --2- 172.21.15.107:0/1643533759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f574c10be00 0x7f574c19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.427+0000 7f575295a640 1 --2- 172.21.15.107:0/1643533759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f574c1033e0 0x7f574c1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.427+0000 7f575295a640 1 --2- 172.21.15.107:0/1643533759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c102ac0 0x7f574c197f70 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.427+0000 7f575295a640 1 -- 172.21.15.107:0/1643533759 >> 172.21.15.107:0/1643533759 conn(0x7f574c077830 msgr2=0x7f574c0fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.427+0000 7f575295a640 1 -- 172.21.15.107:0/1643533759 shutdown_connections 2024-05-09T08:33:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.427+0000 7f575295a640 1 -- 172.21.15.107:0/1643533759 wait complete. 2024-05-09T08:33:28.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:33:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.539+0000 7fa3df561640 1 -- 172.21.15.107:0/3866788479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d8106720 msgr2=0x7fa3d8106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.539+0000 7fa3df561640 1 --2- 172.21.15.107:0/3866788479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d8106720 0x7fa3d8106b00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4009930 tx=0x7fa3d402a420 comp rx=0 tx=0).stop 2024-05-09T08:33:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.539+0000 7fa3df561640 1 -- 172.21.15.107:0/3866788479 shutdown_connections 2024-05-09T08:33:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.539+0000 7fa3df561640 1 --2- 172.21.15.107:0/3866788479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3d81010d0 0x7fa3d810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.539+0000 7fa3df561640 1 --2- 172.21.15.107:0/3866788479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3d8100710 0x7fa3d8100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.539+0000 7fa3df561640 1 --2- 172.21.15.107:0/3866788479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d8106720 0x7fa3d8106b00 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.539+0000 7fa3df561640 1 -- 172.21.15.107:0/3866788479 >> 172.21.15.107:0/3866788479 conn(0x7fa3d8077800 msgr2=0x7fa3d8077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.539+0000 7fa3df561640 1 -- 172.21.15.107:0/3866788479 shutdown_connections 2024-05-09T08:33:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.539+0000 7fa3df561640 1 -- 172.21.15.107:0/3866788479 wait complete. 2024-05-09T08:33:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3df561640 1 Processor -- start 2024-05-09T08:33:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3df561640 1 -- start start 2024-05-09T08:33:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3df561640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3d8100710 0x7fa3d8198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3df561640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3d81010d0 0x7fa3d81985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3df561640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d8106720 0x7fa3d819c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3df561640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d8111ae0 con 0x7fa3d8106720 2024-05-09T08:33:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3df561640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d8111960 con 0x7fa3d8100710 2024-05-09T08:33:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3df561640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d8111c60 con 0x7fa3d81010d0 2024-05-09T08:33:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3ddafe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d8106720 0x7fa3d819c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3ddafe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d8106720 0x7fa3d819c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50580/0 (socket says 172.21.15.107:50580) 2024-05-09T08:33:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3ddafe640 1 -- 172.21.15.107:0/3103440983 learned_addr learned my addr 172.21.15.107:0/3103440983 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3dd2fd640 1 --2- 172.21.15.107:0/3103440983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3d8100710 0x7fa3d8198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3dcafc640 1 --2- 172.21.15.107:0/3103440983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3d81010d0 0x7fa3d81985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3ddafe640 1 -- 172.21.15.107:0/3103440983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3d81010d0 msgr2=0x7fa3d81985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3ddafe640 1 --2- 172.21.15.107:0/3103440983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3d81010d0 0x7fa3d81985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3ddafe640 1 -- 172.21.15.107:0/3103440983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3d8100710 msgr2=0x7fa3d8198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3ddafe640 1 --2- 172.21.15.107:0/3103440983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3d8100710 0x7fa3d8198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.540+0000 7fa3ddafe640 1 -- 172.21.15.107:0/3103440983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d819d070 con 0x7fa3d8106720 2024-05-09T08:33:28.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.541+0000 7fa3ddafe640 1 --2- 172.21.15.107:0/3103440983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d8106720 0x7fa3d819c940 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa3d000cad0 tx=0x7fa3d000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:28.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.541+0000 7fa3ce7fc640 1 -- 172.21.15.107:0/3103440983 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d000ae20 con 0x7fa3d8106720 2024-05-09T08:33:28.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.541+0000 7fa3ce7fc640 1 -- 172.21.15.107:0/3103440983 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3d0005020 con 0x7fa3d8106720 2024-05-09T08:33:28.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.541+0000 7fa3ce7fc640 1 -- 172.21.15.107:0/3103440983 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d000f6b0 con 0x7fa3d8106720 2024-05-09T08:33:28.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.541+0000 7fa3df561640 1 -- 172.21.15.107:0/3103440983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3d819d360 con 0x7fa3d8106720 2024-05-09T08:33:28.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.541+0000 7fa3df561640 1 -- 172.21.15.107:0/3103440983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d81a4bf0 con 0x7fa3d8106720 2024-05-09T08:33:28.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.542+0000 7fa3df561640 1 -- 172.21.15.107:0/3103440983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3d8068aa0 con 0x7fa3d8106720 2024-05-09T08:33:28.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.543+0000 7fa3ce7fc640 1 -- 172.21.15.107:0/3103440983 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7fa3d00051d0 con 0x7fa3d8106720 2024-05-09T08:33:28.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.544+0000 7fa3ce7fc640 1 --2- 172.21.15.107:0/3103440983 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa3c40a2e20 0x7fa3c40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.544+0000 7fa3dd2fd640 1 --2- 172.21.15.107:0/3103440983 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa3c40a2e20 0x7fa3c40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.544+0000 7fa3ce7fc640 1 -- 172.21.15.107:0/3103440983 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fa3d00cfe00 con 0x7fa3d8106720 2024-05-09T08:33:28.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.544+0000 7fa3dd2fd640 1 --2- 172.21.15.107:0/3103440983 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa3c40a2e20 0x7fa3c40a52e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4009900 tx=0x7fa3d4000bc0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:28.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.547+0000 7fa3ce7fc640 1 -- 172.21.15.107:0/3103440983 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa3d0014030 con 0x7fa3d8106720 2024-05-09T08:33:28.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.743+0000 7fa3df561640 1 -- 172.21.15.107:0/3103440983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa3d81086f0 con 0x7fa3d8106720 2024-05-09T08:33:28.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.745+0000 7fa3ce7fc640 1 -- 172.21.15.107:0/3103440983 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25143 (secure 0 0 0) 0x7fa3d009c470 con 0x7fa3d8106720 2024-05-09T08:33:28.744 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:28.744 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":94,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:33:20.828605+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/2923766500":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/2417648229":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1642826815":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/2689615560":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/846948416":"2024-05-10T08:33:20.828572+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/846948416":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:33:28.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.748+0000 7fa3df561640 1 -- 172.21.15.107:0/3103440983 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa3c40a2e20 msgr2=0x7fa3c40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.748+0000 7fa3df561640 1 --2- 172.21.15.107:0/3103440983 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa3c40a2e20 0x7fa3c40a52e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4009900 tx=0x7fa3d4000bc0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.748+0000 7fa3df561640 1 -- 172.21.15.107:0/3103440983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d8106720 msgr2=0x7fa3d819c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.748+0000 7fa3df561640 1 --2- 172.21.15.107:0/3103440983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d8106720 0x7fa3d819c940 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa3d000cad0 tx=0x7fa3d000cf90 comp rx=0 tx=0).stop 2024-05-09T08:33:28.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.748+0000 7fa3df561640 1 -- 172.21.15.107:0/3103440983 shutdown_connections 2024-05-09T08:33:28.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.748+0000 7fa3df561640 1 --2- 172.21.15.107:0/3103440983 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa3c40a2e20 0x7fa3c40a52e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.748+0000 7fa3df561640 1 --2- 172.21.15.107:0/3103440983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3d8106720 0x7fa3d819c940 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.748+0000 7fa3df561640 1 --2- 172.21.15.107:0/3103440983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3d81010d0 0x7fa3d81985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.748+0000 7fa3df561640 1 --2- 172.21.15.107:0/3103440983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3d8100710 0x7fa3d8198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.748+0000 7fa3df561640 1 -- 172.21.15.107:0/3103440983 >> 172.21.15.107:0/3103440983 conn(0x7fa3d8077800 msgr2=0x7fa3d8103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:28.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.748+0000 7fa3df561640 1 -- 172.21.15.107:0/3103440983 shutdown_connections 2024-05-09T08:33:28.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.748+0000 7fa3df561640 1 -- 172.21.15.107:0/3103440983 wait complete. 2024-05-09T08:33:28.758 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:28 smithi107 ceph-mon[30122]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s wr, 0 op/s 2024-05-09T08:33:28.758 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:28 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3259364600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:33:28.758 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:28 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/935590617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:33:28.758 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:28 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1939079705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:33:28.760 DEBUG:tasks.fwd_scrub:Yielding 2024-05-09T08:33:28.760 INFO:teuthology.run_tasks:Running task exec... 2024-05-09T08:33:28.769 INFO:teuthology.task.exec:Executing custom commands... 2024-05-09T08:33:28.770 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-05-09T08:33:28.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-05-09T08:33:28.770 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-05-09T08:33:28.770 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:33:28.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:33:28.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:33:28.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:28 smithi157 ceph-mon[33614]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s wr, 0 op/s 2024-05-09T08:33:28.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:28 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3259364600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:33:28.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:28 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/935590617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:33:28.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:28 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1939079705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:33:28.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:28 smithi110 ceph-mon[33622]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s wr, 0 op/s 2024-05-09T08:33:28.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:28 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3259364600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:33:28.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:28 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/935590617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:33:28.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:28 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1939079705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:33:28.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7f47f6888640 1 -- 172.21.15.107:0/885113410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47f01033e0 msgr2=0x7f47f010b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7f47f6888640 1 --2- 172.21.15.107:0/885113410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47f01033e0 0x7f47f010b8c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009900 tx=0x7f47e402a510 comp rx=0 tx=0).stop 2024-05-09T08:33:28.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7fdd45415640 1 -- 172.21.15.107:0/4058225337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd40102ae0 msgr2=0x7fdd40102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7fdd45415640 1 --2- 172.21.15.107:0/4058225337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd40102ae0 0x7fdd40102ec0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009900 tx=0x7fdd3002a480 comp rx=0 tx=0).stop 2024-05-09T08:33:28.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7f47f6888640 1 -- 172.21.15.107:0/885113410 shutdown_connections 2024-05-09T08:33:28.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7f47f6888640 1 --2- 172.21.15.107:0/885113410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47f010be00 0x7f47f010e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7f47f6888640 1 --2- 172.21.15.107:0/885113410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47f01033e0 0x7f47f010b8c0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7f47f6888640 1 --2- 172.21.15.107:0/885113410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47f0102ac0 0x7f47f0102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7f47f6888640 1 -- 172.21.15.107:0/885113410 >> 172.21.15.107:0/885113410 conn(0x7f47f0076ab0 msgr2=0x7f47f0076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:28.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7fdd45415640 1 -- 172.21.15.107:0/4058225337 shutdown_connections 2024-05-09T08:33:28.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7fdd45415640 1 --2- 172.21.15.107:0/4058225337 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd4010be20 0x7fdd4010e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7fdd45415640 1 --2- 172.21.15.107:0/4058225337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd40103400 0x7fdd4010b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7fdd45415640 1 --2- 172.21.15.107:0/4058225337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd40102ae0 0x7fdd40102ec0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7fdd45415640 1 -- 172.21.15.107:0/4058225337 >> 172.21.15.107:0/4058225337 conn(0x7fdd400778b0 msgr2=0x7fdd40077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:28.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7f47f6888640 1 -- 172.21.15.107:0/885113410 shutdown_connections 2024-05-09T08:33:28.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7f47f6888640 1 -- 172.21.15.107:0/885113410 wait complete. 2024-05-09T08:33:28.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7fdd45415640 1 -- 172.21.15.107:0/4058225337 shutdown_connections 2024-05-09T08:33:28.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.879+0000 7fdd45415640 1 -- 172.21.15.107:0/4058225337 wait complete. 2024-05-09T08:33:28.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7f47f6888640 1 Processor -- start 2024-05-09T08:33:28.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7f47f6888640 1 -- start start 2024-05-09T08:33:28.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7fdd45415640 1 Processor -- start 2024-05-09T08:33:28.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7fdd45415640 1 -- start start 2024-05-09T08:33:28.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7f47f6888640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47f0102ac0 0x7f47f01980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7f47f6888640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47f01033e0 0x7f47f01985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7f47f6888640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47f010be00 0x7f47f019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7f47f6888640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f47f0111b00 con 0x7f47f010be00 2024-05-09T08:33:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7f47f6888640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f47f0111980 con 0x7f47f0102ac0 2024-05-09T08:33:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7f47f6888640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f47f0111c80 con 0x7f47f01033e0 2024-05-09T08:33:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7fdd45415640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd40102ae0 0x7fdd40198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7fdd45415640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd40103400 0x7fdd40198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7f47f4e25640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47f010be00 0x7f47f019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7fdd45415640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd4010be20 0x7fdd4019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7fdd45415640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdd40111aa0 con 0x7fdd4010be20 2024-05-09T08:33:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7fdd45415640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdd40111920 con 0x7fdd40103400 2024-05-09T08:33:28.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7fdd45415640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdd40111c20 con 0x7fdd40102ae0 2024-05-09T08:33:28.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7f47f4e25640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47f010be00 0x7f47f019c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50620/0 (socket says 172.21.15.107:50620) 2024-05-09T08:33:28.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7f47f4e25640 1 -- 172.21.15.107:0/2900032350 learned_addr learned my addr 172.21.15.107:0/2900032350 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:28.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7f47effff640 1 --2- 172.21.15.107:0/2900032350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47f0102ac0 0x7f47f01980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7f47ef7fe640 1 --2- 172.21.15.107:0/2900032350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47f01033e0 0x7f47f01985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7fdd3effd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd40102ae0 0x7fdd40198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7fdd3effd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd40102ae0 0x7fdd40198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:41550/0 (socket says 172.21.15.107:41550) 2024-05-09T08:33:28.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7fdd3effd640 1 -- 172.21.15.107:0/31557377 learned_addr learned my addr 172.21.15.107:0/31557377 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:28.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.880+0000 7fdd3e7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd40103400 0x7fdd40198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7f47f4e25640 1 -- 172.21.15.107:0/2900032350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47f01033e0 msgr2=0x7f47f01985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7f47f4e25640 1 --2- 172.21.15.107:0/2900032350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47f01033e0 0x7f47f01985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7f47f4e25640 1 -- 172.21.15.107:0/2900032350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47f0102ac0 msgr2=0x7f47f01980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7f47f4e25640 1 --2- 172.21.15.107:0/2900032350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47f0102ac0 0x7f47f01980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7fdd3f7fe640 1 --2- 172.21.15.107:0/31557377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd4010be20 0x7fdd4019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7f47f4e25640 1 -- 172.21.15.107:0/2900032350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47f019d090 con 0x7f47f010be00 2024-05-09T08:33:28.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7fdd3effd640 1 -- 172.21.15.107:0/31557377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd40103400 msgr2=0x7fdd40198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7fdd3effd640 1 --2- 172.21.15.107:0/31557377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd40103400 0x7fdd40198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7fdd3effd640 1 -- 172.21.15.107:0/31557377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd4010be20 msgr2=0x7fdd4019c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7fdd3effd640 1 --2- 172.21.15.107:0/31557377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd4010be20 0x7fdd4019c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7fdd3effd640 1 -- 172.21.15.107:0/31557377 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4019cfa0 con 0x7fdd40102ae0 2024-05-09T08:33:28.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7f47f4e25640 1 --2- 172.21.15.107:0/2900032350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47f010be00 0x7f47f019c960 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f47e800eb10 tx=0x7f47e800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:28.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7fdd3effd640 1 --2- 172.21.15.107:0/31557377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd40102ae0 0x7fdd40198000 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdd30005940 tx=0x7fdd300042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:28.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7f47ed7fa640 1 -- 172.21.15.107:0/2900032350 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e8004b20 con 0x7f47f010be00 2024-05-09T08:33:28.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7f47f6888640 1 -- 172.21.15.107:0/2900032350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47f019d380 con 0x7f47f010be00 2024-05-09T08:33:28.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7f47ed7fa640 1 -- 172.21.15.107:0/2900032350 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47e8004cc0 con 0x7f47f010be00 2024-05-09T08:33:28.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7f47ed7fa640 1 -- 172.21.15.107:0/2900032350 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e8010690 con 0x7f47f010be00 2024-05-09T08:33:28.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7f47f6888640 1 -- 172.21.15.107:0/2900032350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47f01a4c10 con 0x7f47f010be00 2024-05-09T08:33:28.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7fdd27fff640 1 -- 172.21.15.107:0/31557377 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3003f030 con 0x7fdd40102ae0 2024-05-09T08:33:28.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7fdd45415640 1 -- 172.21.15.107:0/31557377 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd4019d230 con 0x7fdd40102ae0 2024-05-09T08:33:28.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7fdd45415640 1 -- 172.21.15.107:0/31557377 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd401a4b20 con 0x7fdd40102ae0 2024-05-09T08:33:28.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7fdd27fff640 1 -- 172.21.15.107:0/31557377 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd30003cb0 con 0x7fdd40102ae0 2024-05-09T08:33:28.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.881+0000 7fdd27fff640 1 -- 172.21.15.107:0/31557377 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30049060 con 0x7fdd40102ae0 2024-05-09T08:33:28.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.882+0000 7fdd45415640 1 -- 172.21.15.107:0/31557377 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd2c005290 con 0x7fdd40102ae0 2024-05-09T08:33:28.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7f47f9590740 1 Processor -- start 2024-05-09T08:33:28.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7f47ed7fa640 1 -- 172.21.15.107:0/2900032350 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7f47e8010830 con 0x7f47f010be00 2024-05-09T08:33:28.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7fdd27fff640 1 -- 172.21.15.107:0/31557377 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7fdd30042030 con 0x7fdd40102ae0 2024-05-09T08:33:28.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7f47f9590740 1 -- start start 2024-05-09T08:33:28.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7f47f9590740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e300b0 0x565160e2e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7f47f9590740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e2e9c0 0x565160e2ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7f47f9590740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e39740 0x565160e3bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7f47f9590740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x565160d93990 con 0x565160e39740 2024-05-09T08:33:28.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7f47f9590740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x565160d3b5b0 con 0x565160e300b0 2024-05-09T08:33:28.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7f47f9590740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x565160d8f950 con 0x565160e2e9c0 2024-05-09T08:33:28.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7f47ef7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e300b0 0x565160e2e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7f47effff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e2e9c0 0x565160e2ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7f47effff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e2e9c0 0x565160e2ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:41560/0 (socket says 172.21.15.107:41560) 2024-05-09T08:33:28.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7f47ef7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e300b0 0x565160e2e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:48306/0 (socket says 172.21.15.107:48306) 2024-05-09T08:33:28.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7f47effff640 1 -- 172.21.15.107:0/1891886068 learned_addr learned my addr 172.21.15.107:0/1891886068 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:28.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7f47ef7fe640 1 --2- 172.21.15.107:0/1891886068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e39740 0x565160e3bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7fdd27fff640 1 --2- 172.21.15.107:0/31557377 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdd280a2ec0 0x7fdd280a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.884+0000 7f47ed7fa640 1 --2- 172.21.15.107:0/2900032350 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f47d80a2e20 0x7f47d80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.885+0000 7f47f4e25640 1 --2- 172.21.15.107:0/2900032350 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f47d80a2e20 0x7f47d80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.885+0000 7f47effff640 1 -- 172.21.15.107:0/1891886068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e300b0 msgr2=0x565160e2e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:28.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.885+0000 7f47effff640 1 --2- 172.21.15.107:0/1891886068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e300b0 0x565160e2e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.885+0000 7f47effff640 1 -- 172.21.15.107:0/1891886068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e39740 msgr2=0x565160e3bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:28.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.885+0000 7f47effff640 1 --2- 172.21.15.107:0/1891886068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e39740 0x565160e3bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.885+0000 7f47effff640 1 -- 172.21.15.107:0/1891886068 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565160b6f7e0 con 0x565160e2e9c0 2024-05-09T08:33:28.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.885+0000 7f47ef7fe640 1 --2- 172.21.15.107:0/1891886068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e300b0 0x565160e2e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:33:28.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.885+0000 7fdd27fff640 1 -- 172.21.15.107:0/31557377 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fdd300efbd0 con 0x7fdd40102ae0 2024-05-09T08:33:28.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.885+0000 7fdd3e7fc640 1 --2- 172.21.15.107:0/31557377 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdd280a2ec0 0x7fdd280a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.885+0000 7f47ed7fa640 1 -- 172.21.15.107:0/2900032350 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f47e80cff90 con 0x7f47f010be00 2024-05-09T08:33:28.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.885+0000 7f47effff640 1 --2- 172.21.15.107:0/1891886068 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e2e9c0 0x565160e2ee40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f47e0009d20 tx=0x7f47e0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72c2149cfca80a57 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:28.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.885+0000 7f47f4e25640 1 --2- 172.21.15.107:0/2900032350 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f47d80a2e20 0x7f47d80a52e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f47e80cd260 tx=0x7f47e806dec0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:28.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.885+0000 7fdd3e7fc640 1 --2- 172.21.15.107:0/31557377 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdd280a2ec0 0x7fdd280a5380 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdd401001c0 tx=0x7fdd3400b390 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:28.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.888+0000 7f47d67fc640 1 -- 172.21.15.107:0/1891886068 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e001e070 con 0x565160e2e9c0 2024-05-09T08:33:28.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.888+0000 7f47d67fc640 1 -- 172.21.15.107:0/1891886068 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47e0019ae0 con 0x565160e2e9c0 2024-05-09T08:33:28.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.888+0000 7f47d67fc640 1 -- 172.21.15.107:0/1891886068 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e000ea10 con 0x565160e2e9c0 2024-05-09T08:33:28.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.888+0000 7fdd27fff640 1 -- 172.21.15.107:0/31557377 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdd300bbcc0 con 0x7fdd40102ae0 2024-05-09T08:33:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.889+0000 7f47f9590740 1 -- 172.21.15.107:0/1891886068 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e2e9c0 msgr2=0x565160e2ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.889+0000 7f47f9590740 1 --2- 172.21.15.107:0/1891886068 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e2e9c0 0x565160e2ee40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f47e0009d20 tx=0x7f47e0011810 comp rx=0 tx=0).stop 2024-05-09T08:33:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.889+0000 7f47f9590740 1 -- 172.21.15.107:0/1891886068 shutdown_connections 2024-05-09T08:33:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.889+0000 7f47f9590740 1 --2- 172.21.15.107:0/1891886068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e39740 0x565160e3bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.889+0000 7f47f9590740 1 --2- 172.21.15.107:0/1891886068 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e2e9c0 0x565160e2ee40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.889+0000 7f47f9590740 1 --2- 172.21.15.107:0/1891886068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e300b0 0x565160e2e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.889+0000 7f47f9590740 1 -- 172.21.15.107:0/1891886068 >> 172.21.15.107:0/1891886068 conn(0x565160df70a0 msgr2=0x565160e2af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.889+0000 7f47f9590740 1 -- 172.21.15.107:0/1891886068 shutdown_connections 2024-05-09T08:33:28.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.889+0000 7f47f9590740 1 -- 172.21.15.107:0/1891886068 wait complete. 2024-05-09T08:33:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.889+0000 7f47f9590740 1 -- start start 2024-05-09T08:33:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.889+0000 7f47f9590740 1 Processor -- start 2024-05-09T08:33:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47f9590740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e2e9c0 0x565160e36670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47f9590740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e300b0 0x565160e36bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47f9590740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e39740 0x565160e370f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47f9590740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x565160e3c5d0 con 0x565160e300b0 2024-05-09T08:33:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47f9590740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x565160d4eb40 con 0x565160e2e9c0 2024-05-09T08:33:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47f9590740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x565160e3c750 con 0x565160e39740 2024-05-09T08:33:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47f9590740 10 client.0 _ll_register_callbacks cb 0x565160d430f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e300b0 0x565160e36bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e300b0 0x565160e36bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50636/0 (socket says 172.21.15.107:50636) 2024-05-09T08:33:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47effff640 1 -- 172.21.15.107:0/1408268245 learned_addr learned my addr 172.21.15.107:0/1408268245 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:28.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47ef7fe640 1 --2- 172.21.15.107:0/1408268245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e39740 0x565160e370f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47ef7fe640 1 --2- 172.21.15.107:0/1408268245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e2e9c0 0x565160e36670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47effff640 1 -- 172.21.15.107:0/1408268245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e39740 msgr2=0x565160e370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47effff640 1 --2- 172.21.15.107:0/1408268245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e39740 0x565160e370f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47effff640 1 -- 172.21.15.107:0/1408268245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e2e9c0 msgr2=0x565160e36670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47effff640 1 --2- 172.21.15.107:0/1408268245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e2e9c0 0x565160e36670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47effff640 1 -- 172.21.15.107:0/1408268245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565160bcff70 con 0x565160e300b0 2024-05-09T08:33:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47effff640 1 --2- 172.21.15.107:0/1408268245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e300b0 0x565160e36bb0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f47e0005e00 tx=0x7f47e0019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=740d895a7934c2f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47cf7fe640 1 -- 172.21.15.107:0/1408268245 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e0027020 con 0x565160e300b0 2024-05-09T08:33:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47cf7fe640 1 -- 172.21.15.107:0/1408268245 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f47e0005250 con 0x565160e300b0 2024-05-09T08:33:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.890+0000 7f47f9590740 1 -- 172.21.15.107:0/1408268245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47e0019ae0 con 0x565160e300b0 2024-05-09T08:33:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.891+0000 7f47cf7fe640 1 -- 172.21.15.107:0/1408268245 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f47e0032050 con 0x565160e300b0 2024-05-09T08:33:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.891+0000 7f47f9590740 10 client.15711 fetch_fsmap learned FSMap version 25 2024-05-09T08:33:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.891+0000 7f47f9590740 1 -- 172.21.15.107:0/1408268245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x565160a67170 con 0x565160e300b0 2024-05-09T08:33:28.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.891+0000 7f47cf7fe640 1 -- 172.21.15.107:0/1408268245 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47e0019ae0 con 0x565160e300b0 2024-05-09T08:33:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.891+0000 7f47cf7fe640 1 -- 172.21.15.107:0/1408268245 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f47e00046c0 con 0x565160e300b0 2024-05-09T08:33:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.891+0000 7f47f9590740 10 client.15711 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:33:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.891+0000 7f47f9590740 10 client.15711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:33:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.891+0000 7f47f9590740 20 client.15711 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.891+0000 7f47f9590740 1 --2- 172.21.15.107:0/1408268245 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x565160e49780 0x565160e4bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.891+0000 7f47f9590740 4 client.15711 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:33:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.891+0000 7f47f9590740 1 -- 172.21.15.107:0/1408268245 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565160da10c0 con 0x565160e49780 2024-05-09T08:33:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.891+0000 7f47ef7fe640 1 --2- 172.21.15.107:0/1408268245 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x565160e49780 0x565160e4bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.892+0000 7f47cf7fe640 10 client.15711 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.892+0000 7f47ef7fe640 1 --2- 172.21.15.107:0/1408268245 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x565160e49780 0x565160e4bb70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=decc86138fbe72cf server_cookie=1a2a1c889cc6b718 in_seq=0 out_seq=0 2024-05-09T08:33:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.892+0000 7f47cf7fe640 1 -- 172.21.15.107:0/1408268245 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565160da10c0 con 0x565160e49780 2024-05-09T08:33:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.892+0000 7f47cf7fe640 10 client.15711 handle_command_reply: tid=0 2024-05-09T08:33:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.893+0000 7f47f9590740 1 client.15711 shutdown 2024-05-09T08:33:28.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.893+0000 7f47f9590740 1 -- 172.21.15.107:0/1408268245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e300b0 msgr2=0x565160e36bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.893+0000 7f47f9590740 1 --2- 172.21.15.107:0/1408268245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e300b0 0x565160e36bb0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f47e0005e00 tx=0x7f47e0019660 comp rx=0 tx=0).stop 2024-05-09T08:33:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.893+0000 7f47f9590740 1 -- 172.21.15.107:0/1408268245 shutdown_connections 2024-05-09T08:33:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.893+0000 7f47f9590740 1 --2- 172.21.15.107:0/1408268245 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x565160e49780 0x565160e4bb70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.893+0000 7f47f9590740 1 --2- 172.21.15.107:0/1408268245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e39740 0x565160e370f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.893+0000 7f47cf7fe640 0 client.15711 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.893+0000 7f47f9590740 1 --2- 172.21.15.107:0/1408268245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e300b0 0x565160e36bb0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.893+0000 7f47f9590740 1 --2- 172.21.15.107:0/1408268245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e2e9c0 0x565160e36670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.893+0000 7f47f9590740 1 -- 172.21.15.107:0/1408268245 >> 172.21.15.107:0/1408268245 conn(0x565160df70a0 msgr2=0x565160e30d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.893+0000 7f47f9590740 1 -- 172.21.15.107:0/1408268245 shutdown_connections 2024-05-09T08:33:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.893+0000 7f47f9590740 1 -- 172.21.15.107:0/1408268245 wait complete. 2024-05-09T08:33:28.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.893+0000 7f47f9590740 20 client.15711 trim_cache size 0 max 16384 2024-05-09T08:33:28.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.911+0000 7f47f9590740 1 Processor -- start 2024-05-09T08:33:28.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.911+0000 7f47f9590740 1 -- start start 2024-05-09T08:33:28.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.911+0000 7f47f9590740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e2e9c0 0x565160e454d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.911+0000 7f47f9590740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e300b0 0x565160e29f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.911+0000 7f47f9590740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e39740 0x565160e51290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.911+0000 7f47f9590740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f47f01b1450 con 0x565160e39740 2024-05-09T08:33:28.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.911+0000 7f47f9590740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f47f01aed20 con 0x565160e2e9c0 2024-05-09T08:33:28.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.911+0000 7f47f9590740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f47f01acb70 con 0x565160e300b0 2024-05-09T08:33:28.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.911+0000 7f47ef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e39740 0x565160e51290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.911+0000 7f47ef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e39740 0x565160e51290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50652/0 (socket says 172.21.15.107:50652) 2024-05-09T08:33:28.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.911+0000 7f47ef7fe640 1 -- 172.21.15.107:0/3578092509 learned_addr learned my addr 172.21.15.107:0/3578092509 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:28.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.911+0000 7f47effff640 1 --2- 172.21.15.107:0/3578092509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e300b0 0x565160e29f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.912+0000 7f47ef7fe640 1 --2- 172.21.15.107:0/3578092509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e2e9c0 0x565160e454d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.912+0000 7f47ef7fe640 1 -- 172.21.15.107:0/3578092509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e300b0 msgr2=0x565160e29f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.912+0000 7f47ef7fe640 1 --2- 172.21.15.107:0/3578092509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e300b0 0x565160e29f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.912+0000 7f47ef7fe640 1 -- 172.21.15.107:0/3578092509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e2e9c0 msgr2=0x565160e454d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.912+0000 7f47ef7fe640 1 --2- 172.21.15.107:0/3578092509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e2e9c0 0x565160e454d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.912+0000 7f47ef7fe640 1 -- 172.21.15.107:0/3578092509 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565160e35640 con 0x565160e39740 2024-05-09T08:33:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.912+0000 7f47ef7fe640 1 --2- 172.21.15.107:0/3578092509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e39740 0x565160e51290 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f47e4000c00 tx=0x7f47e4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b58b9402e73a6fa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.912+0000 7f47d5ffb640 1 -- 172.21.15.107:0/3578092509 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e4035040 con 0x565160e39740 2024-05-09T08:33:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.912+0000 7f47d5ffb640 1 -- 172.21.15.107:0/3578092509 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47e4003750 con 0x565160e39740 2024-05-09T08:33:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.912+0000 7f47d5ffb640 1 -- 172.21.15.107:0/3578092509 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e40314a0 con 0x565160e39740 2024-05-09T08:33:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.913+0000 7f47f9590740 1 -- 172.21.15.107:0/3578092509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e39740 msgr2=0x565160e51290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.913+0000 7f47f9590740 1 --2- 172.21.15.107:0/3578092509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e39740 0x565160e51290 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f47e4000c00 tx=0x7f47e4009260 comp rx=0 tx=0).stop 2024-05-09T08:33:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.913+0000 7f47f9590740 1 -- 172.21.15.107:0/3578092509 shutdown_connections 2024-05-09T08:33:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.913+0000 7f47f9590740 1 --2- 172.21.15.107:0/3578092509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e39740 0x565160e51290 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.913+0000 7f47f9590740 1 --2- 172.21.15.107:0/3578092509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e300b0 0x565160e29f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.913+0000 7f47f9590740 1 --2- 172.21.15.107:0/3578092509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e2e9c0 0x565160e454d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.913+0000 7f47f9590740 1 -- 172.21.15.107:0/3578092509 >> 172.21.15.107:0/3578092509 conn(0x565160df70a0 msgr2=0x565160e297f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.913+0000 7f47f9590740 1 -- 172.21.15.107:0/3578092509 shutdown_connections 2024-05-09T08:33:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.913+0000 7f47f9590740 1 -- 172.21.15.107:0/3578092509 wait complete. 2024-05-09T08:33:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.913+0000 7f47f9590740 1 -- start start 2024-05-09T08:33:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.913+0000 7f47f9590740 1 Processor -- start 2024-05-09T08:33:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47f9590740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e2e9c0 0x565160e339a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47f9590740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e300b0 0x565160e33ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47f9590740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e39740 0x565160eaeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47f9590740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f47f01ad1e0 con 0x565160e300b0 2024-05-09T08:33:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47f9590740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f47f01ae650 con 0x565160e2e9c0 2024-05-09T08:33:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47f9590740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f47f01ad8b0 con 0x565160e39740 2024-05-09T08:33:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47f9590740 10 client.0 _ll_register_callbacks cb 0x7f47e001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e300b0 0x565160e33ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e300b0 0x565160e33ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50668/0 (socket says 172.21.15.107:50668) 2024-05-09T08:33:28.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47effff640 1 -- 172.21.15.107:0/2649132987 learned_addr learned my addr 172.21.15.107:0/2649132987 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:28.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47ef7fe640 1 --2- 172.21.15.107:0/2649132987 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e39740 0x565160eaeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47effff640 1 -- 172.21.15.107:0/2649132987 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e39740 msgr2=0x565160eaeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:28.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47effff640 1 --2- 172.21.15.107:0/2649132987 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e39740 0x565160eaeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47effff640 1 -- 172.21.15.107:0/2649132987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e2e9c0 msgr2=0x565160e339a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:28.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47effff640 1 --2- 172.21.15.107:0/2649132987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e2e9c0 0x565160e339a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47effff640 1 -- 172.21.15.107:0/2649132987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565160e2b130 con 0x565160e300b0 2024-05-09T08:33:28.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47effff640 1 --2- 172.21.15.107:0/2649132987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e300b0 0x565160e33ee0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f47e0004a90 tx=0x7f47e00098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdf9b2ddb0bb321 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:28.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:28.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47cf7fe640 1 -- 172.21.15.107:0/2649132987 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e0027040 con 0x565160e300b0 2024-05-09T08:33:28.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47cf7fe640 1 -- 172.21.15.107:0/2649132987 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f47e0004040 con 0x565160e300b0 2024-05-09T08:33:28.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47f9590740 1 -- 172.21.15.107:0/2649132987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47e4003750 con 0x565160e300b0 2024-05-09T08:33:28.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47cf7fe640 1 -- 172.21.15.107:0/2649132987 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f47e001e070 con 0x565160e300b0 2024-05-09T08:33:28.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47f9590740 10 client.15717 fetch_fsmap learned FSMap version 25 2024-05-09T08:33:28.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.914+0000 7f47f9590740 1 -- 172.21.15.107:0/2649132987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x565160eb00c0 con 0x565160e300b0 2024-05-09T08:33:28.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.915+0000 7f47cf7fe640 1 -- 172.21.15.107:0/2649132987 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47e4003750 con 0x565160e300b0 2024-05-09T08:33:28.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.915+0000 7f47cf7fe640 1 -- 172.21.15.107:0/2649132987 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f47e00039d0 con 0x565160e300b0 2024-05-09T08:33:28.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.915+0000 7f47f9590740 10 client.15717 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:33:28.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.915+0000 7f47f9590740 10 client.15717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:33:28.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.915+0000 7f47f9590740 20 client.15717 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:28.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.915+0000 7f47f9590740 1 --2- 172.21.15.107:0/2649132987 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x565160eb0770 0x565160eb2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:28.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.915+0000 7f47f9590740 4 client.15717 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:33:28.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.915+0000 7f47f9590740 1 -- 172.21.15.107:0/2649132987 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x565160d9b410 con 0x565160eb0770 2024-05-09T08:33:28.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.915+0000 7f47ef7fe640 1 --2- 172.21.15.107:0/2649132987 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x565160eb0770 0x565160eb2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:28.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.915+0000 7f47cf7fe640 10 client.15717 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:28.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.915+0000 7f47ef7fe640 1 --2- 172.21.15.107:0/2649132987 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x565160eb0770 0x565160eb2b90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e33964280cdb26c6 server_cookie=82ee337a3d34a11c in_seq=0 out_seq=0 2024-05-09T08:33:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.917+0000 7f47cf7fe640 1 -- 172.21.15.107:0/2649132987 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x565160d9b410 con 0x565160eb0770 2024-05-09T08:33:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.917+0000 7f47cf7fe640 10 client.15717 handle_command_reply: tid=0 2024-05-09T08:33:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.917+0000 7f47f9590740 1 client.15717 shutdown 2024-05-09T08:33:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.917+0000 7f47f9590740 1 -- 172.21.15.107:0/2649132987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e300b0 msgr2=0x565160e33ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.917+0000 7f47f9590740 1 --2- 172.21.15.107:0/2649132987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e300b0 0x565160e33ee0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f47e0004a90 tx=0x7f47e00098f0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.917+0000 7f47f9590740 1 -- 172.21.15.107:0/2649132987 shutdown_connections 2024-05-09T08:33:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.917+0000 7f47f9590740 1 --2- 172.21.15.107:0/2649132987 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x565160eb0770 0x565160eb2b90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.917+0000 7f47f9590740 1 --2- 172.21.15.107:0/2649132987 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565160e39740 0x565160eaeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.917+0000 7f47f9590740 1 --2- 172.21.15.107:0/2649132987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565160e300b0 0x565160e33ee0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.917+0000 7f47cf7fe640 0 client.15717 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.917+0000 7f47f9590740 1 --2- 172.21.15.107:0/2649132987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565160e2e9c0 0x565160e339a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.917+0000 7f47f9590740 1 -- 172.21.15.107:0/2649132987 >> 172.21.15.107:0/2649132987 conn(0x565160df70a0 msgr2=0x565160e2a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:28.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.917+0000 7f47f9590740 1 -- 172.21.15.107:0/2649132987 shutdown_connections 2024-05-09T08:33:28.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.917+0000 7f47f9590740 1 -- 172.21.15.107:0/2649132987 wait complete. 2024-05-09T08:33:28.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.917+0000 7f47f9590740 20 client.15717 trim_cache size 0 max 16384 2024-05-09T08:33:28.922 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:33:28.922 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:33:28.922 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "f9d747b3-16d5-45bc-90a9-5fee9b7986c0", 2024-05-09T08:33:28.922 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:33:28.922 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:33:28.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.918+0000 7f47f6888640 1 -- 172.21.15.107:0/2900032350 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f47d80a2e20 msgr2=0x7f47d80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.918+0000 7f47f6888640 1 --2- 172.21.15.107:0/2900032350 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f47d80a2e20 0x7f47d80a52e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f47e80cd260 tx=0x7f47e806dec0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.918+0000 7f47f6888640 1 -- 172.21.15.107:0/2900032350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47f010be00 msgr2=0x7f47f019c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:28.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.918+0000 7f47f6888640 1 --2- 172.21.15.107:0/2900032350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47f010be00 0x7f47f019c960 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f47e800eb10 tx=0x7f47e800c2d0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.918+0000 7f47f6888640 1 -- 172.21.15.107:0/2900032350 shutdown_connections 2024-05-09T08:33:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.918+0000 7f47f6888640 1 --2- 172.21.15.107:0/2900032350 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f47d80a2e20 0x7f47d80a52e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.918+0000 7f47f6888640 1 --2- 172.21.15.107:0/2900032350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47f010be00 0x7f47f019c960 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.918+0000 7f47f6888640 1 --2- 172.21.15.107:0/2900032350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47f01033e0 0x7f47f01985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.918+0000 7f47f6888640 1 --2- 172.21.15.107:0/2900032350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47f0102ac0 0x7f47f01980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.918+0000 7f47f6888640 1 -- 172.21.15.107:0/2900032350 >> 172.21.15.107:0/2900032350 conn(0x7f47f0076ab0 msgr2=0x7f47f00fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.918+0000 7f47f6888640 1 -- 172.21.15.107:0/2900032350 shutdown_connections 2024-05-09T08:33:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:28.918+0000 7f47f6888640 1 -- 172.21.15.107:0/2900032350 wait complete. 2024-05-09T08:33:28.928 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:33:28.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:33:29.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.033+0000 7f6700177640 1 -- 172.21.15.107:0/1396077356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66f8106710 msgr2=0x7f66f8106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.033+0000 7f6700177640 1 --2- 172.21.15.107:0/1396077356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66f8106710 0x7f66f8106af0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f66f4009930 tx=0x7f66f402a420 comp rx=0 tx=0).stop 2024-05-09T08:33:29.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.033+0000 7f6700177640 1 -- 172.21.15.107:0/1396077356 shutdown_connections 2024-05-09T08:33:29.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.033+0000 7f6700177640 1 --2- 172.21.15.107:0/1396077356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66f81010c0 0x7f66f810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.033+0000 7f6700177640 1 --2- 172.21.15.107:0/1396077356 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66f8100700 0x7f66f8100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.033+0000 7f6700177640 1 --2- 172.21.15.107:0/1396077356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66f8106710 0x7f66f8106af0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.033+0000 7f6700177640 1 -- 172.21.15.107:0/1396077356 >> 172.21.15.107:0/1396077356 conn(0x7f66f8077830 msgr2=0x7f66f8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:29.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.034+0000 7f6700177640 1 -- 172.21.15.107:0/1396077356 shutdown_connections 2024-05-09T08:33:29.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.034+0000 7f6700177640 1 -- 172.21.15.107:0/1396077356 wait complete. 2024-05-09T08:33:29.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.034+0000 7f6700177640 1 Processor -- start 2024-05-09T08:33:29.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.034+0000 7f6700177640 1 -- start start 2024-05-09T08:33:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.034+0000 7f6700177640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66f8100700 0x7f66f8197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.034+0000 7f6700177640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66f81010c0 0x7f66f8198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.034+0000 7f6700177640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66f8106710 0x7f66f819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.034+0000 7f6700177640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f66f8110990 con 0x7f66f8106710 2024-05-09T08:33:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.034+0000 7f6700177640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f66f8110810 con 0x7f66f8100700 2024-05-09T08:33:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.034+0000 7f6700177640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f66f8110b10 con 0x7f66f81010c0 2024-05-09T08:33:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.034+0000 7f66fe714640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66f8106710 0x7f66f819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.034+0000 7f66fe714640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66f8106710 0x7f66f819c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50680/0 (socket says 172.21.15.107:50680) 2024-05-09T08:33:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.034+0000 7f66fe714640 1 -- 172.21.15.107:0/1531338984 learned_addr learned my addr 172.21.15.107:0/1531338984 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.034+0000 7f66fd712640 1 --2- 172.21.15.107:0/1531338984 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66f81010c0 0x7f66f8198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.034+0000 7f66fdf13640 1 --2- 172.21.15.107:0/1531338984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66f8100700 0x7f66f8197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.035+0000 7f66fe714640 1 -- 172.21.15.107:0/1531338984 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66f81010c0 msgr2=0x7f66f8198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.035+0000 7f66fe714640 1 --2- 172.21.15.107:0/1531338984 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66f81010c0 0x7f66f8198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.035+0000 7f66fe714640 1 -- 172.21.15.107:0/1531338984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66f8100700 msgr2=0x7f66f8197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.035+0000 7f66fe714640 1 --2- 172.21.15.107:0/1531338984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66f8100700 0x7f66f8197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.035+0000 7f66fe714640 1 -- 172.21.15.107:0/1531338984 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f819cfb0 con 0x7f66f8106710 2024-05-09T08:33:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.035+0000 7f66fe714640 1 --2- 172.21.15.107:0/1531338984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66f8106710 0x7f66f819c880 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f66f000cad0 tx=0x7f66f000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.035+0000 7f66eeffd640 1 -- 172.21.15.107:0/1531338984 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f000ad60 con 0x7f66f8106710 2024-05-09T08:33:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.035+0000 7f6700177640 1 -- 172.21.15.107:0/1531338984 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66f819d2a0 con 0x7f66f8106710 2024-05-09T08:33:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.035+0000 7f66eeffd640 1 -- 172.21.15.107:0/1531338984 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66f0005020 con 0x7f66f8106710 2024-05-09T08:33:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.035+0000 7f66eeffd640 1 -- 172.21.15.107:0/1531338984 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f000f6b0 con 0x7f66f8106710 2024-05-09T08:33:29.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.035+0000 7f6700177640 1 -- 172.21.15.107:0/1531338984 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66f81a4b30 con 0x7f66f8106710 2024-05-09T08:33:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.037+0000 7f66eeffd640 1 -- 172.21.15.107:0/1531338984 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7f66f00051d0 con 0x7f66f8106710 2024-05-09T08:33:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.037+0000 7f6702f63740 1 Processor -- start 2024-05-09T08:33:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f6702f63740 1 -- start start 2024-05-09T08:33:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f6702f63740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a2c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f6702f63740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a36d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f6702f63740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a37270 0x5627b5a39660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f6702f63740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5627b5991d00 con 0x5627b5a2c280 2024-05-09T08:33:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f6702f63740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5627b59394c0 con 0x5627b5a37270 2024-05-09T08:33:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f6702f63740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5627b598dcc0 con 0x5627b5a2cba0 2024-05-09T08:33:29.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f66fd712640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a2c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f66fd712640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a2c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50692/0 (socket says 172.21.15.107:50692) 2024-05-09T08:33:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f66fd712640 1 -- 172.21.15.107:0/1825632593 learned_addr learned my addr 172.21.15.107:0/1825632593 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f66fdf13640 1 --2- 172.21.15.107:0/1825632593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a36d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f66fd712640 1 --2- 172.21.15.107:0/1825632593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a37270 0x5627b5a39660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f66eeffd640 1 --2- 172.21.15.107:0/1531338984 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f66e40a2e20 0x7f66e40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f66fd712640 1 -- 172.21.15.107:0/1825632593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 msgr2=0x5627b5a36d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f66fd712640 1 --2- 172.21.15.107:0/1825632593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a36d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f66fd712640 1 -- 172.21.15.107:0/1825632593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a37270 msgr2=0x5627b5a39660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f66fd712640 1 --2- 172.21.15.107:0/1825632593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a37270 0x5627b5a39660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f66fd712640 1 -- 172.21.15.107:0/1825632593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627b576f7d0 con 0x5627b5a2c280 2024-05-09T08:33:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.038+0000 7f66fe714640 1 --2- 172.21.15.107:0/1531338984 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f66e40a2e20 0x7f66e40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.039+0000 7f66fd712640 1 --2- 172.21.15.107:0/1825632593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a2c660 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f66e8013610 tx=0x7f66e800a800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f0b720b14e9ee6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.039+0000 7f66eeffd640 1 -- 172.21.15.107:0/1531338984 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f66f00cf990 con 0x7f66f8106710 2024-05-09T08:33:29.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.039+0000 7f66e3fff640 1 -- 172.21.15.107:0/1825632593 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e8019030 con 0x5627b5a2c280 2024-05-09T08:33:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.039+0000 7f66e3fff640 1 -- 172.21.15.107:0/1825632593 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66e80110d0 con 0x5627b5a2c280 2024-05-09T08:33:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.039+0000 7f66e3fff640 1 -- 172.21.15.107:0/1825632593 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e800f7a0 con 0x5627b5a2c280 2024-05-09T08:33:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.039+0000 7f6702f63740 1 -- 172.21.15.107:0/1825632593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 msgr2=0x5627b5a2c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.039+0000 7f6702f63740 1 --2- 172.21.15.107:0/1825632593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a2c660 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f66e8013610 tx=0x7f66e800a800 comp rx=0 tx=0).stop 2024-05-09T08:33:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.039+0000 7f66fe714640 1 --2- 172.21.15.107:0/1531338984 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f66e40a2e20 0x7f66e40a52e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f66f00ccc60 tx=0x7f66f0066530 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.040+0000 7f6702f63740 1 -- 172.21.15.107:0/1825632593 shutdown_connections 2024-05-09T08:33:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.040+0000 7f6702f63740 1 --2- 172.21.15.107:0/1825632593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a37270 0x5627b5a39660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.040+0000 7f6702f63740 1 --2- 172.21.15.107:0/1825632593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a36d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.040+0000 7f6702f63740 1 --2- 172.21.15.107:0/1825632593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a2c660 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.040+0000 7f6702f63740 1 -- 172.21.15.107:0/1825632593 >> 172.21.15.107:0/1825632593 conn(0x5627b5a285a0 msgr2=0x5627b5a28980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.040+0000 7f6702f63740 1 -- 172.21.15.107:0/1825632593 shutdown_connections 2024-05-09T08:33:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.040+0000 7f6702f63740 1 -- 172.21.15.107:0/1825632593 wait complete. 2024-05-09T08:33:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.040+0000 7f6702f63740 1 -- start start 2024-05-09T08:33:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.040+0000 7f6702f63740 1 Processor -- start 2024-05-09T08:33:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.040+0000 7f6702f63740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a34210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.040+0000 7f6702f63740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a34750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.040+0000 7f6702f63740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a37270 0x5627b5a34c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.040+0000 7f6702f63740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5627b5a3a060 con 0x5627b5a2c280 2024-05-09T08:33:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.040+0000 7f6702f63740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5627b594cb90 con 0x5627b5a37270 2024-05-09T08:33:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.040+0000 7f6702f63740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5627b5a3a1e0 con 0x5627b5a2cba0 2024-05-09T08:33:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.040+0000 7f6702f63740 10 client.0 _ll_register_callbacks cb 0x5627b5988da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f66fd712640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a34210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f66fd712640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a34210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50700/0 (socket says 172.21.15.107:50700) 2024-05-09T08:33:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f66fd712640 1 -- 172.21.15.107:0/2250562401 learned_addr learned my addr 172.21.15.107:0/2250562401 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f66fdf13640 1 --2- 172.21.15.107:0/2250562401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a34750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f66fd712640 1 -- 172.21.15.107:0/2250562401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 msgr2=0x5627b5a34750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:29.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f66fd712640 1 --2- 172.21.15.107:0/2250562401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a34750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f66fd712640 1 -- 172.21.15.107:0/2250562401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a37270 msgr2=0x5627b5a34c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f66fd712640 1 --2- 172.21.15.107:0/2250562401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a37270 0x5627b5a34c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f66fd712640 1 -- 172.21.15.107:0/2250562401 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627b56a53f0 con 0x5627b5a2c280 2024-05-09T08:33:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f66fd712640 1 --2- 172.21.15.107:0/2250562401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a34210 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f66e8011fa0 tx=0x7f66e800ae90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a950cd6fca7e2b98 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f66e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f66e17fa640 1 -- 172.21.15.107:0/2250562401 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e8019040 con 0x5627b5a2c280 2024-05-09T08:33:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f66e17fa640 1 -- 172.21.15.107:0/2250562401 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f66e8005240 con 0x5627b5a2c280 2024-05-09T08:33:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f6702f63740 1 -- 172.21.15.107:0/2250562401 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66e80110d0 con 0x5627b5a2c280 2024-05-09T08:33:29.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f66e17fa640 1 -- 172.21.15.107:0/2250562401 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f66e8005440 con 0x5627b5a2c280 2024-05-09T08:33:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f6702f63740 10 client.15729 fetch_fsmap learned FSMap version 25 2024-05-09T08:33:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.041+0000 7f6702f63740 1 -- 172.21.15.107:0/2250562401 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5627b56650f0 con 0x5627b5a2c280 2024-05-09T08:33:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.042+0000 7f66e17fa640 1 -- 172.21.15.107:0/2250562401 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66e80110d0 con 0x5627b5a2c280 2024-05-09T08:33:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.042+0000 7f66e17fa640 1 -- 172.21.15.107:0/2250562401 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f66e800f7b0 con 0x5627b5a2c280 2024-05-09T08:33:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.042+0000 7f6702f63740 10 client.15729 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:33:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.042+0000 7f6702f63740 10 client.15729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:33:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.042+0000 7f6702f63740 20 client.15729 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.042+0000 7f6702f63740 1 --2- 172.21.15.107:0/2250562401 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5627b5a474a0 0x5627b5a49890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.042+0000 7f6702f63740 4 client.15729 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:33:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.042+0000 7f6702f63740 1 -- 172.21.15.107:0/2250562401 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627b599f430 con 0x5627b5a474a0 2024-05-09T08:33:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.042+0000 7f66fdf13640 1 --2- 172.21.15.107:0/2250562401 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5627b5a474a0 0x5627b5a49890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.042+0000 7f66fdf13640 1 --2- 172.21.15.107:0/2250562401 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5627b5a474a0 0x5627b5a49890 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfd3ce1af5b4ca79 server_cookie=c826ff55cff1ed02 in_seq=0 out_seq=0 2024-05-09T08:33:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.042+0000 7f66e17fa640 10 client.15729 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.043+0000 7f66e17fa640 1 -- 172.21.15.107:0/2250562401 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5627b599f430 con 0x5627b5a474a0 2024-05-09T08:33:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.043+0000 7f66e17fa640 10 client.15729 handle_command_reply: tid=0 2024-05-09T08:33:29.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.043+0000 7f6702f63740 1 client.15729 shutdown 2024-05-09T08:33:29.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.043+0000 7f6702f63740 1 -- 172.21.15.107:0/2250562401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 msgr2=0x5627b5a34210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.043+0000 7f6702f63740 1 --2- 172.21.15.107:0/2250562401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a34210 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f66e8011fa0 tx=0x7f66e800ae90 comp rx=0 tx=0).stop 2024-05-09T08:33:29.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.043+0000 7f6702f63740 1 -- 172.21.15.107:0/2250562401 shutdown_connections 2024-05-09T08:33:29.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.043+0000 7f6702f63740 1 --2- 172.21.15.107:0/2250562401 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5627b5a474a0 0x5627b5a49890 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.043+0000 7f6702f63740 1 --2- 172.21.15.107:0/2250562401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a37270 0x5627b5a34c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.043+0000 7f6702f63740 1 --2- 172.21.15.107:0/2250562401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a34750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.043+0000 7f6702f63740 1 --2- 172.21.15.107:0/2250562401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a34210 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.043+0000 7f66e17fa640 0 client.15729 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:29.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.043+0000 7f6702f63740 1 -- 172.21.15.107:0/2250562401 >> 172.21.15.107:0/2250562401 conn(0x5627b5a285a0 msgr2=0x5627b5a2e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:29.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.043+0000 7f6702f63740 1 -- 172.21.15.107:0/2250562401 shutdown_connections 2024-05-09T08:33:29.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.043+0000 7f6702f63740 1 -- 172.21.15.107:0/2250562401 wait complete. 2024-05-09T08:33:29.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.043+0000 7f6702f63740 20 client.15729 trim_cache size 0 max 16384 2024-05-09T08:33:29.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f6702f63740 1 Processor -- start 2024-05-09T08:33:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f6702f63740 1 -- start start 2024-05-09T08:33:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f6702f63740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a2c280 0x5627b5a42f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f6702f63740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a34a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f6702f63740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a37270 0x5627b5a4ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f6702f63740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f66f81b13c0 con 0x5627b5a37270 2024-05-09T08:33:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f6702f63740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f66f81aec90 con 0x5627b5a2c280 2024-05-09T08:33:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f6702f63740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f66f81acae0 con 0x5627b5a2cba0 2024-05-09T08:33:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f66fd712640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a37270 0x5627b5a4ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f66fd712640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a37270 0x5627b5a4ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50708/0 (socket says 172.21.15.107:50708) 2024-05-09T08:33:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f66fd712640 1 -- 172.21.15.107:0/1179307962 learned_addr learned my addr 172.21.15.107:0/1179307962 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f66fd712640 1 --2- 172.21.15.107:0/1179307962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a2c280 0x5627b5a42f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f66fdf13640 1 --2- 172.21.15.107:0/1179307962 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a34a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f66fd712640 1 -- 172.21.15.107:0/1179307962 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 msgr2=0x5627b5a34a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:29.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f66fd712640 1 --2- 172.21.15.107:0/1179307962 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a34a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f66fd712640 1 -- 172.21.15.107:0/1179307962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a2c280 msgr2=0x5627b5a42f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:29.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f66fd712640 1 --2- 172.21.15.107:0/1179307962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a2c280 0x5627b5a42f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f66fd712640 1 -- 172.21.15.107:0/1179307962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627b5a28ee0 con 0x5627b5a37270 2024-05-09T08:33:29.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f66fdf13640 1 --2- 172.21.15.107:0/1179307962 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a34a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:29.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f66fd712640 1 --2- 172.21.15.107:0/1179307962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a37270 0x5627b5a4ee90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f66e8013500 tx=0x7f66e800af50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ddc9b5a2545aef3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:29.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.061+0000 7f66e37fe640 1 -- 172.21.15.107:0/1179307962 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e8011090 con 0x5627b5a37270 2024-05-09T08:33:29.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.062+0000 7f66e37fe640 1 -- 172.21.15.107:0/1179307962 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66e8024070 con 0x5627b5a37270 2024-05-09T08:33:29.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.062+0000 7f66e37fe640 1 -- 172.21.15.107:0/1179307962 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e80048c0 con 0x5627b5a37270 2024-05-09T08:33:29.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.062+0000 7f6702f63740 1 -- 172.21.15.107:0/1179307962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a37270 msgr2=0x5627b5a4ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.062+0000 7f6702f63740 1 --2- 172.21.15.107:0/1179307962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a37270 0x5627b5a4ee90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f66e8013500 tx=0x7f66e800af50 comp rx=0 tx=0).stop 2024-05-09T08:33:29.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.062+0000 7f6702f63740 1 -- 172.21.15.107:0/1179307962 shutdown_connections 2024-05-09T08:33:29.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.062+0000 7f6702f63740 1 --2- 172.21.15.107:0/1179307962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a37270 0x5627b5a4ee90 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.062+0000 7f6702f63740 1 --2- 172.21.15.107:0/1179307962 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a34a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.062+0000 7f6702f63740 1 --2- 172.21.15.107:0/1179307962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a2c280 0x5627b5a42f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.062+0000 7f6702f63740 1 -- 172.21.15.107:0/1179307962 >> 172.21.15.107:0/1179307962 conn(0x5627b5a285a0 msgr2=0x5627b5a2d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.062+0000 7f6702f63740 1 -- 172.21.15.107:0/1179307962 shutdown_connections 2024-05-09T08:33:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.062+0000 7f6702f63740 1 -- 172.21.15.107:0/1179307962 wait complete. 2024-05-09T08:33:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.063+0000 7f6702f63740 1 -- start start 2024-05-09T08:33:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.063+0000 7f6702f63740 1 Processor -- start 2024-05-09T08:33:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.063+0000 7f6702f63740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a3aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.063+0000 7f6702f63740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a3b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.063+0000 7f6702f63740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a37270 0x5627b5a3b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.063+0000 7f6702f63740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f66f81ad150 con 0x5627b5a2c280 2024-05-09T08:33:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.063+0000 7f6702f63740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f66f81ae5c0 con 0x5627b5a37270 2024-05-09T08:33:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.063+0000 7f6702f63740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f66f81ad820 con 0x5627b5a2cba0 2024-05-09T08:33:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.063+0000 7f6702f63740 10 client.0 _ll_register_callbacks cb 0x7f66f4005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.063+0000 7f66fd712640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a3aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.063+0000 7f66fd712640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a3aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50720/0 (socket says 172.21.15.107:50720) 2024-05-09T08:33:29.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.063+0000 7f66fd712640 1 -- 172.21.15.107:0/2229355991 learned_addr learned my addr 172.21.15.107:0/2229355991 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:29.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.063+0000 7f66fdf13640 1 --2- 172.21.15.107:0/2229355991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a3b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.063+0000 7f66fd712640 1 --2- 172.21.15.107:0/2229355991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a37270 0x5627b5a3b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f66fd712640 1 -- 172.21.15.107:0/2229355991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 msgr2=0x5627b5a3b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:29.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f66fd712640 1 --2- 172.21.15.107:0/2229355991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a3b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f66fd712640 1 -- 172.21.15.107:0/2229355991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a37270 msgr2=0x5627b5a3b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:29.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f66fd712640 1 --2- 172.21.15.107:0/2229355991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a37270 0x5627b5a3b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f66fd712640 1 -- 172.21.15.107:0/2229355991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627b5aac780 con 0x5627b5a2c280 2024-05-09T08:33:29.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f66fdf13640 1 --2- 172.21.15.107:0/2229355991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a3b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:29.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f66fd712640 1 --2- 172.21.15.107:0/2229355991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a3aed0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f66e8011330 tx=0x7f66e8005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf495e6716dd9f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:29.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f66e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:29.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f66e17fa640 1 -- 172.21.15.107:0/2229355991 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e8019040 con 0x5627b5a2c280 2024-05-09T08:33:29.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f66e17fa640 1 -- 172.21.15.107:0/2229355991 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f66e800a6f0 con 0x5627b5a2c280 2024-05-09T08:33:29.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f6702f63740 1 -- 172.21.15.107:0/2229355991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66e8024070 con 0x5627b5a2c280 2024-05-09T08:33:29.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f66e17fa640 1 -- 172.21.15.107:0/2229355991 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f66e80055e0 con 0x5627b5a2c280 2024-05-09T08:33:29.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f6702f63740 10 client.15735 fetch_fsmap learned FSMap version 25 2024-05-09T08:33:29.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f6702f63740 1 -- 172.21.15.107:0/2229355991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5627b5aad9e0 con 0x5627b5a2c280 2024-05-09T08:33:29.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f66e17fa640 1 -- 172.21.15.107:0/2229355991 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66e8024070 con 0x5627b5a2c280 2024-05-09T08:33:29.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f66e17fa640 1 -- 172.21.15.107:0/2229355991 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f66e8004bf0 con 0x5627b5a2c280 2024-05-09T08:33:29.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f6702f63740 10 client.15735 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:33:29.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f6702f63740 10 client.15735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:33:29.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.064+0000 7f6702f63740 20 client.15735 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:29.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.065+0000 7f6702f63740 1 --2- 172.21.15.107:0/2229355991 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5627b5aadfe0 0x5627b5ab0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.065+0000 7f6702f63740 4 client.15735 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:33:29.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.065+0000 7f6702f63740 1 -- 172.21.15.107:0/2229355991 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5627b5999780 con 0x5627b5aadfe0 2024-05-09T08:33:29.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.065+0000 7f66fdf13640 1 --2- 172.21.15.107:0/2229355991 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5627b5aadfe0 0x5627b5ab0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.065+0000 7f66fdf13640 1 --2- 172.21.15.107:0/2229355991 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5627b5aadfe0 0x5627b5ab0400 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ef04d2b88f54b1d server_cookie=594c52fbde88530a in_seq=0 out_seq=0 2024-05-09T08:33:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.065+0000 7f66e17fa640 10 client.15735 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.065+0000 7f66e17fa640 1 -- 172.21.15.107:0/2229355991 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5627b5999780 con 0x5627b5aadfe0 2024-05-09T08:33:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.065+0000 7f66e17fa640 10 client.15735 handle_command_reply: tid=0 2024-05-09T08:33:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.065+0000 7f6702f63740 1 client.15735 shutdown 2024-05-09T08:33:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.065+0000 7f6702f63740 1 -- 172.21.15.107:0/2229355991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 msgr2=0x5627b5a3aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.065+0000 7f6702f63740 1 --2- 172.21.15.107:0/2229355991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a3aed0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f66e8011330 tx=0x7f66e8005850 comp rx=0 tx=0).stop 2024-05-09T08:33:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6702f63740 1 -- 172.21.15.107:0/2229355991 shutdown_connections 2024-05-09T08:33:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6702f63740 1 --2- 172.21.15.107:0/2229355991 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5627b5aadfe0 0x5627b5ab0400 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6702f63740 1 --2- 172.21.15.107:0/2229355991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627b5a37270 0x5627b5a3b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f66e17fa640 0 client.15735 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6702f63740 1 --2- 172.21.15.107:0/2229355991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5627b5a2cba0 0x5627b5a3b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6702f63740 1 --2- 172.21.15.107:0/2229355991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5627b5a2c280 0x5627b5a3aed0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6702f63740 1 -- 172.21.15.107:0/2229355991 >> 172.21.15.107:0/2229355991 conn(0x5627b5a285a0 msgr2=0x5627b5a348e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:29.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6702f63740 1 -- 172.21.15.107:0/2229355991 shutdown_connections 2024-05-09T08:33:29.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6702f63740 1 -- 172.21.15.107:0/2229355991 wait complete. 2024-05-09T08:33:29.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6702f63740 20 client.15735 trim_cache size 0 max 16384 2024-05-09T08:33:29.072 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:33:29.072 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:33:29.072 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:33:29.072 INFO:teuthology.orchestra.run.smithi107.stdout: "f9d747b3-16d5-45bc-90a9-5fee9b7986c0": { 2024-05-09T08:33:29.072 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:33:29.072 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "f9d747b3-16d5-45bc-90a9-5fee9b7986c0", 2024-05-09T08:33:29.073 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:33:29.073 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:33:29.073 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:33:29.073 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:33:29.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6700177640 1 -- 172.21.15.107:0/1531338984 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f66e40a2e20 msgr2=0x7f66e40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6700177640 1 --2- 172.21.15.107:0/1531338984 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f66e40a2e20 0x7f66e40a52e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f66f00ccc60 tx=0x7f66f0066530 comp rx=0 tx=0).stop 2024-05-09T08:33:29.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6700177640 1 -- 172.21.15.107:0/1531338984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66f8106710 msgr2=0x7f66f819c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6700177640 1 --2- 172.21.15.107:0/1531338984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66f8106710 0x7f66f819c880 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f66f000cad0 tx=0x7f66f000ced0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6700177640 1 -- 172.21.15.107:0/1531338984 shutdown_connections 2024-05-09T08:33:29.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6700177640 1 --2- 172.21.15.107:0/1531338984 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f66e40a2e20 0x7f66e40a52e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6700177640 1 --2- 172.21.15.107:0/1531338984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66f8106710 0x7f66f819c880 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6700177640 1 --2- 172.21.15.107:0/1531338984 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66f81010c0 0x7f66f8198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6700177640 1 --2- 172.21.15.107:0/1531338984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66f8100700 0x7f66f8197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6700177640 1 -- 172.21.15.107:0/1531338984 >> 172.21.15.107:0/1531338984 conn(0x7f66f8077830 msgr2=0x7f66f80ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:29.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.066+0000 7f6700177640 1 -- 172.21.15.107:0/1531338984 shutdown_connections 2024-05-09T08:33:29.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.067+0000 7f6700177640 1 -- 172.21.15.107:0/1531338984 wait complete. 2024-05-09T08:33:29.080 INFO:tasks.cephfs.filesystem:scrub status for tag:f9d747b3-16d5-45bc-90a9-5fee9b7986c0 - {'path': '/', 'tag': 'f9d747b3-16d5-45bc-90a9-5fee9b7986c0', 'options': 'recursive,force'} 2024-05-09T08:33:29.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.107+0000 7fdd45415640 1 -- 172.21.15.107:0/31557377 --> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fdd2c000bc0 con 0x7fdd280a2ec0 2024-05-09T08:33:29.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.169+0000 7fdd27fff640 1 -- 172.21.15.107:0/31557377 <== mgr.15483 v2:172.21.15.107:6800/165281804 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fdd2c000bc0 con 0x7fdd280a2ec0 2024-05-09T08:33:29.168 INFO:teuthology.orchestra.run.smithi107.stdout:Schedule set for path / 2024-05-09T08:33:29.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.171+0000 7fdd45415640 1 -- 172.21.15.107:0/31557377 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdd280a2ec0 msgr2=0x7fdd280a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.171+0000 7fdd45415640 1 --2- 172.21.15.107:0/31557377 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdd280a2ec0 0x7fdd280a5380 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdd401001c0 tx=0x7fdd3400b390 comp rx=0 tx=0).stop 2024-05-09T08:33:29.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.171+0000 7fdd45415640 1 -- 172.21.15.107:0/31557377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd40102ae0 msgr2=0x7fdd40198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.171+0000 7fdd45415640 1 --2- 172.21.15.107:0/31557377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd40102ae0 0x7fdd40198000 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdd30005940 tx=0x7fdd300042f0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.171+0000 7fdd45415640 1 -- 172.21.15.107:0/31557377 shutdown_connections 2024-05-09T08:33:29.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.171+0000 7fdd45415640 1 --2- 172.21.15.107:0/31557377 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdd280a2ec0 0x7fdd280a5380 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.171+0000 7fdd45415640 1 --2- 172.21.15.107:0/31557377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd4010be20 0x7fdd4019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.171+0000 7fdd45415640 1 --2- 172.21.15.107:0/31557377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd40103400 0x7fdd40198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.171+0000 7fdd45415640 1 --2- 172.21.15.107:0/31557377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd40102ae0 0x7fdd40198000 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.171+0000 7fdd45415640 1 -- 172.21.15.107:0/31557377 >> 172.21.15.107:0/31557377 conn(0x7fdd400778b0 msgr2=0x7fdd400fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:29.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.171+0000 7fdd45415640 1 -- 172.21.15.107:0/31557377 shutdown_connections 2024-05-09T08:33:29.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.171+0000 7fdd45415640 1 -- 172.21.15.107:0/31557377 wait complete. 2024-05-09T08:33:29.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-05-09T08:33:29.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.326+0000 7fe26fc09640 1 -- 172.21.15.107:0/1832849516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe268100950 msgr2=0x7fe268106480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.326+0000 7fe26fc09640 1 --2- 172.21.15.107:0/1832849516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe268100950 0x7fe268106480 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009900 tx=0x7fe25c02a410 comp rx=0 tx=0).stop 2024-05-09T08:33:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.326+0000 7fe26fc09640 1 -- 172.21.15.107:0/1832849516 shutdown_connections 2024-05-09T08:33:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.326+0000 7fe26fc09640 1 --2- 172.21.15.107:0/1832849516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe268106b60 0x7fe26810e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.326+0000 7fe26fc09640 1 --2- 172.21.15.107:0/1832849516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe268100950 0x7fe268106480 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.326+0000 7fe26fc09640 1 --2- 172.21.15.107:0/1832849516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2680fffa0 0x7fe268100380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.326+0000 7fe26fc09640 1 -- 172.21.15.107:0/1832849516 >> 172.21.15.107:0/1832849516 conn(0x7fe268076cc0 msgr2=0x7fe2680fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.326+0000 7fe26fc09640 1 -- 172.21.15.107:0/1832849516 shutdown_connections 2024-05-09T08:33:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.326+0000 7fe26fc09640 1 -- 172.21.15.107:0/1832849516 wait complete. 2024-05-09T08:33:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.327+0000 7fe26fc09640 1 Processor -- start 2024-05-09T08:33:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.327+0000 7fe26fc09640 1 -- start start 2024-05-09T08:33:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.327+0000 7fe26fc09640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2680fffa0 0x7fe26819a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.327+0000 7fe26fc09640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe268106b60 0x7fe26819a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.327+0000 7fe26fc09640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe26819ea40 0x7fe26819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.327+0000 7fe26fc09640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe268111ac0 con 0x7fe2680fffa0 2024-05-09T08:33:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.327+0000 7fe26fc09640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe268111940 con 0x7fe268106b60 2024-05-09T08:33:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.327+0000 7fe26fc09640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe268111c40 con 0x7fe26819ea40 2024-05-09T08:33:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.327+0000 7fe26d9a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2680fffa0 0x7fe26819a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.327+0000 7fe26d9a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2680fffa0 0x7fe26819a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50740/0 (socket says 172.21.15.107:50740) 2024-05-09T08:33:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.327+0000 7fe26d9a5640 1 -- 172.21.15.107:0/1681379746 learned_addr learned my addr 172.21.15.107:0/1681379746 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.327+0000 7fe26d1a4640 1 --2- 172.21.15.107:0/1681379746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe268106b60 0x7fe26819a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.327+0000 7fe26e1a6640 1 --2- 172.21.15.107:0/1681379746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe26819ea40 0x7fe26819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.328+0000 7fe26d9a5640 1 -- 172.21.15.107:0/1681379746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe26819ea40 msgr2=0x7fe26819eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.328+0000 7fe26d9a5640 1 --2- 172.21.15.107:0/1681379746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe26819ea40 0x7fe26819eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.328+0000 7fe26d9a5640 1 -- 172.21.15.107:0/1681379746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe268106b60 msgr2=0x7fe26819a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.328+0000 7fe26d9a5640 1 --2- 172.21.15.107:0/1681379746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe268106b60 0x7fe26819a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.328+0000 7fe26d9a5640 1 -- 172.21.15.107:0/1681379746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26819f530 con 0x7fe2680fffa0 2024-05-09T08:33:29.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.328+0000 7fe26d1a4640 1 --2- 172.21.15.107:0/1681379746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe268106b60 0x7fe26819a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:29.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.328+0000 7fe26d9a5640 1 --2- 172.21.15.107:0/1681379746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2680fffa0 0x7fe26819a180 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe25800ebc0 tx=0x7fe25800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:29.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.328+0000 7fe266ffd640 1 -- 172.21.15.107:0/1681379746 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe258010630 con 0x7fe2680fffa0 2024-05-09T08:33:29.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.328+0000 7fe26fc09640 1 -- 172.21.15.107:0/1681379746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe26819f820 con 0x7fe2680fffa0 2024-05-09T08:33:29.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.328+0000 7fe266ffd640 1 -- 172.21.15.107:0/1681379746 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe258010c80 con 0x7fe2680fffa0 2024-05-09T08:33:29.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.328+0000 7fe266ffd640 1 -- 172.21.15.107:0/1681379746 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe258018b30 con 0x7fe2680fffa0 2024-05-09T08:33:29.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.328+0000 7fe26fc09640 1 -- 172.21.15.107:0/1681379746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2681a6fe0 con 0x7fe2680fffa0 2024-05-09T08:33:29.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.329+0000 7fe26fc09640 1 -- 172.21.15.107:0/1681379746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe26804f5f0 con 0x7fe2680fffa0 2024-05-09T08:33:29.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.330+0000 7fe266ffd640 1 -- 172.21.15.107:0/1681379746 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7fe258018cd0 con 0x7fe2680fffa0 2024-05-09T08:33:29.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.331+0000 7fe266ffd640 1 --2- 172.21.15.107:0/1681379746 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2500a2ec0 0x7fe2500a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.331+0000 7fe26d1a4640 1 --2- 172.21.15.107:0/1681379746 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2500a2ec0 0x7fe2500a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.331+0000 7fe266ffd640 1 -- 172.21.15.107:0/1681379746 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe2580d05e0 con 0x7fe2680fffa0 2024-05-09T08:33:29.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.331+0000 7fe26d1a4640 1 --2- 172.21.15.107:0/1681379746 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2500a2ec0 0x7fe2500a5380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe25c000c00 tx=0x7fe25c009610 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:29.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.335+0000 7fe266ffd640 1 -- 172.21.15.107:0/1681379746 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe25809cc50 con 0x7fe2680fffa0 2024-05-09T08:33:29.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.533+0000 7fe26fc09640 1 -- 172.21.15.107:0/1681379746 --> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fe2681009c0 con 0x7fe2500a2ec0 2024-05-09T08:33:29.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.593+0000 7fe266ffd640 1 -- 172.21.15.107:0/1681379746 <== mgr.15483 v2:172.21.15.107:6800/165281804 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fe2681009c0 con 0x7fe2500a2ec0 2024-05-09T08:33:29.592 INFO:teuthology.orchestra.run.smithi107.stdout:Retention added to path / 2024-05-09T08:33:29.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.595+0000 7fe26fc09640 1 -- 172.21.15.107:0/1681379746 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2500a2ec0 msgr2=0x7fe2500a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.595+0000 7fe26fc09640 1 --2- 172.21.15.107:0/1681379746 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2500a2ec0 0x7fe2500a5380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe25c000c00 tx=0x7fe25c009610 comp rx=0 tx=0).stop 2024-05-09T08:33:29.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.595+0000 7fe26fc09640 1 -- 172.21.15.107:0/1681379746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2680fffa0 msgr2=0x7fe26819a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.595+0000 7fe26fc09640 1 --2- 172.21.15.107:0/1681379746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2680fffa0 0x7fe26819a180 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe25800ebc0 tx=0x7fe25800c6a0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.595+0000 7fe26fc09640 1 -- 172.21.15.107:0/1681379746 shutdown_connections 2024-05-09T08:33:29.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.595+0000 7fe26fc09640 1 --2- 172.21.15.107:0/1681379746 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2500a2ec0 0x7fe2500a5380 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.595+0000 7fe26fc09640 1 --2- 172.21.15.107:0/1681379746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe26819ea40 0x7fe26819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.595+0000 7fe26fc09640 1 --2- 172.21.15.107:0/1681379746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe268106b60 0x7fe26819a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.595+0000 7fe26fc09640 1 --2- 172.21.15.107:0/1681379746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2680fffa0 0x7fe26819a180 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.595+0000 7fe26fc09640 1 -- 172.21.15.107:0/1681379746 >> 172.21.15.107:0/1681379746 conn(0x7fe268076cc0 msgr2=0x7fe2681052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:29.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.595+0000 7fe26fc09640 1 -- 172.21.15.107:0/1681379746 shutdown_connections 2024-05-09T08:33:29.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.595+0000 7fe26fc09640 1 -- 172.21.15.107:0/1681379746 wait complete. 2024-05-09T08:33:29.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-09T08:33:29.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.708+0000 7fd4c5d68640 1 -- 172.21.15.107:0/3730462500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c00fffa0 msgr2=0x7fd4c0100380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.708+0000 7fd4c5d68640 1 --2- 172.21.15.107:0/3730462500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c00fffa0 0x7fd4c0100380 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0009900 tx=0x7fd4b002a410 comp rx=0 tx=0).stop 2024-05-09T08:33:29.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.708+0000 7fd4c5d68640 1 -- 172.21.15.107:0/3730462500 shutdown_connections 2024-05-09T08:33:29.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.708+0000 7fd4c5d68640 1 --2- 172.21.15.107:0/3730462500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4c0106c10 0x7fd4c010e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.708+0000 7fd4c5d68640 1 --2- 172.21.15.107:0/3730462500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c0100950 0x7fd4c01064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.708+0000 7fd4c5d68640 1 --2- 172.21.15.107:0/3730462500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c00fffa0 0x7fd4c0100380 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.708+0000 7fd4c5d68640 1 -- 172.21.15.107:0/3730462500 >> 172.21.15.107:0/3730462500 conn(0x7fd4c0076c90 msgr2=0x7fd4c00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:29.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.708+0000 7fd4c5d68640 1 -- 172.21.15.107:0/3730462500 shutdown_connections 2024-05-09T08:33:29.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.708+0000 7fd4c5d68640 1 -- 172.21.15.107:0/3730462500 wait complete. 2024-05-09T08:33:29.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4c5d68640 1 Processor -- start 2024-05-09T08:33:29.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4c5d68640 1 -- start start 2024-05-09T08:33:29.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4c5d68640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c00fffa0 0x7fd4c0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4c5d68640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c0106c10 0x7fd4c0198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4c5d68640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4c019c8a0 0x7fd4c019cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4c5d68640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c0111b20 con 0x7fd4c00fffa0 2024-05-09T08:33:29.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4c5d68640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c01119a0 con 0x7fd4c019c8a0 2024-05-09T08:33:29.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4c5d68640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c0111ca0 con 0x7fd4c0106c10 2024-05-09T08:33:29.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4bf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c00fffa0 0x7fd4c0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4bf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c00fffa0 0x7fd4c0197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50758/0 (socket says 172.21.15.107:50758) 2024-05-09T08:33:29.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4bf7fe640 1 -- 172.21.15.107:0/2012902137 learned_addr learned my addr 172.21.15.107:0/2012902137 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:29.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4bffff640 1 --2- 172.21.15.107:0/2012902137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4c019c8a0 0x7fd4c019cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4beffd640 1 --2- 172.21.15.107:0/2012902137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c0106c10 0x7fd4c0198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4bf7fe640 1 -- 172.21.15.107:0/2012902137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c0106c10 msgr2=0x7fd4c0198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4bf7fe640 1 --2- 172.21.15.107:0/2012902137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c0106c10 0x7fd4c0198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4bf7fe640 1 -- 172.21.15.107:0/2012902137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4c019c8a0 msgr2=0x7fd4c019cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4bf7fe640 1 --2- 172.21.15.107:0/2012902137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4c019c8a0 0x7fd4c019cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4bf7fe640 1 -- 172.21.15.107:0/2012902137 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c019d390 con 0x7fd4c00fffa0 2024-05-09T08:33:29.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.709+0000 7fd4beffd640 1 --2- 172.21.15.107:0/2012902137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c0106c10 0x7fd4c0198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:29.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.710+0000 7fd4bf7fe640 1 --2- 172.21.15.107:0/2012902137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c00fffa0 0x7fd4c0197fe0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0005940 tx=0x7fd4b0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:29.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.710+0000 7fd4bcff9640 1 -- 172.21.15.107:0/2012902137 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b003f030 con 0x7fd4c00fffa0 2024-05-09T08:33:29.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.710+0000 7fd4bcff9640 1 -- 172.21.15.107:0/2012902137 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4b0003aa0 con 0x7fd4c00fffa0 2024-05-09T08:33:29.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.710+0000 7fd4bcff9640 1 -- 172.21.15.107:0/2012902137 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0035070 con 0x7fd4c00fffa0 2024-05-09T08:33:29.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.710+0000 7fd4c5d68640 1 -- 172.21.15.107:0/2012902137 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c019d620 con 0x7fd4c00fffa0 2024-05-09T08:33:29.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.710+0000 7fd4c5d68640 1 -- 172.21.15.107:0/2012902137 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c01a4de0 con 0x7fd4c00fffa0 2024-05-09T08:33:29.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.711+0000 7fd4c5d68640 1 -- 172.21.15.107:0/2012902137 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c004f5f0 con 0x7fd4c00fffa0 2024-05-09T08:33:29.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.712+0000 7fd4bcff9640 1 -- 172.21.15.107:0/2012902137 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7fd4b0042030 con 0x7fd4c00fffa0 2024-05-09T08:33:29.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.713+0000 7fd4bcff9640 1 --2- 172.21.15.107:0/2012902137 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4a80a2ec0 0x7fd4a80a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:29.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.713+0000 7fd4beffd640 1 --2- 172.21.15.107:0/2012902137 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4a80a2ec0 0x7fd4a80a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:29.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.713+0000 7fd4bcff9640 1 -- 172.21.15.107:0/2012902137 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd4b00eff60 con 0x7fd4c00fffa0 2024-05-09T08:33:29.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.713+0000 7fd4beffd640 1 --2- 172.21.15.107:0/2012902137 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4a80a2ec0 0x7fd4a80a5380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4005d00 tx=0x7fd4b4009550 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:29.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.718+0000 7fd4bcff9640 1 -- 172.21.15.107:0/2012902137 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd4b00bc6a0 con 0x7fd4c00fffa0 2024-05-09T08:33:29.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:29 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1643533759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:33:29.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:29 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3103440983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:29.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:29 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:33:29.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:29 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:33:29.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:29 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:33:29.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:29 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:33:29.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:29 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1643533759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:33:29.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:29 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3103440983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:29.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:29 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:33:29.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:29 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:33:29.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:29 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:33:29.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:29 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:33:29.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:29 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1643533759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:33:29.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:29 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3103440983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:33:29.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:29 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:33:29.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:29 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:33:29.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:29 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:33:29.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:29 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:33:29.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.912+0000 7fd4c5d68640 1 -- 172.21.15.107:0/2012902137 --> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fd4c0100950 con 0x7fd4a80a2ec0 2024-05-09T08:33:29.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.920+0000 7fd4bcff9640 1 -- 172.21.15.107:0/2012902137 <== mgr.15483 v2:172.21.15.107:6800/165281804 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fd4c0100950 con 0x7fd4a80a2ec0 2024-05-09T08:33:29.919 INFO:teuthology.orchestra.run.smithi107.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-09T00:00:00", "created": "2024-05-09T08:33:29", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-05-09T08:33:29.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.922+0000 7fd4c5d68640 1 -- 172.21.15.107:0/2012902137 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4a80a2ec0 msgr2=0x7fd4a80a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.922+0000 7fd4c5d68640 1 --2- 172.21.15.107:0/2012902137 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4a80a2ec0 0x7fd4a80a5380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4005d00 tx=0x7fd4b4009550 comp rx=0 tx=0).stop 2024-05-09T08:33:29.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.922+0000 7fd4c5d68640 1 -- 172.21.15.107:0/2012902137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c00fffa0 msgr2=0x7fd4c0197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:29.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.922+0000 7fd4c5d68640 1 --2- 172.21.15.107:0/2012902137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c00fffa0 0x7fd4c0197fe0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0005940 tx=0x7fd4b0004270 comp rx=0 tx=0).stop 2024-05-09T08:33:29.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.922+0000 7fd4c5d68640 1 -- 172.21.15.107:0/2012902137 shutdown_connections 2024-05-09T08:33:29.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.922+0000 7fd4c5d68640 1 --2- 172.21.15.107:0/2012902137 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4a80a2ec0 0x7fd4a80a5380 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.922+0000 7fd4c5d68640 1 --2- 172.21.15.107:0/2012902137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4c019c8a0 0x7fd4c019cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.922+0000 7fd4c5d68640 1 --2- 172.21.15.107:0/2012902137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4c0106c10 0x7fd4c0198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.922+0000 7fd4c5d68640 1 --2- 172.21.15.107:0/2012902137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c00fffa0 0x7fd4c0197fe0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:29.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.922+0000 7fd4c5d68640 1 -- 172.21.15.107:0/2012902137 >> 172.21.15.107:0/2012902137 conn(0x7fd4c0076c90 msgr2=0x7fd4c00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:29.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.922+0000 7fd4c5d68640 1 -- 172.21.15.107:0/2012902137 shutdown_connections 2024-05-09T08:33:29.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:29.922+0000 7fd4c5d68640 1 -- 172.21.15.107:0/2012902137 wait complete. 2024-05-09T08:33:29.932 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-09T08:33:30.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.035+0000 7f340ea8c640 1 -- 172.21.15.107:0/1850640493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3408101440 msgr2=0x7f340810d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:30.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.035+0000 7f340ea8c640 1 --2- 172.21.15.107:0/1850640493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3408101440 0x7f340810d550 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f3400009ca0 tx=0x7f340002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.035+0000 7f340ea8c640 1 -- 172.21.15.107:0/1850640493 shutdown_connections 2024-05-09T08:33:30.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.035+0000 7f340ea8c640 1 --2- 172.21.15.107:0/1850640493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3408101440 0x7f340810d550 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.035+0000 7f340ea8c640 1 --2- 172.21.15.107:0/1850640493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3408100a80 0x7f3408100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.035+0000 7f340ea8c640 1 --2- 172.21.15.107:0/1850640493 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34081056d0 0x7f3408105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.035+0000 7f340ea8c640 1 -- 172.21.15.107:0/1850640493 >> 172.21.15.107:0/1850640493 conn(0x7f3408076530 msgr2=0x7f3408076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:30.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.035+0000 7f340ea8c640 1 -- 172.21.15.107:0/1850640493 shutdown_connections 2024-05-09T08:33:30.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.036+0000 7f340ea8c640 1 -- 172.21.15.107:0/1850640493 wait complete. 2024-05-09T08:33:30.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.036+0000 7f340ea8c640 1 Processor -- start 2024-05-09T08:33:30.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.036+0000 7f340ea8c640 1 -- start start 2024-05-09T08:33:30.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.036+0000 7f340ea8c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3408100a80 0x7f3408196d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.036+0000 7f340ea8c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3408101440 0x7f3408197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.036+0000 7f340ea8c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34081056d0 0x7f340819b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.036+0000 7f340ea8c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f34081109b0 con 0x7f34081056d0 2024-05-09T08:33:30.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.036+0000 7f340ea8c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3408110830 con 0x7f3408101440 2024-05-09T08:33:30.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.036+0000 7f340ea8c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3408110b30 con 0x7f3408100a80 2024-05-09T08:33:30.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.036+0000 7f340d029640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34081056d0 0x7f340819b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.036+0000 7f340d029640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34081056d0 0x7f340819b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50762/0 (socket says 172.21.15.107:50762) 2024-05-09T08:33:30.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.036+0000 7f340d029640 1 -- 172.21.15.107:0/1007662661 learned_addr learned my addr 172.21.15.107:0/1007662661 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:30.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.036+0000 7f340c828640 1 --2- 172.21.15.107:0/1007662661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3408100a80 0x7f3408196d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.036+0000 7f3407fff640 1 --2- 172.21.15.107:0/1007662661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3408101440 0x7f3408197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.037+0000 7f340d029640 1 -- 172.21.15.107:0/1007662661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3408100a80 msgr2=0x7f3408196d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:30.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.037+0000 7f340d029640 1 --2- 172.21.15.107:0/1007662661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3408100a80 0x7f3408196d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.037+0000 7f340d029640 1 -- 172.21.15.107:0/1007662661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3408101440 msgr2=0x7f3408197290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:30.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.037+0000 7f340d029640 1 --2- 172.21.15.107:0/1007662661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3408101440 0x7f3408197290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.037+0000 7f340d029640 1 -- 172.21.15.107:0/1007662661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340819bd40 con 0x7f34081056d0 2024-05-09T08:33:30.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.037+0000 7f340d029640 1 --2- 172.21.15.107:0/1007662661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34081056d0 0x7f340819b610 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3400009dc0 tx=0x7f3400032ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:30.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.037+0000 7f3405ffb640 1 -- 172.21.15.107:0/1007662661 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3400003bf0 con 0x7f34081056d0 2024-05-09T08:33:30.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.037+0000 7f3405ffb640 1 -- 172.21.15.107:0/1007662661 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3400003d90 con 0x7f34081056d0 2024-05-09T08:33:30.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.037+0000 7f340ea8c640 1 -- 172.21.15.107:0/1007662661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f340819bfd0 con 0x7f34081056d0 2024-05-09T08:33:30.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.037+0000 7f3405ffb640 1 -- 172.21.15.107:0/1007662661 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3400030b80 con 0x7f34081056d0 2024-05-09T08:33:30.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.037+0000 7f340ea8c640 1 -- 172.21.15.107:0/1007662661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34081a38c0 con 0x7f34081056d0 2024-05-09T08:33:30.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.038+0000 7f340ea8c640 1 -- 172.21.15.107:0/1007662661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3408101e30 con 0x7f34081056d0 2024-05-09T08:33:30.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.039+0000 7f3405ffb640 1 -- 172.21.15.107:0/1007662661 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7f3400004040 con 0x7f34081056d0 2024-05-09T08:33:30.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.040+0000 7f3405ffb640 1 --2- 172.21.15.107:0/1007662661 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f33f00a2ec0 0x7f33f00a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:30.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.040+0000 7f340c828640 1 --2- 172.21.15.107:0/1007662661 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f33f00a2ec0 0x7f33f00a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:30.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.041+0000 7f3405ffb640 1 -- 172.21.15.107:0/1007662661 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f34000f0f30 con 0x7f34081056d0 2024-05-09T08:33:30.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.041+0000 7f340c828640 1 --2- 172.21.15.107:0/1007662661 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f33f00a2ec0 0x7f33f00a5380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f33f800bbe0 tx=0x7f33f800b390 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:30.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.045+0000 7f3405ffb640 1 -- 172.21.15.107:0/1007662661 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f34000bd6b0 con 0x7f34081056d0 2024-05-09T08:33:30.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.242+0000 7f340ea8c640 1 -- 172.21.15.107:0/1007662661 --> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f3408061c70 con 0x7f33f00a2ec0 2024-05-09T08:33:30.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.248+0000 7f3405ffb640 1 -- 172.21.15.107:0/1007662661 <== mgr.15483 v2:172.21.15.107:6800/165281804 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f3408061c70 con 0x7f33f00a2ec0 2024-05-09T08:33:30.247 INFO:teuthology.orchestra.run.smithi107.stdout:/ 1m 6m3h 2024-05-09T08:33:30.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.250+0000 7f340ea8c640 1 -- 172.21.15.107:0/1007662661 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f33f00a2ec0 msgr2=0x7f33f00a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:30.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.250+0000 7f340ea8c640 1 --2- 172.21.15.107:0/1007662661 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f33f00a2ec0 0x7f33f00a5380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f33f800bbe0 tx=0x7f33f800b390 comp rx=0 tx=0).stop 2024-05-09T08:33:30.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.250+0000 7f340ea8c640 1 -- 172.21.15.107:0/1007662661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34081056d0 msgr2=0x7f340819b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:30.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.250+0000 7f340ea8c640 1 --2- 172.21.15.107:0/1007662661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34081056d0 0x7f340819b610 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3400009dc0 tx=0x7f3400032ff0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.250+0000 7f340ea8c640 1 -- 172.21.15.107:0/1007662661 shutdown_connections 2024-05-09T08:33:30.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.250+0000 7f340ea8c640 1 --2- 172.21.15.107:0/1007662661 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f33f00a2ec0 0x7f33f00a5380 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.250+0000 7f340ea8c640 1 --2- 172.21.15.107:0/1007662661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34081056d0 0x7f340819b610 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.250+0000 7f340ea8c640 1 --2- 172.21.15.107:0/1007662661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3408101440 0x7f3408197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.250+0000 7f340ea8c640 1 --2- 172.21.15.107:0/1007662661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3408100a80 0x7f3408196d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:30.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.250+0000 7f340ea8c640 1 -- 172.21.15.107:0/1007662661 >> 172.21.15.107:0/1007662661 conn(0x7f3408076530 msgr2=0x7f34080fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:30.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.250+0000 7f340ea8c640 1 -- 172.21.15.107:0/1007662661 shutdown_connections 2024-05-09T08:33:30.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:30.250+0000 7f340ea8c640 1 -- 172.21.15.107:0/1007662661 wait complete. 2024-05-09T08:33:30.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-05-09T08:33:30.289 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-05-09T08:33:30.298 INFO:teuthology.run_tasks:Running task background_exec... 2024-05-09T08:33:30.329 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-05-09T08:33:30.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-05-09T08:33:30.332 INFO:teuthology.run_tasks:Running task workunit... 2024-05-09T08:33:30.382 INFO:tasks.workunit:Pulling workunits from ref be85c202969d25af6e130678e7d73a20e9c4ca11 2024-05-09T08:33:30.383 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-09T08:33:30.383 INFO:tasks.workunit:timeout=3h 2024-05-09T08:33:30.383 INFO:tasks.workunit:cleanup=True 2024-05-09T08:33:30.383 DEBUG:teuthology.orchestra.run.smithi107:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-09T08:33:30.400 INFO:teuthology.orchestra.run.smithi107.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-09T08:33:30.400 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-05-09T08:33:30.400 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 72h/114d Inode: 1 Links: 2 2024-05-09T08:33:30.400 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T08:33:30.400 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:fusefs_t:s0 2024-05-09T08:33:30.400 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-05-09 08:31:21.743757551 +0000 2024-05-09T08:33:30.400 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-05-09 08:31:21.743757551 +0000 2024-05-09T08:33:30.401 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-05-09 08:33:18.932292693 +0000 2024-05-09T08:33:30.401 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2024-05-09T08:33:30.401 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-09T08:33:30.401 DEBUG:teuthology.orchestra.run.smithi107:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-09T08:33:30.474 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout be85c202969d25af6e130678e7d73a20e9c4ca11 2024-05-09T08:33:30.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:30 smithi107 ceph-mon[30122]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-05-09T08:33:30.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:30 smithi107 ceph-mon[30122]: from='client.25267 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:30.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:30 smithi107 ceph-mon[30122]: from='client.15741 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:30.532 INFO:tasks.workunit.client.0.smithi107.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-09T08:33:30.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:30 smithi157 ceph-mon[33614]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-05-09T08:33:30.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:30 smithi157 ceph-mon[33614]: from='client.25267 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:30.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:30 smithi157 ceph-mon[33614]: from='client.15741 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:30.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:30 smithi110 ceph-mon[33622]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-05-09T08:33:30.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:30 smithi110 ceph-mon[33622]: from='client.25267 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:30.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:30 smithi110 ceph-mon[33622]: from='client.15741 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:31 smithi107 ceph-mon[30122]: from='client.15747 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:31 smithi107 ceph-mon[30122]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:31.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:31 smithi157 ceph-mon[33614]: from='client.15747 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:31.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:31 smithi157 ceph-mon[33614]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:31.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:31 smithi110 ceph-mon[33622]: from='client.15747 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:31.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:31 smithi110 ceph-mon[33622]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:33:32.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:32 smithi107 ceph-mon[30122]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 910 B/s wr, 18 op/s 2024-05-09T08:33:32.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:32 smithi157 ceph-mon[33614]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 910 B/s wr, 18 op/s 2024-05-09T08:33:32.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:32 smithi110 ceph-mon[33622]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 910 B/s wr, 18 op/s 2024-05-09T08:33:34.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:34 smithi107 ceph-mon[30122]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 85 KiB/s wr, 52 op/s 2024-05-09T08:33:34.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:34 smithi157 ceph-mon[33614]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 85 KiB/s wr, 52 op/s 2024-05-09T08:33:34.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:34 smithi110 ceph-mon[33622]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 85 KiB/s wr, 52 op/s 2024-05-09T08:33:36.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:36 smithi107 ceph-mon[30122]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-05-09T08:33:36.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:36 smithi157 ceph-mon[33614]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-05-09T08:33:36.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:36 smithi110 ceph-mon[33622]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-05-09T08:33:38.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:38 smithi107 ceph-mon[30122]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 50 op/s 2024-05-09T08:33:38.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:38 smithi157 ceph-mon[33614]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 50 op/s 2024-05-09T08:33:38.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:38 smithi110 ceph-mon[33622]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 50 op/s 2024-05-09T08:33:40.673 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:40 smithi110 ceph-mon[33622]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 50 op/s 2024-05-09T08:33:40.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:40 smithi107 ceph-mon[30122]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 50 op/s 2024-05-09T08:33:40.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:40 smithi157 ceph-mon[33614]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 78 KiB/s wr, 50 op/s 2024-05-09T08:33:41.685 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:41 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:33:41.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:41 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:33:41.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:41 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:33:42.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:42 smithi107 ceph-mon[30122]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-05-09T08:33:42.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:42 smithi157 ceph-mon[33614]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-05-09T08:33:42.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:42 smithi110 ceph-mon[33622]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-05-09T08:33:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.472+0000 7ffaf4328640 1 -- 172.21.15.107:0/4029113719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec06a0a0 msgr2=0x7ffaec06a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.472+0000 7ffaf4328640 1 --2- 172.21.15.107:0/4029113719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec06a0a0 0x7ffaec06a500 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ffadc009900 tx=0x7ffadc02a410 comp rx=0 tx=0).stop 2024-05-09T08:33:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.472+0000 7ffaf4328640 1 -- 172.21.15.107:0/4029113719 shutdown_connections 2024-05-09T08:33:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.472+0000 7ffaf4328640 1 --2- 172.21.15.107:0/4029113719 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaec06aa40 0x7ffaec10e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.472+0000 7ffaf4328640 1 --2- 172.21.15.107:0/4029113719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec06a0a0 0x7ffaec06a500 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.472+0000 7ffaf4328640 1 --2- 172.21.15.107:0/4029113719 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffaec107820 0x7ffaec107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.472+0000 7ffaf4328640 1 -- 172.21.15.107:0/4029113719 >> 172.21.15.107:0/4029113719 conn(0x7ffaec076530 msgr2=0x7ffaec076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.472+0000 7ffaf4328640 1 -- 172.21.15.107:0/4029113719 shutdown_connections 2024-05-09T08:33:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.472+0000 7ffaf4328640 1 -- 172.21.15.107:0/4029113719 wait complete. 2024-05-09T08:33:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.473+0000 7ffaf4328640 1 Processor -- start 2024-05-09T08:33:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.473+0000 7ffaf4328640 1 -- start start 2024-05-09T08:33:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.473+0000 7ffaf4328640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffaec06a0a0 0x7ffaec197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.473+0000 7ffaf4328640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaec06aa40 0x7ffaec198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.473+0000 7ffaf4328640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec107820 0x7ffaec19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.473+0000 7ffaf4328640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec111ac0 con 0x7ffaec107820 2024-05-09T08:33:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.473+0000 7ffaf4328640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec111940 con 0x7ffaec06aa40 2024-05-09T08:33:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.473+0000 7ffaf4328640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec111c40 con 0x7ffaec06a0a0 2024-05-09T08:33:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.473+0000 7ffaf28c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec107820 0x7ffaec19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.473+0000 7ffaf18c3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaec06aa40 0x7ffaec198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.473+0000 7ffaf28c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec107820 0x7ffaec19c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46330/0 (socket says 172.21.15.107:46330) 2024-05-09T08:33:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.473+0000 7ffaf28c5640 1 -- 172.21.15.107:0/1585185893 learned_addr learned my addr 172.21.15.107:0/1585185893 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.473+0000 7ffaf20c4640 1 --2- 172.21.15.107:0/1585185893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffaec06a0a0 0x7ffaec197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.474+0000 7ffaf28c5640 1 -- 172.21.15.107:0/1585185893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffaec06a0a0 msgr2=0x7ffaec197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.474+0000 7ffaf28c5640 1 --2- 172.21.15.107:0/1585185893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffaec06a0a0 0x7ffaec197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.474+0000 7ffaf28c5640 1 -- 172.21.15.107:0/1585185893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaec06aa40 msgr2=0x7ffaec198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.474+0000 7ffaf28c5640 1 --2- 172.21.15.107:0/1585185893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaec06aa40 0x7ffaec198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.474+0000 7ffaf28c5640 1 -- 172.21.15.107:0/1585185893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaec19cf30 con 0x7ffaec107820 2024-05-09T08:33:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.474+0000 7ffaf18c3640 1 --2- 172.21.15.107:0/1585185893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaec06aa40 0x7ffaec198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:33:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.474+0000 7ffaf20c4640 1 --2- 172.21.15.107:0/1585185893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffaec06a0a0 0x7ffaec197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.474+0000 7ffaf28c5640 1 --2- 172.21.15.107:0/1585185893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec107820 0x7ffaec19c800 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ffae400ea60 tx=0x7ffae400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.474+0000 7ffae37fe640 1 -- 172.21.15.107:0/1585185893 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae400ce40 con 0x7ffaec107820 2024-05-09T08:33:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.474+0000 7ffae37fe640 1 -- 172.21.15.107:0/1585185893 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffae4005020 con 0x7ffaec107820 2024-05-09T08:33:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.474+0000 7ffaf4328640 1 -- 172.21.15.107:0/1585185893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaec19d220 con 0x7ffaec107820 2024-05-09T08:33:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.474+0000 7ffae37fe640 1 -- 172.21.15.107:0/1585185893 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae4010670 con 0x7ffaec107820 2024-05-09T08:33:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.474+0000 7ffaf4328640 1 -- 172.21.15.107:0/1585185893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaec1a4ab0 con 0x7ffaec107820 2024-05-09T08:33:43.491 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.477+0000 7ffaf7031740 1 Processor -- start 2024-05-09T08:33:43.491 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.477+0000 7ffae37fe640 1 -- 172.21.15.107:0/1585185893 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7ffae40051d0 con 0x7ffaec107820 2024-05-09T08:33:43.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf7031740 1 -- start start 2024-05-09T08:33:43.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf7031740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6771350 0x55f6b6771730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf7031740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 0x55f6b6780500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf7031740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b6782e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf7031740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f6b66d6340 con 0x55f6b6780a40 2024-05-09T08:33:43.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf7031740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f6b667e290 con 0x55f6b6771350 2024-05-09T08:33:43.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf7031740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f6b66d2300 con 0x55f6b6771c70 2024-05-09T08:33:43.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf18c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b6782e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf18c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b6782e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46334/0 (socket says 172.21.15.107:46334) 2024-05-09T08:33:43.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf18c3640 1 -- 172.21.15.107:0/1420362236 learned_addr learned my addr 172.21.15.107:0/1420362236 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:43.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf18c3640 1 --2- 172.21.15.107:0/1420362236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6771350 0x55f6b6771730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf20c4640 1 --2- 172.21.15.107:0/1420362236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 0x55f6b6780500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffae37fe640 1 --2- 172.21.15.107:0/1585185893 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffad80a2f40 0x7ffad80a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf18c3640 1 -- 172.21.15.107:0/1420362236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 msgr2=0x55f6b6780500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:43.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf18c3640 1 --2- 172.21.15.107:0/1420362236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 0x55f6b6780500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf18c3640 1 -- 172.21.15.107:0/1420362236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6771350 msgr2=0x55f6b6771730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:43.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf18c3640 1 --2- 172.21.15.107:0/1420362236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6771350 0x55f6b6771730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf18c3640 1 -- 172.21.15.107:0/1420362236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6b66a28e0 con 0x55f6b6780a40 2024-05-09T08:33:43.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.478+0000 7ffaf20c4640 1 --2- 172.21.15.107:0/1420362236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 0x55f6b6780500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:43.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.479+0000 7ffaf28c5640 1 --2- 172.21.15.107:0/1585185893 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffad80a2f40 0x7ffad80a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.479+0000 7ffae37fe640 1 -- 172.21.15.107:0/1585185893 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ffae40cff60 con 0x7ffaec107820 2024-05-09T08:33:43.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.479+0000 7ffaf18c3640 1 --2- 172.21.15.107:0/1420362236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b6782e30 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ffadc005c40 tx=0x7ffadc005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ee1c04512cc90f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:43.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.479+0000 7ffae0bfe640 1 -- 172.21.15.107:0/1420362236 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc035070 con 0x55f6b6780a40 2024-05-09T08:33:43.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.479+0000 7ffae0bfe640 1 -- 172.21.15.107:0/1420362236 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffadc045070 con 0x55f6b6780a40 2024-05-09T08:33:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.479+0000 7ffae0bfe640 1 -- 172.21.15.107:0/1420362236 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc03d450 con 0x55f6b6780a40 2024-05-09T08:33:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 -- 172.21.15.107:0/1420362236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 msgr2=0x55f6b6782e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf28c5640 1 --2- 172.21.15.107:0/1585185893 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffad80a2f40 0x7ffad80a5400 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffae40cd230 tx=0x7ffae400ef90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 --2- 172.21.15.107:0/1420362236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b6782e30 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ffadc005c40 tx=0x7ffadc005780 comp rx=0 tx=0).stop 2024-05-09T08:33:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 -- 172.21.15.107:0/1420362236 shutdown_connections 2024-05-09T08:33:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 --2- 172.21.15.107:0/1420362236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b6782e30 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 --2- 172.21.15.107:0/1420362236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 0x55f6b6780500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 --2- 172.21.15.107:0/1420362236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6771350 0x55f6b6771730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 -- 172.21.15.107:0/1420362236 >> 172.21.15.107:0/1420362236 conn(0x55f6b676d960 msgr2=0x55f6b676dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 -- 172.21.15.107:0/1420362236 shutdown_connections 2024-05-09T08:33:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 -- 172.21.15.107:0/1420362236 wait complete. 2024-05-09T08:33:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 -- start start 2024-05-09T08:33:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 Processor -- start 2024-05-09T08:33:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6771350 0x55f6b677d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 0x55f6b677de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b677e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f6b6783610 con 0x55f6b6780a40 2024-05-09T08:33:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f6b6691790 con 0x55f6b6771350 2024-05-09T08:33:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f6b6783790 con 0x55f6b6771c70 2024-05-09T08:33:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.480+0000 7ffaf7031740 10 client.0 _ll_register_callbacks cb 0x55f6b66cca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffaf18c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b677e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffaf18c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b677e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46336/0 (socket says 172.21.15.107:46336) 2024-05-09T08:33:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffaf18c3640 1 -- 172.21.15.107:0/3601760801 learned_addr learned my addr 172.21.15.107:0/3601760801 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffaf20c4640 1 --2- 172.21.15.107:0/3601760801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 0x55f6b677de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffaf18c3640 1 --2- 172.21.15.107:0/3601760801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6771350 0x55f6b677d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffaf18c3640 1 -- 172.21.15.107:0/3601760801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 msgr2=0x55f6b677de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffaf18c3640 1 --2- 172.21.15.107:0/3601760801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 0x55f6b677de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffaf18c3640 1 -- 172.21.15.107:0/3601760801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6771350 msgr2=0x55f6b677d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffaf18c3640 1 --2- 172.21.15.107:0/3601760801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6771350 0x55f6b677d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffaf18c3640 1 -- 172.21.15.107:0/3601760801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6b63ea210 con 0x55f6b6780a40 2024-05-09T08:33:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffaf18c3640 1 --2- 172.21.15.107:0/3601760801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b677e3b0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ffadc000c00 tx=0x7ffadc003bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b640aaa8c60897fd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffad5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffad5ffb640 1 -- 172.21.15.107:0/3601760801 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc035030 con 0x55f6b6780a40 2024-05-09T08:33:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffad5ffb640 1 -- 172.21.15.107:0/3601760801 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ffadc003f50 con 0x55f6b6780a40 2024-05-09T08:33:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffaf7031740 1 -- 172.21.15.107:0/3601760801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffadc045070 con 0x55f6b6780a40 2024-05-09T08:33:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffad5ffb640 1 -- 172.21.15.107:0/3601760801 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ffadc05c050 con 0x55f6b6780a40 2024-05-09T08:33:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffaf7031740 10 client.15765 fetch_fsmap learned FSMap version 25 2024-05-09T08:33:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.481+0000 7ffaf7031740 1 -- 172.21.15.107:0/3601760801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f6b64b6620 con 0x55f6b6780a40 2024-05-09T08:33:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.482+0000 7ffad5ffb640 1 -- 172.21.15.107:0/3601760801 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffadc045070 con 0x55f6b6780a40 2024-05-09T08:33:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.482+0000 7ffad5ffb640 1 -- 172.21.15.107:0/3601760801 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ffadc03da90 con 0x55f6b6780a40 2024-05-09T08:33:43.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.482+0000 7ffaf7031740 10 client.15765 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:33:43.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.482+0000 7ffaf7031740 10 client.15765 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:33:43.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.482+0000 7ffaf7031740 20 client.15765 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:43.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.482+0000 7ffaf7031740 1 --2- 172.21.15.107:0/3601760801 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f6b6790590 0x55f6b6792980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.482+0000 7ffaf7031740 4 client.15765 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:33:43.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.482+0000 7ffaf7031740 1 -- 172.21.15.107:0/3601760801 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6b66e3a70 con 0x55f6b6790590 2024-05-09T08:33:43.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.482+0000 7ffaf20c4640 1 --2- 172.21.15.107:0/3601760801 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f6b6790590 0x55f6b6792980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.483+0000 7ffaf20c4640 1 --2- 172.21.15.107:0/3601760801 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f6b6790590 0x55f6b6792980 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6a16dbf65d58061 server_cookie=d080633af7e7466a in_seq=0 out_seq=0 2024-05-09T08:33:43.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.483+0000 7ffad5ffb640 10 client.15765 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:43.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.483+0000 7ffad5ffb640 1 -- 172.21.15.107:0/3601760801 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f6b66e3a70 con 0x55f6b6790590 2024-05-09T08:33:43.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.483+0000 7ffad5ffb640 10 client.15765 handle_command_reply: tid=0 2024-05-09T08:33:43.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.483+0000 7ffaf7031740 1 client.15765 shutdown 2024-05-09T08:33:43.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.483+0000 7ffaf7031740 1 -- 172.21.15.107:0/3601760801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 msgr2=0x55f6b677e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:43.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.483+0000 7ffaf7031740 1 --2- 172.21.15.107:0/3601760801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b677e3b0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ffadc000c00 tx=0x7ffadc003bb0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.483+0000 7ffaf7031740 1 -- 172.21.15.107:0/3601760801 shutdown_connections 2024-05-09T08:33:43.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.483+0000 7ffaf7031740 1 --2- 172.21.15.107:0/3601760801 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f6b6790590 0x55f6b6792980 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.483+0000 7ffaf7031740 1 --2- 172.21.15.107:0/3601760801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b677e3b0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.483+0000 7ffaf7031740 1 --2- 172.21.15.107:0/3601760801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 0x55f6b677de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.483+0000 7ffad5ffb640 0 client.15765 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:43.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.483+0000 7ffaf7031740 1 --2- 172.21.15.107:0/3601760801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6771350 0x55f6b677d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.484+0000 7ffaf7031740 1 -- 172.21.15.107:0/3601760801 >> 172.21.15.107:0/3601760801 conn(0x55f6b676d960 msgr2=0x55f6b67737d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:43.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.484+0000 7ffaf7031740 1 -- 172.21.15.107:0/3601760801 shutdown_connections 2024-05-09T08:33:43.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.484+0000 7ffaf7031740 1 -- 172.21.15.107:0/3601760801 wait complete. 2024-05-09T08:33:43.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.484+0000 7ffaf7031740 20 client.15765 trim_cache size 0 max 16384 2024-05-09T08:33:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.498+0000 7ffaf7031740 1 Processor -- start 2024-05-09T08:33:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.498+0000 7ffaf7031740 1 -- start start 2024-05-09T08:33:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf7031740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771350 0x55f6b6780060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf7031740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6771c70 0x55f6b677ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf7031740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b677b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf7031740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec1af000 con 0x55f6b6780a40 2024-05-09T08:33:43.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf7031740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec1ac8d0 con 0x55f6b6771c70 2024-05-09T08:33:43.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf7031740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec1aa720 con 0x55f6b6771350 2024-05-09T08:33:43.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf18c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b677b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf18c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b677b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46340/0 (socket says 172.21.15.107:46340) 2024-05-09T08:33:43.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf18c3640 1 -- 172.21.15.107:0/2046536316 learned_addr learned my addr 172.21.15.107:0/2046536316 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:43.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf20c4640 1 --2- 172.21.15.107:0/2046536316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6771c70 0x55f6b677ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf18c3640 1 -- 172.21.15.107:0/2046536316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771350 msgr2=0x55f6b6780060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf18c3640 1 --2- 172.21.15.107:0/2046536316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771350 0x55f6b6780060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf18c3640 1 -- 172.21.15.107:0/2046536316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6771c70 msgr2=0x55f6b677ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf18c3640 1 --2- 172.21.15.107:0/2046536316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6771c70 0x55f6b677ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf18c3640 1 -- 172.21.15.107:0/2046536316 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6b6771aa0 con 0x55f6b6780a40 2024-05-09T08:33:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf20c4640 1 --2- 172.21.15.107:0/2046536316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6771c70 0x55f6b677ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffaf18c3640 1 --2- 172.21.15.107:0/2046536316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b677b0d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ffadc005c40 tx=0x7ffadc005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d2910883158fa2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffad7fff640 1 -- 172.21.15.107:0/2046536316 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc035070 con 0x55f6b6780a40 2024-05-09T08:33:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffad7fff640 1 -- 172.21.15.107:0/2046536316 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffadc045040 con 0x55f6b6780a40 2024-05-09T08:33:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.499+0000 7ffad7fff640 1 -- 172.21.15.107:0/2046536316 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc03d990 con 0x55f6b6780a40 2024-05-09T08:33:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.500+0000 7ffaf7031740 1 -- 172.21.15.107:0/2046536316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 msgr2=0x55f6b677b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.500+0000 7ffaf7031740 1 --2- 172.21.15.107:0/2046536316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b677b0d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ffadc005c40 tx=0x7ffadc005fd0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.500+0000 7ffaf7031740 1 -- 172.21.15.107:0/2046536316 shutdown_connections 2024-05-09T08:33:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.500+0000 7ffaf7031740 1 --2- 172.21.15.107:0/2046536316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6780a40 0x55f6b677b0d0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.500+0000 7ffaf7031740 1 --2- 172.21.15.107:0/2046536316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6771c70 0x55f6b677ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.500+0000 7ffaf7031740 1 --2- 172.21.15.107:0/2046536316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771350 0x55f6b6780060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.500+0000 7ffaf7031740 1 -- 172.21.15.107:0/2046536316 >> 172.21.15.107:0/2046536316 conn(0x55f6b676d960 msgr2=0x55f6b676dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.500+0000 7ffaf7031740 1 -- 172.21.15.107:0/2046536316 shutdown_connections 2024-05-09T08:33:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.500+0000 7ffaf7031740 1 -- 172.21.15.107:0/2046536316 wait complete. 2024-05-09T08:33:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.500+0000 7ffaf7031740 1 -- start start 2024-05-09T08:33:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.500+0000 7ffaf7031740 1 Processor -- start 2024-05-09T08:33:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf7031740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6771350 0x55f6b6795bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf7031740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 0x55f6b6792fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf7031740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6780a40 0x55f6b67934e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf7031740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec1aad90 con 0x55f6b6771350 2024-05-09T08:33:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf7031740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec1ac200 con 0x55f6b6780a40 2024-05-09T08:33:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf7031740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec1ab460 con 0x55f6b6771c70 2024-05-09T08:33:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf7031740 10 client.0 _ll_register_callbacks cb 0x7ffadc031650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf18c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6771350 0x55f6b6795bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf18c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6771350 0x55f6b6795bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46356/0 (socket says 172.21.15.107:46356) 2024-05-09T08:33:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf18c3640 1 -- 172.21.15.107:0/313058193 learned_addr learned my addr 172.21.15.107:0/313058193 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf20c4640 1 --2- 172.21.15.107:0/313058193 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 0x55f6b6792fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf18c3640 1 --2- 172.21.15.107:0/313058193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6780a40 0x55f6b67934e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf18c3640 1 -- 172.21.15.107:0/313058193 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 msgr2=0x55f6b6792fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf18c3640 1 --2- 172.21.15.107:0/313058193 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 0x55f6b6792fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf18c3640 1 -- 172.21.15.107:0/313058193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6780a40 msgr2=0x55f6b67934e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf18c3640 1 --2- 172.21.15.107:0/313058193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6780a40 0x55f6b67934e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf18c3640 1 -- 172.21.15.107:0/313058193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6b6793b20 con 0x55f6b6771350 2024-05-09T08:33:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf20c4640 1 --2- 172.21.15.107:0/313058193 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 0x55f6b6792fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf18c3640 1 --2- 172.21.15.107:0/313058193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6771350 0x55f6b6795bc0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ffadc004720 tx=0x7ffadc03ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfef73f9239356c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffad5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffad5ffb640 1 -- 172.21.15.107:0/313058193 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc035040 con 0x55f6b6771350 2024-05-09T08:33:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffad5ffb640 1 -- 172.21.15.107:0/313058193 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ffadc03d730 con 0x55f6b6771350 2024-05-09T08:33:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.501+0000 7ffaf7031740 1 -- 172.21.15.107:0/313058193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffadc045040 con 0x55f6b6771350 2024-05-09T08:33:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.502+0000 7ffad5ffb640 1 -- 172.21.15.107:0/313058193 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ffadc009520 con 0x55f6b6771350 2024-05-09T08:33:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.502+0000 7ffaf7031740 10 client.15771 fetch_fsmap learned FSMap version 25 2024-05-09T08:33:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.502+0000 7ffaf7031740 1 -- 172.21.15.107:0/313058193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f6b67f6540 con 0x55f6b6771350 2024-05-09T08:33:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.502+0000 7ffad5ffb640 1 -- 172.21.15.107:0/313058193 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffadc045040 con 0x55f6b6771350 2024-05-09T08:33:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.502+0000 7ffad5ffb640 1 -- 172.21.15.107:0/313058193 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ffadc06c060 con 0x55f6b6771350 2024-05-09T08:33:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.502+0000 7ffaf7031740 10 client.15771 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:33:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.502+0000 7ffaf7031740 10 client.15771 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:33:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.502+0000 7ffaf7031740 20 client.15771 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.502+0000 7ffaf7031740 1 --2- 172.21.15.107:0/313058193 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f6b6790590 0x55f6b67f8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.502+0000 7ffaf7031740 4 client.15771 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:33:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.502+0000 7ffaf7031740 1 -- 172.21.15.107:0/313058193 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f6b66dddc0 con 0x55f6b6790590 2024-05-09T08:33:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.502+0000 7ffaf20c4640 1 --2- 172.21.15.107:0/313058193 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f6b6790590 0x55f6b67f8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.503+0000 7ffaf20c4640 1 --2- 172.21.15.107:0/313058193 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f6b6790590 0x55f6b67f8970 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55550871ade82515 server_cookie=cfce9cc64bf64a87 in_seq=0 out_seq=0 2024-05-09T08:33:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.503+0000 7ffad5ffb640 10 client.15771 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.514+0000 7ffad5ffb640 1 -- 172.21.15.107:0/313058193 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f6b66dddc0 con 0x55f6b6790590 2024-05-09T08:33:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.514+0000 7ffad5ffb640 10 client.15771 handle_command_reply: tid=0 2024-05-09T08:33:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf7031740 1 client.15771 shutdown 2024-05-09T08:33:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf7031740 1 -- 172.21.15.107:0/313058193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6771350 msgr2=0x55f6b6795bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf7031740 1 --2- 172.21.15.107:0/313058193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6771350 0x55f6b6795bc0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ffadc004720 tx=0x7ffadc03ffd0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf7031740 1 -- 172.21.15.107:0/313058193 shutdown_connections 2024-05-09T08:33:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf7031740 1 --2- 172.21.15.107:0/313058193 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f6b6790590 0x55f6b67f8970 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf7031740 1 --2- 172.21.15.107:0/313058193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6b6780a40 0x55f6b67934e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffad5ffb640 0 client.15771 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf7031740 1 --2- 172.21.15.107:0/313058193 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6b6771c70 0x55f6b6792fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf7031740 1 --2- 172.21.15.107:0/313058193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6b6771350 0x55f6b6795bc0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf7031740 1 -- 172.21.15.107:0/313058193 >> 172.21.15.107:0/313058193 conn(0x55f6b676d960 msgr2=0x55f6b676c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf7031740 1 -- 172.21.15.107:0/313058193 shutdown_connections 2024-05-09T08:33:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf7031740 1 -- 172.21.15.107:0/313058193 wait complete. 2024-05-09T08:33:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf7031740 20 client.15771 trim_cache size 0 max 16384 2024-05-09T08:33:43.517 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:33:43.517 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:33:43.517 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:33:43.517 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:33:43.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf4328640 1 -- 172.21.15.107:0/1585185893 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffad80a2f40 msgr2=0x7ffad80a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:43.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf4328640 1 --2- 172.21.15.107:0/1585185893 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffad80a2f40 0x7ffad80a5400 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffae40cd230 tx=0x7ffae400ef90 comp rx=0 tx=0).stop 2024-05-09T08:33:43.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf4328640 1 -- 172.21.15.107:0/1585185893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec107820 msgr2=0x7ffaec19c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:43.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf4328640 1 --2- 172.21.15.107:0/1585185893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec107820 0x7ffaec19c800 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ffae400ea60 tx=0x7ffae400ef20 comp rx=0 tx=0).stop 2024-05-09T08:33:43.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf4328640 1 -- 172.21.15.107:0/1585185893 shutdown_connections 2024-05-09T08:33:43.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf4328640 1 --2- 172.21.15.107:0/1585185893 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffad80a2f40 0x7ffad80a5400 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf4328640 1 --2- 172.21.15.107:0/1585185893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffaec107820 0x7ffaec19c800 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf4328640 1 --2- 172.21.15.107:0/1585185893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaec06aa40 0x7ffaec198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.515+0000 7ffaf4328640 1 --2- 172.21.15.107:0/1585185893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffaec06a0a0 0x7ffaec197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:43.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.516+0000 7ffaf4328640 1 -- 172.21.15.107:0/1585185893 >> 172.21.15.107:0/1585185893 conn(0x7ffaec076530 msgr2=0x7ffaec0ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:43.519 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.516+0000 7ffaf4328640 1 -- 172.21.15.107:0/1585185893 shutdown_connections 2024-05-09T08:33:43.519 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:43.516+0000 7ffaf4328640 1 -- 172.21.15.107:0/1585185893 wait complete. 2024-05-09T08:33:44.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:44 smithi107 ceph-mon[30122]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-05-09T08:33:44.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:44 smithi157 ceph-mon[33614]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-05-09T08:33:44.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:44 smithi110 ceph-mon[33622]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-05-09T08:33:46.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:46 smithi107 ceph-mon[30122]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-09T08:33:46.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:46 smithi157 ceph-mon[33614]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-09T08:33:46.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:46 smithi110 ceph-mon[33622]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-09T08:33:48.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:48 smithi107 ceph-mon[30122]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-05-09T08:33:48.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:48 smithi157 ceph-mon[33614]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-05-09T08:33:48.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:48 smithi110 ceph-mon[33622]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-05-09T08:33:50.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:50 smithi107 ceph-mon[30122]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-05-09T08:33:50.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:50 smithi157 ceph-mon[33614]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-05-09T08:33:50.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:50 smithi110 ceph-mon[33622]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-05-09T08:33:52.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:52 smithi107 ceph-mon[30122]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-05-09T08:33:52.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:52 smithi157 ceph-mon[33614]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-05-09T08:33:52.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:52 smithi110 ceph-mon[33622]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-05-09T08:33:54.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:54 smithi107 ceph-mon[30122]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-05-09T08:33:54.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:54 smithi157 ceph-mon[33614]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-05-09T08:33:54.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:54 smithi110 ceph-mon[33622]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-05-09T08:33:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.615+0000 7fb3ad638640 1 -- 172.21.15.107:0/48764477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3a80fffa0 msgr2=0x7fb3a8100380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.615+0000 7fb3ad638640 1 --2- 172.21.15.107:0/48764477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3a80fffa0 0x7fb3a8100380 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb398009ca0 tx=0x7fb39802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.615+0000 7fb3ad638640 1 -- 172.21.15.107:0/48764477 shutdown_connections 2024-05-09T08:33:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.615+0000 7fb3ad638640 1 --2- 172.21.15.107:0/48764477 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3a8106b60 0x7fb3a810e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.615+0000 7fb3ad638640 1 --2- 172.21.15.107:0/48764477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3a8100950 0x7fb3a8106480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.615+0000 7fb3ad638640 1 --2- 172.21.15.107:0/48764477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3a80fffa0 0x7fb3a8100380 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.615+0000 7fb3ad638640 1 -- 172.21.15.107:0/48764477 >> 172.21.15.107:0/48764477 conn(0x7fb3a8076cc0 msgr2=0x7fb3a80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.615+0000 7fb3ad638640 1 -- 172.21.15.107:0/48764477 shutdown_connections 2024-05-09T08:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.615+0000 7fb3ad638640 1 -- 172.21.15.107:0/48764477 wait complete. 2024-05-09T08:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.616+0000 7fb3ad638640 1 Processor -- start 2024-05-09T08:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.616+0000 7fb3ad638640 1 -- start start 2024-05-09T08:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.616+0000 7fb3ad638640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3a80fffa0 0x7fb3a8197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.616+0000 7fb3ad638640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3a8106b60 0x7fb3a81984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.616+0000 7fb3ad638640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3a819c840 0x7fb3a819ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.616+0000 7fb3ad638640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a8111ac0 con 0x7fb3a819c840 2024-05-09T08:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.616+0000 7fb3ad638640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a8111940 con 0x7fb3a80fffa0 2024-05-09T08:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.616+0000 7fb3ad638640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a8111c40 con 0x7fb3a8106b60 2024-05-09T08:33:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb3a77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3a819c840 0x7fb3a819ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb3a77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3a819c840 0x7fb3a819ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33684/0 (socket says 172.21.15.107:33684) 2024-05-09T08:33:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb3a77fe640 1 -- 172.21.15.107:0/3960060018 learned_addr learned my addr 172.21.15.107:0/3960060018 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb3a67fc640 1 --2- 172.21.15.107:0/3960060018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3a8106b60 0x7fb3a81984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb3a6ffd640 1 --2- 172.21.15.107:0/3960060018 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3a80fffa0 0x7fb3a8197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb3a77fe640 1 -- 172.21.15.107:0/3960060018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3a8106b60 msgr2=0x7fb3a81984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb3a77fe640 1 --2- 172.21.15.107:0/3960060018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3a8106b60 0x7fb3a81984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb3a77fe640 1 -- 172.21.15.107:0/3960060018 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3a80fffa0 msgr2=0x7fb3a8197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb3a77fe640 1 --2- 172.21.15.107:0/3960060018 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3a80fffa0 0x7fb3a8197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb3a77fe640 1 -- 172.21.15.107:0/3960060018 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a819d330 con 0x7fb3a819c840 2024-05-09T08:33:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb3a67fc640 1 --2- 172.21.15.107:0/3960060018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3a8106b60 0x7fb3a81984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb3a6ffd640 1 --2- 172.21.15.107:0/3960060018 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3a80fffa0 0x7fb3a8197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb3a77fe640 1 --2- 172.21.15.107:0/3960060018 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3a819c840 0x7fb3a819ccf0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb3a000d750 tx=0x7fb3a000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb38ffff640 1 -- 172.21.15.107:0/3960060018 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0017070 con 0x7fb3a819c840 2024-05-09T08:33:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb38ffff640 1 -- 172.21.15.107:0/3960060018 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb3a00054d0 con 0x7fb3a819c840 2024-05-09T08:33:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb3ad638640 1 -- 172.21.15.107:0/3960060018 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3a819d620 con 0x7fb3a819c840 2024-05-09T08:33:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb38ffff640 1 -- 172.21.15.107:0/3960060018 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0004c60 con 0x7fb3a819c840 2024-05-09T08:33:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.617+0000 7fb3ad638640 1 -- 172.21.15.107:0/3960060018 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3a81a4dc0 con 0x7fb3a819c840 2024-05-09T08:33:56.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.619+0000 7fb38ffff640 1 -- 172.21.15.107:0/3960060018 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7fb3a0004e00 con 0x7fb3a819c840 2024-05-09T08:33:56.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.620+0000 7fb3b0428740 1 Processor -- start 2024-05-09T08:33:56.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.620+0000 7fb3b0428740 1 -- start start 2024-05-09T08:33:56.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.620+0000 7fb3b0428740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089450 0x55d785089830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.620+0000 7fb3b0428740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785089d70 0x55d7850940f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.620+0000 7fb3b0428740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785094630 0x55d785096a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.620+0000 7fb3b0428740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d784fee340 con 0x55d785089450 2024-05-09T08:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.620+0000 7fb3b0428740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d784f96290 con 0x55d785089d70 2024-05-09T08:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.620+0000 7fb3b0428740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d784fea300 con 0x55d785094630 2024-05-09T08:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.620+0000 7fb3a67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089450 0x55d785089830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.620+0000 7fb3a67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089450 0x55d785089830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33688/0 (socket says 172.21.15.107:33688) 2024-05-09T08:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.620+0000 7fb3a67fc640 1 -- 172.21.15.107:0/625310161 learned_addr learned my addr 172.21.15.107:0/625310161 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.620+0000 7fb38ffff640 1 --2- 172.21.15.107:0/3960060018 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb3900a2ec0 0x7fb3900a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.620+0000 7fb3a6ffd640 1 --2- 172.21.15.107:0/625310161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785089d70 0x55d7850940f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.621+0000 7fb3a67fc640 1 --2- 172.21.15.107:0/625310161 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785094630 0x55d785096a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.621+0000 7fb3a77fe640 1 --2- 172.21.15.107:0/3960060018 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb3900a2ec0 0x7fb3900a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.621+0000 7fb3a67fc640 1 -- 172.21.15.107:0/625310161 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785094630 msgr2=0x55d785096a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.621+0000 7fb3a67fc640 1 --2- 172.21.15.107:0/625310161 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785094630 0x55d785096a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.621+0000 7fb3a67fc640 1 -- 172.21.15.107:0/625310161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785089d70 msgr2=0x55d7850940f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.621+0000 7fb3a67fc640 1 --2- 172.21.15.107:0/625310161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785089d70 0x55d7850940f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.621+0000 7fb3a67fc640 1 -- 172.21.15.107:0/625310161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d784fba8e0 con 0x55d785089450 2024-05-09T08:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.621+0000 7fb38ffff640 1 -- 172.21.15.107:0/3960060018 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb3a00cf470 con 0x7fb3a819c840 2024-05-09T08:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.621+0000 7fb3a67fc640 1 --2- 172.21.15.107:0/625310161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089450 0x55d785089830 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb39c005e00 tx=0x7fb39c00b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43577ab2bb141ac5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.621+0000 7fb3a77fe640 1 --2- 172.21.15.107:0/3960060018 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb3900a2ec0 0x7fb3900a5380 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00cc720 tx=0x7fb3a006e430 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.621+0000 7fb38d7fa640 1 -- 172.21.15.107:0/625310161 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c019070 con 0x55d785089450 2024-05-09T08:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.621+0000 7fb38d7fa640 1 -- 172.21.15.107:0/625310161 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb39c004220 con 0x55d785089450 2024-05-09T08:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.621+0000 7fb38d7fa640 1 -- 172.21.15.107:0/625310161 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c011840 con 0x55d785089450 2024-05-09T08:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.622+0000 7fb3b0428740 1 -- 172.21.15.107:0/625310161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089450 msgr2=0x55d785089830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.622+0000 7fb3b0428740 1 --2- 172.21.15.107:0/625310161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089450 0x55d785089830 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb39c005e00 tx=0x7fb39c00b690 comp rx=0 tx=0).stop 2024-05-09T08:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.622+0000 7fb3b0428740 1 -- 172.21.15.107:0/625310161 shutdown_connections 2024-05-09T08:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.622+0000 7fb3b0428740 1 --2- 172.21.15.107:0/625310161 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785094630 0x55d785096a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.622+0000 7fb3b0428740 1 --2- 172.21.15.107:0/625310161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785089d70 0x55d7850940f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.622+0000 7fb3b0428740 1 --2- 172.21.15.107:0/625310161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089450 0x55d785089830 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.622+0000 7fb3b0428740 1 -- 172.21.15.107:0/625310161 >> 172.21.15.107:0/625310161 conn(0x55d785085960 msgr2=0x55d785085d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.622+0000 7fb3b0428740 1 -- 172.21.15.107:0/625310161 shutdown_connections 2024-05-09T08:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.622+0000 7fb3b0428740 1 -- 172.21.15.107:0/625310161 wait complete. 2024-05-09T08:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.622+0000 7fb3b0428740 1 -- start start 2024-05-09T08:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.622+0000 7fb3b0428740 1 Processor -- start 2024-05-09T08:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3b0428740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089450 0x55d785091540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3b0428740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785089d70 0x55d785091a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3b0428740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785094630 0x55d785091fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3b0428740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d785097260 con 0x55d785089450 2024-05-09T08:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3b0428740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d784fa9790 con 0x55d785089d70 2024-05-09T08:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3b0428740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d7850973e0 con 0x55d785094630 2024-05-09T08:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3b0428740 10 client.0 _ll_register_callbacks cb 0x55d784fe4a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3a67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089450 0x55d785091540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3a67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089450 0x55d785091540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33704/0 (socket says 172.21.15.107:33704) 2024-05-09T08:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3a67fc640 1 -- 172.21.15.107:0/2353123667 learned_addr learned my addr 172.21.15.107:0/2353123667 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3a67fc640 1 -- 172.21.15.107:0/2353123667 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785094630 msgr2=0x55d785091fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3a67fc640 1 --2- 172.21.15.107:0/2353123667 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785094630 0x55d785091fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3a67fc640 1 -- 172.21.15.107:0/2353123667 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785089d70 msgr2=0x55d785091a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3a6ffd640 1 --2- 172.21.15.107:0/2353123667 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785089d70 0x55d785091a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3a67fc640 1 --2- 172.21.15.107:0/2353123667 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785089d70 0x55d785091a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3a67fc640 1 -- 172.21.15.107:0/2353123667 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d784d02210 con 0x55d785089450 2024-05-09T08:33:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3a67fc640 1 --2- 172.21.15.107:0/2353123667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089450 0x55d785091540 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb39c004600 tx=0x7fb39c005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47cf500d57e51b03 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb38aff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb38aff5640 1 -- 172.21.15.107:0/2353123667 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c019050 con 0x55d785089450 2024-05-09T08:33:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3b0428740 1 -- 172.21.15.107:0/2353123667 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb39c004220 con 0x55d785089450 2024-05-09T08:33:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb38aff5640 1 -- 172.21.15.107:0/2353123667 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb39c005450 con 0x55d785089450 2024-05-09T08:33:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb38aff5640 1 -- 172.21.15.107:0/2353123667 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb39c031050 con 0x55d785089450 2024-05-09T08:33:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3b0428740 10 client.15783 fetch_fsmap learned FSMap version 25 2024-05-09T08:33:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.623+0000 7fb3b0428740 1 -- 172.21.15.107:0/2353123667 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d784dce620 con 0x55d785089450 2024-05-09T08:33:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.624+0000 7fb38aff5640 1 -- 172.21.15.107:0/2353123667 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb39c004220 con 0x55d785089450 2024-05-09T08:33:56.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.624+0000 7fb38aff5640 1 -- 172.21.15.107:0/2353123667 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb39c004920 con 0x55d785089450 2024-05-09T08:33:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.624+0000 7fb3b0428740 10 client.15783 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:33:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.624+0000 7fb3b0428740 10 client.15783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:33:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.624+0000 7fb3b0428740 20 client.15783 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.624+0000 7fb3b0428740 1 --2- 172.21.15.107:0/2353123667 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d7850a4350 0x55d7850a6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.624+0000 7fb3b0428740 4 client.15783 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:33:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.624+0000 7fb3b0428740 1 -- 172.21.15.107:0/2353123667 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d784ffba70 con 0x55d7850a4350 2024-05-09T08:33:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.624+0000 7fb3a6ffd640 1 --2- 172.21.15.107:0/2353123667 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d7850a4350 0x55d7850a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.625+0000 7fb3a6ffd640 1 --2- 172.21.15.107:0/2353123667 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d7850a4350 0x55d7850a6740 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d6f6861449fd998 server_cookie=79f94a17d2fb784c in_seq=0 out_seq=0 2024-05-09T08:33:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.625+0000 7fb38aff5640 10 client.15783 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.625+0000 7fb38aff5640 1 -- 172.21.15.107:0/2353123667 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d784ffba70 con 0x55d7850a4350 2024-05-09T08:33:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.625+0000 7fb38aff5640 10 client.15783 handle_command_reply: tid=0 2024-05-09T08:33:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.625+0000 7fb3b0428740 1 client.15783 shutdown 2024-05-09T08:33:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.626+0000 7fb3b0428740 1 -- 172.21.15.107:0/2353123667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089450 msgr2=0x55d785091540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.626+0000 7fb3b0428740 1 --2- 172.21.15.107:0/2353123667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089450 0x55d785091540 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb39c004600 tx=0x7fb39c005970 comp rx=0 tx=0).stop 2024-05-09T08:33:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.626+0000 7fb3b0428740 1 -- 172.21.15.107:0/2353123667 shutdown_connections 2024-05-09T08:33:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.626+0000 7fb3b0428740 1 --2- 172.21.15.107:0/2353123667 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d7850a4350 0x55d7850a6740 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.626+0000 7fb3b0428740 1 --2- 172.21.15.107:0/2353123667 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785094630 0x55d785091fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.626+0000 7fb3b0428740 1 --2- 172.21.15.107:0/2353123667 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785089d70 0x55d785091a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.626+0000 7fb3b0428740 1 --2- 172.21.15.107:0/2353123667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089450 0x55d785091540 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.626+0000 7fb38aff5640 0 client.15783 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.626+0000 7fb3b0428740 1 -- 172.21.15.107:0/2353123667 >> 172.21.15.107:0/2353123667 conn(0x55d785085960 msgr2=0x55d78508b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.626+0000 7fb3b0428740 1 -- 172.21.15.107:0/2353123667 shutdown_connections 2024-05-09T08:33:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.626+0000 7fb3b0428740 1 -- 172.21.15.107:0/2353123667 wait complete. 2024-05-09T08:33:56.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.626+0000 7fb3b0428740 20 client.15783 trim_cache size 0 max 16384 2024-05-09T08:33:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.640+0000 7fb3b0428740 1 Processor -- start 2024-05-09T08:33:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.640+0000 7fb3b0428740 1 -- start start 2024-05-09T08:33:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3b0428740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785089450 0x55d785093c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3b0428740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089d70 0x55d7850a0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3b0428740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785094630 0x55d7850906e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3b0428740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a81b1340 con 0x55d785089d70 2024-05-09T08:33:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3b0428740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a81aec10 con 0x55d785094630 2024-05-09T08:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3b0428740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a81aca60 con 0x55d785089450 2024-05-09T08:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3a6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089d70 0x55d7850a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3a6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089d70 0x55d7850a0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33706/0 (socket says 172.21.15.107:33706) 2024-05-09T08:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3a6ffd640 1 -- 172.21.15.107:0/4076013963 learned_addr learned my addr 172.21.15.107:0/4076013963 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3a67fc640 1 --2- 172.21.15.107:0/4076013963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785089450 0x55d785093c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3a6ffd640 1 -- 172.21.15.107:0/4076013963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785089450 msgr2=0x55d785093c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3a6ffd640 1 --2- 172.21.15.107:0/4076013963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785089450 0x55d785093c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3a6ffd640 1 -- 172.21.15.107:0/4076013963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785094630 msgr2=0x55d7850906e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3a6ffd640 1 --2- 172.21.15.107:0/4076013963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785094630 0x55d7850906e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3a6ffd640 1 -- 172.21.15.107:0/4076013963 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d785089ba0 con 0x55d785089d70 2024-05-09T08:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3a67fc640 1 --2- 172.21.15.107:0/4076013963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785089450 0x55d785093c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb3a6ffd640 1 --2- 172.21.15.107:0/4076013963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089d70 0x55d7850a0120 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fb398009ca0 tx=0x7fb398031d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bbdcc8fa48d3cec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb38cff9640 1 -- 172.21.15.107:0/4076013963 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb398031650 con 0x55d785089d70 2024-05-09T08:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb38cff9640 1 -- 172.21.15.107:0/4076013963 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb398036040 con 0x55d785089d70 2024-05-09T08:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.641+0000 7fb38cff9640 1 -- 172.21.15.107:0/4076013963 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb398003cf0 con 0x55d785089d70 2024-05-09T08:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.642+0000 7fb3b0428740 1 -- 172.21.15.107:0/4076013963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089d70 msgr2=0x55d7850a0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.642+0000 7fb3b0428740 1 --2- 172.21.15.107:0/4076013963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089d70 0x55d7850a0120 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fb398009ca0 tx=0x7fb398031d80 comp rx=0 tx=0).stop 2024-05-09T08:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.642+0000 7fb3b0428740 1 -- 172.21.15.107:0/4076013963 shutdown_connections 2024-05-09T08:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.642+0000 7fb3b0428740 1 --2- 172.21.15.107:0/4076013963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785094630 0x55d7850906e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.642+0000 7fb3b0428740 1 --2- 172.21.15.107:0/4076013963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785089d70 0x55d7850a0120 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.642+0000 7fb3b0428740 1 --2- 172.21.15.107:0/4076013963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785089450 0x55d785093c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.642+0000 7fb3b0428740 1 -- 172.21.15.107:0/4076013963 >> 172.21.15.107:0/4076013963 conn(0x55d785085960 msgr2=0x55d785084170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.642+0000 7fb3b0428740 1 -- 172.21.15.107:0/4076013963 shutdown_connections 2024-05-09T08:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.642+0000 7fb3b0428740 1 -- 172.21.15.107:0/4076013963 wait complete. 2024-05-09T08:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.642+0000 7fb3b0428740 1 -- start start 2024-05-09T08:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.642+0000 7fb3b0428740 1 Processor -- start 2024-05-09T08:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3b0428740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785089450 0x55d78509c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3b0428740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785089d70 0x55d78508dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3b0428740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785094630 0x55d78508e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3b0428740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a81ad0d0 con 0x55d785094630 2024-05-09T08:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3b0428740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a81ae540 con 0x55d785089d70 2024-05-09T08:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3b0428740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a81ad7a0 con 0x55d785089450 2024-05-09T08:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3b0428740 10 client.0 _ll_register_callbacks cb 0x7fb398009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3a67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785094630 0x55d78508e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3a67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785094630 0x55d78508e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33716/0 (socket says 172.21.15.107:33716) 2024-05-09T08:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3a67fc640 1 -- 172.21.15.107:0/3113828550 learned_addr learned my addr 172.21.15.107:0/3113828550 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3a67fc640 1 --2- 172.21.15.107:0/3113828550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785089450 0x55d78509c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3a6ffd640 1 --2- 172.21.15.107:0/3113828550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785089d70 0x55d78508dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3a67fc640 1 -- 172.21.15.107:0/3113828550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785089450 msgr2=0x55d78509c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3a67fc640 1 --2- 172.21.15.107:0/3113828550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785089450 0x55d78509c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3a67fc640 1 -- 172.21.15.107:0/3113828550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785089d70 msgr2=0x55d78508dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3a67fc640 1 --2- 172.21.15.107:0/3113828550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785089d70 0x55d78508dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3a67fc640 1 -- 172.21.15.107:0/3113828550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d78508e680 con 0x55d785094630 2024-05-09T08:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3a67fc640 1 --2- 172.21.15.107:0/3113828550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785094630 0x55d78508e140 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb39c004fd0 tx=0x7fb39c00bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96747ac6988176c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb38aff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb38aff5640 1 -- 172.21.15.107:0/3113828550 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c019040 con 0x55d785094630 2024-05-09T08:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb38aff5640 1 -- 172.21.15.107:0/3113828550 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb39c013240 con 0x55d785094630 2024-05-09T08:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3b0428740 1 -- 172.21.15.107:0/3113828550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb398036040 con 0x55d785094630 2024-05-09T08:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb38aff5640 1 -- 172.21.15.107:0/3113828550 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb39c044050 con 0x55d785094630 2024-05-09T08:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.643+0000 7fb3b0428740 10 client.15789 fetch_fsmap learned FSMap version 25 2024-05-09T08:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.644+0000 7fb3b0428740 1 -- 172.21.15.107:0/3113828550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d78510a240 con 0x55d785094630 2024-05-09T08:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.644+0000 7fb38aff5640 1 -- 172.21.15.107:0/3113828550 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb398036040 con 0x55d785094630 2024-05-09T08:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.644+0000 7fb38aff5640 1 -- 172.21.15.107:0/3113828550 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb39c0116d0 con 0x55d785094630 2024-05-09T08:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.644+0000 7fb3b0428740 10 client.15789 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.644+0000 7fb3b0428740 10 client.15789 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.644+0000 7fb3b0428740 20 client.15789 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.644+0000 7fb3b0428740 1 --2- 172.21.15.107:0/3113828550 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d78510a990 0x55d78510cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.644+0000 7fb3b0428740 4 client.15789 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:33:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.644+0000 7fb3b0428740 1 -- 172.21.15.107:0/3113828550 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d784ff5dc0 con 0x55d78510a990 2024-05-09T08:33:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.644+0000 7fb3a6ffd640 1 --2- 172.21.15.107:0/3113828550 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d78510a990 0x55d78510cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.644+0000 7fb38aff5640 10 client.15789 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.644+0000 7fb3a6ffd640 1 --2- 172.21.15.107:0/3113828550 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d78510a990 0x55d78510cdb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc73d819e11527b9 server_cookie=5e9ee29d27e581af in_seq=0 out_seq=0 2024-05-09T08:33:56.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.648+0000 7fb38aff5640 1 -- 172.21.15.107:0/3113828550 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d784ff5dc0 con 0x55d78510a990 2024-05-09T08:33:56.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.648+0000 7fb38aff5640 10 client.15789 handle_command_reply: tid=0 2024-05-09T08:33:56.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.648+0000 7fb3b0428740 1 client.15789 shutdown 2024-05-09T08:33:56.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.648+0000 7fb3b0428740 1 -- 172.21.15.107:0/3113828550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785094630 msgr2=0x55d78508e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:56.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.648+0000 7fb3b0428740 1 --2- 172.21.15.107:0/3113828550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785094630 0x55d78508e140 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb39c004fd0 tx=0x7fb39c00bfb0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.648+0000 7fb3b0428740 1 -- 172.21.15.107:0/3113828550 shutdown_connections 2024-05-09T08:33:56.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.648+0000 7fb3b0428740 1 --2- 172.21.15.107:0/3113828550 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d78510a990 0x55d78510cdb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.648+0000 7fb3b0428740 1 --2- 172.21.15.107:0/3113828550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d785094630 0x55d78508e140 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.648+0000 7fb3b0428740 1 --2- 172.21.15.107:0/3113828550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d785089d70 0x55d78508dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.648+0000 7fb38aff5640 0 client.15789 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:56.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.648+0000 7fb3b0428740 1 --2- 172.21.15.107:0/3113828550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d785089450 0x55d78509c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.649+0000 7fb3b0428740 1 -- 172.21.15.107:0/3113828550 >> 172.21.15.107:0/3113828550 conn(0x55d785085960 msgr2=0x55d785084a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:56.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.649+0000 7fb3b0428740 1 -- 172.21.15.107:0/3113828550 shutdown_connections 2024-05-09T08:33:56.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.649+0000 7fb3b0428740 1 -- 172.21.15.107:0/3113828550 wait complete. 2024-05-09T08:33:56.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.649+0000 7fb3b0428740 20 client.15789 trim_cache size 0 max 16384 2024-05-09T08:33:56.652 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:33:56.652 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:33:56.652 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:33:56.652 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:33:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.649+0000 7fb3ad638640 1 -- 172.21.15.107:0/3960060018 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb3900a2ec0 msgr2=0x7fb3900a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.649+0000 7fb3ad638640 1 --2- 172.21.15.107:0/3960060018 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb3900a2ec0 0x7fb3900a5380 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00cc720 tx=0x7fb3a006e430 comp rx=0 tx=0).stop 2024-05-09T08:33:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.649+0000 7fb3ad638640 1 -- 172.21.15.107:0/3960060018 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3a819c840 msgr2=0x7fb3a819ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.649+0000 7fb3ad638640 1 --2- 172.21.15.107:0/3960060018 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3a819c840 0x7fb3a819ccf0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb3a000d750 tx=0x7fb3a000dc10 comp rx=0 tx=0).stop 2024-05-09T08:33:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.649+0000 7fb3ad638640 1 -- 172.21.15.107:0/3960060018 shutdown_connections 2024-05-09T08:33:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.649+0000 7fb3ad638640 1 --2- 172.21.15.107:0/3960060018 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb3900a2ec0 0x7fb3900a5380 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.649+0000 7fb3ad638640 1 --2- 172.21.15.107:0/3960060018 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3a819c840 0x7fb3a819ccf0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.649+0000 7fb3ad638640 1 --2- 172.21.15.107:0/3960060018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3a8106b60 0x7fb3a81984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.649+0000 7fb3ad638640 1 --2- 172.21.15.107:0/3960060018 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3a80fffa0 0x7fb3a8197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.649+0000 7fb3ad638640 1 -- 172.21.15.107:0/3960060018 >> 172.21.15.107:0/3960060018 conn(0x7fb3a8076cc0 msgr2=0x7fb3a8104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.649+0000 7fb3ad638640 1 -- 172.21.15.107:0/3960060018 shutdown_connections 2024-05-09T08:33:56.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:33:56.649+0000 7fb3ad638640 1 -- 172.21.15.107:0/3960060018 wait complete. 2024-05-09T08:33:56.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:56 smithi107 ceph-mon[30122]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-05-09T08:33:56.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:56 smithi157 ceph-mon[33614]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-05-09T08:33:56.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:56 smithi110 ceph-mon[33622]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-05-09T08:33:58.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:33:58 smithi107 ceph-mon[30122]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-05-09T08:33:58.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:33:58 smithi157 ceph-mon[33614]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-05-09T08:33:58.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:33:58 smithi110 ceph-mon[33622]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-05-09T08:33:59.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:33:59.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:33:59.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.195+0000 7f3096010640 1 -- 172.21.15.107:0/2207330998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30901013d0 msgr2=0x7f309010d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.195+0000 7f3096010640 1 --2- 172.21.15.107:0/2207330998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30901013d0 0x7f309010d5a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3088009cd0 tx=0x7f308802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.195+0000 7f3096010640 1 -- 172.21.15.107:0/2207330998 shutdown_connections 2024-05-09T08:33:59.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.195+0000 7f3096010640 1 --2- 172.21.15.107:0/2207330998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30901013d0 0x7f309010d5a0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.195+0000 7f3096010640 1 --2- 172.21.15.107:0/2207330998 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3090100a10 0x7f3090100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.195+0000 7f3096010640 1 --2- 172.21.15.107:0/2207330998 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3090106900 0x7f3090106ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.195+0000 7f3096010640 1 -- 172.21.15.107:0/2207330998 >> 172.21.15.107:0/2207330998 conn(0x7f3090077710 msgr2=0x7f3090077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.195+0000 7f3096010640 1 -- 172.21.15.107:0/2207330998 shutdown_connections 2024-05-09T08:33:59.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.195+0000 7f3096010640 1 -- 172.21.15.107:0/2207330998 wait complete. 2024-05-09T08:33:59.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.196+0000 7f3096010640 1 Processor -- start 2024-05-09T08:33:59.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.196+0000 7f3096010640 1 -- start start 2024-05-09T08:33:59.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.196+0000 7f3096010640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3090100a10 0x7f3090198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.196+0000 7f3096010640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30901013d0 0x7f3090199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.196+0000 7f3096010640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3090106900 0x7f30901a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.196+0000 7f3096010640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3090110a80 con 0x7f30901013d0 2024-05-09T08:33:59.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.196+0000 7f3096010640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3090110900 con 0x7f3090106900 2024-05-09T08:33:59.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.196+0000 7f3096010640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3090110c00 con 0x7f3090100a10 2024-05-09T08:33:59.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.196+0000 7f308effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30901013d0 0x7f3090199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.196+0000 7f308effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30901013d0 0x7f3090199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33744/0 (socket says 172.21.15.107:33744) 2024-05-09T08:33:59.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.196+0000 7f308effd640 1 -- 172.21.15.107:0/1434644229 learned_addr learned my addr 172.21.15.107:0/1434644229 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.196+0000 7f308f7fe640 1 --2- 172.21.15.107:0/1434644229 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3090100a10 0x7f3090198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.197+0000 7f308ffff640 1 --2- 172.21.15.107:0/1434644229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3090106900 0x7f30901a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.197+0000 7f308effd640 1 -- 172.21.15.107:0/1434644229 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3090100a10 msgr2=0x7f3090198f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.197+0000 7f308effd640 1 --2- 172.21.15.107:0/1434644229 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3090100a10 0x7f3090198f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.197+0000 7f308effd640 1 -- 172.21.15.107:0/1434644229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3090106900 msgr2=0x7f30901a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.197+0000 7f308effd640 1 --2- 172.21.15.107:0/1434644229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3090106900 0x7f30901a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.197+0000 7f308effd640 1 -- 172.21.15.107:0/1434644229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30901a44c0 con 0x7f30901013d0 2024-05-09T08:33:59.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.197+0000 7f308f7fe640 1 --2- 172.21.15.107:0/1434644229 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3090100a10 0x7f3090198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:59.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.197+0000 7f308ffff640 1 --2- 172.21.15.107:0/1434644229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3090106900 0x7f30901a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:59.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.197+0000 7f308effd640 1 --2- 172.21.15.107:0/1434644229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30901013d0 0x7f3090199440 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f308400d770 tx=0x7f308400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.197+0000 7f308cff9640 1 -- 172.21.15.107:0/1434644229 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3084017070 con 0x7f30901013d0 2024-05-09T08:33:59.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.197+0000 7f308cff9640 1 -- 172.21.15.107:0/1434644229 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30840054d0 con 0x7f30901013d0 2024-05-09T08:33:59.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.197+0000 7f3096010640 1 -- 172.21.15.107:0/1434644229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30901a4690 con 0x7f30901013d0 2024-05-09T08:33:59.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.197+0000 7f308cff9640 1 -- 172.21.15.107:0/1434644229 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3084004cf0 con 0x7f30901013d0 2024-05-09T08:33:59.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.197+0000 7f3096010640 1 -- 172.21.15.107:0/1434644229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30901a4b30 con 0x7f30901013d0 2024-05-09T08:33:59.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.199+0000 7f308cff9640 1 -- 172.21.15.107:0/1434644229 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7f3084004e90 con 0x7f30901013d0 2024-05-09T08:33:59.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f3098df4740 1 Processor -- start 2024-05-09T08:33:59.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f3098df4740 1 -- start start 2024-05-09T08:33:59.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f3098df4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0b280 0x5618a3f0b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f3098df4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f0bba0 0x5618a3f15d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f3098df4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f16270 0x5618a3f18660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f3098df4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5618a3e70d00 con 0x5618a3f0bba0 2024-05-09T08:33:59.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f3098df4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5618a3e184c0 con 0x5618a3f16270 2024-05-09T08:33:59.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f3098df4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5618a3e6ccc0 con 0x5618a3f0b280 2024-05-09T08:33:59.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f308cff9640 1 --2- 172.21.15.107:0/1434644229 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f30780a2e20 0x7f30780a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f308f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f0bba0 0x5618a3f15d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f308f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f0bba0 0x5618a3f15d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33754/0 (socket says 172.21.15.107:33754) 2024-05-09T08:33:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f308f7fe640 1 -- 172.21.15.107:0/2151247645 learned_addr learned my addr 172.21.15.107:0/2151247645 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f308ffff640 1 --2- 172.21.15.107:0/1434644229 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f30780a2e20 0x7f30780a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f308effd640 1 --2- 172.21.15.107:0/2151247645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f16270 0x5618a3f18660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f308ffff640 1 --2- 172.21.15.107:0/2151247645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0b280 0x5618a3f0b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f308f7fe640 1 -- 172.21.15.107:0/2151247645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0b280 msgr2=0x5618a3f0b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f308f7fe640 1 --2- 172.21.15.107:0/2151247645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0b280 0x5618a3f0b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f308cff9640 1 -- 172.21.15.107:0/1434644229 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f30840cf530 con 0x7f30901013d0 2024-05-09T08:33:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f308f7fe640 1 -- 172.21.15.107:0/2151247645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f16270 msgr2=0x5618a3f18660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f308f7fe640 1 --2- 172.21.15.107:0/2151247645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f16270 0x5618a3f18660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f308f7fe640 1 -- 172.21.15.107:0/2151247645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618a3c4e7d0 con 0x5618a3f0bba0 2024-05-09T08:33:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.200+0000 7f308f7fe640 1 --2- 172.21.15.107:0/2151247645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f0bba0 0x5618a3f15d30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f308000cc20 tx=0x7f308000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4825f74648c021e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.201+0000 7f308ffff640 1 --2- 172.21.15.107:0/1434644229 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f30780a2e20 0x7f30780a52e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3088033040 tx=0x7f3088032fd0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.201+0000 7f3075ffb640 1 -- 172.21.15.107:0/2151247645 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3080005220 con 0x5618a3f0bba0 2024-05-09T08:33:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.201+0000 7f3075ffb640 1 -- 172.21.15.107:0/2151247645 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3080016070 con 0x5618a3f0bba0 2024-05-09T08:33:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.201+0000 7f3075ffb640 1 -- 172.21.15.107:0/2151247645 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3080010690 con 0x5618a3f0bba0 2024-05-09T08:33:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.201+0000 7f3098df4740 1 -- 172.21.15.107:0/2151247645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f0bba0 msgr2=0x5618a3f15d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.201+0000 7f3098df4740 1 --2- 172.21.15.107:0/2151247645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f0bba0 0x5618a3f15d30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f308000cc20 tx=0x7f308000e720 comp rx=0 tx=0).stop 2024-05-09T08:33:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.201+0000 7f3098df4740 1 -- 172.21.15.107:0/2151247645 shutdown_connections 2024-05-09T08:33:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.201+0000 7f3098df4740 1 --2- 172.21.15.107:0/2151247645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f16270 0x5618a3f18660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.201+0000 7f3098df4740 1 --2- 172.21.15.107:0/2151247645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f0bba0 0x5618a3f15d30 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.201+0000 7f3098df4740 1 --2- 172.21.15.107:0/2151247645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0b280 0x5618a3f0b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.201+0000 7f3098df4740 1 -- 172.21.15.107:0/2151247645 >> 172.21.15.107:0/2151247645 conn(0x5618a3f075a0 msgr2=0x5618a3f07980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.201+0000 7f3098df4740 1 -- 172.21.15.107:0/2151247645 shutdown_connections 2024-05-09T08:33:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.201+0000 7f3098df4740 1 -- 172.21.15.107:0/2151247645 wait complete. 2024-05-09T08:33:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.202+0000 7f3098df4740 1 -- start start 2024-05-09T08:33:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.202+0000 7f3098df4740 1 Processor -- start 2024-05-09T08:33:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.202+0000 7f3098df4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f0b280 0x5618a3f13170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.202+0000 7f3098df4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0bba0 0x5618a3f136b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.202+0000 7f3098df4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f16270 0x5618a3f13bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.202+0000 7f3098df4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5618a3f19060 con 0x5618a3f0b280 2024-05-09T08:33:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.202+0000 7f3098df4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5618a3e2bb90 con 0x5618a3f16270 2024-05-09T08:33:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.202+0000 7f3098df4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5618a3f191e0 con 0x5618a3f0bba0 2024-05-09T08:33:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.202+0000 7f3098df4740 10 client.0 _ll_register_callbacks cb 0x5618a3e67da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.202+0000 7f308f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f0b280 0x5618a3f13170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.202+0000 7f308f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f0b280 0x5618a3f13170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33768/0 (socket says 172.21.15.107:33768) 2024-05-09T08:33:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.202+0000 7f308f7fe640 1 -- 172.21.15.107:0/2980491549 learned_addr learned my addr 172.21.15.107:0/2980491549 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.202+0000 7f308ffff640 1 --2- 172.21.15.107:0/2980491549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f16270 0x5618a3f13bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.202+0000 7f308effd640 1 --2- 172.21.15.107:0/2980491549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0bba0 0x5618a3f136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f308f7fe640 1 -- 172.21.15.107:0/2980491549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0bba0 msgr2=0x5618a3f136b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f308f7fe640 1 --2- 172.21.15.107:0/2980491549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0bba0 0x5618a3f136b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f308f7fe640 1 -- 172.21.15.107:0/2980491549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f16270 msgr2=0x5618a3f13bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f308f7fe640 1 --2- 172.21.15.107:0/2980491549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f16270 0x5618a3f13bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f308f7fe640 1 -- 172.21.15.107:0/2980491549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618a3b843f0 con 0x5618a3f0b280 2024-05-09T08:33:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f308ffff640 1 --2- 172.21.15.107:0/2980491549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f16270 0x5618a3f13bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f308effd640 1 --2- 172.21.15.107:0/2980491549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0bba0 0x5618a3f136b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f308f7fe640 1 --2- 172.21.15.107:0/2980491549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f0b280 0x5618a3f13170 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f308000c960 tx=0x7f30800058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558e1838f17989d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f306effd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f306effd640 1 -- 172.21.15.107:0/2980491549 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f308002a060 con 0x5618a3f0b280 2024-05-09T08:33:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f306effd640 1 -- 172.21.15.107:0/2980491549 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f30800048d0 con 0x5618a3f0b280 2024-05-09T08:33:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f3098df4740 1 -- 172.21.15.107:0/2980491549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3080016070 con 0x5618a3f0b280 2024-05-09T08:33:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f306effd640 1 -- 172.21.15.107:0/2980491549 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3080031050 con 0x5618a3f0b280 2024-05-09T08:33:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f3098df4740 10 client.15801 fetch_fsmap learned FSMap version 25 2024-05-09T08:33:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f3098df4740 1 -- 172.21.15.107:0/2980491549 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5618a3b440f0 con 0x5618a3f0b280 2024-05-09T08:33:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f306effd640 1 -- 172.21.15.107:0/2980491549 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3080016070 con 0x5618a3f0b280 2024-05-09T08:33:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f306effd640 1 -- 172.21.15.107:0/2980491549 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3080025c90 con 0x5618a3f0b280 2024-05-09T08:33:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f3098df4740 10 client.15801 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:33:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f3098df4740 10 client.15801 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:33:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f3098df4740 20 client.15801 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f3098df4740 1 --2- 172.21.15.107:0/2980491549 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5618a3f26490 0x5618a3f28880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f3098df4740 4 client.15801 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:33:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.203+0000 7f3098df4740 1 -- 172.21.15.107:0/2980491549 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618a3e7e430 con 0x5618a3f26490 2024-05-09T08:33:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.204+0000 7f308effd640 1 --2- 172.21.15.107:0/2980491549 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5618a3f26490 0x5618a3f28880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.204+0000 7f308effd640 1 --2- 172.21.15.107:0/2980491549 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5618a3f26490 0x5618a3f28880 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b767c991a02881e0 server_cookie=c74b418d1e528a8 in_seq=0 out_seq=0 2024-05-09T08:33:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.204+0000 7f306effd640 10 client.15801 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.205+0000 7f306effd640 1 -- 172.21.15.107:0/2980491549 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5618a3e7e430 con 0x5618a3f26490 2024-05-09T08:33:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.205+0000 7f306effd640 10 client.15801 handle_command_reply: tid=0 2024-05-09T08:33:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.205+0000 7f3098df4740 1 client.15801 shutdown 2024-05-09T08:33:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.205+0000 7f3098df4740 1 -- 172.21.15.107:0/2980491549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f0b280 msgr2=0x5618a3f13170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.205+0000 7f3098df4740 1 --2- 172.21.15.107:0/2980491549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f0b280 0x5618a3f13170 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f308000c960 tx=0x7f30800058a0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.205+0000 7f3098df4740 1 -- 172.21.15.107:0/2980491549 shutdown_connections 2024-05-09T08:33:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.205+0000 7f3098df4740 1 --2- 172.21.15.107:0/2980491549 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5618a3f26490 0x5618a3f28880 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.205+0000 7f3098df4740 1 --2- 172.21.15.107:0/2980491549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f16270 0x5618a3f13bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.205+0000 7f3098df4740 1 --2- 172.21.15.107:0/2980491549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0bba0 0x5618a3f136b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.205+0000 7f306effd640 0 client.15801 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.205+0000 7f3098df4740 1 --2- 172.21.15.107:0/2980491549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f0b280 0x5618a3f13170 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.205+0000 7f3098df4740 1 -- 172.21.15.107:0/2980491549 >> 172.21.15.107:0/2980491549 conn(0x5618a3f075a0 msgr2=0x5618a3f0d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.205+0000 7f3098df4740 1 -- 172.21.15.107:0/2980491549 shutdown_connections 2024-05-09T08:33:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.205+0000 7f3098df4740 1 -- 172.21.15.107:0/2980491549 wait complete. 2024-05-09T08:33:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.205+0000 7f3098df4740 20 client.15801 trim_cache size 0 max 16384 2024-05-09T08:33:59.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.222+0000 7f3098df4740 1 Processor -- start 2024-05-09T08:33:59.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.222+0000 7f3098df4740 1 -- start start 2024-05-09T08:33:59.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f3098df4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0b280 0x5618a3f21f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f3098df4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f0bba0 0x5618a3f06660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f3098df4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f16270 0x5618a3f2dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f3098df4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f30901b13f0 con 0x5618a3f16270 2024-05-09T08:33:59.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f3098df4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f30901aecc0 con 0x5618a3f0bba0 2024-05-09T08:33:59.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f3098df4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f30901acb10 con 0x5618a3f0b280 2024-05-09T08:33:59.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f308ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f16270 0x5618a3f2dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f308ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f16270 0x5618a3f2dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33780/0 (socket says 172.21.15.107:33780) 2024-05-09T08:33:59.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f308ffff640 1 -- 172.21.15.107:0/1343606317 learned_addr learned my addr 172.21.15.107:0/1343606317 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f308effd640 1 --2- 172.21.15.107:0/1343606317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f0bba0 0x5618a3f06660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f308ffff640 1 -- 172.21.15.107:0/1343606317 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0b280 msgr2=0x5618a3f21f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f308f7fe640 1 --2- 172.21.15.107:0/1343606317 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0b280 0x5618a3f21f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f308ffff640 1 --2- 172.21.15.107:0/1343606317 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0b280 0x5618a3f21f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f308ffff640 1 -- 172.21.15.107:0/1343606317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f0bba0 msgr2=0x5618a3f06660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f308ffff640 1 --2- 172.21.15.107:0/1343606317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f0bba0 0x5618a3f06660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f308ffff640 1 -- 172.21.15.107:0/1343606317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618a3f116c0 con 0x5618a3f16270 2024-05-09T08:33:59.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f308f7fe640 1 --2- 172.21.15.107:0/1343606317 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0b280 0x5618a3f21f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:59.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f308effd640 1 --2- 172.21.15.107:0/1343606317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f0bba0 0x5618a3f06660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:59.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f308ffff640 1 --2- 172.21.15.107:0/1343606317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f16270 0x5618a3f2dee0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3088031ae0 tx=0x7f308803e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c17f0d79f66e92 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f30757fa640 1 -- 172.21.15.107:0/1343606317 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3088031c90 con 0x5618a3f16270 2024-05-09T08:33:59.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f30757fa640 1 -- 172.21.15.107:0/1343606317 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3088036040 con 0x5618a3f16270 2024-05-09T08:33:59.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.223+0000 7f30757fa640 1 -- 172.21.15.107:0/1343606317 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3088030af0 con 0x5618a3f16270 2024-05-09T08:33:59.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.224+0000 7f3098df4740 1 -- 172.21.15.107:0/1343606317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f16270 msgr2=0x5618a3f2dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.224+0000 7f3098df4740 1 --2- 172.21.15.107:0/1343606317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f16270 0x5618a3f2dee0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3088031ae0 tx=0x7f308803e000 comp rx=0 tx=0).stop 2024-05-09T08:33:59.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.224+0000 7f3098df4740 1 -- 172.21.15.107:0/1343606317 shutdown_connections 2024-05-09T08:33:59.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.224+0000 7f3098df4740 1 --2- 172.21.15.107:0/1343606317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f16270 0x5618a3f2dee0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.224+0000 7f3098df4740 1 --2- 172.21.15.107:0/1343606317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f0bba0 0x5618a3f06660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.224+0000 7f3098df4740 1 --2- 172.21.15.107:0/1343606317 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0b280 0x5618a3f21f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.224+0000 7f3098df4740 1 -- 172.21.15.107:0/1343606317 >> 172.21.15.107:0/1343606317 conn(0x5618a3f075a0 msgr2=0x5618a3f05f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.224+0000 7f3098df4740 1 -- 172.21.15.107:0/1343606317 shutdown_connections 2024-05-09T08:33:59.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.224+0000 7f3098df4740 1 -- 172.21.15.107:0/1343606317 wait complete. 2024-05-09T08:33:59.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.224+0000 7f3098df4740 1 -- start start 2024-05-09T08:33:59.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f3098df4740 1 Processor -- start 2024-05-09T08:33:59.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f3098df4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0b280 0x5618a3f1e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f3098df4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f0bba0 0x5618a3f19ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f3098df4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f16270 0x5618a3f19ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f3098df4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f30901ad180 con 0x5618a3f16270 2024-05-09T08:33:59.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f3098df4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f30901ae5f0 con 0x5618a3f0bba0 2024-05-09T08:33:59.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f3098df4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f30901ad850 con 0x5618a3f0b280 2024-05-09T08:33:59.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f3098df4740 10 client.0 _ll_register_callbacks cb 0x7f3080014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:59.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f308ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f16270 0x5618a3f19ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f308ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f16270 0x5618a3f19ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33794/0 (socket says 172.21.15.107:33794) 2024-05-09T08:33:59.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f308ffff640 1 -- 172.21.15.107:0/509378321 learned_addr learned my addr 172.21.15.107:0/509378321 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f308ffff640 1 -- 172.21.15.107:0/509378321 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0b280 msgr2=0x5618a3f1e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f308f7fe640 1 --2- 172.21.15.107:0/509378321 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0b280 0x5618a3f1e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f308ffff640 1 --2- 172.21.15.107:0/509378321 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0b280 0x5618a3f1e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f308ffff640 1 -- 172.21.15.107:0/509378321 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f0bba0 msgr2=0x5618a3f19ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f308effd640 1 --2- 172.21.15.107:0/509378321 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f0bba0 0x5618a3f19ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f308ffff640 1 --2- 172.21.15.107:0/509378321 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f0bba0 0x5618a3f19ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f308ffff640 1 -- 172.21.15.107:0/509378321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618a3f1a7f0 con 0x5618a3f16270 2024-05-09T08:33:59.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f308ffff640 1 --2- 172.21.15.107:0/509378321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f16270 0x5618a3f19ff0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f308802a920 tx=0x7f3088030ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad8012c0b0a9a60 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f306effd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:59.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f306effd640 1 -- 172.21.15.107:0/509378321 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3088060020 con 0x5618a3f16270 2024-05-09T08:33:59.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f306effd640 1 -- 172.21.15.107:0/509378321 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3088004250 con 0x5618a3f16270 2024-05-09T08:33:59.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.225+0000 7f3098df4740 1 -- 172.21.15.107:0/509378321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3088036040 con 0x5618a3f16270 2024-05-09T08:33:59.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.226+0000 7f306effd640 1 -- 172.21.15.107:0/509378321 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f308806b050 con 0x5618a3f16270 2024-05-09T08:33:59.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.226+0000 7f3098df4740 10 client.15807 fetch_fsmap learned FSMap version 25 2024-05-09T08:33:59.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.226+0000 7f3098df4740 1 -- 172.21.15.107:0/509378321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5618a3f8c620 con 0x5618a3f16270 2024-05-09T08:33:59.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.226+0000 7f306effd640 1 -- 172.21.15.107:0/509378321 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3088036040 con 0x5618a3f16270 2024-05-09T08:33:59.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.226+0000 7f306effd640 1 -- 172.21.15.107:0/509378321 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3088030070 con 0x5618a3f16270 2024-05-09T08:33:59.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.226+0000 7f3098df4740 10 client.15807 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:33:59.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.226+0000 7f3098df4740 10 client.15807 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:33:59.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.226+0000 7f3098df4740 20 client.15807 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:59.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.226+0000 7f3098df4740 1 --2- 172.21.15.107:0/509378321 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5618a3f8cd60 0x5618a3f8f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.226+0000 7f3098df4740 4 client.15807 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:33:59.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.226+0000 7f3098df4740 1 -- 172.21.15.107:0/509378321 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5618a3e78780 con 0x5618a3f8cd60 2024-05-09T08:33:59.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.226+0000 7f308f7fe640 1 --2- 172.21.15.107:0/509378321 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5618a3f8cd60 0x5618a3f8f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f308f7fe640 1 --2- 172.21.15.107:0/509378321 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5618a3f8cd60 0x5618a3f8f180 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=425a32e0d61ce10d server_cookie=4f91b2598e62b296 in_seq=0 out_seq=0 2024-05-09T08:33:59.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f306effd640 10 client.15807 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:59.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f306effd640 1 -- 172.21.15.107:0/509378321 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5618a3e78780 con 0x5618a3f8cd60 2024-05-09T08:33:59.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f306effd640 10 client.15807 handle_command_reply: tid=0 2024-05-09T08:33:59.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f3098df4740 1 client.15807 shutdown 2024-05-09T08:33:59.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f3098df4740 1 -- 172.21.15.107:0/509378321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f16270 msgr2=0x5618a3f19ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f3098df4740 1 --2- 172.21.15.107:0/509378321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f16270 0x5618a3f19ff0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f308802a920 tx=0x7f3088030ff0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f3098df4740 1 -- 172.21.15.107:0/509378321 shutdown_connections 2024-05-09T08:33:59.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f3098df4740 1 --2- 172.21.15.107:0/509378321 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5618a3f8cd60 0x5618a3f8f180 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f3098df4740 1 --2- 172.21.15.107:0/509378321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618a3f16270 0x5618a3f19ff0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f3098df4740 1 --2- 172.21.15.107:0/509378321 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a3f0bba0 0x5618a3f19ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f306effd640 0 client.15807 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:59.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f3098df4740 1 --2- 172.21.15.107:0/509378321 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5618a3f0b280 0x5618a3f1e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f3098df4740 1 -- 172.21.15.107:0/509378321 >> 172.21.15.107:0/509378321 conn(0x5618a3f075a0 msgr2=0x5618a3f067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f3098df4740 1 -- 172.21.15.107:0/509378321 shutdown_connections 2024-05-09T08:33:59.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f3098df4740 1 -- 172.21.15.107:0/509378321 wait complete. 2024-05-09T08:33:59.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.227+0000 7f3098df4740 20 client.15807 trim_cache size 0 max 16384 2024-05-09T08:33:59.235 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:33:59.235 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:33:59.235 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:33:59.235 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:33:59.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.228+0000 7f3096010640 1 -- 172.21.15.107:0/1434644229 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f30780a2e20 msgr2=0x7f30780a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.228+0000 7f3096010640 1 --2- 172.21.15.107:0/1434644229 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f30780a2e20 0x7f30780a52e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3088033040 tx=0x7f3088032fd0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.228+0000 7f3096010640 1 -- 172.21.15.107:0/1434644229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30901013d0 msgr2=0x7f3090199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.228+0000 7f3096010640 1 --2- 172.21.15.107:0/1434644229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30901013d0 0x7f3090199440 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f308400d770 tx=0x7f308400dc30 comp rx=0 tx=0).stop 2024-05-09T08:33:59.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.228+0000 7f3096010640 1 -- 172.21.15.107:0/1434644229 shutdown_connections 2024-05-09T08:33:59.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.228+0000 7f3096010640 1 --2- 172.21.15.107:0/1434644229 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f30780a2e20 0x7f30780a52e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.228+0000 7f3096010640 1 --2- 172.21.15.107:0/1434644229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3090106900 0x7f30901a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.228+0000 7f3096010640 1 --2- 172.21.15.107:0/1434644229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30901013d0 0x7f3090199440 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.228+0000 7f3096010640 1 --2- 172.21.15.107:0/1434644229 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3090100a10 0x7f3090198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.228+0000 7f3096010640 1 -- 172.21.15.107:0/1434644229 >> 172.21.15.107:0/1434644229 conn(0x7f3090077710 msgr2=0x7f30900ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.228+0000 7f3096010640 1 -- 172.21.15.107:0/1434644229 shutdown_connections 2024-05-09T08:33:59.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.228+0000 7f3096010640 1 -- 172.21.15.107:0/1434644229 wait complete. 2024-05-09T08:33:59.238 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:33:59.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 -- 172.21.15.107:0/3547440790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53081010d0 msgr2=0x7f530810d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 --2- 172.21.15.107:0/3547440790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53081010d0 0x7f530810d5e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5300009930 tx=0x7f530002a420 comp rx=0 tx=0).stop 2024-05-09T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 -- 172.21.15.107:0/3547440790 shutdown_connections 2024-05-09T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 --2- 172.21.15.107:0/3547440790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53081010d0 0x7f530810d5e0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 --2- 172.21.15.107:0/3547440790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5308100710 0x7f5308100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 --2- 172.21.15.107:0/3547440790 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5308106720 0x7f5308106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 -- 172.21.15.107:0/3547440790 >> 172.21.15.107:0/3547440790 conn(0x7f5308077830 msgr2=0x7f5308077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 -- 172.21.15.107:0/3547440790 shutdown_connections 2024-05-09T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 -- 172.21.15.107:0/3547440790 wait complete. 2024-05-09T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 Processor -- start 2024-05-09T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 -- start start 2024-05-09T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5308100710 0x7f5308198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53081010d0 0x7f53081993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5308106720 0x7f53081a3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f53081109e0 con 0x7f5308106720 2024-05-09T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5308110860 con 0x7f53081010d0 2024-05-09T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.341+0000 7f5310529640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5308110b60 con 0x7f5308100710 2024-05-09T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f530eac6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5308106720 0x7f53081a3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f530eac6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5308106720 0x7f53081a3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33808/0 (socket says 172.21.15.107:33808) 2024-05-09T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f530eac6640 1 -- 172.21.15.107:0/157479097 learned_addr learned my addr 172.21.15.107:0/157479097 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f530dac4640 1 --2- 172.21.15.107:0/157479097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53081010d0 0x7f53081993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f530e2c5640 1 --2- 172.21.15.107:0/157479097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5308100710 0x7f5308198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f530eac6640 1 -- 172.21.15.107:0/157479097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5308100710 msgr2=0x7f5308198e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f530eac6640 1 --2- 172.21.15.107:0/157479097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5308100710 0x7f5308198e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f530eac6640 1 -- 172.21.15.107:0/157479097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53081010d0 msgr2=0x7f53081993c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f530eac6640 1 --2- 172.21.15.107:0/157479097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53081010d0 0x7f53081993c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f530eac6640 1 -- 172.21.15.107:0/157479097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53081a4440 con 0x7f5308106720 2024-05-09T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f530dac4640 1 --2- 172.21.15.107:0/157479097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53081010d0 0x7f53081993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f530eac6640 1 --2- 172.21.15.107:0/157479097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5308106720 0x7f53081a3f00 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5300005940 tx=0x7f5300004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f53077fe640 1 -- 172.21.15.107:0/157479097 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530003f030 con 0x7f5308106720 2024-05-09T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f53077fe640 1 -- 172.21.15.107:0/157479097 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5300003aa0 con 0x7f5308106720 2024-05-09T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f53077fe640 1 -- 172.21.15.107:0/157479097 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300035070 con 0x7f5308106720 2024-05-09T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f5310529640 1 -- 172.21.15.107:0/157479097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53081a4610 con 0x7f5308106720 2024-05-09T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.342+0000 7f5310529640 1 -- 172.21.15.107:0/157479097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53081a4a80 con 0x7f5308106720 2024-05-09T08:33:59.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.343+0000 7f5310529640 1 -- 172.21.15.107:0/157479097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5308068650 con 0x7f5308106720 2024-05-09T08:33:59.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.344+0000 7f53077fe640 1 -- 172.21.15.107:0/157479097 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7f5300042030 con 0x7f5308106720 2024-05-09T08:33:59.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.345+0000 7f53077fe640 1 --2- 172.21.15.107:0/157479097 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f52f00a2e20 0x7f52f00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.345+0000 7f530e2c5640 1 --2- 172.21.15.107:0/157479097 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f52f00a2e20 0x7f52f00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.345+0000 7f53077fe640 1 -- 172.21.15.107:0/157479097 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f53000f0610 con 0x7f5308106720 2024-05-09T08:33:59.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.345+0000 7f530e2c5640 1 --2- 172.21.15.107:0/157479097 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f52f00a2e20 0x7f52f00a52e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f52f80092f0 tx=0x7f52f8009280 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.349+0000 7f53077fe640 1 -- 172.21.15.107:0/157479097 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f53000bcd50 con 0x7f5308106720 2024-05-09T08:33:59.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.566+0000 7f5310529640 1 -- 172.21.15.107:0/157479097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5308108c80 con 0x7f5308106720 2024-05-09T08:33:59.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.567+0000 7f53077fe640 1 -- 172.21.15.107:0/157479097 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11887 (secure 0 0 0) 0x7f53000bc650 con 0x7f5308106720 2024-05-09T08:33:59.566 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:33:59.567 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:33:19.308598+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:33:59.567 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 25 2024-05-09T08:33:59.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.570+0000 7f5310529640 1 -- 172.21.15.107:0/157479097 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f52f00a2e20 msgr2=0x7f52f00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.570+0000 7f5310529640 1 --2- 172.21.15.107:0/157479097 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f52f00a2e20 0x7f52f00a52e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f52f80092f0 tx=0x7f52f8009280 comp rx=0 tx=0).stop 2024-05-09T08:33:59.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.570+0000 7f5310529640 1 -- 172.21.15.107:0/157479097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5308106720 msgr2=0x7f53081a3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.570+0000 7f5310529640 1 --2- 172.21.15.107:0/157479097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5308106720 0x7f53081a3f00 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5300005940 tx=0x7f5300004270 comp rx=0 tx=0).stop 2024-05-09T08:33:59.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.570+0000 7f5310529640 1 -- 172.21.15.107:0/157479097 shutdown_connections 2024-05-09T08:33:59.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.570+0000 7f5310529640 1 --2- 172.21.15.107:0/157479097 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f52f00a2e20 0x7f52f00a52e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.570+0000 7f5310529640 1 --2- 172.21.15.107:0/157479097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5308106720 0x7f53081a3f00 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.570+0000 7f5310529640 1 --2- 172.21.15.107:0/157479097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53081010d0 0x7f53081993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.570+0000 7f5310529640 1 --2- 172.21.15.107:0/157479097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5308100710 0x7f5308198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.570+0000 7f5310529640 1 -- 172.21.15.107:0/157479097 >> 172.21.15.107:0/157479097 conn(0x7f5308077830 msgr2=0x7f53081030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.570+0000 7f5310529640 1 -- 172.21.15.107:0/157479097 shutdown_connections 2024-05-09T08:33:59.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.570+0000 7f5310529640 1 -- 172.21.15.107:0/157479097 wait complete. 2024-05-09T08:33:59.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:33:59.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.689+0000 7f17fdfe9640 1 -- 172.21.15.107:0/3397521362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17f806a8b0 msgr2=0x7f17f810e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.689+0000 7f17fdfe9640 1 --2- 172.21.15.107:0/3397521362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17f806a8b0 0x7f17f810e6e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f17f0009cd0 tx=0x7f17f002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.689+0000 7f17fdfe9640 1 -- 172.21.15.107:0/3397521362 shutdown_connections 2024-05-09T08:33:59.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.689+0000 7f17fdfe9640 1 --2- 172.21.15.107:0/3397521362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17f806a8b0 0x7f17f810e6e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.689+0000 7f17fdfe9640 1 --2- 172.21.15.107:0/3397521362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17f8069ef0 0x7f17f806a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.689+0000 7f17fdfe9640 1 --2- 172.21.15.107:0/3397521362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17f8107820 0x7f17f8107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.689+0000 7f17fdfe9640 1 -- 172.21.15.107:0/3397521362 >> 172.21.15.107:0/3397521362 conn(0x7f17f8077790 msgr2=0x7f17f8077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.689+0000 7f17fdfe9640 1 -- 172.21.15.107:0/3397521362 shutdown_connections 2024-05-09T08:33:59.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.690+0000 7f17fdfe9640 1 -- 172.21.15.107:0/3397521362 wait complete. 2024-05-09T08:33:59.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.690+0000 7f17fdfe9640 1 Processor -- start 2024-05-09T08:33:59.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.690+0000 7f17fdfe9640 1 -- start start 2024-05-09T08:33:59.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.690+0000 7f17fdfe9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17f8069ef0 0x7f17f8198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.690+0000 7f17fdfe9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17f806a8b0 0x7f17f8198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.690+0000 7f17fdfe9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17f8107820 0x7f17f819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.690+0000 7f17fdfe9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f17f8111ae0 con 0x7f17f8107820 2024-05-09T08:33:59.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.690+0000 7f17fdfe9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f17f8111960 con 0x7f17f806a8b0 2024-05-09T08:33:59.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.690+0000 7f17fdfe9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f17f8111c60 con 0x7f17f8069ef0 2024-05-09T08:33:59.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.690+0000 7f17f7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17f8107820 0x7f17f819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.690+0000 7f17f7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17f8107820 0x7f17f819c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33812/0 (socket says 172.21.15.107:33812) 2024-05-09T08:33:59.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.690+0000 7f17f7fff640 1 -- 172.21.15.107:0/759512452 learned_addr learned my addr 172.21.15.107:0/759512452 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.690+0000 7f17f77fe640 1 --2- 172.21.15.107:0/759512452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17f8069ef0 0x7f17f8198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.691+0000 7f17effff640 1 --2- 172.21.15.107:0/759512452 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17f806a8b0 0x7f17f8198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.691+0000 7f17f7fff640 1 -- 172.21.15.107:0/759512452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17f8069ef0 msgr2=0x7f17f8198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.691+0000 7f17f7fff640 1 --2- 172.21.15.107:0/759512452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17f8069ef0 0x7f17f8198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.691+0000 7f17f7fff640 1 -- 172.21.15.107:0/759512452 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17f806a8b0 msgr2=0x7f17f8198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.691+0000 7f17f7fff640 1 --2- 172.21.15.107:0/759512452 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17f806a8b0 0x7f17f8198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.691+0000 7f17f7fff640 1 -- 172.21.15.107:0/759512452 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f819d030 con 0x7f17f8107820 2024-05-09T08:33:59.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.691+0000 7f17f7fff640 1 --2- 172.21.15.107:0/759512452 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17f8107820 0x7f17f819c900 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f17f0005cc0 tx=0x7f17f0003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.691+0000 7f17f57fa640 1 -- 172.21.15.107:0/759512452 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f0004450 con 0x7f17f8107820 2024-05-09T08:33:59.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.691+0000 7f17fdfe9640 1 -- 172.21.15.107:0/759512452 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f819d2c0 con 0x7f17f8107820 2024-05-09T08:33:59.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.691+0000 7f17f57fa640 1 -- 172.21.15.107:0/759512452 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17f0031c90 con 0x7f17f8107820 2024-05-09T08:33:59.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.691+0000 7f17f57fa640 1 -- 172.21.15.107:0/759512452 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f003b400 con 0x7f17f8107820 2024-05-09T08:33:59.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.691+0000 7f17fdfe9640 1 -- 172.21.15.107:0/759512452 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f81a4bb0 con 0x7f17f8107820 2024-05-09T08:33:59.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.693+0000 7f17f57fa640 1 -- 172.21.15.107:0/759512452 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7f17f0004680 con 0x7f17f8107820 2024-05-09T08:33:59.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.693+0000 7f1800dcb740 1 Processor -- start 2024-05-09T08:33:59.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.693+0000 7f1800dcb740 1 -- start start 2024-05-09T08:33:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f1800dcb740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 0x55e94fa74b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f1800dcb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa75060 0x55e94fa7f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f1800dcb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa7f730 0x55e94fa81b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f1800dcb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e94f9d5c40 con 0x55e94fa75060 2024-05-09T08:33:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f1800dcb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e94f97d580 con 0x55e94fa7f730 2024-05-09T08:33:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f1800dcb740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55e94f9d1c00 con 0x55e94fa74740 2024-05-09T08:33:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17f77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa75060 0x55e94fa7f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17f77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa75060 0x55e94fa7f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33826/0 (socket says 172.21.15.107:33826) 2024-05-09T08:33:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17f77fe640 1 -- 172.21.15.107:0/850695112 learned_addr learned my addr 172.21.15.107:0/850695112 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17f57fa640 1 --2- 172.21.15.107:0/759512452 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f17e40a2e20 0x7f17e40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17f7fff640 1 --2- 172.21.15.107:0/759512452 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f17e40a2e20 0x7f17e40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17effff640 1 --2- 172.21.15.107:0/850695112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 0x55e94fa74b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17effff640 1 --2- 172.21.15.107:0/850695112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa7f730 0x55e94fa81b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17f77fe640 1 -- 172.21.15.107:0/850695112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 msgr2=0x55e94fa74b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17f77fe640 1 --2- 172.21.15.107:0/850695112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 0x55e94fa74b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17f77fe640 1 -- 172.21.15.107:0/850695112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa7f730 msgr2=0x55e94fa81b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17f77fe640 1 --2- 172.21.15.107:0/850695112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa7f730 0x55e94fa81b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17f77fe640 1 -- 172.21.15.107:0/850695112 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e94f6e93f0 con 0x55e94fa75060 2024-05-09T08:33:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17effff640 1 --2- 172.21.15.107:0/850695112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa7f730 0x55e94fa81b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17f57fa640 1 -- 172.21.15.107:0/759512452 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f17f002e030 con 0x7f17f8107820 2024-05-09T08:33:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17f77fe640 1 --2- 172.21.15.107:0/850695112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa75060 0x55e94fa7f1f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f17e800bc90 tx=0x7f17e80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c830a4e3c2737218 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17f7fff640 1 --2- 172.21.15.107:0/759512452 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f17e40a2e20 0x7f17e40a52e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f17f00edec0 tx=0x7f17f00b87d0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17edffb640 1 -- 172.21.15.107:0/850695112 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e801c480 con 0x55e94fa75060 2024-05-09T08:33:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.694+0000 7f17edffb640 1 -- 172.21.15.107:0/850695112 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17e801cad0 con 0x55e94fa75060 2024-05-09T08:33:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.695+0000 7f17edffb640 1 -- 172.21.15.107:0/850695112 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8011ae0 con 0x55e94fa75060 2024-05-09T08:33:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.695+0000 7f1800dcb740 1 -- 172.21.15.107:0/850695112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa75060 msgr2=0x55e94fa7f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.695+0000 7f1800dcb740 1 --2- 172.21.15.107:0/850695112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa75060 0x55e94fa7f1f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f17e800bc90 tx=0x7f17e80107a0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.695+0000 7f1800dcb740 1 -- 172.21.15.107:0/850695112 shutdown_connections 2024-05-09T08:33:59.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.695+0000 7f1800dcb740 1 --2- 172.21.15.107:0/850695112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa7f730 0x55e94fa81b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.695+0000 7f1800dcb740 1 --2- 172.21.15.107:0/850695112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa75060 0x55e94fa7f1f0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.695+0000 7f1800dcb740 1 --2- 172.21.15.107:0/850695112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 0x55e94fa74b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.695+0000 7f1800dcb740 1 -- 172.21.15.107:0/850695112 >> 172.21.15.107:0/850695112 conn(0x55e94fa70a60 msgr2=0x55e94fa70e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.695+0000 7f1800dcb740 1 -- 172.21.15.107:0/850695112 shutdown_connections 2024-05-09T08:33:59.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.695+0000 7f1800dcb740 1 -- 172.21.15.107:0/850695112 wait complete. 2024-05-09T08:33:59.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f1800dcb740 1 -- start start 2024-05-09T08:33:59.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f1800dcb740 1 Processor -- start 2024-05-09T08:33:59.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f1800dcb740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 0x55e94fa7c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f1800dcb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa75060 0x55e94fa7cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f1800dcb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa7f730 0x55e94fa7d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f1800dcb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e94fa82520 con 0x55e94fa75060 2024-05-09T08:33:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f1800dcb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e94f990c40 con 0x55e94fa7f730 2024-05-09T08:33:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f1800dcb740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55e94fa826a0 con 0x55e94fa74740 2024-05-09T08:33:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f1800dcb740 10 client.0 _ll_register_callbacks cb 0x55e94f6e8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f17f77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa75060 0x55e94fa7cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f17f77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa75060 0x55e94fa7cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33832/0 (socket says 172.21.15.107:33832) 2024-05-09T08:33:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f17f77fe640 1 -- 172.21.15.107:0/1939741712 learned_addr learned my addr 172.21.15.107:0/1939741712 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f17effff640 1 --2- 172.21.15.107:0/1939741712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa7f730 0x55e94fa7d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f17effff640 1 --2- 172.21.15.107:0/1939741712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 0x55e94fa7c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f17f77fe640 1 -- 172.21.15.107:0/1939741712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 msgr2=0x55e94fa7c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f17f77fe640 1 --2- 172.21.15.107:0/1939741712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 0x55e94fa7c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f17f77fe640 1 -- 172.21.15.107:0/1939741712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa7f730 msgr2=0x55e94fa7d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f17f77fe640 1 --2- 172.21.15.107:0/1939741712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa7f730 0x55e94fa7d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f17f77fe640 1 -- 172.21.15.107:0/1939741712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e94f7b37d0 con 0x55e94fa75060 2024-05-09T08:33:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.696+0000 7f17effff640 1 --2- 172.21.15.107:0/1939741712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 0x55e94fa7c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.697+0000 7f17f77fe640 1 --2- 172.21.15.107:0/1939741712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa75060 0x55e94fa7cbf0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f17e8011010 tx=0x7f17e801c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf130a0899bf84c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.697+0000 7f17d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.697+0000 7f17d6ffd640 1 -- 172.21.15.107:0/1939741712 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8005250 con 0x55e94fa75060 2024-05-09T08:33:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.697+0000 7f17d6ffd640 1 -- 172.21.15.107:0/1939741712 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f17e80053f0 con 0x55e94fa75060 2024-05-09T08:33:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.697+0000 7f1800dcb740 1 -- 172.21.15.107:0/1939741712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17e801cad0 con 0x55e94fa75060 2024-05-09T08:33:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.697+0000 7f17d6ffd640 1 -- 172.21.15.107:0/1939741712 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f17e80055f0 con 0x55e94fa75060 2024-05-09T08:33:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.697+0000 7f1800dcb740 10 client.15825 fetch_fsmap learned FSMap version 25 2024-05-09T08:33:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.697+0000 7f1800dcb740 1 -- 172.21.15.107:0/1939741712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e94fa8e6f0 con 0x55e94fa75060 2024-05-09T08:33:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.697+0000 7f17d6ffd640 1 -- 172.21.15.107:0/1939741712 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17e801cad0 con 0x55e94fa75060 2024-05-09T08:33:59.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.697+0000 7f17d6ffd640 1 -- 172.21.15.107:0/1939741712 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f17e8004990 con 0x55e94fa75060 2024-05-09T08:33:59.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.697+0000 7f1800dcb740 10 client.15825 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:33:59.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.697+0000 7f1800dcb740 10 client.15825 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:33:59.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.697+0000 7f1800dcb740 20 client.15825 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:59.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.697+0000 7f1800dcb740 1 --2- 172.21.15.107:0/1939741712 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55e94fa8faa0 0x55e94fa91e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.697+0000 7f1800dcb740 4 client.15825 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:33:59.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.697+0000 7f1800dcb740 1 -- 172.21.15.107:0/1939741712 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e94f9e3370 con 0x55e94fa8faa0 2024-05-09T08:33:59.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.698+0000 7f17effff640 1 --2- 172.21.15.107:0/1939741712 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55e94fa8faa0 0x55e94fa91e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.698+0000 7f17d6ffd640 10 client.15825 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:33:59.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.698+0000 7f17effff640 1 --2- 172.21.15.107:0/1939741712 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55e94fa8faa0 0x55e94fa91e90 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e3ac689d896291b3 server_cookie=f191f78e85e3449e in_seq=0 out_seq=0 2024-05-09T08:33:59.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.699+0000 7f17d6ffd640 1 -- 172.21.15.107:0/1939741712 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e94f9e3370 con 0x55e94fa8faa0 2024-05-09T08:33:59.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.699+0000 7f17d6ffd640 10 client.15825 handle_command_reply: tid=0 2024-05-09T08:33:59.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.699+0000 7f1800dcb740 1 client.15825 shutdown 2024-05-09T08:33:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.699+0000 7f1800dcb740 1 -- 172.21.15.107:0/1939741712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa75060 msgr2=0x55e94fa7cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.699+0000 7f1800dcb740 1 --2- 172.21.15.107:0/1939741712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa75060 0x55e94fa7cbf0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f17e8011010 tx=0x7f17e801c730 comp rx=0 tx=0).stop 2024-05-09T08:33:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.699+0000 7f1800dcb740 1 -- 172.21.15.107:0/1939741712 shutdown_connections 2024-05-09T08:33:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.699+0000 7f1800dcb740 1 --2- 172.21.15.107:0/1939741712 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55e94fa8faa0 0x55e94fa91e90 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.699+0000 7f1800dcb740 1 --2- 172.21.15.107:0/1939741712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa7f730 0x55e94fa7d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.699+0000 7f1800dcb740 1 --2- 172.21.15.107:0/1939741712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa75060 0x55e94fa7cbf0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.699+0000 7f17d6ffd640 0 client.15825 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:33:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.699+0000 7f1800dcb740 1 --2- 172.21.15.107:0/1939741712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 0x55e94fa7c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.699+0000 7f1800dcb740 1 -- 172.21.15.107:0/1939741712 >> 172.21.15.107:0/1939741712 conn(0x55e94fa70a60 msgr2=0x55e94fa6b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.699+0000 7f1800dcb740 1 -- 172.21.15.107:0/1939741712 shutdown_connections 2024-05-09T08:33:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.699+0000 7f1800dcb740 1 -- 172.21.15.107:0/1939741712 wait complete. 2024-05-09T08:33:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.699+0000 7f1800dcb740 20 client.15825 trim_cache size 0 max 16384 2024-05-09T08:33:59.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f1800dcb740 1 Processor -- start 2024-05-09T08:33:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f1800dcb740 1 -- start start 2024-05-09T08:33:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f1800dcb740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 0x55e94fa8b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f1800dcb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa75060 0x55e94fa76760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f1800dcb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa7f730 0x55e94fa976c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f1800dcb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81af100 con 0x55e94fa7f730 2024-05-09T08:33:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f1800dcb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81ac9d0 con 0x55e94fa75060 2024-05-09T08:33:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f1800dcb740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81aa820 con 0x55e94fa74740 2024-05-09T08:33:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f17effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa7f730 0x55e94fa976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f17effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa7f730 0x55e94fa976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33840/0 (socket says 172.21.15.107:33840) 2024-05-09T08:33:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f17effff640 1 -- 172.21.15.107:0/1282240724 learned_addr learned my addr 172.21.15.107:0/1282240724 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f17effff640 1 -- 172.21.15.107:0/1282240724 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 msgr2=0x55e94fa8b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f17effff640 1 --2- 172.21.15.107:0/1282240724 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 0x55e94fa8b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f17effff640 1 -- 172.21.15.107:0/1282240724 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa75060 msgr2=0x55e94fa76760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f17effff640 1 --2- 172.21.15.107:0/1282240724 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa75060 0x55e94fa76760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f17effff640 1 -- 172.21.15.107:0/1282240724 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e94fa7b660 con 0x55e94fa7f730 2024-05-09T08:33:59.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f17effff640 1 --2- 172.21.15.107:0/1282240724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa7f730 0x55e94fa976c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f17e000f010 tx=0x7f17e00128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e978756f20ed80b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f17ed7fa640 1 -- 172.21.15.107:0/1282240724 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e0021020 con 0x55e94fa7f730 2024-05-09T08:33:59.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f17ed7fa640 1 -- 172.21.15.107:0/1282240724 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17e001d820 con 0x55e94fa7f730 2024-05-09T08:33:59.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.713+0000 7f17ed7fa640 1 -- 172.21.15.107:0/1282240724 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e0010d90 con 0x55e94fa7f730 2024-05-09T08:33:59.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.714+0000 7f1800dcb740 1 -- 172.21.15.107:0/1282240724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa7f730 msgr2=0x55e94fa976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.714+0000 7f1800dcb740 1 --2- 172.21.15.107:0/1282240724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa7f730 0x55e94fa976c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f17e000f010 tx=0x7f17e00128e0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.714+0000 7f1800dcb740 1 -- 172.21.15.107:0/1282240724 shutdown_connections 2024-05-09T08:33:59.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.714+0000 7f1800dcb740 1 --2- 172.21.15.107:0/1282240724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa7f730 0x55e94fa976c0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.714+0000 7f1800dcb740 1 --2- 172.21.15.107:0/1282240724 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa75060 0x55e94fa76760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.714+0000 7f1800dcb740 1 --2- 172.21.15.107:0/1282240724 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 0x55e94fa8b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.714+0000 7f1800dcb740 1 -- 172.21.15.107:0/1282240724 >> 172.21.15.107:0/1282240724 conn(0x55e94fa70a60 msgr2=0x55e94fa70e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.714+0000 7f1800dcb740 1 -- 172.21.15.107:0/1282240724 shutdown_connections 2024-05-09T08:33:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.714+0000 7f1800dcb740 1 -- 172.21.15.107:0/1282240724 wait complete. 2024-05-09T08:33:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.714+0000 7f1800dcb740 1 -- start start 2024-05-09T08:33:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.714+0000 7f1800dcb740 1 Processor -- start 2024-05-09T08:33:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f1800dcb740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 0x55e94fa83ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f1800dcb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa75060 0x55e94fa84020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f1800dcb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa7f730 0x55e94faf4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f1800dcb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81aae90 con 0x55e94fa7f730 2024-05-09T08:33:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f1800dcb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81ac300 con 0x55e94fa75060 2024-05-09T08:33:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f1800dcb740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81ab560 con 0x55e94fa74740 2024-05-09T08:33:59.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f1800dcb740 10 client.0 _ll_register_callbacks cb 0x7f17e800bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:59.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f17effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa7f730 0x55e94faf4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f17effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa7f730 0x55e94faf4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33852/0 (socket says 172.21.15.107:33852) 2024-05-09T08:33:59.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f17effff640 1 -- 172.21.15.107:0/2314888882 learned_addr learned my addr 172.21.15.107:0/2314888882 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f17effff640 1 --2- 172.21.15.107:0/2314888882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 0x55e94fa83ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f17f77fe640 1 --2- 172.21.15.107:0/2314888882 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa75060 0x55e94fa84020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f17effff640 1 -- 172.21.15.107:0/2314888882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 msgr2=0x55e94fa83ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f17effff640 1 --2- 172.21.15.107:0/2314888882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 0x55e94fa83ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f17effff640 1 -- 172.21.15.107:0/2314888882 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa75060 msgr2=0x55e94fa84020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f17effff640 1 --2- 172.21.15.107:0/2314888882 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa75060 0x55e94fa84020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f17effff640 1 -- 172.21.15.107:0/2314888882 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e94faf54f0 con 0x55e94fa7f730 2024-05-09T08:33:59.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f17effff640 1 --2- 172.21.15.107:0/2314888882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa7f730 0x55e94faf4f30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f17e000e8a0 tx=0x7f17e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a34f3d26c1351555 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f17d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:59.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.715+0000 7f17d6ffd640 1 -- 172.21.15.107:0/2314888882 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e0021070 con 0x55e94fa7f730 2024-05-09T08:33:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.716+0000 7f17d6ffd640 1 -- 172.21.15.107:0/2314888882 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f17e00103d0 con 0x55e94fa7f730 2024-05-09T08:33:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.716+0000 7f1800dcb740 1 -- 172.21.15.107:0/2314888882 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17e001d820 con 0x55e94fa7f730 2024-05-09T08:33:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.716+0000 7f17d6ffd640 1 -- 172.21.15.107:0/2314888882 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f17e00105d0 con 0x55e94fa7f730 2024-05-09T08:33:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.716+0000 7f1800dcb740 10 client.15831 fetch_fsmap learned FSMap version 25 2024-05-09T08:33:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.716+0000 7f1800dcb740 1 -- 172.21.15.107:0/2314888882 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e94faf67d0 con 0x55e94fa7f730 2024-05-09T08:33:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.716+0000 7f17d6ffd640 1 -- 172.21.15.107:0/2314888882 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17e001d820 con 0x55e94fa7f730 2024-05-09T08:33:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.716+0000 7f17d6ffd640 1 -- 172.21.15.107:0/2314888882 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f17e000f040 con 0x55e94fa7f730 2024-05-09T08:33:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.716+0000 7f1800dcb740 10 client.15831 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:33:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.716+0000 7f1800dcb740 10 client.15831 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:33:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.716+0000 7f1800dcb740 20 client.15831 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.716+0000 7f1800dcb740 1 --2- 172.21.15.107:0/2314888882 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55e94faf6f80 0x55e94faf93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.716+0000 7f1800dcb740 4 client.15831 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:33:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.716+0000 7f1800dcb740 1 -- 172.21.15.107:0/2314888882 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e94f9dd6c0 con 0x55e94faf6f80 2024-05-09T08:33:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.716+0000 7f17f77fe640 1 --2- 172.21.15.107:0/2314888882 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55e94faf6f80 0x55e94faf93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f17f77fe640 1 --2- 172.21.15.107:0/2314888882 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55e94faf6f80 0x55e94faf93a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b69e2c87ba853fd3 server_cookie=48666b07d6611787 in_seq=0 out_seq=0 2024-05-09T08:33:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f17d6ffd640 10 client.15831 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:33:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f17d6ffd640 1 -- 172.21.15.107:0/2314888882 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e94f9dd6c0 con 0x55e94faf6f80 2024-05-09T08:33:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f17d6ffd640 10 client.15831 handle_command_reply: tid=0 2024-05-09T08:33:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f1800dcb740 1 client.15831 shutdown 2024-05-09T08:33:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f1800dcb740 1 -- 172.21.15.107:0/2314888882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa7f730 msgr2=0x55e94faf4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f1800dcb740 1 --2- 172.21.15.107:0/2314888882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa7f730 0x55e94faf4f30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f17e000e8a0 tx=0x7f17e00046c0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f1800dcb740 1 -- 172.21.15.107:0/2314888882 shutdown_connections 2024-05-09T08:33:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f1800dcb740 1 --2- 172.21.15.107:0/2314888882 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55e94faf6f80 0x55e94faf93a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f1800dcb740 1 --2- 172.21.15.107:0/2314888882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e94fa7f730 0x55e94faf4f30 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f17d6ffd640 0 client.15831 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:33:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f1800dcb740 1 --2- 172.21.15.107:0/2314888882 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e94fa75060 0x55e94fa84020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f1800dcb740 1 --2- 172.21.15.107:0/2314888882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e94fa74740 0x55e94fa83ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f1800dcb740 1 -- 172.21.15.107:0/2314888882 >> 172.21.15.107:0/2314888882 conn(0x55e94fa70a60 msgr2=0x55e94fa7cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f1800dcb740 1 -- 172.21.15.107:0/2314888882 shutdown_connections 2024-05-09T08:33:59.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f1800dcb740 1 -- 172.21.15.107:0/2314888882 wait complete. 2024-05-09T08:33:59.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.717+0000 7f1800dcb740 20 client.15831 trim_cache size 0 max 16384 2024-05-09T08:33:59.723 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:33:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.718+0000 7f17fdfe9640 1 -- 172.21.15.107:0/759512452 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f17e40a2e20 msgr2=0x7f17e40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.718+0000 7f17fdfe9640 1 --2- 172.21.15.107:0/759512452 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f17e40a2e20 0x7f17e40a52e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f17f00edec0 tx=0x7f17f00b87d0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.718+0000 7f17fdfe9640 1 -- 172.21.15.107:0/759512452 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17f8107820 msgr2=0x7f17f819c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.718+0000 7f17fdfe9640 1 --2- 172.21.15.107:0/759512452 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17f8107820 0x7f17f819c900 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f17f0005cc0 tx=0x7f17f0003c20 comp rx=0 tx=0).stop 2024-05-09T08:33:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.718+0000 7f17fdfe9640 1 -- 172.21.15.107:0/759512452 shutdown_connections 2024-05-09T08:33:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.718+0000 7f17fdfe9640 1 --2- 172.21.15.107:0/759512452 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f17e40a2e20 0x7f17e40a52e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.718+0000 7f17fdfe9640 1 --2- 172.21.15.107:0/759512452 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17f8107820 0x7f17f819c900 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.718+0000 7f17fdfe9640 1 --2- 172.21.15.107:0/759512452 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17f806a8b0 0x7f17f8198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.718+0000 7f17fdfe9640 1 --2- 172.21.15.107:0/759512452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17f8069ef0 0x7f17f8198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.718+0000 7f17fdfe9640 1 -- 172.21.15.107:0/759512452 >> 172.21.15.107:0/759512452 conn(0x7f17f8077790 msgr2=0x7f17f80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.718+0000 7f17fdfe9640 1 -- 172.21.15.107:0/759512452 shutdown_connections 2024-05-09T08:33:59.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.718+0000 7f17fdfe9640 1 -- 172.21.15.107:0/759512452 wait complete. 2024-05-09T08:33:59.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:33:59.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.834+0000 7f8996109640 1 -- 172.21.15.107:0/732736392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89901056e0 msgr2=0x7f8990105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.834+0000 7f8996109640 1 --2- 172.21.15.107:0/732736392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89901056e0 0x7f8990105ac0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8980009900 tx=0x7f898002a410 comp rx=0 tx=0).stop 2024-05-09T08:33:59.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.834+0000 7f8996109640 1 -- 172.21.15.107:0/732736392 shutdown_connections 2024-05-09T08:33:59.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.834+0000 7f8996109640 1 --2- 172.21.15.107:0/732736392 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8990101860 0x7f899010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.834+0000 7f8996109640 1 --2- 172.21.15.107:0/732736392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8990100ea0 0x7f8990101320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.834+0000 7f8996109640 1 --2- 172.21.15.107:0/732736392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89901056e0 0x7f8990105ac0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.834+0000 7f8996109640 1 -- 172.21.15.107:0/732736392 >> 172.21.15.107:0/732736392 conn(0x7f8990076cc0 msgr2=0x7f89900fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.834+0000 7f8996109640 1 -- 172.21.15.107:0/732736392 shutdown_connections 2024-05-09T08:33:59.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.834+0000 7f8996109640 1 -- 172.21.15.107:0/732736392 wait complete. 2024-05-09T08:33:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.834+0000 7f8996109640 1 Processor -- start 2024-05-09T08:33:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.834+0000 7f8996109640 1 -- start start 2024-05-09T08:33:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8996109640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8990100ea0 0x7f8990196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8996109640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8990101860 0x7f8990197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8996109640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89901056e0 0x7f899019b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8996109640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8990110990 con 0x7f8990101860 2024-05-09T08:33:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8996109640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8990110810 con 0x7f8990100ea0 2024-05-09T08:33:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8996109640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8990110b10 con 0x7f89901056e0 2024-05-09T08:33:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8987fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8990101860 0x7f8990197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8987fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8990101860 0x7f8990197430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33872/0 (socket says 172.21.15.107:33872) 2024-05-09T08:33:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8987fff640 1 -- 172.21.15.107:0/4257840863 learned_addr learned my addr 172.21.15.107:0/4257840863 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f898f7fe640 1 --2- 172.21.15.107:0/4257840863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8990100ea0 0x7f8990196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8987fff640 1 -- 172.21.15.107:0/4257840863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89901056e0 msgr2=0x7f899019b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8987fff640 1 --2- 172.21.15.107:0/4257840863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89901056e0 0x7f899019b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8987fff640 1 -- 172.21.15.107:0/4257840863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8990100ea0 msgr2=0x7f8990196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8987fff640 1 --2- 172.21.15.107:0/4257840863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8990100ea0 0x7f8990196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8987fff640 1 -- 172.21.15.107:0/4257840863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f899019be90 con 0x7f8990101860 2024-05-09T08:33:59.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f898f7fe640 1 --2- 172.21.15.107:0/4257840863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8990100ea0 0x7f8990196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:59.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8987fff640 1 --2- 172.21.15.107:0/4257840863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8990101860 0x7f8990197430 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f897800d950 tx=0x7f897800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f898d7fa640 1 -- 172.21.15.107:0/4257840863 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8978017070 con 0x7f8990101860 2024-05-09T08:33:59.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8996109640 1 -- 172.21.15.107:0/4257840863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f899019c180 con 0x7f8990101860 2024-05-09T08:33:59.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f898d7fa640 1 -- 172.21.15.107:0/4257840863 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8978009c30 con 0x7f8990101860 2024-05-09T08:33:59.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f898d7fa640 1 -- 172.21.15.107:0/4257840863 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897800f620 con 0x7f8990101860 2024-05-09T08:33:59.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.835+0000 7f8996109640 1 -- 172.21.15.107:0/4257840863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89901a3a10 con 0x7f8990101860 2024-05-09T08:33:59.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.838+0000 7f898d7fa640 1 -- 172.21.15.107:0/4257840863 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7f89780109a0 con 0x7f8990101860 2024-05-09T08:33:59.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.838+0000 7f8998e11740 1 Processor -- start 2024-05-09T08:33:59.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.838+0000 7f8998e11740 1 -- start start 2024-05-09T08:33:59.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f8998e11740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272040 0x563ca5272420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f8998e11740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272960 0x563ca52813e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f8998e11740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5281920 0x563ca5283d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f8998e11740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563ca51d7c40 con 0x563ca5272960 2024-05-09T08:33:59.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f8998e11740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563ca517f580 con 0x563ca5281920 2024-05-09T08:33:59.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f8998e11740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563ca51d3c00 con 0x563ca5272040 2024-05-09T08:33:59.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f898f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272960 0x563ca52813e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f898f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272960 0x563ca52813e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33880/0 (socket says 172.21.15.107:33880) 2024-05-09T08:33:59.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f898f7fe640 1 -- 172.21.15.107:0/4273453741 learned_addr learned my addr 172.21.15.107:0/4273453741 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f898ffff640 1 --2- 172.21.15.107:0/4273453741 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272040 0x563ca5272420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f8987fff640 1 --2- 172.21.15.107:0/4273453741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5281920 0x563ca5283d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f898d7fa640 1 --2- 172.21.15.107:0/4257840863 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f897c0a2e20 0x7f897c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f898f7fe640 1 -- 172.21.15.107:0/4273453741 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272040 msgr2=0x563ca5272420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f898f7fe640 1 --2- 172.21.15.107:0/4273453741 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272040 0x563ca5272420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f898f7fe640 1 -- 172.21.15.107:0/4273453741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5281920 msgr2=0x563ca5283d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f898f7fe640 1 --2- 172.21.15.107:0/4273453741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5281920 0x563ca5283d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f898f7fe640 1 -- 172.21.15.107:0/4273453741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ca4eeb3f0 con 0x563ca5272960 2024-05-09T08:33:59.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f8987fff640 1 --2- 172.21.15.107:0/4273453741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5281920 0x563ca5283d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:59.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f898ffff640 1 --2- 172.21.15.107:0/4257840863 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f897c0a2e20 0x7f897c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f898f7fe640 1 --2- 172.21.15.107:0/4273453741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272960 0x563ca52813e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f8980033000 tx=0x7f8980032e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97e3669145736fe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f898d7fa640 1 -- 172.21.15.107:0/4257840863 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f89780cf740 con 0x7f8990101860 2024-05-09T08:33:59.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.839+0000 7f8985ffb640 1 -- 172.21.15.107:0/4273453741 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898003f070 con 0x563ca5272960 2024-05-09T08:33:59.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.840+0000 7f8985ffb640 1 -- 172.21.15.107:0/4273453741 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8980031630 con 0x563ca5272960 2024-05-09T08:33:59.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.840+0000 7f8985ffb640 1 -- 172.21.15.107:0/4273453741 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8980035070 con 0x563ca5272960 2024-05-09T08:33:59.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.840+0000 7f8998e11740 1 -- 172.21.15.107:0/4273453741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272960 msgr2=0x563ca52813e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.840+0000 7f898ffff640 1 --2- 172.21.15.107:0/4257840863 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f897c0a2e20 0x7f897c0a52e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8988007750 tx=0x7f89880076e0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.840+0000 7f8998e11740 1 --2- 172.21.15.107:0/4273453741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272960 0x563ca52813e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f8980033000 tx=0x7f8980032e00 comp rx=0 tx=0).stop 2024-05-09T08:33:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.840+0000 7f8998e11740 1 -- 172.21.15.107:0/4273453741 shutdown_connections 2024-05-09T08:33:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.840+0000 7f8998e11740 1 --2- 172.21.15.107:0/4273453741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5281920 0x563ca5283d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.840+0000 7f8998e11740 1 --2- 172.21.15.107:0/4273453741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272960 0x563ca52813e0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.840+0000 7f8998e11740 1 --2- 172.21.15.107:0/4273453741 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272040 0x563ca5272420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.840+0000 7f8998e11740 1 -- 172.21.15.107:0/4273453741 >> 172.21.15.107:0/4273453741 conn(0x563ca526e650 msgr2=0x563ca526ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.840+0000 7f8998e11740 1 -- 172.21.15.107:0/4273453741 shutdown_connections 2024-05-09T08:33:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.840+0000 7f8998e11740 1 -- 172.21.15.107:0/4273453741 wait complete. 2024-05-09T08:33:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8998e11740 1 -- start start 2024-05-09T08:33:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8998e11740 1 Processor -- start 2024-05-09T08:33:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8998e11740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272040 0x563ca527e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8998e11740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272960 0x563ca527edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8998e11740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5281920 0x563ca527f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8998e11740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563ca52844f0 con 0x563ca5272960 2024-05-09T08:33:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8998e11740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563ca5192c40 con 0x563ca5281920 2024-05-09T08:33:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8998e11740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563ca5284670 con 0x563ca5272040 2024-05-09T08:33:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8987fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272960 0x563ca527edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8998e11740 10 client.0 _ll_register_callbacks cb 0x563ca4eeabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8987fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272960 0x563ca527edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33884/0 (socket says 172.21.15.107:33884) 2024-05-09T08:33:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8987fff640 1 -- 172.21.15.107:0/1212550490 learned_addr learned my addr 172.21.15.107:0/1212550490 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f898ffff640 1 --2- 172.21.15.107:0/1212550490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5281920 0x563ca527f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f898f7fe640 1 --2- 172.21.15.107:0/1212550490 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272040 0x563ca527e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8987fff640 1 -- 172.21.15.107:0/1212550490 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272040 msgr2=0x563ca527e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8987fff640 1 --2- 172.21.15.107:0/1212550490 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272040 0x563ca527e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8987fff640 1 -- 172.21.15.107:0/1212550490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5281920 msgr2=0x563ca527f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8987fff640 1 --2- 172.21.15.107:0/1212550490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5281920 0x563ca527f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f8987fff640 1 -- 172.21.15.107:0/1212550490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ca4fb57d0 con 0x563ca5272960 2024-05-09T08:33:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.841+0000 7f898f7fe640 1 --2- 172.21.15.107:0/1212550490 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272040 0x563ca527e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.842+0000 7f8987fff640 1 --2- 172.21.15.107:0/1212550490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272960 0x563ca527edd0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f89780cd130 tx=0x7f897809b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c265e1e654180e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.842+0000 7f896effd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.842+0000 7f896effd640 1 -- 172.21.15.107:0/1212550490 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8978020070 con 0x563ca5272960 2024-05-09T08:33:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.842+0000 7f896effd640 1 -- 172.21.15.107:0/1212550490 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f89780ccc90 con 0x563ca5272960 2024-05-09T08:33:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.842+0000 7f8998e11740 1 -- 172.21.15.107:0/1212550490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8980031630 con 0x563ca5272960 2024-05-09T08:33:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.842+0000 7f896effd640 1 -- 172.21.15.107:0/1212550490 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f89780860b0 con 0x563ca5272960 2024-05-09T08:33:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.842+0000 7f8998e11740 10 client.15843 fetch_fsmap learned FSMap version 25 2024-05-09T08:33:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.842+0000 7f8998e11740 1 -- 172.21.15.107:0/1212550490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563ca52906c0 con 0x563ca5272960 2024-05-09T08:33:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.842+0000 7f896effd640 1 -- 172.21.15.107:0/1212550490 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8980031630 con 0x563ca5272960 2024-05-09T08:33:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.842+0000 7f896effd640 1 -- 172.21.15.107:0/1212550490 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f89780bb050 con 0x563ca5272960 2024-05-09T08:33:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.842+0000 7f8998e11740 10 client.15843 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:33:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.842+0000 7f8998e11740 10 client.15843 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:33:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.842+0000 7f8998e11740 20 client.15843 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.842+0000 7f8998e11740 1 --2- 172.21.15.107:0/1212550490 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x563ca5291b80 0x563ca5293f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.842+0000 7f8998e11740 4 client.15843 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:33:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.842+0000 7f8998e11740 1 -- 172.21.15.107:0/1212550490 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ca51e5370 con 0x563ca5291b80 2024-05-09T08:33:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.843+0000 7f898f7fe640 1 --2- 172.21.15.107:0/1212550490 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x563ca5291b80 0x563ca5293f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.843+0000 7f896effd640 10 client.15843 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:33:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.843+0000 7f898f7fe640 1 --2- 172.21.15.107:0/1212550490 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x563ca5291b80 0x563ca5293f70 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c1f99837cb0dbe00 server_cookie=150758a8567e316e in_seq=0 out_seq=0 2024-05-09T08:33:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.844+0000 7f896effd640 1 -- 172.21.15.107:0/1212550490 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563ca51e5370 con 0x563ca5291b80 2024-05-09T08:33:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.844+0000 7f896effd640 10 client.15843 handle_command_reply: tid=0 2024-05-09T08:33:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.844+0000 7f8998e11740 1 client.15843 shutdown 2024-05-09T08:33:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.844+0000 7f8998e11740 1 -- 172.21.15.107:0/1212550490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272960 msgr2=0x563ca527edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.844+0000 7f8998e11740 1 --2- 172.21.15.107:0/1212550490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272960 0x563ca527edd0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f89780cd130 tx=0x7f897809b0d0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.844+0000 7f8998e11740 1 -- 172.21.15.107:0/1212550490 shutdown_connections 2024-05-09T08:33:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.844+0000 7f8998e11740 1 --2- 172.21.15.107:0/1212550490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5281920 0x563ca527f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.844+0000 7f8998e11740 1 --2- 172.21.15.107:0/1212550490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272960 0x563ca527edd0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.844+0000 7f8998e11740 1 --2- 172.21.15.107:0/1212550490 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x563ca5291b80 0x563ca5293f70 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.844+0000 7f8998e11740 1 --2- 172.21.15.107:0/1212550490 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272040 0x563ca527e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.844+0000 7f896effd640 0 client.15843 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:33:59.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.844+0000 7f8998e11740 1 -- 172.21.15.107:0/1212550490 >> 172.21.15.107:0/1212550490 conn(0x563ca526e650 msgr2=0x563ca52743b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.844+0000 7f8998e11740 1 -- 172.21.15.107:0/1212550490 shutdown_connections 2024-05-09T08:33:59.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.844+0000 7f8998e11740 1 -- 172.21.15.107:0/1212550490 wait complete. 2024-05-09T08:33:59.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.844+0000 7f8998e11740 20 client.15843 trim_cache size 0 max 16384 2024-05-09T08:33:59.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f8998e11740 1 Processor -- start 2024-05-09T08:33:59.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f8998e11740 1 -- start start 2024-05-09T08:33:59.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f8998e11740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272040 0x563ca528d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f8998e11740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272960 0x563ca527c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f8998e11740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5281920 0x563ca526d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f8998e11740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f89901af090 con 0x563ca5272040 2024-05-09T08:33:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f8998e11740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f89901ac960 con 0x563ca5281920 2024-05-09T08:33:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f8998e11740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f89901aa7b0 con 0x563ca5272960 2024-05-09T08:33:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f898f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272040 0x563ca528d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f898f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272040 0x563ca528d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33886/0 (socket says 172.21.15.107:33886) 2024-05-09T08:33:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f898f7fe640 1 -- 172.21.15.107:0/3022324366 learned_addr learned my addr 172.21.15.107:0/3022324366 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f898f7fe640 1 -- 172.21.15.107:0/3022324366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272960 msgr2=0x563ca527c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f8987fff640 1 --2- 172.21.15.107:0/3022324366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272960 0x563ca527c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f898f7fe640 1 --2- 172.21.15.107:0/3022324366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272960 0x563ca527c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f898f7fe640 1 -- 172.21.15.107:0/3022324366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5281920 msgr2=0x563ca526d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f898f7fe640 1 --2- 172.21.15.107:0/3022324366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5281920 0x563ca526d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f898f7fe640 1 -- 172.21.15.107:0/3022324366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ca52945a0 con 0x563ca5272040 2024-05-09T08:33:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f8987fff640 1 --2- 172.21.15.107:0/3022324366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272960 0x563ca527c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:59.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.859+0000 7f898f7fe640 1 --2- 172.21.15.107:0/3022324366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272040 0x563ca528d3f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8980005450 tx=0x7f8980005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=308670cf52977f8e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.860+0000 7f89857fa640 1 -- 172.21.15.107:0/3022324366 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898003f040 con 0x563ca5272040 2024-05-09T08:33:59.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.860+0000 7f89857fa640 1 -- 172.21.15.107:0/3022324366 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f89800317f0 con 0x563ca5272040 2024-05-09T08:33:59.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.860+0000 7f89857fa640 1 -- 172.21.15.107:0/3022324366 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8980035040 con 0x563ca5272040 2024-05-09T08:33:59.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.860+0000 7f8998e11740 1 -- 172.21.15.107:0/3022324366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272040 msgr2=0x563ca528d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.860+0000 7f8998e11740 1 --2- 172.21.15.107:0/3022324366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272040 0x563ca528d3f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8980005450 tx=0x7f8980005480 comp rx=0 tx=0).stop 2024-05-09T08:33:59.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.860+0000 7f8998e11740 1 -- 172.21.15.107:0/3022324366 shutdown_connections 2024-05-09T08:33:59.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.860+0000 7f8998e11740 1 --2- 172.21.15.107:0/3022324366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5281920 0x563ca526d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.860+0000 7f8998e11740 1 --2- 172.21.15.107:0/3022324366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272960 0x563ca527c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.860+0000 7f8998e11740 1 --2- 172.21.15.107:0/3022324366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5272040 0x563ca528d3f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.860+0000 7f8998e11740 1 -- 172.21.15.107:0/3022324366 >> 172.21.15.107:0/3022324366 conn(0x563ca526e650 msgr2=0x563ca526cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.860+0000 7f8998e11740 1 -- 172.21.15.107:0/3022324366 shutdown_connections 2024-05-09T08:33:59.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.860+0000 7f8998e11740 1 -- 172.21.15.107:0/3022324366 wait complete. 2024-05-09T08:33:59.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f8998e11740 1 -- start start 2024-05-09T08:33:59.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f8998e11740 1 Processor -- start 2024-05-09T08:33:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f8998e11740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5272040 0x563ca5289e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f8998e11740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272960 0x563ca5298d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f8998e11740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5281920 0x563ca5299480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f8998e11740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f89901aae20 con 0x563ca5281920 2024-05-09T08:33:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f8998e11740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f89901ac290 con 0x563ca5272040 2024-05-09T08:33:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f8998e11740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f89901ab4f0 con 0x563ca5272960 2024-05-09T08:33:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f8998e11740 10 client.0 _ll_register_callbacks cb 0x7f897809b1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f898ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5281920 0x563ca5299480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f898ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5281920 0x563ca5299480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33900/0 (socket says 172.21.15.107:33900) 2024-05-09T08:33:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f898ffff640 1 -- 172.21.15.107:0/2180232118 learned_addr learned my addr 172.21.15.107:0/2180232118 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f8987fff640 1 --2- 172.21.15.107:0/2180232118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272960 0x563ca5298d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f898f7fe640 1 --2- 172.21.15.107:0/2180232118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5272040 0x563ca5289e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f898ffff640 1 -- 172.21.15.107:0/2180232118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272960 msgr2=0x563ca5298d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f898ffff640 1 --2- 172.21.15.107:0/2180232118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272960 0x563ca5298d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f898ffff640 1 -- 172.21.15.107:0/2180232118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5272040 msgr2=0x563ca5289e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f898ffff640 1 --2- 172.21.15.107:0/2180232118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5272040 0x563ca5289e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f898ffff640 1 -- 172.21.15.107:0/2180232118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ca52f7040 con 0x563ca5281920 2024-05-09T08:33:59.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f8987fff640 1 --2- 172.21.15.107:0/2180232118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272960 0x563ca5298d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:33:59.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.861+0000 7f898f7fe640 1 --2- 172.21.15.107:0/2180232118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5272040 0x563ca5289e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:59.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.862+0000 7f898ffff640 1 --2- 172.21.15.107:0/2180232118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5281920 0x563ca5299480 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f89880119f0 tx=0x7f89880074a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4c32842fe07a928 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.862+0000 7f896effd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:59.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.862+0000 7f896effd640 1 -- 172.21.15.107:0/2180232118 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898802b060 con 0x563ca5281920 2024-05-09T08:33:59.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.862+0000 7f8998e11740 1 -- 172.21.15.107:0/2180232118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89800317f0 con 0x563ca5281920 2024-05-09T08:33:59.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.862+0000 7f896effd640 1 -- 172.21.15.107:0/2180232118 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f898801cd00 con 0x563ca5281920 2024-05-09T08:33:59.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.862+0000 7f896effd640 1 -- 172.21.15.107:0/2180232118 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8988032050 con 0x563ca5281920 2024-05-09T08:33:59.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.862+0000 7f8998e11740 10 client.15849 fetch_fsmap learned FSMap version 25 2024-05-09T08:33:59.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.862+0000 7f8998e11740 1 -- 172.21.15.107:0/2180232118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563ca52f8190 con 0x563ca5281920 2024-05-09T08:33:59.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.862+0000 7f896effd640 1 -- 172.21.15.107:0/2180232118 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f89800317f0 con 0x563ca5281920 2024-05-09T08:33:59.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.862+0000 7f896effd640 1 -- 172.21.15.107:0/2180232118 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8988025470 con 0x563ca5281920 2024-05-09T08:33:59.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.862+0000 7f8998e11740 10 client.15849 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:33:59.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.862+0000 7f8998e11740 10 client.15849 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:33:59.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.862+0000 7f8998e11740 20 client.15849 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:59.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.862+0000 7f8998e11740 1 --2- 172.21.15.107:0/2180232118 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x563ca52f8970 0x563ca52fad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.862+0000 7f8998e11740 4 client.15849 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:33:59.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.862+0000 7f8998e11740 1 -- 172.21.15.107:0/2180232118 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563ca51df6c0 con 0x563ca52f8970 2024-05-09T08:33:59.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f898f7fe640 1 --2- 172.21.15.107:0/2180232118 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x563ca52f8970 0x563ca52fad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f896effd640 10 client.15849 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:33:59.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f898f7fe640 1 --2- 172.21.15.107:0/2180232118 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x563ca52f8970 0x563ca52fad90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2f77ad13f629f7c5 server_cookie=937e62a14f038a05 in_seq=0 out_seq=0 2024-05-09T08:33:59.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f896effd640 1 -- 172.21.15.107:0/2180232118 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563ca51df6c0 con 0x563ca52f8970 2024-05-09T08:33:59.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f896effd640 10 client.15849 handle_command_reply: tid=0 2024-05-09T08:33:59.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f8998e11740 1 client.15849 shutdown 2024-05-09T08:33:59.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f8998e11740 1 -- 172.21.15.107:0/2180232118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5281920 msgr2=0x563ca5299480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f8998e11740 1 --2- 172.21.15.107:0/2180232118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5281920 0x563ca5299480 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f89880119f0 tx=0x7f89880074a0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f8998e11740 1 -- 172.21.15.107:0/2180232118 shutdown_connections 2024-05-09T08:33:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f8998e11740 1 --2- 172.21.15.107:0/2180232118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ca5281920 0x563ca5299480 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f8998e11740 1 --2- 172.21.15.107:0/2180232118 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x563ca52f8970 0x563ca52fad90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f8998e11740 1 --2- 172.21.15.107:0/2180232118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ca5272960 0x563ca5298d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f8998e11740 1 --2- 172.21.15.107:0/2180232118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ca5272040 0x563ca5289e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f896effd640 0 client.15849 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:33:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f8998e11740 1 -- 172.21.15.107:0/2180232118 >> 172.21.15.107:0/2180232118 conn(0x563ca526e650 msgr2=0x563ca526d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f8998e11740 1 -- 172.21.15.107:0/2180232118 shutdown_connections 2024-05-09T08:33:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.863+0000 7f8998e11740 1 -- 172.21.15.107:0/2180232118 wait complete. 2024-05-09T08:33:59.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.864+0000 7f8998e11740 20 client.15849 trim_cache size 0 max 16384 2024-05-09T08:33:59.869 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:33:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.864+0000 7f8996109640 1 -- 172.21.15.107:0/4257840863 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f897c0a2e20 msgr2=0x7f897c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.864+0000 7f8996109640 1 --2- 172.21.15.107:0/4257840863 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f897c0a2e20 0x7f897c0a52e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8988007750 tx=0x7f89880076e0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.864+0000 7f8996109640 1 -- 172.21.15.107:0/4257840863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8990101860 msgr2=0x7f8990197430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.864+0000 7f8996109640 1 --2- 172.21.15.107:0/4257840863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8990101860 0x7f8990197430 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f897800d950 tx=0x7f897800de10 comp rx=0 tx=0).stop 2024-05-09T08:33:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.864+0000 7f8996109640 1 -- 172.21.15.107:0/4257840863 shutdown_connections 2024-05-09T08:33:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.864+0000 7f8996109640 1 --2- 172.21.15.107:0/4257840863 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f897c0a2e20 0x7f897c0a52e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.864+0000 7f8996109640 1 --2- 172.21.15.107:0/4257840863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89901056e0 0x7f899019b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.864+0000 7f8996109640 1 --2- 172.21.15.107:0/4257840863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8990101860 0x7f8990197430 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.864+0000 7f8996109640 1 --2- 172.21.15.107:0/4257840863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8990100ea0 0x7f8990196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.864+0000 7f8996109640 1 -- 172.21.15.107:0/4257840863 >> 172.21.15.107:0/4257840863 conn(0x7f8990076cc0 msgr2=0x7f8990103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.864+0000 7f8996109640 1 -- 172.21.15.107:0/4257840863 shutdown_connections 2024-05-09T08:33:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.864+0000 7f8996109640 1 -- 172.21.15.107:0/4257840863 wait complete. 2024-05-09T08:33:59.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:33:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.981+0000 7fa1dba58640 1 -- 172.21.15.107:0/3303369656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d4102240 msgr2=0x7fa1d410e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.981+0000 7fa1dba58640 1 --2- 172.21.15.107:0/3303369656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d4102240 0x7fa1d410e750 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009cd0 tx=0x7fa1cc02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.981+0000 7fa1dba58640 1 -- 172.21.15.107:0/3303369656 shutdown_connections 2024-05-09T08:33:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.981+0000 7fa1dba58640 1 --2- 172.21.15.107:0/3303369656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d4102240 0x7fa1d410e750 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.981+0000 7fa1dba58640 1 --2- 172.21.15.107:0/3303369656 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1d4101880 0x7fa1d4101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.981+0000 7fa1dba58640 1 --2- 172.21.15.107:0/3303369656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1d4107890 0x7fa1d4107c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.981+0000 7fa1dba58640 1 -- 172.21.15.107:0/3303369656 >> 172.21.15.107:0/3303369656 conn(0x7fa1d40fd390 msgr2=0x7fa1d40ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.981+0000 7fa1dba58640 1 -- 172.21.15.107:0/3303369656 shutdown_connections 2024-05-09T08:33:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.981+0000 7fa1dba58640 1 -- 172.21.15.107:0/3303369656 wait complete. 2024-05-09T08:33:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.981+0000 7fa1dba58640 1 Processor -- start 2024-05-09T08:33:59.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.981+0000 7fa1dba58640 1 -- start start 2024-05-09T08:33:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.981+0000 7fa1dba58640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1d4101880 0x7fa1d4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.981+0000 7fa1dba58640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d4102240 0x7fa1d41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1dba58640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1d4107890 0x7fa1d419c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1dba58640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d4111b50 con 0x7fa1d4102240 2024-05-09T08:33:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1dba58640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d41119d0 con 0x7fa1d4107890 2024-05-09T08:33:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1dba58640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d4111cd0 con 0x7fa1d4101880 2024-05-09T08:33:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1d8ff3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d4102240 0x7fa1d41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1d8ff3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d4102240 0x7fa1d41985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33912/0 (socket says 172.21.15.107:33912) 2024-05-09T08:33:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1d8ff3640 1 -- 172.21.15.107:0/4118590516 learned_addr learned my addr 172.21.15.107:0/4118590516 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1d97f4640 1 --2- 172.21.15.107:0/4118590516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1d4101880 0x7fa1d4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1d8ff3640 1 -- 172.21.15.107:0/4118590516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1d4101880 msgr2=0x7fa1d4198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1d8ff3640 1 --2- 172.21.15.107:0/4118590516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1d4101880 0x7fa1d4198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1d8ff3640 1 -- 172.21.15.107:0/4118590516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1d4107890 msgr2=0x7fa1d419c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1d8ff3640 1 --2- 172.21.15.107:0/4118590516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1d4107890 0x7fa1d419c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1d8ff3640 1 -- 172.21.15.107:0/4118590516 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d419d080 con 0x7fa1d4102240 2024-05-09T08:33:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1d8ff3640 1 --2- 172.21.15.107:0/4118590516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d4102240 0x7fa1d41985d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400d770 tx=0x7fa1c400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1ca7fc640 1 -- 172.21.15.107:0/4118590516 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4017070 con 0x7fa1d4102240 2024-05-09T08:33:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1ca7fc640 1 -- 172.21.15.107:0/4118590516 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1c40054d0 con 0x7fa1d4102240 2024-05-09T08:33:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1ca7fc640 1 -- 172.21.15.107:0/4118590516 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4004cf0 con 0x7fa1d4102240 2024-05-09T08:33:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1dba58640 1 -- 172.21.15.107:0/4118590516 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1d419d370 con 0x7fa1d4102240 2024-05-09T08:33:59.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.982+0000 7fa1dba58640 1 -- 172.21.15.107:0/4118590516 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1d41a4c00 con 0x7fa1d4102240 2024-05-09T08:33:59.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.985+0000 7fa1de766740 1 Processor -- start 2024-05-09T08:33:59.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.985+0000 7fa1ca7fc640 1 -- 172.21.15.107:0/4118590516 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7fa1c4004e90 con 0x7fa1d4102240 2024-05-09T08:33:59.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.985+0000 7fa1de766740 1 -- start start 2024-05-09T08:33:59.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.985+0000 7fa1de766740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835330 0x55a8d0835710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.985+0000 7fa1de766740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0835c50 0x55a8d083fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.985+0000 7fa1de766740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d0842710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.985+0000 7fa1de766740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a8d079ac40 con 0x55a8d0840320 2024-05-09T08:33:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.985+0000 7fa1de766740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a8d0742580 con 0x55a8d0835c50 2024-05-09T08:33:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.985+0000 7fa1de766740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a8d0796c00 con 0x55a8d0835330 2024-05-09T08:33:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.985+0000 7fa1d8ff3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d0842710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.985+0000 7fa1d8ff3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d0842710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33924/0 (socket says 172.21.15.107:33924) 2024-05-09T08:33:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.985+0000 7fa1d8ff3640 1 -- 172.21.15.107:0/1898965416 learned_addr learned my addr 172.21.15.107:0/1898965416 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.985+0000 7fa1d97f4640 1 --2- 172.21.15.107:0/1898965416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0835c50 0x55a8d083fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.985+0000 7fa1d9ff5640 1 --2- 172.21.15.107:0/1898965416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835330 0x55a8d0835710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.986+0000 7fa1d8ff3640 1 -- 172.21.15.107:0/1898965416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835330 msgr2=0x55a8d0835710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.986+0000 7fa1d8ff3640 1 --2- 172.21.15.107:0/1898965416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835330 0x55a8d0835710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.986+0000 7fa1d8ff3640 1 -- 172.21.15.107:0/1898965416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0835c50 msgr2=0x55a8d083fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.986+0000 7fa1ca7fc640 1 --2- 172.21.15.107:0/4118590516 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa1c00a2e20 0x7fa1c00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.986+0000 7fa1d8ff3640 1 --2- 172.21.15.107:0/1898965416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0835c50 0x55a8d083fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.986+0000 7fa1d8ff3640 1 -- 172.21.15.107:0/1898965416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8d04ae3f0 con 0x55a8d0840320 2024-05-09T08:33:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.986+0000 7fa1d97f4640 1 --2- 172.21.15.107:0/1898965416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0835c50 0x55a8d083fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:33:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.986+0000 7fa1d9ff5640 1 --2- 172.21.15.107:0/4118590516 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa1c00a2e20 0x7fa1c00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.986+0000 7fa1ca7fc640 1 -- 172.21.15.107:0/4118590516 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fa1c40cf580 con 0x7fa1d4102240 2024-05-09T08:33:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.986+0000 7fa1d8ff3640 1 --2- 172.21.15.107:0/1898965416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d0842710 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa1c40cc900 tx=0x7fa1c4097850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937eb699fa13f95a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.986+0000 7fa1bf7fe640 1 -- 172.21.15.107:0/1898965416 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4020070 con 0x55a8d0840320 2024-05-09T08:33:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.986+0000 7fa1bf7fe640 1 -- 172.21.15.107:0/1898965416 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1c4009c30 con 0x55a8d0840320 2024-05-09T08:33:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.986+0000 7fa1bf7fe640 1 -- 172.21.15.107:0/1898965416 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c409b030 con 0x55a8d0840320 2024-05-09T08:33:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.987+0000 7fa1de766740 1 -- 172.21.15.107:0/1898965416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 msgr2=0x55a8d0842710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.987+0000 7fa1de766740 1 --2- 172.21.15.107:0/1898965416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d0842710 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa1c40cc900 tx=0x7fa1c4097850 comp rx=0 tx=0).stop 2024-05-09T08:33:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.987+0000 7fa1d9ff5640 1 --2- 172.21.15.107:0/4118590516 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa1c00a2e20 0x7fa1c00a52e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009ca0 tx=0x7fa1cc005f30 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.987+0000 7fa1de766740 1 -- 172.21.15.107:0/1898965416 shutdown_connections 2024-05-09T08:33:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.987+0000 7fa1de766740 1 --2- 172.21.15.107:0/1898965416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d0842710 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.987+0000 7fa1de766740 1 --2- 172.21.15.107:0/1898965416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0835c50 0x55a8d083fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.987+0000 7fa1de766740 1 --2- 172.21.15.107:0/1898965416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835330 0x55a8d0835710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.987+0000 7fa1de766740 1 -- 172.21.15.107:0/1898965416 >> 172.21.15.107:0/1898965416 conn(0x55a8d0831650 msgr2=0x55a8d0831a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.987+0000 7fa1de766740 1 -- 172.21.15.107:0/1898965416 shutdown_connections 2024-05-09T08:33:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.987+0000 7fa1de766740 1 -- 172.21.15.107:0/1898965416 wait complete. 2024-05-09T08:33:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.987+0000 7fa1de766740 1 -- start start 2024-05-09T08:33:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.987+0000 7fa1de766740 1 Processor -- start 2024-05-09T08:33:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1de766740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0835330 0x55a8d083d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1de766740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835c50 0x55a8d083d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1de766740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0840320 0x55a8d083dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1de766740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a8d0843110 con 0x55a8d0835330 2024-05-09T08:33:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1de766740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a8d0755c40 con 0x55a8d0840320 2024-05-09T08:33:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1de766740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a8d0843290 con 0x55a8d0835c50 2024-05-09T08:33:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1de766740 10 client.0 _ll_register_callbacks cb 0x55a8d04adbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:33:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1d97f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0835330 0x55a8d083d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1d97f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0835330 0x55a8d083d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33932/0 (socket says 172.21.15.107:33932) 2024-05-09T08:33:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1d97f4640 1 -- 172.21.15.107:0/1561412475 learned_addr learned my addr 172.21.15.107:0/1561412475 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:33:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1d8ff3640 1 --2- 172.21.15.107:0/1561412475 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835c50 0x55a8d083d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1d9ff5640 1 --2- 172.21.15.107:0/1561412475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0840320 0x55a8d083dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1d97f4640 1 -- 172.21.15.107:0/1561412475 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835c50 msgr2=0x55a8d083d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1d97f4640 1 --2- 172.21.15.107:0/1561412475 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835c50 0x55a8d083d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1d97f4640 1 -- 172.21.15.107:0/1561412475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0840320 msgr2=0x55a8d083dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:33:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1d97f4640 1 --2- 172.21.15.107:0/1561412475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0840320 0x55a8d083dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1d97f4640 1 -- 172.21.15.107:0/1561412475 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8d05787d0 con 0x55a8d0835330 2024-05-09T08:33:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1d9ff5640 1 --2- 172.21.15.107:0/1561412475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0840320 0x55a8d083dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:33:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1d97f4640 1 --2- 172.21.15.107:0/1561412475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0835330 0x55a8d083d260 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa1d000cce0 tx=0x7fa1d00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7adeecaf97904128 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:33:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:33:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1bcff9640 1 -- 172.21.15.107:0/1561412475 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0010840 con 0x55a8d0835330 2024-05-09T08:33:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1bcff9640 1 -- 172.21.15.107:0/1561412475 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fa1d00109e0 con 0x55a8d0835330 2024-05-09T08:33:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.988+0000 7fa1de766740 1 -- 172.21.15.107:0/1561412475 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a8d07a8370 con 0x55a8d0835330 2024-05-09T08:33:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.989+0000 7fa1bcff9640 1 -- 172.21.15.107:0/1561412475 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa1d0010be0 con 0x55a8d0835330 2024-05-09T08:33:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.989+0000 7fa1de766740 10 client.15861 fetch_fsmap learned FSMap version 25 2024-05-09T08:33:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.989+0000 7fa1de766740 1 -- 172.21.15.107:0/1561412475 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a8d084f360 con 0x55a8d0835330 2024-05-09T08:33:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.989+0000 7fa1bcff9640 1 -- 172.21.15.107:0/1561412475 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a8d07a8370 con 0x55a8d0835330 2024-05-09T08:33:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.989+0000 7fa1bcff9640 1 -- 172.21.15.107:0/1561412475 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fa1d0018020 con 0x55a8d0835330 2024-05-09T08:33:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.989+0000 7fa1de766740 10 client.15861 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:33:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.989+0000 7fa1de766740 10 client.15861 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:33:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.989+0000 7fa1de766740 20 client.15861 populate_metadata read hostname 'smithi107' 2024-05-09T08:33:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.989+0000 7fa1de766740 1 --2- 172.21.15.107:0/1561412475 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a8d0850870 0x55a8d0852cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:33:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.989+0000 7fa1de766740 4 client.15861 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:33:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.989+0000 7fa1de766740 1 -- 172.21.15.107:0/1561412475 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8d07a26c0 con 0x55a8d0850870 2024-05-09T08:33:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.989+0000 7fa1d8ff3640 1 --2- 172.21.15.107:0/1561412475 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a8d0850870 0x55a8d0852cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:33:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.990+0000 7fa1d8ff3640 1 --2- 172.21.15.107:0/1561412475 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a8d0850870 0x55a8d0852cc0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=447d6806f87cf3e0 server_cookie=21dd9d133170c687 in_seq=0 out_seq=0 2024-05-09T08:33:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.990+0000 7fa1bcff9640 10 client.15861 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.991+0000 7fa1bcff9640 1 -- 172.21.15.107:0/1561412475 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a8d07a26c0 con 0x55a8d0850870 2024-05-09T08:33:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.991+0000 7fa1bcff9640 10 client.15861 handle_command_reply: tid=0 2024-05-09T08:33:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.991+0000 7fa1de766740 1 client.15861 shutdown 2024-05-09T08:33:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.991+0000 7fa1de766740 1 -- 172.21.15.107:0/1561412475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0835330 msgr2=0x55a8d083d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:33:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.991+0000 7fa1de766740 1 --2- 172.21.15.107:0/1561412475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0835330 0x55a8d083d260 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa1d000cce0 tx=0x7fa1d00039c0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.991+0000 7fa1de766740 1 -- 172.21.15.107:0/1561412475 shutdown_connections 2024-05-09T08:33:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.991+0000 7fa1de766740 1 --2- 172.21.15.107:0/1561412475 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a8d0850870 0x55a8d0852cc0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.991+0000 7fa1de766740 1 --2- 172.21.15.107:0/1561412475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0840320 0x55a8d083dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.991+0000 7fa1bcff9640 0 client.15861 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:33:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.991+0000 7fa1de766740 1 --2- 172.21.15.107:0/1561412475 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835c50 0x55a8d083d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.991+0000 7fa1de766740 1 --2- 172.21.15.107:0/1561412475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0835330 0x55a8d083d260 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:33:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.991+0000 7fa1de766740 1 -- 172.21.15.107:0/1561412475 >> 172.21.15.107:0/1561412475 conn(0x55a8d0831650 msgr2=0x55a8d0837450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:33:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.991+0000 7fa1de766740 1 -- 172.21.15.107:0/1561412475 shutdown_connections 2024-05-09T08:33:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.991+0000 7fa1de766740 1 -- 172.21.15.107:0/1561412475 wait complete. 2024-05-09T08:33:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:33:59.991+0000 7fa1de766740 20 client.15861 trim_cache size 0 max 16384 2024-05-09T08:34:00.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.004+0000 7fa1de766740 1 Processor -- start 2024-05-09T08:34:00.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.004+0000 7fa1de766740 1 -- start start 2024-05-09T08:34:00.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.004+0000 7fa1de766740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0835330 0x55a8d083f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:00.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.004+0000 7fa1de766740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835c50 0x55a8d084c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:00.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.004+0000 7fa1de766740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d0837a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:00.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.004+0000 7fa1de766740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d41b1400 con 0x55a8d0840320 2024-05-09T08:34:00.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.004+0000 7fa1de766740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d41aecd0 con 0x55a8d0835330 2024-05-09T08:34:00.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.004+0000 7fa1de766740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d41acb20 con 0x55a8d0835c50 2024-05-09T08:34:00.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.004+0000 7fa1d9ff5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d0837a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.005+0000 7fa1d8ff3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835c50 0x55a8d084c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.005+0000 7fa1d9ff5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d0837a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33940/0 (socket says 172.21.15.107:33940) 2024-05-09T08:34:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.005+0000 7fa1d9ff5640 1 -- 172.21.15.107:0/3641415103 learned_addr learned my addr 172.21.15.107:0/3641415103 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.005+0000 7fa1d97f4640 1 --2- 172.21.15.107:0/3641415103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0835330 0x55a8d083f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.005+0000 7fa1d9ff5640 1 -- 172.21.15.107:0/3641415103 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835c50 msgr2=0x55a8d084c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.005+0000 7fa1d9ff5640 1 --2- 172.21.15.107:0/3641415103 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835c50 0x55a8d084c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.005+0000 7fa1d9ff5640 1 -- 172.21.15.107:0/3641415103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0835330 msgr2=0x55a8d083f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.005+0000 7fa1d9ff5640 1 --2- 172.21.15.107:0/3641415103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0835330 0x55a8d083f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.005+0000 7fa1d9ff5640 1 -- 172.21.15.107:0/3641415103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8d0835a80 con 0x55a8d0840320 2024-05-09T08:34:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.005+0000 7fa1d9ff5640 1 --2- 172.21.15.107:0/3641415103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d0837a90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc031730 tx=0x7fa1cc03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a1f52c6b46a2f62 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.005+0000 7fa1beffd640 1 -- 172.21.15.107:0/3641415103 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc004000 con 0x55a8d0840320 2024-05-09T08:34:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.005+0000 7fa1beffd640 1 -- 172.21.15.107:0/3641415103 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1cc036040 con 0x55a8d0840320 2024-05-09T08:34:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.005+0000 7fa1beffd640 1 -- 172.21.15.107:0/3641415103 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc030d50 con 0x55a8d0840320 2024-05-09T08:34:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.006+0000 7fa1de766740 1 -- 172.21.15.107:0/3641415103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 msgr2=0x55a8d0837a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.006+0000 7fa1de766740 1 --2- 172.21.15.107:0/3641415103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d0837a90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc031730 tx=0x7fa1cc03f000 comp rx=0 tx=0).stop 2024-05-09T08:34:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.006+0000 7fa1de766740 1 -- 172.21.15.107:0/3641415103 shutdown_connections 2024-05-09T08:34:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.006+0000 7fa1de766740 1 --2- 172.21.15.107:0/3641415103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d0837a90 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.006+0000 7fa1de766740 1 --2- 172.21.15.107:0/3641415103 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835c50 0x55a8d084c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.006+0000 7fa1de766740 1 --2- 172.21.15.107:0/3641415103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0835330 0x55a8d083f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.006+0000 7fa1de766740 1 -- 172.21.15.107:0/3641415103 >> 172.21.15.107:0/3641415103 conn(0x55a8d0831650 msgr2=0x55a8d083c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.006+0000 7fa1de766740 1 -- 172.21.15.107:0/3641415103 shutdown_connections 2024-05-09T08:34:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.006+0000 7fa1de766740 1 -- 172.21.15.107:0/3641415103 wait complete. 2024-05-09T08:34:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.006+0000 7fa1de766740 1 -- start start 2024-05-09T08:34:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.006+0000 7fa1de766740 1 Processor -- start 2024-05-09T08:34:00.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1de766740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0835330 0x55a8d0848700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:00.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1de766740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835c50 0x55a8d0839af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:00.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1de766740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d083a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:00.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1de766740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d41ad190 con 0x55a8d0840320 2024-05-09T08:34:00.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1de766740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d41ae600 con 0x55a8d0835330 2024-05-09T08:34:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1de766740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d41ad860 con 0x55a8d0835c50 2024-05-09T08:34:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1de766740 10 client.0 _ll_register_callbacks cb 0x7fa1d0004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1d9ff5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d083a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1d9ff5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d083a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33952/0 (socket says 172.21.15.107:33952) 2024-05-09T08:34:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1d9ff5640 1 -- 172.21.15.107:0/3992038649 learned_addr learned my addr 172.21.15.107:0/3992038649 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1d97f4640 1 --2- 172.21.15.107:0/3992038649 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0835330 0x55a8d0848700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1d8ff3640 1 --2- 172.21.15.107:0/3992038649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835c50 0x55a8d0839af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1d9ff5640 1 -- 172.21.15.107:0/3992038649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835c50 msgr2=0x55a8d0839af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1d9ff5640 1 --2- 172.21.15.107:0/3992038649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835c50 0x55a8d0839af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1d9ff5640 1 -- 172.21.15.107:0/3992038649 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0835330 msgr2=0x55a8d0848700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1d9ff5640 1 --2- 172.21.15.107:0/3992038649 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0835330 0x55a8d0848700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1d9ff5640 1 -- 172.21.15.107:0/3992038649 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8d083a5f0 con 0x55a8d0840320 2024-05-09T08:34:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1d8ff3640 1 --2- 172.21.15.107:0/3992038649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835c50 0x55a8d0839af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:34:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1d9ff5640 1 --2- 172.21.15.107:0/3992038649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d083a030 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009cd0 tx=0x7fa1cc030f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47168bfc9627fc01 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1bcff9640 1 -- 172.21.15.107:0/3992038649 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc058020 con 0x55a8d0840320 2024-05-09T08:34:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1bcff9640 1 -- 172.21.15.107:0/3992038649 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fa1cc004410 con 0x55a8d0840320 2024-05-09T08:34:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.007+0000 7fa1de766740 1 -- 172.21.15.107:0/3992038649 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1cc036040 con 0x55a8d0840320 2024-05-09T08:34:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.008+0000 7fa1bcff9640 1 -- 172.21.15.107:0/3992038649 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa1cc063050 con 0x55a8d0840320 2024-05-09T08:34:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.008+0000 7fa1de766740 10 client.15867 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.008+0000 7fa1de766740 1 -- 172.21.15.107:0/3992038649 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a8d08b6bf0 con 0x55a8d0840320 2024-05-09T08:34:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.008+0000 7fa1bcff9640 1 -- 172.21.15.107:0/3992038649 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1cc036040 con 0x55a8d0840320 2024-05-09T08:34:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.008+0000 7fa1bcff9640 1 -- 172.21.15.107:0/3992038649 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fa1cc030070 con 0x55a8d0840320 2024-05-09T08:34:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.008+0000 7fa1de766740 10 client.15867 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.008+0000 7fa1de766740 10 client.15867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.008+0000 7fa1de766740 20 client.15867 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.008+0000 7fa1de766740 1 --2- 172.21.15.107:0/3992038649 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a8d08b72e0 0x55a8d08b9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.008+0000 7fa1de766740 4 client.15867 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:34:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.008+0000 7fa1de766740 1 -- 172.21.15.107:0/3992038649 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a8d071f820 con 0x55a8d08b72e0 2024-05-09T08:34:00.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.008+0000 7fa1d97f4640 1 --2- 172.21.15.107:0/3992038649 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a8d08b72e0 0x55a8d08b9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:00.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1d97f4640 1 --2- 172.21.15.107:0/3992038649 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a8d08b72e0 0x55a8d08b9700 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=455e748462ee87e8 server_cookie=fd46644d3bb07cd6 in_seq=0 out_seq=0 2024-05-09T08:34:00.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1bcff9640 10 client.15867 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:00.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1bcff9640 1 -- 172.21.15.107:0/3992038649 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a8d071f820 con 0x55a8d08b72e0 2024-05-09T08:34:00.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1bcff9640 10 client.15867 handle_command_reply: tid=0 2024-05-09T08:34:00.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1de766740 1 client.15867 shutdown 2024-05-09T08:34:00.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1de766740 1 -- 172.21.15.107:0/3992038649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 msgr2=0x55a8d083a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:00.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1de766740 1 --2- 172.21.15.107:0/3992038649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d083a030 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009cd0 tx=0x7fa1cc030f40 comp rx=0 tx=0).stop 2024-05-09T08:34:00.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1de766740 1 -- 172.21.15.107:0/3992038649 shutdown_connections 2024-05-09T08:34:00.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1de766740 1 --2- 172.21.15.107:0/3992038649 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a8d08b72e0 0x55a8d08b9700 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:00.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1de766740 1 --2- 172.21.15.107:0/3992038649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8d0840320 0x55a8d083a030 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:00.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1de766740 1 --2- 172.21.15.107:0/3992038649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8d0835c50 0x55a8d0839af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:00.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1bcff9640 0 client.15867 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:00.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1de766740 1 --2- 172.21.15.107:0/3992038649 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8d0835330 0x55a8d0848700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:00.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1de766740 1 -- 172.21.15.107:0/3992038649 >> 172.21.15.107:0/3992038649 conn(0x55a8d0831650 msgr2=0x55a8d083d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:00.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1de766740 1 -- 172.21.15.107:0/3992038649 shutdown_connections 2024-05-09T08:34:00.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1de766740 1 -- 172.21.15.107:0/3992038649 wait complete. 2024-05-09T08:34:00.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.009+0000 7fa1de766740 20 client.15867 trim_cache size 0 max 16384 2024-05-09T08:34:00.015 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:34:00.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.010+0000 7fa1dba58640 1 -- 172.21.15.107:0/4118590516 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa1c00a2e20 msgr2=0x7fa1c00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:00.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.010+0000 7fa1dba58640 1 --2- 172.21.15.107:0/4118590516 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa1c00a2e20 0x7fa1c00a52e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009ca0 tx=0x7fa1cc005f30 comp rx=0 tx=0).stop 2024-05-09T08:34:00.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.010+0000 7fa1dba58640 1 -- 172.21.15.107:0/4118590516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d4102240 msgr2=0x7fa1d41985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:00.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.010+0000 7fa1dba58640 1 --2- 172.21.15.107:0/4118590516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d4102240 0x7fa1d41985d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400d770 tx=0x7fa1c400dc30 comp rx=0 tx=0).stop 2024-05-09T08:34:00.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.010+0000 7fa1dba58640 1 -- 172.21.15.107:0/4118590516 shutdown_connections 2024-05-09T08:34:00.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.010+0000 7fa1dba58640 1 --2- 172.21.15.107:0/4118590516 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa1c00a2e20 0x7fa1c00a52e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:00.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.010+0000 7fa1dba58640 1 --2- 172.21.15.107:0/4118590516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1d4107890 0x7fa1d419c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:00.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.010+0000 7fa1dba58640 1 --2- 172.21.15.107:0/4118590516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1d4102240 0x7fa1d41985d0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:00.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.010+0000 7fa1dba58640 1 --2- 172.21.15.107:0/4118590516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1d4101880 0x7fa1d4198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:00.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.010+0000 7fa1dba58640 1 -- 172.21.15.107:0/4118590516 >> 172.21.15.107:0/4118590516 conn(0x7fa1d40fd390 msgr2=0x7fa1d40fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:00.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.010+0000 7fa1dba58640 1 -- 172.21.15.107:0/4118590516 shutdown_connections 2024-05-09T08:34:00.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:00.010+0000 7fa1dba58640 1 -- 172.21.15.107:0/4118590516 wait complete. 2024-05-09T08:34:00.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:00 smithi107 ceph-mon[30122]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-09T08:34:00.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:00 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/157479097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:34:00.779 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:34:00 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:34:00.602+0000 7fe21cf26700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:34:00.779 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:34:00 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:34:00.602+0000 7fe21cf26700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:34:00.779 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:34:00 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:34:00.602+0000 7fe21cf26700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:34:00.779 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:34:00 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:34:00.602+0000 7fe21cf26700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:34:00.779 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:34:00 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:34:00.602+0000 7fe21cf26700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T08:34:00.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:00 smithi157 ceph-mon[33614]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-09T08:34:00.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:00 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/157479097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:34:00.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:00 smithi110 ceph-mon[33622]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-09T08:34:00.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:00 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/157479097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:34:01.021 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:34:01.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:34:01.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:34:01.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.129+0000 7fc0f38f7640 1 -- 172.21.15.107:0/152551847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ec100700 msgr2=0x7fc0ec100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.129+0000 7fc0f38f7640 1 --2- 172.21.15.107:0/152551847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ec100700 0x7fc0ec100b80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc009930 tx=0x7fc0dc02a420 comp rx=0 tx=0).stop 2024-05-09T08:34:01.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.129+0000 7fc0f38f7640 1 -- 172.21.15.107:0/152551847 shutdown_connections 2024-05-09T08:34:01.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.129+0000 7fc0f38f7640 1 --2- 172.21.15.107:0/152551847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0ec1010c0 0x7fc0ec10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.129+0000 7fc0f38f7640 1 --2- 172.21.15.107:0/152551847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ec100700 0x7fc0ec100b80 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.129+0000 7fc0f38f7640 1 --2- 172.21.15.107:0/152551847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc0ec106710 0x7fc0ec106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.129+0000 7fc0f38f7640 1 -- 172.21.15.107:0/152551847 >> 172.21.15.107:0/152551847 conn(0x7fc0ec077830 msgr2=0x7fc0ec077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:01.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.129+0000 7fc0f38f7640 1 -- 172.21.15.107:0/152551847 shutdown_connections 2024-05-09T08:34:01.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.129+0000 7fc0f38f7640 1 -- 172.21.15.107:0/152551847 wait complete. 2024-05-09T08:34:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f38f7640 1 Processor -- start 2024-05-09T08:34:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f38f7640 1 -- start start 2024-05-09T08:34:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f38f7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ec100700 0x7fc0ec10e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f38f7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc0ec1010c0 0x7fc0ec10e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f38f7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0ec106710 0x7fc0ec10eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f38f7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec111ad0 con 0x7fc0ec100700 2024-05-09T08:34:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f38f7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec111950 con 0x7fc0ec106710 2024-05-09T08:34:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f38f7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec111c50 con 0x7fc0ec1010c0 2024-05-09T08:34:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f1693640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ec100700 0x7fc0ec10e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f1693640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ec100700 0x7fc0ec10e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46864/0 (socket says 172.21.15.107:46864) 2024-05-09T08:34:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f1693640 1 -- 172.21.15.107:0/4231080690 learned_addr learned my addr 172.21.15.107:0/4231080690 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f0e92640 1 --2- 172.21.15.107:0/4231080690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc0ec1010c0 0x7fc0ec10e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f1e94640 1 --2- 172.21.15.107:0/4231080690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0ec106710 0x7fc0ec10eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f1693640 1 -- 172.21.15.107:0/4231080690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc0ec1010c0 msgr2=0x7fc0ec10e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f1693640 1 --2- 172.21.15.107:0/4231080690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc0ec1010c0 0x7fc0ec10e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f1693640 1 -- 172.21.15.107:0/4231080690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0ec106710 msgr2=0x7fc0ec10eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f1693640 1 --2- 172.21.15.107:0/4231080690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0ec106710 0x7fc0ec10eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f1693640 1 -- 172.21.15.107:0/4231080690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0ec1a6520 con 0x7fc0ec100700 2024-05-09T08:34:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.130+0000 7fc0f0e92640 1 --2- 172.21.15.107:0/4231080690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc0ec1010c0 0x7fc0ec10e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.131+0000 7fc0f1693640 1 --2- 172.21.15.107:0/4231080690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ec100700 0x7fc0ec10e060 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc0e800d950 tx=0x7fc0e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.131+0000 7fc0e27fc640 1 -- 172.21.15.107:0/4231080690 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8017030 con 0x7fc0ec100700 2024-05-09T08:34:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.131+0000 7fc0e27fc640 1 -- 172.21.15.107:0/4231080690 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0e8004020 con 0x7fc0ec100700 2024-05-09T08:34:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.131+0000 7fc0f38f7640 1 -- 172.21.15.107:0/4231080690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0ec1a6810 con 0x7fc0ec100700 2024-05-09T08:34:01.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.131+0000 7fc0e27fc640 1 -- 172.21.15.107:0/4231080690 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e80047a0 con 0x7fc0ec100700 2024-05-09T08:34:01.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.131+0000 7fc0f38f7640 1 -- 172.21.15.107:0/4231080690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0ec1a6d30 con 0x7fc0ec100700 2024-05-09T08:34:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.133+0000 7fc0e27fc640 1 -- 172.21.15.107:0/4231080690 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7fc0e80041d0 con 0x7fc0ec100700 2024-05-09T08:34:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.134+0000 7fc0f65fd740 1 Processor -- start 2024-05-09T08:34:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.134+0000 7fc0f65fd740 1 -- start start 2024-05-09T08:34:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.134+0000 7fc0f65fd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576405480b0 0x5576405463f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.134+0000 7fc0f65fd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405469c0 0x557640546e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.134+0000 7fc0f65fd740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557640551740 0x557640553b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.134+0000 7fc0f65fd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5576404ab990 con 0x5576405469c0 2024-05-09T08:34:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.134+0000 7fc0f65fd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5576404535b0 con 0x5576405480b0 2024-05-09T08:34:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.134+0000 7fc0e27fc640 1 --2- 172.21.15.107:0/4231080690 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc0d80a2e20 0x7fc0d80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.134+0000 7fc0f65fd740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5576404a7950 con 0x557640551740 2024-05-09T08:34:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.134+0000 7fc0f1693640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405469c0 0x557640546e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.134+0000 7fc0f1e94640 1 --2- 172.21.15.107:0/4231080690 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc0d80a2e20 0x7fc0d80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.134+0000 7fc0f1e94640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576405480b0 0x5576405463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.134+0000 7fc0f1693640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405469c0 0x557640546e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46868/0 (socket says 172.21.15.107:46868) 2024-05-09T08:34:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.134+0000 7fc0f1693640 1 -- 172.21.15.107:0/3564154408 learned_addr learned my addr 172.21.15.107:0/3564154408 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.134+0000 7fc0e27fc640 1 -- 172.21.15.107:0/4231080690 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fc0e80cf490 con 0x7fc0ec100700 2024-05-09T08:34:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.134+0000 7fc0f0e92640 1 --2- 172.21.15.107:0/3564154408 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557640551740 0x557640553b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.135+0000 7fc0f1693640 1 -- 172.21.15.107:0/3564154408 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557640551740 msgr2=0x557640553b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:01.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.135+0000 7fc0f1693640 1 --2- 172.21.15.107:0/3564154408 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557640551740 0x557640553b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.135+0000 7fc0f1693640 1 -- 172.21.15.107:0/3564154408 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576405480b0 msgr2=0x5576405463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:01.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.135+0000 7fc0f1693640 1 --2- 172.21.15.107:0/3564154408 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576405480b0 0x5576405463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.135+0000 7fc0f1693640 1 -- 172.21.15.107:0/3564154408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576402877e0 con 0x5576405469c0 2024-05-09T08:34:01.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.135+0000 7fc0f1e94640 1 --2- 172.21.15.107:0/3564154408 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576405480b0 0x5576405463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:01.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.135+0000 7fc0f1e94640 1 --2- 172.21.15.107:0/4231080690 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc0d80a2e20 0x7fc0d80a52e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4005fd0 tx=0x7fc0e4009f00 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:01.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.135+0000 7fc0f1693640 1 --2- 172.21.15.107:0/3564154408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405469c0 0x557640546e40 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc0e80cce80 tx=0x7fc0e80c5f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c877a5812df6fd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:01.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.135+0000 7fc0d77fe640 1 -- 172.21.15.107:0/3564154408 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e809af40 con 0x5576405469c0 2024-05-09T08:34:01.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.135+0000 7fc0d77fe640 1 -- 172.21.15.107:0/3564154408 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0e80ad070 con 0x5576405469c0 2024-05-09T08:34:01.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.135+0000 7fc0d77fe640 1 -- 172.21.15.107:0/3564154408 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e809ad60 con 0x5576405469c0 2024-05-09T08:34:01.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.136+0000 7fc0f65fd740 1 -- 172.21.15.107:0/3564154408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405469c0 msgr2=0x557640546e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.136+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/3564154408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405469c0 0x557640546e40 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc0e80cce80 tx=0x7fc0e80c5f90 comp rx=0 tx=0).stop 2024-05-09T08:34:01.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.136+0000 7fc0f65fd740 1 -- 172.21.15.107:0/3564154408 shutdown_connections 2024-05-09T08:34:01.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.136+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/3564154408 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557640551740 0x557640553b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.136+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/3564154408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405469c0 0x557640546e40 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.136+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/3564154408 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576405480b0 0x5576405463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.136+0000 7fc0f65fd740 1 -- 172.21.15.107:0/3564154408 >> 172.21.15.107:0/3564154408 conn(0x55764050f0a0 msgr2=0x557640542f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:01.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.136+0000 7fc0f65fd740 1 -- 172.21.15.107:0/3564154408 shutdown_connections 2024-05-09T08:34:01.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.136+0000 7fc0f65fd740 1 -- 172.21.15.107:0/3564154408 wait complete. 2024-05-09T08:34:01.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.136+0000 7fc0f65fd740 1 -- start start 2024-05-09T08:34:01.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.136+0000 7fc0f65fd740 1 Processor -- start 2024-05-09T08:34:01.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f65fd740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576405469c0 0x55764054e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f65fd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576405480b0 0x55764054ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f65fd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557640551740 0x55764054f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f65fd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557640554570 con 0x557640551740 2024-05-09T08:34:01.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f65fd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557640466b40 con 0x5576405480b0 2024-05-09T08:34:01.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f65fd740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5576405546f0 con 0x5576405469c0 2024-05-09T08:34:01.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f65fd740 10 client.0 _ll_register_callbacks cb 0x55764045b0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:01.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f1e94640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557640551740 0x55764054f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f1e94640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557640551740 0x55764054f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46870/0 (socket says 172.21.15.107:46870) 2024-05-09T08:34:01.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f1e94640 1 -- 172.21.15.107:0/1165808632 learned_addr learned my addr 172.21.15.107:0/1165808632 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:01.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f1693640 1 --2- 172.21.15.107:0/1165808632 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576405469c0 0x55764054e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f1e94640 1 -- 172.21.15.107:0/1165808632 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576405469c0 msgr2=0x55764054e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:01.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f1e94640 1 --2- 172.21.15.107:0/1165808632 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576405469c0 0x55764054e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f1e94640 1 -- 172.21.15.107:0/1165808632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576405480b0 msgr2=0x55764054ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:01.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f1e94640 1 --2- 172.21.15.107:0/1165808632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576405480b0 0x55764054ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f1e94640 1 -- 172.21.15.107:0/1165808632 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576402e7f70 con 0x557640551740 2024-05-09T08:34:01.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f1e94640 1 --2- 172.21.15.107:0/1165808632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557640551740 0x55764054f110 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4003a90 tx=0x7fc0e4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc98c6ead1f72349 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:01.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:01.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0d4ff9640 1 -- 172.21.15.107:0/1165808632 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4024070 con 0x557640551740 2024-05-09T08:34:01.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0d4ff9640 1 -- 172.21.15.107:0/1165808632 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fc0e4029460 con 0x557640551740 2024-05-09T08:34:01.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.137+0000 7fc0f65fd740 1 -- 172.21.15.107:0/1165808632 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0e80ad070 con 0x557640551740 2024-05-09T08:34:01.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.138+0000 7fc0d4ff9640 1 -- 172.21.15.107:0/1165808632 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc0e4032050 con 0x557640551740 2024-05-09T08:34:01.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.138+0000 7fc0f65fd740 10 client.15885 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:01.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.138+0000 7fc0f65fd740 1 -- 172.21.15.107:0/1165808632 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55764017f170 con 0x557640551740 2024-05-09T08:34:01.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.138+0000 7fc0d4ff9640 1 -- 172.21.15.107:0/1165808632 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0e80ad070 con 0x557640551740 2024-05-09T08:34:01.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.138+0000 7fc0d4ff9640 1 -- 172.21.15.107:0/1165808632 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc0e40203f0 con 0x557640551740 2024-05-09T08:34:01.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.138+0000 7fc0f65fd740 10 client.15885 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:01.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.138+0000 7fc0f65fd740 10 client.15885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:01.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.138+0000 7fc0f65fd740 20 client.15885 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:01.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.138+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/1165808632 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557640561800 0x557640563c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.138+0000 7fc0f65fd740 4 client.15885 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:34:01.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.138+0000 7fc0f65fd740 1 -- 172.21.15.107:0/1165808632 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576404b90c0 con 0x557640561800 2024-05-09T08:34:01.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.138+0000 7fc0f1693640 1 --2- 172.21.15.107:0/1165808632 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557640561800 0x557640563c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.139+0000 7fc0d4ff9640 10 client.15885 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:01.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.139+0000 7fc0f1693640 1 --2- 172.21.15.107:0/1165808632 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557640561800 0x557640563c50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=877c71a95824ac5e server_cookie=7a7696581538434c in_seq=0 out_seq=0 2024-05-09T08:34:01.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.139+0000 7fc0d4ff9640 1 -- 172.21.15.107:0/1165808632 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5576404b90c0 con 0x557640561800 2024-05-09T08:34:01.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.139+0000 7fc0d4ff9640 10 client.15885 handle_command_reply: tid=0 2024-05-09T08:34:01.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.139+0000 7fc0f65fd740 1 client.15885 shutdown 2024-05-09T08:34:01.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.140+0000 7fc0f65fd740 1 -- 172.21.15.107:0/1165808632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557640551740 msgr2=0x55764054f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.140+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/1165808632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557640551740 0x55764054f110 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4003a90 tx=0x7fc0e4009500 comp rx=0 tx=0).stop 2024-05-09T08:34:01.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.140+0000 7fc0f65fd740 1 -- 172.21.15.107:0/1165808632 shutdown_connections 2024-05-09T08:34:01.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.140+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/1165808632 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557640561800 0x557640563c50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.140+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/1165808632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557640551740 0x55764054f110 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.140+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/1165808632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576405480b0 0x55764054ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.140+0000 7fc0d4ff9640 0 client.15885 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:01.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.140+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/1165808632 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576405469c0 0x55764054e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.140+0000 7fc0f65fd740 1 -- 172.21.15.107:0/1165808632 >> 172.21.15.107:0/1165808632 conn(0x55764050f0a0 msgr2=0x557640548d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:01.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.140+0000 7fc0f65fd740 1 -- 172.21.15.107:0/1165808632 shutdown_connections 2024-05-09T08:34:01.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.140+0000 7fc0f65fd740 1 -- 172.21.15.107:0/1165808632 wait complete. 2024-05-09T08:34:01.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.140+0000 7fc0f65fd740 20 client.15885 trim_cache size 0 max 16384 2024-05-09T08:34:01.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.158+0000 7fc0f65fd740 1 Processor -- start 2024-05-09T08:34:01.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.158+0000 7fc0f65fd740 1 -- start start 2024-05-09T08:34:01.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.158+0000 7fc0f65fd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405469c0 0x55764055d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.158+0000 7fc0f65fd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576405480b0 0x557640541f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.158+0000 7fc0f65fd740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557640551740 0x557640569520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.158+0000 7fc0f65fd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec1b3640 con 0x5576405469c0 2024-05-09T08:34:01.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.158+0000 7fc0f65fd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec1b0f10 con 0x5576405480b0 2024-05-09T08:34:01.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.158+0000 7fc0f65fd740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec1aed60 con 0x557640551740 2024-05-09T08:34:01.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.158+0000 7fc0f1693640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405469c0 0x55764055d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.158+0000 7fc0f1693640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405469c0 0x55764055d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46882/0 (socket says 172.21.15.107:46882) 2024-05-09T08:34:01.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.158+0000 7fc0f1693640 1 -- 172.21.15.107:0/2201808102 learned_addr learned my addr 172.21.15.107:0/2201808102 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:01.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.158+0000 7fc0f1e94640 1 --2- 172.21.15.107:0/2201808102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557640551740 0x557640569520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.158+0000 7fc0f1693640 1 -- 172.21.15.107:0/2201808102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557640551740 msgr2=0x557640569520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:01.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.158+0000 7fc0f1693640 1 --2- 172.21.15.107:0/2201808102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557640551740 0x557640569520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.158+0000 7fc0f0e92640 1 --2- 172.21.15.107:0/2201808102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576405480b0 0x557640541f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.158+0000 7fc0f1693640 1 -- 172.21.15.107:0/2201808102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576405480b0 msgr2=0x557640541f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:01.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.159+0000 7fc0f1693640 1 --2- 172.21.15.107:0/2201808102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576405480b0 0x557640541f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.159+0000 7fc0f1693640 1 -- 172.21.15.107:0/2201808102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557640543130 con 0x5576405469c0 2024-05-09T08:34:01.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.159+0000 7fc0f1e94640 1 --2- 172.21.15.107:0/2201808102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557640551740 0x557640569520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:01.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.159+0000 7fc0f1693640 1 --2- 172.21.15.107:0/2201808102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405469c0 0x55764055d470 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8097dd0 tx=0x7fc0e80bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7bb2e0564aae7be server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:01.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.159+0000 7fc0d6ffd640 1 -- 172.21.15.107:0/2201808102 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e809ad60 con 0x5576405469c0 2024-05-09T08:34:01.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.159+0000 7fc0d6ffd640 1 -- 172.21.15.107:0/2201808102 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0e80ad040 con 0x5576405469c0 2024-05-09T08:34:01.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.159+0000 7fc0d6ffd640 1 -- 172.21.15.107:0/2201808102 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e809af40 con 0x5576405469c0 2024-05-09T08:34:01.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.160+0000 7fc0f65fd740 1 -- 172.21.15.107:0/2201808102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405469c0 msgr2=0x55764055d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.160+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/2201808102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405469c0 0x55764055d470 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8097dd0 tx=0x7fc0e80bb040 comp rx=0 tx=0).stop 2024-05-09T08:34:01.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.160+0000 7fc0f65fd740 1 -- 172.21.15.107:0/2201808102 shutdown_connections 2024-05-09T08:34:01.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.160+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/2201808102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557640551740 0x557640569520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.160+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/2201808102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576405480b0 0x557640541f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.160+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/2201808102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405469c0 0x55764055d470 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.160+0000 7fc0f65fd740 1 -- 172.21.15.107:0/2201808102 >> 172.21.15.107:0/2201808102 conn(0x55764050f0a0 msgr2=0x5576405417f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:01.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.160+0000 7fc0f65fd740 1 -- 172.21.15.107:0/2201808102 shutdown_connections 2024-05-09T08:34:01.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.160+0000 7fc0f65fd740 1 -- 172.21.15.107:0/2201808102 wait complete. 2024-05-09T08:34:01.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.160+0000 7fc0f65fd740 1 -- start start 2024-05-09T08:34:01.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.160+0000 7fc0f65fd740 1 Processor -- start 2024-05-09T08:34:01.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.160+0000 7fc0f65fd740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576405469c0 0x557640559980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.160+0000 7fc0f65fd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405480b0 0x557640559ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.160+0000 7fc0f65fd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557640551740 0x557640556950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.160+0000 7fc0f65fd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec1af3d0 con 0x5576405480b0 2024-05-09T08:34:01.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f65fd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec1b0840 con 0x557640551740 2024-05-09T08:34:01.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f65fd740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec1afaa0 con 0x5576405469c0 2024-05-09T08:34:01.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f65fd740 10 client.0 _ll_register_callbacks cb 0x7fc0e4003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:01.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f0e92640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405480b0 0x557640559ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f0e92640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405480b0 0x557640559ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46890/0 (socket says 172.21.15.107:46890) 2024-05-09T08:34:01.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f0e92640 1 -- 172.21.15.107:0/3891528017 learned_addr learned my addr 172.21.15.107:0/3891528017 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:01.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f1e94640 1 --2- 172.21.15.107:0/3891528017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557640551740 0x557640556950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f1693640 1 --2- 172.21.15.107:0/3891528017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576405469c0 0x557640559980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f0e92640 1 -- 172.21.15.107:0/3891528017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576405469c0 msgr2=0x557640559980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:01.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f0e92640 1 --2- 172.21.15.107:0/3891528017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576405469c0 0x557640559980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f0e92640 1 -- 172.21.15.107:0/3891528017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557640551740 msgr2=0x557640556950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:01.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f0e92640 1 --2- 172.21.15.107:0/3891528017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557640551740 0x557640556950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f0e92640 1 -- 172.21.15.107:0/3891528017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557640557180 con 0x5576405480b0 2024-05-09T08:34:01.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f0e92640 1 --2- 172.21.15.107:0/3891528017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405480b0 0x557640559ec0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc02a940 tx=0x7fc0dc0316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa834ec67b4714f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:01.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:01.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0d4ff9640 1 -- 172.21.15.107:0/3891528017 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc035070 con 0x5576405480b0 2024-05-09T08:34:01.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0d4ff9640 1 -- 172.21.15.107:0/3891528017 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fc0dc004230 con 0x5576405480b0 2024-05-09T08:34:01.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f65fd740 1 -- 172.21.15.107:0/3891528017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0e80ad040 con 0x5576405480b0 2024-05-09T08:34:01.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0d4ff9640 1 -- 172.21.15.107:0/3891528017 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc0dc04d050 con 0x5576405480b0 2024-05-09T08:34:01.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f65fd740 10 client.15891 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:01.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.161+0000 7fc0f65fd740 1 -- 172.21.15.107:0/3891528017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5576405c88c0 con 0x5576405480b0 2024-05-09T08:34:01.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.162+0000 7fc0d4ff9640 1 -- 172.21.15.107:0/3891528017 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0e80ad040 con 0x5576405480b0 2024-05-09T08:34:01.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.162+0000 7fc0d4ff9640 1 -- 172.21.15.107:0/3891528017 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc0dc004bd0 con 0x5576405480b0 2024-05-09T08:34:01.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.162+0000 7fc0f65fd740 10 client.15891 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:01.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.162+0000 7fc0f65fd740 10 client.15891 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:01.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.162+0000 7fc0f65fd740 20 client.15891 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:01.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.162+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/3891528017 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557640561800 0x5576405cb100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.162+0000 7fc0f65fd740 4 client.15891 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:34:01.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.162+0000 7fc0f65fd740 1 -- 172.21.15.107:0/3891528017 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5576404b3410 con 0x557640561800 2024-05-09T08:34:01.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.162+0000 7fc0f1693640 1 --2- 172.21.15.107:0/3891528017 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557640561800 0x5576405cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.162+0000 7fc0d4ff9640 10 client.15891 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:01.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.162+0000 7fc0f1693640 1 --2- 172.21.15.107:0/3891528017 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557640561800 0x5576405cb100 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=436f04c54d89ff1 server_cookie=42c46c62d03f070e in_seq=0 out_seq=0 2024-05-09T08:34:01.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.164+0000 7fc0d4ff9640 1 -- 172.21.15.107:0/3891528017 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5576404b3410 con 0x557640561800 2024-05-09T08:34:01.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.164+0000 7fc0d4ff9640 10 client.15891 handle_command_reply: tid=0 2024-05-09T08:34:01.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.164+0000 7fc0f65fd740 1 client.15891 shutdown 2024-05-09T08:34:01.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.164+0000 7fc0f65fd740 1 -- 172.21.15.107:0/3891528017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405480b0 msgr2=0x557640559ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.164+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/3891528017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405480b0 0x557640559ec0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc02a940 tx=0x7fc0dc0316d0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.164+0000 7fc0f65fd740 1 -- 172.21.15.107:0/3891528017 shutdown_connections 2024-05-09T08:34:01.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.164+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/3891528017 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557640561800 0x5576405cb100 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.164+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/3891528017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557640551740 0x557640556950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.164+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/3891528017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576405480b0 0x557640559ec0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.164+0000 7fc0d4ff9640 0 client.15891 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:01.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.164+0000 7fc0f65fd740 1 --2- 172.21.15.107:0/3891528017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576405469c0 0x557640559980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.164+0000 7fc0f65fd740 1 -- 172.21.15.107:0/3891528017 >> 172.21.15.107:0/3891528017 conn(0x55764050f0a0 msgr2=0x557640542080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:01.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.164+0000 7fc0f65fd740 1 -- 172.21.15.107:0/3891528017 shutdown_connections 2024-05-09T08:34:01.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.164+0000 7fc0f65fd740 1 -- 172.21.15.107:0/3891528017 wait complete. 2024-05-09T08:34:01.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.164+0000 7fc0f65fd740 20 client.15891 trim_cache size 0 max 16384 2024-05-09T08:34:01.171 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:34:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:34:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "c0c547ab-e6d9-42fd-a0b0-f85a016e7635", 2024-05-09T08:34:01.171 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:34:01.171 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:34:01.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.165+0000 7fc0f38f7640 1 -- 172.21.15.107:0/4231080690 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc0d80a2e20 msgr2=0x7fc0d80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.165+0000 7fc0f38f7640 1 --2- 172.21.15.107:0/4231080690 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc0d80a2e20 0x7fc0d80a52e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4005fd0 tx=0x7fc0e4009f00 comp rx=0 tx=0).stop 2024-05-09T08:34:01.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.165+0000 7fc0f38f7640 1 -- 172.21.15.107:0/4231080690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ec100700 msgr2=0x7fc0ec10e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.165+0000 7fc0f38f7640 1 --2- 172.21.15.107:0/4231080690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ec100700 0x7fc0ec10e060 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc0e800d950 tx=0x7fc0e800de10 comp rx=0 tx=0).stop 2024-05-09T08:34:01.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.165+0000 7fc0f38f7640 1 -- 172.21.15.107:0/4231080690 shutdown_connections 2024-05-09T08:34:01.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.165+0000 7fc0f38f7640 1 --2- 172.21.15.107:0/4231080690 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc0d80a2e20 0x7fc0d80a52e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.165+0000 7fc0f38f7640 1 --2- 172.21.15.107:0/4231080690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0ec106710 0x7fc0ec10eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.165+0000 7fc0f38f7640 1 --2- 172.21.15.107:0/4231080690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc0ec1010c0 0x7fc0ec10e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.165+0000 7fc0f38f7640 1 --2- 172.21.15.107:0/4231080690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ec100700 0x7fc0ec10e060 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.165+0000 7fc0f38f7640 1 -- 172.21.15.107:0/4231080690 >> 172.21.15.107:0/4231080690 conn(0x7fc0ec077830 msgr2=0x7fc0ec103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:01.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.165+0000 7fc0f38f7640 1 -- 172.21.15.107:0/4231080690 shutdown_connections 2024-05-09T08:34:01.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.165+0000 7fc0f38f7640 1 -- 172.21.15.107:0/4231080690 wait complete. 2024-05-09T08:34:01.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:34:01.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:34:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.281+0000 7f046b26b640 1 -- 172.21.15.107:0/2102625253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0464100700 msgr2=0x7f0464100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.281+0000 7f046b26b640 1 --2- 172.21.15.107:0/2102625253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0464100700 0x7f0464100b80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f0454009930 tx=0x7f045402a420 comp rx=0 tx=0).stop 2024-05-09T08:34:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.281+0000 7f046b26b640 1 -- 172.21.15.107:0/2102625253 shutdown_connections 2024-05-09T08:34:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.281+0000 7f046b26b640 1 --2- 172.21.15.107:0/2102625253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04641010c0 0x7f046410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.281+0000 7f046b26b640 1 --2- 172.21.15.107:0/2102625253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0464100700 0x7f0464100b80 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.281+0000 7f046b26b640 1 --2- 172.21.15.107:0/2102625253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0464106710 0x7f0464106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.281+0000 7f046b26b640 1 -- 172.21.15.107:0/2102625253 >> 172.21.15.107:0/2102625253 conn(0x7f0464077830 msgr2=0x7f0464077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:01.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.281+0000 7f046b26b640 1 -- 172.21.15.107:0/2102625253 shutdown_connections 2024-05-09T08:34:01.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.281+0000 7f046b26b640 1 -- 172.21.15.107:0/2102625253 wait complete. 2024-05-09T08:34:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f046b26b640 1 Processor -- start 2024-05-09T08:34:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f046b26b640 1 -- start start 2024-05-09T08:34:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f046b26b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0464100700 0x7f0464196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f046b26b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04641010c0 0x7f0464197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f046b26b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0464106710 0x7f046419b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f046b26b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f04641109d0 con 0x7f0464100700 2024-05-09T08:34:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f046b26b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0464110850 con 0x7f04641010c0 2024-05-09T08:34:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f046b26b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0464110b50 con 0x7f0464106710 2024-05-09T08:34:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f0469007640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0464100700 0x7f0464196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f0469007640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0464100700 0x7f0464196f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46912/0 (socket says 172.21.15.107:46912) 2024-05-09T08:34:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f0469007640 1 -- 172.21.15.107:0/413229612 learned_addr learned my addr 172.21.15.107:0/413229612 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f0469808640 1 --2- 172.21.15.107:0/413229612 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0464106710 0x7f046419b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f0469007640 1 -- 172.21.15.107:0/413229612 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0464106710 msgr2=0x7f046419b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f0468806640 1 --2- 172.21.15.107:0/413229612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04641010c0 0x7f0464197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f0469007640 1 --2- 172.21.15.107:0/413229612 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0464106710 0x7f046419b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f0469007640 1 -- 172.21.15.107:0/413229612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04641010c0 msgr2=0x7f0464197480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f0469007640 1 --2- 172.21.15.107:0/413229612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04641010c0 0x7f0464197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f0469007640 1 -- 172.21.15.107:0/413229612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046419bf30 con 0x7f0464100700 2024-05-09T08:34:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f0469808640 1 --2- 172.21.15.107:0/413229612 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0464106710 0x7f046419b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:34:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.282+0000 7f0469007640 1 --2- 172.21.15.107:0/413229612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0464100700 0x7f0464196f40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f046000e9f0 tx=0x7f046000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.283+0000 7f045a7fc640 1 -- 172.21.15.107:0/413229612 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046000cd80 con 0x7f0464100700 2024-05-09T08:34:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.283+0000 7f045a7fc640 1 -- 172.21.15.107:0/413229612 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f04600054d0 con 0x7f0464100700 2024-05-09T08:34:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.283+0000 7f045a7fc640 1 -- 172.21.15.107:0/413229612 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460010490 con 0x7f0464100700 2024-05-09T08:34:01.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.283+0000 7f046b26b640 1 -- 172.21.15.107:0/413229612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f046419c220 con 0x7f0464100700 2024-05-09T08:34:01.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.283+0000 7f046b26b640 1 -- 172.21.15.107:0/413229612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04641a3ab0 con 0x7f0464100700 2024-05-09T08:34:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.285+0000 7f046df79740 1 Processor -- start 2024-05-09T08:34:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f046df79740 1 -- start start 2024-05-09T08:34:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f045a7fc640 1 -- 172.21.15.107:0/413229612 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 138061+0+0 (secure 0 0 0) 0x7f0460005020 con 0x7f0464100700 2024-05-09T08:34:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f046df79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc6b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f046df79740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc77f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f046df79740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc7a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f046df79740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fbbbbd0d00 con 0x55fbbbc78480 2024-05-09T08:34:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f046df79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55fbbbb784c0 con 0x55fbbbc6aef0 2024-05-09T08:34:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f046df79740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55fbbbbcccc0 con 0x55fbbbc6b8a0 2024-05-09T08:34:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f0468806640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc7a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f0468806640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc7a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46926/0 (socket says 172.21.15.107:46926) 2024-05-09T08:34:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f0468806640 1 -- 172.21.15.107:0/1192057996 learned_addr learned my addr 172.21.15.107:0/1192057996 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f0469808640 1 --2- 172.21.15.107:0/1192057996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc6b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f0469007640 1 --2- 172.21.15.107:0/1192057996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc77f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f0468806640 1 -- 172.21.15.107:0/1192057996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6b8a0 msgr2=0x55fbbbc77f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f0468806640 1 --2- 172.21.15.107:0/1192057996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc77f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f0468806640 1 -- 172.21.15.107:0/1192057996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6aef0 msgr2=0x55fbbbc6b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f0468806640 1 --2- 172.21.15.107:0/1192057996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc6b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f0468806640 1 -- 172.21.15.107:0/1192057996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbbb9ae7d0 con 0x55fbbbc78480 2024-05-09T08:34:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f0469808640 1 --2- 172.21.15.107:0/1192057996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc6b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.286+0000 7f045a7fc640 1 --2- 172.21.15.107:0/413229612 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f04500a2ea0 0x7f04500a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.287+0000 7f0468806640 1 --2- 172.21.15.107:0/1192057996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc7a870 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f0454006010 tx=0x7f0454032b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39fbb6de91577697 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.287+0000 7f0469808640 1 --2- 172.21.15.107:0/413229612 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f04500a2ea0 0x7f04500a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.287+0000 7f045a7fc640 1 -- 172.21.15.107:0/413229612 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f04600cfe40 con 0x7f0464100700 2024-05-09T08:34:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.287+0000 7f044f7fe640 1 -- 172.21.15.107:0/1192057996 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045403f070 con 0x55fbbbc78480 2024-05-09T08:34:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.287+0000 7f044f7fe640 1 -- 172.21.15.107:0/1192057996 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0454004030 con 0x55fbbbc78480 2024-05-09T08:34:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.287+0000 7f044f7fe640 1 -- 172.21.15.107:0/1192057996 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0454035070 con 0x55fbbbc78480 2024-05-09T08:34:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.288+0000 7f046df79740 1 -- 172.21.15.107:0/1192057996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 msgr2=0x55fbbbc7a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.288+0000 7f0469808640 1 --2- 172.21.15.107:0/413229612 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f04500a2ea0 0x7f04500a5360 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f045c005fd0 tx=0x7f045c0086a0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.288+0000 7f046df79740 1 --2- 172.21.15.107:0/1192057996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc7a870 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f0454006010 tx=0x7f0454032b00 comp rx=0 tx=0).stop 2024-05-09T08:34:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.288+0000 7f046df79740 1 -- 172.21.15.107:0/1192057996 shutdown_connections 2024-05-09T08:34:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.288+0000 7f046df79740 1 --2- 172.21.15.107:0/1192057996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc7a870 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.288+0000 7f046df79740 1 --2- 172.21.15.107:0/1192057996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc77f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.288+0000 7f046df79740 1 --2- 172.21.15.107:0/1192057996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc6b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.288+0000 7f046df79740 1 -- 172.21.15.107:0/1192057996 >> 172.21.15.107:0/1192057996 conn(0x55fbbbc675a0 msgr2=0x55fbbbc67980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.288+0000 7f046df79740 1 -- 172.21.15.107:0/1192057996 shutdown_connections 2024-05-09T08:34:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.288+0000 7f046df79740 1 -- 172.21.15.107:0/1192057996 wait complete. 2024-05-09T08:34:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.288+0000 7f046df79740 1 -- start start 2024-05-09T08:34:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.288+0000 7f046df79740 1 Processor -- start 2024-05-09T08:34:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f046df79740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc753e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f046df79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc75920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f046df79740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc75e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f046df79740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fbbbc7b210 con 0x55fbbbc78480 2024-05-09T08:34:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f046df79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55fbbbb8bb90 con 0x55fbbbc6b8a0 2024-05-09T08:34:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f046df79740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55fbbbc7b390 con 0x55fbbbc6aef0 2024-05-09T08:34:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f046df79740 10 client.0 _ll_register_callbacks cb 0x55fbbbbc7da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f0469808640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc75e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f0469808640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc75e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46942/0 (socket says 172.21.15.107:46942) 2024-05-09T08:34:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f0469808640 1 -- 172.21.15.107:0/1211763003 learned_addr learned my addr 172.21.15.107:0/1211763003 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f0469007640 1 --2- 172.21.15.107:0/1211763003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f0468806640 1 --2- 172.21.15.107:0/1211763003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc75920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f0469808640 1 -- 172.21.15.107:0/1211763003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6aef0 msgr2=0x55fbbbc753e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f0469808640 1 --2- 172.21.15.107:0/1211763003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc753e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f0469808640 1 -- 172.21.15.107:0/1211763003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6b8a0 msgr2=0x55fbbbc75920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f0469808640 1 --2- 172.21.15.107:0/1211763003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc75920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f0469808640 1 -- 172.21.15.107:0/1211763003 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbbb8e43f0 con 0x55fbbbc78480 2024-05-09T08:34:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f0469007640 1 --2- 172.21.15.107:0/1211763003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc753e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f0468806640 1 --2- 172.21.15.107:0/1211763003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc75920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:34:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f0469808640 1 --2- 172.21.15.107:0/1211763003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc75e60 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f045c00ef60 tx=0x7f045c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35be7550ff1716d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.289+0000 7f044cff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.290+0000 7f044cff9640 1 -- 172.21.15.107:0/1211763003 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045c01bd30 con 0x55fbbbc78480 2024-05-09T08:34:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.290+0000 7f044cff9640 1 -- 172.21.15.107:0/1211763003 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f045c027480 con 0x55fbbbc78480 2024-05-09T08:34:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.290+0000 7f046df79740 1 -- 172.21.15.107:0/1211763003 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0454004030 con 0x55fbbbc78480 2024-05-09T08:34:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.290+0000 7f044cff9640 1 -- 172.21.15.107:0/1211763003 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f045c030050 con 0x55fbbbc78480 2024-05-09T08:34:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.290+0000 7f046df79740 10 client.15903 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.290+0000 7f046df79740 1 -- 172.21.15.107:0/1211763003 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fbbb8a40f0 con 0x55fbbbc78480 2024-05-09T08:34:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.290+0000 7f044cff9640 1 -- 172.21.15.107:0/1211763003 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0454004030 con 0x55fbbbc78480 2024-05-09T08:34:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.290+0000 7f044cff9640 1 -- 172.21.15.107:0/1211763003 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f045c019390 con 0x55fbbbc78480 2024-05-09T08:34:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.290+0000 7f046df79740 10 client.15903 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.290+0000 7f046df79740 10 client.15903 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.290+0000 7f046df79740 20 client.15903 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.290+0000 7f046df79740 1 --2- 172.21.15.107:0/1211763003 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fbbbc88880 0x55fbbbc8ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.290+0000 7f046df79740 4 client.15903 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:34:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.290+0000 7f046df79740 1 -- 172.21.15.107:0/1211763003 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbbbbde430 con 0x55fbbbc88880 2024-05-09T08:34:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.290+0000 7f0469007640 1 --2- 172.21.15.107:0/1211763003 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fbbbc88880 0x55fbbbc8ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.291+0000 7f0469007640 1 --2- 172.21.15.107:0/1211763003 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fbbbc88880 0x55fbbbc8ad00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38e53edc416fbee6 server_cookie=2f9e07772180c18a in_seq=0 out_seq=0 2024-05-09T08:34:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.291+0000 7f044cff9640 10 client.15903 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.291+0000 7f044cff9640 1 -- 172.21.15.107:0/1211763003 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fbbbbde430 con 0x55fbbbc88880 2024-05-09T08:34:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.291+0000 7f044cff9640 10 client.15903 handle_command_reply: tid=0 2024-05-09T08:34:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.292+0000 7f046df79740 1 client.15903 shutdown 2024-05-09T08:34:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.292+0000 7f046df79740 1 -- 172.21.15.107:0/1211763003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 msgr2=0x55fbbbc75e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.292+0000 7f046df79740 1 --2- 172.21.15.107:0/1211763003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc75e60 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f045c00ef60 tx=0x7f045c004a40 comp rx=0 tx=0).stop 2024-05-09T08:34:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.292+0000 7f046df79740 1 -- 172.21.15.107:0/1211763003 shutdown_connections 2024-05-09T08:34:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.292+0000 7f046df79740 1 --2- 172.21.15.107:0/1211763003 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fbbbc88880 0x55fbbbc8ad00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.292+0000 7f046df79740 1 --2- 172.21.15.107:0/1211763003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc75e60 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.292+0000 7f044cff9640 0 client.15903 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:01.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.292+0000 7f046df79740 1 --2- 172.21.15.107:0/1211763003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc75920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.292+0000 7f046df79740 1 --2- 172.21.15.107:0/1211763003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc753e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.292+0000 7f046df79740 1 -- 172.21.15.107:0/1211763003 >> 172.21.15.107:0/1211763003 conn(0x55fbbbc675a0 msgr2=0x55fbbbc6d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:01.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.292+0000 7f046df79740 1 -- 172.21.15.107:0/1211763003 shutdown_connections 2024-05-09T08:34:01.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.292+0000 7f046df79740 1 -- 172.21.15.107:0/1211763003 wait complete. 2024-05-09T08:34:01.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.292+0000 7f046df79740 20 client.15903 trim_cache size 0 max 16384 2024-05-09T08:34:01.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.309+0000 7f046df79740 1 Processor -- start 2024-05-09T08:34:01.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.309+0000 7f046df79740 1 -- start start 2024-05-09T08:34:01.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.309+0000 7f046df79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc84140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.309+0000 7f046df79740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc905d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.309+0000 7f046df79740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc90b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.309+0000 7f046df79740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f04641af150 con 0x55fbbbc78480 2024-05-09T08:34:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.309+0000 7f046df79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f04641aca20 con 0x55fbbbc6aef0 2024-05-09T08:34:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.309+0000 7f046df79740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f04641aa870 con 0x55fbbbc6b8a0 2024-05-09T08:34:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.310+0000 7f0469808640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc90b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.310+0000 7f0469808640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc90b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46944/0 (socket says 172.21.15.107:46944) 2024-05-09T08:34:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.310+0000 7f0469808640 1 -- 172.21.15.107:0/3853928648 learned_addr learned my addr 172.21.15.107:0/3853928648 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.310+0000 7f0469007640 1 --2- 172.21.15.107:0/3853928648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc84140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.310+0000 7f0468806640 1 --2- 172.21.15.107:0/3853928648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc905d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.310+0000 7f0469808640 1 -- 172.21.15.107:0/3853928648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6b8a0 msgr2=0x55fbbbc905d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.310+0000 7f0469808640 1 --2- 172.21.15.107:0/3853928648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc905d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.310+0000 7f0469808640 1 -- 172.21.15.107:0/3853928648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6aef0 msgr2=0x55fbbbc84140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:01.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.310+0000 7f0469808640 1 --2- 172.21.15.107:0/3853928648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc84140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.310+0000 7f0469808640 1 -- 172.21.15.107:0/3853928648 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbbbc6c4a0 con 0x55fbbbc78480 2024-05-09T08:34:01.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.310+0000 7f0469007640 1 --2- 172.21.15.107:0/3853928648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc84140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:01.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.310+0000 7f0468806640 1 --2- 172.21.15.107:0/3853928648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc905d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:01.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.310+0000 7f0469808640 1 --2- 172.21.15.107:0/3853928648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc90b10 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f045c00e9d0 tx=0x7f045c008f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce2808e41437a91 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:01.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.310+0000 7f044effd640 1 -- 172.21.15.107:0/3853928648 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045c01bb30 con 0x55fbbbc78480 2024-05-09T08:34:01.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.310+0000 7f044effd640 1 -- 172.21.15.107:0/3853928648 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f045c01bcd0 con 0x55fbbbc78480 2024-05-09T08:34:01.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.310+0000 7f044effd640 1 -- 172.21.15.107:0/3853928648 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045c005040 con 0x55fbbbc78480 2024-05-09T08:34:01.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.311+0000 7f046df79740 1 -- 172.21.15.107:0/3853928648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 msgr2=0x55fbbbc90b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.311+0000 7f046df79740 1 --2- 172.21.15.107:0/3853928648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc90b10 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f045c00e9d0 tx=0x7f045c008f90 comp rx=0 tx=0).stop 2024-05-09T08:34:01.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.311+0000 7f046df79740 1 -- 172.21.15.107:0/3853928648 shutdown_connections 2024-05-09T08:34:01.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.311+0000 7f046df79740 1 --2- 172.21.15.107:0/3853928648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc78480 0x55fbbbc90b10 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.311+0000 7f046df79740 1 --2- 172.21.15.107:0/3853928648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc905d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.311+0000 7f046df79740 1 --2- 172.21.15.107:0/3853928648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc84140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.311+0000 7f046df79740 1 -- 172.21.15.107:0/3853928648 >> 172.21.15.107:0/3853928648 conn(0x55fbbbc675a0 msgr2=0x55fbbbc75c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:01.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.311+0000 7f046df79740 1 -- 172.21.15.107:0/3853928648 shutdown_connections 2024-05-09T08:34:01.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.311+0000 7f046df79740 1 -- 172.21.15.107:0/3853928648 wait complete. 2024-05-09T08:34:01.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.311+0000 7f046df79740 1 -- start start 2024-05-09T08:34:01.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.311+0000 7f046df79740 1 Processor -- start 2024-05-09T08:34:01.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f046df79740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc80920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f046df79740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc7d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f046df79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc78480 0x55fbbbc7db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f046df79740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f04641aaee0 con 0x55fbbbc6b8a0 2024-05-09T08:34:01.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f046df79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f04641ac350 con 0x55fbbbc78480 2024-05-09T08:34:01.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f046df79740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f04641ab5b0 con 0x55fbbbc6aef0 2024-05-09T08:34:01.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f046df79740 10 client.0 _ll_register_callbacks cb 0x7f045c008a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:01.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f0468806640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc7d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f0468806640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc7d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46954/0 (socket says 172.21.15.107:46954) 2024-05-09T08:34:01.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f0468806640 1 -- 172.21.15.107:0/1257071426 learned_addr learned my addr 172.21.15.107:0/1257071426 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:01.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f0469808640 1 --2- 172.21.15.107:0/1257071426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc78480 0x55fbbbc7db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f0469007640 1 --2- 172.21.15.107:0/1257071426 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc80920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f0468806640 1 -- 172.21.15.107:0/1257071426 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6aef0 msgr2=0x55fbbbc80920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:01.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f0468806640 1 --2- 172.21.15.107:0/1257071426 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc80920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f0468806640 1 -- 172.21.15.107:0/1257071426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc78480 msgr2=0x55fbbbc7db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:01.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f0468806640 1 --2- 172.21.15.107:0/1257071426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc78480 0x55fbbbc7db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f0468806640 1 -- 172.21.15.107:0/1257071426 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbbbc7e220 con 0x55fbbbc6b8a0 2024-05-09T08:34:01.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.312+0000 7f0469007640 1 --2- 172.21.15.107:0/1257071426 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc80920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:01.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.313+0000 7f0468806640 1 --2- 172.21.15.107:0/1257071426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc7d620 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0454031440 tx=0x7f045402aa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b377acf61e1ca663 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:01.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.313+0000 7f044cff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:01.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.313+0000 7f044cff9640 1 -- 172.21.15.107:0/1257071426 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0454035070 con 0x55fbbbc6b8a0 2024-05-09T08:34:01.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.313+0000 7f044cff9640 1 -- 172.21.15.107:0/1257071426 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f0454004030 con 0x55fbbbc6b8a0 2024-05-09T08:34:01.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.313+0000 7f046df79740 1 -- 172.21.15.107:0/1257071426 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f045c01bcd0 con 0x55fbbbc6b8a0 2024-05-09T08:34:01.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.313+0000 7f044cff9640 1 -- 172.21.15.107:0/1257071426 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0454055050 con 0x55fbbbc6b8a0 2024-05-09T08:34:01.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.313+0000 7f046df79740 10 client.15909 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:01.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.313+0000 7f046df79740 1 -- 172.21.15.107:0/1257071426 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fbbbcefd60 con 0x55fbbbc6b8a0 2024-05-09T08:34:01.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.313+0000 7f044cff9640 1 -- 172.21.15.107:0/1257071426 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f045c01bcd0 con 0x55fbbbc6b8a0 2024-05-09T08:34:01.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.313+0000 7f044cff9640 1 -- 172.21.15.107:0/1257071426 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f0454004750 con 0x55fbbbc6b8a0 2024-05-09T08:34:01.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.313+0000 7f046df79740 10 client.15909 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:01.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.313+0000 7f046df79740 10 client.15909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:01.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.313+0000 7f046df79740 20 client.15909 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:01.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.313+0000 7f046df79740 1 --2- 172.21.15.107:0/1257071426 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fbbbc88880 0x55fbbbcf2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:01.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.313+0000 7f046df79740 4 client.15909 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:34:01.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.313+0000 7f046df79740 1 -- 172.21.15.107:0/1257071426 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fbbbbd8780 con 0x55fbbbc88880 2024-05-09T08:34:01.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f0469007640 1 --2- 172.21.15.107:0/1257071426 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fbbbc88880 0x55fbbbcf2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:01.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f0469007640 1 --2- 172.21.15.107:0/1257071426 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fbbbc88880 0x55fbbbcf2150 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebe32f74f0f85aeb server_cookie=84bbce8978fc63f1 in_seq=0 out_seq=0 2024-05-09T08:34:01.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f044cff9640 10 client.15909 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:01.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f044cff9640 1 -- 172.21.15.107:0/1257071426 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fbbbbd8780 con 0x55fbbbc88880 2024-05-09T08:34:01.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f044cff9640 10 client.15909 handle_command_reply: tid=0 2024-05-09T08:34:01.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f046df79740 1 client.15909 shutdown 2024-05-09T08:34:01.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f046df79740 1 -- 172.21.15.107:0/1257071426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc6b8a0 msgr2=0x55fbbbc7d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f046df79740 1 --2- 172.21.15.107:0/1257071426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc7d620 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0454031440 tx=0x7f045402aa60 comp rx=0 tx=0).stop 2024-05-09T08:34:01.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f046df79740 1 -- 172.21.15.107:0/1257071426 shutdown_connections 2024-05-09T08:34:01.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f046df79740 1 --2- 172.21.15.107:0/1257071426 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fbbbc88880 0x55fbbbcf2150 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f046df79740 1 --2- 172.21.15.107:0/1257071426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fbbbc78480 0x55fbbbc7db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f046df79740 1 --2- 172.21.15.107:0/1257071426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fbbbc6b8a0 0x55fbbbc7d620 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f044cff9640 0 client.15909 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:01.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f046df79740 1 --2- 172.21.15.107:0/1257071426 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fbbbc6aef0 0x55fbbbc80920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f046df79740 1 -- 172.21.15.107:0/1257071426 >> 172.21.15.107:0/1257071426 conn(0x55fbbbc675a0 msgr2=0x55fbbbc65f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:01.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f046df79740 1 -- 172.21.15.107:0/1257071426 shutdown_connections 2024-05-09T08:34:01.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.314+0000 7f046df79740 1 -- 172.21.15.107:0/1257071426 wait complete. 2024-05-09T08:34:01.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.315+0000 7f046df79740 20 client.15909 trim_cache size 0 max 16384 2024-05-09T08:34:01.325 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:34:01.325 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:34:01.325 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:34:01.325 INFO:teuthology.orchestra.run.smithi107.stdout: "c0c547ab-e6d9-42fd-a0b0-f85a016e7635": { 2024-05-09T08:34:01.325 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:34:01.325 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "c0c547ab-e6d9-42fd-a0b0-f85a016e7635", 2024-05-09T08:34:01.326 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:34:01.326 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:34:01.326 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:34:01.326 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:34:01.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.315+0000 7f046b26b640 1 -- 172.21.15.107:0/413229612 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f04500a2ea0 msgr2=0x7f04500a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.315+0000 7f046b26b640 1 --2- 172.21.15.107:0/413229612 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f04500a2ea0 0x7f04500a5360 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f045c005fd0 tx=0x7f045c0086a0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.315+0000 7f046b26b640 1 -- 172.21.15.107:0/413229612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0464100700 msgr2=0x7f0464196f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:01.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.315+0000 7f046b26b640 1 --2- 172.21.15.107:0/413229612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0464100700 0x7f0464196f40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f046000e9f0 tx=0x7f046000edf0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.315+0000 7f046b26b640 1 -- 172.21.15.107:0/413229612 shutdown_connections 2024-05-09T08:34:01.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.315+0000 7f046b26b640 1 --2- 172.21.15.107:0/413229612 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f04500a2ea0 0x7f04500a5360 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.315+0000 7f046b26b640 1 --2- 172.21.15.107:0/413229612 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0464106710 0x7f046419b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.315+0000 7f046b26b640 1 --2- 172.21.15.107:0/413229612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04641010c0 0x7f0464197480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.315+0000 7f046b26b640 1 --2- 172.21.15.107:0/413229612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0464100700 0x7f0464196f40 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:01.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.315+0000 7f046b26b640 1 -- 172.21.15.107:0/413229612 >> 172.21.15.107:0/413229612 conn(0x7f0464077830 msgr2=0x7f0464103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:01.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.315+0000 7f046b26b640 1 -- 172.21.15.107:0/413229612 shutdown_connections 2024-05-09T08:34:01.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:01.315+0000 7f046b26b640 1 -- 172.21.15.107:0/413229612 wait complete. 2024-05-09T08:34:01.328 INFO:tasks.cephfs.filesystem:scrub status for tag:c0c547ab-e6d9-42fd-a0b0-f85a016e7635 - {'path': '/', 'tag': 'c0c547ab-e6d9-42fd-a0b0-f85a016e7635', 'options': 'recursive,force'} 2024-05-09T08:34:02.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:02 smithi107 ceph-mon[30122]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T08:34:02.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:02 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:34:02.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:02 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:34:02.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:02 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:34:02.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:02 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:34:02.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:02 smithi107 ceph-mon[30122]: mgrmap e25: x(active, since 40s), standbys: y, z 2024-05-09T08:34:02.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:02 smithi157 ceph-mon[33614]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T08:34:02.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:02 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:34:02.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:02 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:34:02.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:02 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:34:02.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:02 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:34:02.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:02 smithi157 ceph-mon[33614]: mgrmap e25: x(active, since 40s), standbys: y, z 2024-05-09T08:34:02.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:02 smithi110 ceph-mon[33622]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T08:34:02.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:02 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:34:02.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:02 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:34:02.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:02 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:34:02.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:02 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:34:02.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:02 smithi110 ceph-mon[33622]: mgrmap e25: x(active, since 40s), standbys: y, z 2024-05-09T08:34:04.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:04 smithi107 ceph-mon[30122]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-09T08:34:04.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:04 smithi157 ceph-mon[33614]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-09T08:34:04.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:04 smithi110 ceph-mon[33622]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-09T08:34:06.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:06 smithi107 ceph-mon[30122]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-05-09T08:34:06.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:06 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:34:06.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:06 smithi157 ceph-mon[33614]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-05-09T08:34:06.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:06 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:34:06.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:06 smithi110 ceph-mon[33622]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-05-09T08:34:06.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:06 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:34:08.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:08 smithi107 ceph-mon[30122]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-09T08:34:08.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:08 smithi157 ceph-mon[33614]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-09T08:34:08.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:08 smithi110 ceph-mon[33622]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-09T08:34:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.741+0000 7f41471d5640 1 -- 172.21.15.107:0/1617918519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4140106720 msgr2=0x7f4140106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.741+0000 7f41471d5640 1 --2- 172.21.15.107:0/1617918519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4140106720 0x7f4140106b00 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f413c009930 tx=0x7f413c02a420 comp rx=0 tx=0).stop 2024-05-09T08:34:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.741+0000 7f41471d5640 1 -- 172.21.15.107:0/1617918519 shutdown_connections 2024-05-09T08:34:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.741+0000 7f41471d5640 1 --2- 172.21.15.107:0/1617918519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4140101120 0x7f414010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.741+0000 7f41471d5640 1 --2- 172.21.15.107:0/1617918519 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4140100760 0x7f4140100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.741+0000 7f41471d5640 1 --2- 172.21.15.107:0/1617918519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4140106720 0x7f4140106b00 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.741+0000 7f41471d5640 1 -- 172.21.15.107:0/1617918519 >> 172.21.15.107:0/1617918519 conn(0x7f41400778b0 msgr2=0x7f4140077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.742+0000 7f41471d5640 1 -- 172.21.15.107:0/1617918519 shutdown_connections 2024-05-09T08:34:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.742+0000 7f41471d5640 1 -- 172.21.15.107:0/1617918519 wait complete. 2024-05-09T08:34:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.742+0000 7f41471d5640 1 Processor -- start 2024-05-09T08:34:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.742+0000 7f41471d5640 1 -- start start 2024-05-09T08:34:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.742+0000 7f41471d5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4140100760 0x7f4140198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.742+0000 7f41471d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4140101120 0x7f4140199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.742+0000 7f41471d5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4140106720 0x7f41401a3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.742+0000 7f41471d5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f41401109e0 con 0x7f4140101120 2024-05-09T08:34:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.742+0000 7f41471d5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4140110860 con 0x7f4140106720 2024-05-09T08:34:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.742+0000 7f41471d5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4140110b60 con 0x7f4140100760 2024-05-09T08:34:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f4137fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4140101120 0x7f4140199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f4137fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4140101120 0x7f4140199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46960/0 (socket says 172.21.15.107:46960) 2024-05-09T08:34:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f4137fff640 1 -- 172.21.15.107:0/3723972133 learned_addr learned my addr 172.21.15.107:0/3723972133 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f4144f71640 1 --2- 172.21.15.107:0/3723972133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4140100760 0x7f4140198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f4145772640 1 --2- 172.21.15.107:0/3723972133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4140106720 0x7f41401a3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f4137fff640 1 -- 172.21.15.107:0/3723972133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4140100760 msgr2=0x7f4140198d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f4137fff640 1 --2- 172.21.15.107:0/3723972133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4140100760 0x7f4140198d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f4137fff640 1 -- 172.21.15.107:0/3723972133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4140106720 msgr2=0x7f41401a3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f4137fff640 1 --2- 172.21.15.107:0/3723972133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4140106720 0x7f41401a3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f4137fff640 1 -- 172.21.15.107:0/3723972133 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41401a42c0 con 0x7f4140101120 2024-05-09T08:34:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f4144f71640 1 --2- 172.21.15.107:0/3723972133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4140100760 0x7f4140198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f4137fff640 1 --2- 172.21.15.107:0/3723972133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4140101120 0x7f4140199240 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f413000c920 tx=0x7f413000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f4135ffb640 1 -- 172.21.15.107:0/3723972133 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4130007c70 con 0x7f4140101120 2024-05-09T08:34:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f41471d5640 1 -- 172.21.15.107:0/3723972133 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41401a4490 con 0x7f4140101120 2024-05-09T08:34:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f4135ffb640 1 -- 172.21.15.107:0/3723972133 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4130007e10 con 0x7f4140101120 2024-05-09T08:34:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f4135ffb640 1 -- 172.21.15.107:0/3723972133 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4130018820 con 0x7f4140101120 2024-05-09T08:34:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.743+0000 7f41471d5640 1 -- 172.21.15.107:0/3723972133 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41401a49e0 con 0x7f4140101120 2024-05-09T08:34:09.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.746+0000 7f4135ffb640 1 -- 172.21.15.107:0/3723972133 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 138118+0+0 (secure 0 0 0) 0x7f41300189c0 con 0x7f4140101120 2024-05-09T08:34:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.746+0000 7f4149fc3740 1 Processor -- start 2024-05-09T08:34:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.746+0000 7f4149fc3740 1 -- start start 2024-05-09T08:34:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.746+0000 7f4135ffb640 1 --2- 172.21.15.107:0/3723972133 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f41280a2f50 0x7f41280a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.746+0000 7f4149fc3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645cbc330 0x562645cbc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.746+0000 7f4149fc3740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645ccb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.746+0000 7f4149fc3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645ccba40 0x562645ccde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.746+0000 7f4149fc3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562645c21340 con 0x562645ccba40 2024-05-09T08:34:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.746+0000 7f4149fc3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562645bc9290 con 0x562645cbc330 2024-05-09T08:34:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.746+0000 7f4149fc3740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x562645c1d300 con 0x562645cbcc50 2024-05-09T08:34:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.746+0000 7f4145772640 1 --2- 172.21.15.107:0/3723972133 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f41280a2f50 0x7f41280a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.747+0000 7f4135ffb640 1 -- 172.21.15.107:0/3723972133 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f41300cfcb0 con 0x7f4140101120 2024-05-09T08:34:09.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.747+0000 7f4145772640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645ccba40 0x562645ccde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.747+0000 7f4145772640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645ccba40 0x562645ccde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46962/0 (socket says 172.21.15.107:46962) 2024-05-09T08:34:09.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.747+0000 7f4145772640 1 -- 172.21.15.107:0/80764285 learned_addr learned my addr 172.21.15.107:0/80764285 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:09.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.747+0000 7f4137fff640 1 --2- 172.21.15.107:0/80764285 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645ccb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.747+0000 7f4144f71640 1 --2- 172.21.15.107:0/80764285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645cbc330 0x562645cbc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.747+0000 7f4145772640 1 -- 172.21.15.107:0/80764285 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 msgr2=0x562645ccb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:09.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.747+0000 7f4145772640 1 --2- 172.21.15.107:0/80764285 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645ccb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.747+0000 7f4145772640 1 -- 172.21.15.107:0/80764285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645cbc330 msgr2=0x562645cbc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:09.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.747+0000 7f4145772640 1 --2- 172.21.15.107:0/80764285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645cbc330 0x562645cbc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.747+0000 7f4145772640 1 -- 172.21.15.107:0/80764285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562645bed8e0 con 0x562645ccba40 2024-05-09T08:34:09.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.747+0000 7f4145772640 1 --2- 172.21.15.107:0/3723972133 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f41280a2f50 0x7f41280a5410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f413800bfd0 tx=0x7f413800bd40 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:09.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.747+0000 7f4145772640 1 --2- 172.21.15.107:0/80764285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645ccba40 0x562645ccde30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f4138010dc0 tx=0x7f413801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad5ffc333bcfbec5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:09.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.747+0000 7f4126ffd640 1 -- 172.21.15.107:0/80764285 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413800ee60 con 0x562645ccba40 2024-05-09T08:34:09.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.747+0000 7f4126ffd640 1 -- 172.21.15.107:0/80764285 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41380130d0 con 0x562645ccba40 2024-05-09T08:34:09.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.747+0000 7f4126ffd640 1 -- 172.21.15.107:0/80764285 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4138003d90 con 0x562645ccba40 2024-05-09T08:34:09.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.748+0000 7f4149fc3740 1 -- 172.21.15.107:0/80764285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645ccba40 msgr2=0x562645ccde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:09.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.748+0000 7f4149fc3740 1 --2- 172.21.15.107:0/80764285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645ccba40 0x562645ccde30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f4138010dc0 tx=0x7f413801a000 comp rx=0 tx=0).stop 2024-05-09T08:34:09.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.748+0000 7f4149fc3740 1 -- 172.21.15.107:0/80764285 shutdown_connections 2024-05-09T08:34:09.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.748+0000 7f4149fc3740 1 --2- 172.21.15.107:0/80764285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645ccba40 0x562645ccde30 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.748+0000 7f4149fc3740 1 --2- 172.21.15.107:0/80764285 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645ccb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.748+0000 7f4149fc3740 1 --2- 172.21.15.107:0/80764285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645cbc330 0x562645cbc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.748+0000 7f4149fc3740 1 -- 172.21.15.107:0/80764285 >> 172.21.15.107:0/80764285 conn(0x562645cb8960 msgr2=0x562645cb8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:09.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.748+0000 7f4149fc3740 1 -- 172.21.15.107:0/80764285 shutdown_connections 2024-05-09T08:34:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.748+0000 7f4149fc3740 1 -- 172.21.15.107:0/80764285 wait complete. 2024-05-09T08:34:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.748+0000 7f4149fc3740 1 -- start start 2024-05-09T08:34:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4149fc3740 1 Processor -- start 2024-05-09T08:34:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4149fc3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645cbc330 0x562645cc8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4149fc3740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645cc8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4149fc3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645ccba40 0x562645cc94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4149fc3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562645cce610 con 0x562645ccba40 2024-05-09T08:34:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4149fc3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562645bdc790 con 0x562645cbc330 2024-05-09T08:34:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4149fc3740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x562645cce790 con 0x562645cbcc50 2024-05-09T08:34:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4149fc3740 10 client.0 _ll_register_callbacks cb 0x562645c17a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4145772640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645ccba40 0x562645cc94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4145772640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645ccba40 0x562645cc94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46964/0 (socket says 172.21.15.107:46964) 2024-05-09T08:34:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4145772640 1 -- 172.21.15.107:0/1727469513 learned_addr learned my addr 172.21.15.107:0/1727469513 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4144f71640 1 --2- 172.21.15.107:0/1727469513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645cbc330 0x562645cc8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4137fff640 1 --2- 172.21.15.107:0/1727469513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645cc8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4145772640 1 -- 172.21.15.107:0/1727469513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 msgr2=0x562645cc8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4145772640 1 --2- 172.21.15.107:0/1727469513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645cc8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4145772640 1 -- 172.21.15.107:0/1727469513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645cbc330 msgr2=0x562645cc8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4145772640 1 --2- 172.21.15.107:0/1727469513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645cbc330 0x562645cc8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4145772640 1 -- 172.21.15.107:0/1727469513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562645935210 con 0x562645ccba40 2024-05-09T08:34:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4137fff640 1 --2- 172.21.15.107:0/1727469513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645cc8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.749+0000 7f4144f71640 1 --2- 172.21.15.107:0/1727469513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645cbc330 0x562645cc8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:09.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.750+0000 7f4145772640 1 --2- 172.21.15.107:0/1727469513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645ccba40 0x562645cc94c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4138010d90 tx=0x7f4138005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30428f4002c0b0f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:09.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.750+0000 7f411ffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:09.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.750+0000 7f411ffff640 1 -- 172.21.15.107:0/1727469513 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413802a030 con 0x562645ccba40 2024-05-09T08:34:09.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.750+0000 7f4149fc3740 1 -- 172.21.15.107:0/1727469513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41380130d0 con 0x562645ccba40 2024-05-09T08:34:09.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.750+0000 7f411ffff640 1 -- 172.21.15.107:0/1727469513 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4138022070 con 0x562645ccba40 2024-05-09T08:34:09.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.750+0000 7f411ffff640 1 -- 172.21.15.107:0/1727469513 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4138037050 con 0x562645ccba40 2024-05-09T08:34:09.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.750+0000 7f4149fc3740 10 client.15921 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:09.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.750+0000 7f4149fc3740 1 -- 172.21.15.107:0/1727469513 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562645a01620 con 0x562645ccba40 2024-05-09T08:34:09.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.750+0000 7f411ffff640 1 -- 172.21.15.107:0/1727469513 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41380130d0 con 0x562645ccba40 2024-05-09T08:34:09.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.750+0000 7f411ffff640 1 -- 172.21.15.107:0/1727469513 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f41380314c0 con 0x562645ccba40 2024-05-09T08:34:09.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.750+0000 7f4149fc3740 10 client.15921 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:09.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.750+0000 7f4149fc3740 10 client.15921 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:09.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.750+0000 7f4149fc3740 20 client.15921 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:09.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.751+0000 7f4149fc3740 1 --2- 172.21.15.107:0/1727469513 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562645cdb770 0x562645cddb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.751+0000 7f4149fc3740 4 client.15921 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:34:09.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.751+0000 7f4149fc3740 1 -- 172.21.15.107:0/1727469513 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562645c2ea70 con 0x562645cdb770 2024-05-09T08:34:09.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.751+0000 7f4144f71640 1 --2- 172.21.15.107:0/1727469513 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562645cdb770 0x562645cddb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.751+0000 7f411ffff640 10 client.15921 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:09.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.751+0000 7f4144f71640 1 --2- 172.21.15.107:0/1727469513 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562645cdb770 0x562645cddb60 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9b9d17c05ec71aa server_cookie=5d0f242b25e41d21 in_seq=0 out_seq=0 2024-05-09T08:34:09.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.752+0000 7f411ffff640 1 -- 172.21.15.107:0/1727469513 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562645c2ea70 con 0x562645cdb770 2024-05-09T08:34:09.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.752+0000 7f411ffff640 10 client.15921 handle_command_reply: tid=0 2024-05-09T08:34:09.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.752+0000 7f4149fc3740 1 client.15921 shutdown 2024-05-09T08:34:09.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.752+0000 7f4149fc3740 1 -- 172.21.15.107:0/1727469513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645ccba40 msgr2=0x562645cc94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:09.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.752+0000 7f4149fc3740 1 --2- 172.21.15.107:0/1727469513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645ccba40 0x562645cc94c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4138010d90 tx=0x7f4138005280 comp rx=0 tx=0).stop 2024-05-09T08:34:09.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.752+0000 7f4149fc3740 1 -- 172.21.15.107:0/1727469513 shutdown_connections 2024-05-09T08:34:09.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.752+0000 7f4149fc3740 1 --2- 172.21.15.107:0/1727469513 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562645cdb770 0x562645cddb60 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.752+0000 7f4149fc3740 1 --2- 172.21.15.107:0/1727469513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645ccba40 0x562645cc94c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.752+0000 7f4149fc3740 1 --2- 172.21.15.107:0/1727469513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645cc8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.752+0000 7f4149fc3740 1 --2- 172.21.15.107:0/1727469513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645cbc330 0x562645cc8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.752+0000 7f4149fc3740 1 -- 172.21.15.107:0/1727469513 >> 172.21.15.107:0/1727469513 conn(0x562645cb8960 msgr2=0x562645cbe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:09.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.752+0000 7f411ffff640 0 client.15921 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:09.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.753+0000 7f4149fc3740 1 -- 172.21.15.107:0/1727469513 shutdown_connections 2024-05-09T08:34:09.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.753+0000 7f4149fc3740 1 -- 172.21.15.107:0/1727469513 wait complete. 2024-05-09T08:34:09.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.753+0000 7f4149fc3740 20 client.15921 trim_cache size 0 max 16384 2024-05-09T08:34:09.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4149fc3740 1 Processor -- start 2024-05-09T08:34:09.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4149fc3740 1 -- start start 2024-05-09T08:34:09.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4149fc3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645cbc330 0x562645cd7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4149fc3740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645cc81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4149fc3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645ccba40 0x562645cc9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4149fc3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f41401af060 con 0x562645cbc330 2024-05-09T08:34:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4149fc3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f41401ac930 con 0x562645ccba40 2024-05-09T08:34:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4149fc3740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f41401aa780 con 0x562645cbcc50 2024-05-09T08:34:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4144f71640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645cbc330 0x562645cd7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4144f71640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645cbc330 0x562645cd7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46972/0 (socket says 172.21.15.107:46972) 2024-05-09T08:34:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4144f71640 1 -- 172.21.15.107:0/851281674 learned_addr learned my addr 172.21.15.107:0/851281674 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4145772640 1 --2- 172.21.15.107:0/851281674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645ccba40 0x562645cc9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4137fff640 1 --2- 172.21.15.107:0/851281674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645cc81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4144f71640 1 -- 172.21.15.107:0/851281674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 msgr2=0x562645cc81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4144f71640 1 --2- 172.21.15.107:0/851281674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645cc81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4144f71640 1 -- 172.21.15.107:0/851281674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645ccba40 msgr2=0x562645cc9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4144f71640 1 --2- 172.21.15.107:0/851281674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645ccba40 0x562645cc9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4144f71640 1 -- 172.21.15.107:0/851281674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562645cc6f90 con 0x562645cbc330 2024-05-09T08:34:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.768+0000 7f4144f71640 1 --2- 172.21.15.107:0/851281674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645cbc330 0x562645cd7520 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f413c02a940 tx=0x7f413c0058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0eed72abd28aaa5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.769+0000 7f41267fc640 1 -- 172.21.15.107:0/851281674 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c049070 con 0x562645cbc330 2024-05-09T08:34:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.769+0000 7f41267fc640 1 -- 172.21.15.107:0/851281674 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f413c045c80 con 0x562645cbc330 2024-05-09T08:34:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.769+0000 7f41267fc640 1 -- 172.21.15.107:0/851281674 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c035040 con 0x562645cbc330 2024-05-09T08:34:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.769+0000 7f4149fc3740 1 -- 172.21.15.107:0/851281674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645cbc330 msgr2=0x562645cd7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.769+0000 7f4149fc3740 1 --2- 172.21.15.107:0/851281674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645cbc330 0x562645cd7520 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f413c02a940 tx=0x7f413c0058b0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.769+0000 7f4149fc3740 1 -- 172.21.15.107:0/851281674 shutdown_connections 2024-05-09T08:34:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.769+0000 7f4149fc3740 1 --2- 172.21.15.107:0/851281674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645ccba40 0x562645cc9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.769+0000 7f4149fc3740 1 --2- 172.21.15.107:0/851281674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645cc81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.769+0000 7f4149fc3740 1 --2- 172.21.15.107:0/851281674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645cbc330 0x562645cd7520 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.769+0000 7f4149fc3740 1 -- 172.21.15.107:0/851281674 >> 172.21.15.107:0/851281674 conn(0x562645cb8960 msgr2=0x562645cc8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.769+0000 7f4149fc3740 1 -- 172.21.15.107:0/851281674 shutdown_connections 2024-05-09T08:34:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.770+0000 7f4149fc3740 1 -- 172.21.15.107:0/851281674 wait complete. 2024-05-09T08:34:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.770+0000 7f4149fc3740 1 -- start start 2024-05-09T08:34:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.770+0000 7f4149fc3740 1 Processor -- start 2024-05-09T08:34:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.770+0000 7f4149fc3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645cbc330 0x562645cd3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.770+0000 7f4149fc3740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645cceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.770+0000 7f4149fc3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645ccba40 0x562645ccf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.770+0000 7f4149fc3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f41401aadf0 con 0x562645cbc330 2024-05-09T08:34:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.770+0000 7f4149fc3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f41401ac260 con 0x562645ccba40 2024-05-09T08:34:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.770+0000 7f4149fc3740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f41401ab4c0 con 0x562645cbcc50 2024-05-09T08:34:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.770+0000 7f4149fc3740 10 client.0 _ll_register_callbacks cb 0x7f413c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.770+0000 7f4144f71640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645cbc330 0x562645cd3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.770+0000 7f4144f71640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645cbc330 0x562645cd3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46988/0 (socket says 172.21.15.107:46988) 2024-05-09T08:34:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.770+0000 7f4144f71640 1 -- 172.21.15.107:0/877316246 learned_addr learned my addr 172.21.15.107:0/877316246 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.770+0000 7f4137fff640 1 --2- 172.21.15.107:0/877316246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645cceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.771+0000 7f4145772640 1 --2- 172.21.15.107:0/877316246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645ccba40 0x562645ccf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.771+0000 7f4144f71640 1 -- 172.21.15.107:0/877316246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 msgr2=0x562645cceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.771+0000 7f4144f71640 1 --2- 172.21.15.107:0/877316246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645cceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.771+0000 7f4144f71640 1 -- 172.21.15.107:0/877316246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645ccba40 msgr2=0x562645ccf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.771+0000 7f4144f71640 1 --2- 172.21.15.107:0/877316246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645ccba40 0x562645ccf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.771+0000 7f4144f71640 1 -- 172.21.15.107:0/877316246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562645ccfaa0 con 0x562645cbc330 2024-05-09T08:34:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.771+0000 7f4144f71640 1 --2- 172.21.15.107:0/877316246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645cbc330 0x562645cd3b50 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f413c005940 tx=0x7f413c045ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3748365df12cfc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.771+0000 7f411ffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.771+0000 7f411ffff640 1 -- 172.21.15.107:0/877316246 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c051070 con 0x562645cbc330 2024-05-09T08:34:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.771+0000 7f411ffff640 1 -- 172.21.15.107:0/877316246 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f413c0457a0 con 0x562645cbc330 2024-05-09T08:34:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.771+0000 7f4149fc3740 1 -- 172.21.15.107:0/877316246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f413c045c80 con 0x562645cbc330 2024-05-09T08:34:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.771+0000 7f411ffff640 1 -- 172.21.15.107:0/877316246 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f413c05c050 con 0x562645cbc330 2024-05-09T08:34:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.771+0000 7f4149fc3740 10 client.15927 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.771+0000 7f4149fc3740 1 -- 172.21.15.107:0/877316246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562645d41780 con 0x562645cbc330 2024-05-09T08:34:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.771+0000 7f411ffff640 1 -- 172.21.15.107:0/877316246 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f413c045c80 con 0x562645cbc330 2024-05-09T08:34:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.772+0000 7f411ffff640 1 -- 172.21.15.107:0/877316246 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f413c0054c0 con 0x562645cbc330 2024-05-09T08:34:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.772+0000 7f4149fc3740 10 client.15927 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.772+0000 7f4149fc3740 10 client.15927 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.772+0000 7f4149fc3740 20 client.15927 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.772+0000 7f4149fc3740 1 --2- 172.21.15.107:0/877316246 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562645d41e40 0x562645d44260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.772+0000 7f4149fc3740 4 client.15927 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:34:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.772+0000 7f4149fc3740 1 -- 172.21.15.107:0/877316246 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562645c28dc0 con 0x562645d41e40 2024-05-09T08:34:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.772+0000 7f4137fff640 1 --2- 172.21.15.107:0/877316246 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562645d41e40 0x562645d44260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.772+0000 7f4137fff640 1 --2- 172.21.15.107:0/877316246 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562645d41e40 0x562645d44260 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd2897b2935f9407 server_cookie=c0fba0fd8c219838 in_seq=0 out_seq=0 2024-05-09T08:34:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.772+0000 7f411ffff640 10 client.15927 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.778+0000 7f411ffff640 1 -- 172.21.15.107:0/877316246 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562645c28dc0 con 0x562645d41e40 2024-05-09T08:34:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.778+0000 7f411ffff640 10 client.15927 handle_command_reply: tid=0 2024-05-09T08:34:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.778+0000 7f4149fc3740 1 client.15927 shutdown 2024-05-09T08:34:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.779+0000 7f4149fc3740 1 -- 172.21.15.107:0/877316246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645cbc330 msgr2=0x562645cd3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.779+0000 7f4149fc3740 1 --2- 172.21.15.107:0/877316246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645cbc330 0x562645cd3b50 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f413c005940 tx=0x7f413c045ed0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.779+0000 7f4149fc3740 1 -- 172.21.15.107:0/877316246 shutdown_connections 2024-05-09T08:34:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.779+0000 7f4149fc3740 1 --2- 172.21.15.107:0/877316246 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562645d41e40 0x562645d44260 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.779+0000 7f4149fc3740 1 --2- 172.21.15.107:0/877316246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562645ccba40 0x562645ccf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.779+0000 7f411ffff640 0 client.15927 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.779+0000 7f4149fc3740 1 --2- 172.21.15.107:0/877316246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562645cbcc50 0x562645cceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.779+0000 7f4149fc3740 1 --2- 172.21.15.107:0/877316246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562645cbc330 0x562645cd3b50 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.779+0000 7f4149fc3740 1 -- 172.21.15.107:0/877316246 >> 172.21.15.107:0/877316246 conn(0x562645cb8960 msgr2=0x562645cbe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.779+0000 7f4149fc3740 1 -- 172.21.15.107:0/877316246 shutdown_connections 2024-05-09T08:34:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.779+0000 7f4149fc3740 1 -- 172.21.15.107:0/877316246 wait complete. 2024-05-09T08:34:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.779+0000 7f4149fc3740 20 client.15927 trim_cache size 0 max 16384 2024-05-09T08:34:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:34:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:34:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:34:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:34:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.779+0000 7f41471d5640 1 -- 172.21.15.107:0/3723972133 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f41280a2f50 msgr2=0x7f41280a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.779+0000 7f41471d5640 1 --2- 172.21.15.107:0/3723972133 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f41280a2f50 0x7f41280a5410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f413800bfd0 tx=0x7f413800bd40 comp rx=0 tx=0).stop 2024-05-09T08:34:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.779+0000 7f41471d5640 1 -- 172.21.15.107:0/3723972133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4140101120 msgr2=0x7f4140199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.779+0000 7f41471d5640 1 --2- 172.21.15.107:0/3723972133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4140101120 0x7f4140199240 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f413000c920 tx=0x7f413000cde0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.780+0000 7f41471d5640 1 -- 172.21.15.107:0/3723972133 shutdown_connections 2024-05-09T08:34:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.780+0000 7f41471d5640 1 --2- 172.21.15.107:0/3723972133 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f41280a2f50 0x7f41280a5410 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.780+0000 7f41471d5640 1 --2- 172.21.15.107:0/3723972133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4140106720 0x7f41401a3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.780+0000 7f41471d5640 1 --2- 172.21.15.107:0/3723972133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4140101120 0x7f4140199240 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.780+0000 7f41471d5640 1 --2- 172.21.15.107:0/3723972133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4140100760 0x7f4140198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.780+0000 7f41471d5640 1 -- 172.21.15.107:0/3723972133 >> 172.21.15.107:0/3723972133 conn(0x7f41400778b0 msgr2=0x7f4140103200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.780+0000 7f41471d5640 1 -- 172.21.15.107:0/3723972133 shutdown_connections 2024-05-09T08:34:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:09.780+0000 7f41471d5640 1 -- 172.21.15.107:0/3723972133 wait complete. 2024-05-09T08:34:10.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:10 smithi107 ceph-mon[30122]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-05-09T08:34:10.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:10 smithi157 ceph-mon[33614]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-05-09T08:34:10.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:10 smithi110 ceph-mon[33622]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-05-09T08:34:12.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:12 smithi107 ceph-mon[30122]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-05-09T08:34:12.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:12 smithi157 ceph-mon[33614]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-05-09T08:34:12.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:12 smithi110 ceph-mon[33622]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-05-09T08:34:14.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:14 smithi107 ceph-mon[30122]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-09T08:34:14.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:14 smithi157 ceph-mon[33614]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-09T08:34:14.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:14 smithi110 ceph-mon[33622]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-09T08:34:16.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:16 smithi107 ceph-mon[30122]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-05-09T08:34:16.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:16 smithi157 ceph-mon[33614]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-05-09T08:34:16.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:16 smithi110 ceph-mon[33622]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-05-09T08:34:18.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:18 smithi107 ceph-mon[30122]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 8 op/s 2024-05-09T08:34:18.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:18 smithi157 ceph-mon[33614]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 8 op/s 2024-05-09T08:34:18.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:18 smithi110 ceph-mon[33622]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 8 op/s 2024-05-09T08:34:20.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:20 smithi107 ceph-mon[30122]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-09T08:34:20.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:20 smithi157 ceph-mon[33614]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-09T08:34:20.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:20 smithi110 ceph-mon[33622]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-09T08:34:22.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:22 smithi107 ceph-mon[30122]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-09T08:34:22.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:22 smithi157 ceph-mon[33614]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-09T08:34:22.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:22 smithi110 ceph-mon[33622]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-09T08:34:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.884+0000 7f6ff88cd640 1 -- 172.21.15.107:0/4195954875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff410be00 msgr2=0x7f6ff410e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.884+0000 7f6ff88cd640 1 --2- 172.21.15.107:0/4195954875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff410be00 0x7f6ff410e2a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009ca0 tx=0x7f6fec02a9e0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.884+0000 7f6ff88cd640 1 -- 172.21.15.107:0/4195954875 shutdown_connections 2024-05-09T08:34:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.884+0000 7f6ff88cd640 1 --2- 172.21.15.107:0/4195954875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff410be00 0x7f6ff410e2a0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.884+0000 7f6ff88cd640 1 --2- 172.21.15.107:0/4195954875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff41033e0 0x7f6ff410b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.884+0000 7f6ff88cd640 1 --2- 172.21.15.107:0/4195954875 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ff4102ac0 0x7f6ff4102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.884+0000 7f6ff88cd640 1 -- 172.21.15.107:0/4195954875 >> 172.21.15.107:0/4195954875 conn(0x7f6ff40778b0 msgr2=0x7f6ff4077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:22.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.884+0000 7f6ff88cd640 1 -- 172.21.15.107:0/4195954875 shutdown_connections 2024-05-09T08:34:22.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.884+0000 7f6ff88cd640 1 -- 172.21.15.107:0/4195954875 wait complete. 2024-05-09T08:34:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.885+0000 7f6ff88cd640 1 Processor -- start 2024-05-09T08:34:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.885+0000 7f6ff88cd640 1 -- start start 2024-05-09T08:34:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.885+0000 7f6ff88cd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ff4102ac0 0x7f6ff419d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.885+0000 7f6ff88cd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff41033e0 0x7f6ff419de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.885+0000 7f6ff88cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff410be00 0x7f6ff4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.885+0000 7f6ff88cd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff4111ab0 con 0x7f6ff410be00 2024-05-09T08:34:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.885+0000 7f6ff88cd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff4111930 con 0x7f6ff41033e0 2024-05-09T08:34:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.885+0000 7f6ff88cd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff4111c30 con 0x7f6ff4102ac0 2024-05-09T08:34:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.885+0000 7f6ff2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff410be00 0x7f6ff4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.885+0000 7f6ff2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff410be00 0x7f6ff4197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54934/0 (socket says 172.21.15.107:54934) 2024-05-09T08:34:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.885+0000 7f6ff2d9d640 1 -- 172.21.15.107:0/371367578 learned_addr learned my addr 172.21.15.107:0/371367578 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.885+0000 7f6ff1d9b640 1 --2- 172.21.15.107:0/371367578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff41033e0 0x7f6ff419de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.885+0000 7f6ff259c640 1 --2- 172.21.15.107:0/371367578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ff4102ac0 0x7f6ff419d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.886+0000 7f6ff2d9d640 1 -- 172.21.15.107:0/371367578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ff4102ac0 msgr2=0x7f6ff419d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.886+0000 7f6ff2d9d640 1 --2- 172.21.15.107:0/371367578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ff4102ac0 0x7f6ff419d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.886+0000 7f6ff2d9d640 1 -- 172.21.15.107:0/371367578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff41033e0 msgr2=0x7f6ff419de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.886+0000 7f6ff2d9d640 1 --2- 172.21.15.107:0/371367578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff41033e0 0x7f6ff419de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.886+0000 7f6ff2d9d640 1 -- 172.21.15.107:0/371367578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff4198270 con 0x7f6ff410be00 2024-05-09T08:34:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.886+0000 7f6ff259c640 1 --2- 172.21.15.107:0/371367578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ff4102ac0 0x7f6ff419d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:34:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.886+0000 7f6ff2d9d640 1 --2- 172.21.15.107:0/371367578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff410be00 0x7f6ff4197af0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f6fec004af0 tx=0x7f6fec02e640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.886+0000 7f6fe37fe640 1 -- 172.21.15.107:0/371367578 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fec046070 con 0x7f6ff410be00 2024-05-09T08:34:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.886+0000 7f6fe37fe640 1 -- 172.21.15.107:0/371367578 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6fec003b70 con 0x7f6ff410be00 2024-05-09T08:34:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.886+0000 7f6ff88cd640 1 -- 172.21.15.107:0/371367578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ff4198500 con 0x7f6ff410be00 2024-05-09T08:34:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.886+0000 7f6fe37fe640 1 -- 172.21.15.107:0/371367578 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fec033040 con 0x7f6ff410be00 2024-05-09T08:34:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.886+0000 7f6ff88cd640 1 -- 172.21.15.107:0/371367578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ff41a4800 con 0x7f6ff410be00 2024-05-09T08:34:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.889+0000 7f6fe37fe640 1 -- 172.21.15.107:0/371367578 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 138118+0+0 (secure 0 0 0) 0x7f6fec049070 con 0x7f6ff410be00 2024-05-09T08:34:22.888 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.890+0000 7f6fe37fe640 1 --2- 172.21.15.107:0/371367578 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6fdc0a2fd0 0x7f6fdc0a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.890+0000 7f6ffb5dc740 1 Processor -- start 2024-05-09T08:34:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.890+0000 7f6ff259c640 1 --2- 172.21.15.107:0/371367578 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6fdc0a2fd0 0x7f6fdc0a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.890+0000 7f6ffb5dc740 1 -- start start 2024-05-09T08:34:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ffb5dc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffa6860 0x557a4ffa6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ffb5dc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa7180 0x557a4ffb1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ffb5dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffb1a40 0x557a4ffb3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ffb5dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557a4ff07340 con 0x557a4ffb1a40 2024-05-09T08:34:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ffb5dc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557a4feaf290 con 0x557a4ffa7180 2024-05-09T08:34:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ffb5dc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x557a4ff03300 con 0x557a4ffa6860 2024-05-09T08:34:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ff2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffb1a40 0x557a4ffb3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6fe37fe640 1 -- 172.21.15.107:0/371367578 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6fec0f0640 con 0x7f6ff410be00 2024-05-09T08:34:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ff259c640 1 --2- 172.21.15.107:0/371367578 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6fdc0a2fd0 0x7f6fdc0a5490 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4009b40 tx=0x7f6fe4005e50 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ff2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffb1a40 0x557a4ffb3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54942/0 (socket says 172.21.15.107:54942) 2024-05-09T08:34:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ff2d9d640 1 -- 172.21.15.107:0/1322967937 learned_addr learned my addr 172.21.15.107:0/1322967937 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ff1d9b640 1 --2- 172.21.15.107:0/1322967937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa7180 0x557a4ffb1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ff259c640 1 --2- 172.21.15.107:0/1322967937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffa6860 0x557a4ffa6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ff2d9d640 1 -- 172.21.15.107:0/1322967937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffa6860 msgr2=0x557a4ffa6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ff2d9d640 1 --2- 172.21.15.107:0/1322967937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffa6860 0x557a4ffa6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ff2d9d640 1 -- 172.21.15.107:0/1322967937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa7180 msgr2=0x557a4ffb1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ff2d9d640 1 --2- 172.21.15.107:0/1322967937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa7180 0x557a4ffb1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ff2d9d640 1 -- 172.21.15.107:0/1322967937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a4fed38e0 con 0x557a4ffb1a40 2024-05-09T08:34:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ff259c640 1 --2- 172.21.15.107:0/1322967937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffa6860 0x557a4ffa6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.891+0000 7f6ff2d9d640 1 --2- 172.21.15.107:0/1322967937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffb1a40 0x557a4ffb3e30 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0bc550 tx=0x7f6fec0b8990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ff0e1ddff364696 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.892+0000 7f6fe0ff9640 1 -- 172.21.15.107:0/1322967937 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fec004860 con 0x557a4ffb1a40 2024-05-09T08:34:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.892+0000 7f6fe0ff9640 1 -- 172.21.15.107:0/1322967937 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6fec0098d0 con 0x557a4ffb1a40 2024-05-09T08:34:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.892+0000 7f6fe0ff9640 1 -- 172.21.15.107:0/1322967937 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fec02fcf0 con 0x557a4ffb1a40 2024-05-09T08:34:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.892+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/1322967937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffb1a40 msgr2=0x557a4ffb3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/1322967937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffb1a40 0x557a4ffb3e30 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0bc550 tx=0x7f6fec0b8990 comp rx=0 tx=0).stop 2024-05-09T08:34:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/1322967937 shutdown_connections 2024-05-09T08:34:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/1322967937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffb1a40 0x557a4ffb3e30 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/1322967937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa7180 0x557a4ffb1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/1322967937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffa6860 0x557a4ffa6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/1322967937 >> 172.21.15.107:0/1322967937 conn(0x557a4ffa2d70 msgr2=0x557a4ffa3150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/1322967937 shutdown_connections 2024-05-09T08:34:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/1322967937 wait complete. 2024-05-09T08:34:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 1 -- start start 2024-05-09T08:34:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 1 Processor -- start 2024-05-09T08:34:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffa6860 0x557a4ffb1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffa7180 0x557a4ffae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffb1a40 0x557a4ffaef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557a4ff07340 con 0x557a4ffa6860 2024-05-09T08:34:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557a4fec2790 con 0x557a4ffb1a40 2024-05-09T08:34:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x557a4ffb4430 con 0x557a4ffa7180 2024-05-09T08:34:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.893+0000 7f6ffb5dc740 10 client.0 _ll_register_callbacks cb 0x557a4fefda20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6ff259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffa6860 0x557a4ffb1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6ff259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffa6860 0x557a4ffb1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54950/0 (socket says 172.21.15.107:54950) 2024-05-09T08:34:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6ff259c640 1 -- 172.21.15.107:0/1052558016 learned_addr learned my addr 172.21.15.107:0/1052558016 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6ff2d9d640 1 --2- 172.21.15.107:0/1052558016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffb1a40 0x557a4ffaef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6ff1d9b640 1 --2- 172.21.15.107:0/1052558016 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffa7180 0x557a4ffae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6ff259c640 1 -- 172.21.15.107:0/1052558016 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffa7180 msgr2=0x557a4ffae9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6ff259c640 1 --2- 172.21.15.107:0/1052558016 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffa7180 0x557a4ffae9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6ff259c640 1 -- 172.21.15.107:0/1052558016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffb1a40 msgr2=0x557a4ffaef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6ff259c640 1 --2- 172.21.15.107:0/1052558016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffb1a40 0x557a4ffaef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6ff259c640 1 -- 172.21.15.107:0/1052558016 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a4fc1b210 con 0x557a4ffa6860 2024-05-09T08:34:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6ff1d9b640 1 --2- 172.21.15.107:0/1052558016 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffa7180 0x557a4ffae9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6ff2d9d640 1 --2- 172.21.15.107:0/1052558016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffb1a40 0x557a4ffaef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6ff259c640 1 --2- 172.21.15.107:0/1052558016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffa6860 0x557a4ffb1420 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40118c0 tx=0x7f6fe40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c66af1d4161e89 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6fd9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6fd9ffb640 1 -- 172.21.15.107:0/1052558016 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4023070 con 0x557a4ffa6860 2024-05-09T08:34:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6fd9ffb640 1 -- 172.21.15.107:0/1052558016 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6fe4010d60 con 0x557a4ffa6860 2024-05-09T08:34:22.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.894+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/1052558016 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557a4ff14a70 con 0x557a4ffa6860 2024-05-09T08:34:22.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.895+0000 7f6fd9ffb640 1 -- 172.21.15.107:0/1052558016 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6fe402d050 con 0x557a4ffa6860 2024-05-09T08:34:22.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.895+0000 7f6ffb5dc740 10 client.15939 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:22.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.895+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/1052558016 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557a4fce7620 con 0x557a4ffa6860 2024-05-09T08:34:22.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.895+0000 7f6fd9ffb640 1 -- 172.21.15.107:0/1052558016 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x557a4ff14a70 con 0x557a4ffa6860 2024-05-09T08:34:22.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.895+0000 7f6fd9ffb640 1 -- 172.21.15.107:0/1052558016 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f6fe400f040 con 0x557a4ffa6860 2024-05-09T08:34:22.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.895+0000 7f6ffb5dc740 10 client.15939 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:22.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.895+0000 7f6ffb5dc740 10 client.15939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:22.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.895+0000 7f6ffb5dc740 20 client.15939 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:22.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.895+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/1052558016 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557a4ffc5850 0x557a4ffc7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.895+0000 7f6ffb5dc740 4 client.15939 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:34:22.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.895+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/1052558016 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a4ff0edc0 con 0x557a4ffc5850 2024-05-09T08:34:22.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.895+0000 7f6ff1d9b640 1 --2- 172.21.15.107:0/1052558016 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557a4ffc5850 0x557a4ffc7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.896+0000 7f6fd9ffb640 10 client.15939 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:22.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.896+0000 7f6ff1d9b640 1 --2- 172.21.15.107:0/1052558016 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557a4ffc5850 0x557a4ffc7c40 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69e7dd57434b3506 server_cookie=144e3ae9e6ce24 in_seq=0 out_seq=0 2024-05-09T08:34:22.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.897+0000 7f6fd9ffb640 1 -- 172.21.15.107:0/1052558016 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557a4ff0edc0 con 0x557a4ffc5850 2024-05-09T08:34:22.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.897+0000 7f6fd9ffb640 10 client.15939 handle_command_reply: tid=0 2024-05-09T08:34:22.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.897+0000 7f6ffb5dc740 1 client.15939 shutdown 2024-05-09T08:34:22.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.897+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/1052558016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffa6860 msgr2=0x557a4ffb1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:22.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.897+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/1052558016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffa6860 0x557a4ffb1420 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40118c0 tx=0x7f6fe40046c0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.897+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/1052558016 shutdown_connections 2024-05-09T08:34:22.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.897+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/1052558016 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557a4ffc5850 0x557a4ffc7c40 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.897+0000 7f6fd9ffb640 0 client.15939 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:22.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.897+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/1052558016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffb1a40 0x557a4ffaef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.897+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/1052558016 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffa7180 0x557a4ffae9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.897+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/1052558016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffa6860 0x557a4ffb1420 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.897+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/1052558016 >> 172.21.15.107:0/1052558016 conn(0x557a4ffa2d70 msgr2=0x557a4ffa8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:22.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.897+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/1052558016 shutdown_connections 2024-05-09T08:34:22.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.897+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/1052558016 wait complete. 2024-05-09T08:34:22.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.897+0000 7f6ffb5dc740 20 client.15939 trim_cache size 0 max 16384 2024-05-09T08:34:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.912+0000 7f6ffb5dc740 1 Processor -- start 2024-05-09T08:34:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.912+0000 7f6ffb5dc740 1 -- start start 2024-05-09T08:34:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.912+0000 7f6ffb5dc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa6860 0x557a4ffa8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.912+0000 7f6ffb5dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffa7180 0x557a4ffcd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.912+0000 7f6ffb5dc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffb1a40 0x557a4ffcda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.912+0000 7f6ffb5dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff41aee10 con 0x557a4ffa7180 2024-05-09T08:34:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.912+0000 7f6ffb5dc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff41ac6e0 con 0x557a4ffa6860 2024-05-09T08:34:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.912+0000 7f6ffb5dc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff41aa530 con 0x557a4ffb1a40 2024-05-09T08:34:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.912+0000 7f6ff1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffa7180 0x557a4ffcd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.912+0000 7f6ff1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffa7180 0x557a4ffcd4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54966/0 (socket says 172.21.15.107:54966) 2024-05-09T08:34:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.912+0000 7f6ff1d9b640 1 -- 172.21.15.107:0/4010611629 learned_addr learned my addr 172.21.15.107:0/4010611629 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.912+0000 7f6ff259c640 1 --2- 172.21.15.107:0/4010611629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa6860 0x557a4ffa8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.912+0000 7f6ff2d9d640 1 --2- 172.21.15.107:0/4010611629 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffb1a40 0x557a4ffcda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.913+0000 7f6ff1d9b640 1 -- 172.21.15.107:0/4010611629 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffb1a40 msgr2=0x557a4ffcda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.913+0000 7f6ff1d9b640 1 --2- 172.21.15.107:0/4010611629 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffb1a40 0x557a4ffcda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.913+0000 7f6ff1d9b640 1 -- 172.21.15.107:0/4010611629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa6860 msgr2=0x557a4ffa8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.913+0000 7f6ff1d9b640 1 --2- 172.21.15.107:0/4010611629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa6860 0x557a4ffa8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.913+0000 7f6ff1d9b640 1 -- 172.21.15.107:0/4010611629 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a4ffa6fb0 con 0x557a4ffa7180 2024-05-09T08:34:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.913+0000 7f6ff259c640 1 --2- 172.21.15.107:0/4010611629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa6860 0x557a4ffa8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.913+0000 7f6ff1d9b640 1 --2- 172.21.15.107:0/4010611629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffa7180 0x557a4ffcd4c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6fe800a790 tx=0x7f6fe8010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ec89364e989b65 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:22.914 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.913+0000 7f6fdbfff640 1 -- 172.21.15.107:0/4010611629 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8010e60 con 0x557a4ffa7180 2024-05-09T08:34:22.914 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.913+0000 7f6fdbfff640 1 -- 172.21.15.107:0/4010611629 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6fe8023030 con 0x557a4ffa7180 2024-05-09T08:34:22.914 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.913+0000 7f6fdbfff640 1 -- 172.21.15.107:0/4010611629 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe801ebd0 con 0x557a4ffa7180 2024-05-09T08:34:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.914+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/4010611629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffa7180 msgr2=0x557a4ffcd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.914+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/4010611629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffa7180 0x557a4ffcd4c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6fe800a790 tx=0x7f6fe8010800 comp rx=0 tx=0).stop 2024-05-09T08:34:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.914+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/4010611629 shutdown_connections 2024-05-09T08:34:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.914+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/4010611629 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffb1a40 0x557a4ffcda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.914+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/4010611629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffa7180 0x557a4ffcd4c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.914+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/4010611629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa6860 0x557a4ffa8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.914+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/4010611629 >> 172.21.15.107:0/4010611629 conn(0x557a4ffa2d70 msgr2=0x557a4ffa8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.914+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/4010611629 shutdown_connections 2024-05-09T08:34:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.914+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/4010611629 wait complete. 2024-05-09T08:34:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ffb5dc740 1 -- start start 2024-05-09T08:34:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ffb5dc740 1 Processor -- start 2024-05-09T08:34:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ffb5dc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa6860 0x557a4ffbfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ffb5dc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffb1a40 0x557a4ffc0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ffb5dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffc0670 0x557a5002ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ffb5dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff41aaba0 con 0x557a4ffc0670 2024-05-09T08:34:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ffb5dc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff41ac010 con 0x557a4ffa6860 2024-05-09T08:34:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ffb5dc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff41ab270 con 0x557a4ffb1a40 2024-05-09T08:34:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ffb5dc740 10 client.0 _ll_register_callbacks cb 0x7f6fe4009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ff2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffc0670 0x557a5002ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ff2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffc0670 0x557a5002ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54974/0 (socket says 172.21.15.107:54974) 2024-05-09T08:34:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ff2d9d640 1 -- 172.21.15.107:0/2547085432 learned_addr learned my addr 172.21.15.107:0/2547085432 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ff2d9d640 1 -- 172.21.15.107:0/2547085432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffb1a40 msgr2=0x557a4ffc0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ff259c640 1 --2- 172.21.15.107:0/2547085432 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa6860 0x557a4ffbfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ff2d9d640 1 --2- 172.21.15.107:0/2547085432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffb1a40 0x557a4ffc0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ff2d9d640 1 -- 172.21.15.107:0/2547085432 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa6860 msgr2=0x557a4ffbfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.915+0000 7f6ff2d9d640 1 --2- 172.21.15.107:0/2547085432 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa6860 0x557a4ffbfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.916+0000 7f6ff2d9d640 1 -- 172.21.15.107:0/2547085432 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a4ffc2bc0 con 0x557a4ffc0670 2024-05-09T08:34:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.916+0000 7f6ff2d9d640 1 --2- 172.21.15.107:0/2547085432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffc0670 0x557a5002ad40 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f6fec006010 tx=0x7f6fec004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254859bf4cd757c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.916+0000 7f6fd9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.916+0000 7f6ff259c640 1 --2- 172.21.15.107:0/2547085432 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa6860 0x557a4ffbfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:34:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.916+0000 7f6fd9ffb640 1 -- 172.21.15.107:0/2547085432 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fec0f0430 con 0x557a4ffc0670 2024-05-09T08:34:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.916+0000 7f6fd9ffb640 1 -- 172.21.15.107:0/2547085432 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6fec0d5290 con 0x557a4ffc0670 2024-05-09T08:34:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.916+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/2547085432 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fe8023030 con 0x557a4ffc0670 2024-05-09T08:34:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.916+0000 7f6fd9ffb640 1 -- 172.21.15.107:0/2547085432 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6fec0bc5c0 con 0x557a4ffc0670 2024-05-09T08:34:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.917+0000 7f6ffb5dc740 10 client.15945 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.917+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/2547085432 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557a5002c340 con 0x557a4ffc0670 2024-05-09T08:34:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.917+0000 7f6fd9ffb640 1 -- 172.21.15.107:0/2547085432 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6fe8023030 con 0x557a4ffc0670 2024-05-09T08:34:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.917+0000 7f6fd9ffb640 1 -- 172.21.15.107:0/2547085432 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f6fec040030 con 0x557a4ffc0670 2024-05-09T08:34:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.917+0000 7f6ffb5dc740 10 client.15945 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.917+0000 7f6ffb5dc740 10 client.15945 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.917+0000 7f6ffb5dc740 20 client.15945 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.917+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/2547085432 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557a5002cb60 0x557a5002efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.917+0000 7f6ffb5dc740 4 client.15945 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:34:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.917+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/2547085432 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557a4fe8c8e0 con 0x557a5002cb60 2024-05-09T08:34:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.917+0000 7f6ff259c640 1 --2- 172.21.15.107:0/2547085432 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557a5002cb60 0x557a5002efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.918+0000 7f6fd9ffb640 10 client.15945 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.918+0000 7f6ff259c640 1 --2- 172.21.15.107:0/2547085432 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557a5002cb60 0x557a5002efe0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be9755e888b7e10b server_cookie=2024bd24176a46a in_seq=0 out_seq=0 2024-05-09T08:34:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.921+0000 7f6fd9ffb640 1 -- 172.21.15.107:0/2547085432 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557a4fe8c8e0 con 0x557a5002cb60 2024-05-09T08:34:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.922+0000 7f6fd9ffb640 10 client.15945 handle_command_reply: tid=0 2024-05-09T08:34:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.922+0000 7f6ffb5dc740 1 client.15945 shutdown 2024-05-09T08:34:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.922+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/2547085432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffc0670 msgr2=0x557a5002ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.922+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/2547085432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffc0670 0x557a5002ad40 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f6fec006010 tx=0x7f6fec004e20 comp rx=0 tx=0).stop 2024-05-09T08:34:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.922+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/2547085432 shutdown_connections 2024-05-09T08:34:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.922+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/2547085432 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557a5002cb60 0x557a5002efe0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.922+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/2547085432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a4ffc0670 0x557a5002ad40 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.922+0000 7f6fd9ffb640 0 client.15945 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.922+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/2547085432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a4ffb1a40 0x557a4ffc0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.922+0000 7f6ffb5dc740 1 --2- 172.21.15.107:0/2547085432 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a4ffa6860 0x557a4ffbfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.922+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/2547085432 >> 172.21.15.107:0/2547085432 conn(0x557a4ffa2d70 msgr2=0x557a4ffa8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.922+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/2547085432 shutdown_connections 2024-05-09T08:34:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.922+0000 7f6ffb5dc740 1 -- 172.21.15.107:0/2547085432 wait complete. 2024-05-09T08:34:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.922+0000 7f6ffb5dc740 20 client.15945 trim_cache size 0 max 16384 2024-05-09T08:34:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:34:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:34:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:34:22.924 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:34:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.923+0000 7f6ff88cd640 1 -- 172.21.15.107:0/371367578 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6fdc0a2fd0 msgr2=0x7f6fdc0a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.923+0000 7f6ff88cd640 1 --2- 172.21.15.107:0/371367578 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6fdc0a2fd0 0x7f6fdc0a5490 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4009b40 tx=0x7f6fe4005e50 comp rx=0 tx=0).stop 2024-05-09T08:34:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.923+0000 7f6ff88cd640 1 -- 172.21.15.107:0/371367578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff410be00 msgr2=0x7f6ff4197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.923+0000 7f6ff88cd640 1 --2- 172.21.15.107:0/371367578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff410be00 0x7f6ff4197af0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f6fec004af0 tx=0x7f6fec02e640 comp rx=0 tx=0).stop 2024-05-09T08:34:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.923+0000 7f6ff88cd640 1 -- 172.21.15.107:0/371367578 shutdown_connections 2024-05-09T08:34:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.923+0000 7f6ff88cd640 1 --2- 172.21.15.107:0/371367578 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6fdc0a2fd0 0x7f6fdc0a5490 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.923+0000 7f6ff88cd640 1 --2- 172.21.15.107:0/371367578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff410be00 0x7f6ff4197af0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.923+0000 7f6ff88cd640 1 --2- 172.21.15.107:0/371367578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff41033e0 0x7f6ff419de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.923+0000 7f6ff88cd640 1 --2- 172.21.15.107:0/371367578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ff4102ac0 0x7f6ff419d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.923+0000 7f6ff88cd640 1 -- 172.21.15.107:0/371367578 >> 172.21.15.107:0/371367578 conn(0x7f6ff40778b0 msgr2=0x7f6ff4103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:22.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.923+0000 7f6ff88cd640 1 -- 172.21.15.107:0/371367578 shutdown_connections 2024-05-09T08:34:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:22.923+0000 7f6ff88cd640 1 -- 172.21.15.107:0/371367578 wait complete. 2024-05-09T08:34:24.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:24 smithi107 ceph-mon[30122]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 8 op/s 2024-05-09T08:34:24.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:24 smithi157 ceph-mon[33614]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 8 op/s 2024-05-09T08:34:24.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:24 smithi110 ceph-mon[33622]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 8 op/s 2024-05-09T08:34:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:26 smithi107 ceph-mon[30122]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:34:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:26 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:34:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:26 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:34:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:26 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:34:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:26 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:34:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:26 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:34:26.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:26 smithi157 ceph-mon[33614]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:34:26.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:26 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:34:26.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:26 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:34:26.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:26 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:34:26.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:26 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:34:26.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:26 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:34:26.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:26 smithi110 ceph-mon[33622]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:34:26.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:26 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:34:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:26 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:34:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:26 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:34:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:26 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:34:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:26 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:34:27.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:27 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:34:27.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:27 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:34:27.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:27 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:34:27.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:27 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:34:27.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:27 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:34:27.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:27 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:34:27.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:27 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:34:27.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:27 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:34:27.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:27 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:34:28.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:28 smithi107 ceph-mon[30122]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T08:34:28.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:28 smithi157 ceph-mon[33614]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T08:34:28.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:28 smithi110 ceph-mon[33622]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T08:34:30.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:30 smithi107 ceph-mon[30122]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T08:34:30.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:30 smithi157 ceph-mon[33614]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T08:34:30.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:30 smithi110 ceph-mon[33622]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T08:34:31.330 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:34:31.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:34:31.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.438+0000 7f623899d640 1 -- 172.21.15.107:0/3668462443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6234105700 msgr2=0x7f6234105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:31.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.438+0000 7f623899d640 1 --2- 172.21.15.107:0/3668462443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6234105700 0x7f6234105ae0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f6224009930 tx=0x7f622402a420 comp rx=0 tx=0).stop 2024-05-09T08:34:31.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.438+0000 7f623899d640 1 -- 172.21.15.107:0/3668462443 shutdown_connections 2024-05-09T08:34:31.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.438+0000 7f623899d640 1 --2- 172.21.15.107:0/3668462443 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6234101c30 0x7f623410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.438+0000 7f623899d640 1 --2- 172.21.15.107:0/3668462443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6234101270 0x7f62341016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.438+0000 7f623899d640 1 --2- 172.21.15.107:0/3668462443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6234105700 0x7f6234105ae0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.439+0000 7f623899d640 1 -- 172.21.15.107:0/3668462443 >> 172.21.15.107:0/3668462443 conn(0x7f62340759c0 msgr2=0x7f6234075dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:31.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.439+0000 7f623899d640 1 -- 172.21.15.107:0/3668462443 shutdown_connections 2024-05-09T08:34:31.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.439+0000 7f623899d640 1 -- 172.21.15.107:0/3668462443 wait complete. 2024-05-09T08:34:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.439+0000 7f623899d640 1 Processor -- start 2024-05-09T08:34:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.439+0000 7f623899d640 1 -- start start 2024-05-09T08:34:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f623899d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6234101270 0x7f6234196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f623899d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6234101c30 0x7f6234197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f623899d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6234105700 0x7f623419b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f623899d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f62341109e0 con 0x7f6234101270 2024-05-09T08:34:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f623899d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6234110860 con 0x7f6234105700 2024-05-09T08:34:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f623899d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6234110b60 con 0x7f6234101c30 2024-05-09T08:34:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f623259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6234101270 0x7f6234196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f623259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6234101270 0x7f6234196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38948/0 (socket says 172.21.15.107:38948) 2024-05-09T08:34:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f623259c640 1 -- 172.21.15.107:0/2439820278 learned_addr learned my addr 172.21.15.107:0/2439820278 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f622bfff640 1 --2- 172.21.15.107:0/2439820278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6234101c30 0x7f6234197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f6232d9d640 1 --2- 172.21.15.107:0/2439820278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6234105700 0x7f623419b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f623259c640 1 -- 172.21.15.107:0/2439820278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6234101c30 msgr2=0x7f6234197440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:31.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f623259c640 1 --2- 172.21.15.107:0/2439820278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6234101c30 0x7f6234197440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f623259c640 1 -- 172.21.15.107:0/2439820278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6234105700 msgr2=0x7f623419b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:31.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f623259c640 1 --2- 172.21.15.107:0/2439820278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6234105700 0x7f623419b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f623259c640 1 -- 172.21.15.107:0/2439820278 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f623419bef0 con 0x7f6234101270 2024-05-09T08:34:31.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.440+0000 7f623259c640 1 --2- 172.21.15.107:0/2439820278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6234101270 0x7f6234196f00 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f6224005940 tx=0x7f62240042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:31.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.441+0000 7f622b7fe640 1 -- 172.21.15.107:0/2439820278 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6224035050 con 0x7f6234101270 2024-05-09T08:34:31.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.441+0000 7f622b7fe640 1 -- 172.21.15.107:0/2439820278 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62240394e0 con 0x7f6234101270 2024-05-09T08:34:31.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.441+0000 7f623899d640 1 -- 172.21.15.107:0/2439820278 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f623419c180 con 0x7f6234101270 2024-05-09T08:34:31.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.441+0000 7f622b7fe640 1 -- 172.21.15.107:0/2439820278 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62240044e0 con 0x7f6234101270 2024-05-09T08:34:31.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.441+0000 7f623899d640 1 -- 172.21.15.107:0/2439820278 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6234100160 con 0x7f6234101270 2024-05-09T08:34:31.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.444+0000 7f622b7fe640 1 -- 172.21.15.107:0/2439820278 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 138118+0+0 (secure 0 0 0) 0x7f6224039690 con 0x7f6234101270 2024-05-09T08:34:31.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.444+0000 7f623b782740 1 Processor -- start 2024-05-09T08:34:31.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.444+0000 7f623b782740 1 -- start start 2024-05-09T08:34:31.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f623b782740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e1b280 0x564a35e1b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f623b782740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e1bba0 0x564a35e25d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f623b782740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e26270 0x564a35e28660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f623b782740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564a35d80d00 con 0x564a35e1bba0 2024-05-09T08:34:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f623b782740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564a35d284c0 con 0x564a35e1b280 2024-05-09T08:34:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f623b782740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564a35d7ccc0 con 0x564a35e26270 2024-05-09T08:34:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f623259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e1bba0 0x564a35e25d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f623259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e1bba0 0x564a35e25d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38960/0 (socket says 172.21.15.107:38960) 2024-05-09T08:34:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f623259c640 1 -- 172.21.15.107:0/1069509468 learned_addr learned my addr 172.21.15.107:0/1069509468 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f622bfff640 1 --2- 172.21.15.107:0/1069509468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e26270 0x564a35e28660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f6232d9d640 1 --2- 172.21.15.107:0/1069509468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e1b280 0x564a35e1b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f622b7fe640 1 --2- 172.21.15.107:0/2439820278 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f62200a2f30 0x7f62200a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f623259c640 1 -- 172.21.15.107:0/1069509468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e26270 msgr2=0x564a35e28660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f623259c640 1 --2- 172.21.15.107:0/1069509468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e26270 0x564a35e28660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f623259c640 1 -- 172.21.15.107:0/1069509468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e1b280 msgr2=0x564a35e1b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f623259c640 1 --2- 172.21.15.107:0/1069509468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e1b280 0x564a35e1b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f623259c640 1 -- 172.21.15.107:0/1069509468 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a35b5e7d0 con 0x564a35e1bba0 2024-05-09T08:34:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f6232d9d640 1 --2- 172.21.15.107:0/2439820278 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f62200a2f30 0x7f62200a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f622bfff640 1 --2- 172.21.15.107:0/1069509468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e26270 0x564a35e28660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.445+0000 7f622b7fe640 1 -- 172.21.15.107:0/2439820278 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f62240f1f50 con 0x7f6234101270 2024-05-09T08:34:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.446+0000 7f623259c640 1 --2- 172.21.15.107:0/1069509468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e1bba0 0x564a35e25d30 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f62240ef0a0 tx=0x7f62240e8a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff42c1f672910cb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.446+0000 7f6228ff9640 1 -- 172.21.15.107:0/1069509468 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62240bda10 con 0x564a35e1bba0 2024-05-09T08:34:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.446+0000 7f6232d9d640 1 --2- 172.21.15.107:0/2439820278 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f62200a2f30 0x7f62200a53f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f622c009fd0 tx=0x7f622c004ab0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.446+0000 7f6228ff9640 1 -- 172.21.15.107:0/1069509468 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62240be5d0 con 0x564a35e1bba0 2024-05-09T08:34:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.446+0000 7f6228ff9640 1 -- 172.21.15.107:0/1069509468 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62240f5b30 con 0x564a35e1bba0 2024-05-09T08:34:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 -- 172.21.15.107:0/1069509468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e1bba0 msgr2=0x564a35e25d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 --2- 172.21.15.107:0/1069509468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e1bba0 0x564a35e25d30 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f62240ef0a0 tx=0x7f62240e8a60 comp rx=0 tx=0).stop 2024-05-09T08:34:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 -- 172.21.15.107:0/1069509468 shutdown_connections 2024-05-09T08:34:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 --2- 172.21.15.107:0/1069509468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e26270 0x564a35e28660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 --2- 172.21.15.107:0/1069509468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e1bba0 0x564a35e25d30 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 --2- 172.21.15.107:0/1069509468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e1b280 0x564a35e1b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 -- 172.21.15.107:0/1069509468 >> 172.21.15.107:0/1069509468 conn(0x564a35e175a0 msgr2=0x564a35e17980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:31.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 -- 172.21.15.107:0/1069509468 shutdown_connections 2024-05-09T08:34:31.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 -- 172.21.15.107:0/1069509468 wait complete. 2024-05-09T08:34:31.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 -- start start 2024-05-09T08:34:31.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 Processor -- start 2024-05-09T08:34:31.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e1b280 0x564a35e23350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e23890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e23dd0 0x564a35e31f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564a35d284c0 con 0x564a35e26270 2024-05-09T08:34:31.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564a35d3bb90 con 0x564a35e23dd0 2024-05-09T08:34:31.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.447+0000 7f623b782740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564a35d80d00 con 0x564a35e1b280 2024-05-09T08:34:31.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.448+0000 7f623b782740 10 client.0 _ll_register_callbacks cb 0x564a35d77da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:31.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.448+0000 7f622bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e23890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.448+0000 7f622bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e23890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38962/0 (socket says 172.21.15.107:38962) 2024-05-09T08:34:31.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.448+0000 7f622bfff640 1 -- 172.21.15.107:0/1294342863 learned_addr learned my addr 172.21.15.107:0/1294342863 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:31.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.448+0000 7f623259c640 1 --2- 172.21.15.107:0/1294342863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e1b280 0x564a35e23350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.448+0000 7f6232d9d640 1 --2- 172.21.15.107:0/1294342863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e23dd0 0x564a35e31f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.448+0000 7f622bfff640 1 -- 172.21.15.107:0/1294342863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e1b280 msgr2=0x564a35e23350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:31.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.448+0000 7f622bfff640 1 --2- 172.21.15.107:0/1294342863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e1b280 0x564a35e23350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.448+0000 7f622bfff640 1 -- 172.21.15.107:0/1294342863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e23dd0 msgr2=0x564a35e31f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:31.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.448+0000 7f622bfff640 1 --2- 172.21.15.107:0/1294342863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e23dd0 0x564a35e31f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.448+0000 7f622bfff640 1 -- 172.21.15.107:0/1294342863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a35a943f0 con 0x564a35e26270 2024-05-09T08:34:31.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.448+0000 7f622bfff640 1 --2- 172.21.15.107:0/1294342863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e23890 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f621c0107a0 tx=0x7f621c010ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184d2966827dc5b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:31.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.448+0000 7f6219ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:31.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.448+0000 7f6219ffb640 1 -- 172.21.15.107:0/1294342863 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c00fa40 con 0x564a35e26270 2024-05-09T08:34:31.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.448+0000 7f623b782740 1 -- 172.21.15.107:0/1294342863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62240be5d0 con 0x564a35e26270 2024-05-09T08:34:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.448+0000 7f6219ffb640 1 -- 172.21.15.107:0/1294342863 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f621c00fbe0 con 0x564a35e26270 2024-05-09T08:34:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.449+0000 7f6219ffb640 1 -- 172.21.15.107:0/1294342863 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f621c024050 con 0x564a35e26270 2024-05-09T08:34:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.449+0000 7f623b782740 10 client.15957 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.449+0000 7f623b782740 1 -- 172.21.15.107:0/1294342863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564a35a540f0 con 0x564a35e26270 2024-05-09T08:34:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.449+0000 7f6219ffb640 1 -- 172.21.15.107:0/1294342863 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62240be5d0 con 0x564a35e26270 2024-05-09T08:34:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.449+0000 7f6219ffb640 1 -- 172.21.15.107:0/1294342863 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f621c018370 con 0x564a35e26270 2024-05-09T08:34:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.449+0000 7f623b782740 10 client.15957 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.449+0000 7f623b782740 10 client.15957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.449+0000 7f623b782740 20 client.15957 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.449+0000 7f623b782740 1 --2- 172.21.15.107:0/1294342863 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564a35e36670 0x564a35e38ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.449+0000 7f623b782740 4 client.15957 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:34:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.449+0000 7f623b782740 1 -- 172.21.15.107:0/1294342863 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a35d8e430 con 0x564a35e36670 2024-05-09T08:34:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.449+0000 7f623259c640 1 --2- 172.21.15.107:0/1294342863 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564a35e36670 0x564a35e38ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.450+0000 7f623259c640 1 --2- 172.21.15.107:0/1294342863 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564a35e36670 0x564a35e38ad0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a236f41502d1a4f server_cookie=62336e7bb468cf2c in_seq=0 out_seq=0 2024-05-09T08:34:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.450+0000 7f6219ffb640 10 client.15957 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.451+0000 7f6219ffb640 1 -- 172.21.15.107:0/1294342863 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a35d8e430 con 0x564a35e36670 2024-05-09T08:34:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.451+0000 7f6219ffb640 10 client.15957 handle_command_reply: tid=0 2024-05-09T08:34:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.451+0000 7f623b782740 1 client.15957 shutdown 2024-05-09T08:34:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.451+0000 7f623b782740 1 -- 172.21.15.107:0/1294342863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 msgr2=0x564a35e23890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.451+0000 7f623b782740 1 --2- 172.21.15.107:0/1294342863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e23890 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f621c0107a0 tx=0x7f621c010ba0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.451+0000 7f623b782740 1 -- 172.21.15.107:0/1294342863 shutdown_connections 2024-05-09T08:34:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.451+0000 7f623b782740 1 --2- 172.21.15.107:0/1294342863 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564a35e36670 0x564a35e38ad0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.451+0000 7f6219ffb640 0 client.15957 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.451+0000 7f623b782740 1 --2- 172.21.15.107:0/1294342863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e23dd0 0x564a35e31f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.451+0000 7f623b782740 1 --2- 172.21.15.107:0/1294342863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e23890 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.451+0000 7f623b782740 1 --2- 172.21.15.107:0/1294342863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e1b280 0x564a35e23350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.451+0000 7f623b782740 1 -- 172.21.15.107:0/1294342863 >> 172.21.15.107:0/1294342863 conn(0x564a35e175a0 msgr2=0x564a35e167e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.451+0000 7f623b782740 1 -- 172.21.15.107:0/1294342863 shutdown_connections 2024-05-09T08:34:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.451+0000 7f623b782740 1 -- 172.21.15.107:0/1294342863 wait complete. 2024-05-09T08:34:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.451+0000 7f623b782740 20 client.15957 trim_cache size 0 max 16384 2024-05-09T08:34:31.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.484+0000 7f623b782740 1 Processor -- start 2024-05-09T08:34:31.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.484+0000 7f623b782740 1 -- start start 2024-05-09T08:34:31.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f623b782740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e1b280 0x564a35e16f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f623b782740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e23dd0 0x564a35e17980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f623b782740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e3e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f623b782740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f62341af250 con 0x564a35e26270 2024-05-09T08:34:31.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f623b782740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f62341acb20 con 0x564a35e23dd0 2024-05-09T08:34:31.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f623b782740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f62341aa970 con 0x564a35e1b280 2024-05-09T08:34:31.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f6232d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e3e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f6232d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e3e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38970/0 (socket says 172.21.15.107:38970) 2024-05-09T08:34:31.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f6232d9d640 1 -- 172.21.15.107:0/1490089346 learned_addr learned my addr 172.21.15.107:0/1490089346 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:31.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f623259c640 1 --2- 172.21.15.107:0/1490089346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e1b280 0x564a35e16f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f622bfff640 1 --2- 172.21.15.107:0/1490089346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e23dd0 0x564a35e17980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f6232d9d640 1 -- 172.21.15.107:0/1490089346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e1b280 msgr2=0x564a35e16f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:31.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f6232d9d640 1 --2- 172.21.15.107:0/1490089346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e1b280 0x564a35e16f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f6232d9d640 1 -- 172.21.15.107:0/1490089346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e23dd0 msgr2=0x564a35e17980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:31.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f6232d9d640 1 --2- 172.21.15.107:0/1490089346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e23dd0 0x564a35e17980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f6232d9d640 1 -- 172.21.15.107:0/1490089346 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a35e12f60 con 0x564a35e26270 2024-05-09T08:34:31.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.485+0000 7f6232d9d640 1 --2- 172.21.15.107:0/1490089346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e3e3a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f622c012790 tx=0x7f622c0096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc443113d0d63a16 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:31.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.486+0000 7f621bfff640 1 -- 172.21.15.107:0/1490089346 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c004b20 con 0x564a35e26270 2024-05-09T08:34:31.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.486+0000 7f621bfff640 1 -- 172.21.15.107:0/1490089346 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f622c004cc0 con 0x564a35e26270 2024-05-09T08:34:31.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.486+0000 7f621bfff640 1 -- 172.21.15.107:0/1490089346 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c005520 con 0x564a35e26270 2024-05-09T08:34:31.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.486+0000 7f623b782740 1 -- 172.21.15.107:0/1490089346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 msgr2=0x564a35e3e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:31.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.486+0000 7f623b782740 1 --2- 172.21.15.107:0/1490089346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e3e3a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f622c012790 tx=0x7f622c0096a0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.486+0000 7f623b782740 1 -- 172.21.15.107:0/1490089346 shutdown_connections 2024-05-09T08:34:31.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.486+0000 7f623b782740 1 --2- 172.21.15.107:0/1490089346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e3e3a0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.486+0000 7f623b782740 1 --2- 172.21.15.107:0/1490089346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e23dd0 0x564a35e17980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.486+0000 7f623b782740 1 --2- 172.21.15.107:0/1490089346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e1b280 0x564a35e16f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.486+0000 7f623b782740 1 -- 172.21.15.107:0/1490089346 >> 172.21.15.107:0/1490089346 conn(0x564a35e175a0 msgr2=0x564a35e21c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:31.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.486+0000 7f623b782740 1 -- 172.21.15.107:0/1490089346 shutdown_connections 2024-05-09T08:34:31.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.486+0000 7f623b782740 1 -- 172.21.15.107:0/1490089346 wait complete. 2024-05-09T08:34:31.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.487+0000 7f623b782740 1 -- start start 2024-05-09T08:34:31.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.487+0000 7f623b782740 1 Processor -- start 2024-05-09T08:34:31.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.487+0000 7f623b782740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e1b280 0x564a35e2e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.487+0000 7f623b782740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e23dd0 0x564a35e1fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.487+0000 7f623b782740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e20140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.487+0000 7f623b782740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f62341aafe0 con 0x564a35e26270 2024-05-09T08:34:31.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.487+0000 7f623b782740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f62341ac450 con 0x564a35e23dd0 2024-05-09T08:34:31.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.487+0000 7f623b782740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f62341ab6b0 con 0x564a35e1b280 2024-05-09T08:34:31.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.487+0000 7f6232d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e20140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.487+0000 7f623b782740 10 client.0 _ll_register_callbacks cb 0x7f621c00a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:31.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.487+0000 7f6232d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e20140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38984/0 (socket says 172.21.15.107:38984) 2024-05-09T08:34:31.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.487+0000 7f6232d9d640 1 -- 172.21.15.107:0/1777393034 learned_addr learned my addr 172.21.15.107:0/1777393034 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:31.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.488+0000 7f622bfff640 1 --2- 172.21.15.107:0/1777393034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e23dd0 0x564a35e1fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.488+0000 7f623259c640 1 --2- 172.21.15.107:0/1777393034 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e1b280 0x564a35e2e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.488+0000 7f6232d9d640 1 -- 172.21.15.107:0/1777393034 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e1b280 msgr2=0x564a35e2e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:31.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.488+0000 7f6232d9d640 1 --2- 172.21.15.107:0/1777393034 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e1b280 0x564a35e2e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.488+0000 7f6232d9d640 1 -- 172.21.15.107:0/1777393034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e23dd0 msgr2=0x564a35e1fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:31.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.488+0000 7f6232d9d640 1 --2- 172.21.15.107:0/1777393034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e23dd0 0x564a35e1fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.488+0000 7f6232d9d640 1 -- 172.21.15.107:0/1777393034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a35e20680 con 0x564a35e26270 2024-05-09T08:34:31.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.488+0000 7f6232d9d640 1 --2- 172.21.15.107:0/1777393034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e20140 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f622c01e620 tx=0x7f622c0099a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ddca799ab90cba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:31.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.488+0000 7f6219ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:31.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.488+0000 7f6219ffb640 1 -- 172.21.15.107:0/1777393034 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c01ea10 con 0x564a35e26270 2024-05-09T08:34:31.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.488+0000 7f6219ffb640 1 -- 172.21.15.107:0/1777393034 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f622c01ebb0 con 0x564a35e26270 2024-05-09T08:34:31.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.488+0000 7f623b782740 1 -- 172.21.15.107:0/1777393034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f622c004cc0 con 0x564a35e26270 2024-05-09T08:34:31.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.488+0000 7f6219ffb640 1 -- 172.21.15.107:0/1777393034 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f622c03f050 con 0x564a35e26270 2024-05-09T08:34:31.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.488+0000 7f623b782740 10 client.15963 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:31.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.488+0000 7f623b782740 1 -- 172.21.15.107:0/1777393034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564a35e9c8e0 con 0x564a35e26270 2024-05-09T08:34:31.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.489+0000 7f6219ffb640 1 -- 172.21.15.107:0/1777393034 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f622c036070 con 0x564a35e26270 2024-05-09T08:34:31.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.489+0000 7f6219ffb640 1 -- 172.21.15.107:0/1777393034 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f622c00f040 con 0x564a35e26270 2024-05-09T08:34:31.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.489+0000 7f623b782740 10 client.15963 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:31.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.489+0000 7f623b782740 10 client.15963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:31.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.489+0000 7f623b782740 20 client.15963 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:31.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.489+0000 7f623b782740 1 --2- 172.21.15.107:0/1777393034 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564a35e36670 0x564a35e9efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.489+0000 7f623b782740 4 client.15963 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:34:31.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.489+0000 7f623b782740 1 -- 172.21.15.107:0/1777393034 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f622c004cc0 con 0x564a35e36670 2024-05-09T08:34:31.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.489+0000 7f623259c640 1 --2- 172.21.15.107:0/1777393034 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564a35e36670 0x564a35e9efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f6219ffb640 10 client.15963 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:31.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f623259c640 1 --2- 172.21.15.107:0/1777393034 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564a35e36670 0x564a35e9efe0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b53321908af76c53 server_cookie=921acdcbf3114261 in_seq=0 out_seq=0 2024-05-09T08:34:31.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f6219ffb640 1 -- 172.21.15.107:0/1777393034 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f622c004cc0 con 0x564a35e36670 2024-05-09T08:34:31.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f6219ffb640 10 client.15963 handle_command_reply: tid=0 2024-05-09T08:34:31.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f623b782740 1 client.15963 shutdown 2024-05-09T08:34:31.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f623b782740 1 -- 172.21.15.107:0/1777393034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 msgr2=0x564a35e20140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:31.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f623b782740 1 --2- 172.21.15.107:0/1777393034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e20140 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f622c01e620 tx=0x7f622c0099a0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f623b782740 1 -- 172.21.15.107:0/1777393034 shutdown_connections 2024-05-09T08:34:31.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f623b782740 1 --2- 172.21.15.107:0/1777393034 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564a35e36670 0x564a35e9efe0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f623b782740 1 --2- 172.21.15.107:0/1777393034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564a35e26270 0x564a35e20140 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f623b782740 1 --2- 172.21.15.107:0/1777393034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564a35e23dd0 0x564a35e1fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f6219ffb640 0 client.15963 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:31.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f623b782740 1 --2- 172.21.15.107:0/1777393034 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564a35e1b280 0x564a35e2e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f623b782740 1 -- 172.21.15.107:0/1777393034 >> 172.21.15.107:0/1777393034 conn(0x564a35e175a0 msgr2=0x564a35e1c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:31.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f623b782740 1 -- 172.21.15.107:0/1777393034 shutdown_connections 2024-05-09T08:34:31.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f623b782740 1 -- 172.21.15.107:0/1777393034 wait complete. 2024-05-09T08:34:31.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.490+0000 7f623b782740 20 client.15963 trim_cache size 0 max 16384 2024-05-09T08:34:31.497 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:34:31.498 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:34:31.498 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:34:31.498 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:34:31.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.491+0000 7f623899d640 1 -- 172.21.15.107:0/2439820278 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f62200a2f30 msgr2=0x7f62200a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:31.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.491+0000 7f623899d640 1 --2- 172.21.15.107:0/2439820278 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f62200a2f30 0x7f62200a53f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f622c009fd0 tx=0x7f622c004ab0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.491+0000 7f623899d640 1 -- 172.21.15.107:0/2439820278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6234101270 msgr2=0x7f6234196f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:31.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.491+0000 7f623899d640 1 --2- 172.21.15.107:0/2439820278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6234101270 0x7f6234196f00 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f6224005940 tx=0x7f62240042f0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.491+0000 7f623899d640 1 -- 172.21.15.107:0/2439820278 shutdown_connections 2024-05-09T08:34:31.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.491+0000 7f623899d640 1 --2- 172.21.15.107:0/2439820278 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f62200a2f30 0x7f62200a53f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.491+0000 7f623899d640 1 --2- 172.21.15.107:0/2439820278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6234105700 0x7f623419b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.491+0000 7f623899d640 1 --2- 172.21.15.107:0/2439820278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6234101c30 0x7f6234197440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.491+0000 7f623899d640 1 --2- 172.21.15.107:0/2439820278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6234101270 0x7f6234196f00 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.491+0000 7f623899d640 1 -- 172.21.15.107:0/2439820278 >> 172.21.15.107:0/2439820278 conn(0x7f62340759c0 msgr2=0x7f62340fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:31.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.491+0000 7f623899d640 1 -- 172.21.15.107:0/2439820278 shutdown_connections 2024-05-09T08:34:31.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.491+0000 7f623899d640 1 -- 172.21.15.107:0/2439820278 wait complete. 2024-05-09T08:34:31.501 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:34:31.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:34:31.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.620+0000 7f399847b640 1 -- 172.21.15.107:0/3733869221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39901033e0 msgr2=0x7f399010b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:31.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.620+0000 7f399847b640 1 --2- 172.21.15.107:0/3733869221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39901033e0 0x7f399010b8c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3980009900 tx=0x7f398002a510 comp rx=0 tx=0).stop 2024-05-09T08:34:31.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.620+0000 7f399847b640 1 -- 172.21.15.107:0/3733869221 shutdown_connections 2024-05-09T08:34:31.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.620+0000 7f399847b640 1 --2- 172.21.15.107:0/3733869221 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f399010be00 0x7f399010e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.620+0000 7f399847b640 1 --2- 172.21.15.107:0/3733869221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39901033e0 0x7f399010b8c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.620+0000 7f399847b640 1 --2- 172.21.15.107:0/3733869221 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3990102ac0 0x7f3990102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.620+0000 7f399847b640 1 -- 172.21.15.107:0/3733869221 >> 172.21.15.107:0/3733869221 conn(0x7f3990077830 msgr2=0x7f3990077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:31.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.620+0000 7f399847b640 1 -- 172.21.15.107:0/3733869221 shutdown_connections 2024-05-09T08:34:31.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.620+0000 7f399847b640 1 -- 172.21.15.107:0/3733869221 wait complete. 2024-05-09T08:34:31.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.621+0000 7f399847b640 1 Processor -- start 2024-05-09T08:34:31.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.621+0000 7f399847b640 1 -- start start 2024-05-09T08:34:31.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.621+0000 7f399847b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3990102ac0 0x7f3990100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.621+0000 7f399847b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f39901033e0 0x7f3990100c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.621+0000 7f399847b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f399010be00 0x7f3990101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.621+0000 7f399847b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3990111b00 con 0x7f3990102ac0 2024-05-09T08:34:31.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.621+0000 7f399847b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3990111980 con 0x7f399010be00 2024-05-09T08:34:31.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.621+0000 7f399847b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3990111c80 con 0x7f39901033e0 2024-05-09T08:34:31.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.621+0000 7f3996217640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3990102ac0 0x7f3990100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.621+0000 7f3996217640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3990102ac0 0x7f3990100710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39010/0 (socket says 172.21.15.107:39010) 2024-05-09T08:34:31.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.621+0000 7f3996217640 1 -- 172.21.15.107:0/1027754478 learned_addr learned my addr 172.21.15.107:0/1027754478 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:31.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.621+0000 7f3996a18640 1 --2- 172.21.15.107:0/1027754478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f399010be00 0x7f3990101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.621+0000 7f3995a16640 1 --2- 172.21.15.107:0/1027754478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f39901033e0 0x7f3990100c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.622+0000 7f3996217640 1 -- 172.21.15.107:0/1027754478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f39901033e0 msgr2=0x7f3990100c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:31.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.622+0000 7f3996217640 1 --2- 172.21.15.107:0/1027754478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f39901033e0 0x7f3990100c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.622+0000 7f3996217640 1 -- 172.21.15.107:0/1027754478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f399010be00 msgr2=0x7f3990101190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:31.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.622+0000 7f3996217640 1 --2- 172.21.15.107:0/1027754478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f399010be00 0x7f3990101190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.622+0000 7f3996217640 1 -- 172.21.15.107:0/1027754478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39901a6540 con 0x7f3990102ac0 2024-05-09T08:34:31.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.622+0000 7f3996a18640 1 --2- 172.21.15.107:0/1027754478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f399010be00 0x7f3990101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:31.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.622+0000 7f3996217640 1 --2- 172.21.15.107:0/1027754478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3990102ac0 0x7f3990100710 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f398c00d950 tx=0x7f398c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:31.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.622+0000 7f39877fe640 1 -- 172.21.15.107:0/1027754478 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398c017070 con 0x7f3990102ac0 2024-05-09T08:34:31.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.622+0000 7f39877fe640 1 -- 172.21.15.107:0/1027754478 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f398c009c30 con 0x7f3990102ac0 2024-05-09T08:34:31.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.622+0000 7f399847b640 1 -- 172.21.15.107:0/1027754478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39901a6830 con 0x7f3990102ac0 2024-05-09T08:34:31.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.622+0000 7f39877fe640 1 -- 172.21.15.107:0/1027754478 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398c00f620 con 0x7f3990102ac0 2024-05-09T08:34:31.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.622+0000 7f399847b640 1 -- 172.21.15.107:0/1027754478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39901a6d00 con 0x7f3990102ac0 2024-05-09T08:34:31.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.624+0000 7f399847b640 1 -- 172.21.15.107:0/1027754478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3990068650 con 0x7f3990102ac0 2024-05-09T08:34:31.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.625+0000 7f39877fe640 1 -- 172.21.15.107:0/1027754478 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 138118+0+0 (secure 0 0 0) 0x7f398c0109a0 con 0x7f3990102ac0 2024-05-09T08:34:31.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.626+0000 7f39877fe640 1 --2- 172.21.15.107:0/1027754478 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f397c0a2eb0 0x7f397c0a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:31.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.626+0000 7f3995a16640 1 --2- 172.21.15.107:0/1027754478 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f397c0a2eb0 0x7f397c0a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:31.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.626+0000 7f39877fe640 1 -- 172.21.15.107:0/1027754478 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f398c0cf6e0 con 0x7f3990102ac0 2024-05-09T08:34:31.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.627+0000 7f3995a16640 1 --2- 172.21.15.107:0/1027754478 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f397c0a2eb0 0x7f397c0a5370 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3980005ab0 tx=0x7f3980005a20 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:31.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.635+0000 7f39877fe640 1 -- 172.21.15.107:0/1027754478 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f398c009de0 con 0x7f3990102ac0 2024-05-09T08:34:31.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.966+0000 7f399847b640 1 -- 172.21.15.107:0/1027754478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3990106f80 con 0x7f3990102ac0 2024-05-09T08:34:31.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.967+0000 7f39877fe640 1 -- 172.21.15.107:0/1027754478 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11887 (secure 0 0 0) 0x7f398c09bd60 con 0x7f3990102ac0 2024-05-09T08:34:31.966 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:34:31.967 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:33:19.308598+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:34:31.967 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 25 2024-05-09T08:34:31.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.970+0000 7f399847b640 1 -- 172.21.15.107:0/1027754478 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f397c0a2eb0 msgr2=0x7f397c0a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:31.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.970+0000 7f399847b640 1 --2- 172.21.15.107:0/1027754478 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f397c0a2eb0 0x7f397c0a5370 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3980005ab0 tx=0x7f3980005a20 comp rx=0 tx=0).stop 2024-05-09T08:34:31.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.970+0000 7f399847b640 1 -- 172.21.15.107:0/1027754478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3990102ac0 msgr2=0x7f3990100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:31.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.970+0000 7f399847b640 1 --2- 172.21.15.107:0/1027754478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3990102ac0 0x7f3990100710 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f398c00d950 tx=0x7f398c00de10 comp rx=0 tx=0).stop 2024-05-09T08:34:31.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.970+0000 7f399847b640 1 -- 172.21.15.107:0/1027754478 shutdown_connections 2024-05-09T08:34:31.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.970+0000 7f399847b640 1 --2- 172.21.15.107:0/1027754478 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f397c0a2eb0 0x7f397c0a5370 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.970+0000 7f399847b640 1 --2- 172.21.15.107:0/1027754478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f399010be00 0x7f3990101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.970+0000 7f399847b640 1 --2- 172.21.15.107:0/1027754478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f39901033e0 0x7f3990100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.970+0000 7f399847b640 1 --2- 172.21.15.107:0/1027754478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3990102ac0 0x7f3990100710 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:31.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.970+0000 7f399847b640 1 -- 172.21.15.107:0/1027754478 >> 172.21.15.107:0/1027754478 conn(0x7f3990077830 msgr2=0x7f39900fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:31.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.970+0000 7f399847b640 1 -- 172.21.15.107:0/1027754478 shutdown_connections 2024-05-09T08:34:31.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:31.970+0000 7f399847b640 1 -- 172.21.15.107:0/1027754478 wait complete. 2024-05-09T08:34:31.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:34:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.093+0000 7f36fd542640 1 -- 172.21.15.107:0/1771353387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36f810bdf0 msgr2=0x7f36f810e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.093+0000 7f36fd542640 1 --2- 172.21.15.107:0/1771353387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36f810bdf0 0x7f36f810e290 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f36f0009ca0 tx=0x7f36f002a9e0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.094+0000 7f36fd542640 1 -- 172.21.15.107:0/1771353387 shutdown_connections 2024-05-09T08:34:32.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.094+0000 7f36fd542640 1 --2- 172.21.15.107:0/1771353387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36f810bdf0 0x7f36f810e290 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.094+0000 7f36fd542640 1 --2- 172.21.15.107:0/1771353387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36f81033d0 0x7f36f810b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.094+0000 7f36fd542640 1 --2- 172.21.15.107:0/1771353387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36f8102ab0 0x7f36f8102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.094+0000 7f36fd542640 1 -- 172.21.15.107:0/1771353387 >> 172.21.15.107:0/1771353387 conn(0x7f36f8077830 msgr2=0x7f36f8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.094+0000 7f36fd542640 1 -- 172.21.15.107:0/1771353387 shutdown_connections 2024-05-09T08:34:32.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.094+0000 7f36fd542640 1 -- 172.21.15.107:0/1771353387 wait complete. 2024-05-09T08:34:32.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.094+0000 7f36fd542640 1 Processor -- start 2024-05-09T08:34:32.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.094+0000 7f36fd542640 1 -- start start 2024-05-09T08:34:32.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.095+0000 7f36fd542640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36f8102ab0 0x7f36f8198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.095+0000 7f36fd542640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36f81033d0 0x7f36f8198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.095+0000 7f36fd542640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36f810bdf0 0x7f36f819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.095+0000 7f36fd542640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f36f8111aa0 con 0x7f36f810bdf0 2024-05-09T08:34:32.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.095+0000 7f36fd542640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f36f8111920 con 0x7f36f8102ab0 2024-05-09T08:34:32.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.095+0000 7f36fd542640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f36f8111c20 con 0x7f36f81033d0 2024-05-09T08:34:32.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.095+0000 7f36f77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36f810bdf0 0x7f36f819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.095+0000 7f36f77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36f810bdf0 0x7f36f819c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39026/0 (socket says 172.21.15.107:39026) 2024-05-09T08:34:32.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.095+0000 7f36f77fe640 1 -- 172.21.15.107:0/941848570 learned_addr learned my addr 172.21.15.107:0/941848570 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:32.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.095+0000 7f36f67fc640 1 --2- 172.21.15.107:0/941848570 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36f81033d0 0x7f36f8198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.095+0000 7f36f6ffd640 1 --2- 172.21.15.107:0/941848570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36f8102ab0 0x7f36f8198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.095+0000 7f36f77fe640 1 -- 172.21.15.107:0/941848570 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36f81033d0 msgr2=0x7f36f8198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.095+0000 7f36f77fe640 1 --2- 172.21.15.107:0/941848570 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36f81033d0 0x7f36f8198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.095+0000 7f36f77fe640 1 -- 172.21.15.107:0/941848570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36f8102ab0 msgr2=0x7f36f8198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.095+0000 7f36f77fe640 1 --2- 172.21.15.107:0/941848570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36f8102ab0 0x7f36f8198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.095+0000 7f36f77fe640 1 -- 172.21.15.107:0/941848570 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f819cfc0 con 0x7f36f810bdf0 2024-05-09T08:34:32.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.096+0000 7f36f77fe640 1 --2- 172.21.15.107:0/941848570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36f810bdf0 0x7f36f819c890 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f36f0006010 tx=0x7f36f002fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.096+0000 7f36dffff640 1 -- 172.21.15.107:0/941848570 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f0033040 con 0x7f36f810bdf0 2024-05-09T08:34:32.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.096+0000 7f36fd542640 1 -- 172.21.15.107:0/941848570 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f819d250 con 0x7f36f810bdf0 2024-05-09T08:34:32.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.096+0000 7f36fd542640 1 -- 172.21.15.107:0/941848570 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f81a4b40 con 0x7f36f810bdf0 2024-05-09T08:34:32.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.096+0000 7f36dffff640 1 -- 172.21.15.107:0/941848570 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36f002fcf0 con 0x7f36f810bdf0 2024-05-09T08:34:32.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.096+0000 7f36dffff640 1 -- 172.21.15.107:0/941848570 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f0051060 con 0x7f36f810bdf0 2024-05-09T08:34:32.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.099+0000 7f3700255740 1 Processor -- start 2024-05-09T08:34:32.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.099+0000 7f36dffff640 1 -- 172.21.15.107:0/941848570 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 138118+0+0 (secure 0 0 0) 0x7f36f0036070 con 0x7f36f810bdf0 2024-05-09T08:34:32.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.099+0000 7f3700255740 1 -- start start 2024-05-09T08:34:32.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f3700255740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 0x55bd335b8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f3700255740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335c2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f3700255740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335c3320 0x55bd335c5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f3700255740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55bd3351dc40 con 0x55bd335b8c50 2024-05-09T08:34:32.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f3700255740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55bd334c5580 con 0x55bd335b8330 2024-05-09T08:34:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f3700255740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55bd33519c00 con 0x55bd335c3320 2024-05-09T08:34:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f36f6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335c2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f36f6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335c2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39028/0 (socket says 172.21.15.107:39028) 2024-05-09T08:34:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f36f6ffd640 1 -- 172.21.15.107:0/2547038211 learned_addr learned my addr 172.21.15.107:0/2547038211 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f36f67fc640 1 --2- 172.21.15.107:0/2547038211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335c3320 0x55bd335c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f36f67fc640 1 --2- 172.21.15.107:0/2547038211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 0x55bd335b8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f36dffff640 1 --2- 172.21.15.107:0/941848570 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f36e00a2eb0 0x7f36e00a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f36f6ffd640 1 -- 172.21.15.107:0/2547038211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335c3320 msgr2=0x55bd335c5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f36f6ffd640 1 --2- 172.21.15.107:0/2547038211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335c3320 0x55bd335c5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f36f6ffd640 1 -- 172.21.15.107:0/2547038211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 msgr2=0x55bd335b8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f36f6ffd640 1 --2- 172.21.15.107:0/2547038211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 0x55bd335b8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f36f6ffd640 1 -- 172.21.15.107:0/2547038211 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd332313f0 con 0x55bd335b8c50 2024-05-09T08:34:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f36f67fc640 1 --2- 172.21.15.107:0/2547038211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 0x55bd335b8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:34:32.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f36dffff640 1 -- 172.21.15.107:0/941848570 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f36f00f06a0 con 0x7f36f810bdf0 2024-05-09T08:34:32.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f36f6ffd640 1 --2- 172.21.15.107:0/2547038211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335c2de0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f36e800bd10 tx=0x7f36e80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5713d7ba53222b54 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.100+0000 7f36f77fe640 1 --2- 172.21.15.107:0/941848570 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f36e00a2eb0 0x7f36e00a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.101+0000 7f36dd7fa640 1 -- 172.21.15.107:0/2547038211 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e8004030 con 0x55bd335b8c50 2024-05-09T08:34:32.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.101+0000 7f36dd7fa640 1 -- 172.21.15.107:0/2547038211 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36e80035f0 con 0x55bd335b8c50 2024-05-09T08:34:32.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.101+0000 7f36dd7fa640 1 -- 172.21.15.107:0/2547038211 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e8011770 con 0x55bd335b8c50 2024-05-09T08:34:32.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.102+0000 7f36f77fe640 1 --2- 172.21.15.107:0/941848570 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f36e00a2eb0 0x7f36e00a5370 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f36f00ed8b0 tx=0x7f36f002ff20 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.102+0000 7f3700255740 1 -- 172.21.15.107:0/2547038211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 msgr2=0x55bd335c2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.102+0000 7f3700255740 1 --2- 172.21.15.107:0/2547038211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335c2de0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f36e800bd10 tx=0x7f36e80107a0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.102+0000 7f3700255740 1 -- 172.21.15.107:0/2547038211 shutdown_connections 2024-05-09T08:34:32.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.102+0000 7f3700255740 1 --2- 172.21.15.107:0/2547038211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335c3320 0x55bd335c5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.102+0000 7f3700255740 1 --2- 172.21.15.107:0/2547038211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335c2de0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.102+0000 7f3700255740 1 --2- 172.21.15.107:0/2547038211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 0x55bd335b8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.102+0000 7f3700255740 1 -- 172.21.15.107:0/2547038211 >> 172.21.15.107:0/2547038211 conn(0x55bd335b4650 msgr2=0x55bd335b4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.102+0000 7f3700255740 1 -- 172.21.15.107:0/2547038211 shutdown_connections 2024-05-09T08:34:32.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.102+0000 7f3700255740 1 -- 172.21.15.107:0/2547038211 wait complete. 2024-05-09T08:34:32.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.102+0000 7f3700255740 1 -- start start 2024-05-09T08:34:32.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.102+0000 7f3700255740 1 Processor -- start 2024-05-09T08:34:32.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f3700255740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 0x55bd335c0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f3700255740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335b8c50 0x55bd335c0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f3700255740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335c3320 0x55bd335c0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f36f67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335c3320 0x55bd335c0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f36f67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335c3320 0x55bd335c0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39032/0 (socket says 172.21.15.107:39032) 2024-05-09T08:34:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f36f67fc640 1 -- 172.21.15.107:0/1619326121 learned_addr learned my addr 172.21.15.107:0/1619326121 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f3700255740 1 -- 172.21.15.107:0/1619326121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55bd335ca520 con 0x55bd335c3320 2024-05-09T08:34:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f3700255740 1 -- 172.21.15.107:0/1619326121 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55bd334d8c40 con 0x55bd335b8330 2024-05-09T08:34:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f3700255740 1 -- 172.21.15.107:0/1619326121 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55bd335ca6a0 con 0x55bd335b8c50 2024-05-09T08:34:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f3700255740 10 client.0 _ll_register_callbacks cb 0x55bd33230be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f36f67fc640 1 --2- 172.21.15.107:0/1619326121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 0x55bd335c0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f36f6ffd640 1 --2- 172.21.15.107:0/1619326121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335b8c50 0x55bd335c0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f36f67fc640 1 -- 172.21.15.107:0/1619326121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335b8c50 msgr2=0x55bd335c0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f36f67fc640 1 --2- 172.21.15.107:0/1619326121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335b8c50 0x55bd335c0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f36f67fc640 1 -- 172.21.15.107:0/1619326121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 msgr2=0x55bd335c0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f36f67fc640 1 --2- 172.21.15.107:0/1619326121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 0x55bd335c0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.103+0000 7f36f67fc640 1 -- 172.21.15.107:0/1619326121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd332fb7d0 con 0x55bd335c3320 2024-05-09T08:34:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.104+0000 7f36f67fc640 1 --2- 172.21.15.107:0/1619326121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335c3320 0x55bd335c0cc0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f36ec011f40 tx=0x7f36ec013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c64d54cce4e7b39 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.104+0000 7f36daff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.104+0000 7f36daff5640 1 -- 172.21.15.107:0/1619326121 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec004050 con 0x55bd335c3320 2024-05-09T08:34:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.104+0000 7f36daff5640 1 -- 172.21.15.107:0/1619326121 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f36ec0041f0 con 0x55bd335c3320 2024-05-09T08:34:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.104+0000 7f3700255740 1 -- 172.21.15.107:0/1619326121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36e80035f0 con 0x55bd335c3320 2024-05-09T08:34:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.104+0000 7f36daff5640 1 -- 172.21.15.107:0/1619326121 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f36ec0043f0 con 0x55bd335c3320 2024-05-09T08:34:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.104+0000 7f3700255740 10 client.15981 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.104+0000 7f3700255740 1 -- 172.21.15.107:0/1619326121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bd335d6740 con 0x55bd335c3320 2024-05-09T08:34:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.104+0000 7f36daff5640 1 -- 172.21.15.107:0/1619326121 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36ec0054d0 con 0x55bd335c3320 2024-05-09T08:34:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.105+0000 7f36daff5640 1 -- 172.21.15.107:0/1619326121 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f36ec004cc0 con 0x55bd335c3320 2024-05-09T08:34:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.105+0000 7f3700255740 10 client.15981 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.105+0000 7f3700255740 10 client.15981 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:34:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.105+0000 7f3700255740 20 client.15981 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.105+0000 7f3700255740 1 --2- 172.21.15.107:0/1619326121 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55bd335d7ca0 0x55bd335da090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.105+0000 7f3700255740 4 client.15981 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:34:32.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.105+0000 7f3700255740 1 -- 172.21.15.107:0/1619326121 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f36e80035f0 con 0x55bd335d7ca0 2024-05-09T08:34:32.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.105+0000 7f36f6ffd640 1 --2- 172.21.15.107:0/1619326121 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55bd335d7ca0 0x55bd335da090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.106+0000 7f36daff5640 10 client.15981 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:34:32.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.106+0000 7f36f6ffd640 1 --2- 172.21.15.107:0/1619326121 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55bd335d7ca0 0x55bd335da090 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8e4fda867fea5be6 server_cookie=50d060f1fdd93572 in_seq=0 out_seq=0 2024-05-09T08:34:32.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.107+0000 7f36daff5640 1 -- 172.21.15.107:0/1619326121 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f36e80035f0 con 0x55bd335d7ca0 2024-05-09T08:34:32.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.107+0000 7f36daff5640 10 client.15981 handle_command_reply: tid=0 2024-05-09T08:34:32.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.107+0000 7f3700255740 1 client.15981 shutdown 2024-05-09T08:34:32.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.107+0000 7f3700255740 1 -- 172.21.15.107:0/1619326121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335c3320 msgr2=0x55bd335c0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.107+0000 7f3700255740 1 --2- 172.21.15.107:0/1619326121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335c3320 0x55bd335c0cc0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f36ec011f40 tx=0x7f36ec013040 comp rx=0 tx=0).stop 2024-05-09T08:34:32.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.107+0000 7f3700255740 1 -- 172.21.15.107:0/1619326121 shutdown_connections 2024-05-09T08:34:32.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.107+0000 7f3700255740 1 --2- 172.21.15.107:0/1619326121 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55bd335d7ca0 0x55bd335da090 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.107+0000 7f3700255740 1 --2- 172.21.15.107:0/1619326121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335c3320 0x55bd335c0cc0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.107+0000 7f36daff5640 0 client.15981 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:34:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.107+0000 7f3700255740 1 --2- 172.21.15.107:0/1619326121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335b8c50 0x55bd335c0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.107+0000 7f3700255740 1 --2- 172.21.15.107:0/1619326121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 0x55bd335c0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.107+0000 7f3700255740 1 -- 172.21.15.107:0/1619326121 >> 172.21.15.107:0/1619326121 conn(0x55bd335b4650 msgr2=0x55bd335ba350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.107+0000 7f3700255740 1 -- 172.21.15.107:0/1619326121 shutdown_connections 2024-05-09T08:34:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.107+0000 7f3700255740 1 -- 172.21.15.107:0/1619326121 wait complete. 2024-05-09T08:34:32.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.107+0000 7f3700255740 20 client.15981 trim_cache size 0 max 16384 2024-05-09T08:34:32.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.120+0000 7f3700255740 1 Processor -- start 2024-05-09T08:34:32.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.120+0000 7f3700255740 1 -- start start 2024-05-09T08:34:32.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f3700255740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 0x55bd335c2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f3700255740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335d3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f3700255740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335c3320 0x55bd335bf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f3700255740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f36e4012510 con 0x55bd335b8c50 2024-05-09T08:34:32.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f3700255740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f36e400fde0 con 0x55bd335b8330 2024-05-09T08:34:32.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f3700255740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f36e400dce0 con 0x55bd335c3320 2024-05-09T08:34:32.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f36f6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335d3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f36f6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335d3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39048/0 (socket says 172.21.15.107:39048) 2024-05-09T08:34:32.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f36f6ffd640 1 -- 172.21.15.107:0/1757852489 learned_addr learned my addr 172.21.15.107:0/1757852489 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:32.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f36f67fc640 1 --2- 172.21.15.107:0/1757852489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 0x55bd335c2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f36f67fc640 1 --2- 172.21.15.107:0/1757852489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335c3320 0x55bd335bf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f36f6ffd640 1 -- 172.21.15.107:0/1757852489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335c3320 msgr2=0x55bd335bf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f36f6ffd640 1 --2- 172.21.15.107:0/1757852489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335c3320 0x55bd335bf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f36f6ffd640 1 -- 172.21.15.107:0/1757852489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 msgr2=0x55bd335c2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f36f6ffd640 1 --2- 172.21.15.107:0/1757852489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 0x55bd335c2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f36f6ffd640 1 -- 172.21.15.107:0/1757852489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd335c5a80 con 0x55bd335b8c50 2024-05-09T08:34:32.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.121+0000 7f36f67fc640 1 --2- 172.21.15.107:0/1757852489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 0x55bd335c2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:32.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.122+0000 7f36f6ffd640 1 --2- 172.21.15.107:0/1757852489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335d3470 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f36e800bd10 tx=0x7f36e800b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e15696b8745ed9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.122+0000 7f36dcff9640 1 -- 172.21.15.107:0/1757852489 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e80037f0 con 0x55bd335b8c50 2024-05-09T08:34:32.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.122+0000 7f36dcff9640 1 -- 172.21.15.107:0/1757852489 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36e802f070 con 0x55bd335b8c50 2024-05-09T08:34:32.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.122+0000 7f36dcff9640 1 -- 172.21.15.107:0/1757852489 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e802a410 con 0x55bd335b8c50 2024-05-09T08:34:32.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.123+0000 7f3700255740 1 -- 172.21.15.107:0/1757852489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 msgr2=0x55bd335d3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.123+0000 7f3700255740 1 --2- 172.21.15.107:0/1757852489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335d3470 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f36e800bd10 tx=0x7f36e800b690 comp rx=0 tx=0).stop 2024-05-09T08:34:32.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.123+0000 7f3700255740 1 -- 172.21.15.107:0/1757852489 shutdown_connections 2024-05-09T08:34:32.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.123+0000 7f3700255740 1 --2- 172.21.15.107:0/1757852489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335c3320 0x55bd335bf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.123+0000 7f3700255740 1 --2- 172.21.15.107:0/1757852489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335d3470 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.123+0000 7f3700255740 1 --2- 172.21.15.107:0/1757852489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335b8330 0x55bd335c2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.123+0000 7f3700255740 1 -- 172.21.15.107:0/1757852489 >> 172.21.15.107:0/1757852489 conn(0x55bd335b4650 msgr2=0x55bd335c0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.123+0000 7f3700255740 1 -- 172.21.15.107:0/1757852489 shutdown_connections 2024-05-09T08:34:32.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.123+0000 7f3700255740 1 -- 172.21.15.107:0/1757852489 wait complete. 2024-05-09T08:34:32.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.123+0000 7f3700255740 1 -- start start 2024-05-09T08:34:32.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.123+0000 7f3700255740 1 Processor -- start 2024-05-09T08:34:32.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.123+0000 7f3700255740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335b8330 0x55bd335cfe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.123+0000 7f3700255740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335bcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f3700255740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335c3320 0x55bd335bd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f3700255740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f36e400e2a0 con 0x55bd335b8c50 2024-05-09T08:34:32.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f3700255740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f36e400f710 con 0x55bd335c3320 2024-05-09T08:34:32.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f3700255740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f36e400e970 con 0x55bd335b8330 2024-05-09T08:34:32.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f36f6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335bcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f36f6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335bcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39062/0 (socket says 172.21.15.107:39062) 2024-05-09T08:34:32.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f3700255740 10 client.0 _ll_register_callbacks cb 0x7f36ec009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:32.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f36f6ffd640 1 -- 172.21.15.107:0/313601687 learned_addr learned my addr 172.21.15.107:0/313601687 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:32.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f36f67fc640 1 --2- 172.21.15.107:0/313601687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335b8330 0x55bd335cfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f36f67fc640 1 --2- 172.21.15.107:0/313601687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335c3320 0x55bd335bd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f36f6ffd640 1 -- 172.21.15.107:0/313601687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335b8330 msgr2=0x55bd335cfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f36f6ffd640 1 --2- 172.21.15.107:0/313601687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335b8330 0x55bd335cfe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f36f6ffd640 1 -- 172.21.15.107:0/313601687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335c3320 msgr2=0x55bd335bd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f36f6ffd640 1 --2- 172.21.15.107:0/313601687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335c3320 0x55bd335bd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f36f6ffd640 1 -- 172.21.15.107:0/313601687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd335bda90 con 0x55bd335b8c50 2024-05-09T08:34:32.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f36f67fc640 1 --2- 172.21.15.107:0/313601687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335c3320 0x55bd335bd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:34:32.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f36f6ffd640 1 --2- 172.21.15.107:0/313601687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335bcce0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f36e8011010 tx=0x7f36e800b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9fd37fd76dc4775 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f36daff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:32.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f36daff5640 1 -- 172.21.15.107:0/313601687 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e803a060 con 0x55bd335b8c50 2024-05-09T08:34:32.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f36daff5640 1 -- 172.21.15.107:0/313601687 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f36e80050c0 con 0x55bd335b8c50 2024-05-09T08:34:32.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.124+0000 7f3700255740 1 -- 172.21.15.107:0/313601687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36e802f070 con 0x55bd335b8c50 2024-05-09T08:34:32.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.125+0000 7f36daff5640 1 -- 172.21.15.107:0/313601687 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f36e8003b30 con 0x55bd335b8c50 2024-05-09T08:34:32.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.125+0000 7f3700255740 10 client.15987 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:32.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.125+0000 7f3700255740 1 -- 172.21.15.107:0/313601687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bd3363e260 con 0x55bd335b8c50 2024-05-09T08:34:32.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.125+0000 7f36daff5640 1 -- 172.21.15.107:0/313601687 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36e802f070 con 0x55bd335b8c50 2024-05-09T08:34:32.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.125+0000 7f36daff5640 1 -- 172.21.15.107:0/313601687 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f36e8035460 con 0x55bd335b8c50 2024-05-09T08:34:32.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.125+0000 7f3700255740 10 client.15987 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:32.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.125+0000 7f3700255740 10 client.15987 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:34:32.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.125+0000 7f3700255740 20 client.15987 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:32.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.125+0000 7f3700255740 1 --2- 172.21.15.107:0/313601687 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55bd335d7ca0 0x55bd33640780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.125+0000 7f3700255740 4 client.15987 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:34:32.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.125+0000 7f3700255740 1 -- 172.21.15.107:0/313601687 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bd3352b370 con 0x55bd335d7ca0 2024-05-09T08:34:32.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f36f67fc640 1 --2- 172.21.15.107:0/313601687 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55bd335d7ca0 0x55bd33640780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f36daff5640 10 client.15987 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:34:32.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f36f67fc640 1 --2- 172.21.15.107:0/313601687 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55bd335d7ca0 0x55bd33640780 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=33eba93a19b85152 server_cookie=8d425e924b96aa4a in_seq=0 out_seq=0 2024-05-09T08:34:32.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f36daff5640 1 -- 172.21.15.107:0/313601687 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bd3352b370 con 0x55bd335d7ca0 2024-05-09T08:34:32.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f36daff5640 10 client.15987 handle_command_reply: tid=0 2024-05-09T08:34:32.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f3700255740 1 client.15987 shutdown 2024-05-09T08:34:32.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f3700255740 1 -- 172.21.15.107:0/313601687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 msgr2=0x55bd335bcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f3700255740 1 --2- 172.21.15.107:0/313601687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335bcce0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f36e8011010 tx=0x7f36e800b410 comp rx=0 tx=0).stop 2024-05-09T08:34:32.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f3700255740 1 -- 172.21.15.107:0/313601687 shutdown_connections 2024-05-09T08:34:32.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f3700255740 1 --2- 172.21.15.107:0/313601687 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55bd335d7ca0 0x55bd33640780 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f3700255740 1 --2- 172.21.15.107:0/313601687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bd335c3320 0x55bd335bd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f36daff5640 0 client.15987 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:34:32.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f3700255740 1 --2- 172.21.15.107:0/313601687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bd335b8c50 0x55bd335bcce0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f3700255740 1 --2- 172.21.15.107:0/313601687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bd335b8330 0x55bd335cfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f3700255740 1 -- 172.21.15.107:0/313601687 >> 172.21.15.107:0/313601687 conn(0x55bd335b4650 msgr2=0x55bd335ba310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f3700255740 1 -- 172.21.15.107:0/313601687 shutdown_connections 2024-05-09T08:34:32.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.126+0000 7f3700255740 1 -- 172.21.15.107:0/313601687 wait complete. 2024-05-09T08:34:32.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.127+0000 7f3700255740 20 client.15987 trim_cache size 0 max 16384 2024-05-09T08:34:32.139 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:34:32.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.127+0000 7f36fd542640 1 -- 172.21.15.107:0/941848570 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f36e00a2eb0 msgr2=0x7f36e00a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.127+0000 7f36fd542640 1 --2- 172.21.15.107:0/941848570 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f36e00a2eb0 0x7f36e00a5370 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f36f00ed8b0 tx=0x7f36f002ff20 comp rx=0 tx=0).stop 2024-05-09T08:34:32.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.127+0000 7f36fd542640 1 -- 172.21.15.107:0/941848570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36f810bdf0 msgr2=0x7f36f819c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.127+0000 7f36fd542640 1 --2- 172.21.15.107:0/941848570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36f810bdf0 0x7f36f819c890 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f36f0006010 tx=0x7f36f002fbe0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.127+0000 7f36fd542640 1 -- 172.21.15.107:0/941848570 shutdown_connections 2024-05-09T08:34:32.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.127+0000 7f36fd542640 1 --2- 172.21.15.107:0/941848570 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f36e00a2eb0 0x7f36e00a5370 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.127+0000 7f36fd542640 1 --2- 172.21.15.107:0/941848570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36f810bdf0 0x7f36f819c890 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.127+0000 7f36fd542640 1 --2- 172.21.15.107:0/941848570 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36f81033d0 0x7f36f8198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.127+0000 7f36fd542640 1 --2- 172.21.15.107:0/941848570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36f8102ab0 0x7f36f8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.127+0000 7f36fd542640 1 -- 172.21.15.107:0/941848570 >> 172.21.15.107:0/941848570 conn(0x7f36f8077830 msgr2=0x7f36f80fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.127+0000 7f36fd542640 1 -- 172.21.15.107:0/941848570 shutdown_connections 2024-05-09T08:34:32.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.127+0000 7f36fd542640 1 -- 172.21.15.107:0/941848570 wait complete. 2024-05-09T08:34:32.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:34:32.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.248+0000 7fca11970640 1 -- 172.21.15.107:0/937330705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c106710 msgr2=0x7fca0c106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.248+0000 7fca11970640 1 --2- 172.21.15.107:0/937330705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c106710 0x7fca0c106af0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc009930 tx=0x7fc9fc02a420 comp rx=0 tx=0).stop 2024-05-09T08:34:32.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.248+0000 7fca11970640 1 -- 172.21.15.107:0/937330705 shutdown_connections 2024-05-09T08:34:32.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.248+0000 7fca11970640 1 --2- 172.21.15.107:0/937330705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca0c1010c0 0x7fca0c10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.248+0000 7fca11970640 1 --2- 172.21.15.107:0/937330705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca0c100700 0x7fca0c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.248+0000 7fca11970640 1 --2- 172.21.15.107:0/937330705 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c106710 0x7fca0c106af0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.248+0000 7fca11970640 1 -- 172.21.15.107:0/937330705 >> 172.21.15.107:0/937330705 conn(0x7fca0c077830 msgr2=0x7fca0c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.248+0000 7fca11970640 1 -- 172.21.15.107:0/937330705 shutdown_connections 2024-05-09T08:34:32.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.248+0000 7fca11970640 1 -- 172.21.15.107:0/937330705 wait complete. 2024-05-09T08:34:32.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.248+0000 7fca11970640 1 Processor -- start 2024-05-09T08:34:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.249+0000 7fca11970640 1 -- start start 2024-05-09T08:34:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.249+0000 7fca11970640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca0c100700 0x7fca0c1982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.249+0000 7fca11970640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c1010c0 0x7fca0c1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.249+0000 7fca11970640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca0c19cb60 0x7fca0c19d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.249+0000 7fca11970640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c110990 con 0x7fca0c1010c0 2024-05-09T08:34:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.249+0000 7fca11970640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c110810 con 0x7fca0c100700 2024-05-09T08:34:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.249+0000 7fca11970640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c110b10 con 0x7fca0c19cb60 2024-05-09T08:34:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.249+0000 7fca0a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c1010c0 0x7fca0c1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.249+0000 7fca0a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c1010c0 0x7fca0c1987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39076/0 (socket says 172.21.15.107:39076) 2024-05-09T08:34:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.249+0000 7fca0a7fc640 1 -- 172.21.15.107:0/3814678571 learned_addr learned my addr 172.21.15.107:0/3814678571 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.249+0000 7fca0affd640 1 --2- 172.21.15.107:0/3814678571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca0c100700 0x7fca0c1982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.250+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/3814678571 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca0c19cb60 0x7fca0c19d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.250+0000 7fca0a7fc640 1 -- 172.21.15.107:0/3814678571 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca0c19cb60 msgr2=0x7fca0c19d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.250+0000 7fca0a7fc640 1 --2- 172.21.15.107:0/3814678571 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca0c19cb60 0x7fca0c19d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.250+0000 7fca0a7fc640 1 -- 172.21.15.107:0/3814678571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca0c100700 msgr2=0x7fca0c1982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.250+0000 7fca0a7fc640 1 --2- 172.21.15.107:0/3814678571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca0c100700 0x7fca0c1982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.250+0000 7fca0a7fc640 1 -- 172.21.15.107:0/3814678571 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca0c19d650 con 0x7fca0c1010c0 2024-05-09T08:34:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.250+0000 7fca0affd640 1 --2- 172.21.15.107:0/3814678571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca0c100700 0x7fca0c1982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.250+0000 7fca0a7fc640 1 --2- 172.21.15.107:0/3814678571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c1010c0 0x7fca0c1987e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fca0000c890 tx=0x7fca0000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.250+0000 7fca1096e640 1 -- 172.21.15.107:0/3814678571 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca00004220 con 0x7fca0c1010c0 2024-05-09T08:34:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.250+0000 7fca11970640 1 -- 172.21.15.107:0/3814678571 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca0c19d940 con 0x7fca0c1010c0 2024-05-09T08:34:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.250+0000 7fca1096e640 1 -- 172.21.15.107:0/3814678571 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca000043c0 con 0x7fca0c1010c0 2024-05-09T08:34:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.250+0000 7fca1096e640 1 -- 172.21.15.107:0/3814678571 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca00004a00 con 0x7fca0c1010c0 2024-05-09T08:34:32.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.250+0000 7fca11970640 1 -- 172.21.15.107:0/3814678571 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca0c1a5180 con 0x7fca0c1010c0 2024-05-09T08:34:32.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.253+0000 7fca1096e640 1 -- 172.21.15.107:0/3814678571 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 138118+0+0 (secure 0 0 0) 0x7fca00004ba0 con 0x7fca0c1010c0 2024-05-09T08:34:32.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.254+0000 7fca14322740 1 Processor -- start 2024-05-09T08:34:32.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.254+0000 7fca14322740 1 -- start start 2024-05-09T08:34:32.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.254+0000 7fca14322740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 0x557eeddc6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.254+0000 7fca14322740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddc6c50 0x557eeddd0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.254+0000 7fca14322740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddd1320 0x557eeddd3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.254+0000 7fca14322740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557eedd2bc40 con 0x557eeddc6c50 2024-05-09T08:34:32.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.254+0000 7fca14322740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557eedcd3580 con 0x557eeddc6330 2024-05-09T08:34:32.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.254+0000 7fca14322740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x557eedd27c00 con 0x557eeddd1320 2024-05-09T08:34:32.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.254+0000 7fca0b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 0x557eeddc6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.254+0000 7fca0a7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddd1320 0x557eeddd3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.254+0000 7fca0b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 0x557eeddc6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:46490/0 (socket says 172.21.15.107:46490) 2024-05-09T08:34:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.254+0000 7fca0b7fe640 1 -- 172.21.15.107:0/3690281401 learned_addr learned my addr 172.21.15.107:0/3690281401 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.254+0000 7fca1096e640 1 --2- 172.21.15.107:0/3814678571 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9f40a2eb0 0x7fc9f40a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.254+0000 7fca0affd640 1 --2- 172.21.15.107:0/3690281401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddc6c50 0x557eeddd0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.255+0000 7fca0a7fc640 1 -- 172.21.15.107:0/3690281401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 msgr2=0x557eeddc6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.255+0000 7fca0a7fc640 1 --2- 172.21.15.107:0/3690281401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 0x557eeddc6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.255+0000 7fca0a7fc640 1 -- 172.21.15.107:0/3690281401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddc6c50 msgr2=0x557eeddd0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.255+0000 7fca0a7fc640 1 --2- 172.21.15.107:0/3690281401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddc6c50 0x557eeddd0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.255+0000 7fca0a7fc640 1 -- 172.21.15.107:0/3690281401 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557eeda3f3f0 con 0x557eeddd1320 2024-05-09T08:34:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.255+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/3690281401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 0x557eeddc6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.255+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/3814678571 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9f40a2eb0 0x7fc9f40a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.255+0000 7fca1096e640 1 -- 172.21.15.107:0/3814678571 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fca000cf430 con 0x7fca0c1010c0 2024-05-09T08:34:32.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.255+0000 7fca0a7fc640 1 --2- 172.21.15.107:0/3690281401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddd1320 0x557eeddd3710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fca000cc5e0 tx=0x7fca00097780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41269723147fe28b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.255+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/3814678571 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9f40a2eb0 0x7fc9f40a5370 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fca04009370 tx=0x7fca04004ab0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.255+0000 7fc9f1ffb640 1 -- 172.21.15.107:0/3690281401 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0001d070 con 0x557eeddd1320 2024-05-09T08:34:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.255+0000 7fc9f1ffb640 1 -- 172.21.15.107:0/3690281401 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca0009bab0 con 0x557eeddd1320 2024-05-09T08:34:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.255+0000 7fc9f1ffb640 1 -- 172.21.15.107:0/3690281401 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0006e920 con 0x557eeddd1320 2024-05-09T08:34:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.256+0000 7fca14322740 1 -- 172.21.15.107:0/3690281401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddd1320 msgr2=0x557eeddd3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.256+0000 7fca14322740 1 --2- 172.21.15.107:0/3690281401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddd1320 0x557eeddd3710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fca000cc5e0 tx=0x7fca00097780 comp rx=0 tx=0).stop 2024-05-09T08:34:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.256+0000 7fca14322740 1 -- 172.21.15.107:0/3690281401 shutdown_connections 2024-05-09T08:34:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.256+0000 7fca14322740 1 --2- 172.21.15.107:0/3690281401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddd1320 0x557eeddd3710 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.256+0000 7fca14322740 1 --2- 172.21.15.107:0/3690281401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddc6c50 0x557eeddd0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.256+0000 7fca14322740 1 --2- 172.21.15.107:0/3690281401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 0x557eeddc6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.256+0000 7fca14322740 1 -- 172.21.15.107:0/3690281401 >> 172.21.15.107:0/3690281401 conn(0x557eeddc2650 msgr2=0x557eeddc2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.256+0000 7fca14322740 1 -- 172.21.15.107:0/3690281401 shutdown_connections 2024-05-09T08:34:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.256+0000 7fca14322740 1 -- 172.21.15.107:0/3690281401 wait complete. 2024-05-09T08:34:32.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.257+0000 7fca14322740 1 -- start start 2024-05-09T08:34:32.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.257+0000 7fca14322740 1 Processor -- start 2024-05-09T08:34:32.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.257+0000 7fca14322740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 0x557eeddce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.257+0000 7fca14322740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddc6c50 0x557eeddce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.257+0000 7fca14322740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddced40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.258+0000 7fca14322740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557eeddd4050 con 0x557eeddd1320 2024-05-09T08:34:32.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.258+0000 7fca0b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.258+0000 7fca0b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddced40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39096/0 (socket says 172.21.15.107:39096) 2024-05-09T08:34:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.258+0000 7fca0b7fe640 1 -- 172.21.15.107:0/1191657541 learned_addr learned my addr 172.21.15.107:0/1191657541 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.258+0000 7fca14322740 1 -- 172.21.15.107:0/1191657541 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557eedce6c40 con 0x557eeddc6330 2024-05-09T08:34:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.258+0000 7fca14322740 1 -- 172.21.15.107:0/1191657541 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x557eeddd41d0 con 0x557eeddc6c50 2024-05-09T08:34:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.258+0000 7fca0a7fc640 1 --2- 172.21.15.107:0/1191657541 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddc6c50 0x557eeddce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.258+0000 7fca0affd640 1 --2- 172.21.15.107:0/1191657541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 0x557eeddce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.258+0000 7fca14322740 10 client.0 _ll_register_callbacks cb 0x557eeda3ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.258+0000 7fca0b7fe640 1 -- 172.21.15.107:0/1191657541 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddc6c50 msgr2=0x557eeddce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.258+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/1191657541 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddc6c50 0x557eeddce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.258+0000 7fca0b7fe640 1 -- 172.21.15.107:0/1191657541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 msgr2=0x557eeddce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.258+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/1191657541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 0x557eeddce2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.258+0000 7fca0b7fe640 1 -- 172.21.15.107:0/1191657541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557eedb097d0 con 0x557eeddd1320 2024-05-09T08:34:32.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.258+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/1191657541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddced40 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fca040092c0 tx=0x7fca04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f2d1048f25d67ea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.258+0000 7fc9eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.259+0000 7fc9eaffd640 1 -- 172.21.15.107:0/1191657541 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca04025020 con 0x557eeddd1320 2024-05-09T08:34:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.259+0000 7fca14322740 1 -- 172.21.15.107:0/1191657541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca0009bab0 con 0x557eeddd1320 2024-05-09T08:34:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.259+0000 7fc9eaffd640 1 -- 172.21.15.107:0/1191657541 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fca04011d40 con 0x557eeddd1320 2024-05-09T08:34:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.259+0000 7fc9eaffd640 1 -- 172.21.15.107:0/1191657541 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fca04019070 con 0x557eeddd1320 2024-05-09T08:34:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.259+0000 7fca14322740 10 client.15999 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.259+0000 7fca14322740 1 -- 172.21.15.107:0/1191657541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557eedde02e0 con 0x557eeddd1320 2024-05-09T08:34:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.259+0000 7fc9eaffd640 1 -- 172.21.15.107:0/1191657541 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca0009bab0 con 0x557eeddd1320 2024-05-09T08:34:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.260+0000 7fc9eaffd640 1 -- 172.21.15.107:0/1191657541 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fca040103d0 con 0x557eeddd1320 2024-05-09T08:34:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.260+0000 7fca14322740 10 client.15999 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.260+0000 7fca14322740 10 client.15999 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:34:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.260+0000 7fca14322740 20 client.15999 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.260+0000 7fca14322740 1 --2- 172.21.15.107:0/1191657541 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x557eedde1740 0x557eedde3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.260+0000 7fca14322740 4 client.15999 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:34:32.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.260+0000 7fca14322740 1 -- 172.21.15.107:0/1191657541 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557eedd39370 con 0x557eedde1740 2024-05-09T08:34:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.260+0000 7fca0affd640 1 --2- 172.21.15.107:0/1191657541 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x557eedde1740 0x557eedde3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.261+0000 7fc9eaffd640 10 client.15999 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:34:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.261+0000 7fca0affd640 1 --2- 172.21.15.107:0/1191657541 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x557eedde1740 0x557eedde3be0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f3e6fd1db4519fe server_cookie=43cde9b3f81df62e in_seq=0 out_seq=0 2024-05-09T08:34:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.261+0000 7fc9eaffd640 1 -- 172.21.15.107:0/1191657541 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557eedd39370 con 0x557eedde1740 2024-05-09T08:34:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.261+0000 7fc9eaffd640 10 client.15999 handle_command_reply: tid=0 2024-05-09T08:34:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.261+0000 7fca14322740 1 client.15999 shutdown 2024-05-09T08:34:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.262+0000 7fca14322740 1 -- 172.21.15.107:0/1191657541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 msgr2=0x557eeddced40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.262+0000 7fca14322740 1 --2- 172.21.15.107:0/1191657541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddced40 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fca040092c0 tx=0x7fca04004a40 comp rx=0 tx=0).stop 2024-05-09T08:34:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.262+0000 7fca14322740 1 -- 172.21.15.107:0/1191657541 shutdown_connections 2024-05-09T08:34:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.262+0000 7fca14322740 1 --2- 172.21.15.107:0/1191657541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddced40 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.262+0000 7fca14322740 1 --2- 172.21.15.107:0/1191657541 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x557eedde1740 0x557eedde3be0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.262+0000 7fca14322740 1 --2- 172.21.15.107:0/1191657541 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddc6c50 0x557eeddce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.262+0000 7fc9eaffd640 0 client.15999 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:34:32.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.262+0000 7fca14322740 1 --2- 172.21.15.107:0/1191657541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 0x557eeddce2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.262+0000 7fca14322740 1 -- 172.21.15.107:0/1191657541 >> 172.21.15.107:0/1191657541 conn(0x557eeddc2650 msgr2=0x557eeddc8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.262+0000 7fca14322740 1 -- 172.21.15.107:0/1191657541 shutdown_connections 2024-05-09T08:34:32.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.262+0000 7fca14322740 1 -- 172.21.15.107:0/1191657541 wait complete. 2024-05-09T08:34:32.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.262+0000 7fca14322740 20 client.15999 trim_cache size 0 max 16384 2024-05-09T08:34:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca14322740 1 Processor -- start 2024-05-09T08:34:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca14322740 1 -- start start 2024-05-09T08:34:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca14322740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddc6330 0x557eedddcf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca14322740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6c50 0x557eeddcdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca14322740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddc8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca14322740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c1af910 con 0x557eeddd1320 2024-05-09T08:34:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca14322740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c1ad1e0 con 0x557eeddc6c50 2024-05-09T08:34:32.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca14322740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c1ab030 con 0x557eeddc6330 2024-05-09T08:34:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca0b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddc8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca0b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddc8ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39102/0 (socket says 172.21.15.107:39102) 2024-05-09T08:34:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca0b7fe640 1 -- 172.21.15.107:0/1508038994 learned_addr learned my addr 172.21.15.107:0/1508038994 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca0affd640 1 --2- 172.21.15.107:0/1508038994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddc6330 0x557eedddcf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca0a7fc640 1 --2- 172.21.15.107:0/1508038994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6c50 0x557eeddcdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca0b7fe640 1 -- 172.21.15.107:0/1508038994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddc6330 msgr2=0x557eedddcf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/1508038994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddc6330 0x557eedddcf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca0b7fe640 1 -- 172.21.15.107:0/1508038994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6c50 msgr2=0x557eeddcdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/1508038994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6c50 0x557eeddcdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.288+0000 7fca0b7fe640 1 -- 172.21.15.107:0/1508038994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557eeddd3a80 con 0x557eeddd1320 2024-05-09T08:34:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.289+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/1508038994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddc8ad0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fca04004a00 tx=0x7fca04009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22cb194bfcd05f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.289+0000 7fc9f17fa640 1 -- 172.21.15.107:0/1508038994 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca04025040 con 0x557eeddd1320 2024-05-09T08:34:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.289+0000 7fc9f17fa640 1 -- 172.21.15.107:0/1508038994 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca0401a040 con 0x557eeddd1320 2024-05-09T08:34:32.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.289+0000 7fc9f17fa640 1 -- 172.21.15.107:0/1508038994 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca040346a0 con 0x557eeddd1320 2024-05-09T08:34:32.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.290+0000 7fca14322740 1 -- 172.21.15.107:0/1508038994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 msgr2=0x557eeddc8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.290+0000 7fca14322740 1 --2- 172.21.15.107:0/1508038994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddc8ad0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fca04004a00 tx=0x7fca04009bd0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.290+0000 7fca14322740 1 -- 172.21.15.107:0/1508038994 shutdown_connections 2024-05-09T08:34:32.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.290+0000 7fca14322740 1 --2- 172.21.15.107:0/1508038994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddc8ad0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.290+0000 7fca14322740 1 --2- 172.21.15.107:0/1508038994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6c50 0x557eeddcdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.290+0000 7fca14322740 1 --2- 172.21.15.107:0/1508038994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddc6330 0x557eedddcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.290+0000 7fca14322740 1 -- 172.21.15.107:0/1508038994 >> 172.21.15.107:0/1508038994 conn(0x557eeddc2650 msgr2=0x557eeddc2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.290+0000 7fca14322740 1 -- 172.21.15.107:0/1508038994 shutdown_connections 2024-05-09T08:34:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.290+0000 7fca14322740 1 -- 172.21.15.107:0/1508038994 wait complete. 2024-05-09T08:34:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.290+0000 7fca14322740 1 -- start start 2024-05-09T08:34:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.291+0000 7fca14322740 1 Processor -- start 2024-05-09T08:34:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.291+0000 7fca14322740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 0x557eeddd9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.291+0000 7fca14322740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddc6c50 0x557eeddd9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.291+0000 7fca14322740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.291+0000 7fca14322740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c1ab6a0 con 0x557eeddd1320 2024-05-09T08:34:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.291+0000 7fca14322740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c1acb10 con 0x557eeddc6330 2024-05-09T08:34:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.291+0000 7fca14322740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c1abd70 con 0x557eeddc6c50 2024-05-09T08:34:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.291+0000 7fca14322740 10 client.0 _ll_register_callbacks cb 0x7fca04005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.291+0000 7fca0b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.291+0000 7fca0b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39106/0 (socket says 172.21.15.107:39106) 2024-05-09T08:34:32.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.291+0000 7fca0b7fe640 1 -- 172.21.15.107:0/2407704111 learned_addr learned my addr 172.21.15.107:0/2407704111 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:32.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.291+0000 7fca0affd640 1 --2- 172.21.15.107:0/2407704111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 0x557eeddd9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.291+0000 7fca0a7fc640 1 --2- 172.21.15.107:0/2407704111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddc6c50 0x557eeddd9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.292+0000 7fca0b7fe640 1 -- 172.21.15.107:0/2407704111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddc6c50 msgr2=0x557eeddd9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.292+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/2407704111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddc6c50 0x557eeddd9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.292+0000 7fca0b7fe640 1 -- 172.21.15.107:0/2407704111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 msgr2=0x557eeddd9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.292+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/2407704111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 0x557eeddd9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.292+0000 7fca0b7fe640 1 -- 172.21.15.107:0/2407704111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557eeddcaf50 con 0x557eeddd1320 2024-05-09T08:34:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.292+0000 7fca0a7fc640 1 --2- 172.21.15.107:0/2407704111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddc6c50 0x557eeddd9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:34:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.292+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/2407704111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddca750 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fca04005f00 tx=0x7fca04011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd3c1241f0b7dc60 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.292+0000 7fc9eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.292+0000 7fc9eaffd640 1 -- 172.21.15.107:0/2407704111 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca04041020 con 0x557eeddd1320 2024-05-09T08:34:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.292+0000 7fca14322740 1 -- 172.21.15.107:0/2407704111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca0401a040 con 0x557eeddd1320 2024-05-09T08:34:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.292+0000 7fc9eaffd640 1 -- 172.21.15.107:0/2407704111 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fca0403d670 con 0x557eeddd1320 2024-05-09T08:34:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.292+0000 7fc9eaffd640 1 -- 172.21.15.107:0/2407704111 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fca04025070 con 0x557eeddd1320 2024-05-09T08:34:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.292+0000 7fca14322740 10 client.16005 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.292+0000 7fca14322740 1 -- 172.21.15.107:0/2407704111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557eede47730 con 0x557eeddd1320 2024-05-09T08:34:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.292+0000 7fc9eaffd640 1 -- 172.21.15.107:0/2407704111 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca0401a040 con 0x557eeddd1320 2024-05-09T08:34:32.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.293+0000 7fc9eaffd640 1 -- 172.21.15.107:0/2407704111 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fca04034440 con 0x557eeddd1320 2024-05-09T08:34:32.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.293+0000 7fca14322740 10 client.16005 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:32.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.293+0000 7fca14322740 10 client.16005 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:34:32.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.293+0000 7fca14322740 20 client.16005 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:32.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.293+0000 7fca14322740 1 --2- 172.21.15.107:0/2407704111 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x557eedde1740 0x557eede49df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.293+0000 7fca14322740 4 client.16005 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:34:32.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.293+0000 7fca14322740 1 -- 172.21.15.107:0/2407704111 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557eedd336c0 con 0x557eedde1740 2024-05-09T08:34:32.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.293+0000 7fca0affd640 1 --2- 172.21.15.107:0/2407704111 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x557eedde1740 0x557eede49df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.294+0000 7fca0affd640 1 --2- 172.21.15.107:0/2407704111 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x557eedde1740 0x557eede49df0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8be6f8ada20705b3 server_cookie=16e71e81e36bb0b9 in_seq=0 out_seq=0 2024-05-09T08:34:32.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.294+0000 7fc9eaffd640 10 client.16005 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:34:32.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.294+0000 7fc9eaffd640 1 -- 172.21.15.107:0/2407704111 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557eedd336c0 con 0x557eedde1740 2024-05-09T08:34:32.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.294+0000 7fc9eaffd640 10 client.16005 handle_command_reply: tid=0 2024-05-09T08:34:32.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.294+0000 7fca14322740 1 client.16005 shutdown 2024-05-09T08:34:32.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.294+0000 7fca14322740 1 -- 172.21.15.107:0/2407704111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 msgr2=0x557eeddca750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.294+0000 7fca14322740 1 --2- 172.21.15.107:0/2407704111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddca750 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fca04005f00 tx=0x7fca04011d70 comp rx=0 tx=0).stop 2024-05-09T08:34:32.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.294+0000 7fca14322740 1 -- 172.21.15.107:0/2407704111 shutdown_connections 2024-05-09T08:34:32.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.294+0000 7fca14322740 1 --2- 172.21.15.107:0/2407704111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557eeddd1320 0x557eeddca750 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.294+0000 7fca14322740 1 --2- 172.21.15.107:0/2407704111 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x557eedde1740 0x557eede49df0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.294+0000 7fca14322740 1 --2- 172.21.15.107:0/2407704111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557eeddc6c50 0x557eeddd9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.294+0000 7fca14322740 1 --2- 172.21.15.107:0/2407704111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557eeddc6330 0x557eeddd9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.294+0000 7fca14322740 1 -- 172.21.15.107:0/2407704111 >> 172.21.15.107:0/2407704111 conn(0x557eeddc2650 msgr2=0x557eeddceb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.294+0000 7fc9eaffd640 0 client.16005 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:34:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.294+0000 7fca14322740 1 -- 172.21.15.107:0/2407704111 shutdown_connections 2024-05-09T08:34:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.295+0000 7fca14322740 1 -- 172.21.15.107:0/2407704111 wait complete. 2024-05-09T08:34:32.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.295+0000 7fca14322740 20 client.16005 trim_cache size 0 max 16384 2024-05-09T08:34:32.301 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:34:32.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.295+0000 7fca11970640 1 -- 172.21.15.107:0/3814678571 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9f40a2eb0 msgr2=0x7fc9f40a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.295+0000 7fca11970640 1 --2- 172.21.15.107:0/3814678571 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9f40a2eb0 0x7fc9f40a5370 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fca04009370 tx=0x7fca04004ab0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.295+0000 7fca11970640 1 -- 172.21.15.107:0/3814678571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c1010c0 msgr2=0x7fca0c1987e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.295+0000 7fca11970640 1 --2- 172.21.15.107:0/3814678571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c1010c0 0x7fca0c1987e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fca0000c890 tx=0x7fca0000cd50 comp rx=0 tx=0).stop 2024-05-09T08:34:32.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.295+0000 7fca11970640 1 -- 172.21.15.107:0/3814678571 shutdown_connections 2024-05-09T08:34:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.295+0000 7fca11970640 1 --2- 172.21.15.107:0/3814678571 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9f40a2eb0 0x7fc9f40a5370 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.295+0000 7fca11970640 1 --2- 172.21.15.107:0/3814678571 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca0c19cb60 0x7fca0c19d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.295+0000 7fca11970640 1 --2- 172.21.15.107:0/3814678571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c1010c0 0x7fca0c1987e0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.295+0000 7fca11970640 1 --2- 172.21.15.107:0/3814678571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca0c100700 0x7fca0c1982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.295+0000 7fca11970640 1 -- 172.21.15.107:0/3814678571 >> 172.21.15.107:0/3814678571 conn(0x7fca0c077830 msgr2=0x7fca0c107560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.295+0000 7fca11970640 1 -- 172.21.15.107:0/3814678571 shutdown_connections 2024-05-09T08:34:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.295+0000 7fca11970640 1 -- 172.21.15.107:0/3814678571 wait complete. 2024-05-09T08:34:32.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:34:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.413+0000 7f62b4038640 1 -- 172.21.15.107:0/3481958379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62ac102ab0 msgr2=0x7f62ac102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.413+0000 7f62b4038640 1 --2- 172.21.15.107:0/3481958379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62ac102ab0 0x7f62ac102e90 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f62a8009900 tx=0x7f62a802a510 comp rx=0 tx=0).stop 2024-05-09T08:34:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.413+0000 7f62b4038640 1 -- 172.21.15.107:0/3481958379 shutdown_connections 2024-05-09T08:34:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.413+0000 7f62b4038640 1 --2- 172.21.15.107:0/3481958379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62ac10bdf0 0x7f62ac10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.413+0000 7f62b4038640 1 --2- 172.21.15.107:0/3481958379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62ac1033d0 0x7f62ac10b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.413+0000 7f62b4038640 1 --2- 172.21.15.107:0/3481958379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62ac102ab0 0x7f62ac102e90 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.413+0000 7f62b4038640 1 -- 172.21.15.107:0/3481958379 >> 172.21.15.107:0/3481958379 conn(0x7f62ac077830 msgr2=0x7f62ac077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.413+0000 7f62b4038640 1 -- 172.21.15.107:0/3481958379 shutdown_connections 2024-05-09T08:34:32.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.413+0000 7f62b4038640 1 -- 172.21.15.107:0/3481958379 wait complete. 2024-05-09T08:34:32.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.414+0000 7f62b4038640 1 Processor -- start 2024-05-09T08:34:32.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.414+0000 7f62b4038640 1 -- start start 2024-05-09T08:34:32.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.414+0000 7f62b4038640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62ac102ab0 0x7f62ac1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.414+0000 7f62b4038640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62ac1033d0 0x7f62ac198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.414+0000 7f62b4038640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62ac10bdf0 0x7f62ac19c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.414+0000 7f62b4038640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac111aa0 con 0x7f62ac102ab0 2024-05-09T08:34:32.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.414+0000 7f62b4038640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac111920 con 0x7f62ac10bdf0 2024-05-09T08:34:32.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.414+0000 7f62b4038640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac111c20 con 0x7f62ac1033d0 2024-05-09T08:34:32.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.414+0000 7f62b1dd4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62ac102ab0 0x7f62ac1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.414+0000 7f62b1dd4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62ac102ab0 0x7f62ac1980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39130/0 (socket says 172.21.15.107:39130) 2024-05-09T08:34:32.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.414+0000 7f62b1dd4640 1 -- 172.21.15.107:0/1073550245 learned_addr learned my addr 172.21.15.107:0/1073550245 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:32.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.414+0000 7f62b15d3640 1 --2- 172.21.15.107:0/1073550245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62ac1033d0 0x7f62ac198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.415+0000 7f62b25d5640 1 --2- 172.21.15.107:0/1073550245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62ac10bdf0 0x7f62ac19c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.415+0000 7f62b1dd4640 1 -- 172.21.15.107:0/1073550245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62ac1033d0 msgr2=0x7f62ac198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.415+0000 7f62b1dd4640 1 --2- 172.21.15.107:0/1073550245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62ac1033d0 0x7f62ac198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.415+0000 7f62b1dd4640 1 -- 172.21.15.107:0/1073550245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62ac10bdf0 msgr2=0x7f62ac19c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.415+0000 7f62b1dd4640 1 --2- 172.21.15.107:0/1073550245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62ac10bdf0 0x7f62ac19c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.415+0000 7f62b1dd4640 1 -- 172.21.15.107:0/1073550245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ac19d0c0 con 0x7f62ac102ab0 2024-05-09T08:34:32.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.415+0000 7f62b1dd4640 1 --2- 172.21.15.107:0/1073550245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62ac102ab0 0x7f62ac1980d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f62a8005940 tx=0x7f62a80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.415+0000 7f62a2ffd640 1 -- 172.21.15.107:0/1073550245 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a803f030 con 0x7f62ac102ab0 2024-05-09T08:34:32.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.415+0000 7f62b4038640 1 -- 172.21.15.107:0/1073550245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ac19d350 con 0x7f62ac102ab0 2024-05-09T08:34:32.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.415+0000 7f62a2ffd640 1 -- 172.21.15.107:0/1073550245 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62a80044c0 con 0x7f62ac102ab0 2024-05-09T08:34:32.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.415+0000 7f62b4038640 1 -- 172.21.15.107:0/1073550245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ac1a4c40 con 0x7f62ac102ab0 2024-05-09T08:34:32.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.416+0000 7f62a2ffd640 1 -- 172.21.15.107:0/1073550245 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8035070 con 0x7f62ac102ab0 2024-05-09T08:34:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.418+0000 7f62b6e29740 1 Processor -- start 2024-05-09T08:34:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.418+0000 7f62b6e29740 1 -- start start 2024-05-09T08:34:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62b6e29740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be44330 0x55638be44710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62b6e29740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be44c50 0x55638be4ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62b6e29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be4f320 0x55638be51710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62b6e29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55638bda9c40 con 0x55638be4f320 2024-05-09T08:34:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62b6e29740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55638bd51580 con 0x55638be44c50 2024-05-09T08:34:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62b6e29740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55638bda5c00 con 0x55638be44330 2024-05-09T08:34:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62b15d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be4f320 0x55638be51710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62b15d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be4f320 0x55638be51710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39144/0 (socket says 172.21.15.107:39144) 2024-05-09T08:34:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62b15d3640 1 -- 172.21.15.107:0/2138835152 learned_addr learned my addr 172.21.15.107:0/2138835152 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:32.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62b25d5640 1 --2- 172.21.15.107:0/2138835152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be44330 0x55638be44710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62b15d3640 1 -- 172.21.15.107:0/2138835152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be44330 msgr2=0x55638be44710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62b15d3640 1 --2- 172.21.15.107:0/2138835152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be44330 0x55638be44710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62b15d3640 1 -- 172.21.15.107:0/2138835152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be44c50 msgr2=0x55638be4ede0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:34:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62b15d3640 1 --2- 172.21.15.107:0/2138835152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be44c50 0x55638be4ede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62b15d3640 1 -- 172.21.15.107:0/2138835152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55638babd3f0 con 0x55638be4f320 2024-05-09T08:34:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62b25d5640 1 --2- 172.21.15.107:0/2138835152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be44330 0x55638be44710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:34:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.419+0000 7f62a2ffd640 1 -- 172.21.15.107:0/1073550245 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 138118+0+0 (secure 0 0 0) 0x7f62a8042030 con 0x7f62ac102ab0 2024-05-09T08:34:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.420+0000 7f62b15d3640 1 --2- 172.21.15.107:0/2138835152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be4f320 0x55638be51710 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f629c009c90 tx=0x7f629c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410b8d69b6b52650 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.420+0000 7f6297fff640 1 -- 172.21.15.107:0/2138835152 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629c01e070 con 0x55638be4f320 2024-05-09T08:34:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.420+0000 7f6297fff640 1 -- 172.21.15.107:0/2138835152 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f629c019ae0 con 0x55638be4f320 2024-05-09T08:34:32.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.420+0000 7f6297fff640 1 -- 172.21.15.107:0/2138835152 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629c00e970 con 0x55638be4f320 2024-05-09T08:34:32.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.421+0000 7f62a2ffd640 1 --2- 172.21.15.107:0/1073550245 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f62980a2eb0 0x7f62980a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.421+0000 7f62b6e29740 1 -- 172.21.15.107:0/2138835152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be4f320 msgr2=0x55638be51710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.421+0000 7f62b6e29740 1 --2- 172.21.15.107:0/2138835152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be4f320 0x55638be51710 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f629c009c90 tx=0x7f629c011810 comp rx=0 tx=0).stop 2024-05-09T08:34:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.421+0000 7f62a2ffd640 1 -- 172.21.15.107:0/1073550245 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f62a80efe80 con 0x7f62ac102ab0 2024-05-09T08:34:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.421+0000 7f62b25d5640 1 --2- 172.21.15.107:0/1073550245 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f62980a2eb0 0x7f62980a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.421+0000 7f62b6e29740 1 -- 172.21.15.107:0/2138835152 shutdown_connections 2024-05-09T08:34:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.421+0000 7f62b6e29740 1 --2- 172.21.15.107:0/2138835152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be4f320 0x55638be51710 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.421+0000 7f62b6e29740 1 --2- 172.21.15.107:0/2138835152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be44c50 0x55638be4ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.421+0000 7f62b6e29740 1 --2- 172.21.15.107:0/2138835152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be44330 0x55638be44710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.421+0000 7f62b6e29740 1 -- 172.21.15.107:0/2138835152 >> 172.21.15.107:0/2138835152 conn(0x55638be40650 msgr2=0x55638be40a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.421+0000 7f62b25d5640 1 --2- 172.21.15.107:0/1073550245 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f62980a2eb0 0x7f62980a5370 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f62a4005fd0 tx=0x7f62a4009500 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.421+0000 7f62b6e29740 1 -- 172.21.15.107:0/2138835152 shutdown_connections 2024-05-09T08:34:32.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.421+0000 7f62b6e29740 1 -- 172.21.15.107:0/2138835152 wait complete. 2024-05-09T08:34:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.422+0000 7f62b6e29740 1 -- start start 2024-05-09T08:34:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.422+0000 7f62b6e29740 1 Processor -- start 2024-05-09T08:34:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.422+0000 7f62b6e29740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be44330 0x55638be4eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.422+0000 7f62b6e29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be44c50 0x55638be47ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.422+0000 7f62b6e29740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be4f320 0x55638be48410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.422+0000 7f62b6e29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55638be56520 con 0x55638be44c50 2024-05-09T08:34:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.422+0000 7f62b6e29740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55638bd64c40 con 0x55638be44330 2024-05-09T08:34:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.422+0000 7f62b6e29740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55638be566a0 con 0x55638be4f320 2024-05-09T08:34:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.423+0000 7f62b6e29740 10 client.0 _ll_register_callbacks cb 0x55638babcbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.423+0000 7f62b15d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be44c50 0x55638be47ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.423+0000 7f62b15d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be44c50 0x55638be47ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39148/0 (socket says 172.21.15.107:39148) 2024-05-09T08:34:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.423+0000 7f62b15d3640 1 -- 172.21.15.107:0/1894524896 learned_addr learned my addr 172.21.15.107:0/1894524896 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.423+0000 7f62b25d5640 1 --2- 172.21.15.107:0/1894524896 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be4f320 0x55638be48410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.423+0000 7f62b1dd4640 1 --2- 172.21.15.107:0/1894524896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be44330 0x55638be4eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.423+0000 7f62b15d3640 1 -- 172.21.15.107:0/1894524896 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be4f320 msgr2=0x55638be48410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.423+0000 7f62b15d3640 1 --2- 172.21.15.107:0/1894524896 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be4f320 0x55638be48410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.423+0000 7f62b15d3640 1 -- 172.21.15.107:0/1894524896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be44330 msgr2=0x55638be4eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.423+0000 7f62b15d3640 1 --2- 172.21.15.107:0/1894524896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be44330 0x55638be4eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.423+0000 7f62b15d3640 1 -- 172.21.15.107:0/1894524896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55638bb877d0 con 0x55638be44c50 2024-05-09T08:34:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.423+0000 7f62b15d3640 1 --2- 172.21.15.107:0/1894524896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be44c50 0x55638be47ed0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f629c009890 tx=0x7f629c019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83709a309d77c32 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.423+0000 7f62957fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.423+0000 7f62957fa640 1 -- 172.21.15.107:0/1894524896 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629c01e040 con 0x55638be44c50 2024-05-09T08:34:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.424+0000 7f62957fa640 1 -- 172.21.15.107:0/1894524896 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f629c019850 con 0x55638be44c50 2024-05-09T08:34:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.424+0000 7f62b6e29740 1 -- 172.21.15.107:0/1894524896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f629c019ae0 con 0x55638be44c50 2024-05-09T08:34:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.424+0000 7f62957fa640 1 -- 172.21.15.107:0/1894524896 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f629c033050 con 0x55638be44c50 2024-05-09T08:34:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.424+0000 7f62b6e29740 10 client.16017 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.424+0000 7f62b6e29740 1 -- 172.21.15.107:0/1894524896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55638be66b20 con 0x55638be44c50 2024-05-09T08:34:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.424+0000 7f62957fa640 1 -- 172.21.15.107:0/1894524896 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f629c019ae0 con 0x55638be44c50 2024-05-09T08:34:32.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.424+0000 7f62957fa640 1 -- 172.21.15.107:0/1894524896 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f629c0046c0 con 0x55638be44c50 2024-05-09T08:34:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.424+0000 7f62b6e29740 10 client.16017 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.424+0000 7f62b6e29740 10 client.16017 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.424+0000 7f62b6e29740 20 client.16017 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.424+0000 7f62b6e29740 1 --2- 172.21.15.107:0/1894524896 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55638be67f80 0x55638be6a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.424+0000 7f62b6e29740 4 client.16017 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:34:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.424+0000 7f62b6e29740 1 -- 172.21.15.107:0/1894524896 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55638bdb7370 con 0x55638be67f80 2024-05-09T08:34:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.424+0000 7f62b1dd4640 1 --2- 172.21.15.107:0/1894524896 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55638be67f80 0x55638be6a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.425+0000 7f62b1dd4640 1 --2- 172.21.15.107:0/1894524896 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55638be67f80 0x55638be6a370 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13abbfdb92d31dc7 server_cookie=2b4a54ee0cf30680 in_seq=0 out_seq=0 2024-05-09T08:34:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.425+0000 7f62957fa640 10 client.16017 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.426+0000 7f62957fa640 1 -- 172.21.15.107:0/1894524896 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55638bdb7370 con 0x55638be67f80 2024-05-09T08:34:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.426+0000 7f62957fa640 10 client.16017 handle_command_reply: tid=0 2024-05-09T08:34:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.426+0000 7f62b6e29740 1 client.16017 shutdown 2024-05-09T08:34:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.426+0000 7f62b6e29740 1 -- 172.21.15.107:0/1894524896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be44c50 msgr2=0x55638be47ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.426+0000 7f62b6e29740 1 --2- 172.21.15.107:0/1894524896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be44c50 0x55638be47ed0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f629c009890 tx=0x7f629c019660 comp rx=0 tx=0).stop 2024-05-09T08:34:32.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.426+0000 7f62b6e29740 1 -- 172.21.15.107:0/1894524896 shutdown_connections 2024-05-09T08:34:32.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.426+0000 7f62b6e29740 1 --2- 172.21.15.107:0/1894524896 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55638be67f80 0x55638be6a370 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.426+0000 7f62b6e29740 1 --2- 172.21.15.107:0/1894524896 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be4f320 0x55638be48410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.426+0000 7f62b6e29740 1 --2- 172.21.15.107:0/1894524896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be44c50 0x55638be47ed0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.426+0000 7f62957fa640 0 client.16017 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:32.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.426+0000 7f62b6e29740 1 --2- 172.21.15.107:0/1894524896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be44330 0x55638be4eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.426+0000 7f62b6e29740 1 -- 172.21.15.107:0/1894524896 >> 172.21.15.107:0/1894524896 conn(0x55638be40650 msgr2=0x55638be46450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.426+0000 7f62b6e29740 1 -- 172.21.15.107:0/1894524896 shutdown_connections 2024-05-09T08:34:32.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.426+0000 7f62b6e29740 1 -- 172.21.15.107:0/1894524896 wait complete. 2024-05-09T08:34:32.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.426+0000 7f62b6e29740 20 client.16017 trim_cache size 0 max 16384 2024-05-09T08:34:32.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.440+0000 7f62b6e29740 1 Processor -- start 2024-05-09T08:34:32.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.440+0000 7f62b6e29740 1 -- start start 2024-05-09T08:34:32.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.440+0000 7f62b6e29740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be40650 0x55638be47e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.440+0000 7f62b6e29740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be44330 0x55638be6fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.440+0000 7f62b6e29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be44c50 0x55638be6ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.440+0000 7f62b6e29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac1af260 con 0x55638be44c50 2024-05-09T08:34:32.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.440+0000 7f62b6e29740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac1acb30 con 0x55638be40650 2024-05-09T08:34:32.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.440+0000 7f62b6e29740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac1aa980 con 0x55638be44330 2024-05-09T08:34:32.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.440+0000 7f62b25d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be44c50 0x55638be6ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.440+0000 7f62b25d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be44c50 0x55638be6ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39162/0 (socket says 172.21.15.107:39162) 2024-05-09T08:34:32.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.440+0000 7f62b25d5640 1 -- 172.21.15.107:0/4112345046 learned_addr learned my addr 172.21.15.107:0/4112345046 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:32.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.440+0000 7f62b15d3640 1 --2- 172.21.15.107:0/4112345046 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be44330 0x55638be6fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.440+0000 7f62b1dd4640 1 --2- 172.21.15.107:0/4112345046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be40650 0x55638be47e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.441+0000 7f62b25d5640 1 -- 172.21.15.107:0/4112345046 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be44330 msgr2=0x55638be6fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.441+0000 7f62b25d5640 1 --2- 172.21.15.107:0/4112345046 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be44330 0x55638be6fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.441+0000 7f62b25d5640 1 -- 172.21.15.107:0/4112345046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be40650 msgr2=0x55638be47e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.441+0000 7f62b25d5640 1 --2- 172.21.15.107:0/4112345046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be40650 0x55638be47e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.441+0000 7f62b25d5640 1 -- 172.21.15.107:0/4112345046 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55638be3bf30 con 0x55638be44c50 2024-05-09T08:34:32.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.441+0000 7f62b25d5640 1 --2- 172.21.15.107:0/4112345046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be44c50 0x55638be6ffd0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f62a40129d0 tx=0x7f62a4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40df18a5ea1b197a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.441+0000 7f62977fe640 1 -- 172.21.15.107:0/4112345046 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a4020bb0 con 0x55638be44c50 2024-05-09T08:34:32.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.441+0000 7f62977fe640 1 -- 172.21.15.107:0/4112345046 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62a4003e20 con 0x55638be44c50 2024-05-09T08:34:32.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.441+0000 7f62977fe640 1 -- 172.21.15.107:0/4112345046 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a401b3a0 con 0x55638be44c50 2024-05-09T08:34:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.442+0000 7f62b6e29740 1 -- 172.21.15.107:0/4112345046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be44c50 msgr2=0x55638be6ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.442+0000 7f62b6e29740 1 --2- 172.21.15.107:0/4112345046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be44c50 0x55638be6ffd0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f62a40129d0 tx=0x7f62a4014000 comp rx=0 tx=0).stop 2024-05-09T08:34:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.442+0000 7f62b6e29740 1 -- 172.21.15.107:0/4112345046 shutdown_connections 2024-05-09T08:34:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.442+0000 7f62b6e29740 1 --2- 172.21.15.107:0/4112345046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be44c50 0x55638be6ffd0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.442+0000 7f62b6e29740 1 --2- 172.21.15.107:0/4112345046 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be44330 0x55638be6fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.442+0000 7f62b6e29740 1 --2- 172.21.15.107:0/4112345046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be40650 0x55638be47e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.442+0000 7f62b6e29740 1 -- 172.21.15.107:0/4112345046 >> 172.21.15.107:0/4112345046 conn(0x55638be48570 msgr2=0x55638be4bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.442+0000 7f62b6e29740 1 -- 172.21.15.107:0/4112345046 shutdown_connections 2024-05-09T08:34:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.442+0000 7f62b6e29740 1 -- 172.21.15.107:0/4112345046 wait complete. 2024-05-09T08:34:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.443+0000 7f62b6e29740 1 -- start start 2024-05-09T08:34:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.443+0000 7f62b6e29740 1 Processor -- start 2024-05-09T08:34:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.443+0000 7f62b6e29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be40650 0x55638be60730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.443+0000 7f62b6e29740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be44330 0x55638be60c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.443+0000 7f62b6e29740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be44c50 0x55638becd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.443+0000 7f62b6e29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac1aaff0 con 0x55638be40650 2024-05-09T08:34:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.443+0000 7f62b6e29740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac1ac460 con 0x55638be44330 2024-05-09T08:34:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.443+0000 7f62b1dd4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be40650 0x55638be60730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.443+0000 7f62b6e29740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac1ab6c0 con 0x55638be44c50 2024-05-09T08:34:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.443+0000 7f62b1dd4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be40650 0x55638be60730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39166/0 (socket says 172.21.15.107:39166) 2024-05-09T08:34:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.443+0000 7f62b1dd4640 1 -- 172.21.15.107:0/843348773 learned_addr learned my addr 172.21.15.107:0/843348773 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.443+0000 7f62b6e29740 10 client.0 _ll_register_callbacks cb 0x7f629c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62b15d3640 1 --2- 172.21.15.107:0/843348773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be44330 0x55638be60c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62b25d5640 1 --2- 172.21.15.107:0/843348773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be44c50 0x55638becd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62b1dd4640 1 -- 172.21.15.107:0/843348773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be44c50 msgr2=0x55638becd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62b1dd4640 1 --2- 172.21.15.107:0/843348773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be44c50 0x55638becd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62b1dd4640 1 -- 172.21.15.107:0/843348773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be44330 msgr2=0x55638be60c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62b1dd4640 1 --2- 172.21.15.107:0/843348773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be44330 0x55638be60c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62b1dd4640 1 -- 172.21.15.107:0/843348773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55638becd8c0 con 0x55638be40650 2024-05-09T08:34:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62b15d3640 1 --2- 172.21.15.107:0/843348773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be44330 0x55638be60c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62b1dd4640 1 --2- 172.21.15.107:0/843348773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be40650 0x55638be60730 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f62a803b510 tx=0x7f62a80bb890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4857c308522cc91d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62957fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62957fa640 1 -- 172.21.15.107:0/843348773 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a80b8470 con 0x55638be40650 2024-05-09T08:34:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62957fa640 1 -- 172.21.15.107:0/843348773 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f62a80da6b0 con 0x55638be40650 2024-05-09T08:34:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62b6e29740 1 -- 172.21.15.107:0/843348773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62a4003e20 con 0x55638be40650 2024-05-09T08:34:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62957fa640 1 -- 172.21.15.107:0/843348773 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f62a80b8610 con 0x55638be40650 2024-05-09T08:34:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62b6e29740 10 client.16023 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62b6e29740 1 -- 172.21.15.107:0/843348773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55638beceba0 con 0x55638be40650 2024-05-09T08:34:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.444+0000 7f62957fa640 1 -- 172.21.15.107:0/843348773 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62a80da2a0 con 0x55638be40650 2024-05-09T08:34:32.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.445+0000 7f62957fa640 1 -- 172.21.15.107:0/843348773 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f629c019c90 con 0x55638be40650 2024-05-09T08:34:32.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.446+0000 7f62b6e29740 10 client.16023 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:32.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.446+0000 7f62b6e29740 10 client.16023 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:32.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.446+0000 7f62b6e29740 20 client.16023 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:32.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.446+0000 7f62b6e29740 1 --2- 172.21.15.107:0/843348773 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55638be4f320 0x55638bed12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:32.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.446+0000 7f62b6e29740 4 client.16023 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:34:32.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.446+0000 7f62b6e29740 1 -- 172.21.15.107:0/843348773 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f62a4003e20 con 0x55638be4f320 2024-05-09T08:34:32.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.446+0000 7f62b15d3640 1 --2- 172.21.15.107:0/843348773 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55638be4f320 0x55638bed12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:32.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.446+0000 7f62b15d3640 1 --2- 172.21.15.107:0/843348773 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55638be4f320 0x55638bed12d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89a025ea3bb40c5a server_cookie=5aa171024f8055c9 in_seq=0 out_seq=0 2024-05-09T08:34:32.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.446+0000 7f62957fa640 10 client.16023 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:32.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.446+0000 7f62957fa640 1 -- 172.21.15.107:0/843348773 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f62a4003e20 con 0x55638be4f320 2024-05-09T08:34:32.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.446+0000 7f62957fa640 10 client.16023 handle_command_reply: tid=0 2024-05-09T08:34:32.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.446+0000 7f62b6e29740 1 client.16023 shutdown 2024-05-09T08:34:32.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.447+0000 7f62b6e29740 1 -- 172.21.15.107:0/843348773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be40650 msgr2=0x55638be60730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.447+0000 7f62b6e29740 1 --2- 172.21.15.107:0/843348773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be40650 0x55638be60730 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f62a803b510 tx=0x7f62a80bb890 comp rx=0 tx=0).stop 2024-05-09T08:34:32.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.447+0000 7f62b6e29740 1 -- 172.21.15.107:0/843348773 shutdown_connections 2024-05-09T08:34:32.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.447+0000 7f62b6e29740 1 --2- 172.21.15.107:0/843348773 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55638be4f320 0x55638bed12d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.447+0000 7f62b6e29740 1 --2- 172.21.15.107:0/843348773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55638be44c50 0x55638becd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.447+0000 7f62957fa640 0 client.16023 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:32.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.447+0000 7f62b6e29740 1 --2- 172.21.15.107:0/843348773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55638be44330 0x55638be60c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.447+0000 7f62b6e29740 1 --2- 172.21.15.107:0/843348773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55638be40650 0x55638be60730 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.447+0000 7f62b6e29740 1 -- 172.21.15.107:0/843348773 >> 172.21.15.107:0/843348773 conn(0x55638be48570 msgr2=0x55638be3ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.447+0000 7f62b6e29740 1 -- 172.21.15.107:0/843348773 shutdown_connections 2024-05-09T08:34:32.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.447+0000 7f62b6e29740 1 -- 172.21.15.107:0/843348773 wait complete. 2024-05-09T08:34:32.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.447+0000 7f62b6e29740 20 client.16023 trim_cache size 0 max 16384 2024-05-09T08:34:32.453 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:34:32.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.448+0000 7f62b4038640 1 -- 172.21.15.107:0/1073550245 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f62980a2eb0 msgr2=0x7f62980a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.448+0000 7f62b4038640 1 --2- 172.21.15.107:0/1073550245 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f62980a2eb0 0x7f62980a5370 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f62a4005fd0 tx=0x7f62a4009500 comp rx=0 tx=0).stop 2024-05-09T08:34:32.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.448+0000 7f62b4038640 1 -- 172.21.15.107:0/1073550245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62ac102ab0 msgr2=0x7f62ac1980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:32.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.448+0000 7f62b4038640 1 --2- 172.21.15.107:0/1073550245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62ac102ab0 0x7f62ac1980d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f62a8005940 tx=0x7f62a80040c0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.448+0000 7f62b4038640 1 -- 172.21.15.107:0/1073550245 shutdown_connections 2024-05-09T08:34:32.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.448+0000 7f62b4038640 1 --2- 172.21.15.107:0/1073550245 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f62980a2eb0 0x7f62980a5370 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.448+0000 7f62b4038640 1 --2- 172.21.15.107:0/1073550245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62ac10bdf0 0x7f62ac19c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.448+0000 7f62b4038640 1 --2- 172.21.15.107:0/1073550245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62ac1033d0 0x7f62ac198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.448+0000 7f62b4038640 1 --2- 172.21.15.107:0/1073550245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62ac102ab0 0x7f62ac1980d0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:32.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.448+0000 7f62b4038640 1 -- 172.21.15.107:0/1073550245 >> 172.21.15.107:0/1073550245 conn(0x7f62ac077830 msgr2=0x7f62ac0fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:32.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.448+0000 7f62b4038640 1 -- 172.21.15.107:0/1073550245 shutdown_connections 2024-05-09T08:34:32.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:32.448+0000 7f62b4038640 1 -- 172.21.15.107:0/1073550245 wait complete. 2024-05-09T08:34:32.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:32 smithi107 ceph-mon[30122]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T08:34:32.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:32 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1027754478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:34:32.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:32 smithi157 ceph-mon[33614]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T08:34:32.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:32 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1027754478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:34:32.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:32 smithi110 ceph-mon[33622]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T08:34:32.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:32 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1027754478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:34:33.461 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:34:33.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:34:33.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:34:33.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.576+0000 7f9494d5c640 1 -- 172.21.15.107:0/964253818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f949010e1e0 msgr2=0x7f94901105d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.576+0000 7f9494d5c640 1 --2- 172.21.15.107:0/964253818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f949010e1e0 0x7f94901105d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9488009ca0 tx=0x7f948802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.576+0000 7f9494d5c640 1 -- 172.21.15.107:0/964253818 shutdown_connections 2024-05-09T08:34:33.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.576+0000 7f9494d5c640 1 --2- 172.21.15.107:0/964253818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f949010e1e0 0x7f94901105d0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.576+0000 7f9494d5c640 1 --2- 172.21.15.107:0/964253818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9490100f50 0x7f949010dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.576+0000 7f9494d5c640 1 --2- 172.21.15.107:0/964253818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9490100630 0x7f9490100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.576+0000 7f9494d5c640 1 -- 172.21.15.107:0/964253818 >> 172.21.15.107:0/964253818 conn(0x7f9490077830 msgr2=0x7f9490077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:33.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.576+0000 7f9494d5c640 1 -- 172.21.15.107:0/964253818 shutdown_connections 2024-05-09T08:34:33.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.576+0000 7f9494d5c640 1 -- 172.21.15.107:0/964253818 wait complete. 2024-05-09T08:34:33.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.577+0000 7f9494d5c640 1 Processor -- start 2024-05-09T08:34:33.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.577+0000 7f9494d5c640 1 -- start start 2024-05-09T08:34:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.577+0000 7f9494d5c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9490100630 0x7f949019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.577+0000 7f9494d5c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9490100f50 0x7f949019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.577+0000 7f9494d5c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f949010e1e0 0x7f949019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.577+0000 7f9494d5c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9490113cc0 con 0x7f949010e1e0 2024-05-09T08:34:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.578+0000 7f9494d5c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9490113b40 con 0x7f9490100630 2024-05-09T08:34:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.578+0000 7f9494d5c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9490113e40 con 0x7f9490100f50 2024-05-09T08:34:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.578+0000 7f948ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f949010e1e0 0x7f949019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.578+0000 7f948ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f949010e1e0 0x7f949019eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39190/0 (socket says 172.21.15.107:39190) 2024-05-09T08:34:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.578+0000 7f948ed9d640 1 -- 172.21.15.107:0/154781203 learned_addr learned my addr 172.21.15.107:0/154781203 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.578+0000 7f9487fff640 1 --2- 172.21.15.107:0/154781203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9490100f50 0x7f949019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.578+0000 7f948e59c640 1 --2- 172.21.15.107:0/154781203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9490100630 0x7f949019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.578+0000 7f948ed9d640 1 -- 172.21.15.107:0/154781203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9490100f50 msgr2=0x7f949019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.578+0000 7f948ed9d640 1 --2- 172.21.15.107:0/154781203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9490100f50 0x7f949019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.578+0000 7f948ed9d640 1 -- 172.21.15.107:0/154781203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9490100630 msgr2=0x7f949019a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.578+0000 7f948ed9d640 1 --2- 172.21.15.107:0/154781203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9490100630 0x7f949019a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.578+0000 7f948ed9d640 1 -- 172.21.15.107:0/154781203 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949019f1d0 con 0x7f949010e1e0 2024-05-09T08:34:33.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.578+0000 7f948e59c640 1 --2- 172.21.15.107:0/154781203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9490100630 0x7f949019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:33.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.578+0000 7f948ed9d640 1 --2- 172.21.15.107:0/154781203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f949010e1e0 0x7f949019eaa0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9488032ff0 tx=0x7f9488009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:33.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.578+0000 7f94877fe640 1 -- 172.21.15.107:0/154781203 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9488003cf0 con 0x7f949010e1e0 2024-05-09T08:34:33.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.578+0000 7f94877fe640 1 -- 172.21.15.107:0/154781203 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94880044f0 con 0x7f949010e1e0 2024-05-09T08:34:33.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.579+0000 7f94877fe640 1 -- 172.21.15.107:0/154781203 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9488030c40 con 0x7f949010e1e0 2024-05-09T08:34:33.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.579+0000 7f9494d5c640 1 -- 172.21.15.107:0/154781203 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f949019f460 con 0x7f949010e1e0 2024-05-09T08:34:33.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.579+0000 7f9494d5c640 1 -- 172.21.15.107:0/154781203 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94901a6d50 con 0x7f949010e1e0 2024-05-09T08:34:33.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.582+0000 7f94877fe640 1 -- 172.21.15.107:0/154781203 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 138118+0+0 (secure 0 0 0) 0x7f94880046a0 con 0x7f949010e1e0 2024-05-09T08:34:33.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.582+0000 7f9497a60740 1 Processor -- start 2024-05-09T08:34:33.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.582+0000 7f9497a60740 1 -- start start 2024-05-09T08:34:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.582+0000 7f9497a60740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c3b0b0 0x561494c393f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.582+0000 7f9497a60740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c399c0 0x561494c39e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.582+0000 7f9497a60740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494c46b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f9497a60740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561494b9e990 con 0x561494c44740 2024-05-09T08:34:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f9497a60740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561494b465b0 con 0x561494c399c0 2024-05-09T08:34:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f9497a60740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x561494b9a950 con 0x561494c3b0b0 2024-05-09T08:34:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f9487fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494c46b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f9487fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494c46b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39206/0 (socket says 172.21.15.107:39206) 2024-05-09T08:34:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f9487fff640 1 -- 172.21.15.107:0/263991187 learned_addr learned my addr 172.21.15.107:0/263991187 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f948e59c640 1 --2- 172.21.15.107:0/263991187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c399c0 0x561494c39e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f94877fe640 1 --2- 172.21.15.107:0/154781203 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f947c0a2eb0 0x7f947c0a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f9487fff640 1 --2- 172.21.15.107:0/263991187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c3b0b0 0x561494c393f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f948ed9d640 1 --2- 172.21.15.107:0/154781203 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f947c0a2eb0 0x7f947c0a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f9487fff640 1 -- 172.21.15.107:0/263991187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c3b0b0 msgr2=0x561494c393f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f9487fff640 1 --2- 172.21.15.107:0/263991187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c3b0b0 0x561494c393f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f9487fff640 1 -- 172.21.15.107:0/263991187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c399c0 msgr2=0x561494c39e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f9487fff640 1 --2- 172.21.15.107:0/263991187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c399c0 0x561494c39e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f9487fff640 1 -- 172.21.15.107:0/263991187 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56149497a7e0 con 0x561494c44740 2024-05-09T08:34:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f94877fe640 1 -- 172.21.15.107:0/154781203 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f94880f0f40 con 0x7f949010e1e0 2024-05-09T08:34:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.583+0000 7f9487fff640 1 --2- 172.21.15.107:0/263991187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494c46b30 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9478009f90 tx=0x7f9478009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb6086b10ef3cd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.584+0000 7f9484ff9640 1 -- 172.21.15.107:0/263991187 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9478004050 con 0x561494c44740 2024-05-09T08:34:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.584+0000 7f948ed9d640 1 --2- 172.21.15.107:0/154781203 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f947c0a2eb0 0x7f947c0a5370 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f94880ee260 tx=0x7f94880b2460 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.584+0000 7f9484ff9640 1 -- 172.21.15.107:0/263991187 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94780054d0 con 0x561494c44740 2024-05-09T08:34:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.584+0000 7f9484ff9640 1 -- 172.21.15.107:0/263991187 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9478004cc0 con 0x561494c44740 2024-05-09T08:34:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.585+0000 7f9497a60740 1 -- 172.21.15.107:0/263991187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 msgr2=0x561494c46b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.585+0000 7f9497a60740 1 --2- 172.21.15.107:0/263991187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494c46b30 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9478009f90 tx=0x7f9478009960 comp rx=0 tx=0).stop 2024-05-09T08:34:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.585+0000 7f9497a60740 1 -- 172.21.15.107:0/263991187 shutdown_connections 2024-05-09T08:34:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.585+0000 7f9497a60740 1 --2- 172.21.15.107:0/263991187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494c46b30 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.585+0000 7f9497a60740 1 --2- 172.21.15.107:0/263991187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c399c0 0x561494c39e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.585+0000 7f9497a60740 1 --2- 172.21.15.107:0/263991187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c3b0b0 0x561494c393f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.585+0000 7f9497a60740 1 -- 172.21.15.107:0/263991187 >> 172.21.15.107:0/263991187 conn(0x561494c020a0 msgr2=0x561494c35f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:33.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.585+0000 7f9497a60740 1 -- 172.21.15.107:0/263991187 shutdown_connections 2024-05-09T08:34:33.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.585+0000 7f9497a60740 1 -- 172.21.15.107:0/263991187 wait complete. 2024-05-09T08:34:33.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.585+0000 7f9497a60740 1 -- start start 2024-05-09T08:34:33.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.585+0000 7f9497a60740 1 Processor -- start 2024-05-09T08:34:33.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9497a60740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c399c0 0x561494c416f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9497a60740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c3b0b0 0x561494c41c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9497a60740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494c42170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9497a60740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561494c475d0 con 0x561494c44740 2024-05-09T08:34:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9497a60740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561494b59b40 con 0x561494c3b0b0 2024-05-09T08:34:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9497a60740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x561494c47750 con 0x561494c399c0 2024-05-09T08:34:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9497a60740 10 client.0 _ll_register_callbacks cb 0x561494b4e0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9487fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494c42170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9487fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494c42170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39218/0 (socket says 172.21.15.107:39218) 2024-05-09T08:34:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9487fff640 1 -- 172.21.15.107:0/2955264643 learned_addr learned my addr 172.21.15.107:0/2955264643 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f948e59c640 1 --2- 172.21.15.107:0/2955264643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c3b0b0 0x561494c41c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9487fff640 1 --2- 172.21.15.107:0/2955264643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c399c0 0x561494c416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9487fff640 1 -- 172.21.15.107:0/2955264643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c399c0 msgr2=0x561494c416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9487fff640 1 --2- 172.21.15.107:0/2955264643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c399c0 0x561494c416f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9487fff640 1 -- 172.21.15.107:0/2955264643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c3b0b0 msgr2=0x561494c41c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9487fff640 1 --2- 172.21.15.107:0/2955264643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c3b0b0 0x561494c41c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9487fff640 1 -- 172.21.15.107:0/2955264643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614949daf70 con 0x561494c44740 2024-05-09T08:34:33.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9487fff640 1 --2- 172.21.15.107:0/2955264643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494c42170 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f94780049a0 tx=0x7f94780045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b9226e3c248e67 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:33.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.586+0000 7f9475ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:33.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.587+0000 7f9475ffb640 1 -- 172.21.15.107:0/2955264643 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9478004e50 con 0x561494c44740 2024-05-09T08:34:33.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.587+0000 7f9475ffb640 1 -- 172.21.15.107:0/2955264643 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f947801e460 con 0x561494c44740 2024-05-09T08:34:33.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.587+0000 7f9497a60740 1 -- 172.21.15.107:0/2955264643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94780054d0 con 0x561494c44740 2024-05-09T08:34:33.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.587+0000 7f9475ffb640 1 -- 172.21.15.107:0/2955264643 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9478034050 con 0x561494c44740 2024-05-09T08:34:33.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.587+0000 7f9497a60740 10 client.16035 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:33.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.587+0000 7f9497a60740 1 -- 172.21.15.107:0/2955264643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561494872170 con 0x561494c44740 2024-05-09T08:34:33.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.587+0000 7f9475ffb640 1 -- 172.21.15.107:0/2955264643 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94780054d0 con 0x561494c44740 2024-05-09T08:34:33.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.587+0000 7f9475ffb640 1 -- 172.21.15.107:0/2955264643 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f947802c020 con 0x561494c44740 2024-05-09T08:34:33.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.587+0000 7f9497a60740 10 client.16035 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:33.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.587+0000 7f9497a60740 10 client.16035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:33.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.587+0000 7f9497a60740 20 client.16035 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:33.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.587+0000 7f9497a60740 1 --2- 172.21.15.107:0/2955264643 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561494c54760 0x561494c56b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.587+0000 7f9497a60740 4 client.16035 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:34:33.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.587+0000 7f9497a60740 1 -- 172.21.15.107:0/2955264643 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561494bac0c0 con 0x561494c54760 2024-05-09T08:34:33.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.588+0000 7f948e59c640 1 --2- 172.21.15.107:0/2955264643 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561494c54760 0x561494c56b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.588+0000 7f948e59c640 1 --2- 172.21.15.107:0/2955264643 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561494c54760 0x561494c56b50 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6852e79e147e80ed server_cookie=31eb138e38f64932 in_seq=0 out_seq=0 2024-05-09T08:34:33.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.588+0000 7f9475ffb640 10 client.16035 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:33.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.589+0000 7f9475ffb640 1 -- 172.21.15.107:0/2955264643 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561494bac0c0 con 0x561494c54760 2024-05-09T08:34:33.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.589+0000 7f9475ffb640 10 client.16035 handle_command_reply: tid=0 2024-05-09T08:34:33.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.589+0000 7f9497a60740 1 client.16035 shutdown 2024-05-09T08:34:33.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.589+0000 7f9497a60740 1 -- 172.21.15.107:0/2955264643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 msgr2=0x561494c42170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.589+0000 7f9497a60740 1 --2- 172.21.15.107:0/2955264643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494c42170 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f94780049a0 tx=0x7f94780045e0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.589+0000 7f9497a60740 1 -- 172.21.15.107:0/2955264643 shutdown_connections 2024-05-09T08:34:33.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.589+0000 7f9497a60740 1 --2- 172.21.15.107:0/2955264643 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561494c54760 0x561494c56b50 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.589+0000 7f9497a60740 1 --2- 172.21.15.107:0/2955264643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494c42170 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.589+0000 7f9497a60740 1 --2- 172.21.15.107:0/2955264643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c3b0b0 0x561494c41c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.589+0000 7f9497a60740 1 --2- 172.21.15.107:0/2955264643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c399c0 0x561494c416f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.589+0000 7f9497a60740 1 -- 172.21.15.107:0/2955264643 >> 172.21.15.107:0/2955264643 conn(0x561494c020a0 msgr2=0x561494c3bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:33.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.589+0000 7f9475ffb640 0 client.16035 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:33.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.589+0000 7f9497a60740 1 -- 172.21.15.107:0/2955264643 shutdown_connections 2024-05-09T08:34:33.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.590+0000 7f9497a60740 1 -- 172.21.15.107:0/2955264643 wait complete. 2024-05-09T08:34:33.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.590+0000 7f9497a60740 20 client.16035 trim_cache size 0 max 16384 2024-05-09T08:34:33.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f9497a60740 1 Processor -- start 2024-05-09T08:34:33.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f9497a60740 1 -- start start 2024-05-09T08:34:33.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f9497a60740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c399c0 0x561494c504d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f9497a60740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c3b0b0 0x561494c34f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f9497a60740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c44740 0x561494c5c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f9497a60740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f94901b3600 con 0x561494c3b0b0 2024-05-09T08:34:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f9497a60740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f94901b0ed0 con 0x561494c399c0 2024-05-09T08:34:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f9497a60740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f94901aed20 con 0x561494c44740 2024-05-09T08:34:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f948e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c3b0b0 0x561494c34f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f948e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c3b0b0 0x561494c34f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39232/0 (socket says 172.21.15.107:39232) 2024-05-09T08:34:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f948e59c640 1 -- 172.21.15.107:0/2554542663 learned_addr learned my addr 172.21.15.107:0/2554542663 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f9487fff640 1 --2- 172.21.15.107:0/2554542663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c44740 0x561494c5c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f948e59c640 1 -- 172.21.15.107:0/2554542663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c44740 msgr2=0x561494c5c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f948e59c640 1 --2- 172.21.15.107:0/2554542663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c44740 0x561494c5c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f9487fff640 1 --2- 172.21.15.107:0/2554542663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c399c0 0x561494c504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f948e59c640 1 -- 172.21.15.107:0/2554542663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c399c0 msgr2=0x561494c504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f948e59c640 1 --2- 172.21.15.107:0/2554542663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c399c0 0x561494c504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f948e59c640 1 -- 172.21.15.107:0/2554542663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561494c406a0 con 0x561494c3b0b0 2024-05-09T08:34:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.624+0000 7f9487fff640 1 --2- 172.21.15.107:0/2554542663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c399c0 0x561494c504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:34:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.625+0000 7f948e59c640 1 --2- 172.21.15.107:0/2554542663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c3b0b0 0x561494c34f00 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x561494c3c600 tx=0x7f948000bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50cb55d3e7b3a4c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.625+0000 7f9477fff640 1 -- 172.21.15.107:0/2554542663 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480003ca0 con 0x561494c3b0b0 2024-05-09T08:34:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.625+0000 7f9477fff640 1 -- 172.21.15.107:0/2554542663 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9480022070 con 0x561494c3b0b0 2024-05-09T08:34:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.625+0000 7f9477fff640 1 -- 172.21.15.107:0/2554542663 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480011b60 con 0x561494c3b0b0 2024-05-09T08:34:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.626+0000 7f9497a60740 1 -- 172.21.15.107:0/2554542663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c3b0b0 msgr2=0x561494c34f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.626+0000 7f9497a60740 1 --2- 172.21.15.107:0/2554542663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c3b0b0 0x561494c34f00 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x561494c3c600 tx=0x7f948000bd40 comp rx=0 tx=0).stop 2024-05-09T08:34:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.626+0000 7f9497a60740 1 -- 172.21.15.107:0/2554542663 shutdown_connections 2024-05-09T08:34:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.626+0000 7f9497a60740 1 --2- 172.21.15.107:0/2554542663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c44740 0x561494c5c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.626+0000 7f9497a60740 1 --2- 172.21.15.107:0/2554542663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c3b0b0 0x561494c34f00 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.626+0000 7f9497a60740 1 --2- 172.21.15.107:0/2554542663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c399c0 0x561494c504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.626+0000 7f9497a60740 1 -- 172.21.15.107:0/2554542663 >> 172.21.15.107:0/2554542663 conn(0x561494c020a0 msgr2=0x561494c347f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.626+0000 7f9497a60740 1 -- 172.21.15.107:0/2554542663 shutdown_connections 2024-05-09T08:34:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.626+0000 7f9497a60740 1 -- 172.21.15.107:0/2554542663 wait complete. 2024-05-09T08:34:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.626+0000 7f9497a60740 1 -- start start 2024-05-09T08:34:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.626+0000 7f9497a60740 1 Processor -- start 2024-05-09T08:34:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9497a60740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c399c0 0x561494c48c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9497a60740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c3b0b0 0x561494c491a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9497a60740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494cb39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9497a60740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f94901af390 con 0x561494c44740 2024-05-09T08:34:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9497a60740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f94901b0800 con 0x561494c399c0 2024-05-09T08:34:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9497a60740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f94901afa60 con 0x561494c3b0b0 2024-05-09T08:34:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9497a60740 10 client.0 _ll_register_callbacks cb 0x7f9478011f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9487fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494cb39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9487fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494cb39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39234/0 (socket says 172.21.15.107:39234) 2024-05-09T08:34:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9487fff640 1 -- 172.21.15.107:0/914532845 learned_addr learned my addr 172.21.15.107:0/914532845 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f948e59c640 1 --2- 172.21.15.107:0/914532845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c3b0b0 0x561494c491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9487fff640 1 --2- 172.21.15.107:0/914532845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c399c0 0x561494c48c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9487fff640 1 -- 172.21.15.107:0/914532845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c3b0b0 msgr2=0x561494c491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9487fff640 1 --2- 172.21.15.107:0/914532845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c3b0b0 0x561494c491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9487fff640 1 -- 172.21.15.107:0/914532845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c399c0 msgr2=0x561494c48c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:33.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9487fff640 1 --2- 172.21.15.107:0/914532845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c399c0 0x561494c48c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9487fff640 1 -- 172.21.15.107:0/914532845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561494c36130 con 0x561494c44740 2024-05-09T08:34:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9487fff640 1 --2- 172.21.15.107:0/914532845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494cb39f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f9478009fc0 tx=0x7f94780045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e2c99b54bb307 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.627+0000 7f9475ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.628+0000 7f9475ffb640 1 -- 172.21.15.107:0/914532845 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947800fc70 con 0x561494c44740 2024-05-09T08:34:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.628+0000 7f9475ffb640 1 -- 172.21.15.107:0/914532845 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f9478004c50 con 0x561494c44740 2024-05-09T08:34:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.628+0000 7f9497a60740 1 -- 172.21.15.107:0/914532845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9480022070 con 0x561494c44740 2024-05-09T08:34:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.628+0000 7f9475ffb640 1 -- 172.21.15.107:0/914532845 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9478045050 con 0x561494c44740 2024-05-09T08:34:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.628+0000 7f9497a60740 10 client.16041 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.628+0000 7f9497a60740 1 -- 172.21.15.107:0/914532845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561494cbb110 con 0x561494c44740 2024-05-09T08:34:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.628+0000 7f9475ffb640 1 -- 172.21.15.107:0/914532845 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9480022070 con 0x561494c44740 2024-05-09T08:34:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.628+0000 7f9475ffb640 1 -- 172.21.15.107:0/914532845 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f947802c040 con 0x561494c44740 2024-05-09T08:34:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.628+0000 7f9497a60740 10 client.16041 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.628+0000 7f9497a60740 10 client.16041 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:33.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.628+0000 7f9497a60740 20 client.16041 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.628+0000 7f9497a60740 1 --2- 172.21.15.107:0/914532845 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561494cbb7d0 0x561494cbdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.629+0000 7f9497a60740 4 client.16041 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:34:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.629+0000 7f9497a60740 1 -- 172.21.15.107:0/914532845 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561494ba6410 con 0x561494cbb7d0 2024-05-09T08:34:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.629+0000 7f948e59c640 1 --2- 172.21.15.107:0/914532845 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561494cbb7d0 0x561494cbdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.629+0000 7f9475ffb640 10 client.16041 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.629+0000 7f948e59c640 1 --2- 172.21.15.107:0/914532845 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561494cbb7d0 0x561494cbdbf0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=330dfa9469452418 server_cookie=72b46eca7cdb63f0 in_seq=0 out_seq=0 2024-05-09T08:34:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.631+0000 7f9475ffb640 1 -- 172.21.15.107:0/914532845 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561494ba6410 con 0x561494cbb7d0 2024-05-09T08:34:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.631+0000 7f9475ffb640 10 client.16041 handle_command_reply: tid=0 2024-05-09T08:34:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.631+0000 7f9497a60740 1 client.16041 shutdown 2024-05-09T08:34:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.631+0000 7f9497a60740 1 -- 172.21.15.107:0/914532845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 msgr2=0x561494cb39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.631+0000 7f9497a60740 1 --2- 172.21.15.107:0/914532845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494cb39f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f9478009fc0 tx=0x7f94780045e0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.631+0000 7f9497a60740 1 -- 172.21.15.107:0/914532845 shutdown_connections 2024-05-09T08:34:33.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.631+0000 7f9497a60740 1 --2- 172.21.15.107:0/914532845 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561494cbb7d0 0x561494cbdbf0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.631+0000 7f9497a60740 1 --2- 172.21.15.107:0/914532845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561494c44740 0x561494cb39f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.631+0000 7f9497a60740 1 --2- 172.21.15.107:0/914532845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561494c3b0b0 0x561494c491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.631+0000 7f9497a60740 1 --2- 172.21.15.107:0/914532845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561494c399c0 0x561494c48c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.631+0000 7f9497a60740 1 -- 172.21.15.107:0/914532845 >> 172.21.15.107:0/914532845 conn(0x561494c020a0 msgr2=0x561494c41f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:33.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.631+0000 7f9475ffb640 0 client.16041 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:33.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.631+0000 7f9497a60740 1 -- 172.21.15.107:0/914532845 shutdown_connections 2024-05-09T08:34:33.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.631+0000 7f9497a60740 1 -- 172.21.15.107:0/914532845 wait complete. 2024-05-09T08:34:33.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.631+0000 7f9497a60740 20 client.16041 trim_cache size 0 max 16384 2024-05-09T08:34:33.636 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:34:33.636 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:34:33.636 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "1a795acf-ffd8-4b4e-9cc7-252272f4190e", 2024-05-09T08:34:33.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:34:33.636 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:34:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.632+0000 7f9494d5c640 1 -- 172.21.15.107:0/154781203 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f947c0a2eb0 msgr2=0x7f947c0a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.632+0000 7f9494d5c640 1 --2- 172.21.15.107:0/154781203 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f947c0a2eb0 0x7f947c0a5370 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f94880ee260 tx=0x7f94880b2460 comp rx=0 tx=0).stop 2024-05-09T08:34:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.632+0000 7f9494d5c640 1 -- 172.21.15.107:0/154781203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f949010e1e0 msgr2=0x7f949019eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.632+0000 7f9494d5c640 1 --2- 172.21.15.107:0/154781203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f949010e1e0 0x7f949019eaa0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9488032ff0 tx=0x7f9488009360 comp rx=0 tx=0).stop 2024-05-09T08:34:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.632+0000 7f9494d5c640 1 -- 172.21.15.107:0/154781203 shutdown_connections 2024-05-09T08:34:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.632+0000 7f9494d5c640 1 --2- 172.21.15.107:0/154781203 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f947c0a2eb0 0x7f947c0a5370 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.632+0000 7f9494d5c640 1 --2- 172.21.15.107:0/154781203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f949010e1e0 0x7f949019eaa0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.632+0000 7f9494d5c640 1 --2- 172.21.15.107:0/154781203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9490100f50 0x7f949019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.632+0000 7f9494d5c640 1 --2- 172.21.15.107:0/154781203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9490100630 0x7f949019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.632+0000 7f9494d5c640 1 -- 172.21.15.107:0/154781203 >> 172.21.15.107:0/154781203 conn(0x7f9490077830 msgr2=0x7f949010f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:33.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.632+0000 7f9494d5c640 1 -- 172.21.15.107:0/154781203 shutdown_connections 2024-05-09T08:34:33.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.632+0000 7f9494d5c640 1 -- 172.21.15.107:0/154781203 wait complete. 2024-05-09T08:34:33.643 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:34:33.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:34:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.749+0000 7fa1703a2640 1 -- 172.21.15.107:0/1348997835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa168106710 msgr2=0x7fa168106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.749+0000 7fa1703a2640 1 --2- 172.21.15.107:0/1348997835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa168106710 0x7fa168106af0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fa164009cd0 tx=0x7fa16402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.750+0000 7fa1703a2640 1 -- 172.21.15.107:0/1348997835 shutdown_connections 2024-05-09T08:34:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.750+0000 7fa1703a2640 1 --2- 172.21.15.107:0/1348997835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1681010c0 0x7fa16810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.750+0000 7fa1703a2640 1 --2- 172.21.15.107:0/1348997835 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa168100700 0x7fa168100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.750+0000 7fa1703a2640 1 --2- 172.21.15.107:0/1348997835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa168106710 0x7fa168106af0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.750+0000 7fa1703a2640 1 -- 172.21.15.107:0/1348997835 >> 172.21.15.107:0/1348997835 conn(0x7fa168077830 msgr2=0x7fa168077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.750+0000 7fa1703a2640 1 -- 172.21.15.107:0/1348997835 shutdown_connections 2024-05-09T08:34:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.750+0000 7fa1703a2640 1 -- 172.21.15.107:0/1348997835 wait complete. 2024-05-09T08:34:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.750+0000 7fa1703a2640 1 Processor -- start 2024-05-09T08:34:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.750+0000 7fa1703a2640 1 -- start start 2024-05-09T08:34:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa1703a2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa168100700 0x7fa168196ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa1703a2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1681010c0 0x7fa1681971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa1703a2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa168106710 0x7fa168197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa1703a2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa1681109d0 con 0x7fa1681010c0 2024-05-09T08:34:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa1703a2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa168110850 con 0x7fa168106710 2024-05-09T08:34:33.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa1703a2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa168110b50 con 0x7fa168100700 2024-05-09T08:34:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa16d93d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1681010c0 0x7fa1681971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa16d93d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1681010c0 0x7fa1681971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39264/0 (socket says 172.21.15.107:39264) 2024-05-09T08:34:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa16d93d640 1 -- 172.21.15.107:0/2575958287 learned_addr learned my addr 172.21.15.107:0/2575958287 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa16e13e640 1 --2- 172.21.15.107:0/2575958287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa168100700 0x7fa168196ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa16d93d640 1 -- 172.21.15.107:0/2575958287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa168100700 msgr2=0x7fa168196ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa16d93d640 1 --2- 172.21.15.107:0/2575958287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa168100700 0x7fa168196ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa16d93d640 1 -- 172.21.15.107:0/2575958287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa168106710 msgr2=0x7fa168197720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa16d93d640 1 --2- 172.21.15.107:0/2575958287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa168106710 0x7fa168197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa16d93d640 1 -- 172.21.15.107:0/2575958287 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1681a6510 con 0x7fa1681010c0 2024-05-09T08:34:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa16d93d640 1 --2- 172.21.15.107:0/2575958287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1681010c0 0x7fa1681971e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fa15800e9f0 tx=0x7fa15800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa15f7fe640 1 -- 172.21.15.107:0/2575958287 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15800cd80 con 0x7fa1681010c0 2024-05-09T08:34:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.751+0000 7fa1703a2640 1 -- 172.21.15.107:0/2575958287 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1681a6800 con 0x7fa1681010c0 2024-05-09T08:34:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.752+0000 7fa15f7fe640 1 -- 172.21.15.107:0/2575958287 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1580054d0 con 0x7fa1681010c0 2024-05-09T08:34:33.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.752+0000 7fa15f7fe640 1 -- 172.21.15.107:0/2575958287 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa158010490 con 0x7fa1681010c0 2024-05-09T08:34:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.752+0000 7fa1703a2640 1 -- 172.21.15.107:0/2575958287 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1681a6d50 con 0x7fa1681010c0 2024-05-09T08:34:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.754+0000 7fa15f7fe640 1 -- 172.21.15.107:0/2575958287 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 138118+0+0 (secure 0 0 0) 0x7fa158005020 con 0x7fa1681010c0 2024-05-09T08:34:33.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.755+0000 7fa173194740 1 Processor -- start 2024-05-09T08:34:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.755+0000 7fa173194740 1 -- start start 2024-05-09T08:34:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.755+0000 7fa173194740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94ef280 0x5628b94ef660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.755+0000 7fa173194740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94efba0 0x5628b94f9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.755+0000 7fa173194740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 0x5628b94fc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.755+0000 7fa173194740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5628b9454d00 con 0x5628b94efba0 2024-05-09T08:34:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.755+0000 7fa173194740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5628b93fc4c0 con 0x5628b94ef280 2024-05-09T08:34:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.755+0000 7fa173194740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5628b9450cc0 con 0x5628b94fa270 2024-05-09T08:34:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.756+0000 7fa15f7fe640 1 --2- 172.21.15.107:0/2575958287 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa1540a2eb0 0x7fa1540a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.756+0000 7fa16e13e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94efba0 0x5628b94f9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.756+0000 7fa16e13e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94efba0 0x5628b94f9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39270/0 (socket says 172.21.15.107:39270) 2024-05-09T08:34:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.756+0000 7fa16e13e640 1 -- 172.21.15.107:0/912457736 learned_addr learned my addr 172.21.15.107:0/912457736 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.756+0000 7fa16e93f640 1 --2- 172.21.15.107:0/2575958287 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa1540a2eb0 0x7fa1540a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.756+0000 7fa16e93f640 1 --2- 172.21.15.107:0/912457736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94ef280 0x5628b94ef660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.756+0000 7fa16d93d640 1 --2- 172.21.15.107:0/912457736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 0x5628b94fc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.756+0000 7fa16e13e640 1 -- 172.21.15.107:0/912457736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 msgr2=0x5628b94fc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.756+0000 7fa15f7fe640 1 -- 172.21.15.107:0/2575958287 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fa1580cfea0 con 0x7fa1681010c0 2024-05-09T08:34:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.756+0000 7fa16e13e640 1 --2- 172.21.15.107:0/912457736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 0x5628b94fc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.756+0000 7fa16e13e640 1 -- 172.21.15.107:0/912457736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94ef280 msgr2=0x5628b94ef660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.756+0000 7fa16e13e640 1 --2- 172.21.15.107:0/912457736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94ef280 0x5628b94ef660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.756+0000 7fa16e13e640 1 -- 172.21.15.107:0/912457736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628b92327d0 con 0x5628b94efba0 2024-05-09T08:34:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.756+0000 7fa16d93d640 1 --2- 172.21.15.107:0/912457736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 0x5628b94fc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:34:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.756+0000 7fa16e93f640 1 --2- 172.21.15.107:0/2575958287 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa1540a2eb0 0x7fa1540a5370 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa160005fd0 tx=0x7fa160005c70 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.756+0000 7fa16e13e640 1 --2- 172.21.15.107:0/912457736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94efba0 0x5628b94f9d30 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa164009cd0 tx=0x7fa164005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60facab9de3d31d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.757+0000 7fa15cbfe640 1 -- 172.21.15.107:0/912457736 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164049070 con 0x5628b94efba0 2024-05-09T08:34:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.757+0000 7fa15cbfe640 1 -- 172.21.15.107:0/912457736 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa164036040 con 0x5628b94efba0 2024-05-09T08:34:33.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.757+0000 7fa15cbfe640 1 -- 172.21.15.107:0/912457736 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164004260 con 0x5628b94efba0 2024-05-09T08:34:33.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 -- 172.21.15.107:0/912457736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94efba0 msgr2=0x5628b94f9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 --2- 172.21.15.107:0/912457736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94efba0 0x5628b94f9d30 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa164009cd0 tx=0x7fa164005ef0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 -- 172.21.15.107:0/912457736 shutdown_connections 2024-05-09T08:34:33.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 --2- 172.21.15.107:0/912457736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 0x5628b94fc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 --2- 172.21.15.107:0/912457736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94efba0 0x5628b94f9d30 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 --2- 172.21.15.107:0/912457736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94ef280 0x5628b94ef660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 -- 172.21.15.107:0/912457736 >> 172.21.15.107:0/912457736 conn(0x5628b94eb5a0 msgr2=0x5628b94eb980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:33.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 -- 172.21.15.107:0/912457736 shutdown_connections 2024-05-09T08:34:33.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 -- 172.21.15.107:0/912457736 wait complete. 2024-05-09T08:34:33.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 -- start start 2024-05-09T08:34:33.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 Processor -- start 2024-05-09T08:34:33.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94ef280 0x5628b94f7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94fa270 0x5628b94f79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94f7f60 0x5628b9506470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5628b94fd000 con 0x5628b94fa270 2024-05-09T08:34:33.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5628b940fb90 con 0x5628b94ef280 2024-05-09T08:34:33.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.758+0000 7fa173194740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5628b94fd180 con 0x5628b94f7f60 2024-05-09T08:34:33.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa173194740 10 client.0 _ll_register_callbacks cb 0x5628b944bda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:33.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa16d93d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94fa270 0x5628b94f79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa16e13e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94ef280 0x5628b94f7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa16e93f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94f7f60 0x5628b9506470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa16d93d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94fa270 0x5628b94f79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39276/0 (socket says 172.21.15.107:39276) 2024-05-09T08:34:33.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa16d93d640 1 -- 172.21.15.107:0/2010925275 learned_addr learned my addr 172.21.15.107:0/2010925275 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:33.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa16e13e640 1 -- 172.21.15.107:0/2010925275 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94f7f60 msgr2=0x5628b9506470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:33.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa16e13e640 1 --2- 172.21.15.107:0/2010925275 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94f7f60 0x5628b9506470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa16e13e640 1 -- 172.21.15.107:0/2010925275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94fa270 msgr2=0x5628b94f79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:33.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa16e13e640 1 --2- 172.21.15.107:0/2010925275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94fa270 0x5628b94f79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa16e13e640 1 -- 172.21.15.107:0/2010925275 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628b91683f0 con 0x5628b94ef280 2024-05-09T08:34:33.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa16d93d640 1 --2- 172.21.15.107:0/2010925275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94fa270 0x5628b94f79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:33.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa16e13e640 1 --2- 172.21.15.107:0/2010925275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94ef280 0x5628b94f7490 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x5628b94ec4a0 tx=0x7fa164031a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4df7cafcd6817ab5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:33.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa151ffb640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:34:33.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa151ffb640 1 -- 172.21.15.107:0/2010925275 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164049040 con 0x5628b94ef280 2024-05-09T08:34:33.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa173194740 1 -- 172.21.15.107:0/2010925275 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa164036040 con 0x5628b94ef280 2024-05-09T08:34:33.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.759+0000 7fa151ffb640 1 -- 172.21.15.107:0/2010925275 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fa164041040 con 0x5628b94ef280 2024-05-09T08:34:33.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.760+0000 7fa151ffb640 1 -- 172.21.15.107:0/2010925275 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa164031c70 con 0x5628b94ef280 2024-05-09T08:34:33.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.760+0000 7fa173194740 10 client.25391 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:33.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.760+0000 7fa173194740 1 -- 172.21.15.107:0/2010925275 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5628b91280f0 con 0x5628b94ef280 2024-05-09T08:34:33.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.760+0000 7fa151ffb640 1 -- 172.21.15.107:0/2010925275 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa164036040 con 0x5628b94ef280 2024-05-09T08:34:33.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.760+0000 7fa151ffb640 1 -- 172.21.15.107:0/2010925275 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fa16404c070 con 0x5628b94ef280 2024-05-09T08:34:33.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.760+0000 7fa173194740 10 client.25391 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:33.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.760+0000 7fa173194740 10 client.25391 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:33.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.760+0000 7fa173194740 20 client.25391 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:33.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.760+0000 7fa173194740 1 --2- 172.21.15.107:0/2010925275 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5628b950aac0 0x5628b950cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.760+0000 7fa173194740 4 client.25391 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:34:33.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.760+0000 7fa173194740 1 -- 172.21.15.107:0/2010925275 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628b9462430 con 0x5628b950aac0 2024-05-09T08:34:33.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.760+0000 7fa16d93d640 1 --2- 172.21.15.107:0/2010925275 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5628b950aac0 0x5628b950cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.761+0000 7fa151ffb640 10 client.25391 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:33.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.761+0000 7fa16d93d640 1 --2- 172.21.15.107:0/2010925275 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5628b950aac0 0x5628b950cf30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ce7e06442dc12eb server_cookie=a09a93b5864744e5 in_seq=0 out_seq=0 2024-05-09T08:34:33.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.761+0000 7fa151ffb640 1 -- 172.21.15.107:0/2010925275 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5628b9462430 con 0x5628b950aac0 2024-05-09T08:34:33.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.761+0000 7fa151ffb640 10 client.25391 handle_command_reply: tid=0 2024-05-09T08:34:33.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.762+0000 7fa173194740 1 client.25391 shutdown 2024-05-09T08:34:33.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.762+0000 7fa173194740 1 -- 172.21.15.107:0/2010925275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94ef280 msgr2=0x5628b94f7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.762+0000 7fa173194740 1 --2- 172.21.15.107:0/2010925275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94ef280 0x5628b94f7490 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x5628b94ec4a0 tx=0x7fa164031a80 comp rx=0 tx=0).stop 2024-05-09T08:34:33.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.762+0000 7fa173194740 1 -- 172.21.15.107:0/2010925275 shutdown_connections 2024-05-09T08:34:33.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.762+0000 7fa173194740 1 --2- 172.21.15.107:0/2010925275 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5628b950aac0 0x5628b950cf30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.762+0000 7fa173194740 1 --2- 172.21.15.107:0/2010925275 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94f7f60 0x5628b9506470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.762+0000 7fa173194740 1 --2- 172.21.15.107:0/2010925275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94fa270 0x5628b94f79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.762+0000 7fa173194740 1 --2- 172.21.15.107:0/2010925275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94ef280 0x5628b94f7490 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.762+0000 7fa173194740 1 -- 172.21.15.107:0/2010925275 >> 172.21.15.107:0/2010925275 conn(0x5628b94eb5a0 msgr2=0x5628b94f13a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:33.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.762+0000 7fa151ffb640 0 client.25391 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:33.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.762+0000 7fa173194740 1 -- 172.21.15.107:0/2010925275 shutdown_connections 2024-05-09T08:34:33.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.762+0000 7fa173194740 1 -- 172.21.15.107:0/2010925275 wait complete. 2024-05-09T08:34:33.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.762+0000 7fa173194740 20 client.25391 trim_cache size 0 max 16384 2024-05-09T08:34:33.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.779+0000 7fa173194740 1 Processor -- start 2024-05-09T08:34:33.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.779+0000 7fa173194740 1 -- start start 2024-05-09T08:34:33.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.779+0000 7fa173194740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94ef280 0x5628b94eb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.779+0000 7fa173194740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94f7f60 0x5628b94f1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.779+0000 7fa173194740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 0x5628b9512810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.779+0000 7fa173194740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa1681b34f0 con 0x5628b94f7f60 2024-05-09T08:34:33.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.779+0000 7fa173194740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa1681b0dc0 con 0x5628b94ef280 2024-05-09T08:34:33.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.779+0000 7fa173194740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa1681aec10 con 0x5628b94fa270 2024-05-09T08:34:33.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.780+0000 7fa16d93d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94f7f60 0x5628b94f1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.780+0000 7fa16d93d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94f7f60 0x5628b94f1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39288/0 (socket says 172.21.15.107:39288) 2024-05-09T08:34:33.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.780+0000 7fa16d93d640 1 -- 172.21.15.107:0/2874454555 learned_addr learned my addr 172.21.15.107:0/2874454555 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:33.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.780+0000 7fa16e13e640 1 --2- 172.21.15.107:0/2874454555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94ef280 0x5628b94eb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.780+0000 7fa16e93f640 1 --2- 172.21.15.107:0/2874454555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 0x5628b9512810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.780+0000 7fa16d93d640 1 -- 172.21.15.107:0/2874454555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 msgr2=0x5628b9512810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:33.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.780+0000 7fa16d93d640 1 --2- 172.21.15.107:0/2874454555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 0x5628b9512810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.780+0000 7fa16d93d640 1 -- 172.21.15.107:0/2874454555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94ef280 msgr2=0x5628b94eb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:33.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.780+0000 7fa16d93d640 1 --2- 172.21.15.107:0/2874454555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94ef280 0x5628b94eb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.780+0000 7fa16d93d640 1 -- 172.21.15.107:0/2874454555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628b94f7260 con 0x5628b94f7f60 2024-05-09T08:34:33.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.780+0000 7fa16d93d640 1 --2- 172.21.15.107:0/2874454555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94f7f60 0x5628b94f1520 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa15809bdb0 tx=0x7fa158098140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe2908be77737e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:33.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.780+0000 7fa153fff640 1 -- 172.21.15.107:0/2874454555 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15809c520 con 0x5628b94f7f60 2024-05-09T08:34:33.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.781+0000 7fa153fff640 1 -- 172.21.15.107:0/2874454555 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa15809be20 con 0x5628b94f7f60 2024-05-09T08:34:33.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.781+0000 7fa153fff640 1 -- 172.21.15.107:0/2874454555 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1580bacc0 con 0x5628b94f7f60 2024-05-09T08:34:33.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.781+0000 7fa173194740 1 -- 172.21.15.107:0/2874454555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94f7f60 msgr2=0x5628b94f1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.781+0000 7fa173194740 1 --2- 172.21.15.107:0/2874454555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94f7f60 0x5628b94f1520 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa15809bdb0 tx=0x7fa158098140 comp rx=0 tx=0).stop 2024-05-09T08:34:33.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.781+0000 7fa173194740 1 -- 172.21.15.107:0/2874454555 shutdown_connections 2024-05-09T08:34:33.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.781+0000 7fa173194740 1 --2- 172.21.15.107:0/2874454555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 0x5628b9512810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.781+0000 7fa173194740 1 --2- 172.21.15.107:0/2874454555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94f7f60 0x5628b94f1520 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.781+0000 7fa173194740 1 --2- 172.21.15.107:0/2874454555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94ef280 0x5628b94eb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.781+0000 7fa173194740 1 -- 172.21.15.107:0/2874454555 >> 172.21.15.107:0/2874454555 conn(0x5628b94eb5a0 msgr2=0x5628b94f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:33.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.781+0000 7fa173194740 1 -- 172.21.15.107:0/2874454555 shutdown_connections 2024-05-09T08:34:33.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.781+0000 7fa173194740 1 -- 172.21.15.107:0/2874454555 wait complete. 2024-05-09T08:34:33.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa173194740 1 -- start start 2024-05-09T08:34:33.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa173194740 1 Processor -- start 2024-05-09T08:34:33.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa173194740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94ef280 0x5628b95028f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa173194740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94f7f60 0x5628b94f4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa173194740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 0x5628b94f4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa173194740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa1681af280 con 0x5628b94ef280 2024-05-09T08:34:33.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa173194740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa1681b06f0 con 0x5628b94f7f60 2024-05-09T08:34:33.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa173194740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa1681af950 con 0x5628b94fa270 2024-05-09T08:34:33.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa16e13e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94ef280 0x5628b95028f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa173194740 10 client.0 _ll_register_callbacks cb 0x7fa164047070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:33.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa16e13e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94ef280 0x5628b95028f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39298/0 (socket says 172.21.15.107:39298) 2024-05-09T08:34:33.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa16e13e640 1 -- 172.21.15.107:0/211509952 learned_addr learned my addr 172.21.15.107:0/211509952 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:33.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa16d93d640 1 --2- 172.21.15.107:0/211509952 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94f7f60 0x5628b94f4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa16e93f640 1 --2- 172.21.15.107:0/211509952 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 0x5628b94f4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa16e13e640 1 -- 172.21.15.107:0/211509952 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 msgr2=0x5628b94f4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:33.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa16e13e640 1 --2- 172.21.15.107:0/211509952 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 0x5628b94f4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa16e13e640 1 -- 172.21.15.107:0/211509952 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94f7f60 msgr2=0x5628b94f4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:33.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa16e13e640 1 --2- 172.21.15.107:0/211509952 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94f7f60 0x5628b94f4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa16e13e640 1 -- 172.21.15.107:0/211509952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628b94f3700 con 0x5628b94ef280 2024-05-09T08:34:33.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.782+0000 7fa16e93f640 1 --2- 172.21.15.107:0/211509952 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 0x5628b94f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:34:33.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.783+0000 7fa16e13e640 1 --2- 172.21.15.107:0/211509952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94ef280 0x5628b95028f0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa164004d70 tx=0x7fa164004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca788ea137d8fdb6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:33.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.783+0000 7fa151ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:33.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.783+0000 7fa151ffb640 1 -- 172.21.15.107:0/211509952 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164049070 con 0x5628b94ef280 2024-05-09T08:34:33.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.783+0000 7fa151ffb640 1 -- 172.21.15.107:0/211509952 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fa16404db00 con 0x5628b94ef280 2024-05-09T08:34:33.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.783+0000 7fa173194740 1 -- 172.21.15.107:0/211509952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa15809be20 con 0x5628b94ef280 2024-05-09T08:34:33.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.783+0000 7fa151ffb640 1 -- 172.21.15.107:0/211509952 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa16404dd00 con 0x5628b94ef280 2024-05-09T08:34:33.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.783+0000 7fa173194740 10 client.16059 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:33.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.783+0000 7fa173194740 1 -- 172.21.15.107:0/211509952 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5628b9570320 con 0x5628b94ef280 2024-05-09T08:34:33.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.783+0000 7fa151ffb640 1 -- 172.21.15.107:0/211509952 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa15809be20 con 0x5628b94ef280 2024-05-09T08:34:33.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.783+0000 7fa151ffb640 1 -- 172.21.15.107:0/211509952 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fa16404c040 con 0x5628b94ef280 2024-05-09T08:34:33.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.783+0000 7fa173194740 10 client.16059 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:33.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.783+0000 7fa173194740 10 client.16059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:33.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.783+0000 7fa173194740 20 client.16059 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:33.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.783+0000 7fa173194740 1 --2- 172.21.15.107:0/211509952 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5628b9570b50 0x5628b9572f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:33.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.783+0000 7fa173194740 4 client.16059 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:34:33.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.783+0000 7fa173194740 1 -- 172.21.15.107:0/211509952 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5628b945c780 con 0x5628b9570b50 2024-05-09T08:34:33.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.784+0000 7fa16d93d640 1 --2- 172.21.15.107:0/211509952 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5628b9570b50 0x5628b9572f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:33.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.784+0000 7fa151ffb640 10 client.16059 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:33.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.784+0000 7fa16d93d640 1 --2- 172.21.15.107:0/211509952 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5628b9570b50 0x5628b9572f70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed71b12050849a97 server_cookie=a13cbee3b1404ad7 in_seq=0 out_seq=0 2024-05-09T08:34:33.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.784+0000 7fa151ffb640 1 -- 172.21.15.107:0/211509952 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5628b945c780 con 0x5628b9570b50 2024-05-09T08:34:33.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.784+0000 7fa151ffb640 10 client.16059 handle_command_reply: tid=0 2024-05-09T08:34:33.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.784+0000 7fa173194740 1 client.16059 shutdown 2024-05-09T08:34:33.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.784+0000 7fa173194740 1 -- 172.21.15.107:0/211509952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94ef280 msgr2=0x5628b95028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.784+0000 7fa173194740 1 --2- 172.21.15.107:0/211509952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94ef280 0x5628b95028f0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa164004d70 tx=0x7fa164004da0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.784+0000 7fa173194740 1 -- 172.21.15.107:0/211509952 shutdown_connections 2024-05-09T08:34:33.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa173194740 1 --2- 172.21.15.107:0/211509952 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5628b9570b50 0x5628b9572f70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa173194740 1 --2- 172.21.15.107:0/211509952 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b94fa270 0x5628b94f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa173194740 1 --2- 172.21.15.107:0/211509952 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628b94f7f60 0x5628b94f4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa151ffb640 0 client.16059 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:33.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa173194740 1 --2- 172.21.15.107:0/211509952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5628b94ef280 0x5628b95028f0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa173194740 1 -- 172.21.15.107:0/211509952 >> 172.21.15.107:0/211509952 conn(0x5628b94eb5a0 msgr2=0x5628b94f7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:33.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa173194740 1 -- 172.21.15.107:0/211509952 shutdown_connections 2024-05-09T08:34:33.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa173194740 1 -- 172.21.15.107:0/211509952 wait complete. 2024-05-09T08:34:33.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa173194740 20 client.16059 trim_cache size 0 max 16384 2024-05-09T08:34:33.791 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:34:33.791 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:34:33.791 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:34:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "1a795acf-ffd8-4b4e-9cc7-252272f4190e": { 2024-05-09T08:34:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:34:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "1a795acf-ffd8-4b4e-9cc7-252272f4190e", 2024-05-09T08:34:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:34:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:34:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:34:33.792 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:34:33.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa1703a2640 1 -- 172.21.15.107:0/2575958287 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa1540a2eb0 msgr2=0x7fa1540a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa1703a2640 1 --2- 172.21.15.107:0/2575958287 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa1540a2eb0 0x7fa1540a5370 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa160005fd0 tx=0x7fa160005c70 comp rx=0 tx=0).stop 2024-05-09T08:34:33.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa1703a2640 1 -- 172.21.15.107:0/2575958287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1681010c0 msgr2=0x7fa1681971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:33.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa1703a2640 1 --2- 172.21.15.107:0/2575958287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1681010c0 0x7fa1681971e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fa15800e9f0 tx=0x7fa15800edf0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa1703a2640 1 -- 172.21.15.107:0/2575958287 shutdown_connections 2024-05-09T08:34:33.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa1703a2640 1 --2- 172.21.15.107:0/2575958287 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa1540a2eb0 0x7fa1540a5370 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa1703a2640 1 --2- 172.21.15.107:0/2575958287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa168106710 0x7fa168197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa1703a2640 1 --2- 172.21.15.107:0/2575958287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1681010c0 0x7fa1681971e0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.785+0000 7fa1703a2640 1 --2- 172.21.15.107:0/2575958287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa168100700 0x7fa168196ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:33.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.786+0000 7fa1703a2640 1 -- 172.21.15.107:0/2575958287 >> 172.21.15.107:0/2575958287 conn(0x7fa168077830 msgr2=0x7fa1681030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:33.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.786+0000 7fa1703a2640 1 -- 172.21.15.107:0/2575958287 shutdown_connections 2024-05-09T08:34:33.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:34:33.786+0000 7fa1703a2640 1 -- 172.21.15.107:0/2575958287 wait complete. 2024-05-09T08:34:33.798 INFO:tasks.cephfs.filesystem:scrub status for tag:1a795acf-ffd8-4b4e-9cc7-252272f4190e - {'path': '/', 'tag': '1a795acf-ffd8-4b4e-9cc7-252272f4190e', 'options': 'recursive,force'} 2024-05-09T08:34:34.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:34 smithi107 ceph-mon[30122]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T08:34:34.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:34 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:34:34.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:34 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:34:34.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:34 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:34:34.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:34 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:34:34.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:34 smithi157 ceph-mon[33614]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T08:34:34.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:34 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:34:34.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:34 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:34:34.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:34 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:34:34.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:34 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:34:34.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:34 smithi110 ceph-mon[33622]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T08:34:34.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:34 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:34:34.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:34 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:34:34.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:34 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:34:34.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:34 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:34:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.020+0000 7f1c91cba640 1 -- 172.21.15.107:0/837287425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c1033e0 msgr2=0x7f1c8c10b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.020+0000 7f1c91cba640 1 --2- 172.21.15.107:0/837287425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c1033e0 0x7f1c8c10b8c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1c80009900 tx=0x7f1c8002a510 comp rx=0 tx=0).stop 2024-05-09T08:34:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.020+0000 7f1c91cba640 1 -- 172.21.15.107:0/837287425 shutdown_connections 2024-05-09T08:34:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.020+0000 7f1c91cba640 1 --2- 172.21.15.107:0/837287425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c8c10be00 0x7f1c8c10e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.020+0000 7f1c91cba640 1 --2- 172.21.15.107:0/837287425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c1033e0 0x7f1c8c10b8c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.020+0000 7f1c91cba640 1 --2- 172.21.15.107:0/837287425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c8c102ac0 0x7f1c8c102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.020+0000 7f1c91cba640 1 -- 172.21.15.107:0/837287425 >> 172.21.15.107:0/837287425 conn(0x7f1c8c0778b0 msgr2=0x7f1c8c077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.020+0000 7f1c91cba640 1 -- 172.21.15.107:0/837287425 shutdown_connections 2024-05-09T08:34:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.020+0000 7f1c91cba640 1 -- 172.21.15.107:0/837287425 wait complete. 2024-05-09T08:34:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.021+0000 7f1c91cba640 1 Processor -- start 2024-05-09T08:34:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.021+0000 7f1c91cba640 1 -- start start 2024-05-09T08:34:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.021+0000 7f1c91cba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c102ac0 0x7f1c8c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c91cba640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c8c1033e0 0x7f1c8c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c91cba640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c8c10be00 0x7f1c8c19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c91cba640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c111b00 con 0x7f1c8c102ac0 2024-05-09T08:34:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c91cba640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c111980 con 0x7f1c8c1033e0 2024-05-09T08:34:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c8b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c102ac0 0x7f1c8c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c91cba640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c111c80 con 0x7f1c8c10be00 2024-05-09T08:34:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c8b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c102ac0 0x7f1c8c198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39310/0 (socket says 172.21.15.107:39310) 2024-05-09T08:34:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c8b7fe640 1 -- 172.21.15.107:0/483862805 learned_addr learned my addr 172.21.15.107:0/483862805 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/483862805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c8c10be00 0x7f1c8c19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c8affd640 1 --2- 172.21.15.107:0/483862805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c8c1033e0 0x7f1c8c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c8b7fe640 1 -- 172.21.15.107:0/483862805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c8c10be00 msgr2=0x7f1c8c19c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c8b7fe640 1 --2- 172.21.15.107:0/483862805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c8c10be00 0x7f1c8c19c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c8b7fe640 1 -- 172.21.15.107:0/483862805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c8c1033e0 msgr2=0x7f1c8c1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c8b7fe640 1 --2- 172.21.15.107:0/483862805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c8c1033e0 0x7f1c8c1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c8b7fe640 1 -- 172.21.15.107:0/483862805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8c19d000 con 0x7f1c8c102ac0 2024-05-09T08:34:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c8affd640 1 --2- 172.21.15.107:0/483862805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c8c1033e0 0x7f1c8c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:34:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.022+0000 7f1c8b7fe640 1 --2- 172.21.15.107:0/483862805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c102ac0 0x7f1c8c198060 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00d950 tx=0x7f1c7c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.023+0000 7f1c88ff9640 1 -- 172.21.15.107:0/483862805 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c017030 con 0x7f1c8c102ac0 2024-05-09T08:34:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.023+0000 7f1c91cba640 1 -- 172.21.15.107:0/483862805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c8c19d2f0 con 0x7f1c8c102ac0 2024-05-09T08:34:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.023+0000 7f1c88ff9640 1 -- 172.21.15.107:0/483862805 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c7c004020 con 0x7f1c8c102ac0 2024-05-09T08:34:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.023+0000 7f1c91cba640 1 -- 172.21.15.107:0/483862805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c8c1a4b80 con 0x7f1c8c102ac0 2024-05-09T08:34:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.023+0000 7f1c88ff9640 1 -- 172.21.15.107:0/483862805 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c0047a0 con 0x7f1c8c102ac0 2024-05-09T08:34:36.034 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.026+0000 7f1c949c6740 1 Processor -- start 2024-05-09T08:34:36.034 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.026+0000 7f1c949c6740 1 -- start start 2024-05-09T08:34:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.026+0000 7f1c88ff9640 1 -- 172.21.15.107:0/483862805 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 138118+0+0 (secure 0 0 0) 0x7f1c7c0041d0 con 0x7f1c8c102ac0 2024-05-09T08:34:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.026+0000 7f1c949c6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729450 0x5590eb729830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c949c6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb729d70 0x5590eb7340f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c949c6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb734630 0x5590eb736a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c949c6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5590eb68e340 con 0x5590eb734630 2024-05-09T08:34:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c949c6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5590eb636290 con 0x5590eb729d70 2024-05-09T08:34:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c949c6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5590eb68a300 con 0x5590eb729450 2024-05-09T08:34:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c8bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729450 0x5590eb729830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c8bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729450 0x5590eb729830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:59358/0 (socket says 172.21.15.107:59358) 2024-05-09T08:34:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c8bfff640 1 -- 172.21.15.107:0/2986484842 learned_addr learned my addr 172.21.15.107:0/2986484842 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c8affd640 1 --2- 172.21.15.107:0/2986484842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb734630 0x5590eb736a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c8b7fe640 1 --2- 172.21.15.107:0/2986484842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb729d70 0x5590eb7340f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c8bfff640 1 -- 172.21.15.107:0/2986484842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb729d70 msgr2=0x5590eb7340f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/2986484842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb729d70 0x5590eb7340f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c8bfff640 1 -- 172.21.15.107:0/2986484842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb734630 msgr2=0x5590eb736a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/2986484842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb734630 0x5590eb736a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c8bfff640 1 -- 172.21.15.107:0/2986484842 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590eb65a8e0 con 0x5590eb729450 2024-05-09T08:34:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c8affd640 1 --2- 172.21.15.107:0/2986484842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb734630 0x5590eb736a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:34:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.027+0000 7f1c88ff9640 1 --2- 172.21.15.107:0/483862805 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c740a2f50 0x7f1c740a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.028+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/2986484842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729450 0x5590eb729830 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009f90 tx=0x7f1c84010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97bcfccf12016806 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.028+0000 7f1c8b7fe640 1 --2- 172.21.15.107:0/483862805 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c740a2f50 0x7f1c740a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.028+0000 7f1c88ff9640 1 -- 172.21.15.107:0/483862805 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1c7c0cf440 con 0x7f1c8c102ac0 2024-05-09T08:34:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.028+0000 7f1c71ffb640 1 -- 172.21.15.107:0/2986484842 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c84011700 con 0x5590eb729450 2024-05-09T08:34:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.028+0000 7f1c71ffb640 1 -- 172.21.15.107:0/2986484842 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c84011d50 con 0x5590eb729450 2024-05-09T08:34:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.028+0000 7f1c71ffb640 1 -- 172.21.15.107:0/2986484842 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c84019da0 con 0x5590eb729450 2024-05-09T08:34:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.028+0000 7f1c8b7fe640 1 --2- 172.21.15.107:0/483862805 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c740a2f50 0x7f1c740a5410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0cc6f0 tx=0x7f1c7c097040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.029+0000 7f1c949c6740 1 -- 172.21.15.107:0/2986484842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729450 msgr2=0x5590eb729830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.029+0000 7f1c949c6740 1 --2- 172.21.15.107:0/2986484842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729450 0x5590eb729830 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009f90 tx=0x7f1c84010860 comp rx=0 tx=0).stop 2024-05-09T08:34:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.029+0000 7f1c949c6740 1 -- 172.21.15.107:0/2986484842 shutdown_connections 2024-05-09T08:34:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.029+0000 7f1c949c6740 1 --2- 172.21.15.107:0/2986484842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb734630 0x5590eb736a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.029+0000 7f1c949c6740 1 --2- 172.21.15.107:0/2986484842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb729d70 0x5590eb7340f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.029+0000 7f1c949c6740 1 --2- 172.21.15.107:0/2986484842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729450 0x5590eb729830 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.029+0000 7f1c949c6740 1 -- 172.21.15.107:0/2986484842 >> 172.21.15.107:0/2986484842 conn(0x5590eb725960 msgr2=0x5590eb725d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.029+0000 7f1c949c6740 1 -- 172.21.15.107:0/2986484842 shutdown_connections 2024-05-09T08:34:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.029+0000 7f1c949c6740 1 -- 172.21.15.107:0/2986484842 wait complete. 2024-05-09T08:34:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.030+0000 7f1c949c6740 1 -- start start 2024-05-09T08:34:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.030+0000 7f1c949c6740 1 Processor -- start 2024-05-09T08:34:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.030+0000 7f1c949c6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729450 0x5590eb731520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.030+0000 7f1c949c6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb729d70 0x5590eb731a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.030+0000 7f1c949c6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb734630 0x5590eb731fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.030+0000 7f1c949c6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5590eb737260 con 0x5590eb729d70 2024-05-09T08:34:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.030+0000 7f1c949c6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5590eb649790 con 0x5590eb734630 2024-05-09T08:34:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.030+0000 7f1c949c6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5590eb7373e0 con 0x5590eb729450 2024-05-09T08:34:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.030+0000 7f1c949c6740 10 client.0 _ll_register_callbacks cb 0x5590eb684a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.030+0000 7f1c8bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb734630 0x5590eb731fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.030+0000 7f1c8affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb729d70 0x5590eb731a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.030+0000 7f1c8bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb734630 0x5590eb731fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:46718/0 (socket says 172.21.15.107:46718) 2024-05-09T08:34:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.030+0000 7f1c8bfff640 1 -- 172.21.15.107:0/1629472489 learned_addr learned my addr 172.21.15.107:0/1629472489 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.030+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/1629472489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729450 0x5590eb731520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.031+0000 7f1c8bfff640 1 -- 172.21.15.107:0/1629472489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729450 msgr2=0x5590eb731520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.031+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/1629472489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729450 0x5590eb731520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.031+0000 7f1c8bfff640 1 -- 172.21.15.107:0/1629472489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb729d70 msgr2=0x5590eb731a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.031+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/1629472489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb729d70 0x5590eb731a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.031+0000 7f1c8bfff640 1 -- 172.21.15.107:0/1629472489 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590eb3a2210 con 0x5590eb734630 2024-05-09T08:34:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.031+0000 7f1c8affd640 1 --2- 172.21.15.107:0/1629472489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb729d70 0x5590eb731a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.031+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/1629472489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb734630 0x5590eb731fa0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009d50 tx=0x7f1c84013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6669509bfeb2d263 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.031+0000 7f1c6affd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:34:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.031+0000 7f1c6affd640 1 -- 172.21.15.107:0/1629472489 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c84003970 con 0x5590eb734630 2024-05-09T08:34:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.031+0000 7f1c6affd640 1 -- 172.21.15.107:0/1629472489 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1c84003b10 con 0x5590eb734630 2024-05-09T08:34:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.031+0000 7f1c949c6740 1 -- 172.21.15.107:0/1629472489 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c84011d50 con 0x5590eb734630 2024-05-09T08:34:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.032+0000 7f1c6affd640 1 -- 172.21.15.107:0/1629472489 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1c84011500 con 0x5590eb734630 2024-05-09T08:34:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.032+0000 7f1c949c6740 10 client.25406 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.032+0000 7f1c949c6740 1 -- 172.21.15.107:0/1629472489 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5590eb46e620 con 0x5590eb734630 2024-05-09T08:34:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.032+0000 7f1c6affd640 1 -- 172.21.15.107:0/1629472489 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c84011d50 con 0x5590eb734630 2024-05-09T08:34:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.032+0000 7f1c6affd640 1 -- 172.21.15.107:0/1629472489 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1c8401a970 con 0x5590eb734630 2024-05-09T08:34:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.032+0000 7f1c949c6740 10 client.25406 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.032+0000 7f1c949c6740 10 client.25406 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.032+0000 7f1c949c6740 20 client.25406 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.032+0000 7f1c949c6740 1 --2- 172.21.15.107:0/1629472489 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5590eb744250 0x5590eb746640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.032+0000 7f1c949c6740 4 client.25406 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:34:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.032+0000 7f1c949c6740 1 -- 172.21.15.107:0/1629472489 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590eb69ba70 con 0x5590eb744250 2024-05-09T08:34:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.033+0000 7f1c8affd640 1 --2- 172.21.15.107:0/1629472489 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5590eb744250 0x5590eb746640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.033+0000 7f1c6affd640 10 client.25406 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.033+0000 7f1c8affd640 1 --2- 172.21.15.107:0/1629472489 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5590eb744250 0x5590eb746640 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5518f61a5f28110c server_cookie=fe12c9f60ecb5b82 in_seq=0 out_seq=0 2024-05-09T08:34:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.034+0000 7f1c6affd640 1 -- 172.21.15.107:0/1629472489 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5590eb69ba70 con 0x5590eb744250 2024-05-09T08:34:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.034+0000 7f1c6affd640 10 client.25406 handle_command_reply: tid=0 2024-05-09T08:34:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.034+0000 7f1c949c6740 1 client.25406 shutdown 2024-05-09T08:34:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.034+0000 7f1c949c6740 1 -- 172.21.15.107:0/1629472489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb734630 msgr2=0x5590eb731fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.034+0000 7f1c949c6740 1 --2- 172.21.15.107:0/1629472489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb734630 0x5590eb731fa0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009d50 tx=0x7f1c84013040 comp rx=0 tx=0).stop 2024-05-09T08:34:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.034+0000 7f1c949c6740 1 -- 172.21.15.107:0/1629472489 shutdown_connections 2024-05-09T08:34:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.034+0000 7f1c949c6740 1 --2- 172.21.15.107:0/1629472489 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5590eb744250 0x5590eb746640 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.034+0000 7f1c6affd640 0 client.25406 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.034+0000 7f1c949c6740 1 --2- 172.21.15.107:0/1629472489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb734630 0x5590eb731fa0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.034+0000 7f1c949c6740 1 --2- 172.21.15.107:0/1629472489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb729d70 0x5590eb731a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.034+0000 7f1c949c6740 1 --2- 172.21.15.107:0/1629472489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729450 0x5590eb731520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.034+0000 7f1c949c6740 1 -- 172.21.15.107:0/1629472489 >> 172.21.15.107:0/1629472489 conn(0x5590eb725960 msgr2=0x5590eb72b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.034+0000 7f1c949c6740 1 -- 172.21.15.107:0/1629472489 shutdown_connections 2024-05-09T08:34:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.034+0000 7f1c949c6740 1 -- 172.21.15.107:0/1629472489 wait complete. 2024-05-09T08:34:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.034+0000 7f1c949c6740 20 client.25406 trim_cache size 0 max 16384 2024-05-09T08:34:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.062+0000 7f1c949c6740 1 Processor -- start 2024-05-09T08:34:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c949c6740 1 -- start start 2024-05-09T08:34:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c949c6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb729450 0x5590eb733c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c949c6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb729d70 0x5590eb72be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c949c6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb734630 0x5590eb730d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c949c6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c1b1540 con 0x5590eb729450 2024-05-09T08:34:36.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c949c6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c1aee10 con 0x5590eb729d70 2024-05-09T08:34:36.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c949c6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c1acc60 con 0x5590eb734630 2024-05-09T08:34:36.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c8bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb729450 0x5590eb733c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c8bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb729450 0x5590eb733c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39328/0 (socket says 172.21.15.107:39328) 2024-05-09T08:34:36.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c8bfff640 1 -- 172.21.15.107:0/1175912860 learned_addr learned my addr 172.21.15.107:0/1175912860 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:36.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c8bfff640 1 -- 172.21.15.107:0/1175912860 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb734630 msgr2=0x5590eb730d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:36.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/1175912860 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb734630 0x5590eb730d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c8bfff640 1 -- 172.21.15.107:0/1175912860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb729d70 msgr2=0x5590eb72be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:36.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c8affd640 1 --2- 172.21.15.107:0/1175912860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb729d70 0x5590eb72be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/1175912860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb729d70 0x5590eb72be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c8bfff640 1 -- 172.21.15.107:0/1175912860 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590eb723f00 con 0x5590eb729450 2024-05-09T08:34:36.065 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.063+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/1175912860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb729450 0x5590eb733c50 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1c8401d040 tx=0x7f1c84019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1117a71c0b5bc0e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:36.065 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.064+0000 7f1c717fa640 1 -- 172.21.15.107:0/1175912860 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c84010d60 con 0x5590eb729450 2024-05-09T08:34:36.065 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.064+0000 7f1c717fa640 1 -- 172.21.15.107:0/1175912860 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c84020040 con 0x5590eb729450 2024-05-09T08:34:36.065 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.064+0000 7f1c717fa640 1 -- 172.21.15.107:0/1175912860 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c84011920 con 0x5590eb729450 2024-05-09T08:34:36.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.064+0000 7f1c949c6740 1 -- 172.21.15.107:0/1175912860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb729450 msgr2=0x5590eb733c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:36.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.064+0000 7f1c949c6740 1 --2- 172.21.15.107:0/1175912860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb729450 0x5590eb733c50 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1c8401d040 tx=0x7f1c84019810 comp rx=0 tx=0).stop 2024-05-09T08:34:36.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.064+0000 7f1c949c6740 1 -- 172.21.15.107:0/1175912860 shutdown_connections 2024-05-09T08:34:36.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.064+0000 7f1c949c6740 1 --2- 172.21.15.107:0/1175912860 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb734630 0x5590eb730d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.064+0000 7f1c949c6740 1 --2- 172.21.15.107:0/1175912860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb729d70 0x5590eb72be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.064+0000 7f1c949c6740 1 --2- 172.21.15.107:0/1175912860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb729450 0x5590eb733c50 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.064+0000 7f1c949c6740 1 -- 172.21.15.107:0/1175912860 >> 172.21.15.107:0/1175912860 conn(0x5590eb725960 msgr2=0x5590eb731d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.064+0000 7f1c949c6740 1 -- 172.21.15.107:0/1175912860 shutdown_connections 2024-05-09T08:34:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.064+0000 7f1c949c6740 1 -- 172.21.15.107:0/1175912860 wait complete. 2024-05-09T08:34:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.065+0000 7f1c949c6740 1 -- start start 2024-05-09T08:34:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.065+0000 7f1c949c6740 1 Processor -- start 2024-05-09T08:34:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.065+0000 7f1c949c6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb729450 0x5590eb7a5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.065+0000 7f1c949c6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729d70 0x5590eb73c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.065+0000 7f1c949c6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb734630 0x5590eb73cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.065+0000 7f1c949c6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c1ad2d0 con 0x5590eb734630 2024-05-09T08:34:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.065+0000 7f1c949c6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c1ae740 con 0x5590eb729450 2024-05-09T08:34:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.065+0000 7f1c949c6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c1ad9a0 con 0x5590eb729d70 2024-05-09T08:34:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.065+0000 7f1c949c6740 10 client.0 _ll_register_callbacks cb 0x7f1c84009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.066+0000 7f1c8bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb734630 0x5590eb73cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.066+0000 7f1c8affd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729d70 0x5590eb73c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.066+0000 7f1c8bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb734630 0x5590eb73cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39338/0 (socket says 172.21.15.107:39338) 2024-05-09T08:34:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.066+0000 7f1c8bfff640 1 -- 172.21.15.107:0/1927430531 learned_addr learned my addr 172.21.15.107:0/1927430531 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.066+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/1927430531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb729450 0x5590eb7a5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.066+0000 7f1c8affd640 1 -- 172.21.15.107:0/1927430531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb729450 msgr2=0x5590eb7a5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.066+0000 7f1c8affd640 1 --2- 172.21.15.107:0/1927430531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb729450 0x5590eb7a5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.066+0000 7f1c8affd640 1 -- 172.21.15.107:0/1927430531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb734630 msgr2=0x5590eb73cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.066+0000 7f1c8affd640 1 --2- 172.21.15.107:0/1927430531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb734630 0x5590eb73cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.066+0000 7f1c8affd640 1 -- 172.21.15.107:0/1927430531 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590eb72e240 con 0x5590eb729d70 2024-05-09T08:34:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.066+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/1927430531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb734630 0x5590eb73cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.066+0000 7f1c8affd640 1 --2- 172.21.15.107:0/1927430531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729d70 0x5590eb73c680 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1c80006010 tx=0x7f1c80005110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2618fb1fcbf6a36 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.066+0000 7f1c6affd640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:34:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.066+0000 7f1c6affd640 1 -- 172.21.15.107:0/1927430531 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c80049070 con 0x5590eb729d70 2024-05-09T08:34:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.066+0000 7f1c949c6740 1 -- 172.21.15.107:0/1927430531 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c84020040 con 0x5590eb729d70 2024-05-09T08:34:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.066+0000 7f1c6affd640 1 -- 172.21.15.107:0/1927430531 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1c80045330 con 0x5590eb729d70 2024-05-09T08:34:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.067+0000 7f1c6affd640 1 -- 172.21.15.107:0/1927430531 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1c80053050 con 0x5590eb729d70 2024-05-09T08:34:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.067+0000 7f1c949c6740 10 client.25564 fetch_fsmap learned FSMap version 25 2024-05-09T08:34:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.067+0000 7f1c949c6740 1 -- 172.21.15.107:0/1927430531 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5590eb7aec00 con 0x5590eb729d70 2024-05-09T08:34:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.067+0000 7f1c6affd640 1 -- 172.21.15.107:0/1927430531 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c84020040 con 0x5590eb729d70 2024-05-09T08:34:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.067+0000 7f1c6affd640 1 -- 172.21.15.107:0/1927430531 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1c800035f0 con 0x5590eb729d70 2024-05-09T08:34:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.067+0000 7f1c949c6740 10 client.25564 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T08:34:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.067+0000 7f1c949c6740 10 client.25564 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.067+0000 7f1c949c6740 20 client.25564 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.067+0000 7f1c949c6740 1 --2- 172.21.15.107:0/1927430531 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5590eb744250 0x5590eb7b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:36.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.067+0000 7f1c949c6740 4 client.25564 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:34:36.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.067+0000 7f1c949c6740 1 -- 172.21.15.107:0/1927430531 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5590eb695dc0 con 0x5590eb744250 2024-05-09T08:34:36.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.067+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/1927430531 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5590eb744250 0x5590eb7b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:36.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.068+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/1927430531 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5590eb744250 0x5590eb7b13f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed49e7d969059a4 server_cookie=e0c13408557cdafd in_seq=0 out_seq=0 2024-05-09T08:34:36.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.068+0000 7f1c6affd640 10 client.25564 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:36.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.071+0000 7f1c6affd640 1 -- 172.21.15.107:0/1927430531 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5590eb695dc0 con 0x5590eb744250 2024-05-09T08:34:36.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.071+0000 7f1c6affd640 10 client.25564 handle_command_reply: tid=0 2024-05-09T08:34:36.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.071+0000 7f1c949c6740 1 client.25564 shutdown 2024-05-09T08:34:36.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.071+0000 7f1c949c6740 1 -- 172.21.15.107:0/1927430531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729d70 msgr2=0x5590eb73c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:36.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.071+0000 7f1c949c6740 1 --2- 172.21.15.107:0/1927430531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729d70 0x5590eb73c680 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1c80006010 tx=0x7f1c80005110 comp rx=0 tx=0).stop 2024-05-09T08:34:36.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.072+0000 7f1c949c6740 1 -- 172.21.15.107:0/1927430531 shutdown_connections 2024-05-09T08:34:36.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.072+0000 7f1c949c6740 1 --2- 172.21.15.107:0/1927430531 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5590eb744250 0x5590eb7b13f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.072+0000 7f1c949c6740 1 --2- 172.21.15.107:0/1927430531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590eb734630 0x5590eb73cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.072+0000 7f1c949c6740 1 --2- 172.21.15.107:0/1927430531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590eb729d70 0x5590eb73c680 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.072+0000 7f1c949c6740 1 --2- 172.21.15.107:0/1927430531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590eb729450 0x5590eb7a5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.072+0000 7f1c949c6740 1 -- 172.21.15.107:0/1927430531 >> 172.21.15.107:0/1927430531 conn(0x5590eb725960 msgr2=0x5590eb73f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:36.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.072+0000 7f1c6affd640 0 client.25564 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:36.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.072+0000 7f1c949c6740 1 -- 172.21.15.107:0/1927430531 shutdown_connections 2024-05-09T08:34:36.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.072+0000 7f1c949c6740 1 -- 172.21.15.107:0/1927430531 wait complete. 2024-05-09T08:34:36.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.072+0000 7f1c949c6740 20 client.25564 trim_cache size 0 max 16384 2024-05-09T08:34:36.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:34:36.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:34:36.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:34:36.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:34:36.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.072+0000 7f1c91cba640 1 -- 172.21.15.107:0/483862805 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c740a2f50 msgr2=0x7f1c740a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:36.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.072+0000 7f1c91cba640 1 --2- 172.21.15.107:0/483862805 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c740a2f50 0x7f1c740a5410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0cc6f0 tx=0x7f1c7c097040 comp rx=0 tx=0).stop 2024-05-09T08:34:36.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.072+0000 7f1c91cba640 1 -- 172.21.15.107:0/483862805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c102ac0 msgr2=0x7f1c8c198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:36.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.072+0000 7f1c91cba640 1 --2- 172.21.15.107:0/483862805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c102ac0 0x7f1c8c198060 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00d950 tx=0x7f1c7c00de10 comp rx=0 tx=0).stop 2024-05-09T08:34:36.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.073+0000 7f1c91cba640 1 -- 172.21.15.107:0/483862805 shutdown_connections 2024-05-09T08:34:36.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.073+0000 7f1c91cba640 1 --2- 172.21.15.107:0/483862805 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c740a2f50 0x7f1c740a5410 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.073+0000 7f1c91cba640 1 --2- 172.21.15.107:0/483862805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c8c10be00 0x7f1c8c19c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.073+0000 7f1c91cba640 1 --2- 172.21.15.107:0/483862805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c8c1033e0 0x7f1c8c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.073+0000 7f1c91cba640 1 --2- 172.21.15.107:0/483862805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c102ac0 0x7f1c8c198060 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:36.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.073+0000 7f1c91cba640 1 -- 172.21.15.107:0/483862805 >> 172.21.15.107:0/483862805 conn(0x7f1c8c0778b0 msgr2=0x7f1c8c0fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:36.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.073+0000 7f1c91cba640 1 -- 172.21.15.107:0/483862805 shutdown_connections 2024-05-09T08:34:36.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:36.073+0000 7f1c91cba640 1 -- 172.21.15.107:0/483862805 wait complete. 2024-05-09T08:34:36.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:36 smithi107 ceph-mon[30122]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T08:34:36.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:36 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:34:36.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:36 smithi157 ceph-mon[33614]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T08:34:36.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:36 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:34:36.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:36 smithi110 ceph-mon[33622]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T08:34:36.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:36 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:34:38.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:38 smithi107 ceph-mon[30122]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 511 B/s wr, 9 op/s 2024-05-09T08:34:38.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:38 smithi157 ceph-mon[33614]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 511 B/s wr, 9 op/s 2024-05-09T08:34:38.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:38 smithi110 ceph-mon[33622]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 511 B/s wr, 9 op/s 2024-05-09T08:34:40.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:40 smithi107 ceph-mon[30122]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T08:34:40.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:40 smithi157 ceph-mon[33614]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T08:34:40.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:40 smithi110 ceph-mon[33622]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T08:34:42.416 INFO:tasks.workunit.client.0.smithi107.stderr:Note: switching to 'be85c202969d25af6e130678e7d73a20e9c4ca11'. 2024-05-09T08:34:42.416 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-05-09T08:34:42.416 INFO:tasks.workunit.client.0.smithi107.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-09T08:34:42.416 INFO:tasks.workunit.client.0.smithi107.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-09T08:34:42.416 INFO:tasks.workunit.client.0.smithi107.stderr:state without impacting any branches by switching back to a branch. 2024-05-09T08:34:42.417 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-05-09T08:34:42.417 INFO:tasks.workunit.client.0.smithi107.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-09T08:34:42.417 INFO:tasks.workunit.client.0.smithi107.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-09T08:34:42.417 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-05-09T08:34:42.417 INFO:tasks.workunit.client.0.smithi107.stderr: git switch -c 2024-05-09T08:34:42.417 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-05-09T08:34:42.417 INFO:tasks.workunit.client.0.smithi107.stderr:Or undo this operation with: 2024-05-09T08:34:42.417 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-05-09T08:34:42.417 INFO:tasks.workunit.client.0.smithi107.stderr: git switch - 2024-05-09T08:34:42.417 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-05-09T08:34:42.417 INFO:tasks.workunit.client.0.smithi107.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-09T08:34:42.417 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-05-09T08:34:42.417 INFO:tasks.workunit.client.0.smithi107.stderr:HEAD is now at be85c202969 Merge pull request #57242 from cbodley/wip-65767-squid 2024-05-09T08:34:42.420 DEBUG:teuthology.orchestra.run.smithi107:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-09T08:34:42.482 INFO:tasks.workunit.client.0.smithi107.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-09T08:34:42.485 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-09T08:34:42.485 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-09T08:34:42.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:42 smithi107 ceph-mon[30122]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T08:34:42.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:42 smithi157 ceph-mon[33614]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T08:34:42.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:42 smithi110 ceph-mon[33622]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T08:34:43.165 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-09T08:34:43.220 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-09T08:34:43.259 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-09T08:34:43.261 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-09T08:34:43.261 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-09T08:34:43.299 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-09T08:34:43.303 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:34:43.303 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-09T08:34:43.359 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-05-09T08:34:43.360 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-05-09T08:34:43.360 DEBUG:teuthology.orchestra.run.smithi107:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=be85c202969d25af6e130678e7d73a20e9c4ca11 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-05-09T08:34:43.426 INFO:tasks.workunit.client.0.smithi107.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-05-09T08:34:43.426 INFO:tasks.workunit.client.0.smithi107.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-05-09T08:34:43.426 INFO:tasks.workunit.client.0.smithi107.stderr:+ trap -- retry EXIT 2024-05-09T08:34:43.427 INFO:tasks.workunit.client.0.smithi107.stderr:+ rm -rf ffsb 2024-05-09T08:34:43.427 INFO:tasks.workunit.client.0.smithi107.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-05-09T08:34:43.432 INFO:tasks.workunit.client.0.smithi107.stderr:Cloning into 'ffsb'... 2024-05-09T08:34:44.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:44 smithi107 ceph-mon[30122]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:34:44.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:44 smithi157 ceph-mon[33614]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:34:44.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:44 smithi110 ceph-mon[33622]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T08:34:45.133 INFO:tasks.workunit.client.0.smithi107.stderr:+ trap - EXIT 2024-05-09T08:34:45.133 INFO:tasks.workunit.client.0.smithi107.stderr:+ cd ffsb 2024-05-09T08:34:45.133 INFO:tasks.workunit.client.0.smithi107.stderr:+ ./configure 2024-05-09T08:34:45.263 INFO:tasks.workunit.client.0.smithi107.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-05-09T08:34:46.276 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether build environment is sane... yes 2024-05-09T08:34:46.294 INFO:tasks.workunit.client.0.smithi107.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-05-09T08:34:46.294 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gawk... gawk 2024-05-09T08:34:46.318 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether make sets $(MAKE)... yes 2024-05-09T08:34:46.365 INFO:tasks.workunit.client.0.smithi107.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-05-09T08:34:46.365 INFO:tasks.workunit.client.0.smithi107.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-05-09T08:34:46.387 INFO:tasks.workunit.client.0.smithi107.stdout:checking for style of include used by make... GNU 2024-05-09T08:34:46.392 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc... gcc 2024-05-09T08:34:46.479 INFO:tasks.workunit.client.0.smithi107.stdout:checking for C compiler default output file name... a.out 2024-05-09T08:34:46.481 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether the C compiler works... yes 2024-05-09T08:34:46.485 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are cross compiling... no 2024-05-09T08:34:46.541 INFO:tasks.workunit.client.0.smithi107.stdout:checking for suffix of executables... 2024-05-09T08:34:46.594 INFO:tasks.workunit.client.0.smithi107.stdout:checking for suffix of object files... o 2024-05-09T08:34:46.636 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are using the GNU C compiler... yes 2024-05-09T08:34:46.679 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether gcc accepts -g... yes 2024-05-09T08:34:46.745 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc option to accept ISO C89... none needed 2024-05-09T08:34:46.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:46 smithi157 ceph-mon[33614]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T08:34:46.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:46 smithi110 ceph-mon[33622]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T08:34:46.923 INFO:tasks.workunit.client.0.smithi107.stdout:checking dependency style of gcc... gcc3 2024-05-09T08:34:46.998 INFO:tasks.workunit.client.0.smithi107.stdout:checking how to run the C preprocessor... gcc -E 2024-05-09T08:34:47.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:46 smithi107 ceph-mon[30122]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T08:34:47.058 INFO:tasks.workunit.client.0.smithi107.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-05-09T08:34:47.060 INFO:tasks.workunit.client.0.smithi107.stdout:checking for egrep... /usr/bin/grep -E 2024-05-09T08:34:47.073 INFO:tasks.workunit.client.0.smithi107.stdout:checking for AIX... no 2024-05-09T08:34:47.102 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc... (cached) gcc 2024-05-09T08:34:47.112 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-05-09T08:34:47.113 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether gcc accepts -g... (cached) yes 2024-05-09T08:34:47.113 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-05-09T08:34:47.114 INFO:tasks.workunit.client.0.smithi107.stdout:checking dependency style of gcc... (cached) gcc3 2024-05-09T08:34:47.189 INFO:tasks.workunit.client.0.smithi107.stdout:checking for main in -lm... yes 2024-05-09T08:34:47.264 INFO:tasks.workunit.client.0.smithi107.stdout:checking for main in -lpthread... yes 2024-05-09T08:34:47.474 INFO:tasks.workunit.client.0.smithi107.stdout:checking for ANSI C header files... yes 2024-05-09T08:34:47.545 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-05-09T08:34:47.701 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/types.h... yes 2024-05-09T08:34:47.770 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/stat.h... yes 2024-05-09T08:34:47.838 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdlib.h... yes 2024-05-09T08:34:47.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:47 smithi157 ceph-mon[33614]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:34:47.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:47 smithi110 ceph-mon[33622]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:34:47.904 INFO:tasks.workunit.client.0.smithi107.stdout:checking for string.h... yes 2024-05-09T08:34:47.975 INFO:tasks.workunit.client.0.smithi107.stdout:checking for memory.h... yes 2024-05-09T08:34:48.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:47 smithi107 ceph-mon[30122]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:34:48.045 INFO:tasks.workunit.client.0.smithi107.stdout:checking for strings.h... yes 2024-05-09T08:34:48.115 INFO:tasks.workunit.client.0.smithi107.stdout:checking for inttypes.h... yes 2024-05-09T08:34:48.185 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdint.h... yes 2024-05-09T08:34:48.253 INFO:tasks.workunit.client.0.smithi107.stdout:checking for unistd.h... yes 2024-05-09T08:34:48.339 INFO:tasks.workunit.client.0.smithi107.stdout:checking pthread.h usability... yes 2024-05-09T08:34:48.370 INFO:tasks.workunit.client.0.smithi107.stdout:checking pthread.h presence... yes 2024-05-09T08:34:48.370 INFO:tasks.workunit.client.0.smithi107.stdout:checking for pthread.h... yes 2024-05-09T08:34:48.443 INFO:tasks.workunit.client.0.smithi107.stdout:checking fcntl.h usability... yes 2024-05-09T08:34:48.472 INFO:tasks.workunit.client.0.smithi107.stdout:checking fcntl.h presence... yes 2024-05-09T08:34:48.472 INFO:tasks.workunit.client.0.smithi107.stdout:checking for fcntl.h... yes 2024-05-09T08:34:48.546 INFO:tasks.workunit.client.0.smithi107.stdout:checking limits.h usability... yes 2024-05-09T08:34:48.581 INFO:tasks.workunit.client.0.smithi107.stdout:checking limits.h presence... yes 2024-05-09T08:34:48.581 INFO:tasks.workunit.client.0.smithi107.stdout:checking for limits.h... yes 2024-05-09T08:34:48.606 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdint.h... (cached) yes 2024-05-09T08:34:48.669 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/time.h usability... yes 2024-05-09T08:34:48.699 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/time.h presence... yes 2024-05-09T08:34:48.699 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/time.h... yes 2024-05-09T08:34:48.722 INFO:tasks.workunit.client.0.smithi107.stdout:checking for unistd.h... (cached) yes 2024-05-09T08:34:48.789 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/vfs.h usability... yes 2024-05-09T08:34:48.817 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/vfs.h presence... yes 2024-05-09T08:34:48.817 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/vfs.h... yes 2024-05-09T08:34:48.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:48 smithi157 ceph-mon[33614]: pgmap v46: 129 pgs: 129 active+clean; 4.3 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 275 KiB/s wr, 48 op/s 2024-05-09T08:34:48.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:48 smithi110 ceph-mon[33622]: pgmap v46: 129 pgs: 129 active+clean; 4.3 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 275 KiB/s wr, 48 op/s 2024-05-09T08:34:48.887 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/limits.h usability... no 2024-05-09T08:34:48.921 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/limits.h presence... no 2024-05-09T08:34:48.922 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/limits.h... no 2024-05-09T08:34:48.982 INFO:tasks.workunit.client.0.smithi107.stdout:checking for an ANSI C-conforming const... yes 2024-05-09T08:34:49.030 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-05-09T08:34:49.031 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:48 smithi107 ceph-mon[30122]: pgmap v46: 129 pgs: 129 active+clean; 4.3 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 275 KiB/s wr, 48 op/s 2024-05-09T08:34:49.031 INFO:tasks.workunit.client.0.smithi107.stdout:checking for function prototypes... yes 2024-05-09T08:34:49.077 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether setvbuf arguments are reversed... no 2024-05-09T08:34:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.164+0000 7fea9d170640 1 -- 172.21.15.107:0/4192779494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea981051c0 msgr2=0x7fea9810cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.164+0000 7fea9d170640 1 --2- 172.21.15.107:0/4192779494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea981051c0 0x7fea9810cd90 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fea90009ca0 tx=0x7fea9002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.165+0000 7fea9d170640 1 -- 172.21.15.107:0/4192779494 shutdown_connections 2024-05-09T08:34:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.165+0000 7fea9d170640 1 --2- 172.21.15.107:0/4192779494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea981051c0 0x7fea9810cd90 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.165+0000 7fea9d170640 1 --2- 172.21.15.107:0/4192779494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea98100aa0 0x7fea98104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.165+0000 7fea9d170640 1 --2- 172.21.15.107:0/4192779494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea981000f0 0x7fea981004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.165+0000 7fea9d170640 1 -- 172.21.15.107:0/4192779494 >> 172.21.15.107:0/4192779494 conn(0x7fea98076cc0 msgr2=0x7fea980fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.165+0000 7fea9d170640 1 -- 172.21.15.107:0/4192779494 shutdown_connections 2024-05-09T08:34:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.165+0000 7fea9d170640 1 -- 172.21.15.107:0/4192779494 wait complete. 2024-05-09T08:34:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.165+0000 7fea9d170640 1 Processor -- start 2024-05-09T08:34:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.165+0000 7fea9d170640 1 -- start start 2024-05-09T08:34:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea9d170640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea981000f0 0x7fea98197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea9d170640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea981051c0 0x7fea981984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea9d170640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9819c800 0x7fea9819ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea9d170640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fea9810f8f0 con 0x7fea981000f0 2024-05-09T08:34:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea9d170640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fea9810f770 con 0x7fea981051c0 2024-05-09T08:34:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea9d170640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fea9810fa70 con 0x7fea9819c800 2024-05-09T08:34:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea96d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea981000f0 0x7fea98197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea9659c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea981051c0 0x7fea981984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea9759e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9819c800 0x7fea9819ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea96d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea981000f0 0x7fea98197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41064/0 (socket says 172.21.15.107:41064) 2024-05-09T08:34:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea96d9d640 1 -- 172.21.15.107:0/359405300 learned_addr learned my addr 172.21.15.107:0/359405300 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea96d9d640 1 -- 172.21.15.107:0/359405300 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9819c800 msgr2=0x7fea9819ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea96d9d640 1 --2- 172.21.15.107:0/359405300 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9819c800 0x7fea9819ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea96d9d640 1 -- 172.21.15.107:0/359405300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea981051c0 msgr2=0x7fea981984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea96d9d640 1 --2- 172.21.15.107:0/359405300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea981051c0 0x7fea981984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea96d9d640 1 -- 172.21.15.107:0/359405300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9819d2f0 con 0x7fea981000f0 2024-05-09T08:34:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea9759e640 1 --2- 172.21.15.107:0/359405300 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9819c800 0x7fea9819ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea96d9d640 1 --2- 172.21.15.107:0/359405300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea981000f0 0x7fea98197f90 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fea8800e9f0 tx=0x7fea8800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.166+0000 7fea7ffff640 1 -- 172.21.15.107:0/359405300 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8800cd80 con 0x7fea981000f0 2024-05-09T08:34:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.167+0000 7fea7ffff640 1 -- 172.21.15.107:0/359405300 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea880054d0 con 0x7fea981000f0 2024-05-09T08:34:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.167+0000 7fea7ffff640 1 -- 172.21.15.107:0/359405300 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea88010490 con 0x7fea981000f0 2024-05-09T08:34:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.167+0000 7fea9d170640 1 -- 172.21.15.107:0/359405300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea9819d5e0 con 0x7fea981000f0 2024-05-09T08:34:49.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.167+0000 7fea9d170640 1 -- 172.21.15.107:0/359405300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea981a4d80 con 0x7fea981000f0 2024-05-09T08:34:49.170 INFO:tasks.workunit.client.0.smithi107.stdout:checking for system... yes 2024-05-09T08:34:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.169+0000 7fea7ffff640 1 -- 172.21.15.107:0/359405300 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 138118+0+0 (secure 0 0 0) 0x7fea88005020 con 0x7fea981000f0 2024-05-09T08:34:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.169+0000 7fea9fe7d740 1 Processor -- start 2024-05-09T08:34:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.169+0000 7fea9fe7d740 1 -- start start 2024-05-09T08:34:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea7ffff640 1 --2- 172.21.15.107:0/359405300 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fea800a2f50 0x7fea800a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea9fe7d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4b450 0x562c6ad4b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea9fe7d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad4bd70 0x562c6ad560f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea9fe7d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad56630 0x562c6ad58a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea9fe7d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562c6acb0340 con 0x562c6ad4b450 2024-05-09T08:34:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea9fe7d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562c6ac58290 con 0x562c6ad4bd70 2024-05-09T08:34:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea9fe7d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x562c6acac300 con 0x562c6ad56630 2024-05-09T08:34:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea96d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4b450 0x562c6ad4b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea9759e640 1 --2- 172.21.15.107:0/359405300 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fea800a2f50 0x7fea800a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea96d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4b450 0x562c6ad4b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41078/0 (socket says 172.21.15.107:41078) 2024-05-09T08:34:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea9659c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad4bd70 0x562c6ad560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea96d9d640 1 -- 172.21.15.107:0/2630631347 learned_addr learned my addr 172.21.15.107:0/2630631347 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea7ffff640 1 -- 172.21.15.107:0/359405300 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fea880cfdf0 con 0x7fea981000f0 2024-05-09T08:34:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea9759e640 1 --2- 172.21.15.107:0/2630631347 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad56630 0x562c6ad58a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea96d9d640 1 -- 172.21.15.107:0/2630631347 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad56630 msgr2=0x562c6ad58a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea96d9d640 1 --2- 172.21.15.107:0/2630631347 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad56630 0x562c6ad58a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea96d9d640 1 -- 172.21.15.107:0/2630631347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad4bd70 msgr2=0x562c6ad560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea96d9d640 1 --2- 172.21.15.107:0/2630631347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad4bd70 0x562c6ad560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea96d9d640 1 -- 172.21.15.107:0/2630631347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c6ac7c8e0 con 0x562c6ad4b450 2024-05-09T08:34:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea96d9d640 1 --2- 172.21.15.107:0/2630631347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4b450 0x562c6ad4b830 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fea880981d0 tx=0x7fea8809bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a0a772762021178 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.170+0000 7fea9759e640 1 --2- 172.21.15.107:0/359405300 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fea800a2f50 0x7fea800a5410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fea90006010 tx=0x7fea90031970 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.171+0000 7fea7d7fa640 1 -- 172.21.15.107:0/2630631347 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea880b4f70 con 0x562c6ad4b450 2024-05-09T08:34:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.171+0000 7fea7d7fa640 1 -- 172.21.15.107:0/2630631347 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea88005310 con 0x562c6ad4b450 2024-05-09T08:34:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.171+0000 7fea7d7fa640 1 -- 172.21.15.107:0/2630631347 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea880a49e0 con 0x562c6ad4b450 2024-05-09T08:34:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.171+0000 7fea9fe7d740 1 -- 172.21.15.107:0/2630631347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4b450 msgr2=0x562c6ad4b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.171+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/2630631347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4b450 0x562c6ad4b830 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fea880981d0 tx=0x7fea8809bc00 comp rx=0 tx=0).stop 2024-05-09T08:34:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.171+0000 7fea9fe7d740 1 -- 172.21.15.107:0/2630631347 shutdown_connections 2024-05-09T08:34:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.171+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/2630631347 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad56630 0x562c6ad58a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.171+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/2630631347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad4bd70 0x562c6ad560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.171+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/2630631347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4b450 0x562c6ad4b830 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.171+0000 7fea9fe7d740 1 -- 172.21.15.107:0/2630631347 >> 172.21.15.107:0/2630631347 conn(0x562c6ad47960 msgr2=0x562c6ad47d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.171+0000 7fea9fe7d740 1 -- 172.21.15.107:0/2630631347 shutdown_connections 2024-05-09T08:34:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.171+0000 7fea9fe7d740 1 -- 172.21.15.107:0/2630631347 wait complete. 2024-05-09T08:34:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9fe7d740 1 -- start start 2024-05-09T08:34:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9fe7d740 1 Processor -- start 2024-05-09T08:34:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9fe7d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 0x562c6ad535f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9fe7d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad53b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9fe7d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 0x562c6ad54070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9fe7d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562c6ad59200 con 0x562c6ad4bd70 2024-05-09T08:34:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9fe7d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562c6ac6b790 con 0x562c6ad56630 2024-05-09T08:34:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9fe7d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x562c6ad59380 con 0x562c6ad4b450 2024-05-09T08:34:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9fe7d740 10 client.0 _ll_register_callbacks cb 0x562c6aca6a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad53b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad53b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41080/0 (socket says 172.21.15.107:41080) 2024-05-09T08:34:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9659c640 1 -- 172.21.15.107:0/460915949 learned_addr learned my addr 172.21.15.107:0/460915949 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea96d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 0x562c6ad535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9759e640 1 --2- 172.21.15.107:0/460915949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 0x562c6ad54070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9659c640 1 -- 172.21.15.107:0/460915949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 msgr2=0x562c6ad535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9659c640 1 --2- 172.21.15.107:0/460915949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 0x562c6ad535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9659c640 1 -- 172.21.15.107:0/460915949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 msgr2=0x562c6ad54070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9659c640 1 --2- 172.21.15.107:0/460915949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 0x562c6ad54070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9659c640 1 -- 172.21.15.107:0/460915949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c6a9c4210 con 0x562c6ad4bd70 2024-05-09T08:34:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea96d9d640 1 --2- 172.21.15.107:0/460915949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 0x562c6ad535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.172+0000 7fea9759e640 1 --2- 172.21.15.107:0/460915949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 0x562c6ad54070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.173+0000 7fea9659c640 1 --2- 172.21.15.107:0/460915949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad53b30 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00eaa0 tx=0x7fea8c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f343ff701d61f20a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.173+0000 7fea7aff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:34:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.173+0000 7fea7aff5640 1 -- 172.21.15.107:0/460915949 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c01d060 con 0x562c6ad4bd70 2024-05-09T08:34:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.173+0000 7fea7aff5640 1 -- 172.21.15.107:0/460915949 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fea8c00d950 con 0x562c6ad4bd70 2024-05-09T08:34:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.173+0000 7fea9fe7d740 1 -- 172.21.15.107:0/460915949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562c6acbda70 con 0x562c6ad4bd70 2024-05-09T08:34:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.173+0000 7fea7aff5640 1 -- 172.21.15.107:0/460915949 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fea8c024050 con 0x562c6ad4bd70 2024-05-09T08:34:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.173+0000 7fea9fe7d740 10 client.16089 fetch_fsmap learned FSMap version 26 2024-05-09T08:34:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.173+0000 7fea9fe7d740 1 -- 172.21.15.107:0/460915949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562c6aa90620 con 0x562c6ad4bd70 2024-05-09T08:34:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.173+0000 7fea7aff5640 1 -- 172.21.15.107:0/460915949 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x562c6acbda70 con 0x562c6ad4bd70 2024-05-09T08:34:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.173+0000 7fea7aff5640 1 -- 172.21.15.107:0/460915949 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fea8c017030 con 0x562c6ad4bd70 2024-05-09T08:34:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.173+0000 7fea9fe7d740 10 client.16089 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:34:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.173+0000 7fea9fe7d740 10 client.16089 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.173+0000 7fea9fe7d740 20 client.16089 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.173+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/460915949 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562c6ad66430 0x562c6ad688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.173+0000 7fea9fe7d740 4 client.16089 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:34:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.173+0000 7fea9fe7d740 1 -- 172.21.15.107:0/460915949 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c6acb7dc0 con 0x562c6ad66430 2024-05-09T08:34:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.174+0000 7fea96d9d640 1 --2- 172.21.15.107:0/460915949 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562c6ad66430 0x562c6ad688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.174+0000 7fea7aff5640 10 client.16089 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.174+0000 7fea96d9d640 1 --2- 172.21.15.107:0/460915949 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562c6ad66430 0x562c6ad688a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22644170be782298 server_cookie=42381c33daa7cbe8 in_seq=0 out_seq=0 2024-05-09T08:34:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.175+0000 7fea7aff5640 1 -- 172.21.15.107:0/460915949 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562c6acb7dc0 con 0x562c6ad66430 2024-05-09T08:34:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.175+0000 7fea7aff5640 10 client.16089 handle_command_reply: tid=0 2024-05-09T08:34:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.175+0000 7fea9fe7d740 1 client.16089 shutdown 2024-05-09T08:34:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.175+0000 7fea9fe7d740 1 -- 172.21.15.107:0/460915949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 msgr2=0x562c6ad53b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.175+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/460915949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad53b30 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00eaa0 tx=0x7fea8c00ef60 comp rx=0 tx=0).stop 2024-05-09T08:34:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.175+0000 7fea9fe7d740 1 -- 172.21.15.107:0/460915949 shutdown_connections 2024-05-09T08:34:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.175+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/460915949 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562c6ad66430 0x562c6ad688a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.175+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/460915949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 0x562c6ad54070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.175+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/460915949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad53b30 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.175+0000 7fea7aff5640 0 client.16089 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.175+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/460915949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 0x562c6ad535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.175+0000 7fea9fe7d740 1 -- 172.21.15.107:0/460915949 >> 172.21.15.107:0/460915949 conn(0x562c6ad47960 msgr2=0x562c6ad4d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.175+0000 7fea9fe7d740 1 -- 172.21.15.107:0/460915949 shutdown_connections 2024-05-09T08:34:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.175+0000 7fea9fe7d740 1 -- 172.21.15.107:0/460915949 wait complete. 2024-05-09T08:34:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.175+0000 7fea9fe7d740 20 client.16089 trim_cache size 0 max 16384 2024-05-09T08:34:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.193+0000 7fea9fe7d740 1 Processor -- start 2024-05-09T08:34:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.193+0000 7fea9fe7d740 1 -- start start 2024-05-09T08:34:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.193+0000 7fea9fe7d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 0x562c6ad55d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.193+0000 7fea9fe7d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad620c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.193+0000 7fea9fe7d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 0x562c6ad6e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.193+0000 7fea9fe7d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fea981b1370 con 0x562c6ad4bd70 2024-05-09T08:34:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.193+0000 7fea9659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad620c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.193+0000 7fea9759e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 0x562c6ad6e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.193+0000 7fea9659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad620c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41084/0 (socket says 172.21.15.107:41084) 2024-05-09T08:34:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.193+0000 7fea9659c640 1 -- 172.21.15.107:0/1846271690 learned_addr learned my addr 172.21.15.107:0/1846271690 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.193+0000 7fea96d9d640 1 --2- 172.21.15.107:0/1846271690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 0x562c6ad55d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.193+0000 7fea9fe7d740 1 -- 172.21.15.107:0/1846271690 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fea981aec40 con 0x562c6ad56630 2024-05-09T08:34:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.193+0000 7fea9fe7d740 1 -- 172.21.15.107:0/1846271690 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fea981aca90 con 0x562c6ad4b450 2024-05-09T08:34:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.194+0000 7fea9759e640 1 -- 172.21.15.107:0/1846271690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 msgr2=0x562c6ad55d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.194+0000 7fea9759e640 1 --2- 172.21.15.107:0/1846271690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 0x562c6ad55d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.194+0000 7fea9759e640 1 -- 172.21.15.107:0/1846271690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 msgr2=0x562c6ad620c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.194+0000 7fea9759e640 1 --2- 172.21.15.107:0/1846271690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad620c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.194+0000 7fea9759e640 1 -- 172.21.15.107:0/1846271690 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c6ad4bba0 con 0x562c6ad56630 2024-05-09T08:34:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.194+0000 7fea9659c640 1 --2- 172.21.15.107:0/1846271690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad620c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.194+0000 7fea9759e640 1 --2- 172.21.15.107:0/1846271690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 0x562c6ad6e200 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fea90031040 tx=0x7fea90004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f83ab8b3cd8bd46 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.194+0000 7fea7cff9640 1 -- 172.21.15.107:0/1846271690 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea90055070 con 0x562c6ad56630 2024-05-09T08:34:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.194+0000 7fea7cff9640 1 -- 172.21.15.107:0/1846271690 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea90036030 con 0x562c6ad56630 2024-05-09T08:34:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.194+0000 7fea7cff9640 1 -- 172.21.15.107:0/1846271690 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9004c400 con 0x562c6ad56630 2024-05-09T08:34:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.195+0000 7fea9fe7d740 1 -- 172.21.15.107:0/1846271690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 msgr2=0x562c6ad6e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.195+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/1846271690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 0x562c6ad6e200 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fea90031040 tx=0x7fea90004a40 comp rx=0 tx=0).stop 2024-05-09T08:34:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.195+0000 7fea9fe7d740 1 -- 172.21.15.107:0/1846271690 shutdown_connections 2024-05-09T08:34:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.195+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/1846271690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 0x562c6ad6e200 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.195+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/1846271690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad620c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.195+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/1846271690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 0x562c6ad55d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.195+0000 7fea9fe7d740 1 -- 172.21.15.107:0/1846271690 >> 172.21.15.107:0/1846271690 conn(0x562c6ad47960 msgr2=0x562c6ad47d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.195+0000 7fea9fe7d740 1 -- 172.21.15.107:0/1846271690 shutdown_connections 2024-05-09T08:34:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.195+0000 7fea9fe7d740 1 -- 172.21.15.107:0/1846271690 wait complete. 2024-05-09T08:34:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.195+0000 7fea9fe7d740 1 -- start start 2024-05-09T08:34:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.196+0000 7fea9fe7d740 1 Processor -- start 2024-05-09T08:34:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.196+0000 7fea9fe7d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 0x562c6ad5e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.196+0000 7fea9fe7d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad4fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.196+0000 7fea9fe7d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 0x562c6ad503e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.196+0000 7fea9fe7d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fea981ad100 con 0x562c6ad4bd70 2024-05-09T08:34:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.196+0000 7fea9fe7d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fea981ae570 con 0x562c6ad56630 2024-05-09T08:34:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.196+0000 7fea9fe7d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fea981ad7d0 con 0x562c6ad4b450 2024-05-09T08:34:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.196+0000 7fea9fe7d740 10 client.0 _ll_register_callbacks cb 0x7fea8c00b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:34:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.196+0000 7fea9759e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 0x562c6ad503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.196+0000 7fea9759e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 0x562c6ad503e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:48614/0 (socket says 172.21.15.107:48614) 2024-05-09T08:34:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.196+0000 7fea96d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 0x562c6ad5e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.196+0000 7fea9759e640 1 -- 172.21.15.107:0/1344250218 learned_addr learned my addr 172.21.15.107:0/1344250218 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:34:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.197+0000 7fea9659c640 1 --2- 172.21.15.107:0/1344250218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad4fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.197+0000 7fea96d9d640 1 -- 172.21.15.107:0/1344250218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 msgr2=0x562c6ad503e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.197+0000 7fea96d9d640 1 --2- 172.21.15.107:0/1344250218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 0x562c6ad503e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.197+0000 7fea96d9d640 1 -- 172.21.15.107:0/1344250218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 msgr2=0x562c6ad4fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:34:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.197+0000 7fea96d9d640 1 --2- 172.21.15.107:0/1344250218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad4fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.197+0000 7fea96d9d640 1 -- 172.21.15.107:0/1344250218 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c6ad50b20 con 0x562c6ad4b450 2024-05-09T08:34:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.197+0000 7fea9759e640 1 --2- 172.21.15.107:0/1344250218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 0x562c6ad503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.197+0000 7fea9659c640 1 --2- 172.21.15.107:0/1344250218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad4fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:34:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.197+0000 7fea96d9d640 1 --2- 172.21.15.107:0/1344250218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 0x562c6ad5e8a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x562c6ad4c2e0 tx=0x7fea88097a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff5a2a2c73a8ce6d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:34:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.197+0000 7fea7aff5640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:34:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.197+0000 7fea7aff5640 1 -- 172.21.15.107:0/1344250218 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea880b4f70 con 0x562c6ad4b450 2024-05-09T08:34:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.197+0000 7fea9fe7d740 1 -- 172.21.15.107:0/1344250218 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea90036030 con 0x562c6ad4b450 2024-05-09T08:34:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.197+0000 7fea7aff5640 1 -- 172.21.15.107:0/1344250218 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fea8809bfd0 con 0x562c6ad4b450 2024-05-09T08:34:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.197+0000 7fea7aff5640 1 -- 172.21.15.107:0/1344250218 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fea880a49e0 con 0x562c6ad4b450 2024-05-09T08:34:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.197+0000 7fea9fe7d740 10 client.25582 fetch_fsmap learned FSMap version 26 2024-05-09T08:34:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.197+0000 7fea9fe7d740 1 -- 172.21.15.107:0/1344250218 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562c6add0dc0 con 0x562c6ad4b450 2024-05-09T08:34:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.198+0000 7fea7aff5640 1 -- 172.21.15.107:0/1344250218 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea90036030 con 0x562c6ad4b450 2024-05-09T08:34:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.198+0000 7fea7aff5640 1 -- 172.21.15.107:0/1344250218 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fea8809b1c0 con 0x562c6ad4b450 2024-05-09T08:34:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.198+0000 7fea9fe7d740 10 client.25582 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:34:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.198+0000 7fea9fe7d740 10 client.25582 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:34:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.198+0000 7fea9fe7d740 20 client.25582 populate_metadata read hostname 'smithi107' 2024-05-09T08:34:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.198+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/1344250218 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562c6add1100 0x562c6add34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:34:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.198+0000 7fea9fe7d740 4 client.25582 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:34:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.198+0000 7fea9fe7d740 1 -- 172.21.15.107:0/1344250218 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562c6ac358e0 con 0x562c6add1100 2024-05-09T08:34:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.198+0000 7fea9659c640 1 --2- 172.21.15.107:0/1344250218 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562c6add1100 0x562c6add34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:34:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.198+0000 7fea7aff5640 10 client.25582 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.198+0000 7fea9659c640 1 --2- 172.21.15.107:0/1344250218 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562c6add1100 0x562c6add34f0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=817acac9c3460b1a server_cookie=cb7625d84599ce in_seq=0 out_seq=0 2024-05-09T08:34:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.315+0000 7fea7aff5640 1 -- 172.21.15.107:0/1344250218 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562c6ac358e0 con 0x562c6add1100 2024-05-09T08:34:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.315+0000 7fea7aff5640 10 client.25582 handle_command_reply: tid=0 2024-05-09T08:34:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.315+0000 7fea9fe7d740 1 client.25582 shutdown 2024-05-09T08:34:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.315+0000 7fea9fe7d740 1 -- 172.21.15.107:0/1344250218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 msgr2=0x562c6ad5e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.315+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/1344250218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 0x562c6ad5e8a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x562c6ad4c2e0 tx=0x7fea88097a10 comp rx=0 tx=0).stop 2024-05-09T08:34:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.315+0000 7fea9fe7d740 1 -- 172.21.15.107:0/1344250218 shutdown_connections 2024-05-09T08:34:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.315+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/1344250218 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562c6add1100 0x562c6add34f0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.315+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/1344250218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562c6ad56630 0x562c6ad503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.315+0000 7fea7aff5640 0 client.25582 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:34:49.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.315+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/1344250218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c6ad4bd70 0x562c6ad4fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.315+0000 7fea9fe7d740 1 --2- 172.21.15.107:0/1344250218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c6ad4b450 0x562c6ad5e8a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.315+0000 7fea9fe7d740 1 -- 172.21.15.107:0/1344250218 >> 172.21.15.107:0/1344250218 conn(0x562c6ad47960 msgr2=0x562c6ad46170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:49.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.315+0000 7fea9fe7d740 1 -- 172.21.15.107:0/1344250218 shutdown_connections 2024-05-09T08:34:49.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.315+0000 7fea9fe7d740 1 -- 172.21.15.107:0/1344250218 wait complete. 2024-05-09T08:34:49.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.316+0000 7fea9fe7d740 20 client.25582 trim_cache size 0 max 16384 2024-05-09T08:34:49.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:34:49.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:34:49.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:34:49.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:34:49.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.316+0000 7fea9d170640 1 -- 172.21.15.107:0/359405300 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fea800a2f50 msgr2=0x7fea800a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:49.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.316+0000 7fea9d170640 1 --2- 172.21.15.107:0/359405300 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fea800a2f50 0x7fea800a5410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fea90006010 tx=0x7fea90031970 comp rx=0 tx=0).stop 2024-05-09T08:34:49.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.316+0000 7fea9d170640 1 -- 172.21.15.107:0/359405300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea981000f0 msgr2=0x7fea98197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:34:49.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.316+0000 7fea9d170640 1 --2- 172.21.15.107:0/359405300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea981000f0 0x7fea98197f90 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fea8800e9f0 tx=0x7fea8800edf0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.316+0000 7fea9d170640 1 -- 172.21.15.107:0/359405300 shutdown_connections 2024-05-09T08:34:49.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.316+0000 7fea9d170640 1 --2- 172.21.15.107:0/359405300 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fea800a2f50 0x7fea800a5410 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.316+0000 7fea9d170640 1 --2- 172.21.15.107:0/359405300 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea9819c800 0x7fea9819ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.316+0000 7fea9d170640 1 --2- 172.21.15.107:0/359405300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea981051c0 0x7fea981984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.316+0000 7fea9d170640 1 --2- 172.21.15.107:0/359405300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea981000f0 0x7fea98197f90 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:34:49.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.316+0000 7fea9d170640 1 -- 172.21.15.107:0/359405300 >> 172.21.15.107:0/359405300 conn(0x7fea98076cc0 msgr2=0x7fea98103930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:34:49.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.316+0000 7fea9d170640 1 -- 172.21.15.107:0/359405300 shutdown_connections 2024-05-09T08:34:49.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:34:49.316+0000 7fea9d170640 1 -- 172.21.15.107:0/359405300 wait complete. 2024-05-09T08:34:49.342 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gettimeofday... yes 2024-05-09T08:34:49.431 INFO:tasks.workunit.client.0.smithi107.stdout:checking for mkdir... yes 2024-05-09T08:34:49.520 INFO:tasks.workunit.client.0.smithi107.stdout:checking for strerror... yes 2024-05-09T08:34:49.615 INFO:tasks.workunit.client.0.smithi107.stdout:checking for open64... yes 2024-05-09T08:34:49.704 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stat64... yes 2024-05-09T08:34:49.799 INFO:tasks.workunit.client.0.smithi107.stdout:checking for fseeko64... yes 2024-05-09T08:34:49.884 INFO:tasks.workunit.client.0.smithi107.stdout:checking for lrand48_r... yes 2024-05-09T08:34:49.986 INFO:tasks.workunit.client.0.smithi107.stdout:checking for srand48_r... yes 2024-05-09T08:34:50.018 INFO:tasks.workunit.client.0.smithi107.stdout:configure: creating ./config.status 2024-05-09T08:34:50.155 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: creating Makefile 2024-05-09T08:34:50.172 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: creating config.h 2024-05-09T08:34:50.198 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: executing depfiles commands 2024-05-09T08:34:50.272 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: executing default commands 2024-05-09T08:34:50.347 INFO:tasks.workunit.client.0.smithi107.stderr:+ make 2024-05-09T08:34:50.375 INFO:tasks.workunit.client.0.smithi107.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-05-09T08:34:50.381 INFO:tasks.workunit.client.0.smithi107.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-05-09T08:34:50.382 INFO:tasks.workunit.client.0.smithi107.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-05-09T08:34:50.382 INFO:tasks.workunit.client.0.smithi107.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-05-09T08:34:50.382 INFO:tasks.workunit.client.0.smithi107.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-05-09T08:34:50.383 INFO:tasks.workunit.client.0.smithi107.stderr: any GNU archive site. 2024-05-09T08:34:50.386 INFO:tasks.workunit.client.0.smithi107.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-05-09T08:34:50.390 INFO:tasks.workunit.client.0.smithi107.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-05-09T08:34:50.391 INFO:tasks.workunit.client.0.smithi107.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-05-09T08:34:50.391 INFO:tasks.workunit.client.0.smithi107.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-05-09T08:34:50.391 INFO:tasks.workunit.client.0.smithi107.stderr: You might want to install the `Automake' and `Perl' packages. 2024-05-09T08:34:50.392 INFO:tasks.workunit.client.0.smithi107.stderr: Grab them from any GNU archive site. 2024-05-09T08:34:50.637 INFO:tasks.workunit.client.0.smithi107.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-05-09T08:34:50.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:50 smithi157 ceph-mon[33614]: pgmap v47: 129 pgs: 129 active+clean; 4.3 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 275 KiB/s wr, 44 op/s 2024-05-09T08:34:50.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:50 smithi110 ceph-mon[33622]: pgmap v47: 129 pgs: 129 active+clean; 4.3 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 275 KiB/s wr, 44 op/s 2024-05-09T08:34:51.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:50 smithi107 ceph-mon[30122]: pgmap v47: 129 pgs: 129 active+clean; 4.3 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 275 KiB/s wr, 44 op/s 2024-05-09T08:34:51.054 INFO:tasks.workunit.client.0.smithi107.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-05-09T08:34:51.054 INFO:tasks.workunit.client.0.smithi107.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-05-09T08:34:51.054 INFO:tasks.workunit.client.0.smithi107.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-05-09T08:34:51.054 INFO:tasks.workunit.client.0.smithi107.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-05-09T08:34:51.185 INFO:tasks.workunit.client.0.smithi107.stdout:/bin/sh ./config.status --recheck 2024-05-09T08:34:51.248 INFO:tasks.workunit.client.0.smithi107.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-05-09T08:34:51.402 INFO:tasks.workunit.client.0.smithi107.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-05-09T08:34:52.413 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether build environment is sane... yes 2024-05-09T08:34:52.423 INFO:tasks.workunit.client.0.smithi107.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-05-09T08:34:52.423 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gawk... gawk 2024-05-09T08:34:52.451 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether make sets $(MAKE)... yes 2024-05-09T08:34:52.497 INFO:tasks.workunit.client.0.smithi107.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-05-09T08:34:52.497 INFO:tasks.workunit.client.0.smithi107.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-05-09T08:34:52.519 INFO:tasks.workunit.client.0.smithi107.stdout:checking for style of include used by make... GNU 2024-05-09T08:34:52.525 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc... gcc 2024-05-09T08:34:52.655 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether the C compiler works... yes 2024-05-09T08:34:52.656 INFO:tasks.workunit.client.0.smithi107.stdout:checking for C compiler default output file name... a.out 2024-05-09T08:34:52.712 INFO:tasks.workunit.client.0.smithi107.stdout:checking for suffix of executables... 2024-05-09T08:34:52.765 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are cross compiling... no 2024-05-09T08:34:52.815 INFO:tasks.workunit.client.0.smithi107.stdout:checking for suffix of object files... o 2024-05-09T08:34:52.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:52 smithi157 ceph-mon[33614]: pgmap v48: 129 pgs: 129 active+clean; 4.3 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 275 KiB/s wr, 44 op/s 2024-05-09T08:34:52.848 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are using the GNU C compiler... yes 2024-05-09T08:34:52.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:52 smithi110 ceph-mon[33622]: pgmap v48: 129 pgs: 129 active+clean; 4.3 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 275 KiB/s wr, 44 op/s 2024-05-09T08:34:52.881 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether gcc accepts -g... yes 2024-05-09T08:34:52.926 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc option to accept ISO C89... none needed 2024-05-09T08:34:53.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:52 smithi107 ceph-mon[30122]: pgmap v48: 129 pgs: 129 active+clean; 4.3 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 275 KiB/s wr, 44 op/s 2024-05-09T08:34:53.097 INFO:tasks.workunit.client.0.smithi107.stdout:checking dependency style of gcc... gcc3 2024-05-09T08:34:53.168 INFO:tasks.workunit.client.0.smithi107.stdout:checking how to run the C preprocessor... gcc -E 2024-05-09T08:34:53.244 INFO:tasks.workunit.client.0.smithi107.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-05-09T08:34:53.246 INFO:tasks.workunit.client.0.smithi107.stdout:checking for egrep... /usr/bin/grep -E 2024-05-09T08:34:53.427 INFO:tasks.workunit.client.0.smithi107.stdout:checking for ANSI C header files... yes 2024-05-09T08:34:53.478 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/types.h... yes 2024-05-09T08:34:53.534 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/stat.h... yes 2024-05-09T08:34:53.598 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdlib.h... yes 2024-05-09T08:34:53.655 INFO:tasks.workunit.client.0.smithi107.stdout:checking for string.h... yes 2024-05-09T08:34:53.711 INFO:tasks.workunit.client.0.smithi107.stdout:checking for memory.h... yes 2024-05-09T08:34:53.767 INFO:tasks.workunit.client.0.smithi107.stdout:checking for strings.h... yes 2024-05-09T08:34:53.822 INFO:tasks.workunit.client.0.smithi107.stdout:checking for inttypes.h... yes 2024-05-09T08:34:53.878 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdint.h... yes 2024-05-09T08:34:53.961 INFO:tasks.workunit.client.0.smithi107.stdout:checking for unistd.h... yes 2024-05-09T08:34:54.023 INFO:tasks.workunit.client.0.smithi107.stdout:checking minix/config.h usability... no 2024-05-09T08:34:54.064 INFO:tasks.workunit.client.0.smithi107.stdout:checking minix/config.h presence... no 2024-05-09T08:34:54.064 INFO:tasks.workunit.client.0.smithi107.stdout:checking for minix/config.h... no 2024-05-09T08:34:54.107 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-05-09T08:34:54.117 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc... (cached) gcc 2024-05-09T08:34:54.173 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-05-09T08:34:54.173 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether gcc accepts -g... (cached) yes 2024-05-09T08:34:54.173 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-05-09T08:34:54.173 INFO:tasks.workunit.client.0.smithi107.stdout:checking dependency style of gcc... (cached) gcc3 2024-05-09T08:34:54.234 INFO:tasks.workunit.client.0.smithi107.stdout:checking for main in -lm... yes 2024-05-09T08:34:54.299 INFO:tasks.workunit.client.0.smithi107.stdout:checking for main in -lpthread... yes 2024-05-09T08:34:54.308 INFO:tasks.workunit.client.0.smithi107.stdout:checking for ANSI C header files... (cached) yes 2024-05-09T08:34:54.350 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-05-09T08:34:54.408 INFO:tasks.workunit.client.0.smithi107.stdout:checking pthread.h usability... yes 2024-05-09T08:34:54.435 INFO:tasks.workunit.client.0.smithi107.stdout:checking pthread.h presence... yes 2024-05-09T08:34:54.435 INFO:tasks.workunit.client.0.smithi107.stdout:checking for pthread.h... yes 2024-05-09T08:34:54.504 INFO:tasks.workunit.client.0.smithi107.stdout:checking fcntl.h usability... yes 2024-05-09T08:34:54.528 INFO:tasks.workunit.client.0.smithi107.stdout:checking fcntl.h presence... yes 2024-05-09T08:34:54.528 INFO:tasks.workunit.client.0.smithi107.stdout:checking for fcntl.h... yes 2024-05-09T08:34:54.594 INFO:tasks.workunit.client.0.smithi107.stdout:checking limits.h usability... yes 2024-05-09T08:34:54.617 INFO:tasks.workunit.client.0.smithi107.stdout:checking limits.h presence... yes 2024-05-09T08:34:54.617 INFO:tasks.workunit.client.0.smithi107.stdout:checking for limits.h... yes 2024-05-09T08:34:54.638 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdint.h... (cached) yes 2024-05-09T08:34:54.695 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/time.h usability... yes 2024-05-09T08:34:54.717 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/time.h presence... yes 2024-05-09T08:34:54.718 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/time.h... yes 2024-05-09T08:34:54.740 INFO:tasks.workunit.client.0.smithi107.stdout:checking for unistd.h... (cached) yes 2024-05-09T08:34:54.792 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/vfs.h usability... yes 2024-05-09T08:34:54.815 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/vfs.h presence... yes 2024-05-09T08:34:54.815 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/vfs.h... yes 2024-05-09T08:34:54.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:54 smithi157 ceph-mon[33614]: pgmap v49: 129 pgs: 129 active+clean; 7.1 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 601 KiB/s wr, 213 op/s 2024-05-09T08:34:54.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:54 smithi157 ceph-mon[33614]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-05-09T08:34:54.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:54 smithi110 ceph-mon[33622]: pgmap v49: 129 pgs: 129 active+clean; 7.1 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 601 KiB/s wr, 213 op/s 2024-05-09T08:34:54.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:54 smithi110 ceph-mon[33622]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-05-09T08:34:54.890 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/limits.h usability... no 2024-05-09T08:34:54.939 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/limits.h presence... no 2024-05-09T08:34:54.939 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/limits.h... no 2024-05-09T08:34:54.996 INFO:tasks.workunit.client.0.smithi107.stdout:checking for an ANSI C-conforming const... yes 2024-05-09T08:34:55.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:54 smithi107 ceph-mon[30122]: pgmap v49: 129 pgs: 129 active+clean; 7.1 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 601 KiB/s wr, 213 op/s 2024-05-09T08:34:55.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:54 smithi107 ceph-mon[30122]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-05-09T08:34:55.038 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-05-09T08:34:55.132 INFO:tasks.workunit.client.0.smithi107.stdout:checking for system... yes 2024-05-09T08:34:55.216 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gettimeofday... yes 2024-05-09T08:34:55.294 INFO:tasks.workunit.client.0.smithi107.stdout:checking for mkdir... yes 2024-05-09T08:34:55.371 INFO:tasks.workunit.client.0.smithi107.stdout:checking for strerror... yes 2024-05-09T08:34:55.450 INFO:tasks.workunit.client.0.smithi107.stdout:checking for open64... yes 2024-05-09T08:34:55.530 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stat64... yes 2024-05-09T08:34:55.608 INFO:tasks.workunit.client.0.smithi107.stdout:checking for fseeko64... yes 2024-05-09T08:34:55.693 INFO:tasks.workunit.client.0.smithi107.stdout:checking for lrand48_r... yes 2024-05-09T08:34:55.776 INFO:tasks.workunit.client.0.smithi107.stdout:checking for srand48_r... yes 2024-05-09T08:34:55.804 INFO:tasks.workunit.client.0.smithi107.stdout:configure: creating ./config.status 2024-05-09T08:34:55.907 INFO:tasks.workunit.client.0.smithi107.stdout: /bin/sh ./config.status 2024-05-09T08:34:56.006 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: creating Makefile 2024-05-09T08:34:56.023 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: creating config.h 2024-05-09T08:34:56.057 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: executing depfiles commands 2024-05-09T08:34:56.076 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: executing default commands 2024-05-09T08:34:56.138 INFO:tasks.workunit.client.0.smithi107.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-05-09T08:34:56.304 INFO:tasks.workunit.client.0.smithi107.stdout:rm -f stamp-h1 2024-05-09T08:34:56.307 INFO:tasks.workunit.client.0.smithi107.stdout:touch config.h.in 2024-05-09T08:34:56.309 INFO:tasks.workunit.client.0.smithi107.stdout:cd . && /bin/sh ./config.status config.h 2024-05-09T08:34:56.363 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: creating config.h 2024-05-09T08:34:56.402 INFO:tasks.workunit.client.0.smithi107.stdout:make all-am 2024-05-09T08:34:56.426 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-05-09T08:34:56.426 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-05-09T08:34:56.602 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-05-09T08:34:56.608 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-05-09T08:34:56.715 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-05-09T08:34:56.722 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-05-09T08:34:56.825 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-05-09T08:34:56.830 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-05-09T08:34:56.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:56 smithi157 ceph-mon[33614]: pgmap v50: 129 pgs: 129 active+clean; 7.1 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 601 KiB/s wr, 210 op/s 2024-05-09T08:34:56.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:56 smithi110 ceph-mon[33622]: pgmap v50: 129 pgs: 129 active+clean; 7.1 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 601 KiB/s wr, 210 op/s 2024-05-09T08:34:56.953 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-05-09T08:34:57.189 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-05-09T08:34:57.189 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:56 smithi107 ceph-mon[30122]: pgmap v50: 129 pgs: 129 active+clean; 7.1 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 601 KiB/s wr, 210 op/s 2024-05-09T08:34:57.190 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-05-09T08:34:57.190 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-05-09T08:34:57.190 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-05-09T08:34:57.190 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-05-09T08:34:57.233 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-05-09T08:34:57.239 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-05-09T08:34:57.302 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-05-09T08:34:57.307 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-05-09T08:34:57.551 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-05-09T08:34:57.557 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-05-09T08:34:57.725 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-05-09T08:34:57.731 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-05-09T08:34:57.907 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-05-09T08:34:57.913 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-05-09T08:34:58.013 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-05-09T08:34:58.020 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-05-09T08:34:58.111 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-05-09T08:34:58.119 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-05-09T08:34:58.254 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-05-09T08:34:58.261 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-05-09T08:34:58.304 INFO:tasks.workunit.client.0.smithi107.stderr:In file included from parser.h:22, 2024-05-09T08:34:58.304 INFO:tasks.workunit.client.0.smithi107.stderr: from parser.c:25: 2024-05-09T08:34:58.304 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c: In function ‘init_filesys’: 2024-05-09T08:34:58.306 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-05-09T08:34:58.306 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-05-09T08:34:58.307 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2024-05-09T08:34:58.307 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-05-09T08:34:58.307 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-05-09T08:34:58.307 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2024-05-09T08:34:58.307 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-05-09T08:34:58.307 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2024-05-09T08:34:58.307 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2024-05-09T08:34:58.307 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-05-09T08:34:58.307 INFO:tasks.workunit.client.0.smithi107.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-05-09T08:34:58.307 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2024-05-09T08:34:58.307 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-05-09T08:34:58.307 INFO:tasks.workunit.client.0.smithi107.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-05-09T08:34:58.307 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T08:34:58.308 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-05-09T08:34:58.308 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-05-09T08:34:58.308 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2024-05-09T08:34:58.308 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-05-09T08:34:58.308 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-05-09T08:34:58.308 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2024-05-09T08:34:58.308 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-05-09T08:34:58.308 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2024-05-09T08:34:58.308 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2024-05-09T08:34:58.308 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-05-09T08:34:58.308 INFO:tasks.workunit.client.0.smithi107.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-05-09T08:34:58.308 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2024-05-09T08:34:58.308 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-05-09T08:34:58.308 INFO:tasks.workunit.client.0.smithi107.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-05-09T08:34:58.309 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T08:34:58.309 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-05-09T08:34:58.309 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-05-09T08:34:58.309 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2024-05-09T08:34:58.309 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-05-09T08:34:58.309 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-05-09T08:34:58.309 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2024-05-09T08:34:58.309 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-05-09T08:34:58.309 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2024-05-09T08:34:58.309 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2024-05-09T08:34:58.309 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-05-09T08:34:58.309 INFO:tasks.workunit.client.0.smithi107.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-05-09T08:34:58.309 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2024-05-09T08:34:58.309 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-05-09T08:34:58.310 INFO:tasks.workunit.client.0.smithi107.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-05-09T08:34:58.310 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T08:34:58.310 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-05-09T08:34:58.310 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-05-09T08:34:58.310 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2024-05-09T08:34:58.310 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-05-09T08:34:58.310 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-05-09T08:34:58.310 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2024-05-09T08:34:58.310 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-05-09T08:34:58.310 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2024-05-09T08:34:58.310 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2024-05-09T08:34:58.310 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-05-09T08:34:58.310 INFO:tasks.workunit.client.0.smithi107.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-05-09T08:34:58.311 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2024-05-09T08:34:58.311 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-05-09T08:34:58.311 INFO:tasks.workunit.client.0.smithi107.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-05-09T08:34:58.311 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T08:34:58.311 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c: In function ‘init_tg_stats’: 2024-05-09T08:34:58.311 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-05-09T08:34:58.311 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-05-09T08:34:58.311 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2024-05-09T08:34:58.311 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-05-09T08:34:58.311 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-05-09T08:34:58.311 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2024-05-09T08:34:58.311 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-05-09T08:34:58.311 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2024-05-09T08:34:58.312 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2024-05-09T08:34:58.312 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-05-09T08:34:58.312 INFO:tasks.workunit.client.0.smithi107.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-05-09T08:34:58.312 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2024-05-09T08:34:58.312 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-05-09T08:34:58.312 INFO:tasks.workunit.client.0.smithi107.stderr: 971 | list_for_each_entry(tmp_list, 2024-05-09T08:34:58.312 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T08:34:58.312 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-05-09T08:34:58.312 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-05-09T08:34:58.312 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2024-05-09T08:34:58.312 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-05-09T08:34:58.312 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-05-09T08:34:58.312 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2024-05-09T08:34:58.313 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-05-09T08:34:58.313 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2024-05-09T08:34:58.313 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2024-05-09T08:34:58.313 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-05-09T08:34:58.313 INFO:tasks.workunit.client.0.smithi107.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-05-09T08:34:58.313 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2024-05-09T08:34:58.313 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-05-09T08:34:58.313 INFO:tasks.workunit.client.0.smithi107.stderr: 971 | list_for_each_entry(tmp_list, 2024-05-09T08:34:58.313 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T08:34:58.313 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-05-09T08:34:58.313 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-05-09T08:34:58.313 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2024-05-09T08:34:58.313 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-05-09T08:34:58.313 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-05-09T08:34:58.314 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2024-05-09T08:34:58.314 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-05-09T08:34:58.314 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2024-05-09T08:34:58.314 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2024-05-09T08:34:58.314 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-05-09T08:34:58.314 INFO:tasks.workunit.client.0.smithi107.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-05-09T08:34:58.314 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2024-05-09T08:34:58.314 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-05-09T08:34:58.314 INFO:tasks.workunit.client.0.smithi107.stderr: 980 | list_for_each_entry(tmp_list, 2024-05-09T08:34:58.314 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T08:34:58.314 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-05-09T08:34:58.314 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-05-09T08:34:58.314 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2024-05-09T08:34:58.315 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-05-09T08:34:58.316 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-05-09T08:34:58.316 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2024-05-09T08:34:58.317 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-05-09T08:34:58.317 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2024-05-09T08:34:58.317 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2024-05-09T08:34:58.317 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-05-09T08:34:58.317 INFO:tasks.workunit.client.0.smithi107.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-05-09T08:34:58.317 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2024-05-09T08:34:58.317 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-05-09T08:34:58.317 INFO:tasks.workunit.client.0.smithi107.stderr: 980 | list_for_each_entry(tmp_list, 2024-05-09T08:34:58.317 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T08:34:58.353 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c: In function ‘get_optstr’: 2024-05-09T08:34:58.353 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-05-09T08:34:58.353 INFO:tasks.workunit.client.0.smithi107.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-05-09T08:34:58.353 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T08:34:58.353 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:200:14: note: source object declared here 2024-05-09T08:34:58.353 INFO:tasks.workunit.client.0.smithi107.stderr: 200 | char temp[BUFSIZE]; 2024-05-09T08:34:58.354 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~ 2024-05-09T08:34:58.359 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c: In function ‘size64_convert’: 2024-05-09T08:34:58.359 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-05-09T08:34:58.359 INFO:tasks.workunit.client.0.smithi107.stderr: 119 | memcpy(unit, "\0", 3); 2024-05-09T08:34:58.359 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-05-09T08:34:58.654 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-05-09T08:34:58.659 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-05-09T08:34:58.734 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-05-09T08:34:58.740 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-05-09T08:34:58.842 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-05-09T08:34:58.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:34:58 smithi157 ceph-mon[33614]: pgmap v51: 129 pgs: 129 active+clean; 9.8 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 885 KiB/s wr, 279 op/s 2024-05-09T08:34:58.849 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-05-09T08:34:58.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:34:58 smithi110 ceph-mon[33622]: pgmap v51: 129 pgs: 129 active+clean; 9.8 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 885 KiB/s wr, 279 op/s 2024-05-09T08:34:58.888 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-05-09T08:34:58.895 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-05-09T08:34:58.991 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-05-09T08:34:58.992 INFO:tasks.workunit.client.0.smithi107.stderr:+ cd .. 2024-05-09T08:34:58.993 INFO:tasks.workunit.client.0.smithi107.stderr:+ mkdir tmp 2024-05-09T08:34:58.996 INFO:tasks.workunit.client.0.smithi107.stderr:+ cd tmp 2024-05-09T08:34:58.997 INFO:tasks.workunit.client.0.smithi107.stderr:+ for f in $mydir/*.ffsb 2024-05-09T08:34:58.997 INFO:tasks.workunit.client.0.smithi107.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-05-09T08:34:58.998 INFO:tasks.workunit.client.0.smithi107.stdout:FFSB version 6.0-RC2 started 2024-05-09T08:34:58.998 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:34:59.001 INFO:tasks.workunit.client.0.smithi107.stdout:benchmark time = 300 2024-05-09T08:34:59.001 INFO:tasks.workunit.client.0.smithi107.stdout:ThreadGroup 0 2024-05-09T08:34:59.001 INFO:tasks.workunit.client.0.smithi107.stdout:================ 2024-05-09T08:34:59.002 INFO:tasks.workunit.client.0.smithi107.stdout: num_threads = 32 2024-05-09T08:34:59.002 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:34:59.002 INFO:tasks.workunit.client.0.smithi107.stdout: read_random = off 2024-05-09T08:34:59.002 INFO:tasks.workunit.client.0.smithi107.stdout: read_size = 0 (0B) 2024-05-09T08:34:59.002 INFO:tasks.workunit.client.0.smithi107.stdout: read_blocksize = 0 (0B) 2024-05-09T08:34:59.002 INFO:tasks.workunit.client.0.smithi107.stdout: read_skip = off 2024-05-09T08:34:59.002 INFO:tasks.workunit.client.0.smithi107.stdout: read_skipsize = 0 (0B) 2024-05-09T08:34:59.002 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:34:59.002 INFO:tasks.workunit.client.0.smithi107.stdout: write_random = on 2024-05-09T08:34:59.002 INFO:tasks.workunit.client.0.smithi107.stdout: write_size = 5242880 (5MB) 2024-05-09T08:34:59.002 INFO:tasks.workunit.client.0.smithi107.stdout: fsync_file = 0 2024-05-09T08:34:59.002 INFO:tasks.workunit.client.0.smithi107.stdout: write_blocksize = 4096 (4KB) 2024-05-09T08:34:59.002 INFO:tasks.workunit.client.0.smithi107.stdout: wait time = 0 2024-05-09T08:34:59.002 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:34:59.003 INFO:tasks.workunit.client.0.smithi107.stdout: op weights 2024-05-09T08:34:59.003 INFO:tasks.workunit.client.0.smithi107.stdout: read = 0 (0.00%) 2024-05-09T08:34:59.003 INFO:tasks.workunit.client.0.smithi107.stdout: readall = 0 (0.00%) 2024-05-09T08:34:59.003 INFO:tasks.workunit.client.0.smithi107.stdout: write = 1 (100.00%) 2024-05-09T08:34:59.003 INFO:tasks.workunit.client.0.smithi107.stdout: create = 0 (0.00%) 2024-05-09T08:34:59.003 INFO:tasks.workunit.client.0.smithi107.stdout: append = 0 (0.00%) 2024-05-09T08:34:59.003 INFO:tasks.workunit.client.0.smithi107.stdout: delete = 0 (0.00%) 2024-05-09T08:34:59.003 INFO:tasks.workunit.client.0.smithi107.stdout: metaop = 0 (0.00%) 2024-05-09T08:34:59.003 INFO:tasks.workunit.client.0.smithi107.stdout: createdir = 0 (0.00%) 2024-05-09T08:34:59.003 INFO:tasks.workunit.client.0.smithi107.stdout: stat = 0 (0.00%) 2024-05-09T08:34:59.003 INFO:tasks.workunit.client.0.smithi107.stdout: writeall = 0 (0.00%) 2024-05-09T08:34:59.003 INFO:tasks.workunit.client.0.smithi107.stdout: writeall_fsync = 0 (0.00%) 2024-05-09T08:34:59.003 INFO:tasks.workunit.client.0.smithi107.stdout: open_close = 0 (0.00%) 2024-05-09T08:34:59.003 INFO:tasks.workunit.client.0.smithi107.stdout: write_fsync = 0 (0.00%) 2024-05-09T08:34:59.003 INFO:tasks.workunit.client.0.smithi107.stdout: create_fsync = 0 (0.00%) 2024-05-09T08:34:59.004 INFO:tasks.workunit.client.0.smithi107.stdout: append_fsync = 0 (0.00%) 2024-05-09T08:34:59.004 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:34:59.004 INFO:tasks.workunit.client.0.smithi107.stdout:FileSystem . 2024-05-09T08:34:59.004 INFO:tasks.workunit.client.0.smithi107.stdout:========== 2024-05-09T08:34:59.004 INFO:tasks.workunit.client.0.smithi107.stdout: num_dirs = 0 2024-05-09T08:34:59.004 INFO:tasks.workunit.client.0.smithi107.stdout: starting files = 128 2024-05-09T08:34:59.004 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:34:59.004 INFO:tasks.workunit.client.0.smithi107.stdout: min file size = 104857600 (100MB) 2024-05-09T08:34:59.004 INFO:tasks.workunit.client.0.smithi107.stdout: max file size = 104857600 (100MB) 2024-05-09T08:34:59.004 INFO:tasks.workunit.client.0.smithi107.stdout: directio = off 2024-05-09T08:34:59.004 INFO:tasks.workunit.client.0.smithi107.stdout: alignedio = on 2024-05-09T08:34:59.004 INFO:tasks.workunit.client.0.smithi107.stdout: bufferedio = off 2024-05-09T08:34:59.004 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:34:59.004 INFO:tasks.workunit.client.0.smithi107.stdout: aging is off 2024-05-09T08:34:59.004 INFO:tasks.workunit.client.0.smithi107.stdout: current utilization = 0.04% 2024-05-09T08:34:59.005 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:34:59.005 INFO:tasks.workunit.client.0.smithi107.stdout:checking existing fs: . 2024-05-09T08:34:59.005 INFO:tasks.workunit.client.0.smithi107.stdout:recreating new fileset 2024-05-09T08:34:59.005 INFO:tasks.workunit.client.0.smithi107.stderr:opendir: No such file or directory 2024-05-09T08:34:59.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:34:58 smithi107 ceph-mon[30122]: pgmap v51: 129 pgs: 129 active+clean; 9.8 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 885 KiB/s wr, 279 op/s 2024-05-09T08:35:00.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:00 smithi157 ceph-mon[33614]: pgmap v52: 129 pgs: 129 active+clean; 9.8 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 609 KiB/s wr, 238 op/s 2024-05-09T08:35:00.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:00 smithi110 ceph-mon[33622]: pgmap v52: 129 pgs: 129 active+clean; 9.8 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 609 KiB/s wr, 238 op/s 2024-05-09T08:35:01.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:00 smithi107 ceph-mon[30122]: pgmap v52: 129 pgs: 129 active+clean; 9.8 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 609 KiB/s wr, 238 op/s 2024-05-09T08:35:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.412+0000 7fce156ba640 1 -- 172.21.15.107:0/767045253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce101033d0 msgr2=0x7fce1010b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.412+0000 7fce156ba640 1 --2- 172.21.15.107:0/767045253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce101033d0 0x7fce1010b8b0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fce04009900 tx=0x7fce0402a480 comp rx=0 tx=0).stop 2024-05-09T08:35:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.412+0000 7fce156ba640 1 -- 172.21.15.107:0/767045253 shutdown_connections 2024-05-09T08:35:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.412+0000 7fce156ba640 1 --2- 172.21.15.107:0/767045253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce1010be80 0x7fce1010e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.412+0000 7fce156ba640 1 --2- 172.21.15.107:0/767045253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce101033d0 0x7fce1010b8b0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.412+0000 7fce156ba640 1 --2- 172.21.15.107:0/767045253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce10102ab0 0x7fce10102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.412+0000 7fce156ba640 1 -- 172.21.15.107:0/767045253 >> 172.21.15.107:0/767045253 conn(0x7fce10077880 msgr2=0x7fce10077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.412+0000 7fce156ba640 1 -- 172.21.15.107:0/767045253 shutdown_connections 2024-05-09T08:35:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.412+0000 7fce156ba640 1 -- 172.21.15.107:0/767045253 wait complete. 2024-05-09T08:35:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.413+0000 7fce156ba640 1 Processor -- start 2024-05-09T08:35:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.413+0000 7fce156ba640 1 -- start start 2024-05-09T08:35:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.413+0000 7fce156ba640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce10102ab0 0x7fce10100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.413+0000 7fce156ba640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce101033d0 0x7fce10100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.413+0000 7fce156ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1010be80 0x7fce10101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce156ba640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fce10111ac0 con 0x7fce1010be80 2024-05-09T08:35:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce156ba640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fce10111940 con 0x7fce10102ab0 2024-05-09T08:35:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce156ba640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fce10111c40 con 0x7fce101033d0 2024-05-09T08:35:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce0f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1010be80 0x7fce10101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce0f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1010be80 0x7fce10101180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53688/0 (socket says 172.21.15.107:53688) 2024-05-09T08:35:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce0f7fe640 1 -- 172.21.15.107:0/3668689320 learned_addr learned my addr 172.21.15.107:0/3668689320 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce0effd640 1 --2- 172.21.15.107:0/3668689320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce10102ab0 0x7fce10100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/3668689320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce101033d0 0x7fce10100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce0f7fe640 1 -- 172.21.15.107:0/3668689320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce101033d0 msgr2=0x7fce10100c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce0f7fe640 1 --2- 172.21.15.107:0/3668689320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce101033d0 0x7fce10100c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce0f7fe640 1 -- 172.21.15.107:0/3668689320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce10102ab0 msgr2=0x7fce10100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce0f7fe640 1 --2- 172.21.15.107:0/3668689320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce10102ab0 0x7fce10100700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce0f7fe640 1 -- 172.21.15.107:0/3668689320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce101a64a0 con 0x7fce1010be80 2024-05-09T08:35:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/3668689320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce101033d0 0x7fce10100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce0f7fe640 1 --2- 172.21.15.107:0/3668689320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1010be80 0x7fce10101180 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fce0800eaa0 tx=0x7fce0800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fcdf7fff640 1 -- 172.21.15.107:0/3668689320 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0800cdb0 con 0x7fce1010be80 2024-05-09T08:35:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fcdf7fff640 1 -- 172.21.15.107:0/3668689320 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce08005020 con 0x7fce1010be80 2024-05-09T08:35:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce156ba640 1 -- 172.21.15.107:0/3668689320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce101a6790 con 0x7fce1010be80 2024-05-09T08:35:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fcdf7fff640 1 -- 172.21.15.107:0/3668689320 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce080116d0 con 0x7fce1010be80 2024-05-09T08:35:02.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.414+0000 7fce156ba640 1 -- 172.21.15.107:0/3668689320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce101a6cb0 con 0x7fce1010be80 2024-05-09T08:35:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.417+0000 7fce183c8740 1 Processor -- start 2024-05-09T08:35:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.417+0000 7fcdf7fff640 1 -- 172.21.15.107:0/3668689320 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fce080051d0 con 0x7fce1010be80 2024-05-09T08:35:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.417+0000 7fce183c8740 1 -- start start 2024-05-09T08:35:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.417+0000 7fce183c8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 0x55ce3bfe6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.417+0000 7fce183c8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 0x55ce3bff10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.417+0000 7fce183c8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 0x55ce3bff3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.417+0000 7fce183c8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ce3bf4b340 con 0x55ce3bff1630 2024-05-09T08:35:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.417+0000 7fce183c8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ce3bef3290 con 0x55ce3bfe6d70 2024-05-09T08:35:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.417+0000 7fce183c8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ce3bf47300 con 0x55ce3bfe6450 2024-05-09T08:35:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fce0e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 0x55ce3bff3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fce0e7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 0x55ce3bfe6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fce0e7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 0x55ce3bfe6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:58478/0 (socket says 172.21.15.107:58478) 2024-05-09T08:35:02.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fce0e7fc640 1 -- 172.21.15.107:0/1799903128 learned_addr learned my addr 172.21.15.107:0/1799903128 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:02.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fce0effd640 1 --2- 172.21.15.107:0/1799903128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 0x55ce3bff10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fcdf7fff640 1 --2- 172.21.15.107:0/3668689320 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcdf80a2e50 0x7fcdf80a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fce0e7fc640 1 -- 172.21.15.107:0/1799903128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 msgr2=0x55ce3bff10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:02.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/1799903128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 0x55ce3bff10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fce0e7fc640 1 -- 172.21.15.107:0/1799903128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 msgr2=0x55ce3bff3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:02.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/1799903128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 0x55ce3bff3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fce0e7fc640 1 -- 172.21.15.107:0/1799903128 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce3bf178e0 con 0x55ce3bfe6450 2024-05-09T08:35:02.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fce0effd640 1 --2- 172.21.15.107:0/1799903128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 0x55ce3bff10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:02.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fce0f7fe640 1 --2- 172.21.15.107:0/3668689320 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcdf80a2e50 0x7fcdf80a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fcdf7fff640 1 -- 172.21.15.107:0/3668689320 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fce080cf8a0 con 0x7fce1010be80 2024-05-09T08:35:02.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/1799903128 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 0x55ce3bfe6830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55ce3bfe3af0 tx=0x7fce04009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5cec333bbab853f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:02.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fcdf57fa640 1 -- 172.21.15.107:0/1799903128 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04035040 con 0x55ce3bfe6450 2024-05-09T08:35:02.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fcdf57fa640 1 -- 172.21.15.107:0/1799903128 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce040035f0 con 0x55ce3bfe6450 2024-05-09T08:35:02.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.418+0000 7fcdf57fa640 1 -- 172.21.15.107:0/1799903128 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce040044e0 con 0x55ce3bfe6450 2024-05-09T08:35:02.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.419+0000 7fce0f7fe640 1 --2- 172.21.15.107:0/3668689320 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcdf80a2e50 0x7fcdf80a5310 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fce080a41a0 tx=0x7fce08009270 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:02.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.419+0000 7fce183c8740 1 -- 172.21.15.107:0/1799903128 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 msgr2=0x55ce3bfe6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:02.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.419+0000 7fce183c8740 1 --2- 172.21.15.107:0/1799903128 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 0x55ce3bfe6830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55ce3bfe3af0 tx=0x7fce04009260 comp rx=0 tx=0).stop 2024-05-09T08:35:02.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.419+0000 7fce183c8740 1 -- 172.21.15.107:0/1799903128 shutdown_connections 2024-05-09T08:35:02.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.419+0000 7fce183c8740 1 --2- 172.21.15.107:0/1799903128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 0x55ce3bff3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.419+0000 7fce183c8740 1 --2- 172.21.15.107:0/1799903128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 0x55ce3bff10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.419+0000 7fce183c8740 1 --2- 172.21.15.107:0/1799903128 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 0x55ce3bfe6830 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.419+0000 7fce183c8740 1 -- 172.21.15.107:0/1799903128 >> 172.21.15.107:0/1799903128 conn(0x55ce3bfe2960 msgr2=0x55ce3bfe2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.419+0000 7fce183c8740 1 -- 172.21.15.107:0/1799903128 shutdown_connections 2024-05-09T08:35:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.419+0000 7fce183c8740 1 -- 172.21.15.107:0/1799903128 wait complete. 2024-05-09T08:35:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce183c8740 1 -- start start 2024-05-09T08:35:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce183c8740 1 Processor -- start 2024-05-09T08:35:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce183c8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bfe6450 0x55ce3bfee5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce183c8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6d70 0x55ce3bfeeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce183c8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bff1630 0x55ce3bfef040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce183c8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ce3bff4200 con 0x55ce3bfe6450 2024-05-09T08:35:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce183c8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ce3bf06790 con 0x55ce3bff1630 2024-05-09T08:35:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce183c8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ce3bff4380 con 0x55ce3bfe6d70 2024-05-09T08:35:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce183c8740 10 client.0 _ll_register_callbacks cb 0x55ce3bf41a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce0e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bfe6450 0x55ce3bfee5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce0e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bfe6450 0x55ce3bfee5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53714/0 (socket says 172.21.15.107:53714) 2024-05-09T08:35:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce0effd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6d70 0x55ce3bfeeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce0e7fc640 1 -- 172.21.15.107:0/2998391646 learned_addr learned my addr 172.21.15.107:0/2998391646 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/2998391646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bff1630 0x55ce3bfef040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce0effd640 1 -- 172.21.15.107:0/2998391646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bff1630 msgr2=0x55ce3bfef040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce0effd640 1 --2- 172.21.15.107:0/2998391646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bff1630 0x55ce3bfef040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce0effd640 1 -- 172.21.15.107:0/2998391646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bfe6450 msgr2=0x55ce3bfee5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce0effd640 1 --2- 172.21.15.107:0/2998391646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bfe6450 0x55ce3bfee5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce0effd640 1 -- 172.21.15.107:0/2998391646 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce3bc5f210 con 0x55ce3bfe6d70 2024-05-09T08:35:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.420+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/2998391646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bff1630 0x55ce3bfef040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.421+0000 7fce0effd640 1 --2- 172.21.15.107:0/2998391646 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6d70 0x55ce3bfeeb00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fce0000baa0 tx=0x7fce0000bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8ab1f5952855b85 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.421+0000 7fcdf2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:35:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.421+0000 7fcdf2ff5640 1 -- 172.21.15.107:0/2998391646 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce00003df0 con 0x55ce3bfe6d70 2024-05-09T08:35:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.421+0000 7fcdf2ff5640 1 -- 172.21.15.107:0/2998391646 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fce00005020 con 0x55ce3bfe6d70 2024-05-09T08:35:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.421+0000 7fce183c8740 1 -- 172.21.15.107:0/2998391646 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce040035f0 con 0x55ce3bfe6d70 2024-05-09T08:35:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.421+0000 7fcdf2ff5640 1 -- 172.21.15.107:0/2998391646 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fce00024050 con 0x55ce3bfe6d70 2024-05-09T08:35:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.421+0000 7fce183c8740 10 client.25594 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.421+0000 7fce183c8740 1 -- 172.21.15.107:0/2998391646 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ce3bd2b620 con 0x55ce3bfe6d70 2024-05-09T08:35:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.421+0000 7fcdf2ff5640 1 -- 172.21.15.107:0/2998391646 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce040035f0 con 0x55ce3bfe6d70 2024-05-09T08:35:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.421+0000 7fcdf2ff5640 1 -- 172.21.15.107:0/2998391646 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fce0001b470 con 0x55ce3bfe6d70 2024-05-09T08:35:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.421+0000 7fce183c8740 10 client.25594 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.421+0000 7fce183c8740 10 client.25594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.421+0000 7fce183c8740 20 client.25594 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.421+0000 7fce183c8740 1 --2- 172.21.15.107:0/2998391646 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ce3c0013c0 0x55ce3c0037f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.421+0000 7fce183c8740 4 client.25594 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.421+0000 7fce183c8740 1 -- 172.21.15.107:0/2998391646 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce3bf58a70 con 0x55ce3c0013c0 2024-05-09T08:35:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.422+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/2998391646 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ce3c0013c0 0x55ce3c0037f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.422+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/2998391646 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ce3c0013c0 0x55ce3c0037f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c01d492284ed8a4 server_cookie=4e798f14635c3085 in_seq=0 out_seq=0 2024-05-09T08:35:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.422+0000 7fcdf2ff5640 10 client.25594 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.423+0000 7fcdf2ff5640 1 -- 172.21.15.107:0/2998391646 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ce3bf58a70 con 0x55ce3c0013c0 2024-05-09T08:35:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.423+0000 7fcdf2ff5640 10 client.25594 handle_command_reply: tid=0 2024-05-09T08:35:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.423+0000 7fce183c8740 1 client.25594 shutdown 2024-05-09T08:35:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.423+0000 7fce183c8740 1 -- 172.21.15.107:0/2998391646 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6d70 msgr2=0x55ce3bfeeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.423+0000 7fce183c8740 1 --2- 172.21.15.107:0/2998391646 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6d70 0x55ce3bfeeb00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fce0000baa0 tx=0x7fce0000bf60 comp rx=0 tx=0).stop 2024-05-09T08:35:02.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.423+0000 7fce183c8740 1 -- 172.21.15.107:0/2998391646 shutdown_connections 2024-05-09T08:35:02.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.423+0000 7fce183c8740 1 --2- 172.21.15.107:0/2998391646 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ce3c0013c0 0x55ce3c0037f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.423+0000 7fce183c8740 1 --2- 172.21.15.107:0/2998391646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bff1630 0x55ce3bfef040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.423+0000 7fce183c8740 1 --2- 172.21.15.107:0/2998391646 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6d70 0x55ce3bfeeb00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.423+0000 7fcdf2ff5640 0 client.25594 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:02.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.423+0000 7fce183c8740 1 --2- 172.21.15.107:0/2998391646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bfe6450 0x55ce3bfee5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.423+0000 7fce183c8740 1 -- 172.21.15.107:0/2998391646 >> 172.21.15.107:0/2998391646 conn(0x55ce3bfe2960 msgr2=0x55ce3bfe8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:02.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.423+0000 7fce183c8740 1 -- 172.21.15.107:0/2998391646 shutdown_connections 2024-05-09T08:35:02.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.423+0000 7fce183c8740 1 -- 172.21.15.107:0/2998391646 wait complete. 2024-05-09T08:35:02.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.423+0000 7fce183c8740 20 client.25594 trim_cache size 0 max 16384 2024-05-09T08:35:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.439+0000 7fce183c8740 1 Processor -- start 2024-05-09T08:35:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.439+0000 7fce183c8740 1 -- start start 2024-05-09T08:35:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.439+0000 7fce183c8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 0x55ce3bffd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.439+0000 7fce183c8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 0x55ce3bff03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.439+0000 7fce183c8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 0x55ce3c0090c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.439+0000 7fce183c8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fce101b34c0 con 0x55ce3bff1630 2024-05-09T08:35:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.439+0000 7fce183c8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fce101b0d90 con 0x55ce3bfe6d70 2024-05-09T08:35:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.439+0000 7fce183c8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fce101aebe0 con 0x55ce3bfe6450 2024-05-09T08:35:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.439+0000 7fce0e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 0x55ce3c0090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.439+0000 7fce0e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 0x55ce3c0090c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53720/0 (socket says 172.21.15.107:53720) 2024-05-09T08:35:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.439+0000 7fce0e7fc640 1 -- 172.21.15.107:0/3417665414 learned_addr learned my addr 172.21.15.107:0/3417665414 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.439+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/3417665414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 0x55ce3bffd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.440+0000 7fce0effd640 1 --2- 172.21.15.107:0/3417665414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 0x55ce3bff03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.440+0000 7fce0e7fc640 1 -- 172.21.15.107:0/3417665414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 msgr2=0x55ce3bffd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.440+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/3417665414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 0x55ce3bffd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.440+0000 7fce0e7fc640 1 -- 172.21.15.107:0/3417665414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 msgr2=0x55ce3bff03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.440+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/3417665414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 0x55ce3bff03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.440+0000 7fce0e7fc640 1 -- 172.21.15.107:0/3417665414 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce3bfe0f00 con 0x55ce3bff1630 2024-05-09T08:35:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.440+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/3417665414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 0x55ce3c0090c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fce04006010 tx=0x7fce0404c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efcff7683dbb842f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.440+0000 7fcdf4ff9640 1 -- 172.21.15.107:0/3417665414 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04035040 con 0x55ce3bff1630 2024-05-09T08:35:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.440+0000 7fcdf4ff9640 1 -- 172.21.15.107:0/3417665414 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce04004410 con 0x55ce3bff1630 2024-05-09T08:35:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.440+0000 7fcdf4ff9640 1 -- 172.21.15.107:0/3417665414 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04054790 con 0x55ce3bff1630 2024-05-09T08:35:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.441+0000 7fce183c8740 1 -- 172.21.15.107:0/3417665414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 msgr2=0x55ce3c0090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.441+0000 7fce183c8740 1 --2- 172.21.15.107:0/3417665414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 0x55ce3c0090c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fce04006010 tx=0x7fce0404c040 comp rx=0 tx=0).stop 2024-05-09T08:35:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.441+0000 7fce183c8740 1 -- 172.21.15.107:0/3417665414 shutdown_connections 2024-05-09T08:35:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.441+0000 7fce183c8740 1 --2- 172.21.15.107:0/3417665414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 0x55ce3c0090c0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.441+0000 7fce183c8740 1 --2- 172.21.15.107:0/3417665414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 0x55ce3bff03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.441+0000 7fce183c8740 1 --2- 172.21.15.107:0/3417665414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 0x55ce3bffd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.441+0000 7fce183c8740 1 -- 172.21.15.107:0/3417665414 >> 172.21.15.107:0/3417665414 conn(0x55ce3bfe2960 msgr2=0x55ce3bfefce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.441+0000 7fce183c8740 1 -- 172.21.15.107:0/3417665414 shutdown_connections 2024-05-09T08:35:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.441+0000 7fce183c8740 1 -- 172.21.15.107:0/3417665414 wait complete. 2024-05-09T08:35:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.441+0000 7fce183c8740 1 -- start start 2024-05-09T08:35:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.441+0000 7fce183c8740 1 Processor -- start 2024-05-09T08:35:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce183c8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 0x55ce3bfeb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce183c8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 0x55ce3bfeb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce183c8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 0x55ce3bfebda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce183c8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fce101af250 con 0x55ce3bff1630 2024-05-09T08:35:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce183c8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fce101b06c0 con 0x55ce3bfe6d70 2024-05-09T08:35:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce183c8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fce101af920 con 0x55ce3bfe6450 2024-05-09T08:35:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce183c8740 10 client.0 _ll_register_callbacks cb 0x7fce00004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce0e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 0x55ce3bfebda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce0e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 0x55ce3bfebda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53734/0 (socket says 172.21.15.107:53734) 2024-05-09T08:35:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce0e7fc640 1 -- 172.21.15.107:0/2219372078 learned_addr learned my addr 172.21.15.107:0/2219372078 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce0effd640 1 --2- 172.21.15.107:0/2219372078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 0x55ce3bfeb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/2219372078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 0x55ce3bfeb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce0e7fc640 1 -- 172.21.15.107:0/2219372078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 msgr2=0x55ce3bfeb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/2219372078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 0x55ce3bfeb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce0e7fc640 1 -- 172.21.15.107:0/2219372078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 msgr2=0x55ce3bfeb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/2219372078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 0x55ce3bfeb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce0e7fc640 1 -- 172.21.15.107:0/2219372078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce3c066930 con 0x55ce3bff1630 2024-05-09T08:35:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce0effd640 1 --2- 172.21.15.107:0/2219372078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 0x55ce3bfeb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce0e7fc640 1 --2- 172.21.15.107:0/2219372078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 0x55ce3bfebda0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fce0404b040 tx=0x7fce04005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab2b3c169a4b9f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fcdf2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fcdf2ff5640 1 -- 172.21.15.107:0/2219372078 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04035040 con 0x55ce3bff1630 2024-05-09T08:35:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fcdf2ff5640 1 -- 172.21.15.107:0/2219372078 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fce04031ca0 con 0x55ce3bff1630 2024-05-09T08:35:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.442+0000 7fce183c8740 1 -- 172.21.15.107:0/2219372078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce04004410 con 0x55ce3bff1630 2024-05-09T08:35:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.443+0000 7fcdf2ff5640 1 -- 172.21.15.107:0/2219372078 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fce04005020 con 0x55ce3bff1630 2024-05-09T08:35:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.443+0000 7fce183c8740 10 client.16113 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.443+0000 7fce183c8740 1 -- 172.21.15.107:0/2219372078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ce3c067be0 con 0x55ce3bff1630 2024-05-09T08:35:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.443+0000 7fcdf2ff5640 1 -- 172.21.15.107:0/2219372078 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce04004410 con 0x55ce3bff1630 2024-05-09T08:35:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.443+0000 7fcdf2ff5640 1 -- 172.21.15.107:0/2219372078 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fce04054760 con 0x55ce3bff1630 2024-05-09T08:35:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.443+0000 7fce183c8740 10 client.16113 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.443+0000 7fce183c8740 10 client.16113 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.443+0000 7fce183c8740 20 client.16113 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.443+0000 7fce183c8740 1 --2- 172.21.15.107:0/2219372078 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ce3c067f60 0x55ce3c06a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.443+0000 7fce183c8740 4 client.16113 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:35:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.443+0000 7fce183c8740 1 -- 172.21.15.107:0/2219372078 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ce3bf52dc0 con 0x55ce3c067f60 2024-05-09T08:35:02.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.443+0000 7fce0effd640 1 --2- 172.21.15.107:0/2219372078 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ce3c067f60 0x55ce3c06a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:02.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.444+0000 7fcdf2ff5640 10 client.16113 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:02.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.444+0000 7fce0effd640 1 --2- 172.21.15.107:0/2219372078 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ce3c067f60 0x55ce3c06a380 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=334c1314c1c1fafd server_cookie=b1bfd8410d20894e in_seq=0 out_seq=0 2024-05-09T08:35:02.488 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.488+0000 7fcdf2ff5640 1 -- 172.21.15.107:0/2219372078 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ce3bf52dc0 con 0x55ce3c067f60 2024-05-09T08:35:02.488 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.488+0000 7fcdf2ff5640 10 client.16113 handle_command_reply: tid=0 2024-05-09T08:35:02.489 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.488+0000 7fce183c8740 1 client.16113 shutdown 2024-05-09T08:35:02.489 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.488+0000 7fce183c8740 1 -- 172.21.15.107:0/2219372078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 msgr2=0x55ce3bfebda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:02.489 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.488+0000 7fce183c8740 1 --2- 172.21.15.107:0/2219372078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 0x55ce3bfebda0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fce0404b040 tx=0x7fce04005e50 comp rx=0 tx=0).stop 2024-05-09T08:35:02.489 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.488+0000 7fce183c8740 1 -- 172.21.15.107:0/2219372078 shutdown_connections 2024-05-09T08:35:02.489 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.488+0000 7fce183c8740 1 --2- 172.21.15.107:0/2219372078 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ce3c067f60 0x55ce3c06a380 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.489 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.488+0000 7fce183c8740 1 --2- 172.21.15.107:0/2219372078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce3bff1630 0x55ce3bfebda0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.489 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.488+0000 7fcdf2ff5640 0 client.16113 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:02.489 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.488+0000 7fce183c8740 1 --2- 172.21.15.107:0/2219372078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce3bfe6d70 0x55ce3bfeb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.489 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.488+0000 7fce183c8740 1 --2- 172.21.15.107:0/2219372078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ce3bfe6450 0x55ce3bfeb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.490 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.488+0000 7fce183c8740 1 -- 172.21.15.107:0/2219372078 >> 172.21.15.107:0/2219372078 conn(0x55ce3bfe2960 msgr2=0x55ce3bfe8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:02.490 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.488+0000 7fce183c8740 1 -- 172.21.15.107:0/2219372078 shutdown_connections 2024-05-09T08:35:02.490 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.488+0000 7fce183c8740 1 -- 172.21.15.107:0/2219372078 wait complete. 2024-05-09T08:35:02.490 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.488+0000 7fce183c8740 20 client.16113 trim_cache size 0 max 16384 2024-05-09T08:35:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:35:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:35:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:35:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:35:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.489+0000 7fce156ba640 1 -- 172.21.15.107:0/3668689320 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcdf80a2e50 msgr2=0x7fcdf80a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.489+0000 7fce156ba640 1 --2- 172.21.15.107:0/3668689320 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcdf80a2e50 0x7fcdf80a5310 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fce080a41a0 tx=0x7fce08009270 comp rx=0 tx=0).stop 2024-05-09T08:35:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.489+0000 7fce156ba640 1 -- 172.21.15.107:0/3668689320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1010be80 msgr2=0x7fce10101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.489+0000 7fce156ba640 1 --2- 172.21.15.107:0/3668689320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1010be80 0x7fce10101180 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fce0800eaa0 tx=0x7fce0800ef60 comp rx=0 tx=0).stop 2024-05-09T08:35:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.489+0000 7fce156ba640 1 -- 172.21.15.107:0/3668689320 shutdown_connections 2024-05-09T08:35:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.489+0000 7fce156ba640 1 --2- 172.21.15.107:0/3668689320 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcdf80a2e50 0x7fcdf80a5310 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.489+0000 7fce156ba640 1 --2- 172.21.15.107:0/3668689320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1010be80 0x7fce10101180 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.489+0000 7fce156ba640 1 --2- 172.21.15.107:0/3668689320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce101033d0 0x7fce10100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.489+0000 7fce156ba640 1 --2- 172.21.15.107:0/3668689320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce10102ab0 0x7fce10100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.489+0000 7fce156ba640 1 -- 172.21.15.107:0/3668689320 >> 172.21.15.107:0/3668689320 conn(0x7fce10077880 msgr2=0x7fce100fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.489+0000 7fce156ba640 1 -- 172.21.15.107:0/3668689320 shutdown_connections 2024-05-09T08:35:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:02.489+0000 7fce156ba640 1 -- 172.21.15.107:0/3668689320 wait complete. 2024-05-09T08:35:02.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:02 smithi157 ceph-mon[33614]: pgmap v53: 129 pgs: 129 active+clean; 9.8 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 609 KiB/s wr, 238 op/s 2024-05-09T08:35:02.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:02 smithi110 ceph-mon[33622]: pgmap v53: 129 pgs: 129 active+clean; 9.8 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 609 KiB/s wr, 238 op/s 2024-05-09T08:35:03.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:02 smithi107 ceph-mon[30122]: pgmap v53: 129 pgs: 129 active+clean; 9.8 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 609 KiB/s wr, 238 op/s 2024-05-09T08:35:03.799 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:35:03.799 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:35:03.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.925+0000 7f6d1993b640 1 -- 172.21.15.107:0/1805222231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d141033d0 msgr2=0x7f6d1410b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:03.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.925+0000 7f6d1993b640 1 --2- 172.21.15.107:0/1805222231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d141033d0 0x7f6d1410b8b0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f6d08009900 tx=0x7f6d0802a510 comp rx=0 tx=0).stop 2024-05-09T08:35:03.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.925+0000 7f6d1993b640 1 -- 172.21.15.107:0/1805222231 shutdown_connections 2024-05-09T08:35:03.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.925+0000 7f6d1993b640 1 --2- 172.21.15.107:0/1805222231 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d1410bdf0 0x7f6d1410e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.925+0000 7f6d1993b640 1 --2- 172.21.15.107:0/1805222231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d141033d0 0x7f6d1410b8b0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.925+0000 7f6d1993b640 1 --2- 172.21.15.107:0/1805222231 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d14102ab0 0x7f6d14102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.925+0000 7f6d1993b640 1 -- 172.21.15.107:0/1805222231 >> 172.21.15.107:0/1805222231 conn(0x7f6d14077830 msgr2=0x7f6d14077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:03.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.925+0000 7f6d1993b640 1 -- 172.21.15.107:0/1805222231 shutdown_connections 2024-05-09T08:35:03.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.925+0000 7f6d1993b640 1 -- 172.21.15.107:0/1805222231 wait complete. 2024-05-09T08:35:03.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.926+0000 7f6d1993b640 1 Processor -- start 2024-05-09T08:35:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.926+0000 7f6d1993b640 1 -- start start 2024-05-09T08:35:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.926+0000 7f6d1993b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d14102ab0 0x7f6d14198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.926+0000 7f6d1993b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d141033d0 0x7f6d14198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.926+0000 7f6d1993b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d1410bdf0 0x7f6d1419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.926+0000 7f6d1993b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6d14111aa0 con 0x7f6d141033d0 2024-05-09T08:35:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.926+0000 7f6d1993b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6d14111920 con 0x7f6d14102ab0 2024-05-09T08:35:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.926+0000 7f6d1993b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6d14111c20 con 0x7f6d1410bdf0 2024-05-09T08:35:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.926+0000 7f6d127fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d141033d0 0x7f6d14198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.926+0000 7f6d127fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d141033d0 0x7f6d14198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53742/0 (socket says 172.21.15.107:53742) 2024-05-09T08:35:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.926+0000 7f6d127fc640 1 -- 172.21.15.107:0/475830264 learned_addr learned my addr 172.21.15.107:0/475830264 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.926+0000 7f6d137fe640 1 --2- 172.21.15.107:0/475830264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d1410bdf0 0x7f6d1419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.926+0000 7f6d12ffd640 1 --2- 172.21.15.107:0/475830264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d14102ab0 0x7f6d14198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.927+0000 7f6d127fc640 1 -- 172.21.15.107:0/475830264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d1410bdf0 msgr2=0x7f6d1419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.927+0000 7f6d127fc640 1 --2- 172.21.15.107:0/475830264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d1410bdf0 0x7f6d1419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.927+0000 7f6d127fc640 1 -- 172.21.15.107:0/475830264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d14102ab0 msgr2=0x7f6d14198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.927+0000 7f6d127fc640 1 --2- 172.21.15.107:0/475830264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d14102ab0 0x7f6d14198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.927+0000 7f6d127fc640 1 -- 172.21.15.107:0/475830264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d1419d030 con 0x7f6d141033d0 2024-05-09T08:35:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.927+0000 7f6d127fc640 1 --2- 172.21.15.107:0/475830264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d141033d0 0x7f6d14198580 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f6d08005940 tx=0x7f6d08004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.927+0000 7f6d18939640 1 -- 172.21.15.107:0/475830264 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0803f030 con 0x7f6d141033d0 2024-05-09T08:35:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.927+0000 7f6d1993b640 1 -- 172.21.15.107:0/475830264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d1419d2c0 con 0x7f6d141033d0 2024-05-09T08:35:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.927+0000 7f6d18939640 1 -- 172.21.15.107:0/475830264 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d08003aa0 con 0x7f6d141033d0 2024-05-09T08:35:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.927+0000 7f6d18939640 1 -- 172.21.15.107:0/475830264 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d08035070 con 0x7f6d141033d0 2024-05-09T08:35:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.927+0000 7f6d1993b640 1 -- 172.21.15.107:0/475830264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d141a4bb0 con 0x7f6d141033d0 2024-05-09T08:35:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.929+0000 7f6d18939640 1 -- 172.21.15.107:0/475830264 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6d08042030 con 0x7f6d141033d0 2024-05-09T08:35:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.930+0000 7f6d1c648740 1 Processor -- start 2024-05-09T08:35:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.930+0000 7f6d1c648740 1 -- start start 2024-05-09T08:35:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.930+0000 7f6d18939640 1 --2- 172.21.15.107:0/475830264 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6cfc0a2d00 0x7f6cfc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.930+0000 7f6d1c648740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0160 0x56455b9d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.930+0000 7f6d1c648740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9d0a80 0x56455b9df140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.930+0000 7f6d1c648740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9df680 0x56455b9e1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.930+0000 7f6d1c648740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56455b935d00 con 0x56455b9d0160 2024-05-09T08:35:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.930+0000 7f6d1c648740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56455b8dd4c0 con 0x56455b9df680 2024-05-09T08:35:03.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.930+0000 7f6d1c648740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56455b931cc0 con 0x56455b9d0a80 2024-05-09T08:35:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.930+0000 7f6d18939640 1 -- 172.21.15.107:0/475830264 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6d080efeb0 con 0x7f6d141033d0 2024-05-09T08:35:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.930+0000 7f6d137fe640 1 --2- 172.21.15.107:0/475830264 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6cfc0a2d00 0x7f6cfc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.930+0000 7f6d12ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0160 0x56455b9d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.931+0000 7f6d12ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0160 0x56455b9d0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53750/0 (socket says 172.21.15.107:53750) 2024-05-09T08:35:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.931+0000 7f6d127fc640 1 --2- 172.21.15.107:0/2029588334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9d0a80 0x56455b9df140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.931+0000 7f6d12ffd640 1 -- 172.21.15.107:0/2029588334 learned_addr learned my addr 172.21.15.107:0/2029588334 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.931+0000 7f6d137fe640 1 --2- 172.21.15.107:0/2029588334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9df680 0x56455b9e1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.931+0000 7f6d127fc640 1 -- 172.21.15.107:0/2029588334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9df680 msgr2=0x56455b9e1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.931+0000 7f6d127fc640 1 --2- 172.21.15.107:0/2029588334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9df680 0x56455b9e1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.931+0000 7f6d127fc640 1 -- 172.21.15.107:0/2029588334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0160 msgr2=0x56455b9d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.931+0000 7f6d127fc640 1 --2- 172.21.15.107:0/2029588334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0160 0x56455b9d0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.931+0000 7f6d127fc640 1 -- 172.21.15.107:0/2029588334 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56455b7137d0 con 0x56455b9d0a80 2024-05-09T08:35:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.931+0000 7f6d137fe640 1 --2- 172.21.15.107:0/475830264 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6cfc0a2d00 0x7f6cfc0a51c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c005fd0 tx=0x7f6d0c014b70 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.931+0000 7f6d127fc640 1 --2- 172.21.15.107:0/2029588334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9d0a80 0x56455b9df140 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6d080ed2a0 tx=0x7f6d0803b460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d943ab466e5c2db server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.931+0000 7f6cf9ffb640 1 -- 172.21.15.107:0/2029588334 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d08049040 con 0x56455b9d0a80 2024-05-09T08:35:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.931+0000 7f6cf9ffb640 1 -- 172.21.15.107:0/2029588334 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d080c4ce0 con 0x56455b9d0a80 2024-05-09T08:35:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.931+0000 7f6cf9ffb640 1 -- 172.21.15.107:0/2029588334 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d080ed520 con 0x56455b9d0a80 2024-05-09T08:35:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.932+0000 7f6d1c648740 1 -- 172.21.15.107:0/2029588334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9d0a80 msgr2=0x56455b9df140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.932+0000 7f6d1c648740 1 --2- 172.21.15.107:0/2029588334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9d0a80 0x56455b9df140 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6d080ed2a0 tx=0x7f6d0803b460 comp rx=0 tx=0).stop 2024-05-09T08:35:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.932+0000 7f6d1c648740 1 -- 172.21.15.107:0/2029588334 shutdown_connections 2024-05-09T08:35:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.932+0000 7f6d1c648740 1 --2- 172.21.15.107:0/2029588334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9df680 0x56455b9e1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.932+0000 7f6d1c648740 1 --2- 172.21.15.107:0/2029588334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9d0a80 0x56455b9df140 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.932+0000 7f6d1c648740 1 --2- 172.21.15.107:0/2029588334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0160 0x56455b9d0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.932+0000 7f6d1c648740 1 -- 172.21.15.107:0/2029588334 >> 172.21.15.107:0/2029588334 conn(0x56455b9cc5a0 msgr2=0x56455b9cc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.932+0000 7f6d1c648740 1 -- 172.21.15.107:0/2029588334 shutdown_connections 2024-05-09T08:35:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.932+0000 7f6d1c648740 1 -- 172.21.15.107:0/2029588334 wait complete. 2024-05-09T08:35:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.933+0000 7f6d1c648740 1 -- start start 2024-05-09T08:35:03.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.933+0000 7f6d1c648740 1 Processor -- start 2024-05-09T08:35:03.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.933+0000 7f6d1c648740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9d0160 0x56455b9dc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.933+0000 7f6d1c648740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9dcae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.933+0000 7f6d1c648740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9df680 0x56455b9dd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.933+0000 7f6d1c648740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56455b9e2280 con 0x56455b9d0a80 2024-05-09T08:35:03.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.933+0000 7f6d1c648740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56455b8f0b90 con 0x56455b9d0160 2024-05-09T08:35:03.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.933+0000 7f6d1c648740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56455b9e2400 con 0x56455b9df680 2024-05-09T08:35:03.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.933+0000 7f6d1c648740 10 client.0 _ll_register_callbacks cb 0x56455b92cda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:03.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.933+0000 7f6d127fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9dcae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.933+0000 7f6d127fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9dcae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53752/0 (socket says 172.21.15.107:53752) 2024-05-09T08:35:03.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.933+0000 7f6d127fc640 1 -- 172.21.15.107:0/1666408921 learned_addr learned my addr 172.21.15.107:0/1666408921 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:03.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.933+0000 7f6d137fe640 1 --2- 172.21.15.107:0/1666408921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9df680 0x56455b9dd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.933+0000 7f6d12ffd640 1 --2- 172.21.15.107:0/1666408921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9d0160 0x56455b9dc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6d127fc640 1 -- 172.21.15.107:0/1666408921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9df680 msgr2=0x56455b9dd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:03.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6d127fc640 1 --2- 172.21.15.107:0/1666408921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9df680 0x56455b9dd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6d127fc640 1 -- 172.21.15.107:0/1666408921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9d0160 msgr2=0x56455b9dc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:03.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6d127fc640 1 --2- 172.21.15.107:0/1666408921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9d0160 0x56455b9dc5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6d127fc640 1 -- 172.21.15.107:0/1666408921 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56455b6493f0 con 0x56455b9d0a80 2024-05-09T08:35:03.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6d127fc640 1 --2- 172.21.15.107:0/1666408921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9dcae0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f6d08005c20 tx=0x7f6d08005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89f6cda72cd17d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:03.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6cf2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:03.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6cf2ffd640 1 -- 172.21.15.107:0/1666408921 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d080d5020 con 0x56455b9d0a80 2024-05-09T08:35:03.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6cf2ffd640 1 -- 172.21.15.107:0/1666408921 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6d08003c50 con 0x56455b9d0a80 2024-05-09T08:35:03.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6d1c648740 1 -- 172.21.15.107:0/1666408921 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d080c4ce0 con 0x56455b9d0a80 2024-05-09T08:35:03.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6cf2ffd640 1 -- 172.21.15.107:0/1666408921 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6d08049070 con 0x56455b9d0a80 2024-05-09T08:35:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6d1c648740 10 client.16125 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6d1c648740 1 -- 172.21.15.107:0/1666408921 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56455b6090f0 con 0x56455b9d0a80 2024-05-09T08:35:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6cf2ffd640 1 -- 172.21.15.107:0/1666408921 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d080c4ce0 con 0x56455b9d0a80 2024-05-09T08:35:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6cf2ffd640 1 -- 172.21.15.107:0/1666408921 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f6d080c4850 con 0x56455b9d0a80 2024-05-09T08:35:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6d1c648740 10 client.16125 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.934+0000 7f6d1c648740 10 client.16125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.935+0000 7f6d1c648740 20 client.16125 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.935+0000 7f6d1c648740 1 --2- 172.21.15.107:0/1666408921 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56455b9ef640 0x56455b9f1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.935+0000 7f6d1c648740 4 client.16125 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.935+0000 7f6d1c648740 1 -- 172.21.15.107:0/1666408921 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56455b943430 con 0x56455b9ef640 2024-05-09T08:35:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.935+0000 7f6d12ffd640 1 --2- 172.21.15.107:0/1666408921 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56455b9ef640 0x56455b9f1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.935+0000 7f6cf2ffd640 10 client.16125 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.935+0000 7f6d12ffd640 1 --2- 172.21.15.107:0/1666408921 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56455b9ef640 0x56455b9f1a30 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b26fa7af37180b74 server_cookie=3eed1b330575fb3c in_seq=0 out_seq=0 2024-05-09T08:35:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.936+0000 7f6cf2ffd640 1 -- 172.21.15.107:0/1666408921 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56455b943430 con 0x56455b9ef640 2024-05-09T08:35:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.936+0000 7f6cf2ffd640 10 client.16125 handle_command_reply: tid=0 2024-05-09T08:35:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.936+0000 7f6d1c648740 1 client.16125 shutdown 2024-05-09T08:35:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.936+0000 7f6d1c648740 1 -- 172.21.15.107:0/1666408921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 msgr2=0x56455b9dcae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.936+0000 7f6d1c648740 1 --2- 172.21.15.107:0/1666408921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9dcae0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f6d08005c20 tx=0x7f6d08005c90 comp rx=0 tx=0).stop 2024-05-09T08:35:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.936+0000 7f6d1c648740 1 -- 172.21.15.107:0/1666408921 shutdown_connections 2024-05-09T08:35:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.936+0000 7f6d1c648740 1 --2- 172.21.15.107:0/1666408921 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56455b9ef640 0x56455b9f1a30 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.936+0000 7f6d1c648740 1 --2- 172.21.15.107:0/1666408921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9df680 0x56455b9dd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.936+0000 7f6d1c648740 1 --2- 172.21.15.107:0/1666408921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9dcae0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.936+0000 7f6cf2ffd640 0 client.16125 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:03.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.936+0000 7f6d1c648740 1 --2- 172.21.15.107:0/1666408921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9d0160 0x56455b9dc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.936+0000 7f6d1c648740 1 -- 172.21.15.107:0/1666408921 >> 172.21.15.107:0/1666408921 conn(0x56455b9cc5a0 msgr2=0x56455b9d2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:03.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.936+0000 7f6d1c648740 1 -- 172.21.15.107:0/1666408921 shutdown_connections 2024-05-09T08:35:03.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.937+0000 7f6d1c648740 1 -- 172.21.15.107:0/1666408921 wait complete. 2024-05-09T08:35:03.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.937+0000 7f6d1c648740 20 client.16125 trim_cache size 0 max 16384 2024-05-09T08:35:03.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.957+0000 7f6d1c648740 1 Processor -- start 2024-05-09T08:35:03.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.957+0000 7f6d1c648740 1 -- start start 2024-05-09T08:35:03.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.957+0000 7f6d1c648740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9d0160 0x56455b9eb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.957+0000 7f6d1c648740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9f7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.957+0000 7f6d1c648740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9df680 0x56455b9f7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.957+0000 7f6d1c648740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6d141af1a0 con 0x56455b9d0a80 2024-05-09T08:35:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.957+0000 7f6d1c648740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6d141aca70 con 0x56455b9df680 2024-05-09T08:35:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.957+0000 7f6d1c648740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6d141aa8c0 con 0x56455b9d0160 2024-05-09T08:35:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.957+0000 7f6d127fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9f7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.957+0000 7f6d127fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9f7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53760/0 (socket says 172.21.15.107:53760) 2024-05-09T08:35:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.957+0000 7f6d127fc640 1 -- 172.21.15.107:0/2241028730 learned_addr learned my addr 172.21.15.107:0/2241028730 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.958+0000 7f6d12ffd640 1 --2- 172.21.15.107:0/2241028730 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9d0160 0x56455b9eb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.958+0000 7f6d137fe640 1 --2- 172.21.15.107:0/2241028730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9df680 0x56455b9f7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.958+0000 7f6d127fc640 1 -- 172.21.15.107:0/2241028730 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9d0160 msgr2=0x56455b9eb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.958+0000 7f6d127fc640 1 --2- 172.21.15.107:0/2241028730 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9d0160 0x56455b9eb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.958+0000 7f6d127fc640 1 -- 172.21.15.107:0/2241028730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9df680 msgr2=0x56455b9f7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.958+0000 7f6d127fc640 1 --2- 172.21.15.107:0/2241028730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9df680 0x56455b9f7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.958+0000 7f6d127fc640 1 -- 172.21.15.107:0/2241028730 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56455b9f2090 con 0x56455b9d0a80 2024-05-09T08:35:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.958+0000 7f6d127fc640 1 --2- 172.21.15.107:0/2241028730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9f7320 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f6d0803bfd0 tx=0x7f6d080c00d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f3ff3c3624c69e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.958+0000 7f6cf97fa640 1 -- 172.21.15.107:0/2241028730 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d080d5040 con 0x56455b9d0a80 2024-05-09T08:35:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.958+0000 7f6cf97fa640 1 -- 172.21.15.107:0/2241028730 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d080dc0c0 con 0x56455b9d0a80 2024-05-09T08:35:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.958+0000 7f6cf97fa640 1 -- 172.21.15.107:0/2241028730 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d080b8790 con 0x56455b9d0a80 2024-05-09T08:35:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.959+0000 7f6d1c648740 1 -- 172.21.15.107:0/2241028730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 msgr2=0x56455b9f7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.959+0000 7f6d1c648740 1 --2- 172.21.15.107:0/2241028730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9f7320 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f6d0803bfd0 tx=0x7f6d080c00d0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.959+0000 7f6d1c648740 1 -- 172.21.15.107:0/2241028730 shutdown_connections 2024-05-09T08:35:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.959+0000 7f6d1c648740 1 --2- 172.21.15.107:0/2241028730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9df680 0x56455b9f7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.959+0000 7f6d1c648740 1 --2- 172.21.15.107:0/2241028730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9f7320 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.959+0000 7f6d1c648740 1 --2- 172.21.15.107:0/2241028730 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9d0160 0x56455b9eb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.959+0000 7f6d1c648740 1 -- 172.21.15.107:0/2241028730 >> 172.21.15.107:0/2241028730 conn(0x56455b9cc5a0 msgr2=0x56455b9dcdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.959+0000 7f6d1c648740 1 -- 172.21.15.107:0/2241028730 shutdown_connections 2024-05-09T08:35:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.959+0000 7f6d1c648740 1 -- 172.21.15.107:0/2241028730 wait complete. 2024-05-09T08:35:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.959+0000 7f6d1c648740 1 -- start start 2024-05-09T08:35:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.959+0000 7f6d1c648740 1 Processor -- start 2024-05-09T08:35:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d1c648740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9d0160 0x56455b9e3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d1c648740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9e3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d1c648740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9df680 0x56455b9e3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d1c648740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6d141aaf30 con 0x56455b9d0a80 2024-05-09T08:35:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d1c648740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6d141ac3a0 con 0x56455b9d0160 2024-05-09T08:35:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d1c648740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6d141ab600 con 0x56455b9df680 2024-05-09T08:35:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d1c648740 10 client.0 _ll_register_callbacks cb 0x7f6d0400b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d127fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9e3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d127fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9e3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53764/0 (socket says 172.21.15.107:53764) 2024-05-09T08:35:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d127fc640 1 -- 172.21.15.107:0/1251902363 learned_addr learned my addr 172.21.15.107:0/1251902363 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d12ffd640 1 --2- 172.21.15.107:0/1251902363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9d0160 0x56455b9e3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d137fe640 1 --2- 172.21.15.107:0/1251902363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9df680 0x56455b9e3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d127fc640 1 -- 172.21.15.107:0/1251902363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9df680 msgr2=0x56455b9e3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d127fc640 1 --2- 172.21.15.107:0/1251902363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9df680 0x56455b9e3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d127fc640 1 -- 172.21.15.107:0/1251902363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9d0160 msgr2=0x56455b9e3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d127fc640 1 --2- 172.21.15.107:0/1251902363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9d0160 0x56455b9e3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d127fc640 1 -- 172.21.15.107:0/1251902363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56455b9e1e30 con 0x56455b9d0a80 2024-05-09T08:35:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d137fe640 1 --2- 172.21.15.107:0/1251902363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9df680 0x56455b9e3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:35:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d127fc640 1 --2- 172.21.15.107:0/1251902363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9e3570 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f6d080bc0b0 tx=0x7f6d080dc270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8172a2071438881 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6cf2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6cf2ffd640 1 -- 172.21.15.107:0/1251902363 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d080d5040 con 0x56455b9d0a80 2024-05-09T08:35:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6cf2ffd640 1 -- 172.21.15.107:0/1251902363 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6d080c4850 con 0x56455b9d0a80 2024-05-09T08:35:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6d1c648740 1 -- 172.21.15.107:0/1251902363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d080dc0c0 con 0x56455b9d0a80 2024-05-09T08:35:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.960+0000 7f6cf2ffd640 1 -- 172.21.15.107:0/1251902363 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6d08003da0 con 0x56455b9d0a80 2024-05-09T08:35:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.961+0000 7f6d1c648740 10 client.16131 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.961+0000 7f6d1c648740 1 -- 172.21.15.107:0/1251902363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56455b9e1e30 con 0x56455b9d0a80 2024-05-09T08:35:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.961+0000 7f6cf2ffd640 1 -- 172.21.15.107:0/1251902363 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d080dc0c0 con 0x56455b9d0a80 2024-05-09T08:35:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.961+0000 7f6cf2ffd640 1 -- 172.21.15.107:0/1251902363 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f6d080bafe0 con 0x56455b9d0a80 2024-05-09T08:35:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.961+0000 7f6d1c648740 10 client.16131 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.961+0000 7f6d1c648740 10 client.16131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.961+0000 7f6d1c648740 20 client.16131 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.961+0000 7f6d1c648740 1 --2- 172.21.15.107:0/1251902363 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56455ba56010 0x56455ba58430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.961+0000 7f6d1c648740 4 client.16131 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:35:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.961+0000 7f6d1c648740 1 -- 172.21.15.107:0/1251902363 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56455b93d780 con 0x56455ba56010 2024-05-09T08:35:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.961+0000 7f6d12ffd640 1 --2- 172.21.15.107:0/1251902363 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56455ba56010 0x56455ba58430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6cf2ffd640 10 client.16131 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6d12ffd640 1 --2- 172.21.15.107:0/1251902363 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56455ba56010 0x56455ba58430 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7350da20b4ffaf12 server_cookie=3756f5dcfef3e4f2 in_seq=0 out_seq=0 2024-05-09T08:35:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6cf2ffd640 1 -- 172.21.15.107:0/1251902363 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56455b93d780 con 0x56455ba56010 2024-05-09T08:35:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6cf2ffd640 10 client.16131 handle_command_reply: tid=0 2024-05-09T08:35:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6d1c648740 1 client.16131 shutdown 2024-05-09T08:35:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6d1c648740 1 -- 172.21.15.107:0/1251902363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 msgr2=0x56455b9e3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6d1c648740 1 --2- 172.21.15.107:0/1251902363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9e3570 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f6d080bc0b0 tx=0x7f6d080dc270 comp rx=0 tx=0).stop 2024-05-09T08:35:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6d1c648740 1 -- 172.21.15.107:0/1251902363 shutdown_connections 2024-05-09T08:35:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6d1c648740 1 --2- 172.21.15.107:0/1251902363 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56455ba56010 0x56455ba58430 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6d1c648740 1 --2- 172.21.15.107:0/1251902363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56455b9df680 0x56455b9e3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6cf2ffd640 0 client.16131 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6d1c648740 1 --2- 172.21.15.107:0/1251902363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56455b9d0a80 0x56455b9e3570 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6d1c648740 1 --2- 172.21.15.107:0/1251902363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56455b9d0160 0x56455b9e3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6d1c648740 1 -- 172.21.15.107:0/1251902363 >> 172.21.15.107:0/1251902363 conn(0x56455b9cc5a0 msgr2=0x56455b9d1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6d1c648740 1 -- 172.21.15.107:0/1251902363 shutdown_connections 2024-05-09T08:35:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6d1c648740 1 -- 172.21.15.107:0/1251902363 wait complete. 2024-05-09T08:35:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.962+0000 7f6d1c648740 20 client.16131 trim_cache size 0 max 16384 2024-05-09T08:35:03.972 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:35:03.972 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:35:03.972 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:35:03.972 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:35:03.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.963+0000 7f6d1993b640 1 -- 172.21.15.107:0/475830264 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6cfc0a2d00 msgr2=0x7f6cfc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:03.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.963+0000 7f6d1993b640 1 --2- 172.21.15.107:0/475830264 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6cfc0a2d00 0x7f6cfc0a51c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c005fd0 tx=0x7f6d0c014b70 comp rx=0 tx=0).stop 2024-05-09T08:35:03.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.963+0000 7f6d1993b640 1 -- 172.21.15.107:0/475830264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d141033d0 msgr2=0x7f6d14198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:03.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.963+0000 7f6d1993b640 1 --2- 172.21.15.107:0/475830264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d141033d0 0x7f6d14198580 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f6d08005940 tx=0x7f6d08004270 comp rx=0 tx=0).stop 2024-05-09T08:35:03.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.963+0000 7f6d1993b640 1 -- 172.21.15.107:0/475830264 shutdown_connections 2024-05-09T08:35:03.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.963+0000 7f6d1993b640 1 --2- 172.21.15.107:0/475830264 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6cfc0a2d00 0x7f6cfc0a51c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.963+0000 7f6d1993b640 1 --2- 172.21.15.107:0/475830264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d1410bdf0 0x7f6d1419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.963+0000 7f6d1993b640 1 --2- 172.21.15.107:0/475830264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d141033d0 0x7f6d14198580 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.963+0000 7f6d1993b640 1 --2- 172.21.15.107:0/475830264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d14102ab0 0x7f6d14198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:03.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.963+0000 7f6d1993b640 1 -- 172.21.15.107:0/475830264 >> 172.21.15.107:0/475830264 conn(0x7f6d14077830 msgr2=0x7f6d140fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:03.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.963+0000 7f6d1993b640 1 -- 172.21.15.107:0/475830264 shutdown_connections 2024-05-09T08:35:03.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:03.963+0000 7f6d1993b640 1 -- 172.21.15.107:0/475830264 wait complete. 2024-05-09T08:35:03.975 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:35:03.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:35:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.091+0000 7f566257e640 1 -- 172.21.15.107:0/4148077757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f565c1010d0 msgr2=0x7f565c10d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.091+0000 7f566257e640 1 --2- 172.21.15.107:0/4148077757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f565c1010d0 0x7f565c10d5e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f5654009cd0 tx=0x7f565402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.091+0000 7f566257e640 1 -- 172.21.15.107:0/4148077757 shutdown_connections 2024-05-09T08:35:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.091+0000 7f566257e640 1 --2- 172.21.15.107:0/4148077757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f565c1010d0 0x7f565c10d5e0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.091+0000 7f566257e640 1 --2- 172.21.15.107:0/4148077757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f565c100710 0x7f565c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.091+0000 7f566257e640 1 --2- 172.21.15.107:0/4148077757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f565c106720 0x7f565c106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.091+0000 7f566257e640 1 -- 172.21.15.107:0/4148077757 >> 172.21.15.107:0/4148077757 conn(0x7f565c077830 msgr2=0x7f565c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.091+0000 7f566257e640 1 -- 172.21.15.107:0/4148077757 shutdown_connections 2024-05-09T08:35:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.091+0000 7f566257e640 1 -- 172.21.15.107:0/4148077757 wait complete. 2024-05-09T08:35:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.092+0000 7f566257e640 1 Processor -- start 2024-05-09T08:35:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.092+0000 7f566257e640 1 -- start start 2024-05-09T08:35:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.092+0000 7f566257e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f565c100710 0x7f565c196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.092+0000 7f566257e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f565c1010d0 0x7f565c1971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.092+0000 7f566257e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f565c106720 0x7f565c197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.092+0000 7f566257e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f565c1109e0 con 0x7f565c106720 2024-05-09T08:35:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.092+0000 7f566257e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f565c110860 con 0x7f565c100710 2024-05-09T08:35:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.092+0000 7f566257e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f565c110b60 con 0x7f565c1010d0 2024-05-09T08:35:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.092+0000 7f5660b1b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f565c106720 0x7f565c197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.093+0000 7f5660b1b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f565c106720 0x7f565c197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53772/0 (socket says 172.21.15.107:53772) 2024-05-09T08:35:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.093+0000 7f5660b1b640 1 -- 172.21.15.107:0/1576158327 learned_addr learned my addr 172.21.15.107:0/1576158327 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.093+0000 7f565bfff640 1 --2- 172.21.15.107:0/1576158327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f565c100710 0x7f565c196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.093+0000 7f565b7fe640 1 --2- 172.21.15.107:0/1576158327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f565c1010d0 0x7f565c1971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.093+0000 7f5660b1b640 1 -- 172.21.15.107:0/1576158327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f565c1010d0 msgr2=0x7f565c1971d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.093+0000 7f5660b1b640 1 --2- 172.21.15.107:0/1576158327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f565c1010d0 0x7f565c1971d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.093+0000 7f5660b1b640 1 -- 172.21.15.107:0/1576158327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f565c100710 msgr2=0x7f565c196c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.093+0000 7f5660b1b640 1 --2- 172.21.15.107:0/1576158327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f565c100710 0x7f565c196c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.093+0000 7f5660b1b640 1 -- 172.21.15.107:0/1576158327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f565c1a6500 con 0x7f565c106720 2024-05-09T08:35:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.093+0000 7f565b7fe640 1 --2- 172.21.15.107:0/1576158327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f565c1010d0 0x7f565c1971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.093+0000 7f5660b1b640 1 --2- 172.21.15.107:0/1576158327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f565c106720 0x7f565c197710 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f5654005cc0 tx=0x7f5654003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.094+0000 7f56597fa640 1 -- 172.21.15.107:0/1576158327 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5654004450 con 0x7f565c106720 2024-05-09T08:35:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.094+0000 7f56597fa640 1 -- 172.21.15.107:0/1576158327 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5654031c90 con 0x7f565c106720 2024-05-09T08:35:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.094+0000 7f56597fa640 1 -- 172.21.15.107:0/1576158327 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565403b400 con 0x7f565c106720 2024-05-09T08:35:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.094+0000 7f566257e640 1 -- 172.21.15.107:0/1576158327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f565c1a6790 con 0x7f565c106720 2024-05-09T08:35:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.094+0000 7f566257e640 1 -- 172.21.15.107:0/1576158327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f565c1a6c90 con 0x7f565c106720 2024-05-09T08:35:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.095+0000 7f566257e640 1 -- 172.21.15.107:0/1576158327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f565c068650 con 0x7f565c106720 2024-05-09T08:35:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.096+0000 7f56597fa640 1 -- 172.21.15.107:0/1576158327 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f5654004680 con 0x7f565c106720 2024-05-09T08:35:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.097+0000 7f56597fa640 1 --2- 172.21.15.107:0/1576158327 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f56440a2d00 0x7f56440a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.097+0000 7f565bfff640 1 --2- 172.21.15.107:0/1576158327 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f56440a2d00 0x7f56440a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.097+0000 7f56597fa640 1 -- 172.21.15.107:0/1576158327 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f565402e030 con 0x7f565c106720 2024-05-09T08:35:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.097+0000 7f565bfff640 1 --2- 172.21.15.107:0/1576158327 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f56440a2d00 0x7f56440a51c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f564c00bb60 tx=0x7f564c00b390 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.102+0000 7f56597fa640 1 -- 172.21.15.107:0/1576158327 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f56540bd280 con 0x7f565c106720 2024-05-09T08:35:04.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.397+0000 7f566257e640 1 -- 172.21.15.107:0/1576158327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f565c02d1c0 con 0x7f565c106720 2024-05-09T08:35:04.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.398+0000 7f56597fa640 1 -- 172.21.15.107:0/1576158327 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11890 (secure 0 0 0) 0x7f56540098b0 con 0x7f565c106720 2024-05-09T08:35:04.398 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:35:04.398 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:34:45.925212+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24478,24499]},"id":1}]} 2024-05-09T08:35:04.398 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 26 2024-05-09T08:35:04.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.401+0000 7f566257e640 1 -- 172.21.15.107:0/1576158327 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f56440a2d00 msgr2=0x7f56440a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.401+0000 7f566257e640 1 --2- 172.21.15.107:0/1576158327 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f56440a2d00 0x7f56440a51c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f564c00bb60 tx=0x7f564c00b390 comp rx=0 tx=0).stop 2024-05-09T08:35:04.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.401+0000 7f566257e640 1 -- 172.21.15.107:0/1576158327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f565c106720 msgr2=0x7f565c197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.401+0000 7f566257e640 1 --2- 172.21.15.107:0/1576158327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f565c106720 0x7f565c197710 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f5654005cc0 tx=0x7f5654003c20 comp rx=0 tx=0).stop 2024-05-09T08:35:04.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.401+0000 7f566257e640 1 -- 172.21.15.107:0/1576158327 shutdown_connections 2024-05-09T08:35:04.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.401+0000 7f566257e640 1 --2- 172.21.15.107:0/1576158327 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f56440a2d00 0x7f56440a51c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.401+0000 7f566257e640 1 --2- 172.21.15.107:0/1576158327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f565c106720 0x7f565c197710 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.401+0000 7f566257e640 1 --2- 172.21.15.107:0/1576158327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f565c1010d0 0x7f565c1971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.401+0000 7f566257e640 1 --2- 172.21.15.107:0/1576158327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f565c100710 0x7f565c196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.401+0000 7f566257e640 1 -- 172.21.15.107:0/1576158327 >> 172.21.15.107:0/1576158327 conn(0x7f565c077830 msgr2=0x7f565c1030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.401+0000 7f566257e640 1 -- 172.21.15.107:0/1576158327 shutdown_connections 2024-05-09T08:35:04.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.401+0000 7f566257e640 1 -- 172.21.15.107:0/1576158327 wait complete. 2024-05-09T08:35:04.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:35:04.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.523+0000 7f88738b3640 1 -- 172.21.15.107:0/3169612310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f886c106710 msgr2=0x7f886c106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.523+0000 7f88738b3640 1 --2- 172.21.15.107:0/3169612310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f886c106710 0x7f886c106af0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8868009930 tx=0x7f886802a420 comp rx=0 tx=0).stop 2024-05-09T08:35:04.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.523+0000 7f88738b3640 1 -- 172.21.15.107:0/3169612310 shutdown_connections 2024-05-09T08:35:04.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.523+0000 7f88738b3640 1 --2- 172.21.15.107:0/3169612310 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f886c1010c0 0x7f886c10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.523+0000 7f88738b3640 1 --2- 172.21.15.107:0/3169612310 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f886c100700 0x7f886c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.523+0000 7f88738b3640 1 --2- 172.21.15.107:0/3169612310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f886c106710 0x7f886c106af0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.523+0000 7f88738b3640 1 -- 172.21.15.107:0/3169612310 >> 172.21.15.107:0/3169612310 conn(0x7f886c077830 msgr2=0x7f886c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.523+0000 7f88738b3640 1 -- 172.21.15.107:0/3169612310 shutdown_connections 2024-05-09T08:35:04.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.523+0000 7f88738b3640 1 -- 172.21.15.107:0/3169612310 wait complete. 2024-05-09T08:35:04.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f88738b3640 1 Processor -- start 2024-05-09T08:35:04.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f88738b3640 1 -- start start 2024-05-09T08:35:04.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f88738b3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f886c100700 0x7f886c19a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f88738b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f886c1010c0 0x7f886c19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f88738b3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f886c106710 0x7f886c19eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f88738b3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f886c1109d0 con 0x7f886c1010c0 2024-05-09T08:35:04.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f88738b3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f886c110850 con 0x7f886c106710 2024-05-09T08:35:04.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f88738b3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f886c110b50 con 0x7f886c100700 2024-05-09T08:35:04.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f8870e4e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f886c1010c0 0x7f886c19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f8870e4e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f886c1010c0 0x7f886c19a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53776/0 (socket says 172.21.15.107:53776) 2024-05-09T08:35:04.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f8870e4e640 1 -- 172.21.15.107:0/2474801026 learned_addr learned my addr 172.21.15.107:0/2474801026 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:04.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f8871e50640 1 --2- 172.21.15.107:0/2474801026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f886c106710 0x7f886c19eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f887164f640 1 --2- 172.21.15.107:0/2474801026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f886c100700 0x7f886c19a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f8870e4e640 1 -- 172.21.15.107:0/2474801026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f886c100700 msgr2=0x7f886c19a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f8870e4e640 1 --2- 172.21.15.107:0/2474801026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f886c100700 0x7f886c19a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f8870e4e640 1 -- 172.21.15.107:0/2474801026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f886c106710 msgr2=0x7f886c19eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f8870e4e640 1 --2- 172.21.15.107:0/2474801026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f886c106710 0x7f886c19eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f8870e4e640 1 -- 172.21.15.107:0/2474801026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886c19f1d0 con 0x7f886c1010c0 2024-05-09T08:35:04.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.524+0000 7f887164f640 1 --2- 172.21.15.107:0/2474801026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f886c100700 0x7f886c19a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:35:04.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.525+0000 7f8870e4e640 1 --2- 172.21.15.107:0/2474801026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f886c1010c0 0x7f886c19a720 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f885c00d770 tx=0x7f885c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.525+0000 7f88627fc640 1 -- 172.21.15.107:0/2474801026 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c017070 con 0x7f886c1010c0 2024-05-09T08:35:04.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.525+0000 7f88627fc640 1 -- 172.21.15.107:0/2474801026 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f885c0054d0 con 0x7f886c1010c0 2024-05-09T08:35:04.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.525+0000 7f88738b3640 1 -- 172.21.15.107:0/2474801026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f886c19f4c0 con 0x7f886c1010c0 2024-05-09T08:35:04.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.525+0000 7f88627fc640 1 -- 172.21.15.107:0/2474801026 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c004cf0 con 0x7f886c1010c0 2024-05-09T08:35:04.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.525+0000 7f88738b3640 1 -- 172.21.15.107:0/2474801026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f886c1a6d50 con 0x7f886c1010c0 2024-05-09T08:35:04.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.528+0000 7f88627fc640 1 -- 172.21.15.107:0/2474801026 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f885c004e90 con 0x7f886c1010c0 2024-05-09T08:35:04.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.528+0000 7f88765be740 1 Processor -- start 2024-05-09T08:35:04.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.528+0000 7f88765be740 1 -- start start 2024-05-09T08:35:04.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.528+0000 7f88765be740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a8b330 0x55c309a8b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.528+0000 7f88765be740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8bc50 0x55c309a95de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.528+0000 7f88765be740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a96320 0x55c309a98710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.528+0000 7f88765be740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c3099f0c40 con 0x55c309a96320 2024-05-09T08:35:04.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.528+0000 7f88765be740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c309998580 con 0x55c309a8bc50 2024-05-09T08:35:04.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.528+0000 7f88765be740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c3099ecc00 con 0x55c309a8b330 2024-05-09T08:35:04.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.529+0000 7f88627fc640 1 --2- 172.21.15.107:0/2474801026 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f88580a2d80 0x7f88580a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.529+0000 7f8870e4e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a96320 0x55c309a98710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.529+0000 7f8871e50640 1 --2- 172.21.15.107:0/2474801026 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f88580a2d80 0x7f88580a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.529+0000 7f8870e4e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a96320 0x55c309a98710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53780/0 (socket says 172.21.15.107:53780) 2024-05-09T08:35:04.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.529+0000 7f8870e4e640 1 -- 172.21.15.107:0/3563233666 learned_addr learned my addr 172.21.15.107:0/3563233666 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:04.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.529+0000 7f887164f640 1 --2- 172.21.15.107:0/3563233666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8bc50 0x55c309a95de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.529+0000 7f8871e50640 1 --2- 172.21.15.107:0/3563233666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a8b330 0x55c309a8b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.529+0000 7f88627fc640 1 -- 172.21.15.107:0/2474801026 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f885c0cf1a0 con 0x7f886c1010c0 2024-05-09T08:35:04.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.529+0000 7f887164f640 1 -- 172.21.15.107:0/3563233666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a8b330 msgr2=0x55c309a8b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.529+0000 7f887164f640 1 --2- 172.21.15.107:0/3563233666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a8b330 0x55c309a8b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.529+0000 7f887164f640 1 -- 172.21.15.107:0/3563233666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a96320 msgr2=0x55c309a98710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.529+0000 7f887164f640 1 --2- 172.21.15.107:0/3563233666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a96320 0x55c309a98710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.529+0000 7f887164f640 1 -- 172.21.15.107:0/3563233666 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3097043f0 con 0x55c309a8bc50 2024-05-09T08:35:04.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.529+0000 7f8871e50640 1 --2- 172.21.15.107:0/2474801026 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f88580a2d80 0x7f88580a5240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8864007fd0 tx=0x7f8864007d60 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.529+0000 7f887164f640 1 --2- 172.21.15.107:0/3563233666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8bc50 0x55c309a95de0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8868000c00 tx=0x7f8868032cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9183ac2c712ceb2b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.530+0000 7f88577fe640 1 -- 172.21.15.107:0/3563233666 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886803f070 con 0x55c309a8bc50 2024-05-09T08:35:04.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.530+0000 7f88577fe640 1 -- 172.21.15.107:0/3563233666 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8868032e30 con 0x55c309a8bc50 2024-05-09T08:35:04.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.530+0000 7f88577fe640 1 -- 172.21.15.107:0/3563233666 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868035070 con 0x55c309a8bc50 2024-05-09T08:35:04.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.530+0000 7f88765be740 1 -- 172.21.15.107:0/3563233666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8bc50 msgr2=0x55c309a95de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.530+0000 7f88765be740 1 --2- 172.21.15.107:0/3563233666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8bc50 0x55c309a95de0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8868000c00 tx=0x7f8868032cc0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.530+0000 7f88765be740 1 -- 172.21.15.107:0/3563233666 shutdown_connections 2024-05-09T08:35:04.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.530+0000 7f88765be740 1 --2- 172.21.15.107:0/3563233666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a96320 0x55c309a98710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.530+0000 7f88765be740 1 --2- 172.21.15.107:0/3563233666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8bc50 0x55c309a95de0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.530+0000 7f88765be740 1 --2- 172.21.15.107:0/3563233666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a8b330 0x55c309a8b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.530+0000 7f88765be740 1 -- 172.21.15.107:0/3563233666 >> 172.21.15.107:0/3563233666 conn(0x55c309a87650 msgr2=0x55c309a87a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.530+0000 7f88765be740 1 -- 172.21.15.107:0/3563233666 shutdown_connections 2024-05-09T08:35:04.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.530+0000 7f88765be740 1 -- 172.21.15.107:0/3563233666 wait complete. 2024-05-09T08:35:04.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.531+0000 7f88765be740 1 -- start start 2024-05-09T08:35:04.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.531+0000 7f88765be740 1 Processor -- start 2024-05-09T08:35:04.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.531+0000 7f88765be740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8b330 0x55c309a931e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.531+0000 7f88765be740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a8bc50 0x55c309a93720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.531+0000 7f88765be740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a96320 0x55c309a93c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.531+0000 7f88765be740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c309a990b0 con 0x55c309a8bc50 2024-05-09T08:35:04.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.531+0000 7f88765be740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c3099abc40 con 0x55c309a8b330 2024-05-09T08:35:04.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.531+0000 7f88765be740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c309a99230 con 0x55c309a96320 2024-05-09T08:35:04.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.531+0000 7f88765be740 10 client.0 _ll_register_callbacks cb 0x55c309703be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:04.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.531+0000 7f8870e4e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a8bc50 0x55c309a93720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.531+0000 7f8870e4e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a8bc50 0x55c309a93720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53796/0 (socket says 172.21.15.107:53796) 2024-05-09T08:35:04.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.531+0000 7f8870e4e640 1 -- 172.21.15.107:0/353119891 learned_addr learned my addr 172.21.15.107:0/353119891 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:04.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.531+0000 7f8871e50640 1 --2- 172.21.15.107:0/353119891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a96320 0x55c309a93c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.531+0000 7f887164f640 1 --2- 172.21.15.107:0/353119891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8b330 0x55c309a931e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.532+0000 7f8870e4e640 1 -- 172.21.15.107:0/353119891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a96320 msgr2=0x55c309a93c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.532+0000 7f8870e4e640 1 --2- 172.21.15.107:0/353119891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a96320 0x55c309a93c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.532+0000 7f8870e4e640 1 -- 172.21.15.107:0/353119891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8b330 msgr2=0x55c309a931e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.532+0000 7f8870e4e640 1 --2- 172.21.15.107:0/353119891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8b330 0x55c309a931e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.532+0000 7f8870e4e640 1 -- 172.21.15.107:0/353119891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3097ce7d0 con 0x55c309a8bc50 2024-05-09T08:35:04.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.532+0000 7f8871e50640 1 --2- 172.21.15.107:0/353119891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a96320 0x55c309a93c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:04.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.532+0000 7f8870e4e640 1 --2- 172.21.15.107:0/353119891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a8bc50 0x55c309a93720 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f885c0977c0 tx=0x7f885c06b1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68be9ed0d2604d74 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.532+0000 7f8854ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:04.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.532+0000 7f8854ff9640 1 -- 172.21.15.107:0/353119891 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c020040 con 0x55c309a8bc50 2024-05-09T08:35:04.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.532+0000 7f8854ff9640 1 -- 172.21.15.107:0/353119891 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f885c09a7b0 con 0x55c309a8bc50 2024-05-09T08:35:04.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.532+0000 7f88765be740 1 -- 172.21.15.107:0/353119891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8868032e30 con 0x55c309a8bc50 2024-05-09T08:35:04.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.532+0000 7f8854ff9640 1 -- 172.21.15.107:0/353119891 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f885c09acc0 con 0x55c309a8bc50 2024-05-09T08:35:04.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.532+0000 7f88765be740 10 client.16149 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:04.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.532+0000 7f88765be740 1 -- 172.21.15.107:0/353119891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c309aa51e0 con 0x55c309a8bc50 2024-05-09T08:35:04.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.532+0000 7f8854ff9640 1 -- 172.21.15.107:0/353119891 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8868032e30 con 0x55c309a8bc50 2024-05-09T08:35:04.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.533+0000 7f8854ff9640 1 -- 172.21.15.107:0/353119891 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f885c0ccce0 con 0x55c309a8bc50 2024-05-09T08:35:04.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.533+0000 7f88765be740 10 client.16149 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:04.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.533+0000 7f88765be740 10 client.16149 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:35:04.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.533+0000 7f88765be740 20 client.16149 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:04.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.533+0000 7f88765be740 1 --2- 172.21.15.107:0/353119891 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55c309aa66a0 0x55c309aa8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.533+0000 7f88765be740 4 client.16149 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:04.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.533+0000 7f88765be740 1 -- 172.21.15.107:0/353119891 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3099fe370 con 0x55c309aa66a0 2024-05-09T08:35:04.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.533+0000 7f887164f640 1 --2- 172.21.15.107:0/353119891 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55c309aa66a0 0x55c309aa8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.534+0000 7f8854ff9640 10 client.16149 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:35:04.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.534+0000 7f887164f640 1 --2- 172.21.15.107:0/353119891 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55c309aa66a0 0x55c309aa8a90 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=387bfe0cc6d8d3ff server_cookie=fb1f077e77d1b9ca in_seq=0 out_seq=0 2024-05-09T08:35:04.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.534+0000 7f8854ff9640 1 -- 172.21.15.107:0/353119891 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c3099fe370 con 0x55c309aa66a0 2024-05-09T08:35:04.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.534+0000 7f8854ff9640 10 client.16149 handle_command_reply: tid=0 2024-05-09T08:35:04.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.534+0000 7f88765be740 1 client.16149 shutdown 2024-05-09T08:35:04.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.534+0000 7f88765be740 1 -- 172.21.15.107:0/353119891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a8bc50 msgr2=0x55c309a93720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.534+0000 7f88765be740 1 --2- 172.21.15.107:0/353119891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a8bc50 0x55c309a93720 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f885c0977c0 tx=0x7f885c06b1c0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.535+0000 7f88765be740 1 -- 172.21.15.107:0/353119891 shutdown_connections 2024-05-09T08:35:04.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.535+0000 7f88765be740 1 --2- 172.21.15.107:0/353119891 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55c309aa66a0 0x55c309aa8a90 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.535+0000 7f88765be740 1 --2- 172.21.15.107:0/353119891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a96320 0x55c309a93c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.535+0000 7f88765be740 1 --2- 172.21.15.107:0/353119891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a8bc50 0x55c309a93720 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.535+0000 7f88765be740 1 --2- 172.21.15.107:0/353119891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8b330 0x55c309a931e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.535+0000 7f88765be740 1 -- 172.21.15.107:0/353119891 >> 172.21.15.107:0/353119891 conn(0x55c309a87650 msgr2=0x55c309a8d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.535+0000 7f8854ff9640 0 client.16149 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:35:04.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.535+0000 7f88765be740 1 -- 172.21.15.107:0/353119891 shutdown_connections 2024-05-09T08:35:04.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.535+0000 7f88765be740 1 -- 172.21.15.107:0/353119891 wait complete. 2024-05-09T08:35:04.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.535+0000 7f88765be740 20 client.16149 trim_cache size 0 max 16384 2024-05-09T08:35:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f88765be740 1 Processor -- start 2024-05-09T08:35:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f88765be740 1 -- start start 2024-05-09T08:35:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f88765be740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8b330 0x55c309a95910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f88765be740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a8bc50 0x55c309aa1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f88765be740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a96320 0x55c309a86630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f88765be740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f886c1b35e0 con 0x55c309a8bc50 2024-05-09T08:35:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f88765be740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f886c1b0eb0 con 0x55c309a8b330 2024-05-09T08:35:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f88765be740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f886c1aed00 con 0x55c309a96320 2024-05-09T08:35:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f8870e4e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a8bc50 0x55c309aa1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f8870e4e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a8bc50 0x55c309aa1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53810/0 (socket says 172.21.15.107:53810) 2024-05-09T08:35:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f8870e4e640 1 -- 172.21.15.107:0/3652593682 learned_addr learned my addr 172.21.15.107:0/3652593682 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f887164f640 1 --2- 172.21.15.107:0/3652593682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8b330 0x55c309a95910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f8871e50640 1 --2- 172.21.15.107:0/3652593682 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a96320 0x55c309a86630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f8870e4e640 1 -- 172.21.15.107:0/3652593682 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a96320 msgr2=0x55c309a86630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f8870e4e640 1 --2- 172.21.15.107:0/3652593682 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a96320 0x55c309a86630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f8870e4e640 1 -- 172.21.15.107:0/3652593682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8b330 msgr2=0x55c309a95910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f8870e4e640 1 --2- 172.21.15.107:0/3652593682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8b330 0x55c309a95910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.551+0000 7f8870e4e640 1 -- 172.21.15.107:0/3652593682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c309a87480 con 0x55c309a8bc50 2024-05-09T08:35:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.552+0000 7f8870e4e640 1 --2- 172.21.15.107:0/3652593682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a8bc50 0x55c309aa1f70 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f885c099610 tx=0x7f885c09b490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c0d1189abc2c972 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.552+0000 7f8856ffd640 1 -- 172.21.15.107:0/3652593682 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c0b9070 con 0x55c309a8bc50 2024-05-09T08:35:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.552+0000 7f8856ffd640 1 -- 172.21.15.107:0/3652593682 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f885c009c30 con 0x55c309a8bc50 2024-05-09T08:35:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.552+0000 7f8856ffd640 1 -- 172.21.15.107:0/3652593682 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c020070 con 0x55c309a8bc50 2024-05-09T08:35:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.552+0000 7f88765be740 1 -- 172.21.15.107:0/3652593682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a8bc50 msgr2=0x55c309aa1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 1 --2- 172.21.15.107:0/3652593682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a8bc50 0x55c309aa1f70 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f885c099610 tx=0x7f885c09b490 comp rx=0 tx=0).stop 2024-05-09T08:35:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 1 -- 172.21.15.107:0/3652593682 shutdown_connections 2024-05-09T08:35:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 1 --2- 172.21.15.107:0/3652593682 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a96320 0x55c309a86630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 1 --2- 172.21.15.107:0/3652593682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a8bc50 0x55c309aa1f70 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 1 --2- 172.21.15.107:0/3652593682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8b330 0x55c309a95910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 1 -- 172.21.15.107:0/3652593682 >> 172.21.15.107:0/3652593682 conn(0x55c309a87650 msgr2=0x55c309a85f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 1 -- 172.21.15.107:0/3652593682 shutdown_connections 2024-05-09T08:35:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 1 -- 172.21.15.107:0/3652593682 wait complete. 2024-05-09T08:35:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 1 -- start start 2024-05-09T08:35:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 1 Processor -- start 2024-05-09T08:35:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8b330 0x55c309a99fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a8bc50 0x55c309a9a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a96320 0x55c309a9aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f886c1af370 con 0x55c309a96320 2024-05-09T08:35:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f886c1b07e0 con 0x55c309a8b330 2024-05-09T08:35:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f886c1afa40 con 0x55c309a8bc50 2024-05-09T08:35:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.553+0000 7f88765be740 10 client.0 _ll_register_callbacks cb 0x7f885c012f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f8871e50640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a96320 0x55c309a9aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f8871e50640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a96320 0x55c309a9aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53824/0 (socket says 172.21.15.107:53824) 2024-05-09T08:35:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f8871e50640 1 -- 172.21.15.107:0/766167985 learned_addr learned my addr 172.21.15.107:0/766167985 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f8870e4e640 1 --2- 172.21.15.107:0/766167985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a8bc50 0x55c309a9a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f887164f640 1 --2- 172.21.15.107:0/766167985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8b330 0x55c309a99fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f8871e50640 1 -- 172.21.15.107:0/766167985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a8bc50 msgr2=0x55c309a9a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f8871e50640 1 --2- 172.21.15.107:0/766167985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a8bc50 0x55c309a9a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f8871e50640 1 -- 172.21.15.107:0/766167985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8b330 msgr2=0x55c309a99fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f8871e50640 1 --2- 172.21.15.107:0/766167985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8b330 0x55c309a99fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f8871e50640 1 -- 172.21.15.107:0/766167985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c309b0bcd0 con 0x55c309a96320 2024-05-09T08:35:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f8871e50640 1 --2- 172.21.15.107:0/766167985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a96320 0x55c309a9aa30 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f8864014610 tx=0x7f8864007520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e7329a17b98708a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f8854ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f8854ff9640 1 -- 172.21.15.107:0/766167985 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8864012e60 con 0x55c309a96320 2024-05-09T08:35:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f8854ff9640 1 -- 172.21.15.107:0/766167985 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8864004d20 con 0x55c309a96320 2024-05-09T08:35:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f88765be740 1 -- 172.21.15.107:0/766167985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c3099f86c0 con 0x55c309a96320 2024-05-09T08:35:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f8854ff9640 1 -- 172.21.15.107:0/766167985 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8864021640 con 0x55c309a96320 2024-05-09T08:35:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.554+0000 7f88765be740 10 client.16155 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.555+0000 7f88765be740 1 -- 172.21.15.107:0/766167985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c309b0cfd0 con 0x55c309a96320 2024-05-09T08:35:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.555+0000 7f8854ff9640 1 -- 172.21.15.107:0/766167985 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55c3099f86c0 con 0x55c309a96320 2024-05-09T08:35:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.555+0000 7f8854ff9640 1 -- 172.21.15.107:0/766167985 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f8864029070 con 0x55c309a96320 2024-05-09T08:35:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.555+0000 7f88765be740 10 client.16155 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.555+0000 7f88765be740 10 client.16155 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:35:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.555+0000 7f88765be740 20 client.16155 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.555+0000 7f88765be740 1 --2- 172.21.15.107:0/766167985 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55c309b0d780 0x55c309b0fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.555+0000 7f88765be740 4 client.16155 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:35:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.555+0000 7f88765be740 1 -- 172.21.15.107:0/766167985 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c309975820 con 0x55c309b0d780 2024-05-09T08:35:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.555+0000 7f887164f640 1 --2- 172.21.15.107:0/766167985 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55c309b0d780 0x55c309b0fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f8854ff9640 10 client.16155 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:35:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f887164f640 1 --2- 172.21.15.107:0/766167985 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55c309b0d780 0x55c309b0fba0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3fe88cdcf94e93bc server_cookie=5375adfdd80ea1a1 in_seq=0 out_seq=0 2024-05-09T08:35:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f8854ff9640 1 -- 172.21.15.107:0/766167985 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c309975820 con 0x55c309b0d780 2024-05-09T08:35:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f8854ff9640 10 client.16155 handle_command_reply: tid=0 2024-05-09T08:35:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f88765be740 1 client.16155 shutdown 2024-05-09T08:35:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f88765be740 1 -- 172.21.15.107:0/766167985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a96320 msgr2=0x55c309a9aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f88765be740 1 --2- 172.21.15.107:0/766167985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a96320 0x55c309a9aa30 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f8864014610 tx=0x7f8864007520 comp rx=0 tx=0).stop 2024-05-09T08:35:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f88765be740 1 -- 172.21.15.107:0/766167985 shutdown_connections 2024-05-09T08:35:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f88765be740 1 --2- 172.21.15.107:0/766167985 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55c309b0d780 0x55c309b0fba0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f88765be740 1 --2- 172.21.15.107:0/766167985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c309a96320 0x55c309a9aa30 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f88765be740 1 --2- 172.21.15.107:0/766167985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c309a8bc50 0x55c309a9a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f88765be740 1 --2- 172.21.15.107:0/766167985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c309a8b330 0x55c309a99fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f8854ff9640 0 client.16155 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:35:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f88765be740 1 -- 172.21.15.107:0/766167985 >> 172.21.15.107:0/766167985 conn(0x55c309a87650 msgr2=0x55c309a867b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f88765be740 1 -- 172.21.15.107:0/766167985 shutdown_connections 2024-05-09T08:35:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f88765be740 1 -- 172.21.15.107:0/766167985 wait complete. 2024-05-09T08:35:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.556+0000 7f88765be740 20 client.16155 trim_cache size 0 max 16384 2024-05-09T08:35:04.564 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:35:04.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.557+0000 7f88738b3640 1 -- 172.21.15.107:0/2474801026 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f88580a2d80 msgr2=0x7f88580a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.557+0000 7f88738b3640 1 --2- 172.21.15.107:0/2474801026 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f88580a2d80 0x7f88580a5240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8864007fd0 tx=0x7f8864007d60 comp rx=0 tx=0).stop 2024-05-09T08:35:04.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.557+0000 7f88738b3640 1 -- 172.21.15.107:0/2474801026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f886c1010c0 msgr2=0x7f886c19a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.557+0000 7f88738b3640 1 --2- 172.21.15.107:0/2474801026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f886c1010c0 0x7f886c19a720 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f885c00d770 tx=0x7f885c00dc30 comp rx=0 tx=0).stop 2024-05-09T08:35:04.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.557+0000 7f88738b3640 1 -- 172.21.15.107:0/2474801026 shutdown_connections 2024-05-09T08:35:04.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.557+0000 7f88738b3640 1 --2- 172.21.15.107:0/2474801026 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f88580a2d80 0x7f88580a5240 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.557+0000 7f88738b3640 1 --2- 172.21.15.107:0/2474801026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f886c106710 0x7f886c19eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.557+0000 7f88738b3640 1 --2- 172.21.15.107:0/2474801026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f886c1010c0 0x7f886c19a720 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.557+0000 7f88738b3640 1 --2- 172.21.15.107:0/2474801026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f886c100700 0x7f886c19a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.557+0000 7f88738b3640 1 -- 172.21.15.107:0/2474801026 >> 172.21.15.107:0/2474801026 conn(0x7f886c077830 msgr2=0x7f886c068a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.557+0000 7f88738b3640 1 -- 172.21.15.107:0/2474801026 shutdown_connections 2024-05-09T08:35:04.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.557+0000 7f88738b3640 1 -- 172.21.15.107:0/2474801026 wait complete. 2024-05-09T08:35:04.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:35:04.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.680+0000 7f087fd77640 1 -- 172.21.15.107:0/2722865700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0878106710 msgr2=0x7f0878106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.680+0000 7f087fd77640 1 --2- 172.21.15.107:0/2722865700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0878106710 0x7f0878106af0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0874009930 tx=0x7f087402a420 comp rx=0 tx=0).stop 2024-05-09T08:35:04.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.680+0000 7f087fd77640 1 -- 172.21.15.107:0/2722865700 shutdown_connections 2024-05-09T08:35:04.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.680+0000 7f087fd77640 1 --2- 172.21.15.107:0/2722865700 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08781010c0 0x7f087810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.680+0000 7f087fd77640 1 --2- 172.21.15.107:0/2722865700 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0878100700 0x7f0878100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.680+0000 7f087fd77640 1 --2- 172.21.15.107:0/2722865700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0878106710 0x7f0878106af0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.680+0000 7f087fd77640 1 -- 172.21.15.107:0/2722865700 >> 172.21.15.107:0/2722865700 conn(0x7f0878077830 msgr2=0x7f0878077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.680+0000 7f087fd77640 1 -- 172.21.15.107:0/2722865700 shutdown_connections 2024-05-09T08:35:04.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.680+0000 7f087fd77640 1 -- 172.21.15.107:0/2722865700 wait complete. 2024-05-09T08:35:04.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.681+0000 7f087fd77640 1 Processor -- start 2024-05-09T08:35:04.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.681+0000 7f087fd77640 1 -- start start 2024-05-09T08:35:04.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.681+0000 7f087fd77640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0878100700 0x7f0878198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.681+0000 7f087fd77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08781010c0 0x7f0878199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.681+0000 7f087fd77640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0878106710 0x7f08781a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.681+0000 7f087fd77640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0878110990 con 0x7f08781010c0 2024-05-09T08:35:04.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.681+0000 7f087fd77640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0878110810 con 0x7f0878106710 2024-05-09T08:35:04.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.681+0000 7f087fd77640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0878110b10 con 0x7f0878100700 2024-05-09T08:35:04.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.681+0000 7f087d312640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08781010c0 0x7f0878199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.682+0000 7f087d312640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08781010c0 0x7f0878199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53842/0 (socket says 172.21.15.107:53842) 2024-05-09T08:35:04.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.682+0000 7f087d312640 1 -- 172.21.15.107:0/543458169 learned_addr learned my addr 172.21.15.107:0/543458169 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:04.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.682+0000 7f087e314640 1 --2- 172.21.15.107:0/543458169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0878106710 0x7f08781a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.682+0000 7f087d312640 1 -- 172.21.15.107:0/543458169 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0878100700 msgr2=0x7f0878198f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.682+0000 7f087d312640 1 --2- 172.21.15.107:0/543458169 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0878100700 0x7f0878198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.682+0000 7f087d312640 1 -- 172.21.15.107:0/543458169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0878106710 msgr2=0x7f08781a3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.682+0000 7f087d312640 1 --2- 172.21.15.107:0/543458169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0878106710 0x7f08781a3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.682+0000 7f087d312640 1 -- 172.21.15.107:0/543458169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08781a44f0 con 0x7f08781010c0 2024-05-09T08:35:04.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.682+0000 7f087e314640 1 --2- 172.21.15.107:0/543458169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0878106710 0x7f08781a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:04.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.682+0000 7f087d312640 1 --2- 172.21.15.107:0/543458169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08781010c0 0x7f0878199470 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f086800e9f0 tx=0x7f086800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.682+0000 7f086effd640 1 -- 172.21.15.107:0/543458169 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086800cd80 con 0x7f08781010c0 2024-05-09T08:35:04.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.682+0000 7f087fd77640 1 -- 172.21.15.107:0/543458169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08781a46c0 con 0x7f08781010c0 2024-05-09T08:35:04.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.682+0000 7f086effd640 1 -- 172.21.15.107:0/543458169 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08680108e0 con 0x7f08781010c0 2024-05-09T08:35:04.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.682+0000 7f086effd640 1 -- 172.21.15.107:0/543458169 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0868018920 con 0x7f08781010c0 2024-05-09T08:35:04.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.682+0000 7f087fd77640 1 -- 172.21.15.107:0/543458169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08781a4c30 con 0x7f08781010c0 2024-05-09T08:35:04.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.685+0000 7f086effd640 1 -- 172.21.15.107:0/543458169 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0868018ac0 con 0x7f08781010c0 2024-05-09T08:35:04.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.685+0000 7f0882b65740 1 Processor -- start 2024-05-09T08:35:04.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.686+0000 7f0882b65740 1 -- start start 2024-05-09T08:35:04.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.686+0000 7f0882b65740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a171710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.686+0000 7f0882b65740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 0x55ef1a17bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.686+0000 7f0882b65740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a17c320 0x55ef1a17e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.686+0000 7f0882b65740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ef1a0d6c40 con 0x55ef1a171330 2024-05-09T08:35:04.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.686+0000 7f0882b65740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ef1a07e580 con 0x55ef1a17c320 2024-05-09T08:35:04.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.686+0000 7f0882b65740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ef1a0d2c00 con 0x55ef1a171c50 2024-05-09T08:35:04.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.686+0000 7f086effd640 1 --2- 172.21.15.107:0/543458169 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f08640a2d00 0x7f08640a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.686+0000 7f087e314640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a171710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.686+0000 7f087db13640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 0x55ef1a17bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.686+0000 7f087e314640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a171710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53844/0 (socket says 172.21.15.107:53844) 2024-05-09T08:35:04.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.686+0000 7f087db13640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 0x55ef1a17bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:58686/0 (socket says 172.21.15.107:58686) 2024-05-09T08:35:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.686+0000 7f087e314640 1 -- 172.21.15.107:0/2747950291 learned_addr learned my addr 172.21.15.107:0/2747950291 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.686+0000 7f087e314640 1 --2- 172.21.15.107:0/543458169 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f08640a2d00 0x7f08640a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.686+0000 7f086effd640 1 -- 172.21.15.107:0/543458169 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f08680cfc00 con 0x7f08781010c0 2024-05-09T08:35:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.686+0000 7f087d312640 1 --2- 172.21.15.107:0/2747950291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a17c320 0x55ef1a17e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.687+0000 7f087db13640 1 -- 172.21.15.107:0/2747950291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a17c320 msgr2=0x55ef1a17e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.687+0000 7f087db13640 1 --2- 172.21.15.107:0/2747950291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a17c320 0x55ef1a17e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.687+0000 7f087db13640 1 -- 172.21.15.107:0/2747950291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 msgr2=0x55ef1a171710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.687+0000 7f087db13640 1 --2- 172.21.15.107:0/2747950291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a171710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.687+0000 7f087db13640 1 -- 172.21.15.107:0/2747950291 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef19dea3f0 con 0x55ef1a171c50 2024-05-09T08:35:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.687+0000 7f087e314640 1 --2- 172.21.15.107:0/2747950291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a171710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.687+0000 7f087e314640 1 --2- 172.21.15.107:0/543458169 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f08640a2d00 0x7f08640a51c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0870014e70 tx=0x7f0870009a90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.687+0000 7f087db13640 1 --2- 172.21.15.107:0/2747950291 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 0x55ef1a17bde0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0874009930 tx=0x7f0874000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c265bf26423d0ae0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.687+0000 7f0863fff640 1 -- 172.21.15.107:0/2747950291 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0874040070 con 0x55ef1a171c50 2024-05-09T08:35:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.687+0000 7f0863fff640 1 -- 172.21.15.107:0/2747950291 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0874032d80 con 0x55ef1a171c50 2024-05-09T08:35:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.687+0000 7f0863fff640 1 -- 172.21.15.107:0/2747950291 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0874035040 con 0x55ef1a171c50 2024-05-09T08:35:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.688+0000 7f0882b65740 1 -- 172.21.15.107:0/2747950291 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 msgr2=0x55ef1a17bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.688+0000 7f0882b65740 1 --2- 172.21.15.107:0/2747950291 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 0x55ef1a17bde0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0874009930 tx=0x7f0874000bc0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.688+0000 7f0882b65740 1 -- 172.21.15.107:0/2747950291 shutdown_connections 2024-05-09T08:35:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.688+0000 7f0882b65740 1 --2- 172.21.15.107:0/2747950291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a17c320 0x55ef1a17e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.688+0000 7f0882b65740 1 --2- 172.21.15.107:0/2747950291 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 0x55ef1a17bde0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.688+0000 7f0882b65740 1 --2- 172.21.15.107:0/2747950291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a171710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.688+0000 7f0882b65740 1 -- 172.21.15.107:0/2747950291 >> 172.21.15.107:0/2747950291 conn(0x55ef1a16d650 msgr2=0x55ef1a16da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.688+0000 7f0882b65740 1 -- 172.21.15.107:0/2747950291 shutdown_connections 2024-05-09T08:35:04.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.688+0000 7f0882b65740 1 -- 172.21.15.107:0/2747950291 wait complete. 2024-05-09T08:35:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.688+0000 7f0882b65740 1 -- start start 2024-05-09T08:35:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.688+0000 7f0882b65740 1 Processor -- start 2024-05-09T08:35:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f0882b65740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171330 0x55ef1a179240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f0882b65740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a171c50 0x55ef1a179780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f0882b65740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a17c320 0x55ef1a179cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f0882b65740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ef1a17f0b0 con 0x55ef1a17c320 2024-05-09T08:35:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f0882b65740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ef1a091c40 con 0x55ef1a171c50 2024-05-09T08:35:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f0882b65740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ef1a17f230 con 0x55ef1a171330 2024-05-09T08:35:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f0882b65740 10 client.0 _ll_register_callbacks cb 0x55ef19de9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f087e314640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a17c320 0x55ef1a179cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f087e314640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a17c320 0x55ef1a179cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53852/0 (socket says 172.21.15.107:53852) 2024-05-09T08:35:04.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f087e314640 1 -- 172.21.15.107:0/1541362968 learned_addr learned my addr 172.21.15.107:0/1541362968 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f087db13640 1 --2- 172.21.15.107:0/1541362968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171330 0x55ef1a179240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f087d312640 1 --2- 172.21.15.107:0/1541362968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a171c50 0x55ef1a179780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f087e314640 1 -- 172.21.15.107:0/1541362968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171330 msgr2=0x55ef1a179240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f087e314640 1 --2- 172.21.15.107:0/1541362968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171330 0x55ef1a179240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f087e314640 1 -- 172.21.15.107:0/1541362968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a171c50 msgr2=0x55ef1a179780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f087e314640 1 --2- 172.21.15.107:0/1541362968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a171c50 0x55ef1a179780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f087e314640 1 -- 172.21.15.107:0/1541362968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef19eb47d0 con 0x55ef1a17c320 2024-05-09T08:35:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f087d312640 1 --2- 172.21.15.107:0/1541362968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a171c50 0x55ef1a179780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f087e314640 1 --2- 172.21.15.107:0/1541362968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a17c320 0x55ef1a179cc0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0870009740 tx=0x7f0870009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e33d192ff41007 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.689+0000 7f08617fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.690+0000 7f08617fa640 1 -- 172.21.15.107:0/1541362968 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0870004d20 con 0x55ef1a17c320 2024-05-09T08:35:04.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.690+0000 7f08617fa640 1 -- 172.21.15.107:0/1541362968 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f0870020600 con 0x55ef1a17c320 2024-05-09T08:35:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.690+0000 7f0882b65740 1 -- 172.21.15.107:0/1541362968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0874032d80 con 0x55ef1a17c320 2024-05-09T08:35:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.690+0000 7f08617fa640 1 -- 172.21.15.107:0/1541362968 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f087000fe60 con 0x55ef1a17c320 2024-05-09T08:35:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.690+0000 7f0882b65740 10 client.16167 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.690+0000 7f0882b65740 1 -- 172.21.15.107:0/1541362968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ef1a18b200 con 0x55ef1a17c320 2024-05-09T08:35:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.690+0000 7f08617fa640 1 -- 172.21.15.107:0/1541362968 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0874032d80 con 0x55ef1a17c320 2024-05-09T08:35:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.690+0000 7f08617fa640 1 -- 172.21.15.107:0/1541362968 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f087002a8f0 con 0x55ef1a17c320 2024-05-09T08:35:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.690+0000 7f0882b65740 10 client.16167 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.690+0000 7f0882b65740 10 client.16167 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:35:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.690+0000 7f0882b65740 20 client.16167 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.690+0000 7f0882b65740 1 --2- 172.21.15.107:0/1541362968 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55ef1a18c710 0x55ef1a18eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.690+0000 7f0882b65740 4 client.16167 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:04.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.690+0000 7f0882b65740 1 -- 172.21.15.107:0/1541362968 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef1a0e4370 con 0x55ef1a18c710 2024-05-09T08:35:04.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.691+0000 7f087db13640 1 --2- 172.21.15.107:0/1541362968 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55ef1a18c710 0x55ef1a18eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.691+0000 7f08617fa640 10 client.16167 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:35:04.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.691+0000 7f087db13640 1 --2- 172.21.15.107:0/1541362968 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55ef1a18c710 0x55ef1a18eb20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9ddaa26f02542e8c server_cookie=b69dbf9602a5ebde in_seq=0 out_seq=0 2024-05-09T08:35:04.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.692+0000 7f08617fa640 1 -- 172.21.15.107:0/1541362968 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef1a0e4370 con 0x55ef1a18c710 2024-05-09T08:35:04.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.692+0000 7f08617fa640 10 client.16167 handle_command_reply: tid=0 2024-05-09T08:35:04.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.692+0000 7f0882b65740 1 client.16167 shutdown 2024-05-09T08:35:04.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.692+0000 7f0882b65740 1 -- 172.21.15.107:0/1541362968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a17c320 msgr2=0x55ef1a179cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.692+0000 7f0882b65740 1 --2- 172.21.15.107:0/1541362968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a17c320 0x55ef1a179cc0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0870009740 tx=0x7f0870009790 comp rx=0 tx=0).stop 2024-05-09T08:35:04.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.692+0000 7f0882b65740 1 -- 172.21.15.107:0/1541362968 shutdown_connections 2024-05-09T08:35:04.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.692+0000 7f0882b65740 1 --2- 172.21.15.107:0/1541362968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a17c320 0x55ef1a179cc0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.692+0000 7f0882b65740 1 --2- 172.21.15.107:0/1541362968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a171c50 0x55ef1a179780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.692+0000 7f0882b65740 1 --2- 172.21.15.107:0/1541362968 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55ef1a18c710 0x55ef1a18eb20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.692+0000 7f0882b65740 1 --2- 172.21.15.107:0/1541362968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171330 0x55ef1a179240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.692+0000 7f08617fa640 0 client.16167 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:35:04.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.692+0000 7f0882b65740 1 -- 172.21.15.107:0/1541362968 >> 172.21.15.107:0/1541362968 conn(0x55ef1a16d650 msgr2=0x55ef1a173450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.692+0000 7f0882b65740 1 -- 172.21.15.107:0/1541362968 shutdown_connections 2024-05-09T08:35:04.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.692+0000 7f0882b65740 1 -- 172.21.15.107:0/1541362968 wait complete. 2024-05-09T08:35:04.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.693+0000 7f0882b65740 20 client.16167 trim_cache size 0 max 16384 2024-05-09T08:35:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.710+0000 7f0882b65740 1 Processor -- start 2024-05-09T08:35:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.710+0000 7f0882b65740 1 -- start start 2024-05-09T08:35:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.710+0000 7f0882b65740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a187f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.710+0000 7f0882b65740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 0x55ef1a173ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.710+0000 7f0882b65740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a17c320 0x55ef1a178b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.710+0000 7f0882b65740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f08781b1440 con 0x55ef1a171330 2024-05-09T08:35:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.710+0000 7f0882b65740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f08781aed10 con 0x55ef1a17c320 2024-05-09T08:35:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.710+0000 7f0882b65740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f08781acb60 con 0x55ef1a171c50 2024-05-09T08:35:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.711+0000 7f087db13640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a187f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.711+0000 7f087db13640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a187f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53858/0 (socket says 172.21.15.107:53858) 2024-05-09T08:35:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.711+0000 7f087db13640 1 -- 172.21.15.107:0/2694415072 learned_addr learned my addr 172.21.15.107:0/2694415072 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.711+0000 7f087d312640 1 --2- 172.21.15.107:0/2694415072 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 0x55ef1a173ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.711+0000 7f087e314640 1 --2- 172.21.15.107:0/2694415072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a17c320 0x55ef1a178b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.711+0000 7f087db13640 1 -- 172.21.15.107:0/2694415072 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 msgr2=0x55ef1a173ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.711+0000 7f087db13640 1 --2- 172.21.15.107:0/2694415072 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 0x55ef1a173ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.711+0000 7f087db13640 1 -- 172.21.15.107:0/2694415072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a17c320 msgr2=0x55ef1a178b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.711+0000 7f087db13640 1 --2- 172.21.15.107:0/2694415072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a17c320 0x55ef1a178b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.711+0000 7f087db13640 1 -- 172.21.15.107:0/2694415072 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef1a177790 con 0x55ef1a171330 2024-05-09T08:35:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.711+0000 7f087db13640 1 --2- 172.21.15.107:0/2694415072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a187f70 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f0874009900 tx=0x7f0874031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3863b20a50ed93ad server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.711+0000 7f08637fe640 1 -- 172.21.15.107:0/2694415072 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0874035050 con 0x55ef1a171330 2024-05-09T08:35:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.711+0000 7f08637fe640 1 -- 172.21.15.107:0/2694415072 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08740046c0 con 0x55ef1a171330 2024-05-09T08:35:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.711+0000 7f08637fe640 1 -- 172.21.15.107:0/2694415072 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0874040040 con 0x55ef1a171330 2024-05-09T08:35:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.712+0000 7f0882b65740 1 -- 172.21.15.107:0/2694415072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 msgr2=0x55ef1a187f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.712+0000 7f0882b65740 1 --2- 172.21.15.107:0/2694415072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a187f70 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f0874009900 tx=0x7f0874031400 comp rx=0 tx=0).stop 2024-05-09T08:35:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.712+0000 7f0882b65740 1 -- 172.21.15.107:0/2694415072 shutdown_connections 2024-05-09T08:35:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.712+0000 7f0882b65740 1 --2- 172.21.15.107:0/2694415072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a17c320 0x55ef1a178b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.712+0000 7f0882b65740 1 --2- 172.21.15.107:0/2694415072 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 0x55ef1a173ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.712+0000 7f0882b65740 1 --2- 172.21.15.107:0/2694415072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a187f70 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.712+0000 7f0882b65740 1 -- 172.21.15.107:0/2694415072 >> 172.21.15.107:0/2694415072 conn(0x55ef1a16d650 msgr2=0x55ef1a16c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.712+0000 7f0882b65740 1 -- 172.21.15.107:0/2694415072 shutdown_connections 2024-05-09T08:35:04.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.712+0000 7f0882b65740 1 -- 172.21.15.107:0/2694415072 wait complete. 2024-05-09T08:35:04.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.712+0000 7f0882b65740 1 -- start start 2024-05-09T08:35:04.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f0882b65740 1 Processor -- start 2024-05-09T08:35:04.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f0882b65740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a17f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f0882b65740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 0x55ef1a17fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f0882b65740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a17c320 0x55ef1a1f0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f0882b65740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f08781ad1d0 con 0x55ef1a171330 2024-05-09T08:35:04.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f0882b65740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f08781ae640 con 0x55ef1a17c320 2024-05-09T08:35:04.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f0882b65740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f08781ad8a0 con 0x55ef1a171c50 2024-05-09T08:35:04.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f0882b65740 10 client.0 _ll_register_callbacks cb 0x7f0870009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f087db13640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a17f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f087db13640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a17f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53872/0 (socket says 172.21.15.107:53872) 2024-05-09T08:35:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f087db13640 1 -- 172.21.15.107:0/727777242 learned_addr learned my addr 172.21.15.107:0/727777242 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f087e314640 1 --2- 172.21.15.107:0/727777242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a17c320 0x55ef1a1f0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f087d312640 1 --2- 172.21.15.107:0/727777242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 0x55ef1a17fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f087db13640 1 -- 172.21.15.107:0/727777242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 msgr2=0x55ef1a17fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f087db13640 1 --2- 172.21.15.107:0/727777242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 0x55ef1a17fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f087db13640 1 -- 172.21.15.107:0/727777242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a17c320 msgr2=0x55ef1a1f0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f087db13640 1 --2- 172.21.15.107:0/727777242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a17c320 0x55ef1a1f0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.713+0000 7f087db13640 1 -- 172.21.15.107:0/727777242 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef1a1f1220 con 0x55ef1a171330 2024-05-09T08:35:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.714+0000 7f087db13640 1 --2- 172.21.15.107:0/727777242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a17f630 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f0874005940 tx=0x7f0874005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1eb7f58d0793da server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.714+0000 7f08617fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.714+0000 7f08617fa640 1 -- 172.21.15.107:0/727777242 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0874054020 con 0x55ef1a171330 2024-05-09T08:35:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.714+0000 7f08617fa640 1 -- 172.21.15.107:0/727777242 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f0874004870 con 0x55ef1a171330 2024-05-09T08:35:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.714+0000 7f0882b65740 1 -- 172.21.15.107:0/727777242 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08740046c0 con 0x55ef1a171330 2024-05-09T08:35:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.714+0000 7f08617fa640 1 -- 172.21.15.107:0/727777242 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0874035020 con 0x55ef1a171330 2024-05-09T08:35:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.714+0000 7f0882b65740 10 client.16173 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.714+0000 7f0882b65740 1 -- 172.21.15.107:0/727777242 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ef1a1f2520 con 0x55ef1a171330 2024-05-09T08:35:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.714+0000 7f08617fa640 1 -- 172.21.15.107:0/727777242 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08740046c0 con 0x55ef1a171330 2024-05-09T08:35:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.714+0000 7f08617fa640 1 -- 172.21.15.107:0/727777242 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f08740314a0 con 0x55ef1a171330 2024-05-09T08:35:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.714+0000 7f0882b65740 10 client.16173 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.714+0000 7f0882b65740 10 client.16173 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:35:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.714+0000 7f0882b65740 20 client.16173 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.714+0000 7f0882b65740 1 --2- 172.21.15.107:0/727777242 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55ef1a1f2800 0x55ef1a1f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.714+0000 7f0882b65740 4 client.16173 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:35:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.714+0000 7f0882b65740 1 -- 172.21.15.107:0/727777242 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ef1a0de6c0 con 0x55ef1a1f2800 2024-05-09T08:35:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.715+0000 7f087d312640 1 --2- 172.21.15.107:0/727777242 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55ef1a1f2800 0x55ef1a1f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.715+0000 7f08617fa640 10 client.16173 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:35:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.715+0000 7f087d312640 1 --2- 172.21.15.107:0/727777242 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55ef1a1f2800 0x55ef1a1f4bf0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9d370f609ecd4d7 server_cookie=9853d5f6eebe786 in_seq=0 out_seq=0 2024-05-09T08:35:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.715+0000 7f08617fa640 1 -- 172.21.15.107:0/727777242 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ef1a0de6c0 con 0x55ef1a1f2800 2024-05-09T08:35:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.715+0000 7f08617fa640 10 client.16173 handle_command_reply: tid=0 2024-05-09T08:35:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.715+0000 7f0882b65740 1 client.16173 shutdown 2024-05-09T08:35:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.716+0000 7f0882b65740 1 -- 172.21.15.107:0/727777242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 msgr2=0x55ef1a17f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.716+0000 7f0882b65740 1 --2- 172.21.15.107:0/727777242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a17f630 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f0874005940 tx=0x7f0874005020 comp rx=0 tx=0).stop 2024-05-09T08:35:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.716+0000 7f0882b65740 1 -- 172.21.15.107:0/727777242 shutdown_connections 2024-05-09T08:35:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.716+0000 7f0882b65740 1 --2- 172.21.15.107:0/727777242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef1a17c320 0x55ef1a1f0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.716+0000 7f0882b65740 1 --2- 172.21.15.107:0/727777242 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55ef1a1f2800 0x55ef1a1f4bf0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.716+0000 7f0882b65740 1 --2- 172.21.15.107:0/727777242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef1a171c50 0x55ef1a17fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.716+0000 7f0882b65740 1 --2- 172.21.15.107:0/727777242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef1a171330 0x55ef1a17f630 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.716+0000 7f0882b65740 1 -- 172.21.15.107:0/727777242 >> 172.21.15.107:0/727777242 conn(0x55ef1a16d650 msgr2=0x55ef1a16c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.716+0000 7f08617fa640 0 client.16173 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:35:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.716+0000 7f0882b65740 1 -- 172.21.15.107:0/727777242 shutdown_connections 2024-05-09T08:35:04.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.716+0000 7f0882b65740 1 -- 172.21.15.107:0/727777242 wait complete. 2024-05-09T08:35:04.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.716+0000 7f0882b65740 20 client.16173 trim_cache size 0 max 16384 2024-05-09T08:35:04.722 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:35:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.716+0000 7f087fd77640 1 -- 172.21.15.107:0/543458169 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f08640a2d00 msgr2=0x7f08640a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.716+0000 7f087fd77640 1 --2- 172.21.15.107:0/543458169 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f08640a2d00 0x7f08640a51c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0870014e70 tx=0x7f0870009a90 comp rx=0 tx=0).stop 2024-05-09T08:35:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.716+0000 7f087fd77640 1 -- 172.21.15.107:0/543458169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08781010c0 msgr2=0x7f0878199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.716+0000 7f087fd77640 1 --2- 172.21.15.107:0/543458169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08781010c0 0x7f0878199470 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f086800e9f0 tx=0x7f086800eeb0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.717+0000 7f087fd77640 1 -- 172.21.15.107:0/543458169 shutdown_connections 2024-05-09T08:35:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.717+0000 7f087fd77640 1 --2- 172.21.15.107:0/543458169 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f08640a2d00 0x7f08640a51c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.717+0000 7f087fd77640 1 --2- 172.21.15.107:0/543458169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0878106710 0x7f08781a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.717+0000 7f087fd77640 1 --2- 172.21.15.107:0/543458169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08781010c0 0x7f0878199470 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.717+0000 7f087fd77640 1 --2- 172.21.15.107:0/543458169 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0878100700 0x7f0878198f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.717+0000 7f087fd77640 1 -- 172.21.15.107:0/543458169 >> 172.21.15.107:0/543458169 conn(0x7f0878077830 msgr2=0x7f0878103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.717+0000 7f087fd77640 1 -- 172.21.15.107:0/543458169 shutdown_connections 2024-05-09T08:35:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.717+0000 7f087fd77640 1 -- 172.21.15.107:0/543458169 wait complete. 2024-05-09T08:35:04.725 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:04 smithi107 ceph-mon[30122]: pgmap v54: 129 pgs: 129 active+clean; 91 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 7.4 MiB/s wr, 275 op/s 2024-05-09T08:35:04.726 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:04 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1576158327' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:35:04.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:35:04.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.841+0000 7f6a5eabd640 1 -- 172.21.15.107:0/1591076747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a58101880 msgr2=0x7f6a58101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.841+0000 7f6a5eabd640 1 --2- 172.21.15.107:0/1591076747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a58101880 0x7f6a58101d00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6a44009930 tx=0x7f6a4402a420 comp rx=0 tx=0).stop 2024-05-09T08:35:04.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.841+0000 7f6a5eabd640 1 -- 172.21.15.107:0/1591076747 shutdown_connections 2024-05-09T08:35:04.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.841+0000 7f6a5eabd640 1 --2- 172.21.15.107:0/1591076747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a58102240 0x7f6a5810e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.841+0000 7f6a5eabd640 1 --2- 172.21.15.107:0/1591076747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a58101880 0x7f6a58101d00 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.841+0000 7f6a5eabd640 1 --2- 172.21.15.107:0/1591076747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a58107890 0x7f6a58107c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.841+0000 7f6a5eabd640 1 -- 172.21.15.107:0/1591076747 >> 172.21.15.107:0/1591076747 conn(0x7f6a580fd390 msgr2=0x7f6a580ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.841+0000 7f6a5eabd640 1 -- 172.21.15.107:0/1591076747 shutdown_connections 2024-05-09T08:35:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.841+0000 7f6a5eabd640 1 -- 172.21.15.107:0/1591076747 wait complete. 2024-05-09T08:35:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.842+0000 7f6a5eabd640 1 Processor -- start 2024-05-09T08:35:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.842+0000 7f6a5eabd640 1 -- start start 2024-05-09T08:35:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.842+0000 7f6a5eabd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a58101880 0x7f6a58198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.842+0000 7f6a5eabd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a58102240 0x7f6a581986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.842+0000 7f6a5eabd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a58107890 0x7f6a5819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.842+0000 7f6a5eabd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6a58111b50 con 0x7f6a58102240 2024-05-09T08:35:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.842+0000 7f6a5eabd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6a581119d0 con 0x7f6a58107890 2024-05-09T08:35:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.842+0000 7f6a5eabd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6a58111cd0 con 0x7f6a58101880 2024-05-09T08:35:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.842+0000 7f6a57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a58102240 0x7f6a581986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.842+0000 7f6a57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a58102240 0x7f6a581986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53900/0 (socket says 172.21.15.107:53900) 2024-05-09T08:35:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.842+0000 7f6a57fff640 1 -- 172.21.15.107:0/3168581750 learned_addr learned my addr 172.21.15.107:0/3168581750 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.842+0000 7f6a5c859640 1 --2- 172.21.15.107:0/3168581750 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a58101880 0x7f6a58198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.842+0000 7f6a5d05a640 1 --2- 172.21.15.107:0/3168581750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a58107890 0x7f6a5819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.843+0000 7f6a57fff640 1 -- 172.21.15.107:0/3168581750 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a58101880 msgr2=0x7f6a58198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.843+0000 7f6a57fff640 1 --2- 172.21.15.107:0/3168581750 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a58101880 0x7f6a58198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.843+0000 7f6a57fff640 1 -- 172.21.15.107:0/3168581750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a58107890 msgr2=0x7f6a5819ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.843+0000 7f6a57fff640 1 --2- 172.21.15.107:0/3168581750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a58107890 0x7f6a5819ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.843+0000 7f6a57fff640 1 -- 172.21.15.107:0/3168581750 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a5819d180 con 0x7f6a58102240 2024-05-09T08:35:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.843+0000 7f6a5d05a640 1 --2- 172.21.15.107:0/3168581750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a58107890 0x7f6a5819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.843+0000 7f6a57fff640 1 --2- 172.21.15.107:0/3168581750 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a58102240 0x7f6a581986d0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f6a44004770 tx=0x7f6a440042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.843+0000 7f6a55ffb640 1 -- 172.21.15.107:0/3168581750 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4403f070 con 0x7f6a58102240 2024-05-09T08:35:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.843+0000 7f6a55ffb640 1 -- 172.21.15.107:0/3168581750 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a4403b460 con 0x7f6a58102240 2024-05-09T08:35:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.843+0000 7f6a5eabd640 1 -- 172.21.15.107:0/3168581750 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a5819d410 con 0x7f6a58102240 2024-05-09T08:35:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.843+0000 7f6a55ffb640 1 -- 172.21.15.107:0/3168581750 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a44035070 con 0x7f6a58102240 2024-05-09T08:35:04.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.844+0000 7f6a5eabd640 1 -- 172.21.15.107:0/3168581750 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a581a4d00 con 0x7f6a58102240 2024-05-09T08:35:04.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:04 smithi157 ceph-mon[33614]: pgmap v54: 129 pgs: 129 active+clean; 91 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 7.4 MiB/s wr, 275 op/s 2024-05-09T08:35:04.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:04 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1576158327' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:35:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.846+0000 7f6a55ffb640 1 -- 172.21.15.107:0/3168581750 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6a4403b610 con 0x7f6a58102240 2024-05-09T08:35:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.846+0000 7f6a617d3740 1 Processor -- start 2024-05-09T08:35:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.846+0000 7f6a617d3740 1 -- start start 2024-05-09T08:35:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.846+0000 7f6a617d3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4330 0x5604e82c4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.846+0000 7f6a617d3740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82c4c50 0x5604e82cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.846+0000 7f6a617d3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82cf320 0x5604e82d1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.846+0000 7f6a617d3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5604e8229c40 con 0x5604e82cf320 2024-05-09T08:35:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.846+0000 7f6a617d3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5604e81d1580 con 0x5604e82c4330 2024-05-09T08:35:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.846+0000 7f6a617d3740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5604e8225c00 con 0x5604e82c4c50 2024-05-09T08:35:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.846+0000 7f6a57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82cf320 0x5604e82d1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.847+0000 7f6a5c859640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82c4c50 0x5604e82cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.847+0000 7f6a5c859640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82c4c50 0x5604e82cede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:58736/0 (socket says 172.21.15.107:58736) 2024-05-09T08:35:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.847+0000 7f6a5c859640 1 -- 172.21.15.107:0/608000943 learned_addr learned my addr 172.21.15.107:0/608000943 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.847+0000 7f6a5d05a640 1 --2- 172.21.15.107:0/608000943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4330 0x5604e82c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.847+0000 7f6a55ffb640 1 --2- 172.21.15.107:0/3168581750 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6a480a2d00 0x7f6a480a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.847+0000 7f6a5c859640 1 -- 172.21.15.107:0/608000943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4330 msgr2=0x5604e82c4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.847+0000 7f6a5c859640 1 --2- 172.21.15.107:0/608000943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4330 0x5604e82c4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.847+0000 7f6a5c859640 1 -- 172.21.15.107:0/608000943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82cf320 msgr2=0x5604e82d1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.847+0000 7f6a5c859640 1 --2- 172.21.15.107:0/608000943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82cf320 0x5604e82d1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.847+0000 7f6a5c859640 1 -- 172.21.15.107:0/608000943 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604e7f3d3f0 con 0x5604e82c4c50 2024-05-09T08:35:04.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.847+0000 7f6a5d05a640 1 --2- 172.21.15.107:0/3168581750 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6a480a2d00 0x7f6a480a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.847+0000 7f6a55ffb640 1 -- 172.21.15.107:0/3168581750 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6a440efa90 con 0x7f6a58102240 2024-05-09T08:35:04.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.847+0000 7f6a5c859640 1 --2- 172.21.15.107:0/608000943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82c4c50 0x5604e82cede0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c009a70 tx=0x7f6a4c009f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58eec95b661168c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.847+0000 7f6a42ffd640 1 -- 172.21.15.107:0/608000943 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c017070 con 0x5604e82c4c50 2024-05-09T08:35:04.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.847+0000 7f6a42ffd640 1 -- 172.21.15.107:0/608000943 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a4c013470 con 0x5604e82c4c50 2024-05-09T08:35:04.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.847+0000 7f6a42ffd640 1 -- 172.21.15.107:0/608000943 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c012610 con 0x5604e82c4c50 2024-05-09T08:35:04.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.848+0000 7f6a617d3740 1 -- 172.21.15.107:0/608000943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82c4c50 msgr2=0x5604e82cede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.848+0000 7f6a617d3740 1 --2- 172.21.15.107:0/608000943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82c4c50 0x5604e82cede0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c009a70 tx=0x7f6a4c009f30 comp rx=0 tx=0).stop 2024-05-09T08:35:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.848+0000 7f6a5d05a640 1 --2- 172.21.15.107:0/3168581750 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6a480a2d00 0x7f6a480a51c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6a50005fd0 tx=0x7f6a50004ab0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.848+0000 7f6a617d3740 1 -- 172.21.15.107:0/608000943 shutdown_connections 2024-05-09T08:35:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.848+0000 7f6a617d3740 1 --2- 172.21.15.107:0/608000943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82cf320 0x5604e82d1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.848+0000 7f6a617d3740 1 --2- 172.21.15.107:0/608000943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82c4c50 0x5604e82cede0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.848+0000 7f6a617d3740 1 --2- 172.21.15.107:0/608000943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4330 0x5604e82c4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.848+0000 7f6a617d3740 1 -- 172.21.15.107:0/608000943 >> 172.21.15.107:0/608000943 conn(0x5604e82c0650 msgr2=0x5604e82c0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.848+0000 7f6a617d3740 1 -- 172.21.15.107:0/608000943 shutdown_connections 2024-05-09T08:35:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.848+0000 7f6a617d3740 1 -- 172.21.15.107:0/608000943 wait complete. 2024-05-09T08:35:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.849+0000 7f6a617d3740 1 -- start start 2024-05-09T08:35:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.849+0000 7f6a617d3740 1 Processor -- start 2024-05-09T08:35:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.849+0000 7f6a617d3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82c4330 0x5604e82cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.849+0000 7f6a617d3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4c50 0x5604e82cc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.849+0000 7f6a617d3740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82cf320 0x5604e82ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.849+0000 7f6a617d3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5604e82d2110 con 0x5604e82c4330 2024-05-09T08:35:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.849+0000 7f6a617d3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5604e81e4c40 con 0x5604e82c4c50 2024-05-09T08:35:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.849+0000 7f6a617d3740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5604e82d2290 con 0x5604e82cf320 2024-05-09T08:35:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.849+0000 7f6a617d3740 10 client.0 _ll_register_callbacks cb 0x5604e7f3cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.849+0000 7f6a5c859640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82c4330 0x5604e82cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.849+0000 7f6a5c859640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82c4330 0x5604e82cc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53920/0 (socket says 172.21.15.107:53920) 2024-05-09T08:35:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.849+0000 7f6a5c859640 1 -- 172.21.15.107:0/2042683506 learned_addr learned my addr 172.21.15.107:0/2042683506 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.849+0000 7f6a5d05a640 1 --2- 172.21.15.107:0/2042683506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82cf320 0x5604e82ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.849+0000 7f6a57fff640 1 --2- 172.21.15.107:0/2042683506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4c50 0x5604e82cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.850+0000 7f6a5c859640 1 -- 172.21.15.107:0/2042683506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82cf320 msgr2=0x5604e82ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.850+0000 7f6a5c859640 1 --2- 172.21.15.107:0/2042683506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82cf320 0x5604e82ccc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.850+0000 7f6a5c859640 1 -- 172.21.15.107:0/2042683506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4c50 msgr2=0x5604e82cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.850+0000 7f6a5c859640 1 --2- 172.21.15.107:0/2042683506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4c50 0x5604e82cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.850+0000 7f6a5c859640 1 -- 172.21.15.107:0/2042683506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604e80077d0 con 0x5604e82c4330 2024-05-09T08:35:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.850+0000 7f6a5c859640 1 --2- 172.21.15.107:0/2042683506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82c4330 0x5604e82cc1e0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c013c10 tx=0x7f6a4c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b0ef44e398e6ae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.850+0000 7f6a37fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.850+0000 7f6a37fff640 1 -- 172.21.15.107:0/2042683506 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c017050 con 0x5604e82c4330 2024-05-09T08:35:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.850+0000 7f6a37fff640 1 -- 172.21.15.107:0/2042683506 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6a4c01c450 con 0x5604e82c4330 2024-05-09T08:35:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.850+0000 7f6a617d3740 1 -- 172.21.15.107:0/2042683506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a4c013470 con 0x5604e82c4330 2024-05-09T08:35:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.850+0000 7f6a37fff640 1 -- 172.21.15.107:0/2042683506 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6a4c006d60 con 0x5604e82c4330 2024-05-09T08:35:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.850+0000 7f6a617d3740 10 client.16185 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.850+0000 7f6a617d3740 1 -- 172.21.15.107:0/2042683506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5604e82de270 con 0x5604e82c4330 2024-05-09T08:35:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.850+0000 7f6a37fff640 1 -- 172.21.15.107:0/2042683506 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a4c013470 con 0x5604e82c4330 2024-05-09T08:35:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.851+0000 7f6a37fff640 1 -- 172.21.15.107:0/2042683506 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f6a4c026710 con 0x5604e82c4330 2024-05-09T08:35:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.851+0000 7f6a617d3740 10 client.16185 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.851+0000 7f6a617d3740 10 client.16185 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.851+0000 7f6a617d3740 20 client.16185 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.851+0000 7f6a617d3740 1 --2- 172.21.15.107:0/2042683506 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5604e82df6d0 0x5604e82e1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.851+0000 7f6a617d3740 4 client.16185 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.851+0000 7f6a617d3740 1 -- 172.21.15.107:0/2042683506 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604e8237370 con 0x5604e82df6d0 2024-05-09T08:35:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.851+0000 7f6a57fff640 1 --2- 172.21.15.107:0/2042683506 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5604e82df6d0 0x5604e82e1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.851+0000 7f6a37fff640 10 client.16185 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.851+0000 7f6a57fff640 1 --2- 172.21.15.107:0/2042683506 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5604e82df6d0 0x5604e82e1ac0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18fcd5a71a6a928e server_cookie=a8e43df59db143e5 in_seq=0 out_seq=0 2024-05-09T08:35:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.852+0000 7f6a37fff640 1 -- 172.21.15.107:0/2042683506 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5604e8237370 con 0x5604e82df6d0 2024-05-09T08:35:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.852+0000 7f6a37fff640 10 client.16185 handle_command_reply: tid=0 2024-05-09T08:35:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.852+0000 7f6a617d3740 1 client.16185 shutdown 2024-05-09T08:35:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.852+0000 7f6a617d3740 1 -- 172.21.15.107:0/2042683506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82c4330 msgr2=0x5604e82cc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.852+0000 7f6a617d3740 1 --2- 172.21.15.107:0/2042683506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82c4330 0x5604e82cc1e0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c013c10 tx=0x7f6a4c00edf0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.852+0000 7f6a617d3740 1 -- 172.21.15.107:0/2042683506 shutdown_connections 2024-05-09T08:35:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.852+0000 7f6a617d3740 1 --2- 172.21.15.107:0/2042683506 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5604e82df6d0 0x5604e82e1ac0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.852+0000 7f6a617d3740 1 --2- 172.21.15.107:0/2042683506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82cf320 0x5604e82ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.852+0000 7f6a617d3740 1 --2- 172.21.15.107:0/2042683506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4c50 0x5604e82cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.852+0000 7f6a37fff640 0 client.16185 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.852+0000 7f6a617d3740 1 --2- 172.21.15.107:0/2042683506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82c4330 0x5604e82cc1e0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.852+0000 7f6a617d3740 1 -- 172.21.15.107:0/2042683506 >> 172.21.15.107:0/2042683506 conn(0x5604e82c0650 msgr2=0x5604e82c6450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.852+0000 7f6a617d3740 1 -- 172.21.15.107:0/2042683506 shutdown_connections 2024-05-09T08:35:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.852+0000 7f6a617d3740 1 -- 172.21.15.107:0/2042683506 wait complete. 2024-05-09T08:35:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.852+0000 7f6a617d3740 20 client.16185 trim_cache size 0 max 16384 2024-05-09T08:35:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a617d3740 1 Processor -- start 2024-05-09T08:35:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a617d3740 1 -- start start 2024-05-09T08:35:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a617d3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4330 0x5604e82ce910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a617d3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82c4c50 0x5604e82db000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a617d3740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82cf320 0x5604e82bf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a617d3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6a581b14d0 con 0x5604e82c4c50 2024-05-09T08:35:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a617d3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6a581aeda0 con 0x5604e82c4330 2024-05-09T08:35:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a617d3740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6a581acbf0 con 0x5604e82cf320 2024-05-09T08:35:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82c4c50 0x5604e82db000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82c4c50 0x5604e82db000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53930/0 (socket says 172.21.15.107:53930) 2024-05-09T08:35:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a57fff640 1 -- 172.21.15.107:0/1966200559 learned_addr learned my addr 172.21.15.107:0/1966200559 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a5c859640 1 --2- 172.21.15.107:0/1966200559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4330 0x5604e82ce910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a5d05a640 1 --2- 172.21.15.107:0/1966200559 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82cf320 0x5604e82bf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a57fff640 1 -- 172.21.15.107:0/1966200559 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82cf320 msgr2=0x5604e82bf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a57fff640 1 --2- 172.21.15.107:0/1966200559 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82cf320 0x5604e82bf630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a57fff640 1 -- 172.21.15.107:0/1966200559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4330 msgr2=0x5604e82ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a57fff640 1 --2- 172.21.15.107:0/1966200559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4330 0x5604e82ce910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.869+0000 7f6a57fff640 1 -- 172.21.15.107:0/1966200559 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604e82c0480 con 0x5604e82c4c50 2024-05-09T08:35:04.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.870+0000 7f6a57fff640 1 --2- 172.21.15.107:0/1966200559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82c4c50 0x5604e82db000 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f6a440bb950 tx=0x7f6a440f35d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acdd3323291e6f9f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.870+0000 7f6a427fc640 1 -- 172.21.15.107:0/1966200559 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a44048070 con 0x5604e82c4c50 2024-05-09T08:35:04.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.870+0000 7f6a427fc640 1 -- 172.21.15.107:0/1966200559 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a440bc350 con 0x5604e82c4c50 2024-05-09T08:35:04.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.870+0000 7f6a427fc640 1 -- 172.21.15.107:0/1966200559 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a440f7040 con 0x5604e82c4c50 2024-05-09T08:35:04.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:04 smithi110 ceph-mon[33622]: pgmap v54: 129 pgs: 129 active+clean; 91 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 7.4 MiB/s wr, 275 op/s 2024-05-09T08:35:04.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:04 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1576158327' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:35:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.871+0000 7f6a617d3740 1 -- 172.21.15.107:0/1966200559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82c4c50 msgr2=0x5604e82db000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.871+0000 7f6a617d3740 1 --2- 172.21.15.107:0/1966200559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82c4c50 0x5604e82db000 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f6a440bb950 tx=0x7f6a440f35d0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.871+0000 7f6a617d3740 1 -- 172.21.15.107:0/1966200559 shutdown_connections 2024-05-09T08:35:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.871+0000 7f6a617d3740 1 --2- 172.21.15.107:0/1966200559 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82cf320 0x5604e82bf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.871+0000 7f6a617d3740 1 --2- 172.21.15.107:0/1966200559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82c4c50 0x5604e82db000 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.871+0000 7f6a617d3740 1 --2- 172.21.15.107:0/1966200559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4330 0x5604e82ce910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.871+0000 7f6a617d3740 1 -- 172.21.15.107:0/1966200559 >> 172.21.15.107:0/1966200559 conn(0x5604e82c0650 msgr2=0x5604e82bef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.871+0000 7f6a617d3740 1 -- 172.21.15.107:0/1966200559 shutdown_connections 2024-05-09T08:35:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.871+0000 7f6a617d3740 1 -- 172.21.15.107:0/1966200559 wait complete. 2024-05-09T08:35:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.871+0000 7f6a617d3740 1 -- start start 2024-05-09T08:35:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.871+0000 7f6a617d3740 1 Processor -- start 2024-05-09T08:35:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a617d3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4330 0x5604e82d30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a617d3740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82c4c50 0x5604e82d3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a617d3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82cf320 0x5604e82d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a617d3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6a581ad260 con 0x5604e82cf320 2024-05-09T08:35:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a617d3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6a581ae6d0 con 0x5604e82c4330 2024-05-09T08:35:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a617d3740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6a581ad930 con 0x5604e82c4c50 2024-05-09T08:35:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a617d3740 10 client.0 _ll_register_callbacks cb 0x7f6a440f34a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a5c859640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4330 0x5604e82d30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a57fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82c4c50 0x5604e82d3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a5c859640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4330 0x5604e82d30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:33334/0 (socket says 172.21.15.107:33334) 2024-05-09T08:35:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a5c859640 1 -- 172.21.15.107:0/654499186 learned_addr learned my addr 172.21.15.107:0/654499186 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a5d05a640 1 --2- 172.21.15.107:0/654499186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82cf320 0x5604e82d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a57fff640 1 -- 172.21.15.107:0/654499186 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4330 msgr2=0x5604e82d30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a57fff640 1 --2- 172.21.15.107:0/654499186 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4330 0x5604e82d30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a57fff640 1 -- 172.21.15.107:0/654499186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82cf320 msgr2=0x5604e82d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a57fff640 1 --2- 172.21.15.107:0/654499186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82cf320 0x5604e82d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a57fff640 1 -- 172.21.15.107:0/654499186 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604e8344be0 con 0x5604e82c4c50 2024-05-09T08:35:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a5d05a640 1 --2- 172.21.15.107:0/654499186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82cf320 0x5604e82d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a57fff640 1 --2- 172.21.15.107:0/654499186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82c4c50 0x5604e82d3630 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6a440bc500 tx=0x7f6a44009880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c722d177e1440fcb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a37fff640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:35:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a37fff640 1 -- 172.21.15.107:0/654499186 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a440d1060 con 0x5604e82c4c50 2024-05-09T08:35:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.872+0000 7f6a37fff640 1 -- 172.21.15.107:0/654499186 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6a4403b940 con 0x5604e82c4c50 2024-05-09T08:35:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.873+0000 7f6a617d3740 1 -- 172.21.15.107:0/654499186 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a440bc350 con 0x5604e82c4c50 2024-05-09T08:35:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.873+0000 7f6a37fff640 1 -- 172.21.15.107:0/654499186 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6a44048070 con 0x5604e82c4c50 2024-05-09T08:35:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.873+0000 7f6a617d3740 10 client.25666 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.873+0000 7f6a617d3740 1 -- 172.21.15.107:0/654499186 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5604e8345ea0 con 0x5604e82c4c50 2024-05-09T08:35:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.873+0000 7f6a37fff640 1 -- 172.21.15.107:0/654499186 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a440bc350 con 0x5604e82c4c50 2024-05-09T08:35:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.873+0000 7f6a37fff640 1 -- 172.21.15.107:0/654499186 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f6a4404b060 con 0x5604e82c4c50 2024-05-09T08:35:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.873+0000 7f6a617d3740 10 client.25666 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.873+0000 7f6a617d3740 10 client.25666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.873+0000 7f6a617d3740 20 client.25666 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.873+0000 7f6a617d3740 1 --2- 172.21.15.107:0/654499186 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5604e8346670 0x5604e8348a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.873+0000 7f6a617d3740 4 client.25666 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:35:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.873+0000 7f6a617d3740 1 -- 172.21.15.107:0/654499186 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5604e82316c0 con 0x5604e8346670 2024-05-09T08:35:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.873+0000 7f6a5c859640 1 --2- 172.21.15.107:0/654499186 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5604e8346670 0x5604e8348a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a37fff640 10 client.25666 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a5c859640 1 --2- 172.21.15.107:0/654499186 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5604e8346670 0x5604e8348a90 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac35d310d29373c0 server_cookie=e01433541c1e8a44 in_seq=0 out_seq=0 2024-05-09T08:35:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a37fff640 1 -- 172.21.15.107:0/654499186 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5604e82316c0 con 0x5604e8346670 2024-05-09T08:35:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a37fff640 10 client.25666 handle_command_reply: tid=0 2024-05-09T08:35:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a617d3740 1 client.25666 shutdown 2024-05-09T08:35:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a617d3740 1 -- 172.21.15.107:0/654499186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82c4c50 msgr2=0x5604e82d3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a617d3740 1 --2- 172.21.15.107:0/654499186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82c4c50 0x5604e82d3630 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6a440bc500 tx=0x7f6a44009880 comp rx=0 tx=0).stop 2024-05-09T08:35:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a617d3740 1 -- 172.21.15.107:0/654499186 shutdown_connections 2024-05-09T08:35:04.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a617d3740 1 --2- 172.21.15.107:0/654499186 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5604e8346670 0x5604e8348a90 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a617d3740 1 --2- 172.21.15.107:0/654499186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604e82cf320 0x5604e82d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a617d3740 1 --2- 172.21.15.107:0/654499186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604e82c4c50 0x5604e82d3630 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a37fff640 0 client.25666 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:04.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a617d3740 1 --2- 172.21.15.107:0/654499186 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5604e82c4330 0x5604e82d30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a617d3740 1 -- 172.21.15.107:0/654499186 >> 172.21.15.107:0/654499186 conn(0x5604e82c0650 msgr2=0x5604e82ca670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a617d3740 1 -- 172.21.15.107:0/654499186 shutdown_connections 2024-05-09T08:35:04.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a617d3740 1 -- 172.21.15.107:0/654499186 wait complete. 2024-05-09T08:35:04.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.874+0000 7f6a617d3740 20 client.25666 trim_cache size 0 max 16384 2024-05-09T08:35:04.879 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:35:04.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.875+0000 7f6a5eabd640 1 -- 172.21.15.107:0/3168581750 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6a480a2d00 msgr2=0x7f6a480a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.875+0000 7f6a5eabd640 1 --2- 172.21.15.107:0/3168581750 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6a480a2d00 0x7f6a480a51c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6a50005fd0 tx=0x7f6a50004ab0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.875+0000 7f6a5eabd640 1 -- 172.21.15.107:0/3168581750 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a58102240 msgr2=0x7f6a581986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:04.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.875+0000 7f6a5eabd640 1 --2- 172.21.15.107:0/3168581750 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a58102240 0x7f6a581986d0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f6a44004770 tx=0x7f6a440042f0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.875+0000 7f6a5eabd640 1 -- 172.21.15.107:0/3168581750 shutdown_connections 2024-05-09T08:35:04.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.875+0000 7f6a5eabd640 1 --2- 172.21.15.107:0/3168581750 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6a480a2d00 0x7f6a480a51c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.875+0000 7f6a5eabd640 1 --2- 172.21.15.107:0/3168581750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a58107890 0x7f6a5819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.875+0000 7f6a5eabd640 1 --2- 172.21.15.107:0/3168581750 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a58102240 0x7f6a581986d0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.875+0000 7f6a5eabd640 1 --2- 172.21.15.107:0/3168581750 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a58101880 0x7f6a58198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:04.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.875+0000 7f6a5eabd640 1 -- 172.21.15.107:0/3168581750 >> 172.21.15.107:0/3168581750 conn(0x7f6a580fd390 msgr2=0x7f6a580fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:04.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.875+0000 7f6a5eabd640 1 -- 172.21.15.107:0/3168581750 shutdown_connections 2024-05-09T08:35:04.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:04.875+0000 7f6a5eabd640 1 -- 172.21.15.107:0/3168581750 wait complete. 2024-05-09T08:35:05.885 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:35:05.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:35:05.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:35:06.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.003+0000 7fcf4c038640 1 -- 172.21.15.107:0/1029850110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf44103430 msgr2=0x7fcf4410b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.003+0000 7fcf4c038640 1 --2- 172.21.15.107:0/1029850110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf44103430 0x7fcf4410b910 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009900 tx=0x7fcf3402a480 comp rx=0 tx=0).stop 2024-05-09T08:35:06.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.003+0000 7fcf4c038640 1 -- 172.21.15.107:0/1029850110 shutdown_connections 2024-05-09T08:35:06.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.003+0000 7fcf4c038640 1 --2- 172.21.15.107:0/1029850110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf4410bee0 0x7fcf4410e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.003+0000 7fcf4c038640 1 --2- 172.21.15.107:0/1029850110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf44103430 0x7fcf4410b910 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.003+0000 7fcf4c038640 1 --2- 172.21.15.107:0/1029850110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf44102b10 0x7fcf44102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.003+0000 7fcf4c038640 1 -- 172.21.15.107:0/1029850110 >> 172.21.15.107:0/1029850110 conn(0x7fcf44077860 msgr2=0x7fcf44077c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:06.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.003+0000 7fcf4c038640 1 -- 172.21.15.107:0/1029850110 shutdown_connections 2024-05-09T08:35:06.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.003+0000 7fcf4c038640 1 -- 172.21.15.107:0/1029850110 wait complete. 2024-05-09T08:35:06.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf4c038640 1 Processor -- start 2024-05-09T08:35:06.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf4c038640 1 -- start start 2024-05-09T08:35:06.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf4c038640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf44102b10 0x7fcf44198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf4c038640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf44103430 0x7fcf441986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf4c038640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf4410bee0 0x7fcf4419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf4c038640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcf44111b20 con 0x7fcf4410bee0 2024-05-09T08:35:06.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf4c038640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcf441119a0 con 0x7fcf44103430 2024-05-09T08:35:06.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf4c038640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcf44111ca0 con 0x7fcf44102b10 2024-05-09T08:35:06.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf4a5d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf4410bee0 0x7fcf4419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf4a5d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf4410bee0 0x7fcf4419ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53942/0 (socket says 172.21.15.107:53942) 2024-05-09T08:35:06.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf4a5d5640 1 -- 172.21.15.107:0/861385331 learned_addr learned my addr 172.21.15.107:0/861385331 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:06.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf495d3640 1 --2- 172.21.15.107:0/861385331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf44103430 0x7fcf441986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf49dd4640 1 --2- 172.21.15.107:0/861385331 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf44102b10 0x7fcf44198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf4a5d5640 1 -- 172.21.15.107:0/861385331 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf44102b10 msgr2=0x7fcf44198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf4a5d5640 1 --2- 172.21.15.107:0/861385331 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf44102b10 0x7fcf44198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf4a5d5640 1 -- 172.21.15.107:0/861385331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf44103430 msgr2=0x7fcf441986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf4a5d5640 1 --2- 172.21.15.107:0/861385331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf44103430 0x7fcf441986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.004+0000 7fcf4a5d5640 1 -- 172.21.15.107:0/861385331 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf4419d180 con 0x7fcf4410bee0 2024-05-09T08:35:06.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.005+0000 7fcf4a5d5640 1 --2- 172.21.15.107:0/861385331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf4410bee0 0x7fcf4419ca50 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00e9d0 tx=0x7fcf3c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:06.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.005+0000 7fcf3affd640 1 -- 172.21.15.107:0/861385331 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c00cd80 con 0x7fcf4410bee0 2024-05-09T08:35:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.005+0000 7fcf3affd640 1 -- 172.21.15.107:0/861385331 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf3c005020 con 0x7fcf4410bee0 2024-05-09T08:35:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.005+0000 7fcf4c038640 1 -- 172.21.15.107:0/861385331 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf4419d470 con 0x7fcf4410bee0 2024-05-09T08:35:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.005+0000 7fcf3affd640 1 -- 172.21.15.107:0/861385331 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c0116d0 con 0x7fcf4410bee0 2024-05-09T08:35:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.005+0000 7fcf4c038640 1 -- 172.21.15.107:0/861385331 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf441a4d00 con 0x7fcf4410bee0 2024-05-09T08:35:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.007+0000 7fcf4ee21740 1 Processor -- start 2024-05-09T08:35:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.007+0000 7fcf3affd640 1 -- 172.21.15.107:0/861385331 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fcf3c0051d0 con 0x7fcf4410bee0 2024-05-09T08:35:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.008+0000 7fcf4ee21740 1 -- start start 2024-05-09T08:35:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.008+0000 7fcf4ee21740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26bc0b0 0x559ea26ba3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.008+0000 7fcf4ee21740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26ba9c0 0x559ea26bae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.008+0000 7fcf4ee21740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26c5740 0x559ea26c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.008+0000 7fcf4ee21740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559ea261f990 con 0x559ea26bc0b0 2024-05-09T08:35:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.008+0000 7fcf4ee21740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559ea25c75b0 con 0x559ea26ba9c0 2024-05-09T08:35:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.008+0000 7fcf4ee21740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x559ea261b950 con 0x559ea26c5740 2024-05-09T08:35:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.008+0000 7fcf495d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26bc0b0 0x559ea26ba3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.008+0000 7fcf495d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26bc0b0 0x559ea26ba3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53950/0 (socket says 172.21.15.107:53950) 2024-05-09T08:35:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.008+0000 7fcf495d3640 1 -- 172.21.15.107:0/2387774472 learned_addr learned my addr 172.21.15.107:0/2387774472 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.008+0000 7fcf49dd4640 1 --2- 172.21.15.107:0/2387774472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26ba9c0 0x559ea26bae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.008+0000 7fcf495d3640 1 --2- 172.21.15.107:0/2387774472 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26c5740 0x559ea26c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.009+0000 7fcf3affd640 1 --2- 172.21.15.107:0/861385331 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcf300a2d00 0x7fcf300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.009+0000 7fcf495d3640 1 -- 172.21.15.107:0/2387774472 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26c5740 msgr2=0x559ea26c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.009+0000 7fcf495d3640 1 --2- 172.21.15.107:0/2387774472 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26c5740 0x559ea26c7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.009+0000 7fcf495d3640 1 -- 172.21.15.107:0/2387774472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26ba9c0 msgr2=0x559ea26bae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.009+0000 7fcf495d3640 1 --2- 172.21.15.107:0/2387774472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26ba9c0 0x559ea26bae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.009+0000 7fcf495d3640 1 -- 172.21.15.107:0/2387774472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ea23fb7e0 con 0x559ea26bc0b0 2024-05-09T08:35:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.009+0000 7fcf4a5d5640 1 --2- 172.21.15.107:0/861385331 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcf300a2d00 0x7fcf300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.009+0000 7fcf495d3640 1 --2- 172.21.15.107:0/2387774472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26bc0b0 0x559ea26ba3f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009900 tx=0x7fcf34005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd20991259fbaf1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.009+0000 7fcf3affd640 1 -- 172.21.15.107:0/861385331 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fcf3c0cf890 con 0x7fcf4410bee0 2024-05-09T08:35:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.009+0000 7fcf2ffff640 1 -- 172.21.15.107:0/2387774472 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf34044020 con 0x559ea26bc0b0 2024-05-09T08:35:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.009+0000 7fcf2ffff640 1 -- 172.21.15.107:0/2387774472 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf340317e0 con 0x559ea26bc0b0 2024-05-09T08:35:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.009+0000 7fcf2ffff640 1 -- 172.21.15.107:0/2387774472 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf34035040 con 0x559ea26bc0b0 2024-05-09T08:35:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.010+0000 7fcf4ee21740 1 -- 172.21.15.107:0/2387774472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26bc0b0 msgr2=0x559ea26ba3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.010+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/2387774472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26bc0b0 0x559ea26ba3f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009900 tx=0x7fcf34005c00 comp rx=0 tx=0).stop 2024-05-09T08:35:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.010+0000 7fcf4a5d5640 1 --2- 172.21.15.107:0/861385331 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcf300a2d00 0x7fcf300a51c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c0cd280 tx=0x7fcf3c086040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.010+0000 7fcf4ee21740 1 -- 172.21.15.107:0/2387774472 shutdown_connections 2024-05-09T08:35:06.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.010+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/2387774472 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26c5740 0x559ea26c7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.010+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/2387774472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26ba9c0 0x559ea26bae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.010+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/2387774472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26bc0b0 0x559ea26ba3f0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.010+0000 7fcf4ee21740 1 -- 172.21.15.107:0/2387774472 >> 172.21.15.107:0/2387774472 conn(0x559ea26830a0 msgr2=0x559ea26b6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:06.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.010+0000 7fcf4ee21740 1 -- 172.21.15.107:0/2387774472 shutdown_connections 2024-05-09T08:35:06.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.010+0000 7fcf4ee21740 1 -- 172.21.15.107:0/2387774472 wait complete. 2024-05-09T08:35:06.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.010+0000 7fcf4ee21740 1 -- start start 2024-05-09T08:35:06.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.011+0000 7fcf4ee21740 1 Processor -- start 2024-05-09T08:35:06.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.011+0000 7fcf4ee21740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26ba9c0 0x559ea26c2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.011+0000 7fcf4ee21740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26bc0b0 0x559ea26c2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.011+0000 7fcf4ee21740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26c5740 0x559ea26c30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.011+0000 7fcf4ee21740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559ea26c8570 con 0x559ea26ba9c0 2024-05-09T08:35:06.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.011+0000 7fcf4ee21740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559ea25dab40 con 0x559ea26c5740 2024-05-09T08:35:06.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.011+0000 7fcf4ee21740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x559ea26c86f0 con 0x559ea26bc0b0 2024-05-09T08:35:06.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.011+0000 7fcf4ee21740 10 client.0 _ll_register_callbacks cb 0x559ea25cf0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:06.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.011+0000 7fcf495d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26ba9c0 0x559ea26c2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.011+0000 7fcf495d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26ba9c0 0x559ea26c2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53958/0 (socket says 172.21.15.107:53958) 2024-05-09T08:35:06.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.011+0000 7fcf495d3640 1 -- 172.21.15.107:0/4102886731 learned_addr learned my addr 172.21.15.107:0/4102886731 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:06.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.011+0000 7fcf49dd4640 1 --2- 172.21.15.107:0/4102886731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26bc0b0 0x559ea26c2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.011+0000 7fcf495d3640 1 --2- 172.21.15.107:0/4102886731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26c5740 0x559ea26c30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.012+0000 7fcf495d3640 1 -- 172.21.15.107:0/4102886731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26bc0b0 msgr2=0x559ea26c2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:06.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.012+0000 7fcf495d3640 1 --2- 172.21.15.107:0/4102886731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26bc0b0 0x559ea26c2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.012+0000 7fcf495d3640 1 -- 172.21.15.107:0/4102886731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26c5740 msgr2=0x559ea26c30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:06.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.012+0000 7fcf495d3640 1 --2- 172.21.15.107:0/4102886731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26c5740 0x559ea26c30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.012+0000 7fcf495d3640 1 -- 172.21.15.107:0/4102886731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ea245bf70 con 0x559ea26ba9c0 2024-05-09T08:35:06.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.012+0000 7fcf495d3640 1 --2- 172.21.15.107:0/4102886731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26ba9c0 0x559ea26c2670 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fcf34032fd0 tx=0x7fcf340053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6b39908b77694c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:06.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.012+0000 7fcf2d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:06.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.012+0000 7fcf2d7fa640 1 -- 172.21.15.107:0/4102886731 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf34044070 con 0x559ea26ba9c0 2024-05-09T08:35:06.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.012+0000 7fcf2d7fa640 1 -- 172.21.15.107:0/4102886731 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fcf34004450 con 0x559ea26ba9c0 2024-05-09T08:35:06.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.012+0000 7fcf4ee21740 1 -- 172.21.15.107:0/4102886731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf340317e0 con 0x559ea26ba9c0 2024-05-09T08:35:06.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.012+0000 7fcf2d7fa640 1 -- 172.21.15.107:0/4102886731 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcf34035040 con 0x559ea26ba9c0 2024-05-09T08:35:06.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.012+0000 7fcf4ee21740 10 client.16203 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:06.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.012+0000 7fcf4ee21740 1 -- 172.21.15.107:0/4102886731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559ea22f3170 con 0x559ea26ba9c0 2024-05-09T08:35:06.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.012+0000 7fcf2d7fa640 1 -- 172.21.15.107:0/4102886731 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf340317e0 con 0x559ea26ba9c0 2024-05-09T08:35:06.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.012+0000 7fcf2d7fa640 1 -- 172.21.15.107:0/4102886731 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fcf3404a4a0 con 0x559ea26ba9c0 2024-05-09T08:35:06.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.013+0000 7fcf4ee21740 10 client.16203 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:06.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.013+0000 7fcf4ee21740 10 client.16203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:06.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.013+0000 7fcf4ee21740 20 client.16203 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:06.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.013+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/4102886731 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x559ea26d56a0 0x559ea26d7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.013+0000 7fcf4ee21740 4 client.16203 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:06.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.013+0000 7fcf4ee21740 1 -- 172.21.15.107:0/4102886731 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ea262d0c0 con 0x559ea26d56a0 2024-05-09T08:35:06.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.013+0000 7fcf49dd4640 1 --2- 172.21.15.107:0/4102886731 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x559ea26d56a0 0x559ea26d7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.013+0000 7fcf2d7fa640 10 client.16203 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:06.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.013+0000 7fcf49dd4640 1 --2- 172.21.15.107:0/4102886731 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x559ea26d56a0 0x559ea26d7a90 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23986f594b0e766 server_cookie=f1eadad31be10446 in_seq=0 out_seq=0 2024-05-09T08:35:06.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.014+0000 7fcf2d7fa640 1 -- 172.21.15.107:0/4102886731 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559ea262d0c0 con 0x559ea26d56a0 2024-05-09T08:35:06.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.014+0000 7fcf2d7fa640 10 client.16203 handle_command_reply: tid=0 2024-05-09T08:35:06.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.014+0000 7fcf4ee21740 1 client.16203 shutdown 2024-05-09T08:35:06.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.014+0000 7fcf4ee21740 1 -- 172.21.15.107:0/4102886731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26ba9c0 msgr2=0x559ea26c2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.014+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/4102886731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26ba9c0 0x559ea26c2670 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fcf34032fd0 tx=0x7fcf340053f0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.014+0000 7fcf4ee21740 1 -- 172.21.15.107:0/4102886731 shutdown_connections 2024-05-09T08:35:06.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.014+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/4102886731 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x559ea26d56a0 0x559ea26d7a90 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.014+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/4102886731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26c5740 0x559ea26c30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.015+0000 7fcf2d7fa640 0 client.16203 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:06.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.015+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/4102886731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26bc0b0 0x559ea26c2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.015+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/4102886731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26ba9c0 0x559ea26c2670 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.015+0000 7fcf4ee21740 1 -- 172.21.15.107:0/4102886731 >> 172.21.15.107:0/4102886731 conn(0x559ea26830a0 msgr2=0x559ea26bcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:06.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.015+0000 7fcf4ee21740 1 -- 172.21.15.107:0/4102886731 shutdown_connections 2024-05-09T08:35:06.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.015+0000 7fcf4ee21740 1 -- 172.21.15.107:0/4102886731 wait complete. 2024-05-09T08:35:06.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.015+0000 7fcf4ee21740 20 client.16203 trim_cache size 0 max 16384 2024-05-09T08:35:06.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.032+0000 7fcf4ee21740 1 Processor -- start 2024-05-09T08:35:06.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.032+0000 7fcf4ee21740 1 -- start start 2024-05-09T08:35:06.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.032+0000 7fcf4ee21740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26ba9c0 0x559ea26b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.032+0000 7fcf4ee21740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26bc0b0 0x559ea26bcba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf4ee21740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26c5740 0x559ea26bf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf4ee21740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcf441b13a0 con 0x559ea26bc0b0 2024-05-09T08:35:06.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf4ee21740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcf441aec70 con 0x559ea26c5740 2024-05-09T08:35:06.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf4ee21740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcf441acac0 con 0x559ea26ba9c0 2024-05-09T08:35:06.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf49dd4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26bc0b0 0x559ea26bcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf495d3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26ba9c0 0x559ea26b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf49dd4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26bc0b0 0x559ea26bcba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53966/0 (socket says 172.21.15.107:53966) 2024-05-09T08:35:06.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf49dd4640 1 -- 172.21.15.107:0/514789974 learned_addr learned my addr 172.21.15.107:0/514789974 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:06.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf495d3640 1 --2- 172.21.15.107:0/514789974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26c5740 0x559ea26bf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf49dd4640 1 -- 172.21.15.107:0/514789974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26ba9c0 msgr2=0x559ea26b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:06.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf49dd4640 1 --2- 172.21.15.107:0/514789974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26ba9c0 0x559ea26b5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf49dd4640 1 -- 172.21.15.107:0/514789974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26c5740 msgr2=0x559ea26bf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:06.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf49dd4640 1 --2- 172.21.15.107:0/514789974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26c5740 0x559ea26bf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf49dd4640 1 -- 172.21.15.107:0/514789974 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ea26b7130 con 0x559ea26bc0b0 2024-05-09T08:35:06.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf495d3640 1 --2- 172.21.15.107:0/514789974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26ba9c0 0x559ea26b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:06.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf49dd4640 1 --2- 172.21.15.107:0/514789974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26bc0b0 0x559ea26bcba0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fcf40004580 tx=0x7fcf4000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc1fa02f301fbc74 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:06.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf2f7fe640 1 -- 172.21.15.107:0/514789974 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf40003df0 con 0x559ea26bc0b0 2024-05-09T08:35:06.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf2f7fe640 1 -- 172.21.15.107:0/514789974 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf400054d0 con 0x559ea26bc0b0 2024-05-09T08:35:06.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.033+0000 7fcf2f7fe640 1 -- 172.21.15.107:0/514789974 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf40020160 con 0x559ea26bc0b0 2024-05-09T08:35:06.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.034+0000 7fcf4ee21740 1 -- 172.21.15.107:0/514789974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26bc0b0 msgr2=0x559ea26bcba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.034+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/514789974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26bc0b0 0x559ea26bcba0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fcf40004580 tx=0x7fcf4000b390 comp rx=0 tx=0).stop 2024-05-09T08:35:06.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.034+0000 7fcf4ee21740 1 -- 172.21.15.107:0/514789974 shutdown_connections 2024-05-09T08:35:06.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.034+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/514789974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26c5740 0x559ea26bf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.034+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/514789974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26bc0b0 0x559ea26bcba0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.034+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/514789974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26ba9c0 0x559ea26b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.034+0000 7fcf4ee21740 1 -- 172.21.15.107:0/514789974 >> 172.21.15.107:0/514789974 conn(0x559ea26830a0 msgr2=0x559ea26bc490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.034+0000 7fcf4ee21740 1 -- 172.21.15.107:0/514789974 shutdown_connections 2024-05-09T08:35:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.034+0000 7fcf4ee21740 1 -- 172.21.15.107:0/514789974 wait complete. 2024-05-09T08:35:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.035+0000 7fcf4ee21740 1 -- start start 2024-05-09T08:35:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.035+0000 7fcf4ee21740 1 Processor -- start 2024-05-09T08:35:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.035+0000 7fcf4ee21740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26ba9c0 0x559ea26daed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.035+0000 7fcf4ee21740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26bc0b0 0x559ea26d9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.035+0000 7fcf4ee21740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26c5740 0x559ea26d9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.035+0000 7fcf4ee21740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcf441ad130 con 0x559ea26ba9c0 2024-05-09T08:35:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.035+0000 7fcf4ee21740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcf441ae5a0 con 0x559ea26c5740 2024-05-09T08:35:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.035+0000 7fcf4ee21740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcf441ad800 con 0x559ea26bc0b0 2024-05-09T08:35:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.035+0000 7fcf4ee21740 10 client.0 _ll_register_callbacks cb 0x7fcf4000b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.035+0000 7fcf495d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26ba9c0 0x559ea26daed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.035+0000 7fcf495d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26ba9c0 0x559ea26daed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53978/0 (socket says 172.21.15.107:53978) 2024-05-09T08:35:06.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.035+0000 7fcf495d3640 1 -- 172.21.15.107:0/3411424387 learned_addr learned my addr 172.21.15.107:0/3411424387 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:06.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.035+0000 7fcf49dd4640 1 --2- 172.21.15.107:0/3411424387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26bc0b0 0x559ea26d9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.035+0000 7fcf495d3640 1 --2- 172.21.15.107:0/3411424387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26c5740 0x559ea26d9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf495d3640 1 -- 172.21.15.107:0/3411424387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26bc0b0 msgr2=0x559ea26d9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:06.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf495d3640 1 --2- 172.21.15.107:0/3411424387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26bc0b0 0x559ea26d9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf495d3640 1 -- 172.21.15.107:0/3411424387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26c5740 msgr2=0x559ea26d9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:06.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf495d3640 1 --2- 172.21.15.107:0/3411424387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26c5740 0x559ea26d9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf495d3640 1 -- 172.21.15.107:0/3411424387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ea26da040 con 0x559ea26ba9c0 2024-05-09T08:35:06.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf495d3640 1 --2- 172.21.15.107:0/3411424387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26ba9c0 0x559ea26daed0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009900 tx=0x7fcf340053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd21e92d462322f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:06.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf2d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:06.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf2d7fa640 1 -- 172.21.15.107:0/3411424387 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3405d040 con 0x559ea26ba9c0 2024-05-09T08:35:06.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf4ee21740 1 -- 172.21.15.107:0/3411424387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf400054d0 con 0x559ea26ba9c0 2024-05-09T08:35:06.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf2d7fa640 1 -- 172.21.15.107:0/3411424387 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fcf34032be0 con 0x559ea26ba9c0 2024-05-09T08:35:06.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf2d7fa640 1 -- 172.21.15.107:0/3411424387 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcf34044040 con 0x559ea26ba9c0 2024-05-09T08:35:06.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf4ee21740 10 client.16209 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:06.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf4ee21740 1 -- 172.21.15.107:0/3411424387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559ea273c9a0 con 0x559ea26ba9c0 2024-05-09T08:35:06.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf2d7fa640 1 -- 172.21.15.107:0/3411424387 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf400054d0 con 0x559ea26ba9c0 2024-05-09T08:35:06.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf2d7fa640 1 -- 172.21.15.107:0/3411424387 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fcf34003aa0 con 0x559ea26ba9c0 2024-05-09T08:35:06.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf4ee21740 10 client.16209 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:06.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf4ee21740 10 client.16209 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:06.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf4ee21740 20 client.16209 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:06.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/3411424387 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x559ea26d56a0 0x559ea273f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf4ee21740 4 client.16209 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:35:06.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.036+0000 7fcf4ee21740 1 -- 172.21.15.107:0/3411424387 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559ea2627410 con 0x559ea26d56a0 2024-05-09T08:35:06.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.037+0000 7fcf49dd4640 1 --2- 172.21.15.107:0/3411424387 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x559ea26d56a0 0x559ea273f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.037+0000 7fcf49dd4640 1 --2- 172.21.15.107:0/3411424387 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x559ea26d56a0 0x559ea273f0a0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c50b4a05bc2790b2 server_cookie=32abc01988f34b25 in_seq=0 out_seq=0 2024-05-09T08:35:06.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.037+0000 7fcf2d7fa640 10 client.16209 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:06.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.038+0000 7fcf2d7fa640 1 -- 172.21.15.107:0/3411424387 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559ea2627410 con 0x559ea26d56a0 2024-05-09T08:35:06.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.038+0000 7fcf2d7fa640 10 client.16209 handle_command_reply: tid=0 2024-05-09T08:35:06.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.038+0000 7fcf4ee21740 1 client.16209 shutdown 2024-05-09T08:35:06.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.038+0000 7fcf4ee21740 1 -- 172.21.15.107:0/3411424387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26ba9c0 msgr2=0x559ea26daed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.038+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/3411424387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26ba9c0 0x559ea26daed0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009900 tx=0x7fcf340053f0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.038+0000 7fcf4ee21740 1 -- 172.21.15.107:0/3411424387 shutdown_connections 2024-05-09T08:35:06.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.038+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/3411424387 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x559ea26d56a0 0x559ea273f0a0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.038+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/3411424387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ea26c5740 0x559ea26d9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.038+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/3411424387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559ea26bc0b0 0x559ea26d9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.038+0000 7fcf4ee21740 1 --2- 172.21.15.107:0/3411424387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ea26ba9c0 0x559ea26daed0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.038+0000 7fcf2d7fa640 0 client.16209 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:06.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.038+0000 7fcf4ee21740 1 -- 172.21.15.107:0/3411424387 >> 172.21.15.107:0/3411424387 conn(0x559ea26830a0 msgr2=0x559ea26bcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:06.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.038+0000 7fcf4ee21740 1 -- 172.21.15.107:0/3411424387 shutdown_connections 2024-05-09T08:35:06.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.038+0000 7fcf4ee21740 1 -- 172.21.15.107:0/3411424387 wait complete. 2024-05-09T08:35:06.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.038+0000 7fcf4ee21740 20 client.16209 trim_cache size 0 max 16384 2024-05-09T08:35:06.050 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:35:06.050 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:35:06.050 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "60ac5c9b-e5fb-4198-afa1-36e5a1b6cbce", 2024-05-09T08:35:06.050 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:35:06.050 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:35:06.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.039+0000 7fcf4c038640 1 -- 172.21.15.107:0/861385331 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcf300a2d00 msgr2=0x7fcf300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.039+0000 7fcf4c038640 1 --2- 172.21.15.107:0/861385331 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcf300a2d00 0x7fcf300a51c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c0cd280 tx=0x7fcf3c086040 comp rx=0 tx=0).stop 2024-05-09T08:35:06.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.039+0000 7fcf4c038640 1 -- 172.21.15.107:0/861385331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf4410bee0 msgr2=0x7fcf4419ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.039+0000 7fcf4c038640 1 --2- 172.21.15.107:0/861385331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf4410bee0 0x7fcf4419ca50 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00e9d0 tx=0x7fcf3c00ee90 comp rx=0 tx=0).stop 2024-05-09T08:35:06.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.039+0000 7fcf4c038640 1 -- 172.21.15.107:0/861385331 shutdown_connections 2024-05-09T08:35:06.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.039+0000 7fcf4c038640 1 --2- 172.21.15.107:0/861385331 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcf300a2d00 0x7fcf300a51c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.039+0000 7fcf4c038640 1 --2- 172.21.15.107:0/861385331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf4410bee0 0x7fcf4419ca50 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.039+0000 7fcf4c038640 1 --2- 172.21.15.107:0/861385331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf44103430 0x7fcf441986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.039+0000 7fcf4c038640 1 --2- 172.21.15.107:0/861385331 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf44102b10 0x7fcf44198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.039+0000 7fcf4c038640 1 -- 172.21.15.107:0/861385331 >> 172.21.15.107:0/861385331 conn(0x7fcf44077860 msgr2=0x7fcf440fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:06.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.039+0000 7fcf4c038640 1 -- 172.21.15.107:0/861385331 shutdown_connections 2024-05-09T08:35:06.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.039+0000 7fcf4c038640 1 -- 172.21.15.107:0/861385331 wait complete. 2024-05-09T08:35:06.053 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:35:06.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:35:06.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.172+0000 7f3ba7790640 1 -- 172.21.15.107:0/2274705498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba0102240 msgr2=0x7f3ba010e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.172+0000 7f3ba7790640 1 --2- 172.21.15.107:0/2274705498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba0102240 0x7f3ba010e750 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f3b98009cd0 tx=0x7f3b9802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.172+0000 7f3ba7790640 1 -- 172.21.15.107:0/2274705498 shutdown_connections 2024-05-09T08:35:06.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.172+0000 7f3ba7790640 1 --2- 172.21.15.107:0/2274705498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba0102240 0x7f3ba010e750 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.172+0000 7f3ba7790640 1 --2- 172.21.15.107:0/2274705498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ba0101880 0x7f3ba0101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.172+0000 7f3ba7790640 1 --2- 172.21.15.107:0/2274705498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ba0107890 0x7f3ba0107c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.172+0000 7f3ba7790640 1 -- 172.21.15.107:0/2274705498 >> 172.21.15.107:0/2274705498 conn(0x7f3ba00fd390 msgr2=0x7f3ba00ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:06.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.173+0000 7f3ba7790640 1 -- 172.21.15.107:0/2274705498 shutdown_connections 2024-05-09T08:35:06.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.173+0000 7f3ba7790640 1 -- 172.21.15.107:0/2274705498 wait complete. 2024-05-09T08:35:06.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.173+0000 7f3ba7790640 1 Processor -- start 2024-05-09T08:35:06.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.173+0000 7f3ba7790640 1 -- start start 2024-05-09T08:35:06.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba7790640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ba0101880 0x7f3ba01980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba7790640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ba0102240 0x7f3ba0198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba7790640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba0107890 0x7f3ba019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba7790640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba0111b50 con 0x7f3ba0107890 2024-05-09T08:35:06.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba7790640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba01119d0 con 0x7f3ba0102240 2024-05-09T08:35:06.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba7790640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba0111cd0 con 0x7f3ba0101880 2024-05-09T08:35:06.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba5d2d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba0107890 0x7f3ba019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba5d2d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba0107890 0x7f3ba019c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53994/0 (socket says 172.21.15.107:53994) 2024-05-09T08:35:06.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba5d2d640 1 -- 172.21.15.107:0/2965725770 learned_addr learned my addr 172.21.15.107:0/2965725770 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:06.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba552c640 1 --2- 172.21.15.107:0/2965725770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ba0101880 0x7f3ba01980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/2965725770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ba0102240 0x7f3ba0198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba5d2d640 1 -- 172.21.15.107:0/2965725770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ba0101880 msgr2=0x7f3ba01980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba5d2d640 1 --2- 172.21.15.107:0/2965725770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ba0101880 0x7f3ba01980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba5d2d640 1 -- 172.21.15.107:0/2965725770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ba0102240 msgr2=0x7f3ba0198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba5d2d640 1 --2- 172.21.15.107:0/2965725770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ba0102240 0x7f3ba0198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba5d2d640 1 -- 172.21.15.107:0/2965725770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba019d090 con 0x7f3ba0107890 2024-05-09T08:35:06.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/2965725770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ba0102240 0x7f3ba0198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:06.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba5d2d640 1 --2- 172.21.15.107:0/2965725770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba0107890 0x7f3ba019c960 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f3b98033040 tx=0x7f3b98003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:06.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3b967fc640 1 -- 172.21.15.107:0/2965725770 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98004040 con 0x7f3ba0107890 2024-05-09T08:35:06.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba7790640 1 -- 172.21.15.107:0/2965725770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ba019d320 con 0x7f3ba0107890 2024-05-09T08:35:06.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3b967fc640 1 -- 172.21.15.107:0/2965725770 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b98004690 con 0x7f3ba0107890 2024-05-09T08:35:06.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3b967fc640 1 -- 172.21.15.107:0/2965725770 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98030e10 con 0x7f3ba0107890 2024-05-09T08:35:06.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.174+0000 7f3ba7790640 1 -- 172.21.15.107:0/2965725770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ba01a4c10 con 0x7f3ba0107890 2024-05-09T08:35:06.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.177+0000 7f3b967fc640 1 -- 172.21.15.107:0/2965725770 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3b980305a0 con 0x7f3ba0107890 2024-05-09T08:35:06.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3baa57a740 1 Processor -- start 2024-05-09T08:35:06.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3baa57a740 1 -- start start 2024-05-09T08:35:06.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3baa57a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b0b280 0x562117b0b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3baa57a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b15d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3baa57a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b16270 0x562117b18660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3baa57a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562117a70d00 con 0x562117b0bba0 2024-05-09T08:35:06.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3b967fc640 1 --2- 172.21.15.107:0/2965725770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3b8c0a2d80 0x7f3b8c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3baa57a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562117a184c0 con 0x562117b0b280 2024-05-09T08:35:06.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3baa57a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x562117a6ccc0 con 0x562117b16270 2024-05-09T08:35:06.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3ba5d2d640 1 --2- 172.21.15.107:0/2965725770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3b8c0a2d80 0x7f3b8c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3ba552c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b15d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3ba4d2b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b16270 0x562117b18660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3ba552c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b15d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54010/0 (socket says 172.21.15.107:54010) 2024-05-09T08:35:06.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3ba552c640 1 -- 172.21.15.107:0/3698921478 learned_addr learned my addr 172.21.15.107:0/3698921478 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:06.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3b967fc640 1 -- 172.21.15.107:0/2965725770 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3b9802e030 con 0x7f3ba0107890 2024-05-09T08:35:06.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/3698921478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b0b280 0x562117b0b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3ba4d2b640 1 -- 172.21.15.107:0/3698921478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b0b280 msgr2=0x562117b0b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:06.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.178+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/3698921478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b0b280 0x562117b0b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.179+0000 7f3ba4d2b640 1 -- 172.21.15.107:0/3698921478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 msgr2=0x562117b15d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:06.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.179+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/3698921478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b15d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.179+0000 7f3ba4d2b640 1 -- 172.21.15.107:0/3698921478 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56211784e7d0 con 0x562117b16270 2024-05-09T08:35:06.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.179+0000 7f3ba5d2d640 1 --2- 172.21.15.107:0/2965725770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3b8c0a2d80 0x7f3b8c0a5240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3b980b9490 tx=0x7f3b980dd040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:06.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.179+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/3698921478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b16270 0x562117b18660 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3b90012750 tx=0x7f3b90009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcd7f43b186ae131 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:06.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.179+0000 7f3b8b7fe640 1 -- 172.21.15.107:0/3698921478 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b90012cd0 con 0x562117b16270 2024-05-09T08:35:06.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.179+0000 7f3b8b7fe640 1 -- 172.21.15.107:0/3698921478 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b900054d0 con 0x562117b16270 2024-05-09T08:35:06.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.179+0000 7f3b8b7fe640 1 -- 172.21.15.107:0/3698921478 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b900107d0 con 0x562117b16270 2024-05-09T08:35:06.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.180+0000 7f3baa57a740 1 -- 172.21.15.107:0/3698921478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b16270 msgr2=0x562117b18660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.180+0000 7f3baa57a740 1 --2- 172.21.15.107:0/3698921478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b16270 0x562117b18660 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3b90012750 tx=0x7f3b90009680 comp rx=0 tx=0).stop 2024-05-09T08:35:06.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.180+0000 7f3baa57a740 1 -- 172.21.15.107:0/3698921478 shutdown_connections 2024-05-09T08:35:06.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.180+0000 7f3baa57a740 1 --2- 172.21.15.107:0/3698921478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b16270 0x562117b18660 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.180+0000 7f3baa57a740 1 --2- 172.21.15.107:0/3698921478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b15d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.180+0000 7f3baa57a740 1 --2- 172.21.15.107:0/3698921478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b0b280 0x562117b0b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.180+0000 7f3baa57a740 1 -- 172.21.15.107:0/3698921478 >> 172.21.15.107:0/3698921478 conn(0x562117b075a0 msgr2=0x562117b07980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:06.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.180+0000 7f3baa57a740 1 -- 172.21.15.107:0/3698921478 shutdown_connections 2024-05-09T08:35:06.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.180+0000 7f3baa57a740 1 -- 172.21.15.107:0/3698921478 wait complete. 2024-05-09T08:35:06.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.180+0000 7f3baa57a740 1 -- start start 2024-05-09T08:35:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.180+0000 7f3baa57a740 1 Processor -- start 2024-05-09T08:35:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3baa57a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 0x562117b13210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3baa57a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b13750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3baa57a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 0x562117b13c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3baa57a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562117b19060 con 0x562117b0bba0 2024-05-09T08:35:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3baa57a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562117a2bb90 con 0x562117b16270 2024-05-09T08:35:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3baa57a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x562117b191e0 con 0x562117b0b280 2024-05-09T08:35:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3baa57a740 10 client.0 _ll_register_callbacks cb 0x562117a67da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3ba4d2b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 0x562117b13210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3ba552c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b13750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3ba4d2b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 0x562117b13210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:58850/0 (socket says 172.21.15.107:58850) 2024-05-09T08:35:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3ba4d2b640 1 -- 172.21.15.107:0/3962947045 learned_addr learned my addr 172.21.15.107:0/3962947045 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/3962947045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 0x562117b13c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3ba552c640 1 -- 172.21.15.107:0/3962947045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 msgr2=0x562117b13210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:06.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3ba552c640 1 --2- 172.21.15.107:0/3962947045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 0x562117b13210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3ba552c640 1 -- 172.21.15.107:0/3962947045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 msgr2=0x562117b13c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:06.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3ba552c640 1 --2- 172.21.15.107:0/3962947045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 0x562117b13c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3ba552c640 1 -- 172.21.15.107:0/3962947045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621177843f0 con 0x562117b0bba0 2024-05-09T08:35:06.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/3962947045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 0x562117b13210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:06.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3ba552c640 1 --2- 172.21.15.107:0/3962947045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b13750 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x562117b0c840 tx=0x7f3b9c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb3409a83e27d93 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:06.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.181+0000 7f3b88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:06.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.182+0000 7f3b88ff9640 1 -- 172.21.15.107:0/3962947045 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c004020 con 0x562117b0bba0 2024-05-09T08:35:06.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.182+0000 7f3b88ff9640 1 -- 172.21.15.107:0/3962947045 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3b9c0041c0 con 0x562117b0bba0 2024-05-09T08:35:06.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.182+0000 7f3baa57a740 1 -- 172.21.15.107:0/3962947045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b900054d0 con 0x562117b0bba0 2024-05-09T08:35:06.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.182+0000 7f3b88ff9640 1 -- 172.21.15.107:0/3962947045 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3b9c024050 con 0x562117b0bba0 2024-05-09T08:35:06.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.182+0000 7f3baa57a740 10 client.16221 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:06.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.182+0000 7f3baa57a740 1 -- 172.21.15.107:0/3962947045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5621177440f0 con 0x562117b0bba0 2024-05-09T08:35:06.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.182+0000 7f3b88ff9640 1 -- 172.21.15.107:0/3962947045 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b900054d0 con 0x562117b0bba0 2024-05-09T08:35:06.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.182+0000 7f3b88ff9640 1 -- 172.21.15.107:0/3962947045 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3b9c003820 con 0x562117b0bba0 2024-05-09T08:35:06.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.182+0000 7f3baa57a740 10 client.16221 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:06.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.182+0000 7f3baa57a740 10 client.16221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:06.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.182+0000 7f3baa57a740 20 client.16221 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:06.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.182+0000 7f3baa57a740 1 --2- 172.21.15.107:0/3962947045 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562117b26760 0x562117b28ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.182+0000 7f3baa57a740 4 client.16221 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:06.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.182+0000 7f3baa57a740 1 -- 172.21.15.107:0/3962947045 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562117a7e430 con 0x562117b26760 2024-05-09T08:35:06.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.183+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/3962947045 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562117b26760 0x562117b28ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.183+0000 7f3b88ff9640 10 client.16221 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:06.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.183+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/3962947045 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562117b26760 0x562117b28ba0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28cceca328547d44 server_cookie=81e2aa7f88c6d821 in_seq=0 out_seq=0 2024-05-09T08:35:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.184+0000 7f3b88ff9640 1 -- 172.21.15.107:0/3962947045 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562117a7e430 con 0x562117b26760 2024-05-09T08:35:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.184+0000 7f3b88ff9640 10 client.16221 handle_command_reply: tid=0 2024-05-09T08:35:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.184+0000 7f3baa57a740 1 client.16221 shutdown 2024-05-09T08:35:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.184+0000 7f3baa57a740 1 -- 172.21.15.107:0/3962947045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 msgr2=0x562117b13750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.184+0000 7f3baa57a740 1 --2- 172.21.15.107:0/3962947045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b13750 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x562117b0c840 tx=0x7f3b9c011810 comp rx=0 tx=0).stop 2024-05-09T08:35:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.184+0000 7f3baa57a740 1 -- 172.21.15.107:0/3962947045 shutdown_connections 2024-05-09T08:35:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.184+0000 7f3baa57a740 1 --2- 172.21.15.107:0/3962947045 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562117b26760 0x562117b28ba0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.184+0000 7f3baa57a740 1 --2- 172.21.15.107:0/3962947045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 0x562117b13c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.184+0000 7f3baa57a740 1 --2- 172.21.15.107:0/3962947045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b13750 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.184+0000 7f3baa57a740 1 --2- 172.21.15.107:0/3962947045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 0x562117b13210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.184+0000 7f3baa57a740 1 -- 172.21.15.107:0/3962947045 >> 172.21.15.107:0/3962947045 conn(0x562117b075a0 msgr2=0x562117b067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.184+0000 7f3b88ff9640 0 client.16221 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.184+0000 7f3baa57a740 1 -- 172.21.15.107:0/3962947045 shutdown_connections 2024-05-09T08:35:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.184+0000 7f3baa57a740 1 -- 172.21.15.107:0/3962947045 wait complete. 2024-05-09T08:35:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.184+0000 7f3baa57a740 20 client.16221 trim_cache size 0 max 16384 2024-05-09T08:35:06.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.202+0000 7f3baa57a740 1 Processor -- start 2024-05-09T08:35:06.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.202+0000 7f3baa57a740 1 -- start start 2024-05-09T08:35:06.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.202+0000 7f3baa57a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 0x562117b21f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.202+0000 7f3baa57a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b123b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.202+0000 7f3baa57a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 0x562117b138e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.202+0000 7f3baa57a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba01b1340 con 0x562117b0bba0 2024-05-09T08:35:06.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.202+0000 7f3baa57a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba01aec10 con 0x562117b16270 2024-05-09T08:35:06.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.202+0000 7f3baa57a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba01aca60 con 0x562117b0b280 2024-05-09T08:35:06.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.202+0000 7f3ba552c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b123b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.202+0000 7f3ba552c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b123b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54028/0 (socket says 172.21.15.107:54028) 2024-05-09T08:35:06.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.202+0000 7f3ba552c640 1 -- 172.21.15.107:0/1961149845 learned_addr learned my addr 172.21.15.107:0/1961149845 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:06.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.202+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/1961149845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 0x562117b138e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.203+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/1961149845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 0x562117b21f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.203+0000 7f3ba552c640 1 -- 172.21.15.107:0/1961149845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 msgr2=0x562117b21f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:06.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.203+0000 7f3ba552c640 1 --2- 172.21.15.107:0/1961149845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 0x562117b21f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.203+0000 7f3ba552c640 1 -- 172.21.15.107:0/1961149845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 msgr2=0x562117b138e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:06.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.203+0000 7f3ba552c640 1 --2- 172.21.15.107:0/1961149845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 0x562117b138e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.203+0000 7f3ba552c640 1 -- 172.21.15.107:0/1961149845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562117b117a0 con 0x562117b0bba0 2024-05-09T08:35:06.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.203+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/1961149845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 0x562117b138e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:06.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.203+0000 7f3ba552c640 1 --2- 172.21.15.107:0/1961149845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b123b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c000c00 tx=0x7f3b9c005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94113a6b9e1a306 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:06.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.203+0000 7f3b8affd640 1 -- 172.21.15.107:0/1961149845 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c02c070 con 0x562117b0bba0 2024-05-09T08:35:06.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.203+0000 7f3b8affd640 1 -- 172.21.15.107:0/1961149845 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b9c00b540 con 0x562117b0bba0 2024-05-09T08:35:06.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.203+0000 7f3b8affd640 1 -- 172.21.15.107:0/1961149845 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c005040 con 0x562117b0bba0 2024-05-09T08:35:06.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.204+0000 7f3baa57a740 1 -- 172.21.15.107:0/1961149845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 msgr2=0x562117b123b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.204+0000 7f3baa57a740 1 --2- 172.21.15.107:0/1961149845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b123b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c000c00 tx=0x7f3b9c005c50 comp rx=0 tx=0).stop 2024-05-09T08:35:06.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.204+0000 7f3baa57a740 1 -- 172.21.15.107:0/1961149845 shutdown_connections 2024-05-09T08:35:06.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.204+0000 7f3baa57a740 1 --2- 172.21.15.107:0/1961149845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 0x562117b138e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.204+0000 7f3baa57a740 1 --2- 172.21.15.107:0/1961149845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b123b0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.204+0000 7f3baa57a740 1 --2- 172.21.15.107:0/1961149845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 0x562117b21f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.204+0000 7f3baa57a740 1 -- 172.21.15.107:0/1961149845 >> 172.21.15.107:0/1961149845 conn(0x562117b075a0 msgr2=0x562117b131d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.204+0000 7f3baa57a740 1 -- 172.21.15.107:0/1961149845 shutdown_connections 2024-05-09T08:35:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.204+0000 7f3baa57a740 1 -- 172.21.15.107:0/1961149845 wait complete. 2024-05-09T08:35:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.204+0000 7f3baa57a740 1 -- start start 2024-05-09T08:35:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.204+0000 7f3baa57a740 1 Processor -- start 2024-05-09T08:35:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3baa57a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 0x562117b1e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3baa57a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b1e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3baa57a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 0x562117b0f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3baa57a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba01ad0d0 con 0x562117b0bba0 2024-05-09T08:35:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3baa57a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba01ae540 con 0x562117b16270 2024-05-09T08:35:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3baa57a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba01ad7a0 con 0x562117b0b280 2024-05-09T08:35:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3baa57a740 10 client.0 _ll_register_callbacks cb 0x7f3b9c00b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3ba552c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b1e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3ba552c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b1e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54034/0 (socket says 172.21.15.107:54034) 2024-05-09T08:35:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3ba552c640 1 -- 172.21.15.107:0/509272542 learned_addr learned my addr 172.21.15.107:0/509272542 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/509272542 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 0x562117b1e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/509272542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 0x562117b0f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3ba552c640 1 -- 172.21.15.107:0/509272542 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 msgr2=0x562117b1e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3ba552c640 1 --2- 172.21.15.107:0/509272542 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 0x562117b1e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3ba552c640 1 -- 172.21.15.107:0/509272542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 msgr2=0x562117b0f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3ba552c640 1 --2- 172.21.15.107:0/509272542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 0x562117b0f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3ba552c640 1 -- 172.21.15.107:0/509272542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562117b0fde0 con 0x562117b0bba0 2024-05-09T08:35:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/509272542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 0x562117b0f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3ba552c640 1 --2- 172.21.15.107:0/509272542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b1e9d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c005d00 tx=0x7f3b9c01a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deb290633f841c2f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3b88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3b88ff9640 1 -- 172.21.15.107:0/509272542 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c02c040 con 0x562117b0bba0 2024-05-09T08:35:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3b88ff9640 1 -- 172.21.15.107:0/509272542 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3b9c004280 con 0x562117b0bba0 2024-05-09T08:35:06.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.205+0000 7f3baa57a740 1 -- 172.21.15.107:0/509272542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b9c00b540 con 0x562117b0bba0 2024-05-09T08:35:06.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.206+0000 7f3b88ff9640 1 -- 172.21.15.107:0/509272542 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3b9c041050 con 0x562117b0bba0 2024-05-09T08:35:06.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.206+0000 7f3baa57a740 10 client.16227 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:06.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.206+0000 7f3baa57a740 1 -- 172.21.15.107:0/509272542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562117b90b40 con 0x562117b0bba0 2024-05-09T08:35:06.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.206+0000 7f3b88ff9640 1 -- 172.21.15.107:0/509272542 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b9c00b540 con 0x562117b0bba0 2024-05-09T08:35:06.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.206+0000 7f3b88ff9640 1 -- 172.21.15.107:0/509272542 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3b9c010700 con 0x562117b0bba0 2024-05-09T08:35:06.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.206+0000 7f3baa57a740 10 client.16227 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:06.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.206+0000 7f3baa57a740 10 client.16227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:06.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.206+0000 7f3baa57a740 20 client.16227 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:06.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.206+0000 7f3baa57a740 1 --2- 172.21.15.107:0/509272542 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562117b910f0 0x562117b934e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:06.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.206+0000 7f3baa57a740 4 client.16227 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:35:06.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.206+0000 7f3baa57a740 1 -- 172.21.15.107:0/509272542 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562117a78780 con 0x562117b910f0 2024-05-09T08:35:06.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.206+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/509272542 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562117b910f0 0x562117b934e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:06.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.207+0000 7f3b88ff9640 10 client.16227 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:06.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.207+0000 7f3ba4d2b640 1 --2- 172.21.15.107:0/509272542 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562117b910f0 0x562117b934e0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60773e0167341a7b server_cookie=c97bee5c28cf59c1 in_seq=0 out_seq=0 2024-05-09T08:35:06.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.207+0000 7f3b88ff9640 1 -- 172.21.15.107:0/509272542 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562117a78780 con 0x562117b910f0 2024-05-09T08:35:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.207+0000 7f3b88ff9640 10 client.16227 handle_command_reply: tid=0 2024-05-09T08:35:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.207+0000 7f3baa57a740 1 client.16227 shutdown 2024-05-09T08:35:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.207+0000 7f3baa57a740 1 -- 172.21.15.107:0/509272542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 msgr2=0x562117b1e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.207+0000 7f3baa57a740 1 --2- 172.21.15.107:0/509272542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b1e9d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c005d00 tx=0x7f3b9c01a620 comp rx=0 tx=0).stop 2024-05-09T08:35:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.207+0000 7f3baa57a740 1 -- 172.21.15.107:0/509272542 shutdown_connections 2024-05-09T08:35:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.207+0000 7f3baa57a740 1 --2- 172.21.15.107:0/509272542 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562117b910f0 0x562117b934e0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.207+0000 7f3baa57a740 1 --2- 172.21.15.107:0/509272542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562117b16270 0x562117b0f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.207+0000 7f3b88ff9640 0 client.16227 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.207+0000 7f3baa57a740 1 --2- 172.21.15.107:0/509272542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562117b0bba0 0x562117b1e9d0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.207+0000 7f3baa57a740 1 --2- 172.21.15.107:0/509272542 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562117b0b280 0x562117b1e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.207+0000 7f3baa57a740 1 -- 172.21.15.107:0/509272542 >> 172.21.15.107:0/509272542 conn(0x562117b075a0 msgr2=0x562117b0d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.207+0000 7f3baa57a740 1 -- 172.21.15.107:0/509272542 shutdown_connections 2024-05-09T08:35:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.208+0000 7f3baa57a740 1 -- 172.21.15.107:0/509272542 wait complete. 2024-05-09T08:35:06.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.208+0000 7f3baa57a740 20 client.16227 trim_cache size 0 max 16384 2024-05-09T08:35:06.215 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:35:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:35:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:35:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: "60ac5c9b-e5fb-4198-afa1-36e5a1b6cbce": { 2024-05-09T08:35:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:35:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "60ac5c9b-e5fb-4198-afa1-36e5a1b6cbce", 2024-05-09T08:35:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:35:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:35:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:35:06.215 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:35:06.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.208+0000 7f3ba7790640 1 -- 172.21.15.107:0/2965725770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3b8c0a2d80 msgr2=0x7f3b8c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.208+0000 7f3ba7790640 1 --2- 172.21.15.107:0/2965725770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3b8c0a2d80 0x7f3b8c0a5240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3b980b9490 tx=0x7f3b980dd040 comp rx=0 tx=0).stop 2024-05-09T08:35:06.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.208+0000 7f3ba7790640 1 -- 172.21.15.107:0/2965725770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba0107890 msgr2=0x7f3ba019c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:06.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.208+0000 7f3ba7790640 1 --2- 172.21.15.107:0/2965725770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba0107890 0x7f3ba019c960 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f3b98033040 tx=0x7f3b98003970 comp rx=0 tx=0).stop 2024-05-09T08:35:06.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.208+0000 7f3ba7790640 1 -- 172.21.15.107:0/2965725770 shutdown_connections 2024-05-09T08:35:06.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.208+0000 7f3ba7790640 1 --2- 172.21.15.107:0/2965725770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3b8c0a2d80 0x7f3b8c0a5240 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.208+0000 7f3ba7790640 1 --2- 172.21.15.107:0/2965725770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba0107890 0x7f3ba019c960 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.208+0000 7f3ba7790640 1 --2- 172.21.15.107:0/2965725770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ba0102240 0x7f3ba0198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.208+0000 7f3ba7790640 1 --2- 172.21.15.107:0/2965725770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ba0101880 0x7f3ba01980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:06.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.208+0000 7f3ba7790640 1 -- 172.21.15.107:0/2965725770 >> 172.21.15.107:0/2965725770 conn(0x7f3ba00fd390 msgr2=0x7f3ba00fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:06.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.208+0000 7f3ba7790640 1 -- 172.21.15.107:0/2965725770 shutdown_connections 2024-05-09T08:35:06.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:06.208+0000 7f3ba7790640 1 -- 172.21.15.107:0/2965725770 wait complete. 2024-05-09T08:35:06.219 INFO:tasks.cephfs.filesystem:scrub status for tag:60ac5c9b-e5fb-4198-afa1-36e5a1b6cbce - {'path': '/', 'tag': '60ac5c9b-e5fb-4198-afa1-36e5a1b6cbce', 'options': 'recursive,force'} 2024-05-09T08:35:06.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:06 smithi157 ceph-mon[33614]: pgmap v55: 129 pgs: 129 active+clean; 91 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 7.1 MiB/s wr, 106 op/s 2024-05-09T08:35:06.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:06 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:35:06.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:06 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:06.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:06 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:35:06.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:06 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:06.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:06 smithi110 ceph-mon[33622]: pgmap v55: 129 pgs: 129 active+clean; 91 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 7.1 MiB/s wr, 106 op/s 2024-05-09T08:35:06.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:06 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:35:06.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:06 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:06.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:06 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:35:06.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:06 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:07.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:06 smithi107 ceph-mon[30122]: pgmap v55: 129 pgs: 129 active+clean; 91 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 7.1 MiB/s wr, 106 op/s 2024-05-09T08:35:07.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:06 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:35:07.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:06 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:07.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:06 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:35:07.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:06 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:07.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:07 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:35:07.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:07 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:07.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:07 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:35:07.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:07 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:35:07.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:07 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:07.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:07 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:35:07.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:07 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:07.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:07 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:35:07.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:07 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:07.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:07 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:07.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:07 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:35:07.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:07 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:07.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:07 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:35:07.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:07 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:35:07.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:07 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:07.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:07 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:35:07.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:07 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:07.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:07 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:35:07.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:07 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:07.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:07 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:08.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:07 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:35:08.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:07 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:08.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:07 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:35:08.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:07 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:35:08.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:07 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:08.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:07 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:35:08.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:07 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:08.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:07 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:35:08.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:07 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:08.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:07 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:08.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:08 smithi157 ceph-mon[33614]: pgmap v56: 129 pgs: 129 active+clean; 231 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 19 MiB/s wr, 161 op/s 2024-05-09T08:35:08.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:08 smithi110 ceph-mon[33622]: pgmap v56: 129 pgs: 129 active+clean; 231 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 19 MiB/s wr, 161 op/s 2024-05-09T08:35:09.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:08 smithi107 ceph-mon[30122]: pgmap v56: 129 pgs: 129 active+clean; 231 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 19 MiB/s wr, 161 op/s 2024-05-09T08:35:10.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:10 smithi157 ceph-mon[33614]: pgmap v57: 129 pgs: 129 active+clean; 231 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 19 MiB/s wr, 92 op/s 2024-05-09T08:35:10.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:10 smithi110 ceph-mon[33622]: pgmap v57: 129 pgs: 129 active+clean; 231 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 19 MiB/s wr, 92 op/s 2024-05-09T08:35:11.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:10 smithi107 ceph-mon[30122]: pgmap v57: 129 pgs: 129 active+clean; 231 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 19 MiB/s wr, 92 op/s 2024-05-09T08:35:12.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:12 smithi157 ceph-mon[33614]: pgmap v58: 129 pgs: 129 active+clean; 231 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 19 MiB/s wr, 92 op/s 2024-05-09T08:35:12.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:12 smithi110 ceph-mon[33622]: pgmap v58: 129 pgs: 129 active+clean; 231 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 19 MiB/s wr, 92 op/s 2024-05-09T08:35:13.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:12 smithi107 ceph-mon[30122]: pgmap v58: 129 pgs: 129 active+clean; 231 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 19 MiB/s wr, 92 op/s 2024-05-09T08:35:14.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:14 smithi157 ceph-mon[33614]: pgmap v59: 129 pgs: 129 active+clean; 379 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 31 MiB/s wr, 100 op/s 2024-05-09T08:35:14.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:14 smithi110 ceph-mon[33622]: pgmap v59: 129 pgs: 129 active+clean; 379 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 31 MiB/s wr, 100 op/s 2024-05-09T08:35:15.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:14 smithi107 ceph-mon[30122]: pgmap v59: 129 pgs: 129 active+clean; 379 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 31 MiB/s wr, 100 op/s 2024-05-09T08:35:15.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.595+0000 7f322662a640 1 -- 172.21.15.107:0/805967540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f322010be00 msgr2=0x7f322010e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:15.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.595+0000 7f322662a640 1 --2- 172.21.15.107:0/805967540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f322010be00 0x7f322010e2a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f3218009ca0 tx=0x7f321802a950 comp rx=0 tx=0).stop 2024-05-09T08:35:15.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.595+0000 7f322662a640 1 -- 172.21.15.107:0/805967540 shutdown_connections 2024-05-09T08:35:15.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.595+0000 7f322662a640 1 --2- 172.21.15.107:0/805967540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f322010be00 0x7f322010e2a0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.595+0000 7f322662a640 1 --2- 172.21.15.107:0/805967540 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3220103300 0x7f322010b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.595+0000 7f322662a640 1 --2- 172.21.15.107:0/805967540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32201029e0 0x7f3220102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.595+0000 7f322662a640 1 -- 172.21.15.107:0/805967540 >> 172.21.15.107:0/805967540 conn(0x7f32200fc200 msgr2=0x7f32200fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:15.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.595+0000 7f322662a640 1 -- 172.21.15.107:0/805967540 shutdown_connections 2024-05-09T08:35:15.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.595+0000 7f322662a640 1 -- 172.21.15.107:0/805967540 wait complete. 2024-05-09T08:35:15.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.595+0000 7f322662a640 1 Processor -- start 2024-05-09T08:35:15.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.595+0000 7f322662a640 1 -- start start 2024-05-09T08:35:15.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f322662a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32201029e0 0x7f3220197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f322662a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3220103300 0x7f3220198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f322662a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f322010be00 0x7f322019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f322662a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3220111ab0 con 0x7f3220103300 2024-05-09T08:35:15.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f322662a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3220111930 con 0x7f322010be00 2024-05-09T08:35:15.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f322662a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3220111c30 con 0x7f32201029e0 2024-05-09T08:35:15.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f321f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3220103300 0x7f3220198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f321f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3220103300 0x7f3220198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39698/0 (socket says 172.21.15.107:39698) 2024-05-09T08:35:15.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f321f7fe640 1 -- 172.21.15.107:0/4154901670 learned_addr learned my addr 172.21.15.107:0/4154901670 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:15.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f321ffff640 1 --2- 172.21.15.107:0/4154901670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32201029e0 0x7f3220197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f3224bc7640 1 --2- 172.21.15.107:0/4154901670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f322010be00 0x7f322019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f321f7fe640 1 -- 172.21.15.107:0/4154901670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32201029e0 msgr2=0x7f3220197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:15.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f321f7fe640 1 --2- 172.21.15.107:0/4154901670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32201029e0 0x7f3220197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f321f7fe640 1 -- 172.21.15.107:0/4154901670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f322010be00 msgr2=0x7f322019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:15.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f321f7fe640 1 --2- 172.21.15.107:0/4154901670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f322010be00 0x7f322019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f321f7fe640 1 -- 172.21.15.107:0/4154901670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f322019cf40 con 0x7f3220103300 2024-05-09T08:35:15.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f3224bc7640 1 --2- 172.21.15.107:0/4154901670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f322010be00 0x7f322019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:35:15.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.596+0000 7f321f7fe640 1 --2- 172.21.15.107:0/4154901670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3220103300 0x7f3220198490 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f321400e9f0 tx=0x7f321400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:15.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.597+0000 7f321d7fa640 1 -- 172.21.15.107:0/4154901670 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321400cab0 con 0x7f3220103300 2024-05-09T08:35:15.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.597+0000 7f322662a640 1 -- 172.21.15.107:0/4154901670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f322019d230 con 0x7f3220103300 2024-05-09T08:35:15.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.597+0000 7f321d7fa640 1 -- 172.21.15.107:0/4154901670 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f321400cc50 con 0x7f3220103300 2024-05-09T08:35:15.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.597+0000 7f321d7fa640 1 -- 172.21.15.107:0/4154901670 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3214018750 con 0x7f3220103300 2024-05-09T08:35:15.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.597+0000 7f322662a640 1 -- 172.21.15.107:0/4154901670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32201a4ac0 con 0x7f3220103300 2024-05-09T08:35:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.599+0000 7f321d7fa640 1 -- 172.21.15.107:0/4154901670 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3214018b00 con 0x7f3220103300 2024-05-09T08:35:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.599+0000 7f3229414740 1 Processor -- start 2024-05-09T08:35:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.599+0000 7f3229414740 1 -- start start 2024-05-09T08:35:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.599+0000 7f3229414740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6450 0x5622cd2f6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.599+0000 7f3229414740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6d70 0x5622cd3010f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.599+0000 7f3229414740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd301630 0x5622cd303a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f3229414740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5622cd25b340 con 0x5622cd2f6d70 2024-05-09T08:35:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f3229414740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5622cd203290 con 0x5622cd2f6450 2024-05-09T08:35:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f3229414740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5622cd257300 con 0x5622cd301630 2024-05-09T08:35:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f321ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6d70 0x5622cd3010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f321ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6d70 0x5622cd3010f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39714/0 (socket says 172.21.15.107:39714) 2024-05-09T08:35:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f321ffff640 1 -- 172.21.15.107:0/3628974408 learned_addr learned my addr 172.21.15.107:0/3628974408 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f321f7fe640 1 --2- 172.21.15.107:0/3628974408 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd301630 0x5622cd303a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f3224bc7640 1 --2- 172.21.15.107:0/3628974408 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6450 0x5622cd2f6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f321ffff640 1 -- 172.21.15.107:0/3628974408 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd301630 msgr2=0x5622cd303a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f321ffff640 1 --2- 172.21.15.107:0/3628974408 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd301630 0x5622cd303a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f321ffff640 1 -- 172.21.15.107:0/3628974408 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6450 msgr2=0x5622cd2f6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f321ffff640 1 --2- 172.21.15.107:0/3628974408 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6450 0x5622cd2f6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f321ffff640 1 -- 172.21.15.107:0/3628974408 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622cd2278e0 con 0x5622cd2f6d70 2024-05-09T08:35:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f321d7fa640 1 --2- 172.21.15.107:0/4154901670 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f32080a2da0 0x7f32080a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f3224bc7640 1 --2- 172.21.15.107:0/4154901670 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f32080a2da0 0x7f32080a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f321ffff640 1 --2- 172.21.15.107:0/3628974408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6d70 0x5622cd3010f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f321000ad00 tx=0x7f321000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7f158336be94761 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f321d7fa640 1 -- 172.21.15.107:0/4154901670 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f32140cfc20 con 0x7f3220103300 2024-05-09T08:35:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.600+0000 7f32067fc640 1 -- 172.21.15.107:0/3628974408 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321001d070 con 0x5622cd2f6d70 2024-05-09T08:35:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.601+0000 7f32067fc640 1 -- 172.21.15.107:0/3628974408 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3210018af0 con 0x5622cd2f6d70 2024-05-09T08:35:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.601+0000 7f32067fc640 1 -- 172.21.15.107:0/3628974408 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3210010ba0 con 0x5622cd2f6d70 2024-05-09T08:35:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.601+0000 7f3229414740 1 -- 172.21.15.107:0/3628974408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6d70 msgr2=0x5622cd3010f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.601+0000 7f3224bc7640 1 --2- 172.21.15.107:0/4154901670 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f32080a2da0 0x7f32080a5260 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3218004af0 tx=0x7f3218005d80 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.601+0000 7f3229414740 1 --2- 172.21.15.107:0/3628974408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6d70 0x5622cd3010f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f321000ad00 tx=0x7f321000f790 comp rx=0 tx=0).stop 2024-05-09T08:35:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.601+0000 7f3229414740 1 -- 172.21.15.107:0/3628974408 shutdown_connections 2024-05-09T08:35:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.601+0000 7f3229414740 1 --2- 172.21.15.107:0/3628974408 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd301630 0x5622cd303a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.601+0000 7f3229414740 1 --2- 172.21.15.107:0/3628974408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6d70 0x5622cd3010f0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.601+0000 7f3229414740 1 --2- 172.21.15.107:0/3628974408 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6450 0x5622cd2f6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.601+0000 7f3229414740 1 -- 172.21.15.107:0/3628974408 >> 172.21.15.107:0/3628974408 conn(0x5622cd2f2960 msgr2=0x5622cd2f2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.601+0000 7f3229414740 1 -- 172.21.15.107:0/3628974408 shutdown_connections 2024-05-09T08:35:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.601+0000 7f3229414740 1 -- 172.21.15.107:0/3628974408 wait complete. 2024-05-09T08:35:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.602+0000 7f3229414740 1 -- start start 2024-05-09T08:35:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.602+0000 7f3229414740 1 Processor -- start 2024-05-09T08:35:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.602+0000 7f3229414740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6450 0x5622cd2fe590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.602+0000 7f3229414740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd2f6d70 0x5622cd2fead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.602+0000 7f3229414740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd301630 0x5622cd2ff010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.602+0000 7f3229414740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5622cd304260 con 0x5622cd301630 2024-05-09T08:35:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.602+0000 7f3229414740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5622cd216790 con 0x5622cd2f6450 2024-05-09T08:35:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.602+0000 7f3229414740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5622cd3043e0 con 0x5622cd2f6d70 2024-05-09T08:35:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.602+0000 7f3229414740 10 client.0 _ll_register_callbacks cb 0x5622cd251a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.602+0000 7f321ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6450 0x5622cd2fe590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.602+0000 7f3224bc7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd301630 0x5622cd2ff010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.602+0000 7f321ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6450 0x5622cd2fe590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:55958/0 (socket says 172.21.15.107:55958) 2024-05-09T08:35:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.602+0000 7f321ffff640 1 -- 172.21.15.107:0/3232247890 learned_addr learned my addr 172.21.15.107:0/3232247890 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.602+0000 7f321f7fe640 1 --2- 172.21.15.107:0/3232247890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd2f6d70 0x5622cd2fead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.603+0000 7f3224bc7640 1 -- 172.21.15.107:0/3232247890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd2f6d70 msgr2=0x5622cd2fead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.603+0000 7f3224bc7640 1 --2- 172.21.15.107:0/3232247890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd2f6d70 0x5622cd2fead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.603+0000 7f3224bc7640 1 -- 172.21.15.107:0/3232247890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6450 msgr2=0x5622cd2fe590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.603+0000 7f3224bc7640 1 --2- 172.21.15.107:0/3232247890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6450 0x5622cd2fe590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.603+0000 7f3224bc7640 1 -- 172.21.15.107:0/3232247890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622ccf6f210 con 0x5622cd301630 2024-05-09T08:35:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.603+0000 7f321f7fe640 1 --2- 172.21.15.107:0/3232247890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd2f6d70 0x5622cd2fead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.603+0000 7f321ffff640 1 --2- 172.21.15.107:0/3232247890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6450 0x5622cd2fe590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.603+0000 7f3224bc7640 1 --2- 172.21.15.107:0/3232247890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd301630 0x5622cd2ff010 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3218009ca0 tx=0x7f321802a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f4bca5cd49892f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.603+0000 7f31ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.603+0000 7f31ff7fe640 1 -- 172.21.15.107:0/3232247890 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3218034040 con 0x5622cd301630 2024-05-09T08:35:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.603+0000 7f31ff7fe640 1 -- 172.21.15.107:0/3232247890 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3218051440 con 0x5622cd301630 2024-05-09T08:35:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.603+0000 7f3229414740 1 -- 172.21.15.107:0/3232247890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3210018af0 con 0x5622cd301630 2024-05-09T08:35:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.603+0000 7f31ff7fe640 1 -- 172.21.15.107:0/3232247890 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3218051640 con 0x5622cd301630 2024-05-09T08:35:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.603+0000 7f3229414740 10 client.16239 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.603+0000 7f3229414740 1 -- 172.21.15.107:0/3232247890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5622cd03b620 con 0x5622cd301630 2024-05-09T08:35:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.603+0000 7f31ff7fe640 1 -- 172.21.15.107:0/3232247890 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3210018af0 con 0x5622cd301630 2024-05-09T08:35:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.604+0000 7f31ff7fe640 1 -- 172.21.15.107:0/3232247890 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f321804f020 con 0x5622cd301630 2024-05-09T08:35:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.604+0000 7f3229414740 10 client.16239 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.604+0000 7f3229414740 10 client.16239 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.604+0000 7f3229414740 20 client.16239 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.604+0000 7f3229414740 1 --2- 172.21.15.107:0/3232247890 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5622cd3113a0 0x5622cd3137d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.604+0000 7f3229414740 4 client.16239 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.604+0000 7f3229414740 1 -- 172.21.15.107:0/3232247890 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622cd268a70 con 0x5622cd3113a0 2024-05-09T08:35:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.604+0000 7f321ffff640 1 --2- 172.21.15.107:0/3232247890 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5622cd3113a0 0x5622cd3137d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.604+0000 7f321ffff640 1 --2- 172.21.15.107:0/3232247890 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5622cd3113a0 0x5622cd3137d0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=284864d830885b0e server_cookie=edcb4e3ec5d25f9e in_seq=0 out_seq=0 2024-05-09T08:35:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.604+0000 7f31ff7fe640 10 client.16239 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.605+0000 7f31ff7fe640 1 -- 172.21.15.107:0/3232247890 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5622cd268a70 con 0x5622cd3113a0 2024-05-09T08:35:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.605+0000 7f31ff7fe640 10 client.16239 handle_command_reply: tid=0 2024-05-09T08:35:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.605+0000 7f3229414740 1 client.16239 shutdown 2024-05-09T08:35:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.605+0000 7f3229414740 1 -- 172.21.15.107:0/3232247890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd301630 msgr2=0x5622cd2ff010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.605+0000 7f3229414740 1 --2- 172.21.15.107:0/3232247890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd301630 0x5622cd2ff010 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3218009ca0 tx=0x7f321802a950 comp rx=0 tx=0).stop 2024-05-09T08:35:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.605+0000 7f3229414740 1 -- 172.21.15.107:0/3232247890 shutdown_connections 2024-05-09T08:35:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.605+0000 7f3229414740 1 --2- 172.21.15.107:0/3232247890 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5622cd3113a0 0x5622cd3137d0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.605+0000 7f3229414740 1 --2- 172.21.15.107:0/3232247890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd301630 0x5622cd2ff010 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.605+0000 7f31ff7fe640 0 client.16239 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.605+0000 7f3229414740 1 --2- 172.21.15.107:0/3232247890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd2f6d70 0x5622cd2fead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.605+0000 7f3229414740 1 --2- 172.21.15.107:0/3232247890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6450 0x5622cd2fe590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.605+0000 7f3229414740 1 -- 172.21.15.107:0/3232247890 >> 172.21.15.107:0/3232247890 conn(0x5622cd2f2960 msgr2=0x5622cd2f86e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.606+0000 7f3229414740 1 -- 172.21.15.107:0/3232247890 shutdown_connections 2024-05-09T08:35:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.606+0000 7f3229414740 1 -- 172.21.15.107:0/3232247890 wait complete. 2024-05-09T08:35:15.614 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.606+0000 7f3229414740 20 client.16239 trim_cache size 0 max 16384 2024-05-09T08:35:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f3229414740 1 Processor -- start 2024-05-09T08:35:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f3229414740 1 -- start start 2024-05-09T08:35:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f3229414740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6450 0x5622cd300cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f3229414740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6d70 0x5622cd30d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f3229414740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd301630 0x5622cd30d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f3229414740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f32201b12a0 con 0x5622cd2f6450 2024-05-09T08:35:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f3229414740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f32201aeb70 con 0x5622cd2f6d70 2024-05-09T08:35:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f3229414740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f32201ac9c0 con 0x5622cd301630 2024-05-09T08:35:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f321ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6450 0x5622cd300cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f321ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6450 0x5622cd300cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39732/0 (socket says 172.21.15.107:39732) 2024-05-09T08:35:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f321ffff640 1 -- 172.21.15.107:0/104000620 learned_addr learned my addr 172.21.15.107:0/104000620 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f321f7fe640 1 --2- 172.21.15.107:0/104000620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6d70 0x5622cd30d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f3224bc7640 1 --2- 172.21.15.107:0/104000620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd301630 0x5622cd30d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f321ffff640 1 -- 172.21.15.107:0/104000620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd301630 msgr2=0x5622cd30d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f321ffff640 1 --2- 172.21.15.107:0/104000620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd301630 0x5622cd30d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f321ffff640 1 -- 172.21.15.107:0/104000620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6d70 msgr2=0x5622cd30d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f321ffff640 1 --2- 172.21.15.107:0/104000620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6d70 0x5622cd30d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f321ffff640 1 -- 172.21.15.107:0/104000620 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622cd2f0f00 con 0x5622cd2f6450 2024-05-09T08:35:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f3224bc7640 1 --2- 172.21.15.107:0/104000620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd301630 0x5622cd30d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:35:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.622+0000 7f321f7fe640 1 --2- 172.21.15.107:0/104000620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6d70 0x5622cd30d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.623+0000 7f321ffff640 1 --2- 172.21.15.107:0/104000620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6450 0x5622cd300cc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f321000fc90 tx=0x7f321000a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f04a51fd7d88350 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.623+0000 7f3205ffb640 1 -- 172.21.15.107:0/104000620 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321001d040 con 0x5622cd2f6450 2024-05-09T08:35:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.623+0000 7f3205ffb640 1 -- 172.21.15.107:0/104000620 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3210030070 con 0x5622cd2f6450 2024-05-09T08:35:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.623+0000 7f3205ffb640 1 -- 172.21.15.107:0/104000620 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321002a220 con 0x5622cd2f6450 2024-05-09T08:35:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.623+0000 7f3229414740 1 -- 172.21.15.107:0/104000620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6450 msgr2=0x5622cd300cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.623+0000 7f3229414740 1 --2- 172.21.15.107:0/104000620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6450 0x5622cd300cc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f321000fc90 tx=0x7f321000a2d0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.623+0000 7f3229414740 1 -- 172.21.15.107:0/104000620 shutdown_connections 2024-05-09T08:35:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.623+0000 7f3229414740 1 --2- 172.21.15.107:0/104000620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd301630 0x5622cd30d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.624+0000 7f3229414740 1 --2- 172.21.15.107:0/104000620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd2f6d70 0x5622cd30d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.624+0000 7f3229414740 1 --2- 172.21.15.107:0/104000620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6450 0x5622cd300cc0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.624+0000 7f3229414740 1 -- 172.21.15.107:0/104000620 >> 172.21.15.107:0/104000620 conn(0x5622cd2f2960 msgr2=0x5622cd2ed620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.624+0000 7f3229414740 1 -- 172.21.15.107:0/104000620 shutdown_connections 2024-05-09T08:35:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.624+0000 7f3229414740 1 -- 172.21.15.107:0/104000620 wait complete. 2024-05-09T08:35:15.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.624+0000 7f3229414740 1 -- start start 2024-05-09T08:35:15.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.624+0000 7f3229414740 1 Processor -- start 2024-05-09T08:35:15.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.624+0000 7f3229414740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6450 0x5622cd304880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.624+0000 7f3229414740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd2f6d70 0x5622cd304dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.624+0000 7f3229414740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd301630 0x5622cd36f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.624+0000 7f3229414740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f32201ad030 con 0x5622cd2f6450 2024-05-09T08:35:15.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.624+0000 7f3229414740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f32201ae4a0 con 0x5622cd301630 2024-05-09T08:35:15.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.624+0000 7f3229414740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f32201ad700 con 0x5622cd2f6d70 2024-05-09T08:35:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.624+0000 7f3229414740 10 client.0 _ll_register_callbacks cb 0x7f3214005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f321ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6450 0x5622cd304880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f321ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6450 0x5622cd304880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39744/0 (socket says 172.21.15.107:39744) 2024-05-09T08:35:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f321ffff640 1 -- 172.21.15.107:0/3161896028 learned_addr learned my addr 172.21.15.107:0/3161896028 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f321f7fe640 1 --2- 172.21.15.107:0/3161896028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd2f6d70 0x5622cd304dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f3224bc7640 1 --2- 172.21.15.107:0/3161896028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd301630 0x5622cd36f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f321ffff640 1 -- 172.21.15.107:0/3161896028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd2f6d70 msgr2=0x5622cd304dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f321ffff640 1 --2- 172.21.15.107:0/3161896028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd2f6d70 0x5622cd304dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f321ffff640 1 -- 172.21.15.107:0/3161896028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd301630 msgr2=0x5622cd36f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f321ffff640 1 --2- 172.21.15.107:0/3161896028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd301630 0x5622cd36f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f321ffff640 1 -- 172.21.15.107:0/3161896028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622cd36fed0 con 0x5622cd2f6450 2024-05-09T08:35:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f3224bc7640 1 --2- 172.21.15.107:0/3161896028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd301630 0x5622cd36f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:35:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f321ffff640 1 --2- 172.21.15.107:0/3161896028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6450 0x5622cd304880 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f32100106e0 tx=0x7f321000a920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14552e68f75f7bdc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f31ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f31ff7fe640 1 -- 172.21.15.107:0/3161896028 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3210038020 con 0x5622cd2f6450 2024-05-09T08:35:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f31ff7fe640 1 -- 172.21.15.107:0/3161896028 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3210010740 con 0x5622cd2f6450 2024-05-09T08:35:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f3229414740 1 -- 172.21.15.107:0/3161896028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3210030070 con 0x5622cd2f6450 2024-05-09T08:35:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f31ff7fe640 1 -- 172.21.15.107:0/3161896028 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f321001d040 con 0x5622cd2f6450 2024-05-09T08:35:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f3229414740 10 client.16245 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.625+0000 7f3229414740 1 -- 172.21.15.107:0/3161896028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5622cd377200 con 0x5622cd2f6450 2024-05-09T08:35:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.626+0000 7f31ff7fe640 1 -- 172.21.15.107:0/3161896028 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3210030070 con 0x5622cd2f6450 2024-05-09T08:35:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.626+0000 7f31ff7fe640 1 -- 172.21.15.107:0/3161896028 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f321002a340 con 0x5622cd2f6450 2024-05-09T08:35:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.626+0000 7f3229414740 10 client.16245 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.626+0000 7f3229414740 10 client.16245 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.626+0000 7f3229414740 20 client.16245 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.626+0000 7f3229414740 1 --2- 172.21.15.107:0/3161896028 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5622cd3776f0 0x5622cd379b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.626+0000 7f3229414740 4 client.16245 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:35:15.633 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.626+0000 7f3229414740 1 -- 172.21.15.107:0/3161896028 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5622cd262dc0 con 0x5622cd3776f0 2024-05-09T08:35:15.633 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.626+0000 7f321f7fe640 1 --2- 172.21.15.107:0/3161896028 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5622cd3776f0 0x5622cd379b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:15.633 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.626+0000 7f321f7fe640 1 --2- 172.21.15.107:0/3161896028 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5622cd3776f0 0x5622cd379b10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f53cdb2107bd083a server_cookie=93779c565595d750 in_seq=0 out_seq=0 2024-05-09T08:35:15.633 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.626+0000 7f31ff7fe640 10 client.16245 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.646+0000 7f31ff7fe640 1 -- 172.21.15.107:0/3161896028 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5622cd262dc0 con 0x5622cd3776f0 2024-05-09T08:35:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.646+0000 7f31ff7fe640 10 client.16245 handle_command_reply: tid=0 2024-05-09T08:35:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.646+0000 7f3229414740 1 client.16245 shutdown 2024-05-09T08:35:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.646+0000 7f3229414740 1 -- 172.21.15.107:0/3161896028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6450 msgr2=0x5622cd304880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.646+0000 7f3229414740 1 --2- 172.21.15.107:0/3161896028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6450 0x5622cd304880 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f32100106e0 tx=0x7f321000a920 comp rx=0 tx=0).stop 2024-05-09T08:35:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.646+0000 7f3229414740 1 -- 172.21.15.107:0/3161896028 shutdown_connections 2024-05-09T08:35:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.646+0000 7f3229414740 1 --2- 172.21.15.107:0/3161896028 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5622cd3776f0 0x5622cd379b10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.646+0000 7f3229414740 1 --2- 172.21.15.107:0/3161896028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5622cd301630 0x5622cd36f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.646+0000 7f31ff7fe640 0 client.16245 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.646+0000 7f3229414740 1 --2- 172.21.15.107:0/3161896028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622cd2f6d70 0x5622cd304dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.646+0000 7f3229414740 1 --2- 172.21.15.107:0/3161896028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5622cd2f6450 0x5622cd304880 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.646+0000 7f3229414740 1 -- 172.21.15.107:0/3161896028 >> 172.21.15.107:0/3161896028 conn(0x5622cd2f2960 msgr2=0x5622cd30d730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.646+0000 7f3229414740 1 -- 172.21.15.107:0/3161896028 shutdown_connections 2024-05-09T08:35:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.646+0000 7f3229414740 1 -- 172.21.15.107:0/3161896028 wait complete. 2024-05-09T08:35:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.646+0000 7f3229414740 20 client.16245 trim_cache size 0 max 16384 2024-05-09T08:35:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:35:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:35:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:35:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:35:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.647+0000 7f322662a640 1 -- 172.21.15.107:0/4154901670 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f32080a2da0 msgr2=0x7f32080a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.647+0000 7f322662a640 1 --2- 172.21.15.107:0/4154901670 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f32080a2da0 0x7f32080a5260 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3218004af0 tx=0x7f3218005d80 comp rx=0 tx=0).stop 2024-05-09T08:35:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.647+0000 7f322662a640 1 -- 172.21.15.107:0/4154901670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3220103300 msgr2=0x7f3220198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.647+0000 7f322662a640 1 --2- 172.21.15.107:0/4154901670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3220103300 0x7f3220198490 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f321400e9f0 tx=0x7f321400edf0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.647+0000 7f322662a640 1 -- 172.21.15.107:0/4154901670 shutdown_connections 2024-05-09T08:35:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.647+0000 7f322662a640 1 --2- 172.21.15.107:0/4154901670 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f32080a2da0 0x7f32080a5260 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.647+0000 7f322662a640 1 --2- 172.21.15.107:0/4154901670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f322010be00 0x7f322019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.647+0000 7f322662a640 1 --2- 172.21.15.107:0/4154901670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3220103300 0x7f3220198490 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.647+0000 7f322662a640 1 --2- 172.21.15.107:0/4154901670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32201029e0 0x7f3220197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.647+0000 7f322662a640 1 -- 172.21.15.107:0/4154901670 >> 172.21.15.107:0/4154901670 conn(0x7f32200fc200 msgr2=0x7f32200fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.647+0000 7f322662a640 1 -- 172.21.15.107:0/4154901670 shutdown_connections 2024-05-09T08:35:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:15.647+0000 7f322662a640 1 -- 172.21.15.107:0/4154901670 wait complete. 2024-05-09T08:35:16.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:16 smithi157 ceph-mon[33614]: pgmap v60: 129 pgs: 129 active+clean; 379 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 24 MiB/s wr, 62 op/s 2024-05-09T08:35:16.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:16 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:35:16.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:16 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:35:16.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:16 smithi110 ceph-mon[33622]: pgmap v60: 129 pgs: 129 active+clean; 379 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 24 MiB/s wr, 62 op/s 2024-05-09T08:35:16.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:16 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:35:16.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:16 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:35:17.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:16 smithi107 ceph-mon[30122]: pgmap v60: 129 pgs: 129 active+clean; 379 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 24 MiB/s wr, 62 op/s 2024-05-09T08:35:17.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:16 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:35:17.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:16 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:35:17.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:17 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:35:17.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:17 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:35:18.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:17 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:35:18.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:18 smithi157 ceph-mon[33614]: pgmap v61: 129 pgs: 129 active+clean; 531 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 37 MiB/s wr, 76 op/s 2024-05-09T08:35:18.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:18 smithi110 ceph-mon[33622]: pgmap v61: 129 pgs: 129 active+clean; 531 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 37 MiB/s wr, 76 op/s 2024-05-09T08:35:19.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:18 smithi107 ceph-mon[30122]: pgmap v61: 129 pgs: 129 active+clean; 531 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 37 MiB/s wr, 76 op/s 2024-05-09T08:35:20.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:20 smithi157 ceph-mon[33614]: pgmap v62: 129 pgs: 129 active+clean; 531 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-05-09T08:35:20.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:20 smithi110 ceph-mon[33622]: pgmap v62: 129 pgs: 129 active+clean; 531 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-05-09T08:35:21.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:20 smithi107 ceph-mon[30122]: pgmap v62: 129 pgs: 129 active+clean; 531 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-05-09T08:35:22.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:22 smithi110 ceph-mon[33622]: pgmap v63: 129 pgs: 129 active+clean; 531 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-05-09T08:35:23.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:22 smithi107 ceph-mon[30122]: pgmap v63: 129 pgs: 129 active+clean; 531 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-05-09T08:35:23.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:22 smithi157 ceph-mon[33614]: pgmap v63: 129 pgs: 129 active+clean; 531 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-05-09T08:35:24.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:24 smithi110 ceph-mon[33622]: pgmap v64: 129 pgs: 129 active+clean; 683 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 38 MiB/s wr, 29 op/s 2024-05-09T08:35:25.031 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:24 smithi107 ceph-mon[30122]: pgmap v64: 129 pgs: 129 active+clean; 683 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 38 MiB/s wr, 29 op/s 2024-05-09T08:35:25.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:24 smithi157 ceph-mon[33614]: pgmap v64: 129 pgs: 129 active+clean; 683 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 38 MiB/s wr, 29 op/s 2024-05-09T08:35:26.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:26 smithi110 ceph-mon[33622]: pgmap v65: 129 pgs: 129 active+clean; 683 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-05-09T08:35:27.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:26 smithi107 ceph-mon[30122]: pgmap v65: 129 pgs: 129 active+clean; 683 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-05-09T08:35:27.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:26 smithi157 ceph-mon[33614]: pgmap v65: 129 pgs: 129 active+clean; 683 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-05-09T08:35:27.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:27 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:35:28.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:27 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:35:28.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:27 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:35:28.742 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.741+0000 7f34829e2640 1 -- 172.21.15.107:0/2761515899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347c106720 msgr2=0x7f347c106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.741+0000 7f34829e2640 1 --2- 172.21.15.107:0/2761515899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347c106720 0x7f347c106b00 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f346c009930 tx=0x7f346c02a420 comp rx=0 tx=0).stop 2024-05-09T08:35:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.741+0000 7f34829e2640 1 -- 172.21.15.107:0/2761515899 shutdown_connections 2024-05-09T08:35:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.741+0000 7f34829e2640 1 --2- 172.21.15.107:0/2761515899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f347c101120 0x7f347c10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.741+0000 7f34829e2640 1 --2- 172.21.15.107:0/2761515899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f347c100760 0x7f347c100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.741+0000 7f34829e2640 1 --2- 172.21.15.107:0/2761515899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347c106720 0x7f347c106b00 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.741+0000 7f34829e2640 1 -- 172.21.15.107:0/2761515899 >> 172.21.15.107:0/2761515899 conn(0x7f347c0778b0 msgr2=0x7f347c077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.741+0000 7f34829e2640 1 -- 172.21.15.107:0/2761515899 shutdown_connections 2024-05-09T08:35:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.741+0000 7f34829e2640 1 -- 172.21.15.107:0/2761515899 wait complete. 2024-05-09T08:35:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.742+0000 7f34829e2640 1 Processor -- start 2024-05-09T08:35:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.742+0000 7f34829e2640 1 -- start start 2024-05-09T08:35:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.742+0000 7f34829e2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f347c100760 0x7f347c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.742+0000 7f34829e2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347c101120 0x7f347c198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.742+0000 7f34829e2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f347c106720 0x7f347c19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.742+0000 7f34829e2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f347c1109e0 con 0x7f347c101120 2024-05-09T08:35:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.742+0000 7f34829e2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f347c110860 con 0x7f347c100760 2024-05-09T08:35:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.742+0000 7f34829e2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f347c110b60 con 0x7f347c106720 2024-05-09T08:35:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.742+0000 7f3473fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347c101120 0x7f347c198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.742+0000 7f3473fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347c101120 0x7f347c198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51542/0 (socket says 172.21.15.107:51542) 2024-05-09T08:35:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.742+0000 7f3473fff640 1 -- 172.21.15.107:0/165291621 learned_addr learned my addr 172.21.15.107:0/165291621 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.742+0000 7f3480f7f640 1 --2- 172.21.15.107:0/165291621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f347c106720 0x7f347c19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.742+0000 7f347bfff640 1 --2- 172.21.15.107:0/165291621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f347c100760 0x7f347c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.743+0000 7f3473fff640 1 -- 172.21.15.107:0/165291621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f347c106720 msgr2=0x7f347c19c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.743+0000 7f3473fff640 1 --2- 172.21.15.107:0/165291621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f347c106720 0x7f347c19c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.743+0000 7f3473fff640 1 -- 172.21.15.107:0/165291621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f347c100760 msgr2=0x7f347c198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.743+0000 7f3473fff640 1 --2- 172.21.15.107:0/165291621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f347c100760 0x7f347c198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.743+0000 7f3473fff640 1 -- 172.21.15.107:0/165291621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347c19d000 con 0x7f347c101120 2024-05-09T08:35:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.743+0000 7f3473fff640 1 --2- 172.21.15.107:0/165291621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347c101120 0x7f347c198550 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f346400d950 tx=0x7f346400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.743+0000 7f3479ffb640 1 -- 172.21.15.107:0/165291621 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3464017030 con 0x7f347c101120 2024-05-09T08:35:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.743+0000 7f3479ffb640 1 -- 172.21.15.107:0/165291621 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3464004020 con 0x7f347c101120 2024-05-09T08:35:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.743+0000 7f34829e2640 1 -- 172.21.15.107:0/165291621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f347c19d2f0 con 0x7f347c101120 2024-05-09T08:35:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.743+0000 7f3479ffb640 1 -- 172.21.15.107:0/165291621 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34640047a0 con 0x7f347c101120 2024-05-09T08:35:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.743+0000 7f34829e2640 1 -- 172.21.15.107:0/165291621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f347c1a4b80 con 0x7f347c101120 2024-05-09T08:35:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.745+0000 7f3479ffb640 1 -- 172.21.15.107:0/165291621 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f34640041d0 con 0x7f347c101120 2024-05-09T08:35:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.746+0000 7f34857c5740 1 Processor -- start 2024-05-09T08:35:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.746+0000 7f34857c5740 1 -- start start 2024-05-09T08:35:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.746+0000 7f3479ffb640 1 --2- 172.21.15.107:0/165291621 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f34680a2da0 0x7f34680a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.746+0000 7f34857c5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f16450 0x560406f16830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f34857c5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f16d70 0x560406f210f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f34857c5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f21630 0x560406f23a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f34857c5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560406e7b340 con 0x560406f21630 2024-05-09T08:35:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f34857c5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560406e23290 con 0x560406f16d70 2024-05-09T08:35:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f34857c5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560406e77300 con 0x560406f16450 2024-05-09T08:35:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f3480f7f640 1 --2- 172.21.15.107:0/165291621 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f34680a2da0 0x7f34680a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f3479ffb640 1 -- 172.21.15.107:0/165291621 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f34640cf1a0 con 0x7f347c101120 2024-05-09T08:35:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f3480f7f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f21630 0x560406f23a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f3480f7f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f21630 0x560406f23a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51548/0 (socket says 172.21.15.107:51548) 2024-05-09T08:35:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f3480f7f640 1 -- 172.21.15.107:0/504234982 learned_addr learned my addr 172.21.15.107:0/504234982 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f347bfff640 1 --2- 172.21.15.107:0/504234982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f16450 0x560406f16830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f3473fff640 1 --2- 172.21.15.107:0/504234982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f16d70 0x560406f210f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f3480f7f640 1 -- 172.21.15.107:0/504234982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f16450 msgr2=0x560406f16830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f3480f7f640 1 --2- 172.21.15.107:0/504234982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f16450 0x560406f16830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f3480f7f640 1 -- 172.21.15.107:0/504234982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f16d70 msgr2=0x560406f210f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f3480f7f640 1 --2- 172.21.15.107:0/504234982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f16d70 0x560406f210f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f3480f7f640 1 -- 172.21.15.107:0/504234982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560406e478e0 con 0x560406f21630 2024-05-09T08:35:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f3480f7f640 1 --2- 172.21.15.107:0/165291621 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f34680a2da0 0x7f34680a5260 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3474009fd0 tx=0x7f3474009f40 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f3480f7f640 1 --2- 172.21.15.107:0/504234982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f21630 0x560406f23a20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f3474011750 tx=0x7f347401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71bd17b423cbb2f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.747+0000 7f34727fc640 1 -- 172.21.15.107:0/504234982 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3474024070 con 0x560406f21630 2024-05-09T08:35:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.748+0000 7f34727fc640 1 -- 172.21.15.107:0/504234982 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3474011cb0 con 0x560406f21630 2024-05-09T08:35:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.748+0000 7f34727fc640 1 -- 172.21.15.107:0/504234982 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f347401fd40 con 0x560406f21630 2024-05-09T08:35:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.748+0000 7f34857c5740 1 -- 172.21.15.107:0/504234982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f21630 msgr2=0x560406f23a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.748+0000 7f34857c5740 1 --2- 172.21.15.107:0/504234982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f21630 0x560406f23a20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f3474011750 tx=0x7f347401d000 comp rx=0 tx=0).stop 2024-05-09T08:35:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.748+0000 7f34857c5740 1 -- 172.21.15.107:0/504234982 shutdown_connections 2024-05-09T08:35:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.748+0000 7f34857c5740 1 --2- 172.21.15.107:0/504234982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f21630 0x560406f23a20 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.748+0000 7f34857c5740 1 --2- 172.21.15.107:0/504234982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f16d70 0x560406f210f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.748+0000 7f34857c5740 1 --2- 172.21.15.107:0/504234982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f16450 0x560406f16830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.748+0000 7f34857c5740 1 -- 172.21.15.107:0/504234982 >> 172.21.15.107:0/504234982 conn(0x560406f12960 msgr2=0x560406f12d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.748+0000 7f34857c5740 1 -- 172.21.15.107:0/504234982 shutdown_connections 2024-05-09T08:35:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.748+0000 7f34857c5740 1 -- 172.21.15.107:0/504234982 wait complete. 2024-05-09T08:35:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.749+0000 7f34857c5740 1 -- start start 2024-05-09T08:35:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.749+0000 7f34857c5740 1 Processor -- start 2024-05-09T08:35:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.749+0000 7f34857c5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16450 0x560406f1e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.749+0000 7f34857c5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f16d70 0x560406f1eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.749+0000 7f34857c5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f21630 0x560406f1f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.749+0000 7f34857c5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560406f24200 con 0x560406f16450 2024-05-09T08:35:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.749+0000 7f34857c5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560406e36790 con 0x560406f21630 2024-05-09T08:35:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.749+0000 7f34857c5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560406f24380 con 0x560406f16d70 2024-05-09T08:35:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.749+0000 7f34857c5740 10 client.0 _ll_register_callbacks cb 0x560406e71a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.749+0000 7f347bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16450 0x560406f1e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.749+0000 7f3473fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f16d70 0x560406f1eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.749+0000 7f3473fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f16d70 0x560406f1eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:48800/0 (socket says 172.21.15.107:48800) 2024-05-09T08:35:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.749+0000 7f3473fff640 1 -- 172.21.15.107:0/3751395223 learned_addr learned my addr 172.21.15.107:0/3751395223 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.749+0000 7f3480f7f640 1 --2- 172.21.15.107:0/3751395223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f21630 0x560406f1f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.749+0000 7f347bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16450 0x560406f1e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51564/0 (socket says 172.21.15.107:51564) 2024-05-09T08:35:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f3473fff640 1 -- 172.21.15.107:0/3751395223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f21630 msgr2=0x560406f1f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f3473fff640 1 --2- 172.21.15.107:0/3751395223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f21630 0x560406f1f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f3473fff640 1 -- 172.21.15.107:0/3751395223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16450 msgr2=0x560406f1e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f3473fff640 1 --2- 172.21.15.107:0/3751395223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16450 0x560406f1e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f3473fff640 1 -- 172.21.15.107:0/3751395223 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560406b8f210 con 0x560406f16d70 2024-05-09T08:35:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f3480f7f640 1 --2- 172.21.15.107:0/3751395223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f21630 0x560406f1f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f347bfff640 1 --2- 172.21.15.107:0/3751395223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16450 0x560406f1e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:28.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f3473fff640 1 --2- 172.21.15.107:0/3751395223 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f16d70 0x560406f1eaf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3464009610 tx=0x7f346409adc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c530ac34e833ff6e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:28.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f345b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:35:28.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f345b7fe640 1 -- 172.21.15.107:0/3751395223 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346409f710 con 0x560406f16d70 2024-05-09T08:35:28.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f34857c5740 1 -- 172.21.15.107:0/3751395223 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3474011cb0 con 0x560406f16d70 2024-05-09T08:35:28.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f345b7fe640 1 -- 172.21.15.107:0/3751395223 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f34640979d0 con 0x560406f16d70 2024-05-09T08:35:28.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f345b7fe640 1 -- 172.21.15.107:0/3751395223 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f34640a3fd0 con 0x560406f16d70 2024-05-09T08:35:28.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f34857c5740 10 client.25729 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:28.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f34857c5740 1 -- 172.21.15.107:0/3751395223 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560406c5b620 con 0x560406f16d70 2024-05-09T08:35:28.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f345b7fe640 1 -- 172.21.15.107:0/3751395223 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3474011cb0 con 0x560406f16d70 2024-05-09T08:35:28.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f345b7fe640 1 -- 172.21.15.107:0/3751395223 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f34640ccce0 con 0x560406f16d70 2024-05-09T08:35:28.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f34857c5740 10 client.25729 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:28.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.750+0000 7f34857c5740 10 client.25729 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:28.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.751+0000 7f34857c5740 20 client.25729 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.751+0000 7f34857c5740 1 --2- 172.21.15.107:0/3751395223 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560406f312b0 0x560406f336e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.751+0000 7f34857c5740 4 client.25729 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.751+0000 7f34857c5740 1 -- 172.21.15.107:0/3751395223 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560406e88a70 con 0x560406f312b0 2024-05-09T08:35:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.751+0000 7f347bfff640 1 --2- 172.21.15.107:0/3751395223 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560406f312b0 0x560406f336e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.751+0000 7f345b7fe640 10 client.25729 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.751+0000 7f347bfff640 1 --2- 172.21.15.107:0/3751395223 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560406f312b0 0x560406f336e0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ad7c1d32a8fac03 server_cookie=513351afbba406c4 in_seq=0 out_seq=0 2024-05-09T08:35:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.752+0000 7f345b7fe640 1 -- 172.21.15.107:0/3751395223 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560406e88a70 con 0x560406f312b0 2024-05-09T08:35:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.752+0000 7f345b7fe640 10 client.25729 handle_command_reply: tid=0 2024-05-09T08:35:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.752+0000 7f34857c5740 1 client.25729 shutdown 2024-05-09T08:35:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.752+0000 7f34857c5740 1 -- 172.21.15.107:0/3751395223 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f16d70 msgr2=0x560406f1eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.752+0000 7f34857c5740 1 --2- 172.21.15.107:0/3751395223 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f16d70 0x560406f1eaf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3464009610 tx=0x7f346409adc0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.752+0000 7f34857c5740 1 -- 172.21.15.107:0/3751395223 shutdown_connections 2024-05-09T08:35:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.752+0000 7f34857c5740 1 --2- 172.21.15.107:0/3751395223 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560406f312b0 0x560406f336e0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.752+0000 7f34857c5740 1 --2- 172.21.15.107:0/3751395223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f21630 0x560406f1f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.752+0000 7f345b7fe640 0 client.25729 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.752+0000 7f34857c5740 1 --2- 172.21.15.107:0/3751395223 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f16d70 0x560406f1eaf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.752+0000 7f34857c5740 1 --2- 172.21.15.107:0/3751395223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16450 0x560406f1e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.752+0000 7f34857c5740 1 -- 172.21.15.107:0/3751395223 >> 172.21.15.107:0/3751395223 conn(0x560406f12960 msgr2=0x560406f186e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.752+0000 7f34857c5740 1 -- 172.21.15.107:0/3751395223 shutdown_connections 2024-05-09T08:35:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.752+0000 7f34857c5740 1 -- 172.21.15.107:0/3751395223 wait complete. 2024-05-09T08:35:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.752+0000 7f34857c5740 20 client.25729 trim_cache size 0 max 16384 2024-05-09T08:35:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.771+0000 7f34857c5740 1 Processor -- start 2024-05-09T08:35:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.771+0000 7f34857c5740 1 -- start start 2024-05-09T08:35:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.771+0000 7f34857c5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f16450 0x560406f2d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.771+0000 7f34857c5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16d70 0x560406f18e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.771+0000 7f34857c5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f21630 0x560406f12d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.771+0000 7f34857c5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f347c1b1280 con 0x560406f16d70 2024-05-09T08:35:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.771+0000 7f34857c5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f347c1aeb50 con 0x560406f16450 2024-05-09T08:35:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.771+0000 7f34857c5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f347c1ac9a0 con 0x560406f21630 2024-05-09T08:35:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.772+0000 7f3473fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16d70 0x560406f18e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.772+0000 7f3473fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16d70 0x560406f18e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51576/0 (socket says 172.21.15.107:51576) 2024-05-09T08:35:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.772+0000 7f3473fff640 1 -- 172.21.15.107:0/3960314141 learned_addr learned my addr 172.21.15.107:0/3960314141 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.772+0000 7f3480f7f640 1 --2- 172.21.15.107:0/3960314141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f21630 0x560406f12d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.772+0000 7f347bfff640 1 --2- 172.21.15.107:0/3960314141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f16450 0x560406f2d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.772+0000 7f3473fff640 1 -- 172.21.15.107:0/3960314141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f21630 msgr2=0x560406f12d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.772+0000 7f3473fff640 1 --2- 172.21.15.107:0/3960314141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f21630 0x560406f12d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.772+0000 7f3473fff640 1 -- 172.21.15.107:0/3960314141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f16450 msgr2=0x560406f2d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.772+0000 7f3473fff640 1 --2- 172.21.15.107:0/3960314141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f16450 0x560406f2d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.772+0000 7f3473fff640 1 -- 172.21.15.107:0/3960314141 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560406f1d560 con 0x560406f16d70 2024-05-09T08:35:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.772+0000 7f3473fff640 1 --2- 172.21.15.107:0/3960314141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16d70 0x560406f18e60 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f346409b2f0 tx=0x7f346409b4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f35b8e843283770 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.772+0000 7f3471ffb640 1 -- 172.21.15.107:0/3960314141 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3464014070 con 0x560406f16d70 2024-05-09T08:35:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.772+0000 7f3471ffb640 1 -- 172.21.15.107:0/3960314141 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f346407b050 con 0x560406f16d70 2024-05-09T08:35:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.772+0000 7f3471ffb640 1 -- 172.21.15.107:0/3960314141 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346409f710 con 0x560406f16d70 2024-05-09T08:35:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.773+0000 7f34857c5740 1 -- 172.21.15.107:0/3960314141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16d70 msgr2=0x560406f18e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.773+0000 7f34857c5740 1 --2- 172.21.15.107:0/3960314141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16d70 0x560406f18e60 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f346409b2f0 tx=0x7f346409b4e0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.773+0000 7f34857c5740 1 -- 172.21.15.107:0/3960314141 shutdown_connections 2024-05-09T08:35:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.773+0000 7f34857c5740 1 --2- 172.21.15.107:0/3960314141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f21630 0x560406f12d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.773+0000 7f34857c5740 1 --2- 172.21.15.107:0/3960314141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16d70 0x560406f18e60 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.773+0000 7f34857c5740 1 --2- 172.21.15.107:0/3960314141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f16450 0x560406f2d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.773+0000 7f34857c5740 1 -- 172.21.15.107:0/3960314141 >> 172.21.15.107:0/3960314141 conn(0x560406f12960 msgr2=0x560406f11a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.773+0000 7f34857c5740 1 -- 172.21.15.107:0/3960314141 shutdown_connections 2024-05-09T08:35:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.773+0000 7f34857c5740 1 -- 172.21.15.107:0/3960314141 wait complete. 2024-05-09T08:35:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.774+0000 7f34857c5740 1 -- start start 2024-05-09T08:35:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.774+0000 7f34857c5740 1 Processor -- start 2024-05-09T08:35:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.774+0000 7f34857c5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16450 0x560406f250f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.774+0000 7f34857c5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f16d70 0x560406f25630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.774+0000 7f34857c5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f21630 0x560406f25b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.774+0000 7f34857c5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f347c1ad010 con 0x560406f16450 2024-05-09T08:35:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.774+0000 7f34857c5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f347c1ae480 con 0x560406f16d70 2024-05-09T08:35:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.774+0000 7f34857c5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f347c1ad6e0 con 0x560406f21630 2024-05-09T08:35:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.774+0000 7f34857c5740 10 client.0 _ll_register_callbacks cb 0x7f346409aea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.774+0000 7f347bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16450 0x560406f250f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.774+0000 7f347bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16450 0x560406f250f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51592/0 (socket says 172.21.15.107:51592) 2024-05-09T08:35:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.774+0000 7f347bfff640 1 -- 172.21.15.107:0/460880046 learned_addr learned my addr 172.21.15.107:0/460880046 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:28.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.774+0000 7f3480f7f640 1 --2- 172.21.15.107:0/460880046 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f21630 0x560406f25b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.774+0000 7f3473fff640 1 --2- 172.21.15.107:0/460880046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f16d70 0x560406f25630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.774+0000 7f347bfff640 1 -- 172.21.15.107:0/460880046 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f21630 msgr2=0x560406f25b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:28.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.774+0000 7f347bfff640 1 --2- 172.21.15.107:0/460880046 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f21630 0x560406f25b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.775+0000 7f347bfff640 1 -- 172.21.15.107:0/460880046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f16d70 msgr2=0x560406f25630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:28.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.775+0000 7f347bfff640 1 --2- 172.21.15.107:0/460880046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f16d70 0x560406f25630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.775+0000 7f347bfff640 1 -- 172.21.15.107:0/460880046 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560406f96960 con 0x560406f16450 2024-05-09T08:35:28.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.775+0000 7f347bfff640 1 --2- 172.21.15.107:0/460880046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16450 0x560406f250f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f346c009900 tx=0x7f346c031ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c053eaa009550805 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:28.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.775+0000 7f345b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:28.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.775+0000 7f345b7fe640 1 -- 172.21.15.107:0/460880046 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c04a070 con 0x560406f16450 2024-05-09T08:35:28.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.775+0000 7f345b7fe640 1 -- 172.21.15.107:0/460880046 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f346c0042c0 con 0x560406f16450 2024-05-09T08:35:28.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.775+0000 7f34857c5740 1 -- 172.21.15.107:0/460880046 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f346407b050 con 0x560406f16450 2024-05-09T08:35:28.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.775+0000 7f345b7fe640 1 -- 172.21.15.107:0/460880046 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f346c055050 con 0x560406f16450 2024-05-09T08:35:28.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.775+0000 7f34857c5740 10 client.16263 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:28.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.775+0000 7f34857c5740 1 -- 172.21.15.107:0/460880046 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560406f97cd0 con 0x560406f16450 2024-05-09T08:35:28.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.775+0000 7f345b7fe640 1 -- 172.21.15.107:0/460880046 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f346407b050 con 0x560406f16450 2024-05-09T08:35:28.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.776+0000 7f345b7fe640 1 -- 172.21.15.107:0/460880046 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f346c046750 con 0x560406f16450 2024-05-09T08:35:28.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.776+0000 7f34857c5740 10 client.16263 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:28.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.776+0000 7f34857c5740 10 client.16263 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:28.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.776+0000 7f34857c5740 20 client.16263 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:28.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.776+0000 7f34857c5740 1 --2- 172.21.15.107:0/460880046 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560406f98460 0x560406f9a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:28.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.776+0000 7f34857c5740 4 client.16263 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:35:28.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.776+0000 7f34857c5740 1 -- 172.21.15.107:0/460880046 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560406e82dc0 con 0x560406f98460 2024-05-09T08:35:28.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.776+0000 7f3473fff640 1 --2- 172.21.15.107:0/460880046 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560406f98460 0x560406f9a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:28.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.776+0000 7f345b7fe640 10 client.16263 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:28.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.776+0000 7f3473fff640 1 --2- 172.21.15.107:0/460880046 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560406f98460 0x560406f9a880 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46cd25fbf1be3e11 server_cookie=32fcda2c20bc4c51 in_seq=0 out_seq=0 2024-05-09T08:35:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.792+0000 7f345b7fe640 1 -- 172.21.15.107:0/460880046 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560406e82dc0 con 0x560406f98460 2024-05-09T08:35:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.792+0000 7f345b7fe640 10 client.16263 handle_command_reply: tid=0 2024-05-09T08:35:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.792+0000 7f34857c5740 1 client.16263 shutdown 2024-05-09T08:35:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.792+0000 7f34857c5740 1 -- 172.21.15.107:0/460880046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16450 msgr2=0x560406f250f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.792+0000 7f34857c5740 1 --2- 172.21.15.107:0/460880046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16450 0x560406f250f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f346c009900 tx=0x7f346c031ce0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.792+0000 7f34857c5740 1 -- 172.21.15.107:0/460880046 shutdown_connections 2024-05-09T08:35:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.792+0000 7f34857c5740 1 --2- 172.21.15.107:0/460880046 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560406f98460 0x560406f9a880 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.792+0000 7f34857c5740 1 --2- 172.21.15.107:0/460880046 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560406f21630 0x560406f25b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.792+0000 7f345b7fe640 0 client.16263 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.792+0000 7f34857c5740 1 --2- 172.21.15.107:0/460880046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560406f16d70 0x560406f25630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.792+0000 7f34857c5740 1 --2- 172.21.15.107:0/460880046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560406f16450 0x560406f250f0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.792+0000 7f34857c5740 1 -- 172.21.15.107:0/460880046 >> 172.21.15.107:0/460880046 conn(0x560406f12960 msgr2=0x560406f11a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.792+0000 7f34857c5740 1 -- 172.21.15.107:0/460880046 shutdown_connections 2024-05-09T08:35:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.792+0000 7f34857c5740 1 -- 172.21.15.107:0/460880046 wait complete. 2024-05-09T08:35:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.792+0000 7f34857c5740 20 client.16263 trim_cache size 0 max 16384 2024-05-09T08:35:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:35:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:35:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:35:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:35:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.793+0000 7f34829e2640 1 -- 172.21.15.107:0/165291621 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f34680a2da0 msgr2=0x7f34680a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.793+0000 7f34829e2640 1 --2- 172.21.15.107:0/165291621 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f34680a2da0 0x7f34680a5260 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3474009fd0 tx=0x7f3474009f40 comp rx=0 tx=0).stop 2024-05-09T08:35:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.793+0000 7f34829e2640 1 -- 172.21.15.107:0/165291621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347c101120 msgr2=0x7f347c198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.793+0000 7f34829e2640 1 --2- 172.21.15.107:0/165291621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347c101120 0x7f347c198550 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f346400d950 tx=0x7f346400de10 comp rx=0 tx=0).stop 2024-05-09T08:35:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.793+0000 7f34829e2640 1 -- 172.21.15.107:0/165291621 shutdown_connections 2024-05-09T08:35:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.793+0000 7f34829e2640 1 --2- 172.21.15.107:0/165291621 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f34680a2da0 0x7f34680a5260 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.793+0000 7f34829e2640 1 --2- 172.21.15.107:0/165291621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f347c106720 0x7f347c19c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.793+0000 7f34829e2640 1 --2- 172.21.15.107:0/165291621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347c101120 0x7f347c198550 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.793+0000 7f34829e2640 1 --2- 172.21.15.107:0/165291621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f347c100760 0x7f347c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.793+0000 7f34829e2640 1 -- 172.21.15.107:0/165291621 >> 172.21.15.107:0/165291621 conn(0x7f347c0778b0 msgr2=0x7f347c0ffa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.793+0000 7f34829e2640 1 -- 172.21.15.107:0/165291621 shutdown_connections 2024-05-09T08:35:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:28.793+0000 7f34829e2640 1 -- 172.21.15.107:0/165291621 wait complete. 2024-05-09T08:35:28.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:28 smithi110 ceph-mon[33622]: pgmap v66: 129 pgs: 129 active+clean; 836 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 38 MiB/s wr, 29 op/s 2024-05-09T08:35:28.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:28 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:35:28.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:28 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:35:28.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:28 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:35:28.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:28 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:35:28.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:28 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:35:29.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:28 smithi107 ceph-mon[30122]: pgmap v66: 129 pgs: 129 active+clean; 836 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 38 MiB/s wr, 29 op/s 2024-05-09T08:35:29.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:28 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:35:29.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:28 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:35:29.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:28 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:35:29.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:28 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:35:29.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:28 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:35:29.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:28 smithi157 ceph-mon[33614]: pgmap v66: 129 pgs: 129 active+clean; 836 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 38 MiB/s wr, 29 op/s 2024-05-09T08:35:29.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:28 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:35:29.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:28 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:35:29.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:28 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:35:29.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:28 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:35:29.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:28 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:35:30.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:30 smithi110 ceph-mon[33622]: pgmap v67: 129 pgs: 129 active+clean; 836 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:35:31.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:30 smithi107 ceph-mon[30122]: pgmap v67: 129 pgs: 129 active+clean; 836 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:35:31.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:30 smithi157 ceph-mon[33614]: pgmap v67: 129 pgs: 129 active+clean; 836 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:35:32.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:32 smithi110 ceph-mon[33622]: pgmap v68: 129 pgs: 129 active+clean; 836 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:35:33.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:32 smithi107 ceph-mon[30122]: pgmap v68: 129 pgs: 129 active+clean; 836 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:35:33.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:32 smithi157 ceph-mon[33614]: pgmap v68: 129 pgs: 129 active+clean; 836 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:35:34.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:34 smithi110 ceph-mon[33622]: pgmap v69: 129 pgs: 129 active+clean; 984 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 38 MiB/s wr, 27 op/s 2024-05-09T08:35:35.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:34 smithi107 ceph-mon[30122]: pgmap v69: 129 pgs: 129 active+clean; 984 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 38 MiB/s wr, 27 op/s 2024-05-09T08:35:35.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:34 smithi157 ceph-mon[33614]: pgmap v69: 129 pgs: 129 active+clean; 984 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 38 MiB/s wr, 27 op/s 2024-05-09T08:35:36.219 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:35:36.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:35:36.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.330+0000 7ff506cb3640 1 -- 172.21.15.107:0/2261259647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5001010c0 msgr2=0x7ff50010d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.330+0000 7ff506cb3640 1 --2- 172.21.15.107:0/2261259647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5001010c0 0x7ff50010d5d0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009cd0 tx=0x7ff4f802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.330+0000 7ff506cb3640 1 -- 172.21.15.107:0/2261259647 shutdown_connections 2024-05-09T08:35:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.330+0000 7ff506cb3640 1 --2- 172.21.15.107:0/2261259647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5001010c0 0x7ff50010d5d0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.330+0000 7ff506cb3640 1 --2- 172.21.15.107:0/2261259647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff500100700 0x7ff500100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.330+0000 7ff506cb3640 1 --2- 172.21.15.107:0/2261259647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff500106710 0x7ff500106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.330+0000 7ff506cb3640 1 -- 172.21.15.107:0/2261259647 >> 172.21.15.107:0/2261259647 conn(0x7ff500077830 msgr2=0x7ff500077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.330+0000 7ff506cb3640 1 -- 172.21.15.107:0/2261259647 shutdown_connections 2024-05-09T08:35:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.330+0000 7ff506cb3640 1 -- 172.21.15.107:0/2261259647 wait complete. 2024-05-09T08:35:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.331+0000 7ff506cb3640 1 Processor -- start 2024-05-09T08:35:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.331+0000 7ff506cb3640 1 -- start start 2024-05-09T08:35:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.331+0000 7ff506cb3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff500100700 0x7ff500198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.331+0000 7ff506cb3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5001010c0 0x7ff5001985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.331+0000 7ff506cb3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff500106710 0x7ff50019c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.331+0000 7ff506cb3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff5001109d0 con 0x7ff500106710 2024-05-09T08:35:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.331+0000 7ff506cb3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff500110850 con 0x7ff5001010c0 2024-05-09T08:35:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.331+0000 7ff506cb3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff500110b50 con 0x7ff500100700 2024-05-09T08:35:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.331+0000 7ff505250640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff500106710 0x7ff50019c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.331+0000 7ff505250640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff500106710 0x7ff50019c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41948/0 (socket says 172.21.15.107:41948) 2024-05-09T08:35:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.331+0000 7ff505250640 1 -- 172.21.15.107:0/3747499931 learned_addr learned my addr 172.21.15.107:0/3747499931 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.331+0000 7ff504a4f640 1 --2- 172.21.15.107:0/3747499931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff500100700 0x7ff500198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.331+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/3747499931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5001010c0 0x7ff5001985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.332+0000 7ff505250640 1 -- 172.21.15.107:0/3747499931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff500100700 msgr2=0x7ff500198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.332+0000 7ff505250640 1 --2- 172.21.15.107:0/3747499931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff500100700 0x7ff500198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.332+0000 7ff505250640 1 -- 172.21.15.107:0/3747499931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5001010c0 msgr2=0x7ff5001985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.332+0000 7ff505250640 1 --2- 172.21.15.107:0/3747499931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5001010c0 0x7ff5001985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.332+0000 7ff505250640 1 -- 172.21.15.107:0/3747499931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff50019d050 con 0x7ff500106710 2024-05-09T08:35:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.332+0000 7ff504a4f640 1 --2- 172.21.15.107:0/3747499931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff500100700 0x7ff500198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.332+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/3747499931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5001010c0 0x7ff5001985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.332+0000 7ff505250640 1 --2- 172.21.15.107:0/3747499931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff500106710 0x7ff50019c920 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8033040 tx=0x7ff4f8009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.332+0000 7ff4f5ffb640 1 -- 172.21.15.107:0/3747499931 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8004040 con 0x7ff500106710 2024-05-09T08:35:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.332+0000 7ff506cb3640 1 -- 172.21.15.107:0/3747499931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff50019d2e0 con 0x7ff500106710 2024-05-09T08:35:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.332+0000 7ff4f5ffb640 1 -- 172.21.15.107:0/3747499931 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4f8004690 con 0x7ff500106710 2024-05-09T08:35:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.332+0000 7ff4f5ffb640 1 -- 172.21.15.107:0/3747499931 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8030da0 con 0x7ff500106710 2024-05-09T08:35:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.332+0000 7ff506cb3640 1 -- 172.21.15.107:0/3747499931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5001a4bd0 con 0x7ff500106710 2024-05-09T08:35:36.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.335+0000 7ff4f5ffb640 1 -- 172.21.15.107:0/3747499931 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ff4f80041e0 con 0x7ff500106710 2024-05-09T08:35:36.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.335+0000 7ff5099c4740 1 Processor -- start 2024-05-09T08:35:36.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.335+0000 7ff5099c4740 1 -- start start 2024-05-09T08:35:36.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.335+0000 7ff5099c4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6280 0x55c0c4df6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.335+0000 7ff5099c4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6ba0 0x55c0c4e00d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.335+0000 7ff5099c4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4e01270 0x55c0c4e03660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.335+0000 7ff5099c4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4d5bd00 con 0x55c0c4e01270 2024-05-09T08:35:36.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.335+0000 7ff5099c4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4d034c0 con 0x55c0c4df6280 2024-05-09T08:35:36.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.335+0000 7ff5099c4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4d57cc0 con 0x55c0c4df6ba0 2024-05-09T08:35:36.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.335+0000 7ff4f7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4e01270 0x55c0c4e03660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.335+0000 7ff4f7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4e01270 0x55c0c4e03660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41964/0 (socket says 172.21.15.107:41964) 2024-05-09T08:35:36.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.335+0000 7ff4f7fff640 1 -- 172.21.15.107:0/266249657 learned_addr learned my addr 172.21.15.107:0/266249657 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:36.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.335+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/266249657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6280 0x55c0c4df6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.335+0000 7ff504a4f640 1 --2- 172.21.15.107:0/266249657 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6ba0 0x55c0c4e00d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.336+0000 7ff4f7fff640 1 -- 172.21.15.107:0/266249657 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6ba0 msgr2=0x55c0c4e00d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:36.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.336+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/266249657 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6ba0 0x55c0c4e00d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.336+0000 7ff4f5ffb640 1 --2- 172.21.15.107:0/3747499931 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff4e80a2d00 0x7ff4e80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.336+0000 7ff4f7fff640 1 -- 172.21.15.107:0/266249657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6280 msgr2=0x55c0c4df6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:36.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.336+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/266249657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6280 0x55c0c4df6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.336+0000 7ff4f7fff640 1 -- 172.21.15.107:0/266249657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0c4b397d0 con 0x55c0c4e01270 2024-05-09T08:35:36.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.336+0000 7ff505250640 1 --2- 172.21.15.107:0/3747499931 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff4e80a2d00 0x7ff4e80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.336+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/266249657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4e01270 0x55c0c4e03660 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc012820 tx=0x7ff4fc009720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5544a38ac588c53e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:36.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.336+0000 7ff4f5ffb640 1 -- 172.21.15.107:0/3747499931 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff4f802e030 con 0x7ff500106710 2024-05-09T08:35:36.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.336+0000 7ff4e6ffd640 1 -- 172.21.15.107:0/266249657 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc012e60 con 0x55c0c4e01270 2024-05-09T08:35:36.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.336+0000 7ff4e6ffd640 1 -- 172.21.15.107:0/266249657 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4fc010c80 con 0x55c0c4e01270 2024-05-09T08:35:36.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.336+0000 7ff4e6ffd640 1 -- 172.21.15.107:0/266249657 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc012e60 con 0x55c0c4e01270 2024-05-09T08:35:36.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.337+0000 7ff5099c4740 1 -- 172.21.15.107:0/266249657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4e01270 msgr2=0x55c0c4e03660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.337+0000 7ff505250640 1 --2- 172.21.15.107:0/3747499931 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff4e80a2d00 0x7ff4e80a51c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff4f80edce0 tx=0x7ff4f8087ab0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:36.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.337+0000 7ff5099c4740 1 --2- 172.21.15.107:0/266249657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4e01270 0x55c0c4e03660 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc012820 tx=0x7ff4fc009720 comp rx=0 tx=0).stop 2024-05-09T08:35:36.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.337+0000 7ff5099c4740 1 -- 172.21.15.107:0/266249657 shutdown_connections 2024-05-09T08:35:36.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.337+0000 7ff5099c4740 1 --2- 172.21.15.107:0/266249657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4e01270 0x55c0c4e03660 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.337+0000 7ff5099c4740 1 --2- 172.21.15.107:0/266249657 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6ba0 0x55c0c4e00d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.337+0000 7ff5099c4740 1 --2- 172.21.15.107:0/266249657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6280 0x55c0c4df6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.337+0000 7ff5099c4740 1 -- 172.21.15.107:0/266249657 >> 172.21.15.107:0/266249657 conn(0x55c0c4df25a0 msgr2=0x55c0c4df2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:36.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.337+0000 7ff5099c4740 1 -- 172.21.15.107:0/266249657 shutdown_connections 2024-05-09T08:35:36.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.337+0000 7ff5099c4740 1 -- 172.21.15.107:0/266249657 wait complete. 2024-05-09T08:35:36.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.337+0000 7ff5099c4740 1 -- start start 2024-05-09T08:35:36.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.337+0000 7ff5099c4740 1 Processor -- start 2024-05-09T08:35:36.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff5099c4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6280 0x55c0c4dfe220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff5099c4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4df6ba0 0x55c0c4dfe760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff5099c4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4e01270 0x55c0c4dfeca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff5099c4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4e04060 con 0x55c0c4df6ba0 2024-05-09T08:35:36.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff5099c4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4d16b90 con 0x55c0c4e01270 2024-05-09T08:35:36.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff5099c4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4e041e0 con 0x55c0c4df6280 2024-05-09T08:35:36.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff5099c4740 10 client.0 _ll_register_callbacks cb 0x55c0c4d52da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:36.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff504a4f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4df6ba0 0x55c0c4dfe760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff504a4f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4df6ba0 0x55c0c4dfe760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41970/0 (socket says 172.21.15.107:41970) 2024-05-09T08:35:36.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff504a4f640 1 -- 172.21.15.107:0/656590696 learned_addr learned my addr 172.21.15.107:0/656590696 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:36.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/656590696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6280 0x55c0c4dfe220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/656590696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4e01270 0x55c0c4dfeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff504a4f640 1 -- 172.21.15.107:0/656590696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6280 msgr2=0x55c0c4dfe220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:36.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff504a4f640 1 --2- 172.21.15.107:0/656590696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6280 0x55c0c4dfe220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff504a4f640 1 -- 172.21.15.107:0/656590696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4e01270 msgr2=0x55c0c4dfeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:36.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff504a4f640 1 --2- 172.21.15.107:0/656590696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4e01270 0x55c0c4dfeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.338+0000 7ff504a4f640 1 -- 172.21.15.107:0/656590696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0c4a6f3f0 con 0x55c0c4df6ba0 2024-05-09T08:35:36.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.339+0000 7ff504a4f640 1 --2- 172.21.15.107:0/656590696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4df6ba0 0x55c0c4dfe760 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0012d50 tx=0x7ff4f000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5915635fb20d4f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:36.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.339+0000 7ff4dffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:36.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.339+0000 7ff4dffff640 1 -- 172.21.15.107:0/656590696 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f0012110 con 0x55c0c4df6ba0 2024-05-09T08:35:36.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.339+0000 7ff4dffff640 1 -- 172.21.15.107:0/656590696 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff4f0004050 con 0x55c0c4df6ba0 2024-05-09T08:35:36.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.339+0000 7ff5099c4740 1 -- 172.21.15.107:0/656590696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4fc010c80 con 0x55c0c4df6ba0 2024-05-09T08:35:36.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.339+0000 7ff4dffff640 1 -- 172.21.15.107:0/656590696 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff4f0025050 con 0x55c0c4df6ba0 2024-05-09T08:35:36.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.339+0000 7ff5099c4740 10 client.16275 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:36.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.339+0000 7ff5099c4740 1 -- 172.21.15.107:0/656590696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c0c4a2f0f0 con 0x55c0c4df6ba0 2024-05-09T08:35:36.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.339+0000 7ff4dffff640 1 -- 172.21.15.107:0/656590696 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4fc010c80 con 0x55c0c4df6ba0 2024-05-09T08:35:36.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.339+0000 7ff4dffff640 1 -- 172.21.15.107:0/656590696 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7ff4f0003ca0 con 0x55c0c4df6ba0 2024-05-09T08:35:36.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.339+0000 7ff5099c4740 10 client.16275 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:36.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.339+0000 7ff5099c4740 10 client.16275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:36.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.339+0000 7ff5099c4740 20 client.16275 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:36.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.339+0000 7ff5099c4740 1 --2- 172.21.15.107:0/656590696 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c0c4e117c0 0x55c0c4e13be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.339+0000 7ff5099c4740 4 client.16275 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:36.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.339+0000 7ff5099c4740 1 -- 172.21.15.107:0/656590696 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0c4d69430 con 0x55c0c4e117c0 2024-05-09T08:35:36.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.340+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/656590696 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c0c4e117c0 0x55c0c4e13be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.340+0000 7ff4dffff640 10 client.16275 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.340+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/656590696 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c0c4e117c0 0x55c0c4e13be0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1c0ce8e2b71b647 server_cookie=179cfa20275f48c5 in_seq=0 out_seq=0 2024-05-09T08:35:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.341+0000 7ff4dffff640 1 -- 172.21.15.107:0/656590696 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c0c4d69430 con 0x55c0c4e117c0 2024-05-09T08:35:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.341+0000 7ff4dffff640 10 client.16275 handle_command_reply: tid=0 2024-05-09T08:35:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.341+0000 7ff5099c4740 1 client.16275 shutdown 2024-05-09T08:35:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.341+0000 7ff5099c4740 1 -- 172.21.15.107:0/656590696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4df6ba0 msgr2=0x55c0c4dfe760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.341+0000 7ff5099c4740 1 --2- 172.21.15.107:0/656590696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4df6ba0 0x55c0c4dfe760 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0012d50 tx=0x7ff4f000f790 comp rx=0 tx=0).stop 2024-05-09T08:35:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.341+0000 7ff5099c4740 1 -- 172.21.15.107:0/656590696 shutdown_connections 2024-05-09T08:35:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.341+0000 7ff5099c4740 1 --2- 172.21.15.107:0/656590696 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c0c4e117c0 0x55c0c4e13be0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.341+0000 7ff5099c4740 1 --2- 172.21.15.107:0/656590696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4e01270 0x55c0c4dfeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.341+0000 7ff5099c4740 1 --2- 172.21.15.107:0/656590696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4df6ba0 0x55c0c4dfe760 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.341+0000 7ff4dffff640 0 client.16275 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.341+0000 7ff5099c4740 1 --2- 172.21.15.107:0/656590696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6280 0x55c0c4dfe220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.341+0000 7ff5099c4740 1 -- 172.21.15.107:0/656590696 >> 172.21.15.107:0/656590696 conn(0x55c0c4df25a0 msgr2=0x55c0c4df83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:36.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.341+0000 7ff5099c4740 1 -- 172.21.15.107:0/656590696 shutdown_connections 2024-05-09T08:35:36.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.341+0000 7ff5099c4740 1 -- 172.21.15.107:0/656590696 wait complete. 2024-05-09T08:35:36.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.341+0000 7ff5099c4740 20 client.16275 trim_cache size 0 max 16384 2024-05-09T08:35:36.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.363+0000 7ff5099c4740 1 Processor -- start 2024-05-09T08:35:36.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.363+0000 7ff5099c4740 1 -- start start 2024-05-09T08:35:36.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff5099c4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6280 0x55c0c4e0cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff5099c4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4df6ba0 0x55c0c4dfeac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff5099c4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4e01270 0x55c0c4e194f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff5099c4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff5001b12c0 con 0x55c0c4df6ba0 2024-05-09T08:35:36.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff5099c4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff5001aeb90 con 0x55c0c4df6280 2024-05-09T08:35:36.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff5099c4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff5001ac9e0 con 0x55c0c4e01270 2024-05-09T08:35:36.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff4f7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6280 0x55c0c4e0cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff4f7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6280 0x55c0c4e0cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:36978/0 (socket says 172.21.15.107:36978) 2024-05-09T08:35:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff4f7fff640 1 -- 172.21.15.107:0/2291926156 learned_addr learned my addr 172.21.15.107:0/2291926156 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/2291926156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4e01270 0x55c0c4e194f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff504a4f640 1 --2- 172.21.15.107:0/2291926156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4df6ba0 0x55c0c4dfeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff4f7fff640 1 -- 172.21.15.107:0/2291926156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4e01270 msgr2=0x55c0c4e194f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/2291926156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4e01270 0x55c0c4e194f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff4f7fff640 1 -- 172.21.15.107:0/2291926156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4df6ba0 msgr2=0x55c0c4dfeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/2291926156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4df6ba0 0x55c0c4dfeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff4f7fff640 1 -- 172.21.15.107:0/2291926156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0c4dfc770 con 0x55c0c4df6280 2024-05-09T08:35:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.364+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/2291926156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6280 0x55c0c4e0cf50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc01e580 tx=0x7ff4fc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f311f273f2453cff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.365+0000 7ff4e67fc640 1 -- 172.21.15.107:0/2291926156 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc024040 con 0x55c0c4df6280 2024-05-09T08:35:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.365+0000 7ff4e67fc640 1 -- 172.21.15.107:0/2291926156 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4fc0055a0 con 0x55c0c4df6280 2024-05-09T08:35:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.365+0000 7ff4e67fc640 1 -- 172.21.15.107:0/2291926156 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc0107a0 con 0x55c0c4df6280 2024-05-09T08:35:36.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.365+0000 7ff5099c4740 1 -- 172.21.15.107:0/2291926156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6280 msgr2=0x55c0c4e0cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.365+0000 7ff5099c4740 1 --2- 172.21.15.107:0/2291926156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6280 0x55c0c4e0cf50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc01e580 tx=0x7ff4fc009c00 comp rx=0 tx=0).stop 2024-05-09T08:35:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.365+0000 7ff5099c4740 1 -- 172.21.15.107:0/2291926156 shutdown_connections 2024-05-09T08:35:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.365+0000 7ff5099c4740 1 --2- 172.21.15.107:0/2291926156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4e01270 0x55c0c4e194f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.365+0000 7ff5099c4740 1 --2- 172.21.15.107:0/2291926156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4df6ba0 0x55c0c4dfeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.365+0000 7ff5099c4740 1 --2- 172.21.15.107:0/2291926156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6280 0x55c0c4e0cf50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.365+0000 7ff5099c4740 1 -- 172.21.15.107:0/2291926156 >> 172.21.15.107:0/2291926156 conn(0x55c0c4df25a0 msgr2=0x55c0c4df82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.365+0000 7ff5099c4740 1 -- 172.21.15.107:0/2291926156 shutdown_connections 2024-05-09T08:35:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.366+0000 7ff5099c4740 1 -- 172.21.15.107:0/2291926156 wait complete. 2024-05-09T08:35:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.366+0000 7ff5099c4740 1 -- start start 2024-05-09T08:35:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.366+0000 7ff5099c4740 1 Processor -- start 2024-05-09T08:35:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.366+0000 7ff5099c4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6280 0x55c0c4e099d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.366+0000 7ff5099c4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6ba0 0x55c0c4dfad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.366+0000 7ff5099c4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4e01270 0x55c0c4dfb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.366+0000 7ff5099c4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff5001ad050 con 0x55c0c4e01270 2024-05-09T08:35:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.366+0000 7ff5099c4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff5001ae4c0 con 0x55c0c4df6ba0 2024-05-09T08:35:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.366+0000 7ff5099c4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff5001ad720 con 0x55c0c4df6280 2024-05-09T08:35:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.366+0000 7ff5099c4740 10 client.0 _ll_register_callbacks cb 0x7ff4f0009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.366+0000 7ff4f7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4e01270 0x55c0c4dfb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.366+0000 7ff4f7fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6280 0x55c0c4e099d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.366+0000 7ff4f7fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6280 0x55c0c4e099d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:43464/0 (socket says 172.21.15.107:43464) 2024-05-09T08:35:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.366+0000 7ff4f7fff640 1 -- 172.21.15.107:0/563428279 learned_addr learned my addr 172.21.15.107:0/563428279 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.367+0000 7ff504a4f640 1 --2- 172.21.15.107:0/563428279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6ba0 0x55c0c4dfad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.367+0000 7ff4f7fff640 1 -- 172.21.15.107:0/563428279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6ba0 msgr2=0x55c0c4dfad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.367+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/563428279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6ba0 0x55c0c4dfad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.367+0000 7ff4f7fff640 1 -- 172.21.15.107:0/563428279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4e01270 msgr2=0x55c0c4dfb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.367+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/563428279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4e01270 0x55c0c4dfb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.367+0000 7ff4f7fff640 1 -- 172.21.15.107:0/563428279 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0c4e76d60 con 0x55c0c4df6280 2024-05-09T08:35:36.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.367+0000 7ff4f7fff640 1 --2- 172.21.15.107:0/563428279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6280 0x55c0c4e099d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc0127f0 tx=0x7ff4fc000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fc8e3cf9ca6c363 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:36.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.367+0000 7ff4dffff640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:35:36.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.367+0000 7ff4dffff640 1 -- 172.21.15.107:0/563428279 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc024040 con 0x55c0c4df6280 2024-05-09T08:35:36.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.367+0000 7ff5099c4740 1 -- 172.21.15.107:0/563428279 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4fc0055a0 con 0x55c0c4df6280 2024-05-09T08:35:36.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.367+0000 7ff4dffff640 1 -- 172.21.15.107:0/563428279 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff4fc0050f0 con 0x55c0c4df6280 2024-05-09T08:35:36.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.367+0000 7ff4dffff640 1 -- 172.21.15.107:0/563428279 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff4fc0052f0 con 0x55c0c4df6280 2024-05-09T08:35:36.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.367+0000 7ff5099c4740 10 client.25753 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:36.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.367+0000 7ff5099c4740 1 -- 172.21.15.107:0/563428279 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c0c4e77f10 con 0x55c0c4df6280 2024-05-09T08:35:36.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.368+0000 7ff4dffff640 1 -- 172.21.15.107:0/563428279 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4fc0055a0 con 0x55c0c4df6280 2024-05-09T08:35:36.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.368+0000 7ff4dffff640 1 -- 172.21.15.107:0/563428279 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7ff4fc02a040 con 0x55c0c4df6280 2024-05-09T08:35:36.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.368+0000 7ff5099c4740 10 client.25753 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:36.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.368+0000 7ff5099c4740 10 client.25753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:36.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.368+0000 7ff5099c4740 20 client.25753 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:36.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.368+0000 7ff5099c4740 1 --2- 172.21.15.107:0/563428279 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c0c4e78450 0x55c0c4e7a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.368+0000 7ff5099c4740 4 client.25753 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:35:36.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.368+0000 7ff5099c4740 1 -- 172.21.15.107:0/563428279 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c0c4d63780 con 0x55c0c4e78450 2024-05-09T08:35:36.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.368+0000 7ff504a4f640 1 --2- 172.21.15.107:0/563428279 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c0c4e78450 0x55c0c4e7a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff4dffff640 10 client.25753 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:36.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff504a4f640 1 --2- 172.21.15.107:0/563428279 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c0c4e78450 0x55c0c4e7a870 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3493023c6cd1b80 server_cookie=a56a2c90a5818a22 in_seq=0 out_seq=0 2024-05-09T08:35:36.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff4dffff640 1 -- 172.21.15.107:0/563428279 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c0c4d63780 con 0x55c0c4e78450 2024-05-09T08:35:36.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff4dffff640 10 client.25753 handle_command_reply: tid=0 2024-05-09T08:35:36.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff5099c4740 1 client.25753 shutdown 2024-05-09T08:35:36.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff5099c4740 1 -- 172.21.15.107:0/563428279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6280 msgr2=0x55c0c4e099d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff5099c4740 1 --2- 172.21.15.107:0/563428279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6280 0x55c0c4e099d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc0127f0 tx=0x7ff4fc000bc0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff5099c4740 1 -- 172.21.15.107:0/563428279 shutdown_connections 2024-05-09T08:35:36.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff5099c4740 1 --2- 172.21.15.107:0/563428279 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c0c4e78450 0x55c0c4e7a870 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff5099c4740 1 --2- 172.21.15.107:0/563428279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c0c4e01270 0x55c0c4dfb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff5099c4740 1 --2- 172.21.15.107:0/563428279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c0c4df6ba0 0x55c0c4dfad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff4dffff640 0 client.25753 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff5099c4740 1 --2- 172.21.15.107:0/563428279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c0c4df6280 0x55c0c4e099d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff5099c4740 1 -- 172.21.15.107:0/563428279 >> 172.21.15.107:0/563428279 conn(0x55c0c4df25a0 msgr2=0x55c0c4df0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff5099c4740 1 -- 172.21.15.107:0/563428279 shutdown_connections 2024-05-09T08:35:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff5099c4740 1 -- 172.21.15.107:0/563428279 wait complete. 2024-05-09T08:35:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.369+0000 7ff5099c4740 20 client.25753 trim_cache size 0 max 16384 2024-05-09T08:35:36.375 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:35:36.375 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:35:36.375 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:35:36.375 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:35:36.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.370+0000 7ff506cb3640 1 -- 172.21.15.107:0/3747499931 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff4e80a2d00 msgr2=0x7ff4e80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.370+0000 7ff506cb3640 1 --2- 172.21.15.107:0/3747499931 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff4e80a2d00 0x7ff4e80a51c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff4f80edce0 tx=0x7ff4f8087ab0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.370+0000 7ff506cb3640 1 -- 172.21.15.107:0/3747499931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff500106710 msgr2=0x7ff50019c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.370+0000 7ff506cb3640 1 --2- 172.21.15.107:0/3747499931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff500106710 0x7ff50019c920 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8033040 tx=0x7ff4f8009360 comp rx=0 tx=0).stop 2024-05-09T08:35:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.370+0000 7ff506cb3640 1 -- 172.21.15.107:0/3747499931 shutdown_connections 2024-05-09T08:35:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.370+0000 7ff506cb3640 1 --2- 172.21.15.107:0/3747499931 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff4e80a2d00 0x7ff4e80a51c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.370+0000 7ff506cb3640 1 --2- 172.21.15.107:0/3747499931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff500106710 0x7ff50019c920 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.370+0000 7ff506cb3640 1 --2- 172.21.15.107:0/3747499931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5001010c0 0x7ff5001985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.370+0000 7ff506cb3640 1 --2- 172.21.15.107:0/3747499931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff500100700 0x7ff500198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.370+0000 7ff506cb3640 1 -- 172.21.15.107:0/3747499931 >> 172.21.15.107:0/3747499931 conn(0x7ff500077830 msgr2=0x7ff500103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.370+0000 7ff506cb3640 1 -- 172.21.15.107:0/3747499931 shutdown_connections 2024-05-09T08:35:36.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.370+0000 7ff506cb3640 1 -- 172.21.15.107:0/3747499931 wait complete. 2024-05-09T08:35:36.380 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:35:36.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:35:36.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.502+0000 7fdfad1c8640 1 -- 172.21.15.107:0/3504674812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa8073c60 msgr2=0x7fdfa8074040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.502+0000 7fdfad1c8640 1 --2- 172.21.15.107:0/3504674812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa8073c60 0x7fdfa8074040 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fdf98009900 tx=0x7fdf9802a480 comp rx=0 tx=0).stop 2024-05-09T08:35:36.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.502+0000 7fdfad1c8640 1 -- 172.21.15.107:0/3504674812 shutdown_connections 2024-05-09T08:35:36.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.502+0000 7fdfad1c8640 1 --2- 172.21.15.107:0/3504674812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfa810e050 0x7fdfa81104a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.502+0000 7fdfad1c8640 1 --2- 172.21.15.107:0/3504674812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfa8074580 0x7fdfa810db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.502+0000 7fdfad1c8640 1 --2- 172.21.15.107:0/3504674812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa8073c60 0x7fdfa8074040 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.502+0000 7fdfad1c8640 1 -- 172.21.15.107:0/3504674812 >> 172.21.15.107:0/3504674812 conn(0x7fdfa8078c30 msgr2=0x7fdfa8079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:36.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.502+0000 7fdfad1c8640 1 -- 172.21.15.107:0/3504674812 shutdown_connections 2024-05-09T08:35:36.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.503+0000 7fdfad1c8640 1 -- 172.21.15.107:0/3504674812 wait complete. 2024-05-09T08:35:36.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.503+0000 7fdfad1c8640 1 Processor -- start 2024-05-09T08:35:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.503+0000 7fdfad1c8640 1 -- start start 2024-05-09T08:35:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.503+0000 7fdfad1c8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfa8073c60 0x7fdfa819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.503+0000 7fdfad1c8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfa8074580 0x7fdfa819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.503+0000 7fdfad1c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa810e050 0x7fdfa819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.503+0000 7fdfad1c8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa8113ca0 con 0x7fdfa810e050 2024-05-09T08:35:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.503+0000 7fdfad1c8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa8113b20 con 0x7fdfa8073c60 2024-05-09T08:35:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.503+0000 7fdfad1c8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa8113e20 con 0x7fdfa8074580 2024-05-09T08:35:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdfa759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa810e050 0x7fdfa819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdfa759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa810e050 0x7fdfa819eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42006/0 (socket says 172.21.15.107:42006) 2024-05-09T08:35:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdfa759e640 1 -- 172.21.15.107:0/3194717403 learned_addr learned my addr 172.21.15.107:0/3194717403 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdf9ffff640 1 --2- 172.21.15.107:0/3194717403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfa8074580 0x7fdfa819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdfa6d9d640 1 --2- 172.21.15.107:0/3194717403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfa8073c60 0x7fdfa819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdfa759e640 1 -- 172.21.15.107:0/3194717403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfa8074580 msgr2=0x7fdfa819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdfa759e640 1 --2- 172.21.15.107:0/3194717403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfa8074580 0x7fdfa819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdfa759e640 1 -- 172.21.15.107:0/3194717403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfa8073c60 msgr2=0x7fdfa819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdfa759e640 1 --2- 172.21.15.107:0/3194717403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfa8073c60 0x7fdfa819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdfa759e640 1 -- 172.21.15.107:0/3194717403 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa819f1f0 con 0x7fdfa810e050 2024-05-09T08:35:36.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdf9ffff640 1 --2- 172.21.15.107:0/3194717403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfa8074580 0x7fdfa819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:36.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdfa759e640 1 --2- 172.21.15.107:0/3194717403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa810e050 0x7fdfa819eac0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000eaa0 tx=0x7fdfa000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:36.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdfa4d99640 1 -- 172.21.15.107:0/3194717403 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa000cdb0 con 0x7fdfa810e050 2024-05-09T08:35:36.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdfa4d99640 1 -- 172.21.15.107:0/3194717403 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfa0005020 con 0x7fdfa810e050 2024-05-09T08:35:36.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdfad1c8640 1 -- 172.21.15.107:0/3194717403 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfa819f4e0 con 0x7fdfa810e050 2024-05-09T08:35:36.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdfa4d99640 1 -- 172.21.15.107:0/3194717403 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa00116d0 con 0x7fdfa810e050 2024-05-09T08:35:36.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.504+0000 7fdfad1c8640 1 -- 172.21.15.107:0/3194717403 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfa81a6d70 con 0x7fdfa810e050 2024-05-09T08:35:36.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.506+0000 7fdfad1c8640 1 -- 172.21.15.107:0/3194717403 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfa8067dc0 con 0x7fdfa810e050 2024-05-09T08:35:36.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.507+0000 7fdfa4d99640 1 -- 172.21.15.107:0/3194717403 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fdfa00051d0 con 0x7fdfa810e050 2024-05-09T08:35:36.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.508+0000 7fdfa4d99640 1 --2- 172.21.15.107:0/3194717403 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf940a2d00 0x7fdf940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.508+0000 7fdfa6d9d640 1 --2- 172.21.15.107:0/3194717403 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf940a2d00 0x7fdf940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.508+0000 7fdfa4d99640 1 -- 172.21.15.107:0/3194717403 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fdfa00cf9f0 con 0x7fdfa810e050 2024-05-09T08:35:36.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.508+0000 7fdfa6d9d640 1 --2- 172.21.15.107:0/3194717403 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf940a2d00 0x7fdf940a51c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdf98031440 tx=0x7fdf980095f0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:36.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.513+0000 7fdfa4d99640 1 -- 172.21.15.107:0/3194717403 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdfa0016030 con 0x7fdfa810e050 2024-05-09T08:35:36.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.749+0000 7fdfad1c8640 1 -- 172.21.15.107:0/3194717403 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfa802d1c0 con 0x7fdfa810e050 2024-05-09T08:35:36.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.750+0000 7fdfa4d99640 1 -- 172.21.15.107:0/3194717403 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11890 (secure 0 0 0) 0x7fdfa009c2b0 con 0x7fdfa810e050 2024-05-09T08:35:36.749 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:35:36.749 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:34:45.925212+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24478,24499]},"id":1}]} 2024-05-09T08:35:36.749 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 26 2024-05-09T08:35:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.752+0000 7fdfad1c8640 1 -- 172.21.15.107:0/3194717403 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf940a2d00 msgr2=0x7fdf940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.752+0000 7fdfad1c8640 1 --2- 172.21.15.107:0/3194717403 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf940a2d00 0x7fdf940a51c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdf98031440 tx=0x7fdf980095f0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.752+0000 7fdfad1c8640 1 -- 172.21.15.107:0/3194717403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa810e050 msgr2=0x7fdfa819eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.752+0000 7fdfad1c8640 1 --2- 172.21.15.107:0/3194717403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa810e050 0x7fdfa819eac0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000eaa0 tx=0x7fdfa000ef60 comp rx=0 tx=0).stop 2024-05-09T08:35:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.752+0000 7fdfad1c8640 1 -- 172.21.15.107:0/3194717403 shutdown_connections 2024-05-09T08:35:36.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.752+0000 7fdfad1c8640 1 --2- 172.21.15.107:0/3194717403 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf940a2d00 0x7fdf940a51c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.752+0000 7fdfad1c8640 1 --2- 172.21.15.107:0/3194717403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfa810e050 0x7fdfa819eac0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.752+0000 7fdfad1c8640 1 --2- 172.21.15.107:0/3194717403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfa8074580 0x7fdfa819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.752+0000 7fdfad1c8640 1 --2- 172.21.15.107:0/3194717403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfa8073c60 0x7fdfa819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.752+0000 7fdfad1c8640 1 -- 172.21.15.107:0/3194717403 >> 172.21.15.107:0/3194717403 conn(0x7fdfa8078c30 msgr2=0x7fdfa80ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:36.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.752+0000 7fdfad1c8640 1 -- 172.21.15.107:0/3194717403 shutdown_connections 2024-05-09T08:35:36.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.752+0000 7fdfad1c8640 1 -- 172.21.15.107:0/3194717403 wait complete. 2024-05-09T08:35:36.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:35:36.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:36 smithi110 ceph-mon[33622]: pgmap v70: 129 pgs: 129 active+clean; 984 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:35:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.878+0000 7f8a495ce640 1 -- 172.21.15.107:0/1391344970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a44100700 msgr2=0x7f8a44100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.878+0000 7f8a495ce640 1 --2- 172.21.15.107:0/1391344970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a44100700 0x7f8a44100b80 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f8a38009930 tx=0x7f8a3802a420 comp rx=0 tx=0).stop 2024-05-09T08:35:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.878+0000 7f8a495ce640 1 -- 172.21.15.107:0/1391344970 shutdown_connections 2024-05-09T08:35:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.878+0000 7f8a495ce640 1 --2- 172.21.15.107:0/1391344970 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a441010c0 0x7f8a4410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.878+0000 7f8a495ce640 1 --2- 172.21.15.107:0/1391344970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a44100700 0x7f8a44100b80 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.878+0000 7f8a495ce640 1 --2- 172.21.15.107:0/1391344970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a44106710 0x7f8a44106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.878+0000 7f8a495ce640 1 -- 172.21.15.107:0/1391344970 >> 172.21.15.107:0/1391344970 conn(0x7f8a44077830 msgr2=0x7f8a44077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.878+0000 7f8a495ce640 1 -- 172.21.15.107:0/1391344970 shutdown_connections 2024-05-09T08:35:36.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.878+0000 7f8a495ce640 1 -- 172.21.15.107:0/1391344970 wait complete. 2024-05-09T08:35:36.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.878+0000 7f8a495ce640 1 Processor -- start 2024-05-09T08:35:36.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a495ce640 1 -- start start 2024-05-09T08:35:36.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a495ce640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a44100700 0x7f8a44102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a495ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a441010c0 0x7f8a44103280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a495ce640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a44106710 0x7f8a441037c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a495ce640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441109d0 con 0x7f8a441010c0 2024-05-09T08:35:36.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a495ce640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a44110850 con 0x7f8a44100700 2024-05-09T08:35:36.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a495ce640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8a44110b50 con 0x7f8a44106710 2024-05-09T08:35:36.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a427fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a441010c0 0x7f8a44103280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a427fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a441010c0 0x7f8a44103280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42026/0 (socket says 172.21.15.107:42026) 2024-05-09T08:35:36.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a427fc640 1 -- 172.21.15.107:0/3785614956 learned_addr learned my addr 172.21.15.107:0/3785614956 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:36.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a42ffd640 1 --2- 172.21.15.107:0/3785614956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a44100700 0x7f8a44102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a437fe640 1 --2- 172.21.15.107:0/3785614956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a44106710 0x7f8a441037c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a427fc640 1 -- 172.21.15.107:0/3785614956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a44106710 msgr2=0x7f8a441037c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a427fc640 1 --2- 172.21.15.107:0/3785614956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a44106710 0x7f8a441037c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a427fc640 1 -- 172.21.15.107:0/3785614956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a44100700 msgr2=0x7f8a44102d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a427fc640 1 --2- 172.21.15.107:0/3785614956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a44100700 0x7f8a44102d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.879+0000 7f8a427fc640 1 -- 172.21.15.107:0/3785614956 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a441a64d0 con 0x7f8a441010c0 2024-05-09T08:35:36.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.880+0000 7f8a427fc640 1 --2- 172.21.15.107:0/3785614956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a441010c0 0x7f8a44103280 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f8a38004770 tx=0x7f8a380042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:36.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.880+0000 7f8a2bfff640 1 -- 172.21.15.107:0/3785614956 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3803f070 con 0x7f8a441010c0 2024-05-09T08:35:36.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.880+0000 7f8a2bfff640 1 -- 172.21.15.107:0/3785614956 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a3803b460 con 0x7f8a441010c0 2024-05-09T08:35:36.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.880+0000 7f8a2bfff640 1 -- 172.21.15.107:0/3785614956 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a38035070 con 0x7f8a441010c0 2024-05-09T08:35:36.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.880+0000 7f8a495ce640 1 -- 172.21.15.107:0/3785614956 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a441a6760 con 0x7f8a441010c0 2024-05-09T08:35:36.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.880+0000 7f8a495ce640 1 -- 172.21.15.107:0/3785614956 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a441a6c50 con 0x7f8a441010c0 2024-05-09T08:35:36.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.882+0000 7f8a2bfff640 1 -- 172.21.15.107:0/3785614956 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8a3803b610 con 0x7f8a441010c0 2024-05-09T08:35:36.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.883+0000 7f8a4c3bc740 1 Processor -- start 2024-05-09T08:35:36.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.883+0000 7f8a4c3bc740 1 -- start start 2024-05-09T08:35:36.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.883+0000 7f8a4c3bc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f308a330 0x5561f308a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.883+0000 7f8a4c3bc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308ac50 0x5561f3094de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.883+0000 7f8a4c3bc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f3095320 0x5561f3097710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.883+0000 7f8a4c3bc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5561f2fefc40 con 0x5561f308ac50 2024-05-09T08:35:36.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.883+0000 7f8a4c3bc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5561f2f97580 con 0x5561f3095320 2024-05-09T08:35:36.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.883+0000 7f8a4c3bc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5561f2febc00 con 0x5561f308a330 2024-05-09T08:35:36.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.883+0000 7f8a42ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308ac50 0x5561f3094de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.883+0000 7f8a42ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308ac50 0x5561f3094de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42036/0 (socket says 172.21.15.107:42036) 2024-05-09T08:35:36.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.883+0000 7f8a42ffd640 1 -- 172.21.15.107:0/2227271151 learned_addr learned my addr 172.21.15.107:0/2227271151 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:36.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.883+0000 7f8a2bfff640 1 --2- 172.21.15.107:0/3785614956 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a2c0a2d30 0x7f8a2c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.884+0000 7f8a437fe640 1 --2- 172.21.15.107:0/2227271151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f308a330 0x5561f308a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.884+0000 7f8a427fc640 1 --2- 172.21.15.107:0/2227271151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f3095320 0x5561f3097710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.884+0000 7f8a437fe640 1 --2- 172.21.15.107:0/3785614956 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a2c0a2d30 0x7f8a2c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.884+0000 7f8a42ffd640 1 -- 172.21.15.107:0/2227271151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f308a330 msgr2=0x5561f308a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:36.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.884+0000 7f8a42ffd640 1 --2- 172.21.15.107:0/2227271151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f308a330 0x5561f308a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.884+0000 7f8a42ffd640 1 -- 172.21.15.107:0/2227271151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f3095320 msgr2=0x5561f3097710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:36.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.884+0000 7f8a42ffd640 1 --2- 172.21.15.107:0/2227271151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f3095320 0x5561f3097710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.884+0000 7f8a2bfff640 1 -- 172.21.15.107:0/3785614956 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8a380efb10 con 0x7f8a441010c0 2024-05-09T08:35:36.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.884+0000 7f8a42ffd640 1 -- 172.21.15.107:0/2227271151 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561f2d033f0 con 0x5561f308ac50 2024-05-09T08:35:36.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.884+0000 7f8a427fc640 1 --2- 172.21.15.107:0/2227271151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f3095320 0x5561f3097710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:35:36.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.884+0000 7f8a437fe640 1 --2- 172.21.15.107:0/2227271151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f308a330 0x5561f308a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:36.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.884+0000 7f8a42ffd640 1 --2- 172.21.15.107:0/2227271151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308ac50 0x5561f3094de0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400bc90 tx=0x7f8a340107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c998f3ad8c1af7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:36.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.884+0000 7f8a437fe640 1 --2- 172.21.15.107:0/3785614956 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a2c0a2d30 0x7f8a2c0a51f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c005c40 tx=0x7f8a3c005bd0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:36.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.884+0000 7f8a297fa640 1 -- 172.21.15.107:0/2227271151 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3401c480 con 0x5561f308ac50 2024-05-09T08:35:36.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.884+0000 7f8a297fa640 1 -- 172.21.15.107:0/2227271151 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a3401cad0 con 0x5561f308ac50 2024-05-09T08:35:36.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.884+0000 7f8a297fa640 1 -- 172.21.15.107:0/2227271151 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a34011ae0 con 0x5561f308ac50 2024-05-09T08:35:36.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.885+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/2227271151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308ac50 msgr2=0x5561f3094de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.885+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/2227271151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308ac50 0x5561f3094de0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400bc90 tx=0x7f8a340107a0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.885+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/2227271151 shutdown_connections 2024-05-09T08:35:36.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.885+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/2227271151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f3095320 0x5561f3097710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.885+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/2227271151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308ac50 0x5561f3094de0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.885+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/2227271151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f308a330 0x5561f308a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.885+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/2227271151 >> 172.21.15.107:0/2227271151 conn(0x5561f3086650 msgr2=0x5561f3086a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:36.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.885+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/2227271151 shutdown_connections 2024-05-09T08:35:36.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.885+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/2227271151 wait complete. 2024-05-09T08:35:36.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a4c3bc740 1 -- start start 2024-05-09T08:35:36.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a4c3bc740 1 Processor -- start 2024-05-09T08:35:36.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a4c3bc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308a330 0x5561f3092240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a4c3bc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f308ac50 0x5561f3092780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a4c3bc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f3095320 0x5561f3092cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a4c3bc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5561f3098110 con 0x5561f308a330 2024-05-09T08:35:36.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a4c3bc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5561f2faac40 con 0x5561f308ac50 2024-05-09T08:35:36.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a4c3bc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5561f3098290 con 0x5561f3095320 2024-05-09T08:35:36.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a4c3bc740 10 client.0 _ll_register_callbacks cb 0x5561f2d02be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:36.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a42ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308a330 0x5561f3092240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a42ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308a330 0x5561f3092240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42048/0 (socket says 172.21.15.107:42048) 2024-05-09T08:35:36.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a42ffd640 1 -- 172.21.15.107:0/3195006177 learned_addr learned my addr 172.21.15.107:0/3195006177 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:36.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a427fc640 1 --2- 172.21.15.107:0/3195006177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f308ac50 0x5561f3092780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a437fe640 1 --2- 172.21.15.107:0/3195006177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f3095320 0x5561f3092cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a42ffd640 1 -- 172.21.15.107:0/3195006177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f3095320 msgr2=0x5561f3092cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:36.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a42ffd640 1 --2- 172.21.15.107:0/3195006177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f3095320 0x5561f3092cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a42ffd640 1 -- 172.21.15.107:0/3195006177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f308ac50 msgr2=0x5561f3092780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:36.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a42ffd640 1 --2- 172.21.15.107:0/3195006177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f308ac50 0x5561f3092780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a42ffd640 1 -- 172.21.15.107:0/3195006177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561f2dcd7d0 con 0x5561f308a330 2024-05-09T08:35:36.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.886+0000 7f8a427fc640 1 --2- 172.21.15.107:0/3195006177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f308ac50 0x5561f3092780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:36.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.887+0000 7f8a42ffd640 1 --2- 172.21.15.107:0/3195006177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308a330 0x5561f3092240 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f8a34011700 tx=0x7f8a3400bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe92e226c055ca2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:36.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.887+0000 7f8a26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.887+0000 7f8a26ff5640 1 -- 172.21.15.107:0/3195006177 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3401c680 con 0x5561f308a330 2024-05-09T08:35:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.887+0000 7f8a26ff5640 1 -- 172.21.15.107:0/3195006177 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8a3401c820 con 0x5561f308a330 2024-05-09T08:35:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.887+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/3195006177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a3401cad0 con 0x5561f308a330 2024-05-09T08:35:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.887+0000 7f8a26ff5640 1 -- 172.21.15.107:0/3195006177 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8a34031050 con 0x5561f308a330 2024-05-09T08:35:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.887+0000 7f8a4c3bc740 10 client.16299 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:36.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.887+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/3195006177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5561f30a4210 con 0x5561f308a330 2024-05-09T08:35:36.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.887+0000 7f8a26ff5640 1 -- 172.21.15.107:0/3195006177 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a3401cad0 con 0x5561f308a330 2024-05-09T08:35:36.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.887+0000 7f8a26ff5640 1 -- 172.21.15.107:0/3195006177 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f8a340046c0 con 0x5561f308a330 2024-05-09T08:35:36.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.887+0000 7f8a4c3bc740 10 client.16299 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:36.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.887+0000 7f8a4c3bc740 10 client.16299 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:35:36.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.888+0000 7f8a4c3bc740 20 client.16299 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:36.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.888+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/3195006177 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5561f30a5610 0x5561f30a7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.888+0000 7f8a4c3bc740 4 client.16299 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:36.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.888+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/3195006177 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561f2ffd370 con 0x5561f30a5610 2024-05-09T08:35:36.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.888+0000 7f8a427fc640 1 --2- 172.21.15.107:0/3195006177 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5561f30a5610 0x5561f30a7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.888+0000 7f8a26ff5640 10 client.16299 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:35:36.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.888+0000 7f8a427fc640 1 --2- 172.21.15.107:0/3195006177 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5561f30a5610 0x5561f30a7a00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8556eadc85510d8e server_cookie=ded7b6684cb5a8c7 in_seq=0 out_seq=0 2024-05-09T08:35:36.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.889+0000 7f8a26ff5640 1 -- 172.21.15.107:0/3195006177 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5561f2ffd370 con 0x5561f30a5610 2024-05-09T08:35:36.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.889+0000 7f8a26ff5640 10 client.16299 handle_command_reply: tid=0 2024-05-09T08:35:36.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.889+0000 7f8a4c3bc740 1 client.16299 shutdown 2024-05-09T08:35:36.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.889+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/3195006177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308a330 msgr2=0x5561f3092240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.889+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/3195006177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308a330 0x5561f3092240 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f8a34011700 tx=0x7f8a3400bd80 comp rx=0 tx=0).stop 2024-05-09T08:35:36.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.889+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/3195006177 shutdown_connections 2024-05-09T08:35:36.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.889+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/3195006177 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5561f30a5610 0x5561f30a7a00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.889+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/3195006177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f3095320 0x5561f3092cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.889+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/3195006177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f308ac50 0x5561f3092780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.889+0000 7f8a26ff5640 0 client.16299 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:35:36.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.889+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/3195006177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308a330 0x5561f3092240 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.889+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/3195006177 >> 172.21.15.107:0/3195006177 conn(0x5561f3086650 msgr2=0x5561f308c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:36.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.889+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/3195006177 shutdown_connections 2024-05-09T08:35:36.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.889+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/3195006177 wait complete. 2024-05-09T08:35:36.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.889+0000 7f8a4c3bc740 20 client.16299 trim_cache size 0 max 16384 2024-05-09T08:35:36.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.910+0000 7f8a4c3bc740 1 Processor -- start 2024-05-09T08:35:36.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.910+0000 7f8a4c3bc740 1 -- start start 2024-05-09T08:35:36.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.910+0000 7f8a4c3bc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f308a330 0x5561f3094970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.910+0000 7f8a4c3bc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308ac50 0x5561f30a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.910+0000 7f8a4c3bc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f3095320 0x5561f3091b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.910+0000 7f8a4c3bc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441b3620 con 0x5561f308ac50 2024-05-09T08:35:36.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.910+0000 7f8a4c3bc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441b0ef0 con 0x5561f308a330 2024-05-09T08:35:36.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.910+0000 7f8a4c3bc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441aed40 con 0x5561f3095320 2024-05-09T08:35:36.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.911+0000 7f8a427fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308ac50 0x5561f30a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.911+0000 7f8a427fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308ac50 0x5561f30a0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42050/0 (socket says 172.21.15.107:42050) 2024-05-09T08:35:36.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.911+0000 7f8a427fc640 1 -- 172.21.15.107:0/793649149 learned_addr learned my addr 172.21.15.107:0/793649149 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:36.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.911+0000 7f8a42ffd640 1 --2- 172.21.15.107:0/793649149 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f308a330 0x5561f3094970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.911+0000 7f8a437fe640 1 --2- 172.21.15.107:0/793649149 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f3095320 0x5561f3091b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.911+0000 7f8a427fc640 1 -- 172.21.15.107:0/793649149 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f3095320 msgr2=0x5561f3091b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:36.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.911+0000 7f8a427fc640 1 --2- 172.21.15.107:0/793649149 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f3095320 0x5561f3091b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.911+0000 7f8a427fc640 1 -- 172.21.15.107:0/793649149 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f308a330 msgr2=0x5561f3094970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:36.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.911+0000 7f8a427fc640 1 --2- 172.21.15.107:0/793649149 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f308a330 0x5561f3094970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.911+0000 7f8a427fc640 1 -- 172.21.15.107:0/793649149 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561f3077d20 con 0x5561f308ac50 2024-05-09T08:35:36.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.911+0000 7f8a42ffd640 1 --2- 172.21.15.107:0/793649149 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f308a330 0x5561f3094970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:36.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.911+0000 7f8a427fc640 1 --2- 172.21.15.107:0/793649149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308ac50 0x5561f30a0fd0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8a38005c20 tx=0x7f8a3803b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5c6659ef16fafe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:36.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.911+0000 7f8a28ff9640 1 -- 172.21.15.107:0/793649149 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a38048040 con 0x5561f308ac50 2024-05-09T08:35:36.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.911+0000 7f8a28ff9640 1 -- 172.21.15.107:0/793649149 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a380bbcd0 con 0x5561f308ac50 2024-05-09T08:35:36.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.911+0000 7f8a28ff9640 1 -- 172.21.15.107:0/793649149 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3803ba30 con 0x5561f308ac50 2024-05-09T08:35:36.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.912+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/793649149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308ac50 msgr2=0x5561f30a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.912+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/793649149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308ac50 0x5561f30a0fd0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8a38005c20 tx=0x7f8a3803b9c0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.912+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/793649149 shutdown_connections 2024-05-09T08:35:36.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.912+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/793649149 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f3095320 0x5561f3091b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.912+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/793649149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308ac50 0x5561f30a0fd0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.912+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/793649149 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f308a330 0x5561f3094970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.912+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/793649149 >> 172.21.15.107:0/793649149 conn(0x5561f3086650 msgr2=0x5561f30857b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:36.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.912+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/793649149 shutdown_connections 2024-05-09T08:35:36.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.912+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/793649149 wait complete. 2024-05-09T08:35:36.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.913+0000 7f8a4c3bc740 1 -- start start 2024-05-09T08:35:36.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.913+0000 7f8a4c3bc740 1 Processor -- start 2024-05-09T08:35:36.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.913+0000 7f8a4c3bc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308a330 0x5561f3098750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.913+0000 7f8a4c3bc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f308ac50 0x5561f3098c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.913+0000 7f8a4c3bc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f3095320 0x5561f31038d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.913+0000 7f8a4c3bc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441af3b0 con 0x5561f308a330 2024-05-09T08:35:36.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.913+0000 7f8a4c3bc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441b0820 con 0x5561f3095320 2024-05-09T08:35:36.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.913+0000 7f8a4c3bc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441afa80 con 0x5561f308ac50 2024-05-09T08:35:36.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.913+0000 7f8a4c3bc740 10 client.0 _ll_register_callbacks cb 0x7f8a3400bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:36.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.913+0000 7f8a42ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308a330 0x5561f3098750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.913+0000 7f8a427fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f308ac50 0x5561f3098c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.913+0000 7f8a42ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308a330 0x5561f3098750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42058/0 (socket says 172.21.15.107:42058) 2024-05-09T08:35:36.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.913+0000 7f8a42ffd640 1 -- 172.21.15.107:0/2971914899 learned_addr learned my addr 172.21.15.107:0/2971914899 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:36.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.914+0000 7f8a437fe640 1 --2- 172.21.15.107:0/2971914899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f3095320 0x5561f31038d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.914+0000 7f8a427fc640 1 -- 172.21.15.107:0/2971914899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f3095320 msgr2=0x5561f31038d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.914+0000 7f8a427fc640 1 --2- 172.21.15.107:0/2971914899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f3095320 0x5561f31038d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.914+0000 7f8a427fc640 1 -- 172.21.15.107:0/2971914899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308a330 msgr2=0x5561f3098750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.914+0000 7f8a427fc640 1 --2- 172.21.15.107:0/2971914899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308a330 0x5561f3098750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.914+0000 7f8a427fc640 1 -- 172.21.15.107:0/2971914899 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561f3103f10 con 0x5561f308ac50 2024-05-09T08:35:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.914+0000 7f8a427fc640 1 --2- 172.21.15.107:0/2971914899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f308ac50 0x5561f3098c90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8a38005c20 tx=0x7f8a380095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2eaab88759c760d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.914+0000 7f8a26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:35:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.915+0000 7f8a26ff5640 1 -- 172.21.15.107:0/2971914899 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a38048070 con 0x5561f308ac50 2024-05-09T08:35:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.915+0000 7f8a26ff5640 1 -- 172.21.15.107:0/2971914899 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8a380ed180 con 0x5561f308ac50 2024-05-09T08:35:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.915+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/2971914899 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a380bbcd0 con 0x5561f308ac50 2024-05-09T08:35:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.915+0000 7f8a26ff5640 1 -- 172.21.15.107:0/2971914899 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8a380c4940 con 0x5561f308ac50 2024-05-09T08:35:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.915+0000 7f8a4c3bc740 10 client.25777 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.915+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/2971914899 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5561f310b2e0 con 0x5561f308ac50 2024-05-09T08:35:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.915+0000 7f8a26ff5640 1 -- 172.21.15.107:0/2971914899 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a380bbcd0 con 0x5561f308ac50 2024-05-09T08:35:36.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.916+0000 7f8a26ff5640 1 -- 172.21.15.107:0/2971914899 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f8a3804b060 con 0x5561f308ac50 2024-05-09T08:35:36.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.916+0000 7f8a4c3bc740 10 client.25777 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:36.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.916+0000 7f8a4c3bc740 10 client.25777 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:35:36.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.916+0000 7f8a4c3bc740 20 client.25777 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:36.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.916+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/2971914899 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5561f310b990 0x5561f310ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:36.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.916+0000 7f8a4c3bc740 4 client.25777 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:35:36.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.916+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/2971914899 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5561f2ff76c0 con 0x5561f310b990 2024-05-09T08:35:36.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.916+0000 7f8a42ffd640 1 --2- 172.21.15.107:0/2971914899 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5561f310b990 0x5561f310ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:36.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.917+0000 7f8a26ff5640 10 client.25777 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:35:36.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.917+0000 7f8a42ffd640 1 --2- 172.21.15.107:0/2971914899 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5561f310b990 0x5561f310ddb0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=da4ea1d7d78fc0db server_cookie=b41342f1e8d5f5db in_seq=0 out_seq=0 2024-05-09T08:35:36.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.918+0000 7f8a26ff5640 1 -- 172.21.15.107:0/2971914899 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5561f2ff76c0 con 0x5561f310b990 2024-05-09T08:35:36.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.918+0000 7f8a26ff5640 10 client.25777 handle_command_reply: tid=0 2024-05-09T08:35:36.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.918+0000 7f8a4c3bc740 1 client.25777 shutdown 2024-05-09T08:35:36.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.918+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/2971914899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f308ac50 msgr2=0x5561f3098c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.918+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/2971914899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f308ac50 0x5561f3098c90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8a38005c20 tx=0x7f8a380095d0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.918+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/2971914899 shutdown_connections 2024-05-09T08:35:36.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.918+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/2971914899 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5561f310b990 0x5561f310ddb0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.918+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/2971914899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5561f3095320 0x5561f31038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.918+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/2971914899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5561f308ac50 0x5561f3098c90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.918+0000 7f8a26ff5640 0 client.25777 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:35:36.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.918+0000 7f8a4c3bc740 1 --2- 172.21.15.107:0/2971914899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5561f308a330 0x5561f3098750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.918+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/2971914899 >> 172.21.15.107:0/2971914899 conn(0x5561f3086650 msgr2=0x5561f3092170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:36.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.918+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/2971914899 shutdown_connections 2024-05-09T08:35:36.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.918+0000 7f8a4c3bc740 1 -- 172.21.15.107:0/2971914899 wait complete. 2024-05-09T08:35:36.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.918+0000 7f8a4c3bc740 20 client.25777 trim_cache size 0 max 16384 2024-05-09T08:35:36.922 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:35:36.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.919+0000 7f8a495ce640 1 -- 172.21.15.107:0/3785614956 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a2c0a2d30 msgr2=0x7f8a2c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.919+0000 7f8a495ce640 1 --2- 172.21.15.107:0/3785614956 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a2c0a2d30 0x7f8a2c0a51f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c005c40 tx=0x7f8a3c005bd0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.919+0000 7f8a495ce640 1 -- 172.21.15.107:0/3785614956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a441010c0 msgr2=0x7f8a44103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:36.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.919+0000 7f8a495ce640 1 --2- 172.21.15.107:0/3785614956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a441010c0 0x7f8a44103280 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f8a38004770 tx=0x7f8a380042f0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.919+0000 7f8a495ce640 1 -- 172.21.15.107:0/3785614956 shutdown_connections 2024-05-09T08:35:36.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.919+0000 7f8a495ce640 1 --2- 172.21.15.107:0/3785614956 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a2c0a2d30 0x7f8a2c0a51f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.919+0000 7f8a495ce640 1 --2- 172.21.15.107:0/3785614956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a44106710 0x7f8a441037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.919+0000 7f8a495ce640 1 --2- 172.21.15.107:0/3785614956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a441010c0 0x7f8a44103280 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.919+0000 7f8a495ce640 1 --2- 172.21.15.107:0/3785614956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a44100700 0x7f8a44102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:36.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.919+0000 7f8a495ce640 1 -- 172.21.15.107:0/3785614956 >> 172.21.15.107:0/3785614956 conn(0x7f8a44077830 msgr2=0x7f8a44068da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:36.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.919+0000 7f8a495ce640 1 -- 172.21.15.107:0/3785614956 shutdown_connections 2024-05-09T08:35:36.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:36.919+0000 7f8a495ce640 1 -- 172.21.15.107:0/3785614956 wait complete. 2024-05-09T08:35:36.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:36 smithi107 ceph-mon[30122]: pgmap v70: 129 pgs: 129 active+clean; 984 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:35:36.931 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:35:37.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.050+0000 7f954fc3f640 1 -- 172.21.15.107:0/4271011621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95481010c0 msgr2=0x7f954810d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.050+0000 7f954fc3f640 1 --2- 172.21.15.107:0/4271011621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95481010c0 0x7f954810d5d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9540009cd0 tx=0x7f954002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.050+0000 7f954fc3f640 1 -- 172.21.15.107:0/4271011621 shutdown_connections 2024-05-09T08:35:37.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.050+0000 7f954fc3f640 1 --2- 172.21.15.107:0/4271011621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95481010c0 0x7f954810d5d0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.050+0000 7f954fc3f640 1 --2- 172.21.15.107:0/4271011621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9548100700 0x7f9548100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.050+0000 7f954fc3f640 1 --2- 172.21.15.107:0/4271011621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9548106710 0x7f9548106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.050+0000 7f954fc3f640 1 -- 172.21.15.107:0/4271011621 >> 172.21.15.107:0/4271011621 conn(0x7f9548077830 msgr2=0x7f9548077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.050+0000 7f954fc3f640 1 -- 172.21.15.107:0/4271011621 shutdown_connections 2024-05-09T08:35:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.050+0000 7f954fc3f640 1 -- 172.21.15.107:0/4271011621 wait complete. 2024-05-09T08:35:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.050+0000 7f954fc3f640 1 Processor -- start 2024-05-09T08:35:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954fc3f640 1 -- start start 2024-05-09T08:35:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954fc3f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9548100700 0x7f95480fd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954fc3f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95481010c0 0x7f95480fdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954fc3f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9548106710 0x7f95480fe160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954fc3f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9548110a20 con 0x7f9548106710 2024-05-09T08:35:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954fc3f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f95481108a0 con 0x7f95481010c0 2024-05-09T08:35:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954fc3f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9548110ba0 con 0x7f9548100700 2024-05-09T08:35:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954e1dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9548106710 0x7f95480fe160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954e1dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9548106710 0x7f95480fe160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42086/0 (socket says 172.21.15.107:42086) 2024-05-09T08:35:37.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954e1dc640 1 -- 172.21.15.107:0/1396629544 learned_addr learned my addr 172.21.15.107:0/1396629544 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954d9db640 1 --2- 172.21.15.107:0/1396629544 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9548100700 0x7f95480fd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954d1da640 1 --2- 172.21.15.107:0/1396629544 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95481010c0 0x7f95480fdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954e1dc640 1 -- 172.21.15.107:0/1396629544 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9548100700 msgr2=0x7f95480fd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954e1dc640 1 --2- 172.21.15.107:0/1396629544 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9548100700 0x7f95480fd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954e1dc640 1 -- 172.21.15.107:0/1396629544 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95481010c0 msgr2=0x7f95480fdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954e1dc640 1 --2- 172.21.15.107:0/1396629544 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95481010c0 0x7f95480fdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954e1dc640 1 -- 172.21.15.107:0/1396629544 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95481a6590 con 0x7f9548106710 2024-05-09T08:35:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954d1da640 1 --2- 172.21.15.107:0/1396629544 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95481010c0 0x7f95480fdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.051+0000 7f954e1dc640 1 --2- 172.21.15.107:0/1396629544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9548106710 0x7f95480fe160 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f9540005e30 tx=0x7f9540003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.052+0000 7f953effd640 1 -- 172.21.15.107:0/1396629544 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9540004240 con 0x7f9548106710 2024-05-09T08:35:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.052+0000 7f953effd640 1 -- 172.21.15.107:0/1396629544 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9540031ce0 con 0x7f9548106710 2024-05-09T08:35:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.052+0000 7f953effd640 1 -- 172.21.15.107:0/1396629544 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9540046480 con 0x7f9548106710 2024-05-09T08:35:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.052+0000 7f954fc3f640 1 -- 172.21.15.107:0/1396629544 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95481a6820 con 0x7f9548106710 2024-05-09T08:35:37.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.052+0000 7f954fc3f640 1 -- 172.21.15.107:0/1396629544 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95481a6dc0 con 0x7f9548106710 2024-05-09T08:35:37.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.054+0000 7f953effd640 1 -- 172.21.15.107:0/1396629544 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f95400043e0 con 0x7f9548106710 2024-05-09T08:35:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.054+0000 7f9552a2e740 1 Processor -- start 2024-05-09T08:35:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.055+0000 7f9552a2e740 1 -- start start 2024-05-09T08:35:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.055+0000 7f9552a2e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770eb3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.055+0000 7f9552a2e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770eb3c50 0x55c770ebdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.055+0000 7f9552a2e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770ebe320 0x55c770ec0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.055+0000 7f9552a2e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c770e18c40 con 0x55c770eb3330 2024-05-09T08:35:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.055+0000 7f9552a2e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c770dc0580 con 0x55c770eb3c50 2024-05-09T08:35:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.055+0000 7f9552a2e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c770e14c00 con 0x55c770ebe320 2024-05-09T08:35:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.055+0000 7f954d1da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770eb3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.055+0000 7f954d1da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770eb3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42096/0 (socket says 172.21.15.107:42096) 2024-05-09T08:35:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.055+0000 7f954d1da640 1 -- 172.21.15.107:0/493362378 learned_addr learned my addr 172.21.15.107:0/493362378 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.055+0000 7f954d9db640 1 --2- 172.21.15.107:0/493362378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770eb3c50 0x55c770ebdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.055+0000 7f954d1da640 1 --2- 172.21.15.107:0/493362378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770ebe320 0x55c770ec0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.055+0000 7f954d1da640 1 -- 172.21.15.107:0/493362378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770ebe320 msgr2=0x55c770ec0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.056+0000 7f954d1da640 1 --2- 172.21.15.107:0/493362378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770ebe320 0x55c770ec0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.056+0000 7f954d1da640 1 -- 172.21.15.107:0/493362378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770eb3c50 msgr2=0x55c770ebdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.056+0000 7f954d1da640 1 --2- 172.21.15.107:0/493362378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770eb3c50 0x55c770ebdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.056+0000 7f953effd640 1 --2- 172.21.15.107:0/1396629544 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95340a2db0 0x7f95340a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.056+0000 7f954d1da640 1 -- 172.21.15.107:0/493362378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c770b2c3f0 con 0x55c770eb3330 2024-05-09T08:35:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.056+0000 7f953effd640 1 -- 172.21.15.107:0/1396629544 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f954002e030 con 0x7f9548106710 2024-05-09T08:35:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.056+0000 7f954d1da640 1 --2- 172.21.15.107:0/1396629544 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95340a2db0 0x7f95340a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.056+0000 7f954d1da640 1 --2- 172.21.15.107:0/493362378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770eb3710 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9538013550 tx=0x7f953800b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d53ae678c9af5f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.056+0000 7f9533fff640 1 -- 172.21.15.107:0/493362378 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9538024070 con 0x55c770eb3330 2024-05-09T08:35:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.056+0000 7f9533fff640 1 -- 172.21.15.107:0/493362378 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f953801ed80 con 0x55c770eb3330 2024-05-09T08:35:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.056+0000 7f9533fff640 1 -- 172.21.15.107:0/493362378 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9538019070 con 0x55c770eb3330 2024-05-09T08:35:37.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.057+0000 7f9552a2e740 1 -- 172.21.15.107:0/493362378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 msgr2=0x55c770eb3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.057+0000 7f954d1da640 1 --2- 172.21.15.107:0/1396629544 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95340a2db0 0x7f95340a5270 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9538021000 tx=0x7f953801e000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.057+0000 7f9552a2e740 1 --2- 172.21.15.107:0/493362378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770eb3710 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9538013550 tx=0x7f953800b6b0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.057+0000 7f9552a2e740 1 -- 172.21.15.107:0/493362378 shutdown_connections 2024-05-09T08:35:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.057+0000 7f9552a2e740 1 --2- 172.21.15.107:0/493362378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770ebe320 0x55c770ec0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.057+0000 7f9552a2e740 1 --2- 172.21.15.107:0/493362378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770eb3c50 0x55c770ebdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.057+0000 7f9552a2e740 1 --2- 172.21.15.107:0/493362378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770eb3710 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.057+0000 7f9552a2e740 1 -- 172.21.15.107:0/493362378 >> 172.21.15.107:0/493362378 conn(0x55c770eaf650 msgr2=0x55c770eafa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.057+0000 7f9552a2e740 1 -- 172.21.15.107:0/493362378 shutdown_connections 2024-05-09T08:35:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.057+0000 7f9552a2e740 1 -- 172.21.15.107:0/493362378 wait complete. 2024-05-09T08:35:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.057+0000 7f9552a2e740 1 -- start start 2024-05-09T08:35:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.057+0000 7f9552a2e740 1 Processor -- start 2024-05-09T08:35:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.058+0000 7f9552a2e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770eb3330 0x55c770ebb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.058+0000 7f9552a2e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3c50 0x55c770ebb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.058+0000 7f9552a2e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770ebe320 0x55c770ebbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.058+0000 7f9552a2e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c770ec1110 con 0x55c770eb3c50 2024-05-09T08:35:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.058+0000 7f9552a2e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c770dd3c40 con 0x55c770eb3330 2024-05-09T08:35:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.058+0000 7f9552a2e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c770ec1290 con 0x55c770ebe320 2024-05-09T08:35:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.058+0000 7f9552a2e740 10 client.0 _ll_register_callbacks cb 0x55c770b2bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.058+0000 7f954d9db640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770eb3330 0x55c770ebb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.058+0000 7f954d9db640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770eb3330 0x55c770ebb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:37092/0 (socket says 172.21.15.107:37092) 2024-05-09T08:35:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.058+0000 7f954d9db640 1 -- 172.21.15.107:0/1527756482 learned_addr learned my addr 172.21.15.107:0/1527756482 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.058+0000 7f954d1da640 1 --2- 172.21.15.107:0/1527756482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3c50 0x55c770ebb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.058+0000 7f954e1dc640 1 --2- 172.21.15.107:0/1527756482 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770ebe320 0x55c770ebbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.059+0000 7f954d9db640 1 -- 172.21.15.107:0/1527756482 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770ebe320 msgr2=0x55c770ebbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.059+0000 7f954d9db640 1 --2- 172.21.15.107:0/1527756482 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770ebe320 0x55c770ebbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.059+0000 7f954d9db640 1 -- 172.21.15.107:0/1527756482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3c50 msgr2=0x55c770ebb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:37.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.059+0000 7f954d9db640 1 --2- 172.21.15.107:0/1527756482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3c50 0x55c770ebb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.059+0000 7f954d9db640 1 -- 172.21.15.107:0/1527756482 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c770bf67d0 con 0x55c770eb3330 2024-05-09T08:35:37.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.059+0000 7f954d1da640 1 --2- 172.21.15.107:0/1527756482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3c50 0x55c770ebb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:37.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.059+0000 7f954d9db640 1 --2- 172.21.15.107:0/1527756482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770eb3330 0x55c770ebb280 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f954400b9c0 tx=0x7f954400be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae09ae0e42e039c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:37.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.059+0000 7f95317fa640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:35:37.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.059+0000 7f95317fa640 1 -- 172.21.15.107:0/1527756482 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544003e80 con 0x55c770eb3330 2024-05-09T08:35:37.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.059+0000 7f9552a2e740 1 -- 172.21.15.107:0/1527756482 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f953801ed80 con 0x55c770eb3330 2024-05-09T08:35:37.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.059+0000 7f95317fa640 1 -- 172.21.15.107:0/1527756482 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f9544005020 con 0x55c770eb3330 2024-05-09T08:35:37.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.060+0000 7f95317fa640 1 -- 172.21.15.107:0/1527756482 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9544024050 con 0x55c770eb3330 2024-05-09T08:35:37.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.060+0000 7f9552a2e740 10 client.25622 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:37.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.060+0000 7f9552a2e740 1 -- 172.21.15.107:0/1527756482 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c770ecd3e0 con 0x55c770eb3330 2024-05-09T08:35:37.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.060+0000 7f95317fa640 1 -- 172.21.15.107:0/1527756482 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f953801ed80 con 0x55c770eb3330 2024-05-09T08:35:37.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.060+0000 7f95317fa640 1 -- 172.21.15.107:0/1527756482 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f954401b470 con 0x55c770eb3330 2024-05-09T08:35:37.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.060+0000 7f9552a2e740 10 client.25622 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:37.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.060+0000 7f9552a2e740 10 client.25622 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:35:37.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.060+0000 7f9552a2e740 20 client.25622 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:37.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.060+0000 7f9552a2e740 1 --2- 172.21.15.107:0/1527756482 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55c770ece8a0 0x55c770ed0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.060+0000 7f9552a2e740 4 client.25622 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:37.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.060+0000 7f9552a2e740 1 -- 172.21.15.107:0/1527756482 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c770e26370 con 0x55c770ece8a0 2024-05-09T08:35:37.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.061+0000 7f954d1da640 1 --2- 172.21.15.107:0/1527756482 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55c770ece8a0 0x55c770ed0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.062+0000 7f95317fa640 10 client.25622 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:35:37.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.062+0000 7f954d1da640 1 --2- 172.21.15.107:0/1527756482 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55c770ece8a0 0x55c770ed0d60 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3e20231be7423d1e server_cookie=63c9b1b18e3ce069 in_seq=0 out_seq=0 2024-05-09T08:35:37.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.062+0000 7f95317fa640 1 -- 172.21.15.107:0/1527756482 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c770e26370 con 0x55c770ece8a0 2024-05-09T08:35:37.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.062+0000 7f95317fa640 10 client.25622 handle_command_reply: tid=0 2024-05-09T08:35:37.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.062+0000 7f9552a2e740 1 client.25622 shutdown 2024-05-09T08:35:37.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.063+0000 7f9552a2e740 1 -- 172.21.15.107:0/1527756482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770eb3330 msgr2=0x55c770ebb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.063+0000 7f9552a2e740 1 --2- 172.21.15.107:0/1527756482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770eb3330 0x55c770ebb280 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f954400b9c0 tx=0x7f954400be80 comp rx=0 tx=0).stop 2024-05-09T08:35:37.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.063+0000 7f9552a2e740 1 -- 172.21.15.107:0/1527756482 shutdown_connections 2024-05-09T08:35:37.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.063+0000 7f9552a2e740 1 --2- 172.21.15.107:0/1527756482 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55c770ece8a0 0x55c770ed0d60 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.063+0000 7f9552a2e740 1 --2- 172.21.15.107:0/1527756482 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770ebe320 0x55c770ebbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.063+0000 7f9552a2e740 1 --2- 172.21.15.107:0/1527756482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3c50 0x55c770ebb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.063+0000 7f9552a2e740 1 --2- 172.21.15.107:0/1527756482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770eb3330 0x55c770ebb280 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.063+0000 7f95317fa640 0 client.25622 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:35:37.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.063+0000 7f9552a2e740 1 -- 172.21.15.107:0/1527756482 >> 172.21.15.107:0/1527756482 conn(0x55c770eaf650 msgr2=0x55c770eae7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:37.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.063+0000 7f9552a2e740 1 -- 172.21.15.107:0/1527756482 shutdown_connections 2024-05-09T08:35:37.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.063+0000 7f9552a2e740 1 -- 172.21.15.107:0/1527756482 wait complete. 2024-05-09T08:35:37.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.063+0000 7f9552a2e740 20 client.25622 trim_cache size 0 max 16384 2024-05-09T08:35:37.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.079+0000 7f9552a2e740 1 Processor -- start 2024-05-09T08:35:37.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.079+0000 7f9552a2e740 1 -- start start 2024-05-09T08:35:37.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.079+0000 7f9552a2e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770ebd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.079+0000 7f9552a2e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770eb3c50 0x55c770eca020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.079+0000 7f9552a2e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770ebe320 0x55c770eae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.079+0000 7f9552a2e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f95481b34b0 con 0x55c770eb3330 2024-05-09T08:35:37.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.079+0000 7f9552a2e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f95481b0d80 con 0x55c770ebe320 2024-05-09T08:35:37.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.079+0000 7f9552a2e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f95481aebd0 con 0x55c770eb3c50 2024-05-09T08:35:37.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.079+0000 7f954d9db640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770ebd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.079+0000 7f954d9db640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770ebd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42108/0 (socket says 172.21.15.107:42108) 2024-05-09T08:35:37.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.079+0000 7f954d9db640 1 -- 172.21.15.107:0/1622150569 learned_addr learned my addr 172.21.15.107:0/1622150569 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:37.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.079+0000 7f954e1dc640 1 --2- 172.21.15.107:0/1622150569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770ebe320 0x55c770eae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.079+0000 7f954d1da640 1 --2- 172.21.15.107:0/1622150569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770eb3c50 0x55c770eca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.080+0000 7f954d9db640 1 -- 172.21.15.107:0/1622150569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770eb3c50 msgr2=0x55c770eca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:37.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.080+0000 7f954d9db640 1 --2- 172.21.15.107:0/1622150569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770eb3c50 0x55c770eca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.080+0000 7f954d9db640 1 -- 172.21.15.107:0/1622150569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770ebe320 msgr2=0x55c770eae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:37.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.080+0000 7f954d9db640 1 --2- 172.21.15.107:0/1622150569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770ebe320 0x55c770eae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.080+0000 7f954d9db640 1 -- 172.21.15.107:0/1622150569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c770ea0d20 con 0x55c770eb3330 2024-05-09T08:35:37.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.080+0000 7f954d9db640 1 --2- 172.21.15.107:0/1622150569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770ebd9b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f95440044f0 tx=0x7f954400b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2852c20f813ab3a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:37.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.080+0000 7f95337fe640 1 -- 172.21.15.107:0/1622150569 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544003e80 con 0x55c770eb3330 2024-05-09T08:35:37.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.080+0000 7f95337fe640 1 -- 172.21.15.107:0/1622150569 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f954402d070 con 0x55c770eb3330 2024-05-09T08:35:37.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.080+0000 7f95337fe640 1 -- 172.21.15.107:0/1622150569 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954401bab0 con 0x55c770eb3330 2024-05-09T08:35:37.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.081+0000 7f9552a2e740 1 -- 172.21.15.107:0/1622150569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 msgr2=0x55c770ebd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.081+0000 7f9552a2e740 1 --2- 172.21.15.107:0/1622150569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770ebd9b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f95440044f0 tx=0x7f954400b670 comp rx=0 tx=0).stop 2024-05-09T08:35:37.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.081+0000 7f9552a2e740 1 -- 172.21.15.107:0/1622150569 shutdown_connections 2024-05-09T08:35:37.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.081+0000 7f9552a2e740 1 --2- 172.21.15.107:0/1622150569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770ebe320 0x55c770eae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.081+0000 7f9552a2e740 1 --2- 172.21.15.107:0/1622150569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770eb3c50 0x55c770eca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.081+0000 7f9552a2e740 1 --2- 172.21.15.107:0/1622150569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770ebd9b0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.081+0000 7f9552a2e740 1 -- 172.21.15.107:0/1622150569 >> 172.21.15.107:0/1622150569 conn(0x55c770eaf650 msgr2=0x55c770eadf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:37.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.081+0000 7f9552a2e740 1 -- 172.21.15.107:0/1622150569 shutdown_connections 2024-05-09T08:35:37.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.081+0000 7f9552a2e740 1 -- 172.21.15.107:0/1622150569 wait complete. 2024-05-09T08:35:37.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.081+0000 7f9552a2e740 1 -- start start 2024-05-09T08:35:37.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.081+0000 7f9552a2e740 1 Processor -- start 2024-05-09T08:35:37.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f9552a2e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770ec1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f9552a2e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770eb3c50 0x55c770ec24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f9552a2e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770ebe320 0x55c770ec2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f9552a2e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f95481af240 con 0x55c770eb3330 2024-05-09T08:35:37.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f9552a2e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f95481b06b0 con 0x55c770ebe320 2024-05-09T08:35:37.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f9552a2e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f95481af910 con 0x55c770eb3c50 2024-05-09T08:35:37.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f9552a2e740 10 client.0 _ll_register_callbacks cb 0x7f9544004210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:37.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f954d9db640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770ec1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f954d9db640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770ec1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42124/0 (socket says 172.21.15.107:42124) 2024-05-09T08:35:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f954d9db640 1 -- 172.21.15.107:0/1202751626 learned_addr learned my addr 172.21.15.107:0/1202751626 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f954d1da640 1 --2- 172.21.15.107:0/1202751626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770eb3c50 0x55c770ec24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f954e1dc640 1 --2- 172.21.15.107:0/1202751626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770ebe320 0x55c770ec2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f954d9db640 1 -- 172.21.15.107:0/1202751626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770eb3c50 msgr2=0x55c770ec24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f954d9db640 1 --2- 172.21.15.107:0/1202751626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770eb3c50 0x55c770ec24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f954d9db640 1 -- 172.21.15.107:0/1202751626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770ebe320 msgr2=0x55c770ec2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f954d9db640 1 --2- 172.21.15.107:0/1202751626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770ebe320 0x55c770ec2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f954d9db640 1 -- 172.21.15.107:0/1202751626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c770f33ef0 con 0x55c770eb3330 2024-05-09T08:35:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f954e1dc640 1 --2- 172.21.15.107:0/1202751626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770ebe320 0x55c770ec2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f954d9db640 1 --2- 172.21.15.107:0/1202751626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770ec1f90 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f9544001e90 tx=0x7f9544003e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa1162720e698697 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f95317fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f95317fa640 1 -- 172.21.15.107:0/1202751626 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544035020 con 0x55c770eb3330 2024-05-09T08:35:37.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f95317fa640 1 -- 172.21.15.107:0/1202751626 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f9544017040 con 0x55c770eb3330 2024-05-09T08:35:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.082+0000 7f9552a2e740 1 -- 172.21.15.107:0/1202751626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f954402d070 con 0x55c770eb3330 2024-05-09T08:35:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.083+0000 7f95317fa640 1 -- 172.21.15.107:0/1202751626 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f954403d050 con 0x55c770eb3330 2024-05-09T08:35:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.083+0000 7f9552a2e740 10 client.16323 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.083+0000 7f9552a2e740 1 -- 172.21.15.107:0/1202751626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c770f351d0 con 0x55c770eb3330 2024-05-09T08:35:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.083+0000 7f95317fa640 1 -- 172.21.15.107:0/1202751626 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f954402d070 con 0x55c770eb3330 2024-05-09T08:35:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.083+0000 7f95317fa640 1 -- 172.21.15.107:0/1202751626 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f9544031ae0 con 0x55c770eb3330 2024-05-09T08:35:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.083+0000 7f9552a2e740 10 client.16323 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.083+0000 7f9552a2e740 10 client.16323 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:35:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.083+0000 7f9552a2e740 20 client.16323 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.083+0000 7f9552a2e740 1 --2- 172.21.15.107:0/1202751626 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55c770f35820 0x55c770f37c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.083+0000 7f9552a2e740 4 client.16323 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:35:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.083+0000 7f9552a2e740 1 -- 172.21.15.107:0/1202751626 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c770e206c0 con 0x55c770f35820 2024-05-09T08:35:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.083+0000 7f954d1da640 1 --2- 172.21.15.107:0/1202751626 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55c770f35820 0x55c770f37c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f95317fa640 10 client.16323 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:35:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f954d1da640 1 --2- 172.21.15.107:0/1202751626 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55c770f35820 0x55c770f37c40 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bc39d41e004578ef server_cookie=e311d20ec1a7debe in_seq=0 out_seq=0 2024-05-09T08:35:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f95317fa640 1 -- 172.21.15.107:0/1202751626 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c770e206c0 con 0x55c770f35820 2024-05-09T08:35:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f95317fa640 10 client.16323 handle_command_reply: tid=0 2024-05-09T08:35:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f9552a2e740 1 client.16323 shutdown 2024-05-09T08:35:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f9552a2e740 1 -- 172.21.15.107:0/1202751626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 msgr2=0x55c770ec1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f9552a2e740 1 --2- 172.21.15.107:0/1202751626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770ec1f90 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f9544001e90 tx=0x7f9544003e80 comp rx=0 tx=0).stop 2024-05-09T08:35:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f9552a2e740 1 -- 172.21.15.107:0/1202751626 shutdown_connections 2024-05-09T08:35:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f9552a2e740 1 --2- 172.21.15.107:0/1202751626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c770ebe320 0x55c770ec2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f9552a2e740 1 --2- 172.21.15.107:0/1202751626 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55c770f35820 0x55c770f37c40 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f9552a2e740 1 --2- 172.21.15.107:0/1202751626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c770eb3c50 0x55c770ec24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f9552a2e740 1 --2- 172.21.15.107:0/1202751626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c770eb3330 0x55c770ec1f90 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f95317fa640 0 client.16323 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:35:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f9552a2e740 1 -- 172.21.15.107:0/1202751626 >> 172.21.15.107:0/1202751626 conn(0x55c770eaf650 msgr2=0x55c770eae7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:37.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f9552a2e740 1 -- 172.21.15.107:0/1202751626 shutdown_connections 2024-05-09T08:35:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f9552a2e740 1 -- 172.21.15.107:0/1202751626 wait complete. 2024-05-09T08:35:37.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.084+0000 7f9552a2e740 20 client.16323 trim_cache size 0 max 16384 2024-05-09T08:35:37.090 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:35:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.085+0000 7f954fc3f640 1 -- 172.21.15.107:0/1396629544 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95340a2db0 msgr2=0x7f95340a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.085+0000 7f954fc3f640 1 --2- 172.21.15.107:0/1396629544 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95340a2db0 0x7f95340a5270 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9538021000 tx=0x7f953801e000 comp rx=0 tx=0).stop 2024-05-09T08:35:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.085+0000 7f954fc3f640 1 -- 172.21.15.107:0/1396629544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9548106710 msgr2=0x7f95480fe160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.085+0000 7f954fc3f640 1 --2- 172.21.15.107:0/1396629544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9548106710 0x7f95480fe160 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f9540005e30 tx=0x7f9540003ba0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.085+0000 7f954fc3f640 1 -- 172.21.15.107:0/1396629544 shutdown_connections 2024-05-09T08:35:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.085+0000 7f954fc3f640 1 --2- 172.21.15.107:0/1396629544 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95340a2db0 0x7f95340a5270 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.085+0000 7f954fc3f640 1 --2- 172.21.15.107:0/1396629544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9548106710 0x7f95480fe160 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.085+0000 7f954fc3f640 1 --2- 172.21.15.107:0/1396629544 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95481010c0 0x7f95480fdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.085+0000 7f954fc3f640 1 --2- 172.21.15.107:0/1396629544 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9548100700 0x7f95480fd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.085+0000 7f954fc3f640 1 -- 172.21.15.107:0/1396629544 >> 172.21.15.107:0/1396629544 conn(0x7f9548077830 msgr2=0x7f95480ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.085+0000 7f954fc3f640 1 -- 172.21.15.107:0/1396629544 shutdown_connections 2024-05-09T08:35:37.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.085+0000 7f954fc3f640 1 -- 172.21.15.107:0/1396629544 wait complete. 2024-05-09T08:35:37.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:36 smithi157 ceph-mon[33614]: pgmap v70: 129 pgs: 129 active+clean; 984 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:35:37.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:35:37.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.213+0000 7ff955390640 1 -- 172.21.15.107:0/3687148715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9500fffd0 msgr2=0x7ff9501003b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.213+0000 7ff955390640 1 --2- 172.21.15.107:0/3687148715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9500fffd0 0x7ff9501003b0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff940009930 tx=0x7ff94002a420 comp rx=0 tx=0).stop 2024-05-09T08:35:37.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.213+0000 7ff955390640 1 -- 172.21.15.107:0/3687148715 shutdown_connections 2024-05-09T08:35:37.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.213+0000 7ff955390640 1 --2- 172.21.15.107:0/3687148715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff950106be0 0x7ff95010e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.213+0000 7ff955390640 1 --2- 172.21.15.107:0/3687148715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff950100980 0x7ff9501064b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.213+0000 7ff955390640 1 --2- 172.21.15.107:0/3687148715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9500fffd0 0x7ff9501003b0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.213+0000 7ff955390640 1 -- 172.21.15.107:0/3687148715 >> 172.21.15.107:0/3687148715 conn(0x7ff950076cd0 msgr2=0x7ff9500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:37.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.213+0000 7ff955390640 1 -- 172.21.15.107:0/3687148715 shutdown_connections 2024-05-09T08:35:37.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.213+0000 7ff955390640 1 -- 172.21.15.107:0/3687148715 wait complete. 2024-05-09T08:35:37.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.214+0000 7ff955390640 1 Processor -- start 2024-05-09T08:35:37.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.214+0000 7ff955390640 1 -- start start 2024-05-09T08:35:37.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.214+0000 7ff955390640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9500fffd0 0x7ff95019a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.214+0000 7ff955390640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff950106be0 0x7ff95019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.214+0000 7ff955390640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff95019eb80 0x7ff95019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.214+0000 7ff955390640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff950111aa0 con 0x7ff950106be0 2024-05-09T08:35:37.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.214+0000 7ff955390640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff950111920 con 0x7ff9500fffd0 2024-05-09T08:35:37.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.214+0000 7ff955390640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff950111c20 con 0x7ff95019eb80 2024-05-09T08:35:37.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.214+0000 7ff947fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff950106be0 0x7ff95019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.214+0000 7ff947fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff950106be0 0x7ff95019a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42138/0 (socket says 172.21.15.107:42138) 2024-05-09T08:35:37.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.214+0000 7ff947fff640 1 -- 172.21.15.107:0/1194520111 learned_addr learned my addr 172.21.15.107:0/1194520111 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:37.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.215+0000 7ff947fff640 1 -- 172.21.15.107:0/1194520111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff95019eb80 msgr2=0x7ff95019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.215+0000 7ff947fff640 1 --2- 172.21.15.107:0/1194520111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff95019eb80 0x7ff95019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.215+0000 7ff947fff640 1 -- 172.21.15.107:0/1194520111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9500fffd0 msgr2=0x7ff95019a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.215+0000 7ff947fff640 1 --2- 172.21.15.107:0/1194520111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9500fffd0 0x7ff95019a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.215+0000 7ff947fff640 1 -- 172.21.15.107:0/1194520111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff95019f670 con 0x7ff950106be0 2024-05-09T08:35:37.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.215+0000 7ff947fff640 1 --2- 172.21.15.107:0/1194520111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff950106be0 0x7ff95019a800 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff938005fd0 tx=0x7ff93800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:37.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.215+0000 7ff94cff9640 1 -- 172.21.15.107:0/1194520111 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff938017070 con 0x7ff950106be0 2024-05-09T08:35:37.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.215+0000 7ff94cff9640 1 -- 172.21.15.107:0/1194520111 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff938009c30 con 0x7ff950106be0 2024-05-09T08:35:37.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.215+0000 7ff955390640 1 -- 172.21.15.107:0/1194520111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff95019f960 con 0x7ff950106be0 2024-05-09T08:35:37.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.215+0000 7ff94cff9640 1 -- 172.21.15.107:0/1194520111 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff938003d40 con 0x7ff950106be0 2024-05-09T08:35:37.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.215+0000 7ff955390640 1 -- 172.21.15.107:0/1194520111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9501a7150 con 0x7ff950106be0 2024-05-09T08:35:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.218+0000 7ff94cff9640 1 -- 172.21.15.107:0/1194520111 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ff938003ee0 con 0x7ff950106be0 2024-05-09T08:35:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.218+0000 7ff958180740 1 Processor -- start 2024-05-09T08:35:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.218+0000 7ff958180740 1 -- start start 2024-05-09T08:35:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff958180740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff958180740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9bede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff958180740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9bf320 0x55dc1c9c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff958180740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55dc1c919c40 con 0x55dc1c9b4330 2024-05-09T08:35:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff958180740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55dc1c8c1580 con 0x55dc1c9b4c50 2024-05-09T08:35:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff958180740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55dc1c915c00 con 0x55dc1c9bf320 2024-05-09T08:35:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff94f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff94f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9b4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42144/0 (socket says 172.21.15.107:42144) 2024-05-09T08:35:37.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff94f7fe640 1 -- 172.21.15.107:0/2571525718 learned_addr learned my addr 172.21.15.107:0/2571525718 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:37.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff94effd640 1 --2- 172.21.15.107:0/2571525718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9bede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff947fff640 1 --2- 172.21.15.107:0/2571525718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9bf320 0x55dc1c9c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff94f7fe640 1 -- 172.21.15.107:0/2571525718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9bf320 msgr2=0x55dc1c9c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:37.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff94f7fe640 1 --2- 172.21.15.107:0/2571525718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9bf320 0x55dc1c9c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff94f7fe640 1 -- 172.21.15.107:0/2571525718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9b4c50 msgr2=0x55dc1c9bede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:37.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff94f7fe640 1 --2- 172.21.15.107:0/2571525718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9bede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff94cff9640 1 --2- 172.21.15.107:0/1194520111 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff93c0a2d00 0x7ff93c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff94f7fe640 1 -- 172.21.15.107:0/2571525718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc1c62d3f0 con 0x55dc1c9b4330 2024-05-09T08:35:37.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff947fff640 1 --2- 172.21.15.107:0/2571525718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9bf320 0x55dc1c9c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:37.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff94cff9640 1 -- 172.21.15.107:0/1194520111 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff9380cf520 con 0x7ff950106be0 2024-05-09T08:35:37.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff94f7fe640 1 --2- 172.21.15.107:0/1194520111 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff93c0a2d00 0x7ff93c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.219+0000 7ff94f7fe640 1 --2- 172.21.15.107:0/2571525718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9b4710 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ff948009e20 tx=0x7ff94800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cac21ecbf202fc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:37.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.220+0000 7ff9457fa640 1 -- 172.21.15.107:0/2571525718 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff948020070 con 0x55dc1c9b4330 2024-05-09T08:35:37.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.220+0000 7ff9457fa640 1 -- 172.21.15.107:0/2571525718 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff948010550 con 0x55dc1c9b4330 2024-05-09T08:35:37.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.220+0000 7ff9457fa640 1 -- 172.21.15.107:0/2571525718 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff948019a80 con 0x55dc1c9b4330 2024-05-09T08:35:37.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.221+0000 7ff958180740 1 -- 172.21.15.107:0/2571525718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 msgr2=0x55dc1c9b4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.221+0000 7ff94f7fe640 1 --2- 172.21.15.107:0/1194520111 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff93c0a2d00 0x7ff93c0a51c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff94800f010 tx=0x7ff948020000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:37.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.221+0000 7ff958180740 1 --2- 172.21.15.107:0/2571525718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9b4710 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ff948009e20 tx=0x7ff94800e720 comp rx=0 tx=0).stop 2024-05-09T08:35:37.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.221+0000 7ff958180740 1 -- 172.21.15.107:0/2571525718 shutdown_connections 2024-05-09T08:35:37.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.221+0000 7ff958180740 1 --2- 172.21.15.107:0/2571525718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9bf320 0x55dc1c9c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.221+0000 7ff958180740 1 --2- 172.21.15.107:0/2571525718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9bede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.221+0000 7ff958180740 1 --2- 172.21.15.107:0/2571525718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9b4710 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.221+0000 7ff958180740 1 -- 172.21.15.107:0/2571525718 >> 172.21.15.107:0/2571525718 conn(0x55dc1c9b0650 msgr2=0x55dc1c9b0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:37.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.221+0000 7ff958180740 1 -- 172.21.15.107:0/2571525718 shutdown_connections 2024-05-09T08:35:37.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.221+0000 7ff958180740 1 -- 172.21.15.107:0/2571525718 wait complete. 2024-05-09T08:35:37.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.221+0000 7ff958180740 1 -- start start 2024-05-09T08:35:37.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.221+0000 7ff958180740 1 Processor -- start 2024-05-09T08:35:37.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff958180740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9bc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff958180740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9bc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff958180740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9bf320 0x55dc1c9bcc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff958180740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55dc1c9c2110 con 0x55dc1c9b4330 2024-05-09T08:35:37.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff958180740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55dc1c8d4c40 con 0x55dc1c9bf320 2024-05-09T08:35:37.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff958180740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55dc1c9c2290 con 0x55dc1c9b4c50 2024-05-09T08:35:37.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff958180740 10 client.0 _ll_register_callbacks cb 0x55dc1c62cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:37.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff94effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9bc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff94effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9bc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42146/0 (socket says 172.21.15.107:42146) 2024-05-09T08:35:37.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff94effd640 1 -- 172.21.15.107:0/843890006 learned_addr learned my addr 172.21.15.107:0/843890006 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:37.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff94f7fe640 1 --2- 172.21.15.107:0/843890006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9bf320 0x55dc1c9bcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff947fff640 1 --2- 172.21.15.107:0/843890006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9bc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff94effd640 1 -- 172.21.15.107:0/843890006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9b4c50 msgr2=0x55dc1c9bc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:37.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff94effd640 1 --2- 172.21.15.107:0/843890006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9bc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff94effd640 1 -- 172.21.15.107:0/843890006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9bf320 msgr2=0x55dc1c9bcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:37.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff94effd640 1 --2- 172.21.15.107:0/843890006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9bf320 0x55dc1c9bcc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff94effd640 1 -- 172.21.15.107:0/843890006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc1c6f77d0 con 0x55dc1c9b4330 2024-05-09T08:35:37.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff94f7fe640 1 --2- 172.21.15.107:0/843890006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9bf320 0x55dc1c9bcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:37.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff94effd640 1 --2- 172.21.15.107:0/843890006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9bc1e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7ff940009930 tx=0x7ff940032d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a95d60d7bda7dfa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:37.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.222+0000 7ff9367fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:37.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.223+0000 7ff9367fc640 1 -- 172.21.15.107:0/843890006 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff940040070 con 0x55dc1c9b4330 2024-05-09T08:35:37.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.223+0000 7ff9367fc640 1 -- 172.21.15.107:0/843890006 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff940004030 con 0x55dc1c9b4330 2024-05-09T08:35:37.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.223+0000 7ff958180740 1 -- 172.21.15.107:0/843890006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dc1c927370 con 0x55dc1c9b4330 2024-05-09T08:35:37.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.223+0000 7ff9367fc640 1 -- 172.21.15.107:0/843890006 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff940035040 con 0x55dc1c9b4330 2024-05-09T08:35:37.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.223+0000 7ff958180740 10 client.16335 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:37.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.223+0000 7ff958180740 1 -- 172.21.15.107:0/843890006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55dc1c9ce290 con 0x55dc1c9b4330 2024-05-09T08:35:37.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.223+0000 7ff9367fc640 1 -- 172.21.15.107:0/843890006 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55dc1c927370 con 0x55dc1c9b4330 2024-05-09T08:35:37.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.223+0000 7ff9367fc640 1 -- 172.21.15.107:0/843890006 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7ff9400046c0 con 0x55dc1c9b4330 2024-05-09T08:35:37.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.223+0000 7ff958180740 10 client.16335 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:37.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.223+0000 7ff958180740 10 client.16335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:37.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.223+0000 7ff958180740 20 client.16335 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:37.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.223+0000 7ff958180740 1 --2- 172.21.15.107:0/843890006 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dc1c9cf7a0 0x55dc1c9d1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.223+0000 7ff958180740 4 client.16335 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:37.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.223+0000 7ff958180740 1 -- 172.21.15.107:0/843890006 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc1c9216c0 con 0x55dc1c9cf7a0 2024-05-09T08:35:37.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.223+0000 7ff947fff640 1 --2- 172.21.15.107:0/843890006 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dc1c9cf7a0 0x55dc1c9d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.224+0000 7ff9367fc640 10 client.16335 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:37.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.224+0000 7ff947fff640 1 --2- 172.21.15.107:0/843890006 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dc1c9cf7a0 0x55dc1c9d1c00 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c5b990751c9d3e1 server_cookie=d8180e74c65d7af8 in_seq=0 out_seq=0 2024-05-09T08:35:37.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.224+0000 7ff9367fc640 1 -- 172.21.15.107:0/843890006 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dc1c9216c0 con 0x55dc1c9cf7a0 2024-05-09T08:35:37.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.224+0000 7ff9367fc640 10 client.16335 handle_command_reply: tid=0 2024-05-09T08:35:37.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.224+0000 7ff958180740 1 client.16335 shutdown 2024-05-09T08:35:37.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.225+0000 7ff958180740 1 -- 172.21.15.107:0/843890006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 msgr2=0x55dc1c9bc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.225+0000 7ff958180740 1 --2- 172.21.15.107:0/843890006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9bc1e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7ff940009930 tx=0x7ff940032d30 comp rx=0 tx=0).stop 2024-05-09T08:35:37.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.225+0000 7ff958180740 1 -- 172.21.15.107:0/843890006 shutdown_connections 2024-05-09T08:35:37.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.225+0000 7ff958180740 1 --2- 172.21.15.107:0/843890006 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dc1c9cf7a0 0x55dc1c9d1c00 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.225+0000 7ff958180740 1 --2- 172.21.15.107:0/843890006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9bf320 0x55dc1c9bcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.225+0000 7ff958180740 1 --2- 172.21.15.107:0/843890006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9bc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.225+0000 7ff9367fc640 0 client.16335 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:37.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.225+0000 7ff958180740 1 --2- 172.21.15.107:0/843890006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9bc1e0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.225+0000 7ff958180740 1 -- 172.21.15.107:0/843890006 >> 172.21.15.107:0/843890006 conn(0x55dc1c9b0650 msgr2=0x55dc1c9af7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:37.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.225+0000 7ff958180740 1 -- 172.21.15.107:0/843890006 shutdown_connections 2024-05-09T08:35:37.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.225+0000 7ff958180740 1 -- 172.21.15.107:0/843890006 wait complete. 2024-05-09T08:35:37.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.225+0000 7ff958180740 20 client.16335 trim_cache size 0 max 16384 2024-05-09T08:35:37.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.241+0000 7ff958180740 1 Processor -- start 2024-05-09T08:35:37.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.241+0000 7ff958180740 1 -- start start 2024-05-09T08:35:37.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.241+0000 7ff958180740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9b4330 0x55dc1c9cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.241+0000 7ff958180740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9bb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.241+0000 7ff958180740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9bf320 0x55dc1c9b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.241+0000 7ff958180740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff9501b3700 con 0x55dc1c9b4c50 2024-05-09T08:35:37.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.241+0000 7ff958180740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff9501b0fd0 con 0x55dc1c9bf320 2024-05-09T08:35:37.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.241+0000 7ff958180740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff9501aee20 con 0x55dc1c9b4330 2024-05-09T08:35:37.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.241+0000 7ff947fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9bb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.241+0000 7ff947fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9bb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42158/0 (socket says 172.21.15.107:42158) 2024-05-09T08:35:37.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.241+0000 7ff947fff640 1 -- 172.21.15.107:0/2796719527 learned_addr learned my addr 172.21.15.107:0/2796719527 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:37.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.241+0000 7ff94effd640 1 --2- 172.21.15.107:0/2796719527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9b4330 0x55dc1c9cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.241+0000 7ff94f7fe640 1 --2- 172.21.15.107:0/2796719527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9bf320 0x55dc1c9b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.241+0000 7ff94f7fe640 1 -- 172.21.15.107:0/2796719527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9b4330 msgr2=0x55dc1c9cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:37.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.241+0000 7ff94f7fe640 1 --2- 172.21.15.107:0/2796719527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9b4330 0x55dc1c9cafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.242+0000 7ff94f7fe640 1 -- 172.21.15.107:0/2796719527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4c50 msgr2=0x55dc1c9bb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:37.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.242+0000 7ff94f7fe640 1 --2- 172.21.15.107:0/2796719527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9bb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.242+0000 7ff94f7fe640 1 -- 172.21.15.107:0/2796719527 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc1c9b4a80 con 0x55dc1c9bf320 2024-05-09T08:35:37.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.242+0000 7ff947fff640 1 --2- 172.21.15.107:0/2796719527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9bb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:37.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.242+0000 7ff94f7fe640 1 --2- 172.21.15.107:0/2796719527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9bf320 0x55dc1c9b0a30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff948019660 tx=0x7ff948023000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa1a6dadd1c790fa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:37.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.242+0000 7ff944ff9640 1 -- 172.21.15.107:0/2796719527 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9480193c0 con 0x55dc1c9bf320 2024-05-09T08:35:37.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.242+0000 7ff944ff9640 1 -- 172.21.15.107:0/2796719527 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff948019da0 con 0x55dc1c9bf320 2024-05-09T08:35:37.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.242+0000 7ff944ff9640 1 -- 172.21.15.107:0/2796719527 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94801a5a0 con 0x55dc1c9bf320 2024-05-09T08:35:37.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.243+0000 7ff958180740 1 -- 172.21.15.107:0/2796719527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9bf320 msgr2=0x55dc1c9b0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.243+0000 7ff958180740 1 --2- 172.21.15.107:0/2796719527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9bf320 0x55dc1c9b0a30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff948019660 tx=0x7ff948023000 comp rx=0 tx=0).stop 2024-05-09T08:35:37.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.243+0000 7ff958180740 1 -- 172.21.15.107:0/2796719527 shutdown_connections 2024-05-09T08:35:37.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.243+0000 7ff958180740 1 --2- 172.21.15.107:0/2796719527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9bf320 0x55dc1c9b0a30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.243+0000 7ff958180740 1 --2- 172.21.15.107:0/2796719527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9bb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.243+0000 7ff958180740 1 --2- 172.21.15.107:0/2796719527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9b4330 0x55dc1c9cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.243+0000 7ff958180740 1 -- 172.21.15.107:0/2796719527 >> 172.21.15.107:0/2796719527 conn(0x55dc1c9b0650 msgr2=0x55dc1c9bba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:37.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.243+0000 7ff958180740 1 -- 172.21.15.107:0/2796719527 shutdown_connections 2024-05-09T08:35:37.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.243+0000 7ff958180740 1 -- 172.21.15.107:0/2796719527 wait complete. 2024-05-09T08:35:37.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.243+0000 7ff958180740 1 -- start start 2024-05-09T08:35:37.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.243+0000 7ff958180740 1 Processor -- start 2024-05-09T08:35:37.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff958180740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9c2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff958180740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9c2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff958180740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9bf320 0x55dc1ca33d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff958180740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff9501af490 con 0x55dc1c9b4330 2024-05-09T08:35:37.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff958180740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff9501b0900 con 0x55dc1c9b4c50 2024-05-09T08:35:37.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff958180740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff9501afb60 con 0x55dc1c9bf320 2024-05-09T08:35:37.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff958180740 10 client.0 _ll_register_callbacks cb 0x7ff940009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:37.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff94effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff94effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9c2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42174/0 (socket says 172.21.15.107:42174) 2024-05-09T08:35:37.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff94effd640 1 -- 172.21.15.107:0/4292258631 learned_addr learned my addr 172.21.15.107:0/4292258631 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:37.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff94effd640 1 -- 172.21.15.107:0/4292258631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9bf320 msgr2=0x55dc1ca33d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:37.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff94f7fe640 1 --2- 172.21.15.107:0/4292258631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9bf320 0x55dc1ca33d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff94effd640 1 --2- 172.21.15.107:0/4292258631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9bf320 0x55dc1ca33d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff947fff640 1 --2- 172.21.15.107:0/4292258631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9c2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff94effd640 1 -- 172.21.15.107:0/4292258631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9b4c50 msgr2=0x55dc1c9c2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:37.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff94effd640 1 --2- 172.21.15.107:0/4292258631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9c2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff94effd640 1 -- 172.21.15.107:0/4292258631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc1ca34270 con 0x55dc1c9b4330 2024-05-09T08:35:37.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff94effd640 1 --2- 172.21.15.107:0/4292258631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9c2690 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7ff940004a60 tx=0x7ff940031940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11b21cc34267f89 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:37.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff9367fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:37.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff9367fc640 1 -- 172.21.15.107:0/4292258631 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff940056020 con 0x55dc1c9b4330 2024-05-09T08:35:37.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff9367fc640 1 -- 172.21.15.107:0/4292258631 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff9400050e0 con 0x55dc1c9b4330 2024-05-09T08:35:37.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff958180740 1 -- 172.21.15.107:0/4292258631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff948019da0 con 0x55dc1c9b4330 2024-05-09T08:35:37.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.244+0000 7ff9367fc640 1 -- 172.21.15.107:0/4292258631 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff940040070 con 0x55dc1c9b4330 2024-05-09T08:35:37.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.245+0000 7ff958180740 10 client.16341 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:37.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.245+0000 7ff958180740 1 -- 172.21.15.107:0/4292258631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55dc1ca35520 con 0x55dc1c9b4330 2024-05-09T08:35:37.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.245+0000 7ff9367fc640 1 -- 172.21.15.107:0/4292258631 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff948019da0 con 0x55dc1c9b4330 2024-05-09T08:35:37.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.245+0000 7ff9367fc640 1 -- 172.21.15.107:0/4292258631 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7ff940032020 con 0x55dc1c9b4330 2024-05-09T08:35:37.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.245+0000 7ff958180740 10 client.16341 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:37.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.245+0000 7ff958180740 10 client.16341 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:37.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.245+0000 7ff958180740 20 client.16341 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:37.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.245+0000 7ff958180740 1 --2- 172.21.15.107:0/4292258631 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dc1c9cf7a0 0x55dc1ca37db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:37.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.245+0000 7ff958180740 4 client.16341 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:35:37.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.245+0000 7ff958180740 1 -- 172.21.15.107:0/4292258631 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dc1c89e820 con 0x55dc1c9cf7a0 2024-05-09T08:35:37.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.245+0000 7ff947fff640 1 --2- 172.21.15.107:0/4292258631 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dc1c9cf7a0 0x55dc1ca37db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:37.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff9367fc640 10 client.16341 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:37.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff947fff640 1 --2- 172.21.15.107:0/4292258631 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dc1c9cf7a0 0x55dc1ca37db0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=794b108ff92e3e49 server_cookie=4f0b46a69e6af74e in_seq=0 out_seq=0 2024-05-09T08:35:37.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff9367fc640 1 -- 172.21.15.107:0/4292258631 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dc1c89e820 con 0x55dc1c9cf7a0 2024-05-09T08:35:37.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff9367fc640 10 client.16341 handle_command_reply: tid=0 2024-05-09T08:35:37.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff958180740 1 client.16341 shutdown 2024-05-09T08:35:37.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff958180740 1 -- 172.21.15.107:0/4292258631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 msgr2=0x55dc1c9c2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff958180740 1 --2- 172.21.15.107:0/4292258631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9c2690 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7ff940004a60 tx=0x7ff940031940 comp rx=0 tx=0).stop 2024-05-09T08:35:37.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff958180740 1 -- 172.21.15.107:0/4292258631 shutdown_connections 2024-05-09T08:35:37.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff958180740 1 --2- 172.21.15.107:0/4292258631 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dc1c9cf7a0 0x55dc1ca37db0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff958180740 1 --2- 172.21.15.107:0/4292258631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc1c9bf320 0x55dc1ca33d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff9367fc640 0 client.16341 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:37.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff958180740 1 --2- 172.21.15.107:0/4292258631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc1c9b4c50 0x55dc1c9c2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff958180740 1 --2- 172.21.15.107:0/4292258631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1c9b4330 0x55dc1c9c2690 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff958180740 1 -- 172.21.15.107:0/4292258631 >> 172.21.15.107:0/4292258631 conn(0x55dc1c9b0650 msgr2=0x55dc1c9af7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:37.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff958180740 1 -- 172.21.15.107:0/4292258631 shutdown_connections 2024-05-09T08:35:37.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff958180740 1 -- 172.21.15.107:0/4292258631 wait complete. 2024-05-09T08:35:37.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.246+0000 7ff958180740 20 client.16341 trim_cache size 0 max 16384 2024-05-09T08:35:37.251 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:35:37.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.247+0000 7ff955390640 1 -- 172.21.15.107:0/1194520111 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff93c0a2d00 msgr2=0x7ff93c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.247+0000 7ff955390640 1 --2- 172.21.15.107:0/1194520111 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff93c0a2d00 0x7ff93c0a51c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff94800f010 tx=0x7ff948020000 comp rx=0 tx=0).stop 2024-05-09T08:35:37.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.247+0000 7ff955390640 1 -- 172.21.15.107:0/1194520111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff950106be0 msgr2=0x7ff95019a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:37.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.247+0000 7ff955390640 1 --2- 172.21.15.107:0/1194520111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff950106be0 0x7ff95019a800 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff938005fd0 tx=0x7ff93800dde0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.247+0000 7ff955390640 1 -- 172.21.15.107:0/1194520111 shutdown_connections 2024-05-09T08:35:37.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.247+0000 7ff955390640 1 --2- 172.21.15.107:0/1194520111 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff93c0a2d00 0x7ff93c0a51c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.247+0000 7ff955390640 1 --2- 172.21.15.107:0/1194520111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff95019eb80 0x7ff95019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.247+0000 7ff955390640 1 --2- 172.21.15.107:0/1194520111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff950106be0 0x7ff95019a800 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.247+0000 7ff955390640 1 --2- 172.21.15.107:0/1194520111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9500fffd0 0x7ff95019a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:37.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.247+0000 7ff955390640 1 -- 172.21.15.107:0/1194520111 >> 172.21.15.107:0/1194520111 conn(0x7ff950076cd0 msgr2=0x7ff9501025a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:37.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.247+0000 7ff955390640 1 -- 172.21.15.107:0/1194520111 shutdown_connections 2024-05-09T08:35:37.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:37.247+0000 7ff955390640 1 -- 172.21.15.107:0/1194520111 wait complete. 2024-05-09T08:35:38.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:37 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3194717403' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:35:38.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:37 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3194717403' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:35:38.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:37 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3194717403' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:35:38.258 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:35:38.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:35:38.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:35:38.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.382+0000 7f134899e640 1 -- 172.21.15.107:0/1682985957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13441010c0 msgr2=0x7f134410d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.382+0000 7f134899e640 1 --2- 172.21.15.107:0/1682985957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13441010c0 0x7f134410d5d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f133c009cd0 tx=0x7f133c02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.382+0000 7f134899e640 1 -- 172.21.15.107:0/1682985957 shutdown_connections 2024-05-09T08:35:38.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.382+0000 7f134899e640 1 --2- 172.21.15.107:0/1682985957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13441010c0 0x7f134410d5d0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.382+0000 7f134899e640 1 --2- 172.21.15.107:0/1682985957 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1344100700 0x7f1344100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.382+0000 7f134899e640 1 --2- 172.21.15.107:0/1682985957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1344106710 0x7f1344106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.382+0000 7f134899e640 1 -- 172.21.15.107:0/1682985957 >> 172.21.15.107:0/1682985957 conn(0x7f1344077830 msgr2=0x7f1344077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:38.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.382+0000 7f134899e640 1 -- 172.21.15.107:0/1682985957 shutdown_connections 2024-05-09T08:35:38.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.382+0000 7f134899e640 1 -- 172.21.15.107:0/1682985957 wait complete. 2024-05-09T08:35:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.383+0000 7f134899e640 1 Processor -- start 2024-05-09T08:35:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.383+0000 7f134899e640 1 -- start start 2024-05-09T08:35:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.383+0000 7f134899e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1344100700 0x7f1344102d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.383+0000 7f134899e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13441010c0 0x7f1344103260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.383+0000 7f134899e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1344106710 0x7f13441037a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.383+0000 7f134899e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f13441109d0 con 0x7f1344100700 2024-05-09T08:35:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.383+0000 7f134899e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1344110850 con 0x7f13441010c0 2024-05-09T08:35:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.383+0000 7f134899e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1344110b50 con 0x7f1344106710 2024-05-09T08:35:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.383+0000 7f134259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1344100700 0x7f1344102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.383+0000 7f134259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1344100700 0x7f1344102d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42206/0 (socket says 172.21.15.107:42206) 2024-05-09T08:35:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.383+0000 7f134259c640 1 -- 172.21.15.107:0/3147850578 learned_addr learned my addr 172.21.15.107:0/3147850578 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.383+0000 7f1342d9d640 1 --2- 172.21.15.107:0/3147850578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1344106710 0x7f13441037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.383+0000 7f1341d9b640 1 --2- 172.21.15.107:0/3147850578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13441010c0 0x7f1344103260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.383+0000 7f134259c640 1 -- 172.21.15.107:0/3147850578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1344106710 msgr2=0x7f13441037a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.384+0000 7f134259c640 1 --2- 172.21.15.107:0/3147850578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1344106710 0x7f13441037a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.384+0000 7f134259c640 1 -- 172.21.15.107:0/3147850578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13441010c0 msgr2=0x7f1344103260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.384+0000 7f134259c640 1 --2- 172.21.15.107:0/3147850578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13441010c0 0x7f1344103260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.384+0000 7f134259c640 1 -- 172.21.15.107:0/3147850578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13441a6450 con 0x7f1344100700 2024-05-09T08:35:38.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.384+0000 7f134259c640 1 --2- 172.21.15.107:0/3147850578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1344100700 0x7f1344102d20 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f133400e9f0 tx=0x7f133400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:38.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.384+0000 7f13337fe640 1 -- 172.21.15.107:0/3147850578 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133400cd80 con 0x7f1344100700 2024-05-09T08:35:38.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.384+0000 7f13337fe640 1 -- 172.21.15.107:0/3147850578 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13340054d0 con 0x7f1344100700 2024-05-09T08:35:38.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.384+0000 7f134899e640 1 -- 172.21.15.107:0/3147850578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13441a6740 con 0x7f1344100700 2024-05-09T08:35:38.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.384+0000 7f13337fe640 1 -- 172.21.15.107:0/3147850578 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1334010490 con 0x7f1344100700 2024-05-09T08:35:38.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.384+0000 7f134899e640 1 -- 172.21.15.107:0/3147850578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13441a6c40 con 0x7f1344100700 2024-05-09T08:35:38.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.386+0000 7f13337fe640 1 -- 172.21.15.107:0/3147850578 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1334005020 con 0x7f1344100700 2024-05-09T08:35:38.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.387+0000 7f134b785740 1 Processor -- start 2024-05-09T08:35:38.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.387+0000 7f134b785740 1 -- start start 2024-05-09T08:35:38.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.387+0000 7f134b785740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ed50b0 0x560fe3ed33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.387+0000 7f134b785740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ed39c0 0x560fe3ed3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.387+0000 7f134b785740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ede740 0x560fe3ee0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.387+0000 7f134b785740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560fe3e38990 con 0x560fe3ed50b0 2024-05-09T08:35:38.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.387+0000 7f134b785740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560fe3de05b0 con 0x560fe3ed39c0 2024-05-09T08:35:38.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.387+0000 7f13337fe640 1 --2- 172.21.15.107:0/3147850578 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f132c0a2d00 0x7f132c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.387+0000 7f134b785740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560fe3e34950 con 0x560fe3ede740 2024-05-09T08:35:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.388+0000 7f1342d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ed50b0 0x560fe3ed33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.388+0000 7f1342d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ed50b0 0x560fe3ed33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42210/0 (socket says 172.21.15.107:42210) 2024-05-09T08:35:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.388+0000 7f1341d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ede740 0x560fe3ee0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.388+0000 7f1342d9d640 1 -- 172.21.15.107:0/4265282989 learned_addr learned my addr 172.21.15.107:0/4265282989 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.388+0000 7f1342d9d640 1 --2- 172.21.15.107:0/3147850578 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f132c0a2d00 0x7f132c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.388+0000 7f13337fe640 1 -- 172.21.15.107:0/3147850578 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f13340cfb30 con 0x7f1344100700 2024-05-09T08:35:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.388+0000 7f134259c640 1 --2- 172.21.15.107:0/4265282989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ed39c0 0x560fe3ed3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.388+0000 7f1342d9d640 1 -- 172.21.15.107:0/4265282989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ede740 msgr2=0x560fe3ee0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.388+0000 7f1342d9d640 1 --2- 172.21.15.107:0/4265282989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ede740 0x560fe3ee0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.388+0000 7f1342d9d640 1 -- 172.21.15.107:0/4265282989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ed39c0 msgr2=0x560fe3ed3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.388+0000 7f1342d9d640 1 --2- 172.21.15.107:0/4265282989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ed39c0 0x560fe3ed3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.388+0000 7f1342d9d640 1 -- 172.21.15.107:0/4265282989 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fe3c147e0 con 0x560fe3ed50b0 2024-05-09T08:35:38.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.388+0000 7f1342d9d640 1 --2- 172.21.15.107:0/4265282989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ed50b0 0x560fe3ed33f0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f133c005fe0 tx=0x7f133c031cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b00e32e42985ed server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:38.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.388+0000 7f1342d9d640 1 --2- 172.21.15.107:0/3147850578 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f132c0a2d00 0x7f132c0a51c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f133c005e10 tx=0x7f133c036000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:38.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.388+0000 7f1330ff9640 1 -- 172.21.15.107:0/4265282989 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c04d070 con 0x560fe3ed50b0 2024-05-09T08:35:38.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.389+0000 7f1330ff9640 1 -- 172.21.15.107:0/4265282989 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f133c036070 con 0x560fe3ed50b0 2024-05-09T08:35:38.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.389+0000 7f1330ff9640 1 -- 172.21.15.107:0/4265282989 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0309c0 con 0x560fe3ed50b0 2024-05-09T08:35:38.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.389+0000 7f134b785740 1 -- 172.21.15.107:0/4265282989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ed50b0 msgr2=0x560fe3ed33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.389+0000 7f134b785740 1 --2- 172.21.15.107:0/4265282989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ed50b0 0x560fe3ed33f0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f133c005fe0 tx=0x7f133c031cb0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.389+0000 7f134b785740 1 -- 172.21.15.107:0/4265282989 shutdown_connections 2024-05-09T08:35:38.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.389+0000 7f134b785740 1 --2- 172.21.15.107:0/4265282989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ede740 0x560fe3ee0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.389+0000 7f134b785740 1 --2- 172.21.15.107:0/4265282989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ed39c0 0x560fe3ed3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.389+0000 7f134b785740 1 --2- 172.21.15.107:0/4265282989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ed50b0 0x560fe3ed33f0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.389+0000 7f134b785740 1 -- 172.21.15.107:0/4265282989 >> 172.21.15.107:0/4265282989 conn(0x560fe3e9c0a0 msgr2=0x560fe3ecff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:38.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.389+0000 7f134b785740 1 -- 172.21.15.107:0/4265282989 shutdown_connections 2024-05-09T08:35:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.389+0000 7f134b785740 1 -- 172.21.15.107:0/4265282989 wait complete. 2024-05-09T08:35:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.390+0000 7f134b785740 1 -- start start 2024-05-09T08:35:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.390+0000 7f134b785740 1 Processor -- start 2024-05-09T08:35:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.390+0000 7f134b785740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ed39c0 0x560fe3edb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.390+0000 7f134b785740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ed50b0 0x560fe3edbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.390+0000 7f134b785740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ede740 0x560fe3edc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.390+0000 7f134b785740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560fe3ee1570 con 0x560fe3ed39c0 2024-05-09T08:35:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.390+0000 7f134b785740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560fe3df3b40 con 0x560fe3ede740 2024-05-09T08:35:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.390+0000 7f134b785740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560fe3ee16f0 con 0x560fe3ed50b0 2024-05-09T08:35:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.390+0000 7f134b785740 10 client.0 _ll_register_callbacks cb 0x560fe3de80f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.390+0000 7f134259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ed39c0 0x560fe3edb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.390+0000 7f134259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ed39c0 0x560fe3edb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42226/0 (socket says 172.21.15.107:42226) 2024-05-09T08:35:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.390+0000 7f134259c640 1 -- 172.21.15.107:0/3961421902 learned_addr learned my addr 172.21.15.107:0/3961421902 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:38.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.390+0000 7f1341d9b640 1 --2- 172.21.15.107:0/3961421902 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ed50b0 0x560fe3edbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f134259c640 1 -- 172.21.15.107:0/3961421902 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ed50b0 msgr2=0x560fe3edbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:38.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f134259c640 1 --2- 172.21.15.107:0/3961421902 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ed50b0 0x560fe3edbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f1342d9d640 1 --2- 172.21.15.107:0/3961421902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ede740 0x560fe3edc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f134259c640 1 -- 172.21.15.107:0/3961421902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ede740 msgr2=0x560fe3edc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:38.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f134259c640 1 --2- 172.21.15.107:0/3961421902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ede740 0x560fe3edc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f134259c640 1 -- 172.21.15.107:0/3961421902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fe3c74f70 con 0x560fe3ed39c0 2024-05-09T08:35:38.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f134259c640 1 --2- 172.21.15.107:0/3961421902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ed39c0 0x560fe3edb710 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f13340cd520 tx=0x7f133409b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=235d1aaa5d8f5ab6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:38.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f1329ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:38.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f1329ffb640 1 -- 172.21.15.107:0/3961421902 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1334005680 con 0x560fe3ed39c0 2024-05-09T08:35:38.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f1329ffb640 1 -- 172.21.15.107:0/3961421902 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f13340a44d0 con 0x560fe3ed39c0 2024-05-09T08:35:38.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f134b785740 1 -- 172.21.15.107:0/3961421902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f133c036070 con 0x560fe3ed39c0 2024-05-09T08:35:38.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f1329ffb640 1 -- 172.21.15.107:0/3961421902 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f133409c3f0 con 0x560fe3ed39c0 2024-05-09T08:35:38.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f134b785740 10 client.16353 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:38.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f134b785740 1 -- 172.21.15.107:0/3961421902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560fe3b0c170 con 0x560fe3ed39c0 2024-05-09T08:35:38.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f1329ffb640 1 -- 172.21.15.107:0/3961421902 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f133c036070 con 0x560fe3ed39c0 2024-05-09T08:35:38.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f1329ffb640 1 -- 172.21.15.107:0/3961421902 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f13340a4bb0 con 0x560fe3ed39c0 2024-05-09T08:35:38.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f134b785740 10 client.16353 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:38.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.391+0000 7f134b785740 10 client.16353 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:38.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.392+0000 7f134b785740 20 client.16353 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:38.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.392+0000 7f134b785740 1 --2- 172.21.15.107:0/3961421902 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560fe3eee980 0x560fe3ef0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.392+0000 7f134b785740 4 client.16353 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:38.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.392+0000 7f134b785740 1 -- 172.21.15.107:0/3961421902 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fe3e460c0 con 0x560fe3eee980 2024-05-09T08:35:38.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.392+0000 7f1341d9b640 1 --2- 172.21.15.107:0/3961421902 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560fe3eee980 0x560fe3ef0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.392+0000 7f1341d9b640 1 --2- 172.21.15.107:0/3961421902 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560fe3eee980 0x560fe3ef0e20 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=435772dcb70f54e9 server_cookie=82357cab0243db89 in_seq=0 out_seq=0 2024-05-09T08:35:38.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.392+0000 7f1329ffb640 10 client.16353 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.393+0000 7f1329ffb640 1 -- 172.21.15.107:0/3961421902 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560fe3e460c0 con 0x560fe3eee980 2024-05-09T08:35:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.393+0000 7f1329ffb640 10 client.16353 handle_command_reply: tid=0 2024-05-09T08:35:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.393+0000 7f134b785740 1 client.16353 shutdown 2024-05-09T08:35:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.393+0000 7f134b785740 1 -- 172.21.15.107:0/3961421902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ed39c0 msgr2=0x560fe3edb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.393+0000 7f134b785740 1 --2- 172.21.15.107:0/3961421902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ed39c0 0x560fe3edb710 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f13340cd520 tx=0x7f133409b8b0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.393+0000 7f134b785740 1 -- 172.21.15.107:0/3961421902 shutdown_connections 2024-05-09T08:35:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.393+0000 7f134b785740 1 --2- 172.21.15.107:0/3961421902 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560fe3eee980 0x560fe3ef0e20 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.393+0000 7f134b785740 1 --2- 172.21.15.107:0/3961421902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ede740 0x560fe3edc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.393+0000 7f134b785740 1 --2- 172.21.15.107:0/3961421902 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ed50b0 0x560fe3edbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.393+0000 7f134b785740 1 --2- 172.21.15.107:0/3961421902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ed39c0 0x560fe3edb710 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.393+0000 7f134b785740 1 -- 172.21.15.107:0/3961421902 >> 172.21.15.107:0/3961421902 conn(0x560fe3e9c0a0 msgr2=0x560fe3ed5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.393+0000 7f1329ffb640 0 client.16353 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:38.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.393+0000 7f134b785740 1 -- 172.21.15.107:0/3961421902 shutdown_connections 2024-05-09T08:35:38.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.393+0000 7f134b785740 1 -- 172.21.15.107:0/3961421902 wait complete. 2024-05-09T08:35:38.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.393+0000 7f134b785740 20 client.16353 trim_cache size 0 max 16384 2024-05-09T08:35:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.415+0000 7f134b785740 1 Processor -- start 2024-05-09T08:35:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.415+0000 7f134b785740 1 -- start start 2024-05-09T08:35:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.415+0000 7f134b785740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ed39c0 0x560fe3eea470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.415+0000 7f134b785740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ed50b0 0x560fe3ecee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.415+0000 7f134b785740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ede740 0x560fe3edaee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.415+0000 7f134b785740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f13441b34c0 con 0x560fe3ede740 2024-05-09T08:35:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.415+0000 7f134b785740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f13441b0d90 con 0x560fe3ed50b0 2024-05-09T08:35:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.415+0000 7f134b785740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f13441aebe0 con 0x560fe3ed39c0 2024-05-09T08:35:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.416+0000 7f1342d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ede740 0x560fe3edaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.416+0000 7f1342d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ede740 0x560fe3edaee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42230/0 (socket says 172.21.15.107:42230) 2024-05-09T08:35:38.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.416+0000 7f1342d9d640 1 -- 172.21.15.107:0/2838831837 learned_addr learned my addr 172.21.15.107:0/2838831837 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:38.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.416+0000 7f1341d9b640 1 --2- 172.21.15.107:0/2838831837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ed50b0 0x560fe3ecee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.416+0000 7f134259c640 1 --2- 172.21.15.107:0/2838831837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ed39c0 0x560fe3eea470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.416+0000 7f134259c640 1 -- 172.21.15.107:0/2838831837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ed50b0 msgr2=0x560fe3ecee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:38.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.416+0000 7f134259c640 1 --2- 172.21.15.107:0/2838831837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ed50b0 0x560fe3ecee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.416+0000 7f134259c640 1 -- 172.21.15.107:0/2838831837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ede740 msgr2=0x560fe3edaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:38.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.416+0000 7f134259c640 1 --2- 172.21.15.107:0/2838831837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ede740 0x560fe3edaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.416+0000 7f134259c640 1 -- 172.21.15.107:0/2838831837 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fe3ed10e0 con 0x560fe3ed39c0 2024-05-09T08:35:38.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.417+0000 7f134259c640 1 --2- 172.21.15.107:0/2838831837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ed39c0 0x560fe3eea470 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f133409b9f0 tx=0x7f133409b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8532b637f3e9a85c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.417+0000 7f132bfff640 1 -- 172.21.15.107:0/2838831837 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1334005310 con 0x560fe3ed39c0 2024-05-09T08:35:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.417+0000 7f132bfff640 1 -- 172.21.15.107:0/2838831837 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13340b9070 con 0x560fe3ed39c0 2024-05-09T08:35:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.417+0000 7f132bfff640 1 -- 172.21.15.107:0/2838831837 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1334098360 con 0x560fe3ed39c0 2024-05-09T08:35:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.418+0000 7f134b785740 1 -- 172.21.15.107:0/2838831837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ed39c0 msgr2=0x560fe3eea470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.418+0000 7f134b785740 1 --2- 172.21.15.107:0/2838831837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ed39c0 0x560fe3eea470 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f133409b9f0 tx=0x7f133409b710 comp rx=0 tx=0).stop 2024-05-09T08:35:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.418+0000 7f134b785740 1 -- 172.21.15.107:0/2838831837 shutdown_connections 2024-05-09T08:35:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.418+0000 7f134b785740 1 --2- 172.21.15.107:0/2838831837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ede740 0x560fe3edaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.418+0000 7f134b785740 1 --2- 172.21.15.107:0/2838831837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ed50b0 0x560fe3ecee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.418+0000 7f134b785740 1 --2- 172.21.15.107:0/2838831837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ed39c0 0x560fe3eea470 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.418+0000 7f134b785740 1 -- 172.21.15.107:0/2838831837 >> 172.21.15.107:0/2838831837 conn(0x560fe3e9c0a0 msgr2=0x560fe3ed5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.418+0000 7f134b785740 1 -- 172.21.15.107:0/2838831837 shutdown_connections 2024-05-09T08:35:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.418+0000 7f134b785740 1 -- 172.21.15.107:0/2838831837 wait complete. 2024-05-09T08:35:38.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.418+0000 7f134b785740 1 -- start start 2024-05-09T08:35:38.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f134b785740 1 Processor -- start 2024-05-09T08:35:38.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f134b785740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ed39c0 0x560fe3f50730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f134b785740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ed50b0 0x560fe3ee6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f134b785740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ede740 0x560fe3ee6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f134b785740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f13441af250 con 0x560fe3ede740 2024-05-09T08:35:38.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f134b785740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f13441b06c0 con 0x560fe3ed50b0 2024-05-09T08:35:38.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f134b785740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f13441af920 con 0x560fe3ed39c0 2024-05-09T08:35:38.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f134b785740 10 client.0 _ll_register_callbacks cb 0x7f133409bb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:38.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f1342d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ede740 0x560fe3ee6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f1342d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ede740 0x560fe3ee6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42238/0 (socket says 172.21.15.107:42238) 2024-05-09T08:35:38.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f1342d9d640 1 -- 172.21.15.107:0/1785593489 learned_addr learned my addr 172.21.15.107:0/1785593489 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:38.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f134259c640 1 --2- 172.21.15.107:0/1785593489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ed39c0 0x560fe3f50730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f1341d9b640 1 --2- 172.21.15.107:0/1785593489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ed50b0 0x560fe3ee6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f1342d9d640 1 -- 172.21.15.107:0/1785593489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ed39c0 msgr2=0x560fe3f50730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:38.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f1342d9d640 1 --2- 172.21.15.107:0/1785593489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ed39c0 0x560fe3f50730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f1342d9d640 1 -- 172.21.15.107:0/1785593489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ed50b0 msgr2=0x560fe3ee6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:38.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f1342d9d640 1 --2- 172.21.15.107:0/1785593489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ed50b0 0x560fe3ee6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.419+0000 7f1342d9d640 1 -- 172.21.15.107:0/1785593489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fe3ed8690 con 0x560fe3ede740 2024-05-09T08:35:38.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.420+0000 7f1342d9d640 1 --2- 172.21.15.107:0/1785593489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ede740 0x560fe3ee6ec0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f133c004fc0 tx=0x7f133c02af20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c84efd4e55a46b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:38.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.420+0000 7f1329ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:38.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.420+0000 7f1329ffb640 1 -- 172.21.15.107:0/1785593489 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c04d040 con 0x560fe3ede740 2024-05-09T08:35:38.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.420+0000 7f134b785740 1 -- 172.21.15.107:0/1785593489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13340b9070 con 0x560fe3ede740 2024-05-09T08:35:38.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.420+0000 7f1329ffb640 1 -- 172.21.15.107:0/1785593489 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f133c004410 con 0x560fe3ede740 2024-05-09T08:35:38.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.420+0000 7f1329ffb640 1 -- 172.21.15.107:0/1785593489 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f133c0098e0 con 0x560fe3ede740 2024-05-09T08:35:38.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.420+0000 7f134b785740 10 client.16359 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:38.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.420+0000 7f134b785740 1 -- 172.21.15.107:0/1785593489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560fe3f66430 con 0x560fe3ede740 2024-05-09T08:35:38.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.420+0000 7f1329ffb640 1 -- 172.21.15.107:0/1785593489 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13340b9070 con 0x560fe3ede740 2024-05-09T08:35:38.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.420+0000 7f1329ffb640 1 -- 172.21.15.107:0/1785593489 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f133c030070 con 0x560fe3ede740 2024-05-09T08:35:38.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.420+0000 7f134b785740 10 client.16359 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:38.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.420+0000 7f134b785740 10 client.16359 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:38.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.420+0000 7f134b785740 20 client.16359 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:38.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.420+0000 7f134b785740 1 --2- 172.21.15.107:0/1785593489 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560fe3f66720 0x560fe3f68b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.420+0000 7f134b785740 4 client.16359 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:35:38.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.420+0000 7f134b785740 1 -- 172.21.15.107:0/1785593489 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560fe3e40410 con 0x560fe3f66720 2024-05-09T08:35:38.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.421+0000 7f134259c640 1 --2- 172.21.15.107:0/1785593489 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560fe3f66720 0x560fe3f68b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.421+0000 7f134259c640 1 --2- 172.21.15.107:0/1785593489 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560fe3f66720 0x560fe3f68b10 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a35c435f6f27cb37 server_cookie=823c63b68c7d8763 in_seq=0 out_seq=0 2024-05-09T08:35:38.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.421+0000 7f1329ffb640 10 client.16359 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:38.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.422+0000 7f1329ffb640 1 -- 172.21.15.107:0/1785593489 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560fe3e40410 con 0x560fe3f66720 2024-05-09T08:35:38.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.422+0000 7f1329ffb640 10 client.16359 handle_command_reply: tid=0 2024-05-09T08:35:38.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.422+0000 7f134b785740 1 client.16359 shutdown 2024-05-09T08:35:38.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.423+0000 7f134b785740 1 -- 172.21.15.107:0/1785593489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ede740 msgr2=0x560fe3ee6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.423+0000 7f134b785740 1 --2- 172.21.15.107:0/1785593489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ede740 0x560fe3ee6ec0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f133c004fc0 tx=0x7f133c02af20 comp rx=0 tx=0).stop 2024-05-09T08:35:38.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.423+0000 7f134b785740 1 -- 172.21.15.107:0/1785593489 shutdown_connections 2024-05-09T08:35:38.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.423+0000 7f134b785740 1 --2- 172.21.15.107:0/1785593489 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560fe3f66720 0x560fe3f68b10 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.423+0000 7f134b785740 1 --2- 172.21.15.107:0/1785593489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fe3ede740 0x560fe3ee6ec0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.423+0000 7f134b785740 1 --2- 172.21.15.107:0/1785593489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe3ed50b0 0x560fe3ee6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.423+0000 7f1329ffb640 0 client.16359 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:38.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.423+0000 7f134b785740 1 --2- 172.21.15.107:0/1785593489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560fe3ed39c0 0x560fe3f50730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.423+0000 7f134b785740 1 -- 172.21.15.107:0/1785593489 >> 172.21.15.107:0/1785593489 conn(0x560fe3e9c0a0 msgr2=0x560fe3ed5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:38.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.423+0000 7f134b785740 1 -- 172.21.15.107:0/1785593489 shutdown_connections 2024-05-09T08:35:38.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.423+0000 7f134b785740 1 -- 172.21.15.107:0/1785593489 wait complete. 2024-05-09T08:35:38.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.423+0000 7f134b785740 20 client.16359 trim_cache size 0 max 16384 2024-05-09T08:35:38.427 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:35:38.427 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:35:38.427 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "9f73183a-d1d2-4202-96bd-e4606258c6f7", 2024-05-09T08:35:38.427 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:35:38.427 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:35:38.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.424+0000 7f134899e640 1 -- 172.21.15.107:0/3147850578 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f132c0a2d00 msgr2=0x7f132c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.424+0000 7f134899e640 1 --2- 172.21.15.107:0/3147850578 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f132c0a2d00 0x7f132c0a51c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f133c005e10 tx=0x7f133c036000 comp rx=0 tx=0).stop 2024-05-09T08:35:38.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.424+0000 7f134899e640 1 -- 172.21.15.107:0/3147850578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1344100700 msgr2=0x7f1344102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.424+0000 7f134899e640 1 --2- 172.21.15.107:0/3147850578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1344100700 0x7f1344102d20 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f133400e9f0 tx=0x7f133400edf0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.424+0000 7f134899e640 1 -- 172.21.15.107:0/3147850578 shutdown_connections 2024-05-09T08:35:38.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.424+0000 7f134899e640 1 --2- 172.21.15.107:0/3147850578 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f132c0a2d00 0x7f132c0a51c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.424+0000 7f134899e640 1 --2- 172.21.15.107:0/3147850578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1344106710 0x7f13441037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.424+0000 7f134899e640 1 --2- 172.21.15.107:0/3147850578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13441010c0 0x7f1344103260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.424+0000 7f134899e640 1 --2- 172.21.15.107:0/3147850578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1344100700 0x7f1344102d20 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.424+0000 7f134899e640 1 -- 172.21.15.107:0/3147850578 >> 172.21.15.107:0/3147850578 conn(0x7f1344077830 msgr2=0x7f1344068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:38.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.424+0000 7f134899e640 1 -- 172.21.15.107:0/3147850578 shutdown_connections 2024-05-09T08:35:38.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.424+0000 7f134899e640 1 -- 172.21.15.107:0/3147850578 wait complete. 2024-05-09T08:35:38.435 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:35:38.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:35:38.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.546+0000 7facf5b29640 1 -- 172.21.15.107:0/484726789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf0102ab0 msgr2=0x7facf0102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.546+0000 7facf5b29640 1 --2- 172.21.15.107:0/484726789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf0102ab0 0x7facf0102e90 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7face0009900 tx=0x7face002a510 comp rx=0 tx=0).stop 2024-05-09T08:35:38.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.547+0000 7facf5b29640 1 -- 172.21.15.107:0/484726789 shutdown_connections 2024-05-09T08:35:38.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.547+0000 7facf5b29640 1 --2- 172.21.15.107:0/484726789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facf010bdf0 0x7facf010e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.547+0000 7facf5b29640 1 --2- 172.21.15.107:0/484726789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf01033d0 0x7facf010b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.547+0000 7facf5b29640 1 --2- 172.21.15.107:0/484726789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf0102ab0 0x7facf0102e90 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.547+0000 7facf5b29640 1 -- 172.21.15.107:0/484726789 >> 172.21.15.107:0/484726789 conn(0x7facf0077830 msgr2=0x7facf0077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:38.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.547+0000 7facf5b29640 1 -- 172.21.15.107:0/484726789 shutdown_connections 2024-05-09T08:35:38.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.547+0000 7facf5b29640 1 -- 172.21.15.107:0/484726789 wait complete. 2024-05-09T08:35:38.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.547+0000 7facf5b29640 1 Processor -- start 2024-05-09T08:35:38.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.547+0000 7facf5b29640 1 -- start start 2024-05-09T08:35:38.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.547+0000 7facf5b29640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facf0102ab0 0x7facf0198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.547+0000 7facf5b29640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf01033d0 0x7facf0198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.547+0000 7facf5b29640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf010bdf0 0x7facf019c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.547+0000 7facf5b29640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7facf0111aa0 con 0x7facf010bdf0 2024-05-09T08:35:38.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.547+0000 7facf5b29640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7facf0111920 con 0x7facf0102ab0 2024-05-09T08:35:38.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.547+0000 7facf5b29640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7facf0111c20 con 0x7facf01033d0 2024-05-09T08:35:38.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.548+0000 7faceffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf010bdf0 0x7facf019c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.548+0000 7face7fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf01033d0 0x7facf0198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.548+0000 7faceffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf010bdf0 0x7facf019c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42246/0 (socket says 172.21.15.107:42246) 2024-05-09T08:35:38.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.548+0000 7faceffff640 1 -- 172.21.15.107:0/3319646362 learned_addr learned my addr 172.21.15.107:0/3319646362 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:38.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.548+0000 7facef7fe640 1 --2- 172.21.15.107:0/3319646362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facf0102ab0 0x7facf0198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.548+0000 7faceffff640 1 -- 172.21.15.107:0/3319646362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf01033d0 msgr2=0x7facf0198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.548+0000 7faceffff640 1 --2- 172.21.15.107:0/3319646362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf01033d0 0x7facf0198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.548+0000 7faceffff640 1 -- 172.21.15.107:0/3319646362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facf0102ab0 msgr2=0x7facf0198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.548+0000 7faceffff640 1 --2- 172.21.15.107:0/3319646362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facf0102ab0 0x7facf0198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.548+0000 7faceffff640 1 -- 172.21.15.107:0/3319646362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf019d040 con 0x7facf010bdf0 2024-05-09T08:35:38.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.548+0000 7faceffff640 1 --2- 172.21.15.107:0/3319646362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf010bdf0 0x7facf019c910 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7face800eb10 tx=0x7face800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:38.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.548+0000 7faced7fa640 1 -- 172.21.15.107:0/3319646362 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face8004b20 con 0x7facf010bdf0 2024-05-09T08:35:38.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.548+0000 7faced7fa640 1 -- 172.21.15.107:0/3319646362 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7face8004cc0 con 0x7facf010bdf0 2024-05-09T08:35:38.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.548+0000 7facf5b29640 1 -- 172.21.15.107:0/3319646362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf019d330 con 0x7facf010bdf0 2024-05-09T08:35:38.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.548+0000 7faced7fa640 1 -- 172.21.15.107:0/3319646362 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face8010690 con 0x7facf010bdf0 2024-05-09T08:35:38.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.548+0000 7facf5b29640 1 -- 172.21.15.107:0/3319646362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf01a4bc0 con 0x7facf010bdf0 2024-05-09T08:35:38.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.551+0000 7faced7fa640 1 -- 172.21.15.107:0/3319646362 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7face8005020 con 0x7facf010bdf0 2024-05-09T08:35:38.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.551+0000 7facf883a740 1 Processor -- start 2024-05-09T08:35:38.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.551+0000 7facf883a740 1 -- start start 2024-05-09T08:35:38.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.551+0000 7facf883a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa56160 0x55f24aa56540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.551+0000 7facf883a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa65140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.551+0000 7facf883a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa65680 0x55f24aa67a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.551+0000 7facf883a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f24a9bbd00 con 0x55f24aa65680 2024-05-09T08:35:38.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.551+0000 7facf883a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f24a9634c0 con 0x55f24aa56160 2024-05-09T08:35:38.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.551+0000 7facf883a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f24a9b7cc0 con 0x55f24aa56a80 2024-05-09T08:35:38.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7facef7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa65140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7facef7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa65140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:43748/0 (socket says 172.21.15.107:43748) 2024-05-09T08:35:38.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7facef7fe640 1 -- 172.21.15.107:0/2435349527 learned_addr learned my addr 172.21.15.107:0/2435349527 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:38.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7face7fff640 1 --2- 172.21.15.107:0/2435349527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa65680 0x55f24aa67a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7face7fff640 1 --2- 172.21.15.107:0/2435349527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa56160 0x55f24aa56540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7facef7fe640 1 -- 172.21.15.107:0/2435349527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa56160 msgr2=0x55f24aa56540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:38.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7facef7fe640 1 --2- 172.21.15.107:0/2435349527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa56160 0x55f24aa56540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7faced7fa640 1 --2- 172.21.15.107:0/3319646362 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7facdc0a2d00 0x7facdc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7facef7fe640 1 -- 172.21.15.107:0/2435349527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa65680 msgr2=0x55f24aa67a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:38.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7facef7fe640 1 --2- 172.21.15.107:0/2435349527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa65680 0x55f24aa67a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7facef7fe640 1 -- 172.21.15.107:0/2435349527 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f24a7997d0 con 0x55f24aa56a80 2024-05-09T08:35:38.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7face7fff640 1 --2- 172.21.15.107:0/2435349527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa65680 0x55f24aa67a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:38.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7faceffff640 1 --2- 172.21.15.107:0/3319646362 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7facdc0a2d00 0x7facdc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7facef7fe640 1 --2- 172.21.15.107:0/2435349527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa65140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7face0009900 tx=0x7face0032b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baa4b2ad3b0d3b4b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7faced7fa640 1 -- 172.21.15.107:0/3319646362 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7face80cfd70 con 0x7facf010bdf0 2024-05-09T08:35:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7face5ffb640 1 -- 172.21.15.107:0/2435349527 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face003f070 con 0x55f24aa56a80 2024-05-09T08:35:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.552+0000 7faceffff640 1 --2- 172.21.15.107:0/3319646362 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7facdc0a2d00 0x7facdc0a51c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7face80cd210 tx=0x7face807c040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.553+0000 7face5ffb640 1 -- 172.21.15.107:0/2435349527 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7face0032e20 con 0x55f24aa56a80 2024-05-09T08:35:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.553+0000 7face5ffb640 1 -- 172.21.15.107:0/2435349527 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0035070 con 0x55f24aa56a80 2024-05-09T08:35:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.553+0000 7facf883a740 1 -- 172.21.15.107:0/2435349527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 msgr2=0x55f24aa65140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.553+0000 7facf883a740 1 --2- 172.21.15.107:0/2435349527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa65140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7face0009900 tx=0x7face0032b70 comp rx=0 tx=0).stop 2024-05-09T08:35:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.553+0000 7facf883a740 1 -- 172.21.15.107:0/2435349527 shutdown_connections 2024-05-09T08:35:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.553+0000 7facf883a740 1 --2- 172.21.15.107:0/2435349527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa65680 0x55f24aa67a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.553+0000 7facf883a740 1 --2- 172.21.15.107:0/2435349527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa65140 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.553+0000 7facf883a740 1 --2- 172.21.15.107:0/2435349527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa56160 0x55f24aa56540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.553+0000 7facf883a740 1 -- 172.21.15.107:0/2435349527 >> 172.21.15.107:0/2435349527 conn(0x55f24aa525a0 msgr2=0x55f24aa52980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:38.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.553+0000 7facf883a740 1 -- 172.21.15.107:0/2435349527 shutdown_connections 2024-05-09T08:35:38.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.553+0000 7facf883a740 1 -- 172.21.15.107:0/2435349527 wait complete. 2024-05-09T08:35:38.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.554+0000 7facf883a740 1 -- start start 2024-05-09T08:35:38.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.554+0000 7facf883a740 1 Processor -- start 2024-05-09T08:35:38.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.554+0000 7facf883a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa56160 0x55f24aa62600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.554+0000 7facf883a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa62b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.554+0000 7facf883a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa65680 0x55f24aa63080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.554+0000 7facf883a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f24aa68460 con 0x55f24aa65680 2024-05-09T08:35:38.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.554+0000 7facf883a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f24a976b90 con 0x55f24aa56160 2024-05-09T08:35:38.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.554+0000 7facf883a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f24aa685e0 con 0x55f24aa56a80 2024-05-09T08:35:38.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.554+0000 7facf883a740 10 client.0 _ll_register_callbacks cb 0x55f24a9b2da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:38.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.554+0000 7face7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa56160 0x55f24aa62600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.554+0000 7facef7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa62b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.554+0000 7face7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa56160 0x55f24aa62600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:37238/0 (socket says 172.21.15.107:37238) 2024-05-09T08:35:38.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.554+0000 7face7fff640 1 -- 172.21.15.107:0/3735635434 learned_addr learned my addr 172.21.15.107:0/3735635434 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:38.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.554+0000 7face7fff640 1 --2- 172.21.15.107:0/3735635434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa65680 0x55f24aa63080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.555+0000 7facef7fe640 1 -- 172.21.15.107:0/3735635434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa56160 msgr2=0x55f24aa62600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:38.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.555+0000 7facef7fe640 1 --2- 172.21.15.107:0/3735635434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa56160 0x55f24aa62600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.555+0000 7facef7fe640 1 -- 172.21.15.107:0/3735635434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa65680 msgr2=0x55f24aa63080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:38.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.555+0000 7facef7fe640 1 --2- 172.21.15.107:0/3735635434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa65680 0x55f24aa63080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.555+0000 7facef7fe640 1 -- 172.21.15.107:0/3735635434 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f24a6cf3f0 con 0x55f24aa56a80 2024-05-09T08:35:38.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.555+0000 7face7fff640 1 --2- 172.21.15.107:0/3735635434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa56160 0x55f24aa62600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.555+0000 7facef7fe640 1 --2- 172.21.15.107:0/3735635434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa62b40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7face0009900 tx=0x7face0031750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13ddc77ebb31fdbe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.555+0000 7facceffd640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:35:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.555+0000 7facceffd640 1 -- 172.21.15.107:0/3735635434 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face004a020 con 0x55f24aa56a80 2024-05-09T08:35:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.555+0000 7facf883a740 1 -- 172.21.15.107:0/3735635434 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7face0032e20 con 0x55f24aa56a80 2024-05-09T08:35:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.555+0000 7facceffd640 1 -- 172.21.15.107:0/3735635434 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7face0004450 con 0x55f24aa56a80 2024-05-09T08:35:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.555+0000 7facceffd640 1 -- 172.21.15.107:0/3735635434 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7face0055050 con 0x55f24aa56a80 2024-05-09T08:35:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.555+0000 7facf883a740 10 client.25837 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.555+0000 7facf883a740 1 -- 172.21.15.107:0/3735635434 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f24a68f0f0 con 0x55f24aa56a80 2024-05-09T08:35:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.555+0000 7facceffd640 1 -- 172.21.15.107:0/3735635434 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7face0032e20 con 0x55f24aa56a80 2024-05-09T08:35:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.556+0000 7facceffd640 1 -- 172.21.15.107:0/3735635434 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7face00046c0 con 0x55f24aa56a80 2024-05-09T08:35:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.556+0000 7facf883a740 10 client.25837 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.556+0000 7facf883a740 10 client.25837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.556+0000 7facf883a740 20 client.25837 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.556+0000 7facf883a740 1 --2- 172.21.15.107:0/3735635434 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f24aa758d0 0x55f24aa77cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.556+0000 7facf883a740 4 client.25837 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.556+0000 7facf883a740 1 -- 172.21.15.107:0/3735635434 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f24a9c9430 con 0x55f24aa758d0 2024-05-09T08:35:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.556+0000 7face7fff640 1 --2- 172.21.15.107:0/3735635434 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f24aa758d0 0x55f24aa77cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.556+0000 7face7fff640 1 --2- 172.21.15.107:0/3735635434 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f24aa758d0 0x55f24aa77cc0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dac3ae675eec348 server_cookie=648fada797669850 in_seq=0 out_seq=0 2024-05-09T08:35:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.556+0000 7facceffd640 10 client.25837 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.557+0000 7facceffd640 1 -- 172.21.15.107:0/3735635434 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f24a9c9430 con 0x55f24aa758d0 2024-05-09T08:35:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.557+0000 7facceffd640 10 client.25837 handle_command_reply: tid=0 2024-05-09T08:35:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.557+0000 7facf883a740 1 client.25837 shutdown 2024-05-09T08:35:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.557+0000 7facf883a740 1 -- 172.21.15.107:0/3735635434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 msgr2=0x55f24aa62b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.557+0000 7facf883a740 1 --2- 172.21.15.107:0/3735635434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa62b40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7face0009900 tx=0x7face0031750 comp rx=0 tx=0).stop 2024-05-09T08:35:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.557+0000 7facf883a740 1 -- 172.21.15.107:0/3735635434 shutdown_connections 2024-05-09T08:35:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.557+0000 7facf883a740 1 --2- 172.21.15.107:0/3735635434 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f24aa758d0 0x55f24aa77cc0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.557+0000 7facf883a740 1 --2- 172.21.15.107:0/3735635434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa65680 0x55f24aa63080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.557+0000 7facf883a740 1 --2- 172.21.15.107:0/3735635434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa62b40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.557+0000 7facceffd640 0 client.25837 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:38.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.557+0000 7facf883a740 1 --2- 172.21.15.107:0/3735635434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa56160 0x55f24aa62600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.557+0000 7facf883a740 1 -- 172.21.15.107:0/3735635434 >> 172.21.15.107:0/3735635434 conn(0x55f24aa525a0 msgr2=0x55f24aa58350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:38.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.557+0000 7facf883a740 1 -- 172.21.15.107:0/3735635434 shutdown_connections 2024-05-09T08:35:38.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.557+0000 7facf883a740 1 -- 172.21.15.107:0/3735635434 wait complete. 2024-05-09T08:35:38.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.557+0000 7facf883a740 20 client.25837 trim_cache size 0 max 16384 2024-05-09T08:35:38.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.578+0000 7facf883a740 1 Processor -- start 2024-05-09T08:35:38.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.578+0000 7facf883a740 1 -- start start 2024-05-09T08:35:38.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7facf883a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa56160 0x55f24aa71390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7facf883a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa4d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7facf883a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa65680 0x55f24aa7d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7facf883a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7facf01af1f0 con 0x55f24aa56160 2024-05-09T08:35:38.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7facf883a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7facf01acac0 con 0x55f24aa65680 2024-05-09T08:35:38.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7facf883a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7facf01aa910 con 0x55f24aa56a80 2024-05-09T08:35:38.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7face7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa56160 0x55f24aa71390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7face7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa56160 0x55f24aa71390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42272/0 (socket says 172.21.15.107:42272) 2024-05-09T08:35:38.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7face7fff640 1 -- 172.21.15.107:0/2575997127 learned_addr learned my addr 172.21.15.107:0/2575997127 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:38.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7facef7fe640 1 --2- 172.21.15.107:0/2575997127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa4d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7face7fff640 1 --2- 172.21.15.107:0/2575997127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa65680 0x55f24aa7d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7face7fff640 1 -- 172.21.15.107:0/2575997127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 msgr2=0x55f24aa4d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:38.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7face7fff640 1 --2- 172.21.15.107:0/2575997127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa4d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7face7fff640 1 -- 172.21.15.107:0/2575997127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa65680 msgr2=0x55f24aa7d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:38.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7face7fff640 1 --2- 172.21.15.107:0/2575997127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa65680 0x55f24aa7d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7face7fff640 1 -- 172.21.15.107:0/2575997127 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f24aa615b0 con 0x55f24aa56160 2024-05-09T08:35:38.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7facef7fe640 1 --2- 172.21.15.107:0/2575997127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa4d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:38.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.579+0000 7face7fff640 1 --2- 172.21.15.107:0/2575997127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa56160 0x55f24aa71390 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x55f24aa78290 tx=0x7facd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369d1fcc30b191ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:38.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.580+0000 7face57fa640 1 -- 172.21.15.107:0/2575997127 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd8019040 con 0x55f24aa56160 2024-05-09T08:35:38.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.580+0000 7face57fa640 1 -- 172.21.15.107:0/2575997127 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7facd8004230 con 0x55f24aa56160 2024-05-09T08:35:38.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.580+0000 7face57fa640 1 -- 172.21.15.107:0/2575997127 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd80226b0 con 0x55f24aa56160 2024-05-09T08:35:38.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.580+0000 7facf883a740 1 -- 172.21.15.107:0/2575997127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa56160 msgr2=0x55f24aa71390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.580+0000 7facf883a740 1 --2- 172.21.15.107:0/2575997127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa56160 0x55f24aa71390 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x55f24aa78290 tx=0x7facd80046c0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.580+0000 7facf883a740 1 -- 172.21.15.107:0/2575997127 shutdown_connections 2024-05-09T08:35:38.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.580+0000 7facf883a740 1 --2- 172.21.15.107:0/2575997127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa65680 0x55f24aa7d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.580+0000 7facf883a740 1 --2- 172.21.15.107:0/2575997127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa4d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.580+0000 7facf883a740 1 --2- 172.21.15.107:0/2575997127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa56160 0x55f24aa71390 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.580+0000 7facf883a740 1 -- 172.21.15.107:0/2575997127 >> 172.21.15.107:0/2575997127 conn(0x55f24aa525a0 msgr2=0x55f24aa62e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:38.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.580+0000 7facf883a740 1 -- 172.21.15.107:0/2575997127 shutdown_connections 2024-05-09T08:35:38.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.580+0000 7facf883a740 1 -- 172.21.15.107:0/2575997127 wait complete. 2024-05-09T08:35:38.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.581+0000 7facf883a740 1 -- start start 2024-05-09T08:35:38.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.581+0000 7facf883a740 1 Processor -- start 2024-05-09T08:35:38.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.581+0000 7facf883a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa56160 0x55f24aa69340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.581+0000 7facf883a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa69880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.581+0000 7facf883a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa65680 0x55f24aa69dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.581+0000 7facf883a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7facf01aaf80 con 0x55f24aa56160 2024-05-09T08:35:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.581+0000 7facf883a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7facf01ac3f0 con 0x55f24aa65680 2024-05-09T08:35:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.581+0000 7facf883a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7facf01ab650 con 0x55f24aa56a80 2024-05-09T08:35:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.581+0000 7facf883a740 10 client.0 _ll_register_callbacks cb 0x7facd8009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.581+0000 7face7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa56160 0x55f24aa69340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.581+0000 7facef7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa69880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.581+0000 7face7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa56160 0x55f24aa69340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42278/0 (socket says 172.21.15.107:42278) 2024-05-09T08:35:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.582+0000 7face7fff640 1 -- 172.21.15.107:0/1308041500 learned_addr learned my addr 172.21.15.107:0/1308041500 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.582+0000 7face7fff640 1 --2- 172.21.15.107:0/1308041500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa65680 0x55f24aa69dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.582+0000 7facef7fe640 1 -- 172.21.15.107:0/1308041500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa65680 msgr2=0x55f24aa69dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.582+0000 7facef7fe640 1 --2- 172.21.15.107:0/1308041500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa65680 0x55f24aa69dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.582+0000 7facef7fe640 1 -- 172.21.15.107:0/1308041500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa56160 msgr2=0x55f24aa69340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.582+0000 7facef7fe640 1 --2- 172.21.15.107:0/1308041500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa56160 0x55f24aa69340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.582+0000 7facef7fe640 1 -- 172.21.15.107:0/1308041500 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f24aadabf0 con 0x55f24aa56a80 2024-05-09T08:35:38.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.582+0000 7facef7fe640 1 --2- 172.21.15.107:0/1308041500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa69880 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7face0004b10 tx=0x7face00051c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e358fdd841b4559 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.582+0000 7facceffd640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:35:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.582+0000 7facceffd640 1 -- 172.21.15.107:0/1308041500 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face004a040 con 0x55f24aa56a80 2024-05-09T08:35:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.582+0000 7facf883a740 1 -- 172.21.15.107:0/1308041500 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facd8004230 con 0x55f24aa56a80 2024-05-09T08:35:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.582+0000 7facceffd640 1 -- 172.21.15.107:0/1308041500 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7face0004250 con 0x55f24aa56a80 2024-05-09T08:35:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.582+0000 7facceffd640 1 -- 172.21.15.107:0/1308041500 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7face003f070 con 0x55f24aa56a80 2024-05-09T08:35:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.582+0000 7facf883a740 10 client.25843 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.582+0000 7facf883a740 1 -- 172.21.15.107:0/1308041500 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f24aadbee0 con 0x55f24aa56a80 2024-05-09T08:35:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.583+0000 7facceffd640 1 -- 172.21.15.107:0/1308041500 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7facd8004230 con 0x55f24aa56a80 2024-05-09T08:35:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.583+0000 7facceffd640 1 -- 172.21.15.107:0/1308041500 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7face00035f0 con 0x55f24aa56a80 2024-05-09T08:35:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.583+0000 7facf883a740 10 client.25843 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.583+0000 7facf883a740 10 client.25843 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.583+0000 7facf883a740 20 client.25843 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.583+0000 7facf883a740 1 --2- 172.21.15.107:0/1308041500 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f24aadc660 0x55f24aadea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:38.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.583+0000 7facf883a740 4 client.25843 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:35:38.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.583+0000 7facf883a740 1 -- 172.21.15.107:0/1308041500 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f24a9c3780 con 0x55f24aadc660 2024-05-09T08:35:38.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.583+0000 7face7fff640 1 --2- 172.21.15.107:0/1308041500 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f24aadc660 0x55f24aadea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:38.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.583+0000 7face7fff640 1 --2- 172.21.15.107:0/1308041500 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f24aadc660 0x55f24aadea80 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17b55ed929f71158 server_cookie=76fa7c8c9225a866 in_seq=0 out_seq=0 2024-05-09T08:35:38.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.583+0000 7facceffd640 10 client.25843 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:38.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facceffd640 1 -- 172.21.15.107:0/1308041500 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f24a9c3780 con 0x55f24aadc660 2024-05-09T08:35:38.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facceffd640 10 client.25843 handle_command_reply: tid=0 2024-05-09T08:35:38.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facf883a740 1 client.25843 shutdown 2024-05-09T08:35:38.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facf883a740 1 -- 172.21.15.107:0/1308041500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 msgr2=0x55f24aa69880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facf883a740 1 --2- 172.21.15.107:0/1308041500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa69880 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7face0004b10 tx=0x7face00051c0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facf883a740 1 -- 172.21.15.107:0/1308041500 shutdown_connections 2024-05-09T08:35:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facf883a740 1 --2- 172.21.15.107:0/1308041500 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f24aadc660 0x55f24aadea80 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facf883a740 1 --2- 172.21.15.107:0/1308041500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f24aa65680 0x55f24aa69dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facceffd640 0 client.25843 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facf883a740 1 --2- 172.21.15.107:0/1308041500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f24aa56a80 0x55f24aa69880 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facf883a740 1 --2- 172.21.15.107:0/1308041500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f24aa56160 0x55f24aa69340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facf883a740 1 -- 172.21.15.107:0/1308041500 >> 172.21.15.107:0/1308041500 conn(0x55f24aa525a0 msgr2=0x55f24aa625a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facf883a740 1 -- 172.21.15.107:0/1308041500 shutdown_connections 2024-05-09T08:35:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facf883a740 1 -- 172.21.15.107:0/1308041500 wait complete. 2024-05-09T08:35:38.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facf883a740 20 client.25843 trim_cache size 0 max 16384 2024-05-09T08:35:38.589 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:35:38.590 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:35:38.590 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:35:38.590 INFO:teuthology.orchestra.run.smithi107.stdout: "9f73183a-d1d2-4202-96bd-e4606258c6f7": { 2024-05-09T08:35:38.590 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:35:38.590 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "9f73183a-d1d2-4202-96bd-e4606258c6f7", 2024-05-09T08:35:38.590 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:35:38.590 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:35:38.590 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:35:38.590 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:35:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facf5b29640 1 -- 172.21.15.107:0/3319646362 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7facdc0a2d00 msgr2=0x7facdc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.584+0000 7facf5b29640 1 --2- 172.21.15.107:0/3319646362 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7facdc0a2d00 0x7facdc0a51c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7face80cd210 tx=0x7face807c040 comp rx=0 tx=0).stop 2024-05-09T08:35:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.585+0000 7facf5b29640 1 -- 172.21.15.107:0/3319646362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf010bdf0 msgr2=0x7facf019c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.585+0000 7facf5b29640 1 --2- 172.21.15.107:0/3319646362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf010bdf0 0x7facf019c910 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7face800eb10 tx=0x7face800c2d0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.585+0000 7facf5b29640 1 -- 172.21.15.107:0/3319646362 shutdown_connections 2024-05-09T08:35:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.585+0000 7facf5b29640 1 --2- 172.21.15.107:0/3319646362 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7facdc0a2d00 0x7facdc0a51c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.585+0000 7facf5b29640 1 --2- 172.21.15.107:0/3319646362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facf010bdf0 0x7facf019c910 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.585+0000 7facf5b29640 1 --2- 172.21.15.107:0/3319646362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf01033d0 0x7facf0198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.585+0000 7facf5b29640 1 --2- 172.21.15.107:0/3319646362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facf0102ab0 0x7facf0198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:38.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.585+0000 7facf5b29640 1 -- 172.21.15.107:0/3319646362 >> 172.21.15.107:0/3319646362 conn(0x7facf0077830 msgr2=0x7facf00fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:38.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.585+0000 7facf5b29640 1 -- 172.21.15.107:0/3319646362 shutdown_connections 2024-05-09T08:35:38.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:35:38.585+0000 7facf5b29640 1 -- 172.21.15.107:0/3319646362 wait complete. 2024-05-09T08:35:38.595 INFO:tasks.cephfs.filesystem:scrub status for tag:9f73183a-d1d2-4202-96bd-e4606258c6f7 - {'path': '/', 'tag': '9f73183a-d1d2-4202-96bd-e4606258c6f7', 'options': 'recursive,force'} 2024-05-09T08:35:38.994 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:38 smithi157 ceph-mon[33614]: pgmap v71: 129 pgs: 129 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-05-09T08:35:39.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:38 smithi107 ceph-mon[30122]: pgmap v71: 129 pgs: 129 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-05-09T08:35:39.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:38 smithi110 ceph-mon[33622]: pgmap v71: 129 pgs: 129 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-05-09T08:35:40.022 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:39 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:35:40.022 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:39 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.022 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:39 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:35:40.022 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:39 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:35:40.022 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:39 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.022 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:39 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:35:40.022 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:39 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.022 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:39 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:35:40.023 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:39 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.023 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:39 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:35:40.023 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:39 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.023 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:39 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:35:40.023 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:39 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.023 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:39 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:39 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:35:40.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:39 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:39 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:35:40.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:39 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:35:40.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:39 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:39 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:35:40.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:39 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:39 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:35:40.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:39 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:39 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:35:40.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:39 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:39 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:35:40.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:39 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:39 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:39 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:35:40.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:39 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:39 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:35:40.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:39 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:35:40.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:39 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:39 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:35:40.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:39 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:39 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:35:40.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:39 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:39 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:35:40.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:39 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:39 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:35:40.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:39 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:40.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:39 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:35:41.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:40 smithi107 ceph-mon[30122]: pgmap v72: 129 pgs: 129 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:35:41.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:40 smithi157 ceph-mon[33614]: pgmap v72: 129 pgs: 129 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:35:41.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:40 smithi110 ceph-mon[33622]: pgmap v72: 129 pgs: 129 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:35:41.888 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.886+0000 7f26030a5640 1 -- 172.21.15.107:0/1096683529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25fc101120 msgr2=0x7f25fc10d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:41.888 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.886+0000 7f26030a5640 1 --2- 172.21.15.107:0/1096683529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25fc101120 0x7f25fc10d5e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009cd0 tx=0x7f25f402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.888 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.886+0000 7f26030a5640 1 -- 172.21.15.107:0/1096683529 shutdown_connections 2024-05-09T08:35:41.888 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.886+0000 7f26030a5640 1 --2- 172.21.15.107:0/1096683529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25fc101120 0x7f25fc10d5e0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.888 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.886+0000 7f26030a5640 1 --2- 172.21.15.107:0/1096683529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25fc100760 0x7f25fc100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.888 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.886+0000 7f26030a5640 1 --2- 172.21.15.107:0/1096683529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25fc106720 0x7f25fc106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.886+0000 7f26030a5640 1 -- 172.21.15.107:0/1096683529 >> 172.21.15.107:0/1096683529 conn(0x7f25fc0778b0 msgr2=0x7f25fc077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:41.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.886+0000 7f26030a5640 1 -- 172.21.15.107:0/1096683529 shutdown_connections 2024-05-09T08:35:41.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.886+0000 7f26030a5640 1 -- 172.21.15.107:0/1096683529 wait complete. 2024-05-09T08:35:41.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f26030a5640 1 Processor -- start 2024-05-09T08:35:41.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f26030a5640 1 -- start start 2024-05-09T08:35:41.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f26030a5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25fc100760 0x7f25fc1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f26030a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25fc101120 0x7f25fc1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f26030a5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25fc106720 0x7f25fc19c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f26030a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc110a60 con 0x7f25fc101120 2024-05-09T08:35:41.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f26030a5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1108e0 con 0x7f25fc106720 2024-05-09T08:35:41.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f26030a5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc110be0 con 0x7f25fc100760 2024-05-09T08:35:41.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f25fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25fc101120 0x7f25fc1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f25fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25fc101120 0x7f25fc1985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33274/0 (socket says 172.21.15.107:33274) 2024-05-09T08:35:41.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f25fbfff640 1 -- 172.21.15.107:0/1594290000 learned_addr learned my addr 172.21.15.107:0/1594290000 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:41.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f2600e41640 1 --2- 172.21.15.107:0/1594290000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25fc100760 0x7f25fc1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f25fbfff640 1 -- 172.21.15.107:0/1594290000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25fc100760 msgr2=0x7f25fc1980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:41.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f25fbfff640 1 --2- 172.21.15.107:0/1594290000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25fc100760 0x7f25fc1980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f25fbfff640 1 -- 172.21.15.107:0/1594290000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25fc106720 msgr2=0x7f25fc19c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:41.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f2601642640 1 --2- 172.21.15.107:0/1594290000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25fc106720 0x7f25fc19c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f25fbfff640 1 --2- 172.21.15.107:0/1594290000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25fc106720 0x7f25fc19c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f25fbfff640 1 -- 172.21.15.107:0/1594290000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25fc19d0a0 con 0x7f25fc101120 2024-05-09T08:35:41.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f2600e41640 1 --2- 172.21.15.107:0/1594290000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25fc100760 0x7f25fc1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:35:41.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.887+0000 7f25fbfff640 1 --2- 172.21.15.107:0/1594290000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25fc101120 0x7f25fc1985f0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f25e800b4a0 tx=0x7f25e800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:41.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.888+0000 7f25f9ffb640 1 -- 172.21.15.107:0/1594290000 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e8014070 con 0x7f25fc101120 2024-05-09T08:35:41.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.888+0000 7f25f9ffb640 1 -- 172.21.15.107:0/1594290000 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25e8010ac0 con 0x7f25fc101120 2024-05-09T08:35:41.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.888+0000 7f26030a5640 1 -- 172.21.15.107:0/1594290000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25fc19d390 con 0x7f25fc101120 2024-05-09T08:35:41.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.888+0000 7f25f9ffb640 1 -- 172.21.15.107:0/1594290000 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e8017c30 con 0x7f25fc101120 2024-05-09T08:35:41.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.888+0000 7f26030a5640 1 -- 172.21.15.107:0/1594290000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25fc1a4c20 con 0x7f25fc101120 2024-05-09T08:35:41.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.890+0000 7f25f9ffb640 1 -- 172.21.15.107:0/1594290000 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f25e8017070 con 0x7f25fc101120 2024-05-09T08:35:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.890+0000 7f2605dba740 1 Processor -- start 2024-05-09T08:35:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f2605dba740 1 -- start start 2024-05-09T08:35:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f2605dba740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828b4450 0x561e828b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f2605dba740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4d70 0x561e828bf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f2605dba740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828bf630 0x561e828c1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f2605dba740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561e82819340 con 0x561e828bf630 2024-05-09T08:35:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f2605dba740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561e827c1290 con 0x561e828b4d70 2024-05-09T08:35:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f2605dba740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x561e82815300 con 0x561e828b4450 2024-05-09T08:35:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f25fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828bf630 0x561e828c1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f25fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828bf630 0x561e828c1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33280/0 (socket says 172.21.15.107:33280) 2024-05-09T08:35:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f25fbfff640 1 -- 172.21.15.107:0/451748968 learned_addr learned my addr 172.21.15.107:0/451748968 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f2600e41640 1 --2- 172.21.15.107:0/451748968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4d70 0x561e828bf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f25f9ffb640 1 --2- 172.21.15.107:0/1594290000 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f25ec0a2e20 0x7f25ec0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f2601642640 1 --2- 172.21.15.107:0/451748968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828b4450 0x561e828b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f2601642640 1 --2- 172.21.15.107:0/1594290000 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f25ec0a2e20 0x7f25ec0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f25f9ffb640 1 -- 172.21.15.107:0/1594290000 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f25e80ced80 con 0x7f25fc101120 2024-05-09T08:35:41.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f2600e41640 1 -- 172.21.15.107:0/451748968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828b4450 msgr2=0x561e828b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:41.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f2600e41640 1 --2- 172.21.15.107:0/451748968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828b4450 0x561e828b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f2600e41640 1 -- 172.21.15.107:0/451748968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828bf630 msgr2=0x561e828c1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:41.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f2600e41640 1 --2- 172.21.15.107:0/451748968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828bf630 0x561e828c1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f2600e41640 1 -- 172.21.15.107:0/451748968 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e827e58e0 con 0x561e828b4d70 2024-05-09T08:35:41.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.891+0000 7f25fbfff640 1 --2- 172.21.15.107:0/451748968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828bf630 0x561e828c1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:41.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.892+0000 7f2600e41640 1 --2- 172.21.15.107:0/451748968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4d70 0x561e828bf0f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f25f000a940 tx=0x7f25f000ae00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f48a5741faa7cc0d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:41.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.892+0000 7f2601642640 1 --2- 172.21.15.107:0/1594290000 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f25ec0a2e20 0x7f25ec0a52e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009a60 tx=0x7f25f40095c0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:41.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.892+0000 7f25e6ffd640 1 -- 172.21.15.107:0/451748968 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f0015070 con 0x561e828b4d70 2024-05-09T08:35:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.892+0000 7f25e6ffd640 1 -- 172.21.15.107:0/451748968 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25f000cc00 con 0x561e828b4d70 2024-05-09T08:35:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.892+0000 7f25e6ffd640 1 -- 172.21.15.107:0/451748968 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f0011e60 con 0x561e828b4d70 2024-05-09T08:35:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.893+0000 7f2605dba740 1 -- 172.21.15.107:0/451748968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4d70 msgr2=0x561e828bf0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.893+0000 7f2605dba740 1 --2- 172.21.15.107:0/451748968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4d70 0x561e828bf0f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f25f000a940 tx=0x7f25f000ae00 comp rx=0 tx=0).stop 2024-05-09T08:35:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.893+0000 7f2605dba740 1 -- 172.21.15.107:0/451748968 shutdown_connections 2024-05-09T08:35:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.893+0000 7f2605dba740 1 --2- 172.21.15.107:0/451748968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828bf630 0x561e828c1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.893+0000 7f2605dba740 1 --2- 172.21.15.107:0/451748968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4d70 0x561e828bf0f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.893+0000 7f2605dba740 1 --2- 172.21.15.107:0/451748968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828b4450 0x561e828b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.893+0000 7f2605dba740 1 -- 172.21.15.107:0/451748968 >> 172.21.15.107:0/451748968 conn(0x561e828b0960 msgr2=0x561e828b0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.893+0000 7f2605dba740 1 -- 172.21.15.107:0/451748968 shutdown_connections 2024-05-09T08:35:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.893+0000 7f2605dba740 1 -- 172.21.15.107:0/451748968 wait complete. 2024-05-09T08:35:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.893+0000 7f2605dba740 1 -- start start 2024-05-09T08:35:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.893+0000 7f2605dba740 1 Processor -- start 2024-05-09T08:35:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f2605dba740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828b4450 0x561e828bc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f2605dba740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828b4d70 0x561e828bca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f2605dba740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828bf630 0x561e828bcf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f2605dba740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561e828c2260 con 0x561e828b4d70 2024-05-09T08:35:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f2605dba740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561e827d4790 con 0x561e828bf630 2024-05-09T08:35:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f2605dba740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x561e828c23e0 con 0x561e828b4450 2024-05-09T08:35:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f2605dba740 10 client.0 _ll_register_callbacks cb 0x561e8280fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f25fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828b4d70 0x561e828bca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f25fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828b4d70 0x561e828bca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33290/0 (socket says 172.21.15.107:33290) 2024-05-09T08:35:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f25fbfff640 1 -- 172.21.15.107:0/12247350 learned_addr learned my addr 172.21.15.107:0/12247350 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f2601642640 1 --2- 172.21.15.107:0/12247350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828bf630 0x561e828bcf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f2600e41640 1 --2- 172.21.15.107:0/12247350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828b4450 0x561e828bc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f25fbfff640 1 -- 172.21.15.107:0/12247350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828b4450 msgr2=0x561e828bc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f25fbfff640 1 --2- 172.21.15.107:0/12247350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828b4450 0x561e828bc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f25fbfff640 1 -- 172.21.15.107:0/12247350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828bf630 msgr2=0x561e828bcf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f25fbfff640 1 --2- 172.21.15.107:0/12247350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828bf630 0x561e828bcf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f25fbfff640 1 -- 172.21.15.107:0/12247350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e8252d210 con 0x561e828b4d70 2024-05-09T08:35:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f25fbfff640 1 --2- 172.21.15.107:0/12247350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828b4d70 0x561e828bca40 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f25e80cc330 tx=0x7f25e809b050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f32bcf75b5fa479 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.894+0000 7f25dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.895+0000 7f25dbfff640 1 -- 172.21.15.107:0/12247350 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e801e070 con 0x561e828b4d70 2024-05-09T08:35:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.895+0000 7f25dbfff640 1 -- 172.21.15.107:0/12247350 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f25e8099350 con 0x561e828b4d70 2024-05-09T08:35:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.895+0000 7f2605dba740 1 -- 172.21.15.107:0/12247350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25f000cc00 con 0x561e828b4d70 2024-05-09T08:35:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.895+0000 7f25dbfff640 1 -- 172.21.15.107:0/12247350 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f25e80976c0 con 0x561e828b4d70 2024-05-09T08:35:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.895+0000 7f2605dba740 10 client.16389 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.895+0000 7f2605dba740 1 -- 172.21.15.107:0/12247350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561e825f9620 con 0x561e828b4d70 2024-05-09T08:35:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.895+0000 7f25dbfff640 1 -- 172.21.15.107:0/12247350 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25f000cc00 con 0x561e828b4d70 2024-05-09T08:35:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.895+0000 7f25dbfff640 1 -- 172.21.15.107:0/12247350 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f25e80cc3f0 con 0x561e828b4d70 2024-05-09T08:35:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.895+0000 7f2605dba740 10 client.16389 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.895+0000 7f2605dba740 10 client.16389 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.895+0000 7f2605dba740 20 client.16389 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.895+0000 7f2605dba740 1 --2- 172.21.15.107:0/12247350 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561e828cf780 0x561e828d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.895+0000 7f2605dba740 4 client.16389 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.895+0000 7f2605dba740 1 -- 172.21.15.107:0/12247350 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e82826a70 con 0x561e828cf780 2024-05-09T08:35:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.896+0000 7f2600e41640 1 --2- 172.21.15.107:0/12247350 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561e828cf780 0x561e828d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.896+0000 7f25dbfff640 10 client.16389 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.896+0000 7f2600e41640 1 --2- 172.21.15.107:0/12247350 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561e828cf780 0x561e828d1be0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8cb2f0ec3d2c005 server_cookie=aa23416638aad531 in_seq=0 out_seq=0 2024-05-09T08:35:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.897+0000 7f25dbfff640 1 -- 172.21.15.107:0/12247350 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561e82826a70 con 0x561e828cf780 2024-05-09T08:35:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.897+0000 7f25dbfff640 10 client.16389 handle_command_reply: tid=0 2024-05-09T08:35:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.897+0000 7f2605dba740 1 client.16389 shutdown 2024-05-09T08:35:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.897+0000 7f2605dba740 1 -- 172.21.15.107:0/12247350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828b4d70 msgr2=0x561e828bca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.897+0000 7f2605dba740 1 --2- 172.21.15.107:0/12247350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828b4d70 0x561e828bca40 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f25e80cc330 tx=0x7f25e809b050 comp rx=0 tx=0).stop 2024-05-09T08:35:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.897+0000 7f2605dba740 1 -- 172.21.15.107:0/12247350 shutdown_connections 2024-05-09T08:35:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.897+0000 7f2605dba740 1 --2- 172.21.15.107:0/12247350 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561e828cf780 0x561e828d1be0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.897+0000 7f2605dba740 1 --2- 172.21.15.107:0/12247350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828bf630 0x561e828bcf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.897+0000 7f2605dba740 1 --2- 172.21.15.107:0/12247350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828b4d70 0x561e828bca40 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.897+0000 7f2605dba740 1 --2- 172.21.15.107:0/12247350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828b4450 0x561e828bc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.897+0000 7f2605dba740 1 -- 172.21.15.107:0/12247350 >> 172.21.15.107:0/12247350 conn(0x561e828b0960 msgr2=0x561e828b6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.897+0000 7f25dbfff640 0 client.16389 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.897+0000 7f2605dba740 1 -- 172.21.15.107:0/12247350 shutdown_connections 2024-05-09T08:35:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.897+0000 7f2605dba740 1 -- 172.21.15.107:0/12247350 wait complete. 2024-05-09T08:35:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.897+0000 7f2605dba740 20 client.16389 trim_cache size 0 max 16384 2024-05-09T08:35:41.916 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.915+0000 7f2605dba740 1 Processor -- start 2024-05-09T08:35:41.916 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.915+0000 7f2605dba740 1 -- start start 2024-05-09T08:35:41.916 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.915+0000 7f2605dba740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4450 0x561e828cb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.916 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.915+0000 7f2605dba740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828b4d70 0x561e828b0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.916 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.915+0000 7f2605dba740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828bf630 0x561e828b6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.916 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.915+0000 7f2605dba740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1b13c0 con 0x561e828bf630 2024-05-09T08:35:41.917 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.915+0000 7f2605dba740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1aec90 con 0x561e828b4450 2024-05-09T08:35:41.917 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.915+0000 7f2605dba740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1acae0 con 0x561e828b4d70 2024-05-09T08:35:41.917 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.915+0000 7f2601642640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828bf630 0x561e828b6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.917 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.915+0000 7f2601642640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828bf630 0x561e828b6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33304/0 (socket says 172.21.15.107:33304) 2024-05-09T08:35:41.917 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.915+0000 7f2601642640 1 -- 172.21.15.107:0/42769619 learned_addr learned my addr 172.21.15.107:0/42769619 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:41.917 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.915+0000 7f2600e41640 1 --2- 172.21.15.107:0/42769619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4450 0x561e828cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.917 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.915+0000 7f25fbfff640 1 --2- 172.21.15.107:0/42769619 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828b4d70 0x561e828b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.917 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.915+0000 7f2601642640 1 -- 172.21.15.107:0/42769619 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828b4d70 msgr2=0x561e828b0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:41.917 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.915+0000 7f2601642640 1 --2- 172.21.15.107:0/42769619 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828b4d70 0x561e828b0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.917 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.915+0000 7f2601642640 1 -- 172.21.15.107:0/42769619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4450 msgr2=0x561e828cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:41.917 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.916+0000 7f2601642640 1 --2- 172.21.15.107:0/42769619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4450 0x561e828cb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.916+0000 7f2601642640 1 -- 172.21.15.107:0/42769619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e828aef00 con 0x561e828bf630 2024-05-09T08:35:41.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.916+0000 7f2600e41640 1 --2- 172.21.15.107:0/42769619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4450 0x561e828cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:41.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.916+0000 7f2601642640 1 --2- 172.21.15.107:0/42769619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828bf630 0x561e828b6510 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f25f4031b70 tx=0x7f25f4041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b70a137710e24e57 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:41.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.916+0000 7f25e67fc640 1 -- 172.21.15.107:0/42769619 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f4004040 con 0x561e828bf630 2024-05-09T08:35:41.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.916+0000 7f25e67fc640 1 -- 172.21.15.107:0/42769619 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25f4043070 con 0x561e828bf630 2024-05-09T08:35:41.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.916+0000 7f25e67fc640 1 -- 172.21.15.107:0/42769619 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f4030c60 con 0x561e828bf630 2024-05-09T08:35:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.917+0000 7f2605dba740 1 -- 172.21.15.107:0/42769619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828bf630 msgr2=0x561e828b6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.917+0000 7f2605dba740 1 --2- 172.21.15.107:0/42769619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828bf630 0x561e828b6510 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f25f4031b70 tx=0x7f25f4041000 comp rx=0 tx=0).stop 2024-05-09T08:35:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.917+0000 7f2605dba740 1 -- 172.21.15.107:0/42769619 shutdown_connections 2024-05-09T08:35:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.917+0000 7f2605dba740 1 --2- 172.21.15.107:0/42769619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828bf630 0x561e828b6510 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.917+0000 7f2605dba740 1 --2- 172.21.15.107:0/42769619 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828b4d70 0x561e828b0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.917+0000 7f2605dba740 1 --2- 172.21.15.107:0/42769619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4450 0x561e828cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.917+0000 7f2605dba740 1 -- 172.21.15.107:0/42769619 >> 172.21.15.107:0/42769619 conn(0x561e828b0960 msgr2=0x561e828b5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.917+0000 7f2605dba740 1 -- 172.21.15.107:0/42769619 shutdown_connections 2024-05-09T08:35:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.917+0000 7f2605dba740 1 -- 172.21.15.107:0/42769619 wait complete. 2024-05-09T08:35:41.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.917+0000 7f2605dba740 1 -- start start 2024-05-09T08:35:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f2605dba740 1 Processor -- start 2024-05-09T08:35:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f2605dba740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4450 0x561e828c7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f2605dba740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828b4d70 0x561e828b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f2605dba740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828bf630 0x561e828b93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f2605dba740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1ad150 con 0x561e828b4d70 2024-05-09T08:35:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f2605dba740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1ae5c0 con 0x561e828b4450 2024-05-09T08:35:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f2605dba740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1ad820 con 0x561e828bf630 2024-05-09T08:35:41.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f2605dba740 10 client.0 _ll_register_callbacks cb 0x7f25e809af10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:41.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f25fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828b4d70 0x561e828b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f2601642640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828bf630 0x561e828b93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f25fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828b4d70 0x561e828b8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33312/0 (socket says 172.21.15.107:33312) 2024-05-09T08:35:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f25fbfff640 1 -- 172.21.15.107:0/4175150420 learned_addr learned my addr 172.21.15.107:0/4175150420 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f2600e41640 1 --2- 172.21.15.107:0/4175150420 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4450 0x561e828c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f25fbfff640 1 -- 172.21.15.107:0/4175150420 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828bf630 msgr2=0x561e828b93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f25fbfff640 1 --2- 172.21.15.107:0/4175150420 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828bf630 0x561e828b93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f25fbfff640 1 -- 172.21.15.107:0/4175150420 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4450 msgr2=0x561e828c7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.918+0000 7f25fbfff640 1 --2- 172.21.15.107:0/4175150420 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4450 0x561e828c7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.919+0000 7f25fbfff640 1 -- 172.21.15.107:0/4175150420 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e828b9b30 con 0x561e828b4d70 2024-05-09T08:35:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.919+0000 7f2600e41640 1 --2- 172.21.15.107:0/4175150420 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4450 0x561e828c7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.919+0000 7f25fbfff640 1 --2- 172.21.15.107:0/4175150420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828b4d70 0x561e828b8eb0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f25e8000c00 tx=0x7f25e80047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ce268d4b53f29c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.919+0000 7f25dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.919+0000 7f25dbfff640 1 -- 172.21.15.107:0/4175150420 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e801e070 con 0x561e828b4d70 2024-05-09T08:35:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.919+0000 7f25dbfff640 1 -- 172.21.15.107:0/4175150420 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f25e809a6b0 con 0x561e828b4d70 2024-05-09T08:35:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.919+0000 7f2605dba740 1 -- 172.21.15.107:0/4175150420 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25f4043070 con 0x561e828b4d70 2024-05-09T08:35:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.919+0000 7f25dbfff640 1 -- 172.21.15.107:0/4175150420 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f25e807f050 con 0x561e828b4d70 2024-05-09T08:35:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.919+0000 7f2605dba740 10 client.16395 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.919+0000 7f2605dba740 1 -- 172.21.15.107:0/4175150420 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561e82935cc0 con 0x561e828b4d70 2024-05-09T08:35:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.919+0000 7f25dbfff640 1 -- 172.21.15.107:0/4175150420 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25f4043070 con 0x561e828b4d70 2024-05-09T08:35:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.919+0000 7f25dbfff640 1 -- 172.21.15.107:0/4175150420 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f25e80b6070 con 0x561e828b4d70 2024-05-09T08:35:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.920+0000 7f2605dba740 10 client.16395 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.920+0000 7f2605dba740 10 client.16395 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.920+0000 7f2605dba740 20 client.16395 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.920+0000 7f2605dba740 1 --2- 172.21.15.107:0/4175150420 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561e82936470 0x561e82938890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.920+0000 7f2605dba740 4 client.16395 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:35:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.920+0000 7f2605dba740 1 -- 172.21.15.107:0/4175150420 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561e82820dc0 con 0x561e82936470 2024-05-09T08:35:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.920+0000 7f2600e41640 1 --2- 172.21.15.107:0/4175150420 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561e82936470 0x561e82938890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.920+0000 7f25dbfff640 10 client.16395 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:41.929 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.920+0000 7f2600e41640 1 --2- 172.21.15.107:0/4175150420 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561e82936470 0x561e82938890 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2363672a98873962 server_cookie=6b2a400e98ffa5dc in_seq=0 out_seq=0 2024-05-09T08:35:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.937+0000 7f25dbfff640 1 -- 172.21.15.107:0/4175150420 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561e82820dc0 con 0x561e82936470 2024-05-09T08:35:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.937+0000 7f25dbfff640 10 client.16395 handle_command_reply: tid=0 2024-05-09T08:35:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.937+0000 7f2605dba740 1 client.16395 shutdown 2024-05-09T08:35:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.937+0000 7f2605dba740 1 -- 172.21.15.107:0/4175150420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828b4d70 msgr2=0x561e828b8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.937+0000 7f2605dba740 1 --2- 172.21.15.107:0/4175150420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828b4d70 0x561e828b8eb0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f25e8000c00 tx=0x7f25e80047d0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.937+0000 7f2605dba740 1 -- 172.21.15.107:0/4175150420 shutdown_connections 2024-05-09T08:35:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.937+0000 7f2605dba740 1 --2- 172.21.15.107:0/4175150420 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561e82936470 0x561e82938890 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.937+0000 7f25dbfff640 0 client.16395 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.937+0000 7f2605dba740 1 --2- 172.21.15.107:0/4175150420 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e828bf630 0x561e828b93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.937+0000 7f2605dba740 1 --2- 172.21.15.107:0/4175150420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e828b4d70 0x561e828b8eb0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.937+0000 7f2605dba740 1 --2- 172.21.15.107:0/4175150420 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e828b4450 0x561e828c7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.937+0000 7f2605dba740 1 -- 172.21.15.107:0/4175150420 >> 172.21.15.107:0/4175150420 conn(0x561e828b0960 msgr2=0x561e828b6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:41.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.938+0000 7f2605dba740 1 -- 172.21.15.107:0/4175150420 shutdown_connections 2024-05-09T08:35:41.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.938+0000 7f2605dba740 1 -- 172.21.15.107:0/4175150420 wait complete. 2024-05-09T08:35:41.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.938+0000 7f2605dba740 20 client.16395 trim_cache size 0 max 16384 2024-05-09T08:35:41.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:35:41.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:35:41.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:35:41.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:35:41.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.938+0000 7f26030a5640 1 -- 172.21.15.107:0/1594290000 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f25ec0a2e20 msgr2=0x7f25ec0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:41.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.938+0000 7f26030a5640 1 --2- 172.21.15.107:0/1594290000 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f25ec0a2e20 0x7f25ec0a52e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009a60 tx=0x7f25f40095c0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.938+0000 7f26030a5640 1 -- 172.21.15.107:0/1594290000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25fc101120 msgr2=0x7f25fc1985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:41.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.938+0000 7f26030a5640 1 --2- 172.21.15.107:0/1594290000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25fc101120 0x7f25fc1985f0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f25e800b4a0 tx=0x7f25e800b960 comp rx=0 tx=0).stop 2024-05-09T08:35:41.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.938+0000 7f26030a5640 1 -- 172.21.15.107:0/1594290000 shutdown_connections 2024-05-09T08:35:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.938+0000 7f26030a5640 1 --2- 172.21.15.107:0/1594290000 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f25ec0a2e20 0x7f25ec0a52e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.938+0000 7f26030a5640 1 --2- 172.21.15.107:0/1594290000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25fc106720 0x7f25fc19c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.938+0000 7f26030a5640 1 --2- 172.21.15.107:0/1594290000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25fc101120 0x7f25fc1985f0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.938+0000 7f26030a5640 1 --2- 172.21.15.107:0/1594290000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25fc100760 0x7f25fc1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.938+0000 7f26030a5640 1 -- 172.21.15.107:0/1594290000 >> 172.21.15.107:0/1594290000 conn(0x7f25fc0778b0 msgr2=0x7f25fc0ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.938+0000 7f26030a5640 1 -- 172.21.15.107:0/1594290000 shutdown_connections 2024-05-09T08:35:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:41.938+0000 7f26030a5640 1 -- 172.21.15.107:0/1594290000 wait complete. 2024-05-09T08:35:43.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:42 smithi107 ceph-mon[30122]: pgmap v73: 129 pgs: 129 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:35:43.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:42 smithi157 ceph-mon[33614]: pgmap v73: 129 pgs: 129 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:35:43.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:42 smithi110 ceph-mon[33622]: pgmap v73: 129 pgs: 129 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:35:45.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:44 smithi107 ceph-mon[30122]: pgmap v74: 129 pgs: 129 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:35:45.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:44 smithi157 ceph-mon[33614]: pgmap v74: 129 pgs: 129 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:35:45.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:44 smithi110 ceph-mon[33622]: pgmap v74: 129 pgs: 129 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:35:47.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:46 smithi107 ceph-mon[30122]: pgmap v75: 129 pgs: 129 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 25 MiB/s wr, 41 op/s 2024-05-09T08:35:47.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:46 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:35:47.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:46 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:35:47.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:46 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:35:47.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:46 smithi157 ceph-mon[33614]: pgmap v75: 129 pgs: 129 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 25 MiB/s wr, 41 op/s 2024-05-09T08:35:47.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:46 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:35:47.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:46 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:35:47.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:46 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:35:47.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:46 smithi110 ceph-mon[33622]: pgmap v75: 129 pgs: 129 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 25 MiB/s wr, 41 op/s 2024-05-09T08:35:47.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:46 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:35:47.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:46 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:35:47.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:46 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:35:49.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:48 smithi107 ceph-mon[30122]: pgmap v76: 129 pgs: 129 active+clean; 1.4 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:35:49.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:48 smithi157 ceph-mon[33614]: pgmap v76: 129 pgs: 129 active+clean; 1.4 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:35:49.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:48 smithi110 ceph-mon[33622]: pgmap v76: 129 pgs: 129 active+clean; 1.4 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:35:50.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:49 smithi107 ceph-mon[30122]: pgmap v77: 129 pgs: 129 active+clean; 1.4 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 25 MiB/s wr, 44 op/s 2024-05-09T08:35:50.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:49 smithi157 ceph-mon[33614]: pgmap v77: 129 pgs: 129 active+clean; 1.4 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 25 MiB/s wr, 44 op/s 2024-05-09T08:35:50.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:49 smithi110 ceph-mon[33622]: pgmap v77: 129 pgs: 129 active+clean; 1.4 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 25 MiB/s wr, 44 op/s 2024-05-09T08:35:52.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:51 smithi107 ceph-mon[30122]: pgmap v78: 129 pgs: 129 active+clean; 1.4 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 25 MiB/s wr, 44 op/s 2024-05-09T08:35:52.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:51 smithi157 ceph-mon[33614]: pgmap v78: 129 pgs: 129 active+clean; 1.4 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 25 MiB/s wr, 44 op/s 2024-05-09T08:35:52.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:51 smithi110 ceph-mon[33622]: pgmap v78: 129 pgs: 129 active+clean; 1.4 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 25 MiB/s wr, 44 op/s 2024-05-09T08:35:54.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:53 smithi107 ceph-mon[30122]: pgmap v79: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 37 MiB/s wr, 52 op/s 2024-05-09T08:35:54.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:53 smithi157 ceph-mon[33614]: pgmap v79: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 37 MiB/s wr, 52 op/s 2024-05-09T08:35:54.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:53 smithi110 ceph-mon[33622]: pgmap v79: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 37 MiB/s wr, 52 op/s 2024-05-09T08:35:55.032 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.032+0000 7f4b39f90640 1 -- 172.21.15.107:0/3464392460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b341006f0 msgr2=0x7f4b3410e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:55.032 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.032+0000 7f4b39f90640 1 --2- 172.21.15.107:0/3464392460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b341006f0 0x7f4b3410e6c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009cd0 tx=0x7f4b2c02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.032 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.032+0000 7f4b39f90640 1 -- 172.21.15.107:0/3464392460 shutdown_connections 2024-05-09T08:35:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.032+0000 7f4b39f90640 1 --2- 172.21.15.107:0/3464392460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b341006f0 0x7f4b3410e6c0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.032+0000 7f4b39f90640 1 --2- 172.21.15.107:0/3464392460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b340ffd30 0x7f4b341001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.032+0000 7f4b39f90640 1 --2- 172.21.15.107:0/3464392460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b34106810 0x7f4b34106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.032+0000 7f4b39f90640 1 -- 172.21.15.107:0/3464392460 >> 172.21.15.107:0/3464392460 conn(0x7f4b34076cc0 msgr2=0x7f4b340fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.032+0000 7f4b39f90640 1 -- 172.21.15.107:0/3464392460 shutdown_connections 2024-05-09T08:35:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.032+0000 7f4b39f90640 1 -- 172.21.15.107:0/3464392460 wait complete. 2024-05-09T08:35:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.033+0000 7f4b39f90640 1 Processor -- start 2024-05-09T08:35:55.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.033+0000 7f4b39f90640 1 -- start start 2024-05-09T08:35:55.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.033+0000 7f4b39f90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b340ffd30 0x7f4b34198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.033+0000 7f4b39f90640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b341006f0 0x7f4b34198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.033+0000 7f4b39f90640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b34106810 0x7f4b3419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.033+0000 7f4b39f90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4b34111bd0 con 0x7f4b340ffd30 2024-05-09T08:35:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.033+0000 7f4b39f90640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4b34111a50 con 0x7f4b341006f0 2024-05-09T08:35:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.033+0000 7f4b39f90640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4b34111d50 con 0x7f4b34106810 2024-05-09T08:35:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.033+0000 7f4b337fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b340ffd30 0x7f4b34198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.033+0000 7f4b337fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b340ffd30 0x7f4b34198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51254/0 (socket says 172.21.15.107:51254) 2024-05-09T08:35:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.033+0000 7f4b337fe640 1 -- 172.21.15.107:0/1843955319 learned_addr learned my addr 172.21.15.107:0/1843955319 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.034+0000 7f4b33fff640 1 --2- 172.21.15.107:0/1843955319 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b34106810 0x7f4b3419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.034+0000 7f4b32ffd640 1 --2- 172.21.15.107:0/1843955319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b341006f0 0x7f4b34198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.034+0000 7f4b337fe640 1 -- 172.21.15.107:0/1843955319 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b34106810 msgr2=0x7f4b3419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.034+0000 7f4b337fe640 1 --2- 172.21.15.107:0/1843955319 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b34106810 0x7f4b3419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.034+0000 7f4b337fe640 1 -- 172.21.15.107:0/1843955319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b341006f0 msgr2=0x7f4b34198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.034+0000 7f4b337fe640 1 --2- 172.21.15.107:0/1843955319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b341006f0 0x7f4b34198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.034+0000 7f4b337fe640 1 -- 172.21.15.107:0/1843955319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b3419d000 con 0x7f4b340ffd30 2024-05-09T08:35:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.034+0000 7f4b32ffd640 1 --2- 172.21.15.107:0/1843955319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b341006f0 0x7f4b34198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.034+0000 7f4b337fe640 1 --2- 172.21.15.107:0/1843955319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b340ffd30 0x7f4b34198010 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4b2400d8d0 tx=0x7f4b2400dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.034+0000 7f4b30ff9640 1 -- 172.21.15.107:0/1843955319 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b24017030 con 0x7f4b340ffd30 2024-05-09T08:35:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.034+0000 7f4b30ff9640 1 -- 172.21.15.107:0/1843955319 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b24004020 con 0x7f4b340ffd30 2024-05-09T08:35:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.035+0000 7f4b30ff9640 1 -- 172.21.15.107:0/1843955319 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b240055b0 con 0x7f4b340ffd30 2024-05-09T08:35:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.035+0000 7f4b39f90640 1 -- 172.21.15.107:0/1843955319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b3419d2f0 con 0x7f4b340ffd30 2024-05-09T08:35:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.035+0000 7f4b39f90640 1 -- 172.21.15.107:0/1843955319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b341a4b80 con 0x7f4b340ffd30 2024-05-09T08:35:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.037+0000 7f4b30ff9640 1 -- 172.21.15.107:0/1843955319 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f4b240041d0 con 0x7f4b340ffd30 2024-05-09T08:35:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.038+0000 7f4b3cd7e740 1 Processor -- start 2024-05-09T08:35:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.038+0000 7f4b3cd7e740 1 -- start start 2024-05-09T08:35:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.038+0000 7f4b30ff9640 1 --2- 172.21.15.107:0/1843955319 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4b1c0a2df0 0x7f4b1c0a52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.038+0000 7f4b3cd7e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 0x55adceb13830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.038+0000 7f4b3cd7e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb13d70 0x55adceb1e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.038+0000 7f4b3cd7e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb1e630 0x55adceb20a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.038+0000 7f4b3cd7e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55adcea78340 con 0x55adceb13d70 2024-05-09T08:35:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.038+0000 7f4b3cd7e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55adcea20290 con 0x55adceb1e630 2024-05-09T08:35:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.038+0000 7f4b3cd7e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55adcea74300 con 0x55adceb13450 2024-05-09T08:35:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.038+0000 7f4b33fff640 1 --2- 172.21.15.107:0/1843955319 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4b1c0a2df0 0x7f4b1c0a52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.038+0000 7f4b30ff9640 1 -- 172.21.15.107:0/1843955319 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4b240cf0b0 con 0x7f4b340ffd30 2024-05-09T08:35:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.038+0000 7f4b32ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb13d70 0x55adceb1e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.038+0000 7f4b32ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb13d70 0x55adceb1e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51262/0 (socket says 172.21.15.107:51262) 2024-05-09T08:35:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.038+0000 7f4b32ffd640 1 -- 172.21.15.107:0/3393086087 learned_addr learned my addr 172.21.15.107:0/3393086087 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.038+0000 7f4b337fe640 1 --2- 172.21.15.107:0/3393086087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 0x55adceb13830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.038+0000 7f4b33fff640 1 --2- 172.21.15.107:0/3393086087 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb1e630 0x55adceb20a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.039+0000 7f4b32ffd640 1 -- 172.21.15.107:0/3393086087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 msgr2=0x55adceb13830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.039+0000 7f4b32ffd640 1 --2- 172.21.15.107:0/3393086087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 0x55adceb13830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.039+0000 7f4b32ffd640 1 -- 172.21.15.107:0/3393086087 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb1e630 msgr2=0x55adceb20a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.039+0000 7f4b32ffd640 1 --2- 172.21.15.107:0/3393086087 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb1e630 0x55adceb20a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.039+0000 7f4b32ffd640 1 -- 172.21.15.107:0/3393086087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adcea448e0 con 0x55adceb13d70 2024-05-09T08:35:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.039+0000 7f4b337fe640 1 --2- 172.21.15.107:0/3393086087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 0x55adceb13830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.039+0000 7f4b33fff640 1 --2- 172.21.15.107:0/1843955319 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4b1c0a2df0 0x7f4b1c0a52b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c006010 tx=0x7f4b2c031850 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.039+0000 7f4b32ffd640 1 --2- 172.21.15.107:0/3393086087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb13d70 0x55adceb1e0f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4b2800bd30 tx=0x7f4b2800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622496b417b358f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.039+0000 7f4b19ffb640 1 -- 172.21.15.107:0/3393086087 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b28015070 con 0x55adceb13d70 2024-05-09T08:35:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.039+0000 7f4b19ffb640 1 -- 172.21.15.107:0/3393086087 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b2801a910 con 0x55adceb13d70 2024-05-09T08:35:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.039+0000 7f4b19ffb640 1 -- 172.21.15.107:0/3393086087 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b28011850 con 0x55adceb13d70 2024-05-09T08:35:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.040+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/3393086087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb13d70 msgr2=0x55adceb1e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.040+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/3393086087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb13d70 0x55adceb1e0f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4b2800bd30 tx=0x7f4b2800f730 comp rx=0 tx=0).stop 2024-05-09T08:35:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.040+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/3393086087 shutdown_connections 2024-05-09T08:35:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.040+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/3393086087 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb1e630 0x55adceb20a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.040+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/3393086087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb13d70 0x55adceb1e0f0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.040+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/3393086087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 0x55adceb13830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.040+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/3393086087 >> 172.21.15.107:0/3393086087 conn(0x55adceb0f960 msgr2=0x55adceb0fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.040+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/3393086087 shutdown_connections 2024-05-09T08:35:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.040+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/3393086087 wait complete. 2024-05-09T08:35:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.040+0000 7f4b3cd7e740 1 -- start start 2024-05-09T08:35:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.040+0000 7f4b3cd7e740 1 Processor -- start 2024-05-09T08:35:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b3cd7e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13450 0x55adceb1b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b3cd7e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb13d70 0x55adceb1bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b3cd7e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb1e630 0x55adceb1c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b3cd7e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55adceb21260 con 0x55adceb13d70 2024-05-09T08:35:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b3cd7e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55adcea33790 con 0x55adceb13450 2024-05-09T08:35:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b3cd7e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55adceb213e0 con 0x55adceb1e630 2024-05-09T08:35:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b3cd7e740 10 client.0 _ll_register_callbacks cb 0x55adcea6ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:55.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b32ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb13d70 0x55adceb1bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b32ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb13d70 0x55adceb1bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51268/0 (socket says 172.21.15.107:51268) 2024-05-09T08:35:55.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b32ffd640 1 -- 172.21.15.107:0/4157218421 learned_addr learned my addr 172.21.15.107:0/4157218421 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:55.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b337fe640 1 --2- 172.21.15.107:0/4157218421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13450 0x55adceb1b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b33fff640 1 --2- 172.21.15.107:0/4157218421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb1e630 0x55adceb1c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b32ffd640 1 -- 172.21.15.107:0/4157218421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb1e630 msgr2=0x55adceb1c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:55.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b32ffd640 1 --2- 172.21.15.107:0/4157218421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb1e630 0x55adceb1c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b32ffd640 1 -- 172.21.15.107:0/4157218421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13450 msgr2=0x55adceb1b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:55.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b32ffd640 1 --2- 172.21.15.107:0/4157218421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13450 0x55adceb1b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b32ffd640 1 -- 172.21.15.107:0/4157218421 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adce78c210 con 0x55adceb13d70 2024-05-09T08:35:55.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b337fe640 1 --2- 172.21.15.107:0/4157218421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13450 0x55adceb1b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:35:55.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b32ffd640 1 --2- 172.21.15.107:0/4157218421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb13d70 0x55adceb1bae0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4b2800ff90 tx=0x7f4b2801a460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b16a018c8815f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:55.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:55.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.041+0000 7f4b12ffd640 1 -- 172.21.15.107:0/4157218421 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b28028060 con 0x55adceb13d70 2024-05-09T08:35:55.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.042+0000 7f4b12ffd640 1 -- 172.21.15.107:0/4157218421 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4b2801aac0 con 0x55adceb13d70 2024-05-09T08:35:55.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.042+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/4157218421 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b2801a910 con 0x55adceb13d70 2024-05-09T08:35:55.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.042+0000 7f4b12ffd640 1 -- 172.21.15.107:0/4157218421 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4b2802f050 con 0x55adceb13d70 2024-05-09T08:35:55.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.042+0000 7f4b3cd7e740 10 client.16407 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:55.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.042+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/4157218421 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55adce858620 con 0x55adceb13d70 2024-05-09T08:35:55.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.042+0000 7f4b12ffd640 1 -- 172.21.15.107:0/4157218421 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b2801a910 con 0x55adceb13d70 2024-05-09T08:35:55.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.042+0000 7f4b12ffd640 1 -- 172.21.15.107:0/4157218421 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f4b28004a20 con 0x55adceb13d70 2024-05-09T08:35:55.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.042+0000 7f4b3cd7e740 10 client.16407 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:55.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.042+0000 7f4b3cd7e740 10 client.16407 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:55.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.042+0000 7f4b3cd7e740 20 client.16407 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:55.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.042+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/4157218421 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55adceb2e280 0x55adceb30670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.042+0000 7f4b3cd7e740 4 client.16407 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:35:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.042+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/4157218421 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adcea85a70 con 0x55adceb2e280 2024-05-09T08:35:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.042+0000 7f4b337fe640 1 --2- 172.21.15.107:0/4157218421 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55adceb2e280 0x55adceb30670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.043+0000 7f4b337fe640 1 --2- 172.21.15.107:0/4157218421 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55adceb2e280 0x55adceb30670 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cc6fbd80465a9ca server_cookie=b6fc318ef2454939 in_seq=0 out_seq=0 2024-05-09T08:35:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.043+0000 7f4b12ffd640 10 client.16407 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.043+0000 7f4b12ffd640 1 -- 172.21.15.107:0/4157218421 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55adcea85a70 con 0x55adceb2e280 2024-05-09T08:35:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.043+0000 7f4b12ffd640 10 client.16407 handle_command_reply: tid=0 2024-05-09T08:35:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.043+0000 7f4b3cd7e740 1 client.16407 shutdown 2024-05-09T08:35:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.044+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/4157218421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb13d70 msgr2=0x55adceb1bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.044+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/4157218421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb13d70 0x55adceb1bae0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4b2800ff90 tx=0x7f4b2801a460 comp rx=0 tx=0).stop 2024-05-09T08:35:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.044+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/4157218421 shutdown_connections 2024-05-09T08:35:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.044+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/4157218421 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55adceb2e280 0x55adceb30670 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.044+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/4157218421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb1e630 0x55adceb1c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.044+0000 7f4b12ffd640 0 client.16407 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:55.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.044+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/4157218421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb13d70 0x55adceb1bae0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.044+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/4157218421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13450 0x55adceb1b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.044+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/4157218421 >> 172.21.15.107:0/4157218421 conn(0x55adceb0f960 msgr2=0x55adceb15690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:55.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.044+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/4157218421 shutdown_connections 2024-05-09T08:35:55.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.044+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/4157218421 wait complete. 2024-05-09T08:35:55.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.044+0000 7f4b3cd7e740 20 client.16407 trim_cache size 0 max 16384 2024-05-09T08:35:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.067+0000 7f4b3cd7e740 1 Processor -- start 2024-05-09T08:35:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.067+0000 7f4b3cd7e740 1 -- start start 2024-05-09T08:35:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.067+0000 7f4b3cd7e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 0x55adceb2a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.067+0000 7f4b3cd7e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13d70 0x55adceb1bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.067+0000 7f4b3cd7e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb1e630 0x55adceb35d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.067+0000 7f4b3cd7e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4b341af1a0 con 0x55adceb1e630 2024-05-09T08:35:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.067+0000 7f4b3cd7e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4b341aca70 con 0x55adceb13d70 2024-05-09T08:35:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.067+0000 7f4b3cd7e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4b341aa8c0 con 0x55adceb13450 2024-05-09T08:35:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.067+0000 7f4b33fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb1e630 0x55adceb35d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.067+0000 7f4b33fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb1e630 0x55adceb35d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51270/0 (socket says 172.21.15.107:51270) 2024-05-09T08:35:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.067+0000 7f4b33fff640 1 -- 172.21.15.107:0/2411997256 learned_addr learned my addr 172.21.15.107:0/2411997256 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.067+0000 7f4b32ffd640 1 --2- 172.21.15.107:0/2411997256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13d70 0x55adceb1bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.067+0000 7f4b337fe640 1 --2- 172.21.15.107:0/2411997256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 0x55adceb2a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.068+0000 7f4b33fff640 1 -- 172.21.15.107:0/2411997256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 msgr2=0x55adceb2a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:55.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.068+0000 7f4b33fff640 1 --2- 172.21.15.107:0/2411997256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 0x55adceb2a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.068+0000 7f4b33fff640 1 -- 172.21.15.107:0/2411997256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13d70 msgr2=0x55adceb1bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:55.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.068+0000 7f4b33fff640 1 --2- 172.21.15.107:0/2411997256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13d70 0x55adceb1bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.068+0000 7f4b33fff640 1 -- 172.21.15.107:0/2411997256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adceb0df00 con 0x55adceb1e630 2024-05-09T08:35:55.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.068+0000 7f4b32ffd640 1 --2- 172.21.15.107:0/2411997256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13d70 0x55adceb1bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:55.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.068+0000 7f4b337fe640 1 --2- 172.21.15.107:0/2411997256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 0x55adceb2a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:35:55.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.068+0000 7f4b33fff640 1 --2- 172.21.15.107:0/2411997256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb1e630 0x55adceb35d00 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009cd0 tx=0x7f4b2c049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a01207a982b9ee server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:55.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.068+0000 7f4b197fa640 1 -- 172.21.15.107:0/2411997256 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c051070 con 0x55adceb1e630 2024-05-09T08:35:55.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.068+0000 7f4b197fa640 1 -- 172.21.15.107:0/2411997256 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b2c036070 con 0x55adceb1e630 2024-05-09T08:35:55.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.068+0000 7f4b197fa640 1 -- 172.21.15.107:0/2411997256 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c030d30 con 0x55adceb1e630 2024-05-09T08:35:55.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.069+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/2411997256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb1e630 msgr2=0x55adceb35d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:55.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.069+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/2411997256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb1e630 0x55adceb35d00 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009cd0 tx=0x7f4b2c049000 comp rx=0 tx=0).stop 2024-05-09T08:35:55.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.069+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/2411997256 shutdown_connections 2024-05-09T08:35:55.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.069+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/2411997256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb1e630 0x55adceb35d00 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.069+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/2411997256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13d70 0x55adceb1bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.069+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/2411997256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 0x55adceb2a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.069+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/2411997256 >> 172.21.15.107:0/2411997256 conn(0x55adceb0f960 msgr2=0x55adceb1b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:55.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.069+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/2411997256 shutdown_connections 2024-05-09T08:35:55.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.069+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/2411997256 wait complete. 2024-05-09T08:35:55.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.069+0000 7f4b3cd7e740 1 -- start start 2024-05-09T08:35:55.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.069+0000 7f4b3cd7e740 1 Processor -- start 2024-05-09T08:35:55.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b3cd7e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 0x55adceb26820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b3cd7e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13d70 0x55adceb23600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b3cd7e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb1e630 0x55adceb23b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b3cd7e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4b341aaf30 con 0x55adceb1e630 2024-05-09T08:35:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b3cd7e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4b341ac3a0 con 0x55adceb13d70 2024-05-09T08:35:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b3cd7e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4b341ab600 con 0x55adceb13450 2024-05-09T08:35:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b3cd7e740 10 client.0 _ll_register_callbacks cb 0x7f4b24005750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:35:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b33fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb1e630 0x55adceb23b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b33fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb1e630 0x55adceb23b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51282/0 (socket says 172.21.15.107:51282) 2024-05-09T08:35:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b33fff640 1 -- 172.21.15.107:0/948667003 learned_addr learned my addr 172.21.15.107:0/948667003 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:35:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b337fe640 1 --2- 172.21.15.107:0/948667003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 0x55adceb26820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b32ffd640 1 --2- 172.21.15.107:0/948667003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13d70 0x55adceb23600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b33fff640 1 -- 172.21.15.107:0/948667003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 msgr2=0x55adceb26820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b33fff640 1 --2- 172.21.15.107:0/948667003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 0x55adceb26820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b33fff640 1 -- 172.21.15.107:0/948667003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13d70 msgr2=0x55adceb23600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:35:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b33fff640 1 --2- 172.21.15.107:0/948667003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13d70 0x55adceb23600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b33fff640 1 -- 172.21.15.107:0/948667003 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adceb24080 con 0x55adceb1e630 2024-05-09T08:35:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b32ffd640 1 --2- 172.21.15.107:0/948667003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13d70 0x55adceb23600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:35:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b33fff640 1 --2- 172.21.15.107:0/948667003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb1e630 0x55adceb23b40 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c04f040 tx=0x7f4b2c0099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b81a0bd08b9edf8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:35:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:35:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b12ffd640 1 -- 172.21.15.107:0/948667003 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c051070 con 0x55adceb1e630 2024-05-09T08:35:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b12ffd640 1 -- 172.21.15.107:0/948667003 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4b2c004b40 con 0x55adceb1e630 2024-05-09T08:35:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.070+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/948667003 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b2c036070 con 0x55adceb1e630 2024-05-09T08:35:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.071+0000 7f4b12ffd640 1 -- 172.21.15.107:0/948667003 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4b2c004d80 con 0x55adceb1e630 2024-05-09T08:35:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.071+0000 7f4b3cd7e740 10 client.16413 fetch_fsmap learned FSMap version 26 2024-05-09T08:35:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.071+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/948667003 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55adceb952e0 con 0x55adceb1e630 2024-05-09T08:35:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.071+0000 7f4b12ffd640 1 -- 172.21.15.107:0/948667003 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b2c036070 con 0x55adceb1e630 2024-05-09T08:35:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.071+0000 7f4b12ffd640 1 -- 172.21.15.107:0/948667003 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f4b2c05d030 con 0x55adceb1e630 2024-05-09T08:35:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.071+0000 7f4b3cd7e740 10 client.16413 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:35:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.071+0000 7f4b3cd7e740 10 client.16413 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:35:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.071+0000 7f4b3cd7e740 20 client.16413 populate_metadata read hostname 'smithi107' 2024-05-09T08:35:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.071+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/948667003 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55adceb95ac0 0x55adceb97ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:35:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.071+0000 7f4b3cd7e740 4 client.16413 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:35:55.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.071+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/948667003 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55adcea7fdc0 con 0x55adceb95ac0 2024-05-09T08:35:55.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.071+0000 7f4b337fe640 1 --2- 172.21.15.107:0/948667003 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55adceb95ac0 0x55adceb97ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:35:55.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.072+0000 7f4b12ffd640 10 client.16413 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:55.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.072+0000 7f4b337fe640 1 --2- 172.21.15.107:0/948667003 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55adceb95ac0 0x55adceb97ee0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0667f5950cf130a server_cookie=a76d826cef683370 in_seq=0 out_seq=0 2024-05-09T08:35:55.090 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.090+0000 7f4b12ffd640 1 -- 172.21.15.107:0/948667003 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55adcea7fdc0 con 0x55adceb95ac0 2024-05-09T08:35:55.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.090+0000 7f4b12ffd640 10 client.16413 handle_command_reply: tid=0 2024-05-09T08:35:55.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.090+0000 7f4b3cd7e740 1 client.16413 shutdown 2024-05-09T08:35:55.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.090+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/948667003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb1e630 msgr2=0x55adceb23b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:55.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.090+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/948667003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb1e630 0x55adceb23b40 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c04f040 tx=0x7f4b2c0099f0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.090+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/948667003 shutdown_connections 2024-05-09T08:35:55.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.090+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/948667003 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55adceb95ac0 0x55adceb97ee0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.090+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/948667003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adceb1e630 0x55adceb23b40 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.090+0000 7f4b12ffd640 0 client.16413 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:35:55.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.090+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/948667003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55adceb13d70 0x55adceb23600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.090+0000 7f4b3cd7e740 1 --2- 172.21.15.107:0/948667003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55adceb13450 0x55adceb26820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.090+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/948667003 >> 172.21.15.107:0/948667003 conn(0x55adceb0f960 msgr2=0x55adceb1bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:55.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.090+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/948667003 shutdown_connections 2024-05-09T08:35:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.090+0000 7f4b3cd7e740 1 -- 172.21.15.107:0/948667003 wait complete. 2024-05-09T08:35:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.090+0000 7f4b3cd7e740 20 client.16413 trim_cache size 0 max 16384 2024-05-09T08:35:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:35:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:35:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:35:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:35:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.091+0000 7f4b39f90640 1 -- 172.21.15.107:0/1843955319 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4b1c0a2df0 msgr2=0x7f4b1c0a52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.091+0000 7f4b39f90640 1 --2- 172.21.15.107:0/1843955319 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4b1c0a2df0 0x7f4b1c0a52b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c006010 tx=0x7f4b2c031850 comp rx=0 tx=0).stop 2024-05-09T08:35:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.091+0000 7f4b39f90640 1 -- 172.21.15.107:0/1843955319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b340ffd30 msgr2=0x7f4b34198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:35:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.091+0000 7f4b39f90640 1 --2- 172.21.15.107:0/1843955319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b340ffd30 0x7f4b34198010 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4b2400d8d0 tx=0x7f4b2400dd90 comp rx=0 tx=0).stop 2024-05-09T08:35:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.091+0000 7f4b39f90640 1 -- 172.21.15.107:0/1843955319 shutdown_connections 2024-05-09T08:35:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.091+0000 7f4b39f90640 1 --2- 172.21.15.107:0/1843955319 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4b1c0a2df0 0x7f4b1c0a52b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.091+0000 7f4b39f90640 1 --2- 172.21.15.107:0/1843955319 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b34106810 0x7f4b3419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.091+0000 7f4b39f90640 1 --2- 172.21.15.107:0/1843955319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b341006f0 0x7f4b34198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.091+0000 7f4b39f90640 1 --2- 172.21.15.107:0/1843955319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b340ffd30 0x7f4b34198010 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:35:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.091+0000 7f4b39f90640 1 -- 172.21.15.107:0/1843955319 >> 172.21.15.107:0/1843955319 conn(0x7f4b34076cc0 msgr2=0x7f4b34104870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:35:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.091+0000 7f4b39f90640 1 -- 172.21.15.107:0/1843955319 shutdown_connections 2024-05-09T08:35:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:35:55.091+0000 7f4b39f90640 1 -- 172.21.15.107:0/1843955319 wait complete. 2024-05-09T08:35:56.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:55 smithi107 ceph-mon[30122]: pgmap v80: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:35:56.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:55 smithi157 ceph-mon[33614]: pgmap v80: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:35:56.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:55 smithi110 ceph-mon[33622]: pgmap v80: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:35:58.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:57 smithi107 ceph-mon[30122]: pgmap v81: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 37 MiB/s wr, 31 op/s 2024-05-09T08:35:58.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:57 smithi157 ceph-mon[33614]: pgmap v81: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 37 MiB/s wr, 31 op/s 2024-05-09T08:35:58.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:57 smithi110 ceph-mon[33622]: pgmap v81: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 37 MiB/s wr, 31 op/s 2024-05-09T08:36:00.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:35:59 smithi107 ceph-mon[30122]: pgmap v82: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:36:00.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:35:59 smithi157 ceph-mon[33614]: pgmap v82: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:36:00.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:35:59 smithi110 ceph-mon[33622]: pgmap v82: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:36:02.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:01 smithi107 ceph-mon[30122]: pgmap v83: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:36:02.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:01 smithi157 ceph-mon[33614]: pgmap v83: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:36:02.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:01 smithi110 ceph-mon[33622]: pgmap v83: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:36:04.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:03 smithi107 ceph-mon[30122]: pgmap v84: 129 pgs: 129 active+clean; 1.8 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 MiB/s wr, 42 op/s 2024-05-09T08:36:04.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:03 smithi157 ceph-mon[33614]: pgmap v84: 129 pgs: 129 active+clean; 1.8 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 MiB/s wr, 42 op/s 2024-05-09T08:36:04.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:03 smithi110 ceph-mon[33622]: pgmap v84: 129 pgs: 129 active+clean; 1.8 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 MiB/s wr, 42 op/s 2024-05-09T08:36:06.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:05 smithi107 ceph-mon[30122]: pgmap v85: 129 pgs: 129 active+clean; 1.8 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 34 op/s 2024-05-09T08:36:06.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:05 smithi157 ceph-mon[33614]: pgmap v85: 129 pgs: 129 active+clean; 1.8 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 34 op/s 2024-05-09T08:36:06.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:05 smithi110 ceph-mon[33622]: pgmap v85: 129 pgs: 129 active+clean; 1.8 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 34 op/s 2024-05-09T08:36:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.190+0000 7f2e31674640 1 -- 172.21.15.107:0/2300582626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e2c100f20 msgr2=0x7f2e2c10d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.190+0000 7f2e31674640 1 --2- 172.21.15.107:0/2300582626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e2c100f20 0x7f2e2c10d3e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009ca0 tx=0x7f2e2402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.190+0000 7f2e31674640 1 -- 172.21.15.107:0/2300582626 shutdown_connections 2024-05-09T08:36:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.190+0000 7f2e31674640 1 --2- 172.21.15.107:0/2300582626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e2c100f20 0x7f2e2c10d3e0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.190+0000 7f2e31674640 1 --2- 172.21.15.107:0/2300582626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e2c100560 0x7f2e2c1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.190+0000 7f2e31674640 1 --2- 172.21.15.107:0/2300582626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e2c106620 0x7f2e2c106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.190+0000 7f2e31674640 1 -- 172.21.15.107:0/2300582626 >> 172.21.15.107:0/2300582626 conn(0x7f2e2c0fc050 msgr2=0x7f2e2c0fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.190+0000 7f2e31674640 1 -- 172.21.15.107:0/2300582626 shutdown_connections 2024-05-09T08:36:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.191+0000 7f2e31674640 1 -- 172.21.15.107:0/2300582626 wait complete. 2024-05-09T08:36:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.191+0000 7f2e31674640 1 Processor -- start 2024-05-09T08:36:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.191+0000 7f2e31674640 1 -- start start 2024-05-09T08:36:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e31674640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e2c100560 0x7f2e2c19c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e31674640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e2c100f20 0x7f2e2c19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e31674640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e2c106620 0x7f2e2c196ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e31674640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c1107b0 con 0x7f2e2c100560 2024-05-09T08:36:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e31674640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c110630 con 0x7f2e2c106620 2024-05-09T08:36:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e31674640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c110930 con 0x7f2e2c100f20 2024-05-09T08:36:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e2bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e2c100560 0x7f2e2c19c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e2bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e2c100560 0x7f2e2c19c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57500/0 (socket says 172.21.15.107:57500) 2024-05-09T08:36:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e2bfff640 1 -- 172.21.15.107:0/2336912459 learned_addr learned my addr 172.21.15.107:0/2336912459 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e30e73640 1 --2- 172.21.15.107:0/2336912459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e2c106620 0x7f2e2c196ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e2bfff640 1 -- 172.21.15.107:0/2336912459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e2c100f20 msgr2=0x7f2e2c19ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e2bfff640 1 --2- 172.21.15.107:0/2336912459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e2c100f20 0x7f2e2c19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e2bfff640 1 -- 172.21.15.107:0/2336912459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e2c106620 msgr2=0x7f2e2c196ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e2bfff640 1 --2- 172.21.15.107:0/2336912459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e2c106620 0x7f2e2c196ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e2bfff640 1 -- 172.21.15.107:0/2336912459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e2c197250 con 0x7f2e2c100560 2024-05-09T08:36:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e2bfff640 1 --2- 172.21.15.107:0/2336912459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e2c100560 0x7f2e2c19c920 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c00e9f0 tx=0x7f2e1c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.192+0000 7f2e297fa640 1 -- 172.21.15.107:0/2336912459 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c00cd80 con 0x7f2e2c100560 2024-05-09T08:36:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.193+0000 7f2e297fa640 1 -- 172.21.15.107:0/2336912459 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e1c0054d0 con 0x7f2e2c100560 2024-05-09T08:36:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.193+0000 7f2e297fa640 1 -- 172.21.15.107:0/2336912459 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c010490 con 0x7f2e2c100560 2024-05-09T08:36:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.193+0000 7f2e31674640 1 -- 172.21.15.107:0/2336912459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e2c197540 con 0x7f2e2c100560 2024-05-09T08:36:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.193+0000 7f2e31674640 1 -- 172.21.15.107:0/2336912459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e2c1a37e0 con 0x7f2e2c100560 2024-05-09T08:36:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.196+0000 7f2e297fa640 1 -- 172.21.15.107:0/2336912459 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f2e1c005020 con 0x7f2e2c100560 2024-05-09T08:36:08.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.197+0000 7f2e355e2740 1 Processor -- start 2024-05-09T08:36:08.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.197+0000 7f2e355e2740 1 -- start start 2024-05-09T08:36:08.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.197+0000 7f2e297fa640 1 --2- 172.21.15.107:0/2336912459 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2e140a4f00 0x7f2e140a73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.197+0000 7f2e355e2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 0x55d13c6689b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.197+0000 7f2e355e2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c668ef0 0x55d13c673270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.197+0000 7f2e30e73640 1 --2- 172.21.15.107:0/2336912459 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2e140a4f00 0x7f2e140a73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.197+0000 7f2e355e2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c675ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.197+0000 7f2e355e2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d13c5cd340 con 0x55d13c6737b0 2024-05-09T08:36:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.197+0000 7f2e355e2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d13c575290 con 0x55d13c6685d0 2024-05-09T08:36:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.197+0000 7f2e355e2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d13c5c9300 con 0x55d13c668ef0 2024-05-09T08:36:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.197+0000 7f2e297fa640 1 -- 172.21.15.107:0/2336912459 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f2e1c0cfc50 con 0x7f2e2c100560 2024-05-09T08:36:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.197+0000 7f2e30e73640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c675ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.197+0000 7f2e30e73640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c675ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57510/0 (socket says 172.21.15.107:57510) 2024-05-09T08:36:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.197+0000 7f2e30e73640 1 -- 172.21.15.107:0/2314962041 learned_addr learned my addr 172.21.15.107:0/2314962041 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.198+0000 7f2e2bfff640 1 --2- 172.21.15.107:0/2314962041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 0x55d13c6689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.198+0000 7f2e2b7fe640 1 --2- 172.21.15.107:0/2314962041 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c668ef0 0x55d13c673270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.198+0000 7f2e2bfff640 1 -- 172.21.15.107:0/2314962041 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c668ef0 msgr2=0x55d13c673270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.198+0000 7f2e2bfff640 1 --2- 172.21.15.107:0/2314962041 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c668ef0 0x55d13c673270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.198+0000 7f2e2bfff640 1 -- 172.21.15.107:0/2314962041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 msgr2=0x55d13c675ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.198+0000 7f2e2bfff640 1 --2- 172.21.15.107:0/2314962041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c675ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.199 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.198+0000 7f2e2bfff640 1 -- 172.21.15.107:0/2314962041 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d13c5998e0 con 0x55d13c6685d0 2024-05-09T08:36:08.199 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.198+0000 7f2e30e73640 1 --2- 172.21.15.107:0/2336912459 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2e140a4f00 0x7f2e140a73c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009430 tx=0x7f2e24009800 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:08.199 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.198+0000 7f2e2bfff640 1 --2- 172.21.15.107:0/2314962041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 0x55d13c6689b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c0cd040 tx=0x7f2e1c09b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56a8a5d0599aff42 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:08.199 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.198+0000 7f2e127fc640 1 -- 172.21.15.107:0/2314962041 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c09c510 con 0x55d13c6685d0 2024-05-09T08:36:08.199 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.198+0000 7f2e127fc640 1 -- 172.21.15.107:0/2314962041 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e1c09be10 con 0x55d13c6685d0 2024-05-09T08:36:08.199 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.198+0000 7f2e127fc640 1 -- 172.21.15.107:0/2314962041 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c0cd2c0 con 0x55d13c6685d0 2024-05-09T08:36:08.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.199+0000 7f2e355e2740 1 -- 172.21.15.107:0/2314962041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 msgr2=0x55d13c6689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.199+0000 7f2e355e2740 1 --2- 172.21.15.107:0/2314962041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 0x55d13c6689b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c0cd040 tx=0x7f2e1c09b770 comp rx=0 tx=0).stop 2024-05-09T08:36:08.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.199+0000 7f2e355e2740 1 -- 172.21.15.107:0/2314962041 shutdown_connections 2024-05-09T08:36:08.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.199+0000 7f2e355e2740 1 --2- 172.21.15.107:0/2314962041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c675ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.199+0000 7f2e355e2740 1 --2- 172.21.15.107:0/2314962041 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c668ef0 0x55d13c673270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.199+0000 7f2e355e2740 1 --2- 172.21.15.107:0/2314962041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 0x55d13c6689b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.199+0000 7f2e355e2740 1 -- 172.21.15.107:0/2314962041 >> 172.21.15.107:0/2314962041 conn(0x55d13c664ae0 msgr2=0x55d13c664ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:08.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.199+0000 7f2e355e2740 1 -- 172.21.15.107:0/2314962041 shutdown_connections 2024-05-09T08:36:08.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.199+0000 7f2e355e2740 1 -- 172.21.15.107:0/2314962041 wait complete. 2024-05-09T08:36:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.200+0000 7f2e355e2740 1 -- start start 2024-05-09T08:36:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.200+0000 7f2e355e2740 1 Processor -- start 2024-05-09T08:36:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.200+0000 7f2e355e2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 0x55d13c6706e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.200+0000 7f2e355e2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c668ef0 0x55d13c670c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.200+0000 7f2e355e2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c671160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.200+0000 7f2e355e2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d13c6763e0 con 0x55d13c6737b0 2024-05-09T08:36:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.200+0000 7f2e355e2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d13c588790 con 0x55d13c6685d0 2024-05-09T08:36:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.200+0000 7f2e355e2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d13c676560 con 0x55d13c668ef0 2024-05-09T08:36:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.200+0000 7f2e355e2740 10 client.0 _ll_register_callbacks cb 0x55d13c5c3a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.200+0000 7f2e30e73640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c671160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.200+0000 7f2e2b7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c668ef0 0x55d13c670c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.200+0000 7f2e30e73640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c671160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57514/0 (socket says 172.21.15.107:57514) 2024-05-09T08:36:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.200+0000 7f2e30e73640 1 -- 172.21.15.107:0/3371204594 learned_addr learned my addr 172.21.15.107:0/3371204594 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.200+0000 7f2e2bfff640 1 --2- 172.21.15.107:0/3371204594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 0x55d13c6706e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.201+0000 7f2e30e73640 1 -- 172.21.15.107:0/3371204594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c668ef0 msgr2=0x55d13c670c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.201+0000 7f2e30e73640 1 --2- 172.21.15.107:0/3371204594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c668ef0 0x55d13c670c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.201+0000 7f2e30e73640 1 -- 172.21.15.107:0/3371204594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 msgr2=0x55d13c6706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.201+0000 7f2e30e73640 1 --2- 172.21.15.107:0/3371204594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 0x55d13c6706e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.201+0000 7f2e30e73640 1 -- 172.21.15.107:0/3371204594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d13c2e1210 con 0x55d13c6737b0 2024-05-09T08:36:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.201+0000 7f2e2b7fe640 1 --2- 172.21.15.107:0/3371204594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c668ef0 0x55d13c670c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.201+0000 7f2e30e73640 1 --2- 172.21.15.107:0/3371204594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c671160 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009ca0 tx=0x7f2e2402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=296a95dc837fd505 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.201+0000 7f2e0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.201+0000 7f2e0b7fe640 1 -- 172.21.15.107:0/3371204594 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e2404f030 con 0x55d13c6737b0 2024-05-09T08:36:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.201+0000 7f2e0b7fe640 1 -- 172.21.15.107:0/3371204594 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f2e24046040 con 0x55d13c6737b0 2024-05-09T08:36:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.201+0000 7f2e355e2740 1 -- 172.21.15.107:0/3371204594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e1c09be10 con 0x55d13c6737b0 2024-05-09T08:36:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.201+0000 7f2e0b7fe640 1 -- 172.21.15.107:0/3371204594 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2e2405c050 con 0x55d13c6737b0 2024-05-09T08:36:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.201+0000 7f2e355e2740 10 client.16425 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.201+0000 7f2e355e2740 1 -- 172.21.15.107:0/3371204594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d13c3ad620 con 0x55d13c6737b0 2024-05-09T08:36:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.201+0000 7f2e0b7fe640 1 -- 172.21.15.107:0/3371204594 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e1c09be10 con 0x55d13c6737b0 2024-05-09T08:36:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.202+0000 7f2e0b7fe640 1 -- 172.21.15.107:0/3371204594 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f2e2404a070 con 0x55d13c6737b0 2024-05-09T08:36:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.202+0000 7f2e355e2740 10 client.16425 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.202+0000 7f2e355e2740 10 client.16425 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.202+0000 7f2e355e2740 20 client.16425 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.202+0000 7f2e355e2740 1 --2- 172.21.15.107:0/3371204594 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d13c6835c0 0x55d13c6859e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.202+0000 7f2e355e2740 4 client.16425 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.202+0000 7f2e355e2740 1 -- 172.21.15.107:0/3371204594 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d13c5daa70 con 0x55d13c6835c0 2024-05-09T08:36:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.202+0000 7f2e2bfff640 1 --2- 172.21.15.107:0/3371204594 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d13c6835c0 0x55d13c6859e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.202+0000 7f2e0b7fe640 10 client.16425 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.202+0000 7f2e2bfff640 1 --2- 172.21.15.107:0/3371204594 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d13c6835c0 0x55d13c6859e0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cdf55bc01705863 server_cookie=9cc0e0418284c3c4 in_seq=0 out_seq=0 2024-05-09T08:36:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.203+0000 7f2e0b7fe640 1 -- 172.21.15.107:0/3371204594 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d13c5daa70 con 0x55d13c6835c0 2024-05-09T08:36:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.203+0000 7f2e0b7fe640 10 client.16425 handle_command_reply: tid=0 2024-05-09T08:36:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.203+0000 7f2e355e2740 1 client.16425 shutdown 2024-05-09T08:36:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.203+0000 7f2e355e2740 1 -- 172.21.15.107:0/3371204594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 msgr2=0x55d13c671160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.203+0000 7f2e355e2740 1 --2- 172.21.15.107:0/3371204594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c671160 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009ca0 tx=0x7f2e2402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.203+0000 7f2e355e2740 1 -- 172.21.15.107:0/3371204594 shutdown_connections 2024-05-09T08:36:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.203+0000 7f2e355e2740 1 --2- 172.21.15.107:0/3371204594 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d13c6835c0 0x55d13c6859e0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.203+0000 7f2e355e2740 1 --2- 172.21.15.107:0/3371204594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c671160 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.203+0000 7f2e0b7fe640 0 client.16425 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.203+0000 7f2e355e2740 1 --2- 172.21.15.107:0/3371204594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c668ef0 0x55d13c670c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.203+0000 7f2e355e2740 1 --2- 172.21.15.107:0/3371204594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 0x55d13c6706e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.203+0000 7f2e355e2740 1 -- 172.21.15.107:0/3371204594 >> 172.21.15.107:0/3371204594 conn(0x55d13c664ae0 msgr2=0x55d13c66a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.203+0000 7f2e355e2740 1 -- 172.21.15.107:0/3371204594 shutdown_connections 2024-05-09T08:36:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.203+0000 7f2e355e2740 1 -- 172.21.15.107:0/3371204594 wait complete. 2024-05-09T08:36:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.203+0000 7f2e355e2740 20 client.16425 trim_cache size 0 max 16384 2024-05-09T08:36:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.218+0000 7f2e355e2740 1 Processor -- start 2024-05-09T08:36:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.218+0000 7f2e355e2740 1 -- start start 2024-05-09T08:36:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.218+0000 7f2e355e2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c6685d0 0x55d13c672e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.218+0000 7f2e355e2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c668ef0 0x55d13c67f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.218+0000 7f2e355e2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c66f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.218+0000 7f2e355e2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c1af000 con 0x55d13c6737b0 2024-05-09T08:36:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.218+0000 7f2e355e2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c1ac8d0 con 0x55d13c668ef0 2024-05-09T08:36:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.218+0000 7f2e355e2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c1aa720 con 0x55d13c6685d0 2024-05-09T08:36:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.218+0000 7f2e30e73640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c66f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.218+0000 7f2e30e73640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c66f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57528/0 (socket says 172.21.15.107:57528) 2024-05-09T08:36:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.218+0000 7f2e30e73640 1 -- 172.21.15.107:0/1658539742 learned_addr learned my addr 172.21.15.107:0/1658539742 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.218+0000 7f2e2bfff640 1 --2- 172.21.15.107:0/1658539742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c6685d0 0x55d13c672e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.218+0000 7f2e2b7fe640 1 --2- 172.21.15.107:0/1658539742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c668ef0 0x55d13c67f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.219+0000 7f2e30e73640 1 -- 172.21.15.107:0/1658539742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c6685d0 msgr2=0x55d13c672e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.219+0000 7f2e30e73640 1 --2- 172.21.15.107:0/1658539742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c6685d0 0x55d13c672e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.219+0000 7f2e30e73640 1 -- 172.21.15.107:0/1658539742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c668ef0 msgr2=0x55d13c67f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.219+0000 7f2e30e73640 1 --2- 172.21.15.107:0/1658539742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c668ef0 0x55d13c67f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.219+0000 7f2e30e73640 1 -- 172.21.15.107:0/1658539742 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d13c668d20 con 0x55d13c6737b0 2024-05-09T08:36:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.219+0000 7f2e2bfff640 1 --2- 172.21.15.107:0/1658539742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c6685d0 0x55d13c672e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.219+0000 7f2e30e73640 1 --2- 172.21.15.107:0/1658539742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c66f840 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009ba0 tx=0x7f2e24031ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65263c9ace124428 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.219+0000 7f2e11ffb640 1 -- 172.21.15.107:0/1658539742 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e24031e60 con 0x55d13c6737b0 2024-05-09T08:36:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.219+0000 7f2e11ffb640 1 -- 172.21.15.107:0/1658539742 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e24030dd0 con 0x55d13c6737b0 2024-05-09T08:36:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.219+0000 7f2e11ffb640 1 -- 172.21.15.107:0/1658539742 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e2404f070 con 0x55d13c6737b0 2024-05-09T08:36:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.220+0000 7f2e355e2740 1 -- 172.21.15.107:0/1658539742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 msgr2=0x55d13c66f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.220+0000 7f2e355e2740 1 --2- 172.21.15.107:0/1658539742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c66f840 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009ba0 tx=0x7f2e24031ab0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.220+0000 7f2e355e2740 1 -- 172.21.15.107:0/1658539742 shutdown_connections 2024-05-09T08:36:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.220+0000 7f2e355e2740 1 --2- 172.21.15.107:0/1658539742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c66f840 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.220+0000 7f2e355e2740 1 --2- 172.21.15.107:0/1658539742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c668ef0 0x55d13c67f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.220+0000 7f2e355e2740 1 --2- 172.21.15.107:0/1658539742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c6685d0 0x55d13c672e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.220+0000 7f2e355e2740 1 -- 172.21.15.107:0/1658539742 >> 172.21.15.107:0/1658539742 conn(0x55d13c664ae0 msgr2=0x55d13c663410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.220+0000 7f2e355e2740 1 -- 172.21.15.107:0/1658539742 shutdown_connections 2024-05-09T08:36:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.220+0000 7f2e355e2740 1 -- 172.21.15.107:0/1658539742 wait complete. 2024-05-09T08:36:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.220+0000 7f2e355e2740 1 -- start start 2024-05-09T08:36:08.224 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.220+0000 7f2e355e2740 1 Processor -- start 2024-05-09T08:36:08.224 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e355e2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 0x55d13c676d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.224 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e355e2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c668ef0 0x55d13c6772b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.224 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e355e2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c67b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e355e2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c1aad90 con 0x55d13c6737b0 2024-05-09T08:36:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e355e2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c1ac200 con 0x55d13c6685d0 2024-05-09T08:36:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e355e2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c1ab460 con 0x55d13c668ef0 2024-05-09T08:36:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e355e2740 10 client.0 _ll_register_callbacks cb 0x7f2e24005d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e30e73640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c67b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e30e73640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c67b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57530/0 (socket says 172.21.15.107:57530) 2024-05-09T08:36:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e30e73640 1 -- 172.21.15.107:0/2897279007 learned_addr learned my addr 172.21.15.107:0/2897279007 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e2b7fe640 1 --2- 172.21.15.107:0/2897279007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c668ef0 0x55d13c6772b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e2bfff640 1 --2- 172.21.15.107:0/2897279007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 0x55d13c676d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e30e73640 1 -- 172.21.15.107:0/2897279007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c668ef0 msgr2=0x55d13c6772b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e30e73640 1 --2- 172.21.15.107:0/2897279007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c668ef0 0x55d13c6772b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e30e73640 1 -- 172.21.15.107:0/2897279007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 msgr2=0x55d13c676d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:08.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e30e73640 1 --2- 172.21.15.107:0/2897279007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 0x55d13c676d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e30e73640 1 -- 172.21.15.107:0/2897279007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d13c67b810 con 0x55d13c6737b0 2024-05-09T08:36:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e2bfff640 1 --2- 172.21.15.107:0/2897279007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 0x55d13c676d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:36:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e30e73640 1 --2- 172.21.15.107:0/2897279007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c67b1d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009400 tx=0x7f2e24004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2d57c65709ba5a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.221+0000 7f2e0b7fe640 1 -- 172.21.15.107:0/2897279007 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e24031e60 con 0x55d13c6737b0 2024-05-09T08:36:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.222+0000 7f2e0b7fe640 1 -- 172.21.15.107:0/2897279007 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f2e24004140 con 0x55d13c6737b0 2024-05-09T08:36:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.222+0000 7f2e355e2740 1 -- 172.21.15.107:0/2897279007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e24030dd0 con 0x55d13c6737b0 2024-05-09T08:36:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.222+0000 7f2e0b7fe640 1 -- 172.21.15.107:0/2897279007 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2e2406b050 con 0x55d13c6737b0 2024-05-09T08:36:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.222+0000 7f2e355e2740 10 client.16431 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.222+0000 7f2e355e2740 1 -- 172.21.15.107:0/2897279007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d13c6e9600 con 0x55d13c6737b0 2024-05-09T08:36:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.222+0000 7f2e0b7fe640 1 -- 172.21.15.107:0/2897279007 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e24030dd0 con 0x55d13c6737b0 2024-05-09T08:36:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.222+0000 7f2e0b7fe640 1 -- 172.21.15.107:0/2897279007 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f2e2402f820 con 0x55d13c6737b0 2024-05-09T08:36:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.222+0000 7f2e355e2740 10 client.16431 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:08.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.222+0000 7f2e355e2740 10 client.16431 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:08.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.222+0000 7f2e355e2740 20 client.16431 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:08.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.222+0000 7f2e355e2740 1 --2- 172.21.15.107:0/2897279007 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d13c6835c0 0x55d13c6ebc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.222+0000 7f2e355e2740 4 client.16431 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:36:08.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.222+0000 7f2e355e2740 1 -- 172.21.15.107:0/2897279007 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d13c5d4dc0 con 0x55d13c6835c0 2024-05-09T08:36:08.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.222+0000 7f2e2bfff640 1 --2- 172.21.15.107:0/2897279007 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d13c6835c0 0x55d13c6ebc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.223+0000 7f2e0b7fe640 10 client.16431 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:08.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.223+0000 7f2e2bfff640 1 --2- 172.21.15.107:0/2897279007 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d13c6835c0 0x55d13c6ebc20 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10d5769e19677697 server_cookie=9df10e1a81e810b6 in_seq=0 out_seq=0 2024-05-09T08:36:08.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.240+0000 7f2e0b7fe640 1 -- 172.21.15.107:0/2897279007 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d13c5d4dc0 con 0x55d13c6835c0 2024-05-09T08:36:08.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.240+0000 7f2e0b7fe640 10 client.16431 handle_command_reply: tid=0 2024-05-09T08:36:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.240+0000 7f2e355e2740 1 client.16431 shutdown 2024-05-09T08:36:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.240+0000 7f2e355e2740 1 -- 172.21.15.107:0/2897279007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 msgr2=0x55d13c67b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.240+0000 7f2e355e2740 1 --2- 172.21.15.107:0/2897279007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c67b1d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009400 tx=0x7f2e24004ab0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.240+0000 7f2e355e2740 1 -- 172.21.15.107:0/2897279007 shutdown_connections 2024-05-09T08:36:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.240+0000 7f2e355e2740 1 --2- 172.21.15.107:0/2897279007 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d13c6835c0 0x55d13c6ebc20 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.240+0000 7f2e355e2740 1 --2- 172.21.15.107:0/2897279007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d13c6737b0 0x55d13c67b1d0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.240+0000 7f2e355e2740 1 --2- 172.21.15.107:0/2897279007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d13c668ef0 0x55d13c6772b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.240+0000 7f2e355e2740 1 --2- 172.21.15.107:0/2897279007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d13c6685d0 0x55d13c676d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.240+0000 7f2e355e2740 1 -- 172.21.15.107:0/2897279007 >> 172.21.15.107:0/2897279007 conn(0x55d13c664ae0 msgr2=0x55d13c663ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.240+0000 7f2e0b7fe640 0 client.16431 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.240+0000 7f2e355e2740 1 -- 172.21.15.107:0/2897279007 shutdown_connections 2024-05-09T08:36:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.240+0000 7f2e355e2740 1 -- 172.21.15.107:0/2897279007 wait complete. 2024-05-09T08:36:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.240+0000 7f2e355e2740 20 client.16431 trim_cache size 0 max 16384 2024-05-09T08:36:08.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:36:08.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:36:08.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:36:08.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:36:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.241+0000 7f2e31674640 1 -- 172.21.15.107:0/2336912459 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2e140a4f00 msgr2=0x7f2e140a73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.241+0000 7f2e31674640 1 --2- 172.21.15.107:0/2336912459 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2e140a4f00 0x7f2e140a73c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009430 tx=0x7f2e24009800 comp rx=0 tx=0).stop 2024-05-09T08:36:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.241+0000 7f2e31674640 1 -- 172.21.15.107:0/2336912459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e2c100560 msgr2=0x7f2e2c19c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.241+0000 7f2e31674640 1 --2- 172.21.15.107:0/2336912459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e2c100560 0x7f2e2c19c920 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c00e9f0 tx=0x7f2e1c00edf0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.241+0000 7f2e31674640 1 -- 172.21.15.107:0/2336912459 shutdown_connections 2024-05-09T08:36:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.241+0000 7f2e31674640 1 --2- 172.21.15.107:0/2336912459 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2e140a4f00 0x7f2e140a73c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.241+0000 7f2e31674640 1 --2- 172.21.15.107:0/2336912459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e2c106620 0x7f2e2c196ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.241+0000 7f2e31674640 1 --2- 172.21.15.107:0/2336912459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e2c100f20 0x7f2e2c19ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.241+0000 7f2e31674640 1 --2- 172.21.15.107:0/2336912459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e2c100560 0x7f2e2c19c920 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.241+0000 7f2e31674640 1 -- 172.21.15.107:0/2336912459 >> 172.21.15.107:0/2336912459 conn(0x7f2e2c0fc050 msgr2=0x7f2e2c0fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:08.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.241+0000 7f2e31674640 1 -- 172.21.15.107:0/2336912459 shutdown_connections 2024-05-09T08:36:08.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:08.241+0000 7f2e31674640 1 -- 172.21.15.107:0/2336912459 wait complete. 2024-05-09T08:36:08.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:07 smithi107 ceph-mon[30122]: pgmap v86: 129 pgs: 129 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 38 MiB/s wr, 42 op/s 2024-05-09T08:36:08.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:07 smithi157 ceph-mon[33614]: pgmap v86: 129 pgs: 129 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 38 MiB/s wr, 42 op/s 2024-05-09T08:36:08.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:07 smithi110 ceph-mon[33622]: pgmap v86: 129 pgs: 129 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 38 MiB/s wr, 42 op/s 2024-05-09T08:36:08.595 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:36:08.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:36:08.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.710+0000 7feef96ee640 1 -- 172.21.15.107:0/851453775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feef4100700 msgr2=0x7feef4100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.710+0000 7feef96ee640 1 --2- 172.21.15.107:0/851453775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feef4100700 0x7feef4100b80 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7feee8009930 tx=0x7feee802a420 comp rx=0 tx=0).stop 2024-05-09T08:36:08.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.710+0000 7feef96ee640 1 -- 172.21.15.107:0/851453775 shutdown_connections 2024-05-09T08:36:08.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.710+0000 7feef96ee640 1 --2- 172.21.15.107:0/851453775 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feef41010c0 0x7feef410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.710+0000 7feef96ee640 1 --2- 172.21.15.107:0/851453775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feef4100700 0x7feef4100b80 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.710+0000 7feef96ee640 1 --2- 172.21.15.107:0/851453775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feef4106710 0x7feef4106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.710+0000 7feef96ee640 1 -- 172.21.15.107:0/851453775 >> 172.21.15.107:0/851453775 conn(0x7feef4077830 msgr2=0x7feef4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.710+0000 7feef96ee640 1 -- 172.21.15.107:0/851453775 shutdown_connections 2024-05-09T08:36:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.710+0000 7feef96ee640 1 -- 172.21.15.107:0/851453775 wait complete. 2024-05-09T08:36:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.711+0000 7feef96ee640 1 Processor -- start 2024-05-09T08:36:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.711+0000 7feef96ee640 1 -- start start 2024-05-09T08:36:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.711+0000 7feef96ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feef4100700 0x7feef4197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.711+0000 7feef96ee640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feef41010c0 0x7feef41984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.711+0000 7feef96ee640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feef4106710 0x7feef419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.711+0000 7feef96ee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feef4110990 con 0x7feef4100700 2024-05-09T08:36:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.711+0000 7feef96ee640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feef4110810 con 0x7feef4106710 2024-05-09T08:36:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.711+0000 7feef96ee640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feef4110b10 con 0x7feef41010c0 2024-05-09T08:36:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.711+0000 7feef2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feef4100700 0x7feef4197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.711+0000 7feef2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feef4100700 0x7feef4197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57558/0 (socket says 172.21.15.107:57558) 2024-05-09T08:36:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.711+0000 7feef2ffd640 1 -- 172.21.15.107:0/365465510 learned_addr learned my addr 172.21.15.107:0/365465510 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.711+0000 7feef27fc640 1 --2- 172.21.15.107:0/365465510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feef41010c0 0x7feef41984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.712+0000 7feef37fe640 1 --2- 172.21.15.107:0/365465510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feef4106710 0x7feef419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.712+0000 7feef2ffd640 1 -- 172.21.15.107:0/365465510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feef41010c0 msgr2=0x7feef41984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.712+0000 7feef2ffd640 1 --2- 172.21.15.107:0/365465510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feef41010c0 0x7feef41984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.712+0000 7feef2ffd640 1 -- 172.21.15.107:0/365465510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feef4106710 msgr2=0x7feef419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.712+0000 7feef2ffd640 1 --2- 172.21.15.107:0/365465510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feef4106710 0x7feef419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.712+0000 7feef2ffd640 1 -- 172.21.15.107:0/365465510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef419cf50 con 0x7feef4100700 2024-05-09T08:36:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.712+0000 7feef2ffd640 1 --2- 172.21.15.107:0/365465510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feef4100700 0x7feef4197f60 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7feee400c920 tx=0x7feee400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.712+0000 7feedbfff640 1 -- 172.21.15.107:0/365465510 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4007c70 con 0x7feef4100700 2024-05-09T08:36:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.712+0000 7feedbfff640 1 -- 172.21.15.107:0/365465510 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feee4007e10 con 0x7feef4100700 2024-05-09T08:36:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.712+0000 7feef96ee640 1 -- 172.21.15.107:0/365465510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef419d240 con 0x7feef4100700 2024-05-09T08:36:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.712+0000 7feedbfff640 1 -- 172.21.15.107:0/365465510 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4018820 con 0x7feef4100700 2024-05-09T08:36:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.712+0000 7feef96ee640 1 -- 172.21.15.107:0/365465510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef41a4ad0 con 0x7feef4100700 2024-05-09T08:36:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.714+0000 7feedbfff640 1 -- 172.21.15.107:0/365465510 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7feee40189c0 con 0x7feef4100700 2024-05-09T08:36:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.715+0000 7feefc3f9740 1 Processor -- start 2024-05-09T08:36:08.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.715+0000 7feefc3f9740 1 -- start start 2024-05-09T08:36:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.715+0000 7feefc3f9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 0x55f43a6db660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.715+0000 7feefc3f9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6dbba0 0x55f43a6e5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.715+0000 7feefc3f9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6e6270 0x55f43a6e8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.715+0000 7feefc3f9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f43a640d00 con 0x55f43a6e6270 2024-05-09T08:36:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.715+0000 7feefc3f9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f43a5e84c0 con 0x55f43a6dbba0 2024-05-09T08:36:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.715+0000 7feefc3f9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f43a63ccc0 con 0x55f43a6db280 2024-05-09T08:36:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.715+0000 7feef27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6e6270 0x55f43a6e8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.715+0000 7feef27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6e6270 0x55f43a6e8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57570/0 (socket says 172.21.15.107:57570) 2024-05-09T08:36:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.715+0000 7feef27fc640 1 -- 172.21.15.107:0/918171731 learned_addr learned my addr 172.21.15.107:0/918171731 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.715+0000 7feedbfff640 1 --2- 172.21.15.107:0/365465510 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feedc0a2d00 0x7feedc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.715+0000 7feef37fe640 1 --2- 172.21.15.107:0/918171731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 0x55f43a6db660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.715+0000 7feef2ffd640 1 --2- 172.21.15.107:0/918171731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6dbba0 0x55f43a6e5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.716+0000 7feef37fe640 1 --2- 172.21.15.107:0/365465510 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feedc0a2d00 0x7feedc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.716+0000 7feef27fc640 1 -- 172.21.15.107:0/918171731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 msgr2=0x55f43a6db660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.716+0000 7feef27fc640 1 --2- 172.21.15.107:0/918171731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 0x55f43a6db660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.716+0000 7feedbfff640 1 -- 172.21.15.107:0/365465510 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7feee40cf870 con 0x7feef4100700 2024-05-09T08:36:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.716+0000 7feef27fc640 1 -- 172.21.15.107:0/918171731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6dbba0 msgr2=0x55f43a6e5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.716+0000 7feef27fc640 1 --2- 172.21.15.107:0/918171731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6dbba0 0x55f43a6e5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.716+0000 7feef27fc640 1 -- 172.21.15.107:0/918171731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f43a41e7d0 con 0x55f43a6e6270 2024-05-09T08:36:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.716+0000 7feef37fe640 1 --2- 172.21.15.107:0/918171731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 0x55f43a6db660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.716+0000 7feef27fc640 1 --2- 172.21.15.107:0/918171731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6e6270 0x55f43a6e8660 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7feee8009930 tx=0x7feee8032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73807ed5240d674a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.716+0000 7feef37fe640 1 --2- 172.21.15.107:0/365465510 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feedc0a2d00 0x7feedc0a51c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feeec012940 tx=0x7feeec004ab0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.716+0000 7feed97fa640 1 -- 172.21.15.107:0/918171731 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8035040 con 0x55f43a6e6270 2024-05-09T08:36:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.716+0000 7feed97fa640 1 -- 172.21.15.107:0/918171731 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feee80316a0 con 0x55f43a6e6270 2024-05-09T08:36:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.716+0000 7feed97fa640 1 -- 172.21.15.107:0/918171731 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee80047a0 con 0x55f43a6e6270 2024-05-09T08:36:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.717+0000 7feefc3f9740 1 -- 172.21.15.107:0/918171731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6e6270 msgr2=0x55f43a6e8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.717+0000 7feefc3f9740 1 --2- 172.21.15.107:0/918171731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6e6270 0x55f43a6e8660 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7feee8009930 tx=0x7feee8032d60 comp rx=0 tx=0).stop 2024-05-09T08:36:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.717+0000 7feefc3f9740 1 -- 172.21.15.107:0/918171731 shutdown_connections 2024-05-09T08:36:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.717+0000 7feefc3f9740 1 --2- 172.21.15.107:0/918171731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6e6270 0x55f43a6e8660 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.717+0000 7feefc3f9740 1 --2- 172.21.15.107:0/918171731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6dbba0 0x55f43a6e5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.717+0000 7feefc3f9740 1 --2- 172.21.15.107:0/918171731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 0x55f43a6db660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.717+0000 7feefc3f9740 1 -- 172.21.15.107:0/918171731 >> 172.21.15.107:0/918171731 conn(0x55f43a6d75a0 msgr2=0x55f43a6d7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.717+0000 7feefc3f9740 1 -- 172.21.15.107:0/918171731 shutdown_connections 2024-05-09T08:36:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.717+0000 7feefc3f9740 1 -- 172.21.15.107:0/918171731 wait complete. 2024-05-09T08:36:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.717+0000 7feefc3f9740 1 -- start start 2024-05-09T08:36:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.717+0000 7feefc3f9740 1 Processor -- start 2024-05-09T08:36:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feefc3f9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 0x55f43a6e3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feefc3f9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6dbba0 0x55f43a6e36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feefc3f9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6e6270 0x55f43a6e3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feefc3f9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f43a6e9000 con 0x55f43a6dbba0 2024-05-09T08:36:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feefc3f9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f43a5fbb90 con 0x55f43a6e6270 2024-05-09T08:36:08.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feefc3f9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f43a6e9180 con 0x55f43a6db280 2024-05-09T08:36:08.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feefc3f9740 10 client.0 _ll_register_callbacks cb 0x55f43a637da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:08.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feef27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6dbba0 0x55f43a6e36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feef27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6dbba0 0x55f43a6e36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57574/0 (socket says 172.21.15.107:57574) 2024-05-09T08:36:08.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feef27fc640 1 -- 172.21.15.107:0/260738312 learned_addr learned my addr 172.21.15.107:0/260738312 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:08.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feef37fe640 1 --2- 172.21.15.107:0/260738312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6e6270 0x55f43a6e3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feef2ffd640 1 --2- 172.21.15.107:0/260738312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 0x55f43a6e3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feef27fc640 1 -- 172.21.15.107:0/260738312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 msgr2=0x55f43a6e3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:08.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feef27fc640 1 --2- 172.21.15.107:0/260738312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 0x55f43a6e3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feef27fc640 1 -- 172.21.15.107:0/260738312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6e6270 msgr2=0x55f43a6e3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:08.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feef27fc640 1 --2- 172.21.15.107:0/260738312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6e6270 0x55f43a6e3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feef27fc640 1 -- 172.21.15.107:0/260738312 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f43a3543f0 con 0x55f43a6dbba0 2024-05-09T08:36:08.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feef2ffd640 1 --2- 172.21.15.107:0/260738312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 0x55f43a6e3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:36:08.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feef27fc640 1 --2- 172.21.15.107:0/260738312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6dbba0 0x55f43a6e36b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7feee8031440 tx=0x7feee8005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbde68bdbde9731 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feed6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feed6ff5640 1 -- 172.21.15.107:0/260738312 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee804e020 con 0x55f43a6dbba0 2024-05-09T08:36:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feed6ff5640 1 -- 172.21.15.107:0/260738312 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7feee8004040 con 0x55f43a6dbba0 2024-05-09T08:36:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.718+0000 7feefc3f9740 1 -- 172.21.15.107:0/260738312 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feee80316a0 con 0x55f43a6dbba0 2024-05-09T08:36:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.719+0000 7feed6ff5640 1 -- 172.21.15.107:0/260738312 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7feee8059050 con 0x55f43a6dbba0 2024-05-09T08:36:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.719+0000 7feefc3f9740 10 client.16443 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.719+0000 7feefc3f9740 1 -- 172.21.15.107:0/260738312 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f43a3140f0 con 0x55f43a6dbba0 2024-05-09T08:36:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.719+0000 7feed6ff5640 1 -- 172.21.15.107:0/260738312 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feee80316a0 con 0x55f43a6dbba0 2024-05-09T08:36:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.719+0000 7feed6ff5640 1 -- 172.21.15.107:0/260738312 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7feee80035f0 con 0x55f43a6dbba0 2024-05-09T08:36:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.719+0000 7feefc3f9740 10 client.16443 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.719+0000 7feefc3f9740 10 client.16443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.719+0000 7feefc3f9740 20 client.16443 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.719+0000 7feefc3f9740 1 --2- 172.21.15.107:0/260738312 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f43a6f63b0 0x55f43a6f87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.719+0000 7feefc3f9740 4 client.16443 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.719+0000 7feefc3f9740 1 -- 172.21.15.107:0/260738312 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f43a64e430 con 0x55f43a6f63b0 2024-05-09T08:36:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.719+0000 7feef2ffd640 1 --2- 172.21.15.107:0/260738312 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f43a6f63b0 0x55f43a6f87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.720+0000 7feef2ffd640 1 --2- 172.21.15.107:0/260738312 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f43a6f63b0 0x55f43a6f87a0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89a818c1cf44852d server_cookie=659a8debf54e7aaf in_seq=0 out_seq=0 2024-05-09T08:36:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.720+0000 7feed6ff5640 10 client.16443 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.720+0000 7feed6ff5640 1 -- 172.21.15.107:0/260738312 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f43a64e430 con 0x55f43a6f63b0 2024-05-09T08:36:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.720+0000 7feed6ff5640 10 client.16443 handle_command_reply: tid=0 2024-05-09T08:36:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.720+0000 7feefc3f9740 1 client.16443 shutdown 2024-05-09T08:36:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.721+0000 7feefc3f9740 1 -- 172.21.15.107:0/260738312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6dbba0 msgr2=0x55f43a6e36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.721+0000 7feefc3f9740 1 --2- 172.21.15.107:0/260738312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6dbba0 0x55f43a6e36b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7feee8031440 tx=0x7feee8005020 comp rx=0 tx=0).stop 2024-05-09T08:36:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.721+0000 7feefc3f9740 1 -- 172.21.15.107:0/260738312 shutdown_connections 2024-05-09T08:36:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.721+0000 7feefc3f9740 1 --2- 172.21.15.107:0/260738312 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f43a6f63b0 0x55f43a6f87a0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.721+0000 7feed6ff5640 0 client.16443 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.721+0000 7feefc3f9740 1 --2- 172.21.15.107:0/260738312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6e6270 0x55f43a6e3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.721+0000 7feefc3f9740 1 --2- 172.21.15.107:0/260738312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6dbba0 0x55f43a6e36b0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.721+0000 7feefc3f9740 1 --2- 172.21.15.107:0/260738312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 0x55f43a6e3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.721+0000 7feefc3f9740 1 -- 172.21.15.107:0/260738312 >> 172.21.15.107:0/260738312 conn(0x55f43a6d75a0 msgr2=0x55f43a6dd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:08.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.721+0000 7feefc3f9740 1 -- 172.21.15.107:0/260738312 shutdown_connections 2024-05-09T08:36:08.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.721+0000 7feefc3f9740 1 -- 172.21.15.107:0/260738312 wait complete. 2024-05-09T08:36:08.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.721+0000 7feefc3f9740 20 client.16443 trim_cache size 0 max 16384 2024-05-09T08:36:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.739+0000 7feefc3f9740 1 Processor -- start 2024-05-09T08:36:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.739+0000 7feefc3f9740 1 -- start start 2024-05-09T08:36:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feefc3f9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 0x55f43a6f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feefc3f9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6dbba0 0x55f43a6dda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feefc3f9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6e6270 0x55f43a6fddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feefc3f9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feef41b12c0 con 0x55f43a6e6270 2024-05-09T08:36:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feefc3f9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feef41aeb90 con 0x55f43a6dbba0 2024-05-09T08:36:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feefc3f9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feef41ac9e0 con 0x55f43a6db280 2024-05-09T08:36:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feef37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6e6270 0x55f43a6fddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feef37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6e6270 0x55f43a6fddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57588/0 (socket says 172.21.15.107:57588) 2024-05-09T08:36:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feef37fe640 1 -- 172.21.15.107:0/3632030240 learned_addr learned my addr 172.21.15.107:0/3632030240 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feef2ffd640 1 --2- 172.21.15.107:0/3632030240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 0x55f43a6f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feef27fc640 1 --2- 172.21.15.107:0/3632030240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6dbba0 0x55f43a6dda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feef37fe640 1 -- 172.21.15.107:0/3632030240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 msgr2=0x55f43a6f1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feef37fe640 1 --2- 172.21.15.107:0/3632030240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 0x55f43a6f1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feef37fe640 1 -- 172.21.15.107:0/3632030240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6dbba0 msgr2=0x55f43a6dda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feef37fe640 1 --2- 172.21.15.107:0/3632030240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6dbba0 0x55f43a6dda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feef37fe640 1 -- 172.21.15.107:0/3632030240 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f43a6d73d0 con 0x55f43a6e6270 2024-05-09T08:36:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.740+0000 7feef37fe640 1 --2- 172.21.15.107:0/3632030240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6e6270 0x55f43a6fddd0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7feeec007e90 tx=0x7feeec007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69719a7b9ce6954 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.741+0000 7feed8ff9640 1 -- 172.21.15.107:0/3632030240 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeec025070 con 0x55f43a6e6270 2024-05-09T08:36:08.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.741+0000 7feed8ff9640 1 -- 172.21.15.107:0/3632030240 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feeec019040 con 0x55f43a6e6270 2024-05-09T08:36:08.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.741+0000 7feed8ff9640 1 -- 172.21.15.107:0/3632030240 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeec018070 con 0x55f43a6e6270 2024-05-09T08:36:08.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.741+0000 7feefc3f9740 1 -- 172.21.15.107:0/3632030240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6e6270 msgr2=0x55f43a6fddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.741+0000 7feefc3f9740 1 --2- 172.21.15.107:0/3632030240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6e6270 0x55f43a6fddd0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7feeec007e90 tx=0x7feeec007780 comp rx=0 tx=0).stop 2024-05-09T08:36:08.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feefc3f9740 1 -- 172.21.15.107:0/3632030240 shutdown_connections 2024-05-09T08:36:08.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feefc3f9740 1 --2- 172.21.15.107:0/3632030240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6e6270 0x55f43a6fddd0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feefc3f9740 1 --2- 172.21.15.107:0/3632030240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6dbba0 0x55f43a6dda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feefc3f9740 1 --2- 172.21.15.107:0/3632030240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6db280 0x55f43a6f1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feefc3f9740 1 -- 172.21.15.107:0/3632030240 >> 172.21.15.107:0/3632030240 conn(0x55f43a6d75a0 msgr2=0x55f43a6d67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feefc3f9740 1 -- 172.21.15.107:0/3632030240 shutdown_connections 2024-05-09T08:36:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feefc3f9740 1 -- 172.21.15.107:0/3632030240 wait complete. 2024-05-09T08:36:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feefc3f9740 1 -- start start 2024-05-09T08:36:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feefc3f9740 1 Processor -- start 2024-05-09T08:36:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feefc3f9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6db280 0x55f43a6e9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feefc3f9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6dbba0 0x55f43a6ea4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feefc3f9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6e6270 0x55f43a6eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feefc3f9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feef41ad050 con 0x55f43a6db280 2024-05-09T08:36:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feefc3f9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feef41ae4c0 con 0x55f43a6e6270 2024-05-09T08:36:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feefc3f9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feef41ad720 con 0x55f43a6dbba0 2024-05-09T08:36:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feefc3f9740 10 client.0 _ll_register_callbacks cb 0x7feee8040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feef2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6db280 0x55f43a6e9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feef2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6db280 0x55f43a6e9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57590/0 (socket says 172.21.15.107:57590) 2024-05-09T08:36:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feef2ffd640 1 -- 172.21.15.107:0/967415383 learned_addr learned my addr 172.21.15.107:0/967415383 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.742+0000 7feef27fc640 1 --2- 172.21.15.107:0/967415383 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6dbba0 0x55f43a6ea4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.743+0000 7feef2ffd640 1 -- 172.21.15.107:0/967415383 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6dbba0 msgr2=0x55f43a6ea4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.743+0000 7feef37fe640 1 --2- 172.21.15.107:0/967415383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6e6270 0x55f43a6eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.743+0000 7feef2ffd640 1 --2- 172.21.15.107:0/967415383 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6dbba0 0x55f43a6ea4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.743+0000 7feef2ffd640 1 -- 172.21.15.107:0/967415383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6e6270 msgr2=0x55f43a6eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.743+0000 7feef2ffd640 1 --2- 172.21.15.107:0/967415383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6e6270 0x55f43a6eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.743+0000 7feef2ffd640 1 -- 172.21.15.107:0/967415383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f43a75b7f0 con 0x55f43a6db280 2024-05-09T08:36:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.743+0000 7feef2ffd640 1 --2- 172.21.15.107:0/967415383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6db280 0x55f43a6e9fa0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7feee4097e90 tx=0x7feee401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2d40764b150d3a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.743+0000 7feed6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.743+0000 7feed6ff5640 1 -- 172.21.15.107:0/967415383 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee409b390 con 0x55f43a6db280 2024-05-09T08:36:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.743+0000 7feed6ff5640 1 -- 172.21.15.107:0/967415383 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7feee40980a0 con 0x55f43a6db280 2024-05-09T08:36:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.743+0000 7feefc3f9740 1 -- 172.21.15.107:0/967415383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feeec019040 con 0x55f43a6db280 2024-05-09T08:36:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.743+0000 7feed6ff5640 1 -- 172.21.15.107:0/967415383 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7feee4018cb0 con 0x55f43a6db280 2024-05-09T08:36:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.743+0000 7feefc3f9740 10 client.16449 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.743+0000 7feefc3f9740 1 -- 172.21.15.107:0/967415383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f43a75cb20 con 0x55f43a6db280 2024-05-09T08:36:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.743+0000 7feed6ff5640 1 -- 172.21.15.107:0/967415383 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feeec019040 con 0x55f43a6db280 2024-05-09T08:36:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.744+0000 7feed6ff5640 1 -- 172.21.15.107:0/967415383 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7feee40a4210 con 0x55f43a6db280 2024-05-09T08:36:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.744+0000 7feefc3f9740 10 client.16449 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.744+0000 7feefc3f9740 10 client.16449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.744+0000 7feefc3f9740 20 client.16449 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.744+0000 7feefc3f9740 1 --2- 172.21.15.107:0/967415383 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f43a6f63b0 0x55f43a75f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.744+0000 7feefc3f9740 4 client.16449 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:36:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.744+0000 7feefc3f9740 1 -- 172.21.15.107:0/967415383 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f43a648780 con 0x55f43a6f63b0 2024-05-09T08:36:08.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.744+0000 7feef27fc640 1 --2- 172.21.15.107:0/967415383 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f43a6f63b0 0x55f43a75f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.744+0000 7feef27fc640 1 --2- 172.21.15.107:0/967415383 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f43a6f63b0 0x55f43a75f290 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2c4fbd979d4e487 server_cookie=29a1c3c7cff25c2b in_seq=0 out_seq=0 2024-05-09T08:36:08.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.744+0000 7feed6ff5640 10 client.16449 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:08.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feed6ff5640 1 -- 172.21.15.107:0/967415383 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f43a648780 con 0x55f43a6f63b0 2024-05-09T08:36:08.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feed6ff5640 10 client.16449 handle_command_reply: tid=0 2024-05-09T08:36:08.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feefc3f9740 1 client.16449 shutdown 2024-05-09T08:36:08.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feefc3f9740 1 -- 172.21.15.107:0/967415383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6db280 msgr2=0x55f43a6e9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feefc3f9740 1 --2- 172.21.15.107:0/967415383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6db280 0x55f43a6e9fa0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7feee4097e90 tx=0x7feee401c000 comp rx=0 tx=0).stop 2024-05-09T08:36:08.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feefc3f9740 1 -- 172.21.15.107:0/967415383 shutdown_connections 2024-05-09T08:36:08.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feefc3f9740 1 --2- 172.21.15.107:0/967415383 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f43a6f63b0 0x55f43a75f290 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feefc3f9740 1 --2- 172.21.15.107:0/967415383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f43a6e6270 0x55f43a6eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feefc3f9740 1 --2- 172.21.15.107:0/967415383 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f43a6dbba0 0x55f43a6ea4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feed6ff5640 0 client.16449 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:08.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feefc3f9740 1 --2- 172.21.15.107:0/967415383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f43a6db280 0x55f43a6e9fa0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feefc3f9740 1 -- 172.21.15.107:0/967415383 >> 172.21.15.107:0/967415383 conn(0x55f43a6d75a0 msgr2=0x55f43a6d5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:08.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feefc3f9740 1 -- 172.21.15.107:0/967415383 shutdown_connections 2024-05-09T08:36:08.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feefc3f9740 1 -- 172.21.15.107:0/967415383 wait complete. 2024-05-09T08:36:08.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feefc3f9740 20 client.16449 trim_cache size 0 max 16384 2024-05-09T08:36:08.751 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:36:08.751 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:36:08.751 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:36:08.751 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:36:08.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feef96ee640 1 -- 172.21.15.107:0/365465510 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feedc0a2d00 msgr2=0x7feedc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feef96ee640 1 --2- 172.21.15.107:0/365465510 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feedc0a2d00 0x7feedc0a51c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feeec012940 tx=0x7feeec004ab0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feef96ee640 1 -- 172.21.15.107:0/365465510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feef4100700 msgr2=0x7feef4197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.745+0000 7feef96ee640 1 --2- 172.21.15.107:0/365465510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feef4100700 0x7feef4197f60 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7feee400c920 tx=0x7feee400cde0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.746+0000 7feef96ee640 1 -- 172.21.15.107:0/365465510 shutdown_connections 2024-05-09T08:36:08.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.746+0000 7feef96ee640 1 --2- 172.21.15.107:0/365465510 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feedc0a2d00 0x7feedc0a51c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.746+0000 7feef96ee640 1 --2- 172.21.15.107:0/365465510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feef4106710 0x7feef419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.746+0000 7feef96ee640 1 --2- 172.21.15.107:0/365465510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feef41010c0 0x7feef41984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.746+0000 7feef96ee640 1 --2- 172.21.15.107:0/365465510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feef4100700 0x7feef4197f60 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.746+0000 7feef96ee640 1 -- 172.21.15.107:0/365465510 >> 172.21.15.107:0/365465510 conn(0x7feef4077830 msgr2=0x7feef4068ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:08.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.746+0000 7feef96ee640 1 -- 172.21.15.107:0/365465510 shutdown_connections 2024-05-09T08:36:08.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.746+0000 7feef96ee640 1 -- 172.21.15.107:0/365465510 wait complete. 2024-05-09T08:36:08.757 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:36:08.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:36:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.865+0000 7f29b9541640 1 -- 172.21.15.107:0/1706714121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b41010d0 msgr2=0x7f29b410d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.865+0000 7f29b9541640 1 --2- 172.21.15.107:0/1706714121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b41010d0 0x7f29b410d5e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f29ac009cd0 tx=0x7f29ac02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.865+0000 7f29b9541640 1 -- 172.21.15.107:0/1706714121 shutdown_connections 2024-05-09T08:36:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.865+0000 7f29b9541640 1 --2- 172.21.15.107:0/1706714121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b41010d0 0x7f29b410d5e0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.865+0000 7f29b9541640 1 --2- 172.21.15.107:0/1706714121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29b4100710 0x7f29b4100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.865+0000 7f29b9541640 1 --2- 172.21.15.107:0/1706714121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29b4106720 0x7f29b4106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.865+0000 7f29b9541640 1 -- 172.21.15.107:0/1706714121 >> 172.21.15.107:0/1706714121 conn(0x7f29b4077830 msgr2=0x7f29b4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.865+0000 7f29b9541640 1 -- 172.21.15.107:0/1706714121 shutdown_connections 2024-05-09T08:36:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.865+0000 7f29b9541640 1 -- 172.21.15.107:0/1706714121 wait complete. 2024-05-09T08:36:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.866+0000 7f29b9541640 1 Processor -- start 2024-05-09T08:36:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.866+0000 7f29b9541640 1 -- start start 2024-05-09T08:36:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b9541640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29b4100710 0x7f29b4198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b9541640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29b41010d0 0x7f29b4199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b9541640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4106720 0x7f29b41a3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b9541640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f29b41109e0 con 0x7f29b4106720 2024-05-09T08:36:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b9541640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f29b4110860 con 0x7f29b41010d0 2024-05-09T08:36:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b9541640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f29b4110b60 con 0x7f29b4100710 2024-05-09T08:36:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4106720 0x7f29b41a3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4106720 0x7f29b41a3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57610/0 (socket says 172.21.15.107:57610) 2024-05-09T08:36:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b37fe640 1 -- 172.21.15.107:0/1057786081 learned_addr learned my addr 172.21.15.107:0/1057786081 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b2ffd640 1 --2- 172.21.15.107:0/1057786081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29b4100710 0x7f29b4198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b27fc640 1 --2- 172.21.15.107:0/1057786081 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29b41010d0 0x7f29b4199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b37fe640 1 -- 172.21.15.107:0/1057786081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29b4100710 msgr2=0x7f29b4198ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b37fe640 1 --2- 172.21.15.107:0/1057786081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29b4100710 0x7f29b4198ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b37fe640 1 -- 172.21.15.107:0/1057786081 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29b41010d0 msgr2=0x7f29b4199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:08.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b37fe640 1 --2- 172.21.15.107:0/1057786081 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29b41010d0 0x7f29b4199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:08.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b37fe640 1 -- 172.21.15.107:0/1057786081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b41a4480 con 0x7f29b4106720 2024-05-09T08:36:08.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f29b37fe640 1 --2- 172.21.15.107:0/1057786081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4106720 0x7f29b41a3f40 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f29ac033040 tx=0x7f29ac003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:08.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.867+0000 7f299bfff640 1 -- 172.21.15.107:0/1057786081 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac004040 con 0x7f29b4106720 2024-05-09T08:36:08.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.868+0000 7f29b9541640 1 -- 172.21.15.107:0/1057786081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b41a4650 con 0x7f29b4106720 2024-05-09T08:36:08.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.868+0000 7f29b9541640 1 -- 172.21.15.107:0/1057786081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b41a4b60 con 0x7f29b4106720 2024-05-09T08:36:08.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.868+0000 7f299bfff640 1 -- 172.21.15.107:0/1057786081 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29ac004690 con 0x7f29b4106720 2024-05-09T08:36:08.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.868+0000 7f299bfff640 1 -- 172.21.15.107:0/1057786081 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac030bd0 con 0x7f29b4106720 2024-05-09T08:36:08.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.869+0000 7f29b9541640 1 -- 172.21.15.107:0/1057786081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29a0005290 con 0x7f29b4106720 2024-05-09T08:36:08.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.870+0000 7f299bfff640 1 -- 172.21.15.107:0/1057786081 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f29ac0305a0 con 0x7f29b4106720 2024-05-09T08:36:08.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.870+0000 7f299bfff640 1 --2- 172.21.15.107:0/1057786081 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f299c0a2d00 0x7f299c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:08.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.871+0000 7f29b2ffd640 1 --2- 172.21.15.107:0/1057786081 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f299c0a2d00 0x7f299c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:08.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.871+0000 7f299bfff640 1 -- 172.21.15.107:0/1057786081 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f29ac0f08d0 con 0x7f29b4106720 2024-05-09T08:36:08.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.871+0000 7f29b2ffd640 1 --2- 172.21.15.107:0/1057786081 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f299c0a2d00 0x7f299c0a51c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f29a8009bd0 tx=0x7f29a8005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:08.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:08.875+0000 7f299bfff640 1 -- 172.21.15.107:0/1057786081 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f29ac04b060 con 0x7f29b4106720 2024-05-09T08:36:09.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.142+0000 7f29b9541640 1 -- 172.21.15.107:0/1057786081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29a0005850 con 0x7f29b4106720 2024-05-09T08:36:09.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.142+0000 7f299bfff640 1 -- 172.21.15.107:0/1057786081 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11890 (secure 0 0 0) 0x7f29ac0bd2a0 con 0x7f29b4106720 2024-05-09T08:36:09.141 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:36:09.142 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:34:45.925212+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24478,24499]},"id":1}]} 2024-05-09T08:36:09.142 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 26 2024-05-09T08:36:09.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.145+0000 7f29b9541640 1 -- 172.21.15.107:0/1057786081 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f299c0a2d00 msgr2=0x7f299c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.145+0000 7f29b9541640 1 --2- 172.21.15.107:0/1057786081 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f299c0a2d00 0x7f299c0a51c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f29a8009bd0 tx=0x7f29a8005f90 comp rx=0 tx=0).stop 2024-05-09T08:36:09.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.145+0000 7f29b9541640 1 -- 172.21.15.107:0/1057786081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4106720 msgr2=0x7f29b41a3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.145+0000 7f29b9541640 1 --2- 172.21.15.107:0/1057786081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4106720 0x7f29b41a3f40 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f29ac033040 tx=0x7f29ac003970 comp rx=0 tx=0).stop 2024-05-09T08:36:09.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.145+0000 7f29b9541640 1 -- 172.21.15.107:0/1057786081 shutdown_connections 2024-05-09T08:36:09.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.145+0000 7f29b9541640 1 --2- 172.21.15.107:0/1057786081 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f299c0a2d00 0x7f299c0a51c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.145+0000 7f29b9541640 1 --2- 172.21.15.107:0/1057786081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29b4106720 0x7f29b41a3f40 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.145+0000 7f29b9541640 1 --2- 172.21.15.107:0/1057786081 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29b41010d0 0x7f29b4199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.145+0000 7f29b9541640 1 --2- 172.21.15.107:0/1057786081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29b4100710 0x7f29b4198ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.145+0000 7f29b9541640 1 -- 172.21.15.107:0/1057786081 >> 172.21.15.107:0/1057786081 conn(0x7f29b4077830 msgr2=0x7f29b4102eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.145+0000 7f29b9541640 1 -- 172.21.15.107:0/1057786081 shutdown_connections 2024-05-09T08:36:09.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.145+0000 7f29b9541640 1 -- 172.21.15.107:0/1057786081 wait complete. 2024-05-09T08:36:09.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:36:09.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.273+0000 7faae3970640 1 -- 172.21.15.107:0/4251204897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faadc106710 msgr2=0x7faadc106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.273+0000 7faae3970640 1 --2- 172.21.15.107:0/4251204897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faadc106710 0x7faadc106af0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7faad8009930 tx=0x7faad802a420 comp rx=0 tx=0).stop 2024-05-09T08:36:09.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.273+0000 7faae3970640 1 -- 172.21.15.107:0/4251204897 shutdown_connections 2024-05-09T08:36:09.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.273+0000 7faae3970640 1 --2- 172.21.15.107:0/4251204897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faadc1010c0 0x7faadc10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.273+0000 7faae3970640 1 --2- 172.21.15.107:0/4251204897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faadc100700 0x7faadc100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.273+0000 7faae3970640 1 --2- 172.21.15.107:0/4251204897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faadc106710 0x7faadc106af0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.273+0000 7faae3970640 1 -- 172.21.15.107:0/4251204897 >> 172.21.15.107:0/4251204897 conn(0x7faadc077830 msgr2=0x7faadc077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.273+0000 7faae3970640 1 -- 172.21.15.107:0/4251204897 shutdown_connections 2024-05-09T08:36:09.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.273+0000 7faae3970640 1 -- 172.21.15.107:0/4251204897 wait complete. 2024-05-09T08:36:09.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.273+0000 7faae3970640 1 Processor -- start 2024-05-09T08:36:09.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.273+0000 7faae3970640 1 -- start start 2024-05-09T08:36:09.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae3970640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faadc100700 0x7faadc196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae3970640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faadc1010c0 0x7faadc197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae3970640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faadc106710 0x7faadc1976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae3970640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faadc1109d0 con 0x7faadc100700 2024-05-09T08:36:09.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae3970640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faadc110850 con 0x7faadc1010c0 2024-05-09T08:36:09.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae3970640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faadc110b50 con 0x7faadc106710 2024-05-09T08:36:09.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae170c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faadc100700 0x7faadc196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae170c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faadc100700 0x7faadc196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57630/0 (socket says 172.21.15.107:57630) 2024-05-09T08:36:09.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae170c640 1 -- 172.21.15.107:0/2253727588 learned_addr learned my addr 172.21.15.107:0/2253727588 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:09.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae1f0d640 1 --2- 172.21.15.107:0/2253727588 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faadc106710 0x7faadc1976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae0f0b640 1 --2- 172.21.15.107:0/2253727588 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faadc1010c0 0x7faadc197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae170c640 1 -- 172.21.15.107:0/2253727588 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faadc106710 msgr2=0x7faadc1976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae170c640 1 --2- 172.21.15.107:0/2253727588 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faadc106710 0x7faadc1976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae170c640 1 -- 172.21.15.107:0/2253727588 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faadc1010c0 msgr2=0x7faadc197180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae170c640 1 --2- 172.21.15.107:0/2253727588 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faadc1010c0 0x7faadc197180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae170c640 1 -- 172.21.15.107:0/2253727588 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faadc1a64b0 con 0x7faadc100700 2024-05-09T08:36:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae170c640 1 --2- 172.21.15.107:0/2253727588 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faadc100700 0x7faadc196c40 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7faad8005940 tx=0x7faad80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faad27fc640 1 -- 172.21.15.107:0/2253727588 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad803f030 con 0x7faadc100700 2024-05-09T08:36:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae3970640 1 -- 172.21.15.107:0/2253727588 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faadc1a6740 con 0x7faadc100700 2024-05-09T08:36:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faad27fc640 1 -- 172.21.15.107:0/2253727588 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faad80318a0 con 0x7faadc100700 2024-05-09T08:36:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faad27fc640 1 -- 172.21.15.107:0/2253727588 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad8035070 con 0x7faadc100700 2024-05-09T08:36:09.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.274+0000 7faae3970640 1 -- 172.21.15.107:0/2253727588 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faadc1a6c80 con 0x7faadc100700 2024-05-09T08:36:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.277+0000 7faad27fc640 1 -- 172.21.15.107:0/2253727588 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7faad8042030 con 0x7faadc100700 2024-05-09T08:36:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.277+0000 7faae6684740 1 Processor -- start 2024-05-09T08:36:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.277+0000 7faae6684740 1 -- start start 2024-05-09T08:36:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae6684740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624330 0x563a34624710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae6684740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a34624c50 0x563a3462ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae6684740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a3462f320 0x563a34631710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae6684740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563a34589c40 con 0x563a34624330 2024-05-09T08:36:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae6684740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563a34531580 con 0x563a34624c50 2024-05-09T08:36:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae6684740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563a34585c00 con 0x563a3462f320 2024-05-09T08:36:09.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faad27fc640 1 --2- 172.21.15.107:0/2253727588 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faac80a2d00 0x7faac80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae1f0d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624330 0x563a34624710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae1f0d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624330 0x563a34624710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57636/0 (socket says 172.21.15.107:57636) 2024-05-09T08:36:09.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae1f0d640 1 -- 172.21.15.107:0/711182901 learned_addr learned my addr 172.21.15.107:0/711182901 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:09.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae0f0b640 1 --2- 172.21.15.107:0/711182901 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a3462f320 0x563a34631710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae1f0d640 1 --2- 172.21.15.107:0/2253727588 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faac80a2d00 0x7faac80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faad27fc640 1 -- 172.21.15.107:0/2253727588 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7faad80effa0 con 0x7faadc100700 2024-05-09T08:36:09.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae170c640 1 --2- 172.21.15.107:0/711182901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a34624c50 0x563a3462ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae0f0b640 1 -- 172.21.15.107:0/711182901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a34624c50 msgr2=0x563a3462ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae0f0b640 1 --2- 172.21.15.107:0/711182901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a34624c50 0x563a3462ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae0f0b640 1 -- 172.21.15.107:0/711182901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624330 msgr2=0x563a34624710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae0f0b640 1 --2- 172.21.15.107:0/711182901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624330 0x563a34624710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae0f0b640 1 -- 172.21.15.107:0/711182901 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a3429d3f0 con 0x563a3462f320 2024-05-09T08:36:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.278+0000 7faae1f0d640 1 --2- 172.21.15.107:0/711182901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624330 0x563a34624710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.279+0000 7faae0f0b640 1 --2- 172.21.15.107:0/711182901 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a3462f320 0x563a34631710 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faacc009b40 tx=0x7faacc0113e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3bd4df9d3a3446b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.279+0000 7faae1f0d640 1 --2- 172.21.15.107:0/2253727588 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faac80a2d00 0x7faac80a51c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7faad4014db0 tx=0x7faad4007520 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.279+0000 7faac77fe640 1 -- 172.21.15.107:0/711182901 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc011e10 con 0x563a3462f320 2024-05-09T08:36:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.279+0000 7faac77fe640 1 -- 172.21.15.107:0/711182901 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faacc012900 con 0x563a3462f320 2024-05-09T08:36:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.279+0000 7faac77fe640 1 -- 172.21.15.107:0/711182901 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc00f930 con 0x563a3462f320 2024-05-09T08:36:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.280+0000 7faae6684740 1 -- 172.21.15.107:0/711182901 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a3462f320 msgr2=0x563a34631710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.280+0000 7faae6684740 1 --2- 172.21.15.107:0/711182901 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a3462f320 0x563a34631710 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faacc009b40 tx=0x7faacc0113e0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.280+0000 7faae6684740 1 -- 172.21.15.107:0/711182901 shutdown_connections 2024-05-09T08:36:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.280+0000 7faae6684740 1 --2- 172.21.15.107:0/711182901 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a3462f320 0x563a34631710 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.280+0000 7faae6684740 1 --2- 172.21.15.107:0/711182901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a34624c50 0x563a3462ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.280+0000 7faae6684740 1 --2- 172.21.15.107:0/711182901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624330 0x563a34624710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.280+0000 7faae6684740 1 -- 172.21.15.107:0/711182901 >> 172.21.15.107:0/711182901 conn(0x563a34620650 msgr2=0x563a34620a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.280+0000 7faae6684740 1 -- 172.21.15.107:0/711182901 shutdown_connections 2024-05-09T08:36:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.280+0000 7faae6684740 1 -- 172.21.15.107:0/711182901 wait complete. 2024-05-09T08:36:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.280+0000 7faae6684740 1 -- start start 2024-05-09T08:36:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.280+0000 7faae6684740 1 Processor -- start 2024-05-09T08:36:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae6684740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a34624330 0x563a3462c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae6684740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624c50 0x563a3462c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae6684740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a3462f320 0x563a3462cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae6684740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563a34632110 con 0x563a34624c50 2024-05-09T08:36:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae6684740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563a34544c40 con 0x563a34624330 2024-05-09T08:36:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae6684740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563a34632290 con 0x563a3462f320 2024-05-09T08:36:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae6684740 10 client.0 _ll_register_callbacks cb 0x563a3429cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae0f0b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624c50 0x563a3462c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae0f0b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624c50 0x563a3462c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57638/0 (socket says 172.21.15.107:57638) 2024-05-09T08:36:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae0f0b640 1 -- 172.21.15.107:0/1584939300 learned_addr learned my addr 172.21.15.107:0/1584939300 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae170c640 1 --2- 172.21.15.107:0/1584939300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a34624330 0x563a3462c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae1f0d640 1 --2- 172.21.15.107:0/1584939300 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a3462f320 0x563a3462cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae0f0b640 1 -- 172.21.15.107:0/1584939300 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a3462f320 msgr2=0x563a3462cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae0f0b640 1 --2- 172.21.15.107:0/1584939300 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a3462f320 0x563a3462cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae0f0b640 1 -- 172.21.15.107:0/1584939300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a34624330 msgr2=0x563a3462c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae0f0b640 1 --2- 172.21.15.107:0/1584939300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a34624330 0x563a3462c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.281+0000 7faae0f0b640 1 -- 172.21.15.107:0/1584939300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a343677d0 con 0x563a34624c50 2024-05-09T08:36:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.282+0000 7faae0f0b640 1 --2- 172.21.15.107:0/1584939300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624c50 0x563a3462c730 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7faacc012730 tx=0x7faacc012760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbabf920803516b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.282+0000 7faac4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.282+0000 7faac4ff9640 1 -- 172.21.15.107:0/1584939300 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc028070 con 0x563a34624c50 2024-05-09T08:36:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.282+0000 7faac4ff9640 1 -- 172.21.15.107:0/1584939300 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7faacc005040 con 0x563a34624c50 2024-05-09T08:36:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.282+0000 7faae6684740 1 -- 172.21.15.107:0/1584939300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faacc012900 con 0x563a34624c50 2024-05-09T08:36:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.282+0000 7faac4ff9640 1 -- 172.21.15.107:0/1584939300 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faacc034050 con 0x563a34624c50 2024-05-09T08:36:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.282+0000 7faae6684740 10 client.16467 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.282+0000 7faae6684740 1 -- 172.21.15.107:0/1584939300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563a3463e240 con 0x563a34624c50 2024-05-09T08:36:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.282+0000 7faac4ff9640 1 -- 172.21.15.107:0/1584939300 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faacc012900 con 0x563a34624c50 2024-05-09T08:36:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.282+0000 7faac4ff9640 1 -- 172.21.15.107:0/1584939300 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7faacc004ad0 con 0x563a34624c50 2024-05-09T08:36:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.283+0000 7faae6684740 10 client.16467 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.283+0000 7faae6684740 10 client.16467 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:36:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.283+0000 7faae6684740 20 client.16467 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.283+0000 7faae6684740 1 --2- 172.21.15.107:0/1584939300 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563a3463f650 0x563a34641a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.283+0000 7faae6684740 4 client.16467 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:09.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.283+0000 7faae6684740 1 -- 172.21.15.107:0/1584939300 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a34597370 con 0x563a3463f650 2024-05-09T08:36:09.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.283+0000 7faae170c640 1 --2- 172.21.15.107:0/1584939300 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563a3463f650 0x563a34641a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.283+0000 7faae170c640 1 --2- 172.21.15.107:0/1584939300 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563a3463f650 0x563a34641a40 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=52c147a9538bcefb server_cookie=7b3be370c4b45851 in_seq=0 out_seq=0 2024-05-09T08:36:09.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.283+0000 7faac4ff9640 10 client.16467 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:36:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.284+0000 7faac4ff9640 1 -- 172.21.15.107:0/1584939300 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563a34597370 con 0x563a3463f650 2024-05-09T08:36:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.284+0000 7faac4ff9640 10 client.16467 handle_command_reply: tid=0 2024-05-09T08:36:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.284+0000 7faae6684740 1 client.16467 shutdown 2024-05-09T08:36:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.284+0000 7faae6684740 1 -- 172.21.15.107:0/1584939300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624c50 msgr2=0x563a3462c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.284+0000 7faae6684740 1 --2- 172.21.15.107:0/1584939300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624c50 0x563a3462c730 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7faacc012730 tx=0x7faacc012760 comp rx=0 tx=0).stop 2024-05-09T08:36:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.284+0000 7faae6684740 1 -- 172.21.15.107:0/1584939300 shutdown_connections 2024-05-09T08:36:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.284+0000 7faae6684740 1 --2- 172.21.15.107:0/1584939300 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563a3463f650 0x563a34641a40 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.284+0000 7faae6684740 1 --2- 172.21.15.107:0/1584939300 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a3462f320 0x563a3462cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.284+0000 7faae6684740 1 --2- 172.21.15.107:0/1584939300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624c50 0x563a3462c730 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.284+0000 7faac4ff9640 0 client.16467 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:36:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.284+0000 7faae6684740 1 --2- 172.21.15.107:0/1584939300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a34624330 0x563a3462c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.284+0000 7faae6684740 1 -- 172.21.15.107:0/1584939300 >> 172.21.15.107:0/1584939300 conn(0x563a34620650 msgr2=0x563a34626450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.284+0000 7faae6684740 1 -- 172.21.15.107:0/1584939300 shutdown_connections 2024-05-09T08:36:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.284+0000 7faae6684740 1 -- 172.21.15.107:0/1584939300 wait complete. 2024-05-09T08:36:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.284+0000 7faae6684740 20 client.16467 trim_cache size 0 max 16384 2024-05-09T08:36:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.299+0000 7faae6684740 1 Processor -- start 2024-05-09T08:36:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.299+0000 7faae6684740 1 -- start start 2024-05-09T08:36:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae6684740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624330 0x563a3463afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae6684740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a34624c50 0x563a34647120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae6684740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a3462f320 0x563a34647660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae6684740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faadc1b35c0 con 0x563a34624330 2024-05-09T08:36:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae6684740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faadc1b0e90 con 0x563a34624c50 2024-05-09T08:36:09.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae6684740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faadc1aece0 con 0x563a3462f320 2024-05-09T08:36:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae170c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624330 0x563a3463afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae170c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624330 0x563a3463afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57650/0 (socket says 172.21.15.107:57650) 2024-05-09T08:36:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae170c640 1 -- 172.21.15.107:0/1986121271 learned_addr learned my addr 172.21.15.107:0/1986121271 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae0f0b640 1 --2- 172.21.15.107:0/1986121271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a34624c50 0x563a34647120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae1f0d640 1 --2- 172.21.15.107:0/1986121271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a3462f320 0x563a34647660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae170c640 1 -- 172.21.15.107:0/1986121271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a3462f320 msgr2=0x563a34647660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae170c640 1 --2- 172.21.15.107:0/1986121271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a3462f320 0x563a34647660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae170c640 1 -- 172.21.15.107:0/1986121271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a34624c50 msgr2=0x563a34647120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae170c640 1 --2- 172.21.15.107:0/1986121271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a34624c50 0x563a34647120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae170c640 1 -- 172.21.15.107:0/1986121271 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a34641ea0 con 0x563a34624330 2024-05-09T08:36:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.300+0000 7faae170c640 1 --2- 172.21.15.107:0/1986121271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624330 0x563a3463afd0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7faad8031440 tx=0x7faad80dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802bbf276f788549 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.301+0000 7faac6ffd640 1 -- 172.21.15.107:0/1986121271 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad8049040 con 0x563a34624330 2024-05-09T08:36:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.301+0000 7faac6ffd640 1 -- 172.21.15.107:0/1986121271 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faad803b460 con 0x563a34624330 2024-05-09T08:36:09.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.301+0000 7faac6ffd640 1 -- 172.21.15.107:0/1986121271 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad80ba290 con 0x563a34624330 2024-05-09T08:36:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.301+0000 7faae6684740 1 -- 172.21.15.107:0/1986121271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624330 msgr2=0x563a3463afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.301+0000 7faae6684740 1 --2- 172.21.15.107:0/1986121271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624330 0x563a3463afd0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7faad8031440 tx=0x7faad80dc040 comp rx=0 tx=0).stop 2024-05-09T08:36:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.301+0000 7faae6684740 1 -- 172.21.15.107:0/1986121271 shutdown_connections 2024-05-09T08:36:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.301+0000 7faae6684740 1 --2- 172.21.15.107:0/1986121271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a3462f320 0x563a34647660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.301+0000 7faae6684740 1 --2- 172.21.15.107:0/1986121271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a34624c50 0x563a34647120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.301+0000 7faae6684740 1 --2- 172.21.15.107:0/1986121271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624330 0x563a3463afd0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.301+0000 7faae6684740 1 -- 172.21.15.107:0/1986121271 >> 172.21.15.107:0/1986121271 conn(0x563a34620650 msgr2=0x563a34620a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.301+0000 7faae6684740 1 -- 172.21.15.107:0/1986121271 shutdown_connections 2024-05-09T08:36:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.301+0000 7faae6684740 1 -- 172.21.15.107:0/1986121271 wait complete. 2024-05-09T08:36:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.302+0000 7faae6684740 1 -- start start 2024-05-09T08:36:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.302+0000 7faae6684740 1 Processor -- start 2024-05-09T08:36:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.302+0000 7faae6684740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a34624330 0x563a34637760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.302+0000 7faae6684740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624c50 0x563a34628ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.302+0000 7faae6684740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a3462f320 0x563a34629010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.302+0000 7faae6684740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faadc1af350 con 0x563a34624c50 2024-05-09T08:36:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.302+0000 7faae6684740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faadc1b07c0 con 0x563a3462f320 2024-05-09T08:36:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.302+0000 7faae6684740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faadc1afa20 con 0x563a34624330 2024-05-09T08:36:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.302+0000 7faae6684740 10 client.0 _ll_register_callbacks cb 0x7faad4007e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.302+0000 7faae0f0b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624c50 0x563a34628ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.302+0000 7faae170c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a34624330 0x563a34637760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.302+0000 7faae170c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a34624330 0x563a34637760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:35806/0 (socket says 172.21.15.107:35806) 2024-05-09T08:36:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.302+0000 7faae170c640 1 -- 172.21.15.107:0/149238764 learned_addr learned my addr 172.21.15.107:0/149238764 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.302+0000 7faae1f0d640 1 --2- 172.21.15.107:0/149238764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a3462f320 0x563a34629010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faae170c640 1 -- 172.21.15.107:0/149238764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a3462f320 msgr2=0x563a34629010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faae170c640 1 --2- 172.21.15.107:0/149238764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a3462f320 0x563a34629010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faae170c640 1 -- 172.21.15.107:0/149238764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624c50 msgr2=0x563a34628ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faae170c640 1 --2- 172.21.15.107:0/149238764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624c50 0x563a34628ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faae170c640 1 -- 172.21.15.107:0/149238764 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a34627df0 con 0x563a34624330 2024-05-09T08:36:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faae0f0b640 1 --2- 172.21.15.107:0/149238764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624c50 0x563a34628ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faae1f0d640 1 --2- 172.21.15.107:0/149238764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a3462f320 0x563a34629010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faae170c640 1 --2- 172.21.15.107:0/149238764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a34624330 0x563a34637760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faad80ed990 tx=0x7faad80bc160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2006f9245382148e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faac4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:36:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faac4ff9640 1 -- 172.21.15.107:0/149238764 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad80d4070 con 0x563a34624330 2024-05-09T08:36:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faac4ff9640 1 -- 172.21.15.107:0/149238764 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7faad80ed610 con 0x563a34624330 2024-05-09T08:36:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faae6684740 1 -- 172.21.15.107:0/149238764 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563a345916c0 con 0x563a34624330 2024-05-09T08:36:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faac4ff9640 1 -- 172.21.15.107:0/149238764 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faad8049040 con 0x563a34624330 2024-05-09T08:36:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faae6684740 10 client.25918 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faae6684740 1 -- 172.21.15.107:0/149238764 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563a346a56e0 con 0x563a34624330 2024-05-09T08:36:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faac4ff9640 1 -- 172.21.15.107:0/149238764 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x563a345916c0 con 0x563a34624330 2024-05-09T08:36:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.303+0000 7faac4ff9640 1 -- 172.21.15.107:0/149238764 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7faad80c0150 con 0x563a34624330 2024-05-09T08:36:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faae6684740 10 client.25918 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:09.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faae6684740 10 client.25918 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:36:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faae6684740 20 client.25918 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faae6684740 1 --2- 172.21.15.107:0/149238764 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563a346a5d40 0x563a346a8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faae6684740 4 client.25918 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:36:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faae6684740 1 -- 172.21.15.107:0/149238764 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563a3450e820 con 0x563a346a5d40 2024-05-09T08:36:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faae0f0b640 1 --2- 172.21.15.107:0/149238764 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563a346a5d40 0x563a346a8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faac4ff9640 10 client.25918 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:36:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faae0f0b640 1 --2- 172.21.15.107:0/149238764 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563a346a5d40 0x563a346a8160 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b501cf0f3f7d8fd7 server_cookie=5ebfa2f7f3cb6b79 in_seq=0 out_seq=0 2024-05-09T08:36:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faac4ff9640 1 -- 172.21.15.107:0/149238764 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563a3450e820 con 0x563a346a5d40 2024-05-09T08:36:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faac4ff9640 10 client.25918 handle_command_reply: tid=0 2024-05-09T08:36:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faae6684740 1 client.25918 shutdown 2024-05-09T08:36:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faae6684740 1 -- 172.21.15.107:0/149238764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a34624330 msgr2=0x563a34637760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faae6684740 1 --2- 172.21.15.107:0/149238764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a34624330 0x563a34637760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faad80ed990 tx=0x7faad80bc160 comp rx=0 tx=0).stop 2024-05-09T08:36:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faae6684740 1 -- 172.21.15.107:0/149238764 shutdown_connections 2024-05-09T08:36:09.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faae6684740 1 --2- 172.21.15.107:0/149238764 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563a346a5d40 0x563a346a8160 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faae6684740 1 --2- 172.21.15.107:0/149238764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563a3462f320 0x563a34629010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faae6684740 1 --2- 172.21.15.107:0/149238764 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a34624c50 0x563a34628ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faac4ff9640 0 client.25918 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:36:09.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.304+0000 7faae6684740 1 --2- 172.21.15.107:0/149238764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563a34624330 0x563a34637760 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.305+0000 7faae6684740 1 -- 172.21.15.107:0/149238764 >> 172.21.15.107:0/149238764 conn(0x563a34620650 msgr2=0x563a3462a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.305+0000 7faae6684740 1 -- 172.21.15.107:0/149238764 shutdown_connections 2024-05-09T08:36:09.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.305+0000 7faae6684740 1 -- 172.21.15.107:0/149238764 wait complete. 2024-05-09T08:36:09.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.305+0000 7faae6684740 20 client.25918 trim_cache size 0 max 16384 2024-05-09T08:36:09.310 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:36:09.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.305+0000 7faae3970640 1 -- 172.21.15.107:0/2253727588 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faac80a2d00 msgr2=0x7faac80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.305+0000 7faae3970640 1 --2- 172.21.15.107:0/2253727588 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faac80a2d00 0x7faac80a51c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7faad4014db0 tx=0x7faad4007520 comp rx=0 tx=0).stop 2024-05-09T08:36:09.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.305+0000 7faae3970640 1 -- 172.21.15.107:0/2253727588 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faadc100700 msgr2=0x7faadc196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.305+0000 7faae3970640 1 --2- 172.21.15.107:0/2253727588 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faadc100700 0x7faadc196c40 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7faad8005940 tx=0x7faad80042f0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.305+0000 7faae3970640 1 -- 172.21.15.107:0/2253727588 shutdown_connections 2024-05-09T08:36:09.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.305+0000 7faae3970640 1 --2- 172.21.15.107:0/2253727588 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faac80a2d00 0x7faac80a51c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.305+0000 7faae3970640 1 --2- 172.21.15.107:0/2253727588 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faadc106710 0x7faadc1976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.305+0000 7faae3970640 1 --2- 172.21.15.107:0/2253727588 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faadc1010c0 0x7faadc197180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.305+0000 7faae3970640 1 --2- 172.21.15.107:0/2253727588 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faadc100700 0x7faadc196c40 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.305+0000 7faae3970640 1 -- 172.21.15.107:0/2253727588 >> 172.21.15.107:0/2253727588 conn(0x7faadc077830 msgr2=0x7faadc1030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.305+0000 7faae3970640 1 -- 172.21.15.107:0/2253727588 shutdown_connections 2024-05-09T08:36:09.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.305+0000 7faae3970640 1 -- 172.21.15.107:0/2253727588 wait complete. 2024-05-09T08:36:09.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:36:09.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.432+0000 7f89dae9a640 1 -- 172.21.15.107:0/2805257755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89d4100700 msgr2=0x7f89d4100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.432+0000 7f89dae9a640 1 --2- 172.21.15.107:0/2805257755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89d4100700 0x7f89d4100b80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009930 tx=0x7f89c002a420 comp rx=0 tx=0).stop 2024-05-09T08:36:09.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.432+0000 7f89dae9a640 1 -- 172.21.15.107:0/2805257755 shutdown_connections 2024-05-09T08:36:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.432+0000 7f89dae9a640 1 --2- 172.21.15.107:0/2805257755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d41010c0 0x7f89d410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.432+0000 7f89dae9a640 1 --2- 172.21.15.107:0/2805257755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89d4100700 0x7f89d4100b80 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.432+0000 7f89dae9a640 1 --2- 172.21.15.107:0/2805257755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89d4106710 0x7f89d4106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.432+0000 7f89dae9a640 1 -- 172.21.15.107:0/2805257755 >> 172.21.15.107:0/2805257755 conn(0x7f89d4077830 msgr2=0x7f89d4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.432+0000 7f89dae9a640 1 -- 172.21.15.107:0/2805257755 shutdown_connections 2024-05-09T08:36:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.432+0000 7f89dae9a640 1 -- 172.21.15.107:0/2805257755 wait complete. 2024-05-09T08:36:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.433+0000 7f89dae9a640 1 Processor -- start 2024-05-09T08:36:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.433+0000 7f89dae9a640 1 -- start start 2024-05-09T08:36:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.433+0000 7f89dae9a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89d4100700 0x7f89d4196f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.433+0000 7f89dae9a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89d41010c0 0x7f89d4197470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.433+0000 7f89dae9a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d4106710 0x7f89d419b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.433+0000 7f89dae9a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f89d41109d0 con 0x7f89d41010c0 2024-05-09T08:36:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.433+0000 7f89dae9a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f89d4110850 con 0x7f89d4106710 2024-05-09T08:36:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.433+0000 7f89dae9a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f89d4110b50 con 0x7f89d4100700 2024-05-09T08:36:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.433+0000 7f89d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89d41010c0 0x7f89d4197470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.433+0000 7f89d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89d41010c0 0x7f89d4197470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57686/0 (socket says 172.21.15.107:57686) 2024-05-09T08:36:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.433+0000 7f89d3fff640 1 -- 172.21.15.107:0/4083810641 learned_addr learned my addr 172.21.15.107:0/4083810641 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.433+0000 7f89d8c36640 1 --2- 172.21.15.107:0/4083810641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89d4100700 0x7f89d4196f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.433+0000 7f89d9437640 1 --2- 172.21.15.107:0/4083810641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d4106710 0x7f89d419b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.434+0000 7f89d8c36640 1 -- 172.21.15.107:0/4083810641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d4106710 msgr2=0x7f89d419b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.434+0000 7f89d8c36640 1 --2- 172.21.15.107:0/4083810641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d4106710 0x7f89d419b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.434+0000 7f89d8c36640 1 -- 172.21.15.107:0/4083810641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89d41010c0 msgr2=0x7f89d4197470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.434+0000 7f89d8c36640 1 --2- 172.21.15.107:0/4083810641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89d41010c0 0x7f89d4197470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.434+0000 7f89d8c36640 1 -- 172.21.15.107:0/4083810641 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d419bf20 con 0x7f89d4100700 2024-05-09T08:36:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.434+0000 7f89d3fff640 1 --2- 172.21.15.107:0/4083810641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89d41010c0 0x7f89d4197470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:09.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.434+0000 7f89d8c36640 1 --2- 172.21.15.107:0/4083810641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89d4100700 0x7f89d4196f30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f89c800e9f0 tx=0x7f89c800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.434+0000 7f89d1ffb640 1 -- 172.21.15.107:0/4083810641 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c800ccc0 con 0x7f89d4100700 2024-05-09T08:36:09.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.434+0000 7f89dae9a640 1 -- 172.21.15.107:0/4083810641 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d419c210 con 0x7f89d4100700 2024-05-09T08:36:09.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.434+0000 7f89d1ffb640 1 -- 172.21.15.107:0/4083810641 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f89c80108e0 con 0x7f89d4100700 2024-05-09T08:36:09.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.434+0000 7f89d1ffb640 1 -- 172.21.15.107:0/4083810641 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8018880 con 0x7f89d4100700 2024-05-09T08:36:09.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.434+0000 7f89dae9a640 1 -- 172.21.15.107:0/4083810641 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d41a3aa0 con 0x7f89d4100700 2024-05-09T08:36:09.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.436+0000 7f89d1ffb640 1 -- 172.21.15.107:0/4083810641 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f89c8010a90 con 0x7f89d4100700 2024-05-09T08:36:09.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.437+0000 7f89d1ffb640 1 --2- 172.21.15.107:0/4083810641 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f89c40a2d00 0x7f89c40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.437+0000 7f89d1ffb640 1 -- 172.21.15.107:0/4083810641 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f89c80cfb80 con 0x7f89d4100700 2024-05-09T08:36:09.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.437+0000 7f89d3fff640 1 --2- 172.21.15.107:0/4083810641 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f89c40a2d00 0x7f89c40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.437+0000 7f89ddba7740 1 Processor -- start 2024-05-09T08:36:09.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.437+0000 7f89ddba7740 1 -- start start 2024-05-09T08:36:09.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89ddba7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed455fe0 0x55eaed4563c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89ddba7740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed456900 0x55eaed4653e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89d3fff640 1 --2- 172.21.15.107:0/4083810641 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f89c40a2d00 0x7f89c40a51c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f89c0000c00 tx=0x7f89c00058d0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89ddba7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed465920 0x55eaed467d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89ddba7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55eaed3bbc40 con 0x55eaed455fe0 2024-05-09T08:36:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89ddba7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55eaed363580 con 0x55eaed465920 2024-05-09T08:36:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89ddba7740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55eaed3b7c00 con 0x55eaed456900 2024-05-09T08:36:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89d8c36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed455fe0 0x55eaed4563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89d8c36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed455fe0 0x55eaed4563c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57688/0 (socket says 172.21.15.107:57688) 2024-05-09T08:36:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89d8c36640 1 -- 172.21.15.107:0/46765166 learned_addr learned my addr 172.21.15.107:0/46765166 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89d9437640 1 --2- 172.21.15.107:0/46765166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed465920 0x55eaed467d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89d3fff640 1 --2- 172.21.15.107:0/46765166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed456900 0x55eaed4653e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89d8c36640 1 -- 172.21.15.107:0/46765166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed456900 msgr2=0x55eaed4653e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89d8c36640 1 --2- 172.21.15.107:0/46765166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed456900 0x55eaed4653e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89d8c36640 1 -- 172.21.15.107:0/46765166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed465920 msgr2=0x55eaed467d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89d8c36640 1 --2- 172.21.15.107:0/46765166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed465920 0x55eaed467d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89d8c36640 1 -- 172.21.15.107:0/46765166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eaed0cf3f0 con 0x55eaed455fe0 2024-05-09T08:36:09.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89d3fff640 1 --2- 172.21.15.107:0/46765166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed456900 0x55eaed4653e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:36:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.438+0000 7f89d8c36640 1 --2- 172.21.15.107:0/46765166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed455fe0 0x55eaed4563c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f89c80986b0 tx=0x7f89c809c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ff157671b7e515f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.439+0000 7f89beffd640 1 -- 172.21.15.107:0/46765166 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c80a0090 con 0x55eaed455fe0 2024-05-09T08:36:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.439+0000 7f89beffd640 1 -- 172.21.15.107:0/46765166 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f89c80a4950 con 0x55eaed455fe0 2024-05-09T08:36:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.439+0000 7f89beffd640 1 -- 172.21.15.107:0/46765166 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8097a30 con 0x55eaed455fe0 2024-05-09T08:36:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.439+0000 7f89ddba7740 1 -- 172.21.15.107:0/46765166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed455fe0 msgr2=0x55eaed4563c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.439+0000 7f89ddba7740 1 --2- 172.21.15.107:0/46765166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed455fe0 0x55eaed4563c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f89c80986b0 tx=0x7f89c809c3e0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.439+0000 7f89ddba7740 1 -- 172.21.15.107:0/46765166 shutdown_connections 2024-05-09T08:36:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.439+0000 7f89ddba7740 1 --2- 172.21.15.107:0/46765166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed465920 0x55eaed467d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.439+0000 7f89ddba7740 1 --2- 172.21.15.107:0/46765166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed456900 0x55eaed4653e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.439+0000 7f89ddba7740 1 --2- 172.21.15.107:0/46765166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed455fe0 0x55eaed4563c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.439+0000 7f89ddba7740 1 -- 172.21.15.107:0/46765166 >> 172.21.15.107:0/46765166 conn(0x55eaed452650 msgr2=0x55eaed452a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.440+0000 7f89ddba7740 1 -- 172.21.15.107:0/46765166 shutdown_connections 2024-05-09T08:36:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.440+0000 7f89ddba7740 1 -- 172.21.15.107:0/46765166 wait complete. 2024-05-09T08:36:09.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.440+0000 7f89ddba7740 1 -- start start 2024-05-09T08:36:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.440+0000 7f89ddba7740 1 Processor -- start 2024-05-09T08:36:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.440+0000 7f89ddba7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed455fe0 0x55eaed462820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.440+0000 7f89ddba7740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed456900 0x55eaed462d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.440+0000 7f89ddba7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed465920 0x55eaed4632a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.440+0000 7f89ddba7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55eaed4684f0 con 0x55eaed465920 2024-05-09T08:36:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.440+0000 7f89ddba7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55eaed376c40 con 0x55eaed455fe0 2024-05-09T08:36:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.440+0000 7f89ddba7740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55eaed468670 con 0x55eaed456900 2024-05-09T08:36:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.440+0000 7f89ddba7740 10 client.0 _ll_register_callbacks cb 0x55eaed0cebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89d9437640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed465920 0x55eaed4632a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89d9437640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed465920 0x55eaed4632a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57700/0 (socket says 172.21.15.107:57700) 2024-05-09T08:36:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89d9437640 1 -- 172.21.15.107:0/1646864402 learned_addr learned my addr 172.21.15.107:0/1646864402 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89d8c36640 1 --2- 172.21.15.107:0/1646864402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed455fe0 0x55eaed462820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89d3fff640 1 --2- 172.21.15.107:0/1646864402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed456900 0x55eaed462d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89d9437640 1 -- 172.21.15.107:0/1646864402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed456900 msgr2=0x55eaed462d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89d9437640 1 --2- 172.21.15.107:0/1646864402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed456900 0x55eaed462d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89d9437640 1 -- 172.21.15.107:0/1646864402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed455fe0 msgr2=0x55eaed462820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89d9437640 1 --2- 172.21.15.107:0/1646864402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed455fe0 0x55eaed462820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89d9437640 1 -- 172.21.15.107:0/1646864402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eaed1997d0 con 0x55eaed465920 2024-05-09T08:36:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89d8c36640 1 --2- 172.21.15.107:0/1646864402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed455fe0 0x55eaed462820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89d9437640 1 --2- 172.21.15.107:0/1646864402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed465920 0x55eaed4632a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f89cc007fd0 tx=0x7f89cc011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f6f3b31018656a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89b3fff640 1 -- 172.21.15.107:0/1646864402 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc01e060 con 0x55eaed465920 2024-05-09T08:36:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89b3fff640 1 -- 172.21.15.107:0/1646864402 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f89cc0193f0 con 0x55eaed465920 2024-05-09T08:36:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89ddba7740 1 -- 172.21.15.107:0/1646864402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89c80a4950 con 0x55eaed465920 2024-05-09T08:36:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89b3fff640 1 -- 172.21.15.107:0/1646864402 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f89cc025050 con 0x55eaed465920 2024-05-09T08:36:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89ddba7740 10 client.16485 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.441+0000 7f89ddba7740 1 -- 172.21.15.107:0/1646864402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55eaed474670 con 0x55eaed465920 2024-05-09T08:36:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.442+0000 7f89b3fff640 1 -- 172.21.15.107:0/1646864402 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f89c80a4950 con 0x55eaed465920 2024-05-09T08:36:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.442+0000 7f89b3fff640 1 -- 172.21.15.107:0/1646864402 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f89cc018030 con 0x55eaed465920 2024-05-09T08:36:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.442+0000 7f89ddba7740 10 client.16485 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.442+0000 7f89ddba7740 10 client.16485 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:36:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.442+0000 7f89ddba7740 20 client.16485 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.442+0000 7f89ddba7740 1 --2- 172.21.15.107:0/1646864402 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55eaed475b30 0x55eaed477f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.442+0000 7f89ddba7740 4 client.16485 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.442+0000 7f89ddba7740 1 -- 172.21.15.107:0/1646864402 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eaed3c9370 con 0x55eaed475b30 2024-05-09T08:36:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.442+0000 7f89d8c36640 1 --2- 172.21.15.107:0/1646864402 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55eaed475b30 0x55eaed477f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.442+0000 7f89d8c36640 1 --2- 172.21.15.107:0/1646864402 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55eaed475b30 0x55eaed477f80 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d2f2c7143610724c server_cookie=b1fb69ef19e7406 in_seq=0 out_seq=0 2024-05-09T08:36:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.442+0000 7f89b3fff640 10 client.16485 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:36:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.443+0000 7f89b3fff640 1 -- 172.21.15.107:0/1646864402 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eaed3c9370 con 0x55eaed475b30 2024-05-09T08:36:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.443+0000 7f89b3fff640 10 client.16485 handle_command_reply: tid=0 2024-05-09T08:36:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.443+0000 7f89ddba7740 1 client.16485 shutdown 2024-05-09T08:36:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.443+0000 7f89ddba7740 1 -- 172.21.15.107:0/1646864402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed465920 msgr2=0x55eaed4632a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.443+0000 7f89ddba7740 1 --2- 172.21.15.107:0/1646864402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed465920 0x55eaed4632a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f89cc007fd0 tx=0x7f89cc011ec0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.443+0000 7f89ddba7740 1 -- 172.21.15.107:0/1646864402 shutdown_connections 2024-05-09T08:36:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.443+0000 7f89ddba7740 1 --2- 172.21.15.107:0/1646864402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed465920 0x55eaed4632a0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.443+0000 7f89ddba7740 1 --2- 172.21.15.107:0/1646864402 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55eaed475b30 0x55eaed477f80 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.443+0000 7f89b3fff640 0 client.16485 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:36:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.443+0000 7f89ddba7740 1 --2- 172.21.15.107:0/1646864402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed456900 0x55eaed462d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.443+0000 7f89ddba7740 1 --2- 172.21.15.107:0/1646864402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed455fe0 0x55eaed462820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.443+0000 7f89ddba7740 1 -- 172.21.15.107:0/1646864402 >> 172.21.15.107:0/1646864402 conn(0x55eaed452650 msgr2=0x55eaed4583d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.444+0000 7f89ddba7740 1 -- 172.21.15.107:0/1646864402 shutdown_connections 2024-05-09T08:36:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.444+0000 7f89ddba7740 1 -- 172.21.15.107:0/1646864402 wait complete. 2024-05-09T08:36:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.444+0000 7f89ddba7740 20 client.16485 trim_cache size 0 max 16384 2024-05-09T08:36:09.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.458+0000 7f89ddba7740 1 Processor -- start 2024-05-09T08:36:09.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.458+0000 7f89ddba7740 1 -- start start 2024-05-09T08:36:09.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.458+0000 7f89ddba7740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed455fe0 0x55eaed4713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.458+0000 7f89ddba7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed456900 0x55eaed450f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.458+0000 7f89ddba7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed465920 0x55eaed451460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.458+0000 7f89ddba7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f89d41af170 con 0x55eaed456900 2024-05-09T08:36:09.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.458+0000 7f89ddba7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f89d41aca40 con 0x55eaed465920 2024-05-09T08:36:09.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.458+0000 7f89ddba7740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f89d41aa890 con 0x55eaed455fe0 2024-05-09T08:36:09.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.458+0000 7f89d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed456900 0x55eaed450f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.458+0000 7f89d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed456900 0x55eaed450f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57716/0 (socket says 172.21.15.107:57716) 2024-05-09T08:36:09.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.458+0000 7f89d3fff640 1 -- 172.21.15.107:0/2342399093 learned_addr learned my addr 172.21.15.107:0/2342399093 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:09.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.458+0000 7f89d9437640 1 --2- 172.21.15.107:0/2342399093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed465920 0x55eaed451460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.458+0000 7f89d8c36640 1 --2- 172.21.15.107:0/2342399093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed455fe0 0x55eaed4713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.458+0000 7f89d3fff640 1 -- 172.21.15.107:0/2342399093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed455fe0 msgr2=0x55eaed4713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.458+0000 7f89d3fff640 1 --2- 172.21.15.107:0/2342399093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed455fe0 0x55eaed4713f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.459+0000 7f89d3fff640 1 -- 172.21.15.107:0/2342399093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed465920 msgr2=0x55eaed451460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.459+0000 7f89d3fff640 1 --2- 172.21.15.107:0/2342399093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed465920 0x55eaed451460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.459+0000 7f89d3fff640 1 -- 172.21.15.107:0/2342399093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eaed443d20 con 0x55eaed456900 2024-05-09T08:36:09.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.459+0000 7f89d3fff640 1 --2- 172.21.15.107:0/2342399093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed456900 0x55eaed450f20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f89c0032fd0 tx=0x7f89c003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1fbe63319a00b9f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.459+0000 7f89be7fc640 1 -- 172.21.15.107:0/2342399093 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c0035040 con 0x55eaed456900 2024-05-09T08:36:09.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.459+0000 7f89be7fc640 1 -- 172.21.15.107:0/2342399093 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f89c00047a0 con 0x55eaed456900 2024-05-09T08:36:09.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.459+0000 7f89be7fc640 1 -- 172.21.15.107:0/2342399093 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c00039a0 con 0x55eaed456900 2024-05-09T08:36:09.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 -- 172.21.15.107:0/2342399093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed456900 msgr2=0x55eaed450f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 --2- 172.21.15.107:0/2342399093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed456900 0x55eaed450f20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f89c0032fd0 tx=0x7f89c003d000 comp rx=0 tx=0).stop 2024-05-09T08:36:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 -- 172.21.15.107:0/2342399093 shutdown_connections 2024-05-09T08:36:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 --2- 172.21.15.107:0/2342399093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed465920 0x55eaed451460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 --2- 172.21.15.107:0/2342399093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed456900 0x55eaed450f20 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 --2- 172.21.15.107:0/2342399093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed455fe0 0x55eaed4713f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 -- 172.21.15.107:0/2342399093 >> 172.21.15.107:0/2342399093 conn(0x55eaed452650 msgr2=0x55eaed465dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 -- 172.21.15.107:0/2342399093 shutdown_connections 2024-05-09T08:36:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 -- 172.21.15.107:0/2342399093 wait complete. 2024-05-09T08:36:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 -- start start 2024-05-09T08:36:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 Processor -- start 2024-05-09T08:36:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed455fe0 0x55eaed46daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed456900 0x55eaed47c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed465920 0x55eaed47cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f89d41aaf00 con 0x55eaed456900 2024-05-09T08:36:09.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f89d41ac370 con 0x55eaed455fe0 2024-05-09T08:36:09.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f89d41ab5d0 con 0x55eaed465920 2024-05-09T08:36:09.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.460+0000 7f89ddba7740 10 client.0 _ll_register_callbacks cb 0x7f89cc007900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:09.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed456900 0x55eaed47c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed456900 0x55eaed47c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57726/0 (socket says 172.21.15.107:57726) 2024-05-09T08:36:09.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89d3fff640 1 -- 172.21.15.107:0/2236687474 learned_addr learned my addr 172.21.15.107:0/2236687474 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:09.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89d8c36640 1 --2- 172.21.15.107:0/2236687474 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed455fe0 0x55eaed46daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89d9437640 1 --2- 172.21.15.107:0/2236687474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed465920 0x55eaed47cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89d3fff640 1 -- 172.21.15.107:0/2236687474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed465920 msgr2=0x55eaed47cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89d3fff640 1 --2- 172.21.15.107:0/2236687474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed465920 0x55eaed47cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89d3fff640 1 -- 172.21.15.107:0/2236687474 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed455fe0 msgr2=0x55eaed46daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89d3fff640 1 --2- 172.21.15.107:0/2236687474 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed455fe0 0x55eaed46daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89d3fff640 1 -- 172.21.15.107:0/2236687474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eaed47d2c0 con 0x55eaed456900 2024-05-09T08:36:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89d3fff640 1 --2- 172.21.15.107:0/2236687474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed456900 0x55eaed47c840 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f89c0005940 tx=0x7f89c00038c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4221e4d275e4e522 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89b3fff640 1 -- 172.21.15.107:0/2236687474 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c0058030 con 0x55eaed456900 2024-05-09T08:36:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89b3fff640 1 -- 172.21.15.107:0/2236687474 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f89c0003cc0 con 0x55eaed456900 2024-05-09T08:36:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89ddba7740 1 -- 172.21.15.107:0/2236687474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89c00047a0 con 0x55eaed456900 2024-05-09T08:36:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89b3fff640 1 -- 172.21.15.107:0/2236687474 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f89c0065050 con 0x55eaed456900 2024-05-09T08:36:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89ddba7740 10 client.16491 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.461+0000 7f89ddba7740 1 -- 172.21.15.107:0/2236687474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55eaed4dbde0 con 0x55eaed456900 2024-05-09T08:36:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.462+0000 7f89b3fff640 1 -- 172.21.15.107:0/2236687474 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f89c00047a0 con 0x55eaed456900 2024-05-09T08:36:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.462+0000 7f89b3fff640 1 -- 172.21.15.107:0/2236687474 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f89c0051550 con 0x55eaed456900 2024-05-09T08:36:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.462+0000 7f89ddba7740 10 client.16491 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.462+0000 7f89ddba7740 10 client.16491 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:36:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.462+0000 7f89ddba7740 20 client.16491 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:09.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.462+0000 7f89ddba7740 1 --2- 172.21.15.107:0/2236687474 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55eaed475b30 0x55eaed4de520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.462+0000 7f89ddba7740 4 client.16491 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:36:09.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.462+0000 7f89ddba7740 1 -- 172.21.15.107:0/2236687474 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eaed3c36c0 con 0x55eaed475b30 2024-05-09T08:36:09.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.462+0000 7f89d8c36640 1 --2- 172.21.15.107:0/2236687474 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55eaed475b30 0x55eaed4de520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.462+0000 7f89d8c36640 1 --2- 172.21.15.107:0/2236687474 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55eaed475b30 0x55eaed4de520 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cfe5855c8b9a47fa server_cookie=21013686a9413b88 in_seq=0 out_seq=0 2024-05-09T08:36:09.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.462+0000 7f89b3fff640 10 client.16491 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:36:09.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.463+0000 7f89b3fff640 1 -- 172.21.15.107:0/2236687474 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eaed3c36c0 con 0x55eaed475b30 2024-05-09T08:36:09.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.463+0000 7f89b3fff640 10 client.16491 handle_command_reply: tid=0 2024-05-09T08:36:09.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.463+0000 7f89ddba7740 1 client.16491 shutdown 2024-05-09T08:36:09.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.463+0000 7f89ddba7740 1 -- 172.21.15.107:0/2236687474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed456900 msgr2=0x55eaed47c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.463+0000 7f89ddba7740 1 --2- 172.21.15.107:0/2236687474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed456900 0x55eaed47c840 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f89c0005940 tx=0x7f89c00038c0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.463+0000 7f89ddba7740 1 -- 172.21.15.107:0/2236687474 shutdown_connections 2024-05-09T08:36:09.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.463+0000 7f89ddba7740 1 --2- 172.21.15.107:0/2236687474 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55eaed475b30 0x55eaed4de520 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.463+0000 7f89ddba7740 1 --2- 172.21.15.107:0/2236687474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eaed465920 0x55eaed47cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.463+0000 7f89b3fff640 0 client.16491 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:36:09.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.463+0000 7f89ddba7740 1 --2- 172.21.15.107:0/2236687474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eaed456900 0x55eaed47c840 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.463+0000 7f89ddba7740 1 --2- 172.21.15.107:0/2236687474 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eaed455fe0 0x55eaed46daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.463+0000 7f89ddba7740 1 -- 172.21.15.107:0/2236687474 >> 172.21.15.107:0/2236687474 conn(0x55eaed452650 msgr2=0x55eaed450f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.463+0000 7f89ddba7740 1 -- 172.21.15.107:0/2236687474 shutdown_connections 2024-05-09T08:36:09.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.463+0000 7f89ddba7740 1 -- 172.21.15.107:0/2236687474 wait complete. 2024-05-09T08:36:09.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.463+0000 7f89ddba7740 20 client.16491 trim_cache size 0 max 16384 2024-05-09T08:36:09.469 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:36:09.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.464+0000 7f89dae9a640 1 -- 172.21.15.107:0/4083810641 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f89c40a2d00 msgr2=0x7f89c40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.464+0000 7f89dae9a640 1 --2- 172.21.15.107:0/4083810641 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f89c40a2d00 0x7f89c40a51c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f89c0000c00 tx=0x7f89c00058d0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.464+0000 7f89dae9a640 1 -- 172.21.15.107:0/4083810641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89d4100700 msgr2=0x7f89d4196f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.464+0000 7f89dae9a640 1 --2- 172.21.15.107:0/4083810641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89d4100700 0x7f89d4196f30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f89c800e9f0 tx=0x7f89c800eeb0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.464+0000 7f89dae9a640 1 -- 172.21.15.107:0/4083810641 shutdown_connections 2024-05-09T08:36:09.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.464+0000 7f89dae9a640 1 --2- 172.21.15.107:0/4083810641 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f89c40a2d00 0x7f89c40a51c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.464+0000 7f89dae9a640 1 --2- 172.21.15.107:0/4083810641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d4106710 0x7f89d419b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.464+0000 7f89dae9a640 1 --2- 172.21.15.107:0/4083810641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89d41010c0 0x7f89d4197470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.464+0000 7f89dae9a640 1 --2- 172.21.15.107:0/4083810641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89d4100700 0x7f89d4196f30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.464+0000 7f89dae9a640 1 -- 172.21.15.107:0/4083810641 >> 172.21.15.107:0/4083810641 conn(0x7f89d4077830 msgr2=0x7f89d4103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.464+0000 7f89dae9a640 1 -- 172.21.15.107:0/4083810641 shutdown_connections 2024-05-09T08:36:09.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.464+0000 7f89dae9a640 1 -- 172.21.15.107:0/4083810641 wait complete. 2024-05-09T08:36:09.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:36:09.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.592+0000 7f812321c640 1 -- 172.21.15.107:0/3183314953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811c100700 msgr2=0x7f811c100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.592+0000 7f812321c640 1 --2- 172.21.15.107:0/3183314953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811c100700 0x7f811c100b80 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8108009930 tx=0x7f810802a420 comp rx=0 tx=0).stop 2024-05-09T08:36:09.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.592+0000 7f812321c640 1 -- 172.21.15.107:0/3183314953 shutdown_connections 2024-05-09T08:36:09.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.592+0000 7f812321c640 1 --2- 172.21.15.107:0/3183314953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f811c1010c0 0x7f811c10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.592+0000 7f812321c640 1 --2- 172.21.15.107:0/3183314953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811c100700 0x7f811c100b80 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.592+0000 7f812321c640 1 --2- 172.21.15.107:0/3183314953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f811c106710 0x7f811c106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.592+0000 7f812321c640 1 -- 172.21.15.107:0/3183314953 >> 172.21.15.107:0/3183314953 conn(0x7f811c077830 msgr2=0x7f811c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.592+0000 7f812321c640 1 -- 172.21.15.107:0/3183314953 shutdown_connections 2024-05-09T08:36:09.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.592+0000 7f812321c640 1 -- 172.21.15.107:0/3183314953 wait complete. 2024-05-09T08:36:09.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f812321c640 1 Processor -- start 2024-05-09T08:36:09.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f812321c640 1 -- start start 2024-05-09T08:36:09.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f812321c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811c100700 0x7f811c198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f812321c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f811c1010c0 0x7f811c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f812321c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f811c106710 0x7f811c19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f812321c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f811c1109d0 con 0x7f811c100700 2024-05-09T08:36:09.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f812321c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f811c110850 con 0x7f811c106710 2024-05-09T08:36:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f812321c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f811c110b50 con 0x7f811c1010c0 2024-05-09T08:36:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f8120fb8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811c100700 0x7f811c198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f8120fb8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811c100700 0x7f811c198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57744/0 (socket says 172.21.15.107:57744) 2024-05-09T08:36:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f8120fb8640 1 -- 172.21.15.107:0/1995983961 learned_addr learned my addr 172.21.15.107:0/1995983961 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f811bfff640 1 --2- 172.21.15.107:0/1995983961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f811c1010c0 0x7f811c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f81217b9640 1 --2- 172.21.15.107:0/1995983961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f811c106710 0x7f811c19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f8120fb8640 1 -- 172.21.15.107:0/1995983961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f811c1010c0 msgr2=0x7f811c198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f8120fb8640 1 --2- 172.21.15.107:0/1995983961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f811c1010c0 0x7f811c198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f8120fb8640 1 -- 172.21.15.107:0/1995983961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f811c106710 msgr2=0x7f811c19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f8120fb8640 1 --2- 172.21.15.107:0/1995983961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f811c106710 0x7f811c19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f8120fb8640 1 -- 172.21.15.107:0/1995983961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811c19cff0 con 0x7f811c100700 2024-05-09T08:36:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.593+0000 7f8120fb8640 1 --2- 172.21.15.107:0/1995983961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811c100700 0x7f811c198050 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f811000d950 tx=0x7f811000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.594+0000 7f8119ffb640 1 -- 172.21.15.107:0/1995983961 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110017070 con 0x7f811c100700 2024-05-09T08:36:09.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.594+0000 7f8119ffb640 1 -- 172.21.15.107:0/1995983961 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8110009c30 con 0x7f811c100700 2024-05-09T08:36:09.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.594+0000 7f812321c640 1 -- 172.21.15.107:0/1995983961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811c19d2e0 con 0x7f811c100700 2024-05-09T08:36:09.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.594+0000 7f8119ffb640 1 -- 172.21.15.107:0/1995983961 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811000f620 con 0x7f811c100700 2024-05-09T08:36:09.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.594+0000 7f812321c640 1 -- 172.21.15.107:0/1995983961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f811c1a4b70 con 0x7f811c100700 2024-05-09T08:36:09.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.597+0000 7f8119ffb640 1 -- 172.21.15.107:0/1995983961 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f81100109a0 con 0x7f811c100700 2024-05-09T08:36:09.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.597+0000 7f812600a740 1 Processor -- start 2024-05-09T08:36:09.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.597+0000 7f812600a740 1 -- start start 2024-05-09T08:36:09.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.597+0000 7f812600a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5844210 0x558be58445f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.597+0000 7f812600a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be58531f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.597+0000 7f812600a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 0x558be5855b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.597+0000 7f812600a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558be57a9c40 con 0x558be5853730 2024-05-09T08:36:09.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.597+0000 7f812600a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558be5751580 con 0x558be5844210 2024-05-09T08:36:09.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.597+0000 7f812600a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x558be57a5c00 con 0x558be5844b30 2024-05-09T08:36:09.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.597+0000 7f811bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 0x558be5855b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.597+0000 7f8120fb8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be58531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.597+0000 7f8119ffb640 1 --2- 172.21.15.107:0/1995983961 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f810c0a2d80 0x7f810c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.598+0000 7f8120fb8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be58531f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:35888/0 (socket says 172.21.15.107:35888) 2024-05-09T08:36:09.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.598+0000 7f8120fb8640 1 -- 172.21.15.107:0/414532980 learned_addr learned my addr 172.21.15.107:0/414532980 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:09.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.598+0000 7f81217b9640 1 --2- 172.21.15.107:0/414532980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5844210 0x558be58445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.598+0000 7f81217b9640 1 --2- 172.21.15.107:0/1995983961 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f810c0a2d80 0x7f810c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.598+0000 7f8120fb8640 1 -- 172.21.15.107:0/414532980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5844210 msgr2=0x558be58445f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.598+0000 7f8120fb8640 1 --2- 172.21.15.107:0/414532980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5844210 0x558be58445f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.598+0000 7f8120fb8640 1 -- 172.21.15.107:0/414532980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 msgr2=0x558be5855b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.598+0000 7f8120fb8640 1 --2- 172.21.15.107:0/414532980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 0x558be5855b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.598+0000 7f8120fb8640 1 -- 172.21.15.107:0/414532980 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558be54bd3f0 con 0x558be5844b30 2024-05-09T08:36:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.598+0000 7f8119ffb640 1 -- 172.21.15.107:0/1995983961 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f81100cf230 con 0x7f811c100700 2024-05-09T08:36:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.598+0000 7f81217b9640 1 --2- 172.21.15.107:0/1995983961 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f810c0a2d80 0x7f810c0a5240 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8114005c40 tx=0x7f8114009500 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.598+0000 7f8120fb8640 1 --2- 172.21.15.107:0/414532980 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be58531f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f81100a0890 tx=0x7f8110097140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22310aa8faa23639 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.598+0000 7f8106ffd640 1 -- 172.21.15.107:0/414532980 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110009de0 con 0x558be5844b30 2024-05-09T08:36:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.598+0000 7f8106ffd640 1 -- 172.21.15.107:0/414532980 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f811009b4a0 con 0x558be5844b30 2024-05-09T08:36:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.598+0000 7f8106ffd640 1 -- 172.21.15.107:0/414532980 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110097b10 con 0x558be5844b30 2024-05-09T08:36:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.599+0000 7f812600a740 1 -- 172.21.15.107:0/414532980 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 msgr2=0x558be58531f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.599+0000 7f812600a740 1 --2- 172.21.15.107:0/414532980 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be58531f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f81100a0890 tx=0x7f8110097140 comp rx=0 tx=0).stop 2024-05-09T08:36:09.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.599+0000 7f812600a740 1 -- 172.21.15.107:0/414532980 shutdown_connections 2024-05-09T08:36:09.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.599+0000 7f812600a740 1 --2- 172.21.15.107:0/414532980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 0x558be5855b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.599+0000 7f812600a740 1 --2- 172.21.15.107:0/414532980 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be58531f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.599+0000 7f812600a740 1 --2- 172.21.15.107:0/414532980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5844210 0x558be58445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.599+0000 7f812600a740 1 -- 172.21.15.107:0/414532980 >> 172.21.15.107:0/414532980 conn(0x558be5840650 msgr2=0x558be5840a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.599+0000 7f812600a740 1 -- 172.21.15.107:0/414532980 shutdown_connections 2024-05-09T08:36:09.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.599+0000 7f812600a740 1 -- 172.21.15.107:0/414532980 wait complete. 2024-05-09T08:36:09.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.600+0000 7f812600a740 1 -- start start 2024-05-09T08:36:09.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.600+0000 7f812600a740 1 Processor -- start 2024-05-09T08:36:09.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.600+0000 7f812600a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5844210 0x558be5850650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.600+0000 7f812600a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be5850b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.600+0000 7f812600a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 0x558be58510d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.600+0000 7f812600a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558be58564b0 con 0x558be5853730 2024-05-09T08:36:09.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.600+0000 7f812600a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558be5764c40 con 0x558be5844210 2024-05-09T08:36:09.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.600+0000 7f812600a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x558be5856630 con 0x558be5844b30 2024-05-09T08:36:09.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.600+0000 7f812600a740 10 client.0 _ll_register_callbacks cb 0x558be54bcbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:09.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.600+0000 7f811bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be5850b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.600+0000 7f811bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be5850b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:35900/0 (socket says 172.21.15.107:35900) 2024-05-09T08:36:09.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.600+0000 7f811bfff640 1 -- 172.21.15.107:0/3687496089 learned_addr learned my addr 172.21.15.107:0/3687496089 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:09.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.600+0000 7f8120fb8640 1 --2- 172.21.15.107:0/3687496089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5844210 0x558be5850650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.601+0000 7f81217b9640 1 --2- 172.21.15.107:0/3687496089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 0x558be58510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.601+0000 7f811bfff640 1 -- 172.21.15.107:0/3687496089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5844210 msgr2=0x558be5850650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.601+0000 7f811bfff640 1 --2- 172.21.15.107:0/3687496089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5844210 0x558be5850650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.601+0000 7f811bfff640 1 -- 172.21.15.107:0/3687496089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 msgr2=0x558be58510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.601+0000 7f811bfff640 1 --2- 172.21.15.107:0/3687496089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 0x558be58510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.601+0000 7f811bfff640 1 -- 172.21.15.107:0/3687496089 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558be55877d0 con 0x558be5844b30 2024-05-09T08:36:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.601+0000 7f81217b9640 1 --2- 172.21.15.107:0/3687496089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 0x558be58510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.601+0000 7f811bfff640 1 --2- 172.21.15.107:0/3687496089 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be5850b90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f810802a940 tx=0x7f8108031ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93c3f295fbc94479 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.601+0000 7f80fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:36:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.602+0000 7f80fbfff640 1 -- 172.21.15.107:0/3687496089 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108035040 con 0x558be5844b30 2024-05-09T08:36:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.602+0000 7f812600a740 1 -- 172.21.15.107:0/3687496089 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f811009b4a0 con 0x558be5844b30 2024-05-09T08:36:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.602+0000 7f80fbfff640 1 -- 172.21.15.107:0/3687496089 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f81080317e0 con 0x558be5844b30 2024-05-09T08:36:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.602+0000 7f80fbfff640 1 -- 172.21.15.107:0/3687496089 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8108004040 con 0x558be5844b30 2024-05-09T08:36:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.602+0000 7f812600a740 10 client.25945 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.602+0000 7f812600a740 1 -- 172.21.15.107:0/3687496089 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558be5862700 con 0x558be5844b30 2024-05-09T08:36:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.603+0000 7f80fbfff640 1 -- 172.21.15.107:0/3687496089 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f811009b4a0 con 0x558be5844b30 2024-05-09T08:36:09.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.603+0000 7f80fbfff640 1 -- 172.21.15.107:0/3687496089 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f81080047a0 con 0x558be5844b30 2024-05-09T08:36:09.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.603+0000 7f812600a740 10 client.25945 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:09.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.603+0000 7f812600a740 10 client.25945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:09.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.603+0000 7f812600a740 20 client.25945 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:09.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.603+0000 7f812600a740 1 --2- 172.21.15.107:0/3687496089 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558be5863b10 0x558be5865f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.603+0000 7f812600a740 4 client.25945 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:09.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.603+0000 7f812600a740 1 -- 172.21.15.107:0/3687496089 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558be57b7370 con 0x558be5863b10 2024-05-09T08:36:09.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.604+0000 7f8120fb8640 1 --2- 172.21.15.107:0/3687496089 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558be5863b10 0x558be5865f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.604+0000 7f8120fb8640 1 --2- 172.21.15.107:0/3687496089 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558be5863b10 0x558be5865f90 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8dda7db9cfb8889 server_cookie=7d723b45375b7b83 in_seq=0 out_seq=0 2024-05-09T08:36:09.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.604+0000 7f80fbfff640 10 client.25945 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:09.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.605+0000 7f80fbfff640 1 -- 172.21.15.107:0/3687496089 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558be57b7370 con 0x558be5863b10 2024-05-09T08:36:09.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.605+0000 7f80fbfff640 10 client.25945 handle_command_reply: tid=0 2024-05-09T08:36:09.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.605+0000 7f812600a740 1 client.25945 shutdown 2024-05-09T08:36:09.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.605+0000 7f812600a740 1 -- 172.21.15.107:0/3687496089 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 msgr2=0x558be5850b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.605+0000 7f812600a740 1 --2- 172.21.15.107:0/3687496089 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be5850b90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f810802a940 tx=0x7f8108031ef0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.605+0000 7f812600a740 1 -- 172.21.15.107:0/3687496089 shutdown_connections 2024-05-09T08:36:09.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.605+0000 7f812600a740 1 --2- 172.21.15.107:0/3687496089 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558be5863b10 0x558be5865f90 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.605+0000 7f812600a740 1 --2- 172.21.15.107:0/3687496089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 0x558be58510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.605+0000 7f80fbfff640 0 client.25945 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:09.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.605+0000 7f812600a740 1 --2- 172.21.15.107:0/3687496089 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be5850b90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.605+0000 7f812600a740 1 --2- 172.21.15.107:0/3687496089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5844210 0x558be5850650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.605+0000 7f812600a740 1 -- 172.21.15.107:0/3687496089 >> 172.21.15.107:0/3687496089 conn(0x558be5840650 msgr2=0x558be58463e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.606+0000 7f812600a740 1 -- 172.21.15.107:0/3687496089 shutdown_connections 2024-05-09T08:36:09.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.606+0000 7f812600a740 1 -- 172.21.15.107:0/3687496089 wait complete. 2024-05-09T08:36:09.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.606+0000 7f812600a740 20 client.25945 trim_cache size 0 max 16384 2024-05-09T08:36:09.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.621+0000 7f812600a740 1 Processor -- start 2024-05-09T08:36:09.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.621+0000 7f812600a740 1 -- start start 2024-05-09T08:36:09.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f812600a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5844210 0x558be585f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f812600a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be584f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f812600a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5853730 0x558be583f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f812600a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f811c1af180 con 0x558be5844210 2024-05-09T08:36:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f812600a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f811c1aca50 con 0x558be5853730 2024-05-09T08:36:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f812600a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f811c1aa8a0 con 0x558be5844b30 2024-05-09T08:36:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f8120fb8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5844210 0x558be585f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f8120fb8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5844210 0x558be585f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57772/0 (socket says 172.21.15.107:57772) 2024-05-09T08:36:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f8120fb8640 1 -- 172.21.15.107:0/1859112464 learned_addr learned my addr 172.21.15.107:0/1859112464 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f811bfff640 1 --2- 172.21.15.107:0/1859112464 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be584f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f81217b9640 1 --2- 172.21.15.107:0/1859112464 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5853730 0x558be583f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f8120fb8640 1 -- 172.21.15.107:0/1859112464 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 msgr2=0x558be584f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f8120fb8640 1 --2- 172.21.15.107:0/1859112464 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be584f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f8120fb8640 1 -- 172.21.15.107:0/1859112464 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5853730 msgr2=0x558be583f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f8120fb8640 1 --2- 172.21.15.107:0/1859112464 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5853730 0x558be583f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f8120fb8640 1 -- 172.21.15.107:0/1859112464 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558be5831d20 con 0x558be5844210 2024-05-09T08:36:09.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f81217b9640 1 --2- 172.21.15.107:0/1859112464 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5853730 0x558be583f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:09.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f8120fb8640 1 --2- 172.21.15.107:0/1859112464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5844210 0x558be585f3e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f81100cc6d0 tx=0x7f811009b100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e2db8a67ef168e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f81067fc640 1 -- 172.21.15.107:0/1859112464 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110009de0 con 0x558be5844210 2024-05-09T08:36:09.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f81067fc640 1 -- 172.21.15.107:0/1859112464 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f811009f7a0 con 0x558be5844210 2024-05-09T08:36:09.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.622+0000 7f81067fc640 1 -- 172.21.15.107:0/1859112464 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81100971f0 con 0x558be5844210 2024-05-09T08:36:09.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.623+0000 7f812600a740 1 -- 172.21.15.107:0/1859112464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5844210 msgr2=0x558be585f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.623+0000 7f812600a740 1 --2- 172.21.15.107:0/1859112464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5844210 0x558be585f3e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f81100cc6d0 tx=0x7f811009b100 comp rx=0 tx=0).stop 2024-05-09T08:36:09.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.623+0000 7f812600a740 1 -- 172.21.15.107:0/1859112464 shutdown_connections 2024-05-09T08:36:09.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.623+0000 7f812600a740 1 --2- 172.21.15.107:0/1859112464 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5853730 0x558be583f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.623+0000 7f812600a740 1 --2- 172.21.15.107:0/1859112464 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be584f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.623+0000 7f812600a740 1 --2- 172.21.15.107:0/1859112464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5844210 0x558be585f3e0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.623+0000 7f812600a740 1 -- 172.21.15.107:0/1859112464 >> 172.21.15.107:0/1859112464 conn(0x558be5840650 msgr2=0x558be583ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.623+0000 7f812600a740 1 -- 172.21.15.107:0/1859112464 shutdown_connections 2024-05-09T08:36:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.623+0000 7f812600a740 1 -- 172.21.15.107:0/1859112464 wait complete. 2024-05-09T08:36:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f812600a740 1 -- start start 2024-05-09T08:36:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f812600a740 1 Processor -- start 2024-05-09T08:36:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f812600a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5844210 0x558be5857210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f812600a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be5857750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f812600a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 0x558be5857c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f812600a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f811c1aaf10 con 0x558be5853730 2024-05-09T08:36:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f812600a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f811c1ac380 con 0x558be5844210 2024-05-09T08:36:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f812600a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f811c1ab5e0 con 0x558be5844b30 2024-05-09T08:36:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f812600a740 10 client.0 _ll_register_callbacks cb 0x7f8108031b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f81217b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 0x558be5857c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f81217b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 0x558be5857c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57776/0 (socket says 172.21.15.107:57776) 2024-05-09T08:36:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f81217b9640 1 -- 172.21.15.107:0/2245974915 learned_addr learned my addr 172.21.15.107:0/2245974915 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f8120fb8640 1 --2- 172.21.15.107:0/2245974915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5844210 0x558be5857210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f811bfff640 1 --2- 172.21.15.107:0/2245974915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be5857750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f81217b9640 1 -- 172.21.15.107:0/2245974915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 msgr2=0x558be5857750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f81217b9640 1 --2- 172.21.15.107:0/2245974915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be5857750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f81217b9640 1 -- 172.21.15.107:0/2245974915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5844210 msgr2=0x558be5857210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:09.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.624+0000 7f81217b9640 1 --2- 172.21.15.107:0/2245974915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5844210 0x558be5857210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.625+0000 7f81217b9640 1 -- 172.21.15.107:0/2245974915 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558be58c9110 con 0x558be5853730 2024-05-09T08:36:09.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.625+0000 7f81217b9640 1 --2- 172.21.15.107:0/2245974915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 0x558be5857c90 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f8114003ab0 tx=0x7f8114004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d86fa021af5f95a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:09.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.625+0000 7f80fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:09.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.625+0000 7f80fbfff640 1 -- 172.21.15.107:0/2245974915 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8114030060 con 0x558be5853730 2024-05-09T08:36:09.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.625+0000 7f80fbfff640 1 -- 172.21.15.107:0/2245974915 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f81140045b0 con 0x558be5853730 2024-05-09T08:36:09.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.625+0000 7f812600a740 1 -- 172.21.15.107:0/2245974915 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f811009f7a0 con 0x558be5853730 2024-05-09T08:36:09.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.625+0000 7f80fbfff640 1 -- 172.21.15.107:0/2245974915 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8114037050 con 0x558be5853730 2024-05-09T08:36:09.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.625+0000 7f812600a740 10 client.16509 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:09.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.625+0000 7f812600a740 1 -- 172.21.15.107:0/2245974915 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558be58ca370 con 0x558be5853730 2024-05-09T08:36:09.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.625+0000 7f80fbfff640 1 -- 172.21.15.107:0/2245974915 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f811009f7a0 con 0x558be5853730 2024-05-09T08:36:09.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.626+0000 7f80fbfff640 1 -- 172.21.15.107:0/2245974915 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f811401d040 con 0x558be5853730 2024-05-09T08:36:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.626+0000 7f812600a740 10 client.16509 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.626+0000 7f812600a740 10 client.16509 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.626+0000 7f812600a740 20 client.16509 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.626+0000 7f812600a740 1 --2- 172.21.15.107:0/2245974915 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558be58cab80 0x558be58ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.626+0000 7f812600a740 4 client.16509 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:36:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.626+0000 7f812600a740 1 -- 172.21.15.107:0/2245974915 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558be57b16c0 con 0x558be58cab80 2024-05-09T08:36:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.626+0000 7f8120fb8640 1 --2- 172.21.15.107:0/2245974915 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558be58cab80 0x558be58ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.626+0000 7f80fbfff640 10 client.16509 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.626+0000 7f8120fb8640 1 --2- 172.21.15.107:0/2245974915 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558be58cab80 0x558be58ccfa0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8924130b1a6bc46 server_cookie=54ea4f32f2435b71 in_seq=0 out_seq=0 2024-05-09T08:36:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.626+0000 7f80fbfff640 1 -- 172.21.15.107:0/2245974915 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558be57b16c0 con 0x558be58cab80 2024-05-09T08:36:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.626+0000 7f80fbfff640 10 client.16509 handle_command_reply: tid=0 2024-05-09T08:36:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.626+0000 7f812600a740 1 client.16509 shutdown 2024-05-09T08:36:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812600a740 1 -- 172.21.15.107:0/2245974915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 msgr2=0x558be5857c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812600a740 1 --2- 172.21.15.107:0/2245974915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 0x558be5857c90 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f8114003ab0 tx=0x7f8114004a40 comp rx=0 tx=0).stop 2024-05-09T08:36:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812600a740 1 -- 172.21.15.107:0/2245974915 shutdown_connections 2024-05-09T08:36:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812600a740 1 --2- 172.21.15.107:0/2245974915 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558be58cab80 0x558be58ccfa0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812600a740 1 --2- 172.21.15.107:0/2245974915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558be5853730 0x558be5857c90 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f80fbfff640 0 client.16509 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812600a740 1 --2- 172.21.15.107:0/2245974915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558be5844b30 0x558be5857750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812600a740 1 --2- 172.21.15.107:0/2245974915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558be5844210 0x558be5857210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812600a740 1 -- 172.21.15.107:0/2245974915 >> 172.21.15.107:0/2245974915 conn(0x558be5840650 msgr2=0x558be583f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812600a740 1 -- 172.21.15.107:0/2245974915 shutdown_connections 2024-05-09T08:36:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812600a740 1 -- 172.21.15.107:0/2245974915 wait complete. 2024-05-09T08:36:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812600a740 20 client.16509 trim_cache size 0 max 16384 2024-05-09T08:36:09.634 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:36:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812321c640 1 -- 172.21.15.107:0/1995983961 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f810c0a2d80 msgr2=0x7f810c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812321c640 1 --2- 172.21.15.107:0/1995983961 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f810c0a2d80 0x7f810c0a5240 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8114005c40 tx=0x7f8114009500 comp rx=0 tx=0).stop 2024-05-09T08:36:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812321c640 1 -- 172.21.15.107:0/1995983961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811c100700 msgr2=0x7f811c198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812321c640 1 --2- 172.21.15.107:0/1995983961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811c100700 0x7f811c198050 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f811000d950 tx=0x7f811000de10 comp rx=0 tx=0).stop 2024-05-09T08:36:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812321c640 1 -- 172.21.15.107:0/1995983961 shutdown_connections 2024-05-09T08:36:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812321c640 1 --2- 172.21.15.107:0/1995983961 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f810c0a2d80 0x7f810c0a5240 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812321c640 1 --2- 172.21.15.107:0/1995983961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f811c106710 0x7f811c19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812321c640 1 --2- 172.21.15.107:0/1995983961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f811c1010c0 0x7f811c198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812321c640 1 --2- 172.21.15.107:0/1995983961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f811c100700 0x7f811c198050 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:09.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.627+0000 7f812321c640 1 -- 172.21.15.107:0/1995983961 >> 172.21.15.107:0/1995983961 conn(0x7f811c077830 msgr2=0x7f811c068da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:09.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.628+0000 7f812321c640 1 -- 172.21.15.107:0/1995983961 shutdown_connections 2024-05-09T08:36:09.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:09.628+0000 7f812321c640 1 -- 172.21.15.107:0/1995983961 wait complete. 2024-05-09T08:36:10.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:09 smithi107 ceph-mon[30122]: pgmap v87: 129 pgs: 129 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 25 MiB/s wr, 30 op/s 2024-05-09T08:36:10.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:09 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1057786081' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:36:10.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:09 smithi157 ceph-mon[33614]: pgmap v87: 129 pgs: 129 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 25 MiB/s wr, 30 op/s 2024-05-09T08:36:10.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:09 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1057786081' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:36:10.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:09 smithi110 ceph-mon[33622]: pgmap v87: 129 pgs: 129 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 25 MiB/s wr, 30 op/s 2024-05-09T08:36:10.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:09 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1057786081' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:36:10.639 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:36:10.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:36:10.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:36:10.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.758+0000 7f698df10640 1 -- 172.21.15.107:0/4127775017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6988107820 msgr2=0x7f6988107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.758+0000 7f698df10640 1 --2- 172.21.15.107:0/4127775017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6988107820 0x7f6988107c00 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6978009930 tx=0x7f697802a420 comp rx=0 tx=0).stop 2024-05-09T08:36:10.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.758+0000 7f698df10640 1 -- 172.21.15.107:0/4127775017 shutdown_connections 2024-05-09T08:36:10.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.758+0000 7f698df10640 1 --2- 172.21.15.107:0/4127775017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69881021d0 0x7f698810e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.758+0000 7f698df10640 1 --2- 172.21.15.107:0/4127775017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6988101810 0x7f6988101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.758+0000 7f698df10640 1 --2- 172.21.15.107:0/4127775017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6988107820 0x7f6988107c00 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.758+0000 7f698df10640 1 -- 172.21.15.107:0/4127775017 >> 172.21.15.107:0/4127775017 conn(0x7f6988078c30 msgr2=0x7f6988079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:10.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.758+0000 7f698df10640 1 -- 172.21.15.107:0/4127775017 shutdown_connections 2024-05-09T08:36:10.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.758+0000 7f698df10640 1 -- 172.21.15.107:0/4127775017 wait complete. 2024-05-09T08:36:10.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.759+0000 7f698df10640 1 Processor -- start 2024-05-09T08:36:10.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.759+0000 7f698df10640 1 -- start start 2024-05-09T08:36:10.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.759+0000 7f698df10640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6988101810 0x7f6988198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.759+0000 7f698df10640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69881021d0 0x7f6988198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.759+0000 7f698df10640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6988107820 0x7f698819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.759+0000 7f698df10640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6988111ae0 con 0x7f6988107820 2024-05-09T08:36:10.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.759+0000 7f698df10640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6988111960 con 0x7f6988101810 2024-05-09T08:36:10.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.759+0000 7f698df10640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6988111c60 con 0x7f69881021d0 2024-05-09T08:36:10.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.759+0000 7f6987fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6988107820 0x7f698819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.759+0000 7f6987fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6988107820 0x7f698819c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47480/0 (socket says 172.21.15.107:47480) 2024-05-09T08:36:10.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.759+0000 7f6987fff640 1 -- 172.21.15.107:0/3391324654 learned_addr learned my addr 172.21.15.107:0/3391324654 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:10.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.760+0000 7f6986ffd640 1 --2- 172.21.15.107:0/3391324654 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69881021d0 0x7f6988198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.760+0000 7f69877fe640 1 --2- 172.21.15.107:0/3391324654 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6988101810 0x7f6988198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.760+0000 7f6987fff640 1 -- 172.21.15.107:0/3391324654 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69881021d0 msgr2=0x7f6988198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.760+0000 7f6987fff640 1 --2- 172.21.15.107:0/3391324654 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69881021d0 0x7f6988198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.760+0000 7f6987fff640 1 -- 172.21.15.107:0/3391324654 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6988101810 msgr2=0x7f6988198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.760+0000 7f6987fff640 1 --2- 172.21.15.107:0/3391324654 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6988101810 0x7f6988198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.760+0000 7f6987fff640 1 -- 172.21.15.107:0/3391324654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698819d030 con 0x7f6988107820 2024-05-09T08:36:10.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.760+0000 7f6987fff640 1 --2- 172.21.15.107:0/3391324654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6988107820 0x7f698819c900 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f698000eb10 tx=0x7f698000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:10.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.760+0000 7f6984ff9640 1 -- 172.21.15.107:0/3391324654 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6980004b20 con 0x7f6988107820 2024-05-09T08:36:10.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.760+0000 7f698df10640 1 -- 172.21.15.107:0/3391324654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f698819d320 con 0x7f6988107820 2024-05-09T08:36:10.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.760+0000 7f6984ff9640 1 -- 172.21.15.107:0/3391324654 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6980004cc0 con 0x7f6988107820 2024-05-09T08:36:10.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.760+0000 7f6984ff9640 1 -- 172.21.15.107:0/3391324654 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6980010690 con 0x7f6988107820 2024-05-09T08:36:10.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.760+0000 7f698df10640 1 -- 172.21.15.107:0/3391324654 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69881a4bb0 con 0x7f6988107820 2024-05-09T08:36:10.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.762+0000 7f6984ff9640 1 -- 172.21.15.107:0/3391324654 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6980005020 con 0x7f6988107820 2024-05-09T08:36:10.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f6990c1d740 1 Processor -- start 2024-05-09T08:36:10.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f6990c1d740 1 -- start start 2024-05-09T08:36:10.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f6984ff9640 1 --2- 172.21.15.107:0/3391324654 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f69700a2d00 0x7f69700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f6990c1d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752d10b0 0x55a8752cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f6990c1d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752cf9c0 0x55a8752cfe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f6990c1d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752deb50 0x55a8752e0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f6990c1d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a875234990 con 0x55a8752deb50 2024-05-09T08:36:10.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f6990c1d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a8751dc5b0 con 0x55a8752d10b0 2024-05-09T08:36:10.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f6990c1d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a875230950 con 0x55a8752cf9c0 2024-05-09T08:36:10.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f6986ffd640 1 --2- 172.21.15.107:0/3391324654 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f69700a2d00 0x7f69700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f6984ff9640 1 -- 172.21.15.107:0/3391324654 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f69800cfeb0 con 0x7f6988107820 2024-05-09T08:36:10.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f6987fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752deb50 0x55a8752e0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f69877fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752d10b0 0x55a8752cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f6987fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752deb50 0x55a8752e0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47494/0 (socket says 172.21.15.107:47494) 2024-05-09T08:36:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f6987fff640 1 -- 172.21.15.107:0/1703112453 learned_addr learned my addr 172.21.15.107:0/1703112453 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f69877fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752d10b0 0x55a8752cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:51884/0 (socket says 172.21.15.107:51884) 2024-05-09T08:36:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.763+0000 7f6986ffd640 1 --2- 172.21.15.107:0/1703112453 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752cf9c0 0x55a8752cfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.764+0000 7f6987fff640 1 -- 172.21.15.107:0/1703112453 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752cf9c0 msgr2=0x55a8752cfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.764+0000 7f6987fff640 1 --2- 172.21.15.107:0/1703112453 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752cf9c0 0x55a8752cfe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.764+0000 7f6987fff640 1 -- 172.21.15.107:0/1703112453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752d10b0 msgr2=0x55a8752cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.764+0000 7f6987fff640 1 --2- 172.21.15.107:0/1703112453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752d10b0 0x55a8752cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.764+0000 7f6987fff640 1 -- 172.21.15.107:0/1703112453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8750107e0 con 0x55a8752deb50 2024-05-09T08:36:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.764+0000 7f6986ffd640 1 --2- 172.21.15.107:0/1703112453 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752cf9c0 0x55a8752cfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.764+0000 7f6986ffd640 1 --2- 172.21.15.107:0/3391324654 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f69700a2d00 0x7f69700a51c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f697c00b620 tx=0x7f697c012810 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.764+0000 7f6987fff640 1 --2- 172.21.15.107:0/1703112453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752deb50 0x55a8752e0f40 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f69800a47b0 tx=0x7f698009bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef6326586f4c26c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.764+0000 7f696dffb640 1 -- 172.21.15.107:0/1703112453 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698009c070 con 0x55a8752deb50 2024-05-09T08:36:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.764+0000 7f696dffb640 1 -- 172.21.15.107:0/1703112453 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69800a0420 con 0x55a8752deb50 2024-05-09T08:36:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.764+0000 7f696dffb640 1 -- 172.21.15.107:0/1703112453 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69800a4850 con 0x55a8752deb50 2024-05-09T08:36:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.765+0000 7f6990c1d740 1 -- 172.21.15.107:0/1703112453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752deb50 msgr2=0x55a8752e0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.765+0000 7f6990c1d740 1 --2- 172.21.15.107:0/1703112453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752deb50 0x55a8752e0f40 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f69800a47b0 tx=0x7f698009bf00 comp rx=0 tx=0).stop 2024-05-09T08:36:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.765+0000 7f6990c1d740 1 -- 172.21.15.107:0/1703112453 shutdown_connections 2024-05-09T08:36:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.765+0000 7f6990c1d740 1 --2- 172.21.15.107:0/1703112453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752deb50 0x55a8752e0f40 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.765+0000 7f6990c1d740 1 --2- 172.21.15.107:0/1703112453 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752cf9c0 0x55a8752cfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.765+0000 7f6990c1d740 1 --2- 172.21.15.107:0/1703112453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752d10b0 0x55a8752cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.765+0000 7f6990c1d740 1 -- 172.21.15.107:0/1703112453 >> 172.21.15.107:0/1703112453 conn(0x55a8752980a0 msgr2=0x55a8752cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.765+0000 7f6990c1d740 1 -- 172.21.15.107:0/1703112453 shutdown_connections 2024-05-09T08:36:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.765+0000 7f6990c1d740 1 -- 172.21.15.107:0/1703112453 wait complete. 2024-05-09T08:36:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.765+0000 7f6990c1d740 1 -- start start 2024-05-09T08:36:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6990c1d740 1 Processor -- start 2024-05-09T08:36:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6990c1d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 0x55a8752db9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6990c1d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a8752dbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6990c1d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752deb50 0x55a8752dc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6990c1d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a8752e19e0 con 0x55a8752d10b0 2024-05-09T08:36:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6990c1d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a8751efb40 con 0x55a8752cf9c0 2024-05-09T08:36:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6990c1d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a8752e1b60 con 0x55a8752deb50 2024-05-09T08:36:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6990c1d740 10 client.0 _ll_register_callbacks cb 0x55a8751e40f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6986ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a8752dbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6986ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a8752dbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47498/0 (socket says 172.21.15.107:47498) 2024-05-09T08:36:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6986ffd640 1 -- 172.21.15.107:0/3160623144 learned_addr learned my addr 172.21.15.107:0/3160623144 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f69877fe640 1 --2- 172.21.15.107:0/3160623144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 0x55a8752db9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6987fff640 1 --2- 172.21.15.107:0/3160623144 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752deb50 0x55a8752dc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6986ffd640 1 -- 172.21.15.107:0/3160623144 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752deb50 msgr2=0x55a8752dc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6986ffd640 1 --2- 172.21.15.107:0/3160623144 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752deb50 0x55a8752dc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6986ffd640 1 -- 172.21.15.107:0/3160623144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 msgr2=0x55a8752db9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6986ffd640 1 --2- 172.21.15.107:0/3160623144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 0x55a8752db9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6986ffd640 1 -- 172.21.15.107:0/3160623144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a875070f70 con 0x55a8752d10b0 2024-05-09T08:36:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f6987fff640 1 --2- 172.21.15.107:0/3160623144 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752deb50 0x55a8752dc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:36:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.766+0000 7f69877fe640 1 --2- 172.21.15.107:0/3160623144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 0x55a8752db9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.767+0000 7f6986ffd640 1 --2- 172.21.15.107:0/3160623144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a8752dbf00 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f697c010740 tx=0x7f697c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c883666736dae8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.767+0000 7f6966ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.767+0000 7f6966ffd640 1 -- 172.21.15.107:0/3160623144 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c025070 con 0x55a8752d10b0 2024-05-09T08:36:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.767+0000 7f6966ffd640 1 -- 172.21.15.107:0/3160623144 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f697c01ca80 con 0x55a8752d10b0 2024-05-09T08:36:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.767+0000 7f6990c1d740 1 -- 172.21.15.107:0/3160623144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69800a0420 con 0x55a8752d10b0 2024-05-09T08:36:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.767+0000 7f6966ffd640 1 -- 172.21.15.107:0/3160623144 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f697c031050 con 0x55a8752d10b0 2024-05-09T08:36:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.767+0000 7f6990c1d740 10 client.16521 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.767+0000 7f6990c1d740 1 -- 172.21.15.107:0/3160623144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a874f08170 con 0x55a8752d10b0 2024-05-09T08:36:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.767+0000 7f6966ffd640 1 -- 172.21.15.107:0/3160623144 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69800a0420 con 0x55a8752d10b0 2024-05-09T08:36:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.767+0000 7f6966ffd640 1 -- 172.21.15.107:0/3160623144 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f697c01e070 con 0x55a8752d10b0 2024-05-09T08:36:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.767+0000 7f6990c1d740 10 client.16521 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.767+0000 7f6990c1d740 10 client.16521 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.767+0000 7f6990c1d740 20 client.16521 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.767+0000 7f6990c1d740 1 --2- 172.21.15.107:0/3160623144 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a8752eeb70 0x55a8752f1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.767+0000 7f6990c1d740 4 client.16521 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.767+0000 7f6990c1d740 1 -- 172.21.15.107:0/3160623144 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8752420c0 con 0x55a8752eeb70 2024-05-09T08:36:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.768+0000 7f69877fe640 1 --2- 172.21.15.107:0/3160623144 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a8752eeb70 0x55a8752f1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.768+0000 7f69877fe640 1 --2- 172.21.15.107:0/3160623144 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a8752eeb70 0x55a8752f1010 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b30d2a9b7e9ea84d server_cookie=66e154fa0ba0300a in_seq=0 out_seq=0 2024-05-09T08:36:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.768+0000 7f6966ffd640 10 client.16521 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.769+0000 7f6966ffd640 1 -- 172.21.15.107:0/3160623144 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a8752420c0 con 0x55a8752eeb70 2024-05-09T08:36:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.769+0000 7f6966ffd640 10 client.16521 handle_command_reply: tid=0 2024-05-09T08:36:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.769+0000 7f6990c1d740 1 client.16521 shutdown 2024-05-09T08:36:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.769+0000 7f6990c1d740 1 -- 172.21.15.107:0/3160623144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 msgr2=0x55a8752dbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.769+0000 7f6990c1d740 1 --2- 172.21.15.107:0/3160623144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a8752dbf00 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f697c010740 tx=0x7f697c004730 comp rx=0 tx=0).stop 2024-05-09T08:36:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.769+0000 7f6990c1d740 1 -- 172.21.15.107:0/3160623144 shutdown_connections 2024-05-09T08:36:10.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.769+0000 7f6990c1d740 1 --2- 172.21.15.107:0/3160623144 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a8752eeb70 0x55a8752f1010 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.769+0000 7f6990c1d740 1 --2- 172.21.15.107:0/3160623144 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752deb50 0x55a8752dc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.769+0000 7f6990c1d740 1 --2- 172.21.15.107:0/3160623144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a8752dbf00 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.769+0000 7f6990c1d740 1 --2- 172.21.15.107:0/3160623144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 0x55a8752db9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.769+0000 7f6990c1d740 1 -- 172.21.15.107:0/3160623144 >> 172.21.15.107:0/3160623144 conn(0x55a8752980a0 msgr2=0x55a8752d1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:10.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.769+0000 7f6966ffd640 0 client.16521 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:10.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.769+0000 7f6990c1d740 1 -- 172.21.15.107:0/3160623144 shutdown_connections 2024-05-09T08:36:10.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.769+0000 7f6990c1d740 1 -- 172.21.15.107:0/3160623144 wait complete. 2024-05-09T08:36:10.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.769+0000 7f6990c1d740 20 client.16521 trim_cache size 0 max 16384 2024-05-09T08:36:10.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.789+0000 7f6990c1d740 1 Processor -- start 2024-05-09T08:36:10.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.789+0000 7f6990c1d740 1 -- start start 2024-05-09T08:36:10.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f6990c1d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 0x55a8752ea910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f6990c1d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a8752dab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f6990c1d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752deb50 0x55a8752dd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f6990c1d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f69881af290 con 0x55a8752d10b0 2024-05-09T08:36:10.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f6990c1d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f69881acb60 con 0x55a8752cf9c0 2024-05-09T08:36:10.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f6990c1d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f69881aa9b0 con 0x55a8752deb50 2024-05-09T08:36:10.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f6986ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a8752dab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f6986ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a8752dab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47510/0 (socket says 172.21.15.107:47510) 2024-05-09T08:36:10.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f6986ffd640 1 -- 172.21.15.107:0/3276130117 learned_addr learned my addr 172.21.15.107:0/3276130117 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:10.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f69877fe640 1 --2- 172.21.15.107:0/3276130117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 0x55a8752ea910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f6987fff640 1 --2- 172.21.15.107:0/3276130117 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752deb50 0x55a8752dd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f6986ffd640 1 -- 172.21.15.107:0/3276130117 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752deb50 msgr2=0x55a8752dd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:10.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f6986ffd640 1 --2- 172.21.15.107:0/3276130117 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752deb50 0x55a8752dd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f6986ffd640 1 -- 172.21.15.107:0/3276130117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 msgr2=0x55a8752ea910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:10.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f6986ffd640 1 --2- 172.21.15.107:0/3276130117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 0x55a8752ea910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f6986ffd640 1 -- 172.21.15.107:0/3276130117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8752cc130 con 0x55a8752d10b0 2024-05-09T08:36:10.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.790+0000 7f6986ffd640 1 --2- 172.21.15.107:0/3276130117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a8752dab20 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f697c010710 tx=0x7f697c0046a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e486874ce74a6a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:10.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.791+0000 7f696d7fa640 1 -- 172.21.15.107:0/3276130117 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c03a070 con 0x55a8752d10b0 2024-05-09T08:36:10.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.791+0000 7f696d7fa640 1 -- 172.21.15.107:0/3276130117 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f697c01cd30 con 0x55a8752d10b0 2024-05-09T08:36:10.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.791+0000 7f696d7fa640 1 -- 172.21.15.107:0/3276130117 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c025040 con 0x55a8752d10b0 2024-05-09T08:36:10.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.791+0000 7f6990c1d740 1 -- 172.21.15.107:0/3276130117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 msgr2=0x55a8752dab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.791+0000 7f6990c1d740 1 --2- 172.21.15.107:0/3276130117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a8752dab20 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f697c010710 tx=0x7f697c0046a0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.791+0000 7f6990c1d740 1 -- 172.21.15.107:0/3276130117 shutdown_connections 2024-05-09T08:36:10.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.791+0000 7f6990c1d740 1 --2- 172.21.15.107:0/3276130117 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752deb50 0x55a8752dd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.791+0000 7f6990c1d740 1 --2- 172.21.15.107:0/3276130117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a8752dab20 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.791+0000 7f6990c1d740 1 --2- 172.21.15.107:0/3276130117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 0x55a8752ea910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.791+0000 7f6990c1d740 1 -- 172.21.15.107:0/3276130117 >> 172.21.15.107:0/3276130117 conn(0x55a8752980a0 msgr2=0x55a8752dd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:10.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.791+0000 7f6990c1d740 1 -- 172.21.15.107:0/3276130117 shutdown_connections 2024-05-09T08:36:10.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.791+0000 7f6990c1d740 1 -- 172.21.15.107:0/3276130117 wait complete. 2024-05-09T08:36:10.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.792+0000 7f6990c1d740 1 -- start start 2024-05-09T08:36:10.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.792+0000 7f6990c1d740 1 Processor -- start 2024-05-09T08:36:10.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.792+0000 7f6990c1d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 0x55a87534e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.792+0000 7f6990c1d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a87534ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.792+0000 7f6990c1d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752deb50 0x55a8752e2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.792+0000 7f6990c1d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f69881ab020 con 0x55a8752d10b0 2024-05-09T08:36:10.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.792+0000 7f6990c1d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f69881ac490 con 0x55a8752cf9c0 2024-05-09T08:36:10.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.792+0000 7f6990c1d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f69881ab6f0 con 0x55a8752deb50 2024-05-09T08:36:10.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.792+0000 7f6990c1d740 10 client.0 _ll_register_callbacks cb 0x7f697c010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:10.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.792+0000 7f6986ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a87534ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.792+0000 7f69877fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 0x55a87534e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.792+0000 7f6986ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a87534ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47526/0 (socket says 172.21.15.107:47526) 2024-05-09T08:36:10.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.792+0000 7f6986ffd640 1 -- 172.21.15.107:0/3406372105 learned_addr learned my addr 172.21.15.107:0/3406372105 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:10.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.792+0000 7f6987fff640 1 --2- 172.21.15.107:0/3406372105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752deb50 0x55a8752e2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.793+0000 7f69877fe640 1 -- 172.21.15.107:0/3406372105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752deb50 msgr2=0x55a8752e2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:10.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.793+0000 7f69877fe640 1 --2- 172.21.15.107:0/3406372105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752deb50 0x55a8752e2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.793+0000 7f69877fe640 1 -- 172.21.15.107:0/3406372105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 msgr2=0x55a87534ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:10.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.793+0000 7f69877fe640 1 --2- 172.21.15.107:0/3406372105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a87534ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.793+0000 7f69877fe640 1 -- 172.21.15.107:0/3406372105 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8752e3660 con 0x55a8752cf9c0 2024-05-09T08:36:10.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.793+0000 7f6986ffd640 1 --2- 172.21.15.107:0/3406372105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a87534ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:10.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.793+0000 7f69877fe640 1 --2- 172.21.15.107:0/3406372105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 0x55a87534e4f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6978009900 tx=0x7f6978004120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29a9fc7c87575f86 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:10.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.793+0000 7f6966ffd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:36:10.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.793+0000 7f6966ffd640 1 -- 172.21.15.107:0/3406372105 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6978048070 con 0x55a8752cf9c0 2024-05-09T08:36:10.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.793+0000 7f6990c1d740 1 -- 172.21.15.107:0/3406372105 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f697c01cd30 con 0x55a8752cf9c0 2024-05-09T08:36:10.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.793+0000 7f6966ffd640 1 -- 172.21.15.107:0/3406372105 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6978004210 con 0x55a8752cf9c0 2024-05-09T08:36:10.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.793+0000 7f6966ffd640 1 -- 172.21.15.107:0/3406372105 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6978051050 con 0x55a8752cf9c0 2024-05-09T08:36:10.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.793+0000 7f6990c1d740 10 client.25793 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:10.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.793+0000 7f6990c1d740 1 -- 172.21.15.107:0/3406372105 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a875355d10 con 0x55a8752cf9c0 2024-05-09T08:36:10.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.794+0000 7f6966ffd640 1 -- 172.21.15.107:0/3406372105 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f697c01cd30 con 0x55a8752cf9c0 2024-05-09T08:36:10.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.794+0000 7f6966ffd640 1 -- 172.21.15.107:0/3406372105 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f697804b020 con 0x55a8752cf9c0 2024-05-09T08:36:10.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.794+0000 7f6990c1d740 10 client.25793 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:10.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.794+0000 7f6990c1d740 10 client.25793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:10.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.794+0000 7f6990c1d740 20 client.25793 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:10.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.794+0000 7f6990c1d740 1 --2- 172.21.15.107:0/3406372105 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a875356490 0x55a8753588b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.794+0000 7f6990c1d740 4 client.25793 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:36:10.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.794+0000 7f6990c1d740 1 -- 172.21.15.107:0/3406372105 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a87523c410 con 0x55a875356490 2024-05-09T08:36:10.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.794+0000 7f6986ffd640 1 --2- 172.21.15.107:0/3406372105 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a875356490 0x55a8753588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.795+0000 7f6966ffd640 10 client.25793 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:10.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.795+0000 7f6986ffd640 1 --2- 172.21.15.107:0/3406372105 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a875356490 0x55a8753588b0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c87db772c939fcbd server_cookie=1856d0da75f88947 in_seq=0 out_seq=0 2024-05-09T08:36:10.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.796+0000 7f6966ffd640 1 -- 172.21.15.107:0/3406372105 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a87523c410 con 0x55a875356490 2024-05-09T08:36:10.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.796+0000 7f6966ffd640 10 client.25793 handle_command_reply: tid=0 2024-05-09T08:36:10.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.796+0000 7f6990c1d740 1 client.25793 shutdown 2024-05-09T08:36:10.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.796+0000 7f6990c1d740 1 -- 172.21.15.107:0/3406372105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 msgr2=0x55a87534e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.796+0000 7f6990c1d740 1 --2- 172.21.15.107:0/3406372105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 0x55a87534e4f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6978009900 tx=0x7f6978004120 comp rx=0 tx=0).stop 2024-05-09T08:36:10.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.796+0000 7f6990c1d740 1 -- 172.21.15.107:0/3406372105 shutdown_connections 2024-05-09T08:36:10.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.796+0000 7f6990c1d740 1 --2- 172.21.15.107:0/3406372105 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a875356490 0x55a8753588b0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.796+0000 7f6990c1d740 1 --2- 172.21.15.107:0/3406372105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a8752deb50 0x55a8752e2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.796+0000 7f6990c1d740 1 --2- 172.21.15.107:0/3406372105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a8752d10b0 0x55a87534ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.796+0000 7f6966ffd640 0 client.25793 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:10.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.796+0000 7f6990c1d740 1 --2- 172.21.15.107:0/3406372105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8752cf9c0 0x55a87534e4f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.796+0000 7f6990c1d740 1 -- 172.21.15.107:0/3406372105 >> 172.21.15.107:0/3406372105 conn(0x55a8752980a0 msgr2=0x55a8752db980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:10.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.796+0000 7f6990c1d740 1 -- 172.21.15.107:0/3406372105 shutdown_connections 2024-05-09T08:36:10.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.796+0000 7f6990c1d740 1 -- 172.21.15.107:0/3406372105 wait complete. 2024-05-09T08:36:10.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.796+0000 7f6990c1d740 20 client.25793 trim_cache size 0 max 16384 2024-05-09T08:36:10.800 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:36:10.800 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:36:10.800 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "ae7c21ba-dccb-4450-9568-e7a0294fb1bd", 2024-05-09T08:36:10.800 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:36:10.800 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:36:10.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.797+0000 7f698df10640 1 -- 172.21.15.107:0/3391324654 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f69700a2d00 msgr2=0x7f69700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.797+0000 7f698df10640 1 --2- 172.21.15.107:0/3391324654 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f69700a2d00 0x7f69700a51c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f697c00b620 tx=0x7f697c012810 comp rx=0 tx=0).stop 2024-05-09T08:36:10.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.797+0000 7f698df10640 1 -- 172.21.15.107:0/3391324654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6988107820 msgr2=0x7f698819c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.797+0000 7f698df10640 1 --2- 172.21.15.107:0/3391324654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6988107820 0x7f698819c900 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f698000eb10 tx=0x7f698000c2d0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.797+0000 7f698df10640 1 -- 172.21.15.107:0/3391324654 shutdown_connections 2024-05-09T08:36:10.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.797+0000 7f698df10640 1 --2- 172.21.15.107:0/3391324654 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f69700a2d00 0x7f69700a51c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.797+0000 7f698df10640 1 --2- 172.21.15.107:0/3391324654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6988107820 0x7f698819c900 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.797+0000 7f698df10640 1 --2- 172.21.15.107:0/3391324654 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69881021d0 0x7f6988198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.797+0000 7f698df10640 1 --2- 172.21.15.107:0/3391324654 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6988101810 0x7f6988198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.797+0000 7f698df10640 1 -- 172.21.15.107:0/3391324654 >> 172.21.15.107:0/3391324654 conn(0x7f6988078c30 msgr2=0x7f69881043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:10.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.797+0000 7f698df10640 1 -- 172.21.15.107:0/3391324654 shutdown_connections 2024-05-09T08:36:10.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.797+0000 7f698df10640 1 -- 172.21.15.107:0/3391324654 wait complete. 2024-05-09T08:36:10.808 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:36:10.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:36:10.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.929+0000 7f40d9efc640 1 -- 172.21.15.107:0/2486351799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40d4100700 msgr2=0x7f40d4100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.929+0000 7f40d9efc640 1 --2- 172.21.15.107:0/2486351799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40d4100700 0x7f40d4100b80 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f40c8009930 tx=0x7f40c802a420 comp rx=0 tx=0).stop 2024-05-09T08:36:10.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.929+0000 7f40d9efc640 1 -- 172.21.15.107:0/2486351799 shutdown_connections 2024-05-09T08:36:10.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.929+0000 7f40d9efc640 1 --2- 172.21.15.107:0/2486351799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d41010c0 0x7f40d410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.929+0000 7f40d9efc640 1 --2- 172.21.15.107:0/2486351799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40d4100700 0x7f40d4100b80 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.929+0000 7f40d9efc640 1 --2- 172.21.15.107:0/2486351799 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40d4106710 0x7f40d4106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.929+0000 7f40d9efc640 1 -- 172.21.15.107:0/2486351799 >> 172.21.15.107:0/2486351799 conn(0x7f40d4077830 msgr2=0x7f40d4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:10.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.929+0000 7f40d9efc640 1 -- 172.21.15.107:0/2486351799 shutdown_connections 2024-05-09T08:36:10.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.929+0000 7f40d9efc640 1 -- 172.21.15.107:0/2486351799 wait complete. 2024-05-09T08:36:10.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.930+0000 7f40d9efc640 1 Processor -- start 2024-05-09T08:36:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.930+0000 7f40d9efc640 1 -- start start 2024-05-09T08:36:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.930+0000 7f40d9efc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40d4100700 0x7f40d4198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.930+0000 7f40d9efc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d41010c0 0x7f40d41986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.930+0000 7f40d9efc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40d4106710 0x7f40d419ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.930+0000 7f40d9efc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f40d4111ac0 con 0x7f40d4100700 2024-05-09T08:36:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.930+0000 7f40d9efc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f40d4111940 con 0x7f40d41010c0 2024-05-09T08:36:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.930+0000 7f40d9efc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f40d4111c40 con 0x7f40d4106710 2024-05-09T08:36:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.930+0000 7f40d37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40d4100700 0x7f40d4198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.930+0000 7f40d37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40d4100700 0x7f40d4198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47550/0 (socket says 172.21.15.107:47550) 2024-05-09T08:36:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.930+0000 7f40d37fe640 1 -- 172.21.15.107:0/3521794938 learned_addr learned my addr 172.21.15.107:0/3521794938 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.930+0000 7f40d3fff640 1 --2- 172.21.15.107:0/3521794938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40d4106710 0x7f40d419ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.931+0000 7f40d2ffd640 1 --2- 172.21.15.107:0/3521794938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d41010c0 0x7f40d41986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.931+0000 7f40d37fe640 1 -- 172.21.15.107:0/3521794938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40d4106710 msgr2=0x7f40d419ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.931+0000 7f40d37fe640 1 --2- 172.21.15.107:0/3521794938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40d4106710 0x7f40d419ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.931+0000 7f40d37fe640 1 -- 172.21.15.107:0/3521794938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d41010c0 msgr2=0x7f40d41986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.931+0000 7f40d37fe640 1 --2- 172.21.15.107:0/3521794938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d41010c0 0x7f40d41986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.931+0000 7f40d37fe640 1 -- 172.21.15.107:0/3521794938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d419d150 con 0x7f40d4100700 2024-05-09T08:36:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.931+0000 7f40d37fe640 1 --2- 172.21.15.107:0/3521794938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40d4100700 0x7f40d4198160 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f40c400e9f0 tx=0x7f40c400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.931+0000 7f40d0ff9640 1 -- 172.21.15.107:0/3521794938 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c400cd80 con 0x7f40d4100700 2024-05-09T08:36:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.931+0000 7f40d9efc640 1 -- 172.21.15.107:0/3521794938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d419d440 con 0x7f40d4100700 2024-05-09T08:36:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.931+0000 7f40d0ff9640 1 -- 172.21.15.107:0/3521794938 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f40c40054d0 con 0x7f40d4100700 2024-05-09T08:36:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.931+0000 7f40d0ff9640 1 -- 172.21.15.107:0/3521794938 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c4010490 con 0x7f40d4100700 2024-05-09T08:36:10.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.931+0000 7f40d9efc640 1 -- 172.21.15.107:0/3521794938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d41a4cd0 con 0x7f40d4100700 2024-05-09T08:36:10.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.934+0000 7f40d0ff9640 1 -- 172.21.15.107:0/3521794938 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f40c4005020 con 0x7f40d4100700 2024-05-09T08:36:10.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.934+0000 7f40dcc07740 1 Processor -- start 2024-05-09T08:36:10.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.934+0000 7f40dcc07740 1 -- start start 2024-05-09T08:36:10.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.934+0000 7f40dcc07740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2e160 0x55f88bb2e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.934+0000 7f40dcc07740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb2ea80 0x55f88bb3d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.934+0000 7f40dcc07740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb3d680 0x55f88bb3fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.934+0000 7f40dcc07740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f88ba93d00 con 0x55f88bb3d680 2024-05-09T08:36:10.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.934+0000 7f40dcc07740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f88ba3b4c0 con 0x55f88bb2e160 2024-05-09T08:36:10.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.934+0000 7f40dcc07740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f88ba8fcc0 con 0x55f88bb2ea80 2024-05-09T08:36:10.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.934+0000 7f40d2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb3d680 0x55f88bb3fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.934+0000 7f40d2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb3d680 0x55f88bb3fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47554/0 (socket says 172.21.15.107:47554) 2024-05-09T08:36:10.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.934+0000 7f40d2ffd640 1 -- 172.21.15.107:0/3017671528 learned_addr learned my addr 172.21.15.107:0/3017671528 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:10.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.934+0000 7f40d37fe640 1 --2- 172.21.15.107:0/3017671528 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb2ea80 0x55f88bb3d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.934+0000 7f40d3fff640 1 --2- 172.21.15.107:0/3017671528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2e160 0x55f88bb2e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.934+0000 7f40d0ff9640 1 --2- 172.21.15.107:0/3521794938 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f40bc0a2d00 0x7f40bc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.935+0000 7f40d2ffd640 1 -- 172.21.15.107:0/3017671528 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb2ea80 msgr2=0x55f88bb3d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:10.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.935+0000 7f40d2ffd640 1 --2- 172.21.15.107:0/3017671528 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb2ea80 0x55f88bb3d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.935+0000 7f40d2ffd640 1 -- 172.21.15.107:0/3017671528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2e160 msgr2=0x55f88bb2e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:10.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.935+0000 7f40d2ffd640 1 --2- 172.21.15.107:0/3017671528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2e160 0x55f88bb2e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.935+0000 7f40d2ffd640 1 -- 172.21.15.107:0/3017671528 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f88b8717d0 con 0x55f88bb3d680 2024-05-09T08:36:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.935+0000 7f40d0ff9640 1 -- 172.21.15.107:0/3521794938 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f40c40cfb20 con 0x7f40d4100700 2024-05-09T08:36:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.935+0000 7f40d3fff640 1 --2- 172.21.15.107:0/3521794938 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f40bc0a2d00 0x7f40bc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.935+0000 7f40d2ffd640 1 --2- 172.21.15.107:0/3017671528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb3d680 0x55f88bb3fa70 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f40c8031440 tx=0x7f40c80040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a0b24b35b0124d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.935+0000 7f40b9ffb640 1 -- 172.21.15.107:0/3017671528 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c8035030 con 0x55f88bb3d680 2024-05-09T08:36:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.935+0000 7f40b9ffb640 1 -- 172.21.15.107:0/3017671528 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f40c80044a0 con 0x55f88bb3d680 2024-05-09T08:36:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.935+0000 7f40b9ffb640 1 -- 172.21.15.107:0/3017671528 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c80035f0 con 0x55f88bb3d680 2024-05-09T08:36:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.936+0000 7f40dcc07740 1 -- 172.21.15.107:0/3017671528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb3d680 msgr2=0x55f88bb3fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.936+0000 7f40d3fff640 1 --2- 172.21.15.107:0/3521794938 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f40bc0a2d00 0x7f40bc0a51c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0098b0 tx=0x7f40cc004ab0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.936+0000 7f40dcc07740 1 --2- 172.21.15.107:0/3017671528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb3d680 0x55f88bb3fa70 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f40c8031440 tx=0x7f40c80040b0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.936+0000 7f40dcc07740 1 -- 172.21.15.107:0/3017671528 shutdown_connections 2024-05-09T08:36:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.936+0000 7f40dcc07740 1 --2- 172.21.15.107:0/3017671528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb3d680 0x55f88bb3fa70 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.936+0000 7f40dcc07740 1 --2- 172.21.15.107:0/3017671528 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb2ea80 0x55f88bb3d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.936+0000 7f40dcc07740 1 --2- 172.21.15.107:0/3017671528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2e160 0x55f88bb2e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.936+0000 7f40dcc07740 1 -- 172.21.15.107:0/3017671528 >> 172.21.15.107:0/3017671528 conn(0x55f88bb2a5a0 msgr2=0x55f88bb2a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.936+0000 7f40dcc07740 1 -- 172.21.15.107:0/3017671528 shutdown_connections 2024-05-09T08:36:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.936+0000 7f40dcc07740 1 -- 172.21.15.107:0/3017671528 wait complete. 2024-05-09T08:36:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.936+0000 7f40dcc07740 1 -- start start 2024-05-09T08:36:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.936+0000 7f40dcc07740 1 Processor -- start 2024-05-09T08:36:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40dcc07740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2e160 0x55f88bb3a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40dcc07740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2ea80 0x55f88bb3aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40dcc07740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb3d680 0x55f88bb3b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40dcc07740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f88bb40460 con 0x55f88bb2e160 2024-05-09T08:36:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40dcc07740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f88ba4eb90 con 0x55f88bb2ea80 2024-05-09T08:36:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40dcc07740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f88bb405e0 con 0x55f88bb3d680 2024-05-09T08:36:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40dcc07740 10 client.0 _ll_register_callbacks cb 0x55f88ba8ada0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40d37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2e160 0x55f88bb3a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40d37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2e160 0x55f88bb3a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47562/0 (socket says 172.21.15.107:47562) 2024-05-09T08:36:10.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40d37fe640 1 -- 172.21.15.107:0/4088768021 learned_addr learned my addr 172.21.15.107:0/4088768021 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:10.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40d2ffd640 1 --2- 172.21.15.107:0/4088768021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2ea80 0x55f88bb3aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40d3fff640 1 --2- 172.21.15.107:0/4088768021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb3d680 0x55f88bb3b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40d37fe640 1 -- 172.21.15.107:0/4088768021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb3d680 msgr2=0x55f88bb3b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:10.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40d37fe640 1 --2- 172.21.15.107:0/4088768021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb3d680 0x55f88bb3b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40d37fe640 1 -- 172.21.15.107:0/4088768021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2ea80 msgr2=0x55f88bb3aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:10.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40d37fe640 1 --2- 172.21.15.107:0/4088768021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2ea80 0x55f88bb3aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40d37fe640 1 -- 172.21.15.107:0/4088768021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f88b7a73f0 con 0x55f88bb2e160 2024-05-09T08:36:10.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40d3fff640 1 --2- 172.21.15.107:0/4088768021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb3d680 0x55f88bb3b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:10.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40d37fe640 1 --2- 172.21.15.107:0/4088768021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2e160 0x55f88bb3a5b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f40c40ccf70 tx=0x7f40c4009720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246ed95641427d87 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:10.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:10.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.937+0000 7f40b2ffd640 1 -- 172.21.15.107:0/4088768021 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c409c3e0 con 0x55f88bb2e160 2024-05-09T08:36:10.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.938+0000 7f40b2ffd640 1 -- 172.21.15.107:0/4088768021 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f40c40a44c0 con 0x55f88bb2e160 2024-05-09T08:36:10.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.938+0000 7f40dcc07740 1 -- 172.21.15.107:0/4088768021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40c80044a0 con 0x55f88bb2e160 2024-05-09T08:36:10.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.938+0000 7f40b2ffd640 1 -- 172.21.15.107:0/4088768021 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f40c409bce0 con 0x55f88bb2e160 2024-05-09T08:36:10.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.938+0000 7f40dcc07740 10 client.16539 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:10.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.938+0000 7f40dcc07740 1 -- 172.21.15.107:0/4088768021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f88b7670f0 con 0x55f88bb2e160 2024-05-09T08:36:10.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.938+0000 7f40b2ffd640 1 -- 172.21.15.107:0/4088768021 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f40c80044a0 con 0x55f88bb2e160 2024-05-09T08:36:10.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.938+0000 7f40b2ffd640 1 -- 172.21.15.107:0/4088768021 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f40c4097a30 con 0x55f88bb2e160 2024-05-09T08:36:10.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.938+0000 7f40dcc07740 10 client.16539 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:10.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.938+0000 7f40dcc07740 10 client.16539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:10.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.938+0000 7f40dcc07740 20 client.16539 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:10.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.938+0000 7f40dcc07740 1 --2- 172.21.15.107:0/4088768021 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f88bb4dac0 0x55f88bb4fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.938+0000 7f40dcc07740 4 client.16539 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:10.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.938+0000 7f40dcc07740 1 -- 172.21.15.107:0/4088768021 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f88baa1430 con 0x55f88bb4dac0 2024-05-09T08:36:10.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.938+0000 7f40d2ffd640 1 --2- 172.21.15.107:0/4088768021 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f88bb4dac0 0x55f88bb4fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.939+0000 7f40d2ffd640 1 --2- 172.21.15.107:0/4088768021 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f88bb4dac0 0x55f88bb4fef0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b966126852ef9712 server_cookie=b4db373e74a5ba45 in_seq=0 out_seq=0 2024-05-09T08:36:10.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.939+0000 7f40b2ffd640 10 client.16539 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:10.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.939+0000 7f40b2ffd640 1 -- 172.21.15.107:0/4088768021 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f88baa1430 con 0x55f88bb4dac0 2024-05-09T08:36:10.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.939+0000 7f40b2ffd640 10 client.16539 handle_command_reply: tid=0 2024-05-09T08:36:10.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.940+0000 7f40dcc07740 1 client.16539 shutdown 2024-05-09T08:36:10.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.940+0000 7f40dcc07740 1 -- 172.21.15.107:0/4088768021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2e160 msgr2=0x55f88bb3a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.940+0000 7f40dcc07740 1 --2- 172.21.15.107:0/4088768021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2e160 0x55f88bb3a5b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f40c40ccf70 tx=0x7f40c4009720 comp rx=0 tx=0).stop 2024-05-09T08:36:10.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.940+0000 7f40dcc07740 1 -- 172.21.15.107:0/4088768021 shutdown_connections 2024-05-09T08:36:10.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.940+0000 7f40dcc07740 1 --2- 172.21.15.107:0/4088768021 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f88bb4dac0 0x55f88bb4fef0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.940+0000 7f40dcc07740 1 --2- 172.21.15.107:0/4088768021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb3d680 0x55f88bb3b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.940+0000 7f40b2ffd640 0 client.16539 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:10.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.940+0000 7f40dcc07740 1 --2- 172.21.15.107:0/4088768021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2ea80 0x55f88bb3aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.940+0000 7f40dcc07740 1 --2- 172.21.15.107:0/4088768021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2e160 0x55f88bb3a5b0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.940+0000 7f40dcc07740 1 -- 172.21.15.107:0/4088768021 >> 172.21.15.107:0/4088768021 conn(0x55f88bb2a5a0 msgr2=0x55f88bb30350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:10.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.940+0000 7f40dcc07740 1 -- 172.21.15.107:0/4088768021 shutdown_connections 2024-05-09T08:36:10.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.940+0000 7f40dcc07740 1 -- 172.21.15.107:0/4088768021 wait complete. 2024-05-09T08:36:10.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.940+0000 7f40dcc07740 20 client.16539 trim_cache size 0 max 16384 2024-05-09T08:36:10.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40dcc07740 1 Processor -- start 2024-05-09T08:36:10.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40dcc07740 1 -- start start 2024-05-09T08:36:10.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40dcc07740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2e160 0x55f88bb493f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40dcc07740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2ea80 0x55f88bb3ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40dcc07740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb3d680 0x55f88bb557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40dcc07740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f40d41af180 con 0x55f88bb2ea80 2024-05-09T08:36:10.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40dcc07740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f40d41aca50 con 0x55f88bb2e160 2024-05-09T08:36:10.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40dcc07740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f40d41aa8a0 con 0x55f88bb3d680 2024-05-09T08:36:10.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40d2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2ea80 0x55f88bb3ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40d37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2e160 0x55f88bb493f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40d2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2ea80 0x55f88bb3ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47574/0 (socket says 172.21.15.107:47574) 2024-05-09T08:36:10.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40d37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2e160 0x55f88bb493f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:51954/0 (socket says 172.21.15.107:51954) 2024-05-09T08:36:10.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40d2ffd640 1 -- 172.21.15.107:0/1811640415 learned_addr learned my addr 172.21.15.107:0/1811640415 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:10.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40d3fff640 1 --2- 172.21.15.107:0/1811640415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb3d680 0x55f88bb557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40d37fe640 1 -- 172.21.15.107:0/1811640415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb3d680 msgr2=0x55f88bb557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:10.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40d37fe640 1 --2- 172.21.15.107:0/1811640415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb3d680 0x55f88bb557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40d37fe640 1 -- 172.21.15.107:0/1811640415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2ea80 msgr2=0x55f88bb3ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:10.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40d37fe640 1 --2- 172.21.15.107:0/1811640415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2ea80 0x55f88bb3ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40d37fe640 1 -- 172.21.15.107:0/1811640415 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f88bb2e8b0 con 0x55f88bb2e160 2024-05-09T08:36:10.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.959+0000 7f40d2ffd640 1 --2- 172.21.15.107:0/1811640415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2ea80 0x55f88bb3ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:10.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.960+0000 7f40d37fe640 1 --2- 172.21.15.107:0/1811640415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2e160 0x55f88bb493f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f40c40a4420 tx=0x7f40c4009330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fb214f90a368142 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:10.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.960+0000 7f40b97fa640 1 -- 172.21.15.107:0/1811640415 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c409c3e0 con 0x55f88bb2e160 2024-05-09T08:36:10.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.960+0000 7f40b97fa640 1 -- 172.21.15.107:0/1811640415 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f40c40d4040 con 0x55f88bb2e160 2024-05-09T08:36:10.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.960+0000 7f40b97fa640 1 -- 172.21.15.107:0/1811640415 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c4097a30 con 0x55f88bb2e160 2024-05-09T08:36:10.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.961+0000 7f40dcc07740 1 -- 172.21.15.107:0/1811640415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2e160 msgr2=0x55f88bb493f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.961+0000 7f40dcc07740 1 --2- 172.21.15.107:0/1811640415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2e160 0x55f88bb493f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f40c40a4420 tx=0x7f40c4009330 comp rx=0 tx=0).stop 2024-05-09T08:36:10.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.961+0000 7f40dcc07740 1 -- 172.21.15.107:0/1811640415 shutdown_connections 2024-05-09T08:36:10.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.961+0000 7f40dcc07740 1 --2- 172.21.15.107:0/1811640415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb3d680 0x55f88bb557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.961+0000 7f40dcc07740 1 --2- 172.21.15.107:0/1811640415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2ea80 0x55f88bb3ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.961+0000 7f40dcc07740 1 --2- 172.21.15.107:0/1811640415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb2e160 0x55f88bb493f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.961+0000 7f40dcc07740 1 -- 172.21.15.107:0/1811640415 >> 172.21.15.107:0/1811640415 conn(0x55f88bb2a5a0 msgr2=0x55f88bb3a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:10.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.961+0000 7f40dcc07740 1 -- 172.21.15.107:0/1811640415 shutdown_connections 2024-05-09T08:36:10.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.961+0000 7f40dcc07740 1 -- 172.21.15.107:0/1811640415 wait complete. 2024-05-09T08:36:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.962+0000 7f40dcc07740 1 -- start start 2024-05-09T08:36:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.962+0000 7f40dcc07740 1 Processor -- start 2024-05-09T08:36:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.962+0000 7f40dcc07740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb2e160 0x55f88bb45dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.962+0000 7f40dcc07740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2ea80 0x55f88bb41140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.962+0000 7f40dcc07740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb3d680 0x55f88bb41750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.962+0000 7f40dcc07740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f40d41aaf10 con 0x55f88bb2ea80 2024-05-09T08:36:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.962+0000 7f40dcc07740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f40d41ac380 con 0x55f88bb3d680 2024-05-09T08:36:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.962+0000 7f40dcc07740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f40d41ab5e0 con 0x55f88bb2e160 2024-05-09T08:36:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.962+0000 7f40dcc07740 10 client.0 _ll_register_callbacks cb 0x7f40c400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.962+0000 7f40d2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2ea80 0x55f88bb41140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.962+0000 7f40d2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2ea80 0x55f88bb41140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47580/0 (socket says 172.21.15.107:47580) 2024-05-09T08:36:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.963+0000 7f40d2ffd640 1 -- 172.21.15.107:0/1042569087 learned_addr learned my addr 172.21.15.107:0/1042569087 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.963+0000 7f40d2ffd640 1 -- 172.21.15.107:0/1042569087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb2e160 msgr2=0x55f88bb45dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:10.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.963+0000 7f40d2ffd640 1 --2- 172.21.15.107:0/1042569087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb2e160 0x55f88bb45dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.963+0000 7f40d2ffd640 1 -- 172.21.15.107:0/1042569087 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb3d680 msgr2=0x55f88bb41750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:10.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.963+0000 7f40d2ffd640 1 --2- 172.21.15.107:0/1042569087 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb3d680 0x55f88bb41750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.963+0000 7f40d2ffd640 1 -- 172.21.15.107:0/1042569087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f88bb41f50 con 0x55f88bb2ea80 2024-05-09T08:36:10.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.963+0000 7f40d2ffd640 1 --2- 172.21.15.107:0/1042569087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2ea80 0x55f88bb41140 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f40c8031440 tx=0x7f40c80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10782a09c03f9770 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:10.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.963+0000 7f40b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:10.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.963+0000 7f40b2ffd640 1 -- 172.21.15.107:0/1042569087 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c8054070 con 0x55f88bb2ea80 2024-05-09T08:36:10.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.963+0000 7f40b2ffd640 1 -- 172.21.15.107:0/1042569087 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f40c8050810 con 0x55f88bb2ea80 2024-05-09T08:36:10.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.963+0000 7f40dcc07740 1 -- 172.21.15.107:0/1042569087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40c40d4040 con 0x55f88bb2ea80 2024-05-09T08:36:10.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.963+0000 7f40b2ffd640 1 -- 172.21.15.107:0/1042569087 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f40c8060050 con 0x55f88bb2ea80 2024-05-09T08:36:10.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.964+0000 7f40dcc07740 10 client.16545 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:10.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.964+0000 7f40dcc07740 1 -- 172.21.15.107:0/1042569087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f88bbb41d0 con 0x55f88bb2ea80 2024-05-09T08:36:10.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.964+0000 7f40b2ffd640 1 -- 172.21.15.107:0/1042569087 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f40c40d4040 con 0x55f88bb2ea80 2024-05-09T08:36:10.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.964+0000 7f40b2ffd640 1 -- 172.21.15.107:0/1042569087 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f40c8031760 con 0x55f88bb2ea80 2024-05-09T08:36:10.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.964+0000 7f40dcc07740 10 client.16545 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:10.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.964+0000 7f40dcc07740 10 client.16545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:10.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.964+0000 7f40dcc07740 20 client.16545 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:10.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.964+0000 7f40dcc07740 1 --2- 172.21.15.107:0/1042569087 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f88bb4dac0 0x55f88bbb65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:10.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.964+0000 7f40dcc07740 4 client.16545 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:36:10.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.964+0000 7f40dcc07740 1 -- 172.21.15.107:0/1042569087 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f88ba9b780 con 0x55f88bb4dac0 2024-05-09T08:36:10.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.965+0000 7f40d37fe640 1 --2- 172.21.15.107:0/1042569087 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f88bb4dac0 0x55f88bbb65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:10.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.965+0000 7f40b2ffd640 10 client.16545 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:10.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.965+0000 7f40d37fe640 1 --2- 172.21.15.107:0/1042569087 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f88bb4dac0 0x55f88bbb65d0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33e8caff212be35 server_cookie=9023f46f107bd47b in_seq=0 out_seq=0 2024-05-09T08:36:10.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.965+0000 7f40b2ffd640 1 -- 172.21.15.107:0/1042569087 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f88ba9b780 con 0x55f88bb4dac0 2024-05-09T08:36:10.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.965+0000 7f40b2ffd640 10 client.16545 handle_command_reply: tid=0 2024-05-09T08:36:10.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.965+0000 7f40dcc07740 1 client.16545 shutdown 2024-05-09T08:36:10.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.966+0000 7f40dcc07740 1 -- 172.21.15.107:0/1042569087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2ea80 msgr2=0x55f88bb41140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.966+0000 7f40dcc07740 1 --2- 172.21.15.107:0/1042569087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2ea80 0x55f88bb41140 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f40c8031440 tx=0x7f40c80042f0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.966+0000 7f40dcc07740 1 -- 172.21.15.107:0/1042569087 shutdown_connections 2024-05-09T08:36:10.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.966+0000 7f40dcc07740 1 --2- 172.21.15.107:0/1042569087 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f88bb4dac0 0x55f88bbb65d0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.966+0000 7f40dcc07740 1 --2- 172.21.15.107:0/1042569087 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f88bb3d680 0x55f88bb41750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.966+0000 7f40dcc07740 1 --2- 172.21.15.107:0/1042569087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f88bb2ea80 0x55f88bb41140 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.966+0000 7f40dcc07740 1 --2- 172.21.15.107:0/1042569087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f88bb2e160 0x55f88bb45dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.966+0000 7f40dcc07740 1 -- 172.21.15.107:0/1042569087 >> 172.21.15.107:0/1042569087 conn(0x55f88bb2a5a0 msgr2=0x55f88bb302e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:10.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.966+0000 7f40b2ffd640 0 client.16545 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:10.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.966+0000 7f40dcc07740 1 -- 172.21.15.107:0/1042569087 shutdown_connections 2024-05-09T08:36:10.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.966+0000 7f40dcc07740 1 -- 172.21.15.107:0/1042569087 wait complete. 2024-05-09T08:36:10.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.966+0000 7f40dcc07740 20 client.16545 trim_cache size 0 max 16384 2024-05-09T08:36:10.970 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:36:10.971 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:36:10.971 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:36:10.971 INFO:teuthology.orchestra.run.smithi107.stdout: "ae7c21ba-dccb-4450-9568-e7a0294fb1bd": { 2024-05-09T08:36:10.971 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:36:10.971 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "ae7c21ba-dccb-4450-9568-e7a0294fb1bd", 2024-05-09T08:36:10.971 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:36:10.971 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:36:10.971 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:36:10.971 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:36:10.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.966+0000 7f40d9efc640 1 -- 172.21.15.107:0/3521794938 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f40bc0a2d00 msgr2=0x7f40bc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.966+0000 7f40d9efc640 1 --2- 172.21.15.107:0/3521794938 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f40bc0a2d00 0x7f40bc0a51c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0098b0 tx=0x7f40cc004ab0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.966+0000 7f40d9efc640 1 -- 172.21.15.107:0/3521794938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40d4100700 msgr2=0x7f40d4198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:10.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.966+0000 7f40d9efc640 1 --2- 172.21.15.107:0/3521794938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40d4100700 0x7f40d4198160 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f40c400e9f0 tx=0x7f40c400edf0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.967+0000 7f40d9efc640 1 -- 172.21.15.107:0/3521794938 shutdown_connections 2024-05-09T08:36:10.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.967+0000 7f40d9efc640 1 --2- 172.21.15.107:0/3521794938 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f40bc0a2d00 0x7f40bc0a51c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.967+0000 7f40d9efc640 1 --2- 172.21.15.107:0/3521794938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40d4106710 0x7f40d419ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.967+0000 7f40d9efc640 1 --2- 172.21.15.107:0/3521794938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d41010c0 0x7f40d41986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.967+0000 7f40d9efc640 1 --2- 172.21.15.107:0/3521794938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40d4100700 0x7f40d4198160 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:10.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.967+0000 7f40d9efc640 1 -- 172.21.15.107:0/3521794938 >> 172.21.15.107:0/3521794938 conn(0x7f40d4077830 msgr2=0x7f40d4103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:10.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.967+0000 7f40d9efc640 1 -- 172.21.15.107:0/3521794938 shutdown_connections 2024-05-09T08:36:10.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:10.967+0000 7f40d9efc640 1 -- 172.21.15.107:0/3521794938 wait complete. 2024-05-09T08:36:10.979 INFO:tasks.cephfs.filesystem:scrub status for tag:ae7c21ba-dccb-4450-9568-e7a0294fb1bd - {'path': '/', 'tag': 'ae7c21ba-dccb-4450-9568-e7a0294fb1bd', 'options': 'recursive,force'} 2024-05-09T08:36:12.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:11 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:36:12.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:11 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:11 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:36:12.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:11 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:36:12.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:11 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:11 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:36:12.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:11 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:11 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:36:12.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:11 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:11 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:36:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:11 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:11 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:36:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:11 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:11 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:11 smithi107 ceph-mon[30122]: pgmap v88: 129 pgs: 129 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 25 MiB/s wr, 30 op/s 2024-05-09T08:36:12.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:11 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:36:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:11 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:11 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:36:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:11 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:36:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:11 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:11 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:36:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:11 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:11 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:36:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:11 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:11 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:36:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:11 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:11 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:36:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:11 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:11 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:11 smithi157 ceph-mon[33614]: pgmap v88: 129 pgs: 129 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 25 MiB/s wr, 30 op/s 2024-05-09T08:36:12.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:11 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:36:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:11 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:11 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:36:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:11 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:36:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:11 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:11 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:36:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:11 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:11 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:36:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:11 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:11 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:36:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:11 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:11 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:36:12.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:11 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:11 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:12.372 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:11 smithi110 ceph-mon[33622]: pgmap v88: 129 pgs: 129 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 25 MiB/s wr, 30 op/s 2024-05-09T08:36:14.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:13 smithi107 ceph-mon[30122]: pgmap v89: 129 pgs: 129 active+clean; 2.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 37 MiB/s wr, 68 op/s 2024-05-09T08:36:14.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:13 smithi157 ceph-mon[33614]: pgmap v89: 129 pgs: 129 active+clean; 2.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 37 MiB/s wr, 68 op/s 2024-05-09T08:36:14.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:13 smithi110 ceph-mon[33622]: pgmap v89: 129 pgs: 129 active+clean; 2.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 37 MiB/s wr, 68 op/s 2024-05-09T08:36:16.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:15 smithi107 ceph-mon[30122]: pgmap v90: 129 pgs: 129 active+clean; 2.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 25 MiB/s wr, 45 op/s 2024-05-09T08:36:16.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:15 smithi157 ceph-mon[33614]: pgmap v90: 129 pgs: 129 active+clean; 2.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 25 MiB/s wr, 45 op/s 2024-05-09T08:36:16.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:15 smithi110 ceph-mon[33622]: pgmap v90: 129 pgs: 129 active+clean; 2.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 25 MiB/s wr, 45 op/s 2024-05-09T08:36:17.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:16 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:36:17.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:16 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:36:17.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:16 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:36:17.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:16 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:36:17.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:16 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:36:17.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:16 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:36:17.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:16 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:36:17.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:16 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:36:17.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:16 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:36:18.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:17 smithi107 ceph-mon[30122]: pgmap v91: 129 pgs: 129 active+clean; 2.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:36:18.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:17 smithi157 ceph-mon[33614]: pgmap v91: 129 pgs: 129 active+clean; 2.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:36:18.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:17 smithi110 ceph-mon[33622]: pgmap v91: 129 pgs: 129 active+clean; 2.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:36:20.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:19 smithi107 ceph-mon[30122]: pgmap v92: 129 pgs: 129 active+clean; 2.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 25 MiB/s wr, 45 op/s 2024-05-09T08:36:20.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:19 smithi157 ceph-mon[33614]: pgmap v92: 129 pgs: 129 active+clean; 2.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 25 MiB/s wr, 45 op/s 2024-05-09T08:36:20.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:19 smithi110 ceph-mon[33622]: pgmap v92: 129 pgs: 129 active+clean; 2.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 25 MiB/s wr, 45 op/s 2024-05-09T08:36:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.342+0000 7fd8f31c1640 1 -- 172.21.15.107:0/2000493365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8ec100760 msgr2=0x7fd8ec100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.342+0000 7fd8f31c1640 1 --2- 172.21.15.107:0/2000493365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8ec100760 0x7fd8ec100be0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009930 tx=0x7fd8d802a420 comp rx=0 tx=0).stop 2024-05-09T08:36:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.342+0000 7fd8f31c1640 1 -- 172.21.15.107:0/2000493365 shutdown_connections 2024-05-09T08:36:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.342+0000 7fd8f31c1640 1 --2- 172.21.15.107:0/2000493365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec101120 0x7fd8ec10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.342+0000 7fd8f31c1640 1 --2- 172.21.15.107:0/2000493365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8ec100760 0x7fd8ec100be0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.342+0000 7fd8f31c1640 1 --2- 172.21.15.107:0/2000493365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8ec106720 0x7fd8ec106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.342+0000 7fd8f31c1640 1 -- 172.21.15.107:0/2000493365 >> 172.21.15.107:0/2000493365 conn(0x7fd8ec0778b0 msgr2=0x7fd8ec077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.342+0000 7fd8f31c1640 1 -- 172.21.15.107:0/2000493365 shutdown_connections 2024-05-09T08:36:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.342+0000 7fd8f31c1640 1 -- 172.21.15.107:0/2000493365 wait complete. 2024-05-09T08:36:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.343+0000 7fd8f31c1640 1 Processor -- start 2024-05-09T08:36:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.343+0000 7fd8f31c1640 1 -- start start 2024-05-09T08:36:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.343+0000 7fd8f31c1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec100760 0x7fd8ec198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.343+0000 7fd8f31c1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8ec101120 0x7fd8ec1992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.343+0000 7fd8f31c1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8ec106720 0x7fd8ec1a3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.343+0000 7fd8f31c1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec1109e0 con 0x7fd8ec101120 2024-05-09T08:36:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.343+0000 7fd8f31c1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec110860 con 0x7fd8ec106720 2024-05-09T08:36:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.343+0000 7fd8f31c1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec110b60 con 0x7fd8ec100760 2024-05-09T08:36:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.343+0000 7fd8ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8ec101120 0x7fd8ec1992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.343+0000 7fd8ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8ec101120 0x7fd8ec1992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47564/0 (socket says 172.21.15.107:47564) 2024-05-09T08:36:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.343+0000 7fd8f0f5d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec100760 0x7fd8ec198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.343+0000 7fd8ebfff640 1 -- 172.21.15.107:0/736662575 learned_addr learned my addr 172.21.15.107:0/736662575 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.344+0000 7fd8f175e640 1 --2- 172.21.15.107:0/736662575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8ec106720 0x7fd8ec1a3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.344+0000 7fd8ebfff640 1 -- 172.21.15.107:0/736662575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec100760 msgr2=0x7fd8ec198d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.344+0000 7fd8ebfff640 1 --2- 172.21.15.107:0/736662575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec100760 0x7fd8ec198d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.344+0000 7fd8ebfff640 1 -- 172.21.15.107:0/736662575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8ec106720 msgr2=0x7fd8ec1a3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.344+0000 7fd8ebfff640 1 --2- 172.21.15.107:0/736662575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8ec106720 0x7fd8ec1a3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.344+0000 7fd8ebfff640 1 -- 172.21.15.107:0/736662575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ec1a4340 con 0x7fd8ec101120 2024-05-09T08:36:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.344+0000 7fd8ebfff640 1 --2- 172.21.15.107:0/736662575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8ec101120 0x7fd8ec1992c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8004770 tx=0x7fd8d80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.344+0000 7fd8e9ffb640 1 -- 172.21.15.107:0/736662575 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d803f070 con 0x7fd8ec101120 2024-05-09T08:36:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.344+0000 7fd8e9ffb640 1 -- 172.21.15.107:0/736662575 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8d803b460 con 0x7fd8ec101120 2024-05-09T08:36:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.344+0000 7fd8f31c1640 1 -- 172.21.15.107:0/736662575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8ec1a4510 con 0x7fd8ec101120 2024-05-09T08:36:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.344+0000 7fd8e9ffb640 1 -- 172.21.15.107:0/736662575 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8035070 con 0x7fd8ec101120 2024-05-09T08:36:21.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.344+0000 7fd8f31c1640 1 -- 172.21.15.107:0/736662575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8ec1a4970 con 0x7fd8ec101120 2024-05-09T08:36:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.347+0000 7fd8e9ffb640 1 -- 172.21.15.107:0/736662575 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd8d803b610 con 0x7fd8ec101120 2024-05-09T08:36:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.347+0000 7fd8f5faa740 1 Processor -- start 2024-05-09T08:36:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.347+0000 7fd8f5faa740 1 -- start start 2024-05-09T08:36:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8f5faa740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef0a450 0x55f6fef0a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8f5faa740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0ad70 0x55f6fef150f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8f5faa740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef17a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8f5faa740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f6fee6f340 con 0x55f6fef15630 2024-05-09T08:36:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8f5faa740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f6fee17290 con 0x55f6fef0ad70 2024-05-09T08:36:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8f5faa740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f6fee6b300 con 0x55f6fef0a450 2024-05-09T08:36:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef17a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef17a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47572/0 (socket says 172.21.15.107:47572) 2024-05-09T08:36:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8ebfff640 1 -- 172.21.15.107:0/1763705796 learned_addr learned my addr 172.21.15.107:0/1763705796 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8f175e640 1 --2- 172.21.15.107:0/1763705796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef0a450 0x55f6fef0a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8f0f5d640 1 --2- 172.21.15.107:0/1763705796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0ad70 0x55f6fef150f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8e9ffb640 1 --2- 172.21.15.107:0/736662575 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd8dc0a2da0 0x7fd8dc0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8ebfff640 1 -- 172.21.15.107:0/1763705796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef0a450 msgr2=0x55f6fef0a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8ebfff640 1 --2- 172.21.15.107:0/1763705796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef0a450 0x55f6fef0a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8ebfff640 1 -- 172.21.15.107:0/1763705796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0ad70 msgr2=0x55f6fef150f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8ebfff640 1 --2- 172.21.15.107:0/1763705796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0ad70 0x55f6fef150f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8ebfff640 1 -- 172.21.15.107:0/1763705796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6fee3b8e0 con 0x55f6fef15630 2024-05-09T08:36:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8f0f5d640 1 --2- 172.21.15.107:0/1763705796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0ad70 0x55f6fef150f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8f175e640 1 --2- 172.21.15.107:0/736662575 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd8dc0a2da0 0x7fd8dc0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8e9ffb640 1 -- 172.21.15.107:0/736662575 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd8d80efb40 con 0x7fd8ec101120 2024-05-09T08:36:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.348+0000 7fd8ebfff640 1 --2- 172.21.15.107:0/1763705796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef17a20 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd8d80ecea0 tx=0x7fd8d8086880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bda81163d38e6c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.349+0000 7fd8d6ffd640 1 -- 172.21.15.107:0/1763705796 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d80bb660 con 0x55f6fef15630 2024-05-09T08:36:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.349+0000 7fd8f175e640 1 --2- 172.21.15.107:0/736662575 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd8dc0a2da0 0x7fd8dc0a5260 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4005fd0 tx=0x7fd8e4004a40 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.349+0000 7fd8d6ffd640 1 -- 172.21.15.107:0/1763705796 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8d8086d90 con 0x55f6fef15630 2024-05-09T08:36:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.349+0000 7fd8d6ffd640 1 -- 172.21.15.107:0/1763705796 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d80044e0 con 0x55f6fef15630 2024-05-09T08:36:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.349+0000 7fd8f5faa740 1 -- 172.21.15.107:0/1763705796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 msgr2=0x55f6fef17a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.349+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/1763705796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef17a20 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd8d80ecea0 tx=0x7fd8d8086880 comp rx=0 tx=0).stop 2024-05-09T08:36:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.349+0000 7fd8f5faa740 1 -- 172.21.15.107:0/1763705796 shutdown_connections 2024-05-09T08:36:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.349+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/1763705796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef17a20 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.349+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/1763705796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0ad70 0x55f6fef150f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.349+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/1763705796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef0a450 0x55f6fef0a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.349+0000 7fd8f5faa740 1 -- 172.21.15.107:0/1763705796 >> 172.21.15.107:0/1763705796 conn(0x55f6fef06960 msgr2=0x55f6fef06d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.349+0000 7fd8f5faa740 1 -- 172.21.15.107:0/1763705796 shutdown_connections 2024-05-09T08:36:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.349+0000 7fd8f5faa740 1 -- 172.21.15.107:0/1763705796 wait complete. 2024-05-09T08:36:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.350+0000 7fd8f5faa740 1 -- start start 2024-05-09T08:36:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.350+0000 7fd8f5faa740 1 Processor -- start 2024-05-09T08:36:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.350+0000 7fd8f5faa740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0a450 0x55f6fef124f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.350+0000 7fd8f5faa740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef0ad70 0x55f6fef12a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.350+0000 7fd8f5faa740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef15630 0x55f6fef12f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.350+0000 7fd8f5faa740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f6fef18020 con 0x55f6fef0ad70 2024-05-09T08:36:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.350+0000 7fd8f5faa740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f6fee2a790 con 0x55f6fef0a450 2024-05-09T08:36:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.350+0000 7fd8f5faa740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f6fef181a0 con 0x55f6fef15630 2024-05-09T08:36:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.350+0000 7fd8f5faa740 10 client.0 _ll_register_callbacks cb 0x55f6fee65a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.350+0000 7fd8ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef0ad70 0x55f6fef12a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.350+0000 7fd8ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef0ad70 0x55f6fef12a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47588/0 (socket says 172.21.15.107:47588) 2024-05-09T08:36:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.350+0000 7fd8ebfff640 1 -- 172.21.15.107:0/822565838 learned_addr learned my addr 172.21.15.107:0/822565838 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.350+0000 7fd8f175e640 1 --2- 172.21.15.107:0/822565838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef15630 0x55f6fef12f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.351+0000 7fd8f0f5d640 1 --2- 172.21.15.107:0/822565838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0a450 0x55f6fef124f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.351+0000 7fd8f175e640 1 -- 172.21.15.107:0/822565838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0a450 msgr2=0x55f6fef124f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.351+0000 7fd8f175e640 1 --2- 172.21.15.107:0/822565838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0a450 0x55f6fef124f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.351+0000 7fd8f175e640 1 -- 172.21.15.107:0/822565838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef0ad70 msgr2=0x55f6fef12a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.351+0000 7fd8f175e640 1 --2- 172.21.15.107:0/822565838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef0ad70 0x55f6fef12a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.351+0000 7fd8f175e640 1 -- 172.21.15.107:0/822565838 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6feb83210 con 0x55f6fef15630 2024-05-09T08:36:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.351+0000 7fd8ebfff640 1 --2- 172.21.15.107:0/822565838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef0ad70 0x55f6fef12a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.351+0000 7fd8f0f5d640 1 --2- 172.21.15.107:0/822565838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0a450 0x55f6fef124f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.351+0000 7fd8f175e640 1 --2- 172.21.15.107:0/822565838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef15630 0x55f6fef12f70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4012740 tx=0x7fd8e4004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11a952d6c0564aeb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.351+0000 7fd8cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:36:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.351+0000 7fd8cbfff640 1 -- 172.21.15.107:0/822565838 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e401da30 con 0x55f6fef15630 2024-05-09T08:36:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.351+0000 7fd8cbfff640 1 -- 172.21.15.107:0/822565838 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd8e401dbd0 con 0x55f6fef15630 2024-05-09T08:36:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.351+0000 7fd8f5faa740 1 -- 172.21.15.107:0/822565838 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8d8086d90 con 0x55f6fef15630 2024-05-09T08:36:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.351+0000 7fd8cbfff640 1 -- 172.21.15.107:0/822565838 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd8e4030050 con 0x55f6fef15630 2024-05-09T08:36:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.351+0000 7fd8f5faa740 10 client.25993 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.351+0000 7fd8f5faa740 1 -- 172.21.15.107:0/822565838 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f6fec4f620 con 0x55f6fef15630 2024-05-09T08:36:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.352+0000 7fd8cbfff640 1 -- 172.21.15.107:0/822565838 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8d8086d90 con 0x55f6fef15630 2024-05-09T08:36:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.352+0000 7fd8cbfff640 1 -- 172.21.15.107:0/822565838 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd8e400f040 con 0x55f6fef15630 2024-05-09T08:36:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.352+0000 7fd8f5faa740 10 client.25993 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.352+0000 7fd8f5faa740 10 client.25993 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.352+0000 7fd8f5faa740 20 client.25993 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.352+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/822565838 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f6fef25000 0x55f6fef27410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.352+0000 7fd8f5faa740 4 client.25993 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.352+0000 7fd8f5faa740 1 -- 172.21.15.107:0/822565838 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6fee7ca70 con 0x55f6fef25000 2024-05-09T08:36:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.352+0000 7fd8f0f5d640 1 --2- 172.21.15.107:0/822565838 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f6fef25000 0x55f6fef27410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.353+0000 7fd8cbfff640 10 client.25993 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.353+0000 7fd8f0f5d640 1 --2- 172.21.15.107:0/822565838 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f6fef25000 0x55f6fef27410 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c948dbea779bf03e server_cookie=f17a7069f4ee8679 in_seq=0 out_seq=0 2024-05-09T08:36:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.353+0000 7fd8cbfff640 1 -- 172.21.15.107:0/822565838 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f6fee7ca70 con 0x55f6fef25000 2024-05-09T08:36:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.353+0000 7fd8cbfff640 10 client.25993 handle_command_reply: tid=0 2024-05-09T08:36:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.353+0000 7fd8f5faa740 1 client.25993 shutdown 2024-05-09T08:36:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.353+0000 7fd8f5faa740 1 -- 172.21.15.107:0/822565838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef15630 msgr2=0x55f6fef12f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.353+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/822565838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef15630 0x55f6fef12f70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4012740 tx=0x7fd8e4004ab0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.353+0000 7fd8f5faa740 1 -- 172.21.15.107:0/822565838 shutdown_connections 2024-05-09T08:36:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.353+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/822565838 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f6fef25000 0x55f6fef27410 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.353+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/822565838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef15630 0x55f6fef12f70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.353+0000 7fd8cbfff640 0 client.25993 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.353+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/822565838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef0ad70 0x55f6fef12a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.353+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/822565838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0a450 0x55f6fef124f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.353+0000 7fd8f5faa740 1 -- 172.21.15.107:0/822565838 >> 172.21.15.107:0/822565838 conn(0x55f6fef06960 msgr2=0x55f6fef0c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.354+0000 7fd8f5faa740 1 -- 172.21.15.107:0/822565838 shutdown_connections 2024-05-09T08:36:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.354+0000 7fd8f5faa740 1 -- 172.21.15.107:0/822565838 wait complete. 2024-05-09T08:36:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.354+0000 7fd8f5faa740 20 client.25993 trim_cache size 0 max 16384 2024-05-09T08:36:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f5faa740 1 Processor -- start 2024-05-09T08:36:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f5faa740 1 -- start start 2024-05-09T08:36:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f5faa740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef0a450 0x55f6fef14c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f5faa740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0ad70 0x55f6fef273f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f5faa740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef0ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f5faa740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec1b1350 con 0x55f6fef15630 2024-05-09T08:36:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f5faa740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec1aec20 con 0x55f6fef0ad70 2024-05-09T08:36:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f5faa740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec1aca70 con 0x55f6fef0a450 2024-05-09T08:36:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f175e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef0ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f175e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef0ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47592/0 (socket says 172.21.15.107:47592) 2024-05-09T08:36:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f175e640 1 -- 172.21.15.107:0/797822956 learned_addr learned my addr 172.21.15.107:0/797822956 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f0f5d640 1 --2- 172.21.15.107:0/797822956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef0a450 0x55f6fef14c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8ebfff640 1 --2- 172.21.15.107:0/797822956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0ad70 0x55f6fef273f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f175e640 1 -- 172.21.15.107:0/797822956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef0a450 msgr2=0x55f6fef14c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f175e640 1 --2- 172.21.15.107:0/797822956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef0a450 0x55f6fef14c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f175e640 1 -- 172.21.15.107:0/797822956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0ad70 msgr2=0x55f6fef273f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f175e640 1 --2- 172.21.15.107:0/797822956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0ad70 0x55f6fef273f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f175e640 1 -- 172.21.15.107:0/797822956 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6fef114c0 con 0x55f6fef15630 2024-05-09T08:36:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8ebfff640 1 --2- 172.21.15.107:0/797822956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0ad70 0x55f6fef273f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:36:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.370+0000 7fd8f175e640 1 --2- 172.21.15.107:0/797822956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef0ce10 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd8e401b510 tx=0x7fd8e4005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449555fc3e2d227 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.371+0000 7fd8d67fc640 1 -- 172.21.15.107:0/797822956 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4039070 con 0x55f6fef15630 2024-05-09T08:36:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.371+0000 7fd8d67fc640 1 -- 172.21.15.107:0/797822956 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8e4022070 con 0x55f6fef15630 2024-05-09T08:36:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.371+0000 7fd8d67fc640 1 -- 172.21.15.107:0/797822956 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4027760 con 0x55f6fef15630 2024-05-09T08:36:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.372+0000 7fd8f5faa740 1 -- 172.21.15.107:0/797822956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 msgr2=0x55f6fef0ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.372+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/797822956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef0ce10 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd8e401b510 tx=0x7fd8e4005c50 comp rx=0 tx=0).stop 2024-05-09T08:36:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.372+0000 7fd8f5faa740 1 -- 172.21.15.107:0/797822956 shutdown_connections 2024-05-09T08:36:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.372+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/797822956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef0ce10 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.372+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/797822956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0ad70 0x55f6fef273f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.372+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/797822956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef0a450 0x55f6fef14c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.372+0000 7fd8f5faa740 1 -- 172.21.15.107:0/797822956 >> 172.21.15.107:0/797822956 conn(0x55f6fef06960 msgr2=0x55f6fef05a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.372+0000 7fd8f5faa740 1 -- 172.21.15.107:0/797822956 shutdown_connections 2024-05-09T08:36:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.372+0000 7fd8f5faa740 1 -- 172.21.15.107:0/797822956 wait complete. 2024-05-09T08:36:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.372+0000 7fd8f5faa740 1 -- start start 2024-05-09T08:36:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.372+0000 7fd8f5faa740 1 Processor -- start 2024-05-09T08:36:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f5faa740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0a450 0x55f6fef1d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f5faa740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef0ad70 0x55f6fef0f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f5faa740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef0f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f5faa740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec1ad0e0 con 0x55f6fef15630 2024-05-09T08:36:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f5faa740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec1ae550 con 0x55f6fef0a450 2024-05-09T08:36:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f5faa740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec1ad7b0 con 0x55f6fef0ad70 2024-05-09T08:36:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f5faa740 10 client.0 _ll_register_callbacks cb 0x7fd8e40123f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f175e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef0f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f175e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef0f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47598/0 (socket says 172.21.15.107:47598) 2024-05-09T08:36:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f175e640 1 -- 172.21.15.107:0/4224891684 learned_addr learned my addr 172.21.15.107:0/4224891684 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8ebfff640 1 --2- 172.21.15.107:0/4224891684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef0ad70 0x55f6fef0f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f0f5d640 1 --2- 172.21.15.107:0/4224891684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0a450 0x55f6fef1d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f175e640 1 -- 172.21.15.107:0/4224891684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef0ad70 msgr2=0x55f6fef0f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f175e640 1 --2- 172.21.15.107:0/4224891684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef0ad70 0x55f6fef0f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f175e640 1 -- 172.21.15.107:0/4224891684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0a450 msgr2=0x55f6fef1d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f175e640 1 --2- 172.21.15.107:0/4224891684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0a450 0x55f6fef1d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f175e640 1 -- 172.21.15.107:0/4224891684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6fef8a590 con 0x55f6fef15630 2024-05-09T08:36:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f175e640 1 --2- 172.21.15.107:0/4224891684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef0f790 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4037040 tx=0x7fd8e401bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2eb86a37e2ef98 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8cbfff640 1 -- 172.21.15.107:0/4224891684 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4039070 con 0x55f6fef15630 2024-05-09T08:36:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8cbfff640 1 -- 172.21.15.107:0/4224891684 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd8e4027670 con 0x55f6fef15630 2024-05-09T08:36:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.373+0000 7fd8f5faa740 1 -- 172.21.15.107:0/4224891684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8e4022070 con 0x55f6fef15630 2024-05-09T08:36:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.374+0000 7fd8cbfff640 1 -- 172.21.15.107:0/4224891684 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd8e4009ca0 con 0x55f6fef15630 2024-05-09T08:36:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.374+0000 7fd8f5faa740 10 client.16563 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.374+0000 7fd8f5faa740 1 -- 172.21.15.107:0/4224891684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f6fef8b650 con 0x55f6fef15630 2024-05-09T08:36:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.374+0000 7fd8cbfff640 1 -- 172.21.15.107:0/4224891684 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8e4022070 con 0x55f6fef15630 2024-05-09T08:36:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.374+0000 7fd8cbfff640 1 -- 172.21.15.107:0/4224891684 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd8e4033070 con 0x55f6fef15630 2024-05-09T08:36:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.374+0000 7fd8f5faa740 10 client.16563 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.374+0000 7fd8f5faa740 10 client.16563 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.374+0000 7fd8f5faa740 20 client.16563 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.374+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/4224891684 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f6fef8bd90 0x55f6fef8e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.374+0000 7fd8f5faa740 4 client.16563 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:36:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.374+0000 7fd8f5faa740 1 -- 172.21.15.107:0/4224891684 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f6fee76dc0 con 0x55f6fef8bd90 2024-05-09T08:36:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.374+0000 7fd8f0f5d640 1 --2- 172.21.15.107:0/4224891684 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f6fef8bd90 0x55f6fef8e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.375+0000 7fd8cbfff640 10 client.16563 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.375+0000 7fd8f0f5d640 1 --2- 172.21.15.107:0/4224891684 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f6fef8bd90 0x55f6fef8e1b0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2d352ea5487c280 server_cookie=da7f0ff42de82aca in_seq=0 out_seq=0 2024-05-09T08:36:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.391+0000 7fd8cbfff640 1 -- 172.21.15.107:0/4224891684 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f6fee76dc0 con 0x55f6fef8bd90 2024-05-09T08:36:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.391+0000 7fd8cbfff640 10 client.16563 handle_command_reply: tid=0 2024-05-09T08:36:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.391+0000 7fd8f5faa740 1 client.16563 shutdown 2024-05-09T08:36:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.391+0000 7fd8f5faa740 1 -- 172.21.15.107:0/4224891684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 msgr2=0x55f6fef0f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.391+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/4224891684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef0f790 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4037040 tx=0x7fd8e401bab0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.392+0000 7fd8f5faa740 1 -- 172.21.15.107:0/4224891684 shutdown_connections 2024-05-09T08:36:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.392+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/4224891684 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f6fef8bd90 0x55f6fef8e1b0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.392+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/4224891684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6fef15630 0x55f6fef0f790 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.392+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/4224891684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6fef0ad70 0x55f6fef0f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.392+0000 7fd8cbfff640 0 client.16563 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.392+0000 7fd8f5faa740 1 --2- 172.21.15.107:0/4224891684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6fef0a450 0x55f6fef1d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.392+0000 7fd8f5faa740 1 -- 172.21.15.107:0/4224891684 >> 172.21.15.107:0/4224891684 conn(0x55f6fef06960 msgr2=0x55f6fef05a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.392+0000 7fd8f5faa740 1 -- 172.21.15.107:0/4224891684 shutdown_connections 2024-05-09T08:36:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.392+0000 7fd8f5faa740 1 -- 172.21.15.107:0/4224891684 wait complete. 2024-05-09T08:36:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.392+0000 7fd8f5faa740 20 client.16563 trim_cache size 0 max 16384 2024-05-09T08:36:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:36:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:36:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:36:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:36:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.392+0000 7fd8f31c1640 1 -- 172.21.15.107:0/736662575 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd8dc0a2da0 msgr2=0x7fd8dc0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.392+0000 7fd8f31c1640 1 --2- 172.21.15.107:0/736662575 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd8dc0a2da0 0x7fd8dc0a5260 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4005fd0 tx=0x7fd8e4004a40 comp rx=0 tx=0).stop 2024-05-09T08:36:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.392+0000 7fd8f31c1640 1 -- 172.21.15.107:0/736662575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8ec101120 msgr2=0x7fd8ec1992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.392+0000 7fd8f31c1640 1 --2- 172.21.15.107:0/736662575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8ec101120 0x7fd8ec1992c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8004770 tx=0x7fd8d80042f0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.393+0000 7fd8f31c1640 1 -- 172.21.15.107:0/736662575 shutdown_connections 2024-05-09T08:36:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.393+0000 7fd8f31c1640 1 --2- 172.21.15.107:0/736662575 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd8dc0a2da0 0x7fd8dc0a5260 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.393+0000 7fd8f31c1640 1 --2- 172.21.15.107:0/736662575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8ec106720 0x7fd8ec1a3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.393+0000 7fd8f31c1640 1 --2- 172.21.15.107:0/736662575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8ec101120 0x7fd8ec1992c0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.393+0000 7fd8f31c1640 1 --2- 172.21.15.107:0/736662575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec100760 0x7fd8ec198d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.393+0000 7fd8f31c1640 1 -- 172.21.15.107:0/736662575 >> 172.21.15.107:0/736662575 conn(0x7fd8ec0778b0 msgr2=0x7fd8ec1032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.393+0000 7fd8f31c1640 1 -- 172.21.15.107:0/736662575 shutdown_connections 2024-05-09T08:36:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:21.393+0000 7fd8f31c1640 1 -- 172.21.15.107:0/736662575 wait complete. 2024-05-09T08:36:22.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:21 smithi107 ceph-mon[30122]: pgmap v93: 129 pgs: 129 active+clean; 2.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 25 MiB/s wr, 45 op/s 2024-05-09T08:36:22.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:21 smithi157 ceph-mon[33614]: pgmap v93: 129 pgs: 129 active+clean; 2.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 25 MiB/s wr, 45 op/s 2024-05-09T08:36:22.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:21 smithi110 ceph-mon[33622]: pgmap v93: 129 pgs: 129 active+clean; 2.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 25 MiB/s wr, 45 op/s 2024-05-09T08:36:24.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:23 smithi107 ceph-mon[30122]: pgmap v94: 129 pgs: 129 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 37 MiB/s wr, 57 op/s 2024-05-09T08:36:24.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:23 smithi157 ceph-mon[33614]: pgmap v94: 129 pgs: 129 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 37 MiB/s wr, 57 op/s 2024-05-09T08:36:24.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:23 smithi110 ceph-mon[33622]: pgmap v94: 129 pgs: 129 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 37 MiB/s wr, 57 op/s 2024-05-09T08:36:26.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:25 smithi107 ceph-mon[30122]: pgmap v95: 129 pgs: 129 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:26.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:25 smithi157 ceph-mon[33614]: pgmap v95: 129 pgs: 129 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:26.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:25 smithi110 ceph-mon[33622]: pgmap v95: 129 pgs: 129 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:28.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:27 smithi107 ceph-mon[30122]: pgmap v96: 129 pgs: 129 active+clean; 2.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:36:28.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:27 smithi157 ceph-mon[33614]: pgmap v96: 129 pgs: 129 active+clean; 2.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:36:28.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:27 smithi110 ceph-mon[33622]: pgmap v96: 129 pgs: 129 active+clean; 2.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:36:29.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:28 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:36:29.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:28 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:36:29.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:28 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:36:30.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:29 smithi107 ceph-mon[30122]: pgmap v97: 129 pgs: 129 active+clean; 2.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:30.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:29 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:36:30.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:29 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:36:30.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:29 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:36:30.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:29 smithi157 ceph-mon[33614]: pgmap v97: 129 pgs: 129 active+clean; 2.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:30.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:29 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:36:30.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:29 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:36:30.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:29 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:36:30.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:29 smithi110 ceph-mon[33622]: pgmap v97: 129 pgs: 129 active+clean; 2.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:30.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:29 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:36:30.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:29 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:36:30.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:29 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:36:32.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:31 smithi107 ceph-mon[30122]: pgmap v98: 129 pgs: 129 active+clean; 2.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:32.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:31 smithi157 ceph-mon[33614]: pgmap v98: 129 pgs: 129 active+clean; 2.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:32.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:31 smithi110 ceph-mon[33622]: pgmap v98: 129 pgs: 129 active+clean; 2.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:34.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:33 smithi107 ceph-mon[30122]: pgmap v99: 129 pgs: 129 active+clean; 2.7 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:36:34.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:33 smithi157 ceph-mon[33614]: pgmap v99: 129 pgs: 129 active+clean; 2.7 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:36:34.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:33 smithi110 ceph-mon[33622]: pgmap v99: 129 pgs: 129 active+clean; 2.7 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:36:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.493+0000 7f75f8c3f640 1 -- 172.21.15.107:0/1237112784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f4106720 msgr2=0x7f75f4106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.493+0000 7f75f8c3f640 1 --2- 172.21.15.107:0/1237112784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f4106720 0x7f75f4106b00 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f75e4009930 tx=0x7f75e402a420 comp rx=0 tx=0).stop 2024-05-09T08:36:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.493+0000 7f75f8c3f640 1 -- 172.21.15.107:0/1237112784 shutdown_connections 2024-05-09T08:36:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.493+0000 7f75f8c3f640 1 --2- 172.21.15.107:0/1237112784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75f4101120 0x7f75f410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.493+0000 7f75f8c3f640 1 --2- 172.21.15.107:0/1237112784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75f4100760 0x7f75f4100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.493+0000 7f75f8c3f640 1 --2- 172.21.15.107:0/1237112784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f4106720 0x7f75f4106b00 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.493+0000 7f75f8c3f640 1 -- 172.21.15.107:0/1237112784 >> 172.21.15.107:0/1237112784 conn(0x7f75f40778b0 msgr2=0x7f75f4077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.493+0000 7f75f8c3f640 1 -- 172.21.15.107:0/1237112784 shutdown_connections 2024-05-09T08:36:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.493+0000 7f75f8c3f640 1 -- 172.21.15.107:0/1237112784 wait complete. 2024-05-09T08:36:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f8c3f640 1 Processor -- start 2024-05-09T08:36:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f8c3f640 1 -- start start 2024-05-09T08:36:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f8c3f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75f4100760 0x7f75f4197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f8c3f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f4101120 0x7f75f41984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f8c3f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75f4106720 0x7f75f419c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f8c3f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41109a0 con 0x7f75f4101120 2024-05-09T08:36:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f8c3f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f75f4110820 con 0x7f75f4100760 2024-05-09T08:36:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f8c3f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f75f4110b20 con 0x7f75f4106720 2024-05-09T08:36:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f4101120 0x7f75f41984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f4101120 0x7f75f41984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41306/0 (socket says 172.21.15.107:41306) 2024-05-09T08:36:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f1d9b640 1 -- 172.21.15.107:0/884408328 learned_addr learned my addr 172.21.15.107:0/884408328 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f259c640 1 --2- 172.21.15.107:0/884408328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75f4100760 0x7f75f4197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f2d9d640 1 --2- 172.21.15.107:0/884408328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75f4106720 0x7f75f419c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f1d9b640 1 -- 172.21.15.107:0/884408328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75f4106720 msgr2=0x7f75f419c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f1d9b640 1 --2- 172.21.15.107:0/884408328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75f4106720 0x7f75f419c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f1d9b640 1 -- 172.21.15.107:0/884408328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75f4100760 msgr2=0x7f75f4197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f1d9b640 1 --2- 172.21.15.107:0/884408328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75f4100760 0x7f75f4197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f1d9b640 1 -- 172.21.15.107:0/884408328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f419cf00 con 0x7f75f4101120 2024-05-09T08:36:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.494+0000 7f75f2d9d640 1 --2- 172.21.15.107:0/884408328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75f4106720 0x7f75f419c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.495+0000 7f75f1d9b640 1 --2- 172.21.15.107:0/884408328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f4101120 0x7f75f41984a0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f75e800c920 tx=0x7f75e800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.495+0000 7f75e37fe640 1 -- 172.21.15.107:0/884408328 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8007c70 con 0x7f75f4101120 2024-05-09T08:36:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.495+0000 7f75f8c3f640 1 -- 172.21.15.107:0/884408328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f419d1f0 con 0x7f75f4101120 2024-05-09T08:36:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.495+0000 7f75e37fe640 1 -- 172.21.15.107:0/884408328 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75e8007e10 con 0x7f75f4101120 2024-05-09T08:36:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.495+0000 7f75e37fe640 1 -- 172.21.15.107:0/884408328 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8018820 con 0x7f75f4101120 2024-05-09T08:36:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.495+0000 7f75f8c3f640 1 -- 172.21.15.107:0/884408328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f41a4a80 con 0x7f75f4101120 2024-05-09T08:36:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.498+0000 7f75e37fe640 1 -- 172.21.15.107:0/884408328 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f75e80189c0 con 0x7f75f4101120 2024-05-09T08:36:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.498+0000 7f75fba2e740 1 Processor -- start 2024-05-09T08:36:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.498+0000 7f75fba2e740 1 -- start start 2024-05-09T08:36:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.498+0000 7f75fba2e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 0x55953d244830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.498+0000 7f75fba2e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d24f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.498+0000 7f75fba2e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d251a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75fba2e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55953d1a9340 con 0x55953d244d70 2024-05-09T08:36:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75fba2e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55953d151290 con 0x55953d24f630 2024-05-09T08:36:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75fba2e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55953d1a5300 con 0x55953d244450 2024-05-09T08:36:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75e37fe640 1 --2- 172.21.15.107:0/884408328 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f75dc0a2da0 0x7f75dc0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75f259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d24f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75f259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d24f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41310/0 (socket says 172.21.15.107:41310) 2024-05-09T08:36:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75f259c640 1 -- 172.21.15.107:0/3827929428 learned_addr learned my addr 172.21.15.107:0/3827929428 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75f1d9b640 1 --2- 172.21.15.107:0/3827929428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d251a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75f2d9d640 1 --2- 172.21.15.107:0/3827929428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 0x55953d244830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75f2d9d640 1 --2- 172.21.15.107:0/884408328 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f75dc0a2da0 0x7f75dc0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75f259c640 1 -- 172.21.15.107:0/3827929428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 msgr2=0x55953d244830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75f259c640 1 --2- 172.21.15.107:0/3827929428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 0x55953d244830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75f259c640 1 -- 172.21.15.107:0/3827929428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 msgr2=0x55953d251a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75f259c640 1 --2- 172.21.15.107:0/3827929428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d251a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75f259c640 1 -- 172.21.15.107:0/3827929428 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55953d1758e0 con 0x55953d244d70 2024-05-09T08:36:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75e37fe640 1 -- 172.21.15.107:0/884408328 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f75e80cf940 con 0x7f75f4101120 2024-05-09T08:36:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75f2d9d640 1 --2- 172.21.15.107:0/884408328 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f75dc0a2da0 0x7f75dc0a5260 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f75ec013b30 tx=0x7f75ec005c60 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75f259c640 1 --2- 172.21.15.107:0/3827929428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d24f0f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f75e4009930 tx=0x7f75e4032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879b89a3ca674dc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.499+0000 7f75e0ff9640 1 -- 172.21.15.107:0/3827929428 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4035040 con 0x55953d244d70 2024-05-09T08:36:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.500+0000 7f75e0ff9640 1 -- 172.21.15.107:0/3827929428 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75e40316a0 con 0x55953d244d70 2024-05-09T08:36:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.500+0000 7f75e0ff9640 1 -- 172.21.15.107:0/3827929428 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e40047a0 con 0x55953d244d70 2024-05-09T08:36:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.500+0000 7f75fba2e740 1 -- 172.21.15.107:0/3827929428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 msgr2=0x55953d24f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.500+0000 7f75fba2e740 1 --2- 172.21.15.107:0/3827929428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d24f0f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f75e4009930 tx=0x7f75e4032d60 comp rx=0 tx=0).stop 2024-05-09T08:36:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.500+0000 7f75fba2e740 1 -- 172.21.15.107:0/3827929428 shutdown_connections 2024-05-09T08:36:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.500+0000 7f75fba2e740 1 --2- 172.21.15.107:0/3827929428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d251a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.500+0000 7f75fba2e740 1 --2- 172.21.15.107:0/3827929428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d24f0f0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.500+0000 7f75fba2e740 1 --2- 172.21.15.107:0/3827929428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 0x55953d244830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.500+0000 7f75fba2e740 1 -- 172.21.15.107:0/3827929428 >> 172.21.15.107:0/3827929428 conn(0x55953d240960 msgr2=0x55953d240d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.500+0000 7f75fba2e740 1 -- 172.21.15.107:0/3827929428 shutdown_connections 2024-05-09T08:36:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.501+0000 7f75fba2e740 1 -- 172.21.15.107:0/3827929428 wait complete. 2024-05-09T08:36:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.501+0000 7f75fba2e740 1 -- start start 2024-05-09T08:36:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.501+0000 7f75fba2e740 1 Processor -- start 2024-05-09T08:36:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.501+0000 7f75fba2e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 0x55953d24c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.501+0000 7f75fba2e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d24ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.501+0000 7f75fba2e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d24cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.501+0000 7f75fba2e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55953d252200 con 0x55953d244d70 2024-05-09T08:36:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.501+0000 7f75fba2e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55953d164790 con 0x55953d24f630 2024-05-09T08:36:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.501+0000 7f75fba2e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55953d252380 con 0x55953d244450 2024-05-09T08:36:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.501+0000 7f75fba2e740 10 client.0 _ll_register_callbacks cb 0x55953d19fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.501+0000 7f75f1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d24ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.501+0000 7f75f1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d24ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41322/0 (socket says 172.21.15.107:41322) 2024-05-09T08:36:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.501+0000 7f75f1d9b640 1 -- 172.21.15.107:0/1635025250 learned_addr learned my addr 172.21.15.107:0/1635025250 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.501+0000 7f75f2d9d640 1 --2- 172.21.15.107:0/1635025250 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d24cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.501+0000 7f75f259c640 1 --2- 172.21.15.107:0/1635025250 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 0x55953d24c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.502+0000 7f75f1d9b640 1 -- 172.21.15.107:0/1635025250 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 msgr2=0x55953d24c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.502+0000 7f75f1d9b640 1 --2- 172.21.15.107:0/1635025250 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 0x55953d24c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.502+0000 7f75f1d9b640 1 -- 172.21.15.107:0/1635025250 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 msgr2=0x55953d24cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.502+0000 7f75f1d9b640 1 --2- 172.21.15.107:0/1635025250 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d24cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.502+0000 7f75f1d9b640 1 -- 172.21.15.107:0/1635025250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55953cebd210 con 0x55953d244d70 2024-05-09T08:36:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.502+0000 7f75f259c640 1 --2- 172.21.15.107:0/1635025250 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 0x55953d24c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.502+0000 7f75f2d9d640 1 --2- 172.21.15.107:0/1635025250 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d24cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.502+0000 7f75f1d9b640 1 --2- 172.21.15.107:0/1635025250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d24ca40 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f75e80ccd30 tx=0x7f75e806b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8eb31a28e65ef77 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.502+0000 7f75d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.502+0000 7f75d9ffb640 1 -- 172.21.15.107:0/1635025250 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e80cd160 con 0x55953d244d70 2024-05-09T08:36:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.502+0000 7f75d9ffb640 1 -- 172.21.15.107:0/1635025250 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f75e80a4480 con 0x55953d244d70 2024-05-09T08:36:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.502+0000 7f75fba2e740 1 -- 172.21.15.107:0/1635025250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75e40316a0 con 0x55953d244d70 2024-05-09T08:36:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.502+0000 7f75d9ffb640 1 -- 172.21.15.107:0/1635025250 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f75e809c1a0 con 0x55953d244d70 2024-05-09T08:36:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.502+0000 7f75fba2e740 10 client.16575 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.502+0000 7f75fba2e740 1 -- 172.21.15.107:0/1635025250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55953cf89620 con 0x55953d244d70 2024-05-09T08:36:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.502+0000 7f75d9ffb640 1 -- 172.21.15.107:0/1635025250 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75e40316a0 con 0x55953d244d70 2024-05-09T08:36:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.503+0000 7f75d9ffb640 1 -- 172.21.15.107:0/1635025250 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f75e806ec10 con 0x55953d244d70 2024-05-09T08:36:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.503+0000 7f75fba2e740 10 client.16575 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.503+0000 7f75fba2e740 10 client.16575 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.503+0000 7f75fba2e740 20 client.16575 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.503+0000 7f75fba2e740 1 --2- 172.21.15.107:0/1635025250 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55953d25f260 0x55953d261670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.503+0000 7f75fba2e740 4 client.16575 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.503+0000 7f75fba2e740 1 -- 172.21.15.107:0/1635025250 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55953d1b6a70 con 0x55953d25f260 2024-05-09T08:36:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.503+0000 7f75f259c640 1 --2- 172.21.15.107:0/1635025250 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55953d25f260 0x55953d261670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.504+0000 7f75d9ffb640 10 client.16575 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.504+0000 7f75f259c640 1 --2- 172.21.15.107:0/1635025250 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55953d25f260 0x55953d261670 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=240e3ce135456452 server_cookie=34e3812e34dedc7e in_seq=0 out_seq=0 2024-05-09T08:36:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.504+0000 7f75d9ffb640 1 -- 172.21.15.107:0/1635025250 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55953d1b6a70 con 0x55953d25f260 2024-05-09T08:36:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.504+0000 7f75d9ffb640 10 client.16575 handle_command_reply: tid=0 2024-05-09T08:36:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.504+0000 7f75fba2e740 1 client.16575 shutdown 2024-05-09T08:36:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.504+0000 7f75fba2e740 1 -- 172.21.15.107:0/1635025250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 msgr2=0x55953d24ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.504+0000 7f75fba2e740 1 --2- 172.21.15.107:0/1635025250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d24ca40 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f75e80ccd30 tx=0x7f75e806b900 comp rx=0 tx=0).stop 2024-05-09T08:36:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.504+0000 7f75fba2e740 1 -- 172.21.15.107:0/1635025250 shutdown_connections 2024-05-09T08:36:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.504+0000 7f75fba2e740 1 --2- 172.21.15.107:0/1635025250 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55953d25f260 0x55953d261670 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.504+0000 7f75fba2e740 1 --2- 172.21.15.107:0/1635025250 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d24cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.505+0000 7f75d9ffb640 0 client.16575 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.505+0000 7f75fba2e740 1 --2- 172.21.15.107:0/1635025250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d24ca40 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.505+0000 7f75fba2e740 1 --2- 172.21.15.107:0/1635025250 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 0x55953d24c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.505+0000 7f75fba2e740 1 -- 172.21.15.107:0/1635025250 >> 172.21.15.107:0/1635025250 conn(0x55953d240960 msgr2=0x55953d246690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.505+0000 7f75fba2e740 1 -- 172.21.15.107:0/1635025250 shutdown_connections 2024-05-09T08:36:34.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.505+0000 7f75fba2e740 1 -- 172.21.15.107:0/1635025250 wait complete. 2024-05-09T08:36:34.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.505+0000 7f75fba2e740 20 client.16575 trim_cache size 0 max 16384 2024-05-09T08:36:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.521+0000 7f75fba2e740 1 Processor -- start 2024-05-09T08:36:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.521+0000 7f75fba2e740 1 -- start start 2024-05-09T08:36:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.521+0000 7f75fba2e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 0x55953d261650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.521+0000 7f75fba2e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d25b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75fba2e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d246e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75fba2e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41b12e0 con 0x55953d244d70 2024-05-09T08:36:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75fba2e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41aebb0 con 0x55953d24f630 2024-05-09T08:36:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75fba2e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41aca00 con 0x55953d244450 2024-05-09T08:36:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75f1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d25b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75f1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d25b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41336/0 (socket says 172.21.15.107:41336) 2024-05-09T08:36:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75f1d9b640 1 -- 172.21.15.107:0/849470 learned_addr learned my addr 172.21.15.107:0/849470 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75f2d9d640 1 --2- 172.21.15.107:0/849470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d246e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75f259c640 1 --2- 172.21.15.107:0/849470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 0x55953d261650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75f1d9b640 1 -- 172.21.15.107:0/849470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 msgr2=0x55953d261650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75f1d9b640 1 --2- 172.21.15.107:0/849470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 0x55953d261650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75f1d9b640 1 -- 172.21.15.107:0/849470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 msgr2=0x55953d246e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75f1d9b640 1 --2- 172.21.15.107:0/849470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d246e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75f1d9b640 1 -- 172.21.15.107:0/849470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55953d23ef00 con 0x55953d244d70 2024-05-09T08:36:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75f1d9b640 1 --2- 172.21.15.107:0/849470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d25b0c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f75e8099d50 tx=0x7f75e8084e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c131d2be57417b94 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75dbfff640 1 -- 172.21.15.107:0/849470 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e80cd160 con 0x55953d244d70 2024-05-09T08:36:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75dbfff640 1 -- 172.21.15.107:0/849470 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75e806ed80 con 0x55953d244d70 2024-05-09T08:36:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.522+0000 7f75dbfff640 1 -- 172.21.15.107:0/849470 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e80a44e0 con 0x55953d244d70 2024-05-09T08:36:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.523+0000 7f75fba2e740 1 -- 172.21.15.107:0/849470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 msgr2=0x55953d25b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.523+0000 7f75fba2e740 1 --2- 172.21.15.107:0/849470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d25b0c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f75e8099d50 tx=0x7f75e8084e70 comp rx=0 tx=0).stop 2024-05-09T08:36:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.523+0000 7f75fba2e740 1 -- 172.21.15.107:0/849470 shutdown_connections 2024-05-09T08:36:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.523+0000 7f75fba2e740 1 --2- 172.21.15.107:0/849470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d246e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.523+0000 7f75fba2e740 1 --2- 172.21.15.107:0/849470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d25b0c0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.523+0000 7f75fba2e740 1 --2- 172.21.15.107:0/849470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 0x55953d261650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.523+0000 7f75fba2e740 1 -- 172.21.15.107:0/849470 >> 172.21.15.107:0/849470 conn(0x55953d240960 msgr2=0x55953d23f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.523+0000 7f75fba2e740 1 -- 172.21.15.107:0/849470 shutdown_connections 2024-05-09T08:36:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.523+0000 7f75fba2e740 1 -- 172.21.15.107:0/849470 wait complete. 2024-05-09T08:36:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.523+0000 7f75fba2e740 1 -- start start 2024-05-09T08:36:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75fba2e740 1 Processor -- start 2024-05-09T08:36:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75fba2e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 0x55953d2577c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75fba2e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d248b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75fba2e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d2490c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75fba2e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41ad070 con 0x55953d244d70 2024-05-09T08:36:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75fba2e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41ae4e0 con 0x55953d24f630 2024-05-09T08:36:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75fba2e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41ad740 con 0x55953d244450 2024-05-09T08:36:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75fba2e740 10 client.0 _ll_register_callbacks cb 0x7f75e800c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75f1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d248b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75f1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d248b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41340/0 (socket says 172.21.15.107:41340) 2024-05-09T08:36:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75f1d9b640 1 -- 172.21.15.107:0/3262914757 learned_addr learned my addr 172.21.15.107:0/3262914757 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75f2d9d640 1 --2- 172.21.15.107:0/3262914757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d2490c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75f1d9b640 1 -- 172.21.15.107:0/3262914757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 msgr2=0x55953d2577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75f1d9b640 1 --2- 172.21.15.107:0/3262914757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 0x55953d2577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75f1d9b640 1 -- 172.21.15.107:0/3262914757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 msgr2=0x55953d2490c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75f1d9b640 1 --2- 172.21.15.107:0/3262914757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d2490c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75f1d9b640 1 -- 172.21.15.107:0/3262914757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55953d249600 con 0x55953d244d70 2024-05-09T08:36:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75f1d9b640 1 --2- 172.21.15.107:0/3262914757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d248b80 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f75e8097e20 tx=0x7f75e80a4490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b45f4ff0e979bed server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75d9ffb640 1 -- 172.21.15.107:0/3262914757 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e80b4060 con 0x55953d244d70 2024-05-09T08:36:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75d9ffb640 1 -- 172.21.15.107:0/3262914757 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f75e809b100 con 0x55953d244d70 2024-05-09T08:36:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.524+0000 7f75fba2e740 1 -- 172.21.15.107:0/3262914757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75e806ed80 con 0x55953d244d70 2024-05-09T08:36:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.525+0000 7f75d9ffb640 1 -- 172.21.15.107:0/3262914757 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f75e80cd160 con 0x55953d244d70 2024-05-09T08:36:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.525+0000 7f75fba2e740 10 client.16581 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.525+0000 7f75fba2e740 1 -- 172.21.15.107:0/3262914757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55953d2c54c0 con 0x55953d244d70 2024-05-09T08:36:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.525+0000 7f75d9ffb640 1 -- 172.21.15.107:0/3262914757 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75e806ed80 con 0x55953d244d70 2024-05-09T08:36:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.525+0000 7f75d9ffb640 1 -- 172.21.15.107:0/3262914757 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f75e80a0fe0 con 0x55953d244d70 2024-05-09T08:36:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.525+0000 7f75fba2e740 10 client.16581 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.525+0000 7f75fba2e740 10 client.16581 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.525+0000 7f75fba2e740 20 client.16581 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.525+0000 7f75fba2e740 1 --2- 172.21.15.107:0/3262914757 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55953d25f260 0x55953d2c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.525+0000 7f75fba2e740 4 client.16581 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:36:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.525+0000 7f75fba2e740 1 -- 172.21.15.107:0/3262914757 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55953d1b0dc0 con 0x55953d25f260 2024-05-09T08:36:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.526+0000 7f75f259c640 1 --2- 172.21.15.107:0/3262914757 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55953d25f260 0x55953d2c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.526+0000 7f75d9ffb640 10 client.16581 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.526+0000 7f75f259c640 1 --2- 172.21.15.107:0/3262914757 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55953d25f260 0x55953d2c7a30 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf380a3af371f763 server_cookie=408f70d31033ded0 in_seq=0 out_seq=0 2024-05-09T08:36:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.545+0000 7f75d9ffb640 1 -- 172.21.15.107:0/3262914757 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55953d1b0dc0 con 0x55953d25f260 2024-05-09T08:36:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.545+0000 7f75d9ffb640 10 client.16581 handle_command_reply: tid=0 2024-05-09T08:36:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.545+0000 7f75fba2e740 1 client.16581 shutdown 2024-05-09T08:36:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.545+0000 7f75fba2e740 1 -- 172.21.15.107:0/3262914757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 msgr2=0x55953d248b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.545+0000 7f75fba2e740 1 --2- 172.21.15.107:0/3262914757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d248b80 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f75e8097e20 tx=0x7f75e80a4490 comp rx=0 tx=0).stop 2024-05-09T08:36:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.545+0000 7f75fba2e740 1 -- 172.21.15.107:0/3262914757 shutdown_connections 2024-05-09T08:36:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.545+0000 7f75fba2e740 1 --2- 172.21.15.107:0/3262914757 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55953d25f260 0x55953d2c7a30 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.545+0000 7f75fba2e740 1 --2- 172.21.15.107:0/3262914757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55953d24f630 0x55953d2490c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.545+0000 7f75d9ffb640 0 client.16581 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.545+0000 7f75fba2e740 1 --2- 172.21.15.107:0/3262914757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55953d244d70 0x55953d248b80 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.545+0000 7f75fba2e740 1 --2- 172.21.15.107:0/3262914757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55953d244450 0x55953d2577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.545+0000 7f75fba2e740 1 -- 172.21.15.107:0/3262914757 >> 172.21.15.107:0/3262914757 conn(0x55953d240960 msgr2=0x55953d23fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.546+0000 7f75fba2e740 1 -- 172.21.15.107:0/3262914757 shutdown_connections 2024-05-09T08:36:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.546+0000 7f75fba2e740 1 -- 172.21.15.107:0/3262914757 wait complete. 2024-05-09T08:36:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.546+0000 7f75fba2e740 20 client.16581 trim_cache size 0 max 16384 2024-05-09T08:36:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:36:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:36:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:36:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:36:34.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.546+0000 7f75f8c3f640 1 -- 172.21.15.107:0/884408328 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f75dc0a2da0 msgr2=0x7f75dc0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:34.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.546+0000 7f75f8c3f640 1 --2- 172.21.15.107:0/884408328 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f75dc0a2da0 0x7f75dc0a5260 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f75ec013b30 tx=0x7f75ec005c60 comp rx=0 tx=0).stop 2024-05-09T08:36:34.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.546+0000 7f75f8c3f640 1 -- 172.21.15.107:0/884408328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f4101120 msgr2=0x7f75f41984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.546+0000 7f75f8c3f640 1 --2- 172.21.15.107:0/884408328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f4101120 0x7f75f41984a0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f75e800c920 tx=0x7f75e800cde0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.546+0000 7f75f8c3f640 1 -- 172.21.15.107:0/884408328 shutdown_connections 2024-05-09T08:36:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.546+0000 7f75f8c3f640 1 --2- 172.21.15.107:0/884408328 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f75dc0a2da0 0x7f75dc0a5260 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.546+0000 7f75f8c3f640 1 --2- 172.21.15.107:0/884408328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75f4106720 0x7f75f419c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.546+0000 7f75f8c3f640 1 --2- 172.21.15.107:0/884408328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f4101120 0x7f75f41984a0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.546+0000 7f75f8c3f640 1 --2- 172.21.15.107:0/884408328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75f4100760 0x7f75f4197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.546+0000 7f75f8c3f640 1 -- 172.21.15.107:0/884408328 >> 172.21.15.107:0/884408328 conn(0x7f75f40778b0 msgr2=0x7f75f40ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.546+0000 7f75f8c3f640 1 -- 172.21.15.107:0/884408328 shutdown_connections 2024-05-09T08:36:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:34.546+0000 7f75f8c3f640 1 -- 172.21.15.107:0/884408328 wait complete. 2024-05-09T08:36:36.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:35 smithi107 ceph-mon[30122]: pgmap v100: 129 pgs: 129 active+clean; 2.7 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:36:36.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:35 smithi157 ceph-mon[33614]: pgmap v100: 129 pgs: 129 active+clean; 2.7 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:36:36.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:35 smithi110 ceph-mon[33622]: pgmap v100: 129 pgs: 129 active+clean; 2.7 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:36:38.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:38 smithi107 ceph-mon[30122]: pgmap v101: 129 pgs: 129 active+clean; 2.9 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:36:38.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:38 smithi157 ceph-mon[33614]: pgmap v101: 129 pgs: 129 active+clean; 2.9 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:36:38.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:38 smithi110 ceph-mon[33622]: pgmap v101: 129 pgs: 129 active+clean; 2.9 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:36:40.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:40 smithi107 ceph-mon[30122]: pgmap v102: 129 pgs: 129 active+clean; 2.9 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:40.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:40 smithi157 ceph-mon[33614]: pgmap v102: 129 pgs: 129 active+clean; 2.9 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:40.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:40 smithi110 ceph-mon[33622]: pgmap v102: 129 pgs: 129 active+clean; 2.9 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:40.979 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:36:40.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:36:41.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.097+0000 7f9606b16640 1 -- 172.21.15.107:0/2019167195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96001010c0 msgr2=0x7f960010d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.097+0000 7f9606b16640 1 --2- 172.21.15.107:0/2019167195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96001010c0 0x7f960010d5d0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f95f8009cd0 tx=0x7f95f802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.097+0000 7f9606b16640 1 -- 172.21.15.107:0/2019167195 shutdown_connections 2024-05-09T08:36:41.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.097+0000 7f9606b16640 1 --2- 172.21.15.107:0/2019167195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96001010c0 0x7f960010d5d0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.097+0000 7f9606b16640 1 --2- 172.21.15.107:0/2019167195 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9600100700 0x7f9600100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.097+0000 7f9606b16640 1 --2- 172.21.15.107:0/2019167195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9600106710 0x7f9600106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.097+0000 7f9606b16640 1 -- 172.21.15.107:0/2019167195 >> 172.21.15.107:0/2019167195 conn(0x7f9600077830 msgr2=0x7f9600077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.097+0000 7f9606b16640 1 -- 172.21.15.107:0/2019167195 shutdown_connections 2024-05-09T08:36:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.097+0000 7f9606b16640 1 -- 172.21.15.107:0/2019167195 wait complete. 2024-05-09T08:36:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f9606b16640 1 Processor -- start 2024-05-09T08:36:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f9606b16640 1 -- start start 2024-05-09T08:36:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f9606b16640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9600100700 0x7f9600197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f9606b16640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96001010c0 0x7f9600198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f9606b16640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9600106710 0x7f960019c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f9606b16640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f96001109d0 con 0x7f9600100700 2024-05-09T08:36:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f9606b16640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9600110850 con 0x7f96001010c0 2024-05-09T08:36:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f9606b16640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9600110b50 con 0x7f9600106710 2024-05-09T08:36:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f96048b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9600100700 0x7f9600197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f96048b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9600100700 0x7f9600197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38658/0 (socket says 172.21.15.107:38658) 2024-05-09T08:36:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f96048b2640 1 -- 172.21.15.107:0/777217770 learned_addr learned my addr 172.21.15.107:0/777217770 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f95fffff640 1 --2- 172.21.15.107:0/777217770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96001010c0 0x7f9600198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f96050b3640 1 --2- 172.21.15.107:0/777217770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9600106710 0x7f960019c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f96048b2640 1 -- 172.21.15.107:0/777217770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9600106710 msgr2=0x7f960019c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f96048b2640 1 --2- 172.21.15.107:0/777217770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9600106710 0x7f960019c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f96048b2640 1 -- 172.21.15.107:0/777217770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96001010c0 msgr2=0x7f9600198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f96048b2640 1 --2- 172.21.15.107:0/777217770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96001010c0 0x7f9600198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.098+0000 7f96048b2640 1 -- 172.21.15.107:0/777217770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960019cef0 con 0x7f9600100700 2024-05-09T08:36:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.099+0000 7f96048b2640 1 --2- 172.21.15.107:0/777217770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9600100700 0x7f9600197f00 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f95f400c920 tx=0x7f95f400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.099+0000 7f95fdffb640 1 -- 172.21.15.107:0/777217770 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4007c70 con 0x7f9600100700 2024-05-09T08:36:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.099+0000 7f95fdffb640 1 -- 172.21.15.107:0/777217770 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f95f4007e10 con 0x7f9600100700 2024-05-09T08:36:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.099+0000 7f9606b16640 1 -- 172.21.15.107:0/777217770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f960019d1e0 con 0x7f9600100700 2024-05-09T08:36:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.099+0000 7f95fdffb640 1 -- 172.21.15.107:0/777217770 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4018820 con 0x7f9600100700 2024-05-09T08:36:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.099+0000 7f9606b16640 1 -- 172.21.15.107:0/777217770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96001a4a70 con 0x7f9600100700 2024-05-09T08:36:41.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.102+0000 7f95fdffb640 1 -- 172.21.15.107:0/777217770 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f95f40189c0 con 0x7f9600100700 2024-05-09T08:36:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.102+0000 7f960981b740 1 Processor -- start 2024-05-09T08:36:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.102+0000 7f960981b740 1 -- start start 2024-05-09T08:36:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.102+0000 7f95fdffb640 1 --2- 172.21.15.107:0/777217770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95f00a2d00 0x7f95f00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f960981b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64180 0x55bdfef64560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f960981b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64aa0 0x55bdfef73140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f960981b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef73680 0x55bdfef75a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f960981b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55bdfeec9d00 con 0x55bdfef64180 2024-05-09T08:36:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f960981b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55bdfee714c0 con 0x55bdfef64aa0 2024-05-09T08:36:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f960981b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55bdfeec5cc0 con 0x55bdfef73680 2024-05-09T08:36:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f96048b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64180 0x55bdfef64560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f96050b3640 1 --2- 172.21.15.107:0/777217770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95f00a2d00 0x7f95f00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f96048b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64180 0x55bdfef64560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38674/0 (socket says 172.21.15.107:38674) 2024-05-09T08:36:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f95fdffb640 1 -- 172.21.15.107:0/777217770 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f95f40cf7c0 con 0x7f9600100700 2024-05-09T08:36:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f96048b2640 1 -- 172.21.15.107:0/381553381 learned_addr learned my addr 172.21.15.107:0/381553381 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f95fffff640 1 --2- 172.21.15.107:0/381553381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64aa0 0x55bdfef73140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f96050b3640 1 --2- 172.21.15.107:0/381553381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef73680 0x55bdfef75a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f96048b2640 1 -- 172.21.15.107:0/381553381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef73680 msgr2=0x55bdfef75a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f96048b2640 1 --2- 172.21.15.107:0/381553381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef73680 0x55bdfef75a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f96048b2640 1 -- 172.21.15.107:0/381553381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64aa0 msgr2=0x55bdfef73140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f96048b2640 1 --2- 172.21.15.107:0/381553381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64aa0 0x55bdfef73140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f96048b2640 1 -- 172.21.15.107:0/381553381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdfeca77d0 con 0x55bdfef64180 2024-05-09T08:36:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f96050b3640 1 --2- 172.21.15.107:0/381553381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef73680 0x55bdfef75a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:36:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f96048b2640 1 --2- 172.21.15.107:0/381553381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64180 0x55bdfef64560 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f95f4098350 tx=0x7f95f4020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94598c1a349672e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f96050b3640 1 --2- 172.21.15.107:0/777217770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95f00a2d00 0x7f95f00a51c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f95f8005e10 tx=0x7f95f8005ef0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f95eaffd640 1 -- 172.21.15.107:0/381553381 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4018cb0 con 0x55bdfef64180 2024-05-09T08:36:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f95eaffd640 1 -- 172.21.15.107:0/381553381 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f95f400fb00 con 0x55bdfef64180 2024-05-09T08:36:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.103+0000 7f95eaffd640 1 -- 172.21.15.107:0/381553381 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f409b980 con 0x55bdfef64180 2024-05-09T08:36:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.104+0000 7f960981b740 1 -- 172.21.15.107:0/381553381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64180 msgr2=0x55bdfef64560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.104+0000 7f960981b740 1 --2- 172.21.15.107:0/381553381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64180 0x55bdfef64560 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f95f4098350 tx=0x7f95f4020000 comp rx=0 tx=0).stop 2024-05-09T08:36:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.104+0000 7f960981b740 1 -- 172.21.15.107:0/381553381 shutdown_connections 2024-05-09T08:36:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.104+0000 7f960981b740 1 --2- 172.21.15.107:0/381553381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef73680 0x55bdfef75a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.104+0000 7f960981b740 1 --2- 172.21.15.107:0/381553381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64aa0 0x55bdfef73140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.104+0000 7f960981b740 1 --2- 172.21.15.107:0/381553381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64180 0x55bdfef64560 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.104+0000 7f960981b740 1 -- 172.21.15.107:0/381553381 >> 172.21.15.107:0/381553381 conn(0x55bdfef605a0 msgr2=0x55bdfef60980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.104+0000 7f960981b740 1 -- 172.21.15.107:0/381553381 shutdown_connections 2024-05-09T08:36:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.104+0000 7f960981b740 1 -- 172.21.15.107:0/381553381 wait complete. 2024-05-09T08:36:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.105+0000 7f960981b740 1 -- start start 2024-05-09T08:36:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.105+0000 7f960981b740 1 Processor -- start 2024-05-09T08:36:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.105+0000 7f960981b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64180 0x55bdfef70560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.105+0000 7f960981b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64aa0 0x55bdfef70aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.105+0000 7f960981b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef73680 0x55bdfef70fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.105+0000 7f960981b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55bdfef76440 con 0x55bdfef64aa0 2024-05-09T08:36:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.105+0000 7f960981b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55bdfee84b90 con 0x55bdfef64180 2024-05-09T08:36:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.105+0000 7f960981b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55bdfef765c0 con 0x55bdfef73680 2024-05-09T08:36:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.105+0000 7f960981b740 10 client.0 _ll_register_callbacks cb 0x55bdfeec0da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.105+0000 7f95fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64aa0 0x55bdfef70aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.105+0000 7f95fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64aa0 0x55bdfef70aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38690/0 (socket says 172.21.15.107:38690) 2024-05-09T08:36:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.105+0000 7f95fffff640 1 -- 172.21.15.107:0/1732234963 learned_addr learned my addr 172.21.15.107:0/1732234963 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.105+0000 7f96050b3640 1 --2- 172.21.15.107:0/1732234963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef73680 0x55bdfef70fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.105+0000 7f96048b2640 1 --2- 172.21.15.107:0/1732234963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64180 0x55bdfef70560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.106+0000 7f95fffff640 1 -- 172.21.15.107:0/1732234963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef73680 msgr2=0x55bdfef70fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.106+0000 7f95fffff640 1 --2- 172.21.15.107:0/1732234963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef73680 0x55bdfef70fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.106+0000 7f95fffff640 1 -- 172.21.15.107:0/1732234963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64180 msgr2=0x55bdfef70560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.106+0000 7f95fffff640 1 --2- 172.21.15.107:0/1732234963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64180 0x55bdfef70560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.106+0000 7f95fffff640 1 -- 172.21.15.107:0/1732234963 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdfebdd3f0 con 0x55bdfef64aa0 2024-05-09T08:36:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.106+0000 7f95fffff640 1 --2- 172.21.15.107:0/1732234963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64aa0 0x55bdfef70aa0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00bf80 tx=0x7f95ec0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229fe31fd3716c6b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.106+0000 7f95dffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.106+0000 7f95dffff640 1 -- 172.21.15.107:0/1732234963 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec004020 con 0x55bdfef64aa0 2024-05-09T08:36:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.106+0000 7f95dffff640 1 -- 172.21.15.107:0/1732234963 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f95ec0041c0 con 0x55bdfef64aa0 2024-05-09T08:36:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.106+0000 7f960981b740 1 -- 172.21.15.107:0/1732234963 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bdfeed7430 con 0x55bdfef64aa0 2024-05-09T08:36:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.106+0000 7f95dffff640 1 -- 172.21.15.107:0/1732234963 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f95ec024050 con 0x55bdfef64aa0 2024-05-09T08:36:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.106+0000 7f960981b740 10 client.16593 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.106+0000 7f960981b740 1 -- 172.21.15.107:0/1732234963 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bdfeb9d0f0 con 0x55bdfef64aa0 2024-05-09T08:36:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.106+0000 7f95dffff640 1 -- 172.21.15.107:0/1732234963 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55bdfeed7430 con 0x55bdfef64aa0 2024-05-09T08:36:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.107+0000 7f95dffff640 1 -- 172.21.15.107:0/1732234963 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f95ec0038b0 con 0x55bdfef64aa0 2024-05-09T08:36:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.107+0000 7f960981b740 10 client.16593 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.107+0000 7f960981b740 10 client.16593 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.107+0000 7f960981b740 20 client.16593 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.107+0000 7f960981b740 1 --2- 172.21.15.107:0/1732234963 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55bdfef83960 0x55bdfef85dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.107+0000 7f960981b740 4 client.16593 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.107+0000 7f960981b740 1 -- 172.21.15.107:0/1732234963 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdfeed1780 con 0x55bdfef83960 2024-05-09T08:36:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.107+0000 7f96048b2640 1 --2- 172.21.15.107:0/1732234963 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55bdfef83960 0x55bdfef85dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.107+0000 7f96048b2640 1 --2- 172.21.15.107:0/1732234963 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55bdfef83960 0x55bdfef85dd0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=395f6df46adbeca server_cookie=d270f968244bb869 in_seq=0 out_seq=0 2024-05-09T08:36:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.107+0000 7f95dffff640 10 client.16593 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.108+0000 7f95dffff640 1 -- 172.21.15.107:0/1732234963 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bdfeed1780 con 0x55bdfef83960 2024-05-09T08:36:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.108+0000 7f95dffff640 10 client.16593 handle_command_reply: tid=0 2024-05-09T08:36:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.108+0000 7f960981b740 1 client.16593 shutdown 2024-05-09T08:36:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.108+0000 7f960981b740 1 -- 172.21.15.107:0/1732234963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64aa0 msgr2=0x55bdfef70aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.108+0000 7f960981b740 1 --2- 172.21.15.107:0/1732234963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64aa0 0x55bdfef70aa0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00bf80 tx=0x7f95ec0118d0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.108+0000 7f960981b740 1 -- 172.21.15.107:0/1732234963 shutdown_connections 2024-05-09T08:36:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.108+0000 7f960981b740 1 --2- 172.21.15.107:0/1732234963 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55bdfef83960 0x55bdfef85dd0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.108+0000 7f960981b740 1 --2- 172.21.15.107:0/1732234963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef73680 0x55bdfef70fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.108+0000 7f95dffff640 0 client.16593 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.108+0000 7f960981b740 1 --2- 172.21.15.107:0/1732234963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64aa0 0x55bdfef70aa0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.108+0000 7f960981b740 1 --2- 172.21.15.107:0/1732234963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64180 0x55bdfef70560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.108+0000 7f960981b740 1 -- 172.21.15.107:0/1732234963 >> 172.21.15.107:0/1732234963 conn(0x55bdfef605a0 msgr2=0x55bdfef66300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.108+0000 7f960981b740 1 -- 172.21.15.107:0/1732234963 shutdown_connections 2024-05-09T08:36:41.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.108+0000 7f960981b740 1 -- 172.21.15.107:0/1732234963 wait complete. 2024-05-09T08:36:41.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.108+0000 7f960981b740 20 client.16593 trim_cache size 0 max 16384 2024-05-09T08:36:41.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f960981b740 1 Processor -- start 2024-05-09T08:36:41.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f960981b740 1 -- start start 2024-05-09T08:36:41.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f960981b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef64180 0x55bdfef7f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f960981b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64aa0 0x55bdfef70c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f960981b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef73680 0x55bdfef8b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f960981b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f96001af150 con 0x55bdfef73680 2024-05-09T08:36:41.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f960981b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f96001aca20 con 0x55bdfef64aa0 2024-05-09T08:36:41.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f960981b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f96001aa870 con 0x55bdfef64180 2024-05-09T08:36:41.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f96050b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef73680 0x55bdfef8b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f96050b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef73680 0x55bdfef8b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38692/0 (socket says 172.21.15.107:38692) 2024-05-09T08:36:41.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f96050b3640 1 -- 172.21.15.107:0/1673025569 learned_addr learned my addr 172.21.15.107:0/1673025569 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f96048b2640 1 --2- 172.21.15.107:0/1673025569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef64180 0x55bdfef7f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f95fffff640 1 --2- 172.21.15.107:0/1673025569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64aa0 0x55bdfef70c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f96050b3640 1 -- 172.21.15.107:0/1673025569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef64180 msgr2=0x55bdfef7f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f96050b3640 1 --2- 172.21.15.107:0/1673025569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef64180 0x55bdfef7f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f96050b3640 1 -- 172.21.15.107:0/1673025569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64aa0 msgr2=0x55bdfef70c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f96050b3640 1 --2- 172.21.15.107:0/1673025569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64aa0 0x55bdfef70c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.126+0000 7f96050b3640 1 -- 172.21.15.107:0/1673025569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdfef5bf60 con 0x55bdfef73680 2024-05-09T08:36:41.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.127+0000 7f96050b3640 1 --2- 172.21.15.107:0/1673025569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef73680 0x55bdfef8b6e0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f95f8035000 tx=0x7f95f8040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a7725a37eb4debe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.127+0000 7f95ea7fc640 1 -- 172.21.15.107:0/1673025569 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f8003fb0 con 0x55bdfef73680 2024-05-09T08:36:41.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.127+0000 7f95ea7fc640 1 -- 172.21.15.107:0/1673025569 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f95f8036040 con 0x55bdfef73680 2024-05-09T08:36:41.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.127+0000 7f95ea7fc640 1 -- 172.21.15.107:0/1673025569 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f8030c40 con 0x55bdfef73680 2024-05-09T08:36:41.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.127+0000 7f960981b740 1 -- 172.21.15.107:0/1673025569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef73680 msgr2=0x55bdfef8b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.127+0000 7f960981b740 1 --2- 172.21.15.107:0/1673025569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef73680 0x55bdfef8b6e0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f95f8035000 tx=0x7f95f8040040 comp rx=0 tx=0).stop 2024-05-09T08:36:41.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f960981b740 1 -- 172.21.15.107:0/1673025569 shutdown_connections 2024-05-09T08:36:41.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f960981b740 1 --2- 172.21.15.107:0/1673025569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef73680 0x55bdfef8b6e0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f960981b740 1 --2- 172.21.15.107:0/1673025569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64aa0 0x55bdfef70c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f960981b740 1 --2- 172.21.15.107:0/1673025569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef64180 0x55bdfef7f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f960981b740 1 -- 172.21.15.107:0/1673025569 >> 172.21.15.107:0/1673025569 conn(0x55bdfef605a0 msgr2=0x55bdfef70520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f960981b740 1 -- 172.21.15.107:0/1673025569 shutdown_connections 2024-05-09T08:36:41.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f960981b740 1 -- 172.21.15.107:0/1673025569 wait complete. 2024-05-09T08:36:41.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f960981b740 1 -- start start 2024-05-09T08:36:41.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f960981b740 1 Processor -- start 2024-05-09T08:36:41.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f960981b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64180 0x55bdfef7bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f960981b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64aa0 0x55bdfef6ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f960981b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef73680 0x55bdfef6d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f960981b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f96001aaee0 con 0x55bdfef64aa0 2024-05-09T08:36:41.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f960981b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f96001ac350 con 0x55bdfef64180 2024-05-09T08:36:41.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f960981b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f96001ab5b0 con 0x55bdfef73680 2024-05-09T08:36:41.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f960981b740 10 client.0 _ll_register_callbacks cb 0x7f95ec00bb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:41.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.128+0000 7f95fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64aa0 0x55bdfef6ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f95fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64aa0 0x55bdfef6ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38704/0 (socket says 172.21.15.107:38704) 2024-05-09T08:36:41.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f95fffff640 1 -- 172.21.15.107:0/1799500746 learned_addr learned my addr 172.21.15.107:0/1799500746 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f96050b3640 1 --2- 172.21.15.107:0/1799500746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef73680 0x55bdfef6d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f96048b2640 1 --2- 172.21.15.107:0/1799500746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64180 0x55bdfef7bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f95fffff640 1 -- 172.21.15.107:0/1799500746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef73680 msgr2=0x55bdfef6d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f95fffff640 1 --2- 172.21.15.107:0/1799500746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef73680 0x55bdfef6d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f95fffff640 1 -- 172.21.15.107:0/1799500746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64180 msgr2=0x55bdfef7bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f95fffff640 1 --2- 172.21.15.107:0/1799500746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64180 0x55bdfef7bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f95fffff640 1 -- 172.21.15.107:0/1799500746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdfef6dc10 con 0x55bdfef64aa0 2024-05-09T08:36:41.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f95fffff640 1 --2- 172.21.15.107:0/1799500746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64aa0 0x55bdfef6ced0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f95ec000c00 tx=0x7f95ec003880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885c78fc3ca64442 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f95dffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:41.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f95dffff640 1 -- 172.21.15.107:0/1799500746 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec00b480 con 0x55bdfef64aa0 2024-05-09T08:36:41.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f95dffff640 1 -- 172.21.15.107:0/1799500746 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f95ec003f40 con 0x55bdfef64aa0 2024-05-09T08:36:41.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f960981b740 1 -- 172.21.15.107:0/1799500746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95f8036040 con 0x55bdfef64aa0 2024-05-09T08:36:41.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f95dffff640 1 -- 172.21.15.107:0/1799500746 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f95ec039050 con 0x55bdfef64aa0 2024-05-09T08:36:41.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f960981b740 10 client.16599 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:41.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.129+0000 7f960981b740 1 -- 172.21.15.107:0/1799500746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bdfefe9f30 con 0x55bdfef64aa0 2024-05-09T08:36:41.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.130+0000 7f95dffff640 1 -- 172.21.15.107:0/1799500746 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f95ec02d020 con 0x55bdfef64aa0 2024-05-09T08:36:41.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.130+0000 7f95dffff640 1 -- 172.21.15.107:0/1799500746 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f95ec010850 con 0x55bdfef64aa0 2024-05-09T08:36:41.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.130+0000 7f960981b740 10 client.16599 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.130+0000 7f960981b740 10 client.16599 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.130+0000 7f960981b740 20 client.16599 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.130+0000 7f960981b740 1 --2- 172.21.15.107:0/1799500746 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55bdfef83960 0x55bdfefec7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.130+0000 7f960981b740 4 client.16599 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:36:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.130+0000 7f960981b740 1 -- 172.21.15.107:0/1799500746 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bdfee4e6f0 con 0x55bdfef83960 2024-05-09T08:36:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.130+0000 7f96048b2640 1 --2- 172.21.15.107:0/1799500746 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55bdfef83960 0x55bdfefec7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.130+0000 7f95dffff640 10 client.16599 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.130+0000 7f96048b2640 1 --2- 172.21.15.107:0/1799500746 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55bdfef83960 0x55bdfefec7d0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2628b6e15d5aaa7a server_cookie=95465e24a88648b6 in_seq=0 out_seq=0 2024-05-09T08:36:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.131+0000 7f95dffff640 1 -- 172.21.15.107:0/1799500746 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bdfee4e6f0 con 0x55bdfef83960 2024-05-09T08:36:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.131+0000 7f95dffff640 10 client.16599 handle_command_reply: tid=0 2024-05-09T08:36:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.131+0000 7f960981b740 1 client.16599 shutdown 2024-05-09T08:36:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.131+0000 7f960981b740 1 -- 172.21.15.107:0/1799500746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64aa0 msgr2=0x55bdfef6ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.131+0000 7f960981b740 1 --2- 172.21.15.107:0/1799500746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64aa0 0x55bdfef6ced0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f95ec000c00 tx=0x7f95ec003880 comp rx=0 tx=0).stop 2024-05-09T08:36:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.131+0000 7f960981b740 1 -- 172.21.15.107:0/1799500746 shutdown_connections 2024-05-09T08:36:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.131+0000 7f960981b740 1 --2- 172.21.15.107:0/1799500746 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55bdfef83960 0x55bdfefec7d0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.131+0000 7f960981b740 1 --2- 172.21.15.107:0/1799500746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdfef73680 0x55bdfef6d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.131+0000 7f960981b740 1 --2- 172.21.15.107:0/1799500746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdfef64aa0 0x55bdfef6ced0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.131+0000 7f960981b740 1 --2- 172.21.15.107:0/1799500746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bdfef64180 0x55bdfef7bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.131+0000 7f960981b740 1 -- 172.21.15.107:0/1799500746 >> 172.21.15.107:0/1799500746 conn(0x55bdfef605a0 msgr2=0x55bdfef70d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.131+0000 7f95dffff640 0 client.16599 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:41.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.131+0000 7f960981b740 1 -- 172.21.15.107:0/1799500746 shutdown_connections 2024-05-09T08:36:41.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.131+0000 7f960981b740 1 -- 172.21.15.107:0/1799500746 wait complete. 2024-05-09T08:36:41.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.131+0000 7f960981b740 20 client.16599 trim_cache size 0 max 16384 2024-05-09T08:36:41.136 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:36:41.137 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:36:41.137 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:36:41.137 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:36:41.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.132+0000 7f9606b16640 1 -- 172.21.15.107:0/777217770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95f00a2d00 msgr2=0x7f95f00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.132+0000 7f9606b16640 1 --2- 172.21.15.107:0/777217770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95f00a2d00 0x7f95f00a51c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f95f8005e10 tx=0x7f95f8005ef0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.132+0000 7f9606b16640 1 -- 172.21.15.107:0/777217770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9600100700 msgr2=0x7f9600197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.132+0000 7f9606b16640 1 --2- 172.21.15.107:0/777217770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9600100700 0x7f9600197f00 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f95f400c920 tx=0x7f95f400cde0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.132+0000 7f9606b16640 1 -- 172.21.15.107:0/777217770 shutdown_connections 2024-05-09T08:36:41.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.132+0000 7f9606b16640 1 --2- 172.21.15.107:0/777217770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95f00a2d00 0x7f95f00a51c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.132+0000 7f9606b16640 1 --2- 172.21.15.107:0/777217770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9600106710 0x7f960019c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.132+0000 7f9606b16640 1 --2- 172.21.15.107:0/777217770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96001010c0 0x7f9600198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.132+0000 7f9606b16640 1 --2- 172.21.15.107:0/777217770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9600100700 0x7f9600197f00 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.132+0000 7f9606b16640 1 -- 172.21.15.107:0/777217770 >> 172.21.15.107:0/777217770 conn(0x7f9600077830 msgr2=0x7f9600068c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.132+0000 7f9606b16640 1 -- 172.21.15.107:0/777217770 shutdown_connections 2024-05-09T08:36:41.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.132+0000 7f9606b16640 1 -- 172.21.15.107:0/777217770 wait complete. 2024-05-09T08:36:41.141 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:36:41.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:36:41.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.254+0000 7fc98aa09640 1 -- 172.21.15.107:0/3759356004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9841010d0 msgr2=0x7fc98410d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.254+0000 7fc98aa09640 1 --2- 172.21.15.107:0/3759356004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9841010d0 0x7fc98410d5e0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc97c009cd0 tx=0x7fc97c02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.254+0000 7fc98aa09640 1 -- 172.21.15.107:0/3759356004 shutdown_connections 2024-05-09T08:36:41.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.254+0000 7fc98aa09640 1 --2- 172.21.15.107:0/3759356004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9841010d0 0x7fc98410d5e0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.254+0000 7fc98aa09640 1 --2- 172.21.15.107:0/3759356004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc984100710 0x7fc984100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.254+0000 7fc98aa09640 1 --2- 172.21.15.107:0/3759356004 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc984106720 0x7fc984106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.254+0000 7fc98aa09640 1 -- 172.21.15.107:0/3759356004 >> 172.21.15.107:0/3759356004 conn(0x7fc984077830 msgr2=0x7fc984077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.254+0000 7fc98aa09640 1 -- 172.21.15.107:0/3759356004 shutdown_connections 2024-05-09T08:36:41.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.254+0000 7fc98aa09640 1 -- 172.21.15.107:0/3759356004 wait complete. 2024-05-09T08:36:41.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.254+0000 7fc98aa09640 1 Processor -- start 2024-05-09T08:36:41.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.254+0000 7fc98aa09640 1 -- start start 2024-05-09T08:36:41.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc98aa09640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc984100710 0x7fc984196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc98aa09640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9841010d0 0x7fc984197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc98aa09640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc984106720 0x7fc984197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc98aa09640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841109a0 con 0x7fc984100710 2024-05-09T08:36:41.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc98aa09640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc984110820 con 0x7fc984106720 2024-05-09T08:36:41.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc98aa09640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc984110b20 con 0x7fc9841010d0 2024-05-09T08:36:41.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc983fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc984100710 0x7fc984196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc983fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc984100710 0x7fc984196bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38710/0 (socket says 172.21.15.107:38710) 2024-05-09T08:36:41.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc983fff640 1 -- 172.21.15.107:0/2966612641 learned_addr learned my addr 172.21.15.107:0/2966612641 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc988fa6640 1 --2- 172.21.15.107:0/2966612641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc984106720 0x7fc984197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc97bfff640 1 --2- 172.21.15.107:0/2966612641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9841010d0 0x7fc984197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc983fff640 1 -- 172.21.15.107:0/2966612641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9841010d0 msgr2=0x7fc984197130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc983fff640 1 --2- 172.21.15.107:0/2966612641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9841010d0 0x7fc984197130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc983fff640 1 -- 172.21.15.107:0/2966612641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc984106720 msgr2=0x7fc984197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc983fff640 1 --2- 172.21.15.107:0/2966612641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc984106720 0x7fc984197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc983fff640 1 -- 172.21.15.107:0/2966612641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9841a6460 con 0x7fc984100710 2024-05-09T08:36:41.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc983fff640 1 --2- 172.21.15.107:0/2966612641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc984100710 0x7fc984196bf0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fc97400e9f0 tx=0x7fc97400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc981ffb640 1 -- 172.21.15.107:0/2966612641 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97400cd80 con 0x7fc984100710 2024-05-09T08:36:41.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc981ffb640 1 -- 172.21.15.107:0/2966612641 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc9740108e0 con 0x7fc984100710 2024-05-09T08:36:41.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc98aa09640 1 -- 172.21.15.107:0/2966612641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9841a6750 con 0x7fc984100710 2024-05-09T08:36:41.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.255+0000 7fc981ffb640 1 -- 172.21.15.107:0/2966612641 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974018920 con 0x7fc984100710 2024-05-09T08:36:41.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.256+0000 7fc98aa09640 1 -- 172.21.15.107:0/2966612641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9841a6cf0 con 0x7fc984100710 2024-05-09T08:36:41.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.256+0000 7fc98aa09640 1 -- 172.21.15.107:0/2966612641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc984068650 con 0x7fc984100710 2024-05-09T08:36:41.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.258+0000 7fc981ffb640 1 -- 172.21.15.107:0/2966612641 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fc974010a90 con 0x7fc984100710 2024-05-09T08:36:41.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.259+0000 7fc981ffb640 1 --2- 172.21.15.107:0/2966612641 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9700a2d80 0x7fc9700a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.259+0000 7fc97bfff640 1 --2- 172.21.15.107:0/2966612641 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9700a2d80 0x7fc9700a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.259+0000 7fc981ffb640 1 -- 172.21.15.107:0/2966612641 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fc9740cff40 con 0x7fc984100710 2024-05-09T08:36:41.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.260+0000 7fc97bfff640 1 --2- 172.21.15.107:0/2966612641 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9700a2d80 0x7fc9700a5240 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc96c005fd0 tx=0x7fc96c005850 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.263+0000 7fc981ffb640 1 -- 172.21.15.107:0/2966612641 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc97409c7a0 con 0x7fc984100710 2024-05-09T08:36:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.511+0000 7fc98aa09640 1 -- 172.21.15.107:0/2966612641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc984068fb0 con 0x7fc984100710 2024-05-09T08:36:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.511+0000 7fc981ffb640 1 -- 172.21.15.107:0/2966612641 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11890 (secure 0 0 0) 0x7fc984068fb0 con 0x7fc984100710 2024-05-09T08:36:41.510 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:36:41.510 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:34:45.925212+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24478,24499]},"id":1}]} 2024-05-09T08:36:41.511 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 26 2024-05-09T08:36:41.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.514+0000 7fc98aa09640 1 -- 172.21.15.107:0/2966612641 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9700a2d80 msgr2=0x7fc9700a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.514+0000 7fc98aa09640 1 --2- 172.21.15.107:0/2966612641 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9700a2d80 0x7fc9700a5240 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc96c005fd0 tx=0x7fc96c005850 comp rx=0 tx=0).stop 2024-05-09T08:36:41.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.514+0000 7fc98aa09640 1 -- 172.21.15.107:0/2966612641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc984100710 msgr2=0x7fc984196bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.514+0000 7fc98aa09640 1 --2- 172.21.15.107:0/2966612641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc984100710 0x7fc984196bf0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fc97400e9f0 tx=0x7fc97400eeb0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.514+0000 7fc98aa09640 1 -- 172.21.15.107:0/2966612641 shutdown_connections 2024-05-09T08:36:41.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.514+0000 7fc98aa09640 1 --2- 172.21.15.107:0/2966612641 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9700a2d80 0x7fc9700a5240 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.514+0000 7fc98aa09640 1 --2- 172.21.15.107:0/2966612641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc984106720 0x7fc984197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.514+0000 7fc98aa09640 1 --2- 172.21.15.107:0/2966612641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9841010d0 0x7fc984197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.514+0000 7fc98aa09640 1 --2- 172.21.15.107:0/2966612641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc984100710 0x7fc984196bf0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.514+0000 7fc98aa09640 1 -- 172.21.15.107:0/2966612641 >> 172.21.15.107:0/2966612641 conn(0x7fc984077830 msgr2=0x7fc984102f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.514+0000 7fc98aa09640 1 -- 172.21.15.107:0/2966612641 shutdown_connections 2024-05-09T08:36:41.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.514+0000 7fc98aa09640 1 -- 172.21.15.107:0/2966612641 wait complete. 2024-05-09T08:36:41.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:36:41.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.646+0000 7f1cd07ee640 1 -- 172.21.15.107:0/503101010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cc8107820 msgr2=0x7f1cc8107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.646+0000 7f1cd07ee640 1 --2- 172.21.15.107:0/503101010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cc8107820 0x7f1cc8107c00 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4009930 tx=0x7f1cc402a420 comp rx=0 tx=0).stop 2024-05-09T08:36:41.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.646+0000 7f1cd07ee640 1 -- 172.21.15.107:0/503101010 shutdown_connections 2024-05-09T08:36:41.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.646+0000 7f1cd07ee640 1 --2- 172.21.15.107:0/503101010 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cc81021d0 0x7f1cc810e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.646+0000 7f1cd07ee640 1 --2- 172.21.15.107:0/503101010 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1cc8101810 0x7f1cc8101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.646+0000 7f1cd07ee640 1 --2- 172.21.15.107:0/503101010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cc8107820 0x7f1cc8107c00 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.646+0000 7f1cd07ee640 1 -- 172.21.15.107:0/503101010 >> 172.21.15.107:0/503101010 conn(0x7f1cc8078c30 msgr2=0x7f1cc8079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.646+0000 7f1cd07ee640 1 -- 172.21.15.107:0/503101010 shutdown_connections 2024-05-09T08:36:41.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.646+0000 7f1cd07ee640 1 -- 172.21.15.107:0/503101010 wait complete. 2024-05-09T08:36:41.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.647+0000 7f1cd07ee640 1 Processor -- start 2024-05-09T08:36:41.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.647+0000 7f1cd07ee640 1 -- start start 2024-05-09T08:36:41.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.647+0000 7f1cd07ee640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cc8101810 0x7f1cc819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.647+0000 7f1cd07ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cc81021d0 0x7f1cc819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.647+0000 7f1cd07ee640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1cc8107820 0x7f1cc819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.647+0000 7f1cd07ee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc8111b30 con 0x7f1cc81021d0 2024-05-09T08:36:41.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.647+0000 7f1cd07ee640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc81119b0 con 0x7f1cc8101810 2024-05-09T08:36:41.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.647+0000 7f1cd07ee640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc8111cb0 con 0x7f1cc8107820 2024-05-09T08:36:41.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.647+0000 7f1ccdd89640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cc81021d0 0x7f1cc819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.647+0000 7f1ccdd89640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cc81021d0 0x7f1cc819a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38732/0 (socket says 172.21.15.107:38732) 2024-05-09T08:36:41.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.647+0000 7f1ccdd89640 1 -- 172.21.15.107:0/4100189999 learned_addr learned my addr 172.21.15.107:0/4100189999 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.647+0000 7f1cce58a640 1 --2- 172.21.15.107:0/4100189999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cc8101810 0x7f1cc819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.647+0000 7f1cced8b640 1 --2- 172.21.15.107:0/4100189999 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1cc8107820 0x7f1cc819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.648+0000 7f1ccdd89640 1 -- 172.21.15.107:0/4100189999 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1cc8107820 msgr2=0x7f1cc819eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.648+0000 7f1ccdd89640 1 --2- 172.21.15.107:0/4100189999 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1cc8107820 0x7f1cc819eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.648+0000 7f1ccdd89640 1 -- 172.21.15.107:0/4100189999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cc8101810 msgr2=0x7f1cc819a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.648+0000 7f1ccdd89640 1 --2- 172.21.15.107:0/4100189999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cc8101810 0x7f1cc819a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.648+0000 7f1ccdd89640 1 -- 172.21.15.107:0/4100189999 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc819f2a0 con 0x7f1cc81021d0 2024-05-09T08:36:41.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.648+0000 7f1ccdd89640 1 --2- 172.21.15.107:0/4100189999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cc81021d0 0x7f1cc819a7f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f1cb800e9f0 tx=0x7f1cb800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.648+0000 7f1cbf7fe640 1 -- 172.21.15.107:0/4100189999 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb800cd80 con 0x7f1cc81021d0 2024-05-09T08:36:41.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.648+0000 7f1cd07ee640 1 -- 172.21.15.107:0/4100189999 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cc819f590 con 0x7f1cc81021d0 2024-05-09T08:36:41.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.648+0000 7f1cbf7fe640 1 -- 172.21.15.107:0/4100189999 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1cb80054d0 con 0x7f1cc81021d0 2024-05-09T08:36:41.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.648+0000 7f1cbf7fe640 1 -- 172.21.15.107:0/4100189999 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8010490 con 0x7f1cc81021d0 2024-05-09T08:36:41.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.648+0000 7f1cd07ee640 1 -- 172.21.15.107:0/4100189999 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cc81a6e20 con 0x7f1cc81021d0 2024-05-09T08:36:41.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.650+0000 7f1cbf7fe640 1 -- 172.21.15.107:0/4100189999 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1cb8005020 con 0x7f1cc81021d0 2024-05-09T08:36:41.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.651+0000 7f1cd35de740 1 Processor -- start 2024-05-09T08:36:41.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.651+0000 7f1cd35de740 1 -- start start 2024-05-09T08:36:41.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.651+0000 7f1cbf7fe640 1 --2- 172.21.15.107:0/4100189999 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1cb40a2d00 0x7f1cb40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.651+0000 7f1cd35de740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207330 0x559994207710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.651+0000 7f1cd35de740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207c50 0x559994211de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.651+0000 7f1cd35de740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 0x559994214710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.651+0000 7f1cd35de740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55999416cc40 con 0x559994207c50 2024-05-09T08:36:41.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.651+0000 7f1cd35de740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559994114580 con 0x559994212320 2024-05-09T08:36:41.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.651+0000 7f1cd35de740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x559994168c00 con 0x559994207330 2024-05-09T08:36:41.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cced8b640 1 --2- 172.21.15.107:0/4100189999 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1cb40a2d00 0x7f1cb40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cce58a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207330 0x559994207710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cce58a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207330 0x559994207710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:39192/0 (socket says 172.21.15.107:39192) 2024-05-09T08:36:41.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cce58a640 1 -- 172.21.15.107:0/1031623223 learned_addr learned my addr 172.21.15.107:0/1031623223 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cbf7fe640 1 -- 172.21.15.107:0/4100189999 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1cb80cfbc0 con 0x7f1cc81021d0 2024-05-09T08:36:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1ccdd89640 1 --2- 172.21.15.107:0/1031623223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207c50 0x559994211de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cced8b640 1 --2- 172.21.15.107:0/1031623223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 0x559994214710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cce58a640 1 -- 172.21.15.107:0/1031623223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 msgr2=0x559994214710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cce58a640 1 --2- 172.21.15.107:0/1031623223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 0x559994214710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cce58a640 1 -- 172.21.15.107:0/1031623223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207c50 msgr2=0x559994211de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cce58a640 1 --2- 172.21.15.107:0/1031623223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207c50 0x559994211de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cce58a640 1 -- 172.21.15.107:0/1031623223 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559993e803f0 con 0x559994207330 2024-05-09T08:36:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cced8b640 1 --2- 172.21.15.107:0/1031623223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 0x559994214710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:36:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cced8b640 1 --2- 172.21.15.107:0/4100189999 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1cb40a2d00 0x7f1cb40a51c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x5599942047e0 tx=0x7f1cc0009c80 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cce58a640 1 --2- 172.21.15.107:0/1031623223 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207330 0x559994207710 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4006010 tx=0x7f1cc4004170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18343e09bad9675f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cbcff9640 1 -- 172.21.15.107:0/1031623223 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4035040 con 0x559994207330 2024-05-09T08:36:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cbcff9640 1 -- 172.21.15.107:0/1031623223 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1cc40054f0 con 0x559994207330 2024-05-09T08:36:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.652+0000 7f1cbcff9640 1 -- 172.21.15.107:0/1031623223 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc40035f0 con 0x559994207330 2024-05-09T08:36:41.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.653+0000 7f1cd35de740 1 -- 172.21.15.107:0/1031623223 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207330 msgr2=0x559994207710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.653+0000 7f1cd35de740 1 --2- 172.21.15.107:0/1031623223 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207330 0x559994207710 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4006010 tx=0x7f1cc4004170 comp rx=0 tx=0).stop 2024-05-09T08:36:41.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.653+0000 7f1cd35de740 1 -- 172.21.15.107:0/1031623223 shutdown_connections 2024-05-09T08:36:41.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.653+0000 7f1cd35de740 1 --2- 172.21.15.107:0/1031623223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 0x559994214710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.653+0000 7f1cd35de740 1 --2- 172.21.15.107:0/1031623223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207c50 0x559994211de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.653+0000 7f1cd35de740 1 --2- 172.21.15.107:0/1031623223 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207330 0x559994207710 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.653+0000 7f1cd35de740 1 -- 172.21.15.107:0/1031623223 >> 172.21.15.107:0/1031623223 conn(0x559994203650 msgr2=0x559994203a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.653+0000 7f1cd35de740 1 -- 172.21.15.107:0/1031623223 shutdown_connections 2024-05-09T08:36:41.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.653+0000 7f1cd35de740 1 -- 172.21.15.107:0/1031623223 wait complete. 2024-05-09T08:36:41.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cd35de740 1 -- start start 2024-05-09T08:36:41.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cd35de740 1 Processor -- start 2024-05-09T08:36:41.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cd35de740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x55999420f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cd35de740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207c50 0x55999420f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cd35de740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 0x55999420fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cd35de740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5599942150b0 con 0x559994207330 2024-05-09T08:36:41.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cd35de740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559994127c40 con 0x559994212320 2024-05-09T08:36:41.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cd35de740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x559994215230 con 0x559994207c50 2024-05-09T08:36:41.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cd35de740 10 client.0 _ll_register_callbacks cb 0x559993e7fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:41.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cce58a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x55999420f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cce58a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x55999420f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38750/0 (socket says 172.21.15.107:38750) 2024-05-09T08:36:41.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cce58a640 1 -- 172.21.15.107:0/473131670 learned_addr learned my addr 172.21.15.107:0/473131670 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1ccdd89640 1 --2- 172.21.15.107:0/473131670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207c50 0x55999420f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cced8b640 1 --2- 172.21.15.107:0/473131670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 0x55999420fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cce58a640 1 -- 172.21.15.107:0/473131670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207c50 msgr2=0x55999420f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cce58a640 1 --2- 172.21.15.107:0/473131670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207c50 0x55999420f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cce58a640 1 -- 172.21.15.107:0/473131670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 msgr2=0x55999420fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cce58a640 1 --2- 172.21.15.107:0/473131670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 0x55999420fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cce58a640 1 -- 172.21.15.107:0/473131670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559993f4a7d0 con 0x559994207330 2024-05-09T08:36:41.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.654+0000 7f1cced8b640 1 --2- 172.21.15.107:0/473131670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 0x55999420fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:36:41.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.655+0000 7f1cce58a640 1 --2- 172.21.15.107:0/473131670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x55999420f250 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4005ab0 tx=0x7f1cc40056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85312293a4e7cd4d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.655+0000 7f1cb1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:41.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.655+0000 7f1cb1ffb640 1 -- 172.21.15.107:0/473131670 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4053060 con 0x559994207330 2024-05-09T08:36:41.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.655+0000 7f1cb1ffb640 1 -- 172.21.15.107:0/473131670 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1cc4004d60 con 0x559994207330 2024-05-09T08:36:41.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.655+0000 7f1cd35de740 1 -- 172.21.15.107:0/473131670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cc40054f0 con 0x559994207330 2024-05-09T08:36:41.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.655+0000 7f1cb1ffb640 1 -- 172.21.15.107:0/473131670 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1cc405a050 con 0x559994207330 2024-05-09T08:36:41.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.655+0000 7f1cd35de740 10 client.16617 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:41.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.655+0000 7f1cd35de740 1 -- 172.21.15.107:0/473131670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559994221210 con 0x559994207330 2024-05-09T08:36:41.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.655+0000 7f1cb1ffb640 1 -- 172.21.15.107:0/473131670 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1cc404a070 con 0x559994207330 2024-05-09T08:36:41.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.655+0000 7f1cb1ffb640 1 -- 172.21.15.107:0/473131670 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f1cc40314a0 con 0x559994207330 2024-05-09T08:36:41.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.655+0000 7f1cd35de740 10 client.16617 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:41.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.655+0000 7f1cd35de740 10 client.16617 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:36:41.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.655+0000 7f1cd35de740 20 client.16617 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:41.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.655+0000 7f1cd35de740 1 --2- 172.21.15.107:0/473131670 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5599942226a0 0x559994224a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.655+0000 7f1cd35de740 4 client.16617 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:41.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.655+0000 7f1cd35de740 1 -- 172.21.15.107:0/473131670 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1cc40054f0 con 0x5599942226a0 2024-05-09T08:36:41.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.656+0000 7f1ccdd89640 1 --2- 172.21.15.107:0/473131670 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5599942226a0 0x559994224a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.656+0000 7f1cb1ffb640 10 client.16617 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:36:41.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.656+0000 7f1ccdd89640 1 --2- 172.21.15.107:0/473131670 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5599942226a0 0x559994224a90 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=30f5658db9e778eb server_cookie=1a3ece12399b656c in_seq=0 out_seq=0 2024-05-09T08:36:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.657+0000 7f1cb1ffb640 1 -- 172.21.15.107:0/473131670 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f1cc40054f0 con 0x5599942226a0 2024-05-09T08:36:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.657+0000 7f1cb1ffb640 10 client.16617 handle_command_reply: tid=0 2024-05-09T08:36:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.657+0000 7f1cd35de740 1 client.16617 shutdown 2024-05-09T08:36:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.657+0000 7f1cd35de740 1 -- 172.21.15.107:0/473131670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 msgr2=0x55999420f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.657+0000 7f1cd35de740 1 --2- 172.21.15.107:0/473131670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x55999420f250 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4005ab0 tx=0x7f1cc40056a0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.657+0000 7f1cd35de740 1 -- 172.21.15.107:0/473131670 shutdown_connections 2024-05-09T08:36:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.657+0000 7f1cd35de740 1 --2- 172.21.15.107:0/473131670 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5599942226a0 0x559994224a90 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.657+0000 7f1cd35de740 1 --2- 172.21.15.107:0/473131670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 0x55999420fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.657+0000 7f1cd35de740 1 --2- 172.21.15.107:0/473131670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207c50 0x55999420f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.657+0000 7f1cd35de740 1 --2- 172.21.15.107:0/473131670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x55999420f250 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.657+0000 7f1cd35de740 1 -- 172.21.15.107:0/473131670 >> 172.21.15.107:0/473131670 conn(0x559994203650 msgr2=0x559994209330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.657+0000 7f1cb1ffb640 0 client.16617 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:36:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.657+0000 7f1cd35de740 1 -- 172.21.15.107:0/473131670 shutdown_connections 2024-05-09T08:36:41.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.657+0000 7f1cd35de740 1 -- 172.21.15.107:0/473131670 wait complete. 2024-05-09T08:36:41.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.657+0000 7f1cd35de740 20 client.16617 trim_cache size 0 max 16384 2024-05-09T08:36:41.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cd35de740 1 Processor -- start 2024-05-09T08:36:41.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cd35de740 1 -- start start 2024-05-09T08:36:41.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cd35de740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x559994202630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cd35de740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207c50 0x5599942275e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cd35de740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 0x559994227b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cd35de740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc81b3580 con 0x559994207330 2024-05-09T08:36:41.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cd35de740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc81b0e50 con 0x559994212320 2024-05-09T08:36:41.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cd35de740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc81aeca0 con 0x559994207c50 2024-05-09T08:36:41.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cce58a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x559994202630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cce58a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x559994202630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38756/0 (socket says 172.21.15.107:38756) 2024-05-09T08:36:41.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cce58a640 1 -- 172.21.15.107:0/3091975932 learned_addr learned my addr 172.21.15.107:0/3091975932 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1ccdd89640 1 --2- 172.21.15.107:0/3091975932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207c50 0x5599942275e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cced8b640 1 --2- 172.21.15.107:0/3091975932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 0x559994227b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cce58a640 1 -- 172.21.15.107:0/3091975932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207c50 msgr2=0x5599942275e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cce58a640 1 --2- 172.21.15.107:0/3091975932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207c50 0x5599942275e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cce58a640 1 -- 172.21.15.107:0/3091975932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 msgr2=0x559994227b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cce58a640 1 --2- 172.21.15.107:0/3091975932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 0x559994227b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cce58a640 1 -- 172.21.15.107:0/3091975932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599941ff1a0 con 0x559994207330 2024-05-09T08:36:41.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1cced8b640 1 --2- 172.21.15.107:0/3091975932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 0x559994227b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:36:41.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.672+0000 7f1ccdd89640 1 --2- 172.21.15.107:0/3091975932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207c50 0x5599942275e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:41.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.673+0000 7f1cce58a640 1 --2- 172.21.15.107:0/3091975932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x559994202630 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f1cc40092a0 tx=0x7f1cc4005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168b7d7de58a1933 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.673+0000 7f1cb3fff640 1 -- 172.21.15.107:0/3091975932 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4053040 con 0x559994207330 2024-05-09T08:36:41.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.673+0000 7f1cb3fff640 1 -- 172.21.15.107:0/3091975932 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1cc404a070 con 0x559994207330 2024-05-09T08:36:41.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.673+0000 7f1cb3fff640 1 -- 172.21.15.107:0/3091975932 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4035050 con 0x559994207330 2024-05-09T08:36:41.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 -- 172.21.15.107:0/3091975932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 msgr2=0x559994202630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 --2- 172.21.15.107:0/3091975932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x559994202630 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f1cc40092a0 tx=0x7f1cc4005a20 comp rx=0 tx=0).stop 2024-05-09T08:36:41.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 -- 172.21.15.107:0/3091975932 shutdown_connections 2024-05-09T08:36:41.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 --2- 172.21.15.107:0/3091975932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994212320 0x559994227b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 --2- 172.21.15.107:0/3091975932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994207c50 0x5599942275e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 --2- 172.21.15.107:0/3091975932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x559994202630 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 -- 172.21.15.107:0/3091975932 >> 172.21.15.107:0/3091975932 conn(0x559994203650 msgr2=0x559994201f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 -- 172.21.15.107:0/3091975932 shutdown_connections 2024-05-09T08:36:41.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 -- 172.21.15.107:0/3091975932 wait complete. 2024-05-09T08:36:41.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 -- start start 2024-05-09T08:36:41.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 Processor -- start 2024-05-09T08:36:41.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x5599942278c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994207c50 0x559994224c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994212320 0x559994225190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc81af310 con 0x559994207330 2024-05-09T08:36:41.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc81b0780 con 0x559994207c50 2024-05-09T08:36:41.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc81af9e0 con 0x559994212320 2024-05-09T08:36:41.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cd35de740 10 client.0 _ll_register_callbacks cb 0x7f1cb8004aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:41.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cce58a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x5599942278c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cce58a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x5599942278c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38762/0 (socket says 172.21.15.107:38762) 2024-05-09T08:36:41.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.674+0000 7f1cce58a640 1 -- 172.21.15.107:0/2257718674 learned_addr learned my addr 172.21.15.107:0/2257718674 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1cced8b640 1 --2- 172.21.15.107:0/2257718674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994212320 0x559994225190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1ccdd89640 1 --2- 172.21.15.107:0/2257718674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994207c50 0x559994224c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1cce58a640 1 -- 172.21.15.107:0/2257718674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994212320 msgr2=0x559994225190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1cce58a640 1 --2- 172.21.15.107:0/2257718674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994212320 0x559994225190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1cce58a640 1 -- 172.21.15.107:0/2257718674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994207c50 msgr2=0x559994224c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1cce58a640 1 --2- 172.21.15.107:0/2257718674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994207c50 0x559994224c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1cce58a640 1 -- 172.21.15.107:0/2257718674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559994225880 con 0x559994207330 2024-05-09T08:36:41.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1ccdd89640 1 --2- 172.21.15.107:0/2257718674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994207c50 0x559994224c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:36:41.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1cced8b640 1 --2- 172.21.15.107:0/2257718674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994212320 0x559994225190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:41.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1cce58a640 1 --2- 172.21.15.107:0/2257718674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x5599942278c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f1cc402a9e0 tx=0x7f1cc4031730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71be7c56dab23311 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1cb1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:41.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1cb1ffb640 1 -- 172.21.15.107:0/2257718674 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4035070 con 0x559994207330 2024-05-09T08:36:41.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1cb1ffb640 1 -- 172.21.15.107:0/2257718674 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1cc40037f0 con 0x559994207330 2024-05-09T08:36:41.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1cd35de740 1 -- 172.21.15.107:0/2257718674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cc404a070 con 0x559994207330 2024-05-09T08:36:41.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1cb1ffb640 1 -- 172.21.15.107:0/2257718674 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1cc4053040 con 0x559994207330 2024-05-09T08:36:41.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1cd35de740 10 client.16623 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:41.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1cd35de740 1 -- 172.21.15.107:0/2257718674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5599942886f0 con 0x559994207330 2024-05-09T08:36:41.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.675+0000 7f1cb1ffb640 1 -- 172.21.15.107:0/2257718674 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1cc404a070 con 0x559994207330 2024-05-09T08:36:41.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.676+0000 7f1cb1ffb640 1 -- 172.21.15.107:0/2257718674 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f1cc404e880 con 0x559994207330 2024-05-09T08:36:41.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.676+0000 7f1cd35de740 10 client.16623 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:41.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.676+0000 7f1cd35de740 10 client.16623 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:36:41.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.676+0000 7f1cd35de740 20 client.16623 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:41.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.676+0000 7f1cd35de740 1 --2- 172.21.15.107:0/2257718674 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x559994288bf0 0x55999428b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.676+0000 7f1cd35de740 4 client.16623 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:36:41.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.676+0000 7f1cd35de740 1 -- 172.21.15.107:0/2257718674 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55999417a370 con 0x559994288bf0 2024-05-09T08:36:41.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.676+0000 7f1ccdd89640 1 --2- 172.21.15.107:0/2257718674 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x559994288bf0 0x55999428b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.676+0000 7f1cb1ffb640 10 client.16623 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:36:41.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.676+0000 7f1ccdd89640 1 --2- 172.21.15.107:0/2257718674 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x559994288bf0 0x55999428b010 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e1aa0e9340a0e78 server_cookie=f51b35dd4671b815 in_seq=0 out_seq=0 2024-05-09T08:36:41.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cb1ffb640 1 -- 172.21.15.107:0/2257718674 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55999417a370 con 0x559994288bf0 2024-05-09T08:36:41.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cb1ffb640 10 client.16623 handle_command_reply: tid=0 2024-05-09T08:36:41.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cd35de740 1 client.16623 shutdown 2024-05-09T08:36:41.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cd35de740 1 -- 172.21.15.107:0/2257718674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 msgr2=0x5599942278c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cd35de740 1 --2- 172.21.15.107:0/2257718674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x5599942278c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f1cc402a9e0 tx=0x7f1cc4031730 comp rx=0 tx=0).stop 2024-05-09T08:36:41.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cd35de740 1 -- 172.21.15.107:0/2257718674 shutdown_connections 2024-05-09T08:36:41.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cd35de740 1 --2- 172.21.15.107:0/2257718674 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x559994288bf0 0x55999428b010 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cd35de740 1 --2- 172.21.15.107:0/2257718674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559994212320 0x559994225190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cd35de740 1 --2- 172.21.15.107:0/2257718674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559994207c50 0x559994224c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cd35de740 1 --2- 172.21.15.107:0/2257718674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559994207330 0x5599942278c0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cd35de740 1 -- 172.21.15.107:0/2257718674 >> 172.21.15.107:0/2257718674 conn(0x559994203650 msgr2=0x559994209310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cb1ffb640 0 client.16623 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:36:41.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cd35de740 1 -- 172.21.15.107:0/2257718674 shutdown_connections 2024-05-09T08:36:41.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cd35de740 1 -- 172.21.15.107:0/2257718674 wait complete. 2024-05-09T08:36:41.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cd35de740 20 client.16623 trim_cache size 0 max 16384 2024-05-09T08:36:41.684 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:36:41.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cd07ee640 1 -- 172.21.15.107:0/4100189999 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1cb40a2d00 msgr2=0x7f1cb40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.677+0000 7f1cd07ee640 1 --2- 172.21.15.107:0/4100189999 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1cb40a2d00 0x7f1cb40a51c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x5599942047e0 tx=0x7f1cc0009c80 comp rx=0 tx=0).stop 2024-05-09T08:36:41.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.678+0000 7f1cd07ee640 1 -- 172.21.15.107:0/4100189999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cc81021d0 msgr2=0x7f1cc819a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.678+0000 7f1cd07ee640 1 --2- 172.21.15.107:0/4100189999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cc81021d0 0x7f1cc819a7f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f1cb800e9f0 tx=0x7f1cb800edf0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.678+0000 7f1cd07ee640 1 -- 172.21.15.107:0/4100189999 shutdown_connections 2024-05-09T08:36:41.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.678+0000 7f1cd07ee640 1 --2- 172.21.15.107:0/4100189999 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1cb40a2d00 0x7f1cb40a51c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.678+0000 7f1cd07ee640 1 --2- 172.21.15.107:0/4100189999 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1cc8107820 0x7f1cc819eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.678+0000 7f1cd07ee640 1 --2- 172.21.15.107:0/4100189999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cc81021d0 0x7f1cc819a7f0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.678+0000 7f1cd07ee640 1 --2- 172.21.15.107:0/4100189999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cc8101810 0x7f1cc819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.678+0000 7f1cd07ee640 1 -- 172.21.15.107:0/4100189999 >> 172.21.15.107:0/4100189999 conn(0x7f1cc8078c30 msgr2=0x7f1cc810cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.678+0000 7f1cd07ee640 1 -- 172.21.15.107:0/4100189999 shutdown_connections 2024-05-09T08:36:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.678+0000 7f1cd07ee640 1 -- 172.21.15.107:0/4100189999 wait complete. 2024-05-09T08:36:41.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:36:41.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.806+0000 7fd1a9e6b640 1 -- 172.21.15.107:0/2049814031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1a41021c0 msgr2=0x7fd1a410e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.806+0000 7fd1a9e6b640 1 --2- 172.21.15.107:0/2049814031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1a41021c0 0x7fd1a410e6d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009ca0 tx=0x7fd19c02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.806+0000 7fd1a9e6b640 1 -- 172.21.15.107:0/2049814031 shutdown_connections 2024-05-09T08:36:41.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.806+0000 7fd1a9e6b640 1 --2- 172.21.15.107:0/2049814031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1a41021c0 0x7fd1a410e6d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.806+0000 7fd1a9e6b640 1 --2- 172.21.15.107:0/2049814031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1a4101800 0x7fd1a4101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.806+0000 7fd1a9e6b640 1 --2- 172.21.15.107:0/2049814031 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1a4107810 0x7fd1a4107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.806+0000 7fd1a9e6b640 1 -- 172.21.15.107:0/2049814031 >> 172.21.15.107:0/2049814031 conn(0x7fd1a4078c80 msgr2=0x7fd1a4079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.806+0000 7fd1a9e6b640 1 -- 172.21.15.107:0/2049814031 shutdown_connections 2024-05-09T08:36:41.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.806+0000 7fd1a9e6b640 1 -- 172.21.15.107:0/2049814031 wait complete. 2024-05-09T08:36:41.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.806+0000 7fd1a9e6b640 1 Processor -- start 2024-05-09T08:36:41.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a9e6b640 1 -- start start 2024-05-09T08:36:41.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a9e6b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1a4101800 0x7fd1a4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a9e6b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1a41021c0 0x7fd1a4198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a9e6b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1a4107810 0x7fd1a419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a9e6b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a4111b20 con 0x7fd1a41021c0 2024-05-09T08:36:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a9e6b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a41119a0 con 0x7fd1a4101800 2024-05-09T08:36:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a9e6b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a4111ca0 con 0x7fd1a4107810 2024-05-09T08:36:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1a41021c0 0x7fd1a4198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1a41021c0 0x7fd1a4198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38782/0 (socket says 172.21.15.107:38782) 2024-05-09T08:36:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a2ffd640 1 -- 172.21.15.107:0/2291019014 learned_addr learned my addr 172.21.15.107:0/2291019014 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a3fff640 1 --2- 172.21.15.107:0/2291019014 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1a4107810 0x7fd1a419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a37fe640 1 --2- 172.21.15.107:0/2291019014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1a4101800 0x7fd1a4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a2ffd640 1 -- 172.21.15.107:0/2291019014 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1a4107810 msgr2=0x7fd1a419c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a2ffd640 1 --2- 172.21.15.107:0/2291019014 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1a4107810 0x7fd1a419c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a2ffd640 1 -- 172.21.15.107:0/2291019014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1a4101800 msgr2=0x7fd1a4197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a2ffd640 1 --2- 172.21.15.107:0/2291019014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1a4101800 0x7fd1a4197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a2ffd640 1 -- 172.21.15.107:0/2291019014 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a419cfb0 con 0x7fd1a41021c0 2024-05-09T08:36:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a3fff640 1 --2- 172.21.15.107:0/2291019014 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1a4107810 0x7fd1a419c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:41.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.807+0000 7fd1a37fe640 1 --2- 172.21.15.107:0/2291019014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1a4101800 0x7fd1a4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:41.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.808+0000 7fd1a2ffd640 1 --2- 172.21.15.107:0/2291019014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1a41021c0 0x7fd1a4198500 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd19800e9f0 tx=0x7fd19800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.808+0000 7fd1a0ff9640 1 -- 172.21.15.107:0/2291019014 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19800cd80 con 0x7fd1a41021c0 2024-05-09T08:36:41.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.808+0000 7fd1a0ff9640 1 -- 172.21.15.107:0/2291019014 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1980108e0 con 0x7fd1a41021c0 2024-05-09T08:36:41.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.808+0000 7fd1a9e6b640 1 -- 172.21.15.107:0/2291019014 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1a419d2a0 con 0x7fd1a41021c0 2024-05-09T08:36:41.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.808+0000 7fd1a0ff9640 1 -- 172.21.15.107:0/2291019014 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd198018920 con 0x7fd1a41021c0 2024-05-09T08:36:41.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.808+0000 7fd1a9e6b640 1 -- 172.21.15.107:0/2291019014 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1a41a4b30 con 0x7fd1a41021c0 2024-05-09T08:36:41.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.810+0000 7fd1a0ff9640 1 -- 172.21.15.107:0/2291019014 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd198010a90 con 0x7fd1a41021c0 2024-05-09T08:36:41.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.811+0000 7fd1acb79740 1 Processor -- start 2024-05-09T08:36:41.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.811+0000 7fd1acb79740 1 -- start start 2024-05-09T08:36:41.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.811+0000 7fd1a0ff9640 1 --2- 172.21.15.107:0/2291019014 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd18c0a2d00 0x7fd18c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.811+0000 7fd1acb79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 0x56139bb5b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.811+0000 7fd1acb79740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb5ba30 0x56139bb6a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.811+0000 7fd1acb79740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb6a730 0x56139bb6cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.811+0000 7fd1acb79740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56139bac0c40 con 0x56139bb5ba30 2024-05-09T08:36:41.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.811+0000 7fd1acb79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56139ba68580 con 0x56139bb5b080 2024-05-09T08:36:41.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.811+0000 7fd1acb79740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56139babcc00 con 0x56139bb6a730 2024-05-09T08:36:41.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.811+0000 7fd1a3fff640 1 --2- 172.21.15.107:0/2291019014 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd18c0a2d00 0x7fd18c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.811+0000 7fd1a2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb5ba30 0x56139bb6a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.811+0000 7fd1a0ff9640 1 -- 172.21.15.107:0/2291019014 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd1980cfbe0 con 0x7fd1a41021c0 2024-05-09T08:36:41.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.811+0000 7fd1a2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb5ba30 0x56139bb6a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38798/0 (socket says 172.21.15.107:38798) 2024-05-09T08:36:41.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.811+0000 7fd1a2ffd640 1 -- 172.21.15.107:0/3788009622 learned_addr learned my addr 172.21.15.107:0/3788009622 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.812+0000 7fd1a37fe640 1 --2- 172.21.15.107:0/3788009622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 0x56139bb5b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.812+0000 7fd1a3fff640 1 --2- 172.21.15.107:0/3788009622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb6a730 0x56139bb6cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.812+0000 7fd1a2ffd640 1 -- 172.21.15.107:0/3788009622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb6a730 msgr2=0x56139bb6cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.812+0000 7fd1a2ffd640 1 --2- 172.21.15.107:0/3788009622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb6a730 0x56139bb6cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.812+0000 7fd1a2ffd640 1 -- 172.21.15.107:0/3788009622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 msgr2=0x56139bb5b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.812+0000 7fd1a2ffd640 1 --2- 172.21.15.107:0/3788009622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 0x56139bb5b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.812+0000 7fd1a2ffd640 1 -- 172.21.15.107:0/3788009622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56139b7d43f0 con 0x56139bb5ba30 2024-05-09T08:36:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.812+0000 7fd1a2ffd640 1 --2- 172.21.15.107:0/3788009622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb5ba30 0x56139bb6a1f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd19809bd30 tx=0x7fd198098170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc736a46c8d4f6e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.812+0000 7fd1a3fff640 1 --2- 172.21.15.107:0/2291019014 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd18c0a2d00 0x7fd18c0a51c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd19c02f3a0 tx=0x7fd19c031760 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.812+0000 7fd189ffb640 1 -- 172.21.15.107:0/3788009622 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19809c4a0 con 0x56139bb5ba30 2024-05-09T08:36:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.812+0000 7fd189ffb640 1 -- 172.21.15.107:0/3788009622 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd19809bda0 con 0x56139bb5ba30 2024-05-09T08:36:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.812+0000 7fd189ffb640 1 -- 172.21.15.107:0/3788009622 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd198098410 con 0x56139bb5ba30 2024-05-09T08:36:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.813+0000 7fd1acb79740 1 -- 172.21.15.107:0/3788009622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb5ba30 msgr2=0x56139bb6a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.813+0000 7fd1acb79740 1 --2- 172.21.15.107:0/3788009622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb5ba30 0x56139bb6a1f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd19809bd30 tx=0x7fd198098170 comp rx=0 tx=0).stop 2024-05-09T08:36:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.813+0000 7fd1acb79740 1 -- 172.21.15.107:0/3788009622 shutdown_connections 2024-05-09T08:36:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.813+0000 7fd1acb79740 1 --2- 172.21.15.107:0/3788009622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb6a730 0x56139bb6cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.813+0000 7fd1acb79740 1 --2- 172.21.15.107:0/3788009622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb5ba30 0x56139bb6a1f0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.813+0000 7fd1acb79740 1 --2- 172.21.15.107:0/3788009622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 0x56139bb5b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.813+0000 7fd1acb79740 1 -- 172.21.15.107:0/3788009622 >> 172.21.15.107:0/3788009622 conn(0x56139bb57650 msgr2=0x56139bb57a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.813+0000 7fd1acb79740 1 -- 172.21.15.107:0/3788009622 shutdown_connections 2024-05-09T08:36:41.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.813+0000 7fd1acb79740 1 -- 172.21.15.107:0/3788009622 wait complete. 2024-05-09T08:36:41.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.813+0000 7fd1acb79740 1 -- start start 2024-05-09T08:36:41.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1acb79740 1 Processor -- start 2024-05-09T08:36:41.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1acb79740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb5b080 0x56139bb67670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1acb79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5ba30 0x56139bb67bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1acb79740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb6a730 0x56139bb680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1acb79740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56139bb6d4f0 con 0x56139bb5b080 2024-05-09T08:36:41.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1acb79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56139ba7bc40 con 0x56139bb5ba30 2024-05-09T08:36:41.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1acb79740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56139bb6d670 con 0x56139bb6a730 2024-05-09T08:36:41.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1acb79740 10 client.0 _ll_register_callbacks cb 0x56139b7d3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:41.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1a37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb5b080 0x56139bb67670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1a37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb5b080 0x56139bb67670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38808/0 (socket says 172.21.15.107:38808) 2024-05-09T08:36:41.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1a37fe640 1 -- 172.21.15.107:0/4051817899 learned_addr learned my addr 172.21.15.107:0/4051817899 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1a2ffd640 1 --2- 172.21.15.107:0/4051817899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5ba30 0x56139bb67bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1a3fff640 1 --2- 172.21.15.107:0/4051817899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb6a730 0x56139bb680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1a37fe640 1 -- 172.21.15.107:0/4051817899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb6a730 msgr2=0x56139bb680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1a37fe640 1 --2- 172.21.15.107:0/4051817899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb6a730 0x56139bb680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1a37fe640 1 -- 172.21.15.107:0/4051817899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5ba30 msgr2=0x56139bb67bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1a37fe640 1 --2- 172.21.15.107:0/4051817899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5ba30 0x56139bb67bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1a37fe640 1 -- 172.21.15.107:0/4051817899 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56139b89e7d0 con 0x56139bb5b080 2024-05-09T08:36:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd1a37fe640 1 --2- 172.21.15.107:0/4051817899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb5b080 0x56139bb67670 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fd194009cc0 tx=0x7fd1940039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4320efe94284d46 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.814+0000 7fd182ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.815+0000 7fd182ffd640 1 -- 172.21.15.107:0/4051817899 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194012860 con 0x56139bb5b080 2024-05-09T08:36:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.815+0000 7fd182ffd640 1 -- 172.21.15.107:0/4051817899 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd194012a00 con 0x56139bb5b080 2024-05-09T08:36:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.815+0000 7fd1acb79740 1 -- 172.21.15.107:0/4051817899 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd19809bda0 con 0x56139bb5b080 2024-05-09T08:36:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.815+0000 7fd182ffd640 1 -- 172.21.15.107:0/4051817899 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd194025050 con 0x56139bb5b080 2024-05-09T08:36:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.815+0000 7fd1acb79740 10 client.16635 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:41.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.815+0000 7fd1acb79740 1 -- 172.21.15.107:0/4051817899 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56139bb79730 con 0x56139bb5b080 2024-05-09T08:36:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.815+0000 7fd182ffd640 1 -- 172.21.15.107:0/4051817899 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd19809bda0 con 0x56139bb5b080 2024-05-09T08:36:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.815+0000 7fd182ffd640 1 -- 172.21.15.107:0/4051817899 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd19401a020 con 0x56139bb5b080 2024-05-09T08:36:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.815+0000 7fd1acb79740 10 client.16635 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.815+0000 7fd1acb79740 10 client.16635 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:36:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.815+0000 7fd1acb79740 20 client.16635 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.815+0000 7fd1acb79740 1 --2- 172.21.15.107:0/4051817899 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x56139bb7abf0 0x56139bb7d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.815+0000 7fd1acb79740 4 client.16635 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.815+0000 7fd1acb79740 1 -- 172.21.15.107:0/4051817899 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56139bace370 con 0x56139bb7abf0 2024-05-09T08:36:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.815+0000 7fd1a2ffd640 1 --2- 172.21.15.107:0/4051817899 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x56139bb7abf0 0x56139bb7d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.816+0000 7fd1a2ffd640 1 --2- 172.21.15.107:0/4051817899 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x56139bb7abf0 0x56139bb7d080 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=808950d66c112f3f server_cookie=ee7708075787ef69 in_seq=0 out_seq=0 2024-05-09T08:36:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.816+0000 7fd182ffd640 10 client.16635 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:36:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.817+0000 7fd182ffd640 1 -- 172.21.15.107:0/4051817899 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56139bace370 con 0x56139bb7abf0 2024-05-09T08:36:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.817+0000 7fd182ffd640 10 client.16635 handle_command_reply: tid=0 2024-05-09T08:36:41.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.817+0000 7fd1acb79740 1 client.16635 shutdown 2024-05-09T08:36:41.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.817+0000 7fd1acb79740 1 -- 172.21.15.107:0/4051817899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb5b080 msgr2=0x56139bb67670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.817+0000 7fd1acb79740 1 --2- 172.21.15.107:0/4051817899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb5b080 0x56139bb67670 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fd194009cc0 tx=0x7fd1940039c0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.817+0000 7fd1acb79740 1 -- 172.21.15.107:0/4051817899 shutdown_connections 2024-05-09T08:36:41.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.817+0000 7fd1acb79740 1 --2- 172.21.15.107:0/4051817899 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x56139bb7abf0 0x56139bb7d080 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.817+0000 7fd1acb79740 1 --2- 172.21.15.107:0/4051817899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb6a730 0x56139bb680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.817+0000 7fd1acb79740 1 --2- 172.21.15.107:0/4051817899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5ba30 0x56139bb67bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.817+0000 7fd1acb79740 1 --2- 172.21.15.107:0/4051817899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb5b080 0x56139bb67670 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.817+0000 7fd1acb79740 1 -- 172.21.15.107:0/4051817899 >> 172.21.15.107:0/4051817899 conn(0x56139bb57650 msgr2=0x56139bb5d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.817+0000 7fd182ffd640 0 client.16635 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:36:41.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.817+0000 7fd1acb79740 1 -- 172.21.15.107:0/4051817899 shutdown_connections 2024-05-09T08:36:41.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.817+0000 7fd1acb79740 1 -- 172.21.15.107:0/4051817899 wait complete. 2024-05-09T08:36:41.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.817+0000 7fd1acb79740 20 client.16635 trim_cache size 0 max 16384 2024-05-09T08:36:41.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.832+0000 7fd1acb79740 1 Processor -- start 2024-05-09T08:36:41.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.832+0000 7fd1acb79740 1 -- start start 2024-05-09T08:36:41.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.832+0000 7fd1acb79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 0x56139bb69da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1acb79740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb5ba30 0x56139bb763f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1acb79740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb6a730 0x56139bb82950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1acb79740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a41af250 con 0x56139bb6a730 2024-05-09T08:36:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1acb79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a41acb20 con 0x56139bb5b080 2024-05-09T08:36:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1acb79740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a41aa970 con 0x56139bb5ba30 2024-05-09T08:36:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1a3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb6a730 0x56139bb82950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1a3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb6a730 0x56139bb82950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38816/0 (socket says 172.21.15.107:38816) 2024-05-09T08:36:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1a3fff640 1 -- 172.21.15.107:0/1632646661 learned_addr learned my addr 172.21.15.107:0/1632646661 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1a2ffd640 1 --2- 172.21.15.107:0/1632646661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb5ba30 0x56139bb763f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1a37fe640 1 --2- 172.21.15.107:0/1632646661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 0x56139bb69da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1a3fff640 1 -- 172.21.15.107:0/1632646661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb5ba30 msgr2=0x56139bb763f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1a3fff640 1 --2- 172.21.15.107:0/1632646661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb5ba30 0x56139bb763f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1a3fff640 1 -- 172.21.15.107:0/1632646661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 msgr2=0x56139bb69da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1a3fff640 1 --2- 172.21.15.107:0/1632646661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 0x56139bb69da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1a3fff640 1 -- 172.21.15.107:0/1632646661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56139bb48d20 con 0x56139bb6a730 2024-05-09T08:36:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1a37fe640 1 --2- 172.21.15.107:0/1632646661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 0x56139bb69da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1a3fff640 1 --2- 172.21.15.107:0/1632646661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb6a730 0x56139bb82950 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fd19c035000 tx=0x7fd19c048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=787bb1739f916b5d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1897fa640 1 -- 172.21.15.107:0/1632646661 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19c031c40 con 0x56139bb6a730 2024-05-09T08:36:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1897fa640 1 -- 172.21.15.107:0/1632646661 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd19c036040 con 0x56139bb6a730 2024-05-09T08:36:41.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.833+0000 7fd1897fa640 1 -- 172.21.15.107:0/1632646661 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19c030db0 con 0x56139bb6a730 2024-05-09T08:36:41.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.834+0000 7fd1acb79740 1 -- 172.21.15.107:0/1632646661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb6a730 msgr2=0x56139bb82950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.834+0000 7fd1acb79740 1 --2- 172.21.15.107:0/1632646661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb6a730 0x56139bb82950 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fd19c035000 tx=0x7fd19c048000 comp rx=0 tx=0).stop 2024-05-09T08:36:41.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.834+0000 7fd1acb79740 1 -- 172.21.15.107:0/1632646661 shutdown_connections 2024-05-09T08:36:41.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.834+0000 7fd1acb79740 1 --2- 172.21.15.107:0/1632646661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb6a730 0x56139bb82950 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.834+0000 7fd1acb79740 1 --2- 172.21.15.107:0/1632646661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb5ba30 0x56139bb763f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.834+0000 7fd1acb79740 1 --2- 172.21.15.107:0/1632646661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 0x56139bb69da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.834+0000 7fd1acb79740 1 -- 172.21.15.107:0/1632646661 >> 172.21.15.107:0/1632646661 conn(0x56139bb57650 msgr2=0x56139bb62970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.834+0000 7fd1acb79740 1 -- 172.21.15.107:0/1632646661 shutdown_connections 2024-05-09T08:36:41.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.834+0000 7fd1acb79740 1 -- 172.21.15.107:0/1632646661 wait complete. 2024-05-09T08:36:41.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.835+0000 7fd1acb79740 1 -- start start 2024-05-09T08:36:41.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.835+0000 7fd1acb79740 1 Processor -- start 2024-05-09T08:36:41.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.835+0000 7fd1acb79740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 0x56139bb64220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.835+0000 7fd1acb79740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb5ba30 0x56139bb64760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.835+0000 7fd1acb79740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb6a730 0x56139bb64ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.835+0000 7fd1acb79740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a41aafe0 con 0x56139bb6a730 2024-05-09T08:36:41.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.835+0000 7fd1acb79740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a41ac450 con 0x56139bb5b080 2024-05-09T08:36:41.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.835+0000 7fd1acb79740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a41ab6b0 con 0x56139bb5ba30 2024-05-09T08:36:41.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.835+0000 7fd1acb79740 10 client.0 _ll_register_callbacks cb 0x7fd194004100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:41.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.835+0000 7fd1a3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb6a730 0x56139bb64ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.835+0000 7fd1a2ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb5ba30 0x56139bb64760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.835+0000 7fd1a3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb6a730 0x56139bb64ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38824/0 (socket says 172.21.15.107:38824) 2024-05-09T08:36:41.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.835+0000 7fd1a3fff640 1 -- 172.21.15.107:0/3826640318 learned_addr learned my addr 172.21.15.107:0/3826640318 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd1a37fe640 1 --2- 172.21.15.107:0/3826640318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 0x56139bb64220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd1a3fff640 1 -- 172.21.15.107:0/3826640318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb5ba30 msgr2=0x56139bb64760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd1a3fff640 1 --2- 172.21.15.107:0/3826640318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb5ba30 0x56139bb64760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd1a3fff640 1 -- 172.21.15.107:0/3826640318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 msgr2=0x56139bb64220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd1a3fff640 1 --2- 172.21.15.107:0/3826640318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 0x56139bb64220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd1a3fff640 1 -- 172.21.15.107:0/3826640318 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56139bbe01c0 con 0x56139bb6a730 2024-05-09T08:36:41.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd1a2ffd640 1 --2- 172.21.15.107:0/3826640318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb5ba30 0x56139bb64760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:41.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd1a37fe640 1 --2- 172.21.15.107:0/3826640318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 0x56139bb64220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:36:41.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd1a3fff640 1 --2- 172.21.15.107:0/3826640318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb6a730 0x56139bb64ca0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009960 tx=0x7fd19c009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eea101fd5e9bcac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd182ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:41.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd182ffd640 1 -- 172.21.15.107:0/3826640318 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19c003e00 con 0x56139bb6a730 2024-05-09T08:36:41.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd182ffd640 1 -- 172.21.15.107:0/3826640318 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd19c031ab0 con 0x56139bb6a730 2024-05-09T08:36:41.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd1acb79740 1 -- 172.21.15.107:0/3826640318 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd19c036040 con 0x56139bb6a730 2024-05-09T08:36:41.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd182ffd640 1 -- 172.21.15.107:0/3826640318 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd19c031cb0 con 0x56139bb6a730 2024-05-09T08:36:41.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd1acb79740 10 client.16641 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:41.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd1acb79740 1 -- 172.21.15.107:0/3826640318 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56139bbe1400 con 0x56139bb6a730 2024-05-09T08:36:41.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd182ffd640 1 -- 172.21.15.107:0/3826640318 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd19c036040 con 0x56139bb6a730 2024-05-09T08:36:41.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.836+0000 7fd182ffd640 1 -- 172.21.15.107:0/3826640318 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd19c05d8c0 con 0x56139bb6a730 2024-05-09T08:36:41.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.837+0000 7fd1acb79740 10 client.16641 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:41.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.837+0000 7fd1acb79740 10 client.16641 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:36:41.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.837+0000 7fd1acb79740 20 client.16641 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:41.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.837+0000 7fd1acb79740 1 --2- 172.21.15.107:0/3826640318 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x56139bbe1a60 0x56139bbe3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.837+0000 7fd1acb79740 4 client.16641 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:36:41.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.837+0000 7fd1acb79740 1 -- 172.21.15.107:0/3826640318 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56139bac86c0 con 0x56139bbe1a60 2024-05-09T08:36:41.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.837+0000 7fd1a37fe640 1 --2- 172.21.15.107:0/3826640318 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x56139bbe1a60 0x56139bbe3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.837+0000 7fd182ffd640 10 client.16641 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:36:41.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.837+0000 7fd1a37fe640 1 --2- 172.21.15.107:0/3826640318 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x56139bbe1a60 0x56139bbe3e80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=78ce695c8b968e66 server_cookie=b1c9053c41cc3c8b in_seq=0 out_seq=0 2024-05-09T08:36:41.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.837+0000 7fd182ffd640 1 -- 172.21.15.107:0/3826640318 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56139bac86c0 con 0x56139bbe1a60 2024-05-09T08:36:41.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.837+0000 7fd182ffd640 10 client.16641 handle_command_reply: tid=0 2024-05-09T08:36:41.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.837+0000 7fd1acb79740 1 client.16641 shutdown 2024-05-09T08:36:41.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1acb79740 1 -- 172.21.15.107:0/3826640318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb6a730 msgr2=0x56139bb64ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1acb79740 1 --2- 172.21.15.107:0/3826640318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb6a730 0x56139bb64ca0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009960 tx=0x7fd19c009380 comp rx=0 tx=0).stop 2024-05-09T08:36:41.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1acb79740 1 -- 172.21.15.107:0/3826640318 shutdown_connections 2024-05-09T08:36:41.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1acb79740 1 --2- 172.21.15.107:0/3826640318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56139bb6a730 0x56139bb64ca0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1acb79740 1 --2- 172.21.15.107:0/3826640318 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x56139bbe1a60 0x56139bbe3e80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1acb79740 1 --2- 172.21.15.107:0/3826640318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56139bb5ba30 0x56139bb64760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1acb79740 1 --2- 172.21.15.107:0/3826640318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56139bb5b080 0x56139bb64220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd182ffd640 0 client.16641 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:36:41.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1acb79740 1 -- 172.21.15.107:0/3826640318 >> 172.21.15.107:0/3826640318 conn(0x56139bb57650 msgr2=0x56139bb55f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1acb79740 1 -- 172.21.15.107:0/3826640318 shutdown_connections 2024-05-09T08:36:41.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1acb79740 1 -- 172.21.15.107:0/3826640318 wait complete. 2024-05-09T08:36:41.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1acb79740 20 client.16641 trim_cache size 0 max 16384 2024-05-09T08:36:41.843 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:36:41.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1a9e6b640 1 -- 172.21.15.107:0/2291019014 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd18c0a2d00 msgr2=0x7fd18c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1a9e6b640 1 --2- 172.21.15.107:0/2291019014 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd18c0a2d00 0x7fd18c0a51c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd19c02f3a0 tx=0x7fd19c031760 comp rx=0 tx=0).stop 2024-05-09T08:36:41.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1a9e6b640 1 -- 172.21.15.107:0/2291019014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1a41021c0 msgr2=0x7fd1a4198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1a9e6b640 1 --2- 172.21.15.107:0/2291019014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1a41021c0 0x7fd1a4198500 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd19800e9f0 tx=0x7fd19800eeb0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1a9e6b640 1 -- 172.21.15.107:0/2291019014 shutdown_connections 2024-05-09T08:36:41.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1a9e6b640 1 --2- 172.21.15.107:0/2291019014 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd18c0a2d00 0x7fd18c0a51c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1a9e6b640 1 --2- 172.21.15.107:0/2291019014 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1a4107810 0x7fd1a419c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.838+0000 7fd1a9e6b640 1 --2- 172.21.15.107:0/2291019014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1a41021c0 0x7fd1a4198500 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.839+0000 7fd1a9e6b640 1 --2- 172.21.15.107:0/2291019014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1a4101800 0x7fd1a4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.839+0000 7fd1a9e6b640 1 -- 172.21.15.107:0/2291019014 >> 172.21.15.107:0/2291019014 conn(0x7fd1a4078c80 msgr2=0x7fd1a40ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.839+0000 7fd1a9e6b640 1 -- 172.21.15.107:0/2291019014 shutdown_connections 2024-05-09T08:36:41.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.839+0000 7fd1a9e6b640 1 -- 172.21.15.107:0/2291019014 wait complete. 2024-05-09T08:36:41.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:36:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.963+0000 7f9a67aa5640 1 -- 172.21.15.107:0/450379634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a60106710 msgr2=0x7f9a60106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.963+0000 7f9a67aa5640 1 --2- 172.21.15.107:0/450379634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a60106710 0x7f9a60106af0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009930 tx=0x7f9a5c02a420 comp rx=0 tx=0).stop 2024-05-09T08:36:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.963+0000 7f9a67aa5640 1 -- 172.21.15.107:0/450379634 shutdown_connections 2024-05-09T08:36:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.963+0000 7f9a67aa5640 1 --2- 172.21.15.107:0/450379634 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a601010c0 0x7f9a6010d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.963+0000 7f9a67aa5640 1 --2- 172.21.15.107:0/450379634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a60100700 0x7f9a60100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.963+0000 7f9a67aa5640 1 --2- 172.21.15.107:0/450379634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a60106710 0x7f9a60106af0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.963+0000 7f9a67aa5640 1 -- 172.21.15.107:0/450379634 >> 172.21.15.107:0/450379634 conn(0x7f9a60077830 msgr2=0x7f9a60077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.963+0000 7f9a67aa5640 1 -- 172.21.15.107:0/450379634 shutdown_connections 2024-05-09T08:36:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.963+0000 7f9a67aa5640 1 -- 172.21.15.107:0/450379634 wait complete. 2024-05-09T08:36:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.963+0000 7f9a67aa5640 1 Processor -- start 2024-05-09T08:36:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.963+0000 7f9a67aa5640 1 -- start start 2024-05-09T08:36:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a67aa5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a60100700 0x7f9a60198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a67aa5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a601010c0 0x7f9a601985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a67aa5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a60106710 0x7f9a6019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a67aa5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9a601109d0 con 0x7f9a60106710 2024-05-09T08:36:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a67aa5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9a60110850 con 0x7f9a60100700 2024-05-09T08:36:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a67aa5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9a60110b50 con 0x7f9a601010c0 2024-05-09T08:36:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a66042640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a60106710 0x7f9a6019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a66042640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a60106710 0x7f9a6019c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38844/0 (socket says 172.21.15.107:38844) 2024-05-09T08:36:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a65841640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a60100700 0x7f9a60198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a66042640 1 -- 172.21.15.107:0/1890164935 learned_addr learned my addr 172.21.15.107:0/1890164935 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a65040640 1 --2- 172.21.15.107:0/1890164935 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a601010c0 0x7f9a601985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a66042640 1 -- 172.21.15.107:0/1890164935 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a601010c0 msgr2=0x7f9a601985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a66042640 1 --2- 172.21.15.107:0/1890164935 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a601010c0 0x7f9a601985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a66042640 1 -- 172.21.15.107:0/1890164935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a60100700 msgr2=0x7f9a60198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a66042640 1 --2- 172.21.15.107:0/1890164935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a60100700 0x7f9a60198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a66042640 1 -- 172.21.15.107:0/1890164935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6019d000 con 0x7f9a60106710 2024-05-09T08:36:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.964+0000 7f9a66042640 1 --2- 172.21.15.107:0/1890164935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a60106710 0x7f9a6019c8d0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800eb10 tx=0x7f9a5800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.965+0000 7f9a56ffd640 1 -- 172.21.15.107:0/1890164935 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a58004b20 con 0x7f9a60106710 2024-05-09T08:36:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.965+0000 7f9a67aa5640 1 -- 172.21.15.107:0/1890164935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a6019d2f0 con 0x7f9a60106710 2024-05-09T08:36:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.965+0000 7f9a56ffd640 1 -- 172.21.15.107:0/1890164935 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a58004cc0 con 0x7f9a60106710 2024-05-09T08:36:41.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.965+0000 7f9a56ffd640 1 -- 172.21.15.107:0/1890164935 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a58010690 con 0x7f9a60106710 2024-05-09T08:36:41.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.965+0000 7f9a67aa5640 1 -- 172.21.15.107:0/1890164935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a601a4b80 con 0x7f9a60106710 2024-05-09T08:36:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.968+0000 7f9a56ffd640 1 -- 172.21.15.107:0/1890164935 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f9a58005020 con 0x7f9a60106710 2024-05-09T08:36:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.968+0000 7f9a56ffd640 1 --2- 172.21.15.107:0/1890164935 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9a4c0a2d00 0x7f9a4c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.969+0000 7f9a6a7b4740 1 Processor -- start 2024-05-09T08:36:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.969+0000 7f9a56ffd640 1 -- 172.21.15.107:0/1890164935 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f9a580cfe00 con 0x7f9a60106710 2024-05-09T08:36:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.969+0000 7f9a65841640 1 --2- 172.21.15.107:0/1890164935 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9a4c0a2d00 0x7f9a4c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.969+0000 7f9a6a7b4740 1 -- start start 2024-05-09T08:36:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.969+0000 7f9a6a7b4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6330 0x5570da0d6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.969+0000 7f9a6a7b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0d6c50 0x5570da0e0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.969+0000 7f9a6a7b4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0e1320 0x5570da0e3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.969+0000 7f9a6a7b4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5570da03bc40 con 0x5570da0d6330 2024-05-09T08:36:41.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.969+0000 7f9a6a7b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5570d9fe3580 con 0x5570da0d6c50 2024-05-09T08:36:41.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.969+0000 7f9a6a7b4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5570da037c00 con 0x5570da0e1320 2024-05-09T08:36:41.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.969+0000 7f9a65841640 1 --2- 172.21.15.107:0/1890164935 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9a4c0a2d00 0x7f9a4c0a51c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009900 tx=0x7f9a5c000bc0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.969+0000 7f9a65841640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6330 0x5570da0d6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.969+0000 7f9a65841640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6330 0x5570da0d6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38848/0 (socket says 172.21.15.107:38848) 2024-05-09T08:36:41.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.969+0000 7f9a65841640 1 -- 172.21.15.107:0/814440740 learned_addr learned my addr 172.21.15.107:0/814440740 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.969+0000 7f9a66042640 1 --2- 172.21.15.107:0/814440740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0e1320 0x5570da0e3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.969+0000 7f9a65040640 1 --2- 172.21.15.107:0/814440740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0d6c50 0x5570da0e0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.970+0000 7f9a65841640 1 -- 172.21.15.107:0/814440740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0e1320 msgr2=0x5570da0e3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.970+0000 7f9a65841640 1 --2- 172.21.15.107:0/814440740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0e1320 0x5570da0e3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.970+0000 7f9a65841640 1 -- 172.21.15.107:0/814440740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0d6c50 msgr2=0x5570da0e0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.970+0000 7f9a65841640 1 --2- 172.21.15.107:0/814440740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0d6c50 0x5570da0e0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.970+0000 7f9a65841640 1 -- 172.21.15.107:0/814440740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570d9d4f3f0 con 0x5570da0d6330 2024-05-09T08:36:41.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.970+0000 7f9a65841640 1 --2- 172.21.15.107:0/814440740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6330 0x5570da0d6710 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009930 tx=0x7f9a5c03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e849b152db135ac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.970+0000 7f9a4bfff640 1 -- 172.21.15.107:0/814440740 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c04d070 con 0x5570da0d6330 2024-05-09T08:36:41.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.970+0000 7f9a4bfff640 1 -- 172.21.15.107:0/814440740 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a5c004380 con 0x5570da0d6330 2024-05-09T08:36:41.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.970+0000 7f9a4bfff640 1 -- 172.21.15.107:0/814440740 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c035040 con 0x5570da0d6330 2024-05-09T08:36:41.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.971+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/814440740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6330 msgr2=0x5570da0d6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.971+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/814440740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6330 0x5570da0d6710 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009930 tx=0x7f9a5c03d000 comp rx=0 tx=0).stop 2024-05-09T08:36:41.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.971+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/814440740 shutdown_connections 2024-05-09T08:36:41.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.971+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/814440740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0e1320 0x5570da0e3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.971+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/814440740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0d6c50 0x5570da0e0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.971+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/814440740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6330 0x5570da0d6710 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.971+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/814440740 >> 172.21.15.107:0/814440740 conn(0x5570da0d2650 msgr2=0x5570da0d2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.971+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/814440740 shutdown_connections 2024-05-09T08:36:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.971+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/814440740 wait complete. 2024-05-09T08:36:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.971+0000 7f9a6a7b4740 1 -- start start 2024-05-09T08:36:41.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.971+0000 7f9a6a7b4740 1 Processor -- start 2024-05-09T08:36:41.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a6a7b4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0d6330 0x5570da0de240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a6a7b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0d6c50 0x5570da0de780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a6a7b4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0e1320 0x5570da0decc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a6a7b4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5570da0e40b0 con 0x5570da0e1320 2024-05-09T08:36:41.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a6a7b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5570d9ff6c40 con 0x5570da0d6c50 2024-05-09T08:36:41.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a6a7b4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5570da0e4230 con 0x5570da0d6330 2024-05-09T08:36:41.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a6a7b4740 10 client.0 _ll_register_callbacks cb 0x5570d9d4ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:41.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a66042640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0e1320 0x5570da0decc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a66042640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0e1320 0x5570da0decc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38854/0 (socket says 172.21.15.107:38854) 2024-05-09T08:36:41.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a66042640 1 -- 172.21.15.107:0/624526145 learned_addr learned my addr 172.21.15.107:0/624526145 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a65841640 1 --2- 172.21.15.107:0/624526145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0d6330 0x5570da0de240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a65040640 1 --2- 172.21.15.107:0/624526145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0d6c50 0x5570da0de780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a66042640 1 -- 172.21.15.107:0/624526145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0d6330 msgr2=0x5570da0de240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a66042640 1 --2- 172.21.15.107:0/624526145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0d6330 0x5570da0de240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a66042640 1 -- 172.21.15.107:0/624526145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0d6c50 msgr2=0x5570da0de780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a66042640 1 --2- 172.21.15.107:0/624526145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0d6c50 0x5570da0de780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.972+0000 7f9a66042640 1 -- 172.21.15.107:0/624526145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570d9e197d0 con 0x5570da0e1320 2024-05-09T08:36:41.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.973+0000 7f9a66042640 1 --2- 172.21.15.107:0/624526145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0e1320 0x5570da0decc0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f9a5809bcc0 tx=0x7f9a5809bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79ee46622d1e6cb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.973+0000 7f9a497fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.973+0000 7f9a497fa640 1 -- 172.21.15.107:0/624526145 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a580b4020 con 0x5570da0e1320 2024-05-09T08:36:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.973+0000 7f9a497fa640 1 -- 172.21.15.107:0/624526145 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f9a58005510 con 0x5570da0e1320 2024-05-09T08:36:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.973+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/624526145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a5c004380 con 0x5570da0e1320 2024-05-09T08:36:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.973+0000 7f9a497fa640 1 -- 172.21.15.107:0/624526145 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9a5809bfc0 con 0x5570da0e1320 2024-05-09T08:36:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.973+0000 7f9a6a7b4740 10 client.16653 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.973+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/624526145 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5570da0f02a0 con 0x5570da0e1320 2024-05-09T08:36:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.973+0000 7f9a497fa640 1 -- 172.21.15.107:0/624526145 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a5c004380 con 0x5570da0e1320 2024-05-09T08:36:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.973+0000 7f9a497fa640 1 -- 172.21.15.107:0/624526145 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f9a5809e070 con 0x5570da0e1320 2024-05-09T08:36:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.973+0000 7f9a6a7b4740 10 client.16653 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.973+0000 7f9a6a7b4740 10 client.16653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.973+0000 7f9a6a7b4740 20 client.16653 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:41.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.973+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/624526145 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5570da0f17b0 0x5570da0f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.973+0000 7f9a6a7b4740 4 client.16653 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.973+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/624526145 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570da049370 con 0x5570da0f17b0 2024-05-09T08:36:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.974+0000 7f9a65841640 1 --2- 172.21.15.107:0/624526145 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5570da0f17b0 0x5570da0f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.974+0000 7f9a65841640 1 --2- 172.21.15.107:0/624526145 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5570da0f17b0 0x5570da0f3bd0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94dc8978e21b0c37 server_cookie=f4baab4a43a0704 in_seq=0 out_seq=0 2024-05-09T08:36:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.974+0000 7f9a497fa640 10 client.16653 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.975+0000 7f9a497fa640 1 -- 172.21.15.107:0/624526145 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5570da049370 con 0x5570da0f17b0 2024-05-09T08:36:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.975+0000 7f9a497fa640 10 client.16653 handle_command_reply: tid=0 2024-05-09T08:36:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.975+0000 7f9a6a7b4740 1 client.16653 shutdown 2024-05-09T08:36:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.975+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/624526145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0e1320 msgr2=0x5570da0decc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.975+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/624526145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0e1320 0x5570da0decc0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f9a5809bcc0 tx=0x7f9a5809bbf0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.975+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/624526145 shutdown_connections 2024-05-09T08:36:41.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.975+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/624526145 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5570da0f17b0 0x5570da0f3bd0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.975+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/624526145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0e1320 0x5570da0decc0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.975+0000 7f9a497fa640 0 client.16653 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.975+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/624526145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0d6c50 0x5570da0de780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.975+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/624526145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0d6330 0x5570da0de240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.975+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/624526145 >> 172.21.15.107:0/624526145 conn(0x5570da0d2650 msgr2=0x5570da0d8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.975+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/624526145 shutdown_connections 2024-05-09T08:36:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.975+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/624526145 wait complete. 2024-05-09T08:36:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.975+0000 7f9a6a7b4740 20 client.16653 trim_cache size 0 max 16384 2024-05-09T08:36:41.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a6a7b4740 1 Processor -- start 2024-05-09T08:36:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a6a7b4740 1 -- start start 2024-05-09T08:36:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a6a7b4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6330 0x5570da0ecf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a6a7b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0d6c50 0x5570da0dd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a6a7b4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0e1320 0x5570da0e0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a6a7b4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9a601b1490 con 0x5570da0d6330 2024-05-09T08:36:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a6a7b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9a601aed60 con 0x5570da0d6c50 2024-05-09T08:36:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a6a7b4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9a601acbb0 con 0x5570da0e1320 2024-05-09T08:36:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a65841640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6330 0x5570da0ecf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a65841640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6330 0x5570da0ecf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38866/0 (socket says 172.21.15.107:38866) 2024-05-09T08:36:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a65841640 1 -- 172.21.15.107:0/550489659 learned_addr learned my addr 172.21.15.107:0/550489659 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a65040640 1 --2- 172.21.15.107:0/550489659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0d6c50 0x5570da0dd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a66042640 1 --2- 172.21.15.107:0/550489659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0e1320 0x5570da0e0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a65841640 1 -- 172.21.15.107:0/550489659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0e1320 msgr2=0x5570da0e0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a65841640 1 --2- 172.21.15.107:0/550489659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0e1320 0x5570da0e0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a65841640 1 -- 172.21.15.107:0/550489659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0d6c50 msgr2=0x5570da0dd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a65841640 1 --2- 172.21.15.107:0/550489659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0d6c50 0x5570da0dd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a65841640 1 -- 172.21.15.107:0/550489659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570da0d8fb0 con 0x5570da0d6330 2024-05-09T08:36:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a66042640 1 --2- 172.21.15.107:0/550489659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0e1320 0x5570da0e0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.992+0000 7f9a65841640 1 --2- 172.21.15.107:0/550489659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6330 0x5570da0ecf70 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c004770 tx=0x7f9a5c003f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e2a5175c02260c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.993+0000 7f9a4b7fe640 1 -- 172.21.15.107:0/550489659 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c040040 con 0x5570da0d6330 2024-05-09T08:36:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.993+0000 7f9a4b7fe640 1 -- 172.21.15.107:0/550489659 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a5c004440 con 0x5570da0d6330 2024-05-09T08:36:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.993+0000 7f9a4b7fe640 1 -- 172.21.15.107:0/550489659 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c04d040 con 0x5570da0d6330 2024-05-09T08:36:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.993+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/550489659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6330 msgr2=0x5570da0ecf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.993+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/550489659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6330 0x5570da0ecf70 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c004770 tx=0x7f9a5c003f30 comp rx=0 tx=0).stop 2024-05-09T08:36:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.993+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/550489659 shutdown_connections 2024-05-09T08:36:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.993+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/550489659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0e1320 0x5570da0e0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.993+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/550489659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0d6c50 0x5570da0dd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.993+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/550489659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6330 0x5570da0ecf70 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.993+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/550489659 >> 172.21.15.107:0/550489659 conn(0x5570da0d2650 msgr2=0x5570da0df960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.994+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/550489659 shutdown_connections 2024-05-09T08:36:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.994+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/550489659 wait complete. 2024-05-09T08:36:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.994+0000 7f9a6a7b4740 1 -- start start 2024-05-09T08:36:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.994+0000 7f9a6a7b4740 1 Processor -- start 2024-05-09T08:36:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.994+0000 7f9a6a7b4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0d6330 0x5570da0e9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.994+0000 7f9a6a7b4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6c50 0x5570da0e4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.994+0000 7f9a6a7b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0e1320 0x5570da0e5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.994+0000 7f9a6a7b4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9a601ad220 con 0x5570da0d6c50 2024-05-09T08:36:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.994+0000 7f9a6a7b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9a601ae690 con 0x5570da0e1320 2024-05-09T08:36:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.994+0000 7f9a6a7b4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9a601ad8f0 con 0x5570da0d6330 2024-05-09T08:36:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.994+0000 7f9a6a7b4740 10 client.0 _ll_register_callbacks cb 0x7f9a58098260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a65040640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6c50 0x5570da0e4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a65040640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6c50 0x5570da0e4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38882/0 (socket says 172.21.15.107:38882) 2024-05-09T08:36:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a65040640 1 -- 172.21.15.107:0/710515981 learned_addr learned my addr 172.21.15.107:0/710515981 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a66042640 1 --2- 172.21.15.107:0/710515981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0e1320 0x5570da0e5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a65841640 1 --2- 172.21.15.107:0/710515981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0d6330 0x5570da0e9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a65040640 1 -- 172.21.15.107:0/710515981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0d6330 msgr2=0x5570da0e9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:42.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a65040640 1 --2- 172.21.15.107:0/710515981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0d6330 0x5570da0e9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:42.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a65040640 1 -- 172.21.15.107:0/710515981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0e1320 msgr2=0x5570da0e5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:42.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a65040640 1 --2- 172.21.15.107:0/710515981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0e1320 0x5570da0e5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:42.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a65040640 1 -- 172.21.15.107:0/710515981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570da0e5a90 con 0x5570da0d6c50 2024-05-09T08:36:42.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a66042640 1 --2- 172.21.15.107:0/710515981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0e1320 0x5570da0e5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:42.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a65040640 1 --2- 172.21.15.107:0/710515981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6c50 0x5570da0e4c80 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9a60102930 tx=0x7f9a50010eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa14880ecbcd0a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:42.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a497fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:42.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a497fa640 1 -- 172.21.15.107:0/710515981 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5001f070 con 0x5570da0d6c50 2024-05-09T08:36:42.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a497fa640 1 -- 172.21.15.107:0/710515981 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f9a5000fe20 con 0x5570da0d6c50 2024-05-09T08:36:42.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/710515981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a5c004440 con 0x5570da0d6c50 2024-05-09T08:36:42.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a497fa640 1 -- 172.21.15.107:0/710515981 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9a5002a050 con 0x5570da0d6c50 2024-05-09T08:36:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a6a7b4740 10 client.16659 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.995+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/710515981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5570da157e50 con 0x5570da0d6c50 2024-05-09T08:36:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.996+0000 7f9a497fa640 1 -- 172.21.15.107:0/710515981 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a5c004440 con 0x5570da0d6c50 2024-05-09T08:36:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.996+0000 7f9a497fa640 1 -- 172.21.15.107:0/710515981 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f9a5001a600 con 0x5570da0d6c50 2024-05-09T08:36:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.996+0000 7f9a6a7b4740 10 client.16659 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.996+0000 7f9a6a7b4740 10 client.16659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.996+0000 7f9a6a7b4740 20 client.16659 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.996+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/710515981 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5570da158690 0x5570da15aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.996+0000 7f9a6a7b4740 4 client.16659 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:36:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.996+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/710515981 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5570da0436c0 con 0x5570da158690 2024-05-09T08:36:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.996+0000 7f9a65841640 1 --2- 172.21.15.107:0/710515981 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5570da158690 0x5570da15aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a65841640 1 --2- 172.21.15.107:0/710515981 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5570da158690 0x5570da15aab0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc130703da1a8563 server_cookie=40c702480fb2c9db in_seq=0 out_seq=0 2024-05-09T08:36:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a497fa640 10 client.16659 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a497fa640 1 -- 172.21.15.107:0/710515981 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5570da0436c0 con 0x5570da158690 2024-05-09T08:36:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a497fa640 10 client.16659 handle_command_reply: tid=0 2024-05-09T08:36:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a6a7b4740 1 client.16659 shutdown 2024-05-09T08:36:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/710515981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6c50 msgr2=0x5570da0e4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/710515981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6c50 0x5570da0e4c80 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9a60102930 tx=0x7f9a50010eb0 comp rx=0 tx=0).stop 2024-05-09T08:36:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/710515981 shutdown_connections 2024-05-09T08:36:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/710515981 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5570da158690 0x5570da15aab0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/710515981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da0e1320 0x5570da0e5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/710515981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570da0d6c50 0x5570da0e4c80 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a6a7b4740 1 --2- 172.21.15.107:0/710515981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570da0d6330 0x5570da0e9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:42.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a497fa640 0 client.16659 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:42.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/710515981 >> 172.21.15.107:0/710515981 conn(0x5570da0d2650 msgr2=0x5570da0dea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:42.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/710515981 shutdown_connections 2024-05-09T08:36:42.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a6a7b4740 1 -- 172.21.15.107:0/710515981 wait complete. 2024-05-09T08:36:42.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.997+0000 7f9a6a7b4740 20 client.16659 trim_cache size 0 max 16384 2024-05-09T08:36:42.004 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:36:42.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.998+0000 7f9a67aa5640 1 -- 172.21.15.107:0/1890164935 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9a4c0a2d00 msgr2=0x7f9a4c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:42.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.998+0000 7f9a67aa5640 1 --2- 172.21.15.107:0/1890164935 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9a4c0a2d00 0x7f9a4c0a51c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009900 tx=0x7f9a5c000bc0 comp rx=0 tx=0).stop 2024-05-09T08:36:42.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.998+0000 7f9a67aa5640 1 -- 172.21.15.107:0/1890164935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a60106710 msgr2=0x7f9a6019c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:42.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.998+0000 7f9a67aa5640 1 --2- 172.21.15.107:0/1890164935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a60106710 0x7f9a6019c8d0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800eb10 tx=0x7f9a5800c2d0 comp rx=0 tx=0).stop 2024-05-09T08:36:42.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.998+0000 7f9a67aa5640 1 -- 172.21.15.107:0/1890164935 shutdown_connections 2024-05-09T08:36:42.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.998+0000 7f9a67aa5640 1 --2- 172.21.15.107:0/1890164935 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9a4c0a2d00 0x7f9a4c0a51c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:42.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.998+0000 7f9a67aa5640 1 --2- 172.21.15.107:0/1890164935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a60106710 0x7f9a6019c8d0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:42.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.998+0000 7f9a67aa5640 1 --2- 172.21.15.107:0/1890164935 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a601010c0 0x7f9a601985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:42.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.998+0000 7f9a67aa5640 1 --2- 172.21.15.107:0/1890164935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a60100700 0x7f9a60198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:42.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.998+0000 7f9a67aa5640 1 -- 172.21.15.107:0/1890164935 >> 172.21.15.107:0/1890164935 conn(0x7f9a60077830 msgr2=0x7f9a60068da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:42.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.998+0000 7f9a67aa5640 1 -- 172.21.15.107:0/1890164935 shutdown_connections 2024-05-09T08:36:42.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:41.998+0000 7f9a67aa5640 1 -- 172.21.15.107:0/1890164935 wait complete. 2024-05-09T08:36:42.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:42 smithi107 ceph-mon[30122]: pgmap v103: 129 pgs: 129 active+clean; 2.9 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:42.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:42 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2966612641' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:36:42.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:42 smithi157 ceph-mon[33614]: pgmap v103: 129 pgs: 129 active+clean; 2.9 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:42.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:42 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2966612641' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:36:42.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:42 smithi110 ceph-mon[33622]: pgmap v103: 129 pgs: 129 active+clean; 2.9 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:42.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:42 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2966612641' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:36:43.010 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:36:43.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:36:43.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:36:43.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.128+0000 7fb39e1f9640 1 -- 172.21.15.107:0/259555492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb398106710 msgr2=0x7fb398106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.128+0000 7fb39e1f9640 1 --2- 172.21.15.107:0/259555492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb398106710 0x7fb398106af0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb388009930 tx=0x7fb38802a420 comp rx=0 tx=0).stop 2024-05-09T08:36:43.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.128+0000 7fb39e1f9640 1 -- 172.21.15.107:0/259555492 shutdown_connections 2024-05-09T08:36:43.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.128+0000 7fb39e1f9640 1 --2- 172.21.15.107:0/259555492 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3981010c0 0x7fb39810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.128+0000 7fb39e1f9640 1 --2- 172.21.15.107:0/259555492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb398100700 0x7fb398100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.128+0000 7fb39e1f9640 1 --2- 172.21.15.107:0/259555492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb398106710 0x7fb398106af0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.128+0000 7fb39e1f9640 1 -- 172.21.15.107:0/259555492 >> 172.21.15.107:0/259555492 conn(0x7fb398077830 msgr2=0x7fb398077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:43.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.128+0000 7fb39e1f9640 1 -- 172.21.15.107:0/259555492 shutdown_connections 2024-05-09T08:36:43.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.128+0000 7fb39e1f9640 1 -- 172.21.15.107:0/259555492 wait complete. 2024-05-09T08:36:43.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.129+0000 7fb39e1f9640 1 Processor -- start 2024-05-09T08:36:43.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.129+0000 7fb39e1f9640 1 -- start start 2024-05-09T08:36:43.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.129+0000 7fb39e1f9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb398100700 0x7fb398198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.129+0000 7fb39e1f9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3981010c0 0x7fb3981985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.129+0000 7fb39e1f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb398106710 0x7fb39819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.129+0000 7fb39e1f9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb398110990 con 0x7fb398106710 2024-05-09T08:36:43.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.129+0000 7fb39e1f9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb398110810 con 0x7fb398100700 2024-05-09T08:36:43.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.129+0000 7fb39e1f9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb398110b10 con 0x7fb3981010c0 2024-05-09T08:36:43.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.130+0000 7fb397fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb398106710 0x7fb39819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.130+0000 7fb397fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb398106710 0x7fb39819c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38902/0 (socket says 172.21.15.107:38902) 2024-05-09T08:36:43.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.130+0000 7fb397fff640 1 -- 172.21.15.107:0/724305825 learned_addr learned my addr 172.21.15.107:0/724305825 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:43.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.130+0000 7fb3977fe640 1 --2- 172.21.15.107:0/724305825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb398100700 0x7fb398198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.130+0000 7fb396ffd640 1 --2- 172.21.15.107:0/724305825 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3981010c0 0x7fb3981985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.130+0000 7fb397fff640 1 -- 172.21.15.107:0/724305825 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3981010c0 msgr2=0x7fb3981985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.130+0000 7fb397fff640 1 --2- 172.21.15.107:0/724305825 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3981010c0 0x7fb3981985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.130+0000 7fb397fff640 1 -- 172.21.15.107:0/724305825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb398100700 msgr2=0x7fb398198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.130+0000 7fb397fff640 1 --2- 172.21.15.107:0/724305825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb398100700 0x7fb398198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.130+0000 7fb397fff640 1 -- 172.21.15.107:0/724305825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39819d050 con 0x7fb398106710 2024-05-09T08:36:43.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.130+0000 7fb397fff640 1 --2- 172.21.15.107:0/724305825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb398106710 0x7fb39819c920 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fb39000d750 tx=0x7fb39000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:43.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.130+0000 7fb394ff9640 1 -- 172.21.15.107:0/724305825 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb390017070 con 0x7fb398106710 2024-05-09T08:36:43.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.130+0000 7fb394ff9640 1 -- 172.21.15.107:0/724305825 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb3900054d0 con 0x7fb398106710 2024-05-09T08:36:43.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.130+0000 7fb39e1f9640 1 -- 172.21.15.107:0/724305825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb39819d340 con 0x7fb398106710 2024-05-09T08:36:43.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.130+0000 7fb394ff9640 1 -- 172.21.15.107:0/724305825 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb390004c60 con 0x7fb398106710 2024-05-09T08:36:43.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.130+0000 7fb39e1f9640 1 -- 172.21.15.107:0/724305825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3981a4bd0 con 0x7fb398106710 2024-05-09T08:36:43.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.133+0000 7fb3a0fe1740 1 Processor -- start 2024-05-09T08:36:43.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.133+0000 7fb394ff9640 1 -- 172.21.15.107:0/724305825 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb390004e00 con 0x7fb398106710 2024-05-09T08:36:43.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.133+0000 7fb3a0fe1740 1 -- start start 2024-05-09T08:36:43.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb3a0fe1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb060b0 0x55ad6fb043f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb3a0fe1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb049c0 0x55ad6fb04e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb3a0fe1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb0f740 0x55ad6fb11b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb3a0fe1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ad6fa69990 con 0x55ad6fb0f740 2024-05-09T08:36:43.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb3a0fe1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ad6fa115b0 con 0x55ad6fb049c0 2024-05-09T08:36:43.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb3a0fe1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ad6fa65950 con 0x55ad6fb060b0 2024-05-09T08:36:43.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb396ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb0f740 0x55ad6fb11b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb396ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb0f740 0x55ad6fb11b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38918/0 (socket says 172.21.15.107:38918) 2024-05-09T08:36:43.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb396ffd640 1 -- 172.21.15.107:0/1125884256 learned_addr learned my addr 172.21.15.107:0/1125884256 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:43.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb396ffd640 1 --2- 172.21.15.107:0/1125884256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb060b0 0x55ad6fb043f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb3977fe640 1 --2- 172.21.15.107:0/1125884256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb049c0 0x55ad6fb04e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb396ffd640 1 -- 172.21.15.107:0/1125884256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb060b0 msgr2=0x55ad6fb043f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:43.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb396ffd640 1 --2- 172.21.15.107:0/1125884256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb060b0 0x55ad6fb043f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb396ffd640 1 -- 172.21.15.107:0/1125884256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb049c0 msgr2=0x55ad6fb04e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:43.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb396ffd640 1 --2- 172.21.15.107:0/1125884256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb049c0 0x55ad6fb04e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb394ff9640 1 --2- 172.21.15.107:0/724305825 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb3800a2d00 0x7fb3800a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb396ffd640 1 -- 172.21.15.107:0/1125884256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad6f8457e0 con 0x55ad6fb0f740 2024-05-09T08:36:43.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb394ff9640 1 -- 172.21.15.107:0/724305825 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb3900cf0e0 con 0x7fb398106710 2024-05-09T08:36:43.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.134+0000 7fb396ffd640 1 --2- 172.21.15.107:0/724305825 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb3800a2d00 0x7fb3800a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.135+0000 7fb396ffd640 1 --2- 172.21.15.107:0/1125884256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb0f740 0x55ad6fb11b30 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fb38c014cd0 tx=0x7fb38c0097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=579f25efc46f3bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:43.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.135+0000 7fb37dffb640 1 -- 172.21.15.107:0/1125884256 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c014330 con 0x55ad6fb0f740 2024-05-09T08:36:43.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.135+0000 7fb37dffb640 1 -- 172.21.15.107:0/1125884256 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb38c00fbf0 con 0x55ad6fb0f740 2024-05-09T08:36:43.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.135+0000 7fb37dffb640 1 -- 172.21.15.107:0/1125884256 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c011760 con 0x55ad6fb0f740 2024-05-09T08:36:43.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.136+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/1125884256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb0f740 msgr2=0x55ad6fb11b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.136+0000 7fb396ffd640 1 --2- 172.21.15.107:0/724305825 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb3800a2d00 0x7fb3800a51c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb38c01e000 tx=0x7fb38c0119e0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:43.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.136+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/1125884256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb0f740 0x55ad6fb11b30 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fb38c014cd0 tx=0x7fb38c0097d0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.136+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/1125884256 shutdown_connections 2024-05-09T08:36:43.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.136+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/1125884256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb0f740 0x55ad6fb11b30 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.136+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/1125884256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb049c0 0x55ad6fb04e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.136+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/1125884256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb060b0 0x55ad6fb043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.136+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/1125884256 >> 172.21.15.107:0/1125884256 conn(0x55ad6facd0a0 msgr2=0x55ad6fb00f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:43.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.136+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/1125884256 shutdown_connections 2024-05-09T08:36:43.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.136+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/1125884256 wait complete. 2024-05-09T08:36:43.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.136+0000 7fb3a0fe1740 1 -- start start 2024-05-09T08:36:43.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.136+0000 7fb3a0fe1740 1 Processor -- start 2024-05-09T08:36:43.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb3a0fe1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb049c0 0x55ad6fb0c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb3a0fe1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb060b0 0x55ad6fb0cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb3a0fe1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb0f740 0x55ad6fb0d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb3a0fe1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ad6fb125d0 con 0x55ad6fb060b0 2024-05-09T08:36:43.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb3a0fe1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ad6fa24b40 con 0x55ad6fb049c0 2024-05-09T08:36:43.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb3a0fe1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ad6fb12750 con 0x55ad6fb0f740 2024-05-09T08:36:43.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb3a0fe1740 10 client.0 _ll_register_callbacks cb 0x55ad6fa190f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:43.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb396ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb060b0 0x55ad6fb0cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb3977fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb049c0 0x55ad6fb0c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb396ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb060b0 0x55ad6fb0cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38920/0 (socket says 172.21.15.107:38920) 2024-05-09T08:36:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb396ffd640 1 -- 172.21.15.107:0/550258723 learned_addr learned my addr 172.21.15.107:0/550258723 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb397fff640 1 --2- 172.21.15.107:0/550258723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb0f740 0x55ad6fb0d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb396ffd640 1 -- 172.21.15.107:0/550258723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb0f740 msgr2=0x55ad6fb0d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb396ffd640 1 --2- 172.21.15.107:0/550258723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb0f740 0x55ad6fb0d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb396ffd640 1 -- 172.21.15.107:0/550258723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb049c0 msgr2=0x55ad6fb0c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb396ffd640 1 --2- 172.21.15.107:0/550258723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb049c0 0x55ad6fb0c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb396ffd640 1 -- 172.21.15.107:0/550258723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad6f8a5f70 con 0x55ad6fb060b0 2024-05-09T08:36:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb3977fe640 1 --2- 172.21.15.107:0/550258723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb049c0 0x55ad6fb0c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb396ffd640 1 --2- 172.21.15.107:0/550258723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb060b0 0x55ad6fb0cb00 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00a010 tx=0x7fb38c00e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e660d65225b4116 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.137+0000 7fb376ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.138+0000 7fb376ffd640 1 -- 172.21.15.107:0/550258723 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c00fda0 con 0x55ad6fb060b0 2024-05-09T08:36:43.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.138+0000 7fb376ffd640 1 -- 172.21.15.107:0/550258723 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb38c011440 con 0x55ad6fb060b0 2024-05-09T08:36:43.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.138+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/550258723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb38c00fbf0 con 0x55ad6fb060b0 2024-05-09T08:36:43.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.138+0000 7fb376ffd640 1 -- 172.21.15.107:0/550258723 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb38c03e050 con 0x55ad6fb060b0 2024-05-09T08:36:43.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.138+0000 7fb3a0fe1740 10 client.16671 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:43.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.138+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/550258723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ad6f73d170 con 0x55ad6fb060b0 2024-05-09T08:36:43.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.138+0000 7fb376ffd640 1 -- 172.21.15.107:0/550258723 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb38c00fbf0 con 0x55ad6fb060b0 2024-05-09T08:36:43.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.138+0000 7fb376ffd640 1 -- 172.21.15.107:0/550258723 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb38c0124e0 con 0x55ad6fb060b0 2024-05-09T08:36:43.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.138+0000 7fb3a0fe1740 10 client.16671 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:43.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.138+0000 7fb3a0fe1740 10 client.16671 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:43.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.138+0000 7fb3a0fe1740 20 client.16671 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:43.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.138+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/550258723 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ad6fb1f6b0 0x55ad6fb21aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.138+0000 7fb3a0fe1740 4 client.16671 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:43.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.138+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/550258723 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad6fa770c0 con 0x55ad6fb1f6b0 2024-05-09T08:36:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.139+0000 7fb3977fe640 1 --2- 172.21.15.107:0/550258723 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ad6fb1f6b0 0x55ad6fb21aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.139+0000 7fb3977fe640 1 --2- 172.21.15.107:0/550258723 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ad6fb1f6b0 0x55ad6fb21aa0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a883853e034a7be server_cookie=b212de6b2147cfc7 in_seq=0 out_seq=0 2024-05-09T08:36:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.139+0000 7fb376ffd640 10 client.16671 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.140+0000 7fb376ffd640 1 -- 172.21.15.107:0/550258723 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ad6fa770c0 con 0x55ad6fb1f6b0 2024-05-09T08:36:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.140+0000 7fb376ffd640 10 client.16671 handle_command_reply: tid=0 2024-05-09T08:36:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.140+0000 7fb3a0fe1740 1 client.16671 shutdown 2024-05-09T08:36:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.140+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/550258723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb060b0 msgr2=0x55ad6fb0cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.140+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/550258723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb060b0 0x55ad6fb0cb00 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00a010 tx=0x7fb38c00e000 comp rx=0 tx=0).stop 2024-05-09T08:36:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.140+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/550258723 shutdown_connections 2024-05-09T08:36:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.140+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/550258723 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ad6fb1f6b0 0x55ad6fb21aa0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.140+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/550258723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb0f740 0x55ad6fb0d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.140+0000 7fb376ffd640 0 client.16671 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:43.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.140+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/550258723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb060b0 0x55ad6fb0cb00 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.140+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/550258723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb049c0 0x55ad6fb0c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.140+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/550258723 >> 172.21.15.107:0/550258723 conn(0x55ad6facd0a0 msgr2=0x55ad6fb06d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:43.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.140+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/550258723 shutdown_connections 2024-05-09T08:36:43.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.140+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/550258723 wait complete. 2024-05-09T08:36:43.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.140+0000 7fb3a0fe1740 20 client.16671 trim_cache size 0 max 16384 2024-05-09T08:36:43.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.160+0000 7fb3a0fe1740 1 Processor -- start 2024-05-09T08:36:43.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.160+0000 7fb3a0fe1740 1 -- start start 2024-05-09T08:36:43.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.160+0000 7fb3a0fe1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb049c0 0x55ad6faffe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.160+0000 7fb3a0fe1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb060b0 0x55ad6fb1b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.160+0000 7fb3a0fe1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb0f740 0x55ad6fb0cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.160+0000 7fb3a0fe1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb3981b1480 con 0x55ad6fb049c0 2024-05-09T08:36:43.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.160+0000 7fb3a0fe1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb3981aed50 con 0x55ad6fb0f740 2024-05-09T08:36:43.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.160+0000 7fb3a0fe1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb3981acba0 con 0x55ad6fb060b0 2024-05-09T08:36:43.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.161+0000 7fb3977fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb049c0 0x55ad6faffe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.161+0000 7fb3977fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb049c0 0x55ad6faffe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38926/0 (socket says 172.21.15.107:38926) 2024-05-09T08:36:43.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.161+0000 7fb3977fe640 1 -- 172.21.15.107:0/2516489836 learned_addr learned my addr 172.21.15.107:0/2516489836 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:43.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.161+0000 7fb396ffd640 1 --2- 172.21.15.107:0/2516489836 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb060b0 0x55ad6fb1b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.161+0000 7fb397fff640 1 --2- 172.21.15.107:0/2516489836 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb0f740 0x55ad6fb0cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.161+0000 7fb396ffd640 1 -- 172.21.15.107:0/2516489836 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb0f740 msgr2=0x55ad6fb0cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:43.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.161+0000 7fb396ffd640 1 --2- 172.21.15.107:0/2516489836 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb0f740 0x55ad6fb0cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.161+0000 7fb396ffd640 1 -- 172.21.15.107:0/2516489836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb049c0 msgr2=0x55ad6faffe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:43.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.161+0000 7fb396ffd640 1 --2- 172.21.15.107:0/2516489836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb049c0 0x55ad6faffe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.161+0000 7fb396ffd640 1 -- 172.21.15.107:0/2516489836 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad6fb22100 con 0x55ad6fb060b0 2024-05-09T08:36:43.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.161+0000 7fb3977fe640 1 --2- 172.21.15.107:0/2516489836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb049c0 0x55ad6faffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:43.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.161+0000 7fb396ffd640 1 --2- 172.21.15.107:0/2516489836 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb060b0 0x55ad6fb1b500 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb38c011ee0 tx=0x7fb38c0097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29b022b42eacd217 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:43.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.161+0000 7fb37d7fa640 1 -- 172.21.15.107:0/2516489836 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c012650 con 0x55ad6fb060b0 2024-05-09T08:36:43.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.162+0000 7fb37d7fa640 1 -- 172.21.15.107:0/2516489836 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb38c00fbc0 con 0x55ad6fb060b0 2024-05-09T08:36:43.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.162+0000 7fb37d7fa640 1 -- 172.21.15.107:0/2516489836 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c003bd0 con 0x55ad6fb060b0 2024-05-09T08:36:43.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.162+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/2516489836 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb060b0 msgr2=0x55ad6fb1b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.162+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/2516489836 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb060b0 0x55ad6fb1b500 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb38c011ee0 tx=0x7fb38c0097d0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.163+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/2516489836 shutdown_connections 2024-05-09T08:36:43.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.163+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/2516489836 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb0f740 0x55ad6fb0cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.163+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/2516489836 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb060b0 0x55ad6fb1b500 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.163+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/2516489836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb049c0 0x55ad6faffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.163+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/2516489836 >> 172.21.15.107:0/2516489836 conn(0x55ad6facd0a0 msgr2=0x55ad6fb0c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:43.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.163+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/2516489836 shutdown_connections 2024-05-09T08:36:43.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.163+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/2516489836 wait complete. 2024-05-09T08:36:43.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.163+0000 7fb3a0fe1740 1 -- start start 2024-05-09T08:36:43.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.163+0000 7fb3a0fe1740 1 Processor -- start 2024-05-09T08:36:43.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb3a0fe1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb049c0 0x55ad6fb13b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb3a0fe1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb060b0 0x55ad6fb140b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb3a0fe1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb0f740 0x55ad6fb84ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb3a0fe1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb3981ad210 con 0x55ad6fb060b0 2024-05-09T08:36:43.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb3a0fe1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb3981ae680 con 0x55ad6fb0f740 2024-05-09T08:36:43.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb3a0fe1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb3981ad8e0 con 0x55ad6fb049c0 2024-05-09T08:36:43.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb3a0fe1740 10 client.0 _ll_register_callbacks cb 0x7fb388005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:43.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb396ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb060b0 0x55ad6fb140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb396ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb060b0 0x55ad6fb140b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38940/0 (socket says 172.21.15.107:38940) 2024-05-09T08:36:43.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb396ffd640 1 -- 172.21.15.107:0/3908625719 learned_addr learned my addr 172.21.15.107:0/3908625719 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:43.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb3977fe640 1 --2- 172.21.15.107:0/3908625719 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb049c0 0x55ad6fb13b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb397fff640 1 --2- 172.21.15.107:0/3908625719 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb0f740 0x55ad6fb84ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb396ffd640 1 -- 172.21.15.107:0/3908625719 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb049c0 msgr2=0x55ad6fb13b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:43.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb396ffd640 1 --2- 172.21.15.107:0/3908625719 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb049c0 0x55ad6fb13b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb396ffd640 1 -- 172.21.15.107:0/3908625719 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb0f740 msgr2=0x55ad6fb84ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:43.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb396ffd640 1 --2- 172.21.15.107:0/3908625719 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb0f740 0x55ad6fb84ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb396ffd640 1 -- 172.21.15.107:0/3908625719 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad6fb850e0 con 0x55ad6fb060b0 2024-05-09T08:36:43.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb397fff640 1 --2- 172.21.15.107:0/3908625719 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb0f740 0x55ad6fb84ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:43.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb396ffd640 1 --2- 172.21.15.107:0/3908625719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb060b0 0x55ad6fb140b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0095a0 tx=0x7fb38c00ff80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef13a45028cda65 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:43.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.164+0000 7fb376ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:43.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.165+0000 7fb376ffd640 1 -- 172.21.15.107:0/3908625719 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c012660 con 0x55ad6fb060b0 2024-05-09T08:36:43.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.165+0000 7fb376ffd640 1 -- 172.21.15.107:0/3908625719 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb38c011440 con 0x55ad6fb060b0 2024-05-09T08:36:43.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.165+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/3908625719 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb38c00fbc0 con 0x55ad6fb060b0 2024-05-09T08:36:43.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.165+0000 7fb376ffd640 1 -- 172.21.15.107:0/3908625719 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb38c04f050 con 0x55ad6fb060b0 2024-05-09T08:36:43.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.165+0000 7fb3a0fe1740 10 client.16677 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:43.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.165+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/3908625719 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ad6fb86420 con 0x55ad6fb060b0 2024-05-09T08:36:43.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.165+0000 7fb376ffd640 1 -- 172.21.15.107:0/3908625719 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb38c00fbc0 con 0x55ad6fb060b0 2024-05-09T08:36:43.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.165+0000 7fb376ffd640 1 -- 172.21.15.107:0/3908625719 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb38c0377d0 con 0x55ad6fb060b0 2024-05-09T08:36:43.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.165+0000 7fb3a0fe1740 10 client.16677 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:43.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.165+0000 7fb3a0fe1740 10 client.16677 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:43.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.165+0000 7fb3a0fe1740 20 client.16677 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:43.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.165+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/3908625719 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ad6fb86b10 0x55ad6fb88f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.165+0000 7fb3a0fe1740 4 client.16677 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:36:43.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.165+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/3908625719 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ad6fa71410 con 0x55ad6fb86b10 2024-05-09T08:36:43.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.166+0000 7fb3977fe640 1 --2- 172.21.15.107:0/3908625719 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ad6fb86b10 0x55ad6fb88f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.166+0000 7fb3977fe640 1 --2- 172.21.15.107:0/3908625719 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ad6fb86b10 0x55ad6fb88f30 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f200e5fc6a1a61b server_cookie=8757ce580dfb03cf in_seq=0 out_seq=0 2024-05-09T08:36:43.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.166+0000 7fb376ffd640 10 client.16677 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:43.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.167+0000 7fb376ffd640 1 -- 172.21.15.107:0/3908625719 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ad6fa71410 con 0x55ad6fb86b10 2024-05-09T08:36:43.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.167+0000 7fb376ffd640 10 client.16677 handle_command_reply: tid=0 2024-05-09T08:36:43.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.167+0000 7fb3a0fe1740 1 client.16677 shutdown 2024-05-09T08:36:43.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.168+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/3908625719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb060b0 msgr2=0x55ad6fb140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.168+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/3908625719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb060b0 0x55ad6fb140b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0095a0 tx=0x7fb38c00ff80 comp rx=0 tx=0).stop 2024-05-09T08:36:43.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.168+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/3908625719 shutdown_connections 2024-05-09T08:36:43.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.168+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/3908625719 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ad6fb86b10 0x55ad6fb88f30 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.168+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/3908625719 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad6fb0f740 0x55ad6fb84ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.168+0000 7fb376ffd640 0 client.16677 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:43.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.168+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/3908625719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ad6fb060b0 0x55ad6fb140b0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.168+0000 7fb3a0fe1740 1 --2- 172.21.15.107:0/3908625719 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ad6fb049c0 0x55ad6fb13b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.168+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/3908625719 >> 172.21.15.107:0/3908625719 conn(0x55ad6facd0a0 msgr2=0x55ad6fb06d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:43.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.168+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/3908625719 shutdown_connections 2024-05-09T08:36:43.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.168+0000 7fb3a0fe1740 1 -- 172.21.15.107:0/3908625719 wait complete. 2024-05-09T08:36:43.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.169+0000 7fb3a0fe1740 20 client.16677 trim_cache size 0 max 16384 2024-05-09T08:36:43.179 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:36:43.179 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:36:43.179 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "e92546ad-0df6-4781-9134-8e0bce8dd1a1", 2024-05-09T08:36:43.179 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:36:43.180 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:36:43.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.169+0000 7fb39e1f9640 1 -- 172.21.15.107:0/724305825 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb3800a2d00 msgr2=0x7fb3800a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.169+0000 7fb39e1f9640 1 --2- 172.21.15.107:0/724305825 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb3800a2d00 0x7fb3800a51c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb38c01e000 tx=0x7fb38c0119e0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.169+0000 7fb39e1f9640 1 -- 172.21.15.107:0/724305825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb398106710 msgr2=0x7fb39819c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.169+0000 7fb39e1f9640 1 --2- 172.21.15.107:0/724305825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb398106710 0x7fb39819c920 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fb39000d750 tx=0x7fb39000dc10 comp rx=0 tx=0).stop 2024-05-09T08:36:43.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.169+0000 7fb39e1f9640 1 -- 172.21.15.107:0/724305825 shutdown_connections 2024-05-09T08:36:43.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.169+0000 7fb39e1f9640 1 --2- 172.21.15.107:0/724305825 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb3800a2d00 0x7fb3800a51c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.169+0000 7fb39e1f9640 1 --2- 172.21.15.107:0/724305825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb398106710 0x7fb39819c920 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.170+0000 7fb39e1f9640 1 --2- 172.21.15.107:0/724305825 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3981010c0 0x7fb3981985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.170+0000 7fb39e1f9640 1 --2- 172.21.15.107:0/724305825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb398100700 0x7fb398198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.170+0000 7fb39e1f9640 1 -- 172.21.15.107:0/724305825 >> 172.21.15.107:0/724305825 conn(0x7fb398077830 msgr2=0x7fb398068dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:43.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.170+0000 7fb39e1f9640 1 -- 172.21.15.107:0/724305825 shutdown_connections 2024-05-09T08:36:43.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.170+0000 7fb39e1f9640 1 -- 172.21.15.107:0/724305825 wait complete. 2024-05-09T08:36:43.182 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:36:43.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:36:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.296+0000 7f2ece9c1640 1 -- 172.21.15.107:0/2971651280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec8100760 msgr2=0x7f2ec8100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.296+0000 7f2ece9c1640 1 --2- 172.21.15.107:0/2971651280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec8100760 0x7f2ec8100be0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc009930 tx=0x7f2ebc02a420 comp rx=0 tx=0).stop 2024-05-09T08:36:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.296+0000 7f2ece9c1640 1 -- 172.21.15.107:0/2971651280 shutdown_connections 2024-05-09T08:36:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.296+0000 7f2ece9c1640 1 --2- 172.21.15.107:0/2971651280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ec8101120 0x7f2ec810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.296+0000 7f2ece9c1640 1 --2- 172.21.15.107:0/2971651280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec8100760 0x7f2ec8100be0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.296+0000 7f2ece9c1640 1 --2- 172.21.15.107:0/2971651280 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ec8106720 0x7f2ec8106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.296+0000 7f2ece9c1640 1 -- 172.21.15.107:0/2971651280 >> 172.21.15.107:0/2971651280 conn(0x7f2ec8077890 msgr2=0x7f2ec8077ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.296+0000 7f2ece9c1640 1 -- 172.21.15.107:0/2971651280 shutdown_connections 2024-05-09T08:36:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.296+0000 7f2ece9c1640 1 -- 172.21.15.107:0/2971651280 wait complete. 2024-05-09T08:36:43.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.297+0000 7f2ece9c1640 1 Processor -- start 2024-05-09T08:36:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.297+0000 7f2ece9c1640 1 -- start start 2024-05-09T08:36:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.297+0000 7f2ece9c1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ec8100760 0x7f2ec819a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.297+0000 7f2ece9c1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ec8101120 0x7f2ec819a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.297+0000 7f2ece9c1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec8106720 0x7f2ec819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.297+0000 7f2ece9c1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec81109e0 con 0x7f2ec8106720 2024-05-09T08:36:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.297+0000 7f2ece9c1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec8110860 con 0x7f2ec8101120 2024-05-09T08:36:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.297+0000 7f2ece9c1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec8110b60 con 0x7f2ec8100760 2024-05-09T08:36:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.297+0000 7f2eccf5e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec8106720 0x7f2ec819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.297+0000 7f2eccf5e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec8106720 0x7f2ec819eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38960/0 (socket says 172.21.15.107:38960) 2024-05-09T08:36:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.297+0000 7f2eccf5e640 1 -- 172.21.15.107:0/1307531795 learned_addr learned my addr 172.21.15.107:0/1307531795 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.297+0000 7f2ec77fe640 1 --2- 172.21.15.107:0/1307531795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ec8101120 0x7f2ec819a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.297+0000 7f2ec7fff640 1 --2- 172.21.15.107:0/1307531795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ec8100760 0x7f2ec819a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.298+0000 7f2eccf5e640 1 -- 172.21.15.107:0/1307531795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ec8100760 msgr2=0x7f2ec819a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.298+0000 7f2eccf5e640 1 --2- 172.21.15.107:0/1307531795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ec8100760 0x7f2ec819a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.298+0000 7f2eccf5e640 1 -- 172.21.15.107:0/1307531795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ec8101120 msgr2=0x7f2ec819a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.298+0000 7f2eccf5e640 1 --2- 172.21.15.107:0/1307531795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ec8101120 0x7f2ec819a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.298+0000 7f2eccf5e640 1 -- 172.21.15.107:0/1307531795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec819f2c0 con 0x7f2ec8106720 2024-05-09T08:36:43.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.298+0000 7f2eccf5e640 1 --2- 172.21.15.107:0/1307531795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec8106720 0x7f2ec819eb90 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000cab0 tx=0x7f2ec000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:43.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.298+0000 7f2ec57fa640 1 -- 172.21.15.107:0/1307531795 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec000ae20 con 0x7f2ec8106720 2024-05-09T08:36:43.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.298+0000 7f2ece9c1640 1 -- 172.21.15.107:0/1307531795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ec819f5b0 con 0x7f2ec8106720 2024-05-09T08:36:43.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.298+0000 7f2ec57fa640 1 -- 172.21.15.107:0/1307531795 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ec000f900 con 0x7f2ec8106720 2024-05-09T08:36:43.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.298+0000 7f2ec57fa640 1 -- 172.21.15.107:0/1307531795 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0018a00 con 0x7f2ec8106720 2024-05-09T08:36:43.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.298+0000 7f2ece9c1640 1 -- 172.21.15.107:0/1307531795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ec81a6e40 con 0x7f2ec8106720 2024-05-09T08:36:43.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.300+0000 7f2ec57fa640 1 -- 172.21.15.107:0/1307531795 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f2ec000fab0 con 0x7f2ec8106720 2024-05-09T08:36:43.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.301+0000 7f2ed17ae740 1 Processor -- start 2024-05-09T08:36:43.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.301+0000 7f2ed17ae740 1 -- start start 2024-05-09T08:36:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.301+0000 7f2ed17ae740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a295280 0x56045a295660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.301+0000 7f2ed17ae740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.301+0000 7f2ed17ae740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a2a0270 0x56045a2a2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.301+0000 7f2ed17ae740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56045a1fad00 con 0x56045a295ba0 2024-05-09T08:36:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.301+0000 7f2ed17ae740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56045a1a24c0 con 0x56045a295280 2024-05-09T08:36:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.301+0000 7f2ed17ae740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56045a1f6cc0 con 0x56045a2a0270 2024-05-09T08:36:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.301+0000 7f2ec57fa640 1 --2- 172.21.15.107:0/1307531795 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2eb00a2d00 0x7f2eb00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.301+0000 7f2ec7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.301+0000 7f2ec77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a295280 0x56045a295660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.301+0000 7f2ec7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38966/0 (socket says 172.21.15.107:38966) 2024-05-09T08:36:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.301+0000 7f2eccf5e640 1 --2- 172.21.15.107:0/1307531795 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2eb00a2d00 0x7f2eb00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.301+0000 7f2ec7fff640 1 -- 172.21.15.107:0/3831756662 learned_addr learned my addr 172.21.15.107:0/3831756662 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.302+0000 7f2ec77fe640 1 --2- 172.21.15.107:0/3831756662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a2a0270 0x56045a2a2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.302+0000 7f2ec57fa640 1 -- 172.21.15.107:0/1307531795 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f2ec00cfe30 con 0x7f2ec8106720 2024-05-09T08:36:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.302+0000 7f2ec7fff640 1 -- 172.21.15.107:0/3831756662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a2a0270 msgr2=0x56045a2a2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.302+0000 7f2ec7fff640 1 --2- 172.21.15.107:0/3831756662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a2a0270 0x56045a2a2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.302+0000 7f2ec7fff640 1 -- 172.21.15.107:0/3831756662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a295280 msgr2=0x56045a295660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.302+0000 7f2ec7fff640 1 --2- 172.21.15.107:0/3831756662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a295280 0x56045a295660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.302+0000 7f2ec7fff640 1 -- 172.21.15.107:0/3831756662 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560459fd87d0 con 0x56045a295ba0 2024-05-09T08:36:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.302+0000 7f2ec77fe640 1 --2- 172.21.15.107:0/3831756662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a2a0270 0x56045a2a2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.302+0000 7f2ec7fff640 1 --2- 172.21.15.107:0/3831756662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29fd30 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009ca0 tx=0x7f2eb8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d1c37fe5aec419 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.302+0000 7f2eccf5e640 1 --2- 172.21.15.107:0/1307531795 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2eb00a2d00 0x7f2eb00a51c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000a920 tx=0x7f2ec000a890 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.302+0000 7f2eae7fc640 1 -- 172.21.15.107:0/3831756662 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb801e070 con 0x56045a295ba0 2024-05-09T08:36:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.302+0000 7f2eae7fc640 1 -- 172.21.15.107:0/3831756662 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2eb8019ae0 con 0x56045a295ba0 2024-05-09T08:36:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.302+0000 7f2eae7fc640 1 -- 172.21.15.107:0/3831756662 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb800e990 con 0x56045a295ba0 2024-05-09T08:36:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.303+0000 7f2ed17ae740 1 -- 172.21.15.107:0/3831756662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 msgr2=0x56045a29fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.303+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/3831756662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29fd30 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009ca0 tx=0x7f2eb8011810 comp rx=0 tx=0).stop 2024-05-09T08:36:43.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.303+0000 7f2ed17ae740 1 -- 172.21.15.107:0/3831756662 shutdown_connections 2024-05-09T08:36:43.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.303+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/3831756662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a2a0270 0x56045a2a2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.303+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/3831756662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29fd30 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.303+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/3831756662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a295280 0x56045a295660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.303+0000 7f2ed17ae740 1 -- 172.21.15.107:0/3831756662 >> 172.21.15.107:0/3831756662 conn(0x56045a2915a0 msgr2=0x56045a291980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:43.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.303+0000 7f2ed17ae740 1 -- 172.21.15.107:0/3831756662 shutdown_connections 2024-05-09T08:36:43.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.303+0000 7f2ed17ae740 1 -- 172.21.15.107:0/3831756662 wait complete. 2024-05-09T08:36:43.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.303+0000 7f2ed17ae740 1 -- start start 2024-05-09T08:36:43.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ed17ae740 1 Processor -- start 2024-05-09T08:36:43.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ed17ae740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a295280 0x56045a29d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ed17ae740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ed17ae740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a2a0270 0x56045a29dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ed17ae740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56045a2a3060 con 0x56045a295ba0 2024-05-09T08:36:43.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ed17ae740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56045a1b5b90 con 0x56045a295280 2024-05-09T08:36:43.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ed17ae740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56045a2a31e0 con 0x56045a2a0270 2024-05-09T08:36:43.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ed17ae740 10 client.0 _ll_register_callbacks cb 0x56045a1f1da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:43.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ec7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ec7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38974/0 (socket says 172.21.15.107:38974) 2024-05-09T08:36:43.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ec7fff640 1 -- 172.21.15.107:0/947786773 learned_addr learned my addr 172.21.15.107:0/947786773 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:43.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ec77fe640 1 --2- 172.21.15.107:0/947786773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a295280 0x56045a29d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ec77fe640 1 --2- 172.21.15.107:0/947786773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a2a0270 0x56045a29dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ec7fff640 1 -- 172.21.15.107:0/947786773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a2a0270 msgr2=0x56045a29dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:43.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ec7fff640 1 --2- 172.21.15.107:0/947786773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a2a0270 0x56045a29dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ec7fff640 1 -- 172.21.15.107:0/947786773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a295280 msgr2=0x56045a29d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:43.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ec7fff640 1 --2- 172.21.15.107:0/947786773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a295280 0x56045a29d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.304+0000 7f2ec7fff640 1 -- 172.21.15.107:0/947786773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560459f0e3f0 con 0x56045a295ba0 2024-05-09T08:36:43.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.305+0000 7f2ec77fe640 1 --2- 172.21.15.107:0/947786773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a295280 0x56045a29d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:43.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.305+0000 7f2ec7fff640 1 --2- 172.21.15.107:0/947786773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29d730 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009c70 tx=0x7f2eb8019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13910ea4e295434 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:43.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.305+0000 7f2ea77fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:43.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.305+0000 7f2ea77fe640 1 -- 172.21.15.107:0/947786773 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb801e050 con 0x56045a295ba0 2024-05-09T08:36:43.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.305+0000 7f2ea77fe640 1 -- 172.21.15.107:0/947786773 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f2eb8005240 con 0x56045a295ba0 2024-05-09T08:36:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.305+0000 7f2ed17ae740 1 -- 172.21.15.107:0/947786773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2eb8019ae0 con 0x56045a295ba0 2024-05-09T08:36:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.305+0000 7f2ea77fe640 1 -- 172.21.15.107:0/947786773 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2eb8033050 con 0x56045a295ba0 2024-05-09T08:36:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.305+0000 7f2ed17ae740 10 client.16689 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.305+0000 7f2ed17ae740 1 -- 172.21.15.107:0/947786773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560459ece0f0 con 0x56045a295ba0 2024-05-09T08:36:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.305+0000 7f2ea77fe640 1 -- 172.21.15.107:0/947786773 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2eb8019ae0 con 0x56045a295ba0 2024-05-09T08:36:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.305+0000 7f2ea77fe640 1 -- 172.21.15.107:0/947786773 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f2eb800e6d0 con 0x56045a295ba0 2024-05-09T08:36:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.306+0000 7f2ed17ae740 10 client.16689 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.306+0000 7f2ed17ae740 10 client.16689 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.306+0000 7f2ed17ae740 20 client.16689 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.306+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/947786773 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56045a2b0570 0x56045a2b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.306+0000 7f2ed17ae740 4 client.16689 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.306+0000 7f2ed17ae740 1 -- 172.21.15.107:0/947786773 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56045a208430 con 0x56045a2b0570 2024-05-09T08:36:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.306+0000 7f2ec77fe640 1 --2- 172.21.15.107:0/947786773 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56045a2b0570 0x56045a2b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.306+0000 7f2ea77fe640 10 client.16689 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.306+0000 7f2ec77fe640 1 --2- 172.21.15.107:0/947786773 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56045a2b0570 0x56045a2b2960 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c56595880afd474 server_cookie=71bff69e1cb1e098 in_seq=0 out_seq=0 2024-05-09T08:36:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.307+0000 7f2ea77fe640 1 -- 172.21.15.107:0/947786773 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56045a208430 con 0x56045a2b0570 2024-05-09T08:36:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.307+0000 7f2ea77fe640 10 client.16689 handle_command_reply: tid=0 2024-05-09T08:36:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.307+0000 7f2ed17ae740 1 client.16689 shutdown 2024-05-09T08:36:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.307+0000 7f2ed17ae740 1 -- 172.21.15.107:0/947786773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 msgr2=0x56045a29d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.307+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/947786773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29d730 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009c70 tx=0x7f2eb8019660 comp rx=0 tx=0).stop 2024-05-09T08:36:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.307+0000 7f2ed17ae740 1 -- 172.21.15.107:0/947786773 shutdown_connections 2024-05-09T08:36:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.307+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/947786773 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56045a2b0570 0x56045a2b2960 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.307+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/947786773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a2a0270 0x56045a29dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.307+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/947786773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29d730 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.307+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/947786773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a295280 0x56045a29d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.307+0000 7f2ea77fe640 0 client.16689 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.307+0000 7f2ed17ae740 1 -- 172.21.15.107:0/947786773 >> 172.21.15.107:0/947786773 conn(0x56045a2915a0 msgr2=0x56045a2907e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:43.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.307+0000 7f2ed17ae740 1 -- 172.21.15.107:0/947786773 shutdown_connections 2024-05-09T08:36:43.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.307+0000 7f2ed17ae740 1 -- 172.21.15.107:0/947786773 wait complete. 2024-05-09T08:36:43.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.307+0000 7f2ed17ae740 20 client.16689 trim_cache size 0 max 16384 2024-05-09T08:36:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.330+0000 7f2ed17ae740 1 Processor -- start 2024-05-09T08:36:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.330+0000 7f2ed17ae740 1 -- start start 2024-05-09T08:36:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.330+0000 7f2ed17ae740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a295280 0x56045a2abf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.330+0000 7f2ed17ae740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a28c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.330+0000 7f2ed17ae740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a2a0270 0x56045a2b80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.330+0000 7f2ed17ae740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec81b3410 con 0x56045a295ba0 2024-05-09T08:36:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.330+0000 7f2ed17ae740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec81b0ce0 con 0x56045a2a0270 2024-05-09T08:36:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.330+0000 7f2ed17ae740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec81aeb30 con 0x56045a295280 2024-05-09T08:36:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.330+0000 7f2ec7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a28c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.330+0000 7f2ec7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a28c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38980/0 (socket says 172.21.15.107:38980) 2024-05-09T08:36:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.330+0000 7f2ec7fff640 1 -- 172.21.15.107:0/396012996 learned_addr learned my addr 172.21.15.107:0/396012996 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.330+0000 7f2ec77fe640 1 --2- 172.21.15.107:0/396012996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a2a0270 0x56045a2b80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.330+0000 7f2ec77fe640 1 --2- 172.21.15.107:0/396012996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a295280 0x56045a2abf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.331+0000 7f2ec7fff640 1 -- 172.21.15.107:0/396012996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a295280 msgr2=0x56045a2abf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.331+0000 7f2ec7fff640 1 --2- 172.21.15.107:0/396012996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a295280 0x56045a2abf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.331+0000 7f2ec7fff640 1 -- 172.21.15.107:0/396012996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a2a0270 msgr2=0x56045a2b80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.331+0000 7f2ec7fff640 1 --2- 172.21.15.107:0/396012996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a2a0270 0x56045a2b80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.331+0000 7f2ec7fff640 1 -- 172.21.15.107:0/396012996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56045a29c1a0 con 0x56045a295ba0 2024-05-09T08:36:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.331+0000 7f2ec77fe640 1 --2- 172.21.15.107:0/396012996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a2a0270 0x56045a2b80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.331+0000 7f2ec7fff640 1 --2- 172.21.15.107:0/396012996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a28c670 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8011850 tx=0x7f2eb8011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be8ffe2d44b39250 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.331+0000 7f2eadffb640 1 -- 172.21.15.107:0/396012996 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb801e040 con 0x56045a295ba0 2024-05-09T08:36:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.331+0000 7f2eadffb640 1 -- 172.21.15.107:0/396012996 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2eb802a070 con 0x56045a295ba0 2024-05-09T08:36:43.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.331+0000 7f2eadffb640 1 -- 172.21.15.107:0/396012996 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8005370 con 0x56045a295ba0 2024-05-09T08:36:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.332+0000 7f2ed17ae740 1 -- 172.21.15.107:0/396012996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 msgr2=0x56045a28c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.332+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/396012996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a28c670 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8011850 tx=0x7f2eb8011e40 comp rx=0 tx=0).stop 2024-05-09T08:36:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.332+0000 7f2ed17ae740 1 -- 172.21.15.107:0/396012996 shutdown_connections 2024-05-09T08:36:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.332+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/396012996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a2a0270 0x56045a2b80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.332+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/396012996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a28c670 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.332+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/396012996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a295280 0x56045a2abf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.332+0000 7f2ed17ae740 1 -- 172.21.15.107:0/396012996 >> 172.21.15.107:0/396012996 conn(0x56045a2915a0 msgr2=0x56045a29f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:43.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.332+0000 7f2ed17ae740 1 -- 172.21.15.107:0/396012996 shutdown_connections 2024-05-09T08:36:43.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.332+0000 7f2ed17ae740 1 -- 172.21.15.107:0/396012996 wait complete. 2024-05-09T08:36:43.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.332+0000 7f2ed17ae740 1 -- start start 2024-05-09T08:36:43.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ed17ae740 1 Processor -- start 2024-05-09T08:36:43.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ed17ae740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a295280 0x56045a299fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ed17ae740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ed17ae740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a2a0270 0x56045a29aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ed17ae740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec81af1a0 con 0x56045a295ba0 2024-05-09T08:36:43.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ed17ae740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec81b0610 con 0x56045a2a0270 2024-05-09T08:36:43.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ed17ae740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec81af870 con 0x56045a295280 2024-05-09T08:36:43.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ed17ae740 10 client.0 _ll_register_callbacks cb 0x7f2eb801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:43.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ec7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ec7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38984/0 (socket says 172.21.15.107:38984) 2024-05-09T08:36:43.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ec7fff640 1 -- 172.21.15.107:0/3560471624 learned_addr learned my addr 172.21.15.107:0/3560471624 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ec77fe640 1 --2- 172.21.15.107:0/3560471624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a295280 0x56045a299fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ec77fe640 1 --2- 172.21.15.107:0/3560471624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a2a0270 0x56045a29aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ec7fff640 1 -- 172.21.15.107:0/3560471624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a295280 msgr2=0x56045a299fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ec7fff640 1 --2- 172.21.15.107:0/3560471624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a295280 0x56045a299fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ec7fff640 1 -- 172.21.15.107:0/3560471624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a2a0270 msgr2=0x56045a29aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ec7fff640 1 --2- 172.21.15.107:0/3560471624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a2a0270 0x56045a29aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ec7fff640 1 -- 172.21.15.107:0/3560471624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56045a315a60 con 0x56045a295ba0 2024-05-09T08:36:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ec7fff640 1 --2- 172.21.15.107:0/3560471624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29a4f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009f50 tx=0x7f2eb8009c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67732de157c71865 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.333+0000 7f2ea77fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.334+0000 7f2ea77fe640 1 -- 172.21.15.107:0/3560471624 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb800e560 con 0x56045a295ba0 2024-05-09T08:36:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.334+0000 7f2ed17ae740 1 -- 172.21.15.107:0/3560471624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2eb802a070 con 0x56045a295ba0 2024-05-09T08:36:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.334+0000 7f2ea77fe640 1 -- 172.21.15.107:0/3560471624 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f2eb800e700 con 0x56045a295ba0 2024-05-09T08:36:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.334+0000 7f2ea77fe640 1 -- 172.21.15.107:0/3560471624 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2eb8048050 con 0x56045a295ba0 2024-05-09T08:36:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.334+0000 7f2ed17ae740 10 client.16695 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:43.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.334+0000 7f2ed17ae740 1 -- 172.21.15.107:0/3560471624 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56045a316d50 con 0x56045a295ba0 2024-05-09T08:36:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.334+0000 7f2ea77fe640 1 -- 172.21.15.107:0/3560471624 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2eb802a070 con 0x56045a295ba0 2024-05-09T08:36:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.334+0000 7f2ea77fe640 1 -- 172.21.15.107:0/3560471624 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f2eb8003890 con 0x56045a295ba0 2024-05-09T08:36:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.334+0000 7f2ed17ae740 10 client.16695 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.334+0000 7f2ed17ae740 10 client.16695 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.334+0000 7f2ed17ae740 20 client.16695 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.334+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/3560471624 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56045a317390 0x56045a3197b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.334+0000 7f2ed17ae740 4 client.16695 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:36:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.334+0000 7f2ed17ae740 1 -- 172.21.15.107:0/3560471624 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56045a202780 con 0x56045a317390 2024-05-09T08:36:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.335+0000 7f2ec77fe640 1 --2- 172.21.15.107:0/3560471624 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56045a317390 0x56045a3197b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.335+0000 7f2ec77fe640 1 --2- 172.21.15.107:0/3560471624 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56045a317390 0x56045a3197b0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4d58b164bea22cf server_cookie=781a970f52fe5c6c in_seq=0 out_seq=0 2024-05-09T08:36:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.335+0000 7f2ea77fe640 10 client.16695 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.335+0000 7f2ea77fe640 1 -- 172.21.15.107:0/3560471624 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56045a202780 con 0x56045a317390 2024-05-09T08:36:43.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.335+0000 7f2ea77fe640 10 client.16695 handle_command_reply: tid=0 2024-05-09T08:36:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.335+0000 7f2ed17ae740 1 client.16695 shutdown 2024-05-09T08:36:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.335+0000 7f2ed17ae740 1 -- 172.21.15.107:0/3560471624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 msgr2=0x56045a29a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.335+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/3560471624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29a4f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009f50 tx=0x7f2eb8009c40 comp rx=0 tx=0).stop 2024-05-09T08:36:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.335+0000 7f2ed17ae740 1 -- 172.21.15.107:0/3560471624 shutdown_connections 2024-05-09T08:36:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.335+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/3560471624 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56045a317390 0x56045a3197b0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.335+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/3560471624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56045a2a0270 0x56045a29aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.335+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/3560471624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56045a295ba0 0x56045a29a4f0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.335+0000 7f2ed17ae740 1 --2- 172.21.15.107:0/3560471624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56045a295280 0x56045a299fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.335+0000 7f2ed17ae740 1 -- 172.21.15.107:0/3560471624 >> 172.21.15.107:0/3560471624 conn(0x56045a2915a0 msgr2=0x56045a29d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.335+0000 7f2ea77fe640 0 client.16695 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.336+0000 7f2ed17ae740 1 -- 172.21.15.107:0/3560471624 shutdown_connections 2024-05-09T08:36:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.336+0000 7f2ed17ae740 1 -- 172.21.15.107:0/3560471624 wait complete. 2024-05-09T08:36:43.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.336+0000 7f2ed17ae740 20 client.16695 trim_cache size 0 max 16384 2024-05-09T08:36:43.342 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:36:43.342 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:36:43.342 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:36:43.342 INFO:teuthology.orchestra.run.smithi107.stdout: "e92546ad-0df6-4781-9134-8e0bce8dd1a1": { 2024-05-09T08:36:43.342 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:36:43.342 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "e92546ad-0df6-4781-9134-8e0bce8dd1a1", 2024-05-09T08:36:43.342 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:36:43.342 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:36:43.342 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:36:43.342 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:36:43.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.336+0000 7f2ece9c1640 1 -- 172.21.15.107:0/1307531795 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2eb00a2d00 msgr2=0x7f2eb00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.336+0000 7f2ece9c1640 1 --2- 172.21.15.107:0/1307531795 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2eb00a2d00 0x7f2eb00a51c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000a920 tx=0x7f2ec000a890 comp rx=0 tx=0).stop 2024-05-09T08:36:43.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.336+0000 7f2ece9c1640 1 -- 172.21.15.107:0/1307531795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec8106720 msgr2=0x7f2ec819eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:43.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.336+0000 7f2ece9c1640 1 --2- 172.21.15.107:0/1307531795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec8106720 0x7f2ec819eb90 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000cab0 tx=0x7f2ec000cf70 comp rx=0 tx=0).stop 2024-05-09T08:36:43.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.336+0000 7f2ece9c1640 1 -- 172.21.15.107:0/1307531795 shutdown_connections 2024-05-09T08:36:43.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.336+0000 7f2ece9c1640 1 --2- 172.21.15.107:0/1307531795 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2eb00a2d00 0x7f2eb00a51c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.336+0000 7f2ece9c1640 1 --2- 172.21.15.107:0/1307531795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ec8106720 0x7f2ec819eb90 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.336+0000 7f2ece9c1640 1 --2- 172.21.15.107:0/1307531795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ec8101120 0x7f2ec819a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.336+0000 7f2ece9c1640 1 --2- 172.21.15.107:0/1307531795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ec8100760 0x7f2ec819a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:43.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.336+0000 7f2ece9c1640 1 -- 172.21.15.107:0/1307531795 >> 172.21.15.107:0/1307531795 conn(0x7f2ec8077890 msgr2=0x7f2ec810bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:43.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.336+0000 7f2ece9c1640 1 -- 172.21.15.107:0/1307531795 shutdown_connections 2024-05-09T08:36:43.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:36:43.336+0000 7f2ece9c1640 1 -- 172.21.15.107:0/1307531795 wait complete. 2024-05-09T08:36:43.346 INFO:tasks.cephfs.filesystem:scrub status for tag:e92546ad-0df6-4781-9134-8e0bce8dd1a1 - {'path': '/', 'tag': 'e92546ad-0df6-4781-9134-8e0bce8dd1a1', 'options': 'recursive,force'} 2024-05-09T08:36:44.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:44 smithi107 ceph-mon[30122]: pgmap v104: 129 pgs: 129 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:36:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:44 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:36:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:44 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:44 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:36:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:44 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:36:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:44 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:44 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:36:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:44 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:44 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:36:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:44 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:44 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:36:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:44 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:44 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:36:44.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:44 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:44 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:44 smithi157 ceph-mon[33614]: pgmap v104: 129 pgs: 129 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:36:44.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:44 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:36:44.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:44 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:44 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:36:44.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:44 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:36:44.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:44 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:44 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:36:44.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:44 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:44 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:36:44.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:44 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:44 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:36:44.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:44 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:44 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:36:44.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:44 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:44 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:44 smithi110 ceph-mon[33622]: pgmap v104: 129 pgs: 129 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:36:44.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:44 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:36:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:44 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:44 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:36:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:44 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:36:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:44 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:44 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:36:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:44 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:44 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:36:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:44 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:44 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:36:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:44 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:44 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:36:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:44 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:44.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:44 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:36:46.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:46 smithi107 ceph-mon[30122]: pgmap v105: 129 pgs: 129 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:46.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:46 smithi157 ceph-mon[33614]: pgmap v105: 129 pgs: 129 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:46.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:46 smithi110 ceph-mon[33622]: pgmap v105: 129 pgs: 129 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:36:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.646+0000 7f8a2011b640 1 -- 172.21.15.107:0/2440541351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a1810be00 msgr2=0x7f8a1810e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.646+0000 7f8a2011b640 1 --2- 172.21.15.107:0/2440541351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a1810be00 0x7f8a1810e2a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f8a10009ca0 tx=0x7f8a1002a9e0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.646+0000 7f8a2011b640 1 -- 172.21.15.107:0/2440541351 shutdown_connections 2024-05-09T08:36:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.646+0000 7f8a2011b640 1 --2- 172.21.15.107:0/2440541351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a1810be00 0x7f8a1810e2a0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.646+0000 7f8a2011b640 1 --2- 172.21.15.107:0/2440541351 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a181033e0 0x7f8a1810b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.646+0000 7f8a2011b640 1 --2- 172.21.15.107:0/2440541351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a18102ac0 0x7f8a18102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.646+0000 7f8a2011b640 1 -- 172.21.15.107:0/2440541351 >> 172.21.15.107:0/2440541351 conn(0x7f8a180778b0 msgr2=0x7f8a18077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.646+0000 7f8a2011b640 1 -- 172.21.15.107:0/2440541351 shutdown_connections 2024-05-09T08:36:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.646+0000 7f8a2011b640 1 -- 172.21.15.107:0/2440541351 wait complete. 2024-05-09T08:36:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.647+0000 7f8a2011b640 1 Processor -- start 2024-05-09T08:36:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.647+0000 7f8a2011b640 1 -- start start 2024-05-09T08:36:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.647+0000 7f8a2011b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a18102ac0 0x7f8a18197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.647+0000 7f8a2011b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a181033e0 0x7f8a18198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.647+0000 7f8a2011b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a1810be00 0x7f8a1819c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.647+0000 7f8a2011b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a18111ab0 con 0x7f8a181033e0 2024-05-09T08:36:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.647+0000 7f8a2011b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a18111930 con 0x7f8a1810be00 2024-05-09T08:36:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.647+0000 7f8a2011b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8a18111c30 con 0x7f8a18102ac0 2024-05-09T08:36:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.647+0000 7f8a1d6b6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a181033e0 0x7f8a18198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.647+0000 7f8a1d6b6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a181033e0 0x7f8a18198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39008/0 (socket says 172.21.15.107:39008) 2024-05-09T08:36:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.647+0000 7f8a1d6b6640 1 -- 172.21.15.107:0/2832584079 learned_addr learned my addr 172.21.15.107:0/2832584079 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.648+0000 7f8a1deb7640 1 --2- 172.21.15.107:0/2832584079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a18102ac0 0x7f8a18197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.648+0000 7f8a1e6b8640 1 --2- 172.21.15.107:0/2832584079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a1810be00 0x7f8a1819c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.648+0000 7f8a1d6b6640 1 -- 172.21.15.107:0/2832584079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a18102ac0 msgr2=0x7f8a18197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.648+0000 7f8a1d6b6640 1 --2- 172.21.15.107:0/2832584079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a18102ac0 0x7f8a18197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.648+0000 7f8a1d6b6640 1 -- 172.21.15.107:0/2832584079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a1810be00 msgr2=0x7f8a1819c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.648+0000 7f8a1d6b6640 1 --2- 172.21.15.107:0/2832584079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a1810be00 0x7f8a1819c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.648+0000 7f8a1d6b6640 1 -- 172.21.15.107:0/2832584079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a1819ce90 con 0x7f8a181033e0 2024-05-09T08:36:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.648+0000 7f8a1d6b6640 1 --2- 172.21.15.107:0/2832584079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a181033e0 0x7f8a18198430 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800d950 tx=0x7f8a0800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.648+0000 7f8a0effd640 1 -- 172.21.15.107:0/2832584079 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a08017030 con 0x7f8a181033e0 2024-05-09T08:36:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.648+0000 7f8a2011b640 1 -- 172.21.15.107:0/2832584079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a1819d180 con 0x7f8a181033e0 2024-05-09T08:36:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.648+0000 7f8a0effd640 1 -- 172.21.15.107:0/2832584079 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a08004020 con 0x7f8a181033e0 2024-05-09T08:36:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.648+0000 7f8a0effd640 1 -- 172.21.15.107:0/2832584079 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a080047a0 con 0x7f8a181033e0 2024-05-09T08:36:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.648+0000 7f8a2011b640 1 -- 172.21.15.107:0/2832584079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a18100640 con 0x7f8a181033e0 2024-05-09T08:36:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.651+0000 7f8a0effd640 1 -- 172.21.15.107:0/2832584079 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8a080041d0 con 0x7f8a181033e0 2024-05-09T08:36:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.651+0000 7f8a22e2f740 1 Processor -- start 2024-05-09T08:36:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a22e2f740 1 -- start start 2024-05-09T08:36:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a0effd640 1 --2- 172.21.15.107:0/2832584079 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a040a2da0 0x7f8a040a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a22e2f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x560396016830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a22e2f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396016d70 0x5603960210f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a22e2f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396021630 0x560396023a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a22e2f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560395f7b340 con 0x560396016450 2024-05-09T08:36:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a22e2f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560395f23290 con 0x560396021630 2024-05-09T08:36:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a22e2f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560395f77300 con 0x560396016d70 2024-05-09T08:36:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a1e6b8640 1 --2- 172.21.15.107:0/2832584079 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a040a2da0 0x7f8a040a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a1deb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x560396016830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a1deb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x560396016830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39014/0 (socket says 172.21.15.107:39014) 2024-05-09T08:36:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a1deb7640 1 -- 172.21.15.107:0/4077439697 learned_addr learned my addr 172.21.15.107:0/4077439697 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a0effd640 1 -- 172.21.15.107:0/2832584079 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8a080cf290 con 0x7f8a181033e0 2024-05-09T08:36:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a1d6b6640 1 --2- 172.21.15.107:0/4077439697 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396016d70 0x5603960210f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a1deb7640 1 -- 172.21.15.107:0/4077439697 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396016d70 msgr2=0x5603960210f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a1deb7640 1 --2- 172.21.15.107:0/4077439697 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396016d70 0x5603960210f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a1deb7640 1 -- 172.21.15.107:0/4077439697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396021630 msgr2=0x560396023a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a1deb7640 1 --2- 172.21.15.107:0/4077439697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396021630 0x560396023a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a1deb7640 1 -- 172.21.15.107:0/4077439697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560395f478e0 con 0x560396016450 2024-05-09T08:36:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a1d6b6640 1 --2- 172.21.15.107:0/4077439697 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396016d70 0x5603960210f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.652+0000 7f8a1e6b8640 1 --2- 172.21.15.107:0/2832584079 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a040a2da0 0x7f8a040a5260 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8a10005fa0 tx=0x7f8a10009360 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.653+0000 7f8a1deb7640 1 --2- 172.21.15.107:0/4077439697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x560396016830 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400cc60 tx=0x7f8a1400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7846fef31cd5c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.653+0000 7f8a03fff640 1 -- 172.21.15.107:0/4077439697 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14014070 con 0x560396016450 2024-05-09T08:36:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.653+0000 7f8a03fff640 1 -- 172.21.15.107:0/4077439697 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a140047a0 con 0x560396016450 2024-05-09T08:36:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.653+0000 7f8a03fff640 1 -- 172.21.15.107:0/4077439697 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14010690 con 0x560396016450 2024-05-09T08:36:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.653+0000 7f8a22e2f740 1 -- 172.21.15.107:0/4077439697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 msgr2=0x560396016830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.653+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/4077439697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x560396016830 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400cc60 tx=0x7f8a1400e720 comp rx=0 tx=0).stop 2024-05-09T08:36:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.653+0000 7f8a22e2f740 1 -- 172.21.15.107:0/4077439697 shutdown_connections 2024-05-09T08:36:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.653+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/4077439697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396021630 0x560396023a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.653+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/4077439697 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396016d70 0x5603960210f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.653+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/4077439697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x560396016830 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.653+0000 7f8a22e2f740 1 -- 172.21.15.107:0/4077439697 >> 172.21.15.107:0/4077439697 conn(0x560396012960 msgr2=0x560396012d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.654+0000 7f8a22e2f740 1 -- 172.21.15.107:0/4077439697 shutdown_connections 2024-05-09T08:36:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.654+0000 7f8a22e2f740 1 -- 172.21.15.107:0/4077439697 wait complete. 2024-05-09T08:36:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.654+0000 7f8a22e2f740 1 -- start start 2024-05-09T08:36:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.654+0000 7f8a22e2f740 1 Processor -- start 2024-05-09T08:36:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.654+0000 7f8a22e2f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x56039601e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.654+0000 7f8a22e2f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396016d70 0x56039601eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.654+0000 7f8a22e2f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396021630 0x56039601eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.654+0000 7f8a22e2f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560396024260 con 0x560396016450 2024-05-09T08:36:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.654+0000 7f8a22e2f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560395f36790 con 0x560396016d70 2024-05-09T08:36:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.654+0000 7f8a22e2f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5603960243e0 con 0x560396021630 2024-05-09T08:36:47.664 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.654+0000 7f8a22e2f740 10 client.0 _ll_register_callbacks cb 0x560395f71a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:47.664 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.654+0000 7f8a1deb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x56039601e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.664 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a1deb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x56039601e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39022/0 (socket says 172.21.15.107:39022) 2024-05-09T08:36:47.664 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a1deb7640 1 -- 172.21.15.107:0/759700866 learned_addr learned my addr 172.21.15.107:0/759700866 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:47.664 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a1e6b8640 1 --2- 172.21.15.107:0/759700866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396021630 0x56039601eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.664 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a1d6b6640 1 --2- 172.21.15.107:0/759700866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396016d70 0x56039601eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a1deb7640 1 -- 172.21.15.107:0/759700866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396021630 msgr2=0x56039601eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a1deb7640 1 --2- 172.21.15.107:0/759700866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396021630 0x56039601eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a1deb7640 1 -- 172.21.15.107:0/759700866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396016d70 msgr2=0x56039601eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a1deb7640 1 --2- 172.21.15.107:0/759700866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396016d70 0x56039601eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a1deb7640 1 -- 172.21.15.107:0/759700866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560395c8f210 con 0x560396016450 2024-05-09T08:36:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a1d6b6640 1 --2- 172.21.15.107:0/759700866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396016d70 0x56039601eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:36:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a1deb7640 1 --2- 172.21.15.107:0/759700866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x56039601e570 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f8a14010bf0 tx=0x7f8a14010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e51d68aa48b6c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a017fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a017fa640 1 -- 172.21.15.107:0/759700866 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14014070 con 0x560396016450 2024-05-09T08:36:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a017fa640 1 -- 172.21.15.107:0/759700866 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8a14004020 con 0x560396016450 2024-05-09T08:36:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a22e2f740 1 -- 172.21.15.107:0/759700866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a140047a0 con 0x560396016450 2024-05-09T08:36:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a017fa640 1 -- 172.21.15.107:0/759700866 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8a1402d050 con 0x560396016450 2024-05-09T08:36:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a22e2f740 10 client.16707 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.655+0000 7f8a22e2f740 1 -- 172.21.15.107:0/759700866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560395d5b620 con 0x560396016450 2024-05-09T08:36:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.656+0000 7f8a017fa640 1 -- 172.21.15.107:0/759700866 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a14010e40 con 0x560396016450 2024-05-09T08:36:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.656+0000 7f8a017fa640 1 -- 172.21.15.107:0/759700866 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f8a140035f0 con 0x560396016450 2024-05-09T08:36:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.656+0000 7f8a22e2f740 10 client.16707 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.656+0000 7f8a22e2f740 10 client.16707 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.656+0000 7f8a22e2f740 20 client.16707 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.656+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/759700866 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5603960311d0 0x5603960335c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.656+0000 7f8a22e2f740 4 client.16707 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:36:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.656+0000 7f8a22e2f740 1 -- 172.21.15.107:0/759700866 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8a140047a0 con 0x5603960311d0 2024-05-09T08:36:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.656+0000 7f8a1d6b6640 1 --2- 172.21.15.107:0/759700866 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5603960311d0 0x5603960335c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.657+0000 7f8a1d6b6640 1 --2- 172.21.15.107:0/759700866 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5603960311d0 0x5603960335c0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6af229a7d552b882 server_cookie=44a86fc109e15dd0 in_seq=0 out_seq=0 2024-05-09T08:36:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.657+0000 7f8a017fa640 10 client.16707 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.657+0000 7f8a017fa640 1 -- 172.21.15.107:0/759700866 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f8a140047a0 con 0x5603960311d0 2024-05-09T08:36:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.657+0000 7f8a017fa640 10 client.16707 handle_command_reply: tid=0 2024-05-09T08:36:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.657+0000 7f8a22e2f740 1 client.16707 shutdown 2024-05-09T08:36:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.658+0000 7f8a22e2f740 1 -- 172.21.15.107:0/759700866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 msgr2=0x56039601e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.658+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/759700866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x56039601e570 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f8a14010bf0 tx=0x7f8a14010c20 comp rx=0 tx=0).stop 2024-05-09T08:36:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.658+0000 7f8a22e2f740 1 -- 172.21.15.107:0/759700866 shutdown_connections 2024-05-09T08:36:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.658+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/759700866 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5603960311d0 0x5603960335c0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.658+0000 7f8a017fa640 0 client.16707 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.658+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/759700866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396021630 0x56039601eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.658+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/759700866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396016d70 0x56039601eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.658+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/759700866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x56039601e570 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.658+0000 7f8a22e2f740 1 -- 172.21.15.107:0/759700866 >> 172.21.15.107:0/759700866 conn(0x560396012960 msgr2=0x560396018690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.658+0000 7f8a22e2f740 1 -- 172.21.15.107:0/759700866 shutdown_connections 2024-05-09T08:36:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.658+0000 7f8a22e2f740 1 -- 172.21.15.107:0/759700866 wait complete. 2024-05-09T08:36:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.658+0000 7f8a22e2f740 20 client.16707 trim_cache size 0 max 16384 2024-05-09T08:36:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.675+0000 7f8a22e2f740 1 Processor -- start 2024-05-09T08:36:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.675+0000 7f8a22e2f740 1 -- start start 2024-05-09T08:36:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a22e2f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x560396020ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a22e2f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396016d70 0x56039602d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a22e2f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396021630 0x56039601d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a22e2f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a181af190 con 0x560396016450 2024-05-09T08:36:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a22e2f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a181aca60 con 0x560396021630 2024-05-09T08:36:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a22e2f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8a181aa8b0 con 0x560396016d70 2024-05-09T08:36:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a1deb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x560396020ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a1deb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x560396020ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39024/0 (socket says 172.21.15.107:39024) 2024-05-09T08:36:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a1d6b6640 1 --2- 172.21.15.107:0/226957503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396016d70 0x56039602d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a1deb7640 1 -- 172.21.15.107:0/226957503 learned_addr learned my addr 172.21.15.107:0/226957503 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a1e6b8640 1 --2- 172.21.15.107:0/226957503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396021630 0x56039601d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a1d6b6640 1 -- 172.21.15.107:0/226957503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396021630 msgr2=0x56039601d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a1d6b6640 1 --2- 172.21.15.107:0/226957503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396021630 0x56039601d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a1d6b6640 1 -- 172.21.15.107:0/226957503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 msgr2=0x560396020ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a1d6b6640 1 --2- 172.21.15.107:0/226957503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x560396020ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a1d6b6640 1 -- 172.21.15.107:0/226957503 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560396016b60 con 0x560396016d70 2024-05-09T08:36:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a1deb7640 1 --2- 172.21.15.107:0/226957503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x560396020ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:36:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a1e6b8640 1 --2- 172.21.15.107:0/226957503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396021630 0x56039601d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:36:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a1d6b6640 1 --2- 172.21.15.107:0/226957503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396016d70 0x56039602d120 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8a080cc650 tx=0x7f8a08097040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc15716f5742ad71 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.676+0000 7f8a037fe640 1 -- 172.21.15.107:0/226957503 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a0809bb50 con 0x560396016d70 2024-05-09T08:36:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.677+0000 7f8a037fe640 1 -- 172.21.15.107:0/226957503 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a08020160 con 0x560396016d70 2024-05-09T08:36:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.677+0000 7f8a037fe640 1 -- 172.21.15.107:0/226957503 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a08097ac0 con 0x560396016d70 2024-05-09T08:36:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.677+0000 7f8a22e2f740 1 -- 172.21.15.107:0/226957503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396016d70 msgr2=0x56039602d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.677+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/226957503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396016d70 0x56039602d120 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8a080cc650 tx=0x7f8a08097040 comp rx=0 tx=0).stop 2024-05-09T08:36:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.677+0000 7f8a22e2f740 1 -- 172.21.15.107:0/226957503 shutdown_connections 2024-05-09T08:36:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.677+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/226957503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396021630 0x56039601d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.677+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/226957503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396016d70 0x56039602d120 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.677+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/226957503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016450 0x560396020ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.677+0000 7f8a22e2f740 1 -- 172.21.15.107:0/226957503 >> 172.21.15.107:0/226957503 conn(0x560396012960 msgr2=0x5603960110e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.677+0000 7f8a22e2f740 1 -- 172.21.15.107:0/226957503 shutdown_connections 2024-05-09T08:36:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.677+0000 7f8a22e2f740 1 -- 172.21.15.107:0/226957503 wait complete. 2024-05-09T08:36:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.678+0000 7f8a22e2f740 1 -- start start 2024-05-09T08:36:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.678+0000 7f8a22e2f740 1 Processor -- start 2024-05-09T08:36:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.678+0000 7f8a22e2f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396016450 0x56039601b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.678+0000 7f8a22e2f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016d70 0x56039601b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.678+0000 7f8a22e2f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396021630 0x56039601bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.678+0000 7f8a22e2f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a181aaf20 con 0x560396016d70 2024-05-09T08:36:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.678+0000 7f8a22e2f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a181ac390 con 0x560396016450 2024-05-09T08:36:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.678+0000 7f8a22e2f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8a181ab5f0 con 0x560396021630 2024-05-09T08:36:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.678+0000 7f8a22e2f740 10 client.0 _ll_register_callbacks cb 0x7f8a1400cab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:36:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.678+0000 7f8a1d6b6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016d70 0x56039601b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.678+0000 7f8a1d6b6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016d70 0x56039601b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39038/0 (socket says 172.21.15.107:39038) 2024-05-09T08:36:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.678+0000 7f8a1d6b6640 1 -- 172.21.15.107:0/3485026668 learned_addr learned my addr 172.21.15.107:0/3485026668 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:36:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.678+0000 7f8a1e6b8640 1 --2- 172.21.15.107:0/3485026668 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396021630 0x56039601bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.679+0000 7f8a1deb7640 1 --2- 172.21.15.107:0/3485026668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396016450 0x56039601b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.679+0000 7f8a1d6b6640 1 -- 172.21.15.107:0/3485026668 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396021630 msgr2=0x56039601bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.679+0000 7f8a1d6b6640 1 --2- 172.21.15.107:0/3485026668 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396021630 0x56039601bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.679+0000 7f8a1d6b6640 1 -- 172.21.15.107:0/3485026668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396016450 msgr2=0x56039601b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:36:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.679+0000 7f8a1d6b6640 1 --2- 172.21.15.107:0/3485026668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396016450 0x56039601b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.679+0000 7f8a1d6b6640 1 -- 172.21.15.107:0/3485026668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560396096490 con 0x560396016d70 2024-05-09T08:36:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.679+0000 7f8a1deb7640 1 --2- 172.21.15.107:0/3485026668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396016450 0x56039601b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:36:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.679+0000 7f8a1d6b6640 1 --2- 172.21.15.107:0/3485026668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016d70 0x56039601b660 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f8a080cf260 tx=0x7f8a080050c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=550e2959d5425fdc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:36:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.679+0000 7f8a017fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:36:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.679+0000 7f8a017fa640 1 -- 172.21.15.107:0/3485026668 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a080171d0 con 0x560396016d70 2024-05-09T08:36:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.679+0000 7f8a017fa640 1 -- 172.21.15.107:0/3485026668 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8a08097ac0 con 0x560396016d70 2024-05-09T08:36:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.679+0000 7f8a22e2f740 1 -- 172.21.15.107:0/3485026668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a08020160 con 0x560396016d70 2024-05-09T08:36:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.679+0000 7f8a017fa640 1 -- 172.21.15.107:0/3485026668 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8a0809b450 con 0x560396016d70 2024-05-09T08:36:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.679+0000 7f8a22e2f740 10 client.16713 fetch_fsmap learned FSMap version 26 2024-05-09T08:36:47.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.679+0000 7f8a22e2f740 1 -- 172.21.15.107:0/3485026668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560396097650 con 0x560396016d70 2024-05-09T08:36:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.679+0000 7f8a017fa640 1 -- 172.21.15.107:0/3485026668 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a08020160 con 0x560396016d70 2024-05-09T08:36:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.680+0000 7f8a017fa640 1 -- 172.21.15.107:0/3485026668 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f8a080971a0 con 0x560396016d70 2024-05-09T08:36:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.680+0000 7f8a22e2f740 10 client.16713 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:36:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.680+0000 7f8a22e2f740 10 client.16713 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:36:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.680+0000 7f8a22e2f740 20 client.16713 populate_metadata read hostname 'smithi107' 2024-05-09T08:36:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.680+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/3485026668 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5603960311d0 0x560396099e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:36:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.680+0000 7f8a22e2f740 4 client.16713 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:36:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.680+0000 7f8a22e2f740 1 -- 172.21.15.107:0/3485026668 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560395f88a70 con 0x5603960311d0 2024-05-09T08:36:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.680+0000 7f8a1deb7640 1 --2- 172.21.15.107:0/3485026668 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5603960311d0 0x560396099e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:36:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.680+0000 7f8a017fa640 10 client.16713 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.680+0000 7f8a1deb7640 1 --2- 172.21.15.107:0/3485026668 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5603960311d0 0x560396099e60 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f64afb8190696b0 server_cookie=d623999d9de96c2b in_seq=0 out_seq=0 2024-05-09T08:36:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.697+0000 7f8a017fa640 1 -- 172.21.15.107:0/3485026668 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560395f88a70 con 0x5603960311d0 2024-05-09T08:36:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.697+0000 7f8a017fa640 10 client.16713 handle_command_reply: tid=0 2024-05-09T08:36:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.697+0000 7f8a22e2f740 1 client.16713 shutdown 2024-05-09T08:36:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.697+0000 7f8a22e2f740 1 -- 172.21.15.107:0/3485026668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016d70 msgr2=0x56039601b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.697+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/3485026668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016d70 0x56039601b660 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f8a080cf260 tx=0x7f8a080050c0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.697+0000 7f8a22e2f740 1 -- 172.21.15.107:0/3485026668 shutdown_connections 2024-05-09T08:36:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.697+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/3485026668 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5603960311d0 0x560396099e60 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.697+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/3485026668 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560396021630 0x56039601bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.697+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/3485026668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560396016d70 0x56039601b660 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.697+0000 7f8a22e2f740 1 --2- 172.21.15.107:0/3485026668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560396016450 0x56039601b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.697+0000 7f8a22e2f740 1 -- 172.21.15.107:0/3485026668 >> 172.21.15.107:0/3485026668 conn(0x560396012960 msgr2=0x560396011070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.697+0000 7f8a017fa640 0 client.16713 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:36:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.697+0000 7f8a22e2f740 1 -- 172.21.15.107:0/3485026668 shutdown_connections 2024-05-09T08:36:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.697+0000 7f8a22e2f740 1 -- 172.21.15.107:0/3485026668 wait complete. 2024-05-09T08:36:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.697+0000 7f8a22e2f740 20 client.16713 trim_cache size 0 max 16384 2024-05-09T08:36:47.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:36:47.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:36:47.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:36:47.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:36:47.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.698+0000 7f8a2011b640 1 -- 172.21.15.107:0/2832584079 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a040a2da0 msgr2=0x7f8a040a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:47.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.698+0000 7f8a2011b640 1 --2- 172.21.15.107:0/2832584079 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a040a2da0 0x7f8a040a5260 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8a10005fa0 tx=0x7f8a10009360 comp rx=0 tx=0).stop 2024-05-09T08:36:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.698+0000 7f8a2011b640 1 -- 172.21.15.107:0/2832584079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a181033e0 msgr2=0x7f8a18198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:36:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.698+0000 7f8a2011b640 1 --2- 172.21.15.107:0/2832584079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a181033e0 0x7f8a18198430 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800d950 tx=0x7f8a0800de10 comp rx=0 tx=0).stop 2024-05-09T08:36:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.698+0000 7f8a2011b640 1 -- 172.21.15.107:0/2832584079 shutdown_connections 2024-05-09T08:36:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.698+0000 7f8a2011b640 1 --2- 172.21.15.107:0/2832584079 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a040a2da0 0x7f8a040a5260 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.698+0000 7f8a2011b640 1 --2- 172.21.15.107:0/2832584079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a1810be00 0x7f8a1819c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.698+0000 7f8a2011b640 1 --2- 172.21.15.107:0/2832584079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a181033e0 0x7f8a18198430 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.698+0000 7f8a2011b640 1 --2- 172.21.15.107:0/2832584079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a18102ac0 0x7f8a18197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:36:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.698+0000 7f8a2011b640 1 -- 172.21.15.107:0/2832584079 >> 172.21.15.107:0/2832584079 conn(0x7f8a180778b0 msgr2=0x7f8a180fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:36:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.698+0000 7f8a2011b640 1 -- 172.21.15.107:0/2832584079 shutdown_connections 2024-05-09T08:36:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:36:47.698+0000 7f8a2011b640 1 -- 172.21.15.107:0/2832584079 wait complete. 2024-05-09T08:36:48.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:48 smithi107 ceph-mon[30122]: pgmap v106: 129 pgs: 129 active+clean; 3.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:36:48.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:48 smithi157 ceph-mon[33614]: pgmap v106: 129 pgs: 129 active+clean; 3.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:36:48.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:48 smithi110 ceph-mon[33622]: pgmap v106: 129 pgs: 129 active+clean; 3.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:36:50.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:50 smithi107 ceph-mon[30122]: pgmap v107: 129 pgs: 129 active+clean; 3.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 25 MiB/s wr, 41 op/s 2024-05-09T08:36:50.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:50 smithi157 ceph-mon[33614]: pgmap v107: 129 pgs: 129 active+clean; 3.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 25 MiB/s wr, 41 op/s 2024-05-09T08:36:50.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:50 smithi110 ceph-mon[33622]: pgmap v107: 129 pgs: 129 active+clean; 3.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 25 MiB/s wr, 41 op/s 2024-05-09T08:36:51.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:51 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:36:51.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:51 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:36:51.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:51 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:36:52.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:52 smithi107 ceph-mon[30122]: pgmap v108: 129 pgs: 129 active+clean; 3.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 25 MiB/s wr, 41 op/s 2024-05-09T08:36:52.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:52 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:36:52.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:52 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:36:52.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:52 smithi157 ceph-mon[33614]: pgmap v108: 129 pgs: 129 active+clean; 3.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 25 MiB/s wr, 41 op/s 2024-05-09T08:36:52.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:52 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:36:52.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:52 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:36:52.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:52 smithi110 ceph-mon[33622]: pgmap v108: 129 pgs: 129 active+clean; 3.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 25 MiB/s wr, 41 op/s 2024-05-09T08:36:52.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:52 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:36:52.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:52 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:36:54.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:54 smithi107 ceph-mon[30122]: pgmap v109: 129 pgs: 129 active+clean; 3.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:36:54.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:54 smithi157 ceph-mon[33614]: pgmap v109: 129 pgs: 129 active+clean; 3.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:36:54.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:54 smithi110 ceph-mon[33622]: pgmap v109: 129 pgs: 129 active+clean; 3.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:36:56.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:56 smithi107 ceph-mon[30122]: pgmap v110: 129 pgs: 129 active+clean; 3.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 25 MiB/s wr, 45 op/s 2024-05-09T08:36:56.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:56 smithi157 ceph-mon[33614]: pgmap v110: 129 pgs: 129 active+clean; 3.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 25 MiB/s wr, 45 op/s 2024-05-09T08:36:56.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:56 smithi110 ceph-mon[33622]: pgmap v110: 129 pgs: 129 active+clean; 3.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 25 MiB/s wr, 45 op/s 2024-05-09T08:36:58.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:36:58 smithi107 ceph-mon[30122]: pgmap v111: 129 pgs: 129 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:36:58.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:36:58 smithi157 ceph-mon[33614]: pgmap v111: 129 pgs: 129 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:36:58.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:36:58 smithi110 ceph-mon[33622]: pgmap v111: 129 pgs: 129 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 37 MiB/s wr, 53 op/s 2024-05-09T08:37:00.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:00 smithi107 ceph-mon[30122]: pgmap v112: 129 pgs: 129 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:37:00.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:00 smithi157 ceph-mon[33614]: pgmap v112: 129 pgs: 129 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:37:00.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:00 smithi110 ceph-mon[33622]: pgmap v112: 129 pgs: 129 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:37:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.798+0000 7fd719709640 1 -- 172.21.15.107:0/1012413005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd714102ac0 msgr2=0x7fd714102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.798+0000 7fd719709640 1 --2- 172.21.15.107:0/1012413005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd714102ac0 0x7fd714102ea0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fd704009900 tx=0x7fd70402a510 comp rx=0 tx=0).stop 2024-05-09T08:37:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.798+0000 7fd719709640 1 -- 172.21.15.107:0/1012413005 shutdown_connections 2024-05-09T08:37:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.798+0000 7fd719709640 1 --2- 172.21.15.107:0/1012413005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd71410be00 0x7fd71410e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.798+0000 7fd719709640 1 --2- 172.21.15.107:0/1012413005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7141033e0 0x7fd71410b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.798+0000 7fd719709640 1 --2- 172.21.15.107:0/1012413005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd714102ac0 0x7fd714102ea0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.798+0000 7fd719709640 1 -- 172.21.15.107:0/1012413005 >> 172.21.15.107:0/1012413005 conn(0x7fd7140778b0 msgr2=0x7fd714077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.798+0000 7fd719709640 1 -- 172.21.15.107:0/1012413005 shutdown_connections 2024-05-09T08:37:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.798+0000 7fd719709640 1 -- 172.21.15.107:0/1012413005 wait complete. 2024-05-09T08:37:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.798+0000 7fd719709640 1 Processor -- start 2024-05-09T08:37:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd719709640 1 -- start start 2024-05-09T08:37:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd719709640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd714102ac0 0x7fd714197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd719709640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7141033e0 0x7fd714198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd719709640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd71410be00 0x7fd71419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd719709640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd714111ab0 con 0x7fd7141033e0 2024-05-09T08:37:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd719709640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd714111930 con 0x7fd714102ac0 2024-05-09T08:37:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd719709640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd714111c30 con 0x7fd71410be00 2024-05-09T08:37:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd70bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7141033e0 0x7fd714198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd70bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7141033e0 0x7fd714198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41442/0 (socket says 172.21.15.107:41442) 2024-05-09T08:37:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd70bfff640 1 -- 172.21.15.107:0/2323279804 learned_addr learned my addr 172.21.15.107:0/2323279804 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd712ffd640 1 --2- 172.21.15.107:0/2323279804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd714102ac0 0x7fd714197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd7137fe640 1 --2- 172.21.15.107:0/2323279804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd71410be00 0x7fd71419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd70bfff640 1 -- 172.21.15.107:0/2323279804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd71410be00 msgr2=0x7fd71419c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd70bfff640 1 --2- 172.21.15.107:0/2323279804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd71410be00 0x7fd71419c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd70bfff640 1 -- 172.21.15.107:0/2323279804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd714102ac0 msgr2=0x7fd714197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd70bfff640 1 --2- 172.21.15.107:0/2323279804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd714102ac0 0x7fd714197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd70bfff640 1 -- 172.21.15.107:0/2323279804 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71419ced0 con 0x7fd7141033e0 2024-05-09T08:37:00.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd70bfff640 1 --2- 172.21.15.107:0/2323279804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7141033e0 0x7fd714198470 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc00c920 tx=0x7fd6fc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:00.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.799+0000 7fd710ff9640 1 -- 172.21.15.107:0/2323279804 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc007c70 con 0x7fd7141033e0 2024-05-09T08:37:00.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.800+0000 7fd710ff9640 1 -- 172.21.15.107:0/2323279804 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6fc007e10 con 0x7fd7141033e0 2024-05-09T08:37:00.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.800+0000 7fd719709640 1 -- 172.21.15.107:0/2323279804 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71419d1c0 con 0x7fd7141033e0 2024-05-09T08:37:00.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.800+0000 7fd710ff9640 1 -- 172.21.15.107:0/2323279804 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc018820 con 0x7fd7141033e0 2024-05-09T08:37:00.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.800+0000 7fd719709640 1 -- 172.21.15.107:0/2323279804 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7141a4a50 con 0x7fd7141033e0 2024-05-09T08:37:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.802+0000 7fd710ff9640 1 -- 172.21.15.107:0/2323279804 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd6fc0189c0 con 0x7fd7141033e0 2024-05-09T08:37:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.802+0000 7fd71c413740 1 Processor -- start 2024-05-09T08:37:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.802+0000 7fd71c413740 1 -- start start 2024-05-09T08:37:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd71c413740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24a860 0x55c6ae24ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd71c413740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24b180 0x55c6ae255500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd71c413740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae257e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd71c413740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c6ae1ab340 con 0x55c6ae24b180 2024-05-09T08:37:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd71c413740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c6ae153290 con 0x55c6ae24a860 2024-05-09T08:37:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd71c413740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c6ae1a7300 con 0x55c6ae255a40 2024-05-09T08:37:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd712ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24b180 0x55c6ae255500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd7137fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24a860 0x55c6ae24ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd712ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24b180 0x55c6ae255500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41450/0 (socket says 172.21.15.107:41450) 2024-05-09T08:37:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd712ffd640 1 -- 172.21.15.107:0/2602495708 learned_addr learned my addr 172.21.15.107:0/2602495708 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd70bfff640 1 --2- 172.21.15.107:0/2602495708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae257e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd710ff9640 1 --2- 172.21.15.107:0/2323279804 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd7000a2e20 0x7fd7000a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd7137fe640 1 -- 172.21.15.107:0/2602495708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 msgr2=0x55c6ae257e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd7137fe640 1 --2- 172.21.15.107:0/2602495708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae257e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd7137fe640 1 -- 172.21.15.107:0/2602495708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24b180 msgr2=0x55c6ae255500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd7137fe640 1 --2- 172.21.15.107:0/2602495708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24b180 0x55c6ae255500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.803+0000 7fd7137fe640 1 -- 172.21.15.107:0/2602495708 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6ae1778e0 con 0x55c6ae24a860 2024-05-09T08:37:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.804+0000 7fd7137fe640 1 --2- 172.21.15.107:0/2323279804 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd7000a2e20 0x7fd7000a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.804+0000 7fd710ff9640 1 -- 172.21.15.107:0/2323279804 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd6fc0cf930 con 0x7fd7141033e0 2024-05-09T08:37:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.804+0000 7fd7137fe640 1 --2- 172.21.15.107:0/2602495708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24a860 0x55c6ae24ac40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd70c005fd0 tx=0x7fd70c00d710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=766ee7e37cba33fd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.804+0000 7fd7097fa640 1 -- 172.21.15.107:0/2602495708 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c011e60 con 0x55c6ae24a860 2024-05-09T08:37:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.804+0000 7fd7097fa640 1 -- 172.21.15.107:0/2602495708 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd70c01ad90 con 0x55c6ae24a860 2024-05-09T08:37:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.804+0000 7fd7097fa640 1 -- 172.21.15.107:0/2602495708 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c00ddc0 con 0x55c6ae24a860 2024-05-09T08:37:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.805+0000 7fd71c413740 1 -- 172.21.15.107:0/2602495708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24a860 msgr2=0x55c6ae24ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.805+0000 7fd71c413740 1 --2- 172.21.15.107:0/2602495708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24a860 0x55c6ae24ac40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd70c005fd0 tx=0x7fd70c00d710 comp rx=0 tx=0).stop 2024-05-09T08:37:00.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.805+0000 7fd7137fe640 1 --2- 172.21.15.107:0/2323279804 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd7000a2e20 0x7fd7000a52e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd70c01e000 tx=0x7fd70c01f000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.805+0000 7fd71c413740 1 -- 172.21.15.107:0/2602495708 shutdown_connections 2024-05-09T08:37:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.805+0000 7fd71c413740 1 --2- 172.21.15.107:0/2602495708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae257e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.805+0000 7fd71c413740 1 --2- 172.21.15.107:0/2602495708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24b180 0x55c6ae255500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.805+0000 7fd71c413740 1 --2- 172.21.15.107:0/2602495708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24a860 0x55c6ae24ac40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.805+0000 7fd71c413740 1 -- 172.21.15.107:0/2602495708 >> 172.21.15.107:0/2602495708 conn(0x55c6ae246d70 msgr2=0x55c6ae247150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.805+0000 7fd71c413740 1 -- 172.21.15.107:0/2602495708 shutdown_connections 2024-05-09T08:37:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.805+0000 7fd71c413740 1 -- 172.21.15.107:0/2602495708 wait complete. 2024-05-09T08:37:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.805+0000 7fd71c413740 1 -- start start 2024-05-09T08:37:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.805+0000 7fd71c413740 1 Processor -- start 2024-05-09T08:37:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd71c413740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae2529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd71c413740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24b180 0x55c6ae252f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd71c413740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae253450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd71c413740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c6ae258670 con 0x55c6ae24a860 2024-05-09T08:37:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd71c413740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c6ae166790 con 0x55c6ae24b180 2024-05-09T08:37:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd71c413740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c6ae2587f0 con 0x55c6ae255a40 2024-05-09T08:37:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd71c413740 10 client.0 _ll_register_callbacks cb 0x55c6ae1a1a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd712ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae2529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd712ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae2529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41458/0 (socket says 172.21.15.107:41458) 2024-05-09T08:37:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd712ffd640 1 -- 172.21.15.107:0/3624340692 learned_addr learned my addr 172.21.15.107:0/3624340692 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd70bfff640 1 --2- 172.21.15.107:0/3624340692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24b180 0x55c6ae252f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd7137fe640 1 --2- 172.21.15.107:0/3624340692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae253450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd712ffd640 1 -- 172.21.15.107:0/3624340692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 msgr2=0x55c6ae253450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd712ffd640 1 --2- 172.21.15.107:0/3624340692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae253450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd712ffd640 1 -- 172.21.15.107:0/3624340692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24b180 msgr2=0x55c6ae252f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd712ffd640 1 --2- 172.21.15.107:0/3624340692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24b180 0x55c6ae252f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd712ffd640 1 -- 172.21.15.107:0/3624340692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6adebf210 con 0x55c6ae24a860 2024-05-09T08:37:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd7137fe640 1 --2- 172.21.15.107:0/3624340692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae253450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd70bfff640 1 --2- 172.21.15.107:0/3624340692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24b180 0x55c6ae252f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd712ffd640 1 --2- 172.21.15.107:0/3624340692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae2529d0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd704033000 tx=0x7fd704031760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e8f96b0afb8dc9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd6fa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd6fa7fc640 1 -- 172.21.15.107:0/3624340692 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd704035040 con 0x55c6ae24a860 2024-05-09T08:37:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd6fa7fc640 1 -- 172.21.15.107:0/3624340692 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd704004460 con 0x55c6ae24a860 2024-05-09T08:37:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.806+0000 7fd71c413740 1 -- 172.21.15.107:0/3624340692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd70c01ad90 con 0x55c6ae24a860 2024-05-09T08:37:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.807+0000 7fd6fa7fc640 1 -- 172.21.15.107:0/3624340692 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd704005020 con 0x55c6ae24a860 2024-05-09T08:37:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.807+0000 7fd71c413740 10 client.16725 fetch_fsmap learned FSMap version 26 2024-05-09T08:37:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.807+0000 7fd71c413740 1 -- 172.21.15.107:0/3624340692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c6adf8b620 con 0x55c6ae24a860 2024-05-09T08:37:00.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.807+0000 7fd6fa7fc640 1 -- 172.21.15.107:0/3624340692 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd70c01ad90 con 0x55c6ae24a860 2024-05-09T08:37:00.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.807+0000 7fd6fa7fc640 1 -- 172.21.15.107:0/3624340692 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd7040035f0 con 0x55c6ae24a860 2024-05-09T08:37:00.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.807+0000 7fd71c413740 10 client.16725 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:37:00.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.807+0000 7fd71c413740 10 client.16725 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:00.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.807+0000 7fd71c413740 20 client.16725 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:00.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.807+0000 7fd71c413740 1 --2- 172.21.15.107:0/3624340692 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c6ae265790 0x55c6ae267c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.807+0000 7fd71c413740 4 client.16725 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:00.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.807+0000 7fd71c413740 1 -- 172.21.15.107:0/3624340692 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6ae1b8a70 con 0x55c6ae265790 2024-05-09T08:37:00.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.807+0000 7fd70bfff640 1 --2- 172.21.15.107:0/3624340692 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c6ae265790 0x55c6ae267c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.808+0000 7fd6fa7fc640 10 client.16725 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:00.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.808+0000 7fd70bfff640 1 --2- 172.21.15.107:0/3624340692 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c6ae265790 0x55c6ae267c20 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=858309dc25eec216 server_cookie=945de4056ed51dd2 in_seq=0 out_seq=0 2024-05-09T08:37:00.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.809+0000 7fd6fa7fc640 1 -- 172.21.15.107:0/3624340692 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c6ae1b8a70 con 0x55c6ae265790 2024-05-09T08:37:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.809+0000 7fd6fa7fc640 10 client.16725 handle_command_reply: tid=0 2024-05-09T08:37:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.809+0000 7fd71c413740 1 client.16725 shutdown 2024-05-09T08:37:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.809+0000 7fd71c413740 1 -- 172.21.15.107:0/3624340692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 msgr2=0x55c6ae2529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.809+0000 7fd71c413740 1 --2- 172.21.15.107:0/3624340692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae2529d0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd704033000 tx=0x7fd704031760 comp rx=0 tx=0).stop 2024-05-09T08:37:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.809+0000 7fd71c413740 1 -- 172.21.15.107:0/3624340692 shutdown_connections 2024-05-09T08:37:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.809+0000 7fd71c413740 1 --2- 172.21.15.107:0/3624340692 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c6ae265790 0x55c6ae267c20 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.809+0000 7fd71c413740 1 --2- 172.21.15.107:0/3624340692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae253450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.809+0000 7fd71c413740 1 --2- 172.21.15.107:0/3624340692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24b180 0x55c6ae252f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.809+0000 7fd6fa7fc640 0 client.16725 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.809+0000 7fd71c413740 1 --2- 172.21.15.107:0/3624340692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae2529d0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.809+0000 7fd71c413740 1 -- 172.21.15.107:0/3624340692 >> 172.21.15.107:0/3624340692 conn(0x55c6ae246d70 msgr2=0x55c6ae24caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.809+0000 7fd71c413740 1 -- 172.21.15.107:0/3624340692 shutdown_connections 2024-05-09T08:37:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.809+0000 7fd71c413740 1 -- 172.21.15.107:0/3624340692 wait complete. 2024-05-09T08:37:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.809+0000 7fd71c413740 20 client.16725 trim_cache size 0 max 16384 2024-05-09T08:37:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.829+0000 7fd71c413740 1 Processor -- start 2024-05-09T08:37:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.829+0000 7fd71c413740 1 -- start start 2024-05-09T08:37:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.829+0000 7fd71c413740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae255100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.829+0000 7fd71c413740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24b180 0x55c6ae261530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.829+0000 7fd71c413740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae2530a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.829+0000 7fd71c413740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd7141af090 con 0x55c6ae24a860 2024-05-09T08:37:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.829+0000 7fd71c413740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd7141ac960 con 0x55c6ae24b180 2024-05-09T08:37:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.829+0000 7fd71c413740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd7141aa7b0 con 0x55c6ae255a40 2024-05-09T08:37:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.829+0000 7fd712ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae255100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.829+0000 7fd712ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae255100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41466/0 (socket says 172.21.15.107:41466) 2024-05-09T08:37:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.829+0000 7fd712ffd640 1 -- 172.21.15.107:0/2124183147 learned_addr learned my addr 172.21.15.107:0/2124183147 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:00.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.829+0000 7fd70bfff640 1 --2- 172.21.15.107:0/2124183147 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24b180 0x55c6ae261530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.830+0000 7fd7137fe640 1 --2- 172.21.15.107:0/2124183147 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae2530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.830+0000 7fd712ffd640 1 -- 172.21.15.107:0/2124183147 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 msgr2=0x55c6ae2530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:00.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.830+0000 7fd712ffd640 1 --2- 172.21.15.107:0/2124183147 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae2530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.830+0000 7fd712ffd640 1 -- 172.21.15.107:0/2124183147 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24b180 msgr2=0x55c6ae261530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:00.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.830+0000 7fd712ffd640 1 --2- 172.21.15.107:0/2124183147 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24b180 0x55c6ae261530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.830+0000 7fd712ffd640 1 -- 172.21.15.107:0/2124183147 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6ae24d810 con 0x55c6ae24a860 2024-05-09T08:37:00.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.830+0000 7fd7137fe640 1 --2- 172.21.15.107:0/2124183147 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae2530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:00.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.830+0000 7fd712ffd640 1 --2- 172.21.15.107:0/2124183147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae255100 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fd704000c00 tx=0x7fd70402a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f3b1545831912c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:00.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.830+0000 7fd708ff9640 1 -- 172.21.15.107:0/2124183147 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd704035040 con 0x55c6ae24a860 2024-05-09T08:37:00.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.830+0000 7fd708ff9640 1 -- 172.21.15.107:0/2124183147 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd704040070 con 0x55c6ae24a860 2024-05-09T08:37:00.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.830+0000 7fd708ff9640 1 -- 172.21.15.107:0/2124183147 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd704003e90 con 0x55c6ae24a860 2024-05-09T08:37:00.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.831+0000 7fd71c413740 1 -- 172.21.15.107:0/2124183147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 msgr2=0x55c6ae255100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:00.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.831+0000 7fd71c413740 1 --2- 172.21.15.107:0/2124183147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae255100 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fd704000c00 tx=0x7fd70402a510 comp rx=0 tx=0).stop 2024-05-09T08:37:00.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.831+0000 7fd71c413740 1 -- 172.21.15.107:0/2124183147 shutdown_connections 2024-05-09T08:37:00.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.831+0000 7fd71c413740 1 --2- 172.21.15.107:0/2124183147 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae2530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.831+0000 7fd71c413740 1 --2- 172.21.15.107:0/2124183147 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24b180 0x55c6ae261530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.831+0000 7fd71c413740 1 --2- 172.21.15.107:0/2124183147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae255100 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.831+0000 7fd71c413740 1 -- 172.21.15.107:0/2124183147 >> 172.21.15.107:0/2124183147 conn(0x55c6ae246d70 msgr2=0x55c6ae252990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.831+0000 7fd71c413740 1 -- 172.21.15.107:0/2124183147 shutdown_connections 2024-05-09T08:37:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.831+0000 7fd71c413740 1 -- 172.21.15.107:0/2124183147 wait complete. 2024-05-09T08:37:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.832+0000 7fd71c413740 1 -- start start 2024-05-09T08:37:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.832+0000 7fd71c413740 1 Processor -- start 2024-05-09T08:37:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.832+0000 7fd71c413740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae24f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.832+0000 7fd71c413740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24b180 0x55c6ae24fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.832+0000 7fd71c413740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae2501e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.832+0000 7fd71c413740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd7141aae20 con 0x55c6ae24a860 2024-05-09T08:37:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.832+0000 7fd71c413740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd7141ac290 con 0x55c6ae24b180 2024-05-09T08:37:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.832+0000 7fd71c413740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd7141ab4f0 con 0x55c6ae255a40 2024-05-09T08:37:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.832+0000 7fd71c413740 10 client.0 _ll_register_callbacks cb 0x7fd704005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.832+0000 7fd712ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae24f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.832+0000 7fd712ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae24f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41480/0 (socket says 172.21.15.107:41480) 2024-05-09T08:37:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.832+0000 7fd712ffd640 1 -- 172.21.15.107:0/1313658958 learned_addr learned my addr 172.21.15.107:0/1313658958 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.833+0000 7fd70bfff640 1 --2- 172.21.15.107:0/1313658958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24b180 0x55c6ae24fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.833+0000 7fd7137fe640 1 --2- 172.21.15.107:0/1313658958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae2501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.833+0000 7fd712ffd640 1 -- 172.21.15.107:0/1313658958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 msgr2=0x55c6ae2501e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.833+0000 7fd712ffd640 1 --2- 172.21.15.107:0/1313658958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae2501e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.833+0000 7fd712ffd640 1 -- 172.21.15.107:0/1313658958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24b180 msgr2=0x55c6ae24fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.833+0000 7fd712ffd640 1 --2- 172.21.15.107:0/1313658958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24b180 0x55c6ae24fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.833+0000 7fd712ffd640 1 -- 172.21.15.107:0/1313658958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6ae2cae50 con 0x55c6ae24a860 2024-05-09T08:37:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.833+0000 7fd712ffd640 1 --2- 172.21.15.107:0/1313658958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae24f760 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd7040059d0 tx=0x7fd704009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1edf5fbfba914ed2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.833+0000 7fd6fa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.833+0000 7fd6fa7fc640 1 -- 172.21.15.107:0/1313658958 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd704035070 con 0x55c6ae24a860 2024-05-09T08:37:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.833+0000 7fd6fa7fc640 1 -- 172.21.15.107:0/1313658958 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd704004260 con 0x55c6ae24a860 2024-05-09T08:37:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.833+0000 7fd71c413740 1 -- 172.21.15.107:0/1313658958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd704040070 con 0x55c6ae24a860 2024-05-09T08:37:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.833+0000 7fd6fa7fc640 1 -- 172.21.15.107:0/1313658958 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd70406e050 con 0x55c6ae24a860 2024-05-09T08:37:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.833+0000 7fd71c413740 10 client.16731 fetch_fsmap learned FSMap version 26 2024-05-09T08:37:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.833+0000 7fd71c413740 1 -- 172.21.15.107:0/1313658958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c6ae242330 con 0x55c6ae24a860 2024-05-09T08:37:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.834+0000 7fd6fa7fc640 1 -- 172.21.15.107:0/1313658958 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd704040070 con 0x55c6ae24a860 2024-05-09T08:37:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.834+0000 7fd6fa7fc640 1 -- 172.21.15.107:0/1313658958 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd704004810 con 0x55c6ae24a860 2024-05-09T08:37:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.834+0000 7fd71c413740 10 client.16731 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T08:37:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.834+0000 7fd71c413740 10 client.16731 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.834+0000 7fd71c413740 20 client.16731 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.834+0000 7fd71c413740 1 --2- 172.21.15.107:0/1313658958 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c6ae242b40 0x55c6ae244f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.834+0000 7fd71c413740 4 client.16731 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:37:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.834+0000 7fd71c413740 1 -- 172.21.15.107:0/1313658958 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c6ae1b2dc0 con 0x55c6ae242b40 2024-05-09T08:37:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.835+0000 7fd70bfff640 1 --2- 172.21.15.107:0/1313658958 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c6ae242b40 0x55c6ae244f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.835+0000 7fd6fa7fc640 10 client.16731 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.835+0000 7fd70bfff640 1 --2- 172.21.15.107:0/1313658958 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c6ae242b40 0x55c6ae244f60 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa5be576e5000c57 server_cookie=e74bd5ceaa5a45d in_seq=0 out_seq=0 2024-05-09T08:37:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.943+0000 7fd6fa7fc640 1 -- 172.21.15.107:0/1313658958 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c6ae1b2dc0 con 0x55c6ae242b40 2024-05-09T08:37:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.943+0000 7fd6fa7fc640 10 client.16731 handle_command_reply: tid=0 2024-05-09T08:37:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.943+0000 7fd71c413740 1 client.16731 shutdown 2024-05-09T08:37:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.943+0000 7fd71c413740 1 -- 172.21.15.107:0/1313658958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 msgr2=0x55c6ae24f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.943+0000 7fd71c413740 1 --2- 172.21.15.107:0/1313658958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae24f760 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd7040059d0 tx=0x7fd704009590 comp rx=0 tx=0).stop 2024-05-09T08:37:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.943+0000 7fd71c413740 1 -- 172.21.15.107:0/1313658958 shutdown_connections 2024-05-09T08:37:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.943+0000 7fd71c413740 1 --2- 172.21.15.107:0/1313658958 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c6ae242b40 0x55c6ae244f60 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.943+0000 7fd71c413740 1 --2- 172.21.15.107:0/1313658958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6ae255a40 0x55c6ae2501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.943+0000 7fd6fa7fc640 0 client.16731 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.943+0000 7fd71c413740 1 --2- 172.21.15.107:0/1313658958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6ae24b180 0x55c6ae24fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.943+0000 7fd71c413740 1 --2- 172.21.15.107:0/1313658958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6ae24a860 0x55c6ae24f760 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.943+0000 7fd71c413740 1 -- 172.21.15.107:0/1313658958 >> 172.21.15.107:0/1313658958 conn(0x55c6ae246d70 msgr2=0x55c6ae253200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.944+0000 7fd71c413740 1 -- 172.21.15.107:0/1313658958 shutdown_connections 2024-05-09T08:37:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.944+0000 7fd71c413740 1 -- 172.21.15.107:0/1313658958 wait complete. 2024-05-09T08:37:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.944+0000 7fd71c413740 20 client.16731 trim_cache size 0 max 16384 2024-05-09T08:37:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:37:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:37:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:37:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:37:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.944+0000 7fd719709640 1 -- 172.21.15.107:0/2323279804 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd7000a2e20 msgr2=0x7fd7000a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.944+0000 7fd719709640 1 --2- 172.21.15.107:0/2323279804 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd7000a2e20 0x7fd7000a52e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd70c01e000 tx=0x7fd70c01f000 comp rx=0 tx=0).stop 2024-05-09T08:37:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.944+0000 7fd719709640 1 -- 172.21.15.107:0/2323279804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7141033e0 msgr2=0x7fd714198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.944+0000 7fd719709640 1 --2- 172.21.15.107:0/2323279804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7141033e0 0x7fd714198470 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc00c920 tx=0x7fd6fc00cde0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.944+0000 7fd719709640 1 -- 172.21.15.107:0/2323279804 shutdown_connections 2024-05-09T08:37:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.944+0000 7fd719709640 1 --2- 172.21.15.107:0/2323279804 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd7000a2e20 0x7fd7000a52e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.944+0000 7fd719709640 1 --2- 172.21.15.107:0/2323279804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd71410be00 0x7fd71419c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.944+0000 7fd719709640 1 --2- 172.21.15.107:0/2323279804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7141033e0 0x7fd714198470 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.944+0000 7fd719709640 1 --2- 172.21.15.107:0/2323279804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd714102ac0 0x7fd714197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.944+0000 7fd719709640 1 -- 172.21.15.107:0/2323279804 >> 172.21.15.107:0/2323279804 conn(0x7fd7140778b0 msgr2=0x7fd7140fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.944+0000 7fd719709640 1 -- 172.21.15.107:0/2323279804 shutdown_connections 2024-05-09T08:37:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:00.944+0000 7fd719709640 1 -- 172.21.15.107:0/2323279804 wait complete. 2024-05-09T08:37:02.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:02 smithi157 ceph-mon[33614]: pgmap v113: 129 pgs: 129 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:37:02.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:02 smithi110 ceph-mon[33622]: pgmap v113: 129 pgs: 129 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:37:02.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:02 smithi107 ceph-mon[30122]: pgmap v113: 129 pgs: 129 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:37:04.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:04 smithi157 ceph-mon[33614]: pgmap v114: 129 pgs: 129 active+clean; 3.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 37 MiB/s wr, 46 op/s 2024-05-09T08:37:04.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:04 smithi110 ceph-mon[33622]: pgmap v114: 129 pgs: 129 active+clean; 3.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 37 MiB/s wr, 46 op/s 2024-05-09T08:37:04.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:04 smithi107 ceph-mon[30122]: pgmap v114: 129 pgs: 129 active+clean; 3.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 37 MiB/s wr, 46 op/s 2024-05-09T08:37:06.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:06 smithi157 ceph-mon[33614]: pgmap v115: 129 pgs: 129 active+clean; 3.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 34 op/s 2024-05-09T08:37:06.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:06 smithi110 ceph-mon[33622]: pgmap v115: 129 pgs: 129 active+clean; 3.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 34 op/s 2024-05-09T08:37:06.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:06 smithi107 ceph-mon[30122]: pgmap v115: 129 pgs: 129 active+clean; 3.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 34 op/s 2024-05-09T08:37:08.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:08 smithi157 ceph-mon[33614]: pgmap v116: 129 pgs: 129 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 37 MiB/s wr, 42 op/s 2024-05-09T08:37:08.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:08 smithi110 ceph-mon[33622]: pgmap v116: 129 pgs: 129 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 37 MiB/s wr, 42 op/s 2024-05-09T08:37:08.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:08 smithi107 ceph-mon[30122]: pgmap v116: 129 pgs: 129 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 37 MiB/s wr, 42 op/s 2024-05-09T08:37:10.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:10 smithi157 ceph-mon[33614]: pgmap v117: 129 pgs: 129 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 34 op/s 2024-05-09T08:37:10.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:10 smithi110 ceph-mon[33622]: pgmap v117: 129 pgs: 129 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 34 op/s 2024-05-09T08:37:10.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:10 smithi107 ceph-mon[30122]: pgmap v117: 129 pgs: 129 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 34 op/s 2024-05-09T08:37:12.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:12 smithi157 ceph-mon[33614]: pgmap v118: 129 pgs: 129 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 34 op/s 2024-05-09T08:37:12.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:12 smithi157 ceph-mon[33614]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:37:12.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:12 smithi110 ceph-mon[33622]: pgmap v118: 129 pgs: 129 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 34 op/s 2024-05-09T08:37:12.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:12 smithi110 ceph-mon[33622]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:37:12.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:12 smithi107 ceph-mon[30122]: pgmap v118: 129 pgs: 129 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 34 op/s 2024-05-09T08:37:12.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:12 smithi107 ceph-mon[30122]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:37:13.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:37:13.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:37:13.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.464+0000 7feb6039e640 1 -- 172.21.15.107:0/1232379067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb58100c10 msgr2=0x7feb58101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:13.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.464+0000 7feb6039e640 1 --2- 172.21.15.107:0/1232379067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb58100c10 0x7feb58101090 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7feb48009930 tx=0x7feb4802a420 comp rx=0 tx=0).stop 2024-05-09T08:37:13.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.464+0000 7feb6039e640 1 -- 172.21.15.107:0/1232379067 shutdown_connections 2024-05-09T08:37:13.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.464+0000 7feb6039e640 1 --2- 172.21.15.107:0/1232379067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb581015d0 0x7feb5810d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.464+0000 7feb6039e640 1 --2- 172.21.15.107:0/1232379067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb58100c10 0x7feb58101090 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.464+0000 7feb6039e640 1 --2- 172.21.15.107:0/1232379067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb58106700 0x7feb58106ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.464+0000 7feb6039e640 1 -- 172.21.15.107:0/1232379067 >> 172.21.15.107:0/1232379067 conn(0x7feb580762b0 msgr2=0x7feb580766c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:13.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.464+0000 7feb6039e640 1 -- 172.21.15.107:0/1232379067 shutdown_connections 2024-05-09T08:37:13.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.464+0000 7feb6039e640 1 -- 172.21.15.107:0/1232379067 wait complete. 2024-05-09T08:37:13.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.465+0000 7feb6039e640 1 Processor -- start 2024-05-09T08:37:13.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.465+0000 7feb6039e640 1 -- start start 2024-05-09T08:37:13.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.465+0000 7feb6039e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb58100c10 0x7feb58198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.465+0000 7feb6039e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb581015d0 0x7feb581993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.465+0000 7feb6039e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb58106700 0x7feb581a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.465+0000 7feb6039e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feb581109c0 con 0x7feb58106700 2024-05-09T08:37:13.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.465+0000 7feb6039e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feb58110840 con 0x7feb58100c10 2024-05-09T08:37:13.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.465+0000 7feb6039e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feb58110b40 con 0x7feb581015d0 2024-05-09T08:37:13.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb5e93b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb58106700 0x7feb581a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb5e93b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb58106700 0x7feb581a3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50134/0 (socket says 172.21.15.107:50134) 2024-05-09T08:37:13.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb5e93b640 1 -- 172.21.15.107:0/1021641299 learned_addr learned my addr 172.21.15.107:0/1021641299 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:13.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb5e13a640 1 --2- 172.21.15.107:0/1021641299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb58100c10 0x7feb58198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb5d939640 1 --2- 172.21.15.107:0/1021641299 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb581015d0 0x7feb581993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb5e93b640 1 -- 172.21.15.107:0/1021641299 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb581015d0 msgr2=0x7feb581993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:13.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb5e93b640 1 --2- 172.21.15.107:0/1021641299 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb581015d0 0x7feb581993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb5e93b640 1 -- 172.21.15.107:0/1021641299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb58100c10 msgr2=0x7feb58198e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:13.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb5e93b640 1 --2- 172.21.15.107:0/1021641299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb58100c10 0x7feb58198e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb5e93b640 1 -- 172.21.15.107:0/1021641299 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb581a4420 con 0x7feb58106700 2024-05-09T08:37:13.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb5d939640 1 --2- 172.21.15.107:0/1021641299 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb581015d0 0x7feb581993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:13.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb5e93b640 1 --2- 172.21.15.107:0/1021641299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb58106700 0x7feb581a3ee0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7feb50009fd0 tx=0x7feb5000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:13.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb4f7fe640 1 -- 172.21.15.107:0/1021641299 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb50017070 con 0x7feb58106700 2024-05-09T08:37:13.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb4f7fe640 1 -- 172.21.15.107:0/1021641299 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb50003e20 con 0x7feb58106700 2024-05-09T08:37:13.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb6039e640 1 -- 172.21.15.107:0/1021641299 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb581a4650 con 0x7feb58106700 2024-05-09T08:37:13.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb4f7fe640 1 -- 172.21.15.107:0/1021641299 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb50010460 con 0x7feb58106700 2024-05-09T08:37:13.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.466+0000 7feb6039e640 1 -- 172.21.15.107:0/1021641299 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb581a4b70 con 0x7feb58106700 2024-05-09T08:37:13.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.469+0000 7feb63187740 1 Processor -- start 2024-05-09T08:37:13.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.469+0000 7feb4f7fe640 1 -- 172.21.15.107:0/1021641299 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7feb50003970 con 0x7feb58106700 2024-05-09T08:37:13.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.469+0000 7feb63187740 1 -- start start 2024-05-09T08:37:13.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb63187740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 0x561ef389e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb63187740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef389eba0 0x561ef38a8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb63187740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef38a9270 0x561ef38ab660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb63187740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561ef3803d00 con 0x561ef38a9270 2024-05-09T08:37:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb63187740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561ef37ab4c0 con 0x561ef389e280 2024-05-09T08:37:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb63187740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x561ef37ffcc0 con 0x561ef389eba0 2024-05-09T08:37:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb5d939640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef38a9270 0x561ef38ab660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb5d939640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef38a9270 0x561ef38ab660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50150/0 (socket says 172.21.15.107:50150) 2024-05-09T08:37:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb5d939640 1 -- 172.21.15.107:0/1728554891 learned_addr learned my addr 172.21.15.107:0/1728554891 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb5d939640 1 --2- 172.21.15.107:0/1728554891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 0x561ef389e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb5e13a640 1 --2- 172.21.15.107:0/1728554891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef389eba0 0x561ef38a8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb4f7fe640 1 --2- 172.21.15.107:0/1021641299 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feb440a2d80 0x7feb440a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb5d939640 1 -- 172.21.15.107:0/1728554891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef389eba0 msgr2=0x561ef38a8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb5d939640 1 --2- 172.21.15.107:0/1728554891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef389eba0 0x561ef38a8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb5d939640 1 -- 172.21.15.107:0/1728554891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 msgr2=0x561ef389e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:13.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb5d939640 1 --2- 172.21.15.107:0/1728554891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 0x561ef389e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.470+0000 7feb5d939640 1 -- 172.21.15.107:0/1728554891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ef35e17d0 con 0x561ef38a9270 2024-05-09T08:37:13.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.471+0000 7feb5e93b640 1 --2- 172.21.15.107:0/1021641299 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feb440a2d80 0x7feb440a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.471+0000 7feb5d939640 1 --2- 172.21.15.107:0/1728554891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef38a9270 0x561ef38ab660 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7feb480057c0 tx=0x7feb480058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75838ed14a1e3dd8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:13.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.471+0000 7feb4cbfe640 1 -- 172.21.15.107:0/1728554891 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb48035040 con 0x561ef38a9270 2024-05-09T08:37:13.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.471+0000 7feb4f7fe640 1 -- 172.21.15.107:0/1021641299 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7feb500cf590 con 0x7feb58106700 2024-05-09T08:37:13.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.471+0000 7feb4cbfe640 1 -- 172.21.15.107:0/1728554891 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb480035f0 con 0x561ef38a9270 2024-05-09T08:37:13.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.471+0000 7feb4cbfe640 1 -- 172.21.15.107:0/1728554891 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb480044e0 con 0x561ef38a9270 2024-05-09T08:37:13.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.472+0000 7feb63187740 1 -- 172.21.15.107:0/1728554891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef38a9270 msgr2=0x561ef38ab660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:13.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.472+0000 7feb63187740 1 --2- 172.21.15.107:0/1728554891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef38a9270 0x561ef38ab660 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7feb480057c0 tx=0x7feb480058d0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.472+0000 7feb5e93b640 1 --2- 172.21.15.107:0/1021641299 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feb440a2d80 0x7feb440a5240 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7feb50097bb0 tx=0x7feb500979f0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:13.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.472+0000 7feb63187740 1 -- 172.21.15.107:0/1728554891 shutdown_connections 2024-05-09T08:37:13.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.472+0000 7feb63187740 1 --2- 172.21.15.107:0/1728554891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef38a9270 0x561ef38ab660 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.472+0000 7feb63187740 1 --2- 172.21.15.107:0/1728554891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef389eba0 0x561ef38a8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.472+0000 7feb63187740 1 --2- 172.21.15.107:0/1728554891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 0x561ef389e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.472+0000 7feb63187740 1 -- 172.21.15.107:0/1728554891 >> 172.21.15.107:0/1728554891 conn(0x561ef389a5a0 msgr2=0x561ef389a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:13.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.472+0000 7feb63187740 1 -- 172.21.15.107:0/1728554891 shutdown_connections 2024-05-09T08:37:13.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.472+0000 7feb63187740 1 -- 172.21.15.107:0/1728554891 wait complete. 2024-05-09T08:37:13.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.472+0000 7feb63187740 1 -- start start 2024-05-09T08:37:13.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.472+0000 7feb63187740 1 Processor -- start 2024-05-09T08:37:13.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.472+0000 7feb63187740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef389e280 0x561ef38a6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.472+0000 7feb63187740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef389eba0 0x561ef38a6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.472+0000 7feb63187740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef38a9270 0x561ef38a6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb63187740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561ef38ac000 con 0x561ef389eba0 2024-05-09T08:37:13.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb63187740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561ef37beb90 con 0x561ef38a9270 2024-05-09T08:37:13.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb63187740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x561ef38ac180 con 0x561ef389e280 2024-05-09T08:37:13.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb63187740 10 client.0 _ll_register_callbacks cb 0x561ef37fada0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:13.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb5e13a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef389eba0 0x561ef38a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb5e13a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef389eba0 0x561ef38a6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50154/0 (socket says 172.21.15.107:50154) 2024-05-09T08:37:13.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb5e13a640 1 -- 172.21.15.107:0/1696087578 learned_addr learned my addr 172.21.15.107:0/1696087578 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:13.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb5d939640 1 --2- 172.21.15.107:0/1696087578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef389e280 0x561ef38a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb5d939640 1 --2- 172.21.15.107:0/1696087578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef38a9270 0x561ef38a6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb5e13a640 1 -- 172.21.15.107:0/1696087578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef389e280 msgr2=0x561ef38a6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:13.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb5e13a640 1 --2- 172.21.15.107:0/1696087578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef389e280 0x561ef38a6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb5e13a640 1 -- 172.21.15.107:0/1696087578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef38a9270 msgr2=0x561ef38a6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:13.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb5e13a640 1 --2- 172.21.15.107:0/1696087578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef38a9270 0x561ef38a6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb5e13a640 1 -- 172.21.15.107:0/1696087578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ef35173f0 con 0x561ef389eba0 2024-05-09T08:37:13.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb5d939640 1 --2- 172.21.15.107:0/1696087578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef38a9270 0x561ef38a6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:13.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb5e13a640 1 --2- 172.21.15.107:0/1696087578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef389eba0 0x561ef38a6760 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7feb54010850 tx=0x7feb54010d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38844b6c88d71022 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:13.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:13.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb41ffb640 1 -- 172.21.15.107:0/1696087578 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5400fb10 con 0x561ef389eba0 2024-05-09T08:37:13.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb41ffb640 1 -- 172.21.15.107:0/1696087578 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7feb5400fcb0 con 0x561ef389eba0 2024-05-09T08:37:13.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb63187740 1 -- 172.21.15.107:0/1696087578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb480035f0 con 0x561ef389eba0 2024-05-09T08:37:13.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb41ffb640 1 -- 172.21.15.107:0/1696087578 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feb54024050 con 0x561ef389eba0 2024-05-09T08:37:13.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb63187740 10 client.16743 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:13.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.473+0000 7feb63187740 1 -- 172.21.15.107:0/1696087578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561ef34d70f0 con 0x561ef389eba0 2024-05-09T08:37:13.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.474+0000 7feb41ffb640 1 -- 172.21.15.107:0/1696087578 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb480035f0 con 0x561ef389eba0 2024-05-09T08:37:13.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.474+0000 7feb41ffb640 1 -- 172.21.15.107:0/1696087578 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7feb54018490 con 0x561ef389eba0 2024-05-09T08:37:13.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.474+0000 7feb63187740 10 client.16743 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:13.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.474+0000 7feb63187740 10 client.16743 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:13.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.474+0000 7feb63187740 20 client.16743 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:13.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.474+0000 7feb63187740 1 --2- 172.21.15.107:0/1696087578 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561ef38b9730 0x561ef38bbb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.474+0000 7feb63187740 4 client.16743 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:13.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.474+0000 7feb63187740 1 -- 172.21.15.107:0/1696087578 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ef3811430 con 0x561ef38b9730 2024-05-09T08:37:13.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.474+0000 7feb5d939640 1 --2- 172.21.15.107:0/1696087578 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561ef38b9730 0x561ef38bbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.475+0000 7feb5d939640 1 --2- 172.21.15.107:0/1696087578 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561ef38b9730 0x561ef38bbb50 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b91b2fdf6b5e002b server_cookie=e73f21f7a6a53761 in_seq=0 out_seq=0 2024-05-09T08:37:13.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.475+0000 7feb41ffb640 10 client.16743 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:13.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.475+0000 7feb41ffb640 1 -- 172.21.15.107:0/1696087578 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ef3811430 con 0x561ef38b9730 2024-05-09T08:37:13.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.475+0000 7feb41ffb640 10 client.16743 handle_command_reply: tid=0 2024-05-09T08:37:13.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.475+0000 7feb63187740 1 client.16743 shutdown 2024-05-09T08:37:13.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.475+0000 7feb63187740 1 -- 172.21.15.107:0/1696087578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef389eba0 msgr2=0x561ef38a6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:13.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.475+0000 7feb63187740 1 --2- 172.21.15.107:0/1696087578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef389eba0 0x561ef38a6760 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7feb54010850 tx=0x7feb54010d10 comp rx=0 tx=0).stop 2024-05-09T08:37:13.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.475+0000 7feb63187740 1 -- 172.21.15.107:0/1696087578 shutdown_connections 2024-05-09T08:37:13.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.475+0000 7feb63187740 1 --2- 172.21.15.107:0/1696087578 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561ef38b9730 0x561ef38bbb50 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.475+0000 7feb63187740 1 --2- 172.21.15.107:0/1696087578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef38a9270 0x561ef38a6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.475+0000 7feb63187740 1 --2- 172.21.15.107:0/1696087578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef389eba0 0x561ef38a6760 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.475+0000 7feb63187740 1 --2- 172.21.15.107:0/1696087578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef389e280 0x561ef38a6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.475+0000 7feb63187740 1 -- 172.21.15.107:0/1696087578 >> 172.21.15.107:0/1696087578 conn(0x561ef389a5a0 msgr2=0x561ef38a03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:13.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.475+0000 7feb41ffb640 0 client.16743 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:13.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.476+0000 7feb63187740 1 -- 172.21.15.107:0/1696087578 shutdown_connections 2024-05-09T08:37:13.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.476+0000 7feb63187740 1 -- 172.21.15.107:0/1696087578 wait complete. 2024-05-09T08:37:13.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.476+0000 7feb63187740 20 client.16743 trim_cache size 0 max 16384 2024-05-09T08:37:13.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.497+0000 7feb63187740 1 Processor -- start 2024-05-09T08:37:13.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.497+0000 7feb63187740 1 -- start start 2024-05-09T08:37:13.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.497+0000 7feb63187740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 0x561ef38a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.497+0000 7feb63187740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef389eba0 0x561ef38a6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.497+0000 7feb63187740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef38a9270 0x561ef38c1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.497+0000 7feb63187740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feb581b1380 con 0x561ef389eba0 2024-05-09T08:37:13.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.497+0000 7feb63187740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feb581aec50 con 0x561ef389e280 2024-05-09T08:37:13.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.497+0000 7feb63187740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feb581acaa0 con 0x561ef38a9270 2024-05-09T08:37:13.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.497+0000 7feb5e13a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef389eba0 0x561ef38a6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.497+0000 7feb5e13a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef389eba0 0x561ef38a6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50162/0 (socket says 172.21.15.107:50162) 2024-05-09T08:37:13.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.497+0000 7feb5e13a640 1 -- 172.21.15.107:0/2721310703 learned_addr learned my addr 172.21.15.107:0/2721310703 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:13.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.497+0000 7feb5d939640 1 --2- 172.21.15.107:0/2721310703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 0x561ef38a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.497+0000 7feb5d939640 1 --2- 172.21.15.107:0/2721310703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef38a9270 0x561ef38c1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.498+0000 7feb5e13a640 1 -- 172.21.15.107:0/2721310703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef38a9270 msgr2=0x561ef38c1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:13.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.498+0000 7feb5e13a640 1 --2- 172.21.15.107:0/2721310703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef38a9270 0x561ef38c1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.498+0000 7feb5e13a640 1 -- 172.21.15.107:0/2721310703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 msgr2=0x561ef38a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.498+0000 7feb5e13a640 1 --2- 172.21.15.107:0/2721310703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 0x561ef38a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.498+0000 7feb5e13a640 1 -- 172.21.15.107:0/2721310703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ef38b8230 con 0x561ef389eba0 2024-05-09T08:37:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.498+0000 7feb5d939640 1 --2- 172.21.15.107:0/2721310703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef38a9270 0x561ef38c1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:13.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.498+0000 7feb5e13a640 1 --2- 172.21.15.107:0/2721310703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef389eba0 0x561ef38a6ac0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7feb5400acb0 tx=0x7feb54010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da8723d2526882fe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:13.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.498+0000 7feb43fff640 1 -- 172.21.15.107:0/2721310703 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5402d070 con 0x561ef389eba0 2024-05-09T08:37:13.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.498+0000 7feb43fff640 1 -- 172.21.15.107:0/2721310703 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb54018af0 con 0x561ef389eba0 2024-05-09T08:37:13.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.498+0000 7feb43fff640 1 -- 172.21.15.107:0/2721310703 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb54005290 con 0x561ef389eba0 2024-05-09T08:37:13.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 -- 172.21.15.107:0/2721310703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef389eba0 msgr2=0x561ef38a6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:13.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 --2- 172.21.15.107:0/2721310703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef389eba0 0x561ef38a6ac0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7feb5400acb0 tx=0x7feb54010430 comp rx=0 tx=0).stop 2024-05-09T08:37:13.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 -- 172.21.15.107:0/2721310703 shutdown_connections 2024-05-09T08:37:13.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 --2- 172.21.15.107:0/2721310703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef38a9270 0x561ef38c1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 --2- 172.21.15.107:0/2721310703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef389eba0 0x561ef38a6ac0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 --2- 172.21.15.107:0/2721310703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 0x561ef38a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 -- 172.21.15.107:0/2721310703 >> 172.21.15.107:0/2721310703 conn(0x561ef389a5a0 msgr2=0x561ef38997e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:13.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 -- 172.21.15.107:0/2721310703 shutdown_connections 2024-05-09T08:37:13.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 -- 172.21.15.107:0/2721310703 wait complete. 2024-05-09T08:37:13.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 -- start start 2024-05-09T08:37:13.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 Processor -- start 2024-05-09T08:37:13.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 0x561ef38a2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef389eba0 0x561ef38a3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef38a9270 0x561ef38a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feb581ad110 con 0x561ef38a9270 2024-05-09T08:37:13.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feb581ae580 con 0x561ef389e280 2024-05-09T08:37:13.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.499+0000 7feb63187740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feb581ad7e0 con 0x561ef389eba0 2024-05-09T08:37:13.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb63187740 10 client.0 _ll_register_callbacks cb 0x7feb5400a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:13.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb5d939640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef38a9270 0x561ef38a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb5d939640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 0x561ef38a2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb5d939640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 0x561ef38a2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:47908/0 (socket says 172.21.15.107:47908) 2024-05-09T08:37:13.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb5d939640 1 -- 172.21.15.107:0/4120455909 learned_addr learned my addr 172.21.15.107:0/4120455909 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:13.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb5e13a640 1 --2- 172.21.15.107:0/4120455909 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef389eba0 0x561ef38a3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb5d939640 1 -- 172.21.15.107:0/4120455909 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef389eba0 msgr2=0x561ef38a3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:13.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb5d939640 1 --2- 172.21.15.107:0/4120455909 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef389eba0 0x561ef38a3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb5d939640 1 -- 172.21.15.107:0/4120455909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef38a9270 msgr2=0x561ef38a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:13.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb5d939640 1 --2- 172.21.15.107:0/4120455909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef38a9270 0x561ef38a38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb5d939640 1 -- 172.21.15.107:0/4120455909 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ef391ec90 con 0x561ef389e280 2024-05-09T08:37:13.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb5e13a640 1 --2- 172.21.15.107:0/4120455909 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef389eba0 0x561ef38a3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:13.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb5d939640 1 --2- 172.21.15.107:0/4120455909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 0x561ef38a2e50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7feb48031a70 tx=0x7feb48031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74582bbe3fe0e027 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:13.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:37:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb41ffb640 1 -- 172.21.15.107:0/4120455909 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb48035040 con 0x561ef389e280 2024-05-09T08:37:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb63187740 1 -- 172.21.15.107:0/4120455909 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb54018af0 con 0x561ef389e280 2024-05-09T08:37:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb41ffb640 1 -- 172.21.15.107:0/4120455909 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7feb48004300 con 0x561ef389e280 2024-05-09T08:37:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb41ffb640 1 -- 172.21.15.107:0/4120455909 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feb480035f0 con 0x561ef389e280 2024-05-09T08:37:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb63187740 10 client.25961 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.500+0000 7feb63187740 1 -- 172.21.15.107:0/4120455909 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561ef391fee0 con 0x561ef389e280 2024-05-09T08:37:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.501+0000 7feb41ffb640 1 -- 172.21.15.107:0/4120455909 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb54018af0 con 0x561ef389e280 2024-05-09T08:37:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.501+0000 7feb41ffb640 1 -- 172.21.15.107:0/4120455909 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7feb4803a040 con 0x561ef389e280 2024-05-09T08:37:13.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.501+0000 7feb63187740 10 client.25961 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:13.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.501+0000 7feb63187740 10 client.25961 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:13.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.501+0000 7feb63187740 20 client.25961 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:13.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.501+0000 7feb63187740 1 --2- 172.21.15.107:0/4120455909 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561ef3920170 0x561ef3922560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.501+0000 7feb63187740 4 client.25961 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:37:13.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.501+0000 7feb63187740 1 -- 172.21.15.107:0/4120455909 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561ef380b780 con 0x561ef3920170 2024-05-09T08:37:13.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.501+0000 7feb5e13a640 1 --2- 172.21.15.107:0/4120455909 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561ef3920170 0x561ef3922560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb5e13a640 1 --2- 172.21.15.107:0/4120455909 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561ef3920170 0x561ef3922560 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ccd9ad709ad3388 server_cookie=49eff9792c085a6a in_seq=0 out_seq=0 2024-05-09T08:37:13.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb41ffb640 10 client.25961 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:13.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb41ffb640 1 -- 172.21.15.107:0/4120455909 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561ef380b780 con 0x561ef3920170 2024-05-09T08:37:13.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb41ffb640 10 client.25961 handle_command_reply: tid=0 2024-05-09T08:37:13.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb63187740 1 client.25961 shutdown 2024-05-09T08:37:13.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb63187740 1 -- 172.21.15.107:0/4120455909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 msgr2=0x561ef38a2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:13.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb63187740 1 --2- 172.21.15.107:0/4120455909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 0x561ef38a2e50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7feb48031a70 tx=0x7feb48031400 comp rx=0 tx=0).stop 2024-05-09T08:37:13.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb63187740 1 -- 172.21.15.107:0/4120455909 shutdown_connections 2024-05-09T08:37:13.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb63187740 1 --2- 172.21.15.107:0/4120455909 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561ef3920170 0x561ef3922560 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb63187740 1 --2- 172.21.15.107:0/4120455909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ef38a9270 0x561ef38a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb41ffb640 0 client.25961 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:13.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb63187740 1 --2- 172.21.15.107:0/4120455909 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561ef389eba0 0x561ef38a3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb63187740 1 --2- 172.21.15.107:0/4120455909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561ef389e280 0x561ef38a2e50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb63187740 1 -- 172.21.15.107:0/4120455909 >> 172.21.15.107:0/4120455909 conn(0x561ef389a5a0 msgr2=0x561ef38a7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:13.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb63187740 1 -- 172.21.15.107:0/4120455909 shutdown_connections 2024-05-09T08:37:13.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb63187740 1 -- 172.21.15.107:0/4120455909 wait complete. 2024-05-09T08:37:13.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.502+0000 7feb63187740 20 client.25961 trim_cache size 0 max 16384 2024-05-09T08:37:13.515 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:37:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:37:13.515 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:37:13.516 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:37:13.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.503+0000 7feb6039e640 1 -- 172.21.15.107:0/1021641299 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feb440a2d80 msgr2=0x7feb440a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:13.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.503+0000 7feb6039e640 1 --2- 172.21.15.107:0/1021641299 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feb440a2d80 0x7feb440a5240 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7feb50097bb0 tx=0x7feb500979f0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.503+0000 7feb6039e640 1 -- 172.21.15.107:0/1021641299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb58106700 msgr2=0x7feb581a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:13.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.503+0000 7feb6039e640 1 --2- 172.21.15.107:0/1021641299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb58106700 0x7feb581a3ee0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7feb50009fd0 tx=0x7feb5000b260 comp rx=0 tx=0).stop 2024-05-09T08:37:13.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.503+0000 7feb6039e640 1 -- 172.21.15.107:0/1021641299 shutdown_connections 2024-05-09T08:37:13.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.503+0000 7feb6039e640 1 --2- 172.21.15.107:0/1021641299 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feb440a2d80 0x7feb440a5240 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.503+0000 7feb6039e640 1 --2- 172.21.15.107:0/1021641299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb58106700 0x7feb581a3ee0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.503+0000 7feb6039e640 1 --2- 172.21.15.107:0/1021641299 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb581015d0 0x7feb581993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.503+0000 7feb6039e640 1 --2- 172.21.15.107:0/1021641299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb58100c10 0x7feb58198e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.503+0000 7feb6039e640 1 -- 172.21.15.107:0/1021641299 >> 172.21.15.107:0/1021641299 conn(0x7feb580762b0 msgr2=0x7feb580fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:13.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.503+0000 7feb6039e640 1 -- 172.21.15.107:0/1021641299 shutdown_connections 2024-05-09T08:37:13.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.503+0000 7feb6039e640 1 -- 172.21.15.107:0/1021641299 wait complete. 2024-05-09T08:37:13.519 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:37:13.519 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:37:13.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.643+0000 7f171789e640 1 -- 172.21.15.107:0/2146306306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17101033b0 msgr2=0x7f171010b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:13.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.643+0000 7f171789e640 1 --2- 172.21.15.107:0/2146306306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17101033b0 0x7f171010b890 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f1700009900 tx=0x7f170002a480 comp rx=0 tx=0).stop 2024-05-09T08:37:13.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.643+0000 7f171789e640 1 -- 172.21.15.107:0/2146306306 shutdown_connections 2024-05-09T08:37:13.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.643+0000 7f171789e640 1 --2- 172.21.15.107:0/2146306306 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f171010be60 0x7f171010e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.643+0000 7f171789e640 1 --2- 172.21.15.107:0/2146306306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17101033b0 0x7f171010b890 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.643+0000 7f171789e640 1 --2- 172.21.15.107:0/2146306306 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1710102a90 0x7f1710102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.643+0000 7f171789e640 1 -- 172.21.15.107:0/2146306306 >> 172.21.15.107:0/2146306306 conn(0x7f1710076ab0 msgr2=0x7f1710076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:13.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.643+0000 7f171789e640 1 -- 172.21.15.107:0/2146306306 shutdown_connections 2024-05-09T08:37:13.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.644+0000 7f171789e640 1 -- 172.21.15.107:0/2146306306 wait complete. 2024-05-09T08:37:13.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.644+0000 7f171789e640 1 Processor -- start 2024-05-09T08:37:13.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.644+0000 7f171789e640 1 -- start start 2024-05-09T08:37:13.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.644+0000 7f171789e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1710102a90 0x7f17101006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.644+0000 7f171789e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17101033b0 0x7f1710100c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.644+0000 7f171789e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f171010be60 0x7f1710101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.644+0000 7f171789e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1710111aa0 con 0x7f171010be60 2024-05-09T08:37:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.644+0000 7f171789e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1710111920 con 0x7f1710102a90 2024-05-09T08:37:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.644+0000 7f171789e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1710111c20 con 0x7f17101033b0 2024-05-09T08:37:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.644+0000 7f1715e3b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f171010be60 0x7f1710101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.644+0000 7f1715e3b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f171010be60 0x7f1710101160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50186/0 (socket says 172.21.15.107:50186) 2024-05-09T08:37:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.645+0000 7f1715e3b640 1 -- 172.21.15.107:0/1208395572 learned_addr learned my addr 172.21.15.107:0/1208395572 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:13.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.645+0000 7f1714e39640 1 --2- 172.21.15.107:0/1208395572 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17101033b0 0x7f1710100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.645+0000 7f171563a640 1 --2- 172.21.15.107:0/1208395572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1710102a90 0x7f17101006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.645+0000 7f1715e3b640 1 -- 172.21.15.107:0/1208395572 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17101033b0 msgr2=0x7f1710100c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:13.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.645+0000 7f1715e3b640 1 --2- 172.21.15.107:0/1208395572 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17101033b0 0x7f1710100c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.645+0000 7f1715e3b640 1 -- 172.21.15.107:0/1208395572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1710102a90 msgr2=0x7f17101006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:13.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.645+0000 7f1715e3b640 1 --2- 172.21.15.107:0/1208395572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1710102a90 0x7f17101006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.645+0000 7f1715e3b640 1 -- 172.21.15.107:0/1208395572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17101aa9e0 con 0x7f171010be60 2024-05-09T08:37:13.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.645+0000 7f171563a640 1 --2- 172.21.15.107:0/1208395572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1710102a90 0x7f17101006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:13.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.645+0000 7f1715e3b640 1 --2- 172.21.15.107:0/1208395572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f171010be60 0x7f1710101160 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f170800e970 tx=0x7f170800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:13.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.645+0000 7f17067fc640 1 -- 172.21.15.107:0/1208395572 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170800ccc0 con 0x7f171010be60 2024-05-09T08:37:13.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.645+0000 7f17067fc640 1 -- 172.21.15.107:0/1208395572 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17080054d0 con 0x7f171010be60 2024-05-09T08:37:13.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.645+0000 7f171789e640 1 -- 172.21.15.107:0/1208395572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17101aac10 con 0x7f171010be60 2024-05-09T08:37:13.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.645+0000 7f17067fc640 1 -- 172.21.15.107:0/1208395572 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1708010430 con 0x7f171010be60 2024-05-09T08:37:13.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.645+0000 7f171789e640 1 -- 172.21.15.107:0/1208395572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17101ab0e0 con 0x7f171010be60 2024-05-09T08:37:13.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.646+0000 7f171789e640 1 -- 172.21.15.107:0/1208395572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17100696e0 con 0x7f171010be60 2024-05-09T08:37:13.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.648+0000 7f17067fc640 1 -- 172.21.15.107:0/1208395572 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1708005020 con 0x7f171010be60 2024-05-09T08:37:13.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.648+0000 7f17067fc640 1 --2- 172.21.15.107:0/1208395572 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f16fc0a2d00 0x7f16fc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:13.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.649+0000 7f171563a640 1 --2- 172.21.15.107:0/1208395572 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f16fc0a2d00 0x7f16fc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:13.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.649+0000 7f17067fc640 1 -- 172.21.15.107:0/1208395572 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f17080cfb30 con 0x7f171010be60 2024-05-09T08:37:13.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.649+0000 7f171563a640 1 --2- 172.21.15.107:0/1208395572 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f16fc0a2d00 0x7f16fc0a51c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f170c005fd0 tx=0x7f170c0093e0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:13.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.654+0000 7f17067fc640 1 -- 172.21.15.107:0/1208395572 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f170809c3f0 con 0x7f171010be60 2024-05-09T08:37:13.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.922+0000 7f171789e640 1 -- 172.21.15.107:0/1208395572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1710068aa0 con 0x7f171010be60 2024-05-09T08:37:13.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.922+0000 7f17067fc640 1 -- 172.21.15.107:0/1208395572 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11887 (secure 0 0 0) 0x7f170809bcf0 con 0x7f171010be60 2024-05-09T08:37:13.921 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:37:13.921 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:37:10.930317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:37:13.921 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2024-05-09T08:37:13.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.925+0000 7f171789e640 1 -- 172.21.15.107:0/1208395572 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f16fc0a2d00 msgr2=0x7f16fc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:13.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.925+0000 7f171789e640 1 --2- 172.21.15.107:0/1208395572 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f16fc0a2d00 0x7f16fc0a51c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f170c005fd0 tx=0x7f170c0093e0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.925+0000 7f171789e640 1 -- 172.21.15.107:0/1208395572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f171010be60 msgr2=0x7f1710101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:13.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.925+0000 7f171789e640 1 --2- 172.21.15.107:0/1208395572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f171010be60 0x7f1710101160 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f170800e970 tx=0x7f170800ee30 comp rx=0 tx=0).stop 2024-05-09T08:37:13.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.925+0000 7f171789e640 1 -- 172.21.15.107:0/1208395572 shutdown_connections 2024-05-09T08:37:13.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.925+0000 7f171789e640 1 --2- 172.21.15.107:0/1208395572 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f16fc0a2d00 0x7f16fc0a51c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.925+0000 7f171789e640 1 --2- 172.21.15.107:0/1208395572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f171010be60 0x7f1710101160 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.925+0000 7f171789e640 1 --2- 172.21.15.107:0/1208395572 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17101033b0 0x7f1710100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.925+0000 7f171789e640 1 --2- 172.21.15.107:0/1208395572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1710102a90 0x7f17101006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:13.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.925+0000 7f171789e640 1 -- 172.21.15.107:0/1208395572 >> 172.21.15.107:0/1208395572 conn(0x7f1710076ab0 msgr2=0x7f17100fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:13.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.925+0000 7f171789e640 1 -- 172.21.15.107:0/1208395572 shutdown_connections 2024-05-09T08:37:13.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:13.925+0000 7f171789e640 1 -- 172.21.15.107:0/1208395572 wait complete. 2024-05-09T08:37:13.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:37:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.042+0000 7f13c113b640 1 -- 172.21.15.107:0/246025253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13bc105700 msgr2=0x7f13bc105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.042+0000 7f13c113b640 1 --2- 172.21.15.107:0/246025253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13bc105700 0x7f13bc105ae0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f13ac009900 tx=0x7f13ac02a410 comp rx=0 tx=0).stop 2024-05-09T08:37:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.042+0000 7f13c113b640 1 -- 172.21.15.107:0/246025253 shutdown_connections 2024-05-09T08:37:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.042+0000 7f13c113b640 1 --2- 172.21.15.107:0/246025253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13bc1013d0 0x7f13bc10d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.042+0000 7f13c113b640 1 --2- 172.21.15.107:0/246025253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13bc100a10 0x7f13bc100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.042+0000 7f13c113b640 1 --2- 172.21.15.107:0/246025253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13bc105700 0x7f13bc105ae0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.042+0000 7f13c113b640 1 -- 172.21.15.107:0/246025253 >> 172.21.15.107:0/246025253 conn(0x7f13bc077510 msgr2=0x7f13bc077920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.042+0000 7f13c113b640 1 -- 172.21.15.107:0/246025253 shutdown_connections 2024-05-09T08:37:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.042+0000 7f13c113b640 1 -- 172.21.15.107:0/246025253 wait complete. 2024-05-09T08:37:14.043 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.043+0000 7f13c113b640 1 Processor -- start 2024-05-09T08:37:14.043 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.043+0000 7f13c113b640 1 -- start start 2024-05-09T08:37:14.043 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.043+0000 7f13c113b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13bc100a10 0x7f13bc196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.043 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.043+0000 7f13c113b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13bc1013d0 0x7f13bc1972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.043+0000 7f13c113b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13bc105700 0x7f13bc19b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.043+0000 7f13c113b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc110960 con 0x7f13bc100a10 2024-05-09T08:37:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.043+0000 7f13c113b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc1107e0 con 0x7f13bc105700 2024-05-09T08:37:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.043+0000 7f13c113b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc110ae0 con 0x7f13bc1013d0 2024-05-09T08:37:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.043+0000 7f13bad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13bc100a10 0x7f13bc196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.043+0000 7f13bad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13bc100a10 0x7f13bc196d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50214/0 (socket says 172.21.15.107:50214) 2024-05-09T08:37:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.043+0000 7f13bad9d640 1 -- 172.21.15.107:0/1729957519 learned_addr learned my addr 172.21.15.107:0/1729957519 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.043+0000 7f13ba59c640 1 --2- 172.21.15.107:0/1729957519 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13bc1013d0 0x7f13bc1972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.043+0000 7f13bb59e640 1 --2- 172.21.15.107:0/1729957519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13bc105700 0x7f13bc19b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.044+0000 7f13bad9d640 1 -- 172.21.15.107:0/1729957519 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13bc1013d0 msgr2=0x7f13bc1972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.044+0000 7f13bad9d640 1 --2- 172.21.15.107:0/1729957519 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13bc1013d0 0x7f13bc1972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.044+0000 7f13bad9d640 1 -- 172.21.15.107:0/1729957519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13bc105700 msgr2=0x7f13bc19b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.044+0000 7f13bad9d640 1 --2- 172.21.15.107:0/1729957519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13bc105700 0x7f13bc19b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.044+0000 7f13bad9d640 1 -- 172.21.15.107:0/1729957519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13bc19bd20 con 0x7f13bc100a10 2024-05-09T08:37:14.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.044+0000 7f13bad9d640 1 --2- 172.21.15.107:0/1729957519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13bc100a10 0x7f13bc196d80 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f13ac005940 tx=0x7f13ac004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.044+0000 7f13a3fff640 1 -- 172.21.15.107:0/1729957519 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac047070 con 0x7f13bc100a10 2024-05-09T08:37:14.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.044+0000 7f13c113b640 1 -- 172.21.15.107:0/1729957519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13bc19bfb0 con 0x7f13bc100a10 2024-05-09T08:37:14.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.044+0000 7f13a3fff640 1 -- 172.21.15.107:0/1729957519 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13ac003ca0 con 0x7f13bc100a10 2024-05-09T08:37:14.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.044+0000 7f13a3fff640 1 -- 172.21.15.107:0/1729957519 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac035070 con 0x7f13bc100a10 2024-05-09T08:37:14.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.044+0000 7f13c113b640 1 -- 172.21.15.107:0/1729957519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13bc06a080 con 0x7f13bc100a10 2024-05-09T08:37:14.048 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.047+0000 7f13c3e46740 1 Processor -- start 2024-05-09T08:37:14.048 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.047+0000 7f13c3e46740 1 -- start start 2024-05-09T08:37:14.048 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.047+0000 7f13a3fff640 1 -- 172.21.15.107:0/1729957519 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f13ac04a070 con 0x7f13bc100a10 2024-05-09T08:37:14.048 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.047+0000 7f13c3e46740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849450 0x5599d9849830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.047+0000 7f13c3e46740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9849d70 0x5599d98540f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.047+0000 7f13c3e46740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9854630 0x5599d9856a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.047+0000 7f13c3e46740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5599d97ae340 con 0x5599d9849d70 2024-05-09T08:37:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.047+0000 7f13c3e46740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5599d9756290 con 0x5599d9854630 2024-05-09T08:37:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.047+0000 7f13c3e46740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5599d97aa300 con 0x5599d9849450 2024-05-09T08:37:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.047+0000 7f13bad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9849d70 0x5599d98540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.048+0000 7f13bad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9849d70 0x5599d98540f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50224/0 (socket says 172.21.15.107:50224) 2024-05-09T08:37:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.048+0000 7f13bad9d640 1 -- 172.21.15.107:0/1063358752 learned_addr learned my addr 172.21.15.107:0/1063358752 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.048+0000 7f13bb59e640 1 --2- 172.21.15.107:0/1063358752 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849450 0x5599d9849830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.048+0000 7f13ba59c640 1 --2- 172.21.15.107:0/1063358752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9854630 0x5599d9856a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.048+0000 7f13bad9d640 1 -- 172.21.15.107:0/1063358752 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849450 msgr2=0x5599d9849830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.048+0000 7f13bad9d640 1 --2- 172.21.15.107:0/1063358752 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849450 0x5599d9849830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.048+0000 7f13bad9d640 1 -- 172.21.15.107:0/1063358752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9854630 msgr2=0x5599d9856a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.048+0000 7f13bad9d640 1 --2- 172.21.15.107:0/1063358752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9854630 0x5599d9856a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.048+0000 7f13bad9d640 1 -- 172.21.15.107:0/1063358752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599d977a8e0 con 0x5599d9849d70 2024-05-09T08:37:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.048+0000 7f13bad9d640 1 --2- 172.21.15.107:0/1063358752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9849d70 0x5599d98540f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0ee8e0 tx=0x7f13ac0f7cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83e885441d9701b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.048+0000 7f13a3fff640 1 --2- 172.21.15.107:0/1729957519 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13a40a2da0 0x7f13a40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.048+0000 7f13bb59e640 1 --2- 172.21.15.107:0/1729957519 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13a40a2da0 0x7f13a40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.048+0000 7f13a17fa640 1 -- 172.21.15.107:0/1063358752 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac0b9a90 con 0x5599d9849d70 2024-05-09T08:37:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.048+0000 7f13a3fff640 1 -- 172.21.15.107:0/1729957519 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f13ac0f1580 con 0x7f13bc100a10 2024-05-09T08:37:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.048+0000 7f13a17fa640 1 -- 172.21.15.107:0/1063358752 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13ac101020 con 0x5599d9849d70 2024-05-09T08:37:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.049+0000 7f13a17fa640 1 -- 172.21.15.107:0/1063358752 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac0bc4f0 con 0x5599d9849d70 2024-05-09T08:37:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.049+0000 7f13c3e46740 1 -- 172.21.15.107:0/1063358752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9849d70 msgr2=0x5599d98540f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.049+0000 7f13bb59e640 1 --2- 172.21.15.107:0/1729957519 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13a40a2da0 0x7f13a40a5260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f13b4007f00 tx=0x7f13b4007520 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.049+0000 7f13c3e46740 1 --2- 172.21.15.107:0/1063358752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9849d70 0x5599d98540f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0ee8e0 tx=0x7f13ac0f7cd0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.049+0000 7f13c3e46740 1 -- 172.21.15.107:0/1063358752 shutdown_connections 2024-05-09T08:37:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.049+0000 7f13c3e46740 1 --2- 172.21.15.107:0/1063358752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9854630 0x5599d9856a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.049+0000 7f13c3e46740 1 --2- 172.21.15.107:0/1063358752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9849d70 0x5599d98540f0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.049+0000 7f13c3e46740 1 --2- 172.21.15.107:0/1063358752 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849450 0x5599d9849830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.049+0000 7f13c3e46740 1 -- 172.21.15.107:0/1063358752 >> 172.21.15.107:0/1063358752 conn(0x5599d9845960 msgr2=0x5599d9845d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.049+0000 7f13c3e46740 1 -- 172.21.15.107:0/1063358752 shutdown_connections 2024-05-09T08:37:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.049+0000 7f13c3e46740 1 -- 172.21.15.107:0/1063358752 wait complete. 2024-05-09T08:37:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.050+0000 7f13c3e46740 1 -- start start 2024-05-09T08:37:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.050+0000 7f13c3e46740 1 Processor -- start 2024-05-09T08:37:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.050+0000 7f13c3e46740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9849450 0x5599d98514f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.050+0000 7f13c3e46740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849d70 0x5599d9851a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.050+0000 7f13c3e46740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9854630 0x5599d9851f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.050+0000 7f13c3e46740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5599d97ae340 con 0x5599d9849450 2024-05-09T08:37:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.050+0000 7f13c3e46740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5599d9769790 con 0x5599d9849d70 2024-05-09T08:37:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.050+0000 7f13c3e46740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5599d9857020 con 0x5599d9854630 2024-05-09T08:37:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.050+0000 7f13bad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9849450 0x5599d98514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.050+0000 7f13c3e46740 10 client.0 _ll_register_callbacks cb 0x5599d97a4a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.050+0000 7f13bad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9849450 0x5599d98514f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50230/0 (socket says 172.21.15.107:50230) 2024-05-09T08:37:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.050+0000 7f13bad9d640 1 -- 172.21.15.107:0/4085926499 learned_addr learned my addr 172.21.15.107:0/4085926499 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.050+0000 7f13ba59c640 1 --2- 172.21.15.107:0/4085926499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849d70 0x5599d9851a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.050+0000 7f13bb59e640 1 --2- 172.21.15.107:0/4085926499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9854630 0x5599d9851f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.051+0000 7f13bad9d640 1 -- 172.21.15.107:0/4085926499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9854630 msgr2=0x5599d9851f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.051+0000 7f13bad9d640 1 --2- 172.21.15.107:0/4085926499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9854630 0x5599d9851f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.051+0000 7f13bad9d640 1 -- 172.21.15.107:0/4085926499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849d70 msgr2=0x5599d9851a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.051+0000 7f13bad9d640 1 --2- 172.21.15.107:0/4085926499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849d70 0x5599d9851a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.051+0000 7f13bad9d640 1 -- 172.21.15.107:0/4085926499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599d94c2210 con 0x5599d9849450 2024-05-09T08:37:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.051+0000 7f13ba59c640 1 --2- 172.21.15.107:0/4085926499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849d70 0x5599d9851a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.051+0000 7f13bad9d640 1 --2- 172.21.15.107:0/4085926499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9849450 0x5599d98514f0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0bd780 tx=0x7f13ac0f18f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0149625eecca162 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.051+0000 7f139eff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.054+0000 7f139eff5640 1 -- 172.21.15.107:0/4085926499 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac100060 con 0x5599d9849450 2024-05-09T08:37:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.054+0000 7f139eff5640 1 -- 172.21.15.107:0/4085926499 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f13ac0bbaf0 con 0x5599d9849450 2024-05-09T08:37:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.054+0000 7f13c3e46740 1 -- 172.21.15.107:0/4085926499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13ac101020 con 0x5599d9849450 2024-05-09T08:37:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.055+0000 7f139eff5640 1 -- 172.21.15.107:0/4085926499 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f13ac0bd150 con 0x5599d9849450 2024-05-09T08:37:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.055+0000 7f13c3e46740 10 client.16767 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.055+0000 7f13c3e46740 1 -- 172.21.15.107:0/4085926499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5599d958e620 con 0x5599d9849450 2024-05-09T08:37:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.055+0000 7f139eff5640 1 -- 172.21.15.107:0/4085926499 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13ac101020 con 0x5599d9849450 2024-05-09T08:37:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.055+0000 7f139eff5640 1 -- 172.21.15.107:0/4085926499 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f13ac041030 con 0x5599d9849450 2024-05-09T08:37:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.055+0000 7f13c3e46740 10 client.16767 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.055+0000 7f13c3e46740 10 client.16767 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.055+0000 7f13c3e46740 20 client.16767 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.055+0000 7f13c3e46740 1 --2- 172.21.15.107:0/4085926499 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5599d9863f30 0x5599d9866380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.055+0000 7f13c3e46740 4 client.16767 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.055+0000 7f13c3e46740 1 -- 172.21.15.107:0/4085926499 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599d97bba70 con 0x5599d9863f30 2024-05-09T08:37:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.055+0000 7f13ba59c640 1 --2- 172.21.15.107:0/4085926499 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5599d9863f30 0x5599d9866380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.056+0000 7f13ba59c640 1 --2- 172.21.15.107:0/4085926499 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5599d9863f30 0x5599d9866380 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5a91a29eb47c542 server_cookie=2b22d1dc4974ed23 in_seq=0 out_seq=0 2024-05-09T08:37:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.056+0000 7f139eff5640 10 client.16767 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.056+0000 7f139eff5640 1 -- 172.21.15.107:0/4085926499 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5599d97bba70 con 0x5599d9863f30 2024-05-09T08:37:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.056+0000 7f139eff5640 10 client.16767 handle_command_reply: tid=0 2024-05-09T08:37:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f13c3e46740 1 client.16767 shutdown 2024-05-09T08:37:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f13c3e46740 1 -- 172.21.15.107:0/4085926499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9849450 msgr2=0x5599d98514f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f13c3e46740 1 --2- 172.21.15.107:0/4085926499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9849450 0x5599d98514f0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0bd780 tx=0x7f13ac0f18f0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f13c3e46740 1 -- 172.21.15.107:0/4085926499 shutdown_connections 2024-05-09T08:37:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f13c3e46740 1 --2- 172.21.15.107:0/4085926499 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5599d9863f30 0x5599d9866380 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f13c3e46740 1 --2- 172.21.15.107:0/4085926499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9854630 0x5599d9851f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f13c3e46740 1 --2- 172.21.15.107:0/4085926499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849d70 0x5599d9851a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f139eff5640 0 client.16767 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f13c3e46740 1 --2- 172.21.15.107:0/4085926499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9849450 0x5599d98514f0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f13c3e46740 1 -- 172.21.15.107:0/4085926499 >> 172.21.15.107:0/4085926499 conn(0x5599d9845960 msgr2=0x5599d984b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f13c3e46740 1 -- 172.21.15.107:0/4085926499 shutdown_connections 2024-05-09T08:37:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f13c3e46740 1 -- 172.21.15.107:0/4085926499 wait complete. 2024-05-09T08:37:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f13c3e46740 20 client.16767 trim_cache size 0 max 16384 2024-05-09T08:37:14.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f8b6e07d640 1 -- 172.21.15.107:0/903423191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b68100960 msgr2=0x7f8b68106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f8b6e07d640 1 --2- 172.21.15.107:0/903423191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b68100960 0x7f8b68106490 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009900 tx=0x7f8b5c02a410 comp rx=0 tx=0).stop 2024-05-09T08:37:14.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f8b6e07d640 1 -- 172.21.15.107:0/903423191 shutdown_connections 2024-05-09T08:37:14.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f8b6e07d640 1 --2- 172.21.15.107:0/903423191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b68106bc0 0x7f8b6810e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f8b6e07d640 1 --2- 172.21.15.107:0/903423191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b68100960 0x7f8b68106490 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f8b6e07d640 1 --2- 172.21.15.107:0/903423191 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b680fffb0 0x7f8b68100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f8b6e07d640 1 -- 172.21.15.107:0/903423191 >> 172.21.15.107:0/903423191 conn(0x7f8b68076cd0 msgr2=0x7f8b680fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f8b6e07d640 1 -- 172.21.15.107:0/903423191 shutdown_connections 2024-05-09T08:37:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.057+0000 7f8b6e07d640 1 -- 172.21.15.107:0/903423191 wait complete. 2024-05-09T08:37:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.058+0000 7f8b6e07d640 1 Processor -- start 2024-05-09T08:37:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.058+0000 7f8b6e07d640 1 -- start start 2024-05-09T08:37:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.058+0000 7f8b6e07d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b680fffb0 0x7f8b6819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.058+0000 7f8b6e07d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b68106bc0 0x7f8b6819a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b6e07d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b6819eb90 0x7f8b6819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b6e07d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8b68111ad0 con 0x7f8b6819eb90 2024-05-09T08:37:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b6e07d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8b68111950 con 0x7f8b68106bc0 2024-05-09T08:37:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b6e07d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8b68111c50 con 0x7f8b680fffb0 2024-05-09T08:37:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b67fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b6819eb90 0x7f8b6819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b67fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b6819eb90 0x7f8b6819f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50242/0 (socket says 172.21.15.107:50242) 2024-05-09T08:37:14.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b67fff640 1 -- 172.21.15.107:0/1675870708 learned_addr learned my addr 172.21.15.107:0/1675870708 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b677fe640 1 --2- 172.21.15.107:0/1675870708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b680fffb0 0x7f8b6819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b66ffd640 1 --2- 172.21.15.107:0/1675870708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b68106bc0 0x7f8b6819a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b67fff640 1 -- 172.21.15.107:0/1675870708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b680fffb0 msgr2=0x7f8b6819a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b67fff640 1 --2- 172.21.15.107:0/1675870708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b680fffb0 0x7f8b6819a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b67fff640 1 -- 172.21.15.107:0/1675870708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b68106bc0 msgr2=0x7f8b6819a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b67fff640 1 --2- 172.21.15.107:0/1675870708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b68106bc0 0x7f8b6819a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b67fff640 1 -- 172.21.15.107:0/1675870708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b6819f680 con 0x7f8b6819eb90 2024-05-09T08:37:14.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b67fff640 1 --2- 172.21.15.107:0/1675870708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b6819eb90 0x7f8b6819f040 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f8b6000efd0 tx=0x7f8b6000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b64ff9640 1 -- 172.21.15.107:0/1675870708 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60003b70 con 0x7f8b6819eb90 2024-05-09T08:37:14.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.059+0000 7f8b64ff9640 1 -- 172.21.15.107:0/1675870708 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b60003d10 con 0x7f8b6819eb90 2024-05-09T08:37:14.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.060+0000 7f8b6e07d640 1 -- 172.21.15.107:0/1675870708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b6819f970 con 0x7f8b6819eb90 2024-05-09T08:37:14.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.060+0000 7f8b64ff9640 1 -- 172.21.15.107:0/1675870708 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60010980 con 0x7f8b6819eb90 2024-05-09T08:37:14.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.060+0000 7f8b6e07d640 1 -- 172.21.15.107:0/1675870708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b681a70e0 con 0x7f8b6819eb90 2024-05-09T08:37:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.062+0000 7f8b64ff9640 1 -- 172.21.15.107:0/1675870708 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8b60004020 con 0x7f8b6819eb90 2024-05-09T08:37:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.063+0000 7f8b64ff9640 1 --2- 172.21.15.107:0/1675870708 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8b500a2d00 0x7f8b500a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.063+0000 7f8b677fe640 1 --2- 172.21.15.107:0/1675870708 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8b500a2d00 0x7f8b500a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.063+0000 7f8b64ff9640 1 -- 172.21.15.107:0/1675870708 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8b600cfb50 con 0x7f8b6819eb90 2024-05-09T08:37:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.063+0000 7f8b70d8d740 1 Processor -- start 2024-05-09T08:37:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.063+0000 7f8b70d8d740 1 -- start start 2024-05-09T08:37:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b70d8d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a1b330 0x560379a1b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b70d8d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a25de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b70d8d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a26320 0x560379a28710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b70d8d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560379980c40 con 0x560379a1bc50 2024-05-09T08:37:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b70d8d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560379928580 con 0x560379a1b330 2024-05-09T08:37:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b70d8d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56037997cc00 con 0x560379a26320 2024-05-09T08:37:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b677fe640 1 --2- 172.21.15.107:0/1675870708 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8b500a2d00 0x7f8b500a51c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8b580095f0 tx=0x7f8b58005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b66ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a25de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b66ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a25de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50256/0 (socket says 172.21.15.107:50256) 2024-05-09T08:37:14.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b66ffd640 1 -- 172.21.15.107:0/1708127205 learned_addr learned my addr 172.21.15.107:0/1708127205 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b677fe640 1 --2- 172.21.15.107:0/1708127205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a1b330 0x560379a1b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b67fff640 1 --2- 172.21.15.107:0/1708127205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a26320 0x560379a28710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b66ffd640 1 -- 172.21.15.107:0/1708127205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a26320 msgr2=0x560379a28710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b66ffd640 1 --2- 172.21.15.107:0/1708127205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a26320 0x560379a28710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b66ffd640 1 -- 172.21.15.107:0/1708127205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a1b330 msgr2=0x560379a1b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b66ffd640 1 --2- 172.21.15.107:0/1708127205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a1b330 0x560379a1b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b66ffd640 1 -- 172.21.15.107:0/1708127205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603796943f0 con 0x560379a1bc50 2024-05-09T08:37:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b67fff640 1 --2- 172.21.15.107:0/1708127205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a26320 0x560379a28710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.064+0000 7f8b66ffd640 1 --2- 172.21.15.107:0/1708127205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a25de0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009900 tx=0x7f8b5c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc2991e24a07f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.065+0000 7f8b4dffb640 1 -- 172.21.15.107:0/1708127205 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c040070 con 0x560379a1bc50 2024-05-09T08:37:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.065+0000 7f8b4dffb640 1 -- 172.21.15.107:0/1708127205 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b5c032e00 con 0x560379a1bc50 2024-05-09T08:37:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.065+0000 7f8b4dffb640 1 -- 172.21.15.107:0/1708127205 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c035040 con 0x560379a1bc50 2024-05-09T08:37:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.065+0000 7f8b70d8d740 1 -- 172.21.15.107:0/1708127205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 msgr2=0x560379a25de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.065+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/1708127205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a25de0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009900 tx=0x7f8b5c000bc0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.066+0000 7f8b70d8d740 1 -- 172.21.15.107:0/1708127205 shutdown_connections 2024-05-09T08:37:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.066+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/1708127205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a26320 0x560379a28710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.066+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/1708127205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a25de0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.066+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/1708127205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a1b330 0x560379a1b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.066+0000 7f8b70d8d740 1 -- 172.21.15.107:0/1708127205 >> 172.21.15.107:0/1708127205 conn(0x560379a17650 msgr2=0x560379a17a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.066+0000 7f8b70d8d740 1 -- 172.21.15.107:0/1708127205 shutdown_connections 2024-05-09T08:37:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.066+0000 7f8b70d8d740 1 -- 172.21.15.107:0/1708127205 wait complete. 2024-05-09T08:37:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.066+0000 7f8b70d8d740 1 -- start start 2024-05-09T08:37:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.066+0000 7f8b70d8d740 1 Processor -- start 2024-05-09T08:37:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b70d8d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a1b330 0x560379a232d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b70d8d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a23810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b70d8d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a26320 0x560379a23d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b70d8d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560379a290b0 con 0x560379a1bc50 2024-05-09T08:37:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b70d8d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56037993bc40 con 0x560379a26320 2024-05-09T08:37:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b70d8d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560379a29230 con 0x560379a1b330 2024-05-09T08:37:14.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b70d8d740 10 client.0 _ll_register_callbacks cb 0x560379693be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:14.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b66ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a23810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b66ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a23810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50260/0 (socket says 172.21.15.107:50260) 2024-05-09T08:37:14.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b66ffd640 1 -- 172.21.15.107:0/1198549995 learned_addr learned my addr 172.21.15.107:0/1198549995 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b677fe640 1 --2- 172.21.15.107:0/1198549995 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a1b330 0x560379a232d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b67fff640 1 --2- 172.21.15.107:0/1198549995 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a26320 0x560379a23d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b66ffd640 1 -- 172.21.15.107:0/1198549995 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a1b330 msgr2=0x560379a232d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b66ffd640 1 --2- 172.21.15.107:0/1198549995 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a1b330 0x560379a232d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b66ffd640 1 -- 172.21.15.107:0/1198549995 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a26320 msgr2=0x560379a23d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b66ffd640 1 --2- 172.21.15.107:0/1198549995 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a26320 0x560379a23d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b66ffd640 1 -- 172.21.15.107:0/1198549995 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56037975e7d0 con 0x560379a1bc50 2024-05-09T08:37:14.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b67fff640 1 --2- 172.21.15.107:0/1198549995 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a26320 0x560379a23d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:14.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b66ffd640 1 --2- 172.21.15.107:0/1198549995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a23810 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c031780 tx=0x7f8b5c0317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54eb864591cbbe4b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:14.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b46ffd640 1 -- 172.21.15.107:0/1198549995 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c040070 con 0x560379a1bc50 2024-05-09T08:37:14.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b46ffd640 1 -- 172.21.15.107:0/1198549995 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8b5c0053d0 con 0x560379a1bc50 2024-05-09T08:37:14.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.067+0000 7f8b70d8d740 1 -- 172.21.15.107:0/1198549995 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b5c032e00 con 0x560379a1bc50 2024-05-09T08:37:14.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.068+0000 7f8b46ffd640 1 -- 172.21.15.107:0/1198549995 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8b5c056050 con 0x560379a1bc50 2024-05-09T08:37:14.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.068+0000 7f8b70d8d740 10 client.16779 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:14.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.068+0000 7f8b70d8d740 1 -- 172.21.15.107:0/1198549995 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560379a352d0 con 0x560379a1bc50 2024-05-09T08:37:14.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.068+0000 7f8b46ffd640 1 -- 172.21.15.107:0/1198549995 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b5c032e00 con 0x560379a1bc50 2024-05-09T08:37:14.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.068+0000 7f8b46ffd640 1 -- 172.21.15.107:0/1198549995 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8b5c03c440 con 0x560379a1bc50 2024-05-09T08:37:14.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.068+0000 7f8b70d8d740 10 client.16779 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:14.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.068+0000 7f8b70d8d740 10 client.16779 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:37:14.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.068+0000 7f8b70d8d740 20 client.16779 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:14.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.068+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/1198549995 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x560379a36620 0x560379a38a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.068+0000 7f8b70d8d740 4 client.16779 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:14.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.068+0000 7f8b70d8d740 1 -- 172.21.15.107:0/1198549995 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56037998e370 con 0x560379a36620 2024-05-09T08:37:14.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.068+0000 7f8b677fe640 1 --2- 172.21.15.107:0/1198549995 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x560379a36620 0x560379a38a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.069+0000 7f8b677fe640 1 --2- 172.21.15.107:0/1198549995 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x560379a36620 0x560379a38a10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c7b327e767385cc0 server_cookie=2abe51b8bed5b7f9 in_seq=0 out_seq=0 2024-05-09T08:37:14.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.069+0000 7f8b46ffd640 10 client.16779 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:37:14.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.070+0000 7f8b46ffd640 1 -- 172.21.15.107:0/1198549995 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56037998e370 con 0x560379a36620 2024-05-09T08:37:14.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.070+0000 7f8b46ffd640 10 client.16779 handle_command_reply: tid=0 2024-05-09T08:37:14.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.070+0000 7f8b70d8d740 1 client.16779 shutdown 2024-05-09T08:37:14.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.070+0000 7f8b70d8d740 1 -- 172.21.15.107:0/1198549995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 msgr2=0x560379a23810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.070+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/1198549995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a23810 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c031780 tx=0x7f8b5c0317b0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.070+0000 7f8b70d8d740 1 -- 172.21.15.107:0/1198549995 shutdown_connections 2024-05-09T08:37:14.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.070+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/1198549995 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x560379a36620 0x560379a38a10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.070+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/1198549995 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a26320 0x560379a23d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.070+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/1198549995 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a23810 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.070+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/1198549995 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a1b330 0x560379a232d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.070+0000 7f8b70d8d740 1 -- 172.21.15.107:0/1198549995 >> 172.21.15.107:0/1198549995 conn(0x560379a17650 msgr2=0x560379a1d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.070+0000 7f8b46ffd640 0 client.16779 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:37:14.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.070+0000 7f8b70d8d740 1 -- 172.21.15.107:0/1198549995 shutdown_connections 2024-05-09T08:37:14.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.070+0000 7f8b70d8d740 1 -- 172.21.15.107:0/1198549995 wait complete. 2024-05-09T08:37:14.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.070+0000 7f8b70d8d740 20 client.16779 trim_cache size 0 max 16384 2024-05-09T08:37:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.077+0000 7f13c3e46740 1 Processor -- start 2024-05-09T08:37:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.077+0000 7f13c3e46740 1 -- start start 2024-05-09T08:37:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.077+0000 7f13c3e46740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849450 0x5599d9853c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.077+0000 7f13c3e46740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849d70 0x5599d985fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.077+0000 7f13c3e46740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9854630 0x5599d986bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.077+0000 7f13c3e46740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc1af030 con 0x5599d9854630 2024-05-09T08:37:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.077+0000 7f13c3e46740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc1ac900 con 0x5599d9849450 2024-05-09T08:37:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.077+0000 7f13c3e46740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc1aa750 con 0x5599d9849d70 2024-05-09T08:37:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.077+0000 7f13bb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9854630 0x5599d986bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.077+0000 7f13bb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9854630 0x5599d986bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50276/0 (socket says 172.21.15.107:50276) 2024-05-09T08:37:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.077+0000 7f13bb59e640 1 -- 172.21.15.107:0/1449240611 learned_addr learned my addr 172.21.15.107:0/1449240611 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.077+0000 7f13bad9d640 1 --2- 172.21.15.107:0/1449240611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849450 0x5599d9853c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.077+0000 7f13ba59c640 1 --2- 172.21.15.107:0/1449240611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849d70 0x5599d985fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.078+0000 7f13bb59e640 1 -- 172.21.15.107:0/1449240611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849d70 msgr2=0x5599d985fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.078+0000 7f13bb59e640 1 --2- 172.21.15.107:0/1449240611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849d70 0x5599d985fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.078+0000 7f13bb59e640 1 -- 172.21.15.107:0/1449240611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849450 msgr2=0x5599d9853c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.078+0000 7f13bb59e640 1 --2- 172.21.15.107:0/1449240611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849450 0x5599d9853c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.078+0000 7f13bb59e640 1 -- 172.21.15.107:0/1449240611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599d9849ba0 con 0x5599d9854630 2024-05-09T08:37:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.078+0000 7f13bad9d640 1 --2- 172.21.15.107:0/1449240611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849450 0x5599d9853c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.078+0000 7f13ba59c640 1 --2- 172.21.15.107:0/1449240611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849d70 0x5599d985fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.078+0000 7f13bb59e640 1 --2- 172.21.15.107:0/1449240611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9854630 0x5599d986bc60 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f13b401e790 tx=0x7f13b4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b540237636d756a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.078+0000 7f13a0ff9640 1 -- 172.21.15.107:0/1449240611 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b401e020 con 0x5599d9854630 2024-05-09T08:37:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.078+0000 7f13a0ff9640 1 -- 172.21.15.107:0/1449240611 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13b4003e20 con 0x5599d9854630 2024-05-09T08:37:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.078+0000 7f13a0ff9640 1 -- 172.21.15.107:0/1449240611 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b4004140 con 0x5599d9854630 2024-05-09T08:37:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 -- 172.21.15.107:0/1449240611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9854630 msgr2=0x5599d986bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 --2- 172.21.15.107:0/1449240611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9854630 0x5599d986bc60 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f13b401e790 tx=0x7f13b4015000 comp rx=0 tx=0).stop 2024-05-09T08:37:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 -- 172.21.15.107:0/1449240611 shutdown_connections 2024-05-09T08:37:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 --2- 172.21.15.107:0/1449240611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9854630 0x5599d986bc60 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.082 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 --2- 172.21.15.107:0/1449240611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849d70 0x5599d985fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 --2- 172.21.15.107:0/1449240611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849450 0x5599d9853c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 -- 172.21.15.107:0/1449240611 >> 172.21.15.107:0/1449240611 conn(0x5599d9845960 msgr2=0x5599d9845d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 -- 172.21.15.107:0/1449240611 shutdown_connections 2024-05-09T08:37:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 -- 172.21.15.107:0/1449240611 wait complete. 2024-05-09T08:37:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 -- start start 2024-05-09T08:37:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 Processor -- start 2024-05-09T08:37:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849450 0x5599d985c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849d70 0x5599d98550d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9854630 0x5599d9855610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc1aadc0 con 0x5599d9854630 2024-05-09T08:37:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc1ac230 con 0x5599d9849d70 2024-05-09T08:37:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.079+0000 7f13c3e46740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc1ab490 con 0x5599d9849450 2024-05-09T08:37:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f13c3e46740 10 client.0 _ll_register_callbacks cb 0x7f13b000b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f13bb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9854630 0x5599d9855610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f13bb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9854630 0x5599d9855610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50284/0 (socket says 172.21.15.107:50284) 2024-05-09T08:37:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f13bb59e640 1 -- 172.21.15.107:0/4273304829 learned_addr learned my addr 172.21.15.107:0/4273304829 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f13ba59c640 1 --2- 172.21.15.107:0/4273304829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849d70 0x5599d98550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f13bad9d640 1 --2- 172.21.15.107:0/4273304829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849450 0x5599d985c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f13bb59e640 1 -- 172.21.15.107:0/4273304829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849450 msgr2=0x5599d985c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f13bb59e640 1 --2- 172.21.15.107:0/4273304829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849450 0x5599d985c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f13bb59e640 1 -- 172.21.15.107:0/4273304829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849d70 msgr2=0x5599d98550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f13bb59e640 1 --2- 172.21.15.107:0/4273304829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849d70 0x5599d98550d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f13bb59e640 1 -- 172.21.15.107:0/4273304829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599d9855e10 con 0x5599d9854630 2024-05-09T08:37:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f13bad9d640 1 --2- 172.21.15.107:0/4273304829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849450 0x5599d985c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f13bb59e640 1 --2- 172.21.15.107:0/4273304829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9854630 0x5599d9855610 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f13b4017040 tx=0x7f13b4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ecadfe9d75dd02 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f139eff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f139eff5640 1 -- 172.21.15.107:0/4273304829 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b4036060 con 0x5599d9854630 2024-05-09T08:37:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f139eff5640 1 -- 172.21.15.107:0/4273304829 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f13b4024040 con 0x5599d9854630 2024-05-09T08:37:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.080+0000 7f13c3e46740 1 -- 172.21.15.107:0/4273304829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13b4003e20 con 0x5599d9854630 2024-05-09T08:37:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.081+0000 7f139eff5640 1 -- 172.21.15.107:0/4273304829 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f13b403d050 con 0x5599d9854630 2024-05-09T08:37:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.081+0000 7f13c3e46740 10 client.16785 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.081+0000 7f13c3e46740 1 -- 172.21.15.107:0/4273304829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5599d98ca4d0 con 0x5599d9854630 2024-05-09T08:37:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.081+0000 7f139eff5640 1 -- 172.21.15.107:0/4273304829 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13b4003e20 con 0x5599d9854630 2024-05-09T08:37:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.081+0000 7f139eff5640 1 -- 172.21.15.107:0/4273304829 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f13b4004460 con 0x5599d9854630 2024-05-09T08:37:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.081+0000 7f13c3e46740 10 client.16785 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.081+0000 7f13c3e46740 10 client.16785 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.081+0000 7f13c3e46740 20 client.16785 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.081+0000 7f13c3e46740 1 --2- 172.21.15.107:0/4273304829 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5599d98cab50 0x5599d98ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.081+0000 7f13c3e46740 4 client.16785 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:37:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.081+0000 7f13c3e46740 1 -- 172.21.15.107:0/4273304829 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5599d97b5dc0 con 0x5599d98cab50 2024-05-09T08:37:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.081+0000 7f13bad9d640 1 --2- 172.21.15.107:0/4273304829 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5599d98cab50 0x5599d98ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.082+0000 7f139eff5640 10 client.16785 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.082+0000 7f13bad9d640 1 --2- 172.21.15.107:0/4273304829 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5599d98cab50 0x5599d98ccf70 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74d8f71f4d2e4db5 server_cookie=3b1763c18ccf9a58 in_seq=0 out_seq=0 2024-05-09T08:37:14.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.088+0000 7f8b70d8d740 1 Processor -- start 2024-05-09T08:37:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.088+0000 7f8b70d8d740 1 -- start start 2024-05-09T08:37:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.089+0000 7f8b70d8d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a1b330 0x560379a31f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.089+0000 7f8b70d8d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a1bc50 0x560379a1dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.089+0000 7f8b70d8d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a26320 0x560379a25090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.089+0000 7f8b67fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a26320 0x560379a25090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.089+0000 7f8b67fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a26320 0x560379a25090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50286/0 (socket says 172.21.15.107:50286) 2024-05-09T08:37:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.089+0000 7f8b67fff640 1 -- 172.21.15.107:0/774820375 learned_addr learned my addr 172.21.15.107:0/774820375 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.089+0000 7f8b70d8d740 1 -- 172.21.15.107:0/774820375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8b681b3570 con 0x560379a26320 2024-05-09T08:37:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.089+0000 7f8b70d8d740 1 -- 172.21.15.107:0/774820375 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8b681b0e40 con 0x560379a1b330 2024-05-09T08:37:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.089+0000 7f8b70d8d740 1 -- 172.21.15.107:0/774820375 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8b681aec90 con 0x560379a1bc50 2024-05-09T08:37:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.089+0000 7f8b66ffd640 1 --2- 172.21.15.107:0/774820375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a1bc50 0x560379a1dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.089+0000 7f8b677fe640 1 --2- 172.21.15.107:0/774820375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a1b330 0x560379a31f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.089+0000 7f8b67fff640 1 -- 172.21.15.107:0/774820375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a1bc50 msgr2=0x560379a1dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.089+0000 7f8b67fff640 1 --2- 172.21.15.107:0/774820375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a1bc50 0x560379a1dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.089+0000 7f8b67fff640 1 -- 172.21.15.107:0/774820375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a1b330 msgr2=0x560379a31f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.089+0000 7f8b67fff640 1 --2- 172.21.15.107:0/774820375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a1b330 0x560379a31f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.089+0000 7f8b67fff640 1 -- 172.21.15.107:0/774820375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560379a28710 con 0x560379a26320 2024-05-09T08:37:14.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.090+0000 7f8b67fff640 1 --2- 172.21.15.107:0/774820375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a26320 0x560379a25090 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f8b6009bca0 tx=0x7f8b6009b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f4de05f66552276 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.090+0000 7f8b4d7fa640 1 -- 172.21.15.107:0/774820375 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b600b8020 con 0x560379a26320 2024-05-09T08:37:14.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.090+0000 7f8b4d7fa640 1 -- 172.21.15.107:0/774820375 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b6009c410 con 0x560379a26320 2024-05-09T08:37:14.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.090+0000 7f8b4d7fa640 1 -- 172.21.15.107:0/774820375 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60098380 con 0x560379a26320 2024-05-09T08:37:14.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.091+0000 7f8b70d8d740 1 -- 172.21.15.107:0/774820375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a26320 msgr2=0x560379a25090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.091+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/774820375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a26320 0x560379a25090 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f8b6009bca0 tx=0x7f8b6009b780 comp rx=0 tx=0).stop 2024-05-09T08:37:14.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.092+0000 7f8b70d8d740 1 -- 172.21.15.107:0/774820375 shutdown_connections 2024-05-09T08:37:14.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.092+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/774820375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a26320 0x560379a25090 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.092+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/774820375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a1bc50 0x560379a1dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.092+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/774820375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a1b330 0x560379a31f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.092+0000 7f8b70d8d740 1 -- 172.21.15.107:0/774820375 >> 172.21.15.107:0/774820375 conn(0x560379a17650 msgr2=0x560379a24980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.092+0000 7f8b70d8d740 1 -- 172.21.15.107:0/774820375 shutdown_connections 2024-05-09T08:37:14.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.092+0000 7f8b70d8d740 1 -- 172.21.15.107:0/774820375 wait complete. 2024-05-09T08:37:14.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.092+0000 7f8b70d8d740 1 -- start start 2024-05-09T08:37:14.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.093+0000 7f8b70d8d740 1 Processor -- start 2024-05-09T08:37:14.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.093+0000 7f8b70d8d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a1b330 0x560379a2e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.093+0000 7f8b70d8d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a1fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.093+0000 7f8b66ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a1fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.093+0000 7f8b70d8d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a26320 0x560379a20180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.093+0000 7f8b66ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a1fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50300/0 (socket says 172.21.15.107:50300) 2024-05-09T08:37:14.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.093+0000 7f8b66ffd640 1 -- 172.21.15.107:0/3465773425 learned_addr learned my addr 172.21.15.107:0/3465773425 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.093+0000 7f8b70d8d740 1 -- 172.21.15.107:0/3465773425 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8b681af300 con 0x560379a1bc50 2024-05-09T08:37:14.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.093+0000 7f8b70d8d740 1 -- 172.21.15.107:0/3465773425 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8b681b0770 con 0x560379a26320 2024-05-09T08:37:14.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.093+0000 7f8b70d8d740 1 -- 172.21.15.107:0/3465773425 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8b681af9d0 con 0x560379a1b330 2024-05-09T08:37:14.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.093+0000 7f8b677fe640 1 --2- 172.21.15.107:0/3465773425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a1b330 0x560379a2e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.093+0000 7f8b70d8d740 10 client.0 _ll_register_callbacks cb 0x7f8b58009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:14.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.094+0000 7f8b67fff640 1 --2- 172.21.15.107:0/3465773425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a26320 0x560379a20180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.094+0000 7f8b66ffd640 1 -- 172.21.15.107:0/3465773425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a1b330 msgr2=0x560379a2e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.094+0000 7f8b66ffd640 1 --2- 172.21.15.107:0/3465773425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a1b330 0x560379a2e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.094+0000 7f8b66ffd640 1 -- 172.21.15.107:0/3465773425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a26320 msgr2=0x560379a20180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.094+0000 7f8b66ffd640 1 --2- 172.21.15.107:0/3465773425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a26320 0x560379a20180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.094+0000 7f8b66ffd640 1 -- 172.21.15.107:0/3465773425 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560379a20950 con 0x560379a1bc50 2024-05-09T08:37:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.094+0000 7f8b67fff640 1 --2- 172.21.15.107:0/3465773425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a26320 0x560379a20180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.094+0000 7f8b66ffd640 1 --2- 172.21.15.107:0/3465773425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a1fc40 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c03ca00 tx=0x7f8b5c005150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd959e565d72a4e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.094+0000 7f8b46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.094+0000 7f8b46ffd640 1 -- 172.21.15.107:0/3465773425 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c040050 con 0x560379a1bc50 2024-05-09T08:37:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.094+0000 7f8b46ffd640 1 -- 172.21.15.107:0/3465773425 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8b5c0055d0 con 0x560379a1bc50 2024-05-09T08:37:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.094+0000 7f8b70d8d740 1 -- 172.21.15.107:0/3465773425 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b6009c410 con 0x560379a1bc50 2024-05-09T08:37:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.094+0000 7f8b46ffd640 1 -- 172.21.15.107:0/3465773425 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8b5c067050 con 0x560379a1bc50 2024-05-09T08:37:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.095+0000 7f8b70d8d740 10 client.16791 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.095+0000 7f8b70d8d740 1 -- 172.21.15.107:0/3465773425 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560379a9c840 con 0x560379a1bc50 2024-05-09T08:37:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.095+0000 7f8b46ffd640 1 -- 172.21.15.107:0/3465773425 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b6009c410 con 0x560379a1bc50 2024-05-09T08:37:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.095+0000 7f8b46ffd640 1 -- 172.21.15.107:0/3465773425 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8b5c004ad0 con 0x560379a1bc50 2024-05-09T08:37:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.095+0000 7f8b70d8d740 10 client.16791 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:14.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.095+0000 7f8b70d8d740 10 client.16791 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:37:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.095+0000 7f8b70d8d740 20 client.16791 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.095+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/3465773425 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x560379a9ce70 0x560379a9f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.095+0000 7f8b70d8d740 4 client.16791 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:37:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.095+0000 7f8b70d8d740 1 -- 172.21.15.107:0/3465773425 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560379a38e00 con 0x560379a9ce70 2024-05-09T08:37:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.095+0000 7f8b677fe640 1 --2- 172.21.15.107:0/3465773425 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x560379a9ce70 0x560379a9f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.096+0000 7f8b46ffd640 10 client.16791 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:37:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.096+0000 7f8b677fe640 1 --2- 172.21.15.107:0/3465773425 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x560379a9ce70 0x560379a9f290 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8d3b7836c1b6ffaa server_cookie=36227b584b16abe9 in_seq=0 out_seq=0 2024-05-09T08:37:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.096+0000 7f8b46ffd640 1 -- 172.21.15.107:0/3465773425 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560379a38e00 con 0x560379a9ce70 2024-05-09T08:37:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.096+0000 7f8b46ffd640 10 client.16791 handle_command_reply: tid=0 2024-05-09T08:37:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.096+0000 7f8b70d8d740 1 client.16791 shutdown 2024-05-09T08:37:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.096+0000 7f8b70d8d740 1 -- 172.21.15.107:0/3465773425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 msgr2=0x560379a1fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.096+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/3465773425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a1fc40 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c03ca00 tx=0x7f8b5c005150 comp rx=0 tx=0).stop 2024-05-09T08:37:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.096+0000 7f8b70d8d740 1 -- 172.21.15.107:0/3465773425 shutdown_connections 2024-05-09T08:37:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.096+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/3465773425 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x560379a9ce70 0x560379a9f290 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.096+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/3465773425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560379a26320 0x560379a20180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.096+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/3465773425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560379a1bc50 0x560379a1fc40 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.096+0000 7f8b70d8d740 1 --2- 172.21.15.107:0/3465773425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560379a1b330 0x560379a2e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.096+0000 7f8b70d8d740 1 -- 172.21.15.107:0/3465773425 >> 172.21.15.107:0/3465773425 conn(0x560379a17650 msgr2=0x560379a23b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.096+0000 7f8b46ffd640 0 client.16791 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:37:14.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.096+0000 7f8b70d8d740 1 -- 172.21.15.107:0/3465773425 shutdown_connections 2024-05-09T08:37:14.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.097+0000 7f8b70d8d740 1 -- 172.21.15.107:0/3465773425 wait complete. 2024-05-09T08:37:14.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.097+0000 7f8b70d8d740 20 client.16791 trim_cache size 0 max 16384 2024-05-09T08:37:14.099 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:37:14.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.097+0000 7f8b6e07d640 1 -- 172.21.15.107:0/1675870708 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8b500a2d00 msgr2=0x7f8b500a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.097+0000 7f8b6e07d640 1 --2- 172.21.15.107:0/1675870708 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8b500a2d00 0x7f8b500a51c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8b580095f0 tx=0x7f8b58005f90 comp rx=0 tx=0).stop 2024-05-09T08:37:14.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.097+0000 7f8b6e07d640 1 -- 172.21.15.107:0/1675870708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b6819eb90 msgr2=0x7f8b6819f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.097+0000 7f8b6e07d640 1 --2- 172.21.15.107:0/1675870708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b6819eb90 0x7f8b6819f040 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f8b6000efd0 tx=0x7f8b6000c6a0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.097+0000 7f8b6e07d640 1 -- 172.21.15.107:0/1675870708 shutdown_connections 2024-05-09T08:37:14.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.097+0000 7f8b6e07d640 1 --2- 172.21.15.107:0/1675870708 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8b500a2d00 0x7f8b500a51c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.097+0000 7f8b6e07d640 1 --2- 172.21.15.107:0/1675870708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b6819eb90 0x7f8b6819f040 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.097+0000 7f8b6e07d640 1 --2- 172.21.15.107:0/1675870708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b68106bc0 0x7f8b6819a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.097+0000 7f8b6e07d640 1 --2- 172.21.15.107:0/1675870708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b680fffb0 0x7f8b6819a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.097+0000 7f8b6e07d640 1 -- 172.21.15.107:0/1675870708 >> 172.21.15.107:0/1675870708 conn(0x7f8b68076cd0 msgr2=0x7f8b681050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.097+0000 7f8b6e07d640 1 -- 172.21.15.107:0/1675870708 shutdown_connections 2024-05-09T08:37:14.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.097+0000 7f8b6e07d640 1 -- 172.21.15.107:0/1675870708 wait complete. 2024-05-09T08:37:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.099+0000 7f139eff5640 1 -- 172.21.15.107:0/4273304829 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5599d97b5dc0 con 0x5599d98cab50 2024-05-09T08:37:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.099+0000 7f139eff5640 10 client.16785 handle_command_reply: tid=0 2024-05-09T08:37:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.099+0000 7f13c3e46740 1 client.16785 shutdown 2024-05-09T08:37:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.100+0000 7f13c3e46740 1 -- 172.21.15.107:0/4273304829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9854630 msgr2=0x5599d9855610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.100+0000 7f13c3e46740 1 --2- 172.21.15.107:0/4273304829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9854630 0x5599d9855610 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f13b4017040 tx=0x7f13b4003c20 comp rx=0 tx=0).stop 2024-05-09T08:37:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.100+0000 7f13c3e46740 1 -- 172.21.15.107:0/4273304829 shutdown_connections 2024-05-09T08:37:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.100+0000 7f13c3e46740 1 --2- 172.21.15.107:0/4273304829 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5599d98cab50 0x5599d98ccf70 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.100+0000 7f139eff5640 0 client.16785 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.100+0000 7f13c3e46740 1 --2- 172.21.15.107:0/4273304829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599d9854630 0x5599d9855610 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.100+0000 7f13c3e46740 1 --2- 172.21.15.107:0/4273304829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599d9849d70 0x5599d98550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.100+0000 7f13c3e46740 1 --2- 172.21.15.107:0/4273304829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599d9849450 0x5599d985c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.100+0000 7f13c3e46740 1 -- 172.21.15.107:0/4273304829 >> 172.21.15.107:0/4273304829 conn(0x5599d9845960 msgr2=0x5599d9851490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.100+0000 7f13c3e46740 1 -- 172.21.15.107:0/4273304829 shutdown_connections 2024-05-09T08:37:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.100+0000 7f13c3e46740 1 -- 172.21.15.107:0/4273304829 wait complete. 2024-05-09T08:37:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.100+0000 7f13c3e46740 20 client.16785 trim_cache size 0 max 16384 2024-05-09T08:37:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:37:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:37:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:37:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:37:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.101+0000 7f13c113b640 1 -- 172.21.15.107:0/1729957519 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13a40a2da0 msgr2=0x7f13a40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.101+0000 7f13c113b640 1 --2- 172.21.15.107:0/1729957519 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13a40a2da0 0x7f13a40a5260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f13b4007f00 tx=0x7f13b4007520 comp rx=0 tx=0).stop 2024-05-09T08:37:14.105 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.101+0000 7f13c113b640 1 -- 172.21.15.107:0/1729957519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13bc100a10 msgr2=0x7f13bc196d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.105 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.101+0000 7f13c113b640 1 --2- 172.21.15.107:0/1729957519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13bc100a10 0x7f13bc196d80 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f13ac005940 tx=0x7f13ac004270 comp rx=0 tx=0).stop 2024-05-09T08:37:14.105 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.101+0000 7f13c113b640 1 -- 172.21.15.107:0/1729957519 shutdown_connections 2024-05-09T08:37:14.105 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.101+0000 7f13c113b640 1 --2- 172.21.15.107:0/1729957519 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13a40a2da0 0x7f13a40a5260 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.105 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.101+0000 7f13c113b640 1 --2- 172.21.15.107:0/1729957519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13bc105700 0x7f13bc19b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.105 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.101+0000 7f13c113b640 1 --2- 172.21.15.107:0/1729957519 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13bc1013d0 0x7f13bc1972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.105 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.101+0000 7f13c113b640 1 --2- 172.21.15.107:0/1729957519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13bc100a10 0x7f13bc196d80 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.105 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.101+0000 7f13c113b640 1 -- 172.21.15.107:0/1729957519 >> 172.21.15.107:0/1729957519 conn(0x7f13bc077510 msgr2=0x7f13bc0fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.105 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.101+0000 7f13c113b640 1 -- 172.21.15.107:0/1729957519 shutdown_connections 2024-05-09T08:37:14.105 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:14.101+0000 7f13c113b640 1 -- 172.21.15.107:0/1729957519 wait complete. 2024-05-09T08:37:14.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:37:14.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.224+0000 7fd712734640 1 -- 172.21.15.107:0/2584800863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd70c106710 msgr2=0x7fd70c106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.224+0000 7fd712734640 1 --2- 172.21.15.107:0/2584800863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd70c106710 0x7fd70c106af0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009930 tx=0x7fd6fc02a420 comp rx=0 tx=0).stop 2024-05-09T08:37:14.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.224+0000 7fd712734640 1 -- 172.21.15.107:0/2584800863 shutdown_connections 2024-05-09T08:37:14.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.224+0000 7fd712734640 1 --2- 172.21.15.107:0/2584800863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd70c1010c0 0x7fd70c10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.224+0000 7fd712734640 1 --2- 172.21.15.107:0/2584800863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd70c100700 0x7fd70c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.224+0000 7fd712734640 1 --2- 172.21.15.107:0/2584800863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd70c106710 0x7fd70c106af0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.224+0000 7fd712734640 1 -- 172.21.15.107:0/2584800863 >> 172.21.15.107:0/2584800863 conn(0x7fd70c077830 msgr2=0x7fd70c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.224+0000 7fd712734640 1 -- 172.21.15.107:0/2584800863 shutdown_connections 2024-05-09T08:37:14.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.224+0000 7fd712734640 1 -- 172.21.15.107:0/2584800863 wait complete. 2024-05-09T08:37:14.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.225+0000 7fd712734640 1 Processor -- start 2024-05-09T08:37:14.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.225+0000 7fd712734640 1 -- start start 2024-05-09T08:37:14.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.225+0000 7fd712734640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd70c100700 0x7fd70c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.225+0000 7fd712734640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd70c1010c0 0x7fd70c1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.225+0000 7fd712734640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd70c106710 0x7fd70c19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.225+0000 7fd712734640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c1109d0 con 0x7fd70c106710 2024-05-09T08:37:14.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.225+0000 7fd712734640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c110850 con 0x7fd70c100700 2024-05-09T08:37:14.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.225+0000 7fd712734640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c110b50 con 0x7fd70c1010c0 2024-05-09T08:37:14.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.226+0000 7fd710cd1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd70c106710 0x7fd70c19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.226+0000 7fd710cd1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd70c106710 0x7fd70c19c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50316/0 (socket says 172.21.15.107:50316) 2024-05-09T08:37:14.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.226+0000 7fd710cd1640 1 -- 172.21.15.107:0/3356592340 learned_addr learned my addr 172.21.15.107:0/3356592340 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.226+0000 7fd70b7fe640 1 --2- 172.21.15.107:0/3356592340 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd70c1010c0 0x7fd70c1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.226+0000 7fd70bfff640 1 --2- 172.21.15.107:0/3356592340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd70c100700 0x7fd70c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.226+0000 7fd710cd1640 1 -- 172.21.15.107:0/3356592340 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd70c1010c0 msgr2=0x7fd70c1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.226+0000 7fd710cd1640 1 --2- 172.21.15.107:0/3356592340 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd70c1010c0 0x7fd70c1985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.226+0000 7fd710cd1640 1 -- 172.21.15.107:0/3356592340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd70c100700 msgr2=0x7fd70c198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.226+0000 7fd710cd1640 1 --2- 172.21.15.107:0/3356592340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd70c100700 0x7fd70c198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.226+0000 7fd710cd1640 1 -- 172.21.15.107:0/3356592340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd70c19d020 con 0x7fd70c106710 2024-05-09T08:37:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.226+0000 7fd710cd1640 1 --2- 172.21.15.107:0/3356592340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd70c106710 0x7fd70c19c8f0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd70400eb10 tx=0x7fd70400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.226+0000 7fd7097fa640 1 -- 172.21.15.107:0/3356592340 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd704004b20 con 0x7fd70c106710 2024-05-09T08:37:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.226+0000 7fd712734640 1 -- 172.21.15.107:0/3356592340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd70c19d310 con 0x7fd70c106710 2024-05-09T08:37:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.226+0000 7fd7097fa640 1 -- 172.21.15.107:0/3356592340 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd704004cc0 con 0x7fd70c106710 2024-05-09T08:37:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.226+0000 7fd7097fa640 1 -- 172.21.15.107:0/3356592340 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd704010690 con 0x7fd70c106710 2024-05-09T08:37:14.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.226+0000 7fd712734640 1 -- 172.21.15.107:0/3356592340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd70c1a4ba0 con 0x7fd70c106710 2024-05-09T08:37:14.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.229+0000 7fd7097fa640 1 -- 172.21.15.107:0/3356592340 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd704005020 con 0x7fd70c106710 2024-05-09T08:37:14.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.229+0000 7fd715441740 1 Processor -- start 2024-05-09T08:37:14.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.229+0000 7fd715441740 1 -- start start 2024-05-09T08:37:14.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.229+0000 7fd715441740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9e330 0x55c193c9e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.229+0000 7fd715441740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9ec50 0x55c193ca8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.229+0000 7fd715441740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 0x55c193cab710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.229+0000 7fd7097fa640 1 --2- 172.21.15.107:0/3356592340 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd6f40a2d00 0x7fd6f40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.229+0000 7fd715441740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c193c03c40 con 0x55c193c9e330 2024-05-09T08:37:14.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.229+0000 7fd715441740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c193bab580 con 0x55c193ca9320 2024-05-09T08:37:14.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.229+0000 7fd715441740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c193bffc00 con 0x55c193c9ec50 2024-05-09T08:37:14.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd70b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9e330 0x55c193c9e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd70b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9e330 0x55c193c9e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50318/0 (socket says 172.21.15.107:50318) 2024-05-09T08:37:14.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd70b7fe640 1 -- 172.21.15.107:0/496292944 learned_addr learned my addr 172.21.15.107:0/496292944 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd70bfff640 1 --2- 172.21.15.107:0/496292944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9ec50 0x55c193ca8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd7097fa640 1 -- 172.21.15.107:0/3356592340 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd7040cfeb0 con 0x7fd70c106710 2024-05-09T08:37:14.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd70b7fe640 1 --2- 172.21.15.107:0/3356592340 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd6f40a2d00 0x7fd6f40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd710cd1640 1 --2- 172.21.15.107:0/496292944 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 0x55c193cab710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd70b7fe640 1 -- 172.21.15.107:0/496292944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9ec50 msgr2=0x55c193ca8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd70b7fe640 1 --2- 172.21.15.107:0/496292944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9ec50 0x55c193ca8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd70b7fe640 1 -- 172.21.15.107:0/496292944 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 msgr2=0x55c193cab710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd70b7fe640 1 --2- 172.21.15.107:0/496292944 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 0x55c193cab710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd70b7fe640 1 -- 172.21.15.107:0/496292944 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1939173f0 con 0x55c193c9e330 2024-05-09T08:37:14.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd70b7fe640 1 --2- 172.21.15.107:0/496292944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9e330 0x55c193c9e710 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fd700011a20 tx=0x7fd700011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39189a60182014c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd70b7fe640 1 --2- 172.21.15.107:0/3356592340 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd6f40a2d00 0x7fd6f40a51c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd70000f790 tx=0x7fd70001d000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd6f27fc640 1 -- 172.21.15.107:0/496292944 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70002a070 con 0x55c193c9e330 2024-05-09T08:37:14.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd6f27fc640 1 -- 172.21.15.107:0/496292944 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd70000ac90 con 0x55c193c9e330 2024-05-09T08:37:14.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.230+0000 7fd6f27fc640 1 -- 172.21.15.107:0/496292944 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7000259b0 con 0x55c193c9e330 2024-05-09T08:37:14.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.231+0000 7fd715441740 1 -- 172.21.15.107:0/496292944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9e330 msgr2=0x55c193c9e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.231+0000 7fd715441740 1 --2- 172.21.15.107:0/496292944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9e330 0x55c193c9e710 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fd700011a20 tx=0x7fd700011e20 comp rx=0 tx=0).stop 2024-05-09T08:37:14.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.231+0000 7fd715441740 1 -- 172.21.15.107:0/496292944 shutdown_connections 2024-05-09T08:37:14.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.231+0000 7fd715441740 1 --2- 172.21.15.107:0/496292944 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 0x55c193cab710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.231+0000 7fd715441740 1 --2- 172.21.15.107:0/496292944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9ec50 0x55c193ca8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.231+0000 7fd715441740 1 --2- 172.21.15.107:0/496292944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9e330 0x55c193c9e710 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.231+0000 7fd715441740 1 -- 172.21.15.107:0/496292944 >> 172.21.15.107:0/496292944 conn(0x55c193c9a650 msgr2=0x55c193c9aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.231+0000 7fd715441740 1 -- 172.21.15.107:0/496292944 shutdown_connections 2024-05-09T08:37:14.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.231+0000 7fd715441740 1 -- 172.21.15.107:0/496292944 wait complete. 2024-05-09T08:37:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.232+0000 7fd715441740 1 -- start start 2024-05-09T08:37:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.232+0000 7fd715441740 1 Processor -- start 2024-05-09T08:37:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.232+0000 7fd715441740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 0x55c193ca6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.232+0000 7fd715441740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9ec50 0x55c193ca6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.232+0000 7fd715441740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 0x55c193ca6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.232+0000 7fd715441740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c193cac0b0 con 0x55c193c9ec50 2024-05-09T08:37:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.232+0000 7fd715441740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c193bbec40 con 0x55c193ca9320 2024-05-09T08:37:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.232+0000 7fd715441740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c193cac230 con 0x55c193c9e330 2024-05-09T08:37:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.232+0000 7fd715441740 10 client.0 _ll_register_callbacks cb 0x55c193916be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.232+0000 7fd70b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9ec50 0x55c193ca6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.232+0000 7fd70b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9ec50 0x55c193ca6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50330/0 (socket says 172.21.15.107:50330) 2024-05-09T08:37:14.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.232+0000 7fd70b7fe640 1 -- 172.21.15.107:0/2214855357 learned_addr learned my addr 172.21.15.107:0/2214855357 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.232+0000 7fd710cd1640 1 --2- 172.21.15.107:0/2214855357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 0x55c193ca6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.232+0000 7fd70bfff640 1 --2- 172.21.15.107:0/2214855357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 0x55c193ca6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.233+0000 7fd70b7fe640 1 -- 172.21.15.107:0/2214855357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 msgr2=0x55c193ca6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.233+0000 7fd70b7fe640 1 --2- 172.21.15.107:0/2214855357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 0x55c193ca6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.233+0000 7fd70b7fe640 1 -- 172.21.15.107:0/2214855357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 msgr2=0x55c193ca6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.233+0000 7fd70b7fe640 1 --2- 172.21.15.107:0/2214855357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 0x55c193ca6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.233+0000 7fd70b7fe640 1 -- 172.21.15.107:0/2214855357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1939e17d0 con 0x55c193c9ec50 2024-05-09T08:37:14.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.233+0000 7fd710cd1640 1 --2- 172.21.15.107:0/2214855357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 0x55c193ca6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:14.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.233+0000 7fd70bfff640 1 --2- 172.21.15.107:0/2214855357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 0x55c193ca6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:14.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.233+0000 7fd70b7fe640 1 --2- 172.21.15.107:0/2214855357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9ec50 0x55c193ca6780 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fd70000ae40 tx=0x7fd700005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e99a5a1502ba8ad3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.233+0000 7fd6eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:14.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.233+0000 7fd6eb7fe640 1 -- 172.21.15.107:0/2214855357 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70002a030 con 0x55c193c9ec50 2024-05-09T08:37:14.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.233+0000 7fd6eb7fe640 1 -- 172.21.15.107:0/2214855357 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd70001cb80 con 0x55c193c9ec50 2024-05-09T08:37:14.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.233+0000 7fd715441740 1 -- 172.21.15.107:0/2214855357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd70000ac90 con 0x55c193c9ec50 2024-05-09T08:37:14.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.233+0000 7fd6eb7fe640 1 -- 172.21.15.107:0/2214855357 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd70001cd80 con 0x55c193c9ec50 2024-05-09T08:37:14.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.233+0000 7fd715441740 10 client.16803 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:14.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.233+0000 7fd715441740 1 -- 172.21.15.107:0/2214855357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c193cb8240 con 0x55c193c9ec50 2024-05-09T08:37:14.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.233+0000 7fd6eb7fe640 1 -- 172.21.15.107:0/2214855357 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd70000ac90 con 0x55c193c9ec50 2024-05-09T08:37:14.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.234+0000 7fd6eb7fe640 1 -- 172.21.15.107:0/2214855357 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd700030670 con 0x55c193c9ec50 2024-05-09T08:37:14.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.234+0000 7fd715441740 10 client.16803 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:14.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.234+0000 7fd715441740 10 client.16803 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:37:14.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.234+0000 7fd715441740 20 client.16803 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:14.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.234+0000 7fd715441740 1 --2- 172.21.15.107:0/2214855357 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55c193cb9700 0x55c193cbbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.234+0000 7fd715441740 4 client.16803 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:14.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.234+0000 7fd715441740 1 -- 172.21.15.107:0/2214855357 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c193c11370 con 0x55c193cb9700 2024-05-09T08:37:14.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.234+0000 7fd70bfff640 1 --2- 172.21.15.107:0/2214855357 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55c193cb9700 0x55c193cbbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.234+0000 7fd6eb7fe640 10 client.16803 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:37:14.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.234+0000 7fd70bfff640 1 --2- 172.21.15.107:0/2214855357 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55c193cb9700 0x55c193cbbaf0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=545e0a2b02644e91 server_cookie=fa5e2b0f8f4533ee in_seq=0 out_seq=0 2024-05-09T08:37:14.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.235+0000 7fd6eb7fe640 1 -- 172.21.15.107:0/2214855357 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c193c11370 con 0x55c193cb9700 2024-05-09T08:37:14.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.235+0000 7fd6eb7fe640 10 client.16803 handle_command_reply: tid=0 2024-05-09T08:37:14.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.235+0000 7fd715441740 1 client.16803 shutdown 2024-05-09T08:37:14.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.235+0000 7fd715441740 1 -- 172.21.15.107:0/2214855357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9ec50 msgr2=0x55c193ca6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.235+0000 7fd715441740 1 --2- 172.21.15.107:0/2214855357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9ec50 0x55c193ca6780 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fd70000ae40 tx=0x7fd700005dd0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.235+0000 7fd715441740 1 -- 172.21.15.107:0/2214855357 shutdown_connections 2024-05-09T08:37:14.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.235+0000 7fd715441740 1 --2- 172.21.15.107:0/2214855357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 0x55c193ca6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.235+0000 7fd715441740 1 --2- 172.21.15.107:0/2214855357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9ec50 0x55c193ca6780 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.235+0000 7fd715441740 1 --2- 172.21.15.107:0/2214855357 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55c193cb9700 0x55c193cbbaf0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.235+0000 7fd715441740 1 --2- 172.21.15.107:0/2214855357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 0x55c193ca6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.235+0000 7fd6eb7fe640 0 client.16803 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:37:14.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.235+0000 7fd715441740 1 -- 172.21.15.107:0/2214855357 >> 172.21.15.107:0/2214855357 conn(0x55c193c9a650 msgr2=0x55c193ca0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.235+0000 7fd715441740 1 -- 172.21.15.107:0/2214855357 shutdown_connections 2024-05-09T08:37:14.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.235+0000 7fd715441740 1 -- 172.21.15.107:0/2214855357 wait complete. 2024-05-09T08:37:14.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.236+0000 7fd715441740 20 client.16803 trim_cache size 0 max 16384 2024-05-09T08:37:14.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd715441740 1 Processor -- start 2024-05-09T08:37:14.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd715441740 1 -- start start 2024-05-09T08:37:14.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd715441740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 0x55c193cb4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd715441740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193c9ec50 0x55c193ca53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd715441740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193ca9320 0x55c193cbbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd715441740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c1b1580 con 0x55c193ca9320 2024-05-09T08:37:14.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd715441740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c1aee50 con 0x55c193c9ec50 2024-05-09T08:37:14.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd715441740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c1acca0 con 0x55c193c9e330 2024-05-09T08:37:14.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd710cd1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193ca9320 0x55c193cbbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd710cd1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193ca9320 0x55c193cbbaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50340/0 (socket says 172.21.15.107:50340) 2024-05-09T08:37:14.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd710cd1640 1 -- 172.21.15.107:0/756221523 learned_addr learned my addr 172.21.15.107:0/756221523 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd70b7fe640 1 --2- 172.21.15.107:0/756221523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193c9ec50 0x55c193ca53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd70bfff640 1 --2- 172.21.15.107:0/756221523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 0x55c193cb4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd710cd1640 1 -- 172.21.15.107:0/756221523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 msgr2=0x55c193cb4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd710cd1640 1 --2- 172.21.15.107:0/756221523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 0x55c193cb4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd710cd1640 1 -- 172.21.15.107:0/756221523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193c9ec50 msgr2=0x55c193ca53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd710cd1640 1 --2- 172.21.15.107:0/756221523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193c9ec50 0x55c193ca53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd710cd1640 1 -- 172.21.15.107:0/756221523 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c193c9a480 con 0x55c193ca9320 2024-05-09T08:37:14.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.249+0000 7fd70bfff640 1 --2- 172.21.15.107:0/756221523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 0x55c193cb4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:14.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.250+0000 7fd710cd1640 1 --2- 172.21.15.107:0/756221523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193ca9320 0x55c193cbbaf0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fd7040984d0 tx=0x7fd704004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1ddd6550ed0caeb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.250+0000 7fd6f1ffb640 1 -- 172.21.15.107:0/756221523 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7040c6e60 con 0x55c193ca9320 2024-05-09T08:37:14.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.250+0000 7fd6f1ffb640 1 -- 172.21.15.107:0/756221523 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd70409c770 con 0x55c193ca9320 2024-05-09T08:37:14.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.250+0000 7fd6f1ffb640 1 -- 172.21.15.107:0/756221523 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7040986e0 con 0x55c193ca9320 2024-05-09T08:37:14.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 -- 172.21.15.107:0/756221523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193ca9320 msgr2=0x55c193cbbaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 --2- 172.21.15.107:0/756221523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193ca9320 0x55c193cbbaf0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fd7040984d0 tx=0x7fd704004a40 comp rx=0 tx=0).stop 2024-05-09T08:37:14.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 -- 172.21.15.107:0/756221523 shutdown_connections 2024-05-09T08:37:14.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 --2- 172.21.15.107:0/756221523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193ca9320 0x55c193cbbaf0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 --2- 172.21.15.107:0/756221523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193c9ec50 0x55c193ca53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 --2- 172.21.15.107:0/756221523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 0x55c193cb4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 -- 172.21.15.107:0/756221523 >> 172.21.15.107:0/756221523 conn(0x55c193c9a650 msgr2=0x55c193ca5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 -- 172.21.15.107:0/756221523 shutdown_connections 2024-05-09T08:37:14.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 -- 172.21.15.107:0/756221523 wait complete. 2024-05-09T08:37:14.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 -- start start 2024-05-09T08:37:14.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 Processor -- start 2024-05-09T08:37:14.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 0x55c193cb1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9ec50 0x55c193ca35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 0x55c193ca3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c1ad310 con 0x55c193c9ec50 2024-05-09T08:37:14.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c1ae780 con 0x55c193ca9320 2024-05-09T08:37:14.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c1ad9e0 con 0x55c193c9e330 2024-05-09T08:37:14.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.251+0000 7fd715441740 10 client.0 _ll_register_callbacks cb 0x7fd700028070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:14.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd70b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9ec50 0x55c193ca35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd70b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9ec50 0x55c193ca35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50342/0 (socket says 172.21.15.107:50342) 2024-05-09T08:37:14.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd70b7fe640 1 -- 172.21.15.107:0/580717734 learned_addr learned my addr 172.21.15.107:0/580717734 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd710cd1640 1 --2- 172.21.15.107:0/580717734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 0x55c193ca3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd70bfff640 1 --2- 172.21.15.107:0/580717734 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 0x55c193cb1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd70b7fe640 1 -- 172.21.15.107:0/580717734 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 msgr2=0x55c193cb1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd70b7fe640 1 --2- 172.21.15.107:0/580717734 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 0x55c193cb1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd70b7fe640 1 -- 172.21.15.107:0/580717734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 msgr2=0x55c193ca3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd70b7fe640 1 --2- 172.21.15.107:0/580717734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 0x55c193ca3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd70b7fe640 1 -- 172.21.15.107:0/580717734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c193ca24d0 con 0x55c193c9ec50 2024-05-09T08:37:14.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd710cd1640 1 --2- 172.21.15.107:0/580717734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 0x55c193ca3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:14.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd70b7fe640 1 --2- 172.21.15.107:0/580717734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9ec50 0x55c193ca35c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fd70001cfd0 tx=0x7fd70000f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43828f3cd30aa8d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd6eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd6eb7fe640 1 -- 172.21.15.107:0/580717734 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700044060 con 0x55c193c9ec50 2024-05-09T08:37:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd6eb7fe640 1 -- 172.21.15.107:0/580717734 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd700025660 con 0x55c193c9ec50 2024-05-09T08:37:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd715441740 1 -- 172.21.15.107:0/580717734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd70409c770 con 0x55c193c9ec50 2024-05-09T08:37:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd6eb7fe640 1 -- 172.21.15.107:0/580717734 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd70004b050 con 0x55c193c9ec50 2024-05-09T08:37:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd715441740 10 client.16809 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.252+0000 7fd715441740 1 -- 172.21.15.107:0/580717734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c193d1ecd0 con 0x55c193c9ec50 2024-05-09T08:37:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.253+0000 7fd6eb7fe640 1 -- 172.21.15.107:0/580717734 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd70409c770 con 0x55c193c9ec50 2024-05-09T08:37:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.253+0000 7fd6eb7fe640 1 -- 172.21.15.107:0/580717734 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd70003f500 con 0x55c193c9ec50 2024-05-09T08:37:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.253+0000 7fd715441740 10 client.16809 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.253+0000 7fd715441740 10 client.16809 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:37:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.253+0000 7fd715441740 20 client.16809 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:14.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.253+0000 7fd715441740 1 --2- 172.21.15.107:0/580717734 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55c193d1f340 0x55c193d21760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.253+0000 7fd715441740 4 client.16809 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:37:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.253+0000 7fd715441740 1 -- 172.21.15.107:0/580717734 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c193c0b6c0 con 0x55c193d1f340 2024-05-09T08:37:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.253+0000 7fd70bfff640 1 --2- 172.21.15.107:0/580717734 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55c193d1f340 0x55c193d21760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.253+0000 7fd6eb7fe640 10 client.16809 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:37:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.253+0000 7fd70bfff640 1 --2- 172.21.15.107:0/580717734 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55c193d1f340 0x55c193d21760 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b9a95bf8d507cc9c server_cookie=cd062984eb0a723d in_seq=0 out_seq=0 2024-05-09T08:37:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.253+0000 7fd6eb7fe640 1 -- 172.21.15.107:0/580717734 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c193c0b6c0 con 0x55c193d1f340 2024-05-09T08:37:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd6eb7fe640 10 client.16809 handle_command_reply: tid=0 2024-05-09T08:37:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd715441740 1 client.16809 shutdown 2024-05-09T08:37:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd715441740 1 -- 172.21.15.107:0/580717734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9ec50 msgr2=0x55c193ca35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd715441740 1 --2- 172.21.15.107:0/580717734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9ec50 0x55c193ca35c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fd70001cfd0 tx=0x7fd70000f840 comp rx=0 tx=0).stop 2024-05-09T08:37:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd715441740 1 -- 172.21.15.107:0/580717734 shutdown_connections 2024-05-09T08:37:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd715441740 1 --2- 172.21.15.107:0/580717734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c193ca9320 0x55c193ca3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd715441740 1 --2- 172.21.15.107:0/580717734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c193c9ec50 0x55c193ca35c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd715441740 1 --2- 172.21.15.107:0/580717734 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55c193d1f340 0x55c193d21760 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd715441740 1 --2- 172.21.15.107:0/580717734 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c193c9e330 0x55c193cb1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd715441740 1 -- 172.21.15.107:0/580717734 >> 172.21.15.107:0/580717734 conn(0x55c193c9a650 msgr2=0x55c193c98f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd6eb7fe640 0 client.16809 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:37:14.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd715441740 1 -- 172.21.15.107:0/580717734 shutdown_connections 2024-05-09T08:37:14.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd715441740 1 -- 172.21.15.107:0/580717734 wait complete. 2024-05-09T08:37:14.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd715441740 20 client.16809 trim_cache size 0 max 16384 2024-05-09T08:37:14.261 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:37:14.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd712734640 1 -- 172.21.15.107:0/3356592340 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd6f40a2d00 msgr2=0x7fd6f40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd712734640 1 --2- 172.21.15.107:0/3356592340 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd6f40a2d00 0x7fd6f40a51c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd70000f790 tx=0x7fd70001d000 comp rx=0 tx=0).stop 2024-05-09T08:37:14.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd712734640 1 -- 172.21.15.107:0/3356592340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd70c106710 msgr2=0x7fd70c19c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.254+0000 7fd712734640 1 --2- 172.21.15.107:0/3356592340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd70c106710 0x7fd70c19c8f0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd70400eb10 tx=0x7fd70400c2d0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.255+0000 7fd712734640 1 -- 172.21.15.107:0/3356592340 shutdown_connections 2024-05-09T08:37:14.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.255+0000 7fd712734640 1 --2- 172.21.15.107:0/3356592340 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd6f40a2d00 0x7fd6f40a51c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.255+0000 7fd712734640 1 --2- 172.21.15.107:0/3356592340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd70c106710 0x7fd70c19c8f0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.255+0000 7fd712734640 1 --2- 172.21.15.107:0/3356592340 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd70c1010c0 0x7fd70c1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.255+0000 7fd712734640 1 --2- 172.21.15.107:0/3356592340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd70c100700 0x7fd70c198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.255+0000 7fd712734640 1 -- 172.21.15.107:0/3356592340 >> 172.21.15.107:0/3356592340 conn(0x7fd70c077830 msgr2=0x7fd70c0ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.255+0000 7fd712734640 1 -- 172.21.15.107:0/3356592340 shutdown_connections 2024-05-09T08:37:14.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.255+0000 7fd712734640 1 -- 172.21.15.107:0/3356592340 wait complete. 2024-05-09T08:37:14.264 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:14 smithi107 ceph-mon[30122]: pgmap v119: 129 pgs: 129 active+clean; 3.9 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 37 MiB/s wr, 41 op/s 2024-05-09T08:37:14.264 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:14 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1208395572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:37:14.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:37:14.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:14 smithi157 ceph-mon[33614]: pgmap v119: 129 pgs: 129 active+clean; 3.9 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 37 MiB/s wr, 41 op/s 2024-05-09T08:37:14.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:14 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1208395572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:37:14.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:14 smithi110 ceph-mon[33622]: pgmap v119: 129 pgs: 129 active+clean; 3.9 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 37 MiB/s wr, 41 op/s 2024-05-09T08:37:14.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:14 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1208395572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:37:14.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.385+0000 7f4eb222c640 1 -- 172.21.15.107:0/618509033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4eac1021b0 msgr2=0x7f4eac10e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.385+0000 7f4eb222c640 1 --2- 172.21.15.107:0/618509033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4eac1021b0 0x7f4eac10e6c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4009cd0 tx=0x7f4ea402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.385+0000 7f4eb222c640 1 -- 172.21.15.107:0/618509033 shutdown_connections 2024-05-09T08:37:14.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.385+0000 7f4eb222c640 1 --2- 172.21.15.107:0/618509033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4eac1021b0 0x7f4eac10e6c0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.385+0000 7f4eb222c640 1 --2- 172.21.15.107:0/618509033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4eac1017f0 0x7f4eac101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.385+0000 7f4eb222c640 1 --2- 172.21.15.107:0/618509033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4eac107800 0x7f4eac107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.385+0000 7f4eb222c640 1 -- 172.21.15.107:0/618509033 >> 172.21.15.107:0/618509033 conn(0x7f4eac0fd2a0 msgr2=0x7f4eac0ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.385+0000 7f4eb222c640 1 -- 172.21.15.107:0/618509033 shutdown_connections 2024-05-09T08:37:14.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.385+0000 7f4eb222c640 1 -- 172.21.15.107:0/618509033 wait complete. 2024-05-09T08:37:14.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.385+0000 7f4eb222c640 1 Processor -- start 2024-05-09T08:37:14.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eb222c640 1 -- start start 2024-05-09T08:37:14.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eb222c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4eac1017f0 0x7f4eac19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eb222c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4eac1021b0 0x7f4eac19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eb222c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4eac107800 0x7f4eac19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eb222c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac111ac0 con 0x7f4eac1021b0 2024-05-09T08:37:14.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eb222c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac111940 con 0x7f4eac107800 2024-05-09T08:37:14.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eb222c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac111c40 con 0x7f4eac1017f0 2024-05-09T08:37:14.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eaaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4eac1021b0 0x7f4eac19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eaaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4eac1021b0 0x7f4eac19a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50360/0 (socket says 172.21.15.107:50360) 2024-05-09T08:37:14.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eaaffd640 1 -- 172.21.15.107:0/1249897619 learned_addr learned my addr 172.21.15.107:0/1249897619 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eab7fe640 1 --2- 172.21.15.107:0/1249897619 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4eac1017f0 0x7f4eac19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eabfff640 1 --2- 172.21.15.107:0/1249897619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4eac107800 0x7f4eac19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eaaffd640 1 -- 172.21.15.107:0/1249897619 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4eac1017f0 msgr2=0x7f4eac19a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eaaffd640 1 --2- 172.21.15.107:0/1249897619 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4eac1017f0 0x7f4eac19a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eaaffd640 1 -- 172.21.15.107:0/1249897619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4eac107800 msgr2=0x7f4eac19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eaaffd640 1 --2- 172.21.15.107:0/1249897619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4eac107800 0x7f4eac19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.386+0000 7f4eaaffd640 1 -- 172.21.15.107:0/1249897619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eac19f210 con 0x7f4eac1021b0 2024-05-09T08:37:14.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.387+0000 7f4eaaffd640 1 --2- 172.21.15.107:0/1249897619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4eac1021b0 0x7f4eac19a760 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000d770 tx=0x7f4ea000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.387+0000 7f4ea8ff9640 1 -- 172.21.15.107:0/1249897619 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0017070 con 0x7f4eac1021b0 2024-05-09T08:37:14.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.387+0000 7f4eb222c640 1 -- 172.21.15.107:0/1249897619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eac19f500 con 0x7f4eac1021b0 2024-05-09T08:37:14.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.387+0000 7f4ea8ff9640 1 -- 172.21.15.107:0/1249897619 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ea00054d0 con 0x7f4eac1021b0 2024-05-09T08:37:14.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.387+0000 7f4ea8ff9640 1 -- 172.21.15.107:0/1249897619 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0004cf0 con 0x7f4eac1021b0 2024-05-09T08:37:14.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.387+0000 7f4eb222c640 1 -- 172.21.15.107:0/1249897619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eac1a6d90 con 0x7f4eac1021b0 2024-05-09T08:37:14.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.389+0000 7f4eb500f740 1 Processor -- start 2024-05-09T08:37:14.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.389+0000 7f4ea8ff9640 1 -- 172.21.15.107:0/1249897619 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f4ea0004e90 con 0x7f4eac1021b0 2024-05-09T08:37:14.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.389+0000 7f4eb500f740 1 -- start start 2024-05-09T08:37:14.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eb500f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb30330 0x55eeceb30710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eb500f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb30c50 0x55eeceb3ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eb500f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb3b320 0x55eeceb3d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eb500f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55eecea95c40 con 0x55eeceb3b320 2024-05-09T08:37:14.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eb500f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55eecea3d580 con 0x55eeceb30330 2024-05-09T08:37:14.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eb500f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55eecea91c00 con 0x55eeceb30c50 2024-05-09T08:37:14.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eaaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb3b320 0x55eeceb3d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eaaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb3b320 0x55eeceb3d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50364/0 (socket says 172.21.15.107:50364) 2024-05-09T08:37:14.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eaaffd640 1 -- 172.21.15.107:0/3216340056 learned_addr learned my addr 172.21.15.107:0/3216340056 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eab7fe640 1 --2- 172.21.15.107:0/3216340056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb30c50 0x55eeceb3ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eabfff640 1 --2- 172.21.15.107:0/3216340056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb30330 0x55eeceb30710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4ea8ff9640 1 --2- 172.21.15.107:0/1249897619 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4e940a2d00 0x7f4e940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eaaffd640 1 -- 172.21.15.107:0/3216340056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb30c50 msgr2=0x55eeceb3ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eaaffd640 1 --2- 172.21.15.107:0/3216340056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb30c50 0x55eeceb3ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eaaffd640 1 -- 172.21.15.107:0/3216340056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb30330 msgr2=0x55eeceb30710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eaaffd640 1 --2- 172.21.15.107:0/3216340056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb30330 0x55eeceb30710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eaaffd640 1 -- 172.21.15.107:0/3216340056 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eece7a93f0 con 0x55eeceb3b320 2024-05-09T08:37:14.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4eabfff640 1 --2- 172.21.15.107:0/1249897619 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4e940a2d00 0x7f4e940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.390+0000 7f4ea8ff9640 1 -- 172.21.15.107:0/1249897619 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4ea00cf220 con 0x7f4eac1021b0 2024-05-09T08:37:14.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.391+0000 7f4eaaffd640 1 --2- 172.21.15.107:0/3216340056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb3b320 0x55eeceb3d710 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4ea00cc580 tx=0x7f4ea0066010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec6bdac988261f64 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.391+0000 7f4e91ffb640 1 -- 172.21.15.107:0/3216340056 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0020070 con 0x55eeceb3b320 2024-05-09T08:37:14.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.391+0000 7f4eabfff640 1 --2- 172.21.15.107:0/1249897619 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4e940a2d00 0x7f4e940a51c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4005e30 tx=0x7f4ea402a8e0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.391+0000 7f4e91ffb640 1 -- 172.21.15.107:0/3216340056 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ea0009c30 con 0x55eeceb3b320 2024-05-09T08:37:14.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.391+0000 7f4e91ffb640 1 -- 172.21.15.107:0/3216340056 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea009af20 con 0x55eeceb3b320 2024-05-09T08:37:14.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.391+0000 7f4eb500f740 1 -- 172.21.15.107:0/3216340056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb3b320 msgr2=0x55eeceb3d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.391+0000 7f4eb500f740 1 --2- 172.21.15.107:0/3216340056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb3b320 0x55eeceb3d710 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4ea00cc580 tx=0x7f4ea0066010 comp rx=0 tx=0).stop 2024-05-09T08:37:14.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.392+0000 7f4eb500f740 1 -- 172.21.15.107:0/3216340056 shutdown_connections 2024-05-09T08:37:14.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.392+0000 7f4eb500f740 1 --2- 172.21.15.107:0/3216340056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb3b320 0x55eeceb3d710 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.392+0000 7f4eb500f740 1 --2- 172.21.15.107:0/3216340056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb30c50 0x55eeceb3ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.392+0000 7f4eb500f740 1 --2- 172.21.15.107:0/3216340056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb30330 0x55eeceb30710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.392+0000 7f4eb500f740 1 -- 172.21.15.107:0/3216340056 >> 172.21.15.107:0/3216340056 conn(0x55eeceb2c650 msgr2=0x55eeceb2ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.392+0000 7f4eb500f740 1 -- 172.21.15.107:0/3216340056 shutdown_connections 2024-05-09T08:37:14.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.392+0000 7f4eb500f740 1 -- 172.21.15.107:0/3216340056 wait complete. 2024-05-09T08:37:14.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.392+0000 7f4eb500f740 1 -- start start 2024-05-09T08:37:14.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.392+0000 7f4eb500f740 1 Processor -- start 2024-05-09T08:37:14.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.392+0000 7f4eb500f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb30330 0x55eeceb38260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eb500f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb30c50 0x55eeceb387a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eb500f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb3b320 0x55eeceb38ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eb500f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55eeceb3e0b0 con 0x55eeceb30c50 2024-05-09T08:37:14.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eb500f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55eecea50c40 con 0x55eeceb30330 2024-05-09T08:37:14.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eb500f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55eeceb3e230 con 0x55eeceb3b320 2024-05-09T08:37:14.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eb500f740 10 client.0 _ll_register_callbacks cb 0x55eece7a8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:14.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eaaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb30c50 0x55eeceb387a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eaaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb30c50 0x55eeceb387a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50380/0 (socket says 172.21.15.107:50380) 2024-05-09T08:37:14.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eaaffd640 1 -- 172.21.15.107:0/3468014165 learned_addr learned my addr 172.21.15.107:0/3468014165 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eabfff640 1 --2- 172.21.15.107:0/3468014165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb3b320 0x55eeceb38ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eab7fe640 1 --2- 172.21.15.107:0/3468014165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb30330 0x55eeceb38260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eaaffd640 1 -- 172.21.15.107:0/3468014165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb3b320 msgr2=0x55eeceb38ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eaaffd640 1 --2- 172.21.15.107:0/3468014165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb3b320 0x55eeceb38ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eaaffd640 1 -- 172.21.15.107:0/3468014165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb30330 msgr2=0x55eeceb38260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eaaffd640 1 --2- 172.21.15.107:0/3468014165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb30330 0x55eeceb38260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eaaffd640 1 -- 172.21.15.107:0/3468014165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eece8737d0 con 0x55eeceb30c50 2024-05-09T08:37:14.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eabfff640 1 --2- 172.21.15.107:0/3468014165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb3b320 0x55eeceb38ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:14.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eaaffd640 1 --2- 172.21.15.107:0/3468014165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb30c50 0x55eeceb387a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f4ea00cc6c0 tx=0x7f4ea00cf590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7343d2a906a2d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4e8affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:14.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4e8affd640 1 -- 172.21.15.107:0/3468014165 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0020050 con 0x55eeceb30c50 2024-05-09T08:37:14.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4e8affd640 1 -- 172.21.15.107:0/3468014165 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4ea00d2a00 con 0x55eeceb30c50 2024-05-09T08:37:14.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4eb500f740 1 -- 172.21.15.107:0/3468014165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55eeceaa3370 con 0x55eeceb30c50 2024-05-09T08:37:14.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.393+0000 7f4e8affd640 1 -- 172.21.15.107:0/3468014165 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4ea0005180 con 0x55eeceb30c50 2024-05-09T08:37:14.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.394+0000 7f4eb500f740 10 client.16821 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:14.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.394+0000 7f4eb500f740 1 -- 172.21.15.107:0/3468014165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55eeceb4a270 con 0x55eeceb30c50 2024-05-09T08:37:14.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.394+0000 7f4e8affd640 1 -- 172.21.15.107:0/3468014165 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55eeceaa3370 con 0x55eeceb30c50 2024-05-09T08:37:14.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.394+0000 7f4e8affd640 1 -- 172.21.15.107:0/3468014165 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f4ea00cc890 con 0x55eeceb30c50 2024-05-09T08:37:14.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.394+0000 7f4eb500f740 10 client.16821 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:14.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.394+0000 7f4eb500f740 10 client.16821 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:14.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.394+0000 7f4eb500f740 20 client.16821 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:14.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.394+0000 7f4eb500f740 1 --2- 172.21.15.107:0/3468014165 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55eeceb4b5c0 0x55eeceb4d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.394+0000 7f4eb500f740 4 client.16821 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:14.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.394+0000 7f4eb500f740 1 -- 172.21.15.107:0/3468014165 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eecea9d6c0 con 0x55eeceb4b5c0 2024-05-09T08:37:14.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.394+0000 7f4eab7fe640 1 --2- 172.21.15.107:0/3468014165 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55eeceb4b5c0 0x55eeceb4d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.395+0000 7f4e8affd640 10 client.16821 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:14.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.395+0000 7f4eab7fe640 1 --2- 172.21.15.107:0/3468014165 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55eeceb4b5c0 0x55eeceb4d9b0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e426fd4366e23a87 server_cookie=3a8a13eb941c03eb in_seq=0 out_seq=0 2024-05-09T08:37:14.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.395+0000 7f4e8affd640 1 -- 172.21.15.107:0/3468014165 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eecea9d6c0 con 0x55eeceb4b5c0 2024-05-09T08:37:14.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.396+0000 7f4e8affd640 10 client.16821 handle_command_reply: tid=0 2024-05-09T08:37:14.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.396+0000 7f4eb500f740 1 client.16821 shutdown 2024-05-09T08:37:14.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.396+0000 7f4eb500f740 1 -- 172.21.15.107:0/3468014165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb30c50 msgr2=0x55eeceb387a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.396+0000 7f4eb500f740 1 --2- 172.21.15.107:0/3468014165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb30c50 0x55eeceb387a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f4ea00cc6c0 tx=0x7f4ea00cf590 comp rx=0 tx=0).stop 2024-05-09T08:37:14.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.396+0000 7f4eb500f740 1 -- 172.21.15.107:0/3468014165 shutdown_connections 2024-05-09T08:37:14.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.396+0000 7f4eb500f740 1 --2- 172.21.15.107:0/3468014165 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55eeceb4b5c0 0x55eeceb4d9b0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.396+0000 7f4eb500f740 1 --2- 172.21.15.107:0/3468014165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb3b320 0x55eeceb38ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.396+0000 7f4eb500f740 1 --2- 172.21.15.107:0/3468014165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb30c50 0x55eeceb387a0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.396+0000 7f4eb500f740 1 --2- 172.21.15.107:0/3468014165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb30330 0x55eeceb38260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.396+0000 7f4e8affd640 0 client.16821 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:14.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.396+0000 7f4eb500f740 1 -- 172.21.15.107:0/3468014165 >> 172.21.15.107:0/3468014165 conn(0x55eeceb2c650 msgr2=0x55eeceb32450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.396+0000 7f4eb500f740 1 -- 172.21.15.107:0/3468014165 shutdown_connections 2024-05-09T08:37:14.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.396+0000 7f4eb500f740 1 -- 172.21.15.107:0/3468014165 wait complete. 2024-05-09T08:37:14.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.396+0000 7f4eb500f740 20 client.16821 trim_cache size 0 max 16384 2024-05-09T08:37:14.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eb500f740 1 Processor -- start 2024-05-09T08:37:14.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eb500f740 1 -- start start 2024-05-09T08:37:14.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eb500f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb30330 0x55eeceb46f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eb500f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb30c50 0x55eeceb373c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eb500f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb3b320 0x55eeceb32a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eb500f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac1b3570 con 0x55eeceb30330 2024-05-09T08:37:14.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eb500f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac1b0e40 con 0x55eeceb3b320 2024-05-09T08:37:14.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eb500f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac1aec90 con 0x55eeceb30c50 2024-05-09T08:37:14.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eab7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb30330 0x55eeceb46f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eab7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb30330 0x55eeceb46f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50388/0 (socket says 172.21.15.107:50388) 2024-05-09T08:37:14.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eab7fe640 1 -- 172.21.15.107:0/3154750125 learned_addr learned my addr 172.21.15.107:0/3154750125 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eabfff640 1 --2- 172.21.15.107:0/3154750125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb3b320 0x55eeceb32a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eaaffd640 1 --2- 172.21.15.107:0/3154750125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb30c50 0x55eeceb373c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eab7fe640 1 -- 172.21.15.107:0/3154750125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb30c50 msgr2=0x55eeceb373c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eab7fe640 1 --2- 172.21.15.107:0/3154750125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb30c50 0x55eeceb373c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eab7fe640 1 -- 172.21.15.107:0/3154750125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb3b320 msgr2=0x55eeceb32a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eab7fe640 1 --2- 172.21.15.107:0/3154750125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb3b320 0x55eeceb32a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.411+0000 7f4eab7fe640 1 -- 172.21.15.107:0/3154750125 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eeceb4dd20 con 0x55eeceb30330 2024-05-09T08:37:14.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.412+0000 7f4eaaffd640 1 --2- 172.21.15.107:0/3154750125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb30c50 0x55eeceb373c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:14.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.412+0000 7f4eab7fe640 1 --2- 172.21.15.107:0/3154750125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb30330 0x55eeceb46f70 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x55eeceb4df20 tx=0x7f4e9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4553cc2aebb45324 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.412+0000 7f4e917fa640 1 -- 172.21.15.107:0/3154750125 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c025070 con 0x55eeceb30330 2024-05-09T08:37:14.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.412+0000 7f4e917fa640 1 -- 172.21.15.107:0/3154750125 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e9c003d20 con 0x55eeceb30330 2024-05-09T08:37:14.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.412+0000 7f4e917fa640 1 -- 172.21.15.107:0/3154750125 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c010850 con 0x55eeceb30330 2024-05-09T08:37:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.413+0000 7f4eb500f740 1 -- 172.21.15.107:0/3154750125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb30330 msgr2=0x55eeceb46f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.413+0000 7f4eb500f740 1 --2- 172.21.15.107:0/3154750125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb30330 0x55eeceb46f70 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x55eeceb4df20 tx=0x7f4e9c0046c0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.413+0000 7f4eb500f740 1 -- 172.21.15.107:0/3154750125 shutdown_connections 2024-05-09T08:37:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.413+0000 7f4eb500f740 1 --2- 172.21.15.107:0/3154750125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb3b320 0x55eeceb32a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.413+0000 7f4eb500f740 1 --2- 172.21.15.107:0/3154750125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb30c50 0x55eeceb373c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.413+0000 7f4eb500f740 1 --2- 172.21.15.107:0/3154750125 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb30330 0x55eeceb46f70 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.413+0000 7f4eb500f740 1 -- 172.21.15.107:0/3154750125 >> 172.21.15.107:0/3154750125 conn(0x55eeceb2c650 msgr2=0x55eeceb2af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.413+0000 7f4eb500f740 1 -- 172.21.15.107:0/3154750125 shutdown_connections 2024-05-09T08:37:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.413+0000 7f4eb500f740 1 -- 172.21.15.107:0/3154750125 wait complete. 2024-05-09T08:37:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.413+0000 7f4eb500f740 1 -- start start 2024-05-09T08:37:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.413+0000 7f4eb500f740 1 Processor -- start 2024-05-09T08:37:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eb500f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb30330 0x55eeceb43670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eb500f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb30c50 0x55eeceb34a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eb500f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb3b320 0x55eeceb34fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eb500f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac1af300 con 0x55eeceb3b320 2024-05-09T08:37:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eb500f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac1b0770 con 0x55eeceb30c50 2024-05-09T08:37:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eb500f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac1af9d0 con 0x55eeceb30330 2024-05-09T08:37:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eb500f740 10 client.0 _ll_register_callbacks cb 0x7f4ea0095f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eabfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb3b320 0x55eeceb34fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eabfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb3b320 0x55eeceb34fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50404/0 (socket says 172.21.15.107:50404) 2024-05-09T08:37:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eabfff640 1 -- 172.21.15.107:0/198803015 learned_addr learned my addr 172.21.15.107:0/198803015 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eaaffd640 1 --2- 172.21.15.107:0/198803015 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb30c50 0x55eeceb34a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eab7fe640 1 --2- 172.21.15.107:0/198803015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb30330 0x55eeceb43670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eabfff640 1 -- 172.21.15.107:0/198803015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb30330 msgr2=0x55eeceb43670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eabfff640 1 --2- 172.21.15.107:0/198803015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb30330 0x55eeceb43670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eabfff640 1 -- 172.21.15.107:0/198803015 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb30c50 msgr2=0x55eeceb34a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eabfff640 1 --2- 172.21.15.107:0/198803015 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb30c50 0x55eeceb34a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eabfff640 1 -- 172.21.15.107:0/198803015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eeceb35590 con 0x55eeceb3b320 2024-05-09T08:37:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eabfff640 1 --2- 172.21.15.107:0/198803015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb3b320 0x55eeceb34fd0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4009cd0 tx=0x7f4ea40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d38d0cf25837f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4e8affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4e8affd640 1 -- 172.21.15.107:0/198803015 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4004720 con 0x55eeceb3b320 2024-05-09T08:37:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4e8affd640 1 -- 172.21.15.107:0/198803015 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4ea40305a0 con 0x55eeceb3b320 2024-05-09T08:37:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.414+0000 7f4eb500f740 1 -- 172.21.15.107:0/198803015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e9c003d20 con 0x55eeceb3b320 2024-05-09T08:37:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.415+0000 7f4e8affd640 1 -- 172.21.15.107:0/198803015 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4ea4004b40 con 0x55eeceb3b320 2024-05-09T08:37:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.415+0000 7f4eb500f740 10 client.16827 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:14.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.415+0000 7f4eb500f740 1 -- 172.21.15.107:0/198803015 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55eecebb1540 con 0x55eeceb3b320 2024-05-09T08:37:14.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.415+0000 7f4e8affd640 1 -- 172.21.15.107:0/198803015 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e9c003d20 con 0x55eeceb3b320 2024-05-09T08:37:14.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.415+0000 7f4e8affd640 1 -- 172.21.15.107:0/198803015 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f4ea4047040 con 0x55eeceb3b320 2024-05-09T08:37:14.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.415+0000 7f4eb500f740 10 client.16827 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:14.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.415+0000 7f4eb500f740 10 client.16827 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:14.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.415+0000 7f4eb500f740 20 client.16827 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:14.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.415+0000 7f4eb500f740 1 --2- 172.21.15.107:0/198803015 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55eecebb1d70 0x55eecebb4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:14.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.415+0000 7f4eb500f740 4 client.16827 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:37:14.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.415+0000 7f4eb500f740 1 -- 172.21.15.107:0/198803015 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eecea1a820 con 0x55eecebb1d70 2024-05-09T08:37:14.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.415+0000 7f4eab7fe640 1 --2- 172.21.15.107:0/198803015 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55eecebb1d70 0x55eecebb4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:14.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4e8affd640 10 client.16827 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:14.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4eab7fe640 1 --2- 172.21.15.107:0/198803015 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55eecebb1d70 0x55eecebb4190 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=991e03de9ead764 server_cookie=beb8d8d9b8690dd7 in_seq=0 out_seq=0 2024-05-09T08:37:14.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4e8affd640 1 -- 172.21.15.107:0/198803015 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eecea1a820 con 0x55eecebb1d70 2024-05-09T08:37:14.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4e8affd640 10 client.16827 handle_command_reply: tid=0 2024-05-09T08:37:14.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4eb500f740 1 client.16827 shutdown 2024-05-09T08:37:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4eb500f740 1 -- 172.21.15.107:0/198803015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb3b320 msgr2=0x55eeceb34fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4eb500f740 1 --2- 172.21.15.107:0/198803015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb3b320 0x55eeceb34fd0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4009cd0 tx=0x7f4ea40095c0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4eb500f740 1 -- 172.21.15.107:0/198803015 shutdown_connections 2024-05-09T08:37:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4eb500f740 1 --2- 172.21.15.107:0/198803015 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55eecebb1d70 0x55eecebb4190 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4eb500f740 1 --2- 172.21.15.107:0/198803015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eeceb3b320 0x55eeceb34fd0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4eb500f740 1 --2- 172.21.15.107:0/198803015 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eeceb30c50 0x55eeceb34a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4e8affd640 0 client.16827 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4eb500f740 1 --2- 172.21.15.107:0/198803015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eeceb30330 0x55eeceb43670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4eb500f740 1 -- 172.21.15.107:0/198803015 >> 172.21.15.107:0/198803015 conn(0x55eeceb2c650 msgr2=0x55eeceb2b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4eb500f740 1 -- 172.21.15.107:0/198803015 shutdown_connections 2024-05-09T08:37:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4eb500f740 1 -- 172.21.15.107:0/198803015 wait complete. 2024-05-09T08:37:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.416+0000 7f4eb500f740 20 client.16827 trim_cache size 0 max 16384 2024-05-09T08:37:14.422 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:37:14.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.417+0000 7f4eb222c640 1 -- 172.21.15.107:0/1249897619 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4e940a2d00 msgr2=0x7f4e940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.417+0000 7f4eb222c640 1 --2- 172.21.15.107:0/1249897619 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4e940a2d00 0x7f4e940a51c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4005e30 tx=0x7f4ea402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.417+0000 7f4eb222c640 1 -- 172.21.15.107:0/1249897619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4eac1021b0 msgr2=0x7f4eac19a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.417+0000 7f4eb222c640 1 --2- 172.21.15.107:0/1249897619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4eac1021b0 0x7f4eac19a760 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000d770 tx=0x7f4ea000dc30 comp rx=0 tx=0).stop 2024-05-09T08:37:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.417+0000 7f4eb222c640 1 -- 172.21.15.107:0/1249897619 shutdown_connections 2024-05-09T08:37:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.417+0000 7f4eb222c640 1 --2- 172.21.15.107:0/1249897619 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4e940a2d00 0x7f4e940a51c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.417+0000 7f4eb222c640 1 --2- 172.21.15.107:0/1249897619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4eac107800 0x7f4eac19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.417+0000 7f4eb222c640 1 --2- 172.21.15.107:0/1249897619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4eac1021b0 0x7f4eac19a760 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.417+0000 7f4eb222c640 1 --2- 172.21.15.107:0/1249897619 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4eac1017f0 0x7f4eac19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.417+0000 7f4eb222c640 1 -- 172.21.15.107:0/1249897619 >> 172.21.15.107:0/1249897619 conn(0x7f4eac0fd2a0 msgr2=0x7f4eac10cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.417+0000 7f4eb222c640 1 -- 172.21.15.107:0/1249897619 shutdown_connections 2024-05-09T08:37:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:14.417+0000 7f4eb222c640 1 -- 172.21.15.107:0/1249897619 wait complete. 2024-05-09T08:37:15.428 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:37:15.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:37:15.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:37:15.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.555+0000 7f41899b6640 1 -- 172.21.15.107:0/2169411028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41841010b0 msgr2=0x7f418410d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.555+0000 7f41899b6640 1 --2- 172.21.15.107:0/2169411028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41841010b0 0x7f418410d5c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f417c009cd0 tx=0x7f417c02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.555+0000 7f41899b6640 1 -- 172.21.15.107:0/2169411028 shutdown_connections 2024-05-09T08:37:15.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.555+0000 7f41899b6640 1 --2- 172.21.15.107:0/2169411028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41841010b0 0x7f418410d5c0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.555+0000 7f41899b6640 1 --2- 172.21.15.107:0/2169411028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41841006f0 0x7f4184100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.555+0000 7f41899b6640 1 --2- 172.21.15.107:0/2169411028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4184106700 0x7f4184106ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.555+0000 7f41899b6640 1 -- 172.21.15.107:0/2169411028 >> 172.21.15.107:0/2169411028 conn(0x7f41840777d0 msgr2=0x7f4184077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:15.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.555+0000 7f41899b6640 1 -- 172.21.15.107:0/2169411028 shutdown_connections 2024-05-09T08:37:15.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.555+0000 7f41899b6640 1 -- 172.21.15.107:0/2169411028 wait complete. 2024-05-09T08:37:15.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.555+0000 7f41899b6640 1 Processor -- start 2024-05-09T08:37:15.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.555+0000 7f41899b6640 1 -- start start 2024-05-09T08:37:15.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.555+0000 7f41899b6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41841006f0 0x7f4184198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41899b6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41841010b0 0x7f4184199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41899b6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4184106700 0x7f41841a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41899b6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4184110980 con 0x7f4184106700 2024-05-09T08:37:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41899b6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4184110800 con 0x7f41841010b0 2024-05-09T08:37:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41899b6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4184110b00 con 0x7f41841006f0 2024-05-09T08:37:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41837fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4184106700 0x7f41841a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41837fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4184106700 0x7f41841a3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50426/0 (socket says 172.21.15.107:50426) 2024-05-09T08:37:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41837fe640 1 -- 172.21.15.107:0/588813409 learned_addr learned my addr 172.21.15.107:0/588813409 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f417bfff640 1 --2- 172.21.15.107:0/588813409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41841010b0 0x7f4184199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f4182ffd640 1 --2- 172.21.15.107:0/588813409 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41841006f0 0x7f4184198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41837fe640 1 -- 172.21.15.107:0/588813409 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41841006f0 msgr2=0x7f4184198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41837fe640 1 --2- 172.21.15.107:0/588813409 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41841006f0 0x7f4184198dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41837fe640 1 -- 172.21.15.107:0/588813409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41841010b0 msgr2=0x7f4184199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41837fe640 1 --2- 172.21.15.107:0/588813409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41841010b0 0x7f4184199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41837fe640 1 -- 172.21.15.107:0/588813409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41841a4380 con 0x7f4184106700 2024-05-09T08:37:15.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f417bfff640 1 --2- 172.21.15.107:0/588813409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41841010b0 0x7f4184199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:15.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f4182ffd640 1 --2- 172.21.15.107:0/588813409 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41841006f0 0x7f4184198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:15.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41837fe640 1 --2- 172.21.15.107:0/588813409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4184106700 0x7f41841a3e40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f417c005cc0 tx=0x7f417c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:15.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41889b4640 1 -- 172.21.15.107:0/588813409 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417c004450 con 0x7f4184106700 2024-05-09T08:37:15.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41889b4640 1 -- 172.21.15.107:0/588813409 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f417c031c90 con 0x7f4184106700 2024-05-09T08:37:15.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41899b6640 1 -- 172.21.15.107:0/588813409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41841a4550 con 0x7f4184106700 2024-05-09T08:37:15.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.556+0000 7f41889b4640 1 -- 172.21.15.107:0/588813409 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417c03b400 con 0x7f4184106700 2024-05-09T08:37:15.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.557+0000 7f41899b6640 1 -- 172.21.15.107:0/588813409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41841a4a10 con 0x7f4184106700 2024-05-09T08:37:15.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.559+0000 7f41889b4640 1 -- 172.21.15.107:0/588813409 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f417c004680 con 0x7f4184106700 2024-05-09T08:37:15.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.559+0000 7f418c79b740 1 Processor -- start 2024-05-09T08:37:15.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.559+0000 7f418c79b740 1 -- start start 2024-05-09T08:37:15.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f418c79b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 0x55b83df813f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f418c79b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df819c0 0x55b83df81e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f418c79b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df90b50 0x55b83df92f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f418c79b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b83dee6990 con 0x55b83df90b50 2024-05-09T08:37:15.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f418c79b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b83de8e5b0 con 0x55b83df819c0 2024-05-09T08:37:15.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f418c79b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55b83dee2950 con 0x55b83df830b0 2024-05-09T08:37:15.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f417bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df90b50 0x55b83df92f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f417bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df90b50 0x55b83df92f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50432/0 (socket says 172.21.15.107:50432) 2024-05-09T08:37:15.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f41889b4640 1 --2- 172.21.15.107:0/588813409 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f41700a2d00 0x7f41700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f417bfff640 1 -- 172.21.15.107:0/3682558093 learned_addr learned my addr 172.21.15.107:0/3682558093 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:15.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f417bfff640 1 --2- 172.21.15.107:0/3682558093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 0x55b83df813f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f4182ffd640 1 --2- 172.21.15.107:0/3682558093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df819c0 0x55b83df81e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f41837fe640 1 --2- 172.21.15.107:0/588813409 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f41700a2d00 0x7f41700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f41889b4640 1 -- 172.21.15.107:0/588813409 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f417c02e030 con 0x7f4184106700 2024-05-09T08:37:15.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f417bfff640 1 -- 172.21.15.107:0/3682558093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 msgr2=0x55b83df813f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:15.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f417bfff640 1 --2- 172.21.15.107:0/3682558093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 0x55b83df813f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f417bfff640 1 -- 172.21.15.107:0/3682558093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df819c0 msgr2=0x55b83df81e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:15.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f417bfff640 1 --2- 172.21.15.107:0/3682558093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df819c0 0x55b83df81e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f417bfff640 1 -- 172.21.15.107:0/3682558093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b83dcc27e0 con 0x55b83df90b50 2024-05-09T08:37:15.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.560+0000 7f417bfff640 1 --2- 172.21.15.107:0/3682558093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df90b50 0x55b83df92f40 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f416c014a30 tx=0x7f416c00c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8612c71bdd52f584 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:15.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.561+0000 7f4179ffb640 1 -- 172.21.15.107:0/3682558093 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c0142d0 con 0x55b83df90b50 2024-05-09T08:37:15.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.561+0000 7f41837fe640 1 --2- 172.21.15.107:0/588813409 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f41700a2d00 0x7f41700a51c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f417c0edf20 tx=0x7f417c04c000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:15.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.561+0000 7f4179ffb640 1 -- 172.21.15.107:0/3682558093 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f416c00f820 con 0x55b83df90b50 2024-05-09T08:37:15.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.561+0000 7f4179ffb640 1 -- 172.21.15.107:0/3682558093 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c003850 con 0x55b83df90b50 2024-05-09T08:37:15.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.561+0000 7f418c79b740 1 -- 172.21.15.107:0/3682558093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df90b50 msgr2=0x55b83df92f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.561+0000 7f418c79b740 1 --2- 172.21.15.107:0/3682558093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df90b50 0x55b83df92f40 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f416c014a30 tx=0x7f416c00c570 comp rx=0 tx=0).stop 2024-05-09T08:37:15.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.561+0000 7f418c79b740 1 -- 172.21.15.107:0/3682558093 shutdown_connections 2024-05-09T08:37:15.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.561+0000 7f418c79b740 1 --2- 172.21.15.107:0/3682558093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df90b50 0x55b83df92f40 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.561+0000 7f418c79b740 1 --2- 172.21.15.107:0/3682558093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df819c0 0x55b83df81e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.561+0000 7f418c79b740 1 --2- 172.21.15.107:0/3682558093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 0x55b83df813f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.561+0000 7f418c79b740 1 -- 172.21.15.107:0/3682558093 >> 172.21.15.107:0/3682558093 conn(0x55b83df4a0a0 msgr2=0x55b83df7df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:15.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.561+0000 7f418c79b740 1 -- 172.21.15.107:0/3682558093 shutdown_connections 2024-05-09T08:37:15.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.561+0000 7f418c79b740 1 -- 172.21.15.107:0/3682558093 wait complete. 2024-05-09T08:37:15.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.562+0000 7f418c79b740 1 -- start start 2024-05-09T08:37:15.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.562+0000 7f418c79b740 1 Processor -- start 2024-05-09T08:37:15.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.562+0000 7f418c79b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df819c0 0x55b83df8da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.562+0000 7f418c79b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 0x55b83df8dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.562+0000 7f418c79b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df90b50 0x55b83df8e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.562+0000 7f418c79b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b83df93980 con 0x55b83df819c0 2024-05-09T08:37:15.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.562+0000 7f418c79b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b83dea1b40 con 0x55b83df90b50 2024-05-09T08:37:15.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.562+0000 7f418c79b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55b83df93b00 con 0x55b83df830b0 2024-05-09T08:37:15.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.562+0000 7f418c79b740 10 client.0 _ll_register_callbacks cb 0x55b83de960f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:15.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.562+0000 7f417bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df819c0 0x55b83df8da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.562+0000 7f4182ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 0x55b83df8dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.562+0000 7f417bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df819c0 0x55b83df8da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50434/0 (socket says 172.21.15.107:50434) 2024-05-09T08:37:15.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.562+0000 7f417bfff640 1 -- 172.21.15.107:0/2210001855 learned_addr learned my addr 172.21.15.107:0/2210001855 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:15.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.563+0000 7f417bfff640 1 --2- 172.21.15.107:0/2210001855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df90b50 0x55b83df8e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.563+0000 7f417bfff640 1 -- 172.21.15.107:0/2210001855 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 msgr2=0x55b83df8dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:15.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.563+0000 7f417bfff640 1 --2- 172.21.15.107:0/2210001855 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 0x55b83df8dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.563+0000 7f417bfff640 1 -- 172.21.15.107:0/2210001855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df90b50 msgr2=0x55b83df8e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:15.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.563+0000 7f417bfff640 1 --2- 172.21.15.107:0/2210001855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df90b50 0x55b83df8e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.563+0000 7f417bfff640 1 -- 172.21.15.107:0/2210001855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b83dd22f70 con 0x55b83df819c0 2024-05-09T08:37:15.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.563+0000 7f4182ffd640 1 --2- 172.21.15.107:0/2210001855 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 0x55b83df8dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:15.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.563+0000 7f417bfff640 1 --2- 172.21.15.107:0/2210001855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df819c0 0x55b83df8da90 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f416c00d010 tx=0x7f416c00cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8513fccbb24d074 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:15.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.563+0000 7f4162ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:15.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.563+0000 7f4162ffd640 1 -- 172.21.15.107:0/2210001855 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c005050 con 0x55b83df819c0 2024-05-09T08:37:15.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.563+0000 7f4162ffd640 1 -- 172.21.15.107:0/2210001855 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f416c0051f0 con 0x55b83df819c0 2024-05-09T08:37:15.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.563+0000 7f418c79b740 1 -- 172.21.15.107:0/2210001855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f416c00f820 con 0x55b83df819c0 2024-05-09T08:37:15.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.563+0000 7f4162ffd640 1 -- 172.21.15.107:0/2210001855 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f416c0053f0 con 0x55b83df819c0 2024-05-09T08:37:15.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.563+0000 7f418c79b740 10 client.16839 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.563+0000 7f418c79b740 1 -- 172.21.15.107:0/2210001855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b83dbba170 con 0x55b83df819c0 2024-05-09T08:37:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.563+0000 7f4162ffd640 1 -- 172.21.15.107:0/2210001855 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f416c00f820 con 0x55b83df819c0 2024-05-09T08:37:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.564+0000 7f4162ffd640 1 -- 172.21.15.107:0/2210001855 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f416c012450 con 0x55b83df819c0 2024-05-09T08:37:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.564+0000 7f418c79b740 10 client.16839 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.564+0000 7f418c79b740 10 client.16839 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.564+0000 7f418c79b740 20 client.16839 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.564+0000 7f418c79b740 1 --2- 172.21.15.107:0/2210001855 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b83dfa0b50 0x55b83dfa2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.564+0000 7f418c79b740 4 client.16839 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.564+0000 7f418c79b740 1 -- 172.21.15.107:0/2210001855 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b83def40c0 con 0x55b83dfa0b50 2024-05-09T08:37:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.564+0000 7f4182ffd640 1 --2- 172.21.15.107:0/2210001855 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b83dfa0b50 0x55b83dfa2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.564+0000 7f4182ffd640 1 --2- 172.21.15.107:0/2210001855 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b83dfa0b50 0x55b83dfa2f40 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bb7abc940018be5 server_cookie=f3334ca00d0dd1d9 in_seq=0 out_seq=0 2024-05-09T08:37:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.564+0000 7f4162ffd640 10 client.16839 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.565+0000 7f4162ffd640 1 -- 172.21.15.107:0/2210001855 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b83def40c0 con 0x55b83dfa0b50 2024-05-09T08:37:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.565+0000 7f4162ffd640 10 client.16839 handle_command_reply: tid=0 2024-05-09T08:37:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.565+0000 7f418c79b740 1 client.16839 shutdown 2024-05-09T08:37:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.565+0000 7f418c79b740 1 -- 172.21.15.107:0/2210001855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df819c0 msgr2=0x55b83df8da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.565+0000 7f418c79b740 1 --2- 172.21.15.107:0/2210001855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df819c0 0x55b83df8da90 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f416c00d010 tx=0x7f416c00cd60 comp rx=0 tx=0).stop 2024-05-09T08:37:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.565+0000 7f418c79b740 1 -- 172.21.15.107:0/2210001855 shutdown_connections 2024-05-09T08:37:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.565+0000 7f418c79b740 1 --2- 172.21.15.107:0/2210001855 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b83dfa0b50 0x55b83dfa2f40 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.565+0000 7f418c79b740 1 --2- 172.21.15.107:0/2210001855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df90b50 0x55b83df8e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.565+0000 7f418c79b740 1 --2- 172.21.15.107:0/2210001855 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 0x55b83df8dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.565+0000 7f418c79b740 1 --2- 172.21.15.107:0/2210001855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df819c0 0x55b83df8da90 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.565+0000 7f418c79b740 1 -- 172.21.15.107:0/2210001855 >> 172.21.15.107:0/2210001855 conn(0x55b83df4a0a0 msgr2=0x55b83df83d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.565+0000 7f4162ffd640 0 client.16839 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:15.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.565+0000 7f418c79b740 1 -- 172.21.15.107:0/2210001855 shutdown_connections 2024-05-09T08:37:15.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.565+0000 7f418c79b740 1 -- 172.21.15.107:0/2210001855 wait complete. 2024-05-09T08:37:15.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.566+0000 7f418c79b740 20 client.16839 trim_cache size 0 max 16384 2024-05-09T08:37:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.587+0000 7f418c79b740 1 Processor -- start 2024-05-09T08:37:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.587+0000 7f418c79b740 1 -- start start 2024-05-09T08:37:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f418c79b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df819c0 0x55b83df9c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f418c79b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 0x55b83df7ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f418c79b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df90b50 0x55b83dfa8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f418c79b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f41841af060 con 0x55b83df819c0 2024-05-09T08:37:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f418c79b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f41841ac930 con 0x55b83df90b50 2024-05-09T08:37:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f418c79b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f41841aa780 con 0x55b83df830b0 2024-05-09T08:37:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f417bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df819c0 0x55b83df9c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f417bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df819c0 0x55b83df9c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50436/0 (socket says 172.21.15.107:50436) 2024-05-09T08:37:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f417bfff640 1 -- 172.21.15.107:0/2930920635 learned_addr learned my addr 172.21.15.107:0/2930920635 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f4182ffd640 1 --2- 172.21.15.107:0/2930920635 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 0x55b83df7ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f417bfff640 1 --2- 172.21.15.107:0/2930920635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df90b50 0x55b83dfa8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f417bfff640 1 -- 172.21.15.107:0/2930920635 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 msgr2=0x55b83df7ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f417bfff640 1 --2- 172.21.15.107:0/2930920635 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 0x55b83df7ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f417bfff640 1 -- 172.21.15.107:0/2930920635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df90b50 msgr2=0x55b83dfa8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f417bfff640 1 --2- 172.21.15.107:0/2930920635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df90b50 0x55b83dfa8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f417bfff640 1 -- 172.21.15.107:0/2930920635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b83df7c470 con 0x55b83df819c0 2024-05-09T08:37:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f4182ffd640 1 --2- 172.21.15.107:0/2930920635 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 0x55b83df7ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f417bfff640 1 --2- 172.21.15.107:0/2930920635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df819c0 0x55b83df9c880 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f416c000c00 tx=0x7f416c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218f315979a57472 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.588+0000 7f41797fa640 1 -- 172.21.15.107:0/2930920635 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c00f820 con 0x55b83df819c0 2024-05-09T08:37:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.589+0000 7f41797fa640 1 -- 172.21.15.107:0/2930920635 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f416c00ca10 con 0x55b83df819c0 2024-05-09T08:37:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.589+0000 7f41797fa640 1 -- 172.21.15.107:0/2930920635 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c00fe40 con 0x55b83df819c0 2024-05-09T08:37:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.589+0000 7f418c79b740 1 -- 172.21.15.107:0/2930920635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df819c0 msgr2=0x55b83df9c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.589+0000 7f418c79b740 1 --2- 172.21.15.107:0/2930920635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df819c0 0x55b83df9c880 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f416c000c00 tx=0x7f416c005fd0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.589+0000 7f418c79b740 1 -- 172.21.15.107:0/2930920635 shutdown_connections 2024-05-09T08:37:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.589+0000 7f418c79b740 1 --2- 172.21.15.107:0/2930920635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df90b50 0x55b83dfa8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.589+0000 7f418c79b740 1 --2- 172.21.15.107:0/2930920635 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df830b0 0x55b83df7ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.589+0000 7f418c79b740 1 --2- 172.21.15.107:0/2930920635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df819c0 0x55b83df9c880 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.589+0000 7f418c79b740 1 -- 172.21.15.107:0/2930920635 >> 172.21.15.107:0/2930920635 conn(0x55b83df4a0a0 msgr2=0x55b83df8cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.590+0000 7f418c79b740 1 -- 172.21.15.107:0/2930920635 shutdown_connections 2024-05-09T08:37:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.590+0000 7f418c79b740 1 -- 172.21.15.107:0/2930920635 wait complete. 2024-05-09T08:37:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.590+0000 7f418c79b740 1 -- start start 2024-05-09T08:37:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.590+0000 7f418c79b740 1 Processor -- start 2024-05-09T08:37:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.590+0000 7f418c79b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df819c0 0x55b83df94990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.590+0000 7f418c79b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df830b0 0x55b83df94ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.590+0000 7f418c79b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df90b50 0x55b83df95410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.590+0000 7f418c79b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f41841aadf0 con 0x55b83df830b0 2024-05-09T08:37:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.590+0000 7f418c79b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f41841ac260 con 0x55b83df819c0 2024-05-09T08:37:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.590+0000 7f418c79b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f41841ab4c0 con 0x55b83df90b50 2024-05-09T08:37:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.590+0000 7f418c79b740 10 client.0 _ll_register_callbacks cb 0x7f416c014470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.590+0000 7f4182ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df830b0 0x55b83df94ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f4182ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df830b0 0x55b83df94ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50446/0 (socket says 172.21.15.107:50446) 2024-05-09T08:37:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f4182ffd640 1 -- 172.21.15.107:0/1438776506 learned_addr learned my addr 172.21.15.107:0/1438776506 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f417bfff640 1 --2- 172.21.15.107:0/1438776506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df90b50 0x55b83df95410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f417bfff640 1 --2- 172.21.15.107:0/1438776506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df819c0 0x55b83df94990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f4182ffd640 1 -- 172.21.15.107:0/1438776506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df90b50 msgr2=0x55b83df95410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f4182ffd640 1 --2- 172.21.15.107:0/1438776506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df90b50 0x55b83df95410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f4182ffd640 1 -- 172.21.15.107:0/1438776506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df819c0 msgr2=0x55b83df94990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f4182ffd640 1 --2- 172.21.15.107:0/1438776506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df819c0 0x55b83df94990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f4182ffd640 1 -- 172.21.15.107:0/1438776506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b83e0060a0 con 0x55b83df830b0 2024-05-09T08:37:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f417bfff640 1 --2- 172.21.15.107:0/1438776506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df819c0 0x55b83df94990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f4182ffd640 1 --2- 172.21.15.107:0/1438776506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df830b0 0x55b83df94ed0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f417400c780 tx=0x7f4174011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a8bd00d3813ac7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f4162ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f4162ffd640 1 -- 172.21.15.107:0/1438776506 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4174004320 con 0x55b83df830b0 2024-05-09T08:37:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f4162ffd640 1 -- 172.21.15.107:0/1438776506 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f417401d420 con 0x55b83df830b0 2024-05-09T08:37:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f418c79b740 1 -- 172.21.15.107:0/1438776506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f416c00ca10 con 0x55b83df830b0 2024-05-09T08:37:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f4162ffd640 1 -- 172.21.15.107:0/1438776506 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f417402c050 con 0x55b83df830b0 2024-05-09T08:37:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f418c79b740 10 client.16845 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f418c79b740 1 -- 172.21.15.107:0/1438776506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b83e007450 con 0x55b83df830b0 2024-05-09T08:37:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.591+0000 7f4162ffd640 1 -- 172.21.15.107:0/1438776506 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f416c00ca10 con 0x55b83df830b0 2024-05-09T08:37:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.592+0000 7f4162ffd640 1 -- 172.21.15.107:0/1438776506 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f4174003610 con 0x55b83df830b0 2024-05-09T08:37:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.592+0000 7f418c79b740 10 client.16845 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.592+0000 7f418c79b740 10 client.16845 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.592+0000 7f418c79b740 20 client.16845 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.592+0000 7f418c79b740 1 --2- 172.21.15.107:0/1438776506 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b83e007c50 0x55b83e00a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.592+0000 7f418c79b740 4 client.16845 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:37:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.592+0000 7f418c79b740 1 -- 172.21.15.107:0/1438776506 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b83deee410 con 0x55b83e007c50 2024-05-09T08:37:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.592+0000 7f417bfff640 1 --2- 172.21.15.107:0/1438776506 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b83e007c50 0x55b83e00a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.592+0000 7f4162ffd640 10 client.16845 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.592+0000 7f417bfff640 1 --2- 172.21.15.107:0/1438776506 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b83e007c50 0x55b83e00a070 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c83ada50febdec4a server_cookie=b00d1f829aeda4ff in_seq=0 out_seq=0 2024-05-09T08:37:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.593+0000 7f4162ffd640 1 -- 172.21.15.107:0/1438776506 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b83deee410 con 0x55b83e007c50 2024-05-09T08:37:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.593+0000 7f4162ffd640 10 client.16845 handle_command_reply: tid=0 2024-05-09T08:37:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.593+0000 7f418c79b740 1 client.16845 shutdown 2024-05-09T08:37:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.594+0000 7f418c79b740 1 -- 172.21.15.107:0/1438776506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df830b0 msgr2=0x55b83df94ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.594+0000 7f418c79b740 1 --2- 172.21.15.107:0/1438776506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df830b0 0x55b83df94ed0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f417400c780 tx=0x7f4174011d90 comp rx=0 tx=0).stop 2024-05-09T08:37:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.594+0000 7f418c79b740 1 -- 172.21.15.107:0/1438776506 shutdown_connections 2024-05-09T08:37:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.594+0000 7f418c79b740 1 --2- 172.21.15.107:0/1438776506 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b83e007c50 0x55b83e00a070 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.594+0000 7f418c79b740 1 --2- 172.21.15.107:0/1438776506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b83df90b50 0x55b83df95410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.594+0000 7f418c79b740 1 --2- 172.21.15.107:0/1438776506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b83df830b0 0x55b83df94ed0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.594+0000 7f418c79b740 1 --2- 172.21.15.107:0/1438776506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b83df819c0 0x55b83df94990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.594+0000 7f418c79b740 1 -- 172.21.15.107:0/1438776506 >> 172.21.15.107:0/1438776506 conn(0x55b83df4a0a0 msgr2=0x55b83df83490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.594+0000 7f4162ffd640 0 client.16845 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.594+0000 7f418c79b740 1 -- 172.21.15.107:0/1438776506 shutdown_connections 2024-05-09T08:37:15.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.594+0000 7f418c79b740 1 -- 172.21.15.107:0/1438776506 wait complete. 2024-05-09T08:37:15.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.594+0000 7f418c79b740 20 client.16845 trim_cache size 0 max 16384 2024-05-09T08:37:15.601 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:37:15.601 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:37:15.601 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "99f0370a-c553-401c-ac78-d98d5ffabeab", 2024-05-09T08:37:15.601 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:37:15.601 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:37:15.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.595+0000 7f41899b6640 1 -- 172.21.15.107:0/588813409 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f41700a2d00 msgr2=0x7f41700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.595+0000 7f41899b6640 1 --2- 172.21.15.107:0/588813409 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f41700a2d00 0x7f41700a51c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f417c0edf20 tx=0x7f417c04c000 comp rx=0 tx=0).stop 2024-05-09T08:37:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.595+0000 7f41899b6640 1 -- 172.21.15.107:0/588813409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4184106700 msgr2=0x7f41841a3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.595+0000 7f41899b6640 1 --2- 172.21.15.107:0/588813409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4184106700 0x7f41841a3e40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f417c005cc0 tx=0x7f417c003c20 comp rx=0 tx=0).stop 2024-05-09T08:37:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.595+0000 7f41899b6640 1 -- 172.21.15.107:0/588813409 shutdown_connections 2024-05-09T08:37:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.595+0000 7f41899b6640 1 --2- 172.21.15.107:0/588813409 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f41700a2d00 0x7f41700a51c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.595+0000 7f41899b6640 1 --2- 172.21.15.107:0/588813409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4184106700 0x7f41841a3e40 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.595+0000 7f41899b6640 1 --2- 172.21.15.107:0/588813409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41841010b0 0x7f4184199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.595+0000 7f41899b6640 1 --2- 172.21.15.107:0/588813409 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41841006f0 0x7f4184198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.595+0000 7f41899b6640 1 -- 172.21.15.107:0/588813409 >> 172.21.15.107:0/588813409 conn(0x7f41840777d0 msgr2=0x7f41840fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.595+0000 7f41899b6640 1 -- 172.21.15.107:0/588813409 shutdown_connections 2024-05-09T08:37:15.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.595+0000 7f41899b6640 1 -- 172.21.15.107:0/588813409 wait complete. 2024-05-09T08:37:15.605 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:37:15.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:37:15.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.725+0000 7f5149aee640 1 -- 172.21.15.107:0/3914386043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5144101800 msgr2=0x7f5144101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.725+0000 7f5149aee640 1 --2- 172.21.15.107:0/3914386043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5144101800 0x7f5144101c80 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f5138009930 tx=0x7f513802a420 comp rx=0 tx=0).stop 2024-05-09T08:37:15.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.725+0000 7f5149aee640 1 -- 172.21.15.107:0/3914386043 shutdown_connections 2024-05-09T08:37:15.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.725+0000 7f5149aee640 1 --2- 172.21.15.107:0/3914386043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51441021c0 0x7f514410e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.725+0000 7f5149aee640 1 --2- 172.21.15.107:0/3914386043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5144101800 0x7f5144101c80 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.725+0000 7f5149aee640 1 --2- 172.21.15.107:0/3914386043 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5144107810 0x7f5144107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.725+0000 7f5149aee640 1 -- 172.21.15.107:0/3914386043 >> 172.21.15.107:0/3914386043 conn(0x7f5144078c80 msgr2=0x7f5144079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:15.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.725+0000 7f5149aee640 1 -- 172.21.15.107:0/3914386043 shutdown_connections 2024-05-09T08:37:15.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.725+0000 7f5149aee640 1 -- 172.21.15.107:0/3914386043 wait complete. 2024-05-09T08:37:15.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.726+0000 7f5149aee640 1 Processor -- start 2024-05-09T08:37:15.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.726+0000 7f5149aee640 1 -- start start 2024-05-09T08:37:15.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.726+0000 7f5149aee640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5144101800 0x7f51441980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.726+0000 7f5149aee640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51441021c0 0x7f51441985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.726+0000 7f5149aee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5144107810 0x7f514419c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.726+0000 7f5149aee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5144111ad0 con 0x7f5144107810 2024-05-09T08:37:15.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.726+0000 7f5149aee640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5144111950 con 0x7f51441021c0 2024-05-09T08:37:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.726+0000 7f5149aee640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5144111c50 con 0x7f5144101800 2024-05-09T08:37:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.727+0000 7f5143fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5144107810 0x7f514419c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.727+0000 7f5143fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5144107810 0x7f514419c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50464/0 (socket says 172.21.15.107:50464) 2024-05-09T08:37:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.727+0000 7f5143fff640 1 -- 172.21.15.107:0/1943682405 learned_addr learned my addr 172.21.15.107:0/1943682405 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.727+0000 7f51437fe640 1 --2- 172.21.15.107:0/1943682405 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5144101800 0x7f51441980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.727+0000 7f5142ffd640 1 --2- 172.21.15.107:0/1943682405 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51441021c0 0x7f51441985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.727+0000 7f5143fff640 1 -- 172.21.15.107:0/1943682405 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5144101800 msgr2=0x7f51441980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.727+0000 7f5143fff640 1 --2- 172.21.15.107:0/1943682405 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5144101800 0x7f51441980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.727+0000 7f5143fff640 1 -- 172.21.15.107:0/1943682405 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51441021c0 msgr2=0x7f51441985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.727+0000 7f5143fff640 1 --2- 172.21.15.107:0/1943682405 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51441021c0 0x7f51441985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.727+0000 7f5143fff640 1 -- 172.21.15.107:0/1943682405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514419d090 con 0x7f5144107810 2024-05-09T08:37:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.727+0000 7f5143fff640 1 --2- 172.21.15.107:0/1943682405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5144107810 0x7f514419c960 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f513c00eaa0 tx=0x7f513c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.727+0000 7f5140ff9640 1 -- 172.21.15.107:0/1943682405 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c00cdb0 con 0x7f5144107810 2024-05-09T08:37:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.727+0000 7f5149aee640 1 -- 172.21.15.107:0/1943682405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f514419d380 con 0x7f5144107810 2024-05-09T08:37:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.727+0000 7f5140ff9640 1 -- 172.21.15.107:0/1943682405 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f513c005020 con 0x7f5144107810 2024-05-09T08:37:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.727+0000 7f5140ff9640 1 -- 172.21.15.107:0/1943682405 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c0116d0 con 0x7f5144107810 2024-05-09T08:37:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.727+0000 7f5149aee640 1 -- 172.21.15.107:0/1943682405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51441a4c10 con 0x7f5144107810 2024-05-09T08:37:15.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.730+0000 7f5140ff9640 1 -- 172.21.15.107:0/1943682405 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f513c0051d0 con 0x7f5144107810 2024-05-09T08:37:15.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.730+0000 7f514c7fc740 1 Processor -- start 2024-05-09T08:37:15.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.730+0000 7f514c7fc740 1 -- start start 2024-05-09T08:37:15.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.730+0000 7f514c7fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba8d160 0x5636cba8d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.730+0000 7f514c7fc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8da80 0x5636cba9c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.730+0000 7f514c7fc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba9c680 0x5636cba9ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.730+0000 7f514c7fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5636cb9f2d00 con 0x5636cba8d160 2024-05-09T08:37:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.730+0000 7f514c7fc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5636cb99a4c0 con 0x5636cba8da80 2024-05-09T08:37:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.730+0000 7f514c7fc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5636cb9eecc0 con 0x5636cba9c680 2024-05-09T08:37:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.730+0000 7f5142ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba8d160 0x5636cba8d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.730+0000 7f5142ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba8d160 0x5636cba8d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50470/0 (socket says 172.21.15.107:50470) 2024-05-09T08:37:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.730+0000 7f5142ffd640 1 -- 172.21.15.107:0/3135789989 learned_addr learned my addr 172.21.15.107:0/3135789989 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.731+0000 7f51437fe640 1 --2- 172.21.15.107:0/3135789989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8da80 0x5636cba9c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.731+0000 7f5142ffd640 1 --2- 172.21.15.107:0/3135789989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba9c680 0x5636cba9ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.731+0000 7f5140ff9640 1 --2- 172.21.15.107:0/1943682405 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f512c0a2d00 0x7f512c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.731+0000 7f51437fe640 1 -- 172.21.15.107:0/3135789989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba9c680 msgr2=0x5636cba9ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.731+0000 7f51437fe640 1 --2- 172.21.15.107:0/3135789989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba9c680 0x5636cba9ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.731+0000 7f51437fe640 1 -- 172.21.15.107:0/3135789989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba8d160 msgr2=0x5636cba8d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.731+0000 7f51437fe640 1 --2- 172.21.15.107:0/3135789989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba8d160 0x5636cba8d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.731+0000 7f51437fe640 1 -- 172.21.15.107:0/3135789989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636cb7d07d0 con 0x5636cba8da80 2024-05-09T08:37:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.731+0000 7f5142ffd640 1 --2- 172.21.15.107:0/3135789989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba8d160 0x5636cba8d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.731+0000 7f5143fff640 1 --2- 172.21.15.107:0/1943682405 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f512c0a2d00 0x7f512c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.731+0000 7f5140ff9640 1 -- 172.21.15.107:0/1943682405 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f513c0cf920 con 0x7f5144107810 2024-05-09T08:37:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.731+0000 7f51437fe640 1 --2- 172.21.15.107:0/3135789989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8da80 0x5636cba9c140 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f513400aac0 tx=0x7f513400af80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a44a8a8ea275ae67 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.731+0000 7f5129ffb640 1 -- 172.21.15.107:0/3135789989 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5134004020 con 0x5636cba8da80 2024-05-09T08:37:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.731+0000 7f5129ffb640 1 -- 172.21.15.107:0/3135789989 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51340041c0 con 0x5636cba8da80 2024-05-09T08:37:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.731+0000 7f5129ffb640 1 -- 172.21.15.107:0/3135789989 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5134018680 con 0x5636cba8da80 2024-05-09T08:37:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.732+0000 7f514c7fc740 1 -- 172.21.15.107:0/3135789989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8da80 msgr2=0x5636cba9c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.732+0000 7f514c7fc740 1 --2- 172.21.15.107:0/3135789989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8da80 0x5636cba9c140 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f513400aac0 tx=0x7f513400af80 comp rx=0 tx=0).stop 2024-05-09T08:37:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.732+0000 7f5143fff640 1 --2- 172.21.15.107:0/1943682405 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f512c0a2d00 0x7f512c0a51c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f513c0ccdc0 tx=0x7f513c091020 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.732+0000 7f514c7fc740 1 -- 172.21.15.107:0/3135789989 shutdown_connections 2024-05-09T08:37:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.732+0000 7f514c7fc740 1 --2- 172.21.15.107:0/3135789989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba9c680 0x5636cba9ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.732+0000 7f514c7fc740 1 --2- 172.21.15.107:0/3135789989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8da80 0x5636cba9c140 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.732+0000 7f514c7fc740 1 --2- 172.21.15.107:0/3135789989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba8d160 0x5636cba8d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.732+0000 7f514c7fc740 1 -- 172.21.15.107:0/3135789989 >> 172.21.15.107:0/3135789989 conn(0x5636cba895a0 msgr2=0x5636cba89980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.732+0000 7f514c7fc740 1 -- 172.21.15.107:0/3135789989 shutdown_connections 2024-05-09T08:37:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.732+0000 7f514c7fc740 1 -- 172.21.15.107:0/3135789989 wait complete. 2024-05-09T08:37:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.732+0000 7f514c7fc740 1 -- start start 2024-05-09T08:37:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f514c7fc740 1 Processor -- start 2024-05-09T08:37:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f514c7fc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 0x5636cba99570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f514c7fc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba8da80 0x5636cba99ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f514c7fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba9c680 0x5636cba99ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f514c7fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5636cba9f4c0 con 0x5636cba9c680 2024-05-09T08:37:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f514c7fc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5636cb9adb90 con 0x5636cba8d160 2024-05-09T08:37:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f514c7fc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5636cba9f640 con 0x5636cba8da80 2024-05-09T08:37:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f514c7fc740 10 client.0 _ll_register_callbacks cb 0x5636cb9e9da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f5142ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 0x5636cba99570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f5142ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 0x5636cba99570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:48280/0 (socket says 172.21.15.107:48280) 2024-05-09T08:37:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f5142ffd640 1 -- 172.21.15.107:0/2612616012 learned_addr learned my addr 172.21.15.107:0/2612616012 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f51437fe640 1 --2- 172.21.15.107:0/2612616012 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba8da80 0x5636cba99ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f5142ffd640 1 --2- 172.21.15.107:0/2612616012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba9c680 0x5636cba99ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f5142ffd640 1 -- 172.21.15.107:0/2612616012 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba8da80 msgr2=0x5636cba99ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f5142ffd640 1 --2- 172.21.15.107:0/2612616012 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba8da80 0x5636cba99ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f5142ffd640 1 -- 172.21.15.107:0/2612616012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba9c680 msgr2=0x5636cba99ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f5142ffd640 1 --2- 172.21.15.107:0/2612616012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba9c680 0x5636cba99ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f5142ffd640 1 -- 172.21.15.107:0/2612616012 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636cb7063f0 con 0x5636cba8d160 2024-05-09T08:37:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.733+0000 7f51437fe640 1 --2- 172.21.15.107:0/2612616012 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba8da80 0x5636cba99ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.734+0000 7f5142ffd640 1 --2- 172.21.15.107:0/2612616012 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 0x5636cba99570 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f51380057c0 tx=0x7f5138040040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db5b93bd45161ec6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.734+0000 7f5122ffd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:37:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.734+0000 7f5122ffd640 1 -- 172.21.15.107:0/2612616012 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138035040 con 0x5636cba8d160 2024-05-09T08:37:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.734+0000 7f514c7fc740 1 -- 172.21.15.107:0/2612616012 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51340041c0 con 0x5636cba8d160 2024-05-09T08:37:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.734+0000 7f5122ffd640 1 -- 172.21.15.107:0/2612616012 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f513803eac0 con 0x5636cba8d160 2024-05-09T08:37:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.734+0000 7f5122ffd640 1 -- 172.21.15.107:0/2612616012 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5138054050 con 0x5636cba8d160 2024-05-09T08:37:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.734+0000 7f514c7fc740 10 client.26051 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.734+0000 7f514c7fc740 1 -- 172.21.15.107:0/2612616012 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5636cb6c60f0 con 0x5636cba8d160 2024-05-09T08:37:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.734+0000 7f5122ffd640 1 -- 172.21.15.107:0/2612616012 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51340041c0 con 0x5636cba8d160 2024-05-09T08:37:15.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.734+0000 7f5122ffd640 1 -- 172.21.15.107:0/2612616012 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f513804b440 con 0x5636cba8d160 2024-05-09T08:37:15.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.734+0000 7f514c7fc740 10 client.26051 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:15.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.734+0000 7f514c7fc740 10 client.26051 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:15.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.734+0000 7f514c7fc740 20 client.26051 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:15.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.734+0000 7f514c7fc740 1 --2- 172.21.15.107:0/2612616012 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5636cbaac9f0 0x5636cbaaee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.734+0000 7f514c7fc740 4 client.26051 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:15.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.734+0000 7f514c7fc740 1 -- 172.21.15.107:0/2612616012 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636cba00430 con 0x5636cbaac9f0 2024-05-09T08:37:15.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.735+0000 7f51437fe640 1 --2- 172.21.15.107:0/2612616012 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5636cbaac9f0 0x5636cbaaee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.735+0000 7f5122ffd640 10 client.26051 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:15.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.735+0000 7f51437fe640 1 --2- 172.21.15.107:0/2612616012 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5636cbaac9f0 0x5636cbaaee00 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc69331fcb94f5b4 server_cookie=4a7a0f24fd09a2b0 in_seq=0 out_seq=0 2024-05-09T08:37:15.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.736+0000 7f5122ffd640 1 -- 172.21.15.107:0/2612616012 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5636cba00430 con 0x5636cbaac9f0 2024-05-09T08:37:15.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.736+0000 7f5122ffd640 10 client.26051 handle_command_reply: tid=0 2024-05-09T08:37:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.736+0000 7f514c7fc740 1 client.26051 shutdown 2024-05-09T08:37:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.736+0000 7f514c7fc740 1 -- 172.21.15.107:0/2612616012 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 msgr2=0x5636cba99570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.736+0000 7f514c7fc740 1 --2- 172.21.15.107:0/2612616012 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 0x5636cba99570 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f51380057c0 tx=0x7f5138040040 comp rx=0 tx=0).stop 2024-05-09T08:37:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.736+0000 7f514c7fc740 1 -- 172.21.15.107:0/2612616012 shutdown_connections 2024-05-09T08:37:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.736+0000 7f514c7fc740 1 --2- 172.21.15.107:0/2612616012 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5636cbaac9f0 0x5636cbaaee00 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.736+0000 7f514c7fc740 1 --2- 172.21.15.107:0/2612616012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba9c680 0x5636cba99ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.736+0000 7f514c7fc740 1 --2- 172.21.15.107:0/2612616012 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba8da80 0x5636cba99ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.736+0000 7f5122ffd640 0 client.26051 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.736+0000 7f514c7fc740 1 --2- 172.21.15.107:0/2612616012 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 0x5636cba99570 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.736+0000 7f514c7fc740 1 -- 172.21.15.107:0/2612616012 >> 172.21.15.107:0/2612616012 conn(0x5636cba895a0 msgr2=0x5636cba8f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.736+0000 7f514c7fc740 1 -- 172.21.15.107:0/2612616012 shutdown_connections 2024-05-09T08:37:15.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.736+0000 7f514c7fc740 1 -- 172.21.15.107:0/2612616012 wait complete. 2024-05-09T08:37:15.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.736+0000 7f514c7fc740 20 client.26051 trim_cache size 0 max 16384 2024-05-09T08:37:15.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.757+0000 7f514c7fc740 1 Processor -- start 2024-05-09T08:37:15.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.757+0000 7f514c7fc740 1 -- start start 2024-05-09T08:37:15.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f514c7fc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 0x5636cbaa8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f514c7fc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba8da80 0x5636cba98e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f514c7fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba9c680 0x5636cbab46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f514c7fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f51441af2a0 con 0x5636cba9c680 2024-05-09T08:37:15.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f514c7fc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f51441acb70 con 0x5636cba8d160 2024-05-09T08:37:15.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f514c7fc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f51441aa9c0 con 0x5636cba8da80 2024-05-09T08:37:15.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f5142ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba9c680 0x5636cbab46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f5142ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba9c680 0x5636cbab46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50488/0 (socket says 172.21.15.107:50488) 2024-05-09T08:37:15.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f5142ffd640 1 -- 172.21.15.107:0/2689394352 learned_addr learned my addr 172.21.15.107:0/2689394352 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:15.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f5142ffd640 1 --2- 172.21.15.107:0/2689394352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 0x5636cbaa8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f51437fe640 1 --2- 172.21.15.107:0/2689394352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba8da80 0x5636cba98e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f5142ffd640 1 -- 172.21.15.107:0/2689394352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba8da80 msgr2=0x5636cba98e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:15.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f5142ffd640 1 --2- 172.21.15.107:0/2689394352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba8da80 0x5636cba98e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f5142ffd640 1 -- 172.21.15.107:0/2689394352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 msgr2=0x5636cbaa8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:15.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f5142ffd640 1 --2- 172.21.15.107:0/2689394352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 0x5636cbaa8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f5142ffd640 1 -- 172.21.15.107:0/2689394352 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636cba8e720 con 0x5636cba9c680 2024-05-09T08:37:15.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f5142ffd640 1 --2- 172.21.15.107:0/2689394352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba9c680 0x5636cbab46d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x5636cba9c5b0 tx=0x7f513802a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f71ff4aec978e96 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:15.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f51297fa640 1 -- 172.21.15.107:0/2689394352 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513804be60 con 0x5636cba9c680 2024-05-09T08:37:15.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.758+0000 7f51297fa640 1 -- 172.21.15.107:0/2689394352 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51380057f0 con 0x5636cba9c680 2024-05-09T08:37:15.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.759+0000 7f51297fa640 1 -- 172.21.15.107:0/2689394352 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138035040 con 0x5636cba9c680 2024-05-09T08:37:15.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.759+0000 7f514c7fc740 1 -- 172.21.15.107:0/2689394352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba9c680 msgr2=0x5636cbab46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.759+0000 7f514c7fc740 1 --2- 172.21.15.107:0/2689394352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba9c680 0x5636cbab46d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x5636cba9c5b0 tx=0x7f513802a420 comp rx=0 tx=0).stop 2024-05-09T08:37:15.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.759+0000 7f514c7fc740 1 -- 172.21.15.107:0/2689394352 shutdown_connections 2024-05-09T08:37:15.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.759+0000 7f514c7fc740 1 --2- 172.21.15.107:0/2689394352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba9c680 0x5636cbab46d0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.759+0000 7f514c7fc740 1 --2- 172.21.15.107:0/2689394352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba8da80 0x5636cba98e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.759+0000 7f514c7fc740 1 --2- 172.21.15.107:0/2689394352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 0x5636cbaa8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.759+0000 7f514c7fc740 1 -- 172.21.15.107:0/2689394352 >> 172.21.15.107:0/2689394352 conn(0x5636cba895a0 msgr2=0x5636cba8f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:15.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.759+0000 7f514c7fc740 1 -- 172.21.15.107:0/2689394352 shutdown_connections 2024-05-09T08:37:15.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.759+0000 7f514c7fc740 1 -- 172.21.15.107:0/2689394352 wait complete. 2024-05-09T08:37:15.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.760+0000 7f514c7fc740 1 -- start start 2024-05-09T08:37:15.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.760+0000 7f514c7fc740 1 Processor -- start 2024-05-09T08:37:15.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.760+0000 7f514c7fc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 0x5636cbaa0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.760+0000 7f514c7fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba8da80 0x5636cbaa08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.760+0000 7f514c7fc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba9c680 0x5636cbaa0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.760+0000 7f514c7fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f51441ab030 con 0x5636cba8da80 2024-05-09T08:37:15.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.760+0000 7f514c7fc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f51441ac4a0 con 0x5636cba8d160 2024-05-09T08:37:15.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.760+0000 7f514c7fc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f51441ab700 con 0x5636cba9c680 2024-05-09T08:37:15.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.760+0000 7f514c7fc740 10 client.0 _ll_register_callbacks cb 0x7f513803e540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:15.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.760+0000 7f51437fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba8da80 0x5636cbaa08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.760+0000 7f51437fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba8da80 0x5636cbaa08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50490/0 (socket says 172.21.15.107:50490) 2024-05-09T08:37:15.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.760+0000 7f51437fe640 1 -- 172.21.15.107:0/3260702176 learned_addr learned my addr 172.21.15.107:0/3260702176 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:15.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.760+0000 7f5142ffd640 1 --2- 172.21.15.107:0/3260702176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 0x5636cbaa0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.760+0000 7f5142ffd640 1 --2- 172.21.15.107:0/3260702176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba9c680 0x5636cbaa0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.761+0000 7f5142ffd640 1 -- 172.21.15.107:0/3260702176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba9c680 msgr2=0x5636cbaa0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:15.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.761+0000 7f5142ffd640 1 --2- 172.21.15.107:0/3260702176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba9c680 0x5636cbaa0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.761+0000 7f5142ffd640 1 -- 172.21.15.107:0/3260702176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba8da80 msgr2=0x5636cbaa08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:15.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.761+0000 7f5142ffd640 1 --2- 172.21.15.107:0/3260702176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba8da80 0x5636cbaa08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.761+0000 7f5142ffd640 1 -- 172.21.15.107:0/3260702176 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636cbb11f70 con 0x5636cba8d160 2024-05-09T08:37:15.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.761+0000 7f51437fe640 1 --2- 172.21.15.107:0/3260702176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba8da80 0x5636cbaa08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:15.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.761+0000 7f5142ffd640 1 --2- 172.21.15.107:0/3260702176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 0x5636cbaa0380 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f513804b960 tx=0x7f5138000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fbb82cf40247118 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:15.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.761+0000 7f5122ffd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:37:15.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.761+0000 7f5122ffd640 1 -- 172.21.15.107:0/3260702176 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513804be60 con 0x5636cba8d160 2024-05-09T08:37:15.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.761+0000 7f514c7fc740 1 -- 172.21.15.107:0/3260702176 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51380057f0 con 0x5636cba8d160 2024-05-09T08:37:15.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.761+0000 7f5122ffd640 1 -- 172.21.15.107:0/3260702176 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f513803eac0 con 0x5636cba8d160 2024-05-09T08:37:15.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.761+0000 7f5122ffd640 1 -- 172.21.15.107:0/3260702176 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f513803ecc0 con 0x5636cba8d160 2024-05-09T08:37:15.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.761+0000 7f514c7fc740 10 client.26057 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:15.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.761+0000 7f514c7fc740 1 -- 172.21.15.107:0/3260702176 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5636cbb13260 con 0x5636cba8d160 2024-05-09T08:37:15.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.761+0000 7f5122ffd640 1 -- 172.21.15.107:0/3260702176 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51380057f0 con 0x5636cba8d160 2024-05-09T08:37:15.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.762+0000 7f5122ffd640 1 -- 172.21.15.107:0/3260702176 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f513803e0b0 con 0x5636cba8d160 2024-05-09T08:37:15.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.762+0000 7f514c7fc740 10 client.26057 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:15.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.762+0000 7f514c7fc740 10 client.26057 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:15.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.762+0000 7f514c7fc740 20 client.26057 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:15.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.762+0000 7f514c7fc740 1 --2- 172.21.15.107:0/3260702176 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5636cbb13910 0x5636cbb15d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:15.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.762+0000 7f514c7fc740 4 client.26057 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:37:15.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.762+0000 7f514c7fc740 1 -- 172.21.15.107:0/3260702176 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5636cb9fa780 con 0x5636cbb13910 2024-05-09T08:37:15.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.762+0000 7f51437fe640 1 --2- 172.21.15.107:0/3260702176 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5636cbb13910 0x5636cbb15d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:15.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.762+0000 7f5122ffd640 10 client.26057 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:15.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.762+0000 7f51437fe640 1 --2- 172.21.15.107:0/3260702176 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5636cbb13910 0x5636cbb15d30 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87c49dc2db927887 server_cookie=d0cc7679fbbe13ee in_seq=0 out_seq=0 2024-05-09T08:37:15.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.763+0000 7f5122ffd640 1 -- 172.21.15.107:0/3260702176 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5636cb9fa780 con 0x5636cbb13910 2024-05-09T08:37:15.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.763+0000 7f5122ffd640 10 client.26057 handle_command_reply: tid=0 2024-05-09T08:37:15.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.763+0000 7f514c7fc740 1 client.26057 shutdown 2024-05-09T08:37:15.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.763+0000 7f514c7fc740 1 -- 172.21.15.107:0/3260702176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 msgr2=0x5636cbaa0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.763+0000 7f514c7fc740 1 --2- 172.21.15.107:0/3260702176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 0x5636cbaa0380 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f513804b960 tx=0x7f5138000bc0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.763+0000 7f514c7fc740 1 -- 172.21.15.107:0/3260702176 shutdown_connections 2024-05-09T08:37:15.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.763+0000 7f514c7fc740 1 --2- 172.21.15.107:0/3260702176 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5636cbb13910 0x5636cbb15d30 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.763+0000 7f514c7fc740 1 --2- 172.21.15.107:0/3260702176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636cba9c680 0x5636cbaa0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.763+0000 7f5122ffd640 0 client.26057 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:15.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.763+0000 7f514c7fc740 1 --2- 172.21.15.107:0/3260702176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5636cba8da80 0x5636cbaa08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.763+0000 7f514c7fc740 1 --2- 172.21.15.107:0/3260702176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636cba8d160 0x5636cbaa0380 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.763+0000 7f514c7fc740 1 -- 172.21.15.107:0/3260702176 >> 172.21.15.107:0/3260702176 conn(0x5636cba895a0 msgr2=0x5636cba87f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:15.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.763+0000 7f514c7fc740 1 -- 172.21.15.107:0/3260702176 shutdown_connections 2024-05-09T08:37:15.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.763+0000 7f514c7fc740 1 -- 172.21.15.107:0/3260702176 wait complete. 2024-05-09T08:37:15.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.763+0000 7f514c7fc740 20 client.26057 trim_cache size 0 max 16384 2024-05-09T08:37:15.769 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:37:15.769 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:37:15.769 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:37:15.769 INFO:teuthology.orchestra.run.smithi107.stdout: "99f0370a-c553-401c-ac78-d98d5ffabeab": { 2024-05-09T08:37:15.769 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:37:15.769 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "99f0370a-c553-401c-ac78-d98d5ffabeab", 2024-05-09T08:37:15.769 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:37:15.769 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:37:15.769 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:37:15.769 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:37:15.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.764+0000 7f5149aee640 1 -- 172.21.15.107:0/1943682405 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f512c0a2d00 msgr2=0x7f512c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.764+0000 7f5149aee640 1 --2- 172.21.15.107:0/1943682405 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f512c0a2d00 0x7f512c0a51c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f513c0ccdc0 tx=0x7f513c091020 comp rx=0 tx=0).stop 2024-05-09T08:37:15.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.764+0000 7f5149aee640 1 -- 172.21.15.107:0/1943682405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5144107810 msgr2=0x7f514419c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:15.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.764+0000 7f5149aee640 1 --2- 172.21.15.107:0/1943682405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5144107810 0x7f514419c960 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f513c00eaa0 tx=0x7f513c00ef60 comp rx=0 tx=0).stop 2024-05-09T08:37:15.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.764+0000 7f5149aee640 1 -- 172.21.15.107:0/1943682405 shutdown_connections 2024-05-09T08:37:15.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.764+0000 7f5149aee640 1 --2- 172.21.15.107:0/1943682405 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f512c0a2d00 0x7f512c0a51c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.764+0000 7f5149aee640 1 --2- 172.21.15.107:0/1943682405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5144107810 0x7f514419c960 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.764+0000 7f5149aee640 1 --2- 172.21.15.107:0/1943682405 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51441021c0 0x7f51441985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.764+0000 7f5149aee640 1 --2- 172.21.15.107:0/1943682405 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5144101800 0x7f51441980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:15.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.764+0000 7f5149aee640 1 -- 172.21.15.107:0/1943682405 >> 172.21.15.107:0/1943682405 conn(0x7f5144078c80 msgr2=0x7f51440ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:15.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.764+0000 7f5149aee640 1 -- 172.21.15.107:0/1943682405 shutdown_connections 2024-05-09T08:37:15.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:15.764+0000 7f5149aee640 1 -- 172.21.15.107:0/1943682405 wait complete. 2024-05-09T08:37:15.774 INFO:tasks.cephfs.filesystem:scrub status for tag:99f0370a-c553-401c-ac78-d98d5ffabeab - {'path': '/', 'tag': '99f0370a-c553-401c-ac78-d98d5ffabeab', 'options': 'recursive,force'} 2024-05-09T08:37:16.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:16 smithi157 ceph-mon[33614]: pgmap v120: 129 pgs: 129 active+clean; 3.9 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:37:16.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:16 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:37:16.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:16 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:16.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:16 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:37:16.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:16 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:37:16.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:16 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:16.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:16 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:37:16.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:16 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:16.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:16 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:16.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:16 smithi110 ceph-mon[33622]: pgmap v120: 129 pgs: 129 active+clean; 3.9 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:37:16.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:16 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:37:16.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:16 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:16.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:16 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:37:16.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:16 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:37:16.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:16 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:16.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:16 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:37:16.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:16 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:16.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:16 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:16.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:16 smithi107 ceph-mon[30122]: pgmap v120: 129 pgs: 129 active+clean; 3.9 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:37:16.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:16 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:37:16.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:16 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:16.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:16 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:37:16.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:16 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:37:16.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:16 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:16.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:16 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:37:16.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:16 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:16.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:16 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:17.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:17 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:37:17.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:17 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:17.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:17 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:37:17.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:17 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:17.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:17 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:37:17.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:17 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:17.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:17 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:37:17.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:17 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:17.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:17 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:37:17.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:17 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:17.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:17 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:37:17.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:17 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:17.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:17 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:37:17.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:17 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:17.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:17 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:37:17.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:17 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:17.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:17 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:37:17.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:17 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:18.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:18 smithi157 ceph-mon[33614]: pgmap v121: 129 pgs: 129 active+clean; 4.0 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 37 MiB/s wr, 54 op/s 2024-05-09T08:37:18.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:18 smithi110 ceph-mon[33622]: pgmap v121: 129 pgs: 129 active+clean; 4.0 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 37 MiB/s wr, 54 op/s 2024-05-09T08:37:18.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:18 smithi107 ceph-mon[30122]: pgmap v121: 129 pgs: 129 active+clean; 4.0 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 37 MiB/s wr, 54 op/s 2024-05-09T08:37:20.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:20 smithi157 ceph-mon[33614]: pgmap v122: 129 pgs: 129 active+clean; 4.0 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 24 MiB/s wr, 46 op/s 2024-05-09T08:37:20.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:20 smithi110 ceph-mon[33622]: pgmap v122: 129 pgs: 129 active+clean; 4.0 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 24 MiB/s wr, 46 op/s 2024-05-09T08:37:20.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:20 smithi107 ceph-mon[30122]: pgmap v122: 129 pgs: 129 active+clean; 4.0 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 24 MiB/s wr, 46 op/s 2024-05-09T08:37:21.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:21 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:37:21.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:21 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:37:21.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:21 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:37:22.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:22 smithi157 ceph-mon[33614]: pgmap v123: 129 pgs: 129 active+clean; 4.0 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 24 MiB/s wr, 46 op/s 2024-05-09T08:37:22.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:22 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:37:22.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:22 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:37:22.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:22 smithi110 ceph-mon[33622]: pgmap v123: 129 pgs: 129 active+clean; 4.0 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 24 MiB/s wr, 46 op/s 2024-05-09T08:37:22.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:22 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:37:22.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:22 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:37:22.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:22 smithi107 ceph-mon[30122]: pgmap v123: 129 pgs: 129 active+clean; 4.0 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 24 MiB/s wr, 46 op/s 2024-05-09T08:37:22.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:22 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:37:22.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:22 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:37:24.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:24 smithi157 ceph-mon[33614]: pgmap v124: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 37 MiB/s wr, 54 op/s 2024-05-09T08:37:24.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:24 smithi110 ceph-mon[33622]: pgmap v124: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 37 MiB/s wr, 54 op/s 2024-05-09T08:37:24.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:24 smithi107 ceph-mon[30122]: pgmap v124: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 37 MiB/s wr, 54 op/s 2024-05-09T08:37:26.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:26 smithi157 ceph-mon[33614]: pgmap v125: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 25 MiB/s wr, 47 op/s 2024-05-09T08:37:26.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:26 smithi110 ceph-mon[33622]: pgmap v125: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 25 MiB/s wr, 47 op/s 2024-05-09T08:37:26.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:26 smithi107 ceph-mon[30122]: pgmap v125: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 25 MiB/s wr, 47 op/s 2024-05-09T08:37:27.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.200+0000 7f14677d5640 1 -- 172.21.15.107:0/1177773617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1460101820 msgr2=0x7f1460101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:27.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.200+0000 7f14677d5640 1 --2- 172.21.15.107:0/1177773617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1460101820 0x7f1460101ca0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f1450009930 tx=0x7f145002a420 comp rx=0 tx=0).stop 2024-05-09T08:37:27.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.200+0000 7f14677d5640 1 -- 172.21.15.107:0/1177773617 shutdown_connections 2024-05-09T08:37:27.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.200+0000 7f14677d5640 1 --2- 172.21.15.107:0/1177773617 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14601021e0 0x7f146010e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.200+0000 7f14677d5640 1 --2- 172.21.15.107:0/1177773617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1460101820 0x7f1460101ca0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.200+0000 7f14677d5640 1 --2- 172.21.15.107:0/1177773617 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1460107830 0x7f1460107c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.200+0000 7f14677d5640 1 -- 172.21.15.107:0/1177773617 >> 172.21.15.107:0/1177773617 conn(0x7f14600fd2f0 msgr2=0x7f14600ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.200+0000 7f14677d5640 1 -- 172.21.15.107:0/1177773617 shutdown_connections 2024-05-09T08:37:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.200+0000 7f14677d5640 1 -- 172.21.15.107:0/1177773617 wait complete. 2024-05-09T08:37:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.201+0000 7f14677d5640 1 Processor -- start 2024-05-09T08:37:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.201+0000 7f14677d5640 1 -- start start 2024-05-09T08:37:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.201+0000 7f14677d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1460101820 0x7f1460198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.201+0000 7f14677d5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14601021e0 0x7f14601985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.201+0000 7f14677d5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1460107830 0x7f146019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.201+0000 7f14677d5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1460111b40 con 0x7f1460101820 2024-05-09T08:37:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.201+0000 7f14677d5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f14601119c0 con 0x7f1460107830 2024-05-09T08:37:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.201+0000 7f14677d5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1460111cc0 con 0x7f14601021e0 2024-05-09T08:37:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.201+0000 7f1465571640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1460101820 0x7f1460198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.201+0000 7f1465571640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1460101820 0x7f1460198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44080/0 (socket says 172.21.15.107:44080) 2024-05-09T08:37:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.201+0000 7f1465571640 1 -- 172.21.15.107:0/3367400502 learned_addr learned my addr 172.21.15.107:0/3367400502 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.201+0000 7f1464d70640 1 --2- 172.21.15.107:0/3367400502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14601021e0 0x7f14601985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.202+0000 7f1465d72640 1 --2- 172.21.15.107:0/3367400502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1460107830 0x7f146019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.202+0000 7f1465571640 1 -- 172.21.15.107:0/3367400502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14601021e0 msgr2=0x7f14601985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.202+0000 7f1465571640 1 --2- 172.21.15.107:0/3367400502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14601021e0 0x7f14601985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.202+0000 7f1465571640 1 -- 172.21.15.107:0/3367400502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1460107830 msgr2=0x7f146019c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.202+0000 7f1465571640 1 --2- 172.21.15.107:0/3367400502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1460107830 0x7f146019c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.202+0000 7f1465571640 1 -- 172.21.15.107:0/3367400502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f146019d000 con 0x7f1460101820 2024-05-09T08:37:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.202+0000 7f1465571640 1 --2- 172.21.15.107:0/3367400502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1460101820 0x7f1460198060 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f145c00e9f0 tx=0x7f145c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.202+0000 7f14567fc640 1 -- 172.21.15.107:0/3367400502 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c00cd80 con 0x7f1460101820 2024-05-09T08:37:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.202+0000 7f14677d5640 1 -- 172.21.15.107:0/3367400502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f146019d2f0 con 0x7f1460101820 2024-05-09T08:37:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.202+0000 7f14567fc640 1 -- 172.21.15.107:0/3367400502 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f145c0108e0 con 0x7f1460101820 2024-05-09T08:37:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.202+0000 7f14567fc640 1 -- 172.21.15.107:0/3367400502 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c018920 con 0x7f1460101820 2024-05-09T08:37:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.202+0000 7f14677d5640 1 -- 172.21.15.107:0/3367400502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14601a4b80 con 0x7f1460101820 2024-05-09T08:37:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.205+0000 7f14567fc640 1 -- 172.21.15.107:0/3367400502 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f145c010a90 con 0x7f1460101820 2024-05-09T08:37:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.205+0000 7f146a5c3740 1 Processor -- start 2024-05-09T08:37:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.205+0000 7f146a5c3740 1 -- start start 2024-05-09T08:37:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f146a5c3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409450 0x56117f409830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f146a5c3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f409d70 0x56117f4140f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f146a5c3740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f414630 0x56117f416a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f146a5c3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56117f36e340 con 0x56117f409d70 2024-05-09T08:37:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f146a5c3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56117f316290 con 0x56117f409450 2024-05-09T08:37:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f146a5c3740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56117f36a300 con 0x56117f414630 2024-05-09T08:37:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f14567fc640 1 --2- 172.21.15.107:0/3367400502 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f144c0a2da0 0x7f144c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f1465571640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f409d70 0x56117f4140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f1465d72640 1 --2- 172.21.15.107:0/3367400502 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f144c0a2da0 0x7f144c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f1465571640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f409d70 0x56117f4140f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44084/0 (socket says 172.21.15.107:44084) 2024-05-09T08:37:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f1465571640 1 -- 172.21.15.107:0/1464276598 learned_addr learned my addr 172.21.15.107:0/1464276598 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f1465d72640 1 --2- 172.21.15.107:0/1464276598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409450 0x56117f409830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f1464d70640 1 --2- 172.21.15.107:0/1464276598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f414630 0x56117f416a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f14567fc640 1 -- 172.21.15.107:0/3367400502 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f145c0cfb10 con 0x7f1460101820 2024-05-09T08:37:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f1465571640 1 -- 172.21.15.107:0/1464276598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f414630 msgr2=0x56117f416a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f1465571640 1 --2- 172.21.15.107:0/1464276598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f414630 0x56117f416a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f1465571640 1 -- 172.21.15.107:0/1464276598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409450 msgr2=0x56117f409830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f1465571640 1 --2- 172.21.15.107:0/1464276598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409450 0x56117f409830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f1465571640 1 -- 172.21.15.107:0/1464276598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56117f33a8e0 con 0x56117f409d70 2024-05-09T08:37:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.206+0000 7f1465d72640 1 --2- 172.21.15.107:0/1464276598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409450 0x56117f409830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.207+0000 7f1465d72640 1 --2- 172.21.15.107:0/3367400502 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f144c0a2da0 0x7f144c0a5260 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1458009fd0 tx=0x7f1458009dc0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.207+0000 7f1465571640 1 --2- 172.21.15.107:0/1464276598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f409d70 0x56117f4140f0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f145c0cce20 tx=0x7f145c010f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b748a9eb29fdd30 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.207+0000 7f144b7fe640 1 -- 172.21.15.107:0/1464276598 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c026040 con 0x56117f409d70 2024-05-09T08:37:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.207+0000 7f144b7fe640 1 -- 172.21.15.107:0/1464276598 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f145c0a0060 con 0x56117f409d70 2024-05-09T08:37:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.207+0000 7f144b7fe640 1 -- 172.21.15.107:0/1464276598 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c09b360 con 0x56117f409d70 2024-05-09T08:37:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.207+0000 7f146a5c3740 1 -- 172.21.15.107:0/1464276598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f409d70 msgr2=0x56117f4140f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.207+0000 7f146a5c3740 1 --2- 172.21.15.107:0/1464276598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f409d70 0x56117f4140f0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f145c0cce20 tx=0x7f145c010f20 comp rx=0 tx=0).stop 2024-05-09T08:37:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.207+0000 7f146a5c3740 1 -- 172.21.15.107:0/1464276598 shutdown_connections 2024-05-09T08:37:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.207+0000 7f146a5c3740 1 --2- 172.21.15.107:0/1464276598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f414630 0x56117f416a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.207+0000 7f146a5c3740 1 --2- 172.21.15.107:0/1464276598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f409d70 0x56117f4140f0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.207+0000 7f146a5c3740 1 --2- 172.21.15.107:0/1464276598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409450 0x56117f409830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.208+0000 7f146a5c3740 1 -- 172.21.15.107:0/1464276598 >> 172.21.15.107:0/1464276598 conn(0x56117f405960 msgr2=0x56117f405d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.208+0000 7f146a5c3740 1 -- 172.21.15.107:0/1464276598 shutdown_connections 2024-05-09T08:37:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.208+0000 7f146a5c3740 1 -- 172.21.15.107:0/1464276598 wait complete. 2024-05-09T08:37:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.208+0000 7f146a5c3740 1 -- start start 2024-05-09T08:37:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.208+0000 7f146a5c3740 1 Processor -- start 2024-05-09T08:37:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.208+0000 7f146a5c3740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409450 0x56117f4115a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.208+0000 7f146a5c3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409d70 0x56117f411ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.208+0000 7f146a5c3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f412020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.208+0000 7f146a5c3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56117f417260 con 0x56117f414630 2024-05-09T08:37:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.208+0000 7f146a5c3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56117f329790 con 0x56117f409d70 2024-05-09T08:37:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.208+0000 7f146a5c3740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56117f4173e0 con 0x56117f409450 2024-05-09T08:37:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.208+0000 7f146a5c3740 10 client.0 _ll_register_callbacks cb 0x56117f364a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1465d72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f412020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1465571640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409450 0x56117f4115a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1465571640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409450 0x56117f4115a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:33432/0 (socket says 172.21.15.107:33432) 2024-05-09T08:37:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1465d72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f412020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44092/0 (socket says 172.21.15.107:44092) 2024-05-09T08:37:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1465571640 1 -- 172.21.15.107:0/698361462 learned_addr learned my addr 172.21.15.107:0/698361462 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1464d70640 1 --2- 172.21.15.107:0/698361462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409d70 0x56117f411ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1465571640 1 -- 172.21.15.107:0/698361462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409d70 msgr2=0x56117f411ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1465571640 1 --2- 172.21.15.107:0/698361462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409d70 0x56117f411ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1465571640 1 -- 172.21.15.107:0/698361462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 msgr2=0x56117f412020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1465571640 1 --2- 172.21.15.107:0/698361462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f412020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1465571640 1 -- 172.21.15.107:0/698361462 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56117f082210 con 0x56117f409450 2024-05-09T08:37:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1465d72640 1 --2- 172.21.15.107:0/698361462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f412020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1465571640 1 --2- 172.21.15.107:0/698361462 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409450 0x56117f4115a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f145c098110 tx=0x7f145c0cfde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac58f1a65560cb9b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1448ff9640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:37:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1448ff9640 1 -- 172.21.15.107:0/698361462 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c099fb0 con 0x56117f409450 2024-05-09T08:37:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f146a5c3740 1 -- 172.21.15.107:0/698361462 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f145c0a0060 con 0x56117f409450 2024-05-09T08:37:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1448ff9640 1 -- 172.21.15.107:0/698361462 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f145c09b100 con 0x56117f409450 2024-05-09T08:37:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f1448ff9640 1 -- 172.21.15.107:0/698361462 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f145c09b300 con 0x56117f409450 2024-05-09T08:37:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f146a5c3740 10 client.26272 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.209+0000 7f146a5c3740 1 -- 172.21.15.107:0/698361462 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56117f14e620 con 0x56117f409450 2024-05-09T08:37:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.210+0000 7f1448ff9640 1 -- 172.21.15.107:0/698361462 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f145c0a0060 con 0x56117f409450 2024-05-09T08:37:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.210+0000 7f1448ff9640 1 -- 172.21.15.107:0/698361462 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f145c083070 con 0x56117f409450 2024-05-09T08:37:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.210+0000 7f146a5c3740 10 client.26272 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.210+0000 7f146a5c3740 10 client.26272 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.210+0000 7f146a5c3740 20 client.26272 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.210+0000 7f146a5c3740 1 --2- 172.21.15.107:0/698361462 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56117f4241e0 0x56117f4265d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.210+0000 7f146a5c3740 4 client.26272 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.210+0000 7f146a5c3740 1 -- 172.21.15.107:0/698361462 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56117f37ba70 con 0x56117f4241e0 2024-05-09T08:37:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.210+0000 7f1464d70640 1 --2- 172.21.15.107:0/698361462 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56117f4241e0 0x56117f4265d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.210+0000 7f1464d70640 1 --2- 172.21.15.107:0/698361462 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56117f4241e0 0x56117f4265d0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ef5cd23fb0882e1 server_cookie=a15ae0f82568afa in_seq=0 out_seq=0 2024-05-09T08:37:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.210+0000 7f1448ff9640 10 client.26272 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.211+0000 7f1448ff9640 1 -- 172.21.15.107:0/698361462 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56117f37ba70 con 0x56117f4241e0 2024-05-09T08:37:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.211+0000 7f1448ff9640 10 client.26272 handle_command_reply: tid=0 2024-05-09T08:37:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.211+0000 7f146a5c3740 1 client.26272 shutdown 2024-05-09T08:37:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.211+0000 7f146a5c3740 1 -- 172.21.15.107:0/698361462 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409450 msgr2=0x56117f4115a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.211+0000 7f146a5c3740 1 --2- 172.21.15.107:0/698361462 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409450 0x56117f4115a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f145c098110 tx=0x7f145c0cfde0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.211+0000 7f146a5c3740 1 -- 172.21.15.107:0/698361462 shutdown_connections 2024-05-09T08:37:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.211+0000 7f146a5c3740 1 --2- 172.21.15.107:0/698361462 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56117f4241e0 0x56117f4265d0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.211+0000 7f146a5c3740 1 --2- 172.21.15.107:0/698361462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f412020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.211+0000 7f1448ff9640 0 client.26272 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.211+0000 7f146a5c3740 1 --2- 172.21.15.107:0/698361462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409d70 0x56117f411ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.211+0000 7f146a5c3740 1 --2- 172.21.15.107:0/698361462 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409450 0x56117f4115a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.211+0000 7f146a5c3740 1 -- 172.21.15.107:0/698361462 >> 172.21.15.107:0/698361462 conn(0x56117f405960 msgr2=0x56117f40b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.212+0000 7f146a5c3740 1 -- 172.21.15.107:0/698361462 shutdown_connections 2024-05-09T08:37:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.212+0000 7f146a5c3740 1 -- 172.21.15.107:0/698361462 wait complete. 2024-05-09T08:37:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.212+0000 7f146a5c3740 20 client.26272 trim_cache size 0 max 16384 2024-05-09T08:37:27.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.226+0000 7f146a5c3740 1 Processor -- start 2024-05-09T08:37:27.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.226+0000 7f146a5c3740 1 -- start start 2024-05-09T08:37:27.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.226+0000 7f146a5c3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409450 0x56117f413cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.226+0000 7f146a5c3740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409d70 0x56117f40bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.226+0000 7f146a5c3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f40e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.226+0000 7f146a5c3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f14601af1b0 con 0x56117f414630 2024-05-09T08:37:27.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.227+0000 7f146a5c3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f14601aca80 con 0x56117f409450 2024-05-09T08:37:27.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.227+0000 7f146a5c3740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f14601aa8d0 con 0x56117f409d70 2024-05-09T08:37:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.227+0000 7f1465d72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f40e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.227+0000 7f1465d72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f40e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44104/0 (socket says 172.21.15.107:44104) 2024-05-09T08:37:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.227+0000 7f1465d72640 1 -- 172.21.15.107:0/4133111337 learned_addr learned my addr 172.21.15.107:0/4133111337 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.227+0000 7f1464d70640 1 --2- 172.21.15.107:0/4133111337 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409d70 0x56117f40bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.227+0000 7f1465571640 1 --2- 172.21.15.107:0/4133111337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409450 0x56117f413cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.227+0000 7f1465d72640 1 -- 172.21.15.107:0/4133111337 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409d70 msgr2=0x56117f40bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.227+0000 7f1465d72640 1 --2- 172.21.15.107:0/4133111337 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409d70 0x56117f40bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.227+0000 7f1465d72640 1 -- 172.21.15.107:0/4133111337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409450 msgr2=0x56117f413cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.227+0000 7f1465d72640 1 --2- 172.21.15.107:0/4133111337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409450 0x56117f413cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.227+0000 7f1465d72640 1 -- 172.21.15.107:0/4133111337 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56117f403f00 con 0x56117f414630 2024-05-09T08:37:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.227+0000 7f1465d72640 1 --2- 172.21.15.107:0/4133111337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f40e820 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f14580145f0 tx=0x7f145801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce52249daca9657 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.227+0000 7f144affd640 1 -- 172.21.15.107:0/4133111337 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1458014110 con 0x56117f414630 2024-05-09T08:37:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.227+0000 7f144affd640 1 -- 172.21.15.107:0/4133111337 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f14580142b0 con 0x56117f414630 2024-05-09T08:37:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.227+0000 7f144affd640 1 -- 172.21.15.107:0/4133111337 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14580054c0 con 0x56117f414630 2024-05-09T08:37:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.228+0000 7f146a5c3740 1 -- 172.21.15.107:0/4133111337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 msgr2=0x56117f40e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.228+0000 7f146a5c3740 1 --2- 172.21.15.107:0/4133111337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f40e820 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f14580145f0 tx=0x7f145801e000 comp rx=0 tx=0).stop 2024-05-09T08:37:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.228+0000 7f146a5c3740 1 -- 172.21.15.107:0/4133111337 shutdown_connections 2024-05-09T08:37:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.228+0000 7f146a5c3740 1 --2- 172.21.15.107:0/4133111337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f40e820 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.228+0000 7f146a5c3740 1 --2- 172.21.15.107:0/4133111337 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409d70 0x56117f40bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.228+0000 7f146a5c3740 1 --2- 172.21.15.107:0/4133111337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409450 0x56117f413cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.228+0000 7f146a5c3740 1 -- 172.21.15.107:0/4133111337 >> 172.21.15.107:0/4133111337 conn(0x56117f405960 msgr2=0x56117f404a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.228+0000 7f146a5c3740 1 -- 172.21.15.107:0/4133111337 shutdown_connections 2024-05-09T08:37:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.228+0000 7f146a5c3740 1 -- 172.21.15.107:0/4133111337 wait complete. 2024-05-09T08:37:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.229+0000 7f146a5c3740 1 -- start start 2024-05-09T08:37:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.229+0000 7f146a5c3740 1 Processor -- start 2024-05-09T08:37:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.229+0000 7f146a5c3740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409450 0x56117f4827a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.229+0000 7f146a5c3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409d70 0x56117f482ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.229+0000 7f146a5c3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f4269e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.229+0000 7f146a5c3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f14601aaf40 con 0x56117f414630 2024-05-09T08:37:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.229+0000 7f146a5c3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f14601ac3b0 con 0x56117f409d70 2024-05-09T08:37:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.229+0000 7f146a5c3740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f14601ab610 con 0x56117f409450 2024-05-09T08:37:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.229+0000 7f146a5c3740 10 client.0 _ll_register_callbacks cb 0x7f1458014690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.229+0000 7f1465d72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f4269e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.229+0000 7f1465571640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409450 0x56117f4827a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.229+0000 7f1464d70640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409d70 0x56117f482ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.229+0000 7f1465d72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f4269e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44110/0 (socket says 172.21.15.107:44110) 2024-05-09T08:37:27.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.229+0000 7f1465d72640 1 -- 172.21.15.107:0/895472732 learned_addr learned my addr 172.21.15.107:0/895472732 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:27.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.230+0000 7f1465571640 1 -- 172.21.15.107:0/895472732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409d70 msgr2=0x56117f482ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:27.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.230+0000 7f1465571640 1 --2- 172.21.15.107:0/895472732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409d70 0x56117f482ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.230+0000 7f1465571640 1 -- 172.21.15.107:0/895472732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 msgr2=0x56117f4269e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:27.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.230+0000 7f1465571640 1 --2- 172.21.15.107:0/895472732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f4269e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.230+0000 7f1465571640 1 -- 172.21.15.107:0/895472732 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56117f4270d0 con 0x56117f409450 2024-05-09T08:37:27.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.230+0000 7f1465571640 1 --2- 172.21.15.107:0/895472732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409450 0x56117f4827a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f145c09b9b0 tx=0x7f145c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10903ffc1a70c57c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:27.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.230+0000 7f1448ff9640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:37:27.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.230+0000 7f1448ff9640 1 -- 172.21.15.107:0/895472732 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c099fb0 con 0x56117f409450 2024-05-09T08:37:27.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.230+0000 7f1448ff9640 1 -- 172.21.15.107:0/895472732 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f145c09b100 con 0x56117f409450 2024-05-09T08:37:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.230+0000 7f146a5c3740 1 -- 172.21.15.107:0/895472732 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14580142b0 con 0x56117f409450 2024-05-09T08:37:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.230+0000 7f1448ff9640 1 -- 172.21.15.107:0/895472732 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f145c09ba80 con 0x56117f409450 2024-05-09T08:37:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.230+0000 7f146a5c3740 10 client.26278 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.230+0000 7f146a5c3740 1 -- 172.21.15.107:0/895472732 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56117f489f50 con 0x56117f409450 2024-05-09T08:37:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.230+0000 7f1448ff9640 1 -- 172.21.15.107:0/895472732 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f14580142b0 con 0x56117f409450 2024-05-09T08:37:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.231+0000 7f1448ff9640 1 -- 172.21.15.107:0/895472732 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f145c083040 con 0x56117f409450 2024-05-09T08:37:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.231+0000 7f146a5c3740 10 client.26278 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.231+0000 7f146a5c3740 10 client.26278 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.231+0000 7f146a5c3740 20 client.26278 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.231+0000 7f146a5c3740 1 --2- 172.21.15.107:0/895472732 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56117f48a380 0x56117f48c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.231+0000 7f146a5c3740 4 client.26278 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:37:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.231+0000 7f146a5c3740 1 -- 172.21.15.107:0/895472732 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56117f375dc0 con 0x56117f48a380 2024-05-09T08:37:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.231+0000 7f1464d70640 1 --2- 172.21.15.107:0/895472732 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56117f48a380 0x56117f48c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.231+0000 7f1448ff9640 10 client.26278 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.231+0000 7f1464d70640 1 --2- 172.21.15.107:0/895472732 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56117f48a380 0x56117f48c770 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bc89de930ce8d56 server_cookie=d26206b279c6af63 in_seq=0 out_seq=0 2024-05-09T08:37:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.248+0000 7f1448ff9640 1 -- 172.21.15.107:0/895472732 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56117f375dc0 con 0x56117f48a380 2024-05-09T08:37:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.248+0000 7f1448ff9640 10 client.26278 handle_command_reply: tid=0 2024-05-09T08:37:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.248+0000 7f146a5c3740 1 client.26278 shutdown 2024-05-09T08:37:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f146a5c3740 1 -- 172.21.15.107:0/895472732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409450 msgr2=0x56117f4827a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f146a5c3740 1 --2- 172.21.15.107:0/895472732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409450 0x56117f4827a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f145c09b9b0 tx=0x7f145c000bc0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f146a5c3740 1 -- 172.21.15.107:0/895472732 shutdown_connections 2024-05-09T08:37:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f146a5c3740 1 --2- 172.21.15.107:0/895472732 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56117f48a380 0x56117f48c770 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f146a5c3740 1 --2- 172.21.15.107:0/895472732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56117f414630 0x56117f4269e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f146a5c3740 1 --2- 172.21.15.107:0/895472732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56117f409d70 0x56117f482ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f1448ff9640 0 client.26278 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f146a5c3740 1 --2- 172.21.15.107:0/895472732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56117f409450 0x56117f4827a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f146a5c3740 1 -- 172.21.15.107:0/895472732 >> 172.21.15.107:0/895472732 conn(0x56117f405960 msgr2=0x56117f41f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f146a5c3740 1 -- 172.21.15.107:0/895472732 shutdown_connections 2024-05-09T08:37:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f146a5c3740 1 -- 172.21.15.107:0/895472732 wait complete. 2024-05-09T08:37:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f146a5c3740 20 client.26278 trim_cache size 0 max 16384 2024-05-09T08:37:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:37:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:37:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:37:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:37:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f14677d5640 1 -- 172.21.15.107:0/3367400502 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f144c0a2da0 msgr2=0x7f144c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f14677d5640 1 --2- 172.21.15.107:0/3367400502 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f144c0a2da0 0x7f144c0a5260 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1458009fd0 tx=0x7f1458009dc0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f14677d5640 1 -- 172.21.15.107:0/3367400502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1460101820 msgr2=0x7f1460198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f14677d5640 1 --2- 172.21.15.107:0/3367400502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1460101820 0x7f1460198060 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f145c00e9f0 tx=0x7f145c00eeb0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f14677d5640 1 -- 172.21.15.107:0/3367400502 shutdown_connections 2024-05-09T08:37:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f14677d5640 1 --2- 172.21.15.107:0/3367400502 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f144c0a2da0 0x7f144c0a5260 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f14677d5640 1 --2- 172.21.15.107:0/3367400502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1460107830 0x7f146019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f14677d5640 1 --2- 172.21.15.107:0/3367400502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14601021e0 0x7f14601985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.249+0000 7f14677d5640 1 --2- 172.21.15.107:0/3367400502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1460101820 0x7f1460198060 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.250+0000 7f14677d5640 1 -- 172.21.15.107:0/3367400502 >> 172.21.15.107:0/3367400502 conn(0x7f14600fd2f0 msgr2=0x7f14600fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.250+0000 7f14677d5640 1 -- 172.21.15.107:0/3367400502 shutdown_connections 2024-05-09T08:37:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:27.250+0000 7f14677d5640 1 -- 172.21.15.107:0/3367400502 wait complete. 2024-05-09T08:37:28.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:28 smithi157 ceph-mon[33614]: pgmap v126: 129 pgs: 129 active+clean; 4.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 37 MiB/s wr, 54 op/s 2024-05-09T08:37:28.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:28 smithi110 ceph-mon[33622]: pgmap v126: 129 pgs: 129 active+clean; 4.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 37 MiB/s wr, 54 op/s 2024-05-09T08:37:28.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:28 smithi107 ceph-mon[30122]: pgmap v126: 129 pgs: 129 active+clean; 4.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 37 MiB/s wr, 54 op/s 2024-05-09T08:37:30.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:30 smithi157 ceph-mon[33614]: pgmap v127: 129 pgs: 129 active+clean; 4.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:37:30.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:30 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:37:30.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:30 smithi110 ceph-mon[33622]: pgmap v127: 129 pgs: 129 active+clean; 4.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:37:30.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:30 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:37:30.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:30 smithi107 ceph-mon[30122]: pgmap v127: 129 pgs: 129 active+clean; 4.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:37:30.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:30 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:37:31.342 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:31 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:37:31.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:31 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:37:31.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:31 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:37:31.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:31 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:37:31.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:31 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:37:31.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:31 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:37:31.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:31 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:37:31.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:31 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:37:31.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:31 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:37:32.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:32 smithi107 ceph-mon[30122]: pgmap v128: 129 pgs: 129 active+clean; 4.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:37:32.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:32 smithi157 ceph-mon[33614]: pgmap v128: 129 pgs: 129 active+clean; 4.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:37:32.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:32 smithi110 ceph-mon[33622]: pgmap v128: 129 pgs: 129 active+clean; 4.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:37:34.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:34 smithi107 ceph-mon[30122]: pgmap v129: 129 pgs: 129 active+clean; 4.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:37:34.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:34 smithi157 ceph-mon[33614]: pgmap v129: 129 pgs: 129 active+clean; 4.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:37:34.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:34 smithi110 ceph-mon[33622]: pgmap v129: 129 pgs: 129 active+clean; 4.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:37:36.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:36 smithi107 ceph-mon[30122]: pgmap v130: 129 pgs: 129 active+clean; 4.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-05-09T08:37:36.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:36 smithi157 ceph-mon[33614]: pgmap v130: 129 pgs: 129 active+clean; 4.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-05-09T08:37:36.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:36 smithi110 ceph-mon[33622]: pgmap v130: 129 pgs: 129 active+clean; 4.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-05-09T08:37:38.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:38 smithi107 ceph-mon[30122]: pgmap v131: 129 pgs: 129 active+clean; 4.6 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:37:38.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:38 smithi157 ceph-mon[33614]: pgmap v131: 129 pgs: 129 active+clean; 4.6 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:37:38.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:38 smithi110 ceph-mon[33622]: pgmap v131: 129 pgs: 129 active+clean; 4.6 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:37:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.351+0000 7f601dbed640 1 -- 172.21.15.107:0/3833645114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6018106720 msgr2=0x7f6018106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.351+0000 7f601dbed640 1 --2- 172.21.15.107:0/3833645114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6018106720 0x7f6018106b00 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f6008009930 tx=0x7f600802a420 comp rx=0 tx=0).stop 2024-05-09T08:37:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.351+0000 7f601dbed640 1 -- 172.21.15.107:0/3833645114 shutdown_connections 2024-05-09T08:37:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.351+0000 7f601dbed640 1 --2- 172.21.15.107:0/3833645114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6018101120 0x7f601810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.351+0000 7f601dbed640 1 --2- 172.21.15.107:0/3833645114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6018100760 0x7f6018100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.351+0000 7f601dbed640 1 --2- 172.21.15.107:0/3833645114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6018106720 0x7f6018106b00 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.351+0000 7f601dbed640 1 -- 172.21.15.107:0/3833645114 >> 172.21.15.107:0/3833645114 conn(0x7f60180778b0 msgr2=0x7f6018077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.351+0000 7f601dbed640 1 -- 172.21.15.107:0/3833645114 shutdown_connections 2024-05-09T08:37:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.351+0000 7f601dbed640 1 -- 172.21.15.107:0/3833645114 wait complete. 2024-05-09T08:37:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.351+0000 7f601dbed640 1 Processor -- start 2024-05-09T08:37:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f601dbed640 1 -- start start 2024-05-09T08:37:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f601dbed640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6018100760 0x7f601819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f601dbed640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6018101120 0x7f601819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f601dbed640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6018106720 0x7f601819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f601dbed640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6018110a30 con 0x7f6018106720 2024-05-09T08:37:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f601dbed640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f60181108b0 con 0x7f6018100760 2024-05-09T08:37:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f601dbed640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6018110bb0 con 0x7f6018101120 2024-05-09T08:37:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f60177fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6018100760 0x7f601819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f6017fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6018106720 0x7f601819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f60177fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6018100760 0x7f601819a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44190/0 (socket says 172.21.15.107:44190) 2024-05-09T08:37:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f60177fe640 1 -- 172.21.15.107:0/2882572969 learned_addr learned my addr 172.21.15.107:0/2882572969 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f6016ffd640 1 --2- 172.21.15.107:0/2882572969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6018101120 0x7f601819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f6017fff640 1 -- 172.21.15.107:0/2882572969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6018101120 msgr2=0x7f601819a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f6017fff640 1 --2- 172.21.15.107:0/2882572969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6018101120 0x7f601819a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f6017fff640 1 -- 172.21.15.107:0/2882572969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6018100760 msgr2=0x7f601819a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f6017fff640 1 --2- 172.21.15.107:0/2882572969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6018100760 0x7f601819a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f6017fff640 1 -- 172.21.15.107:0/2882572969 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601819f280 con 0x7f6018106720 2024-05-09T08:37:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f6016ffd640 1 --2- 172.21.15.107:0/2882572969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6018101120 0x7f601819a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.352+0000 7f6017fff640 1 --2- 172.21.15.107:0/2882572969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6018106720 0x7f601819eb50 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f601000e940 tx=0x7f601000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.353+0000 7f6014ff9640 1 -- 172.21.15.107:0/2882572969 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6010004d20 con 0x7f6018106720 2024-05-09T08:37:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.353+0000 7f601dbed640 1 -- 172.21.15.107:0/2882572969 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f601819f570 con 0x7f6018106720 2024-05-09T08:37:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.353+0000 7f6014ff9640 1 -- 172.21.15.107:0/2882572969 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6010005020 con 0x7f6018106720 2024-05-09T08:37:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.353+0000 7f6014ff9640 1 -- 172.21.15.107:0/2882572969 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60100116d0 con 0x7f6018106720 2024-05-09T08:37:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.353+0000 7f601dbed640 1 -- 172.21.15.107:0/2882572969 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60181a6e00 con 0x7f6018106720 2024-05-09T08:37:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.355+0000 7f6014ff9640 1 -- 172.21.15.107:0/2882572969 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f601000cb20 con 0x7f6018106720 2024-05-09T08:37:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f6014ff9640 1 --2- 172.21.15.107:0/2882572969 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f60000a2da0 0x7f60000a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f60209d0740 1 Processor -- start 2024-05-09T08:37:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f6014ff9640 1 -- 172.21.15.107:0/2882572969 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f60100cfb60 con 0x7f6018106720 2024-05-09T08:37:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f60177fe640 1 --2- 172.21.15.107:0/2882572969 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f60000a2da0 0x7f60000a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f60209d0740 1 -- start start 2024-05-09T08:37:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f60209d0740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5af450 0x5635af5af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f60209d0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5afd70 0x5635af5ba0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f60209d0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5ba630 0x5635af5bca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f60209d0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5635af514340 con 0x5635af5ba630 2024-05-09T08:37:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f60209d0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5635af4bc290 con 0x5635af5afd70 2024-05-09T08:37:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f60209d0740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5635af510300 con 0x5635af5af450 2024-05-09T08:37:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f6017fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5ba630 0x5635af5bca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f6017fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5ba630 0x5635af5bca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45796/0 (socket says 172.21.15.107:45796) 2024-05-09T08:37:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f6017fff640 1 -- 172.21.15.107:0/4030243460 learned_addr learned my addr 172.21.15.107:0/4030243460 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f6016ffd640 1 --2- 172.21.15.107:0/4030243460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5afd70 0x5635af5ba0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f60177fe640 1 --2- 172.21.15.107:0/2882572969 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f60000a2da0 0x7f60000a5260 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6008000c00 tx=0x7f6008005a20 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.356+0000 7f60177fe640 1 --2- 172.21.15.107:0/4030243460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5af450 0x5635af5af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.357+0000 7f6017fff640 1 -- 172.21.15.107:0/4030243460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5af450 msgr2=0x5635af5af830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.357+0000 7f6017fff640 1 --2- 172.21.15.107:0/4030243460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5af450 0x5635af5af830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.357+0000 7f6017fff640 1 -- 172.21.15.107:0/4030243460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5afd70 msgr2=0x5635af5ba0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.357+0000 7f6017fff640 1 --2- 172.21.15.107:0/4030243460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5afd70 0x5635af5ba0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.357+0000 7f6017fff640 1 -- 172.21.15.107:0/4030243460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635af4e08e0 con 0x5635af5ba630 2024-05-09T08:37:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.357+0000 7f6016ffd640 1 --2- 172.21.15.107:0/4030243460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5afd70 0x5635af5ba0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.357+0000 7f6017fff640 1 --2- 172.21.15.107:0/4030243460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5ba630 0x5635af5bca20 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f60100a4460 tx=0x7f601009b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b6360fae55b295c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.357+0000 7f5ffdffb640 1 -- 172.21.15.107:0/4030243460 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601001f070 con 0x5635af5ba630 2024-05-09T08:37:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.357+0000 7f5ffdffb640 1 -- 172.21.15.107:0/4030243460 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60100a00d0 con 0x5635af5ba630 2024-05-09T08:37:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.357+0000 7f5ffdffb640 1 -- 172.21.15.107:0/4030243460 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6010098390 con 0x5635af5ba630 2024-05-09T08:37:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.358+0000 7f60209d0740 1 -- 172.21.15.107:0/4030243460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5ba630 msgr2=0x5635af5bca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.358+0000 7f60209d0740 1 --2- 172.21.15.107:0/4030243460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5ba630 0x5635af5bca20 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f60100a4460 tx=0x7f601009b740 comp rx=0 tx=0).stop 2024-05-09T08:37:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.358+0000 7f60209d0740 1 -- 172.21.15.107:0/4030243460 shutdown_connections 2024-05-09T08:37:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.358+0000 7f60209d0740 1 --2- 172.21.15.107:0/4030243460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5ba630 0x5635af5bca20 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.358+0000 7f60209d0740 1 --2- 172.21.15.107:0/4030243460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5afd70 0x5635af5ba0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.358+0000 7f60209d0740 1 --2- 172.21.15.107:0/4030243460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5af450 0x5635af5af830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.358+0000 7f60209d0740 1 -- 172.21.15.107:0/4030243460 >> 172.21.15.107:0/4030243460 conn(0x5635af5ab960 msgr2=0x5635af5abd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.358+0000 7f60209d0740 1 -- 172.21.15.107:0/4030243460 shutdown_connections 2024-05-09T08:37:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.358+0000 7f60209d0740 1 -- 172.21.15.107:0/4030243460 wait complete. 2024-05-09T08:37:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.358+0000 7f60209d0740 1 -- start start 2024-05-09T08:37:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.358+0000 7f60209d0740 1 Processor -- start 2024-05-09T08:37:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f60209d0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5af450 0x5635af5b75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f60209d0740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5afd70 0x5635af5b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f60209d0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5ba630 0x5635af5b8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f60209d0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5635af5bd260 con 0x5635af5af450 2024-05-09T08:37:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f60209d0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5635af4cf790 con 0x5635af5ba630 2024-05-09T08:37:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f60209d0740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5635af5bd3e0 con 0x5635af5afd70 2024-05-09T08:37:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f60209d0740 10 client.0 _ll_register_callbacks cb 0x5635af50aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f60177fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5af450 0x5635af5b75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f60177fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5af450 0x5635af5b75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45798/0 (socket says 172.21.15.107:45798) 2024-05-09T08:37:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f60177fe640 1 -- 172.21.15.107:0/1910091105 learned_addr learned my addr 172.21.15.107:0/1910091105 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f6017fff640 1 --2- 172.21.15.107:0/1910091105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5ba630 0x5635af5b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f6016ffd640 1 --2- 172.21.15.107:0/1910091105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5afd70 0x5635af5b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f60177fe640 1 -- 172.21.15.107:0/1910091105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5afd70 msgr2=0x5635af5b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f60177fe640 1 --2- 172.21.15.107:0/1910091105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5afd70 0x5635af5b7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f60177fe640 1 -- 172.21.15.107:0/1910091105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5ba630 msgr2=0x5635af5b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f60177fe640 1 --2- 172.21.15.107:0/1910091105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5ba630 0x5635af5b8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f60177fe640 1 -- 172.21.15.107:0/1910091105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635af228210 con 0x5635af5af450 2024-05-09T08:37:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f60177fe640 1 --2- 172.21.15.107:0/1910091105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5af450 0x5635af5b75d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6008009900 tx=0x7f6008009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bdbf6f49cf72d3d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.359+0000 7f5ff6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.360+0000 7f5ff6ffd640 1 -- 172.21.15.107:0/1910091105 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6008035040 con 0x5635af5af450 2024-05-09T08:37:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.360+0000 7f5ff6ffd640 1 -- 172.21.15.107:0/1910091105 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f60080318b0 con 0x5635af5af450 2024-05-09T08:37:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.360+0000 7f60209d0740 1 -- 172.21.15.107:0/1910091105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60100a00d0 con 0x5635af5af450 2024-05-09T08:37:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.360+0000 7f5ff6ffd640 1 -- 172.21.15.107:0/1910091105 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f600805b050 con 0x5635af5af450 2024-05-09T08:37:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.360+0000 7f60209d0740 10 client.16893 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.360+0000 7f60209d0740 1 -- 172.21.15.107:0/1910091105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5635af2f4620 con 0x5635af5af450 2024-05-09T08:37:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.360+0000 7f5ff6ffd640 1 -- 172.21.15.107:0/1910091105 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60100a00d0 con 0x5635af5af450 2024-05-09T08:37:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.360+0000 7f5ff6ffd640 1 -- 172.21.15.107:0/1910091105 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f600803cbb0 con 0x5635af5af450 2024-05-09T08:37:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.360+0000 7f60209d0740 10 client.16893 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.360+0000 7f60209d0740 10 client.16893 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.360+0000 7f60209d0740 20 client.16893 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.360+0000 7f60209d0740 1 --2- 172.21.15.107:0/1910091105 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5635af5ca3d0 0x5635af5cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.360+0000 7f60209d0740 4 client.16893 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.360+0000 7f60209d0740 1 -- 172.21.15.107:0/1910091105 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635af521a70 con 0x5635af5ca3d0 2024-05-09T08:37:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.361+0000 7f6016ffd640 1 --2- 172.21.15.107:0/1910091105 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5635af5ca3d0 0x5635af5cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.361+0000 7f5ff6ffd640 10 client.16893 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.361+0000 7f6016ffd640 1 --2- 172.21.15.107:0/1910091105 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5635af5ca3d0 0x5635af5cc7c0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e9708d6412fc98a server_cookie=5fc9e062fe2eee78 in_seq=0 out_seq=0 2024-05-09T08:37:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.362+0000 7f5ff6ffd640 1 -- 172.21.15.107:0/1910091105 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5635af521a70 con 0x5635af5ca3d0 2024-05-09T08:37:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.362+0000 7f5ff6ffd640 10 client.16893 handle_command_reply: tid=0 2024-05-09T08:37:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.362+0000 7f60209d0740 1 client.16893 shutdown 2024-05-09T08:37:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.362+0000 7f60209d0740 1 -- 172.21.15.107:0/1910091105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5af450 msgr2=0x5635af5b75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.362+0000 7f60209d0740 1 --2- 172.21.15.107:0/1910091105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5af450 0x5635af5b75d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6008009900 tx=0x7f6008009260 comp rx=0 tx=0).stop 2024-05-09T08:37:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.362+0000 7f60209d0740 1 -- 172.21.15.107:0/1910091105 shutdown_connections 2024-05-09T08:37:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.362+0000 7f60209d0740 1 --2- 172.21.15.107:0/1910091105 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5635af5ca3d0 0x5635af5cc7c0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.362+0000 7f60209d0740 1 --2- 172.21.15.107:0/1910091105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5ba630 0x5635af5b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.362+0000 7f5ff6ffd640 0 client.16893 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.362+0000 7f60209d0740 1 --2- 172.21.15.107:0/1910091105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5afd70 0x5635af5b7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.362+0000 7f60209d0740 1 --2- 172.21.15.107:0/1910091105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5af450 0x5635af5b75d0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.362+0000 7f60209d0740 1 -- 172.21.15.107:0/1910091105 >> 172.21.15.107:0/1910091105 conn(0x5635af5ab960 msgr2=0x5635af5b1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.362+0000 7f60209d0740 1 -- 172.21.15.107:0/1910091105 shutdown_connections 2024-05-09T08:37:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.362+0000 7f60209d0740 1 -- 172.21.15.107:0/1910091105 wait complete. 2024-05-09T08:37:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.362+0000 7f60209d0740 20 client.16893 trim_cache size 0 max 16384 2024-05-09T08:37:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.380+0000 7f60209d0740 1 Processor -- start 2024-05-09T08:37:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.380+0000 7f60209d0740 1 -- start start 2024-05-09T08:37:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.380+0000 7f60209d0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5af450 0x5635af5c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.380+0000 7f60209d0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5afd70 0x5635af5b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.380+0000 7f60209d0740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5ba630 0x5635af5d20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.380+0000 7f60209d0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f60181b3630 con 0x5635af5af450 2024-05-09T08:37:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.380+0000 7f60209d0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f60181b0f00 con 0x5635af5afd70 2024-05-09T08:37:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.380+0000 7f60209d0740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f60181aed50 con 0x5635af5ba630 2024-05-09T08:37:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.380+0000 7f60177fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5af450 0x5635af5c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.380+0000 7f60177fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5af450 0x5635af5c6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45804/0 (socket says 172.21.15.107:45804) 2024-05-09T08:37:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.380+0000 7f60177fe640 1 -- 172.21.15.107:0/3774559689 learned_addr learned my addr 172.21.15.107:0/3774559689 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.380+0000 7f6016ffd640 1 --2- 172.21.15.107:0/3774559689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5afd70 0x5635af5b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.381+0000 7f60177fe640 1 -- 172.21.15.107:0/3774559689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5ba630 msgr2=0x5635af5d20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.381+0000 7f6017fff640 1 --2- 172.21.15.107:0/3774559689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5ba630 0x5635af5d20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.381+0000 7f60177fe640 1 --2- 172.21.15.107:0/3774559689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5ba630 0x5635af5d20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.381+0000 7f60177fe640 1 -- 172.21.15.107:0/3774559689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5afd70 msgr2=0x5635af5b1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.381+0000 7f60177fe640 1 --2- 172.21.15.107:0/3774559689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5afd70 0x5635af5b1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.381+0000 7f60177fe640 1 -- 172.21.15.107:0/3774559689 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635af5cce40 con 0x5635af5af450 2024-05-09T08:37:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.381+0000 7f6016ffd640 1 --2- 172.21.15.107:0/3774559689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5afd70 0x5635af5b1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.381+0000 7f6017fff640 1 --2- 172.21.15.107:0/3774559689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5ba630 0x5635af5d20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.381+0000 7f60177fe640 1 --2- 172.21.15.107:0/3774559689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5af450 0x5635af5c6120 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f6008031440 tx=0x7f6008009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f106f4aa3b398082 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.381+0000 7f5ffd7fa640 1 -- 172.21.15.107:0/3774559689 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6008035030 con 0x5635af5af450 2024-05-09T08:37:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.381+0000 7f5ffd7fa640 1 -- 172.21.15.107:0/3774559689 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f600804a7f0 con 0x5635af5af450 2024-05-09T08:37:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.381+0000 7f5ffd7fa640 1 -- 172.21.15.107:0/3774559689 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6008031de0 con 0x5635af5af450 2024-05-09T08:37:40.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.382+0000 7f60209d0740 1 -- 172.21.15.107:0/3774559689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5af450 msgr2=0x5635af5c6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:40.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.382+0000 7f60209d0740 1 --2- 172.21.15.107:0/3774559689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5af450 0x5635af5c6120 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f6008031440 tx=0x7f6008009830 comp rx=0 tx=0).stop 2024-05-09T08:37:40.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.382+0000 7f60209d0740 1 -- 172.21.15.107:0/3774559689 shutdown_connections 2024-05-09T08:37:40.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.382+0000 7f60209d0740 1 --2- 172.21.15.107:0/3774559689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5ba630 0x5635af5d20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.382+0000 7f60209d0740 1 --2- 172.21.15.107:0/3774559689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5afd70 0x5635af5b1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.382+0000 7f60209d0740 1 --2- 172.21.15.107:0/3774559689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5af450 0x5635af5c6120 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.382+0000 7f60209d0740 1 -- 172.21.15.107:0/3774559689 >> 172.21.15.107:0/3774559689 conn(0x5635af5ab960 msgr2=0x5635af5aaa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:40.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.382+0000 7f60209d0740 1 -- 172.21.15.107:0/3774559689 shutdown_connections 2024-05-09T08:37:40.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.382+0000 7f60209d0740 1 -- 172.21.15.107:0/3774559689 wait complete. 2024-05-09T08:37:40.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.382+0000 7f60209d0740 1 -- start start 2024-05-09T08:37:40.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.382+0000 7f60209d0740 1 Processor -- start 2024-05-09T08:37:40.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f60209d0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5af450 0x5635af5b4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f60209d0740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5afd70 0x5635af5b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f60209d0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5ba630 0x5635af5b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f60209d0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f60181af3c0 con 0x5635af5ba630 2024-05-09T08:37:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f60209d0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f60181b0830 con 0x5635af5af450 2024-05-09T08:37:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f60209d0740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f60181afa90 con 0x5635af5afd70 2024-05-09T08:37:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f60209d0740 10 client.0 _ll_register_callbacks cb 0x7f6008004290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f6017fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5ba630 0x5635af5b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f6017fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5ba630 0x5635af5b4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45810/0 (socket says 172.21.15.107:45810) 2024-05-09T08:37:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f60177fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5af450 0x5635af5b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f6017fff640 1 -- 172.21.15.107:0/3617250347 learned_addr learned my addr 172.21.15.107:0/3617250347 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f6016ffd640 1 --2- 172.21.15.107:0/3617250347 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5afd70 0x5635af5b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f6017fff640 1 -- 172.21.15.107:0/3617250347 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5afd70 msgr2=0x5635af5b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f6017fff640 1 --2- 172.21.15.107:0/3617250347 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5afd70 0x5635af5b4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f6017fff640 1 -- 172.21.15.107:0/3617250347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5af450 msgr2=0x5635af5b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f6017fff640 1 --2- 172.21.15.107:0/3617250347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5af450 0x5635af5b4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f6017fff640 1 -- 172.21.15.107:0/3617250347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635af5b0820 con 0x5635af5ba630 2024-05-09T08:37:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f60177fe640 1 --2- 172.21.15.107:0/3617250347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5af450 0x5635af5b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f6016ffd640 1 --2- 172.21.15.107:0/3617250347 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5afd70 0x5635af5b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f6017fff640 1 --2- 172.21.15.107:0/3617250347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5ba630 0x5635af5b4fc0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f60100cd000 tx=0x7f601009be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e2e98ddb1b9696 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.383+0000 7f5ff6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.384+0000 7f5ff6ffd640 1 -- 172.21.15.107:0/3617250347 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601007f060 con 0x5635af5ba630 2024-05-09T08:37:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.384+0000 7f5ff6ffd640 1 -- 172.21.15.107:0/3617250347 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f60100cd1d0 con 0x5635af5ba630 2024-05-09T08:37:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.384+0000 7f60209d0740 1 -- 172.21.15.107:0/3617250347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f600804a7f0 con 0x5635af5ba630 2024-05-09T08:37:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.384+0000 7f5ff6ffd640 1 -- 172.21.15.107:0/3617250347 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f601001f040 con 0x5635af5ba630 2024-05-09T08:37:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.384+0000 7f60209d0740 10 client.16899 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.384+0000 7f60209d0740 1 -- 172.21.15.107:0/3617250347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5635af630c70 con 0x5635af5ba630 2024-05-09T08:37:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.384+0000 7f5ff6ffd640 1 -- 172.21.15.107:0/3617250347 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f600804a7f0 con 0x5635af5ba630 2024-05-09T08:37:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.384+0000 7f5ff6ffd640 1 -- 172.21.15.107:0/3617250347 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f601000f040 con 0x5635af5ba630 2024-05-09T08:37:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.384+0000 7f60209d0740 10 client.16899 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.384+0000 7f60209d0740 10 client.16899 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.384+0000 7f60209d0740 20 client.16899 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.384+0000 7f60209d0740 1 --2- 172.21.15.107:0/3617250347 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5635af630f50 0x5635af633340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.384+0000 7f60209d0740 4 client.16899 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:37:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.384+0000 7f60209d0740 1 -- 172.21.15.107:0/3617250347 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5635af51bdc0 con 0x5635af630f50 2024-05-09T08:37:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.385+0000 7f60177fe640 1 --2- 172.21.15.107:0/3617250347 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5635af630f50 0x5635af633340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.385+0000 7f5ff6ffd640 10 client.16899 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.385+0000 7f60177fe640 1 --2- 172.21.15.107:0/3617250347 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5635af630f50 0x5635af633340 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cb009ebdff807e4 server_cookie=a576eddcb01b4e27 in_seq=0 out_seq=0 2024-05-09T08:37:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.403+0000 7f5ff6ffd640 1 -- 172.21.15.107:0/3617250347 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5635af51bdc0 con 0x5635af630f50 2024-05-09T08:37:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.403+0000 7f5ff6ffd640 10 client.16899 handle_command_reply: tid=0 2024-05-09T08:37:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.403+0000 7f60209d0740 1 client.16899 shutdown 2024-05-09T08:37:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.403+0000 7f60209d0740 1 -- 172.21.15.107:0/3617250347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5ba630 msgr2=0x5635af5b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.403+0000 7f60209d0740 1 --2- 172.21.15.107:0/3617250347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5ba630 0x5635af5b4fc0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f60100cd000 tx=0x7f601009be50 comp rx=0 tx=0).stop 2024-05-09T08:37:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.404+0000 7f60209d0740 1 -- 172.21.15.107:0/3617250347 shutdown_connections 2024-05-09T08:37:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.404+0000 7f60209d0740 1 --2- 172.21.15.107:0/3617250347 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5635af630f50 0x5635af633340 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.404+0000 7f60209d0740 1 --2- 172.21.15.107:0/3617250347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5635af5ba630 0x5635af5b4fc0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.404+0000 7f60209d0740 1 --2- 172.21.15.107:0/3617250347 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5635af5afd70 0x5635af5b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.404+0000 7f60209d0740 1 --2- 172.21.15.107:0/3617250347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635af5af450 0x5635af5b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.404+0000 7f60209d0740 1 -- 172.21.15.107:0/3617250347 >> 172.21.15.107:0/3617250347 conn(0x5635af5ab960 msgr2=0x5635af5b7590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.404+0000 7f5ff6ffd640 0 client.16899 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.404+0000 7f60209d0740 1 -- 172.21.15.107:0/3617250347 shutdown_connections 2024-05-09T08:37:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.404+0000 7f60209d0740 1 -- 172.21.15.107:0/3617250347 wait complete. 2024-05-09T08:37:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.404+0000 7f60209d0740 20 client.16899 trim_cache size 0 max 16384 2024-05-09T08:37:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:37:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:37:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:37:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:37:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.404+0000 7f601dbed640 1 -- 172.21.15.107:0/2882572969 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f60000a2da0 msgr2=0x7f60000a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.404+0000 7f601dbed640 1 --2- 172.21.15.107:0/2882572969 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f60000a2da0 0x7f60000a5260 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6008000c00 tx=0x7f6008005a20 comp rx=0 tx=0).stop 2024-05-09T08:37:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.404+0000 7f601dbed640 1 -- 172.21.15.107:0/2882572969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6018106720 msgr2=0x7f601819eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.404+0000 7f601dbed640 1 --2- 172.21.15.107:0/2882572969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6018106720 0x7f601819eb50 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f601000e940 tx=0x7f601000ee00 comp rx=0 tx=0).stop 2024-05-09T08:37:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.405+0000 7f601dbed640 1 -- 172.21.15.107:0/2882572969 shutdown_connections 2024-05-09T08:37:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.405+0000 7f601dbed640 1 --2- 172.21.15.107:0/2882572969 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f60000a2da0 0x7f60000a5260 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.405+0000 7f601dbed640 1 --2- 172.21.15.107:0/2882572969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6018106720 0x7f601819eb50 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.405+0000 7f601dbed640 1 --2- 172.21.15.107:0/2882572969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6018101120 0x7f601819a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.405+0000 7f601dbed640 1 --2- 172.21.15.107:0/2882572969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6018100760 0x7f601819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.405+0000 7f601dbed640 1 -- 172.21.15.107:0/2882572969 >> 172.21.15.107:0/2882572969 conn(0x7f60180778b0 msgr2=0x7f601810bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.405+0000 7f601dbed640 1 -- 172.21.15.107:0/2882572969 shutdown_connections 2024-05-09T08:37:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:40.405+0000 7f601dbed640 1 -- 172.21.15.107:0/2882572969 wait complete. 2024-05-09T08:37:40.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:40 smithi107 ceph-mon[30122]: pgmap v132: 129 pgs: 129 active+clean; 4.6 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:37:40.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:40 smithi157 ceph-mon[33614]: pgmap v132: 129 pgs: 129 active+clean; 4.6 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:37:40.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:40 smithi110 ceph-mon[33622]: pgmap v132: 129 pgs: 129 active+clean; 4.6 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:37:42.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:42 smithi107 ceph-mon[30122]: pgmap v133: 129 pgs: 129 active+clean; 4.6 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:37:42.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:42 smithi157 ceph-mon[33614]: pgmap v133: 129 pgs: 129 active+clean; 4.6 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:37:42.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:42 smithi110 ceph-mon[33622]: pgmap v133: 129 pgs: 129 active+clean; 4.6 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:37:44.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:44 smithi107 ceph-mon[30122]: pgmap v134: 129 pgs: 129 active+clean; 4.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 37 MiB/s wr, 31 op/s 2024-05-09T08:37:44.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:44 smithi157 ceph-mon[33614]: pgmap v134: 129 pgs: 129 active+clean; 4.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 37 MiB/s wr, 31 op/s 2024-05-09T08:37:44.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:44 smithi110 ceph-mon[33622]: pgmap v134: 129 pgs: 129 active+clean; 4.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 37 MiB/s wr, 31 op/s 2024-05-09T08:37:45.776 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:37:45.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:37:45.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.893+0000 7f7159d09640 1 -- 172.21.15.107:0/250831280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71541033d0 msgr2=0x7f715410b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:45.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.893+0000 7f7159d09640 1 --2- 172.21.15.107:0/250831280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71541033d0 0x7f715410b8b0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f7148009900 tx=0x7f714802a510 comp rx=0 tx=0).stop 2024-05-09T08:37:45.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.893+0000 7f7159d09640 1 -- 172.21.15.107:0/250831280 shutdown_connections 2024-05-09T08:37:45.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.893+0000 7f7159d09640 1 --2- 172.21.15.107:0/250831280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f715410bdf0 0x7f715410e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.893+0000 7f7159d09640 1 --2- 172.21.15.107:0/250831280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71541033d0 0x7f715410b8b0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.893+0000 7f7159d09640 1 --2- 172.21.15.107:0/250831280 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7154102ab0 0x7f7154102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.893+0000 7f7159d09640 1 -- 172.21.15.107:0/250831280 >> 172.21.15.107:0/250831280 conn(0x7f7154077830 msgr2=0x7f7154077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:45.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.893+0000 7f7159d09640 1 -- 172.21.15.107:0/250831280 shutdown_connections 2024-05-09T08:37:45.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.893+0000 7f7159d09640 1 -- 172.21.15.107:0/250831280 wait complete. 2024-05-09T08:37:45.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.894+0000 7f7159d09640 1 Processor -- start 2024-05-09T08:37:45.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.894+0000 7f7159d09640 1 -- start start 2024-05-09T08:37:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7159d09640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7154102ab0 0x7f7154198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7159d09640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71541033d0 0x7f7154198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7159d09640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f715410bdf0 0x7f715419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7159d09640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7154111aa0 con 0x7f715410bdf0 2024-05-09T08:37:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7159d09640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7154111920 con 0x7f7154102ab0 2024-05-09T08:37:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7159d09640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7154111c20 con 0x7f71541033d0 2024-05-09T08:37:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7153fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f715410bdf0 0x7f715419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7153fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f715410bdf0 0x7f715419c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44814/0 (socket says 172.21.15.107:44814) 2024-05-09T08:37:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7153fff640 1 -- 172.21.15.107:0/2975096522 learned_addr learned my addr 172.21.15.107:0/2975096522 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7152ffd640 1 --2- 172.21.15.107:0/2975096522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71541033d0 0x7f7154198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f71537fe640 1 --2- 172.21.15.107:0/2975096522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7154102ab0 0x7f7154198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:45.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7153fff640 1 -- 172.21.15.107:0/2975096522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71541033d0 msgr2=0x7f7154198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:45.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7153fff640 1 --2- 172.21.15.107:0/2975096522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71541033d0 0x7f7154198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7153fff640 1 -- 172.21.15.107:0/2975096522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7154102ab0 msgr2=0x7f7154198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:45.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7153fff640 1 --2- 172.21.15.107:0/2975096522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7154102ab0 0x7f7154198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7153fff640 1 -- 172.21.15.107:0/2975096522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f715419d040 con 0x7f715410bdf0 2024-05-09T08:37:45.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7153fff640 1 --2- 172.21.15.107:0/2975096522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f715410bdf0 0x7f715419c910 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f714c00cce0 tx=0x7f714c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:45.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7150ff9640 1 -- 172.21.15.107:0/2975096522 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714c004b20 con 0x7f715410bdf0 2024-05-09T08:37:45.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7150ff9640 1 -- 172.21.15.107:0/2975096522 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f714c004cc0 con 0x7f715410bdf0 2024-05-09T08:37:45.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.895+0000 7f7150ff9640 1 -- 172.21.15.107:0/2975096522 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714c005420 con 0x7f715410bdf0 2024-05-09T08:37:45.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.896+0000 7f7159d09640 1 -- 172.21.15.107:0/2975096522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f715419d330 con 0x7f715410bdf0 2024-05-09T08:37:45.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.896+0000 7f7159d09640 1 -- 172.21.15.107:0/2975096522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7154100630 con 0x7f715410bdf0 2024-05-09T08:37:45.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f7150ff9640 1 -- 172.21.15.107:0/2975096522 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f714c0055c0 con 0x7f715410bdf0 2024-05-09T08:37:45.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f715ca0c740 1 Processor -- start 2024-05-09T08:37:45.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f715ca0c740 1 -- start start 2024-05-09T08:37:45.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f715ca0c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 0x55f23c085660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f715ca0c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c085ba0 0x55f23c08fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f715ca0c740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c090270 0x55f23c092660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f715ca0c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f23bfead00 con 0x55f23c085ba0 2024-05-09T08:37:45.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f715ca0c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f23bf924c0 con 0x55f23c085280 2024-05-09T08:37:45.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f715ca0c740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f23bfe6cc0 con 0x55f23c090270 2024-05-09T08:37:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f71537fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c085ba0 0x55f23c08fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f71537fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c085ba0 0x55f23c08fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44822/0 (socket says 172.21.15.107:44822) 2024-05-09T08:37:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f71537fe640 1 -- 172.21.15.107:0/2779110425 learned_addr learned my addr 172.21.15.107:0/2779110425 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f7150ff9640 1 --2- 172.21.15.107:0/2975096522 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f713c0a2d00 0x7f713c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f71537fe640 1 -- 172.21.15.107:0/2779110425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c090270 msgr2=0x55f23c092660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f71537fe640 1 --2- 172.21.15.107:0/2779110425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c090270 0x55f23c092660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f71537fe640 1 -- 172.21.15.107:0/2779110425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 msgr2=0x55f23c085660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f71537fe640 1 --2- 172.21.15.107:0/2779110425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 0x55f23c085660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.899+0000 7f71537fe640 1 -- 172.21.15.107:0/2779110425 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f23bdc87d0 con 0x55f23c085ba0 2024-05-09T08:37:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.900+0000 7f7153fff640 1 --2- 172.21.15.107:0/2975096522 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f713c0a2d00 0x7f713c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.900+0000 7f7150ff9640 1 -- 172.21.15.107:0/2975096522 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f714c0d3790 con 0x7f715410bdf0 2024-05-09T08:37:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.900+0000 7f71537fe640 1 --2- 172.21.15.107:0/2779110425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c085ba0 0x55f23c08fd30 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f714400cbe0 tx=0x7f714400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18181dc34e360ca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.900+0000 7f7139ffb640 1 -- 172.21.15.107:0/2779110425 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7144010630 con 0x55f23c085ba0 2024-05-09T08:37:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.900+0000 7f7139ffb640 1 -- 172.21.15.107:0/2779110425 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7144010c80 con 0x55f23c085ba0 2024-05-09T08:37:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.900+0000 7f7139ffb640 1 -- 172.21.15.107:0/2779110425 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714401ab10 con 0x55f23c085ba0 2024-05-09T08:37:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.901+0000 7f715ca0c740 1 -- 172.21.15.107:0/2779110425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c085ba0 msgr2=0x55f23c08fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.901+0000 7f7153fff640 1 --2- 172.21.15.107:0/2975096522 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f713c0a2d00 0x7f713c0a51c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f714c0d0d40 tx=0x7f714c06e990 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.901+0000 7f715ca0c740 1 --2- 172.21.15.107:0/2779110425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c085ba0 0x55f23c08fd30 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f714400cbe0 tx=0x7f714400e720 comp rx=0 tx=0).stop 2024-05-09T08:37:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.901+0000 7f715ca0c740 1 -- 172.21.15.107:0/2779110425 shutdown_connections 2024-05-09T08:37:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.901+0000 7f715ca0c740 1 --2- 172.21.15.107:0/2779110425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c090270 0x55f23c092660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.901+0000 7f715ca0c740 1 --2- 172.21.15.107:0/2779110425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c085ba0 0x55f23c08fd30 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.901+0000 7f715ca0c740 1 --2- 172.21.15.107:0/2779110425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 0x55f23c085660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.901+0000 7f715ca0c740 1 -- 172.21.15.107:0/2779110425 >> 172.21.15.107:0/2779110425 conn(0x55f23c0815a0 msgr2=0x55f23c081980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.901+0000 7f715ca0c740 1 -- 172.21.15.107:0/2779110425 shutdown_connections 2024-05-09T08:37:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.901+0000 7f715ca0c740 1 -- 172.21.15.107:0/2779110425 wait complete. 2024-05-09T08:37:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.901+0000 7f715ca0c740 1 -- start start 2024-05-09T08:37:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f715ca0c740 1 Processor -- start 2024-05-09T08:37:45.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f715ca0c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 0x55f23c08d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f715ca0c740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c085ba0 0x55f23c08d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f715ca0c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c090270 0x55f23c08dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f715ca0c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f23c093060 con 0x55f23c090270 2024-05-09T08:37:45.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f715ca0c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f23bfa5b90 con 0x55f23c085280 2024-05-09T08:37:45.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f715ca0c740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f23c0931e0 con 0x55f23c085ba0 2024-05-09T08:37:45.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f715ca0c740 10 client.0 _ll_register_callbacks cb 0x55f23bfe1da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:45.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f7152ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c090270 0x55f23c08dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:45.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f7152ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c090270 0x55f23c08dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44828/0 (socket says 172.21.15.107:44828) 2024-05-09T08:37:45.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f7152ffd640 1 -- 172.21.15.107:0/1259477858 learned_addr learned my addr 172.21.15.107:0/1259477858 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:45.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f7152ffd640 1 --2- 172.21.15.107:0/1259477858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 0x55f23c08d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:45.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f71537fe640 1 --2- 172.21.15.107:0/1259477858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c085ba0 0x55f23c08d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:45.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f7152ffd640 1 -- 172.21.15.107:0/1259477858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c085ba0 msgr2=0x55f23c08d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:45.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f7152ffd640 1 --2- 172.21.15.107:0/1259477858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c085ba0 0x55f23c08d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f7152ffd640 1 -- 172.21.15.107:0/1259477858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 msgr2=0x55f23c08d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:45.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f7152ffd640 1 --2- 172.21.15.107:0/1259477858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 0x55f23c08d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f7152ffd640 1 -- 172.21.15.107:0/1259477858 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f23bcfe3f0 con 0x55f23c090270 2024-05-09T08:37:45.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f7152ffd640 1 --2- 172.21.15.107:0/1259477858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c090270 0x55f23c08dc90 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f71480096c0 tx=0x7f7148005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=129faecd189bc303 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:45.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.902+0000 7f7132ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:45.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.903+0000 7f7132ffd640 1 -- 172.21.15.107:0/1259477858 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7148035050 con 0x55f23c090270 2024-05-09T08:37:45.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.903+0000 7f7132ffd640 1 -- 172.21.15.107:0/1259477858 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f71480042d0 con 0x55f23c090270 2024-05-09T08:37:45.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.903+0000 7f715ca0c740 1 -- 172.21.15.107:0/1259477858 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7144010c80 con 0x55f23c090270 2024-05-09T08:37:45.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.903+0000 7f7132ffd640 1 -- 172.21.15.107:0/1259477858 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f71480044d0 con 0x55f23c090270 2024-05-09T08:37:45.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.903+0000 7f715ca0c740 10 client.16911 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:45.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.903+0000 7f715ca0c740 1 -- 172.21.15.107:0/1259477858 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f23bcbe0f0 con 0x55f23c090270 2024-05-09T08:37:45.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.903+0000 7f7132ffd640 1 -- 172.21.15.107:0/1259477858 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7144010c80 con 0x55f23c090270 2024-05-09T08:37:45.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.903+0000 7f7132ffd640 1 -- 172.21.15.107:0/1259477858 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f714803d680 con 0x55f23c090270 2024-05-09T08:37:45.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.903+0000 7f715ca0c740 10 client.16911 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:45.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.903+0000 7f715ca0c740 10 client.16911 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:45.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.903+0000 7f715ca0c740 20 client.16911 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:45.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.903+0000 7f715ca0c740 1 --2- 172.21.15.107:0/1259477858 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f23c0a0610 0x55f23c0a2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.903+0000 7f715ca0c740 4 client.16911 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:45.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.903+0000 7f715ca0c740 1 -- 172.21.15.107:0/1259477858 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f23bff8430 con 0x55f23c0a0610 2024-05-09T08:37:45.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.904+0000 7f71537fe640 1 --2- 172.21.15.107:0/1259477858 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f23c0a0610 0x55f23c0a2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:45.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.904+0000 7f7132ffd640 10 client.16911 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:45.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.904+0000 7f71537fe640 1 --2- 172.21.15.107:0/1259477858 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f23c0a0610 0x55f23c0a2a00 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fca047b70b1d6a4 server_cookie=e2249b2719fd8860 in_seq=0 out_seq=0 2024-05-09T08:37:45.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.905+0000 7f7132ffd640 1 -- 172.21.15.107:0/1259477858 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f23bff8430 con 0x55f23c0a0610 2024-05-09T08:37:45.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.905+0000 7f7132ffd640 10 client.16911 handle_command_reply: tid=0 2024-05-09T08:37:45.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.905+0000 7f715ca0c740 1 client.16911 shutdown 2024-05-09T08:37:45.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.905+0000 7f715ca0c740 1 -- 172.21.15.107:0/1259477858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c090270 msgr2=0x55f23c08dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:45.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.905+0000 7f715ca0c740 1 --2- 172.21.15.107:0/1259477858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c090270 0x55f23c08dc90 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f71480096c0 tx=0x7f7148005b50 comp rx=0 tx=0).stop 2024-05-09T08:37:45.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.905+0000 7f715ca0c740 1 -- 172.21.15.107:0/1259477858 shutdown_connections 2024-05-09T08:37:45.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.905+0000 7f715ca0c740 1 --2- 172.21.15.107:0/1259477858 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f23c0a0610 0x55f23c0a2a00 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.905+0000 7f715ca0c740 1 --2- 172.21.15.107:0/1259477858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c090270 0x55f23c08dc90 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.905+0000 7f715ca0c740 1 --2- 172.21.15.107:0/1259477858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c085ba0 0x55f23c08d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.905+0000 7f715ca0c740 1 --2- 172.21.15.107:0/1259477858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 0x55f23c08d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.905+0000 7f7132ffd640 0 client.16911 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:45.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.905+0000 7f715ca0c740 1 -- 172.21.15.107:0/1259477858 >> 172.21.15.107:0/1259477858 conn(0x55f23c0815a0 msgr2=0x55f23c0873a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:45.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.905+0000 7f715ca0c740 1 -- 172.21.15.107:0/1259477858 shutdown_connections 2024-05-09T08:37:45.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.905+0000 7f715ca0c740 1 -- 172.21.15.107:0/1259477858 wait complete. 2024-05-09T08:37:45.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.905+0000 7f715ca0c740 20 client.16911 trim_cache size 0 max 16384 2024-05-09T08:37:45.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.925+0000 7f715ca0c740 1 Processor -- start 2024-05-09T08:37:45.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.925+0000 7f715ca0c740 1 -- start start 2024-05-09T08:37:45.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f715ca0c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 0x55f23c09bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f715ca0c740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c085ba0 0x55f23c080660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f715ca0c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c090270 0x55f23c0a8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f715ca0c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f71541af290 con 0x55f23c090270 2024-05-09T08:37:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f715ca0c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f71541acb60 con 0x55f23c085280 2024-05-09T08:37:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f715ca0c740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f71541aa9b0 con 0x55f23c085ba0 2024-05-09T08:37:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f7152ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c090270 0x55f23c0a8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f7152ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c090270 0x55f23c0a8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44844/0 (socket says 172.21.15.107:44844) 2024-05-09T08:37:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f7152ffd640 1 -- 172.21.15.107:0/2010033939 learned_addr learned my addr 172.21.15.107:0/2010033939 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f71537fe640 1 --2- 172.21.15.107:0/2010033939 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c085ba0 0x55f23c080660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f7152ffd640 1 --2- 172.21.15.107:0/2010033939 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 0x55f23c09bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f7152ffd640 1 -- 172.21.15.107:0/2010033939 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c085ba0 msgr2=0x55f23c080660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f7152ffd640 1 --2- 172.21.15.107:0/2010033939 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c085ba0 0x55f23c080660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f7152ffd640 1 -- 172.21.15.107:0/2010033939 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 msgr2=0x55f23c09bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f7152ffd640 1 --2- 172.21.15.107:0/2010033939 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 0x55f23c09bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f7152ffd640 1 -- 172.21.15.107:0/2010033939 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f23c08b760 con 0x55f23c090270 2024-05-09T08:37:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.926+0000 7f7152ffd640 1 --2- 172.21.15.107:0/2010033939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c090270 0x55f23c0a8310 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7148031540 tx=0x7f714802a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3399a5bf8a3ba015 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.927+0000 7f71397fa640 1 -- 172.21.15.107:0/2010033939 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7148057070 con 0x55f23c090270 2024-05-09T08:37:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.927+0000 7f71397fa640 1 -- 172.21.15.107:0/2010033939 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71480042d0 con 0x55f23c090270 2024-05-09T08:37:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.927+0000 7f71397fa640 1 -- 172.21.15.107:0/2010033939 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7148005650 con 0x55f23c090270 2024-05-09T08:37:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.927+0000 7f715ca0c740 1 -- 172.21.15.107:0/2010033939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c090270 msgr2=0x55f23c0a8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.927+0000 7f715ca0c740 1 --2- 172.21.15.107:0/2010033939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c090270 0x55f23c0a8310 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7148031540 tx=0x7f714802a510 comp rx=0 tx=0).stop 2024-05-09T08:37:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.927+0000 7f715ca0c740 1 -- 172.21.15.107:0/2010033939 shutdown_connections 2024-05-09T08:37:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.927+0000 7f715ca0c740 1 --2- 172.21.15.107:0/2010033939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c090270 0x55f23c0a8310 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.927+0000 7f715ca0c740 1 --2- 172.21.15.107:0/2010033939 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c085ba0 0x55f23c080660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.927+0000 7f715ca0c740 1 --2- 172.21.15.107:0/2010033939 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 0x55f23c09bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.927+0000 7f715ca0c740 1 -- 172.21.15.107:0/2010033939 >> 172.21.15.107:0/2010033939 conn(0x55f23c0815a0 msgr2=0x55f23c07ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.927+0000 7f715ca0c740 1 -- 172.21.15.107:0/2010033939 shutdown_connections 2024-05-09T08:37:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.927+0000 7f715ca0c740 1 -- 172.21.15.107:0/2010033939 wait complete. 2024-05-09T08:37:45.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.928+0000 7f715ca0c740 1 -- start start 2024-05-09T08:37:45.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.928+0000 7f715ca0c740 1 Processor -- start 2024-05-09T08:37:45.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.928+0000 7f715ca0c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 0x55f23c098620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.928+0000 7f715ca0c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c085ba0 0x55f23c089aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.928+0000 7f715ca0c740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c090270 0x55f23c089fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.928+0000 7f715ca0c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f71541ab020 con 0x55f23c085ba0 2024-05-09T08:37:45.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.928+0000 7f715ca0c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f71541ac490 con 0x55f23c085280 2024-05-09T08:37:45.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.928+0000 7f715ca0c740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f71541ab6f0 con 0x55f23c090270 2024-05-09T08:37:45.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.928+0000 7f715ca0c740 10 client.0 _ll_register_callbacks cb 0x7f7148009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:45.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.928+0000 7f71537fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c085ba0 0x55f23c089aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.928+0000 7f71537fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c085ba0 0x55f23c089aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44848/0 (socket says 172.21.15.107:44848) 2024-05-09T08:37:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.928+0000 7f71537fe640 1 -- 172.21.15.107:0/3794532593 learned_addr learned my addr 172.21.15.107:0/3794532593 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.928+0000 7f7152ffd640 1 --2- 172.21.15.107:0/3794532593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c090270 0x55f23c089fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.929+0000 7f7152ffd640 1 --2- 172.21.15.107:0/3794532593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 0x55f23c098620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.929+0000 7f71537fe640 1 -- 172.21.15.107:0/3794532593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c090270 msgr2=0x55f23c089fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.929+0000 7f71537fe640 1 --2- 172.21.15.107:0/3794532593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c090270 0x55f23c089fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.929+0000 7f71537fe640 1 -- 172.21.15.107:0/3794532593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 msgr2=0x55f23c098620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.929+0000 7f71537fe640 1 --2- 172.21.15.107:0/3794532593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 0x55f23c098620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.929+0000 7f71537fe640 1 -- 172.21.15.107:0/3794532593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f23c08a520 con 0x55f23c085ba0 2024-05-09T08:37:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.929+0000 7f7152ffd640 1 --2- 172.21.15.107:0/3794532593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 0x55f23c098620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.929+0000 7f71537fe640 1 --2- 172.21.15.107:0/3794532593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c085ba0 0x55f23c089aa0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f714400cbe0 tx=0x7f7144010f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db060950e6869591 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.929+0000 7f7132ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.929+0000 7f7132ffd640 1 -- 172.21.15.107:0/3794532593 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714402c070 con 0x55f23c085ba0 2024-05-09T08:37:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.929+0000 7f715ca0c740 1 -- 172.21.15.107:0/3794532593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71480042d0 con 0x55f23c085ba0 2024-05-09T08:37:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.929+0000 7f7132ffd640 1 -- 172.21.15.107:0/3794532593 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f7144005550 con 0x55f23c085ba0 2024-05-09T08:37:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.929+0000 7f7132ffd640 1 -- 172.21.15.107:0/3794532593 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7144038050 con 0x55f23c085ba0 2024-05-09T08:37:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.929+0000 7f715ca0c740 10 client.16917 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.929+0000 7f715ca0c740 1 -- 172.21.15.107:0/3794532593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f23c1068a0 con 0x55f23c085ba0 2024-05-09T08:37:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.929+0000 7f7132ffd640 1 -- 172.21.15.107:0/3794532593 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71480042d0 con 0x55f23c085ba0 2024-05-09T08:37:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.930+0000 7f7132ffd640 1 -- 172.21.15.107:0/3794532593 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7144010810 con 0x55f23c085ba0 2024-05-09T08:37:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.930+0000 7f715ca0c740 10 client.16917 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.930+0000 7f715ca0c740 10 client.16917 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.930+0000 7f715ca0c740 20 client.16917 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.930+0000 7f715ca0c740 1 --2- 172.21.15.107:0/3794532593 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f23c106d70 0x55f23c109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.930+0000 7f715ca0c740 4 client.16917 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:37:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.930+0000 7f715ca0c740 1 -- 172.21.15.107:0/3794532593 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f23bff2780 con 0x55f23c106d70 2024-05-09T08:37:45.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.930+0000 7f7152ffd640 1 --2- 172.21.15.107:0/3794532593 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f23c106d70 0x55f23c109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:45.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.930+0000 7f7152ffd640 1 --2- 172.21.15.107:0/3794532593 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f23c106d70 0x55f23c109190 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22fecb0aff8df960 server_cookie=10f8e4d0c5096a7a in_seq=0 out_seq=0 2024-05-09T08:37:45.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.930+0000 7f7132ffd640 10 client.16917 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:45.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.931+0000 7f7132ffd640 1 -- 172.21.15.107:0/3794532593 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f23bff2780 con 0x55f23c106d70 2024-05-09T08:37:45.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.931+0000 7f7132ffd640 10 client.16917 handle_command_reply: tid=0 2024-05-09T08:37:45.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.931+0000 7f715ca0c740 1 client.16917 shutdown 2024-05-09T08:37:45.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.931+0000 7f715ca0c740 1 -- 172.21.15.107:0/3794532593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c085ba0 msgr2=0x55f23c089aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:45.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.931+0000 7f715ca0c740 1 --2- 172.21.15.107:0/3794532593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c085ba0 0x55f23c089aa0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f714400cbe0 tx=0x7f7144010f10 comp rx=0 tx=0).stop 2024-05-09T08:37:45.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.931+0000 7f715ca0c740 1 -- 172.21.15.107:0/3794532593 shutdown_connections 2024-05-09T08:37:45.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.931+0000 7f715ca0c740 1 --2- 172.21.15.107:0/3794532593 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f23c106d70 0x55f23c109190 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.931+0000 7f715ca0c740 1 --2- 172.21.15.107:0/3794532593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f23c090270 0x55f23c089fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.931+0000 7f715ca0c740 1 --2- 172.21.15.107:0/3794532593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23c085ba0 0x55f23c089aa0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.931+0000 7f715ca0c740 1 --2- 172.21.15.107:0/3794532593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f23c085280 0x55f23c098620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.931+0000 7f715ca0c740 1 -- 172.21.15.107:0/3794532593 >> 172.21.15.107:0/3794532593 conn(0x55f23c0815a0 msgr2=0x55f23c08da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:45.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.931+0000 7f7132ffd640 0 client.16917 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:45.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.931+0000 7f715ca0c740 1 -- 172.21.15.107:0/3794532593 shutdown_connections 2024-05-09T08:37:45.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.931+0000 7f715ca0c740 1 -- 172.21.15.107:0/3794532593 wait complete. 2024-05-09T08:37:45.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.931+0000 7f715ca0c740 20 client.16917 trim_cache size 0 max 16384 2024-05-09T08:37:45.938 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:37:45.938 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:37:45.938 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:37:45.938 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:37:45.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.932+0000 7f7159d09640 1 -- 172.21.15.107:0/2975096522 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f713c0a2d00 msgr2=0x7f713c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:45.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.932+0000 7f7159d09640 1 --2- 172.21.15.107:0/2975096522 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f713c0a2d00 0x7f713c0a51c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f714c0d0d40 tx=0x7f714c06e990 comp rx=0 tx=0).stop 2024-05-09T08:37:45.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.932+0000 7f7159d09640 1 -- 172.21.15.107:0/2975096522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f715410bdf0 msgr2=0x7f715419c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:45.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.932+0000 7f7159d09640 1 --2- 172.21.15.107:0/2975096522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f715410bdf0 0x7f715419c910 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f714c00cce0 tx=0x7f714c00a620 comp rx=0 tx=0).stop 2024-05-09T08:37:45.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.932+0000 7f7159d09640 1 -- 172.21.15.107:0/2975096522 shutdown_connections 2024-05-09T08:37:45.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.932+0000 7f7159d09640 1 --2- 172.21.15.107:0/2975096522 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f713c0a2d00 0x7f713c0a51c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.932+0000 7f7159d09640 1 --2- 172.21.15.107:0/2975096522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f715410bdf0 0x7f715419c910 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.932+0000 7f7159d09640 1 --2- 172.21.15.107:0/2975096522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71541033d0 0x7f7154198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.932+0000 7f7159d09640 1 --2- 172.21.15.107:0/2975096522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7154102ab0 0x7f7154198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:45.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.932+0000 7f7159d09640 1 -- 172.21.15.107:0/2975096522 >> 172.21.15.107:0/2975096522 conn(0x7f7154077830 msgr2=0x7f71540fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:45.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.932+0000 7f7159d09640 1 -- 172.21.15.107:0/2975096522 shutdown_connections 2024-05-09T08:37:45.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:45.932+0000 7f7159d09640 1 -- 172.21.15.107:0/2975096522 wait complete. 2024-05-09T08:37:45.945 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:37:45.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:37:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.048+0000 7ff801b3b640 1 -- 172.21.15.107:0/2111070354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7fc106720 msgr2=0x7ff7fc106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.048+0000 7ff801b3b640 1 --2- 172.21.15.107:0/2111070354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7fc106720 0x7ff7fc106b00 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009930 tx=0x7ff7ec02a420 comp rx=0 tx=0).stop 2024-05-09T08:37:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.048+0000 7ff801b3b640 1 -- 172.21.15.107:0/2111070354 shutdown_connections 2024-05-09T08:37:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.048+0000 7ff801b3b640 1 --2- 172.21.15.107:0/2111070354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc1010d0 0x7ff7fc10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.048+0000 7ff801b3b640 1 --2- 172.21.15.107:0/2111070354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7fc100710 0x7ff7fc100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.048+0000 7ff801b3b640 1 --2- 172.21.15.107:0/2111070354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7fc106720 0x7ff7fc106b00 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.048+0000 7ff801b3b640 1 -- 172.21.15.107:0/2111070354 >> 172.21.15.107:0/2111070354 conn(0x7ff7fc077830 msgr2=0x7ff7fc077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.048+0000 7ff801b3b640 1 -- 172.21.15.107:0/2111070354 shutdown_connections 2024-05-09T08:37:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.048+0000 7ff801b3b640 1 -- 172.21.15.107:0/2111070354 wait complete. 2024-05-09T08:37:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.048+0000 7ff801b3b640 1 Processor -- start 2024-05-09T08:37:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff801b3b640 1 -- start start 2024-05-09T08:37:46.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff801b3b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7fc100710 0x7ff7fc198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff801b3b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7fc1010d0 0x7ff7fc1993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff801b3b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc106720 0x7ff7fc1a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff801b3b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc1109e0 con 0x7ff7fc100710 2024-05-09T08:37:46.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff801b3b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc110860 con 0x7ff7fc1010d0 2024-05-09T08:37:46.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff801b3b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc110b60 con 0x7ff7fc106720 2024-05-09T08:37:46.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff7fb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7fc100710 0x7ff7fc198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff7fb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7fc100710 0x7ff7fc198e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44862/0 (socket says 172.21.15.107:44862) 2024-05-09T08:37:46.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff7fb7fe640 1 -- 172.21.15.107:0/3696497105 learned_addr learned my addr 172.21.15.107:0/3696497105 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:46.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff7faffd640 1 --2- 172.21.15.107:0/3696497105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7fc1010d0 0x7ff7fc1993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff7fbfff640 1 --2- 172.21.15.107:0/3696497105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc106720 0x7ff7fc1a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff7fb7fe640 1 -- 172.21.15.107:0/3696497105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc106720 msgr2=0x7ff7fc1a3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff7fb7fe640 1 --2- 172.21.15.107:0/3696497105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc106720 0x7ff7fc1a3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff7fb7fe640 1 -- 172.21.15.107:0/3696497105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7fc1010d0 msgr2=0x7ff7fc1993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff7fb7fe640 1 --2- 172.21.15.107:0/3696497105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7fc1010d0 0x7ff7fc1993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff7fb7fe640 1 -- 172.21.15.107:0/3696497105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7fc1a4420 con 0x7ff7fc100710 2024-05-09T08:37:46.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff7faffd640 1 --2- 172.21.15.107:0/3696497105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7fc1010d0 0x7ff7fc1993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:46.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff7fbfff640 1 --2- 172.21.15.107:0/3696497105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc106720 0x7ff7fc1a3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:46.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff7fb7fe640 1 --2- 172.21.15.107:0/3696497105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7fc100710 0x7ff7fc198e60 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec004770 tx=0x7ff7ec0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.049+0000 7ff7f8ff9640 1 -- 172.21.15.107:0/3696497105 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec03f070 con 0x7ff7fc100710 2024-05-09T08:37:46.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.050+0000 7ff801b3b640 1 -- 172.21.15.107:0/3696497105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7fc1a45f0 con 0x7ff7fc100710 2024-05-09T08:37:46.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.050+0000 7ff7f8ff9640 1 -- 172.21.15.107:0/3696497105 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7ec03b460 con 0x7ff7fc100710 2024-05-09T08:37:46.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.050+0000 7ff7f8ff9640 1 -- 172.21.15.107:0/3696497105 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec035070 con 0x7ff7fc100710 2024-05-09T08:37:46.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.050+0000 7ff801b3b640 1 -- 172.21.15.107:0/3696497105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7fc1a4a50 con 0x7ff7fc100710 2024-05-09T08:37:46.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.051+0000 7ff801b3b640 1 -- 172.21.15.107:0/3696497105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7fc101e50 con 0x7ff7fc100710 2024-05-09T08:37:46.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.052+0000 7ff7f8ff9640 1 -- 172.21.15.107:0/3696497105 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ff7ec03b610 con 0x7ff7fc100710 2024-05-09T08:37:46.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.053+0000 7ff7f8ff9640 1 --2- 172.21.15.107:0/3696497105 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff7e40a2d00 0x7ff7e40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.053+0000 7ff7faffd640 1 --2- 172.21.15.107:0/3696497105 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff7e40a2d00 0x7ff7e40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.053+0000 7ff7f8ff9640 1 -- 172.21.15.107:0/3696497105 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff7ec0efbd0 con 0x7ff7fc100710 2024-05-09T08:37:46.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.053+0000 7ff7faffd640 1 --2- 172.21.15.107:0/3696497105 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff7e40a2d00 0x7ff7e40a51c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc069510 tx=0x7ff7f0005e50 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.057+0000 7ff7f8ff9640 1 -- 172.21.15.107:0/3696497105 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff7ec0bc490 con 0x7ff7fc100710 2024-05-09T08:37:46.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.325+0000 7ff801b3b640 1 -- 172.21.15.107:0/3696497105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7fc108cb0 con 0x7ff7fc100710 2024-05-09T08:37:46.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.326+0000 7ff7f8ff9640 1 -- 172.21.15.107:0/3696497105 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11887 (secure 0 0 0) 0x7ff7ec0bbd90 con 0x7ff7fc100710 2024-05-09T08:37:46.326 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:37:46.326 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:37:10.930317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:37:46.326 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2024-05-09T08:37:46.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.328+0000 7ff801b3b640 1 -- 172.21.15.107:0/3696497105 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff7e40a2d00 msgr2=0x7ff7e40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.328+0000 7ff801b3b640 1 --2- 172.21.15.107:0/3696497105 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff7e40a2d00 0x7ff7e40a51c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc069510 tx=0x7ff7f0005e50 comp rx=0 tx=0).stop 2024-05-09T08:37:46.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.328+0000 7ff801b3b640 1 -- 172.21.15.107:0/3696497105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7fc100710 msgr2=0x7ff7fc198e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.328+0000 7ff801b3b640 1 --2- 172.21.15.107:0/3696497105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7fc100710 0x7ff7fc198e60 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec004770 tx=0x7ff7ec0042f0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.328+0000 7ff801b3b640 1 -- 172.21.15.107:0/3696497105 shutdown_connections 2024-05-09T08:37:46.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.328+0000 7ff801b3b640 1 --2- 172.21.15.107:0/3696497105 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff7e40a2d00 0x7ff7e40a51c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.328+0000 7ff801b3b640 1 --2- 172.21.15.107:0/3696497105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc106720 0x7ff7fc1a3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.328+0000 7ff801b3b640 1 --2- 172.21.15.107:0/3696497105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7fc1010d0 0x7ff7fc1993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.328+0000 7ff801b3b640 1 --2- 172.21.15.107:0/3696497105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7fc100710 0x7ff7fc198e60 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.328+0000 7ff801b3b640 1 -- 172.21.15.107:0/3696497105 >> 172.21.15.107:0/3696497105 conn(0x7ff7fc077830 msgr2=0x7ff7fc1030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.329+0000 7ff801b3b640 1 -- 172.21.15.107:0/3696497105 shutdown_connections 2024-05-09T08:37:46.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.329+0000 7ff801b3b640 1 -- 172.21.15.107:0/3696497105 wait complete. 2024-05-09T08:37:46.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:37:46.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.453+0000 7f9f6c493640 1 -- 172.21.15.107:0/343505399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f64102ab0 msgr2=0x7f9f64102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.453+0000 7f9f6c493640 1 --2- 172.21.15.107:0/343505399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f64102ab0 0x7f9f64102e90 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009900 tx=0x7f9f6002a510 comp rx=0 tx=0).stop 2024-05-09T08:37:46.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.453+0000 7f9f6c493640 1 -- 172.21.15.107:0/343505399 shutdown_connections 2024-05-09T08:37:46.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.453+0000 7f9f6c493640 1 --2- 172.21.15.107:0/343505399 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f6410bdf0 0x7f9f6410e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.453+0000 7f9f6c493640 1 --2- 172.21.15.107:0/343505399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f641033d0 0x7f9f6410b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.453+0000 7f9f6c493640 1 --2- 172.21.15.107:0/343505399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f64102ab0 0x7f9f64102e90 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.453+0000 7f9f6c493640 1 -- 172.21.15.107:0/343505399 >> 172.21.15.107:0/343505399 conn(0x7f9f64077830 msgr2=0x7f9f64077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.453+0000 7f9f6c493640 1 -- 172.21.15.107:0/343505399 shutdown_connections 2024-05-09T08:37:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.453+0000 7f9f6c493640 1 -- 172.21.15.107:0/343505399 wait complete. 2024-05-09T08:37:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.454+0000 7f9f6c493640 1 Processor -- start 2024-05-09T08:37:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.454+0000 7f9f6c493640 1 -- start start 2024-05-09T08:37:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.454+0000 7f9f6c493640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f64102ab0 0x7f9f64100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.454+0000 7f9f6c493640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f641033d0 0x7f9f64100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.454+0000 7f9f6c493640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f6410bdf0 0x7f9f64101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.454+0000 7f9f6c493640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9f64111aa0 con 0x7f9f641033d0 2024-05-09T08:37:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.454+0000 7f9f6c493640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9f64111920 con 0x7f9f64102ab0 2024-05-09T08:37:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.454+0000 7f9f6c493640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9f64111c20 con 0x7f9f6410bdf0 2024-05-09T08:37:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.454+0000 7f9f69a2e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f641033d0 0x7f9f64100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.454+0000 7f9f69a2e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f641033d0 0x7f9f64100c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44878/0 (socket says 172.21.15.107:44878) 2024-05-09T08:37:46.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.454+0000 7f9f69a2e640 1 -- 172.21.15.107:0/329631253 learned_addr learned my addr 172.21.15.107:0/329631253 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:46.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.454+0000 7f9f6a22f640 1 --2- 172.21.15.107:0/329631253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f64102ab0 0x7f9f64100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.454+0000 7f9f6aa30640 1 --2- 172.21.15.107:0/329631253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f6410bdf0 0x7f9f64101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.455+0000 7f9f69a2e640 1 -- 172.21.15.107:0/329631253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f6410bdf0 msgr2=0x7f9f64101180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.455+0000 7f9f69a2e640 1 --2- 172.21.15.107:0/329631253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f6410bdf0 0x7f9f64101180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.455+0000 7f9f69a2e640 1 -- 172.21.15.107:0/329631253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f64102ab0 msgr2=0x7f9f64100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.455+0000 7f9f69a2e640 1 --2- 172.21.15.107:0/329631253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f64102ab0 0x7f9f64100700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.455+0000 7f9f69a2e640 1 -- 172.21.15.107:0/329631253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f641a6500 con 0x7f9f641033d0 2024-05-09T08:37:46.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.455+0000 7f9f69a2e640 1 --2- 172.21.15.107:0/329631253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f641033d0 0x7f9f64100c40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f9f5400e9f0 tx=0x7f9f5400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.455+0000 7f9f5b7fe640 1 -- 172.21.15.107:0/329631253 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5400cd80 con 0x7f9f641033d0 2024-05-09T08:37:46.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.455+0000 7f9f6c493640 1 -- 172.21.15.107:0/329631253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f641a67f0 con 0x7f9f641033d0 2024-05-09T08:37:46.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.455+0000 7f9f5b7fe640 1 -- 172.21.15.107:0/329631253 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f540108e0 con 0x7f9f641033d0 2024-05-09T08:37:46.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.455+0000 7f9f5b7fe640 1 -- 172.21.15.107:0/329631253 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f54018920 con 0x7f9f641033d0 2024-05-09T08:37:46.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.455+0000 7f9f6c493640 1 -- 172.21.15.107:0/329631253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f641a6d90 con 0x7f9f641033d0 2024-05-09T08:37:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.457+0000 7f9f5b7fe640 1 -- 172.21.15.107:0/329631253 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f9f54010a90 con 0x7f9f641033d0 2024-05-09T08:37:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.458+0000 7f9f6f1a0740 1 Processor -- start 2024-05-09T08:37:46.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.458+0000 7f9f6f1a0740 1 -- start start 2024-05-09T08:37:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.458+0000 7f9f6f1a0740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3b330 0x564753c3b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.458+0000 7f9f6f1a0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c3bc50 0x564753c45de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.458+0000 7f9f6f1a0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c46320 0x564753c48710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.458+0000 7f9f6f1a0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564753ba0c40 con 0x564753c46320 2024-05-09T08:37:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.458+0000 7f9f6f1a0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564753b48580 con 0x564753c3bc50 2024-05-09T08:37:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.458+0000 7f9f6f1a0740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564753b9cc00 con 0x564753c3b330 2024-05-09T08:37:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.458+0000 7f9f69a2e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c46320 0x564753c48710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.458+0000 7f9f6aa30640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3b330 0x564753c3b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.459+0000 7f9f6aa30640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3b330 0x564753c3b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:37306/0 (socket says 172.21.15.107:37306) 2024-05-09T08:37:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.459+0000 7f9f6aa30640 1 -- 172.21.15.107:0/3177753045 learned_addr learned my addr 172.21.15.107:0/3177753045 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.459+0000 7f9f6a22f640 1 --2- 172.21.15.107:0/3177753045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c3bc50 0x564753c45de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.459+0000 7f9f6aa30640 1 -- 172.21.15.107:0/3177753045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c3bc50 msgr2=0x564753c45de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.459+0000 7f9f6aa30640 1 --2- 172.21.15.107:0/3177753045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c3bc50 0x564753c45de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.459+0000 7f9f5b7fe640 1 --2- 172.21.15.107:0/329631253 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9f500a2d00 0x7f9f500a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.459+0000 7f9f6aa30640 1 -- 172.21.15.107:0/3177753045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c46320 msgr2=0x564753c48710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.459+0000 7f9f6aa30640 1 --2- 172.21.15.107:0/3177753045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c46320 0x564753c48710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.459+0000 7f9f6aa30640 1 -- 172.21.15.107:0/3177753045 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647538b43f0 con 0x564753c3b330 2024-05-09T08:37:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.459+0000 7f9f69a2e640 1 --2- 172.21.15.107:0/3177753045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c46320 0x564753c48710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.459+0000 7f9f5b7fe640 1 -- 172.21.15.107:0/329631253 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f9f540cfa80 con 0x7f9f641033d0 2024-05-09T08:37:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.459+0000 7f9f6aa30640 1 --2- 172.21.15.107:0/329631253 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9f500a2d00 0x7f9f500a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.459+0000 7f9f6aa30640 1 --2- 172.21.15.107:0/3177753045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3b330 0x564753c3b710 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009f70 tx=0x7f9f5c010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31d55a969507e96b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.460+0000 7f9f58bfe640 1 -- 172.21.15.107:0/3177753045 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c022070 con 0x564753c3b330 2024-05-09T08:37:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.460+0000 7f9f58bfe640 1 -- 172.21.15.107:0/3177753045 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f5c0113e0 con 0x564753c3b330 2024-05-09T08:37:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.460+0000 7f9f58bfe640 1 -- 172.21.15.107:0/3177753045 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c016070 con 0x564753c3b330 2024-05-09T08:37:46.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.460+0000 7f9f6aa30640 1 --2- 172.21.15.107:0/329631253 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9f500a2d00 0x7f9f500a51c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c01f000 tx=0x7f9f5c022000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.460+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/3177753045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3b330 msgr2=0x564753c3b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.460+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/3177753045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3b330 0x564753c3b710 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009f70 tx=0x7f9f5c010800 comp rx=0 tx=0).stop 2024-05-09T08:37:46.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.460+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/3177753045 shutdown_connections 2024-05-09T08:37:46.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.460+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/3177753045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c46320 0x564753c48710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.460+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/3177753045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c3bc50 0x564753c45de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.460+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/3177753045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3b330 0x564753c3b710 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.460+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/3177753045 >> 172.21.15.107:0/3177753045 conn(0x564753c37650 msgr2=0x564753c37a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.461+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/3177753045 shutdown_connections 2024-05-09T08:37:46.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.461+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/3177753045 wait complete. 2024-05-09T08:37:46.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.461+0000 7f9f6f1a0740 1 -- start start 2024-05-09T08:37:46.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.461+0000 7f9f6f1a0740 1 Processor -- start 2024-05-09T08:37:46.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.461+0000 7f9f6f1a0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c3b330 0x564753c43270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.461+0000 7f9f6f1a0740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3bc50 0x564753c437b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.461+0000 7f9f6f1a0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c46320 0x564753c43cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.461+0000 7f9f6f1a0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564753c49110 con 0x564753c46320 2024-05-09T08:37:46.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.461+0000 7f9f6f1a0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564753b5bc40 con 0x564753c3b330 2024-05-09T08:37:46.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.461+0000 7f9f6f1a0740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564753c49290 con 0x564753c3bc50 2024-05-09T08:37:46.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.461+0000 7f9f6f1a0740 10 client.0 _ll_register_callbacks cb 0x5647538b3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:46.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.461+0000 7f9f6aa30640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c46320 0x564753c43cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.461+0000 7f9f6aa30640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c46320 0x564753c43cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44906/0 (socket says 172.21.15.107:44906) 2024-05-09T08:37:46.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.461+0000 7f9f6aa30640 1 -- 172.21.15.107:0/782665350 learned_addr learned my addr 172.21.15.107:0/782665350 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:46.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.461+0000 7f9f69a2e640 1 --2- 172.21.15.107:0/782665350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3bc50 0x564753c437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.461+0000 7f9f6a22f640 1 --2- 172.21.15.107:0/782665350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c3b330 0x564753c43270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.462+0000 7f9f69a2e640 1 -- 172.21.15.107:0/782665350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c3b330 msgr2=0x564753c43270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.462+0000 7f9f69a2e640 1 --2- 172.21.15.107:0/782665350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c3b330 0x564753c43270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.462+0000 7f9f69a2e640 1 -- 172.21.15.107:0/782665350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c46320 msgr2=0x564753c43cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.462+0000 7f9f69a2e640 1 --2- 172.21.15.107:0/782665350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c46320 0x564753c43cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.462+0000 7f9f69a2e640 1 -- 172.21.15.107:0/782665350 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56475397e7d0 con 0x564753c3bc50 2024-05-09T08:37:46.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.462+0000 7f9f6aa30640 1 --2- 172.21.15.107:0/782665350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c46320 0x564753c43cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:46.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.462+0000 7f9f69a2e640 1 --2- 172.21.15.107:0/782665350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3bc50 0x564753c437b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9f540cce70 tx=0x7f9f54009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9684c4b2730a9909 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.462+0000 7f9f4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:37:46.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.462+0000 7f9f4dffb640 1 -- 172.21.15.107:0/782665350 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f540d7040 con 0x564753c3bc50 2024-05-09T08:37:46.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.462+0000 7f9f4dffb640 1 -- 172.21.15.107:0/782665350 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f9f54010430 con 0x564753c3bc50 2024-05-09T08:37:46.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.462+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/782665350 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f5c0113e0 con 0x564753c3bc50 2024-05-09T08:37:46.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.462+0000 7f9f4dffb640 1 -- 172.21.15.107:0/782665350 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9f54010de0 con 0x564753c3bc50 2024-05-09T08:37:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.462+0000 7f9f6f1a0740 10 client.26320 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.462+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/782665350 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564753c552d0 con 0x564753c3bc50 2024-05-09T08:37:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.463+0000 7f9f4dffb640 1 -- 172.21.15.107:0/782665350 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f5c0113e0 con 0x564753c3bc50 2024-05-09T08:37:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.463+0000 7f9f4dffb640 1 -- 172.21.15.107:0/782665350 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9f54097a40 con 0x564753c3bc50 2024-05-09T08:37:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.463+0000 7f9f6f1a0740 10 client.26320 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.463+0000 7f9f6f1a0740 10 client.26320 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:37:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.463+0000 7f9f6f1a0740 20 client.26320 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.463+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/782665350 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x564753c567e0 0x564753c58bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.463+0000 7f9f6f1a0740 4 client.26320 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.463+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/782665350 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564753bae370 con 0x564753c567e0 2024-05-09T08:37:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.463+0000 7f9f6a22f640 1 --2- 172.21.15.107:0/782665350 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x564753c567e0 0x564753c58bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.464+0000 7f9f6a22f640 1 --2- 172.21.15.107:0/782665350 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x564753c567e0 0x564753c58bf0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=788e4dcfaaf5b208 server_cookie=33fcbbc5db2551dc in_seq=0 out_seq=0 2024-05-09T08:37:46.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.464+0000 7f9f4dffb640 10 client.26320 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:37:46.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.464+0000 7f9f4dffb640 1 -- 172.21.15.107:0/782665350 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564753bae370 con 0x564753c567e0 2024-05-09T08:37:46.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.464+0000 7f9f4dffb640 10 client.26320 handle_command_reply: tid=0 2024-05-09T08:37:46.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.464+0000 7f9f6f1a0740 1 client.26320 shutdown 2024-05-09T08:37:46.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.464+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/782665350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3bc50 msgr2=0x564753c437b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.464+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/782665350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3bc50 0x564753c437b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9f540cce70 tx=0x7f9f54009310 comp rx=0 tx=0).stop 2024-05-09T08:37:46.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.465+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/782665350 shutdown_connections 2024-05-09T08:37:46.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.465+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/782665350 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x564753c567e0 0x564753c58bf0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.465+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/782665350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c46320 0x564753c43cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.465+0000 7f9f4dffb640 0 client.26320 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:37:46.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.465+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/782665350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3bc50 0x564753c437b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.465+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/782665350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c3b330 0x564753c43270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.465+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/782665350 >> 172.21.15.107:0/782665350 conn(0x564753c37650 msgr2=0x564753c3d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.465+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/782665350 shutdown_connections 2024-05-09T08:37:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.465+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/782665350 wait complete. 2024-05-09T08:37:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.465+0000 7f9f6f1a0740 20 client.26320 trim_cache size 0 max 16384 2024-05-09T08:37:46.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.479+0000 7f9f6f1a0740 1 Processor -- start 2024-05-09T08:37:46.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.479+0000 7f9f6f1a0740 1 -- start start 2024-05-09T08:37:46.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f6f1a0740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3b330 0x564753c51fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f6f1a0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c3bc50 0x564753c42250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f6f1a0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c46320 0x564753c5e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f6f1a0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9f641b3600 con 0x564753c3bc50 2024-05-09T08:37:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f6f1a0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9f641b0ed0 con 0x564753c46320 2024-05-09T08:37:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f6f1a0740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9f641aed20 con 0x564753c3b330 2024-05-09T08:37:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f69a2e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c3bc50 0x564753c42250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f69a2e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c3bc50 0x564753c42250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44916/0 (socket says 172.21.15.107:44916) 2024-05-09T08:37:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f69a2e640 1 -- 172.21.15.107:0/3031719556 learned_addr learned my addr 172.21.15.107:0/3031719556 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f6a22f640 1 --2- 172.21.15.107:0/3031719556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3b330 0x564753c51fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f6aa30640 1 --2- 172.21.15.107:0/3031719556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c46320 0x564753c5e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f69a2e640 1 -- 172.21.15.107:0/3031719556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3b330 msgr2=0x564753c51fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f69a2e640 1 --2- 172.21.15.107:0/3031719556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3b330 0x564753c51fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f69a2e640 1 -- 172.21.15.107:0/3031719556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c46320 msgr2=0x564753c5e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f69a2e640 1 --2- 172.21.15.107:0/3031719556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c46320 0x564753c5e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f69a2e640 1 -- 172.21.15.107:0/3031719556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564753c37480 con 0x564753c3bc50 2024-05-09T08:37:46.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.480+0000 7f9f69a2e640 1 --2- 172.21.15.107:0/3031719556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c3bc50 0x564753c42250 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f9f5400c6a0 tx=0x7f9f540d3180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc75c4afddb44d17 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.481+0000 7f9f4ffff640 1 -- 172.21.15.107:0/3031719556 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5409b830 con 0x564753c3bc50 2024-05-09T08:37:46.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.481+0000 7f9f4ffff640 1 -- 172.21.15.107:0/3031719556 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f5409fff0 con 0x564753c3bc50 2024-05-09T08:37:46.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.481+0000 7f9f4ffff640 1 -- 172.21.15.107:0/3031719556 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5409b140 con 0x564753c3bc50 2024-05-09T08:37:46.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.481+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/3031719556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c3bc50 msgr2=0x564753c42250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.481+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/3031719556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c3bc50 0x564753c42250 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f9f5400c6a0 tx=0x7f9f540d3180 comp rx=0 tx=0).stop 2024-05-09T08:37:46.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.481+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/3031719556 shutdown_connections 2024-05-09T08:37:46.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.481+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/3031719556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c46320 0x564753c5e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.481+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/3031719556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c3bc50 0x564753c42250 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.481+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/3031719556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c3b330 0x564753c51fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.481+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/3031719556 >> 172.21.15.107:0/3031719556 conn(0x564753c37650 msgr2=0x564753c41b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.482+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/3031719556 shutdown_connections 2024-05-09T08:37:46.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.482+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/3031719556 wait complete. 2024-05-09T08:37:46.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.482+0000 7f9f6f1a0740 1 -- start start 2024-05-09T08:37:46.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.482+0000 7f9f6f1a0740 1 Processor -- start 2024-05-09T08:37:46.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.482+0000 7f9f6f1a0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c3b330 0x564753c4e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.482+0000 7f9f6f1a0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c3bc50 0x564753c4e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.482+0000 7f9f6f1a0740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c46320 0x564753c49640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.482+0000 7f9f6f1a0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9f641af390 con 0x564753c3bc50 2024-05-09T08:37:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.482+0000 7f9f6f1a0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9f641b0800 con 0x564753c3b330 2024-05-09T08:37:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.482+0000 7f9f6f1a0740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9f641afa60 con 0x564753c46320 2024-05-09T08:37:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.482+0000 7f9f6f1a0740 10 client.0 _ll_register_callbacks cb 0x7f9f540d32c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.482+0000 7f9f69a2e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c3bc50 0x564753c4e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.482+0000 7f9f69a2e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c3bc50 0x564753c4e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44930/0 (socket says 172.21.15.107:44930) 2024-05-09T08:37:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.482+0000 7f9f69a2e640 1 -- 172.21.15.107:0/870058237 learned_addr learned my addr 172.21.15.107:0/870058237 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.482+0000 7f9f6a22f640 1 --2- 172.21.15.107:0/870058237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c3b330 0x564753c4e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.482+0000 7f9f6aa30640 1 --2- 172.21.15.107:0/870058237 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c46320 0x564753c49640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.483+0000 7f9f69a2e640 1 -- 172.21.15.107:0/870058237 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c46320 msgr2=0x564753c49640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.483+0000 7f9f69a2e640 1 --2- 172.21.15.107:0/870058237 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c46320 0x564753c49640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.483+0000 7f9f69a2e640 1 -- 172.21.15.107:0/870058237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c3b330 msgr2=0x564753c4e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.483+0000 7f9f69a2e640 1 --2- 172.21.15.107:0/870058237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c3b330 0x564753c4e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.483+0000 7f9f69a2e640 1 -- 172.21.15.107:0/870058237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564753c49db0 con 0x564753c3bc50 2024-05-09T08:37:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.483+0000 7f9f6a22f640 1 --2- 172.21.15.107:0/870058237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c3b330 0x564753c4e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.483+0000 7f9f69a2e640 1 --2- 172.21.15.107:0/870058237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c3bc50 0x564753c4e6c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f9f54022000 tx=0x7f9f540cfe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da868566812973c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.483+0000 7f9f4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.483+0000 7f9f4dffb640 1 -- 172.21.15.107:0/870058237 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f540a9060 con 0x564753c3bc50 2024-05-09T08:37:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.483+0000 7f9f4dffb640 1 -- 172.21.15.107:0/870058237 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f9f540ba350 con 0x564753c3bc50 2024-05-09T08:37:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.483+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/870058237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f5409fff0 con 0x564753c3bc50 2024-05-09T08:37:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.483+0000 7f9f4dffb640 1 -- 172.21.15.107:0/870058237 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9f540b8070 con 0x564753c3bc50 2024-05-09T08:37:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.483+0000 7f9f6f1a0740 10 client.16941 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.483+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/870058237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564753c49db0 con 0x564753c3bc50 2024-05-09T08:37:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.483+0000 7f9f4dffb640 1 -- 172.21.15.107:0/870058237 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f5409fff0 con 0x564753c3bc50 2024-05-09T08:37:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.483+0000 7f9f4dffb640 1 -- 172.21.15.107:0/870058237 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9f5409b140 con 0x564753c3bc50 2024-05-09T08:37:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.484+0000 7f9f6f1a0740 10 client.16941 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.484+0000 7f9f6f1a0740 10 client.16941 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:37:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.484+0000 7f9f6f1a0740 20 client.16941 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.484+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/870058237 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x564753cbcae0 0x564753cbef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.484+0000 7f9f6f1a0740 4 client.16941 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:37:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.484+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/870058237 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564753ba86c0 con 0x564753cbcae0 2024-05-09T08:37:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.484+0000 7f9f6a22f640 1 --2- 172.21.15.107:0/870058237 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x564753cbcae0 0x564753cbef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.484+0000 7f9f6a22f640 1 --2- 172.21.15.107:0/870058237 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x564753cbcae0 0x564753cbef00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=153c2a8bed2a47e7 server_cookie=a1ee06079be647f5 in_seq=0 out_seq=0 2024-05-09T08:37:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.484+0000 7f9f4dffb640 10 client.16941 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:37:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.484+0000 7f9f4dffb640 1 -- 172.21.15.107:0/870058237 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564753ba86c0 con 0x564753cbcae0 2024-05-09T08:37:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.484+0000 7f9f4dffb640 10 client.16941 handle_command_reply: tid=0 2024-05-09T08:37:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.484+0000 7f9f6f1a0740 1 client.16941 shutdown 2024-05-09T08:37:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/870058237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c3bc50 msgr2=0x564753c4e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/870058237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c3bc50 0x564753c4e6c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f9f54022000 tx=0x7f9f540cfe70 comp rx=0 tx=0).stop 2024-05-09T08:37:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/870058237 shutdown_connections 2024-05-09T08:37:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/870058237 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x564753cbcae0 0x564753cbef00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/870058237 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564753c46320 0x564753c49640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/870058237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564753c3bc50 0x564753c4e6c0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6f1a0740 1 --2- 172.21.15.107:0/870058237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564753c3b330 0x564753c4e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/870058237 >> 172.21.15.107:0/870058237 conn(0x564753c37650 msgr2=0x564753c423d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f4dffb640 0 client.16941 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:37:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/870058237 shutdown_connections 2024-05-09T08:37:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6f1a0740 1 -- 172.21.15.107:0/870058237 wait complete. 2024-05-09T08:37:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6f1a0740 20 client.16941 trim_cache size 0 max 16384 2024-05-09T08:37:46.492 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:37:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6c493640 1 -- 172.21.15.107:0/329631253 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9f500a2d00 msgr2=0x7f9f500a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6c493640 1 --2- 172.21.15.107:0/329631253 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9f500a2d00 0x7f9f500a51c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c01f000 tx=0x7f9f5c022000 comp rx=0 tx=0).stop 2024-05-09T08:37:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6c493640 1 -- 172.21.15.107:0/329631253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f641033d0 msgr2=0x7f9f64100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6c493640 1 --2- 172.21.15.107:0/329631253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f641033d0 0x7f9f64100c40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f9f5400e9f0 tx=0x7f9f5400eeb0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6c493640 1 -- 172.21.15.107:0/329631253 shutdown_connections 2024-05-09T08:37:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6c493640 1 --2- 172.21.15.107:0/329631253 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9f500a2d00 0x7f9f500a51c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6c493640 1 --2- 172.21.15.107:0/329631253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f6410bdf0 0x7f9f64101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6c493640 1 --2- 172.21.15.107:0/329631253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f641033d0 0x7f9f64100c40 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6c493640 1 --2- 172.21.15.107:0/329631253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f64102ab0 0x7f9f64100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.485+0000 7f9f6c493640 1 -- 172.21.15.107:0/329631253 >> 172.21.15.107:0/329631253 conn(0x7f9f64077830 msgr2=0x7f9f640fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.486+0000 7f9f6c493640 1 -- 172.21.15.107:0/329631253 shutdown_connections 2024-05-09T08:37:46.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.486+0000 7f9f6c493640 1 -- 172.21.15.107:0/329631253 wait complete. 2024-05-09T08:37:46.495 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:46 smithi107 ceph-mon[30122]: pgmap v135: 129 pgs: 129 active+clean; 4.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:37:46.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:37:46.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:46 smithi157 ceph-mon[33614]: pgmap v135: 129 pgs: 129 active+clean; 4.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:37:46.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.612+0000 7f53c3a15640 1 -- 172.21.15.107:0/3019364014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53bc107810 msgr2=0x7f53bc107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.612+0000 7f53c3a15640 1 --2- 172.21.15.107:0/3019364014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53bc107810 0x7f53bc107bf0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f53b8009930 tx=0x7f53b802a420 comp rx=0 tx=0).stop 2024-05-09T08:37:46.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.612+0000 7f53c3a15640 1 -- 172.21.15.107:0/3019364014 shutdown_connections 2024-05-09T08:37:46.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.612+0000 7f53c3a15640 1 --2- 172.21.15.107:0/3019364014 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53bc1021c0 0x7f53bc10e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.612+0000 7f53c3a15640 1 --2- 172.21.15.107:0/3019364014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53bc101800 0x7f53bc101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.612+0000 7f53c3a15640 1 --2- 172.21.15.107:0/3019364014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53bc107810 0x7f53bc107bf0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.612+0000 7f53c3a15640 1 -- 172.21.15.107:0/3019364014 >> 172.21.15.107:0/3019364014 conn(0x7f53bc078c80 msgr2=0x7f53bc079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.612+0000 7f53c3a15640 1 -- 172.21.15.107:0/3019364014 shutdown_connections 2024-05-09T08:37:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.613+0000 7f53c3a15640 1 -- 172.21.15.107:0/3019364014 wait complete. 2024-05-09T08:37:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.613+0000 7f53c3a15640 1 Processor -- start 2024-05-09T08:37:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.613+0000 7f53c3a15640 1 -- start start 2024-05-09T08:37:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.613+0000 7f53c3a15640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53bc101800 0x7f53bc198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.613+0000 7f53c3a15640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53bc1021c0 0x7f53bc198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.613+0000 7f53c3a15640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53bc107810 0x7f53bc19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.613+0000 7f53c3a15640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc111a90 con 0x7f53bc1021c0 2024-05-09T08:37:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.613+0000 7f53c3a15640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc111910 con 0x7f53bc107810 2024-05-09T08:37:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.613+0000 7f53c3a15640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc111c10 con 0x7f53bc101800 2024-05-09T08:37:46.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.613+0000 7f53c0fb0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53bc1021c0 0x7f53bc198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.613+0000 7f53c0fb0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53bc1021c0 0x7f53bc198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44938/0 (socket says 172.21.15.107:44938) 2024-05-09T08:37:46.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.613+0000 7f53c0fb0640 1 -- 172.21.15.107:0/2386602040 learned_addr learned my addr 172.21.15.107:0/2386602040 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:46.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.614+0000 7f53c17b1640 1 --2- 172.21.15.107:0/2386602040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53bc101800 0x7f53bc198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.614+0000 7f53c1fb2640 1 --2- 172.21.15.107:0/2386602040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53bc107810 0x7f53bc19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.614+0000 7f53c0fb0640 1 -- 172.21.15.107:0/2386602040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53bc101800 msgr2=0x7f53bc198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.614+0000 7f53c0fb0640 1 --2- 172.21.15.107:0/2386602040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53bc101800 0x7f53bc198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.614+0000 7f53c0fb0640 1 -- 172.21.15.107:0/2386602040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53bc107810 msgr2=0x7f53bc19c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.614+0000 7f53c0fb0640 1 --2- 172.21.15.107:0/2386602040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53bc107810 0x7f53bc19c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.614+0000 7f53c0fb0640 1 -- 172.21.15.107:0/2386602040 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53bc19d040 con 0x7f53bc1021c0 2024-05-09T08:37:46.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.614+0000 7f53c1fb2640 1 --2- 172.21.15.107:0/2386602040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53bc107810 0x7f53bc19c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:46.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.614+0000 7f53c0fb0640 1 --2- 172.21.15.107:0/2386602040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53bc1021c0 0x7f53bc198590 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00ea60 tx=0x7f53ac00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.614+0000 7f53b27fc640 1 -- 172.21.15.107:0/2386602040 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac00cde0 con 0x7f53bc1021c0 2024-05-09T08:37:46.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.614+0000 7f53c3a15640 1 -- 172.21.15.107:0/2386602040 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53bc19d330 con 0x7f53bc1021c0 2024-05-09T08:37:46.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.614+0000 7f53b27fc640 1 -- 172.21.15.107:0/2386602040 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53ac0035f0 con 0x7f53bc1021c0 2024-05-09T08:37:46.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.614+0000 7f53b27fc640 1 -- 172.21.15.107:0/2386602040 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac010690 con 0x7f53bc1021c0 2024-05-09T08:37:46.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.614+0000 7f53c3a15640 1 -- 172.21.15.107:0/2386602040 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53bc1a4bc0 con 0x7f53bc1021c0 2024-05-09T08:37:46.626 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:46 smithi110 ceph-mon[33622]: pgmap v135: 129 pgs: 129 active+clean; 4.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:37:46.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.617+0000 7f53b27fc640 1 -- 172.21.15.107:0/2386602040 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f53ac0037a0 con 0x7f53bc1021c0 2024-05-09T08:37:46.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.617+0000 7f53c67fe740 1 Processor -- start 2024-05-09T08:37:46.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.617+0000 7f53c67fe740 1 -- start start 2024-05-09T08:37:46.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53c67fe740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08af330 0x55cac08af710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53c67fe740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08afc50 0x55cac08b9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53c67fe740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08ba320 0x55cac08bc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53c67fe740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cac0814c40 con 0x55cac08af330 2024-05-09T08:37:46.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53c67fe740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cac07bc580 con 0x55cac08ba320 2024-05-09T08:37:46.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53c67fe740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55cac0810c00 con 0x55cac08afc50 2024-05-09T08:37:46.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53c1fb2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08af330 0x55cac08af710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53c1fb2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08af330 0x55cac08af710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44942/0 (socket says 172.21.15.107:44942) 2024-05-09T08:37:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53c1fb2640 1 -- 172.21.15.107:0/1700806002 learned_addr learned my addr 172.21.15.107:0/1700806002 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53c17b1640 1 --2- 172.21.15.107:0/1700806002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08afc50 0x55cac08b9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53c0fb0640 1 --2- 172.21.15.107:0/1700806002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08ba320 0x55cac08bc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53c1fb2640 1 -- 172.21.15.107:0/1700806002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08afc50 msgr2=0x55cac08b9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53c1fb2640 1 --2- 172.21.15.107:0/1700806002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08afc50 0x55cac08b9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53c1fb2640 1 -- 172.21.15.107:0/1700806002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08ba320 msgr2=0x55cac08bc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53c1fb2640 1 --2- 172.21.15.107:0/1700806002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08ba320 0x55cac08bc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53c1fb2640 1 -- 172.21.15.107:0/1700806002 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cac05283f0 con 0x55cac08af330 2024-05-09T08:37:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.618+0000 7f53b27fc640 1 --2- 172.21.15.107:0/2386602040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f53a80a2d00 0x7f53a80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.619+0000 7f53c17b1640 1 --2- 172.21.15.107:0/2386602040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f53a80a2d00 0x7f53a80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.619+0000 7f53c1fb2640 1 --2- 172.21.15.107:0/1700806002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08af330 0x55cac08af710 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009e40 tx=0x7f53b400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298bc3e587a5bec1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.619+0000 7f53b27fc640 1 -- 172.21.15.107:0/2386602040 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f53ac0cfb80 con 0x7f53bc1021c0 2024-05-09T08:37:46.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.619+0000 7f53a77fe640 1 -- 172.21.15.107:0/1700806002 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b4004d20 con 0x55cac08af330 2024-05-09T08:37:46.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.619+0000 7f53a77fe640 1 -- 172.21.15.107:0/1700806002 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53b40054d0 con 0x55cac08af330 2024-05-09T08:37:46.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.619+0000 7f53a77fe640 1 -- 172.21.15.107:0/1700806002 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b4011800 con 0x55cac08af330 2024-05-09T08:37:46.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 -- 172.21.15.107:0/1700806002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08af330 msgr2=0x55cac08af710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c17b1640 1 --2- 172.21.15.107:0/2386602040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f53a80a2d00 0x7f53a80a51c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f53b8000c00 tx=0x7f53b8005a40 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 --2- 172.21.15.107:0/1700806002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08af330 0x55cac08af710 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009e40 tx=0x7f53b400f730 comp rx=0 tx=0).stop 2024-05-09T08:37:46.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 -- 172.21.15.107:0/1700806002 shutdown_connections 2024-05-09T08:37:46.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 --2- 172.21.15.107:0/1700806002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08ba320 0x55cac08bc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 --2- 172.21.15.107:0/1700806002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08afc50 0x55cac08b9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 --2- 172.21.15.107:0/1700806002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08af330 0x55cac08af710 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 -- 172.21.15.107:0/1700806002 >> 172.21.15.107:0/1700806002 conn(0x55cac08ab650 msgr2=0x55cac08aba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 -- 172.21.15.107:0/1700806002 shutdown_connections 2024-05-09T08:37:46.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 -- 172.21.15.107:0/1700806002 wait complete. 2024-05-09T08:37:46.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 -- start start 2024-05-09T08:37:46.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 Processor -- start 2024-05-09T08:37:46.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08af330 0x55cac08b7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08afc50 0x55cac08b7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08ba320 0x55cac08b7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cac08bd110 con 0x55cac08af330 2024-05-09T08:37:46.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cac07cfc40 con 0x55cac08ba320 2024-05-09T08:37:46.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.620+0000 7f53c67fe740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55cac08bd290 con 0x55cac08afc50 2024-05-09T08:37:46.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.621+0000 7f53c67fe740 10 client.0 _ll_register_callbacks cb 0x55cac0527be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:46.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.621+0000 7f53c17b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08af330 0x55cac08b7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.621+0000 7f53c1fb2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08ba320 0x55cac08b7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.621+0000 7f53c17b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08af330 0x55cac08b7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44950/0 (socket says 172.21.15.107:44950) 2024-05-09T08:37:46.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.621+0000 7f53c17b1640 1 -- 172.21.15.107:0/2715860777 learned_addr learned my addr 172.21.15.107:0/2715860777 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:46.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.621+0000 7f53c0fb0640 1 --2- 172.21.15.107:0/2715860777 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08afc50 0x55cac08b7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.621+0000 7f53c1fb2640 1 -- 172.21.15.107:0/2715860777 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08afc50 msgr2=0x55cac08b7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.621+0000 7f53c1fb2640 1 --2- 172.21.15.107:0/2715860777 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08afc50 0x55cac08b7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.621+0000 7f53c1fb2640 1 -- 172.21.15.107:0/2715860777 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08af330 msgr2=0x55cac08b7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.621+0000 7f53c1fb2640 1 --2- 172.21.15.107:0/2715860777 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08af330 0x55cac08b7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.621+0000 7f53c1fb2640 1 -- 172.21.15.107:0/2715860777 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cac05f27d0 con 0x55cac08ba320 2024-05-09T08:37:46.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.621+0000 7f53c1fb2640 1 --2- 172.21.15.107:0/2715860777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08ba320 0x55cac08b7cc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f53b4000f80 tx=0x7f53b40058b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6af43819634ef9d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.621+0000 7f53a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:37:46.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.621+0000 7f53a4ff9640 1 -- 172.21.15.107:0/2715860777 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b4028060 con 0x55cac08ba320 2024-05-09T08:37:46.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.621+0000 7f53c67fe740 1 -- 172.21.15.107:0/2715860777 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53b40054d0 con 0x55cac08ba320 2024-05-09T08:37:46.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.621+0000 7f53a4ff9640 1 -- 172.21.15.107:0/2715860777 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f53b4005210 con 0x55cac08ba320 2024-05-09T08:37:46.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.622+0000 7f53a4ff9640 1 -- 172.21.15.107:0/2715860777 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f53b402f050 con 0x55cac08ba320 2024-05-09T08:37:46.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.622+0000 7f53c67fe740 10 client.26126 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.622+0000 7f53c67fe740 1 -- 172.21.15.107:0/2715860777 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cac08c92a0 con 0x55cac08ba320 2024-05-09T08:37:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.622+0000 7f53a4ff9640 1 -- 172.21.15.107:0/2715860777 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53b40054d0 con 0x55cac08ba320 2024-05-09T08:37:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.622+0000 7f53a4ff9640 1 -- 172.21.15.107:0/2715860777 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f53b4011a40 con 0x55cac08ba320 2024-05-09T08:37:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.622+0000 7f53c67fe740 10 client.26126 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.622+0000 7f53c67fe740 10 client.26126 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:37:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.622+0000 7f53c67fe740 20 client.26126 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.622+0000 7f53c67fe740 1 --2- 172.21.15.107:0/2715860777 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cac08ca700 0x55cac08ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.622+0000 7f53c67fe740 4 client.26126 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.622+0000 7f53c67fe740 1 -- 172.21.15.107:0/2715860777 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cac0822370 con 0x55cac08ca700 2024-05-09T08:37:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.622+0000 7f53c17b1640 1 --2- 172.21.15.107:0/2715860777 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cac08ca700 0x55cac08ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.623+0000 7f53a4ff9640 10 client.26126 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:37:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.623+0000 7f53c17b1640 1 --2- 172.21.15.107:0/2715860777 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cac08ca700 0x55cac08ccaf0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2b025d52a470e0c3 server_cookie=bab2c0c870552027 in_seq=0 out_seq=0 2024-05-09T08:37:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.623+0000 7f53a4ff9640 1 -- 172.21.15.107:0/2715860777 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cac0822370 con 0x55cac08ca700 2024-05-09T08:37:46.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.623+0000 7f53a4ff9640 10 client.26126 handle_command_reply: tid=0 2024-05-09T08:37:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.623+0000 7f53c67fe740 1 client.26126 shutdown 2024-05-09T08:37:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.623+0000 7f53c67fe740 1 -- 172.21.15.107:0/2715860777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08ba320 msgr2=0x55cac08b7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.623+0000 7f53c67fe740 1 --2- 172.21.15.107:0/2715860777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08ba320 0x55cac08b7cc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f53b4000f80 tx=0x7f53b40058b0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.623+0000 7f53c67fe740 1 -- 172.21.15.107:0/2715860777 shutdown_connections 2024-05-09T08:37:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.623+0000 7f53c67fe740 1 --2- 172.21.15.107:0/2715860777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08ba320 0x55cac08b7cc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.623+0000 7f53c67fe740 1 --2- 172.21.15.107:0/2715860777 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cac08ca700 0x55cac08ccaf0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.623+0000 7f53c67fe740 1 --2- 172.21.15.107:0/2715860777 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08afc50 0x55cac08b7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.623+0000 7f53a4ff9640 0 client.26126 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:37:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.623+0000 7f53c67fe740 1 --2- 172.21.15.107:0/2715860777 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08af330 0x55cac08b7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.623+0000 7f53c67fe740 1 -- 172.21.15.107:0/2715860777 >> 172.21.15.107:0/2715860777 conn(0x55cac08ab650 msgr2=0x55cac08b1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.624+0000 7f53c67fe740 1 -- 172.21.15.107:0/2715860777 shutdown_connections 2024-05-09T08:37:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.624+0000 7f53c67fe740 1 -- 172.21.15.107:0/2715860777 wait complete. 2024-05-09T08:37:46.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.624+0000 7f53c67fe740 20 client.26126 trim_cache size 0 max 16384 2024-05-09T08:37:46.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.637+0000 7f53c67fe740 1 Processor -- start 2024-05-09T08:37:46.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.637+0000 7f53c67fe740 1 -- start start 2024-05-09T08:37:46.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.637+0000 7f53c67fe740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08af330 0x55cac08c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.637+0000 7f53c67fe740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08afc50 0x55cac08d2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.637+0000 7f53c67fe740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08ba320 0x55cac08d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53c67fe740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc1b1550 con 0x55cac08ba320 2024-05-09T08:37:46.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53c67fe740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc1aee20 con 0x55cac08afc50 2024-05-09T08:37:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53c67fe740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc1acc70 con 0x55cac08af330 2024-05-09T08:37:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53c1fb2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08ba320 0x55cac08d25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53c1fb2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08ba320 0x55cac08d25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44962/0 (socket says 172.21.15.107:44962) 2024-05-09T08:37:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53c1fb2640 1 -- 172.21.15.107:0/4049907338 learned_addr learned my addr 172.21.15.107:0/4049907338 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53c17b1640 1 --2- 172.21.15.107:0/4049907338 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08af330 0x55cac08c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53c0fb0640 1 --2- 172.21.15.107:0/4049907338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08afc50 0x55cac08d2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53c1fb2640 1 -- 172.21.15.107:0/4049907338 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08af330 msgr2=0x55cac08c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53c1fb2640 1 --2- 172.21.15.107:0/4049907338 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08af330 0x55cac08c5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53c1fb2640 1 -- 172.21.15.107:0/4049907338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08afc50 msgr2=0x55cac08d2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53c1fb2640 1 --2- 172.21.15.107:0/4049907338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08afc50 0x55cac08d2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53c1fb2640 1 -- 172.21.15.107:0/4049907338 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cac08afa80 con 0x55cac08ba320 2024-05-09T08:37:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53c0fb0640 1 --2- 172.21.15.107:0/4049907338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08afc50 0x55cac08d2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:46.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53c1fb2640 1 --2- 172.21.15.107:0/4049907338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08ba320 0x55cac08d25d0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f53b400a010 tx=0x7f53b4019430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d5aed3603ada08f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53a6ffd640 1 -- 172.21.15.107:0/4049907338 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b4028070 con 0x55cac08ba320 2024-05-09T08:37:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53a6ffd640 1 -- 172.21.15.107:0/4049907338 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53b4005920 con 0x55cac08ba320 2024-05-09T08:37:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.638+0000 7f53a6ffd640 1 -- 172.21.15.107:0/4049907338 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b4019880 con 0x55cac08ba320 2024-05-09T08:37:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.639+0000 7f53c67fe740 1 -- 172.21.15.107:0/4049907338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08ba320 msgr2=0x55cac08d25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.639+0000 7f53c67fe740 1 --2- 172.21.15.107:0/4049907338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08ba320 0x55cac08d25d0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f53b400a010 tx=0x7f53b4019430 comp rx=0 tx=0).stop 2024-05-09T08:37:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.639+0000 7f53c67fe740 1 -- 172.21.15.107:0/4049907338 shutdown_connections 2024-05-09T08:37:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.639+0000 7f53c67fe740 1 --2- 172.21.15.107:0/4049907338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08ba320 0x55cac08d25d0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.639+0000 7f53c67fe740 1 --2- 172.21.15.107:0/4049907338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08afc50 0x55cac08d2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.639+0000 7f53c67fe740 1 --2- 172.21.15.107:0/4049907338 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08af330 0x55cac08c5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.639+0000 7f53c67fe740 1 -- 172.21.15.107:0/4049907338 >> 172.21.15.107:0/4049907338 conn(0x55cac08ab650 msgr2=0x55cac08aa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.639+0000 7f53c67fe740 1 -- 172.21.15.107:0/4049907338 shutdown_connections 2024-05-09T08:37:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.639+0000 7f53c67fe740 1 -- 172.21.15.107:0/4049907338 wait complete. 2024-05-09T08:37:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.639+0000 7f53c67fe740 1 -- start start 2024-05-09T08:37:46.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.639+0000 7f53c67fe740 1 Processor -- start 2024-05-09T08:37:46.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c67fe740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08af330 0x55cac08bdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c67fe740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08afc50 0x55cac08be460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c67fe740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08ba320 0x55cac08be9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c67fe740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc1ad2e0 con 0x55cac08ba320 2024-05-09T08:37:46.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c67fe740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc1ae750 con 0x55cac08af330 2024-05-09T08:37:46.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c67fe740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc1ad9b0 con 0x55cac08afc50 2024-05-09T08:37:46.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c67fe740 10 client.0 _ll_register_callbacks cb 0x7f53ac0d35a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:46.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c1fb2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08ba320 0x55cac08be9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c1fb2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08ba320 0x55cac08be9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44974/0 (socket says 172.21.15.107:44974) 2024-05-09T08:37:46.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c1fb2640 1 -- 172.21.15.107:0/3699477325 learned_addr learned my addr 172.21.15.107:0/3699477325 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:46.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c0fb0640 1 --2- 172.21.15.107:0/3699477325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08afc50 0x55cac08be460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c17b1640 1 --2- 172.21.15.107:0/3699477325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08af330 0x55cac08bdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c1fb2640 1 -- 172.21.15.107:0/3699477325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08afc50 msgr2=0x55cac08be460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c1fb2640 1 --2- 172.21.15.107:0/3699477325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08afc50 0x55cac08be460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c1fb2640 1 -- 172.21.15.107:0/3699477325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08af330 msgr2=0x55cac08bdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c1fb2640 1 --2- 172.21.15.107:0/3699477325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08af330 0x55cac08bdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.640+0000 7f53c1fb2640 1 -- 172.21.15.107:0/3699477325 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cac092f900 con 0x55cac08ba320 2024-05-09T08:37:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.641+0000 7f53c1fb2640 1 --2- 172.21.15.107:0/3699477325 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08ba320 0x55cac08be9a0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f53b4004f70 tx=0x7f53b4011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6786aa0fd5f7dd4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.641+0000 7f53a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.641+0000 7f53a4ff9640 1 -- 172.21.15.107:0/3699477325 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b4028070 con 0x55cac08ba320 2024-05-09T08:37:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.641+0000 7f53a4ff9640 1 -- 172.21.15.107:0/3699477325 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f53b4005210 con 0x55cac08ba320 2024-05-09T08:37:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.641+0000 7f53c67fe740 1 -- 172.21.15.107:0/3699477325 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53b4005920 con 0x55cac08ba320 2024-05-09T08:37:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.641+0000 7f53a4ff9640 1 -- 172.21.15.107:0/3699477325 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f53b4005410 con 0x55cac08ba320 2024-05-09T08:37:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.641+0000 7f53c67fe740 10 client.16959 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.641+0000 7f53c67fe740 1 -- 172.21.15.107:0/3699477325 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cac0930b40 con 0x55cac08ba320 2024-05-09T08:37:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.641+0000 7f53a4ff9640 1 -- 172.21.15.107:0/3699477325 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53b4005920 con 0x55cac08ba320 2024-05-09T08:37:46.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.641+0000 7f53a4ff9640 1 -- 172.21.15.107:0/3699477325 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f53b4011440 con 0x55cac08ba320 2024-05-09T08:37:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.641+0000 7f53c67fe740 10 client.16959 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.641+0000 7f53c67fe740 10 client.16959 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:37:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.641+0000 7f53c67fe740 20 client.16959 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.641+0000 7f53c67fe740 1 --2- 172.21.15.107:0/3699477325 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cac0931170 0x55cac0933590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.641+0000 7f53c67fe740 4 client.16959 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:37:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.641+0000 7f53c67fe740 1 -- 172.21.15.107:0/3699477325 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cac081c6c0 con 0x55cac0931170 2024-05-09T08:37:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53c17b1640 1 --2- 172.21.15.107:0/3699477325 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cac0931170 0x55cac0933590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53a4ff9640 10 client.16959 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:37:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53c17b1640 1 --2- 172.21.15.107:0/3699477325 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cac0931170 0x55cac0933590 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f8b1d63e640dbacd server_cookie=6a504b9c51e86463 in_seq=0 out_seq=0 2024-05-09T08:37:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53a4ff9640 1 -- 172.21.15.107:0/3699477325 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cac081c6c0 con 0x55cac0931170 2024-05-09T08:37:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53a4ff9640 10 client.16959 handle_command_reply: tid=0 2024-05-09T08:37:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53c67fe740 1 client.16959 shutdown 2024-05-09T08:37:46.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53c67fe740 1 -- 172.21.15.107:0/3699477325 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08ba320 msgr2=0x55cac08be9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53c67fe740 1 --2- 172.21.15.107:0/3699477325 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08ba320 0x55cac08be9a0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f53b4004f70 tx=0x7f53b4011f00 comp rx=0 tx=0).stop 2024-05-09T08:37:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53c67fe740 1 -- 172.21.15.107:0/3699477325 shutdown_connections 2024-05-09T08:37:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53c67fe740 1 --2- 172.21.15.107:0/3699477325 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cac08ba320 0x55cac08be9a0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53c67fe740 1 --2- 172.21.15.107:0/3699477325 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cac0931170 0x55cac0933590 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53c67fe740 1 --2- 172.21.15.107:0/3699477325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac08afc50 0x55cac08be460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53c67fe740 1 --2- 172.21.15.107:0/3699477325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cac08af330 0x55cac08bdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53c67fe740 1 -- 172.21.15.107:0/3699477325 >> 172.21.15.107:0/3699477325 conn(0x55cac08ab650 msgr2=0x55cac08b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53a4ff9640 0 client.16959 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:37:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53c67fe740 1 -- 172.21.15.107:0/3699477325 shutdown_connections 2024-05-09T08:37:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.642+0000 7f53c67fe740 1 -- 172.21.15.107:0/3699477325 wait complete. 2024-05-09T08:37:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.643+0000 7f53c67fe740 20 client.16959 trim_cache size 0 max 16384 2024-05-09T08:37:46.650 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:37:46.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.643+0000 7f53c3a15640 1 -- 172.21.15.107:0/2386602040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f53a80a2d00 msgr2=0x7f53a80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.643+0000 7f53c3a15640 1 --2- 172.21.15.107:0/2386602040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f53a80a2d00 0x7f53a80a51c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f53b8000c00 tx=0x7f53b8005a40 comp rx=0 tx=0).stop 2024-05-09T08:37:46.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.643+0000 7f53c3a15640 1 -- 172.21.15.107:0/2386602040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53bc1021c0 msgr2=0x7f53bc198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.643+0000 7f53c3a15640 1 --2- 172.21.15.107:0/2386602040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53bc1021c0 0x7f53bc198590 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00ea60 tx=0x7f53ac00ef20 comp rx=0 tx=0).stop 2024-05-09T08:37:46.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.643+0000 7f53c3a15640 1 -- 172.21.15.107:0/2386602040 shutdown_connections 2024-05-09T08:37:46.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.643+0000 7f53c3a15640 1 --2- 172.21.15.107:0/2386602040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f53a80a2d00 0x7f53a80a51c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.643+0000 7f53c3a15640 1 --2- 172.21.15.107:0/2386602040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53bc107810 0x7f53bc19c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.643+0000 7f53c3a15640 1 --2- 172.21.15.107:0/2386602040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53bc1021c0 0x7f53bc198590 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.643+0000 7f53c3a15640 1 --2- 172.21.15.107:0/2386602040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53bc101800 0x7f53bc198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.643+0000 7f53c3a15640 1 -- 172.21.15.107:0/2386602040 >> 172.21.15.107:0/2386602040 conn(0x7f53bc078c80 msgr2=0x7f53bc0ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.643+0000 7f53c3a15640 1 -- 172.21.15.107:0/2386602040 shutdown_connections 2024-05-09T08:37:46.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.643+0000 7f53c3a15640 1 -- 172.21.15.107:0/2386602040 wait complete. 2024-05-09T08:37:46.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:37:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.773+0000 7fefa851b640 1 -- 172.21.15.107:0/2607753498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefa0100fb0 msgr2=0x7fefa0104ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.773+0000 7fefa851b640 1 --2- 172.21.15.107:0/2607753498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefa0100fb0 0x7fefa0104ad0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fef9000aa10 tx=0x7fef9002b720 comp rx=0 tx=0).stop 2024-05-09T08:37:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.773+0000 7fefa851b640 1 -- 172.21.15.107:0/2607753498 shutdown_connections 2024-05-09T08:37:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.773+0000 7fefa851b640 1 --2- 172.21.15.107:0/2607753498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefa01051b0 0x7fefa010cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.773+0000 7fefa851b640 1 --2- 172.21.15.107:0/2607753498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefa0100fb0 0x7fefa0104ad0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.773+0000 7fefa851b640 1 --2- 172.21.15.107:0/2607753498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefa0100600 0x7fefa01009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.773+0000 7fefa851b640 1 -- 172.21.15.107:0/2607753498 >> 172.21.15.107:0/2607753498 conn(0x7fefa0075a30 msgr2=0x7fefa0075e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.773+0000 7fefa851b640 1 -- 172.21.15.107:0/2607753498 shutdown_connections 2024-05-09T08:37:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.773+0000 7fefa851b640 1 -- 172.21.15.107:0/2607753498 wait complete. 2024-05-09T08:37:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa851b640 1 Processor -- start 2024-05-09T08:37:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa851b640 1 -- start start 2024-05-09T08:37:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa851b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefa0100600 0x7fefa0195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa851b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefa01051b0 0x7fefa0196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa851b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefa019a6f0 0x7fefa019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa851b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fefa010f850 con 0x7fefa0100600 2024-05-09T08:37:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa851b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fefa010f6d0 con 0x7fefa01051b0 2024-05-09T08:37:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa851b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fefa010f9d0 con 0x7fefa019a6f0 2024-05-09T08:37:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa62b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefa0100600 0x7fefa0195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa62b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefa0100600 0x7fefa0195e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44988/0 (socket says 172.21.15.107:44988) 2024-05-09T08:37:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa62b7640 1 -- 172.21.15.107:0/3709638651 learned_addr learned my addr 172.21.15.107:0/3709638651 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa5ab6640 1 --2- 172.21.15.107:0/3709638651 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefa01051b0 0x7fefa0196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa6ab8640 1 --2- 172.21.15.107:0/3709638651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefa019a6f0 0x7fefa019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa62b7640 1 -- 172.21.15.107:0/3709638651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefa019a6f0 msgr2=0x7fefa019aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa62b7640 1 --2- 172.21.15.107:0/3709638651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefa019a6f0 0x7fefa019aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa62b7640 1 -- 172.21.15.107:0/3709638651 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefa01051b0 msgr2=0x7fefa0196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa62b7640 1 --2- 172.21.15.107:0/3709638651 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefa01051b0 0x7fefa0196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.774+0000 7fefa62b7640 1 -- 172.21.15.107:0/3709638651 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa019b1e0 con 0x7fefa0100600 2024-05-09T08:37:46.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.775+0000 7fefa62b7640 1 --2- 172.21.15.107:0/3709638651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefa0100600 0x7fefa0195e30 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fef9c00d950 tx=0x7fef9c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.775+0000 7fef977fe640 1 -- 172.21.15.107:0/3709638651 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c017070 con 0x7fefa0100600 2024-05-09T08:37:46.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.775+0000 7fef977fe640 1 -- 172.21.15.107:0/3709638651 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef9c009c30 con 0x7fefa0100600 2024-05-09T08:37:46.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.775+0000 7fef977fe640 1 -- 172.21.15.107:0/3709638651 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c00f620 con 0x7fefa0100600 2024-05-09T08:37:46.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.775+0000 7fefa851b640 1 -- 172.21.15.107:0/3709638651 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefa019b4d0 con 0x7fefa0100600 2024-05-09T08:37:46.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.775+0000 7fefa851b640 1 -- 172.21.15.107:0/3709638651 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefa01a2ce0 con 0x7fefa0100600 2024-05-09T08:37:46.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.777+0000 7fef977fe640 1 -- 172.21.15.107:0/3709638651 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fef9c0109a0 con 0x7fefa0100600 2024-05-09T08:37:46.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.778+0000 7fefab22e740 1 Processor -- start 2024-05-09T08:37:46.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.778+0000 7fef977fe640 1 --2- 172.21.15.107:0/3709638651 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fef8c0a2d80 0x7fef8c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.778+0000 7fef977fe640 1 -- 172.21.15.107:0/3709638651 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fef9c0cf2d0 con 0x7fefa0100600 2024-05-09T08:37:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.778+0000 7fefa6ab8640 1 --2- 172.21.15.107:0/3709638651 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fef8c0a2d80 0x7fef8c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.778+0000 7fefab22e740 1 -- start start 2024-05-09T08:37:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.778+0000 7fefab22e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d73499380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.778+0000 7fefab22e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d73499950 0x557d734a5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.778+0000 7fefab22e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d734a6530 0x557d734a8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.778+0000 7fefab22e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557d733fec40 con 0x557d73498fa0 2024-05-09T08:37:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.778+0000 7fefab22e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557d733a6580 con 0x557d73499950 2024-05-09T08:37:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.778+0000 7fefab22e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x557d733fac00 con 0x557d734a6530 2024-05-09T08:37:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.779+0000 7fefa62b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d73499380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.779+0000 7fefa6ab8640 1 --2- 172.21.15.107:0/3709638651 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fef8c0a2d80 0x7fef8c0a5240 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fef98005fd0 tx=0x7fef98009500 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.779+0000 7fefa62b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d73499380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45000/0 (socket says 172.21.15.107:45000) 2024-05-09T08:37:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.779+0000 7fefa62b7640 1 -- 172.21.15.107:0/3815604053 learned_addr learned my addr 172.21.15.107:0/3815604053 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.779+0000 7fefa6ab8640 1 --2- 172.21.15.107:0/3815604053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d734a6530 0x557d734a8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.779+0000 7fefa5ab6640 1 --2- 172.21.15.107:0/3815604053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d73499950 0x557d734a5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.779+0000 7fefa62b7640 1 -- 172.21.15.107:0/3815604053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d734a6530 msgr2=0x557d734a8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.779+0000 7fefa62b7640 1 --2- 172.21.15.107:0/3815604053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d734a6530 0x557d734a8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.779+0000 7fefa62b7640 1 -- 172.21.15.107:0/3815604053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d73499950 msgr2=0x557d734a5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.779+0000 7fefa62b7640 1 --2- 172.21.15.107:0/3815604053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d73499950 0x557d734a5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.779+0000 7fefa62b7640 1 -- 172.21.15.107:0/3815604053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d731123f0 con 0x557d73498fa0 2024-05-09T08:37:46.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.779+0000 7fefa5ab6640 1 --2- 172.21.15.107:0/3815604053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d73499950 0x557d734a5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:46.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.779+0000 7fefa62b7640 1 --2- 172.21.15.107:0/3815604053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d73499380 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fef9c097f10 tx=0x7fef9c09af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9b957bb477bbe85 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.779+0000 7fef94bfe640 1 -- 172.21.15.107:0/3815604053 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c020040 con 0x557d73498fa0 2024-05-09T08:37:46.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.779+0000 7fef94bfe640 1 -- 172.21.15.107:0/3815604053 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef9c0a4100 con 0x557d73498fa0 2024-05-09T08:37:46.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.779+0000 7fef94bfe640 1 -- 172.21.15.107:0/3815604053 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c097490 con 0x557d73498fa0 2024-05-09T08:37:46.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.780+0000 7fefab22e740 1 -- 172.21.15.107:0/3815604053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 msgr2=0x557d73499380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.780+0000 7fefab22e740 1 --2- 172.21.15.107:0/3815604053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d73499380 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fef9c097f10 tx=0x7fef9c09af70 comp rx=0 tx=0).stop 2024-05-09T08:37:46.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.780+0000 7fefab22e740 1 -- 172.21.15.107:0/3815604053 shutdown_connections 2024-05-09T08:37:46.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.780+0000 7fefab22e740 1 --2- 172.21.15.107:0/3815604053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d734a6530 0x557d734a8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.780+0000 7fefab22e740 1 --2- 172.21.15.107:0/3815604053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d73499950 0x557d734a5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.780+0000 7fefab22e740 1 --2- 172.21.15.107:0/3815604053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d73499380 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.780+0000 7fefab22e740 1 -- 172.21.15.107:0/3815604053 >> 172.21.15.107:0/3815604053 conn(0x557d73495650 msgr2=0x557d73495a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.780+0000 7fefab22e740 1 -- 172.21.15.107:0/3815604053 shutdown_connections 2024-05-09T08:37:46.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.780+0000 7fefab22e740 1 -- 172.21.15.107:0/3815604053 wait complete. 2024-05-09T08:37:46.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.781+0000 7fefab22e740 1 -- start start 2024-05-09T08:37:46.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.781+0000 7fefab22e740 1 Processor -- start 2024-05-09T08:37:46.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.781+0000 7fefab22e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d734a3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.781+0000 7fefab22e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d73499950 0x557d734a3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.781+0000 7fefab22e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d734a6530 0x557d734a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.781+0000 7fefab22e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557d734a9260 con 0x557d73498fa0 2024-05-09T08:37:46.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.781+0000 7fefab22e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557d733b9c40 con 0x557d734a6530 2024-05-09T08:37:46.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.781+0000 7fefab22e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x557d734a93e0 con 0x557d73499950 2024-05-09T08:37:46.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.781+0000 7fefab22e740 10 client.0 _ll_register_callbacks cb 0x557d73111be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:46.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.781+0000 7fefa62b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d734a3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.781+0000 7fefa62b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d734a3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45016/0 (socket says 172.21.15.107:45016) 2024-05-09T08:37:46.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.781+0000 7fefa62b7640 1 -- 172.21.15.107:0/2796255009 learned_addr learned my addr 172.21.15.107:0/2796255009 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:46.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.781+0000 7fefa5ab6640 1 --2- 172.21.15.107:0/2796255009 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d73499950 0x557d734a3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.781+0000 7fefa6ab8640 1 --2- 172.21.15.107:0/2796255009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d734a6530 0x557d734a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fefa62b7640 1 -- 172.21.15.107:0/2796255009 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d73499950 msgr2=0x557d734a3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fefa62b7640 1 --2- 172.21.15.107:0/2796255009 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d73499950 0x557d734a3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fefa62b7640 1 -- 172.21.15.107:0/2796255009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d734a6530 msgr2=0x557d734a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fefa62b7640 1 --2- 172.21.15.107:0/2796255009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d734a6530 0x557d734a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fefa62b7640 1 -- 172.21.15.107:0/2796255009 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d731dc7d0 con 0x557d73498fa0 2024-05-09T08:37:46.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fefa5ab6640 1 --2- 172.21.15.107:0/2796255009 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d73499950 0x557d734a3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:46.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fefa62b7640 1 --2- 172.21.15.107:0/2796255009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d734a3500 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0cf5f0 tx=0x7fef9c0ccbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7024568f0314f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fef89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:46.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fef89ffb640 1 -- 172.21.15.107:0/2796255009 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c020070 con 0x557d73498fa0 2024-05-09T08:37:46.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fef89ffb640 1 -- 172.21.15.107:0/2796255009 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fef9c09aba0 con 0x557d73498fa0 2024-05-09T08:37:46.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fefab22e740 1 -- 172.21.15.107:0/2796255009 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef9c0a4100 con 0x557d73498fa0 2024-05-09T08:37:46.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fef89ffb640 1 -- 172.21.15.107:0/2796255009 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fef9c082050 con 0x557d73498fa0 2024-05-09T08:37:46.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fefab22e740 10 client.16971 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:46.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fefab22e740 1 -- 172.21.15.107:0/2796255009 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557d734b54e0 con 0x557d73498fa0 2024-05-09T08:37:46.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fef89ffb640 1 -- 172.21.15.107:0/2796255009 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef9c0a4100 con 0x557d73498fa0 2024-05-09T08:37:46.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fef89ffb640 1 -- 172.21.15.107:0/2796255009 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fef9c010460 con 0x557d73498fa0 2024-05-09T08:37:46.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fefab22e740 10 client.16971 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:46.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fefab22e740 10 client.16971 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:46.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.782+0000 7fefab22e740 20 client.16971 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:46.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.783+0000 7fefab22e740 1 --2- 172.21.15.107:0/2796255009 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557d734b6830 0x557d734b8c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.783+0000 7fefab22e740 4 client.16971 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:46.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.783+0000 7fefab22e740 1 -- 172.21.15.107:0/2796255009 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d7340c370 con 0x557d734b6830 2024-05-09T08:37:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.783+0000 7fefa5ab6640 1 --2- 172.21.15.107:0/2796255009 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557d734b6830 0x557d734b8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.783+0000 7fef89ffb640 10 client.16971 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.783+0000 7fefa5ab6640 1 --2- 172.21.15.107:0/2796255009 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557d734b6830 0x557d734b8c20 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3965f9e4c65b295f server_cookie=c28a51d183dd3319 in_seq=0 out_seq=0 2024-05-09T08:37:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.784+0000 7fef89ffb640 1 -- 172.21.15.107:0/2796255009 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557d7340c370 con 0x557d734b6830 2024-05-09T08:37:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.784+0000 7fef89ffb640 10 client.16971 handle_command_reply: tid=0 2024-05-09T08:37:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.784+0000 7fefab22e740 1 client.16971 shutdown 2024-05-09T08:37:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.784+0000 7fefab22e740 1 -- 172.21.15.107:0/2796255009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 msgr2=0x557d734a3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.784+0000 7fefab22e740 1 --2- 172.21.15.107:0/2796255009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d734a3500 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0cf5f0 tx=0x7fef9c0ccbf0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.784+0000 7fefab22e740 1 -- 172.21.15.107:0/2796255009 shutdown_connections 2024-05-09T08:37:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.784+0000 7fefab22e740 1 --2- 172.21.15.107:0/2796255009 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557d734b6830 0x557d734b8c20 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.784+0000 7fefab22e740 1 --2- 172.21.15.107:0/2796255009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d734a6530 0x557d734a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.784+0000 7fefab22e740 1 --2- 172.21.15.107:0/2796255009 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d73499950 0x557d734a3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.784+0000 7fefab22e740 1 --2- 172.21.15.107:0/2796255009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d734a3500 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.784+0000 7fefab22e740 1 -- 172.21.15.107:0/2796255009 >> 172.21.15.107:0/2796255009 conn(0x557d73495650 msgr2=0x557d7349b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.784+0000 7fef89ffb640 0 client.16971 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:46.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.784+0000 7fefab22e740 1 -- 172.21.15.107:0/2796255009 shutdown_connections 2024-05-09T08:37:46.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.784+0000 7fefab22e740 1 -- 172.21.15.107:0/2796255009 wait complete. 2024-05-09T08:37:46.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.784+0000 7fefab22e740 20 client.16971 trim_cache size 0 max 16384 2024-05-09T08:37:46.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefab22e740 1 Processor -- start 2024-05-09T08:37:46.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefab22e740 1 -- start start 2024-05-09T08:37:46.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefab22e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d7349b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefab22e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d73499950 0x557d734a5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefab22e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d734a6530 0x557d734bb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefab22e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fefa01af170 con 0x557d73498fa0 2024-05-09T08:37:46.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefab22e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fefa01aca40 con 0x557d73499950 2024-05-09T08:37:46.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefab22e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fefa01aa890 con 0x557d734a6530 2024-05-09T08:37:46.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefa62b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d7349b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefa62b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d7349b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45028/0 (socket says 172.21.15.107:45028) 2024-05-09T08:37:46.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefa62b7640 1 -- 172.21.15.107:0/2251136148 learned_addr learned my addr 172.21.15.107:0/2251136148 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:46.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefa6ab8640 1 --2- 172.21.15.107:0/2251136148 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d734a6530 0x557d734bb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefa5ab6640 1 --2- 172.21.15.107:0/2251136148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d73499950 0x557d734a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefa62b7640 1 -- 172.21.15.107:0/2251136148 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d734a6530 msgr2=0x557d734bb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefa62b7640 1 --2- 172.21.15.107:0/2251136148 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d734a6530 0x557d734bb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefa62b7640 1 -- 172.21.15.107:0/2251136148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d73499950 msgr2=0x557d734a5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefa62b7640 1 --2- 172.21.15.107:0/2251136148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d73499950 0x557d734a5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefa62b7640 1 -- 172.21.15.107:0/2251136148 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d73490f30 con 0x557d73498fa0 2024-05-09T08:37:46.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.804+0000 7fefa62b7640 1 --2- 172.21.15.107:0/2251136148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d7349b7a0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fef9c09b240 tx=0x7fef9c0cf220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c1c7d727487bd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.805+0000 7fef8bfff640 1 -- 172.21.15.107:0/2251136148 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c020030 con 0x557d73498fa0 2024-05-09T08:37:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.805+0000 7fef8bfff640 1 -- 172.21.15.107:0/2251136148 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef9c09aba0 con 0x557d73498fa0 2024-05-09T08:37:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.805+0000 7fef8bfff640 1 -- 172.21.15.107:0/2251136148 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c010460 con 0x557d73498fa0 2024-05-09T08:37:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.805+0000 7fefab22e740 1 -- 172.21.15.107:0/2251136148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 msgr2=0x557d7349b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.805+0000 7fefab22e740 1 --2- 172.21.15.107:0/2251136148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d7349b7a0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fef9c09b240 tx=0x7fef9c0cf220 comp rx=0 tx=0).stop 2024-05-09T08:37:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.805+0000 7fefab22e740 1 -- 172.21.15.107:0/2251136148 shutdown_connections 2024-05-09T08:37:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.805+0000 7fefab22e740 1 --2- 172.21.15.107:0/2251136148 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d734a6530 0x557d734bb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.805+0000 7fefab22e740 1 --2- 172.21.15.107:0/2251136148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d73499950 0x557d734a5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.805+0000 7fefab22e740 1 --2- 172.21.15.107:0/2251136148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d7349b7a0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.805+0000 7fefab22e740 1 -- 172.21.15.107:0/2251136148 >> 172.21.15.107:0/2251136148 conn(0x557d73495650 msgr2=0x557d734a4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.805+0000 7fefab22e740 1 -- 172.21.15.107:0/2251136148 shutdown_connections 2024-05-09T08:37:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.805+0000 7fefab22e740 1 -- 172.21.15.107:0/2251136148 wait complete. 2024-05-09T08:37:46.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefab22e740 1 -- start start 2024-05-09T08:37:46.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefab22e740 1 Processor -- start 2024-05-09T08:37:46.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefab22e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d734bb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefab22e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d73499950 0x557d734b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefab22e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d734a6530 0x557d734b92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefab22e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fefa01aaf00 con 0x557d73498fa0 2024-05-09T08:37:46.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefab22e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fefa01ac370 con 0x557d734a6530 2024-05-09T08:37:46.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefab22e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fefa01ab5d0 con 0x557d73499950 2024-05-09T08:37:46.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefab22e740 10 client.0 _ll_register_callbacks cb 0x7fef9c09b400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:46.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefa62b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d734bb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefa62b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d734bb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45032/0 (socket says 172.21.15.107:45032) 2024-05-09T08:37:46.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefa62b7640 1 -- 172.21.15.107:0/4087123877 learned_addr learned my addr 172.21.15.107:0/4087123877 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:46.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefa6ab8640 1 --2- 172.21.15.107:0/4087123877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d734a6530 0x557d734b92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefa5ab6640 1 --2- 172.21.15.107:0/4087123877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d73499950 0x557d734b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefa62b7640 1 -- 172.21.15.107:0/4087123877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d73499950 msgr2=0x557d734b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefa62b7640 1 --2- 172.21.15.107:0/4087123877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d73499950 0x557d734b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefa62b7640 1 -- 172.21.15.107:0/4087123877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d734a6530 msgr2=0x557d734b92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefa62b7640 1 --2- 172.21.15.107:0/4087123877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d734a6530 0x557d734b92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefa62b7640 1 -- 172.21.15.107:0/4087123877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d734b9870 con 0x557d73498fa0 2024-05-09T08:37:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.806+0000 7fefa5ab6640 1 --2- 172.21.15.107:0/4087123877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d73499950 0x557d734b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.807+0000 7fefa62b7640 1 --2- 172.21.15.107:0/4087123877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d734bb9e0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fef9c09aec0 tx=0x7fef9c09ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f4a4a598f86f45c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.807+0000 7fef89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:46.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.807+0000 7fef89ffb640 1 -- 172.21.15.107:0/4087123877 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c020030 con 0x557d73498fa0 2024-05-09T08:37:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.807+0000 7fef89ffb640 1 -- 172.21.15.107:0/4087123877 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fef9c097bb0 con 0x557d73498fa0 2024-05-09T08:37:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.807+0000 7fefab22e740 1 -- 172.21.15.107:0/4087123877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef9c09aba0 con 0x557d73498fa0 2024-05-09T08:37:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.807+0000 7fef89ffb640 1 -- 172.21.15.107:0/4087123877 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fef9c0ad050 con 0x557d73498fa0 2024-05-09T08:37:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.807+0000 7fefab22e740 10 client.16977 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.807+0000 7fefab22e740 1 -- 172.21.15.107:0/4087123877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557d734b9870 con 0x557d73498fa0 2024-05-09T08:37:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.807+0000 7fef89ffb640 1 -- 172.21.15.107:0/4087123877 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef9c09aba0 con 0x557d73498fa0 2024-05-09T08:37:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.807+0000 7fef89ffb640 1 -- 172.21.15.107:0/4087123877 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fef9c0a09d0 con 0x557d73498fa0 2024-05-09T08:37:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.807+0000 7fefab22e740 10 client.16977 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:46.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.807+0000 7fefab22e740 10 client.16977 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:46.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.807+0000 7fefab22e740 20 client.16977 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:46.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.807+0000 7fefab22e740 1 --2- 172.21.15.107:0/4087123877 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557d7351cab0 0x557d7351eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:46.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.807+0000 7fefab22e740 4 client.16977 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:37:46.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.807+0000 7fefab22e740 1 -- 172.21.15.107:0/4087123877 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557d734066c0 con 0x557d7351cab0 2024-05-09T08:37:46.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fefa5ab6640 1 --2- 172.21.15.107:0/4087123877 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557d7351cab0 0x557d7351eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:46.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fefa5ab6640 1 --2- 172.21.15.107:0/4087123877 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557d7351cab0 0x557d7351eea0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e6e793993f982b3 server_cookie=998fe2a484941a1c in_seq=0 out_seq=0 2024-05-09T08:37:46.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fef89ffb640 10 client.16977 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:46.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fef89ffb640 1 -- 172.21.15.107:0/4087123877 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557d734066c0 con 0x557d7351cab0 2024-05-09T08:37:46.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fef89ffb640 10 client.16977 handle_command_reply: tid=0 2024-05-09T08:37:46.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fefab22e740 1 client.16977 shutdown 2024-05-09T08:37:46.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fefab22e740 1 -- 172.21.15.107:0/4087123877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 msgr2=0x557d734bb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fefab22e740 1 --2- 172.21.15.107:0/4087123877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d734bb9e0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fef9c09aec0 tx=0x7fef9c09ad50 comp rx=0 tx=0).stop 2024-05-09T08:37:46.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fefab22e740 1 -- 172.21.15.107:0/4087123877 shutdown_connections 2024-05-09T08:37:46.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fefab22e740 1 --2- 172.21.15.107:0/4087123877 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557d7351cab0 0x557d7351eea0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fefab22e740 1 --2- 172.21.15.107:0/4087123877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557d734a6530 0x557d734b92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fefab22e740 1 --2- 172.21.15.107:0/4087123877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557d73499950 0x557d734b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fef89ffb640 0 client.16977 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fefab22e740 1 --2- 172.21.15.107:0/4087123877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d73498fa0 0x557d734bb9e0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fefab22e740 1 -- 172.21.15.107:0/4087123877 >> 172.21.15.107:0/4087123877 conn(0x557d73495650 msgr2=0x557d734947b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fefab22e740 1 -- 172.21.15.107:0/4087123877 shutdown_connections 2024-05-09T08:37:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fefab22e740 1 -- 172.21.15.107:0/4087123877 wait complete. 2024-05-09T08:37:46.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.808+0000 7fefab22e740 20 client.16977 trim_cache size 0 max 16384 2024-05-09T08:37:46.816 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:37:46.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.809+0000 7fefa851b640 1 -- 172.21.15.107:0/3709638651 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fef8c0a2d80 msgr2=0x7fef8c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.809+0000 7fefa851b640 1 --2- 172.21.15.107:0/3709638651 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fef8c0a2d80 0x7fef8c0a5240 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fef98005fd0 tx=0x7fef98009500 comp rx=0 tx=0).stop 2024-05-09T08:37:46.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.809+0000 7fefa851b640 1 -- 172.21.15.107:0/3709638651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefa0100600 msgr2=0x7fefa0195e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.809+0000 7fefa851b640 1 --2- 172.21.15.107:0/3709638651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefa0100600 0x7fefa0195e30 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fef9c00d950 tx=0x7fef9c00de10 comp rx=0 tx=0).stop 2024-05-09T08:37:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.809+0000 7fefa851b640 1 -- 172.21.15.107:0/3709638651 shutdown_connections 2024-05-09T08:37:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.809+0000 7fefa851b640 1 --2- 172.21.15.107:0/3709638651 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fef8c0a2d80 0x7fef8c0a5240 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.809+0000 7fefa851b640 1 --2- 172.21.15.107:0/3709638651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fefa019a6f0 0x7fefa019aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.809+0000 7fefa851b640 1 --2- 172.21.15.107:0/3709638651 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefa01051b0 0x7fefa0196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.809+0000 7fefa851b640 1 --2- 172.21.15.107:0/3709638651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefa0100600 0x7fefa0195e30 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.809+0000 7fefa851b640 1 -- 172.21.15.107:0/3709638651 >> 172.21.15.107:0/3709638651 conn(0x7fefa0075a30 msgr2=0x7fefa00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.809+0000 7fefa851b640 1 -- 172.21.15.107:0/3709638651 shutdown_connections 2024-05-09T08:37:46.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:46.809+0000 7fefa851b640 1 -- 172.21.15.107:0/3709638651 wait complete. 2024-05-09T08:37:47.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:47 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3696497105' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:37:47.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:47 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3696497105' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:37:47.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:47 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3696497105' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:37:47.822 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:37:47.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:37:47.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:37:47.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.944+0000 7f85d19c1640 1 -- 172.21.15.107:0/794008608 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc0fffb0 msgr2=0x7f85cc100390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:47.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.944+0000 7f85d19c1640 1 --2- 172.21.15.107:0/794008608 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc0fffb0 0x7f85cc100390 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f85bc009900 tx=0x7f85bc02a410 comp rx=0 tx=0).stop 2024-05-09T08:37:47.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.944+0000 7f85d19c1640 1 -- 172.21.15.107:0/794008608 shutdown_connections 2024-05-09T08:37:47.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.944+0000 7f85d19c1640 1 --2- 172.21.15.107:0/794008608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85cc106bc0 0x7f85cc10e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.944+0000 7f85d19c1640 1 --2- 172.21.15.107:0/794008608 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc100960 0x7f85cc106490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.944+0000 7f85d19c1640 1 --2- 172.21.15.107:0/794008608 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc0fffb0 0x7f85cc100390 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.944+0000 7f85d19c1640 1 -- 172.21.15.107:0/794008608 >> 172.21.15.107:0/794008608 conn(0x7f85cc076cd0 msgr2=0x7f85cc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.944+0000 7f85d19c1640 1 -- 172.21.15.107:0/794008608 shutdown_connections 2024-05-09T08:37:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.944+0000 7f85d19c1640 1 -- 172.21.15.107:0/794008608 wait complete. 2024-05-09T08:37:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.945+0000 7f85d19c1640 1 Processor -- start 2024-05-09T08:37:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.945+0000 7f85d19c1640 1 -- start start 2024-05-09T08:37:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.945+0000 7f85d19c1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc0fffb0 0x7f85cc198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.945+0000 7f85d19c1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc106bc0 0x7f85cc198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.945+0000 7f85d19c1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85cc19c8d0 0x7f85cc19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.945+0000 7f85d19c1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc111ad0 con 0x7f85cc0fffb0 2024-05-09T08:37:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.945+0000 7f85d19c1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc111950 con 0x7f85cc106bc0 2024-05-09T08:37:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.945+0000 7f85d19c1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc111c50 con 0x7f85cc19c8d0 2024-05-09T08:37:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.945+0000 7f85caffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc0fffb0 0x7f85cc198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.945+0000 7f85caffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc0fffb0 0x7f85cc198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45052/0 (socket says 172.21.15.107:45052) 2024-05-09T08:37:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.945+0000 7f85caffd640 1 -- 172.21.15.107:0/3831610303 learned_addr learned my addr 172.21.15.107:0/3831610303 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.945+0000 7f85c27fc640 1 --2- 172.21.15.107:0/3831610303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc106bc0 0x7f85cc198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.945+0000 7f85cb7fe640 1 --2- 172.21.15.107:0/3831610303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85cc19c8d0 0x7f85cc19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.946+0000 7f85caffd640 1 -- 172.21.15.107:0/3831610303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85cc19c8d0 msgr2=0x7f85cc19cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.946+0000 7f85caffd640 1 --2- 172.21.15.107:0/3831610303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85cc19c8d0 0x7f85cc19cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.946+0000 7f85caffd640 1 -- 172.21.15.107:0/3831610303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc106bc0 msgr2=0x7f85cc198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.946+0000 7f85caffd640 1 --2- 172.21.15.107:0/3831610303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc106bc0 0x7f85cc198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.946+0000 7f85caffd640 1 -- 172.21.15.107:0/3831610303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85cc19d3c0 con 0x7f85cc0fffb0 2024-05-09T08:37:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.946+0000 7f85cb7fe640 1 --2- 172.21.15.107:0/3831610303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85cc19c8d0 0x7f85cc19cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.946+0000 7f85caffd640 1 --2- 172.21.15.107:0/3831610303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc0fffb0 0x7f85cc198010 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f85bc005940 tx=0x7f85bc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.946+0000 7f85d09bf640 1 -- 172.21.15.107:0/3831610303 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc03f030 con 0x7f85cc0fffb0 2024-05-09T08:37:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.946+0000 7f85d19c1640 1 -- 172.21.15.107:0/3831610303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85cc19d650 con 0x7f85cc0fffb0 2024-05-09T08:37:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.946+0000 7f85d09bf640 1 -- 172.21.15.107:0/3831610303 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85bc003cb0 con 0x7f85cc0fffb0 2024-05-09T08:37:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.946+0000 7f85d09bf640 1 -- 172.21.15.107:0/3831610303 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc035070 con 0x7f85cc0fffb0 2024-05-09T08:37:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.946+0000 7f85d19c1640 1 -- 172.21.15.107:0/3831610303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85cc1a4e70 con 0x7f85cc0fffb0 2024-05-09T08:37:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.948+0000 7f85d09bf640 1 -- 172.21.15.107:0/3831610303 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f85bc042030 con 0x7f85cc0fffb0 2024-05-09T08:37:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.949+0000 7f85d09bf640 1 --2- 172.21.15.107:0/3831610303 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f85b80a2d00 0x7f85b80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.949+0000 7f85d47ad740 1 Processor -- start 2024-05-09T08:37:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.949+0000 7f85d47ad740 1 -- start start 2024-05-09T08:37:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.949+0000 7f85c27fc640 1 --2- 172.21.15.107:0/3831610303 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f85b80a2d00 0x7f85b80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.949+0000 7f85d09bf640 1 -- 172.21.15.107:0/3831610303 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f85bc0eff70 con 0x7f85cc0fffb0 2024-05-09T08:37:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.949+0000 7f85d47ad740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da7690b0 0x5645da7673f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.949+0000 7f85d47ad740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da7679c0 0x5645da767e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.949+0000 7f85d47ad740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da776b50 0x5645da778f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.949+0000 7f85d47ad740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5645da6cc990 con 0x5645da7690b0 2024-05-09T08:37:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.949+0000 7f85d47ad740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5645da6745b0 con 0x5645da776b50 2024-05-09T08:37:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.949+0000 7f85d47ad740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5645da6c8950 con 0x5645da7679c0 2024-05-09T08:37:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.950+0000 7f85caffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da7690b0 0x5645da7673f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.950+0000 7f85caffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da7690b0 0x5645da7673f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45062/0 (socket says 172.21.15.107:45062) 2024-05-09T08:37:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.950+0000 7f85caffd640 1 -- 172.21.15.107:0/1700395643 learned_addr learned my addr 172.21.15.107:0/1700395643 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.950+0000 7f85c27fc640 1 --2- 172.21.15.107:0/1700395643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da7679c0 0x5645da767e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.950+0000 7f85c27fc640 1 --2- 172.21.15.107:0/3831610303 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f85b80a2d00 0x7f85b80a51c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0ff2e0 tx=0x7f85b4005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.950+0000 7f85cb7fe640 1 --2- 172.21.15.107:0/1700395643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da776b50 0x5645da778f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.950+0000 7f85caffd640 1 -- 172.21.15.107:0/1700395643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da7679c0 msgr2=0x5645da767e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.950+0000 7f85caffd640 1 --2- 172.21.15.107:0/1700395643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da7679c0 0x5645da767e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.950+0000 7f85caffd640 1 -- 172.21.15.107:0/1700395643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da776b50 msgr2=0x5645da778f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.950+0000 7f85caffd640 1 --2- 172.21.15.107:0/1700395643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da776b50 0x5645da778f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.950+0000 7f85caffd640 1 -- 172.21.15.107:0/1700395643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645da4a87e0 con 0x5645da7690b0 2024-05-09T08:37:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.950+0000 7f85c27fc640 1 --2- 172.21.15.107:0/1700395643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da7679c0 0x5645da767e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:37:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.950+0000 7f85caffd640 1 --2- 172.21.15.107:0/1700395643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da7690b0 0x5645da7673f0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0c4870 tx=0x7f85bc0bba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2db41cfe62adfa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.950+0000 7f85c1ffb640 1 -- 172.21.15.107:0/1700395643 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc049040 con 0x5645da7690b0 2024-05-09T08:37:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.950+0000 7f85c1ffb640 1 -- 172.21.15.107:0/1700395643 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85bc0bc830 con 0x5645da7690b0 2024-05-09T08:37:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.950+0000 7f85c1ffb640 1 -- 172.21.15.107:0/1700395643 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc0dc070 con 0x5645da7690b0 2024-05-09T08:37:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.951+0000 7f85d47ad740 1 -- 172.21.15.107:0/1700395643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da7690b0 msgr2=0x5645da7673f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.951+0000 7f85d47ad740 1 --2- 172.21.15.107:0/1700395643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da7690b0 0x5645da7673f0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0c4870 tx=0x7f85bc0bba90 comp rx=0 tx=0).stop 2024-05-09T08:37:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.951+0000 7f85d47ad740 1 -- 172.21.15.107:0/1700395643 shutdown_connections 2024-05-09T08:37:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.951+0000 7f85d47ad740 1 --2- 172.21.15.107:0/1700395643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da776b50 0x5645da778f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.951+0000 7f85d47ad740 1 --2- 172.21.15.107:0/1700395643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da7679c0 0x5645da767e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.951+0000 7f85d47ad740 1 --2- 172.21.15.107:0/1700395643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da7690b0 0x5645da7673f0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.951+0000 7f85d47ad740 1 -- 172.21.15.107:0/1700395643 >> 172.21.15.107:0/1700395643 conn(0x5645da7300a0 msgr2=0x5645da763f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.951+0000 7f85d47ad740 1 -- 172.21.15.107:0/1700395643 shutdown_connections 2024-05-09T08:37:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.951+0000 7f85d47ad740 1 -- 172.21.15.107:0/1700395643 wait complete. 2024-05-09T08:37:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.951+0000 7f85d47ad740 1 -- start start 2024-05-09T08:37:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85d47ad740 1 Processor -- start 2024-05-09T08:37:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85d47ad740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da7679c0 0x5645da773a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85d47ad740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da7690b0 0x5645da773fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85d47ad740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da776b50 0x5645da774510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85d47ad740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5645da7797a0 con 0x5645da7679c0 2024-05-09T08:37:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85d47ad740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5645da687b40 con 0x5645da776b50 2024-05-09T08:37:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85d47ad740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5645da779920 con 0x5645da7690b0 2024-05-09T08:37:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85caffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da7679c0 0x5645da773a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85d47ad740 10 client.0 _ll_register_callbacks cb 0x5645da67c0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85caffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da7679c0 0x5645da773a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45076/0 (socket says 172.21.15.107:45076) 2024-05-09T08:37:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85caffd640 1 -- 172.21.15.107:0/904583969 learned_addr learned my addr 172.21.15.107:0/904583969 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85cb7fe640 1 --2- 172.21.15.107:0/904583969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da776b50 0x5645da774510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85c27fc640 1 --2- 172.21.15.107:0/904583969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da7690b0 0x5645da773fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85caffd640 1 -- 172.21.15.107:0/904583969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da7690b0 msgr2=0x5645da773fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85caffd640 1 --2- 172.21.15.107:0/904583969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da7690b0 0x5645da773fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85caffd640 1 -- 172.21.15.107:0/904583969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da776b50 msgr2=0x5645da774510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85caffd640 1 --2- 172.21.15.107:0/904583969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da776b50 0x5645da774510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85caffd640 1 -- 172.21.15.107:0/904583969 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645da508f70 con 0x5645da7679c0 2024-05-09T08:37:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85caffd640 1 --2- 172.21.15.107:0/904583969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da7679c0 0x5645da773a90 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0b8590 tx=0x7f85bc0b89d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ccd23ae6c9053ca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.952+0000 7f85aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.953+0000 7f85aaffd640 1 -- 172.21.15.107:0/904583969 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc049070 con 0x5645da7679c0 2024-05-09T08:37:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.953+0000 7f85aaffd640 1 -- 172.21.15.107:0/904583969 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f85bc0dc050 con 0x5645da7679c0 2024-05-09T08:37:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.953+0000 7f85d47ad740 1 -- 172.21.15.107:0/904583969 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85bc0bc830 con 0x5645da7679c0 2024-05-09T08:37:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.953+0000 7f85aaffd640 1 -- 172.21.15.107:0/904583969 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f85bc0ed750 con 0x5645da7679c0 2024-05-09T08:37:47.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.953+0000 7f85d47ad740 10 client.16989 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:47.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.953+0000 7f85d47ad740 1 -- 172.21.15.107:0/904583969 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5645da3a0170 con 0x5645da7679c0 2024-05-09T08:37:47.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.953+0000 7f85aaffd640 1 -- 172.21.15.107:0/904583969 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85bc0bc830 con 0x5645da7679c0 2024-05-09T08:37:47.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.953+0000 7f85aaffd640 1 -- 172.21.15.107:0/904583969 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f85bc0c4910 con 0x5645da7679c0 2024-05-09T08:37:47.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.953+0000 7f85d47ad740 10 client.16989 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:47.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.953+0000 7f85d47ad740 10 client.16989 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:47.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.953+0000 7f85d47ad740 20 client.16989 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:47.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.953+0000 7f85d47ad740 1 --2- 172.21.15.107:0/904583969 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5645da786800 0x5645da788bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.953+0000 7f85d47ad740 4 client.16989 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:47.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.953+0000 7f85d47ad740 1 -- 172.21.15.107:0/904583969 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645da6da0c0 con 0x5645da786800 2024-05-09T08:37:47.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.953+0000 7f85c27fc640 1 --2- 172.21.15.107:0/904583969 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5645da786800 0x5645da788bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.954+0000 7f85aaffd640 10 client.16989 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:47.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.954+0000 7f85c27fc640 1 --2- 172.21.15.107:0/904583969 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5645da786800 0x5645da788bf0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d51878321e97bf86 server_cookie=2a0f46f4f9ba9043 in_seq=0 out_seq=0 2024-05-09T08:37:47.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.955+0000 7f85aaffd640 1 -- 172.21.15.107:0/904583969 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5645da6da0c0 con 0x5645da786800 2024-05-09T08:37:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.955+0000 7f85aaffd640 10 client.16989 handle_command_reply: tid=0 2024-05-09T08:37:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.955+0000 7f85d47ad740 1 client.16989 shutdown 2024-05-09T08:37:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.955+0000 7f85d47ad740 1 -- 172.21.15.107:0/904583969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da7679c0 msgr2=0x5645da773a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.955+0000 7f85d47ad740 1 --2- 172.21.15.107:0/904583969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da7679c0 0x5645da773a90 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0b8590 tx=0x7f85bc0b89d0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.955+0000 7f85d47ad740 1 -- 172.21.15.107:0/904583969 shutdown_connections 2024-05-09T08:37:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.955+0000 7f85d47ad740 1 --2- 172.21.15.107:0/904583969 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5645da786800 0x5645da788bf0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.955+0000 7f85d47ad740 1 --2- 172.21.15.107:0/904583969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da776b50 0x5645da774510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.955+0000 7f85d47ad740 1 --2- 172.21.15.107:0/904583969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da7690b0 0x5645da773fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.955+0000 7f85aaffd640 0 client.16989 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.955+0000 7f85d47ad740 1 --2- 172.21.15.107:0/904583969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da7679c0 0x5645da773a90 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.955+0000 7f85d47ad740 1 -- 172.21.15.107:0/904583969 >> 172.21.15.107:0/904583969 conn(0x5645da7300a0 msgr2=0x5645da769d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.955+0000 7f85d47ad740 1 -- 172.21.15.107:0/904583969 shutdown_connections 2024-05-09T08:37:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.955+0000 7f85d47ad740 1 -- 172.21.15.107:0/904583969 wait complete. 2024-05-09T08:37:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.955+0000 7f85d47ad740 20 client.16989 trim_cache size 0 max 16384 2024-05-09T08:37:47.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.972+0000 7f85d47ad740 1 Processor -- start 2024-05-09T08:37:47.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85d47ad740 1 -- start start 2024-05-09T08:37:47.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85d47ad740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da7690b0 0x5645da782660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85d47ad740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da776b50 0x5645da762f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85d47ad740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da763eb0 0x5645da78e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85d47ad740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc1af180 con 0x5645da763eb0 2024-05-09T08:37:47.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85d47ad740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc1aca50 con 0x5645da7690b0 2024-05-09T08:37:47.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85d47ad740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc1aa8a0 con 0x5645da776b50 2024-05-09T08:37:47.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85cb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da763eb0 0x5645da78e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85cb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da763eb0 0x5645da78e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45082/0 (socket says 172.21.15.107:45082) 2024-05-09T08:37:47.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85cb7fe640 1 -- 172.21.15.107:0/1654061568 learned_addr learned my addr 172.21.15.107:0/1654061568 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:47.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85c27fc640 1 --2- 172.21.15.107:0/1654061568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da776b50 0x5645da762f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85caffd640 1 --2- 172.21.15.107:0/1654061568 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da7690b0 0x5645da782660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85cb7fe640 1 -- 172.21.15.107:0/1654061568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da776b50 msgr2=0x5645da762f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:47.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85cb7fe640 1 --2- 172.21.15.107:0/1654061568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da776b50 0x5645da762f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85cb7fe640 1 -- 172.21.15.107:0/1654061568 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da7690b0 msgr2=0x5645da782660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:47.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85cb7fe640 1 --2- 172.21.15.107:0/1654061568 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da7690b0 0x5645da782660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85cb7fe640 1 -- 172.21.15.107:0/1654061568 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645da768410 con 0x5645da763eb0 2024-05-09T08:37:47.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.973+0000 7f85cb7fe640 1 --2- 172.21.15.107:0/1654061568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da763eb0 0x5645da78e470 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f85c4009e60 tx=0x7f85c4009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398411e6b5572012 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:47.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.974+0000 7f85c17fa640 1 -- 172.21.15.107:0/1654061568 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c400fd70 con 0x5645da763eb0 2024-05-09T08:37:47.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.974+0000 7f85c17fa640 1 -- 172.21.15.107:0/1654061568 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85c400e8c0 con 0x5645da763eb0 2024-05-09T08:37:47.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.974+0000 7f85c17fa640 1 -- 172.21.15.107:0/1654061568 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c400c8d0 con 0x5645da763eb0 2024-05-09T08:37:47.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.974+0000 7f85d47ad740 1 -- 172.21.15.107:0/1654061568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da763eb0 msgr2=0x5645da78e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:47.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.974+0000 7f85d47ad740 1 --2- 172.21.15.107:0/1654061568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da763eb0 0x5645da78e470 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f85c4009e60 tx=0x7f85c4009f50 comp rx=0 tx=0).stop 2024-05-09T08:37:47.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.974+0000 7f85d47ad740 1 -- 172.21.15.107:0/1654061568 shutdown_connections 2024-05-09T08:37:47.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.974+0000 7f85d47ad740 1 --2- 172.21.15.107:0/1654061568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da763eb0 0x5645da78e470 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.974+0000 7f85d47ad740 1 --2- 172.21.15.107:0/1654061568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da776b50 0x5645da762f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.974+0000 7f85d47ad740 1 --2- 172.21.15.107:0/1654061568 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da7690b0 0x5645da782660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.974+0000 7f85d47ad740 1 -- 172.21.15.107:0/1654061568 >> 172.21.15.107:0/1654061568 conn(0x5645da7300a0 msgr2=0x5645da7627f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:47.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.974+0000 7f85d47ad740 1 -- 172.21.15.107:0/1654061568 shutdown_connections 2024-05-09T08:37:47.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.974+0000 7f85d47ad740 1 -- 172.21.15.107:0/1654061568 wait complete. 2024-05-09T08:37:47.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.975+0000 7f85d47ad740 1 -- start start 2024-05-09T08:37:47.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.975+0000 7f85d47ad740 1 Processor -- start 2024-05-09T08:37:47.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.975+0000 7f85d47ad740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da763eb0 0x5645da77ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.975+0000 7f85d47ad740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da7690b0 0x5645da77bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.975+0000 7f85d47ad740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da776b50 0x5645da77c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.975+0000 7f85d47ad740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc1aaf10 con 0x5645da763eb0 2024-05-09T08:37:47.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.975+0000 7f85d47ad740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc1ac380 con 0x5645da7690b0 2024-05-09T08:37:47.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.975+0000 7f85d47ad740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc1ab5e0 con 0x5645da776b50 2024-05-09T08:37:47.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85d47ad740 10 client.0 _ll_register_callbacks cb 0x7f85bc0bbf40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:47.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85caffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da763eb0 0x5645da77ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85caffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da763eb0 0x5645da77ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45096/0 (socket says 172.21.15.107:45096) 2024-05-09T08:37:47.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85caffd640 1 -- 172.21.15.107:0/1062358381 learned_addr learned my addr 172.21.15.107:0/1062358381 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:47.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85cb7fe640 1 --2- 172.21.15.107:0/1062358381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da776b50 0x5645da77c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85c27fc640 1 --2- 172.21.15.107:0/1062358381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da7690b0 0x5645da77bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85caffd640 1 -- 172.21.15.107:0/1062358381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da776b50 msgr2=0x5645da77c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:47.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85caffd640 1 --2- 172.21.15.107:0/1062358381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da776b50 0x5645da77c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85caffd640 1 -- 172.21.15.107:0/1062358381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da7690b0 msgr2=0x5645da77bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:47.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85caffd640 1 --2- 172.21.15.107:0/1062358381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da7690b0 0x5645da77bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85caffd640 1 -- 172.21.15.107:0/1062358381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645da77c5f0 con 0x5645da763eb0 2024-05-09T08:37:47.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85cb7fe640 1 --2- 172.21.15.107:0/1062358381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da776b50 0x5645da77c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:47.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85caffd640 1 --2- 172.21.15.107:0/1062358381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da763eb0 0x5645da77ed60 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0ed410 tx=0x7f85bc0c0380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3858a1e8247983ce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:47.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:47.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85aaffd640 1 -- 172.21.15.107:0/1062358381 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc049070 con 0x5645da763eb0 2024-05-09T08:37:47.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85aaffd640 1 -- 172.21.15.107:0/1062358381 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f85bc0bb810 con 0x5645da763eb0 2024-05-09T08:37:47.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85d47ad740 1 -- 172.21.15.107:0/1062358381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85c400e8c0 con 0x5645da763eb0 2024-05-09T08:37:47.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85aaffd640 1 -- 172.21.15.107:0/1062358381 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f85bc003e60 con 0x5645da763eb0 2024-05-09T08:37:47.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85d47ad740 10 client.16995 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:47.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.976+0000 7f85d47ad740 1 -- 172.21.15.107:0/1062358381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5645da77c5f0 con 0x5645da763eb0 2024-05-09T08:37:47.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.977+0000 7f85aaffd640 1 -- 172.21.15.107:0/1062358381 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85c400e8c0 con 0x5645da763eb0 2024-05-09T08:37:47.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.977+0000 7f85aaffd640 1 -- 172.21.15.107:0/1062358381 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f85bc0dc0b0 con 0x5645da763eb0 2024-05-09T08:37:47.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.977+0000 7f85d47ad740 10 client.16995 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:47.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.977+0000 7f85d47ad740 10 client.16995 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:47.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.977+0000 7f85d47ad740 20 client.16995 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:47.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.977+0000 7f85d47ad740 1 --2- 172.21.15.107:0/1062358381 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5645da7edee0 0x5645da7f02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:47.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.977+0000 7f85d47ad740 4 client.16995 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:37:47.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.977+0000 7f85d47ad740 1 -- 172.21.15.107:0/1062358381 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5645da6d4410 con 0x5645da7edee0 2024-05-09T08:37:47.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.977+0000 7f85c27fc640 1 --2- 172.21.15.107:0/1062358381 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5645da7edee0 0x5645da7f02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:47.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.978+0000 7f85aaffd640 10 client.16995 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:47.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.978+0000 7f85c27fc640 1 --2- 172.21.15.107:0/1062358381 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5645da7edee0 0x5645da7f02d0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=156f84a3c31c9b0c server_cookie=4423e61a11232b4 in_seq=0 out_seq=0 2024-05-09T08:37:47.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.978+0000 7f85aaffd640 1 -- 172.21.15.107:0/1062358381 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5645da6d4410 con 0x5645da7edee0 2024-05-09T08:37:47.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.979+0000 7f85aaffd640 10 client.16995 handle_command_reply: tid=0 2024-05-09T08:37:47.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.979+0000 7f85d47ad740 1 client.16995 shutdown 2024-05-09T08:37:47.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.979+0000 7f85d47ad740 1 -- 172.21.15.107:0/1062358381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da763eb0 msgr2=0x5645da77ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:47.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.979+0000 7f85d47ad740 1 --2- 172.21.15.107:0/1062358381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da763eb0 0x5645da77ed60 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0ed410 tx=0x7f85bc0c0380 comp rx=0 tx=0).stop 2024-05-09T08:37:47.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.979+0000 7f85d47ad740 1 -- 172.21.15.107:0/1062358381 shutdown_connections 2024-05-09T08:37:47.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.979+0000 7f85d47ad740 1 --2- 172.21.15.107:0/1062358381 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5645da7edee0 0x5645da7f02d0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.979+0000 7f85d47ad740 1 --2- 172.21.15.107:0/1062358381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5645da776b50 0x5645da77c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.979+0000 7f85d47ad740 1 --2- 172.21.15.107:0/1062358381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5645da7690b0 0x5645da77bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.979+0000 7f85d47ad740 1 --2- 172.21.15.107:0/1062358381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645da763eb0 0x5645da77ed60 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.979+0000 7f85aaffd640 0 client.16995 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:47.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.979+0000 7f85d47ad740 1 -- 172.21.15.107:0/1062358381 >> 172.21.15.107:0/1062358381 conn(0x5645da7300a0 msgr2=0x5645da763080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:47.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.979+0000 7f85d47ad740 1 -- 172.21.15.107:0/1062358381 shutdown_connections 2024-05-09T08:37:47.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.979+0000 7f85d47ad740 1 -- 172.21.15.107:0/1062358381 wait complete. 2024-05-09T08:37:47.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.979+0000 7f85d47ad740 20 client.16995 trim_cache size 0 max 16384 2024-05-09T08:37:47.984 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:37:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:37:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "dbf46e95-13ff-4f10-8354-ab8e4994df62", 2024-05-09T08:37:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:37:47.985 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:37:47.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.980+0000 7f85d19c1640 1 -- 172.21.15.107:0/3831610303 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f85b80a2d00 msgr2=0x7f85b80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:47.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.980+0000 7f85d19c1640 1 --2- 172.21.15.107:0/3831610303 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f85b80a2d00 0x7f85b80a51c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0ff2e0 tx=0x7f85b4005f90 comp rx=0 tx=0).stop 2024-05-09T08:37:47.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.980+0000 7f85d19c1640 1 -- 172.21.15.107:0/3831610303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc0fffb0 msgr2=0x7f85cc198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:47.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.980+0000 7f85d19c1640 1 --2- 172.21.15.107:0/3831610303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc0fffb0 0x7f85cc198010 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f85bc005940 tx=0x7f85bc0042f0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.980+0000 7f85d19c1640 1 -- 172.21.15.107:0/3831610303 shutdown_connections 2024-05-09T08:37:47.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.980+0000 7f85d19c1640 1 --2- 172.21.15.107:0/3831610303 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f85b80a2d00 0x7f85b80a51c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.980+0000 7f85d19c1640 1 --2- 172.21.15.107:0/3831610303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85cc19c8d0 0x7f85cc19cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.980+0000 7f85d19c1640 1 --2- 172.21.15.107:0/3831610303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc106bc0 0x7f85cc198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.980+0000 7f85d19c1640 1 --2- 172.21.15.107:0/3831610303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc0fffb0 0x7f85cc198010 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:47.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.980+0000 7f85d19c1640 1 -- 172.21.15.107:0/3831610303 >> 172.21.15.107:0/3831610303 conn(0x7f85cc076cd0 msgr2=0x7f85cc104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:47.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.980+0000 7f85d19c1640 1 -- 172.21.15.107:0/3831610303 shutdown_connections 2024-05-09T08:37:47.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:47.980+0000 7f85d19c1640 1 -- 172.21.15.107:0/3831610303 wait complete. 2024-05-09T08:37:47.992 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:37:47.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:37:48.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.109+0000 7fd448f7e640 1 -- 172.21.15.107:0/3080087911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4441033d0 msgr2=0x7fd44410b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:48.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.109+0000 7fd448f7e640 1 --2- 172.21.15.107:0/3080087911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4441033d0 0x7fd44410b8b0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fd438009900 tx=0x7fd43802a510 comp rx=0 tx=0).stop 2024-05-09T08:37:48.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.109+0000 7fd448f7e640 1 -- 172.21.15.107:0/3080087911 shutdown_connections 2024-05-09T08:37:48.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.109+0000 7fd448f7e640 1 --2- 172.21.15.107:0/3080087911 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd44410bdf0 0x7fd44410e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.109+0000 7fd448f7e640 1 --2- 172.21.15.107:0/3080087911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4441033d0 0x7fd44410b8b0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.109+0000 7fd448f7e640 1 --2- 172.21.15.107:0/3080087911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd444102ab0 0x7fd444102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.109+0000 7fd448f7e640 1 -- 172.21.15.107:0/3080087911 >> 172.21.15.107:0/3080087911 conn(0x7fd444077830 msgr2=0x7fd444077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:48.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.109+0000 7fd448f7e640 1 -- 172.21.15.107:0/3080087911 shutdown_connections 2024-05-09T08:37:48.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.109+0000 7fd448f7e640 1 -- 172.21.15.107:0/3080087911 wait complete. 2024-05-09T08:37:48.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.109+0000 7fd448f7e640 1 Processor -- start 2024-05-09T08:37:48.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd448f7e640 1 -- start start 2024-05-09T08:37:48.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd448f7e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd444102ab0 0x7fd4441980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd448f7e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4441033d0 0x7fd4441985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd448f7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd44410bdf0 0x7fd44419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd448f7e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd444111af0 con 0x7fd44410bdf0 2024-05-09T08:37:48.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd448f7e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd444111970 con 0x7fd4441033d0 2024-05-09T08:37:48.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd448f7e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd444111c70 con 0x7fd444102ab0 2024-05-09T08:37:48.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd442d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd44410bdf0 0x7fd44419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd441d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4441033d0 0x7fd4441985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd441d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4441033d0 0x7fd4441985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:55954/0 (socket says 172.21.15.107:55954) 2024-05-09T08:37:48.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd441d9b640 1 -- 172.21.15.107:0/243451334 learned_addr learned my addr 172.21.15.107:0/243451334 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:48.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd44259c640 1 --2- 172.21.15.107:0/243451334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd444102ab0 0x7fd4441980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd442d9d640 1 -- 172.21.15.107:0/243451334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd444102ab0 msgr2=0x7fd4441980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:48.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd442d9d640 1 --2- 172.21.15.107:0/243451334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd444102ab0 0x7fd4441980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd442d9d640 1 -- 172.21.15.107:0/243451334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4441033d0 msgr2=0x7fd4441985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:48.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd442d9d640 1 --2- 172.21.15.107:0/243451334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4441033d0 0x7fd4441985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd442d9d640 1 -- 172.21.15.107:0/243451334 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44419d050 con 0x7fd44410bdf0 2024-05-09T08:37:48.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd441d9b640 1 --2- 172.21.15.107:0/243451334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4441033d0 0x7fd4441985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:48.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.110+0000 7fd442d9d640 1 --2- 172.21.15.107:0/243451334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd44410bdf0 0x7fd44419c920 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fd43c009fd0 tx=0x7fd43c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:48.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.111+0000 7fd4337fe640 1 -- 172.21.15.107:0/243451334 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c00ce20 con 0x7fd44410bdf0 2024-05-09T08:37:48.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.111+0000 7fd448f7e640 1 -- 172.21.15.107:0/243451334 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd44419d340 con 0x7fd44410bdf0 2024-05-09T08:37:48.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.111+0000 7fd4337fe640 1 -- 172.21.15.107:0/243451334 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd43c005020 con 0x7fd44410bdf0 2024-05-09T08:37:48.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.111+0000 7fd4337fe640 1 -- 172.21.15.107:0/243451334 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c010690 con 0x7fd44410bdf0 2024-05-09T08:37:48.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.111+0000 7fd448f7e640 1 -- 172.21.15.107:0/243451334 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4441a4bd0 con 0x7fd44410bdf0 2024-05-09T08:37:48.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.113+0000 7fd4337fe640 1 -- 172.21.15.107:0/243451334 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd43c0051d0 con 0x7fd44410bdf0 2024-05-09T08:37:48.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd44bd6b740 1 Processor -- start 2024-05-09T08:37:48.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd4337fe640 1 --2- 172.21.15.107:0/243451334 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd42c0a2d00 0x7fd42c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd44bd6b740 1 -- start start 2024-05-09T08:37:48.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd441d9b640 1 --2- 172.21.15.107:0/243451334 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd42c0a2d00 0x7fd42c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd44bd6b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a358280 0x555f9a358660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd44bd6b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358ba0 0x555f9a362d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd4337fe640 1 -- 172.21.15.107:0/243451334 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd43c0cfe10 con 0x7fd44410bdf0 2024-05-09T08:37:48.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd44bd6b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a363270 0x555f9a365660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd44bd6b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555f9a2bdd00 con 0x555f9a358ba0 2024-05-09T08:37:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd44bd6b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555f9a2654c0 con 0x555f9a363270 2024-05-09T08:37:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd44bd6b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x555f9a2b9cc0 con 0x555f9a358280 2024-05-09T08:37:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd44259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a358280 0x555f9a358660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd441d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358ba0 0x555f9a362d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd442d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a363270 0x555f9a365660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd44259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a358280 0x555f9a358660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:37514/0 (socket says 172.21.15.107:37514) 2024-05-09T08:37:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd441d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358ba0 0x555f9a362d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45108/0 (socket says 172.21.15.107:45108) 2024-05-09T08:37:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.114+0000 7fd44259c640 1 -- 172.21.15.107:0/2074712278 learned_addr learned my addr 172.21.15.107:0/2074712278 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.115+0000 7fd442d9d640 1 -- 172.21.15.107:0/2074712278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a358280 msgr2=0x555f9a358660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.115+0000 7fd442d9d640 1 --2- 172.21.15.107:0/2074712278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a358280 0x555f9a358660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.115+0000 7fd442d9d640 1 -- 172.21.15.107:0/2074712278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358ba0 msgr2=0x555f9a362d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.115+0000 7fd442d9d640 1 --2- 172.21.15.107:0/2074712278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358ba0 0x555f9a362d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.115+0000 7fd442d9d640 1 -- 172.21.15.107:0/2074712278 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f9a09b7d0 con 0x555f9a363270 2024-05-09T08:37:48.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.115+0000 7fd44259c640 1 --2- 172.21.15.107:0/2074712278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a358280 0x555f9a358660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:48.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.115+0000 7fd441d9b640 1 --2- 172.21.15.107:0/243451334 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd42c0a2d00 0x7fd42c0a51c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd44410dc80 tx=0x7fd438003860 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:48.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.115+0000 7fd442d9d640 1 --2- 172.21.15.107:0/2074712278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a363270 0x555f9a365660 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd43c0cd200 tx=0x7fd43c09c090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1323cb3c59590a46 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:48.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.115+0000 7fd430ff9640 1 -- 172.21.15.107:0/2074712278 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c0b4020 con 0x555f9a363270 2024-05-09T08:37:48.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.115+0000 7fd430ff9640 1 -- 172.21.15.107:0/2074712278 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd43c0bc070 con 0x555f9a363270 2024-05-09T08:37:48.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.115+0000 7fd430ff9640 1 -- 172.21.15.107:0/2074712278 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c0a47b0 con 0x555f9a363270 2024-05-09T08:37:48.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.116+0000 7fd44bd6b740 1 -- 172.21.15.107:0/2074712278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a363270 msgr2=0x555f9a365660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:48.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.116+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/2074712278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a363270 0x555f9a365660 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd43c0cd200 tx=0x7fd43c09c090 comp rx=0 tx=0).stop 2024-05-09T08:37:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.116+0000 7fd44bd6b740 1 -- 172.21.15.107:0/2074712278 shutdown_connections 2024-05-09T08:37:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.116+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/2074712278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a363270 0x555f9a365660 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.116+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/2074712278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358ba0 0x555f9a362d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.116+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/2074712278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a358280 0x555f9a358660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.116+0000 7fd44bd6b740 1 -- 172.21.15.107:0/2074712278 >> 172.21.15.107:0/2074712278 conn(0x555f9a3545a0 msgr2=0x555f9a354980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.116+0000 7fd44bd6b740 1 -- 172.21.15.107:0/2074712278 shutdown_connections 2024-05-09T08:37:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.116+0000 7fd44bd6b740 1 -- 172.21.15.107:0/2074712278 wait complete. 2024-05-09T08:37:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.116+0000 7fd44bd6b740 1 -- start start 2024-05-09T08:37:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44bd6b740 1 Processor -- start 2024-05-09T08:37:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44bd6b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358280 0x555f9a3601d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44bd6b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a358ba0 0x555f9a360710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44bd6b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a363270 0x555f9a360c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44bd6b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555f9a366060 con 0x555f9a358280 2024-05-09T08:37:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44bd6b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555f9a278b90 con 0x555f9a358ba0 2024-05-09T08:37:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44bd6b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x555f9a3661e0 con 0x555f9a363270 2024-05-09T08:37:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44bd6b740 10 client.0 _ll_register_callbacks cb 0x555f9a2b4da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358280 0x555f9a3601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358280 0x555f9a3601d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45112/0 (socket says 172.21.15.107:45112) 2024-05-09T08:37:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44259c640 1 -- 172.21.15.107:0/2128036234 learned_addr learned my addr 172.21.15.107:0/2128036234 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd441d9b640 1 --2- 172.21.15.107:0/2128036234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a358ba0 0x555f9a360710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd442d9d640 1 --2- 172.21.15.107:0/2128036234 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a363270 0x555f9a360c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44259c640 1 -- 172.21.15.107:0/2128036234 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a363270 msgr2=0x555f9a360c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44259c640 1 --2- 172.21.15.107:0/2128036234 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a363270 0x555f9a360c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44259c640 1 -- 172.21.15.107:0/2128036234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a358ba0 msgr2=0x555f9a360710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44259c640 1 --2- 172.21.15.107:0/2128036234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a358ba0 0x555f9a360710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44259c640 1 -- 172.21.15.107:0/2128036234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f99fd13f0 con 0x555f9a358280 2024-05-09T08:37:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd44259c640 1 --2- 172.21.15.107:0/2128036234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358280 0x555f9a3601d0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fd43400ba80 tx=0x7fd43400bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a161c115ac27490 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.117+0000 7fd429ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.118+0000 7fd429ffb640 1 -- 172.21.15.107:0/2128036234 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd434011440 con 0x555f9a358280 2024-05-09T08:37:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.118+0000 7fd429ffb640 1 -- 172.21.15.107:0/2128036234 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd4340115e0 con 0x555f9a358280 2024-05-09T08:37:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.118+0000 7fd44bd6b740 1 -- 172.21.15.107:0/2128036234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd43c0bc070 con 0x555f9a358280 2024-05-09T08:37:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.118+0000 7fd429ffb640 1 -- 172.21.15.107:0/2128036234 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd434005020 con 0x555f9a358280 2024-05-09T08:37:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.118+0000 7fd44bd6b740 10 client.17007 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.118+0000 7fd44bd6b740 1 -- 172.21.15.107:0/2128036234 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555f99f910f0 con 0x555f9a358280 2024-05-09T08:37:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.118+0000 7fd429ffb640 1 -- 172.21.15.107:0/2128036234 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd43c0bc070 con 0x555f9a358280 2024-05-09T08:37:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.118+0000 7fd429ffb640 1 -- 172.21.15.107:0/2128036234 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd4340047a0 con 0x555f9a358280 2024-05-09T08:37:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.118+0000 7fd44bd6b740 10 client.17007 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.118+0000 7fd44bd6b740 10 client.17007 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.118+0000 7fd44bd6b740 20 client.17007 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.118+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/2128036234 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555f9a373620 0x555f9a375a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.118+0000 7fd44bd6b740 4 client.17007 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.118+0000 7fd44bd6b740 1 -- 172.21.15.107:0/2128036234 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f9a2cb430 con 0x555f9a373620 2024-05-09T08:37:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.118+0000 7fd441d9b640 1 --2- 172.21.15.107:0/2128036234 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555f9a373620 0x555f9a375a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.119+0000 7fd429ffb640 10 client.17007 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.119+0000 7fd441d9b640 1 --2- 172.21.15.107:0/2128036234 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555f9a373620 0x555f9a375a90 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26824fca50801930 server_cookie=3956844f58b7ab98 in_seq=0 out_seq=0 2024-05-09T08:37:48.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.119+0000 7fd429ffb640 1 -- 172.21.15.107:0/2128036234 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f9a2cb430 con 0x555f9a373620 2024-05-09T08:37:48.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.119+0000 7fd429ffb640 10 client.17007 handle_command_reply: tid=0 2024-05-09T08:37:48.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.120+0000 7fd44bd6b740 1 client.17007 shutdown 2024-05-09T08:37:48.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.120+0000 7fd44bd6b740 1 -- 172.21.15.107:0/2128036234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358280 msgr2=0x555f9a3601d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:48.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.120+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/2128036234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358280 0x555f9a3601d0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fd43400ba80 tx=0x7fd43400bf40 comp rx=0 tx=0).stop 2024-05-09T08:37:48.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.120+0000 7fd44bd6b740 1 -- 172.21.15.107:0/2128036234 shutdown_connections 2024-05-09T08:37:48.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.120+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/2128036234 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555f9a373620 0x555f9a375a90 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.120+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/2128036234 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a363270 0x555f9a360c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.120+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/2128036234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a358ba0 0x555f9a360710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.120+0000 7fd429ffb640 0 client.17007 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:48.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.120+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/2128036234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358280 0x555f9a3601d0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.120+0000 7fd44bd6b740 1 -- 172.21.15.107:0/2128036234 >> 172.21.15.107:0/2128036234 conn(0x555f9a3545a0 msgr2=0x555f9a35a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:48.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.120+0000 7fd44bd6b740 1 -- 172.21.15.107:0/2128036234 shutdown_connections 2024-05-09T08:37:48.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.120+0000 7fd44bd6b740 1 -- 172.21.15.107:0/2128036234 wait complete. 2024-05-09T08:37:48.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.120+0000 7fd44bd6b740 20 client.17007 trim_cache size 0 max 16384 2024-05-09T08:37:48.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd44bd6b740 1 Processor -- start 2024-05-09T08:37:48.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd44bd6b740 1 -- start start 2024-05-09T08:37:48.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd44bd6b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358280 0x555f9a36ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd44bd6b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a358ba0 0x555f9a35f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd44bd6b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a363270 0x555f9a37b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd44bd6b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd4441b1470 con 0x555f9a358280 2024-05-09T08:37:48.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd44bd6b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd4441aed40 con 0x555f9a358ba0 2024-05-09T08:37:48.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd44bd6b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd4441acb90 con 0x555f9a363270 2024-05-09T08:37:48.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd44259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358280 0x555f9a36ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd44259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358280 0x555f9a36ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45116/0 (socket says 172.21.15.107:45116) 2024-05-09T08:37:48.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd44259c640 1 -- 172.21.15.107:0/1552769143 learned_addr learned my addr 172.21.15.107:0/1552769143 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:48.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd442d9d640 1 --2- 172.21.15.107:0/1552769143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a363270 0x555f9a37b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd441d9b640 1 --2- 172.21.15.107:0/1552769143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a358ba0 0x555f9a35f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd44259c640 1 -- 172.21.15.107:0/1552769143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a363270 msgr2=0x555f9a37b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:48.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd44259c640 1 --2- 172.21.15.107:0/1552769143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a363270 0x555f9a37b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd44259c640 1 -- 172.21.15.107:0/1552769143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a358ba0 msgr2=0x555f9a35f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:48.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd44259c640 1 --2- 172.21.15.107:0/1552769143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a358ba0 0x555f9a35f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd44259c640 1 -- 172.21.15.107:0/1552769143 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f9a3543d0 con 0x555f9a358280 2024-05-09T08:37:48.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.141+0000 7fd442d9d640 1 --2- 172.21.15.107:0/1552769143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a363270 0x555f9a37b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:37:48.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.142+0000 7fd44259c640 1 --2- 172.21.15.107:0/1552769143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358280 0x555f9a36ef20 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fd434003b00 tx=0x7fd434003b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=135d33634a3d1ac7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:48.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.142+0000 7fd42bfff640 1 -- 172.21.15.107:0/1552769143 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43402c020 con 0x555f9a358280 2024-05-09T08:37:48.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.142+0000 7fd42bfff640 1 -- 172.21.15.107:0/1552769143 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd43400b480 con 0x555f9a358280 2024-05-09T08:37:48.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.142+0000 7fd42bfff640 1 -- 172.21.15.107:0/1552769143 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43401b8d0 con 0x555f9a358280 2024-05-09T08:37:48.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.142+0000 7fd44bd6b740 1 -- 172.21.15.107:0/1552769143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358280 msgr2=0x555f9a36ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:48.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.142+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/1552769143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358280 0x555f9a36ef20 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fd434003b00 tx=0x7fd434003b30 comp rx=0 tx=0).stop 2024-05-09T08:37:48.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd44bd6b740 1 -- 172.21.15.107:0/1552769143 shutdown_connections 2024-05-09T08:37:48.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/1552769143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a363270 0x555f9a37b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/1552769143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a358ba0 0x555f9a35f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/1552769143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a358280 0x555f9a36ef20 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd44bd6b740 1 -- 172.21.15.107:0/1552769143 >> 172.21.15.107:0/1552769143 conn(0x555f9a3545a0 msgr2=0x555f9a360a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:48.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd44bd6b740 1 -- 172.21.15.107:0/1552769143 shutdown_connections 2024-05-09T08:37:48.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd44bd6b740 1 -- 172.21.15.107:0/1552769143 wait complete. 2024-05-09T08:37:48.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd44bd6b740 1 -- start start 2024-05-09T08:37:48.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd44bd6b740 1 Processor -- start 2024-05-09T08:37:48.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd44bd6b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a358280 0x555f9a366f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd44bd6b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a358ba0 0x555f9a3674d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd44bd6b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a363270 0x555f9a367a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd44bd6b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd4441ad200 con 0x555f9a363270 2024-05-09T08:37:48.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd44bd6b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd4441ae670 con 0x555f9a358280 2024-05-09T08:37:48.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd44bd6b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd4441ad8d0 con 0x555f9a358ba0 2024-05-09T08:37:48.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd44bd6b740 10 client.0 _ll_register_callbacks cb 0x7fd434004270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:48.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd442d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a363270 0x555f9a367a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd442d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a363270 0x555f9a367a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45122/0 (socket says 172.21.15.107:45122) 2024-05-09T08:37:48.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd442d9d640 1 -- 172.21.15.107:0/3941524355 learned_addr learned my addr 172.21.15.107:0/3941524355 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:48.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.143+0000 7fd441d9b640 1 --2- 172.21.15.107:0/3941524355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a358ba0 0x555f9a3674d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.144+0000 7fd44259c640 1 --2- 172.21.15.107:0/3941524355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a358280 0x555f9a366f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.144+0000 7fd442d9d640 1 -- 172.21.15.107:0/3941524355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a358ba0 msgr2=0x555f9a3674d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:48.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.144+0000 7fd442d9d640 1 --2- 172.21.15.107:0/3941524355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a358ba0 0x555f9a3674d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.144+0000 7fd442d9d640 1 -- 172.21.15.107:0/3941524355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a358280 msgr2=0x555f9a366f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:48.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.144+0000 7fd442d9d640 1 --2- 172.21.15.107:0/3941524355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a358280 0x555f9a366f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.144+0000 7fd442d9d640 1 -- 172.21.15.107:0/3941524355 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f9a3d8d30 con 0x555f9a363270 2024-05-09T08:37:48.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.144+0000 7fd442d9d640 1 --2- 172.21.15.107:0/3941524355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a363270 0x555f9a367a10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fd43c0cd800 tx=0x7fd43c009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c981f8f741fbbb95 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:48.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.144+0000 7fd429ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:48.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.144+0000 7fd429ffb640 1 -- 172.21.15.107:0/3941524355 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c0b4040 con 0x555f9a363270 2024-05-09T08:37:48.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.144+0000 7fd429ffb640 1 -- 172.21.15.107:0/3941524355 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd43c098640 con 0x555f9a363270 2024-05-09T08:37:48.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.144+0000 7fd44bd6b740 1 -- 172.21.15.107:0/3941524355 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd43400b480 con 0x555f9a363270 2024-05-09T08:37:48.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.144+0000 7fd429ffb640 1 -- 172.21.15.107:0/3941524355 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd43c09bda0 con 0x555f9a363270 2024-05-09T08:37:48.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.144+0000 7fd44bd6b740 10 client.17013 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:48.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.144+0000 7fd44bd6b740 1 -- 172.21.15.107:0/3941524355 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555f9a3da020 con 0x555f9a363270 2024-05-09T08:37:48.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.144+0000 7fd429ffb640 1 -- 172.21.15.107:0/3941524355 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd43400b480 con 0x555f9a363270 2024-05-09T08:37:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.145+0000 7fd429ffb640 1 -- 172.21.15.107:0/3941524355 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd43c0a4820 con 0x555f9a363270 2024-05-09T08:37:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.145+0000 7fd44bd6b740 10 client.17013 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.145+0000 7fd44bd6b740 10 client.17013 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.145+0000 7fd44bd6b740 20 client.17013 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.145+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/3941524355 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555f9a3da3b0 0x555f9a3dc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.145+0000 7fd44bd6b740 4 client.17013 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:37:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.145+0000 7fd44bd6b740 1 -- 172.21.15.107:0/3941524355 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555f9a2c5780 con 0x555f9a3da3b0 2024-05-09T08:37:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.145+0000 7fd44259c640 1 --2- 172.21.15.107:0/3941524355 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555f9a3da3b0 0x555f9a3dc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.145+0000 7fd429ffb640 10 client.17013 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.145+0000 7fd44259c640 1 --2- 172.21.15.107:0/3941524355 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555f9a3da3b0 0x555f9a3dc7a0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53a56fb1d5fbbbda server_cookie=8a7977fa2f83f5d9 in_seq=0 out_seq=0 2024-05-09T08:37:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.145+0000 7fd429ffb640 1 -- 172.21.15.107:0/3941524355 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555f9a2c5780 con 0x555f9a3da3b0 2024-05-09T08:37:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.145+0000 7fd429ffb640 10 client.17013 handle_command_reply: tid=0 2024-05-09T08:37:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.145+0000 7fd44bd6b740 1 client.17013 shutdown 2024-05-09T08:37:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.145+0000 7fd44bd6b740 1 -- 172.21.15.107:0/3941524355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a363270 msgr2=0x555f9a367a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:48.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.145+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/3941524355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a363270 0x555f9a367a10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fd43c0cd800 tx=0x7fd43c009590 comp rx=0 tx=0).stop 2024-05-09T08:37:48.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.145+0000 7fd44bd6b740 1 -- 172.21.15.107:0/3941524355 shutdown_connections 2024-05-09T08:37:48.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/3941524355 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555f9a3da3b0 0x555f9a3dc7a0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/3941524355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f9a363270 0x555f9a367a10 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/3941524355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f9a358ba0 0x555f9a3674d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd429ffb640 0 client.17013 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:48.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd44bd6b740 1 --2- 172.21.15.107:0/3941524355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f9a358280 0x555f9a366f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd44bd6b740 1 -- 172.21.15.107:0/3941524355 >> 172.21.15.107:0/3941524355 conn(0x555f9a3545a0 msgr2=0x555f9a359a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:48.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd44bd6b740 1 -- 172.21.15.107:0/3941524355 shutdown_connections 2024-05-09T08:37:48.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd44bd6b740 1 -- 172.21.15.107:0/3941524355 wait complete. 2024-05-09T08:37:48.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd44bd6b740 20 client.17013 trim_cache size 0 max 16384 2024-05-09T08:37:48.153 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:37:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:37:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:37:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "dbf46e95-13ff-4f10-8354-ab8e4994df62": { 2024-05-09T08:37:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:37:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "dbf46e95-13ff-4f10-8354-ab8e4994df62", 2024-05-09T08:37:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:37:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:37:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:37:48.153 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:37:48.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd448f7e640 1 -- 172.21.15.107:0/243451334 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd42c0a2d00 msgr2=0x7fd42c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:48.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd448f7e640 1 --2- 172.21.15.107:0/243451334 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd42c0a2d00 0x7fd42c0a51c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd44410dc80 tx=0x7fd438003860 comp rx=0 tx=0).stop 2024-05-09T08:37:48.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd448f7e640 1 -- 172.21.15.107:0/243451334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd44410bdf0 msgr2=0x7fd44419c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:48.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd448f7e640 1 --2- 172.21.15.107:0/243451334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd44410bdf0 0x7fd44419c920 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fd43c009fd0 tx=0x7fd43c00ef10 comp rx=0 tx=0).stop 2024-05-09T08:37:48.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd448f7e640 1 -- 172.21.15.107:0/243451334 shutdown_connections 2024-05-09T08:37:48.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd448f7e640 1 --2- 172.21.15.107:0/243451334 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd42c0a2d00 0x7fd42c0a51c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd448f7e640 1 --2- 172.21.15.107:0/243451334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd44410bdf0 0x7fd44419c920 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd448f7e640 1 --2- 172.21.15.107:0/243451334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4441033d0 0x7fd4441985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd448f7e640 1 --2- 172.21.15.107:0/243451334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd444102ab0 0x7fd4441980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:48.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd448f7e640 1 -- 172.21.15.107:0/243451334 >> 172.21.15.107:0/243451334 conn(0x7fd444077830 msgr2=0x7fd4440fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:48.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd448f7e640 1 -- 172.21.15.107:0/243451334 shutdown_connections 2024-05-09T08:37:48.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:37:48.146+0000 7fd448f7e640 1 -- 172.21.15.107:0/243451334 wait complete. 2024-05-09T08:37:48.158 INFO:tasks.cephfs.filesystem:scrub status for tag:dbf46e95-13ff-4f10-8354-ab8e4994df62 - {'path': '/', 'tag': 'dbf46e95-13ff-4f10-8354-ab8e4994df62', 'options': 'recursive,force'} 2024-05-09T08:37:48.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:48 smithi107 ceph-mon[30122]: pgmap v136: 129 pgs: 129 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:37:48.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:48 smithi157 ceph-mon[33614]: pgmap v136: 129 pgs: 129 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:37:48.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:48 smithi110 ceph-mon[33622]: pgmap v136: 129 pgs: 129 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:37:49.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:49 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:37:49.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:49 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:49 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:37:49.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:49 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:37:49.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:49 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:49 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:37:49.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:49 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:49 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:37:49.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:49 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:49 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:37:49.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:49 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:49 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:37:49.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:49 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:49 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:49 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:37:49.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:49 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:49 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:37:49.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:49 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:37:49.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:49 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:49 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:37:49.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:49 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:49 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:37:49.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:49 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:49 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:37:49.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:49 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:49 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:37:49.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:49 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:49 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:49 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:37:49.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:49 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:49 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:37:49.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:49 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:37:49.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:49 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:49 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:37:49.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:49 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:49 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:37:49.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:49 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:49 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:37:49.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:49 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:49 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:37:49.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:49 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:49.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:49 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:37:50.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:50 smithi107 ceph-mon[30122]: pgmap v137: 129 pgs: 129 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-05-09T08:37:50.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:50 smithi157 ceph-mon[33614]: pgmap v137: 129 pgs: 129 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-05-09T08:37:50.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:50 smithi110 ceph-mon[33622]: pgmap v137: 129 pgs: 129 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-05-09T08:37:52.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:52 smithi107 ceph-mon[30122]: pgmap v138: 129 pgs: 129 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-05-09T08:37:52.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:52 smithi157 ceph-mon[33614]: pgmap v138: 129 pgs: 129 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-05-09T08:37:52.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:52 smithi110 ceph-mon[33622]: pgmap v138: 129 pgs: 129 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-05-09T08:37:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.511+0000 7f3bc1df4640 1 -- 172.21.15.107:0/468569949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bbc106720 msgr2=0x7f3bbc106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.511+0000 7f3bc1df4640 1 --2- 172.21.15.107:0/468569949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bbc106720 0x7f3bbc106b00 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f3bac009930 tx=0x7f3bac02a420 comp rx=0 tx=0).stop 2024-05-09T08:37:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.511+0000 7f3bc1df4640 1 -- 172.21.15.107:0/468569949 shutdown_connections 2024-05-09T08:37:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.511+0000 7f3bc1df4640 1 --2- 172.21.15.107:0/468569949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bbc101120 0x7f3bbc10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.511+0000 7f3bc1df4640 1 --2- 172.21.15.107:0/468569949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3bbc100760 0x7f3bbc100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.511+0000 7f3bc1df4640 1 --2- 172.21.15.107:0/468569949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bbc106720 0x7f3bbc106b00 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.511+0000 7f3bc1df4640 1 -- 172.21.15.107:0/468569949 >> 172.21.15.107:0/468569949 conn(0x7f3bbc0778b0 msgr2=0x7f3bbc077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.511+0000 7f3bc1df4640 1 -- 172.21.15.107:0/468569949 shutdown_connections 2024-05-09T08:37:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.511+0000 7f3bc1df4640 1 -- 172.21.15.107:0/468569949 wait complete. 2024-05-09T08:37:53.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.511+0000 7f3bc1df4640 1 Processor -- start 2024-05-09T08:37:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bc1df4640 1 -- start start 2024-05-09T08:37:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bc1df4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bbc100760 0x7f3bbc198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bc1df4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bbc101120 0x7f3bbc1993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bc1df4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3bbc106720 0x7f3bbc1a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bc1df4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc1109e0 con 0x7f3bbc101120 2024-05-09T08:37:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bc1df4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc110860 con 0x7f3bbc106720 2024-05-09T08:37:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bc1df4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc110b60 con 0x7f3bbc100760 2024-05-09T08:37:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bbaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bbc101120 0x7f3bbc1993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bbaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bbc101120 0x7f3bbc1993a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35124/0 (socket says 172.21.15.107:35124) 2024-05-09T08:37:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bbaffd640 1 -- 172.21.15.107:0/4189687891 learned_addr learned my addr 172.21.15.107:0/4189687891 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bbbfff640 1 --2- 172.21.15.107:0/4189687891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3bbc106720 0x7f3bbc1a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bbb7fe640 1 --2- 172.21.15.107:0/4189687891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bbc100760 0x7f3bbc198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bbaffd640 1 -- 172.21.15.107:0/4189687891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bbc100760 msgr2=0x7f3bbc198e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/4189687891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bbc100760 0x7f3bbc198e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bbaffd640 1 -- 172.21.15.107:0/4189687891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3bbc106720 msgr2=0x7f3bbc1a3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/4189687891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3bbc106720 0x7f3bbc1a3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.512+0000 7f3bbaffd640 1 -- 172.21.15.107:0/4189687891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bbc1a4420 con 0x7f3bbc101120 2024-05-09T08:37:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.513+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/4189687891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bbc101120 0x7f3bbc1993a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f3bb000d950 tx=0x7f3bb000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.513+0000 7f3bb8ff9640 1 -- 172.21.15.107:0/4189687891 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb0017030 con 0x7f3bbc101120 2024-05-09T08:37:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.513+0000 7f3bb8ff9640 1 -- 172.21.15.107:0/4189687891 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3bb0004020 con 0x7f3bbc101120 2024-05-09T08:37:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.513+0000 7f3bc1df4640 1 -- 172.21.15.107:0/4189687891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bbc1a45f0 con 0x7f3bbc101120 2024-05-09T08:37:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.513+0000 7f3bb8ff9640 1 -- 172.21.15.107:0/4189687891 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb00047a0 con 0x7f3bbc101120 2024-05-09T08:37:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.513+0000 7f3bc1df4640 1 -- 172.21.15.107:0/4189687891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bbc1a4b90 con 0x7f3bbc101120 2024-05-09T08:37:53.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.515+0000 7f3bb8ff9640 1 -- 172.21.15.107:0/4189687891 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3bb00041d0 con 0x7f3bbc101120 2024-05-09T08:37:53.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.516+0000 7f3bb8ff9640 1 --2- 172.21.15.107:0/4189687891 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3ba40a2e20 0x7f3ba40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.516+0000 7f3bc4bd6740 1 Processor -- start 2024-05-09T08:37:53.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.516+0000 7f3bbb7fe640 1 --2- 172.21.15.107:0/4189687891 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3ba40a2e20 0x7f3ba40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.516+0000 7f3bb8ff9640 1 -- 172.21.15.107:0/4189687891 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3bb00cf170 con 0x7f3bbc101120 2024-05-09T08:37:53.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.516+0000 7f3bc4bd6740 1 -- start start 2024-05-09T08:37:53.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.516+0000 7f3bc4bd6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1e2f0 0x555eb1e1e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.516+0000 7f3bc4bd6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1ec10 0x555eb1e2d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.516+0000 7f3bc4bd6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e2da40 0x555eb1e2fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.516+0000 7f3bc4bd6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555eb1d83340 con 0x555eb1e1ec10 2024-05-09T08:37:53.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.516+0000 7f3bc4bd6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555eb1d2b290 con 0x555eb1e2da40 2024-05-09T08:37:53.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.516+0000 7f3bc4bd6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x555eb1d7f300 con 0x555eb1e1e2f0 2024-05-09T08:37:53.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.516+0000 7f3bbaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1ec10 0x555eb1e2d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.516+0000 7f3bbaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1ec10 0x555eb1e2d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35138/0 (socket says 172.21.15.107:35138) 2024-05-09T08:37:53.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.516+0000 7f3bbaffd640 1 -- 172.21.15.107:0/826192050 learned_addr learned my addr 172.21.15.107:0/826192050 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:53.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.517+0000 7f3bbbfff640 1 --2- 172.21.15.107:0/826192050 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e2da40 0x555eb1e2fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.517+0000 7f3bbb7fe640 1 --2- 172.21.15.107:0/4189687891 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3ba40a2e20 0x7f3ba40a52e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3bac009900 tx=0x7f3bac000bc0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:53.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.517+0000 7f3bbaffd640 1 -- 172.21.15.107:0/826192050 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1e2f0 msgr2=0x555eb1e1e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:53.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.517+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/826192050 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1e2f0 0x555eb1e1e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.517+0000 7f3bbaffd640 1 -- 172.21.15.107:0/826192050 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e2da40 msgr2=0x555eb1e2fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:53.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.517+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/826192050 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e2da40 0x555eb1e2fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.517+0000 7f3bbaffd640 1 -- 172.21.15.107:0/826192050 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555eb1d4f8e0 con 0x555eb1e1ec10 2024-05-09T08:37:53.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.517+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/826192050 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1ec10 0x555eb1e2d500 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0097ca0 tx=0x7f3bb009af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb86eb7a9f8384a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.517+0000 7f3ba1ffb640 1 -- 172.21.15.107:0/826192050 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb00a3f40 con 0x555eb1e1ec10 2024-05-09T08:37:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.517+0000 7f3ba1ffb640 1 -- 172.21.15.107:0/826192050 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3bb00cc780 con 0x555eb1e1ec10 2024-05-09T08:37:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.517+0000 7f3ba1ffb640 1 -- 172.21.15.107:0/826192050 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb00bb070 con 0x555eb1e1ec10 2024-05-09T08:37:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.518+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/826192050 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1ec10 msgr2=0x555eb1e2d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.518+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/826192050 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1ec10 0x555eb1e2d500 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0097ca0 tx=0x7f3bb009af40 comp rx=0 tx=0).stop 2024-05-09T08:37:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.518+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/826192050 shutdown_connections 2024-05-09T08:37:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.518+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/826192050 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e2da40 0x555eb1e2fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.518+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/826192050 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1ec10 0x555eb1e2d500 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.518+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/826192050 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1e2f0 0x555eb1e1e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.518+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/826192050 >> 172.21.15.107:0/826192050 conn(0x555eb1e1a960 msgr2=0x555eb1e1ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.518+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/826192050 shutdown_connections 2024-05-09T08:37:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.518+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/826192050 wait complete. 2024-05-09T08:37:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.518+0000 7f3bc4bd6740 1 -- start start 2024-05-09T08:37:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.518+0000 7f3bc4bd6740 1 Processor -- start 2024-05-09T08:37:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bc4bd6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1e2f0 0x555eb1e2aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bc4bd6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1ec10 0x555eb1e2af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bc4bd6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e2da40 0x555eb1e2b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bc4bd6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555eb1e305b0 con 0x555eb1e1ec10 2024-05-09T08:37:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bc4bd6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555eb1d3e790 con 0x555eb1e2da40 2024-05-09T08:37:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bc4bd6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x555eb1e30730 con 0x555eb1e1e2f0 2024-05-09T08:37:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bc4bd6740 10 client.0 _ll_register_callbacks cb 0x555eb1d79a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bbaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1ec10 0x555eb1e2af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bbaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1ec10 0x555eb1e2af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35150/0 (socket says 172.21.15.107:35150) 2024-05-09T08:37:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bbaffd640 1 -- 172.21.15.107:0/718928357 learned_addr learned my addr 172.21.15.107:0/718928357 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bbb7fe640 1 --2- 172.21.15.107:0/718928357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1e2f0 0x555eb1e2aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bbbfff640 1 --2- 172.21.15.107:0/718928357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e2da40 0x555eb1e2b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bbaffd640 1 -- 172.21.15.107:0/718928357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1e2f0 msgr2=0x555eb1e2aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/718928357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1e2f0 0x555eb1e2aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bbaffd640 1 -- 172.21.15.107:0/718928357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e2da40 msgr2=0x555eb1e2b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/718928357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e2da40 0x555eb1e2b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bbaffd640 1 -- 172.21.15.107:0/718928357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555eb1a97210 con 0x555eb1e1ec10 2024-05-09T08:37:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bbbfff640 1 --2- 172.21.15.107:0/718928357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e2da40 0x555eb1e2b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:37:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/718928357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1ec10 0x555eb1e2af60 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009610 tx=0x7f3bb009f7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e851c64700410b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.519+0000 7f3b9affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.520+0000 7f3b9affd640 1 -- 172.21.15.107:0/718928357 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb00b9060 con 0x555eb1e1ec10 2024-05-09T08:37:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.520+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/718928357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bb00cc780 con 0x555eb1e1ec10 2024-05-09T08:37:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.520+0000 7f3b9affd640 1 -- 172.21.15.107:0/718928357 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3bb0097940 con 0x555eb1e1ec10 2024-05-09T08:37:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.520+0000 7f3b9affd640 1 -- 172.21.15.107:0/718928357 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3bb00a3f40 con 0x555eb1e1ec10 2024-05-09T08:37:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.520+0000 7f3bc4bd6740 10 client.17025 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.520+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/718928357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555eb1b63620 con 0x555eb1e1ec10 2024-05-09T08:37:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.520+0000 7f3b9affd640 1 -- 172.21.15.107:0/718928357 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3bb00cc780 con 0x555eb1e1ec10 2024-05-09T08:37:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.520+0000 7f3b9affd640 1 -- 172.21.15.107:0/718928357 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3bb00bb250 con 0x555eb1e1ec10 2024-05-09T08:37:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.520+0000 7f3bc4bd6740 10 client.17025 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.520+0000 7f3bc4bd6740 10 client.17025 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.520+0000 7f3bc4bd6740 20 client.17025 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.520+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/718928357 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555eb1e3d5f0 0x555eb1e3f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.520+0000 7f3bc4bd6740 4 client.17025 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:37:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.520+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/718928357 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555eb1d90a70 con 0x555eb1e3d5f0 2024-05-09T08:37:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.521+0000 7f3bbb7fe640 1 --2- 172.21.15.107:0/718928357 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555eb1e3d5f0 0x555eb1e3f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.521+0000 7f3b9affd640 10 client.17025 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.521+0000 7f3bbb7fe640 1 --2- 172.21.15.107:0/718928357 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555eb1e3d5f0 0x555eb1e3f9e0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ff2f51d4faabe84 server_cookie=bc6d56826ae297b1 in_seq=0 out_seq=0 2024-05-09T08:37:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.522+0000 7f3b9affd640 1 -- 172.21.15.107:0/718928357 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555eb1d90a70 con 0x555eb1e3d5f0 2024-05-09T08:37:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.522+0000 7f3b9affd640 10 client.17025 handle_command_reply: tid=0 2024-05-09T08:37:53.530 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.522+0000 7f3bc4bd6740 1 client.17025 shutdown 2024-05-09T08:37:53.530 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.522+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/718928357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1ec10 msgr2=0x555eb1e2af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:53.530 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.522+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/718928357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1ec10 0x555eb1e2af60 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009610 tx=0x7f3bb009f7b0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.530 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.522+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/718928357 shutdown_connections 2024-05-09T08:37:53.530 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.522+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/718928357 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555eb1e3d5f0 0x555eb1e3f9e0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.530 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.522+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/718928357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e2da40 0x555eb1e2b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.530 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.522+0000 7f3b9affd640 0 client.17025 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:53.530 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.522+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/718928357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1ec10 0x555eb1e2af60 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.530 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.522+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/718928357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1e2f0 0x555eb1e2aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.531 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.522+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/718928357 >> 172.21.15.107:0/718928357 conn(0x555eb1e1a960 msgr2=0x555eb1e207d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:53.531 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.522+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/718928357 shutdown_connections 2024-05-09T08:37:53.531 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.522+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/718928357 wait complete. 2024-05-09T08:37:53.531 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.522+0000 7f3bc4bd6740 20 client.17025 trim_cache size 0 max 16384 2024-05-09T08:37:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.537+0000 7f3bc4bd6740 1 Processor -- start 2024-05-09T08:37:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.537+0000 7f3bc4bd6740 1 -- start start 2024-05-09T08:37:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.537+0000 7f3bc4bd6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e1e2f0 0x555eb1e2d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.537+0000 7f3bc4bd6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1ec10 0x555eb1e2b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.537+0000 7f3bc4bd6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e2da40 0x555eb1e42a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.537+0000 7f3bc4bd6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc1af1c0 con 0x555eb1e2da40 2024-05-09T08:37:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.537+0000 7f3bc4bd6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc1aca90 con 0x555eb1e1e2f0 2024-05-09T08:37:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.537+0000 7f3bc4bd6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc1aa8e0 con 0x555eb1e1ec10 2024-05-09T08:37:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.537+0000 7f3bbbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e2da40 0x555eb1e42a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.537+0000 7f3bbbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e2da40 0x555eb1e42a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35154/0 (socket says 172.21.15.107:35154) 2024-05-09T08:37:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.537+0000 7f3bbbfff640 1 -- 172.21.15.107:0/103303114 learned_addr learned my addr 172.21.15.107:0/103303114 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.537+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/103303114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1ec10 0x555eb1e2b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.538+0000 7f3bbb7fe640 1 --2- 172.21.15.107:0/103303114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e1e2f0 0x555eb1e2d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.538+0000 7f3bbbfff640 1 -- 172.21.15.107:0/103303114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1ec10 msgr2=0x555eb1e2b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.538+0000 7f3bbbfff640 1 --2- 172.21.15.107:0/103303114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1ec10 0x555eb1e2b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.538+0000 7f3bbbfff640 1 -- 172.21.15.107:0/103303114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e1e2f0 msgr2=0x555eb1e2d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.538+0000 7f3bbbfff640 1 --2- 172.21.15.107:0/103303114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e1e2f0 0x555eb1e2d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.538+0000 7f3bbbfff640 1 -- 172.21.15.107:0/103303114 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555eb1b63620 con 0x555eb1e2da40 2024-05-09T08:37:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.538+0000 7f3bbbfff640 1 --2- 172.21.15.107:0/103303114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e2da40 0x555eb1e42a00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4009f10 tx=0x7f3bb40127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33cf9b1bb662e28a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.538+0000 7f3ba17fa640 1 -- 172.21.15.107:0/103303114 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4004d20 con 0x555eb1e2da40 2024-05-09T08:37:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.538+0000 7f3ba17fa640 1 -- 172.21.15.107:0/103303114 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3bb4003e20 con 0x555eb1e2da40 2024-05-09T08:37:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.538+0000 7f3ba17fa640 1 -- 172.21.15.107:0/103303114 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4005280 con 0x555eb1e2da40 2024-05-09T08:37:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.539+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/103303114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e2da40 msgr2=0x555eb1e42a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.539+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/103303114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e2da40 0x555eb1e42a00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4009f10 tx=0x7f3bb40127c0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.539+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/103303114 shutdown_connections 2024-05-09T08:37:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.539+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/103303114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e2da40 0x555eb1e42a00 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.539+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/103303114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1ec10 0x555eb1e2b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.539+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/103303114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e1e2f0 0x555eb1e2d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.539+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/103303114 >> 172.21.15.107:0/103303114 conn(0x555eb1e1a960 msgr2=0x555eb1e39210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.539+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/103303114 shutdown_connections 2024-05-09T08:37:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.539+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/103303114 wait complete. 2024-05-09T08:37:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.539+0000 7f3bc4bd6740 1 -- start start 2024-05-09T08:37:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bc4bd6740 1 Processor -- start 2024-05-09T08:37:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bc4bd6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1e2f0 0x555eb1e42ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bc4bd6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1ec10 0x555eb1e40e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bc4bd6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e2da40 0x555eb1e41390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bc4bd6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc1aaf50 con 0x555eb1e1e2f0 2024-05-09T08:37:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bc4bd6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc1ac3c0 con 0x555eb1e2da40 2024-05-09T08:37:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bc4bd6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc1ab620 con 0x555eb1e1ec10 2024-05-09T08:37:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bc4bd6740 10 client.0 _ll_register_callbacks cb 0x7f3bb009ad00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:37:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bbb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1e2f0 0x555eb1e42ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bbb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1e2f0 0x555eb1e42ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35170/0 (socket says 172.21.15.107:35170) 2024-05-09T08:37:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bbb7fe640 1 -- 172.21.15.107:0/2109075744 learned_addr learned my addr 172.21.15.107:0/2109075744 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:37:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bbbfff640 1 --2- 172.21.15.107:0/2109075744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e2da40 0x555eb1e41390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bbb7fe640 1 -- 172.21.15.107:0/2109075744 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1ec10 msgr2=0x555eb1e40e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/2109075744 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1ec10 0x555eb1e40e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bbb7fe640 1 --2- 172.21.15.107:0/2109075744 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1ec10 0x555eb1e40e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bbb7fe640 1 -- 172.21.15.107:0/2109075744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e2da40 msgr2=0x555eb1e41390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:37:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bbb7fe640 1 --2- 172.21.15.107:0/2109075744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e2da40 0x555eb1e41390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.540+0000 7f3bbb7fe640 1 -- 172.21.15.107:0/2109075744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555eb1e418d0 con 0x555eb1e1e2f0 2024-05-09T08:37:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.541+0000 7f3bbb7fe640 1 --2- 172.21.15.107:0/2109075744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1e2f0 0x555eb1e42ab0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f3bac031c60 tx=0x7f3bac009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9e231a67b026ec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:37:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.541+0000 7f3b9affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:37:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.541+0000 7f3b9affd640 1 -- 172.21.15.107:0/2109075744 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac05b060 con 0x555eb1e1e2f0 2024-05-09T08:37:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.541+0000 7f3b9affd640 1 -- 172.21.15.107:0/2109075744 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3bac0037f0 con 0x555eb1e1e2f0 2024-05-09T08:37:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.541+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/2109075744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bb4003e20 con 0x555eb1e1e2f0 2024-05-09T08:37:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.541+0000 7f3b9affd640 1 -- 172.21.15.107:0/2109075744 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3bac062050 con 0x555eb1e1e2f0 2024-05-09T08:37:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.541+0000 7f3bc4bd6740 10 client.17031 fetch_fsmap learned FSMap version 27 2024-05-09T08:37:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.541+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/2109075744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555eb1ea4510 con 0x555eb1e1e2f0 2024-05-09T08:37:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.541+0000 7f3b9affd640 1 -- 172.21.15.107:0/2109075744 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3bb4003e20 con 0x555eb1e1e2f0 2024-05-09T08:37:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.541+0000 7f3b9affd640 1 -- 172.21.15.107:0/2109075744 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3bac056970 con 0x555eb1e1e2f0 2024-05-09T08:37:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.541+0000 7f3bc4bd6740 10 client.17031 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:37:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.541+0000 7f3bc4bd6740 10 client.17031 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:37:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.541+0000 7f3bc4bd6740 20 client.17031 populate_metadata read hostname 'smithi107' 2024-05-09T08:37:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.541+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/2109075744 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555eb1e3d5f0 0x555eb1ea6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:37:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.541+0000 7f3bc4bd6740 4 client.17031 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:37:53.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.541+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/2109075744 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555eb1d8adc0 con 0x555eb1e3d5f0 2024-05-09T08:37:53.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.542+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/2109075744 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555eb1e3d5f0 0x555eb1ea6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:37:53.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.542+0000 7f3b9affd640 10 client.17031 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:53.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.542+0000 7f3bbaffd640 1 --2- 172.21.15.107:0/2109075744 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555eb1e3d5f0 0x555eb1ea6c50 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bab6ecc6140c8ff2 server_cookie=681dbc3d4fed956 in_seq=0 out_seq=0 2024-05-09T08:37:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.561+0000 7f3b9affd640 1 -- 172.21.15.107:0/2109075744 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555eb1d8adc0 con 0x555eb1e3d5f0 2024-05-09T08:37:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.561+0000 7f3b9affd640 10 client.17031 handle_command_reply: tid=0 2024-05-09T08:37:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.561+0000 7f3bc4bd6740 1 client.17031 shutdown 2024-05-09T08:37:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.561+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/2109075744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1e2f0 msgr2=0x555eb1e42ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.561+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/2109075744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1e2f0 0x555eb1e42ab0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f3bac031c60 tx=0x7f3bac009260 comp rx=0 tx=0).stop 2024-05-09T08:37:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.561+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/2109075744 shutdown_connections 2024-05-09T08:37:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.561+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/2109075744 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555eb1e3d5f0 0x555eb1ea6c50 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.561+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/2109075744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555eb1e2da40 0x555eb1e41390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.561+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/2109075744 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555eb1e1ec10 0x555eb1e40e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.562 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.561+0000 7f3b9affd640 0 client.17031 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:37:53.562 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.561+0000 7f3bc4bd6740 1 --2- 172.21.15.107:0/2109075744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555eb1e1e2f0 0x555eb1e42ab0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.562 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.561+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/2109075744 >> 172.21.15.107:0/2109075744 conn(0x555eb1e1a960 msgr2=0x555eb1e1fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:53.562 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.561+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/2109075744 shutdown_connections 2024-05-09T08:37:53.562 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.561+0000 7f3bc4bd6740 1 -- 172.21.15.107:0/2109075744 wait complete. 2024-05-09T08:37:53.562 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.561+0000 7f3bc4bd6740 20 client.17031 trim_cache size 0 max 16384 2024-05-09T08:37:53.562 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:37:53.562 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:37:53.562 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:37:53.562 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:37:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.562+0000 7f3bc1df4640 1 -- 172.21.15.107:0/4189687891 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3ba40a2e20 msgr2=0x7f3ba40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.562+0000 7f3bc1df4640 1 --2- 172.21.15.107:0/4189687891 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3ba40a2e20 0x7f3ba40a52e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3bac009900 tx=0x7f3bac000bc0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.562+0000 7f3bc1df4640 1 -- 172.21.15.107:0/4189687891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bbc101120 msgr2=0x7f3bbc1993a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:37:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.562+0000 7f3bc1df4640 1 --2- 172.21.15.107:0/4189687891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bbc101120 0x7f3bbc1993a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f3bb000d950 tx=0x7f3bb000de10 comp rx=0 tx=0).stop 2024-05-09T08:37:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.562+0000 7f3bc1df4640 1 -- 172.21.15.107:0/4189687891 shutdown_connections 2024-05-09T08:37:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.562+0000 7f3bc1df4640 1 --2- 172.21.15.107:0/4189687891 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3ba40a2e20 0x7f3ba40a52e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.562+0000 7f3bc1df4640 1 --2- 172.21.15.107:0/4189687891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3bbc106720 0x7f3bbc1a3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.562+0000 7f3bc1df4640 1 --2- 172.21.15.107:0/4189687891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bbc101120 0x7f3bbc1993a0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.562+0000 7f3bc1df4640 1 --2- 172.21.15.107:0/4189687891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bbc100760 0x7f3bbc198e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:37:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.562+0000 7f3bc1df4640 1 -- 172.21.15.107:0/4189687891 >> 172.21.15.107:0/4189687891 conn(0x7f3bbc0778b0 msgr2=0x7f3bbc103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:37:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.562+0000 7f3bc1df4640 1 -- 172.21.15.107:0/4189687891 shutdown_connections 2024-05-09T08:37:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:37:53.562+0000 7f3bc1df4640 1 -- 172.21.15.107:0/4189687891 wait complete. 2024-05-09T08:37:54.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:54 smithi107 ceph-mon[30122]: pgmap v139: 129 pgs: 129 active+clean; 5.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 36 MiB/s wr, 55 op/s 2024-05-09T08:37:54.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:54 smithi157 ceph-mon[33614]: pgmap v139: 129 pgs: 129 active+clean; 5.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 36 MiB/s wr, 55 op/s 2024-05-09T08:37:54.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:54 smithi110 ceph-mon[33622]: pgmap v139: 129 pgs: 129 active+clean; 5.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 36 MiB/s wr, 55 op/s 2024-05-09T08:37:56.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:56 smithi107 ceph-mon[30122]: pgmap v140: 129 pgs: 129 active+clean; 5.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 24 MiB/s wr, 43 op/s 2024-05-09T08:37:56.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:56 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:37:56.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:56 smithi157 ceph-mon[33614]: pgmap v140: 129 pgs: 129 active+clean; 5.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 24 MiB/s wr, 43 op/s 2024-05-09T08:37:56.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:56 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:37:56.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:56 smithi110 ceph-mon[33622]: pgmap v140: 129 pgs: 129 active+clean; 5.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 24 MiB/s wr, 43 op/s 2024-05-09T08:37:56.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:56 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:37:57.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:57 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:37:57.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:57 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:37:57.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:57 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:37:57.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:57 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:37:57.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:57 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:37:57.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:57 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:37:58.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:37:58 smithi107 ceph-mon[30122]: pgmap v141: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 36 MiB/s wr, 54 op/s 2024-05-09T08:37:58.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:37:58 smithi157 ceph-mon[33614]: pgmap v141: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 36 MiB/s wr, 54 op/s 2024-05-09T08:37:58.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:37:58 smithi110 ceph-mon[33622]: pgmap v141: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 36 MiB/s wr, 54 op/s 2024-05-09T08:38:00.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:00 smithi107 ceph-mon[30122]: pgmap v142: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 24 MiB/s wr, 47 op/s 2024-05-09T08:38:00.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:00 smithi157 ceph-mon[33614]: pgmap v142: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 24 MiB/s wr, 47 op/s 2024-05-09T08:38:00.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:00 smithi110 ceph-mon[33622]: pgmap v142: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 24 MiB/s wr, 47 op/s 2024-05-09T08:38:02.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:02 smithi107 ceph-mon[30122]: pgmap v143: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 24 MiB/s wr, 47 op/s 2024-05-09T08:38:02.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:02 smithi157 ceph-mon[33614]: pgmap v143: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 24 MiB/s wr, 47 op/s 2024-05-09T08:38:02.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:02 smithi110 ceph-mon[33622]: pgmap v143: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 24 MiB/s wr, 47 op/s 2024-05-09T08:38:04.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:04 smithi107 ceph-mon[30122]: pgmap v144: 129 pgs: 129 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 37 MiB/s wr, 70 op/s 2024-05-09T08:38:04.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:04 smithi157 ceph-mon[33614]: pgmap v144: 129 pgs: 129 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 37 MiB/s wr, 70 op/s 2024-05-09T08:38:04.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:04 smithi110 ceph-mon[33622]: pgmap v144: 129 pgs: 129 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 37 MiB/s wr, 70 op/s 2024-05-09T08:38:06.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:06 smithi157 ceph-mon[33614]: pgmap v145: 129 pgs: 129 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 25 MiB/s wr, 34 op/s 2024-05-09T08:38:06.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:06 smithi110 ceph-mon[33622]: pgmap v145: 129 pgs: 129 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 25 MiB/s wr, 34 op/s 2024-05-09T08:38:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.663+0000 7fda41509640 1 -- 172.21.15.107:0/3920698302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda3c101120 msgr2=0x7fda3c10d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.663+0000 7fda41509640 1 --2- 172.21.15.107:0/3920698302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda3c101120 0x7fda3c10d5e0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fda34009cd0 tx=0x7fda3402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.663+0000 7fda41509640 1 -- 172.21.15.107:0/3920698302 shutdown_connections 2024-05-09T08:38:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.663+0000 7fda41509640 1 --2- 172.21.15.107:0/3920698302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda3c101120 0x7fda3c10d5e0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.663+0000 7fda41509640 1 --2- 172.21.15.107:0/3920698302 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda3c100760 0x7fda3c100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.663+0000 7fda41509640 1 --2- 172.21.15.107:0/3920698302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda3c106720 0x7fda3c106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.663+0000 7fda41509640 1 -- 172.21.15.107:0/3920698302 >> 172.21.15.107:0/3920698302 conn(0x7fda3c0778b0 msgr2=0x7fda3c077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.663+0000 7fda41509640 1 -- 172.21.15.107:0/3920698302 shutdown_connections 2024-05-09T08:38:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.663+0000 7fda41509640 1 -- 172.21.15.107:0/3920698302 wait complete. 2024-05-09T08:38:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.664+0000 7fda41509640 1 Processor -- start 2024-05-09T08:38:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.664+0000 7fda41509640 1 -- start start 2024-05-09T08:38:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.664+0000 7fda41509640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda3c100760 0x7fda3c198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.664+0000 7fda41509640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda3c101120 0x7fda3c199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.664+0000 7fda41509640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda3c106720 0x7fda3c1a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.664+0000 7fda41509640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c1109a0 con 0x7fda3c100760 2024-05-09T08:38:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.664+0000 7fda41509640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c110820 con 0x7fda3c106720 2024-05-09T08:38:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.664+0000 7fda41509640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c110b20 con 0x7fda3c101120 2024-05-09T08:38:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.664+0000 7fda3affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda3c100760 0x7fda3c198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.664+0000 7fda3affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda3c100760 0x7fda3c198dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33922/0 (socket says 172.21.15.107:33922) 2024-05-09T08:38:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.664+0000 7fda3affd640 1 -- 172.21.15.107:0/860219984 learned_addr learned my addr 172.21.15.107:0/860219984 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.664+0000 7fda3a7fc640 1 --2- 172.21.15.107:0/860219984 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda3c101120 0x7fda3c199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.665+0000 7fda3b7fe640 1 --2- 172.21.15.107:0/860219984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda3c106720 0x7fda3c1a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.665+0000 7fda3affd640 1 -- 172.21.15.107:0/860219984 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda3c101120 msgr2=0x7fda3c199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.665+0000 7fda3affd640 1 --2- 172.21.15.107:0/860219984 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda3c101120 0x7fda3c199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.665+0000 7fda3affd640 1 -- 172.21.15.107:0/860219984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda3c106720 msgr2=0x7fda3c1a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.665+0000 7fda3affd640 1 --2- 172.21.15.107:0/860219984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda3c106720 0x7fda3c1a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.665+0000 7fda3affd640 1 -- 172.21.15.107:0/860219984 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda3c1a4380 con 0x7fda3c100760 2024-05-09T08:38:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.665+0000 7fda3b7fe640 1 --2- 172.21.15.107:0/860219984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda3c106720 0x7fda3c1a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:38:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.665+0000 7fda3affd640 1 --2- 172.21.15.107:0/860219984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda3c100760 0x7fda3c198dc0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fda2c00c890 tx=0x7fda2c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.665+0000 7fda23fff640 1 -- 172.21.15.107:0/860219984 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2c004220 con 0x7fda3c100760 2024-05-09T08:38:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.665+0000 7fda41509640 1 -- 172.21.15.107:0/860219984 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda3c1a4550 con 0x7fda3c100760 2024-05-09T08:38:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.665+0000 7fda23fff640 1 -- 172.21.15.107:0/860219984 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fda2c0043c0 con 0x7fda3c100760 2024-05-09T08:38:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.665+0000 7fda23fff640 1 -- 172.21.15.107:0/860219984 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2c004a00 con 0x7fda3c100760 2024-05-09T08:38:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.665+0000 7fda41509640 1 -- 172.21.15.107:0/860219984 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda3c1a4ac0 con 0x7fda3c100760 2024-05-09T08:38:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.667+0000 7fda23fff640 1 -- 172.21.15.107:0/860219984 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fda2c004ba0 con 0x7fda3c100760 2024-05-09T08:38:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.668+0000 7fda23fff640 1 --2- 172.21.15.107:0/860219984 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fda240a2da0 0x7fda240a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.668+0000 7fda3a7fc640 1 --2- 172.21.15.107:0/860219984 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fda240a2da0 0x7fda240a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.668+0000 7fda23fff640 1 -- 172.21.15.107:0/860219984 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fda2c0cf440 con 0x7fda3c100760 2024-05-09T08:38:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.668+0000 7fda4420d740 1 Processor -- start 2024-05-09T08:38:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.668+0000 7fda3a7fc640 1 --2- 172.21.15.107:0/860219984 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fda240a2da0 0x7fda240a5260 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fda3c069a30 tx=0x7fda3000b390 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.668+0000 7fda4420d740 1 -- start start 2024-05-09T08:38:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda4420d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63450 0x556e87a63830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda4420d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a63d70 0x556e87a6e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda4420d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a6e630 0x556e87a70a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda4420d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556e879c8340 con 0x556e87a63450 2024-05-09T08:38:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda4420d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556e87970290 con 0x556e87a6e630 2024-05-09T08:38:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda4420d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x556e879c4300 con 0x556e87a63d70 2024-05-09T08:38:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda3affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63450 0x556e87a63830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda3affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63450 0x556e87a63830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33924/0 (socket says 172.21.15.107:33924) 2024-05-09T08:38:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda3affd640 1 -- 172.21.15.107:0/404679367 learned_addr learned my addr 172.21.15.107:0/404679367 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda3b7fe640 1 --2- 172.21.15.107:0/404679367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a6e630 0x556e87a70a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda3affd640 1 -- 172.21.15.107:0/404679367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a63d70 msgr2=0x556e87a6e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda3a7fc640 1 --2- 172.21.15.107:0/404679367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a63d70 0x556e87a6e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda3affd640 1 --2- 172.21.15.107:0/404679367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a63d70 0x556e87a6e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda3affd640 1 -- 172.21.15.107:0/404679367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a6e630 msgr2=0x556e87a70a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda3affd640 1 --2- 172.21.15.107:0/404679367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a6e630 0x556e87a70a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda3affd640 1 -- 172.21.15.107:0/404679367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e879948e0 con 0x556e87a63450 2024-05-09T08:38:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda3affd640 1 --2- 172.21.15.107:0/404679367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63450 0x556e87a63830 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fda2c0a3d40 tx=0x7fda2c09b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd742af135b2e92 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda217fa640 1 -- 172.21.15.107:0/404679367 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2c01d040 con 0x556e87a63450 2024-05-09T08:38:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda217fa640 1 -- 172.21.15.107:0/404679367 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fda2c09f9b0 con 0x556e87a63450 2024-05-09T08:38:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.669+0000 7fda217fa640 1 -- 172.21.15.107:0/404679367 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2c097040 con 0x556e87a63450 2024-05-09T08:38:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.670+0000 7fda4420d740 1 -- 172.21.15.107:0/404679367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63450 msgr2=0x556e87a63830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.670+0000 7fda4420d740 1 --2- 172.21.15.107:0/404679367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63450 0x556e87a63830 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fda2c0a3d40 tx=0x7fda2c09b260 comp rx=0 tx=0).stop 2024-05-09T08:38:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.670+0000 7fda4420d740 1 -- 172.21.15.107:0/404679367 shutdown_connections 2024-05-09T08:38:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.670+0000 7fda4420d740 1 --2- 172.21.15.107:0/404679367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a6e630 0x556e87a70a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.670+0000 7fda4420d740 1 --2- 172.21.15.107:0/404679367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a63d70 0x556e87a6e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.670+0000 7fda4420d740 1 --2- 172.21.15.107:0/404679367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63450 0x556e87a63830 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.670+0000 7fda4420d740 1 -- 172.21.15.107:0/404679367 >> 172.21.15.107:0/404679367 conn(0x556e87a5f960 msgr2=0x556e87a5fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.670+0000 7fda4420d740 1 -- 172.21.15.107:0/404679367 shutdown_connections 2024-05-09T08:38:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.670+0000 7fda4420d740 1 -- 172.21.15.107:0/404679367 wait complete. 2024-05-09T08:38:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda4420d740 1 -- start start 2024-05-09T08:38:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda4420d740 1 Processor -- start 2024-05-09T08:38:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda4420d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63450 0x556e87a6b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda4420d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a63d70 0x556e87a6bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda4420d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a6e630 0x556e87a6c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda4420d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556e87a71260 con 0x556e87a63450 2024-05-09T08:38:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda4420d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556e87983790 con 0x556e87a6e630 2024-05-09T08:38:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda4420d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x556e87a713e0 con 0x556e87a63d70 2024-05-09T08:38:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda4420d740 10 client.0 _ll_register_callbacks cb 0x556e879bea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda3affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63450 0x556e87a6b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda3affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63450 0x556e87a6b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33940/0 (socket says 172.21.15.107:33940) 2024-05-09T08:38:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda3affd640 1 -- 172.21.15.107:0/1524048715 learned_addr learned my addr 172.21.15.107:0/1524048715 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda3b7fe640 1 --2- 172.21.15.107:0/1524048715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a6e630 0x556e87a6c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda3a7fc640 1 --2- 172.21.15.107:0/1524048715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a63d70 0x556e87a6bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda3affd640 1 -- 172.21.15.107:0/1524048715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a63d70 msgr2=0x556e87a6bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda3affd640 1 --2- 172.21.15.107:0/1524048715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a63d70 0x556e87a6bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda3affd640 1 -- 172.21.15.107:0/1524048715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a6e630 msgr2=0x556e87a6c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda3affd640 1 --2- 172.21.15.107:0/1524048715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a6e630 0x556e87a6c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.671+0000 7fda3affd640 1 -- 172.21.15.107:0/1524048715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e876dc210 con 0x556e87a63450 2024-05-09T08:38:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.672+0000 7fda3affd640 1 --2- 172.21.15.107:0/1524048715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63450 0x556e87a6b640 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fda2c09b590 tx=0x7fda2c086060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c5b023404925d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.672+0000 7fda1eff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.672+0000 7fda1eff5640 1 -- 172.21.15.107:0/1524048715 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2c0b3070 con 0x556e87a63450 2024-05-09T08:38:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.672+0000 7fda1eff5640 1 -- 172.21.15.107:0/1524048715 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fda2c097c70 con 0x556e87a63450 2024-05-09T08:38:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.672+0000 7fda4420d740 1 -- 172.21.15.107:0/1524048715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda2c09f9b0 con 0x556e87a63450 2024-05-09T08:38:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.672+0000 7fda1eff5640 1 -- 172.21.15.107:0/1524048715 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fda2c01d040 con 0x556e87a63450 2024-05-09T08:38:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.672+0000 7fda4420d740 10 client.17043 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.672+0000 7fda4420d740 1 -- 172.21.15.107:0/1524048715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556e877a8620 con 0x556e87a63450 2024-05-09T08:38:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.672+0000 7fda1eff5640 1 -- 172.21.15.107:0/1524048715 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fda2c09f9b0 con 0x556e87a63450 2024-05-09T08:38:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.672+0000 7fda1eff5640 1 -- 172.21.15.107:0/1524048715 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fda2c0afb20 con 0x556e87a63450 2024-05-09T08:38:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.672+0000 7fda4420d740 10 client.17043 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.672+0000 7fda4420d740 10 client.17043 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.672+0000 7fda4420d740 20 client.17043 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.672+0000 7fda4420d740 1 --2- 172.21.15.107:0/1524048715 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556e87a7e290 0x556e87a80680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.672+0000 7fda4420d740 4 client.17043 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.672+0000 7fda4420d740 1 -- 172.21.15.107:0/1524048715 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e879d5a70 con 0x556e87a7e290 2024-05-09T08:38:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.673+0000 7fda3a7fc640 1 --2- 172.21.15.107:0/1524048715 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556e87a7e290 0x556e87a80680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.673+0000 7fda1eff5640 10 client.17043 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.673+0000 7fda3a7fc640 1 --2- 172.21.15.107:0/1524048715 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556e87a7e290 0x556e87a80680 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ccecfc02819fb89 server_cookie=21c1b467e14fc257 in_seq=0 out_seq=0 2024-05-09T08:38:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.674+0000 7fda1eff5640 1 -- 172.21.15.107:0/1524048715 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556e879d5a70 con 0x556e87a7e290 2024-05-09T08:38:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.674+0000 7fda1eff5640 10 client.17043 handle_command_reply: tid=0 2024-05-09T08:38:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.674+0000 7fda4420d740 1 client.17043 shutdown 2024-05-09T08:38:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.674+0000 7fda4420d740 1 -- 172.21.15.107:0/1524048715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63450 msgr2=0x556e87a6b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.674+0000 7fda4420d740 1 --2- 172.21.15.107:0/1524048715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63450 0x556e87a6b640 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fda2c09b590 tx=0x7fda2c086060 comp rx=0 tx=0).stop 2024-05-09T08:38:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.674+0000 7fda4420d740 1 -- 172.21.15.107:0/1524048715 shutdown_connections 2024-05-09T08:38:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.674+0000 7fda4420d740 1 --2- 172.21.15.107:0/1524048715 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556e87a7e290 0x556e87a80680 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.674+0000 7fda4420d740 1 --2- 172.21.15.107:0/1524048715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a6e630 0x556e87a6c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.674+0000 7fda1eff5640 0 client.17043 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.674+0000 7fda4420d740 1 --2- 172.21.15.107:0/1524048715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a63d70 0x556e87a6bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.674+0000 7fda4420d740 1 --2- 172.21.15.107:0/1524048715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63450 0x556e87a6b640 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.674+0000 7fda4420d740 1 -- 172.21.15.107:0/1524048715 >> 172.21.15.107:0/1524048715 conn(0x556e87a5f960 msgr2=0x556e87a65690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.674+0000 7fda4420d740 1 -- 172.21.15.107:0/1524048715 shutdown_connections 2024-05-09T08:38:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.674+0000 7fda4420d740 1 -- 172.21.15.107:0/1524048715 wait complete. 2024-05-09T08:38:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.674+0000 7fda4420d740 20 client.17043 trim_cache size 0 max 16384 2024-05-09T08:38:06.692 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda4420d740 1 Processor -- start 2024-05-09T08:38:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda4420d740 1 -- start start 2024-05-09T08:38:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda4420d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a63450 0x556e87a6dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda4420d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63d70 0x556e87a7a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda4420d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a6e630 0x556e87a5e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda4420d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c1b13b0 con 0x556e87a63d70 2024-05-09T08:38:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda4420d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c1aec80 con 0x556e87a63450 2024-05-09T08:38:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda4420d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c1acad0 con 0x556e87a6e630 2024-05-09T08:38:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda3a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63d70 0x556e87a7a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda3a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63d70 0x556e87a7a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33946/0 (socket says 172.21.15.107:33946) 2024-05-09T08:38:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda3a7fc640 1 -- 172.21.15.107:0/1707821839 learned_addr learned my addr 172.21.15.107:0/1707821839 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda3affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a63450 0x556e87a6dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda3b7fe640 1 --2- 172.21.15.107:0/1707821839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a6e630 0x556e87a5e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda3a7fc640 1 -- 172.21.15.107:0/1707821839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a6e630 msgr2=0x556e87a5e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda3a7fc640 1 --2- 172.21.15.107:0/1707821839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a6e630 0x556e87a5e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda3a7fc640 1 -- 172.21.15.107:0/1707821839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a63450 msgr2=0x556e87a6dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda3a7fc640 1 --2- 172.21.15.107:0/1707821839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a63450 0x556e87a6dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.693+0000 7fda3a7fc640 1 -- 172.21.15.107:0/1707821839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e87a5df00 con 0x556e87a63d70 2024-05-09T08:38:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.694+0000 7fda3a7fc640 1 --2- 172.21.15.107:0/1707821839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63d70 0x556e87a7a120 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x556e87a6aea0 tx=0x7fda30013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bfc177c96e5a97 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.694+0000 7fda20ff9640 1 -- 172.21.15.107:0/1707821839 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda30011da0 con 0x556e87a63d70 2024-05-09T08:38:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.694+0000 7fda20ff9640 1 -- 172.21.15.107:0/1707821839 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fda30005020 con 0x556e87a63d70 2024-05-09T08:38:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.694+0000 7fda20ff9640 1 -- 172.21.15.107:0/1707821839 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda300276d0 con 0x556e87a63d70 2024-05-09T08:38:06.697 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 -- 172.21.15.107:0/1707821839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63d70 msgr2=0x556e87a7a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:06.697 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 --2- 172.21.15.107:0/1707821839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63d70 0x556e87a7a120 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x556e87a6aea0 tx=0x7fda30013000 comp rx=0 tx=0).stop 2024-05-09T08:38:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 -- 172.21.15.107:0/1707821839 shutdown_connections 2024-05-09T08:38:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 --2- 172.21.15.107:0/1707821839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a6e630 0x556e87a5e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 --2- 172.21.15.107:0/1707821839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63d70 0x556e87a7a120 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 --2- 172.21.15.107:0/1707821839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a63450 0x556e87a6dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 -- 172.21.15.107:0/1707821839 >> 172.21.15.107:0/1707821839 conn(0x556e87a5f960 msgr2=0x556e87a5e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 -- 172.21.15.107:0/1707821839 shutdown_connections 2024-05-09T08:38:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 -- 172.21.15.107:0/1707821839 wait complete. 2024-05-09T08:38:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 -- start start 2024-05-09T08:38:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 Processor -- start 2024-05-09T08:38:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a63450 0x556e87a68a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63d70 0x556e87a68f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a6e630 0x556e87ae7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c1ad140 con 0x556e87a63d70 2024-05-09T08:38:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c1ae5b0 con 0x556e87a63450 2024-05-09T08:38:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c1ad810 con 0x556e87a6e630 2024-05-09T08:38:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda4420d740 10 client.0 _ll_register_callbacks cb 0x7fda2c09f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.695+0000 7fda3a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63d70 0x556e87a68f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda3a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63d70 0x556e87a68f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33960/0 (socket says 172.21.15.107:33960) 2024-05-09T08:38:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda3a7fc640 1 -- 172.21.15.107:0/233777795 learned_addr learned my addr 172.21.15.107:0/233777795 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda3b7fe640 1 --2- 172.21.15.107:0/233777795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a6e630 0x556e87ae7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda3affd640 1 --2- 172.21.15.107:0/233777795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a63450 0x556e87a68a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda3a7fc640 1 -- 172.21.15.107:0/233777795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a6e630 msgr2=0x556e87ae7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda3a7fc640 1 --2- 172.21.15.107:0/233777795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a6e630 0x556e87ae7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda3a7fc640 1 -- 172.21.15.107:0/233777795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a63450 msgr2=0x556e87a68a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda3a7fc640 1 --2- 172.21.15.107:0/233777795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a63450 0x556e87a68a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda3a7fc640 1 -- 172.21.15.107:0/233777795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e87ae7e70 con 0x556e87a63d70 2024-05-09T08:38:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda3a7fc640 1 --2- 172.21.15.107:0/233777795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63d70 0x556e87a68f70 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fda30010a10 tx=0x7fda30011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a196db52ebb8f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda1eff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda1eff5640 1 -- 172.21.15.107:0/233777795 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda30034020 con 0x556e87a63d70 2024-05-09T08:38:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda1eff5640 1 -- 172.21.15.107:0/233777795 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fda300053e0 con 0x556e87a63d70 2024-05-09T08:38:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda4420d740 1 -- 172.21.15.107:0/233777795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda30005020 con 0x556e87a63d70 2024-05-09T08:38:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda1eff5640 1 -- 172.21.15.107:0/233777795 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fda3003f050 con 0x556e87a63d70 2024-05-09T08:38:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda4420d740 10 client.17049 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.696+0000 7fda4420d740 1 -- 172.21.15.107:0/233777795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556e87ae9130 con 0x556e87a63d70 2024-05-09T08:38:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.697+0000 7fda1eff5640 1 -- 172.21.15.107:0/233777795 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fda30005020 con 0x556e87a63d70 2024-05-09T08:38:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.697+0000 7fda1eff5640 1 -- 172.21.15.107:0/233777795 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fda30027450 con 0x556e87a63d70 2024-05-09T08:38:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.697+0000 7fda4420d740 10 client.17049 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.697+0000 7fda4420d740 10 client.17049 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.697+0000 7fda4420d740 20 client.17049 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.697+0000 7fda4420d740 1 --2- 172.21.15.107:0/233777795 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556e87ae9930 0x556e87aebd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.697+0000 7fda4420d740 4 client.17049 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:38:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.697+0000 7fda4420d740 1 -- 172.21.15.107:0/233777795 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556e879cfdc0 con 0x556e87ae9930 2024-05-09T08:38:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.697+0000 7fda3affd640 1 --2- 172.21.15.107:0/233777795 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556e87ae9930 0x556e87aebd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.698+0000 7fda3affd640 1 --2- 172.21.15.107:0/233777795 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556e87ae9930 0x556e87aebd50 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=522d7e1d08b129a server_cookie=29d5a6846bf9e65 in_seq=0 out_seq=0 2024-05-09T08:38:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.698+0000 7fda1eff5640 10 client.17049 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.713+0000 7fda1eff5640 1 -- 172.21.15.107:0/233777795 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556e879cfdc0 con 0x556e87ae9930 2024-05-09T08:38:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.713+0000 7fda1eff5640 10 client.17049 handle_command_reply: tid=0 2024-05-09T08:38:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.713+0000 7fda4420d740 1 client.17049 shutdown 2024-05-09T08:38:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.714+0000 7fda4420d740 1 -- 172.21.15.107:0/233777795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63d70 msgr2=0x556e87a68f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.714+0000 7fda4420d740 1 --2- 172.21.15.107:0/233777795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63d70 0x556e87a68f70 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fda30010a10 tx=0x7fda30011ba0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.714+0000 7fda4420d740 1 -- 172.21.15.107:0/233777795 shutdown_connections 2024-05-09T08:38:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.714+0000 7fda4420d740 1 --2- 172.21.15.107:0/233777795 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556e87ae9930 0x556e87aebd50 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.714+0000 7fda4420d740 1 --2- 172.21.15.107:0/233777795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556e87a6e630 0x556e87ae7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.714+0000 7fda1eff5640 0 client.17049 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.714+0000 7fda4420d740 1 --2- 172.21.15.107:0/233777795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556e87a63d70 0x556e87a68f70 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.714+0000 7fda4420d740 1 --2- 172.21.15.107:0/233777795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556e87a63450 0x556e87a68a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.714+0000 7fda4420d740 1 -- 172.21.15.107:0/233777795 >> 172.21.15.107:0/233777795 conn(0x556e87a5f960 msgr2=0x556e87a5ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.714+0000 7fda4420d740 1 -- 172.21.15.107:0/233777795 shutdown_connections 2024-05-09T08:38:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.714+0000 7fda4420d740 1 -- 172.21.15.107:0/233777795 wait complete. 2024-05-09T08:38:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.714+0000 7fda4420d740 20 client.17049 trim_cache size 0 max 16384 2024-05-09T08:38:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:38:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:38:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:38:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:38:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.715+0000 7fda41509640 1 -- 172.21.15.107:0/860219984 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fda240a2da0 msgr2=0x7fda240a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.715+0000 7fda41509640 1 --2- 172.21.15.107:0/860219984 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fda240a2da0 0x7fda240a5260 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fda3c069a30 tx=0x7fda3000b390 comp rx=0 tx=0).stop 2024-05-09T08:38:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.715+0000 7fda41509640 1 -- 172.21.15.107:0/860219984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda3c100760 msgr2=0x7fda3c198dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.715+0000 7fda41509640 1 --2- 172.21.15.107:0/860219984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda3c100760 0x7fda3c198dc0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fda2c00c890 tx=0x7fda2c00cd50 comp rx=0 tx=0).stop 2024-05-09T08:38:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.715+0000 7fda41509640 1 -- 172.21.15.107:0/860219984 shutdown_connections 2024-05-09T08:38:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.715+0000 7fda41509640 1 --2- 172.21.15.107:0/860219984 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fda240a2da0 0x7fda240a5260 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.715+0000 7fda41509640 1 --2- 172.21.15.107:0/860219984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda3c106720 0x7fda3c1a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.715+0000 7fda41509640 1 --2- 172.21.15.107:0/860219984 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda3c101120 0x7fda3c199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.715+0000 7fda41509640 1 --2- 172.21.15.107:0/860219984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda3c100760 0x7fda3c198dc0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.715+0000 7fda41509640 1 -- 172.21.15.107:0/860219984 >> 172.21.15.107:0/860219984 conn(0x7fda3c0778b0 msgr2=0x7fda3c1032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.715+0000 7fda41509640 1 -- 172.21.15.107:0/860219984 shutdown_connections 2024-05-09T08:38:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:06.715+0000 7fda41509640 1 -- 172.21.15.107:0/860219984 wait complete. 2024-05-09T08:38:06.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:06 smithi107 ceph-mon[30122]: pgmap v145: 129 pgs: 129 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 25 MiB/s wr, 34 op/s 2024-05-09T08:38:08.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:08 smithi157 ceph-mon[33614]: pgmap v146: 129 pgs: 129 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 37 MiB/s wr, 47 op/s 2024-05-09T08:38:08.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:08 smithi110 ceph-mon[33622]: pgmap v146: 129 pgs: 129 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 37 MiB/s wr, 47 op/s 2024-05-09T08:38:08.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:08 smithi107 ceph-mon[30122]: pgmap v146: 129 pgs: 129 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 37 MiB/s wr, 47 op/s 2024-05-09T08:38:10.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:10 smithi157 ceph-mon[33614]: pgmap v147: 129 pgs: 129 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 25 MiB/s wr, 35 op/s 2024-05-09T08:38:10.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:10 smithi110 ceph-mon[33622]: pgmap v147: 129 pgs: 129 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 25 MiB/s wr, 35 op/s 2024-05-09T08:38:10.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:10 smithi107 ceph-mon[30122]: pgmap v147: 129 pgs: 129 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 25 MiB/s wr, 35 op/s 2024-05-09T08:38:12.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:12 smithi157 ceph-mon[33614]: pgmap v148: 129 pgs: 129 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 25 MiB/s wr, 35 op/s 2024-05-09T08:38:12.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:12 smithi110 ceph-mon[33622]: pgmap v148: 129 pgs: 129 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 25 MiB/s wr, 35 op/s 2024-05-09T08:38:12.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:12 smithi107 ceph-mon[30122]: pgmap v148: 129 pgs: 129 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 25 MiB/s wr, 35 op/s 2024-05-09T08:38:14.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:14 smithi157 ceph-mon[33614]: pgmap v149: 129 pgs: 129 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 37 MiB/s wr, 43 op/s 2024-05-09T08:38:14.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:14 smithi110 ceph-mon[33622]: pgmap v149: 129 pgs: 129 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 37 MiB/s wr, 43 op/s 2024-05-09T08:38:14.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:14 smithi107 ceph-mon[30122]: pgmap v149: 129 pgs: 129 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 37 MiB/s wr, 43 op/s 2024-05-09T08:38:16.580 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:16 smithi107 ceph-mon[30122]: pgmap v150: 129 pgs: 129 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:38:16.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:16 smithi157 ceph-mon[33614]: pgmap v150: 129 pgs: 129 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:38:16.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:16 smithi110 ceph-mon[33622]: pgmap v150: 129 pgs: 129 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:38:18.159 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:38:18.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:38:18.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.279+0000 7fe4cf789640 1 -- 172.21.15.107:0/2323541586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c8100700 msgr2=0x7fe4c8100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.279+0000 7fe4cf789640 1 --2- 172.21.15.107:0/2323541586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c8100700 0x7fe4c8100b80 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009930 tx=0x7fe4b802a420 comp rx=0 tx=0).stop 2024-05-09T08:38:18.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.279+0000 7fe4cf789640 1 -- 172.21.15.107:0/2323541586 shutdown_connections 2024-05-09T08:38:18.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.279+0000 7fe4cf789640 1 --2- 172.21.15.107:0/2323541586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4c81010c0 0x7fe4c810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.279+0000 7fe4cf789640 1 --2- 172.21.15.107:0/2323541586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c8100700 0x7fe4c8100b80 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.279+0000 7fe4cf789640 1 --2- 172.21.15.107:0/2323541586 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4c8106710 0x7fe4c8106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.279+0000 7fe4cf789640 1 -- 172.21.15.107:0/2323541586 >> 172.21.15.107:0/2323541586 conn(0x7fe4c8077830 msgr2=0x7fe4c8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.279+0000 7fe4cf789640 1 -- 172.21.15.107:0/2323541586 shutdown_connections 2024-05-09T08:38:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.279+0000 7fe4cf789640 1 -- 172.21.15.107:0/2323541586 wait complete. 2024-05-09T08:38:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cf789640 1 Processor -- start 2024-05-09T08:38:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cf789640 1 -- start start 2024-05-09T08:38:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cf789640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4c8100700 0x7fe4c81980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cf789640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4c81010c0 0x7fe4c8198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cf789640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c8106710 0x7fe4c819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cf789640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c81109d0 con 0x7fe4c8106710 2024-05-09T08:38:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cf789640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c8110850 con 0x7fe4c81010c0 2024-05-09T08:38:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cf789640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c8110b50 con 0x7fe4c8100700 2024-05-09T08:38:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cdd26640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c8106710 0x7fe4c819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4ccd24640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4c81010c0 0x7fe4c8198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cdd26640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c8106710 0x7fe4c819c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46442/0 (socket says 172.21.15.107:46442) 2024-05-09T08:38:18.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cdd26640 1 -- 172.21.15.107:0/3127321470 learned_addr learned my addr 172.21.15.107:0/3127321470 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cd525640 1 --2- 172.21.15.107:0/3127321470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4c8100700 0x7fe4c81980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cdd26640 1 -- 172.21.15.107:0/3127321470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4c8100700 msgr2=0x7fe4c81980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cdd26640 1 --2- 172.21.15.107:0/3127321470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4c8100700 0x7fe4c81980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cdd26640 1 -- 172.21.15.107:0/3127321470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4c81010c0 msgr2=0x7fe4c8198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cdd26640 1 --2- 172.21.15.107:0/3127321470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4c81010c0 0x7fe4c8198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4cdd26640 1 -- 172.21.15.107:0/3127321470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c819d080 con 0x7fe4c8106710 2024-05-09T08:38:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.280+0000 7fe4ccd24640 1 --2- 172.21.15.107:0/3127321470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4c81010c0 0x7fe4c8198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.281+0000 7fe4cdd26640 1 --2- 172.21.15.107:0/3127321470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c8106710 0x7fe4c819c950 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000eb10 tx=0x7fe4c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.281+0000 7fe4be7fc640 1 -- 172.21.15.107:0/3127321470 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0004b20 con 0x7fe4c8106710 2024-05-09T08:38:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.281+0000 7fe4cf789640 1 -- 172.21.15.107:0/3127321470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c819d370 con 0x7fe4c8106710 2024-05-09T08:38:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.281+0000 7fe4be7fc640 1 -- 172.21.15.107:0/3127321470 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4c0004cc0 con 0x7fe4c8106710 2024-05-09T08:38:18.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.281+0000 7fe4be7fc640 1 -- 172.21.15.107:0/3127321470 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0010690 con 0x7fe4c8106710 2024-05-09T08:38:18.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.281+0000 7fe4cf789640 1 -- 172.21.15.107:0/3127321470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c81a4c00 con 0x7fe4c8106710 2024-05-09T08:38:18.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.284+0000 7fe4d2571740 1 Processor -- start 2024-05-09T08:38:18.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.284+0000 7fe4be7fc640 1 -- 172.21.15.107:0/3127321470 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe4c0005020 con 0x7fe4c8106710 2024-05-09T08:38:18.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.284+0000 7fe4d2571740 1 -- start start 2024-05-09T08:38:18.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.284+0000 7fe4d2571740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16160 0x5607b7a16540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.284+0000 7fe4d2571740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16a80 0x5607b7a25140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.284+0000 7fe4d2571740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a27a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.284+0000 7fe4d2571740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5607b797bd00 con 0x5607b7a16160 2024-05-09T08:38:18.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.284+0000 7fe4d2571740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5607b79234c0 con 0x5607b7a16a80 2024-05-09T08:38:18.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.284+0000 7fe4d2571740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5607b7977cc0 con 0x5607b7a25680 2024-05-09T08:38:18.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.284+0000 7fe4ccd24640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16160 0x5607b7a16540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.284+0000 7fe4ccd24640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16160 0x5607b7a16540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46456/0 (socket says 172.21.15.107:46456) 2024-05-09T08:38:18.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.284+0000 7fe4cd525640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16a80 0x5607b7a25140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.284+0000 7fe4ccd24640 1 -- 172.21.15.107:0/22841121 learned_addr learned my addr 172.21.15.107:0/22841121 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:18.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.284+0000 7fe4be7fc640 1 --2- 172.21.15.107:0/3127321470 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4b40a2d00 0x7fe4b40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.284+0000 7fe4ccd24640 1 --2- 172.21.15.107:0/22841121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a27a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.284+0000 7fe4cd525640 1 -- 172.21.15.107:0/22841121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 msgr2=0x5607b7a27a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:18.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.285+0000 7fe4cd525640 1 --2- 172.21.15.107:0/22841121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a27a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.285+0000 7fe4cd525640 1 -- 172.21.15.107:0/22841121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16160 msgr2=0x5607b7a16540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:18.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.285+0000 7fe4cd525640 1 --2- 172.21.15.107:0/22841121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16160 0x5607b7a16540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.285+0000 7fe4cd525640 1 -- 172.21.15.107:0/22841121 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607b77597d0 con 0x5607b7a16a80 2024-05-09T08:38:18.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.285+0000 7fe4ccd24640 1 --2- 172.21.15.107:0/22841121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a27a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:38:18.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.285+0000 7fe4be7fc640 1 -- 172.21.15.107:0/3127321470 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe4c00cfdf0 con 0x7fe4c8106710 2024-05-09T08:38:18.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.285+0000 7fe4cdd26640 1 --2- 172.21.15.107:0/3127321470 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4b40a2d00 0x7fe4b40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.285+0000 7fe4cd525640 1 --2- 172.21.15.107:0/22841121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16a80 0x5607b7a25140 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4009af0 tx=0x7fe4c40113e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9615c62c86edd7f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:18.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.285+0000 7fe4b37fe640 1 -- 172.21.15.107:0/22841121 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4011e10 con 0x5607b7a16a80 2024-05-09T08:38:18.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.285+0000 7fe4b37fe640 1 -- 172.21.15.107:0/22841121 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4c4012900 con 0x5607b7a16a80 2024-05-09T08:38:18.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.285+0000 7fe4b37fe640 1 -- 172.21.15.107:0/22841121 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c400f970 con 0x5607b7a16a80 2024-05-09T08:38:18.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.285+0000 7fe4cdd26640 1 --2- 172.21.15.107:0/3127321470 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4b40a2d00 0x7fe4b40a51c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe4c00cd290 tx=0x7fe4c000efd0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:18.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.286+0000 7fe4d2571740 1 -- 172.21.15.107:0/22841121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16a80 msgr2=0x5607b7a25140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.286+0000 7fe4d2571740 1 --2- 172.21.15.107:0/22841121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16a80 0x5607b7a25140 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4009af0 tx=0x7fe4c40113e0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.286+0000 7fe4d2571740 1 -- 172.21.15.107:0/22841121 shutdown_connections 2024-05-09T08:38:18.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.286+0000 7fe4d2571740 1 --2- 172.21.15.107:0/22841121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a27a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.286+0000 7fe4d2571740 1 --2- 172.21.15.107:0/22841121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16a80 0x5607b7a25140 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.286+0000 7fe4d2571740 1 --2- 172.21.15.107:0/22841121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16160 0x5607b7a16540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.286+0000 7fe4d2571740 1 -- 172.21.15.107:0/22841121 >> 172.21.15.107:0/22841121 conn(0x5607b7a125a0 msgr2=0x5607b7a12980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:18.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.286+0000 7fe4d2571740 1 -- 172.21.15.107:0/22841121 shutdown_connections 2024-05-09T08:38:18.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.286+0000 7fe4d2571740 1 -- 172.21.15.107:0/22841121 wait complete. 2024-05-09T08:38:18.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.286+0000 7fe4d2571740 1 -- start start 2024-05-09T08:38:18.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.286+0000 7fe4d2571740 1 Processor -- start 2024-05-09T08:38:18.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4d2571740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16160 0x5607b7a22560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4d2571740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16a80 0x5607b7a22aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4d2571740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a22fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4d2571740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5607b7a284c0 con 0x5607b7a16160 2024-05-09T08:38:18.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4d2571740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5607b7936b90 con 0x5607b7a16a80 2024-05-09T08:38:18.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4d2571740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5607b7a28640 con 0x5607b7a25680 2024-05-09T08:38:18.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4d2571740 10 client.0 _ll_register_callbacks cb 0x5607b7972da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:18.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4ccd24640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16160 0x5607b7a22560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4ccd24640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16160 0x5607b7a22560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46460/0 (socket says 172.21.15.107:46460) 2024-05-09T08:38:18.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4ccd24640 1 -- 172.21.15.107:0/30121110 learned_addr learned my addr 172.21.15.107:0/30121110 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:18.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4cd525640 1 --2- 172.21.15.107:0/30121110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16a80 0x5607b7a22aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4ccd24640 1 --2- 172.21.15.107:0/30121110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a22fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4ccd24640 1 -- 172.21.15.107:0/30121110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 msgr2=0x5607b7a22fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:18.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4ccd24640 1 --2- 172.21.15.107:0/30121110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a22fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4ccd24640 1 -- 172.21.15.107:0/30121110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16a80 msgr2=0x5607b7a22aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:18.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4ccd24640 1 --2- 172.21.15.107:0/30121110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16a80 0x5607b7a22aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4ccd24640 1 -- 172.21.15.107:0/30121110 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607b768f3f0 con 0x5607b7a16160 2024-05-09T08:38:18.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4ccd24640 1 --2- 172.21.15.107:0/30121110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16160 0x5607b7a22560 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009900 tx=0x7fe4b8040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d723e9d59120811 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.287+0000 7fe4b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.288+0000 7fe4b0ff9640 1 -- 172.21.15.107:0/30121110 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8035040 con 0x5607b7a16160 2024-05-09T08:38:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.288+0000 7fe4d2571740 1 -- 172.21.15.107:0/30121110 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4c4012900 con 0x5607b7a16160 2024-05-09T08:38:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.288+0000 7fe4b0ff9640 1 -- 172.21.15.107:0/30121110 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe4b8004050 con 0x5607b7a16160 2024-05-09T08:38:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.288+0000 7fe4b0ff9640 1 -- 172.21.15.107:0/30121110 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe4b8051050 con 0x5607b7a16160 2024-05-09T08:38:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.288+0000 7fe4d2571740 10 client.17061 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.288+0000 7fe4d2571740 1 -- 172.21.15.107:0/30121110 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5607b764f0f0 con 0x5607b7a16160 2024-05-09T08:38:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.288+0000 7fe4b0ff9640 1 -- 172.21.15.107:0/30121110 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4c4012900 con 0x5607b7a16160 2024-05-09T08:38:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.288+0000 7fe4b0ff9640 1 -- 172.21.15.107:0/30121110 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe4b80035f0 con 0x5607b7a16160 2024-05-09T08:38:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.288+0000 7fe4d2571740 10 client.17061 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.288+0000 7fe4d2571740 10 client.17061 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.288+0000 7fe4d2571740 20 client.17061 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:18.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.288+0000 7fe4d2571740 1 --2- 172.21.15.107:0/30121110 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607b7a35970 0x5607b7a37d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.288+0000 7fe4d2571740 4 client.17061 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.288+0000 7fe4d2571740 1 -- 172.21.15.107:0/30121110 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607b7989430 con 0x5607b7a35970 2024-05-09T08:38:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.289+0000 7fe4cd525640 1 --2- 172.21.15.107:0/30121110 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607b7a35970 0x5607b7a37d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.289+0000 7fe4cd525640 1 --2- 172.21.15.107:0/30121110 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607b7a35970 0x5607b7a37d60 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa4a8ce67faa8100 server_cookie=e0edb88daa2b7a42 in_seq=0 out_seq=0 2024-05-09T08:38:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.289+0000 7fe4b0ff9640 10 client.17061 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.290+0000 7fe4b0ff9640 1 -- 172.21.15.107:0/30121110 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5607b7989430 con 0x5607b7a35970 2024-05-09T08:38:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.290+0000 7fe4b0ff9640 10 client.17061 handle_command_reply: tid=0 2024-05-09T08:38:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.290+0000 7fe4d2571740 1 client.17061 shutdown 2024-05-09T08:38:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.290+0000 7fe4d2571740 1 -- 172.21.15.107:0/30121110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16160 msgr2=0x5607b7a22560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.290+0000 7fe4d2571740 1 --2- 172.21.15.107:0/30121110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16160 0x5607b7a22560 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009900 tx=0x7fe4b8040040 comp rx=0 tx=0).stop 2024-05-09T08:38:18.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.290+0000 7fe4d2571740 1 -- 172.21.15.107:0/30121110 shutdown_connections 2024-05-09T08:38:18.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.290+0000 7fe4d2571740 1 --2- 172.21.15.107:0/30121110 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607b7a35970 0x5607b7a37d60 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.290+0000 7fe4d2571740 1 --2- 172.21.15.107:0/30121110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a22fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.290+0000 7fe4d2571740 1 --2- 172.21.15.107:0/30121110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16a80 0x5607b7a22aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.290+0000 7fe4b0ff9640 0 client.17061 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.290+0000 7fe4d2571740 1 --2- 172.21.15.107:0/30121110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16160 0x5607b7a22560 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.290+0000 7fe4d2571740 1 -- 172.21.15.107:0/30121110 >> 172.21.15.107:0/30121110 conn(0x5607b7a125a0 msgr2=0x5607b7a18350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.290+0000 7fe4d2571740 1 -- 172.21.15.107:0/30121110 shutdown_connections 2024-05-09T08:38:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.290+0000 7fe4d2571740 1 -- 172.21.15.107:0/30121110 wait complete. 2024-05-09T08:38:18.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.290+0000 7fe4d2571740 20 client.17061 trim_cache size 0 max 16384 2024-05-09T08:38:18.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4d2571740 1 Processor -- start 2024-05-09T08:38:18.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4d2571740 1 -- start start 2024-05-09T08:38:18.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4d2571740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16160 0x5607b7a31450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4d2571740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16a80 0x5607b7a3d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4d2571740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a3db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4d2571740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c81af0e0 con 0x5607b7a16a80 2024-05-09T08:38:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4d2571740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c81ac9b0 con 0x5607b7a16160 2024-05-09T08:38:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4d2571740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c81aa800 con 0x5607b7a25680 2024-05-09T08:38:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4cd525640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16a80 0x5607b7a3d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4cd525640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16a80 0x5607b7a3d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46468/0 (socket says 172.21.15.107:46468) 2024-05-09T08:38:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4cd525640 1 -- 172.21.15.107:0/1366903369 learned_addr learned my addr 172.21.15.107:0/1366903369 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4ccd24640 1 --2- 172.21.15.107:0/1366903369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16160 0x5607b7a31450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4ccd24640 1 --2- 172.21.15.107:0/1366903369 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a3db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4cd525640 1 -- 172.21.15.107:0/1366903369 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 msgr2=0x5607b7a3db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4cd525640 1 --2- 172.21.15.107:0/1366903369 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a3db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4cd525640 1 -- 172.21.15.107:0/1366903369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16160 msgr2=0x5607b7a31450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4cd525640 1 --2- 172.21.15.107:0/1366903369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16160 0x5607b7a31450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.311+0000 7fe4cd525640 1 -- 172.21.15.107:0/1366903369 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607b7a38390 con 0x5607b7a16a80 2024-05-09T08:38:18.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.312+0000 7fe4ccd24640 1 --2- 172.21.15.107:0/1366903369 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a3db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:38:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.312+0000 7fe4cd525640 1 --2- 172.21.15.107:0/1366903369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16a80 0x5607b7a3d5c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe4c400f590 tx=0x7fe4c4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a94c67dd0953def server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.312+0000 7fe4b2ffd640 1 -- 172.21.15.107:0/1366903369 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c40052b0 con 0x5607b7a16a80 2024-05-09T08:38:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.312+0000 7fe4b2ffd640 1 -- 172.21.15.107:0/1366903369 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4c4005450 con 0x5607b7a16a80 2024-05-09T08:38:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.312+0000 7fe4b2ffd640 1 -- 172.21.15.107:0/1366903369 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c40035f0 con 0x5607b7a16a80 2024-05-09T08:38:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.313+0000 7fe4d2571740 1 -- 172.21.15.107:0/1366903369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16a80 msgr2=0x5607b7a3d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.313+0000 7fe4d2571740 1 --2- 172.21.15.107:0/1366903369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16a80 0x5607b7a3d5c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe4c400f590 tx=0x7fe4c4005f90 comp rx=0 tx=0).stop 2024-05-09T08:38:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.313+0000 7fe4d2571740 1 -- 172.21.15.107:0/1366903369 shutdown_connections 2024-05-09T08:38:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.313+0000 7fe4d2571740 1 --2- 172.21.15.107:0/1366903369 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a3db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.313+0000 7fe4d2571740 1 --2- 172.21.15.107:0/1366903369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16a80 0x5607b7a3d5c0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.313+0000 7fe4d2571740 1 --2- 172.21.15.107:0/1366903369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16160 0x5607b7a31450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.313+0000 7fe4d2571740 1 -- 172.21.15.107:0/1366903369 >> 172.21.15.107:0/1366903369 conn(0x5607b7a125a0 msgr2=0x5607b7a216c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.313+0000 7fe4d2571740 1 -- 172.21.15.107:0/1366903369 shutdown_connections 2024-05-09T08:38:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.313+0000 7fe4d2571740 1 -- 172.21.15.107:0/1366903369 wait complete. 2024-05-09T08:38:18.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.313+0000 7fe4d2571740 1 -- start start 2024-05-09T08:38:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.313+0000 7fe4d2571740 1 Processor -- start 2024-05-09T08:38:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.313+0000 7fe4d2571740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16160 0x5607b7a2db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4d2571740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16a80 0x5607b7a1edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4d2571740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a1f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4d2571740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c81aae70 con 0x5607b7a16a80 2024-05-09T08:38:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4d2571740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c81ac2e0 con 0x5607b7a16160 2024-05-09T08:38:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4d2571740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c81ab540 con 0x5607b7a25680 2024-05-09T08:38:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4d2571740 10 client.0 _ll_register_callbacks cb 0x7fe4b8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4cd525640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16a80 0x5607b7a1edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4cd525640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16a80 0x5607b7a1edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46472/0 (socket says 172.21.15.107:46472) 2024-05-09T08:38:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4cd525640 1 -- 172.21.15.107:0/2118853766 learned_addr learned my addr 172.21.15.107:0/2118853766 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4ccd24640 1 --2- 172.21.15.107:0/2118853766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16160 0x5607b7a2db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4ccd24640 1 --2- 172.21.15.107:0/2118853766 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a1f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4cd525640 1 -- 172.21.15.107:0/2118853766 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 msgr2=0x5607b7a1f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:18.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4cd525640 1 --2- 172.21.15.107:0/2118853766 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a1f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4cd525640 1 -- 172.21.15.107:0/2118853766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16160 msgr2=0x5607b7a2db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:18.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4cd525640 1 --2- 172.21.15.107:0/2118853766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16160 0x5607b7a2db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4cd525640 1 -- 172.21.15.107:0/2118853766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607b7a1f870 con 0x5607b7a16a80 2024-05-09T08:38:18.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4ccd24640 1 --2- 172.21.15.107:0/2118853766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16160 0x5607b7a2db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:18.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4cd525640 1 --2- 172.21.15.107:0/2118853766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16a80 0x5607b7a1edf0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4027040 tx=0x7fe4c4005600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68957d98ba7d0b1a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:18.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:18.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4b0ff9640 1 -- 172.21.15.107:0/2118853766 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c400f5f0 con 0x5607b7a16a80 2024-05-09T08:38:18.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4b0ff9640 1 -- 172.21.15.107:0/2118853766 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe4c400f790 con 0x5607b7a16a80 2024-05-09T08:38:18.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4d2571740 1 -- 172.21.15.107:0/2118853766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4c4005450 con 0x5607b7a16a80 2024-05-09T08:38:18.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4b0ff9640 1 -- 172.21.15.107:0/2118853766 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe4c400f9b0 con 0x5607b7a16a80 2024-05-09T08:38:18.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.314+0000 7fe4d2571740 10 client.17067 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:18.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.315+0000 7fe4d2571740 1 -- 172.21.15.107:0/2118853766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5607b7a9ba70 con 0x5607b7a16a80 2024-05-09T08:38:18.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.315+0000 7fe4b0ff9640 1 -- 172.21.15.107:0/2118853766 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4c4005450 con 0x5607b7a16a80 2024-05-09T08:38:18.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.315+0000 7fe4b0ff9640 1 -- 172.21.15.107:0/2118853766 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe4c4012450 con 0x5607b7a16a80 2024-05-09T08:38:18.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.315+0000 7fe4d2571740 10 client.17067 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:18.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.315+0000 7fe4d2571740 10 client.17067 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:18.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.315+0000 7fe4d2571740 20 client.17067 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:18.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.315+0000 7fe4d2571740 1 --2- 172.21.15.107:0/2118853766 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607b7a9bf40 0x5607b7a9e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.315+0000 7fe4d2571740 4 client.17067 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:38:18.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.315+0000 7fe4d2571740 1 -- 172.21.15.107:0/2118853766 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5607b7983780 con 0x5607b7a9bf40 2024-05-09T08:38:18.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.315+0000 7fe4ccd24640 1 --2- 172.21.15.107:0/2118853766 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607b7a9bf40 0x5607b7a9e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4ccd24640 1 --2- 172.21.15.107:0/2118853766 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607b7a9bf40 0x5607b7a9e330 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f96811c5d9314733 server_cookie=3fa1772c04a8fcef in_seq=0 out_seq=0 2024-05-09T08:38:18.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4b0ff9640 10 client.17067 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:18.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4b0ff9640 1 -- 172.21.15.107:0/2118853766 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5607b7983780 con 0x5607b7a9bf40 2024-05-09T08:38:18.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4b0ff9640 10 client.17067 handle_command_reply: tid=0 2024-05-09T08:38:18.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4d2571740 1 client.17067 shutdown 2024-05-09T08:38:18.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4d2571740 1 -- 172.21.15.107:0/2118853766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16a80 msgr2=0x5607b7a1edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4d2571740 1 --2- 172.21.15.107:0/2118853766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16a80 0x5607b7a1edf0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4027040 tx=0x7fe4c4005600 comp rx=0 tx=0).stop 2024-05-09T08:38:18.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4d2571740 1 -- 172.21.15.107:0/2118853766 shutdown_connections 2024-05-09T08:38:18.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4d2571740 1 --2- 172.21.15.107:0/2118853766 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607b7a9bf40 0x5607b7a9e330 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4d2571740 1 --2- 172.21.15.107:0/2118853766 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607b7a25680 0x5607b7a1f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4d2571740 1 --2- 172.21.15.107:0/2118853766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607b7a16a80 0x5607b7a1edf0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4b0ff9640 0 client.17067 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:18.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4d2571740 1 --2- 172.21.15.107:0/2118853766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607b7a16160 0x5607b7a2db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4d2571740 1 -- 172.21.15.107:0/2118853766 >> 172.21.15.107:0/2118853766 conn(0x5607b7a125a0 msgr2=0x5607b7a22520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:18.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4d2571740 1 -- 172.21.15.107:0/2118853766 shutdown_connections 2024-05-09T08:38:18.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4d2571740 1 -- 172.21.15.107:0/2118853766 wait complete. 2024-05-09T08:38:18.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.316+0000 7fe4d2571740 20 client.17067 trim_cache size 0 max 16384 2024-05-09T08:38:18.322 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:38:18.322 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:38:18.322 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:38:18.322 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:38:18.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.317+0000 7fe4cf789640 1 -- 172.21.15.107:0/3127321470 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4b40a2d00 msgr2=0x7fe4b40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.317+0000 7fe4cf789640 1 --2- 172.21.15.107:0/3127321470 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4b40a2d00 0x7fe4b40a51c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe4c00cd290 tx=0x7fe4c000efd0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.317+0000 7fe4cf789640 1 -- 172.21.15.107:0/3127321470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c8106710 msgr2=0x7fe4c819c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.317+0000 7fe4cf789640 1 --2- 172.21.15.107:0/3127321470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c8106710 0x7fe4c819c950 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000eb10 tx=0x7fe4c000c2d0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.317+0000 7fe4cf789640 1 -- 172.21.15.107:0/3127321470 shutdown_connections 2024-05-09T08:38:18.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.317+0000 7fe4cf789640 1 --2- 172.21.15.107:0/3127321470 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4b40a2d00 0x7fe4b40a51c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.317+0000 7fe4cf789640 1 --2- 172.21.15.107:0/3127321470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c8106710 0x7fe4c819c950 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.317+0000 7fe4cf789640 1 --2- 172.21.15.107:0/3127321470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4c81010c0 0x7fe4c8198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.317+0000 7fe4cf789640 1 --2- 172.21.15.107:0/3127321470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4c8100700 0x7fe4c81980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.317+0000 7fe4cf789640 1 -- 172.21.15.107:0/3127321470 >> 172.21.15.107:0/3127321470 conn(0x7fe4c8077830 msgr2=0x7fe4c80ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:18.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.317+0000 7fe4cf789640 1 -- 172.21.15.107:0/3127321470 shutdown_connections 2024-05-09T08:38:18.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.317+0000 7fe4cf789640 1 -- 172.21.15.107:0/3127321470 wait complete. 2024-05-09T08:38:18.329 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:38:18.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:38:18.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.450+0000 7f8fc3cfc640 1 -- 172.21.15.107:0/4172738936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fbc100960 msgr2=0x7f8fbc106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.450+0000 7f8fc3cfc640 1 --2- 172.21.15.107:0/4172738936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fbc100960 0x7f8fbc106490 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f8fac009900 tx=0x7f8fac02a410 comp rx=0 tx=0).stop 2024-05-09T08:38:18.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.450+0000 7f8fc3cfc640 1 -- 172.21.15.107:0/4172738936 shutdown_connections 2024-05-09T08:38:18.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.450+0000 7f8fc3cfc640 1 --2- 172.21.15.107:0/4172738936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fbc106bc0 0x7f8fbc10e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.450+0000 7f8fc3cfc640 1 --2- 172.21.15.107:0/4172738936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fbc100960 0x7f8fbc106490 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.450+0000 7f8fc3cfc640 1 --2- 172.21.15.107:0/4172738936 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fbc0fffb0 0x7f8fbc100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.450+0000 7f8fc3cfc640 1 -- 172.21.15.107:0/4172738936 >> 172.21.15.107:0/4172738936 conn(0x7f8fbc076cd0 msgr2=0x7f8fbc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:18.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.450+0000 7f8fc3cfc640 1 -- 172.21.15.107:0/4172738936 shutdown_connections 2024-05-09T08:38:18.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.450+0000 7f8fc3cfc640 1 -- 172.21.15.107:0/4172738936 wait complete. 2024-05-09T08:38:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc3cfc640 1 Processor -- start 2024-05-09T08:38:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc3cfc640 1 -- start start 2024-05-09T08:38:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc3cfc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fbc0fffb0 0x7f8fbc198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc3cfc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fbc106bc0 0x7f8fbc1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc3cfc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fbc19c920 0x7f8fbc19cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc3cfc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc111b20 con 0x7f8fbc106bc0 2024-05-09T08:38:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc3cfc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc1119a0 con 0x7f8fbc19c920 2024-05-09T08:38:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc3cfc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc111ca0 con 0x7f8fbc0fffb0 2024-05-09T08:38:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc1297640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fbc106bc0 0x7f8fbc1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc1297640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fbc106bc0 0x7f8fbc1985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46498/0 (socket says 172.21.15.107:46498) 2024-05-09T08:38:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc1297640 1 -- 172.21.15.107:0/176855977 learned_addr learned my addr 172.21.15.107:0/176855977 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc1a98640 1 --2- 172.21.15.107:0/176855977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fbc0fffb0 0x7f8fbc198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc2299640 1 --2- 172.21.15.107:0/176855977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fbc19c920 0x7f8fbc19cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc1297640 1 -- 172.21.15.107:0/176855977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fbc0fffb0 msgr2=0x7f8fbc198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc1297640 1 --2- 172.21.15.107:0/176855977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fbc0fffb0 0x7f8fbc198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc1297640 1 -- 172.21.15.107:0/176855977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fbc19c920 msgr2=0x7f8fbc19cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc1297640 1 --2- 172.21.15.107:0/176855977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fbc19c920 0x7f8fbc19cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.451+0000 7f8fc1297640 1 -- 172.21.15.107:0/176855977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fbc19d410 con 0x7f8fbc106bc0 2024-05-09T08:38:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.452+0000 7f8fc2299640 1 --2- 172.21.15.107:0/176855977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fbc19c920 0x7f8fbc19cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:38:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.452+0000 7f8fc1297640 1 --2- 172.21.15.107:0/176855977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fbc106bc0 0x7f8fbc1985a0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f8fac005ab0 tx=0x7f8fac003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.452+0000 7f8fb2ffd640 1 -- 172.21.15.107:0/176855977 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac03f070 con 0x7f8fbc106bc0 2024-05-09T08:38:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.452+0000 7f8fb2ffd640 1 -- 172.21.15.107:0/176855977 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fac031d60 con 0x7f8fbc106bc0 2024-05-09T08:38:18.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.452+0000 7f8fc3cfc640 1 -- 172.21.15.107:0/176855977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fbc19d6a0 con 0x7f8fbc106bc0 2024-05-09T08:38:18.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.452+0000 7f8fb2ffd640 1 -- 172.21.15.107:0/176855977 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac035030 con 0x7f8fbc106bc0 2024-05-09T08:38:18.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.452+0000 7f8fc3cfc640 1 -- 172.21.15.107:0/176855977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fbc1a4ec0 con 0x7f8fbc106bc0 2024-05-09T08:38:18.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.453+0000 7f8fc3cfc640 1 -- 172.21.15.107:0/176855977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fbc067dc0 con 0x7f8fbc106bc0 2024-05-09T08:38:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.454+0000 7f8fb2ffd640 1 -- 172.21.15.107:0/176855977 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8fac042070 con 0x7f8fbc106bc0 2024-05-09T08:38:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.455+0000 7f8fb2ffd640 1 --2- 172.21.15.107:0/176855977 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fa80a2d80 0x7f8fa80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.455+0000 7f8fc1a98640 1 --2- 172.21.15.107:0/176855977 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fa80a2d80 0x7f8fa80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.455+0000 7f8fb2ffd640 1 -- 172.21.15.107:0/176855977 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8fac0efb40 con 0x7f8fbc106bc0 2024-05-09T08:38:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.455+0000 7f8fc1a98640 1 --2- 172.21.15.107:0/176855977 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fa80a2d80 0x7f8fa80a5240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8005fd0 tx=0x7f8fb800b390 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:18.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.460+0000 7f8fb2ffd640 1 -- 172.21.15.107:0/176855977 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8fac0bc400 con 0x7f8fbc106bc0 2024-05-09T08:38:18.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:18 smithi107 ceph-mon[30122]: pgmap v151: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-05-09T08:38:18.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:18 smithi157 ceph-mon[33614]: pgmap v151: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-05-09T08:38:18.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:18 smithi110 ceph-mon[33622]: pgmap v151: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-05-09T08:38:18.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.717+0000 7f8fc3cfc640 1 -- 172.21.15.107:0/176855977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fbc1993b0 con 0x7f8fbc106bc0 2024-05-09T08:38:18.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.718+0000 7f8fb2ffd640 1 -- 172.21.15.107:0/176855977 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11887 (secure 0 0 0) 0x7f8fac0bbd00 con 0x7f8fbc106bc0 2024-05-09T08:38:18.717 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:38:18.717 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:37:10.930317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:38:18.717 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2024-05-09T08:38:18.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.720+0000 7f8fc3cfc640 1 -- 172.21.15.107:0/176855977 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fa80a2d80 msgr2=0x7f8fa80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.720+0000 7f8fc3cfc640 1 --2- 172.21.15.107:0/176855977 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fa80a2d80 0x7f8fa80a5240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8005fd0 tx=0x7f8fb800b390 comp rx=0 tx=0).stop 2024-05-09T08:38:18.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.720+0000 7f8fc3cfc640 1 -- 172.21.15.107:0/176855977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fbc106bc0 msgr2=0x7f8fbc1985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.720+0000 7f8fc3cfc640 1 --2- 172.21.15.107:0/176855977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fbc106bc0 0x7f8fbc1985a0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f8fac005ab0 tx=0x7f8fac003cd0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.720+0000 7f8fc3cfc640 1 -- 172.21.15.107:0/176855977 shutdown_connections 2024-05-09T08:38:18.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.720+0000 7f8fc3cfc640 1 --2- 172.21.15.107:0/176855977 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fa80a2d80 0x7f8fa80a5240 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.720+0000 7f8fc3cfc640 1 --2- 172.21.15.107:0/176855977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fbc19c920 0x7f8fbc19cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.720+0000 7f8fc3cfc640 1 --2- 172.21.15.107:0/176855977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fbc106bc0 0x7f8fbc1985a0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.720+0000 7f8fc3cfc640 1 --2- 172.21.15.107:0/176855977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fbc0fffb0 0x7f8fbc198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.720+0000 7f8fc3cfc640 1 -- 172.21.15.107:0/176855977 >> 172.21.15.107:0/176855977 conn(0x7f8fbc076cd0 msgr2=0x7f8fbc104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:18.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.721+0000 7f8fc3cfc640 1 -- 172.21.15.107:0/176855977 shutdown_connections 2024-05-09T08:38:18.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.721+0000 7f8fc3cfc640 1 -- 172.21.15.107:0/176855977 wait complete. 2024-05-09T08:38:18.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:38:18.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.846+0000 7f8119f14640 1 -- 172.21.15.107:0/4271837224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8114107820 msgr2=0x7f8114107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.846+0000 7f8119f14640 1 --2- 172.21.15.107:0/4271837224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8114107820 0x7f8114107c00 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f8104009930 tx=0x7f810402a420 comp rx=0 tx=0).stop 2024-05-09T08:38:18.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.846+0000 7f8119f14640 1 -- 172.21.15.107:0/4271837224 shutdown_connections 2024-05-09T08:38:18.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.846+0000 7f8119f14640 1 --2- 172.21.15.107:0/4271837224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81141021d0 0x7f811410e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.846+0000 7f8119f14640 1 --2- 172.21.15.107:0/4271837224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8114101810 0x7f8114101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.846+0000 7f8119f14640 1 --2- 172.21.15.107:0/4271837224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8114107820 0x7f8114107c00 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.846+0000 7f8119f14640 1 -- 172.21.15.107:0/4271837224 >> 172.21.15.107:0/4271837224 conn(0x7f8114078c30 msgr2=0x7f8114079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:18.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.846+0000 7f8119f14640 1 -- 172.21.15.107:0/4271837224 shutdown_connections 2024-05-09T08:38:18.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.846+0000 7f8119f14640 1 -- 172.21.15.107:0/4271837224 wait complete. 2024-05-09T08:38:18.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.847+0000 7f8119f14640 1 Processor -- start 2024-05-09T08:38:18.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.847+0000 7f8119f14640 1 -- start start 2024-05-09T08:38:18.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.847+0000 7f8119f14640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8114101810 0x7f8114198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.847+0000 7f8119f14640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81141021d0 0x7f81141985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.847+0000 7f8119f14640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8114107820 0x7f811419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.847+0000 7f8119f14640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8114111ae0 con 0x7f81141021d0 2024-05-09T08:38:18.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.847+0000 7f8119f14640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8114111960 con 0x7f8114107820 2024-05-09T08:38:18.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.847+0000 7f8119f14640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8114111c60 con 0x7f8114101810 2024-05-09T08:38:18.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f8112ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81141021d0 0x7f81141985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f8112ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81141021d0 0x7f81141985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46510/0 (socket says 172.21.15.107:46510) 2024-05-09T08:38:18.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f8112ffd640 1 -- 172.21.15.107:0/3636973548 learned_addr learned my addr 172.21.15.107:0/3636973548 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:18.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f81137fe640 1 --2- 172.21.15.107:0/3636973548 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8114101810 0x7f8114198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f8113fff640 1 --2- 172.21.15.107:0/3636973548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8114107820 0x7f811419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f8112ffd640 1 -- 172.21.15.107:0/3636973548 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8114101810 msgr2=0x7f8114198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f8112ffd640 1 --2- 172.21.15.107:0/3636973548 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8114101810 0x7f8114198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f8112ffd640 1 -- 172.21.15.107:0/3636973548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8114107820 msgr2=0x7f811419c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f8112ffd640 1 --2- 172.21.15.107:0/3636973548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8114107820 0x7f811419c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f8112ffd640 1 -- 172.21.15.107:0/3636973548 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811419d070 con 0x7f81141021d0 2024-05-09T08:38:18.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f8113fff640 1 --2- 172.21.15.107:0/3636973548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8114107820 0x7f811419c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:18.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f8112ffd640 1 --2- 172.21.15.107:0/3636973548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81141021d0 0x7f81141985c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f810800e9f0 tx=0x7f810800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:18.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f8110ff9640 1 -- 172.21.15.107:0/3636973548 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810800cd80 con 0x7f81141021d0 2024-05-09T08:38:18.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f8119f14640 1 -- 172.21.15.107:0/3636973548 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811419d360 con 0x7f81141021d0 2024-05-09T08:38:18.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f8110ff9640 1 -- 172.21.15.107:0/3636973548 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f81080054d0 con 0x7f81141021d0 2024-05-09T08:38:18.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f8119f14640 1 -- 172.21.15.107:0/3636973548 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81141a4bf0 con 0x7f81141021d0 2024-05-09T08:38:18.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.848+0000 7f8110ff9640 1 -- 172.21.15.107:0/3636973548 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108018630 con 0x7f81141021d0 2024-05-09T08:38:18.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.851+0000 7f8110ff9640 1 -- 172.21.15.107:0/3636973548 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8108004e60 con 0x7f81141021d0 2024-05-09T08:38:18.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.851+0000 7f811cc24740 1 Processor -- start 2024-05-09T08:38:18.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.851+0000 7f811cc24740 1 -- start start 2024-05-09T08:38:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.851+0000 7f811cc24740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0420d230 0x55ff0420d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.851+0000 7f811cc24740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420db50 0x55ff0421c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.851+0000 7f811cc24740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0421c730 0x55ff0421eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.851+0000 7f811cc24740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ff04172c40 con 0x55ff0420d230 2024-05-09T08:38:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.851+0000 7f811cc24740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ff0411a580 con 0x55ff0420db50 2024-05-09T08:38:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f811cc24740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ff0416ec00 con 0x55ff0421c730 2024-05-09T08:38:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f8110ff9640 1 --2- 172.21.15.107:0/3636973548 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f80fc0a2d00 0x7f80fc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f8113fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0420d230 0x55ff0420d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f8113fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0420d230 0x55ff0420d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46526/0 (socket says 172.21.15.107:46526) 2024-05-09T08:38:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f8113fff640 1 -- 172.21.15.107:0/2931300424 learned_addr learned my addr 172.21.15.107:0/2931300424 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f8110ff9640 1 -- 172.21.15.107:0/3636973548 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f81080cfcc0 con 0x7f81141021d0 2024-05-09T08:38:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f8112ffd640 1 --2- 172.21.15.107:0/2931300424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0421c730 0x55ff0421eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f8113fff640 1 --2- 172.21.15.107:0/3636973548 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f80fc0a2d00 0x7f80fc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f81137fe640 1 --2- 172.21.15.107:0/2931300424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420db50 0x55ff0421c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f8112ffd640 1 -- 172.21.15.107:0/2931300424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420db50 msgr2=0x55ff0421c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f8112ffd640 1 --2- 172.21.15.107:0/2931300424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420db50 0x55ff0421c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f8112ffd640 1 -- 172.21.15.107:0/2931300424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0420d230 msgr2=0x55ff0420d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f8112ffd640 1 --2- 172.21.15.107:0/2931300424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0420d230 0x55ff0420d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f8112ffd640 1 -- 172.21.15.107:0/2931300424 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff03e863f0 con 0x55ff0421c730 2024-05-09T08:38:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f8113fff640 1 --2- 172.21.15.107:0/2931300424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0420d230 0x55ff0420d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f8112ffd640 1 --2- 172.21.15.107:0/2931300424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0421c730 0x55ff0421eb20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f81080cd080 tx=0x7f81080097b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88da22b9aed54e0a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f8113fff640 1 --2- 172.21.15.107:0/3636973548 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f80fc0a2d00 0x7f80fc0a51c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f810c012d10 tx=0x7f810c009500 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.852+0000 7f80f9ffb640 1 -- 172.21.15.107:0/2931300424 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81080b8030 con 0x55ff0421c730 2024-05-09T08:38:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.853+0000 7f80f9ffb640 1 -- 172.21.15.107:0/2931300424 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f810809c580 con 0x55ff0421c730 2024-05-09T08:38:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.853+0000 7f80f9ffb640 1 -- 172.21.15.107:0/2931300424 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81080cd330 con 0x55ff0421c730 2024-05-09T08:38:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.853+0000 7f811cc24740 1 -- 172.21.15.107:0/2931300424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0421c730 msgr2=0x55ff0421eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.853+0000 7f811cc24740 1 --2- 172.21.15.107:0/2931300424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0421c730 0x55ff0421eb20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f81080cd080 tx=0x7f81080097b0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.853+0000 7f811cc24740 1 -- 172.21.15.107:0/2931300424 shutdown_connections 2024-05-09T08:38:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.853+0000 7f811cc24740 1 --2- 172.21.15.107:0/2931300424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0421c730 0x55ff0421eb20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.853+0000 7f811cc24740 1 --2- 172.21.15.107:0/2931300424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420db50 0x55ff0421c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.853+0000 7f811cc24740 1 --2- 172.21.15.107:0/2931300424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0420d230 0x55ff0420d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.853+0000 7f811cc24740 1 -- 172.21.15.107:0/2931300424 >> 172.21.15.107:0/2931300424 conn(0x55ff04209650 msgr2=0x55ff04209a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.853+0000 7f811cc24740 1 -- 172.21.15.107:0/2931300424 shutdown_connections 2024-05-09T08:38:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.853+0000 7f811cc24740 1 -- 172.21.15.107:0/2931300424 wait complete. 2024-05-09T08:38:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.854+0000 7f811cc24740 1 -- start start 2024-05-09T08:38:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.854+0000 7f811cc24740 1 Processor -- start 2024-05-09T08:38:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.854+0000 7f811cc24740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420d230 0x55ff04219660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.854+0000 7f811cc24740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0420db50 0x55ff04219ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.854+0000 7f811cc24740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0421c730 0x55ff0421a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.854+0000 7f811cc24740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ff0421f4f0 con 0x55ff0421c730 2024-05-09T08:38:18.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.854+0000 7f811cc24740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ff0412dc40 con 0x55ff0420d230 2024-05-09T08:38:18.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.854+0000 7f811cc24740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ff0421f670 con 0x55ff0420db50 2024-05-09T08:38:18.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.854+0000 7f811cc24740 10 client.0 _ll_register_callbacks cb 0x55ff03e85be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:18.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.854+0000 7f8113fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0421c730 0x55ff0421a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.854+0000 7f81137fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420d230 0x55ff04219660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.854+0000 7f8113fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0421c730 0x55ff0421a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46536/0 (socket says 172.21.15.107:46536) 2024-05-09T08:38:18.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.854+0000 7f8113fff640 1 -- 172.21.15.107:0/3700107618 learned_addr learned my addr 172.21.15.107:0/3700107618 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:18.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.854+0000 7f81137fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420d230 0x55ff04219660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:46556/0 (socket says 172.21.15.107:46556) 2024-05-09T08:38:18.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.855+0000 7f8112ffd640 1 --2- 172.21.15.107:0/3700107618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0420db50 0x55ff04219ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.855+0000 7f81137fe640 1 -- 172.21.15.107:0/3700107618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0420db50 msgr2=0x55ff04219ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:18.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.855+0000 7f81137fe640 1 --2- 172.21.15.107:0/3700107618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0420db50 0x55ff04219ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.855+0000 7f81137fe640 1 -- 172.21.15.107:0/3700107618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0421c730 msgr2=0x55ff0421a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:18.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.855+0000 7f81137fe640 1 --2- 172.21.15.107:0/3700107618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0421c730 0x55ff0421a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.855+0000 7f81137fe640 1 -- 172.21.15.107:0/3700107618 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff03f507d0 con 0x55ff0420d230 2024-05-09T08:38:18.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.855+0000 7f8113fff640 1 --2- 172.21.15.107:0/3700107618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0421c730 0x55ff0421a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:18.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.855+0000 7f81137fe640 1 --2- 172.21.15.107:0/3700107618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420d230 0x55ff04219660 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8104031440 tx=0x7f8104031ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53584bbe46497b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:18.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.855+0000 7f80f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:38:18.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.855+0000 7f80f2ffd640 1 -- 172.21.15.107:0/3700107618 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8104035040 con 0x55ff0420d230 2024-05-09T08:38:18.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.855+0000 7f80f2ffd640 1 -- 172.21.15.107:0/3700107618 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8104031720 con 0x55ff0420d230 2024-05-09T08:38:18.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.855+0000 7f811cc24740 1 -- 172.21.15.107:0/3700107618 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f810809c580 con 0x55ff0420d230 2024-05-09T08:38:18.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.855+0000 7f80f2ffd640 1 -- 172.21.15.107:0/3700107618 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f810404f050 con 0x55ff0420d230 2024-05-09T08:38:18.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.855+0000 7f811cc24740 10 client.26237 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:18.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.855+0000 7f811cc24740 1 -- 172.21.15.107:0/3700107618 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ff0422b6d0 con 0x55ff0420d230 2024-05-09T08:38:18.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.856+0000 7f80f2ffd640 1 -- 172.21.15.107:0/3700107618 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f810809c580 con 0x55ff0420d230 2024-05-09T08:38:18.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.856+0000 7f80f2ffd640 1 -- 172.21.15.107:0/3700107618 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f81040047a0 con 0x55ff0420d230 2024-05-09T08:38:18.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.856+0000 7f811cc24740 10 client.26237 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:18.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.856+0000 7f811cc24740 10 client.26237 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:38:18.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.856+0000 7f811cc24740 20 client.26237 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:18.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.856+0000 7f811cc24740 1 --2- 172.21.15.107:0/3700107618 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55ff0422cb90 0x55ff0422ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.856+0000 7f811cc24740 4 client.26237 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:18.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.856+0000 7f811cc24740 1 -- 172.21.15.107:0/3700107618 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff04180370 con 0x55ff0422cb90 2024-05-09T08:38:18.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.856+0000 7f8112ffd640 1 --2- 172.21.15.107:0/3700107618 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55ff0422cb90 0x55ff0422ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.857+0000 7f80f2ffd640 10 client.26237 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:38:18.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.857+0000 7f8112ffd640 1 --2- 172.21.15.107:0/3700107618 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55ff0422cb90 0x55ff0422ef80 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4665216219521b22 server_cookie=adaa22cdc2fff73a in_seq=0 out_seq=0 2024-05-09T08:38:18.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.857+0000 7f80f2ffd640 1 -- 172.21.15.107:0/3700107618 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ff04180370 con 0x55ff0422cb90 2024-05-09T08:38:18.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.857+0000 7f80f2ffd640 10 client.26237 handle_command_reply: tid=0 2024-05-09T08:38:18.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.857+0000 7f811cc24740 1 client.26237 shutdown 2024-05-09T08:38:18.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.857+0000 7f811cc24740 1 -- 172.21.15.107:0/3700107618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420d230 msgr2=0x55ff04219660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.857+0000 7f811cc24740 1 --2- 172.21.15.107:0/3700107618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420d230 0x55ff04219660 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8104031440 tx=0x7f8104031ef0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.858+0000 7f811cc24740 1 -- 172.21.15.107:0/3700107618 shutdown_connections 2024-05-09T08:38:18.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.858+0000 7f811cc24740 1 --2- 172.21.15.107:0/3700107618 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55ff0422cb90 0x55ff0422ef80 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.858+0000 7f811cc24740 1 --2- 172.21.15.107:0/3700107618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0421c730 0x55ff0421a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.858+0000 7f811cc24740 1 --2- 172.21.15.107:0/3700107618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0420db50 0x55ff04219ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.858+0000 7f811cc24740 1 --2- 172.21.15.107:0/3700107618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420d230 0x55ff04219660 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.858+0000 7f811cc24740 1 -- 172.21.15.107:0/3700107618 >> 172.21.15.107:0/3700107618 conn(0x55ff04209650 msgr2=0x55ff0420f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:18.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.858+0000 7f80f2ffd640 0 client.26237 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:38:18.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.858+0000 7f811cc24740 1 -- 172.21.15.107:0/3700107618 shutdown_connections 2024-05-09T08:38:18.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.858+0000 7f811cc24740 1 -- 172.21.15.107:0/3700107618 wait complete. 2024-05-09T08:38:18.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.858+0000 7f811cc24740 20 client.26237 trim_cache size 0 max 16384 2024-05-09T08:38:18.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.874+0000 7f811cc24740 1 Processor -- start 2024-05-09T08:38:18.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.874+0000 7f811cc24740 1 -- start start 2024-05-09T08:38:18.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.874+0000 7f811cc24740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0420d230 0x55ff042283f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.874+0000 7f811cc24740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420db50 0x55ff04208630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.874+0000 7f811cc24740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0421c730 0x55ff04234850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.874+0000 7f811cc24740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f81141af250 con 0x55ff0421c730 2024-05-09T08:38:18.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.874+0000 7f811cc24740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f81141acb20 con 0x55ff0420db50 2024-05-09T08:38:18.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.874+0000 7f811cc24740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f81141aa970 con 0x55ff0420d230 2024-05-09T08:38:18.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.875+0000 7f8113fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0421c730 0x55ff04234850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.875+0000 7f8113fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0421c730 0x55ff04234850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46546/0 (socket says 172.21.15.107:46546) 2024-05-09T08:38:18.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.875+0000 7f8113fff640 1 -- 172.21.15.107:0/690534089 learned_addr learned my addr 172.21.15.107:0/690534089 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:18.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.875+0000 7f8112ffd640 1 --2- 172.21.15.107:0/690534089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420db50 0x55ff04208630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.875+0000 7f81137fe640 1 --2- 172.21.15.107:0/690534089 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0420d230 0x55ff042283f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.875+0000 7f8113fff640 1 -- 172.21.15.107:0/690534089 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0420d230 msgr2=0x55ff042283f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:18.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.875+0000 7f8113fff640 1 --2- 172.21.15.107:0/690534089 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0420d230 0x55ff042283f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.875+0000 7f8113fff640 1 -- 172.21.15.107:0/690534089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420db50 msgr2=0x55ff04208630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:18.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.875+0000 7f8113fff640 1 --2- 172.21.15.107:0/690534089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420db50 0x55ff04208630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.875+0000 7f8113fff640 1 -- 172.21.15.107:0/690534089 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff041fad20 con 0x55ff0421c730 2024-05-09T08:38:18.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.875+0000 7f81137fe640 1 --2- 172.21.15.107:0/690534089 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0420d230 0x55ff042283f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:18.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.875+0000 7f8113fff640 1 --2- 172.21.15.107:0/690534089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0421c730 0x55ff04234850 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f810c009fb0 tx=0x7f810c009a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf92ba8015dcce68 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:18.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.875+0000 7f80f97fa640 1 -- 172.21.15.107:0/690534089 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810c012e00 con 0x55ff0421c730 2024-05-09T08:38:18.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.875+0000 7f80f97fa640 1 -- 172.21.15.107:0/690534089 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f810c00fd60 con 0x55ff0421c730 2024-05-09T08:38:18.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.875+0000 7f80f97fa640 1 -- 172.21.15.107:0/690534089 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810c0056f0 con 0x55ff0421c730 2024-05-09T08:38:18.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.876+0000 7f811cc24740 1 -- 172.21.15.107:0/690534089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0421c730 msgr2=0x55ff04234850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.876+0000 7f811cc24740 1 --2- 172.21.15.107:0/690534089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0421c730 0x55ff04234850 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f810c009fb0 tx=0x7f810c009a40 comp rx=0 tx=0).stop 2024-05-09T08:38:18.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.876+0000 7f811cc24740 1 -- 172.21.15.107:0/690534089 shutdown_connections 2024-05-09T08:38:18.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.876+0000 7f811cc24740 1 --2- 172.21.15.107:0/690534089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0421c730 0x55ff04234850 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.876+0000 7f811cc24740 1 --2- 172.21.15.107:0/690534089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0420db50 0x55ff04208630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.876+0000 7f811cc24740 1 --2- 172.21.15.107:0/690534089 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0420d230 0x55ff042283f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.876+0000 7f811cc24740 1 -- 172.21.15.107:0/690534089 >> 172.21.15.107:0/690534089 conn(0x55ff04209650 msgr2=0x55ff04207f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:18.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.876+0000 7f811cc24740 1 -- 172.21.15.107:0/690534089 shutdown_connections 2024-05-09T08:38:18.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.876+0000 7f811cc24740 1 -- 172.21.15.107:0/690534089 wait complete. 2024-05-09T08:38:18.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.876+0000 7f811cc24740 1 -- start start 2024-05-09T08:38:18.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f811cc24740 1 Processor -- start 2024-05-09T08:38:18.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f811cc24740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0420d230 0x55ff04224d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f811cc24740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0420db50 0x55ff04220090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f811cc24740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0421c730 0x55ff042205f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f811cc24740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f81141aafe0 con 0x55ff0420db50 2024-05-09T08:38:18.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f811cc24740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f81141ac450 con 0x55ff0421c730 2024-05-09T08:38:18.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f811cc24740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f81141ab6b0 con 0x55ff0420d230 2024-05-09T08:38:18.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f811cc24740 10 client.0 _ll_register_callbacks cb 0x7f8104009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:18.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f8112ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0420db50 0x55ff04220090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f8112ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0420db50 0x55ff04220090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46548/0 (socket says 172.21.15.107:46548) 2024-05-09T08:38:18.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f8112ffd640 1 -- 172.21.15.107:0/107145325 learned_addr learned my addr 172.21.15.107:0/107145325 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:18.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f8113fff640 1 --2- 172.21.15.107:0/107145325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0421c730 0x55ff042205f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f81137fe640 1 --2- 172.21.15.107:0/107145325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0420d230 0x55ff04224d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f8112ffd640 1 -- 172.21.15.107:0/107145325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0420d230 msgr2=0x55ff04224d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:18.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f8112ffd640 1 --2- 172.21.15.107:0/107145325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0420d230 0x55ff04224d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f8112ffd640 1 -- 172.21.15.107:0/107145325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0421c730 msgr2=0x55ff042205f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:18.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f8112ffd640 1 --2- 172.21.15.107:0/107145325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0421c730 0x55ff042205f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f8112ffd640 1 -- 172.21.15.107:0/107145325 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff04220e70 con 0x55ff0420db50 2024-05-09T08:38:18.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.877+0000 7f8113fff640 1 --2- 172.21.15.107:0/107145325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0421c730 0x55ff042205f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:18.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.878+0000 7f8112ffd640 1 --2- 172.21.15.107:0/107145325 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0420db50 0x55ff04220090 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f81080cd160 tx=0x7f81080cfec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3756c995683e81c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:18.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.878+0000 7f80f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:18.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.878+0000 7f80f2ffd640 1 -- 172.21.15.107:0/107145325 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81080b8040 con 0x55ff0420db50 2024-05-09T08:38:18.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.878+0000 7f811cc24740 1 -- 172.21.15.107:0/107145325 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f810c00fd60 con 0x55ff0420db50 2024-05-09T08:38:18.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.878+0000 7f80f2ffd640 1 -- 172.21.15.107:0/107145325 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f81080046c0 con 0x55ff0420db50 2024-05-09T08:38:18.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.878+0000 7f80f2ffd640 1 -- 172.21.15.107:0/107145325 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f810809c580 con 0x55ff0420db50 2024-05-09T08:38:18.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.878+0000 7f811cc24740 10 client.17091 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:18.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.878+0000 7f811cc24740 1 -- 172.21.15.107:0/107145325 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ff04293130 con 0x55ff0420db50 2024-05-09T08:38:18.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.878+0000 7f80f2ffd640 1 -- 172.21.15.107:0/107145325 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f810c00fd60 con 0x55ff0420db50 2024-05-09T08:38:18.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.878+0000 7f80f2ffd640 1 -- 172.21.15.107:0/107145325 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f81080cd330 con 0x55ff0420db50 2024-05-09T08:38:18.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.878+0000 7f811cc24740 10 client.17091 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:18.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.878+0000 7f811cc24740 10 client.17091 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:38:18.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.878+0000 7f811cc24740 20 client.17091 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:18.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.878+0000 7f811cc24740 1 --2- 172.21.15.107:0/107145325 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55ff042933c0 0x55ff042957b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:18.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.878+0000 7f811cc24740 4 client.17091 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:38:18.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.878+0000 7f811cc24740 1 -- 172.21.15.107:0/107145325 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ff0417a6c0 con 0x55ff042933c0 2024-05-09T08:38:18.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.879+0000 7f81137fe640 1 --2- 172.21.15.107:0/107145325 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55ff042933c0 0x55ff042957b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:18.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.879+0000 7f81137fe640 1 --2- 172.21.15.107:0/107145325 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55ff042933c0 0x55ff042957b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=41d23fee1d7e8ee9 server_cookie=7b979370ca4538af in_seq=0 out_seq=0 2024-05-09T08:38:18.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.879+0000 7f80f2ffd640 10 client.17091 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:38:18.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.879+0000 7f80f2ffd640 1 -- 172.21.15.107:0/107145325 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ff0417a6c0 con 0x55ff042933c0 2024-05-09T08:38:18.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.879+0000 7f80f2ffd640 10 client.17091 handle_command_reply: tid=0 2024-05-09T08:38:18.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.879+0000 7f811cc24740 1 client.17091 shutdown 2024-05-09T08:38:18.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.879+0000 7f811cc24740 1 -- 172.21.15.107:0/107145325 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0420db50 msgr2=0x55ff04220090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.879+0000 7f811cc24740 1 --2- 172.21.15.107:0/107145325 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0420db50 0x55ff04220090 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f81080cd160 tx=0x7f81080cfec0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.879+0000 7f811cc24740 1 -- 172.21.15.107:0/107145325 shutdown_connections 2024-05-09T08:38:18.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.879+0000 7f811cc24740 1 --2- 172.21.15.107:0/107145325 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55ff042933c0 0x55ff042957b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.879+0000 7f811cc24740 1 --2- 172.21.15.107:0/107145325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ff0421c730 0x55ff042205f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.879+0000 7f811cc24740 1 --2- 172.21.15.107:0/107145325 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ff0420db50 0x55ff04220090 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.879+0000 7f811cc24740 1 --2- 172.21.15.107:0/107145325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff0420d230 0x55ff04224d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.879+0000 7f811cc24740 1 -- 172.21.15.107:0/107145325 >> 172.21.15.107:0/107145325 conn(0x55ff04209650 msgr2=0x55ff042087b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:18.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.879+0000 7f80f2ffd640 0 client.17091 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:38:18.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.880+0000 7f811cc24740 1 -- 172.21.15.107:0/107145325 shutdown_connections 2024-05-09T08:38:18.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.880+0000 7f811cc24740 1 -- 172.21.15.107:0/107145325 wait complete. 2024-05-09T08:38:18.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.880+0000 7f811cc24740 20 client.17091 trim_cache size 0 max 16384 2024-05-09T08:38:18.886 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:38:18.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.880+0000 7f8119f14640 1 -- 172.21.15.107:0/3636973548 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f80fc0a2d00 msgr2=0x7f80fc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.880+0000 7f8119f14640 1 --2- 172.21.15.107:0/3636973548 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f80fc0a2d00 0x7f80fc0a51c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f810c012d10 tx=0x7f810c009500 comp rx=0 tx=0).stop 2024-05-09T08:38:18.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.880+0000 7f8119f14640 1 -- 172.21.15.107:0/3636973548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81141021d0 msgr2=0x7f81141985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:18.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.880+0000 7f8119f14640 1 --2- 172.21.15.107:0/3636973548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81141021d0 0x7f81141985c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f810800e9f0 tx=0x7f810800edf0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.880+0000 7f8119f14640 1 -- 172.21.15.107:0/3636973548 shutdown_connections 2024-05-09T08:38:18.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.880+0000 7f8119f14640 1 --2- 172.21.15.107:0/3636973548 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f80fc0a2d00 0x7f80fc0a51c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.880+0000 7f8119f14640 1 --2- 172.21.15.107:0/3636973548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8114107820 0x7f811419c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.880+0000 7f8119f14640 1 --2- 172.21.15.107:0/3636973548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81141021d0 0x7f81141985c0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.880+0000 7f8119f14640 1 --2- 172.21.15.107:0/3636973548 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8114101810 0x7f8114198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:18.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.880+0000 7f8119f14640 1 -- 172.21.15.107:0/3636973548 >> 172.21.15.107:0/3636973548 conn(0x7f8114078c30 msgr2=0x7f81141043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:18.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.880+0000 7f8119f14640 1 -- 172.21.15.107:0/3636973548 shutdown_connections 2024-05-09T08:38:18.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:18.880+0000 7f8119f14640 1 -- 172.21.15.107:0/3636973548 wait complete. 2024-05-09T08:38:18.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:38:19.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.011+0000 7f5e1371d640 1 -- 172.21.15.107:0/1853259093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c069b60 msgr2=0x7f5e0c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.011+0000 7f5e1371d640 1 --2- 172.21.15.107:0/1853259093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c069b60 0x7f5e0c069fe0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc009900 tx=0x7f5dfc02a410 comp rx=0 tx=0).stop 2024-05-09T08:38:19.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.011+0000 7f5e1371d640 1 -- 172.21.15.107:0/1853259093 shutdown_connections 2024-05-09T08:38:19.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.011+0000 7f5e1371d640 1 --2- 172.21.15.107:0/1853259093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e0c06a520 0x7f5e0c10e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.011+0000 7f5e1371d640 1 --2- 172.21.15.107:0/1853259093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c069b60 0x7f5e0c069fe0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.011+0000 7f5e1371d640 1 --2- 172.21.15.107:0/1853259093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e0c107880 0x7f5e0c107c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.011+0000 7f5e1371d640 1 -- 172.21.15.107:0/1853259093 >> 172.21.15.107:0/1853259093 conn(0x7f5e0c0762b0 msgr2=0x7f5e0c0766c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.011+0000 7f5e1371d640 1 -- 172.21.15.107:0/1853259093 shutdown_connections 2024-05-09T08:38:19.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.011+0000 7f5e1371d640 1 -- 172.21.15.107:0/1853259093 wait complete. 2024-05-09T08:38:19.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.011+0000 7f5e1371d640 1 Processor -- start 2024-05-09T08:38:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e1371d640 1 -- start start 2024-05-09T08:38:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e1371d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c069b60 0x7f5e0c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e1371d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e0c06a520 0x7f5e0c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e1371d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e0c107880 0x7f5e0c19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e1371d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c111ac0 con 0x7f5e0c069b60 2024-05-09T08:38:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e1371d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c111940 con 0x7f5e0c107880 2024-05-09T08:38:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e1371d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c111c40 con 0x7f5e0c06a520 2024-05-09T08:38:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e114b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c069b60 0x7f5e0c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e114b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c069b60 0x7f5e0c198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46552/0 (socket says 172.21.15.107:46552) 2024-05-09T08:38:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e114b9640 1 -- 172.21.15.107:0/3362807790 learned_addr learned my addr 172.21.15.107:0/3362807790 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e11cba640 1 --2- 172.21.15.107:0/3362807790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e0c107880 0x7f5e0c19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e10cb8640 1 --2- 172.21.15.107:0/3362807790 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e0c06a520 0x7f5e0c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e114b9640 1 -- 172.21.15.107:0/3362807790 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e0c06a520 msgr2=0x7f5e0c198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e114b9640 1 --2- 172.21.15.107:0/3362807790 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e0c06a520 0x7f5e0c198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e114b9640 1 -- 172.21.15.107:0/3362807790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e0c107880 msgr2=0x7f5e0c19c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e114b9640 1 --2- 172.21.15.107:0/3362807790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e0c107880 0x7f5e0c19c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e114b9640 1 -- 172.21.15.107:0/3362807790 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e0c19d010 con 0x7f5e0c069b60 2024-05-09T08:38:19.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.012+0000 7f5e114b9640 1 --2- 172.21.15.107:0/3362807790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c069b60 0x7f5e0c198020 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f5e0800ea60 tx=0x7f5e0800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.013+0000 7f5e027fc640 1 -- 172.21.15.107:0/3362807790 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0800cde0 con 0x7f5e0c069b60 2024-05-09T08:38:19.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.013+0000 7f5e027fc640 1 -- 172.21.15.107:0/3362807790 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e080035f0 con 0x7f5e0c069b60 2024-05-09T08:38:19.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.013+0000 7f5e1371d640 1 -- 172.21.15.107:0/3362807790 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e0c19d300 con 0x7f5e0c069b60 2024-05-09T08:38:19.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.013+0000 7f5e027fc640 1 -- 172.21.15.107:0/3362807790 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08010690 con 0x7f5e0c069b60 2024-05-09T08:38:19.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.013+0000 7f5e1371d640 1 -- 172.21.15.107:0/3362807790 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e0c1a4b90 con 0x7f5e0c069b60 2024-05-09T08:38:19.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.015+0000 7f5e027fc640 1 -- 172.21.15.107:0/3362807790 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f5e080037a0 con 0x7f5e0c069b60 2024-05-09T08:38:19.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.016+0000 7f5e1641f740 1 Processor -- start 2024-05-09T08:38:19.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.016+0000 7f5e1641f740 1 -- start start 2024-05-09T08:38:19.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.016+0000 7f5e1641f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d656710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.016+0000 7f5e1641f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 0x56175d660de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.016+0000 7f5e1641f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d661320 0x56175d663710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.016+0000 7f5e1641f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56175d5bbc40 con 0x56175d656330 2024-05-09T08:38:19.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.016+0000 7f5e027fc640 1 --2- 172.21.15.107:0/3362807790 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5df80a2d00 0x7f5df80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.016+0000 7f5e1641f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56175d563580 con 0x56175d656c50 2024-05-09T08:38:19.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.016+0000 7f5e1641f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56175d5b7c00 con 0x56175d661320 2024-05-09T08:38:19.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.016+0000 7f5e11cba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d656710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.016+0000 7f5e11cba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d656710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46562/0 (socket says 172.21.15.107:46562) 2024-05-09T08:38:19.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.016+0000 7f5e11cba640 1 -- 172.21.15.107:0/3776530651 learned_addr learned my addr 172.21.15.107:0/3776530651 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:19.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.016+0000 7f5e11cba640 1 --2- 172.21.15.107:0/3362807790 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5df80a2d00 0x7f5df80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.017+0000 7f5e027fc640 1 -- 172.21.15.107:0/3362807790 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f5e080cfae0 con 0x7f5e0c069b60 2024-05-09T08:38:19.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.017+0000 7f5e114b9640 1 --2- 172.21.15.107:0/3776530651 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 0x56175d660de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.017+0000 7f5e10cb8640 1 --2- 172.21.15.107:0/3776530651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d661320 0x56175d663710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.017+0000 7f5e11cba640 1 -- 172.21.15.107:0/3776530651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d661320 msgr2=0x56175d663710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.017+0000 7f5e11cba640 1 --2- 172.21.15.107:0/3776530651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d661320 0x56175d663710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.017+0000 7f5e11cba640 1 -- 172.21.15.107:0/3776530651 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 msgr2=0x56175d660de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.017+0000 7f5e11cba640 1 --2- 172.21.15.107:0/3776530651 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 0x56175d660de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.017+0000 7f5e11cba640 1 -- 172.21.15.107:0/3776530651 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56175d2cf3f0 con 0x56175d656330 2024-05-09T08:38:19.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.017+0000 7f5e114b9640 1 --2- 172.21.15.107:0/3776530651 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 0x56175d660de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:19.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.017+0000 7f5e10cb8640 1 --2- 172.21.15.107:0/3776530651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d661320 0x56175d663710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:38:19.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.017+0000 7f5e11cba640 1 --2- 172.21.15.107:0/3776530651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d656710 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5e04012da0 tx=0x7f5e040107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a0e627804a6f6f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.017+0000 7f5e11cba640 1 --2- 172.21.15.107:0/3362807790 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5df80a2d00 0x7f5df80a51c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5e04005c40 tx=0x7f5e0401d000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.017+0000 7f5df77fe640 1 -- 172.21.15.107:0/3776530651 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0402a070 con 0x56175d656330 2024-05-09T08:38:19.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.017+0000 7f5df77fe640 1 -- 172.21.15.107:0/3776530651 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e04009dc0 con 0x56175d656330 2024-05-09T08:38:19.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.017+0000 7f5df77fe640 1 -- 172.21.15.107:0/3776530651 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e04004290 con 0x56175d656330 2024-05-09T08:38:19.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.018+0000 7f5e1641f740 1 -- 172.21.15.107:0/3776530651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 msgr2=0x56175d656710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.018+0000 7f5e1641f740 1 --2- 172.21.15.107:0/3776530651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d656710 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5e04012da0 tx=0x7f5e040107a0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.018+0000 7f5e1641f740 1 -- 172.21.15.107:0/3776530651 shutdown_connections 2024-05-09T08:38:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.018+0000 7f5e1641f740 1 --2- 172.21.15.107:0/3776530651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d661320 0x56175d663710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.018+0000 7f5e1641f740 1 --2- 172.21.15.107:0/3776530651 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 0x56175d660de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.018+0000 7f5e1641f740 1 --2- 172.21.15.107:0/3776530651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d656710 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.018+0000 7f5e1641f740 1 -- 172.21.15.107:0/3776530651 >> 172.21.15.107:0/3776530651 conn(0x56175d652650 msgr2=0x56175d652a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.018+0000 7f5e1641f740 1 -- 172.21.15.107:0/3776530651 shutdown_connections 2024-05-09T08:38:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.018+0000 7f5e1641f740 1 -- 172.21.15.107:0/3776530651 wait complete. 2024-05-09T08:38:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.018+0000 7f5e1641f740 1 -- start start 2024-05-09T08:38:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e1641f740 1 Processor -- start 2024-05-09T08:38:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e1641f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d65e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e1641f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 0x56175d65e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e1641f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d661320 0x56175d65ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e1641f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56175d6640b0 con 0x56175d656330 2024-05-09T08:38:19.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e1641f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56175d576c40 con 0x56175d656c50 2024-05-09T08:38:19.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e1641f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56175d664230 con 0x56175d661320 2024-05-09T08:38:19.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e114b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d65e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e114b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d65e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46564/0 (socket says 172.21.15.107:46564) 2024-05-09T08:38:19.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e114b9640 1 -- 172.21.15.107:0/2027530424 learned_addr learned my addr 172.21.15.107:0/2027530424 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:19.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e1641f740 10 client.0 _ll_register_callbacks cb 0x56175d2cebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:19.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e11cba640 1 --2- 172.21.15.107:0/2027530424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d661320 0x56175d65ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e10cb8640 1 --2- 172.21.15.107:0/2027530424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 0x56175d65e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e114b9640 1 -- 172.21.15.107:0/2027530424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d661320 msgr2=0x56175d65ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e114b9640 1 --2- 172.21.15.107:0/2027530424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d661320 0x56175d65ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e114b9640 1 -- 172.21.15.107:0/2027530424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 msgr2=0x56175d65e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e114b9640 1 --2- 172.21.15.107:0/2027530424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 0x56175d65e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e114b9640 1 -- 172.21.15.107:0/2027530424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56175d3997d0 con 0x56175d656330 2024-05-09T08:38:19.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.019+0000 7f5e10cb8640 1 --2- 172.21.15.107:0/2027530424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 0x56175d65e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:38:19.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.020+0000 7f5e114b9640 1 --2- 172.21.15.107:0/2027530424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d65e2e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f5e08098670 tx=0x7f5e0809bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f105deebace90bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.020+0000 7f5df4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:19.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.020+0000 7f5df4ff9640 1 -- 172.21.15.107:0/2027530424 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0809c3a0 con 0x56175d656330 2024-05-09T08:38:19.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.020+0000 7f5df4ff9640 1 -- 172.21.15.107:0/2027530424 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f5e080cd150 con 0x56175d656330 2024-05-09T08:38:19.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.020+0000 7f5e1641f740 1 -- 172.21.15.107:0/2027530424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e04009dc0 con 0x56175d656330 2024-05-09T08:38:19.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.020+0000 7f5df4ff9640 1 -- 172.21.15.107:0/2027530424 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5e0809bca0 con 0x56175d656330 2024-05-09T08:38:19.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.020+0000 7f5e1641f740 10 client.17103 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:19.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.020+0000 7f5e1641f740 1 -- 172.21.15.107:0/2027530424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56175d670260 con 0x56175d656330 2024-05-09T08:38:19.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.020+0000 7f5df4ff9640 1 -- 172.21.15.107:0/2027530424 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e04009dc0 con 0x56175d656330 2024-05-09T08:38:19.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.020+0000 7f5df4ff9640 1 -- 172.21.15.107:0/2027530424 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5e08098310 con 0x56175d656330 2024-05-09T08:38:19.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.021+0000 7f5e1641f740 10 client.17103 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:19.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.021+0000 7f5e1641f740 10 client.17103 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:38:19.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.021+0000 7f5e1641f740 20 client.17103 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:19.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.021+0000 7f5e1641f740 1 --2- 172.21.15.107:0/2027530424 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x56175d6717c0 0x56175d673c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.021+0000 7f5e1641f740 4 client.17103 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:19.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.021+0000 7f5e1641f740 1 -- 172.21.15.107:0/2027530424 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56175d5c9370 con 0x56175d6717c0 2024-05-09T08:38:19.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.021+0000 7f5e10cb8640 1 --2- 172.21.15.107:0/2027530424 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x56175d6717c0 0x56175d673c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.022+0000 7f5df4ff9640 10 client.17103 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:38:19.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.022+0000 7f5e10cb8640 1 --2- 172.21.15.107:0/2027530424 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x56175d6717c0 0x56175d673c30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=551a9190322c632a server_cookie=2ecf275009000070 in_seq=0 out_seq=0 2024-05-09T08:38:19.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.022+0000 7f5df4ff9640 1 -- 172.21.15.107:0/2027530424 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56175d5c9370 con 0x56175d6717c0 2024-05-09T08:38:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.022+0000 7f5df4ff9640 10 client.17103 handle_command_reply: tid=0 2024-05-09T08:38:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.022+0000 7f5e1641f740 1 client.17103 shutdown 2024-05-09T08:38:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.022+0000 7f5e1641f740 1 -- 172.21.15.107:0/2027530424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 msgr2=0x56175d65e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.022+0000 7f5e1641f740 1 --2- 172.21.15.107:0/2027530424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d65e2e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f5e08098670 tx=0x7f5e0809bb30 comp rx=0 tx=0).stop 2024-05-09T08:38:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.022+0000 7f5e1641f740 1 -- 172.21.15.107:0/2027530424 shutdown_connections 2024-05-09T08:38:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.022+0000 7f5e1641f740 1 --2- 172.21.15.107:0/2027530424 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x56175d6717c0 0x56175d673c30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.022+0000 7f5e1641f740 1 --2- 172.21.15.107:0/2027530424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d661320 0x56175d65ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.022+0000 7f5e1641f740 1 --2- 172.21.15.107:0/2027530424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 0x56175d65e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.022+0000 7f5df4ff9640 0 client.17103 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:38:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.022+0000 7f5e1641f740 1 --2- 172.21.15.107:0/2027530424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d65e2e0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.023+0000 7f5e1641f740 1 -- 172.21.15.107:0/2027530424 >> 172.21.15.107:0/2027530424 conn(0x56175d652650 msgr2=0x56175d658350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.023+0000 7f5e1641f740 1 -- 172.21.15.107:0/2027530424 shutdown_connections 2024-05-09T08:38:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.023+0000 7f5e1641f740 1 -- 172.21.15.107:0/2027530424 wait complete. 2024-05-09T08:38:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.023+0000 7f5e1641f740 20 client.17103 trim_cache size 0 max 16384 2024-05-09T08:38:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.043+0000 7f5e1641f740 1 Processor -- start 2024-05-09T08:38:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.043+0000 7f5e1641f740 1 -- start start 2024-05-09T08:38:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.043+0000 7f5e1641f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d656330 0x56175d66cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.043+0000 7f5e1641f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 0x56175d651630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.043+0000 7f5e1641f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d661320 0x56175d6794d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.043+0000 7f5e1641f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c1b1420 con 0x56175d661320 2024-05-09T08:38:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.043+0000 7f5e1641f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c1aecf0 con 0x56175d656c50 2024-05-09T08:38:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.043+0000 7f5e1641f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c1acb40 con 0x56175d656330 2024-05-09T08:38:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.043+0000 7f5e11cba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d661320 0x56175d6794d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.043+0000 7f5e11cba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d661320 0x56175d6794d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46580/0 (socket says 172.21.15.107:46580) 2024-05-09T08:38:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.043+0000 7f5e11cba640 1 -- 172.21.15.107:0/1658062595 learned_addr learned my addr 172.21.15.107:0/1658062595 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.044+0000 7f5e10cb8640 1 --2- 172.21.15.107:0/1658062595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 0x56175d651630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.044+0000 7f5e114b9640 1 --2- 172.21.15.107:0/1658062595 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d656330 0x56175d66cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.044+0000 7f5e11cba640 1 -- 172.21.15.107:0/1658062595 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d656330 msgr2=0x56175d66cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.044+0000 7f5e11cba640 1 --2- 172.21.15.107:0/1658062595 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d656330 0x56175d66cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.044+0000 7f5e11cba640 1 -- 172.21.15.107:0/1658062595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 msgr2=0x56175d651630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.044+0000 7f5e11cba640 1 --2- 172.21.15.107:0/1658062595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 0x56175d651630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.044+0000 7f5e11cba640 1 -- 172.21.15.107:0/1658062595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56175d656130 con 0x56175d661320 2024-05-09T08:38:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.044+0000 7f5e10cb8640 1 --2- 172.21.15.107:0/1658062595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 0x56175d651630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:38:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.044+0000 7f5e114b9640 1 --2- 172.21.15.107:0/1658062595 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d656330 0x56175d66cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:38:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.044+0000 7f5e11cba640 1 --2- 172.21.15.107:0/1658062595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d661320 0x56175d6794d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f5e04010ca0 tx=0x7f5e04005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f308fc6dd9d07e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.044+0000 7f5df6ffd640 1 -- 172.21.15.107:0/1658062595 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e04033070 con 0x56175d661320 2024-05-09T08:38:19.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.044+0000 7f5df6ffd640 1 -- 172.21.15.107:0/1658062595 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e04003e20 con 0x56175d661320 2024-05-09T08:38:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.044+0000 7f5df6ffd640 1 -- 172.21.15.107:0/1658062595 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0402a040 con 0x56175d661320 2024-05-09T08:38:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.045+0000 7f5e1641f740 1 -- 172.21.15.107:0/1658062595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d661320 msgr2=0x56175d6794d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.045+0000 7f5e1641f740 1 --2- 172.21.15.107:0/1658062595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d661320 0x56175d6794d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f5e04010ca0 tx=0x7f5e04005df0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.045+0000 7f5e1641f740 1 -- 172.21.15.107:0/1658062595 shutdown_connections 2024-05-09T08:38:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.045+0000 7f5e1641f740 1 --2- 172.21.15.107:0/1658062595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d661320 0x56175d6794d0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.045+0000 7f5e1641f740 1 --2- 172.21.15.107:0/1658062595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d656c50 0x56175d651630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.045+0000 7f5e1641f740 1 --2- 172.21.15.107:0/1658062595 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d656330 0x56175d66cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.045+0000 7f5e1641f740 1 -- 172.21.15.107:0/1658062595 >> 172.21.15.107:0/1658062595 conn(0x56175d652650 msgr2=0x56175d650f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.045+0000 7f5e1641f740 1 -- 172.21.15.107:0/1658062595 shutdown_connections 2024-05-09T08:38:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.045+0000 7f5e1641f740 1 -- 172.21.15.107:0/1658062595 wait complete. 2024-05-09T08:38:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.045+0000 7f5e1641f740 1 -- start start 2024-05-09T08:38:19.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.045+0000 7f5e1641f740 1 Processor -- start 2024-05-09T08:38:19.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e1641f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d664eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e1641f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d656c50 0x56175d6653f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e1641f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d661320 0x56175d665930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e1641f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c1ad1b0 con 0x56175d656330 2024-05-09T08:38:19.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e1641f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c1ae620 con 0x56175d661320 2024-05-09T08:38:19.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e1641f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c1ad880 con 0x56175d656c50 2024-05-09T08:38:19.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e1641f740 10 client.0 _ll_register_callbacks cb 0x7f5e0800e9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:19.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e114b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d664eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e114b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d664eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46582/0 (socket says 172.21.15.107:46582) 2024-05-09T08:38:19.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e114b9640 1 -- 172.21.15.107:0/102417636 learned_addr learned my addr 172.21.15.107:0/102417636 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:19.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e10cb8640 1 --2- 172.21.15.107:0/102417636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d656c50 0x56175d6653f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e11cba640 1 --2- 172.21.15.107:0/102417636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d661320 0x56175d665930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e114b9640 1 -- 172.21.15.107:0/102417636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d656c50 msgr2=0x56175d6653f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e114b9640 1 --2- 172.21.15.107:0/102417636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d656c50 0x56175d6653f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e114b9640 1 -- 172.21.15.107:0/102417636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d661320 msgr2=0x56175d665930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e114b9640 1 --2- 172.21.15.107:0/102417636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d661320 0x56175d665930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e114b9640 1 -- 172.21.15.107:0/102417636 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56175d6d6d70 con 0x56175d656330 2024-05-09T08:38:19.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5e114b9640 1 --2- 172.21.15.107:0/102417636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d664eb0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f5e0801fd20 tx=0x7f5e08097e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffc8b6e1259e1a31 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5df4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:19.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.046+0000 7f5df4ff9640 1 -- 172.21.15.107:0/102417636 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e080979f0 con 0x56175d656330 2024-05-09T08:38:19.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.047+0000 7f5df4ff9640 1 -- 172.21.15.107:0/102417636 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f5e08098310 con 0x56175d656330 2024-05-09T08:38:19.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.047+0000 7f5e1641f740 1 -- 172.21.15.107:0/102417636 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e04003e20 con 0x56175d656330 2024-05-09T08:38:19.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.047+0000 7f5df4ff9640 1 -- 172.21.15.107:0/102417636 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5e0809b600 con 0x56175d656330 2024-05-09T08:38:19.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.047+0000 7f5e1641f740 10 client.17109 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:19.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.047+0000 7f5e1641f740 1 -- 172.21.15.107:0/102417636 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56175d6d7fa0 con 0x56175d656330 2024-05-09T08:38:19.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.047+0000 7f5df4ff9640 1 -- 172.21.15.107:0/102417636 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e04003e20 con 0x56175d656330 2024-05-09T08:38:19.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.047+0000 7f5df4ff9640 1 -- 172.21.15.107:0/102417636 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5e080a4b60 con 0x56175d656330 2024-05-09T08:38:19.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.047+0000 7f5e1641f740 10 client.17109 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:19.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.047+0000 7f5e1641f740 10 client.17109 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:38:19.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.047+0000 7f5e1641f740 20 client.17109 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:19.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.047+0000 7f5e1641f740 1 --2- 172.21.15.107:0/102417636 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x56175d6717c0 0x56175d6da490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.047+0000 7f5e1641f740 4 client.17109 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:38:19.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.047+0000 7f5e1641f740 1 -- 172.21.15.107:0/102417636 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56175d5c36c0 con 0x56175d6717c0 2024-05-09T08:38:19.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.047+0000 7f5e10cb8640 1 --2- 172.21.15.107:0/102417636 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x56175d6717c0 0x56175d6da490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5e10cb8640 1 --2- 172.21.15.107:0/102417636 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x56175d6717c0 0x56175d6da490 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dec9d9fa30e7a6ed server_cookie=bdbe19681ff6a187 in_seq=0 out_seq=0 2024-05-09T08:38:19.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5df4ff9640 10 client.17109 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:38:19.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5df4ff9640 1 -- 172.21.15.107:0/102417636 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56175d5c36c0 con 0x56175d6717c0 2024-05-09T08:38:19.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5df4ff9640 10 client.17109 handle_command_reply: tid=0 2024-05-09T08:38:19.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5e1641f740 1 client.17109 shutdown 2024-05-09T08:38:19.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5e1641f740 1 -- 172.21.15.107:0/102417636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 msgr2=0x56175d664eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5e1641f740 1 --2- 172.21.15.107:0/102417636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d664eb0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f5e0801fd20 tx=0x7f5e08097e10 comp rx=0 tx=0).stop 2024-05-09T08:38:19.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5e1641f740 1 -- 172.21.15.107:0/102417636 shutdown_connections 2024-05-09T08:38:19.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5e1641f740 1 --2- 172.21.15.107:0/102417636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56175d661320 0x56175d665930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5e1641f740 1 --2- 172.21.15.107:0/102417636 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x56175d6717c0 0x56175d6da490 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5e1641f740 1 --2- 172.21.15.107:0/102417636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56175d656c50 0x56175d6653f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5e1641f740 1 --2- 172.21.15.107:0/102417636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56175d656330 0x56175d664eb0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5df4ff9640 0 client.17109 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:38:19.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5e1641f740 1 -- 172.21.15.107:0/102417636 >> 172.21.15.107:0/102417636 conn(0x56175d652650 msgr2=0x56175d6517b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5e1641f740 1 -- 172.21.15.107:0/102417636 shutdown_connections 2024-05-09T08:38:19.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5e1641f740 1 -- 172.21.15.107:0/102417636 wait complete. 2024-05-09T08:38:19.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.048+0000 7f5e1641f740 20 client.17109 trim_cache size 0 max 16384 2024-05-09T08:38:19.055 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:38:19.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.049+0000 7f5e1371d640 1 -- 172.21.15.107:0/3362807790 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5df80a2d00 msgr2=0x7f5df80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.049+0000 7f5e1371d640 1 --2- 172.21.15.107:0/3362807790 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5df80a2d00 0x7f5df80a51c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5e04005c40 tx=0x7f5e0401d000 comp rx=0 tx=0).stop 2024-05-09T08:38:19.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.049+0000 7f5e1371d640 1 -- 172.21.15.107:0/3362807790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c069b60 msgr2=0x7f5e0c198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.049+0000 7f5e1371d640 1 --2- 172.21.15.107:0/3362807790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c069b60 0x7f5e0c198020 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f5e0800ea60 tx=0x7f5e0800ef20 comp rx=0 tx=0).stop 2024-05-09T08:38:19.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.049+0000 7f5e1371d640 1 -- 172.21.15.107:0/3362807790 shutdown_connections 2024-05-09T08:38:19.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.049+0000 7f5e1371d640 1 --2- 172.21.15.107:0/3362807790 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5df80a2d00 0x7f5df80a51c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.049+0000 7f5e1371d640 1 --2- 172.21.15.107:0/3362807790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e0c107880 0x7f5e0c19c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.049+0000 7f5e1371d640 1 --2- 172.21.15.107:0/3362807790 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e0c06a520 0x7f5e0c198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.049+0000 7f5e1371d640 1 --2- 172.21.15.107:0/3362807790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e0c069b60 0x7f5e0c198020 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.049+0000 7f5e1371d640 1 -- 172.21.15.107:0/3362807790 >> 172.21.15.107:0/3362807790 conn(0x7f5e0c0762b0 msgr2=0x7f5e0c0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.049+0000 7f5e1371d640 1 -- 172.21.15.107:0/3362807790 shutdown_connections 2024-05-09T08:38:19.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.049+0000 7f5e1371d640 1 -- 172.21.15.107:0/3362807790 wait complete. 2024-05-09T08:38:19.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:38:19.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.177+0000 7fe6f0f09640 1 -- 172.21.15.107:0/1986644730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6ec100640 msgr2=0x7fe6ec100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.177+0000 7fe6f0f09640 1 --2- 172.21.15.107:0/1986644730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6ec100640 0x7fe6ec100a20 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc009900 tx=0x7fe6dc02a410 comp rx=0 tx=0).stop 2024-05-09T08:38:19.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.177+0000 7fe6f0f09640 1 -- 172.21.15.107:0/1986644730 shutdown_connections 2024-05-09T08:38:19.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.177+0000 7fe6f0f09640 1 --2- 172.21.15.107:0/1986644730 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6ec10e1f0 0x7fe6ec1105e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.177+0000 7fe6f0f09640 1 --2- 172.21.15.107:0/1986644730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6ec100f60 0x7fe6ec10dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.177+0000 7fe6f0f09640 1 --2- 172.21.15.107:0/1986644730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6ec100640 0x7fe6ec100a20 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.177+0000 7fe6f0f09640 1 -- 172.21.15.107:0/1986644730 >> 172.21.15.107:0/1986644730 conn(0x7fe6ec076ab0 msgr2=0x7fe6ec076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.177+0000 7fe6f0f09640 1 -- 172.21.15.107:0/1986644730 shutdown_connections 2024-05-09T08:38:19.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.177+0000 7fe6f0f09640 1 -- 172.21.15.107:0/1986644730 wait complete. 2024-05-09T08:38:19.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6f0f09640 1 Processor -- start 2024-05-09T08:38:19.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6f0f09640 1 -- start start 2024-05-09T08:38:19.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6f0f09640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6ec100640 0x7fe6ec19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6f0f09640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6ec100f60 0x7fe6ec19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6f0f09640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6ec10e1f0 0x7fe6ec19eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6f0f09640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec113c90 con 0x7fe6ec100f60 2024-05-09T08:38:19.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6f0f09640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec113b10 con 0x7fe6ec10e1f0 2024-05-09T08:38:19.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6f0f09640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec113e10 con 0x7fe6ec100640 2024-05-09T08:38:19.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6ec100f60 0x7fe6ec19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6ec100f60 0x7fe6ec19a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46606/0 (socket says 172.21.15.107:46606) 2024-05-09T08:38:19.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6e3fff640 1 -- 172.21.15.107:0/3694187887 learned_addr learned my addr 172.21.15.107:0/3694187887 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:19.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6ea59c640 1 --2- 172.21.15.107:0/3694187887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6ec100640 0x7fe6ec19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6ead9d640 1 --2- 172.21.15.107:0/3694187887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6ec10e1f0 0x7fe6ec19eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6e3fff640 1 -- 172.21.15.107:0/3694187887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6ec100640 msgr2=0x7fe6ec19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6e3fff640 1 --2- 172.21.15.107:0/3694187887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6ec100640 0x7fe6ec19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6e3fff640 1 -- 172.21.15.107:0/3694187887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6ec10e1f0 msgr2=0x7fe6ec19eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6e3fff640 1 --2- 172.21.15.107:0/3694187887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6ec10e1f0 0x7fe6ec19eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6e3fff640 1 -- 172.21.15.107:0/3694187887 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ec19f240 con 0x7fe6ec100f60 2024-05-09T08:38:19.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.178+0000 7fe6e3fff640 1 --2- 172.21.15.107:0/3694187887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6ec100f60 0x7fe6ec19a790 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fe6d400c890 tx=0x7fe6d400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.179+0000 7fe6e37fe640 1 -- 172.21.15.107:0/3694187887 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4004220 con 0x7fe6ec100f60 2024-05-09T08:38:19.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.179+0000 7fe6f0f09640 1 -- 172.21.15.107:0/3694187887 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6ec19f530 con 0x7fe6ec100f60 2024-05-09T08:38:19.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.179+0000 7fe6e37fe640 1 -- 172.21.15.107:0/3694187887 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6d40043c0 con 0x7fe6ec100f60 2024-05-09T08:38:19.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.179+0000 7fe6e37fe640 1 -- 172.21.15.107:0/3694187887 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4004a00 con 0x7fe6ec100f60 2024-05-09T08:38:19.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.179+0000 7fe6f0f09640 1 -- 172.21.15.107:0/3694187887 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6ec1a6dc0 con 0x7fe6ec100f60 2024-05-09T08:38:19.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.181+0000 7fe6e37fe640 1 -- 172.21.15.107:0/3694187887 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe6d4004ba0 con 0x7fe6ec100f60 2024-05-09T08:38:19.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.181+0000 7fe6f3c0e740 1 Processor -- start 2024-05-09T08:38:19.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.182+0000 7fe6f3c0e740 1 -- start start 2024-05-09T08:38:19.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.182+0000 7fe6f3c0e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9b4330 0x55f3ff9b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.182+0000 7fe6f3c0e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.182+0000 7fe6f3c0e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9bf320 0x55f3ff9c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.182+0000 7fe6f3c0e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f3ff919c40 con 0x55f3ff9b4c50 2024-05-09T08:38:19.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.182+0000 7fe6f3c0e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f3ff8c1580 con 0x55f3ff9bf320 2024-05-09T08:38:19.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.182+0000 7fe6f3c0e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f3ff915c00 con 0x55f3ff9b4330 2024-05-09T08:38:19.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.182+0000 7fe6ea59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.182+0000 7fe6ea59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46608/0 (socket says 172.21.15.107:46608) 2024-05-09T08:38:19.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.182+0000 7fe6ea59c640 1 -- 172.21.15.107:0/59765 learned_addr learned my addr 172.21.15.107:0/59765 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:19.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.182+0000 7fe6ead9d640 1 --2- 172.21.15.107:0/59765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9b4330 0x55f3ff9b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.182+0000 7fe6e3fff640 1 --2- 172.21.15.107:0/59765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9bf320 0x55f3ff9c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.182+0000 7fe6e37fe640 1 --2- 172.21.15.107:0/3694187887 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6d80a2d00 0x7fe6d80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.183+0000 7fe6ea59c640 1 -- 172.21.15.107:0/59765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9b4330 msgr2=0x55f3ff9b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.183+0000 7fe6ea59c640 1 --2- 172.21.15.107:0/59765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9b4330 0x55f3ff9b4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.183+0000 7fe6ea59c640 1 -- 172.21.15.107:0/59765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9bf320 msgr2=0x55f3ff9c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.183+0000 7fe6ea59c640 1 --2- 172.21.15.107:0/59765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9bf320 0x55f3ff9c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.183+0000 7fe6ea59c640 1 -- 172.21.15.107:0/59765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3ff62d3f0 con 0x55f3ff9b4c50 2024-05-09T08:38:19.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.183+0000 7fe6ead9d640 1 --2- 172.21.15.107:0/59765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9b4330 0x55f3ff9b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:19.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.183+0000 7fe6e37fe640 1 -- 172.21.15.107:0/3694187887 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe6d40cf470 con 0x7fe6ec100f60 2024-05-09T08:38:19.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.183+0000 7fe6ead9d640 1 --2- 172.21.15.107:0/3694187887 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6d80a2d00 0x7fe6d80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.183+0000 7fe6ea59c640 1 --2- 172.21.15.107:0/59765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bede0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc009900 tx=0x7fe6dc032e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc8f02902fd17f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.183+0000 7fe6ead9d640 1 --2- 172.21.15.107:0/3694187887 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6d80a2d00 0x7fe6d80a51c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec069bd0 tx=0x7fe6e4004a40 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.183+0000 7fe6e0ff9640 1 -- 172.21.15.107:0/59765 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc040070 con 0x55f3ff9b4c50 2024-05-09T08:38:19.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.183+0000 7fe6e0ff9640 1 -- 172.21.15.107:0/59765 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6dc0316a0 con 0x55f3ff9b4c50 2024-05-09T08:38:19.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.183+0000 7fe6e0ff9640 1 -- 172.21.15.107:0/59765 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc035040 con 0x55f3ff9b4c50 2024-05-09T08:38:19.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.184+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/59765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4c50 msgr2=0x55f3ff9bede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.184+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/59765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bede0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc009900 tx=0x7fe6dc032e00 comp rx=0 tx=0).stop 2024-05-09T08:38:19.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.184+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/59765 shutdown_connections 2024-05-09T08:38:19.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.184+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/59765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9bf320 0x55f3ff9c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.184+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/59765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bede0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.184+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/59765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9b4330 0x55f3ff9b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.184+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/59765 >> 172.21.15.107:0/59765 conn(0x55f3ff9b0650 msgr2=0x55f3ff9b0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.184+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/59765 shutdown_connections 2024-05-09T08:38:19.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.184+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/59765 wait complete. 2024-05-09T08:38:19.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.184+0000 7fe6f3c0e740 1 -- start start 2024-05-09T08:38:19.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.184+0000 7fe6f3c0e740 1 Processor -- start 2024-05-09T08:38:19.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6f3c0e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4330 0x55f3ff9bc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6f3c0e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6f3c0e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9bf320 0x55f3ff9bccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6f3c0e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f3ff9c20b0 con 0x55f3ff9b4330 2024-05-09T08:38:19.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6f3c0e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f3ff8d4c40 con 0x55f3ff9b4c50 2024-05-09T08:38:19.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6f3c0e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f3ff9c2230 con 0x55f3ff9bf320 2024-05-09T08:38:19.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6f3c0e740 10 client.0 _ll_register_callbacks cb 0x55f3ff62cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:19.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6ea59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4330 0x55f3ff9bc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6ea59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4330 0x55f3ff9bc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46612/0 (socket says 172.21.15.107:46612) 2024-05-09T08:38:19.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6ea59c640 1 -- 172.21.15.107:0/2664748769 learned_addr learned my addr 172.21.15.107:0/2664748769 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:19.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6e3fff640 1 --2- 172.21.15.107:0/2664748769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6ead9d640 1 --2- 172.21.15.107:0/2664748769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9bf320 0x55f3ff9bccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6ea59c640 1 -- 172.21.15.107:0/2664748769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9bf320 msgr2=0x55f3ff9bccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6ea59c640 1 --2- 172.21.15.107:0/2664748769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9bf320 0x55f3ff9bccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6ea59c640 1 -- 172.21.15.107:0/2664748769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9b4c50 msgr2=0x55f3ff9bc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6ea59c640 1 --2- 172.21.15.107:0/2664748769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6ea59c640 1 -- 172.21.15.107:0/2664748769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3ff6f77d0 con 0x55f3ff9b4330 2024-05-09T08:38:19.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6e3fff640 1 --2- 172.21.15.107:0/2664748769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:19.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6ea59c640 1 --2- 172.21.15.107:0/2664748769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4330 0x55f3ff9bc270 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc005c00 tx=0x7fe6dc005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218c783814200de6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.185+0000 7fe6d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:19.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.186+0000 7fe6d1ffb640 1 -- 172.21.15.107:0/2664748769 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc04c060 con 0x55f3ff9b4330 2024-05-09T08:38:19.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.186+0000 7fe6d1ffb640 1 -- 172.21.15.107:0/2664748769 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe6dc031ab0 con 0x55f3ff9b4330 2024-05-09T08:38:19.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.186+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/2664748769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6dc0316a0 con 0x55f3ff9b4330 2024-05-09T08:38:19.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.186+0000 7fe6d1ffb640 1 -- 172.21.15.107:0/2664748769 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe6dc053050 con 0x55f3ff9b4330 2024-05-09T08:38:19.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.186+0000 7fe6f3c0e740 10 client.17121 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:19.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.186+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/2664748769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f3ff9ce2a0 con 0x55f3ff9b4330 2024-05-09T08:38:19.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.186+0000 7fe6d1ffb640 1 -- 172.21.15.107:0/2664748769 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6dc0316a0 con 0x55f3ff9b4330 2024-05-09T08:38:19.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.186+0000 7fe6d1ffb640 1 -- 172.21.15.107:0/2664748769 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe6dc003a40 con 0x55f3ff9b4330 2024-05-09T08:38:19.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.186+0000 7fe6f3c0e740 10 client.17121 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:19.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.186+0000 7fe6f3c0e740 10 client.17121 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:19.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.186+0000 7fe6f3c0e740 20 client.17121 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:19.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.186+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/2664748769 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f3ff9cf5f0 0x55f3ff9d19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.186+0000 7fe6f3c0e740 4 client.17121 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:19.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.186+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/2664748769 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3ff927370 con 0x55f3ff9cf5f0 2024-05-09T08:38:19.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.187+0000 7fe6e3fff640 1 --2- 172.21.15.107:0/2664748769 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f3ff9cf5f0 0x55f3ff9d19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.187+0000 7fe6d1ffb640 10 client.17121 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:19.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.187+0000 7fe6e3fff640 1 --2- 172.21.15.107:0/2664748769 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f3ff9cf5f0 0x55f3ff9d19e0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3b23690637320e8 server_cookie=c3cc09ee02130220 in_seq=0 out_seq=0 2024-05-09T08:38:19.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.188+0000 7fe6d1ffb640 1 -- 172.21.15.107:0/2664748769 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f3ff927370 con 0x55f3ff9cf5f0 2024-05-09T08:38:19.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.188+0000 7fe6d1ffb640 10 client.17121 handle_command_reply: tid=0 2024-05-09T08:38:19.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.188+0000 7fe6f3c0e740 1 client.17121 shutdown 2024-05-09T08:38:19.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.188+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/2664748769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4330 msgr2=0x55f3ff9bc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.188+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/2664748769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4330 0x55f3ff9bc270 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc005c00 tx=0x7fe6dc005fd0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.188+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/2664748769 shutdown_connections 2024-05-09T08:38:19.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.188+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/2664748769 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f3ff9cf5f0 0x55f3ff9d19e0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.188+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/2664748769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9bf320 0x55f3ff9bccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.188+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/2664748769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.188+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/2664748769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4330 0x55f3ff9bc270 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.188+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/2664748769 >> 172.21.15.107:0/2664748769 conn(0x55f3ff9b0650 msgr2=0x55f3ff9af7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.188+0000 7fe6d1ffb640 0 client.17121 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:19.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.188+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/2664748769 shutdown_connections 2024-05-09T08:38:19.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.188+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/2664748769 wait complete. 2024-05-09T08:38:19.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.188+0000 7fe6f3c0e740 20 client.17121 trim_cache size 0 max 16384 2024-05-09T08:38:19.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.202+0000 7fe6f3c0e740 1 Processor -- start 2024-05-09T08:38:19.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.202+0000 7fe6f3c0e740 1 -- start start 2024-05-09T08:38:19.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.202+0000 7fe6f3c0e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9b4330 0x55f3ff9cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.202+0000 7fe6f3c0e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.202+0000 7fe6f3c0e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9bf320 0x55f3ff9d6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.202+0000 7fe6f3c0e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec1b35d0 con 0x55f3ff9b4c50 2024-05-09T08:38:19.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.202+0000 7fe6f3c0e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec1b0ea0 con 0x55f3ff9b4330 2024-05-09T08:38:19.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.202+0000 7fe6f3c0e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec1aecf0 con 0x55f3ff9bf320 2024-05-09T08:38:19.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.202+0000 7fe6e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.202+0000 7fe6e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46624/0 (socket says 172.21.15.107:46624) 2024-05-09T08:38:19.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.202+0000 7fe6e3fff640 1 -- 172.21.15.107:0/1412727551 learned_addr learned my addr 172.21.15.107:0/1412727551 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:19.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.203+0000 7fe6ead9d640 1 --2- 172.21.15.107:0/1412727551 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9bf320 0x55f3ff9d6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.203+0000 7fe6ea59c640 1 --2- 172.21.15.107:0/1412727551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9b4330 0x55f3ff9cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.203+0000 7fe6e3fff640 1 -- 172.21.15.107:0/1412727551 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9bf320 msgr2=0x55f3ff9d6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.203+0000 7fe6e3fff640 1 --2- 172.21.15.107:0/1412727551 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9bf320 0x55f3ff9d6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.203+0000 7fe6e3fff640 1 -- 172.21.15.107:0/1412727551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9b4330 msgr2=0x55f3ff9cafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.203+0000 7fe6e3fff640 1 --2- 172.21.15.107:0/1412727551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9b4330 0x55f3ff9cafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.203+0000 7fe6e3fff640 1 -- 172.21.15.107:0/1412727551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3ff9b4a80 con 0x55f3ff9b4c50 2024-05-09T08:38:19.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.203+0000 7fe6e3fff640 1 --2- 172.21.15.107:0/1412727551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bb3d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fe6d40cc8c0 tx=0x7fe6d40d19f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b2f73dbc4028623 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.203+0000 7fe6d3fff640 1 -- 172.21.15.107:0/1412727551 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d40b3070 con 0x55f3ff9b4c50 2024-05-09T08:38:19.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.203+0000 7fe6d3fff640 1 -- 172.21.15.107:0/1412727551 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6d409b630 con 0x55f3ff9b4c50 2024-05-09T08:38:19.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.203+0000 7fe6d3fff640 1 -- 172.21.15.107:0/1412727551 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d401d040 con 0x55f3ff9b4c50 2024-05-09T08:38:19.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.204+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/1412727551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4c50 msgr2=0x55f3ff9bb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.204+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/1412727551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bb3d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fe6d40cc8c0 tx=0x7fe6d40d19f0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.204+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/1412727551 shutdown_connections 2024-05-09T08:38:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.204+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/1412727551 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9bf320 0x55f3ff9d6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.204+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/1412727551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9bb3d0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.204+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/1412727551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9b4330 0x55f3ff9cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.204+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/1412727551 >> 172.21.15.107:0/1412727551 conn(0x55f3ff9b0650 msgr2=0x55f3ff9b0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.204+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/1412727551 shutdown_connections 2024-05-09T08:38:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.204+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/1412727551 wait complete. 2024-05-09T08:38:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.204+0000 7fe6f3c0e740 1 -- start start 2024-05-09T08:38:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.204+0000 7fe6f3c0e740 1 Processor -- start 2024-05-09T08:38:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6f3c0e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9b4330 0x55f3ff9c7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6f3c0e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6f3c0e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9bf320 0x55f3ff9c2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6f3c0e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec1af360 con 0x55f3ff9bf320 2024-05-09T08:38:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6f3c0e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec1b07d0 con 0x55f3ff9b4c50 2024-05-09T08:38:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6f3c0e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec1afa30 con 0x55f3ff9b4330 2024-05-09T08:38:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6f3c0e740 10 client.0 _ll_register_callbacks cb 0x7fe6d409b400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6ead9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9bf320 0x55f3ff9c2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6ead9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9bf320 0x55f3ff9c2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46630/0 (socket says 172.21.15.107:46630) 2024-05-09T08:38:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6ea59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9b4330 0x55f3ff9c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6ead9d640 1 -- 172.21.15.107:0/1698878644 learned_addr learned my addr 172.21.15.107:0/1698878644 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6e3fff640 1 --2- 172.21.15.107:0/1698878644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6ead9d640 1 -- 172.21.15.107:0/1698878644 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9b4330 msgr2=0x55f3ff9c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6ead9d640 1 --2- 172.21.15.107:0/1698878644 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9b4330 0x55f3ff9c7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6ead9d640 1 -- 172.21.15.107:0/1698878644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9b4c50 msgr2=0x55f3ff9c79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6ead9d640 1 --2- 172.21.15.107:0/1698878644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9c79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6ead9d640 1 -- 172.21.15.107:0/1698878644 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3ff9c32d0 con 0x55f3ff9bf320 2024-05-09T08:38:19.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6ead9d640 1 --2- 172.21.15.107:0/1698878644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9bf320 0x55f3ff9c2aa0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4013970 tx=0x7fe6e4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bba2440445de76d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:19.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6d1ffb640 1 -- 172.21.15.107:0/1698878644 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4020b70 con 0x55f3ff9bf320 2024-05-09T08:38:19.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6d1ffb640 1 -- 172.21.15.107:0/1698878644 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe6e4020d10 con 0x55f3ff9bf320 2024-05-09T08:38:19.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.205+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/1698878644 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6d409b630 con 0x55f3ff9bf320 2024-05-09T08:38:19.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.206+0000 7fe6d1ffb640 1 -- 172.21.15.107:0/1698878644 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe6e401e450 con 0x55f3ff9bf320 2024-05-09T08:38:19.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.206+0000 7fe6f3c0e740 10 client.17127 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:19.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.206+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/1698878644 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f3ffa35490 con 0x55f3ff9bf320 2024-05-09T08:38:19.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.206+0000 7fe6d1ffb640 1 -- 172.21.15.107:0/1698878644 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6d409b630 con 0x55f3ff9bf320 2024-05-09T08:38:19.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.206+0000 7fe6d1ffb640 1 -- 172.21.15.107:0/1698878644 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe6e400f040 con 0x55f3ff9bf320 2024-05-09T08:38:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.206+0000 7fe6f3c0e740 10 client.17127 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.206+0000 7fe6f3c0e740 10 client.17127 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.206+0000 7fe6f3c0e740 20 client.17127 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.206+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/1698878644 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f3ffa35c90 0x55f3ffa380b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.206+0000 7fe6f3c0e740 4 client.17127 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:38:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.206+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/1698878644 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f3ff9216c0 con 0x55f3ffa35c90 2024-05-09T08:38:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.206+0000 7fe6ea59c640 1 --2- 172.21.15.107:0/1698878644 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f3ffa35c90 0x55f3ffa380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6d1ffb640 10 client.17127 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6ea59c640 1 --2- 172.21.15.107:0/1698878644 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f3ffa35c90 0x55f3ffa380b0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd8cebfa2780d9a8 server_cookie=df4ab087c3f014d8 in_seq=0 out_seq=0 2024-05-09T08:38:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6d1ffb640 1 -- 172.21.15.107:0/1698878644 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f3ff9216c0 con 0x55f3ffa35c90 2024-05-09T08:38:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6d1ffb640 10 client.17127 handle_command_reply: tid=0 2024-05-09T08:38:19.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6f3c0e740 1 client.17127 shutdown 2024-05-09T08:38:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/1698878644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9bf320 msgr2=0x55f3ff9c2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/1698878644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9bf320 0x55f3ff9c2aa0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4013970 tx=0x7fe6e4004ab0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/1698878644 shutdown_connections 2024-05-09T08:38:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/1698878644 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f3ffa35c90 0x55f3ffa380b0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/1698878644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f3ff9bf320 0x55f3ff9c2aa0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/1698878644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3ff9b4c50 0x55f3ff9c79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6d1ffb640 0 client.17127 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:19.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6f3c0e740 1 --2- 172.21.15.107:0/1698878644 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f3ff9b4330 0x55f3ff9c7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/1698878644 >> 172.21.15.107:0/1698878644 conn(0x55f3ff9b0650 msgr2=0x55f3ff9aef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/1698878644 shutdown_connections 2024-05-09T08:38:19.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6f3c0e740 1 -- 172.21.15.107:0/1698878644 wait complete. 2024-05-09T08:38:19.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.207+0000 7fe6f3c0e740 20 client.17127 trim_cache size 0 max 16384 2024-05-09T08:38:19.232 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:38:19.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.208+0000 7fe6f0f09640 1 -- 172.21.15.107:0/3694187887 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6d80a2d00 msgr2=0x7fe6d80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.208+0000 7fe6f0f09640 1 --2- 172.21.15.107:0/3694187887 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6d80a2d00 0x7fe6d80a51c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec069bd0 tx=0x7fe6e4004a40 comp rx=0 tx=0).stop 2024-05-09T08:38:19.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.208+0000 7fe6f0f09640 1 -- 172.21.15.107:0/3694187887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6ec100f60 msgr2=0x7fe6ec19a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.208+0000 7fe6f0f09640 1 --2- 172.21.15.107:0/3694187887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6ec100f60 0x7fe6ec19a790 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fe6d400c890 tx=0x7fe6d400cd50 comp rx=0 tx=0).stop 2024-05-09T08:38:19.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.208+0000 7fe6f0f09640 1 -- 172.21.15.107:0/3694187887 shutdown_connections 2024-05-09T08:38:19.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.208+0000 7fe6f0f09640 1 --2- 172.21.15.107:0/3694187887 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6d80a2d00 0x7fe6d80a51c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.208+0000 7fe6f0f09640 1 --2- 172.21.15.107:0/3694187887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6ec10e1f0 0x7fe6ec19eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.208+0000 7fe6f0f09640 1 --2- 172.21.15.107:0/3694187887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6ec100f60 0x7fe6ec19a790 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.208+0000 7fe6f0f09640 1 --2- 172.21.15.107:0/3694187887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6ec100640 0x7fe6ec19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.208+0000 7fe6f0f09640 1 -- 172.21.15.107:0/3694187887 >> 172.21.15.107:0/3694187887 conn(0x7fe6ec076ab0 msgr2=0x7fe6ec10f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.208+0000 7fe6f0f09640 1 -- 172.21.15.107:0/3694187887 shutdown_connections 2024-05-09T08:38:19.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:19.208+0000 7fe6f0f09640 1 -- 172.21.15.107:0/3694187887 wait complete. 2024-05-09T08:38:19.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:19 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/176855977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:38:19.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:19 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/176855977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:38:19.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:19 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/176855977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:38:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.819+0000 7f1362390640 1 -- 172.21.15.107:0/930837371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f135c107820 msgr2=0x7f135c107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.819+0000 7f1362390640 1 --2- 172.21.15.107:0/930837371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f135c107820 0x7f135c107c00 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f134c009930 tx=0x7f134c02a420 comp rx=0 tx=0).stop 2024-05-09T08:38:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.819+0000 7f1362390640 1 -- 172.21.15.107:0/930837371 shutdown_connections 2024-05-09T08:38:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.819+0000 7f1362390640 1 --2- 172.21.15.107:0/930837371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f135c1021d0 0x7f135c10e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.819+0000 7f1362390640 1 --2- 172.21.15.107:0/930837371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f135c101810 0x7f135c101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.819+0000 7f1362390640 1 --2- 172.21.15.107:0/930837371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f135c107820 0x7f135c107c00 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.819+0000 7f1362390640 1 -- 172.21.15.107:0/930837371 >> 172.21.15.107:0/930837371 conn(0x7f135c0fd2e0 msgr2=0x7f135c0ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.819+0000 7f1362390640 1 -- 172.21.15.107:0/930837371 shutdown_connections 2024-05-09T08:38:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.819+0000 7f1362390640 1 -- 172.21.15.107:0/930837371 wait complete. 2024-05-09T08:38:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.820+0000 7f1362390640 1 Processor -- start 2024-05-09T08:38:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.820+0000 7f1362390640 1 -- start start 2024-05-09T08:38:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.820+0000 7f1362390640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f135c101810 0x7f135c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.820+0000 7f1362390640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f135c1021d0 0x7f135c198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.820+0000 7f1362390640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f135c107820 0x7f135c19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.820+0000 7f1362390640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f135c111aa0 con 0x7f135c101810 2024-05-09T08:38:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.820+0000 7f1362390640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f135c111920 con 0x7f135c1021d0 2024-05-09T08:38:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.820+0000 7f1362390640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f135c111c20 con 0x7f135c107820 2024-05-09T08:38:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f135bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f135c101810 0x7f135c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f135bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f135c101810 0x7f135c197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46646/0 (socket says 172.21.15.107:46646) 2024-05-09T08:38:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f135bfff640 1 -- 172.21.15.107:0/3967703824 learned_addr learned my addr 172.21.15.107:0/3967703824 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f135b7fe640 1 --2- 172.21.15.107:0/3967703824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f135c1021d0 0x7f135c198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f136092d640 1 --2- 172.21.15.107:0/3967703824 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f135c107820 0x7f135c19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f135bfff640 1 -- 172.21.15.107:0/3967703824 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f135c107820 msgr2=0x7f135c19c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f135bfff640 1 --2- 172.21.15.107:0/3967703824 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f135c107820 0x7f135c19c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f135bfff640 1 -- 172.21.15.107:0/3967703824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f135c1021d0 msgr2=0x7f135c198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f135bfff640 1 --2- 172.21.15.107:0/3967703824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f135c1021d0 0x7f135c198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f135bfff640 1 -- 172.21.15.107:0/3967703824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f135c19cfb0 con 0x7f135c101810 2024-05-09T08:38:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f136092d640 1 --2- 172.21.15.107:0/3967703824 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f135c107820 0x7f135c19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f135bfff640 1 --2- 172.21.15.107:0/3967703824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f135c101810 0x7f135c197fc0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f134c004770 tx=0x7f134c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f13597fa640 1 -- 172.21.15.107:0/3967703824 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c03f070 con 0x7f135c101810 2024-05-09T08:38:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f1362390640 1 -- 172.21.15.107:0/3967703824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f135c19d240 con 0x7f135c101810 2024-05-09T08:38:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f13597fa640 1 -- 172.21.15.107:0/3967703824 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f134c03b460 con 0x7f135c101810 2024-05-09T08:38:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f13597fa640 1 -- 172.21.15.107:0/3967703824 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c035070 con 0x7f135c101810 2024-05-09T08:38:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.821+0000 7f1362390640 1 -- 172.21.15.107:0/3967703824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f135c1a4b30 con 0x7f135c101810 2024-05-09T08:38:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.824+0000 7f13597fa640 1 -- 172.21.15.107:0/3967703824 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f134c03b610 con 0x7f135c101810 2024-05-09T08:38:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.824+0000 7f1365179740 1 Processor -- start 2024-05-09T08:38:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.824+0000 7f1365179740 1 -- start start 2024-05-09T08:38:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.824+0000 7f1365179740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795b1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.824+0000 7f1365179740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795b1c70 0x55ef795c0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.824+0000 7f1365179740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795c0a40 0x55ef795c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f1365179740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ef79516340 con 0x55ef795c0a40 2024-05-09T08:38:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f1365179740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ef794be290 con 0x55ef795b1c70 2024-05-09T08:38:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f13597fa640 1 --2- 172.21.15.107:0/3967703824 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13440a2da0 0x7f13440a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f1365179740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ef79512300 con 0x55ef795b1350 2024-05-09T08:38:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f135b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795c0a40 0x55ef795c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f136092d640 1 --2- 172.21.15.107:0/3967703824 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13440a2da0 0x7f13440a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f135b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795c0a40 0x55ef795c2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46662/0 (socket says 172.21.15.107:46662) 2024-05-09T08:38:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f135b7fe640 1 -- 172.21.15.107:0/3795880385 learned_addr learned my addr 172.21.15.107:0/3795880385 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f136092d640 1 --2- 172.21.15.107:0/3795880385 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795b1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f13597fa640 1 -- 172.21.15.107:0/3967703824 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f134c0efbd0 con 0x7f135c101810 2024-05-09T08:38:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f135bfff640 1 --2- 172.21.15.107:0/3795880385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795b1c70 0x55ef795c0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f135b7fe640 1 -- 172.21.15.107:0/3795880385 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 msgr2=0x55ef795b1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f135b7fe640 1 --2- 172.21.15.107:0/3795880385 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795b1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f135b7fe640 1 -- 172.21.15.107:0/3795880385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795b1c70 msgr2=0x55ef795c0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f135b7fe640 1 --2- 172.21.15.107:0/3795880385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795b1c70 0x55ef795c0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f135b7fe640 1 -- 172.21.15.107:0/3795880385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef794e28e0 con 0x55ef795c0a40 2024-05-09T08:38:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f135b7fe640 1 --2- 172.21.15.107:0/3795880385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795c0a40 0x55ef795c2e30 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f135000dc80 tx=0x7f135000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425693e47ab5f832 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f136092d640 1 --2- 172.21.15.107:0/3967703824 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13440a2da0 0x7f13440a5260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f13540092f0 tx=0x7f1354009280 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.825+0000 7f13427fc640 1 -- 172.21.15.107:0/3795880385 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1350015070 con 0x55ef795c0a40 2024-05-09T08:38:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.826+0000 7f13427fc640 1 -- 172.21.15.107:0/3795880385 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f135001a910 con 0x55ef795c0a40 2024-05-09T08:38:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.826+0000 7f13427fc640 1 -- 172.21.15.107:0/3795880385 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1350011a10 con 0x55ef795c0a40 2024-05-09T08:38:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.826+0000 7f1365179740 1 -- 172.21.15.107:0/3795880385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795c0a40 msgr2=0x55ef795c2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.826+0000 7f1365179740 1 --2- 172.21.15.107:0/3795880385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795c0a40 0x55ef795c2e30 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f135000dc80 tx=0x7f135000f730 comp rx=0 tx=0).stop 2024-05-09T08:38:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.826+0000 7f1365179740 1 -- 172.21.15.107:0/3795880385 shutdown_connections 2024-05-09T08:38:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.826+0000 7f1365179740 1 --2- 172.21.15.107:0/3795880385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795c0a40 0x55ef795c2e30 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.826+0000 7f1365179740 1 --2- 172.21.15.107:0/3795880385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795b1c70 0x55ef795c0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.826+0000 7f1365179740 1 --2- 172.21.15.107:0/3795880385 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795b1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.826+0000 7f1365179740 1 -- 172.21.15.107:0/3795880385 >> 172.21.15.107:0/3795880385 conn(0x55ef795ad960 msgr2=0x55ef795add40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.826+0000 7f1365179740 1 -- 172.21.15.107:0/3795880385 shutdown_connections 2024-05-09T08:38:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.826+0000 7f1365179740 1 -- 172.21.15.107:0/3795880385 wait complete. 2024-05-09T08:38:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.827+0000 7f1365179740 1 -- start start 2024-05-09T08:38:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.827+0000 7f1365179740 1 Processor -- start 2024-05-09T08:38:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.827+0000 7f1365179740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795bd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.827+0000 7f1365179740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795bde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.827+0000 7f1365179740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795c0a40 0x55ef795be3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.827+0000 7f1365179740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ef795c3670 con 0x55ef795b1c70 2024-05-09T08:38:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.827+0000 7f1365179740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ef794d1790 con 0x55ef795c0a40 2024-05-09T08:38:19.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.827+0000 7f1365179740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ef795c37f0 con 0x55ef795b1350 2024-05-09T08:38:19.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.827+0000 7f1365179740 10 client.0 _ll_register_callbacks cb 0x55ef7950ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:19.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.827+0000 7f135b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795bde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.827+0000 7f135b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795bde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46670/0 (socket says 172.21.15.107:46670) 2024-05-09T08:38:19.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.827+0000 7f135b7fe640 1 -- 172.21.15.107:0/2731562152 learned_addr learned my addr 172.21.15.107:0/2731562152 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:19.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f135bfff640 1 --2- 172.21.15.107:0/2731562152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795bd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f136092d640 1 --2- 172.21.15.107:0/2731562152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795c0a40 0x55ef795be3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f135b7fe640 1 -- 172.21.15.107:0/2731562152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 msgr2=0x55ef795bd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f135b7fe640 1 --2- 172.21.15.107:0/2731562152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795bd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f135b7fe640 1 -- 172.21.15.107:0/2731562152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795c0a40 msgr2=0x55ef795be3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f135b7fe640 1 --2- 172.21.15.107:0/2731562152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795c0a40 0x55ef795be3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f135b7fe640 1 -- 172.21.15.107:0/2731562152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef7922a210 con 0x55ef795b1c70 2024-05-09T08:38:19.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f136092d640 1 --2- 172.21.15.107:0/2731562152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795c0a40 0x55ef795be3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:38:19.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f135b7fe640 1 --2- 172.21.15.107:0/2731562152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795bde70 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f135000dfa0 tx=0x7f135000dfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98352dff35ee029 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f133b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:19.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f133b7fe640 1 -- 172.21.15.107:0/2731562152 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1350015070 con 0x55ef795b1c70 2024-05-09T08:38:19.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f133b7fe640 1 -- 172.21.15.107:0/2731562152 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f135001a670 con 0x55ef795b1c70 2024-05-09T08:38:19.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f1365179740 1 -- 172.21.15.107:0/2731562152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f135001a910 con 0x55ef795b1c70 2024-05-09T08:38:19.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f133b7fe640 1 -- 172.21.15.107:0/2731562152 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f135001aac0 con 0x55ef795b1c70 2024-05-09T08:38:19.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f1365179740 10 client.17139 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:19.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f1365179740 1 -- 172.21.15.107:0/2731562152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ef792f6620 con 0x55ef795b1c70 2024-05-09T08:38:19.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.828+0000 7f133b7fe640 1 -- 172.21.15.107:0/2731562152 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f135001a910 con 0x55ef795b1c70 2024-05-09T08:38:19.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.829+0000 7f133b7fe640 1 -- 172.21.15.107:0/2731562152 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f13500119a0 con 0x55ef795b1c70 2024-05-09T08:38:19.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.829+0000 7f1365179740 10 client.17139 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:19.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.829+0000 7f1365179740 10 client.17139 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.829+0000 7f1365179740 20 client.17139 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.829+0000 7f1365179740 1 --2- 172.21.15.107:0/2731562152 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ef795d0650 0x55ef795d2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.829+0000 7f1365179740 4 client.17139 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.829+0000 7f1365179740 1 -- 172.21.15.107:0/2731562152 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef79523a70 con 0x55ef795d0650 2024-05-09T08:38:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.829+0000 7f135bfff640 1 --2- 172.21.15.107:0/2731562152 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ef795d0650 0x55ef795d2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.829+0000 7f135bfff640 1 --2- 172.21.15.107:0/2731562152 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ef795d0650 0x55ef795d2a40 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d10777853e5543fe server_cookie=b271d239d37d36ff in_seq=0 out_seq=0 2024-05-09T08:38:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.829+0000 7f133b7fe640 10 client.17139 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.830+0000 7f133b7fe640 1 -- 172.21.15.107:0/2731562152 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef79523a70 con 0x55ef795d0650 2024-05-09T08:38:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.830+0000 7f133b7fe640 10 client.17139 handle_command_reply: tid=0 2024-05-09T08:38:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.830+0000 7f1365179740 1 client.17139 shutdown 2024-05-09T08:38:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.830+0000 7f1365179740 1 -- 172.21.15.107:0/2731562152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 msgr2=0x55ef795bde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.830+0000 7f1365179740 1 --2- 172.21.15.107:0/2731562152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795bde70 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f135000dfa0 tx=0x7f135000dfd0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.830+0000 7f1365179740 1 -- 172.21.15.107:0/2731562152 shutdown_connections 2024-05-09T08:38:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.830+0000 7f1365179740 1 --2- 172.21.15.107:0/2731562152 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ef795d0650 0x55ef795d2a40 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.830+0000 7f1365179740 1 --2- 172.21.15.107:0/2731562152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795c0a40 0x55ef795be3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.830+0000 7f1365179740 1 --2- 172.21.15.107:0/2731562152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795bde70 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.830+0000 7f1365179740 1 --2- 172.21.15.107:0/2731562152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795bd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.830+0000 7f1365179740 1 -- 172.21.15.107:0/2731562152 >> 172.21.15.107:0/2731562152 conn(0x55ef795ad960 msgr2=0x55ef795b37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.830+0000 7f133b7fe640 0 client.17139 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.831+0000 7f1365179740 1 -- 172.21.15.107:0/2731562152 shutdown_connections 2024-05-09T08:38:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.831+0000 7f1365179740 1 -- 172.21.15.107:0/2731562152 wait complete. 2024-05-09T08:38:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.831+0000 7f1365179740 20 client.17139 trim_cache size 0 max 16384 2024-05-09T08:38:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f1365179740 1 Processor -- start 2024-05-09T08:38:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f1365179740 1 -- start start 2024-05-09T08:38:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f1365179740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795cc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f1365179740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795bc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f1365179740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795c0a40 0x55ef795be1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f1365179740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f135c1af210 con 0x55ef795b1c70 2024-05-09T08:38:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f1365179740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f135c1acae0 con 0x55ef795c0a40 2024-05-09T08:38:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f1365179740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f135c1aa930 con 0x55ef795b1350 2024-05-09T08:38:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f135b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795bc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f135b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795bc200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46682/0 (socket says 172.21.15.107:46682) 2024-05-09T08:38:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f135b7fe640 1 -- 172.21.15.107:0/3534886470 learned_addr learned my addr 172.21.15.107:0/3534886470 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f135bfff640 1 --2- 172.21.15.107:0/3534886470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795cc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f135b7fe640 1 -- 172.21.15.107:0/3534886470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 msgr2=0x55ef795cc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f135b7fe640 1 --2- 172.21.15.107:0/3534886470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795cc530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f136092d640 1 --2- 172.21.15.107:0/3534886470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795c0a40 0x55ef795be1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f135b7fe640 1 -- 172.21.15.107:0/3534886470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795c0a40 msgr2=0x55ef795be1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f135b7fe640 1 --2- 172.21.15.107:0/3534886470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795c0a40 0x55ef795be1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.846+0000 7f135b7fe640 1 -- 172.21.15.107:0/3534886470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef795bc8e0 con 0x55ef795b1c70 2024-05-09T08:38:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.847+0000 7f135b7fe640 1 --2- 172.21.15.107:0/3534886470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795bc200 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f1350011e40 tx=0x7f1350001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227de09d6253eb0f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.847+0000 7f1341ffb640 1 -- 172.21.15.107:0/3534886470 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1350037070 con 0x55ef795b1c70 2024-05-09T08:38:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.847+0000 7f1341ffb640 1 -- 172.21.15.107:0/3534886470 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1350011b20 con 0x55ef795b1c70 2024-05-09T08:38:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.847+0000 7f1341ffb640 1 -- 172.21.15.107:0/3534886470 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1350015070 con 0x55ef795b1c70 2024-05-09T08:38:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.848+0000 7f1365179740 1 -- 172.21.15.107:0/3534886470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 msgr2=0x55ef795bc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.848+0000 7f1365179740 1 --2- 172.21.15.107:0/3534886470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795bc200 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f1350011e40 tx=0x7f1350001e50 comp rx=0 tx=0).stop 2024-05-09T08:38:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.848+0000 7f1365179740 1 -- 172.21.15.107:0/3534886470 shutdown_connections 2024-05-09T08:38:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.848+0000 7f1365179740 1 --2- 172.21.15.107:0/3534886470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795c0a40 0x55ef795be1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.848+0000 7f1365179740 1 --2- 172.21.15.107:0/3534886470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795bc200 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.848+0000 7f1365179740 1 --2- 172.21.15.107:0/3534886470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795cc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.848+0000 7f1365179740 1 -- 172.21.15.107:0/3534886470 >> 172.21.15.107:0/3534886470 conn(0x55ef795ad960 msgr2=0x55ef795ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.848+0000 7f1365179740 1 -- 172.21.15.107:0/3534886470 shutdown_connections 2024-05-09T08:38:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.848+0000 7f1365179740 1 -- 172.21.15.107:0/3534886470 wait complete. 2024-05-09T08:38:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.848+0000 7f1365179740 1 -- start start 2024-05-09T08:38:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.848+0000 7f1365179740 1 Processor -- start 2024-05-09T08:38:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.848+0000 7f1365179740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.848+0000 7f1365179740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795c41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.848+0000 7f1365179740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795c0a40 0x55ef795c4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f1365179740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f135c1aafa0 con 0x55ef795b1c70 2024-05-09T08:38:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f1365179740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f135c1ac410 con 0x55ef795c0a40 2024-05-09T08:38:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f1365179740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f135c1ab670 con 0x55ef795b1350 2024-05-09T08:38:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f1365179740 10 client.0 _ll_register_callbacks cb 0x7f134c005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f135b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795c41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f135b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795c41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46690/0 (socket says 172.21.15.107:46690) 2024-05-09T08:38:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f135b7fe640 1 -- 172.21.15.107:0/1713641274 learned_addr learned my addr 172.21.15.107:0/1713641274 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f135bfff640 1 --2- 172.21.15.107:0/1713641274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f135b7fe640 1 -- 172.21.15.107:0/1713641274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 msgr2=0x55ef795c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f135b7fe640 1 --2- 172.21.15.107:0/1713641274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795c8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f135b7fe640 1 -- 172.21.15.107:0/1713641274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795c0a40 msgr2=0x55ef795c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f135b7fe640 1 --2- 172.21.15.107:0/1713641274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795c0a40 0x55ef795c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f135b7fe640 1 -- 172.21.15.107:0/1713641274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef795c4fb0 con 0x55ef795b1c70 2024-05-09T08:38:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f135bfff640 1 --2- 172.21.15.107:0/1713641274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f135b7fe640 1 --2- 172.21.15.107:0/1713641274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795c41d0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f1350005890 tx=0x7f13500048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36df696baf8778ce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f133b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f133b7fe640 1 -- 172.21.15.107:0/1713641274 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1350015070 con 0x55ef795b1c70 2024-05-09T08:38:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f133b7fe640 1 -- 172.21.15.107:0/1713641274 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1350011670 con 0x55ef795b1c70 2024-05-09T08:38:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f1365179740 1 -- 172.21.15.107:0/1713641274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1350011b20 con 0x55ef795b1c70 2024-05-09T08:38:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f133b7fe640 1 -- 172.21.15.107:0/1713641274 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1350048050 con 0x55ef795b1c70 2024-05-09T08:38:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f1365179740 10 client.17145 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.849+0000 7f1365179740 1 -- 172.21.15.107:0/1713641274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ef79636aa0 con 0x55ef795b1c70 2024-05-09T08:38:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.850+0000 7f133b7fe640 1 -- 172.21.15.107:0/1713641274 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1350011b20 con 0x55ef795b1c70 2024-05-09T08:38:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.850+0000 7f133b7fe640 1 -- 172.21.15.107:0/1713641274 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f1350043030 con 0x55ef795b1c70 2024-05-09T08:38:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.850+0000 7f1365179740 10 client.17145 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.850+0000 7f1365179740 10 client.17145 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.850+0000 7f1365179740 20 client.17145 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.850+0000 7f1365179740 1 --2- 172.21.15.107:0/1713641274 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ef79636f70 0x55ef796393b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.850+0000 7f1365179740 4 client.17145 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:38:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.850+0000 7f1365179740 1 -- 172.21.15.107:0/1713641274 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ef7951ddc0 con 0x55ef79636f70 2024-05-09T08:38:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.850+0000 7f135bfff640 1 --2- 172.21.15.107:0/1713641274 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ef79636f70 0x55ef796393b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.851+0000 7f133b7fe640 10 client.17145 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.851+0000 7f135bfff640 1 --2- 172.21.15.107:0/1713641274 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ef79636f70 0x55ef796393b0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5986ef97edd8ead server_cookie=9006dc8fa4db8f42 in_seq=0 out_seq=0 2024-05-09T08:38:19.869 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.870+0000 7f133b7fe640 1 -- 172.21.15.107:0/1713641274 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ef7951ddc0 con 0x55ef79636f70 2024-05-09T08:38:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.870+0000 7f133b7fe640 10 client.17145 handle_command_reply: tid=0 2024-05-09T08:38:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.870+0000 7f1365179740 1 client.17145 shutdown 2024-05-09T08:38:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.870+0000 7f1365179740 1 -- 172.21.15.107:0/1713641274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 msgr2=0x55ef795c41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.870+0000 7f1365179740 1 --2- 172.21.15.107:0/1713641274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795c41d0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f1350005890 tx=0x7f13500048b0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.870+0000 7f1365179740 1 -- 172.21.15.107:0/1713641274 shutdown_connections 2024-05-09T08:38:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.870+0000 7f1365179740 1 --2- 172.21.15.107:0/1713641274 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ef79636f70 0x55ef796393b0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.870+0000 7f1365179740 1 --2- 172.21.15.107:0/1713641274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ef795c0a40 0x55ef795c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.870+0000 7f1365179740 1 --2- 172.21.15.107:0/1713641274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ef795b1c70 0x55ef795c41d0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.870+0000 7f133b7fe640 0 client.17145 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.870+0000 7f1365179740 1 --2- 172.21.15.107:0/1713641274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ef795b1350 0x55ef795c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.870+0000 7f1365179740 1 -- 172.21.15.107:0/1713641274 >> 172.21.15.107:0/1713641274 conn(0x55ef795ad960 msgr2=0x55ef795b3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.870+0000 7f1365179740 1 -- 172.21.15.107:0/1713641274 shutdown_connections 2024-05-09T08:38:19.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.870+0000 7f1365179740 1 -- 172.21.15.107:0/1713641274 wait complete. 2024-05-09T08:38:19.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.870+0000 7f1365179740 20 client.17145 trim_cache size 0 max 16384 2024-05-09T08:38:19.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:38:19.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:38:19.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:38:19.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:38:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.871+0000 7f1362390640 1 -- 172.21.15.107:0/3967703824 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13440a2da0 msgr2=0x7f13440a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.871+0000 7f1362390640 1 --2- 172.21.15.107:0/3967703824 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13440a2da0 0x7f13440a5260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f13540092f0 tx=0x7f1354009280 comp rx=0 tx=0).stop 2024-05-09T08:38:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.871+0000 7f1362390640 1 -- 172.21.15.107:0/3967703824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f135c101810 msgr2=0x7f135c197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.871+0000 7f1362390640 1 --2- 172.21.15.107:0/3967703824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f135c101810 0x7f135c197fc0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f134c004770 tx=0x7f134c0042f0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.871+0000 7f1362390640 1 -- 172.21.15.107:0/3967703824 shutdown_connections 2024-05-09T08:38:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.871+0000 7f1362390640 1 --2- 172.21.15.107:0/3967703824 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13440a2da0 0x7f13440a5260 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.871+0000 7f1362390640 1 --2- 172.21.15.107:0/3967703824 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f135c107820 0x7f135c19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.871+0000 7f1362390640 1 --2- 172.21.15.107:0/3967703824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f135c1021d0 0x7f135c198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.871+0000 7f1362390640 1 --2- 172.21.15.107:0/3967703824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f135c101810 0x7f135c197fc0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.871+0000 7f1362390640 1 -- 172.21.15.107:0/3967703824 >> 172.21.15.107:0/3967703824 conn(0x7f135c0fd2e0 msgr2=0x7f135c0fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.871+0000 7f1362390640 1 -- 172.21.15.107:0/3967703824 shutdown_connections 2024-05-09T08:38:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:19.871+0000 7f1362390640 1 -- 172.21.15.107:0/3967703824 wait complete. 2024-05-09T08:38:20.236 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:38:20.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:38:20.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:38:20.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.355+0000 7f885be4d640 1 -- 172.21.15.107:0/3759291416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88541010c0 msgr2=0x7f885410d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.355+0000 7f885be4d640 1 --2- 172.21.15.107:0/3759291416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88541010c0 0x7f885410d5d0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f884c009cd0 tx=0x7f884c02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.355+0000 7f885be4d640 1 -- 172.21.15.107:0/3759291416 shutdown_connections 2024-05-09T08:38:20.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.355+0000 7f885be4d640 1 --2- 172.21.15.107:0/3759291416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88541010c0 0x7f885410d5d0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.355+0000 7f885be4d640 1 --2- 172.21.15.107:0/3759291416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8854100700 0x7f8854100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.355+0000 7f885be4d640 1 --2- 172.21.15.107:0/3759291416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8854106710 0x7f8854106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.355+0000 7f885be4d640 1 -- 172.21.15.107:0/3759291416 >> 172.21.15.107:0/3759291416 conn(0x7f8854077830 msgr2=0x7f8854077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:20.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.355+0000 7f885be4d640 1 -- 172.21.15.107:0/3759291416 shutdown_connections 2024-05-09T08:38:20.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.355+0000 7f885be4d640 1 -- 172.21.15.107:0/3759291416 wait complete. 2024-05-09T08:38:20.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f885be4d640 1 Processor -- start 2024-05-09T08:38:20.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f885be4d640 1 -- start start 2024-05-09T08:38:20.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f885be4d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8854100700 0x7f8854196c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f885be4d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88541010c0 0x7f8854197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f885be4d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8854106710 0x7f88541976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f885be4d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f88541109d0 con 0x7f8854106710 2024-05-09T08:38:20.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f885be4d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8854110850 con 0x7f88541010c0 2024-05-09T08:38:20.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f885be4d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8854110b50 con 0x7f8854100700 2024-05-09T08:38:20.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f885a3ea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8854106710 0x7f88541976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f885a3ea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8854106710 0x7f88541976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46710/0 (socket says 172.21.15.107:46710) 2024-05-09T08:38:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f885a3ea640 1 -- 172.21.15.107:0/1217322833 learned_addr learned my addr 172.21.15.107:0/1217322833 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f88593e8640 1 --2- 172.21.15.107:0/1217322833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88541010c0 0x7f8854197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f8859be9640 1 --2- 172.21.15.107:0/1217322833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8854100700 0x7f8854196c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f885a3ea640 1 -- 172.21.15.107:0/1217322833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8854100700 msgr2=0x7f8854196c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f885a3ea640 1 --2- 172.21.15.107:0/1217322833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8854100700 0x7f8854196c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f885a3ea640 1 -- 172.21.15.107:0/1217322833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88541010c0 msgr2=0x7f8854197160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f885a3ea640 1 --2- 172.21.15.107:0/1217322833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88541010c0 0x7f8854197160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.356+0000 7f885a3ea640 1 -- 172.21.15.107:0/1217322833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88541a6490 con 0x7f8854106710 2024-05-09T08:38:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.357+0000 7f885a3ea640 1 --2- 172.21.15.107:0/1217322833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8854106710 0x7f88541976a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f884c033040 tx=0x7f884c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.357+0000 7f884affd640 1 -- 172.21.15.107:0/1217322833 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c004040 con 0x7f8854106710 2024-05-09T08:38:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.357+0000 7f885be4d640 1 -- 172.21.15.107:0/1217322833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88541a6720 con 0x7f8854106710 2024-05-09T08:38:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.357+0000 7f884affd640 1 -- 172.21.15.107:0/1217322833 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f884c004690 con 0x7f8854106710 2024-05-09T08:38:20.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.357+0000 7f884affd640 1 -- 172.21.15.107:0/1217322833 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c030e10 con 0x7f8854106710 2024-05-09T08:38:20.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.357+0000 7f885be4d640 1 -- 172.21.15.107:0/1217322833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88541a6cc0 con 0x7f8854106710 2024-05-09T08:38:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.360+0000 7f884affd640 1 -- 172.21.15.107:0/1217322833 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f884c0305a0 con 0x7f8854106710 2024-05-09T08:38:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.360+0000 7f885ec3a740 1 Processor -- start 2024-05-09T08:38:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.360+0000 7f885ec3a740 1 -- start start 2024-05-09T08:38:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.360+0000 7f885ec3a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265d80b0 0x55c6265d63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.360+0000 7f885ec3a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d69c0 0x55c6265d6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.360+0000 7f885ec3a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265e1740 0x55c6265e3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.360+0000 7f885ec3a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c62653b990 con 0x55c6265e1740 2024-05-09T08:38:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.360+0000 7f885ec3a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c6264e35b0 con 0x55c6265d69c0 2024-05-09T08:38:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.360+0000 7f885ec3a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c626537950 con 0x55c6265d80b0 2024-05-09T08:38:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f88593e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265e1740 0x55c6265e3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f88593e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265e1740 0x55c6265e3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46726/0 (socket says 172.21.15.107:46726) 2024-05-09T08:38:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f88593e8640 1 -- 172.21.15.107:0/2988915643 learned_addr learned my addr 172.21.15.107:0/2988915643 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f8859be9640 1 --2- 172.21.15.107:0/2988915643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d69c0 0x55c6265d6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f88593e8640 1 --2- 172.21.15.107:0/2988915643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265d80b0 0x55c6265d63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f884affd640 1 --2- 172.21.15.107:0/1217322833 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f88400a2d00 0x7f88400a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f88593e8640 1 -- 172.21.15.107:0/2988915643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265d80b0 msgr2=0x55c6265d63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f88593e8640 1 --2- 172.21.15.107:0/2988915643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265d80b0 0x55c6265d63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f88593e8640 1 -- 172.21.15.107:0/2988915643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d69c0 msgr2=0x55c6265d6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f88593e8640 1 --2- 172.21.15.107:0/2988915643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d69c0 0x55c6265d6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f88593e8640 1 -- 172.21.15.107:0/2988915643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6263177e0 con 0x55c6265e1740 2024-05-09T08:38:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f8859be9640 1 --2- 172.21.15.107:0/2988915643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d69c0 0x55c6265d6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f885a3ea640 1 --2- 172.21.15.107:0/1217322833 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f88400a2d00 0x7f88400a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f884affd640 1 -- 172.21.15.107:0/1217322833 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f884c02e030 con 0x7f8854106710 2024-05-09T08:38:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f88593e8640 1 --2- 172.21.15.107:0/2988915643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265e1740 0x55c6265e3b30 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f88440127c0 tx=0x7f884400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85281ebe0556710d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f883ffff640 1 -- 172.21.15.107:0/2988915643 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8844012e00 con 0x55c6265e1740 2024-05-09T08:38:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f883ffff640 1 -- 172.21.15.107:0/2988915643 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8844010c20 con 0x55c6265e1740 2024-05-09T08:38:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.361+0000 7f883ffff640 1 -- 172.21.15.107:0/2988915643 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884401eab0 con 0x55c6265e1740 2024-05-09T08:38:20.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.362+0000 7f885ec3a740 1 -- 172.21.15.107:0/2988915643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265e1740 msgr2=0x55c6265e3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.362+0000 7f885a3ea640 1 --2- 172.21.15.107:0/1217322833 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f88400a2d00 0x7f88400a51c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f884c0bcfe0 tx=0x7f884c08ceb0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:20.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.362+0000 7f885ec3a740 1 --2- 172.21.15.107:0/2988915643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265e1740 0x55c6265e3b30 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f88440127c0 tx=0x7f884400b7c0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.362+0000 7f885ec3a740 1 -- 172.21.15.107:0/2988915643 shutdown_connections 2024-05-09T08:38:20.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.362+0000 7f885ec3a740 1 --2- 172.21.15.107:0/2988915643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265e1740 0x55c6265e3b30 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.362+0000 7f885ec3a740 1 --2- 172.21.15.107:0/2988915643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d69c0 0x55c6265d6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.362+0000 7f885ec3a740 1 --2- 172.21.15.107:0/2988915643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265d80b0 0x55c6265d63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.362+0000 7f885ec3a740 1 -- 172.21.15.107:0/2988915643 >> 172.21.15.107:0/2988915643 conn(0x55c62659f0a0 msgr2=0x55c6265d2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:20.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.362+0000 7f885ec3a740 1 -- 172.21.15.107:0/2988915643 shutdown_connections 2024-05-09T08:38:20.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.362+0000 7f885ec3a740 1 -- 172.21.15.107:0/2988915643 wait complete. 2024-05-09T08:38:20.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.363+0000 7f885ec3a740 1 -- start start 2024-05-09T08:38:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.363+0000 7f885ec3a740 1 Processor -- start 2024-05-09T08:38:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.363+0000 7f885ec3a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265d69c0 0x55c6265de6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.363+0000 7f885ec3a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d80b0 0x55c6265debf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.363+0000 7f885ec3a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265e1740 0x55c6265df130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.363+0000 7f885ec3a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c6265e45d0 con 0x55c6265e1740 2024-05-09T08:38:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.363+0000 7f885ec3a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c6264f6b40 con 0x55c6265d80b0 2024-05-09T08:38:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.363+0000 7f885ec3a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c6265e4750 con 0x55c6265d69c0 2024-05-09T08:38:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.363+0000 7f885ec3a740 10 client.0 _ll_register_callbacks cb 0x55c6264eb0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.363+0000 7f88593e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265e1740 0x55c6265df130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.363+0000 7f88593e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265e1740 0x55c6265df130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46740/0 (socket says 172.21.15.107:46740) 2024-05-09T08:38:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.363+0000 7f88593e8640 1 -- 172.21.15.107:0/2000551248 learned_addr learned my addr 172.21.15.107:0/2000551248 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.363+0000 7f88593e8640 1 --2- 172.21.15.107:0/2000551248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265d69c0 0x55c6265de6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.364+0000 7f8859be9640 1 --2- 172.21.15.107:0/2000551248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d80b0 0x55c6265debf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.364+0000 7f88593e8640 1 -- 172.21.15.107:0/2000551248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265d69c0 msgr2=0x55c6265de6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.364+0000 7f88593e8640 1 --2- 172.21.15.107:0/2000551248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265d69c0 0x55c6265de6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.364+0000 7f88593e8640 1 -- 172.21.15.107:0/2000551248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d80b0 msgr2=0x55c6265debf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.364+0000 7f88593e8640 1 --2- 172.21.15.107:0/2000551248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d80b0 0x55c6265debf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.364+0000 7f88593e8640 1 -- 172.21.15.107:0/2000551248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c626377f70 con 0x55c6265e1740 2024-05-09T08:38:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.364+0000 7f8859be9640 1 --2- 172.21.15.107:0/2000551248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d80b0 0x55c6265debf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.364+0000 7f88593e8640 1 --2- 172.21.15.107:0/2000551248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265e1740 0x55c6265df130 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f884400b450 tx=0x7f8844014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8446031f8fd858d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.364+0000 7f883d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.364+0000 7f883d7fa640 1 -- 172.21.15.107:0/2000551248 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8844024040 con 0x55c6265e1740 2024-05-09T08:38:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.364+0000 7f883d7fa640 1 -- 172.21.15.107:0/2000551248 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f884401e810 con 0x55c6265e1740 2024-05-09T08:38:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.364+0000 7f885ec3a740 1 -- 172.21.15.107:0/2000551248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8844010c20 con 0x55c6265e1740 2024-05-09T08:38:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.364+0000 7f883d7fa640 1 -- 172.21.15.107:0/2000551248 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f884401ea10 con 0x55c6265e1740 2024-05-09T08:38:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.364+0000 7f885ec3a740 10 client.17157 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.364+0000 7f885ec3a740 1 -- 172.21.15.107:0/2000551248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c62620f170 con 0x55c6265e1740 2024-05-09T08:38:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.365+0000 7f883d7fa640 1 -- 172.21.15.107:0/2000551248 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8844010c20 con 0x55c6265e1740 2024-05-09T08:38:20.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.365+0000 7f883d7fa640 1 -- 172.21.15.107:0/2000551248 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f884402a040 con 0x55c6265e1740 2024-05-09T08:38:20.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.365+0000 7f885ec3a740 10 client.17157 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:20.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.365+0000 7f885ec3a740 10 client.17157 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:20.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.365+0000 7f885ec3a740 20 client.17157 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:20.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.365+0000 7f885ec3a740 1 --2- 172.21.15.107:0/2000551248 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c6265f1730 0x55c6265f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.365+0000 7f885ec3a740 4 client.17157 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:20.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.365+0000 7f885ec3a740 1 -- 172.21.15.107:0/2000551248 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6265490c0 con 0x55c6265f1730 2024-05-09T08:38:20.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.365+0000 7f8859be9640 1 --2- 172.21.15.107:0/2000551248 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c6265f1730 0x55c6265f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.366+0000 7f8859be9640 1 --2- 172.21.15.107:0/2000551248 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c6265f1730 0x55c6265f3b20 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9fa0ab095c32aa8 server_cookie=2bec8e55bf379591 in_seq=0 out_seq=0 2024-05-09T08:38:20.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.366+0000 7f883d7fa640 10 client.17157 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:20.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.366+0000 7f883d7fa640 1 -- 172.21.15.107:0/2000551248 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c6265490c0 con 0x55c6265f1730 2024-05-09T08:38:20.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.366+0000 7f883d7fa640 10 client.17157 handle_command_reply: tid=0 2024-05-09T08:38:20.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.366+0000 7f885ec3a740 1 client.17157 shutdown 2024-05-09T08:38:20.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.367+0000 7f885ec3a740 1 -- 172.21.15.107:0/2000551248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265e1740 msgr2=0x55c6265df130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.367+0000 7f885ec3a740 1 --2- 172.21.15.107:0/2000551248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265e1740 0x55c6265df130 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f884400b450 tx=0x7f8844014040 comp rx=0 tx=0).stop 2024-05-09T08:38:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.367+0000 7f885ec3a740 1 -- 172.21.15.107:0/2000551248 shutdown_connections 2024-05-09T08:38:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.367+0000 7f885ec3a740 1 --2- 172.21.15.107:0/2000551248 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c6265f1730 0x55c6265f3b20 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.367+0000 7f885ec3a740 1 --2- 172.21.15.107:0/2000551248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265e1740 0x55c6265df130 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.367+0000 7f885ec3a740 1 --2- 172.21.15.107:0/2000551248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d80b0 0x55c6265debf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.367+0000 7f885ec3a740 1 --2- 172.21.15.107:0/2000551248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265d69c0 0x55c6265de6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.367+0000 7f883d7fa640 0 client.17157 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.367+0000 7f885ec3a740 1 -- 172.21.15.107:0/2000551248 >> 172.21.15.107:0/2000551248 conn(0x55c62659f0a0 msgr2=0x55c6265d8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.367+0000 7f885ec3a740 1 -- 172.21.15.107:0/2000551248 shutdown_connections 2024-05-09T08:38:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.367+0000 7f885ec3a740 1 -- 172.21.15.107:0/2000551248 wait complete. 2024-05-09T08:38:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.367+0000 7f885ec3a740 20 client.17157 trim_cache size 0 max 16384 2024-05-09T08:38:20.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f885ec3a740 1 Processor -- start 2024-05-09T08:38:20.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f885ec3a740 1 -- start start 2024-05-09T08:38:20.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f885ec3a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265d69c0 0x55c6265ed500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f885ec3a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265d80b0 0x55c6265d1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f885ec3a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265e1740 0x55c6265f9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f885ec3a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f88541b34e0 con 0x55c6265d80b0 2024-05-09T08:38:20.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f885ec3a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f88541b0db0 con 0x55c6265e1740 2024-05-09T08:38:20.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f885ec3a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f88541aec00 con 0x55c6265d69c0 2024-05-09T08:38:20.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f8859be9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265d80b0 0x55c6265d1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f8859be9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265d80b0 0x55c6265d1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46746/0 (socket says 172.21.15.107:46746) 2024-05-09T08:38:20.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f88593e8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265d69c0 0x55c6265ed500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f8859be9640 1 -- 172.21.15.107:0/1624920561 learned_addr learned my addr 172.21.15.107:0/1624920561 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:20.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f88593e8640 1 --2- 172.21.15.107:0/1624920561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265e1740 0x55c6265f9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f8859be9640 1 -- 172.21.15.107:0/1624920561 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265d69c0 msgr2=0x55c6265ed500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:20.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f8859be9640 1 --2- 172.21.15.107:0/1624920561 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265d69c0 0x55c6265ed500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f8859be9640 1 -- 172.21.15.107:0/1624920561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265e1740 msgr2=0x55c6265f9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:20.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f8859be9640 1 --2- 172.21.15.107:0/1624920561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265e1740 0x55c6265f9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.388+0000 7f8859be9640 1 -- 172.21.15.107:0/1624920561 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6265e3f90 con 0x55c6265d80b0 2024-05-09T08:38:20.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.389+0000 7f8859be9640 1 --2- 172.21.15.107:0/1624920561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265d80b0 0x55c6265d1e60 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x55c6265f41b0 tx=0x7f8850005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f6942573196d40 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:20.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.389+0000 7f883f7fe640 1 -- 172.21.15.107:0/1624920561 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850004250 con 0x55c6265d80b0 2024-05-09T08:38:20.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.389+0000 7f883f7fe640 1 -- 172.21.15.107:0/1624920561 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f88500043f0 con 0x55c6265d80b0 2024-05-09T08:38:20.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.389+0000 7f883f7fe640 1 -- 172.21.15.107:0/1624920561 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850020110 con 0x55c6265d80b0 2024-05-09T08:38:20.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.390+0000 7f885ec3a740 1 -- 172.21.15.107:0/1624920561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265d80b0 msgr2=0x55c6265d1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.390+0000 7f885ec3a740 1 --2- 172.21.15.107:0/1624920561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265d80b0 0x55c6265d1e60 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x55c6265f41b0 tx=0x7f8850005f90 comp rx=0 tx=0).stop 2024-05-09T08:38:20.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.390+0000 7f885ec3a740 1 -- 172.21.15.107:0/1624920561 shutdown_connections 2024-05-09T08:38:20.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.390+0000 7f885ec3a740 1 --2- 172.21.15.107:0/1624920561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265e1740 0x55c6265f9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.390+0000 7f885ec3a740 1 --2- 172.21.15.107:0/1624920561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265d80b0 0x55c6265d1e60 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.390+0000 7f885ec3a740 1 --2- 172.21.15.107:0/1624920561 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265d69c0 0x55c6265ed500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.390+0000 7f885ec3a740 1 -- 172.21.15.107:0/1624920561 >> 172.21.15.107:0/1624920561 conn(0x55c62659f0a0 msgr2=0x55c6265e0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:20.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.390+0000 7f885ec3a740 1 -- 172.21.15.107:0/1624920561 shutdown_connections 2024-05-09T08:38:20.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.390+0000 7f885ec3a740 1 -- 172.21.15.107:0/1624920561 wait complete. 2024-05-09T08:38:20.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.390+0000 7f885ec3a740 1 -- start start 2024-05-09T08:38:20.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.390+0000 7f885ec3a740 1 Processor -- start 2024-05-09T08:38:20.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f885ec3a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265d69c0 0x55c6265e9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f885ec3a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d80b0 0x55c6265e69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f885ec3a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265e1740 0x55c6265e6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f885ec3a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f88541af270 con 0x55c6265d69c0 2024-05-09T08:38:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f885ec3a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f88541b06e0 con 0x55c6265d80b0 2024-05-09T08:38:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f885ec3a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f88541af940 con 0x55c6265e1740 2024-05-09T08:38:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f885ec3a740 10 client.0 _ll_register_callbacks cb 0x7f8850009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f88593e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265d69c0 0x55c6265e9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f88593e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265d69c0 0x55c6265e9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46762/0 (socket says 172.21.15.107:46762) 2024-05-09T08:38:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f88593e8640 1 -- 172.21.15.107:0/601582801 learned_addr learned my addr 172.21.15.107:0/601582801 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f8859be9640 1 --2- 172.21.15.107:0/601582801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d80b0 0x55c6265e69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f88593e8640 1 -- 172.21.15.107:0/601582801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265e1740 msgr2=0x55c6265e6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f88593e8640 1 --2- 172.21.15.107:0/601582801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265e1740 0x55c6265e6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f88593e8640 1 -- 172.21.15.107:0/601582801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d80b0 msgr2=0x55c6265e69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f88593e8640 1 --2- 172.21.15.107:0/601582801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d80b0 0x55c6265e69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f88593e8640 1 -- 172.21.15.107:0/601582801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6265e7700 con 0x55c6265d69c0 2024-05-09T08:38:20.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f88593e8640 1 --2- 172.21.15.107:0/601582801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265d69c0 0x55c6265e9e20 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f8844014010 tx=0x7f884400bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a3ba6815b7c1eb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:20.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f883d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:20.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f883d7fa640 1 -- 172.21.15.107:0/601582801 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884402de60 con 0x55c6265d69c0 2024-05-09T08:38:20.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f883d7fa640 1 -- 172.21.15.107:0/601582801 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f884401ebb0 con 0x55c6265d69c0 2024-05-09T08:38:20.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.391+0000 7f885ec3a740 1 -- 172.21.15.107:0/601582801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88500043f0 con 0x55c6265d69c0 2024-05-09T08:38:20.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.392+0000 7f883d7fa640 1 -- 172.21.15.107:0/601582801 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8844046050 con 0x55c6265d69c0 2024-05-09T08:38:20.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.392+0000 7f885ec3a740 10 client.17163 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:20.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.392+0000 7f885ec3a740 1 -- 172.21.15.107:0/601582801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c626658ae0 con 0x55c6265d69c0 2024-05-09T08:38:20.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.392+0000 7f883d7fa640 1 -- 172.21.15.107:0/601582801 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f88500043f0 con 0x55c6265d69c0 2024-05-09T08:38:20.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.392+0000 7f883d7fa640 1 -- 172.21.15.107:0/601582801 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f884402a070 con 0x55c6265d69c0 2024-05-09T08:38:20.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.392+0000 7f885ec3a740 10 client.17163 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:20.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.392+0000 7f885ec3a740 10 client.17163 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:20.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.392+0000 7f885ec3a740 20 client.17163 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:20.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.392+0000 7f885ec3a740 1 --2- 172.21.15.107:0/601582801 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c6266591a0 0x55c62665b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.392+0000 7f885ec3a740 4 client.17163 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:38:20.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.392+0000 7f885ec3a740 1 -- 172.21.15.107:0/601582801 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c626543410 con 0x55c6266591a0 2024-05-09T08:38:20.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.392+0000 7f8859be9640 1 --2- 172.21.15.107:0/601582801 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c6266591a0 0x55c62665b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.393+0000 7f883d7fa640 10 client.17163 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:20.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.393+0000 7f8859be9640 1 --2- 172.21.15.107:0/601582801 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c6266591a0 0x55c62665b590 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ac5f3e909091977 server_cookie=bc2cb791c6df946a in_seq=0 out_seq=0 2024-05-09T08:38:20.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.394+0000 7f883d7fa640 1 -- 172.21.15.107:0/601582801 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c626543410 con 0x55c6266591a0 2024-05-09T08:38:20.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.394+0000 7f883d7fa640 10 client.17163 handle_command_reply: tid=0 2024-05-09T08:38:20.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.394+0000 7f885ec3a740 1 client.17163 shutdown 2024-05-09T08:38:20.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885ec3a740 1 -- 172.21.15.107:0/601582801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265d69c0 msgr2=0x55c6265e9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885ec3a740 1 --2- 172.21.15.107:0/601582801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265d69c0 0x55c6265e9e20 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f8844014010 tx=0x7f884400bde0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885ec3a740 1 -- 172.21.15.107:0/601582801 shutdown_connections 2024-05-09T08:38:20.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885ec3a740 1 --2- 172.21.15.107:0/601582801 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c6266591a0 0x55c62665b590 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885ec3a740 1 --2- 172.21.15.107:0/601582801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6265e1740 0x55c6265e6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885ec3a740 1 --2- 172.21.15.107:0/601582801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c6265d80b0 0x55c6265e69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885ec3a740 1 --2- 172.21.15.107:0/601582801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c6265d69c0 0x55c6265e9e20 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885ec3a740 1 -- 172.21.15.107:0/601582801 >> 172.21.15.107:0/601582801 conn(0x55c62659f0a0 msgr2=0x55c6265de670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:20.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f883d7fa640 0 client.17163 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:20.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885ec3a740 1 -- 172.21.15.107:0/601582801 shutdown_connections 2024-05-09T08:38:20.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885ec3a740 1 -- 172.21.15.107:0/601582801 wait complete. 2024-05-09T08:38:20.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885ec3a740 20 client.17163 trim_cache size 0 max 16384 2024-05-09T08:38:20.398 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:38:20.398 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:38:20.398 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "17dc1c9d-52e8-433e-9d70-dfe8adcdad28", 2024-05-09T08:38:20.398 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:38:20.398 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:38:20.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885be4d640 1 -- 172.21.15.107:0/1217322833 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f88400a2d00 msgr2=0x7f88400a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885be4d640 1 --2- 172.21.15.107:0/1217322833 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f88400a2d00 0x7f88400a51c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f884c0bcfe0 tx=0x7f884c08ceb0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885be4d640 1 -- 172.21.15.107:0/1217322833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8854106710 msgr2=0x7f88541976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885be4d640 1 --2- 172.21.15.107:0/1217322833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8854106710 0x7f88541976a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f884c033040 tx=0x7f884c003970 comp rx=0 tx=0).stop 2024-05-09T08:38:20.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885be4d640 1 -- 172.21.15.107:0/1217322833 shutdown_connections 2024-05-09T08:38:20.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885be4d640 1 --2- 172.21.15.107:0/1217322833 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f88400a2d00 0x7f88400a51c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885be4d640 1 --2- 172.21.15.107:0/1217322833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8854106710 0x7f88541976a0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885be4d640 1 --2- 172.21.15.107:0/1217322833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88541010c0 0x7f8854197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885be4d640 1 --2- 172.21.15.107:0/1217322833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8854100700 0x7f8854196c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.395+0000 7f885be4d640 1 -- 172.21.15.107:0/1217322833 >> 172.21.15.107:0/1217322833 conn(0x7f8854077830 msgr2=0x7f8854102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:20.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.396+0000 7f885be4d640 1 -- 172.21.15.107:0/1217322833 shutdown_connections 2024-05-09T08:38:20.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.396+0000 7f885be4d640 1 -- 172.21.15.107:0/1217322833 wait complete. 2024-05-09T08:38:20.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:38:20.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:38:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.527+0000 7f6c95163640 1 -- 172.21.15.107:0/4017429514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c901010c0 msgr2=0x7f6c9010d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.527+0000 7f6c95163640 1 --2- 172.21.15.107:0/4017429514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c901010c0 0x7f6c9010d5d0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009cd0 tx=0x7f6c8802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.527+0000 7f6c95163640 1 -- 172.21.15.107:0/4017429514 shutdown_connections 2024-05-09T08:38:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.527+0000 7f6c95163640 1 --2- 172.21.15.107:0/4017429514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c901010c0 0x7f6c9010d5d0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.527+0000 7f6c95163640 1 --2- 172.21.15.107:0/4017429514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c90100700 0x7f6c90100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.527+0000 7f6c95163640 1 --2- 172.21.15.107:0/4017429514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c90106710 0x7f6c90106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.527+0000 7f6c95163640 1 -- 172.21.15.107:0/4017429514 >> 172.21.15.107:0/4017429514 conn(0x7f6c90077830 msgr2=0x7f6c90077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.527+0000 7f6c95163640 1 -- 172.21.15.107:0/4017429514 shutdown_connections 2024-05-09T08:38:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.527+0000 7f6c95163640 1 -- 172.21.15.107:0/4017429514 wait complete. 2024-05-09T08:38:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.528+0000 7f6c95163640 1 Processor -- start 2024-05-09T08:38:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.528+0000 7f6c95163640 1 -- start start 2024-05-09T08:38:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c95163640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c90100700 0x7f6c90198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c95163640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c901010c0 0x7f6c90198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c95163640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c90106710 0x7f6c9019c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c95163640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6c901109d0 con 0x7f6c901010c0 2024-05-09T08:38:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c95163640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6c90110850 con 0x7f6c90100700 2024-05-09T08:38:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c95163640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6c90110b50 con 0x7f6c90106710 2024-05-09T08:38:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c8e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c901010c0 0x7f6c90198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c8e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c901010c0 0x7f6c90198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46780/0 (socket says 172.21.15.107:46780) 2024-05-09T08:38:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c8e59c640 1 -- 172.21.15.107:0/642923240 learned_addr learned my addr 172.21.15.107:0/642923240 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c8ed9d640 1 --2- 172.21.15.107:0/642923240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c90100700 0x7f6c90198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c8f59e640 1 --2- 172.21.15.107:0/642923240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c90106710 0x7f6c9019c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c8ed9d640 1 -- 172.21.15.107:0/642923240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c90106710 msgr2=0x7f6c9019c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c8ed9d640 1 --2- 172.21.15.107:0/642923240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c90106710 0x7f6c9019c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c8ed9d640 1 -- 172.21.15.107:0/642923240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c901010c0 msgr2=0x7f6c90198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c8ed9d640 1 --2- 172.21.15.107:0/642923240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c901010c0 0x7f6c90198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c8ed9d640 1 -- 172.21.15.107:0/642923240 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9019d010 con 0x7f6c90100700 2024-05-09T08:38:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c8e59c640 1 --2- 172.21.15.107:0/642923240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c901010c0 0x7f6c90198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c8f59e640 1 --2- 172.21.15.107:0/642923240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c90106710 0x7f6c9019c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.529+0000 7f6c8ed9d640 1 --2- 172.21.15.107:0/642923240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c90100700 0x7f6c90198020 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000e9f0 tx=0x7f6c8000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:20.532 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:20 smithi107 ceph-mon[30122]: pgmap v152: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:38:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.530+0000 7f6c77fff640 1 -- 172.21.15.107:0/642923240 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8000ccc0 con 0x7f6c90100700 2024-05-09T08:38:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.530+0000 7f6c77fff640 1 -- 172.21.15.107:0/642923240 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c800108e0 con 0x7f6c90100700 2024-05-09T08:38:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.530+0000 7f6c77fff640 1 -- 172.21.15.107:0/642923240 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c80018880 con 0x7f6c90100700 2024-05-09T08:38:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.530+0000 7f6c95163640 1 -- 172.21.15.107:0/642923240 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c9019d300 con 0x7f6c90100700 2024-05-09T08:38:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.530+0000 7f6c95163640 1 -- 172.21.15.107:0/642923240 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c901a4b90 con 0x7f6c90100700 2024-05-09T08:38:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.532+0000 7f6c77fff640 1 -- 172.21.15.107:0/642923240 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6c80010a90 con 0x7f6c90100700 2024-05-09T08:38:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.532+0000 7f6c97e76740 1 Processor -- start 2024-05-09T08:38:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c97e76740 1 -- start start 2024-05-09T08:38:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c97e76740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667efd0 0x55e38667f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c97e76740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38667f980 0x55e38668e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c97e76740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38668e680 0x55e386690a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c97e76740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e3865e4d00 con 0x55e38667efd0 2024-05-09T08:38:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c97e76740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e38658c4c0 con 0x55e38668e680 2024-05-09T08:38:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c97e76740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55e3865e0cc0 con 0x55e38667f980 2024-05-09T08:38:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c8f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667efd0 0x55e38667f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c8e59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38668e680 0x55e386690a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c8f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667efd0 0x55e38667f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46794/0 (socket says 172.21.15.107:46794) 2024-05-09T08:38:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c8f59e640 1 -- 172.21.15.107:0/2415367867 learned_addr learned my addr 172.21.15.107:0/2415367867 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c77fff640 1 --2- 172.21.15.107:0/642923240 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6c780a2d80 0x7f6c780a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c8ed9d640 1 --2- 172.21.15.107:0/2415367867 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38667f980 0x55e38668e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c8e59c640 1 -- 172.21.15.107:0/2415367867 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38667f980 msgr2=0x55e38668e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c8e59c640 1 --2- 172.21.15.107:0/2415367867 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38667f980 0x55e38668e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c8e59c640 1 -- 172.21.15.107:0/2415367867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667efd0 msgr2=0x55e38667f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c8e59c640 1 --2- 172.21.15.107:0/2415367867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667efd0 0x55e38667f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c8e59c640 1 -- 172.21.15.107:0/2415367867 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3863c27d0 con 0x55e38668e680 2024-05-09T08:38:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c8f59e640 1 --2- 172.21.15.107:0/642923240 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6c780a2d80 0x7f6c780a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.533+0000 7f6c77fff640 1 -- 172.21.15.107:0/642923240 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6c800cfae0 con 0x7f6c90100700 2024-05-09T08:38:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.534+0000 7f6c8e59c640 1 --2- 172.21.15.107:0/2415367867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38668e680 0x55e386690a70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6c90100300 tx=0x7f6c840119a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=268ee4f04475a839 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.534+0000 7f6c757fa640 1 -- 172.21.15.107:0/2415367867 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8401e070 con 0x55e38668e680 2024-05-09T08:38:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.534+0000 7f6c757fa640 1 -- 172.21.15.107:0/2415367867 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c8400fa70 con 0x55e38668e680 2024-05-09T08:38:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.534+0000 7f6c8f59e640 1 --2- 172.21.15.107:0/642923240 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6c780a2d80 0x7f6c780a5240 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009a60 tx=0x7f6c8803f040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.534+0000 7f6c757fa640 1 -- 172.21.15.107:0/2415367867 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c84019b00 con 0x55e38668e680 2024-05-09T08:38:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.534+0000 7f6c97e76740 1 -- 172.21.15.107:0/2415367867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38668e680 msgr2=0x55e386690a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.534+0000 7f6c97e76740 1 --2- 172.21.15.107:0/2415367867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38668e680 0x55e386690a70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6c90100300 tx=0x7f6c840119a0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.535+0000 7f6c97e76740 1 -- 172.21.15.107:0/2415367867 shutdown_connections 2024-05-09T08:38:20.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.535+0000 7f6c97e76740 1 --2- 172.21.15.107:0/2415367867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38668e680 0x55e386690a70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.535+0000 7f6c97e76740 1 --2- 172.21.15.107:0/2415367867 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38667f980 0x55e38668e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.535+0000 7f6c97e76740 1 --2- 172.21.15.107:0/2415367867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667efd0 0x55e38667f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.535+0000 7f6c97e76740 1 -- 172.21.15.107:0/2415367867 >> 172.21.15.107:0/2415367867 conn(0x55e38667b5a0 msgr2=0x55e38667b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:20.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.535+0000 7f6c97e76740 1 -- 172.21.15.107:0/2415367867 shutdown_connections 2024-05-09T08:38:20.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.535+0000 7f6c97e76740 1 -- 172.21.15.107:0/2415367867 wait complete. 2024-05-09T08:38:20.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.535+0000 7f6c97e76740 1 -- start start 2024-05-09T08:38:20.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.535+0000 7f6c97e76740 1 Processor -- start 2024-05-09T08:38:20.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.535+0000 7f6c97e76740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667efd0 0x55e38668b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.535+0000 7f6c97e76740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667f980 0x55e38668bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.535+0000 7f6c97e76740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38668e680 0x55e38668c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.535+0000 7f6c97e76740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e386691490 con 0x55e38667f980 2024-05-09T08:38:20.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.535+0000 7f6c97e76740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e38659fb90 con 0x55e38667efd0 2024-05-09T08:38:20.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.535+0000 7f6c97e76740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55e386691610 con 0x55e38668e680 2024-05-09T08:38:20.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.535+0000 7f6c97e76740 10 client.0 _ll_register_callbacks cb 0x55e3865dbda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:20.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c8ed9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667efd0 0x55e38668b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c8ed9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667efd0 0x55e38668b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:46822/0 (socket says 172.21.15.107:46822) 2024-05-09T08:38:20.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c8e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667f980 0x55e38668bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c8ed9d640 1 -- 172.21.15.107:0/208841063 learned_addr learned my addr 172.21.15.107:0/208841063 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:20.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c8f59e640 1 --2- 172.21.15.107:0/208841063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38668e680 0x55e38668c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c8ed9d640 1 -- 172.21.15.107:0/208841063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38668e680 msgr2=0x55e38668c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:20.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c8ed9d640 1 --2- 172.21.15.107:0/208841063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38668e680 0x55e38668c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c8ed9d640 1 -- 172.21.15.107:0/208841063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667f980 msgr2=0x55e38668bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:20.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c8ed9d640 1 --2- 172.21.15.107:0/208841063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667f980 0x55e38668bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c8ed9d640 1 -- 172.21.15.107:0/208841063 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3862f83f0 con 0x55e38667efd0 2024-05-09T08:38:20.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c8f59e640 1 --2- 172.21.15.107:0/208841063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38668e680 0x55e38668c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:20.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c8ed9d640 1 --2- 172.21.15.107:0/208841063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667efd0 0x55e38668b590 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6c800980a0 tx=0x7f6c8009fff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee582e1605165afa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:20.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c72ff5640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:38:20.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c72ff5640 1 -- 172.21.15.107:0/208841063 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c800cd2f0 con 0x55e38667efd0 2024-05-09T08:38:20.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c97e76740 1 -- 172.21.15.107:0/208841063 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c8400fa70 con 0x55e38667efd0 2024-05-09T08:38:20.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c72ff5640 1 -- 172.21.15.107:0/208841063 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6c80010430 con 0x55e38667efd0 2024-05-09T08:38:20.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c72ff5640 1 -- 172.21.15.107:0/208841063 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6c8009bc40 con 0x55e38667efd0 2024-05-09T08:38:20.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c97e76740 10 client.26309 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:20.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c97e76740 1 -- 172.21.15.107:0/208841063 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e3862b80f0 con 0x55e38667efd0 2024-05-09T08:38:20.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.536+0000 7f6c72ff5640 1 -- 172.21.15.107:0/208841063 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c8400fa70 con 0x55e38667efd0 2024-05-09T08:38:20.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.537+0000 7f6c72ff5640 1 -- 172.21.15.107:0/208841063 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f6c80097990 con 0x55e38667efd0 2024-05-09T08:38:20.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.537+0000 7f6c97e76740 10 client.26309 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:20.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.537+0000 7f6c97e76740 10 client.26309 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:20.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.537+0000 7f6c97e76740 20 client.26309 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:20.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.537+0000 7f6c97e76740 1 --2- 172.21.15.107:0/208841063 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55e38669ea10 0x55e3866a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.537+0000 7f6c97e76740 4 client.26309 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:20.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.537+0000 7f6c97e76740 1 -- 172.21.15.107:0/208841063 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3865f2430 con 0x55e38669ea10 2024-05-09T08:38:20.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.537+0000 7f6c8e59c640 1 --2- 172.21.15.107:0/208841063 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55e38669ea10 0x55e3866a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.537+0000 7f6c8e59c640 1 --2- 172.21.15.107:0/208841063 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55e38669ea10 0x55e3866a0e00 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90af68f2956acd8d server_cookie=65e84c96d458d7c7 in_seq=0 out_seq=0 2024-05-09T08:38:20.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.537+0000 7f6c72ff5640 10 client.26309 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:20.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.538+0000 7f6c72ff5640 1 -- 172.21.15.107:0/208841063 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e3865f2430 con 0x55e38669ea10 2024-05-09T08:38:20.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.538+0000 7f6c72ff5640 10 client.26309 handle_command_reply: tid=0 2024-05-09T08:38:20.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.538+0000 7f6c97e76740 1 client.26309 shutdown 2024-05-09T08:38:20.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.538+0000 7f6c97e76740 1 -- 172.21.15.107:0/208841063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667efd0 msgr2=0x55e38668b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.538+0000 7f6c97e76740 1 --2- 172.21.15.107:0/208841063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667efd0 0x55e38668b590 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6c800980a0 tx=0x7f6c8009fff0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.538+0000 7f6c97e76740 1 -- 172.21.15.107:0/208841063 shutdown_connections 2024-05-09T08:38:20.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.538+0000 7f6c97e76740 1 --2- 172.21.15.107:0/208841063 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55e38669ea10 0x55e3866a0e00 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.538+0000 7f6c97e76740 1 --2- 172.21.15.107:0/208841063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38668e680 0x55e38668c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.538+0000 7f6c97e76740 1 --2- 172.21.15.107:0/208841063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667f980 0x55e38668bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.538+0000 7f6c72ff5640 0 client.26309 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:20.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.538+0000 7f6c97e76740 1 --2- 172.21.15.107:0/208841063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667efd0 0x55e38668b590 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.538+0000 7f6c97e76740 1 -- 172.21.15.107:0/208841063 >> 172.21.15.107:0/208841063 conn(0x55e38667b5a0 msgr2=0x55e3866813a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:20.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.538+0000 7f6c97e76740 1 -- 172.21.15.107:0/208841063 shutdown_connections 2024-05-09T08:38:20.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.538+0000 7f6c97e76740 1 -- 172.21.15.107:0/208841063 wait complete. 2024-05-09T08:38:20.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.538+0000 7f6c97e76740 20 client.26309 trim_cache size 0 max 16384 2024-05-09T08:38:20.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c97e76740 1 Processor -- start 2024-05-09T08:38:20.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c97e76740 1 -- start start 2024-05-09T08:38:20.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c97e76740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667efd0 0x55e38669a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c97e76740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38667f980 0x55e38668bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c97e76740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38668e680 0x55e3866a66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c97e76740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6c901af240 con 0x55e38668e680 2024-05-09T08:38:20.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c97e76740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6c901acb10 con 0x55e38667efd0 2024-05-09T08:38:20.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c97e76740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6c901aa960 con 0x55e38667f980 2024-05-09T08:38:20.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c8f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38668e680 0x55e3866a66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c8e59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38667f980 0x55e38668bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c8f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38668e680 0x55e3866a66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46806/0 (socket says 172.21.15.107:46806) 2024-05-09T08:38:20.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c8f59e640 1 -- 172.21.15.107:0/396119663 learned_addr learned my addr 172.21.15.107:0/396119663 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:20.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c8e59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38667f980 0x55e38668bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:60580/0 (socket says 172.21.15.107:60580) 2024-05-09T08:38:20.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c8ed9d640 1 --2- 172.21.15.107:0/396119663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667efd0 0x55e38669a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c8e59c640 1 -- 172.21.15.107:0/396119663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667efd0 msgr2=0x55e38669a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:20.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c8e59c640 1 --2- 172.21.15.107:0/396119663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667efd0 0x55e38669a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c8e59c640 1 -- 172.21.15.107:0/396119663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38668e680 msgr2=0x55e3866a66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:20.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c8e59c640 1 --2- 172.21.15.107:0/396119663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38668e680 0x55e3866a66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c8e59c640 1 -- 172.21.15.107:0/396119663 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e386690de0 con 0x55e38667f980 2024-05-09T08:38:20.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.556+0000 7f6c8ed9d640 1 --2- 172.21.15.107:0/396119663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667efd0 0x55e38669a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:20.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.557+0000 7f6c8e59c640 1 --2- 172.21.15.107:0/396119663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38667f980 0x55e38668bc60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6c8401c040 tx=0x7f6c84019780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65dd9aa8d8349ef0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:20.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.557+0000 7f6c74ff9640 1 -- 172.21.15.107:0/396119663 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8401e040 con 0x55e38667f980 2024-05-09T08:38:20.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.557+0000 7f6c74ff9640 1 -- 172.21.15.107:0/396119663 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c8402e070 con 0x55e38667f980 2024-05-09T08:38:20.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.557+0000 7f6c74ff9640 1 -- 172.21.15.107:0/396119663 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c840055d0 con 0x55e38667f980 2024-05-09T08:38:20.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 -- 172.21.15.107:0/396119663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38667f980 msgr2=0x55e38668bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 --2- 172.21.15.107:0/396119663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38667f980 0x55e38668bc60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6c8401c040 tx=0x7f6c84019780 comp rx=0 tx=0).stop 2024-05-09T08:38:20.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 -- 172.21.15.107:0/396119663 shutdown_connections 2024-05-09T08:38:20.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 --2- 172.21.15.107:0/396119663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38668e680 0x55e3866a66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 --2- 172.21.15.107:0/396119663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38667f980 0x55e38668bc60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 --2- 172.21.15.107:0/396119663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667efd0 0x55e38669a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 -- 172.21.15.107:0/396119663 >> 172.21.15.107:0/396119663 conn(0x55e38667b5a0 msgr2=0x55e38668b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:20.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 -- 172.21.15.107:0/396119663 shutdown_connections 2024-05-09T08:38:20.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 -- 172.21.15.107:0/396119663 wait complete. 2024-05-09T08:38:20.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 -- start start 2024-05-09T08:38:20.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 Processor -- start 2024-05-09T08:38:20.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667efd0 0x55e3866968d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667f980 0x55e386696e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38668e680 0x55e3866938a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6c901aafd0 con 0x55e38667efd0 2024-05-09T08:38:20.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6c901ac440 con 0x55e38667f980 2024-05-09T08:38:20.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.558+0000 7f6c97e76740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6c901ab6a0 con 0x55e38668e680 2024-05-09T08:38:20.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c97e76740 10 client.0 _ll_register_callbacks cb 0x7f6c8000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:20.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c8ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667efd0 0x55e3866968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c8ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667efd0 0x55e3866968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46822/0 (socket says 172.21.15.107:46822) 2024-05-09T08:38:20.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c8ed9d640 1 -- 172.21.15.107:0/3840678046 learned_addr learned my addr 172.21.15.107:0/3840678046 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:20.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c8e59c640 1 --2- 172.21.15.107:0/3840678046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667f980 0x55e386696e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c8f59e640 1 --2- 172.21.15.107:0/3840678046 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38668e680 0x55e3866938a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c8ed9d640 1 -- 172.21.15.107:0/3840678046 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38668e680 msgr2=0x55e3866938a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:20.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c8ed9d640 1 --2- 172.21.15.107:0/3840678046 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38668e680 0x55e3866938a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c8ed9d640 1 -- 172.21.15.107:0/3840678046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667f980 msgr2=0x55e386696e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c8ed9d640 1 --2- 172.21.15.107:0/3840678046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667f980 0x55e386696e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c8ed9d640 1 -- 172.21.15.107:0/3840678046 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e386693e30 con 0x55e38667efd0 2024-05-09T08:38:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c8ed9d640 1 --2- 172.21.15.107:0/3840678046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667efd0 0x55e3866968d0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f6c800cfea0 tx=0x7f6c80097bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69227a68889c8fcb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c72ff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c72ff5640 1 -- 172.21.15.107:0/3840678046 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8009b780 con 0x55e38667efd0 2024-05-09T08:38:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c72ff5640 1 -- 172.21.15.107:0/3840678046 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6c800a4d00 con 0x55e38667efd0 2024-05-09T08:38:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c97e76740 1 -- 172.21.15.107:0/3840678046 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c8402e070 con 0x55e38667efd0 2024-05-09T08:38:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c72ff5640 1 -- 172.21.15.107:0/3840678046 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6c800cd300 con 0x55e38667efd0 2024-05-09T08:38:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c97e76740 10 client.17181 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.559+0000 7f6c97e76740 1 -- 172.21.15.107:0/3840678046 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e3867057c0 con 0x55e38667efd0 2024-05-09T08:38:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.560+0000 7f6c72ff5640 1 -- 172.21.15.107:0/3840678046 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c8402e070 con 0x55e38667efd0 2024-05-09T08:38:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.560+0000 7f6c72ff5640 1 -- 172.21.15.107:0/3840678046 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f6c8009ab30 con 0x55e38667efd0 2024-05-09T08:38:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.560+0000 7f6c97e76740 10 client.17181 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.560+0000 7f6c97e76740 10 client.17181 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:20.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.560+0000 7f6c97e76740 20 client.17181 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:20.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.560+0000 7f6c97e76740 1 --2- 172.21.15.107:0/3840678046 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55e386705f10 0x55e386708330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:20.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.560+0000 7f6c97e76740 4 client.17181 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:38:20.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.560+0000 7f6c97e76740 1 -- 172.21.15.107:0/3840678046 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e3865ec780 con 0x55e386705f10 2024-05-09T08:38:20.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.560+0000 7f6c8e59c640 1 --2- 172.21.15.107:0/3840678046 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55e386705f10 0x55e386708330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:20.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.560+0000 7f6c72ff5640 10 client.17181 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:20.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.560+0000 7f6c8e59c640 1 --2- 172.21.15.107:0/3840678046 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55e386705f10 0x55e386708330 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9f85f4fda944913 server_cookie=924c4265d007c518 in_seq=0 out_seq=0 2024-05-09T08:38:20.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.561+0000 7f6c72ff5640 1 -- 172.21.15.107:0/3840678046 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e3865ec780 con 0x55e386705f10 2024-05-09T08:38:20.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.561+0000 7f6c72ff5640 10 client.17181 handle_command_reply: tid=0 2024-05-09T08:38:20.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.561+0000 7f6c97e76740 1 client.17181 shutdown 2024-05-09T08:38:20.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.561+0000 7f6c97e76740 1 -- 172.21.15.107:0/3840678046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667efd0 msgr2=0x55e3866968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.561+0000 7f6c97e76740 1 --2- 172.21.15.107:0/3840678046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667efd0 0x55e3866968d0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f6c800cfea0 tx=0x7f6c80097bc0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.561+0000 7f6c97e76740 1 -- 172.21.15.107:0/3840678046 shutdown_connections 2024-05-09T08:38:20.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.561+0000 7f6c97e76740 1 --2- 172.21.15.107:0/3840678046 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55e386705f10 0x55e386708330 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.561+0000 7f6c97e76740 1 --2- 172.21.15.107:0/3840678046 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e38668e680 0x55e3866938a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.561+0000 7f6c72ff5640 0 client.17181 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:20.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.561+0000 7f6c97e76740 1 --2- 172.21.15.107:0/3840678046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e38667f980 0x55e386696e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.561+0000 7f6c97e76740 1 --2- 172.21.15.107:0/3840678046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e38667efd0 0x55e3866968d0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.561+0000 7f6c97e76740 1 -- 172.21.15.107:0/3840678046 >> 172.21.15.107:0/3840678046 conn(0x55e38667b5a0 msgr2=0x55e38668bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:20.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.561+0000 7f6c97e76740 1 -- 172.21.15.107:0/3840678046 shutdown_connections 2024-05-09T08:38:20.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.561+0000 7f6c97e76740 1 -- 172.21.15.107:0/3840678046 wait complete. 2024-05-09T08:38:20.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.561+0000 7f6c97e76740 20 client.17181 trim_cache size 0 max 16384 2024-05-09T08:38:20.567 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:38:20.568 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:38:20.568 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:38:20.568 INFO:teuthology.orchestra.run.smithi107.stdout: "17dc1c9d-52e8-433e-9d70-dfe8adcdad28": { 2024-05-09T08:38:20.568 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:38:20.568 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "17dc1c9d-52e8-433e-9d70-dfe8adcdad28", 2024-05-09T08:38:20.568 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:38:20.568 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:38:20.568 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:38:20.568 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:38:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.562+0000 7f6c95163640 1 -- 172.21.15.107:0/642923240 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6c780a2d80 msgr2=0x7f6c780a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.562+0000 7f6c95163640 1 --2- 172.21.15.107:0/642923240 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6c780a2d80 0x7f6c780a5240 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009a60 tx=0x7f6c8803f040 comp rx=0 tx=0).stop 2024-05-09T08:38:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.562+0000 7f6c95163640 1 -- 172.21.15.107:0/642923240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c90100700 msgr2=0x7f6c90198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.562+0000 7f6c95163640 1 --2- 172.21.15.107:0/642923240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c90100700 0x7f6c90198020 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000e9f0 tx=0x7f6c8000eeb0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.562+0000 7f6c95163640 1 -- 172.21.15.107:0/642923240 shutdown_connections 2024-05-09T08:38:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.562+0000 7f6c95163640 1 --2- 172.21.15.107:0/642923240 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6c780a2d80 0x7f6c780a5240 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.562+0000 7f6c95163640 1 --2- 172.21.15.107:0/642923240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c90106710 0x7f6c9019c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.562+0000 7f6c95163640 1 --2- 172.21.15.107:0/642923240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c901010c0 0x7f6c90198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.562+0000 7f6c95163640 1 --2- 172.21.15.107:0/642923240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c90100700 0x7f6c90198020 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.562+0000 7f6c95163640 1 -- 172.21.15.107:0/642923240 >> 172.21.15.107:0/642923240 conn(0x7f6c90077830 msgr2=0x7f6c90068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.562+0000 7f6c95163640 1 -- 172.21.15.107:0/642923240 shutdown_connections 2024-05-09T08:38:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:20.562+0000 7f6c95163640 1 -- 172.21.15.107:0/642923240 wait complete. 2024-05-09T08:38:20.573 INFO:tasks.cephfs.filesystem:scrub status for tag:17dc1c9d-52e8-433e-9d70-dfe8adcdad28 - {'path': '/', 'tag': '17dc1c9d-52e8-433e-9d70-dfe8adcdad28', 'options': 'recursive,force'} 2024-05-09T08:38:20.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:20 smithi157 ceph-mon[33614]: pgmap v152: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:38:20.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:20 smithi110 ceph-mon[33622]: pgmap v152: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:38:21.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:21 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:38:21.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:21 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:21.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:21 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:38:21.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:21 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:38:21.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:21 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:21.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:21 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:38:21.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:21 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:21.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:21 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:38:21.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:21 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:21.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:21 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:21.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:21 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:38:21.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:21 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:21.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:21 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:38:21.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:21 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:38:21.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:21 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:21.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:21 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:38:21.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:21 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:21.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:21 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:38:21.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:21 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:21.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:21 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:21.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:21 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:38:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:21 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:21 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:38:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:21 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:38:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:21 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:21 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:38:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:21 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:21 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:38:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:21 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:21.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:21 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:22.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:22 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:38:22.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:22 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:22.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:22 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:38:22.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:22 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:22.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:22 smithi157 ceph-mon[33614]: pgmap v153: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:38:22.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:22 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:38:22.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:22 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:22.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:22 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:38:22.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:22 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:22.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:22 smithi110 ceph-mon[33622]: pgmap v153: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:38:22.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:22 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:38:22.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:22 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:22.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:22 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:38:22.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:22 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:22.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:22 smithi107 ceph-mon[30122]: pgmap v153: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:38:24.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:24 smithi157 ceph-mon[33614]: pgmap v154: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 37 MiB/s wr, 56 op/s 2024-05-09T08:38:24.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:24 smithi110 ceph-mon[33622]: pgmap v154: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 37 MiB/s wr, 56 op/s 2024-05-09T08:38:24.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:24 smithi107 ceph-mon[30122]: pgmap v154: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 37 MiB/s wr, 56 op/s 2024-05-09T08:38:26.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:26 smithi157 ceph-mon[33614]: pgmap v155: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 25 MiB/s wr, 48 op/s 2024-05-09T08:38:26.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:26 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:38:26.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:26 smithi110 ceph-mon[33622]: pgmap v155: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 25 MiB/s wr, 48 op/s 2024-05-09T08:38:26.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:26 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:38:26.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:26 smithi107 ceph-mon[30122]: pgmap v155: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 25 MiB/s wr, 48 op/s 2024-05-09T08:38:26.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:26 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:38:27.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:27 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:38:27.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:27 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:38:27.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:27 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:38:27.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:27 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:38:27.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:27 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:38:27.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:27 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:38:28.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:28 smithi157 ceph-mon[33614]: pgmap v156: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 37 MiB/s wr, 56 op/s 2024-05-09T08:38:28.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:28 smithi110 ceph-mon[33622]: pgmap v156: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 37 MiB/s wr, 56 op/s 2024-05-09T08:38:28.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:28 smithi107 ceph-mon[30122]: pgmap v156: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 37 MiB/s wr, 56 op/s 2024-05-09T08:38:30.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:30 smithi157 ceph-mon[33614]: pgmap v157: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 25 MiB/s wr, 48 op/s 2024-05-09T08:38:30.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:30 smithi110 ceph-mon[33622]: pgmap v157: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 25 MiB/s wr, 48 op/s 2024-05-09T08:38:30.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:30 smithi107 ceph-mon[30122]: pgmap v157: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 25 MiB/s wr, 48 op/s 2024-05-09T08:38:31.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:31 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:38:31.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:31 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:38:31.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:31 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:38:31.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:31 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:38:31.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:31 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:38:31.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:31 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:38:31.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:31 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:38:31.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:31 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:38:31.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:31 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:38:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:31 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:38:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:31 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:38:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:31 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:38:32.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:32 smithi157 ceph-mon[33614]: pgmap v158: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 25 MiB/s wr, 48 op/s 2024-05-09T08:38:32.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:32 smithi110 ceph-mon[33622]: pgmap v158: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 25 MiB/s wr, 48 op/s 2024-05-09T08:38:32.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:32 smithi107 ceph-mon[30122]: pgmap v158: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 25 MiB/s wr, 48 op/s 2024-05-09T08:38:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.971+0000 7f5ca6770640 1 -- 172.21.15.107:0/1615398396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca0101120 msgr2=0x7f5ca010d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.971+0000 7f5ca6770640 1 --2- 172.21.15.107:0/1615398396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca0101120 0x7f5ca010d5e0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f5c98009cd0 tx=0x7f5c9802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.971+0000 7f5ca6770640 1 -- 172.21.15.107:0/1615398396 shutdown_connections 2024-05-09T08:38:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.971+0000 7f5ca6770640 1 --2- 172.21.15.107:0/1615398396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca0101120 0x7f5ca010d5e0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.971+0000 7f5ca6770640 1 --2- 172.21.15.107:0/1615398396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ca0100760 0x7f5ca0100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.971+0000 7f5ca6770640 1 --2- 172.21.15.107:0/1615398396 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ca0106720 0x7f5ca0106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.971+0000 7f5ca6770640 1 -- 172.21.15.107:0/1615398396 >> 172.21.15.107:0/1615398396 conn(0x7f5ca00778b0 msgr2=0x7f5ca0077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.971+0000 7f5ca6770640 1 -- 172.21.15.107:0/1615398396 shutdown_connections 2024-05-09T08:38:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.971+0000 7f5ca6770640 1 -- 172.21.15.107:0/1615398396 wait complete. 2024-05-09T08:38:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.972+0000 7f5ca6770640 1 Processor -- start 2024-05-09T08:38:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.972+0000 7f5ca6770640 1 -- start start 2024-05-09T08:38:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.972+0000 7f5ca6770640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca0100760 0x7f5ca019a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.972+0000 7f5ca6770640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ca0101120 0x7f5ca019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5ca6770640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ca0106720 0x7f5ca019e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5ca6770640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca01109e0 con 0x7f5ca0100760 2024-05-09T08:38:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5ca6770640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca0110860 con 0x7f5ca0106720 2024-05-09T08:38:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5ca6770640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca0110b60 con 0x7f5ca0101120 2024-05-09T08:38:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5c9ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca0100760 0x7f5ca019a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5c9ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca0100760 0x7f5ca019a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39768/0 (socket says 172.21.15.107:39768) 2024-05-09T08:38:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5c9ffff640 1 -- 172.21.15.107:0/2515264556 learned_addr learned my addr 172.21.15.107:0/2515264556 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5ca4d0d640 1 --2- 172.21.15.107:0/2515264556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ca0106720 0x7f5ca019e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5c9f7fe640 1 --2- 172.21.15.107:0/2515264556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ca0101120 0x7f5ca019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5c9ffff640 1 -- 172.21.15.107:0/2515264556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ca0101120 msgr2=0x7f5ca019a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5c9ffff640 1 --2- 172.21.15.107:0/2515264556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ca0101120 0x7f5ca019a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5c9ffff640 1 -- 172.21.15.107:0/2515264556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ca0106720 msgr2=0x7f5ca019e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5c9ffff640 1 --2- 172.21.15.107:0/2515264556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ca0106720 0x7f5ca019e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5c9ffff640 1 -- 172.21.15.107:0/2515264556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ca019f0b0 con 0x7f5ca0100760 2024-05-09T08:38:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5ca4d0d640 1 --2- 172.21.15.107:0/2515264556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ca0106720 0x7f5ca019e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5c9ffff640 1 --2- 172.21.15.107:0/2515264556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca0100760 0x7f5ca019a0c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f5c9000e9f0 tx=0x7f5c9000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5c9d7fa640 1 -- 172.21.15.107:0/2515264556 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c9000cd80 con 0x7f5ca0100760 2024-05-09T08:38:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5ca6770640 1 -- 172.21.15.107:0/2515264556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ca019f3a0 con 0x7f5ca0100760 2024-05-09T08:38:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5c9d7fa640 1 -- 172.21.15.107:0/2515264556 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5c900054d0 con 0x7f5ca0100760 2024-05-09T08:38:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5c9d7fa640 1 -- 172.21.15.107:0/2515264556 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c90010490 con 0x7f5ca0100760 2024-05-09T08:38:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.973+0000 7f5ca6770640 1 -- 172.21.15.107:0/2515264556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ca01a6c30 con 0x7f5ca0100760 2024-05-09T08:38:32.982 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.976+0000 7f5c9d7fa640 1 -- 172.21.15.107:0/2515264556 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f5c90005020 con 0x7f5ca0100760 2024-05-09T08:38:32.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.976+0000 7f5ca9484740 1 Processor -- start 2024-05-09T08:38:32.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.976+0000 7f5ca9484740 1 -- start start 2024-05-09T08:38:32.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.976+0000 7f5ca9484740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e835450 0x56145e835830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:32.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.976+0000 7f5ca9484740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835d70 0x56145e8400f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:32.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.976+0000 7f5ca9484740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e840630 0x56145e842a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:32.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.976+0000 7f5ca9484740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56145e79a340 con 0x56145e835d70 2024-05-09T08:38:32.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.976+0000 7f5ca9484740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56145e742290 con 0x56145e840630 2024-05-09T08:38:32.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.976+0000 7f5ca9484740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56145e796300 con 0x56145e835450 2024-05-09T08:38:32.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.976+0000 7f5c9d7fa640 1 --2- 172.21.15.107:0/2515264556 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5c880a2da0 0x7f5c880a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:32.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5c9ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835d70 0x56145e8400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:32.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5ca4d0d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e835450 0x56145e835830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:32.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5c9ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835d70 0x56145e8400f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39784/0 (socket says 172.21.15.107:39784) 2024-05-09T08:38:32.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5c9ffff640 1 -- 172.21.15.107:0/2974010171 learned_addr learned my addr 172.21.15.107:0/2974010171 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5ca4d0d640 1 --2- 172.21.15.107:0/2515264556 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5c880a2da0 0x7f5c880a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5c9d7fa640 1 -- 172.21.15.107:0/2515264556 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f5c900cfbf0 con 0x7f5ca0100760 2024-05-09T08:38:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5c9f7fe640 1 --2- 172.21.15.107:0/2974010171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e840630 0x56145e842a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5ca4d0d640 1 -- 172.21.15.107:0/2974010171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e840630 msgr2=0x56145e842a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5ca4d0d640 1 --2- 172.21.15.107:0/2974010171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e840630 0x56145e842a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5ca4d0d640 1 -- 172.21.15.107:0/2974010171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835d70 msgr2=0x56145e8400f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5ca4d0d640 1 --2- 172.21.15.107:0/2974010171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835d70 0x56145e8400f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5ca4d0d640 1 -- 172.21.15.107:0/2974010171 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56145e7668e0 con 0x56145e835450 2024-05-09T08:38:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5c9ffff640 1 --2- 172.21.15.107:0/2974010171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835d70 0x56145e8400f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5c9f7fe640 1 --2- 172.21.15.107:0/2974010171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e840630 0x56145e842a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:38:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5ca4d0d640 1 --2- 172.21.15.107:0/2974010171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e835450 0x56145e835830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5c98006010 tx=0x7f5c98031ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99f8eb31e250fbef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5ca4d0d640 1 --2- 172.21.15.107:0/2515264556 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5c880a2da0 0x7f5c880a5260 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5c98009ca0 tx=0x7f5c98048000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5c867fc640 1 -- 172.21.15.107:0/2974010171 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c9804d070 con 0x56145e835450 2024-05-09T08:38:32.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5c867fc640 1 -- 172.21.15.107:0/2974010171 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5c98036040 con 0x56145e835450 2024-05-09T08:38:32.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.977+0000 7f5c867fc640 1 -- 172.21.15.107:0/2974010171 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c98030800 con 0x56145e835450 2024-05-09T08:38:32.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.978+0000 7f5ca9484740 1 -- 172.21.15.107:0/2974010171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e835450 msgr2=0x56145e835830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:32.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.978+0000 7f5ca9484740 1 --2- 172.21.15.107:0/2974010171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e835450 0x56145e835830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5c98006010 tx=0x7f5c98031ca0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.978+0000 7f5ca9484740 1 -- 172.21.15.107:0/2974010171 shutdown_connections 2024-05-09T08:38:32.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.978+0000 7f5ca9484740 1 --2- 172.21.15.107:0/2974010171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e840630 0x56145e842a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.978+0000 7f5ca9484740 1 --2- 172.21.15.107:0/2974010171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835d70 0x56145e8400f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.978+0000 7f5ca9484740 1 --2- 172.21.15.107:0/2974010171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e835450 0x56145e835830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.978+0000 7f5ca9484740 1 -- 172.21.15.107:0/2974010171 >> 172.21.15.107:0/2974010171 conn(0x56145e831960 msgr2=0x56145e831d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:32.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.978+0000 7f5ca9484740 1 -- 172.21.15.107:0/2974010171 shutdown_connections 2024-05-09T08:38:32.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.978+0000 7f5ca9484740 1 -- 172.21.15.107:0/2974010171 wait complete. 2024-05-09T08:38:32.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.979+0000 7f5ca9484740 1 -- start start 2024-05-09T08:38:32.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.979+0000 7f5ca9484740 1 Processor -- start 2024-05-09T08:38:32.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.979+0000 7f5ca9484740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e835450 0x56145e83d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:32.986 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.979+0000 7f5ca9484740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835d70 0x56145e83dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:32.986 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.979+0000 7f5ca9484740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e840630 0x56145e83dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:32.986 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.979+0000 7f5ca9484740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56145e843200 con 0x56145e835d70 2024-05-09T08:38:32.986 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.979+0000 7f5ca9484740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56145e755790 con 0x56145e835450 2024-05-09T08:38:32.986 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.979+0000 7f5ca9484740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56145e843380 con 0x56145e840630 2024-05-09T08:38:32.986 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.979+0000 7f5c9f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835d70 0x56145e83dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:32.986 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.979+0000 7f5ca4d0d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e840630 0x56145e83dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:32.986 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.979+0000 7f5ca9484740 10 client.0 _ll_register_callbacks cb 0x56145e790a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:32.986 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.979+0000 7f5ca4d0d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e840630 0x56145e83dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:34350/0 (socket says 172.21.15.107:34350) 2024-05-09T08:38:32.986 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.979+0000 7f5c9f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835d70 0x56145e83dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39800/0 (socket says 172.21.15.107:39800) 2024-05-09T08:38:32.986 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.979+0000 7f5ca4d0d640 1 -- 172.21.15.107:0/3968237160 learned_addr learned my addr 172.21.15.107:0/3968237160 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:32.986 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.979+0000 7f5c9ffff640 1 --2- 172.21.15.107:0/3968237160 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e835450 0x56145e83d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:32.987 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.980+0000 7f5ca4d0d640 1 -- 172.21.15.107:0/3968237160 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e835450 msgr2=0x56145e83d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:32.987 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.980+0000 7f5ca4d0d640 1 --2- 172.21.15.107:0/3968237160 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e835450 0x56145e83d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.987 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.980+0000 7f5ca4d0d640 1 -- 172.21.15.107:0/3968237160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835d70 msgr2=0x56145e83dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:32.987 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.980+0000 7f5ca4d0d640 1 --2- 172.21.15.107:0/3968237160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835d70 0x56145e83dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.987 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.980+0000 7f5ca4d0d640 1 -- 172.21.15.107:0/3968237160 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56145e4ae210 con 0x56145e840630 2024-05-09T08:38:32.987 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.980+0000 7f5c9f7fe640 1 --2- 172.21.15.107:0/3968237160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835d70 0x56145e83dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:32.987 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.980+0000 7f5ca4d0d640 1 --2- 172.21.15.107:0/3968237160 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e840630 0x56145e83dff0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5c98005cc0 tx=0x7f5c98004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1bb031132a3ed1f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:32.987 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.980+0000 7f5c7f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:38:32.987 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.980+0000 7f5c7f7fe640 1 -- 172.21.15.107:0/3968237160 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c98056070 con 0x56145e840630 2024-05-09T08:38:32.987 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.980+0000 7f5c7f7fe640 1 -- 172.21.15.107:0/3968237160 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f5c980307a0 con 0x56145e840630 2024-05-09T08:38:32.987 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.980+0000 7f5ca9484740 1 -- 172.21.15.107:0/3968237160 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c98036040 con 0x56145e840630 2024-05-09T08:38:32.987 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.980+0000 7f5c7f7fe640 1 -- 172.21.15.107:0/3968237160 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5c9804d070 con 0x56145e840630 2024-05-09T08:38:32.987 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.980+0000 7f5ca9484740 10 client.26542 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:32.987 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.980+0000 7f5ca9484740 1 -- 172.21.15.107:0/3968237160 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56145e57a620 con 0x56145e840630 2024-05-09T08:38:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.980+0000 7f5c7f7fe640 1 -- 172.21.15.107:0/3968237160 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5c98036040 con 0x56145e840630 2024-05-09T08:38:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.981+0000 7f5c7f7fe640 1 -- 172.21.15.107:0/3968237160 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5c98030070 con 0x56145e840630 2024-05-09T08:38:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.981+0000 7f5ca9484740 10 client.26542 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.981+0000 7f5ca9484740 10 client.26542 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.981+0000 7f5ca9484740 20 client.26542 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.981+0000 7f5ca9484740 1 --2- 172.21.15.107:0/3968237160 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56145e850150 0x56145e852540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.981+0000 7f5ca9484740 4 client.26542 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.981+0000 7f5ca9484740 1 -- 172.21.15.107:0/3968237160 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56145e7a7a70 con 0x56145e850150 2024-05-09T08:38:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.981+0000 7f5c9ffff640 1 --2- 172.21.15.107:0/3968237160 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56145e850150 0x56145e852540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.981+0000 7f5c9ffff640 1 --2- 172.21.15.107:0/3968237160 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56145e850150 0x56145e852540 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9c028760642dfd5 server_cookie=f738cdcb87be6f0c in_seq=0 out_seq=0 2024-05-09T08:38:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.981+0000 7f5c7f7fe640 10 client.26542 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.982+0000 7f5c7f7fe640 1 -- 172.21.15.107:0/3968237160 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56145e7a7a70 con 0x56145e850150 2024-05-09T08:38:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.982+0000 7f5c7f7fe640 10 client.26542 handle_command_reply: tid=0 2024-05-09T08:38:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.982+0000 7f5ca9484740 1 client.26542 shutdown 2024-05-09T08:38:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.982+0000 7f5ca9484740 1 -- 172.21.15.107:0/3968237160 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e840630 msgr2=0x56145e83dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.982+0000 7f5ca9484740 1 --2- 172.21.15.107:0/3968237160 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e840630 0x56145e83dff0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5c98005cc0 tx=0x7f5c98004270 comp rx=0 tx=0).stop 2024-05-09T08:38:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.982+0000 7f5ca9484740 1 -- 172.21.15.107:0/3968237160 shutdown_connections 2024-05-09T08:38:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.982+0000 7f5ca9484740 1 --2- 172.21.15.107:0/3968237160 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56145e850150 0x56145e852540 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.982+0000 7f5ca9484740 1 --2- 172.21.15.107:0/3968237160 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e840630 0x56145e83dff0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.982+0000 7f5ca9484740 1 --2- 172.21.15.107:0/3968237160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835d70 0x56145e83dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.982+0000 7f5ca9484740 1 --2- 172.21.15.107:0/3968237160 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e835450 0x56145e83d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.982+0000 7f5c7f7fe640 0 client.26542 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.982+0000 7f5ca9484740 1 -- 172.21.15.107:0/3968237160 >> 172.21.15.107:0/3968237160 conn(0x56145e831960 msgr2=0x56145e8376e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.982+0000 7f5ca9484740 1 -- 172.21.15.107:0/3968237160 shutdown_connections 2024-05-09T08:38:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.982+0000 7f5ca9484740 1 -- 172.21.15.107:0/3968237160 wait complete. 2024-05-09T08:38:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.982+0000 7f5ca9484740 20 client.26542 trim_cache size 0 max 16384 2024-05-09T08:38:33.002 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.998+0000 7f5ca9484740 1 Processor -- start 2024-05-09T08:38:33.002 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.998+0000 7f5ca9484740 1 -- start start 2024-05-09T08:38:33.002 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.998+0000 7f5ca9484740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835450 0x56145e84c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:33.002 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.998+0000 7f5ca9484740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e835d70 0x56145e83c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:33.002 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.998+0000 7f5ca9484740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e840630 0x56145e83de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:33.002 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.998+0000 7f5ca9484740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca01b3540 con 0x56145e835450 2024-05-09T08:38:33.002 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.998+0000 7f5ca9484740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca01b0e10 con 0x56145e835d70 2024-05-09T08:38:33.002 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.998+0000 7f5ca9484740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca01aec60 con 0x56145e840630 2024-05-09T08:38:33.002 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.998+0000 7f5c9ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835450 0x56145e84c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:33.002 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.998+0000 7f5c9ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835450 0x56145e84c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39814/0 (socket says 172.21.15.107:39814) 2024-05-09T08:38:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.998+0000 7f5c9ffff640 1 -- 172.21.15.107:0/3699623848 learned_addr learned my addr 172.21.15.107:0/3699623848 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.998+0000 7f5ca4d0d640 1 --2- 172.21.15.107:0/3699623848 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e840630 0x56145e83de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.999+0000 7f5c9f7fe640 1 --2- 172.21.15.107:0/3699623848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e835d70 0x56145e83c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.999+0000 7f5ca4d0d640 1 -- 172.21.15.107:0/3699623848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e835d70 msgr2=0x56145e83c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.999+0000 7f5ca4d0d640 1 --2- 172.21.15.107:0/3699623848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e835d70 0x56145e83c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.999+0000 7f5ca4d0d640 1 -- 172.21.15.107:0/3699623848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835450 msgr2=0x56145e84c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.999+0000 7f5ca4d0d640 1 --2- 172.21.15.107:0/3699623848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835450 0x56145e84c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.999+0000 7f5ca4d0d640 1 -- 172.21.15.107:0/3699623848 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56145e835b60 con 0x56145e840630 2024-05-09T08:38:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.999+0000 7f5c9f7fe640 1 --2- 172.21.15.107:0/3699623848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e835d70 0x56145e83c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.999+0000 7f5c9ffff640 1 --2- 172.21.15.107:0/3699623848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835450 0x56145e84c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.999+0000 7f5ca4d0d640 1 --2- 172.21.15.107:0/3699623848 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e840630 0x56145e83de10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5c9802ae00 tx=0x7f5c98005d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffaa6dba9d6e4171 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.999+0000 7f5c85ffb640 1 -- 172.21.15.107:0/3699623848 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c98056040 con 0x56145e840630 2024-05-09T08:38:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.999+0000 7f5c85ffb640 1 -- 172.21.15.107:0/3699623848 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5c98036040 con 0x56145e840630 2024-05-09T08:38:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:32.999+0000 7f5c85ffb640 1 -- 172.21.15.107:0/3699623848 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c98052a40 con 0x56145e840630 2024-05-09T08:38:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.000+0000 7f5ca9484740 1 -- 172.21.15.107:0/3699623848 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e840630 msgr2=0x56145e83de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.000+0000 7f5ca9484740 1 --2- 172.21.15.107:0/3699623848 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e840630 0x56145e83de10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5c9802ae00 tx=0x7f5c98005d10 comp rx=0 tx=0).stop 2024-05-09T08:38:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.000+0000 7f5ca9484740 1 -- 172.21.15.107:0/3699623848 shutdown_connections 2024-05-09T08:38:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.000+0000 7f5ca9484740 1 --2- 172.21.15.107:0/3699623848 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e840630 0x56145e83de10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.000+0000 7f5ca9484740 1 --2- 172.21.15.107:0/3699623848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e835d70 0x56145e83c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.000+0000 7f5ca9484740 1 --2- 172.21.15.107:0/3699623848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835450 0x56145e84c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.000+0000 7f5ca9484740 1 -- 172.21.15.107:0/3699623848 >> 172.21.15.107:0/3699623848 conn(0x56145e831960 msgr2=0x56145e830170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.000+0000 7f5ca9484740 1 -- 172.21.15.107:0/3699623848 shutdown_connections 2024-05-09T08:38:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.000+0000 7f5ca9484740 1 -- 172.21.15.107:0/3699623848 wait complete. 2024-05-09T08:38:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.000+0000 7f5ca9484740 1 -- start start 2024-05-09T08:38:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5ca9484740 1 Processor -- start 2024-05-09T08:38:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5ca9484740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835450 0x56145e848600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5ca9484740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e835d70 0x56145e848b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5ca9484740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e840630 0x56145e839ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5ca9484740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca01af2d0 con 0x56145e835450 2024-05-09T08:38:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5ca9484740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca01b0740 con 0x56145e840630 2024-05-09T08:38:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5ca9484740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca01af9a0 con 0x56145e835d70 2024-05-09T08:38:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5ca9484740 10 client.0 _ll_register_callbacks cb 0x7f5c900979b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5c9ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835450 0x56145e848600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5c9ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835450 0x56145e848600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39822/0 (socket says 172.21.15.107:39822) 2024-05-09T08:38:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5c9ffff640 1 -- 172.21.15.107:0/1242928070 learned_addr learned my addr 172.21.15.107:0/1242928070 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5c9f7fe640 1 --2- 172.21.15.107:0/1242928070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e835d70 0x56145e848b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5ca4d0d640 1 --2- 172.21.15.107:0/1242928070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e840630 0x56145e839ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5c9ffff640 1 -- 172.21.15.107:0/1242928070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e835d70 msgr2=0x56145e848b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5c9ffff640 1 --2- 172.21.15.107:0/1242928070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e835d70 0x56145e848b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5c9ffff640 1 -- 172.21.15.107:0/1242928070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e840630 msgr2=0x56145e839ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5c9ffff640 1 --2- 172.21.15.107:0/1242928070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e840630 0x56145e839ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5c9ffff640 1 -- 172.21.15.107:0/1242928070 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56145e83a700 con 0x56145e835450 2024-05-09T08:38:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5ca4d0d640 1 --2- 172.21.15.107:0/1242928070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e840630 0x56145e839ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:38:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5c9f7fe640 1 --2- 172.21.15.107:0/1242928070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e835d70 0x56145e848b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5c9ffff640 1 --2- 172.21.15.107:0/1242928070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835450 0x56145e848600 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f5c90098780 tx=0x7f5c90021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9ae8c095e48555 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.001+0000 7f5c7f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.002+0000 7f5c7f7fe640 1 -- 172.21.15.107:0/1242928070 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c9009c4b0 con 0x56145e835450 2024-05-09T08:38:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.002+0000 7f5c7f7fe640 1 -- 172.21.15.107:0/1242928070 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f5c90016db0 con 0x56145e835450 2024-05-09T08:38:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.002+0000 7f5ca9484740 1 -- 172.21.15.107:0/1242928070 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c98036040 con 0x56145e835450 2024-05-09T08:38:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.002+0000 7f5c7f7fe640 1 -- 172.21.15.107:0/1242928070 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5c9009bdb0 con 0x56145e835450 2024-05-09T08:38:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.002+0000 7f5ca9484740 10 client.17199 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.002+0000 7f5ca9484740 1 -- 172.21.15.107:0/1242928070 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56145e8b5ff0 con 0x56145e835450 2024-05-09T08:38:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.002+0000 7f5c7f7fe640 1 -- 172.21.15.107:0/1242928070 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5c98036040 con 0x56145e835450 2024-05-09T08:38:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.002+0000 7f5c7f7fe640 1 -- 172.21.15.107:0/1242928070 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5c900cc070 con 0x56145e835450 2024-05-09T08:38:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.002+0000 7f5ca9484740 10 client.17199 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.002+0000 7f5ca9484740 10 client.17199 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.002+0000 7f5ca9484740 20 client.17199 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.002+0000 7f5ca9484740 1 --2- 172.21.15.107:0/1242928070 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56145e8b6710 0x56145e8b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.002+0000 7f5ca9484740 4 client.17199 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:38:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.002+0000 7f5ca9484740 1 -- 172.21.15.107:0/1242928070 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56145e7a1dc0 con 0x56145e8b6710 2024-05-09T08:38:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.002+0000 7f5c9f7fe640 1 --2- 172.21.15.107:0/1242928070 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56145e8b6710 0x56145e8b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.003+0000 7f5c7f7fe640 10 client.17199 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.003+0000 7f5c9f7fe640 1 --2- 172.21.15.107:0/1242928070 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56145e8b6710 0x56145e8b8b30 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a301d8151033f74 server_cookie=74a33d913fb30072 in_seq=0 out_seq=0 2024-05-09T08:38:33.017 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.018+0000 7f5c7f7fe640 1 -- 172.21.15.107:0/1242928070 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56145e7a1dc0 con 0x56145e8b6710 2024-05-09T08:38:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.018+0000 7f5c7f7fe640 10 client.17199 handle_command_reply: tid=0 2024-05-09T08:38:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.018+0000 7f5ca9484740 1 client.17199 shutdown 2024-05-09T08:38:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.018+0000 7f5ca9484740 1 -- 172.21.15.107:0/1242928070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835450 msgr2=0x56145e848600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.018+0000 7f5ca9484740 1 --2- 172.21.15.107:0/1242928070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835450 0x56145e848600 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f5c90098780 tx=0x7f5c90021000 comp rx=0 tx=0).stop 2024-05-09T08:38:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.018+0000 7f5ca9484740 1 -- 172.21.15.107:0/1242928070 shutdown_connections 2024-05-09T08:38:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.018+0000 7f5ca9484740 1 --2- 172.21.15.107:0/1242928070 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56145e8b6710 0x56145e8b8b30 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.018+0000 7f5ca9484740 1 --2- 172.21.15.107:0/1242928070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56145e840630 0x56145e839ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.018+0000 7f5ca9484740 1 --2- 172.21.15.107:0/1242928070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56145e835d70 0x56145e848b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.018+0000 7f5c7f7fe640 0 client.17199 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.018+0000 7f5ca9484740 1 --2- 172.21.15.107:0/1242928070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56145e835450 0x56145e848600 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.018+0000 7f5ca9484740 1 -- 172.21.15.107:0/1242928070 >> 172.21.15.107:0/1242928070 conn(0x56145e831960 msgr2=0x56145e830a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.018+0000 7f5ca9484740 1 -- 172.21.15.107:0/1242928070 shutdown_connections 2024-05-09T08:38:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.018+0000 7f5ca9484740 1 -- 172.21.15.107:0/1242928070 wait complete. 2024-05-09T08:38:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.018+0000 7f5ca9484740 20 client.17199 trim_cache size 0 max 16384 2024-05-09T08:38:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:38:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:38:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:38:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:38:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.019+0000 7f5ca6770640 1 -- 172.21.15.107:0/2515264556 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5c880a2da0 msgr2=0x7f5c880a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.019+0000 7f5ca6770640 1 --2- 172.21.15.107:0/2515264556 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5c880a2da0 0x7f5c880a5260 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5c98009ca0 tx=0x7f5c98048000 comp rx=0 tx=0).stop 2024-05-09T08:38:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.019+0000 7f5ca6770640 1 -- 172.21.15.107:0/2515264556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca0100760 msgr2=0x7f5ca019a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.019+0000 7f5ca6770640 1 --2- 172.21.15.107:0/2515264556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca0100760 0x7f5ca019a0c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f5c9000e9f0 tx=0x7f5c9000edf0 comp rx=0 tx=0).stop 2024-05-09T08:38:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.019+0000 7f5ca6770640 1 -- 172.21.15.107:0/2515264556 shutdown_connections 2024-05-09T08:38:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.019+0000 7f5ca6770640 1 --2- 172.21.15.107:0/2515264556 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5c880a2da0 0x7f5c880a5260 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.019+0000 7f5ca6770640 1 --2- 172.21.15.107:0/2515264556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ca0106720 0x7f5ca019e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.019+0000 7f5ca6770640 1 --2- 172.21.15.107:0/2515264556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ca0101120 0x7f5ca019a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.019+0000 7f5ca6770640 1 --2- 172.21.15.107:0/2515264556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ca0100760 0x7f5ca019a0c0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.019+0000 7f5ca6770640 1 -- 172.21.15.107:0/2515264556 >> 172.21.15.107:0/2515264556 conn(0x7f5ca00778b0 msgr2=0x7f5ca0068ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.019+0000 7f5ca6770640 1 -- 172.21.15.107:0/2515264556 shutdown_connections 2024-05-09T08:38:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:33.019+0000 7f5ca6770640 1 -- 172.21.15.107:0/2515264556 wait complete. 2024-05-09T08:38:34.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:34 smithi157 ceph-mon[33614]: pgmap v159: 129 pgs: 129 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 37 MiB/s wr, 56 op/s 2024-05-09T08:38:34.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:34 smithi110 ceph-mon[33622]: pgmap v159: 129 pgs: 129 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 37 MiB/s wr, 56 op/s 2024-05-09T08:38:34.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:34 smithi107 ceph-mon[30122]: pgmap v159: 129 pgs: 129 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 37 MiB/s wr, 56 op/s 2024-05-09T08:38:36.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:36 smithi110 ceph-mon[33622]: pgmap v160: 129 pgs: 129 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:38:36.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:36 smithi107 ceph-mon[30122]: pgmap v160: 129 pgs: 129 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:38:36.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:36 smithi157 ceph-mon[33614]: pgmap v160: 129 pgs: 129 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:38:38.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:38 smithi110 ceph-mon[33622]: pgmap v161: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 37 MiB/s wr, 26 op/s 2024-05-09T08:38:38.645 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:38 smithi157 ceph-mon[33614]: pgmap v161: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 37 MiB/s wr, 26 op/s 2024-05-09T08:38:38.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:38 smithi107 ceph-mon[30122]: pgmap v161: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 37 MiB/s wr, 26 op/s 2024-05-09T08:38:40.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:40 smithi110 ceph-mon[33622]: pgmap v162: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:40.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:40 smithi107 ceph-mon[30122]: pgmap v162: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:40.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:40 smithi157 ceph-mon[33614]: pgmap v162: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:42.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:42 smithi110 ceph-mon[33622]: pgmap v163: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:42.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:42 smithi107 ceph-mon[30122]: pgmap v163: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:42.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:42 smithi157 ceph-mon[33614]: pgmap v163: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:44.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:44 smithi110 ceph-mon[33622]: pgmap v164: 129 pgs: 129 active+clean; 6.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:38:44.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:44 smithi107 ceph-mon[30122]: pgmap v164: 129 pgs: 129 active+clean; 6.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:38:44.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:44 smithi157 ceph-mon[33614]: pgmap v164: 129 pgs: 129 active+clean; 6.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:38:46.121 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.120+0000 7fe1178a5640 1 -- 172.21.15.107:0/2080143820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1101033e0 msgr2=0x7fe11010b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:46.121 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.120+0000 7fe1178a5640 1 --2- 172.21.15.107:0/2080143820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1101033e0 0x7fe11010b8c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fe100009900 tx=0x7fe10002a510 comp rx=0 tx=0).stop 2024-05-09T08:38:46.121 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.120+0000 7fe1178a5640 1 -- 172.21.15.107:0/2080143820 shutdown_connections 2024-05-09T08:38:46.121 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.120+0000 7fe1178a5640 1 --2- 172.21.15.107:0/2080143820 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe11010be00 0x7fe11010e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.121 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.120+0000 7fe1178a5640 1 --2- 172.21.15.107:0/2080143820 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1101033e0 0x7fe11010b8c0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.120+0000 7fe1178a5640 1 --2- 172.21.15.107:0/2080143820 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe110102ac0 0x7fe110102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.120+0000 7fe1178a5640 1 -- 172.21.15.107:0/2080143820 >> 172.21.15.107:0/2080143820 conn(0x7fe1100778b0 msgr2=0x7fe110077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.120+0000 7fe1178a5640 1 -- 172.21.15.107:0/2080143820 shutdown_connections 2024-05-09T08:38:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.120+0000 7fe1178a5640 1 -- 172.21.15.107:0/2080143820 wait complete. 2024-05-09T08:38:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe1178a5640 1 Processor -- start 2024-05-09T08:38:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe1178a5640 1 -- start start 2024-05-09T08:38:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe1178a5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe110102ac0 0x7fe110197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe1178a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1101033e0 0x7fe110198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe1178a5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe11010be00 0x7fe11019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe1178a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe110111ab0 con 0x7fe1101033e0 2024-05-09T08:38:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe1178a5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe110111930 con 0x7fe110102ac0 2024-05-09T08:38:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe1178a5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe110111c30 con 0x7fe11010be00 2024-05-09T08:38:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe114e40640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1101033e0 0x7fe110198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe114e40640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1101033e0 0x7fe110198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40160/0 (socket says 172.21.15.107:40160) 2024-05-09T08:38:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe114e40640 1 -- 172.21.15.107:0/721258417 learned_addr learned my addr 172.21.15.107:0/721258417 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe115641640 1 --2- 172.21.15.107:0/721258417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe110102ac0 0x7fe110197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe115e42640 1 --2- 172.21.15.107:0/721258417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe11010be00 0x7fe11019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe115641640 1 -- 172.21.15.107:0/721258417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe11010be00 msgr2=0x7fe11019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe115641640 1 --2- 172.21.15.107:0/721258417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe11010be00 0x7fe11019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe115641640 1 -- 172.21.15.107:0/721258417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1101033e0 msgr2=0x7fe110198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe115641640 1 --2- 172.21.15.107:0/721258417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1101033e0 0x7fe110198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.121+0000 7fe115641640 1 -- 172.21.15.107:0/721258417 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11019cfc0 con 0x7fe110102ac0 2024-05-09T08:38:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.122+0000 7fe115641640 1 --2- 172.21.15.107:0/721258417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe110102ac0 0x7fe110197fd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00d720 tx=0x7fe10c00dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.122+0000 7fe1067fc640 1 -- 172.21.15.107:0/721258417 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c017070 con 0x7fe110102ac0 2024-05-09T08:38:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.122+0000 7fe1178a5640 1 -- 172.21.15.107:0/721258417 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe11019d2b0 con 0x7fe110102ac0 2024-05-09T08:38:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.122+0000 7fe1178a5640 1 -- 172.21.15.107:0/721258417 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1101a4b40 con 0x7fe110102ac0 2024-05-09T08:38:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.122+0000 7fe1067fc640 1 -- 172.21.15.107:0/721258417 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe10c00de30 con 0x7fe110102ac0 2024-05-09T08:38:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.122+0000 7fe1067fc640 1 -- 172.21.15.107:0/721258417 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c004e00 con 0x7fe110102ac0 2024-05-09T08:38:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe11a5b4740 1 Processor -- start 2024-05-09T08:38:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe1067fc640 1 -- 172.21.15.107:0/721258417 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe10c0099d0 con 0x7fe110102ac0 2024-05-09T08:38:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe11a5b4740 1 -- start start 2024-05-09T08:38:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe11a5b4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a290450 0x56253a290830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe11a5b4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290d70 0x56253a29b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe11a5b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a29b630 0x56253a29da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe11a5b4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56253a1f5340 con 0x56253a290d70 2024-05-09T08:38:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe11a5b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56253a19d290 con 0x56253a29b630 2024-05-09T08:38:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe11a5b4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56253a1f1300 con 0x56253a290450 2024-05-09T08:38:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe115641640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290d70 0x56253a29b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe114e40640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a29b630 0x56253a29da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe115641640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290d70 0x56253a29b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40162/0 (socket says 172.21.15.107:40162) 2024-05-09T08:38:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe115641640 1 -- 172.21.15.107:0/860535943 learned_addr learned my addr 172.21.15.107:0/860535943 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe115e42640 1 --2- 172.21.15.107:0/860535943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a290450 0x56253a290830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe115641640 1 -- 172.21.15.107:0/860535943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a290450 msgr2=0x56253a290830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe115641640 1 --2- 172.21.15.107:0/860535943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a290450 0x56253a290830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe115641640 1 -- 172.21.15.107:0/860535943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a29b630 msgr2=0x56253a29da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe115641640 1 --2- 172.21.15.107:0/860535943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a29b630 0x56253a29da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.125+0000 7fe115641640 1 -- 172.21.15.107:0/860535943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56253a1c18e0 con 0x56253a290d70 2024-05-09T08:38:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.126+0000 7fe114e40640 1 --2- 172.21.15.107:0/860535943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a29b630 0x56253a29da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.126+0000 7fe1067fc640 1 --2- 172.21.15.107:0/721258417 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe0fc0a2da0 0x7fe0fc0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.126+0000 7fe115641640 1 --2- 172.21.15.107:0/860535943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290d70 0x56253a29b0f0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0cc800 tx=0x7fe10c0d5b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c135add0517904cc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.126+0000 7fe115e42640 1 --2- 172.21.15.107:0/721258417 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe0fc0a2da0 0x7fe0fc0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.126+0000 7fe1067fc640 1 -- 172.21.15.107:0/721258417 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe10c0cf4a0 con 0x7fe110102ac0 2024-05-09T08:38:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.126+0000 7fe0fb7fe640 1 -- 172.21.15.107:0/860535943 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c097900 con 0x56253a290d70 2024-05-09T08:38:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.126+0000 7fe0fb7fe640 1 -- 172.21.15.107:0/860535943 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe10c004400 con 0x56253a290d70 2024-05-09T08:38:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.126+0000 7fe0fb7fe640 1 -- 172.21.15.107:0/860535943 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c09afc0 con 0x56253a290d70 2024-05-09T08:38:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.127+0000 7fe11a5b4740 1 -- 172.21.15.107:0/860535943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290d70 msgr2=0x56253a29b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.127+0000 7fe115e42640 1 --2- 172.21.15.107:0/721258417 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe0fc0a2da0 0x7fe0fc0a5260 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe108005fd0 tx=0x7fe108009500 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.127+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/860535943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290d70 0x56253a29b0f0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0cc800 tx=0x7fe10c0d5b10 comp rx=0 tx=0).stop 2024-05-09T08:38:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.127+0000 7fe11a5b4740 1 -- 172.21.15.107:0/860535943 shutdown_connections 2024-05-09T08:38:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.127+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/860535943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a29b630 0x56253a29da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.127+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/860535943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290d70 0x56253a29b0f0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.127+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/860535943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a290450 0x56253a290830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.127+0000 7fe11a5b4740 1 -- 172.21.15.107:0/860535943 >> 172.21.15.107:0/860535943 conn(0x56253a28c960 msgr2=0x56253a28cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.127+0000 7fe11a5b4740 1 -- 172.21.15.107:0/860535943 shutdown_connections 2024-05-09T08:38:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.127+0000 7fe11a5b4740 1 -- 172.21.15.107:0/860535943 wait complete. 2024-05-09T08:38:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.127+0000 7fe11a5b4740 1 -- start start 2024-05-09T08:38:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.127+0000 7fe11a5b4740 1 Processor -- start 2024-05-09T08:38:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe11a5b4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290450 0x56253a298610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe11a5b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a290d70 0x56253a298b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe11a5b4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a29b630 0x56253a299090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe11a5b4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56253a29e260 con 0x56253a290450 2024-05-09T08:38:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe11a5b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56253a1b0790 con 0x56253a290d70 2024-05-09T08:38:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe11a5b4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56253a29e3e0 con 0x56253a29b630 2024-05-09T08:38:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe11a5b4740 10 client.0 _ll_register_callbacks cb 0x56253a1eba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe115641640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290450 0x56253a298610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe115641640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290450 0x56253a298610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40172/0 (socket says 172.21.15.107:40172) 2024-05-09T08:38:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe115641640 1 -- 172.21.15.107:0/3683613982 learned_addr learned my addr 172.21.15.107:0/3683613982 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe114e40640 1 --2- 172.21.15.107:0/3683613982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a290d70 0x56253a298b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe115e42640 1 --2- 172.21.15.107:0/3683613982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a29b630 0x56253a299090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe115641640 1 -- 172.21.15.107:0/3683613982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a29b630 msgr2=0x56253a299090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe115641640 1 --2- 172.21.15.107:0/3683613982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a29b630 0x56253a299090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe115641640 1 -- 172.21.15.107:0/3683613982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a290d70 msgr2=0x56253a298b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe115641640 1 --2- 172.21.15.107:0/3683613982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a290d70 0x56253a298b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe115641640 1 -- 172.21.15.107:0/3683613982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562539f09210 con 0x56253a290450 2024-05-09T08:38:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe115641640 1 --2- 172.21.15.107:0/3683613982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290450 0x56253a298610 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0cce90 tx=0x7fe10c0d2e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d17d869dca49533 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.128+0000 7fe0f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.129+0000 7fe0f8ff9640 1 -- 172.21.15.107:0/3683613982 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c09b1a0 con 0x56253a290450 2024-05-09T08:38:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.129+0000 7fe0f8ff9640 1 -- 172.21.15.107:0/3683613982 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe10c0046c0 con 0x56253a290450 2024-05-09T08:38:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.129+0000 7fe11a5b4740 1 -- 172.21.15.107:0/3683613982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56253a202a70 con 0x56253a290450 2024-05-09T08:38:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.129+0000 7fe0f8ff9640 1 -- 172.21.15.107:0/3683613982 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe10c09b430 con 0x56253a290450 2024-05-09T08:38:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.129+0000 7fe11a5b4740 10 client.17211 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.129+0000 7fe11a5b4740 1 -- 172.21.15.107:0/3683613982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562539fd5620 con 0x56253a290450 2024-05-09T08:38:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.129+0000 7fe0f8ff9640 1 -- 172.21.15.107:0/3683613982 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56253a202a70 con 0x56253a290450 2024-05-09T08:38:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.129+0000 7fe0f8ff9640 1 -- 172.21.15.107:0/3683613982 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe10c0ccb10 con 0x56253a290450 2024-05-09T08:38:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.129+0000 7fe11a5b4740 10 client.17211 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.129+0000 7fe11a5b4740 10 client.17211 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.129+0000 7fe11a5b4740 20 client.17211 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.129+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/3683613982 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56253a2ab260 0x56253a2ad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.129+0000 7fe11a5b4740 4 client.17211 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.129+0000 7fe11a5b4740 1 -- 172.21.15.107:0/3683613982 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56253a1fcdc0 con 0x56253a2ab260 2024-05-09T08:38:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.130+0000 7fe114e40640 1 --2- 172.21.15.107:0/3683613982 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56253a2ab260 0x56253a2ad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.130+0000 7fe114e40640 1 --2- 172.21.15.107:0/3683613982 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56253a2ab260 0x56253a2ad650 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9022b5456ccd55fb server_cookie=68ac6f837eea3a20 in_seq=0 out_seq=0 2024-05-09T08:38:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.130+0000 7fe0f8ff9640 10 client.17211 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.131+0000 7fe0f8ff9640 1 -- 172.21.15.107:0/3683613982 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56253a1fcdc0 con 0x56253a2ab260 2024-05-09T08:38:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.131+0000 7fe0f8ff9640 10 client.17211 handle_command_reply: tid=0 2024-05-09T08:38:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.131+0000 7fe11a5b4740 1 client.17211 shutdown 2024-05-09T08:38:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.131+0000 7fe11a5b4740 1 -- 172.21.15.107:0/3683613982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290450 msgr2=0x56253a298610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.131+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/3683613982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290450 0x56253a298610 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0cce90 tx=0x7fe10c0d2e40 comp rx=0 tx=0).stop 2024-05-09T08:38:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.131+0000 7fe11a5b4740 1 -- 172.21.15.107:0/3683613982 shutdown_connections 2024-05-09T08:38:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.131+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/3683613982 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56253a2ab260 0x56253a2ad650 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.131+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/3683613982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a29b630 0x56253a299090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.131+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/3683613982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a290d70 0x56253a298b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.131+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/3683613982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290450 0x56253a298610 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.131+0000 7fe0f8ff9640 0 client.17211 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.131+0000 7fe11a5b4740 1 -- 172.21.15.107:0/3683613982 >> 172.21.15.107:0/3683613982 conn(0x56253a28c960 msgr2=0x56253a292690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.131+0000 7fe11a5b4740 1 -- 172.21.15.107:0/3683613982 shutdown_connections 2024-05-09T08:38:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.131+0000 7fe11a5b4740 1 -- 172.21.15.107:0/3683613982 wait complete. 2024-05-09T08:38:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.131+0000 7fe11a5b4740 20 client.17211 trim_cache size 0 max 16384 2024-05-09T08:38:46.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.150+0000 7fe11a5b4740 1 Processor -- start 2024-05-09T08:38:46.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.150+0000 7fe11a5b4740 1 -- start start 2024-05-09T08:38:46.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.150+0000 7fe11a5b4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a290450 0x56253a29ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.150+0000 7fe11a5b4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290d70 0x56253a2b0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.150+0000 7fe11a5b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a29b630 0x56253a2b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.150+0000 7fe11a5b4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101b12b0 con 0x56253a290d70 2024-05-09T08:38:46.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.150+0000 7fe11a5b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101aeb80 con 0x56253a29b630 2024-05-09T08:38:46.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.150+0000 7fe11a5b4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101ac9d0 con 0x56253a290450 2024-05-09T08:38:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.150+0000 7fe114e40640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290d70 0x56253a2b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.150+0000 7fe114e40640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290d70 0x56253a2b0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40188/0 (socket says 172.21.15.107:40188) 2024-05-09T08:38:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.150+0000 7fe114e40640 1 -- 172.21.15.107:0/3578559779 learned_addr learned my addr 172.21.15.107:0/3578559779 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.150+0000 7fe115641640 1 --2- 172.21.15.107:0/3578559779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a290450 0x56253a29ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.150+0000 7fe115e42640 1 --2- 172.21.15.107:0/3578559779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a29b630 0x56253a2b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.151+0000 7fe114e40640 1 -- 172.21.15.107:0/3578559779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a290450 msgr2=0x56253a29ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.151+0000 7fe114e40640 1 --2- 172.21.15.107:0/3578559779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a290450 0x56253a29ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.151+0000 7fe114e40640 1 -- 172.21.15.107:0/3578559779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a29b630 msgr2=0x56253a2b07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.151+0000 7fe114e40640 1 --2- 172.21.15.107:0/3578559779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a29b630 0x56253a2b07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.151+0000 7fe114e40640 1 -- 172.21.15.107:0/3578559779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56253a2975c0 con 0x56253a290d70 2024-05-09T08:38:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.151+0000 7fe114e40640 1 --2- 172.21.15.107:0/3578559779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290d70 0x56253a2b0290 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fe100006010 tx=0x7fe1000058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc2a0cc728ce9db server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.151+0000 7fe0faffd640 1 -- 172.21.15.107:0/3578559779 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe100049030 con 0x56253a290d70 2024-05-09T08:38:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.151+0000 7fe0faffd640 1 -- 172.21.15.107:0/3578559779 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1000043e0 con 0x56253a290d70 2024-05-09T08:38:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.151+0000 7fe0faffd640 1 -- 172.21.15.107:0/3578559779 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe100035040 con 0x56253a290d70 2024-05-09T08:38:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.152+0000 7fe11a5b4740 1 -- 172.21.15.107:0/3578559779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290d70 msgr2=0x56253a2b0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.152+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/3578559779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290d70 0x56253a2b0290 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fe100006010 tx=0x7fe1000058d0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.152+0000 7fe11a5b4740 1 -- 172.21.15.107:0/3578559779 shutdown_connections 2024-05-09T08:38:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.152+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/3578559779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a29b630 0x56253a2b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.152+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/3578559779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290d70 0x56253a2b0290 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.152+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/3578559779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a290450 0x56253a29ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.152+0000 7fe11a5b4740 1 -- 172.21.15.107:0/3578559779 >> 172.21.15.107:0/3578559779 conn(0x56253a28c960 msgr2=0x56253a28cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.152+0000 7fe11a5b4740 1 -- 172.21.15.107:0/3578559779 shutdown_connections 2024-05-09T08:38:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.152+0000 7fe11a5b4740 1 -- 172.21.15.107:0/3578559779 wait complete. 2024-05-09T08:38:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.152+0000 7fe11a5b4740 1 -- start start 2024-05-09T08:38:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.152+0000 7fe11a5b4740 1 Processor -- start 2024-05-09T08:38:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe11a5b4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290450 0x56253a2b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe11a5b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a290d70 0x56253a2ad970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe11a5b4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a29b630 0x56253a2adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe11a5b4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101ad040 con 0x56253a290450 2024-05-09T08:38:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe11a5b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101ae4b0 con 0x56253a290d70 2024-05-09T08:38:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe11a5b4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101ad710 con 0x56253a29b630 2024-05-09T08:38:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe11a5b4740 10 client.0 _ll_register_callbacks cb 0x7fe100009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe115641640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290450 0x56253a2b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe115641640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290450 0x56253a2b05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40200/0 (socket says 172.21.15.107:40200) 2024-05-09T08:38:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe115641640 1 -- 172.21.15.107:0/1060477587 learned_addr learned my addr 172.21.15.107:0/1060477587 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe114e40640 1 --2- 172.21.15.107:0/1060477587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a290d70 0x56253a2ad970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe115e42640 1 --2- 172.21.15.107:0/1060477587 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a29b630 0x56253a2adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe115641640 1 -- 172.21.15.107:0/1060477587 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a29b630 msgr2=0x56253a2adeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe115641640 1 --2- 172.21.15.107:0/1060477587 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a29b630 0x56253a2adeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe115641640 1 -- 172.21.15.107:0/1060477587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a290d70 msgr2=0x56253a2ad970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe115641640 1 --2- 172.21.15.107:0/1060477587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a290d70 0x56253a2ad970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe115641640 1 -- 172.21.15.107:0/1060477587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56253a293400 con 0x56253a290450 2024-05-09T08:38:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe115641640 1 --2- 172.21.15.107:0/1060477587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290450 0x56253a2b05e0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0d53a0 tx=0x7fe10c00d6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f0bec13f358629 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe0f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.153+0000 7fe0f8ff9640 1 -- 172.21.15.107:0/1060477587 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c09b1a0 con 0x56253a290450 2024-05-09T08:38:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.154+0000 7fe0f8ff9640 1 -- 172.21.15.107:0/1060477587 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe10c0ccb10 con 0x56253a290450 2024-05-09T08:38:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.154+0000 7fe11a5b4740 1 -- 172.21.15.107:0/1060477587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1000043e0 con 0x56253a290450 2024-05-09T08:38:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.154+0000 7fe0f8ff9640 1 -- 172.21.15.107:0/1060477587 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe10c09afc0 con 0x56253a290450 2024-05-09T08:38:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.154+0000 7fe11a5b4740 10 client.17217 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.154+0000 7fe11a5b4740 1 -- 172.21.15.107:0/1060477587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56253a311200 con 0x56253a290450 2024-05-09T08:38:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.154+0000 7fe0f8ff9640 1 -- 172.21.15.107:0/1060477587 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1000043e0 con 0x56253a290450 2024-05-09T08:38:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.154+0000 7fe0f8ff9640 1 -- 172.21.15.107:0/1060477587 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe10c097cd0 con 0x56253a290450 2024-05-09T08:38:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.154+0000 7fe11a5b4740 10 client.17217 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.154+0000 7fe11a5b4740 10 client.17217 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.154+0000 7fe11a5b4740 20 client.17217 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.154+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/1060477587 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56253a311700 0x56253a313b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.154+0000 7fe11a5b4740 4 client.17217 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:38:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.154+0000 7fe11a5b4740 1 -- 172.21.15.107:0/1060477587 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56253a17a8e0 con 0x56253a311700 2024-05-09T08:38:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.154+0000 7fe114e40640 1 --2- 172.21.15.107:0/1060477587 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56253a311700 0x56253a313b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.155+0000 7fe0f8ff9640 10 client.17217 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.155+0000 7fe114e40640 1 --2- 172.21.15.107:0/1060477587 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56253a311700 0x56253a313b20 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16136f8a53b5066 server_cookie=1cfa181adaa1282e in_seq=0 out_seq=0 2024-05-09T08:38:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.169+0000 7fe0f8ff9640 1 -- 172.21.15.107:0/1060477587 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56253a17a8e0 con 0x56253a311700 2024-05-09T08:38:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.169+0000 7fe0f8ff9640 10 client.17217 handle_command_reply: tid=0 2024-05-09T08:38:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.169+0000 7fe11a5b4740 1 client.17217 shutdown 2024-05-09T08:38:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.169+0000 7fe11a5b4740 1 -- 172.21.15.107:0/1060477587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290450 msgr2=0x56253a2b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.169+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/1060477587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290450 0x56253a2b05e0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0d53a0 tx=0x7fe10c00d6a0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.169+0000 7fe11a5b4740 1 -- 172.21.15.107:0/1060477587 shutdown_connections 2024-05-09T08:38:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.169+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/1060477587 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56253a311700 0x56253a313b20 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.169+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/1060477587 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56253a29b630 0x56253a2adeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.169+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/1060477587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56253a290d70 0x56253a2ad970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.169+0000 7fe0f8ff9640 0 client.17217 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.169+0000 7fe11a5b4740 1 --2- 172.21.15.107:0/1060477587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56253a290450 0x56253a2b05e0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.169+0000 7fe11a5b4740 1 -- 172.21.15.107:0/1060477587 >> 172.21.15.107:0/1060477587 conn(0x56253a28c960 msgr2=0x56253a291d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.169+0000 7fe11a5b4740 1 -- 172.21.15.107:0/1060477587 shutdown_connections 2024-05-09T08:38:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.169+0000 7fe11a5b4740 1 -- 172.21.15.107:0/1060477587 wait complete. 2024-05-09T08:38:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.170+0000 7fe11a5b4740 20 client.17217 trim_cache size 0 max 16384 2024-05-09T08:38:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:38:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:38:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:38:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:38:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.170+0000 7fe1178a5640 1 -- 172.21.15.107:0/721258417 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe0fc0a2da0 msgr2=0x7fe0fc0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.170+0000 7fe1178a5640 1 --2- 172.21.15.107:0/721258417 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe0fc0a2da0 0x7fe0fc0a5260 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe108005fd0 tx=0x7fe108009500 comp rx=0 tx=0).stop 2024-05-09T08:38:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.170+0000 7fe1178a5640 1 -- 172.21.15.107:0/721258417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe110102ac0 msgr2=0x7fe110197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.170+0000 7fe1178a5640 1 --2- 172.21.15.107:0/721258417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe110102ac0 0x7fe110197fd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00d720 tx=0x7fe10c00dbe0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.170+0000 7fe1178a5640 1 -- 172.21.15.107:0/721258417 shutdown_connections 2024-05-09T08:38:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.170+0000 7fe1178a5640 1 --2- 172.21.15.107:0/721258417 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe0fc0a2da0 0x7fe0fc0a5260 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.170+0000 7fe1178a5640 1 --2- 172.21.15.107:0/721258417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe11010be00 0x7fe11019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.170+0000 7fe1178a5640 1 --2- 172.21.15.107:0/721258417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1101033e0 0x7fe110198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.170+0000 7fe1178a5640 1 --2- 172.21.15.107:0/721258417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe110102ac0 0x7fe110197fd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.170+0000 7fe1178a5640 1 -- 172.21.15.107:0/721258417 >> 172.21.15.107:0/721258417 conn(0x7fe1100778b0 msgr2=0x7fe1100fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.170+0000 7fe1178a5640 1 -- 172.21.15.107:0/721258417 shutdown_connections 2024-05-09T08:38:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:46.170+0000 7fe1178a5640 1 -- 172.21.15.107:0/721258417 wait complete. 2024-05-09T08:38:46.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:46 smithi110 ceph-mon[33622]: pgmap v165: 129 pgs: 129 active+clean; 6.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:46.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:46 smithi107 ceph-mon[30122]: pgmap v165: 129 pgs: 129 active+clean; 6.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:46.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:46 smithi157 ceph-mon[33614]: pgmap v165: 129 pgs: 129 active+clean; 6.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:48.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:48 smithi110 ceph-mon[33622]: pgmap v166: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 37 MiB/s wr, 31 op/s 2024-05-09T08:38:48.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:48 smithi107 ceph-mon[30122]: pgmap v166: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 37 MiB/s wr, 31 op/s 2024-05-09T08:38:48.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:48 smithi157 ceph-mon[33614]: pgmap v166: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 37 MiB/s wr, 31 op/s 2024-05-09T08:38:50.575 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:38:50.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:38:50.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:50 smithi110 ceph-mon[33622]: pgmap v167: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:50.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.689+0000 7f0414c9a640 1 -- 172.21.15.107:0/1544196429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04101010c0 msgr2=0x7f041010d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:50.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.689+0000 7f0414c9a640 1 --2- 172.21.15.107:0/1544196429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04101010c0 0x7f041010d5d0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f0408009cd0 tx=0x7f040802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.689+0000 7f0414c9a640 1 -- 172.21.15.107:0/1544196429 shutdown_connections 2024-05-09T08:38:50.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.689+0000 7f0414c9a640 1 --2- 172.21.15.107:0/1544196429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04101010c0 0x7f041010d5d0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.689+0000 7f0414c9a640 1 --2- 172.21.15.107:0/1544196429 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0410100700 0x7f0410100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.689+0000 7f0414c9a640 1 --2- 172.21.15.107:0/1544196429 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0410106710 0x7f0410106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.689+0000 7f0414c9a640 1 -- 172.21.15.107:0/1544196429 >> 172.21.15.107:0/1544196429 conn(0x7f0410077830 msgr2=0x7f0410077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:50.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.689+0000 7f0414c9a640 1 -- 172.21.15.107:0/1544196429 shutdown_connections 2024-05-09T08:38:50.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.689+0000 7f0414c9a640 1 -- 172.21.15.107:0/1544196429 wait complete. 2024-05-09T08:38:50.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f0414c9a640 1 Processor -- start 2024-05-09T08:38:50.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f0414c9a640 1 -- start start 2024-05-09T08:38:50.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f0414c9a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0410100700 0x7f0410198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f0414c9a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04101010c0 0x7f0410199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f0414c9a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0410106710 0x7f04101a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f0414c9a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0410110990 con 0x7f0410100700 2024-05-09T08:38:50.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f0414c9a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0410110810 con 0x7f04101010c0 2024-05-09T08:38:50.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f0414c9a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0410110b10 con 0x7f0410106710 2024-05-09T08:38:50.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f040e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0410100700 0x7f0410198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f040e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0410100700 0x7f0410198e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54192/0 (socket says 172.21.15.107:54192) 2024-05-09T08:38:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f040e59c640 1 -- 172.21.15.107:0/1574820245 learned_addr learned my addr 172.21.15.107:0/1574820245 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f040dd9b640 1 --2- 172.21.15.107:0/1574820245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04101010c0 0x7f0410199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f040ed9d640 1 --2- 172.21.15.107:0/1574820245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0410106710 0x7f04101a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f040e59c640 1 -- 172.21.15.107:0/1574820245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0410106710 msgr2=0x7f04101a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f040e59c640 1 --2- 172.21.15.107:0/1574820245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0410106710 0x7f04101a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f040e59c640 1 -- 172.21.15.107:0/1574820245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04101010c0 msgr2=0x7f0410199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f040e59c640 1 --2- 172.21.15.107:0/1574820245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04101010c0 0x7f0410199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f040e59c640 1 -- 172.21.15.107:0/1574820245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04101a43f0 con 0x7f0410100700 2024-05-09T08:38:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f040ed9d640 1 --2- 172.21.15.107:0/1574820245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0410106710 0x7f04101a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:38:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.690+0000 7f040e59c640 1 --2- 172.21.15.107:0/1574820245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0410100700 0x7f0410198e30 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f040000d950 tx=0x7f040000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.691+0000 7f03ff7fe640 1 -- 172.21.15.107:0/1574820245 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0400017070 con 0x7f0410100700 2024-05-09T08:38:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.691+0000 7f03ff7fe640 1 -- 172.21.15.107:0/1574820245 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0400009c30 con 0x7f0410100700 2024-05-09T08:38:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.691+0000 7f03ff7fe640 1 -- 172.21.15.107:0/1574820245 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040000f620 con 0x7f0410100700 2024-05-09T08:38:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.691+0000 7f0414c9a640 1 -- 172.21.15.107:0/1574820245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04101a45c0 con 0x7f0410100700 2024-05-09T08:38:50.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.691+0000 7f0414c9a640 1 -- 172.21.15.107:0/1574820245 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04101a4b30 con 0x7f0410100700 2024-05-09T08:38:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.693+0000 7f03ff7fe640 1 -- 172.21.15.107:0/1574820245 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f04000109a0 con 0x7f0410100700 2024-05-09T08:38:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.694+0000 7f04179a7740 1 Processor -- start 2024-05-09T08:38:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.694+0000 7f04179a7740 1 -- start start 2024-05-09T08:38:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.694+0000 7f03ff7fe640 1 --2- 172.21.15.107:0/1574820245 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f03f80a2d80 0x7f03f80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.694+0000 7f04179a7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424280 0x564701424660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.694+0000 7f04179a7740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564701424ba0 0x56470142ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.694+0000 7f04179a7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56470142f270 0x564701431660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.694+0000 7f04179a7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564701389d00 con 0x564701424280 2024-05-09T08:38:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.694+0000 7f04179a7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5647013314c0 con 0x56470142f270 2024-05-09T08:38:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.694+0000 7f04179a7740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564701385cc0 con 0x564701424ba0 2024-05-09T08:38:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.694+0000 7f040ed9d640 1 --2- 172.21.15.107:0/1574820245 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f03f80a2d80 0x7f03f80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.694+0000 7f03ff7fe640 1 -- 172.21.15.107:0/1574820245 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f04000cf2e0 con 0x7f0410100700 2024-05-09T08:38:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.694+0000 7f040e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424280 0x564701424660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.695+0000 7f040e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424280 0x564701424660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54196/0 (socket says 172.21.15.107:54196) 2024-05-09T08:38:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.695+0000 7f040e59c640 1 -- 172.21.15.107:0/1482169503 learned_addr learned my addr 172.21.15.107:0/1482169503 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.695+0000 7f040dd9b640 1 --2- 172.21.15.107:0/1482169503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564701424ba0 0x56470142ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.695+0000 7f040ed9d640 1 --2- 172.21.15.107:0/1482169503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56470142f270 0x564701431660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.695+0000 7f040e59c640 1 -- 172.21.15.107:0/1482169503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564701424ba0 msgr2=0x56470142ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.695+0000 7f040e59c640 1 --2- 172.21.15.107:0/1482169503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564701424ba0 0x56470142ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.695+0000 7f040e59c640 1 -- 172.21.15.107:0/1482169503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56470142f270 msgr2=0x564701431660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.695+0000 7f040e59c640 1 --2- 172.21.15.107:0/1482169503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56470142f270 0x564701431660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.695+0000 7f040e59c640 1 -- 172.21.15.107:0/1482169503 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647011677d0 con 0x564701424280 2024-05-09T08:38:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.695+0000 7f040dd9b640 1 --2- 172.21.15.107:0/1482169503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564701424ba0 0x56470142ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.695+0000 7f040ed9d640 1 --2- 172.21.15.107:0/1574820245 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f03f80a2d80 0x7f03f80a5240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0408006010 tx=0x7f0408031640 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.695+0000 7f040e59c640 1 --2- 172.21.15.107:0/1482169503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424280 0x564701424660 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f04000a3be0 tx=0x7f040009b110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8119a5cb71431da5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.695+0000 7f03fcff9640 1 -- 172.21.15.107:0/1482169503 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0400020040 con 0x564701424280 2024-05-09T08:38:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.695+0000 7f03fcff9640 1 -- 172.21.15.107:0/1482169503 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f040006f8d0 con 0x564701424280 2024-05-09T08:38:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.695+0000 7f03fcff9640 1 -- 172.21.15.107:0/1482169503 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040009b4a0 con 0x564701424280 2024-05-09T08:38:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.696+0000 7f04179a7740 1 -- 172.21.15.107:0/1482169503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424280 msgr2=0x564701424660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.696+0000 7f04179a7740 1 --2- 172.21.15.107:0/1482169503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424280 0x564701424660 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f04000a3be0 tx=0x7f040009b110 comp rx=0 tx=0).stop 2024-05-09T08:38:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.696+0000 7f04179a7740 1 -- 172.21.15.107:0/1482169503 shutdown_connections 2024-05-09T08:38:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.696+0000 7f04179a7740 1 --2- 172.21.15.107:0/1482169503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56470142f270 0x564701431660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.696+0000 7f04179a7740 1 --2- 172.21.15.107:0/1482169503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564701424ba0 0x56470142ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.696+0000 7f04179a7740 1 --2- 172.21.15.107:0/1482169503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424280 0x564701424660 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.696+0000 7f04179a7740 1 -- 172.21.15.107:0/1482169503 >> 172.21.15.107:0/1482169503 conn(0x5647014205a0 msgr2=0x564701420980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.696+0000 7f04179a7740 1 -- 172.21.15.107:0/1482169503 shutdown_connections 2024-05-09T08:38:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.696+0000 7f04179a7740 1 -- 172.21.15.107:0/1482169503 wait complete. 2024-05-09T08:38:50.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.696+0000 7f04179a7740 1 -- start start 2024-05-09T08:38:50.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f04179a7740 1 Processor -- start 2024-05-09T08:38:50.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f04179a7740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564701424280 0x56470142c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f04179a7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x56470142c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f04179a7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56470142f270 0x56470142cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f04179a7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564701432060 con 0x564701424ba0 2024-05-09T08:38:50.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f04179a7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564701344b90 con 0x56470142f270 2024-05-09T08:38:50.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f04179a7740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5647014321e0 con 0x564701424280 2024-05-09T08:38:50.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f04179a7740 10 client.0 _ll_register_callbacks cb 0x564701380da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:50.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f040dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x56470142c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f040dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x56470142c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54198/0 (socket says 172.21.15.107:54198) 2024-05-09T08:38:50.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f040dd9b640 1 -- 172.21.15.107:0/753244113 learned_addr learned my addr 172.21.15.107:0/753244113 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:50.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f040ed9d640 1 --2- 172.21.15.107:0/753244113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56470142f270 0x56470142cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f040e59c640 1 --2- 172.21.15.107:0/753244113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564701424280 0x56470142c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f040dd9b640 1 -- 172.21.15.107:0/753244113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564701424280 msgr2=0x56470142c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:50.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f040dd9b640 1 --2- 172.21.15.107:0/753244113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564701424280 0x56470142c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f040dd9b640 1 -- 172.21.15.107:0/753244113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56470142f270 msgr2=0x56470142cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:50.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f040dd9b640 1 --2- 172.21.15.107:0/753244113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56470142f270 0x56470142cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f040dd9b640 1 -- 172.21.15.107:0/753244113 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56470109d3f0 con 0x564701424ba0 2024-05-09T08:38:50.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f040dd9b640 1 --2- 172.21.15.107:0/753244113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x56470142c7c0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f0404012720 tx=0x7f0404012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75fcf2c40b5a39ed server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:50.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.697+0000 7f03f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:50.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.698+0000 7f03f5ffb640 1 -- 172.21.15.107:0/753244113 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0404004030 con 0x564701424ba0 2024-05-09T08:38:50.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.698+0000 7f03f5ffb640 1 -- 172.21.15.107:0/753244113 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f04040041d0 con 0x564701424ba0 2024-05-09T08:38:50.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.698+0000 7f04179a7740 1 -- 172.21.15.107:0/753244113 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f040006f8d0 con 0x564701424ba0 2024-05-09T08:38:50.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.698+0000 7f03f5ffb640 1 -- 172.21.15.107:0/753244113 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0404025050 con 0x564701424ba0 2024-05-09T08:38:50.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.698+0000 7f04179a7740 10 client.17229 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:50.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.698+0000 7f04179a7740 1 -- 172.21.15.107:0/753244113 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56470105d0f0 con 0x564701424ba0 2024-05-09T08:38:50.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.698+0000 7f03f5ffb640 1 -- 172.21.15.107:0/753244113 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f040006f8d0 con 0x564701424ba0 2024-05-09T08:38:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.698+0000 7f03f5ffb640 1 -- 172.21.15.107:0/753244113 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f0404003a70 con 0x564701424ba0 2024-05-09T08:38:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.698+0000 7f04179a7740 10 client.17229 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.698+0000 7f04179a7740 10 client.17229 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.698+0000 7f04179a7740 20 client.17229 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.698+0000 7f04179a7740 1 --2- 172.21.15.107:0/753244113 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56470143f770 0x564701441b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.698+0000 7f04179a7740 4 client.17229 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.698+0000 7f04179a7740 1 -- 172.21.15.107:0/753244113 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564701397430 con 0x56470143f770 2024-05-09T08:38:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.699+0000 7f040e59c640 1 --2- 172.21.15.107:0/753244113 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56470143f770 0x564701441b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.699+0000 7f040e59c640 1 --2- 172.21.15.107:0/753244113 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56470143f770 0x564701441b90 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a382c0972c959b server_cookie=79be901034cfdb65 in_seq=0 out_seq=0 2024-05-09T08:38:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.699+0000 7f03f5ffb640 10 client.17229 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.700+0000 7f03f5ffb640 1 -- 172.21.15.107:0/753244113 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564701397430 con 0x56470143f770 2024-05-09T08:38:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.700+0000 7f03f5ffb640 10 client.17229 handle_command_reply: tid=0 2024-05-09T08:38:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.700+0000 7f04179a7740 1 client.17229 shutdown 2024-05-09T08:38:50.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.700+0000 7f04179a7740 1 -- 172.21.15.107:0/753244113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 msgr2=0x56470142c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:50.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.700+0000 7f04179a7740 1 --2- 172.21.15.107:0/753244113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x56470142c7c0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f0404012720 tx=0x7f0404012be0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.700+0000 7f04179a7740 1 -- 172.21.15.107:0/753244113 shutdown_connections 2024-05-09T08:38:50.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.700+0000 7f04179a7740 1 --2- 172.21.15.107:0/753244113 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56470143f770 0x564701441b90 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.700+0000 7f04179a7740 1 --2- 172.21.15.107:0/753244113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56470142f270 0x56470142cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.700+0000 7f04179a7740 1 --2- 172.21.15.107:0/753244113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x56470142c7c0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.700+0000 7f04179a7740 1 --2- 172.21.15.107:0/753244113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564701424280 0x56470142c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.700+0000 7f04179a7740 1 -- 172.21.15.107:0/753244113 >> 172.21.15.107:0/753244113 conn(0x5647014205a0 msgr2=0x5647014263a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:50.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.700+0000 7f03f5ffb640 0 client.17229 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:50.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.700+0000 7f04179a7740 1 -- 172.21.15.107:0/753244113 shutdown_connections 2024-05-09T08:38:50.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.700+0000 7f04179a7740 1 -- 172.21.15.107:0/753244113 wait complete. 2024-05-09T08:38:50.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.700+0000 7f04179a7740 20 client.17229 trim_cache size 0 max 16384 2024-05-09T08:38:50.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.719+0000 7f04179a7740 1 Processor -- start 2024-05-09T08:38:50.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.719+0000 7f04179a7740 1 -- start start 2024-05-09T08:38:50.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.719+0000 7f04179a7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564701424280 0x56470143af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.719+0000 7f04179a7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x564701447460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.719+0000 7f04179a7740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56470142f270 0x5647014479a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.719+0000 7f04179a7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f04101b1490 con 0x564701424ba0 2024-05-09T08:38:50.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.719+0000 7f04179a7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f04101aed60 con 0x564701424280 2024-05-09T08:38:50.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.719+0000 7f04179a7740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f04101acbb0 con 0x56470142f270 2024-05-09T08:38:50.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.719+0000 7f040dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x564701447460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.719+0000 7f040dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x564701447460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54210/0 (socket says 172.21.15.107:54210) 2024-05-09T08:38:50.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.719+0000 7f040dd9b640 1 -- 172.21.15.107:0/4164554261 learned_addr learned my addr 172.21.15.107:0/4164554261 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:50.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.719+0000 7f040ed9d640 1 --2- 172.21.15.107:0/4164554261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56470142f270 0x5647014479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.719+0000 7f040e59c640 1 --2- 172.21.15.107:0/4164554261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564701424280 0x56470143af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.720+0000 7f040dd9b640 1 -- 172.21.15.107:0/4164554261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56470142f270 msgr2=0x5647014479a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:50.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.720+0000 7f040dd9b640 1 --2- 172.21.15.107:0/4164554261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56470142f270 0x5647014479a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.720+0000 7f040dd9b640 1 -- 172.21.15.107:0/4164554261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564701424280 msgr2=0x56470143af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:50.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.720+0000 7f040dd9b640 1 --2- 172.21.15.107:0/4164554261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564701424280 0x56470143af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.720+0000 7f040dd9b640 1 -- 172.21.15.107:0/4164554261 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564701431660 con 0x564701424ba0 2024-05-09T08:38:50.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.720+0000 7f040ed9d640 1 --2- 172.21.15.107:0/4164554261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56470142f270 0x5647014479a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:50.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.720+0000 7f040dd9b640 1 --2- 172.21.15.107:0/4164554261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x564701447460 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f0404000c00 tx=0x7f040400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4d75aaf39ab700b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:50.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.720+0000 7f03f7fff640 1 -- 172.21.15.107:0/4164554261 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040402e070 con 0x564701424ba0 2024-05-09T08:38:50.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.720+0000 7f03f7fff640 1 -- 172.21.15.107:0/4164554261 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0404004460 con 0x564701424ba0 2024-05-09T08:38:50.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.720+0000 7f03f7fff640 1 -- 172.21.15.107:0/4164554261 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0404022040 con 0x564701424ba0 2024-05-09T08:38:50.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 -- 172.21.15.107:0/4164554261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 msgr2=0x564701447460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:50.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 --2- 172.21.15.107:0/4164554261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x564701447460 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f0404000c00 tx=0x7f040400f790 comp rx=0 tx=0).stop 2024-05-09T08:38:50.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 -- 172.21.15.107:0/4164554261 shutdown_connections 2024-05-09T08:38:50.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 --2- 172.21.15.107:0/4164554261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56470142f270 0x5647014479a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 --2- 172.21.15.107:0/4164554261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x564701447460 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 --2- 172.21.15.107:0/4164554261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564701424280 0x56470143af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 -- 172.21.15.107:0/4164554261 >> 172.21.15.107:0/4164554261 conn(0x5647014205a0 msgr2=0x56470142e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:50.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 -- 172.21.15.107:0/4164554261 shutdown_connections 2024-05-09T08:38:50.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 -- 172.21.15.107:0/4164554261 wait complete. 2024-05-09T08:38:50.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 -- start start 2024-05-09T08:38:50.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 Processor -- start 2024-05-09T08:38:50.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564701424280 0x564701437430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x564701437970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56470142f270 0x5647014285d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f04101ad220 con 0x564701424ba0 2024-05-09T08:38:50.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f04101ae690 con 0x56470142f270 2024-05-09T08:38:50.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.721+0000 7f04179a7740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f04101ad8f0 con 0x564701424280 2024-05-09T08:38:50.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f04179a7740 10 client.0 _ll_register_callbacks cb 0x7f0404009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:50.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f040dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x564701437970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f040dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x564701437970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54224/0 (socket says 172.21.15.107:54224) 2024-05-09T08:38:50.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f040dd9b640 1 -- 172.21.15.107:0/1868192576 learned_addr learned my addr 172.21.15.107:0/1868192576 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:50.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f040e59c640 1 --2- 172.21.15.107:0/1868192576 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564701424280 0x564701437430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f040ed9d640 1 --2- 172.21.15.107:0/1868192576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56470142f270 0x5647014285d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f040dd9b640 1 -- 172.21.15.107:0/1868192576 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564701424280 msgr2=0x564701437430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:50.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f040dd9b640 1 --2- 172.21.15.107:0/1868192576 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564701424280 0x564701437430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f040dd9b640 1 -- 172.21.15.107:0/1868192576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56470142f270 msgr2=0x5647014285d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:50.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f040dd9b640 1 --2- 172.21.15.107:0/1868192576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56470142f270 0x5647014285d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f040dd9b640 1 -- 172.21.15.107:0/1868192576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564701428c40 con 0x564701424ba0 2024-05-09T08:38:50.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f040ed9d640 1 --2- 172.21.15.107:0/1868192576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56470142f270 0x5647014285d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:38:50.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f040dd9b640 1 --2- 172.21.15.107:0/1868192576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x564701437970 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f0404006010 tx=0x7f0404009e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385213bdc7d4abfe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:50.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f03f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:50.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f03f5ffb640 1 -- 172.21.15.107:0/1868192576 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040402e070 con 0x564701424ba0 2024-05-09T08:38:50.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f03f5ffb640 1 -- 172.21.15.107:0/1868192576 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f0404003fb0 con 0x564701424ba0 2024-05-09T08:38:50.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f04179a7740 1 -- 172.21.15.107:0/1868192576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0404004460 con 0x564701424ba0 2024-05-09T08:38:50.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f03f5ffb640 1 -- 172.21.15.107:0/1868192576 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f040400f920 con 0x564701424ba0 2024-05-09T08:38:50.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f04179a7740 10 client.17235 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:50.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.722+0000 7f04179a7740 1 -- 172.21.15.107:0/1868192576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5647014a5640 con 0x564701424ba0 2024-05-09T08:38:50.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.723+0000 7f03f5ffb640 1 -- 172.21.15.107:0/1868192576 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0404004460 con 0x564701424ba0 2024-05-09T08:38:50.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.723+0000 7f03f5ffb640 1 -- 172.21.15.107:0/1868192576 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f0404010b70 con 0x564701424ba0 2024-05-09T08:38:50.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.723+0000 7f04179a7740 10 client.17235 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:50.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.723+0000 7f04179a7740 10 client.17235 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:50.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.723+0000 7f04179a7740 20 client.17235 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:50.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.723+0000 7f04179a7740 1 --2- 172.21.15.107:0/1868192576 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5647014a5a80 0x5647014a7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.723+0000 7f04179a7740 4 client.17235 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:38:50.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.723+0000 7f04179a7740 1 -- 172.21.15.107:0/1868192576 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564701391780 con 0x5647014a5a80 2024-05-09T08:38:50.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.723+0000 7f040e59c640 1 --2- 172.21.15.107:0/1868192576 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5647014a5a80 0x5647014a7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f03f5ffb640 10 client.17235 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:50.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f040e59c640 1 --2- 172.21.15.107:0/1868192576 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5647014a5a80 0x5647014a7ea0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b10a06254e227fb server_cookie=2c4244acb44cd00f in_seq=0 out_seq=0 2024-05-09T08:38:50.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f03f5ffb640 1 -- 172.21.15.107:0/1868192576 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564701391780 con 0x5647014a5a80 2024-05-09T08:38:50.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f03f5ffb640 10 client.17235 handle_command_reply: tid=0 2024-05-09T08:38:50.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f04179a7740 1 client.17235 shutdown 2024-05-09T08:38:50.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f04179a7740 1 -- 172.21.15.107:0/1868192576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 msgr2=0x564701437970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:50.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f04179a7740 1 --2- 172.21.15.107:0/1868192576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x564701437970 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f0404006010 tx=0x7f0404009e50 comp rx=0 tx=0).stop 2024-05-09T08:38:50.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f04179a7740 1 -- 172.21.15.107:0/1868192576 shutdown_connections 2024-05-09T08:38:50.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f04179a7740 1 --2- 172.21.15.107:0/1868192576 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5647014a5a80 0x5647014a7ea0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f04179a7740 1 --2- 172.21.15.107:0/1868192576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56470142f270 0x5647014285d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f03f5ffb640 0 client.17235 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:50.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f04179a7740 1 --2- 172.21.15.107:0/1868192576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564701424ba0 0x564701437970 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f04179a7740 1 --2- 172.21.15.107:0/1868192576 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564701424280 0x564701437430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f04179a7740 1 -- 172.21.15.107:0/1868192576 >> 172.21.15.107:0/1868192576 conn(0x5647014205a0 msgr2=0x5647014259d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:50.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f04179a7740 1 -- 172.21.15.107:0/1868192576 shutdown_connections 2024-05-09T08:38:50.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f04179a7740 1 -- 172.21.15.107:0/1868192576 wait complete. 2024-05-09T08:38:50.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.724+0000 7f04179a7740 20 client.17235 trim_cache size 0 max 16384 2024-05-09T08:38:50.734 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:38:50.734 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:38:50.734 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:38:50.734 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:38:50.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.725+0000 7f0414c9a640 1 -- 172.21.15.107:0/1574820245 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f03f80a2d80 msgr2=0x7f03f80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:50.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.725+0000 7f0414c9a640 1 --2- 172.21.15.107:0/1574820245 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f03f80a2d80 0x7f03f80a5240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0408006010 tx=0x7f0408031640 comp rx=0 tx=0).stop 2024-05-09T08:38:50.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.725+0000 7f0414c9a640 1 -- 172.21.15.107:0/1574820245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0410100700 msgr2=0x7f0410198e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:50.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.725+0000 7f0414c9a640 1 --2- 172.21.15.107:0/1574820245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0410100700 0x7f0410198e30 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f040000d950 tx=0x7f040000de10 comp rx=0 tx=0).stop 2024-05-09T08:38:50.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.725+0000 7f0414c9a640 1 -- 172.21.15.107:0/1574820245 shutdown_connections 2024-05-09T08:38:50.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.725+0000 7f0414c9a640 1 --2- 172.21.15.107:0/1574820245 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f03f80a2d80 0x7f03f80a5240 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.725+0000 7f0414c9a640 1 --2- 172.21.15.107:0/1574820245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0410106710 0x7f04101a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.725+0000 7f0414c9a640 1 --2- 172.21.15.107:0/1574820245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04101010c0 0x7f0410199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.725+0000 7f0414c9a640 1 --2- 172.21.15.107:0/1574820245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0410100700 0x7f0410198e30 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.725+0000 7f0414c9a640 1 -- 172.21.15.107:0/1574820245 >> 172.21.15.107:0/1574820245 conn(0x7f0410077830 msgr2=0x7f0410102ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:50.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.725+0000 7f0414c9a640 1 -- 172.21.15.107:0/1574820245 shutdown_connections 2024-05-09T08:38:50.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.725+0000 7f0414c9a640 1 -- 172.21.15.107:0/1574820245 wait complete. 2024-05-09T08:38:50.736 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:50 smithi107 ceph-mon[30122]: pgmap v167: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:50.737 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:38:50.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:38:50.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:50 smithi157 ceph-mon[33614]: pgmap v167: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:50.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.851+0000 7f196ac15640 1 -- 172.21.15.107:0/2980355262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1964100960 msgr2=0x7f1964106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.851+0000 7f196ac15640 1 --2- 172.21.15.107:0/2980355262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1964100960 0x7f1964106490 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f1950009900 tx=0x7f195002a410 comp rx=0 tx=0).stop 2024-05-09T08:38:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.851+0000 7f196ac15640 1 -- 172.21.15.107:0/2980355262 shutdown_connections 2024-05-09T08:38:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.851+0000 7f196ac15640 1 --2- 172.21.15.107:0/2980355262 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1964106bc0 0x7f196410e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.851+0000 7f196ac15640 1 --2- 172.21.15.107:0/2980355262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1964100960 0x7f1964106490 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.851+0000 7f196ac15640 1 --2- 172.21.15.107:0/2980355262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19640fffb0 0x7f1964100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.851+0000 7f196ac15640 1 -- 172.21.15.107:0/2980355262 >> 172.21.15.107:0/2980355262 conn(0x7f1964076cd0 msgr2=0x7f19640fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.851+0000 7f196ac15640 1 -- 172.21.15.107:0/2980355262 shutdown_connections 2024-05-09T08:38:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.851+0000 7f196ac15640 1 -- 172.21.15.107:0/2980355262 wait complete. 2024-05-09T08:38:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.851+0000 7f196ac15640 1 Processor -- start 2024-05-09T08:38:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.851+0000 7f196ac15640 1 -- start start 2024-05-09T08:38:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f196ac15640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19640fffb0 0x7f1964100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f196ac15640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1964106bc0 0x7f1964101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f196ac15640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19641019a0 0x7f19641a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f196ac15640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1964111a90 con 0x7f1964106bc0 2024-05-09T08:38:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f196ac15640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1964111910 con 0x7f19640fffb0 2024-05-09T08:38:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f196ac15640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1964111c10 con 0x7f19641019a0 2024-05-09T08:38:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f1963fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1964106bc0 0x7f1964101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f1963fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1964106bc0 0x7f1964101460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54254/0 (socket says 172.21.15.107:54254) 2024-05-09T08:38:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f1963fff640 1 -- 172.21.15.107:0/3623908991 learned_addr learned my addr 172.21.15.107:0/3623908991 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f19689b1640 1 --2- 172.21.15.107:0/3623908991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19640fffb0 0x7f1964100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f19691b2640 1 --2- 172.21.15.107:0/3623908991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19641019a0 0x7f19641a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f1963fff640 1 -- 172.21.15.107:0/3623908991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19641019a0 msgr2=0x7f19641a6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f1963fff640 1 --2- 172.21.15.107:0/3623908991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19641019a0 0x7f19641a6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f1963fff640 1 -- 172.21.15.107:0/3623908991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19640fffb0 msgr2=0x7f1964100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f1963fff640 1 --2- 172.21.15.107:0/3623908991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19640fffb0 0x7f1964100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f1963fff640 1 -- 172.21.15.107:0/3623908991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19641a6960 con 0x7f1964106bc0 2024-05-09T08:38:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f1963fff640 1 --2- 172.21.15.107:0/3623908991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1964106bc0 0x7f1964101460 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f1950004270 tx=0x7f19500042a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.852+0000 7f1961ffb640 1 -- 172.21.15.107:0/3623908991 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195003f030 con 0x7f1964106bc0 2024-05-09T08:38:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.853+0000 7f1961ffb640 1 -- 172.21.15.107:0/3623908991 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19500044c0 con 0x7f1964106bc0 2024-05-09T08:38:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.853+0000 7f196ac15640 1 -- 172.21.15.107:0/3623908991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19641a6bf0 con 0x7f1964106bc0 2024-05-09T08:38:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.853+0000 7f1961ffb640 1 -- 172.21.15.107:0/3623908991 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950035070 con 0x7f1964106bc0 2024-05-09T08:38:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.853+0000 7f196ac15640 1 -- 172.21.15.107:0/3623908991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19641a70e0 con 0x7f1964106bc0 2024-05-09T08:38:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.854+0000 7f196ac15640 1 -- 172.21.15.107:0/3623908991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1964067dc0 con 0x7f1964106bc0 2024-05-09T08:38:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.855+0000 7f1961ffb640 1 -- 172.21.15.107:0/3623908991 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1950042030 con 0x7f1964106bc0 2024-05-09T08:38:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.856+0000 7f1961ffb640 1 --2- 172.21.15.107:0/3623908991 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f19540a2d00 0x7f19540a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.856+0000 7f19689b1640 1 --2- 172.21.15.107:0/3623908991 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f19540a2d00 0x7f19540a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.856+0000 7f1961ffb640 1 -- 172.21.15.107:0/3623908991 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f19500f0070 con 0x7f1964106bc0 2024-05-09T08:38:50.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.857+0000 7f19689b1640 1 --2- 172.21.15.107:0/3623908991 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f19540a2d00 0x7f19540a51c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1958005d00 tx=0x7f195800b410 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:50.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:50.860+0000 7f1961ffb640 1 -- 172.21.15.107:0/3623908991 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f19500bc930 con 0x7f1964106bc0 2024-05-09T08:38:51.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.134+0000 7f196ac15640 1 -- 172.21.15.107:0/3623908991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1964109d70 con 0x7f1964106bc0 2024-05-09T08:38:51.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.135+0000 7f1961ffb640 1 -- 172.21.15.107:0/3623908991 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11887 (secure 0 0 0) 0x7f19500bc230 con 0x7f1964106bc0 2024-05-09T08:38:51.135 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:38:51.135 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:37:10.930317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:38:51.135 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2024-05-09T08:38:51.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.137+0000 7f196ac15640 1 -- 172.21.15.107:0/3623908991 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f19540a2d00 msgr2=0x7f19540a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.137+0000 7f196ac15640 1 --2- 172.21.15.107:0/3623908991 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f19540a2d00 0x7f19540a51c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1958005d00 tx=0x7f195800b410 comp rx=0 tx=0).stop 2024-05-09T08:38:51.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.137+0000 7f196ac15640 1 -- 172.21.15.107:0/3623908991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1964106bc0 msgr2=0x7f1964101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.137+0000 7f196ac15640 1 --2- 172.21.15.107:0/3623908991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1964106bc0 0x7f1964101460 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f1950004270 tx=0x7f19500042a0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.137+0000 7f196ac15640 1 -- 172.21.15.107:0/3623908991 shutdown_connections 2024-05-09T08:38:51.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.137+0000 7f196ac15640 1 --2- 172.21.15.107:0/3623908991 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f19540a2d00 0x7f19540a51c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.137+0000 7f196ac15640 1 --2- 172.21.15.107:0/3623908991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19641019a0 0x7f19641a6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.137+0000 7f196ac15640 1 --2- 172.21.15.107:0/3623908991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1964106bc0 0x7f1964101460 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.137+0000 7f196ac15640 1 --2- 172.21.15.107:0/3623908991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19640fffb0 0x7f1964100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.137+0000 7f196ac15640 1 -- 172.21.15.107:0/3623908991 >> 172.21.15.107:0/3623908991 conn(0x7f1964076cd0 msgr2=0x7f1964104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.137+0000 7f196ac15640 1 -- 172.21.15.107:0/3623908991 shutdown_connections 2024-05-09T08:38:51.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.137+0000 7f196ac15640 1 -- 172.21.15.107:0/3623908991 wait complete. 2024-05-09T08:38:51.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:38:51.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.263+0000 7feec5438640 1 -- 172.21.15.107:0/1283208180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feec0100700 msgr2=0x7feec0100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.263+0000 7feec5438640 1 --2- 172.21.15.107:0/1283208180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feec0100700 0x7feec0100b80 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7feea8009930 tx=0x7feea802a420 comp rx=0 tx=0).stop 2024-05-09T08:38:51.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.263+0000 7feec5438640 1 -- 172.21.15.107:0/1283208180 shutdown_connections 2024-05-09T08:38:51.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.263+0000 7feec5438640 1 --2- 172.21.15.107:0/1283208180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec01010c0 0x7feec010d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.263+0000 7feec5438640 1 --2- 172.21.15.107:0/1283208180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feec0100700 0x7feec0100b80 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.263+0000 7feec5438640 1 --2- 172.21.15.107:0/1283208180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feec0106710 0x7feec0106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.263+0000 7feec5438640 1 -- 172.21.15.107:0/1283208180 >> 172.21.15.107:0/1283208180 conn(0x7feec0077830 msgr2=0x7feec0077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.263+0000 7feec5438640 1 -- 172.21.15.107:0/1283208180 shutdown_connections 2024-05-09T08:38:51.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.263+0000 7feec5438640 1 -- 172.21.15.107:0/1283208180 wait complete. 2024-05-09T08:38:51.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.264+0000 7feec5438640 1 Processor -- start 2024-05-09T08:38:51.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.264+0000 7feec5438640 1 -- start start 2024-05-09T08:38:51.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.264+0000 7feec5438640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec0100700 0x7feec0198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.264+0000 7feec5438640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feec01010c0 0x7feec01993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.264+0000 7feec5438640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feec0106710 0x7feec01a3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.264+0000 7feec5438640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feec01109d0 con 0x7feec0106710 2024-05-09T08:38:51.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.264+0000 7feec5438640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feec0110850 con 0x7feec01010c0 2024-05-09T08:38:51.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.264+0000 7feec5438640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feec0110b50 con 0x7feec0100700 2024-05-09T08:38:51.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.264+0000 7feebf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feec0106710 0x7feec01a3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.264+0000 7feebeffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec0100700 0x7feec0198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.264+0000 7feebf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feec0106710 0x7feec01a3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54272/0 (socket says 172.21.15.107:54272) 2024-05-09T08:38:51.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.264+0000 7feebf7fe640 1 -- 172.21.15.107:0/3251667315 learned_addr learned my addr 172.21.15.107:0/3251667315 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:51.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.265+0000 7feeb7fff640 1 --2- 172.21.15.107:0/3251667315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feec01010c0 0x7feec01993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.265+0000 7feebf7fe640 1 -- 172.21.15.107:0/3251667315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec0100700 msgr2=0x7feec0198ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.265+0000 7feebf7fe640 1 --2- 172.21.15.107:0/3251667315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec0100700 0x7feec0198ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.265+0000 7feebf7fe640 1 -- 172.21.15.107:0/3251667315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feec01010c0 msgr2=0x7feec01993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.265+0000 7feebf7fe640 1 --2- 172.21.15.107:0/3251667315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feec01010c0 0x7feec01993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.265+0000 7feebf7fe640 1 -- 172.21.15.107:0/3251667315 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec01a4460 con 0x7feec0106710 2024-05-09T08:38:51.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.265+0000 7feeb7fff640 1 --2- 172.21.15.107:0/3251667315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feec01010c0 0x7feec01993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:51.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.265+0000 7feebeffd640 1 --2- 172.21.15.107:0/3251667315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec0100700 0x7feec0198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:51.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.265+0000 7feebf7fe640 1 --2- 172.21.15.107:0/3251667315 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feec0106710 0x7feec01a3f20 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7feeb800eb10 tx=0x7feeb800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.265+0000 7feebcff9640 1 -- 172.21.15.107:0/3251667315 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8004b20 con 0x7feec0106710 2024-05-09T08:38:51.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.265+0000 7feec5438640 1 -- 172.21.15.107:0/3251667315 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feec01a4630 con 0x7feec0106710 2024-05-09T08:38:51.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.265+0000 7feebcff9640 1 -- 172.21.15.107:0/3251667315 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feeb8004cc0 con 0x7feec0106710 2024-05-09T08:38:51.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.265+0000 7feebcff9640 1 -- 172.21.15.107:0/3251667315 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8010690 con 0x7feec0106710 2024-05-09T08:38:51.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.265+0000 7feec5438640 1 -- 172.21.15.107:0/3251667315 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feec01a4ba0 con 0x7feec0106710 2024-05-09T08:38:51.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.267+0000 7feebcff9640 1 -- 172.21.15.107:0/3251667315 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7feeb8005020 con 0x7feec0106710 2024-05-09T08:38:51.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.268+0000 7feec8221740 1 Processor -- start 2024-05-09T08:38:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.268+0000 7feec8221740 1 -- start start 2024-05-09T08:38:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.268+0000 7feebcff9640 1 --2- 172.21.15.107:0/3251667315 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feeac0a2d00 0x7feeac0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.268+0000 7feec8221740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91210 0x563faed915f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.268+0000 7feec8221740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faed91b30 0x563faeda01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.268+0000 7feec8221740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faeda0730 0x563faeda2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feec8221740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563faecf6c40 con 0x563faed91210 2024-05-09T08:38:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feec8221740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563faec9e580 con 0x563faed91b30 2024-05-09T08:38:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feec8221740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563faecf2c00 con 0x563faeda0730 2024-05-09T08:38:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feebeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91210 0x563faed915f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feebcff9640 1 -- 172.21.15.107:0/3251667315 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7feeb80cff00 con 0x7feec0106710 2024-05-09T08:38:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feeb7fff640 1 --2- 172.21.15.107:0/3251667315 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feeac0a2d00 0x7feeac0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feebeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91210 0x563faed915f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54288/0 (socket says 172.21.15.107:54288) 2024-05-09T08:38:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feebeffd640 1 -- 172.21.15.107:0/966898865 learned_addr learned my addr 172.21.15.107:0/966898865 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feebf7fe640 1 --2- 172.21.15.107:0/966898865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faeda0730 0x563faeda2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feebeffd640 1 -- 172.21.15.107:0/966898865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faeda0730 msgr2=0x563faeda2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feebeffd640 1 --2- 172.21.15.107:0/966898865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faeda0730 0x563faeda2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feebeffd640 1 -- 172.21.15.107:0/966898865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faed91b30 msgr2=0x563faeda01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feebeffd640 1 --2- 172.21.15.107:0/966898865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faed91b30 0x563faeda01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feebeffd640 1 -- 172.21.15.107:0/966898865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563faea0a3f0 con 0x563faed91210 2024-05-09T08:38:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feebeffd640 1 --2- 172.21.15.107:0/966898865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91210 0x563faed915f0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7feeb00114f0 tx=0x7feeb00119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12aa1d4bf22ab0c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feeb7fff640 1 --2- 172.21.15.107:0/3251667315 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feeac0a2d00 0x7feeac0a51c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feea8000c00 tx=0x7feea8032c60 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feeb57fa640 1 -- 172.21.15.107:0/966898865 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb001e070 con 0x563faed91210 2024-05-09T08:38:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feeb57fa640 1 -- 172.21.15.107:0/966898865 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feeb0011030 con 0x563faed91210 2024-05-09T08:38:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.269+0000 7feeb57fa640 1 -- 172.21.15.107:0/966898865 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb00196b0 con 0x563faed91210 2024-05-09T08:38:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.270+0000 7feec8221740 1 -- 172.21.15.107:0/966898865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91210 msgr2=0x563faed915f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.270+0000 7feec8221740 1 --2- 172.21.15.107:0/966898865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91210 0x563faed915f0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7feeb00114f0 tx=0x7feeb00119b0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.270+0000 7feec8221740 1 -- 172.21.15.107:0/966898865 shutdown_connections 2024-05-09T08:38:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.270+0000 7feec8221740 1 --2- 172.21.15.107:0/966898865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faeda0730 0x563faeda2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.270+0000 7feec8221740 1 --2- 172.21.15.107:0/966898865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faed91b30 0x563faeda01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.270+0000 7feec8221740 1 --2- 172.21.15.107:0/966898865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91210 0x563faed915f0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.270+0000 7feec8221740 1 -- 172.21.15.107:0/966898865 >> 172.21.15.107:0/966898865 conn(0x563faed8d650 msgr2=0x563faed8da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.271+0000 7feec8221740 1 -- 172.21.15.107:0/966898865 shutdown_connections 2024-05-09T08:38:51.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.271+0000 7feec8221740 1 -- 172.21.15.107:0/966898865 wait complete. 2024-05-09T08:38:51.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.271+0000 7feec8221740 1 -- start start 2024-05-09T08:38:51.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.271+0000 7feec8221740 1 Processor -- start 2024-05-09T08:38:51.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.271+0000 7feec8221740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faed91210 0x563faed9d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.271+0000 7feec8221740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91b30 0x563faed9dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.271+0000 7feec8221740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faeda0730 0x563faed9e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.271+0000 7feec8221740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563faeda3570 con 0x563faed91b30 2024-05-09T08:38:51.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.271+0000 7feec8221740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563faecb1c40 con 0x563faed91210 2024-05-09T08:38:51.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.271+0000 7feec8221740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563faeda36f0 con 0x563faeda0730 2024-05-09T08:38:51.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.271+0000 7feec8221740 10 client.0 _ll_register_callbacks cb 0x563faea09be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:51.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.271+0000 7feeb7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91b30 0x563faed9dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.271+0000 7feeb7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91b30 0x563faed9dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54300/0 (socket says 172.21.15.107:54300) 2024-05-09T08:38:51.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.271+0000 7feebf7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faeda0730 0x563faed9e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.271+0000 7feeb7fff640 1 -- 172.21.15.107:0/1948038891 learned_addr learned my addr 172.21.15.107:0/1948038891 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:51.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.271+0000 7feebeffd640 1 --2- 172.21.15.107:0/1948038891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faed91210 0x563faed9d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.272+0000 7feebf7fe640 1 -- 172.21.15.107:0/1948038891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faed91210 msgr2=0x563faed9d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.272+0000 7feebf7fe640 1 --2- 172.21.15.107:0/1948038891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faed91210 0x563faed9d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.272+0000 7feebf7fe640 1 -- 172.21.15.107:0/1948038891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91b30 msgr2=0x563faed9dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.272+0000 7feebf7fe640 1 --2- 172.21.15.107:0/1948038891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91b30 0x563faed9dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.272+0000 7feebf7fe640 1 -- 172.21.15.107:0/1948038891 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563faead47d0 con 0x563faeda0730 2024-05-09T08:38:51.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.272+0000 7feebf7fe640 1 --2- 172.21.15.107:0/1948038891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faeda0730 0x563faed9e0e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feeb80cd2f0 tx=0x7feeb809bcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fc8c353ebfee1b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.272+0000 7feea67fc640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:38:51.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.272+0000 7feea67fc640 1 -- 172.21.15.107:0/1948038891 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb809c0c0 con 0x563faeda0730 2024-05-09T08:38:51.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.272+0000 7feec8221740 1 -- 172.21.15.107:0/1948038891 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feeb0011030 con 0x563faeda0730 2024-05-09T08:38:51.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.272+0000 7feea67fc640 1 -- 172.21.15.107:0/1948038891 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7feeb80a0120 con 0x563faeda0730 2024-05-09T08:38:51.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.272+0000 7feea67fc640 1 -- 172.21.15.107:0/1948038891 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feeb80a0470 con 0x563faeda0730 2024-05-09T08:38:51.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.272+0000 7feec8221740 10 client.26599 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:51.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.272+0000 7feec8221740 1 -- 172.21.15.107:0/1948038891 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563faedaf6f0 con 0x563faeda0730 2024-05-09T08:38:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.272+0000 7feea67fc640 1 -- 172.21.15.107:0/1948038891 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feeb0011030 con 0x563faeda0730 2024-05-09T08:38:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.273+0000 7feea67fc640 1 -- 172.21.15.107:0/1948038891 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7feeb80a48a0 con 0x563faeda0730 2024-05-09T08:38:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.273+0000 7feec8221740 10 client.26599 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.273+0000 7feec8221740 10 client.26599 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:38:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.273+0000 7feec8221740 20 client.26599 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.273+0000 7feec8221740 1 --2- 172.21.15.107:0/1948038891 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563faedb0b50 0x563faedb2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.273+0000 7feec8221740 4 client.26599 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.273+0000 7feec8221740 1 -- 172.21.15.107:0/1948038891 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563faed04370 con 0x563faedb0b50 2024-05-09T08:38:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.273+0000 7feebeffd640 1 --2- 172.21.15.107:0/1948038891 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563faedb0b50 0x563faedb2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.273+0000 7feea67fc640 10 client.26599 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:38:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.273+0000 7feebeffd640 1 --2- 172.21.15.107:0/1948038891 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563faedb0b50 0x563faedb2fb0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=43bb3f0ae563e49b server_cookie=65478b30e2ec4e3 in_seq=0 out_seq=0 2024-05-09T08:38:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.274+0000 7feea67fc640 1 -- 172.21.15.107:0/1948038891 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563faed04370 con 0x563faedb0b50 2024-05-09T08:38:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.274+0000 7feea67fc640 10 client.26599 handle_command_reply: tid=0 2024-05-09T08:38:51.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.274+0000 7feec8221740 1 client.26599 shutdown 2024-05-09T08:38:51.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.274+0000 7feec8221740 1 -- 172.21.15.107:0/1948038891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faeda0730 msgr2=0x563faed9e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.274+0000 7feec8221740 1 --2- 172.21.15.107:0/1948038891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faeda0730 0x563faed9e0e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feeb80cd2f0 tx=0x7feeb809bcc0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.274+0000 7feec8221740 1 -- 172.21.15.107:0/1948038891 shutdown_connections 2024-05-09T08:38:51.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.274+0000 7feec8221740 1 --2- 172.21.15.107:0/1948038891 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563faedb0b50 0x563faedb2fb0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.274+0000 7feec8221740 1 --2- 172.21.15.107:0/1948038891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faeda0730 0x563faed9e0e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.274+0000 7feea67fc640 0 client.26599 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:38:51.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.274+0000 7feec8221740 1 --2- 172.21.15.107:0/1948038891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91b30 0x563faed9dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.274+0000 7feec8221740 1 --2- 172.21.15.107:0/1948038891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faed91210 0x563faed9d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.274+0000 7feec8221740 1 -- 172.21.15.107:0/1948038891 >> 172.21.15.107:0/1948038891 conn(0x563faed8d650 msgr2=0x563faed93400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.274+0000 7feec8221740 1 -- 172.21.15.107:0/1948038891 shutdown_connections 2024-05-09T08:38:51.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.274+0000 7feec8221740 1 -- 172.21.15.107:0/1948038891 wait complete. 2024-05-09T08:38:51.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.274+0000 7feec8221740 20 client.26599 trim_cache size 0 max 16384 2024-05-09T08:38:51.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.289+0000 7feec8221740 1 Processor -- start 2024-05-09T08:38:51.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.289+0000 7feec8221740 1 -- start start 2024-05-09T08:38:51.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.290+0000 7feec8221740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faed91210 0x563faedac4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.290+0000 7feec8221740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91b30 0x563faed9c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.290+0000 7feec8221740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faeda0730 0x563faed8c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.290+0000 7feec8221740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feec01af250 con 0x563faed91b30 2024-05-09T08:38:51.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.290+0000 7feec8221740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feec01acb20 con 0x563faeda0730 2024-05-09T08:38:51.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.290+0000 7feec8221740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feec01aa970 con 0x563faed91210 2024-05-09T08:38:51.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.290+0000 7feeb7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91b30 0x563faed9c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.290+0000 7feeb7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91b30 0x563faed9c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54306/0 (socket says 172.21.15.107:54306) 2024-05-09T08:38:51.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.290+0000 7feeb7fff640 1 -- 172.21.15.107:0/2800395349 learned_addr learned my addr 172.21.15.107:0/2800395349 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:51.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.290+0000 7feebf7fe640 1 --2- 172.21.15.107:0/2800395349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faeda0730 0x563faed8c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.290+0000 7feebeffd640 1 --2- 172.21.15.107:0/2800395349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faed91210 0x563faedac4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.290+0000 7feeb7fff640 1 -- 172.21.15.107:0/2800395349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faed91210 msgr2=0x563faedac4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.290+0000 7feeb7fff640 1 --2- 172.21.15.107:0/2800395349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faed91210 0x563faedac4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.290+0000 7feeb7fff640 1 -- 172.21.15.107:0/2800395349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faeda0730 msgr2=0x563faed8c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.290+0000 7feeb7fff640 1 --2- 172.21.15.107:0/2800395349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faeda0730 0x563faed8c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.290+0000 7feeb7fff640 1 -- 172.21.15.107:0/2800395349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563faed91960 con 0x563faed91b30 2024-05-09T08:38:51.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.291+0000 7feeb7fff640 1 --2- 172.21.15.107:0/2800395349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91b30 0x563faed9c7c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7feea8009900 tx=0x7feea803e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffafce27df53e7f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.291+0000 7feeb4ff9640 1 -- 172.21.15.107:0/2800395349 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea804e070 con 0x563faed91b30 2024-05-09T08:38:51.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.291+0000 7feeb4ff9640 1 -- 172.21.15.107:0/2800395349 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feea8004440 con 0x563faed91b30 2024-05-09T08:38:51.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.291+0000 7feeb4ff9640 1 -- 172.21.15.107:0/2800395349 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea8035050 con 0x563faed91b30 2024-05-09T08:38:51.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 -- 172.21.15.107:0/2800395349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91b30 msgr2=0x563faed9c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 --2- 172.21.15.107:0/2800395349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91b30 0x563faed9c7c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7feea8009900 tx=0x7feea803e000 comp rx=0 tx=0).stop 2024-05-09T08:38:51.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 -- 172.21.15.107:0/2800395349 shutdown_connections 2024-05-09T08:38:51.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 --2- 172.21.15.107:0/2800395349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faeda0730 0x563faed8c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 --2- 172.21.15.107:0/2800395349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91b30 0x563faed9c7c0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 --2- 172.21.15.107:0/2800395349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faed91210 0x563faedac4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 -- 172.21.15.107:0/2800395349 >> 172.21.15.107:0/2800395349 conn(0x563faed8d650 msgr2=0x563faed8bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 -- 172.21.15.107:0/2800395349 shutdown_connections 2024-05-09T08:38:51.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 -- 172.21.15.107:0/2800395349 wait complete. 2024-05-09T08:38:51.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 -- start start 2024-05-09T08:38:51.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 Processor -- start 2024-05-09T08:38:51.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91210 0x563faed9a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faed91b30 0x563faed9a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faeda0730 0x563faed9abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feec01aafe0 con 0x563faed91210 2024-05-09T08:38:51.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feec01ac450 con 0x563faeda0730 2024-05-09T08:38:51.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feec01ab6b0 con 0x563faed91b30 2024-05-09T08:38:51.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feec8221740 10 client.0 _ll_register_callbacks cb 0x7feeb809be90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:51.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feebeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91210 0x563faed9a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feebeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91210 0x563faed9a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54316/0 (socket says 172.21.15.107:54316) 2024-05-09T08:38:51.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.292+0000 7feebeffd640 1 -- 172.21.15.107:0/2895769391 learned_addr learned my addr 172.21.15.107:0/2895769391 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:51.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.293+0000 7feebf7fe640 1 --2- 172.21.15.107:0/2895769391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faeda0730 0x563faed9abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.293+0000 7feeb7fff640 1 --2- 172.21.15.107:0/2895769391 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faed91b30 0x563faed9a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.293+0000 7feebeffd640 1 -- 172.21.15.107:0/2895769391 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faed91b30 msgr2=0x563faed9a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.293+0000 7feebeffd640 1 --2- 172.21.15.107:0/2895769391 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faed91b30 0x563faed9a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.293+0000 7feebeffd640 1 -- 172.21.15.107:0/2895769391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faeda0730 msgr2=0x563faed9abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.293+0000 7feebeffd640 1 --2- 172.21.15.107:0/2895769391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faeda0730 0x563faed9abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.293+0000 7feebeffd640 1 -- 172.21.15.107:0/2895769391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563faee160f0 con 0x563faed91210 2024-05-09T08:38:51.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.293+0000 7feebeffd640 1 --2- 172.21.15.107:0/2895769391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91210 0x563faed9a170 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7feeb00058c0 tx=0x7feeb0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10297f8b6843930e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.293+0000 7feea67fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:51.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.293+0000 7feea67fc640 1 -- 172.21.15.107:0/2895769391 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb002f060 con 0x563faed91210 2024-05-09T08:38:51.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.293+0000 7feea67fc640 1 -- 172.21.15.107:0/2895769391 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7feeb0005040 con 0x563faed91210 2024-05-09T08:38:51.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.293+0000 7feec8221740 1 -- 172.21.15.107:0/2895769391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feea8004440 con 0x563faed91210 2024-05-09T08:38:51.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.293+0000 7feea67fc640 1 -- 172.21.15.107:0/2895769391 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feeb0036050 con 0x563faed91210 2024-05-09T08:38:51.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.293+0000 7feec8221740 10 client.17259 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:51.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.293+0000 7feec8221740 1 -- 172.21.15.107:0/2895769391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563faee17230 con 0x563faed91210 2024-05-09T08:38:51.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.293+0000 7feea67fc640 1 -- 172.21.15.107:0/2895769391 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feea8004440 con 0x563faed91210 2024-05-09T08:38:51.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.294+0000 7feea67fc640 1 -- 172.21.15.107:0/2895769391 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7feeb002a170 con 0x563faed91210 2024-05-09T08:38:51.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.294+0000 7feec8221740 10 client.17259 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:51.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.294+0000 7feec8221740 10 client.17259 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:38:51.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.294+0000 7feec8221740 20 client.17259 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:51.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.294+0000 7feec8221740 1 --2- 172.21.15.107:0/2895769391 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563faee177c0 0x563faee19bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.294+0000 7feec8221740 4 client.17259 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:38:51.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.294+0000 7feec8221740 1 -- 172.21.15.107:0/2895769391 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563faecfe6c0 con 0x563faee177c0 2024-05-09T08:38:51.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.294+0000 7feeb7fff640 1 --2- 172.21.15.107:0/2895769391 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563faee177c0 0x563faee19bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.294+0000 7feea67fc640 10 client.17259 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:38:51.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.294+0000 7feeb7fff640 1 --2- 172.21.15.107:0/2895769391 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563faee177c0 0x563faee19bb0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8e4597cf3ccf2c50 server_cookie=24e8712e9d1c2c19 in_seq=0 out_seq=0 2024-05-09T08:38:51.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feea67fc640 1 -- 172.21.15.107:0/2895769391 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563faecfe6c0 con 0x563faee177c0 2024-05-09T08:38:51.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feea67fc640 10 client.17259 handle_command_reply: tid=0 2024-05-09T08:38:51.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feec8221740 1 client.17259 shutdown 2024-05-09T08:38:51.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feec8221740 1 -- 172.21.15.107:0/2895769391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91210 msgr2=0x563faed9a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feec8221740 1 --2- 172.21.15.107:0/2895769391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91210 0x563faed9a170 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7feeb00058c0 tx=0x7feeb0009310 comp rx=0 tx=0).stop 2024-05-09T08:38:51.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feec8221740 1 -- 172.21.15.107:0/2895769391 shutdown_connections 2024-05-09T08:38:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feec8221740 1 --2- 172.21.15.107:0/2895769391 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563faee177c0 0x563faee19bb0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feec8221740 1 --2- 172.21.15.107:0/2895769391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563faeda0730 0x563faed9abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feec8221740 1 --2- 172.21.15.107:0/2895769391 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563faed91b30 0x563faed9a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feea67fc640 0 client.17259 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:38:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feec8221740 1 --2- 172.21.15.107:0/2895769391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563faed91210 0x563faed9a170 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feec8221740 1 -- 172.21.15.107:0/2895769391 >> 172.21.15.107:0/2895769391 conn(0x563faed8d650 msgr2=0x563faed9d600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feec8221740 1 -- 172.21.15.107:0/2895769391 shutdown_connections 2024-05-09T08:38:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feec8221740 1 -- 172.21.15.107:0/2895769391 wait complete. 2024-05-09T08:38:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feec8221740 20 client.17259 trim_cache size 0 max 16384 2024-05-09T08:38:51.301 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:38:51.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feec5438640 1 -- 172.21.15.107:0/3251667315 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feeac0a2d00 msgr2=0x7feeac0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feec5438640 1 --2- 172.21.15.107:0/3251667315 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feeac0a2d00 0x7feeac0a51c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feea8000c00 tx=0x7feea8032c60 comp rx=0 tx=0).stop 2024-05-09T08:38:51.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feec5438640 1 -- 172.21.15.107:0/3251667315 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feec0106710 msgr2=0x7feec01a3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.295+0000 7feec5438640 1 --2- 172.21.15.107:0/3251667315 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feec0106710 0x7feec01a3f20 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7feeb800eb10 tx=0x7feeb800c2d0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.296+0000 7feec5438640 1 -- 172.21.15.107:0/3251667315 shutdown_connections 2024-05-09T08:38:51.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.296+0000 7feec5438640 1 --2- 172.21.15.107:0/3251667315 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feeac0a2d00 0x7feeac0a51c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.296+0000 7feec5438640 1 --2- 172.21.15.107:0/3251667315 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feec0106710 0x7feec01a3f20 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.296+0000 7feec5438640 1 --2- 172.21.15.107:0/3251667315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feec01010c0 0x7feec01993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.296+0000 7feec5438640 1 --2- 172.21.15.107:0/3251667315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec0100700 0x7feec0198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.296+0000 7feec5438640 1 -- 172.21.15.107:0/3251667315 >> 172.21.15.107:0/3251667315 conn(0x7feec0077830 msgr2=0x7feec01030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.296+0000 7feec5438640 1 -- 172.21.15.107:0/3251667315 shutdown_connections 2024-05-09T08:38:51.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.296+0000 7feec5438640 1 -- 172.21.15.107:0/3251667315 wait complete. 2024-05-09T08:38:51.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:38:51.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.424+0000 7faed6d77640 1 -- 172.21.15.107:0/3835521716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed01010c0 msgr2=0x7faed010d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.424+0000 7faed6d77640 1 --2- 172.21.15.107:0/3835521716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed01010c0 0x7faed010d5d0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7faec8009cd0 tx=0x7faec802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.424+0000 7faed6d77640 1 -- 172.21.15.107:0/3835521716 shutdown_connections 2024-05-09T08:38:51.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.424+0000 7faed6d77640 1 --2- 172.21.15.107:0/3835521716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed01010c0 0x7faed010d5d0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.424+0000 7faed6d77640 1 --2- 172.21.15.107:0/3835521716 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faed0100700 0x7faed0100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.424+0000 7faed6d77640 1 --2- 172.21.15.107:0/3835521716 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed0106710 0x7faed0106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.424+0000 7faed6d77640 1 -- 172.21.15.107:0/3835521716 >> 172.21.15.107:0/3835521716 conn(0x7faed0077830 msgr2=0x7faed0077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.424+0000 7faed6d77640 1 -- 172.21.15.107:0/3835521716 shutdown_connections 2024-05-09T08:38:51.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.424+0000 7faed6d77640 1 -- 172.21.15.107:0/3835521716 wait complete. 2024-05-09T08:38:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.424+0000 7faed6d77640 1 Processor -- start 2024-05-09T08:38:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.424+0000 7faed6d77640 1 -- start start 2024-05-09T08:38:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.424+0000 7faed6d77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed0100700 0x7faed0198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.424+0000 7faed6d77640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed01010c0 0x7faed0198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.424+0000 7faed6d77640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faed0106710 0x7faed019c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faed6d77640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faed01109d0 con 0x7faed0100700 2024-05-09T08:38:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faed6d77640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faed0110850 con 0x7faed01010c0 2024-05-09T08:38:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faed6d77640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faed0110b50 con 0x7faed0106710 2024-05-09T08:38:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faed4b13640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed0100700 0x7faed0198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faed4b13640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed0100700 0x7faed0198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54330/0 (socket says 172.21.15.107:54330) 2024-05-09T08:38:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faed4b13640 1 -- 172.21.15.107:0/1535428212 learned_addr learned my addr 172.21.15.107:0/1535428212 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faed5314640 1 --2- 172.21.15.107:0/1535428212 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faed0106710 0x7faed019c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faed4b13640 1 -- 172.21.15.107:0/1535428212 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faed0106710 msgr2=0x7faed019c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faed4b13640 1 --2- 172.21.15.107:0/1535428212 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faed0106710 0x7faed019c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faed4b13640 1 -- 172.21.15.107:0/1535428212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed01010c0 msgr2=0x7faed0198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faecffff640 1 --2- 172.21.15.107:0/1535428212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed01010c0 0x7faed0198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faed4b13640 1 --2- 172.21.15.107:0/1535428212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed01010c0 0x7faed0198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faed4b13640 1 -- 172.21.15.107:0/1535428212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed019d010 con 0x7faed0100700 2024-05-09T08:38:51.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faed4b13640 1 --2- 172.21.15.107:0/1535428212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed0100700 0x7faed0198020 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7faec400d950 tx=0x7faec400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faecdffb640 1 -- 172.21.15.107:0/1535428212 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec4017070 con 0x7faed0100700 2024-05-09T08:38:51.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faecdffb640 1 -- 172.21.15.107:0/1535428212 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faec4009c30 con 0x7faed0100700 2024-05-09T08:38:51.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faed6d77640 1 -- 172.21.15.107:0/1535428212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed019d300 con 0x7faed0100700 2024-05-09T08:38:51.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.425+0000 7faecdffb640 1 -- 172.21.15.107:0/1535428212 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec400f620 con 0x7faed0100700 2024-05-09T08:38:51.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.426+0000 7faed6d77640 1 -- 172.21.15.107:0/1535428212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed01a4b90 con 0x7faed0100700 2024-05-09T08:38:51.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.428+0000 7faecdffb640 1 -- 172.21.15.107:0/1535428212 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7faec40109a0 con 0x7faed0100700 2024-05-09T08:38:51.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.428+0000 7faed9b65740 1 Processor -- start 2024-05-09T08:38:51.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.428+0000 7faed9b65740 1 -- start start 2024-05-09T08:38:51.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.428+0000 7faecdffb640 1 --2- 172.21.15.107:0/1535428212 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faec00a2d00 0x7faec00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.428+0000 7faed9b65740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60330 0x5614fef60710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.428+0000 7faed9b65740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60c50 0x5614fef6ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.428+0000 7faed9b65740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef6b320 0x5614fef6d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.428+0000 7faed9b65740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5614feec5c40 con 0x5614fef60330 2024-05-09T08:38:51.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.428+0000 7faed9b65740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5614fee6d580 con 0x5614fef60c50 2024-05-09T08:38:51.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.428+0000 7faed9b65740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5614feec1c00 con 0x5614fef6b320 2024-05-09T08:38:51.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faed5314640 1 --2- 172.21.15.107:0/1535428212 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faec00a2d00 0x7faec00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faecdffb640 1 -- 172.21.15.107:0/1535428212 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7faec40cf430 con 0x7faed0100700 2024-05-09T08:38:51.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faed4b13640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60330 0x5614fef60710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faed4b13640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60330 0x5614fef60710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54336/0 (socket says 172.21.15.107:54336) 2024-05-09T08:38:51.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faed4b13640 1 -- 172.21.15.107:0/4127238357 learned_addr learned my addr 172.21.15.107:0/4127238357 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:51.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faed5314640 1 --2- 172.21.15.107:0/4127238357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef6b320 0x5614fef6d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faecffff640 1 --2- 172.21.15.107:0/4127238357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60c50 0x5614fef6ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faed4b13640 1 -- 172.21.15.107:0/4127238357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef6b320 msgr2=0x5614fef6d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faed4b13640 1 --2- 172.21.15.107:0/4127238357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef6b320 0x5614fef6d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faed4b13640 1 -- 172.21.15.107:0/4127238357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60c50 msgr2=0x5614fef6ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faed4b13640 1 --2- 172.21.15.107:0/4127238357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60c50 0x5614fef6ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faed4b13640 1 -- 172.21.15.107:0/4127238357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614febd93f0 con 0x5614fef60330 2024-05-09T08:38:51.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faed5314640 1 --2- 172.21.15.107:0/4127238357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef6b320 0x5614fef6d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:51.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faed4b13640 1 --2- 172.21.15.107:0/4127238357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60330 0x5614fef60710 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7faec40a3d30 tx=0x7faec409b230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a85910203d15407 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faed5314640 1 --2- 172.21.15.107:0/1535428212 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faec00a2d00 0x7faec00a51c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faec8005fe0 tx=0x7faec8005c30 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faebaffd640 1 -- 172.21.15.107:0/4127238357 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec4020040 con 0x5614fef60330 2024-05-09T08:38:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faebaffd640 1 -- 172.21.15.107:0/4127238357 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faec40c9f00 con 0x5614fef60330 2024-05-09T08:38:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.429+0000 7faebaffd640 1 -- 172.21.15.107:0/4127238357 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec4097c60 con 0x5614fef60330 2024-05-09T08:38:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.430+0000 7faed9b65740 1 -- 172.21.15.107:0/4127238357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60330 msgr2=0x5614fef60710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.430+0000 7faed9b65740 1 --2- 172.21.15.107:0/4127238357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60330 0x5614fef60710 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7faec40a3d30 tx=0x7faec409b230 comp rx=0 tx=0).stop 2024-05-09T08:38:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.430+0000 7faed9b65740 1 -- 172.21.15.107:0/4127238357 shutdown_connections 2024-05-09T08:38:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.430+0000 7faed9b65740 1 --2- 172.21.15.107:0/4127238357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef6b320 0x5614fef6d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.430+0000 7faed9b65740 1 --2- 172.21.15.107:0/4127238357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60c50 0x5614fef6ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.430+0000 7faed9b65740 1 --2- 172.21.15.107:0/4127238357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60330 0x5614fef60710 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.430+0000 7faed9b65740 1 -- 172.21.15.107:0/4127238357 >> 172.21.15.107:0/4127238357 conn(0x5614fef5c650 msgr2=0x5614fef5ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.430+0000 7faed9b65740 1 -- 172.21.15.107:0/4127238357 shutdown_connections 2024-05-09T08:38:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.430+0000 7faed9b65740 1 -- 172.21.15.107:0/4127238357 wait complete. 2024-05-09T08:38:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.431+0000 7faed9b65740 1 -- start start 2024-05-09T08:38:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.431+0000 7faed9b65740 1 Processor -- start 2024-05-09T08:38:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.431+0000 7faed9b65740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef60330 0x5614fef682a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.431+0000 7faed9b65740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60c50 0x5614fef687e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.431+0000 7faed9b65740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef6b320 0x5614fef68d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.431+0000 7faed9b65740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5614fef6e050 con 0x5614fef60c50 2024-05-09T08:38:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.431+0000 7faed9b65740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5614fee80c40 con 0x5614fef6b320 2024-05-09T08:38:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.431+0000 7faed9b65740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5614fef6e1d0 con 0x5614fef60330 2024-05-09T08:38:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.431+0000 7faed9b65740 10 client.0 _ll_register_callbacks cb 0x5614febd8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.431+0000 7faecffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60c50 0x5614fef687e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.431+0000 7faecffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60c50 0x5614fef687e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54346/0 (socket says 172.21.15.107:54346) 2024-05-09T08:38:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.431+0000 7faecffff640 1 -- 172.21.15.107:0/536588749 learned_addr learned my addr 172.21.15.107:0/536588749 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.431+0000 7faed4b13640 1 --2- 172.21.15.107:0/536588749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef60330 0x5614fef682a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.431+0000 7faed5314640 1 --2- 172.21.15.107:0/536588749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef6b320 0x5614fef68d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.432+0000 7faecffff640 1 -- 172.21.15.107:0/536588749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef60330 msgr2=0x5614fef682a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.432+0000 7faecffff640 1 --2- 172.21.15.107:0/536588749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef60330 0x5614fef682a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.432+0000 7faecffff640 1 -- 172.21.15.107:0/536588749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef6b320 msgr2=0x5614fef68d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.432+0000 7faecffff640 1 --2- 172.21.15.107:0/536588749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef6b320 0x5614fef68d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.432+0000 7faecffff640 1 -- 172.21.15.107:0/536588749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614feca37d0 con 0x5614fef60c50 2024-05-09T08:38:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.432+0000 7faed5314640 1 --2- 172.21.15.107:0/536588749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef6b320 0x5614fef68d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.432+0000 7faecffff640 1 --2- 172.21.15.107:0/536588749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60c50 0x5614fef687e0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7faebc0136e0 tx=0x7faebc013ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f245ddcfcc4b9b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.432+0000 7faeaffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.432+0000 7faeaffff640 1 -- 172.21.15.107:0/536588749 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faebc011dd0 con 0x5614fef60c50 2024-05-09T08:38:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.432+0000 7faeaffff640 1 -- 172.21.15.107:0/536588749 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7faebc004020 con 0x5614fef60c50 2024-05-09T08:38:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.432+0000 7faed9b65740 1 -- 172.21.15.107:0/536588749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faec40c9f00 con 0x5614fef60c50 2024-05-09T08:38:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.432+0000 7faeaffff640 1 -- 172.21.15.107:0/536588749 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faebc004220 con 0x5614fef60c50 2024-05-09T08:38:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.432+0000 7faed9b65740 10 client.17271 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.432+0000 7faed9b65740 1 -- 172.21.15.107:0/536588749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5614fef7a290 con 0x5614fef60c50 2024-05-09T08:38:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.432+0000 7faeaffff640 1 -- 172.21.15.107:0/536588749 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faec40c9f00 con 0x5614fef60c50 2024-05-09T08:38:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.432+0000 7faeaffff640 1 -- 172.21.15.107:0/536588749 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7faebc01a070 con 0x5614fef60c50 2024-05-09T08:38:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.433+0000 7faed9b65740 10 client.17271 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.433+0000 7faed9b65740 10 client.17271 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:38:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.433+0000 7faed9b65740 20 client.17271 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.433+0000 7faed9b65740 1 --2- 172.21.15.107:0/536588749 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5614fef7b7a0 0x5614fef7dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.433+0000 7faed9b65740 4 client.17271 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.433+0000 7faed9b65740 1 -- 172.21.15.107:0/536588749 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614feed3370 con 0x5614fef7b7a0 2024-05-09T08:38:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.433+0000 7faed4b13640 1 --2- 172.21.15.107:0/536588749 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5614fef7b7a0 0x5614fef7dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.433+0000 7faed4b13640 1 --2- 172.21.15.107:0/536588749 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5614fef7b7a0 0x5614fef7dc20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e79cd92bb67bc8c9 server_cookie=7b26cc08f3a15c2b in_seq=0 out_seq=0 2024-05-09T08:38:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.433+0000 7faeaffff640 10 client.17271 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:38:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.434+0000 7faeaffff640 1 -- 172.21.15.107:0/536588749 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5614feed3370 con 0x5614fef7b7a0 2024-05-09T08:38:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.434+0000 7faeaffff640 10 client.17271 handle_command_reply: tid=0 2024-05-09T08:38:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.434+0000 7faed9b65740 1 client.17271 shutdown 2024-05-09T08:38:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.434+0000 7faed9b65740 1 -- 172.21.15.107:0/536588749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60c50 msgr2=0x5614fef687e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.434+0000 7faed9b65740 1 --2- 172.21.15.107:0/536588749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60c50 0x5614fef687e0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7faebc0136e0 tx=0x7faebc013ba0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.434+0000 7faed9b65740 1 -- 172.21.15.107:0/536588749 shutdown_connections 2024-05-09T08:38:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.434+0000 7faed9b65740 1 --2- 172.21.15.107:0/536588749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef6b320 0x5614fef68d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.434+0000 7faed9b65740 1 --2- 172.21.15.107:0/536588749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60c50 0x5614fef687e0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.434+0000 7faed9b65740 1 --2- 172.21.15.107:0/536588749 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5614fef7b7a0 0x5614fef7dc20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.434+0000 7faed9b65740 1 --2- 172.21.15.107:0/536588749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef60330 0x5614fef682a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.434+0000 7faeaffff640 0 client.17271 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:38:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.434+0000 7faed9b65740 1 -- 172.21.15.107:0/536588749 >> 172.21.15.107:0/536588749 conn(0x5614fef5c650 msgr2=0x5614fef62350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.434+0000 7faed9b65740 1 -- 172.21.15.107:0/536588749 shutdown_connections 2024-05-09T08:38:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.434+0000 7faed9b65740 1 -- 172.21.15.107:0/536588749 wait complete. 2024-05-09T08:38:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.434+0000 7faed9b65740 20 client.17271 trim_cache size 0 max 16384 2024-05-09T08:38:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.450+0000 7faed9b65740 1 Processor -- start 2024-05-09T08:38:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.450+0000 7faed9b65740 1 -- start start 2024-05-09T08:38:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.450+0000 7faed9b65740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60330 0x5614fef76f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.450+0000 7faed9b65740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60c50 0x5614fef67400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.450+0000 7faed9b65740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef6b320 0x5614fef62a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.450+0000 7faed9b65740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faed01b1440 con 0x5614fef60330 2024-05-09T08:38:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.450+0000 7faed9b65740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faed01aed10 con 0x5614fef60c50 2024-05-09T08:38:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.450+0000 7faed9b65740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faed01acb60 con 0x5614fef6b320 2024-05-09T08:38:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.451+0000 7faed4b13640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60330 0x5614fef76f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.451+0000 7faed4b13640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60330 0x5614fef76f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54358/0 (socket says 172.21.15.107:54358) 2024-05-09T08:38:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.451+0000 7faed4b13640 1 -- 172.21.15.107:0/1661995397 learned_addr learned my addr 172.21.15.107:0/1661995397 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.451+0000 7faed5314640 1 --2- 172.21.15.107:0/1661995397 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef6b320 0x5614fef62a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.451+0000 7faecffff640 1 --2- 172.21.15.107:0/1661995397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60c50 0x5614fef67400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.451+0000 7faed4b13640 1 -- 172.21.15.107:0/1661995397 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef6b320 msgr2=0x5614fef62a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.451+0000 7faed4b13640 1 --2- 172.21.15.107:0/1661995397 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef6b320 0x5614fef62a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.451+0000 7faed4b13640 1 -- 172.21.15.107:0/1661995397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60c50 msgr2=0x5614fef67400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.451+0000 7faed4b13640 1 --2- 172.21.15.107:0/1661995397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60c50 0x5614fef67400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.451+0000 7faed4b13640 1 -- 172.21.15.107:0/1661995397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614fef4dd20 con 0x5614fef60330 2024-05-09T08:38:51.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.451+0000 7faecffff640 1 --2- 172.21.15.107:0/1661995397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60c50 0x5614fef67400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:38:51.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.451+0000 7faed4b13640 1 --2- 172.21.15.107:0/1661995397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60330 0x5614fef76f10 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7faec4097fc0 tx=0x7faec4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695a1f0311ed342d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.451+0000 7faeba7fc640 1 -- 172.21.15.107:0/1661995397 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec4020040 con 0x5614fef60330 2024-05-09T08:38:51.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.451+0000 7faeba7fc640 1 -- 172.21.15.107:0/1661995397 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faec40a4260 con 0x5614fef60330 2024-05-09T08:38:51.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.451+0000 7faeba7fc640 1 -- 172.21.15.107:0/1661995397 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec4097c60 con 0x5614fef60330 2024-05-09T08:38:51.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.452+0000 7faed9b65740 1 -- 172.21.15.107:0/1661995397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60330 msgr2=0x5614fef76f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.452+0000 7faed9b65740 1 --2- 172.21.15.107:0/1661995397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60330 0x5614fef76f10 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7faec4097fc0 tx=0x7faec4005880 comp rx=0 tx=0).stop 2024-05-09T08:38:51.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.452+0000 7faed9b65740 1 -- 172.21.15.107:0/1661995397 shutdown_connections 2024-05-09T08:38:51.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.452+0000 7faed9b65740 1 --2- 172.21.15.107:0/1661995397 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef6b320 0x5614fef62a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.452+0000 7faed9b65740 1 --2- 172.21.15.107:0/1661995397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60c50 0x5614fef67400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.452+0000 7faed9b65740 1 --2- 172.21.15.107:0/1661995397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef60330 0x5614fef76f10 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.452+0000 7faed9b65740 1 -- 172.21.15.107:0/1661995397 >> 172.21.15.107:0/1661995397 conn(0x5614fef5c650 msgr2=0x5614fef68260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.452+0000 7faed9b65740 1 -- 172.21.15.107:0/1661995397 shutdown_connections 2024-05-09T08:38:51.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.452+0000 7faed9b65740 1 -- 172.21.15.107:0/1661995397 wait complete. 2024-05-09T08:38:51.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.452+0000 7faed9b65740 1 -- start start 2024-05-09T08:38:51.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.452+0000 7faed9b65740 1 Processor -- start 2024-05-09T08:38:51.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed9b65740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60330 0x5614fef73540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed9b65740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef60c50 0x5614fef6e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed9b65740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef6b320 0x5614fef6ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed9b65740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faed01ad1d0 con 0x5614fef6b320 2024-05-09T08:38:51.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed9b65740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faed01ae640 con 0x5614fef60330 2024-05-09T08:38:51.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed9b65740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faed01ad8a0 con 0x5614fef60c50 2024-05-09T08:38:51.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed9b65740 10 client.0 _ll_register_callbacks cb 0x7faebc005870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:51.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed5314640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef6b320 0x5614fef6ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed4b13640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60330 0x5614fef73540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed5314640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef6b320 0x5614fef6ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54362/0 (socket says 172.21.15.107:54362) 2024-05-09T08:38:51.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed5314640 1 -- 172.21.15.107:0/4087123580 learned_addr learned my addr 172.21.15.107:0/4087123580 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:51.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faecffff640 1 --2- 172.21.15.107:0/4087123580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef60c50 0x5614fef6e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed5314640 1 -- 172.21.15.107:0/4087123580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef60c50 msgr2=0x5614fef6e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed5314640 1 --2- 172.21.15.107:0/4087123580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef60c50 0x5614fef6e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed5314640 1 -- 172.21.15.107:0/4087123580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60330 msgr2=0x5614fef73540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed5314640 1 --2- 172.21.15.107:0/4087123580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60330 0x5614fef73540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed5314640 1 -- 172.21.15.107:0/4087123580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614fef6f490 con 0x5614fef6b320 2024-05-09T08:38:51.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.453+0000 7faed4b13640 1 --2- 172.21.15.107:0/4087123580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60330 0x5614fef73540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:51.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.454+0000 7faed5314640 1 --2- 172.21.15.107:0/4087123580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef6b320 0x5614fef6ef20 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7faec8009cd0 tx=0x7faec80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdde6fed3596b9b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.454+0000 7faeaffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:51.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.454+0000 7faeaffff640 1 -- 172.21.15.107:0/4087123580 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec804f070 con 0x5614fef6b320 2024-05-09T08:38:51.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.454+0000 7faeaffff640 1 -- 172.21.15.107:0/4087123580 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7faec8003e00 con 0x5614fef6b320 2024-05-09T08:38:51.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.454+0000 7faed9b65740 1 -- 172.21.15.107:0/4087123580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faec40a4260 con 0x5614fef6b320 2024-05-09T08:38:51.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.454+0000 7faeaffff640 1 -- 172.21.15.107:0/4087123580 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faec80047a0 con 0x5614fef6b320 2024-05-09T08:38:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.454+0000 7faed9b65740 10 client.17277 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.454+0000 7faed9b65740 1 -- 172.21.15.107:0/4087123580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5614fefe1ab0 con 0x5614fef6b320 2024-05-09T08:38:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.454+0000 7faeaffff640 1 -- 172.21.15.107:0/4087123580 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faec40a4260 con 0x5614fef6b320 2024-05-09T08:38:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.454+0000 7faeaffff640 1 -- 172.21.15.107:0/4087123580 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7faec8030070 con 0x5614fef6b320 2024-05-09T08:38:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.454+0000 7faed9b65740 10 client.17277 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.454+0000 7faed9b65740 10 client.17277 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:38:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.454+0000 7faed9b65740 20 client.17277 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.454+0000 7faed9b65740 1 --2- 172.21.15.107:0/4087123580 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5614fefe2290 0x5614fefe46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.454+0000 7faed9b65740 4 client.17277 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:38:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.454+0000 7faed9b65740 1 -- 172.21.15.107:0/4087123580 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5614feecd6c0 con 0x5614fefe2290 2024-05-09T08:38:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.455+0000 7faed4b13640 1 --2- 172.21.15.107:0/4087123580 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5614fefe2290 0x5614fefe46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.455+0000 7faed4b13640 1 --2- 172.21.15.107:0/4087123580 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5614fefe2290 0x5614fefe46b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3805465f6489f2de server_cookie=7e410089c2128fa4 in_seq=0 out_seq=0 2024-05-09T08:38:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.455+0000 7faeaffff640 10 client.17277 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:38:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.455+0000 7faeaffff640 1 -- 172.21.15.107:0/4087123580 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5614feecd6c0 con 0x5614fefe2290 2024-05-09T08:38:51.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.455+0000 7faeaffff640 10 client.17277 handle_command_reply: tid=0 2024-05-09T08:38:51.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.455+0000 7faed9b65740 1 client.17277 shutdown 2024-05-09T08:38:51.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.455+0000 7faed9b65740 1 -- 172.21.15.107:0/4087123580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef6b320 msgr2=0x5614fef6ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.456+0000 7faed9b65740 1 --2- 172.21.15.107:0/4087123580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef6b320 0x5614fef6ef20 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7faec8009cd0 tx=0x7faec80095c0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.456+0000 7faed9b65740 1 -- 172.21.15.107:0/4087123580 shutdown_connections 2024-05-09T08:38:51.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.456+0000 7faed9b65740 1 --2- 172.21.15.107:0/4087123580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5614fef6b320 0x5614fef6ef20 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.456+0000 7faed9b65740 1 --2- 172.21.15.107:0/4087123580 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5614fefe2290 0x5614fefe46b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.456+0000 7faed9b65740 1 --2- 172.21.15.107:0/4087123580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614fef60c50 0x5614fef6e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.456+0000 7faeaffff640 0 client.17277 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:38:51.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.456+0000 7faed9b65740 1 --2- 172.21.15.107:0/4087123580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614fef60330 0x5614fef73540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.456+0000 7faed9b65740 1 -- 172.21.15.107:0/4087123580 >> 172.21.15.107:0/4087123580 conn(0x5614fef5c650 msgr2=0x5614fef67280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.456+0000 7faed9b65740 1 -- 172.21.15.107:0/4087123580 shutdown_connections 2024-05-09T08:38:51.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.456+0000 7faed9b65740 1 -- 172.21.15.107:0/4087123580 wait complete. 2024-05-09T08:38:51.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.456+0000 7faed9b65740 20 client.17277 trim_cache size 0 max 16384 2024-05-09T08:38:51.500 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:38:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.456+0000 7faed6d77640 1 -- 172.21.15.107:0/1535428212 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faec00a2d00 msgr2=0x7faec00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.457+0000 7faed6d77640 1 --2- 172.21.15.107:0/1535428212 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faec00a2d00 0x7faec00a51c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faec8005fe0 tx=0x7faec8005c30 comp rx=0 tx=0).stop 2024-05-09T08:38:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.457+0000 7faed6d77640 1 -- 172.21.15.107:0/1535428212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed0100700 msgr2=0x7faed0198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.457+0000 7faed6d77640 1 --2- 172.21.15.107:0/1535428212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed0100700 0x7faed0198020 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7faec400d950 tx=0x7faec400de10 comp rx=0 tx=0).stop 2024-05-09T08:38:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.457+0000 7faed6d77640 1 -- 172.21.15.107:0/1535428212 shutdown_connections 2024-05-09T08:38:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.457+0000 7faed6d77640 1 --2- 172.21.15.107:0/1535428212 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faec00a2d00 0x7faec00a51c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.457+0000 7faed6d77640 1 --2- 172.21.15.107:0/1535428212 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faed0106710 0x7faed019c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.457+0000 7faed6d77640 1 --2- 172.21.15.107:0/1535428212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed01010c0 0x7faed0198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.457+0000 7faed6d77640 1 --2- 172.21.15.107:0/1535428212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed0100700 0x7faed0198020 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.457+0000 7faed6d77640 1 -- 172.21.15.107:0/1535428212 >> 172.21.15.107:0/1535428212 conn(0x7faed0077830 msgr2=0x7faed0068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.457+0000 7faed6d77640 1 -- 172.21.15.107:0/1535428212 shutdown_connections 2024-05-09T08:38:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.457+0000 7faed6d77640 1 -- 172.21.15.107:0/1535428212 wait complete. 2024-05-09T08:38:51.502 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:51 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3623908991' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:38:51.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:38:51.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.619+0000 7f3fc37c8640 1 -- 172.21.15.107:0/2278938881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fbc100710 msgr2=0x7f3fbc100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.619+0000 7f3fc37c8640 1 --2- 172.21.15.107:0/2278938881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fbc100710 0x7f3fbc100b90 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f3fac009930 tx=0x7f3fac02a420 comp rx=0 tx=0).stop 2024-05-09T08:38:51.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.619+0000 7f3fc37c8640 1 -- 172.21.15.107:0/2278938881 shutdown_connections 2024-05-09T08:38:51.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.619+0000 7f3fc37c8640 1 --2- 172.21.15.107:0/2278938881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3fbc1010d0 0x7f3fbc10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.619+0000 7f3fc37c8640 1 --2- 172.21.15.107:0/2278938881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fbc100710 0x7f3fbc100b90 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.619+0000 7f3fc37c8640 1 --2- 172.21.15.107:0/2278938881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fbc106720 0x7f3fbc106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.619+0000 7f3fc37c8640 1 -- 172.21.15.107:0/2278938881 >> 172.21.15.107:0/2278938881 conn(0x7f3fbc077790 msgr2=0x7f3fbc077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.619+0000 7f3fc37c8640 1 -- 172.21.15.107:0/2278938881 shutdown_connections 2024-05-09T08:38:51.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.619+0000 7f3fc37c8640 1 -- 172.21.15.107:0/2278938881 wait complete. 2024-05-09T08:38:51.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.619+0000 7f3fc37c8640 1 Processor -- start 2024-05-09T08:38:51.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.619+0000 7f3fc37c8640 1 -- start start 2024-05-09T08:38:51.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc37c8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3fbc100710 0x7f3fbc198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc37c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fbc1010d0 0x7f3fbc1993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc37c8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fbc106720 0x7f3fbc1a3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc37c8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc1109e0 con 0x7f3fbc1010d0 2024-05-09T08:38:51.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc37c8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc110860 con 0x7f3fbc106720 2024-05-09T08:38:51.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc37c8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc110b60 con 0x7f3fbc100710 2024-05-09T08:38:51.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc0d63640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fbc1010d0 0x7f3fbc1993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc1564640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3fbc100710 0x7f3fbc198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc0d63640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fbc1010d0 0x7f3fbc1993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54380/0 (socket says 172.21.15.107:54380) 2024-05-09T08:38:51.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc0d63640 1 -- 172.21.15.107:0/4095598256 learned_addr learned my addr 172.21.15.107:0/4095598256 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:51.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc1564640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3fbc100710 0x7f3fbc198e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:37798/0 (socket says 172.21.15.107:37798) 2024-05-09T08:38:51.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc1d65640 1 --2- 172.21.15.107:0/4095598256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fbc106720 0x7f3fbc1a3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc0d63640 1 -- 172.21.15.107:0/4095598256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3fbc100710 msgr2=0x7f3fbc198e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc0d63640 1 --2- 172.21.15.107:0/4095598256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3fbc100710 0x7f3fbc198e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc0d63640 1 -- 172.21.15.107:0/4095598256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fbc106720 msgr2=0x7f3fbc1a3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc0d63640 1 --2- 172.21.15.107:0/4095598256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fbc106720 0x7f3fbc1a3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc0d63640 1 -- 172.21.15.107:0/4095598256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fbc1a4440 con 0x7f3fbc1010d0 2024-05-09T08:38:51.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc1d65640 1 --2- 172.21.15.107:0/4095598256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fbc106720 0x7f3fbc1a3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:51.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.620+0000 7f3fc0d63640 1 --2- 172.21.15.107:0/4095598256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fbc1010d0 0x7f3fbc1993c0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f3fac005940 tx=0x7f3fac0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.621+0000 7f3fb27fc640 1 -- 172.21.15.107:0/4095598256 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fac03f030 con 0x7f3fbc1010d0 2024-05-09T08:38:51.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.621+0000 7f3fb27fc640 1 -- 172.21.15.107:0/4095598256 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fac0318a0 con 0x7f3fbc1010d0 2024-05-09T08:38:51.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.621+0000 7f3fc37c8640 1 -- 172.21.15.107:0/4095598256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fbc1a4610 con 0x7f3fbc1010d0 2024-05-09T08:38:51.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.621+0000 7f3fb27fc640 1 -- 172.21.15.107:0/4095598256 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fac035070 con 0x7f3fbc1010d0 2024-05-09T08:38:51.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.621+0000 7f3fc37c8640 1 -- 172.21.15.107:0/4095598256 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fbc1a4ad0 con 0x7f3fbc1010d0 2024-05-09T08:38:51.624 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:51 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3623908991' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:38:51.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.624+0000 7f3fc65b2740 1 Processor -- start 2024-05-09T08:38:51.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.624+0000 7f3fb27fc640 1 -- 172.21.15.107:0/4095598256 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3fac042030 con 0x7f3fbc1010d0 2024-05-09T08:38:51.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.624+0000 7f3fc65b2740 1 -- start start 2024-05-09T08:38:51.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.624+0000 7f3fc65b2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3330 0x55b9be5a3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.624+0000 7f3fc65b2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5a3c50 0x55b9be5adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.624+0000 7f3fc65b2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5ae320 0x55b9be5b0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.624+0000 7f3fc65b2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b9be508c40 con 0x55b9be5ae320 2024-05-09T08:38:51.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.624+0000 7f3fc65b2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b9be4b0580 con 0x55b9be5a3330 2024-05-09T08:38:51.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.624+0000 7f3fc65b2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55b9be504c00 con 0x55b9be5a3c50 2024-05-09T08:38:51.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fc0d63640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5ae320 0x55b9be5b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fc1564640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5a3c50 0x55b9be5adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fc1d65640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3330 0x55b9be5a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fc1564640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5a3c50 0x55b9be5adde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:37806/0 (socket says 172.21.15.107:37806) 2024-05-09T08:38:51.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fc1d65640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3330 0x55b9be5a3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:54576/0 (socket says 172.21.15.107:54576) 2024-05-09T08:38:51.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fc1564640 1 -- 172.21.15.107:0/874756413 learned_addr learned my addr 172.21.15.107:0/874756413 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:51.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fc1564640 1 -- 172.21.15.107:0/874756413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3330 msgr2=0x55b9be5a3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fc1564640 1 --2- 172.21.15.107:0/874756413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3330 0x55b9be5a3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fc1564640 1 -- 172.21.15.107:0/874756413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5ae320 msgr2=0x55b9be5b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fc1564640 1 --2- 172.21.15.107:0/874756413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5ae320 0x55b9be5b0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fc1564640 1 -- 172.21.15.107:0/874756413 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9be21c3f0 con 0x55b9be5a3c50 2024-05-09T08:38:51.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fb27fc640 1 --2- 172.21.15.107:0/4095598256 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3fa80a2d80 0x7f3fa80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fc1d65640 1 --2- 172.21.15.107:0/4095598256 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3fa80a2d80 0x7f3fa80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fc1564640 1 --2- 172.21.15.107:0/874756413 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5a3c50 0x55b9be5adde0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800bc90 tx=0x7f3fb80107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a516af7ed45f2c98 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fb27fc640 1 -- 172.21.15.107:0/4095598256 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3fac0eff10 con 0x7f3fbc1010d0 2024-05-09T08:38:51.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fa77fe640 1 -- 172.21.15.107:0/874756413 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb801c480 con 0x55b9be5a3c50 2024-05-09T08:38:51.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fa77fe640 1 -- 172.21.15.107:0/874756413 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fb801cad0 con 0x55b9be5a3c50 2024-05-09T08:38:51.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.625+0000 7f3fa77fe640 1 -- 172.21.15.107:0/874756413 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8011ae0 con 0x55b9be5a3c50 2024-05-09T08:38:51.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.626+0000 7f3fc65b2740 1 -- 172.21.15.107:0/874756413 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5a3c50 msgr2=0x55b9be5adde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.626+0000 7f3fc1d65640 1 --2- 172.21.15.107:0/4095598256 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3fa80a2d80 0x7f3fa80a5240 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4005fd0 tx=0x7f3fb4009500 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.626+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/874756413 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5a3c50 0x55b9be5adde0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800bc90 tx=0x7f3fb80107a0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.626+0000 7f3fc65b2740 1 -- 172.21.15.107:0/874756413 shutdown_connections 2024-05-09T08:38:51.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.626+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/874756413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5ae320 0x55b9be5b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.626+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/874756413 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5a3c50 0x55b9be5adde0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.626+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/874756413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3330 0x55b9be5a3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.626+0000 7f3fc65b2740 1 -- 172.21.15.107:0/874756413 >> 172.21.15.107:0/874756413 conn(0x55b9be59f650 msgr2=0x55b9be59fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.626+0000 7f3fc65b2740 1 -- 172.21.15.107:0/874756413 shutdown_connections 2024-05-09T08:38:51.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.626+0000 7f3fc65b2740 1 -- 172.21.15.107:0/874756413 wait complete. 2024-05-09T08:38:51.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.627+0000 7f3fc65b2740 1 -- start start 2024-05-09T08:38:51.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.627+0000 7f3fc65b2740 1 Processor -- start 2024-05-09T08:38:51.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.627+0000 7f3fc65b2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3330 0x55b9be5ab260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.627+0000 7f3fc65b2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3c50 0x55b9be5ab7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.627+0000 7f3fc65b2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5ae320 0x55b9be5abce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.627+0000 7f3fc65b2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b9be5b1110 con 0x55b9be5a3c50 2024-05-09T08:38:51.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.627+0000 7f3fc65b2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b9be4c3c40 con 0x55b9be5a3330 2024-05-09T08:38:51.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.627+0000 7f3fc65b2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55b9be5b1290 con 0x55b9be5ae320 2024-05-09T08:38:51.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.627+0000 7f3fc65b2740 10 client.0 _ll_register_callbacks cb 0x55b9be21bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:51.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.627+0000 7f3fc0d63640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3c50 0x55b9be5ab7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.627+0000 7f3fc0d63640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3c50 0x55b9be5ab7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54404/0 (socket says 172.21.15.107:54404) 2024-05-09T08:38:51.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.627+0000 7f3fc1d65640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5ae320 0x55b9be5abce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.627+0000 7f3fc0d63640 1 -- 172.21.15.107:0/3813557101 learned_addr learned my addr 172.21.15.107:0/3813557101 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:51.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.627+0000 7f3fc1564640 1 --2- 172.21.15.107:0/3813557101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3330 0x55b9be5ab260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.628+0000 7f3fc0d63640 1 -- 172.21.15.107:0/3813557101 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5ae320 msgr2=0x55b9be5abce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.628+0000 7f3fc0d63640 1 --2- 172.21.15.107:0/3813557101 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5ae320 0x55b9be5abce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.628+0000 7f3fc0d63640 1 -- 172.21.15.107:0/3813557101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3330 msgr2=0x55b9be5ab260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.628+0000 7f3fc0d63640 1 --2- 172.21.15.107:0/3813557101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3330 0x55b9be5ab260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.628+0000 7f3fc0d63640 1 -- 172.21.15.107:0/3813557101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9be2e67d0 con 0x55b9be5a3c50 2024-05-09T08:38:51.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.628+0000 7f3fc0d63640 1 --2- 172.21.15.107:0/3813557101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3c50 0x55b9be5ab7a0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f3fac031440 tx=0x7f3fac0b7bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f95d2474be9c138 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.628+0000 7f3fa4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:51.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.628+0000 7f3fa4ff9640 1 -- 172.21.15.107:0/3813557101 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fac049050 con 0x55b9be5a3c50 2024-05-09T08:38:51.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.628+0000 7f3fa4ff9640 1 -- 172.21.15.107:0/3813557101 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3fac0ed520 con 0x55b9be5a3c50 2024-05-09T08:38:51.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.628+0000 7f3fc65b2740 1 -- 172.21.15.107:0/3813557101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fb801cad0 con 0x55b9be5a3c50 2024-05-09T08:38:51.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.628+0000 7f3fa4ff9640 1 -- 172.21.15.107:0/3813557101 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3fac0bbbb0 con 0x55b9be5a3c50 2024-05-09T08:38:51.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.628+0000 7f3fc65b2740 10 client.17289 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:51.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.628+0000 7f3fc65b2740 1 -- 172.21.15.107:0/3813557101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b9be5bd300 con 0x55b9be5a3c50 2024-05-09T08:38:51.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.628+0000 7f3fa4ff9640 1 -- 172.21.15.107:0/3813557101 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fb801cad0 con 0x55b9be5a3c50 2024-05-09T08:38:51.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.628+0000 7f3fa4ff9640 1 -- 172.21.15.107:0/3813557101 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3fac0bc2d0 con 0x55b9be5a3c50 2024-05-09T08:38:51.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.629+0000 7f3fc65b2740 10 client.17289 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:51.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.629+0000 7f3fc65b2740 10 client.17289 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:51.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.629+0000 7f3fc65b2740 20 client.17289 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:51.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.629+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/3813557101 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b9be5be650 0x55b9be5c0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.629+0000 7f3fc65b2740 4 client.17289 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:51.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.629+0000 7f3fc65b2740 1 -- 172.21.15.107:0/3813557101 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9be516370 con 0x55b9be5be650 2024-05-09T08:38:51.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.629+0000 7f3fc1564640 1 --2- 172.21.15.107:0/3813557101 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b9be5be650 0x55b9be5c0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.629+0000 7f3fc1564640 1 --2- 172.21.15.107:0/3813557101 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b9be5be650 0x55b9be5c0aa0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2630c8f001236a1 server_cookie=6b88cbe615e35908 in_seq=0 out_seq=0 2024-05-09T08:38:51.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.629+0000 7f3fa4ff9640 10 client.17289 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:51.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.630+0000 7f3fa4ff9640 1 -- 172.21.15.107:0/3813557101 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b9be516370 con 0x55b9be5be650 2024-05-09T08:38:51.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.630+0000 7f3fa4ff9640 10 client.17289 handle_command_reply: tid=0 2024-05-09T08:38:51.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.630+0000 7f3fc65b2740 1 client.17289 shutdown 2024-05-09T08:38:51.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.630+0000 7f3fc65b2740 1 -- 172.21.15.107:0/3813557101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3c50 msgr2=0x55b9be5ab7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.630+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/3813557101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3c50 0x55b9be5ab7a0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f3fac031440 tx=0x7f3fac0b7bf0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.630+0000 7f3fc65b2740 1 -- 172.21.15.107:0/3813557101 shutdown_connections 2024-05-09T08:38:51.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.630+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/3813557101 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b9be5be650 0x55b9be5c0aa0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.630+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/3813557101 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5ae320 0x55b9be5abce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.630+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/3813557101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3c50 0x55b9be5ab7a0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.630+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/3813557101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3330 0x55b9be5ab260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.630+0000 7f3fc65b2740 1 -- 172.21.15.107:0/3813557101 >> 172.21.15.107:0/3813557101 conn(0x55b9be59f650 msgr2=0x55b9be5a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.630+0000 7f3fa4ff9640 0 client.17289 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:51.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.630+0000 7f3fc65b2740 1 -- 172.21.15.107:0/3813557101 shutdown_connections 2024-05-09T08:38:51.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.630+0000 7f3fc65b2740 1 -- 172.21.15.107:0/3813557101 wait complete. 2024-05-09T08:38:51.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.630+0000 7f3fc65b2740 20 client.17289 trim_cache size 0 max 16384 2024-05-09T08:38:51.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.645+0000 7f3fc65b2740 1 Processor -- start 2024-05-09T08:38:51.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.645+0000 7f3fc65b2740 1 -- start start 2024-05-09T08:38:51.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc65b2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3330 0x55b9be5ba000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc65b2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3c50 0x55b9be5c6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc65b2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5ae320 0x55b9be5c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc65b2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc1b13d0 con 0x55b9be5a3330 2024-05-09T08:38:51.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc65b2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc1aeca0 con 0x55b9be5a3c50 2024-05-09T08:38:51.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc65b2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc1acaf0 con 0x55b9be5ae320 2024-05-09T08:38:51.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc1564640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3330 0x55b9be5ba000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc1564640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3330 0x55b9be5ba000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54412/0 (socket says 172.21.15.107:54412) 2024-05-09T08:38:51.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc1564640 1 -- 172.21.15.107:0/4268855146 learned_addr learned my addr 172.21.15.107:0/4268855146 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:51.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc1d65640 1 --2- 172.21.15.107:0/4268855146 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5ae320 0x55b9be5c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc1564640 1 -- 172.21.15.107:0/4268855146 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5ae320 msgr2=0x55b9be5c68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc0d63640 1 --2- 172.21.15.107:0/4268855146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3c50 0x55b9be5c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc1564640 1 --2- 172.21.15.107:0/4268855146 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5ae320 0x55b9be5c68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc1564640 1 -- 172.21.15.107:0/4268855146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3c50 msgr2=0x55b9be5c6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc1564640 1 --2- 172.21.15.107:0/4268855146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3c50 0x55b9be5c6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc1564640 1 -- 172.21.15.107:0/4268855146 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9be590d20 con 0x55b9be5a3330 2024-05-09T08:38:51.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc0d63640 1 --2- 172.21.15.107:0/4268855146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3c50 0x55b9be5c6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:38:51.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fc1564640 1 --2- 172.21.15.107:0/4268855146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3330 0x55b9be5ba000 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x55b9be5a48f0 tx=0x7f3fb800b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa0ca9dec29cdc4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.646+0000 7f3fa6ffd640 1 -- 172.21.15.107:0/4268855146 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb802e070 con 0x55b9be5a3330 2024-05-09T08:38:51.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.647+0000 7f3fa6ffd640 1 -- 172.21.15.107:0/4268855146 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fb80298a0 con 0x55b9be5a3330 2024-05-09T08:38:51.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.647+0000 7f3fa6ffd640 1 -- 172.21.15.107:0/4268855146 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8023040 con 0x55b9be5a3330 2024-05-09T08:38:51.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.647+0000 7f3fc65b2740 1 -- 172.21.15.107:0/4268855146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3330 msgr2=0x55b9be5ba000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.647+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/4268855146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3330 0x55b9be5ba000 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x55b9be5a48f0 tx=0x7f3fb800b390 comp rx=0 tx=0).stop 2024-05-09T08:38:51.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.647+0000 7f3fc65b2740 1 -- 172.21.15.107:0/4268855146 shutdown_connections 2024-05-09T08:38:51.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.647+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/4268855146 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5ae320 0x55b9be5c68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.647+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/4268855146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3c50 0x55b9be5c6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.647+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/4268855146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3330 0x55b9be5ba000 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.647+0000 7f3fc65b2740 1 -- 172.21.15.107:0/4268855146 >> 172.21.15.107:0/4268855146 conn(0x55b9be59f650 msgr2=0x55b9be5ad210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.647+0000 7f3fc65b2740 1 -- 172.21.15.107:0/4268855146 shutdown_connections 2024-05-09T08:38:51.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.647+0000 7f3fc65b2740 1 -- 172.21.15.107:0/4268855146 wait complete. 2024-05-09T08:38:51.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.648+0000 7f3fc65b2740 1 -- start start 2024-05-09T08:38:51.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.648+0000 7f3fc65b2740 1 Processor -- start 2024-05-09T08:38:51.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.648+0000 7f3fc65b2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3330 0x55b9be5a8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.648+0000 7f3fc65b2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3c50 0x55b9be5a86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.648+0000 7f3fc65b2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5ae320 0x55b9be5a8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.648+0000 7f3fc65b2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc1ad160 con 0x55b9be5a3330 2024-05-09T08:38:51.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.648+0000 7f3fc65b2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc1ae5d0 con 0x55b9be5a3c50 2024-05-09T08:38:51.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.648+0000 7f3fc65b2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc1ad830 con 0x55b9be5ae320 2024-05-09T08:38:51.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.648+0000 7f3fc65b2740 10 client.0 _ll_register_callbacks cb 0x7f3fac009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:51.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.648+0000 7f3fc1564640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3330 0x55b9be5a8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.648+0000 7f3fc1564640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3330 0x55b9be5a8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54414/0 (socket says 172.21.15.107:54414) 2024-05-09T08:38:51.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.648+0000 7f3fc1564640 1 -- 172.21.15.107:0/4240924421 learned_addr learned my addr 172.21.15.107:0/4240924421 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.648+0000 7f3fc1d65640 1 --2- 172.21.15.107:0/4240924421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5ae320 0x55b9be5a8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.648+0000 7f3fc0d63640 1 --2- 172.21.15.107:0/4240924421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3c50 0x55b9be5a86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.649+0000 7f3fc1564640 1 -- 172.21.15.107:0/4240924421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5ae320 msgr2=0x55b9be5a8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.649+0000 7f3fc1564640 1 --2- 172.21.15.107:0/4240924421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5ae320 0x55b9be5a8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.649+0000 7f3fc1564640 1 -- 172.21.15.107:0/4240924421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3c50 msgr2=0x55b9be5a86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.649+0000 7f3fc1564640 1 --2- 172.21.15.107:0/4240924421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3c50 0x55b9be5a86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.649+0000 7f3fc1564640 1 -- 172.21.15.107:0/4240924421 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9be623e30 con 0x55b9be5a3330 2024-05-09T08:38:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.649+0000 7f3fc1564640 1 --2- 172.21.15.107:0/4240924421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3330 0x55b9be5a8180 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8000c00 tx=0x7f3fb80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce9bc0576c88ffe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.649+0000 7f3fa4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.649+0000 7f3fa4ff9640 1 -- 172.21.15.107:0/4240924421 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8037060 con 0x55b9be5a3330 2024-05-09T08:38:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.649+0000 7f3fa4ff9640 1 -- 172.21.15.107:0/4240924421 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3fb8010ca0 con 0x55b9be5a3330 2024-05-09T08:38:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.649+0000 7f3fc65b2740 1 -- 172.21.15.107:0/4240924421 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fb80298a0 con 0x55b9be5a3330 2024-05-09T08:38:51.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.649+0000 7f3fa4ff9640 1 -- 172.21.15.107:0/4240924421 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3fb803e050 con 0x55b9be5a3330 2024-05-09T08:38:51.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.649+0000 7f3fc65b2740 10 client.17295 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:51.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.649+0000 7f3fc65b2740 1 -- 172.21.15.107:0/4240924421 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b9be625080 con 0x55b9be5a3330 2024-05-09T08:38:51.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.649+0000 7f3fa4ff9640 1 -- 172.21.15.107:0/4240924421 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fb80298a0 con 0x55b9be5a3330 2024-05-09T08:38:51.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.650+0000 7f3fa4ff9640 1 -- 172.21.15.107:0/4240924421 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3fb8003790 con 0x55b9be5a3330 2024-05-09T08:38:51.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.650+0000 7f3fc65b2740 10 client.17295 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:51.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.650+0000 7f3fc65b2740 10 client.17295 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:51.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.650+0000 7f3fc65b2740 20 client.17295 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:51.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.650+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/4240924421 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b9be6255c0 0x55b9be6279e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:51.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.650+0000 7f3fc65b2740 4 client.17295 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:38:51.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.650+0000 7f3fc65b2740 1 -- 172.21.15.107:0/4240924421 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b9be5106c0 con 0x55b9be6255c0 2024-05-09T08:38:51.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.650+0000 7f3fc0d63640 1 --2- 172.21.15.107:0/4240924421 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b9be6255c0 0x55b9be6279e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:51.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.650+0000 7f3fa4ff9640 10 client.17295 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:51.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.650+0000 7f3fc0d63640 1 --2- 172.21.15.107:0/4240924421 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b9be6255c0 0x55b9be6279e0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2ff22272c3c88a8 server_cookie=146975cc5dc40934 in_seq=0 out_seq=0 2024-05-09T08:38:51.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.651+0000 7f3fa4ff9640 1 -- 172.21.15.107:0/4240924421 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b9be5106c0 con 0x55b9be6255c0 2024-05-09T08:38:51.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.651+0000 7f3fa4ff9640 10 client.17295 handle_command_reply: tid=0 2024-05-09T08:38:51.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.651+0000 7f3fc65b2740 1 client.17295 shutdown 2024-05-09T08:38:51.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.651+0000 7f3fc65b2740 1 -- 172.21.15.107:0/4240924421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3330 msgr2=0x55b9be5a8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.651+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/4240924421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3330 0x55b9be5a8180 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8000c00 tx=0x7f3fb80107a0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.651+0000 7f3fc65b2740 1 -- 172.21.15.107:0/4240924421 shutdown_connections 2024-05-09T08:38:51.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.651+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/4240924421 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b9be6255c0 0x55b9be6279e0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.651+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/4240924421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b9be5ae320 0x55b9be5a8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.651+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/4240924421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b9be5a3c50 0x55b9be5a86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.651+0000 7f3fa4ff9640 0 client.17295 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:51.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.651+0000 7f3fc65b2740 1 --2- 172.21.15.107:0/4240924421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9be5a3330 0x55b9be5a8180 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.651+0000 7f3fc65b2740 1 -- 172.21.15.107:0/4240924421 >> 172.21.15.107:0/4240924421 conn(0x55b9be59f650 msgr2=0x55b9be5ab220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.651+0000 7f3fc65b2740 1 -- 172.21.15.107:0/4240924421 shutdown_connections 2024-05-09T08:38:51.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.651+0000 7f3fc65b2740 1 -- 172.21.15.107:0/4240924421 wait complete. 2024-05-09T08:38:51.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.651+0000 7f3fc65b2740 20 client.17295 trim_cache size 0 max 16384 2024-05-09T08:38:51.658 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:38:51.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.652+0000 7f3fc37c8640 1 -- 172.21.15.107:0/4095598256 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3fa80a2d80 msgr2=0x7f3fa80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.652+0000 7f3fc37c8640 1 --2- 172.21.15.107:0/4095598256 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3fa80a2d80 0x7f3fa80a5240 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4005fd0 tx=0x7f3fb4009500 comp rx=0 tx=0).stop 2024-05-09T08:38:51.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.652+0000 7f3fc37c8640 1 -- 172.21.15.107:0/4095598256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fbc1010d0 msgr2=0x7f3fbc1993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:51.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.652+0000 7f3fc37c8640 1 --2- 172.21.15.107:0/4095598256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fbc1010d0 0x7f3fbc1993c0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f3fac005940 tx=0x7f3fac0042f0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.652+0000 7f3fc37c8640 1 -- 172.21.15.107:0/4095598256 shutdown_connections 2024-05-09T08:38:51.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.652+0000 7f3fc37c8640 1 --2- 172.21.15.107:0/4095598256 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3fa80a2d80 0x7f3fa80a5240 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.652+0000 7f3fc37c8640 1 --2- 172.21.15.107:0/4095598256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fbc106720 0x7f3fbc1a3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.652+0000 7f3fc37c8640 1 --2- 172.21.15.107:0/4095598256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fbc1010d0 0x7f3fbc1993c0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.652+0000 7f3fc37c8640 1 --2- 172.21.15.107:0/4095598256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3fbc100710 0x7f3fbc198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:51.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.652+0000 7f3fc37c8640 1 -- 172.21.15.107:0/4095598256 >> 172.21.15.107:0/4095598256 conn(0x7f3fbc077790 msgr2=0x7f3fbc103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:51.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.652+0000 7f3fc37c8640 1 -- 172.21.15.107:0/4095598256 shutdown_connections 2024-05-09T08:38:51.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:51.652+0000 7f3fc37c8640 1 -- 172.21.15.107:0/4095598256 wait complete. 2024-05-09T08:38:51.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:51 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3623908991' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:38:52.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:52 smithi110 ceph-mon[33622]: pgmap v168: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:52.665 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:38:52.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:38:52.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:38:52.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:52 smithi107 ceph-mon[30122]: pgmap v168: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:52.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.780+0000 7ff5380ee640 1 -- 172.21.15.107:0/645766432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5301033d0 msgr2=0x7ff53010b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:52.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.780+0000 7ff5380ee640 1 --2- 172.21.15.107:0/645766432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5301033d0 0x7ff53010b8b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7ff520009900 tx=0x7ff52002a510 comp rx=0 tx=0).stop 2024-05-09T08:38:52.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.780+0000 7ff5380ee640 1 -- 172.21.15.107:0/645766432 shutdown_connections 2024-05-09T08:38:52.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.780+0000 7ff5380ee640 1 --2- 172.21.15.107:0/645766432 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff53010bdf0 0x7ff53010e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.780+0000 7ff5380ee640 1 --2- 172.21.15.107:0/645766432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5301033d0 0x7ff53010b8b0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.780+0000 7ff5380ee640 1 --2- 172.21.15.107:0/645766432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff530102ab0 0x7ff530102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.780+0000 7ff5380ee640 1 -- 172.21.15.107:0/645766432 >> 172.21.15.107:0/645766432 conn(0x7ff530077830 msgr2=0x7ff530077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:52.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.780+0000 7ff5380ee640 1 -- 172.21.15.107:0/645766432 shutdown_connections 2024-05-09T08:38:52.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.780+0000 7ff5380ee640 1 -- 172.21.15.107:0/645766432 wait complete. 2024-05-09T08:38:52.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.781+0000 7ff5380ee640 1 Processor -- start 2024-05-09T08:38:52.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.781+0000 7ff5380ee640 1 -- start start 2024-05-09T08:38:52.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.781+0000 7ff5380ee640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff530102ab0 0x7ff530100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.781+0000 7ff5380ee640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5301033d0 0x7ff530100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.781+0000 7ff5380ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53010bdf0 0x7ff530101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.781+0000 7ff5380ee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff530111aa0 con 0x7ff53010bdf0 2024-05-09T08:38:52.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.781+0000 7ff5380ee640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff530111920 con 0x7ff530102ab0 2024-05-09T08:38:52.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.781+0000 7ff5380ee640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff530111c20 con 0x7ff5301033d0 2024-05-09T08:38:52.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.781+0000 7ff53668b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53010bdf0 0x7ff530101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.781+0000 7ff53668b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53010bdf0 0x7ff530101180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54422/0 (socket says 172.21.15.107:54422) 2024-05-09T08:38:52.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.781+0000 7ff53668b640 1 -- 172.21.15.107:0/292208973 learned_addr learned my addr 172.21.15.107:0/292208973 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:52.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.781+0000 7ff535e8a640 1 --2- 172.21.15.107:0/292208973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff530102ab0 0x7ff530100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.781+0000 7ff535689640 1 --2- 172.21.15.107:0/292208973 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5301033d0 0x7ff530100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.782+0000 7ff53668b640 1 -- 172.21.15.107:0/292208973 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5301033d0 msgr2=0x7ff530100c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:52.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.782+0000 7ff53668b640 1 --2- 172.21.15.107:0/292208973 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5301033d0 0x7ff530100c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.782+0000 7ff53668b640 1 -- 172.21.15.107:0/292208973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff530102ab0 msgr2=0x7ff530100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:52.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.782+0000 7ff53668b640 1 --2- 172.21.15.107:0/292208973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff530102ab0 0x7ff530100700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.782+0000 7ff53668b640 1 -- 172.21.15.107:0/292208973 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5301a6550 con 0x7ff53010bdf0 2024-05-09T08:38:52.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.782+0000 7ff53668b640 1 --2- 172.21.15.107:0/292208973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53010bdf0 0x7ff530101180 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ff528009fd0 tx=0x7ff52800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:52.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.782+0000 7ff526ffd640 1 -- 172.21.15.107:0/292208973 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff528017070 con 0x7ff53010bdf0 2024-05-09T08:38:52.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.782+0000 7ff526ffd640 1 -- 172.21.15.107:0/292208973 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff528003e20 con 0x7ff53010bdf0 2024-05-09T08:38:52.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.782+0000 7ff5380ee640 1 -- 172.21.15.107:0/292208973 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5301a6840 con 0x7ff53010bdf0 2024-05-09T08:38:52.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.782+0000 7ff526ffd640 1 -- 172.21.15.107:0/292208973 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff528010460 con 0x7ff53010bdf0 2024-05-09T08:38:52.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.782+0000 7ff5380ee640 1 -- 172.21.15.107:0/292208973 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5301a6d10 con 0x7ff53010bdf0 2024-05-09T08:38:52.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.785+0000 7ff526ffd640 1 -- 172.21.15.107:0/292208973 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ff528003970 con 0x7ff53010bdf0 2024-05-09T08:38:52.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.785+0000 7ff53adfa740 1 Processor -- start 2024-05-09T08:38:52.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff53adfa740 1 -- start start 2024-05-09T08:38:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff53adfa740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee670b0 0x5596dee653f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff53adfa740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee659c0 0x5596dee65e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff53adfa740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee70740 0x5596dee72b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff53adfa740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5596dedca990 con 0x5596dee670b0 2024-05-09T08:38:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff53adfa740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5596ded725b0 con 0x5596dee70740 2024-05-09T08:38:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff53adfa740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5596dedc6950 con 0x5596dee659c0 2024-05-09T08:38:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff526ffd640 1 --2- 172.21.15.107:0/292208973 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff51c0a2d00 0x7ff51c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff535689640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee670b0 0x5596dee653f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff535e8a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee659c0 0x5596dee65e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff535689640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee670b0 0x5596dee653f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54428/0 (socket says 172.21.15.107:54428) 2024-05-09T08:38:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff535689640 1 -- 172.21.15.107:0/915717674 learned_addr learned my addr 172.21.15.107:0/915717674 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff53668b640 1 --2- 172.21.15.107:0/292208973 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff51c0a2d00 0x7ff51c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff535689640 1 --2- 172.21.15.107:0/915717674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee70740 0x5596dee72b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff526ffd640 1 -- 172.21.15.107:0/292208973 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff5280cf520 con 0x7ff53010bdf0 2024-05-09T08:38:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff535e8a640 1 -- 172.21.15.107:0/915717674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee70740 msgr2=0x5596dee72b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff535e8a640 1 --2- 172.21.15.107:0/915717674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee70740 0x5596dee72b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff535e8a640 1 -- 172.21.15.107:0/915717674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee670b0 msgr2=0x5596dee653f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff535e8a640 1 --2- 172.21.15.107:0/915717674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee670b0 0x5596dee653f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.786+0000 7ff535e8a640 1 -- 172.21.15.107:0/915717674 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596deba67e0 con 0x5596dee659c0 2024-05-09T08:38:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.787+0000 7ff53668b640 1 --2- 172.21.15.107:0/292208973 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff51c0a2d00 0x7ff51c0a51c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff52809b490 tx=0x7ff52806e800 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.787+0000 7ff535e8a640 1 --2- 172.21.15.107:0/915717674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee659c0 0x5596dee65e40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff52c00ab40 tx=0x7ff52c00f3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61c4aa5166025af1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:52.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.787+0000 7ff51bfff640 1 -- 172.21.15.107:0/915717674 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52c004020 con 0x5596dee659c0 2024-05-09T08:38:52.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.787+0000 7ff51bfff640 1 -- 172.21.15.107:0/915717674 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff52c0041c0 con 0x5596dee659c0 2024-05-09T08:38:52.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.787+0000 7ff51bfff640 1 -- 172.21.15.107:0/915717674 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52c0186a0 con 0x5596dee659c0 2024-05-09T08:38:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.788+0000 7ff53adfa740 1 -- 172.21.15.107:0/915717674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee659c0 msgr2=0x5596dee65e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.788+0000 7ff53adfa740 1 --2- 172.21.15.107:0/915717674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee659c0 0x5596dee65e40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff52c00ab40 tx=0x7ff52c00f3c0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.788+0000 7ff53adfa740 1 -- 172.21.15.107:0/915717674 shutdown_connections 2024-05-09T08:38:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.788+0000 7ff53adfa740 1 --2- 172.21.15.107:0/915717674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee70740 0x5596dee72b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.788+0000 7ff53adfa740 1 --2- 172.21.15.107:0/915717674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee659c0 0x5596dee65e40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.788+0000 7ff53adfa740 1 --2- 172.21.15.107:0/915717674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee670b0 0x5596dee653f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.788+0000 7ff53adfa740 1 -- 172.21.15.107:0/915717674 >> 172.21.15.107:0/915717674 conn(0x5596dee2e0a0 msgr2=0x5596dee61f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.788+0000 7ff53adfa740 1 -- 172.21.15.107:0/915717674 shutdown_connections 2024-05-09T08:38:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.788+0000 7ff53adfa740 1 -- 172.21.15.107:0/915717674 wait complete. 2024-05-09T08:38:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.788+0000 7ff53adfa740 1 -- start start 2024-05-09T08:38:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.788+0000 7ff53adfa740 1 Processor -- start 2024-05-09T08:38:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff53adfa740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee659c0 0x5596dee6d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff53adfa740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee670b0 0x5596dee6db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff53adfa740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee70740 0x5596dee6e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff53adfa740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5596dee735d0 con 0x5596dee70740 2024-05-09T08:38:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff53adfa740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5596ded85b40 con 0x5596dee659c0 2024-05-09T08:38:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff53adfa740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5596dee73750 con 0x5596dee670b0 2024-05-09T08:38:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff53adfa740 10 client.0 _ll_register_callbacks cb 0x5596ded7a0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff535689640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee70740 0x5596dee6e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff535689640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee70740 0x5596dee6e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54440/0 (socket says 172.21.15.107:54440) 2024-05-09T08:38:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff535689640 1 -- 172.21.15.107:0/2796041968 learned_addr learned my addr 172.21.15.107:0/2796041968 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff535e8a640 1 --2- 172.21.15.107:0/2796041968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee670b0 0x5596dee6db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff535689640 1 --2- 172.21.15.107:0/2796041968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee659c0 0x5596dee6d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff535689640 1 -- 172.21.15.107:0/2796041968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee670b0 msgr2=0x5596dee6db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff535689640 1 --2- 172.21.15.107:0/2796041968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee670b0 0x5596dee6db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff535689640 1 -- 172.21.15.107:0/2796041968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee659c0 msgr2=0x5596dee6d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff535689640 1 --2- 172.21.15.107:0/2796041968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee659c0 0x5596dee6d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff535689640 1 -- 172.21.15.107:0/2796041968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596dec06f70 con 0x5596dee70740 2024-05-09T08:38:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff535689640 1 --2- 172.21.15.107:0/2796041968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee70740 0x5596dee6e090 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ff520000c00 tx=0x7ff520005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbb67d54e4e62b6e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.789+0000 7ff5197fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.790+0000 7ff5197fa640 1 -- 172.21.15.107:0/2796041968 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff520035070 con 0x5596dee70740 2024-05-09T08:38:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.790+0000 7ff53adfa740 1 -- 172.21.15.107:0/2796041968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff52c0041c0 con 0x5596dee70740 2024-05-09T08:38:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.790+0000 7ff5197fa640 1 -- 172.21.15.107:0/2796041968 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff520005030 con 0x5596dee70740 2024-05-09T08:38:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.790+0000 7ff5197fa640 1 -- 172.21.15.107:0/2796041968 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff520055050 con 0x5596dee70740 2024-05-09T08:38:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.790+0000 7ff53adfa740 10 client.17307 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.790+0000 7ff53adfa740 1 -- 172.21.15.107:0/2796041968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5596dea9e170 con 0x5596dee70740 2024-05-09T08:38:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.790+0000 7ff5197fa640 1 -- 172.21.15.107:0/2796041968 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff52c0041c0 con 0x5596dee70740 2024-05-09T08:38:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.790+0000 7ff5197fa640 1 -- 172.21.15.107:0/2796041968 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff5200035f0 con 0x5596dee70740 2024-05-09T08:38:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.790+0000 7ff53adfa740 10 client.17307 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.790+0000 7ff53adfa740 10 client.17307 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.790+0000 7ff53adfa740 20 client.17307 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.790+0000 7ff53adfa740 1 --2- 172.21.15.107:0/2796041968 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5596dee80770 0x5596dee82b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.790+0000 7ff53adfa740 4 client.17307 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.790+0000 7ff53adfa740 1 -- 172.21.15.107:0/2796041968 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596dedd80c0 con 0x5596dee80770 2024-05-09T08:38:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.791+0000 7ff535e8a640 1 --2- 172.21.15.107:0/2796041968 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5596dee80770 0x5596dee82b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.791+0000 7ff5197fa640 10 client.17307 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.791+0000 7ff535e8a640 1 --2- 172.21.15.107:0/2796041968 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5596dee80770 0x5596dee82b60 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=972e823f801a8b79 server_cookie=cd81f96eea394a60 in_seq=0 out_seq=0 2024-05-09T08:38:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.792+0000 7ff5197fa640 1 -- 172.21.15.107:0/2796041968 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5596dedd80c0 con 0x5596dee80770 2024-05-09T08:38:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.792+0000 7ff5197fa640 10 client.17307 handle_command_reply: tid=0 2024-05-09T08:38:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.792+0000 7ff53adfa740 1 client.17307 shutdown 2024-05-09T08:38:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.792+0000 7ff53adfa740 1 -- 172.21.15.107:0/2796041968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee70740 msgr2=0x5596dee6e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.792+0000 7ff53adfa740 1 --2- 172.21.15.107:0/2796041968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee70740 0x5596dee6e090 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ff520000c00 tx=0x7ff520005b50 comp rx=0 tx=0).stop 2024-05-09T08:38:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.792+0000 7ff53adfa740 1 -- 172.21.15.107:0/2796041968 shutdown_connections 2024-05-09T08:38:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.792+0000 7ff53adfa740 1 --2- 172.21.15.107:0/2796041968 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5596dee80770 0x5596dee82b60 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.792+0000 7ff53adfa740 1 --2- 172.21.15.107:0/2796041968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee70740 0x5596dee6e090 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.792+0000 7ff53adfa740 1 --2- 172.21.15.107:0/2796041968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee670b0 0x5596dee6db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.792+0000 7ff5197fa640 0 client.17307 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.792+0000 7ff53adfa740 1 --2- 172.21.15.107:0/2796041968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee659c0 0x5596dee6d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.792+0000 7ff53adfa740 1 -- 172.21.15.107:0/2796041968 >> 172.21.15.107:0/2796041968 conn(0x5596dee2e0a0 msgr2=0x5596dee67d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.792+0000 7ff53adfa740 1 -- 172.21.15.107:0/2796041968 shutdown_connections 2024-05-09T08:38:52.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.792+0000 7ff53adfa740 1 -- 172.21.15.107:0/2796041968 wait complete. 2024-05-09T08:38:52.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.792+0000 7ff53adfa740 20 client.17307 trim_cache size 0 max 16384 2024-05-09T08:38:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.810+0000 7ff53adfa740 1 Processor -- start 2024-05-09T08:38:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.810+0000 7ff53adfa740 1 -- start start 2024-05-09T08:38:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.810+0000 7ff53adfa740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee659c0 0x5596dee7c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.810+0000 7ff53adfa740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee670b0 0x5596dee60f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.810+0000 7ff53adfa740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee70740 0x5596dee88410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.810+0000 7ff53adfa740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff5301b35d0 con 0x5596dee70740 2024-05-09T08:38:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.810+0000 7ff53adfa740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff5301b0ea0 con 0x5596dee659c0 2024-05-09T08:38:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.810+0000 7ff53adfa740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff5301aecf0 con 0x5596dee670b0 2024-05-09T08:38:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.810+0000 7ff535689640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee70740 0x5596dee88410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.811+0000 7ff535689640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee70740 0x5596dee88410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54444/0 (socket says 172.21.15.107:54444) 2024-05-09T08:38:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.811+0000 7ff535689640 1 -- 172.21.15.107:0/1792416173 learned_addr learned my addr 172.21.15.107:0/1792416173 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.811+0000 7ff535e8a640 1 --2- 172.21.15.107:0/1792416173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee670b0 0x5596dee60f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.811+0000 7ff535689640 1 --2- 172.21.15.107:0/1792416173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee659c0 0x5596dee7c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.811+0000 7ff535689640 1 -- 172.21.15.107:0/1792416173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee670b0 msgr2=0x5596dee60f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.811+0000 7ff535689640 1 --2- 172.21.15.107:0/1792416173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee670b0 0x5596dee60f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.811+0000 7ff535689640 1 -- 172.21.15.107:0/1792416173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee659c0 msgr2=0x5596dee7c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.811+0000 7ff535689640 1 --2- 172.21.15.107:0/1792416173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee659c0 0x5596dee7c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.811+0000 7ff535689640 1 -- 172.21.15.107:0/1792416173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596dee82ed0 con 0x5596dee70740 2024-05-09T08:38:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.811+0000 7ff535e8a640 1 --2- 172.21.15.107:0/1792416173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee670b0 0x5596dee60f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.811+0000 7ff535689640 1 --2- 172.21.15.107:0/1792416173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee70740 0x5596dee88410 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7ff520004010 tx=0x7ff52003f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60de4dccc213ce38 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.811+0000 7ff51b7fe640 1 -- 172.21.15.107:0/1792416173 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff520035070 con 0x5596dee70740 2024-05-09T08:38:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.811+0000 7ff51b7fe640 1 -- 172.21.15.107:0/1792416173 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff520032ca0 con 0x5596dee70740 2024-05-09T08:38:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.811+0000 7ff51b7fe640 1 -- 172.21.15.107:0/1792416173 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff520032020 con 0x5596dee70740 2024-05-09T08:38:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.812+0000 7ff53adfa740 1 -- 172.21.15.107:0/1792416173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee70740 msgr2=0x5596dee88410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.812+0000 7ff53adfa740 1 --2- 172.21.15.107:0/1792416173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee70740 0x5596dee88410 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7ff520004010 tx=0x7ff52003f040 comp rx=0 tx=0).stop 2024-05-09T08:38:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.812+0000 7ff53adfa740 1 -- 172.21.15.107:0/1792416173 shutdown_connections 2024-05-09T08:38:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.812+0000 7ff53adfa740 1 --2- 172.21.15.107:0/1792416173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee70740 0x5596dee88410 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.812+0000 7ff53adfa740 1 --2- 172.21.15.107:0/1792416173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee670b0 0x5596dee60f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.812+0000 7ff53adfa740 1 --2- 172.21.15.107:0/1792416173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee659c0 0x5596dee7c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.812+0000 7ff53adfa740 1 -- 172.21.15.107:0/1792416173 >> 172.21.15.107:0/1792416173 conn(0x5596dee2e0a0 msgr2=0x5596dee607f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.812+0000 7ff53adfa740 1 -- 172.21.15.107:0/1792416173 shutdown_connections 2024-05-09T08:38:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.812+0000 7ff53adfa740 1 -- 172.21.15.107:0/1792416173 wait complete. 2024-05-09T08:38:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.812+0000 7ff53adfa740 1 -- start start 2024-05-09T08:38:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.812+0000 7ff53adfa740 1 Processor -- start 2024-05-09T08:38:52.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff53adfa740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee659c0 0x5596dee6a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff53adfa740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee670b0 0x5596dee6aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff53adfa740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee70740 0x5596dee6b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff53adfa740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff5301af360 con 0x5596dee659c0 2024-05-09T08:38:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff53adfa740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff5301b07d0 con 0x5596dee670b0 2024-05-09T08:38:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff53adfa740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff5301afa30 con 0x5596dee70740 2024-05-09T08:38:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff53adfa740 10 client.0 _ll_register_callbacks cb 0x7ff520004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff535689640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee659c0 0x5596dee6a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff535689640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee659c0 0x5596dee6a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54456/0 (socket says 172.21.15.107:54456) 2024-05-09T08:38:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff535689640 1 -- 172.21.15.107:0/4126740241 learned_addr learned my addr 172.21.15.107:0/4126740241 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff535e8a640 1 --2- 172.21.15.107:0/4126740241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee670b0 0x5596dee6aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff535689640 1 --2- 172.21.15.107:0/4126740241 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee70740 0x5596dee6b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff535689640 1 -- 172.21.15.107:0/4126740241 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee70740 msgr2=0x5596dee6b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff535689640 1 --2- 172.21.15.107:0/4126740241 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee70740 0x5596dee6b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff535689640 1 -- 172.21.15.107:0/4126740241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee670b0 msgr2=0x5596dee6aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff535689640 1 --2- 172.21.15.107:0/4126740241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee670b0 0x5596dee6aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff535689640 1 -- 172.21.15.107:0/4126740241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596deee5f20 con 0x5596dee659c0 2024-05-09T08:38:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff535e8a640 1 --2- 172.21.15.107:0/4126740241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee670b0 0x5596dee6aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff535689640 1 --2- 172.21.15.107:0/4126740241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee659c0 0x5596dee6a580 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7ff520009900 tx=0x7ff52002a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7fee0b4bcacaf4f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff5197fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff5197fa640 1 -- 172.21.15.107:0/4126740241 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff520035040 con 0x5596dee659c0 2024-05-09T08:38:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff5197fa640 1 -- 172.21.15.107:0/4126740241 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff520031a70 con 0x5596dee659c0 2024-05-09T08:38:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.813+0000 7ff53adfa740 1 -- 172.21.15.107:0/4126740241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff520032ca0 con 0x5596dee659c0 2024-05-09T08:38:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.814+0000 7ff5197fa640 1 -- 172.21.15.107:0/4126740241 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff5200037f0 con 0x5596dee659c0 2024-05-09T08:38:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.814+0000 7ff53adfa740 10 client.17313 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.814+0000 7ff53adfa740 1 -- 172.21.15.107:0/4126740241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5596deee7240 con 0x5596dee659c0 2024-05-09T08:38:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.814+0000 7ff5197fa640 1 -- 172.21.15.107:0/4126740241 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff520032ca0 con 0x5596dee659c0 2024-05-09T08:38:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.814+0000 7ff5197fa640 1 -- 172.21.15.107:0/4126740241 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff52004c430 con 0x5596dee659c0 2024-05-09T08:38:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.814+0000 7ff53adfa740 10 client.17313 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.814+0000 7ff53adfa740 10 client.17313 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.814+0000 7ff53adfa740 20 client.17313 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:52.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.814+0000 7ff53adfa740 1 --2- 172.21.15.107:0/4126740241 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5596deee77f0 0x5596deee9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.814+0000 7ff53adfa740 4 client.17313 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:38:52.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.814+0000 7ff53adfa740 1 -- 172.21.15.107:0/4126740241 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5596dedd2410 con 0x5596deee77f0 2024-05-09T08:38:52.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.814+0000 7ff535e8a640 1 --2- 172.21.15.107:0/4126740241 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5596deee77f0 0x5596deee9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.815+0000 7ff5197fa640 10 client.17313 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:52.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.815+0000 7ff535e8a640 1 --2- 172.21.15.107:0/4126740241 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5596deee77f0 0x5596deee9be0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79ef1769181944ab server_cookie=33fac3c058a8cfc3 in_seq=0 out_seq=0 2024-05-09T08:38:52.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.816+0000 7ff5197fa640 1 -- 172.21.15.107:0/4126740241 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5596dedd2410 con 0x5596deee77f0 2024-05-09T08:38:52.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.816+0000 7ff5197fa640 10 client.17313 handle_command_reply: tid=0 2024-05-09T08:38:52.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.816+0000 7ff53adfa740 1 client.17313 shutdown 2024-05-09T08:38:52.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.816+0000 7ff53adfa740 1 -- 172.21.15.107:0/4126740241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee659c0 msgr2=0x5596dee6a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:52.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.816+0000 7ff53adfa740 1 --2- 172.21.15.107:0/4126740241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee659c0 0x5596dee6a580 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7ff520009900 tx=0x7ff52002a510 comp rx=0 tx=0).stop 2024-05-09T08:38:52.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.816+0000 7ff53adfa740 1 -- 172.21.15.107:0/4126740241 shutdown_connections 2024-05-09T08:38:52.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.816+0000 7ff53adfa740 1 --2- 172.21.15.107:0/4126740241 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5596deee77f0 0x5596deee9be0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.816+0000 7ff53adfa740 1 --2- 172.21.15.107:0/4126740241 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5596dee70740 0x5596dee6b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.816+0000 7ff53adfa740 1 --2- 172.21.15.107:0/4126740241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596dee670b0 0x5596dee6aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.816+0000 7ff5197fa640 0 client.17313 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:52.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.816+0000 7ff53adfa740 1 --2- 172.21.15.107:0/4126740241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5596dee659c0 0x5596dee6a580 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.816+0000 7ff53adfa740 1 -- 172.21.15.107:0/4126740241 >> 172.21.15.107:0/4126740241 conn(0x5596dee2e0a0 msgr2=0x5596dee61080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:52.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.816+0000 7ff53adfa740 1 -- 172.21.15.107:0/4126740241 shutdown_connections 2024-05-09T08:38:52.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.816+0000 7ff53adfa740 1 -- 172.21.15.107:0/4126740241 wait complete. 2024-05-09T08:38:52.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.816+0000 7ff53adfa740 20 client.17313 trim_cache size 0 max 16384 2024-05-09T08:38:52.822 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:38:52.822 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:38:52.823 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "9381b4cc-ae03-440b-98e5-8b16a1ee06bf", 2024-05-09T08:38:52.823 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:38:52.823 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:38:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.817+0000 7ff5380ee640 1 -- 172.21.15.107:0/292208973 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff51c0a2d00 msgr2=0x7ff51c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.817+0000 7ff5380ee640 1 --2- 172.21.15.107:0/292208973 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff51c0a2d00 0x7ff51c0a51c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff52809b490 tx=0x7ff52806e800 comp rx=0 tx=0).stop 2024-05-09T08:38:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.817+0000 7ff5380ee640 1 -- 172.21.15.107:0/292208973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53010bdf0 msgr2=0x7ff530101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.817+0000 7ff5380ee640 1 --2- 172.21.15.107:0/292208973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53010bdf0 0x7ff530101180 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ff528009fd0 tx=0x7ff52800b260 comp rx=0 tx=0).stop 2024-05-09T08:38:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.817+0000 7ff5380ee640 1 -- 172.21.15.107:0/292208973 shutdown_connections 2024-05-09T08:38:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.817+0000 7ff5380ee640 1 --2- 172.21.15.107:0/292208973 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff51c0a2d00 0x7ff51c0a51c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.817+0000 7ff5380ee640 1 --2- 172.21.15.107:0/292208973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53010bdf0 0x7ff530101180 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.817+0000 7ff5380ee640 1 --2- 172.21.15.107:0/292208973 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5301033d0 0x7ff530100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.817+0000 7ff5380ee640 1 --2- 172.21.15.107:0/292208973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff530102ab0 0x7ff530100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.817+0000 7ff5380ee640 1 -- 172.21.15.107:0/292208973 >> 172.21.15.107:0/292208973 conn(0x7ff530077830 msgr2=0x7ff5300fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.817+0000 7ff5380ee640 1 -- 172.21.15.107:0/292208973 shutdown_connections 2024-05-09T08:38:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.817+0000 7ff5380ee640 1 -- 172.21.15.107:0/292208973 wait complete. 2024-05-09T08:38:52.829 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:38:52.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:38:52.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:52 smithi157 ceph-mon[33614]: pgmap v168: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.995+0000 7f409ea15640 1 -- 172.21.15.107:0/3835867086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4098106710 msgr2=0x7f4098106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.995+0000 7f409ea15640 1 --2- 172.21.15.107:0/3835867086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4098106710 0x7f4098106af0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f4088009930 tx=0x7f408802a420 comp rx=0 tx=0).stop 2024-05-09T08:38:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.995+0000 7f409ea15640 1 -- 172.21.15.107:0/3835867086 shutdown_connections 2024-05-09T08:38:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.995+0000 7f409ea15640 1 --2- 172.21.15.107:0/3835867086 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40981010c0 0x7f409810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.995+0000 7f409ea15640 1 --2- 172.21.15.107:0/3835867086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4098100700 0x7f4098100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.995+0000 7f409ea15640 1 --2- 172.21.15.107:0/3835867086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4098106710 0x7f4098106af0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.995+0000 7f409ea15640 1 -- 172.21.15.107:0/3835867086 >> 172.21.15.107:0/3835867086 conn(0x7f4098077830 msgr2=0x7f4098077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.995+0000 7f409ea15640 1 -- 172.21.15.107:0/3835867086 shutdown_connections 2024-05-09T08:38:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.995+0000 7f409ea15640 1 -- 172.21.15.107:0/3835867086 wait complete. 2024-05-09T08:38:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.996+0000 7f409ea15640 1 Processor -- start 2024-05-09T08:38:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.996+0000 7f409ea15640 1 -- start start 2024-05-09T08:38:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.996+0000 7f409ea15640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4098100700 0x7f4098198eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.996+0000 7f409ea15640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40981010c0 0x7f40981993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.996+0000 7f409ea15640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4098106710 0x7f40981a3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.996+0000 7f409ea15640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4098110990 con 0x7f4098106710 2024-05-09T08:38:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.996+0000 7f409ea15640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4098110810 con 0x7f40981010c0 2024-05-09T08:38:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.996+0000 7f409ea15640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4098110b10 con 0x7f4098100700 2024-05-09T08:38:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.996+0000 7f409cfb2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4098106710 0x7f40981a3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.996+0000 7f4097fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4098100700 0x7f4098198eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.996+0000 7f409cfb2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4098106710 0x7f40981a3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54468/0 (socket says 172.21.15.107:54468) 2024-05-09T08:38:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.996+0000 7f4097fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4098100700 0x7f4098198eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:37882/0 (socket says 172.21.15.107:37882) 2024-05-09T08:38:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.996+0000 7f409cfb2640 1 -- 172.21.15.107:0/2892852388 learned_addr learned my addr 172.21.15.107:0/2892852388 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.996+0000 7f408ffff640 1 --2- 172.21.15.107:0/2892852388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40981010c0 0x7f40981993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.996+0000 7f409cfb2640 1 -- 172.21.15.107:0/2892852388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4098100700 msgr2=0x7f4098198eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.996+0000 7f409cfb2640 1 --2- 172.21.15.107:0/2892852388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4098100700 0x7f4098198eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.997+0000 7f409cfb2640 1 -- 172.21.15.107:0/2892852388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40981010c0 msgr2=0x7f40981993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.997+0000 7f409cfb2640 1 --2- 172.21.15.107:0/2892852388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40981010c0 0x7f40981993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.997+0000 7f409cfb2640 1 -- 172.21.15.107:0/2892852388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40981a4470 con 0x7f4098106710 2024-05-09T08:38:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.997+0000 7f4097fff640 1 --2- 172.21.15.107:0/2892852388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4098100700 0x7f4098198eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.997+0000 7f409cfb2640 1 --2- 172.21.15.107:0/2892852388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4098106710 0x7f40981a3f30 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f409000eb10 tx=0x7f409000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.997+0000 7f4095ffb640 1 -- 172.21.15.107:0/2892852388 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4090004b20 con 0x7f4098106710 2024-05-09T08:38:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.997+0000 7f409ea15640 1 -- 172.21.15.107:0/2892852388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40981a4640 con 0x7f4098106710 2024-05-09T08:38:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.997+0000 7f4095ffb640 1 -- 172.21.15.107:0/2892852388 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4090004cc0 con 0x7f4098106710 2024-05-09T08:38:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.997+0000 7f4095ffb640 1 -- 172.21.15.107:0/2892852388 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4090010690 con 0x7f4098106710 2024-05-09T08:38:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.997+0000 7f409ea15640 1 -- 172.21.15.107:0/2892852388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40981a4b60 con 0x7f4098106710 2024-05-09T08:38:53.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:52.999+0000 7f4095ffb640 1 -- 172.21.15.107:0/2892852388 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f4090005020 con 0x7f4098106710 2024-05-09T08:38:53.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.000+0000 7f40a1801740 1 Processor -- start 2024-05-09T08:38:53.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.000+0000 7f40a1801740 1 -- start start 2024-05-09T08:38:53.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.000+0000 7f40a1801740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125da280 0x555b125da660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:53.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.000+0000 7f40a1801740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125daba0 0x555b125e4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:53.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.000+0000 7f40a1801740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125e5270 0x555b125e7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:53.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.000+0000 7f40a1801740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555b1253fd00 con 0x555b125daba0 2024-05-09T08:38:53.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.000+0000 7f40a1801740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555b124e74c0 con 0x555b125e5270 2024-05-09T08:38:53.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.000+0000 7f40a1801740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x555b1253bcc0 con 0x555b125da280 2024-05-09T08:38:53.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.000+0000 7f4097fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125daba0 0x555b125e4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.000+0000 7f4095ffb640 1 --2- 172.21.15.107:0/2892852388 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f40840a2d00 0x7f40840a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.000+0000 7f4097fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125daba0 0x555b125e4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54476/0 (socket says 172.21.15.107:54476) 2024-05-09T08:38:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.000+0000 7f4097fff640 1 -- 172.21.15.107:0/2140376416 learned_addr learned my addr 172.21.15.107:0/2140376416 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.000+0000 7f408ffff640 1 --2- 172.21.15.107:0/2140376416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125da280 0x555b125da660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.000+0000 7f409cfb2640 1 --2- 172.21.15.107:0/2892852388 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f40840a2d00 0x7f40840a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.000+0000 7f408ffff640 1 --2- 172.21.15.107:0/2140376416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125e5270 0x555b125e7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.001+0000 7f4095ffb640 1 -- 172.21.15.107:0/2892852388 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f40900cfcc0 con 0x7f4098106710 2024-05-09T08:38:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.001+0000 7f4097fff640 1 -- 172.21.15.107:0/2140376416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125da280 msgr2=0x555b125da660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.001+0000 7f4097fff640 1 --2- 172.21.15.107:0/2140376416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125da280 0x555b125da660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.001+0000 7f4097fff640 1 -- 172.21.15.107:0/2140376416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125e5270 msgr2=0x555b125e7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.001+0000 7f4097fff640 1 --2- 172.21.15.107:0/2140376416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125e5270 0x555b125e7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.001+0000 7f4097fff640 1 -- 172.21.15.107:0/2140376416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b1231d7d0 con 0x555b125daba0 2024-05-09T08:38:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.001+0000 7f4097fff640 1 --2- 172.21.15.107:0/2140376416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125daba0 0x555b125e4d30 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f4088006010 tx=0x7f4088032eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9d76900f3260ca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.001+0000 7f409cfb2640 1 --2- 172.21.15.107:0/2892852388 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f40840a2d00 0x7f40840a51c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f40900982e0 tx=0x7f409000efd0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.001+0000 7f408e7fc640 1 -- 172.21.15.107:0/2140376416 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4088035040 con 0x555b125daba0 2024-05-09T08:38:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.001+0000 7f408e7fc640 1 -- 172.21.15.107:0/2140376416 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f40880317e0 con 0x555b125daba0 2024-05-09T08:38:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.001+0000 7f408e7fc640 1 -- 172.21.15.107:0/2140376416 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40880047a0 con 0x555b125daba0 2024-05-09T08:38:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.002+0000 7f40a1801740 1 -- 172.21.15.107:0/2140376416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125daba0 msgr2=0x555b125e4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.002+0000 7f40a1801740 1 --2- 172.21.15.107:0/2140376416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125daba0 0x555b125e4d30 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f4088006010 tx=0x7f4088032eb0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.002+0000 7f40a1801740 1 -- 172.21.15.107:0/2140376416 shutdown_connections 2024-05-09T08:38:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.002+0000 7f40a1801740 1 --2- 172.21.15.107:0/2140376416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125e5270 0x555b125e7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.002+0000 7f40a1801740 1 --2- 172.21.15.107:0/2140376416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125daba0 0x555b125e4d30 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.002+0000 7f40a1801740 1 --2- 172.21.15.107:0/2140376416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125da280 0x555b125da660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.002+0000 7f40a1801740 1 -- 172.21.15.107:0/2140376416 >> 172.21.15.107:0/2140376416 conn(0x555b125d65a0 msgr2=0x555b125d6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.002+0000 7f40a1801740 1 -- 172.21.15.107:0/2140376416 shutdown_connections 2024-05-09T08:38:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.002+0000 7f40a1801740 1 -- 172.21.15.107:0/2140376416 wait complete. 2024-05-09T08:38:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.003+0000 7f40a1801740 1 -- start start 2024-05-09T08:38:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.003+0000 7f40a1801740 1 Processor -- start 2024-05-09T08:38:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.003+0000 7f40a1801740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125da280 0x555b125e21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.003+0000 7f40a1801740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125daba0 0x555b125e26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.003+0000 7f40a1801740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125e5270 0x555b125e2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.003+0000 7f40a1801740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555b125e80a0 con 0x555b125da280 2024-05-09T08:38:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.003+0000 7f40a1801740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555b124fab90 con 0x555b125daba0 2024-05-09T08:38:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.003+0000 7f40a1801740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x555b125e8220 con 0x555b125e5270 2024-05-09T08:38:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.003+0000 7f40a1801740 10 client.0 _ll_register_callbacks cb 0x555b12536da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.003+0000 7f408ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125da280 0x555b125e21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.003+0000 7f408ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125da280 0x555b125e21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54486/0 (socket says 172.21.15.107:54486) 2024-05-09T08:38:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.003+0000 7f408ffff640 1 -- 172.21.15.107:0/2498628388 learned_addr learned my addr 172.21.15.107:0/2498628388 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.003+0000 7f408ffff640 1 --2- 172.21.15.107:0/2498628388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125e5270 0x555b125e2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.003+0000 7f4097fff640 1 --2- 172.21.15.107:0/2498628388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125daba0 0x555b125e26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.004+0000 7f408ffff640 1 -- 172.21.15.107:0/2498628388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125e5270 msgr2=0x555b125e2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.004+0000 7f408ffff640 1 --2- 172.21.15.107:0/2498628388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125e5270 0x555b125e2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.004+0000 7f408ffff640 1 -- 172.21.15.107:0/2498628388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125daba0 msgr2=0x555b125e26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.004+0000 7f408ffff640 1 --2- 172.21.15.107:0/2498628388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125daba0 0x555b125e26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.004+0000 7f408ffff640 1 -- 172.21.15.107:0/2498628388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b122533f0 con 0x555b125da280 2024-05-09T08:38:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.004+0000 7f408ffff640 1 --2- 172.21.15.107:0/2498628388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125da280 0x555b125e21a0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f408001cf10 tx=0x7f4080005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff6d87dae2487de server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.004+0000 7f40777fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.004+0000 7f40777fe640 1 -- 172.21.15.107:0/2498628388 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4080005020 con 0x555b125da280 2024-05-09T08:38:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.004+0000 7f40777fe640 1 -- 172.21.15.107:0/2498628388 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4080025040 con 0x555b125da280 2024-05-09T08:38:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.004+0000 7f40a1801740 1 -- 172.21.15.107:0/2498628388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40880317e0 con 0x555b125da280 2024-05-09T08:38:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.004+0000 7f40777fe640 1 -- 172.21.15.107:0/2498628388 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f40800051c0 con 0x555b125da280 2024-05-09T08:38:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.004+0000 7f40a1801740 10 client.17325 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.004+0000 7f40a1801740 1 -- 172.21.15.107:0/2498628388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555b122130f0 con 0x555b125da280 2024-05-09T08:38:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.004+0000 7f40777fe640 1 -- 172.21.15.107:0/2498628388 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f40880317e0 con 0x555b125da280 2024-05-09T08:38:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.004+0000 7f40777fe640 1 -- 172.21.15.107:0/2498628388 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f4080010430 con 0x555b125da280 2024-05-09T08:38:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.005+0000 7f40a1801740 10 client.17325 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.005+0000 7f40a1801740 10 client.17325 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.005+0000 7f40a1801740 20 client.17325 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.005+0000 7f40a1801740 1 --2- 172.21.15.107:0/2498628388 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555b125f5630 0x555b125f7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.005+0000 7f40a1801740 4 client.17325 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.005+0000 7f40a1801740 1 -- 172.21.15.107:0/2498628388 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b1254d430 con 0x555b125f5630 2024-05-09T08:38:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.005+0000 7f4097fff640 1 --2- 172.21.15.107:0/2498628388 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555b125f5630 0x555b125f7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.005+0000 7f4097fff640 1 --2- 172.21.15.107:0/2498628388 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555b125f5630 0x555b125f7a20 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=358605880d92eb83 server_cookie=31af1a8a773341cc in_seq=0 out_seq=0 2024-05-09T08:38:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.005+0000 7f40777fe640 10 client.17325 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.006+0000 7f40777fe640 1 -- 172.21.15.107:0/2498628388 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555b1254d430 con 0x555b125f5630 2024-05-09T08:38:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.006+0000 7f40777fe640 10 client.17325 handle_command_reply: tid=0 2024-05-09T08:38:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.006+0000 7f40a1801740 1 client.17325 shutdown 2024-05-09T08:38:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.006+0000 7f40a1801740 1 -- 172.21.15.107:0/2498628388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125da280 msgr2=0x555b125e21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.006+0000 7f40a1801740 1 --2- 172.21.15.107:0/2498628388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125da280 0x555b125e21a0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f408001cf10 tx=0x7f4080005f90 comp rx=0 tx=0).stop 2024-05-09T08:38:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.006+0000 7f40a1801740 1 -- 172.21.15.107:0/2498628388 shutdown_connections 2024-05-09T08:38:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.006+0000 7f40a1801740 1 --2- 172.21.15.107:0/2498628388 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555b125f5630 0x555b125f7a20 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.006+0000 7f40a1801740 1 --2- 172.21.15.107:0/2498628388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125e5270 0x555b125e2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.006+0000 7f40a1801740 1 --2- 172.21.15.107:0/2498628388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125daba0 0x555b125e26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.006+0000 7f40777fe640 0 client.17325 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.006+0000 7f40a1801740 1 --2- 172.21.15.107:0/2498628388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125da280 0x555b125e21a0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.006+0000 7f40a1801740 1 -- 172.21.15.107:0/2498628388 >> 172.21.15.107:0/2498628388 conn(0x555b125d65a0 msgr2=0x555b125dc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.006+0000 7f40a1801740 1 -- 172.21.15.107:0/2498628388 shutdown_connections 2024-05-09T08:38:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.006+0000 7f40a1801740 1 -- 172.21.15.107:0/2498628388 wait complete. 2024-05-09T08:38:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.006+0000 7f40a1801740 20 client.17325 trim_cache size 0 max 16384 2024-05-09T08:38:53.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.031+0000 7f40a1801740 1 Processor -- start 2024-05-09T08:38:53.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f40a1801740 1 -- start start 2024-05-09T08:38:53.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f40a1801740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125da280 0x555b125e10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:53.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f40a1801740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125daba0 0x555b125f0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:53.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f40a1801740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125e5270 0x555b125dd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:53.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f40a1801740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f40981af260 con 0x555b125e5270 2024-05-09T08:38:53.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f40a1801740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f40981acb30 con 0x555b125daba0 2024-05-09T08:38:53.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f40a1801740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f40981aa980 con 0x555b125da280 2024-05-09T08:38:53.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f408ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125e5270 0x555b125dd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:53.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f408ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125e5270 0x555b125dd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54502/0 (socket says 172.21.15.107:54502) 2024-05-09T08:38:53.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f408ffff640 1 -- 172.21.15.107:0/3921985168 learned_addr learned my addr 172.21.15.107:0/3921985168 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:53.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f4097fff640 1 --2- 172.21.15.107:0/3921985168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125daba0 0x555b125f0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:53.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f408ffff640 1 --2- 172.21.15.107:0/3921985168 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125da280 0x555b125e10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:53.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f408ffff640 1 -- 172.21.15.107:0/3921985168 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125da280 msgr2=0x555b125e10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:53.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f408ffff640 1 --2- 172.21.15.107:0/3921985168 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125da280 0x555b125e10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f408ffff640 1 -- 172.21.15.107:0/3921985168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125daba0 msgr2=0x555b125f0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:53.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f408ffff640 1 --2- 172.21.15.107:0/3921985168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125daba0 0x555b125f0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f408ffff640 1 -- 172.21.15.107:0/3921985168 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b125d9890 con 0x555b125e5270 2024-05-09T08:38:53.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f4097fff640 1 --2- 172.21.15.107:0/3921985168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125daba0 0x555b125f0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:38:53.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.032+0000 7f408ffff640 1 --2- 172.21.15.107:0/3921985168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125e5270 0x555b125dd4b0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f40800097c0 tx=0x7f4080012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb0980cdedab7297 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:53.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.033+0000 7f408dffb640 1 -- 172.21.15.107:0/3921985168 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4080010430 con 0x555b125e5270 2024-05-09T08:38:53.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.033+0000 7f408dffb640 1 -- 172.21.15.107:0/3921985168 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f40800105d0 con 0x555b125e5270 2024-05-09T08:38:53.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.033+0000 7f408dffb640 1 -- 172.21.15.107:0/3921985168 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4080004c70 con 0x555b125e5270 2024-05-09T08:38:53.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.033+0000 7f40a1801740 1 -- 172.21.15.107:0/3921985168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125e5270 msgr2=0x555b125dd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:53.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.033+0000 7f40a1801740 1 --2- 172.21.15.107:0/3921985168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125e5270 0x555b125dd4b0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f40800097c0 tx=0x7f4080012040 comp rx=0 tx=0).stop 2024-05-09T08:38:53.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.033+0000 7f40a1801740 1 -- 172.21.15.107:0/3921985168 shutdown_connections 2024-05-09T08:38:53.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.033+0000 7f40a1801740 1 --2- 172.21.15.107:0/3921985168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125e5270 0x555b125dd4b0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.033+0000 7f40a1801740 1 --2- 172.21.15.107:0/3921985168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125daba0 0x555b125f0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.033+0000 7f40a1801740 1 --2- 172.21.15.107:0/3921985168 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125da280 0x555b125e10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.033+0000 7f40a1801740 1 -- 172.21.15.107:0/3921985168 >> 172.21.15.107:0/3921985168 conn(0x555b125d65a0 msgr2=0x555b125e29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:53.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.034+0000 7f40a1801740 1 -- 172.21.15.107:0/3921985168 shutdown_connections 2024-05-09T08:38:53.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.034+0000 7f40a1801740 1 -- 172.21.15.107:0/3921985168 wait complete. 2024-05-09T08:38:53.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.034+0000 7f40a1801740 1 -- start start 2024-05-09T08:38:53.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.034+0000 7f40a1801740 1 Processor -- start 2024-05-09T08:38:53.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.034+0000 7f40a1801740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125da280 0x555b125ed800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:53.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.034+0000 7f40a1801740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125daba0 0x555b125e8b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:53.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.034+0000 7f40a1801740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125e5270 0x555b125e90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:53.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.034+0000 7f40a1801740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f40981aaff0 con 0x555b125da280 2024-05-09T08:38:53.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.034+0000 7f40a1801740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f40981ac460 con 0x555b125e5270 2024-05-09T08:38:53.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.034+0000 7f40a1801740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f40981ab6c0 con 0x555b125daba0 2024-05-09T08:38:53.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.034+0000 7f40a1801740 10 client.0 _ll_register_callbacks cb 0x7f408001caf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:53.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.034+0000 7f408ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125da280 0x555b125ed800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:53.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.034+0000 7f408ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125da280 0x555b125ed800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54508/0 (socket says 172.21.15.107:54508) 2024-05-09T08:38:53.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.034+0000 7f408ffff640 1 -- 172.21.15.107:0/3196229261 learned_addr learned my addr 172.21.15.107:0/3196229261 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:53.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.035+0000 7f4097fff640 1 --2- 172.21.15.107:0/3196229261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125daba0 0x555b125e8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:53.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.035+0000 7f408ffff640 1 --2- 172.21.15.107:0/3196229261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125e5270 0x555b125e90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:53.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.035+0000 7f408ffff640 1 -- 172.21.15.107:0/3196229261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125daba0 msgr2=0x555b125e8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:53.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.035+0000 7f408ffff640 1 --2- 172.21.15.107:0/3196229261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125daba0 0x555b125e8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.035+0000 7f408ffff640 1 -- 172.21.15.107:0/3196229261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125e5270 msgr2=0x555b125e90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:53.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.035+0000 7f408ffff640 1 --2- 172.21.15.107:0/3196229261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125e5270 0x555b125e90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.035+0000 7f408ffff640 1 -- 172.21.15.107:0/3196229261 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b125e98c0 con 0x555b125da280 2024-05-09T08:38:53.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.035+0000 7f408ffff640 1 --2- 172.21.15.107:0/3196229261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125da280 0x555b125ed800 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f40800108d0 tx=0x7f408001ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cff905ba8c13b2e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:53.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.035+0000 7f40777fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:53.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.035+0000 7f40777fe640 1 -- 172.21.15.107:0/3196229261 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4080010430 con 0x555b125da280 2024-05-09T08:38:53.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.035+0000 7f40a1801740 1 -- 172.21.15.107:0/3196229261 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40800105d0 con 0x555b125da280 2024-05-09T08:38:53.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.035+0000 7f40777fe640 1 -- 172.21.15.107:0/3196229261 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4080005360 con 0x555b125da280 2024-05-09T08:38:53.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.035+0000 7f40777fe640 1 -- 172.21.15.107:0/3196229261 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4080045050 con 0x555b125da280 2024-05-09T08:38:53.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.035+0000 7f40a1801740 10 client.17331 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:53.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.035+0000 7f40a1801740 1 -- 172.21.15.107:0/3196229261 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555b1265bb80 con 0x555b125da280 2024-05-09T08:38:53.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.035+0000 7f40777fe640 1 -- 172.21.15.107:0/3196229261 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f40800105d0 con 0x555b125da280 2024-05-09T08:38:53.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.036+0000 7f40777fe640 1 -- 172.21.15.107:0/3196229261 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f4080035470 con 0x555b125da280 2024-05-09T08:38:53.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.036+0000 7f40a1801740 10 client.17331 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:53.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.036+0000 7f40a1801740 10 client.17331 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:53.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.036+0000 7f40a1801740 20 client.17331 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:53.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.036+0000 7f40a1801740 1 --2- 172.21.15.107:0/3196229261 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555b1265c250 0x555b1265e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:53.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.036+0000 7f40a1801740 4 client.17331 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:38:53.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.036+0000 7f40a1801740 1 -- 172.21.15.107:0/3196229261 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555b12547780 con 0x555b1265c250 2024-05-09T08:38:53.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.036+0000 7f4097fff640 1 --2- 172.21.15.107:0/3196229261 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555b1265c250 0x555b1265e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:53.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.036+0000 7f4097fff640 1 --2- 172.21.15.107:0/3196229261 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555b1265c250 0x555b1265e640 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a22962490d4477a1 server_cookie=b28659925aaea029 in_seq=0 out_seq=0 2024-05-09T08:38:53.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.036+0000 7f40777fe640 10 client.17331 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:53.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.036+0000 7f40777fe640 1 -- 172.21.15.107:0/3196229261 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555b12547780 con 0x555b1265c250 2024-05-09T08:38:53.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.036+0000 7f40777fe640 10 client.17331 handle_command_reply: tid=0 2024-05-09T08:38:53.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.036+0000 7f40a1801740 1 client.17331 shutdown 2024-05-09T08:38:53.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.037+0000 7f40a1801740 1 -- 172.21.15.107:0/3196229261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125da280 msgr2=0x555b125ed800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:53.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.037+0000 7f40a1801740 1 --2- 172.21.15.107:0/3196229261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125da280 0x555b125ed800 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f40800108d0 tx=0x7f408001ca20 comp rx=0 tx=0).stop 2024-05-09T08:38:53.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.037+0000 7f40a1801740 1 -- 172.21.15.107:0/3196229261 shutdown_connections 2024-05-09T08:38:53.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.037+0000 7f40a1801740 1 --2- 172.21.15.107:0/3196229261 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555b1265c250 0x555b1265e640 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.037+0000 7f40a1801740 1 --2- 172.21.15.107:0/3196229261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555b125e5270 0x555b125e90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.037+0000 7f40a1801740 1 --2- 172.21.15.107:0/3196229261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b125daba0 0x555b125e8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.037+0000 7f40777fe640 0 client.17331 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:53.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.037+0000 7f40a1801740 1 --2- 172.21.15.107:0/3196229261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b125da280 0x555b125ed800 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.037+0000 7f40a1801740 1 -- 172.21.15.107:0/3196229261 >> 172.21.15.107:0/3196229261 conn(0x555b125d65a0 msgr2=0x555b125db9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:53.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.037+0000 7f40a1801740 1 -- 172.21.15.107:0/3196229261 shutdown_connections 2024-05-09T08:38:53.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.037+0000 7f40a1801740 1 -- 172.21.15.107:0/3196229261 wait complete. 2024-05-09T08:38:53.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.037+0000 7f40a1801740 20 client.17331 trim_cache size 0 max 16384 2024-05-09T08:38:53.044 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:38:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:38:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:38:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: "9381b4cc-ae03-440b-98e5-8b16a1ee06bf": { 2024-05-09T08:38:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:38:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "9381b4cc-ae03-440b-98e5-8b16a1ee06bf", 2024-05-09T08:38:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:38:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:38:53.044 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:38:53.044 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:38:53.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.037+0000 7f409ea15640 1 -- 172.21.15.107:0/2892852388 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f40840a2d00 msgr2=0x7f40840a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:53.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.037+0000 7f409ea15640 1 --2- 172.21.15.107:0/2892852388 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f40840a2d00 0x7f40840a51c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f40900982e0 tx=0x7f409000efd0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.037+0000 7f409ea15640 1 -- 172.21.15.107:0/2892852388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4098106710 msgr2=0x7f40981a3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:53.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.037+0000 7f409ea15640 1 --2- 172.21.15.107:0/2892852388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4098106710 0x7f40981a3f30 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f409000eb10 tx=0x7f409000c2d0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.038+0000 7f409ea15640 1 -- 172.21.15.107:0/2892852388 shutdown_connections 2024-05-09T08:38:53.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.038+0000 7f409ea15640 1 --2- 172.21.15.107:0/2892852388 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f40840a2d00 0x7f40840a51c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.038+0000 7f409ea15640 1 --2- 172.21.15.107:0/2892852388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4098106710 0x7f40981a3f30 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.038+0000 7f409ea15640 1 --2- 172.21.15.107:0/2892852388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40981010c0 0x7f40981993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.038+0000 7f409ea15640 1 --2- 172.21.15.107:0/2892852388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4098100700 0x7f4098198eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:53.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.038+0000 7f409ea15640 1 -- 172.21.15.107:0/2892852388 >> 172.21.15.107:0/2892852388 conn(0x7f4098077830 msgr2=0x7f4098103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:53.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.038+0000 7f409ea15640 1 -- 172.21.15.107:0/2892852388 shutdown_connections 2024-05-09T08:38:53.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:38:53.038+0000 7f409ea15640 1 -- 172.21.15.107:0/2892852388 wait complete. 2024-05-09T08:38:53.048 INFO:tasks.cephfs.filesystem:scrub status for tag:9381b4cc-ae03-440b-98e5-8b16a1ee06bf - {'path': '/', 'tag': '9381b4cc-ae03-440b-98e5-8b16a1ee06bf', 'options': 'recursive,force'} 2024-05-09T08:38:53.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:53 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:38:53.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:53 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:53.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:53 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:38:53.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:53 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:38:53.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:53 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:53.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:53 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:38:53.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:53 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:53.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:53 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:53.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:53 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:38:53.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:53 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:53.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:53 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:38:53.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:53 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:38:53.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:53 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:53.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:53 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:38:53.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:53 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:53.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:53 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:53.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:53 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:38:53.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:53 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:53.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:53 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:38:53.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:53 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:38:53.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:53 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:53.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:53 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:38:53.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:53 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:53.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:53 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:54.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:54 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:38:54.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:54 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:54.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:54 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:38:54.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:54 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:54.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:54 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:38:54.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:54 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:54.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:54 smithi107 ceph-mon[30122]: pgmap v169: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:38:54.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:54 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:38:54.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:54 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:54.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:54 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:38:54.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:54 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:54.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:54 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:38:54.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:54 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:54.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:54 smithi157 ceph-mon[33614]: pgmap v169: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:38:54.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:54 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:38:54.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:54 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:54.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:54 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:38:54.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:54 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:54.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:54 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:38:54.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:54 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:38:54.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:54 smithi110 ceph-mon[33622]: pgmap v169: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:38:56.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:56 smithi107 ceph-mon[30122]: pgmap v170: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:56.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:56 smithi157 ceph-mon[33614]: pgmap v170: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:56.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:56 smithi110 ceph-mon[33622]: pgmap v170: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:38:58.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:38:58 smithi107 ceph-mon[30122]: pgmap v171: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 37 MiB/s wr, 56 op/s 2024-05-09T08:38:58.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:38:58 smithi157 ceph-mon[33614]: pgmap v171: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 37 MiB/s wr, 56 op/s 2024-05-09T08:38:58.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:38:58 smithi110 ceph-mon[33622]: pgmap v171: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 37 MiB/s wr, 56 op/s 2024-05-09T08:38:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.267+0000 7fa5089f9640 1 -- 172.21.15.107:0/1166476372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5041033e0 msgr2=0x7fa50410b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.267+0000 7fa5089f9640 1 --2- 172.21.15.107:0/1166476372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5041033e0 0x7fa50410b8c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009900 tx=0x7fa4f802a510 comp rx=0 tx=0).stop 2024-05-09T08:38:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.267+0000 7fa5089f9640 1 -- 172.21.15.107:0/1166476372 shutdown_connections 2024-05-09T08:38:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.267+0000 7fa5089f9640 1 --2- 172.21.15.107:0/1166476372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa50410be00 0x7fa50410e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.267+0000 7fa5089f9640 1 --2- 172.21.15.107:0/1166476372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5041033e0 0x7fa50410b8c0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.267+0000 7fa5089f9640 1 --2- 172.21.15.107:0/1166476372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa504102ac0 0x7fa504102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.267+0000 7fa5089f9640 1 -- 172.21.15.107:0/1166476372 >> 172.21.15.107:0/1166476372 conn(0x7fa5040778b0 msgr2=0x7fa504077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.267+0000 7fa5089f9640 1 -- 172.21.15.107:0/1166476372 shutdown_connections 2024-05-09T08:38:59.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.267+0000 7fa5089f9640 1 -- 172.21.15.107:0/1166476372 wait complete. 2024-05-09T08:38:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.268+0000 7fa5089f9640 1 Processor -- start 2024-05-09T08:38:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.268+0000 7fa5089f9640 1 -- start start 2024-05-09T08:38:59.270 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.268+0000 7fa5089f9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa504102ac0 0x7fa504197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa5089f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5041033e0 0x7fa5041984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa5089f9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa50410be00 0x7fa50419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa5089f9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa504111ab0 con 0x7fa5041033e0 2024-05-09T08:38:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa5089f9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa504111930 con 0x7fa50410be00 2024-05-09T08:38:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa5089f9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa504111c30 con 0x7fa504102ac0 2024-05-09T08:38:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa501d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5041033e0 0x7fa5041984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa501d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5041033e0 0x7fa5041984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54520/0 (socket says 172.21.15.107:54520) 2024-05-09T08:38:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa501d9b640 1 -- 172.21.15.107:0/686075775 learned_addr learned my addr 172.21.15.107:0/686075775 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa502d9d640 1 --2- 172.21.15.107:0/686075775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa50410be00 0x7fa50419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa50259c640 1 --2- 172.21.15.107:0/686075775 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa504102ac0 0x7fa504197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa501d9b640 1 -- 172.21.15.107:0/686075775 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa504102ac0 msgr2=0x7fa504197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa501d9b640 1 --2- 172.21.15.107:0/686075775 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa504102ac0 0x7fa504197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa501d9b640 1 -- 172.21.15.107:0/686075775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa50410be00 msgr2=0x7fa50419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa501d9b640 1 --2- 172.21.15.107:0/686075775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa50410be00 0x7fa50419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa501d9b640 1 -- 172.21.15.107:0/686075775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50419cfa0 con 0x7fa5041033e0 2024-05-09T08:38:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa501d9b640 1 --2- 172.21.15.107:0/686075775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5041033e0 0x7fa5041984f0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8005940 tx=0x7fa4f80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa4f37fe640 1 -- 172.21.15.107:0/686075775 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f803f030 con 0x7fa5041033e0 2024-05-09T08:38:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa5089f9640 1 -- 172.21.15.107:0/686075775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa50419d230 con 0x7fa5041033e0 2024-05-09T08:38:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa4f37fe640 1 -- 172.21.15.107:0/686075775 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4f8003cb0 con 0x7fa5041033e0 2024-05-09T08:38:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa4f37fe640 1 -- 172.21.15.107:0/686075775 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8035070 con 0x7fa5041033e0 2024-05-09T08:38:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.269+0000 7fa5089f9640 1 -- 172.21.15.107:0/686075775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5041a4b20 con 0x7fa5041033e0 2024-05-09T08:38:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.272+0000 7fa4f37fe640 1 -- 172.21.15.107:0/686075775 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fa4f8042030 con 0x7fa5041033e0 2024-05-09T08:38:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.272+0000 7fa50b7e1740 1 Processor -- start 2024-05-09T08:38:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.272+0000 7fa50b7e1740 1 -- start start 2024-05-09T08:38:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.272+0000 7fa50b7e1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a916830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.272+0000 7fa50b7e1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916d70 0x55e14a9210f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.272+0000 7fa50b7e1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a921630 0x55e14a923a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.272+0000 7fa50b7e1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e14a87b340 con 0x55e14a916450 2024-05-09T08:38:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.272+0000 7fa50b7e1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e14a823290 con 0x55e14a916d70 2024-05-09T08:38:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.272+0000 7fa50b7e1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55e14a877300 con 0x55e14a921630 2024-05-09T08:38:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.273+0000 7fa502d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a916830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.273+0000 7fa502d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a916830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54522/0 (socket says 172.21.15.107:54522) 2024-05-09T08:38:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.273+0000 7fa502d9d640 1 -- 172.21.15.107:0/789223881 learned_addr learned my addr 172.21.15.107:0/789223881 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.273+0000 7fa501d9b640 1 --2- 172.21.15.107:0/789223881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a921630 0x55e14a923a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.273+0000 7fa50259c640 1 --2- 172.21.15.107:0/789223881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916d70 0x55e14a9210f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.273+0000 7fa502d9d640 1 -- 172.21.15.107:0/789223881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a921630 msgr2=0x55e14a923a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.273+0000 7fa502d9d640 1 --2- 172.21.15.107:0/789223881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a921630 0x55e14a923a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.273+0000 7fa502d9d640 1 -- 172.21.15.107:0/789223881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916d70 msgr2=0x55e14a9210f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.273+0000 7fa502d9d640 1 --2- 172.21.15.107:0/789223881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916d70 0x55e14a9210f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.273+0000 7fa502d9d640 1 -- 172.21.15.107:0/789223881 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e14a8478e0 con 0x55e14a916450 2024-05-09T08:38:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.273+0000 7fa50259c640 1 --2- 172.21.15.107:0/789223881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916d70 0x55e14a9210f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.273+0000 7fa4f37fe640 1 --2- 172.21.15.107:0/686075775 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa4ec0a2da0 0x7fa4ec0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.273+0000 7fa502d9d640 1 --2- 172.21.15.107:0/789223881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a916830 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009f50 tx=0x7fa4fc0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1cc43f6ebd85e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.273+0000 7fa50259c640 1 --2- 172.21.15.107:0/686075775 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa4ec0a2da0 0x7fa4ec0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.273+0000 7fa4f37fe640 1 -- 172.21.15.107:0/686075775 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fa4f80efec0 con 0x7fa5041033e0 2024-05-09T08:38:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.274+0000 7fa4f0ff9640 1 -- 172.21.15.107:0/789223881 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc004d20 con 0x55e14a916450 2024-05-09T08:38:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.274+0000 7fa4f0ff9640 1 -- 172.21.15.107:0/789223881 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4fc004050 con 0x55e14a916450 2024-05-09T08:38:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.274+0000 7fa4f0ff9640 1 -- 172.21.15.107:0/789223881 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc005280 con 0x55e14a916450 2024-05-09T08:38:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.274+0000 7fa50b7e1740 1 -- 172.21.15.107:0/789223881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 msgr2=0x55e14a916830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.274+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/789223881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a916830 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009f50 tx=0x7fa4fc0118d0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.274+0000 7fa50259c640 1 --2- 172.21.15.107:0/686075775 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa4ec0a2da0 0x7fa4ec0a5260 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x55e14a9179f0 tx=0x7fa4f400b390 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.274+0000 7fa50b7e1740 1 -- 172.21.15.107:0/789223881 shutdown_connections 2024-05-09T08:38:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.274+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/789223881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a921630 0x55e14a923a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.274+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/789223881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916d70 0x55e14a9210f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.274+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/789223881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a916830 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.274+0000 7fa50b7e1740 1 -- 172.21.15.107:0/789223881 >> 172.21.15.107:0/789223881 conn(0x55e14a912960 msgr2=0x55e14a912d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.275+0000 7fa50b7e1740 1 -- 172.21.15.107:0/789223881 shutdown_connections 2024-05-09T08:38:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.275+0000 7fa50b7e1740 1 -- 172.21.15.107:0/789223881 wait complete. 2024-05-09T08:38:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.275+0000 7fa50b7e1740 1 -- start start 2024-05-09T08:38:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.275+0000 7fa50b7e1740 1 Processor -- start 2024-05-09T08:38:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.275+0000 7fa50b7e1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a91e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.275+0000 7fa50b7e1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916d70 0x55e14a91ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.275+0000 7fa50b7e1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a921630 0x55e14a91efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.275+0000 7fa50b7e1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e14a924260 con 0x55e14a916450 2024-05-09T08:38:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.275+0000 7fa50b7e1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e14a836790 con 0x55e14a916d70 2024-05-09T08:38:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.275+0000 7fa50b7e1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55e14a9243e0 con 0x55e14a921630 2024-05-09T08:38:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.275+0000 7fa50b7e1740 10 client.0 _ll_register_callbacks cb 0x55e14a871a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.275+0000 7fa50259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a91e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.275+0000 7fa50259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a91e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54532/0 (socket says 172.21.15.107:54532) 2024-05-09T08:38:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.275+0000 7fa50259c640 1 -- 172.21.15.107:0/185911235 learned_addr learned my addr 172.21.15.107:0/185911235 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.275+0000 7fa502d9d640 1 --2- 172.21.15.107:0/185911235 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a921630 0x55e14a91efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.276+0000 7fa501d9b640 1 --2- 172.21.15.107:0/185911235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916d70 0x55e14a91ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.276+0000 7fa50259c640 1 -- 172.21.15.107:0/185911235 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a921630 msgr2=0x55e14a91efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:59.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.276+0000 7fa50259c640 1 --2- 172.21.15.107:0/185911235 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a921630 0x55e14a91efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.276+0000 7fa50259c640 1 -- 172.21.15.107:0/185911235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916d70 msgr2=0x55e14a91ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:59.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.276+0000 7fa50259c640 1 --2- 172.21.15.107:0/185911235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916d70 0x55e14a91ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.276+0000 7fa50259c640 1 -- 172.21.15.107:0/185911235 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e14a58f210 con 0x55e14a916450 2024-05-09T08:38:59.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.276+0000 7fa501d9b640 1 --2- 172.21.15.107:0/185911235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916d70 0x55e14a91ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:38:59.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.276+0000 7fa50259c640 1 --2- 172.21.15.107:0/185911235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a91e530 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400b5a0 tx=0x7fa4f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be8d43dd5e92c72d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:59.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.276+0000 7fa4e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:59.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.276+0000 7fa4e9ffb640 1 -- 172.21.15.107:0/185911235 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f401b660 con 0x55e14a916450 2024-05-09T08:38:59.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.276+0000 7fa50b7e1740 1 -- 172.21.15.107:0/185911235 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4fc004050 con 0x55e14a916450 2024-05-09T08:38:59.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.276+0000 7fa4e9ffb640 1 -- 172.21.15.107:0/185911235 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fa4f401b800 con 0x55e14a916450 2024-05-09T08:38:59.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.276+0000 7fa4e9ffb640 1 -- 172.21.15.107:0/185911235 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa4f401ba00 con 0x55e14a916450 2024-05-09T08:38:59.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.276+0000 7fa50b7e1740 10 client.17343 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:59.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.276+0000 7fa50b7e1740 1 -- 172.21.15.107:0/185911235 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e14a65b620 con 0x55e14a916450 2024-05-09T08:38:59.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.276+0000 7fa4e9ffb640 1 -- 172.21.15.107:0/185911235 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4fc004050 con 0x55e14a916450 2024-05-09T08:38:59.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.277+0000 7fa4e9ffb640 1 -- 172.21.15.107:0/185911235 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa4f4024070 con 0x55e14a916450 2024-05-09T08:38:59.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.277+0000 7fa50b7e1740 10 client.17343 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:59.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.277+0000 7fa50b7e1740 10 client.17343 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:59.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.277+0000 7fa50b7e1740 20 client.17343 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:59.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.277+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/185911235 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55e14a9313a0 0x55e14a9337d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.277+0000 7fa50b7e1740 4 client.17343 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:38:59.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.277+0000 7fa50b7e1740 1 -- 172.21.15.107:0/185911235 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e14a888a70 con 0x55e14a9313a0 2024-05-09T08:38:59.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.277+0000 7fa501d9b640 1 --2- 172.21.15.107:0/185911235 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55e14a9313a0 0x55e14a9337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.277+0000 7fa4e9ffb640 10 client.17343 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:59.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.277+0000 7fa501d9b640 1 --2- 172.21.15.107:0/185911235 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55e14a9313a0 0x55e14a9337d0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9c61d8db46752dc server_cookie=998d7ee6cad94cd3 in_seq=0 out_seq=0 2024-05-09T08:38:59.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.278+0000 7fa4e9ffb640 1 -- 172.21.15.107:0/185911235 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e14a888a70 con 0x55e14a9313a0 2024-05-09T08:38:59.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.278+0000 7fa4e9ffb640 10 client.17343 handle_command_reply: tid=0 2024-05-09T08:38:59.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.278+0000 7fa50b7e1740 1 client.17343 shutdown 2024-05-09T08:38:59.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.278+0000 7fa50b7e1740 1 -- 172.21.15.107:0/185911235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 msgr2=0x55e14a91e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:59.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.278+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/185911235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a91e530 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400b5a0 tx=0x7fa4f40046c0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.278+0000 7fa50b7e1740 1 -- 172.21.15.107:0/185911235 shutdown_connections 2024-05-09T08:38:59.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.278+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/185911235 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55e14a9313a0 0x55e14a9337d0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.278+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/185911235 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a921630 0x55e14a91efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.278+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/185911235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916d70 0x55e14a91ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.278+0000 7fa4e9ffb640 0 client.17343 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:59.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.278+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/185911235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a91e530 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.278+0000 7fa50b7e1740 1 -- 172.21.15.107:0/185911235 >> 172.21.15.107:0/185911235 conn(0x55e14a912960 msgr2=0x55e14a9186e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:59.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.278+0000 7fa50b7e1740 1 -- 172.21.15.107:0/185911235 shutdown_connections 2024-05-09T08:38:59.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.278+0000 7fa50b7e1740 1 -- 172.21.15.107:0/185911235 wait complete. 2024-05-09T08:38:59.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.279+0000 7fa50b7e1740 20 client.17343 trim_cache size 0 max 16384 2024-05-09T08:38:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.295+0000 7fa50b7e1740 1 Processor -- start 2024-05-09T08:38:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.295+0000 7fa50b7e1740 1 -- start start 2024-05-09T08:38:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa50b7e1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a92d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa50b7e1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a916d70 0x55e14a918e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa50b7e1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a921630 0x55e14a91d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa50b7e1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa5041b1380 con 0x55e14a916450 2024-05-09T08:38:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa50b7e1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa5041aec50 con 0x55e14a921630 2024-05-09T08:38:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa50b7e1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa5041acaa0 con 0x55e14a916d70 2024-05-09T08:38:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa50259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a92d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa50259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a92d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54546/0 (socket says 172.21.15.107:54546) 2024-05-09T08:38:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa50259c640 1 -- 172.21.15.107:0/1812100238 learned_addr learned my addr 172.21.15.107:0/1812100238 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa501d9b640 1 --2- 172.21.15.107:0/1812100238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a916d70 0x55e14a918e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa502d9d640 1 --2- 172.21.15.107:0/1812100238 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a921630 0x55e14a91d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa50259c640 1 -- 172.21.15.107:0/1812100238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a916d70 msgr2=0x55e14a918e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa50259c640 1 --2- 172.21.15.107:0/1812100238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a916d70 0x55e14a918e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa50259c640 1 -- 172.21.15.107:0/1812100238 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a921630 msgr2=0x55e14a91d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa50259c640 1 --2- 172.21.15.107:0/1812100238 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a921630 0x55e14a91d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa50259c640 1 -- 172.21.15.107:0/1812100238 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e14a92ff40 con 0x55e14a916450 2024-05-09T08:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa50259c640 1 --2- 172.21.15.107:0/1812100238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a92d120 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400bfd0 tx=0x7fa4f4003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152d2a563bb8d7e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa4ebfff640 1 -- 172.21.15.107:0/1812100238 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f4039070 con 0x55e14a916450 2024-05-09T08:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa4ebfff640 1 -- 172.21.15.107:0/1812100238 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4f4026440 con 0x55e14a916450 2024-05-09T08:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.296+0000 7fa4ebfff640 1 -- 172.21.15.107:0/1812100238 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f401be60 con 0x55e14a916450 2024-05-09T08:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.297+0000 7fa50b7e1740 1 -- 172.21.15.107:0/1812100238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 msgr2=0x55e14a92d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.297+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/1812100238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a92d120 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400bfd0 tx=0x7fa4f4003ea0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.297+0000 7fa50b7e1740 1 -- 172.21.15.107:0/1812100238 shutdown_connections 2024-05-09T08:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.297+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/1812100238 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a921630 0x55e14a91d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.297+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/1812100238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a916d70 0x55e14a918e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.297+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/1812100238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916450 0x55e14a92d120 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.297+0000 7fa50b7e1740 1 -- 172.21.15.107:0/1812100238 >> 172.21.15.107:0/1812100238 conn(0x55e14a912960 msgr2=0x55e14a91fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.297+0000 7fa50b7e1740 1 -- 172.21.15.107:0/1812100238 shutdown_connections 2024-05-09T08:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.297+0000 7fa50b7e1740 1 -- 172.21.15.107:0/1812100238 wait complete. 2024-05-09T08:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.297+0000 7fa50b7e1740 1 -- start start 2024-05-09T08:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa50b7e1740 1 Processor -- start 2024-05-09T08:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa50b7e1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916450 0x55e14a91b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa50b7e1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916d70 0x55e14a91b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa50b7e1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a921630 0x55e14a91bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa50b7e1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa5041ad110 con 0x55e14a916d70 2024-05-09T08:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa50b7e1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa5041ae580 con 0x55e14a916450 2024-05-09T08:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa50b7e1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa5041ad7e0 con 0x55e14a921630 2024-05-09T08:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa50b7e1740 10 client.0 _ll_register_callbacks cb 0x7fa4f40043e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa501d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916d70 0x55e14a91b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa501d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916d70 0x55e14a91b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54558/0 (socket says 172.21.15.107:54558) 2024-05-09T08:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa501d9b640 1 -- 172.21.15.107:0/3543464981 learned_addr learned my addr 172.21.15.107:0/3543464981 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa50259c640 1 --2- 172.21.15.107:0/3543464981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916450 0x55e14a91b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa502d9d640 1 --2- 172.21.15.107:0/3543464981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a921630 0x55e14a91bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa501d9b640 1 -- 172.21.15.107:0/3543464981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a921630 msgr2=0x55e14a91bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa501d9b640 1 --2- 172.21.15.107:0/3543464981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a921630 0x55e14a91bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa501d9b640 1 -- 172.21.15.107:0/3543464981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916450 msgr2=0x55e14a91b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa501d9b640 1 --2- 172.21.15.107:0/3543464981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916450 0x55e14a91b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa501d9b640 1 -- 172.21.15.107:0/3543464981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e14a996950 con 0x55e14a916d70 2024-05-09T08:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa501d9b640 1 --2- 172.21.15.107:0/3543464981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916d70 0x55e14a91b6d0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fa4f80bbe30 tx=0x7fa4f80f23d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b930012f189857e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.298+0000 7fa4e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.299+0000 7fa4e9ffb640 1 -- 172.21.15.107:0/3543464981 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8049040 con 0x55e14a916d70 2024-05-09T08:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.299+0000 7fa4e9ffb640 1 -- 172.21.15.107:0/3543464981 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fa4f80ba430 con 0x55e14a916d70 2024-05-09T08:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.299+0000 7fa50b7e1740 1 -- 172.21.15.107:0/3543464981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4f4026440 con 0x55e14a916d70 2024-05-09T08:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.299+0000 7fa4e9ffb640 1 -- 172.21.15.107:0/3543464981 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa4f80bbc70 con 0x55e14a916d70 2024-05-09T08:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.299+0000 7fa50b7e1740 10 client.17349 fetch_fsmap learned FSMap version 27 2024-05-09T08:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.299+0000 7fa50b7e1740 1 -- 172.21.15.107:0/3543464981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e14a997c70 con 0x55e14a916d70 2024-05-09T08:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.299+0000 7fa4e9ffb640 1 -- 172.21.15.107:0/3543464981 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4f4026440 con 0x55e14a916d70 2024-05-09T08:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.299+0000 7fa4e9ffb640 1 -- 172.21.15.107:0/3543464981 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa4f80ed530 con 0x55e14a916d70 2024-05-09T08:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.299+0000 7fa50b7e1740 10 client.17349 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.299+0000 7fa50b7e1740 10 client.17349 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.299+0000 7fa50b7e1740 20 client.17349 populate_metadata read hostname 'smithi107' 2024-05-09T08:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.299+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/3543464981 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55e14a9983f0 0x55e14a99a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.299+0000 7fa50b7e1740 4 client.17349 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.299+0000 7fa50b7e1740 1 -- 172.21.15.107:0/3543464981 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e14a882dc0 con 0x55e14a9983f0 2024-05-09T08:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.299+0000 7fa50259c640 1 --2- 172.21.15.107:0/3543464981 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55e14a9983f0 0x55e14a99a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:38:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.300+0000 7fa4e9ffb640 10 client.17349 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.300+0000 7fa50259c640 1 --2- 172.21.15.107:0/3543464981 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55e14a9983f0 0x55e14a99a810 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8042d8a74835369 server_cookie=d7de19968d4ed6eb in_seq=0 out_seq=0 2024-05-09T08:38:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.316+0000 7fa4e9ffb640 1 -- 172.21.15.107:0/3543464981 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e14a882dc0 con 0x55e14a9983f0 2024-05-09T08:38:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.316+0000 7fa4e9ffb640 10 client.17349 handle_command_reply: tid=0 2024-05-09T08:38:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.316+0000 7fa50b7e1740 1 client.17349 shutdown 2024-05-09T08:38:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.316+0000 7fa50b7e1740 1 -- 172.21.15.107:0/3543464981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916d70 msgr2=0x55e14a91b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.316+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/3543464981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916d70 0x55e14a91b6d0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fa4f80bbe30 tx=0x7fa4f80f23d0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.316+0000 7fa50b7e1740 1 -- 172.21.15.107:0/3543464981 shutdown_connections 2024-05-09T08:38:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.316+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/3543464981 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55e14a9983f0 0x55e14a99a810 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.316+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/3543464981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e14a921630 0x55e14a91bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.316+0000 7fa4e9ffb640 0 client.17349 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:38:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.316+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/3543464981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e14a916d70 0x55e14a91b6d0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.316+0000 7fa50b7e1740 1 --2- 172.21.15.107:0/3543464981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e14a916450 0x55e14a91b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.316+0000 7fa50b7e1740 1 -- 172.21.15.107:0/3543464981 >> 172.21.15.107:0/3543464981 conn(0x55e14a912960 msgr2=0x55e14a911170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.316+0000 7fa50b7e1740 1 -- 172.21.15.107:0/3543464981 shutdown_connections 2024-05-09T08:38:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.316+0000 7fa50b7e1740 1 -- 172.21.15.107:0/3543464981 wait complete. 2024-05-09T08:38:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.316+0000 7fa50b7e1740 20 client.17349 trim_cache size 0 max 16384 2024-05-09T08:38:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:38:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:38:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:38:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:38:59.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.317+0000 7fa5089f9640 1 -- 172.21.15.107:0/686075775 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa4ec0a2da0 msgr2=0x7fa4ec0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:59.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.317+0000 7fa5089f9640 1 --2- 172.21.15.107:0/686075775 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa4ec0a2da0 0x7fa4ec0a5260 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x55e14a9179f0 tx=0x7fa4f400b390 comp rx=0 tx=0).stop 2024-05-09T08:38:59.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.317+0000 7fa5089f9640 1 -- 172.21.15.107:0/686075775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5041033e0 msgr2=0x7fa5041984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:38:59.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.317+0000 7fa5089f9640 1 --2- 172.21.15.107:0/686075775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5041033e0 0x7fa5041984f0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8005940 tx=0x7fa4f80042f0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.317+0000 7fa5089f9640 1 -- 172.21.15.107:0/686075775 shutdown_connections 2024-05-09T08:38:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.317+0000 7fa5089f9640 1 --2- 172.21.15.107:0/686075775 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa4ec0a2da0 0x7fa4ec0a5260 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.317+0000 7fa5089f9640 1 --2- 172.21.15.107:0/686075775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa50410be00 0x7fa50419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.317+0000 7fa5089f9640 1 --2- 172.21.15.107:0/686075775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5041033e0 0x7fa5041984f0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.317+0000 7fa5089f9640 1 --2- 172.21.15.107:0/686075775 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa504102ac0 0x7fa504197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:38:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.317+0000 7fa5089f9640 1 -- 172.21.15.107:0/686075775 >> 172.21.15.107:0/686075775 conn(0x7fa5040778b0 msgr2=0x7fa5040fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:38:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.317+0000 7fa5089f9640 1 -- 172.21.15.107:0/686075775 shutdown_connections 2024-05-09T08:38:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:38:59.317+0000 7fa5089f9640 1 -- 172.21.15.107:0/686075775 wait complete. 2024-05-09T08:39:00.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:00 smithi107 ceph-mon[30122]: pgmap v172: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 25 MiB/s wr, 44 op/s 2024-05-09T08:39:00.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:00 smithi157 ceph-mon[33614]: pgmap v172: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 25 MiB/s wr, 44 op/s 2024-05-09T08:39:00.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:00 smithi110 ceph-mon[33622]: pgmap v172: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 25 MiB/s wr, 44 op/s 2024-05-09T08:39:01.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:01 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:39:01.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:01 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:39:01.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:01 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:39:01.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:01 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:39:01.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:01 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:39:01.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:01 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:39:02.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:02 smithi107 ceph-mon[30122]: pgmap v173: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 25 MiB/s wr, 44 op/s 2024-05-09T08:39:02.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:02 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:39:02.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:02 smithi157 ceph-mon[33614]: pgmap v173: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 25 MiB/s wr, 44 op/s 2024-05-09T08:39:02.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:02 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:39:02.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:02 smithi110 ceph-mon[33622]: pgmap v173: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 25 MiB/s wr, 44 op/s 2024-05-09T08:39:02.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:02 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:39:04.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:04 smithi107 ceph-mon[30122]: pgmap v174: 129 pgs: 129 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 37 MiB/s wr, 71 op/s 2024-05-09T08:39:04.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:04 smithi157 ceph-mon[33614]: pgmap v174: 129 pgs: 129 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 37 MiB/s wr, 71 op/s 2024-05-09T08:39:04.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:04 smithi110 ceph-mon[33622]: pgmap v174: 129 pgs: 129 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 37 MiB/s wr, 71 op/s 2024-05-09T08:39:06.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:06 smithi107 ceph-mon[30122]: pgmap v175: 129 pgs: 129 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 24 MiB/s wr, 64 op/s 2024-05-09T08:39:06.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:06 smithi157 ceph-mon[33614]: pgmap v175: 129 pgs: 129 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 24 MiB/s wr, 64 op/s 2024-05-09T08:39:06.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:06 smithi110 ceph-mon[33622]: pgmap v175: 129 pgs: 129 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 24 MiB/s wr, 64 op/s 2024-05-09T08:39:08.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:08 smithi107 ceph-mon[30122]: pgmap v176: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 37 MiB/s wr, 71 op/s 2024-05-09T08:39:08.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:08 smithi157 ceph-mon[33614]: pgmap v176: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 37 MiB/s wr, 71 op/s 2024-05-09T08:39:08.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:08 smithi110 ceph-mon[33622]: pgmap v176: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 37 MiB/s wr, 71 op/s 2024-05-09T08:39:10.672 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:10 smithi110 ceph-mon[33622]: pgmap v177: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 24 MiB/s wr, 34 op/s 2024-05-09T08:39:10.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:10 smithi107 ceph-mon[30122]: pgmap v177: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 24 MiB/s wr, 34 op/s 2024-05-09T08:39:10.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:10 smithi157 ceph-mon[33614]: pgmap v177: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 24 MiB/s wr, 34 op/s 2024-05-09T08:39:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.415+0000 7fe519ba5640 1 -- 172.21.15.107:0/1531132654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5140ffd30 msgr2=0x7fe5141001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.415+0000 7fe519ba5640 1 --2- 172.21.15.107:0/1531132654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5140ffd30 0x7fe5141001b0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fe508009930 tx=0x7fe50802a420 comp rx=0 tx=0).stop 2024-05-09T08:39:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.415+0000 7fe519ba5640 1 -- 172.21.15.107:0/1531132654 shutdown_connections 2024-05-09T08:39:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.415+0000 7fe519ba5640 1 --2- 172.21.15.107:0/1531132654 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5141006f0 0x7fe51410e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.415+0000 7fe519ba5640 1 --2- 172.21.15.107:0/1531132654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5140ffd30 0x7fe5141001b0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.415+0000 7fe519ba5640 1 --2- 172.21.15.107:0/1531132654 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe514106810 0x7fe514106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.415+0000 7fe519ba5640 1 -- 172.21.15.107:0/1531132654 >> 172.21.15.107:0/1531132654 conn(0x7fe514076cc0 msgr2=0x7fe5140fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.415+0000 7fe519ba5640 1 -- 172.21.15.107:0/1531132654 shutdown_connections 2024-05-09T08:39:12.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.415+0000 7fe519ba5640 1 -- 172.21.15.107:0/1531132654 wait complete. 2024-05-09T08:39:12.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.416+0000 7fe519ba5640 1 Processor -- start 2024-05-09T08:39:12.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.416+0000 7fe519ba5640 1 -- start start 2024-05-09T08:39:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.416+0000 7fe519ba5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe5140ffd30 0x7fe514198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.416+0000 7fe519ba5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5141006f0 0x7fe514198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.416+0000 7fe519ba5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe514106810 0x7fe51419c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.417+0000 7fe519ba5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe514111bf0 con 0x7fe5141006f0 2024-05-09T08:39:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.417+0000 7fe519ba5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe514111a70 con 0x7fe514106810 2024-05-09T08:39:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.417+0000 7fe519ba5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe514111d70 con 0x7fe5140ffd30 2024-05-09T08:39:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.417+0000 7fe512ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5141006f0 0x7fe514198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.417+0000 7fe512ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5141006f0 0x7fe514198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38252/0 (socket says 172.21.15.107:38252) 2024-05-09T08:39:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.417+0000 7fe512ffd640 1 -- 172.21.15.107:0/1654704494 learned_addr learned my addr 172.21.15.107:0/1654704494 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.417+0000 7fe513fff640 1 --2- 172.21.15.107:0/1654704494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe514106810 0x7fe51419c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.417+0000 7fe512ffd640 1 -- 172.21.15.107:0/1654704494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe5140ffd30 msgr2=0x7fe514198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.417+0000 7fe512ffd640 1 --2- 172.21.15.107:0/1654704494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe5140ffd30 0x7fe514198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.417+0000 7fe512ffd640 1 -- 172.21.15.107:0/1654704494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe514106810 msgr2=0x7fe51419c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.417+0000 7fe512ffd640 1 --2- 172.21.15.107:0/1654704494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe514106810 0x7fe51419c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.417+0000 7fe512ffd640 1 -- 172.21.15.107:0/1654704494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51419d0f0 con 0x7fe5141006f0 2024-05-09T08:39:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.418+0000 7fe512ffd640 1 --2- 172.21.15.107:0/1654704494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5141006f0 0x7fe514198690 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fe508004770 tx=0x7fe5080042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.418+0000 7fe510ff9640 1 -- 172.21.15.107:0/1654704494 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50803f070 con 0x7fe5141006f0 2024-05-09T08:39:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.418+0000 7fe519ba5640 1 -- 172.21.15.107:0/1654704494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51419d380 con 0x7fe5141006f0 2024-05-09T08:39:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.418+0000 7fe510ff9640 1 -- 172.21.15.107:0/1654704494 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe50803b460 con 0x7fe5141006f0 2024-05-09T08:39:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.418+0000 7fe519ba5640 1 -- 172.21.15.107:0/1654704494 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5141a4c70 con 0x7fe5141006f0 2024-05-09T08:39:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.418+0000 7fe510ff9640 1 -- 172.21.15.107:0/1654704494 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508035070 con 0x7fe5141006f0 2024-05-09T08:39:12.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.420+0000 7fe510ff9640 1 -- 172.21.15.107:0/1654704494 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe50803b610 con 0x7fe5141006f0 2024-05-09T08:39:12.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.421+0000 7fe51c98f740 1 Processor -- start 2024-05-09T08:39:12.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.421+0000 7fe51c98f740 1 -- start start 2024-05-09T08:39:12.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.421+0000 7fe51c98f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece008f450 0x55ece008f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.421+0000 7fe51c98f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008fd70 0x55ece009a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.421+0000 7fe51c98f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece009a630 0x55ece009ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.421+0000 7fe51c98f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ecdfff4340 con 0x55ece008fd70 2024-05-09T08:39:12.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.421+0000 7fe51c98f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ecdff9c290 con 0x55ece009a630 2024-05-09T08:39:12.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.421+0000 7fe51c98f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ecdfff0300 con 0x55ece008f450 2024-05-09T08:39:12.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.421+0000 7fe510ff9640 1 --2- 172.21.15.107:0/1654704494 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4fc0a2e20 0x7fe4fc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.421+0000 7fe5137fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008fd70 0x55ece009a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.421+0000 7fe5137fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008fd70 0x55ece009a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38254/0 (socket says 172.21.15.107:38254) 2024-05-09T08:39:12.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.421+0000 7fe512ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece009a630 0x55ece009ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.421+0000 7fe5137fe640 1 -- 172.21.15.107:0/2864062230 learned_addr learned my addr 172.21.15.107:0/2864062230 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:12.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.421+0000 7fe513fff640 1 --2- 172.21.15.107:0/2864062230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece008f450 0x55ece008f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.421+0000 7fe513fff640 1 --2- 172.21.15.107:0/1654704494 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4fc0a2e20 0x7fe4fc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.422+0000 7fe5137fe640 1 -- 172.21.15.107:0/2864062230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece008f450 msgr2=0x55ece008f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.422+0000 7fe5137fe640 1 --2- 172.21.15.107:0/2864062230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece008f450 0x55ece008f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.422+0000 7fe5137fe640 1 -- 172.21.15.107:0/2864062230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece009a630 msgr2=0x55ece009ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.422+0000 7fe5137fe640 1 --2- 172.21.15.107:0/2864062230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece009a630 0x55ece009ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.422+0000 7fe5137fe640 1 -- 172.21.15.107:0/2864062230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecdffc08e0 con 0x55ece008fd70 2024-05-09T08:39:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.422+0000 7fe510ff9640 1 -- 172.21.15.107:0/1654704494 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe5080efac0 con 0x7fe5141006f0 2024-05-09T08:39:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.422+0000 7fe5137fe640 1 --2- 172.21.15.107:0/2864062230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008fd70 0x55ece009a0f0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe50400bb10 tx=0x7fe50400d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d971c88c14a75bc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.422+0000 7fe513fff640 1 --2- 172.21.15.107:0/1654704494 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4fc0a2e20 0x7fe4fc0a52e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe50c009900 tx=0x7fe50c009650 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.422+0000 7fe4f9ffb640 1 -- 172.21.15.107:0/2864062230 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe504016070 con 0x55ece008fd70 2024-05-09T08:39:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.422+0000 7fe4f9ffb640 1 -- 172.21.15.107:0/2864062230 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe50400dce0 con 0x55ece008fd70 2024-05-09T08:39:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.422+0000 7fe4f9ffb640 1 -- 172.21.15.107:0/2864062230 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe504010460 con 0x55ece008fd70 2024-05-09T08:39:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.423+0000 7fe51c98f740 1 -- 172.21.15.107:0/2864062230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008fd70 msgr2=0x55ece009a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.423+0000 7fe51c98f740 1 --2- 172.21.15.107:0/2864062230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008fd70 0x55ece009a0f0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe50400bb10 tx=0x7fe50400d2e0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.423+0000 7fe51c98f740 1 -- 172.21.15.107:0/2864062230 shutdown_connections 2024-05-09T08:39:12.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.423+0000 7fe51c98f740 1 --2- 172.21.15.107:0/2864062230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece009a630 0x55ece009ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.423+0000 7fe51c98f740 1 --2- 172.21.15.107:0/2864062230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008fd70 0x55ece009a0f0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.423+0000 7fe51c98f740 1 --2- 172.21.15.107:0/2864062230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece008f450 0x55ece008f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.423+0000 7fe51c98f740 1 -- 172.21.15.107:0/2864062230 >> 172.21.15.107:0/2864062230 conn(0x55ece008b960 msgr2=0x55ece008bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.423+0000 7fe51c98f740 1 -- 172.21.15.107:0/2864062230 shutdown_connections 2024-05-09T08:39:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.423+0000 7fe51c98f740 1 -- 172.21.15.107:0/2864062230 wait complete. 2024-05-09T08:39:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.423+0000 7fe51c98f740 1 -- start start 2024-05-09T08:39:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe51c98f740 1 Processor -- start 2024-05-09T08:39:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe51c98f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece008f450 0x55ece0097560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe51c98f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008fd70 0x55ece0097aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe51c98f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece009a630 0x55ece0097fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe51c98f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ece009d260 con 0x55ece009a630 2024-05-09T08:39:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe51c98f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ecdffaf790 con 0x55ece008fd70 2024-05-09T08:39:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe51c98f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ece009d3e0 con 0x55ece008f450 2024-05-09T08:39:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe51c98f740 10 client.0 _ll_register_callbacks cb 0x55ecdffeaa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe513fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece009a630 0x55ece0097fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe512ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008fd70 0x55ece0097aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe513fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece009a630 0x55ece0097fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38264/0 (socket says 172.21.15.107:38264) 2024-05-09T08:39:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe513fff640 1 -- 172.21.15.107:0/87902490 learned_addr learned my addr 172.21.15.107:0/87902490 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe5137fe640 1 --2- 172.21.15.107:0/87902490 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece008f450 0x55ece0097560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe513fff640 1 -- 172.21.15.107:0/87902490 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece008f450 msgr2=0x55ece0097560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe513fff640 1 --2- 172.21.15.107:0/87902490 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece008f450 0x55ece0097560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe513fff640 1 -- 172.21.15.107:0/87902490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008fd70 msgr2=0x55ece0097aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe513fff640 1 --2- 172.21.15.107:0/87902490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008fd70 0x55ece0097aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe513fff640 1 -- 172.21.15.107:0/87902490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecdfd08210 con 0x55ece009a630 2024-05-09T08:39:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.424+0000 7fe512ffd640 1 --2- 172.21.15.107:0/87902490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008fd70 0x55ece0097aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.425+0000 7fe513fff640 1 --2- 172.21.15.107:0/87902490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece009a630 0x55ece0097fe0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fe50c008840 tx=0x7fe50c004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d346f5b6625bb36 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.425+0000 7fe4f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.425+0000 7fe4f2ffd640 1 -- 172.21.15.107:0/87902490 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50c010040 con 0x55ece009a630 2024-05-09T08:39:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.425+0000 7fe4f2ffd640 1 -- 172.21.15.107:0/87902490 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe50c007830 con 0x55ece009a630 2024-05-09T08:39:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.425+0000 7fe51c98f740 1 -- 172.21.15.107:0/87902490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe50400dce0 con 0x55ece009a630 2024-05-09T08:39:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.425+0000 7fe4f2ffd640 1 -- 172.21.15.107:0/87902490 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe50c030050 con 0x55ece009a630 2024-05-09T08:39:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.425+0000 7fe51c98f740 10 client.17361 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.425+0000 7fe51c98f740 1 -- 172.21.15.107:0/87902490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ecdfdd4620 con 0x55ece009a630 2024-05-09T08:39:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.425+0000 7fe4f2ffd640 1 -- 172.21.15.107:0/87902490 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe50400dce0 con 0x55ece009a630 2024-05-09T08:39:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.425+0000 7fe4f2ffd640 1 -- 172.21.15.107:0/87902490 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe50c0227b0 con 0x55ece009a630 2024-05-09T08:39:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.425+0000 7fe51c98f740 10 client.17361 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.425+0000 7fe51c98f740 10 client.17361 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.425+0000 7fe51c98f740 20 client.17361 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.425+0000 7fe51c98f740 1 --2- 172.21.15.107:0/87902490 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ece00aa370 0x55ece00ac780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.425+0000 7fe51c98f740 4 client.17361 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.425+0000 7fe51c98f740 1 -- 172.21.15.107:0/87902490 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ece0001a70 con 0x55ece00aa370 2024-05-09T08:39:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.426+0000 7fe5137fe640 1 --2- 172.21.15.107:0/87902490 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ece00aa370 0x55ece00ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.426+0000 7fe4f2ffd640 10 client.17361 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.426+0000 7fe5137fe640 1 --2- 172.21.15.107:0/87902490 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ece00aa370 0x55ece00ac780 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54edb102ae4b1195 server_cookie=35bd050cc239d72e in_seq=0 out_seq=0 2024-05-09T08:39:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.427+0000 7fe4f2ffd640 1 -- 172.21.15.107:0/87902490 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ece0001a70 con 0x55ece00aa370 2024-05-09T08:39:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.427+0000 7fe4f2ffd640 10 client.17361 handle_command_reply: tid=0 2024-05-09T08:39:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.427+0000 7fe51c98f740 1 client.17361 shutdown 2024-05-09T08:39:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.427+0000 7fe51c98f740 1 -- 172.21.15.107:0/87902490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece009a630 msgr2=0x55ece0097fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.427+0000 7fe51c98f740 1 --2- 172.21.15.107:0/87902490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece009a630 0x55ece0097fe0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fe50c008840 tx=0x7fe50c004b50 comp rx=0 tx=0).stop 2024-05-09T08:39:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.427+0000 7fe51c98f740 1 -- 172.21.15.107:0/87902490 shutdown_connections 2024-05-09T08:39:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.427+0000 7fe51c98f740 1 --2- 172.21.15.107:0/87902490 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ece00aa370 0x55ece00ac780 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.427+0000 7fe51c98f740 1 --2- 172.21.15.107:0/87902490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece009a630 0x55ece0097fe0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.427+0000 7fe4f2ffd640 0 client.17361 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.427+0000 7fe51c98f740 1 --2- 172.21.15.107:0/87902490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008fd70 0x55ece0097aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.427+0000 7fe51c98f740 1 --2- 172.21.15.107:0/87902490 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece008f450 0x55ece0097560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.427+0000 7fe51c98f740 1 -- 172.21.15.107:0/87902490 >> 172.21.15.107:0/87902490 conn(0x55ece008b960 msgr2=0x55ece0091690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.427+0000 7fe51c98f740 1 -- 172.21.15.107:0/87902490 shutdown_connections 2024-05-09T08:39:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.427+0000 7fe51c98f740 1 -- 172.21.15.107:0/87902490 wait complete. 2024-05-09T08:39:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.427+0000 7fe51c98f740 20 client.17361 trim_cache size 0 max 16384 2024-05-09T08:39:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.442+0000 7fe51c98f740 1 Processor -- start 2024-05-09T08:39:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.442+0000 7fe51c98f740 1 -- start start 2024-05-09T08:39:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.442+0000 7fe51c98f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008f450 0x55ece00a6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.442+0000 7fe51c98f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008fd70 0x55ece008b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.442+0000 7fe51c98f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece009a630 0x55ece0091510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.442+0000 7fe51c98f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe5141af170 con 0x55ece008fd70 2024-05-09T08:39:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.442+0000 7fe51c98f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe5141aca40 con 0x55ece008f450 2024-05-09T08:39:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.442+0000 7fe51c98f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe5141aa890 con 0x55ece009a630 2024-05-09T08:39:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.442+0000 7fe512ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008fd70 0x55ece008b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.442+0000 7fe512ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008fd70 0x55ece008b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38266/0 (socket says 172.21.15.107:38266) 2024-05-09T08:39:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.442+0000 7fe512ffd640 1 -- 172.21.15.107:0/3698272795 learned_addr learned my addr 172.21.15.107:0/3698272795 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.442+0000 7fe5137fe640 1 --2- 172.21.15.107:0/3698272795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008f450 0x55ece00a6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.442+0000 7fe513fff640 1 --2- 172.21.15.107:0/3698272795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece009a630 0x55ece0091510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.443+0000 7fe512ffd640 1 -- 172.21.15.107:0/3698272795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece009a630 msgr2=0x55ece0091510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.443+0000 7fe512ffd640 1 --2- 172.21.15.107:0/3698272795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece009a630 0x55ece0091510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.443+0000 7fe512ffd640 1 -- 172.21.15.107:0/3698272795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008f450 msgr2=0x55ece00a6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.443+0000 7fe512ffd640 1 --2- 172.21.15.107:0/3698272795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008f450 0x55ece00a6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.443+0000 7fe512ffd640 1 -- 172.21.15.107:0/3698272795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ece0089f00 con 0x55ece008fd70 2024-05-09T08:39:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.443+0000 7fe512ffd640 1 --2- 172.21.15.107:0/3698272795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008fd70 0x55ece008b180 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fe5080ecf60 tx=0x7fe5080bbb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b0617d92e11547 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.443+0000 7fe4f97fa640 1 -- 172.21.15.107:0/3698272795 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508048040 con 0x55ece008fd70 2024-05-09T08:39:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.443+0000 7fe4f97fa640 1 -- 172.21.15.107:0/3698272795 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5080c48f0 con 0x55ece008fd70 2024-05-09T08:39:12.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.443+0000 7fe4f97fa640 1 -- 172.21.15.107:0/3698272795 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5080b8440 con 0x55ece008fd70 2024-05-09T08:39:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.444+0000 7fe51c98f740 1 -- 172.21.15.107:0/3698272795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008fd70 msgr2=0x55ece008b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.444+0000 7fe51c98f740 1 --2- 172.21.15.107:0/3698272795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008fd70 0x55ece008b180 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fe5080ecf60 tx=0x7fe5080bbb80 comp rx=0 tx=0).stop 2024-05-09T08:39:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.444+0000 7fe51c98f740 1 -- 172.21.15.107:0/3698272795 shutdown_connections 2024-05-09T08:39:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.444+0000 7fe51c98f740 1 --2- 172.21.15.107:0/3698272795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece009a630 0x55ece0091510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.444+0000 7fe51c98f740 1 --2- 172.21.15.107:0/3698272795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008fd70 0x55ece008b180 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.444+0000 7fe51c98f740 1 --2- 172.21.15.107:0/3698272795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008f450 0x55ece00a6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.444+0000 7fe51c98f740 1 -- 172.21.15.107:0/3698272795 >> 172.21.15.107:0/3698272795 conn(0x55ece008b960 msgr2=0x55ece0090e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.444+0000 7fe51c98f740 1 -- 172.21.15.107:0/3698272795 shutdown_connections 2024-05-09T08:39:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.444+0000 7fe51c98f740 1 -- 172.21.15.107:0/3698272795 wait complete. 2024-05-09T08:39:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.444+0000 7fe51c98f740 1 -- start start 2024-05-09T08:39:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.444+0000 7fe51c98f740 1 Processor -- start 2024-05-09T08:39:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.444+0000 7fe51c98f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008f450 0x55ece00a2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.444+0000 7fe51c98f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008fd70 0x55ece0093eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe51c98f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece009a630 0x55ece00943f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe51c98f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe5141aaf00 con 0x55ece008f450 2024-05-09T08:39:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe51c98f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe5141ac370 con 0x55ece008fd70 2024-05-09T08:39:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe51c98f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe5141ab5d0 con 0x55ece009a630 2024-05-09T08:39:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe51c98f740 10 client.0 _ll_register_callbacks cb 0x7fe50c008700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe5137fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008f450 0x55ece00a2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe5137fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008f450 0x55ece00a2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38270/0 (socket says 172.21.15.107:38270) 2024-05-09T08:39:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe5137fe640 1 -- 172.21.15.107:0/423533318 learned_addr learned my addr 172.21.15.107:0/423533318 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe512ffd640 1 --2- 172.21.15.107:0/423533318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008fd70 0x55ece0093eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe513fff640 1 --2- 172.21.15.107:0/423533318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece009a630 0x55ece00943f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe5137fe640 1 -- 172.21.15.107:0/423533318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece009a630 msgr2=0x55ece00943f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe5137fe640 1 --2- 172.21.15.107:0/423533318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece009a630 0x55ece00943f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe5137fe640 1 -- 172.21.15.107:0/423533318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008fd70 msgr2=0x55ece0093eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe5137fe640 1 --2- 172.21.15.107:0/423533318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008fd70 0x55ece0093eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe5137fe640 1 -- 172.21.15.107:0/423533318 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ece0094a90 con 0x55ece008f450 2024-05-09T08:39:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe512ffd640 1 --2- 172.21.15.107:0/423533318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008fd70 0x55ece0093eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe5137fe640 1 --2- 172.21.15.107:0/423533318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008f450 0x55ece00a2890 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fe504006010 tx=0x7fe50400b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d254cd3e31b0d9b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe4f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe4f2ffd640 1 -- 172.21.15.107:0/423533318 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe504016040 con 0x55ece008f450 2024-05-09T08:39:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe4f2ffd640 1 -- 172.21.15.107:0/423533318 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe504004040 con 0x55ece008f450 2024-05-09T08:39:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe51c98f740 1 -- 172.21.15.107:0/423533318 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5080c48f0 con 0x55ece008f450 2024-05-09T08:39:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.445+0000 7fe4f2ffd640 1 -- 172.21.15.107:0/423533318 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe504035050 con 0x55ece008f450 2024-05-09T08:39:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.446+0000 7fe51c98f740 10 client.17367 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.446+0000 7fe51c98f740 1 -- 172.21.15.107:0/423533318 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ece01107a0 con 0x55ece008f450 2024-05-09T08:39:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.446+0000 7fe4f2ffd640 1 -- 172.21.15.107:0/423533318 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5080c48f0 con 0x55ece008f450 2024-05-09T08:39:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.446+0000 7fe4f2ffd640 1 -- 172.21.15.107:0/423533318 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe5040035f0 con 0x55ece008f450 2024-05-09T08:39:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.446+0000 7fe51c98f740 10 client.17367 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.446+0000 7fe51c98f740 10 client.17367 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.446+0000 7fe51c98f740 20 client.17367 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.446+0000 7fe51c98f740 1 --2- 172.21.15.107:0/423533318 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ece00aa370 0x55ece0112c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.446+0000 7fe51c98f740 4 client.17367 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:39:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.446+0000 7fe51c98f740 1 -- 172.21.15.107:0/423533318 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ecdfffbdc0 con 0x55ece00aa370 2024-05-09T08:39:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.446+0000 7fe512ffd640 1 --2- 172.21.15.107:0/423533318 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ece00aa370 0x55ece0112c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.447+0000 7fe512ffd640 1 --2- 172.21.15.107:0/423533318 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ece00aa370 0x55ece0112c00 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfd7fb647a15ea86 server_cookie=123b917640a8041c in_seq=0 out_seq=0 2024-05-09T08:39:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.447+0000 7fe4f2ffd640 10 client.17367 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:12.464 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.465+0000 7fe4f2ffd640 1 -- 172.21.15.107:0/423533318 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ecdfffbdc0 con 0x55ece00aa370 2024-05-09T08:39:12.464 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.465+0000 7fe4f2ffd640 10 client.17367 handle_command_reply: tid=0 2024-05-09T08:39:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.465+0000 7fe51c98f740 1 client.17367 shutdown 2024-05-09T08:39:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.465+0000 7fe51c98f740 1 -- 172.21.15.107:0/423533318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008f450 msgr2=0x55ece00a2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.465+0000 7fe51c98f740 1 --2- 172.21.15.107:0/423533318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008f450 0x55ece00a2890 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fe504006010 tx=0x7fe50400b460 comp rx=0 tx=0).stop 2024-05-09T08:39:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.465+0000 7fe51c98f740 1 -- 172.21.15.107:0/423533318 shutdown_connections 2024-05-09T08:39:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.465+0000 7fe51c98f740 1 --2- 172.21.15.107:0/423533318 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ece00aa370 0x55ece0112c00 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.465+0000 7fe51c98f740 1 --2- 172.21.15.107:0/423533318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ece009a630 0x55ece00943f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.465+0000 7fe4f2ffd640 0 client.17367 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.465+0000 7fe51c98f740 1 --2- 172.21.15.107:0/423533318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ece008fd70 0x55ece0093eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.465+0000 7fe51c98f740 1 --2- 172.21.15.107:0/423533318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ece008f450 0x55ece00a2890 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.466+0000 7fe51c98f740 1 -- 172.21.15.107:0/423533318 >> 172.21.15.107:0/423533318 conn(0x55ece008b960 msgr2=0x55ece0091690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.466+0000 7fe51c98f740 1 -- 172.21.15.107:0/423533318 shutdown_connections 2024-05-09T08:39:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.466+0000 7fe51c98f740 1 -- 172.21.15.107:0/423533318 wait complete. 2024-05-09T08:39:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.466+0000 7fe51c98f740 20 client.17367 trim_cache size 0 max 16384 2024-05-09T08:39:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:39:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:39:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:39:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:39:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.466+0000 7fe519ba5640 1 -- 172.21.15.107:0/1654704494 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4fc0a2e20 msgr2=0x7fe4fc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.466+0000 7fe519ba5640 1 --2- 172.21.15.107:0/1654704494 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4fc0a2e20 0x7fe4fc0a52e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe50c009900 tx=0x7fe50c009650 comp rx=0 tx=0).stop 2024-05-09T08:39:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.466+0000 7fe519ba5640 1 -- 172.21.15.107:0/1654704494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5141006f0 msgr2=0x7fe514198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.466+0000 7fe519ba5640 1 --2- 172.21.15.107:0/1654704494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5141006f0 0x7fe514198690 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fe508004770 tx=0x7fe5080042f0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.466+0000 7fe519ba5640 1 -- 172.21.15.107:0/1654704494 shutdown_connections 2024-05-09T08:39:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.466+0000 7fe519ba5640 1 --2- 172.21.15.107:0/1654704494 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4fc0a2e20 0x7fe4fc0a52e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.466+0000 7fe519ba5640 1 --2- 172.21.15.107:0/1654704494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe514106810 0x7fe51419c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.466+0000 7fe519ba5640 1 --2- 172.21.15.107:0/1654704494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5141006f0 0x7fe514198690 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.466+0000 7fe519ba5640 1 --2- 172.21.15.107:0/1654704494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe5140ffd30 0x7fe514198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:12.468 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.466+0000 7fe519ba5640 1 -- 172.21.15.107:0/1654704494 >> 172.21.15.107:0/1654704494 conn(0x7fe514076cc0 msgr2=0x7fe5140fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:12.468 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.467+0000 7fe519ba5640 1 -- 172.21.15.107:0/1654704494 shutdown_connections 2024-05-09T08:39:12.468 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:12.467+0000 7fe519ba5640 1 -- 172.21.15.107:0/1654704494 wait complete. 2024-05-09T08:39:12.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:12 smithi107 ceph-mon[30122]: pgmap v178: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 24 MiB/s wr, 34 op/s 2024-05-09T08:39:12.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:12 smithi157 ceph-mon[33614]: pgmap v178: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 24 MiB/s wr, 34 op/s 2024-05-09T08:39:12.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:12 smithi110 ceph-mon[33622]: pgmap v178: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 24 MiB/s wr, 34 op/s 2024-05-09T08:39:14.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:14 smithi107 ceph-mon[30122]: pgmap v179: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 37 MiB/s wr, 42 op/s 2024-05-09T08:39:14.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:14 smithi157 ceph-mon[33614]: pgmap v179: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 37 MiB/s wr, 42 op/s 2024-05-09T08:39:14.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:14 smithi110 ceph-mon[33622]: pgmap v179: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 37 MiB/s wr, 42 op/s 2024-05-09T08:39:16.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:16 smithi107 ceph-mon[30122]: pgmap v180: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:39:16.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:16 smithi157 ceph-mon[33614]: pgmap v180: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:39:16.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:16 smithi110 ceph-mon[33622]: pgmap v180: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:39:18.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:18 smithi107 ceph-mon[30122]: pgmap v181: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:18.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:18 smithi157 ceph-mon[33614]: pgmap v181: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:18.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:18 smithi110 ceph-mon[33622]: pgmap v181: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:20.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:20 smithi107 ceph-mon[30122]: pgmap v182: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:39:20.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:20 smithi157 ceph-mon[33614]: pgmap v182: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:39:20.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:20 smithi110 ceph-mon[33622]: pgmap v182: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:39:22.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:22 smithi107 ceph-mon[30122]: pgmap v183: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:39:22.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:22 smithi157 ceph-mon[33614]: pgmap v183: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:39:22.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:22 smithi110 ceph-mon[33622]: pgmap v183: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:39:23.049 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:39:23.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:39:23.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.170+0000 7f4c03322640 1 -- 172.21.15.107:0/1533743430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bfc107810 msgr2=0x7f4bfc107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.170+0000 7f4c03322640 1 --2- 172.21.15.107:0/1533743430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bfc107810 0x7f4bfc107bf0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009930 tx=0x7f4bf802a420 comp rx=0 tx=0).stop 2024-05-09T08:39:23.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.170+0000 7f4c03322640 1 -- 172.21.15.107:0/1533743430 shutdown_connections 2024-05-09T08:39:23.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.170+0000 7f4c03322640 1 --2- 172.21.15.107:0/1533743430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bfc06a7f0 0x7f4bfc10e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.170+0000 7f4c03322640 1 --2- 172.21.15.107:0/1533743430 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4bfc069e30 0x7f4bfc06a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.170+0000 7f4c03322640 1 --2- 172.21.15.107:0/1533743430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bfc107810 0x7f4bfc107bf0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.170+0000 7f4c03322640 1 -- 172.21.15.107:0/1533743430 >> 172.21.15.107:0/1533743430 conn(0x7f4bfc076530 msgr2=0x7f4bfc076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.171+0000 7f4c03322640 1 -- 172.21.15.107:0/1533743430 shutdown_connections 2024-05-09T08:39:23.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.171+0000 7f4c03322640 1 -- 172.21.15.107:0/1533743430 wait complete. 2024-05-09T08:39:23.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.171+0000 7f4c03322640 1 Processor -- start 2024-05-09T08:39:23.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.171+0000 7f4c03322640 1 -- start start 2024-05-09T08:39:23.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.171+0000 7f4c03322640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4bfc069e30 0x7f4bfc198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.171+0000 7f4c03322640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bfc06a7f0 0x7f4bfc1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.171+0000 7f4c03322640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bfc107810 0x7f4bfc19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.171+0000 7f4c03322640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc111ad0 con 0x7f4bfc107810 2024-05-09T08:39:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.171+0000 7f4c03322640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc111950 con 0x7f4bfc06a7f0 2024-05-09T08:39:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.171+0000 7f4c03322640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc111c50 con 0x7f4bfc069e30 2024-05-09T08:39:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4c018bf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bfc107810 0x7f4bfc19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4c018bf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bfc107810 0x7f4bfc19c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35510/0 (socket says 172.21.15.107:35510) 2024-05-09T08:39:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4c018bf640 1 -- 172.21.15.107:0/1517255731 learned_addr learned my addr 172.21.15.107:0/1517255731 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4c008bd640 1 --2- 172.21.15.107:0/1517255731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bfc06a7f0 0x7f4bfc1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4c010be640 1 --2- 172.21.15.107:0/1517255731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4bfc069e30 0x7f4bfc198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4c018bf640 1 -- 172.21.15.107:0/1517255731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4bfc069e30 msgr2=0x7f4bfc198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4c018bf640 1 --2- 172.21.15.107:0/1517255731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4bfc069e30 0x7f4bfc198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4c018bf640 1 -- 172.21.15.107:0/1517255731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bfc06a7f0 msgr2=0x7f4bfc1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4c018bf640 1 --2- 172.21.15.107:0/1517255731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bfc06a7f0 0x7f4bfc1985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4c018bf640 1 -- 172.21.15.107:0/1517255731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bfc19d060 con 0x7f4bfc107810 2024-05-09T08:39:23.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4c010be640 1 --2- 172.21.15.107:0/1517255731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4bfc069e30 0x7f4bfc198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:39:23.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4c018bf640 1 --2- 172.21.15.107:0/1517255731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bfc107810 0x7f4bfc19c930 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400eb10 tx=0x7f4bf400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4bf27fc640 1 -- 172.21.15.107:0/1517255731 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4004b20 con 0x7f4bfc107810 2024-05-09T08:39:23.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4bf27fc640 1 -- 172.21.15.107:0/1517255731 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bf4004cc0 con 0x7f4bfc107810 2024-05-09T08:39:23.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4c03322640 1 -- 172.21.15.107:0/1517255731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bfc19d350 con 0x7f4bfc107810 2024-05-09T08:39:23.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4bf27fc640 1 -- 172.21.15.107:0/1517255731 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4010690 con 0x7f4bfc107810 2024-05-09T08:39:23.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.172+0000 7f4c03322640 1 -- 172.21.15.107:0/1517255731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bfc1a4be0 con 0x7f4bfc107810 2024-05-09T08:39:23.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.175+0000 7f4bf27fc640 1 -- 172.21.15.107:0/1517255731 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f4bf4005020 con 0x7f4bfc107810 2024-05-09T08:39:23.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.175+0000 7f4c06035740 1 Processor -- start 2024-05-09T08:39:23.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.175+0000 7f4c06035740 1 -- start start 2024-05-09T08:39:23.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c06035740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a6c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c06035740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a76d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c06035740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a77270 0x55a7e6a79660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c06035740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a7e69d1d00 con 0x55a7e6a6c280 2024-05-09T08:39:23.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c06035740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a7e69794c0 con 0x55a7e6a6cba0 2024-05-09T08:39:23.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c06035740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a7e69cdcc0 con 0x55a7e6a77270 2024-05-09T08:39:23.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c008bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a6c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c008bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a6c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35512/0 (socket says 172.21.15.107:35512) 2024-05-09T08:39:23.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c008bd640 1 -- 172.21.15.107:0/3639080897 learned_addr learned my addr 172.21.15.107:0/3639080897 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:23.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4bf27fc640 1 --2- 172.21.15.107:0/1517255731 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4be80a2d00 0x7f4be80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c010be640 1 --2- 172.21.15.107:0/3639080897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a76d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c008bd640 1 --2- 172.21.15.107:0/3639080897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a77270 0x55a7e6a79660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c018bf640 1 --2- 172.21.15.107:0/1517255731 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4be80a2d00 0x7f4be80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c008bd640 1 -- 172.21.15.107:0/3639080897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a77270 msgr2=0x55a7e6a79660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c008bd640 1 --2- 172.21.15.107:0/3639080897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a77270 0x55a7e6a79660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c008bd640 1 -- 172.21.15.107:0/3639080897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a6cba0 msgr2=0x55a7e6a76d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c008bd640 1 --2- 172.21.15.107:0/3639080897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a76d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c008bd640 1 -- 172.21.15.107:0/3639080897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7e67af7d0 con 0x55a7e6a6c280 2024-05-09T08:39:23.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4bf27fc640 1 -- 172.21.15.107:0/1517255731 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4bf40cfd70 con 0x7f4bfc107810 2024-05-09T08:39:23.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.176+0000 7f4c008bd640 1 --2- 172.21.15.107:0/3639080897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a6c660 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f4bec0135a0 tx=0x7f4bec009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10d3c9553d67475 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.177+0000 7f4c018bf640 1 --2- 172.21.15.107:0/1517255731 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4be80a2d00 0x7f4be80a51c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4bf409bc30 tx=0x7f4bf400efd0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.177+0000 7f4be77fe640 1 -- 172.21.15.107:0/3639080897 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec004050 con 0x55a7e6a6c280 2024-05-09T08:39:23.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.177+0000 7f4be77fe640 1 -- 172.21.15.107:0/3639080897 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bec0041f0 con 0x55a7e6a6c280 2024-05-09T08:39:23.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.177+0000 7f4be77fe640 1 -- 172.21.15.107:0/3639080897 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec00f7e0 con 0x55a7e6a6c280 2024-05-09T08:39:23.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.178+0000 7f4c06035740 1 -- 172.21.15.107:0/3639080897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 msgr2=0x55a7e6a6c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.178+0000 7f4c06035740 1 --2- 172.21.15.107:0/3639080897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a6c660 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f4bec0135a0 tx=0x7f4bec009830 comp rx=0 tx=0).stop 2024-05-09T08:39:23.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.178+0000 7f4c06035740 1 -- 172.21.15.107:0/3639080897 shutdown_connections 2024-05-09T08:39:23.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.178+0000 7f4c06035740 1 --2- 172.21.15.107:0/3639080897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a77270 0x55a7e6a79660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.178+0000 7f4c06035740 1 --2- 172.21.15.107:0/3639080897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a76d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.178+0000 7f4c06035740 1 --2- 172.21.15.107:0/3639080897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a6c660 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.178+0000 7f4c06035740 1 -- 172.21.15.107:0/3639080897 >> 172.21.15.107:0/3639080897 conn(0x55a7e6a685a0 msgr2=0x55a7e6a68980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.178+0000 7f4c06035740 1 -- 172.21.15.107:0/3639080897 shutdown_connections 2024-05-09T08:39:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.178+0000 7f4c06035740 1 -- 172.21.15.107:0/3639080897 wait complete. 2024-05-09T08:39:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.178+0000 7f4c06035740 1 -- start start 2024-05-09T08:39:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.178+0000 7f4c06035740 1 Processor -- start 2024-05-09T08:39:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c06035740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a741d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c06035740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a74710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c06035740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a77270 0x55a7e6a74c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c06035740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a7e6a7a060 con 0x55a7e6a6c280 2024-05-09T08:39:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c06035740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a7e698cb90 con 0x55a7e6a77270 2024-05-09T08:39:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c06035740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a7e6a7a1e0 con 0x55a7e6a6cba0 2024-05-09T08:39:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c06035740 10 client.0 _ll_register_callbacks cb 0x55a7e69c8da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c008bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a741d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c008bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a741d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35520/0 (socket says 172.21.15.107:35520) 2024-05-09T08:39:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c008bd640 1 -- 172.21.15.107:0/3396868956 learned_addr learned my addr 172.21.15.107:0/3396868956 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:23.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c010be640 1 --2- 172.21.15.107:0/3396868956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a74710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c008bd640 1 --2- 172.21.15.107:0/3396868956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a77270 0x55a7e6a74c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c008bd640 1 -- 172.21.15.107:0/3396868956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a6cba0 msgr2=0x55a7e6a74710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c008bd640 1 --2- 172.21.15.107:0/3396868956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a74710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c008bd640 1 -- 172.21.15.107:0/3396868956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a77270 msgr2=0x55a7e6a74c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c008bd640 1 --2- 172.21.15.107:0/3396868956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a77270 0x55a7e6a74c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.179+0000 7f4c008bd640 1 -- 172.21.15.107:0/3396868956 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7e66e53f0 con 0x55a7e6a6c280 2024-05-09T08:39:23.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.180+0000 7f4c008bd640 1 --2- 172.21.15.107:0/3396868956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a741d0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f4bec009710 tx=0x7f4bec009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62cb26ab83849da8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.180+0000 7f4be4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:23.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.180+0000 7f4be4ff9640 1 -- 172.21.15.107:0/3396868956 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec02e060 con 0x55a7e6a6c280 2024-05-09T08:39:23.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.180+0000 7f4be4ff9640 1 -- 172.21.15.107:0/3396868956 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4bec003a00 con 0x55a7e6a6c280 2024-05-09T08:39:23.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.180+0000 7f4c06035740 1 -- 172.21.15.107:0/3396868956 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bec0041f0 con 0x55a7e6a6c280 2024-05-09T08:39:23.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.180+0000 7f4be4ff9640 1 -- 172.21.15.107:0/3396868956 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4bec024070 con 0x55a7e6a6c280 2024-05-09T08:39:23.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.180+0000 7f4c06035740 10 client.17379 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:23.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.180+0000 7f4c06035740 1 -- 172.21.15.107:0/3396868956 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a7e66a50f0 con 0x55a7e6a6c280 2024-05-09T08:39:23.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.180+0000 7f4be4ff9640 1 -- 172.21.15.107:0/3396868956 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bec0041f0 con 0x55a7e6a6c280 2024-05-09T08:39:23.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.180+0000 7f4be4ff9640 1 -- 172.21.15.107:0/3396868956 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f4bec01e070 con 0x55a7e6a6c280 2024-05-09T08:39:23.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.180+0000 7f4c06035740 10 client.17379 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:23.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.180+0000 7f4c06035740 10 client.17379 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:23.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.180+0000 7f4c06035740 20 client.17379 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:23.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.180+0000 7f4c06035740 1 --2- 172.21.15.107:0/3396868956 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a7e6a87550 0x55a7e6a89940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.180+0000 7f4c06035740 4 client.17379 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:23.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.181+0000 7f4c06035740 1 -- 172.21.15.107:0/3396868956 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7e69df430 con 0x55a7e6a87550 2024-05-09T08:39:23.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.181+0000 7f4c010be640 1 --2- 172.21.15.107:0/3396868956 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a7e6a87550 0x55a7e6a89940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.182+0000 7f4c010be640 1 --2- 172.21.15.107:0/3396868956 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a7e6a87550 0x55a7e6a89940 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82368f518f1e732b server_cookie=94b9a9a1ff7b52ed in_seq=0 out_seq=0 2024-05-09T08:39:23.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.182+0000 7f4be4ff9640 10 client.17379 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:23.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.182+0000 7f4be4ff9640 1 -- 172.21.15.107:0/3396868956 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a7e69df430 con 0x55a7e6a87550 2024-05-09T08:39:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.182+0000 7f4be4ff9640 10 client.17379 handle_command_reply: tid=0 2024-05-09T08:39:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.182+0000 7f4c06035740 1 client.17379 shutdown 2024-05-09T08:39:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.182+0000 7f4c06035740 1 -- 172.21.15.107:0/3396868956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 msgr2=0x55a7e6a741d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.182+0000 7f4c06035740 1 --2- 172.21.15.107:0/3396868956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a741d0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f4bec009710 tx=0x7f4bec009660 comp rx=0 tx=0).stop 2024-05-09T08:39:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.183+0000 7f4c06035740 1 -- 172.21.15.107:0/3396868956 shutdown_connections 2024-05-09T08:39:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.183+0000 7f4c06035740 1 --2- 172.21.15.107:0/3396868956 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a7e6a87550 0x55a7e6a89940 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.183+0000 7f4c06035740 1 --2- 172.21.15.107:0/3396868956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a77270 0x55a7e6a74c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.183+0000 7f4be4ff9640 0 client.17379 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.183+0000 7f4c06035740 1 --2- 172.21.15.107:0/3396868956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a74710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.183+0000 7f4c06035740 1 --2- 172.21.15.107:0/3396868956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a741d0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.183+0000 7f4c06035740 1 -- 172.21.15.107:0/3396868956 >> 172.21.15.107:0/3396868956 conn(0x55a7e6a685a0 msgr2=0x55a7e6a6e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.183+0000 7f4c06035740 1 -- 172.21.15.107:0/3396868956 shutdown_connections 2024-05-09T08:39:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.183+0000 7f4c06035740 1 -- 172.21.15.107:0/3396868956 wait complete. 2024-05-09T08:39:23.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.183+0000 7f4c06035740 20 client.17379 trim_cache size 0 max 16384 2024-05-09T08:39:23.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.209+0000 7f4c06035740 1 Processor -- start 2024-05-09T08:39:23.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.209+0000 7f4c06035740 1 -- start start 2024-05-09T08:39:23.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.209+0000 7f4c06035740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a6c280 0x55a7e6a82f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.210+0000 7f4c06035740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a73200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.210+0000 7f4c06035740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a77270 0x55a7e6a748a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.210+0000 7f4c06035740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc1b14a0 con 0x55a7e6a6cba0 2024-05-09T08:39:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.210+0000 7f4c06035740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc1aed70 con 0x55a7e6a6c280 2024-05-09T08:39:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.210+0000 7f4c06035740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc1acbc0 con 0x55a7e6a77270 2024-05-09T08:39:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.210+0000 7f4c010be640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a73200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.210+0000 7f4c010be640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a73200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35526/0 (socket says 172.21.15.107:35526) 2024-05-09T08:39:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.210+0000 7f4c010be640 1 -- 172.21.15.107:0/3620378655 learned_addr learned my addr 172.21.15.107:0/3620378655 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.210+0000 7f4c008bd640 1 --2- 172.21.15.107:0/3620378655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a77270 0x55a7e6a748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.210+0000 7f4c010be640 1 -- 172.21.15.107:0/3620378655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a77270 msgr2=0x55a7e6a748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.210+0000 7f4c010be640 1 --2- 172.21.15.107:0/3620378655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a77270 0x55a7e6a748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.210+0000 7f4c010be640 1 -- 172.21.15.107:0/3620378655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a6c280 msgr2=0x55a7e6a82f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.210+0000 7f4c010be640 1 --2- 172.21.15.107:0/3620378655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a6c280 0x55a7e6a82f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.210+0000 7f4c010be640 1 -- 172.21.15.107:0/3620378655 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7e6a683d0 con 0x55a7e6a6cba0 2024-05-09T08:39:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.210+0000 7f4c010be640 1 --2- 172.21.15.107:0/3620378655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a73200 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009930 tx=0x7f4bf8005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f52f44da8fe9844b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.210+0000 7f4be6ffd640 1 -- 172.21.15.107:0/3620378655 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf804a070 con 0x55a7e6a6cba0 2024-05-09T08:39:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.211+0000 7f4be6ffd640 1 -- 172.21.15.107:0/3620378655 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bf8045dc0 con 0x55a7e6a6cba0 2024-05-09T08:39:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.211+0000 7f4be6ffd640 1 -- 172.21.15.107:0/3620378655 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8035040 con 0x55a7e6a6cba0 2024-05-09T08:39:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.211+0000 7f4c06035740 1 -- 172.21.15.107:0/3620378655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6cba0 msgr2=0x55a7e6a73200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.211+0000 7f4c06035740 1 --2- 172.21.15.107:0/3620378655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a73200 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009930 tx=0x7f4bf8005b50 comp rx=0 tx=0).stop 2024-05-09T08:39:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.211+0000 7f4c06035740 1 -- 172.21.15.107:0/3620378655 shutdown_connections 2024-05-09T08:39:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.211+0000 7f4c06035740 1 --2- 172.21.15.107:0/3620378655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a77270 0x55a7e6a748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.211+0000 7f4c06035740 1 --2- 172.21.15.107:0/3620378655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a73200 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.211+0000 7f4c06035740 1 --2- 172.21.15.107:0/3620378655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a6c280 0x55a7e6a82f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.211+0000 7f4c06035740 1 -- 172.21.15.107:0/3620378655 >> 172.21.15.107:0/3620378655 conn(0x55a7e6a685a0 msgr2=0x55a7e6a74190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.211+0000 7f4c06035740 1 -- 172.21.15.107:0/3620378655 shutdown_connections 2024-05-09T08:39:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.211+0000 7f4c06035740 1 -- 172.21.15.107:0/3620378655 wait complete. 2024-05-09T08:39:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.212+0000 7f4c06035740 1 -- start start 2024-05-09T08:39:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.212+0000 7f4c06035740 1 Processor -- start 2024-05-09T08:39:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.212+0000 7f4c06035740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a7a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.212+0000 7f4c06035740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a7ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.212+0000 7f4c06035740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a77270 0x55a7e6aeb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.212+0000 7f4c06035740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc1ad230 con 0x55a7e6a6c280 2024-05-09T08:39:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.212+0000 7f4c06035740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc1ae6a0 con 0x55a7e6a6cba0 2024-05-09T08:39:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.212+0000 7f4c06035740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc1ad900 con 0x55a7e6a77270 2024-05-09T08:39:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.212+0000 7f4c06035740 10 client.0 _ll_register_callbacks cb 0x7f4bf40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.212+0000 7f4c008bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a7a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.212+0000 7f4c008bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a7a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35528/0 (socket says 172.21.15.107:35528) 2024-05-09T08:39:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.212+0000 7f4c008bd640 1 -- 172.21.15.107:0/36782183 learned_addr learned my addr 172.21.15.107:0/36782183 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.213+0000 7f4c010be640 1 --2- 172.21.15.107:0/36782183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a7ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.213+0000 7f4c008bd640 1 -- 172.21.15.107:0/36782183 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a77270 msgr2=0x55a7e6aeb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.213+0000 7f4c008bd640 1 --2- 172.21.15.107:0/36782183 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a77270 0x55a7e6aeb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.213+0000 7f4c008bd640 1 -- 172.21.15.107:0/36782183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a6cba0 msgr2=0x55a7e6a7ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.213+0000 7f4c008bd640 1 --2- 172.21.15.107:0/36782183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a7ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.213+0000 7f4c008bd640 1 -- 172.21.15.107:0/36782183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7e6aebe20 con 0x55a7e6a6c280 2024-05-09T08:39:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.213+0000 7f4c010be640 1 --2- 172.21.15.107:0/36782183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a7ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.213+0000 7f4c008bd640 1 --2- 172.21.15.107:0/36782183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a7a620 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f4bec004cb0 tx=0x7f4bec00f5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce6d7cc9b28492fa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.213+0000 7f4be4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.213+0000 7f4be4ff9640 1 -- 172.21.15.107:0/36782183 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec02e070 con 0x55a7e6a6c280 2024-05-09T08:39:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.213+0000 7f4be4ff9640 1 -- 172.21.15.107:0/36782183 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4bec0051a0 con 0x55a7e6a6c280 2024-05-09T08:39:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.213+0000 7f4c06035740 1 -- 172.21.15.107:0/36782183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bf8045dc0 con 0x55a7e6a6c280 2024-05-09T08:39:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.213+0000 7f4be4ff9640 1 -- 172.21.15.107:0/36782183 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4bec043050 con 0x55a7e6a6c280 2024-05-09T08:39:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.213+0000 7f4c06035740 10 client.17385 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.213+0000 7f4c06035740 1 -- 172.21.15.107:0/36782183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a7e6aed110 con 0x55a7e6a6c280 2024-05-09T08:39:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.214+0000 7f4be4ff9640 1 -- 172.21.15.107:0/36782183 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bf8045dc0 con 0x55a7e6a6c280 2024-05-09T08:39:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.214+0000 7f4be4ff9640 1 -- 172.21.15.107:0/36782183 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f4bec01e040 con 0x55a7e6a6c280 2024-05-09T08:39:23.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.214+0000 7f4c06035740 10 client.17385 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:23.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.214+0000 7f4c06035740 10 client.17385 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:23.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.214+0000 7f4c06035740 20 client.17385 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:23.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.214+0000 7f4c06035740 1 --2- 172.21.15.107:0/36782183 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a7e6aed860 0x55a7e6aefc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.214+0000 7f4c06035740 4 client.17385 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:39:23.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.214+0000 7f4c06035740 1 -- 172.21.15.107:0/36782183 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a7e6a73880 con 0x55a7e6aed860 2024-05-09T08:39:23.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.215+0000 7f4c010be640 1 --2- 172.21.15.107:0/36782183 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a7e6aed860 0x55a7e6aefc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.216+0000 7f4c010be640 1 --2- 172.21.15.107:0/36782183 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a7e6aed860 0x55a7e6aefc80 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dbd71ef4a1246f4 server_cookie=550b28c9cb1a8639 in_seq=0 out_seq=0 2024-05-09T08:39:23.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.216+0000 7f4be4ff9640 10 client.17385 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:23.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.217+0000 7f4be4ff9640 1 -- 172.21.15.107:0/36782183 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a7e6a73880 con 0x55a7e6aed860 2024-05-09T08:39:23.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.217+0000 7f4be4ff9640 10 client.17385 handle_command_reply: tid=0 2024-05-09T08:39:23.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.217+0000 7f4c06035740 1 client.17385 shutdown 2024-05-09T08:39:23.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.217+0000 7f4c06035740 1 -- 172.21.15.107:0/36782183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 msgr2=0x55a7e6a7a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.217+0000 7f4c06035740 1 --2- 172.21.15.107:0/36782183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a7a620 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f4bec004cb0 tx=0x7f4bec00f5f0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.217+0000 7f4c06035740 1 -- 172.21.15.107:0/36782183 shutdown_connections 2024-05-09T08:39:23.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.217+0000 7f4c06035740 1 --2- 172.21.15.107:0/36782183 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a7e6aed860 0x55a7e6aefc80 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.217+0000 7f4c06035740 1 --2- 172.21.15.107:0/36782183 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a7e6a77270 0x55a7e6aeb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.217+0000 7f4c06035740 1 --2- 172.21.15.107:0/36782183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a7e6a6cba0 0x55a7e6a7ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.217+0000 7f4be4ff9640 0 client.17385 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:23.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.217+0000 7f4c06035740 1 --2- 172.21.15.107:0/36782183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a7e6a6c280 0x55a7e6a7a620 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.217+0000 7f4c06035740 1 -- 172.21.15.107:0/36782183 >> 172.21.15.107:0/36782183 conn(0x55a7e6a685a0 msgr2=0x55a7e6a74a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.217+0000 7f4c06035740 1 -- 172.21.15.107:0/36782183 shutdown_connections 2024-05-09T08:39:23.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.217+0000 7f4c06035740 1 -- 172.21.15.107:0/36782183 wait complete. 2024-05-09T08:39:23.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.217+0000 7f4c06035740 20 client.17385 trim_cache size 0 max 16384 2024-05-09T08:39:23.220 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:39:23.220 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:39:23.220 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:39:23.220 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:39:23.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.218+0000 7f4c03322640 1 -- 172.21.15.107:0/1517255731 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4be80a2d00 msgr2=0x7f4be80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.218+0000 7f4c03322640 1 --2- 172.21.15.107:0/1517255731 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4be80a2d00 0x7f4be80a51c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4bf409bc30 tx=0x7f4bf400efd0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.218+0000 7f4c03322640 1 -- 172.21.15.107:0/1517255731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bfc107810 msgr2=0x7f4bfc19c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.218+0000 7f4c03322640 1 --2- 172.21.15.107:0/1517255731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bfc107810 0x7f4bfc19c930 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400eb10 tx=0x7f4bf400c2d0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.218+0000 7f4c03322640 1 -- 172.21.15.107:0/1517255731 shutdown_connections 2024-05-09T08:39:23.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.218+0000 7f4c03322640 1 --2- 172.21.15.107:0/1517255731 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4be80a2d00 0x7f4be80a51c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.218+0000 7f4c03322640 1 --2- 172.21.15.107:0/1517255731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bfc107810 0x7f4bfc19c930 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.218+0000 7f4c03322640 1 --2- 172.21.15.107:0/1517255731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bfc06a7f0 0x7f4bfc1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.218+0000 7f4c03322640 1 --2- 172.21.15.107:0/1517255731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4bfc069e30 0x7f4bfc198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.218+0000 7f4c03322640 1 -- 172.21.15.107:0/1517255731 >> 172.21.15.107:0/1517255731 conn(0x7f4bfc076530 msgr2=0x7f4bfc0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.218+0000 7f4c03322640 1 -- 172.21.15.107:0/1517255731 shutdown_connections 2024-05-09T08:39:23.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.218+0000 7f4c03322640 1 -- 172.21.15.107:0/1517255731 wait complete. 2024-05-09T08:39:23.227 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:39:23.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:39:23.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.333+0000 7f4a4e710640 1 -- 172.21.15.107:0/2609879212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a48106720 msgr2=0x7f4a48106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.333+0000 7f4a4e710640 1 --2- 172.21.15.107:0/2609879212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a48106720 0x7f4a48106b00 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f4a38009930 tx=0x7f4a3802a420 comp rx=0 tx=0).stop 2024-05-09T08:39:23.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.334+0000 7f4a4e710640 1 -- 172.21.15.107:0/2609879212 shutdown_connections 2024-05-09T08:39:23.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.334+0000 7f4a4e710640 1 --2- 172.21.15.107:0/2609879212 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a481010d0 0x7f4a4810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.334+0000 7f4a4e710640 1 --2- 172.21.15.107:0/2609879212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a48100710 0x7f4a48100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.334+0000 7f4a4e710640 1 --2- 172.21.15.107:0/2609879212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a48106720 0x7f4a48106b00 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.334+0000 7f4a4e710640 1 -- 172.21.15.107:0/2609879212 >> 172.21.15.107:0/2609879212 conn(0x7f4a48077830 msgr2=0x7f4a48077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.334+0000 7f4a4e710640 1 -- 172.21.15.107:0/2609879212 shutdown_connections 2024-05-09T08:39:23.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.334+0000 7f4a4e710640 1 -- 172.21.15.107:0/2609879212 wait complete. 2024-05-09T08:39:23.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.334+0000 7f4a4e710640 1 Processor -- start 2024-05-09T08:39:23.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.334+0000 7f4a4e710640 1 -- start start 2024-05-09T08:39:23.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a4e710640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a48100710 0x7f4a48198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a4e710640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a481010d0 0x7f4a48199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a4e710640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a48106720 0x7f4a481a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a4e710640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4a481109a0 con 0x7f4a48100710 2024-05-09T08:39:23.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a4e710640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4a48110820 con 0x7f4a481010d0 2024-05-09T08:39:23.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a4e710640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4a48110b20 con 0x7f4a48106720 2024-05-09T08:39:23.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a47fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a48100710 0x7f4a48198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a47fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a48100710 0x7f4a48198e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35556/0 (socket says 172.21.15.107:35556) 2024-05-09T08:39:23.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a47fff640 1 -- 172.21.15.107:0/3450068770 learned_addr learned my addr 172.21.15.107:0/3450068770 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:23.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a3ffff640 1 --2- 172.21.15.107:0/3450068770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a481010d0 0x7f4a48199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a4ccad640 1 --2- 172.21.15.107:0/3450068770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a48106720 0x7f4a481a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a47fff640 1 -- 172.21.15.107:0/3450068770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a48106720 msgr2=0x7f4a481a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a47fff640 1 --2- 172.21.15.107:0/3450068770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a48106720 0x7f4a481a3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a47fff640 1 -- 172.21.15.107:0/3450068770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a481010d0 msgr2=0x7f4a48199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a47fff640 1 --2- 172.21.15.107:0/3450068770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a481010d0 0x7f4a48199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a47fff640 1 -- 172.21.15.107:0/3450068770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a481a43e0 con 0x7f4a48100710 2024-05-09T08:39:23.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a4ccad640 1 --2- 172.21.15.107:0/3450068770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a48106720 0x7f4a481a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:23.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a3ffff640 1 --2- 172.21.15.107:0/3450068770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a481010d0 0x7f4a48199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:39:23.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a47fff640 1 --2- 172.21.15.107:0/3450068770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a48100710 0x7f4a48198e20 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f4a38005940 tx=0x7f4a380042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a45ffb640 1 -- 172.21.15.107:0/3450068770 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3803f030 con 0x7f4a48100710 2024-05-09T08:39:23.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a45ffb640 1 -- 172.21.15.107:0/3450068770 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a38003cb0 con 0x7f4a48100710 2024-05-09T08:39:23.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.335+0000 7f4a4e710640 1 -- 172.21.15.107:0/3450068770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a481a45b0 con 0x7f4a48100710 2024-05-09T08:39:23.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.336+0000 7f4a45ffb640 1 -- 172.21.15.107:0/3450068770 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a38035070 con 0x7f4a48100710 2024-05-09T08:39:23.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.336+0000 7f4a4e710640 1 -- 172.21.15.107:0/3450068770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a481a4a10 con 0x7f4a48100710 2024-05-09T08:39:23.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.337+0000 7f4a4e710640 1 -- 172.21.15.107:0/3450068770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a48068650 con 0x7f4a48100710 2024-05-09T08:39:23.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.338+0000 7f4a45ffb640 1 -- 172.21.15.107:0/3450068770 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f4a38042030 con 0x7f4a48100710 2024-05-09T08:39:23.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.339+0000 7f4a45ffb640 1 --2- 172.21.15.107:0/3450068770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4a340a2d00 0x7f4a340a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.339+0000 7f4a3ffff640 1 --2- 172.21.15.107:0/3450068770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4a340a2d00 0x7f4a340a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.339+0000 7f4a45ffb640 1 -- 172.21.15.107:0/3450068770 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4a380efe20 con 0x7f4a48100710 2024-05-09T08:39:23.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.340+0000 7f4a3ffff640 1 --2- 172.21.15.107:0/3450068770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4a340a2d00 0x7f4a340a51c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4a48069510 tx=0x7f4a3000a5c0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.343+0000 7f4a45ffb640 1 -- 172.21.15.107:0/3450068770 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4a380bc6e0 con 0x7f4a48100710 2024-05-09T08:39:23.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.595+0000 7f4a4e710640 1 -- 172.21.15.107:0/3450068770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a4802d1c0 con 0x7f4a48100710 2024-05-09T08:39:23.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.596+0000 7f4a45ffb640 1 -- 172.21.15.107:0/3450068770 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11887 (secure 0 0 0) 0x7f4a380bbfe0 con 0x7f4a48100710 2024-05-09T08:39:23.595 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:39:23.595 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:37:10.930317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:39:23.596 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2024-05-09T08:39:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.599+0000 7f4a4e710640 1 -- 172.21.15.107:0/3450068770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4a340a2d00 msgr2=0x7f4a340a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.599+0000 7f4a4e710640 1 --2- 172.21.15.107:0/3450068770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4a340a2d00 0x7f4a340a51c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4a48069510 tx=0x7f4a3000a5c0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.599+0000 7f4a4e710640 1 -- 172.21.15.107:0/3450068770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a48100710 msgr2=0x7f4a48198e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.599+0000 7f4a4e710640 1 --2- 172.21.15.107:0/3450068770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a48100710 0x7f4a48198e20 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f4a38005940 tx=0x7f4a380042f0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.599+0000 7f4a4e710640 1 -- 172.21.15.107:0/3450068770 shutdown_connections 2024-05-09T08:39:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.599+0000 7f4a4e710640 1 --2- 172.21.15.107:0/3450068770 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4a340a2d00 0x7f4a340a51c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.599+0000 7f4a4e710640 1 --2- 172.21.15.107:0/3450068770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a48106720 0x7f4a481a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.599+0000 7f4a4e710640 1 --2- 172.21.15.107:0/3450068770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a481010d0 0x7f4a48199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.599+0000 7f4a4e710640 1 --2- 172.21.15.107:0/3450068770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a48100710 0x7f4a48198e20 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.599+0000 7f4a4e710640 1 -- 172.21.15.107:0/3450068770 >> 172.21.15.107:0/3450068770 conn(0x7f4a48077830 msgr2=0x7f4a48103030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.599+0000 7f4a4e710640 1 -- 172.21.15.107:0/3450068770 shutdown_connections 2024-05-09T08:39:23.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.599+0000 7f4a4e710640 1 -- 172.21.15.107:0/3450068770 wait complete. 2024-05-09T08:39:23.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:39:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.728+0000 7fe291932640 1 -- 172.21.15.107:0/3550246136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe28c1033d0 msgr2=0x7fe28c10b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.728+0000 7fe291932640 1 --2- 172.21.15.107:0/3550246136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe28c1033d0 0x7fe28c10b8b0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fe280009900 tx=0x7fe28002a510 comp rx=0 tx=0).stop 2024-05-09T08:39:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.728+0000 7fe291932640 1 -- 172.21.15.107:0/3550246136 shutdown_connections 2024-05-09T08:39:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.728+0000 7fe291932640 1 --2- 172.21.15.107:0/3550246136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe28c10bdf0 0x7fe28c10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.728+0000 7fe291932640 1 --2- 172.21.15.107:0/3550246136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe28c1033d0 0x7fe28c10b8b0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.728+0000 7fe291932640 1 --2- 172.21.15.107:0/3550246136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe28c102ab0 0x7fe28c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.728+0000 7fe291932640 1 -- 172.21.15.107:0/3550246136 >> 172.21.15.107:0/3550246136 conn(0x7fe28c077830 msgr2=0x7fe28c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.728+0000 7fe291932640 1 -- 172.21.15.107:0/3550246136 shutdown_connections 2024-05-09T08:39:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.728+0000 7fe291932640 1 -- 172.21.15.107:0/3550246136 wait complete. 2024-05-09T08:39:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.729+0000 7fe291932640 1 Processor -- start 2024-05-09T08:39:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.729+0000 7fe291932640 1 -- start start 2024-05-09T08:39:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.729+0000 7fe291932640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe28c102ab0 0x7fe28c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.729+0000 7fe291932640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe28c1033d0 0x7fe28c1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.729+0000 7fe291932640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe28c10bdf0 0x7fe28c19c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.729+0000 7fe291932640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c111af0 con 0x7fe28c10bdf0 2024-05-09T08:39:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.729+0000 7fe291932640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c111970 con 0x7fe28c102ab0 2024-05-09T08:39:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.729+0000 7fe291932640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c111c70 con 0x7fe28c1033d0 2024-05-09T08:39:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.729+0000 7fe28b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe28c10bdf0 0x7fe28c19c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.729+0000 7fe28a7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe28c1033d0 0x7fe28c1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.729+0000 7fe28b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe28c10bdf0 0x7fe28c19c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35570/0 (socket says 172.21.15.107:35570) 2024-05-09T08:39:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.729+0000 7fe28b7fe640 1 -- 172.21.15.107:0/3095663115 learned_addr learned my addr 172.21.15.107:0/3095663115 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.729+0000 7fe28affd640 1 --2- 172.21.15.107:0/3095663115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe28c102ab0 0x7fe28c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.730+0000 7fe28b7fe640 1 -- 172.21.15.107:0/3095663115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe28c1033d0 msgr2=0x7fe28c1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.730+0000 7fe28b7fe640 1 --2- 172.21.15.107:0/3095663115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe28c1033d0 0x7fe28c1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.730+0000 7fe28b7fe640 1 -- 172.21.15.107:0/3095663115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe28c102ab0 msgr2=0x7fe28c198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.730+0000 7fe28b7fe640 1 --2- 172.21.15.107:0/3095663115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe28c102ab0 0x7fe28c198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.730+0000 7fe28b7fe640 1 -- 172.21.15.107:0/3095663115 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28c19d080 con 0x7fe28c10bdf0 2024-05-09T08:39:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.730+0000 7fe28b7fe640 1 --2- 172.21.15.107:0/3095663115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe28c10bdf0 0x7fe28c19c950 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fe284009fd0 tx=0x7fe28400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.730+0000 7fe290930640 1 -- 172.21.15.107:0/3095663115 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284017070 con 0x7fe28c10bdf0 2024-05-09T08:39:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.730+0000 7fe291932640 1 -- 172.21.15.107:0/3095663115 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28c19d370 con 0x7fe28c10bdf0 2024-05-09T08:39:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.730+0000 7fe290930640 1 -- 172.21.15.107:0/3095663115 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe284003e20 con 0x7fe28c10bdf0 2024-05-09T08:39:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.730+0000 7fe290930640 1 -- 172.21.15.107:0/3095663115 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284010460 con 0x7fe28c10bdf0 2024-05-09T08:39:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.730+0000 7fe291932640 1 -- 172.21.15.107:0/3095663115 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe28c1a4c00 con 0x7fe28c10bdf0 2024-05-09T08:39:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.733+0000 7fe290930640 1 -- 172.21.15.107:0/3095663115 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe284003970 con 0x7fe28c10bdf0 2024-05-09T08:39:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.733+0000 7fe29463d740 1 Processor -- start 2024-05-09T08:39:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.733+0000 7fe29463d740 1 -- start start 2024-05-09T08:39:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.733+0000 7fe29463d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89364230 0x557f89364610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.733+0000 7fe29463d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 0x557f893731f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.733+0000 7fe29463d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89373730 0x557f89375b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.733+0000 7fe29463d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557f892c9c40 con 0x557f89364230 2024-05-09T08:39:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.733+0000 7fe29463d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557f89271580 con 0x557f89364b50 2024-05-09T08:39:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.733+0000 7fe29463d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x557f892c5c00 con 0x557f89373730 2024-05-09T08:39:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.733+0000 7fe28a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89364230 0x557f89364610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.733+0000 7fe28a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89364230 0x557f89364610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35578/0 (socket says 172.21.15.107:35578) 2024-05-09T08:39:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.733+0000 7fe28a7fc640 1 -- 172.21.15.107:0/3286060694 learned_addr learned my addr 172.21.15.107:0/3286060694 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.733+0000 7fe28affd640 1 --2- 172.21.15.107:0/3286060694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 0x557f893731f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.733+0000 7fe28a7fc640 1 --2- 172.21.15.107:0/3286060694 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89373730 0x557f89375b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.734+0000 7fe290930640 1 --2- 172.21.15.107:0/3095663115 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2740a2d00 0x7fe2740a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.734+0000 7fe28a7fc640 1 -- 172.21.15.107:0/3286060694 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89373730 msgr2=0x557f89375b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.734+0000 7fe28a7fc640 1 --2- 172.21.15.107:0/3286060694 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89373730 0x557f89375b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.734+0000 7fe28a7fc640 1 -- 172.21.15.107:0/3286060694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 msgr2=0x557f893731f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.734+0000 7fe28a7fc640 1 --2- 172.21.15.107:0/3286060694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 0x557f893731f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.734+0000 7fe28a7fc640 1 -- 172.21.15.107:0/3286060694 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f88fdd3f0 con 0x557f89364230 2024-05-09T08:39:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.734+0000 7fe28b7fe640 1 --2- 172.21.15.107:0/3095663115 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2740a2d00 0x7fe2740a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.734+0000 7fe28a7fc640 1 --2- 172.21.15.107:0/3286060694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89364230 0x557f89364610 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fe280000c00 tx=0x7fe280009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee224976618de18 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.734+0000 7fe290930640 1 -- 172.21.15.107:0/3095663115 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe2840cf4a0 con 0x7fe28c10bdf0 2024-05-09T08:39:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.734+0000 7fe271ffb640 1 -- 172.21.15.107:0/3286060694 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280044020 con 0x557f89364230 2024-05-09T08:39:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.734+0000 7fe271ffb640 1 -- 172.21.15.107:0/3286060694 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe280004770 con 0x557f89364230 2024-05-09T08:39:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.734+0000 7fe271ffb640 1 -- 172.21.15.107:0/3286060694 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280035040 con 0x557f89364230 2024-05-09T08:39:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.735+0000 7fe29463d740 1 -- 172.21.15.107:0/3286060694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89364230 msgr2=0x557f89364610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.735+0000 7fe29463d740 1 --2- 172.21.15.107:0/3286060694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89364230 0x557f89364610 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fe280000c00 tx=0x7fe280009560 comp rx=0 tx=0).stop 2024-05-09T08:39:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.735+0000 7fe28b7fe640 1 --2- 172.21.15.107:0/3095663115 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2740a2d00 0x7fe2740a51c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe28409b410 tx=0x7fe284004280 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.735+0000 7fe29463d740 1 -- 172.21.15.107:0/3286060694 shutdown_connections 2024-05-09T08:39:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.735+0000 7fe29463d740 1 --2- 172.21.15.107:0/3286060694 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89373730 0x557f89375b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.735+0000 7fe29463d740 1 --2- 172.21.15.107:0/3286060694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 0x557f893731f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.735+0000 7fe29463d740 1 --2- 172.21.15.107:0/3286060694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89364230 0x557f89364610 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.735+0000 7fe29463d740 1 -- 172.21.15.107:0/3286060694 >> 172.21.15.107:0/3286060694 conn(0x557f89360650 msgr2=0x557f89360a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.735+0000 7fe29463d740 1 -- 172.21.15.107:0/3286060694 shutdown_connections 2024-05-09T08:39:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.735+0000 7fe29463d740 1 -- 172.21.15.107:0/3286060694 wait complete. 2024-05-09T08:39:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.735+0000 7fe29463d740 1 -- start start 2024-05-09T08:39:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe29463d740 1 Processor -- start 2024-05-09T08:39:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe29463d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89364230 0x557f89370630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe29463d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 0x557f89370b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe29463d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89373730 0x557f893710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe29463d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557f89376490 con 0x557f89373730 2024-05-09T08:39:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe29463d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557f89284c40 con 0x557f89364b50 2024-05-09T08:39:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe29463d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x557f89376610 con 0x557f89364230 2024-05-09T08:39:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe29463d740 10 client.0 _ll_register_callbacks cb 0x557f88fdcbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe28a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89373730 0x557f893710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe28a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89373730 0x557f893710b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35588/0 (socket says 172.21.15.107:35588) 2024-05-09T08:39:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe28a7fc640 1 -- 172.21.15.107:0/328709390 learned_addr learned my addr 172.21.15.107:0/328709390 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe28a7fc640 1 --2- 172.21.15.107:0/328709390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89364230 0x557f89370630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe28affd640 1 --2- 172.21.15.107:0/328709390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 0x557f89370b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe28a7fc640 1 -- 172.21.15.107:0/328709390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89364230 msgr2=0x557f89370630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe28a7fc640 1 --2- 172.21.15.107:0/328709390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89364230 0x557f89370630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe28a7fc640 1 -- 172.21.15.107:0/328709390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 msgr2=0x557f89370b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe28a7fc640 1 --2- 172.21.15.107:0/328709390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 0x557f89370b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe28a7fc640 1 -- 172.21.15.107:0/328709390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f890a77d0 con 0x557f89373730 2024-05-09T08:39:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.736+0000 7fe28affd640 1 --2- 172.21.15.107:0/328709390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 0x557f89370b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:39:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.737+0000 7fe28a7fc640 1 --2- 172.21.15.107:0/328709390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89373730 0x557f893710b0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fe280009340 tx=0x7fe280005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8f263737f46e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.737+0000 7fe26affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.737+0000 7fe26affd640 1 -- 172.21.15.107:0/328709390 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280044070 con 0x557f89373730 2024-05-09T08:39:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.737+0000 7fe26affd640 1 -- 172.21.15.107:0/328709390 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe280005240 con 0x557f89373730 2024-05-09T08:39:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.737+0000 7fe29463d740 1 -- 172.21.15.107:0/328709390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe280004770 con 0x557f89373730 2024-05-09T08:39:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.737+0000 7fe26affd640 1 -- 172.21.15.107:0/328709390 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe280035040 con 0x557f89373730 2024-05-09T08:39:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.737+0000 7fe29463d740 10 client.17403 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.737+0000 7fe29463d740 1 -- 172.21.15.107:0/328709390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557f893826a0 con 0x557f89373730 2024-05-09T08:39:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.737+0000 7fe26affd640 1 -- 172.21.15.107:0/328709390 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe280004770 con 0x557f89373730 2024-05-09T08:39:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.737+0000 7fe26affd640 1 -- 172.21.15.107:0/328709390 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe2800315c0 con 0x557f89373730 2024-05-09T08:39:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.737+0000 7fe29463d740 10 client.17403 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.737+0000 7fe29463d740 10 client.17403 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:39:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.737+0000 7fe29463d740 20 client.17403 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.737+0000 7fe29463d740 1 --2- 172.21.15.107:0/328709390 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x557f893839f0 0x557f89385de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.737+0000 7fe29463d740 4 client.17403 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.737+0000 7fe29463d740 1 -- 172.21.15.107:0/328709390 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f892d7370 con 0x557f893839f0 2024-05-09T08:39:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.738+0000 7fe28affd640 1 --2- 172.21.15.107:0/328709390 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x557f893839f0 0x557f89385de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.738+0000 7fe28affd640 1 --2- 172.21.15.107:0/328709390 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x557f893839f0 0x557f89385de0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d37170d72b8c0c86 server_cookie=9eba64f74390d4f in_seq=0 out_seq=0 2024-05-09T08:39:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.738+0000 7fe26affd640 10 client.17403 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:39:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.739+0000 7fe26affd640 1 -- 172.21.15.107:0/328709390 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557f892d7370 con 0x557f893839f0 2024-05-09T08:39:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.739+0000 7fe26affd640 10 client.17403 handle_command_reply: tid=0 2024-05-09T08:39:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.739+0000 7fe29463d740 1 client.17403 shutdown 2024-05-09T08:39:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.739+0000 7fe29463d740 1 -- 172.21.15.107:0/328709390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89373730 msgr2=0x557f893710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.739+0000 7fe29463d740 1 --2- 172.21.15.107:0/328709390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89373730 0x557f893710b0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fe280009340 tx=0x7fe280005d50 comp rx=0 tx=0).stop 2024-05-09T08:39:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.739+0000 7fe29463d740 1 -- 172.21.15.107:0/328709390 shutdown_connections 2024-05-09T08:39:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.739+0000 7fe29463d740 1 --2- 172.21.15.107:0/328709390 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x557f893839f0 0x557f89385de0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.739+0000 7fe29463d740 1 --2- 172.21.15.107:0/328709390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89373730 0x557f893710b0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.739+0000 7fe29463d740 1 --2- 172.21.15.107:0/328709390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 0x557f89370b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.739+0000 7fe26affd640 0 client.17403 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:39:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.739+0000 7fe29463d740 1 --2- 172.21.15.107:0/328709390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89364230 0x557f89370630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.739+0000 7fe29463d740 1 -- 172.21.15.107:0/328709390 >> 172.21.15.107:0/328709390 conn(0x557f89360650 msgr2=0x557f893663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.739+0000 7fe29463d740 1 -- 172.21.15.107:0/328709390 shutdown_connections 2024-05-09T08:39:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.739+0000 7fe29463d740 1 -- 172.21.15.107:0/328709390 wait complete. 2024-05-09T08:39:23.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.739+0000 7fe29463d740 20 client.17403 trim_cache size 0 max 16384 2024-05-09T08:39:23.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.754+0000 7fe29463d740 1 Processor -- start 2024-05-09T08:39:23.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.754+0000 7fe29463d740 1 -- start start 2024-05-09T08:39:23.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.754+0000 7fe29463d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89364230 0x557f8936f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.754+0000 7fe29463d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89364b50 0x557f8936d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.754+0000 7fe29463d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89373730 0x557f8936de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.754+0000 7fe29463d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1af090 con 0x557f89364230 2024-05-09T08:39:23.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.754+0000 7fe29463d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1ac960 con 0x557f89373730 2024-05-09T08:39:23.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.754+0000 7fe29463d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1aa7b0 con 0x557f89364b50 2024-05-09T08:39:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.754+0000 7fe28a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89364230 0x557f8936f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.754+0000 7fe28a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89364230 0x557f8936f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35598/0 (socket says 172.21.15.107:35598) 2024-05-09T08:39:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.754+0000 7fe28a7fc640 1 -- 172.21.15.107:0/2684881948 learned_addr learned my addr 172.21.15.107:0/2684881948 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.754+0000 7fe28affd640 1 --2- 172.21.15.107:0/2684881948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89364b50 0x557f8936d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.754+0000 7fe28a7fc640 1 --2- 172.21.15.107:0/2684881948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89373730 0x557f8936de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.755+0000 7fe28a7fc640 1 -- 172.21.15.107:0/2684881948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89364b50 msgr2=0x557f8936d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.755+0000 7fe28a7fc640 1 --2- 172.21.15.107:0/2684881948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89364b50 0x557f8936d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.755+0000 7fe28a7fc640 1 -- 172.21.15.107:0/2684881948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89373730 msgr2=0x557f8936de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.755+0000 7fe28a7fc640 1 --2- 172.21.15.107:0/2684881948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89373730 0x557f8936de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.755+0000 7fe28a7fc640 1 -- 172.21.15.107:0/2684881948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f89364980 con 0x557f89364230 2024-05-09T08:39:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.755+0000 7fe28a7fc640 1 --2- 172.21.15.107:0/2684881948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89364230 0x557f8936f7d0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fe2800095a0 tx=0x7fe28002a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75f8069e05425ed server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.755+0000 7fe2717fa640 1 -- 172.21.15.107:0/2684881948 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280044040 con 0x557f89364230 2024-05-09T08:39:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.755+0000 7fe2717fa640 1 -- 172.21.15.107:0/2684881948 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2800041e0 con 0x557f89364230 2024-05-09T08:39:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.755+0000 7fe2717fa640 1 -- 172.21.15.107:0/2684881948 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280059040 con 0x557f89364230 2024-05-09T08:39:23.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.756+0000 7fe29463d740 1 -- 172.21.15.107:0/2684881948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89364230 msgr2=0x557f8936f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.756+0000 7fe29463d740 1 --2- 172.21.15.107:0/2684881948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89364230 0x557f8936f7d0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fe2800095a0 tx=0x7fe28002a510 comp rx=0 tx=0).stop 2024-05-09T08:39:23.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.756+0000 7fe29463d740 1 -- 172.21.15.107:0/2684881948 shutdown_connections 2024-05-09T08:39:23.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.756+0000 7fe29463d740 1 --2- 172.21.15.107:0/2684881948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89373730 0x557f8936de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.756+0000 7fe29463d740 1 --2- 172.21.15.107:0/2684881948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89364b50 0x557f8936d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.756+0000 7fe29463d740 1 --2- 172.21.15.107:0/2684881948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89364230 0x557f8936f7d0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.756+0000 7fe29463d740 1 -- 172.21.15.107:0/2684881948 >> 172.21.15.107:0/2684881948 conn(0x557f89360650 msgr2=0x557f89360a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.756+0000 7fe29463d740 1 -- 172.21.15.107:0/2684881948 shutdown_connections 2024-05-09T08:39:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.756+0000 7fe29463d740 1 -- 172.21.15.107:0/2684881948 wait complete. 2024-05-09T08:39:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.756+0000 7fe29463d740 1 -- start start 2024-05-09T08:39:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.756+0000 7fe29463d740 1 Processor -- start 2024-05-09T08:39:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe29463d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89364230 0x557f89389240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe29463d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 0x557f893865d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe29463d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89373730 0x557f89386b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe29463d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1aae20 con 0x557f89373730 2024-05-09T08:39:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe29463d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1ac290 con 0x557f89364b50 2024-05-09T08:39:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe29463d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c1ab4f0 con 0x557f89364230 2024-05-09T08:39:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe29463d740 10 client.0 _ll_register_callbacks cb 0x7fe27c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe28a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89373730 0x557f89386b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe28a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89373730 0x557f89386b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35604/0 (socket says 172.21.15.107:35604) 2024-05-09T08:39:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe28a7fc640 1 -- 172.21.15.107:0/2233573296 learned_addr learned my addr 172.21.15.107:0/2233573296 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe28affd640 1 --2- 172.21.15.107:0/2233573296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 0x557f893865d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe28a7fc640 1 --2- 172.21.15.107:0/2233573296 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89364230 0x557f89389240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe28affd640 1 -- 172.21.15.107:0/2233573296 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89364230 msgr2=0x557f89389240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe28affd640 1 --2- 172.21.15.107:0/2233573296 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89364230 0x557f89389240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe28affd640 1 -- 172.21.15.107:0/2233573296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89373730 msgr2=0x557f89386b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe28affd640 1 --2- 172.21.15.107:0/2233573296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89373730 0x557f89386b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe28affd640 1 -- 172.21.15.107:0/2233573296 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f89387360 con 0x557f89364b50 2024-05-09T08:39:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.757+0000 7fe28a7fc640 1 --2- 172.21.15.107:0/2233573296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89373730 0x557f89386b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.758+0000 7fe28affd640 1 --2- 172.21.15.107:0/2233573296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 0x557f893865d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x557f89376670 tx=0x7fe27c0108c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3574c98bb6e5a09a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.758+0000 7fe26affd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:39:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.758+0000 7fe26affd640 1 -- 172.21.15.107:0/2233573296 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c020300 con 0x557f89364b50 2024-05-09T08:39:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.758+0000 7fe29463d740 1 -- 172.21.15.107:0/2233573296 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557f892d16c0 con 0x557f89364b50 2024-05-09T08:39:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.758+0000 7fe26affd640 1 -- 172.21.15.107:0/2233573296 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe27c0204a0 con 0x557f89364b50 2024-05-09T08:39:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.758+0000 7fe26affd640 1 -- 172.21.15.107:0/2233573296 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe27c02f050 con 0x557f89364b50 2024-05-09T08:39:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.758+0000 7fe29463d740 10 client.26501 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.758+0000 7fe29463d740 1 -- 172.21.15.107:0/2233573296 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557f893e9cc0 con 0x557f89364b50 2024-05-09T08:39:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.758+0000 7fe26affd640 1 -- 172.21.15.107:0/2233573296 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x557f892d16c0 con 0x557f89364b50 2024-05-09T08:39:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.758+0000 7fe26affd640 1 -- 172.21.15.107:0/2233573296 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe27c01ec70 con 0x557f89364b50 2024-05-09T08:39:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.758+0000 7fe29463d740 10 client.26501 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.758+0000 7fe29463d740 10 client.26501 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:39:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.758+0000 7fe29463d740 20 client.26501 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.758+0000 7fe29463d740 1 --2- 172.21.15.107:0/2233573296 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x557f893ea3e0 0x557f893ec800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.758+0000 7fe29463d740 4 client.26501 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:39:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.758+0000 7fe29463d740 1 -- 172.21.15.107:0/2233573296 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557f8924e820 con 0x557f893ea3e0 2024-05-09T08:39:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe28a7fc640 1 --2- 172.21.15.107:0/2233573296 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x557f893ea3e0 0x557f893ec800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe26affd640 10 client.26501 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:39:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe28a7fc640 1 --2- 172.21.15.107:0/2233573296 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x557f893ea3e0 0x557f893ec800 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9f8d5c12f4fcc5b2 server_cookie=272cb2d83270e5eb in_seq=0 out_seq=0 2024-05-09T08:39:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe26affd640 1 -- 172.21.15.107:0/2233573296 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557f8924e820 con 0x557f893ea3e0 2024-05-09T08:39:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe26affd640 10 client.26501 handle_command_reply: tid=0 2024-05-09T08:39:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe29463d740 1 client.26501 shutdown 2024-05-09T08:39:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe29463d740 1 -- 172.21.15.107:0/2233573296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 msgr2=0x557f893865d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe29463d740 1 --2- 172.21.15.107:0/2233573296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 0x557f893865d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x557f89376670 tx=0x7fe27c0108c0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe29463d740 1 -- 172.21.15.107:0/2233573296 shutdown_connections 2024-05-09T08:39:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe29463d740 1 --2- 172.21.15.107:0/2233573296 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x557f893ea3e0 0x557f893ec800 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe29463d740 1 --2- 172.21.15.107:0/2233573296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557f89373730 0x557f89386b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe29463d740 1 --2- 172.21.15.107:0/2233573296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557f89364b50 0x557f893865d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe29463d740 1 --2- 172.21.15.107:0/2233573296 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557f89364230 0x557f89389240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe26affd640 0 client.26501 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:39:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe29463d740 1 -- 172.21.15.107:0/2233573296 >> 172.21.15.107:0/2233573296 conn(0x557f89360650 msgr2=0x557f8935ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe29463d740 1 -- 172.21.15.107:0/2233573296 shutdown_connections 2024-05-09T08:39:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.759+0000 7fe29463d740 1 -- 172.21.15.107:0/2233573296 wait complete. 2024-05-09T08:39:23.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.760+0000 7fe29463d740 20 client.26501 trim_cache size 0 max 16384 2024-05-09T08:39:23.765 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:39:23.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.760+0000 7fe291932640 1 -- 172.21.15.107:0/3095663115 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2740a2d00 msgr2=0x7fe2740a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.760+0000 7fe291932640 1 --2- 172.21.15.107:0/3095663115 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2740a2d00 0x7fe2740a51c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe28409b410 tx=0x7fe284004280 comp rx=0 tx=0).stop 2024-05-09T08:39:23.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.760+0000 7fe291932640 1 -- 172.21.15.107:0/3095663115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe28c10bdf0 msgr2=0x7fe28c19c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.760+0000 7fe291932640 1 --2- 172.21.15.107:0/3095663115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe28c10bdf0 0x7fe28c19c950 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fe284009fd0 tx=0x7fe28400b260 comp rx=0 tx=0).stop 2024-05-09T08:39:23.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.760+0000 7fe291932640 1 -- 172.21.15.107:0/3095663115 shutdown_connections 2024-05-09T08:39:23.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.760+0000 7fe291932640 1 --2- 172.21.15.107:0/3095663115 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2740a2d00 0x7fe2740a51c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.760+0000 7fe291932640 1 --2- 172.21.15.107:0/3095663115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe28c10bdf0 0x7fe28c19c950 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.760+0000 7fe291932640 1 --2- 172.21.15.107:0/3095663115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe28c1033d0 0x7fe28c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.760+0000 7fe291932640 1 --2- 172.21.15.107:0/3095663115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe28c102ab0 0x7fe28c198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.760+0000 7fe291932640 1 -- 172.21.15.107:0/3095663115 >> 172.21.15.107:0/3095663115 conn(0x7fe28c077830 msgr2=0x7fe28c0fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.760+0000 7fe291932640 1 -- 172.21.15.107:0/3095663115 shutdown_connections 2024-05-09T08:39:23.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.760+0000 7fe291932640 1 -- 172.21.15.107:0/3095663115 wait complete. 2024-05-09T08:39:23.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:39:23.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.894+0000 7fbe2d763640 1 -- 172.21.15.107:0/3587733825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe28102ab0 msgr2=0x7fbe28102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.894+0000 7fbe2d763640 1 --2- 172.21.15.107:0/3587733825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe28102ab0 0x7fbe28102e90 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009900 tx=0x7fbe1802a510 comp rx=0 tx=0).stop 2024-05-09T08:39:23.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.894+0000 7fbe2d763640 1 -- 172.21.15.107:0/3587733825 shutdown_connections 2024-05-09T08:39:23.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.894+0000 7fbe2d763640 1 --2- 172.21.15.107:0/3587733825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe2810bdf0 0x7fbe2810e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.894+0000 7fbe2d763640 1 --2- 172.21.15.107:0/3587733825 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe281033d0 0x7fbe2810b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.894+0000 7fbe2d763640 1 --2- 172.21.15.107:0/3587733825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe28102ab0 0x7fbe28102e90 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.894+0000 7fbe2d763640 1 -- 172.21.15.107:0/3587733825 >> 172.21.15.107:0/3587733825 conn(0x7fbe28077830 msgr2=0x7fbe28077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.894+0000 7fbe2d763640 1 -- 172.21.15.107:0/3587733825 shutdown_connections 2024-05-09T08:39:23.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.894+0000 7fbe2d763640 1 -- 172.21.15.107:0/3587733825 wait complete. 2024-05-09T08:39:23.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe2d763640 1 Processor -- start 2024-05-09T08:39:23.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe2d763640 1 -- start start 2024-05-09T08:39:23.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe2d763640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe28102ab0 0x7fbe28198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe2d763640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe281033d0 0x7fbe28198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe2d763640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe2810bdf0 0x7fbe2819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe2d763640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbe28111ae0 con 0x7fbe28102ab0 2024-05-09T08:39:23.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe2d763640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbe28111960 con 0x7fbe281033d0 2024-05-09T08:39:23.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe2d763640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbe28111c60 con 0x7fbe2810bdf0 2024-05-09T08:39:23.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe26ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe28102ab0 0x7fbe28198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe26ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe28102ab0 0x7fbe28198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35616/0 (socket says 172.21.15.107:35616) 2024-05-09T08:39:23.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe26ffd640 1 -- 172.21.15.107:0/2972924900 learned_addr learned my addr 172.21.15.107:0/2972924900 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:23.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe1ffff640 1 --2- 172.21.15.107:0/2972924900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe281033d0 0x7fbe28198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe277fe640 1 --2- 172.21.15.107:0/2972924900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe2810bdf0 0x7fbe2819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe26ffd640 1 -- 172.21.15.107:0/2972924900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe2810bdf0 msgr2=0x7fbe2819c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe26ffd640 1 --2- 172.21.15.107:0/2972924900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe2810bdf0 0x7fbe2819c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe26ffd640 1 -- 172.21.15.107:0/2972924900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe281033d0 msgr2=0x7fbe28198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe26ffd640 1 --2- 172.21.15.107:0/2972924900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe281033d0 0x7fbe28198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe26ffd640 1 -- 172.21.15.107:0/2972924900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2819d030 con 0x7fbe28102ab0 2024-05-09T08:39:23.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe277fe640 1 --2- 172.21.15.107:0/2972924900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe2810bdf0 0x7fbe2819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:23.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.895+0000 7fbe26ffd640 1 --2- 172.21.15.107:0/2972924900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe28102ab0 0x7fbe28198040 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fbe18006010 tx=0x7fbe18004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.896+0000 7fbe24ff9640 1 -- 172.21.15.107:0/2972924900 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1803f030 con 0x7fbe28102ab0 2024-05-09T08:39:23.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.896+0000 7fbe24ff9640 1 -- 172.21.15.107:0/2972924900 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe180035f0 con 0x7fbe28102ab0 2024-05-09T08:39:23.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.896+0000 7fbe24ff9640 1 -- 172.21.15.107:0/2972924900 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe18035070 con 0x7fbe28102ab0 2024-05-09T08:39:23.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.896+0000 7fbe2d763640 1 -- 172.21.15.107:0/2972924900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe2819d2c0 con 0x7fbe28102ab0 2024-05-09T08:39:23.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.896+0000 7fbe2d763640 1 -- 172.21.15.107:0/2972924900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe281a4bb0 con 0x7fbe28102ab0 2024-05-09T08:39:23.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.899+0000 7fbe3047b740 1 Processor -- start 2024-05-09T08:39:23.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.899+0000 7fbe3047b740 1 -- start start 2024-05-09T08:39:23.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.899+0000 7fbe3047b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9127740 0x5643d9127b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.899+0000 7fbe3047b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9128060 0x5643d91321f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.899+0000 7fbe24ff9640 1 -- 172.21.15.107:0/2972924900 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fbe18003b10 con 0x7fbe28102ab0 2024-05-09T08:39:23.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.899+0000 7fbe3047b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9132730 0x5643d9134b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.899+0000 7fbe3047b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5643d9088c40 con 0x5643d9127740 2024-05-09T08:39:23.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.899+0000 7fbe3047b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5643d9030580 con 0x5643d9132730 2024-05-09T08:39:23.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.899+0000 7fbe3047b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5643d9084c00 con 0x5643d9128060 2024-05-09T08:39:23.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.899+0000 7fbe277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9127740 0x5643d9127b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.899+0000 7fbe26ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9128060 0x5643d91321f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.899+0000 7fbe277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9127740 0x5643d9127b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35618/0 (socket says 172.21.15.107:35618) 2024-05-09T08:39:23.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.899+0000 7fbe277fe640 1 -- 172.21.15.107:0/1160123747 learned_addr learned my addr 172.21.15.107:0/1160123747 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:23.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.899+0000 7fbe1ffff640 1 --2- 172.21.15.107:0/1160123747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9132730 0x5643d9134b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.900+0000 7fbe277fe640 1 -- 172.21.15.107:0/1160123747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9128060 msgr2=0x5643d91321f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.900+0000 7fbe277fe640 1 --2- 172.21.15.107:0/1160123747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9128060 0x5643d91321f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.900+0000 7fbe277fe640 1 -- 172.21.15.107:0/1160123747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9132730 msgr2=0x5643d9134b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.900+0000 7fbe277fe640 1 --2- 172.21.15.107:0/1160123747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9132730 0x5643d9134b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.900+0000 7fbe277fe640 1 -- 172.21.15.107:0/1160123747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643d8d9c3f0 con 0x5643d9127740 2024-05-09T08:39:23.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.900+0000 7fbe26ffd640 1 --2- 172.21.15.107:0/1160123747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9128060 0x5643d91321f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:23.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.900+0000 7fbe1ffff640 1 --2- 172.21.15.107:0/1160123747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9132730 0x5643d9134b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:23.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.900+0000 7fbe277fe640 1 --2- 172.21.15.107:0/1160123747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9127740 0x5643d9127b20 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fbe20008e60 tx=0x7fbe2000d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd821ac99b657cf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.900+0000 7fbe24ff9640 1 --2- 172.21.15.107:0/2972924900 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbe140a2d00 0x7fbe140a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.900+0000 7fbe1d7fa640 1 -- 172.21.15.107:0/1160123747 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2000f830 con 0x5643d9127740 2024-05-09T08:39:23.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.900+0000 7fbe1d7fa640 1 -- 172.21.15.107:0/1160123747 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe20018480 con 0x5643d9127740 2024-05-09T08:39:23.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.900+0000 7fbe1d7fa640 1 -- 172.21.15.107:0/1160123747 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2000e5d0 con 0x5643d9127740 2024-05-09T08:39:23.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.900+0000 7fbe26ffd640 1 --2- 172.21.15.107:0/2972924900 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbe140a2d00 0x7fbe140a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.900+0000 7fbe24ff9640 1 -- 172.21.15.107:0/2972924900 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fbe180f0300 con 0x7fbe28102ab0 2024-05-09T08:39:23.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.901+0000 7fbe3047b740 1 -- 172.21.15.107:0/1160123747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9127740 msgr2=0x5643d9127b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.901+0000 7fbe3047b740 1 --2- 172.21.15.107:0/1160123747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9127740 0x5643d9127b20 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fbe20008e60 tx=0x7fbe2000d710 comp rx=0 tx=0).stop 2024-05-09T08:39:23.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.901+0000 7fbe26ffd640 1 --2- 172.21.15.107:0/2972924900 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbe140a2d00 0x7fbe140a51c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbe180ed7a0 tx=0x7fbe180f3e50 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.901+0000 7fbe3047b740 1 -- 172.21.15.107:0/1160123747 shutdown_connections 2024-05-09T08:39:23.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.901+0000 7fbe3047b740 1 --2- 172.21.15.107:0/1160123747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9132730 0x5643d9134b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.901+0000 7fbe3047b740 1 --2- 172.21.15.107:0/1160123747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9128060 0x5643d91321f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.901+0000 7fbe3047b740 1 --2- 172.21.15.107:0/1160123747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9127740 0x5643d9127b20 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.901+0000 7fbe3047b740 1 -- 172.21.15.107:0/1160123747 >> 172.21.15.107:0/1160123747 conn(0x5643d9123a60 msgr2=0x5643d9123e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.901+0000 7fbe3047b740 1 -- 172.21.15.107:0/1160123747 shutdown_connections 2024-05-09T08:39:23.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.901+0000 7fbe3047b740 1 -- 172.21.15.107:0/1160123747 wait complete. 2024-05-09T08:39:23.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe3047b740 1 -- start start 2024-05-09T08:39:23.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe3047b740 1 Processor -- start 2024-05-09T08:39:23.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe3047b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9127740 0x5643d912f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe3047b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9128060 0x5643d912fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe3047b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9132730 0x5643d9130140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe3047b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5643d9135520 con 0x5643d9127740 2024-05-09T08:39:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe3047b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5643d9043c40 con 0x5643d9132730 2024-05-09T08:39:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe3047b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5643d91356a0 con 0x5643d9128060 2024-05-09T08:39:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe3047b740 10 client.0 _ll_register_callbacks cb 0x5643d8d9bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9127740 0x5643d912f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9127740 0x5643d912f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35630/0 (socket says 172.21.15.107:35630) 2024-05-09T08:39:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe277fe640 1 -- 172.21.15.107:0/3946208038 learned_addr learned my addr 172.21.15.107:0/3946208038 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe1ffff640 1 --2- 172.21.15.107:0/3946208038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9128060 0x5643d912fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe277fe640 1 --2- 172.21.15.107:0/3946208038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9132730 0x5643d9130140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe277fe640 1 -- 172.21.15.107:0/3946208038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9128060 msgr2=0x5643d912fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe277fe640 1 --2- 172.21.15.107:0/3946208038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9128060 0x5643d912fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe277fe640 1 -- 172.21.15.107:0/3946208038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9132730 msgr2=0x5643d9130140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe277fe640 1 --2- 172.21.15.107:0/3946208038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9132730 0x5643d9130140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.902+0000 7fbe277fe640 1 -- 172.21.15.107:0/3946208038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643d8e667d0 con 0x5643d9127740 2024-05-09T08:39:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.903+0000 7fbe277fe640 1 --2- 172.21.15.107:0/3946208038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9127740 0x5643d912f6c0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fbe20018b70 tx=0x7fbe20018ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a918f020a42afaa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.903+0000 7fbe0e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.903+0000 7fbe0e7fc640 1 -- 172.21.15.107:0/3946208038 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe20004e50 con 0x5643d9127740 2024-05-09T08:39:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.903+0000 7fbe0e7fc640 1 -- 172.21.15.107:0/3946208038 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fbe2000f630 con 0x5643d9127740 2024-05-09T08:39:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.903+0000 7fbe3047b740 1 -- 172.21.15.107:0/3946208038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe20018480 con 0x5643d9127740 2024-05-09T08:39:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.903+0000 7fbe0e7fc640 1 -- 172.21.15.107:0/3946208038 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbe2000f830 con 0x5643d9127740 2024-05-09T08:39:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.903+0000 7fbe3047b740 10 client.17421 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.903+0000 7fbe3047b740 1 -- 172.21.15.107:0/3946208038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5643d9141720 con 0x5643d9127740 2024-05-09T08:39:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.903+0000 7fbe0e7fc640 1 -- 172.21.15.107:0/3946208038 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe20018480 con 0x5643d9127740 2024-05-09T08:39:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.903+0000 7fbe0e7fc640 1 -- 172.21.15.107:0/3946208038 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fbe20026020 con 0x5643d9127740 2024-05-09T08:39:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.903+0000 7fbe3047b740 10 client.17421 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:23.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.903+0000 7fbe3047b740 10 client.17421 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:39:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.903+0000 7fbe3047b740 20 client.17421 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.903+0000 7fbe3047b740 1 --2- 172.21.15.107:0/3946208038 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5643d9142b80 0x5643d9144f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.903+0000 7fbe3047b740 4 client.17421 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.903+0000 7fbe3047b740 1 -- 172.21.15.107:0/3946208038 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643d9096370 con 0x5643d9142b80 2024-05-09T08:39:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.904+0000 7fbe1ffff640 1 --2- 172.21.15.107:0/3946208038 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5643d9142b80 0x5643d9144f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.904+0000 7fbe0e7fc640 10 client.17421 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:39:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.904+0000 7fbe1ffff640 1 --2- 172.21.15.107:0/3946208038 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5643d9142b80 0x5643d9144f70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f1834dce00e4a6 server_cookie=a54d48e9d4a946be in_seq=0 out_seq=0 2024-05-09T08:39:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.905+0000 7fbe0e7fc640 1 -- 172.21.15.107:0/3946208038 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5643d9096370 con 0x5643d9142b80 2024-05-09T08:39:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.905+0000 7fbe0e7fc640 10 client.17421 handle_command_reply: tid=0 2024-05-09T08:39:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.905+0000 7fbe3047b740 1 client.17421 shutdown 2024-05-09T08:39:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.905+0000 7fbe3047b740 1 -- 172.21.15.107:0/3946208038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9127740 msgr2=0x5643d912f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.905+0000 7fbe3047b740 1 --2- 172.21.15.107:0/3946208038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9127740 0x5643d912f6c0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fbe20018b70 tx=0x7fbe20018ec0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.905+0000 7fbe3047b740 1 -- 172.21.15.107:0/3946208038 shutdown_connections 2024-05-09T08:39:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.905+0000 7fbe3047b740 1 --2- 172.21.15.107:0/3946208038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9132730 0x5643d9130140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.905+0000 7fbe3047b740 1 --2- 172.21.15.107:0/3946208038 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5643d9142b80 0x5643d9144f70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.905+0000 7fbe3047b740 1 --2- 172.21.15.107:0/3946208038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9128060 0x5643d912fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.905+0000 7fbe3047b740 1 --2- 172.21.15.107:0/3946208038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9127740 0x5643d912f6c0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.905+0000 7fbe3047b740 1 -- 172.21.15.107:0/3946208038 >> 172.21.15.107:0/3946208038 conn(0x5643d9123a60 msgr2=0x5643d9129860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.905+0000 7fbe0e7fc640 0 client.17421 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:39:23.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.905+0000 7fbe3047b740 1 -- 172.21.15.107:0/3946208038 shutdown_connections 2024-05-09T08:39:23.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.905+0000 7fbe3047b740 1 -- 172.21.15.107:0/3946208038 wait complete. 2024-05-09T08:39:23.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.905+0000 7fbe3047b740 20 client.17421 trim_cache size 0 max 16384 2024-05-09T08:39:23.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe3047b740 1 Processor -- start 2024-05-09T08:39:23.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe3047b740 1 -- start start 2024-05-09T08:39:23.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe3047b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9127740 0x5643d9131df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe3047b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9128060 0x5643d913e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe3047b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9132730 0x5643d912e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe3047b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbe281af1f0 con 0x5643d9128060 2024-05-09T08:39:23.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe3047b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbe281acac0 con 0x5643d9127740 2024-05-09T08:39:23.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe3047b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbe281aa910 con 0x5643d9132730 2024-05-09T08:39:23.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe1ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9128060 0x5643d913e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe1ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9128060 0x5643d913e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35634/0 (socket says 172.21.15.107:35634) 2024-05-09T08:39:23.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe1ffff640 1 -- 172.21.15.107:0/4143754038 learned_addr learned my addr 172.21.15.107:0/4143754038 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:23.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe277fe640 1 --2- 172.21.15.107:0/4143754038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9132730 0x5643d912e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe1ffff640 1 -- 172.21.15.107:0/4143754038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9132730 msgr2=0x5643d912e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe1ffff640 1 --2- 172.21.15.107:0/4143754038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9132730 0x5643d912e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe1ffff640 1 -- 172.21.15.107:0/4143754038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9127740 msgr2=0x5643d9131df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe1ffff640 1 --2- 172.21.15.107:0/4143754038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9127740 0x5643d9131df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.919+0000 7fbe1ffff640 1 -- 172.21.15.107:0/4143754038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643d911af30 con 0x5643d9128060 2024-05-09T08:39:23.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.920+0000 7fbe1ffff640 1 --2- 172.21.15.107:0/4143754038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9128060 0x5643d913e3e0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x5643d9124bf0 tx=0x7fbe10009110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=900cec234094cc11 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.920+0000 7fbe1cff9640 1 -- 172.21.15.107:0/4143754038 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe10024070 con 0x5643d9128060 2024-05-09T08:39:23.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.920+0000 7fbe1cff9640 1 -- 172.21.15.107:0/4143754038 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe10015dc0 con 0x5643d9128060 2024-05-09T08:39:23.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.920+0000 7fbe1cff9640 1 -- 172.21.15.107:0/4143754038 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1001f720 con 0x5643d9128060 2024-05-09T08:39:23.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 -- 172.21.15.107:0/4143754038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9128060 msgr2=0x5643d913e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 --2- 172.21.15.107:0/4143754038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9128060 0x5643d913e3e0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x5643d9124bf0 tx=0x7fbe10009110 comp rx=0 tx=0).stop 2024-05-09T08:39:23.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 -- 172.21.15.107:0/4143754038 shutdown_connections 2024-05-09T08:39:23.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 --2- 172.21.15.107:0/4143754038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9132730 0x5643d912e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 --2- 172.21.15.107:0/4143754038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9128060 0x5643d913e3e0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 --2- 172.21.15.107:0/4143754038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9127740 0x5643d9131df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 -- 172.21.15.107:0/4143754038 >> 172.21.15.107:0/4143754038 conn(0x5643d9123a60 msgr2=0x5643d9128fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 -- 172.21.15.107:0/4143754038 shutdown_connections 2024-05-09T08:39:23.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 -- 172.21.15.107:0/4143754038 wait complete. 2024-05-09T08:39:23.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 -- start start 2024-05-09T08:39:23.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 Processor -- start 2024-05-09T08:39:23.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9127740 0x5643d913ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9128060 0x5643d9135f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9132730 0x5643d9136470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbe281aaf80 con 0x5643d9132730 2024-05-09T08:39:23.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbe281ac3f0 con 0x5643d9128060 2024-05-09T08:39:23.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbe281ab650 con 0x5643d9127740 2024-05-09T08:39:23.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe3047b740 10 client.0 _ll_register_callbacks cb 0x7fbe100093c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:23.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9132730 0x5643d9136470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9132730 0x5643d9136470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35638/0 (socket says 172.21.15.107:35638) 2024-05-09T08:39:23.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.921+0000 7fbe277fe640 1 -- 172.21.15.107:0/1435166374 learned_addr learned my addr 172.21.15.107:0/1435166374 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:23.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.922+0000 7fbe277fe640 1 --2- 172.21.15.107:0/1435166374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9127740 0x5643d913ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.922+0000 7fbe1ffff640 1 --2- 172.21.15.107:0/1435166374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9128060 0x5643d9135f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.922+0000 7fbe277fe640 1 -- 172.21.15.107:0/1435166374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9127740 msgr2=0x5643d913ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.922+0000 7fbe277fe640 1 --2- 172.21.15.107:0/1435166374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9127740 0x5643d913ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.922+0000 7fbe277fe640 1 -- 172.21.15.107:0/1435166374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9128060 msgr2=0x5643d9135f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:23.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.922+0000 7fbe277fe640 1 --2- 172.21.15.107:0/1435166374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9128060 0x5643d9135f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.922+0000 7fbe277fe640 1 -- 172.21.15.107:0/1435166374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643d9136bb0 con 0x5643d9132730 2024-05-09T08:39:23.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.922+0000 7fbe277fe640 1 --2- 172.21.15.107:0/1435166374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9132730 0x5643d9136470 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fbe20018cd0 tx=0x7fbe2000d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48cda1fec5576057 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:23.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.922+0000 7fbe0e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:23.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.922+0000 7fbe0e7fc640 1 -- 172.21.15.107:0/1435166374 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe20018480 con 0x5643d9132730 2024-05-09T08:39:23.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.922+0000 7fbe0e7fc640 1 -- 172.21.15.107:0/1435166374 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fbe200286a0 con 0x5643d9132730 2024-05-09T08:39:23.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.922+0000 7fbe3047b740 1 -- 172.21.15.107:0/1435166374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe10015dc0 con 0x5643d9132730 2024-05-09T08:39:23.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.922+0000 7fbe0e7fc640 1 -- 172.21.15.107:0/1435166374 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbe20042050 con 0x5643d9132730 2024-05-09T08:39:23.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.922+0000 7fbe3047b740 10 client.17427 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:23.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.922+0000 7fbe3047b740 1 -- 172.21.15.107:0/1435166374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5643d91a8ce0 con 0x5643d9132730 2024-05-09T08:39:23.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.922+0000 7fbe0e7fc640 1 -- 172.21.15.107:0/1435166374 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe10015dc0 con 0x5643d9132730 2024-05-09T08:39:23.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.923+0000 7fbe0e7fc640 1 -- 172.21.15.107:0/1435166374 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fbe20026070 con 0x5643d9132730 2024-05-09T08:39:23.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.923+0000 7fbe3047b740 10 client.17427 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:23.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.923+0000 7fbe3047b740 10 client.17427 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:39:23.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.923+0000 7fbe3047b740 20 client.17427 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:23.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.923+0000 7fbe3047b740 1 --2- 172.21.15.107:0/1435166374 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5643d91a9320 0x5643d91ab740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:23.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.923+0000 7fbe3047b740 4 client.17427 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:39:23.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.923+0000 7fbe3047b740 1 -- 172.21.15.107:0/1435166374 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5643d90906c0 con 0x5643d91a9320 2024-05-09T08:39:23.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.923+0000 7fbe1ffff640 1 --2- 172.21.15.107:0/1435166374 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5643d91a9320 0x5643d91ab740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:23.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.923+0000 7fbe1ffff640 1 --2- 172.21.15.107:0/1435166374 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5643d91a9320 0x5643d91ab740 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7f959db6f0f60703 server_cookie=3a300db968eaa31b in_seq=0 out_seq=0 2024-05-09T08:39:23.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.923+0000 7fbe0e7fc640 10 client.17427 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:39:23.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe0e7fc640 1 -- 172.21.15.107:0/1435166374 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5643d90906c0 con 0x5643d91a9320 2024-05-09T08:39:23.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe0e7fc640 10 client.17427 handle_command_reply: tid=0 2024-05-09T08:39:23.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe3047b740 1 client.17427 shutdown 2024-05-09T08:39:23.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe3047b740 1 -- 172.21.15.107:0/1435166374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9132730 msgr2=0x5643d9136470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe3047b740 1 --2- 172.21.15.107:0/1435166374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9132730 0x5643d9136470 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fbe20018cd0 tx=0x7fbe2000d710 comp rx=0 tx=0).stop 2024-05-09T08:39:23.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe3047b740 1 -- 172.21.15.107:0/1435166374 shutdown_connections 2024-05-09T08:39:23.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe3047b740 1 --2- 172.21.15.107:0/1435166374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643d9132730 0x5643d9136470 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe3047b740 1 --2- 172.21.15.107:0/1435166374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643d9128060 0x5643d9135f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe3047b740 1 --2- 172.21.15.107:0/1435166374 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5643d91a9320 0x5643d91ab740 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe3047b740 1 --2- 172.21.15.107:0/1435166374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643d9127740 0x5643d913ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe0e7fc640 0 client.17427 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:39:23.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe3047b740 1 -- 172.21.15.107:0/1435166374 >> 172.21.15.107:0/1435166374 conn(0x5643d9123a60 msgr2=0x5643d911df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe3047b740 1 -- 172.21.15.107:0/1435166374 shutdown_connections 2024-05-09T08:39:23.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe3047b740 1 -- 172.21.15.107:0/1435166374 wait complete. 2024-05-09T08:39:23.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe3047b740 20 client.17427 trim_cache size 0 max 16384 2024-05-09T08:39:23.937 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:39:23.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe2d763640 1 -- 172.21.15.107:0/2972924900 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbe140a2d00 msgr2=0x7fbe140a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe2d763640 1 --2- 172.21.15.107:0/2972924900 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbe140a2d00 0x7fbe140a51c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbe180ed7a0 tx=0x7fbe180f3e50 comp rx=0 tx=0).stop 2024-05-09T08:39:23.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe2d763640 1 -- 172.21.15.107:0/2972924900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe28102ab0 msgr2=0x7fbe28198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:23.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe2d763640 1 --2- 172.21.15.107:0/2972924900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe28102ab0 0x7fbe28198040 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fbe18006010 tx=0x7fbe18004270 comp rx=0 tx=0).stop 2024-05-09T08:39:23.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe2d763640 1 -- 172.21.15.107:0/2972924900 shutdown_connections 2024-05-09T08:39:23.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe2d763640 1 --2- 172.21.15.107:0/2972924900 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbe140a2d00 0x7fbe140a51c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe2d763640 1 --2- 172.21.15.107:0/2972924900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe2810bdf0 0x7fbe2819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe2d763640 1 --2- 172.21.15.107:0/2972924900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe281033d0 0x7fbe28198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe2d763640 1 --2- 172.21.15.107:0/2972924900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe28102ab0 0x7fbe28198040 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:23.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.924+0000 7fbe2d763640 1 -- 172.21.15.107:0/2972924900 >> 172.21.15.107:0/2972924900 conn(0x7fbe28077830 msgr2=0x7fbe280fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:23.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.925+0000 7fbe2d763640 1 -- 172.21.15.107:0/2972924900 shutdown_connections 2024-05-09T08:39:23.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:23.925+0000 7fbe2d763640 1 -- 172.21.15.107:0/2972924900 wait complete. 2024-05-09T08:39:23.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:39:24.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.059+0000 7fe234168640 1 -- 172.21.15.107:0/2364983024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe22c106710 msgr2=0x7fe22c106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:24.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.059+0000 7fe234168640 1 --2- 172.21.15.107:0/2364983024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe22c106710 0x7fe22c106af0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fe228009930 tx=0x7fe22802a420 comp rx=0 tx=0).stop 2024-05-09T08:39:24.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.059+0000 7fe234168640 1 -- 172.21.15.107:0/2364983024 shutdown_connections 2024-05-09T08:39:24.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.059+0000 7fe234168640 1 --2- 172.21.15.107:0/2364983024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe22c1010c0 0x7fe22c10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.059+0000 7fe234168640 1 --2- 172.21.15.107:0/2364983024 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe22c100700 0x7fe22c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.059+0000 7fe234168640 1 --2- 172.21.15.107:0/2364983024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe22c106710 0x7fe22c106af0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.059+0000 7fe234168640 1 -- 172.21.15.107:0/2364983024 >> 172.21.15.107:0/2364983024 conn(0x7fe22c077830 msgr2=0x7fe22c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:24.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.059+0000 7fe234168640 1 -- 172.21.15.107:0/2364983024 shutdown_connections 2024-05-09T08:39:24.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.059+0000 7fe234168640 1 -- 172.21.15.107:0/2364983024 wait complete. 2024-05-09T08:39:24.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.060+0000 7fe234168640 1 Processor -- start 2024-05-09T08:39:24.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.060+0000 7fe234168640 1 -- start start 2024-05-09T08:39:24.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.060+0000 7fe234168640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe22c100700 0x7fe22c198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.060+0000 7fe234168640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe22c1010c0 0x7fe22c1993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.060+0000 7fe234168640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe22c106710 0x7fe22c1a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.060+0000 7fe234168640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe22c1109d0 con 0x7fe22c106710 2024-05-09T08:39:24.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.060+0000 7fe234168640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe22c110850 con 0x7fe22c100700 2024-05-09T08:39:24.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.060+0000 7fe234168640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe22c110b50 con 0x7fe22c1010c0 2024-05-09T08:39:24.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.060+0000 7fe232705640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe22c106710 0x7fe22c1a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:24.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.060+0000 7fe232705640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe22c106710 0x7fe22c1a3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35656/0 (socket says 172.21.15.107:35656) 2024-05-09T08:39:24.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.060+0000 7fe232705640 1 -- 172.21.15.107:0/497463176 learned_addr learned my addr 172.21.15.107:0/497463176 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:24.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.060+0000 7fe231703640 1 --2- 172.21.15.107:0/497463176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe22c1010c0 0x7fe22c1993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:24.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.060+0000 7fe231f04640 1 --2- 172.21.15.107:0/497463176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe22c100700 0x7fe22c198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:24.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.061+0000 7fe232705640 1 -- 172.21.15.107:0/497463176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe22c1010c0 msgr2=0x7fe22c1993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:24.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.061+0000 7fe232705640 1 --2- 172.21.15.107:0/497463176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe22c1010c0 0x7fe22c1993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.061+0000 7fe232705640 1 -- 172.21.15.107:0/497463176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe22c100700 msgr2=0x7fe22c198e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:24.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.061+0000 7fe232705640 1 --2- 172.21.15.107:0/497463176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe22c100700 0x7fe22c198e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.061+0000 7fe232705640 1 -- 172.21.15.107:0/497463176 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe22c1a4430 con 0x7fe22c106710 2024-05-09T08:39:24.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.061+0000 7fe232705640 1 --2- 172.21.15.107:0/497463176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe22c106710 0x7fe22c1a3ef0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fe22400eb10 tx=0x7fe22400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:24.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.061+0000 7fe222ffd640 1 -- 172.21.15.107:0/497463176 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe224004b20 con 0x7fe22c106710 2024-05-09T08:39:24.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.061+0000 7fe222ffd640 1 -- 172.21.15.107:0/497463176 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe224004cc0 con 0x7fe22c106710 2024-05-09T08:39:24.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.061+0000 7fe234168640 1 -- 172.21.15.107:0/497463176 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe22c1a4600 con 0x7fe22c106710 2024-05-09T08:39:24.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.061+0000 7fe222ffd640 1 -- 172.21.15.107:0/497463176 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe224010690 con 0x7fe22c106710 2024-05-09T08:39:24.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.061+0000 7fe234168640 1 -- 172.21.15.107:0/497463176 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe22c1a4b00 con 0x7fe22c106710 2024-05-09T08:39:24.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.064+0000 7fe236e72740 1 Processor -- start 2024-05-09T08:39:24.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.064+0000 7fe222ffd640 1 -- 172.21.15.107:0/497463176 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe224005020 con 0x7fe22c106710 2024-05-09T08:39:24.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.064+0000 7fe236e72740 1 -- start start 2024-05-09T08:39:24.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.064+0000 7fe236e72740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a774210 0x55794a7745f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.064+0000 7fe236e72740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774b30 0x55794a7831f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.064+0000 7fe236e72740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a783730 0x55794a785b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.064+0000 7fe236e72740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55794a6d9c40 con 0x55794a774b30 2024-05-09T08:39:24.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.064+0000 7fe236e72740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55794a681580 con 0x55794a783730 2024-05-09T08:39:24.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.064+0000 7fe236e72740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55794a6d5c00 con 0x55794a774210 2024-05-09T08:39:24.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.064+0000 7fe231f04640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774b30 0x55794a7831f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:24.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.064+0000 7fe231f04640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774b30 0x55794a7831f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35658/0 (socket says 172.21.15.107:35658) 2024-05-09T08:39:24.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.064+0000 7fe231f04640 1 -- 172.21.15.107:0/2176829611 learned_addr learned my addr 172.21.15.107:0/2176829611 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:24.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.065+0000 7fe231703640 1 --2- 172.21.15.107:0/2176829611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a783730 0x55794a785b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:24.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.065+0000 7fe222ffd640 1 --2- 172.21.15.107:0/497463176 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2180a2d00 0x7fe2180a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.065+0000 7fe231703640 1 --2- 172.21.15.107:0/2176829611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a774210 0x55794a7745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:24.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.065+0000 7fe231f04640 1 -- 172.21.15.107:0/2176829611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a774210 msgr2=0x55794a7745f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:24.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.065+0000 7fe231f04640 1 --2- 172.21.15.107:0/2176829611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a774210 0x55794a7745f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.065+0000 7fe231f04640 1 -- 172.21.15.107:0/2176829611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a783730 msgr2=0x55794a785b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:24.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.065+0000 7fe231f04640 1 --2- 172.21.15.107:0/2176829611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a783730 0x55794a785b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.065+0000 7fe231f04640 1 -- 172.21.15.107:0/2176829611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55794a3ed3f0 con 0x55794a774b30 2024-05-09T08:39:24.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.065+0000 7fe232705640 1 --2- 172.21.15.107:0/497463176 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2180a2d00 0x7fe2180a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:24.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.065+0000 7fe222ffd640 1 -- 172.21.15.107:0/497463176 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe2240cfcc0 con 0x7fe22c106710 2024-05-09T08:39:24.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.065+0000 7fe231f04640 1 --2- 172.21.15.107:0/2176829611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774b30 0x55794a7831f0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fe22802a940 tx=0x7fe228031ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d33085e8e096811 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:24.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.065+0000 7fe217fff640 1 -- 172.21.15.107:0/2176829611 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe228040070 con 0x55794a774b30 2024-05-09T08:39:24.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.065+0000 7fe232705640 1 --2- 172.21.15.107:0/497463176 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2180a2d00 0x7fe2180a51c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe2240cd6b0 tx=0x7fe224009a70 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:24.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.065+0000 7fe217fff640 1 -- 172.21.15.107:0/2176829611 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe228031e50 con 0x55794a774b30 2024-05-09T08:39:24.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.065+0000 7fe217fff640 1 -- 172.21.15.107:0/2176829611 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe228035040 con 0x55794a774b30 2024-05-09T08:39:24.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.066+0000 7fe236e72740 1 -- 172.21.15.107:0/2176829611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774b30 msgr2=0x55794a7831f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:24.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.066+0000 7fe236e72740 1 --2- 172.21.15.107:0/2176829611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774b30 0x55794a7831f0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fe22802a940 tx=0x7fe228031ba0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.066+0000 7fe236e72740 1 -- 172.21.15.107:0/2176829611 shutdown_connections 2024-05-09T08:39:24.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.066+0000 7fe236e72740 1 --2- 172.21.15.107:0/2176829611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a783730 0x55794a785b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.066+0000 7fe236e72740 1 --2- 172.21.15.107:0/2176829611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774b30 0x55794a7831f0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.066+0000 7fe236e72740 1 --2- 172.21.15.107:0/2176829611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a774210 0x55794a7745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.066+0000 7fe236e72740 1 -- 172.21.15.107:0/2176829611 >> 172.21.15.107:0/2176829611 conn(0x55794a770650 msgr2=0x55794a770a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:24.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.066+0000 7fe236e72740 1 -- 172.21.15.107:0/2176829611 shutdown_connections 2024-05-09T08:39:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.066+0000 7fe236e72740 1 -- 172.21.15.107:0/2176829611 wait complete. 2024-05-09T08:39:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.066+0000 7fe236e72740 1 -- start start 2024-05-09T08:39:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.067+0000 7fe236e72740 1 Processor -- start 2024-05-09T08:39:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.067+0000 7fe236e72740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774210 0x55794a7805e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.067+0000 7fe236e72740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a774b30 0x55794a780b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.067+0000 7fe236e72740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a783730 0x55794a781060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.067+0000 7fe236e72740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55794a786510 con 0x55794a774210 2024-05-09T08:39:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.067+0000 7fe236e72740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55794a694c40 con 0x55794a783730 2024-05-09T08:39:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.067+0000 7fe236e72740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55794a786690 con 0x55794a774b30 2024-05-09T08:39:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.067+0000 7fe236e72740 10 client.0 _ll_register_callbacks cb 0x55794a3ecbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.067+0000 7fe231703640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774210 0x55794a7805e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.067+0000 7fe231703640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774210 0x55794a7805e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35670/0 (socket says 172.21.15.107:35670) 2024-05-09T08:39:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.067+0000 7fe231703640 1 -- 172.21.15.107:0/2302388529 learned_addr learned my addr 172.21.15.107:0/2302388529 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.067+0000 7fe231f04640 1 --2- 172.21.15.107:0/2302388529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a774b30 0x55794a780b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.067+0000 7fe231703640 1 -- 172.21.15.107:0/2302388529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a774b30 msgr2=0x55794a780b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.067+0000 7fe231703640 1 --2- 172.21.15.107:0/2302388529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a774b30 0x55794a780b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.067+0000 7fe231703640 1 -- 172.21.15.107:0/2302388529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a783730 msgr2=0x55794a781060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.067+0000 7fe231703640 1 --2- 172.21.15.107:0/2302388529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a783730 0x55794a781060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe231703640 1 -- 172.21.15.107:0/2302388529 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55794a4b77d0 con 0x55794a774210 2024-05-09T08:39:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe231703640 1 --2- 172.21.15.107:0/2302388529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774210 0x55794a7805e0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fe21c004430 tx=0x7fe21c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad193319ee8969f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe2157fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe2157fa640 1 -- 172.21.15.107:0/2302388529 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c012870 con 0x55794a774210 2024-05-09T08:39:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe2157fa640 1 -- 172.21.15.107:0/2302388529 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe21c012a10 con 0x55794a774210 2024-05-09T08:39:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe236e72740 1 -- 172.21.15.107:0/2302388529 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe228031e50 con 0x55794a774210 2024-05-09T08:39:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe2157fa640 1 -- 172.21.15.107:0/2302388529 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe21c027050 con 0x55794a774210 2024-05-09T08:39:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe236e72740 10 client.17439 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe236e72740 1 -- 172.21.15.107:0/2302388529 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55794a792710 con 0x55794a774210 2024-05-09T08:39:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe2157fa640 1 -- 172.21.15.107:0/2302388529 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe228031e50 con 0x55794a774210 2024-05-09T08:39:24.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe2157fa640 1 -- 172.21.15.107:0/2302388529 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe21c01c020 con 0x55794a774210 2024-05-09T08:39:24.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe236e72740 10 client.17439 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:24.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe236e72740 10 client.17439 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:24.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe236e72740 20 client.17439 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:24.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe236e72740 1 --2- 172.21.15.107:0/2302388529 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55794a793bd0 0x55794a795fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe236e72740 4 client.17439 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:24.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.068+0000 7fe236e72740 1 -- 172.21.15.107:0/2302388529 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55794a6e7370 con 0x55794a793bd0 2024-05-09T08:39:24.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.069+0000 7fe231f04640 1 --2- 172.21.15.107:0/2302388529 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55794a793bd0 0x55794a795fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:24.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.069+0000 7fe231f04640 1 --2- 172.21.15.107:0/2302388529 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55794a793bd0 0x55794a795fe0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fa197ba755c0629 server_cookie=d62f816fb8fb2691 in_seq=0 out_seq=0 2024-05-09T08:39:24.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.069+0000 7fe2157fa640 10 client.17439 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:24.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.070+0000 7fe2157fa640 1 -- 172.21.15.107:0/2302388529 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55794a6e7370 con 0x55794a793bd0 2024-05-09T08:39:24.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.070+0000 7fe2157fa640 10 client.17439 handle_command_reply: tid=0 2024-05-09T08:39:24.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.070+0000 7fe236e72740 1 client.17439 shutdown 2024-05-09T08:39:24.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.070+0000 7fe236e72740 1 -- 172.21.15.107:0/2302388529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774210 msgr2=0x55794a7805e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.070+0000 7fe236e72740 1 --2- 172.21.15.107:0/2302388529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774210 0x55794a7805e0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fe21c004430 tx=0x7fe21c0039c0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.070+0000 7fe236e72740 1 -- 172.21.15.107:0/2302388529 shutdown_connections 2024-05-09T08:39:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.070+0000 7fe236e72740 1 --2- 172.21.15.107:0/2302388529 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55794a793bd0 0x55794a795fe0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.070+0000 7fe236e72740 1 --2- 172.21.15.107:0/2302388529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a783730 0x55794a781060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.070+0000 7fe236e72740 1 --2- 172.21.15.107:0/2302388529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a774b30 0x55794a780b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.070+0000 7fe2157fa640 0 client.17439 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.070+0000 7fe236e72740 1 --2- 172.21.15.107:0/2302388529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774210 0x55794a7805e0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.070+0000 7fe236e72740 1 -- 172.21.15.107:0/2302388529 >> 172.21.15.107:0/2302388529 conn(0x55794a770650 msgr2=0x55794a776400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.070+0000 7fe236e72740 1 -- 172.21.15.107:0/2302388529 shutdown_connections 2024-05-09T08:39:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.070+0000 7fe236e72740 1 -- 172.21.15.107:0/2302388529 wait complete. 2024-05-09T08:39:24.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.070+0000 7fe236e72740 20 client.17439 trim_cache size 0 max 16384 2024-05-09T08:39:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe236e72740 1 Processor -- start 2024-05-09T08:39:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe236e72740 1 -- start start 2024-05-09T08:39:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe236e72740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a774210 0x55794a795fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe236e72740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774b30 0x55794a78f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe236e72740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a783730 0x55794a79b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe236e72740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe22c1af240 con 0x55794a774b30 2024-05-09T08:39:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe236e72740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe22c1acb10 con 0x55794a774210 2024-05-09T08:39:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe236e72740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe22c1aa960 con 0x55794a783730 2024-05-09T08:39:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe231f04640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774b30 0x55794a78f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe231f04640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774b30 0x55794a78f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35682/0 (socket says 172.21.15.107:35682) 2024-05-09T08:39:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe231f04640 1 -- 172.21.15.107:0/2543390400 learned_addr learned my addr 172.21.15.107:0/2543390400 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe231703640 1 --2- 172.21.15.107:0/2543390400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a774210 0x55794a795fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe231f04640 1 -- 172.21.15.107:0/2543390400 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a783730 msgr2=0x55794a79b8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:39:24.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe231f04640 1 --2- 172.21.15.107:0/2543390400 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a783730 0x55794a79b8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe231f04640 1 -- 172.21.15.107:0/2543390400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a774210 msgr2=0x55794a795fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:24.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe231f04640 1 --2- 172.21.15.107:0/2543390400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a774210 0x55794a795fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe231f04640 1 -- 172.21.15.107:0/2543390400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55794a774960 con 0x55794a774b30 2024-05-09T08:39:24.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.086+0000 7fe231703640 1 --2- 172.21.15.107:0/2543390400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a774210 0x55794a795fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:24.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.087+0000 7fe231f04640 1 --2- 172.21.15.107:0/2543390400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774b30 0x55794a78f440 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fe228000c00 tx=0x7fe228047040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e454d13367954e16 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:24.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.087+0000 7fe2177fe640 1 -- 172.21.15.107:0/2543390400 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe228055070 con 0x55794a774b30 2024-05-09T08:39:24.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.087+0000 7fe2177fe640 1 -- 172.21.15.107:0/2543390400 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe228031e50 con 0x55794a774b30 2024-05-09T08:39:24.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.087+0000 7fe2177fe640 1 -- 172.21.15.107:0/2543390400 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe228040040 con 0x55794a774b30 2024-05-09T08:39:24.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.088+0000 7fe236e72740 1 -- 172.21.15.107:0/2543390400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774b30 msgr2=0x55794a78f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:24.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.088+0000 7fe236e72740 1 --2- 172.21.15.107:0/2543390400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774b30 0x55794a78f440 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fe228000c00 tx=0x7fe228047040 comp rx=0 tx=0).stop 2024-05-09T08:39:24.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.088+0000 7fe236e72740 1 -- 172.21.15.107:0/2543390400 shutdown_connections 2024-05-09T08:39:24.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.088+0000 7fe236e72740 1 --2- 172.21.15.107:0/2543390400 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a783730 0x55794a79b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.088+0000 7fe236e72740 1 --2- 172.21.15.107:0/2543390400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a774b30 0x55794a78f440 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.088+0000 7fe236e72740 1 --2- 172.21.15.107:0/2543390400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a774210 0x55794a795fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.088+0000 7fe236e72740 1 -- 172.21.15.107:0/2543390400 >> 172.21.15.107:0/2543390400 conn(0x55794a770650 msgr2=0x55794a76f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.088+0000 7fe236e72740 1 -- 172.21.15.107:0/2543390400 shutdown_connections 2024-05-09T08:39:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.088+0000 7fe236e72740 1 -- 172.21.15.107:0/2543390400 wait complete. 2024-05-09T08:39:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.088+0000 7fe236e72740 1 -- start start 2024-05-09T08:39:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.088+0000 7fe236e72740 1 Processor -- start 2024-05-09T08:39:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe236e72740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a774210 0x55794a787320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe236e72740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a774b30 0x55794a787860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe236e72740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a783730 0x55794a787da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe236e72740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe22c1aafd0 con 0x55794a783730 2024-05-09T08:39:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe236e72740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe22c1ac440 con 0x55794a774b30 2024-05-09T08:39:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe236e72740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe22c1ab6a0 con 0x55794a774210 2024-05-09T08:39:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe236e72740 10 client.0 _ll_register_callbacks cb 0x7fe21c00bd20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe231703640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a783730 0x55794a787da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe231703640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a783730 0x55794a787da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35686/0 (socket says 172.21.15.107:35686) 2024-05-09T08:39:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe231703640 1 -- 172.21.15.107:0/843828950 learned_addr learned my addr 172.21.15.107:0/843828950 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe231f04640 1 --2- 172.21.15.107:0/843828950 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a774b30 0x55794a787860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe231703640 1 -- 172.21.15.107:0/843828950 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a774210 msgr2=0x55794a787320 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:39:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe231703640 1 --2- 172.21.15.107:0/843828950 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a774210 0x55794a787320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe231703640 1 -- 172.21.15.107:0/843828950 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a774b30 msgr2=0x55794a787860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe231703640 1 --2- 172.21.15.107:0/843828950 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a774b30 0x55794a787860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe231703640 1 -- 172.21.15.107:0/843828950 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55794a7f9120 con 0x55794a783730 2024-05-09T08:39:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe231703640 1 --2- 172.21.15.107:0/843828950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a783730 0x55794a787da0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fe21c005d00 tx=0x7fe21c012e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29cbd2d061cc4ff1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.089+0000 7fe2157fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.090+0000 7fe2157fa640 1 -- 172.21.15.107:0/843828950 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c003fb0 con 0x55794a783730 2024-05-09T08:39:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.090+0000 7fe2157fa640 1 -- 172.21.15.107:0/843828950 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe21c00b390 con 0x55794a783730 2024-05-09T08:39:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.090+0000 7fe236e72740 1 -- 172.21.15.107:0/843828950 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe228031e50 con 0x55794a783730 2024-05-09T08:39:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.090+0000 7fe2157fa640 1 -- 172.21.15.107:0/843828950 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe21c039050 con 0x55794a783730 2024-05-09T08:39:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.090+0000 7fe236e72740 10 client.17445 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.090+0000 7fe236e72740 1 -- 172.21.15.107:0/843828950 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55794a7fa3c0 con 0x55794a783730 2024-05-09T08:39:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.090+0000 7fe2157fa640 1 -- 172.21.15.107:0/843828950 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe228031e50 con 0x55794a783730 2024-05-09T08:39:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.090+0000 7fe2157fa640 1 -- 172.21.15.107:0/843828950 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe21c01c040 con 0x55794a783730 2024-05-09T08:39:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.090+0000 7fe236e72740 10 client.17445 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.090+0000 7fe236e72740 10 client.17445 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.090+0000 7fe236e72740 20 client.17445 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.090+0000 7fe236e72740 1 --2- 172.21.15.107:0/843828950 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55794a7fab40 0x55794a7fcf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.090+0000 7fe236e72740 4 client.17445 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:39:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.090+0000 7fe236e72740 1 -- 172.21.15.107:0/843828950 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55794a6e16c0 con 0x55794a7fab40 2024-05-09T08:39:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.092+0000 7fe231f04640 1 --2- 172.21.15.107:0/843828950 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55794a7fab40 0x55794a7fcf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.094+0000 7fe2157fa640 10 client.17445 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.094+0000 7fe231f04640 1 --2- 172.21.15.107:0/843828950 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55794a7fab40 0x55794a7fcf60 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c50ea0acf27653a1 server_cookie=5b9526b7cbf1d19 in_seq=0 out_seq=0 2024-05-09T08:39:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.095+0000 7fe2157fa640 1 -- 172.21.15.107:0/843828950 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55794a6e16c0 con 0x55794a7fab40 2024-05-09T08:39:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.095+0000 7fe2157fa640 10 client.17445 handle_command_reply: tid=0 2024-05-09T08:39:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.095+0000 7fe236e72740 1 client.17445 shutdown 2024-05-09T08:39:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.095+0000 7fe236e72740 1 -- 172.21.15.107:0/843828950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a783730 msgr2=0x55794a787da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.095+0000 7fe236e72740 1 --2- 172.21.15.107:0/843828950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a783730 0x55794a787da0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fe21c005d00 tx=0x7fe21c012e50 comp rx=0 tx=0).stop 2024-05-09T08:39:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.095+0000 7fe236e72740 1 -- 172.21.15.107:0/843828950 shutdown_connections 2024-05-09T08:39:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.095+0000 7fe236e72740 1 --2- 172.21.15.107:0/843828950 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55794a7fab40 0x55794a7fcf60 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.095+0000 7fe236e72740 1 --2- 172.21.15.107:0/843828950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55794a783730 0x55794a787da0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.095+0000 7fe236e72740 1 --2- 172.21.15.107:0/843828950 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55794a774b30 0x55794a787860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.095+0000 7fe2157fa640 0 client.17445 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.095+0000 7fe236e72740 1 --2- 172.21.15.107:0/843828950 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55794a774210 0x55794a787320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.095+0000 7fe236e72740 1 -- 172.21.15.107:0/843828950 >> 172.21.15.107:0/843828950 conn(0x55794a770650 msgr2=0x55794a77ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.095+0000 7fe236e72740 1 -- 172.21.15.107:0/843828950 shutdown_connections 2024-05-09T08:39:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.095+0000 7fe236e72740 1 -- 172.21.15.107:0/843828950 wait complete. 2024-05-09T08:39:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.095+0000 7fe236e72740 20 client.17445 trim_cache size 0 max 16384 2024-05-09T08:39:24.096 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:39:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.096+0000 7fe234168640 1 -- 172.21.15.107:0/497463176 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2180a2d00 msgr2=0x7fe2180a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.096+0000 7fe234168640 1 --2- 172.21.15.107:0/497463176 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2180a2d00 0x7fe2180a51c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe2240cd6b0 tx=0x7fe224009a70 comp rx=0 tx=0).stop 2024-05-09T08:39:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.096+0000 7fe234168640 1 -- 172.21.15.107:0/497463176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe22c106710 msgr2=0x7fe22c1a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.096+0000 7fe234168640 1 --2- 172.21.15.107:0/497463176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe22c106710 0x7fe22c1a3ef0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fe22400eb10 tx=0x7fe22400c2d0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.096+0000 7fe234168640 1 -- 172.21.15.107:0/497463176 shutdown_connections 2024-05-09T08:39:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.096+0000 7fe234168640 1 --2- 172.21.15.107:0/497463176 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2180a2d00 0x7fe2180a51c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.096+0000 7fe234168640 1 --2- 172.21.15.107:0/497463176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe22c106710 0x7fe22c1a3ef0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.096+0000 7fe234168640 1 --2- 172.21.15.107:0/497463176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe22c1010c0 0x7fe22c1993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.096+0000 7fe234168640 1 --2- 172.21.15.107:0/497463176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe22c100700 0x7fe22c198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.096+0000 7fe234168640 1 -- 172.21.15.107:0/497463176 >> 172.21.15.107:0/497463176 conn(0x7fe22c077830 msgr2=0x7fe22c103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.096+0000 7fe234168640 1 -- 172.21.15.107:0/497463176 shutdown_connections 2024-05-09T08:39:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:24.096+0000 7fe234168640 1 -- 172.21.15.107:0/497463176 wait complete. 2024-05-09T08:39:24.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:24 smithi107 ceph-mon[30122]: pgmap v184: 129 pgs: 129 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:24.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:24 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3450068770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:39:24.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:24 smithi157 ceph-mon[33614]: pgmap v184: 129 pgs: 129 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:24.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:24 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3450068770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:39:24.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:24 smithi110 ceph-mon[33622]: pgmap v184: 129 pgs: 129 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:24.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:24 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3450068770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:39:25.108 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:39:25.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:39:25.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:39:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.232+0000 7f209edb0640 1 -- 172.21.15.107:0/2999680734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2098100700 msgr2=0x7f2098100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.232+0000 7f209edb0640 1 --2- 172.21.15.107:0/2999680734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2098100700 0x7f2098100b80 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f2088009930 tx=0x7f208802a420 comp rx=0 tx=0).stop 2024-05-09T08:39:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.232+0000 7f209edb0640 1 -- 172.21.15.107:0/2999680734 shutdown_connections 2024-05-09T08:39:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.232+0000 7f209edb0640 1 --2- 172.21.15.107:0/2999680734 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20981010c0 0x7f209810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.232+0000 7f209edb0640 1 --2- 172.21.15.107:0/2999680734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2098100700 0x7f2098100b80 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.232+0000 7f209edb0640 1 --2- 172.21.15.107:0/2999680734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2098106710 0x7f2098106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.232+0000 7f209edb0640 1 -- 172.21.15.107:0/2999680734 >> 172.21.15.107:0/2999680734 conn(0x7f2098077830 msgr2=0x7f2098077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.232+0000 7f209edb0640 1 -- 172.21.15.107:0/2999680734 shutdown_connections 2024-05-09T08:39:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.232+0000 7f209edb0640 1 -- 172.21.15.107:0/2999680734 wait complete. 2024-05-09T08:39:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.233+0000 7f209edb0640 1 Processor -- start 2024-05-09T08:39:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.233+0000 7f209edb0640 1 -- start start 2024-05-09T08:39:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.233+0000 7f209edb0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2098100700 0x7f2098102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.233+0000 7f209edb0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20981010c0 0x7f2098103200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.233+0000 7f209edb0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2098106710 0x7f2098103740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.233+0000 7f209edb0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2098110990 con 0x7f2098100700 2024-05-09T08:39:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.233+0000 7f209edb0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2098110810 con 0x7f2098106710 2024-05-09T08:39:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.233+0000 7f209edb0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2098110b10 con 0x7f20981010c0 2024-05-09T08:39:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.233+0000 7f209cb4c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2098100700 0x7f2098102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.233+0000 7f209cb4c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2098100700 0x7f2098102cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35702/0 (socket says 172.21.15.107:35702) 2024-05-09T08:39:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.233+0000 7f209cb4c640 1 -- 172.21.15.107:0/444671948 learned_addr learned my addr 172.21.15.107:0/444671948 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:25.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.233+0000 7f208ffff640 1 --2- 172.21.15.107:0/444671948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20981010c0 0x7f2098103200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.233+0000 7f209d34d640 1 --2- 172.21.15.107:0/444671948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2098106710 0x7f2098103740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.234+0000 7f209cb4c640 1 -- 172.21.15.107:0/444671948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20981010c0 msgr2=0x7f2098103200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.234+0000 7f209cb4c640 1 --2- 172.21.15.107:0/444671948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20981010c0 0x7f2098103200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.234+0000 7f209cb4c640 1 -- 172.21.15.107:0/444671948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2098106710 msgr2=0x7f2098103740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.234+0000 7f209cb4c640 1 --2- 172.21.15.107:0/444671948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2098106710 0x7f2098103740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.234+0000 7f209cb4c640 1 -- 172.21.15.107:0/444671948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20981a6580 con 0x7f2098100700 2024-05-09T08:39:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.234+0000 7f209cb4c640 1 --2- 172.21.15.107:0/444671948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2098100700 0x7f2098102cc0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f209400cce0 tx=0x7f209400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.234+0000 7f208dffb640 1 -- 172.21.15.107:0/444671948 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209400f650 con 0x7f2098100700 2024-05-09T08:39:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.234+0000 7f209edb0640 1 -- 172.21.15.107:0/444671948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20981a6870 con 0x7f2098100700 2024-05-09T08:39:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.234+0000 7f208dffb640 1 -- 172.21.15.107:0/444671948 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f209400fca0 con 0x7f2098100700 2024-05-09T08:39:25.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.234+0000 7f208dffb640 1 -- 172.21.15.107:0/444671948 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2094018b90 con 0x7f2098100700 2024-05-09T08:39:25.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.234+0000 7f209edb0640 1 -- 172.21.15.107:0/444671948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20981a6d90 con 0x7f2098100700 2024-05-09T08:39:25.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.236+0000 7f208dffb640 1 -- 172.21.15.107:0/444671948 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f209400f7f0 con 0x7f2098100700 2024-05-09T08:39:25.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.237+0000 7f20a1b9f740 1 Processor -- start 2024-05-09T08:39:25.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.237+0000 7f20a1b9f740 1 -- start start 2024-05-09T08:39:25.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.237+0000 7f208dffb640 1 --2- 172.21.15.107:0/444671948 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f20800a2d00 0x7f20800a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.237+0000 7f20a1b9f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a90b0 0x55c00e2a73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.237+0000 7f20a1b9f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2a79c0 0x55c00e2a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.237+0000 7f20a1b9f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2b2740 0x55c00e2b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.237+0000 7f20a1b9f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c00e20c990 con 0x55c00e2a90b0 2024-05-09T08:39:25.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.237+0000 7f20a1b9f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c00e1b45b0 con 0x55c00e2a79c0 2024-05-09T08:39:25.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.237+0000 7f208dffb640 1 -- 172.21.15.107:0/444671948 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f20940cfc30 con 0x7f2098100700 2024-05-09T08:39:25.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.237+0000 7f20a1b9f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c00e208950 con 0x55c00e2b2740 2024-05-09T08:39:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.237+0000 7f209d34d640 1 --2- 172.21.15.107:0/444671948 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f20800a2d00 0x7f20800a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.237+0000 7f209cb4c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a90b0 0x55c00e2a73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.237+0000 7f209cb4c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a90b0 0x55c00e2a73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35710/0 (socket says 172.21.15.107:35710) 2024-05-09T08:39:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.237+0000 7f209cb4c640 1 -- 172.21.15.107:0/2400675217 learned_addr learned my addr 172.21.15.107:0/2400675217 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.238+0000 7f209d34d640 1 --2- 172.21.15.107:0/2400675217 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2b2740 0x55c00e2b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.238+0000 7f208ffff640 1 --2- 172.21.15.107:0/2400675217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2a79c0 0x55c00e2a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.238+0000 7f209cb4c640 1 -- 172.21.15.107:0/2400675217 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2b2740 msgr2=0x55c00e2b4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.238+0000 7f209cb4c640 1 --2- 172.21.15.107:0/2400675217 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2b2740 0x55c00e2b4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.238+0000 7f209cb4c640 1 -- 172.21.15.107:0/2400675217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2a79c0 msgr2=0x55c00e2a7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.238+0000 7f209cb4c640 1 --2- 172.21.15.107:0/2400675217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2a79c0 0x55c00e2a7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.238+0000 7f209cb4c640 1 -- 172.21.15.107:0/2400675217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c00dfe87e0 con 0x55c00e2a90b0 2024-05-09T08:39:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.238+0000 7f208ffff640 1 --2- 172.21.15.107:0/2400675217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2a79c0 0x55c00e2a7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.238+0000 7f209d34d640 1 --2- 172.21.15.107:0/444671948 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f20800a2d00 0x7f20800a51c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f20900058c0 tx=0x7f20900127c0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.238+0000 7f209cb4c640 1 --2- 172.21.15.107:0/2400675217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a90b0 0x55c00e2a73f0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f2094098250 tx=0x7f209409bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aafff19a66a167c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.238+0000 7f207effd640 1 -- 172.21.15.107:0/2400675217 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209407fad0 con 0x55c00e2a90b0 2024-05-09T08:39:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.238+0000 7f207effd640 1 -- 172.21.15.107:0/2400675217 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f209400fae0 con 0x55c00e2a90b0 2024-05-09T08:39:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.238+0000 7f207effd640 1 -- 172.21.15.107:0/2400675217 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2094018d30 con 0x55c00e2a90b0 2024-05-09T08:39:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.239+0000 7f20a1b9f740 1 -- 172.21.15.107:0/2400675217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a90b0 msgr2=0x55c00e2a73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.239+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/2400675217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a90b0 0x55c00e2a73f0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f2094098250 tx=0x7f209409bc80 comp rx=0 tx=0).stop 2024-05-09T08:39:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.239+0000 7f20a1b9f740 1 -- 172.21.15.107:0/2400675217 shutdown_connections 2024-05-09T08:39:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.239+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/2400675217 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2b2740 0x55c00e2b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.239+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/2400675217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2a79c0 0x55c00e2a7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.239+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/2400675217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a90b0 0x55c00e2a73f0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.239+0000 7f20a1b9f740 1 -- 172.21.15.107:0/2400675217 >> 172.21.15.107:0/2400675217 conn(0x55c00e2700a0 msgr2=0x55c00e2a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.239+0000 7f20a1b9f740 1 -- 172.21.15.107:0/2400675217 shutdown_connections 2024-05-09T08:39:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.239+0000 7f20a1b9f740 1 -- 172.21.15.107:0/2400675217 wait complete. 2024-05-09T08:39:25.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.239+0000 7f20a1b9f740 1 -- start start 2024-05-09T08:39:25.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.239+0000 7f20a1b9f740 1 Processor -- start 2024-05-09T08:39:25.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f20a1b9f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a79c0 0x55c00e2af690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f20a1b9f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2a90b0 0x55c00e2afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f20a1b9f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2b2740 0x55c00e2b0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f20a1b9f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c00e2b55d0 con 0x55c00e2a79c0 2024-05-09T08:39:25.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f20a1b9f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c00e1c7b40 con 0x55c00e2b2740 2024-05-09T08:39:25.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f20a1b9f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c00e2b5750 con 0x55c00e2a90b0 2024-05-09T08:39:25.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f20a1b9f740 10 client.0 _ll_register_callbacks cb 0x55c00e1bc0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:25.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f209cb4c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a79c0 0x55c00e2af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f209cb4c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a79c0 0x55c00e2af690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35726/0 (socket says 172.21.15.107:35726) 2024-05-09T08:39:25.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f209cb4c640 1 -- 172.21.15.107:0/2777307063 learned_addr learned my addr 172.21.15.107:0/2777307063 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:25.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f209d34d640 1 --2- 172.21.15.107:0/2777307063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2b2740 0x55c00e2b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f208ffff640 1 --2- 172.21.15.107:0/2777307063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2a90b0 0x55c00e2afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f209cb4c640 1 -- 172.21.15.107:0/2777307063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2a90b0 msgr2=0x55c00e2afbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f209cb4c640 1 --2- 172.21.15.107:0/2777307063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2a90b0 0x55c00e2afbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f209cb4c640 1 -- 172.21.15.107:0/2777307063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2b2740 msgr2=0x55c00e2b0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f209cb4c640 1 --2- 172.21.15.107:0/2777307063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2b2740 0x55c00e2b0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.240+0000 7f209cb4c640 1 -- 172.21.15.107:0/2777307063 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c00e048f70 con 0x55c00e2a79c0 2024-05-09T08:39:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.241+0000 7f209cb4c640 1 --2- 172.21.15.107:0/2777307063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a79c0 0x55c00e2af690 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f209400af80 tx=0x7f2094018f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39173d6765284d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.241+0000 7f2077fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.241+0000 7f2077fff640 1 -- 172.21.15.107:0/2777307063 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209407e060 con 0x55c00e2a79c0 2024-05-09T08:39:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.241+0000 7f2077fff640 1 -- 172.21.15.107:0/2777307063 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f20940a45d0 con 0x55c00e2a79c0 2024-05-09T08:39:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.241+0000 7f20a1b9f740 1 -- 172.21.15.107:0/2777307063 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c00e21a0c0 con 0x55c00e2a79c0 2024-05-09T08:39:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.241+0000 7f2077fff640 1 -- 172.21.15.107:0/2777307063 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2094099f00 con 0x55c00e2a79c0 2024-05-09T08:39:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.241+0000 7f20a1b9f740 10 client.17457 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.241+0000 7f20a1b9f740 1 -- 172.21.15.107:0/2777307063 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c00dee0170 con 0x55c00e2a79c0 2024-05-09T08:39:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.241+0000 7f2077fff640 1 -- 172.21.15.107:0/2777307063 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55c00e21a0c0 con 0x55c00e2a79c0 2024-05-09T08:39:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.241+0000 7f2077fff640 1 -- 172.21.15.107:0/2777307063 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f2094062050 con 0x55c00e2a79c0 2024-05-09T08:39:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.241+0000 7f20a1b9f740 10 client.17457 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.241+0000 7f20a1b9f740 10 client.17457 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.241+0000 7f20a1b9f740 20 client.17457 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.241+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/2777307063 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c00e2c2660 0x55c00e2c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.241+0000 7f20a1b9f740 4 client.17457 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.241+0000 7f20a1b9f740 1 -- 172.21.15.107:0/2777307063 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c00e214410 con 0x55c00e2c2660 2024-05-09T08:39:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.242+0000 7f208ffff640 1 --2- 172.21.15.107:0/2777307063 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c00e2c2660 0x55c00e2c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.242+0000 7f2077fff640 10 client.17457 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.242+0000 7f208ffff640 1 --2- 172.21.15.107:0/2777307063 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c00e2c2660 0x55c00e2c4a50 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=829199163f72295 server_cookie=cfea20b0e86c729 in_seq=0 out_seq=0 2024-05-09T08:39:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.243+0000 7f2077fff640 1 -- 172.21.15.107:0/2777307063 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c00e214410 con 0x55c00e2c2660 2024-05-09T08:39:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.243+0000 7f2077fff640 10 client.17457 handle_command_reply: tid=0 2024-05-09T08:39:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.243+0000 7f20a1b9f740 1 client.17457 shutdown 2024-05-09T08:39:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.243+0000 7f20a1b9f740 1 -- 172.21.15.107:0/2777307063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a79c0 msgr2=0x55c00e2af690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.243+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/2777307063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a79c0 0x55c00e2af690 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f209400af80 tx=0x7f2094018f60 comp rx=0 tx=0).stop 2024-05-09T08:39:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.243+0000 7f20a1b9f740 1 -- 172.21.15.107:0/2777307063 shutdown_connections 2024-05-09T08:39:25.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.243+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/2777307063 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c00e2c2660 0x55c00e2c4a50 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.243+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/2777307063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2b2740 0x55c00e2b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.243+0000 7f2077fff640 0 client.17457 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.243+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/2777307063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2a90b0 0x55c00e2afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.243+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/2777307063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a79c0 0x55c00e2af690 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.243+0000 7f20a1b9f740 1 -- 172.21.15.107:0/2777307063 >> 172.21.15.107:0/2777307063 conn(0x55c00e2700a0 msgr2=0x55c00e2a9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.243+0000 7f20a1b9f740 1 -- 172.21.15.107:0/2777307063 shutdown_connections 2024-05-09T08:39:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.243+0000 7f20a1b9f740 1 -- 172.21.15.107:0/2777307063 wait complete. 2024-05-09T08:39:25.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.243+0000 7f20a1b9f740 20 client.17457 trim_cache size 0 max 16384 2024-05-09T08:39:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f20a1b9f740 1 Processor -- start 2024-05-09T08:39:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f20a1b9f740 1 -- start start 2024-05-09T08:39:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f20a1b9f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2a79c0 0x55c00e2ac910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f20a1b9f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2a90b0 0x55c00e2ace50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f20a1b9f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2b2740 0x55c00e2bd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f20a1b9f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f20981b3560 con 0x55c00e2b2740 2024-05-09T08:39:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f20a1b9f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f20981b0e30 con 0x55c00e2a79c0 2024-05-09T08:39:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f20a1b9f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f20981aec80 con 0x55c00e2a90b0 2024-05-09T08:39:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f209d34d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2b2740 0x55c00e2bd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f209d34d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2b2740 0x55c00e2bd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35728/0 (socket says 172.21.15.107:35728) 2024-05-09T08:39:25.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f209d34d640 1 -- 172.21.15.107:0/1800829564 learned_addr learned my addr 172.21.15.107:0/1800829564 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:25.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f208ffff640 1 --2- 172.21.15.107:0/1800829564 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2a90b0 0x55c00e2ace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f209cb4c640 1 --2- 172.21.15.107:0/1800829564 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2a79c0 0x55c00e2ac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f209d34d640 1 -- 172.21.15.107:0/1800829564 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2a90b0 msgr2=0x55c00e2ace50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f209d34d640 1 --2- 172.21.15.107:0/1800829564 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2a90b0 0x55c00e2ace50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f209d34d640 1 -- 172.21.15.107:0/1800829564 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2a79c0 msgr2=0x55c00e2ac910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.265+0000 7f209d34d640 1 --2- 172.21.15.107:0/1800829564 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2a79c0 0x55c00e2ac910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.266+0000 7f209d34d640 1 -- 172.21.15.107:0/1800829564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c00e2ad390 con 0x55c00e2b2740 2024-05-09T08:39:25.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.266+0000 7f209d34d640 1 --2- 172.21.15.107:0/1800829564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2b2740 0x55c00e2bd9e0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f20900206c0 tx=0x7f209001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f4f6b8670fbac9e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.266+0000 7f207e7fc640 1 -- 172.21.15.107:0/1800829564 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2090020220 con 0x55c00e2b2740 2024-05-09T08:39:25.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.266+0000 7f207e7fc640 1 -- 172.21.15.107:0/1800829564 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f20900054d0 con 0x55c00e2b2740 2024-05-09T08:39:25.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.266+0000 7f207e7fc640 1 -- 172.21.15.107:0/1800829564 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209001e730 con 0x55c00e2b2740 2024-05-09T08:39:25.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.267+0000 7f20a1b9f740 1 -- 172.21.15.107:0/1800829564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2b2740 msgr2=0x55c00e2bd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.267+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/1800829564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2b2740 0x55c00e2bd9e0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f20900206c0 tx=0x7f209001c000 comp rx=0 tx=0).stop 2024-05-09T08:39:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.267+0000 7f20a1b9f740 1 -- 172.21.15.107:0/1800829564 shutdown_connections 2024-05-09T08:39:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.267+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/1800829564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2b2740 0x55c00e2bd9e0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.267+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/1800829564 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2a90b0 0x55c00e2ace50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.267+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/1800829564 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2a79c0 0x55c00e2ac910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.267+0000 7f20a1b9f740 1 -- 172.21.15.107:0/1800829564 >> 172.21.15.107:0/1800829564 conn(0x55c00e2700a0 msgr2=0x55c00e2a3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.267+0000 7f20a1b9f740 1 -- 172.21.15.107:0/1800829564 shutdown_connections 2024-05-09T08:39:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.267+0000 7f20a1b9f740 1 -- 172.21.15.107:0/1800829564 wait complete. 2024-05-09T08:39:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.267+0000 7f20a1b9f740 1 -- start start 2024-05-09T08:39:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.267+0000 7f20a1b9f740 1 Processor -- start 2024-05-09T08:39:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.267+0000 7f20a1b9f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a79c0 0x55c00e2c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.267+0000 7f20a1b9f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2a90b0 0x55c00e2c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.267+0000 7f20a1b9f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2b2740 0x55c00e327840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.267+0000 7f20a1b9f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f20981af2f0 con 0x55c00e2a79c0 2024-05-09T08:39:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f20a1b9f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f20981b0760 con 0x55c00e2b2740 2024-05-09T08:39:25.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f20a1b9f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f20981af9c0 con 0x55c00e2a90b0 2024-05-09T08:39:25.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f20a1b9f740 10 client.0 _ll_register_callbacks cb 0x7f209400fe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:25.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f209cb4c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a79c0 0x55c00e2c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f209cb4c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a79c0 0x55c00e2c57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35730/0 (socket says 172.21.15.107:35730) 2024-05-09T08:39:25.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f209cb4c640 1 -- 172.21.15.107:0/3035147242 learned_addr learned my addr 172.21.15.107:0/3035147242 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:25.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f209d34d640 1 --2- 172.21.15.107:0/3035147242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2b2740 0x55c00e327840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f208ffff640 1 --2- 172.21.15.107:0/3035147242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2a90b0 0x55c00e2c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f209cb4c640 1 -- 172.21.15.107:0/3035147242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2a90b0 msgr2=0x55c00e2c5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f209cb4c640 1 --2- 172.21.15.107:0/3035147242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2a90b0 0x55c00e2c5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f209cb4c640 1 -- 172.21.15.107:0/3035147242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2b2740 msgr2=0x55c00e327840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f209cb4c640 1 --2- 172.21.15.107:0/3035147242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2b2740 0x55c00e327840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f209cb4c640 1 -- 172.21.15.107:0/3035147242 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c00e2c6590 con 0x55c00e2a79c0 2024-05-09T08:39:25.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f209cb4c640 1 --2- 172.21.15.107:0/3035147242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a79c0 0x55c00e2c57c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f209409a0a0 tx=0x7f20940a4a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d74b8bb1e4d180 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f2077fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:25.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f2077fff640 1 -- 172.21.15.107:0/3035147242 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209407e040 con 0x55c00e2a79c0 2024-05-09T08:39:25.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f2077fff640 1 -- 172.21.15.107:0/3035147242 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f209409b240 con 0x55c00e2a79c0 2024-05-09T08:39:25.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f20a1b9f740 1 -- 172.21.15.107:0/3035147242 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20900054d0 con 0x55c00e2a79c0 2024-05-09T08:39:25.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f2077fff640 1 -- 172.21.15.107:0/3035147242 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f209409b750 con 0x55c00e2a79c0 2024-05-09T08:39:25.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f20a1b9f740 10 client.17463 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:25.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.268+0000 7f20a1b9f740 1 -- 172.21.15.107:0/3035147242 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c00e328e90 con 0x55c00e2a79c0 2024-05-09T08:39:25.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.269+0000 7f2077fff640 1 -- 172.21.15.107:0/3035147242 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f20900054d0 con 0x55c00e2a79c0 2024-05-09T08:39:25.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.269+0000 7f2077fff640 1 -- 172.21.15.107:0/3035147242 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f2094062050 con 0x55c00e2a79c0 2024-05-09T08:39:25.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.269+0000 7f20a1b9f740 10 client.17463 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:25.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.269+0000 7f20a1b9f740 10 client.17463 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:25.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.269+0000 7f20a1b9f740 20 client.17463 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:25.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.269+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/3035147242 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c00e2c2660 0x55c00e32b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.269+0000 7f20a1b9f740 4 client.17463 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:39:25.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.269+0000 7f20a1b9f740 1 -- 172.21.15.107:0/3035147242 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c00e191da0 con 0x55c00e2c2660 2024-05-09T08:39:25.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.269+0000 7f208ffff640 1 --2- 172.21.15.107:0/3035147242 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c00e2c2660 0x55c00e32b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.269+0000 7f2077fff640 10 client.17463 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:25.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.269+0000 7f208ffff640 1 --2- 172.21.15.107:0/3035147242 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c00e2c2660 0x55c00e32b420 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbd1b82f1be25a14 server_cookie=88c134db65e7f1c5 in_seq=0 out_seq=0 2024-05-09T08:39:25.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.270+0000 7f2077fff640 1 -- 172.21.15.107:0/3035147242 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c00e191da0 con 0x55c00e2c2660 2024-05-09T08:39:25.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.270+0000 7f2077fff640 10 client.17463 handle_command_reply: tid=0 2024-05-09T08:39:25.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.271+0000 7f20a1b9f740 1 client.17463 shutdown 2024-05-09T08:39:25.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.271+0000 7f20a1b9f740 1 -- 172.21.15.107:0/3035147242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a79c0 msgr2=0x55c00e2c57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.271+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/3035147242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a79c0 0x55c00e2c57c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f209409a0a0 tx=0x7f20940a4a60 comp rx=0 tx=0).stop 2024-05-09T08:39:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.271+0000 7f20a1b9f740 1 -- 172.21.15.107:0/3035147242 shutdown_connections 2024-05-09T08:39:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.271+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/3035147242 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c00e2c2660 0x55c00e32b420 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.271+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/3035147242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c00e2b2740 0x55c00e327840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.271+0000 7f2077fff640 0 client.17463 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.271+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/3035147242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c00e2a90b0 0x55c00e2c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.271+0000 7f20a1b9f740 1 --2- 172.21.15.107:0/3035147242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c00e2a79c0 0x55c00e2c57c0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.271+0000 7f20a1b9f740 1 -- 172.21.15.107:0/3035147242 >> 172.21.15.107:0/3035147242 conn(0x55c00e2700a0 msgr2=0x55c00e2a27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.271+0000 7f20a1b9f740 1 -- 172.21.15.107:0/3035147242 shutdown_connections 2024-05-09T08:39:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.271+0000 7f20a1b9f740 1 -- 172.21.15.107:0/3035147242 wait complete. 2024-05-09T08:39:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.271+0000 7f20a1b9f740 20 client.17463 trim_cache size 0 max 16384 2024-05-09T08:39:25.276 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:39:25.277 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:39:25.277 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "4e9e571e-05c0-48ca-8145-3edc36000e1d", 2024-05-09T08:39:25.277 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:39:25.277 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:39:25.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.272+0000 7f209edb0640 1 -- 172.21.15.107:0/444671948 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f20800a2d00 msgr2=0x7f20800a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.272+0000 7f209edb0640 1 --2- 172.21.15.107:0/444671948 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f20800a2d00 0x7f20800a51c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f20900058c0 tx=0x7f20900127c0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.272+0000 7f209edb0640 1 -- 172.21.15.107:0/444671948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2098100700 msgr2=0x7f2098102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.272+0000 7f209edb0640 1 --2- 172.21.15.107:0/444671948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2098100700 0x7f2098102cc0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f209400cce0 tx=0x7f209400a620 comp rx=0 tx=0).stop 2024-05-09T08:39:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.272+0000 7f209edb0640 1 -- 172.21.15.107:0/444671948 shutdown_connections 2024-05-09T08:39:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.272+0000 7f209edb0640 1 --2- 172.21.15.107:0/444671948 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f20800a2d00 0x7f20800a51c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.272+0000 7f209edb0640 1 --2- 172.21.15.107:0/444671948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2098106710 0x7f2098103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.272+0000 7f209edb0640 1 --2- 172.21.15.107:0/444671948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20981010c0 0x7f2098103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.272+0000 7f209edb0640 1 --2- 172.21.15.107:0/444671948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2098100700 0x7f2098102cc0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.272+0000 7f209edb0640 1 -- 172.21.15.107:0/444671948 >> 172.21.15.107:0/444671948 conn(0x7f2098077830 msgr2=0x7f2098068da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.272+0000 7f209edb0640 1 -- 172.21.15.107:0/444671948 shutdown_connections 2024-05-09T08:39:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.272+0000 7f209edb0640 1 -- 172.21.15.107:0/444671948 wait complete. 2024-05-09T08:39:25.282 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:39:25.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:39:25.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.398+0000 7f6127d09640 1 -- 172.21.15.107:0/2249864198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61201010c0 msgr2=0x7f612010d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.398+0000 7f6127d09640 1 --2- 172.21.15.107:0/2249864198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61201010c0 0x7f612010d5d0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f6118009cd0 tx=0x7f611802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.398+0000 7f6127d09640 1 -- 172.21.15.107:0/2249864198 shutdown_connections 2024-05-09T08:39:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.398+0000 7f6127d09640 1 --2- 172.21.15.107:0/2249864198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61201010c0 0x7f612010d5d0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.398+0000 7f6127d09640 1 --2- 172.21.15.107:0/2249864198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6120100700 0x7f6120100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.398+0000 7f6127d09640 1 --2- 172.21.15.107:0/2249864198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6120106710 0x7f6120106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.398+0000 7f6127d09640 1 -- 172.21.15.107:0/2249864198 >> 172.21.15.107:0/2249864198 conn(0x7f6120077830 msgr2=0x7f6120077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.398+0000 7f6127d09640 1 -- 172.21.15.107:0/2249864198 shutdown_connections 2024-05-09T08:39:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.398+0000 7f6127d09640 1 -- 172.21.15.107:0/2249864198 wait complete. 2024-05-09T08:39:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f6127d09640 1 Processor -- start 2024-05-09T08:39:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f6127d09640 1 -- start start 2024-05-09T08:39:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f6127d09640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6120100700 0x7f6120198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f6127d09640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61201010c0 0x7f6120199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f6127d09640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6120106710 0x7f61201a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f6127d09640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6120110990 con 0x7f6120100700 2024-05-09T08:39:25.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f6127d09640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6120110810 con 0x7f61201010c0 2024-05-09T08:39:25.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f6127d09640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6120110b10 con 0x7f6120106710 2024-05-09T08:39:25.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f6125aa5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6120100700 0x7f6120198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f6125aa5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6120100700 0x7f6120198e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35740/0 (socket says 172.21.15.107:35740) 2024-05-09T08:39:25.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f6125aa5640 1 -- 172.21.15.107:0/2057383976 learned_addr learned my addr 172.21.15.107:0/2057383976 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:25.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f61262a6640 1 --2- 172.21.15.107:0/2057383976 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6120106710 0x7f61201a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f61252a4640 1 --2- 172.21.15.107:0/2057383976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61201010c0 0x7f6120199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f6125aa5640 1 -- 172.21.15.107:0/2057383976 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6120106710 msgr2=0x7f61201a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f6125aa5640 1 --2- 172.21.15.107:0/2057383976 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6120106710 0x7f61201a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f6125aa5640 1 -- 172.21.15.107:0/2057383976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61201010c0 msgr2=0x7f6120199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f6125aa5640 1 --2- 172.21.15.107:0/2057383976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61201010c0 0x7f6120199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.399+0000 7f6125aa5640 1 -- 172.21.15.107:0/2057383976 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61201a43f0 con 0x7f6120100700 2024-05-09T08:39:25.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.400+0000 7f6125aa5640 1 --2- 172.21.15.107:0/2057383976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6120100700 0x7f6120198e30 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f611c00d950 tx=0x7f611c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.400+0000 7f6116ffd640 1 -- 172.21.15.107:0/2057383976 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c017030 con 0x7f6120100700 2024-05-09T08:39:25.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.400+0000 7f6116ffd640 1 -- 172.21.15.107:0/2057383976 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f611c004020 con 0x7f6120100700 2024-05-09T08:39:25.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.400+0000 7f6127d09640 1 -- 172.21.15.107:0/2057383976 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61201a45c0 con 0x7f6120100700 2024-05-09T08:39:25.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.400+0000 7f6116ffd640 1 -- 172.21.15.107:0/2057383976 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c0047a0 con 0x7f6120100700 2024-05-09T08:39:25.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.400+0000 7f6127d09640 1 -- 172.21.15.107:0/2057383976 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61201a4b60 con 0x7f6120100700 2024-05-09T08:39:25.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.402+0000 7f6116ffd640 1 -- 172.21.15.107:0/2057383976 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f611c0041d0 con 0x7f6120100700 2024-05-09T08:39:25.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.403+0000 7f6116ffd640 1 --2- 172.21.15.107:0/2057383976 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f610c0a2d80 0x7f610c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.403+0000 7f612aa0e740 1 Processor -- start 2024-05-09T08:39:25.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.403+0000 7f61252a4640 1 --2- 172.21.15.107:0/2057383976 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f610c0a2d80 0x7f610c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.403+0000 7f6116ffd640 1 -- 172.21.15.107:0/2057383976 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f611c0cf0d0 con 0x7f6120100700 2024-05-09T08:39:25.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.403+0000 7f612aa0e740 1 -- start start 2024-05-09T08:39:25.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.403+0000 7f612aa0e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890708fb0 0x55b890709390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.403+0000 7f612aa0e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890718140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f612aa0e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890718680 0x55b89071aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f612aa0e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b89066ed00 con 0x55b890709960 2024-05-09T08:39:25.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f612aa0e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b8906164c0 con 0x55b890718680 2024-05-09T08:39:25.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f612aa0e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55b89066acc0 con 0x55b890708fb0 2024-05-09T08:39:25.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f61252a4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890718140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f61252a4640 1 --2- 172.21.15.107:0/2057383976 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f610c0a2d80 0x7f610c0a5240 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6120069530 tx=0x7f611000b390 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f61252a4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890718140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35752/0 (socket says 172.21.15.107:35752) 2024-05-09T08:39:25.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f61252a4640 1 -- 172.21.15.107:0/493502008 learned_addr learned my addr 172.21.15.107:0/493502008 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f6125aa5640 1 --2- 172.21.15.107:0/493502008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890708fb0 0x55b890709390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f61262a6640 1 --2- 172.21.15.107:0/493502008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890718680 0x55b89071aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f6125aa5640 1 -- 172.21.15.107:0/493502008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890718680 msgr2=0x55b89071aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f6125aa5640 1 --2- 172.21.15.107:0/493502008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890718680 0x55b89071aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f6125aa5640 1 -- 172.21.15.107:0/493502008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 msgr2=0x55b890718140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f6125aa5640 1 --2- 172.21.15.107:0/493502008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890718140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f6125aa5640 1 -- 172.21.15.107:0/493502008 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b89044c7d0 con 0x55b890708fb0 2024-05-09T08:39:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f6125aa5640 1 --2- 172.21.15.107:0/493502008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890708fb0 0x55b890709390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f611c0cc490 tx=0x7f611c09b190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe0ffdf89f99abb8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.404+0000 7f610bfff640 1 -- 172.21.15.107:0/493502008 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c09b290 con 0x55b890708fb0 2024-05-09T08:39:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.405+0000 7f610bfff640 1 -- 172.21.15.107:0/493502008 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f611c09f640 con 0x55b890708fb0 2024-05-09T08:39:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.405+0000 7f610bfff640 1 -- 172.21.15.107:0/493502008 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c009a30 con 0x55b890708fb0 2024-05-09T08:39:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.405+0000 7f612aa0e740 1 -- 172.21.15.107:0/493502008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890708fb0 msgr2=0x55b890709390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.405+0000 7f612aa0e740 1 --2- 172.21.15.107:0/493502008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890708fb0 0x55b890709390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f611c0cc490 tx=0x7f611c09b190 comp rx=0 tx=0).stop 2024-05-09T08:39:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.405+0000 7f612aa0e740 1 -- 172.21.15.107:0/493502008 shutdown_connections 2024-05-09T08:39:25.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.405+0000 7f612aa0e740 1 --2- 172.21.15.107:0/493502008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890718680 0x55b89071aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.405+0000 7f612aa0e740 1 --2- 172.21.15.107:0/493502008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890718140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.405+0000 7f612aa0e740 1 --2- 172.21.15.107:0/493502008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890708fb0 0x55b890709390 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.405+0000 7f612aa0e740 1 -- 172.21.15.107:0/493502008 >> 172.21.15.107:0/493502008 conn(0x55b8907055a0 msgr2=0x55b890705980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.405+0000 7f612aa0e740 1 -- 172.21.15.107:0/493502008 shutdown_connections 2024-05-09T08:39:25.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.405+0000 7f612aa0e740 1 -- 172.21.15.107:0/493502008 wait complete. 2024-05-09T08:39:25.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.406+0000 7f612aa0e740 1 -- start start 2024-05-09T08:39:25.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.406+0000 7f612aa0e740 1 Processor -- start 2024-05-09T08:39:25.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.406+0000 7f612aa0e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890708fb0 0x55b8907156b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.406+0000 7f612aa0e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890715bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.406+0000 7f612aa0e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890718680 0x55b890716130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.406+0000 7f612aa0e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b89071b440 con 0x55b890709960 2024-05-09T08:39:25.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.406+0000 7f612aa0e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b890629b90 con 0x55b890718680 2024-05-09T08:39:25.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.406+0000 7f612aa0e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55b89071b5c0 con 0x55b890708fb0 2024-05-09T08:39:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.406+0000 7f612aa0e740 10 client.0 _ll_register_callbacks cb 0x55b890665da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.406+0000 7f61252a4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890715bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.406+0000 7f6125aa5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890708fb0 0x55b8907156b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.406+0000 7f61252a4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890715bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35766/0 (socket says 172.21.15.107:35766) 2024-05-09T08:39:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.406+0000 7f61252a4640 1 -- 172.21.15.107:0/3155724241 learned_addr learned my addr 172.21.15.107:0/3155724241 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.406+0000 7f61262a6640 1 --2- 172.21.15.107:0/3155724241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890718680 0x55b890716130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.407+0000 7f61252a4640 1 -- 172.21.15.107:0/3155724241 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890708fb0 msgr2=0x55b8907156b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.407+0000 7f61252a4640 1 --2- 172.21.15.107:0/3155724241 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890708fb0 0x55b8907156b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.407+0000 7f61252a4640 1 -- 172.21.15.107:0/3155724241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890718680 msgr2=0x55b890716130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.407+0000 7f61252a4640 1 --2- 172.21.15.107:0/3155724241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890718680 0x55b890716130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.407+0000 7f61252a4640 1 -- 172.21.15.107:0/3155724241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8903823f0 con 0x55b890709960 2024-05-09T08:39:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.407+0000 7f6125aa5640 1 --2- 172.21.15.107:0/3155724241 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890708fb0 0x55b8907156b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.407+0000 7f61262a6640 1 --2- 172.21.15.107:0/3155724241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890718680 0x55b890716130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:39:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.407+0000 7f61252a4640 1 --2- 172.21.15.107:0/3155724241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890715bf0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f61100117c0 tx=0x7f6110004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc05aca2e0feb1e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.407+0000 7f61097fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.407+0000 7f61097fa640 1 -- 172.21.15.107:0/3155724241 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611001e940 con 0x55b890709960 2024-05-09T08:39:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.407+0000 7f612aa0e740 1 -- 172.21.15.107:0/3155724241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f611c09f640 con 0x55b890709960 2024-05-09T08:39:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.407+0000 7f61097fa640 1 -- 172.21.15.107:0/3155724241 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f611001eae0 con 0x55b890709960 2024-05-09T08:39:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.407+0000 7f61097fa640 1 -- 172.21.15.107:0/3155724241 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f611001ece0 con 0x55b890709960 2024-05-09T08:39:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.407+0000 7f612aa0e740 10 client.17475 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.407+0000 7f612aa0e740 1 -- 172.21.15.107:0/3155724241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b8903420f0 con 0x55b890709960 2024-05-09T08:39:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.407+0000 7f61097fa640 1 -- 172.21.15.107:0/3155724241 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f611c09f640 con 0x55b890709960 2024-05-09T08:39:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.408+0000 7f61097fa640 1 -- 172.21.15.107:0/3155724241 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f611001f430 con 0x55b890709960 2024-05-09T08:39:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.408+0000 7f612aa0e740 10 client.17475 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.408+0000 7f612aa0e740 10 client.17475 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.408+0000 7f612aa0e740 20 client.17475 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.408+0000 7f612aa0e740 1 --2- 172.21.15.107:0/3155724241 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b890728af0 0x55b89072af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.408+0000 7f612aa0e740 4 client.17475 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.408+0000 7f612aa0e740 1 -- 172.21.15.107:0/3155724241 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b89067c430 con 0x55b890728af0 2024-05-09T08:39:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.408+0000 7f6125aa5640 1 --2- 172.21.15.107:0/3155724241 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b890728af0 0x55b89072af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.408+0000 7f6125aa5640 1 --2- 172.21.15.107:0/3155724241 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b890728af0 0x55b89072af50 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc61425d79167a12 server_cookie=b2730454f07983ce in_seq=0 out_seq=0 2024-05-09T08:39:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.408+0000 7f61097fa640 10 client.17475 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.409+0000 7f61097fa640 1 -- 172.21.15.107:0/3155724241 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b89067c430 con 0x55b890728af0 2024-05-09T08:39:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.409+0000 7f61097fa640 10 client.17475 handle_command_reply: tid=0 2024-05-09T08:39:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.409+0000 7f612aa0e740 1 client.17475 shutdown 2024-05-09T08:39:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.409+0000 7f612aa0e740 1 -- 172.21.15.107:0/3155724241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 msgr2=0x55b890715bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.409+0000 7f612aa0e740 1 --2- 172.21.15.107:0/3155724241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890715bf0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f61100117c0 tx=0x7f6110004730 comp rx=0 tx=0).stop 2024-05-09T08:39:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.409+0000 7f612aa0e740 1 -- 172.21.15.107:0/3155724241 shutdown_connections 2024-05-09T08:39:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.409+0000 7f612aa0e740 1 --2- 172.21.15.107:0/3155724241 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b890728af0 0x55b89072af50 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.409+0000 7f612aa0e740 1 --2- 172.21.15.107:0/3155724241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890718680 0x55b890716130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.409+0000 7f612aa0e740 1 --2- 172.21.15.107:0/3155724241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890715bf0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.409+0000 7f612aa0e740 1 --2- 172.21.15.107:0/3155724241 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890708fb0 0x55b8907156b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.409+0000 7f612aa0e740 1 -- 172.21.15.107:0/3155724241 >> 172.21.15.107:0/3155724241 conn(0x55b8907055a0 msgr2=0x55b89070b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.409+0000 7f61097fa640 0 client.17475 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.409+0000 7f612aa0e740 1 -- 172.21.15.107:0/3155724241 shutdown_connections 2024-05-09T08:39:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.409+0000 7f612aa0e740 1 -- 172.21.15.107:0/3155724241 wait complete. 2024-05-09T08:39:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.410+0000 7f612aa0e740 20 client.17475 trim_cache size 0 max 16384 2024-05-09T08:39:25.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.431+0000 7f612aa0e740 1 Processor -- start 2024-05-09T08:39:25.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.431+0000 7f612aa0e740 1 -- start start 2024-05-09T08:39:25.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.431+0000 7f612aa0e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890708fb0 0x55b890724340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.431+0000 7f612aa0e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890700670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.431+0000 7f612aa0e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890718680 0x55b890704660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.431+0000 7f612aa0e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f61201af260 con 0x55b890709960 2024-05-09T08:39:25.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.431+0000 7f612aa0e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f61201acb30 con 0x55b890708fb0 2024-05-09T08:39:25.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.431+0000 7f612aa0e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f61201aa980 con 0x55b890718680 2024-05-09T08:39:25.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.431+0000 7f61252a4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890700670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.431+0000 7f61252a4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890700670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35768/0 (socket says 172.21.15.107:35768) 2024-05-09T08:39:25.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.431+0000 7f61252a4640 1 -- 172.21.15.107:0/3415023919 learned_addr learned my addr 172.21.15.107:0/3415023919 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:25.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.432+0000 7f61262a6640 1 --2- 172.21.15.107:0/3415023919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890718680 0x55b890704660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.432+0000 7f6125aa5640 1 --2- 172.21.15.107:0/3415023919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890708fb0 0x55b890724340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.432+0000 7f61252a4640 1 -- 172.21.15.107:0/3415023919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890718680 msgr2=0x55b890704660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.432+0000 7f61252a4640 1 --2- 172.21.15.107:0/3415023919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890718680 0x55b890704660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.432+0000 7f61252a4640 1 -- 172.21.15.107:0/3415023919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890708fb0 msgr2=0x55b890724340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.432+0000 7f61252a4640 1 --2- 172.21.15.107:0/3415023919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890708fb0 0x55b890724340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.432+0000 7f61252a4640 1 -- 172.21.15.107:0/3415023919 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b890714660 con 0x55b890709960 2024-05-09T08:39:25.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.432+0000 7f61252a4640 1 --2- 172.21.15.107:0/3415023919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890700670 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f6110010e50 tx=0x7f611000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12121e00fe97837a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.432+0000 7f610b7fe640 1 -- 172.21.15.107:0/3415023919 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611003a070 con 0x55b890709960 2024-05-09T08:39:25.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.432+0000 7f610b7fe640 1 -- 172.21.15.107:0/3415023919 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f611000b400 con 0x55b890709960 2024-05-09T08:39:25.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.432+0000 7f610b7fe640 1 -- 172.21.15.107:0/3415023919 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6110028690 con 0x55b890709960 2024-05-09T08:39:25.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.433+0000 7f612aa0e740 1 -- 172.21.15.107:0/3415023919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 msgr2=0x55b890700670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.433+0000 7f612aa0e740 1 --2- 172.21.15.107:0/3415023919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890700670 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f6110010e50 tx=0x7f611000b8f0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.433+0000 7f612aa0e740 1 -- 172.21.15.107:0/3415023919 shutdown_connections 2024-05-09T08:39:25.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.433+0000 7f612aa0e740 1 --2- 172.21.15.107:0/3415023919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890718680 0x55b890704660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.433+0000 7f612aa0e740 1 --2- 172.21.15.107:0/3415023919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890709960 0x55b890700670 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.433+0000 7f612aa0e740 1 --2- 172.21.15.107:0/3415023919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890708fb0 0x55b890724340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.433+0000 7f612aa0e740 1 -- 172.21.15.107:0/3415023919 >> 172.21.15.107:0/3415023919 conn(0x55b8907055a0 msgr2=0x55b890703f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.433+0000 7f612aa0e740 1 -- 172.21.15.107:0/3415023919 shutdown_connections 2024-05-09T08:39:25.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.433+0000 7f612aa0e740 1 -- 172.21.15.107:0/3415023919 wait complete. 2024-05-09T08:39:25.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.433+0000 7f612aa0e740 1 -- start start 2024-05-09T08:39:25.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.433+0000 7f612aa0e740 1 Processor -- start 2024-05-09T08:39:25.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f612aa0e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890708fb0 0x55b890720a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f612aa0e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890709960 0x55b89071bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f612aa0e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890718680 0x55b89071c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f612aa0e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f61201aaff0 con 0x55b890708fb0 2024-05-09T08:39:25.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f612aa0e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f61201ac460 con 0x55b890709960 2024-05-09T08:39:25.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f612aa0e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f61201ab6c0 con 0x55b890718680 2024-05-09T08:39:25.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f612aa0e740 10 client.0 _ll_register_callbacks cb 0x7f6110010d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:25.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f6125aa5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890708fb0 0x55b890720a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f6125aa5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890708fb0 0x55b890720a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35772/0 (socket says 172.21.15.107:35772) 2024-05-09T08:39:25.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f6125aa5640 1 -- 172.21.15.107:0/1186433580 learned_addr learned my addr 172.21.15.107:0/1186433580 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:25.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f61252a4640 1 --2- 172.21.15.107:0/1186433580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890709960 0x55b89071bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f61262a6640 1 --2- 172.21.15.107:0/1186433580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890718680 0x55b89071c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f6125aa5640 1 -- 172.21.15.107:0/1186433580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890718680 msgr2=0x55b89071c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f6125aa5640 1 --2- 172.21.15.107:0/1186433580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890718680 0x55b89071c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f6125aa5640 1 -- 172.21.15.107:0/1186433580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890709960 msgr2=0x55b89071bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f6125aa5640 1 --2- 172.21.15.107:0/1186433580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890709960 0x55b89071bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f6125aa5640 1 -- 172.21.15.107:0/1186433580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b89071c970 con 0x55b890708fb0 2024-05-09T08:39:25.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f61252a4640 1 --2- 172.21.15.107:0/1186433580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890709960 0x55b89071bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:25.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f6125aa5640 1 --2- 172.21.15.107:0/1186433580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890708fb0 0x55b890720a80 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f611c0976f0 tx=0x7f611c09b060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=313d1b6880b4710 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.434+0000 7f61097fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:25.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.435+0000 7f61097fa640 1 -- 172.21.15.107:0/1186433580 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c09f640 con 0x55b890708fb0 2024-05-09T08:39:25.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.435+0000 7f61097fa640 1 -- 172.21.15.107:0/1186433580 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f611c097970 con 0x55b890708fb0 2024-05-09T08:39:25.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.435+0000 7f612aa0e740 1 -- 172.21.15.107:0/1186433580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f611000b400 con 0x55b890708fb0 2024-05-09T08:39:25.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.435+0000 7f61097fa640 1 -- 172.21.15.107:0/1186433580 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f611c09b290 con 0x55b890708fb0 2024-05-09T08:39:25.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.435+0000 7f612aa0e740 10 client.17481 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:25.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.435+0000 7f612aa0e740 1 -- 172.21.15.107:0/1186433580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b89078ef10 con 0x55b890708fb0 2024-05-09T08:39:25.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.435+0000 7f61097fa640 1 -- 172.21.15.107:0/1186433580 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f611000b400 con 0x55b890708fb0 2024-05-09T08:39:25.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.435+0000 7f61097fa640 1 -- 172.21.15.107:0/1186433580 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f611c080bc0 con 0x55b890708fb0 2024-05-09T08:39:25.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.435+0000 7f612aa0e740 10 client.17481 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:25.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.435+0000 7f612aa0e740 10 client.17481 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:25.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.435+0000 7f612aa0e740 20 client.17481 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:25.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.435+0000 7f612aa0e740 1 --2- 172.21.15.107:0/1186433580 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b89078f1f0 0x55b8907915e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.435+0000 7f612aa0e740 4 client.17481 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:39:25.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.435+0000 7f612aa0e740 1 -- 172.21.15.107:0/1186433580 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b890676780 con 0x55b89078f1f0 2024-05-09T08:39:25.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.435+0000 7f61252a4640 1 --2- 172.21.15.107:0/1186433580 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b89078f1f0 0x55b8907915e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f61252a4640 1 --2- 172.21.15.107:0/1186433580 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b89078f1f0 0x55b8907915e0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6db2a675a0d3f73b server_cookie=4cdfd01bc0318fe6 in_seq=0 out_seq=0 2024-05-09T08:39:25.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f61097fa640 10 client.17481 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:25.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f61097fa640 1 -- 172.21.15.107:0/1186433580 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b890676780 con 0x55b89078f1f0 2024-05-09T08:39:25.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f61097fa640 10 client.17481 handle_command_reply: tid=0 2024-05-09T08:39:25.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f612aa0e740 1 client.17481 shutdown 2024-05-09T08:39:25.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f612aa0e740 1 -- 172.21.15.107:0/1186433580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890708fb0 msgr2=0x55b890720a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f612aa0e740 1 --2- 172.21.15.107:0/1186433580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890708fb0 0x55b890720a80 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f611c0976f0 tx=0x7f611c09b060 comp rx=0 tx=0).stop 2024-05-09T08:39:25.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f612aa0e740 1 -- 172.21.15.107:0/1186433580 shutdown_connections 2024-05-09T08:39:25.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f612aa0e740 1 --2- 172.21.15.107:0/1186433580 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b89078f1f0 0x55b8907915e0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f612aa0e740 1 --2- 172.21.15.107:0/1186433580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b890718680 0x55b89071c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f61097fa640 0 client.17481 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:25.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f612aa0e740 1 --2- 172.21.15.107:0/1186433580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b890709960 0x55b89071bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f612aa0e740 1 --2- 172.21.15.107:0/1186433580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b890708fb0 0x55b890720a80 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f612aa0e740 1 -- 172.21.15.107:0/1186433580 >> 172.21.15.107:0/1186433580 conn(0x55b8907055a0 msgr2=0x55b89070b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f612aa0e740 1 -- 172.21.15.107:0/1186433580 shutdown_connections 2024-05-09T08:39:25.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f612aa0e740 1 -- 172.21.15.107:0/1186433580 wait complete. 2024-05-09T08:39:25.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.436+0000 7f612aa0e740 20 client.17481 trim_cache size 0 max 16384 2024-05-09T08:39:25.442 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:39:25.443 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:39:25.443 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:39:25.443 INFO:teuthology.orchestra.run.smithi107.stdout: "4e9e571e-05c0-48ca-8145-3edc36000e1d": { 2024-05-09T08:39:25.443 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:39:25.443 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "4e9e571e-05c0-48ca-8145-3edc36000e1d", 2024-05-09T08:39:25.443 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:39:25.443 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:39:25.443 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:39:25.443 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:39:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.437+0000 7f6127d09640 1 -- 172.21.15.107:0/2057383976 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f610c0a2d80 msgr2=0x7f610c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.437+0000 7f6127d09640 1 --2- 172.21.15.107:0/2057383976 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f610c0a2d80 0x7f610c0a5240 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6120069530 tx=0x7f611000b390 comp rx=0 tx=0).stop 2024-05-09T08:39:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.437+0000 7f6127d09640 1 -- 172.21.15.107:0/2057383976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6120100700 msgr2=0x7f6120198e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.437+0000 7f6127d09640 1 --2- 172.21.15.107:0/2057383976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6120100700 0x7f6120198e30 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f611c00d950 tx=0x7f611c00de10 comp rx=0 tx=0).stop 2024-05-09T08:39:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.437+0000 7f6127d09640 1 -- 172.21.15.107:0/2057383976 shutdown_connections 2024-05-09T08:39:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.437+0000 7f6127d09640 1 --2- 172.21.15.107:0/2057383976 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f610c0a2d80 0x7f610c0a5240 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.437+0000 7f6127d09640 1 --2- 172.21.15.107:0/2057383976 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6120106710 0x7f61201a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.437+0000 7f6127d09640 1 --2- 172.21.15.107:0/2057383976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61201010c0 0x7f6120199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.437+0000 7f6127d09640 1 --2- 172.21.15.107:0/2057383976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6120100700 0x7f6120198e30 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.437+0000 7f6127d09640 1 -- 172.21.15.107:0/2057383976 >> 172.21.15.107:0/2057383976 conn(0x7f6120077830 msgr2=0x7f6120102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.437+0000 7f6127d09640 1 -- 172.21.15.107:0/2057383976 shutdown_connections 2024-05-09T08:39:25.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:25.437+0000 7f6127d09640 1 -- 172.21.15.107:0/2057383976 wait complete. 2024-05-09T08:39:25.447 INFO:tasks.cephfs.filesystem:scrub status for tag:4e9e571e-05c0-48ca-8145-3edc36000e1d - {'path': '/', 'tag': '4e9e571e-05c0-48ca-8145-3edc36000e1d', 'options': 'recursive,force'} 2024-05-09T08:39:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.564+0000 7fafb6722640 1 -- 172.21.15.107:0/2451746523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafb0101a80 msgr2=0x7fafb0105350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.564+0000 7fafb6722640 1 --2- 172.21.15.107:0/2451746523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafb0101a80 0x7fafb0105350 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fafa4009930 tx=0x7fafa402a420 comp rx=0 tx=0).stop 2024-05-09T08:39:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.565+0000 7fafb6722640 1 -- 172.21.15.107:0/2451746523 shutdown_connections 2024-05-09T08:39:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.565+0000 7fafb6722640 1 --2- 172.21.15.107:0/2451746523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafb0105a80 0x7fafb010d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.565+0000 7fafb6722640 1 --2- 172.21.15.107:0/2451746523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafb0101a80 0x7fafb0105350 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.565+0000 7fafb6722640 1 --2- 172.21.15.107:0/2451746523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafb01010d0 0x7fafb01014b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.565+0000 7fafb6722640 1 -- 172.21.15.107:0/2451746523 >> 172.21.15.107:0/2451746523 conn(0x7fafb0076cc0 msgr2=0x7fafb00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.565+0000 7fafb6722640 1 -- 172.21.15.107:0/2451746523 shutdown_connections 2024-05-09T08:39:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.565+0000 7fafb6722640 1 -- 172.21.15.107:0/2451746523 wait complete. 2024-05-09T08:39:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.565+0000 7fafb6722640 1 Processor -- start 2024-05-09T08:39:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.565+0000 7fafb6722640 1 -- start start 2024-05-09T08:39:25.567 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafb6722640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafb01010d0 0x7fafb0197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.567 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafb6722640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafb0105a80 0x7fafb0198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.567 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafb6722640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafb019c7f0 0x7fafb019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.567 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafb6722640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fafb01109f0 con 0x7fafb0105a80 2024-05-09T08:39:25.567 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafb6722640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fafb0110870 con 0x7fafb01010d0 2024-05-09T08:39:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafb6722640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fafb0110b70 con 0x7fafb019c7f0 2024-05-09T08:39:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafaf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafb0105a80 0x7fafb0198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafaf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafb0105a80 0x7fafb0198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35798/0 (socket says 172.21.15.107:35798) 2024-05-09T08:39:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafaf7fe640 1 -- 172.21.15.107:0/4071444796 learned_addr learned my addr 172.21.15.107:0/4071444796 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafaffff640 1 --2- 172.21.15.107:0/4071444796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafb01010d0 0x7fafb0197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafb4cbf640 1 --2- 172.21.15.107:0/4071444796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafb019c7f0 0x7fafb019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafaf7fe640 1 -- 172.21.15.107:0/4071444796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafb019c7f0 msgr2=0x7fafb019cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafaf7fe640 1 --2- 172.21.15.107:0/4071444796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafb019c7f0 0x7fafb019cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafaf7fe640 1 -- 172.21.15.107:0/4071444796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafb01010d0 msgr2=0x7fafb0197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafaf7fe640 1 --2- 172.21.15.107:0/4071444796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafb01010d0 0x7fafb0197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafaf7fe640 1 -- 172.21.15.107:0/4071444796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafb019d2e0 con 0x7fafb0105a80 2024-05-09T08:39:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafb4cbf640 1 --2- 172.21.15.107:0/4071444796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafb019c7f0 0x7fafb019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafaffff640 1 --2- 172.21.15.107:0/4071444796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafb01010d0 0x7fafb0197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafaf7fe640 1 --2- 172.21.15.107:0/4071444796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafb0105a80 0x7fafb0198470 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fafa4031440 tx=0x7fafa4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafad7fa640 1 -- 172.21.15.107:0/4071444796 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa403f030 con 0x7fafb0105a80 2024-05-09T08:39:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafad7fa640 1 -- 172.21.15.107:0/4071444796 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafa4003aa0 con 0x7fafb0105a80 2024-05-09T08:39:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.566+0000 7fafb6722640 1 -- 172.21.15.107:0/4071444796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafb019d570 con 0x7fafb0105a80 2024-05-09T08:39:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.567+0000 7fafad7fa640 1 -- 172.21.15.107:0/4071444796 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa4035070 con 0x7fafb0105a80 2024-05-09T08:39:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.567+0000 7fafb6722640 1 -- 172.21.15.107:0/4071444796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafb01a4d30 con 0x7fafb0105a80 2024-05-09T08:39:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.569+0000 7fafad7fa640 1 -- 172.21.15.107:0/4071444796 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fafa4042030 con 0x7fafb0105a80 2024-05-09T08:39:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.569+0000 7fafb9435740 1 Processor -- start 2024-05-09T08:39:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.569+0000 7fafb9435740 1 -- start start 2024-05-09T08:39:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafb9435740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd79a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafb9435740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd79ad70 0x5640bd7a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafb9435740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd7a5630 0x5640bd7a7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafb9435740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5640bd6ff340 con 0x5640bd79a450 2024-05-09T08:39:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafb9435740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5640bd6a7290 con 0x5640bd7a5630 2024-05-09T08:39:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafb9435740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5640bd6fb300 con 0x5640bd79ad70 2024-05-09T08:39:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafad7fa640 1 --2- 172.21.15.107:0/4071444796 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf980a2da0 0x7faf980a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafb4cbf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd79a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafb4cbf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd79a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35810/0 (socket says 172.21.15.107:35810) 2024-05-09T08:39:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafb4cbf640 1 -- 172.21.15.107:0/3781871796 learned_addr learned my addr 172.21.15.107:0/3781871796 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafaf7fe640 1 --2- 172.21.15.107:0/3781871796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd7a5630 0x5640bd7a7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafaffff640 1 --2- 172.21.15.107:0/3781871796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd79ad70 0x5640bd7a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafb4cbf640 1 --2- 172.21.15.107:0/4071444796 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf980a2da0 0x7faf980a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafb4cbf640 1 -- 172.21.15.107:0/3781871796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd79ad70 msgr2=0x5640bd7a50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafb4cbf640 1 --2- 172.21.15.107:0/3781871796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd79ad70 0x5640bd7a50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafb4cbf640 1 -- 172.21.15.107:0/3781871796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd7a5630 msgr2=0x5640bd7a7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafad7fa640 1 -- 172.21.15.107:0/4071444796 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fafa40efb40 con 0x7fafb0105a80 2024-05-09T08:39:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafb4cbf640 1 --2- 172.21.15.107:0/3781871796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd7a5630 0x5640bd7a7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafb4cbf640 1 -- 172.21.15.107:0/3781871796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640bd6cb8e0 con 0x5640bd79a450 2024-05-09T08:39:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.570+0000 7fafb4cbf640 1 --2- 172.21.15.107:0/3781871796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd79a830 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fafa8013a60 tx=0x7fafa8013f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295609c7f9e0d983 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.571+0000 7faf967fc640 1 -- 172.21.15.107:0/3781871796 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa8024070 con 0x5640bd79a450 2024-05-09T08:39:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.571+0000 7faf967fc640 1 -- 172.21.15.107:0/3781871796 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafa801fc90 con 0x5640bd79a450 2024-05-09T08:39:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.571+0000 7faf967fc640 1 -- 172.21.15.107:0/3781871796 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa8012aa0 con 0x5640bd79a450 2024-05-09T08:39:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.571+0000 7fafb9435740 1 -- 172.21.15.107:0/3781871796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 msgr2=0x5640bd79a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.571+0000 7fafb9435740 1 --2- 172.21.15.107:0/3781871796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd79a830 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fafa8013a60 tx=0x7fafa8013f20 comp rx=0 tx=0).stop 2024-05-09T08:39:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.571+0000 7fafb4cbf640 1 --2- 172.21.15.107:0/4071444796 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf980a2da0 0x7faf980a5260 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fafa8021000 tx=0x7fafa801e000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.571+0000 7fafb9435740 1 -- 172.21.15.107:0/3781871796 shutdown_connections 2024-05-09T08:39:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.571+0000 7fafb9435740 1 --2- 172.21.15.107:0/3781871796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd7a5630 0x5640bd7a7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.571+0000 7fafb9435740 1 --2- 172.21.15.107:0/3781871796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd79ad70 0x5640bd7a50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.571+0000 7fafb9435740 1 --2- 172.21.15.107:0/3781871796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd79a830 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.571+0000 7fafb9435740 1 -- 172.21.15.107:0/3781871796 >> 172.21.15.107:0/3781871796 conn(0x5640bd796960 msgr2=0x5640bd796d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.571+0000 7fafb9435740 1 -- 172.21.15.107:0/3781871796 shutdown_connections 2024-05-09T08:39:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.571+0000 7fafb9435740 1 -- 172.21.15.107:0/3781871796 wait complete. 2024-05-09T08:39:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.572+0000 7fafb9435740 1 -- start start 2024-05-09T08:39:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.572+0000 7fafb9435740 1 Processor -- start 2024-05-09T08:39:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.572+0000 7fafb9435740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd79a450 0x5640bd7a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.572+0000 7fafb9435740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79ad70 0x5640bd7a2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.572+0000 7fafb9435740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd7a5630 0x5640bd7a2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.572+0000 7fafb9435740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5640bd7a8260 con 0x5640bd79ad70 2024-05-09T08:39:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.572+0000 7fafb9435740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5640bd6ba790 con 0x5640bd7a5630 2024-05-09T08:39:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.572+0000 7fafb9435740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5640bd7a83e0 con 0x5640bd79a450 2024-05-09T08:39:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.572+0000 7fafb9435740 10 client.0 _ll_register_callbacks cb 0x5640bd6f5a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.572+0000 7fafaf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79ad70 0x5640bd7a2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.572+0000 7fafaf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79ad70 0x5640bd7a2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35812/0 (socket says 172.21.15.107:35812) 2024-05-09T08:39:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.572+0000 7fafaffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd79a450 0x5640bd7a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.572+0000 7fafaf7fe640 1 -- 172.21.15.107:0/1776406686 learned_addr learned my addr 172.21.15.107:0/1776406686 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.573+0000 7fafb4cbf640 1 --2- 172.21.15.107:0/1776406686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd7a5630 0x5640bd7a2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.573+0000 7fafaf7fe640 1 -- 172.21.15.107:0/1776406686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd79a450 msgr2=0x5640bd7a24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.573+0000 7fafaf7fe640 1 --2- 172.21.15.107:0/1776406686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd79a450 0x5640bd7a24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.573+0000 7fafaf7fe640 1 -- 172.21.15.107:0/1776406686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd7a5630 msgr2=0x5640bd7a2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.573+0000 7fafaf7fe640 1 --2- 172.21.15.107:0/1776406686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd7a5630 0x5640bd7a2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.573+0000 7fafaf7fe640 1 -- 172.21.15.107:0/1776406686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640bd413210 con 0x5640bd79ad70 2024-05-09T08:39:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.573+0000 7fafb4cbf640 1 --2- 172.21.15.107:0/1776406686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd7a5630 0x5640bd7a2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.573+0000 7fafaf7fe640 1 --2- 172.21.15.107:0/1776406686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79ad70 0x5640bd7a2a20 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fafa40b8100 tx=0x7fafa408abd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0c5bf62ea279a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.573+0000 7faf8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.573+0000 7faf8f7fe640 1 -- 172.21.15.107:0/1776406686 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa40c0050 con 0x5640bd79ad70 2024-05-09T08:39:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.573+0000 7faf8f7fe640 1 -- 172.21.15.107:0/1776406686 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fafa40b8310 con 0x5640bd79ad70 2024-05-09T08:39:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.573+0000 7fafb9435740 1 -- 172.21.15.107:0/1776406686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafa801fc90 con 0x5640bd79ad70 2024-05-09T08:39:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.573+0000 7faf8f7fe640 1 -- 172.21.15.107:0/1776406686 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fafa40c4910 con 0x5640bd79ad70 2024-05-09T08:39:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.573+0000 7fafb9435740 10 client.17493 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.573+0000 7fafb9435740 1 -- 172.21.15.107:0/1776406686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5640bd4df620 con 0x5640bd79ad70 2024-05-09T08:39:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.573+0000 7faf8f7fe640 1 -- 172.21.15.107:0/1776406686 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafa801fc90 con 0x5640bd79ad70 2024-05-09T08:39:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.574+0000 7faf8f7fe640 1 -- 172.21.15.107:0/1776406686 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fafa40ad040 con 0x5640bd79ad70 2024-05-09T08:39:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.574+0000 7fafb9435740 10 client.17493 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.574+0000 7fafb9435740 10 client.17493 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.574+0000 7fafb9435740 20 client.17493 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.574+0000 7fafb9435740 1 --2- 172.21.15.107:0/1776406686 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5640bd7b5320 0x5640bd7b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.574+0000 7fafb9435740 4 client.17493 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.574+0000 7fafb9435740 1 -- 172.21.15.107:0/1776406686 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640bd70ca70 con 0x5640bd7b5320 2024-05-09T08:39:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.574+0000 7fafaffff640 1 --2- 172.21.15.107:0/1776406686 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5640bd7b5320 0x5640bd7b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.574+0000 7fafaffff640 1 --2- 172.21.15.107:0/1776406686 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5640bd7b5320 0x5640bd7b7760 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d15211c100d24bfd server_cookie=d8b7e6dbc04ebe62 in_seq=0 out_seq=0 2024-05-09T08:39:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.574+0000 7faf8f7fe640 10 client.17493 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.575+0000 7faf8f7fe640 1 -- 172.21.15.107:0/1776406686 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5640bd70ca70 con 0x5640bd7b5320 2024-05-09T08:39:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.575+0000 7faf8f7fe640 10 client.17493 handle_command_reply: tid=0 2024-05-09T08:39:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.575+0000 7fafb9435740 1 client.17493 shutdown 2024-05-09T08:39:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.575+0000 7fafb9435740 1 -- 172.21.15.107:0/1776406686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79ad70 msgr2=0x5640bd7a2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.575+0000 7fafb9435740 1 --2- 172.21.15.107:0/1776406686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79ad70 0x5640bd7a2a20 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fafa40b8100 tx=0x7fafa408abd0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.575+0000 7fafb9435740 1 -- 172.21.15.107:0/1776406686 shutdown_connections 2024-05-09T08:39:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.575+0000 7fafb9435740 1 --2- 172.21.15.107:0/1776406686 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5640bd7b5320 0x5640bd7b7760 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.575+0000 7fafb9435740 1 --2- 172.21.15.107:0/1776406686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd7a5630 0x5640bd7a2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.575+0000 7fafb9435740 1 --2- 172.21.15.107:0/1776406686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79ad70 0x5640bd7a2a20 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.575+0000 7faf8f7fe640 0 client.17493 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.575+0000 7fafb9435740 1 --2- 172.21.15.107:0/1776406686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd79a450 0x5640bd7a24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.575+0000 7fafb9435740 1 -- 172.21.15.107:0/1776406686 >> 172.21.15.107:0/1776406686 conn(0x5640bd796960 msgr2=0x5640bd79c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.575+0000 7fafb9435740 1 -- 172.21.15.107:0/1776406686 shutdown_connections 2024-05-09T08:39:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.575+0000 7fafb9435740 1 -- 172.21.15.107:0/1776406686 wait complete. 2024-05-09T08:39:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.575+0000 7fafb9435740 20 client.17493 trim_cache size 0 max 16384 2024-05-09T08:39:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.590+0000 7fafb9435740 1 Processor -- start 2024-05-09T08:39:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.590+0000 7fafb9435740 1 -- start start 2024-05-09T08:39:25.591 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafb9435740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd7b1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafb9435740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd79ad70 0x5640bd79ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafb9435740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd7a5630 0x5640bd7a2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafb9435740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fafb01aefb0 con 0x5640bd79a450 2024-05-09T08:39:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafb9435740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fafb01ac880 con 0x5640bd79ad70 2024-05-09T08:39:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafb9435740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fafb01aa6d0 con 0x5640bd7a5630 2024-05-09T08:39:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafaffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd7b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafaffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd7b1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35828/0 (socket says 172.21.15.107:35828) 2024-05-09T08:39:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafaffff640 1 -- 172.21.15.107:0/3749434969 learned_addr learned my addr 172.21.15.107:0/3749434969 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafaf7fe640 1 --2- 172.21.15.107:0/3749434969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd79ad70 0x5640bd79ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafb4cbf640 1 --2- 172.21.15.107:0/3749434969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd7a5630 0x5640bd7a2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafaffff640 1 -- 172.21.15.107:0/3749434969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd7a5630 msgr2=0x5640bd7a2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafaffff640 1 --2- 172.21.15.107:0/3749434969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd7a5630 0x5640bd7a2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafaffff640 1 -- 172.21.15.107:0/3749434969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd79ad70 msgr2=0x5640bd79ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafaffff640 1 --2- 172.21.15.107:0/3749434969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd79ad70 0x5640bd79ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafaffff640 1 -- 172.21.15.107:0/3749434969 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640bd7b7de0 con 0x5640bd79a450 2024-05-09T08:39:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7fafaffff640 1 --2- 172.21.15.107:0/3749434969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd7b1120 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x5640bd7b7fb0 tx=0x7fafa0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e74f543475122f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.591+0000 7faf95ffb640 1 -- 172.21.15.107:0/3749434969 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa0003ae0 con 0x5640bd79a450 2024-05-09T08:39:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.592+0000 7faf95ffb640 1 -- 172.21.15.107:0/3749434969 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafa0029020 con 0x5640bd79a450 2024-05-09T08:39:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.592+0000 7faf95ffb640 1 -- 172.21.15.107:0/3749434969 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa0022650 con 0x5640bd79a450 2024-05-09T08:39:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.592+0000 7fafb9435740 1 -- 172.21.15.107:0/3749434969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 msgr2=0x5640bd7b1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.592+0000 7fafb9435740 1 --2- 172.21.15.107:0/3749434969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd7b1120 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x5640bd7b7fb0 tx=0x7fafa0005f90 comp rx=0 tx=0).stop 2024-05-09T08:39:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.592+0000 7fafb9435740 1 -- 172.21.15.107:0/3749434969 shutdown_connections 2024-05-09T08:39:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.592+0000 7fafb9435740 1 --2- 172.21.15.107:0/3749434969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd7a5630 0x5640bd7a2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.592+0000 7fafb9435740 1 --2- 172.21.15.107:0/3749434969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd79ad70 0x5640bd79ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.592+0000 7fafb9435740 1 --2- 172.21.15.107:0/3749434969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd7b1120 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.592+0000 7fafb9435740 1 -- 172.21.15.107:0/3749434969 >> 172.21.15.107:0/3749434969 conn(0x5640bd796960 msgr2=0x5640bd7a24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.592+0000 7fafb9435740 1 -- 172.21.15.107:0/3749434969 shutdown_connections 2024-05-09T08:39:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.592+0000 7fafb9435740 1 -- 172.21.15.107:0/3749434969 wait complete. 2024-05-09T08:39:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.593+0000 7fafb9435740 1 -- start start 2024-05-09T08:39:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.593+0000 7fafb9435740 1 Processor -- start 2024-05-09T08:39:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.593+0000 7fafb9435740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd7adb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.593+0000 7fafb9435740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd79ad70 0x5640bd79eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.593+0000 7fafb9435740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd7a5630 0x5640bd79f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.593+0000 7fafb9435740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fafb01aad40 con 0x5640bd79a450 2024-05-09T08:39:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.593+0000 7fafb9435740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fafb01ac1b0 con 0x5640bd7a5630 2024-05-09T08:39:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.593+0000 7fafb9435740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fafb01ab410 con 0x5640bd79ad70 2024-05-09T08:39:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.593+0000 7fafb9435740 10 client.0 _ll_register_callbacks cb 0x7fafa4009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.593+0000 7fafaffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd7adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.593+0000 7fafaffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd7adb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35840/0 (socket says 172.21.15.107:35840) 2024-05-09T08:39:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.593+0000 7fafaffff640 1 -- 172.21.15.107:0/3197411765 learned_addr learned my addr 172.21.15.107:0/3197411765 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.593+0000 7fafaf7fe640 1 --2- 172.21.15.107:0/3197411765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd79ad70 0x5640bd79eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.593+0000 7fafb4cbf640 1 --2- 172.21.15.107:0/3197411765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd7a5630 0x5640bd79f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.594+0000 7fafaffff640 1 -- 172.21.15.107:0/3197411765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd79ad70 msgr2=0x5640bd79eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.594+0000 7fafaffff640 1 --2- 172.21.15.107:0/3197411765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd79ad70 0x5640bd79eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.594+0000 7fafaffff640 1 -- 172.21.15.107:0/3197411765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd7a5630 msgr2=0x5640bd79f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.594+0000 7fafaffff640 1 --2- 172.21.15.107:0/3197411765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd7a5630 0x5640bd79f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.594+0000 7fafaffff640 1 -- 172.21.15.107:0/3197411765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640bd81a8e0 con 0x5640bd79a450 2024-05-09T08:39:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.594+0000 7fafaffff640 1 --2- 172.21.15.107:0/3197411765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd7adb80 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fafa0004550 tx=0x7fafa0004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0305074475cd3f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.594+0000 7faf8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.594+0000 7faf8f7fe640 1 -- 172.21.15.107:0/3197411765 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa0003ae0 con 0x5640bd79a450 2024-05-09T08:39:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.594+0000 7faf8f7fe640 1 -- 172.21.15.107:0/3197411765 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fafa0003c80 con 0x5640bd79a450 2024-05-09T08:39:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.594+0000 7fafb9435740 1 -- 172.21.15.107:0/3197411765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafa0029020 con 0x5640bd79a450 2024-05-09T08:39:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.594+0000 7faf8f7fe640 1 -- 172.21.15.107:0/3197411765 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fafa0041050 con 0x5640bd79a450 2024-05-09T08:39:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.594+0000 7fafb9435740 10 client.17499 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.594+0000 7fafb9435740 1 -- 172.21.15.107:0/3197411765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5640bd81ba10 con 0x5640bd79a450 2024-05-09T08:39:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.594+0000 7faf8f7fe640 1 -- 172.21.15.107:0/3197411765 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafa0029020 con 0x5640bd79a450 2024-05-09T08:39:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.595+0000 7faf8f7fe640 1 -- 172.21.15.107:0/3197411765 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fafa0004930 con 0x5640bd79a450 2024-05-09T08:39:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.595+0000 7fafb9435740 10 client.17499 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.595+0000 7fafb9435740 10 client.17499 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.595+0000 7fafb9435740 20 client.17499 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.595+0000 7fafb9435740 1 --2- 172.21.15.107:0/3197411765 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5640bd81c0a0 0x5640bd81e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.595+0000 7fafb9435740 4 client.17499 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:39:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.595+0000 7fafb9435740 1 -- 172.21.15.107:0/3197411765 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5640bd706dc0 con 0x5640bd81c0a0 2024-05-09T08:39:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.595+0000 7fafaf7fe640 1 --2- 172.21.15.107:0/3197411765 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5640bd81c0a0 0x5640bd81e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.595+0000 7fafaf7fe640 1 --2- 172.21.15.107:0/3197411765 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5640bd81c0a0 0x5640bd81e4c0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbbde3d7b07ed654 server_cookie=9199c970c88b6e8f in_seq=0 out_seq=0 2024-05-09T08:39:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.595+0000 7faf8f7fe640 10 client.17499 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.609+0000 7faf8f7fe640 1 -- 172.21.15.107:0/3197411765 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5640bd706dc0 con 0x5640bd81c0a0 2024-05-09T08:39:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.609+0000 7faf8f7fe640 10 client.17499 handle_command_reply: tid=0 2024-05-09T08:39:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.609+0000 7fafb9435740 1 client.17499 shutdown 2024-05-09T08:39:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.609+0000 7fafb9435740 1 -- 172.21.15.107:0/3197411765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 msgr2=0x5640bd7adb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.609+0000 7fafb9435740 1 --2- 172.21.15.107:0/3197411765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd7adb80 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fafa0004550 tx=0x7fafa0004580 comp rx=0 tx=0).stop 2024-05-09T08:39:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb9435740 1 -- 172.21.15.107:0/3197411765 shutdown_connections 2024-05-09T08:39:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb9435740 1 --2- 172.21.15.107:0/3197411765 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5640bd81c0a0 0x5640bd81e4c0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb9435740 1 --2- 172.21.15.107:0/3197411765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5640bd7a5630 0x5640bd79f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7faf8f7fe640 0 client.17499 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb9435740 1 --2- 172.21.15.107:0/3197411765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5640bd79ad70 0x5640bd79eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb9435740 1 --2- 172.21.15.107:0/3197411765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640bd79a450 0x5640bd7adb80 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb9435740 1 -- 172.21.15.107:0/3197411765 >> 172.21.15.107:0/3197411765 conn(0x5640bd796960 msgr2=0x5640bd7a2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb9435740 1 -- 172.21.15.107:0/3197411765 shutdown_connections 2024-05-09T08:39:25.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb9435740 1 -- 172.21.15.107:0/3197411765 wait complete. 2024-05-09T08:39:25.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb9435740 20 client.17499 trim_cache size 0 max 16384 2024-05-09T08:39:25.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:39:25.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:39:25.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:39:25.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:39:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb6722640 1 -- 172.21.15.107:0/4071444796 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf980a2da0 msgr2=0x7faf980a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb6722640 1 --2- 172.21.15.107:0/4071444796 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf980a2da0 0x7faf980a5260 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fafa8021000 tx=0x7fafa801e000 comp rx=0 tx=0).stop 2024-05-09T08:39:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb6722640 1 -- 172.21.15.107:0/4071444796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafb0105a80 msgr2=0x7fafb0198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb6722640 1 --2- 172.21.15.107:0/4071444796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafb0105a80 0x7fafb0198470 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fafa4031440 tx=0x7fafa4004040 comp rx=0 tx=0).stop 2024-05-09T08:39:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb6722640 1 -- 172.21.15.107:0/4071444796 shutdown_connections 2024-05-09T08:39:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb6722640 1 --2- 172.21.15.107:0/4071444796 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf980a2da0 0x7faf980a5260 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb6722640 1 --2- 172.21.15.107:0/4071444796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafb019c7f0 0x7fafb019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb6722640 1 --2- 172.21.15.107:0/4071444796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafb0105a80 0x7fafb0198470 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb6722640 1 --2- 172.21.15.107:0/4071444796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafb01010d0 0x7fafb0197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.610+0000 7fafb6722640 1 -- 172.21.15.107:0/4071444796 >> 172.21.15.107:0/4071444796 conn(0x7fafb0076cc0 msgr2=0x7fafb00ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.611+0000 7fafb6722640 1 -- 172.21.15.107:0/4071444796 shutdown_connections 2024-05-09T08:39:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:25.611+0000 7fafb6722640 1 -- 172.21.15.107:0/4071444796 wait complete. 2024-05-09T08:39:25.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:25 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:39:25.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:25 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:25.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:25 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:39:25.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:25 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:25.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:25 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:39:25.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:25 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:25.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:25 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:39:25.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:25 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:25.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:25 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:39:25.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:25 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:25.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:25 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:39:25.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:25 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:26.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:26 smithi107 ceph-mon[30122]: pgmap v185: 129 pgs: 129 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:39:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:26 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:39:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:26 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:26 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:39:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:26 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:39:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:26 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:26 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:39:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:26 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:26 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:39:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:26 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:26.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:26 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:26.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:26 smithi157 ceph-mon[33614]: pgmap v185: 129 pgs: 129 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:39:26.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:26 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:39:26.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:26 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:26.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:26 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:39:26.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:26 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:39:26.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:26 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:26 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:39:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:26 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:26 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:39:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:26 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:26.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:26 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:26.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:26 smithi110 ceph-mon[33622]: pgmap v185: 129 pgs: 129 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:39:26.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:26 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:39:26.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:26 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:26 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:39:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:26 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:39:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:26 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:26 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:39:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:26 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:26 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:39:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:26 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:26.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:26 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:28.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:28 smithi107 ceph-mon[30122]: pgmap v186: 129 pgs: 129 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 37 MiB/s wr, 61 op/s 2024-05-09T08:39:28.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:28 smithi157 ceph-mon[33614]: pgmap v186: 129 pgs: 129 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 37 MiB/s wr, 61 op/s 2024-05-09T08:39:28.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:28 smithi110 ceph-mon[33622]: pgmap v186: 129 pgs: 129 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 37 MiB/s wr, 61 op/s 2024-05-09T08:39:30.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:30 smithi107 ceph-mon[30122]: pgmap v187: 129 pgs: 129 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 24 MiB/s wr, 49 op/s 2024-05-09T08:39:30.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:30 smithi157 ceph-mon[33614]: pgmap v187: 129 pgs: 129 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 24 MiB/s wr, 49 op/s 2024-05-09T08:39:30.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:30 smithi110 ceph-mon[33622]: pgmap v187: 129 pgs: 129 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 24 MiB/s wr, 49 op/s 2024-05-09T08:39:31.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:31 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:39:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:31 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:39:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:31 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:39:31.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:31 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:39:31.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:31 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:39:31.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:31 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:39:31.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:31 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:39:31.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:31 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:39:31.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:31 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:39:32.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:32 smithi107 ceph-mon[30122]: pgmap v188: 129 pgs: 129 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 24 MiB/s wr, 49 op/s 2024-05-09T08:39:32.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:32 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:39:32.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:32 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:39:32.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:32 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:39:32.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:32 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:39:32.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:32 smithi157 ceph-mon[33614]: pgmap v188: 129 pgs: 129 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 24 MiB/s wr, 49 op/s 2024-05-09T08:39:32.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:32 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:39:32.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:32 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:39:32.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:32 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:39:32.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:32 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:39:32.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:32 smithi110 ceph-mon[33622]: pgmap v188: 129 pgs: 129 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 24 MiB/s wr, 49 op/s 2024-05-09T08:39:32.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:32 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:39:32.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:32 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:39:32.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:32 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:39:32.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:32 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:39:34.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:34 smithi107 ceph-mon[30122]: pgmap v189: 129 pgs: 129 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 36 MiB/s wr, 57 op/s 2024-05-09T08:39:34.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:34 smithi157 ceph-mon[33614]: pgmap v189: 129 pgs: 129 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 36 MiB/s wr, 57 op/s 2024-05-09T08:39:34.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:34 smithi110 ceph-mon[33622]: pgmap v189: 129 pgs: 129 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 36 MiB/s wr, 57 op/s 2024-05-09T08:39:36.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:36 smithi107 ceph-mon[30122]: pgmap v190: 129 pgs: 129 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 24 MiB/s wr, 49 op/s 2024-05-09T08:39:36.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:36 smithi157 ceph-mon[33614]: pgmap v190: 129 pgs: 129 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 24 MiB/s wr, 49 op/s 2024-05-09T08:39:36.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:36 smithi110 ceph-mon[33622]: pgmap v190: 129 pgs: 129 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 24 MiB/s wr, 49 op/s 2024-05-09T08:39:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 -- 172.21.15.107:0/96878498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc101120 msgr2=0x7fdbcc10d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 --2- 172.21.15.107:0/96878498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc101120 0x7fdbcc10d5e0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4009cd0 tx=0x7fdbc402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 -- 172.21.15.107:0/96878498 shutdown_connections 2024-05-09T08:39:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 --2- 172.21.15.107:0/96878498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc101120 0x7fdbcc10d5e0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 --2- 172.21.15.107:0/96878498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdbcc100760 0x7fdbcc100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 --2- 172.21.15.107:0/96878498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbcc106720 0x7fdbcc106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 -- 172.21.15.107:0/96878498 >> 172.21.15.107:0/96878498 conn(0x7fdbcc0778b0 msgr2=0x7fdbcc077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 -- 172.21.15.107:0/96878498 shutdown_connections 2024-05-09T08:39:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 -- 172.21.15.107:0/96878498 wait complete. 2024-05-09T08:39:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 Processor -- start 2024-05-09T08:39:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 -- start start 2024-05-09T08:39:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc100760 0x7fdbcc197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbcc101120 0x7fdbcc198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdbcc106720 0x7fdbcc19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc111ae0 con 0x7fdbcc100760 2024-05-09T08:39:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc111960 con 0x7fdbcc101120 2024-05-09T08:39:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.716+0000 7fdbd3cd2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc111c60 con 0x7fdbcc106720 2024-05-09T08:39:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.717+0000 7fdbd1a6e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc100760 0x7fdbcc197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.717+0000 7fdbd1a6e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc100760 0x7fdbcc197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36820/0 (socket says 172.21.15.107:36820) 2024-05-09T08:39:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.717+0000 7fdbd1a6e640 1 -- 172.21.15.107:0/2542299033 learned_addr learned my addr 172.21.15.107:0/2542299033 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.717+0000 7fdbd126d640 1 --2- 172.21.15.107:0/2542299033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbcc101120 0x7fdbcc198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.717+0000 7fdbd226f640 1 --2- 172.21.15.107:0/2542299033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdbcc106720 0x7fdbcc19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.717+0000 7fdbd1a6e640 1 -- 172.21.15.107:0/2542299033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdbcc106720 msgr2=0x7fdbcc19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.717+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/2542299033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdbcc106720 0x7fdbcc19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.717+0000 7fdbd1a6e640 1 -- 172.21.15.107:0/2542299033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbcc101120 msgr2=0x7fdbcc198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.717+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/2542299033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbcc101120 0x7fdbcc198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.717+0000 7fdbd1a6e640 1 -- 172.21.15.107:0/2542299033 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbcc19cf60 con 0x7fdbcc100760 2024-05-09T08:39:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.717+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/2542299033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc100760 0x7fdbcc197fc0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fdbc800d950 tx=0x7fdbc800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.717+0000 7fdbc2ffd640 1 -- 172.21.15.107:0/2542299033 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8017070 con 0x7fdbcc100760 2024-05-09T08:39:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.717+0000 7fdbd3cd2640 1 -- 172.21.15.107:0/2542299033 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbcc19d250 con 0x7fdbcc100760 2024-05-09T08:39:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.717+0000 7fdbc2ffd640 1 -- 172.21.15.107:0/2542299033 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdbc8009c30 con 0x7fdbcc100760 2024-05-09T08:39:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.717+0000 7fdbc2ffd640 1 -- 172.21.15.107:0/2542299033 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc800f620 con 0x7fdbcc100760 2024-05-09T08:39:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.717+0000 7fdbd3cd2640 1 -- 172.21.15.107:0/2542299033 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbcc1a4ae0 con 0x7fdbcc100760 2024-05-09T08:39:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.720+0000 7fdbc2ffd640 1 -- 172.21.15.107:0/2542299033 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fdbc80109a0 con 0x7fdbcc100760 2024-05-09T08:39:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.720+0000 7fdbd69de740 1 Processor -- start 2024-05-09T08:39:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.721+0000 7fdbd69de740 1 -- start start 2024-05-09T08:39:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.721+0000 7fdbc2ffd640 1 --2- 172.21.15.107:0/2542299033 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdbb80a2da0 0x7fdbb80a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.721+0000 7fdbd69de740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49330 0x55ffe0e49710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.721+0000 7fdbd69de740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e49c50 0x55ffe0e58500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.721+0000 7fdbd69de740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e58a40 0x55ffe0e5ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.721+0000 7fdbd69de740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ffe0dae340 con 0x55ffe0e58a40 2024-05-09T08:39:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.721+0000 7fdbd69de740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ffe0d56290 con 0x55ffe0e49330 2024-05-09T08:39:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.721+0000 7fdbd69de740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ffe0daa300 con 0x55ffe0e49c50 2024-05-09T08:39:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.721+0000 7fdbd226f640 1 --2- 172.21.15.107:0/2542299033 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdbb80a2da0 0x7fdbb80a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.721+0000 7fdbc2ffd640 1 -- 172.21.15.107:0/2542299033 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fdbc80cf2d0 con 0x7fdbcc100760 2024-05-09T08:39:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.721+0000 7fdbd226f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e58a40 0x55ffe0e5ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.721+0000 7fdbd226f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e58a40 0x55ffe0e5ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36834/0 (socket says 172.21.15.107:36834) 2024-05-09T08:39:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.721+0000 7fdbd226f640 1 -- 172.21.15.107:0/2630824902 learned_addr learned my addr 172.21.15.107:0/2630824902 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.721+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/2630824902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49330 0x55ffe0e49710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.721+0000 7fdbd126d640 1 --2- 172.21.15.107:0/2630824902 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e49c50 0x55ffe0e58500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.722+0000 7fdbd1a6e640 1 -- 172.21.15.107:0/2630824902 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e49c50 msgr2=0x55ffe0e58500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.722+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/2630824902 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e49c50 0x55ffe0e58500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.722+0000 7fdbd1a6e640 1 -- 172.21.15.107:0/2630824902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e58a40 msgr2=0x55ffe0e5ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.722+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/2630824902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e58a40 0x55ffe0e5ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.722+0000 7fdbd1a6e640 1 -- 172.21.15.107:0/2630824902 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffe0d7a8e0 con 0x55ffe0e49330 2024-05-09T08:39:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.722+0000 7fdbd226f640 1 --2- 172.21.15.107:0/2630824902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e58a40 0x55ffe0e5ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.722+0000 7fdbd226f640 1 --2- 172.21.15.107:0/2542299033 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdbb80a2da0 0x7fdbb80a5260 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdbc402ae00 tx=0x7fdbc40316d0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.722+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/2630824902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49330 0x55ffe0e49710 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdbc80cc6c0 tx=0x7fdbc8009de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5dae3241b813b3c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.722+0000 7fdbb7fff640 1 -- 172.21.15.107:0/2630824902 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8020040 con 0x55ffe0e49330 2024-05-09T08:39:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.722+0000 7fdbb7fff640 1 -- 172.21.15.107:0/2630824902 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdbc80a4100 con 0x55ffe0e49330 2024-05-09T08:39:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.722+0000 7fdbb7fff640 1 -- 172.21.15.107:0/2630824902 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8097290 con 0x55ffe0e49330 2024-05-09T08:39:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.723+0000 7fdbd69de740 1 -- 172.21.15.107:0/2630824902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49330 msgr2=0x55ffe0e49710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.723+0000 7fdbd69de740 1 --2- 172.21.15.107:0/2630824902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49330 0x55ffe0e49710 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdbc80cc6c0 tx=0x7fdbc8009de0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.723+0000 7fdbd69de740 1 -- 172.21.15.107:0/2630824902 shutdown_connections 2024-05-09T08:39:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.723+0000 7fdbd69de740 1 --2- 172.21.15.107:0/2630824902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e58a40 0x55ffe0e5ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.723+0000 7fdbd69de740 1 --2- 172.21.15.107:0/2630824902 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e49c50 0x55ffe0e58500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.723+0000 7fdbd69de740 1 --2- 172.21.15.107:0/2630824902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49330 0x55ffe0e49710 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.723+0000 7fdbd69de740 1 -- 172.21.15.107:0/2630824902 >> 172.21.15.107:0/2630824902 conn(0x55ffe0e45960 msgr2=0x55ffe0e45d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.723+0000 7fdbd69de740 1 -- 172.21.15.107:0/2630824902 shutdown_connections 2024-05-09T08:39:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.723+0000 7fdbd69de740 1 -- 172.21.15.107:0/2630824902 wait complete. 2024-05-09T08:39:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.723+0000 7fdbd69de740 1 -- start start 2024-05-09T08:39:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.723+0000 7fdbd69de740 1 Processor -- start 2024-05-09T08:39:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd69de740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e49330 0x55ffe0e559a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd69de740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e49c50 0x55ffe0e55ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd69de740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e58a40 0x55ffe0e56420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd69de740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ffe0e5b670 con 0x55ffe0e49c50 2024-05-09T08:39:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd69de740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ffe0d69790 con 0x55ffe0e58a40 2024-05-09T08:39:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd69de740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ffe0e5b7f0 con 0x55ffe0e49330 2024-05-09T08:39:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd69de740 10 client.0 _ll_register_callbacks cb 0x55ffe0da4a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd126d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e49c50 0x55ffe0e55ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd126d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e49c50 0x55ffe0e55ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36840/0 (socket says 172.21.15.107:36840) 2024-05-09T08:39:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd126d640 1 -- 172.21.15.107:0/1486070338 learned_addr learned my addr 172.21.15.107:0/1486070338 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd226f640 1 --2- 172.21.15.107:0/1486070338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e58a40 0x55ffe0e56420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/1486070338 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e49330 0x55ffe0e559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd126d640 1 -- 172.21.15.107:0/1486070338 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e49330 msgr2=0x55ffe0e559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd126d640 1 --2- 172.21.15.107:0/1486070338 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e49330 0x55ffe0e559a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd126d640 1 -- 172.21.15.107:0/1486070338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e58a40 msgr2=0x55ffe0e56420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd126d640 1 --2- 172.21.15.107:0/1486070338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e58a40 0x55ffe0e56420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd126d640 1 -- 172.21.15.107:0/1486070338 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffe0ac2210 con 0x55ffe0e49c50 2024-05-09T08:39:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd226f640 1 --2- 172.21.15.107:0/1486070338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e58a40 0x55ffe0e56420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/1486070338 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e49330 0x55ffe0e559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:39:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd126d640 1 --2- 172.21.15.107:0/1486070338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e49c50 0x55ffe0e55ee0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc00ba40 tx=0x7fdbbc00bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b678f77e259fccc8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbb57fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbb57fa640 1 -- 172.21.15.107:0/1486070338 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc011440 con 0x55ffe0e49c50 2024-05-09T08:39:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbb57fa640 1 -- 172.21.15.107:0/1486070338 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fdbbc0115e0 con 0x55ffe0e49c50 2024-05-09T08:39:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.724+0000 7fdbd69de740 1 -- 172.21.15.107:0/1486070338 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbc80a4100 con 0x55ffe0e49c50 2024-05-09T08:39:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.725+0000 7fdbb57fa640 1 -- 172.21.15.107:0/1486070338 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdbbc0117e0 con 0x55ffe0e49c50 2024-05-09T08:39:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.725+0000 7fdbd69de740 10 client.17511 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.725+0000 7fdbd69de740 1 -- 172.21.15.107:0/1486070338 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ffe0b8e620 con 0x55ffe0e49c50 2024-05-09T08:39:38.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.725+0000 7fdbb57fa640 1 -- 172.21.15.107:0/1486070338 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdbc80a4100 con 0x55ffe0e49c50 2024-05-09T08:39:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.725+0000 7fdbb57fa640 1 -- 172.21.15.107:0/1486070338 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fdbbc01b980 con 0x55ffe0e49c50 2024-05-09T08:39:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.725+0000 7fdbd69de740 10 client.17511 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.725+0000 7fdbd69de740 10 client.17511 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.725+0000 7fdbd69de740 20 client.17511 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.725+0000 7fdbd69de740 1 --2- 172.21.15.107:0/1486070338 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ffe0e688e0 0x55ffe0e6acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.725+0000 7fdbd69de740 4 client.17511 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.725+0000 7fdbd69de740 1 -- 172.21.15.107:0/1486070338 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffe0dbba70 con 0x55ffe0e688e0 2024-05-09T08:39:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.726+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/1486070338 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ffe0e688e0 0x55ffe0e6acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.726+0000 7fdbb57fa640 10 client.17511 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.726+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/1486070338 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ffe0e688e0 0x55ffe0e6acd0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b95110348cbc57f4 server_cookie=df01cf4d639e0a24 in_seq=0 out_seq=0 2024-05-09T08:39:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.727+0000 7fdbb57fa640 1 -- 172.21.15.107:0/1486070338 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ffe0dbba70 con 0x55ffe0e688e0 2024-05-09T08:39:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.727+0000 7fdbb57fa640 10 client.17511 handle_command_reply: tid=0 2024-05-09T08:39:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.727+0000 7fdbd69de740 1 client.17511 shutdown 2024-05-09T08:39:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.727+0000 7fdbd69de740 1 -- 172.21.15.107:0/1486070338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e49c50 msgr2=0x55ffe0e55ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.727+0000 7fdbd69de740 1 --2- 172.21.15.107:0/1486070338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e49c50 0x55ffe0e55ee0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc00ba40 tx=0x7fdbbc00bf00 comp rx=0 tx=0).stop 2024-05-09T08:39:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.727+0000 7fdbd69de740 1 -- 172.21.15.107:0/1486070338 shutdown_connections 2024-05-09T08:39:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.727+0000 7fdbd69de740 1 --2- 172.21.15.107:0/1486070338 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ffe0e688e0 0x55ffe0e6acd0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.727+0000 7fdbd69de740 1 --2- 172.21.15.107:0/1486070338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e58a40 0x55ffe0e56420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.727+0000 7fdbd69de740 1 --2- 172.21.15.107:0/1486070338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e49c50 0x55ffe0e55ee0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.727+0000 7fdbd69de740 1 --2- 172.21.15.107:0/1486070338 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e49330 0x55ffe0e559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.727+0000 7fdbb57fa640 0 client.17511 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.727+0000 7fdbd69de740 1 -- 172.21.15.107:0/1486070338 >> 172.21.15.107:0/1486070338 conn(0x55ffe0e45960 msgr2=0x55ffe0e4b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.727+0000 7fdbd69de740 1 -- 172.21.15.107:0/1486070338 shutdown_connections 2024-05-09T08:39:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.727+0000 7fdbd69de740 1 -- 172.21.15.107:0/1486070338 wait complete. 2024-05-09T08:39:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.727+0000 7fdbd69de740 20 client.17511 trim_cache size 0 max 16384 2024-05-09T08:39:38.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.743+0000 7fdbd69de740 1 Processor -- start 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.743+0000 7fdbd69de740 1 -- start start 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd69de740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e49330 0x55ffe0e580d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd69de740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49c50 0x55ffe0e56240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd69de740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e58a40 0x55ffe0e4b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd69de740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1af210 con 0x55ffe0e49330 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd69de740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1acae0 con 0x55ffe0e49c50 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd69de740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1aa930 con 0x55ffe0e58a40 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd1a6e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e49330 0x55ffe0e580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd1a6e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e49330 0x55ffe0e580d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36848/0 (socket says 172.21.15.107:36848) 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd1a6e640 1 -- 172.21.15.107:0/2228575404 learned_addr learned my addr 172.21.15.107:0/2228575404 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd126d640 1 --2- 172.21.15.107:0/2228575404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49c50 0x55ffe0e56240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd1a6e640 1 -- 172.21.15.107:0/2228575404 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e58a40 msgr2=0x55ffe0e4b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/2228575404 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e58a40 0x55ffe0e4b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd1a6e640 1 -- 172.21.15.107:0/2228575404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49c50 msgr2=0x55ffe0e56240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/2228575404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49c50 0x55ffe0e56240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd1a6e640 1 -- 172.21.15.107:0/2228575404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffe0e40bc0 con 0x55ffe0e49330 2024-05-09T08:39:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd126d640 1 --2- 172.21.15.107:0/2228575404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49c50 0x55ffe0e56240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:39:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/2228575404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e49330 0x55ffe0e580d0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8097f10 tx=0x7fdbc8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=470b713a6440f9e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.744+0000 7fdbb77fe640 1 -- 172.21.15.107:0/2228575404 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8020040 con 0x55ffe0e49330 2024-05-09T08:39:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.745+0000 7fdbb77fe640 1 -- 172.21.15.107:0/2228575404 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdbc80a4100 con 0x55ffe0e49330 2024-05-09T08:39:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.745+0000 7fdbb77fe640 1 -- 172.21.15.107:0/2228575404 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8010460 con 0x55ffe0e49330 2024-05-09T08:39:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.745+0000 7fdbd69de740 1 -- 172.21.15.107:0/2228575404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e49330 msgr2=0x55ffe0e580d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.745+0000 7fdbd69de740 1 --2- 172.21.15.107:0/2228575404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e49330 0x55ffe0e580d0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8097f10 tx=0x7fdbc8005880 comp rx=0 tx=0).stop 2024-05-09T08:39:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.745+0000 7fdbd69de740 1 -- 172.21.15.107:0/2228575404 shutdown_connections 2024-05-09T08:39:38.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.745+0000 7fdbd69de740 1 --2- 172.21.15.107:0/2228575404 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e58a40 0x55ffe0e4b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.745+0000 7fdbd69de740 1 --2- 172.21.15.107:0/2228575404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49c50 0x55ffe0e56240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.745+0000 7fdbd69de740 1 --2- 172.21.15.107:0/2228575404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e49330 0x55ffe0e580d0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.745+0000 7fdbd69de740 1 -- 172.21.15.107:0/2228575404 >> 172.21.15.107:0/2228575404 conn(0x55ffe0e45960 msgr2=0x55ffe0e4aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:38.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.745+0000 7fdbd69de740 1 -- 172.21.15.107:0/2228575404 shutdown_connections 2024-05-09T08:39:38.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.745+0000 7fdbd69de740 1 -- 172.21.15.107:0/2228575404 wait complete. 2024-05-09T08:39:38.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.746+0000 7fdbd69de740 1 -- start start 2024-05-09T08:39:38.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.746+0000 7fdbd69de740 1 Processor -- start 2024-05-09T08:39:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.746+0000 7fdbd69de740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49330 0x55ffe0e5c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.746+0000 7fdbd69de740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e49c50 0x55ffe0e5cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.746+0000 7fdbd69de740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e58a40 0x55ffe0e5d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.746+0000 7fdbd69de740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1aafa0 con 0x55ffe0e58a40 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.746+0000 7fdbd69de740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1ac410 con 0x55ffe0e49330 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.746+0000 7fdbd69de740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1ab670 con 0x55ffe0e49c50 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.746+0000 7fdbd69de740 10 client.0 _ll_register_callbacks cb 0x7fdbbc0042b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.746+0000 7fdbd1a6e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49330 0x55ffe0e5c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.746+0000 7fdbd226f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e58a40 0x55ffe0e5d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.747+0000 7fdbd1a6e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49330 0x55ffe0e5c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:50186/0 (socket says 172.21.15.107:50186) 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.747+0000 7fdbd1a6e640 1 -- 172.21.15.107:0/1281772505 learned_addr learned my addr 172.21.15.107:0/1281772505 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.747+0000 7fdbd126d640 1 --2- 172.21.15.107:0/1281772505 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e49c50 0x55ffe0e5cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.747+0000 7fdbd1a6e640 1 -- 172.21.15.107:0/1281772505 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e49c50 msgr2=0x55ffe0e5cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.747+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/1281772505 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e49c50 0x55ffe0e5cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.747+0000 7fdbd1a6e640 1 -- 172.21.15.107:0/1281772505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e58a40 msgr2=0x55ffe0e5d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:38.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.747+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/1281772505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e58a40 0x55ffe0e5d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.747+0000 7fdbd1a6e640 1 -- 172.21.15.107:0/1281772505 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffe0e49a80 con 0x55ffe0e49330 2024-05-09T08:39:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.747+0000 7fdbd226f640 1 --2- 172.21.15.107:0/1281772505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e58a40 0x55ffe0e5d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.747+0000 7fdbd1a6e640 1 --2- 172.21.15.107:0/1281772505 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49330 0x55ffe0e5c5d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8009f90 tx=0x7fdbc800bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2aeca991836a64b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.747+0000 7fdbb57fa640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:39:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.747+0000 7fdbb57fa640 1 -- 172.21.15.107:0/1281772505 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8020070 con 0x55ffe0e49330 2024-05-09T08:39:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.747+0000 7fdbd69de740 1 -- 172.21.15.107:0/1281772505 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbc80a4100 con 0x55ffe0e49330 2024-05-09T08:39:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.747+0000 7fdbb57fa640 1 -- 172.21.15.107:0/1281772505 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fdbc80b4220 con 0x55ffe0e49330 2024-05-09T08:39:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.747+0000 7fdbb57fa640 1 -- 172.21.15.107:0/1281772505 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdbc80b4460 con 0x55ffe0e49330 2024-05-09T08:39:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.747+0000 7fdbd69de740 10 client.26591 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.748+0000 7fdbd69de740 1 -- 172.21.15.107:0/1281772505 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ffe0ecf080 con 0x55ffe0e49330 2024-05-09T08:39:38.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.748+0000 7fdbb57fa640 1 -- 172.21.15.107:0/1281772505 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdbc80a4100 con 0x55ffe0e49330 2024-05-09T08:39:38.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.748+0000 7fdbb57fa640 1 -- 172.21.15.107:0/1281772505 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fdbc809a400 con 0x55ffe0e49330 2024-05-09T08:39:38.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.748+0000 7fdbd69de740 10 client.26591 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:38.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.748+0000 7fdbd69de740 10 client.26591 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:38.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.748+0000 7fdbd69de740 20 client.26591 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:38.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.748+0000 7fdbd69de740 1 --2- 172.21.15.107:0/1281772505 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ffe0ecf2b0 0x55ffe0ed16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:38.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.748+0000 7fdbd69de740 4 client.26591 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:39:38.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.748+0000 7fdbd69de740 1 -- 172.21.15.107:0/1281772505 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ffe0db5dc0 con 0x55ffe0ecf2b0 2024-05-09T08:39:38.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.749+0000 7fdbd126d640 1 --2- 172.21.15.107:0/1281772505 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ffe0ecf2b0 0x55ffe0ed16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:38.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.749+0000 7fdbb57fa640 10 client.26591 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:38.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.749+0000 7fdbd126d640 1 --2- 172.21.15.107:0/1281772505 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ffe0ecf2b0 0x55ffe0ed16a0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6aa3b62b198c5f0f server_cookie=cc4f0822000521e3 in_seq=0 out_seq=0 2024-05-09T08:39:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.764+0000 7fdbb57fa640 1 -- 172.21.15.107:0/1281772505 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ffe0db5dc0 con 0x55ffe0ecf2b0 2024-05-09T08:39:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.764+0000 7fdbb57fa640 10 client.26591 handle_command_reply: tid=0 2024-05-09T08:39:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.764+0000 7fdbd69de740 1 client.26591 shutdown 2024-05-09T08:39:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.764+0000 7fdbd69de740 1 -- 172.21.15.107:0/1281772505 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49330 msgr2=0x55ffe0e5c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.764+0000 7fdbd69de740 1 --2- 172.21.15.107:0/1281772505 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49330 0x55ffe0e5c5d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8009f90 tx=0x7fdbc800bf60 comp rx=0 tx=0).stop 2024-05-09T08:39:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.764+0000 7fdbd69de740 1 -- 172.21.15.107:0/1281772505 shutdown_connections 2024-05-09T08:39:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.764+0000 7fdbd69de740 1 --2- 172.21.15.107:0/1281772505 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ffe0ecf2b0 0x55ffe0ed16a0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.764+0000 7fdbd69de740 1 --2- 172.21.15.107:0/1281772505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ffe0e58a40 0x55ffe0e5d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.764+0000 7fdbb57fa640 0 client.26591 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.764+0000 7fdbd69de740 1 --2- 172.21.15.107:0/1281772505 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ffe0e49c50 0x55ffe0e5cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.764+0000 7fdbd69de740 1 --2- 172.21.15.107:0/1281772505 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ffe0e49330 0x55ffe0e5c5d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.764+0000 7fdbd69de740 1 -- 172.21.15.107:0/1281772505 >> 172.21.15.107:0/1281772505 conn(0x55ffe0e45960 msgr2=0x55ffe0e4b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:38.765 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.764+0000 7fdbd69de740 1 -- 172.21.15.107:0/1281772505 shutdown_connections 2024-05-09T08:39:38.765 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.764+0000 7fdbd69de740 1 -- 172.21.15.107:0/1281772505 wait complete. 2024-05-09T08:39:38.765 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.764+0000 7fdbd69de740 20 client.26591 trim_cache size 0 max 16384 2024-05-09T08:39:38.765 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:39:38.765 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:39:38.765 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:39:38.765 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:39:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.765+0000 7fdbd3cd2640 1 -- 172.21.15.107:0/2542299033 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdbb80a2da0 msgr2=0x7fdbb80a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.765+0000 7fdbd3cd2640 1 --2- 172.21.15.107:0/2542299033 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdbb80a2da0 0x7fdbb80a5260 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdbc402ae00 tx=0x7fdbc40316d0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.765+0000 7fdbd3cd2640 1 -- 172.21.15.107:0/2542299033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc100760 msgr2=0x7fdbcc197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.765+0000 7fdbd3cd2640 1 --2- 172.21.15.107:0/2542299033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc100760 0x7fdbcc197fc0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fdbc800d950 tx=0x7fdbc800de10 comp rx=0 tx=0).stop 2024-05-09T08:39:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.765+0000 7fdbd3cd2640 1 -- 172.21.15.107:0/2542299033 shutdown_connections 2024-05-09T08:39:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.765+0000 7fdbd3cd2640 1 --2- 172.21.15.107:0/2542299033 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdbb80a2da0 0x7fdbb80a5260 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.765+0000 7fdbd3cd2640 1 --2- 172.21.15.107:0/2542299033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdbcc106720 0x7fdbcc19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.765+0000 7fdbd3cd2640 1 --2- 172.21.15.107:0/2542299033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbcc101120 0x7fdbcc198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.765+0000 7fdbd3cd2640 1 --2- 172.21.15.107:0/2542299033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc100760 0x7fdbcc197fc0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.765+0000 7fdbd3cd2640 1 -- 172.21.15.107:0/2542299033 >> 172.21.15.107:0/2542299033 conn(0x7fdbcc0778b0 msgr2=0x7fdbcc103280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.765+0000 7fdbd3cd2640 1 -- 172.21.15.107:0/2542299033 shutdown_connections 2024-05-09T08:39:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:38.765+0000 7fdbd3cd2640 1 -- 172.21.15.107:0/2542299033 wait complete. 2024-05-09T08:39:38.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:38 smithi107 ceph-mon[30122]: pgmap v191: 129 pgs: 129 active+clean; 8.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 37 MiB/s wr, 57 op/s 2024-05-09T08:39:38.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:38 smithi157 ceph-mon[33614]: pgmap v191: 129 pgs: 129 active+clean; 8.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 37 MiB/s wr, 57 op/s 2024-05-09T08:39:38.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:38 smithi110 ceph-mon[33622]: pgmap v191: 129 pgs: 129 active+clean; 8.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 37 MiB/s wr, 57 op/s 2024-05-09T08:39:40.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:40 smithi107 ceph-mon[30122]: pgmap v192: 129 pgs: 129 active+clean; 8.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:39:40.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:40 smithi157 ceph-mon[33614]: pgmap v192: 129 pgs: 129 active+clean; 8.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:39:40.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:40 smithi110 ceph-mon[33622]: pgmap v192: 129 pgs: 129 active+clean; 8.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:39:42.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:42 smithi107 ceph-mon[30122]: pgmap v193: 129 pgs: 129 active+clean; 8.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:39:42.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:42 smithi157 ceph-mon[33614]: pgmap v193: 129 pgs: 129 active+clean; 8.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:39:42.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:42 smithi110 ceph-mon[33622]: pgmap v193: 129 pgs: 129 active+clean; 8.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:39:44.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:44 smithi107 ceph-mon[30122]: pgmap v194: 129 pgs: 129 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:44.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:44 smithi157 ceph-mon[33614]: pgmap v194: 129 pgs: 129 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:44.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:44 smithi110 ceph-mon[33622]: pgmap v194: 129 pgs: 129 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:46.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:46 smithi107 ceph-mon[30122]: pgmap v195: 129 pgs: 129 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:39:46.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:46 smithi157 ceph-mon[33614]: pgmap v195: 129 pgs: 129 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:39:46.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:46 smithi110 ceph-mon[33622]: pgmap v195: 129 pgs: 129 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:39:48.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:48 smithi107 ceph-mon[30122]: pgmap v196: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:48.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:48 smithi157 ceph-mon[33614]: pgmap v196: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:48.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:48 smithi110 ceph-mon[33622]: pgmap v196: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:50.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:50 smithi107 ceph-mon[30122]: pgmap v197: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-05-09T08:39:50.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:50 smithi157 ceph-mon[33614]: pgmap v197: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-05-09T08:39:50.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:50 smithi110 ceph-mon[33622]: pgmap v197: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-05-09T08:39:51.870 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.868+0000 7f7cc842c640 1 -- 172.21.15.107:0/3780232873 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cc0106720 msgr2=0x7f7cc0106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.868+0000 7f7cc842c640 1 --2- 172.21.15.107:0/3780232873 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cc0106720 0x7f7cc0106b00 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc009cd0 tx=0x7f7cbc02a910 comp rx=0 tx=0).stop 2024-05-09T08:39:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.868+0000 7f7cc842c640 1 -- 172.21.15.107:0/3780232873 shutdown_connections 2024-05-09T08:39:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.868+0000 7f7cc842c640 1 --2- 172.21.15.107:0/3780232873 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cc0101120 0x7f7cc010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.868+0000 7f7cc842c640 1 --2- 172.21.15.107:0/3780232873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7cc0100760 0x7f7cc0100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.868+0000 7f7cc842c640 1 --2- 172.21.15.107:0/3780232873 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cc0106720 0x7f7cc0106b00 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.868+0000 7f7cc842c640 1 -- 172.21.15.107:0/3780232873 >> 172.21.15.107:0/3780232873 conn(0x7f7cc00778b0 msgr2=0x7f7cc0077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.868+0000 7f7cc842c640 1 -- 172.21.15.107:0/3780232873 shutdown_connections 2024-05-09T08:39:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.868+0000 7f7cc842c640 1 -- 172.21.15.107:0/3780232873 wait complete. 2024-05-09T08:39:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.869+0000 7f7cc842c640 1 Processor -- start 2024-05-09T08:39:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.869+0000 7f7cc842c640 1 -- start start 2024-05-09T08:39:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.869+0000 7f7cc842c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cc0100760 0x7f7cc0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.869+0000 7f7cc842c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7cc0101120 0x7f7cc0198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.869+0000 7f7cc842c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cc0106720 0x7f7cc019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.869+0000 7f7cc842c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc00698e0 con 0x7f7cc0106720 2024-05-09T08:39:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.869+0000 7f7cc842c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc0069760 con 0x7f7cc0100760 2024-05-09T08:39:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.869+0000 7f7cc842c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc0069a60 con 0x7f7cc0101120 2024-05-09T08:39:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.869+0000 7f7cc69c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cc0106720 0x7f7cc019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.869+0000 7f7cc69c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cc0106720 0x7f7cc019c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33250/0 (socket says 172.21.15.107:33250) 2024-05-09T08:39:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.869+0000 7f7cc69c9640 1 -- 172.21.15.107:0/1521718000 learned_addr learned my addr 172.21.15.107:0/1521718000 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.869+0000 7f7cc61c8640 1 --2- 172.21.15.107:0/1521718000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cc0100760 0x7f7cc0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.870+0000 7f7cc59c7640 1 --2- 172.21.15.107:0/1521718000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7cc0101120 0x7f7cc0198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.870+0000 7f7cc69c9640 1 -- 172.21.15.107:0/1521718000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7cc0101120 msgr2=0x7f7cc0198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.870+0000 7f7cc69c9640 1 --2- 172.21.15.107:0/1521718000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7cc0101120 0x7f7cc0198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.870+0000 7f7cc69c9640 1 -- 172.21.15.107:0/1521718000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cc0100760 msgr2=0x7f7cc0198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:51.873 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.870+0000 7f7cc69c9640 1 --2- 172.21.15.107:0/1521718000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cc0100760 0x7f7cc0198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.873 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.870+0000 7f7cc69c9640 1 -- 172.21.15.107:0/1521718000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc019d030 con 0x7f7cc0106720 2024-05-09T08:39:51.873 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.870+0000 7f7cc69c9640 1 --2- 172.21.15.107:0/1521718000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cc0106720 0x7f7cc019c900 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f7cb800d750 tx=0x7f7cb800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:51.873 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.870+0000 7f7cb77fe640 1 -- 172.21.15.107:0/1521718000 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8017070 con 0x7f7cc0106720 2024-05-09T08:39:51.873 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.870+0000 7f7cc842c640 1 -- 172.21.15.107:0/1521718000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cc019d320 con 0x7f7cc0106720 2024-05-09T08:39:51.873 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.870+0000 7f7cb77fe640 1 -- 172.21.15.107:0/1521718000 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cb80054d0 con 0x7f7cc0106720 2024-05-09T08:39:51.873 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.870+0000 7f7cb77fe640 1 -- 172.21.15.107:0/1521718000 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8004c60 con 0x7f7cc0106720 2024-05-09T08:39:51.873 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.870+0000 7f7cc842c640 1 -- 172.21.15.107:0/1521718000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cc01a4bb0 con 0x7f7cc0106720 2024-05-09T08:39:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.872+0000 7f7cb77fe640 1 -- 172.21.15.107:0/1521718000 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7cb8004e00 con 0x7f7cc0106720 2024-05-09T08:39:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.873+0000 7f7cb77fe640 1 --2- 172.21.15.107:0/1521718000 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7cac0a2da0 0x7f7cac0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.873+0000 7f7ccb20e740 1 Processor -- start 2024-05-09T08:39:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7cc61c8640 1 --2- 172.21.15.107:0/1521718000 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7cac0a2da0 0x7f7cac0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7cb77fe640 1 -- 172.21.15.107:0/1521718000 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f7cb80cf170 con 0x7f7cc0106720 2024-05-09T08:39:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7ccb20e740 1 -- start start 2024-05-09T08:39:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7ccb20e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193b0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7ccb20e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 0x5648193bb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7ccb20e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193bfa40 0x5648193c1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7ccb20e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564819315340 con 0x5648193b0450 2024-05-09T08:39:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7ccb20e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5648192bd290 con 0x5648193b0d70 2024-05-09T08:39:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7ccb20e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564819311300 con 0x5648193bfa40 2024-05-09T08:39:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7cc59c7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 0x5648193bb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7cc59c7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 0x5648193bb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:36908/0 (socket says 172.21.15.107:36908) 2024-05-09T08:39:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7cc59c7640 1 -- 172.21.15.107:0/961580129 learned_addr learned my addr 172.21.15.107:0/961580129 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7cc61c8640 1 --2- 172.21.15.107:0/961580129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7cc69c9640 1 --2- 172.21.15.107:0/961580129 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193bfa40 0x5648193c1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7cc59c7640 1 -- 172.21.15.107:0/961580129 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193bfa40 msgr2=0x5648193c1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7cc59c7640 1 --2- 172.21.15.107:0/961580129 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193bfa40 0x5648193c1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7cc61c8640 1 --2- 172.21.15.107:0/1521718000 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7cac0a2da0 0x7f7cac0a5260 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc005e30 tx=0x7f7cbc009360 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7cc59c7640 1 -- 172.21.15.107:0/961580129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 msgr2=0x5648193b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7cc59c7640 1 --2- 172.21.15.107:0/961580129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.874+0000 7f7cc59c7640 1 -- 172.21.15.107:0/961580129 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648192e18e0 con 0x5648193b0d70 2024-05-09T08:39:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.875+0000 7f7cc59c7640 1 --2- 172.21.15.107:0/961580129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 0x5648193bb0f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7cb000b7f0 tx=0x7f7cb000bcb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc38093d34f04499 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.875+0000 7f7cb4bfe640 1 -- 172.21.15.107:0/961580129 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb000d950 con 0x5648193b0d70 2024-05-09T08:39:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.875+0000 7f7cb4bfe640 1 -- 172.21.15.107:0/961580129 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cb0018070 con 0x5648193b0d70 2024-05-09T08:39:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.875+0000 7f7cb4bfe640 1 -- 172.21.15.107:0/961580129 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb00123e0 con 0x5648193b0d70 2024-05-09T08:39:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.876+0000 7f7ccb20e740 1 -- 172.21.15.107:0/961580129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 msgr2=0x5648193bb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.876+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/961580129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 0x5648193bb0f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7cb000b7f0 tx=0x7f7cb000bcb0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.876+0000 7f7ccb20e740 1 -- 172.21.15.107:0/961580129 shutdown_connections 2024-05-09T08:39:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.876+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/961580129 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193bfa40 0x5648193c1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.876+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/961580129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 0x5648193bb0f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.876+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/961580129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.876+0000 7f7ccb20e740 1 -- 172.21.15.107:0/961580129 >> 172.21.15.107:0/961580129 conn(0x5648193ac960 msgr2=0x5648193acd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.876+0000 7f7ccb20e740 1 -- 172.21.15.107:0/961580129 shutdown_connections 2024-05-09T08:39:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.876+0000 7f7ccb20e740 1 -- 172.21.15.107:0/961580129 wait complete. 2024-05-09T08:39:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.876+0000 7f7ccb20e740 1 -- start start 2024-05-09T08:39:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7ccb20e740 1 Processor -- start 2024-05-09T08:39:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7ccb20e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193b84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7ccb20e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 0x5648193b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7ccb20e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193bfa40 0x5648193b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7ccb20e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5648193c2670 con 0x5648193b0450 2024-05-09T08:39:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7ccb20e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5648192d0790 con 0x5648193b0d70 2024-05-09T08:39:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7ccb20e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5648193c27f0 con 0x5648193bfa40 2024-05-09T08:39:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7cc61c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193b84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7ccb20e740 10 client.0 _ll_register_callbacks cb 0x56481930ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7cc61c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193b84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33270/0 (socket says 172.21.15.107:33270) 2024-05-09T08:39:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7cc61c8640 1 -- 172.21.15.107:0/1836757332 learned_addr learned my addr 172.21.15.107:0/1836757332 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7cc59c7640 1 --2- 172.21.15.107:0/1836757332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 0x5648193b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7cc69c9640 1 --2- 172.21.15.107:0/1836757332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193bfa40 0x5648193b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7cc59c7640 1 -- 172.21.15.107:0/1836757332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193bfa40 msgr2=0x5648193b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7cc59c7640 1 --2- 172.21.15.107:0/1836757332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193bfa40 0x5648193b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7cc59c7640 1 -- 172.21.15.107:0/1836757332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 msgr2=0x5648193b84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7cc59c7640 1 --2- 172.21.15.107:0/1836757332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193b84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.877+0000 7f7cc59c7640 1 -- 172.21.15.107:0/1836757332 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564819029210 con 0x5648193b0d70 2024-05-09T08:39:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.878+0000 7f7cc59c7640 1 --2- 172.21.15.107:0/1836757332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 0x5648193b8a10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0000c00 tx=0x7f7cb0013730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d69993e09f140d02 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.878+0000 7f7ca9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:39:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.878+0000 7f7ca9ffb640 1 -- 172.21.15.107:0/1836757332 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb000dd60 con 0x5648193b0d70 2024-05-09T08:39:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.878+0000 7f7ccb20e740 1 -- 172.21.15.107:0/1836757332 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cb0018070 con 0x5648193b0d70 2024-05-09T08:39:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.878+0000 7f7ca9ffb640 1 -- 172.21.15.107:0/1836757332 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f7cb001c3f0 con 0x5648193b0d70 2024-05-09T08:39:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.878+0000 7f7ca9ffb640 1 -- 172.21.15.107:0/1836757332 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7cb001c5f0 con 0x5648193b0d70 2024-05-09T08:39:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.878+0000 7f7ccb20e740 10 client.26603 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.878+0000 7f7ccb20e740 1 -- 172.21.15.107:0/1836757332 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5648190f5620 con 0x5648193b0d70 2024-05-09T08:39:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.878+0000 7f7ca9ffb640 1 -- 172.21.15.107:0/1836757332 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cb0018070 con 0x5648193b0d70 2024-05-09T08:39:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.878+0000 7f7ca9ffb640 1 -- 172.21.15.107:0/1836757332 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7cb0024850 con 0x5648193b0d70 2024-05-09T08:39:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.878+0000 7f7ccb20e740 10 client.26603 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.878+0000 7f7ccb20e740 10 client.26603 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.878+0000 7f7ccb20e740 20 client.26603 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.878+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/1836757332 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5648193cf650 0x5648193d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.878+0000 7f7ccb20e740 4 client.26603 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.878+0000 7f7ccb20e740 1 -- 172.21.15.107:0/1836757332 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564819322a70 con 0x5648193cf650 2024-05-09T08:39:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.879+0000 7f7cc61c8640 1 --2- 172.21.15.107:0/1836757332 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5648193cf650 0x5648193d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.881+0000 7f7ca9ffb640 10 client.26603 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.881+0000 7f7cc61c8640 1 --2- 172.21.15.107:0/1836757332 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5648193cf650 0x5648193d1a40 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6deadf83f646e126 server_cookie=96b9900b065baa54 in_seq=0 out_seq=0 2024-05-09T08:39:51.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.882+0000 7f7ca9ffb640 1 -- 172.21.15.107:0/1836757332 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564819322a70 con 0x5648193cf650 2024-05-09T08:39:51.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.882+0000 7f7ca9ffb640 10 client.26603 handle_command_reply: tid=0 2024-05-09T08:39:51.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.882+0000 7f7ccb20e740 1 client.26603 shutdown 2024-05-09T08:39:51.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.882+0000 7f7ccb20e740 1 -- 172.21.15.107:0/1836757332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 msgr2=0x5648193b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:51.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.882+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/1836757332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 0x5648193b8a10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0000c00 tx=0x7f7cb0013730 comp rx=0 tx=0).stop 2024-05-09T08:39:51.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.882+0000 7f7ccb20e740 1 -- 172.21.15.107:0/1836757332 shutdown_connections 2024-05-09T08:39:51.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.882+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/1836757332 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5648193cf650 0x5648193d1a40 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.882+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/1836757332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193bfa40 0x5648193b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.882+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/1836757332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 0x5648193b8a10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.882+0000 7f7ca9ffb640 0 client.26603 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:51.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.882+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/1836757332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193b84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.882+0000 7f7ccb20e740 1 -- 172.21.15.107:0/1836757332 >> 172.21.15.107:0/1836757332 conn(0x5648193ac960 msgr2=0x5648193b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:51.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.882+0000 7f7ccb20e740 1 -- 172.21.15.107:0/1836757332 shutdown_connections 2024-05-09T08:39:51.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.882+0000 7f7ccb20e740 1 -- 172.21.15.107:0/1836757332 wait complete. 2024-05-09T08:39:51.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.882+0000 7f7ccb20e740 20 client.26603 trim_cache size 0 max 16384 2024-05-09T08:39:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.902+0000 7f7ccb20e740 1 Processor -- start 2024-05-09T08:39:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.902+0000 7f7ccb20e740 1 -- start start 2024-05-09T08:39:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.902+0000 7f7ccb20e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193cb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.902+0000 7f7ccb20e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193b0d70 0x5648193b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.902+0000 7f7ccb20e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193bfa40 0x5648193ab880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.902+0000 7f7ccb20e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc01af1c0 con 0x5648193b0450 2024-05-09T08:39:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.902+0000 7f7ccb20e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc01aca90 con 0x5648193bfa40 2024-05-09T08:39:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.902+0000 7f7ccb20e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc01aa8e0 con 0x5648193b0d70 2024-05-09T08:39:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.902+0000 7f7cc61c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.902+0000 7f7cc61c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193cb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33272/0 (socket says 172.21.15.107:33272) 2024-05-09T08:39:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.902+0000 7f7cc61c8640 1 -- 172.21.15.107:0/2900632709 learned_addr learned my addr 172.21.15.107:0/2900632709 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.902+0000 7f7cc69c9640 1 --2- 172.21.15.107:0/2900632709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193bfa40 0x5648193ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.902+0000 7f7cc59c7640 1 --2- 172.21.15.107:0/2900632709 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193b0d70 0x5648193b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.903+0000 7f7cc69c9640 1 -- 172.21.15.107:0/2900632709 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193b0d70 msgr2=0x5648193b2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.903+0000 7f7cc69c9640 1 --2- 172.21.15.107:0/2900632709 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193b0d70 0x5648193b2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.903+0000 7f7cc69c9640 1 -- 172.21.15.107:0/2900632709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 msgr2=0x5648193cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.903+0000 7f7cc69c9640 1 --2- 172.21.15.107:0/2900632709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.903+0000 7f7cc69c9640 1 -- 172.21.15.107:0/2900632709 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648193b0ba0 con 0x5648193bfa40 2024-05-09T08:39:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.903+0000 7f7cc61c8640 1 --2- 172.21.15.107:0/2900632709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.903+0000 7f7cc69c9640 1 --2- 172.21.15.107:0/2900632709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193bfa40 0x5648193ab880 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7cb80cc560 tx=0x7f7cb80bb040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8be76a4c5c4c3aaa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.903+0000 7f7cabfff640 1 -- 172.21.15.107:0/2900632709 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8020040 con 0x5648193bfa40 2024-05-09T08:39:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.903+0000 7f7cabfff640 1 -- 172.21.15.107:0/2900632709 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cb809f6e0 con 0x5648193bfa40 2024-05-09T08:39:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.903+0000 7f7cabfff640 1 -- 172.21.15.107:0/2900632709 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb80979a0 con 0x5648193bfa40 2024-05-09T08:39:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 -- 172.21.15.107:0/2900632709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193bfa40 msgr2=0x5648193ab880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/2900632709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193bfa40 0x5648193ab880 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7cb80cc560 tx=0x7f7cb80bb040 comp rx=0 tx=0).stop 2024-05-09T08:39:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 -- 172.21.15.107:0/2900632709 shutdown_connections 2024-05-09T08:39:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/2900632709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193bfa40 0x5648193ab880 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/2900632709 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193b0d70 0x5648193b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/2900632709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 -- 172.21.15.107:0/2900632709 >> 172.21.15.107:0/2900632709 conn(0x5648193ac960 msgr2=0x5648193ab170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 -- 172.21.15.107:0/2900632709 shutdown_connections 2024-05-09T08:39:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 -- 172.21.15.107:0/2900632709 wait complete. 2024-05-09T08:39:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 -- start start 2024-05-09T08:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 Processor -- start 2024-05-09T08:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 0x5648193b5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193bfa40 0x5648193b5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc01aaf50 con 0x5648193b0450 2024-05-09T08:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc01ac3c0 con 0x5648193b0d70 2024-05-09T08:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.904+0000 7f7ccb20e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc01ab620 con 0x5648193bfa40 2024-05-09T08:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7ccb20e740 10 client.0 _ll_register_callbacks cb 0x7f7cb000b670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7cc61c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7cc61c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193b51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33288/0 (socket says 172.21.15.107:33288) 2024-05-09T08:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7cc61c8640 1 -- 172.21.15.107:0/2929510876 learned_addr learned my addr 172.21.15.107:0/2929510876 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7cc59c7640 1 --2- 172.21.15.107:0/2929510876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 0x5648193b5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7cc69c9640 1 --2- 172.21.15.107:0/2929510876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193bfa40 0x5648193b5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7cc61c8640 1 -- 172.21.15.107:0/2929510876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193bfa40 msgr2=0x5648193b5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7cc61c8640 1 --2- 172.21.15.107:0/2929510876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193bfa40 0x5648193b5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7cc61c8640 1 -- 172.21.15.107:0/2929510876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 msgr2=0x5648193b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7cc61c8640 1 --2- 172.21.15.107:0/2929510876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 0x5648193b5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7cc61c8640 1 -- 172.21.15.107:0/2929510876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564819434a50 con 0x5648193b0450 2024-05-09T08:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7cc59c7640 1 --2- 172.21.15.107:0/2929510876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 0x5648193b5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7cc61c8640 1 --2- 172.21.15.107:0/2929510876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193b51f0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc009cd0 tx=0x7f7cbc03e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5662e4d313af7d33 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7ca9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7ca9ffb640 1 -- 172.21.15.107:0/2929510876 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc0542f0 con 0x5648193b0450 2024-05-09T08:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7ca9ffb640 1 -- 172.21.15.107:0/2929510876 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f7cbc054490 con 0x5648193b0450 2024-05-09T08:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.905+0000 7f7ccb20e740 1 -- 172.21.15.107:0/2929510876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cb809f6e0 con 0x5648193b0450 2024-05-09T08:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.906+0000 7f7ca9ffb640 1 -- 172.21.15.107:0/2929510876 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7cbc064050 con 0x5648193b0450 2024-05-09T08:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.906+0000 7f7ccb20e740 10 client.17535 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.906+0000 7f7ccb20e740 1 -- 172.21.15.107:0/2929510876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564819435d60 con 0x5648193b0450 2024-05-09T08:39:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.906+0000 7f7ca9ffb640 1 -- 172.21.15.107:0/2929510876 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cb809f6e0 con 0x5648193b0450 2024-05-09T08:39:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.906+0000 7f7ca9ffb640 1 -- 172.21.15.107:0/2929510876 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7cbc043040 con 0x5648193b0450 2024-05-09T08:39:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.906+0000 7f7ccb20e740 10 client.17535 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.906+0000 7f7ccb20e740 10 client.17535 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.906+0000 7f7ccb20e740 20 client.17535 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.906+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/2929510876 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564819436510 0x564819438930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.906+0000 7f7ccb20e740 4 client.17535 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:39:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.906+0000 7f7ccb20e740 1 -- 172.21.15.107:0/2929510876 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56481931cdc0 con 0x564819436510 2024-05-09T08:39:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.906+0000 7f7cc59c7640 1 --2- 172.21.15.107:0/2929510876 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564819436510 0x564819438930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.907+0000 7f7ca9ffb640 10 client.17535 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.907+0000 7f7cc59c7640 1 --2- 172.21.15.107:0/2929510876 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564819436510 0x564819438930 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d085c116b87bba62 server_cookie=4c55cf2879d0f01b in_seq=0 out_seq=0 2024-05-09T08:39:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.921+0000 7f7ca9ffb640 1 -- 172.21.15.107:0/2929510876 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56481931cdc0 con 0x564819436510 2024-05-09T08:39:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.921+0000 7f7ca9ffb640 10 client.17535 handle_command_reply: tid=0 2024-05-09T08:39:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.922+0000 7f7ccb20e740 1 client.17535 shutdown 2024-05-09T08:39:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.922+0000 7f7ccb20e740 1 -- 172.21.15.107:0/2929510876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 msgr2=0x5648193b51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.922+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/2929510876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193b51f0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc009cd0 tx=0x7f7cbc03e040 comp rx=0 tx=0).stop 2024-05-09T08:39:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.922+0000 7f7ccb20e740 1 -- 172.21.15.107:0/2929510876 shutdown_connections 2024-05-09T08:39:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.922+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/2929510876 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564819436510 0x564819438930 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.922+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/2929510876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5648193bfa40 0x5648193b5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.922+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/2929510876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648193b0d70 0x5648193b5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.922+0000 7f7ca9ffb640 0 client.17535 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.922+0000 7f7ccb20e740 1 --2- 172.21.15.107:0/2929510876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5648193b0450 0x5648193b51f0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.922+0000 7f7ccb20e740 1 -- 172.21.15.107:0/2929510876 >> 172.21.15.107:0/2929510876 conn(0x5648193ac960 msgr2=0x5648193aba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.922+0000 7f7ccb20e740 1 -- 172.21.15.107:0/2929510876 shutdown_connections 2024-05-09T08:39:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.922+0000 7f7ccb20e740 1 -- 172.21.15.107:0/2929510876 wait complete. 2024-05-09T08:39:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.922+0000 7f7ccb20e740 20 client.17535 trim_cache size 0 max 16384 2024-05-09T08:39:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:39:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:39:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:39:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:39:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.923+0000 7f7cc842c640 1 -- 172.21.15.107:0/1521718000 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7cac0a2da0 msgr2=0x7f7cac0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.923+0000 7f7cc842c640 1 --2- 172.21.15.107:0/1521718000 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7cac0a2da0 0x7f7cac0a5260 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc005e30 tx=0x7f7cbc009360 comp rx=0 tx=0).stop 2024-05-09T08:39:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.923+0000 7f7cc842c640 1 -- 172.21.15.107:0/1521718000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cc0106720 msgr2=0x7f7cc019c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.923+0000 7f7cc842c640 1 --2- 172.21.15.107:0/1521718000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cc0106720 0x7f7cc019c900 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f7cb800d750 tx=0x7f7cb800dc10 comp rx=0 tx=0).stop 2024-05-09T08:39:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.923+0000 7f7cc842c640 1 -- 172.21.15.107:0/1521718000 shutdown_connections 2024-05-09T08:39:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.923+0000 7f7cc842c640 1 --2- 172.21.15.107:0/1521718000 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7cac0a2da0 0x7f7cac0a5260 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.923+0000 7f7cc842c640 1 --2- 172.21.15.107:0/1521718000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cc0106720 0x7f7cc019c900 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.923+0000 7f7cc842c640 1 --2- 172.21.15.107:0/1521718000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7cc0101120 0x7f7cc0198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.923+0000 7f7cc842c640 1 --2- 172.21.15.107:0/1521718000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cc0100760 0x7f7cc0198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.923+0000 7f7cc842c640 1 -- 172.21.15.107:0/1521718000 >> 172.21.15.107:0/1521718000 conn(0x7f7cc00778b0 msgr2=0x7f7cc01032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.923+0000 7f7cc842c640 1 -- 172.21.15.107:0/1521718000 shutdown_connections 2024-05-09T08:39:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:39:51.923+0000 7f7cc842c640 1 -- 172.21.15.107:0/1521718000 wait complete. 2024-05-09T08:39:52.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:52 smithi107 ceph-mon[30122]: pgmap v198: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-05-09T08:39:52.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:52 smithi157 ceph-mon[33614]: pgmap v198: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-05-09T08:39:52.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:52 smithi110 ceph-mon[33622]: pgmap v198: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-05-09T08:39:54.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:54 smithi107 ceph-mon[30122]: pgmap v199: 129 pgs: 129 active+clean; 8.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:54.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:54 smithi157 ceph-mon[33614]: pgmap v199: 129 pgs: 129 active+clean; 8.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:54.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:54 smithi110 ceph-mon[33622]: pgmap v199: 129 pgs: 129 active+clean; 8.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:55.448 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:39:55.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:39:55.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.562+0000 7f9dcee6b640 1 -- 172.21.15.107:0/4003804708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dc8100700 msgr2=0x7f9dc8100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:55.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.562+0000 7f9dcee6b640 1 --2- 172.21.15.107:0/4003804708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dc8100700 0x7f9dc8100b80 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f9db4009930 tx=0x7f9db402a420 comp rx=0 tx=0).stop 2024-05-09T08:39:55.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.562+0000 7f9dcee6b640 1 -- 172.21.15.107:0/4003804708 shutdown_connections 2024-05-09T08:39:55.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.562+0000 7f9dcee6b640 1 --2- 172.21.15.107:0/4003804708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9dc81010c0 0x7f9dc810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.562+0000 7f9dcee6b640 1 --2- 172.21.15.107:0/4003804708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dc8100700 0x7f9dc8100b80 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.562+0000 7f9dcee6b640 1 --2- 172.21.15.107:0/4003804708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9dc8106710 0x7f9dc8106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.562+0000 7f9dcee6b640 1 -- 172.21.15.107:0/4003804708 >> 172.21.15.107:0/4003804708 conn(0x7f9dc8077830 msgr2=0x7f9dc8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:55.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.562+0000 7f9dcee6b640 1 -- 172.21.15.107:0/4003804708 shutdown_connections 2024-05-09T08:39:55.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.562+0000 7f9dcee6b640 1 -- 172.21.15.107:0/4003804708 wait complete. 2024-05-09T08:39:55.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.562+0000 7f9dcee6b640 1 Processor -- start 2024-05-09T08:39:55.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.562+0000 7f9dcee6b640 1 -- start start 2024-05-09T08:39:55.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dcee6b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9dc8100700 0x7f9dc8198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dcee6b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9dc81010c0 0x7f9dc8199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dcee6b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dc8106710 0x7f9dc81a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dcee6b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc81109d0 con 0x7f9dc8106710 2024-05-09T08:39:55.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dcee6b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc8110850 con 0x7f9dc8100700 2024-05-09T08:39:55.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dcee6b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc8110b50 con 0x7f9dc81010c0 2024-05-09T08:39:55.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dcd408640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dc8106710 0x7f9dc81a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dcd408640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dc8106710 0x7f9dc81a3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33298/0 (socket says 172.21.15.107:33298) 2024-05-09T08:39:55.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dcd408640 1 -- 172.21.15.107:0/336852067 learned_addr learned my addr 172.21.15.107:0/336852067 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:55.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dc7fff640 1 --2- 172.21.15.107:0/336852067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9dc81010c0 0x7f9dc8199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dccc07640 1 --2- 172.21.15.107:0/336852067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9dc8100700 0x7f9dc8198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dcd408640 1 -- 172.21.15.107:0/336852067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9dc81010c0 msgr2=0x7f9dc8199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:55.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dcd408640 1 --2- 172.21.15.107:0/336852067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9dc81010c0 0x7f9dc8199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dcd408640 1 -- 172.21.15.107:0/336852067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9dc8100700 msgr2=0x7f9dc8198f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:55.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dcd408640 1 --2- 172.21.15.107:0/336852067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9dc8100700 0x7f9dc8198f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dcd408640 1 -- 172.21.15.107:0/336852067 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc81a44d0 con 0x7f9dc8106710 2024-05-09T08:39:55.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dcd408640 1 --2- 172.21.15.107:0/336852067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dc8106710 0x7f9dc81a3f90 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f9dc000cad0 tx=0x7f9dc000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:55.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dc5ffb640 1 -- 172.21.15.107:0/336852067 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc000ad60 con 0x7f9dc8106710 2024-05-09T08:39:55.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.563+0000 7f9dc5ffb640 1 -- 172.21.15.107:0/336852067 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9dc0005020 con 0x7f9dc8106710 2024-05-09T08:39:55.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.564+0000 7f9dcee6b640 1 -- 172.21.15.107:0/336852067 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dc81a46a0 con 0x7f9dc8106710 2024-05-09T08:39:55.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.564+0000 7f9dc5ffb640 1 -- 172.21.15.107:0/336852067 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc000f6b0 con 0x7f9dc8106710 2024-05-09T08:39:55.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.564+0000 7f9dcee6b640 1 -- 172.21.15.107:0/336852067 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dc81a4b70 con 0x7f9dc8106710 2024-05-09T08:39:55.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.566+0000 7f9dc5ffb640 1 -- 172.21.15.107:0/336852067 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f9dc00051d0 con 0x7f9dc8106710 2024-05-09T08:39:55.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dd1b78740 1 Processor -- start 2024-05-09T08:39:55.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dc5ffb640 1 --2- 172.21.15.107:0/336852067 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9db80a2d00 0x7f9db80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dd1b78740 1 -- start start 2024-05-09T08:39:55.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dc7fff640 1 --2- 172.21.15.107:0/336852067 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9db80a2d00 0x7f9db80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dd1b78740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f18280 0x562e13f18660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dc5ffb640 1 -- 172.21.15.107:0/336852067 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f9dc00cf7a0 con 0x7f9dc8106710 2024-05-09T08:39:55.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dd1b78740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f18ba0 0x562e13f22d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dd1b78740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f23270 0x562e13f25660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dd1b78740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562e13e7dd00 con 0x562e13f18280 2024-05-09T08:39:55.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dd1b78740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562e13e254c0 con 0x562e13f18ba0 2024-05-09T08:39:55.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dd1b78740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x562e13e79cc0 con 0x562e13f23270 2024-05-09T08:39:55.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dccc07640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f18280 0x562e13f18660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dccc07640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f18280 0x562e13f18660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33306/0 (socket says 172.21.15.107:33306) 2024-05-09T08:39:55.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dccc07640 1 -- 172.21.15.107:0/1419739345 learned_addr learned my addr 172.21.15.107:0/1419739345 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:55.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dc7fff640 1 --2- 172.21.15.107:0/1419739345 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f18ba0 0x562e13f22d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dcd408640 1 --2- 172.21.15.107:0/1419739345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f23270 0x562e13f25660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dccc07640 1 -- 172.21.15.107:0/1419739345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f23270 msgr2=0x562e13f25660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:55.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dccc07640 1 --2- 172.21.15.107:0/1419739345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f23270 0x562e13f25660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dccc07640 1 -- 172.21.15.107:0/1419739345 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f18ba0 msgr2=0x562e13f22d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:55.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dccc07640 1 --2- 172.21.15.107:0/1419739345 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f18ba0 0x562e13f22d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dccc07640 1 -- 172.21.15.107:0/1419739345 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e13c5b7d0 con 0x562e13f18280 2024-05-09T08:39:55.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dcd408640 1 --2- 172.21.15.107:0/1419739345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f23270 0x562e13f25660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:55.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.567+0000 7f9dc7fff640 1 --2- 172.21.15.107:0/336852067 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9db80a2d00 0x7f9db80a51c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9db4006010 tx=0x7f9db4005e80 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:55.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.568+0000 7f9dccc07640 1 --2- 172.21.15.107:0/1419739345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f18280 0x562e13f18660 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009a70 tx=0x7f9dbc009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb4745359b1bf4d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:55.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.568+0000 7f9db2ffd640 1 -- 172.21.15.107:0/1419739345 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc017070 con 0x562e13f18280 2024-05-09T08:39:55.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.568+0000 7f9db2ffd640 1 -- 172.21.15.107:0/1419739345 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9dbc013470 con 0x562e13f18280 2024-05-09T08:39:55.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.568+0000 7f9db2ffd640 1 -- 172.21.15.107:0/1419739345 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc012630 con 0x562e13f18280 2024-05-09T08:39:55.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.568+0000 7f9dd1b78740 1 -- 172.21.15.107:0/1419739345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f18280 msgr2=0x562e13f18660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:55.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.569+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/1419739345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f18280 0x562e13f18660 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009a70 tx=0x7f9dbc009f30 comp rx=0 tx=0).stop 2024-05-09T08:39:55.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.569+0000 7f9dd1b78740 1 -- 172.21.15.107:0/1419739345 shutdown_connections 2024-05-09T08:39:55.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.569+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/1419739345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f23270 0x562e13f25660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.569+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/1419739345 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f18ba0 0x562e13f22d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.569+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/1419739345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f18280 0x562e13f18660 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.569+0000 7f9dd1b78740 1 -- 172.21.15.107:0/1419739345 >> 172.21.15.107:0/1419739345 conn(0x562e13f145a0 msgr2=0x562e13f14980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:55.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.569+0000 7f9dd1b78740 1 -- 172.21.15.107:0/1419739345 shutdown_connections 2024-05-09T08:39:55.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.569+0000 7f9dd1b78740 1 -- 172.21.15.107:0/1419739345 wait complete. 2024-05-09T08:39:55.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.569+0000 7f9dd1b78740 1 -- start start 2024-05-09T08:39:55.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.569+0000 7f9dd1b78740 1 Processor -- start 2024-05-09T08:39:55.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.569+0000 7f9dd1b78740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f18280 0x562e13f201c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.569+0000 7f9dd1b78740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18ba0 0x562e13f20700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.569+0000 7f9dd1b78740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f23270 0x562e13f20c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.569+0000 7f9dd1b78740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562e13f26060 con 0x562e13f18280 2024-05-09T08:39:55.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dd1b78740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562e13e38b90 con 0x562e13f23270 2024-05-09T08:39:55.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dd1b78740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x562e13f261e0 con 0x562e13f18ba0 2024-05-09T08:39:55.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dd1b78740 10 client.0 _ll_register_callbacks cb 0x562e13e74da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:55.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dccc07640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f18280 0x562e13f201c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dccc07640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f18280 0x562e13f201c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33316/0 (socket says 172.21.15.107:33316) 2024-05-09T08:39:55.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dccc07640 1 -- 172.21.15.107:0/3507304500 learned_addr learned my addr 172.21.15.107:0/3507304500 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:55.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dc7fff640 1 --2- 172.21.15.107:0/3507304500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18ba0 0x562e13f20700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dcd408640 1 --2- 172.21.15.107:0/3507304500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f23270 0x562e13f20c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dccc07640 1 -- 172.21.15.107:0/3507304500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18ba0 msgr2=0x562e13f20700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:55.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dccc07640 1 --2- 172.21.15.107:0/3507304500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18ba0 0x562e13f20700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dccc07640 1 -- 172.21.15.107:0/3507304500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f23270 msgr2=0x562e13f20c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:55.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dccc07640 1 --2- 172.21.15.107:0/3507304500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f23270 0x562e13f20c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dccc07640 1 -- 172.21.15.107:0/3507304500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e13b913f0 con 0x562e13f18280 2024-05-09T08:39:55.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dc7fff640 1 --2- 172.21.15.107:0/3507304500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18ba0 0x562e13f20700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:55.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dcd408640 1 --2- 172.21.15.107:0/3507304500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f23270 0x562e13f20c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:55.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dccc07640 1 --2- 172.21.15.107:0/3507304500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f18280 0x562e13f201c0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc004700 tx=0x7f9dbc00ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cdf447e989728fb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:55.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9da7fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:55.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9da7fff640 1 -- 172.21.15.107:0/3507304500 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc017050 con 0x562e13f18280 2024-05-09T08:39:55.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9da7fff640 1 -- 172.21.15.107:0/3507304500 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f9dbc01c650 con 0x562e13f18280 2024-05-09T08:39:55.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dd1b78740 1 -- 172.21.15.107:0/3507304500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dbc013470 con 0x562e13f18280 2024-05-09T08:39:55.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9da7fff640 1 -- 172.21.15.107:0/3507304500 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9dbc02f050 con 0x562e13f18280 2024-05-09T08:39:55.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dd1b78740 10 client.17547 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:55.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.570+0000 7f9dd1b78740 1 -- 172.21.15.107:0/3507304500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562e13b510f0 con 0x562e13f18280 2024-05-09T08:39:55.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.571+0000 7f9da7fff640 1 -- 172.21.15.107:0/3507304500 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9dbc013470 con 0x562e13f18280 2024-05-09T08:39:55.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.571+0000 7f9da7fff640 1 -- 172.21.15.107:0/3507304500 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9dbc0128d0 con 0x562e13f18280 2024-05-09T08:39:55.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.571+0000 7f9dd1b78740 10 client.17547 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:55.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.571+0000 7f9dd1b78740 10 client.17547 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:55.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.571+0000 7f9dd1b78740 20 client.17547 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:55.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.571+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/3507304500 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562e13f33400 0x562e13f357f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.571+0000 7f9dd1b78740 4 client.17547 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:55.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.571+0000 7f9dd1b78740 1 -- 172.21.15.107:0/3507304500 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e13e8b430 con 0x562e13f33400 2024-05-09T08:39:55.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.571+0000 7f9dc7fff640 1 --2- 172.21.15.107:0/3507304500 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562e13f33400 0x562e13f357f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.572+0000 7f9da7fff640 10 client.17547 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:55.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.572+0000 7f9dc7fff640 1 --2- 172.21.15.107:0/3507304500 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562e13f33400 0x562e13f357f0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab87b8a308b98541 server_cookie=dcfc8e229dc07249 in_seq=0 out_seq=0 2024-05-09T08:39:55.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.572+0000 7f9da7fff640 1 -- 172.21.15.107:0/3507304500 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e13e8b430 con 0x562e13f33400 2024-05-09T08:39:55.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.572+0000 7f9da7fff640 10 client.17547 handle_command_reply: tid=0 2024-05-09T08:39:55.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.572+0000 7f9dd1b78740 1 client.17547 shutdown 2024-05-09T08:39:55.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.572+0000 7f9dd1b78740 1 -- 172.21.15.107:0/3507304500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f18280 msgr2=0x562e13f201c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:55.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.572+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/3507304500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f18280 0x562e13f201c0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc004700 tx=0x7f9dbc00ef30 comp rx=0 tx=0).stop 2024-05-09T08:39:55.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.572+0000 7f9dd1b78740 1 -- 172.21.15.107:0/3507304500 shutdown_connections 2024-05-09T08:39:55.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.572+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/3507304500 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562e13f33400 0x562e13f357f0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.572+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/3507304500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f23270 0x562e13f20c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.572+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/3507304500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18ba0 0x562e13f20700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.572+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/3507304500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f18280 0x562e13f201c0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.572+0000 7f9da7fff640 0 client.17547 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:55.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.572+0000 7f9dd1b78740 1 -- 172.21.15.107:0/3507304500 >> 172.21.15.107:0/3507304500 conn(0x562e13f145a0 msgr2=0x562e13f1a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:55.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.573+0000 7f9dd1b78740 1 -- 172.21.15.107:0/3507304500 shutdown_connections 2024-05-09T08:39:55.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.573+0000 7f9dd1b78740 1 -- 172.21.15.107:0/3507304500 wait complete. 2024-05-09T08:39:55.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.573+0000 7f9dd1b78740 20 client.17547 trim_cache size 0 max 16384 2024-05-09T08:39:55.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dd1b78740 1 Processor -- start 2024-05-09T08:39:55.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dd1b78740 1 -- start start 2024-05-09T08:39:55.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dd1b78740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f18280 0x562e13f2ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dd1b78740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18ba0 0x562e13f13660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dd1b78740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f23270 0x562e13f3ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dd1b78740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc81b14f0 con 0x562e13f23270 2024-05-09T08:39:55.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dd1b78740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc81aedc0 con 0x562e13f18280 2024-05-09T08:39:55.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dd1b78740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc81acc10 con 0x562e13f18ba0 2024-05-09T08:39:55.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dcd408640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f23270 0x562e13f3ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dcd408640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f23270 0x562e13f3ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33322/0 (socket says 172.21.15.107:33322) 2024-05-09T08:39:55.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dcd408640 1 -- 172.21.15.107:0/1532760743 learned_addr learned my addr 172.21.15.107:0/1532760743 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:55.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dc7fff640 1 --2- 172.21.15.107:0/1532760743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18ba0 0x562e13f13660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dccc07640 1 --2- 172.21.15.107:0/1532760743 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f18280 0x562e13f2ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dcd408640 1 -- 172.21.15.107:0/1532760743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18ba0 msgr2=0x562e13f13660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:55.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dcd408640 1 --2- 172.21.15.107:0/1532760743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18ba0 0x562e13f13660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dcd408640 1 -- 172.21.15.107:0/1532760743 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f18280 msgr2=0x562e13f2ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dcd408640 1 --2- 172.21.15.107:0/1532760743 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f18280 0x562e13f2ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.595+0000 7f9dcd408640 1 -- 172.21.15.107:0/1532760743 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e13f189d0 con 0x562e13f23270 2024-05-09T08:39:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.596+0000 7f9dcd408640 1 --2- 172.21.15.107:0/1532760743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f23270 0x562e13f3ae70 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f9dc00ccbf0 tx=0x7f9dc009ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c0a9ec6ea88a7b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.596+0000 7f9db27fc640 1 -- 172.21.15.107:0/1532760743 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc001d030 con 0x562e13f23270 2024-05-09T08:39:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.596+0000 7f9db27fc640 1 -- 172.21.15.107:0/1532760743 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9dc0005560 con 0x562e13f23270 2024-05-09T08:39:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.596+0000 7f9db27fc640 1 -- 172.21.15.107:0/1532760743 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc00860e0 con 0x562e13f23270 2024-05-09T08:39:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 -- 172.21.15.107:0/1532760743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f23270 msgr2=0x562e13f3ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/1532760743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f23270 0x562e13f3ae70 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f9dc00ccbf0 tx=0x7f9dc009ba20 comp rx=0 tx=0).stop 2024-05-09T08:39:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 -- 172.21.15.107:0/1532760743 shutdown_connections 2024-05-09T08:39:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/1532760743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f23270 0x562e13f3ae70 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/1532760743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18ba0 0x562e13f13660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/1532760743 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f18280 0x562e13f2ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 -- 172.21.15.107:0/1532760743 >> 172.21.15.107:0/1532760743 conn(0x562e13f145a0 msgr2=0x562e13f12f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 -- 172.21.15.107:0/1532760743 shutdown_connections 2024-05-09T08:39:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 -- 172.21.15.107:0/1532760743 wait complete. 2024-05-09T08:39:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 -- start start 2024-05-09T08:39:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 Processor -- start 2024-05-09T08:39:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18280 0x562e13f2b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f18ba0 0x562e13f1cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f23270 0x562e13f1d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc81ad280 con 0x562e13f23270 2024-05-09T08:39:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc81ae6f0 con 0x562e13f18ba0 2024-05-09T08:39:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc81ad950 con 0x562e13f18280 2024-05-09T08:39:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.597+0000 7f9dd1b78740 10 client.0 _ll_register_callbacks cb 0x7f9dbc0098f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dccc07640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18280 0x562e13f2b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dcd408640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f23270 0x562e13f1d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dcd408640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f23270 0x562e13f1d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33330/0 (socket says 172.21.15.107:33330) 2024-05-09T08:39:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dcd408640 1 -- 172.21.15.107:0/3807867035 learned_addr learned my addr 172.21.15.107:0/3807867035 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dccc07640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18280 0x562e13f2b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:53236/0 (socket says 172.21.15.107:53236) 2024-05-09T08:39:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dc7fff640 1 --2- 172.21.15.107:0/3807867035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f18ba0 0x562e13f1cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dcd408640 1 -- 172.21.15.107:0/3807867035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18280 msgr2=0x562e13f2b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dcd408640 1 --2- 172.21.15.107:0/3807867035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18280 0x562e13f2b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dcd408640 1 -- 172.21.15.107:0/3807867035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f18ba0 msgr2=0x562e13f1cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dcd408640 1 --2- 172.21.15.107:0/3807867035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f18ba0 0x562e13f1cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dcd408640 1 -- 172.21.15.107:0/3807867035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e13f1d900 con 0x562e13f23270 2024-05-09T08:39:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dccc07640 1 --2- 172.21.15.107:0/3807867035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18280 0x562e13f2b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dc7fff640 1 --2- 172.21.15.107:0/3807867035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f18ba0 0x562e13f1cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dcd408640 1 --2- 172.21.15.107:0/3807867035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f23270 0x562e13f1d0d0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f9dc00a40a0 tx=0x7f9dc009fe10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a9abeaba04ee7ba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9da7fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9da7fff640 1 -- 172.21.15.107:0/3807867035 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc00b3070 con 0x562e13f23270 2024-05-09T08:39:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9da7fff640 1 -- 172.21.15.107:0/3807867035 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f9dc0086070 con 0x562e13f23270 2024-05-09T08:39:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dd1b78740 1 -- 172.21.15.107:0/3807867035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dc0005560 con 0x562e13f23270 2024-05-09T08:39:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9da7fff640 1 -- 172.21.15.107:0/3807867035 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9dc001d070 con 0x562e13f23270 2024-05-09T08:39:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dd1b78740 10 client.17553 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.598+0000 7f9dd1b78740 1 -- 172.21.15.107:0/3807867035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562e13f996d0 con 0x562e13f23270 2024-05-09T08:39:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.599+0000 7f9da7fff640 1 -- 172.21.15.107:0/3807867035 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9dc0005560 con 0x562e13f23270 2024-05-09T08:39:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.599+0000 7f9da7fff640 1 -- 172.21.15.107:0/3807867035 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9dc0097740 con 0x562e13f23270 2024-05-09T08:39:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.599+0000 7f9dd1b78740 10 client.17553 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.599+0000 7f9dd1b78740 10 client.17553 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.599+0000 7f9dd1b78740 20 client.17553 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.599+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/3807867035 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562e13f99d40 0x562e13f9c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.599+0000 7f9dd1b78740 4 client.17553 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:39:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.599+0000 7f9dd1b78740 1 -- 172.21.15.107:0/3807867035 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562e13e85780 con 0x562e13f99d40 2024-05-09T08:39:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.599+0000 7f9dccc07640 1 --2- 172.21.15.107:0/3807867035 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562e13f99d40 0x562e13f9c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.599+0000 7f9da7fff640 10 client.17553 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.599+0000 7f9dccc07640 1 --2- 172.21.15.107:0/3807867035 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562e13f99d40 0x562e13f9c130 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9a5b261f431888a server_cookie=4d42c9538255427c in_seq=0 out_seq=0 2024-05-09T08:39:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.600+0000 7f9da7fff640 1 -- 172.21.15.107:0/3807867035 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562e13e85780 con 0x562e13f99d40 2024-05-09T08:39:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.600+0000 7f9da7fff640 10 client.17553 handle_command_reply: tid=0 2024-05-09T08:39:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.600+0000 7f9dd1b78740 1 client.17553 shutdown 2024-05-09T08:39:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.600+0000 7f9dd1b78740 1 -- 172.21.15.107:0/3807867035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f23270 msgr2=0x562e13f1d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.600+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/3807867035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f23270 0x562e13f1d0d0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f9dc00a40a0 tx=0x7f9dc009fe10 comp rx=0 tx=0).stop 2024-05-09T08:39:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.600+0000 7f9dd1b78740 1 -- 172.21.15.107:0/3807867035 shutdown_connections 2024-05-09T08:39:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.600+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/3807867035 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562e13f99d40 0x562e13f9c130 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.600+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/3807867035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e13f23270 0x562e13f1d0d0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.600+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/3807867035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e13f18ba0 0x562e13f1cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.600+0000 7f9dd1b78740 1 --2- 172.21.15.107:0/3807867035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e13f18280 0x562e13f2b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.600+0000 7f9da7fff640 0 client.17553 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:55.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.600+0000 7f9dd1b78740 1 -- 172.21.15.107:0/3807867035 >> 172.21.15.107:0/3807867035 conn(0x562e13f145a0 msgr2=0x562e13f137e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:55.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.600+0000 7f9dd1b78740 1 -- 172.21.15.107:0/3807867035 shutdown_connections 2024-05-09T08:39:55.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.600+0000 7f9dd1b78740 1 -- 172.21.15.107:0/3807867035 wait complete. 2024-05-09T08:39:55.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.600+0000 7f9dd1b78740 20 client.17553 trim_cache size 0 max 16384 2024-05-09T08:39:55.607 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:39:55.607 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:39:55.607 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:39:55.607 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:39:55.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.601+0000 7f9dcee6b640 1 -- 172.21.15.107:0/336852067 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9db80a2d00 msgr2=0x7f9db80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:55.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.601+0000 7f9dcee6b640 1 --2- 172.21.15.107:0/336852067 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9db80a2d00 0x7f9db80a51c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9db4006010 tx=0x7f9db4005e80 comp rx=0 tx=0).stop 2024-05-09T08:39:55.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.601+0000 7f9dcee6b640 1 -- 172.21.15.107:0/336852067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dc8106710 msgr2=0x7f9dc81a3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:55.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.601+0000 7f9dcee6b640 1 --2- 172.21.15.107:0/336852067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dc8106710 0x7f9dc81a3f90 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f9dc000cad0 tx=0x7f9dc000ced0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.601+0000 7f9dcee6b640 1 -- 172.21.15.107:0/336852067 shutdown_connections 2024-05-09T08:39:55.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.601+0000 7f9dcee6b640 1 --2- 172.21.15.107:0/336852067 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9db80a2d00 0x7f9db80a51c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.601+0000 7f9dcee6b640 1 --2- 172.21.15.107:0/336852067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9dc8106710 0x7f9dc81a3f90 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.601+0000 7f9dcee6b640 1 --2- 172.21.15.107:0/336852067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9dc81010c0 0x7f9dc8199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.601+0000 7f9dcee6b640 1 --2- 172.21.15.107:0/336852067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9dc8100700 0x7f9dc8198f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.601+0000 7f9dcee6b640 1 -- 172.21.15.107:0/336852067 >> 172.21.15.107:0/336852067 conn(0x7f9dc8077830 msgr2=0x7f9dc8103130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:55.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.601+0000 7f9dcee6b640 1 -- 172.21.15.107:0/336852067 shutdown_connections 2024-05-09T08:39:55.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.601+0000 7f9dcee6b640 1 -- 172.21.15.107:0/336852067 wait complete. 2024-05-09T08:39:55.612 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:39:55.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:39:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.735+0000 7f884a309640 1 -- 172.21.15.107:0/3982492809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8844106720 msgr2=0x7f8844106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.735+0000 7f884a309640 1 --2- 172.21.15.107:0/3982492809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8844106720 0x7f8844106b00 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f8834009930 tx=0x7f883402a420 comp rx=0 tx=0).stop 2024-05-09T08:39:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.735+0000 7f884a309640 1 -- 172.21.15.107:0/3982492809 shutdown_connections 2024-05-09T08:39:55.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.735+0000 7f884a309640 1 --2- 172.21.15.107:0/3982492809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88441010d0 0x7f884410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.735+0000 7f884a309640 1 --2- 172.21.15.107:0/3982492809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8844100710 0x7f8844100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.735+0000 7f884a309640 1 --2- 172.21.15.107:0/3982492809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8844106720 0x7f8844106b00 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.735+0000 7f884a309640 1 -- 172.21.15.107:0/3982492809 >> 172.21.15.107:0/3982492809 conn(0x7f8844077830 msgr2=0x7f8844077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:55.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.735+0000 7f884a309640 1 -- 172.21.15.107:0/3982492809 shutdown_connections 2024-05-09T08:39:55.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.735+0000 7f884a309640 1 -- 172.21.15.107:0/3982492809 wait complete. 2024-05-09T08:39:55.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.735+0000 7f884a309640 1 Processor -- start 2024-05-09T08:39:55.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.735+0000 7f884a309640 1 -- start start 2024-05-09T08:39:55.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f884a309640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8844100710 0x7f8844196f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f884a309640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88441010d0 0x7f8844197470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f884a309640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8844106720 0x7f884419b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f884a309640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f88441109f0 con 0x7f88441010d0 2024-05-09T08:39:55.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f884a309640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8844110870 con 0x7f8844100710 2024-05-09T08:39:55.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f884a309640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8844110b70 con 0x7f8844106720 2024-05-09T08:39:55.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f88437fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88441010d0 0x7f8844197470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f88437fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88441010d0 0x7f8844197470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33352/0 (socket says 172.21.15.107:33352) 2024-05-09T08:39:55.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f88437fe640 1 -- 172.21.15.107:0/290745735 learned_addr learned my addr 172.21.15.107:0/290745735 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:55.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f8843fff640 1 --2- 172.21.15.107:0/290745735 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8844100710 0x7f8844196f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f88488a6640 1 --2- 172.21.15.107:0/290745735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8844106720 0x7f884419b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f88437fe640 1 -- 172.21.15.107:0/290745735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8844106720 msgr2=0x7f884419b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:55.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f88437fe640 1 --2- 172.21.15.107:0/290745735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8844106720 0x7f884419b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f88437fe640 1 -- 172.21.15.107:0/290745735 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8844100710 msgr2=0x7f8844196f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:55.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f88437fe640 1 --2- 172.21.15.107:0/290745735 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8844100710 0x7f8844196f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:55.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f88437fe640 1 -- 172.21.15.107:0/290745735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884419bf20 con 0x7f88441010d0 2024-05-09T08:39:55.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f8843fff640 1 --2- 172.21.15.107:0/290745735 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8844100710 0x7f8844196f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:55.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.736+0000 7f88437fe640 1 --2- 172.21.15.107:0/290745735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88441010d0 0x7f8844197470 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f883800ea60 tx=0x7f883800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.737+0000 7f88417fa640 1 -- 172.21.15.107:0/290745735 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883800cde0 con 0x7f88441010d0 2024-05-09T08:39:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.737+0000 7f884a309640 1 -- 172.21.15.107:0/290745735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f884419c210 con 0x7f88441010d0 2024-05-09T08:39:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.737+0000 7f88417fa640 1 -- 172.21.15.107:0/290745735 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f88380035f0 con 0x7f88441010d0 2024-05-09T08:39:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.737+0000 7f88417fa640 1 -- 172.21.15.107:0/290745735 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8838010690 con 0x7f88441010d0 2024-05-09T08:39:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.737+0000 7f884a309640 1 -- 172.21.15.107:0/290745735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88441a3aa0 con 0x7f88441010d0 2024-05-09T08:39:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.738+0000 7f884a309640 1 -- 172.21.15.107:0/290745735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8844068650 con 0x7f88441010d0 2024-05-09T08:39:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.739+0000 7f88417fa640 1 -- 172.21.15.107:0/290745735 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f88380037a0 con 0x7f88441010d0 2024-05-09T08:39:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.740+0000 7f88417fa640 1 --2- 172.21.15.107:0/290745735 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f882c0a2d40 0x7f882c0a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.740+0000 7f8843fff640 1 --2- 172.21.15.107:0/290745735 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f882c0a2d40 0x7f882c0a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.740+0000 7f88417fa640 1 -- 172.21.15.107:0/290745735 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f88380cfae0 con 0x7f88441010d0 2024-05-09T08:39:55.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.740+0000 7f8843fff640 1 --2- 172.21.15.107:0/290745735 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f882c0a2d40 0x7f882c0a5200 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8834031440 tx=0x7f8834000bc0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:55.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:55.744+0000 7f88417fa640 1 -- 172.21.15.107:0/290745735 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f883809c3a0 con 0x7f88441010d0 2024-05-09T08:39:56.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.023+0000 7f884a309640 1 -- 172.21.15.107:0/290745735 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8844108ce0 con 0x7f88441010d0 2024-05-09T08:39:56.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.024+0000 7f88417fa640 1 -- 172.21.15.107:0/290745735 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11887 (secure 0 0 0) 0x7f883809bca0 con 0x7f88441010d0 2024-05-09T08:39:56.024 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:39:56.024 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:37:10.930317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:39:56.024 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2024-05-09T08:39:56.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.026+0000 7f884a309640 1 -- 172.21.15.107:0/290745735 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f882c0a2d40 msgr2=0x7f882c0a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.026+0000 7f884a309640 1 --2- 172.21.15.107:0/290745735 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f882c0a2d40 0x7f882c0a5200 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8834031440 tx=0x7f8834000bc0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.026+0000 7f884a309640 1 -- 172.21.15.107:0/290745735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88441010d0 msgr2=0x7f8844197470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.026+0000 7f884a309640 1 --2- 172.21.15.107:0/290745735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88441010d0 0x7f8844197470 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f883800ea60 tx=0x7f883800ef20 comp rx=0 tx=0).stop 2024-05-09T08:39:56.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.027+0000 7f884a309640 1 -- 172.21.15.107:0/290745735 shutdown_connections 2024-05-09T08:39:56.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.027+0000 7f884a309640 1 --2- 172.21.15.107:0/290745735 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f882c0a2d40 0x7f882c0a5200 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.027+0000 7f884a309640 1 --2- 172.21.15.107:0/290745735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8844106720 0x7f884419b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.027+0000 7f884a309640 1 --2- 172.21.15.107:0/290745735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88441010d0 0x7f8844197470 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.027+0000 7f884a309640 1 --2- 172.21.15.107:0/290745735 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8844100710 0x7f8844196f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.027+0000 7f884a309640 1 -- 172.21.15.107:0/290745735 >> 172.21.15.107:0/290745735 conn(0x7f8844077830 msgr2=0x7f8844103100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.027+0000 7f884a309640 1 -- 172.21.15.107:0/290745735 shutdown_connections 2024-05-09T08:39:56.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.027+0000 7f884a309640 1 -- 172.21.15.107:0/290745735 wait complete. 2024-05-09T08:39:56.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:39:56.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.150+0000 7fbc6569e640 1 -- 172.21.15.107:0/798953378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc6010bdf0 msgr2=0x7fbc6010e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.150+0000 7fbc6569e640 1 --2- 172.21.15.107:0/798953378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc6010bdf0 0x7fbc6010e290 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fbc58009ca0 tx=0x7fbc5802a9e0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.150+0000 7fbc6569e640 1 -- 172.21.15.107:0/798953378 shutdown_connections 2024-05-09T08:39:56.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.150+0000 7fbc6569e640 1 --2- 172.21.15.107:0/798953378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc6010bdf0 0x7fbc6010e290 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.150+0000 7fbc6569e640 1 --2- 172.21.15.107:0/798953378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc601033d0 0x7fbc6010b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.150+0000 7fbc6569e640 1 --2- 172.21.15.107:0/798953378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc60102ab0 0x7fbc60102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.150+0000 7fbc6569e640 1 -- 172.21.15.107:0/798953378 >> 172.21.15.107:0/798953378 conn(0x7fbc60077830 msgr2=0x7fbc60077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.150+0000 7fbc6569e640 1 -- 172.21.15.107:0/798953378 shutdown_connections 2024-05-09T08:39:56.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.150+0000 7fbc6569e640 1 -- 172.21.15.107:0/798953378 wait complete. 2024-05-09T08:39:56.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.151+0000 7fbc6569e640 1 Processor -- start 2024-05-09T08:39:56.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.151+0000 7fbc6569e640 1 -- start start 2024-05-09T08:39:56.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.151+0000 7fbc6569e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc60102ab0 0x7fbc60198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.151+0000 7fbc6569e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc601033d0 0x7fbc60198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.151+0000 7fbc6569e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc6010bdf0 0x7fbc6019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.151+0000 7fbc6569e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbc60111aa0 con 0x7fbc60102ab0 2024-05-09T08:39:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.151+0000 7fbc6569e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbc60111920 con 0x7fbc601033d0 2024-05-09T08:39:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.151+0000 7fbc6569e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbc60111c20 con 0x7fbc6010bdf0 2024-05-09T08:39:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.151+0000 7fbc5effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc60102ab0 0x7fbc60198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.151+0000 7fbc5effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc60102ab0 0x7fbc60198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33362/0 (socket says 172.21.15.107:33362) 2024-05-09T08:39:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.151+0000 7fbc5effd640 1 -- 172.21.15.107:0/196042773 learned_addr learned my addr 172.21.15.107:0/196042773 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.151+0000 7fbc5f7fe640 1 --2- 172.21.15.107:0/196042773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc6010bdf0 0x7fbc6019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.152+0000 7fbc5e7fc640 1 --2- 172.21.15.107:0/196042773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc601033d0 0x7fbc60198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.152+0000 7fbc5effd640 1 -- 172.21.15.107:0/196042773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc6010bdf0 msgr2=0x7fbc6019c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.152+0000 7fbc5effd640 1 --2- 172.21.15.107:0/196042773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc6010bdf0 0x7fbc6019c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.152+0000 7fbc5effd640 1 -- 172.21.15.107:0/196042773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc601033d0 msgr2=0x7fbc60198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.152+0000 7fbc5effd640 1 --2- 172.21.15.107:0/196042773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc601033d0 0x7fbc60198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.152+0000 7fbc5effd640 1 -- 172.21.15.107:0/196042773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc6019d030 con 0x7fbc60102ab0 2024-05-09T08:39:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.152+0000 7fbc5e7fc640 1 --2- 172.21.15.107:0/196042773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc601033d0 0x7fbc60198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.152+0000 7fbc5effd640 1 --2- 172.21.15.107:0/196042773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc60102ab0 0x7fbc60198040 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000d950 tx=0x7fbc5000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.152+0000 7fbc47fff640 1 -- 172.21.15.107:0/196042773 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc50017070 con 0x7fbc60102ab0 2024-05-09T08:39:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.152+0000 7fbc47fff640 1 -- 172.21.15.107:0/196042773 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc50009c30 con 0x7fbc60102ab0 2024-05-09T08:39:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.152+0000 7fbc6569e640 1 -- 172.21.15.107:0/196042773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc6019d320 con 0x7fbc60102ab0 2024-05-09T08:39:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.152+0000 7fbc47fff640 1 -- 172.21.15.107:0/196042773 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5000f620 con 0x7fbc60102ab0 2024-05-09T08:39:56.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.152+0000 7fbc6569e640 1 -- 172.21.15.107:0/196042773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc601a4bb0 con 0x7fbc60102ab0 2024-05-09T08:39:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc47fff640 1 -- 172.21.15.107:0/196042773 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fbc500109a0 con 0x7fbc60102ab0 2024-05-09T08:39:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc683ad740 1 Processor -- start 2024-05-09T08:39:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc683ad740 1 -- start start 2024-05-09T08:39:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc683ad740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e5e230 0x564eb1e5e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc683ad740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e5eb50 0x564eb1e6d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc683ad740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e6d730 0x564eb1e6fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc683ad740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564eb1dc3c40 con 0x564eb1e6d730 2024-05-09T08:39:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc683ad740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564eb1d6b580 con 0x564eb1e5e230 2024-05-09T08:39:56.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc683ad740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564eb1dbfc00 con 0x564eb1e5eb50 2024-05-09T08:39:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc5e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e6d730 0x564eb1e6fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc5e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e6d730 0x564eb1e6fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33364/0 (socket says 172.21.15.107:33364) 2024-05-09T08:39:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc5e7fc640 1 -- 172.21.15.107:0/1876622639 learned_addr learned my addr 172.21.15.107:0/1876622639 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc5f7fe640 1 --2- 172.21.15.107:0/1876622639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e5e230 0x564eb1e5e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc5effd640 1 --2- 172.21.15.107:0/1876622639 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e5eb50 0x564eb1e6d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc47fff640 1 --2- 172.21.15.107:0/196042773 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbc480a2d00 0x7fbc480a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc5e7fc640 1 -- 172.21.15.107:0/1876622639 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e5eb50 msgr2=0x564eb1e6d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc5e7fc640 1 --2- 172.21.15.107:0/1876622639 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e5eb50 0x564eb1e6d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc5e7fc640 1 -- 172.21.15.107:0/1876622639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e5e230 msgr2=0x564eb1e5e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc5e7fc640 1 --2- 172.21.15.107:0/1876622639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e5e230 0x564eb1e5e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.155+0000 7fbc5e7fc640 1 -- 172.21.15.107:0/1876622639 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564eb1ad73f0 con 0x564eb1e6d730 2024-05-09T08:39:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.156+0000 7fbc5f7fe640 1 --2- 172.21.15.107:0/196042773 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbc480a2d00 0x7fbc480a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.156+0000 7fbc47fff640 1 -- 172.21.15.107:0/196042773 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fbc500cf3e0 con 0x7fbc60102ab0 2024-05-09T08:39:56.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.156+0000 7fbc5e7fc640 1 --2- 172.21.15.107:0/1876622639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e6d730 0x564eb1e6fb20 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009ba0 tx=0x7fbc5400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ff0d8e5dad97d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.156+0000 7fbc457fa640 1 -- 172.21.15.107:0/1876622639 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54016070 con 0x564eb1e6d730 2024-05-09T08:39:56.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.156+0000 7fbc457fa640 1 -- 172.21.15.107:0/1876622639 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc54011920 con 0x564eb1e6d730 2024-05-09T08:39:56.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.156+0000 7fbc457fa640 1 -- 172.21.15.107:0/1876622639 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc540107b0 con 0x564eb1e6d730 2024-05-09T08:39:56.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 -- 172.21.15.107:0/1876622639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e6d730 msgr2=0x564eb1e6fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc5f7fe640 1 --2- 172.21.15.107:0/196042773 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbc480a2d00 0x7fbc480a51c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbc58006010 tx=0x7fbc580095c0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 --2- 172.21.15.107:0/1876622639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e6d730 0x564eb1e6fb20 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009ba0 tx=0x7fbc5400e6c0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 -- 172.21.15.107:0/1876622639 shutdown_connections 2024-05-09T08:39:56.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 --2- 172.21.15.107:0/1876622639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e6d730 0x564eb1e6fb20 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 --2- 172.21.15.107:0/1876622639 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e5eb50 0x564eb1e6d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 --2- 172.21.15.107:0/1876622639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e5e230 0x564eb1e5e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 -- 172.21.15.107:0/1876622639 >> 172.21.15.107:0/1876622639 conn(0x564eb1e5a650 msgr2=0x564eb1e5aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 -- 172.21.15.107:0/1876622639 shutdown_connections 2024-05-09T08:39:56.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 -- 172.21.15.107:0/1876622639 wait complete. 2024-05-09T08:39:56.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 -- start start 2024-05-09T08:39:56.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 Processor -- start 2024-05-09T08:39:56.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5e230 0x564eb1e6a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e5eb50 0x564eb1e6ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e6d730 0x564eb1e6b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564eb1e704f0 con 0x564eb1e5e230 2024-05-09T08:39:56.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564eb1d7ec40 con 0x564eb1e6d730 2024-05-09T08:39:56.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564eb1e70670 con 0x564eb1e5eb50 2024-05-09T08:39:56.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc683ad740 10 client.0 _ll_register_callbacks cb 0x564eb1ad6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:56.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.157+0000 7fbc5effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5e230 0x564eb1e6a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc5effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5e230 0x564eb1e6a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33370/0 (socket says 172.21.15.107:33370) 2024-05-09T08:39:56.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc5effd640 1 -- 172.21.15.107:0/529261520 learned_addr learned my addr 172.21.15.107:0/529261520 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:56.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc5f7fe640 1 --2- 172.21.15.107:0/529261520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e6d730 0x564eb1e6b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc5e7fc640 1 --2- 172.21.15.107:0/529261520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e5eb50 0x564eb1e6ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc5effd640 1 -- 172.21.15.107:0/529261520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e5eb50 msgr2=0x564eb1e6ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc5effd640 1 --2- 172.21.15.107:0/529261520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e5eb50 0x564eb1e6ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc5effd640 1 -- 172.21.15.107:0/529261520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e6d730 msgr2=0x564eb1e6b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc5effd640 1 --2- 172.21.15.107:0/529261520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e6d730 0x564eb1e6b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc5effd640 1 -- 172.21.15.107:0/529261520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564eb1ba17d0 con 0x564eb1e5e230 2024-05-09T08:39:56.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc5effd640 1 --2- 172.21.15.107:0/529261520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5e230 0x564eb1e6a630 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fbc500cc830 tx=0x7fbc5009aff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc91dabbc52fd7e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc42ff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:56.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc42ff5640 1 -- 172.21.15.107:0/529261520 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc50020070 con 0x564eb1e5e230 2024-05-09T08:39:56.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc42ff5640 1 -- 172.21.15.107:0/529261520 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fbc500cf120 con 0x564eb1e5e230 2024-05-09T08:39:56.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc683ad740 1 -- 172.21.15.107:0/529261520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc54011920 con 0x564eb1e5e230 2024-05-09T08:39:56.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc42ff5640 1 -- 172.21.15.107:0/529261520 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbc5009bca0 con 0x564eb1e5e230 2024-05-09T08:39:56.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc683ad740 10 client.17571 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:56.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.158+0000 7fbc683ad740 1 -- 172.21.15.107:0/529261520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564eb1e7c740 con 0x564eb1e5e230 2024-05-09T08:39:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.159+0000 7fbc42ff5640 1 -- 172.21.15.107:0/529261520 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc54011920 con 0x564eb1e5e230 2024-05-09T08:39:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.159+0000 7fbc42ff5640 1 -- 172.21.15.107:0/529261520 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fbc500cca50 con 0x564eb1e5e230 2024-05-09T08:39:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.159+0000 7fbc683ad740 10 client.17571 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.159+0000 7fbc683ad740 10 client.17571 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:39:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.159+0000 7fbc683ad740 20 client.17571 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.159+0000 7fbc683ad740 1 --2- 172.21.15.107:0/529261520 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x564eb1e7dc00 0x564eb1e7fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.159+0000 7fbc683ad740 4 client.17571 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.159+0000 7fbc683ad740 1 -- 172.21.15.107:0/529261520 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564eb1dd1370 con 0x564eb1e7dc00 2024-05-09T08:39:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.159+0000 7fbc5e7fc640 1 --2- 172.21.15.107:0/529261520 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x564eb1e7dc00 0x564eb1e7fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.160+0000 7fbc5e7fc640 1 --2- 172.21.15.107:0/529261520 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x564eb1e7dc00 0x564eb1e7fff0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=422b93e96d4a46d2 server_cookie=53c622df9b1fcfa2 in_seq=0 out_seq=0 2024-05-09T08:39:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.160+0000 7fbc42ff5640 10 client.17571 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:39:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.160+0000 7fbc42ff5640 1 -- 172.21.15.107:0/529261520 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564eb1dd1370 con 0x564eb1e7dc00 2024-05-09T08:39:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.160+0000 7fbc42ff5640 10 client.17571 handle_command_reply: tid=0 2024-05-09T08:39:56.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.160+0000 7fbc683ad740 1 client.17571 shutdown 2024-05-09T08:39:56.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.160+0000 7fbc683ad740 1 -- 172.21.15.107:0/529261520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5e230 msgr2=0x564eb1e6a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.160+0000 7fbc683ad740 1 --2- 172.21.15.107:0/529261520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5e230 0x564eb1e6a630 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fbc500cc830 tx=0x7fbc5009aff0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.160+0000 7fbc683ad740 1 -- 172.21.15.107:0/529261520 shutdown_connections 2024-05-09T08:39:56.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.160+0000 7fbc683ad740 1 --2- 172.21.15.107:0/529261520 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x564eb1e7dc00 0x564eb1e7fff0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.160+0000 7fbc683ad740 1 --2- 172.21.15.107:0/529261520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e6d730 0x564eb1e6b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.160+0000 7fbc683ad740 1 --2- 172.21.15.107:0/529261520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e5eb50 0x564eb1e6ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.160+0000 7fbc683ad740 1 --2- 172.21.15.107:0/529261520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5e230 0x564eb1e6a630 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.160+0000 7fbc683ad740 1 -- 172.21.15.107:0/529261520 >> 172.21.15.107:0/529261520 conn(0x564eb1e5a650 msgr2=0x564eb1e60480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.161+0000 7fbc42ff5640 0 client.17571 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:39:56.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.161+0000 7fbc683ad740 1 -- 172.21.15.107:0/529261520 shutdown_connections 2024-05-09T08:39:56.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.161+0000 7fbc683ad740 1 -- 172.21.15.107:0/529261520 wait complete. 2024-05-09T08:39:56.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.161+0000 7fbc683ad740 20 client.17571 trim_cache size 0 max 16384 2024-05-09T08:39:56.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc683ad740 1 Processor -- start 2024-05-09T08:39:56.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc683ad740 1 -- start start 2024-05-09T08:39:56.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc683ad740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5e230 0x564eb1e793f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc683ad740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e5eb50 0x564eb1e69f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc683ad740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e6d730 0x564eb1e85760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc683ad740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbc601af2a0 con 0x564eb1e5e230 2024-05-09T08:39:56.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc683ad740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbc601acb70 con 0x564eb1e6d730 2024-05-09T08:39:56.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc683ad740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbc601aa9c0 con 0x564eb1e5eb50 2024-05-09T08:39:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc5effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5e230 0x564eb1e793f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc5effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5e230 0x564eb1e793f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33382/0 (socket says 172.21.15.107:33382) 2024-05-09T08:39:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc5effd640 1 -- 172.21.15.107:0/3049908303 learned_addr learned my addr 172.21.15.107:0/3049908303 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc5e7fc640 1 --2- 172.21.15.107:0/3049908303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e5eb50 0x564eb1e69f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc5f7fe640 1 --2- 172.21.15.107:0/3049908303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e6d730 0x564eb1e85760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc5effd640 1 -- 172.21.15.107:0/3049908303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e5eb50 msgr2=0x564eb1e69f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc5effd640 1 --2- 172.21.15.107:0/3049908303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e5eb50 0x564eb1e69f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc5effd640 1 -- 172.21.15.107:0/3049908303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e6d730 msgr2=0x564eb1e85760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc5effd640 1 --2- 172.21.15.107:0/3049908303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e6d730 0x564eb1e85760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc5effd640 1 -- 172.21.15.107:0/3049908303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564eb1e7a140 con 0x564eb1e5e230 2024-05-09T08:39:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.177+0000 7fbc5f7fe640 1 --2- 172.21.15.107:0/3049908303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e6d730 0x564eb1e85760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.178+0000 7fbc5effd640 1 --2- 172.21.15.107:0/3049908303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5e230 0x564eb1e793f0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fbc500ccdd0 tx=0x7fbc500a4210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2139b7a1a3a75e90 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.178+0000 7fbc44ff9640 1 -- 172.21.15.107:0/3049908303 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc50014020 con 0x564eb1e5e230 2024-05-09T08:39:56.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.178+0000 7fbc44ff9640 1 -- 172.21.15.107:0/3049908303 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc5009f950 con 0x564eb1e5e230 2024-05-09T08:39:56.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.178+0000 7fbc44ff9640 1 -- 172.21.15.107:0/3049908303 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc500b4030 con 0x564eb1e5e230 2024-05-09T08:39:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 -- 172.21.15.107:0/3049908303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5e230 msgr2=0x564eb1e793f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 --2- 172.21.15.107:0/3049908303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5e230 0x564eb1e793f0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fbc500ccdd0 tx=0x7fbc500a4210 comp rx=0 tx=0).stop 2024-05-09T08:39:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 -- 172.21.15.107:0/3049908303 shutdown_connections 2024-05-09T08:39:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 --2- 172.21.15.107:0/3049908303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e6d730 0x564eb1e85760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 --2- 172.21.15.107:0/3049908303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e5eb50 0x564eb1e69f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 --2- 172.21.15.107:0/3049908303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5e230 0x564eb1e793f0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 -- 172.21.15.107:0/3049908303 >> 172.21.15.107:0/3049908303 conn(0x564eb1e5a650 msgr2=0x564eb1e597b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 -- 172.21.15.107:0/3049908303 shutdown_connections 2024-05-09T08:39:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 -- 172.21.15.107:0/3049908303 wait complete. 2024-05-09T08:39:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 -- start start 2024-05-09T08:39:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 Processor -- start 2024-05-09T08:39:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e5e230 0x564eb1e75900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5eb50 0x564eb1e75e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e6d730 0x564eb1e66cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbc601ab030 con 0x564eb1e5eb50 2024-05-09T08:39:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbc601ac4a0 con 0x564eb1e5e230 2024-05-09T08:39:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbc601ab700 con 0x564eb1e6d730 2024-05-09T08:39:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc683ad740 10 client.0 _ll_register_callbacks cb 0x7fbc5009b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc5e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5eb50 0x564eb1e75e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc5e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5eb50 0x564eb1e75e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33396/0 (socket says 172.21.15.107:33396) 2024-05-09T08:39:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc5e7fc640 1 -- 172.21.15.107:0/901921869 learned_addr learned my addr 172.21.15.107:0/901921869 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.179+0000 7fbc5f7fe640 1 --2- 172.21.15.107:0/901921869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e6d730 0x564eb1e66cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.180+0000 7fbc5effd640 1 --2- 172.21.15.107:0/901921869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e5e230 0x564eb1e75900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.180+0000 7fbc5e7fc640 1 -- 172.21.15.107:0/901921869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e6d730 msgr2=0x564eb1e66cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.180+0000 7fbc5e7fc640 1 --2- 172.21.15.107:0/901921869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e6d730 0x564eb1e66cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.180+0000 7fbc5e7fc640 1 -- 172.21.15.107:0/901921869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e5e230 msgr2=0x564eb1e75900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.180+0000 7fbc5e7fc640 1 --2- 172.21.15.107:0/901921869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e5e230 0x564eb1e75900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.180+0000 7fbc5e7fc640 1 -- 172.21.15.107:0/901921869 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564eb1e67500 con 0x564eb1e5eb50 2024-05-09T08:39:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.180+0000 7fbc5e7fc640 1 --2- 172.21.15.107:0/901921869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5eb50 0x564eb1e75e40 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009af0 tx=0x7fbc540098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9893da07c351865d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.180+0000 7fbc42ff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.180+0000 7fbc42ff5640 1 -- 172.21.15.107:0/901921869 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5402d070 con 0x564eb1e5eb50 2024-05-09T08:39:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.180+0000 7fbc42ff5640 1 -- 172.21.15.107:0/901921869 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fbc540094c0 con 0x564eb1e5eb50 2024-05-09T08:39:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.180+0000 7fbc683ad740 1 -- 172.21.15.107:0/901921869 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc5009f950 con 0x564eb1e5eb50 2024-05-09T08:39:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.180+0000 7fbc42ff5640 1 -- 172.21.15.107:0/901921869 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbc54036050 con 0x564eb1e5eb50 2024-05-09T08:39:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.180+0000 7fbc683ad740 10 client.17577 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.180+0000 7fbc683ad740 1 -- 172.21.15.107:0/901921869 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564eb1ee3ac0 con 0x564eb1e5eb50 2024-05-09T08:39:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.180+0000 7fbc42ff5640 1 -- 172.21.15.107:0/901921869 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc5009f950 con 0x564eb1e5eb50 2024-05-09T08:39:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.181+0000 7fbc42ff5640 1 -- 172.21.15.107:0/901921869 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fbc54030020 con 0x564eb1e5eb50 2024-05-09T08:39:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.181+0000 7fbc683ad740 10 client.17577 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.181+0000 7fbc683ad740 10 client.17577 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:39:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.181+0000 7fbc683ad740 20 client.17577 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.181+0000 7fbc683ad740 1 --2- 172.21.15.107:0/901921869 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x564eb1ee4010 0x564eb1ee6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.181+0000 7fbc683ad740 4 client.17577 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:39:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.181+0000 7fbc683ad740 1 -- 172.21.15.107:0/901921869 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564eb1dcb6c0 con 0x564eb1ee4010 2024-05-09T08:39:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.181+0000 7fbc5effd640 1 --2- 172.21.15.107:0/901921869 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x564eb1ee4010 0x564eb1ee6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.181+0000 7fbc5effd640 1 --2- 172.21.15.107:0/901921869 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x564eb1ee4010 0x564eb1ee6430 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=368d3f9d0c6275a3 server_cookie=60e263b25a769313 in_seq=0 out_seq=0 2024-05-09T08:39:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.181+0000 7fbc42ff5640 10 client.17577 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:39:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.181+0000 7fbc42ff5640 1 -- 172.21.15.107:0/901921869 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564eb1dcb6c0 con 0x564eb1ee4010 2024-05-09T08:39:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.181+0000 7fbc42ff5640 10 client.17577 handle_command_reply: tid=0 2024-05-09T08:39:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.181+0000 7fbc683ad740 1 client.17577 shutdown 2024-05-09T08:39:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc683ad740 1 -- 172.21.15.107:0/901921869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5eb50 msgr2=0x564eb1e75e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc683ad740 1 --2- 172.21.15.107:0/901921869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5eb50 0x564eb1e75e40 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009af0 tx=0x7fbc540098a0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc683ad740 1 -- 172.21.15.107:0/901921869 shutdown_connections 2024-05-09T08:39:56.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc683ad740 1 --2- 172.21.15.107:0/901921869 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x564eb1ee4010 0x564eb1ee6430 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc683ad740 1 --2- 172.21.15.107:0/901921869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564eb1e6d730 0x564eb1e66cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc683ad740 1 --2- 172.21.15.107:0/901921869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb1e5eb50 0x564eb1e75e40 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc683ad740 1 --2- 172.21.15.107:0/901921869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564eb1e5e230 0x564eb1e75900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc683ad740 1 -- 172.21.15.107:0/901921869 >> 172.21.15.107:0/901921869 conn(0x564eb1e5a650 msgr2=0x564eb1e6bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc42ff5640 0 client.17577 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:39:56.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc683ad740 1 -- 172.21.15.107:0/901921869 shutdown_connections 2024-05-09T08:39:56.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc683ad740 1 -- 172.21.15.107:0/901921869 wait complete. 2024-05-09T08:39:56.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc683ad740 20 client.17577 trim_cache size 0 max 16384 2024-05-09T08:39:56.189 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:39:56.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc6569e640 1 -- 172.21.15.107:0/196042773 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbc480a2d00 msgr2=0x7fbc480a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc6569e640 1 --2- 172.21.15.107:0/196042773 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbc480a2d00 0x7fbc480a51c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbc58006010 tx=0x7fbc580095c0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc6569e640 1 -- 172.21.15.107:0/196042773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc60102ab0 msgr2=0x7fbc60198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc6569e640 1 --2- 172.21.15.107:0/196042773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc60102ab0 0x7fbc60198040 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000d950 tx=0x7fbc5000de10 comp rx=0 tx=0).stop 2024-05-09T08:39:56.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc6569e640 1 -- 172.21.15.107:0/196042773 shutdown_connections 2024-05-09T08:39:56.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc6569e640 1 --2- 172.21.15.107:0/196042773 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbc480a2d00 0x7fbc480a51c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc6569e640 1 --2- 172.21.15.107:0/196042773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc6010bdf0 0x7fbc6019c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc6569e640 1 --2- 172.21.15.107:0/196042773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc601033d0 0x7fbc60198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc6569e640 1 --2- 172.21.15.107:0/196042773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc60102ab0 0x7fbc60198040 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.182+0000 7fbc6569e640 1 -- 172.21.15.107:0/196042773 >> 172.21.15.107:0/196042773 conn(0x7fbc60077830 msgr2=0x7fbc600feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.183+0000 7fbc6569e640 1 -- 172.21.15.107:0/196042773 shutdown_connections 2024-05-09T08:39:56.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.183+0000 7fbc6569e640 1 -- 172.21.15.107:0/196042773 wait complete. 2024-05-09T08:39:56.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:39:56.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.311+0000 7f37dce07640 1 -- 172.21.15.107:0/535401742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d8106710 msgr2=0x7f37d8106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.311+0000 7f37dce07640 1 --2- 172.21.15.107:0/535401742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d8106710 0x7f37d8106af0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009930 tx=0x7f37c802a420 comp rx=0 tx=0).stop 2024-05-09T08:39:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.311+0000 7f37dce07640 1 -- 172.21.15.107:0/535401742 shutdown_connections 2024-05-09T08:39:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.311+0000 7f37dce07640 1 --2- 172.21.15.107:0/535401742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37d81010c0 0x7f37d810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.311+0000 7f37dce07640 1 --2- 172.21.15.107:0/535401742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37d8100700 0x7f37d8100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.311+0000 7f37dce07640 1 --2- 172.21.15.107:0/535401742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d8106710 0x7f37d8106af0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.311+0000 7f37dce07640 1 -- 172.21.15.107:0/535401742 >> 172.21.15.107:0/535401742 conn(0x7f37d8077830 msgr2=0x7f37d8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.311+0000 7f37dce07640 1 -- 172.21.15.107:0/535401742 shutdown_connections 2024-05-09T08:39:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.311+0000 7f37dce07640 1 -- 172.21.15.107:0/535401742 wait complete. 2024-05-09T08:39:56.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.312+0000 7f37dce07640 1 Processor -- start 2024-05-09T08:39:56.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.312+0000 7f37dce07640 1 -- start start 2024-05-09T08:39:56.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.312+0000 7f37dce07640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37d8100700 0x7f37d819a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.312+0000 7f37dce07640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37d81010c0 0x7f37d819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.312+0000 7f37dce07640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d8106710 0x7f37d819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.312+0000 7f37dce07640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f37d8110990 con 0x7f37d8106710 2024-05-09T08:39:56.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.312+0000 7f37dce07640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f37d8110810 con 0x7f37d8100700 2024-05-09T08:39:56.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.312+0000 7f37dce07640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f37d8110b10 con 0x7f37d81010c0 2024-05-09T08:39:56.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37d6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d8106710 0x7f37d819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37d6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d8106710 0x7f37d819ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33418/0 (socket says 172.21.15.107:33418) 2024-05-09T08:39:56.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37d6d9d640 1 -- 172.21.15.107:0/3853443693 learned_addr learned my addr 172.21.15.107:0/3853443693 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:56.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37d659c640 1 --2- 172.21.15.107:0/3853443693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37d8100700 0x7f37d819a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/3853443693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37d81010c0 0x7f37d819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37d6d9d640 1 -- 172.21.15.107:0/3853443693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37d81010c0 msgr2=0x7f37d819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37d6d9d640 1 --2- 172.21.15.107:0/3853443693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37d81010c0 0x7f37d819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37d6d9d640 1 -- 172.21.15.107:0/3853443693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37d8100700 msgr2=0x7f37d819a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37d6d9d640 1 --2- 172.21.15.107:0/3853443693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37d8100700 0x7f37d819a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37d6d9d640 1 -- 172.21.15.107:0/3853443693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d819f200 con 0x7f37d8106710 2024-05-09T08:39:56.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/3853443693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37d81010c0 0x7f37d819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:56.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37d6d9d640 1 --2- 172.21.15.107:0/3853443693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d8106710 0x7f37d819ead0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f37d0009fd0 tx=0x7f37d000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37c77fe640 1 -- 172.21.15.107:0/3853443693 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d0017070 con 0x7f37d8106710 2024-05-09T08:39:56.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37c77fe640 1 -- 172.21.15.107:0/3853443693 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37d0003e20 con 0x7f37d8106710 2024-05-09T08:39:56.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37c77fe640 1 -- 172.21.15.107:0/3853443693 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d0010460 con 0x7f37d8106710 2024-05-09T08:39:56.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37dce07640 1 -- 172.21.15.107:0/3853443693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37d819f4f0 con 0x7f37d8106710 2024-05-09T08:39:56.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.313+0000 7f37dce07640 1 -- 172.21.15.107:0/3853443693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37d81a6d80 con 0x7f37d8106710 2024-05-09T08:39:56.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.316+0000 7f37c77fe640 1 -- 172.21.15.107:0/3853443693 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f37d0003970 con 0x7f37d8106710 2024-05-09T08:39:56.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.316+0000 7f37dfbf2740 1 Processor -- start 2024-05-09T08:39:56.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.316+0000 7f37dfbf2740 1 -- start start 2024-05-09T08:39:56.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.316+0000 7f37dfbf2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943d0330 0x555f943d0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.316+0000 7f37dfbf2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943d0c50 0x555f943dade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.316+0000 7f37dfbf2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943db320 0x555f943dd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.316+0000 7f37dfbf2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555f94335c40 con 0x555f943db320 2024-05-09T08:39:56.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.316+0000 7f37dfbf2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555f942dd580 con 0x555f943d0330 2024-05-09T08:39:56.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.316+0000 7f37dfbf2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x555f94331c00 con 0x555f943d0c50 2024-05-09T08:39:56.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.317+0000 7f37d5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943db320 0x555f943dd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.317+0000 7f37c77fe640 1 --2- 172.21.15.107:0/3853443693 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f37c00a2d00 0x7f37c00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.317+0000 7f37d5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943db320 0x555f943dd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33422/0 (socket says 172.21.15.107:33422) 2024-05-09T08:39:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.317+0000 7f37d5d9b640 1 -- 172.21.15.107:0/196788368 learned_addr learned my addr 172.21.15.107:0/196788368 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.317+0000 7f37d659c640 1 --2- 172.21.15.107:0/196788368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943d0c50 0x555f943dade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.317+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/196788368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943d0330 0x555f943d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.317+0000 7f37d6d9d640 1 --2- 172.21.15.107:0/3853443693 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f37c00a2d00 0x7f37c00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.317+0000 7f37c77fe640 1 -- 172.21.15.107:0/3853443693 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f37d00cf5b0 con 0x7f37d8106710 2024-05-09T08:39:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.317+0000 7f37d659c640 1 -- 172.21.15.107:0/196788368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943d0330 msgr2=0x555f943d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.317+0000 7f37d659c640 1 --2- 172.21.15.107:0/196788368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943d0330 0x555f943d0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.317+0000 7f37d659c640 1 -- 172.21.15.107:0/196788368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943db320 msgr2=0x555f943dd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.317+0000 7f37d659c640 1 --2- 172.21.15.107:0/196788368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943db320 0x555f943dd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.317+0000 7f37d659c640 1 -- 172.21.15.107:0/196788368 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f940493f0 con 0x555f943d0c50 2024-05-09T08:39:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.317+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/196788368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943d0330 0x555f943d0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:56.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.317+0000 7f37d6d9d640 1 --2- 172.21.15.107:0/3853443693 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f37c00a2d00 0x7f37c00a51c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f37d00ccfa0 tx=0x7f37d0086040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.317+0000 7f37d659c640 1 --2- 172.21.15.107:0/196788368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943d0c50 0x555f943dade0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f37c8031440 tx=0x7f37c8005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d92fc8ffbbae8945 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.318+0000 7f37c4ff9640 1 -- 172.21.15.107:0/196788368 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c8040070 con 0x555f943d0c50 2024-05-09T08:39:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.318+0000 7f37c4ff9640 1 -- 172.21.15.107:0/196788368 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37c80316a0 con 0x555f943d0c50 2024-05-09T08:39:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.318+0000 7f37c4ff9640 1 -- 172.21.15.107:0/196788368 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c8035040 con 0x555f943d0c50 2024-05-09T08:39:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.318+0000 7f37dfbf2740 1 -- 172.21.15.107:0/196788368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943d0c50 msgr2=0x555f943dade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.318+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/196788368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943d0c50 0x555f943dade0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f37c8031440 tx=0x7f37c8005c00 comp rx=0 tx=0).stop 2024-05-09T08:39:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.318+0000 7f37dfbf2740 1 -- 172.21.15.107:0/196788368 shutdown_connections 2024-05-09T08:39:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.318+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/196788368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943db320 0x555f943dd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.318+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/196788368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943d0c50 0x555f943dade0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.318+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/196788368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943d0330 0x555f943d0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.318+0000 7f37dfbf2740 1 -- 172.21.15.107:0/196788368 >> 172.21.15.107:0/196788368 conn(0x555f943cc650 msgr2=0x555f943cca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.319+0000 7f37dfbf2740 1 -- 172.21.15.107:0/196788368 shutdown_connections 2024-05-09T08:39:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.319+0000 7f37dfbf2740 1 -- 172.21.15.107:0/196788368 wait complete. 2024-05-09T08:39:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.319+0000 7f37dfbf2740 1 -- start start 2024-05-09T08:39:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.319+0000 7f37dfbf2740 1 Processor -- start 2024-05-09T08:39:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.319+0000 7f37dfbf2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943d0330 0x555f943d8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.319+0000 7f37dfbf2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943d0c50 0x555f943d87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.319+0000 7f37dfbf2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943db320 0x555f943d8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.319+0000 7f37dfbf2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555f943de0b0 con 0x555f943d0330 2024-05-09T08:39:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.319+0000 7f37dfbf2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555f942f0c40 con 0x555f943db320 2024-05-09T08:39:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.319+0000 7f37dfbf2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x555f943de230 con 0x555f943d0c50 2024-05-09T08:39:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.319+0000 7f37dfbf2740 10 client.0 _ll_register_callbacks cb 0x555f94048be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.319+0000 7f37d5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943d0330 0x555f943d8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37d5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943d0330 0x555f943d8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33430/0 (socket says 172.21.15.107:33430) 2024-05-09T08:39:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37d5d9b640 1 -- 172.21.15.107:0/1445777977 learned_addr learned my addr 172.21.15.107:0/1445777977 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/1445777977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943db320 0x555f943d8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37d659c640 1 --2- 172.21.15.107:0/1445777977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943d0c50 0x555f943d87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37d5d9b640 1 -- 172.21.15.107:0/1445777977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943d0c50 msgr2=0x555f943d87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/1445777977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943d0c50 0x555f943d87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37d5d9b640 1 -- 172.21.15.107:0/1445777977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943db320 msgr2=0x555f943d8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/1445777977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943db320 0x555f943d8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37d5d9b640 1 -- 172.21.15.107:0/1445777977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f941137d0 con 0x555f943d0330 2024-05-09T08:39:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/1445777977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943d0330 0x555f943d8290 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f37cc013ea0 tx=0x7f37cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5360f5feb824f00 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37bdffb640 1 -- 172.21.15.107:0/1445777977 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37cc005e50 con 0x555f943d0330 2024-05-09T08:39:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37bdffb640 1 -- 172.21.15.107:0/1445777977 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f37cc005020 con 0x555f943d0330 2024-05-09T08:39:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37dfbf2740 1 -- 172.21.15.107:0/1445777977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37c80316a0 con 0x555f943d0330 2024-05-09T08:39:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37bdffb640 1 -- 172.21.15.107:0/1445777977 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f37cc005220 con 0x555f943d0330 2024-05-09T08:39:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37dfbf2740 10 client.17589 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.320+0000 7f37dfbf2740 1 -- 172.21.15.107:0/1445777977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555f943ea2e0 con 0x555f943d0330 2024-05-09T08:39:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.321+0000 7f37bdffb640 1 -- 172.21.15.107:0/1445777977 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37c80316a0 con 0x555f943d0330 2024-05-09T08:39:56.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.321+0000 7f37bdffb640 1 -- 172.21.15.107:0/1445777977 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f37cc00f040 con 0x555f943d0330 2024-05-09T08:39:56.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.321+0000 7f37dfbf2740 10 client.17589 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:56.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.321+0000 7f37dfbf2740 10 client.17589 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:39:56.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.321+0000 7f37dfbf2740 20 client.17589 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:56.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.321+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/1445777977 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x555f943eb7a0 0x555f943edbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.321+0000 7f37dfbf2740 4 client.17589 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:56.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.321+0000 7f37dfbf2740 1 -- 172.21.15.107:0/1445777977 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f94343370 con 0x555f943eb7a0 2024-05-09T08:39:56.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.321+0000 7f37d659c640 1 --2- 172.21.15.107:0/1445777977 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x555f943eb7a0 0x555f943edbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.322+0000 7f37bdffb640 10 client.17589 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:39:56.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.322+0000 7f37d659c640 1 --2- 172.21.15.107:0/1445777977 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x555f943eb7a0 0x555f943edbb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2b3f3f9a5dd521d2 server_cookie=483dfaa7192f39f9 in_seq=0 out_seq=0 2024-05-09T08:39:56.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.322+0000 7f37bdffb640 1 -- 172.21.15.107:0/1445777977 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f94343370 con 0x555f943eb7a0 2024-05-09T08:39:56.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.322+0000 7f37bdffb640 10 client.17589 handle_command_reply: tid=0 2024-05-09T08:39:56.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.322+0000 7f37dfbf2740 1 client.17589 shutdown 2024-05-09T08:39:56.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.322+0000 7f37dfbf2740 1 -- 172.21.15.107:0/1445777977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943d0330 msgr2=0x555f943d8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.322+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/1445777977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943d0330 0x555f943d8290 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f37cc013ea0 tx=0x7f37cc004730 comp rx=0 tx=0).stop 2024-05-09T08:39:56.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.322+0000 7f37dfbf2740 1 -- 172.21.15.107:0/1445777977 shutdown_connections 2024-05-09T08:39:56.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.322+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/1445777977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943db320 0x555f943d8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.322+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/1445777977 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x555f943eb7a0 0x555f943edbb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.322+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/1445777977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943d0c50 0x555f943d87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.322+0000 7f37bdffb640 0 client.17589 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:39:56.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.322+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/1445777977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943d0330 0x555f943d8290 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.323+0000 7f37dfbf2740 1 -- 172.21.15.107:0/1445777977 >> 172.21.15.107:0/1445777977 conn(0x555f943cc650 msgr2=0x555f943d2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.323+0000 7f37dfbf2740 1 -- 172.21.15.107:0/1445777977 shutdown_connections 2024-05-09T08:39:56.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.323+0000 7f37dfbf2740 1 -- 172.21.15.107:0/1445777977 wait complete. 2024-05-09T08:39:56.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.323+0000 7f37dfbf2740 20 client.17589 trim_cache size 0 max 16384 2024-05-09T08:39:56.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.339+0000 7f37dfbf2740 1 Processor -- start 2024-05-09T08:39:56.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.339+0000 7f37dfbf2740 1 -- start start 2024-05-09T08:39:56.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.339+0000 7f37dfbf2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943d0330 0x555f943e6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.339+0000 7f37dfbf2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943d0c50 0x555f943d2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.339+0000 7f37dfbf2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943db320 0x555f943f3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.339+0000 7f37dfbf2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f37d81b3620 con 0x555f943db320 2024-05-09T08:39:56.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.339+0000 7f37dfbf2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f37d81b0ef0 con 0x555f943d0330 2024-05-09T08:39:56.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.339+0000 7f37dfbf2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f37d81aed40 con 0x555f943d0c50 2024-05-09T08:39:56.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.339+0000 7f37d5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943db320 0x555f943f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.339+0000 7f37d5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943db320 0x555f943f3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33438/0 (socket says 172.21.15.107:33438) 2024-05-09T08:39:56.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.339+0000 7f37d5d9b640 1 -- 172.21.15.107:0/1336502980 learned_addr learned my addr 172.21.15.107:0/1336502980 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:56.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.339+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/1336502980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943d0330 0x555f943e6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.339+0000 7f37d659c640 1 --2- 172.21.15.107:0/1336502980 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943d0c50 0x555f943d2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.340+0000 7f37d5d9b640 1 -- 172.21.15.107:0/1336502980 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943d0c50 msgr2=0x555f943d2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.340+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/1336502980 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943d0c50 0x555f943d2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.340+0000 7f37d5d9b640 1 -- 172.21.15.107:0/1336502980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943d0330 msgr2=0x555f943e6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.340+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/1336502980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943d0330 0x555f943e6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.340+0000 7f37d5d9b640 1 -- 172.21.15.107:0/1336502980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f943bdd20 con 0x555f943db320 2024-05-09T08:39:56.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.340+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/1336502980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943db320 0x555f943f3480 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f37cc013c70 tx=0x7f37cc0053c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c05f5ab667086f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.340+0000 7f37bffff640 1 -- 172.21.15.107:0/1336502980 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37cc013690 con 0x555f943db320 2024-05-09T08:39:56.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.340+0000 7f37bffff640 1 -- 172.21.15.107:0/1336502980 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37cc005470 con 0x555f943db320 2024-05-09T08:39:56.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.340+0000 7f37bffff640 1 -- 172.21.15.107:0/1336502980 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37cc004a00 con 0x555f943db320 2024-05-09T08:39:56.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 -- 172.21.15.107:0/1336502980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943db320 msgr2=0x555f943f3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/1336502980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943db320 0x555f943f3480 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f37cc013c70 tx=0x7f37cc0053c0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 -- 172.21.15.107:0/1336502980 shutdown_connections 2024-05-09T08:39:56.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/1336502980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943db320 0x555f943f3480 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/1336502980 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943d0c50 0x555f943d2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/1336502980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943d0330 0x555f943e6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 -- 172.21.15.107:0/1336502980 >> 172.21.15.107:0/1336502980 conn(0x555f943cc650 msgr2=0x555f943d8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 -- 172.21.15.107:0/1336502980 shutdown_connections 2024-05-09T08:39:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 -- 172.21.15.107:0/1336502980 wait complete. 2024-05-09T08:39:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 -- start start 2024-05-09T08:39:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 Processor -- start 2024-05-09T08:39:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943d0330 0x555f943e36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943d0c50 0x555f943d4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943db320 0x555f943d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f37d81af3b0 con 0x555f943d0c50 2024-05-09T08:39:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f37d81b0820 con 0x555f943d0330 2024-05-09T08:39:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f37d81afa80 con 0x555f943db320 2024-05-09T08:39:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.341+0000 7f37dfbf2740 10 client.0 _ll_register_callbacks cb 0x7f37cc013d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37d659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943d0c50 0x555f943d4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37d659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943d0c50 0x555f943d4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33450/0 (socket says 172.21.15.107:33450) 2024-05-09T08:39:56.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37d659c640 1 -- 172.21.15.107:0/2184150110 learned_addr learned my addr 172.21.15.107:0/2184150110 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:56.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/2184150110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943d0330 0x555f943e36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/2184150110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943db320 0x555f943d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37d659c640 1 -- 172.21.15.107:0/2184150110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943db320 msgr2=0x555f943d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37d659c640 1 --2- 172.21.15.107:0/2184150110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943db320 0x555f943d5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37d659c640 1 -- 172.21.15.107:0/2184150110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943d0330 msgr2=0x555f943e36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37d659c640 1 --2- 172.21.15.107:0/2184150110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943d0330 0x555f943e36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37d659c640 1 -- 172.21.15.107:0/2184150110 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f943d56b0 con 0x555f943d0c50 2024-05-09T08:39:56.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/2184150110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943db320 0x555f943d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:56.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37d659c640 1 --2- 172.21.15.107:0/2184150110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943d0c50 0x555f943d4b00 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009930 tx=0x7f37c8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7d0ae7aebdb32a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:56.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37bdffb640 1 -- 172.21.15.107:0/2184150110 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c8053070 con 0x555f943d0c50 2024-05-09T08:39:56.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37bdffb640 1 -- 172.21.15.107:0/2184150110 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f37c804f3c0 con 0x555f943d0c50 2024-05-09T08:39:56.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37dfbf2740 1 -- 172.21.15.107:0/2184150110 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37cc005470 con 0x555f943d0c50 2024-05-09T08:39:56.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37bdffb640 1 -- 172.21.15.107:0/2184150110 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f37c805c050 con 0x555f943d0c50 2024-05-09T08:39:56.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37dfbf2740 10 client.17595 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:56.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.342+0000 7f37dfbf2740 1 -- 172.21.15.107:0/2184150110 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555f94451b80 con 0x555f943d0c50 2024-05-09T08:39:56.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.343+0000 7f37bdffb640 1 -- 172.21.15.107:0/2184150110 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37cc005470 con 0x555f943d0c50 2024-05-09T08:39:56.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.343+0000 7f37bdffb640 1 -- 172.21.15.107:0/2184150110 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f37c8056020 con 0x555f943d0c50 2024-05-09T08:39:56.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.343+0000 7f37dfbf2740 10 client.17595 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:56.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.343+0000 7f37dfbf2740 10 client.17595 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:39:56.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.343+0000 7f37dfbf2740 20 client.17595 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:56.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.343+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/2184150110 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x555f94451f80 0x555f94454370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.343+0000 7f37dfbf2740 4 client.17595 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:39:56.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.343+0000 7f37dfbf2740 1 -- 172.21.15.107:0/2184150110 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555f9433d6c0 con 0x555f94451f80 2024-05-09T08:39:56.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.343+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/2184150110 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x555f94451f80 0x555f94454370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.343+0000 7f37d5d9b640 1 --2- 172.21.15.107:0/2184150110 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x555f94451f80 0x555f94454370 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1dc73f0c10dbff50 server_cookie=e6582b6333b3f207 in_seq=0 out_seq=0 2024-05-09T08:39:56.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.343+0000 7f37bdffb640 10 client.17595 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:39:56.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37bdffb640 1 -- 172.21.15.107:0/2184150110 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555f9433d6c0 con 0x555f94451f80 2024-05-09T08:39:56.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37bdffb640 10 client.17595 handle_command_reply: tid=0 2024-05-09T08:39:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dfbf2740 1 client.17595 shutdown 2024-05-09T08:39:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dfbf2740 1 -- 172.21.15.107:0/2184150110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943d0c50 msgr2=0x555f943d4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/2184150110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943d0c50 0x555f943d4b00 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009930 tx=0x7f37c8000bc0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dfbf2740 1 -- 172.21.15.107:0/2184150110 shutdown_connections 2024-05-09T08:39:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/2184150110 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x555f94451f80 0x555f94454370 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/2184150110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f943db320 0x555f943d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/2184150110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f943d0c50 0x555f943d4b00 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dfbf2740 1 --2- 172.21.15.107:0/2184150110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f943d0330 0x555f943e36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dfbf2740 1 -- 172.21.15.107:0/2184150110 >> 172.21.15.107:0/2184150110 conn(0x555f943cc650 msgr2=0x555f943d8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37bdffb640 0 client.17595 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:39:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dfbf2740 1 -- 172.21.15.107:0/2184150110 shutdown_connections 2024-05-09T08:39:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dfbf2740 1 -- 172.21.15.107:0/2184150110 wait complete. 2024-05-09T08:39:56.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dfbf2740 20 client.17595 trim_cache size 0 max 16384 2024-05-09T08:39:56.351 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:39:56.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dce07640 1 -- 172.21.15.107:0/3853443693 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f37c00a2d00 msgr2=0x7f37c00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dce07640 1 --2- 172.21.15.107:0/3853443693 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f37c00a2d00 0x7f37c00a51c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f37d00ccfa0 tx=0x7f37d0086040 comp rx=0 tx=0).stop 2024-05-09T08:39:56.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dce07640 1 -- 172.21.15.107:0/3853443693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d8106710 msgr2=0x7f37d819ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dce07640 1 --2- 172.21.15.107:0/3853443693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d8106710 0x7f37d819ead0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f37d0009fd0 tx=0x7f37d000b260 comp rx=0 tx=0).stop 2024-05-09T08:39:56.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.344+0000 7f37dce07640 1 -- 172.21.15.107:0/3853443693 shutdown_connections 2024-05-09T08:39:56.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.345+0000 7f37dce07640 1 --2- 172.21.15.107:0/3853443693 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f37c00a2d00 0x7f37c00a51c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.345+0000 7f37dce07640 1 --2- 172.21.15.107:0/3853443693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d8106710 0x7f37d819ead0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.345+0000 7f37dce07640 1 --2- 172.21.15.107:0/3853443693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37d81010c0 0x7f37d819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.345+0000 7f37dce07640 1 --2- 172.21.15.107:0/3853443693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37d8100700 0x7f37d819a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.345+0000 7f37dce07640 1 -- 172.21.15.107:0/3853443693 >> 172.21.15.107:0/3853443693 conn(0x7f37d8077830 msgr2=0x7f37d810c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.345+0000 7f37dce07640 1 -- 172.21.15.107:0/3853443693 shutdown_connections 2024-05-09T08:39:56.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.345+0000 7f37dce07640 1 -- 172.21.15.107:0/3853443693 wait complete. 2024-05-09T08:39:56.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:39:56.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.477+0000 7f5aad509640 1 -- 172.21.15.107:0/3976697197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa80fffa0 msgr2=0x7f5aa8100380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.477+0000 7f5aad509640 1 --2- 172.21.15.107:0/3976697197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa80fffa0 0x7f5aa8100380 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f5a98009900 tx=0x7f5a9802a410 comp rx=0 tx=0).stop 2024-05-09T08:39:56.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.477+0000 7f5aad509640 1 -- 172.21.15.107:0/3976697197 shutdown_connections 2024-05-09T08:39:56.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.477+0000 7f5aad509640 1 --2- 172.21.15.107:0/3976697197 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5aa8106bb0 0x7f5aa810e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.477+0000 7f5aad509640 1 --2- 172.21.15.107:0/3976697197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5aa8100950 0x7f5aa8106480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.477+0000 7f5aad509640 1 --2- 172.21.15.107:0/3976697197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa80fffa0 0x7f5aa8100380 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.477+0000 7f5aad509640 1 -- 172.21.15.107:0/3976697197 >> 172.21.15.107:0/3976697197 conn(0x7f5aa8076cf0 msgr2=0x7f5aa80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.478+0000 7f5aad509640 1 -- 172.21.15.107:0/3976697197 shutdown_connections 2024-05-09T08:39:56.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.478+0000 7f5aad509640 1 -- 172.21.15.107:0/3976697197 wait complete. 2024-05-09T08:39:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.478+0000 7f5aad509640 1 Processor -- start 2024-05-09T08:39:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.478+0000 7f5aad509640 1 -- start start 2024-05-09T08:39:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aad509640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5aa80fffa0 0x7f5aa8198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aad509640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5aa8106bb0 0x7f5aa8198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aad509640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa819c9c0 0x7f5aa819ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aad509640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa8111ac0 con 0x7f5aa819c9c0 2024-05-09T08:39:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aad509640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa8111940 con 0x7f5aa80fffa0 2024-05-09T08:39:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aad509640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa8111c40 con 0x7f5aa8106bb0 2024-05-09T08:39:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aa77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa819c9c0 0x7f5aa819ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aa77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa819c9c0 0x7f5aa819ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33458/0 (socket says 172.21.15.107:33458) 2024-05-09T08:39:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aa77fe640 1 -- 172.21.15.107:0/3745225787 learned_addr learned my addr 172.21.15.107:0/3745225787 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aa6ffd640 1 --2- 172.21.15.107:0/3745225787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5aa80fffa0 0x7f5aa8198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aa67fc640 1 --2- 172.21.15.107:0/3745225787 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5aa8106bb0 0x7f5aa8198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aa77fe640 1 -- 172.21.15.107:0/3745225787 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5aa8106bb0 msgr2=0x7f5aa8198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aa77fe640 1 --2- 172.21.15.107:0/3745225787 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5aa8106bb0 0x7f5aa8198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aa77fe640 1 -- 172.21.15.107:0/3745225787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5aa80fffa0 msgr2=0x7f5aa8198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aa77fe640 1 --2- 172.21.15.107:0/3745225787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5aa80fffa0 0x7f5aa8198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aa77fe640 1 -- 172.21.15.107:0/3745225787 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa819d4b0 con 0x7f5aa819c9c0 2024-05-09T08:39:56.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aa6ffd640 1 --2- 172.21.15.107:0/3745225787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5aa80fffa0 0x7f5aa8198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:39:56.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5aa77fe640 1 --2- 172.21.15.107:0/3745225787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa819c9c0 0x7f5aa819ce70 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f5aa000eaa0 tx=0x7f5aa000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.479+0000 7f5a8ffff640 1 -- 172.21.15.107:0/3745225787 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa000cdb0 con 0x7f5aa819c9c0 2024-05-09T08:39:56.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.480+0000 7f5aad509640 1 -- 172.21.15.107:0/3745225787 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aa819d7a0 con 0x7f5aa819c9c0 2024-05-09T08:39:56.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.480+0000 7f5a8ffff640 1 -- 172.21.15.107:0/3745225787 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5aa0005020 con 0x7f5aa819c9c0 2024-05-09T08:39:56.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.480+0000 7f5a8ffff640 1 -- 172.21.15.107:0/3745225787 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa00116d0 con 0x7f5aa819c9c0 2024-05-09T08:39:56.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.480+0000 7f5aad509640 1 -- 172.21.15.107:0/3745225787 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aa81a4fe0 con 0x7f5aa819c9c0 2024-05-09T08:39:56.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.482+0000 7f5ab0211740 1 Processor -- start 2024-05-09T08:39:56.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.482+0000 7f5a8ffff640 1 -- 172.21.15.107:0/3745225787 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f5aa00051d0 con 0x7f5aa819c9c0 2024-05-09T08:39:56.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.482+0000 7f5ab0211740 1 -- start start 2024-05-09T08:39:56.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5ab0211740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356330 0x563f9d356710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5ab0211740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 0x563f9d360de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5ab0211740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d361320 0x563f9d363710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5ab0211740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563f9d2bbc40 con 0x563f9d356330 2024-05-09T08:39:56.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5ab0211740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563f9d263580 con 0x563f9d361320 2024-05-09T08:39:56.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5ab0211740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563f9d2b7c00 con 0x563f9d356c50 2024-05-09T08:39:56.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5aa67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356330 0x563f9d356710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5aa67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356330 0x563f9d356710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33464/0 (socket says 172.21.15.107:33464) 2024-05-09T08:39:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5aa67fc640 1 -- 172.21.15.107:0/878844456 learned_addr learned my addr 172.21.15.107:0/878844456 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5aa6ffd640 1 --2- 172.21.15.107:0/878844456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 0x563f9d360de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5aa67fc640 1 -- 172.21.15.107:0/878844456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 msgr2=0x563f9d360de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5aa67fc640 1 --2- 172.21.15.107:0/878844456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 0x563f9d360de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5aa67fc640 1 -- 172.21.15.107:0/878844456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d361320 msgr2=0x563f9d363710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5aa67fc640 1 --2- 172.21.15.107:0/878844456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d361320 0x563f9d363710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5a8ffff640 1 --2- 172.21.15.107:0/3745225787 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5a900a2d00 0x7f5a900a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5aa67fc640 1 -- 172.21.15.107:0/878844456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f9cfcf3f0 con 0x563f9d356330 2024-05-09T08:39:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5aa6ffd640 1 --2- 172.21.15.107:0/878844456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 0x563f9d360de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:39:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5aa77fe640 1 --2- 172.21.15.107:0/3745225787 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5a900a2d00 0x7f5a900a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5aa67fc640 1 --2- 172.21.15.107:0/878844456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356330 0x563f9d356710 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00cbe0 tx=0x7f5a9c010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be1f54b6fd74b0c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5a8ffff640 1 -- 172.21.15.107:0/3745225787 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f5aa00cf960 con 0x7f5aa819c9c0 2024-05-09T08:39:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5a8d7fa640 1 -- 172.21.15.107:0/878844456 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c016030 con 0x563f9d356330 2024-05-09T08:39:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5a8d7fa640 1 -- 172.21.15.107:0/878844456 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a9c0054d0 con 0x563f9d356330 2024-05-09T08:39:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.483+0000 7f5a8d7fa640 1 -- 172.21.15.107:0/878844456 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c012480 con 0x563f9d356330 2024-05-09T08:39:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.484+0000 7f5aa77fe640 1 --2- 172.21.15.107:0/3745225787 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5a900a2d00 0x7f5a900a51c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00cd350 tx=0x7f5aa006aa50 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.484+0000 7f5ab0211740 1 -- 172.21.15.107:0/878844456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356330 msgr2=0x563f9d356710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.484+0000 7f5ab0211740 1 --2- 172.21.15.107:0/878844456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356330 0x563f9d356710 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00cbe0 tx=0x7f5a9c010740 comp rx=0 tx=0).stop 2024-05-09T08:39:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.484+0000 7f5ab0211740 1 -- 172.21.15.107:0/878844456 shutdown_connections 2024-05-09T08:39:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.484+0000 7f5ab0211740 1 --2- 172.21.15.107:0/878844456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d361320 0x563f9d363710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.484+0000 7f5ab0211740 1 --2- 172.21.15.107:0/878844456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 0x563f9d360de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.484+0000 7f5ab0211740 1 --2- 172.21.15.107:0/878844456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356330 0x563f9d356710 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.484+0000 7f5ab0211740 1 -- 172.21.15.107:0/878844456 >> 172.21.15.107:0/878844456 conn(0x563f9d352650 msgr2=0x563f9d352a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.484+0000 7f5ab0211740 1 -- 172.21.15.107:0/878844456 shutdown_connections 2024-05-09T08:39:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.484+0000 7f5ab0211740 1 -- 172.21.15.107:0/878844456 wait complete. 2024-05-09T08:39:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.485+0000 7f5ab0211740 1 -- start start 2024-05-09T08:39:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.485+0000 7f5ab0211740 1 Processor -- start 2024-05-09T08:39:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.485+0000 7f5ab0211740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356330 0x563f9d35e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.485+0000 7f5ab0211740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 0x563f9d35e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.485+0000 7f5ab0211740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d361320 0x563f9d35ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.485+0000 7f5ab0211740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563f9d364110 con 0x563f9d356330 2024-05-09T08:39:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.485+0000 7f5ab0211740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563f9d276c40 con 0x563f9d361320 2024-05-09T08:39:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.485+0000 7f5ab0211740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563f9d364290 con 0x563f9d356c50 2024-05-09T08:39:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.485+0000 7f5ab0211740 10 client.0 _ll_register_callbacks cb 0x563f9cfcebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.485+0000 7f5aa67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356330 0x563f9d35e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.485+0000 7f5aa67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356330 0x563f9d35e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33478/0 (socket says 172.21.15.107:33478) 2024-05-09T08:39:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.485+0000 7f5aa67fc640 1 -- 172.21.15.107:0/3772536476 learned_addr learned my addr 172.21.15.107:0/3772536476 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.485+0000 7f5aa6ffd640 1 --2- 172.21.15.107:0/3772536476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 0x563f9d35e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.485+0000 7f5aa67fc640 1 -- 172.21.15.107:0/3772536476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 msgr2=0x563f9d35e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5aa67fc640 1 --2- 172.21.15.107:0/3772536476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 0x563f9d35e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5aa67fc640 1 -- 172.21.15.107:0/3772536476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d361320 msgr2=0x563f9d35ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5aa67fc640 1 --2- 172.21.15.107:0/3772536476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d361320 0x563f9d35ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5aa67fc640 1 -- 172.21.15.107:0/3772536476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f9d0997d0 con 0x563f9d356330 2024-05-09T08:39:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5aa6ffd640 1 --2- 172.21.15.107:0/3772536476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 0x563f9d35e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:39:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5aa67fc640 1 --2- 172.21.15.107:0/3772536476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356330 0x563f9d35e2d0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c004b20 tx=0x7f5a9c0056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e526b5bf0c483f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5a8aff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5a8aff5640 1 -- 172.21.15.107:0/3772536476 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c016070 con 0x563f9d356330 2024-05-09T08:39:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5a8aff5640 1 -- 172.21.15.107:0/3772536476 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f5a9c012480 con 0x563f9d356330 2024-05-09T08:39:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5ab0211740 1 -- 172.21.15.107:0/3772536476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a9c0054d0 con 0x563f9d356330 2024-05-09T08:39:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5a8aff5640 1 -- 172.21.15.107:0/3772536476 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5a9c02f050 con 0x563f9d356330 2024-05-09T08:39:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5ab0211740 10 client.17607 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5ab0211740 1 -- 172.21.15.107:0/3772536476 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563f9d370330 con 0x563f9d356330 2024-05-09T08:39:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5a8aff5640 1 -- 172.21.15.107:0/3772536476 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a9c0054d0 con 0x563f9d356330 2024-05-09T08:39:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5a8aff5640 1 -- 172.21.15.107:0/3772536476 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5a9c01b6f0 con 0x563f9d356330 2024-05-09T08:39:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5ab0211740 10 client.17607 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:56.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.486+0000 7f5ab0211740 10 client.17607 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.487+0000 7f5ab0211740 20 client.17607 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.487+0000 7f5ab0211740 1 --2- 172.21.15.107:0/3772536476 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563f9d3716e0 0x563f9d373ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.487+0000 7f5ab0211740 4 client.17607 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.487+0000 7f5ab0211740 1 -- 172.21.15.107:0/3772536476 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f9d2c9370 con 0x563f9d3716e0 2024-05-09T08:39:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.487+0000 7f5aa6ffd640 1 --2- 172.21.15.107:0/3772536476 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563f9d3716e0 0x563f9d373ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.487+0000 7f5aa6ffd640 1 --2- 172.21.15.107:0/3772536476 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563f9d3716e0 0x563f9d373ad0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af7ad5f79502413b server_cookie=536190480709e8f0 in_seq=0 out_seq=0 2024-05-09T08:39:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.487+0000 7f5a8aff5640 10 client.17607 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.488+0000 7f5a8aff5640 1 -- 172.21.15.107:0/3772536476 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563f9d2c9370 con 0x563f9d3716e0 2024-05-09T08:39:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.488+0000 7f5a8aff5640 10 client.17607 handle_command_reply: tid=0 2024-05-09T08:39:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.488+0000 7f5ab0211740 1 client.17607 shutdown 2024-05-09T08:39:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.488+0000 7f5ab0211740 1 -- 172.21.15.107:0/3772536476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356330 msgr2=0x563f9d35e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.488+0000 7f5ab0211740 1 --2- 172.21.15.107:0/3772536476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356330 0x563f9d35e2d0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c004b20 tx=0x7f5a9c0056f0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.488+0000 7f5ab0211740 1 -- 172.21.15.107:0/3772536476 shutdown_connections 2024-05-09T08:39:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.488+0000 7f5ab0211740 1 --2- 172.21.15.107:0/3772536476 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563f9d3716e0 0x563f9d373ad0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.488+0000 7f5a8aff5640 0 client.17607 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:56.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.488+0000 7f5ab0211740 1 --2- 172.21.15.107:0/3772536476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d361320 0x563f9d35ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.488+0000 7f5ab0211740 1 --2- 172.21.15.107:0/3772536476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 0x563f9d35e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.488+0000 7f5ab0211740 1 --2- 172.21.15.107:0/3772536476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356330 0x563f9d35e2d0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.488+0000 7f5ab0211740 1 -- 172.21.15.107:0/3772536476 >> 172.21.15.107:0/3772536476 conn(0x563f9d352650 msgr2=0x563f9d358450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.488+0000 7f5ab0211740 1 -- 172.21.15.107:0/3772536476 shutdown_connections 2024-05-09T08:39:56.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.488+0000 7f5ab0211740 1 -- 172.21.15.107:0/3772536476 wait complete. 2024-05-09T08:39:56.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.488+0000 7f5ab0211740 20 client.17607 trim_cache size 0 max 16384 2024-05-09T08:39:56.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5ab0211740 1 Processor -- start 2024-05-09T08:39:56.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5ab0211740 1 -- start start 2024-05-09T08:39:56.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5ab0211740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356330 0x563f9d36cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5ab0211740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356c50 0x563f9d35da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5ab0211740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d361320 0x563f9d3516f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5ab0211740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa81b1520 con 0x563f9d356c50 2024-05-09T08:39:56.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5ab0211740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa81aedf0 con 0x563f9d361320 2024-05-09T08:39:56.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5ab0211740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa81acc40 con 0x563f9d356330 2024-05-09T08:39:56.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5aa6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356c50 0x563f9d35da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5aa6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356c50 0x563f9d35da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33494/0 (socket says 172.21.15.107:33494) 2024-05-09T08:39:56.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5aa6ffd640 1 -- 172.21.15.107:0/1995623470 learned_addr learned my addr 172.21.15.107:0/1995623470 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:56.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5aa67fc640 1 --2- 172.21.15.107:0/1995623470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356330 0x563f9d36cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5aa67fc640 1 --2- 172.21.15.107:0/1995623470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d361320 0x563f9d3516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5aa6ffd640 1 -- 172.21.15.107:0/1995623470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356330 msgr2=0x563f9d36cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5aa6ffd640 1 --2- 172.21.15.107:0/1995623470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356330 0x563f9d36cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5aa6ffd640 1 -- 172.21.15.107:0/1995623470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d361320 msgr2=0x563f9d3516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5aa6ffd640 1 --2- 172.21.15.107:0/1995623470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d361320 0x563f9d3516f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5aa6ffd640 1 -- 172.21.15.107:0/1995623470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f9d343d20 con 0x563f9d356c50 2024-05-09T08:39:56.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.504+0000 7f5aa67fc640 1 --2- 172.21.15.107:0/1995623470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356330 0x563f9d36cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:56.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.505+0000 7f5aa6ffd640 1 --2- 172.21.15.107:0/1995623470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356c50 0x563f9d35da80 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f5a980096c0 tx=0x7f5a98005a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b802e41eaba791f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.505+0000 7f5a8cff9640 1 -- 172.21.15.107:0/1995623470 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a98035070 con 0x563f9d356c50 2024-05-09T08:39:56.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.505+0000 7f5a8cff9640 1 -- 172.21.15.107:0/1995623470 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a98048070 con 0x563f9d356c50 2024-05-09T08:39:56.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.505+0000 7f5a8cff9640 1 -- 172.21.15.107:0/1995623470 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a98005440 con 0x563f9d356c50 2024-05-09T08:39:56.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 -- 172.21.15.107:0/1995623470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356c50 msgr2=0x563f9d35da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 --2- 172.21.15.107:0/1995623470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356c50 0x563f9d35da80 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f5a980096c0 tx=0x7f5a98005a60 comp rx=0 tx=0).stop 2024-05-09T08:39:56.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 -- 172.21.15.107:0/1995623470 shutdown_connections 2024-05-09T08:39:56.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 --2- 172.21.15.107:0/1995623470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d361320 0x563f9d3516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 --2- 172.21.15.107:0/1995623470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d356c50 0x563f9d35da80 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 --2- 172.21.15.107:0/1995623470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356330 0x563f9d36cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 -- 172.21.15.107:0/1995623470 >> 172.21.15.107:0/1995623470 conn(0x563f9d352650 msgr2=0x563f9d358420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 -- 172.21.15.107:0/1995623470 shutdown_connections 2024-05-09T08:39:56.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 -- 172.21.15.107:0/1995623470 wait complete. 2024-05-09T08:39:56.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 -- start start 2024-05-09T08:39:56.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 Processor -- start 2024-05-09T08:39:56.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d356330 0x563f9d369950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 0x563f9d35ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d361320 0x563f9d35b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa81ad2b0 con 0x563f9d361320 2024-05-09T08:39:56.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa81ae720 con 0x563f9d356330 2024-05-09T08:39:56.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa81ad980 con 0x563f9d356c50 2024-05-09T08:39:56.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5ab0211740 10 client.0 _ll_register_callbacks cb 0x7f5a9c00cae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:56.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5aa67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d361320 0x563f9d35b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5aa67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d361320 0x563f9d35b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33498/0 (socket says 172.21.15.107:33498) 2024-05-09T08:39:56.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.506+0000 7f5aa67fc640 1 -- 172.21.15.107:0/3736799726 learned_addr learned my addr 172.21.15.107:0/3736799726 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:56.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.507+0000 7f5aa6ffd640 1 --2- 172.21.15.107:0/3736799726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 0x563f9d35ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.507+0000 7f5aa67fc640 1 --2- 172.21.15.107:0/3736799726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d356330 0x563f9d369950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.507+0000 7f5aa67fc640 1 -- 172.21.15.107:0/3736799726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 msgr2=0x563f9d35ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.507+0000 7f5aa67fc640 1 --2- 172.21.15.107:0/3736799726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 0x563f9d35ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.507+0000 7f5aa67fc640 1 -- 172.21.15.107:0/3736799726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d356330 msgr2=0x563f9d369950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:56.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.507+0000 7f5aa67fc640 1 --2- 172.21.15.107:0/3736799726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d356330 0x563f9d369950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.507+0000 7f5aa67fc640 1 -- 172.21.15.107:0/3736799726 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f9d35bab0 con 0x563f9d361320 2024-05-09T08:39:56.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.507+0000 7f5aa67fc640 1 --2- 172.21.15.107:0/3736799726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d361320 0x563f9d35b280 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00c460 tx=0x7f5a9c0056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad02013a0959a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:56.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.507+0000 7f5a8aff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:56.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.507+0000 7f5a8aff5640 1 -- 172.21.15.107:0/3736799726 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c016070 con 0x563f9d361320 2024-05-09T08:39:56.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.507+0000 7f5a8aff5640 1 -- 172.21.15.107:0/3736799726 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f5a9c00c6c0 con 0x563f9d361320 2024-05-09T08:39:56.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.507+0000 7f5ab0211740 1 -- 172.21.15.107:0/3736799726 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a98048070 con 0x563f9d361320 2024-05-09T08:39:56.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.507+0000 7f5a8aff5640 1 -- 172.21.15.107:0/3736799726 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5a9c005230 con 0x563f9d361320 2024-05-09T08:39:56.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.507+0000 7f5ab0211740 10 client.17613 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:56.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.507+0000 7f5ab0211740 1 -- 172.21.15.107:0/3736799726 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563f9d3d7910 con 0x563f9d361320 2024-05-09T08:39:56.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.507+0000 7f5a8aff5640 1 -- 172.21.15.107:0/3736799726 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a98048070 con 0x563f9d361320 2024-05-09T08:39:56.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.508+0000 7f5a8aff5640 1 -- 172.21.15.107:0/3736799726 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5a9c012890 con 0x563f9d361320 2024-05-09T08:39:56.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.508+0000 7f5ab0211740 10 client.17613 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:56.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.508+0000 7f5ab0211740 10 client.17613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:56.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.508+0000 7f5ab0211740 20 client.17613 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:56.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.508+0000 7f5ab0211740 1 --2- 172.21.15.107:0/3736799726 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563f9d3716e0 0x563f9d3d9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:56.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.508+0000 7f5ab0211740 4 client.17613 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:39:56.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.508+0000 7f5ab0211740 1 -- 172.21.15.107:0/3736799726 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563f9d2c36c0 con 0x563f9d3716e0 2024-05-09T08:39:56.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.508+0000 7f5aa6ffd640 1 --2- 172.21.15.107:0/3736799726 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563f9d3716e0 0x563f9d3d9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:56.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.508+0000 7f5a8aff5640 10 client.17613 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:56.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.508+0000 7f5aa6ffd640 1 --2- 172.21.15.107:0/3736799726 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563f9d3716e0 0x563f9d3d9fb0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39aa224bbabb8c6b server_cookie=9e8bd901e47b2ec0 in_seq=0 out_seq=0 2024-05-09T08:39:56.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.509+0000 7f5a8aff5640 1 -- 172.21.15.107:0/3736799726 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563f9d2c36c0 con 0x563f9d3716e0 2024-05-09T08:39:56.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.509+0000 7f5a8aff5640 10 client.17613 handle_command_reply: tid=0 2024-05-09T08:39:56.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.509+0000 7f5ab0211740 1 client.17613 shutdown 2024-05-09T08:39:56.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.509+0000 7f5ab0211740 1 -- 172.21.15.107:0/3736799726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d361320 msgr2=0x563f9d35b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.509+0000 7f5ab0211740 1 --2- 172.21.15.107:0/3736799726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d361320 0x563f9d35b280 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00c460 tx=0x7f5a9c0056f0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.509+0000 7f5ab0211740 1 -- 172.21.15.107:0/3736799726 shutdown_connections 2024-05-09T08:39:56.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.509+0000 7f5ab0211740 1 --2- 172.21.15.107:0/3736799726 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563f9d3716e0 0x563f9d3d9fb0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.509+0000 7f5ab0211740 1 --2- 172.21.15.107:0/3736799726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f9d361320 0x563f9d35b280 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.509+0000 7f5ab0211740 1 --2- 172.21.15.107:0/3736799726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f9d356c50 0x563f9d35ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.509+0000 7f5ab0211740 1 --2- 172.21.15.107:0/3736799726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f9d356330 0x563f9d369950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.509+0000 7f5ab0211740 1 -- 172.21.15.107:0/3736799726 >> 172.21.15.107:0/3736799726 conn(0x563f9d352650 msgr2=0x563f9d35eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.509+0000 7f5a8aff5640 0 client.17613 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:56.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.509+0000 7f5ab0211740 1 -- 172.21.15.107:0/3736799726 shutdown_connections 2024-05-09T08:39:56.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.509+0000 7f5ab0211740 1 -- 172.21.15.107:0/3736799726 wait complete. 2024-05-09T08:39:56.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.509+0000 7f5ab0211740 20 client.17613 trim_cache size 0 max 16384 2024-05-09T08:39:56.517 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:39:56.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.510+0000 7f5aad509640 1 -- 172.21.15.107:0/3745225787 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5a900a2d00 msgr2=0x7f5a900a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.510+0000 7f5aad509640 1 --2- 172.21.15.107:0/3745225787 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5a900a2d00 0x7f5a900a51c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00cd350 tx=0x7f5aa006aa50 comp rx=0 tx=0).stop 2024-05-09T08:39:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.510+0000 7f5aad509640 1 -- 172.21.15.107:0/3745225787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa819c9c0 msgr2=0x7f5aa819ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.510+0000 7f5aad509640 1 --2- 172.21.15.107:0/3745225787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa819c9c0 0x7f5aa819ce70 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f5aa000eaa0 tx=0x7f5aa000ef60 comp rx=0 tx=0).stop 2024-05-09T08:39:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.510+0000 7f5aad509640 1 -- 172.21.15.107:0/3745225787 shutdown_connections 2024-05-09T08:39:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.510+0000 7f5aad509640 1 --2- 172.21.15.107:0/3745225787 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5a900a2d00 0x7f5a900a51c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.510+0000 7f5aad509640 1 --2- 172.21.15.107:0/3745225787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa819c9c0 0x7f5aa819ce70 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.510+0000 7f5aad509640 1 --2- 172.21.15.107:0/3745225787 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5aa8106bb0 0x7f5aa8198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.510+0000 7f5aad509640 1 --2- 172.21.15.107:0/3745225787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5aa80fffa0 0x7f5aa8198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.510+0000 7f5aad509640 1 -- 172.21.15.107:0/3745225787 >> 172.21.15.107:0/3745225787 conn(0x7f5aa8076cf0 msgr2=0x7f5aa8105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.510+0000 7f5aad509640 1 -- 172.21.15.107:0/3745225787 shutdown_connections 2024-05-09T08:39:56.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:56.510+0000 7f5aad509640 1 -- 172.21.15.107:0/3745225787 wait complete. 2024-05-09T08:39:56.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:56 smithi107 ceph-mon[30122]: pgmap v200: 129 pgs: 129 active+clean; 8.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:39:56.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:56 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/290745735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:39:56.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:56 smithi157 ceph-mon[33614]: pgmap v200: 129 pgs: 129 active+clean; 8.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:39:56.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:56 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/290745735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:39:56.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:56 smithi110 ceph-mon[33622]: pgmap v200: 129 pgs: 129 active+clean; 8.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 25 MiB/s wr, 15 op/s 2024-05-09T08:39:56.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:56 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/290745735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:39:57.522 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:39:57.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:39:57.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:39:57.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.645+0000 7fc8936c4640 1 -- 172.21.15.107:0/1365955377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc88c100f50 msgr2=0x7fc88c10dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.645+0000 7fc8936c4640 1 --2- 172.21.15.107:0/1365955377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc88c100f50 0x7fc88c10dca0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fc87c009900 tx=0x7fc87c02a410 comp rx=0 tx=0).stop 2024-05-09T08:39:57.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.645+0000 7fc8936c4640 1 -- 172.21.15.107:0/1365955377 shutdown_connections 2024-05-09T08:39:57.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.645+0000 7fc8936c4640 1 --2- 172.21.15.107:0/1365955377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc88c10e1e0 0x7fc88c1105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.645+0000 7fc8936c4640 1 --2- 172.21.15.107:0/1365955377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc88c100f50 0x7fc88c10dca0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.645+0000 7fc8936c4640 1 --2- 172.21.15.107:0/1365955377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc88c100630 0x7fc88c100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.645+0000 7fc8936c4640 1 -- 172.21.15.107:0/1365955377 >> 172.21.15.107:0/1365955377 conn(0x7fc88c077830 msgr2=0x7fc88c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:57.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.645+0000 7fc8936c4640 1 -- 172.21.15.107:0/1365955377 shutdown_connections 2024-05-09T08:39:57.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.645+0000 7fc8936c4640 1 -- 172.21.15.107:0/1365955377 wait complete. 2024-05-09T08:39:57.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.645+0000 7fc8936c4640 1 Processor -- start 2024-05-09T08:39:57.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.645+0000 7fc8936c4640 1 -- start start 2024-05-09T08:39:57.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc8936c4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc88c100630 0x7fc88c19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc8936c4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc88c100f50 0x7fc88c19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc8936c4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc88c10e1e0 0x7fc88c19ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc8936c4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c113cc0 con 0x7fc88c100630 2024-05-09T08:39:57.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc8936c4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c113b40 con 0x7fc88c100f50 2024-05-09T08:39:57.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc8936c4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c113e40 con 0x7fc88c10e1e0 2024-05-09T08:39:57.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc891460640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc88c100630 0x7fc88c19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc891460640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc88c100630 0x7fc88c19a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33522/0 (socket says 172.21.15.107:33522) 2024-05-09T08:39:57.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc891460640 1 -- 172.21.15.107:0/351432631 learned_addr learned my addr 172.21.15.107:0/351432631 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:57.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc891c61640 1 --2- 172.21.15.107:0/351432631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc88c10e1e0 0x7fc88c19ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc890c5f640 1 --2- 172.21.15.107:0/351432631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc88c100f50 0x7fc88c19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc891460640 1 -- 172.21.15.107:0/351432631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc88c10e1e0 msgr2=0x7fc88c19ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc891460640 1 --2- 172.21.15.107:0/351432631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc88c10e1e0 0x7fc88c19ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc891460640 1 -- 172.21.15.107:0/351432631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc88c100f50 msgr2=0x7fc88c19a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc891460640 1 --2- 172.21.15.107:0/351432631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc88c100f50 0x7fc88c19a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc891460640 1 -- 172.21.15.107:0/351432631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88c19f200 con 0x7fc88c100630 2024-05-09T08:39:57.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc891460640 1 --2- 172.21.15.107:0/351432631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc88c100630 0x7fc88c19a260 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc88800d770 tx=0x7fc88800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:57.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.646+0000 7fc8827fc640 1 -- 172.21.15.107:0/351432631 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc888017070 con 0x7fc88c100630 2024-05-09T08:39:57.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.647+0000 7fc8827fc640 1 -- 172.21.15.107:0/351432631 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc8880054d0 con 0x7fc88c100630 2024-05-09T08:39:57.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.647+0000 7fc8936c4640 1 -- 172.21.15.107:0/351432631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc88c19f4f0 con 0x7fc88c100630 2024-05-09T08:39:57.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.647+0000 7fc8827fc640 1 -- 172.21.15.107:0/351432631 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc888004cf0 con 0x7fc88c100630 2024-05-09T08:39:57.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.647+0000 7fc8936c4640 1 -- 172.21.15.107:0/351432631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc88c1a6d80 con 0x7fc88c100630 2024-05-09T08:39:57.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.650+0000 7fc8827fc640 1 -- 172.21.15.107:0/351432631 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fc888004e90 con 0x7fc88c100630 2024-05-09T08:39:57.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.650+0000 7fc8963d8740 1 Processor -- start 2024-05-09T08:39:57.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.650+0000 7fc8963d8740 1 -- start start 2024-05-09T08:39:57.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.650+0000 7fc8963d8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc700b0 0x55faadc6e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.650+0000 7fc8963d8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc6e9c0 0x55faadc6ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.650+0000 7fc8963d8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadc7bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.650+0000 7fc8963d8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55faadbd3990 con 0x55faadc79740 2024-05-09T08:39:57.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.650+0000 7fc8963d8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55faadb7b5b0 con 0x55faadc700b0 2024-05-09T08:39:57.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.650+0000 7fc8963d8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55faadbcf950 con 0x55faadc6e9c0 2024-05-09T08:39:57.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc890c5f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadc7bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc891c61640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc700b0 0x55faadc6e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc890c5f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadc7bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33530/0 (socket says 172.21.15.107:33530) 2024-05-09T08:39:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc890c5f640 1 -- 172.21.15.107:0/822547858 learned_addr learned my addr 172.21.15.107:0/822547858 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc891460640 1 --2- 172.21.15.107:0/822547858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc6e9c0 0x55faadc6ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc8827fc640 1 --2- 172.21.15.107:0/351432631 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc8780a2d80 0x7fc8780a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc891c61640 1 --2- 172.21.15.107:0/351432631 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc8780a2d80 0x7fc8780a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc890c5f640 1 -- 172.21.15.107:0/822547858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc6e9c0 msgr2=0x55faadc6ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc890c5f640 1 --2- 172.21.15.107:0/822547858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc6e9c0 0x55faadc6ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc890c5f640 1 -- 172.21.15.107:0/822547858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc700b0 msgr2=0x55faadc6e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc890c5f640 1 --2- 172.21.15.107:0/822547858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc700b0 0x55faadc6e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc890c5f640 1 -- 172.21.15.107:0/822547858 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faad9af7e0 con 0x55faadc79740 2024-05-09T08:39:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc8827fc640 1 -- 172.21.15.107:0/351432631 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fc8880cf0f0 con 0x7fc88c100630 2024-05-09T08:39:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc891c61640 1 --2- 172.21.15.107:0/822547858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc700b0 0x55faadc6e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc890c5f640 1 --2- 172.21.15.107:0/822547858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadc7bb30 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc87c009900 tx=0x7fc87c032b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a668e695c85e633 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:57.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc8777fe640 1 -- 172.21.15.107:0/822547858 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c040070 con 0x55faadc79740 2024-05-09T08:39:57.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc891c61640 1 --2- 172.21.15.107:0/351432631 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc8780a2d80 0x7fc8780a5240 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc88c06afc0 tx=0x7fc884004ab0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:57.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc8777fe640 1 -- 172.21.15.107:0/822547858 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc87c004030 con 0x55faadc79740 2024-05-09T08:39:57.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.651+0000 7fc8777fe640 1 -- 172.21.15.107:0/822547858 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c035040 con 0x55faadc79740 2024-05-09T08:39:57.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.652+0000 7fc8963d8740 1 -- 172.21.15.107:0/822547858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 msgr2=0x55faadc7bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.652+0000 7fc8963d8740 1 --2- 172.21.15.107:0/822547858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadc7bb30 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc87c009900 tx=0x7fc87c032b70 comp rx=0 tx=0).stop 2024-05-09T08:39:57.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.652+0000 7fc8963d8740 1 -- 172.21.15.107:0/822547858 shutdown_connections 2024-05-09T08:39:57.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.652+0000 7fc8963d8740 1 --2- 172.21.15.107:0/822547858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadc7bb30 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.652+0000 7fc8963d8740 1 --2- 172.21.15.107:0/822547858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc6e9c0 0x55faadc6ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.652+0000 7fc8963d8740 1 --2- 172.21.15.107:0/822547858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc700b0 0x55faadc6e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.652+0000 7fc8963d8740 1 -- 172.21.15.107:0/822547858 >> 172.21.15.107:0/822547858 conn(0x55faadc370a0 msgr2=0x55faadc6af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:57.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.652+0000 7fc8963d8740 1 -- 172.21.15.107:0/822547858 shutdown_connections 2024-05-09T08:39:57.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.652+0000 7fc8963d8740 1 -- 172.21.15.107:0/822547858 wait complete. 2024-05-09T08:39:57.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.653+0000 7fc8963d8740 1 -- start start 2024-05-09T08:39:57.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.653+0000 7fc8963d8740 1 Processor -- start 2024-05-09T08:39:57.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.653+0000 7fc8963d8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc6e9c0 0x55faadc76640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.653+0000 7fc8963d8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc700b0 0x55faadc76b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.653+0000 7fc8963d8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc79740 0x55faadc770c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.653+0000 7fc8963d8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55faadc7c570 con 0x55faadc6e9c0 2024-05-09T08:39:57.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.653+0000 7fc8963d8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55faadb8eb40 con 0x55faadc79740 2024-05-09T08:39:57.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.653+0000 7fc8963d8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55faadc7c6f0 con 0x55faadc700b0 2024-05-09T08:39:57.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.653+0000 7fc8963d8740 10 client.0 _ll_register_callbacks cb 0x55faadb830f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:57.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.653+0000 7fc891460640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc6e9c0 0x55faadc76640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.653+0000 7fc891460640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc6e9c0 0x55faadc76640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33538/0 (socket says 172.21.15.107:33538) 2024-05-09T08:39:57.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.653+0000 7fc891460640 1 -- 172.21.15.107:0/4290905274 learned_addr learned my addr 172.21.15.107:0/4290905274 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:57.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.653+0000 7fc890c5f640 1 --2- 172.21.15.107:0/4290905274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc700b0 0x55faadc76b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.653+0000 7fc891c61640 1 --2- 172.21.15.107:0/4290905274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc79740 0x55faadc770c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.653+0000 7fc891460640 1 -- 172.21.15.107:0/4290905274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc700b0 msgr2=0x55faadc76b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:57.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc891460640 1 --2- 172.21.15.107:0/4290905274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc700b0 0x55faadc76b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc891460640 1 -- 172.21.15.107:0/4290905274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc79740 msgr2=0x55faadc770c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc891460640 1 --2- 172.21.15.107:0/4290905274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc79740 0x55faadc770c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc891460640 1 -- 172.21.15.107:0/4290905274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faada0ff70 con 0x55faadc6e9c0 2024-05-09T08:39:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc891460640 1 --2- 172.21.15.107:0/4290905274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc6e9c0 0x55faadc76640 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fc8880ccae0 tx=0x7fc888097720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928a538c24bf2c53 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc874ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc874ff9640 1 -- 172.21.15.107:0/4290905274 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc888020040 con 0x55faadc6e9c0 2024-05-09T08:39:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc874ff9640 1 -- 172.21.15.107:0/4290905274 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fc8880979d0 con 0x55faadc6e9c0 2024-05-09T08:39:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc8963d8740 1 -- 172.21.15.107:0/4290905274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc87c004030 con 0x55faadc6e9c0 2024-05-09T08:39:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc874ff9640 1 -- 172.21.15.107:0/4290905274 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc88809acc0 con 0x55faadc6e9c0 2024-05-09T08:39:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc8963d8740 10 client.17625 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc8963d8740 1 -- 172.21.15.107:0/4290905274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55faad8a7170 con 0x55faadc6e9c0 2024-05-09T08:39:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc874ff9640 1 -- 172.21.15.107:0/4290905274 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc87c004030 con 0x55faadc6e9c0 2024-05-09T08:39:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc874ff9640 1 -- 172.21.15.107:0/4290905274 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc8880b40f0 con 0x55faadc6e9c0 2024-05-09T08:39:57.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc8963d8740 10 client.17625 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:57.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc8963d8740 10 client.17625 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:57.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc8963d8740 20 client.17625 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:57.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.654+0000 7fc8963d8740 1 --2- 172.21.15.107:0/4290905274 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55faadc897b0 0x55faadc8bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.655+0000 7fc8963d8740 4 client.17625 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:57.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.655+0000 7fc8963d8740 1 -- 172.21.15.107:0/4290905274 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55faadbe10c0 con 0x55faadc897b0 2024-05-09T08:39:57.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.655+0000 7fc890c5f640 1 --2- 172.21.15.107:0/4290905274 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55faadc897b0 0x55faadc8bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.655+0000 7fc874ff9640 10 client.17625 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:57.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.655+0000 7fc890c5f640 1 --2- 172.21.15.107:0/4290905274 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55faadc897b0 0x55faadc8bba0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c59cfaae73b49a2 server_cookie=5158c76ff1bc644 in_seq=0 out_seq=0 2024-05-09T08:39:57.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.656+0000 7fc874ff9640 1 -- 172.21.15.107:0/4290905274 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55faadbe10c0 con 0x55faadc897b0 2024-05-09T08:39:57.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.656+0000 7fc874ff9640 10 client.17625 handle_command_reply: tid=0 2024-05-09T08:39:57.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.656+0000 7fc8963d8740 1 client.17625 shutdown 2024-05-09T08:39:57.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.656+0000 7fc8963d8740 1 -- 172.21.15.107:0/4290905274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc6e9c0 msgr2=0x55faadc76640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.656+0000 7fc8963d8740 1 --2- 172.21.15.107:0/4290905274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc6e9c0 0x55faadc76640 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fc8880ccae0 tx=0x7fc888097720 comp rx=0 tx=0).stop 2024-05-09T08:39:57.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.656+0000 7fc8963d8740 1 -- 172.21.15.107:0/4290905274 shutdown_connections 2024-05-09T08:39:57.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.656+0000 7fc8963d8740 1 --2- 172.21.15.107:0/4290905274 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55faadc897b0 0x55faadc8bba0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.656+0000 7fc8963d8740 1 --2- 172.21.15.107:0/4290905274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc79740 0x55faadc770c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.656+0000 7fc874ff9640 0 client.17625 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:57.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.656+0000 7fc8963d8740 1 --2- 172.21.15.107:0/4290905274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc700b0 0x55faadc76b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.656+0000 7fc8963d8740 1 --2- 172.21.15.107:0/4290905274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc6e9c0 0x55faadc76640 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.656+0000 7fc8963d8740 1 -- 172.21.15.107:0/4290905274 >> 172.21.15.107:0/4290905274 conn(0x55faadc370a0 msgr2=0x55faadc70d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:57.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.656+0000 7fc8963d8740 1 -- 172.21.15.107:0/4290905274 shutdown_connections 2024-05-09T08:39:57.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.656+0000 7fc8963d8740 1 -- 172.21.15.107:0/4290905274 wait complete. 2024-05-09T08:39:57.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.657+0000 7fc8963d8740 20 client.17625 trim_cache size 0 max 16384 2024-05-09T08:39:57.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.676+0000 7fc8963d8740 1 Processor -- start 2024-05-09T08:39:57.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.676+0000 7fc8963d8740 1 -- start start 2024-05-09T08:39:57.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.676+0000 7fc8963d8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc700b0 0x55faadc69e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.676+0000 7fc8963d8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadc85470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.676+0000 7fc8963d8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc78580 0x55faadc91470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.676+0000 7fc8963d8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1b34e0 con 0x55faadc79740 2024-05-09T08:39:57.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.676+0000 7fc8963d8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1b0db0 con 0x55faadc700b0 2024-05-09T08:39:57.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.676+0000 7fc8963d8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1aec00 con 0x55faadc78580 2024-05-09T08:39:57.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.676+0000 7fc890c5f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadc85470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.676+0000 7fc890c5f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadc85470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33548/0 (socket says 172.21.15.107:33548) 2024-05-09T08:39:57.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.676+0000 7fc891c61640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc78580 0x55faadc91470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.676+0000 7fc890c5f640 1 -- 172.21.15.107:0/100321943 learned_addr learned my addr 172.21.15.107:0/100321943 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:57.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.676+0000 7fc891460640 1 --2- 172.21.15.107:0/100321943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc700b0 0x55faadc69e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.676+0000 7fc890c5f640 1 -- 172.21.15.107:0/100321943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc78580 msgr2=0x55faadc91470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:57.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.677+0000 7fc890c5f640 1 --2- 172.21.15.107:0/100321943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc78580 0x55faadc91470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.677+0000 7fc890c5f640 1 -- 172.21.15.107:0/100321943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc700b0 msgr2=0x55faadc69e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:57.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.677+0000 7fc890c5f640 1 --2- 172.21.15.107:0/100321943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc700b0 0x55faadc69e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.677+0000 7fc890c5f640 1 -- 172.21.15.107:0/100321943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faadc6f260 con 0x55faadc79740 2024-05-09T08:39:57.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.677+0000 7fc890c5f640 1 --2- 172.21.15.107:0/100321943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadc85470 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fc87c043000 tx=0x7fc87c005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc419723bb508ed3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:57.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.677+0000 7fc876ffd640 1 -- 172.21.15.107:0/100321943 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c035040 con 0x55faadc79740 2024-05-09T08:39:57.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.677+0000 7fc876ffd640 1 -- 172.21.15.107:0/100321943 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc87c0314a0 con 0x55faadc79740 2024-05-09T08:39:57.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.677+0000 7fc876ffd640 1 -- 172.21.15.107:0/100321943 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c040040 con 0x55faadc79740 2024-05-09T08:39:57.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.678+0000 7fc8963d8740 1 -- 172.21.15.107:0/100321943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 msgr2=0x55faadc85470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.678+0000 7fc8963d8740 1 --2- 172.21.15.107:0/100321943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadc85470 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fc87c043000 tx=0x7fc87c005db0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.678+0000 7fc8963d8740 1 -- 172.21.15.107:0/100321943 shutdown_connections 2024-05-09T08:39:57.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.678+0000 7fc8963d8740 1 --2- 172.21.15.107:0/100321943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc78580 0x55faadc91470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.678+0000 7fc8963d8740 1 --2- 172.21.15.107:0/100321943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadc85470 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.678+0000 7fc8963d8740 1 --2- 172.21.15.107:0/100321943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc700b0 0x55faadc69e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.678+0000 7fc8963d8740 1 -- 172.21.15.107:0/100321943 >> 172.21.15.107:0/100321943 conn(0x55faadc370a0 msgr2=0x55faadc77e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:57.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.678+0000 7fc8963d8740 1 -- 172.21.15.107:0/100321943 shutdown_connections 2024-05-09T08:39:57.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.678+0000 7fc8963d8740 1 -- 172.21.15.107:0/100321943 wait complete. 2024-05-09T08:39:57.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.678+0000 7fc8963d8740 1 -- start start 2024-05-09T08:39:57.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.678+0000 7fc8963d8740 1 Processor -- start 2024-05-09T08:39:57.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc8963d8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc700b0 0x55faadc7ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc8963d8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc78580 0x55faadc7ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc8963d8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadcefcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc8963d8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1af270 con 0x55faadc79740 2024-05-09T08:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc8963d8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1b06e0 con 0x55faadc78580 2024-05-09T08:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc8963d8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1af940 con 0x55faadc700b0 2024-05-09T08:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc8963d8740 10 client.0 _ll_register_callbacks cb 0x7fc888009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc891c61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadcefcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc890c5f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc78580 0x55faadc7ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc891c61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadcefcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33558/0 (socket says 172.21.15.107:33558) 2024-05-09T08:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc891c61640 1 -- 172.21.15.107:0/3285896569 learned_addr learned my addr 172.21.15.107:0/3285896569 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc891460640 1 --2- 172.21.15.107:0/3285896569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc700b0 0x55faadc7ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc891c61640 1 -- 172.21.15.107:0/3285896569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc700b0 msgr2=0x55faadc7ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc891c61640 1 --2- 172.21.15.107:0/3285896569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc700b0 0x55faadc7ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc891c61640 1 -- 172.21.15.107:0/3285896569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc78580 msgr2=0x55faadc7ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc891c61640 1 --2- 172.21.15.107:0/3285896569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc78580 0x55faadc7ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc891c61640 1 -- 172.21.15.107:0/3285896569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faadcf0230 con 0x55faadc79740 2024-05-09T08:39:57.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc890c5f640 1 --2- 172.21.15.107:0/3285896569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc78580 0x55faadc7ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc891460640 1 --2- 172.21.15.107:0/3285896569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc700b0 0x55faadc7ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc891c61640 1 --2- 172.21.15.107:0/3285896569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadcefcf0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fc884012c50 tx=0x7fc884004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75d6838ad58c94a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc874ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc874ff9640 1 -- 172.21.15.107:0/3285896569 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88401a070 con 0x55faadc79740 2024-05-09T08:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc874ff9640 1 -- 172.21.15.107:0/3285896569 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fc884014020 con 0x55faadc79740 2024-05-09T08:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.679+0000 7fc8963d8740 1 -- 172.21.15.107:0/3285896569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc87c0314a0 con 0x55faadc79740 2024-05-09T08:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.680+0000 7fc874ff9640 1 -- 172.21.15.107:0/3285896569 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc884014220 con 0x55faadc79740 2024-05-09T08:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.680+0000 7fc8963d8740 10 client.17631 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.680+0000 7fc8963d8740 1 -- 172.21.15.107:0/3285896569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55faadcf1540 con 0x55faadc79740 2024-05-09T08:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.680+0000 7fc874ff9640 1 -- 172.21.15.107:0/3285896569 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc87c0314a0 con 0x55faadc79740 2024-05-09T08:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.680+0000 7fc874ff9640 1 -- 172.21.15.107:0/3285896569 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc88401d070 con 0x55faadc79740 2024-05-09T08:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.680+0000 7fc8963d8740 10 client.17631 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.680+0000 7fc8963d8740 10 client.17631 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.680+0000 7fc8963d8740 20 client.17631 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:57.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.680+0000 7fc8963d8740 1 --2- 172.21.15.107:0/3285896569 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55faadcf1cf0 0x55faadcf4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.680+0000 7fc8963d8740 4 client.17631 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:39:57.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.680+0000 7fc8963d8740 1 -- 172.21.15.107:0/3285896569 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55faadbdb410 con 0x55faadcf1cf0 2024-05-09T08:39:57.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.680+0000 7fc891460640 1 --2- 172.21.15.107:0/3285896569 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55faadcf1cf0 0x55faadcf4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.681+0000 7fc891460640 1 --2- 172.21.15.107:0/3285896569 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55faadcf1cf0 0x55faadcf4110 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a079aeb6a7bb41d server_cookie=46e8504f6040f32a in_seq=0 out_seq=0 2024-05-09T08:39:57.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.681+0000 7fc874ff9640 10 client.17631 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:57.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.682+0000 7fc874ff9640 1 -- 172.21.15.107:0/3285896569 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55faadbdb410 con 0x55faadcf1cf0 2024-05-09T08:39:57.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.682+0000 7fc874ff9640 10 client.17631 handle_command_reply: tid=0 2024-05-09T08:39:57.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.682+0000 7fc8963d8740 1 client.17631 shutdown 2024-05-09T08:39:57.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.682+0000 7fc8963d8740 1 -- 172.21.15.107:0/3285896569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 msgr2=0x55faadcefcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.682+0000 7fc8963d8740 1 --2- 172.21.15.107:0/3285896569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadcefcf0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fc884012c50 tx=0x7fc884004a40 comp rx=0 tx=0).stop 2024-05-09T08:39:57.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.682+0000 7fc8963d8740 1 -- 172.21.15.107:0/3285896569 shutdown_connections 2024-05-09T08:39:57.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.682+0000 7fc8963d8740 1 --2- 172.21.15.107:0/3285896569 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55faadcf1cf0 0x55faadcf4110 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.682+0000 7fc8963d8740 1 --2- 172.21.15.107:0/3285896569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55faadc79740 0x55faadcefcf0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.682+0000 7fc8963d8740 1 --2- 172.21.15.107:0/3285896569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faadc78580 0x55faadc7ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.682+0000 7fc8963d8740 1 --2- 172.21.15.107:0/3285896569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faadc700b0 0x55faadc7ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.682+0000 7fc8963d8740 1 -- 172.21.15.107:0/3285896569 >> 172.21.15.107:0/3285896569 conn(0x55faadc370a0 msgr2=0x55faadc76600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:57.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.682+0000 7fc874ff9640 0 client.17631 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:57.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.682+0000 7fc8963d8740 1 -- 172.21.15.107:0/3285896569 shutdown_connections 2024-05-09T08:39:57.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.682+0000 7fc8963d8740 1 -- 172.21.15.107:0/3285896569 wait complete. 2024-05-09T08:39:57.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.682+0000 7fc8963d8740 20 client.17631 trim_cache size 0 max 16384 2024-05-09T08:39:57.691 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:39:57.691 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:39:57.691 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "ff8f8b46-ad21-412f-9eff-94ddcf0491a3", 2024-05-09T08:39:57.692 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:39:57.692 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:39:57.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.683+0000 7fc8936c4640 1 -- 172.21.15.107:0/351432631 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc8780a2d80 msgr2=0x7fc8780a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.683+0000 7fc8936c4640 1 --2- 172.21.15.107:0/351432631 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc8780a2d80 0x7fc8780a5240 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc88c06afc0 tx=0x7fc884004ab0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.683+0000 7fc8936c4640 1 -- 172.21.15.107:0/351432631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc88c100630 msgr2=0x7fc88c19a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.683+0000 7fc8936c4640 1 --2- 172.21.15.107:0/351432631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc88c100630 0x7fc88c19a260 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc88800d770 tx=0x7fc88800dc30 comp rx=0 tx=0).stop 2024-05-09T08:39:57.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.683+0000 7fc8936c4640 1 -- 172.21.15.107:0/351432631 shutdown_connections 2024-05-09T08:39:57.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.683+0000 7fc8936c4640 1 --2- 172.21.15.107:0/351432631 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc8780a2d80 0x7fc8780a5240 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.683+0000 7fc8936c4640 1 --2- 172.21.15.107:0/351432631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc88c10e1e0 0x7fc88c19ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.683+0000 7fc8936c4640 1 --2- 172.21.15.107:0/351432631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc88c100f50 0x7fc88c19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.683+0000 7fc8936c4640 1 --2- 172.21.15.107:0/351432631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc88c100630 0x7fc88c19a260 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.683+0000 7fc8936c4640 1 -- 172.21.15.107:0/351432631 >> 172.21.15.107:0/351432631 conn(0x7fc88c077830 msgr2=0x7fc88c10f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:57.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.683+0000 7fc8936c4640 1 -- 172.21.15.107:0/351432631 shutdown_connections 2024-05-09T08:39:57.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.683+0000 7fc8936c4640 1 -- 172.21.15.107:0/351432631 wait complete. 2024-05-09T08:39:57.695 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:39:57.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:39:57.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.815+0000 7f82f4009640 1 -- 172.21.15.107:0/4166987814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec106710 msgr2=0x7f82ec106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.815+0000 7f82f4009640 1 --2- 172.21.15.107:0/4166987814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec106710 0x7f82ec106af0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f82e8009930 tx=0x7f82e802a420 comp rx=0 tx=0).stop 2024-05-09T08:39:57.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.815+0000 7f82f4009640 1 -- 172.21.15.107:0/4166987814 shutdown_connections 2024-05-09T08:39:57.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.815+0000 7f82f4009640 1 --2- 172.21.15.107:0/4166987814 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82ec1010c0 0x7f82ec10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.815+0000 7f82f4009640 1 --2- 172.21.15.107:0/4166987814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82ec100700 0x7f82ec100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.815+0000 7f82f4009640 1 --2- 172.21.15.107:0/4166987814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec106710 0x7f82ec106af0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.815+0000 7f82f4009640 1 -- 172.21.15.107:0/4166987814 >> 172.21.15.107:0/4166987814 conn(0x7f82ec077830 msgr2=0x7f82ec077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:57.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.815+0000 7f82f4009640 1 -- 172.21.15.107:0/4166987814 shutdown_connections 2024-05-09T08:39:57.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.815+0000 7f82f4009640 1 -- 172.21.15.107:0/4166987814 wait complete. 2024-05-09T08:39:57.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.815+0000 7f82f4009640 1 Processor -- start 2024-05-09T08:39:57.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f4009640 1 -- start start 2024-05-09T08:39:57.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f4009640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82ec100700 0x7f82ec197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f4009640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec1010c0 0x7f82ec198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f4009640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82ec106710 0x7f82ec19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f4009640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec1109e0 con 0x7f82ec1010c0 2024-05-09T08:39:57.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f4009640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec110860 con 0x7f82ec100700 2024-05-09T08:39:57.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f4009640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec110b60 con 0x7f82ec106710 2024-05-09T08:39:57.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f15a4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec1010c0 0x7f82ec198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f1da5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82ec100700 0x7f82ec197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f15a4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec1010c0 0x7f82ec198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33576/0 (socket says 172.21.15.107:33576) 2024-05-09T08:39:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f1da5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82ec100700 0x7f82ec197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:37232/0 (socket says 172.21.15.107:37232) 2024-05-09T08:39:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f15a4640 1 -- 172.21.15.107:0/1054435805 learned_addr learned my addr 172.21.15.107:0/1054435805 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f25a6640 1 --2- 172.21.15.107:0/1054435805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82ec106710 0x7f82ec19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f15a4640 1 -- 172.21.15.107:0/1054435805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82ec106710 msgr2=0x7f82ec19c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f15a4640 1 --2- 172.21.15.107:0/1054435805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82ec106710 0x7f82ec19c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f15a4640 1 -- 172.21.15.107:0/1054435805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82ec100700 msgr2=0x7f82ec197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f15a4640 1 --2- 172.21.15.107:0/1054435805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82ec100700 0x7f82ec197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f15a4640 1 -- 172.21.15.107:0/1054435805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82ec19cfd0 con 0x7f82ec1010c0 2024-05-09T08:39:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f25a6640 1 --2- 172.21.15.107:0/1054435805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82ec106710 0x7f82ec19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.816+0000 7f82f15a4640 1 --2- 172.21.15.107:0/1054435805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec1010c0 0x7f82ec198520 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00d950 tx=0x7f82dc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.817+0000 7f82e2ffd640 1 -- 172.21.15.107:0/1054435805 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc017030 con 0x7f82ec1010c0 2024-05-09T08:39:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.817+0000 7f82e2ffd640 1 -- 172.21.15.107:0/1054435805 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82dc004020 con 0x7f82ec1010c0 2024-05-09T08:39:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.817+0000 7f82f4009640 1 -- 172.21.15.107:0/1054435805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82ec19d2c0 con 0x7f82ec1010c0 2024-05-09T08:39:57.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.817+0000 7f82e2ffd640 1 -- 172.21.15.107:0/1054435805 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc0047a0 con 0x7f82ec1010c0 2024-05-09T08:39:57.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.817+0000 7f82f4009640 1 -- 172.21.15.107:0/1054435805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82ec1a4b50 con 0x7f82ec1010c0 2024-05-09T08:39:57.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.819+0000 7f82e2ffd640 1 -- 172.21.15.107:0/1054435805 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f82dc0041d0 con 0x7f82ec1010c0 2024-05-09T08:39:57.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.820+0000 7f82f6df1740 1 Processor -- start 2024-05-09T08:39:57.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.820+0000 7f82f6df1740 1 -- start start 2024-05-09T08:39:57.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.820+0000 7f82f6df1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a36280 0x563900a36660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.820+0000 7f82f6df1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36ba0 0x563900a40d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.820+0000 7f82f6df1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a41270 0x563900a43660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.820+0000 7f82f6df1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56390099bd00 con 0x563900a36280 2024-05-09T08:39:57.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.820+0000 7f82f6df1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5639009434c0 con 0x563900a36ba0 2024-05-09T08:39:57.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.820+0000 7f82f6df1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563900997cc0 con 0x563900a41270 2024-05-09T08:39:57.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.820+0000 7f82e2ffd640 1 --2- 172.21.15.107:0/1054435805 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f82d80a2d00 0x7f82d80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.821+0000 7f82f25a6640 1 --2- 172.21.15.107:0/1054435805 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f82d80a2d00 0x7f82d80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.821+0000 7f82f15a4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a41270 0x563900a43660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.821+0000 7f82f1da5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36ba0 0x563900a40d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.821+0000 7f82f25a6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a36280 0x563900a36660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.821+0000 7f82e2ffd640 1 -- 172.21.15.107:0/1054435805 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f82dc0cf250 con 0x7f82ec1010c0 2024-05-09T08:39:57.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.821+0000 7f82f1da5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36ba0 0x563900a40d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:37238/0 (socket says 172.21.15.107:37238) 2024-05-09T08:39:57.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.821+0000 7f82f15a4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a41270 0x563900a43660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:53488/0 (socket says 172.21.15.107:53488) 2024-05-09T08:39:57.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.821+0000 7f82f1da5640 1 -- 172.21.15.107:0/3726922247 learned_addr learned my addr 172.21.15.107:0/3726922247 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:57.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.821+0000 7f82f1da5640 1 -- 172.21.15.107:0/3726922247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a41270 msgr2=0x563900a43660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:57.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.821+0000 7f82f1da5640 1 --2- 172.21.15.107:0/3726922247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a41270 0x563900a43660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.821+0000 7f82f1da5640 1 -- 172.21.15.107:0/3726922247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a36280 msgr2=0x563900a36660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:57.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.821+0000 7f82f1da5640 1 --2- 172.21.15.107:0/3726922247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a36280 0x563900a36660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.821+0000 7f82f1da5640 1 -- 172.21.15.107:0/3726922247 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639007797d0 con 0x563900a36ba0 2024-05-09T08:39:57.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.821+0000 7f82f25a6640 1 --2- 172.21.15.107:0/1054435805 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f82d80a2d00 0x7f82d80a51c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f82e4012a20 tx=0x7f82e400ac70 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:57.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.821+0000 7f82f1da5640 1 --2- 172.21.15.107:0/3726922247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36ba0 0x563900a40d30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f82e802aa10 tx=0x7f82e8004310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=810a91e6b5b97009 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:57.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.822+0000 7f82d7fff640 1 -- 172.21.15.107:0/3726922247 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8035040 con 0x563900a36ba0 2024-05-09T08:39:57.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.822+0000 7f82d7fff640 1 -- 172.21.15.107:0/3726922247 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82e80054f0 con 0x563900a36ba0 2024-05-09T08:39:57.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.822+0000 7f82d7fff640 1 -- 172.21.15.107:0/3726922247 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8003850 con 0x563900a36ba0 2024-05-09T08:39:57.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.822+0000 7f82f6df1740 1 -- 172.21.15.107:0/3726922247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36ba0 msgr2=0x563900a40d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.822+0000 7f82f6df1740 1 --2- 172.21.15.107:0/3726922247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36ba0 0x563900a40d30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f82e802aa10 tx=0x7f82e8004310 comp rx=0 tx=0).stop 2024-05-09T08:39:57.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.822+0000 7f82f6df1740 1 -- 172.21.15.107:0/3726922247 shutdown_connections 2024-05-09T08:39:57.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.822+0000 7f82f6df1740 1 --2- 172.21.15.107:0/3726922247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a41270 0x563900a43660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.822+0000 7f82f6df1740 1 --2- 172.21.15.107:0/3726922247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36ba0 0x563900a40d30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.822+0000 7f82f6df1740 1 --2- 172.21.15.107:0/3726922247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a36280 0x563900a36660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.822+0000 7f82f6df1740 1 -- 172.21.15.107:0/3726922247 >> 172.21.15.107:0/3726922247 conn(0x563900a325a0 msgr2=0x563900a32980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:57.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.823+0000 7f82f6df1740 1 -- 172.21.15.107:0/3726922247 shutdown_connections 2024-05-09T08:39:57.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.823+0000 7f82f6df1740 1 -- 172.21.15.107:0/3726922247 wait complete. 2024-05-09T08:39:57.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.823+0000 7f82f6df1740 1 -- start start 2024-05-09T08:39:57.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.823+0000 7f82f6df1740 1 Processor -- start 2024-05-09T08:39:57.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.823+0000 7f82f6df1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36280 0x563900a3e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.823+0000 7f82f6df1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 0x563900a3e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.823+0000 7f82f6df1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a41270 0x563900a3ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.823+0000 7f82f6df1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563900a44000 con 0x563900a41270 2024-05-09T08:39:57.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.823+0000 7f82f6df1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563900956b90 con 0x563900a36280 2024-05-09T08:39:57.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.823+0000 7f82f6df1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563900a44180 con 0x563900a36ba0 2024-05-09T08:39:57.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.823+0000 7f82f6df1740 10 client.0 _ll_register_callbacks cb 0x563900992da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:57.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82f25a6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a41270 0x563900a3ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82f25a6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a41270 0x563900a3ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33582/0 (socket says 172.21.15.107:33582) 2024-05-09T08:39:57.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82f25a6640 1 -- 172.21.15.107:0/2090481372 learned_addr learned my addr 172.21.15.107:0/2090481372 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:57.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82f1da5640 1 --2- 172.21.15.107:0/2090481372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36280 0x563900a3e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82f15a4640 1 --2- 172.21.15.107:0/2090481372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 0x563900a3e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82f25a6640 1 -- 172.21.15.107:0/2090481372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 msgr2=0x563900a3e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:57.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82f25a6640 1 --2- 172.21.15.107:0/2090481372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 0x563900a3e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82f25a6640 1 -- 172.21.15.107:0/2090481372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36280 msgr2=0x563900a3e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:57.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82f25a6640 1 --2- 172.21.15.107:0/2090481372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36280 0x563900a3e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82f25a6640 1 -- 172.21.15.107:0/2090481372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639006af3f0 con 0x563900a41270 2024-05-09T08:39:57.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82f15a4640 1 --2- 172.21.15.107:0/2090481372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 0x563900a3e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:39:57.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82f25a6640 1 --2- 172.21.15.107:0/2090481372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a41270 0x563900a3ec60 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f82e400fe90 tx=0x7f82e4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4502ad18d15adff7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:57.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:57.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82d57fa640 1 -- 172.21.15.107:0/2090481372 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e4004b20 con 0x563900a41270 2024-05-09T08:39:57.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82d57fa640 1 -- 172.21.15.107:0/2090481372 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f82e4004cc0 con 0x563900a41270 2024-05-09T08:39:57.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82f6df1740 1 -- 172.21.15.107:0/2090481372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82e80054f0 con 0x563900a41270 2024-05-09T08:39:57.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82d57fa640 1 -- 172.21.15.107:0/2090481372 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f82e401c5f0 con 0x563900a41270 2024-05-09T08:39:57.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82f6df1740 10 client.17643 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:57.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.824+0000 7f82f6df1740 1 -- 172.21.15.107:0/2090481372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56390066f0f0 con 0x563900a41270 2024-05-09T08:39:57.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.825+0000 7f82d57fa640 1 -- 172.21.15.107:0/2090481372 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82e80054f0 con 0x563900a41270 2024-05-09T08:39:57.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.825+0000 7f82d57fa640 1 -- 172.21.15.107:0/2090481372 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f82e401b040 con 0x563900a41270 2024-05-09T08:39:57.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.825+0000 7f82f6df1740 10 client.17643 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:57.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.825+0000 7f82f6df1740 10 client.17643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:57.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.825+0000 7f82f6df1740 20 client.17643 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:57.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.825+0000 7f82f6df1740 1 --2- 172.21.15.107:0/2090481372 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563900a515c0 0x563900a53a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.825+0000 7f82f6df1740 4 client.17643 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:39:57.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.825+0000 7f82f6df1740 1 -- 172.21.15.107:0/2090481372 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639009a9430 con 0x563900a515c0 2024-05-09T08:39:57.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.825+0000 7f82f1da5640 1 --2- 172.21.15.107:0/2090481372 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563900a515c0 0x563900a53a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.826+0000 7f82d57fa640 10 client.17643 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:57.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.826+0000 7f82f1da5640 1 --2- 172.21.15.107:0/2090481372 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563900a515c0 0x563900a53a10 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0407bc271fc164c server_cookie=b05d7cc3d1e11ec1 in_seq=0 out_seq=0 2024-05-09T08:39:57.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.826+0000 7f82d57fa640 1 -- 172.21.15.107:0/2090481372 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5639009a9430 con 0x563900a515c0 2024-05-09T08:39:57.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.826+0000 7f82d57fa640 10 client.17643 handle_command_reply: tid=0 2024-05-09T08:39:57.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.826+0000 7f82f6df1740 1 client.17643 shutdown 2024-05-09T08:39:57.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.827+0000 7f82f6df1740 1 -- 172.21.15.107:0/2090481372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a41270 msgr2=0x563900a3ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.827+0000 7f82f6df1740 1 --2- 172.21.15.107:0/2090481372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a41270 0x563900a3ec60 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f82e400fe90 tx=0x7f82e4005f90 comp rx=0 tx=0).stop 2024-05-09T08:39:57.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.827+0000 7f82f6df1740 1 -- 172.21.15.107:0/2090481372 shutdown_connections 2024-05-09T08:39:57.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.827+0000 7f82f6df1740 1 --2- 172.21.15.107:0/2090481372 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563900a515c0 0x563900a53a10 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.827+0000 7f82f6df1740 1 --2- 172.21.15.107:0/2090481372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a41270 0x563900a3ec60 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.827+0000 7f82f6df1740 1 --2- 172.21.15.107:0/2090481372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 0x563900a3e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.827+0000 7f82f6df1740 1 --2- 172.21.15.107:0/2090481372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36280 0x563900a3e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.827+0000 7f82d57fa640 0 client.17643 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:57.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.827+0000 7f82f6df1740 1 -- 172.21.15.107:0/2090481372 >> 172.21.15.107:0/2090481372 conn(0x563900a325a0 msgr2=0x563900a317e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:57.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.827+0000 7f82f6df1740 1 -- 172.21.15.107:0/2090481372 shutdown_connections 2024-05-09T08:39:57.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.827+0000 7f82f6df1740 1 -- 172.21.15.107:0/2090481372 wait complete. 2024-05-09T08:39:57.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.827+0000 7f82f6df1740 20 client.17643 trim_cache size 0 max 16384 2024-05-09T08:39:57.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.847+0000 7f82f6df1740 1 Processor -- start 2024-05-09T08:39:57.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.847+0000 7f82f6df1740 1 -- start start 2024-05-09T08:39:57.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.847+0000 7f82f6df1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a36280 0x563900a4cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.847+0000 7f82f6df1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 0x563900a389e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.847+0000 7f82f6df1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a41270 0x563900a592e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.847+0000 7f82f6df1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec1b13d0 con 0x563900a36280 2024-05-09T08:39:57.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.847+0000 7f82f6df1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec1aeca0 con 0x563900a41270 2024-05-09T08:39:57.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.847+0000 7f82f6df1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec1acaf0 con 0x563900a36ba0 2024-05-09T08:39:57.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.848+0000 7f82f1da5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a36280 0x563900a4cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.848+0000 7f82f1da5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a36280 0x563900a4cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33588/0 (socket says 172.21.15.107:33588) 2024-05-09T08:39:57.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.848+0000 7f82f1da5640 1 -- 172.21.15.107:0/2068670836 learned_addr learned my addr 172.21.15.107:0/2068670836 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:57.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.848+0000 7f82f25a6640 1 --2- 172.21.15.107:0/2068670836 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a41270 0x563900a592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.848+0000 7f82f15a4640 1 --2- 172.21.15.107:0/2068670836 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 0x563900a389e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.848+0000 7f82f1da5640 1 -- 172.21.15.107:0/2068670836 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 msgr2=0x563900a389e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:57.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.848+0000 7f82f1da5640 1 --2- 172.21.15.107:0/2068670836 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 0x563900a389e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.848+0000 7f82f1da5640 1 -- 172.21.15.107:0/2068670836 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a41270 msgr2=0x563900a592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:57.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.848+0000 7f82f1da5640 1 --2- 172.21.15.107:0/2068670836 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a41270 0x563900a592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.848+0000 7f82f1da5640 1 -- 172.21.15.107:0/2068670836 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563900a370a0 con 0x563900a36280 2024-05-09T08:39:57.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.848+0000 7f82f15a4640 1 --2- 172.21.15.107:0/2068670836 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 0x563900a389e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:57.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.848+0000 7f82f25a6640 1 --2- 172.21.15.107:0/2068670836 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a41270 0x563900a592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:57.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.848+0000 7f82f1da5640 1 --2- 172.21.15.107:0/2068670836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a36280 0x563900a4cec0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f82e8006010 tx=0x7f82e80048e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d713fcf8663db670 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:57.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.848+0000 7f82d77fe640 1 -- 172.21.15.107:0/2068670836 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8056070 con 0x563900a36280 2024-05-09T08:39:57.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.848+0000 7f82d77fe640 1 -- 172.21.15.107:0/2068670836 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82e8051520 con 0x563900a36280 2024-05-09T08:39:57.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.848+0000 7f82d77fe640 1 -- 172.21.15.107:0/2068670836 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8035040 con 0x563900a36280 2024-05-09T08:39:57.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.849+0000 7f82f6df1740 1 -- 172.21.15.107:0/2068670836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a36280 msgr2=0x563900a4cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.849+0000 7f82f6df1740 1 --2- 172.21.15.107:0/2068670836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a36280 0x563900a4cec0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f82e8006010 tx=0x7f82e80048e0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.849+0000 7f82f6df1740 1 -- 172.21.15.107:0/2068670836 shutdown_connections 2024-05-09T08:39:57.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.849+0000 7f82f6df1740 1 --2- 172.21.15.107:0/2068670836 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a41270 0x563900a592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.849+0000 7f82f6df1740 1 --2- 172.21.15.107:0/2068670836 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 0x563900a389e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.849+0000 7f82f6df1740 1 --2- 172.21.15.107:0/2068670836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a36280 0x563900a4cec0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.849+0000 7f82f6df1740 1 -- 172.21.15.107:0/2068670836 >> 172.21.15.107:0/2068670836 conn(0x563900a325a0 msgr2=0x563900a3ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:57.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.849+0000 7f82f6df1740 1 -- 172.21.15.107:0/2068670836 shutdown_connections 2024-05-09T08:39:57.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.849+0000 7f82f6df1740 1 -- 172.21.15.107:0/2068670836 wait complete. 2024-05-09T08:39:57.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.849+0000 7f82f6df1740 1 -- start start 2024-05-09T08:39:57.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.849+0000 7f82f6df1740 1 Processor -- start 2024-05-09T08:39:57.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f6df1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36280 0x563900a3b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f6df1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 0x563900a3b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f6df1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a41270 0x563900a3baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f6df1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec1ad160 con 0x563900a41270 2024-05-09T08:39:57.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f6df1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec1ae5d0 con 0x563900a36280 2024-05-09T08:39:57.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f6df1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec1ad830 con 0x563900a36ba0 2024-05-09T08:39:57.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f6df1740 10 client.0 _ll_register_callbacks cb 0x7f82e400ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:39:57.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f25a6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a41270 0x563900a3baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f25a6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a41270 0x563900a3baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33590/0 (socket says 172.21.15.107:33590) 2024-05-09T08:39:57.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f25a6640 1 -- 172.21.15.107:0/3047996601 learned_addr learned my addr 172.21.15.107:0/3047996601 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:39:57.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f1da5640 1 --2- 172.21.15.107:0/3047996601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36280 0x563900a3b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f15a4640 1 --2- 172.21.15.107:0/3047996601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 0x563900a3b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f25a6640 1 -- 172.21.15.107:0/3047996601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 msgr2=0x563900a3b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:57.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f25a6640 1 --2- 172.21.15.107:0/3047996601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 0x563900a3b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f25a6640 1 -- 172.21.15.107:0/3047996601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36280 msgr2=0x563900a3b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:39:57.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f25a6640 1 --2- 172.21.15.107:0/3047996601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36280 0x563900a3b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f25a6640 1 -- 172.21.15.107:0/3047996601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563900ab6d00 con 0x563900a41270 2024-05-09T08:39:57.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f15a4640 1 --2- 172.21.15.107:0/3047996601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 0x563900a3b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:39:57.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82f25a6640 1 --2- 172.21.15.107:0/3047996601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a41270 0x563900a3baa0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f82e4005490 tx=0x7f82e4012660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a03431b5048e532 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:39:57.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:39:57.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.850+0000 7f82d57fa640 1 -- 172.21.15.107:0/3047996601 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e4012240 con 0x563900a41270 2024-05-09T08:39:57.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.851+0000 7f82f6df1740 1 -- 172.21.15.107:0/3047996601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82e8051520 con 0x563900a41270 2024-05-09T08:39:57.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.851+0000 7f82d57fa640 1 -- 172.21.15.107:0/3047996601 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f82e4004ab0 con 0x563900a41270 2024-05-09T08:39:57.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.851+0000 7f82d57fa640 1 -- 172.21.15.107:0/3047996601 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f82e4041050 con 0x563900a41270 2024-05-09T08:39:57.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.851+0000 7f82f6df1740 10 client.17649 fetch_fsmap learned FSMap version 27 2024-05-09T08:39:57.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.851+0000 7f82f6df1740 1 -- 172.21.15.107:0/3047996601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563900ab7f40 con 0x563900a41270 2024-05-09T08:39:57.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.851+0000 7f82d57fa640 1 -- 172.21.15.107:0/3047996601 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82e8051520 con 0x563900a41270 2024-05-09T08:39:57.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.851+0000 7f82d57fa640 1 -- 172.21.15.107:0/3047996601 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f82e401b070 con 0x563900a41270 2024-05-09T08:39:57.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.851+0000 7f82f6df1740 10 client.17649 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:39:57.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.851+0000 7f82f6df1740 10 client.17649 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:39:57.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.851+0000 7f82f6df1740 20 client.17649 populate_metadata read hostname 'smithi107' 2024-05-09T08:39:57.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.851+0000 7f82f6df1740 1 --2- 172.21.15.107:0/3047996601 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563900ab8740 0x563900abab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:39:57.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.851+0000 7f82f6df1740 4 client.17649 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:39:57.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.851+0000 7f82f6df1740 1 -- 172.21.15.107:0/3047996601 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5639009a3780 con 0x563900ab8740 2024-05-09T08:39:57.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.851+0000 7f82f1da5640 1 --2- 172.21.15.107:0/3047996601 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563900ab8740 0x563900abab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:39:57.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82d57fa640 10 client.17649 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:57.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82f1da5640 1 --2- 172.21.15.107:0/3047996601 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563900ab8740 0x563900abab60 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72e53c47841dc144 server_cookie=875056ebb9b5f8ea in_seq=0 out_seq=0 2024-05-09T08:39:57.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82d57fa640 1 -- 172.21.15.107:0/3047996601 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5639009a3780 con 0x563900ab8740 2024-05-09T08:39:57.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82d57fa640 10 client.17649 handle_command_reply: tid=0 2024-05-09T08:39:57.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82f6df1740 1 client.17649 shutdown 2024-05-09T08:39:57.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82f6df1740 1 -- 172.21.15.107:0/3047996601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a41270 msgr2=0x563900a3baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82f6df1740 1 --2- 172.21.15.107:0/3047996601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a41270 0x563900a3baa0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f82e4005490 tx=0x7f82e4012660 comp rx=0 tx=0).stop 2024-05-09T08:39:57.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82f6df1740 1 -- 172.21.15.107:0/3047996601 shutdown_connections 2024-05-09T08:39:57.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82f6df1740 1 --2- 172.21.15.107:0/3047996601 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563900ab8740 0x563900abab60 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82f6df1740 1 --2- 172.21.15.107:0/3047996601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563900a41270 0x563900a3baa0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82f6df1740 1 --2- 172.21.15.107:0/3047996601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563900a36ba0 0x563900a3b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82f6df1740 1 --2- 172.21.15.107:0/3047996601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563900a36280 0x563900a3b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82d57fa640 0 client.17649 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:39:57.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82f6df1740 1 -- 172.21.15.107:0/3047996601 >> 172.21.15.107:0/3047996601 conn(0x563900a325a0 msgr2=0x563900a30f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:57.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82f6df1740 1 -- 172.21.15.107:0/3047996601 shutdown_connections 2024-05-09T08:39:57.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82f6df1740 1 -- 172.21.15.107:0/3047996601 wait complete. 2024-05-09T08:39:57.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.852+0000 7f82f6df1740 20 client.17649 trim_cache size 0 max 16384 2024-05-09T08:39:57.860 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:39:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:39:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:39:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: "ff8f8b46-ad21-412f-9eff-94ddcf0491a3": { 2024-05-09T08:39:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:39:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "ff8f8b46-ad21-412f-9eff-94ddcf0491a3", 2024-05-09T08:39:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:39:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:39:57.860 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:39:57.860 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:39:57.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.853+0000 7f82f4009640 1 -- 172.21.15.107:0/1054435805 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f82d80a2d00 msgr2=0x7f82d80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.853+0000 7f82f4009640 1 --2- 172.21.15.107:0/1054435805 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f82d80a2d00 0x7f82d80a51c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f82e4012a20 tx=0x7f82e400ac70 comp rx=0 tx=0).stop 2024-05-09T08:39:57.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.853+0000 7f82f4009640 1 -- 172.21.15.107:0/1054435805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec1010c0 msgr2=0x7f82ec198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:39:57.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.853+0000 7f82f4009640 1 --2- 172.21.15.107:0/1054435805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec1010c0 0x7f82ec198520 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00d950 tx=0x7f82dc00de10 comp rx=0 tx=0).stop 2024-05-09T08:39:57.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.853+0000 7f82f4009640 1 -- 172.21.15.107:0/1054435805 shutdown_connections 2024-05-09T08:39:57.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.853+0000 7f82f4009640 1 --2- 172.21.15.107:0/1054435805 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f82d80a2d00 0x7f82d80a51c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.853+0000 7f82f4009640 1 --2- 172.21.15.107:0/1054435805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82ec106710 0x7f82ec19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.853+0000 7f82f4009640 1 --2- 172.21.15.107:0/1054435805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec1010c0 0x7f82ec198520 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.853+0000 7f82f4009640 1 --2- 172.21.15.107:0/1054435805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82ec100700 0x7f82ec197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:39:57.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.853+0000 7f82f4009640 1 -- 172.21.15.107:0/1054435805 >> 172.21.15.107:0/1054435805 conn(0x7f82ec077830 msgr2=0x7f82ec1023a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:39:57.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.853+0000 7f82f4009640 1 -- 172.21.15.107:0/1054435805 shutdown_connections 2024-05-09T08:39:57.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:39:57.853+0000 7f82f4009640 1 -- 172.21.15.107:0/1054435805 wait complete. 2024-05-09T08:39:57.865 INFO:tasks.cephfs.filesystem:scrub status for tag:ff8f8b46-ad21-412f-9eff-94ddcf0491a3 - {'path': '/', 'tag': 'ff8f8b46-ad21-412f-9eff-94ddcf0491a3', 'options': 'recursive,force'} 2024-05-09T08:39:58.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:58 smithi107 ceph-mon[30122]: pgmap v201: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:58.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:58 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:39:58.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:58 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:58.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:58 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:39:58.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:58 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:39:58.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:58 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:58.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:58 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:39:58.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:58 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:58.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:58 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:58.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:58 smithi157 ceph-mon[33614]: pgmap v201: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:58.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:58 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:39:58.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:58 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:58.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:58 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:39:58.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:58 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:39:58.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:58 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:58.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:58 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:39:58.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:58 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:58.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:58 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:58.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:58 smithi110 ceph-mon[33622]: pgmap v201: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:39:58.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:58 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:39:58.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:58 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:58.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:58 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:39:58.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:58 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:39:58.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:58 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:58.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:58 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:39:58.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:58 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:58.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:58 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:59.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:59 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:39:59.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:59 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:59.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:59 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:39:59.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:59 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:59.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:59 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:39:59.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:39:59 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:59.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:59 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:39:59.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:59 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:59.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:59 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:39:59.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:59 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:59.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:59 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:39:59.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:39:59 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:59.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:59 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:39:59.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:59 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:59.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:59 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:39:59.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:59 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:39:59.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:59 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:39:59.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:39:59 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:00.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:00 smithi107 ceph-mon[30122]: pgmap v202: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:40:00.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:00 smithi107 ceph-mon[30122]: overall HEALTH_OK 2024-05-09T08:40:00.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:00 smithi157 ceph-mon[33614]: pgmap v202: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:40:00.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:00 smithi157 ceph-mon[33614]: overall HEALTH_OK 2024-05-09T08:40:00.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:00 smithi110 ceph-mon[33622]: pgmap v202: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:40:00.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:00 smithi110 ceph-mon[33622]: overall HEALTH_OK 2024-05-09T08:40:02.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:02 smithi107 ceph-mon[30122]: pgmap v203: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:40:02.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:02 smithi107 ceph-mon[30122]: osdmap e95: 12 total, 12 up, 12 in 2024-05-09T08:40:02.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:02 smithi157 ceph-mon[33614]: pgmap v203: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:40:02.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:02 smithi157 ceph-mon[33614]: osdmap e95: 12 total, 12 up, 12 in 2024-05-09T08:40:02.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:02 smithi110 ceph-mon[33622]: pgmap v203: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:40:02.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:02 smithi110 ceph-mon[33622]: osdmap e95: 12 total, 12 up, 12 in 2024-05-09T08:40:04.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:04 smithi107 ceph-mon[30122]: pgmap v205: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 29 MiB/s wr, 81 op/s 2024-05-09T08:40:04.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:04 smithi157 ceph-mon[33614]: pgmap v205: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 29 MiB/s wr, 81 op/s 2024-05-09T08:40:04.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:04 smithi110 ceph-mon[33622]: pgmap v205: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 29 MiB/s wr, 81 op/s 2024-05-09T08:40:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.021+0000 7fbd95ab3640 1 -- 172.21.15.107:0/1856672096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd901056d0 msgr2=0x7fbd90105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.021+0000 7fbd95ab3640 1 --2- 172.21.15.107:0/1856672096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd901056d0 0x7fbd90105ab0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fbd80009900 tx=0x7fbd8002a410 comp rx=0 tx=0).stop 2024-05-09T08:40:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.021+0000 7fbd95ab3640 1 -- 172.21.15.107:0/1856672096 shutdown_connections 2024-05-09T08:40:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.021+0000 7fbd95ab3640 1 --2- 172.21.15.107:0/1856672096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd90101440 0x7fbd9010d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.021+0000 7fbd95ab3640 1 --2- 172.21.15.107:0/1856672096 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd90100a80 0x7fbd90100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.021+0000 7fbd95ab3640 1 --2- 172.21.15.107:0/1856672096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd901056d0 0x7fbd90105ab0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.021+0000 7fbd95ab3640 1 -- 172.21.15.107:0/1856672096 >> 172.21.15.107:0/1856672096 conn(0x7fbd90076530 msgr2=0x7fbd90076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.021+0000 7fbd95ab3640 1 -- 172.21.15.107:0/1856672096 shutdown_connections 2024-05-09T08:40:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.021+0000 7fbd95ab3640 1 -- 172.21.15.107:0/1856672096 wait complete. 2024-05-09T08:40:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.021+0000 7fbd95ab3640 1 Processor -- start 2024-05-09T08:40:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.021+0000 7fbd95ab3640 1 -- start start 2024-05-09T08:40:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.021+0000 7fbd95ab3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd90100a80 0x7fbd90198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd95ab3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd90101440 0x7fbd90199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd95ab3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd901056d0 0x7fbd901a3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd95ab3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901109b0 con 0x7fbd901056d0 2024-05-09T08:40:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd95ab3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbd90110830 con 0x7fbd90100a80 2024-05-09T08:40:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd95ab3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbd90110b30 con 0x7fbd90101440 2024-05-09T08:40:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd8ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd901056d0 0x7fbd901a3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd8ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd901056d0 0x7fbd901a3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59774/0 (socket says 172.21.15.107:59774) 2024-05-09T08:40:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd8ffff640 1 -- 172.21.15.107:0/3859880708 learned_addr learned my addr 172.21.15.107:0/3859880708 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd87fff640 1 --2- 172.21.15.107:0/3859880708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd90101440 0x7fbd90199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd8f7fe640 1 --2- 172.21.15.107:0/3859880708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd90100a80 0x7fbd90198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd8ffff640 1 -- 172.21.15.107:0/3859880708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd90101440 msgr2=0x7fbd90199230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd8ffff640 1 --2- 172.21.15.107:0/3859880708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd90101440 0x7fbd90199230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd8ffff640 1 -- 172.21.15.107:0/3859880708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd90100a80 msgr2=0x7fbd90198cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd8ffff640 1 --2- 172.21.15.107:0/3859880708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd90100a80 0x7fbd90198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd8ffff640 1 -- 172.21.15.107:0/3859880708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd901a42b0 con 0x7fbd901056d0 2024-05-09T08:40:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd8ffff640 1 --2- 172.21.15.107:0/3859880708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd901056d0 0x7fbd901a3d70 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009fd0 tx=0x7fbd8800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd8d7fa640 1 -- 172.21.15.107:0/3859880708 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd88017070 con 0x7fbd901056d0 2024-05-09T08:40:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd8d7fa640 1 -- 172.21.15.107:0/3859880708 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd88003e20 con 0x7fbd901056d0 2024-05-09T08:40:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd95ab3640 1 -- 172.21.15.107:0/3859880708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd901a44e0 con 0x7fbd901056d0 2024-05-09T08:40:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd8d7fa640 1 -- 172.21.15.107:0/3859880708 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd88010460 con 0x7fbd901056d0 2024-05-09T08:40:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.022+0000 7fbd95ab3640 1 -- 172.21.15.107:0/3859880708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd901a49b0 con 0x7fbd901056d0 2024-05-09T08:40:05.033 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.025+0000 7fbd8d7fa640 1 -- 172.21.15.107:0/3859880708 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fbd88003970 con 0x7fbd901056d0 2024-05-09T08:40:05.033 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.025+0000 7fbd987c4740 1 Processor -- start 2024-05-09T08:40:05.033 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.025+0000 7fbd987c4740 1 -- start start 2024-05-09T08:40:05.033 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.025+0000 7fbd987c4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93b7450 0x560da93b7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.033 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.025+0000 7fbd987c4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93b7d70 0x560da93c20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.025+0000 7fbd987c4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93c2630 0x560da93c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.025+0000 7fbd987c4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560da931c340 con 0x560da93b7d70 2024-05-09T08:40:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.025+0000 7fbd987c4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560da92c4290 con 0x560da93c2630 2024-05-09T08:40:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.025+0000 7fbd987c4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560da9318300 con 0x560da93b7450 2024-05-09T08:40:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd8f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93b7d70 0x560da93c20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd8f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93b7d70 0x560da93c20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59776/0 (socket says 172.21.15.107:59776) 2024-05-09T08:40:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd8d7fa640 1 --2- 172.21.15.107:0/3859880708 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbd7c0a2da0 0x7fbd7c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd8f7fe640 1 -- 172.21.15.107:0/3484782510 learned_addr learned my addr 172.21.15.107:0/3484782510 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd87fff640 1 --2- 172.21.15.107:0/3484782510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93b7450 0x560da93b7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd8ffff640 1 --2- 172.21.15.107:0/3859880708 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbd7c0a2da0 0x7fbd7c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd87fff640 1 --2- 172.21.15.107:0/3484782510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93c2630 0x560da93c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd8f7fe640 1 -- 172.21.15.107:0/3484782510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93b7450 msgr2=0x560da93b7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd8f7fe640 1 --2- 172.21.15.107:0/3484782510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93b7450 0x560da93b7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd8f7fe640 1 -- 172.21.15.107:0/3484782510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93c2630 msgr2=0x560da93c4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd8f7fe640 1 --2- 172.21.15.107:0/3484782510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93c2630 0x560da93c4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd8d7fa640 1 -- 172.21.15.107:0/3859880708 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7fbd880cf710 con 0x7fbd901056d0 2024-05-09T08:40:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd8f7fe640 1 -- 172.21.15.107:0/3484782510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560da92e88e0 con 0x560da93b7d70 2024-05-09T08:40:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd87fff640 1 --2- 172.21.15.107:0/3484782510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93c2630 0x560da93c4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:40:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd8f7fe640 1 --2- 172.21.15.107:0/3484782510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93b7d70 0x560da93c20f0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fbd80006010 tx=0x7fbd80032c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac15728837d17104 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd8ffff640 1 --2- 172.21.15.107:0/3859880708 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbd7c0a2da0 0x7fbd7c0a5260 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbd880ccb00 tx=0x7fbd88086040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd85ffb640 1 -- 172.21.15.107:0/3484782510 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8003f070 con 0x560da93b7d70 2024-05-09T08:40:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd85ffb640 1 -- 172.21.15.107:0/3484782510 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd80032e20 con 0x560da93b7d70 2024-05-09T08:40:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.026+0000 7fbd85ffb640 1 -- 172.21.15.107:0/3484782510 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80035070 con 0x560da93b7d70 2024-05-09T08:40:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.027+0000 7fbd987c4740 1 -- 172.21.15.107:0/3484782510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93b7d70 msgr2=0x560da93c20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.027+0000 7fbd987c4740 1 --2- 172.21.15.107:0/3484782510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93b7d70 0x560da93c20f0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fbd80006010 tx=0x7fbd80032c30 comp rx=0 tx=0).stop 2024-05-09T08:40:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.027+0000 7fbd987c4740 1 -- 172.21.15.107:0/3484782510 shutdown_connections 2024-05-09T08:40:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.027+0000 7fbd987c4740 1 --2- 172.21.15.107:0/3484782510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93c2630 0x560da93c4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.027+0000 7fbd987c4740 1 --2- 172.21.15.107:0/3484782510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93b7d70 0x560da93c20f0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.027+0000 7fbd987c4740 1 --2- 172.21.15.107:0/3484782510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93b7450 0x560da93b7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.027+0000 7fbd987c4740 1 -- 172.21.15.107:0/3484782510 >> 172.21.15.107:0/3484782510 conn(0x560da93b3960 msgr2=0x560da93b3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.027+0000 7fbd987c4740 1 -- 172.21.15.107:0/3484782510 shutdown_connections 2024-05-09T08:40:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.027+0000 7fbd987c4740 1 -- 172.21.15.107:0/3484782510 wait complete. 2024-05-09T08:40:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.027+0000 7fbd987c4740 1 -- start start 2024-05-09T08:40:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd987c4740 1 Processor -- start 2024-05-09T08:40:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd987c4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93b7450 0x560da93bf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd987c4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93b7d70 0x560da93bfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd987c4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93c2630 0x560da93c0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd987c4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560da93c5200 con 0x560da93c2630 2024-05-09T08:40:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd987c4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560da92d7790 con 0x560da93b7450 2024-05-09T08:40:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd987c4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560da93c5380 con 0x560da93b7d70 2024-05-09T08:40:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd987c4740 10 client.0 _ll_register_callbacks cb 0x560da9312a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd87fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93c2630 0x560da93c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd87fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93c2630 0x560da93c0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59792/0 (socket says 172.21.15.107:59792) 2024-05-09T08:40:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd87fff640 1 -- 172.21.15.107:0/3635749006 learned_addr learned my addr 172.21.15.107:0/3635749006 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd87fff640 1 --2- 172.21.15.107:0/3635749006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93b7450 0x560da93bf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd8f7fe640 1 --2- 172.21.15.107:0/3635749006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93b7d70 0x560da93bfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd87fff640 1 -- 172.21.15.107:0/3635749006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93b7d70 msgr2=0x560da93bfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd87fff640 1 --2- 172.21.15.107:0/3635749006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93b7d70 0x560da93bfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd87fff640 1 -- 172.21.15.107:0/3635749006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93b7450 msgr2=0x560da93bf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:05.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd87fff640 1 --2- 172.21.15.107:0/3635749006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93b7450 0x560da93bf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.028+0000 7fbd87fff640 1 -- 172.21.15.107:0/3635749006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560da9030210 con 0x560da93c2630 2024-05-09T08:40:05.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.029+0000 7fbd87fff640 1 --2- 172.21.15.107:0/3635749006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93c2630 0x560da93c0030 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fbd78005cc0 tx=0x7fbd78005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c02eda2f8746eaf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:05.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.029+0000 7fbd6effd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:05.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.029+0000 7fbd6effd640 1 -- 172.21.15.107:0/3635749006 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7800fe00 con 0x560da93c2630 2024-05-09T08:40:05.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.029+0000 7fbd6effd640 1 -- 172.21.15.107:0/3635749006 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7fbd78011440 con 0x560da93c2630 2024-05-09T08:40:05.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.029+0000 7fbd987c4740 1 -- 172.21.15.107:0/3635749006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd80032e20 con 0x560da93c2630 2024-05-09T08:40:05.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.029+0000 7fbd6effd640 1 -- 172.21.15.107:0/3635749006 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbd78028050 con 0x560da93c2630 2024-05-09T08:40:05.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.029+0000 7fbd987c4740 10 client.17661 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:05.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.029+0000 7fbd987c4740 1 -- 172.21.15.107:0/3635749006 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560da90fc620 con 0x560da93c2630 2024-05-09T08:40:05.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.029+0000 7fbd6effd640 1 -- 172.21.15.107:0/3635749006 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd80032e20 con 0x560da93c2630 2024-05-09T08:40:05.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.029+0000 7fbd6effd640 1 -- 172.21.15.107:0/3635749006 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fbd78020510 con 0x560da93c2630 2024-05-09T08:40:05.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.029+0000 7fbd987c4740 10 client.17661 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:05.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.029+0000 7fbd987c4740 10 client.17661 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:40:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.029+0000 7fbd987c4740 20 client.17661 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.029+0000 7fbd987c4740 1 --2- 172.21.15.107:0/3635749006 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560da93d2280 0x560da93d4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.029+0000 7fbd987c4740 4 client.17661 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:40:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.029+0000 7fbd987c4740 1 -- 172.21.15.107:0/3635749006 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560da9329a70 con 0x560da93d2280 2024-05-09T08:40:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.030+0000 7fbd8f7fe640 1 --2- 172.21.15.107:0/3635749006 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560da93d2280 0x560da93d4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.030+0000 7fbd6effd640 10 client.17661 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.030+0000 7fbd8f7fe640 1 --2- 172.21.15.107:0/3635749006 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560da93d2280 0x560da93d4670 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30407adcf1834fad server_cookie=1899b15fc231c0a3 in_seq=0 out_seq=0 2024-05-09T08:40:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.031+0000 7fbd6effd640 1 -- 172.21.15.107:0/3635749006 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560da9329a70 con 0x560da93d2280 2024-05-09T08:40:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.031+0000 7fbd6effd640 10 client.17661 handle_command_reply: tid=0 2024-05-09T08:40:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.031+0000 7fbd987c4740 1 client.17661 shutdown 2024-05-09T08:40:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.031+0000 7fbd987c4740 1 -- 172.21.15.107:0/3635749006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93c2630 msgr2=0x560da93c0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.031+0000 7fbd987c4740 1 --2- 172.21.15.107:0/3635749006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93c2630 0x560da93c0030 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fbd78005cc0 tx=0x7fbd78005c50 comp rx=0 tx=0).stop 2024-05-09T08:40:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.031+0000 7fbd987c4740 1 -- 172.21.15.107:0/3635749006 shutdown_connections 2024-05-09T08:40:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.031+0000 7fbd987c4740 1 --2- 172.21.15.107:0/3635749006 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560da93d2280 0x560da93d4670 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.031+0000 7fbd987c4740 1 --2- 172.21.15.107:0/3635749006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93c2630 0x560da93c0030 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.031+0000 7fbd987c4740 1 --2- 172.21.15.107:0/3635749006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93b7d70 0x560da93bfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.031+0000 7fbd6effd640 0 client.17661 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.031+0000 7fbd987c4740 1 --2- 172.21.15.107:0/3635749006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93b7450 0x560da93bf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.031+0000 7fbd987c4740 1 -- 172.21.15.107:0/3635749006 >> 172.21.15.107:0/3635749006 conn(0x560da93b3960 msgr2=0x560da93b9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.031+0000 7fbd987c4740 1 -- 172.21.15.107:0/3635749006 shutdown_connections 2024-05-09T08:40:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.031+0000 7fbd987c4740 1 -- 172.21.15.107:0/3635749006 wait complete. 2024-05-09T08:40:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.031+0000 7fbd987c4740 20 client.17661 trim_cache size 0 max 16384 2024-05-09T08:40:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.046+0000 7fbd987c4740 1 Processor -- start 2024-05-09T08:40:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.046+0000 7fbd987c4740 1 -- start start 2024-05-09T08:40:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd987c4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93b7450 0x560da93ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd987c4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93b7d70 0x560da93be4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd987c4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93c2630 0x560da93b9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd987c4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901b1540 con 0x560da93c2630 2024-05-09T08:40:05.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd987c4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901aee10 con 0x560da93b7450 2024-05-09T08:40:05.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd987c4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901acc80 con 0x560da93b7d70 2024-05-09T08:40:05.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd87fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93c2630 0x560da93b9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd87fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93c2630 0x560da93b9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59794/0 (socket says 172.21.15.107:59794) 2024-05-09T08:40:05.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd87fff640 1 -- 172.21.15.107:0/164240644 learned_addr learned my addr 172.21.15.107:0/164240644 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:05.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd87fff640 1 --2- 172.21.15.107:0/164240644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93b7450 0x560da93ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd8f7fe640 1 --2- 172.21.15.107:0/164240644 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93b7d70 0x560da93be4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd87fff640 1 -- 172.21.15.107:0/164240644 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93b7d70 msgr2=0x560da93be4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:05.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd87fff640 1 --2- 172.21.15.107:0/164240644 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93b7d70 0x560da93be4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd87fff640 1 -- 172.21.15.107:0/164240644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93b7450 msgr2=0x560da93ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:05.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd87fff640 1 --2- 172.21.15.107:0/164240644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93b7450 0x560da93ce0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd87fff640 1 -- 172.21.15.107:0/164240644 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560da93d49e0 con 0x560da93c2630 2024-05-09T08:40:05.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd8f7fe640 1 --2- 172.21.15.107:0/164240644 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93b7d70 0x560da93be4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:40:05.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd87fff640 1 --2- 172.21.15.107:0/164240644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93c2630 0x560da93b9e10 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800ce30 tx=0x7fbd78012b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b75918bd97776c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:05.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd857fa640 1 -- 172.21.15.107:0/164240644 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd78012c30 con 0x560da93c2630 2024-05-09T08:40:05.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd857fa640 1 -- 172.21.15.107:0/164240644 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd7800fe00 con 0x560da93c2630 2024-05-09T08:40:05.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.047+0000 7fbd857fa640 1 -- 172.21.15.107:0/164240644 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd78005290 con 0x560da93c2630 2024-05-09T08:40:05.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.048+0000 7fbd987c4740 1 -- 172.21.15.107:0/164240644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93c2630 msgr2=0x560da93b9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:05.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.048+0000 7fbd987c4740 1 --2- 172.21.15.107:0/164240644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93c2630 0x560da93b9e10 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800ce30 tx=0x7fbd78012b10 comp rx=0 tx=0).stop 2024-05-09T08:40:05.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.048+0000 7fbd987c4740 1 -- 172.21.15.107:0/164240644 shutdown_connections 2024-05-09T08:40:05.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.048+0000 7fbd987c4740 1 --2- 172.21.15.107:0/164240644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93c2630 0x560da93b9e10 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.048+0000 7fbd987c4740 1 --2- 172.21.15.107:0/164240644 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93b7d70 0x560da93be4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.048+0000 7fbd987c4740 1 --2- 172.21.15.107:0/164240644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93b7450 0x560da93ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.048+0000 7fbd987c4740 1 -- 172.21.15.107:0/164240644 >> 172.21.15.107:0/164240644 conn(0x560da93b3960 msgr2=0x560da93b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:05.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.048+0000 7fbd987c4740 1 -- 172.21.15.107:0/164240644 shutdown_connections 2024-05-09T08:40:05.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.048+0000 7fbd987c4740 1 -- 172.21.15.107:0/164240644 wait complete. 2024-05-09T08:40:05.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.048+0000 7fbd987c4740 1 -- start start 2024-05-09T08:40:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd987c4740 1 Processor -- start 2024-05-09T08:40:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd987c4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93b7450 0x560da93bc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd987c4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93b7d70 0x560da93bc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd987c4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93c2630 0x560da93bcc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd987c4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901ad2d0 con 0x560da93b7d70 2024-05-09T08:40:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd987c4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901ae740 con 0x560da93b7450 2024-05-09T08:40:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd987c4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901ad9a0 con 0x560da93c2630 2024-05-09T08:40:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd987c4740 10 client.0 _ll_register_callbacks cb 0x7fbd7800cc40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd8f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93b7d70 0x560da93bc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd8f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93b7d70 0x560da93bc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59802/0 (socket says 172.21.15.107:59802) 2024-05-09T08:40:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd8f7fe640 1 -- 172.21.15.107:0/1274217763 learned_addr learned my addr 172.21.15.107:0/1274217763 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd87fff640 1 --2- 172.21.15.107:0/1274217763 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93c2630 0x560da93bcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd87fff640 1 --2- 172.21.15.107:0/1274217763 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93b7450 0x560da93bc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd8f7fe640 1 -- 172.21.15.107:0/1274217763 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93c2630 msgr2=0x560da93bcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd8f7fe640 1 --2- 172.21.15.107:0/1274217763 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93c2630 0x560da93bcc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd8f7fe640 1 -- 172.21.15.107:0/1274217763 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93b7450 msgr2=0x560da93bc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd8f7fe640 1 --2- 172.21.15.107:0/1274217763 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93b7450 0x560da93bc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd8f7fe640 1 -- 172.21.15.107:0/1274217763 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560da93b7ba0 con 0x560da93b7d70 2024-05-09T08:40:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.049+0000 7fbd87fff640 1 --2- 172.21.15.107:0/1274217763 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93b7450 0x560da93bc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:40:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.050+0000 7fbd8f7fe640 1 --2- 172.21.15.107:0/1274217763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93b7d70 0x560da93bc730 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fbd80000c00 tx=0x7fbd80009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec5b9c8f33cf3b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.050+0000 7fbd6effd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.050+0000 7fbd6effd640 1 -- 172.21.15.107:0/1274217763 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80050030 con 0x560da93b7d70 2024-05-09T08:40:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.050+0000 7fbd6effd640 1 -- 172.21.15.107:0/1274217763 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7fbd8004cdd0 con 0x560da93b7d70 2024-05-09T08:40:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.050+0000 7fbd987c4740 1 -- 172.21.15.107:0/1274217763 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd7800fe00 con 0x560da93b7d70 2024-05-09T08:40:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.050+0000 7fbd6effd640 1 -- 172.21.15.107:0/1274217763 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbd8005d050 con 0x560da93b7d70 2024-05-09T08:40:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.050+0000 7fbd987c4740 10 client.17667 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.050+0000 7fbd987c4740 1 -- 172.21.15.107:0/1274217763 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560da9438830 con 0x560da93b7d70 2024-05-09T08:40:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.050+0000 7fbd6effd640 1 -- 172.21.15.107:0/1274217763 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd7800fe00 con 0x560da93b7d70 2024-05-09T08:40:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.050+0000 7fbd6effd640 1 -- 172.21.15.107:0/1274217763 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fbd8004c740 con 0x560da93b7d70 2024-05-09T08:40:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.050+0000 7fbd987c4740 10 client.17667 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.050+0000 7fbd987c4740 10 client.17667 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:40:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.050+0000 7fbd987c4740 20 client.17667 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.050+0000 7fbd987c4740 1 --2- 172.21.15.107:0/1274217763 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560da9438be0 0x560da943afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.050+0000 7fbd987c4740 4 client.17667 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:40:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.050+0000 7fbd987c4740 1 -- 172.21.15.107:0/1274217763 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560da9323dc0 con 0x560da9438be0 2024-05-09T08:40:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.051+0000 7fbd87fff640 1 --2- 172.21.15.107:0/1274217763 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560da9438be0 0x560da943afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.051+0000 7fbd6effd640 10 client.17667 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:05.051+0000 7fbd87fff640 1 --2- 172.21.15.107:0/1274217763 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560da9438be0 0x560da943afd0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd2b9c0a7f7749c4 server_cookie=2c0642c76dfc14a0 in_seq=0 out_seq=0 2024-05-09T08:40:06.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:06 smithi107 ceph-mon[30122]: pgmap v206: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 29 MiB/s wr, 81 op/s 2024-05-09T08:40:06.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:06 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:40:06.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:06 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:40:06.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:06 smithi157 ceph-mon[33614]: pgmap v206: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 29 MiB/s wr, 81 op/s 2024-05-09T08:40:06.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:06 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:40:06.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:06 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:40:06.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:06 smithi110 ceph-mon[33622]: pgmap v206: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 29 MiB/s wr, 81 op/s 2024-05-09T08:40:06.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:06 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:40:06.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:06 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:40:07.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:07 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:40:07.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:07 smithi107 ceph-mon[30122]: osdmap e96: 12 total, 12 up, 12 in 2024-05-09T08:40:07.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:07 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:40:07.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:07 smithi157 ceph-mon[33614]: osdmap e96: 12 total, 12 up, 12 in 2024-05-09T08:40:07.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:07 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:40:07.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:07 smithi110 ceph-mon[33622]: osdmap e96: 12 total, 12 up, 12 in 2024-05-09T08:40:08.745 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:08 smithi157 ceph-mon[33614]: pgmap v208: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 21 MiB/s wr, 93 op/s 2024-05-09T08:40:08.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:08 smithi107 ceph-mon[30122]: pgmap v208: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 21 MiB/s wr, 93 op/s 2024-05-09T08:40:08.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:08 smithi110 ceph-mon[33622]: pgmap v208: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 21 MiB/s wr, 93 op/s 2024-05-09T08:40:10.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:10 smithi107 ceph-mon[30122]: pgmap v209: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 21 MiB/s wr, 93 op/s 2024-05-09T08:40:10.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:10 smithi157 ceph-mon[33614]: pgmap v209: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 21 MiB/s wr, 93 op/s 2024-05-09T08:40:10.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:10 smithi110 ceph-mon[33622]: pgmap v209: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 21 MiB/s wr, 93 op/s 2024-05-09T08:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.955+0000 7fbd6effd640 1 -- 172.21.15.107:0/1274217763 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560da9323dc0 con 0x560da9438be0 2024-05-09T08:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.955+0000 7fbd6effd640 10 client.17667 handle_command_reply: tid=0 2024-05-09T08:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.955+0000 7fbd987c4740 1 client.17667 shutdown 2024-05-09T08:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.956+0000 7fbd987c4740 1 -- 172.21.15.107:0/1274217763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93b7d70 msgr2=0x560da93bc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.956+0000 7fbd987c4740 1 --2- 172.21.15.107:0/1274217763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93b7d70 0x560da93bc730 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fbd80000c00 tx=0x7fbd80009290 comp rx=0 tx=0).stop 2024-05-09T08:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.956+0000 7fbd987c4740 1 -- 172.21.15.107:0/1274217763 shutdown_connections 2024-05-09T08:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.956+0000 7fbd987c4740 1 --2- 172.21.15.107:0/1274217763 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560da9438be0 0x560da943afd0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.956+0000 7fbd987c4740 1 --2- 172.21.15.107:0/1274217763 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560da93c2630 0x560da93bcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.956+0000 7fbd987c4740 1 --2- 172.21.15.107:0/1274217763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560da93b7d70 0x560da93bc730 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.956+0000 7fbd987c4740 1 --2- 172.21.15.107:0/1274217763 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560da93b7450 0x560da93bc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.956+0000 7fbd6effd640 0 client.17667 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.956+0000 7fbd987c4740 1 -- 172.21.15.107:0/1274217763 >> 172.21.15.107:0/1274217763 conn(0x560da93b3960 msgr2=0x560da93bf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.956+0000 7fbd987c4740 1 -- 172.21.15.107:0/1274217763 shutdown_connections 2024-05-09T08:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.956+0000 7fbd987c4740 1 -- 172.21.15.107:0/1274217763 wait complete. 2024-05-09T08:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.956+0000 7fbd987c4740 20 client.17667 trim_cache size 0 max 16384 2024-05-09T08:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.957+0000 7fbd95ab3640 1 -- 172.21.15.107:0/3859880708 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbd7c0a2da0 msgr2=0x7fbd7c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.957+0000 7fbd95ab3640 1 --2- 172.21.15.107:0/3859880708 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbd7c0a2da0 0x7fbd7c0a5260 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbd880ccb00 tx=0x7fbd88086040 comp rx=0 tx=0).stop 2024-05-09T08:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.957+0000 7fbd95ab3640 1 -- 172.21.15.107:0/3859880708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd901056d0 msgr2=0x7fbd901a3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.957+0000 7fbd95ab3640 1 --2- 172.21.15.107:0/3859880708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd901056d0 0x7fbd901a3d70 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009fd0 tx=0x7fbd8800b260 comp rx=0 tx=0).stop 2024-05-09T08:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.957+0000 7fbd95ab3640 1 -- 172.21.15.107:0/3859880708 shutdown_connections 2024-05-09T08:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.957+0000 7fbd95ab3640 1 --2- 172.21.15.107:0/3859880708 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbd7c0a2da0 0x7fbd7c0a5260 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.957+0000 7fbd95ab3640 1 --2- 172.21.15.107:0/3859880708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd901056d0 0x7fbd901a3d70 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.957+0000 7fbd95ab3640 1 --2- 172.21.15.107:0/3859880708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd90101440 0x7fbd90199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.957+0000 7fbd95ab3640 1 --2- 172.21.15.107:0/3859880708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd90100a80 0x7fbd90198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.957+0000 7fbd95ab3640 1 -- 172.21.15.107:0/3859880708 >> 172.21.15.107:0/3859880708 conn(0x7fbd90076530 msgr2=0x7fbd900fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.957+0000 7fbd95ab3640 1 -- 172.21.15.107:0/3859880708 shutdown_connections 2024-05-09T08:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:10.957+0000 7fbd95ab3640 1 -- 172.21.15.107:0/3859880708 wait complete. 2024-05-09T08:40:12.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:12 smithi107 ceph-mon[30122]: pgmap v210: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 18 MiB/s wr, 78 op/s 2024-05-09T08:40:12.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:12 smithi157 ceph-mon[33614]: pgmap v210: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 18 MiB/s wr, 78 op/s 2024-05-09T08:40:12.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:12 smithi110 ceph-mon[33622]: pgmap v210: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 18 MiB/s wr, 78 op/s 2024-05-09T08:40:14.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:14 smithi107 ceph-mon[30122]: pgmap v211: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 4.0 MiB/s wr, 15 op/s 2024-05-09T08:40:14.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:14 smithi157 ceph-mon[33614]: pgmap v211: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 4.0 MiB/s wr, 15 op/s 2024-05-09T08:40:14.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:14 smithi110 ceph-mon[33622]: pgmap v211: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 4.0 MiB/s wr, 15 op/s 2024-05-09T08:40:16.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:16 smithi107 ceph-mon[30122]: pgmap v212: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 4.0 MiB/s wr, 15 op/s 2024-05-09T08:40:16.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:16 smithi157 ceph-mon[33614]: pgmap v212: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 4.0 MiB/s wr, 15 op/s 2024-05-09T08:40:16.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:16 smithi110 ceph-mon[33622]: pgmap v212: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 4.0 MiB/s wr, 15 op/s 2024-05-09T08:40:18.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:18 smithi107 ceph-mon[30122]: pgmap v213: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 17 MiB/s wr, 16 op/s 2024-05-09T08:40:18.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:18 smithi157 ceph-mon[33614]: pgmap v213: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 17 MiB/s wr, 16 op/s 2024-05-09T08:40:18.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:18 smithi110 ceph-mon[33622]: pgmap v213: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 17 MiB/s wr, 16 op/s 2024-05-09T08:40:20.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:20 smithi107 ceph-mon[30122]: pgmap v214: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 14 MiB/s wr, 13 op/s 2024-05-09T08:40:20.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:20 smithi157 ceph-mon[33614]: pgmap v214: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 14 MiB/s wr, 13 op/s 2024-05-09T08:40:20.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:20 smithi110 ceph-mon[33622]: pgmap v214: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 14 MiB/s wr, 13 op/s 2024-05-09T08:40:22.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:22 smithi107 ceph-mon[30122]: pgmap v215: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 14 MiB/s wr, 13 op/s 2024-05-09T08:40:22.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:22 smithi157 ceph-mon[33614]: pgmap v215: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 14 MiB/s wr, 13 op/s 2024-05-09T08:40:22.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:22 smithi110 ceph-mon[33622]: pgmap v215: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 14 MiB/s wr, 13 op/s 2024-05-09T08:40:24.061 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.059+0000 7fd5048a5640 1 -- 172.21.15.107:0/4177708538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd500100760 msgr2=0x7fd500100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:24.062 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.059+0000 7fd5048a5640 1 --2- 172.21.15.107:0/4177708538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd500100760 0x7fd500100be0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4009930 tx=0x7fd4f402a420 comp rx=0 tx=0).stop 2024-05-09T08:40:24.062 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.059+0000 7fd5048a5640 1 -- 172.21.15.107:0/4177708538 shutdown_connections 2024-05-09T08:40:24.062 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.059+0000 7fd5048a5640 1 --2- 172.21.15.107:0/4177708538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd500101120 0x7fd50010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.062 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.059+0000 7fd5048a5640 1 --2- 172.21.15.107:0/4177708538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd500100760 0x7fd500100be0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.062 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.059+0000 7fd5048a5640 1 --2- 172.21.15.107:0/4177708538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd500106720 0x7fd500106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.062 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.059+0000 7fd5048a5640 1 -- 172.21.15.107:0/4177708538 >> 172.21.15.107:0/4177708538 conn(0x7fd5000778b0 msgr2=0x7fd500077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:24.062 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.059+0000 7fd5048a5640 1 -- 172.21.15.107:0/4177708538 shutdown_connections 2024-05-09T08:40:24.062 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.059+0000 7fd5048a5640 1 -- 172.21.15.107:0/4177708538 wait complete. 2024-05-09T08:40:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.060+0000 7fd5048a5640 1 Processor -- start 2024-05-09T08:40:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.060+0000 7fd5048a5640 1 -- start start 2024-05-09T08:40:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.060+0000 7fd5048a5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd500100760 0x7fd500196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.060+0000 7fd5048a5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd500101120 0x7fd500197450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.060+0000 7fd5048a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd500106720 0x7fd50019b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.060+0000 7fd5048a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd500110a30 con 0x7fd500106720 2024-05-09T08:40:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.060+0000 7fd5048a5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd5001108b0 con 0x7fd500101120 2024-05-09T08:40:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.060+0000 7fd5048a5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd500110bb0 con 0x7fd500100760 2024-05-09T08:40:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.061+0000 7fd4fed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd500106720 0x7fd50019b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.061+0000 7fd4fed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd500106720 0x7fd50019b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43726/0 (socket says 172.21.15.107:43726) 2024-05-09T08:40:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.061+0000 7fd4fed9d640 1 -- 172.21.15.107:0/3528428954 learned_addr learned my addr 172.21.15.107:0/3528428954 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.061+0000 7fd4fe59c640 1 --2- 172.21.15.107:0/3528428954 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd500100760 0x7fd500196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.061+0000 7fd4fdd9b640 1 --2- 172.21.15.107:0/3528428954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd500101120 0x7fd500197450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.061+0000 7fd4fed9d640 1 -- 172.21.15.107:0/3528428954 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd500100760 msgr2=0x7fd500196f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.061+0000 7fd4fed9d640 1 --2- 172.21.15.107:0/3528428954 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd500100760 0x7fd500196f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.061+0000 7fd4fed9d640 1 -- 172.21.15.107:0/3528428954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd500101120 msgr2=0x7fd500197450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.061+0000 7fd4fed9d640 1 --2- 172.21.15.107:0/3528428954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd500101120 0x7fd500197450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.061+0000 7fd4fed9d640 1 -- 172.21.15.107:0/3528428954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd50019beb0 con 0x7fd500106720 2024-05-09T08:40:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.061+0000 7fd4fed9d640 1 --2- 172.21.15.107:0/3528428954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd500106720 0x7fd50019b780 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8007cb0 tx=0x7fd4f800e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.061+0000 7fd4ef7fe640 1 -- 172.21.15.107:0/3528428954 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f800eb30 con 0x7fd500106720 2024-05-09T08:40:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.061+0000 7fd5048a5640 1 -- 172.21.15.107:0/3528428954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd50019c1a0 con 0x7fd500106720 2024-05-09T08:40:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.061+0000 7fd4ef7fe640 1 -- 172.21.15.107:0/3528428954 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4f800ecd0 con 0x7fd500106720 2024-05-09T08:40:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.061+0000 7fd4ef7fe640 1 -- 172.21.15.107:0/3528428954 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f80153f0 con 0x7fd500106720 2024-05-09T08:40:24.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.061+0000 7fd5048a5640 1 -- 172.21.15.107:0/3528428954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd500068ee0 con 0x7fd500106720 2024-05-09T08:40:24.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.064+0000 7fd4ef7fe640 1 -- 172.21.15.107:0/3528428954 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd4f800a440 con 0x7fd500106720 2024-05-09T08:40:24.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.064+0000 7fd5075b9740 1 Processor -- start 2024-05-09T08:40:24.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd5075b9740 1 -- start start 2024-05-09T08:40:24.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd4ef7fe640 1 --2- 172.21.15.107:0/3528428954 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4e80a2da0 0x7fd4e80a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd5075b9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a74730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd5075b9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a74c70 0x55d245a83500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd5075b9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a83a40 0x55d245a85e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd5075b9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d2459d9340 con 0x55d245a74350 2024-05-09T08:40:24.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd5075b9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d245981290 con 0x55d245a83a40 2024-05-09T08:40:24.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd5075b9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d2459d5300 con 0x55d245a74c70 2024-05-09T08:40:24.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd4fdd9b640 1 --2- 172.21.15.107:0/3528428954 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4e80a2da0 0x7fd4e80a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd4ef7fe640 1 -- 172.21.15.107:0/3528428954 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fd4f80d14d0 con 0x7fd500106720 2024-05-09T08:40:24.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd4fe59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a74730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd4fe59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a74730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43736/0 (socket says 172.21.15.107:43736) 2024-05-09T08:40:24.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd4fe59c640 1 -- 172.21.15.107:0/1291920690 learned_addr learned my addr 172.21.15.107:0/1291920690 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:24.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd4fdd9b640 1 --2- 172.21.15.107:0/1291920690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a74c70 0x55d245a83500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd4fed9d640 1 --2- 172.21.15.107:0/1291920690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a83a40 0x55d245a85e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd4fe59c640 1 -- 172.21.15.107:0/1291920690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a74c70 msgr2=0x55d245a83500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:24.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd4fe59c640 1 --2- 172.21.15.107:0/1291920690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a74c70 0x55d245a83500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd4fe59c640 1 -- 172.21.15.107:0/1291920690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a83a40 msgr2=0x55d245a85e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:24.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd4fe59c640 1 --2- 172.21.15.107:0/1291920690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a83a40 0x55d245a85e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.065+0000 7fd4fe59c640 1 -- 172.21.15.107:0/1291920690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2459a58e0 con 0x55d245a74350 2024-05-09T08:40:24.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.066+0000 7fd4fdd9b640 1 --2- 172.21.15.107:0/3528428954 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4e80a2da0 0x7fd4e80a5260 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4006010 tx=0x7fd4f40045a0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:24.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.066+0000 7fd4fe59c640 1 --2- 172.21.15.107:0/1291920690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a74730 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000bc60 tx=0x7fd4f000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0689d57a6c497a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:24.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.066+0000 7fd4ecff9640 1 -- 172.21.15.107:0/1291920690 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0015070 con 0x55d245a74350 2024-05-09T08:40:24.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.066+0000 7fd4ecff9640 1 -- 172.21.15.107:0/1291920690 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4f001a910 con 0x55d245a74350 2024-05-09T08:40:24.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.066+0000 7fd4ecff9640 1 -- 172.21.15.107:0/1291920690 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0011880 con 0x55d245a74350 2024-05-09T08:40:24.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.066+0000 7fd5075b9740 1 -- 172.21.15.107:0/1291920690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 msgr2=0x55d245a74730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:24.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.066+0000 7fd5075b9740 1 --2- 172.21.15.107:0/1291920690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a74730 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000bc60 tx=0x7fd4f000f730 comp rx=0 tx=0).stop 2024-05-09T08:40:24.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.067+0000 7fd5075b9740 1 -- 172.21.15.107:0/1291920690 shutdown_connections 2024-05-09T08:40:24.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.067+0000 7fd5075b9740 1 --2- 172.21.15.107:0/1291920690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a83a40 0x55d245a85e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.067+0000 7fd5075b9740 1 --2- 172.21.15.107:0/1291920690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a74c70 0x55d245a83500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.067+0000 7fd5075b9740 1 --2- 172.21.15.107:0/1291920690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a74730 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.067+0000 7fd5075b9740 1 -- 172.21.15.107:0/1291920690 >> 172.21.15.107:0/1291920690 conn(0x55d245a70960 msgr2=0x55d245a70d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:24.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.067+0000 7fd5075b9740 1 -- 172.21.15.107:0/1291920690 shutdown_connections 2024-05-09T08:40:24.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.067+0000 7fd5075b9740 1 -- 172.21.15.107:0/1291920690 wait complete. 2024-05-09T08:40:24.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.067+0000 7fd5075b9740 1 -- start start 2024-05-09T08:40:24.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.067+0000 7fd5075b9740 1 Processor -- start 2024-05-09T08:40:24.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.067+0000 7fd5075b9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74350 0x55d245a80a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.067+0000 7fd5075b9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a74c70 0x55d245a80f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.067+0000 7fd5075b9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a83a40 0x55d245a81480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.067+0000 7fd5075b9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d245a86670 con 0x55d245a83a40 2024-05-09T08:40:24.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.067+0000 7fd5075b9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d245994790 con 0x55d245a74350 2024-05-09T08:40:24.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.067+0000 7fd5075b9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d245a867f0 con 0x55d245a74c70 2024-05-09T08:40:24.076 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.067+0000 7fd5075b9740 10 client.0 _ll_register_callbacks cb 0x55d2459cfa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd4fed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a83a40 0x55d245a81480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd4fed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a83a40 0x55d245a81480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43738/0 (socket says 172.21.15.107:43738) 2024-05-09T08:40:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd4fed9d640 1 -- 172.21.15.107:0/1373112071 learned_addr learned my addr 172.21.15.107:0/1373112071 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd4fdd9b640 1 --2- 172.21.15.107:0/1373112071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a74c70 0x55d245a80f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd4fe59c640 1 --2- 172.21.15.107:0/1373112071 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74350 0x55d245a80a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd4fed9d640 1 -- 172.21.15.107:0/1373112071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a74c70 msgr2=0x55d245a80f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd4fed9d640 1 --2- 172.21.15.107:0/1373112071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a74c70 0x55d245a80f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd4fed9d640 1 -- 172.21.15.107:0/1373112071 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74350 msgr2=0x55d245a80a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd4fed9d640 1 --2- 172.21.15.107:0/1373112071 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74350 0x55d245a80a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd4fed9d640 1 -- 172.21.15.107:0/1373112071 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2456ed210 con 0x55d245a83a40 2024-05-09T08:40:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd4fe59c640 1 --2- 172.21.15.107:0/1373112071 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74350 0x55d245a80a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:40:24.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd4fed9d640 1 --2- 172.21.15.107:0/1373112071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a83a40 0x55d245a81480 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8007c80 tx=0x7fd4f809cfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b413db93b71cfab0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:24.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd4e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:24.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd4e5ffb640 1 -- 172.21.15.107:0/1373112071 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f809dd40 con 0x55d245a83a40 2024-05-09T08:40:24.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd4e5ffb640 1 -- 172.21.15.107:0/1373112071 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fd4f80ceaf0 con 0x55d245a83a40 2024-05-09T08:40:24.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd5075b9740 1 -- 172.21.15.107:0/1373112071 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4f001a910 con 0x55d245a83a40 2024-05-09T08:40:24.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.068+0000 7fd4e5ffb640 1 -- 172.21.15.107:0/1373112071 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd4f809d640 con 0x55d245a83a40 2024-05-09T08:40:24.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.069+0000 7fd5075b9740 10 client.17679 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:24.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.069+0000 7fd5075b9740 1 -- 172.21.15.107:0/1373112071 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d2457b9620 con 0x55d245a83a40 2024-05-09T08:40:24.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.069+0000 7fd4e5ffb640 1 -- 172.21.15.107:0/1373112071 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4f001a910 con 0x55d245a83a40 2024-05-09T08:40:24.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.069+0000 7fd4e5ffb640 1 -- 172.21.15.107:0/1373112071 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd4f8014070 con 0x55d245a83a40 2024-05-09T08:40:24.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.069+0000 7fd5075b9740 10 client.17679 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:24.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.069+0000 7fd5075b9740 10 client.17679 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:40:24.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.069+0000 7fd5075b9740 20 client.17679 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.069+0000 7fd5075b9740 1 --2- 172.21.15.107:0/1373112071 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d245a938f0 0x55d245a95d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.069+0000 7fd5075b9740 4 client.17679 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:40:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.069+0000 7fd5075b9740 1 -- 172.21.15.107:0/1373112071 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2459e6a70 con 0x55d245a938f0 2024-05-09T08:40:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.070+0000 7fd4fe59c640 1 --2- 172.21.15.107:0/1373112071 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d245a938f0 0x55d245a95d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.070+0000 7fd4e5ffb640 10 client.17679 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.070+0000 7fd4fe59c640 1 --2- 172.21.15.107:0/1373112071 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d245a938f0 0x55d245a95d00 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20e998d18cba061e server_cookie=215044ac34588330 in_seq=0 out_seq=0 2024-05-09T08:40:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.071+0000 7fd4e5ffb640 1 -- 172.21.15.107:0/1373112071 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d2459e6a70 con 0x55d245a938f0 2024-05-09T08:40:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.071+0000 7fd4e5ffb640 10 client.17679 handle_command_reply: tid=0 2024-05-09T08:40:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.071+0000 7fd5075b9740 1 client.17679 shutdown 2024-05-09T08:40:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.072+0000 7fd5075b9740 1 -- 172.21.15.107:0/1373112071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a83a40 msgr2=0x55d245a81480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.072+0000 7fd5075b9740 1 --2- 172.21.15.107:0/1373112071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a83a40 0x55d245a81480 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8007c80 tx=0x7fd4f809cfa0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.072+0000 7fd5075b9740 1 -- 172.21.15.107:0/1373112071 shutdown_connections 2024-05-09T08:40:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.072+0000 7fd5075b9740 1 --2- 172.21.15.107:0/1373112071 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d245a938f0 0x55d245a95d00 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.072+0000 7fd5075b9740 1 --2- 172.21.15.107:0/1373112071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a83a40 0x55d245a81480 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.072+0000 7fd5075b9740 1 --2- 172.21.15.107:0/1373112071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a74c70 0x55d245a80f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.072+0000 7fd4e5ffb640 0 client.17679 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.072+0000 7fd5075b9740 1 --2- 172.21.15.107:0/1373112071 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74350 0x55d245a80a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.072+0000 7fd5075b9740 1 -- 172.21.15.107:0/1373112071 >> 172.21.15.107:0/1373112071 conn(0x55d245a70960 msgr2=0x55d245a767d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.072+0000 7fd5075b9740 1 -- 172.21.15.107:0/1373112071 shutdown_connections 2024-05-09T08:40:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.072+0000 7fd5075b9740 1 -- 172.21.15.107:0/1373112071 wait complete. 2024-05-09T08:40:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.072+0000 7fd5075b9740 20 client.17679 trim_cache size 0 max 16384 2024-05-09T08:40:24.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.091+0000 7fd5075b9740 1 Processor -- start 2024-05-09T08:40:24.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.091+0000 7fd5075b9740 1 -- start start 2024-05-09T08:40:24.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.091+0000 7fd5075b9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a8f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.091+0000 7fd5075b9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74c70 0x55d245a829b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.091+0000 7fd5075b9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a83a40 0x55d245a9b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.091+0000 7fd5075b9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd5001af220 con 0x55d245a74350 2024-05-09T08:40:24.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.091+0000 7fd5075b9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd5001acaf0 con 0x55d245a74c70 2024-05-09T08:40:24.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.091+0000 7fd5075b9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd5001aa960 con 0x55d245a83a40 2024-05-09T08:40:24.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.092+0000 7fd4fe59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a8f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.092+0000 7fd4fe59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a8f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43752/0 (socket says 172.21.15.107:43752) 2024-05-09T08:40:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.092+0000 7fd4fe59c640 1 -- 172.21.15.107:0/2184842226 learned_addr learned my addr 172.21.15.107:0/2184842226 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.092+0000 7fd4fdd9b640 1 --2- 172.21.15.107:0/2184842226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74c70 0x55d245a829b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.092+0000 7fd4fed9d640 1 --2- 172.21.15.107:0/2184842226 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a83a40 0x55d245a9b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.092+0000 7fd4fe59c640 1 -- 172.21.15.107:0/2184842226 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a83a40 msgr2=0x55d245a9b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.092+0000 7fd4fe59c640 1 --2- 172.21.15.107:0/2184842226 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a83a40 0x55d245a9b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.092+0000 7fd4fe59c640 1 -- 172.21.15.107:0/2184842226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74c70 msgr2=0x55d245a829b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.092+0000 7fd4fe59c640 1 --2- 172.21.15.107:0/2184842226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74c70 0x55d245a829b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.092+0000 7fd4fe59c640 1 -- 172.21.15.107:0/2184842226 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d245a7f9b0 con 0x55d245a74350 2024-05-09T08:40:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.092+0000 7fd4fe59c640 1 --2- 172.21.15.107:0/2184842226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a8f530 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fd4f00114b0 tx=0x7fd4f0011580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c6820ff5f322f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.092+0000 7fd4e7fff640 1 -- 172.21.15.107:0/2184842226 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0015040 con 0x55d245a74350 2024-05-09T08:40:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.092+0000 7fd4e7fff640 1 -- 172.21.15.107:0/2184842226 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4f000be50 con 0x55d245a74350 2024-05-09T08:40:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.092+0000 7fd4e7fff640 1 -- 172.21.15.107:0/2184842226 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0004c30 con 0x55d245a74350 2024-05-09T08:40:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.093+0000 7fd5075b9740 1 -- 172.21.15.107:0/2184842226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 msgr2=0x55d245a8f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.093+0000 7fd5075b9740 1 --2- 172.21.15.107:0/2184842226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a8f530 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fd4f00114b0 tx=0x7fd4f0011580 comp rx=0 tx=0).stop 2024-05-09T08:40:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.093+0000 7fd5075b9740 1 -- 172.21.15.107:0/2184842226 shutdown_connections 2024-05-09T08:40:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.093+0000 7fd5075b9740 1 --2- 172.21.15.107:0/2184842226 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a83a40 0x55d245a9b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.093+0000 7fd5075b9740 1 --2- 172.21.15.107:0/2184842226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74c70 0x55d245a829b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.093+0000 7fd5075b9740 1 --2- 172.21.15.107:0/2184842226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a8f530 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.093+0000 7fd5075b9740 1 -- 172.21.15.107:0/2184842226 >> 172.21.15.107:0/2184842226 conn(0x55d245a70960 msgr2=0x55d245a70d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.093+0000 7fd5075b9740 1 -- 172.21.15.107:0/2184842226 shutdown_connections 2024-05-09T08:40:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.093+0000 7fd5075b9740 1 -- 172.21.15.107:0/2184842226 wait complete. 2024-05-09T08:40:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd5075b9740 1 -- start start 2024-05-09T08:40:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd5075b9740 1 Processor -- start 2024-05-09T08:40:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd5075b9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a7d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd5075b9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74c70 0x55d245a7dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd5075b9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a83a40 0x55d245a7e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd5075b9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd5001aafb0 con 0x55d245a74350 2024-05-09T08:40:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd5075b9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd5001ac420 con 0x55d245a74c70 2024-05-09T08:40:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd5075b9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd5001ab680 con 0x55d245a83a40 2024-05-09T08:40:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd5075b9740 10 client.0 _ll_register_callbacks cb 0x7fd4f8007ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd4fe59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a7d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd4fdd9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74c70 0x55d245a7dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd4fe59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a7d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43768/0 (socket says 172.21.15.107:43768) 2024-05-09T08:40:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd4fe59c640 1 -- 172.21.15.107:0/1788438013 learned_addr learned my addr 172.21.15.107:0/1788438013 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd4fdd9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74c70 0x55d245a7dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:51384/0 (socket says 172.21.15.107:51384) 2024-05-09T08:40:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd4fed9d640 1 --2- 172.21.15.107:0/1788438013 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a83a40 0x55d245a7e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd4fe59c640 1 -- 172.21.15.107:0/1788438013 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a83a40 msgr2=0x55d245a7e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd4fe59c640 1 --2- 172.21.15.107:0/1788438013 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a83a40 0x55d245a7e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd4fe59c640 1 -- 172.21.15.107:0/1788438013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74c70 msgr2=0x55d245a7dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd4fe59c640 1 --2- 172.21.15.107:0/1788438013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74c70 0x55d245a7dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd4fe59c640 1 -- 172.21.15.107:0/1788438013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d245af8fd0 con 0x55d245a74350 2024-05-09T08:40:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.094+0000 7fd4fdd9b640 1 --2- 172.21.15.107:0/1788438013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74c70 0x55d245a7dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:40:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.095+0000 7fd4fe59c640 1 --2- 172.21.15.107:0/1788438013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a7d820 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000ffa0 tx=0x7fd4f000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec775131694578e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.095+0000 7fd4e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.095+0000 7fd4e5ffb640 1 -- 172.21.15.107:0/1788438013 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0015050 con 0x55d245a74350 2024-05-09T08:40:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.095+0000 7fd4e5ffb640 1 -- 172.21.15.107:0/1788438013 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fd4f00117f0 con 0x55d245a74350 2024-05-09T08:40:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.095+0000 7fd5075b9740 1 -- 172.21.15.107:0/1788438013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4f000be50 con 0x55d245a74350 2024-05-09T08:40:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.095+0000 7fd4e5ffb640 1 -- 172.21.15.107:0/1788438013 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd4f00119f0 con 0x55d245a74350 2024-05-09T08:40:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.095+0000 7fd5075b9740 10 client.17685 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.095+0000 7fd5075b9740 1 -- 172.21.15.107:0/1788438013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d245afa2a0 con 0x55d245a74350 2024-05-09T08:40:24.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.095+0000 7fd4e5ffb640 1 -- 172.21.15.107:0/1788438013 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4f000be50 con 0x55d245a74350 2024-05-09T08:40:24.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.095+0000 7fd4e5ffb640 1 -- 172.21.15.107:0/1788438013 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd4f0036030 con 0x55d245a74350 2024-05-09T08:40:24.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.095+0000 7fd5075b9740 10 client.17685 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:24.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.095+0000 7fd5075b9740 10 client.17685 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:40:24.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.095+0000 7fd5075b9740 20 client.17685 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:24.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.095+0000 7fd5075b9740 1 --2- 172.21.15.107:0/1788438013 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d245afa710 0x55d245afcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:24.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.095+0000 7fd5075b9740 4 client.17685 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:40:24.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.095+0000 7fd5075b9740 1 -- 172.21.15.107:0/1788438013 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d2459e0dc0 con 0x55d245afa710 2024-05-09T08:40:24.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.096+0000 7fd4fdd9b640 1 --2- 172.21.15.107:0/1788438013 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d245afa710 0x55d245afcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:24.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.096+0000 7fd4fdd9b640 1 --2- 172.21.15.107:0/1788438013 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d245afa710 0x55d245afcb00 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de2ebd90248731aa server_cookie=3c0573e3a3e1ecc6 in_seq=0 out_seq=0 2024-05-09T08:40:24.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.096+0000 7fd4e5ffb640 10 client.17685 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.113+0000 7fd4e5ffb640 1 -- 172.21.15.107:0/1788438013 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d2459e0dc0 con 0x55d245afa710 2024-05-09T08:40:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.113+0000 7fd4e5ffb640 10 client.17685 handle_command_reply: tid=0 2024-05-09T08:40:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.113+0000 7fd5075b9740 1 client.17685 shutdown 2024-05-09T08:40:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.113+0000 7fd5075b9740 1 -- 172.21.15.107:0/1788438013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 msgr2=0x55d245a7d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.113+0000 7fd5075b9740 1 --2- 172.21.15.107:0/1788438013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a7d820 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000ffa0 tx=0x7fd4f000b900 comp rx=0 tx=0).stop 2024-05-09T08:40:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.113+0000 7fd5075b9740 1 -- 172.21.15.107:0/1788438013 shutdown_connections 2024-05-09T08:40:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.113+0000 7fd5075b9740 1 --2- 172.21.15.107:0/1788438013 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d245afa710 0x55d245afcb00 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.113+0000 7fd5075b9740 1 --2- 172.21.15.107:0/1788438013 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d245a83a40 0x55d245a7e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.113+0000 7fd4e5ffb640 0 client.17685 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.113+0000 7fd5075b9740 1 --2- 172.21.15.107:0/1788438013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d245a74c70 0x55d245a7dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.113+0000 7fd5075b9740 1 --2- 172.21.15.107:0/1788438013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d245a74350 0x55d245a7d820 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.113+0000 7fd5075b9740 1 -- 172.21.15.107:0/1788438013 >> 172.21.15.107:0/1788438013 conn(0x55d245a70960 msgr2=0x55d245a76730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.113+0000 7fd5075b9740 1 -- 172.21.15.107:0/1788438013 shutdown_connections 2024-05-09T08:40:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.113+0000 7fd5075b9740 1 -- 172.21.15.107:0/1788438013 wait complete. 2024-05-09T08:40:24.114 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.113+0000 7fd5075b9740 20 client.17685 trim_cache size 0 max 16384 2024-05-09T08:40:24.114 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:40:24.114 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:40:24.114 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:40:24.114 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:40:24.115 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.114+0000 7fd5048a5640 1 -- 172.21.15.107:0/3528428954 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4e80a2da0 msgr2=0x7fd4e80a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:24.115 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.114+0000 7fd5048a5640 1 --2- 172.21.15.107:0/3528428954 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4e80a2da0 0x7fd4e80a5260 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4006010 tx=0x7fd4f40045a0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.115 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.114+0000 7fd5048a5640 1 -- 172.21.15.107:0/3528428954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd500106720 msgr2=0x7fd50019b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:24.115 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.114+0000 7fd5048a5640 1 --2- 172.21.15.107:0/3528428954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd500106720 0x7fd50019b780 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8007cb0 tx=0x7fd4f800e420 comp rx=0 tx=0).stop 2024-05-09T08:40:24.115 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.114+0000 7fd5048a5640 1 -- 172.21.15.107:0/3528428954 shutdown_connections 2024-05-09T08:40:24.115 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.114+0000 7fd5048a5640 1 --2- 172.21.15.107:0/3528428954 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4e80a2da0 0x7fd4e80a5260 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.114+0000 7fd5048a5640 1 --2- 172.21.15.107:0/3528428954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd500106720 0x7fd50019b780 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.114+0000 7fd5048a5640 1 --2- 172.21.15.107:0/3528428954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd500101120 0x7fd500197450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.114+0000 7fd5048a5640 1 --2- 172.21.15.107:0/3528428954 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd500100760 0x7fd500196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.114+0000 7fd5048a5640 1 -- 172.21.15.107:0/3528428954 >> 172.21.15.107:0/3528428954 conn(0x7fd5000778b0 msgr2=0x7fd500103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.114+0000 7fd5048a5640 1 -- 172.21.15.107:0/3528428954 shutdown_connections 2024-05-09T08:40:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:24.114+0000 7fd5048a5640 1 -- 172.21.15.107:0/3528428954 wait complete. 2024-05-09T08:40:24.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:24 smithi107 ceph-mon[30122]: pgmap v216: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 26 MiB/s wr, 21 op/s 2024-05-09T08:40:24.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:24 smithi157 ceph-mon[33614]: pgmap v216: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 26 MiB/s wr, 21 op/s 2024-05-09T08:40:24.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:24 smithi110 ceph-mon[33622]: pgmap v216: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 26 MiB/s wr, 21 op/s 2024-05-09T08:40:26.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:26 smithi107 ceph-mon[30122]: pgmap v217: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 25 MiB/s wr, 14 op/s 2024-05-09T08:40:26.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:26 smithi157 ceph-mon[33614]: pgmap v217: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 25 MiB/s wr, 14 op/s 2024-05-09T08:40:26.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:26 smithi110 ceph-mon[33622]: pgmap v217: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 25 MiB/s wr, 14 op/s 2024-05-09T08:40:27.867 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:40:27.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:40:27.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.986+0000 7f9556bf9640 1 -- 172.21.15.107:0/2189959410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9550106bc0 msgr2=0x7f955010e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:27.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.986+0000 7f9556bf9640 1 --2- 172.21.15.107:0/2189959410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9550106bc0 0x7f955010e740 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f9548009ca0 tx=0x7f954802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:40:27.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.987+0000 7f9556bf9640 1 -- 172.21.15.107:0/2189959410 shutdown_connections 2024-05-09T08:40:27.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.987+0000 7f9556bf9640 1 --2- 172.21.15.107:0/2189959410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9550106bc0 0x7f955010e740 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:27.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.987+0000 7f9556bf9640 1 --2- 172.21.15.107:0/2189959410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9550100960 0x7f9550106490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:27.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.987+0000 7f9556bf9640 1 --2- 172.21.15.107:0/2189959410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95500fffb0 0x7f9550100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:27.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.987+0000 7f9556bf9640 1 -- 172.21.15.107:0/2189959410 >> 172.21.15.107:0/2189959410 conn(0x7f9550076cd0 msgr2=0x7f95500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:27.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.987+0000 7f9556bf9640 1 -- 172.21.15.107:0/2189959410 shutdown_connections 2024-05-09T08:40:27.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.987+0000 7f9556bf9640 1 -- 172.21.15.107:0/2189959410 wait complete. 2024-05-09T08:40:27.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.987+0000 7f9556bf9640 1 Processor -- start 2024-05-09T08:40:27.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.987+0000 7f9556bf9640 1 -- start start 2024-05-09T08:40:27.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9556bf9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95500fffb0 0x7f9550198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:27.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9556bf9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9550106bc0 0x7f9550198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:27.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9556bf9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f955019c8f0 0x7f955019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:27.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9556bf9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9550111ad0 con 0x7f95500fffb0 2024-05-09T08:40:27.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9556bf9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9550111950 con 0x7f9550106bc0 2024-05-09T08:40:27.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9556bf9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9550111c50 con 0x7f955019c8f0 2024-05-09T08:40:27.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9554995640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95500fffb0 0x7f9550198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:27.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9554995640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95500fffb0 0x7f9550198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43776/0 (socket says 172.21.15.107:43776) 2024-05-09T08:40:27.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9554995640 1 -- 172.21.15.107:0/2738978663 learned_addr learned my addr 172.21.15.107:0/2738978663 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:27.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f954ffff640 1 --2- 172.21.15.107:0/2738978663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9550106bc0 0x7f9550198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:27.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9555196640 1 --2- 172.21.15.107:0/2738978663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f955019c8f0 0x7f955019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:27.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9554995640 1 -- 172.21.15.107:0/2738978663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f955019c8f0 msgr2=0x7f955019cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:27.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9554995640 1 --2- 172.21.15.107:0/2738978663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f955019c8f0 0x7f955019cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:27.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9554995640 1 -- 172.21.15.107:0/2738978663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9550106bc0 msgr2=0x7f9550198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:27.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9554995640 1 --2- 172.21.15.107:0/2738978663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9550106bc0 0x7f9550198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:27.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9554995640 1 -- 172.21.15.107:0/2738978663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f955019d3e0 con 0x7f95500fffb0 2024-05-09T08:40:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9554995640 1 --2- 172.21.15.107:0/2738978663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95500fffb0 0x7f9550198030 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f954400d950 tx=0x7f954400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f954dffb640 1 -- 172.21.15.107:0/2738978663 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544017030 con 0x7f95500fffb0 2024-05-09T08:40:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f9556bf9640 1 -- 172.21.15.107:0/2738978663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f955019d6d0 con 0x7f95500fffb0 2024-05-09T08:40:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f954dffb640 1 -- 172.21.15.107:0/2738978663 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9544004020 con 0x7f95500fffb0 2024-05-09T08:40:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.988+0000 7f954dffb640 1 -- 172.21.15.107:0/2738978663 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95440047a0 con 0x7f95500fffb0 2024-05-09T08:40:27.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.989+0000 7f9556bf9640 1 -- 172.21.15.107:0/2738978663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95501a4ec0 con 0x7f95500fffb0 2024-05-09T08:40:27.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.991+0000 7f954dffb640 1 -- 172.21.15.107:0/2738978663 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f95440041d0 con 0x7f95500fffb0 2024-05-09T08:40:27.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.992+0000 7f95599e6740 1 Processor -- start 2024-05-09T08:40:27.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.992+0000 7f95599e6740 1 -- start start 2024-05-09T08:40:27.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.992+0000 7f95599e6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3ce280 0x55c08a3ce660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:27.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.992+0000 7f95599e6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3ceba0 0x55c08a3d8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:27.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.992+0000 7f95599e6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3d9270 0x55c08a3db660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:27.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.992+0000 7f95599e6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c08a333d00 con 0x55c08a3ceba0 2024-05-09T08:40:27.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.992+0000 7f95599e6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c08a2db4c0 con 0x55c08a3d9270 2024-05-09T08:40:27.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.992+0000 7f95599e6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c08a32fcc0 con 0x55c08a3ce280 2024-05-09T08:40:27.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.992+0000 7f9554995640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3ceba0 0x55c08a3d8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:27.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.992+0000 7f9554995640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3ceba0 0x55c08a3d8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43788/0 (socket says 172.21.15.107:43788) 2024-05-09T08:40:27.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.992+0000 7f954dffb640 1 --2- 172.21.15.107:0/2738978663 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95400a2d00 0x7f95400a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:27.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.992+0000 7f9554995640 1 -- 172.21.15.107:0/3728635214 learned_addr learned my addr 172.21.15.107:0/3728635214 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:27.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.992+0000 7f9555196640 1 --2- 172.21.15.107:0/3728635214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3ce280 0x55c08a3ce660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:27.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.992+0000 7f954ffff640 1 --2- 172.21.15.107:0/3728635214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3d9270 0x55c08a3db660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:27.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.993+0000 7f9555196640 1 --2- 172.21.15.107:0/2738978663 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95400a2d00 0x7f95400a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:27.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.993+0000 7f954dffb640 1 -- 172.21.15.107:0/2738978663 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f95440cf340 con 0x7f95500fffb0 2024-05-09T08:40:27.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.993+0000 7f9554995640 1 -- 172.21.15.107:0/3728635214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3ce280 msgr2=0x55c08a3ce660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:27.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.993+0000 7f9554995640 1 --2- 172.21.15.107:0/3728635214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3ce280 0x55c08a3ce660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:27.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.993+0000 7f9554995640 1 -- 172.21.15.107:0/3728635214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3d9270 msgr2=0x55c08a3db660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:27.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.993+0000 7f9554995640 1 --2- 172.21.15.107:0/3728635214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3d9270 0x55c08a3db660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:27.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.993+0000 7f9554995640 1 -- 172.21.15.107:0/3728635214 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c08a1117d0 con 0x55c08a3ceba0 2024-05-09T08:40:27.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.993+0000 7f9555196640 1 --2- 172.21.15.107:0/3728635214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3ce280 0x55c08a3ce660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:40:27.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.993+0000 7f954ffff640 1 --2- 172.21.15.107:0/3728635214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3d9270 0x55c08a3db660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:40:27.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.993+0000 7f9554995640 1 --2- 172.21.15.107:0/3728635214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3ceba0 0x55c08a3d8d30 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f954409b440 tx=0x7f954409ae10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e337c5613839bbb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:27.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.993+0000 7f9555196640 1 --2- 172.21.15.107:0/2738978663 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95400a2d00 0x7f95400a51c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9548005e30 tx=0x7f954802a8c0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:27.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.993+0000 7f953affd640 1 -- 172.21.15.107:0/3728635214 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954409aff0 con 0x55c08a3ceba0 2024-05-09T08:40:28.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.993+0000 7f953affd640 1 -- 172.21.15.107:0/3728635214 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f954409b4b0 con 0x55c08a3ceba0 2024-05-09T08:40:28.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.993+0000 7f953affd640 1 -- 172.21.15.107:0/3728635214 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95440a3c90 con 0x55c08a3ceba0 2024-05-09T08:40:28.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.994+0000 7f95599e6740 1 -- 172.21.15.107:0/3728635214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3ceba0 msgr2=0x55c08a3d8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.994+0000 7f95599e6740 1 --2- 172.21.15.107:0/3728635214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3ceba0 0x55c08a3d8d30 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f954409b440 tx=0x7f954409ae10 comp rx=0 tx=0).stop 2024-05-09T08:40:28.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.994+0000 7f95599e6740 1 -- 172.21.15.107:0/3728635214 shutdown_connections 2024-05-09T08:40:28.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.994+0000 7f95599e6740 1 --2- 172.21.15.107:0/3728635214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3d9270 0x55c08a3db660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.994+0000 7f95599e6740 1 --2- 172.21.15.107:0/3728635214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3ceba0 0x55c08a3d8d30 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.994+0000 7f95599e6740 1 --2- 172.21.15.107:0/3728635214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3ce280 0x55c08a3ce660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.994+0000 7f95599e6740 1 -- 172.21.15.107:0/3728635214 >> 172.21.15.107:0/3728635214 conn(0x55c08a3ca5a0 msgr2=0x55c08a3ca980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.994+0000 7f95599e6740 1 -- 172.21.15.107:0/3728635214 shutdown_connections 2024-05-09T08:40:28.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.994+0000 7f95599e6740 1 -- 172.21.15.107:0/3728635214 wait complete. 2024-05-09T08:40:28.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.995+0000 7f95599e6740 1 -- start start 2024-05-09T08:40:28.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.995+0000 7f95599e6740 1 Processor -- start 2024-05-09T08:40:28.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.995+0000 7f95599e6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3ce280 0x55c08a3d6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.995+0000 7f95599e6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3ceba0 0x55c08a3d67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.995+0000 7f95599e6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3d9270 0x55c08a3d6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.995+0000 7f95599e6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c08a3dc000 con 0x55c08a3d9270 2024-05-09T08:40:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.995+0000 7f95599e6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c08a2eeb90 con 0x55c08a3ceba0 2024-05-09T08:40:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.995+0000 7f95599e6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c08a3dc180 con 0x55c08a3ce280 2024-05-09T08:40:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.995+0000 7f95599e6740 10 client.0 _ll_register_callbacks cb 0x55c08a32ada0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.995+0000 7f9555196640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3d9270 0x55c08a3d6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.995+0000 7f9555196640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3d9270 0x55c08a3d6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43792/0 (socket says 172.21.15.107:43792) 2024-05-09T08:40:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.995+0000 7f9555196640 1 -- 172.21.15.107:0/284609259 learned_addr learned my addr 172.21.15.107:0/284609259 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.995+0000 7f9554995640 1 --2- 172.21.15.107:0/284609259 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3ce280 0x55c08a3d6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.995+0000 7f954ffff640 1 --2- 172.21.15.107:0/284609259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3ceba0 0x55c08a3d67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.996+0000 7f9555196640 1 -- 172.21.15.107:0/284609259 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3ce280 msgr2=0x55c08a3d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.996+0000 7f9555196640 1 --2- 172.21.15.107:0/284609259 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3ce280 0x55c08a3d6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.996+0000 7f9555196640 1 -- 172.21.15.107:0/284609259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3ceba0 msgr2=0x55c08a3d67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.996+0000 7f9555196640 1 --2- 172.21.15.107:0/284609259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3ceba0 0x55c08a3d67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.996+0000 7f9555196640 1 -- 172.21.15.107:0/284609259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c08a0473f0 con 0x55c08a3d9270 2024-05-09T08:40:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.996+0000 7f954ffff640 1 --2- 172.21.15.107:0/284609259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3ceba0 0x55c08a3d67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:40:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.996+0000 7f9555196640 1 --2- 172.21.15.107:0/284609259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3d9270 0x55c08a3d6d10 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f9548003bf0 tx=0x7f95480095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f7b4a1630a9020 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.996+0000 7f952ffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.996+0000 7f952ffff640 1 -- 172.21.15.107:0/284609259 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9548004640 con 0x55c08a3d9270 2024-05-09T08:40:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.996+0000 7f952ffff640 1 -- 172.21.15.107:0/284609259 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f95480047e0 con 0x55c08a3d9270 2024-05-09T08:40:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.996+0000 7f95599e6740 1 -- 172.21.15.107:0/284609259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f954409b4b0 con 0x55c08a3d9270 2024-05-09T08:40:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.996+0000 7f952ffff640 1 -- 172.21.15.107:0/284609259 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9548059050 con 0x55c08a3d9270 2024-05-09T08:40:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.996+0000 7f95599e6740 10 client.17697 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.996+0000 7f95599e6740 1 -- 172.21.15.107:0/284609259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c08a0070f0 con 0x55c08a3d9270 2024-05-09T08:40:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.996+0000 7f952ffff640 1 -- 172.21.15.107:0/284609259 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f954409b4b0 con 0x55c08a3d9270 2024-05-09T08:40:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.997+0000 7f952ffff640 1 -- 172.21.15.107:0/284609259 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f954802f6c0 con 0x55c08a3d9270 2024-05-09T08:40:28.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.997+0000 7f95599e6740 10 client.17697 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.997+0000 7f95599e6740 10 client.17697 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:40:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.997+0000 7f95599e6740 20 client.17697 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.997+0000 7f95599e6740 1 --2- 172.21.15.107:0/284609259 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c08a3e96c0 0x55c08a3ebae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.997+0000 7f95599e6740 4 client.17697 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:40:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.997+0000 7f95599e6740 1 -- 172.21.15.107:0/284609259 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c08a341430 con 0x55c08a3e96c0 2024-05-09T08:40:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.997+0000 7f9554995640 1 --2- 172.21.15.107:0/284609259 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c08a3e96c0 0x55c08a3ebae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.997+0000 7f9554995640 1 --2- 172.21.15.107:0/284609259 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c08a3e96c0 0x55c08a3ebae0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ccfd0bf2be6c240 server_cookie=36cc2fea538de3b in_seq=0 out_seq=0 2024-05-09T08:40:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.997+0000 7f952ffff640 10 client.17697 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.998+0000 7f952ffff640 1 -- 172.21.15.107:0/284609259 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c08a341430 con 0x55c08a3e96c0 2024-05-09T08:40:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.998+0000 7f952ffff640 10 client.17697 handle_command_reply: tid=0 2024-05-09T08:40:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.998+0000 7f95599e6740 1 client.17697 shutdown 2024-05-09T08:40:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.998+0000 7f95599e6740 1 -- 172.21.15.107:0/284609259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3d9270 msgr2=0x55c08a3d6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.998+0000 7f95599e6740 1 --2- 172.21.15.107:0/284609259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3d9270 0x55c08a3d6d10 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f9548003bf0 tx=0x7f95480095c0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.998+0000 7f95599e6740 1 -- 172.21.15.107:0/284609259 shutdown_connections 2024-05-09T08:40:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.998+0000 7f95599e6740 1 --2- 172.21.15.107:0/284609259 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c08a3e96c0 0x55c08a3ebae0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.998+0000 7f95599e6740 1 --2- 172.21.15.107:0/284609259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3d9270 0x55c08a3d6d10 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.998+0000 7f95599e6740 1 --2- 172.21.15.107:0/284609259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3ceba0 0x55c08a3d67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.998+0000 7f95599e6740 1 --2- 172.21.15.107:0/284609259 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3ce280 0x55c08a3d6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.998+0000 7f95599e6740 1 -- 172.21.15.107:0/284609259 >> 172.21.15.107:0/284609259 conn(0x55c08a3ca5a0 msgr2=0x55c08a3d03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.998+0000 7f952ffff640 0 client.17697 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.998+0000 7f95599e6740 1 -- 172.21.15.107:0/284609259 shutdown_connections 2024-05-09T08:40:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.998+0000 7f95599e6740 1 -- 172.21.15.107:0/284609259 wait complete. 2024-05-09T08:40:28.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:27.998+0000 7f95599e6740 20 client.17697 trim_cache size 0 max 16384 2024-05-09T08:40:28.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.020+0000 7f95599e6740 1 Processor -- start 2024-05-09T08:40:28.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f95599e6740 1 -- start start 2024-05-09T08:40:28.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f95599e6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3ce280 0x55c08a3c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f95599e6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3ceba0 0x55c08a3e4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f95599e6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3d9270 0x55c08a3d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f95599e6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f95501b15b0 con 0x55c08a3ce280 2024-05-09T08:40:28.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f95599e6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f95501aee80 con 0x55c08a3ceba0 2024-05-09T08:40:28.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f95599e6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f95501accf0 con 0x55c08a3d9270 2024-05-09T08:40:28.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f9554995640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3ce280 0x55c08a3c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f9554995640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3ce280 0x55c08a3c5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43804/0 (socket says 172.21.15.107:43804) 2024-05-09T08:40:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f9554995640 1 -- 172.21.15.107:0/2294281456 learned_addr learned my addr 172.21.15.107:0/2294281456 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f954ffff640 1 --2- 172.21.15.107:0/2294281456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3ceba0 0x55c08a3e4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f9555196640 1 --2- 172.21.15.107:0/2294281456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3d9270 0x55c08a3d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f9554995640 1 -- 172.21.15.107:0/2294281456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3d9270 msgr2=0x55c08a3d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f9554995640 1 --2- 172.21.15.107:0/2294281456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3d9270 0x55c08a3d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f9554995640 1 -- 172.21.15.107:0/2294281456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3ceba0 msgr2=0x55c08a3e4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f9554995640 1 --2- 172.21.15.107:0/2294281456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3ceba0 0x55c08a3e4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f9554995640 1 -- 172.21.15.107:0/2294281456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c08a3d5240 con 0x55c08a3ce280 2024-05-09T08:40:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.021+0000 7f9554995640 1 --2- 172.21.15.107:0/2294281456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3ce280 0x55c08a3c5670 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f95440cc790 tx=0x7f954409af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b34c650ff2659 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.022+0000 7f953a7fc640 1 -- 172.21.15.107:0/2294281456 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954409aff0 con 0x55c08a3ce280 2024-05-09T08:40:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.022+0000 7f953a7fc640 1 -- 172.21.15.107:0/2294281456 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f95440d3020 con 0x55c08a3ce280 2024-05-09T08:40:28.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.022+0000 7f953a7fc640 1 -- 172.21.15.107:0/2294281456 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544097b20 con 0x55c08a3ce280 2024-05-09T08:40:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.022+0000 7f95599e6740 1 -- 172.21.15.107:0/2294281456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3ce280 msgr2=0x55c08a3c5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.022+0000 7f95599e6740 1 --2- 172.21.15.107:0/2294281456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3ce280 0x55c08a3c5670 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f95440cc790 tx=0x7f954409af40 comp rx=0 tx=0).stop 2024-05-09T08:40:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.022+0000 7f95599e6740 1 -- 172.21.15.107:0/2294281456 shutdown_connections 2024-05-09T08:40:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.022+0000 7f95599e6740 1 --2- 172.21.15.107:0/2294281456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3d9270 0x55c08a3d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.022+0000 7f95599e6740 1 --2- 172.21.15.107:0/2294281456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3ceba0 0x55c08a3e4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.022+0000 7f95599e6740 1 --2- 172.21.15.107:0/2294281456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3ce280 0x55c08a3c5670 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.022+0000 7f95599e6740 1 -- 172.21.15.107:0/2294281456 >> 172.21.15.107:0/2294281456 conn(0x55c08a3ca5a0 msgr2=0x55c08a3c8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.022+0000 7f95599e6740 1 -- 172.21.15.107:0/2294281456 shutdown_connections 2024-05-09T08:40:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.022+0000 7f95599e6740 1 -- 172.21.15.107:0/2294281456 wait complete. 2024-05-09T08:40:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.023+0000 7f95599e6740 1 -- start start 2024-05-09T08:40:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.023+0000 7f95599e6740 1 Processor -- start 2024-05-09T08:40:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.023+0000 7f95599e6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3ce280 0x55c08a3e1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.023+0000 7f95599e6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3ceba0 0x55c08a3d2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.023+0000 7f95599e6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3d9270 0x55c08a3d30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.023+0000 7f95599e6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f95501ad340 con 0x55c08a3d9270 2024-05-09T08:40:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.023+0000 7f95599e6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f95501ae7b0 con 0x55c08a3ce280 2024-05-09T08:40:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.023+0000 7f95599e6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f95501ada10 con 0x55c08a3ceba0 2024-05-09T08:40:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.023+0000 7f95599e6740 10 client.0 _ll_register_callbacks cb 0x7f9548009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.023+0000 7f9555196640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3d9270 0x55c08a3d30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.023+0000 7f9555196640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3d9270 0x55c08a3d30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43812/0 (socket says 172.21.15.107:43812) 2024-05-09T08:40:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.023+0000 7f9555196640 1 -- 172.21.15.107:0/1363251563 learned_addr learned my addr 172.21.15.107:0/1363251563 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.023+0000 7f954ffff640 1 --2- 172.21.15.107:0/1363251563 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3ceba0 0x55c08a3d2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.024+0000 7f9554995640 1 --2- 172.21.15.107:0/1363251563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3ce280 0x55c08a3e1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.024+0000 7f9555196640 1 -- 172.21.15.107:0/1363251563 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3ceba0 msgr2=0x55c08a3d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.024+0000 7f9555196640 1 --2- 172.21.15.107:0/1363251563 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3ceba0 0x55c08a3d2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.024+0000 7f9555196640 1 -- 172.21.15.107:0/1363251563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3ce280 msgr2=0x55c08a3e1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.024+0000 7f9555196640 1 --2- 172.21.15.107:0/1363251563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3ce280 0x55c08a3e1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.024+0000 7f9555196640 1 -- 172.21.15.107:0/1363251563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c08a3d3890 con 0x55c08a3d9270 2024-05-09T08:40:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.024+0000 7f9555196640 1 --2- 172.21.15.107:0/1363251563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3d9270 0x55c08a3d30c0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f954802fa80 tx=0x7f9548003ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=566ff83148607212 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.024+0000 7f952ffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.024+0000 7f952ffff640 1 -- 172.21.15.107:0/1363251563 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9548061020 con 0x55c08a3d9270 2024-05-09T08:40:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.024+0000 7f952ffff640 1 -- 172.21.15.107:0/1363251563 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f954802f6c0 con 0x55c08a3d9270 2024-05-09T08:40:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.024+0000 7f95599e6740 1 -- 172.21.15.107:0/1363251563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95440d3020 con 0x55c08a3d9270 2024-05-09T08:40:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.024+0000 7f952ffff640 1 -- 172.21.15.107:0/1363251563 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f954806c050 con 0x55c08a3d9270 2024-05-09T08:40:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.024+0000 7f95599e6740 10 client.17703 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.024+0000 7f95599e6740 1 -- 172.21.15.107:0/1363251563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c08a453fe0 con 0x55c08a3d9270 2024-05-09T08:40:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.024+0000 7f952ffff640 1 -- 172.21.15.107:0/1363251563 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f95440d3020 con 0x55c08a3d9270 2024-05-09T08:40:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.025+0000 7f952ffff640 1 -- 172.21.15.107:0/1363251563 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9548030900 con 0x55c08a3d9270 2024-05-09T08:40:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.025+0000 7f95599e6740 10 client.17703 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.025+0000 7f95599e6740 10 client.17703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:40:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.025+0000 7f95599e6740 20 client.17703 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.025+0000 7f95599e6740 1 --2- 172.21.15.107:0/1363251563 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c08a4546c0 0x55c08a456ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.025+0000 7f95599e6740 4 client.17703 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:40:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.025+0000 7f95599e6740 1 -- 172.21.15.107:0/1363251563 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c08a33b780 con 0x55c08a4546c0 2024-05-09T08:40:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.025+0000 7f9554995640 1 --2- 172.21.15.107:0/1363251563 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c08a4546c0 0x55c08a456ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.025+0000 7f952ffff640 10 client.17703 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.025+0000 7f9554995640 1 --2- 172.21.15.107:0/1363251563 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c08a4546c0 0x55c08a456ae0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebb3fae13b6b6bf2 server_cookie=41141a829961ba93 in_seq=0 out_seq=0 2024-05-09T08:40:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.025+0000 7f952ffff640 1 -- 172.21.15.107:0/1363251563 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c08a33b780 con 0x55c08a4546c0 2024-05-09T08:40:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.025+0000 7f952ffff640 10 client.17703 handle_command_reply: tid=0 2024-05-09T08:40:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.025+0000 7f95599e6740 1 client.17703 shutdown 2024-05-09T08:40:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.026+0000 7f95599e6740 1 -- 172.21.15.107:0/1363251563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3d9270 msgr2=0x55c08a3d30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.026+0000 7f95599e6740 1 --2- 172.21.15.107:0/1363251563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3d9270 0x55c08a3d30c0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f954802fa80 tx=0x7f9548003ef0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.026+0000 7f95599e6740 1 -- 172.21.15.107:0/1363251563 shutdown_connections 2024-05-09T08:40:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.026+0000 7f95599e6740 1 --2- 172.21.15.107:0/1363251563 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c08a4546c0 0x55c08a456ae0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.026+0000 7f95599e6740 1 --2- 172.21.15.107:0/1363251563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c08a3d9270 0x55c08a3d30c0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.026+0000 7f95599e6740 1 --2- 172.21.15.107:0/1363251563 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c08a3ceba0 0x55c08a3d2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.026+0000 7f952ffff640 0 client.17703 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.026+0000 7f95599e6740 1 --2- 172.21.15.107:0/1363251563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c08a3ce280 0x55c08a3e1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.026+0000 7f95599e6740 1 -- 172.21.15.107:0/1363251563 >> 172.21.15.107:0/1363251563 conn(0x55c08a3ca5a0 msgr2=0x55c08a3d6ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.026+0000 7f95599e6740 1 -- 172.21.15.107:0/1363251563 shutdown_connections 2024-05-09T08:40:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.026+0000 7f95599e6740 1 -- 172.21.15.107:0/1363251563 wait complete. 2024-05-09T08:40:28.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.026+0000 7f95599e6740 20 client.17703 trim_cache size 0 max 16384 2024-05-09T08:40:28.032 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:40:28.032 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:40:28.032 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:40:28.032 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:40:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.027+0000 7f9556bf9640 1 -- 172.21.15.107:0/2738978663 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95400a2d00 msgr2=0x7f95400a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.027+0000 7f9556bf9640 1 --2- 172.21.15.107:0/2738978663 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95400a2d00 0x7f95400a51c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9548005e30 tx=0x7f954802a8c0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.027+0000 7f9556bf9640 1 -- 172.21.15.107:0/2738978663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95500fffb0 msgr2=0x7f9550198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.027+0000 7f9556bf9640 1 --2- 172.21.15.107:0/2738978663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95500fffb0 0x7f9550198030 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f954400d950 tx=0x7f954400de10 comp rx=0 tx=0).stop 2024-05-09T08:40:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.027+0000 7f9556bf9640 1 -- 172.21.15.107:0/2738978663 shutdown_connections 2024-05-09T08:40:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.027+0000 7f9556bf9640 1 --2- 172.21.15.107:0/2738978663 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f95400a2d00 0x7f95400a51c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.027+0000 7f9556bf9640 1 --2- 172.21.15.107:0/2738978663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f955019c8f0 0x7f955019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.027+0000 7f9556bf9640 1 --2- 172.21.15.107:0/2738978663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9550106bc0 0x7f9550198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.027+0000 7f9556bf9640 1 --2- 172.21.15.107:0/2738978663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95500fffb0 0x7f9550198030 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.027+0000 7f9556bf9640 1 -- 172.21.15.107:0/2738978663 >> 172.21.15.107:0/2738978663 conn(0x7f9550076cd0 msgr2=0x7f9550104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.027+0000 7f9556bf9640 1 -- 172.21.15.107:0/2738978663 shutdown_connections 2024-05-09T08:40:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.027+0000 7f9556bf9640 1 -- 172.21.15.107:0/2738978663 wait complete. 2024-05-09T08:40:28.039 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:40:28.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:40:28.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.151+0000 7feb1b241640 1 -- 172.21.15.107:0/228197953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb14106720 msgr2=0x7feb14106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.151+0000 7feb1b241640 1 --2- 172.21.15.107:0/228197953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb14106720 0x7feb14106b00 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7feb08009930 tx=0x7feb0802a420 comp rx=0 tx=0).stop 2024-05-09T08:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.151+0000 7feb1b241640 1 -- 172.21.15.107:0/228197953 shutdown_connections 2024-05-09T08:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.151+0000 7feb1b241640 1 --2- 172.21.15.107:0/228197953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb141010d0 0x7feb1410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.151+0000 7feb1b241640 1 --2- 172.21.15.107:0/228197953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb14100710 0x7feb14100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.151+0000 7feb1b241640 1 --2- 172.21.15.107:0/228197953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb14106720 0x7feb14106b00 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.151+0000 7feb1b241640 1 -- 172.21.15.107:0/228197953 >> 172.21.15.107:0/228197953 conn(0x7feb14077830 msgr2=0x7feb14077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.151+0000 7feb1b241640 1 -- 172.21.15.107:0/228197953 shutdown_connections 2024-05-09T08:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.151+0000 7feb1b241640 1 -- 172.21.15.107:0/228197953 wait complete. 2024-05-09T08:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb1b241640 1 Processor -- start 2024-05-09T08:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb1b241640 1 -- start start 2024-05-09T08:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb1b241640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb14100710 0x7feb14198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb1b241640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb141010d0 0x7feb14198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb1b241640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb14106720 0x7feb1419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb1b241640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feb14069770 con 0x7feb14106720 2024-05-09T08:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb1b241640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feb140695f0 con 0x7feb14100710 2024-05-09T08:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb1b241640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feb140698f0 con 0x7feb141010d0 2024-05-09T08:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb197de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb14106720 0x7feb1419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb197de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb14106720 0x7feb1419c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43840/0 (socket says 172.21.15.107:43840) 2024-05-09T08:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb197de640 1 -- 172.21.15.107:0/2721248592 learned_addr learned my addr 172.21.15.107:0/2721248592 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb18fdd640 1 --2- 172.21.15.107:0/2721248592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb14100710 0x7feb14198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb13fff640 1 --2- 172.21.15.107:0/2721248592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb141010d0 0x7feb14198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb197de640 1 -- 172.21.15.107:0/2721248592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb141010d0 msgr2=0x7feb14198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb197de640 1 --2- 172.21.15.107:0/2721248592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb141010d0 0x7feb14198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb197de640 1 -- 172.21.15.107:0/2721248592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb14100710 msgr2=0x7feb14198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb197de640 1 --2- 172.21.15.107:0/2721248592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb14100710 0x7feb14198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb197de640 1 -- 172.21.15.107:0/2721248592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb1419d030 con 0x7feb14106720 2024-05-09T08:40:28.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.152+0000 7feb13fff640 1 --2- 172.21.15.107:0/2721248592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb141010d0 0x7feb14198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:40:28.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.153+0000 7feb197de640 1 --2- 172.21.15.107:0/2721248592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb14106720 0x7feb1419c900 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009fd0 tx=0x7feb0c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.153+0000 7feb11ffb640 1 -- 172.21.15.107:0/2721248592 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c017070 con 0x7feb14106720 2024-05-09T08:40:28.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.153+0000 7feb11ffb640 1 -- 172.21.15.107:0/2721248592 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb0c003e20 con 0x7feb14106720 2024-05-09T08:40:28.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.153+0000 7feb1b241640 1 -- 172.21.15.107:0/2721248592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb1419d320 con 0x7feb14106720 2024-05-09T08:40:28.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.153+0000 7feb11ffb640 1 -- 172.21.15.107:0/2721248592 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c010460 con 0x7feb14106720 2024-05-09T08:40:28.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.153+0000 7feb1b241640 1 -- 172.21.15.107:0/2721248592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb141a4bb0 con 0x7feb14106720 2024-05-09T08:40:28.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.154+0000 7feb1b241640 1 -- 172.21.15.107:0/2721248592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb14068650 con 0x7feb14106720 2024-05-09T08:40:28.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.156+0000 7feb11ffb640 1 -- 172.21.15.107:0/2721248592 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7feb0c003970 con 0x7feb14106720 2024-05-09T08:40:28.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.157+0000 7feb11ffb640 1 --2- 172.21.15.107:0/2721248592 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feb040a2d80 0x7feb040a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.157+0000 7feb18fdd640 1 --2- 172.21.15.107:0/2721248592 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feb040a2d80 0x7feb040a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.157+0000 7feb11ffb640 1 -- 172.21.15.107:0/2721248592 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7feb0c0cf580 con 0x7feb14106720 2024-05-09T08:40:28.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.157+0000 7feb18fdd640 1 --2- 172.21.15.107:0/2721248592 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feb040a2d80 0x7feb040a5240 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7feb08009900 tx=0x7feb08000bc0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.161+0000 7feb11ffb640 1 -- 172.21.15.107:0/2721248592 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7feb0c09bdf0 con 0x7feb14106720 2024-05-09T08:40:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.444+0000 7feb1b241640 1 -- 172.21.15.107:0/2721248592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb1402d1c0 con 0x7feb14106720 2024-05-09T08:40:28.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.445+0000 7feb11ffb640 1 -- 172.21.15.107:0/2721248592 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11887 (secure 0 0 0) 0x7feb0c09b6f0 con 0x7feb14106720 2024-05-09T08:40:28.444 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:40:28.444 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:37:10.930317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:40:28.445 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2024-05-09T08:40:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.447+0000 7feb1b241640 1 -- 172.21.15.107:0/2721248592 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feb040a2d80 msgr2=0x7feb040a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.447+0000 7feb1b241640 1 --2- 172.21.15.107:0/2721248592 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feb040a2d80 0x7feb040a5240 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7feb08009900 tx=0x7feb08000bc0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.447+0000 7feb1b241640 1 -- 172.21.15.107:0/2721248592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb14106720 msgr2=0x7feb1419c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.447+0000 7feb1b241640 1 --2- 172.21.15.107:0/2721248592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb14106720 0x7feb1419c900 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009fd0 tx=0x7feb0c00b260 comp rx=0 tx=0).stop 2024-05-09T08:40:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.447+0000 7feb1b241640 1 -- 172.21.15.107:0/2721248592 shutdown_connections 2024-05-09T08:40:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.447+0000 7feb1b241640 1 --2- 172.21.15.107:0/2721248592 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feb040a2d80 0x7feb040a5240 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.447+0000 7feb1b241640 1 --2- 172.21.15.107:0/2721248592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb14106720 0x7feb1419c900 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.447+0000 7feb1b241640 1 --2- 172.21.15.107:0/2721248592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb141010d0 0x7feb14198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.447+0000 7feb1b241640 1 --2- 172.21.15.107:0/2721248592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb14100710 0x7feb14198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.447+0000 7feb1b241640 1 -- 172.21.15.107:0/2721248592 >> 172.21.15.107:0/2721248592 conn(0x7feb14077830 msgr2=0x7feb14103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.447+0000 7feb1b241640 1 -- 172.21.15.107:0/2721248592 shutdown_connections 2024-05-09T08:40:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.447+0000 7feb1b241640 1 -- 172.21.15.107:0/2721248592 wait complete. 2024-05-09T08:40:28.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:40:28.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.587+0000 7ff61762a640 1 -- 172.21.15.107:0/2747294355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff610106780 msgr2=0x7ff610106b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.587+0000 7ff61762a640 1 --2- 172.21.15.107:0/2747294355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff610106780 0x7ff610106b60 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009900 tx=0x7ff60c02a410 comp rx=0 tx=0).stop 2024-05-09T08:40:28.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.587+0000 7ff61762a640 1 -- 172.21.15.107:0/2747294355 shutdown_connections 2024-05-09T08:40:28.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.587+0000 7ff61762a640 1 --2- 172.21.15.107:0/2747294355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff610101420 0x7ff61010d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.587+0000 7ff61762a640 1 --2- 172.21.15.107:0/2747294355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff610100a80 0x7ff610100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.587+0000 7ff61762a640 1 --2- 172.21.15.107:0/2747294355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff610106780 0x7ff610106b60 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.587+0000 7ff61762a640 1 -- 172.21.15.107:0/2747294355 >> 172.21.15.107:0/2747294355 conn(0x7ff610076530 msgr2=0x7ff610076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.587+0000 7ff61762a640 1 -- 172.21.15.107:0/2747294355 shutdown_connections 2024-05-09T08:40:28.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.587+0000 7ff61762a640 1 -- 172.21.15.107:0/2747294355 wait complete. 2024-05-09T08:40:28.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff61762a640 1 Processor -- start 2024-05-09T08:40:28.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff61762a640 1 -- start start 2024-05-09T08:40:28.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff61762a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff610100a80 0x7ff610198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff61762a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff610101420 0x7ff6101992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff61762a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff610106780 0x7ff6101a3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff61762a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff610110980 con 0x7ff610100a80 2024-05-09T08:40:28.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff61762a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff610110800 con 0x7ff610106780 2024-05-09T08:40:28.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff61762a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff610110b00 con 0x7ff610101420 2024-05-09T08:40:28.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff6153c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff610100a80 0x7ff610198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff6153c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff610100a80 0x7ff610198d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43856/0 (socket says 172.21.15.107:43856) 2024-05-09T08:40:28.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff6153c6640 1 -- 172.21.15.107:0/16224303 learned_addr learned my addr 172.21.15.107:0/16224303 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff614bc5640 1 --2- 172.21.15.107:0/16224303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff610101420 0x7ff6101992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff615bc7640 1 --2- 172.21.15.107:0/16224303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff610106780 0x7ff6101a3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff6153c6640 1 -- 172.21.15.107:0/16224303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff610101420 msgr2=0x7ff6101992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff6153c6640 1 --2- 172.21.15.107:0/16224303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff610101420 0x7ff6101992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff6153c6640 1 -- 172.21.15.107:0/16224303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff610106780 msgr2=0x7ff6101a3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff6153c6640 1 --2- 172.21.15.107:0/16224303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff610106780 0x7ff6101a3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff6153c6640 1 -- 172.21.15.107:0/16224303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6101a4340 con 0x7ff610100a80 2024-05-09T08:40:28.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.588+0000 7ff615bc7640 1 --2- 172.21.15.107:0/16224303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff610106780 0x7ff6101a3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:40:28.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.589+0000 7ff6153c6640 1 --2- 172.21.15.107:0/16224303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff610100a80 0x7ff610198d80 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7ff60c005940 tx=0x7ff60c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.589+0000 7ff6067fc640 1 -- 172.21.15.107:0/16224303 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60c03f030 con 0x7ff610100a80 2024-05-09T08:40:28.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.589+0000 7ff6067fc640 1 -- 172.21.15.107:0/16224303 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff60c003aa0 con 0x7ff610100a80 2024-05-09T08:40:28.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.589+0000 7ff61762a640 1 -- 172.21.15.107:0/16224303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6101a4510 con 0x7ff610100a80 2024-05-09T08:40:28.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.589+0000 7ff6067fc640 1 -- 172.21.15.107:0/16224303 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60c035070 con 0x7ff610100a80 2024-05-09T08:40:28.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.589+0000 7ff61762a640 1 -- 172.21.15.107:0/16224303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6101a4970 con 0x7ff610100a80 2024-05-09T08:40:28.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.591+0000 7ff6067fc640 1 -- 172.21.15.107:0/16224303 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ff60c042030 con 0x7ff610100a80 2024-05-09T08:40:28.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.592+0000 7ff61a412740 1 Processor -- start 2024-05-09T08:40:28.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.592+0000 7ff61a412740 1 -- start start 2024-05-09T08:40:28.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.592+0000 7ff61a412740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4c330 0x557220b4c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.592+0000 7ff61a412740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b4cc50 0x557220b56de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.592+0000 7ff61a412740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b57320 0x557220b59710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.592+0000 7ff61a412740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557220ab1c40 con 0x557220b4c330 2024-05-09T08:40:28.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.592+0000 7ff61a412740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557220a59580 con 0x557220b4cc50 2024-05-09T08:40:28.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.592+0000 7ff61a412740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x557220aadc00 con 0x557220b57320 2024-05-09T08:40:28.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.592+0000 7ff6067fc640 1 --2- 172.21.15.107:0/16224303 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff5fc0a2d00 0x7ff5fc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.592+0000 7ff615bc7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4c330 0x557220b4c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.592+0000 7ff615bc7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4c330 0x557220b4c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43864/0 (socket says 172.21.15.107:43864) 2024-05-09T08:40:28.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.592+0000 7ff615bc7640 1 -- 172.21.15.107:0/3615868112 learned_addr learned my addr 172.21.15.107:0/3615868112 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.592+0000 7ff614bc5640 1 --2- 172.21.15.107:0/3615868112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b57320 0x557220b59710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.592+0000 7ff615bc7640 1 --2- 172.21.15.107:0/16224303 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff5fc0a2d00 0x7ff5fc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.593+0000 7ff6067fc640 1 -- 172.21.15.107:0/16224303 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7ff60c0efc40 con 0x7ff610100a80 2024-05-09T08:40:28.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.593+0000 7ff6153c6640 1 --2- 172.21.15.107:0/3615868112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b4cc50 0x557220b56de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.593+0000 7ff614bc5640 1 -- 172.21.15.107:0/3615868112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b4cc50 msgr2=0x557220b56de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.593+0000 7ff614bc5640 1 --2- 172.21.15.107:0/3615868112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b4cc50 0x557220b56de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.593+0000 7ff614bc5640 1 -- 172.21.15.107:0/3615868112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4c330 msgr2=0x557220b4c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.593+0000 7ff614bc5640 1 --2- 172.21.15.107:0/3615868112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4c330 0x557220b4c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.593+0000 7ff614bc5640 1 -- 172.21.15.107:0/3615868112 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572207c53f0 con 0x557220b57320 2024-05-09T08:40:28.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.593+0000 7ff615bc7640 1 --2- 172.21.15.107:0/16224303 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff5fc0a2d00 0x7ff5fc0a51c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff608007fd0 tx=0x7ff608004ab0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.593+0000 7ff614bc5640 1 --2- 172.21.15.107:0/3615868112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b57320 0x557220b59710 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff600005fd0 tx=0x7ff60000bb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6840c15b45e3d4d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.593+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3615868112 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600003bd0 con 0x557220b57320 2024-05-09T08:40:28.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.593+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3615868112 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff600015070 con 0x557220b57320 2024-05-09T08:40:28.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.593+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3615868112 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600004c40 con 0x557220b57320 2024-05-09T08:40:28.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.594+0000 7ff61a412740 1 -- 172.21.15.107:0/3615868112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b57320 msgr2=0x557220b59710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.594+0000 7ff61a412740 1 --2- 172.21.15.107:0/3615868112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b57320 0x557220b59710 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff600005fd0 tx=0x7ff60000bb20 comp rx=0 tx=0).stop 2024-05-09T08:40:28.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.594+0000 7ff61a412740 1 -- 172.21.15.107:0/3615868112 shutdown_connections 2024-05-09T08:40:28.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.594+0000 7ff61a412740 1 --2- 172.21.15.107:0/3615868112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b57320 0x557220b59710 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.594+0000 7ff61a412740 1 --2- 172.21.15.107:0/3615868112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b4cc50 0x557220b56de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.594+0000 7ff61a412740 1 --2- 172.21.15.107:0/3615868112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4c330 0x557220b4c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.594+0000 7ff61a412740 1 -- 172.21.15.107:0/3615868112 >> 172.21.15.107:0/3615868112 conn(0x557220b48650 msgr2=0x557220b48a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.594+0000 7ff61a412740 1 -- 172.21.15.107:0/3615868112 shutdown_connections 2024-05-09T08:40:28.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.594+0000 7ff61a412740 1 -- 172.21.15.107:0/3615868112 wait complete. 2024-05-09T08:40:28.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.594+0000 7ff61a412740 1 -- start start 2024-05-09T08:40:28.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff61a412740 1 Processor -- start 2024-05-09T08:40:28.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff61a412740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b4c330 0x557220b54240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff61a412740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b4cc50 0x557220b54780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff61a412740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b57320 0x557220b54cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff61a412740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557220b5a110 con 0x557220b57320 2024-05-09T08:40:28.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff61a412740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557220a6cc40 con 0x557220b4c330 2024-05-09T08:40:28.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff61a412740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x557220b5a290 con 0x557220b4cc50 2024-05-09T08:40:28.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff61a412740 10 client.0 _ll_register_callbacks cb 0x5572207c4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:28.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff615bc7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b57320 0x557220b54cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff615bc7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b57320 0x557220b54cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43866/0 (socket says 172.21.15.107:43866) 2024-05-09T08:40:28.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff615bc7640 1 -- 172.21.15.107:0/1584121206 learned_addr learned my addr 172.21.15.107:0/1584121206 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff614bc5640 1 --2- 172.21.15.107:0/1584121206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b4cc50 0x557220b54780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff6153c6640 1 --2- 172.21.15.107:0/1584121206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b4c330 0x557220b54240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff615bc7640 1 -- 172.21.15.107:0/1584121206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b4cc50 msgr2=0x557220b54780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff615bc7640 1 --2- 172.21.15.107:0/1584121206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b4cc50 0x557220b54780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff615bc7640 1 -- 172.21.15.107:0/1584121206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b4c330 msgr2=0x557220b54240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff615bc7640 1 --2- 172.21.15.107:0/1584121206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b4c330 0x557220b54240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.595+0000 7ff615bc7640 1 -- 172.21.15.107:0/1584121206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55722088f7d0 con 0x557220b57320 2024-05-09T08:40:28.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.596+0000 7ff615bc7640 1 --2- 172.21.15.107:0/1584121206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b57320 0x557220b54cc0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7ff6080073c0 tx=0x7ff608004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a23cfc331ab1b359 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.596+0000 7ff5f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:28.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.596+0000 7ff5f8ff9640 1 -- 172.21.15.107:0/1584121206 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff608004b20 con 0x557220b57320 2024-05-09T08:40:28.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.596+0000 7ff5f8ff9640 1 -- 172.21.15.107:0/1584121206 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7ff608004cc0 con 0x557220b57320 2024-05-09T08:40:28.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.596+0000 7ff61a412740 1 -- 172.21.15.107:0/1584121206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff600015070 con 0x557220b57320 2024-05-09T08:40:28.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.596+0000 7ff5f8ff9640 1 -- 172.21.15.107:0/1584121206 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff608031050 con 0x557220b57320 2024-05-09T08:40:28.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.596+0000 7ff61a412740 10 client.17721 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:28.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.596+0000 7ff61a412740 1 -- 172.21.15.107:0/1584121206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557220b66330 con 0x557220b57320 2024-05-09T08:40:28.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.596+0000 7ff5f8ff9640 1 -- 172.21.15.107:0/1584121206 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff600015070 con 0x557220b57320 2024-05-09T08:40:28.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.596+0000 7ff5f8ff9640 1 -- 172.21.15.107:0/1584121206 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff60801c040 con 0x557220b57320 2024-05-09T08:40:28.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.597+0000 7ff61a412740 10 client.17721 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:28.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.597+0000 7ff61a412740 10 client.17721 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:40:28.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.597+0000 7ff61a412740 20 client.17721 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:28.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.597+0000 7ff61a412740 1 --2- 172.21.15.107:0/1584121206 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x557220b677f0 0x557220b69c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.597+0000 7ff61a412740 4 client.17721 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:40:28.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.597+0000 7ff61a412740 1 -- 172.21.15.107:0/1584121206 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557220abf370 con 0x557220b677f0 2024-05-09T08:40:28.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.597+0000 7ff6153c6640 1 --2- 172.21.15.107:0/1584121206 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x557220b677f0 0x557220b69c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.597+0000 7ff5f8ff9640 10 client.17721 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:40:28.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.597+0000 7ff6153c6640 1 --2- 172.21.15.107:0/1584121206 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x557220b677f0 0x557220b69c20 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=429de6778e950ff9 server_cookie=fa201b3fbb3d7334 in_seq=0 out_seq=0 2024-05-09T08:40:28.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.598+0000 7ff5f8ff9640 1 -- 172.21.15.107:0/1584121206 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557220abf370 con 0x557220b677f0 2024-05-09T08:40:28.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.598+0000 7ff5f8ff9640 10 client.17721 handle_command_reply: tid=0 2024-05-09T08:40:28.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.598+0000 7ff61a412740 1 client.17721 shutdown 2024-05-09T08:40:28.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.598+0000 7ff61a412740 1 -- 172.21.15.107:0/1584121206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b57320 msgr2=0x557220b54cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.598+0000 7ff61a412740 1 --2- 172.21.15.107:0/1584121206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b57320 0x557220b54cc0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7ff6080073c0 tx=0x7ff608004a40 comp rx=0 tx=0).stop 2024-05-09T08:40:28.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.598+0000 7ff61a412740 1 -- 172.21.15.107:0/1584121206 shutdown_connections 2024-05-09T08:40:28.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.598+0000 7ff61a412740 1 --2- 172.21.15.107:0/1584121206 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x557220b677f0 0x557220b69c20 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.598+0000 7ff61a412740 1 --2- 172.21.15.107:0/1584121206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b57320 0x557220b54cc0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.598+0000 7ff61a412740 1 --2- 172.21.15.107:0/1584121206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b4cc50 0x557220b54780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.598+0000 7ff5f8ff9640 0 client.17721 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:40:28.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.598+0000 7ff61a412740 1 --2- 172.21.15.107:0/1584121206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b4c330 0x557220b54240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.598+0000 7ff61a412740 1 -- 172.21.15.107:0/1584121206 >> 172.21.15.107:0/1584121206 conn(0x557220b48650 msgr2=0x557220b4e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.598+0000 7ff61a412740 1 -- 172.21.15.107:0/1584121206 shutdown_connections 2024-05-09T08:40:28.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.598+0000 7ff61a412740 1 -- 172.21.15.107:0/1584121206 wait complete. 2024-05-09T08:40:28.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.598+0000 7ff61a412740 20 client.17721 trim_cache size 0 max 16384 2024-05-09T08:40:28.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.614+0000 7ff61a412740 1 Processor -- start 2024-05-09T08:40:28.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.614+0000 7ff61a412740 1 -- start start 2024-05-09T08:40:28.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.614+0000 7ff61a412740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b4c330 0x557220b63000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.614+0000 7ff61a412740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4cc50 0x557220b4ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.614+0000 7ff61a412740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b57320 0x557220b6f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.614+0000 7ff61a412740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101b13b0 con 0x557220b4cc50 2024-05-09T08:40:28.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.614+0000 7ff61a412740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101aec80 con 0x557220b4c330 2024-05-09T08:40:28.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.614+0000 7ff61a412740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101acaf0 con 0x557220b57320 2024-05-09T08:40:28.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.615+0000 7ff614bc5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4cc50 0x557220b4ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.615+0000 7ff614bc5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4cc50 0x557220b4ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43882/0 (socket says 172.21.15.107:43882) 2024-05-09T08:40:28.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.615+0000 7ff614bc5640 1 -- 172.21.15.107:0/4126464500 learned_addr learned my addr 172.21.15.107:0/4126464500 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.615+0000 7ff615bc7640 1 --2- 172.21.15.107:0/4126464500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b57320 0x557220b6f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.615+0000 7ff6153c6640 1 --2- 172.21.15.107:0/4126464500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b4c330 0x557220b63000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.615+0000 7ff614bc5640 1 -- 172.21.15.107:0/4126464500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b57320 msgr2=0x557220b6f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.615+0000 7ff614bc5640 1 --2- 172.21.15.107:0/4126464500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b57320 0x557220b6f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.615+0000 7ff614bc5640 1 -- 172.21.15.107:0/4126464500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b4c330 msgr2=0x557220b63000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.615+0000 7ff614bc5640 1 --2- 172.21.15.107:0/4126464500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b4c330 0x557220b63000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.615+0000 7ff614bc5640 1 -- 172.21.15.107:0/4126464500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557220b4d030 con 0x557220b4cc50 2024-05-09T08:40:28.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.615+0000 7ff614bc5640 1 --2- 172.21.15.107:0/4126464500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4cc50 0x557220b4ea90 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7ff60000b620 tx=0x7ff600005d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beffcbfa9b5538aa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.615+0000 7ff5faffd640 1 -- 172.21.15.107:0/4126464500 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600029070 con 0x557220b4cc50 2024-05-09T08:40:28.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.615+0000 7ff5faffd640 1 -- 172.21.15.107:0/4126464500 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff600015050 con 0x557220b4cc50 2024-05-09T08:40:28.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.615+0000 7ff5faffd640 1 -- 172.21.15.107:0/4126464500 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600024d20 con 0x557220b4cc50 2024-05-09T08:40:28.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.616+0000 7ff61a412740 1 -- 172.21.15.107:0/4126464500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4cc50 msgr2=0x557220b4ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.616+0000 7ff61a412740 1 --2- 172.21.15.107:0/4126464500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4cc50 0x557220b4ea90 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7ff60000b620 tx=0x7ff600005d20 comp rx=0 tx=0).stop 2024-05-09T08:40:28.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.616+0000 7ff61a412740 1 -- 172.21.15.107:0/4126464500 shutdown_connections 2024-05-09T08:40:28.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.616+0000 7ff61a412740 1 --2- 172.21.15.107:0/4126464500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b57320 0x557220b6f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.616+0000 7ff61a412740 1 --2- 172.21.15.107:0/4126464500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4cc50 0x557220b4ea90 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.616+0000 7ff61a412740 1 --2- 172.21.15.107:0/4126464500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b4c330 0x557220b63000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.616+0000 7ff61a412740 1 -- 172.21.15.107:0/4126464500 >> 172.21.15.107:0/4126464500 conn(0x557220b48650 msgr2=0x557220b477b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.616+0000 7ff61a412740 1 -- 172.21.15.107:0/4126464500 shutdown_connections 2024-05-09T08:40:28.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.616+0000 7ff61a412740 1 -- 172.21.15.107:0/4126464500 wait complete. 2024-05-09T08:40:28.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.616+0000 7ff61a412740 1 -- start start 2024-05-09T08:40:28.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.616+0000 7ff61a412740 1 Processor -- start 2024-05-09T08:40:28.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff61a412740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b4c330 0x557220b5fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff61a412740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4cc50 0x557220b50d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff61a412740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b57320 0x557220b51470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff61a412740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101ad140 con 0x557220b4cc50 2024-05-09T08:40:28.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff61a412740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101ae5b0 con 0x557220b57320 2024-05-09T08:40:28.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff61a412740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101ad810 con 0x557220b4c330 2024-05-09T08:40:28.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff61a412740 10 client.0 _ll_register_callbacks cb 0x7ff608010d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:28.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff614bc5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4cc50 0x557220b50d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff614bc5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4cc50 0x557220b50d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43888/0 (socket says 172.21.15.107:43888) 2024-05-09T08:40:28.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff614bc5640 1 -- 172.21.15.107:0/2204274721 learned_addr learned my addr 172.21.15.107:0/2204274721 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff6153c6640 1 --2- 172.21.15.107:0/2204274721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b4c330 0x557220b5fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff615bc7640 1 --2- 172.21.15.107:0/2204274721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b57320 0x557220b51470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff614bc5640 1 -- 172.21.15.107:0/2204274721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b4c330 msgr2=0x557220b5fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff614bc5640 1 --2- 172.21.15.107:0/2204274721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b4c330 0x557220b5fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff614bc5640 1 -- 172.21.15.107:0/2204274721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b57320 msgr2=0x557220b51470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff614bc5640 1 --2- 172.21.15.107:0/2204274721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b57320 0x557220b51470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.617+0000 7ff614bc5640 1 -- 172.21.15.107:0/2204274721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557220bccd60 con 0x557220b4cc50 2024-05-09T08:40:28.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.618+0000 7ff614bc5640 1 --2- 172.21.15.107:0/2204274721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4cc50 0x557220b50d30 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ff600024980 tx=0x7ff6000043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=215ab44a8e7ae28c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.618+0000 7ff5f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:28.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.618+0000 7ff5f8ff9640 1 -- 172.21.15.107:0/2204274721 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600032060 con 0x557220b4cc50 2024-05-09T08:40:28.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.618+0000 7ff5f8ff9640 1 -- 172.21.15.107:0/2204274721 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7ff600024a20 con 0x557220b4cc50 2024-05-09T08:40:28.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.618+0000 7ff61a412740 1 -- 172.21.15.107:0/2204274721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff600015050 con 0x557220b4cc50 2024-05-09T08:40:28.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.618+0000 7ff5f8ff9640 1 -- 172.21.15.107:0/2204274721 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff600039050 con 0x557220b4cc50 2024-05-09T08:40:28.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.618+0000 7ff61a412740 10 client.17727 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:28.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.618+0000 7ff61a412740 1 -- 172.21.15.107:0/2204274721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557220bcdec0 con 0x557220b4cc50 2024-05-09T08:40:28.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.618+0000 7ff5f8ff9640 1 -- 172.21.15.107:0/2204274721 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff600015050 con 0x557220b4cc50 2024-05-09T08:40:28.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.618+0000 7ff5f8ff9640 1 -- 172.21.15.107:0/2204274721 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff60002c070 con 0x557220b4cc50 2024-05-09T08:40:28.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.618+0000 7ff61a412740 10 client.17727 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:28.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.618+0000 7ff61a412740 10 client.17727 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:40:28.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.618+0000 7ff61a412740 20 client.17727 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:28.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.618+0000 7ff61a412740 1 --2- 172.21.15.107:0/2204274721 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x557220bce320 0x557220bd0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.618+0000 7ff61a412740 4 client.17727 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:40:28.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.618+0000 7ff61a412740 1 -- 172.21.15.107:0/2204274721 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557220ab96c0 con 0x557220bce320 2024-05-09T08:40:28.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.619+0000 7ff6153c6640 1 --2- 172.21.15.107:0/2204274721 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x557220bce320 0x557220bd0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.619+0000 7ff5f8ff9640 10 client.17727 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:40:28.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.619+0000 7ff6153c6640 1 --2- 172.21.15.107:0/2204274721 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x557220bce320 0x557220bd0740 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ceadbc9225ba9cb1 server_cookie=23375007fbf1e1bc in_seq=0 out_seq=0 2024-05-09T08:40:28.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.619+0000 7ff5f8ff9640 1 -- 172.21.15.107:0/2204274721 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557220ab96c0 con 0x557220bce320 2024-05-09T08:40:28.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.619+0000 7ff5f8ff9640 10 client.17727 handle_command_reply: tid=0 2024-05-09T08:40:28.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.619+0000 7ff61a412740 1 client.17727 shutdown 2024-05-09T08:40:28.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.619+0000 7ff61a412740 1 -- 172.21.15.107:0/2204274721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4cc50 msgr2=0x557220b50d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.619+0000 7ff61a412740 1 --2- 172.21.15.107:0/2204274721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4cc50 0x557220b50d30 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ff600024980 tx=0x7ff6000043b0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61a412740 1 -- 172.21.15.107:0/2204274721 shutdown_connections 2024-05-09T08:40:28.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61a412740 1 --2- 172.21.15.107:0/2204274721 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x557220bce320 0x557220bd0740 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61a412740 1 --2- 172.21.15.107:0/2204274721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557220b57320 0x557220b51470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61a412740 1 --2- 172.21.15.107:0/2204274721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557220b4cc50 0x557220b50d30 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61a412740 1 --2- 172.21.15.107:0/2204274721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557220b4c330 0x557220b5fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61a412740 1 -- 172.21.15.107:0/2204274721 >> 172.21.15.107:0/2204274721 conn(0x557220b48650 msgr2=0x557220b54200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff5f8ff9640 0 client.17727 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:40:28.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61a412740 1 -- 172.21.15.107:0/2204274721 shutdown_connections 2024-05-09T08:40:28.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61a412740 1 -- 172.21.15.107:0/2204274721 wait complete. 2024-05-09T08:40:28.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61a412740 20 client.17727 trim_cache size 0 max 16384 2024-05-09T08:40:28.627 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:40:28.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61762a640 1 -- 172.21.15.107:0/16224303 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff5fc0a2d00 msgr2=0x7ff5fc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61762a640 1 --2- 172.21.15.107:0/16224303 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff5fc0a2d00 0x7ff5fc0a51c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff608007fd0 tx=0x7ff608004ab0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61762a640 1 -- 172.21.15.107:0/16224303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff610100a80 msgr2=0x7ff610198d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61762a640 1 --2- 172.21.15.107:0/16224303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff610100a80 0x7ff610198d80 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7ff60c005940 tx=0x7ff60c004270 comp rx=0 tx=0).stop 2024-05-09T08:40:28.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61762a640 1 -- 172.21.15.107:0/16224303 shutdown_connections 2024-05-09T08:40:28.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61762a640 1 --2- 172.21.15.107:0/16224303 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff5fc0a2d00 0x7ff5fc0a51c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61762a640 1 --2- 172.21.15.107:0/16224303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff610106780 0x7ff6101a3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61762a640 1 --2- 172.21.15.107:0/16224303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff610101420 0x7ff6101992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61762a640 1 --2- 172.21.15.107:0/16224303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff610100a80 0x7ff610198d80 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61762a640 1 -- 172.21.15.107:0/16224303 >> 172.21.15.107:0/16224303 conn(0x7ff610076530 msgr2=0x7ff6100ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.620+0000 7ff61762a640 1 -- 172.21.15.107:0/16224303 shutdown_connections 2024-05-09T08:40:28.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.621+0000 7ff61762a640 1 -- 172.21.15.107:0/16224303 wait complete. 2024-05-09T08:40:28.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:40:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.750+0000 7f4a8e3d5640 1 -- 172.21.15.107:0/3081267603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a881051b0 msgr2=0x7f4a8810cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.750+0000 7f4a8e3d5640 1 --2- 172.21.15.107:0/3081267603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a881051b0 0x7f4a8810cd80 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f4a80009ca0 tx=0x7f4a8002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.750+0000 7f4a8e3d5640 1 -- 172.21.15.107:0/3081267603 shutdown_connections 2024-05-09T08:40:28.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.750+0000 7f4a8e3d5640 1 --2- 172.21.15.107:0/3081267603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a881051b0 0x7f4a8810cd80 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.750+0000 7f4a8e3d5640 1 --2- 172.21.15.107:0/3081267603 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a88100a90 0x7f4a88104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.750+0000 7f4a8e3d5640 1 --2- 172.21.15.107:0/3081267603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a881000e0 0x7f4a881004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.750+0000 7f4a8e3d5640 1 -- 172.21.15.107:0/3081267603 >> 172.21.15.107:0/3081267603 conn(0x7f4a88076cd0 msgr2=0x7f4a880fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.750+0000 7f4a8e3d5640 1 -- 172.21.15.107:0/3081267603 shutdown_connections 2024-05-09T08:40:28.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.750+0000 7f4a8e3d5640 1 -- 172.21.15.107:0/3081267603 wait complete. 2024-05-09T08:40:28.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.751+0000 7f4a8e3d5640 1 Processor -- start 2024-05-09T08:40:28.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.751+0000 7f4a8e3d5640 1 -- start start 2024-05-09T08:40:28.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.751+0000 7f4a8e3d5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a881000e0 0x7f4a88197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.751+0000 7f4a8e3d5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a881051b0 0x7f4a881984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.751+0000 7f4a8e3d5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8819c870 0x7f4a8819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.751+0000 7f4a8e3d5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4a88111ab0 con 0x7f4a881000e0 2024-05-09T08:40:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.751+0000 7f4a8e3d5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4a88111930 con 0x7f4a8819c870 2024-05-09T08:40:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.751+0000 7f4a8e3d5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4a88111c30 con 0x7f4a881051b0 2024-05-09T08:40:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.751+0000 7f4a87fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a881000e0 0x7f4a88197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.752+0000 7f4a87fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a881000e0 0x7f4a88197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43906/0 (socket says 172.21.15.107:43906) 2024-05-09T08:40:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.752+0000 7f4a87fff640 1 -- 172.21.15.107:0/1288174295 learned_addr learned my addr 172.21.15.107:0/1288174295 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.752+0000 7f4a8c972640 1 --2- 172.21.15.107:0/1288174295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8819c870 0x7f4a8819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.752+0000 7f4a87fff640 1 -- 172.21.15.107:0/1288174295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a881051b0 msgr2=0x7f4a881984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.752+0000 7f4a87fff640 1 --2- 172.21.15.107:0/1288174295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a881051b0 0x7f4a881984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.752+0000 7f4a87fff640 1 -- 172.21.15.107:0/1288174295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8819c870 msgr2=0x7f4a8819cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.752+0000 7f4a87fff640 1 --2- 172.21.15.107:0/1288174295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8819c870 0x7f4a8819cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.752+0000 7f4a87fff640 1 -- 172.21.15.107:0/1288174295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8819d360 con 0x7f4a881000e0 2024-05-09T08:40:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.752+0000 7f4a8c972640 1 --2- 172.21.15.107:0/1288174295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8819c870 0x7f4a8819cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:40:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.752+0000 7f4a87fff640 1 --2- 172.21.15.107:0/1288174295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a881000e0 0x7f4a88197fb0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800c920 tx=0x7f4a7800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.752+0000 7f4a857fa640 1 -- 172.21.15.107:0/1288174295 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a78007c70 con 0x7f4a881000e0 2024-05-09T08:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.752+0000 7f4a857fa640 1 -- 172.21.15.107:0/1288174295 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a78007e10 con 0x7f4a881000e0 2024-05-09T08:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.752+0000 7f4a8e3d5640 1 -- 172.21.15.107:0/1288174295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a8819d650 con 0x7f4a881000e0 2024-05-09T08:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.752+0000 7f4a857fa640 1 -- 172.21.15.107:0/1288174295 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a78018820 con 0x7f4a881000e0 2024-05-09T08:40:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.752+0000 7f4a8e3d5640 1 -- 172.21.15.107:0/1288174295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a881a4e90 con 0x7f4a881000e0 2024-05-09T08:40:28.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.755+0000 7f4a911be740 1 Processor -- start 2024-05-09T08:40:28.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.755+0000 7f4a911be740 1 -- start start 2024-05-09T08:40:28.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.755+0000 7f4a857fa640 1 -- 172.21.15.107:0/1288174295 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f4a780189c0 con 0x7f4a881000e0 2024-05-09T08:40:28.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.755+0000 7f4a911be740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182da330 0x55b8182da710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.755+0000 7f4a911be740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182dac50 0x55b8182e4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.755+0000 7f4a911be740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182e5320 0x55b8182e7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.755+0000 7f4a911be740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b81823fc40 con 0x55b8182dac50 2024-05-09T08:40:28.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.755+0000 7f4a911be740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b8181e7580 con 0x55b8182da330 2024-05-09T08:40:28.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.755+0000 7f4a911be740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55b81823bc00 con 0x55b8182e5320 2024-05-09T08:40:28.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.755+0000 7f4a87fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182dac50 0x55b8182e4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.755+0000 7f4a87fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182dac50 0x55b8182e4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43920/0 (socket says 172.21.15.107:43920) 2024-05-09T08:40:28.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.755+0000 7f4a87fff640 1 -- 172.21.15.107:0/3871524861 learned_addr learned my addr 172.21.15.107:0/3871524861 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.755+0000 7f4a877fe640 1 --2- 172.21.15.107:0/3871524861 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182e5320 0x55b8182e7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.755+0000 7f4a8c972640 1 --2- 172.21.15.107:0/3871524861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182da330 0x55b8182da710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.756+0000 7f4a87fff640 1 -- 172.21.15.107:0/3871524861 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182e5320 msgr2=0x55b8182e7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.756+0000 7f4a87fff640 1 --2- 172.21.15.107:0/3871524861 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182e5320 0x55b8182e7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.756+0000 7f4a87fff640 1 -- 172.21.15.107:0/3871524861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182da330 msgr2=0x55b8182da710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.756+0000 7f4a87fff640 1 --2- 172.21.15.107:0/3871524861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182da330 0x55b8182da710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.756+0000 7f4a87fff640 1 -- 172.21.15.107:0/3871524861 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b817f533f0 con 0x55b8182dac50 2024-05-09T08:40:28.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.756+0000 7f4a87fff640 1 --2- 172.21.15.107:0/3871524861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182dac50 0x55b8182e4de0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f4a780cccb0 tx=0x7f4a780d5db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47dfd1c44f0c45a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.756+0000 7f4a857fa640 1 --2- 172.21.15.107:0/1288174295 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4a700a2d80 0x7f4a700a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.756+0000 7f4a6e7fc640 1 -- 172.21.15.107:0/3871524861 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a78097db0 con 0x55b8182dac50 2024-05-09T08:40:28.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.756+0000 7f4a8c972640 1 --2- 172.21.15.107:0/1288174295 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4a700a2d80 0x7f4a700a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.756+0000 7f4a6e7fc640 1 -- 172.21.15.107:0/3871524861 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a780dc070 con 0x55b8182dac50 2024-05-09T08:40:28.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.756+0000 7f4a6e7fc640 1 -- 172.21.15.107:0/3871524861 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7809af60 con 0x55b8182dac50 2024-05-09T08:40:28.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.756+0000 7f4a857fa640 1 -- 172.21.15.107:0/1288174295 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f4a780cf9a0 con 0x7f4a881000e0 2024-05-09T08:40:28.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.757+0000 7f4a8c972640 1 --2- 172.21.15.107:0/1288174295 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4a700a2d80 0x7f4a700a5240 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4a80005e30 tx=0x7f4a80005f80 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.757+0000 7f4a911be740 1 -- 172.21.15.107:0/3871524861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182dac50 msgr2=0x55b8182e4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.757+0000 7f4a911be740 1 --2- 172.21.15.107:0/3871524861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182dac50 0x55b8182e4de0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f4a780cccb0 tx=0x7f4a780d5db0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.757+0000 7f4a911be740 1 -- 172.21.15.107:0/3871524861 shutdown_connections 2024-05-09T08:40:28.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.757+0000 7f4a911be740 1 --2- 172.21.15.107:0/3871524861 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182e5320 0x55b8182e7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.757+0000 7f4a911be740 1 --2- 172.21.15.107:0/3871524861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182dac50 0x55b8182e4de0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.757+0000 7f4a911be740 1 --2- 172.21.15.107:0/3871524861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182da330 0x55b8182da710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.757+0000 7f4a911be740 1 -- 172.21.15.107:0/3871524861 >> 172.21.15.107:0/3871524861 conn(0x55b8182d6650 msgr2=0x55b8182d6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.757+0000 7f4a911be740 1 -- 172.21.15.107:0/3871524861 shutdown_connections 2024-05-09T08:40:28.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.757+0000 7f4a911be740 1 -- 172.21.15.107:0/3871524861 wait complete. 2024-05-09T08:40:28.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.757+0000 7f4a911be740 1 -- start start 2024-05-09T08:40:28.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a911be740 1 Processor -- start 2024-05-09T08:40:28.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a911be740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182da330 0x55b8182e21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a911be740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182dac50 0x55b8182e2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a911be740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182e5320 0x55b8182e2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a911be740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b8182e8110 con 0x55b8182dac50 2024-05-09T08:40:28.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a911be740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b8181fac40 con 0x55b8182e5320 2024-05-09T08:40:28.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a911be740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55b8182e8290 con 0x55b8182da330 2024-05-09T08:40:28.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a911be740 10 client.0 _ll_register_callbacks cb 0x55b817f52be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:28.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a877fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182dac50 0x55b8182e2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a877fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182dac50 0x55b8182e2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43936/0 (socket says 172.21.15.107:43936) 2024-05-09T08:40:28.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a877fe640 1 -- 172.21.15.107:0/2229371030 learned_addr learned my addr 172.21.15.107:0/2229371030 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a8c972640 1 --2- 172.21.15.107:0/2229371030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182e5320 0x55b8182e2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a87fff640 1 --2- 172.21.15.107:0/2229371030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182da330 0x55b8182e21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a877fe640 1 -- 172.21.15.107:0/2229371030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182da330 msgr2=0x55b8182e21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a877fe640 1 --2- 172.21.15.107:0/2229371030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182da330 0x55b8182e21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a877fe640 1 -- 172.21.15.107:0/2229371030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182e5320 msgr2=0x55b8182e2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a877fe640 1 --2- 172.21.15.107:0/2229371030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182e5320 0x55b8182e2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a877fe640 1 -- 172.21.15.107:0/2229371030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b81801d7d0 con 0x55b8182dac50 2024-05-09T08:40:28.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.758+0000 7f4a87fff640 1 --2- 172.21.15.107:0/2229371030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182da330 0x55b8182e21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:40:28.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.759+0000 7f4a877fe640 1 --2- 172.21.15.107:0/2229371030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182dac50 0x55b8182e2720 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00b880 tx=0x7f4a7c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3b06fc6fa368e27 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.759+0000 7f4a677fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:28.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.759+0000 7f4a677fe640 1 -- 172.21.15.107:0/2229371030 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c01a470 con 0x55b8182dac50 2024-05-09T08:40:28.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.759+0000 7f4a677fe640 1 -- 172.21.15.107:0/2229371030 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f4a7c01a610 con 0x55b8182dac50 2024-05-09T08:40:28.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.759+0000 7f4a911be740 1 -- 172.21.15.107:0/2229371030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a780dc070 con 0x55b8182dac50 2024-05-09T08:40:28.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.759+0000 7f4a677fe640 1 -- 172.21.15.107:0/2229371030 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4a7c023050 con 0x55b8182dac50 2024-05-09T08:40:28.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.759+0000 7f4a911be740 10 client.17739 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:28.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.759+0000 7f4a911be740 1 -- 172.21.15.107:0/2229371030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b8182f4380 con 0x55b8182dac50 2024-05-09T08:40:28.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.759+0000 7f4a677fe640 1 -- 172.21.15.107:0/2229371030 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a780dc070 con 0x55b8182dac50 2024-05-09T08:40:28.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.759+0000 7f4a677fe640 1 -- 172.21.15.107:0/2229371030 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f4a7c011970 con 0x55b8182dac50 2024-05-09T08:40:28.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.759+0000 7f4a911be740 10 client.17739 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:28.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.759+0000 7f4a911be740 10 client.17739 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:40:28.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.759+0000 7f4a911be740 20 client.17739 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:28.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.759+0000 7f4a911be740 1 --2- 172.21.15.107:0/2229371030 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55b8182f5840 0x55b8182f7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.759+0000 7f4a911be740 4 client.17739 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:40:28.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.759+0000 7f4a911be740 1 -- 172.21.15.107:0/2229371030 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b81824d370 con 0x55b8182f5840 2024-05-09T08:40:28.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.760+0000 7f4a87fff640 1 --2- 172.21.15.107:0/2229371030 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55b8182f5840 0x55b8182f7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.760+0000 7f4a87fff640 1 --2- 172.21.15.107:0/2229371030 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55b8182f5840 0x55b8182f7cb0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=979f2329c3010f36 server_cookie=23a72b3e5973e1ca in_seq=0 out_seq=0 2024-05-09T08:40:28.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.760+0000 7f4a677fe640 10 client.17739 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:40:28.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.761+0000 7f4a677fe640 1 -- 172.21.15.107:0/2229371030 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b81824d370 con 0x55b8182f5840 2024-05-09T08:40:28.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.761+0000 7f4a677fe640 10 client.17739 handle_command_reply: tid=0 2024-05-09T08:40:28.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.761+0000 7f4a911be740 1 client.17739 shutdown 2024-05-09T08:40:28.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.761+0000 7f4a911be740 1 -- 172.21.15.107:0/2229371030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182dac50 msgr2=0x55b8182e2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.761+0000 7f4a911be740 1 --2- 172.21.15.107:0/2229371030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182dac50 0x55b8182e2720 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00b880 tx=0x7f4a7c00bd40 comp rx=0 tx=0).stop 2024-05-09T08:40:28.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.761+0000 7f4a911be740 1 -- 172.21.15.107:0/2229371030 shutdown_connections 2024-05-09T08:40:28.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.761+0000 7f4a911be740 1 --2- 172.21.15.107:0/2229371030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182e5320 0x55b8182e2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.761+0000 7f4a911be740 1 --2- 172.21.15.107:0/2229371030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182dac50 0x55b8182e2720 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.761+0000 7f4a911be740 1 --2- 172.21.15.107:0/2229371030 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55b8182f5840 0x55b8182f7cb0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.761+0000 7f4a911be740 1 --2- 172.21.15.107:0/2229371030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182da330 0x55b8182e21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.761+0000 7f4a677fe640 0 client.17739 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:40:28.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.761+0000 7f4a911be740 1 -- 172.21.15.107:0/2229371030 >> 172.21.15.107:0/2229371030 conn(0x55b8182d6650 msgr2=0x55b8182d57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.761+0000 7f4a911be740 1 -- 172.21.15.107:0/2229371030 shutdown_connections 2024-05-09T08:40:28.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.761+0000 7f4a911be740 1 -- 172.21.15.107:0/2229371030 wait complete. 2024-05-09T08:40:28.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.761+0000 7f4a911be740 20 client.17739 trim_cache size 0 max 16384 2024-05-09T08:40:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.778+0000 7f4a911be740 1 Processor -- start 2024-05-09T08:40:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.778+0000 7f4a911be740 1 -- start start 2024-05-09T08:40:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.778+0000 7f4a911be740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182da330 0x55b8182f1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.778+0000 7f4a911be740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182dac50 0x55b8182fd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.778+0000 7f4a911be740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182e5320 0x55b8182fdb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.778+0000 7f4a911be740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4a881b1480 con 0x55b8182da330 2024-05-09T08:40:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.778+0000 7f4a911be740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4a881aed50 con 0x55b8182e5320 2024-05-09T08:40:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.778+0000 7f4a911be740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4a881acbc0 con 0x55b8182dac50 2024-05-09T08:40:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.779+0000 7f4a87fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182da330 0x55b8182f1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.779+0000 7f4a87fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182da330 0x55b8182f1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43952/0 (socket says 172.21.15.107:43952) 2024-05-09T08:40:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.779+0000 7f4a87fff640 1 -- 172.21.15.107:0/1845482745 learned_addr learned my addr 172.21.15.107:0/1845482745 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.779+0000 7f4a877fe640 1 --2- 172.21.15.107:0/1845482745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182dac50 0x55b8182fd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.779+0000 7f4a8c972640 1 --2- 172.21.15.107:0/1845482745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182e5320 0x55b8182fdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.779+0000 7f4a87fff640 1 -- 172.21.15.107:0/1845482745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182dac50 msgr2=0x55b8182fd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.779+0000 7f4a87fff640 1 --2- 172.21.15.107:0/1845482745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182dac50 0x55b8182fd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.779+0000 7f4a87fff640 1 -- 172.21.15.107:0/1845482745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182e5320 msgr2=0x55b8182fdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.779+0000 7f4a87fff640 1 --2- 172.21.15.107:0/1845482745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182e5320 0x55b8182fdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.779+0000 7f4a87fff640 1 -- 172.21.15.107:0/1845482745 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8182daa80 con 0x55b8182da330 2024-05-09T08:40:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.779+0000 7f4a877fe640 1 --2- 172.21.15.107:0/1845482745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182dac50 0x55b8182fd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:40:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.779+0000 7f4a8c972640 1 --2- 172.21.15.107:0/1845482745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182e5320 0x55b8182fdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:40:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.779+0000 7f4a87fff640 1 --2- 172.21.15.107:0/1845482745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182da330 0x55b8182f1060 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f4a78097f70 tx=0x7f4a780d5630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68c28d6dd6c4987 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.779+0000 7f4a6dffb640 1 -- 172.21.15.107:0/1845482745 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7804c070 con 0x55b8182da330 2024-05-09T08:40:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.779+0000 7f4a6dffb640 1 -- 172.21.15.107:0/1845482745 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a780dc040 con 0x55b8182da330 2024-05-09T08:40:28.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.779+0000 7f4a6dffb640 1 -- 172.21.15.107:0/1845482745 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7809b290 con 0x55b8182da330 2024-05-09T08:40:28.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:28 smithi107 ceph-mon[30122]: pgmap v218: 129 pgs: 129 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:40:28.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:28 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2721248592' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:40:28.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.780+0000 7f4a911be740 1 -- 172.21.15.107:0/1845482745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182da330 msgr2=0x55b8182f1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.780+0000 7f4a911be740 1 --2- 172.21.15.107:0/1845482745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182da330 0x55b8182f1060 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f4a78097f70 tx=0x7f4a780d5630 comp rx=0 tx=0).stop 2024-05-09T08:40:28.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.780+0000 7f4a911be740 1 -- 172.21.15.107:0/1845482745 shutdown_connections 2024-05-09T08:40:28.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.780+0000 7f4a911be740 1 --2- 172.21.15.107:0/1845482745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182e5320 0x55b8182fdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.780+0000 7f4a911be740 1 --2- 172.21.15.107:0/1845482745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182dac50 0x55b8182fd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.780+0000 7f4a911be740 1 --2- 172.21.15.107:0/1845482745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182da330 0x55b8182f1060 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.780+0000 7f4a911be740 1 -- 172.21.15.107:0/1845482745 >> 172.21.15.107:0/1845482745 conn(0x55b8182d6650 msgr2=0x55b8182d6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.780+0000 7f4a911be740 1 -- 172.21.15.107:0/1845482745 shutdown_connections 2024-05-09T08:40:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.780+0000 7f4a911be740 1 -- 172.21.15.107:0/1845482745 wait complete. 2024-05-09T08:40:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.780+0000 7f4a911be740 1 -- start start 2024-05-09T08:40:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a911be740 1 Processor -- start 2024-05-09T08:40:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a911be740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182da330 0x55b8182defd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a911be740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182dac50 0x55b8182df510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a911be740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182e5320 0x55b8182dfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a911be740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4a881ad210 con 0x55b8182e5320 2024-05-09T08:40:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a911be740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4a881ae680 con 0x55b8182da330 2024-05-09T08:40:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a911be740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4a881ad8e0 con 0x55b8182dac50 2024-05-09T08:40:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a911be740 10 client.0 _ll_register_callbacks cb 0x7f4a7c004450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:28.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a8c972640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182e5320 0x55b8182dfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a8c972640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182e5320 0x55b8182dfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43956/0 (socket says 172.21.15.107:43956) 2024-05-09T08:40:28.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a8c972640 1 -- 172.21.15.107:0/1208227639 learned_addr learned my addr 172.21.15.107:0/1208227639 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a87fff640 1 --2- 172.21.15.107:0/1208227639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182da330 0x55b8182defd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a877fe640 1 --2- 172.21.15.107:0/1208227639 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182dac50 0x55b8182df510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a8c972640 1 -- 172.21.15.107:0/1208227639 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182dac50 msgr2=0x55b8182df510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a8c972640 1 --2- 172.21.15.107:0/1208227639 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182dac50 0x55b8182df510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a8c972640 1 -- 172.21.15.107:0/1208227639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182da330 msgr2=0x55b8182defd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a8c972640 1 --2- 172.21.15.107:0/1208227639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182da330 0x55b8182defd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.781+0000 7f4a8c972640 1 -- 172.21.15.107:0/1208227639 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b81835af00 con 0x55b8182e5320 2024-05-09T08:40:28.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.782+0000 7f4a8c972640 1 --2- 172.21.15.107:0/1208227639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182e5320 0x55b8182dfa50 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f4a80030840 tx=0x7f4a800095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90321d32536fa78d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.782+0000 7f4a677fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:28.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.782+0000 7f4a677fe640 1 -- 172.21.15.107:0/1208227639 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8002fdd0 con 0x55b8182e5320 2024-05-09T08:40:28.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.782+0000 7f4a677fe640 1 -- 172.21.15.107:0/1208227639 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f4a80045450 con 0x55b8182e5320 2024-05-09T08:40:28.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.782+0000 7f4a911be740 1 -- 172.21.15.107:0/1208227639 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a780dc040 con 0x55b8182e5320 2024-05-09T08:40:28.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.782+0000 7f4a677fe640 1 -- 172.21.15.107:0/1208227639 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4a80030e80 con 0x55b8182e5320 2024-05-09T08:40:28.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.782+0000 7f4a911be740 10 client.17745 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:28.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.782+0000 7f4a911be740 1 -- 172.21.15.107:0/1208227639 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b81835c200 con 0x55b8182e5320 2024-05-09T08:40:28.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.782+0000 7f4a677fe640 1 -- 172.21.15.107:0/1208227639 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a780dc040 con 0x55b8182e5320 2024-05-09T08:40:28.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.782+0000 7f4a677fe640 1 -- 172.21.15.107:0/1208227639 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f4a80043040 con 0x55b8182e5320 2024-05-09T08:40:28.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.782+0000 7f4a911be740 10 client.17745 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:28.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.782+0000 7f4a911be740 10 client.17745 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:40:28.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.782+0000 7f4a911be740 20 client.17745 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:28.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.782+0000 7f4a911be740 1 --2- 172.21.15.107:0/1208227639 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55b81835c9b0 0x55b81835edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.782+0000 7f4a911be740 4 client.17745 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:40:28.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.782+0000 7f4a911be740 1 -- 172.21.15.107:0/1208227639 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b8182476c0 con 0x55b81835c9b0 2024-05-09T08:40:28.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a87fff640 1 --2- 172.21.15.107:0/1208227639 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55b81835c9b0 0x55b81835edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a87fff640 1 --2- 172.21.15.107:0/1208227639 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55b81835c9b0 0x55b81835edd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e1f16adad97a6c8e server_cookie=2ebb85f9f7c45b80 in_seq=0 out_seq=0 2024-05-09T08:40:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a677fe640 10 client.17745 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:40:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a677fe640 1 -- 172.21.15.107:0/1208227639 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b8182476c0 con 0x55b81835c9b0 2024-05-09T08:40:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a677fe640 10 client.17745 handle_command_reply: tid=0 2024-05-09T08:40:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a911be740 1 client.17745 shutdown 2024-05-09T08:40:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a911be740 1 -- 172.21.15.107:0/1208227639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182e5320 msgr2=0x55b8182dfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a911be740 1 --2- 172.21.15.107:0/1208227639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182e5320 0x55b8182dfa50 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f4a80030840 tx=0x7f4a800095c0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a911be740 1 -- 172.21.15.107:0/1208227639 shutdown_connections 2024-05-09T08:40:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a911be740 1 --2- 172.21.15.107:0/1208227639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8182e5320 0x55b8182dfa50 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a911be740 1 --2- 172.21.15.107:0/1208227639 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55b81835c9b0 0x55b81835edd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a911be740 1 --2- 172.21.15.107:0/1208227639 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b8182dac50 0x55b8182df510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a911be740 1 --2- 172.21.15.107:0/1208227639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b8182da330 0x55b8182defd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a911be740 1 -- 172.21.15.107:0/1208227639 >> 172.21.15.107:0/1208227639 conn(0x55b8182d6650 msgr2=0x55b8182e2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a677fe640 0 client.17745 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:40:28.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a911be740 1 -- 172.21.15.107:0/1208227639 shutdown_connections 2024-05-09T08:40:28.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.783+0000 7f4a911be740 1 -- 172.21.15.107:0/1208227639 wait complete. 2024-05-09T08:40:28.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.784+0000 7f4a911be740 20 client.17745 trim_cache size 0 max 16384 2024-05-09T08:40:28.790 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:40:28.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.784+0000 7f4a8e3d5640 1 -- 172.21.15.107:0/1288174295 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4a700a2d80 msgr2=0x7f4a700a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.784+0000 7f4a8e3d5640 1 --2- 172.21.15.107:0/1288174295 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4a700a2d80 0x7f4a700a5240 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4a80005e30 tx=0x7f4a80005f80 comp rx=0 tx=0).stop 2024-05-09T08:40:28.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.784+0000 7f4a8e3d5640 1 -- 172.21.15.107:0/1288174295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a881000e0 msgr2=0x7f4a88197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.784+0000 7f4a8e3d5640 1 --2- 172.21.15.107:0/1288174295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a881000e0 0x7f4a88197fb0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800c920 tx=0x7f4a7800cde0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.784+0000 7f4a8e3d5640 1 -- 172.21.15.107:0/1288174295 shutdown_connections 2024-05-09T08:40:28.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.784+0000 7f4a8e3d5640 1 --2- 172.21.15.107:0/1288174295 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4a700a2d80 0x7f4a700a5240 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.784+0000 7f4a8e3d5640 1 --2- 172.21.15.107:0/1288174295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8819c870 0x7f4a8819cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.784+0000 7f4a8e3d5640 1 --2- 172.21.15.107:0/1288174295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a881051b0 0x7f4a881984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.784+0000 7f4a8e3d5640 1 --2- 172.21.15.107:0/1288174295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a881000e0 0x7f4a88197fb0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.784+0000 7f4a8e3d5640 1 -- 172.21.15.107:0/1288174295 >> 172.21.15.107:0/1288174295 conn(0x7f4a88076cd0 msgr2=0x7f4a881032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.784+0000 7f4a8e3d5640 1 -- 172.21.15.107:0/1288174295 shutdown_connections 2024-05-09T08:40:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.784+0000 7f4a8e3d5640 1 -- 172.21.15.107:0/1288174295 wait complete. 2024-05-09T08:40:28.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:40:28.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:28 smithi157 ceph-mon[33614]: pgmap v218: 129 pgs: 129 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:40:28.845 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:28 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2721248592' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:40:28.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:28 smithi110 ceph-mon[33622]: pgmap v218: 129 pgs: 129 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:40:28.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:28 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2721248592' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:40:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.913+0000 7ff653c7d640 1 -- 172.21.15.107:0/3315990512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff64c106710 msgr2=0x7ff64c106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.913+0000 7ff653c7d640 1 --2- 172.21.15.107:0/3315990512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff64c106710 0x7ff64c106af0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7ff648009930 tx=0x7ff64802a420 comp rx=0 tx=0).stop 2024-05-09T08:40:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.913+0000 7ff653c7d640 1 -- 172.21.15.107:0/3315990512 shutdown_connections 2024-05-09T08:40:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.913+0000 7ff653c7d640 1 --2- 172.21.15.107:0/3315990512 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff64c1010c0 0x7ff64c10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.913+0000 7ff653c7d640 1 --2- 172.21.15.107:0/3315990512 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff64c100700 0x7ff64c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.913+0000 7ff653c7d640 1 --2- 172.21.15.107:0/3315990512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff64c106710 0x7ff64c106af0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.913+0000 7ff653c7d640 1 -- 172.21.15.107:0/3315990512 >> 172.21.15.107:0/3315990512 conn(0x7ff64c077830 msgr2=0x7ff64c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.913+0000 7ff653c7d640 1 -- 172.21.15.107:0/3315990512 shutdown_connections 2024-05-09T08:40:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.913+0000 7ff653c7d640 1 -- 172.21.15.107:0/3315990512 wait complete. 2024-05-09T08:40:28.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.914+0000 7ff653c7d640 1 Processor -- start 2024-05-09T08:40:28.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.914+0000 7ff653c7d640 1 -- start start 2024-05-09T08:40:28.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.914+0000 7ff653c7d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff64c100700 0x7ff64c198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.914+0000 7ff653c7d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff64c1010c0 0x7ff64c199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.914+0000 7ff653c7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff64c106710 0x7ff64c1a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.914+0000 7ff653c7d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c1109d0 con 0x7ff64c106710 2024-05-09T08:40:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.914+0000 7ff653c7d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c110850 con 0x7ff64c1010c0 2024-05-09T08:40:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.914+0000 7ff653c7d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c110b50 con 0x7ff64c100700 2024-05-09T08:40:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.914+0000 7ff65221a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff64c106710 0x7ff64c1a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.914+0000 7ff65221a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff64c106710 0x7ff64c1a3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43962/0 (socket says 172.21.15.107:43962) 2024-05-09T08:40:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.914+0000 7ff65221a640 1 -- 172.21.15.107:0/1988990637 learned_addr learned my addr 172.21.15.107:0/1988990637 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.914+0000 7ff651a19640 1 --2- 172.21.15.107:0/1988990637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff64c100700 0x7ff64c198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.914+0000 7ff651218640 1 --2- 172.21.15.107:0/1988990637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff64c1010c0 0x7ff64c199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.915+0000 7ff65221a640 1 -- 172.21.15.107:0/1988990637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff64c100700 msgr2=0x7ff64c198ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.915+0000 7ff65221a640 1 --2- 172.21.15.107:0/1988990637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff64c100700 0x7ff64c198ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.915+0000 7ff65221a640 1 -- 172.21.15.107:0/1988990637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff64c1010c0 msgr2=0x7ff64c199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.915+0000 7ff65221a640 1 --2- 172.21.15.107:0/1988990637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff64c1010c0 0x7ff64c199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.915+0000 7ff65221a640 1 -- 172.21.15.107:0/1988990637 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64c1a4490 con 0x7ff64c106710 2024-05-09T08:40:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.915+0000 7ff651218640 1 --2- 172.21.15.107:0/1988990637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff64c1010c0 0x7ff64c199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:40:28.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.915+0000 7ff651a19640 1 --2- 172.21.15.107:0/1988990637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff64c100700 0x7ff64c198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:40:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.915+0000 7ff65221a640 1 --2- 172.21.15.107:0/1988990637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff64c106710 0x7ff64c1a3f50 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7ff64400eaa0 tx=0x7ff64400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.915+0000 7ff642ffd640 1 -- 172.21.15.107:0/1988990637 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64400cdb0 con 0x7ff64c106710 2024-05-09T08:40:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.915+0000 7ff642ffd640 1 -- 172.21.15.107:0/1988990637 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff644005020 con 0x7ff64c106710 2024-05-09T08:40:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.915+0000 7ff653c7d640 1 -- 172.21.15.107:0/1988990637 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff64c1a4660 con 0x7ff64c106710 2024-05-09T08:40:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.915+0000 7ff642ffd640 1 -- 172.21.15.107:0/1988990637 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6440116d0 con 0x7ff64c106710 2024-05-09T08:40:28.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.915+0000 7ff653c7d640 1 -- 172.21.15.107:0/1988990637 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff64c1a4ad0 con 0x7ff64c106710 2024-05-09T08:40:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.917+0000 7ff642ffd640 1 -- 172.21.15.107:0/1988990637 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ff6440051d0 con 0x7ff64c106710 2024-05-09T08:40:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.918+0000 7ff65698c740 1 Processor -- start 2024-05-09T08:40:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.918+0000 7ff642ffd640 1 --2- 172.21.15.107:0/1988990637 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff6380a2d00 0x7ff6380a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.918+0000 7ff65698c740 1 -- start start 2024-05-09T08:40:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.918+0000 7ff651218640 1 --2- 172.21.15.107:0/1988990637 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff6380a2d00 0x7ff6380a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.918+0000 7ff65698c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a73210 0x561586a735f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.918+0000 7ff642ffd640 1 -- 172.21.15.107:0/1988990637 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7ff6440cfa60 con 0x7ff64c106710 2024-05-09T08:40:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff65698c740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 0x561586a821f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff65698c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a82730 0x561586a84b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff65698c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5615869d8c40 con 0x561586a82730 2024-05-09T08:40:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff65698c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561586980580 con 0x561586a73210 2024-05-09T08:40:28.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff65698c740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5615869d4c00 con 0x561586a73b30 2024-05-09T08:40:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff65221a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a82730 0x561586a84b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff65221a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a82730 0x561586a84b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43978/0 (socket says 172.21.15.107:43978) 2024-05-09T08:40:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff65221a640 1 -- 172.21.15.107:0/3163375327 learned_addr learned my addr 172.21.15.107:0/3163375327 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff651a19640 1 --2- 172.21.15.107:0/3163375327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a73210 0x561586a735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff651218640 1 --2- 172.21.15.107:0/3163375327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 0x561586a821f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff65221a640 1 -- 172.21.15.107:0/3163375327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 msgr2=0x561586a821f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff65221a640 1 --2- 172.21.15.107:0/3163375327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 0x561586a821f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff651218640 1 --2- 172.21.15.107:0/1988990637 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff6380a2d00 0x7ff6380a51c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0694c0 tx=0x7ff63c00be80 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff65221a640 1 -- 172.21.15.107:0/3163375327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a73210 msgr2=0x561586a735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff65221a640 1 --2- 172.21.15.107:0/3163375327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a73210 0x561586a735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff65221a640 1 -- 172.21.15.107:0/3163375327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615866ec3f0 con 0x561586a82730 2024-05-09T08:40:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff65221a640 1 --2- 172.21.15.107:0/3163375327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a82730 0x561586a84b20 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7ff6440cd400 tx=0x7ff64409c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf1a45bdaf75428 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff637fff640 1 -- 172.21.15.107:0/3163375327 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64401f070 con 0x561586a82730 2024-05-09T08:40:28.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff637fff640 1 -- 172.21.15.107:0/3163375327 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff644005560 con 0x561586a82730 2024-05-09T08:40:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.919+0000 7ff637fff640 1 -- 172.21.15.107:0/3163375327 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff644097b80 con 0x561586a82730 2024-05-09T08:40:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.920+0000 7ff65698c740 1 -- 172.21.15.107:0/3163375327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a82730 msgr2=0x561586a84b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.920+0000 7ff65698c740 1 --2- 172.21.15.107:0/3163375327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a82730 0x561586a84b20 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7ff6440cd400 tx=0x7ff64409c430 comp rx=0 tx=0).stop 2024-05-09T08:40:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.920+0000 7ff65698c740 1 -- 172.21.15.107:0/3163375327 shutdown_connections 2024-05-09T08:40:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.920+0000 7ff65698c740 1 --2- 172.21.15.107:0/3163375327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a82730 0x561586a84b20 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.920+0000 7ff65698c740 1 --2- 172.21.15.107:0/3163375327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 0x561586a821f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.920+0000 7ff65698c740 1 --2- 172.21.15.107:0/3163375327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a73210 0x561586a735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.920+0000 7ff65698c740 1 -- 172.21.15.107:0/3163375327 >> 172.21.15.107:0/3163375327 conn(0x561586a6f650 msgr2=0x561586a6fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.920+0000 7ff65698c740 1 -- 172.21.15.107:0/3163375327 shutdown_connections 2024-05-09T08:40:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.920+0000 7ff65698c740 1 -- 172.21.15.107:0/3163375327 wait complete. 2024-05-09T08:40:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.921+0000 7ff65698c740 1 -- start start 2024-05-09T08:40:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.921+0000 7ff65698c740 1 Processor -- start 2024-05-09T08:40:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.921+0000 7ff65698c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a73210 0x561586a7f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.921+0000 7ff65698c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73b30 0x561586a7fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.921+0000 7ff65698c740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a82730 0x561586a800f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.921+0000 7ff65698c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561586a85570 con 0x561586a73b30 2024-05-09T08:40:28.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.921+0000 7ff65698c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561586993c40 con 0x561586a73210 2024-05-09T08:40:28.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.921+0000 7ff65698c740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x561586a856f0 con 0x561586a82730 2024-05-09T08:40:28.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.921+0000 7ff65698c740 10 client.0 _ll_register_callbacks cb 0x5615866ebbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:28.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.921+0000 7ff651218640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73b30 0x561586a7fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.921+0000 7ff651218640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73b30 0x561586a7fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43994/0 (socket says 172.21.15.107:43994) 2024-05-09T08:40:28.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.921+0000 7ff651218640 1 -- 172.21.15.107:0/1964598856 learned_addr learned my addr 172.21.15.107:0/1964598856 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff651a19640 1 --2- 172.21.15.107:0/1964598856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a73210 0x561586a7f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff65221a640 1 --2- 172.21.15.107:0/1964598856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a82730 0x561586a800f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff651218640 1 -- 172.21.15.107:0/1964598856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a82730 msgr2=0x561586a800f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff651218640 1 --2- 172.21.15.107:0/1964598856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a82730 0x561586a800f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff651218640 1 -- 172.21.15.107:0/1964598856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a73210 msgr2=0x561586a7f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff651218640 1 --2- 172.21.15.107:0/1964598856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a73210 0x561586a7f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff651218640 1 -- 172.21.15.107:0/1964598856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615867b67d0 con 0x561586a73b30 2024-05-09T08:40:28.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff65221a640 1 --2- 172.21.15.107:0/1964598856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a82730 0x561586a800f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:40:28.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff651218640 1 --2- 172.21.15.107:0/1964598856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73b30 0x561586a7fbb0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7ff63c011a50 tx=0x7ff63c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b86afd3e116944 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff6357fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:28.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff6357fa640 1 -- 172.21.15.107:0/1964598856 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c010d60 con 0x561586a73b30 2024-05-09T08:40:28.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff6357fa640 1 -- 172.21.15.107:0/1964598856 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7ff63c01d430 con 0x561586a73b30 2024-05-09T08:40:28.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff65698c740 1 -- 172.21.15.107:0/1964598856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff644005560 con 0x561586a73b30 2024-05-09T08:40:28.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff6357fa640 1 -- 172.21.15.107:0/1964598856 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff63c030050 con 0x561586a73b30 2024-05-09T08:40:28.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff65698c740 10 client.17757 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:28.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff65698c740 1 -- 172.21.15.107:0/1964598856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561586a91770 con 0x561586a73b30 2024-05-09T08:40:28.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.922+0000 7ff6357fa640 1 -- 172.21.15.107:0/1964598856 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff644005560 con 0x561586a73b30 2024-05-09T08:40:28.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.923+0000 7ff6357fa640 1 -- 172.21.15.107:0/1964598856 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff63c0255f0 con 0x561586a73b30 2024-05-09T08:40:28.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.923+0000 7ff65698c740 10 client.17757 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:28.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.923+0000 7ff65698c740 10 client.17757 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:40:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.923+0000 7ff65698c740 20 client.17757 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.923+0000 7ff65698c740 1 --2- 172.21.15.107:0/1964598856 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561586a92d20 0x561586a95160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.923+0000 7ff65698c740 4 client.17757 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:40:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.923+0000 7ff65698c740 1 -- 172.21.15.107:0/1964598856 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615869e6370 con 0x561586a92d20 2024-05-09T08:40:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.923+0000 7ff651a19640 1 --2- 172.21.15.107:0/1964598856 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561586a92d20 0x561586a95160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.923+0000 7ff6357fa640 10 client.17757 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.923+0000 7ff651a19640 1 --2- 172.21.15.107:0/1964598856 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561586a92d20 0x561586a95160 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c154bae63e38095 server_cookie=41bf76d0fb269574 in_seq=0 out_seq=0 2024-05-09T08:40:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.924+0000 7ff6357fa640 1 -- 172.21.15.107:0/1964598856 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5615869e6370 con 0x561586a92d20 2024-05-09T08:40:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.924+0000 7ff6357fa640 10 client.17757 handle_command_reply: tid=0 2024-05-09T08:40:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.924+0000 7ff65698c740 1 client.17757 shutdown 2024-05-09T08:40:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.924+0000 7ff65698c740 1 -- 172.21.15.107:0/1964598856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73b30 msgr2=0x561586a7fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.924+0000 7ff65698c740 1 --2- 172.21.15.107:0/1964598856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73b30 0x561586a7fbb0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7ff63c011a50 tx=0x7ff63c004730 comp rx=0 tx=0).stop 2024-05-09T08:40:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.924+0000 7ff65698c740 1 -- 172.21.15.107:0/1964598856 shutdown_connections 2024-05-09T08:40:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.924+0000 7ff65698c740 1 --2- 172.21.15.107:0/1964598856 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561586a92d20 0x561586a95160 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.925+0000 7ff65698c740 1 --2- 172.21.15.107:0/1964598856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a82730 0x561586a800f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.925+0000 7ff65698c740 1 --2- 172.21.15.107:0/1964598856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73b30 0x561586a7fbb0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.925+0000 7ff65698c740 1 --2- 172.21.15.107:0/1964598856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a73210 0x561586a7f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.925+0000 7ff6357fa640 0 client.17757 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:28.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.925+0000 7ff65698c740 1 -- 172.21.15.107:0/1964598856 >> 172.21.15.107:0/1964598856 conn(0x561586a6f650 msgr2=0x561586a75400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.925+0000 7ff65698c740 1 -- 172.21.15.107:0/1964598856 shutdown_connections 2024-05-09T08:40:28.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.925+0000 7ff65698c740 1 -- 172.21.15.107:0/1964598856 wait complete. 2024-05-09T08:40:28.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.925+0000 7ff65698c740 20 client.17757 trim_cache size 0 max 16384 2024-05-09T08:40:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff65698c740 1 Processor -- start 2024-05-09T08:40:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff65698c740 1 -- start start 2024-05-09T08:40:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff65698c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73210 0x561586a8e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff65698c740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 0x561586a9aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff65698c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a82730 0x561586a9af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff65698c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c1af320 con 0x561586a73210 2024-05-09T08:40:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff65698c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c1acbf0 con 0x561586a82730 2024-05-09T08:40:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff65698c740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c1aaa60 con 0x561586a73b30 2024-05-09T08:40:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff651a19640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73210 0x561586a8e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff651a19640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73210 0x561586a8e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44008/0 (socket says 172.21.15.107:44008) 2024-05-09T08:40:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff651a19640 1 -- 172.21.15.107:0/2133308491 learned_addr learned my addr 172.21.15.107:0/2133308491 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff65221a640 1 --2- 172.21.15.107:0/2133308491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a82730 0x561586a9af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff651218640 1 --2- 172.21.15.107:0/2133308491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 0x561586a9aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff651a19640 1 -- 172.21.15.107:0/2133308491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 msgr2=0x561586a9aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff651a19640 1 --2- 172.21.15.107:0/2133308491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 0x561586a9aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff651a19640 1 -- 172.21.15.107:0/2133308491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a82730 msgr2=0x561586a9af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff651a19640 1 --2- 172.21.15.107:0/2133308491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a82730 0x561586a9af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff651a19640 1 -- 172.21.15.107:0/2133308491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561586a84b20 con 0x561586a73210 2024-05-09T08:40:28.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff65221a640 1 --2- 172.21.15.107:0/2133308491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a82730 0x561586a9af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:40:28.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff651218640 1 --2- 172.21.15.107:0/2133308491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 0x561586a9aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:40:28.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.940+0000 7ff651a19640 1 --2- 172.21.15.107:0/2133308491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73210 0x561586a8e470 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7ff648031440 tx=0x7ff648005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e32a77e6687880 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.941+0000 7ff6377fe640 1 -- 172.21.15.107:0/2133308491 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648035040 con 0x561586a73210 2024-05-09T08:40:28.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.941+0000 7ff6377fe640 1 -- 172.21.15.107:0/2133308491 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff6480041d0 con 0x561586a73210 2024-05-09T08:40:28.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.941+0000 7ff6377fe640 1 -- 172.21.15.107:0/2133308491 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648005560 con 0x561586a73210 2024-05-09T08:40:28.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.941+0000 7ff65698c740 1 -- 172.21.15.107:0/2133308491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73210 msgr2=0x561586a8e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.941+0000 7ff65698c740 1 --2- 172.21.15.107:0/2133308491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73210 0x561586a8e470 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7ff648031440 tx=0x7ff648005a20 comp rx=0 tx=0).stop 2024-05-09T08:40:28.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.941+0000 7ff65698c740 1 -- 172.21.15.107:0/2133308491 shutdown_connections 2024-05-09T08:40:28.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.941+0000 7ff65698c740 1 --2- 172.21.15.107:0/2133308491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a82730 0x561586a9af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.941+0000 7ff65698c740 1 --2- 172.21.15.107:0/2133308491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 0x561586a9aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.941+0000 7ff65698c740 1 --2- 172.21.15.107:0/2133308491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73210 0x561586a8e470 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.941+0000 7ff65698c740 1 -- 172.21.15.107:0/2133308491 >> 172.21.15.107:0/2133308491 conn(0x561586a6f650 msgr2=0x561586a6e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.941+0000 7ff65698c740 1 -- 172.21.15.107:0/2133308491 shutdown_connections 2024-05-09T08:40:28.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.941+0000 7ff65698c740 1 -- 172.21.15.107:0/2133308491 wait complete. 2024-05-09T08:40:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.942+0000 7ff65698c740 1 -- start start 2024-05-09T08:40:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.942+0000 7ff65698c740 1 Processor -- start 2024-05-09T08:40:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.942+0000 7ff65698c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73210 0x561586a7ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.942+0000 7ff65698c740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 0x561586a7cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.942+0000 7ff65698c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a82730 0x561586af82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.942+0000 7ff65698c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c1ab0b0 con 0x561586a73210 2024-05-09T08:40:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.942+0000 7ff65698c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c1ac520 con 0x561586a82730 2024-05-09T08:40:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.942+0000 7ff65698c740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c1ab780 con 0x561586a73b30 2024-05-09T08:40:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.942+0000 7ff65698c740 10 client.0 _ll_register_callbacks cb 0x7ff63c00dac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.942+0000 7ff651a19640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73210 0x561586a7ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.942+0000 7ff651a19640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73210 0x561586a7ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44018/0 (socket says 172.21.15.107:44018) 2024-05-09T08:40:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.942+0000 7ff651a19640 1 -- 172.21.15.107:0/1944392966 learned_addr learned my addr 172.21.15.107:0/1944392966 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.942+0000 7ff65221a640 1 --2- 172.21.15.107:0/1944392966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a82730 0x561586af82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.943+0000 7ff651218640 1 --2- 172.21.15.107:0/1944392966 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 0x561586a7cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.943+0000 7ff651a19640 1 -- 172.21.15.107:0/1944392966 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 msgr2=0x561586a7cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.943+0000 7ff651a19640 1 --2- 172.21.15.107:0/1944392966 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 0x561586a7cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.943+0000 7ff651a19640 1 -- 172.21.15.107:0/1944392966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a82730 msgr2=0x561586af82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.943+0000 7ff651a19640 1 --2- 172.21.15.107:0/1944392966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a82730 0x561586af82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.943+0000 7ff651a19640 1 -- 172.21.15.107:0/1944392966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561586af87e0 con 0x561586a73210 2024-05-09T08:40:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.943+0000 7ff651218640 1 --2- 172.21.15.107:0/1944392966 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 0x561586a7cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:40:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.943+0000 7ff651a19640 1 --2- 172.21.15.107:0/1944392966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73210 0x561586a7ca90 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7ff648000c00 tx=0x7ff648009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a42e52452d06fb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.943+0000 7ff6357fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.943+0000 7ff6357fa640 1 -- 172.21.15.107:0/1944392966 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648035050 con 0x561586a73210 2024-05-09T08:40:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.943+0000 7ff6357fa640 1 -- 172.21.15.107:0/1944392966 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7ff648043040 con 0x561586a73210 2024-05-09T08:40:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.943+0000 7ff65698c740 1 -- 172.21.15.107:0/1944392966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6480041d0 con 0x561586a73210 2024-05-09T08:40:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.943+0000 7ff6357fa640 1 -- 172.21.15.107:0/1944392966 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff6480035f0 con 0x561586a73210 2024-05-09T08:40:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.943+0000 7ff65698c740 10 client.17763 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:28.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.943+0000 7ff65698c740 1 -- 172.21.15.107:0/1944392966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561586af9a50 con 0x561586a73210 2024-05-09T08:40:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.943+0000 7ff6357fa640 1 -- 172.21.15.107:0/1944392966 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff6480041d0 con 0x561586a73210 2024-05-09T08:40:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.944+0000 7ff6357fa640 1 -- 172.21.15.107:0/1944392966 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff648054030 con 0x561586a73210 2024-05-09T08:40:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.944+0000 7ff65698c740 10 client.17763 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.944+0000 7ff65698c740 10 client.17763 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:40:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.944+0000 7ff65698c740 20 client.17763 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.944+0000 7ff65698c740 1 --2- 172.21.15.107:0/1944392966 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561586afa130 0x561586afc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.944+0000 7ff65698c740 4 client.17763 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:40:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.944+0000 7ff65698c740 1 -- 172.21.15.107:0/1944392966 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5615869e06c0 con 0x561586afa130 2024-05-09T08:40:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.944+0000 7ff651218640 1 --2- 172.21.15.107:0/1944392966 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561586afa130 0x561586afc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.944+0000 7ff6357fa640 10 client.17763 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.944+0000 7ff651218640 1 --2- 172.21.15.107:0/1944392966 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561586afa130 0x561586afc550 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=941ce97b1bd7d206 server_cookie=61a46e7b5e128cf4 in_seq=0 out_seq=0 2024-05-09T08:40:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.945+0000 7ff6357fa640 1 -- 172.21.15.107:0/1944392966 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5615869e06c0 con 0x561586afa130 2024-05-09T08:40:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.945+0000 7ff6357fa640 10 client.17763 handle_command_reply: tid=0 2024-05-09T08:40:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.945+0000 7ff65698c740 1 client.17763 shutdown 2024-05-09T08:40:28.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.945+0000 7ff65698c740 1 -- 172.21.15.107:0/1944392966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73210 msgr2=0x561586a7ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.945+0000 7ff65698c740 1 --2- 172.21.15.107:0/1944392966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73210 0x561586a7ca90 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7ff648000c00 tx=0x7ff648009680 comp rx=0 tx=0).stop 2024-05-09T08:40:28.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.945+0000 7ff65698c740 1 -- 172.21.15.107:0/1944392966 shutdown_connections 2024-05-09T08:40:28.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.945+0000 7ff65698c740 1 --2- 172.21.15.107:0/1944392966 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561586afa130 0x561586afc550 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.945+0000 7ff65698c740 1 --2- 172.21.15.107:0/1944392966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561586a82730 0x561586af82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.945+0000 7ff65698c740 1 --2- 172.21.15.107:0/1944392966 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561586a73b30 0x561586a7cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.945+0000 7ff65698c740 1 --2- 172.21.15.107:0/1944392966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561586a73210 0x561586a7ca90 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.945+0000 7ff6357fa640 0 client.17763 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:28.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.945+0000 7ff65698c740 1 -- 172.21.15.107:0/1944392966 >> 172.21.15.107:0/1944392966 conn(0x561586a6f650 msgr2=0x561586a7f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.945+0000 7ff65698c740 1 -- 172.21.15.107:0/1944392966 shutdown_connections 2024-05-09T08:40:28.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.945+0000 7ff65698c740 1 -- 172.21.15.107:0/1944392966 wait complete. 2024-05-09T08:40:28.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.945+0000 7ff65698c740 20 client.17763 trim_cache size 0 max 16384 2024-05-09T08:40:28.951 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:40:28.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.946+0000 7ff653c7d640 1 -- 172.21.15.107:0/1988990637 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff6380a2d00 msgr2=0x7ff6380a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.946+0000 7ff653c7d640 1 --2- 172.21.15.107:0/1988990637 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff6380a2d00 0x7ff6380a51c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0694c0 tx=0x7ff63c00be80 comp rx=0 tx=0).stop 2024-05-09T08:40:28.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.946+0000 7ff653c7d640 1 -- 172.21.15.107:0/1988990637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff64c106710 msgr2=0x7ff64c1a3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:28.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.946+0000 7ff653c7d640 1 --2- 172.21.15.107:0/1988990637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff64c106710 0x7ff64c1a3f50 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7ff64400eaa0 tx=0x7ff64400ef60 comp rx=0 tx=0).stop 2024-05-09T08:40:28.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.946+0000 7ff653c7d640 1 -- 172.21.15.107:0/1988990637 shutdown_connections 2024-05-09T08:40:28.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.946+0000 7ff653c7d640 1 --2- 172.21.15.107:0/1988990637 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff6380a2d00 0x7ff6380a51c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.946+0000 7ff653c7d640 1 --2- 172.21.15.107:0/1988990637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff64c106710 0x7ff64c1a3f50 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.946+0000 7ff653c7d640 1 --2- 172.21.15.107:0/1988990637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff64c1010c0 0x7ff64c199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.946+0000 7ff653c7d640 1 --2- 172.21.15.107:0/1988990637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff64c100700 0x7ff64c198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:28.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.946+0000 7ff653c7d640 1 -- 172.21.15.107:0/1988990637 >> 172.21.15.107:0/1988990637 conn(0x7ff64c077830 msgr2=0x7ff64c103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:28.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.946+0000 7ff653c7d640 1 -- 172.21.15.107:0/1988990637 shutdown_connections 2024-05-09T08:40:28.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:28.946+0000 7ff653c7d640 1 -- 172.21.15.107:0/1988990637 wait complete. 2024-05-09T08:40:29.958 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:40:29.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:40:29.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:40:30.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.074+0000 7fe2e16d4640 1 -- 172.21.15.107:0/2717606365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2dc100710 msgr2=0x7fe2dc100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.074+0000 7fe2e16d4640 1 --2- 172.21.15.107:0/2717606365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2dc100710 0x7fe2dc100b90 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009930 tx=0x7fe2c402a420 comp rx=0 tx=0).stop 2024-05-09T08:40:30.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.074+0000 7fe2e16d4640 1 -- 172.21.15.107:0/2717606365 shutdown_connections 2024-05-09T08:40:30.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.074+0000 7fe2e16d4640 1 --2- 172.21.15.107:0/2717606365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2dc1010d0 0x7fe2dc10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.074+0000 7fe2e16d4640 1 --2- 172.21.15.107:0/2717606365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2dc100710 0x7fe2dc100b90 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.074+0000 7fe2e16d4640 1 --2- 172.21.15.107:0/2717606365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2dc106720 0x7fe2dc106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.074+0000 7fe2e16d4640 1 -- 172.21.15.107:0/2717606365 >> 172.21.15.107:0/2717606365 conn(0x7fe2dc076ab0 msgr2=0x7fe2dc076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.074+0000 7fe2e16d4640 1 -- 172.21.15.107:0/2717606365 shutdown_connections 2024-05-09T08:40:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.074+0000 7fe2e16d4640 1 -- 172.21.15.107:0/2717606365 wait complete. 2024-05-09T08:40:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.074+0000 7fe2e16d4640 1 Processor -- start 2024-05-09T08:40:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.074+0000 7fe2e16d4640 1 -- start start 2024-05-09T08:40:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2e16d4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2dc100710 0x7fe2dc198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2e16d4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2dc1010d0 0x7fe2dc199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2e16d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2dc106720 0x7fe2dc1a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2e16d4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe2dc1109e0 con 0x7fe2dc106720 2024-05-09T08:40:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2e16d4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe2dc110860 con 0x7fe2dc1010d0 2024-05-09T08:40:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2e16d4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe2dc110b60 con 0x7fe2dc100710 2024-05-09T08:40:30.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2db7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2dc106720 0x7fe2dc1a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2db7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2dc106720 0x7fe2dc1a3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44030/0 (socket says 172.21.15.107:44030) 2024-05-09T08:40:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2db7fe640 1 -- 172.21.15.107:0/4204416947 learned_addr learned my addr 172.21.15.107:0/4204416947 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2d3fff640 1 --2- 172.21.15.107:0/4204416947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2dc1010d0 0x7fe2dc199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2daffd640 1 --2- 172.21.15.107:0/4204416947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2dc100710 0x7fe2dc198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2db7fe640 1 -- 172.21.15.107:0/4204416947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2dc100710 msgr2=0x7fe2dc198f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2db7fe640 1 --2- 172.21.15.107:0/4204416947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2dc100710 0x7fe2dc198f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2db7fe640 1 -- 172.21.15.107:0/4204416947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2dc1010d0 msgr2=0x7fe2dc199490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2db7fe640 1 --2- 172.21.15.107:0/4204416947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2dc1010d0 0x7fe2dc199490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2db7fe640 1 -- 172.21.15.107:0/4204416947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2dc1a4510 con 0x7fe2dc106720 2024-05-09T08:40:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2db7fe640 1 --2- 172.21.15.107:0/4204416947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2dc106720 0x7fe2dc1a3fd0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4009fd0 tx=0x7fe2d400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2d8ff9640 1 -- 172.21.15.107:0/4204416947 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4017070 con 0x7fe2dc106720 2024-05-09T08:40:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.075+0000 7fe2e16d4640 1 -- 172.21.15.107:0/4204416947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2dc1a46e0 con 0x7fe2dc106720 2024-05-09T08:40:30.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.076+0000 7fe2d8ff9640 1 -- 172.21.15.107:0/4204416947 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2d4003e20 con 0x7fe2dc106720 2024-05-09T08:40:30.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.076+0000 7fe2d8ff9640 1 -- 172.21.15.107:0/4204416947 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4020060 con 0x7fe2dc106720 2024-05-09T08:40:30.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.076+0000 7fe2e16d4640 1 -- 172.21.15.107:0/4204416947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2dc1a4c00 con 0x7fe2dc106720 2024-05-09T08:40:30.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.078+0000 7fe2d8ff9640 1 -- 172.21.15.107:0/4204416947 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe2d4003970 con 0x7fe2dc106720 2024-05-09T08:40:30.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.078+0000 7fe2e43e9740 1 Processor -- start 2024-05-09T08:40:30.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.078+0000 7fe2d8ff9640 1 --2- 172.21.15.107:0/4204416947 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2c80a2d00 0x7fe2c80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.078+0000 7fe2e43e9740 1 -- start start 2024-05-09T08:40:30.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.079+0000 7fe2d8ff9640 1 -- 172.21.15.107:0/4204416947 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fe2d40cf9a0 con 0x7fe2dc106720 2024-05-09T08:40:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.079+0000 7fe2d3fff640 1 --2- 172.21.15.107:0/4204416947 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2c80a2d00 0x7fe2c80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.079+0000 7fe2e43e9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5609447460b0 0x5609447443f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.079+0000 7fe2e43e9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447449c0 0x560944744e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.079+0000 7fe2e43e9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56094474f740 0x560944751b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.079+0000 7fe2e43e9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5609446a9990 con 0x5609447449c0 2024-05-09T08:40:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.079+0000 7fe2e43e9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5609446515b0 con 0x56094474f740 2024-05-09T08:40:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.079+0000 7fe2e43e9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5609446a5950 con 0x5609447460b0 2024-05-09T08:40:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.079+0000 7fe2daffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5609447460b0 0x5609447443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.079+0000 7fe2db7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56094474f740 0x560944751b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.079+0000 7fe2daffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5609447460b0 0x5609447443f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:33620/0 (socket says 172.21.15.107:33620) 2024-05-09T08:40:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.079+0000 7fe2daffd640 1 -- 172.21.15.107:0/2656498746 learned_addr learned my addr 172.21.15.107:0/2656498746 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.079+0000 7fe2d3fff640 1 --2- 172.21.15.107:0/2656498746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447449c0 0x560944744e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.079+0000 7fe2d3fff640 1 --2- 172.21.15.107:0/4204416947 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2c80a2d00 0x7fe2c80a51c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009900 tx=0x7fe2c4000bc0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.080+0000 7fe2db7fe640 1 -- 172.21.15.107:0/2656498746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5609447460b0 msgr2=0x5609447443f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.080+0000 7fe2db7fe640 1 --2- 172.21.15.107:0/2656498746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5609447460b0 0x5609447443f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.080+0000 7fe2db7fe640 1 -- 172.21.15.107:0/2656498746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447449c0 msgr2=0x560944744e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.080+0000 7fe2db7fe640 1 --2- 172.21.15.107:0/2656498746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447449c0 0x560944744e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.080+0000 7fe2db7fe640 1 -- 172.21.15.107:0/2656498746 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609444857e0 con 0x56094474f740 2024-05-09T08:40:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.080+0000 7fe2db7fe640 1 --2- 172.21.15.107:0/2656498746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56094474f740 0x560944751b30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe2d40ccd40 tx=0x7fe2d4003d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f2766238aa9088d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.080+0000 7fe2d17fa640 1 -- 172.21.15.107:0/2656498746 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d409fec0 con 0x56094474f740 2024-05-09T08:40:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.080+0000 7fe2d17fa640 1 -- 172.21.15.107:0/2656498746 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2d40b4020 con 0x56094474f740 2024-05-09T08:40:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.080+0000 7fe2d17fa640 1 -- 172.21.15.107:0/2656498746 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d40a42f0 con 0x56094474f740 2024-05-09T08:40:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.081+0000 7fe2e43e9740 1 -- 172.21.15.107:0/2656498746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56094474f740 msgr2=0x560944751b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.081+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/2656498746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56094474f740 0x560944751b30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe2d40ccd40 tx=0x7fe2d4003d20 comp rx=0 tx=0).stop 2024-05-09T08:40:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.081+0000 7fe2e43e9740 1 -- 172.21.15.107:0/2656498746 shutdown_connections 2024-05-09T08:40:30.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.081+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/2656498746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56094474f740 0x560944751b30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.081+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/2656498746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447449c0 0x560944744e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.081+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/2656498746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5609447460b0 0x5609447443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.081+0000 7fe2e43e9740 1 -- 172.21.15.107:0/2656498746 >> 172.21.15.107:0/2656498746 conn(0x56094470d0a0 msgr2=0x560944740f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:30.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.081+0000 7fe2e43e9740 1 -- 172.21.15.107:0/2656498746 shutdown_connections 2024-05-09T08:40:30.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.081+0000 7fe2e43e9740 1 -- 172.21.15.107:0/2656498746 wait complete. 2024-05-09T08:40:30.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.081+0000 7fe2e43e9740 1 -- start start 2024-05-09T08:40:30.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.081+0000 7fe2e43e9740 1 Processor -- start 2024-05-09T08:40:30.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.081+0000 7fe2e43e9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447449c0 0x56094474c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.081+0000 7fe2e43e9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5609447460b0 0x56094474cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2e43e9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56094474f740 0x56094474d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2e43e9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560944752570 con 0x5609447449c0 2024-05-09T08:40:30.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2e43e9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560944664b40 con 0x5609447460b0 2024-05-09T08:40:30.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2e43e9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5609447526f0 con 0x56094474f740 2024-05-09T08:40:30.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2e43e9740 10 client.0 _ll_register_callbacks cb 0x5609446590f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:30.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2daffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447449c0 0x56094474c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2daffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447449c0 0x56094474c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44052/0 (socket says 172.21.15.107:44052) 2024-05-09T08:40:30.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2daffd640 1 -- 172.21.15.107:0/1789227380 learned_addr learned my addr 172.21.15.107:0/1789227380 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:30.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2d3fff640 1 --2- 172.21.15.107:0/1789227380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5609447460b0 0x56094474cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2db7fe640 1 --2- 172.21.15.107:0/1789227380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56094474f740 0x56094474d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2daffd640 1 -- 172.21.15.107:0/1789227380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56094474f740 msgr2=0x56094474d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:30.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2daffd640 1 --2- 172.21.15.107:0/1789227380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56094474f740 0x56094474d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2daffd640 1 -- 172.21.15.107:0/1789227380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5609447460b0 msgr2=0x56094474cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:30.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2daffd640 1 --2- 172.21.15.107:0/1789227380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5609447460b0 0x56094474cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2daffd640 1 -- 172.21.15.107:0/1789227380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609444e5f70 con 0x5609447449c0 2024-05-09T08:40:30.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2daffd640 1 --2- 172.21.15.107:0/1789227380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447449c0 0x56094474c670 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc0126d0 tx=0x7fe2cc012b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aecb32ed81cf01a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:30.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2c27fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:30.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2c27fc640 1 -- 172.21.15.107:0/1789227380 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc011bb0 con 0x5609447449c0 2024-05-09T08:40:30.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2c27fc640 1 -- 172.21.15.107:0/1789227380 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fe2cc011d50 con 0x5609447449c0 2024-05-09T08:40:30.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2e43e9740 1 -- 172.21.15.107:0/1789227380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2d40b4020 con 0x5609447449c0 2024-05-09T08:40:30.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.082+0000 7fe2c27fc640 1 -- 172.21.15.107:0/1789227380 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe2cc01c420 con 0x5609447449c0 2024-05-09T08:40:30.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.083+0000 7fe2e43e9740 10 client.17775 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:30.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.083+0000 7fe2e43e9740 1 -- 172.21.15.107:0/1789227380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56094437d170 con 0x5609447449c0 2024-05-09T08:40:30.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.083+0000 7fe2c27fc640 1 -- 172.21.15.107:0/1789227380 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2d40b4020 con 0x5609447449c0 2024-05-09T08:40:30.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.083+0000 7fe2c27fc640 1 -- 172.21.15.107:0/1789227380 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe2cc00f6c0 con 0x5609447449c0 2024-05-09T08:40:30.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.083+0000 7fe2e43e9740 10 client.17775 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:30.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.083+0000 7fe2e43e9740 10 client.17775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:40:30.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.083+0000 7fe2e43e9740 20 client.17775 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:30.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.083+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/1789227380 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56094475f8e0 0x560944761d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.083+0000 7fe2e43e9740 4 client.17775 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:40:30.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.083+0000 7fe2e43e9740 1 -- 172.21.15.107:0/1789227380 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609446b70c0 con 0x56094475f8e0 2024-05-09T08:40:30.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.083+0000 7fe2d3fff640 1 --2- 172.21.15.107:0/1789227380 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56094475f8e0 0x560944761d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.084+0000 7fe2c27fc640 10 client.17775 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:30.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.084+0000 7fe2d3fff640 1 --2- 172.21.15.107:0/1789227380 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56094475f8e0 0x560944761d70 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46dc2db6270aa7e4 server_cookie=d41bea2a18e89833 in_seq=0 out_seq=0 2024-05-09T08:40:30.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.084+0000 7fe2c27fc640 1 -- 172.21.15.107:0/1789227380 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5609446b70c0 con 0x56094475f8e0 2024-05-09T08:40:30.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.084+0000 7fe2c27fc640 10 client.17775 handle_command_reply: tid=0 2024-05-09T08:40:30.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.084+0000 7fe2e43e9740 1 client.17775 shutdown 2024-05-09T08:40:30.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.084+0000 7fe2e43e9740 1 -- 172.21.15.107:0/1789227380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447449c0 msgr2=0x56094474c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.084+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/1789227380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447449c0 0x56094474c670 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc0126d0 tx=0x7fe2cc012b90 comp rx=0 tx=0).stop 2024-05-09T08:40:30.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.084+0000 7fe2e43e9740 1 -- 172.21.15.107:0/1789227380 shutdown_connections 2024-05-09T08:40:30.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.084+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/1789227380 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56094475f8e0 0x560944761d70 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.084+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/1789227380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56094474f740 0x56094474d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.084+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/1789227380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5609447460b0 0x56094474cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.084+0000 7fe2c27fc640 0 client.17775 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:30.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.084+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/1789227380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447449c0 0x56094474c670 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.085+0000 7fe2e43e9740 1 -- 172.21.15.107:0/1789227380 >> 172.21.15.107:0/1789227380 conn(0x56094470d0a0 msgr2=0x560944746d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:30.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.085+0000 7fe2e43e9740 1 -- 172.21.15.107:0/1789227380 shutdown_connections 2024-05-09T08:40:30.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.085+0000 7fe2e43e9740 1 -- 172.21.15.107:0/1789227380 wait complete. 2024-05-09T08:40:30.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.085+0000 7fe2e43e9740 20 client.17775 trim_cache size 0 max 16384 2024-05-09T08:40:30.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2e43e9740 1 Processor -- start 2024-05-09T08:40:30.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2e43e9740 1 -- start start 2024-05-09T08:40:30.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2e43e9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5609447449c0 0x56094473fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2e43e9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447460b0 0x56094475b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2e43e9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56094474f740 0x56094474e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2e43e9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe2dc1b1590 con 0x5609447460b0 2024-05-09T08:40:30.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2e43e9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe2dc1aee60 con 0x5609447449c0 2024-05-09T08:40:30.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2e43e9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe2dc1accd0 con 0x56094474f740 2024-05-09T08:40:30.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447460b0 0x56094475b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447460b0 0x56094475b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44060/0 (socket says 172.21.15.107:44060) 2024-05-09T08:40:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2d3fff640 1 -- 172.21.15.107:0/3507735462 learned_addr learned my addr 172.21.15.107:0/3507735462 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2daffd640 1 --2- 172.21.15.107:0/3507735462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5609447449c0 0x56094473fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2db7fe640 1 --2- 172.21.15.107:0/3507735462 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56094474f740 0x56094474e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2d3fff640 1 -- 172.21.15.107:0/3507735462 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56094474f740 msgr2=0x56094474e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2d3fff640 1 --2- 172.21.15.107:0/3507735462 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56094474f740 0x56094474e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2d3fff640 1 -- 172.21.15.107:0/3507735462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5609447449c0 msgr2=0x56094473fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.105+0000 7fe2d3fff640 1 --2- 172.21.15.107:0/3507735462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5609447449c0 0x56094473fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.106+0000 7fe2d3fff640 1 -- 172.21.15.107:0/3507735462 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609447526e0 con 0x5609447460b0 2024-05-09T08:40:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.106+0000 7fe2d3fff640 1 --2- 172.21.15.107:0/3507735462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447460b0 0x56094475b470 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fe2c402a940 tx=0x7fe2c403d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcc65d2953069d72 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.106+0000 7fe2d0ff9640 1 -- 172.21.15.107:0/3507735462 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4059070 con 0x5609447460b0 2024-05-09T08:40:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.106+0000 7fe2d0ff9640 1 -- 172.21.15.107:0/3507735462 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2c40532d0 con 0x5609447460b0 2024-05-09T08:40:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.106+0000 7fe2d0ff9640 1 -- 172.21.15.107:0/3507735462 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4035040 con 0x5609447460b0 2024-05-09T08:40:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 -- 172.21.15.107:0/3507735462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447460b0 msgr2=0x56094475b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/3507735462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447460b0 0x56094475b470 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fe2c402a940 tx=0x7fe2c403d000 comp rx=0 tx=0).stop 2024-05-09T08:40:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 -- 172.21.15.107:0/3507735462 shutdown_connections 2024-05-09T08:40:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/3507735462 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56094474f740 0x56094474e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/3507735462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447460b0 0x56094475b470 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/3507735462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5609447449c0 0x56094473fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 -- 172.21.15.107:0/3507735462 >> 172.21.15.107:0/3507735462 conn(0x56094470d0a0 msgr2=0x56094474dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 -- 172.21.15.107:0/3507735462 shutdown_connections 2024-05-09T08:40:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 -- 172.21.15.107:0/3507735462 wait complete. 2024-05-09T08:40:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 -- start start 2024-05-09T08:40:30.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 Processor -- start 2024-05-09T08:40:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5609447449c0 0x5609447c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447460b0 0x5609447c47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56094474f740 0x560944749040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe2dc1ad320 con 0x5609447460b0 2024-05-09T08:40:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe2dc1ae790 con 0x56094474f740 2024-05-09T08:40:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe2dc1ad9f0 con 0x5609447449c0 2024-05-09T08:40:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.107+0000 7fe2e43e9740 10 client.0 _ll_register_callbacks cb 0x7fe2cc0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447460b0 0x5609447c47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447460b0 0x5609447c47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44074/0 (socket says 172.21.15.107:44074) 2024-05-09T08:40:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2d3fff640 1 -- 172.21.15.107:0/4020497126 learned_addr learned my addr 172.21.15.107:0/4020497126 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2db7fe640 1 --2- 172.21.15.107:0/4020497126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56094474f740 0x560944749040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2daffd640 1 --2- 172.21.15.107:0/4020497126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5609447449c0 0x5609447c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2d3fff640 1 -- 172.21.15.107:0/4020497126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5609447449c0 msgr2=0x5609447c42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2d3fff640 1 --2- 172.21.15.107:0/4020497126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5609447449c0 0x5609447c42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2d3fff640 1 -- 172.21.15.107:0/4020497126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56094474f740 msgr2=0x560944749040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2d3fff640 1 --2- 172.21.15.107:0/4020497126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56094474f740 0x560944749040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2d3fff640 1 -- 172.21.15.107:0/4020497126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56094475c020 con 0x5609447460b0 2024-05-09T08:40:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2d3fff640 1 --2- 172.21.15.107:0/4020497126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447460b0 0x5609447c47f0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4004770 tx=0x7fe2c4031530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc09c5a95b12805 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2c27fc640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2c27fc640 1 -- 172.21.15.107:0/4020497126 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4059070 con 0x5609447460b0 2024-05-09T08:40:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2c27fc640 1 -- 172.21.15.107:0/4020497126 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fe2c4053480 con 0x5609447460b0 2024-05-09T08:40:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2e43e9740 1 -- 172.21.15.107:0/4020497126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2c40532d0 con 0x5609447460b0 2024-05-09T08:40:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2c27fc640 1 -- 172.21.15.107:0/4020497126 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe2c404e030 con 0x5609447460b0 2024-05-09T08:40:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2e43e9740 10 client.17781 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.108+0000 7fe2e43e9740 1 -- 172.21.15.107:0/4020497126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5609447cb8a0 con 0x5609447460b0 2024-05-09T08:40:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.109+0000 7fe2c27fc640 1 -- 172.21.15.107:0/4020497126 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2c40532d0 con 0x5609447460b0 2024-05-09T08:40:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.109+0000 7fe2c27fc640 1 -- 172.21.15.107:0/4020497126 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe2c405e460 con 0x5609447460b0 2024-05-09T08:40:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.109+0000 7fe2e43e9740 10 client.17781 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.109+0000 7fe2e43e9740 10 client.17781 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:40:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.109+0000 7fe2e43e9740 20 client.17781 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.109+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/4020497126 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5609447cbfc0 0x5609447ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.109+0000 7fe2e43e9740 4 client.17781 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:40:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.109+0000 7fe2e43e9740 1 -- 172.21.15.107:0/4020497126 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5609446b1410 con 0x5609447cbfc0 2024-05-09T08:40:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.109+0000 7fe2daffd640 1 --2- 172.21.15.107:0/4020497126 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5609447cbfc0 0x5609447ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.109+0000 7fe2c27fc640 10 client.17781 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.109+0000 7fe2daffd640 1 --2- 172.21.15.107:0/4020497126 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5609447cbfc0 0x5609447ce3e0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a104319db0a0fe7 server_cookie=74df63ce18a703ac in_seq=0 out_seq=0 2024-05-09T08:40:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.111+0000 7fe2c27fc640 1 -- 172.21.15.107:0/4020497126 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5609446b1410 con 0x5609447cbfc0 2024-05-09T08:40:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.111+0000 7fe2c27fc640 10 client.17781 handle_command_reply: tid=0 2024-05-09T08:40:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.111+0000 7fe2e43e9740 1 client.17781 shutdown 2024-05-09T08:40:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.111+0000 7fe2e43e9740 1 -- 172.21.15.107:0/4020497126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447460b0 msgr2=0x5609447c47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.111+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/4020497126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447460b0 0x5609447c47f0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4004770 tx=0x7fe2c4031530 comp rx=0 tx=0).stop 2024-05-09T08:40:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.111+0000 7fe2e43e9740 1 -- 172.21.15.107:0/4020497126 shutdown_connections 2024-05-09T08:40:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.111+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/4020497126 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5609447cbfc0 0x5609447ce3e0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.111+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/4020497126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56094474f740 0x560944749040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.111+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/4020497126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609447460b0 0x5609447c47f0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.111+0000 7fe2e43e9740 1 --2- 172.21.15.107:0/4020497126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5609447449c0 0x5609447c42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.111+0000 7fe2c27fc640 0 client.17781 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.111+0000 7fe2e43e9740 1 -- 172.21.15.107:0/4020497126 >> 172.21.15.107:0/4020497126 conn(0x56094470d0a0 msgr2=0x56094474e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.111+0000 7fe2e43e9740 1 -- 172.21.15.107:0/4020497126 shutdown_connections 2024-05-09T08:40:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.111+0000 7fe2e43e9740 1 -- 172.21.15.107:0/4020497126 wait complete. 2024-05-09T08:40:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.111+0000 7fe2e43e9740 20 client.17781 trim_cache size 0 max 16384 2024-05-09T08:40:30.116 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:40:30.116 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:40:30.116 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "780ca629-fec8-45ed-8a09-b26053188d11", 2024-05-09T08:40:30.116 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:40:30.116 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:40:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.112+0000 7fe2e16d4640 1 -- 172.21.15.107:0/4204416947 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2c80a2d00 msgr2=0x7fe2c80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.112+0000 7fe2e16d4640 1 --2- 172.21.15.107:0/4204416947 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2c80a2d00 0x7fe2c80a51c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009900 tx=0x7fe2c4000bc0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.112+0000 7fe2e16d4640 1 -- 172.21.15.107:0/4204416947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2dc106720 msgr2=0x7fe2dc1a3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.112+0000 7fe2e16d4640 1 --2- 172.21.15.107:0/4204416947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2dc106720 0x7fe2dc1a3fd0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4009fd0 tx=0x7fe2d400b260 comp rx=0 tx=0).stop 2024-05-09T08:40:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.112+0000 7fe2e16d4640 1 -- 172.21.15.107:0/4204416947 shutdown_connections 2024-05-09T08:40:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.112+0000 7fe2e16d4640 1 --2- 172.21.15.107:0/4204416947 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2c80a2d00 0x7fe2c80a51c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.112+0000 7fe2e16d4640 1 --2- 172.21.15.107:0/4204416947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2dc106720 0x7fe2dc1a3fd0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.112+0000 7fe2e16d4640 1 --2- 172.21.15.107:0/4204416947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2dc1010d0 0x7fe2dc199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.112+0000 7fe2e16d4640 1 --2- 172.21.15.107:0/4204416947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2dc100710 0x7fe2dc198f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.112+0000 7fe2e16d4640 1 -- 172.21.15.107:0/4204416947 >> 172.21.15.107:0/4204416947 conn(0x7fe2dc076ab0 msgr2=0x7fe2dc0fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.112+0000 7fe2e16d4640 1 -- 172.21.15.107:0/4204416947 shutdown_connections 2024-05-09T08:40:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.112+0000 7fe2e16d4640 1 -- 172.21.15.107:0/4204416947 wait complete. 2024-05-09T08:40:30.123 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:40:30.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:40:30.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.249+0000 7fae511a9640 1 -- 172.21.15.107:0/3219968385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c1010c0 msgr2=0x7fae4c10d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.249+0000 7fae511a9640 1 --2- 172.21.15.107:0/3219968385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c1010c0 0x7fae4c10d5d0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fae44009cd0 tx=0x7fae4402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.249+0000 7fae511a9640 1 -- 172.21.15.107:0/3219968385 shutdown_connections 2024-05-09T08:40:30.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.249+0000 7fae511a9640 1 --2- 172.21.15.107:0/3219968385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c1010c0 0x7fae4c10d5d0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.250+0000 7fae511a9640 1 --2- 172.21.15.107:0/3219968385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae4c100700 0x7fae4c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.250+0000 7fae511a9640 1 --2- 172.21.15.107:0/3219968385 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae4c106710 0x7fae4c106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.250+0000 7fae511a9640 1 -- 172.21.15.107:0/3219968385 >> 172.21.15.107:0/3219968385 conn(0x7fae4c077830 msgr2=0x7fae4c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:30.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.250+0000 7fae511a9640 1 -- 172.21.15.107:0/3219968385 shutdown_connections 2024-05-09T08:40:30.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.250+0000 7fae511a9640 1 -- 172.21.15.107:0/3219968385 wait complete. 2024-05-09T08:40:30.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.250+0000 7fae511a9640 1 Processor -- start 2024-05-09T08:40:30.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae511a9640 1 -- start start 2024-05-09T08:40:30.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae511a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c100700 0x7fae4c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae511a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae4c1010c0 0x7fae4c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae511a9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae4c106710 0x7fae4c19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae511a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c110990 con 0x7fae4c100700 2024-05-09T08:40:30.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae511a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c110810 con 0x7fae4c1010c0 2024-05-09T08:40:30.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae511a9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c110b10 con 0x7fae4c106710 2024-05-09T08:40:30.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae4ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c100700 0x7fae4c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae4ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c100700 0x7fae4c198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44092/0 (socket says 172.21.15.107:44092) 2024-05-09T08:40:30.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae4a59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae4c1010c0 0x7fae4c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae4ad9d640 1 -- 172.21.15.107:0/1114000219 learned_addr learned my addr 172.21.15.107:0/1114000219 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:30.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae4b59e640 1 --2- 172.21.15.107:0/1114000219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae4c106710 0x7fae4c19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae4ad9d640 1 -- 172.21.15.107:0/1114000219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae4c106710 msgr2=0x7fae4c19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae4ad9d640 1 --2- 172.21.15.107:0/1114000219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae4c106710 0x7fae4c19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae4ad9d640 1 -- 172.21.15.107:0/1114000219 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae4c1010c0 msgr2=0x7fae4c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae4ad9d640 1 --2- 172.21.15.107:0/1114000219 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae4c1010c0 0x7fae4c198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae4ad9d640 1 -- 172.21.15.107:0/1114000219 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae4c19cff0 con 0x7fae4c100700 2024-05-09T08:40:30.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.251+0000 7fae4ad9d640 1 --2- 172.21.15.107:0/1114000219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c100700 0x7fae4c198000 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00d950 tx=0x7fae3c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:30.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.252+0000 7fae33fff640 1 -- 172.21.15.107:0/1114000219 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3c017030 con 0x7fae4c100700 2024-05-09T08:40:30.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.252+0000 7fae511a9640 1 -- 172.21.15.107:0/1114000219 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae4c19d2e0 con 0x7fae4c100700 2024-05-09T08:40:30.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.252+0000 7fae33fff640 1 -- 172.21.15.107:0/1114000219 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae3c004020 con 0x7fae4c100700 2024-05-09T08:40:30.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.252+0000 7fae33fff640 1 -- 172.21.15.107:0/1114000219 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3c0047a0 con 0x7fae4c100700 2024-05-09T08:40:30.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.252+0000 7fae511a9640 1 -- 172.21.15.107:0/1114000219 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae4c1a4b70 con 0x7fae4c100700 2024-05-09T08:40:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.254+0000 7fae33fff640 1 -- 172.21.15.107:0/1114000219 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fae3c0041d0 con 0x7fae4c100700 2024-05-09T08:40:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.254+0000 7fae53f9a740 1 Processor -- start 2024-05-09T08:40:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.254+0000 7fae53f9a740 1 -- start start 2024-05-09T08:40:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae53f9a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4be160 0x55d49e4be540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae53f9a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4bea80 0x55d49e4cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae53f9a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4cd680 0x55d49e4cfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae53f9a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d49e423d00 con 0x55d49e4be160 2024-05-09T08:40:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae53f9a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d49e3cb4c0 con 0x55d49e4bea80 2024-05-09T08:40:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae53f9a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d49e41fcc0 con 0x55d49e4cd680 2024-05-09T08:40:30.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae4b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4be160 0x55d49e4be540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae4b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4be160 0x55d49e4be540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44098/0 (socket says 172.21.15.107:44098) 2024-05-09T08:40:30.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae4b59e640 1 -- 172.21.15.107:0/3840022767 learned_addr learned my addr 172.21.15.107:0/3840022767 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:30.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae4ad9d640 1 --2- 172.21.15.107:0/3840022767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4bea80 0x55d49e4cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae4a59c640 1 --2- 172.21.15.107:0/3840022767 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4cd680 0x55d49e4cfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae4b59e640 1 -- 172.21.15.107:0/3840022767 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4cd680 msgr2=0x55d49e4cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:30.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae33fff640 1 --2- 172.21.15.107:0/1114000219 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae340a2d00 0x7fae340a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae4b59e640 1 --2- 172.21.15.107:0/3840022767 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4cd680 0x55d49e4cfa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae4b59e640 1 -- 172.21.15.107:0/3840022767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4bea80 msgr2=0x55d49e4cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:30.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae4b59e640 1 --2- 172.21.15.107:0/3840022767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4bea80 0x55d49e4cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae4b59e640 1 -- 172.21.15.107:0/3840022767 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d49e2017d0 con 0x55d49e4be160 2024-05-09T08:40:30.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae4ad9d640 1 --2- 172.21.15.107:0/3840022767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4bea80 0x55d49e4cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:40:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.255+0000 7fae33fff640 1 -- 172.21.15.107:0/1114000219 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fae3c0cf220 con 0x7fae4c100700 2024-05-09T08:40:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.256+0000 7fae4b59e640 1 --2- 172.21.15.107:0/1114000219 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae340a2d00 0x7fae340a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.256+0000 7fae4b59e640 1 --2- 172.21.15.107:0/3840022767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4be160 0x55d49e4be540 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fae44035000 tx=0x7fae44009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7f25daa3207093 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.256+0000 7fae317fa640 1 -- 172.21.15.107:0/3840022767 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4404c070 con 0x55d49e4be160 2024-05-09T08:40:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.256+0000 7fae317fa640 1 -- 172.21.15.107:0/3840022767 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae44036040 con 0x55d49e4be160 2024-05-09T08:40:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.256+0000 7fae317fa640 1 -- 172.21.15.107:0/3840022767 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae440309a0 con 0x55d49e4be160 2024-05-09T08:40:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.257+0000 7fae53f9a740 1 -- 172.21.15.107:0/3840022767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4be160 msgr2=0x55d49e4be540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.257+0000 7fae53f9a740 1 --2- 172.21.15.107:0/3840022767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4be160 0x55d49e4be540 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fae44035000 tx=0x7fae44009a20 comp rx=0 tx=0).stop 2024-05-09T08:40:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.257+0000 7fae4b59e640 1 --2- 172.21.15.107:0/1114000219 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae340a2d00 0x7fae340a51c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fae44005cc0 tx=0x7fae4404a000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.257+0000 7fae53f9a740 1 -- 172.21.15.107:0/3840022767 shutdown_connections 2024-05-09T08:40:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.257+0000 7fae53f9a740 1 --2- 172.21.15.107:0/3840022767 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4cd680 0x55d49e4cfa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.257+0000 7fae53f9a740 1 --2- 172.21.15.107:0/3840022767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4bea80 0x55d49e4cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.257+0000 7fae53f9a740 1 --2- 172.21.15.107:0/3840022767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4be160 0x55d49e4be540 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.257+0000 7fae53f9a740 1 -- 172.21.15.107:0/3840022767 >> 172.21.15.107:0/3840022767 conn(0x55d49e4ba5a0 msgr2=0x55d49e4ba980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.257+0000 7fae53f9a740 1 -- 172.21.15.107:0/3840022767 shutdown_connections 2024-05-09T08:40:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.257+0000 7fae53f9a740 1 -- 172.21.15.107:0/3840022767 wait complete. 2024-05-09T08:40:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.257+0000 7fae53f9a740 1 -- start start 2024-05-09T08:40:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.257+0000 7fae53f9a740 1 Processor -- start 2024-05-09T08:40:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae53f9a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4be160 0x55d49e4ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae53f9a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4bea80 0x55d49e4caae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae53f9a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4cd680 0x55d49e4cb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae53f9a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d49e4d04c0 con 0x55d49e4be160 2024-05-09T08:40:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae53f9a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d49e3deb90 con 0x55d49e4cd680 2024-05-09T08:40:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae53f9a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d49e4d0640 con 0x55d49e4bea80 2024-05-09T08:40:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae53f9a740 10 client.0 _ll_register_callbacks cb 0x55d49e41ada0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae4ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4be160 0x55d49e4ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae4ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4be160 0x55d49e4ca5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44104/0 (socket says 172.21.15.107:44104) 2024-05-09T08:40:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae4ad9d640 1 -- 172.21.15.107:0/156501574 learned_addr learned my addr 172.21.15.107:0/156501574 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae4a59c640 1 --2- 172.21.15.107:0/156501574 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4bea80 0x55d49e4caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae4b59e640 1 --2- 172.21.15.107:0/156501574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4cd680 0x55d49e4cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae4ad9d640 1 -- 172.21.15.107:0/156501574 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4bea80 msgr2=0x55d49e4caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae4ad9d640 1 --2- 172.21.15.107:0/156501574 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4bea80 0x55d49e4caae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae4ad9d640 1 -- 172.21.15.107:0/156501574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4cd680 msgr2=0x55d49e4cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae4ad9d640 1 --2- 172.21.15.107:0/156501574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4cd680 0x55d49e4cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae4ad9d640 1 -- 172.21.15.107:0/156501574 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d49e1373f0 con 0x55d49e4be160 2024-05-09T08:40:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae4ad9d640 1 --2- 172.21.15.107:0/156501574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4be160 0x55d49e4ca5a0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fae3c009610 tx=0x7fae3c09b220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba8d2fde2c65c0a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae2eff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae2eff5640 1 -- 172.21.15.107:0/156501574 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3c09b390 con 0x55d49e4be160 2024-05-09T08:40:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae2eff5640 1 -- 172.21.15.107:0/156501574 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fae3c09a7e0 con 0x55d49e4be160 2024-05-09T08:40:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae53f9a740 1 -- 172.21.15.107:0/156501574 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae44036040 con 0x55d49e4be160 2024-05-09T08:40:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae2eff5640 1 -- 172.21.15.107:0/156501574 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fae3c09f740 con 0x55d49e4be160 2024-05-09T08:40:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae53f9a740 10 client.17793 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.258+0000 7fae53f9a740 1 -- 172.21.15.107:0/156501574 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d49e0f70f0 con 0x55d49e4be160 2024-05-09T08:40:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.259+0000 7fae2eff5640 1 -- 172.21.15.107:0/156501574 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae44036040 con 0x55d49e4be160 2024-05-09T08:40:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.259+0000 7fae2eff5640 1 -- 172.21.15.107:0/156501574 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fae3c0b41f0 con 0x55d49e4be160 2024-05-09T08:40:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.259+0000 7fae53f9a740 10 client.17793 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.259+0000 7fae53f9a740 10 client.17793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:40:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.259+0000 7fae53f9a740 20 client.17793 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.259+0000 7fae53f9a740 1 --2- 172.21.15.107:0/156501574 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d49e4ddab0 0x55d49e4dff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.259+0000 7fae53f9a740 4 client.17793 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:40:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.259+0000 7fae53f9a740 1 -- 172.21.15.107:0/156501574 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d49e431430 con 0x55d49e4ddab0 2024-05-09T08:40:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.259+0000 7fae4a59c640 1 --2- 172.21.15.107:0/156501574 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d49e4ddab0 0x55d49e4dff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.260+0000 7fae2eff5640 10 client.17793 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.260+0000 7fae4a59c640 1 --2- 172.21.15.107:0/156501574 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d49e4ddab0 0x55d49e4dff00 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d24f94cd844e3fe server_cookie=cac671c6299895b8 in_seq=0 out_seq=0 2024-05-09T08:40:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.260+0000 7fae2eff5640 1 -- 172.21.15.107:0/156501574 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d49e431430 con 0x55d49e4ddab0 2024-05-09T08:40:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.260+0000 7fae2eff5640 10 client.17793 handle_command_reply: tid=0 2024-05-09T08:40:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.260+0000 7fae53f9a740 1 client.17793 shutdown 2024-05-09T08:40:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.260+0000 7fae53f9a740 1 -- 172.21.15.107:0/156501574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4be160 msgr2=0x55d49e4ca5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.260+0000 7fae53f9a740 1 --2- 172.21.15.107:0/156501574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4be160 0x55d49e4ca5a0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fae3c009610 tx=0x7fae3c09b220 comp rx=0 tx=0).stop 2024-05-09T08:40:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.261+0000 7fae53f9a740 1 -- 172.21.15.107:0/156501574 shutdown_connections 2024-05-09T08:40:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.261+0000 7fae53f9a740 1 --2- 172.21.15.107:0/156501574 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d49e4ddab0 0x55d49e4dff00 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.261+0000 7fae2eff5640 0 client.17793 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.261+0000 7fae53f9a740 1 --2- 172.21.15.107:0/156501574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4cd680 0x55d49e4cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.261+0000 7fae53f9a740 1 --2- 172.21.15.107:0/156501574 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4bea80 0x55d49e4caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.261+0000 7fae53f9a740 1 --2- 172.21.15.107:0/156501574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4be160 0x55d49e4ca5a0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.261+0000 7fae53f9a740 1 -- 172.21.15.107:0/156501574 >> 172.21.15.107:0/156501574 conn(0x55d49e4ba5a0 msgr2=0x55d49e4c0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.261+0000 7fae53f9a740 1 -- 172.21.15.107:0/156501574 shutdown_connections 2024-05-09T08:40:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.261+0000 7fae53f9a740 1 -- 172.21.15.107:0/156501574 wait complete. 2024-05-09T08:40:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.261+0000 7fae53f9a740 20 client.17793 trim_cache size 0 max 16384 2024-05-09T08:40:30.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.280+0000 7fae53f9a740 1 Processor -- start 2024-05-09T08:40:30.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.280+0000 7fae53f9a740 1 -- start start 2024-05-09T08:40:30.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.280+0000 7fae53f9a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4ba5a0 0x55d49e4d93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.280+0000 7fae53f9a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4be160 0x55d49e4cac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.280+0000 7fae53f9a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4bea80 0x55d49e4e5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.280+0000 7fae53f9a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c1af320 con 0x55d49e4bea80 2024-05-09T08:40:30.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.280+0000 7fae53f9a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c1acbf0 con 0x55d49e4be160 2024-05-09T08:40:30.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.280+0000 7fae53f9a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c1aaa60 con 0x55d49e4ba5a0 2024-05-09T08:40:30.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.281+0000 7fae4b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4bea80 0x55d49e4e5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.281+0000 7fae4b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4bea80 0x55d49e4e5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44120/0 (socket says 172.21.15.107:44120) 2024-05-09T08:40:30.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.281+0000 7fae4b59e640 1 -- 172.21.15.107:0/1434143977 learned_addr learned my addr 172.21.15.107:0/1434143977 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:30.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.281+0000 7fae4ad9d640 1 --2- 172.21.15.107:0/1434143977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4ba5a0 0x55d49e4d93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.281+0000 7fae4a59c640 1 --2- 172.21.15.107:0/1434143977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4be160 0x55d49e4cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.281+0000 7fae4b59e640 1 -- 172.21.15.107:0/1434143977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4ba5a0 msgr2=0x55d49e4d93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.281+0000 7fae4b59e640 1 --2- 172.21.15.107:0/1434143977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4ba5a0 0x55d49e4d93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.281+0000 7fae4b59e640 1 -- 172.21.15.107:0/1434143977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4be160 msgr2=0x55d49e4cac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.281+0000 7fae4b59e640 1 --2- 172.21.15.107:0/1434143977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4be160 0x55d49e4cac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.281+0000 7fae4b59e640 1 -- 172.21.15.107:0/1434143977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d49e4ba3d0 con 0x55d49e4bea80 2024-05-09T08:40:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.281+0000 7fae4a59c640 1 --2- 172.21.15.107:0/1434143977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4be160 0x55d49e4cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:40:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.281+0000 7fae4b59e640 1 --2- 172.21.15.107:0/1434143977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4bea80 0x55d49e4e5780 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fae44005e30 tx=0x7fae44051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4420c5d0f6b46de5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.281+0000 7fae30ff9640 1 -- 172.21.15.107:0/1434143977 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae44060070 con 0x55d49e4bea80 2024-05-09T08:40:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.281+0000 7fae30ff9640 1 -- 172.21.15.107:0/1434143977 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae44036040 con 0x55d49e4bea80 2024-05-09T08:40:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.281+0000 7fae30ff9640 1 -- 172.21.15.107:0/1434143977 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4404c040 con 0x55d49e4bea80 2024-05-09T08:40:30.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.282+0000 7fae53f9a740 1 -- 172.21.15.107:0/1434143977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4bea80 msgr2=0x55d49e4e5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.282+0000 7fae53f9a740 1 --2- 172.21.15.107:0/1434143977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4bea80 0x55d49e4e5780 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fae44005e30 tx=0x7fae44051000 comp rx=0 tx=0).stop 2024-05-09T08:40:30.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.282+0000 7fae53f9a740 1 -- 172.21.15.107:0/1434143977 shutdown_connections 2024-05-09T08:40:30.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.282+0000 7fae53f9a740 1 --2- 172.21.15.107:0/1434143977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4bea80 0x55d49e4e5780 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.282+0000 7fae53f9a740 1 --2- 172.21.15.107:0/1434143977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4be160 0x55d49e4cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.282+0000 7fae53f9a740 1 --2- 172.21.15.107:0/1434143977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4ba5a0 0x55d49e4d93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.282+0000 7fae53f9a740 1 -- 172.21.15.107:0/1434143977 >> 172.21.15.107:0/1434143977 conn(0x55d49e4b59e0 msgr2=0x55d49e4ca560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:30.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.282+0000 7fae53f9a740 1 -- 172.21.15.107:0/1434143977 shutdown_connections 2024-05-09T08:40:30.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.282+0000 7fae53f9a740 1 -- 172.21.15.107:0/1434143977 wait complete. 2024-05-09T08:40:30.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae53f9a740 1 -- start start 2024-05-09T08:40:30.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae53f9a740 1 Processor -- start 2024-05-09T08:40:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae53f9a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4ba5a0 0x55d49e4d1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae53f9a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4be160 0x55d49e4d1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae53f9a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4bea80 0x55d49e4d1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae53f9a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c1ab0b0 con 0x55d49e4ba5a0 2024-05-09T08:40:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae53f9a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c1ac520 con 0x55d49e4be160 2024-05-09T08:40:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae53f9a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c1ab780 con 0x55d49e4bea80 2024-05-09T08:40:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae53f9a740 10 client.0 _ll_register_callbacks cb 0x7fae3c004540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae4ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4ba5a0 0x55d49e4d1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae4ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4ba5a0 0x55d49e4d1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44122/0 (socket says 172.21.15.107:44122) 2024-05-09T08:40:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae4ad9d640 1 -- 172.21.15.107:0/469977297 learned_addr learned my addr 172.21.15.107:0/469977297 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae4b59e640 1 --2- 172.21.15.107:0/469977297 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4bea80 0x55d49e4d1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae4a59c640 1 --2- 172.21.15.107:0/469977297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4be160 0x55d49e4d1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae4ad9d640 1 -- 172.21.15.107:0/469977297 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4bea80 msgr2=0x55d49e4d1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae4ad9d640 1 --2- 172.21.15.107:0/469977297 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4bea80 0x55d49e4d1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae4ad9d640 1 -- 172.21.15.107:0/469977297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4be160 msgr2=0x55d49e4d1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae4ad9d640 1 --2- 172.21.15.107:0/469977297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4be160 0x55d49e4d1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae4ad9d640 1 -- 172.21.15.107:0/469977297 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d49e5430c0 con 0x55d49e4ba5a0 2024-05-09T08:40:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae4ad9d640 1 --2- 172.21.15.107:0/469977297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4ba5a0 0x55d49e4d1430 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fae3c0045f0 tx=0x7fae3c097c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b253d8a8741d9e51 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.283+0000 7fae2eff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.284+0000 7fae2eff5640 1 -- 172.21.15.107:0/469977297 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3c097630 con 0x55d49e4ba5a0 2024-05-09T08:40:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.284+0000 7fae2eff5640 1 -- 172.21.15.107:0/469977297 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fae3c0972e0 con 0x55d49e4ba5a0 2024-05-09T08:40:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.284+0000 7fae53f9a740 1 -- 172.21.15.107:0/469977297 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae44036040 con 0x55d49e4ba5a0 2024-05-09T08:40:30.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.284+0000 7fae2eff5640 1 -- 172.21.15.107:0/469977297 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fae3c0b8050 con 0x55d49e4ba5a0 2024-05-09T08:40:30.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.284+0000 7fae53f9a740 10 client.17799 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:30.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.284+0000 7fae53f9a740 1 -- 172.21.15.107:0/469977297 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d49e544320 con 0x55d49e4ba5a0 2024-05-09T08:40:30.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.284+0000 7fae2eff5640 1 -- 172.21.15.107:0/469977297 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae44036040 con 0x55d49e4ba5a0 2024-05-09T08:40:30.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.284+0000 7fae2eff5640 1 -- 172.21.15.107:0/469977297 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fae3c0a08d0 con 0x55d49e4ba5a0 2024-05-09T08:40:30.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.284+0000 7fae53f9a740 10 client.17799 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:30.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.284+0000 7fae53f9a740 10 client.17799 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:40:30.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.284+0000 7fae53f9a740 20 client.17799 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:30.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.284+0000 7fae53f9a740 1 --2- 172.21.15.107:0/469977297 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d49e4cd680 0x55d49e5468c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:30.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.284+0000 7fae53f9a740 4 client.17799 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:40:30.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.284+0000 7fae53f9a740 1 -- 172.21.15.107:0/469977297 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d49e42b780 con 0x55d49e4cd680 2024-05-09T08:40:30.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae4a59c640 1 --2- 172.21.15.107:0/469977297 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d49e4cd680 0x55d49e5468c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:30.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae2eff5640 10 client.17799 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:30.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae4a59c640 1 --2- 172.21.15.107:0/469977297 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d49e4cd680 0x55d49e5468c0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c97cd4bb9ef4cfdd server_cookie=7ffa7f522d935ad in_seq=0 out_seq=0 2024-05-09T08:40:30.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae2eff5640 1 -- 172.21.15.107:0/469977297 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d49e42b780 con 0x55d49e4cd680 2024-05-09T08:40:30.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae2eff5640 10 client.17799 handle_command_reply: tid=0 2024-05-09T08:40:30.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae53f9a740 1 client.17799 shutdown 2024-05-09T08:40:30.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae53f9a740 1 -- 172.21.15.107:0/469977297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4ba5a0 msgr2=0x55d49e4d1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae53f9a740 1 --2- 172.21.15.107:0/469977297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4ba5a0 0x55d49e4d1430 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fae3c0045f0 tx=0x7fae3c097c30 comp rx=0 tx=0).stop 2024-05-09T08:40:30.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae53f9a740 1 -- 172.21.15.107:0/469977297 shutdown_connections 2024-05-09T08:40:30.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae53f9a740 1 --2- 172.21.15.107:0/469977297 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d49e4cd680 0x55d49e5468c0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae53f9a740 1 --2- 172.21.15.107:0/469977297 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d49e4bea80 0x55d49e4d1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae2eff5640 0 client.17799 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:30.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae53f9a740 1 --2- 172.21.15.107:0/469977297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d49e4be160 0x55d49e4d1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae53f9a740 1 --2- 172.21.15.107:0/469977297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d49e4ba5a0 0x55d49e4d1430 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae53f9a740 1 -- 172.21.15.107:0/469977297 >> 172.21.15.107:0/469977297 conn(0x55d49e4b59e0 msgr2=0x55d49e4c0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:30.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae53f9a740 1 -- 172.21.15.107:0/469977297 shutdown_connections 2024-05-09T08:40:30.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae53f9a740 1 -- 172.21.15.107:0/469977297 wait complete. 2024-05-09T08:40:30.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.285+0000 7fae53f9a740 20 client.17799 trim_cache size 0 max 16384 2024-05-09T08:40:30.295 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:40:30.295 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:40:30.295 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:40:30.295 INFO:teuthology.orchestra.run.smithi107.stdout: "780ca629-fec8-45ed-8a09-b26053188d11": { 2024-05-09T08:40:30.295 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:40:30.295 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "780ca629-fec8-45ed-8a09-b26053188d11", 2024-05-09T08:40:30.295 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:40:30.296 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:40:30.296 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:40:30.296 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:40:30.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.286+0000 7fae511a9640 1 -- 172.21.15.107:0/1114000219 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae340a2d00 msgr2=0x7fae340a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.286+0000 7fae511a9640 1 --2- 172.21.15.107:0/1114000219 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae340a2d00 0x7fae340a51c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fae44005cc0 tx=0x7fae4404a000 comp rx=0 tx=0).stop 2024-05-09T08:40:30.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.286+0000 7fae511a9640 1 -- 172.21.15.107:0/1114000219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c100700 msgr2=0x7fae4c198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:30.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.286+0000 7fae511a9640 1 --2- 172.21.15.107:0/1114000219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c100700 0x7fae4c198000 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00d950 tx=0x7fae3c00de10 comp rx=0 tx=0).stop 2024-05-09T08:40:30.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.286+0000 7fae511a9640 1 -- 172.21.15.107:0/1114000219 shutdown_connections 2024-05-09T08:40:30.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.286+0000 7fae511a9640 1 --2- 172.21.15.107:0/1114000219 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae340a2d00 0x7fae340a51c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.286+0000 7fae511a9640 1 --2- 172.21.15.107:0/1114000219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae4c106710 0x7fae4c19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.286+0000 7fae511a9640 1 --2- 172.21.15.107:0/1114000219 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae4c1010c0 0x7fae4c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.286+0000 7fae511a9640 1 --2- 172.21.15.107:0/1114000219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae4c100700 0x7fae4c198000 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:30.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.286+0000 7fae511a9640 1 -- 172.21.15.107:0/1114000219 >> 172.21.15.107:0/1114000219 conn(0x7fae4c077830 msgr2=0x7fae4c068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:30.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.286+0000 7fae511a9640 1 -- 172.21.15.107:0/1114000219 shutdown_connections 2024-05-09T08:40:30.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:40:30.286+0000 7fae511a9640 1 -- 172.21.15.107:0/1114000219 wait complete. 2024-05-09T08:40:30.298 INFO:tasks.cephfs.filesystem:scrub status for tag:780ca629-fec8-45ed-8a09-b26053188d11 - {'path': '/', 'tag': '780ca629-fec8-45ed-8a09-b26053188d11', 'options': 'recursive,force'} 2024-05-09T08:40:30.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:30 smithi107 ceph-mon[30122]: pgmap v219: 129 pgs: 129 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:40:30.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:30 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:40:30.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:30 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:30.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:30 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:40:30.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:30 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:30.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:30 smithi157 ceph-mon[33614]: pgmap v219: 129 pgs: 129 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:40:30.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:30 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:40:30.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:30 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:30.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:30 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:40:30.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:30 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:30.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:30 smithi110 ceph-mon[33622]: pgmap v219: 129 pgs: 129 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:40:30.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:30 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:40:30.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:30 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:30.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:30 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:40:30.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:30 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:31 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:40:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:31 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:31 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:40:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:31 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:40:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:31 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:31 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:40:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:31 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:31 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:40:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:31 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:31.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:31 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:31.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:31 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:40:31.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:31 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:31.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:31 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:40:31.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:31 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:40:31.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:31 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:31.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:31 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:40:31.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:31 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:31.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:31 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:40:31.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:31 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:31.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:31 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:31.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:31 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:40:31.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:31 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:31.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:31 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:40:31.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:31 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:40:31.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:31 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:31.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:31 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:40:31.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:31 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:31.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:31 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:40:31.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:31 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:31.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:31 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:40:32.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:32 smithi107 ceph-mon[30122]: pgmap v220: 129 pgs: 129 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:40:32.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:32 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:40:32.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:32 smithi157 ceph-mon[33614]: pgmap v220: 129 pgs: 129 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:40:32.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:32 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:40:32.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:32 smithi110 ceph-mon[33622]: pgmap v220: 129 pgs: 129 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-05-09T08:40:32.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:32 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:40:33.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:33 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:40:33.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:33 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:40:33.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:33 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:40:33.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:33 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:40:33.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:33 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:40:33.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:33 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:40:33.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:33 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:40:33.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:33 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:40:33.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:33 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:40:34.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:34 smithi107 ceph-mon[30122]: pgmap v221: 129 pgs: 129 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 37 MiB/s wr, 59 op/s 2024-05-09T08:40:34.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:34 smithi157 ceph-mon[33614]: pgmap v221: 129 pgs: 129 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 37 MiB/s wr, 59 op/s 2024-05-09T08:40:34.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:34 smithi110 ceph-mon[33622]: pgmap v221: 129 pgs: 129 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 37 MiB/s wr, 59 op/s 2024-05-09T08:40:36.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:36 smithi107 ceph-mon[30122]: pgmap v222: 129 pgs: 129 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 25 MiB/s wr, 52 op/s 2024-05-09T08:40:36.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:36 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:40:36.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:36 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:40:36.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:36 smithi157 ceph-mon[33614]: pgmap v222: 129 pgs: 129 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 25 MiB/s wr, 52 op/s 2024-05-09T08:40:36.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:36 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:40:36.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:36 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:40:36.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:36 smithi110 ceph-mon[33622]: pgmap v222: 129 pgs: 129 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 25 MiB/s wr, 52 op/s 2024-05-09T08:40:36.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:36 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:40:36.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:36 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:40:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.217+0000 7f60fda31640 1 -- 172.21.15.107:0/4080120329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60f8106c10 msgr2=0x7f60f810e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.217+0000 7f60fda31640 1 --2- 172.21.15.107:0/4080120329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60f8106c10 0x7f60f810e790 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009ca0 tx=0x7f60f002a8c0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.217+0000 7f60fda31640 1 -- 172.21.15.107:0/4080120329 shutdown_connections 2024-05-09T08:40:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.217+0000 7f60fda31640 1 --2- 172.21.15.107:0/4080120329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60f8106c10 0x7f60f810e790 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.217+0000 7f60fda31640 1 --2- 172.21.15.107:0/4080120329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60f8100950 0x7f60f81064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.217+0000 7f60fda31640 1 --2- 172.21.15.107:0/4080120329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60f80fffa0 0x7f60f8100380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.217+0000 7f60fda31640 1 -- 172.21.15.107:0/4080120329 >> 172.21.15.107:0/4080120329 conn(0x7f60f8076cc0 msgr2=0x7f60f80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.217+0000 7f60fda31640 1 -- 172.21.15.107:0/4080120329 shutdown_connections 2024-05-09T08:40:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.217+0000 7f60fda31640 1 -- 172.21.15.107:0/4080120329 wait complete. 2024-05-09T08:40:37.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.218+0000 7f60fda31640 1 Processor -- start 2024-05-09T08:40:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.218+0000 7f60fda31640 1 -- start start 2024-05-09T08:40:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.218+0000 7f60fda31640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60f80fffa0 0x7f60f8197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.218+0000 7f60fda31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60f8106c10 0x7f60f8198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.218+0000 7f60fda31640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60f819c750 0x7f60f819cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.218+0000 7f60fda31640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f60f8111b20 con 0x7f60f8106c10 2024-05-09T08:40:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.218+0000 7f60fda31640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f60f81119a0 con 0x7f60f80fffa0 2024-05-09T08:40:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.218+0000 7f60fda31640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f60f8111ca0 con 0x7f60f819c750 2024-05-09T08:40:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60f8106c10 0x7f60f8198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60f8106c10 0x7f60f8198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47176/0 (socket says 172.21.15.107:47176) 2024-05-09T08:40:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60effff640 1 -- 172.21.15.107:0/547432328 learned_addr learned my addr 172.21.15.107:0/547432328 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60f77fe640 1 --2- 172.21.15.107:0/547432328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60f819c750 0x7f60f819cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60f6ffd640 1 --2- 172.21.15.107:0/547432328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60f80fffa0 0x7f60f8197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60effff640 1 -- 172.21.15.107:0/547432328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60f819c750 msgr2=0x7f60f819cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60effff640 1 --2- 172.21.15.107:0/547432328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60f819c750 0x7f60f819cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60effff640 1 -- 172.21.15.107:0/547432328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60f80fffa0 msgr2=0x7f60f8197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60effff640 1 --2- 172.21.15.107:0/547432328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60f80fffa0 0x7f60f8197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60effff640 1 -- 172.21.15.107:0/547432328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f819d240 con 0x7f60f8106c10 2024-05-09T08:40:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60f6ffd640 1 --2- 172.21.15.107:0/547432328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60f80fffa0 0x7f60f8197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:40:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60effff640 1 --2- 172.21.15.107:0/547432328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60f8106c10 0x7f60f8198420 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f60e000c8c0 tx=0x7f60e000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60fca2f640 1 -- 172.21.15.107:0/547432328 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e0007bb0 con 0x7f60f8106c10 2024-05-09T08:40:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60fca2f640 1 -- 172.21.15.107:0/547432328 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60e0007d50 con 0x7f60f8106c10 2024-05-09T08:40:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60fda31640 1 -- 172.21.15.107:0/547432328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60f819d530 con 0x7f60f8106c10 2024-05-09T08:40:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60fca2f640 1 -- 172.21.15.107:0/547432328 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e00187c0 con 0x7f60f8106c10 2024-05-09T08:40:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.219+0000 7f60fda31640 1 -- 172.21.15.107:0/547432328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60f81a4d20 con 0x7f60f8106c10 2024-05-09T08:40:37.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.222+0000 7f610081a740 1 Processor -- start 2024-05-09T08:40:37.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.222+0000 7f610081a740 1 -- start start 2024-05-09T08:40:37.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.222+0000 7f60fca2f640 1 -- 172.21.15.107:0/547432328 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f60e000f450 con 0x7f60f8106c10 2024-05-09T08:40:37.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.222+0000 7f610081a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555749011450 0x555749011830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.222+0000 7f610081a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555749011d70 0x55574901c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.222+0000 7f610081a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55574901c630 0x55574901ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.222+0000 7f610081a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555748f76340 con 0x55574901c630 2024-05-09T08:40:37.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.222+0000 7f610081a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555748f1e290 con 0x555749011450 2024-05-09T08:40:37.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.222+0000 7f610081a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x555748f72300 con 0x555749011d70 2024-05-09T08:40:37.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.222+0000 7f60effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55574901c630 0x55574901ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.222+0000 7f60effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55574901c630 0x55574901ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47188/0 (socket says 172.21.15.107:47188) 2024-05-09T08:40:37.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.222+0000 7f60effff640 1 -- 172.21.15.107:0/1354122810 learned_addr learned my addr 172.21.15.107:0/1354122810 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:37.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.222+0000 7f60f6ffd640 1 --2- 172.21.15.107:0/1354122810 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555749011d70 0x55574901c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.223+0000 7f60f77fe640 1 --2- 172.21.15.107:0/1354122810 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555749011450 0x555749011830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.223+0000 7f60effff640 1 -- 172.21.15.107:0/1354122810 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555749011d70 msgr2=0x55574901c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:37.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.223+0000 7f60effff640 1 --2- 172.21.15.107:0/1354122810 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555749011d70 0x55574901c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.223+0000 7f60effff640 1 -- 172.21.15.107:0/1354122810 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555749011450 msgr2=0x555749011830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:37.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.223+0000 7f60effff640 1 --2- 172.21.15.107:0/1354122810 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555749011450 0x555749011830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.223+0000 7f60effff640 1 -- 172.21.15.107:0/1354122810 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555748f428e0 con 0x55574901c630 2024-05-09T08:40:37.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.223+0000 7f60fca2f640 1 --2- 172.21.15.107:0/547432328 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f60e40a2da0 0x7f60e40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.223+0000 7f60effff640 1 --2- 172.21.15.107:0/1354122810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55574901c630 0x55574901ea20 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f60e00ccb60 tx=0x7f60e00d5cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb6068ac5f8cf66 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:37.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.223+0000 7f60f77fe640 1 --2- 172.21.15.107:0/547432328 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f60e40a2da0 0x7f60e40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.223+0000 7f60edffb640 1 -- 172.21.15.107:0/1354122810 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e0097c60 con 0x55574901c630 2024-05-09T08:40:37.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.223+0000 7f60fca2f640 1 -- 172.21.15.107:0/547432328 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f60e00cf850 con 0x7f60f8106c10 2024-05-09T08:40:37.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.223+0000 7f60edffb640 1 -- 172.21.15.107:0/1354122810 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60e009c0c0 con 0x55574901c630 2024-05-09T08:40:37.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.223+0000 7f60edffb640 1 -- 172.21.15.107:0/1354122810 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e0067680 con 0x55574901c630 2024-05-09T08:40:37.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.224+0000 7f610081a740 1 -- 172.21.15.107:0/1354122810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55574901c630 msgr2=0x55574901ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:37.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.224+0000 7f610081a740 1 --2- 172.21.15.107:0/1354122810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55574901c630 0x55574901ea20 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f60e00ccb60 tx=0x7f60e00d5cd0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.224+0000 7f60f77fe640 1 --2- 172.21.15.107:0/547432328 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f60e40a2da0 0x7f60e40a5260 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009430 tx=0x7f60f0005ef0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:37.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.224+0000 7f610081a740 1 -- 172.21.15.107:0/1354122810 shutdown_connections 2024-05-09T08:40:37.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.224+0000 7f610081a740 1 --2- 172.21.15.107:0/1354122810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55574901c630 0x55574901ea20 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.224+0000 7f610081a740 1 --2- 172.21.15.107:0/1354122810 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555749011d70 0x55574901c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.224+0000 7f610081a740 1 --2- 172.21.15.107:0/1354122810 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555749011450 0x555749011830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.224+0000 7f610081a740 1 -- 172.21.15.107:0/1354122810 >> 172.21.15.107:0/1354122810 conn(0x55574900d960 msgr2=0x55574900dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:37.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.224+0000 7f610081a740 1 -- 172.21.15.107:0/1354122810 shutdown_connections 2024-05-09T08:40:37.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.224+0000 7f610081a740 1 -- 172.21.15.107:0/1354122810 wait complete. 2024-05-09T08:40:37.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.225+0000 7f610081a740 1 -- start start 2024-05-09T08:40:37.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.225+0000 7f610081a740 1 Processor -- start 2024-05-09T08:40:37.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.225+0000 7f610081a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555749011450 0x555749019580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.225+0000 7f610081a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011d70 0x555749019ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.225+0000 7f610081a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55574901c630 0x55574901a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.225+0000 7f610081a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55574901f260 con 0x555749011d70 2024-05-09T08:40:37.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.225+0000 7f610081a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555748f31790 con 0x55574901c630 2024-05-09T08:40:37.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.225+0000 7f610081a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55574901f3e0 con 0x555749011450 2024-05-09T08:40:37.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.225+0000 7f610081a740 10 client.0 _ll_register_callbacks cb 0x555748f6ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:37.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.225+0000 7f60effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011d70 0x555749019ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.225+0000 7f60f6ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555749011450 0x555749019580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.225+0000 7f60effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011d70 0x555749019ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47198/0 (socket says 172.21.15.107:47198) 2024-05-09T08:40:37.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.225+0000 7f60effff640 1 -- 172.21.15.107:0/3455290784 learned_addr learned my addr 172.21.15.107:0/3455290784 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:37.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.225+0000 7f60f77fe640 1 --2- 172.21.15.107:0/3455290784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55574901c630 0x55574901a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f60effff640 1 -- 172.21.15.107:0/3455290784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555749011450 msgr2=0x555749019580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:37.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f60effff640 1 --2- 172.21.15.107:0/3455290784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555749011450 0x555749019580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f60effff640 1 -- 172.21.15.107:0/3455290784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55574901c630 msgr2=0x55574901a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:37.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f60effff640 1 --2- 172.21.15.107:0/3455290784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55574901c630 0x55574901a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f60effff640 1 -- 172.21.15.107:0/3455290784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555748c8a210 con 0x555749011d70 2024-05-09T08:40:37.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f60f6ffd640 1 --2- 172.21.15.107:0/3455290784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555749011450 0x555749019580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:40:37.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f60f77fe640 1 --2- 172.21.15.107:0/3455290784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55574901c630 0x55574901a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:40:37.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f60effff640 1 --2- 172.21.15.107:0/3455290784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011d70 0x555749019ac0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f60e00ccca0 tx=0x7f60e0067c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd93813be7e3a17 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:37.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f60d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:37.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f60d6ffd640 1 -- 172.21.15.107:0/3455290784 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e00678b0 con 0x555749011d70 2024-05-09T08:40:37.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f60d6ffd640 1 -- 172.21.15.107:0/3455290784 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f60e0099a60 con 0x555749011d70 2024-05-09T08:40:37.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f610081a740 1 -- 172.21.15.107:0/3455290784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60e009c0c0 con 0x555749011d70 2024-05-09T08:40:37.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f60d6ffd640 1 -- 172.21.15.107:0/3455290784 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f60e0067ae0 con 0x555749011d70 2024-05-09T08:40:37.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f610081a740 10 client.17811 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:37.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f610081a740 1 -- 172.21.15.107:0/3455290784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555748d56620 con 0x555749011d70 2024-05-09T08:40:37.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f60d6ffd640 1 -- 172.21.15.107:0/3455290784 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60e009c0c0 con 0x555749011d70 2024-05-09T08:40:37.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f60d6ffd640 1 -- 172.21.15.107:0/3455290784 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f60e006e030 con 0x555749011d70 2024-05-09T08:40:37.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f610081a740 10 client.17811 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:37.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f610081a740 10 client.17811 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:40:37.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.226+0000 7f610081a740 20 client.17811 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:37.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.227+0000 7f610081a740 1 --2- 172.21.15.107:0/3455290784 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55574902c1d0 0x55574902e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.227+0000 7f610081a740 4 client.17811 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:40:37.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.227+0000 7f610081a740 1 -- 172.21.15.107:0/3455290784 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555748f83a70 con 0x55574902c1d0 2024-05-09T08:40:37.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.227+0000 7f60f6ffd640 1 --2- 172.21.15.107:0/3455290784 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55574902c1d0 0x55574902e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.227+0000 7f60f6ffd640 1 --2- 172.21.15.107:0/3455290784 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55574902c1d0 0x55574902e5c0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc1566ea51dc8d67 server_cookie=f12435b3bbf66b85 in_seq=0 out_seq=0 2024-05-09T08:40:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.227+0000 7f60d6ffd640 10 client.17811 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.228+0000 7f60d6ffd640 1 -- 172.21.15.107:0/3455290784 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555748f83a70 con 0x55574902c1d0 2024-05-09T08:40:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.228+0000 7f60d6ffd640 10 client.17811 handle_command_reply: tid=0 2024-05-09T08:40:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.228+0000 7f610081a740 1 client.17811 shutdown 2024-05-09T08:40:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.228+0000 7f610081a740 1 -- 172.21.15.107:0/3455290784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011d70 msgr2=0x555749019ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.228+0000 7f610081a740 1 --2- 172.21.15.107:0/3455290784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011d70 0x555749019ac0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f60e00ccca0 tx=0x7f60e0067c80 comp rx=0 tx=0).stop 2024-05-09T08:40:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.228+0000 7f610081a740 1 -- 172.21.15.107:0/3455290784 shutdown_connections 2024-05-09T08:40:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.228+0000 7f610081a740 1 --2- 172.21.15.107:0/3455290784 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55574902c1d0 0x55574902e5c0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.228+0000 7f610081a740 1 --2- 172.21.15.107:0/3455290784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55574901c630 0x55574901a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.228+0000 7f610081a740 1 --2- 172.21.15.107:0/3455290784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011d70 0x555749019ac0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.228+0000 7f610081a740 1 --2- 172.21.15.107:0/3455290784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555749011450 0x555749019580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.228+0000 7f610081a740 1 -- 172.21.15.107:0/3455290784 >> 172.21.15.107:0/3455290784 conn(0x55574900d960 msgr2=0x555749013690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.228+0000 7f60d6ffd640 0 client.17811 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:37.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.228+0000 7f610081a740 1 -- 172.21.15.107:0/3455290784 shutdown_connections 2024-05-09T08:40:37.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.228+0000 7f610081a740 1 -- 172.21.15.107:0/3455290784 wait complete. 2024-05-09T08:40:37.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.228+0000 7f610081a740 20 client.17811 trim_cache size 0 max 16384 2024-05-09T08:40:37.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f610081a740 1 Processor -- start 2024-05-09T08:40:37.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f610081a740 1 -- start start 2024-05-09T08:40:37.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f610081a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555749011450 0x55574901bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f610081a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011d70 0x555749018740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f610081a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55574901c630 0x5557490315e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f610081a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f60f81b1450 con 0x555749011d70 2024-05-09T08:40:37.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f610081a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f60f81aed20 con 0x555749011450 2024-05-09T08:40:37.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f610081a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f60f81acb90 con 0x55574901c630 2024-05-09T08:40:37.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f60effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011d70 0x555749018740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f60effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011d70 0x555749018740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47206/0 (socket says 172.21.15.107:47206) 2024-05-09T08:40:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f60effff640 1 -- 172.21.15.107:0/3689457784 learned_addr learned my addr 172.21.15.107:0/3689457784 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f60f77fe640 1 --2- 172.21.15.107:0/3689457784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55574901c630 0x5557490315e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f60effff640 1 -- 172.21.15.107:0/3689457784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55574901c630 msgr2=0x5557490315e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f60effff640 1 --2- 172.21.15.107:0/3689457784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55574901c630 0x5557490315e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f60effff640 1 -- 172.21.15.107:0/3689457784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555749011450 msgr2=0x55574901bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f60effff640 1 --2- 172.21.15.107:0/3689457784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555749011450 0x55574901bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.243+0000 7f60effff640 1 -- 172.21.15.107:0/3689457784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55574900bf00 con 0x555749011d70 2024-05-09T08:40:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.244+0000 7f60effff640 1 --2- 172.21.15.107:0/3689457784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011d70 0x555749018740 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f60e0097e20 tx=0x7f60e0098240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5542e67de171731 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.244+0000 7f60ed7fa640 1 -- 172.21.15.107:0/3689457784 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e0067d40 con 0x555749011d70 2024-05-09T08:40:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.244+0000 7f60ed7fa640 1 -- 172.21.15.107:0/3689457784 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60e009c0c0 con 0x555749011d70 2024-05-09T08:40:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.244+0000 7f60ed7fa640 1 -- 172.21.15.107:0/3689457784 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e009b320 con 0x555749011d70 2024-05-09T08:40:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.245+0000 7f610081a740 1 -- 172.21.15.107:0/3689457784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011d70 msgr2=0x555749018740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.245+0000 7f610081a740 1 --2- 172.21.15.107:0/3689457784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011d70 0x555749018740 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f60e0097e20 tx=0x7f60e0098240 comp rx=0 tx=0).stop 2024-05-09T08:40:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.245+0000 7f610081a740 1 -- 172.21.15.107:0/3689457784 shutdown_connections 2024-05-09T08:40:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.245+0000 7f610081a740 1 --2- 172.21.15.107:0/3689457784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55574901c630 0x5557490315e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.245+0000 7f610081a740 1 --2- 172.21.15.107:0/3689457784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011d70 0x555749018740 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.245+0000 7f610081a740 1 --2- 172.21.15.107:0/3689457784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555749011450 0x55574901bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.245+0000 7f610081a740 1 -- 172.21.15.107:0/3689457784 >> 172.21.15.107:0/3689457784 conn(0x55574900d960 msgr2=0x55574900dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.245+0000 7f610081a740 1 -- 172.21.15.107:0/3689457784 shutdown_connections 2024-05-09T08:40:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.245+0000 7f610081a740 1 -- 172.21.15.107:0/3689457784 wait complete. 2024-05-09T08:40:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.245+0000 7f610081a740 1 -- start start 2024-05-09T08:40:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.245+0000 7f610081a740 1 Processor -- start 2024-05-09T08:40:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.245+0000 7f610081a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011450 0x555749031810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f610081a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555749011d70 0x55574902eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f610081a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55574901c630 0x55574902f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f610081a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f60f81ad1e0 con 0x555749011450 2024-05-09T08:40:37.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f610081a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f60f81ae650 con 0x555749011d70 2024-05-09T08:40:37.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f610081a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f60f81ad8b0 con 0x55574901c630 2024-05-09T08:40:37.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f610081a740 10 client.0 _ll_register_callbacks cb 0x7f60f0009360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:37.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f60f6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011450 0x555749031810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f60f6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011450 0x555749031810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47208/0 (socket says 172.21.15.107:47208) 2024-05-09T08:40:37.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f60f6ffd640 1 -- 172.21.15.107:0/1442598833 learned_addr learned my addr 172.21.15.107:0/1442598833 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:37.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f60effff640 1 --2- 172.21.15.107:0/1442598833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555749011d70 0x55574902eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f60f77fe640 1 --2- 172.21.15.107:0/1442598833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55574901c630 0x55574902f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f60f6ffd640 1 -- 172.21.15.107:0/1442598833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55574901c630 msgr2=0x55574902f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:37.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f60f6ffd640 1 --2- 172.21.15.107:0/1442598833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55574901c630 0x55574902f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f60f6ffd640 1 -- 172.21.15.107:0/1442598833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555749011d70 msgr2=0x55574902eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:37.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f60f6ffd640 1 --2- 172.21.15.107:0/1442598833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555749011d70 0x55574902eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f60f6ffd640 1 -- 172.21.15.107:0/1442598833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55574902f720 con 0x555749011450 2024-05-09T08:40:37.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f60effff640 1 --2- 172.21.15.107:0/1442598833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555749011d70 0x55574902eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:40:37.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f60f6ffd640 1 --2- 172.21.15.107:0/1442598833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011450 0x555749031810 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x55574901f420 tx=0x7f60e8012e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f172a99f77f57068 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:37.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f60d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:37.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f60d6ffd640 1 -- 172.21.15.107:0/1442598833 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e801e910 con 0x555749011450 2024-05-09T08:40:37.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f60d6ffd640 1 -- 172.21.15.107:0/1442598833 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f60e800f950 con 0x555749011450 2024-05-09T08:40:37.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f610081a740 1 -- 172.21.15.107:0/1442598833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60e009c0c0 con 0x555749011450 2024-05-09T08:40:37.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f60d6ffd640 1 -- 172.21.15.107:0/1442598833 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f60e802e050 con 0x555749011450 2024-05-09T08:40:37.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f610081a740 10 client.17817 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:37.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.246+0000 7f610081a740 1 -- 172.21.15.107:0/1442598833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555749092290 con 0x555749011450 2024-05-09T08:40:37.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.247+0000 7f60d6ffd640 1 -- 172.21.15.107:0/1442598833 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60e009c0c0 con 0x555749011450 2024-05-09T08:40:37.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.247+0000 7f60d6ffd640 1 -- 172.21.15.107:0/1442598833 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f60e80048a0 con 0x555749011450 2024-05-09T08:40:37.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.247+0000 7f610081a740 10 client.17817 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:37.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.247+0000 7f610081a740 10 client.17817 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:40:37.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.247+0000 7f610081a740 20 client.17817 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:37.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.247+0000 7f610081a740 1 --2- 172.21.15.107:0/1442598833 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555749092a10 0x555749094e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:37.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.247+0000 7f610081a740 4 client.17817 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:40:37.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.247+0000 7f610081a740 1 -- 172.21.15.107:0/1442598833 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555748f7ddc0 con 0x555749092a10 2024-05-09T08:40:37.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.247+0000 7f60effff640 1 --2- 172.21.15.107:0/1442598833 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555749092a10 0x555749094e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:37.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.248+0000 7f60effff640 1 --2- 172.21.15.107:0/1442598833 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555749092a10 0x555749094e30 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f50738958627cac server_cookie=6999dbed42058e12 in_seq=0 out_seq=0 2024-05-09T08:40:37.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.248+0000 7f60d6ffd640 10 client.17817 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:37.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.264+0000 7f60d6ffd640 1 -- 172.21.15.107:0/1442598833 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555748f7ddc0 con 0x555749092a10 2024-05-09T08:40:37.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.264+0000 7f60d6ffd640 10 client.17817 handle_command_reply: tid=0 2024-05-09T08:40:37.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.264+0000 7f610081a740 1 client.17817 shutdown 2024-05-09T08:40:37.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.264+0000 7f610081a740 1 -- 172.21.15.107:0/1442598833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011450 msgr2=0x555749031810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.264+0000 7f610081a740 1 --2- 172.21.15.107:0/1442598833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011450 0x555749031810 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x55574901f420 tx=0x7f60e8012e10 comp rx=0 tx=0).stop 2024-05-09T08:40:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.264+0000 7f610081a740 1 -- 172.21.15.107:0/1442598833 shutdown_connections 2024-05-09T08:40:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.264+0000 7f610081a740 1 --2- 172.21.15.107:0/1442598833 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555749092a10 0x555749094e30 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.264+0000 7f610081a740 1 --2- 172.21.15.107:0/1442598833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55574901c630 0x55574902f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.264+0000 7f610081a740 1 --2- 172.21.15.107:0/1442598833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555749011d70 0x55574902eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.264+0000 7f60d6ffd640 0 client.17817 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.264+0000 7f610081a740 1 --2- 172.21.15.107:0/1442598833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555749011450 0x555749031810 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.264+0000 7f610081a740 1 -- 172.21.15.107:0/1442598833 >> 172.21.15.107:0/1442598833 conn(0x55574900d960 msgr2=0x555749012e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.264+0000 7f610081a740 1 -- 172.21.15.107:0/1442598833 shutdown_connections 2024-05-09T08:40:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.264+0000 7f610081a740 1 -- 172.21.15.107:0/1442598833 wait complete. 2024-05-09T08:40:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.264+0000 7f610081a740 20 client.17817 trim_cache size 0 max 16384 2024-05-09T08:40:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:40:37.265 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:40:37.265 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:40:37.265 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:40:37.265 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.265+0000 7f60fda31640 1 -- 172.21.15.107:0/547432328 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f60e40a2da0 msgr2=0x7f60e40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:37.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.265+0000 7f60fda31640 1 --2- 172.21.15.107:0/547432328 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f60e40a2da0 0x7f60e40a5260 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009430 tx=0x7f60f0005ef0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.265+0000 7f60fda31640 1 -- 172.21.15.107:0/547432328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60f8106c10 msgr2=0x7f60f8198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:37.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.265+0000 7f60fda31640 1 --2- 172.21.15.107:0/547432328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60f8106c10 0x7f60f8198420 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f60e000c8c0 tx=0x7f60e000cd80 comp rx=0 tx=0).stop 2024-05-09T08:40:37.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.265+0000 7f60fda31640 1 -- 172.21.15.107:0/547432328 shutdown_connections 2024-05-09T08:40:37.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.265+0000 7f60fda31640 1 --2- 172.21.15.107:0/547432328 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f60e40a2da0 0x7f60e40a5260 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.265+0000 7f60fda31640 1 --2- 172.21.15.107:0/547432328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60f819c750 0x7f60f819cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.265+0000 7f60fda31640 1 --2- 172.21.15.107:0/547432328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60f8106c10 0x7f60f8198420 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.265+0000 7f60fda31640 1 --2- 172.21.15.107:0/547432328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60f80fffa0 0x7f60f8197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:37.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.265+0000 7f60fda31640 1 -- 172.21.15.107:0/547432328 >> 172.21.15.107:0/547432328 conn(0x7f60f8076cc0 msgr2=0x7f60f8104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:37.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.265+0000 7f60fda31640 1 -- 172.21.15.107:0/547432328 shutdown_connections 2024-05-09T08:40:37.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:37.265+0000 7f60fda31640 1 -- 172.21.15.107:0/547432328 wait complete. 2024-05-09T08:40:37.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:37 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:40:37.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:37 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:40:38.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:37 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:40:38.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:38 smithi157 ceph-mon[33614]: pgmap v223: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 37 MiB/s wr, 59 op/s 2024-05-09T08:40:38.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:38 smithi110 ceph-mon[33622]: pgmap v223: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 37 MiB/s wr, 59 op/s 2024-05-09T08:40:39.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:38 smithi107 ceph-mon[30122]: pgmap v223: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 37 MiB/s wr, 59 op/s 2024-05-09T08:40:40.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:40 smithi157 ceph-mon[33614]: pgmap v224: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 24 MiB/s wr, 47 op/s 2024-05-09T08:40:40.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:40 smithi110 ceph-mon[33622]: pgmap v224: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 24 MiB/s wr, 47 op/s 2024-05-09T08:40:41.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:40 smithi107 ceph-mon[30122]: pgmap v224: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 24 MiB/s wr, 47 op/s 2024-05-09T08:40:42.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:42 smithi157 ceph-mon[33614]: pgmap v225: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 24 MiB/s wr, 47 op/s 2024-05-09T08:40:42.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:42 smithi110 ceph-mon[33622]: pgmap v225: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 24 MiB/s wr, 47 op/s 2024-05-09T08:40:43.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:42 smithi107 ceph-mon[30122]: pgmap v225: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 24 MiB/s wr, 47 op/s 2024-05-09T08:40:44.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:44 smithi157 ceph-mon[33614]: pgmap v226: 129 pgs: 129 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 37 MiB/s wr, 59 op/s 2024-05-09T08:40:44.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:44 smithi110 ceph-mon[33622]: pgmap v226: 129 pgs: 129 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 37 MiB/s wr, 59 op/s 2024-05-09T08:40:45.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:44 smithi107 ceph-mon[30122]: pgmap v226: 129 pgs: 129 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 37 MiB/s wr, 59 op/s 2024-05-09T08:40:46.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:46 smithi107 ceph-mon[30122]: pgmap v227: 129 pgs: 129 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:40:46.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:46 smithi157 ceph-mon[33614]: pgmap v227: 129 pgs: 129 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:40:46.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:46 smithi110 ceph-mon[33622]: pgmap v227: 129 pgs: 129 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:40:48.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:48 smithi157 ceph-mon[33614]: pgmap v228: 129 pgs: 129 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:40:48.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:48 smithi110 ceph-mon[33622]: pgmap v228: 129 pgs: 129 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:40:49.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:48 smithi107 ceph-mon[30122]: pgmap v228: 129 pgs: 129 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:40:50.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.359+0000 7f6dabb16640 1 -- 172.21.15.107:0/203230101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6da4100760 msgr2=0x7f6da4100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:50.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.359+0000 7f6dabb16640 1 --2- 172.21.15.107:0/203230101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6da4100760 0x7f6da4100be0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f6d94009930 tx=0x7f6d9402a420 comp rx=0 tx=0).stop 2024-05-09T08:40:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.360+0000 7f6dabb16640 1 -- 172.21.15.107:0/203230101 shutdown_connections 2024-05-09T08:40:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.360+0000 7f6dabb16640 1 --2- 172.21.15.107:0/203230101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6da4101120 0x7f6da410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.360+0000 7f6dabb16640 1 --2- 172.21.15.107:0/203230101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6da4100760 0x7f6da4100be0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.360+0000 7f6dabb16640 1 --2- 172.21.15.107:0/203230101 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6da4106720 0x7f6da4106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.360+0000 7f6dabb16640 1 -- 172.21.15.107:0/203230101 >> 172.21.15.107:0/203230101 conn(0x7f6da40778b0 msgr2=0x7f6da4077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.360+0000 7f6dabb16640 1 -- 172.21.15.107:0/203230101 shutdown_connections 2024-05-09T08:40:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.360+0000 7f6dabb16640 1 -- 172.21.15.107:0/203230101 wait complete. 2024-05-09T08:40:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.360+0000 7f6dabb16640 1 Processor -- start 2024-05-09T08:40:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.360+0000 7f6dabb16640 1 -- start start 2024-05-09T08:40:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.360+0000 7f6dabb16640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6da4100760 0x7f6da4198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.360+0000 7f6dabb16640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6da4101120 0x7f6da4199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6dabb16640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6da4106720 0x7f6da41a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6dabb16640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6da41109e0 con 0x7f6da4106720 2024-05-09T08:40:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6dabb16640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6da4110860 con 0x7f6da4100760 2024-05-09T08:40:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6dabb16640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6da4110b60 con 0x7f6da4101120 2024-05-09T08:40:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6daa0b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6da4106720 0x7f6da41a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6daa0b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6da4106720 0x7f6da41a3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58746/0 (socket says 172.21.15.107:58746) 2024-05-09T08:40:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6daa0b3640 1 -- 172.21.15.107:0/2731948607 learned_addr learned my addr 172.21.15.107:0/2731948607 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6da98b2640 1 --2- 172.21.15.107:0/2731948607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6da4100760 0x7f6da4198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6da90b1640 1 --2- 172.21.15.107:0/2731948607 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6da4101120 0x7f6da4199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6daa0b3640 1 -- 172.21.15.107:0/2731948607 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6da4101120 msgr2=0x7f6da4199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6daa0b3640 1 --2- 172.21.15.107:0/2731948607 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6da4101120 0x7f6da4199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6daa0b3640 1 -- 172.21.15.107:0/2731948607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6da4100760 msgr2=0x7f6da4198d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6daa0b3640 1 --2- 172.21.15.107:0/2731948607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6da4100760 0x7f6da4198d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6daa0b3640 1 -- 172.21.15.107:0/2731948607 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da41a42e0 con 0x7f6da4106720 2024-05-09T08:40:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6daa0b3640 1 --2- 172.21.15.107:0/2731948607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6da4106720 0x7f6da41a3da0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c00cce0 tx=0x7f6d9c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6d9affd640 1 -- 172.21.15.107:0/2731948607 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c00f450 con 0x7f6da4106720 2024-05-09T08:40:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6d9affd640 1 -- 172.21.15.107:0/2731948607 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d9c00faa0 con 0x7f6da4106720 2024-05-09T08:40:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6dabb16640 1 -- 172.21.15.107:0/2731948607 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da41a4510 con 0x7f6da4106720 2024-05-09T08:40:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6d9affd640 1 -- 172.21.15.107:0/2731948607 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c018b70 con 0x7f6da4106720 2024-05-09T08:40:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.361+0000 7f6dabb16640 1 -- 172.21.15.107:0/2731948607 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da41a49e0 con 0x7f6da4106720 2024-05-09T08:40:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.364+0000 7f6d9affd640 1 -- 172.21.15.107:0/2731948607 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6d9c018d10 con 0x7f6da4106720 2024-05-09T08:40:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.364+0000 7f6d9affd640 1 --2- 172.21.15.107:0/2731948607 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6d900a2da0 0x7f6d900a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.365+0000 7f6dae81a740 1 Processor -- start 2024-05-09T08:40:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.365+0000 7f6da98b2640 1 --2- 172.21.15.107:0/2731948607 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6d900a2da0 0x7f6d900a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.365+0000 7f6d9affd640 1 -- 172.21.15.107:0/2731948607 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f6d9c0cfc50 con 0x7f6da4106720 2024-05-09T08:40:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.365+0000 7f6dae81a740 1 -- start start 2024-05-09T08:40:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.365+0000 7f6dae81a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae839390 0x5590ae839770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.365+0000 7f6dae81a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae848500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.365+0000 7f6dae81a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae848a40 0x5590ae84ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.365+0000 7f6dae81a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5590ae79e340 con 0x5590ae839cb0 2024-05-09T08:40:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.365+0000 7f6dae81a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5590ae746290 con 0x5590ae839390 2024-05-09T08:40:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.365+0000 7f6dae81a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5590ae79a300 con 0x5590ae848a40 2024-05-09T08:40:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.365+0000 7f6da98b2640 1 --2- 172.21.15.107:0/2731948607 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6d900a2da0 0x7f6d900a5260 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6da0005fd0 tx=0x7f6da000b390 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.365+0000 7f6da90b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae848500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.365+0000 7f6da90b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae848500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58756/0 (socket says 172.21.15.107:58756) 2024-05-09T08:40:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.365+0000 7f6da90b1640 1 -- 172.21.15.107:0/1180732132 learned_addr learned my addr 172.21.15.107:0/1180732132 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.365+0000 7f6da98b2640 1 --2- 172.21.15.107:0/1180732132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae839390 0x5590ae839770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.365+0000 7f6daa0b3640 1 --2- 172.21.15.107:0/1180732132 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae848a40 0x5590ae84ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.366+0000 7f6da90b1640 1 -- 172.21.15.107:0/1180732132 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae848a40 msgr2=0x5590ae84ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.366+0000 7f6da90b1640 1 --2- 172.21.15.107:0/1180732132 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae848a40 0x5590ae84ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.366+0000 7f6da90b1640 1 -- 172.21.15.107:0/1180732132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae839390 msgr2=0x5590ae839770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.366+0000 7f6da90b1640 1 --2- 172.21.15.107:0/1180732132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae839390 0x5590ae839770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.366+0000 7f6da90b1640 1 -- 172.21.15.107:0/1180732132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590ae76a8e0 con 0x5590ae839cb0 2024-05-09T08:40:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.366+0000 7f6da98b2640 1 --2- 172.21.15.107:0/1180732132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae839390 0x5590ae839770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:40:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.366+0000 7f6da90b1640 1 --2- 172.21.15.107:0/1180732132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae848500 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f6d94009930 tx=0x7f6d94032cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4e7d14fab219a5b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.366+0000 7f6d8ffff640 1 -- 172.21.15.107:0/1180732132 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9403f070 con 0x5590ae839cb0 2024-05-09T08:40:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.366+0000 7f6d8ffff640 1 -- 172.21.15.107:0/1180732132 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d940316a0 con 0x5590ae839cb0 2024-05-09T08:40:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.366+0000 7f6d8ffff640 1 -- 172.21.15.107:0/1180732132 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94035070 con 0x5590ae839cb0 2024-05-09T08:40:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.367+0000 7f6dae81a740 1 -- 172.21.15.107:0/1180732132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 msgr2=0x5590ae848500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.367+0000 7f6dae81a740 1 --2- 172.21.15.107:0/1180732132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae848500 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f6d94009930 tx=0x7f6d94032cc0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.367+0000 7f6dae81a740 1 -- 172.21.15.107:0/1180732132 shutdown_connections 2024-05-09T08:40:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.367+0000 7f6dae81a740 1 --2- 172.21.15.107:0/1180732132 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae848a40 0x5590ae84ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.367+0000 7f6dae81a740 1 --2- 172.21.15.107:0/1180732132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae848500 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.367+0000 7f6dae81a740 1 --2- 172.21.15.107:0/1180732132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae839390 0x5590ae839770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.367+0000 7f6dae81a740 1 -- 172.21.15.107:0/1180732132 >> 172.21.15.107:0/1180732132 conn(0x5590ae835960 msgr2=0x5590ae835d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.367+0000 7f6dae81a740 1 -- 172.21.15.107:0/1180732132 shutdown_connections 2024-05-09T08:40:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.367+0000 7f6dae81a740 1 -- 172.21.15.107:0/1180732132 wait complete. 2024-05-09T08:40:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.367+0000 7f6dae81a740 1 -- start start 2024-05-09T08:40:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.367+0000 7f6dae81a740 1 Processor -- start 2024-05-09T08:40:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6dae81a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839390 0x5590ae845990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6dae81a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae839cb0 0x5590ae845ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6dae81a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae848a40 0x5590ae846410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6dae81a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5590ae84b5b0 con 0x5590ae839390 2024-05-09T08:40:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6dae81a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5590ae759790 con 0x5590ae848a40 2024-05-09T08:40:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6dae81a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5590ae84b730 con 0x5590ae839cb0 2024-05-09T08:40:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6da98b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839390 0x5590ae845990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6da98b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839390 0x5590ae845990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58768/0 (socket says 172.21.15.107:58768) 2024-05-09T08:40:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6da98b2640 1 -- 172.21.15.107:0/1008963594 learned_addr learned my addr 172.21.15.107:0/1008963594 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6dae81a740 10 client.0 _ll_register_callbacks cb 0x5590ae794a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6daa0b3640 1 --2- 172.21.15.107:0/1008963594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae848a40 0x5590ae846410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6da90b1640 1 --2- 172.21.15.107:0/1008963594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae839cb0 0x5590ae845ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6da98b2640 1 -- 172.21.15.107:0/1008963594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae839cb0 msgr2=0x5590ae845ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6da98b2640 1 --2- 172.21.15.107:0/1008963594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae839cb0 0x5590ae845ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6da98b2640 1 -- 172.21.15.107:0/1008963594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae848a40 msgr2=0x5590ae846410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6da98b2640 1 --2- 172.21.15.107:0/1008963594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae848a40 0x5590ae846410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6da98b2640 1 -- 172.21.15.107:0/1008963594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590ae4b2210 con 0x5590ae839390 2024-05-09T08:40:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6daa0b3640 1 --2- 172.21.15.107:0/1008963594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae848a40 0x5590ae846410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:40:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6da90b1640 1 --2- 172.21.15.107:0/1008963594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae839cb0 0x5590ae845ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:40:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6da98b2640 1 --2- 172.21.15.107:0/1008963594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839390 0x5590ae845990 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f6da0003f90 tx=0x7f6da0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a58dcf7d6c2de9c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6d8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.368+0000 7f6d8d7fa640 1 -- 172.21.15.107:0/1008963594 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da001f840 con 0x5590ae839390 2024-05-09T08:40:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.369+0000 7f6d8d7fa640 1 -- 172.21.15.107:0/1008963594 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f6da001f9e0 con 0x5590ae839390 2024-05-09T08:40:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.369+0000 7f6dae81a740 1 -- 172.21.15.107:0/1008963594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d940316a0 con 0x5590ae839390 2024-05-09T08:40:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.369+0000 7f6d8d7fa640 1 -- 172.21.15.107:0/1008963594 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6da0031050 con 0x5590ae839390 2024-05-09T08:40:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.369+0000 7f6dae81a740 10 client.17829 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.369+0000 7f6dae81a740 1 -- 172.21.15.107:0/1008963594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5590ae57e620 con 0x5590ae839390 2024-05-09T08:40:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.369+0000 7f6d8d7fa640 1 -- 172.21.15.107:0/1008963594 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d940316a0 con 0x5590ae839390 2024-05-09T08:40:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.369+0000 7f6d8d7fa640 1 -- 172.21.15.107:0/1008963594 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f6da0028ca0 con 0x5590ae839390 2024-05-09T08:40:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.369+0000 7f6dae81a740 10 client.17829 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.369+0000 7f6dae81a740 10 client.17829 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:40:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.369+0000 7f6dae81a740 20 client.17829 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.369+0000 7f6dae81a740 1 --2- 172.21.15.107:0/1008963594 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5590ae8587f0 0x5590ae85ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.369+0000 7f6dae81a740 4 client.17829 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:40:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.369+0000 7f6dae81a740 1 -- 172.21.15.107:0/1008963594 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590ae7aba70 con 0x5590ae8587f0 2024-05-09T08:40:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.369+0000 7f6da90b1640 1 --2- 172.21.15.107:0/1008963594 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5590ae8587f0 0x5590ae85ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.370+0000 7f6d8d7fa640 10 client.17829 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.370+0000 7f6da90b1640 1 --2- 172.21.15.107:0/1008963594 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5590ae8587f0 0x5590ae85ac40 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d6c168b70c99f4f server_cookie=bb2fc5848de4b2ff in_seq=0 out_seq=0 2024-05-09T08:40:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.371+0000 7f6d8d7fa640 1 -- 172.21.15.107:0/1008963594 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5590ae7aba70 con 0x5590ae8587f0 2024-05-09T08:40:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.371+0000 7f6d8d7fa640 10 client.17829 handle_command_reply: tid=0 2024-05-09T08:40:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.371+0000 7f6dae81a740 1 client.17829 shutdown 2024-05-09T08:40:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.371+0000 7f6dae81a740 1 -- 172.21.15.107:0/1008963594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839390 msgr2=0x5590ae845990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.371+0000 7f6dae81a740 1 --2- 172.21.15.107:0/1008963594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839390 0x5590ae845990 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f6da0003f90 tx=0x7f6da0004730 comp rx=0 tx=0).stop 2024-05-09T08:40:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.371+0000 7f6dae81a740 1 -- 172.21.15.107:0/1008963594 shutdown_connections 2024-05-09T08:40:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.371+0000 7f6dae81a740 1 --2- 172.21.15.107:0/1008963594 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5590ae8587f0 0x5590ae85ac40 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.371+0000 7f6dae81a740 1 --2- 172.21.15.107:0/1008963594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae848a40 0x5590ae846410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.371+0000 7f6dae81a740 1 --2- 172.21.15.107:0/1008963594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae839cb0 0x5590ae845ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.371+0000 7f6d8d7fa640 0 client.17829 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.371+0000 7f6dae81a740 1 --2- 172.21.15.107:0/1008963594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839390 0x5590ae845990 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.371+0000 7f6dae81a740 1 -- 172.21.15.107:0/1008963594 >> 172.21.15.107:0/1008963594 conn(0x5590ae835960 msgr2=0x5590ae83b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.371+0000 7f6dae81a740 1 -- 172.21.15.107:0/1008963594 shutdown_connections 2024-05-09T08:40:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.371+0000 7f6dae81a740 1 -- 172.21.15.107:0/1008963594 wait complete. 2024-05-09T08:40:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.371+0000 7f6dae81a740 20 client.17829 trim_cache size 0 max 16384 2024-05-09T08:40:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6dae81a740 1 Processor -- start 2024-05-09T08:40:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6dae81a740 1 -- start start 2024-05-09T08:40:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6dae81a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae839390 0x5590ae8480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6dae81a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae8544b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6dae81a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae848a40 0x5590ae844b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6dae81a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6da41af0a0 con 0x5590ae839cb0 2024-05-09T08:40:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6dae81a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6da41ac970 con 0x5590ae839390 2024-05-09T08:40:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6dae81a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6da41aa7e0 con 0x5590ae848a40 2024-05-09T08:40:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6da90b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae8544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6da90b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae8544b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58780/0 (socket says 172.21.15.107:58780) 2024-05-09T08:40:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6da90b1640 1 -- 172.21.15.107:0/3660513677 learned_addr learned my addr 172.21.15.107:0/3660513677 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6da98b2640 1 --2- 172.21.15.107:0/3660513677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae839390 0x5590ae8480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6daa0b3640 1 --2- 172.21.15.107:0/3660513677 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae848a40 0x5590ae844b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6da90b1640 1 -- 172.21.15.107:0/3660513677 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae848a40 msgr2=0x5590ae844b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6da90b1640 1 --2- 172.21.15.107:0/3660513677 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae848a40 0x5590ae844b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6da90b1640 1 -- 172.21.15.107:0/3660513677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae839390 msgr2=0x5590ae8480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6da90b1640 1 --2- 172.21.15.107:0/3660513677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae839390 0x5590ae8480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6da90b1640 1 -- 172.21.15.107:0/3660513677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590ae830bc0 con 0x5590ae839cb0 2024-05-09T08:40:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.386+0000 7f6da90b1640 1 --2- 172.21.15.107:0/3660513677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae8544b0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f6d94031440 tx=0x7f6d94000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef00bf3e7830a92 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.387+0000 7f6d8f7fe640 1 -- 172.21.15.107:0/3660513677 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94035040 con 0x5590ae839cb0 2024-05-09T08:40:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.387+0000 7f6d8f7fe640 1 -- 172.21.15.107:0/3660513677 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d94004020 con 0x5590ae839cb0 2024-05-09T08:40:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.387+0000 7f6d8f7fe640 1 -- 172.21.15.107:0/3660513677 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9403f040 con 0x5590ae839cb0 2024-05-09T08:40:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.387+0000 7f6dae81a740 1 -- 172.21.15.107:0/3660513677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 msgr2=0x5590ae8544b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.387+0000 7f6dae81a740 1 --2- 172.21.15.107:0/3660513677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae8544b0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f6d94031440 tx=0x7f6d94000bc0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.387+0000 7f6dae81a740 1 -- 172.21.15.107:0/3660513677 shutdown_connections 2024-05-09T08:40:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.387+0000 7f6dae81a740 1 --2- 172.21.15.107:0/3660513677 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae848a40 0x5590ae844b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.387+0000 7f6dae81a740 1 --2- 172.21.15.107:0/3660513677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae8544b0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.387+0000 7f6dae81a740 1 --2- 172.21.15.107:0/3660513677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae839390 0x5590ae8480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.387+0000 7f6dae81a740 1 -- 172.21.15.107:0/3660513677 >> 172.21.15.107:0/3660513677 conn(0x5590ae835960 msgr2=0x5590ae845950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.387+0000 7f6dae81a740 1 -- 172.21.15.107:0/3660513677 shutdown_connections 2024-05-09T08:40:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.387+0000 7f6dae81a740 1 -- 172.21.15.107:0/3660513677 wait complete. 2024-05-09T08:40:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.388+0000 7f6dae81a740 1 -- start start 2024-05-09T08:40:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.388+0000 7f6dae81a740 1 Processor -- start 2024-05-09T08:40:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.388+0000 7f6dae81a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae839390 0x5590ae8509c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.388+0000 7f6dae81a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae850f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.388+0000 7f6dae81a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae848a40 0x5590ae842070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.388+0000 7f6dae81a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6da41aae30 con 0x5590ae839cb0 2024-05-09T08:40:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.388+0000 7f6dae81a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6da41ac2a0 con 0x5590ae848a40 2024-05-09T08:40:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.388+0000 7f6dae81a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6da41ab500 con 0x5590ae839390 2024-05-09T08:40:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.388+0000 7f6dae81a740 10 client.0 _ll_register_callbacks cb 0x7f6da0003870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:40:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.388+0000 7f6da90b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae850f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.388+0000 7f6da98b2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae839390 0x5590ae8509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.388+0000 7f6da90b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae850f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58794/0 (socket says 172.21.15.107:58794) 2024-05-09T08:40:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.388+0000 7f6da90b1640 1 -- 172.21.15.107:0/1568599970 learned_addr learned my addr 172.21.15.107:0/1568599970 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:40:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.388+0000 7f6daa0b3640 1 --2- 172.21.15.107:0/1568599970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae848a40 0x5590ae842070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.389+0000 7f6da90b1640 1 -- 172.21.15.107:0/1568599970 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae839390 msgr2=0x5590ae8509c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.389+0000 7f6da90b1640 1 --2- 172.21.15.107:0/1568599970 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae839390 0x5590ae8509c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.389+0000 7f6da90b1640 1 -- 172.21.15.107:0/1568599970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae848a40 msgr2=0x5590ae842070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:40:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.389+0000 7f6da90b1640 1 --2- 172.21.15.107:0/1568599970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae848a40 0x5590ae842070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.389+0000 7f6da90b1640 1 -- 172.21.15.107:0/1568599970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590ae8428a0 con 0x5590ae839cb0 2024-05-09T08:40:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.389+0000 7f6da98b2640 1 --2- 172.21.15.107:0/1568599970 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae839390 0x5590ae8509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:40:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.389+0000 7f6da90b1640 1 --2- 172.21.15.107:0/1568599970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae850f00 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f6d94005ab0 tx=0x7f6d94032c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebcab509633ff53f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:40:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.389+0000 7f6d8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:40:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.389+0000 7f6d8d7fa640 1 -- 172.21.15.107:0/1568599970 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94059060 con 0x5590ae839cb0 2024-05-09T08:40:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.389+0000 7f6dae81a740 1 -- 172.21.15.107:0/1568599970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d94004020 con 0x5590ae839cb0 2024-05-09T08:40:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.389+0000 7f6d8d7fa640 1 -- 172.21.15.107:0/1568599970 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f6d94005250 con 0x5590ae839cb0 2024-05-09T08:40:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.390+0000 7f6d8d7fa640 1 -- 172.21.15.107:0/1568599970 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6d9403f070 con 0x5590ae839cb0 2024-05-09T08:40:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.390+0000 7f6dae81a740 10 client.17835 fetch_fsmap learned FSMap version 27 2024-05-09T08:40:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.390+0000 7f6dae81a740 1 -- 172.21.15.107:0/1568599970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5590ae8be8f0 con 0x5590ae839cb0 2024-05-09T08:40:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.391+0000 7f6d8d7fa640 1 -- 172.21.15.107:0/1568599970 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d94004020 con 0x5590ae839cb0 2024-05-09T08:40:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.391+0000 7f6d8d7fa640 1 -- 172.21.15.107:0/1568599970 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f6d940035f0 con 0x5590ae839cb0 2024-05-09T08:40:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.391+0000 7f6dae81a740 10 client.17835 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:40:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.391+0000 7f6dae81a740 10 client.17835 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:40:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.391+0000 7f6dae81a740 20 client.17835 populate_metadata read hostname 'smithi107' 2024-05-09T08:40:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.391+0000 7f6dae81a740 1 --2- 172.21.15.107:0/1568599970 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5590ae8587f0 0x5590ae8c0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:40:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.391+0000 7f6dae81a740 4 client.17835 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:40:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.391+0000 7f6dae81a740 1 -- 172.21.15.107:0/1568599970 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5590ae7a5dc0 con 0x5590ae8587f0 2024-05-09T08:40:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.392+0000 7f6da98b2640 1 --2- 172.21.15.107:0/1568599970 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5590ae8587f0 0x5590ae8c0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:40:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.392+0000 7f6d8d7fa640 10 client.17835 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.392+0000 7f6da98b2640 1 --2- 172.21.15.107:0/1568599970 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5590ae8587f0 0x5590ae8c0c30 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46986526e3d13fa3 server_cookie=363a15ec660316c9 in_seq=0 out_seq=0 2024-05-09T08:40:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.411+0000 7f6d8d7fa640 1 -- 172.21.15.107:0/1568599970 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5590ae7a5dc0 con 0x5590ae8587f0 2024-05-09T08:40:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.411+0000 7f6d8d7fa640 10 client.17835 handle_command_reply: tid=0 2024-05-09T08:40:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.411+0000 7f6dae81a740 1 client.17835 shutdown 2024-05-09T08:40:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.412+0000 7f6dae81a740 1 -- 172.21.15.107:0/1568599970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 msgr2=0x5590ae850f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.412+0000 7f6dae81a740 1 --2- 172.21.15.107:0/1568599970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae850f00 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f6d94005ab0 tx=0x7f6d94032c40 comp rx=0 tx=0).stop 2024-05-09T08:40:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.412+0000 7f6dae81a740 1 -- 172.21.15.107:0/1568599970 shutdown_connections 2024-05-09T08:40:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.412+0000 7f6dae81a740 1 --2- 172.21.15.107:0/1568599970 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5590ae8587f0 0x5590ae8c0c30 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.412+0000 7f6dae81a740 1 --2- 172.21.15.107:0/1568599970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5590ae848a40 0x5590ae842070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.412+0000 7f6dae81a740 1 --2- 172.21.15.107:0/1568599970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5590ae839cb0 0x5590ae850f00 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.412+0000 7f6dae81a740 1 --2- 172.21.15.107:0/1568599970 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5590ae839390 0x5590ae8509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.412+0000 7f6dae81a740 1 -- 172.21.15.107:0/1568599970 >> 172.21.15.107:0/1568599970 conn(0x5590ae835960 msgr2=0x5590ae8449d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.412+0000 7f6d8d7fa640 0 client.17835 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:40:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.412+0000 7f6dae81a740 1 -- 172.21.15.107:0/1568599970 shutdown_connections 2024-05-09T08:40:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.412+0000 7f6dae81a740 1 -- 172.21.15.107:0/1568599970 wait complete. 2024-05-09T08:40:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.412+0000 7f6dae81a740 20 client.17835 trim_cache size 0 max 16384 2024-05-09T08:40:50.413 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:40:50.413 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:40:50.413 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:40:50.413 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:40:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.413+0000 7f6dabb16640 1 -- 172.21.15.107:0/2731948607 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6d900a2da0 msgr2=0x7f6d900a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.413+0000 7f6dabb16640 1 --2- 172.21.15.107:0/2731948607 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6d900a2da0 0x7f6d900a5260 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6da0005fd0 tx=0x7f6da000b390 comp rx=0 tx=0).stop 2024-05-09T08:40:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.413+0000 7f6dabb16640 1 -- 172.21.15.107:0/2731948607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6da4106720 msgr2=0x7f6da41a3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:40:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.413+0000 7f6dabb16640 1 --2- 172.21.15.107:0/2731948607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6da4106720 0x7f6da41a3da0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c00cce0 tx=0x7f6d9c00a620 comp rx=0 tx=0).stop 2024-05-09T08:40:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.413+0000 7f6dabb16640 1 -- 172.21.15.107:0/2731948607 shutdown_connections 2024-05-09T08:40:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.413+0000 7f6dabb16640 1 --2- 172.21.15.107:0/2731948607 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6d900a2da0 0x7f6d900a5260 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.413+0000 7f6dabb16640 1 --2- 172.21.15.107:0/2731948607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6da4106720 0x7f6da41a3da0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.413+0000 7f6dabb16640 1 --2- 172.21.15.107:0/2731948607 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6da4101120 0x7f6da4199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.413+0000 7f6dabb16640 1 --2- 172.21.15.107:0/2731948607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6da4100760 0x7f6da4198d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:40:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.413+0000 7f6dabb16640 1 -- 172.21.15.107:0/2731948607 >> 172.21.15.107:0/2731948607 conn(0x7f6da40778b0 msgr2=0x7f6da41032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:40:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.413+0000 7f6dabb16640 1 -- 172.21.15.107:0/2731948607 shutdown_connections 2024-05-09T08:40:50.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:40:50.413+0000 7f6dabb16640 1 -- 172.21.15.107:0/2731948607 wait complete. 2024-05-09T08:40:50.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:50 smithi157 ceph-mon[33614]: pgmap v229: 129 pgs: 129 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:40:50.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:50 smithi110 ceph-mon[33622]: pgmap v229: 129 pgs: 129 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:40:51.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:50 smithi107 ceph-mon[30122]: pgmap v229: 129 pgs: 129 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:40:52.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:52 smithi157 ceph-mon[33614]: pgmap v230: 129 pgs: 129 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:40:52.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:52 smithi110 ceph-mon[33622]: pgmap v230: 129 pgs: 129 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:40:53.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:52 smithi107 ceph-mon[30122]: pgmap v230: 129 pgs: 129 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:40:54.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:54 smithi157 ceph-mon[33614]: pgmap v231: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 37 MiB/s wr, 33 op/s 2024-05-09T08:40:54.914 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:54 smithi110 ceph-mon[33622]: pgmap v231: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 37 MiB/s wr, 33 op/s 2024-05-09T08:40:55.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:54 smithi107 ceph-mon[30122]: pgmap v231: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 37 MiB/s wr, 33 op/s 2024-05-09T08:40:56.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:56 smithi157 ceph-mon[33614]: pgmap v232: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 20 op/s 2024-05-09T08:40:56.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:56 smithi110 ceph-mon[33622]: pgmap v232: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 20 op/s 2024-05-09T08:40:57.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:56 smithi107 ceph-mon[30122]: pgmap v232: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 20 op/s 2024-05-09T08:40:58.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:40:58 smithi157 ceph-mon[33614]: pgmap v233: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-05-09T08:40:58.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:40:58 smithi110 ceph-mon[33622]: pgmap v233: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-05-09T08:40:59.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:40:58 smithi107 ceph-mon[30122]: pgmap v233: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-05-09T08:41:00.302 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:41:00.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:41:00.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.539+0000 7f7d5aa6d640 1 -- 172.21.15.107:0/16890332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d541010c0 msgr2=0x7f7d5410d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:00.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.539+0000 7f7d5aa6d640 1 --2- 172.21.15.107:0/16890332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d541010c0 0x7f7d5410d5d0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c009cd0 tx=0x7f7d4c02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.539+0000 7f7d5aa6d640 1 -- 172.21.15.107:0/16890332 shutdown_connections 2024-05-09T08:41:00.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.539+0000 7f7d5aa6d640 1 --2- 172.21.15.107:0/16890332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d541010c0 0x7f7d5410d5d0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.539+0000 7f7d5aa6d640 1 --2- 172.21.15.107:0/16890332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d54100700 0x7f7d54100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.539+0000 7f7d5aa6d640 1 --2- 172.21.15.107:0/16890332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d54106710 0x7f7d54106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.539+0000 7f7d5aa6d640 1 -- 172.21.15.107:0/16890332 >> 172.21.15.107:0/16890332 conn(0x7f7d54077830 msgr2=0x7f7d54077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:00.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.539+0000 7f7d5aa6d640 1 -- 172.21.15.107:0/16890332 shutdown_connections 2024-05-09T08:41:00.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.539+0000 7f7d5aa6d640 1 -- 172.21.15.107:0/16890332 wait complete. 2024-05-09T08:41:00.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.540+0000 7f7d5aa6d640 1 Processor -- start 2024-05-09T08:41:00.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.540+0000 7f7d5aa6d640 1 -- start start 2024-05-09T08:41:00.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.540+0000 7f7d5aa6d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d54100700 0x7f7d54198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.540+0000 7f7d5aa6d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d541010c0 0x7f7d54198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.540+0000 7f7d5aa6d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d54106710 0x7f7d5419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.540+0000 7f7d5aa6d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7d541109d0 con 0x7f7d541010c0 2024-05-09T08:41:00.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.540+0000 7f7d5aa6d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7d54110850 con 0x7f7d54100700 2024-05-09T08:41:00.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.540+0000 7f7d5aa6d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7d54110b50 con 0x7f7d54106710 2024-05-09T08:41:00.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.540+0000 7f7d53fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d541010c0 0x7f7d54198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.540+0000 7f7d53fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d541010c0 0x7f7d54198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45356/0 (socket says 172.21.15.107:45356) 2024-05-09T08:41:00.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.540+0000 7f7d53fff640 1 -- 172.21.15.107:0/422598717 learned_addr learned my addr 172.21.15.107:0/422598717 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:00.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.541+0000 7f7d58809640 1 --2- 172.21.15.107:0/422598717 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d54100700 0x7f7d54198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.541+0000 7f7d5900a640 1 --2- 172.21.15.107:0/422598717 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d54106710 0x7f7d5419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.541+0000 7f7d53fff640 1 -- 172.21.15.107:0/422598717 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d54106710 msgr2=0x7f7d5419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:00.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.541+0000 7f7d53fff640 1 --2- 172.21.15.107:0/422598717 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d54106710 0x7f7d5419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.541+0000 7f7d53fff640 1 -- 172.21.15.107:0/422598717 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d54100700 msgr2=0x7f7d54198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:00.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.541+0000 7f7d53fff640 1 --2- 172.21.15.107:0/422598717 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d54100700 0x7f7d54198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.541+0000 7f7d53fff640 1 -- 172.21.15.107:0/422598717 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d5419d030 con 0x7f7d541010c0 2024-05-09T08:41:00.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.541+0000 7f7d53fff640 1 --2- 172.21.15.107:0/422598717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d541010c0 0x7f7d54198580 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f7d4000d950 tx=0x7f7d4000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:00.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.541+0000 7f7d51ffb640 1 -- 172.21.15.107:0/422598717 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d40017070 con 0x7f7d541010c0 2024-05-09T08:41:00.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.541+0000 7f7d51ffb640 1 -- 172.21.15.107:0/422598717 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d40009c30 con 0x7f7d541010c0 2024-05-09T08:41:00.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.541+0000 7f7d5aa6d640 1 -- 172.21.15.107:0/422598717 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d5419d320 con 0x7f7d541010c0 2024-05-09T08:41:00.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.541+0000 7f7d51ffb640 1 -- 172.21.15.107:0/422598717 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d4000f620 con 0x7f7d541010c0 2024-05-09T08:41:00.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.541+0000 7f7d5aa6d640 1 -- 172.21.15.107:0/422598717 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d541a4bb0 con 0x7f7d541010c0 2024-05-09T08:41:00.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.544+0000 7f7d51ffb640 1 -- 172.21.15.107:0/422598717 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7d400109a0 con 0x7f7d541010c0 2024-05-09T08:41:00.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.544+0000 7f7d51ffb640 1 --2- 172.21.15.107:0/422598717 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7d440a2d00 0x7f7d440a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.544+0000 7f7d5d776740 1 Processor -- start 2024-05-09T08:41:00.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.544+0000 7f7d5d776740 1 -- start start 2024-05-09T08:41:00.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.544+0000 7f7d58809640 1 --2- 172.21.15.107:0/422598717 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7d440a2d00 0x7f7d440a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.544+0000 7f7d51ffb640 1 -- 172.21.15.107:0/422598717 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f7d400cf440 con 0x7f7d541010c0 2024-05-09T08:41:00.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d5d776740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0160 0x5564d76a0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d5d776740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76a0a80 0x5564d76af140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d5d776740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76af680 0x5564d76b1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d5d776740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5564d7605d00 con 0x5564d76a0a80 2024-05-09T08:41:00.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d5d776740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5564d75ad4c0 con 0x5564d76af680 2024-05-09T08:41:00.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d5d776740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5564d7601cc0 con 0x5564d76a0160 2024-05-09T08:41:00.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d53fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76a0a80 0x5564d76af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d53fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76a0a80 0x5564d76af140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45358/0 (socket says 172.21.15.107:45358) 2024-05-09T08:41:00.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d53fff640 1 -- 172.21.15.107:0/1370150978 learned_addr learned my addr 172.21.15.107:0/1370150978 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:00.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d5900a640 1 --2- 172.21.15.107:0/1370150978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76af680 0x5564d76b1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d58809640 1 --2- 172.21.15.107:0/422598717 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7d440a2d00 0x7f7d440a51c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009ba0 tx=0x7f7d48005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:00.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d58809640 1 --2- 172.21.15.107:0/1370150978 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0160 0x5564d76a0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d53fff640 1 -- 172.21.15.107:0/1370150978 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0160 msgr2=0x5564d76a0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:00.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d53fff640 1 --2- 172.21.15.107:0/1370150978 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0160 0x5564d76a0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d53fff640 1 -- 172.21.15.107:0/1370150978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76af680 msgr2=0x5564d76b1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:00.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d53fff640 1 --2- 172.21.15.107:0/1370150978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76af680 0x5564d76b1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d53fff640 1 -- 172.21.15.107:0/1370150978 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564d73e37d0 con 0x5564d76a0a80 2024-05-09T08:41:00.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d5900a640 1 --2- 172.21.15.107:0/1370150978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76af680 0x5564d76b1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:00.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d53fff640 1 --2- 172.21.15.107:0/1370150978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76a0a80 0x5564d76af140 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f7d40097a10 tx=0x7f7d40009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c986284867681da9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:00.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.545+0000 7f7d3effd640 1 -- 172.21.15.107:0/1370150978 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d40020040 con 0x5564d76a0a80 2024-05-09T08:41:00.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.546+0000 7f7d3effd640 1 -- 172.21.15.107:0/1370150978 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d4009f960 con 0x5564d76a0a80 2024-05-09T08:41:00.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.546+0000 7f7d3effd640 1 -- 172.21.15.107:0/1370150978 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d400ccbc0 con 0x5564d76a0a80 2024-05-09T08:41:00.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.546+0000 7f7d5d776740 1 -- 172.21.15.107:0/1370150978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76a0a80 msgr2=0x5564d76af140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:00.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.546+0000 7f7d5d776740 1 --2- 172.21.15.107:0/1370150978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76a0a80 0x5564d76af140 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f7d40097a10 tx=0x7f7d40009ea0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.546+0000 7f7d5d776740 1 -- 172.21.15.107:0/1370150978 shutdown_connections 2024-05-09T08:41:00.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.546+0000 7f7d5d776740 1 --2- 172.21.15.107:0/1370150978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76af680 0x5564d76b1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.546+0000 7f7d5d776740 1 --2- 172.21.15.107:0/1370150978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76a0a80 0x5564d76af140 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.546+0000 7f7d5d776740 1 --2- 172.21.15.107:0/1370150978 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0160 0x5564d76a0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.546+0000 7f7d5d776740 1 -- 172.21.15.107:0/1370150978 >> 172.21.15.107:0/1370150978 conn(0x5564d769c5a0 msgr2=0x5564d769c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:00.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.546+0000 7f7d5d776740 1 -- 172.21.15.107:0/1370150978 shutdown_connections 2024-05-09T08:41:00.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.547+0000 7f7d5d776740 1 -- 172.21.15.107:0/1370150978 wait complete. 2024-05-09T08:41:00.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.547+0000 7f7d5d776740 1 -- start start 2024-05-09T08:41:00.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.547+0000 7f7d5d776740 1 Processor -- start 2024-05-09T08:41:00.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.547+0000 7f7d5d776740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76a0160 0x5564d76ac5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.547+0000 7f7d5d776740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0a80 0x5564d76acb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d5d776740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76af680 0x5564d76ad060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d5d776740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5564d76b2400 con 0x5564d76af680 2024-05-09T08:41:00.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d5d776740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5564d75c0b90 con 0x5564d76a0160 2024-05-09T08:41:00.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d5d776740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5564d76b2580 con 0x5564d76a0a80 2024-05-09T08:41:00.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d5d776740 10 client.0 _ll_register_callbacks cb 0x5564d75fcda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:00.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d5900a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76af680 0x5564d76ad060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d5900a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76af680 0x5564d76ad060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45362/0 (socket says 172.21.15.107:45362) 2024-05-09T08:41:00.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d5900a640 1 -- 172.21.15.107:0/3551718812 learned_addr learned my addr 172.21.15.107:0/3551718812 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:00.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d53fff640 1 --2- 172.21.15.107:0/3551718812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0a80 0x5564d76acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d58809640 1 --2- 172.21.15.107:0/3551718812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76a0160 0x5564d76ac5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d5900a640 1 -- 172.21.15.107:0/3551718812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0a80 msgr2=0x5564d76acb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:00.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d5900a640 1 --2- 172.21.15.107:0/3551718812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0a80 0x5564d76acb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d5900a640 1 -- 172.21.15.107:0/3551718812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76a0160 msgr2=0x5564d76ac5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:00.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d5900a640 1 --2- 172.21.15.107:0/3551718812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76a0160 0x5564d76ac5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d5900a640 1 -- 172.21.15.107:0/3551718812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564d73193f0 con 0x5564d76af680 2024-05-09T08:41:00.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d5900a640 1 --2- 172.21.15.107:0/3551718812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76af680 0x5564d76ad060 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c009cd0 tx=0x7f7d4c031a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd59de3138b05e3a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:00.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d33fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:00.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d33fff640 1 -- 172.21.15.107:0/3551718812 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c049060 con 0x5564d76af680 2024-05-09T08:41:00.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d33fff640 1 -- 172.21.15.107:0/3551718812 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f7d4c003970 con 0x5564d76af680 2024-05-09T08:41:00.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.548+0000 7f7d5d776740 1 -- 172.21.15.107:0/3551718812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d4009f960 con 0x5564d76af680 2024-05-09T08:41:00.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.549+0000 7f7d33fff640 1 -- 172.21.15.107:0/3551718812 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7d4c050050 con 0x5564d76af680 2024-05-09T08:41:00.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.549+0000 7f7d5d776740 10 client.17847 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:00.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.549+0000 7f7d5d776740 1 -- 172.21.15.107:0/3551718812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5564d72d90f0 con 0x5564d76af680 2024-05-09T08:41:00.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.549+0000 7f7d33fff640 1 -- 172.21.15.107:0/3551718812 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d4009f960 con 0x5564d76af680 2024-05-09T08:41:00.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.549+0000 7f7d33fff640 1 -- 172.21.15.107:0/3551718812 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7d4c043070 con 0x5564d76af680 2024-05-09T08:41:00.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.549+0000 7f7d5d776740 10 client.17847 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:00.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.549+0000 7f7d5d776740 10 client.17847 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:00.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.549+0000 7f7d5d776740 20 client.17847 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:00.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.549+0000 7f7d5d776740 1 --2- 172.21.15.107:0/3551718812 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5564d76bf9e0 0x5564d76c1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.549+0000 7f7d5d776740 4 client.17847 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:00.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.549+0000 7f7d5d776740 1 -- 172.21.15.107:0/3551718812 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564d7613430 con 0x5564d76bf9e0 2024-05-09T08:41:00.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.549+0000 7f7d58809640 1 --2- 172.21.15.107:0/3551718812 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5564d76bf9e0 0x5564d76c1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.550+0000 7f7d58809640 1 --2- 172.21.15.107:0/3551718812 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5564d76bf9e0 0x5564d76c1e20 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2be789bc2c6b3ea server_cookie=e3b80901ffd25c4c in_seq=0 out_seq=0 2024-05-09T08:41:00.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.550+0000 7f7d33fff640 10 client.17847 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:00.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.550+0000 7f7d33fff640 1 -- 172.21.15.107:0/3551718812 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5564d7613430 con 0x5564d76bf9e0 2024-05-09T08:41:00.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.550+0000 7f7d33fff640 10 client.17847 handle_command_reply: tid=0 2024-05-09T08:41:00.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.550+0000 7f7d5d776740 1 client.17847 shutdown 2024-05-09T08:41:00.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.551+0000 7f7d5d776740 1 -- 172.21.15.107:0/3551718812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76af680 msgr2=0x5564d76ad060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:00.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.551+0000 7f7d5d776740 1 --2- 172.21.15.107:0/3551718812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76af680 0x5564d76ad060 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c009cd0 tx=0x7f7d4c031a80 comp rx=0 tx=0).stop 2024-05-09T08:41:00.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.551+0000 7f7d5d776740 1 -- 172.21.15.107:0/3551718812 shutdown_connections 2024-05-09T08:41:00.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.551+0000 7f7d5d776740 1 --2- 172.21.15.107:0/3551718812 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5564d76bf9e0 0x5564d76c1e20 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.551+0000 7f7d5d776740 1 --2- 172.21.15.107:0/3551718812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76af680 0x5564d76ad060 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.551+0000 7f7d33fff640 0 client.17847 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:00.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.551+0000 7f7d5d776740 1 --2- 172.21.15.107:0/3551718812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0a80 0x5564d76acb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.551+0000 7f7d5d776740 1 --2- 172.21.15.107:0/3551718812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76a0160 0x5564d76ac5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.551+0000 7f7d5d776740 1 -- 172.21.15.107:0/3551718812 >> 172.21.15.107:0/3551718812 conn(0x5564d769c5a0 msgr2=0x5564d76a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:00.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.551+0000 7f7d5d776740 1 -- 172.21.15.107:0/3551718812 shutdown_connections 2024-05-09T08:41:00.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.551+0000 7f7d5d776740 1 -- 172.21.15.107:0/3551718812 wait complete. 2024-05-09T08:41:00.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.551+0000 7f7d5d776740 20 client.17847 trim_cache size 0 max 16384 2024-05-09T08:41:00.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.572+0000 7f7d5d776740 1 Processor -- start 2024-05-09T08:41:00.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d5d776740 1 -- start start 2024-05-09T08:41:00.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d5d776740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0160 0x5564d769bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d5d776740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76a0a80 0x5564d76ace80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d5d776740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76af680 0x5564d76a2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d5d776740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7d541af210 con 0x5564d76af680 2024-05-09T08:41:00.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d5d776740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7d541acae0 con 0x5564d76a0a80 2024-05-09T08:41:00.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d5d776740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7d541aa950 con 0x5564d76a0160 2024-05-09T08:41:00.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d5900a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76af680 0x5564d76a2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d5900a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76af680 0x5564d76a2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45374/0 (socket says 172.21.15.107:45374) 2024-05-09T08:41:00.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d5900a640 1 -- 172.21.15.107:0/4242916812 learned_addr learned my addr 172.21.15.107:0/4242916812 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:00.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d53fff640 1 --2- 172.21.15.107:0/4242916812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76a0a80 0x5564d76ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d58809640 1 --2- 172.21.15.107:0/4242916812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0160 0x5564d769bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d5900a640 1 -- 172.21.15.107:0/4242916812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0160 msgr2=0x5564d769bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:00.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d5900a640 1 --2- 172.21.15.107:0/4242916812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0160 0x5564d769bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d5900a640 1 -- 172.21.15.107:0/4242916812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76a0a80 msgr2=0x5564d76ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:00.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d5900a640 1 --2- 172.21.15.107:0/4242916812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76a0a80 0x5564d76ace80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d5900a640 1 -- 172.21.15.107:0/4242916812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564d769f890 con 0x5564d76af680 2024-05-09T08:41:00.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d53fff640 1 --2- 172.21.15.107:0/4242916812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76a0a80 0x5564d76ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:00.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.573+0000 7f7d5900a640 1 --2- 172.21.15.107:0/4242916812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76af680 0x5564d76a2350 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c02ae00 tx=0x7f7d4c031e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e25f542437a5db6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:00.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.574+0000 7f7d3e7fc640 1 -- 172.21.15.107:0/4242916812 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c049040 con 0x5564d76af680 2024-05-09T08:41:00.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.574+0000 7f7d3e7fc640 1 -- 172.21.15.107:0/4242916812 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d4c036040 con 0x5564d76af680 2024-05-09T08:41:00.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.574+0000 7f7d3e7fc640 1 -- 172.21.15.107:0/4242916812 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c044430 con 0x5564d76af680 2024-05-09T08:41:00.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.574+0000 7f7d5d776740 1 -- 172.21.15.107:0/4242916812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76af680 msgr2=0x5564d76a2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:00.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.574+0000 7f7d5d776740 1 --2- 172.21.15.107:0/4242916812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76af680 0x5564d76a2350 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c02ae00 tx=0x7f7d4c031e00 comp rx=0 tx=0).stop 2024-05-09T08:41:00.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.575+0000 7f7d5d776740 1 -- 172.21.15.107:0/4242916812 shutdown_connections 2024-05-09T08:41:00.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.575+0000 7f7d5d776740 1 --2- 172.21.15.107:0/4242916812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76af680 0x5564d76a2350 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.575+0000 7f7d5d776740 1 --2- 172.21.15.107:0/4242916812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76a0a80 0x5564d76ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.575+0000 7f7d5d776740 1 --2- 172.21.15.107:0/4242916812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0160 0x5564d769bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.575+0000 7f7d5d776740 1 -- 172.21.15.107:0/4242916812 >> 172.21.15.107:0/4242916812 conn(0x5564d769c5a0 msgr2=0x5564d76ab520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:00.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.575+0000 7f7d5d776740 1 -- 172.21.15.107:0/4242916812 shutdown_connections 2024-05-09T08:41:00.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.575+0000 7f7d5d776740 1 -- 172.21.15.107:0/4242916812 wait complete. 2024-05-09T08:41:00.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.575+0000 7f7d5d776740 1 -- start start 2024-05-09T08:41:00.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.575+0000 7f7d5d776740 1 Processor -- start 2024-05-09T08:41:00.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d5d776740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76a0160 0x5564d76b7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d5d776740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0a80 0x5564d76b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d5d776740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76af680 0x5564d76b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d5d776740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7d541aafa0 con 0x5564d76a0160 2024-05-09T08:41:00.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d5d776740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7d541ac410 con 0x5564d76af680 2024-05-09T08:41:00.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d5d776740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7d541ab670 con 0x5564d76a0a80 2024-05-09T08:41:00.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d5d776740 10 client.0 _ll_register_callbacks cb 0x7f7d4c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:00.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d58809640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76a0160 0x5564d76b7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d58809640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76a0160 0x5564d76b7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45390/0 (socket says 172.21.15.107:45390) 2024-05-09T08:41:00.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d58809640 1 -- 172.21.15.107:0/3721891205 learned_addr learned my addr 172.21.15.107:0/3721891205 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:00.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d53fff640 1 --2- 172.21.15.107:0/3721891205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0a80 0x5564d76b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d5900a640 1 --2- 172.21.15.107:0/3721891205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76af680 0x5564d76b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d58809640 1 -- 172.21.15.107:0/3721891205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0a80 msgr2=0x5564d76b30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:00.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d58809640 1 --2- 172.21.15.107:0/3721891205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0a80 0x5564d76b30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d58809640 1 -- 172.21.15.107:0/3721891205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76af680 msgr2=0x5564d76b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:00.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d58809640 1 --2- 172.21.15.107:0/3721891205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76af680 0x5564d76b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d58809640 1 -- 172.21.15.107:0/3721891205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564d7724f60 con 0x5564d76a0160 2024-05-09T08:41:00.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d5900a640 1 --2- 172.21.15.107:0/3721891205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76af680 0x5564d76b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:00.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d58809640 1 --2- 172.21.15.107:0/3721891205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76a0160 0x5564d76b7d70 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f7d48011970 tx=0x7f7d48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc4e1dd067f2815 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:00.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d33fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:00.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d33fff640 1 -- 172.21.15.107:0/3721891205 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d4802c360 con 0x5564d76a0160 2024-05-09T08:41:00.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d33fff640 1 -- 172.21.15.107:0/3721891205 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f7d4802c500 con 0x5564d76a0160 2024-05-09T08:41:00.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.576+0000 7f7d5d776740 1 -- 172.21.15.107:0/3721891205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d4c036040 con 0x5564d76a0160 2024-05-09T08:41:00.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.577+0000 7f7d33fff640 1 -- 172.21.15.107:0/3721891205 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7d4803d050 con 0x5564d76a0160 2024-05-09T08:41:00.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.577+0000 7f7d5d776740 10 client.17853 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:00.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.577+0000 7f7d5d776740 1 -- 172.21.15.107:0/3721891205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5564d7726170 con 0x5564d76a0160 2024-05-09T08:41:00.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.577+0000 7f7d33fff640 1 -- 172.21.15.107:0/3721891205 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d4c036040 con 0x5564d76a0160 2024-05-09T08:41:00.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.577+0000 7f7d33fff640 1 -- 172.21.15.107:0/3721891205 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7d48035bd0 con 0x5564d76a0160 2024-05-09T08:41:00.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.577+0000 7f7d5d776740 10 client.17853 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:00.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.577+0000 7f7d5d776740 10 client.17853 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:00.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.577+0000 7f7d5d776740 20 client.17853 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:00.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.577+0000 7f7d5d776740 1 --2- 172.21.15.107:0/3721891205 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5564d76bf9e0 0x5564d7728970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.577+0000 7f7d5d776740 4 client.17853 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:41:00.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.577+0000 7f7d5d776740 1 -- 172.21.15.107:0/3721891205 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5564d760d780 con 0x5564d76bf9e0 2024-05-09T08:41:00.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.577+0000 7f7d53fff640 1 --2- 172.21.15.107:0/3721891205 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5564d76bf9e0 0x5564d7728970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d33fff640 10 client.17853 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:00.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d53fff640 1 --2- 172.21.15.107:0/3721891205 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5564d76bf9e0 0x5564d7728970 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49a8f3373f4cc50d server_cookie=5d25cb0e43d683d6 in_seq=0 out_seq=0 2024-05-09T08:41:00.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d33fff640 1 -- 172.21.15.107:0/3721891205 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5564d760d780 con 0x5564d76bf9e0 2024-05-09T08:41:00.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d33fff640 10 client.17853 handle_command_reply: tid=0 2024-05-09T08:41:00.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d5d776740 1 client.17853 shutdown 2024-05-09T08:41:00.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d5d776740 1 -- 172.21.15.107:0/3721891205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76a0160 msgr2=0x5564d76b7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:00.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d5d776740 1 --2- 172.21.15.107:0/3721891205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76a0160 0x5564d76b7d70 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f7d48011970 tx=0x7f7d48004730 comp rx=0 tx=0).stop 2024-05-09T08:41:00.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d5d776740 1 -- 172.21.15.107:0/3721891205 shutdown_connections 2024-05-09T08:41:00.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d5d776740 1 --2- 172.21.15.107:0/3721891205 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5564d76bf9e0 0x5564d7728970 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d5d776740 1 --2- 172.21.15.107:0/3721891205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564d76af680 0x5564d76b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d33fff640 0 client.17853 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:00.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d5d776740 1 --2- 172.21.15.107:0/3721891205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5564d76a0a80 0x5564d76b30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d5d776740 1 --2- 172.21.15.107:0/3721891205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5564d76a0160 0x5564d76b7d70 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d5d776740 1 -- 172.21.15.107:0/3721891205 >> 172.21.15.107:0/3721891205 conn(0x5564d769c5a0 msgr2=0x5564d76a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:00.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d5d776740 1 -- 172.21.15.107:0/3721891205 shutdown_connections 2024-05-09T08:41:00.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d5d776740 1 -- 172.21.15.107:0/3721891205 wait complete. 2024-05-09T08:41:00.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.578+0000 7f7d5d776740 20 client.17853 trim_cache size 0 max 16384 2024-05-09T08:41:00.585 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:41:00.585 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:41:00.585 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:41:00.585 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:41:00.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.579+0000 7f7d5aa6d640 1 -- 172.21.15.107:0/422598717 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7d440a2d00 msgr2=0x7f7d440a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:00.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.579+0000 7f7d5aa6d640 1 --2- 172.21.15.107:0/422598717 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7d440a2d00 0x7f7d440a51c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009ba0 tx=0x7f7d48005f90 comp rx=0 tx=0).stop 2024-05-09T08:41:00.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.579+0000 7f7d5aa6d640 1 -- 172.21.15.107:0/422598717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d541010c0 msgr2=0x7f7d54198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:00.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.579+0000 7f7d5aa6d640 1 --2- 172.21.15.107:0/422598717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d541010c0 0x7f7d54198580 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f7d4000d950 tx=0x7f7d4000de10 comp rx=0 tx=0).stop 2024-05-09T08:41:00.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.579+0000 7f7d5aa6d640 1 -- 172.21.15.107:0/422598717 shutdown_connections 2024-05-09T08:41:00.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.579+0000 7f7d5aa6d640 1 --2- 172.21.15.107:0/422598717 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7d440a2d00 0x7f7d440a51c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.579+0000 7f7d5aa6d640 1 --2- 172.21.15.107:0/422598717 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d54106710 0x7f7d5419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.579+0000 7f7d5aa6d640 1 --2- 172.21.15.107:0/422598717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d541010c0 0x7f7d54198580 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.579+0000 7f7d5aa6d640 1 --2- 172.21.15.107:0/422598717 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d54100700 0x7f7d54198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.579+0000 7f7d5aa6d640 1 -- 172.21.15.107:0/422598717 >> 172.21.15.107:0/422598717 conn(0x7f7d54077830 msgr2=0x7f7d540ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:00.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.579+0000 7f7d5aa6d640 1 -- 172.21.15.107:0/422598717 shutdown_connections 2024-05-09T08:41:00.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.579+0000 7f7d5aa6d640 1 -- 172.21.15.107:0/422598717 wait complete. 2024-05-09T08:41:00.591 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:00 smithi107 ceph-mon[30122]: pgmap v234: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 20 op/s 2024-05-09T08:41:00.592 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:41:00.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:41:00.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.708+0000 7f7b7f59e640 1 -- 172.21.15.107:0/3225823188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b80104210 msgr2=0x7f7b80110720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:00.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.708+0000 7f7b7f59e640 1 --2- 172.21.15.107:0/3225823188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b80104210 0x7f7b80110720 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f7b78009cd0 tx=0x7f7b7802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.709+0000 7f7b7f59e640 1 -- 172.21.15.107:0/3225823188 shutdown_connections 2024-05-09T08:41:00.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.709+0000 7f7b7f59e640 1 --2- 172.21.15.107:0/3225823188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b80104210 0x7f7b80110720 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.709+0000 7f7b7f59e640 1 --2- 172.21.15.107:0/3225823188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b80103850 0x7f7b80103cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.709+0000 7f7b7f59e640 1 --2- 172.21.15.107:0/3225823188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b80109860 0x7f7b80109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.709+0000 7f7b7f59e640 1 -- 172.21.15.107:0/3225823188 >> 172.21.15.107:0/3225823188 conn(0x7f7b800ff300 msgr2=0x7f7b80101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:00.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.709+0000 7f7b7f59e640 1 -- 172.21.15.107:0/3225823188 shutdown_connections 2024-05-09T08:41:00.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.709+0000 7f7b7f59e640 1 -- 172.21.15.107:0/3225823188 wait complete. 2024-05-09T08:41:00.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.709+0000 7f7b7f59e640 1 Processor -- start 2024-05-09T08:41:00.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.709+0000 7f7b7f59e640 1 -- start start 2024-05-09T08:41:00.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b80103850 0x7f7b8019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7f59e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b80104210 0x7f7b8019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7f59e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b80109860 0x7f7b8019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7f59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7b80113b20 con 0x7f7b80103850 2024-05-09T08:41:00.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7f59e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7b801139a0 con 0x7f7b80104210 2024-05-09T08:41:00.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7f59e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7b80113ca0 con 0x7f7b80109860 2024-05-09T08:41:00.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b80103850 0x7f7b8019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b80103850 0x7f7b8019a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47474/0 (socket says 172.21.15.107:47474) 2024-05-09T08:41:00.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7e59c640 1 -- 172.21.15.107:0/99070469 learned_addr learned my addr 172.21.15.107:0/99070469 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:00.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7dd9b640 1 --2- 172.21.15.107:0/99070469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b80104210 0x7f7b8019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7ed9d640 1 --2- 172.21.15.107:0/99070469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b80109860 0x7f7b8019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7e59c640 1 -- 172.21.15.107:0/99070469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b80109860 msgr2=0x7f7b8019e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:00.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7e59c640 1 --2- 172.21.15.107:0/99070469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b80109860 0x7f7b8019e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7e59c640 1 -- 172.21.15.107:0/99070469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b80104210 msgr2=0x7f7b8019a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:00.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7e59c640 1 --2- 172.21.15.107:0/99070469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b80104210 0x7f7b8019a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7e59c640 1 -- 172.21.15.107:0/99070469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8019f070 con 0x7f7b80103850 2024-05-09T08:41:00.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7ed9d640 1 --2- 172.21.15.107:0/99070469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b80109860 0x7f7b8019e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:00.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7e59c640 1 --2- 172.21.15.107:0/99070469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b80103850 0x7f7b8019a080 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000e9f0 tx=0x7f7b7000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:00.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b6f7fe640 1 -- 172.21.15.107:0/99070469 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7000cd80 con 0x7f7b80103850 2024-05-09T08:41:00.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b6f7fe640 1 -- 172.21.15.107:0/99070469 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b700054d0 con 0x7f7b80103850 2024-05-09T08:41:00.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7f59e640 1 -- 172.21.15.107:0/99070469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b8019f360 con 0x7f7b80103850 2024-05-09T08:41:00.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b6f7fe640 1 -- 172.21.15.107:0/99070469 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b70010490 con 0x7f7b80103850 2024-05-09T08:41:00.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.710+0000 7f7b7f59e640 1 -- 172.21.15.107:0/99070469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b801a6bf0 con 0x7f7b80103850 2024-05-09T08:41:00.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.712+0000 7f7b7f59e640 1 -- 172.21.15.107:0/99070469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b80068570 con 0x7f7b80103850 2024-05-09T08:41:00.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.713+0000 7f7b6f7fe640 1 -- 172.21.15.107:0/99070469 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7b70005020 con 0x7f7b80103850 2024-05-09T08:41:00.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.714+0000 7f7b6f7fe640 1 --2- 172.21.15.107:0/99070469 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b680a2d80 0x7f7b680a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:00.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.714+0000 7f7b7dd9b640 1 --2- 172.21.15.107:0/99070469 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b680a2d80 0x7f7b680a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:00.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.714+0000 7f7b6f7fe640 1 -- 172.21.15.107:0/99070469 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f7b700cfbc0 con 0x7f7b80103850 2024-05-09T08:41:00.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.714+0000 7f7b7dd9b640 1 --2- 172.21.15.107:0/99070469 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b680a2d80 0x7f7b680a5240 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7b801019f0 tx=0x7f7b74005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:00.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.718+0000 7f7b6f7fe640 1 -- 172.21.15.107:0/99070469 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7b7009c430 con 0x7f7b80103850 2024-05-09T08:41:00.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:00 smithi157 ceph-mon[33614]: pgmap v234: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 20 op/s 2024-05-09T08:41:00.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:00 smithi110 ceph-mon[33622]: pgmap v234: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 20 op/s 2024-05-09T08:41:00.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.989+0000 7f7b7f59e640 1 -- 172.21.15.107:0/99070469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b8010bde0 con 0x7f7b80103850 2024-05-09T08:41:00.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.990+0000 7f7b6f7fe640 1 -- 172.21.15.107:0/99070469 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11887 (secure 0 0 0) 0x7f7b7009bd30 con 0x7f7b80103850 2024-05-09T08:41:00.990 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:41:00.990 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:37:10.930317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:41:00.990 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2024-05-09T08:41:00.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.992+0000 7f7b7f59e640 1 -- 172.21.15.107:0/99070469 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b680a2d80 msgr2=0x7f7b680a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:00.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.992+0000 7f7b7f59e640 1 --2- 172.21.15.107:0/99070469 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b680a2d80 0x7f7b680a5240 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7b801019f0 tx=0x7f7b74005f90 comp rx=0 tx=0).stop 2024-05-09T08:41:00.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.992+0000 7f7b7f59e640 1 -- 172.21.15.107:0/99070469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b80103850 msgr2=0x7f7b8019a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:00.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.992+0000 7f7b7f59e640 1 --2- 172.21.15.107:0/99070469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b80103850 0x7f7b8019a080 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000e9f0 tx=0x7f7b7000edf0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.992+0000 7f7b7f59e640 1 -- 172.21.15.107:0/99070469 shutdown_connections 2024-05-09T08:41:00.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.992+0000 7f7b7f59e640 1 --2- 172.21.15.107:0/99070469 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b680a2d80 0x7f7b680a5240 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.992+0000 7f7b7f59e640 1 --2- 172.21.15.107:0/99070469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b80109860 0x7f7b8019e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.992+0000 7f7b7f59e640 1 --2- 172.21.15.107:0/99070469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b80104210 0x7f7b8019a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.992+0000 7f7b7f59e640 1 --2- 172.21.15.107:0/99070469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b80103850 0x7f7b8019a080 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:00.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.992+0000 7f7b7f59e640 1 -- 172.21.15.107:0/99070469 >> 172.21.15.107:0/99070469 conn(0x7f7b800ff300 msgr2=0x7f7b800ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:00.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.992+0000 7f7b7f59e640 1 -- 172.21.15.107:0/99070469 shutdown_connections 2024-05-09T08:41:00.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:00.992+0000 7f7b7f59e640 1 -- 172.21.15.107:0/99070469 wait complete. 2024-05-09T08:41:01.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:41:01.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.124+0000 7fdce63ce640 1 -- 172.21.15.107:0/2407364349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdce01010c0 msgr2=0x7fdce010d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.124+0000 7fdce63ce640 1 --2- 172.21.15.107:0/2407364349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdce01010c0 0x7fdce010d5d0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8009cd0 tx=0x7fdcd802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.124+0000 7fdce63ce640 1 -- 172.21.15.107:0/2407364349 shutdown_connections 2024-05-09T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.124+0000 7fdce63ce640 1 --2- 172.21.15.107:0/2407364349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdce01010c0 0x7fdce010d5d0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.124+0000 7fdce63ce640 1 --2- 172.21.15.107:0/2407364349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdce0100700 0x7fdce0100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.124+0000 7fdce63ce640 1 --2- 172.21.15.107:0/2407364349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdce0106710 0x7fdce0106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.124+0000 7fdce63ce640 1 -- 172.21.15.107:0/2407364349 >> 172.21.15.107:0/2407364349 conn(0x7fdce0077830 msgr2=0x7fdce0077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.124+0000 7fdce63ce640 1 -- 172.21.15.107:0/2407364349 shutdown_connections 2024-05-09T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.124+0000 7fdce63ce640 1 -- 172.21.15.107:0/2407364349 wait complete. 2024-05-09T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.125+0000 7fdce63ce640 1 Processor -- start 2024-05-09T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.125+0000 7fdce63ce640 1 -- start start 2024-05-09T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.125+0000 7fdce63ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdce0100700 0x7fdce0196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.125+0000 7fdce63ce640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdce01010c0 0x7fdce0197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.125+0000 7fdce63ce640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdce0106710 0x7fdce019b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.125+0000 7fdce63ce640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdce01109d0 con 0x7fdce0100700 2024-05-09T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.125+0000 7fdce63ce640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdce0110850 con 0x7fdce0106710 2024-05-09T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.125+0000 7fdce63ce640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdce0110b50 con 0x7fdce01010c0 2024-05-09T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.125+0000 7fdcdffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdce0100700 0x7fdce0196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.125+0000 7fdcdffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdce0100700 0x7fdce0196f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47494/0 (socket says 172.21.15.107:47494) 2024-05-09T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.125+0000 7fdcdffff640 1 -- 172.21.15.107:0/2695785354 learned_addr learned my addr 172.21.15.107:0/2695785354 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.125+0000 7fdcd7fff640 1 --2- 172.21.15.107:0/2695785354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdce01010c0 0x7fdce0197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.126+0000 7fdce496b640 1 --2- 172.21.15.107:0/2695785354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdce0106710 0x7fdce019b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.126+0000 7fdcdffff640 1 -- 172.21.15.107:0/2695785354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdce01010c0 msgr2=0x7fdce0197480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.126+0000 7fdcdffff640 1 --2- 172.21.15.107:0/2695785354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdce01010c0 0x7fdce0197480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.126+0000 7fdcdffff640 1 -- 172.21.15.107:0/2695785354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdce0106710 msgr2=0x7fdce019b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.126+0000 7fdcdffff640 1 --2- 172.21.15.107:0/2695785354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdce0106710 0x7fdce019b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.126+0000 7fdcdffff640 1 -- 172.21.15.107:0/2695785354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce019bf30 con 0x7fdce0100700 2024-05-09T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.126+0000 7fdcd7fff640 1 --2- 172.21.15.107:0/2695785354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdce01010c0 0x7fdce0197480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.126+0000 7fdce496b640 1 --2- 172.21.15.107:0/2695785354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdce0106710 0x7fdce019b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.126+0000 7fdcdffff640 1 --2- 172.21.15.107:0/2695785354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdce0100700 0x7fdce0196f40 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fdcd000eb00 tx=0x7fdcd000ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.126+0000 7fdcddffb640 1 -- 172.21.15.107:0/2695785354 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd000bc30 con 0x7fdce0100700 2024-05-09T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.126+0000 7fdce63ce640 1 -- 172.21.15.107:0/2695785354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdce019c220 con 0x7fdce0100700 2024-05-09T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.126+0000 7fdcddffb640 1 -- 172.21.15.107:0/2695785354 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcd000bdd0 con 0x7fdce0100700 2024-05-09T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.126+0000 7fdcddffb640 1 -- 172.21.15.107:0/2695785354 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0017410 con 0x7fdce0100700 2024-05-09T08:41:01.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.126+0000 7fdce63ce640 1 -- 172.21.15.107:0/2695785354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdce00689c0 con 0x7fdce0100700 2024-05-09T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.128+0000 7fdcddffb640 1 -- 172.21.15.107:0/2695785354 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fdcd0004f70 con 0x7fdce0100700 2024-05-09T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.129+0000 7fdcddffb640 1 --2- 172.21.15.107:0/2695785354 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdccc0a2d00 0x7fdccc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.129+0000 7fdce91b9740 1 Processor -- start 2024-05-09T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.129+0000 7fdce91b9740 1 -- start start 2024-05-09T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.129+0000 7fdcd7fff640 1 --2- 172.21.15.107:0/2695785354 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdccc0a2d00 0x7fdccc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.129+0000 7fdcddffb640 1 -- 172.21.15.107:0/2695785354 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fdcd00d11b0 con 0x7fdce0100700 2024-05-09T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.129+0000 7fdce91b9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563692851330 0x563692851710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.129+0000 7fdce91b9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851c50 0x56369285bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.129+0000 7fdce91b9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56369285c320 0x56369285e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.129+0000 7fdce91b9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5636927b6c40 con 0x563692851330 2024-05-09T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.129+0000 7fdce91b9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56369275e580 con 0x563692851c50 2024-05-09T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.129+0000 7fdce91b9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5636927b2c00 con 0x56369285c320 2024-05-09T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.129+0000 7fdcdffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563692851330 0x563692851710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.129+0000 7fdcdffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563692851330 0x563692851710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47500/0 (socket says 172.21.15.107:47500) 2024-05-09T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.129+0000 7fdcdffff640 1 -- 172.21.15.107:0/2725506768 learned_addr learned my addr 172.21.15.107:0/2725506768 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.129+0000 7fdce496b640 1 --2- 172.21.15.107:0/2725506768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56369285c320 0x56369285e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.130+0000 7fdcdffff640 1 -- 172.21.15.107:0/2725506768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56369285c320 msgr2=0x56369285e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.130+0000 7fdcdffff640 1 --2- 172.21.15.107:0/2725506768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56369285c320 0x56369285e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.130+0000 7fdcdffff640 1 -- 172.21.15.107:0/2725506768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851c50 msgr2=0x56369285bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.130+0000 7fdcdffff640 1 --2- 172.21.15.107:0/2725506768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851c50 0x56369285bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.130+0000 7fdcd7fff640 1 --2- 172.21.15.107:0/2695785354 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdccc0a2d00 0x7fdccc0a51c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80058c0 tx=0x7fdcc8005850 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.130+0000 7fdcdffff640 1 -- 172.21.15.107:0/2725506768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636924ca3f0 con 0x563692851330 2024-05-09T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.130+0000 7fdce496b640 1 --2- 172.21.15.107:0/2725506768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56369285c320 0x56369285e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.130+0000 7fdcdffff640 1 --2- 172.21.15.107:0/2725506768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563692851330 0x563692851710 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fdcd00a5a60 tx=0x7fdcd009dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48852fe79e04a067 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.130+0000 7fdcd67fc640 1 -- 172.21.15.107:0/2725506768 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd00a16d0 con 0x563692851330 2024-05-09T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.130+0000 7fdcd67fc640 1 -- 172.21.15.107:0/2725506768 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcd00a5f90 con 0x563692851330 2024-05-09T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.130+0000 7fdcd67fc640 1 -- 172.21.15.107:0/2725506768 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd00bd0d0 con 0x563692851330 2024-05-09T08:41:01.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.131+0000 7fdce91b9740 1 -- 172.21.15.107:0/2725506768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563692851330 msgr2=0x563692851710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.131+0000 7fdce91b9740 1 --2- 172.21.15.107:0/2725506768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563692851330 0x563692851710 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fdcd00a5a60 tx=0x7fdcd009dae0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.131+0000 7fdce91b9740 1 -- 172.21.15.107:0/2725506768 shutdown_connections 2024-05-09T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.131+0000 7fdce91b9740 1 --2- 172.21.15.107:0/2725506768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56369285c320 0x56369285e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.131+0000 7fdce91b9740 1 --2- 172.21.15.107:0/2725506768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851c50 0x56369285bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.131+0000 7fdce91b9740 1 --2- 172.21.15.107:0/2725506768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563692851330 0x563692851710 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.131+0000 7fdce91b9740 1 -- 172.21.15.107:0/2725506768 >> 172.21.15.107:0/2725506768 conn(0x56369284d650 msgr2=0x56369284da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.131+0000 7fdce91b9740 1 -- 172.21.15.107:0/2725506768 shutdown_connections 2024-05-09T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.131+0000 7fdce91b9740 1 -- 172.21.15.107:0/2725506768 wait complete. 2024-05-09T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.131+0000 7fdce91b9740 1 -- start start 2024-05-09T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdce91b9740 1 Processor -- start 2024-05-09T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdce91b9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563692851330 0x5636928592f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdce91b9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851c50 0x563692859830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdce91b9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56369285c320 0x563692859d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdce91b9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56369285f110 con 0x563692851330 2024-05-09T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdce91b9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563692771c40 con 0x563692851c50 2024-05-09T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdce91b9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56369285f290 con 0x56369285c320 2024-05-09T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdce91b9740 10 client.0 _ll_register_callbacks cb 0x5636924c9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdcdffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563692851330 0x5636928592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdcdffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563692851330 0x5636928592f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47516/0 (socket says 172.21.15.107:47516) 2024-05-09T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdcdffff640 1 -- 172.21.15.107:0/2305092028 learned_addr learned my addr 172.21.15.107:0/2305092028 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdcd7fff640 1 --2- 172.21.15.107:0/2305092028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851c50 0x563692859830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdce496b640 1 --2- 172.21.15.107:0/2305092028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56369285c320 0x563692859d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdcdffff640 1 -- 172.21.15.107:0/2305092028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56369285c320 msgr2=0x563692859d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdcdffff640 1 --2- 172.21.15.107:0/2305092028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56369285c320 0x563692859d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdcdffff640 1 -- 172.21.15.107:0/2305092028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851c50 msgr2=0x563692859830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdcdffff640 1 --2- 172.21.15.107:0/2305092028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851c50 0x563692859830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdcdffff640 1 -- 172.21.15.107:0/2305092028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636925947d0 con 0x563692851330 2024-05-09T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdce496b640 1 --2- 172.21.15.107:0/2305092028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56369285c320 0x563692859d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdcdffff640 1 --2- 172.21.15.107:0/2305092028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563692851330 0x5636928592f0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0099780 tx=0x7fdcd0099ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c344a154908846d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.132+0000 7fdcbf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.133+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/2305092028 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd00b5070 con 0x563692851330 2024-05-09T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.133+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/2305092028 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fdcd0005260 con 0x563692851330 2024-05-09T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.133+0000 7fdce91b9740 1 -- 172.21.15.107:0/2305092028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcd00a5f90 con 0x563692851330 2024-05-09T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.133+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/2305092028 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdcd0005460 con 0x563692851330 2024-05-09T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.133+0000 7fdce91b9740 10 client.17871 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.133+0000 7fdce91b9740 1 -- 172.21.15.107:0/2305092028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56369286b350 con 0x563692851330 2024-05-09T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.133+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/2305092028 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcd00a5f90 con 0x563692851330 2024-05-09T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.133+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/2305092028 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fdcd00a1120 con 0x563692851330 2024-05-09T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.133+0000 7fdce91b9740 10 client.17871 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:01.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.133+0000 7fdce91b9740 10 client.17871 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.133+0000 7fdce91b9740 20 client.17871 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.133+0000 7fdce91b9740 1 --2- 172.21.15.107:0/2305092028 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x56369286c6a0 0x56369286ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.133+0000 7fdce91b9740 4 client.17871 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.133+0000 7fdce91b9740 1 -- 172.21.15.107:0/2305092028 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636927c4370 con 0x56369286c6a0 2024-05-09T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.133+0000 7fdcd7fff640 1 --2- 172.21.15.107:0/2305092028 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x56369286c6a0 0x56369286ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.134+0000 7fdcd7fff640 1 --2- 172.21.15.107:0/2305092028 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x56369286c6a0 0x56369286ea90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1aa58e7a6bc6cc7 server_cookie=f4b3e842cbdf3982 in_seq=0 out_seq=0 2024-05-09T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.134+0000 7fdcbf7fe640 10 client.17871 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.134+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/2305092028 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5636927c4370 con 0x56369286c6a0 2024-05-09T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.134+0000 7fdcbf7fe640 10 client.17871 handle_command_reply: tid=0 2024-05-09T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.134+0000 7fdce91b9740 1 client.17871 shutdown 2024-05-09T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.135+0000 7fdce91b9740 1 -- 172.21.15.107:0/2305092028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563692851330 msgr2=0x5636928592f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.135+0000 7fdce91b9740 1 --2- 172.21.15.107:0/2305092028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563692851330 0x5636928592f0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0099780 tx=0x7fdcd0099ab0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.135+0000 7fdce91b9740 1 -- 172.21.15.107:0/2305092028 shutdown_connections 2024-05-09T08:41:01.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.135+0000 7fdce91b9740 1 --2- 172.21.15.107:0/2305092028 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x56369286c6a0 0x56369286ea90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.135+0000 7fdce91b9740 1 --2- 172.21.15.107:0/2305092028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56369285c320 0x563692859d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.135+0000 7fdce91b9740 1 --2- 172.21.15.107:0/2305092028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851c50 0x563692859830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.135+0000 7fdcbf7fe640 0 client.17871 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.135+0000 7fdce91b9740 1 --2- 172.21.15.107:0/2305092028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563692851330 0x5636928592f0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.135+0000 7fdce91b9740 1 -- 172.21.15.107:0/2305092028 >> 172.21.15.107:0/2305092028 conn(0x56369284d650 msgr2=0x563692853350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.135+0000 7fdce91b9740 1 -- 172.21.15.107:0/2305092028 shutdown_connections 2024-05-09T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.135+0000 7fdce91b9740 1 -- 172.21.15.107:0/2305092028 wait complete. 2024-05-09T08:41:01.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.135+0000 7fdce91b9740 20 client.17871 trim_cache size 0 max 16384 2024-05-09T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdce91b9740 1 Processor -- start 2024-05-09T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdce91b9740 1 -- start start 2024-05-09T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdce91b9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851330 0x56369285ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdce91b9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563692851c50 0x56369284cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdce91b9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56369285c320 0x563692853a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdce91b9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdce01af260 con 0x56369285c320 2024-05-09T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdce91b9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdce01acb30 con 0x563692851330 2024-05-09T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdce91b9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdce01aa9a0 con 0x563692851c50 2024-05-09T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdce496b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56369285c320 0x563692853a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdce496b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56369285c320 0x563692853a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47530/0 (socket says 172.21.15.107:47530) 2024-05-09T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdce496b640 1 -- 172.21.15.107:0/881976364 learned_addr learned my addr 172.21.15.107:0/881976364 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdcdffff640 1 --2- 172.21.15.107:0/881976364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851330 0x56369285ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdcd7fff640 1 --2- 172.21.15.107:0/881976364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563692851c50 0x56369284cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdce496b640 1 -- 172.21.15.107:0/881976364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563692851c50 msgr2=0x56369284cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdce496b640 1 --2- 172.21.15.107:0/881976364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563692851c50 0x56369284cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdce496b640 1 -- 172.21.15.107:0/881976364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851330 msgr2=0x56369285ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdce496b640 1 --2- 172.21.15.107:0/881976364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851330 0x56369285ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.150+0000 7fdce496b640 1 -- 172.21.15.107:0/881976364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56369285a250 con 0x56369285c320 2024-05-09T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.151+0000 7fdce496b640 1 --2- 172.21.15.107:0/881976364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56369285c320 0x563692853a00 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8035000 tx=0x7fdcd8009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3714ef1da296d35 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.151+0000 7fdcd5ffb640 1 -- 172.21.15.107:0/881976364 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8031560 con 0x56369285c320 2024-05-09T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.151+0000 7fdcd5ffb640 1 -- 172.21.15.107:0/881976364 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcd8036040 con 0x56369285c320 2024-05-09T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.151+0000 7fdcd5ffb640 1 -- 172.21.15.107:0/881976364 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8004310 con 0x56369285c320 2024-05-09T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 -- 172.21.15.107:0/881976364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56369285c320 msgr2=0x563692853a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 --2- 172.21.15.107:0/881976364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56369285c320 0x563692853a00 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8035000 tx=0x7fdcd8009a20 comp rx=0 tx=0).stop 2024-05-09T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 -- 172.21.15.107:0/881976364 shutdown_connections 2024-05-09T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 --2- 172.21.15.107:0/881976364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56369285c320 0x563692853a00 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 --2- 172.21.15.107:0/881976364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563692851c50 0x56369284cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 --2- 172.21.15.107:0/881976364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851330 0x56369285ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 -- 172.21.15.107:0/881976364 >> 172.21.15.107:0/881976364 conn(0x56369284d650 msgr2=0x563692859b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 -- 172.21.15.107:0/881976364 shutdown_connections 2024-05-09T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 -- 172.21.15.107:0/881976364 wait complete. 2024-05-09T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 -- start start 2024-05-09T08:41:01.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 Processor -- start 2024-05-09T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563692851330 0x563692864970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851c50 0x56369285fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56369285c320 0x5636928602f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdce01aaff0 con 0x56369285c320 2024-05-09T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdce01ac460 con 0x563692851c50 2024-05-09T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.152+0000 7fdce91b9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdce01ab6c0 con 0x563692851330 2024-05-09T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdce91b9740 10 client.0 _ll_register_callbacks cb 0x7fdcd009cc80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdce496b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56369285c320 0x5636928602f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdcdffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563692851330 0x563692864970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdce496b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56369285c320 0x5636928602f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47542/0 (socket says 172.21.15.107:47542) 2024-05-09T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdce496b640 1 -- 172.21.15.107:0/2480326374 learned_addr learned my addr 172.21.15.107:0/2480326374 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdcd7fff640 1 --2- 172.21.15.107:0/2480326374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851c50 0x56369285fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdce496b640 1 -- 172.21.15.107:0/2480326374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563692851330 msgr2=0x563692864970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdce496b640 1 --2- 172.21.15.107:0/2480326374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563692851330 0x563692864970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdce496b640 1 -- 172.21.15.107:0/2480326374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851c50 msgr2=0x56369285fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdce496b640 1 --2- 172.21.15.107:0/2480326374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851c50 0x56369285fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdce496b640 1 -- 172.21.15.107:0/2480326374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563692860af0 con 0x56369285c320 2024-05-09T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdcdffff640 1 --2- 172.21.15.107:0/2480326374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563692851330 0x563692864970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdce496b640 1 --2- 172.21.15.107:0/2480326374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56369285c320 0x5636928602f0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8009430 tx=0x7fdcd8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=461235d912d2ea33 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdcbf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/2480326374 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8050020 con 0x56369285c320 2024-05-09T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdce91b9740 1 -- 172.21.15.107:0/2480326374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcd8036040 con 0x56369285c320 2024-05-09T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/2480326374 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fdcd8031a60 con 0x56369285c320 2024-05-09T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/2480326374 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdcd8031c60 con 0x56369285c320 2024-05-09T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdce91b9740 10 client.17877 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.153+0000 7fdce91b9740 1 -- 172.21.15.107:0/2480326374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5636928d29f0 con 0x56369285c320 2024-05-09T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.154+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/2480326374 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcd8036040 con 0x56369285c320 2024-05-09T08:41:01.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.154+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/2480326374 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fdcd8042070 con 0x56369285c320 2024-05-09T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.154+0000 7fdce91b9740 10 client.17877 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.154+0000 7fdce91b9740 10 client.17877 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.154+0000 7fdce91b9740 20 client.17877 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.154+0000 7fdce91b9740 1 --2- 172.21.15.107:0/2480326374 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5636928d3100 0x5636928d54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.154+0000 7fdce91b9740 4 client.17877 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.154+0000 7fdce91b9740 1 -- 172.21.15.107:0/2480326374 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5636927be6c0 con 0x5636928d3100 2024-05-09T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.154+0000 7fdcdffff640 1 --2- 172.21.15.107:0/2480326374 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5636928d3100 0x5636928d54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.154+0000 7fdcbf7fe640 10 client.17877 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.154+0000 7fdcdffff640 1 --2- 172.21.15.107:0/2480326374 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5636928d3100 0x5636928d54f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7ac8e4b84b1e4d06 server_cookie=d8a6e76288a9b8b1 in_seq=0 out_seq=0 2024-05-09T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/2480326374 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5636927be6c0 con 0x5636928d3100 2024-05-09T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdcbf7fe640 10 client.17877 handle_command_reply: tid=0 2024-05-09T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdce91b9740 1 client.17877 shutdown 2024-05-09T08:41:01.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdce91b9740 1 -- 172.21.15.107:0/2480326374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56369285c320 msgr2=0x5636928602f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdce91b9740 1 --2- 172.21.15.107:0/2480326374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56369285c320 0x5636928602f0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8009430 tx=0x7fdcd8004a40 comp rx=0 tx=0).stop 2024-05-09T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdce91b9740 1 -- 172.21.15.107:0/2480326374 shutdown_connections 2024-05-09T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdce91b9740 1 --2- 172.21.15.107:0/2480326374 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5636928d3100 0x5636928d54f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdce91b9740 1 --2- 172.21.15.107:0/2480326374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56369285c320 0x5636928602f0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdce91b9740 1 --2- 172.21.15.107:0/2480326374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563692851c50 0x56369285fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdce91b9740 1 --2- 172.21.15.107:0/2480326374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563692851330 0x563692864970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdce91b9740 1 -- 172.21.15.107:0/2480326374 >> 172.21.15.107:0/2480326374 conn(0x56369284d650 msgr2=0x56369284bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdcbf7fe640 0 client.17877 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdce91b9740 1 -- 172.21.15.107:0/2480326374 shutdown_connections 2024-05-09T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdce91b9740 1 -- 172.21.15.107:0/2480326374 wait complete. 2024-05-09T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdce91b9740 20 client.17877 trim_cache size 0 max 16384 2024-05-09T08:41:01.161 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdce63ce640 1 -- 172.21.15.107:0/2695785354 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdccc0a2d00 msgr2=0x7fdccc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdce63ce640 1 --2- 172.21.15.107:0/2695785354 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdccc0a2d00 0x7fdccc0a51c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80058c0 tx=0x7fdcc8005850 comp rx=0 tx=0).stop 2024-05-09T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdce63ce640 1 -- 172.21.15.107:0/2695785354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdce0100700 msgr2=0x7fdce0196f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.155+0000 7fdce63ce640 1 --2- 172.21.15.107:0/2695785354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdce0100700 0x7fdce0196f40 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fdcd000eb00 tx=0x7fdcd000ef00 comp rx=0 tx=0).stop 2024-05-09T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.156+0000 7fdce63ce640 1 -- 172.21.15.107:0/2695785354 shutdown_connections 2024-05-09T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.156+0000 7fdce63ce640 1 --2- 172.21.15.107:0/2695785354 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdccc0a2d00 0x7fdccc0a51c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.156+0000 7fdce63ce640 1 --2- 172.21.15.107:0/2695785354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdce0106710 0x7fdce019b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.156+0000 7fdce63ce640 1 --2- 172.21.15.107:0/2695785354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdce01010c0 0x7fdce0197480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.156+0000 7fdce63ce640 1 --2- 172.21.15.107:0/2695785354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdce0100700 0x7fdce0196f40 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.156+0000 7fdce63ce640 1 -- 172.21.15.107:0/2695785354 >> 172.21.15.107:0/2695785354 conn(0x7fdce0077830 msgr2=0x7fdce0103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.156+0000 7fdce63ce640 1 -- 172.21.15.107:0/2695785354 shutdown_connections 2024-05-09T08:41:01.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.156+0000 7fdce63ce640 1 -- 172.21.15.107:0/2695785354 wait complete. 2024-05-09T08:41:01.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:41:01.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.280+0000 7fde6c1c8640 1 -- 172.21.15.107:0/1782650210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde64100700 msgr2=0x7fde64100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.280+0000 7fde6c1c8640 1 --2- 172.21.15.107:0/1782650210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde64100700 0x7fde64100b80 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fde54009930 tx=0x7fde5402a420 comp rx=0 tx=0).stop 2024-05-09T08:41:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.280+0000 7fde6c1c8640 1 -- 172.21.15.107:0/1782650210 shutdown_connections 2024-05-09T08:41:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.280+0000 7fde6c1c8640 1 --2- 172.21.15.107:0/1782650210 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde641010c0 0x7fde6410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.280+0000 7fde6c1c8640 1 --2- 172.21.15.107:0/1782650210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde64100700 0x7fde64100b80 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.280+0000 7fde6c1c8640 1 --2- 172.21.15.107:0/1782650210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde64106710 0x7fde64106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.280+0000 7fde6c1c8640 1 -- 172.21.15.107:0/1782650210 >> 172.21.15.107:0/1782650210 conn(0x7fde64077830 msgr2=0x7fde64077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.280+0000 7fde6c1c8640 1 -- 172.21.15.107:0/1782650210 shutdown_connections 2024-05-09T08:41:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.280+0000 7fde6c1c8640 1 -- 172.21.15.107:0/1782650210 wait complete. 2024-05-09T08:41:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.281+0000 7fde6c1c8640 1 Processor -- start 2024-05-09T08:41:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.281+0000 7fde6c1c8640 1 -- start start 2024-05-09T08:41:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.281+0000 7fde6c1c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde64100700 0x7fde64198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.281+0000 7fde6c1c8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde641010c0 0x7fde641993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.281+0000 7fde6c1c8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde64106710 0x7fde641a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.281+0000 7fde6c1c8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fde64110990 con 0x7fde64100700 2024-05-09T08:41:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.281+0000 7fde6c1c8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fde64110810 con 0x7fde64106710 2024-05-09T08:41:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.281+0000 7fde6c1c8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fde64110b10 con 0x7fde641010c0 2024-05-09T08:41:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.281+0000 7fde69f64640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde64100700 0x7fde64198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.281+0000 7fde69763640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde641010c0 0x7fde641993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.281+0000 7fde69f64640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde64100700 0x7fde64198e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47564/0 (socket says 172.21.15.107:47564) 2024-05-09T08:41:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.281+0000 7fde69763640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde641010c0 0x7fde641993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:57080/0 (socket says 172.21.15.107:57080) 2024-05-09T08:41:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.281+0000 7fde69f64640 1 -- 172.21.15.107:0/1422524867 learned_addr learned my addr 172.21.15.107:0/1422524867 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.281+0000 7fde6a765640 1 --2- 172.21.15.107:0/1422524867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde64106710 0x7fde641a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.282+0000 7fde69f64640 1 -- 172.21.15.107:0/1422524867 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde641010c0 msgr2=0x7fde641993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.282+0000 7fde69f64640 1 --2- 172.21.15.107:0/1422524867 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde641010c0 0x7fde641993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.282+0000 7fde69f64640 1 -- 172.21.15.107:0/1422524867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde64106710 msgr2=0x7fde641a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.282+0000 7fde69f64640 1 --2- 172.21.15.107:0/1422524867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde64106710 0x7fde641a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.282+0000 7fde69f64640 1 -- 172.21.15.107:0/1422524867 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde641a4430 con 0x7fde64100700 2024-05-09T08:41:01.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.282+0000 7fde69f64640 1 --2- 172.21.15.107:0/1422524867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde64100700 0x7fde64198e70 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fde6000d950 tx=0x7fde6000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.282+0000 7fde5affd640 1 -- 172.21.15.107:0/1422524867 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde60017030 con 0x7fde64100700 2024-05-09T08:41:01.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.282+0000 7fde5affd640 1 -- 172.21.15.107:0/1422524867 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde60004020 con 0x7fde64100700 2024-05-09T08:41:01.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.282+0000 7fde6c1c8640 1 -- 172.21.15.107:0/1422524867 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde641a4600 con 0x7fde64100700 2024-05-09T08:41:01.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.282+0000 7fde5affd640 1 -- 172.21.15.107:0/1422524867 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde600047a0 con 0x7fde64100700 2024-05-09T08:41:01.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.282+0000 7fde6c1c8640 1 -- 172.21.15.107:0/1422524867 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde641a4b70 con 0x7fde64100700 2024-05-09T08:41:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.284+0000 7fde5affd640 1 -- 172.21.15.107:0/1422524867 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fde600041d0 con 0x7fde64100700 2024-05-09T08:41:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.285+0000 7fde6efb1740 1 Processor -- start 2024-05-09T08:41:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.285+0000 7fde6efb1740 1 -- start start 2024-05-09T08:41:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.285+0000 7fde6efb1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdcf330 0x5599fbdcf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.285+0000 7fde6efb1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdcfc50 0x5599fbdd9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.285+0000 7fde6efb1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdda320 0x5599fbddc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.285+0000 7fde6efb1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5599fbd34c40 con 0x5599fbdda320 2024-05-09T08:41:01.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.285+0000 7fde6efb1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5599fbcdc580 con 0x5599fbdcfc50 2024-05-09T08:41:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.285+0000 7fde6efb1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5599fbd30c00 con 0x5599fbdcf330 2024-05-09T08:41:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.285+0000 7fde69763640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdda320 0x5599fbddc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.285+0000 7fde69763640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdda320 0x5599fbddc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47570/0 (socket says 172.21.15.107:47570) 2024-05-09T08:41:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.285+0000 7fde69763640 1 -- 172.21.15.107:0/2599815309 learned_addr learned my addr 172.21.15.107:0/2599815309 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.285+0000 7fde6a765640 1 --2- 172.21.15.107:0/2599815309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdcf330 0x5599fbdcf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.285+0000 7fde5affd640 1 --2- 172.21.15.107:0/1422524867 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fde500a2d80 0x7fde500a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.285+0000 7fde69f64640 1 --2- 172.21.15.107:0/2599815309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdcfc50 0x5599fbdd9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.286+0000 7fde6a765640 1 --2- 172.21.15.107:0/1422524867 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fde500a2d80 0x7fde500a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.286+0000 7fde69763640 1 -- 172.21.15.107:0/2599815309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdcf330 msgr2=0x5599fbdcf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.286+0000 7fde69763640 1 --2- 172.21.15.107:0/2599815309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdcf330 0x5599fbdcf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.286+0000 7fde69763640 1 -- 172.21.15.107:0/2599815309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdcfc50 msgr2=0x5599fbdd9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.286+0000 7fde69763640 1 --2- 172.21.15.107:0/2599815309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdcfc50 0x5599fbdd9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.286+0000 7fde69763640 1 -- 172.21.15.107:0/2599815309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599fba483f0 con 0x5599fbdda320 2024-05-09T08:41:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.286+0000 7fde6a765640 1 --2- 172.21.15.107:0/2599815309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdcf330 0x5599fbdcf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.286+0000 7fde5affd640 1 -- 172.21.15.107:0/1422524867 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fde600cf100 con 0x7fde64100700 2024-05-09T08:41:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.286+0000 7fde69763640 1 --2- 172.21.15.107:0/2599815309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdda320 0x5599fbddc710 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fde5402a940 tx=0x7fde54031ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a344610d3127f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.286+0000 7fde6a765640 1 --2- 172.21.15.107:0/1422524867 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fde500a2d80 0x7fde500a5240 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fde5c005c40 tx=0x7fde5c004ab0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.286+0000 7fde4ffff640 1 -- 172.21.15.107:0/2599815309 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde54040070 con 0x5599fbdda320 2024-05-09T08:41:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.286+0000 7fde4ffff640 1 -- 172.21.15.107:0/2599815309 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde54031e50 con 0x5599fbdda320 2024-05-09T08:41:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.286+0000 7fde4ffff640 1 -- 172.21.15.107:0/2599815309 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde54035040 con 0x5599fbdda320 2024-05-09T08:41:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.287+0000 7fde6efb1740 1 -- 172.21.15.107:0/2599815309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdda320 msgr2=0x5599fbddc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.287+0000 7fde6efb1740 1 --2- 172.21.15.107:0/2599815309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdda320 0x5599fbddc710 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fde5402a940 tx=0x7fde54031ba0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.287+0000 7fde6efb1740 1 -- 172.21.15.107:0/2599815309 shutdown_connections 2024-05-09T08:41:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.287+0000 7fde6efb1740 1 --2- 172.21.15.107:0/2599815309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdda320 0x5599fbddc710 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.287+0000 7fde6efb1740 1 --2- 172.21.15.107:0/2599815309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdcfc50 0x5599fbdd9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.287+0000 7fde6efb1740 1 --2- 172.21.15.107:0/2599815309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdcf330 0x5599fbdcf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.287+0000 7fde6efb1740 1 -- 172.21.15.107:0/2599815309 >> 172.21.15.107:0/2599815309 conn(0x5599fbdcb650 msgr2=0x5599fbdcba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.287+0000 7fde6efb1740 1 -- 172.21.15.107:0/2599815309 shutdown_connections 2024-05-09T08:41:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.287+0000 7fde6efb1740 1 -- 172.21.15.107:0/2599815309 wait complete. 2024-05-09T08:41:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.288+0000 7fde6efb1740 1 -- start start 2024-05-09T08:41:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.288+0000 7fde6efb1740 1 Processor -- start 2024-05-09T08:41:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.288+0000 7fde6efb1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcf330 0x5599fbdd71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.288+0000 7fde6efb1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdcfc50 0x5599fbdd7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.288+0000 7fde6efb1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdda320 0x5599fbdd7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.288+0000 7fde6efb1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5599fbddd0b0 con 0x5599fbdcf330 2024-05-09T08:41:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.288+0000 7fde6efb1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5599fbcefc40 con 0x5599fbdda320 2024-05-09T08:41:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.288+0000 7fde6efb1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5599fbddd230 con 0x5599fbdcfc50 2024-05-09T08:41:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.288+0000 7fde6efb1740 10 client.0 _ll_register_callbacks cb 0x5599fba47be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.288+0000 7fde69f64640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcf330 0x5599fbdd71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.288+0000 7fde69f64640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcf330 0x5599fbdd71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47586/0 (socket says 172.21.15.107:47586) 2024-05-09T08:41:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.288+0000 7fde69f64640 1 -- 172.21.15.107:0/246616411 learned_addr learned my addr 172.21.15.107:0/246616411 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.288+0000 7fde69763640 1 --2- 172.21.15.107:0/246616411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdcfc50 0x5599fbdd7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.288+0000 7fde6a765640 1 --2- 172.21.15.107:0/246616411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdda320 0x5599fbdd7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.289+0000 7fde69f64640 1 -- 172.21.15.107:0/246616411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdcfc50 msgr2=0x5599fbdd7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.289+0000 7fde69f64640 1 --2- 172.21.15.107:0/246616411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdcfc50 0x5599fbdd7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.289+0000 7fde69f64640 1 -- 172.21.15.107:0/246616411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdda320 msgr2=0x5599fbdd7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.289+0000 7fde69f64640 1 --2- 172.21.15.107:0/246616411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdda320 0x5599fbdd7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.289+0000 7fde69f64640 1 -- 172.21.15.107:0/246616411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599fbb127d0 con 0x5599fbdcf330 2024-05-09T08:41:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.289+0000 7fde69f64640 1 --2- 172.21.15.107:0/246616411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcf330 0x5599fbdd71f0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fde600cc550 tx=0x7fde600cf300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2b9348dcf715419 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.289+0000 7fde4d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.289+0000 7fde4d7fa640 1 -- 172.21.15.107:0/246616411 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde600a3ee0 con 0x5599fbdcf330 2024-05-09T08:41:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.289+0000 7fde4d7fa640 1 -- 172.21.15.107:0/246616411 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fde60009a30 con 0x5599fbdcf330 2024-05-09T08:41:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.289+0000 7fde6efb1740 1 -- 172.21.15.107:0/246616411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde54031e50 con 0x5599fbdcf330 2024-05-09T08:41:01.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.289+0000 7fde4d7fa640 1 -- 172.21.15.107:0/246616411 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fde6009b270 con 0x5599fbdcf330 2024-05-09T08:41:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.289+0000 7fde6efb1740 10 client.17889 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.289+0000 7fde6efb1740 1 -- 172.21.15.107:0/246616411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5599fbde9230 con 0x5599fbdcf330 2024-05-09T08:41:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.289+0000 7fde4d7fa640 1 -- 172.21.15.107:0/246616411 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde54031e50 con 0x5599fbdcf330 2024-05-09T08:41:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.289+0000 7fde4d7fa640 1 -- 172.21.15.107:0/246616411 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fde600a3ae0 con 0x5599fbdcf330 2024-05-09T08:41:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.290+0000 7fde6efb1740 10 client.17889 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.290+0000 7fde6efb1740 10 client.17889 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:41:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.290+0000 7fde6efb1740 20 client.17889 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.290+0000 7fde6efb1740 1 --2- 172.21.15.107:0/246616411 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5599fbdea6f0 0x5599fbdecae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.290+0000 7fde6efb1740 4 client.17889 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.290+0000 7fde6efb1740 1 -- 172.21.15.107:0/246616411 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599fbd42370 con 0x5599fbdea6f0 2024-05-09T08:41:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.290+0000 7fde69763640 1 --2- 172.21.15.107:0/246616411 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5599fbdea6f0 0x5599fbdecae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.290+0000 7fde69763640 1 --2- 172.21.15.107:0/246616411 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5599fbdea6f0 0x5599fbdecae0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1f430b72d83d89f6 server_cookie=e4b5e96d330bfd3e in_seq=0 out_seq=0 2024-05-09T08:41:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.290+0000 7fde4d7fa640 10 client.17889 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:41:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.291+0000 7fde4d7fa640 1 -- 172.21.15.107:0/246616411 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5599fbd42370 con 0x5599fbdea6f0 2024-05-09T08:41:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.291+0000 7fde4d7fa640 10 client.17889 handle_command_reply: tid=0 2024-05-09T08:41:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.291+0000 7fde6efb1740 1 client.17889 shutdown 2024-05-09T08:41:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.291+0000 7fde6efb1740 1 -- 172.21.15.107:0/246616411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcf330 msgr2=0x5599fbdd71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.291+0000 7fde6efb1740 1 --2- 172.21.15.107:0/246616411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcf330 0x5599fbdd71f0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fde600cc550 tx=0x7fde600cf300 comp rx=0 tx=0).stop 2024-05-09T08:41:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.291+0000 7fde6efb1740 1 -- 172.21.15.107:0/246616411 shutdown_connections 2024-05-09T08:41:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.291+0000 7fde6efb1740 1 --2- 172.21.15.107:0/246616411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdda320 0x5599fbdd7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.291+0000 7fde6efb1740 1 --2- 172.21.15.107:0/246616411 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5599fbdea6f0 0x5599fbdecae0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.291+0000 7fde6efb1740 1 --2- 172.21.15.107:0/246616411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdcfc50 0x5599fbdd7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.291+0000 7fde6efb1740 1 --2- 172.21.15.107:0/246616411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcf330 0x5599fbdd71f0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.291+0000 7fde4d7fa640 0 client.17889 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:41:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.291+0000 7fde6efb1740 1 -- 172.21.15.107:0/246616411 >> 172.21.15.107:0/246616411 conn(0x5599fbdcb650 msgr2=0x5599fbdd1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.291+0000 7fde6efb1740 1 -- 172.21.15.107:0/246616411 shutdown_connections 2024-05-09T08:41:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.291+0000 7fde6efb1740 1 -- 172.21.15.107:0/246616411 wait complete. 2024-05-09T08:41:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.292+0000 7fde6efb1740 20 client.17889 trim_cache size 0 max 16384 2024-05-09T08:41:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.305+0000 7fde6efb1740 1 Processor -- start 2024-05-09T08:41:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.305+0000 7fde6efb1740 1 -- start start 2024-05-09T08:41:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.305+0000 7fde6efb1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdcf330 0x5599fbde5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.305+0000 7fde6efb1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcfc50 0x5599fbdd1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.305+0000 7fde6efb1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdda320 0x5599fbdca630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.305+0000 7fde6efb1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fde641af310 con 0x5599fbdcfc50 2024-05-09T08:41:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.305+0000 7fde6efb1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fde641acbe0 con 0x5599fbdda320 2024-05-09T08:41:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.305+0000 7fde6efb1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fde641aaa50 con 0x5599fbdcf330 2024-05-09T08:41:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.305+0000 7fde69763640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcfc50 0x5599fbdd1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.305+0000 7fde69763640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcfc50 0x5599fbdd1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47594/0 (socket says 172.21.15.107:47594) 2024-05-09T08:41:01.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.305+0000 7fde69763640 1 -- 172.21.15.107:0/2488087023 learned_addr learned my addr 172.21.15.107:0/2488087023 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:01.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.306+0000 7fde6a765640 1 --2- 172.21.15.107:0/2488087023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdda320 0x5599fbdca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.306+0000 7fde69f64640 1 --2- 172.21.15.107:0/2488087023 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdcf330 0x5599fbde5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.306+0000 7fde69763640 1 -- 172.21.15.107:0/2488087023 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdcf330 msgr2=0x5599fbde5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.306+0000 7fde69763640 1 --2- 172.21.15.107:0/2488087023 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdcf330 0x5599fbde5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.306+0000 7fde69763640 1 -- 172.21.15.107:0/2488087023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdda320 msgr2=0x5599fbdca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.306+0000 7fde69763640 1 --2- 172.21.15.107:0/2488087023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdda320 0x5599fbdca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.306+0000 7fde69763640 1 -- 172.21.15.107:0/2488087023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599fbdece50 con 0x5599fbdcfc50 2024-05-09T08:41:01.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.306+0000 7fde6a765640 1 --2- 172.21.15.107:0/2488087023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdda320 0x5599fbdca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:01.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.306+0000 7fde69763640 1 --2- 172.21.15.107:0/2488087023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcfc50 0x5599fbdd1ba0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fde54031440 tx=0x7fde54005500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e82dfb2ea7c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.306+0000 7fde4f7fe640 1 -- 172.21.15.107:0/2488087023 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde54040040 con 0x5599fbdcfc50 2024-05-09T08:41:01.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.306+0000 7fde4f7fe640 1 -- 172.21.15.107:0/2488087023 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde54031e50 con 0x5599fbdcfc50 2024-05-09T08:41:01.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.306+0000 7fde4f7fe640 1 -- 172.21.15.107:0/2488087023 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde54035040 con 0x5599fbdcfc50 2024-05-09T08:41:01.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.307+0000 7fde6efb1740 1 -- 172.21.15.107:0/2488087023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcfc50 msgr2=0x5599fbdd1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.307+0000 7fde6efb1740 1 --2- 172.21.15.107:0/2488087023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcfc50 0x5599fbdd1ba0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fde54031440 tx=0x7fde54005500 comp rx=0 tx=0).stop 2024-05-09T08:41:01.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.307+0000 7fde6efb1740 1 -- 172.21.15.107:0/2488087023 shutdown_connections 2024-05-09T08:41:01.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.307+0000 7fde6efb1740 1 --2- 172.21.15.107:0/2488087023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdda320 0x5599fbdca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.307+0000 7fde6efb1740 1 --2- 172.21.15.107:0/2488087023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcfc50 0x5599fbdd1ba0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.307+0000 7fde6efb1740 1 --2- 172.21.15.107:0/2488087023 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdcf330 0x5599fbde5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.307+0000 7fde6efb1740 1 -- 172.21.15.107:0/2488087023 >> 172.21.15.107:0/2488087023 conn(0x5599fbdcb650 msgr2=0x5599fbdc9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.307+0000 7fde6efb1740 1 -- 172.21.15.107:0/2488087023 shutdown_connections 2024-05-09T08:41:01.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.307+0000 7fde6efb1740 1 -- 172.21.15.107:0/2488087023 wait complete. 2024-05-09T08:41:01.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.307+0000 7fde6efb1740 1 -- start start 2024-05-09T08:41:01.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.307+0000 7fde6efb1740 1 Processor -- start 2024-05-09T08:41:01.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde6efb1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdcf330 0x5599fbdd4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde6efb1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcfc50 0x5599fbdd4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde6efb1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdda320 0x5599fbe4fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde6efb1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fde641ab0a0 con 0x5599fbdcfc50 2024-05-09T08:41:01.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde6efb1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fde641ac510 con 0x5599fbdcf330 2024-05-09T08:41:01.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde6efb1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fde641ab770 con 0x5599fbdda320 2024-05-09T08:41:01.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde6efb1740 10 client.0 _ll_register_callbacks cb 0x7fde600046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:01.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde69763640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcfc50 0x5599fbdd4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde69763640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcfc50 0x5599fbdd4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47600/0 (socket says 172.21.15.107:47600) 2024-05-09T08:41:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde69763640 1 -- 172.21.15.107:0/803070402 learned_addr learned my addr 172.21.15.107:0/803070402 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde6a765640 1 --2- 172.21.15.107:0/803070402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdda320 0x5599fbe4fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde69f64640 1 --2- 172.21.15.107:0/803070402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdcf330 0x5599fbdd4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde69763640 1 -- 172.21.15.107:0/803070402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdda320 msgr2=0x5599fbe4fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde69763640 1 --2- 172.21.15.107:0/803070402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdda320 0x5599fbe4fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde69763640 1 -- 172.21.15.107:0/803070402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdcf330 msgr2=0x5599fbdd4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde69763640 1 --2- 172.21.15.107:0/803070402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdcf330 0x5599fbdd4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde69763640 1 -- 172.21.15.107:0/803070402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599fbe500a0 con 0x5599fbdcfc50 2024-05-09T08:41:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde69f64640 1 --2- 172.21.15.107:0/803070402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdcf330 0x5599fbdd4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde69763640 1 --2- 172.21.15.107:0/803070402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcfc50 0x5599fbdd4a50 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fde54005290 tx=0x7fde540053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ca99138ce7a161 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde4d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.308+0000 7fde4d7fa640 1 -- 172.21.15.107:0/803070402 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde54035030 con 0x5599fbdcfc50 2024-05-09T08:41:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.309+0000 7fde4d7fa640 1 -- 172.21.15.107:0/803070402 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fde540316a0 con 0x5599fbdcfc50 2024-05-09T08:41:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.309+0000 7fde6efb1740 1 -- 172.21.15.107:0/803070402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde54031e50 con 0x5599fbdcfc50 2024-05-09T08:41:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.309+0000 7fde4d7fa640 1 -- 172.21.15.107:0/803070402 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fde54066050 con 0x5599fbdcfc50 2024-05-09T08:41:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.309+0000 7fde6efb1740 10 client.17895 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.309+0000 7fde6efb1740 1 -- 172.21.15.107:0/803070402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5599fbe512d0 con 0x5599fbdcfc50 2024-05-09T08:41:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.309+0000 7fde4d7fa640 1 -- 172.21.15.107:0/803070402 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde54031e50 con 0x5599fbdcfc50 2024-05-09T08:41:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.309+0000 7fde4d7fa640 1 -- 172.21.15.107:0/803070402 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fde54004770 con 0x5599fbdcfc50 2024-05-09T08:41:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.309+0000 7fde6efb1740 10 client.17895 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.309+0000 7fde6efb1740 10 client.17895 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:41:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.309+0000 7fde6efb1740 20 client.17895 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.309+0000 7fde6efb1740 1 --2- 172.21.15.107:0/803070402 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5599fbe51550 0x5599fbe53940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.309+0000 7fde6efb1740 4 client.17895 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:41:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.309+0000 7fde6efb1740 1 -- 172.21.15.107:0/803070402 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5599fbd3c6c0 con 0x5599fbe51550 2024-05-09T08:41:01.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.309+0000 7fde69f64640 1 --2- 172.21.15.107:0/803070402 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5599fbe51550 0x5599fbe53940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde4d7fa640 10 client.17895 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:41:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde69f64640 1 --2- 172.21.15.107:0/803070402 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5599fbe51550 0x5599fbe53940 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=feb592f50ab3f2df server_cookie=d4b5f00b16706edb in_seq=0 out_seq=0 2024-05-09T08:41:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde4d7fa640 1 -- 172.21.15.107:0/803070402 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5599fbd3c6c0 con 0x5599fbe51550 2024-05-09T08:41:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde4d7fa640 10 client.17895 handle_command_reply: tid=0 2024-05-09T08:41:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde6efb1740 1 client.17895 shutdown 2024-05-09T08:41:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde6efb1740 1 -- 172.21.15.107:0/803070402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcfc50 msgr2=0x5599fbdd4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde6efb1740 1 --2- 172.21.15.107:0/803070402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcfc50 0x5599fbdd4a50 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fde54005290 tx=0x7fde540053f0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde6efb1740 1 -- 172.21.15.107:0/803070402 shutdown_connections 2024-05-09T08:41:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde6efb1740 1 --2- 172.21.15.107:0/803070402 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5599fbe51550 0x5599fbe53940 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde6efb1740 1 --2- 172.21.15.107:0/803070402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5599fbdda320 0x5599fbe4fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde6efb1740 1 --2- 172.21.15.107:0/803070402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5599fbdcfc50 0x5599fbdd4a50 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde6efb1740 1 --2- 172.21.15.107:0/803070402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599fbdcf330 0x5599fbdd4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde4d7fa640 0 client.17895 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:41:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde6efb1740 1 -- 172.21.15.107:0/803070402 >> 172.21.15.107:0/803070402 conn(0x5599fbdcb650 msgr2=0x5599fbdca7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde6efb1740 1 -- 172.21.15.107:0/803070402 shutdown_connections 2024-05-09T08:41:01.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde6efb1740 1 -- 172.21.15.107:0/803070402 wait complete. 2024-05-09T08:41:01.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.310+0000 7fde6efb1740 20 client.17895 trim_cache size 0 max 16384 2024-05-09T08:41:01.317 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:41:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.311+0000 7fde6c1c8640 1 -- 172.21.15.107:0/1422524867 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fde500a2d80 msgr2=0x7fde500a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.311+0000 7fde6c1c8640 1 --2- 172.21.15.107:0/1422524867 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fde500a2d80 0x7fde500a5240 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fde5c005c40 tx=0x7fde5c004ab0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.311+0000 7fde6c1c8640 1 -- 172.21.15.107:0/1422524867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde64100700 msgr2=0x7fde64198e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.311+0000 7fde6c1c8640 1 --2- 172.21.15.107:0/1422524867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde64100700 0x7fde64198e70 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fde6000d950 tx=0x7fde6000de10 comp rx=0 tx=0).stop 2024-05-09T08:41:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.311+0000 7fde6c1c8640 1 -- 172.21.15.107:0/1422524867 shutdown_connections 2024-05-09T08:41:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.311+0000 7fde6c1c8640 1 --2- 172.21.15.107:0/1422524867 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fde500a2d80 0x7fde500a5240 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.311+0000 7fde6c1c8640 1 --2- 172.21.15.107:0/1422524867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde64106710 0x7fde641a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.311+0000 7fde6c1c8640 1 --2- 172.21.15.107:0/1422524867 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde641010c0 0x7fde641993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.311+0000 7fde6c1c8640 1 --2- 172.21.15.107:0/1422524867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde64100700 0x7fde64198e70 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.311+0000 7fde6c1c8640 1 -- 172.21.15.107:0/1422524867 >> 172.21.15.107:0/1422524867 conn(0x7fde64077830 msgr2=0x7fde64103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.311+0000 7fde6c1c8640 1 -- 172.21.15.107:0/1422524867 shutdown_connections 2024-05-09T08:41:01.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.311+0000 7fde6c1c8640 1 -- 172.21.15.107:0/1422524867 wait complete. 2024-05-09T08:41:01.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:41:01.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.447+0000 7f32ef2da640 1 -- 172.21.15.107:0/371154158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32e8106710 msgr2=0x7f32e8106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.447+0000 7f32ef2da640 1 --2- 172.21.15.107:0/371154158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32e8106710 0x7f32e8106af0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f32e4009930 tx=0x7f32e402a420 comp rx=0 tx=0).stop 2024-05-09T08:41:01.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.447+0000 7f32ef2da640 1 -- 172.21.15.107:0/371154158 shutdown_connections 2024-05-09T08:41:01.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.447+0000 7f32ef2da640 1 --2- 172.21.15.107:0/371154158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32e81010c0 0x7f32e810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.447+0000 7f32ef2da640 1 --2- 172.21.15.107:0/371154158 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32e8100700 0x7f32e8100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.447+0000 7f32ef2da640 1 --2- 172.21.15.107:0/371154158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32e8106710 0x7f32e8106af0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.447+0000 7f32ef2da640 1 -- 172.21.15.107:0/371154158 >> 172.21.15.107:0/371154158 conn(0x7f32e8077830 msgr2=0x7f32e8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.447+0000 7f32ef2da640 1 -- 172.21.15.107:0/371154158 shutdown_connections 2024-05-09T08:41:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.447+0000 7f32ef2da640 1 -- 172.21.15.107:0/371154158 wait complete. 2024-05-09T08:41:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ef2da640 1 Processor -- start 2024-05-09T08:41:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ef2da640 1 -- start start 2024-05-09T08:41:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ef2da640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32e8100700 0x7f32e81980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ef2da640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32e81010c0 0x7f32e8198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ef2da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32e8106710 0x7f32e819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ef2da640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81109d0 con 0x7f32e8106710 2024-05-09T08:41:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ef2da640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f32e8110850 con 0x7f32e8100700 2024-05-09T08:41:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ef2da640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f32e8110b50 con 0x7f32e81010c0 2024-05-09T08:41:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ed877640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32e8106710 0x7f32e819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ed877640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32e8106710 0x7f32e819c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47610/0 (socket says 172.21.15.107:47610) 2024-05-09T08:41:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ed877640 1 -- 172.21.15.107:0/1205042757 learned_addr learned my addr 172.21.15.107:0/1205042757 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ec875640 1 --2- 172.21.15.107:0/1205042757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32e81010c0 0x7f32e8198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ed076640 1 --2- 172.21.15.107:0/1205042757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32e8100700 0x7f32e81980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ed877640 1 -- 172.21.15.107:0/1205042757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32e81010c0 msgr2=0x7f32e8198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ed877640 1 --2- 172.21.15.107:0/1205042757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32e81010c0 0x7f32e8198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ed877640 1 -- 172.21.15.107:0/1205042757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32e8100700 msgr2=0x7f32e81980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ed877640 1 --2- 172.21.15.107:0/1205042757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32e8100700 0x7f32e81980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.448+0000 7f32ed877640 1 -- 172.21.15.107:0/1205042757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e819d060 con 0x7f32e8106710 2024-05-09T08:41:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.449+0000 7f32ed877640 1 --2- 172.21.15.107:0/1205042757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32e8106710 0x7f32e819c930 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f32e000cce0 tx=0x7f32e000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.449+0000 7f32de7fc640 1 -- 172.21.15.107:0/1205042757 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e000f450 con 0x7f32e8106710 2024-05-09T08:41:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.449+0000 7f32ef2da640 1 -- 172.21.15.107:0/1205042757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e819d350 con 0x7f32e8106710 2024-05-09T08:41:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.449+0000 7f32de7fc640 1 -- 172.21.15.107:0/1205042757 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32e000faa0 con 0x7f32e8106710 2024-05-09T08:41:01.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.449+0000 7f32de7fc640 1 -- 172.21.15.107:0/1205042757 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e0018b70 con 0x7f32e8106710 2024-05-09T08:41:01.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.449+0000 7f32ef2da640 1 -- 172.21.15.107:0/1205042757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e81a4be0 con 0x7f32e8106710 2024-05-09T08:41:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.451+0000 7f32de7fc640 1 -- 172.21.15.107:0/1205042757 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f32e0018d10 con 0x7f32e8106710 2024-05-09T08:41:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.452+0000 7f32f1fed740 1 Processor -- start 2024-05-09T08:41:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.452+0000 7f32f1fed740 1 -- start start 2024-05-09T08:41:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.452+0000 7f32f1fed740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6ed210 0x557ebd6ed5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.452+0000 7f32f1fed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6edb30 0x557ebd6fc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.452+0000 7f32f1fed740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6feb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.452+0000 7f32f1fed740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557ebd652c40 con 0x557ebd6fc730 2024-05-09T08:41:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.452+0000 7f32f1fed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557ebd5fa580 con 0x557ebd6edb30 2024-05-09T08:41:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.452+0000 7f32f1fed740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x557ebd64ec00 con 0x557ebd6ed210 2024-05-09T08:41:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.452+0000 7f32ec875640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6feb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.452+0000 7f32ec875640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6feb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47624/0 (socket says 172.21.15.107:47624) 2024-05-09T08:41:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.452+0000 7f32ec875640 1 -- 172.21.15.107:0/465353058 learned_addr learned my addr 172.21.15.107:0/465353058 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.452+0000 7f32ec875640 1 --2- 172.21.15.107:0/465353058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6ed210 0x557ebd6ed5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.452+0000 7f32ed076640 1 --2- 172.21.15.107:0/465353058 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6edb30 0x557ebd6fc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.452+0000 7f32de7fc640 1 --2- 172.21.15.107:0/1205042757 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f32d40a2d00 0x7f32d40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.453+0000 7f32ed877640 1 --2- 172.21.15.107:0/1205042757 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f32d40a2d00 0x7f32d40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.453+0000 7f32ec875640 1 -- 172.21.15.107:0/465353058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6ed210 msgr2=0x557ebd6ed5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.453+0000 7f32ec875640 1 --2- 172.21.15.107:0/465353058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6ed210 0x557ebd6ed5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.453+0000 7f32ec875640 1 -- 172.21.15.107:0/465353058 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6edb30 msgr2=0x557ebd6fc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.453+0000 7f32ec875640 1 --2- 172.21.15.107:0/465353058 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6edb30 0x557ebd6fc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.453+0000 7f32ec875640 1 -- 172.21.15.107:0/465353058 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ebd3663f0 con 0x557ebd6fc730 2024-05-09T08:41:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.453+0000 7f32de7fc640 1 -- 172.21.15.107:0/1205042757 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f32e00cfbd0 con 0x7f32e8106710 2024-05-09T08:41:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.453+0000 7f32ec875640 1 --2- 172.21.15.107:0/465353058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6feb20 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f32d80127b0 tx=0x7f32d800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dfa1dce478ec099 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.453+0000 7f32d37fe640 1 -- 172.21.15.107:0/465353058 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8012df0 con 0x557ebd6fc730 2024-05-09T08:41:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.453+0000 7f32ed877640 1 --2- 172.21.15.107:0/1205042757 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f32d40a2d00 0x7f32d40a51c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f32e00cd570 tx=0x7f32e000af30 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.453+0000 7f32d37fe640 1 -- 172.21.15.107:0/465353058 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32d8005020 con 0x557ebd6fc730 2024-05-09T08:41:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.453+0000 7f32d37fe640 1 -- 172.21.15.107:0/465353058 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8004a00 con 0x557ebd6fc730 2024-05-09T08:41:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.454+0000 7f32f1fed740 1 -- 172.21.15.107:0/465353058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 msgr2=0x557ebd6feb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.454+0000 7f32f1fed740 1 --2- 172.21.15.107:0/465353058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6feb20 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f32d80127b0 tx=0x7f32d800b7c0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.454+0000 7f32f1fed740 1 -- 172.21.15.107:0/465353058 shutdown_connections 2024-05-09T08:41:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.454+0000 7f32f1fed740 1 --2- 172.21.15.107:0/465353058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6feb20 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.454+0000 7f32f1fed740 1 --2- 172.21.15.107:0/465353058 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6edb30 0x557ebd6fc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.454+0000 7f32f1fed740 1 --2- 172.21.15.107:0/465353058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6ed210 0x557ebd6ed5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.454+0000 7f32f1fed740 1 -- 172.21.15.107:0/465353058 >> 172.21.15.107:0/465353058 conn(0x557ebd6e9650 msgr2=0x557ebd6e9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.454+0000 7f32f1fed740 1 -- 172.21.15.107:0/465353058 shutdown_connections 2024-05-09T08:41:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.454+0000 7f32f1fed740 1 -- 172.21.15.107:0/465353058 wait complete. 2024-05-09T08:41:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.454+0000 7f32f1fed740 1 -- start start 2024-05-09T08:41:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.454+0000 7f32f1fed740 1 Processor -- start 2024-05-09T08:41:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32f1fed740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6ed210 0x557ebd6f9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32f1fed740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6edb30 0x557ebd6f9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32f1fed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6fc730 0x557ebd6fa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32f1fed740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557ebd6ff570 con 0x557ebd6ed210 2024-05-09T08:41:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32f1fed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557ebd60dc40 con 0x557ebd6fc730 2024-05-09T08:41:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32f1fed740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x557ebd6ff6f0 con 0x557ebd6edb30 2024-05-09T08:41:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32f1fed740 10 client.0 _ll_register_callbacks cb 0x557ebd365be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32ec875640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6ed210 0x557ebd6f9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32ec875640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6ed210 0x557ebd6f9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47636/0 (socket says 172.21.15.107:47636) 2024-05-09T08:41:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32ec875640 1 -- 172.21.15.107:0/1125466303 learned_addr learned my addr 172.21.15.107:0/1125466303 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32ed076640 1 --2- 172.21.15.107:0/1125466303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6edb30 0x557ebd6f9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32ec875640 1 --2- 172.21.15.107:0/1125466303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6fc730 0x557ebd6fa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32ec875640 1 -- 172.21.15.107:0/1125466303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6edb30 msgr2=0x557ebd6f9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32ec875640 1 --2- 172.21.15.107:0/1125466303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6edb30 0x557ebd6f9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32ec875640 1 -- 172.21.15.107:0/1125466303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6fc730 msgr2=0x557ebd6fa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32ec875640 1 --2- 172.21.15.107:0/1125466303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6fc730 0x557ebd6fa0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32ec875640 1 -- 172.21.15.107:0/1125466303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ebd4307d0 con 0x557ebd6ed210 2024-05-09T08:41:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32ec875640 1 --2- 172.21.15.107:0/1125466303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6ed210 0x557ebd6f9660 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f32d800bac0 tx=0x7f32d800b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a613f758396273c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.455+0000 7f32d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.456+0000 7f32d0ff9640 1 -- 172.21.15.107:0/1125466303 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8010970 con 0x557ebd6ed210 2024-05-09T08:41:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.456+0000 7f32d0ff9640 1 -- 172.21.15.107:0/1125466303 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f32d8010b10 con 0x557ebd6ed210 2024-05-09T08:41:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.456+0000 7f32f1fed740 1 -- 172.21.15.107:0/1125466303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32d8005020 con 0x557ebd6ed210 2024-05-09T08:41:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.456+0000 7f32d0ff9640 1 -- 172.21.15.107:0/1125466303 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f32d8034050 con 0x557ebd6ed210 2024-05-09T08:41:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.456+0000 7f32f1fed740 10 client.17907 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.456+0000 7f32f1fed740 1 -- 172.21.15.107:0/1125466303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557ebd70b7e0 con 0x557ebd6ed210 2024-05-09T08:41:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.456+0000 7f32d0ff9640 1 -- 172.21.15.107:0/1125466303 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32d8005020 con 0x557ebd6ed210 2024-05-09T08:41:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.456+0000 7f32d0ff9640 1 -- 172.21.15.107:0/1125466303 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f32d8029020 con 0x557ebd6ed210 2024-05-09T08:41:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.456+0000 7f32f1fed740 10 client.17907 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.456+0000 7f32f1fed740 10 client.17907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.456+0000 7f32f1fed740 20 client.17907 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.456+0000 7f32f1fed740 1 --2- 172.21.15.107:0/1125466303 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557ebd70cb30 0x557ebd70ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.456+0000 7f32f1fed740 4 client.17907 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.456+0000 7f32f1fed740 1 -- 172.21.15.107:0/1125466303 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ebd660370 con 0x557ebd70cb30 2024-05-09T08:41:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.457+0000 7f32ed076640 1 --2- 172.21.15.107:0/1125466303 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557ebd70cb30 0x557ebd70ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.457+0000 7f32d0ff9640 10 client.17907 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.457+0000 7f32ed076640 1 --2- 172.21.15.107:0/1125466303 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557ebd70cb30 0x557ebd70ef20 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a82e64585ac6d06 server_cookie=78a5b916ada009a6 in_seq=0 out_seq=0 2024-05-09T08:41:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.458+0000 7f32d0ff9640 1 -- 172.21.15.107:0/1125466303 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557ebd660370 con 0x557ebd70cb30 2024-05-09T08:41:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.458+0000 7f32d0ff9640 10 client.17907 handle_command_reply: tid=0 2024-05-09T08:41:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.458+0000 7f32f1fed740 1 client.17907 shutdown 2024-05-09T08:41:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.458+0000 7f32f1fed740 1 -- 172.21.15.107:0/1125466303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6ed210 msgr2=0x557ebd6f9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.458+0000 7f32f1fed740 1 --2- 172.21.15.107:0/1125466303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6ed210 0x557ebd6f9660 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f32d800bac0 tx=0x7f32d800b720 comp rx=0 tx=0).stop 2024-05-09T08:41:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.458+0000 7f32f1fed740 1 -- 172.21.15.107:0/1125466303 shutdown_connections 2024-05-09T08:41:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.458+0000 7f32f1fed740 1 --2- 172.21.15.107:0/1125466303 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557ebd70cb30 0x557ebd70ef20 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.458+0000 7f32f1fed740 1 --2- 172.21.15.107:0/1125466303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6fc730 0x557ebd6fa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.458+0000 7f32f1fed740 1 --2- 172.21.15.107:0/1125466303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6edb30 0x557ebd6f9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.458+0000 7f32d0ff9640 0 client.17907 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.458+0000 7f32f1fed740 1 --2- 172.21.15.107:0/1125466303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6ed210 0x557ebd6f9660 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.458+0000 7f32f1fed740 1 -- 172.21.15.107:0/1125466303 >> 172.21.15.107:0/1125466303 conn(0x557ebd6e9650 msgr2=0x557ebd6ef400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.458+0000 7f32f1fed740 1 -- 172.21.15.107:0/1125466303 shutdown_connections 2024-05-09T08:41:01.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.458+0000 7f32f1fed740 1 -- 172.21.15.107:0/1125466303 wait complete. 2024-05-09T08:41:01.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.458+0000 7f32f1fed740 20 client.17907 trim_cache size 0 max 16384 2024-05-09T08:41:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32f1fed740 1 Processor -- start 2024-05-09T08:41:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32f1fed740 1 -- start start 2024-05-09T08:41:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32f1fed740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6ed210 0x557ebd708500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32f1fed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6edb30 0x557ebd6e7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32f1fed740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6f8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32f1fed740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81af2e0 con 0x557ebd6fc730 2024-05-09T08:41:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32f1fed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81acbb0 con 0x557ebd6edb30 2024-05-09T08:41:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32f1fed740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81aaa20 con 0x557ebd6ed210 2024-05-09T08:41:01.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32ec875640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6f8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32ec875640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6f8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47648/0 (socket says 172.21.15.107:47648) 2024-05-09T08:41:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32ec875640 1 -- 172.21.15.107:0/4218379355 learned_addr learned my addr 172.21.15.107:0/4218379355 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32ed076640 1 --2- 172.21.15.107:0/4218379355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6edb30 0x557ebd6e7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32ec875640 1 --2- 172.21.15.107:0/4218379355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6ed210 0x557ebd708500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32ec875640 1 -- 172.21.15.107:0/4218379355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6ed210 msgr2=0x557ebd708500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32ec875640 1 --2- 172.21.15.107:0/4218379355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6ed210 0x557ebd708500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32ec875640 1 -- 172.21.15.107:0/4218379355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6edb30 msgr2=0x557ebd6e7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32ec875640 1 --2- 172.21.15.107:0/4218379355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6edb30 0x557ebd6e7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.473+0000 7f32ec875640 1 -- 172.21.15.107:0/4218379355 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ebd6dad20 con 0x557ebd6fc730 2024-05-09T08:41:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.474+0000 7f32ec875640 1 --2- 172.21.15.107:0/4218379355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6f8e50 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f32d80057c0 tx=0x7f32d8014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28130873d2d4384a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.474+0000 7f32d2ffd640 1 -- 172.21.15.107:0/4218379355 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d800b790 con 0x557ebd6fc730 2024-05-09T08:41:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.474+0000 7f32d2ffd640 1 -- 172.21.15.107:0/4218379355 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32d800bd40 con 0x557ebd6fc730 2024-05-09T08:41:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.474+0000 7f32d2ffd640 1 -- 172.21.15.107:0/4218379355 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8010ac0 con 0x557ebd6fc730 2024-05-09T08:41:01.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.475+0000 7f32f1fed740 1 -- 172.21.15.107:0/4218379355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 msgr2=0x557ebd6f8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.475+0000 7f32f1fed740 1 --2- 172.21.15.107:0/4218379355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6f8e50 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f32d80057c0 tx=0x7f32d8014040 comp rx=0 tx=0).stop 2024-05-09T08:41:01.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.475+0000 7f32f1fed740 1 -- 172.21.15.107:0/4218379355 shutdown_connections 2024-05-09T08:41:01.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.475+0000 7f32f1fed740 1 --2- 172.21.15.107:0/4218379355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6f8e50 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.475+0000 7f32f1fed740 1 --2- 172.21.15.107:0/4218379355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6edb30 0x557ebd6e7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.475+0000 7f32f1fed740 1 --2- 172.21.15.107:0/4218379355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6ed210 0x557ebd708500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.475+0000 7f32f1fed740 1 -- 172.21.15.107:0/4218379355 >> 172.21.15.107:0/4218379355 conn(0x557ebd6e9650 msgr2=0x557ebd6f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.475+0000 7f32f1fed740 1 -- 172.21.15.107:0/4218379355 shutdown_connections 2024-05-09T08:41:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.475+0000 7f32f1fed740 1 -- 172.21.15.107:0/4218379355 wait complete. 2024-05-09T08:41:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.475+0000 7f32f1fed740 1 -- start start 2024-05-09T08:41:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.475+0000 7f32f1fed740 1 Processor -- start 2024-05-09T08:41:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.475+0000 7f32f1fed740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6ed210 0x557ebd704c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.475+0000 7f32f1fed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6edb30 0x557ebd6f5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.475+0000 7f32f1fed740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6f6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32f1fed740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81ab070 con 0x557ebd6fc730 2024-05-09T08:41:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32f1fed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81ac4e0 con 0x557ebd6edb30 2024-05-09T08:41:01.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32f1fed740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81ab740 con 0x557ebd6ed210 2024-05-09T08:41:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32f1fed740 10 client.0 _ll_register_callbacks cb 0x7f32e4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32ec875640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6f6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32ec875640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6f6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47650/0 (socket says 172.21.15.107:47650) 2024-05-09T08:41:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32ec875640 1 -- 172.21.15.107:0/3502135395 learned_addr learned my addr 172.21.15.107:0/3502135395 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32ed076640 1 --2- 172.21.15.107:0/3502135395 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6edb30 0x557ebd6f5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32ec875640 1 --2- 172.21.15.107:0/3502135395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6ed210 0x557ebd704c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32ec875640 1 -- 172.21.15.107:0/3502135395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6ed210 msgr2=0x557ebd704c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32ec875640 1 --2- 172.21.15.107:0/3502135395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6ed210 0x557ebd704c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32ec875640 1 -- 172.21.15.107:0/3502135395 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6edb30 msgr2=0x557ebd6f5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32ec875640 1 --2- 172.21.15.107:0/3502135395 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6edb30 0x557ebd6f5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32ec875640 1 -- 172.21.15.107:0/3502135395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ebd6f6990 con 0x557ebd6fc730 2024-05-09T08:41:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32ed076640 1 --2- 172.21.15.107:0/3502135395 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6edb30 0x557ebd6f5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32ec875640 1 --2- 172.21.15.107:0/3502135395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6f6450 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f32d801f760 tx=0x7f32d8021040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd931f3b15854959 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:01.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32d0ff9640 1 -- 172.21.15.107:0/3502135395 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d800b390 con 0x557ebd6fc730 2024-05-09T08:41:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32f1fed740 1 -- 172.21.15.107:0/3502135395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32d800bd40 con 0x557ebd6fc730 2024-05-09T08:41:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32d0ff9640 1 -- 172.21.15.107:0/3502135395 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f32d801f9a0 con 0x557ebd6fc730 2024-05-09T08:41:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32d0ff9640 1 -- 172.21.15.107:0/3502135395 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f32d801fba0 con 0x557ebd6fc730 2024-05-09T08:41:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32f1fed740 10 client.17913 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.476+0000 7f32f1fed740 1 -- 172.21.15.107:0/3502135395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557ebd772c50 con 0x557ebd6fc730 2024-05-09T08:41:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.477+0000 7f32d0ff9640 1 -- 172.21.15.107:0/3502135395 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32d800bd40 con 0x557ebd6fc730 2024-05-09T08:41:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.477+0000 7f32d0ff9640 1 -- 172.21.15.107:0/3502135395 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f32d8029070 con 0x557ebd6fc730 2024-05-09T08:41:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.477+0000 7f32f1fed740 10 client.17913 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.477+0000 7f32f1fed740 10 client.17913 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.477+0000 7f32f1fed740 20 client.17913 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.477+0000 7f32f1fed740 1 --2- 172.21.15.107:0/3502135395 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557ebd773260 0x557ebd775680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.477+0000 7f32f1fed740 4 client.17913 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:41:01.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.477+0000 7f32f1fed740 1 -- 172.21.15.107:0/3502135395 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557ebd65a6c0 con 0x557ebd773260 2024-05-09T08:41:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.477+0000 7f32ed076640 1 --2- 172.21.15.107:0/3502135395 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557ebd773260 0x557ebd775680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32d0ff9640 10 client.17913 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32ed076640 1 --2- 172.21.15.107:0/3502135395 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557ebd773260 0x557ebd775680 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=681f09687acae4c1 server_cookie=193af34a7589ce91 in_seq=0 out_seq=0 2024-05-09T08:41:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32d0ff9640 1 -- 172.21.15.107:0/3502135395 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557ebd65a6c0 con 0x557ebd773260 2024-05-09T08:41:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32d0ff9640 10 client.17913 handle_command_reply: tid=0 2024-05-09T08:41:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32f1fed740 1 client.17913 shutdown 2024-05-09T08:41:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32f1fed740 1 -- 172.21.15.107:0/3502135395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 msgr2=0x557ebd6f6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32f1fed740 1 --2- 172.21.15.107:0/3502135395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6f6450 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f32d801f760 tx=0x7f32d8021040 comp rx=0 tx=0).stop 2024-05-09T08:41:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32f1fed740 1 -- 172.21.15.107:0/3502135395 shutdown_connections 2024-05-09T08:41:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32f1fed740 1 --2- 172.21.15.107:0/3502135395 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557ebd773260 0x557ebd775680 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32f1fed740 1 --2- 172.21.15.107:0/3502135395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557ebd6fc730 0x557ebd6f6450 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32f1fed740 1 --2- 172.21.15.107:0/3502135395 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557ebd6edb30 0x557ebd6f5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32f1fed740 1 --2- 172.21.15.107:0/3502135395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557ebd6ed210 0x557ebd704c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32f1fed740 1 -- 172.21.15.107:0/3502135395 >> 172.21.15.107:0/3502135395 conn(0x557ebd6e9650 msgr2=0x557ebd6eeab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32d0ff9640 0 client.17913 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:01.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32f1fed740 1 -- 172.21.15.107:0/3502135395 shutdown_connections 2024-05-09T08:41:01.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32f1fed740 1 -- 172.21.15.107:0/3502135395 wait complete. 2024-05-09T08:41:01.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.478+0000 7f32f1fed740 20 client.17913 trim_cache size 0 max 16384 2024-05-09T08:41:01.485 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:41:01.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.479+0000 7f32ef2da640 1 -- 172.21.15.107:0/1205042757 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f32d40a2d00 msgr2=0x7f32d40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.479+0000 7f32ef2da640 1 --2- 172.21.15.107:0/1205042757 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f32d40a2d00 0x7f32d40a51c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f32e00cd570 tx=0x7f32e000af30 comp rx=0 tx=0).stop 2024-05-09T08:41:01.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.479+0000 7f32ef2da640 1 -- 172.21.15.107:0/1205042757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32e8106710 msgr2=0x7f32e819c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:01.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.479+0000 7f32ef2da640 1 --2- 172.21.15.107:0/1205042757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32e8106710 0x7f32e819c930 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f32e000cce0 tx=0x7f32e000a620 comp rx=0 tx=0).stop 2024-05-09T08:41:01.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.479+0000 7f32ef2da640 1 -- 172.21.15.107:0/1205042757 shutdown_connections 2024-05-09T08:41:01.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.479+0000 7f32ef2da640 1 --2- 172.21.15.107:0/1205042757 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f32d40a2d00 0x7f32d40a51c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.479+0000 7f32ef2da640 1 --2- 172.21.15.107:0/1205042757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32e8106710 0x7f32e819c930 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.479+0000 7f32ef2da640 1 --2- 172.21.15.107:0/1205042757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32e81010c0 0x7f32e8198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.479+0000 7f32ef2da640 1 --2- 172.21.15.107:0/1205042757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32e8100700 0x7f32e81980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.479+0000 7f32ef2da640 1 -- 172.21.15.107:0/1205042757 >> 172.21.15.107:0/1205042757 conn(0x7f32e8077830 msgr2=0x7f32e8068e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.479+0000 7f32ef2da640 1 -- 172.21.15.107:0/1205042757 shutdown_connections 2024-05-09T08:41:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:01.479+0000 7f32ef2da640 1 -- 172.21.15.107:0/1205042757 wait complete. 2024-05-09T08:41:01.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:01 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/99070469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:41:01.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:01 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/99070469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:41:01.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:01 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/99070469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:41:02.491 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:41:02.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:41:02.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:41:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.617+0000 7f9bfbfc1640 1 -- 172.21.15.107:0/469246406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bf4106710 msgr2=0x7f9bf4106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.617+0000 7f9bfbfc1640 1 --2- 172.21.15.107:0/469246406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bf4106710 0x7f9bf4106af0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009930 tx=0x7f9bf002a420 comp rx=0 tx=0).stop 2024-05-09T08:41:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.617+0000 7f9bfbfc1640 1 -- 172.21.15.107:0/469246406 shutdown_connections 2024-05-09T08:41:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.617+0000 7f9bfbfc1640 1 --2- 172.21.15.107:0/469246406 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bf41010c0 0x7f9bf410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.617+0000 7f9bfbfc1640 1 --2- 172.21.15.107:0/469246406 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bf4100700 0x7f9bf4100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.617+0000 7f9bfbfc1640 1 --2- 172.21.15.107:0/469246406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bf4106710 0x7f9bf4106af0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.617+0000 7f9bfbfc1640 1 -- 172.21.15.107:0/469246406 >> 172.21.15.107:0/469246406 conn(0x7f9bf4077830 msgr2=0x7f9bf4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.617+0000 7f9bfbfc1640 1 -- 172.21.15.107:0/469246406 shutdown_connections 2024-05-09T08:41:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.617+0000 7f9bfbfc1640 1 -- 172.21.15.107:0/469246406 wait complete. 2024-05-09T08:41:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.617+0000 7f9bfbfc1640 1 Processor -- start 2024-05-09T08:41:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.617+0000 7f9bfbfc1640 1 -- start start 2024-05-09T08:41:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bfbfc1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bf4100700 0x7f9bf4198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bfbfc1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bf41010c0 0x7f9bf4199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bfbfc1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bf4106710 0x7f9bf41a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bfbfc1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf4110990 con 0x7f9bf4106710 2024-05-09T08:41:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bfbfc1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf4110810 con 0x7f9bf41010c0 2024-05-09T08:41:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bfbfc1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf4110b10 con 0x7f9bf4100700 2024-05-09T08:41:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bfa55e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bf4106710 0x7f9bf41a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bfa55e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bf4106710 0x7f9bf41a3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47658/0 (socket says 172.21.15.107:47658) 2024-05-09T08:41:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bfa55e640 1 -- 172.21.15.107:0/4147795582 learned_addr learned my addr 172.21.15.107:0/4147795582 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bf9d5d640 1 --2- 172.21.15.107:0/4147795582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bf4100700 0x7f9bf4198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bf955c640 1 --2- 172.21.15.107:0/4147795582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bf41010c0 0x7f9bf4199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bfa55e640 1 -- 172.21.15.107:0/4147795582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bf4100700 msgr2=0x7f9bf4198e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bfa55e640 1 --2- 172.21.15.107:0/4147795582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bf4100700 0x7f9bf4198e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bfa55e640 1 -- 172.21.15.107:0/4147795582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bf41010c0 msgr2=0x7f9bf4199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bfa55e640 1 --2- 172.21.15.107:0/4147795582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bf41010c0 0x7f9bf4199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bfa55e640 1 -- 172.21.15.107:0/4147795582 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf41a43f0 con 0x7f9bf4106710 2024-05-09T08:41:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.618+0000 7f9bfa55e640 1 --2- 172.21.15.107:0/4147795582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bf4106710 0x7f9bf41a3eb0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00cce0 tx=0x7f9bec00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.619+0000 7f9beaffd640 1 -- 172.21.15.107:0/4147795582 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bec00f450 con 0x7f9bf4106710 2024-05-09T08:41:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.619+0000 7f9beaffd640 1 -- 172.21.15.107:0/4147795582 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bec00faa0 con 0x7f9bf4106710 2024-05-09T08:41:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.619+0000 7f9bfbfc1640 1 -- 172.21.15.107:0/4147795582 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bf41a45c0 con 0x7f9bf4106710 2024-05-09T08:41:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.619+0000 7f9beaffd640 1 -- 172.21.15.107:0/4147795582 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bec018b70 con 0x7f9bf4106710 2024-05-09T08:41:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.619+0000 7f9bfbfc1640 1 -- 172.21.15.107:0/4147795582 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bf41a4a90 con 0x7f9bf4106710 2024-05-09T08:41:02.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.621+0000 7f9bfedab740 1 Processor -- start 2024-05-09T08:41:02.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.621+0000 7f9beaffd640 1 -- 172.21.15.107:0/4147795582 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f9bec018d10 con 0x7f9bf4106710 2024-05-09T08:41:02.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.621+0000 7f9bfedab740 1 -- start start 2024-05-09T08:41:02.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bfedab740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d3550b0 0x55597d3533f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bfedab740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d3539c0 0x55597d353e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bfedab740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d35e740 0x55597d360b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bfedab740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55597d2b8990 con 0x55597d35e740 2024-05-09T08:41:02.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bfedab740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55597d2605b0 con 0x55597d3550b0 2024-05-09T08:41:02.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bfedab740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55597d2b4950 con 0x55597d3539c0 2024-05-09T08:41:02.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bf955c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d35e740 0x55597d360b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bf955c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d35e740 0x55597d360b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47670/0 (socket says 172.21.15.107:47670) 2024-05-09T08:41:02.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bf955c640 1 -- 172.21.15.107:0/2248448622 learned_addr learned my addr 172.21.15.107:0/2248448622 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:02.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bf955c640 1 --2- 172.21.15.107:0/2248448622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d3550b0 0x55597d3533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bf9d5d640 1 --2- 172.21.15.107:0/2248448622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d3539c0 0x55597d353e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9beaffd640 1 --2- 172.21.15.107:0/4147795582 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9be00a2d00 0x7f9be00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bf955c640 1 -- 172.21.15.107:0/2248448622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d3539c0 msgr2=0x55597d353e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bf955c640 1 --2- 172.21.15.107:0/2248448622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d3539c0 0x55597d353e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bf955c640 1 -- 172.21.15.107:0/2248448622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d3550b0 msgr2=0x55597d3533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bf955c640 1 --2- 172.21.15.107:0/2248448622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d3550b0 0x55597d3533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bf955c640 1 -- 172.21.15.107:0/2248448622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55597d0947e0 con 0x55597d35e740 2024-05-09T08:41:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.622+0000 7f9bfa55e640 1 --2- 172.21.15.107:0/4147795582 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9be00a2d00 0x7f9be00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.623+0000 7f9bf955c640 1 --2- 172.21.15.107:0/2248448622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d35e740 0x55597d360b30 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f9be40149c0 tx=0x7f9be400c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc8bfce3a2e39b18 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.623+0000 7f9beaffd640 1 -- 172.21.15.107:0/4147795582 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f9bec0cfc10 con 0x7f9bf4106710 2024-05-09T08:41:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.623+0000 7f9bdffff640 1 -- 172.21.15.107:0/2248448622 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be40142d0 con 0x55597d35e740 2024-05-09T08:41:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.623+0000 7f9bdffff640 1 -- 172.21.15.107:0/2248448622 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9be400fc10 con 0x55597d35e740 2024-05-09T08:41:02.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.623+0000 7f9bdffff640 1 -- 172.21.15.107:0/2248448622 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be4012af0 con 0x55597d35e740 2024-05-09T08:41:02.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.624+0000 7f9bfedab740 1 -- 172.21.15.107:0/2248448622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d35e740 msgr2=0x55597d360b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.624+0000 7f9bfa55e640 1 --2- 172.21.15.107:0/4147795582 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9be00a2d00 0x7f9be00a51c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9bec09ba20 tx=0x7f9bec06dce0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.624+0000 7f9bfedab740 1 --2- 172.21.15.107:0/2248448622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d35e740 0x55597d360b30 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f9be40149c0 tx=0x7f9be400c4d0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.624+0000 7f9bfedab740 1 -- 172.21.15.107:0/2248448622 shutdown_connections 2024-05-09T08:41:02.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.624+0000 7f9bfedab740 1 --2- 172.21.15.107:0/2248448622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d35e740 0x55597d360b30 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.624+0000 7f9bfedab740 1 --2- 172.21.15.107:0/2248448622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d3539c0 0x55597d353e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.624+0000 7f9bfedab740 1 --2- 172.21.15.107:0/2248448622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d3550b0 0x55597d3533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.624+0000 7f9bfedab740 1 -- 172.21.15.107:0/2248448622 >> 172.21.15.107:0/2248448622 conn(0x55597d31c0a0 msgr2=0x55597d34ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.624+0000 7f9bfedab740 1 -- 172.21.15.107:0/2248448622 shutdown_connections 2024-05-09T08:41:02.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.624+0000 7f9bfedab740 1 -- 172.21.15.107:0/2248448622 wait complete. 2024-05-09T08:41:02.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.624+0000 7f9bfedab740 1 -- start start 2024-05-09T08:41:02.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.624+0000 7f9bfedab740 1 Processor -- start 2024-05-09T08:41:02.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bfedab740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3539c0 0x55597d35b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bfedab740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d3550b0 0x55597d35bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bfedab740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d35e740 0x55597d35c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bfedab740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55597d3615d0 con 0x55597d3539c0 2024-05-09T08:41:02.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bfedab740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55597d273b40 con 0x55597d3550b0 2024-05-09T08:41:02.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bfedab740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55597d361750 con 0x55597d35e740 2024-05-09T08:41:02.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bfedab740 10 client.0 _ll_register_callbacks cb 0x55597d2680f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:02.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bf955c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3539c0 0x55597d35b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bf955c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3539c0 0x55597d35b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47678/0 (socket says 172.21.15.107:47678) 2024-05-09T08:41:02.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bf955c640 1 -- 172.21.15.107:0/2429252063 learned_addr learned my addr 172.21.15.107:0/2429252063 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:02.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bf9d5d640 1 --2- 172.21.15.107:0/2429252063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d3550b0 0x55597d35bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bf955c640 1 -- 172.21.15.107:0/2429252063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d35e740 msgr2=0x55597d35c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bf955c640 1 --2- 172.21.15.107:0/2429252063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d35e740 0x55597d35c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bf955c640 1 -- 172.21.15.107:0/2429252063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d3550b0 msgr2=0x55597d35bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bf955c640 1 --2- 172.21.15.107:0/2429252063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d3550b0 0x55597d35bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bf955c640 1 -- 172.21.15.107:0/2429252063 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55597d0f4f70 con 0x55597d3539c0 2024-05-09T08:41:02.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bf955c640 1 --2- 172.21.15.107:0/2429252063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3539c0 0x55597d35b6f0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f9be4005d00 tx=0x7f9be400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=318192f71da9967a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.625+0000 7f9bdd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:02.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.626+0000 7f9bdd7fa640 1 -- 172.21.15.107:0/2429252063 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be40140d0 con 0x55597d3539c0 2024-05-09T08:41:02.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.626+0000 7f9bdd7fa640 1 -- 172.21.15.107:0/2429252063 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f9be4014270 con 0x55597d3539c0 2024-05-09T08:41:02.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.626+0000 7f9bfedab740 1 -- 172.21.15.107:0/2429252063 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9be400fc10 con 0x55597d3539c0 2024-05-09T08:41:02.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.626+0000 7f9bdd7fa640 1 -- 172.21.15.107:0/2429252063 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9be4031050 con 0x55597d3539c0 2024-05-09T08:41:02.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.626+0000 7f9bfedab740 10 client.17925 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:02.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.626+0000 7f9bfedab740 1 -- 172.21.15.107:0/2429252063 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55597cf8c170 con 0x55597d3539c0 2024-05-09T08:41:02.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.626+0000 7f9bdd7fa640 1 -- 172.21.15.107:0/2429252063 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9be400fc10 con 0x55597d3539c0 2024-05-09T08:41:02.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.626+0000 7f9bdd7fa640 1 -- 172.21.15.107:0/2429252063 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9be4012aa0 con 0x55597d3539c0 2024-05-09T08:41:02.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.626+0000 7f9bfedab740 10 client.17925 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:02.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.626+0000 7f9bfedab740 10 client.17925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:02.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.626+0000 7f9bfedab740 20 client.17925 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:02.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.626+0000 7f9bfedab740 1 --2- 172.21.15.107:0/2429252063 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55597d36e760 0x55597d370b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.626+0000 7f9bfedab740 4 client.17925 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:02.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.626+0000 7f9bfedab740 1 -- 172.21.15.107:0/2429252063 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55597d2c60c0 con 0x55597d36e760 2024-05-09T08:41:02.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.627+0000 7f9bf9d5d640 1 --2- 172.21.15.107:0/2429252063 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55597d36e760 0x55597d370b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.627+0000 7f9bdd7fa640 10 client.17925 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:02.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.627+0000 7f9bf9d5d640 1 --2- 172.21.15.107:0/2429252063 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55597d36e760 0x55597d370b50 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44758a906fe34c17 server_cookie=dc1c095e78e98315 in_seq=0 out_seq=0 2024-05-09T08:41:02.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.628+0000 7f9bdd7fa640 1 -- 172.21.15.107:0/2429252063 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55597d2c60c0 con 0x55597d36e760 2024-05-09T08:41:02.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.628+0000 7f9bdd7fa640 10 client.17925 handle_command_reply: tid=0 2024-05-09T08:41:02.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.628+0000 7f9bfedab740 1 client.17925 shutdown 2024-05-09T08:41:02.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.628+0000 7f9bfedab740 1 -- 172.21.15.107:0/2429252063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3539c0 msgr2=0x55597d35b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.628+0000 7f9bfedab740 1 --2- 172.21.15.107:0/2429252063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3539c0 0x55597d35b6f0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f9be4005d00 tx=0x7f9be400ce30 comp rx=0 tx=0).stop 2024-05-09T08:41:02.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.628+0000 7f9bfedab740 1 -- 172.21.15.107:0/2429252063 shutdown_connections 2024-05-09T08:41:02.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.628+0000 7f9bfedab740 1 --2- 172.21.15.107:0/2429252063 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55597d36e760 0x55597d370b50 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.628+0000 7f9bfedab740 1 --2- 172.21.15.107:0/2429252063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d35e740 0x55597d35c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.628+0000 7f9bdd7fa640 0 client.17925 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:02.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.628+0000 7f9bfedab740 1 --2- 172.21.15.107:0/2429252063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d3550b0 0x55597d35bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.628+0000 7f9bfedab740 1 --2- 172.21.15.107:0/2429252063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3539c0 0x55597d35b6f0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.628+0000 7f9bfedab740 1 -- 172.21.15.107:0/2429252063 >> 172.21.15.107:0/2429252063 conn(0x55597d31c0a0 msgr2=0x55597d355d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.628+0000 7f9bfedab740 1 -- 172.21.15.107:0/2429252063 shutdown_connections 2024-05-09T08:41:02.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.628+0000 7f9bfedab740 1 -- 172.21.15.107:0/2429252063 wait complete. 2024-05-09T08:41:02.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.629+0000 7f9bfedab740 20 client.17925 trim_cache size 0 max 16384 2024-05-09T08:41:02.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.652+0000 7f9bfedab740 1 Processor -- start 2024-05-09T08:41:02.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.652+0000 7f9bfedab740 1 -- start start 2024-05-09T08:41:02.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.652+0000 7f9bfedab740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d3539c0 0x55597d36a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.652+0000 7f9bfedab740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3550b0 0x55597d35a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.652+0000 7f9bfedab740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d35e740 0x55597d34ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.652+0000 7f9bfedab740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf41b15b0 con 0x55597d3550b0 2024-05-09T08:41:02.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.652+0000 7f9bfedab740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf41aee80 con 0x55597d35e740 2024-05-09T08:41:02.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.652+0000 7f9bfedab740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf41accf0 con 0x55597d3539c0 2024-05-09T08:41:02.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.652+0000 7f9bf9d5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3550b0 0x55597d35a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.652+0000 7f9bf9d5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3550b0 0x55597d35a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47694/0 (socket says 172.21.15.107:47694) 2024-05-09T08:41:02.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.652+0000 7f9bf9d5d640 1 -- 172.21.15.107:0/3198136139 learned_addr learned my addr 172.21.15.107:0/3198136139 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:02.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.652+0000 7f9bf955c640 1 --2- 172.21.15.107:0/3198136139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d35e740 0x55597d34ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.652+0000 7f9bf955c640 1 --2- 172.21.15.107:0/3198136139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d3539c0 0x55597d36a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.653+0000 7f9bf9d5d640 1 -- 172.21.15.107:0/3198136139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d3539c0 msgr2=0x55597d36a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.653+0000 7f9bf9d5d640 1 --2- 172.21.15.107:0/3198136139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d3539c0 0x55597d36a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.653+0000 7f9bf9d5d640 1 -- 172.21.15.107:0/3198136139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d35e740 msgr2=0x55597d34ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.653+0000 7f9bf9d5d640 1 --2- 172.21.15.107:0/3198136139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d35e740 0x55597d34ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.653+0000 7f9bf9d5d640 1 -- 172.21.15.107:0/3198136139 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55597d34e470 con 0x55597d3550b0 2024-05-09T08:41:02.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.653+0000 7f9bf9d5d640 1 --2- 172.21.15.107:0/3198136139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3550b0 0x55597d35a850 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f9bf002a940 tx=0x7f9bf0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84164449938c314 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.653+0000 7f9bdf7fe640 1 -- 172.21.15.107:0/3198136139 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf004a020 con 0x55597d3550b0 2024-05-09T08:41:02.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.653+0000 7f9bdf7fe640 1 -- 172.21.15.107:0/3198136139 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bf0031950 con 0x55597d3550b0 2024-05-09T08:41:02.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.653+0000 7f9bdf7fe640 1 -- 172.21.15.107:0/3198136139 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0035040 con 0x55597d3550b0 2024-05-09T08:41:02.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.654+0000 7f9bfedab740 1 -- 172.21.15.107:0/3198136139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3550b0 msgr2=0x55597d35a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.654+0000 7f9bfedab740 1 --2- 172.21.15.107:0/3198136139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3550b0 0x55597d35a850 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f9bf002a940 tx=0x7f9bf0000bc0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.654+0000 7f9bfedab740 1 -- 172.21.15.107:0/3198136139 shutdown_connections 2024-05-09T08:41:02.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.654+0000 7f9bfedab740 1 --2- 172.21.15.107:0/3198136139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d35e740 0x55597d34ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.654+0000 7f9bfedab740 1 --2- 172.21.15.107:0/3198136139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3550b0 0x55597d35a850 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.654+0000 7f9bfedab740 1 --2- 172.21.15.107:0/3198136139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d3539c0 0x55597d36a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.654+0000 7f9bfedab740 1 -- 172.21.15.107:0/3198136139 >> 172.21.15.107:0/3198136139 conn(0x55597d31c0a0 msgr2=0x55597d34e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.654+0000 7f9bfedab740 1 -- 172.21.15.107:0/3198136139 shutdown_connections 2024-05-09T08:41:02.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.654+0000 7f9bfedab740 1 -- 172.21.15.107:0/3198136139 wait complete. 2024-05-09T08:41:02.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.654+0000 7f9bfedab740 1 -- start start 2024-05-09T08:41:02.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.654+0000 7f9bfedab740 1 Processor -- start 2024-05-09T08:41:02.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bfedab740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3539c0 0x55597d362770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bfedab740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d3550b0 0x55597d362cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bfedab740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d35e740 0x55597d3d3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bfedab740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf41ad340 con 0x55597d3539c0 2024-05-09T08:41:02.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bfedab740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf41ae7b0 con 0x55597d3550b0 2024-05-09T08:41:02.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bfedab740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf41ada10 con 0x55597d35e740 2024-05-09T08:41:02.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bfedab740 10 client.0 _ll_register_callbacks cb 0x7f9be400cf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:02.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bf955c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3539c0 0x55597d362770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bf955c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3539c0 0x55597d362770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47700/0 (socket says 172.21.15.107:47700) 2024-05-09T08:41:02.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bf955c640 1 -- 172.21.15.107:0/3099758032 learned_addr learned my addr 172.21.15.107:0/3099758032 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:02.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bf9d5d640 1 --2- 172.21.15.107:0/3099758032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d3550b0 0x55597d362cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bf955c640 1 --2- 172.21.15.107:0/3099758032 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d35e740 0x55597d3d3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bf955c640 1 -- 172.21.15.107:0/3099758032 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d35e740 msgr2=0x55597d3d3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bf955c640 1 --2- 172.21.15.107:0/3099758032 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d35e740 0x55597d3d3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bf955c640 1 -- 172.21.15.107:0/3099758032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d3550b0 msgr2=0x55597d362cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bf955c640 1 --2- 172.21.15.107:0/3099758032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d3550b0 0x55597d362cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bf955c640 1 -- 172.21.15.107:0/3099758032 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55597d363580 con 0x55597d3539c0 2024-05-09T08:41:02.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bf9d5d640 1 --2- 172.21.15.107:0/3099758032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d3550b0 0x55597d362cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:02.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bf955c640 1 --2- 172.21.15.107:0/3099758032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3539c0 0x55597d362770 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f9be4001eb0 tx=0x7f9be4005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ee1be7c71d862c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.655+0000 7f9bdd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:02.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.656+0000 7f9bdd7fa640 1 -- 172.21.15.107:0/3099758032 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be40140c0 con 0x55597d3539c0 2024-05-09T08:41:02.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.656+0000 7f9bdd7fa640 1 -- 172.21.15.107:0/3099758032 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f9be4014260 con 0x55597d3539c0 2024-05-09T08:41:02.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.656+0000 7f9bfedab740 1 -- 172.21.15.107:0/3099758032 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bf0031950 con 0x55597d3539c0 2024-05-09T08:41:02.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.656+0000 7f9bdd7fa640 1 -- 172.21.15.107:0/3099758032 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9be4014460 con 0x55597d3539c0 2024-05-09T08:41:02.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.656+0000 7f9bfedab740 10 client.17931 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:02.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.656+0000 7f9bfedab740 1 -- 172.21.15.107:0/3099758032 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55597d3d5090 con 0x55597d3539c0 2024-05-09T08:41:02.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.656+0000 7f9bdd7fa640 1 -- 172.21.15.107:0/3099758032 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9be403d070 con 0x55597d3539c0 2024-05-09T08:41:02.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.656+0000 7f9bdd7fa640 1 -- 172.21.15.107:0/3099758032 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9be40053c0 con 0x55597d3539c0 2024-05-09T08:41:02.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.656+0000 7f9bfedab740 10 client.17931 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:02.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.656+0000 7f9bfedab740 10 client.17931 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:02.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.656+0000 7f9bfedab740 20 client.17931 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.656+0000 7f9bfedab740 1 --2- 172.21.15.107:0/3099758032 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55597d36e760 0x55597d3d7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.656+0000 7f9bfedab740 4 client.17931 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:41:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.656+0000 7f9bfedab740 1 -- 172.21.15.107:0/3099758032 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55597d2c0410 con 0x55597d36e760 2024-05-09T08:41:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.657+0000 7f9bf9d5d640 1 --2- 172.21.15.107:0/3099758032 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55597d36e760 0x55597d3d7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.657+0000 7f9bdd7fa640 10 client.17931 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.657+0000 7f9bf9d5d640 1 --2- 172.21.15.107:0/3099758032 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55597d36e760 0x55597d3d7730 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff7b8de89359188d server_cookie=42c84fbfced766 in_seq=0 out_seq=0 2024-05-09T08:41:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.658+0000 7f9bdd7fa640 1 -- 172.21.15.107:0/3099758032 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55597d2c0410 con 0x55597d36e760 2024-05-09T08:41:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.658+0000 7f9bdd7fa640 10 client.17931 handle_command_reply: tid=0 2024-05-09T08:41:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.658+0000 7f9bfedab740 1 client.17931 shutdown 2024-05-09T08:41:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.658+0000 7f9bfedab740 1 -- 172.21.15.107:0/3099758032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3539c0 msgr2=0x55597d362770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.658+0000 7f9bfedab740 1 --2- 172.21.15.107:0/3099758032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3539c0 0x55597d362770 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f9be4001eb0 tx=0x7f9be4005f30 comp rx=0 tx=0).stop 2024-05-09T08:41:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.658+0000 7f9bfedab740 1 -- 172.21.15.107:0/3099758032 shutdown_connections 2024-05-09T08:41:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.658+0000 7f9bfedab740 1 --2- 172.21.15.107:0/3099758032 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55597d36e760 0x55597d3d7730 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.658+0000 7f9bfedab740 1 --2- 172.21.15.107:0/3099758032 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55597d35e740 0x55597d3d3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.658+0000 7f9bdd7fa640 0 client.17931 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:02.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.658+0000 7f9bfedab740 1 --2- 172.21.15.107:0/3099758032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55597d3550b0 0x55597d362cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.658+0000 7f9bfedab740 1 --2- 172.21.15.107:0/3099758032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55597d3539c0 0x55597d362770 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.658+0000 7f9bfedab740 1 -- 172.21.15.107:0/3099758032 >> 172.21.15.107:0/3099758032 conn(0x55597d31c0a0 msgr2=0x55597d34e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.659+0000 7f9bfedab740 1 -- 172.21.15.107:0/3099758032 shutdown_connections 2024-05-09T08:41:02.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.659+0000 7f9bfedab740 1 -- 172.21.15.107:0/3099758032 wait complete. 2024-05-09T08:41:02.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.659+0000 7f9bfedab740 20 client.17931 trim_cache size 0 max 16384 2024-05-09T08:41:02.664 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:41:02.664 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:41:02.665 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "bf9f267e-0705-497a-b064-ac8e913a8753", 2024-05-09T08:41:02.665 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:41:02.665 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:41:02.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.659+0000 7f9bfbfc1640 1 -- 172.21.15.107:0/4147795582 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9be00a2d00 msgr2=0x7f9be00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.659+0000 7f9bfbfc1640 1 --2- 172.21.15.107:0/4147795582 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9be00a2d00 0x7f9be00a51c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9bec09ba20 tx=0x7f9bec06dce0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.659+0000 7f9bfbfc1640 1 -- 172.21.15.107:0/4147795582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bf4106710 msgr2=0x7f9bf41a3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.659+0000 7f9bfbfc1640 1 --2- 172.21.15.107:0/4147795582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bf4106710 0x7f9bf41a3eb0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00cce0 tx=0x7f9bec00a620 comp rx=0 tx=0).stop 2024-05-09T08:41:02.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.659+0000 7f9bfbfc1640 1 -- 172.21.15.107:0/4147795582 shutdown_connections 2024-05-09T08:41:02.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.659+0000 7f9bfbfc1640 1 --2- 172.21.15.107:0/4147795582 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9be00a2d00 0x7f9be00a51c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.659+0000 7f9bfbfc1640 1 --2- 172.21.15.107:0/4147795582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bf4106710 0x7f9bf41a3eb0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.659+0000 7f9bfbfc1640 1 --2- 172.21.15.107:0/4147795582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bf41010c0 0x7f9bf4199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.659+0000 7f9bfbfc1640 1 --2- 172.21.15.107:0/4147795582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bf4100700 0x7f9bf4198e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.659+0000 7f9bfbfc1640 1 -- 172.21.15.107:0/4147795582 >> 172.21.15.107:0/4147795582 conn(0x7f9bf4077830 msgr2=0x7f9bf41030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.659+0000 7f9bfbfc1640 1 -- 172.21.15.107:0/4147795582 shutdown_connections 2024-05-09T08:41:02.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.659+0000 7f9bfbfc1640 1 -- 172.21.15.107:0/4147795582 wait complete. 2024-05-09T08:41:02.672 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:41:02.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:41:02.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:02 smithi107 ceph-mon[30122]: pgmap v235: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 20 op/s 2024-05-09T08:41:02.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:02 smithi107 ceph-mon[30122]: osdmap e97: 12 total, 12 up, 12 in 2024-05-09T08:41:02.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:02 smithi157 ceph-mon[33614]: pgmap v235: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 20 op/s 2024-05-09T08:41:02.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:02 smithi157 ceph-mon[33614]: osdmap e97: 12 total, 12 up, 12 in 2024-05-09T08:41:02.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.863+0000 7ffb09041640 1 -- 172.21.15.107:0/506309097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb04106710 msgr2=0x7ffb04106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.863+0000 7ffb09041640 1 --2- 172.21.15.107:0/506309097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb04106710 0x7ffb04106af0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4009930 tx=0x7ffaf402a420 comp rx=0 tx=0).stop 2024-05-09T08:41:02.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.863+0000 7ffb09041640 1 -- 172.21.15.107:0/506309097 shutdown_connections 2024-05-09T08:41:02.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.863+0000 7ffb09041640 1 --2- 172.21.15.107:0/506309097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb041010c0 0x7ffb0410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.863+0000 7ffb09041640 1 --2- 172.21.15.107:0/506309097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb04100700 0x7ffb04100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.863+0000 7ffb09041640 1 --2- 172.21.15.107:0/506309097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb04106710 0x7ffb04106af0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.863+0000 7ffb09041640 1 -- 172.21.15.107:0/506309097 >> 172.21.15.107:0/506309097 conn(0x7ffb04077830 msgr2=0x7ffb04077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.863+0000 7ffb09041640 1 -- 172.21.15.107:0/506309097 shutdown_connections 2024-05-09T08:41:02.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.863+0000 7ffb09041640 1 -- 172.21.15.107:0/506309097 wait complete. 2024-05-09T08:41:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.864+0000 7ffb09041640 1 Processor -- start 2024-05-09T08:41:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.864+0000 7ffb09041640 1 -- start start 2024-05-09T08:41:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.864+0000 7ffb09041640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb04100700 0x7ffb04198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.864+0000 7ffb09041640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb041010c0 0x7ffb04198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.864+0000 7ffb09041640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb04106710 0x7ffb0419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.864+0000 7ffb09041640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffb04110a20 con 0x7ffb04106710 2024-05-09T08:41:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.864+0000 7ffb09041640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffb041108a0 con 0x7ffb041010c0 2024-05-09T08:41:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.864+0000 7ffb09041640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffb04110ba0 con 0x7ffb04100700 2024-05-09T08:41:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.864+0000 7ffb0359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb04106710 0x7ffb0419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.864+0000 7ffb0359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb04106710 0x7ffb0419c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47714/0 (socket says 172.21.15.107:47714) 2024-05-09T08:41:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.864+0000 7ffb0359e640 1 -- 172.21.15.107:0/3989489226 learned_addr learned my addr 172.21.15.107:0/3989489226 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.864+0000 7ffb0259c640 1 --2- 172.21.15.107:0/3989489226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb041010c0 0x7ffb04198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.864+0000 7ffb02d9d640 1 --2- 172.21.15.107:0/3989489226 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb04100700 0x7ffb04198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.865+0000 7ffb0359e640 1 -- 172.21.15.107:0/3989489226 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb04100700 msgr2=0x7ffb04198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.865+0000 7ffb0359e640 1 --2- 172.21.15.107:0/3989489226 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb04100700 0x7ffb04198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.865+0000 7ffb0359e640 1 -- 172.21.15.107:0/3989489226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb041010c0 msgr2=0x7ffb04198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.865+0000 7ffb0359e640 1 --2- 172.21.15.107:0/3989489226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb041010c0 0x7ffb04198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.865+0000 7ffb0359e640 1 -- 172.21.15.107:0/3989489226 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb0419d000 con 0x7ffb04106710 2024-05-09T08:41:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.865+0000 7ffb02d9d640 1 --2- 172.21.15.107:0/3989489226 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb04100700 0x7ffb04198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.865+0000 7ffb0359e640 1 --2- 172.21.15.107:0/3989489226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb04106710 0x7ffb0419c8d0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7ffafc00eaa0 tx=0x7ffafc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.865+0000 7ffaebfff640 1 -- 172.21.15.107:0/3989489226 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc00cdb0 con 0x7ffb04106710 2024-05-09T08:41:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.865+0000 7ffb09041640 1 -- 172.21.15.107:0/3989489226 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb0419d2f0 con 0x7ffb04106710 2024-05-09T08:41:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.865+0000 7ffaebfff640 1 -- 172.21.15.107:0/3989489226 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffafc005020 con 0x7ffb04106710 2024-05-09T08:41:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.865+0000 7ffaebfff640 1 -- 172.21.15.107:0/3989489226 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc0116d0 con 0x7ffb04106710 2024-05-09T08:41:02.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.865+0000 7ffb09041640 1 -- 172.21.15.107:0/3989489226 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb041a4b80 con 0x7ffb04106710 2024-05-09T08:41:02.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.868+0000 7ffb0be1f740 1 Processor -- start 2024-05-09T08:41:02.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.868+0000 7ffaebfff640 1 -- 172.21.15.107:0/3989489226 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ffafc0051d0 con 0x7ffb04106710 2024-05-09T08:41:02.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.868+0000 7ffb0be1f740 1 -- start start 2024-05-09T08:41:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.868+0000 7ffb0be1f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab45d160 0x5632ab45d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.868+0000 7ffb0be1f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45da80 0x5632ab46c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb0be1f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab46c680 0x5632ab46ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb0be1f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5632ab3c2d00 con 0x5632ab46c680 2024-05-09T08:41:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb0be1f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5632ab36a4c0 con 0x5632ab45da80 2024-05-09T08:41:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb0be1f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5632ab3becc0 con 0x5632ab45d160 2024-05-09T08:41:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb0259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab46c680 0x5632ab46ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb0259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab46c680 0x5632ab46ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47718/0 (socket says 172.21.15.107:47718) 2024-05-09T08:41:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb0259c640 1 -- 172.21.15.107:0/1479072906 learned_addr learned my addr 172.21.15.107:0/1479072906 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb02d9d640 1 --2- 172.21.15.107:0/1479072906 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45da80 0x5632ab46c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb0259c640 1 --2- 172.21.15.107:0/1479072906 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab45d160 0x5632ab45d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffaebfff640 1 --2- 172.21.15.107:0/3989489226 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffaec0a2d00 0x7ffaec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb0259c640 1 -- 172.21.15.107:0/1479072906 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab45d160 msgr2=0x5632ab45d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb0259c640 1 --2- 172.21.15.107:0/1479072906 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab45d160 0x5632ab45d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb0259c640 1 -- 172.21.15.107:0/1479072906 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45da80 msgr2=0x5632ab46c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb0259c640 1 --2- 172.21.15.107:0/1479072906 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45da80 0x5632ab46c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb0259c640 1 -- 172.21.15.107:0/1479072906 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632ab1a07d0 con 0x5632ab46c680 2024-05-09T08:41:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb02d9d640 1 --2- 172.21.15.107:0/1479072906 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45da80 0x5632ab46c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb0359e640 1 --2- 172.21.15.107:0/3989489226 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffaec0a2d00 0x7ffaec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffaebfff640 1 -- 172.21.15.107:0/3989489226 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7ffafc0cfa10 con 0x7ffb04106710 2024-05-09T08:41:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.869+0000 7ffb0259c640 1 --2- 172.21.15.107:0/1479072906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab46c680 0x5632ab46ea70 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7ffaf80058c0 tx=0x7ffaf8005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73bafe582471893 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.870+0000 7ffae97fa640 1 -- 172.21.15.107:0/1479072906 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8004030 con 0x5632ab46c680 2024-05-09T08:41:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.870+0000 7ffae97fa640 1 -- 172.21.15.107:0/1479072906 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffaf8005020 con 0x5632ab46c680 2024-05-09T08:41:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.870+0000 7ffae97fa640 1 -- 172.21.15.107:0/1479072906 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8004c10 con 0x5632ab46c680 2024-05-09T08:41:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.870+0000 7ffb0be1f740 1 -- 172.21.15.107:0/1479072906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab46c680 msgr2=0x5632ab46ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.870+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/1479072906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab46c680 0x5632ab46ea70 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7ffaf80058c0 tx=0x7ffaf8005850 comp rx=0 tx=0).stop 2024-05-09T08:41:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.870+0000 7ffb0359e640 1 --2- 172.21.15.107:0/3989489226 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffaec0a2d00 0x7ffaec0a51c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ffafc0984f0 tx=0x7ffafc086650 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.870+0000 7ffb0be1f740 1 -- 172.21.15.107:0/1479072906 shutdown_connections 2024-05-09T08:41:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.870+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/1479072906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab46c680 0x5632ab46ea70 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.870+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/1479072906 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45da80 0x5632ab46c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.870+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/1479072906 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab45d160 0x5632ab45d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.870+0000 7ffb0be1f740 1 -- 172.21.15.107:0/1479072906 >> 172.21.15.107:0/1479072906 conn(0x5632ab4595a0 msgr2=0x5632ab459980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.870+0000 7ffb0be1f740 1 -- 172.21.15.107:0/1479072906 shutdown_connections 2024-05-09T08:41:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.870+0000 7ffb0be1f740 1 -- 172.21.15.107:0/1479072906 wait complete. 2024-05-09T08:41:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0be1f740 1 -- start start 2024-05-09T08:41:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0be1f740 1 Processor -- start 2024-05-09T08:41:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0be1f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45d160 0x5632ab469620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0be1f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab45da80 0x5632ab469b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0be1f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab46c680 0x5632ab46a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0be1f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5632ab46f4c0 con 0x5632ab46c680 2024-05-09T08:41:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0be1f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5632ab37db90 con 0x5632ab45d160 2024-05-09T08:41:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0be1f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5632ab46f640 con 0x5632ab45da80 2024-05-09T08:41:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0be1f740 10 client.0 _ll_register_callbacks cb 0x5632ab3b9da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab46c680 0x5632ab46a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45d160 0x5632ab469620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb02d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab45da80 0x5632ab469b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45d160 0x5632ab469620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:55424/0 (socket says 172.21.15.107:55424) 2024-05-09T08:41:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0259c640 1 -- 172.21.15.107:0/2430996318 learned_addr learned my addr 172.21.15.107:0/2430996318 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0259c640 1 -- 172.21.15.107:0/2430996318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab45da80 msgr2=0x5632ab469b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0259c640 1 --2- 172.21.15.107:0/2430996318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab45da80 0x5632ab469b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0259c640 1 -- 172.21.15.107:0/2430996318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab46c680 msgr2=0x5632ab46a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0259c640 1 --2- 172.21.15.107:0/2430996318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab46c680 0x5632ab46a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.871+0000 7ffb0259c640 1 -- 172.21.15.107:0/2430996318 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632ab0d63f0 con 0x5632ab45d160 2024-05-09T08:41:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.872+0000 7ffb0259c640 1 --2- 172.21.15.107:0/2430996318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45d160 0x5632ab469620 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8004fd0 tx=0x7ffaf8005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfee0047c4f3bee0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.872+0000 7ffae6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:41:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.872+0000 7ffae6ff5640 1 -- 172.21.15.107:0/2430996318 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8025070 con 0x5632ab45d160 2024-05-09T08:41:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.872+0000 7ffb0be1f740 1 -- 172.21.15.107:0/2430996318 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaf8005020 con 0x5632ab45d160 2024-05-09T08:41:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.872+0000 7ffae6ff5640 1 -- 172.21.15.107:0/2430996318 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7ffaf80051d0 con 0x5632ab45d160 2024-05-09T08:41:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.872+0000 7ffae6ff5640 1 -- 172.21.15.107:0/2430996318 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffaf8028040 con 0x5632ab45d160 2024-05-09T08:41:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.872+0000 7ffb0be1f740 10 client.26927 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.872+0000 7ffb0be1f740 1 -- 172.21.15.107:0/2430996318 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5632ab0960f0 con 0x5632ab45d160 2024-05-09T08:41:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.872+0000 7ffae6ff5640 1 -- 172.21.15.107:0/2430996318 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffaf8005020 con 0x5632ab45d160 2024-05-09T08:41:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.872+0000 7ffae6ff5640 1 -- 172.21.15.107:0/2430996318 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ffaf801f440 con 0x5632ab45d160 2024-05-09T08:41:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.872+0000 7ffb0be1f740 10 client.26927 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.872+0000 7ffb0be1f740 10 client.26927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.872+0000 7ffb0be1f740 20 client.26927 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.872+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/2430996318 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5632ab47c9b0 0x5632ab47eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.872+0000 7ffb0be1f740 4 client.26927 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.872+0000 7ffb0be1f740 1 -- 172.21.15.107:0/2430996318 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632ab3d0430 con 0x5632ab47c9b0 2024-05-09T08:41:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.873+0000 7ffb02d9d640 1 --2- 172.21.15.107:0/2430996318 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5632ab47c9b0 0x5632ab47eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.873+0000 7ffb02d9d640 1 --2- 172.21.15.107:0/2430996318 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5632ab47c9b0 0x5632ab47eda0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddadb5fb7859c113 server_cookie=489401de37be7537 in_seq=0 out_seq=0 2024-05-09T08:41:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.873+0000 7ffae6ff5640 10 client.26927 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:02.879 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:02 smithi110 ceph-mon[33622]: pgmap v235: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 20 op/s 2024-05-09T08:41:02.880 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:02 smithi110 ceph-mon[33622]: osdmap e97: 12 total, 12 up, 12 in 2024-05-09T08:41:02.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.874+0000 7ffae6ff5640 1 -- 172.21.15.107:0/2430996318 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5632ab3d0430 con 0x5632ab47c9b0 2024-05-09T08:41:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.874+0000 7ffae6ff5640 10 client.26927 handle_command_reply: tid=0 2024-05-09T08:41:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.874+0000 7ffb0be1f740 1 client.26927 shutdown 2024-05-09T08:41:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.874+0000 7ffb0be1f740 1 -- 172.21.15.107:0/2430996318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45d160 msgr2=0x5632ab469620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.874+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/2430996318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45d160 0x5632ab469620 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8004fd0 tx=0x7ffaf8005f90 comp rx=0 tx=0).stop 2024-05-09T08:41:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.874+0000 7ffb0be1f740 1 -- 172.21.15.107:0/2430996318 shutdown_connections 2024-05-09T08:41:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.874+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/2430996318 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5632ab47c9b0 0x5632ab47eda0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.874+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/2430996318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab46c680 0x5632ab46a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.874+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/2430996318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab45da80 0x5632ab469b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.874+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/2430996318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45d160 0x5632ab469620 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.874+0000 7ffae6ff5640 0 client.26927 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.874+0000 7ffb0be1f740 1 -- 172.21.15.107:0/2430996318 >> 172.21.15.107:0/2430996318 conn(0x5632ab4595a0 msgr2=0x5632ab45f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.874+0000 7ffb0be1f740 1 -- 172.21.15.107:0/2430996318 shutdown_connections 2024-05-09T08:41:02.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.874+0000 7ffb0be1f740 1 -- 172.21.15.107:0/2430996318 wait complete. 2024-05-09T08:41:02.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.874+0000 7ffb0be1f740 20 client.26927 trim_cache size 0 max 16384 2024-05-09T08:41:02.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.894+0000 7ffb0be1f740 1 Processor -- start 2024-05-09T08:41:02.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.894+0000 7ffb0be1f740 1 -- start start 2024-05-09T08:41:02.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb0be1f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab4595a0 0x5632ab478450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb0be1f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab45d160 0x5632ab4844e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb0be1f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45da80 0x5632ab484a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb0be1f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffb041af4e0 con 0x5632ab45d160 2024-05-09T08:41:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb0be1f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffb041acdb0 con 0x5632ab45da80 2024-05-09T08:41:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb0be1f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffb041aac20 con 0x5632ab4595a0 2024-05-09T08:41:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb02d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab45d160 0x5632ab4844e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb02d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab45d160 0x5632ab4844e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47738/0 (socket says 172.21.15.107:47738) 2024-05-09T08:41:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb02d9d640 1 -- 172.21.15.107:0/3802430248 learned_addr learned my addr 172.21.15.107:0/3802430248 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb0259c640 1 --2- 172.21.15.107:0/3802430248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab4595a0 0x5632ab478450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb0259c640 1 --2- 172.21.15.107:0/3802430248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45da80 0x5632ab484a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb02d9d640 1 -- 172.21.15.107:0/3802430248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab4595a0 msgr2=0x5632ab478450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb02d9d640 1 --2- 172.21.15.107:0/3802430248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab4595a0 0x5632ab478450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb02d9d640 1 -- 172.21.15.107:0/3802430248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45da80 msgr2=0x5632ab484a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb02d9d640 1 --2- 172.21.15.107:0/3802430248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45da80 0x5632ab484a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb02d9d640 1 -- 172.21.15.107:0/3802430248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632ab46f3a0 con 0x5632ab45d160 2024-05-09T08:41:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb0259c640 1 --2- 172.21.15.107:0/3802430248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab4595a0 0x5632ab478450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.895+0000 7ffb02d9d640 1 --2- 172.21.15.107:0/3802430248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab45d160 0x5632ab4844e0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4006010 tx=0x7ffaf4005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3f7c3804ee4752 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.896+0000 7ffae8ff9640 1 -- 172.21.15.107:0/3802430248 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4035040 con 0x5632ab45d160 2024-05-09T08:41:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.896+0000 7ffae8ff9640 1 -- 172.21.15.107:0/3802430248 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffaf4045830 con 0x5632ab45d160 2024-05-09T08:41:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.896+0000 7ffae8ff9640 1 -- 172.21.15.107:0/3802430248 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf40046c0 con 0x5632ab45d160 2024-05-09T08:41:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.896+0000 7ffb0be1f740 1 -- 172.21.15.107:0/3802430248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab45d160 msgr2=0x5632ab4844e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.896+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/3802430248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab45d160 0x5632ab4844e0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4006010 tx=0x7ffaf4005b50 comp rx=0 tx=0).stop 2024-05-09T08:41:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.896+0000 7ffb0be1f740 1 -- 172.21.15.107:0/3802430248 shutdown_connections 2024-05-09T08:41:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.896+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/3802430248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45da80 0x5632ab484a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.896+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/3802430248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab45d160 0x5632ab4844e0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.896+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/3802430248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab4595a0 0x5632ab478450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.896+0000 7ffb0be1f740 1 -- 172.21.15.107:0/3802430248 >> 172.21.15.107:0/3802430248 conn(0x5632ab4549e0 msgr2=0x5632ab457f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.897+0000 7ffb0be1f740 1 -- 172.21.15.107:0/3802430248 shutdown_connections 2024-05-09T08:41:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.897+0000 7ffb0be1f740 1 -- 172.21.15.107:0/3802430248 wait complete. 2024-05-09T08:41:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.897+0000 7ffb0be1f740 1 -- start start 2024-05-09T08:41:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.897+0000 7ffb0be1f740 1 Processor -- start 2024-05-09T08:41:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.897+0000 7ffb0be1f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab4595a0 0x5632ab470ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.897+0000 7ffb0be1f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45d160 0x5632ab470ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.897+0000 7ffb0be1f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab45da80 0x5632ab4e1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.897+0000 7ffb0be1f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffb041ab270 con 0x5632ab45da80 2024-05-09T08:41:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.897+0000 7ffb0be1f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffb041ac6e0 con 0x5632ab45d160 2024-05-09T08:41:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.897+0000 7ffb0be1f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffb041ab940 con 0x5632ab4595a0 2024-05-09T08:41:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.897+0000 7ffb0be1f740 10 client.0 _ll_register_callbacks cb 0x7ffaf8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.897+0000 7ffb0259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab45da80 0x5632ab4e1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.897+0000 7ffb0259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab45da80 0x5632ab4e1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47740/0 (socket says 172.21.15.107:47740) 2024-05-09T08:41:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.897+0000 7ffb0259c640 1 -- 172.21.15.107:0/2952667282 learned_addr learned my addr 172.21.15.107:0/2952667282 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.897+0000 7ffb0259c640 1 --2- 172.21.15.107:0/2952667282 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab4595a0 0x5632ab470ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.898+0000 7ffb02d9d640 1 --2- 172.21.15.107:0/2952667282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45d160 0x5632ab470ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.898+0000 7ffb0259c640 1 -- 172.21.15.107:0/2952667282 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab4595a0 msgr2=0x5632ab470ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.898+0000 7ffb0259c640 1 --2- 172.21.15.107:0/2952667282 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab4595a0 0x5632ab470ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.898+0000 7ffb0259c640 1 -- 172.21.15.107:0/2952667282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45d160 msgr2=0x5632ab470ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.898+0000 7ffb0259c640 1 --2- 172.21.15.107:0/2952667282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45d160 0x5632ab470ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.898+0000 7ffb0259c640 1 -- 172.21.15.107:0/2952667282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632ab4e2290 con 0x5632ab45da80 2024-05-09T08:41:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.898+0000 7ffb0259c640 1 --2- 172.21.15.107:0/2952667282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab45da80 0x5632ab4e1d50 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7ffaf80056c0 tx=0x7ffaf80097c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a04aed3eb0a914 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.898+0000 7ffae6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.898+0000 7ffae6ff5640 1 -- 172.21.15.107:0/2952667282 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf801fe60 con 0x5632ab45da80 2024-05-09T08:41:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.898+0000 7ffae6ff5640 1 -- 172.21.15.107:0/2952667282 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7ffaf8005210 con 0x5632ab45da80 2024-05-09T08:41:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.898+0000 7ffb0be1f740 1 -- 172.21.15.107:0/2952667282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaf4045830 con 0x5632ab45da80 2024-05-09T08:41:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.898+0000 7ffae6ff5640 1 -- 172.21.15.107:0/2952667282 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffaf8025040 con 0x5632ab45da80 2024-05-09T08:41:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.898+0000 7ffb0be1f740 10 client.17949 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.898+0000 7ffb0be1f740 1 -- 172.21.15.107:0/2952667282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5632ab4e3510 con 0x5632ab45da80 2024-05-09T08:41:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.898+0000 7ffae6ff5640 1 -- 172.21.15.107:0/2952667282 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffaf4045830 con 0x5632ab45da80 2024-05-09T08:41:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.899+0000 7ffae6ff5640 1 -- 172.21.15.107:0/2952667282 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ffaf8004370 con 0x5632ab45da80 2024-05-09T08:41:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.899+0000 7ffb0be1f740 10 client.17949 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.899+0000 7ffb0be1f740 10 client.17949 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.899+0000 7ffb0be1f740 20 client.17949 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:02.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.899+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/2952667282 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5632ab46c680 0x5632ab4e5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.899+0000 7ffb0be1f740 4 client.17949 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:41:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.899+0000 7ffb0be1f740 1 -- 172.21.15.107:0/2952667282 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5632ab3ca780 con 0x5632ab46c680 2024-05-09T08:41:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.899+0000 7ffb02d9d640 1 --2- 172.21.15.107:0/2952667282 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5632ab46c680 0x5632ab4e5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.899+0000 7ffae6ff5640 10 client.17949 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.899+0000 7ffb02d9d640 1 --2- 172.21.15.107:0/2952667282 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5632ab46c680 0x5632ab4e5b10 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acf01986da7a349b server_cookie=6b0ae57204915eff in_seq=0 out_seq=0 2024-05-09T08:41:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.899+0000 7ffae6ff5640 1 -- 172.21.15.107:0/2952667282 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5632ab3ca780 con 0x5632ab46c680 2024-05-09T08:41:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.899+0000 7ffae6ff5640 10 client.17949 handle_command_reply: tid=0 2024-05-09T08:41:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.899+0000 7ffb0be1f740 1 client.17949 shutdown 2024-05-09T08:41:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb0be1f740 1 -- 172.21.15.107:0/2952667282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab45da80 msgr2=0x5632ab4e1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/2952667282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab45da80 0x5632ab4e1d50 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7ffaf80056c0 tx=0x7ffaf80097c0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb0be1f740 1 -- 172.21.15.107:0/2952667282 shutdown_connections 2024-05-09T08:41:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/2952667282 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5632ab46c680 0x5632ab4e5b10 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffae6ff5640 0 client.17949 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/2952667282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632ab45da80 0x5632ab4e1d50 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/2952667282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632ab45d160 0x5632ab470ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb0be1f740 1 --2- 172.21.15.107:0/2952667282 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632ab4595a0 0x5632ab470ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb0be1f740 1 -- 172.21.15.107:0/2952667282 >> 172.21.15.107:0/2952667282 conn(0x5632ab4549e0 msgr2=0x5632ab4695e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb0be1f740 1 -- 172.21.15.107:0/2952667282 shutdown_connections 2024-05-09T08:41:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb0be1f740 1 -- 172.21.15.107:0/2952667282 wait complete. 2024-05-09T08:41:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb0be1f740 20 client.17949 trim_cache size 0 max 16384 2024-05-09T08:41:02.909 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:41:02.910 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:41:02.910 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:41:02.910 INFO:teuthology.orchestra.run.smithi107.stdout: "bf9f267e-0705-497a-b064-ac8e913a8753": { 2024-05-09T08:41:02.910 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:41:02.910 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "bf9f267e-0705-497a-b064-ac8e913a8753", 2024-05-09T08:41:02.910 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:41:02.910 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:41:02.910 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:41:02.910 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:41:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb09041640 1 -- 172.21.15.107:0/3989489226 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffaec0a2d00 msgr2=0x7ffaec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb09041640 1 --2- 172.21.15.107:0/3989489226 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffaec0a2d00 0x7ffaec0a51c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ffafc0984f0 tx=0x7ffafc086650 comp rx=0 tx=0).stop 2024-05-09T08:41:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb09041640 1 -- 172.21.15.107:0/3989489226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb04106710 msgr2=0x7ffb0419c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb09041640 1 --2- 172.21.15.107:0/3989489226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb04106710 0x7ffb0419c8d0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7ffafc00eaa0 tx=0x7ffafc00ef60 comp rx=0 tx=0).stop 2024-05-09T08:41:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb09041640 1 -- 172.21.15.107:0/3989489226 shutdown_connections 2024-05-09T08:41:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb09041640 1 --2- 172.21.15.107:0/3989489226 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffaec0a2d00 0x7ffaec0a51c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb09041640 1 --2- 172.21.15.107:0/3989489226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffb04106710 0x7ffb0419c8d0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb09041640 1 --2- 172.21.15.107:0/3989489226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb041010c0 0x7ffb04198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb09041640 1 --2- 172.21.15.107:0/3989489226 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb04100700 0x7ffb04198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.900+0000 7ffb09041640 1 -- 172.21.15.107:0/3989489226 >> 172.21.15.107:0/3989489226 conn(0x7ffb04077830 msgr2=0x7ffb04068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.901+0000 7ffb09041640 1 -- 172.21.15.107:0/3989489226 shutdown_connections 2024-05-09T08:41:02.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:02.901+0000 7ffb09041640 1 -- 172.21.15.107:0/3989489226 wait complete. 2024-05-09T08:41:02.913 INFO:tasks.cephfs.filesystem:scrub status for tag:bf9f267e-0705-497a-b064-ac8e913a8753 - {'path': '/', 'tag': 'bf9f267e-0705-497a-b064-ac8e913a8753', 'options': 'recursive,force'} 2024-05-09T08:41:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.509+0000 7f9bbc6c3640 1 -- 172.21.15.107:0/2080819878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bb4100710 msgr2=0x7f9bb4100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.509+0000 7f9bbc6c3640 1 --2- 172.21.15.107:0/2080819878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bb4100710 0x7f9bb4100b90 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4009930 tx=0x7f9ba402a420 comp rx=0 tx=0).stop 2024-05-09T08:41:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.509+0000 7f9bbc6c3640 1 -- 172.21.15.107:0/2080819878 shutdown_connections 2024-05-09T08:41:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.509+0000 7f9bbc6c3640 1 --2- 172.21.15.107:0/2080819878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bb41010d0 0x7f9bb410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.509+0000 7f9bbc6c3640 1 --2- 172.21.15.107:0/2080819878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bb4100710 0x7f9bb4100b90 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.509+0000 7f9bbc6c3640 1 --2- 172.21.15.107:0/2080819878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bb4106720 0x7f9bb4106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.509+0000 7f9bbc6c3640 1 -- 172.21.15.107:0/2080819878 >> 172.21.15.107:0/2080819878 conn(0x7f9bb40fc200 msgr2=0x7f9bb40fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.509+0000 7f9bbc6c3640 1 -- 172.21.15.107:0/2080819878 shutdown_connections 2024-05-09T08:41:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.509+0000 7f9bbc6c3640 1 -- 172.21.15.107:0/2080819878 wait complete. 2024-05-09T08:41:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.509+0000 7f9bbc6c3640 1 Processor -- start 2024-05-09T08:41:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.509+0000 7f9bbc6c3640 1 -- start start 2024-05-09T08:41:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.509+0000 7f9bbc6c3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bb4100710 0x7f9bb4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bbc6c3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bb41010d0 0x7f9bb4198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bbc6c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bb4106720 0x7f9bb419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bbc6c3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb41109e0 con 0x7f9bb4106720 2024-05-09T08:41:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bbc6c3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb4110860 con 0x7f9bb41010d0 2024-05-09T08:41:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bbc6c3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb4110b60 con 0x7f9bb4100710 2024-05-09T08:41:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bbac60640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bb4106720 0x7f9bb419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bbac60640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bb4106720 0x7f9bb419c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47758/0 (socket says 172.21.15.107:47758) 2024-05-09T08:41:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bbac60640 1 -- 172.21.15.107:0/2269423102 learned_addr learned my addr 172.21.15.107:0/2269423102 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bba45f640 1 --2- 172.21.15.107:0/2269423102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bb4100710 0x7f9bb4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bb9c5e640 1 --2- 172.21.15.107:0/2269423102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bb41010d0 0x7f9bb4198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bbac60640 1 -- 172.21.15.107:0/2269423102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bb4100710 msgr2=0x7f9bb4197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bbac60640 1 --2- 172.21.15.107:0/2269423102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bb4100710 0x7f9bb4197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bbac60640 1 -- 172.21.15.107:0/2269423102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bb41010d0 msgr2=0x7f9bb4198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bbac60640 1 --2- 172.21.15.107:0/2269423102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bb41010d0 0x7f9bb4198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bbac60640 1 -- 172.21.15.107:0/2269423102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb419cfb0 con 0x7f9bb4106720 2024-05-09T08:41:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bbac60640 1 --2- 172.21.15.107:0/2269423102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bb4106720 0x7f9bb419c880 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f9bac009fd0 tx=0x7f9bac00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bab7fe640 1 -- 172.21.15.107:0/2269423102 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bac017070 con 0x7f9bb4106720 2024-05-09T08:41:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bab7fe640 1 -- 172.21.15.107:0/2269423102 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bac003e20 con 0x7f9bb4106720 2024-05-09T08:41:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bbc6c3640 1 -- 172.21.15.107:0/2269423102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bb419d2a0 con 0x7f9bb4106720 2024-05-09T08:41:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bab7fe640 1 -- 172.21.15.107:0/2269423102 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bac010460 con 0x7f9bb4106720 2024-05-09T08:41:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.510+0000 7f9bbc6c3640 1 -- 172.21.15.107:0/2269423102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bb41a4b30 con 0x7f9bb4106720 2024-05-09T08:41:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.513+0000 7f9bab7fe640 1 -- 172.21.15.107:0/2269423102 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f9bac003970 con 0x7f9bb4106720 2024-05-09T08:41:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.513+0000 7f9bbf3cb740 1 Processor -- start 2024-05-09T08:41:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bbf3cb740 1 -- start start 2024-05-09T08:41:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bab7fe640 1 --2- 172.21.15.107:0/2269423102 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9ba00a2da0 0x7f9ba00a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bbf3cb740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d392e450 0x5582d392e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bbf3cb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d392ed70 0x5582d39390f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bbf3cb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d3939630 0x5582d393ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bbf3cb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5582d3893340 con 0x5582d3939630 2024-05-09T08:41:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bbf3cb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5582d383b290 con 0x5582d392ed70 2024-05-09T08:41:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bbf3cb740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5582d388f300 con 0x5582d392e450 2024-05-09T08:41:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bb9c5e640 1 --2- 172.21.15.107:0/2269423102 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9ba00a2da0 0x7f9ba00a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bbac60640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d3939630 0x5582d393ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bbac60640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d3939630 0x5582d393ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47766/0 (socket says 172.21.15.107:47766) 2024-05-09T08:41:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bab7fe640 1 -- 172.21.15.107:0/2269423102 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f9bac0cf610 con 0x7f9bb4106720 2024-05-09T08:41:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bbac60640 1 -- 172.21.15.107:0/594464039 learned_addr learned my addr 172.21.15.107:0/594464039 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bb9c5e640 1 --2- 172.21.15.107:0/594464039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d392ed70 0x5582d39390f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bbac60640 1 -- 172.21.15.107:0/594464039 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d392e450 msgr2=0x5582d392e830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:41:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bbac60640 1 --2- 172.21.15.107:0/594464039 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d392e450 0x5582d392e830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bbac60640 1 -- 172.21.15.107:0/594464039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d392ed70 msgr2=0x5582d39390f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bbac60640 1 --2- 172.21.15.107:0/594464039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d392ed70 0x5582d39390f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.514+0000 7f9bbac60640 1 -- 172.21.15.107:0/594464039 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582d385f8e0 con 0x5582d3939630 2024-05-09T08:41:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.515+0000 7f9bb9c5e640 1 --2- 172.21.15.107:0/2269423102 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9ba00a2da0 0x7f9ba00a5260 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4009900 tx=0x7f9ba4031ea0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.515+0000 7f9bbac60640 1 --2- 172.21.15.107:0/594464039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d3939630 0x5582d393ba20 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f9bac09b4d0 tx=0x7f9bac097440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=566298b1eca498a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.515+0000 7f9ba8ff9640 1 -- 172.21.15.107:0/594464039 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bac020070 con 0x5582d3939630 2024-05-09T08:41:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.515+0000 7f9ba8ff9640 1 -- 172.21.15.107:0/594464039 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bac003c60 con 0x5582d3939630 2024-05-09T08:41:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.515+0000 7f9ba8ff9640 1 -- 172.21.15.107:0/594464039 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bac0a4390 con 0x5582d3939630 2024-05-09T08:41:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/594464039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d3939630 msgr2=0x5582d393ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/594464039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d3939630 0x5582d393ba20 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f9bac09b4d0 tx=0x7f9bac097440 comp rx=0 tx=0).stop 2024-05-09T08:41:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/594464039 shutdown_connections 2024-05-09T08:41:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/594464039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d3939630 0x5582d393ba20 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/594464039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d392ed70 0x5582d39390f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/594464039 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d392e450 0x5582d392e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/594464039 >> 172.21.15.107:0/594464039 conn(0x5582d392a960 msgr2=0x5582d392ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/594464039 shutdown_connections 2024-05-09T08:41:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/594464039 wait complete. 2024-05-09T08:41:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 -- start start 2024-05-09T08:41:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 Processor -- start 2024-05-09T08:41:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3936490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d392ed70 0x5582d39369d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d3939630 0x5582d3936f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5582d393c260 con 0x5582d392e450 2024-05-09T08:41:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5582d384e790 con 0x5582d392ed70 2024-05-09T08:41:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.516+0000 7f9bbf3cb740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5582d393c3e0 con 0x5582d3939630 2024-05-09T08:41:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9bbf3cb740 10 client.0 _ll_register_callbacks cb 0x5582d3889a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:03.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9bba45f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3936490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:03.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9bba45f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3936490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47774/0 (socket says 172.21.15.107:47774) 2024-05-09T08:41:03.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9bba45f640 1 -- 172.21.15.107:0/2503756727 learned_addr learned my addr 172.21.15.107:0/2503756727 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:03.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9bb9c5e640 1 --2- 172.21.15.107:0/2503756727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d392ed70 0x5582d39369d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:03.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9bba45f640 1 -- 172.21.15.107:0/2503756727 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d3939630 msgr2=0x5582d3936f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:03.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9bba45f640 1 --2- 172.21.15.107:0/2503756727 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d3939630 0x5582d3936f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9bba45f640 1 -- 172.21.15.107:0/2503756727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d392ed70 msgr2=0x5582d39369d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:03.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9bba45f640 1 --2- 172.21.15.107:0/2503756727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d392ed70 0x5582d39369d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9bba45f640 1 -- 172.21.15.107:0/2503756727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582d35a7210 con 0x5582d392e450 2024-05-09T08:41:03.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9bb9c5e640 1 --2- 172.21.15.107:0/2503756727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d392ed70 0x5582d39369d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:03.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9bba45f640 1 --2- 172.21.15.107:0/2503756727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3936490 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f9bb00116f0 tx=0x7f9bb0011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b753119e8009fc8d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:03.524 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9b9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9b9dffb640 1 -- 172.21.15.107:0/2503756727 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb000f920 con 0x5582d392e450 2024-05-09T08:41:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9b9dffb640 1 -- 172.21.15.107:0/2503756727 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f9bb000fac0 con 0x5582d392e450 2024-05-09T08:41:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/2503756727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5582d38a0a70 con 0x5582d392e450 2024-05-09T08:41:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9b9dffb640 1 -- 172.21.15.107:0/2503756727 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9bb000fce0 con 0x5582d392e450 2024-05-09T08:41:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9bbf3cb740 10 client.17961 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.517+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/2503756727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5582d3673620 con 0x5582d392e450 2024-05-09T08:41:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.518+0000 7f9b9dffb640 1 -- 172.21.15.107:0/2503756727 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5582d38a0a70 con 0x5582d392e450 2024-05-09T08:41:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.518+0000 7f9b9dffb640 1 -- 172.21.15.107:0/2503756727 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9bb0018070 con 0x5582d392e450 2024-05-09T08:41:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.518+0000 7f9bbf3cb740 10 client.17961 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.518+0000 7f9bbf3cb740 10 client.17961 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.518+0000 7f9bbf3cb740 20 client.17961 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.518+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/2503756727 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5582d39492b0 0x5582d394b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.518+0000 7f9bbf3cb740 4 client.17961 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.518+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/2503756727 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582d389adc0 con 0x5582d39492b0 2024-05-09T08:41:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.518+0000 7f9bb9c5e640 1 --2- 172.21.15.107:0/2503756727 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5582d39492b0 0x5582d394b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.519+0000 7f9bb9c5e640 1 --2- 172.21.15.107:0/2503756727 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5582d39492b0 0x5582d394b720 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5356b721770224 server_cookie=95900bf3f1c3b57d in_seq=0 out_seq=0 2024-05-09T08:41:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.519+0000 7f9b9dffb640 10 client.17961 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.519+0000 7f9b9dffb640 1 -- 172.21.15.107:0/2503756727 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5582d389adc0 con 0x5582d39492b0 2024-05-09T08:41:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.519+0000 7f9b9dffb640 10 client.17961 handle_command_reply: tid=0 2024-05-09T08:41:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.519+0000 7f9bbf3cb740 1 client.17961 shutdown 2024-05-09T08:41:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.519+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/2503756727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 msgr2=0x5582d3936490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.519+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/2503756727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3936490 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f9bb00116f0 tx=0x7f9bb0011bb0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.519+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/2503756727 shutdown_connections 2024-05-09T08:41:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.519+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/2503756727 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5582d39492b0 0x5582d394b720 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.519+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/2503756727 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d3939630 0x5582d3936f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.519+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/2503756727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d392ed70 0x5582d39369d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.519+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/2503756727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3936490 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.519+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/2503756727 >> 172.21.15.107:0/2503756727 conn(0x5582d392a960 msgr2=0x5582d39306e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.519+0000 7f9b9dffb640 0 client.17961 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.520+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/2503756727 shutdown_connections 2024-05-09T08:41:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.520+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/2503756727 wait complete. 2024-05-09T08:41:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.520+0000 7f9bbf3cb740 20 client.17961 trim_cache size 0 max 16384 2024-05-09T08:41:03.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.536+0000 7f9bbf3cb740 1 Processor -- start 2024-05-09T08:41:03.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bbf3cb740 1 -- start start 2024-05-09T08:41:03.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bbf3cb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3930e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bbf3cb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d392ed70 0x5582d3951080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bbf3cb740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d3939630 0x5582d39515c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bbf3cb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb41b1650 con 0x5582d392e450 2024-05-09T08:41:03.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bbf3cb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb41aef20 con 0x5582d392ed70 2024-05-09T08:41:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bbf3cb740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb41acd90 con 0x5582d3939630 2024-05-09T08:41:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bba45f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3930e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bba45f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3930e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47782/0 (socket says 172.21.15.107:47782) 2024-05-09T08:41:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bba45f640 1 -- 172.21.15.107:0/195760977 learned_addr learned my addr 172.21.15.107:0/195760977 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bb9c5e640 1 --2- 172.21.15.107:0/195760977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d392ed70 0x5582d3951080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bbac60640 1 --2- 172.21.15.107:0/195760977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d3939630 0x5582d39515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bba45f640 1 -- 172.21.15.107:0/195760977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d3939630 msgr2=0x5582d39515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bba45f640 1 --2- 172.21.15.107:0/195760977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d3939630 0x5582d39515c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bba45f640 1 -- 172.21.15.107:0/195760977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d392ed70 msgr2=0x5582d3951080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bba45f640 1 --2- 172.21.15.107:0/195760977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d392ed70 0x5582d3951080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bba45f640 1 -- 172.21.15.107:0/195760977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582d3928f00 con 0x5582d392e450 2024-05-09T08:41:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.537+0000 7f9bba45f640 1 --2- 172.21.15.107:0/195760977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3930e10 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f9bb001a5a0 tx=0x7f9bb00096f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ddd25082be2295f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.538+0000 7f9b9ffff640 1 -- 172.21.15.107:0/195760977 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb002c070 con 0x5582d392e450 2024-05-09T08:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.538+0000 7f9b9ffff640 1 -- 172.21.15.107:0/195760977 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bb00054f0 con 0x5582d392e450 2024-05-09T08:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.538+0000 7f9b9ffff640 1 -- 172.21.15.107:0/195760977 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb001ae20 con 0x5582d392e450 2024-05-09T08:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.538+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/195760977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 msgr2=0x5582d3930e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.538+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/195760977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3930e10 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f9bb001a5a0 tx=0x7f9bb00096f0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.538+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/195760977 shutdown_connections 2024-05-09T08:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.538+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/195760977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d3939630 0x5582d39515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.538+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/195760977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d392ed70 0x5582d3951080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.538+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/195760977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3930e10 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.538+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/195760977 >> 172.21.15.107:0/195760977 conn(0x5582d392a960 msgr2=0x5582d3929a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.538+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/195760977 shutdown_connections 2024-05-09T08:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.538+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/195760977 wait complete. 2024-05-09T08:41:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bbf3cb740 1 -- start start 2024-05-09T08:41:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bbf3cb740 1 Processor -- start 2024-05-09T08:41:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bbf3cb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3941930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bbf3cb740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d392ed70 0x5582d393a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bbf3cb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d3939630 0x5582d393a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bbf3cb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb41ad3e0 con 0x5582d392e450 2024-05-09T08:41:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bbf3cb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb41ae850 con 0x5582d3939630 2024-05-09T08:41:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bbf3cb740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb41adab0 con 0x5582d392ed70 2024-05-09T08:41:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bbf3cb740 10 client.0 _ll_register_callbacks cb 0x7f9bb0009d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bba45f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3941930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bba45f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3941930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47788/0 (socket says 172.21.15.107:47788) 2024-05-09T08:41:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bba45f640 1 -- 172.21.15.107:0/245869843 learned_addr learned my addr 172.21.15.107:0/245869843 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bb9c5e640 1 --2- 172.21.15.107:0/245869843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d392ed70 0x5582d393a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bbac60640 1 --2- 172.21.15.107:0/245869843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d3939630 0x5582d393a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bba45f640 1 -- 172.21.15.107:0/245869843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d392ed70 msgr2=0x5582d393a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bba45f640 1 --2- 172.21.15.107:0/245869843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d392ed70 0x5582d393a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bba45f640 1 -- 172.21.15.107:0/245869843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d3939630 msgr2=0x5582d393a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bba45f640 1 --2- 172.21.15.107:0/245869843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d3939630 0x5582d393a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.539+0000 7f9bba45f640 1 -- 172.21.15.107:0/245869843 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582d393aec0 con 0x5582d392e450 2024-05-09T08:41:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.540+0000 7f9bba45f640 1 --2- 172.21.15.107:0/245869843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3941930 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0009560 tx=0x7f9bb001be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a5b4ebad7c7f15 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.540+0000 7f9b9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.540+0000 7f9b9dffb640 1 -- 172.21.15.107:0/245869843 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0036020 con 0x5582d392e450 2024-05-09T08:41:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.540+0000 7f9b9dffb640 1 -- 172.21.15.107:0/245869843 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f9bb00035f0 con 0x5582d392e450 2024-05-09T08:41:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.540+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/245869843 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bb00054f0 con 0x5582d392e450 2024-05-09T08:41:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.540+0000 7f9b9dffb640 1 -- 172.21.15.107:0/245869843 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9bb002c040 con 0x5582d392e450 2024-05-09T08:41:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.540+0000 7f9bbf3cb740 10 client.17967 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.540+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/245869843 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5582d39af8f0 con 0x5582d392e450 2024-05-09T08:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.540+0000 7f9b9dffb640 1 -- 172.21.15.107:0/245869843 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bb00054f0 con 0x5582d392e450 2024-05-09T08:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.540+0000 7f9b9dffb640 1 -- 172.21.15.107:0/245869843 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9bb0026070 con 0x5582d392e450 2024-05-09T08:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.541+0000 7f9bbf3cb740 10 client.17967 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.541+0000 7f9bbf3cb740 10 client.17967 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.541+0000 7f9bbf3cb740 20 client.17967 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.541+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/245869843 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5582d39aff40 0x5582d39b2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.541+0000 7f9bbf3cb740 4 client.17967 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.541+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/245869843 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5582d38188e0 con 0x5582d39aff40 2024-05-09T08:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.541+0000 7f9bb9c5e640 1 --2- 172.21.15.107:0/245869843 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5582d39aff40 0x5582d39b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.541+0000 7f9b9dffb640 10 client.17967 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.541+0000 7f9bb9c5e640 1 --2- 172.21.15.107:0/245869843 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5582d39aff40 0x5582d39b2330 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fced7ec8a48b56ae server_cookie=43c21e196b84ca53 in_seq=0 out_seq=0 2024-05-09T08:41:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.559+0000 7f9b9dffb640 1 -- 172.21.15.107:0/245869843 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5582d38188e0 con 0x5582d39aff40 2024-05-09T08:41:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.559+0000 7f9b9dffb640 10 client.17967 handle_command_reply: tid=0 2024-05-09T08:41:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.559+0000 7f9bbf3cb740 1 client.17967 shutdown 2024-05-09T08:41:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.559+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/245869843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 msgr2=0x5582d3941930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.559+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/245869843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3941930 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0009560 tx=0x7f9bb001be10 comp rx=0 tx=0).stop 2024-05-09T08:41:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.559+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/245869843 shutdown_connections 2024-05-09T08:41:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.559+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/245869843 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5582d39aff40 0x5582d39b2330 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.559+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/245869843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5582d3939630 0x5582d393a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.559+0000 7f9b9dffb640 0 client.17967 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.559+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/245869843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5582d392ed70 0x5582d393a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.559+0000 7f9bbf3cb740 1 --2- 172.21.15.107:0/245869843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5582d392e450 0x5582d3941930 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.559+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/245869843 >> 172.21.15.107:0/245869843 conn(0x5582d392a960 msgr2=0x5582d3936430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.559+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/245869843 shutdown_connections 2024-05-09T08:41:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.559+0000 7f9bbf3cb740 1 -- 172.21.15.107:0/245869843 wait complete. 2024-05-09T08:41:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.559+0000 7f9bbf3cb740 20 client.17967 trim_cache size 0 max 16384 2024-05-09T08:41:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:41:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:41:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:41:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:41:03.562 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.560+0000 7f9bbc6c3640 1 -- 172.21.15.107:0/2269423102 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9ba00a2da0 msgr2=0x7f9ba00a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:03.562 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.560+0000 7f9bbc6c3640 1 --2- 172.21.15.107:0/2269423102 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9ba00a2da0 0x7f9ba00a5260 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4009900 tx=0x7f9ba4031ea0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.562 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.560+0000 7f9bbc6c3640 1 -- 172.21.15.107:0/2269423102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bb4106720 msgr2=0x7f9bb419c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.560+0000 7f9bbc6c3640 1 --2- 172.21.15.107:0/2269423102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bb4106720 0x7f9bb419c880 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f9bac009fd0 tx=0x7f9bac00b260 comp rx=0 tx=0).stop 2024-05-09T08:41:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.560+0000 7f9bbc6c3640 1 -- 172.21.15.107:0/2269423102 shutdown_connections 2024-05-09T08:41:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.560+0000 7f9bbc6c3640 1 --2- 172.21.15.107:0/2269423102 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9ba00a2da0 0x7f9ba00a5260 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.560+0000 7f9bbc6c3640 1 --2- 172.21.15.107:0/2269423102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bb4106720 0x7f9bb419c880 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.560+0000 7f9bbc6c3640 1 --2- 172.21.15.107:0/2269423102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bb41010d0 0x7f9bb4198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.560+0000 7f9bbc6c3640 1 --2- 172.21.15.107:0/2269423102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bb4100710 0x7f9bb4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.560+0000 7f9bbc6c3640 1 -- 172.21.15.107:0/2269423102 >> 172.21.15.107:0/2269423102 conn(0x7f9bb40fc200 msgr2=0x7f9bb41045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.560+0000 7f9bbc6c3640 1 -- 172.21.15.107:0/2269423102 shutdown_connections 2024-05-09T08:41:03.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:03.560+0000 7f9bbc6c3640 1 -- 172.21.15.107:0/2269423102 wait complete. 2024-05-09T08:41:03.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:03 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:41:03.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:03 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:03.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:03 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:41:03.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:03 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:41:03.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:03 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:03.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:03 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:41:03.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:03 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:03.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:03 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:03.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:03 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:41:03.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:03 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:03.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:03 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:41:03.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:03 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:41:03.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:03 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:03.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:03 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:41:03.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:03 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:03.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:03 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:04.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:03 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:41:04.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:03 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:04.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:03 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:41:04.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:03 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:41:04.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:03 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:04.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:03 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:41:04.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:03 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:04.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:03 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:04.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:04 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:41:04.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:04 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:04.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:04 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:41:04.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:04 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:04.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:04 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:41:04.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:04 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:04.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:04 smithi157 ceph-mon[33614]: pgmap v237: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 29 MiB/s wr, 38 op/s 2024-05-09T08:41:04.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:04 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:41:04.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:04 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:04.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:04 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:41:04.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:04 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:04.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:04 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:41:04.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:04 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:04.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:04 smithi110 ceph-mon[33622]: pgmap v237: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 29 MiB/s wr, 38 op/s 2024-05-09T08:41:05.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:04 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:41:05.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:04 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:05.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:04 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:41:05.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:04 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:05.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:04 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:41:05.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:04 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:05.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:04 smithi107 ceph-mon[30122]: pgmap v237: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 29 MiB/s wr, 38 op/s 2024-05-09T08:41:06.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:06 smithi157 ceph-mon[33614]: pgmap v238: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 29 MiB/s wr, 38 op/s 2024-05-09T08:41:06.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:06 smithi110 ceph-mon[33622]: pgmap v238: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 29 MiB/s wr, 38 op/s 2024-05-09T08:41:07.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:06 smithi107 ceph-mon[30122]: pgmap v238: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 29 MiB/s wr, 38 op/s 2024-05-09T08:41:08.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:07 smithi107 ceph-mon[30122]: osdmap e98: 12 total, 12 up, 12 in 2024-05-09T08:41:08.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:07 smithi107 ceph-mon[30122]: pgmap v240: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 36 MiB/s wr, 105 op/s 2024-05-09T08:41:08.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:07 smithi157 ceph-mon[33614]: osdmap e98: 12 total, 12 up, 12 in 2024-05-09T08:41:08.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:07 smithi157 ceph-mon[33614]: pgmap v240: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 36 MiB/s wr, 105 op/s 2024-05-09T08:41:08.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:07 smithi110 ceph-mon[33622]: osdmap e98: 12 total, 12 up, 12 in 2024-05-09T08:41:08.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:07 smithi110 ceph-mon[33622]: pgmap v240: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 36 MiB/s wr, 105 op/s 2024-05-09T08:41:10.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:10 smithi107 ceph-mon[30122]: pgmap v241: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 36 MiB/s wr, 105 op/s 2024-05-09T08:41:10.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:10 smithi157 ceph-mon[33614]: pgmap v241: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 36 MiB/s wr, 105 op/s 2024-05-09T08:41:10.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:10 smithi110 ceph-mon[33622]: pgmap v241: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 36 MiB/s wr, 105 op/s 2024-05-09T08:41:11.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:11 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:41:11.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:11 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:41:11.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:11 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:41:12.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:12 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:41:12.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:12 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:41:12.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:12 smithi107 ceph-mon[30122]: pgmap v242: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 30 MiB/s wr, 90 op/s 2024-05-09T08:41:12.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:12 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:41:12.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:12 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:41:12.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:12 smithi157 ceph-mon[33614]: pgmap v242: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 30 MiB/s wr, 90 op/s 2024-05-09T08:41:12.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:12 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:41:12.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:12 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:41:12.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:12 smithi110 ceph-mon[33622]: pgmap v242: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 30 MiB/s wr, 90 op/s 2024-05-09T08:41:14.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:14 smithi107 ceph-mon[30122]: pgmap v243: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 29 MiB/s wr, 64 op/s 2024-05-09T08:41:14.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:14 smithi157 ceph-mon[33614]: pgmap v243: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 29 MiB/s wr, 64 op/s 2024-05-09T08:41:14.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:14 smithi110 ceph-mon[33622]: pgmap v243: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 29 MiB/s wr, 64 op/s 2024-05-09T08:41:16.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:16 smithi107 ceph-mon[30122]: pgmap v244: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 29 MiB/s wr, 64 op/s 2024-05-09T08:41:16.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:16 smithi157 ceph-mon[33614]: pgmap v244: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 29 MiB/s wr, 64 op/s 2024-05-09T08:41:16.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:16 smithi110 ceph-mon[33622]: pgmap v244: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 29 MiB/s wr, 64 op/s 2024-05-09T08:41:16.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.727+0000 7f01fd1c1640 1 -- 172.21.15.107:0/2496266090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01f8100760 msgr2=0x7f01f8100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:16.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.727+0000 7f01fd1c1640 1 --2- 172.21.15.107:0/2496266090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01f8100760 0x7f01f8100be0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009930 tx=0x7f01ec02a420 comp rx=0 tx=0).stop 2024-05-09T08:41:16.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.727+0000 7f01fd1c1640 1 -- 172.21.15.107:0/2496266090 shutdown_connections 2024-05-09T08:41:16.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.727+0000 7f01fd1c1640 1 --2- 172.21.15.107:0/2496266090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01f8101120 0x7f01f810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.727+0000 7f01fd1c1640 1 --2- 172.21.15.107:0/2496266090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01f8100760 0x7f01f8100be0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.727+0000 7f01fd1c1640 1 --2- 172.21.15.107:0/2496266090 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f8106720 0x7f01f8106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.727+0000 7f01fd1c1640 1 -- 172.21.15.107:0/2496266090 >> 172.21.15.107:0/2496266090 conn(0x7f01f80778b0 msgr2=0x7f01f8077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:16.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.727+0000 7f01fd1c1640 1 -- 172.21.15.107:0/2496266090 shutdown_connections 2024-05-09T08:41:16.727 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.727+0000 7f01fd1c1640 1 -- 172.21.15.107:0/2496266090 wait complete. 2024-05-09T08:41:16.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.728+0000 7f01fd1c1640 1 Processor -- start 2024-05-09T08:41:16.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.728+0000 7f01fd1c1640 1 -- start start 2024-05-09T08:41:16.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.728+0000 7f01fd1c1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01f8100760 0x7f01f819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.728+0000 7f01fd1c1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f8101120 0x7f01f819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.728+0000 7f01fd1c1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01f8106720 0x7f01f819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.728+0000 7f01fd1c1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f01f8113ce0 con 0x7f01f8106720 2024-05-09T08:41:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.728+0000 7f01fd1c1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f01f8113b60 con 0x7f01f8101120 2024-05-09T08:41:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.728+0000 7f01fd1c1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f01f8113e60 con 0x7f01f8100760 2024-05-09T08:41:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.728+0000 7f01f759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01f8106720 0x7f01f819eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.729+0000 7f01f759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01f8106720 0x7f01f819eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58906/0 (socket says 172.21.15.107:58906) 2024-05-09T08:41:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.729+0000 7f01f759e640 1 -- 172.21.15.107:0/4047804768 learned_addr learned my addr 172.21.15.107:0/4047804768 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.729+0000 7f01f659c640 1 --2- 172.21.15.107:0/4047804768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f8101120 0x7f01f819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.729+0000 7f01f6d9d640 1 --2- 172.21.15.107:0/4047804768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01f8100760 0x7f01f819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.729+0000 7f01f759e640 1 -- 172.21.15.107:0/4047804768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01f8100760 msgr2=0x7f01f819a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.729+0000 7f01f759e640 1 --2- 172.21.15.107:0/4047804768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01f8100760 0x7f01f819a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.729+0000 7f01f759e640 1 -- 172.21.15.107:0/4047804768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f8101120 msgr2=0x7f01f819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.729+0000 7f01f759e640 1 --2- 172.21.15.107:0/4047804768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f8101120 0x7f01f819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.729+0000 7f01f759e640 1 -- 172.21.15.107:0/4047804768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f819f230 con 0x7f01f8106720 2024-05-09T08:41:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.729+0000 7f01f6d9d640 1 --2- 172.21.15.107:0/4047804768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01f8100760 0x7f01f819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.729+0000 7f01f759e640 1 --2- 172.21.15.107:0/4047804768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01f8106720 0x7f01f819eb00 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f01f0009fd0 tx=0x7f01f000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.729+0000 7f01dffff640 1 -- 172.21.15.107:0/4047804768 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f0017070 con 0x7f01f8106720 2024-05-09T08:41:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.729+0000 7f01dffff640 1 -- 172.21.15.107:0/4047804768 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01f0003e20 con 0x7f01f8106720 2024-05-09T08:41:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.729+0000 7f01fd1c1640 1 -- 172.21.15.107:0/4047804768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01f819f520 con 0x7f01f8106720 2024-05-09T08:41:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.729+0000 7f01dffff640 1 -- 172.21.15.107:0/4047804768 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f0010460 con 0x7f01f8106720 2024-05-09T08:41:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.729+0000 7f01fd1c1640 1 -- 172.21.15.107:0/4047804768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01f81a6db0 con 0x7f01f8106720 2024-05-09T08:41:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.732+0000 7f01dffff640 1 -- 172.21.15.107:0/4047804768 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f01f0003970 con 0x7f01f8106720 2024-05-09T08:41:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.732+0000 7f01dffff640 1 --2- 172.21.15.107:0/4047804768 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f01e00a2da0 0x7f01e00a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.733+0000 7f01f6d9d640 1 --2- 172.21.15.107:0/4047804768 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f01e00a2da0 0x7f01e00a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.733+0000 7f01dffff640 1 -- 172.21.15.107:0/4047804768 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f01f00cf650 con 0x7f01f8106720 2024-05-09T08:41:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.733+0000 7f01f6d9d640 1 --2- 172.21.15.107:0/4047804768 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f01e00a2da0 0x7f01e00a5260 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f01e8005fd0 tx=0x7f01e80094d0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:16.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01fffa9740 1 Processor -- start 2024-05-09T08:41:16.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01fffa9740 1 -- start start 2024-05-09T08:41:16.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01fffa9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a122450 0x55aa6a122830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01fffa9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a122d70 0x55aa6a12d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01fffa9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a12d630 0x55aa6a12fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01fffa9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55aa6a087340 con 0x55aa6a122450 2024-05-09T08:41:16.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01fffa9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55aa6a02f290 con 0x55aa6a12d630 2024-05-09T08:41:16.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01fffa9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55aa6a083300 con 0x55aa6a122d70 2024-05-09T08:41:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01f6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a122450 0x55aa6a122830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01f6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a122450 0x55aa6a122830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58918/0 (socket says 172.21.15.107:58918) 2024-05-09T08:41:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01f6d9d640 1 -- 172.21.15.107:0/3604932038 learned_addr learned my addr 172.21.15.107:0/3604932038 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01f759e640 1 --2- 172.21.15.107:0/3604932038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a12d630 0x55aa6a12fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01f659c640 1 --2- 172.21.15.107:0/3604932038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a122d70 0x55aa6a12d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01f6d9d640 1 -- 172.21.15.107:0/3604932038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a122d70 msgr2=0x55aa6a12d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01f6d9d640 1 --2- 172.21.15.107:0/3604932038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a122d70 0x55aa6a12d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01f6d9d640 1 -- 172.21.15.107:0/3604932038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a12d630 msgr2=0x55aa6a12fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01f6d9d640 1 --2- 172.21.15.107:0/3604932038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a12d630 0x55aa6a12fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01f6d9d640 1 -- 172.21.15.107:0/3604932038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa6a0538e0 con 0x55aa6a122450 2024-05-09T08:41:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.748+0000 7f01f6d9d640 1 --2- 172.21.15.107:0/3604932038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a122450 0x55aa6a122830 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f01e800eed0 tx=0x7f01e8011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51cccda2726ef2fa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.749+0000 7f01dd7fa640 1 -- 172.21.15.107:0/3604932038 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8005020 con 0x55aa6a122450 2024-05-09T08:41:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.749+0000 7f01dd7fa640 1 -- 172.21.15.107:0/3604932038 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01e8017030 con 0x55aa6a122450 2024-05-09T08:41:16.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.749+0000 7f01dd7fa640 1 -- 172.21.15.107:0/3604932038 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e801e440 con 0x55aa6a122450 2024-05-09T08:41:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.749+0000 7f01fffa9740 1 -- 172.21.15.107:0/3604932038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a122450 msgr2=0x55aa6a122830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.749+0000 7f01fffa9740 1 --2- 172.21.15.107:0/3604932038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a122450 0x55aa6a122830 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f01e800eed0 tx=0x7f01e8011000 comp rx=0 tx=0).stop 2024-05-09T08:41:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.750+0000 7f01fffa9740 1 -- 172.21.15.107:0/3604932038 shutdown_connections 2024-05-09T08:41:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.750+0000 7f01fffa9740 1 --2- 172.21.15.107:0/3604932038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a12d630 0x55aa6a12fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.750+0000 7f01fffa9740 1 --2- 172.21.15.107:0/3604932038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a122d70 0x55aa6a12d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.750+0000 7f01fffa9740 1 --2- 172.21.15.107:0/3604932038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a122450 0x55aa6a122830 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.750+0000 7f01fffa9740 1 -- 172.21.15.107:0/3604932038 >> 172.21.15.107:0/3604932038 conn(0x55aa6a11e960 msgr2=0x55aa6a11ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.750+0000 7f01fffa9740 1 -- 172.21.15.107:0/3604932038 shutdown_connections 2024-05-09T08:41:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.750+0000 7f01fffa9740 1 -- 172.21.15.107:0/3604932038 wait complete. 2024-05-09T08:41:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.750+0000 7f01fffa9740 1 -- start start 2024-05-09T08:41:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.750+0000 7f01fffa9740 1 Processor -- start 2024-05-09T08:41:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.750+0000 7f01fffa9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a122450 0x55aa6a12a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.750+0000 7f01fffa9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a122d70 0x55aa6a12aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.750+0000 7f01fffa9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a12d630 0x55aa6a12af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.750+0000 7f01fffa9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55aa6a130200 con 0x55aa6a12d630 2024-05-09T08:41:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.750+0000 7f01fffa9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55aa6a042790 con 0x55aa6a122d70 2024-05-09T08:41:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.750+0000 7f01fffa9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55aa6a130380 con 0x55aa6a122450 2024-05-09T08:41:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.750+0000 7f01fffa9740 10 client.0 _ll_register_callbacks cb 0x55aa6a07da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01f759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a12d630 0x55aa6a12af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01f6d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a122450 0x55aa6a12a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01f759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a12d630 0x55aa6a12af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58932/0 (socket says 172.21.15.107:58932) 2024-05-09T08:41:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01f6d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a122450 0x55aa6a12a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:35744/0 (socket says 172.21.15.107:35744) 2024-05-09T08:41:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01f759e640 1 -- 172.21.15.107:0/3811677198 learned_addr learned my addr 172.21.15.107:0/3811677198 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01f659c640 1 --2- 172.21.15.107:0/3811677198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a122d70 0x55aa6a12aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01f759e640 1 -- 172.21.15.107:0/3811677198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a122450 msgr2=0x55aa6a12a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01f759e640 1 --2- 172.21.15.107:0/3811677198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a122450 0x55aa6a12a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01f759e640 1 -- 172.21.15.107:0/3811677198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a122d70 msgr2=0x55aa6a12aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01f759e640 1 --2- 172.21.15.107:0/3811677198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a122d70 0x55aa6a12aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01f759e640 1 -- 172.21.15.107:0/3811677198 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa69d9b210 con 0x55aa6a12d630 2024-05-09T08:41:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01f759e640 1 --2- 172.21.15.107:0/3811677198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a12d630 0x55aa6a12af70 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f01f000b760 tx=0x7f01f0003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a979b41e4475ca9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01daff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01daff5640 1 -- 172.21.15.107:0/3811677198 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f0020040 con 0x55aa6a12d630 2024-05-09T08:41:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01daff5640 1 -- 172.21.15.107:0/3811677198 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f01f00b4660 con 0x55aa6a12d630 2024-05-09T08:41:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01fffa9740 1 -- 172.21.15.107:0/3811677198 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01e8017030 con 0x55aa6a12d630 2024-05-09T08:41:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01daff5640 1 -- 172.21.15.107:0/3811677198 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f01f00b9050 con 0x55aa6a12d630 2024-05-09T08:41:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01fffa9740 10 client.17979 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.751+0000 7f01fffa9740 1 -- 172.21.15.107:0/3811677198 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55aa69e67620 con 0x55aa6a12d630 2024-05-09T08:41:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.752+0000 7f01daff5640 1 -- 172.21.15.107:0/3811677198 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01e8017030 con 0x55aa6a12d630 2024-05-09T08:41:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.752+0000 7f01daff5640 1 -- 172.21.15.107:0/3811677198 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f01f00afce0 con 0x55aa6a12d630 2024-05-09T08:41:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.752+0000 7f01fffa9740 10 client.17979 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.752+0000 7f01fffa9740 10 client.17979 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.752+0000 7f01fffa9740 20 client.17979 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.752+0000 7f01fffa9740 1 --2- 172.21.15.107:0/3811677198 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55aa6a13d250 0x55aa6a13f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.752+0000 7f01fffa9740 4 client.17979 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.752+0000 7f01fffa9740 1 -- 172.21.15.107:0/3811677198 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa6a094a70 con 0x55aa6a13d250 2024-05-09T08:41:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.752+0000 7f01f6d9d640 1 --2- 172.21.15.107:0/3811677198 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55aa6a13d250 0x55aa6a13f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.753+0000 7f01f6d9d640 1 --2- 172.21.15.107:0/3811677198 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55aa6a13d250 0x55aa6a13f640 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0e2ebe2c05121d8 server_cookie=f4457a7729c41818 in_seq=0 out_seq=0 2024-05-09T08:41:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.753+0000 7f01daff5640 10 client.17979 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.753+0000 7f01daff5640 1 -- 172.21.15.107:0/3811677198 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aa6a094a70 con 0x55aa6a13d250 2024-05-09T08:41:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.753+0000 7f01daff5640 10 client.17979 handle_command_reply: tid=0 2024-05-09T08:41:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.753+0000 7f01fffa9740 1 client.17979 shutdown 2024-05-09T08:41:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.753+0000 7f01fffa9740 1 -- 172.21.15.107:0/3811677198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a12d630 msgr2=0x55aa6a12af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.753+0000 7f01fffa9740 1 --2- 172.21.15.107:0/3811677198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a12d630 0x55aa6a12af70 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f01f000b760 tx=0x7f01f0003ce0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.753+0000 7f01fffa9740 1 -- 172.21.15.107:0/3811677198 shutdown_connections 2024-05-09T08:41:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.753+0000 7f01fffa9740 1 --2- 172.21.15.107:0/3811677198 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55aa6a13d250 0x55aa6a13f640 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.753+0000 7f01fffa9740 1 --2- 172.21.15.107:0/3811677198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a12d630 0x55aa6a12af70 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.753+0000 7f01daff5640 0 client.17979 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.753+0000 7f01fffa9740 1 --2- 172.21.15.107:0/3811677198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a122d70 0x55aa6a12aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.754+0000 7f01fffa9740 1 --2- 172.21.15.107:0/3811677198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a122450 0x55aa6a12a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.754+0000 7f01fffa9740 1 -- 172.21.15.107:0/3811677198 >> 172.21.15.107:0/3811677198 conn(0x55aa6a11e960 msgr2=0x55aa6a124690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.754+0000 7f01fffa9740 1 -- 172.21.15.107:0/3811677198 shutdown_connections 2024-05-09T08:41:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.754+0000 7f01fffa9740 1 -- 172.21.15.107:0/3811677198 wait complete. 2024-05-09T08:41:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.754+0000 7f01fffa9740 20 client.17979 trim_cache size 0 max 16384 2024-05-09T08:41:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.768+0000 7f01fffa9740 1 Processor -- start 2024-05-09T08:41:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.768+0000 7f01fffa9740 1 -- start start 2024-05-09T08:41:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.768+0000 7f01fffa9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a122450 0x55aa6a1390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.768+0000 7f01fffa9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a122d70 0x55aa6a124e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.768+0000 7f01fffa9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a12d630 0x55aa6a144eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.768+0000 7f01fffa9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f01f81b3710 con 0x55aa6a12d630 2024-05-09T08:41:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.768+0000 7f01fffa9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f01f81b0fe0 con 0x55aa6a122d70 2024-05-09T08:41:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.768+0000 7f01fffa9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f01f81aee50 con 0x55aa6a122450 2024-05-09T08:41:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.768+0000 7f01f759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a12d630 0x55aa6a144eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.768+0000 7f01f759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a12d630 0x55aa6a144eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58944/0 (socket says 172.21.15.107:58944) 2024-05-09T08:41:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.768+0000 7f01f759e640 1 -- 172.21.15.107:0/3704923360 learned_addr learned my addr 172.21.15.107:0/3704923360 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.769+0000 7f01f6d9d640 1 --2- 172.21.15.107:0/3704923360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a122450 0x55aa6a1390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.769+0000 7f01f659c640 1 --2- 172.21.15.107:0/3704923360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a122d70 0x55aa6a124e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.769+0000 7f01f759e640 1 -- 172.21.15.107:0/3704923360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a122450 msgr2=0x55aa6a1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.769+0000 7f01f759e640 1 --2- 172.21.15.107:0/3704923360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a122450 0x55aa6a1390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.769+0000 7f01f759e640 1 -- 172.21.15.107:0/3704923360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a122d70 msgr2=0x55aa6a124e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.769+0000 7f01f759e640 1 --2- 172.21.15.107:0/3704923360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a122d70 0x55aa6a124e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.769+0000 7f01f759e640 1 -- 172.21.15.107:0/3704923360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa6a13f9b0 con 0x55aa6a12d630 2024-05-09T08:41:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.769+0000 7f01f759e640 1 --2- 172.21.15.107:0/3704923360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a12d630 0x55aa6a144eb0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f01f0099a70 tx=0x7f01f009b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999b81c27a5c72d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.769+0000 7f01dcff9640 1 -- 172.21.15.107:0/3704923360 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f009bec0 con 0x55aa6a12d630 2024-05-09T08:41:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.769+0000 7f01dcff9640 1 -- 172.21.15.107:0/3704923360 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01f0015070 con 0x55aa6a12d630 2024-05-09T08:41:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.769+0000 7f01dcff9640 1 -- 172.21.15.107:0/3704923360 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f009caf0 con 0x55aa6a12d630 2024-05-09T08:41:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.770+0000 7f01fffa9740 1 -- 172.21.15.107:0/3704923360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a12d630 msgr2=0x55aa6a144eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.770+0000 7f01fffa9740 1 --2- 172.21.15.107:0/3704923360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a12d630 0x55aa6a144eb0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f01f0099a70 tx=0x7f01f009b940 comp rx=0 tx=0).stop 2024-05-09T08:41:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.770+0000 7f01fffa9740 1 -- 172.21.15.107:0/3704923360 shutdown_connections 2024-05-09T08:41:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.770+0000 7f01fffa9740 1 --2- 172.21.15.107:0/3704923360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a12d630 0x55aa6a144eb0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.770+0000 7f01fffa9740 1 --2- 172.21.15.107:0/3704923360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a122d70 0x55aa6a124e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.770+0000 7f01fffa9740 1 --2- 172.21.15.107:0/3704923360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a122450 0x55aa6a1390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.770+0000 7f01fffa9740 1 -- 172.21.15.107:0/3704923360 >> 172.21.15.107:0/3704923360 conn(0x55aa6a11e960 msgr2=0x55aa6a11da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.770+0000 7f01fffa9740 1 -- 172.21.15.107:0/3704923360 shutdown_connections 2024-05-09T08:41:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.770+0000 7f01fffa9740 1 -- 172.21.15.107:0/3704923360 wait complete. 2024-05-09T08:41:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.770+0000 7f01fffa9740 1 -- start start 2024-05-09T08:41:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.770+0000 7f01fffa9740 1 Processor -- start 2024-05-09T08:41:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01fffa9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a122450 0x55aa6a1312d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01fffa9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a122d70 0x55aa6a131810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01fffa9740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a12d630 0x55aa6a131d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01fffa9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f01f81af4a0 con 0x55aa6a122450 2024-05-09T08:41:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01fffa9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f01f81b0910 con 0x55aa6a122d70 2024-05-09T08:41:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01fffa9740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f01f81afb70 con 0x55aa6a12d630 2024-05-09T08:41:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01f6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a122450 0x55aa6a1312d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01fffa9740 10 client.0 _ll_register_callbacks cb 0x7f01f00973a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01f6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a122450 0x55aa6a1312d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58956/0 (socket says 172.21.15.107:58956) 2024-05-09T08:41:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01f6d9d640 1 -- 172.21.15.107:0/1828513230 learned_addr learned my addr 172.21.15.107:0/1828513230 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01f759e640 1 --2- 172.21.15.107:0/1828513230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a12d630 0x55aa6a131d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01f659c640 1 --2- 172.21.15.107:0/1828513230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a122d70 0x55aa6a131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01f6d9d640 1 -- 172.21.15.107:0/1828513230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a12d630 msgr2=0x55aa6a131d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01f6d9d640 1 --2- 172.21.15.107:0/1828513230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a12d630 0x55aa6a131d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01f6d9d640 1 -- 172.21.15.107:0/1828513230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a122d70 msgr2=0x55aa6a131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01f6d9d640 1 --2- 172.21.15.107:0/1828513230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a122d70 0x55aa6a131810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01f6d9d640 1 -- 172.21.15.107:0/1828513230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa6a1a2a10 con 0x55aa6a122450 2024-05-09T08:41:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01f759e640 1 --2- 172.21.15.107:0/1828513230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a12d630 0x55aa6a131d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01f6d9d640 1 --2- 172.21.15.107:0/1828513230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a122450 0x55aa6a1312d0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f01e800ee70 tx=0x7f01e8004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b150ef45abd87da9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01daff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01daff5640 1 -- 172.21.15.107:0/1828513230 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8035030 con 0x55aa6a122450 2024-05-09T08:41:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01fffa9740 1 -- 172.21.15.107:0/1828513230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01f0015070 con 0x55aa6a122450 2024-05-09T08:41:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.771+0000 7f01daff5640 1 -- 172.21.15.107:0/1828513230 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f01e8023040 con 0x55aa6a122450 2024-05-09T08:41:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.772+0000 7f01daff5640 1 -- 172.21.15.107:0/1828513230 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f01e8041050 con 0x55aa6a122450 2024-05-09T08:41:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.772+0000 7f01fffa9740 10 client.17985 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.772+0000 7f01fffa9740 1 -- 172.21.15.107:0/1828513230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55aa6a1a3d20 con 0x55aa6a122450 2024-05-09T08:41:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.772+0000 7f01daff5640 1 -- 172.21.15.107:0/1828513230 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01f0015070 con 0x55aa6a122450 2024-05-09T08:41:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.772+0000 7f01daff5640 1 -- 172.21.15.107:0/1828513230 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f01e800f040 con 0x55aa6a122450 2024-05-09T08:41:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.772+0000 7f01fffa9740 10 client.17985 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.772+0000 7f01fffa9740 10 client.17985 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.772+0000 7f01fffa9740 20 client.17985 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.772+0000 7f01fffa9740 1 --2- 172.21.15.107:0/1828513230 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55aa6a13d250 0x55aa6a1a6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.772+0000 7f01fffa9740 4 client.17985 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:41:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.772+0000 7f01fffa9740 1 -- 172.21.15.107:0/1828513230 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aa6a08edc0 con 0x55aa6a13d250 2024-05-09T08:41:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.772+0000 7f01f659c640 1 --2- 172.21.15.107:0/1828513230 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55aa6a13d250 0x55aa6a1a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.773+0000 7f01daff5640 10 client.17985 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.773+0000 7f01f659c640 1 --2- 172.21.15.107:0/1828513230 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55aa6a13d250 0x55aa6a1a6180 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0cb6e51d444ee4e server_cookie=7bcf7c94f6bc0950 in_seq=0 out_seq=0 2024-05-09T08:41:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.791+0000 7f01daff5640 1 -- 172.21.15.107:0/1828513230 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55aa6a08edc0 con 0x55aa6a13d250 2024-05-09T08:41:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.792+0000 7f01daff5640 10 client.17985 handle_command_reply: tid=0 2024-05-09T08:41:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.792+0000 7f01fffa9740 1 client.17985 shutdown 2024-05-09T08:41:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.792+0000 7f01fffa9740 1 -- 172.21.15.107:0/1828513230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a122450 msgr2=0x55aa6a1312d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:16.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.792+0000 7f01fffa9740 1 --2- 172.21.15.107:0/1828513230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a122450 0x55aa6a1312d0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f01e800ee70 tx=0x7f01e8004770 comp rx=0 tx=0).stop 2024-05-09T08:41:16.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.792+0000 7f01fffa9740 1 -- 172.21.15.107:0/1828513230 shutdown_connections 2024-05-09T08:41:16.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.792+0000 7f01fffa9740 1 --2- 172.21.15.107:0/1828513230 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55aa6a13d250 0x55aa6a1a6180 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.792+0000 7f01fffa9740 1 --2- 172.21.15.107:0/1828513230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aa6a12d630 0x55aa6a131d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.792+0000 7f01daff5640 0 client.17985 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:16.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.792+0000 7f01fffa9740 1 --2- 172.21.15.107:0/1828513230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aa6a122d70 0x55aa6a131810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.792+0000 7f01fffa9740 1 --2- 172.21.15.107:0/1828513230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aa6a122450 0x55aa6a1312d0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.792+0000 7f01fffa9740 1 -- 172.21.15.107:0/1828513230 >> 172.21.15.107:0/1828513230 conn(0x55aa6a11e960 msgr2=0x55aa6a12a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:16.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.792+0000 7f01fffa9740 1 -- 172.21.15.107:0/1828513230 shutdown_connections 2024-05-09T08:41:16.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.792+0000 7f01fffa9740 1 -- 172.21.15.107:0/1828513230 wait complete. 2024-05-09T08:41:16.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.792+0000 7f01fffa9740 20 client.17985 trim_cache size 0 max 16384 2024-05-09T08:41:16.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:41:16.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:41:16.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:41:16.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:41:16.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.792+0000 7f01fd1c1640 1 -- 172.21.15.107:0/4047804768 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f01e00a2da0 msgr2=0x7f01e00a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:16.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.792+0000 7f01fd1c1640 1 --2- 172.21.15.107:0/4047804768 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f01e00a2da0 0x7f01e00a5260 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f01e8005fd0 tx=0x7f01e80094d0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.793+0000 7f01fd1c1640 1 -- 172.21.15.107:0/4047804768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01f8106720 msgr2=0x7f01f819eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:16.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.793+0000 7f01fd1c1640 1 --2- 172.21.15.107:0/4047804768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01f8106720 0x7f01f819eb00 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f01f0009fd0 tx=0x7f01f000b260 comp rx=0 tx=0).stop 2024-05-09T08:41:16.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.793+0000 7f01fd1c1640 1 -- 172.21.15.107:0/4047804768 shutdown_connections 2024-05-09T08:41:16.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.793+0000 7f01fd1c1640 1 --2- 172.21.15.107:0/4047804768 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f01e00a2da0 0x7f01e00a5260 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.793+0000 7f01fd1c1640 1 --2- 172.21.15.107:0/4047804768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01f8106720 0x7f01f819eb00 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.793+0000 7f01fd1c1640 1 --2- 172.21.15.107:0/4047804768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f8101120 0x7f01f819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.793+0000 7f01fd1c1640 1 --2- 172.21.15.107:0/4047804768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01f8100760 0x7f01f819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:16.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.793+0000 7f01fd1c1640 1 -- 172.21.15.107:0/4047804768 >> 172.21.15.107:0/4047804768 conn(0x7f01f80778b0 msgr2=0x7f01f8103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:16.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.793+0000 7f01fd1c1640 1 -- 172.21.15.107:0/4047804768 shutdown_connections 2024-05-09T08:41:16.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:16.793+0000 7f01fd1c1640 1 -- 172.21.15.107:0/4047804768 wait complete. 2024-05-09T08:41:18.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:18 smithi107 ceph-mon[30122]: pgmap v245: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 30 MiB/s wr, 25 op/s 2024-05-09T08:41:18.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:18 smithi157 ceph-mon[33614]: pgmap v245: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 30 MiB/s wr, 25 op/s 2024-05-09T08:41:18.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:18 smithi110 ceph-mon[33622]: pgmap v245: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 30 MiB/s wr, 25 op/s 2024-05-09T08:41:20.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:20 smithi107 ceph-mon[30122]: pgmap v246: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-05-09T08:41:20.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:20 smithi157 ceph-mon[33614]: pgmap v246: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-05-09T08:41:20.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:20 smithi110 ceph-mon[33622]: pgmap v246: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-05-09T08:41:22.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:22 smithi107 ceph-mon[30122]: pgmap v247: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-05-09T08:41:22.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:22 smithi157 ceph-mon[33614]: pgmap v247: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-05-09T08:41:22.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:22 smithi110 ceph-mon[33622]: pgmap v247: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-05-09T08:41:24.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:24 smithi107 ceph-mon[30122]: pgmap v248: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-05-09T08:41:24.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:24 smithi157 ceph-mon[33614]: pgmap v248: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-05-09T08:41:24.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:24 smithi110 ceph-mon[33622]: pgmap v248: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-05-09T08:41:26.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:26 smithi107 ceph-mon[30122]: pgmap v249: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:41:26.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:26 smithi157 ceph-mon[33614]: pgmap v249: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:41:26.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:26 smithi110 ceph-mon[33622]: pgmap v249: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:41:28.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:28 smithi157 ceph-mon[33614]: pgmap v250: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-05-09T08:41:28.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:28 smithi110 ceph-mon[33622]: pgmap v250: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-05-09T08:41:28.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:28 smithi107 ceph-mon[30122]: pgmap v250: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 37 MiB/s wr, 28 op/s 2024-05-09T08:41:29.997 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.996+0000 7f57b3212640 1 -- 172.21.15.107:0/3386453867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57ac106760 msgr2=0x7f57ac106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:29.997 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.996+0000 7f57b3212640 1 --2- 172.21.15.107:0/3386453867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57ac106760 0x7f57ac106b40 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f57a0009930 tx=0x7f57a002a420 comp rx=0 tx=0).stop 2024-05-09T08:41:29.997 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.996+0000 7f57b3212640 1 -- 172.21.15.107:0/3386453867 shutdown_connections 2024-05-09T08:41:29.997 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.996+0000 7f57b3212640 1 --2- 172.21.15.107:0/3386453867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57ac101110 0x7f57ac10d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.997 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.996+0000 7f57b3212640 1 --2- 172.21.15.107:0/3386453867 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57ac100750 0x7f57ac100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.997 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.996+0000 7f57b3212640 1 --2- 172.21.15.107:0/3386453867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57ac106760 0x7f57ac106b40 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.997 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.996+0000 7f57b3212640 1 -- 172.21.15.107:0/3386453867 >> 172.21.15.107:0/3386453867 conn(0x7f57ac076ab0 msgr2=0x7f57ac076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:29.997 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.996+0000 7f57b3212640 1 -- 172.21.15.107:0/3386453867 shutdown_connections 2024-05-09T08:41:29.997 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.996+0000 7f57b3212640 1 -- 172.21.15.107:0/3386453867 wait complete. 2024-05-09T08:41:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.996+0000 7f57b3212640 1 Processor -- start 2024-05-09T08:41:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.996+0000 7f57b3212640 1 -- start start 2024-05-09T08:41:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57b3212640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57ac100750 0x7f57ac198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57b3212640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57ac101110 0x7f57ac199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57b3212640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57ac106760 0x7f57ac1a3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57b3212640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac110a20 con 0x7f57ac101110 2024-05-09T08:41:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57b3212640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac1108a0 con 0x7f57ac106760 2024-05-09T08:41:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57b3212640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac110ba0 con 0x7f57ac100750 2024-05-09T08:41:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57ac101110 0x7f57ac199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57ac101110 0x7f57ac199280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58820/0 (socket says 172.21.15.107:58820) 2024-05-09T08:41:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57abfff640 1 -- 172.21.15.107:0/13604812 learned_addr learned my addr 172.21.15.107:0/13604812 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57b0fae640 1 --2- 172.21.15.107:0/13604812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57ac100750 0x7f57ac198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57b17af640 1 --2- 172.21.15.107:0/13604812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57ac106760 0x7f57ac1a3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57abfff640 1 -- 172.21.15.107:0/13604812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57ac100750 msgr2=0x7f57ac198d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57abfff640 1 --2- 172.21.15.107:0/13604812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57ac100750 0x7f57ac198d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57abfff640 1 -- 172.21.15.107:0/13604812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57ac106760 msgr2=0x7f57ac1a3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57abfff640 1 --2- 172.21.15.107:0/13604812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57ac106760 0x7f57ac1a3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57abfff640 1 -- 172.21.15.107:0/13604812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57ac1a4300 con 0x7f57ac101110 2024-05-09T08:41:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57b0fae640 1 --2- 172.21.15.107:0/13604812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57ac100750 0x7f57ac198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57abfff640 1 --2- 172.21.15.107:0/13604812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57ac101110 0x7f57ac199280 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f579800e9f0 tx=0x7f579800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57a9ffb640 1 -- 172.21.15.107:0/13604812 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579800cd80 con 0x7f57ac101110 2024-05-09T08:41:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57a9ffb640 1 -- 172.21.15.107:0/13604812 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57980054d0 con 0x7f57ac101110 2024-05-09T08:41:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57b3212640 1 -- 172.21.15.107:0/13604812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57ac1a4530 con 0x7f57ac101110 2024-05-09T08:41:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.997+0000 7f57a9ffb640 1 -- 172.21.15.107:0/13604812 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5798010490 con 0x7f57ac101110 2024-05-09T08:41:29.999 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:29.998+0000 7f57b3212640 1 -- 172.21.15.107:0/13604812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57ac1a4a50 con 0x7f57ac101110 2024-05-09T08:41:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.000+0000 7f57a9ffb640 1 -- 172.21.15.107:0/13604812 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f5798005020 con 0x7f57ac101110 2024-05-09T08:41:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.000+0000 7f57a9ffb640 1 --2- 172.21.15.107:0/13604812 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f579c0a2e20 0x7f579c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b0fae640 1 --2- 172.21.15.107:0/13604812 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f579c0a2e20 0x7f579c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57a9ffb640 1 -- 172.21.15.107:0/13604812 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f57980cfc40 con 0x7f57ac101110 2024-05-09T08:41:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b5ffc740 1 Processor -- start 2024-05-09T08:41:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b5ffc740 1 -- start start 2024-05-09T08:41:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b5ffc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89ad450 0x5644d89ad830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b5ffc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89add70 0x5644d89b80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b5ffc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89b8630 0x5644d89baa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b5ffc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5644d8912340 con 0x5644d89b8630 2024-05-09T08:41:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b5ffc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5644d88ba290 con 0x5644d89add70 2024-05-09T08:41:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b5ffc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5644d890e300 con 0x5644d89ad450 2024-05-09T08:41:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b0fae640 1 --2- 172.21.15.107:0/13604812 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f579c0a2e20 0x7f579c0a52e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f57a0000c00 tx=0x7f57a0009610 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b17af640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89b8630 0x5644d89baa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b17af640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89b8630 0x5644d89baa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58828/0 (socket says 172.21.15.107:58828) 2024-05-09T08:41:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b17af640 1 -- 172.21.15.107:0/2614197728 learned_addr learned my addr 172.21.15.107:0/2614197728 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57abfff640 1 --2- 172.21.15.107:0/2614197728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89add70 0x5644d89b80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b0fae640 1 --2- 172.21.15.107:0/2614197728 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89ad450 0x5644d89ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b17af640 1 -- 172.21.15.107:0/2614197728 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89ad450 msgr2=0x5644d89ad830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b17af640 1 --2- 172.21.15.107:0/2614197728 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89ad450 0x5644d89ad830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b17af640 1 -- 172.21.15.107:0/2614197728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89add70 msgr2=0x5644d89b80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.001+0000 7f57b17af640 1 --2- 172.21.15.107:0/2614197728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89add70 0x5644d89b80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.002+0000 7f57b17af640 1 -- 172.21.15.107:0/2614197728 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644d88de8e0 con 0x5644d89b8630 2024-05-09T08:41:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.002+0000 7f57b17af640 1 --2- 172.21.15.107:0/2614197728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89b8630 0x5644d89baa20 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f57a4007dd0 tx=0x7f57a400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=736a559e468e99f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.002+0000 7f5796ffd640 1 -- 172.21.15.107:0/2614197728 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a4014070 con 0x5644d89b8630 2024-05-09T08:41:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.002+0000 7f5796ffd640 1 -- 172.21.15.107:0/2614197728 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57a4004d20 con 0x5644d89b8630 2024-05-09T08:41:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.002+0000 7f5796ffd640 1 -- 172.21.15.107:0/2614197728 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a40107b0 con 0x5644d89b8630 2024-05-09T08:41:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.003+0000 7f57b5ffc740 1 -- 172.21.15.107:0/2614197728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89b8630 msgr2=0x5644d89baa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.003+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/2614197728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89b8630 0x5644d89baa20 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f57a4007dd0 tx=0x7f57a400e720 comp rx=0 tx=0).stop 2024-05-09T08:41:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.003+0000 7f57b5ffc740 1 -- 172.21.15.107:0/2614197728 shutdown_connections 2024-05-09T08:41:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.003+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/2614197728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89b8630 0x5644d89baa20 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.003+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/2614197728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89add70 0x5644d89b80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.003+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/2614197728 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89ad450 0x5644d89ad830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.003+0000 7f57b5ffc740 1 -- 172.21.15.107:0/2614197728 >> 172.21.15.107:0/2614197728 conn(0x5644d89a9960 msgr2=0x5644d89a9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.003+0000 7f57b5ffc740 1 -- 172.21.15.107:0/2614197728 shutdown_connections 2024-05-09T08:41:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.003+0000 7f57b5ffc740 1 -- 172.21.15.107:0/2614197728 wait complete. 2024-05-09T08:41:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.003+0000 7f57b5ffc740 1 -- start start 2024-05-09T08:41:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.003+0000 7f57b5ffc740 1 Processor -- start 2024-05-09T08:41:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57b5ffc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89ad450 0x5644d89b5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57b5ffc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89add70 0x5644d89b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57b5ffc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89b8630 0x5644d89b6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57b5ffc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5644d89bb260 con 0x5644d89add70 2024-05-09T08:41:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57b5ffc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5644d88cd790 con 0x5644d89ad450 2024-05-09T08:41:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57b5ffc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5644d89bb3e0 con 0x5644d89b8630 2024-05-09T08:41:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57b5ffc740 10 client.0 _ll_register_callbacks cb 0x5644d8908a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89add70 0x5644d89b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89add70 0x5644d89b5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58836/0 (socket says 172.21.15.107:58836) 2024-05-09T08:41:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57abfff640 1 -- 172.21.15.107:0/997850681 learned_addr learned my addr 172.21.15.107:0/997850681 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57b0fae640 1 --2- 172.21.15.107:0/997850681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89ad450 0x5644d89b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57b17af640 1 --2- 172.21.15.107:0/997850681 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89b8630 0x5644d89b6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57abfff640 1 -- 172.21.15.107:0/997850681 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89b8630 msgr2=0x5644d89b6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57abfff640 1 --2- 172.21.15.107:0/997850681 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89b8630 0x5644d89b6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57abfff640 1 -- 172.21.15.107:0/997850681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89ad450 msgr2=0x5644d89b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57abfff640 1 --2- 172.21.15.107:0/997850681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89ad450 0x5644d89b5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57abfff640 1 -- 172.21.15.107:0/997850681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644d8626210 con 0x5644d89add70 2024-05-09T08:41:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57b17af640 1 --2- 172.21.15.107:0/997850681 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89b8630 0x5644d89b6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57abfff640 1 --2- 172.21.15.107:0/997850681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89add70 0x5644d89b5b50 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f579809bd40 tx=0x7f5798005450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f3c6c84330c80c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f578bfff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f578bfff640 1 -- 172.21.15.107:0/997850681 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57980a0160 con 0x5644d89add70 2024-05-09T08:41:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f578bfff640 1 -- 172.21.15.107:0/997850681 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f5798098420 con 0x5644d89add70 2024-05-09T08:41:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.004+0000 7f57b5ffc740 1 -- 172.21.15.107:0/997850681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57a4004d20 con 0x5644d89add70 2024-05-09T08:41:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.005+0000 7f578bfff640 1 -- 172.21.15.107:0/997850681 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f57980a4a20 con 0x5644d89add70 2024-05-09T08:41:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.005+0000 7f57b5ffc740 10 client.17997 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.005+0000 7f57b5ffc740 1 -- 172.21.15.107:0/997850681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5644d86f2620 con 0x5644d89add70 2024-05-09T08:41:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.005+0000 7f578bfff640 1 -- 172.21.15.107:0/997850681 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57a4004d20 con 0x5644d89add70 2024-05-09T08:41:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.005+0000 7f578bfff640 1 -- 172.21.15.107:0/997850681 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5798086040 con 0x5644d89add70 2024-05-09T08:41:30.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.005+0000 7f57b5ffc740 10 client.17997 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:30.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.005+0000 7f57b5ffc740 10 client.17997 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:30.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.005+0000 7f57b5ffc740 20 client.17997 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:30.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.005+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/997850681 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5644d89c83e0 0x5644d89ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:30.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.005+0000 7f57b5ffc740 4 client.17997 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:30.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.005+0000 7f57b5ffc740 1 -- 172.21.15.107:0/997850681 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644d891fa70 con 0x5644d89c83e0 2024-05-09T08:41:30.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.005+0000 7f57b0fae640 1 --2- 172.21.15.107:0/997850681 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5644d89c83e0 0x5644d89ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:30.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.006+0000 7f57b0fae640 1 --2- 172.21.15.107:0/997850681 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5644d89c83e0 0x5644d89ca7d0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97635e0a9bdaf55d server_cookie=ca6362875c2931c2 in_seq=0 out_seq=0 2024-05-09T08:41:30.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.006+0000 7f578bfff640 10 client.17997 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:30.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.006+0000 7f578bfff640 1 -- 172.21.15.107:0/997850681 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5644d891fa70 con 0x5644d89c83e0 2024-05-09T08:41:30.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.007+0000 7f578bfff640 10 client.17997 handle_command_reply: tid=0 2024-05-09T08:41:30.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.007+0000 7f57b5ffc740 1 client.17997 shutdown 2024-05-09T08:41:30.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.007+0000 7f57b5ffc740 1 -- 172.21.15.107:0/997850681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89add70 msgr2=0x5644d89b5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:30.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.007+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/997850681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89add70 0x5644d89b5b50 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f579809bd40 tx=0x7f5798005450 comp rx=0 tx=0).stop 2024-05-09T08:41:30.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.007+0000 7f57b5ffc740 1 -- 172.21.15.107:0/997850681 shutdown_connections 2024-05-09T08:41:30.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.007+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/997850681 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5644d89c83e0 0x5644d89ca7d0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.007+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/997850681 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89b8630 0x5644d89b6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.007+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/997850681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89add70 0x5644d89b5b50 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.007+0000 7f578bfff640 0 client.17997 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:30.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.007+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/997850681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89ad450 0x5644d89b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.007+0000 7f57b5ffc740 1 -- 172.21.15.107:0/997850681 >> 172.21.15.107:0/997850681 conn(0x5644d89a9960 msgr2=0x5644d89af690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:30.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.007+0000 7f57b5ffc740 1 -- 172.21.15.107:0/997850681 shutdown_connections 2024-05-09T08:41:30.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.007+0000 7f57b5ffc740 1 -- 172.21.15.107:0/997850681 wait complete. 2024-05-09T08:41:30.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.007+0000 7f57b5ffc740 20 client.17997 trim_cache size 0 max 16384 2024-05-09T08:41:30.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.029+0000 7f57b5ffc740 1 Processor -- start 2024-05-09T08:41:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.029+0000 7f57b5ffc740 1 -- start start 2024-05-09T08:41:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.029+0000 7f57b5ffc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89ad450 0x5644d89c4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.029+0000 7f57b5ffc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89add70 0x5644d89afe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.029+0000 7f57b5ffc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89b8630 0x5644d89b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.029+0000 7f57b5ffc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac1b1320 con 0x5644d89ad450 2024-05-09T08:41:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.029+0000 7f57b5ffc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac1aebf0 con 0x5644d89add70 2024-05-09T08:41:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.029+0000 7f57b5ffc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac1aca60 con 0x5644d89b8630 2024-05-09T08:41:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.029+0000 7f57b0fae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89ad450 0x5644d89c4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.029+0000 7f57b0fae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89ad450 0x5644d89c4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58850/0 (socket says 172.21.15.107:58850) 2024-05-09T08:41:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.029+0000 7f57b0fae640 1 -- 172.21.15.107:0/1435343703 learned_addr learned my addr 172.21.15.107:0/1435343703 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.030+0000 7f57b17af640 1 --2- 172.21.15.107:0/1435343703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89b8630 0x5644d89b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.030+0000 7f57abfff640 1 --2- 172.21.15.107:0/1435343703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89add70 0x5644d89afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.030+0000 7f57b0fae640 1 -- 172.21.15.107:0/1435343703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89b8630 msgr2=0x5644d89b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.030+0000 7f57b0fae640 1 --2- 172.21.15.107:0/1435343703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89b8630 0x5644d89b5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.030+0000 7f57b0fae640 1 -- 172.21.15.107:0/1435343703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89add70 msgr2=0x5644d89afe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.030+0000 7f57b0fae640 1 --2- 172.21.15.107:0/1435343703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89add70 0x5644d89afe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.030+0000 7f57b0fae640 1 -- 172.21.15.107:0/1435343703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644d89b45c0 con 0x5644d89ad450 2024-05-09T08:41:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.030+0000 7f57abfff640 1 --2- 172.21.15.107:0/1435343703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89add70 0x5644d89afe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.030+0000 7f57b0fae640 1 --2- 172.21.15.107:0/1435343703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89ad450 0x5644d89c4120 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f57a002a940 tx=0x7f57a003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1333ee71203ba70 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.030+0000 7f57967fc640 1 -- 172.21.15.107:0/1435343703 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0056070 con 0x5644d89ad450 2024-05-09T08:41:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.030+0000 7f57967fc640 1 -- 172.21.15.107:0/1435343703 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57a0005040 con 0x5644d89ad450 2024-05-09T08:41:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.030+0000 7f57967fc640 1 -- 172.21.15.107:0/1435343703 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0035050 con 0x5644d89ad450 2024-05-09T08:41:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.031+0000 7f57b5ffc740 1 -- 172.21.15.107:0/1435343703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89ad450 msgr2=0x5644d89c4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.031+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/1435343703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89ad450 0x5644d89c4120 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f57a002a940 tx=0x7f57a003f000 comp rx=0 tx=0).stop 2024-05-09T08:41:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.031+0000 7f57b5ffc740 1 -- 172.21.15.107:0/1435343703 shutdown_connections 2024-05-09T08:41:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.031+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/1435343703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89b8630 0x5644d89b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.031+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/1435343703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89add70 0x5644d89afe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.031+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/1435343703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89ad450 0x5644d89c4120 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.031+0000 7f57b5ffc740 1 -- 172.21.15.107:0/1435343703 >> 172.21.15.107:0/1435343703 conn(0x5644d89a9960 msgr2=0x5644d89a9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.031+0000 7f57b5ffc740 1 -- 172.21.15.107:0/1435343703 shutdown_connections 2024-05-09T08:41:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.031+0000 7f57b5ffc740 1 -- 172.21.15.107:0/1435343703 wait complete. 2024-05-09T08:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.031+0000 7f57b5ffc740 1 -- start start 2024-05-09T08:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.031+0000 7f57b5ffc740 1 Processor -- start 2024-05-09T08:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57b5ffc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89ad450 0x5644d89b2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57b5ffc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89add70 0x5644d89b2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57b5ffc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89b8630 0x5644d89b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57b5ffc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac1ad0b0 con 0x5644d89add70 2024-05-09T08:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57b5ffc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac1ae520 con 0x5644d89b8630 2024-05-09T08:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57b5ffc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac1ad780 con 0x5644d89ad450 2024-05-09T08:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57b5ffc740 10 client.0 _ll_register_callbacks cb 0x7f579800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89add70 0x5644d89b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57b0fae640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89ad450 0x5644d89b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89add70 0x5644d89b2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58854/0 (socket says 172.21.15.107:58854) 2024-05-09T08:41:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57b0fae640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89ad450 0x5644d89b2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:56332/0 (socket says 172.21.15.107:56332) 2024-05-09T08:41:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57abfff640 1 -- 172.21.15.107:0/1405992851 learned_addr learned my addr 172.21.15.107:0/1405992851 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57b17af640 1 --2- 172.21.15.107:0/1405992851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89b8630 0x5644d89b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57abfff640 1 -- 172.21.15.107:0/1405992851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89ad450 msgr2=0x5644d89b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57abfff640 1 --2- 172.21.15.107:0/1405992851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89ad450 0x5644d89b2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57abfff640 1 -- 172.21.15.107:0/1405992851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89b8630 msgr2=0x5644d89b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57abfff640 1 --2- 172.21.15.107:0/1405992851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89b8630 0x5644d89b2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57abfff640 1 -- 172.21.15.107:0/1405992851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644d8a2d940 con 0x5644d89add70 2024-05-09T08:41:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57b17af640 1 --2- 172.21.15.107:0/1405992851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89b8630 0x5644d89b2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f57abfff640 1 --2- 172.21.15.107:0/1405992851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89add70 0x5644d89b2850 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f5798098780 tx=0x7f5798016fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd4001ea7595b2a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.032+0000 7f578bfff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.033+0000 7f578bfff640 1 -- 172.21.15.107:0/1405992851 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5798098050 con 0x5644d89add70 2024-05-09T08:41:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.033+0000 7f578bfff640 1 -- 172.21.15.107:0/1405992851 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f579809afb0 con 0x5644d89add70 2024-05-09T08:41:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.033+0000 7f57b5ffc740 1 -- 172.21.15.107:0/1405992851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57a0005040 con 0x5644d89add70 2024-05-09T08:41:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.033+0000 7f578bfff640 1 -- 172.21.15.107:0/1405992851 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f57980a0160 con 0x5644d89add70 2024-05-09T08:41:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.033+0000 7f57b5ffc740 10 client.18003 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.033+0000 7f57b5ffc740 1 -- 172.21.15.107:0/1405992851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5644d8a2eb80 con 0x5644d89add70 2024-05-09T08:41:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.033+0000 7f578bfff640 1 -- 172.21.15.107:0/1405992851 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57a0005040 con 0x5644d89add70 2024-05-09T08:41:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.033+0000 7f578bfff640 1 -- 172.21.15.107:0/1405992851 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5798086040 con 0x5644d89add70 2024-05-09T08:41:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.033+0000 7f57b5ffc740 10 client.18003 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.033+0000 7f57b5ffc740 10 client.18003 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.033+0000 7f57b5ffc740 20 client.18003 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.033+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/1405992851 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5644d8a2f130 0x5644d8a31550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.033+0000 7f57b5ffc740 4 client.18003 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:41:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.033+0000 7f57b5ffc740 1 -- 172.21.15.107:0/1405992851 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5644d8919dc0 con 0x5644d8a2f130 2024-05-09T08:41:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.034+0000 7f57b0fae640 1 --2- 172.21.15.107:0/1405992851 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5644d8a2f130 0x5644d8a31550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.034+0000 7f578bfff640 10 client.18003 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.034+0000 7f57b0fae640 1 --2- 172.21.15.107:0/1405992851 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5644d8a2f130 0x5644d8a31550 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8773aebfd9ded856 server_cookie=4fd1646438e78960 in_seq=0 out_seq=0 2024-05-09T08:41:30.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.051+0000 7f578bfff640 1 -- 172.21.15.107:0/1405992851 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5644d8919dc0 con 0x5644d8a2f130 2024-05-09T08:41:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.051+0000 7f578bfff640 10 client.18003 handle_command_reply: tid=0 2024-05-09T08:41:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.051+0000 7f57b5ffc740 1 client.18003 shutdown 2024-05-09T08:41:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.051+0000 7f57b5ffc740 1 -- 172.21.15.107:0/1405992851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89add70 msgr2=0x5644d89b2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.051+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/1405992851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89add70 0x5644d89b2850 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f5798098780 tx=0x7f5798016fc0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.051+0000 7f57b5ffc740 1 -- 172.21.15.107:0/1405992851 shutdown_connections 2024-05-09T08:41:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.051+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/1405992851 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5644d8a2f130 0x5644d8a31550 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.051+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/1405992851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5644d89b8630 0x5644d89b2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.051+0000 7f578bfff640 0 client.18003 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.051+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/1405992851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5644d89add70 0x5644d89b2850 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.051+0000 7f57b5ffc740 1 --2- 172.21.15.107:0/1405992851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5644d89ad450 0x5644d89b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.051+0000 7f57b5ffc740 1 -- 172.21.15.107:0/1405992851 >> 172.21.15.107:0/1405992851 conn(0x5644d89a9960 msgr2=0x5644d89b5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.051+0000 7f57b5ffc740 1 -- 172.21.15.107:0/1405992851 shutdown_connections 2024-05-09T08:41:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.051+0000 7f57b5ffc740 1 -- 172.21.15.107:0/1405992851 wait complete. 2024-05-09T08:41:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.051+0000 7f57b5ffc740 20 client.18003 trim_cache size 0 max 16384 2024-05-09T08:41:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:41:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:41:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:41:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:41:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.052+0000 7f57b3212640 1 -- 172.21.15.107:0/13604812 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f579c0a2e20 msgr2=0x7f579c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.052+0000 7f57b3212640 1 --2- 172.21.15.107:0/13604812 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f579c0a2e20 0x7f579c0a52e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f57a0000c00 tx=0x7f57a0009610 comp rx=0 tx=0).stop 2024-05-09T08:41:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.052+0000 7f57b3212640 1 -- 172.21.15.107:0/13604812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57ac101110 msgr2=0x7f57ac199280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.052+0000 7f57b3212640 1 --2- 172.21.15.107:0/13604812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57ac101110 0x7f57ac199280 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f579800e9f0 tx=0x7f579800edf0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.052+0000 7f57b3212640 1 -- 172.21.15.107:0/13604812 shutdown_connections 2024-05-09T08:41:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.052+0000 7f57b3212640 1 --2- 172.21.15.107:0/13604812 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f579c0a2e20 0x7f579c0a52e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.052+0000 7f57b3212640 1 --2- 172.21.15.107:0/13604812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57ac106760 0x7f57ac1a3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.052+0000 7f57b3212640 1 --2- 172.21.15.107:0/13604812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57ac101110 0x7f57ac199280 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.052+0000 7f57b3212640 1 --2- 172.21.15.107:0/13604812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57ac100750 0x7f57ac198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.052+0000 7f57b3212640 1 -- 172.21.15.107:0/13604812 >> 172.21.15.107:0/13604812 conn(0x7f57ac076ab0 msgr2=0x7f57ac0fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.052+0000 7f57b3212640 1 -- 172.21.15.107:0/13604812 shutdown_connections 2024-05-09T08:41:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:30.052+0000 7f57b3212640 1 -- 172.21.15.107:0/13604812 wait complete. 2024-05-09T08:41:30.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:30 smithi157 ceph-mon[33614]: pgmap v251: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 24 MiB/s wr, 15 op/s 2024-05-09T08:41:30.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:30 smithi110 ceph-mon[33622]: pgmap v251: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 24 MiB/s wr, 15 op/s 2024-05-09T08:41:30.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:30 smithi107 ceph-mon[30122]: pgmap v251: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 24 MiB/s wr, 15 op/s 2024-05-09T08:41:32.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:32 smithi157 ceph-mon[33614]: pgmap v252: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 24 MiB/s wr, 15 op/s 2024-05-09T08:41:32.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:32 smithi110 ceph-mon[33622]: pgmap v252: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 24 MiB/s wr, 15 op/s 2024-05-09T08:41:32.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:32 smithi107 ceph-mon[30122]: pgmap v252: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 24 MiB/s wr, 15 op/s 2024-05-09T08:41:32.914 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:41:32.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:41:33.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.659+0000 7fc314810640 1 -- 172.21.15.107:0/309157063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc310100700 msgr2=0x7fc310100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:33.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.659+0000 7fc314810640 1 --2- 172.21.15.107:0/309157063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc310100700 0x7fc310100b80 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8009930 tx=0x7fc2f802a420 comp rx=0 tx=0).stop 2024-05-09T08:41:33.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.659+0000 7fc314810640 1 -- 172.21.15.107:0/309157063 shutdown_connections 2024-05-09T08:41:33.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.659+0000 7fc314810640 1 --2- 172.21.15.107:0/309157063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3101010c0 0x7fc31010d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.659+0000 7fc314810640 1 --2- 172.21.15.107:0/309157063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc310100700 0x7fc310100b80 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.659+0000 7fc314810640 1 --2- 172.21.15.107:0/309157063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc310106710 0x7fc310106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.659+0000 7fc314810640 1 -- 172.21.15.107:0/309157063 >> 172.21.15.107:0/309157063 conn(0x7fc310077830 msgr2=0x7fc310077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:33.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.660+0000 7fc314810640 1 -- 172.21.15.107:0/309157063 shutdown_connections 2024-05-09T08:41:33.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.660+0000 7fc314810640 1 -- 172.21.15.107:0/309157063 wait complete. 2024-05-09T08:41:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.660+0000 7fc314810640 1 Processor -- start 2024-05-09T08:41:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc314810640 1 -- start start 2024-05-09T08:41:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc314810640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc310100700 0x7fc310198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc314810640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3101010c0 0x7fc3101985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc314810640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc310106710 0x7fc31019c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc314810640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc310111ae0 con 0x7fc310100700 2024-05-09T08:41:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc314810640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc310111960 con 0x7fc3101010c0 2024-05-09T08:41:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc314810640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc310111c60 con 0x7fc310106710 2024-05-09T08:41:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc30e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc310100700 0x7fc310198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc30e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc310100700 0x7fc310198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57822/0 (socket says 172.21.15.107:57822) 2024-05-09T08:41:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc30e59c640 1 -- 172.21.15.107:0/977096197 learned_addr learned my addr 172.21.15.107:0/977096197 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc305d9b640 1 --2- 172.21.15.107:0/977096197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3101010c0 0x7fc3101985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc30ed9d640 1 --2- 172.21.15.107:0/977096197 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc310106710 0x7fc31019c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc30e59c640 1 -- 172.21.15.107:0/977096197 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc310106710 msgr2=0x7fc31019c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc30e59c640 1 --2- 172.21.15.107:0/977096197 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc310106710 0x7fc31019c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc30e59c640 1 -- 172.21.15.107:0/977096197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3101010c0 msgr2=0x7fc3101985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc30e59c640 1 --2- 172.21.15.107:0/977096197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3101010c0 0x7fc3101985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc30e59c640 1 -- 172.21.15.107:0/977096197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31019d050 con 0x7fc310100700 2024-05-09T08:41:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.661+0000 7fc30ed9d640 1 --2- 172.21.15.107:0/977096197 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc310106710 0x7fc31019c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.662+0000 7fc30e59c640 1 --2- 172.21.15.107:0/977096197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc310100700 0x7fc310198060 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fc30000d770 tx=0x7fc30000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.662+0000 7fc307fff640 1 -- 172.21.15.107:0/977096197 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc300017070 con 0x7fc310100700 2024-05-09T08:41:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.662+0000 7fc314810640 1 -- 172.21.15.107:0/977096197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31019d340 con 0x7fc310100700 2024-05-09T08:41:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.662+0000 7fc307fff640 1 -- 172.21.15.107:0/977096197 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc3000054d0 con 0x7fc310100700 2024-05-09T08:41:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.662+0000 7fc314810640 1 -- 172.21.15.107:0/977096197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc31010de10 con 0x7fc310100700 2024-05-09T08:41:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.662+0000 7fc307fff640 1 -- 172.21.15.107:0/977096197 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc300004cf0 con 0x7fc310100700 2024-05-09T08:41:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.665+0000 7fc307fff640 1 -- 172.21.15.107:0/977096197 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fc300004e90 con 0x7fc310100700 2024-05-09T08:41:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.665+0000 7fc3175f4740 1 Processor -- start 2024-05-09T08:41:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.666+0000 7fc3175f4740 1 -- start start 2024-05-09T08:41:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.666+0000 7fc3175f4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c23660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.666+0000 7fc3175f4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c2dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.666+0000 7fc3175f4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c2e270 0x55a3a4c30660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.666+0000 7fc3175f4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a3a4b88d00 con 0x55a3a4c23ba0 2024-05-09T08:41:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.666+0000 7fc3175f4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a3a4b304c0 con 0x55a3a4c23280 2024-05-09T08:41:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.666+0000 7fc3175f4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a3a4b84cc0 con 0x55a3a4c2e270 2024-05-09T08:41:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.666+0000 7fc30e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c2dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.666+0000 7fc30ed9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c23660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.666+0000 7fc30ed9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c23660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:59116/0 (socket says 172.21.15.107:59116) 2024-05-09T08:41:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.666+0000 7fc30ed9d640 1 -- 172.21.15.107:0/136924009 learned_addr learned my addr 172.21.15.107:0/136924009 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.666+0000 7fc307fff640 1 --2- 172.21.15.107:0/977096197 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc2fc0a2d80 0x7fc2fc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.667+0000 7fc305d9b640 1 --2- 172.21.15.107:0/136924009 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c2e270 0x55a3a4c30660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.667+0000 7fc30ed9d640 1 -- 172.21.15.107:0/136924009 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c2e270 msgr2=0x55a3a4c30660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.667+0000 7fc307fff640 1 -- 172.21.15.107:0/977096197 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fc3000cf110 con 0x7fc310100700 2024-05-09T08:41:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.667+0000 7fc30ed9d640 1 --2- 172.21.15.107:0/136924009 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c2e270 0x55a3a4c30660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.667+0000 7fc30ed9d640 1 -- 172.21.15.107:0/136924009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c23ba0 msgr2=0x55a3a4c2dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.667+0000 7fc30ed9d640 1 --2- 172.21.15.107:0/136924009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c2dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.667+0000 7fc30ed9d640 1 -- 172.21.15.107:0/136924009 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3a49667d0 con 0x55a3a4c23280 2024-05-09T08:41:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.667+0000 7fc305d9b640 1 --2- 172.21.15.107:0/136924009 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c2e270 0x55a3a4c30660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.667+0000 7fc30ed9d640 1 --2- 172.21.15.107:0/977096197 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc2fc0a2d80 0x7fc2fc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.667+0000 7fc30ed9d640 1 --2- 172.21.15.107:0/136924009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c23660 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc308009710 tx=0x7fc308009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c9e3defdf716f09 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.667+0000 7fc304d99640 1 -- 172.21.15.107:0/136924009 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30801d530 con 0x55a3a4c23280 2024-05-09T08:41:33.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.667+0000 7fc304d99640 1 -- 172.21.15.107:0/136924009 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc30801db80 con 0x55a3a4c23280 2024-05-09T08:41:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.668+0000 7fc304d99640 1 -- 172.21.15.107:0/136924009 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3080267f0 con 0x55a3a4c23280 2024-05-09T08:41:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.668+0000 7fc30ed9d640 1 --2- 172.21.15.107:0/977096197 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc2fc0a2d80 0x7fc2fc0a5240 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc308025000 tx=0x7fc30801e000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.668+0000 7fc3175f4740 1 -- 172.21.15.107:0/136924009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 msgr2=0x55a3a4c23660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.668+0000 7fc3175f4740 1 --2- 172.21.15.107:0/136924009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c23660 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc308009710 tx=0x7fc308009bd0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.668+0000 7fc3175f4740 1 -- 172.21.15.107:0/136924009 shutdown_connections 2024-05-09T08:41:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.669+0000 7fc3175f4740 1 --2- 172.21.15.107:0/136924009 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c2e270 0x55a3a4c30660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.669+0000 7fc3175f4740 1 --2- 172.21.15.107:0/136924009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c2dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.669+0000 7fc3175f4740 1 --2- 172.21.15.107:0/136924009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c23660 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.669+0000 7fc3175f4740 1 -- 172.21.15.107:0/136924009 >> 172.21.15.107:0/136924009 conn(0x55a3a4c1f5a0 msgr2=0x55a3a4c1f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.669+0000 7fc3175f4740 1 -- 172.21.15.107:0/136924009 shutdown_connections 2024-05-09T08:41:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.669+0000 7fc3175f4740 1 -- 172.21.15.107:0/136924009 wait complete. 2024-05-09T08:41:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.669+0000 7fc3175f4740 1 -- start start 2024-05-09T08:41:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.670+0000 7fc3175f4740 1 Processor -- start 2024-05-09T08:41:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.670+0000 7fc3175f4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c23280 0x55a3a4c2b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.670+0000 7fc3175f4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c2b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.670+0000 7fc3175f4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c2e270 0x55a3a4c2bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.670+0000 7fc3175f4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a3a4c31060 con 0x55a3a4c23280 2024-05-09T08:41:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.670+0000 7fc3175f4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a3a4b43b90 con 0x55a3a4c23ba0 2024-05-09T08:41:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.670+0000 7fc3175f4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a3a4c311e0 con 0x55a3a4c2e270 2024-05-09T08:41:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.670+0000 7fc3175f4740 10 client.0 _ll_register_callbacks cb 0x55a3a4b7fda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.670+0000 7fc305d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c2b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.670+0000 7fc305d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c2b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:59118/0 (socket says 172.21.15.107:59118) 2024-05-09T08:41:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.670+0000 7fc305d9b640 1 -- 172.21.15.107:0/3061173497 learned_addr learned my addr 172.21.15.107:0/3061173497 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.670+0000 7fc30ed9d640 1 --2- 172.21.15.107:0/3061173497 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c2e270 0x55a3a4c2bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.671+0000 7fc30e59c640 1 --2- 172.21.15.107:0/3061173497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c23280 0x55a3a4c2b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.671+0000 7fc305d9b640 1 -- 172.21.15.107:0/3061173497 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c2e270 msgr2=0x55a3a4c2bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.671+0000 7fc305d9b640 1 --2- 172.21.15.107:0/3061173497 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c2e270 0x55a3a4c2bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.671+0000 7fc305d9b640 1 -- 172.21.15.107:0/3061173497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c23280 msgr2=0x55a3a4c2b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.671+0000 7fc305d9b640 1 --2- 172.21.15.107:0/3061173497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c23280 0x55a3a4c2b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.671+0000 7fc305d9b640 1 -- 172.21.15.107:0/3061173497 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3a489c3f0 con 0x55a3a4c23ba0 2024-05-09T08:41:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.671+0000 7fc30e59c640 1 --2- 172.21.15.107:0/3061173497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c23280 0x55a3a4c2b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.671+0000 7fc305d9b640 1 --2- 172.21.15.107:0/3061173497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c2b670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8009930 tx=0x7fc2f80314a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f3abda236056a92 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.671+0000 7fc2f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:41:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.671+0000 7fc2f5ffb640 1 -- 172.21.15.107:0/3061173497 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8035040 con 0x55a3a4c23ba0 2024-05-09T08:41:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.671+0000 7fc2f5ffb640 1 -- 172.21.15.107:0/3061173497 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fc2f8004050 con 0x55a3a4c23ba0 2024-05-09T08:41:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.671+0000 7fc3175f4740 1 -- 172.21.15.107:0/3061173497 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc30801db80 con 0x55a3a4c23ba0 2024-05-09T08:41:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.671+0000 7fc2f5ffb640 1 -- 172.21.15.107:0/3061173497 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc2f804b050 con 0x55a3a4c23ba0 2024-05-09T08:41:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.672+0000 7fc3175f4740 10 client.26993 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.672+0000 7fc3175f4740 1 -- 172.21.15.107:0/3061173497 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a3a485c0f0 con 0x55a3a4c23ba0 2024-05-09T08:41:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.672+0000 7fc2f5ffb640 1 -- 172.21.15.107:0/3061173497 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc30801db80 con 0x55a3a4c23ba0 2024-05-09T08:41:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.672+0000 7fc2f5ffb640 1 -- 172.21.15.107:0/3061173497 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc2f803c440 con 0x55a3a4c23ba0 2024-05-09T08:41:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.672+0000 7fc3175f4740 10 client.26993 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.672+0000 7fc3175f4740 10 client.26993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.672+0000 7fc3175f4740 20 client.26993 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.672+0000 7fc3175f4740 1 --2- 172.21.15.107:0/3061173497 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a3a4c3e490 0x55a3a4c408a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.672+0000 7fc3175f4740 4 client.26993 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.672+0000 7fc3175f4740 1 -- 172.21.15.107:0/3061173497 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3a4b96430 con 0x55a3a4c3e490 2024-05-09T08:41:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.672+0000 7fc30e59c640 1 --2- 172.21.15.107:0/3061173497 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a3a4c3e490 0x55a3a4c408a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.673+0000 7fc2f5ffb640 10 client.26993 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.673+0000 7fc30e59c640 1 --2- 172.21.15.107:0/3061173497 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a3a4c3e490 0x55a3a4c408a0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c830f3676ab7b041 server_cookie=d92c36e266e1343a in_seq=0 out_seq=0 2024-05-09T08:41:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.673+0000 7fc2f5ffb640 1 -- 172.21.15.107:0/3061173497 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a3a4b96430 con 0x55a3a4c3e490 2024-05-09T08:41:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.673+0000 7fc2f5ffb640 10 client.26993 handle_command_reply: tid=0 2024-05-09T08:41:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.673+0000 7fc3175f4740 1 client.26993 shutdown 2024-05-09T08:41:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.674+0000 7fc3175f4740 1 -- 172.21.15.107:0/3061173497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23ba0 msgr2=0x55a3a4c2b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:33.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.674+0000 7fc3175f4740 1 --2- 172.21.15.107:0/3061173497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c2b670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8009930 tx=0x7fc2f80314a0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.674+0000 7fc3175f4740 1 -- 172.21.15.107:0/3061173497 shutdown_connections 2024-05-09T08:41:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.674+0000 7fc3175f4740 1 --2- 172.21.15.107:0/3061173497 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a3a4c3e490 0x55a3a4c408a0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.674+0000 7fc3175f4740 1 --2- 172.21.15.107:0/3061173497 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c2e270 0x55a3a4c2bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.674+0000 7fc2f5ffb640 0 client.26993 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.674+0000 7fc3175f4740 1 --2- 172.21.15.107:0/3061173497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c2b670 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.674+0000 7fc3175f4740 1 --2- 172.21.15.107:0/3061173497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c23280 0x55a3a4c2b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.674+0000 7fc3175f4740 1 -- 172.21.15.107:0/3061173497 >> 172.21.15.107:0/3061173497 conn(0x55a3a4c1f5a0 msgr2=0x55a3a4c252a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.674+0000 7fc3175f4740 1 -- 172.21.15.107:0/3061173497 shutdown_connections 2024-05-09T08:41:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.674+0000 7fc3175f4740 1 -- 172.21.15.107:0/3061173497 wait complete. 2024-05-09T08:41:33.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.674+0000 7fc3175f4740 20 client.26993 trim_cache size 0 max 16384 2024-05-09T08:41:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.709+0000 7fc3175f4740 1 Processor -- start 2024-05-09T08:41:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.709+0000 7fc3175f4740 1 -- start start 2024-05-09T08:41:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.709+0000 7fc3175f4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c39f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc3175f4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c461b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc3175f4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c2e270 0x55a3a4c466f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc3175f4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f0012780 con 0x55a3a4c2e270 2024-05-09T08:41:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc3175f4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f0010050 con 0x55a3a4c23280 2024-05-09T08:41:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc3175f4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f000dec0 con 0x55a3a4c23ba0 2024-05-09T08:41:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc30ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c2e270 0x55a3a4c466f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc30e59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c39f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc30ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c2e270 0x55a3a4c466f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57846/0 (socket says 172.21.15.107:57846) 2024-05-09T08:41:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc30ed9d640 1 -- 172.21.15.107:0/346822375 learned_addr learned my addr 172.21.15.107:0/346822375 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc305d9b640 1 --2- 172.21.15.107:0/346822375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc30ed9d640 1 -- 172.21.15.107:0/346822375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c23ba0 msgr2=0x55a3a4c461b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc30ed9d640 1 --2- 172.21.15.107:0/346822375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c461b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc30ed9d640 1 -- 172.21.15.107:0/346822375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 msgr2=0x55a3a4c39f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc30ed9d640 1 --2- 172.21.15.107:0/346822375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c39f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc30ed9d640 1 -- 172.21.15.107:0/346822375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3a4c2a0e0 con 0x55a3a4c2e270 2024-05-09T08:41:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc305d9b640 1 --2- 172.21.15.107:0/346822375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c461b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc30e59c640 1 --2- 172.21.15.107:0/346822375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c39f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:33.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.710+0000 7fc30ed9d640 1 --2- 172.21.15.107:0/346822375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c2e270 0x55a3a4c466f0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fc308005a10 tx=0x7fc30800e7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9bb845be4f2c66c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:33.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.711+0000 7fc2f7fff640 1 -- 172.21.15.107:0/346822375 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30800e960 con 0x55a3a4c2e270 2024-05-09T08:41:33.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.711+0000 7fc2f7fff640 1 -- 172.21.15.107:0/346822375 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc308036070 con 0x55a3a4c2e270 2024-05-09T08:41:33.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.711+0000 7fc2f7fff640 1 -- 172.21.15.107:0/346822375 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30801d530 con 0x55a3a4c2e270 2024-05-09T08:41:33.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.711+0000 7fc3175f4740 1 -- 172.21.15.107:0/346822375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c2e270 msgr2=0x55a3a4c466f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:33.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.711+0000 7fc3175f4740 1 --2- 172.21.15.107:0/346822375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c2e270 0x55a3a4c466f0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fc308005a10 tx=0x7fc30800e7c0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.711+0000 7fc3175f4740 1 -- 172.21.15.107:0/346822375 shutdown_connections 2024-05-09T08:41:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.711+0000 7fc3175f4740 1 --2- 172.21.15.107:0/346822375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c2e270 0x55a3a4c466f0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.711+0000 7fc3175f4740 1 --2- 172.21.15.107:0/346822375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c461b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.711+0000 7fc3175f4740 1 --2- 172.21.15.107:0/346822375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c39f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.712+0000 7fc3175f4740 1 -- 172.21.15.107:0/346822375 >> 172.21.15.107:0/346822375 conn(0x55a3a4c1f5a0 msgr2=0x55a3a4c2b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.712+0000 7fc3175f4740 1 -- 172.21.15.107:0/346822375 shutdown_connections 2024-05-09T08:41:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.712+0000 7fc3175f4740 1 -- 172.21.15.107:0/346822375 wait complete. 2024-05-09T08:41:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.712+0000 7fc3175f4740 1 -- start start 2024-05-09T08:41:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.712+0000 7fc3175f4740 1 Processor -- start 2024-05-09T08:41:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.712+0000 7fc3175f4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c32130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.712+0000 7fc3175f4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c32670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.712+0000 7fc3175f4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c2e270 0x55a3a4c32bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.712+0000 7fc3175f4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f000e510 con 0x55a3a4c2e270 2024-05-09T08:41:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.712+0000 7fc3175f4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f000f980 con 0x55a3a4c23280 2024-05-09T08:41:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.712+0000 7fc3175f4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f000ebe0 con 0x55a3a4c23ba0 2024-05-09T08:41:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc3175f4740 10 client.0 _ll_register_callbacks cb 0x7fc2f8031c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc30e59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c32130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc30e59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c32130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:59142/0 (socket says 172.21.15.107:59142) 2024-05-09T08:41:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc30e59c640 1 -- 172.21.15.107:0/3925271829 learned_addr learned my addr 172.21.15.107:0/3925271829 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc305d9b640 1 --2- 172.21.15.107:0/3925271829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c32670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc30ed9d640 1 --2- 172.21.15.107:0/3925271829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c2e270 0x55a3a4c32bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc305d9b640 1 -- 172.21.15.107:0/3925271829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 msgr2=0x55a3a4c32130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc305d9b640 1 --2- 172.21.15.107:0/3925271829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c32130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc305d9b640 1 -- 172.21.15.107:0/3925271829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c2e270 msgr2=0x55a3a4c32bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc305d9b640 1 --2- 172.21.15.107:0/3925271829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c2e270 0x55a3a4c32bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc305d9b640 1 -- 172.21.15.107:0/3925271829 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3a4ca3d40 con 0x55a3a4c23ba0 2024-05-09T08:41:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc30e59c640 1 --2- 172.21.15.107:0/3925271829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c32130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc30ed9d640 1 --2- 172.21.15.107:0/3925271829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c2e270 0x55a3a4c32bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc305d9b640 1 --2- 172.21.15.107:0/3925271829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c32670 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc2f803c960 tx=0x7fc2f803cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8fec100766a6a4e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc2f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:41:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc2f5ffb640 1 -- 172.21.15.107:0/3925271829 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8035040 con 0x55a3a4c23ba0 2024-05-09T08:41:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc2f5ffb640 1 -- 172.21.15.107:0/3925271829 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fc2f8005030 con 0x55a3a4c23ba0 2024-05-09T08:41:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.713+0000 7fc3175f4740 1 -- 172.21.15.107:0/3925271829 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc308036070 con 0x55a3a4c23ba0 2024-05-09T08:41:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.714+0000 7fc2f5ffb640 1 -- 172.21.15.107:0/3925271829 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc2f805c050 con 0x55a3a4c23ba0 2024-05-09T08:41:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.714+0000 7fc3175f4740 10 client.27232 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.714+0000 7fc3175f4740 1 -- 172.21.15.107:0/3925271829 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a3a4ca4ff0 con 0x55a3a4c23ba0 2024-05-09T08:41:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.714+0000 7fc2f5ffb640 1 -- 172.21.15.107:0/3925271829 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc308036070 con 0x55a3a4c23ba0 2024-05-09T08:41:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.714+0000 7fc2f5ffb640 1 -- 172.21.15.107:0/3925271829 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc2f8004ac0 con 0x55a3a4c23ba0 2024-05-09T08:41:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.714+0000 7fc3175f4740 10 client.27232 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.714+0000 7fc3175f4740 10 client.27232 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.714+0000 7fc3175f4740 20 client.27232 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.714+0000 7fc3175f4740 1 --2- 172.21.15.107:0/3925271829 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a3a4c3e490 0x55a3a4ca7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.714+0000 7fc3175f4740 4 client.27232 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:41:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.714+0000 7fc3175f4740 1 -- 172.21.15.107:0/3925271829 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a3a4b90780 con 0x55a3a4c3e490 2024-05-09T08:41:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.714+0000 7fc30e59c640 1 --2- 172.21.15.107:0/3925271829 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a3a4c3e490 0x55a3a4ca7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc2f5ffb640 10 client.27232 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc30e59c640 1 --2- 172.21.15.107:0/3925271829 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a3a4c3e490 0x55a3a4ca7740 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a3208aacce775f5 server_cookie=fef35fdcbc6ba489 in_seq=0 out_seq=0 2024-05-09T08:41:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc2f5ffb640 1 -- 172.21.15.107:0/3925271829 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a3a4b90780 con 0x55a3a4c3e490 2024-05-09T08:41:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc2f5ffb640 10 client.27232 handle_command_reply: tid=0 2024-05-09T08:41:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc3175f4740 1 client.27232 shutdown 2024-05-09T08:41:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc3175f4740 1 -- 172.21.15.107:0/3925271829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c23ba0 msgr2=0x55a3a4c32670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc3175f4740 1 --2- 172.21.15.107:0/3925271829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c32670 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc2f803c960 tx=0x7fc2f803cc10 comp rx=0 tx=0).stop 2024-05-09T08:41:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc3175f4740 1 -- 172.21.15.107:0/3925271829 shutdown_connections 2024-05-09T08:41:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc3175f4740 1 --2- 172.21.15.107:0/3925271829 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a3a4c3e490 0x55a3a4ca7740 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc3175f4740 1 --2- 172.21.15.107:0/3925271829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a3a4c2e270 0x55a3a4c32bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc3175f4740 1 --2- 172.21.15.107:0/3925271829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a3a4c23ba0 0x55a3a4c32670 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc3175f4740 1 --2- 172.21.15.107:0/3925271829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a3a4c23280 0x55a3a4c32130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc2f5ffb640 0 client.27232 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc3175f4740 1 -- 172.21.15.107:0/3925271829 >> 172.21.15.107:0/3925271829 conn(0x55a3a4c1f5a0 msgr2=0x55a3a4c1df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc3175f4740 1 -- 172.21.15.107:0/3925271829 shutdown_connections 2024-05-09T08:41:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc3175f4740 1 -- 172.21.15.107:0/3925271829 wait complete. 2024-05-09T08:41:33.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.715+0000 7fc3175f4740 20 client.27232 trim_cache size 0 max 16384 2024-05-09T08:41:33.721 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:41:33.721 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:41:33.721 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:41:33.722 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:41:33.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.716+0000 7fc314810640 1 -- 172.21.15.107:0/977096197 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc2fc0a2d80 msgr2=0x7fc2fc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:33.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.716+0000 7fc314810640 1 --2- 172.21.15.107:0/977096197 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc2fc0a2d80 0x7fc2fc0a5240 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc308025000 tx=0x7fc30801e000 comp rx=0 tx=0).stop 2024-05-09T08:41:33.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.716+0000 7fc314810640 1 -- 172.21.15.107:0/977096197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc310100700 msgr2=0x7fc310198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:33.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.716+0000 7fc314810640 1 --2- 172.21.15.107:0/977096197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc310100700 0x7fc310198060 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fc30000d770 tx=0x7fc30000dc30 comp rx=0 tx=0).stop 2024-05-09T08:41:33.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.716+0000 7fc314810640 1 -- 172.21.15.107:0/977096197 shutdown_connections 2024-05-09T08:41:33.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.716+0000 7fc314810640 1 --2- 172.21.15.107:0/977096197 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc2fc0a2d80 0x7fc2fc0a5240 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.716+0000 7fc314810640 1 --2- 172.21.15.107:0/977096197 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc310106710 0x7fc31019c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.716+0000 7fc314810640 1 --2- 172.21.15.107:0/977096197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3101010c0 0x7fc3101985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.716+0000 7fc314810640 1 --2- 172.21.15.107:0/977096197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc310100700 0x7fc310198060 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.716+0000 7fc314810640 1 -- 172.21.15.107:0/977096197 >> 172.21.15.107:0/977096197 conn(0x7fc310077830 msgr2=0x7fc310103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:33.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.716+0000 7fc314810640 1 -- 172.21.15.107:0/977096197 shutdown_connections 2024-05-09T08:41:33.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.716+0000 7fc314810640 1 -- 172.21.15.107:0/977096197 wait complete. 2024-05-09T08:41:33.797 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:41:33.797 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:41:33.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.928+0000 7fc9f3148640 1 -- 172.21.15.107:0/3113064613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ec106720 msgr2=0x7fc9ec106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:33.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.928+0000 7fc9f3148640 1 --2- 172.21.15.107:0/3113064613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ec106720 0x7fc9ec106b00 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0009930 tx=0x7fc9e002a420 comp rx=0 tx=0).stop 2024-05-09T08:41:33.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.928+0000 7fc9f3148640 1 -- 172.21.15.107:0/3113064613 shutdown_connections 2024-05-09T08:41:33.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.928+0000 7fc9f3148640 1 --2- 172.21.15.107:0/3113064613 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9ec1010d0 0x7fc9ec10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.928+0000 7fc9f3148640 1 --2- 172.21.15.107:0/3113064613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9ec100710 0x7fc9ec100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.928+0000 7fc9f3148640 1 --2- 172.21.15.107:0/3113064613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ec106720 0x7fc9ec106b00 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.928+0000 7fc9f3148640 1 -- 172.21.15.107:0/3113064613 >> 172.21.15.107:0/3113064613 conn(0x7fc9ec077830 msgr2=0x7fc9ec077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:33.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.928+0000 7fc9f3148640 1 -- 172.21.15.107:0/3113064613 shutdown_connections 2024-05-09T08:41:33.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.928+0000 7fc9f3148640 1 -- 172.21.15.107:0/3113064613 wait complete. 2024-05-09T08:41:33.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.929+0000 7fc9f3148640 1 Processor -- start 2024-05-09T08:41:33.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.929+0000 7fc9f3148640 1 -- start start 2024-05-09T08:41:33.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.929+0000 7fc9f3148640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9ec100710 0x7fc9ec198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.929+0000 7fc9f3148640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ec1010d0 0x7fc9ec1993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.929+0000 7fc9f3148640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9ec106720 0x7fc9ec1a3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.929+0000 7fc9f3148640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec1109e0 con 0x7fc9ec1010d0 2024-05-09T08:41:33.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.929+0000 7fc9f3148640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec110860 con 0x7fc9ec106720 2024-05-09T08:41:33.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.929+0000 7fc9f3148640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec110b60 con 0x7fc9ec100710 2024-05-09T08:41:33.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.929+0000 7fc9ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ec1010d0 0x7fc9ec1993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.929+0000 7fc9ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ec1010d0 0x7fc9ec1993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57876/0 (socket says 172.21.15.107:57876) 2024-05-09T08:41:33.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.929+0000 7fc9ebfff640 1 -- 172.21.15.107:0/3405354151 learned_addr learned my addr 172.21.15.107:0/3405354151 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:33.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.930+0000 7fc9f0ee4640 1 --2- 172.21.15.107:0/3405354151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9ec100710 0x7fc9ec198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.930+0000 7fc9f16e5640 1 --2- 172.21.15.107:0/3405354151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9ec106720 0x7fc9ec1a3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.930+0000 7fc9ebfff640 1 -- 172.21.15.107:0/3405354151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9ec100710 msgr2=0x7fc9ec198ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:33.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.930+0000 7fc9ebfff640 1 --2- 172.21.15.107:0/3405354151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9ec100710 0x7fc9ec198ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.930+0000 7fc9ebfff640 1 -- 172.21.15.107:0/3405354151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9ec106720 msgr2=0x7fc9ec1a3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:33.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.930+0000 7fc9ebfff640 1 --2- 172.21.15.107:0/3405354151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9ec106720 0x7fc9ec1a3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:33.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.930+0000 7fc9ebfff640 1 -- 172.21.15.107:0/3405354151 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9ec1a4460 con 0x7fc9ec1010d0 2024-05-09T08:41:33.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.930+0000 7fc9f16e5640 1 --2- 172.21.15.107:0/3405354151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9ec106720 0x7fc9ec1a3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:33.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.930+0000 7fc9ebfff640 1 --2- 172.21.15.107:0/3405354151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ec1010d0 0x7fc9ec1993e0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fc9d800d770 tx=0x7fc9d800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:33.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.930+0000 7fc9e9ffb640 1 -- 172.21.15.107:0/3405354151 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8017070 con 0x7fc9ec1010d0 2024-05-09T08:41:33.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.930+0000 7fc9e9ffb640 1 -- 172.21.15.107:0/3405354151 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc9d80054d0 con 0x7fc9ec1010d0 2024-05-09T08:41:33.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.930+0000 7fc9f3148640 1 -- 172.21.15.107:0/3405354151 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9ec1a4630 con 0x7fc9ec1010d0 2024-05-09T08:41:33.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.930+0000 7fc9e9ffb640 1 -- 172.21.15.107:0/3405354151 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8004cf0 con 0x7fc9ec1010d0 2024-05-09T08:41:33.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.931+0000 7fc9f3148640 1 -- 172.21.15.107:0/3405354151 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9ec1a4b50 con 0x7fc9ec1010d0 2024-05-09T08:41:33.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.932+0000 7fc9f3148640 1 -- 172.21.15.107:0/3405354151 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9ec068650 con 0x7fc9ec1010d0 2024-05-09T08:41:33.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.933+0000 7fc9e9ffb640 1 -- 172.21.15.107:0/3405354151 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fc9d8004e90 con 0x7fc9ec1010d0 2024-05-09T08:41:33.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.934+0000 7fc9e9ffb640 1 --2- 172.21.15.107:0/3405354151 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9dc0a2d80 0x7fc9dc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:33.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.934+0000 7fc9f0ee4640 1 --2- 172.21.15.107:0/3405354151 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9dc0a2d80 0x7fc9dc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:33.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.934+0000 7fc9e9ffb640 1 -- 172.21.15.107:0/3405354151 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fc9d80cf1f0 con 0x7fc9ec1010d0 2024-05-09T08:41:33.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.934+0000 7fc9f0ee4640 1 --2- 172.21.15.107:0/3405354151 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9dc0a2d80 0x7fc9dc0a5240 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0009900 tx=0x7fc9e0000bc0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:33.938+0000 7fc9e9ffb640 1 -- 172.21.15.107:0/3405354151 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc9d809ba60 con 0x7fc9ec1010d0 2024-05-09T08:41:34.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.218+0000 7fc9f3148640 1 -- 172.21.15.107:0/3405354151 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9ec108c80 con 0x7fc9ec1010d0 2024-05-09T08:41:34.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.219+0000 7fc9e9ffb640 1 -- 172.21.15.107:0/3405354151 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11887 (secure 0 0 0) 0x7fc9d809b360 con 0x7fc9ec1010d0 2024-05-09T08:41:34.218 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:41:34.218 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:37:10.930317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:41:34.218 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2024-05-09T08:41:34.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.221+0000 7fc9f3148640 1 -- 172.21.15.107:0/3405354151 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9dc0a2d80 msgr2=0x7fc9dc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.221+0000 7fc9f3148640 1 --2- 172.21.15.107:0/3405354151 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9dc0a2d80 0x7fc9dc0a5240 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0009900 tx=0x7fc9e0000bc0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.221+0000 7fc9f3148640 1 -- 172.21.15.107:0/3405354151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ec1010d0 msgr2=0x7fc9ec1993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.221+0000 7fc9f3148640 1 --2- 172.21.15.107:0/3405354151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ec1010d0 0x7fc9ec1993e0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fc9d800d770 tx=0x7fc9d800dc30 comp rx=0 tx=0).stop 2024-05-09T08:41:34.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.221+0000 7fc9f3148640 1 -- 172.21.15.107:0/3405354151 shutdown_connections 2024-05-09T08:41:34.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.221+0000 7fc9f3148640 1 --2- 172.21.15.107:0/3405354151 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9dc0a2d80 0x7fc9dc0a5240 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.221+0000 7fc9f3148640 1 --2- 172.21.15.107:0/3405354151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9ec106720 0x7fc9ec1a3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.221+0000 7fc9f3148640 1 --2- 172.21.15.107:0/3405354151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9ec1010d0 0x7fc9ec1993e0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.221+0000 7fc9f3148640 1 --2- 172.21.15.107:0/3405354151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9ec100710 0x7fc9ec198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.221+0000 7fc9f3148640 1 -- 172.21.15.107:0/3405354151 >> 172.21.15.107:0/3405354151 conn(0x7fc9ec077830 msgr2=0x7fc9ec103100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.221+0000 7fc9f3148640 1 -- 172.21.15.107:0/3405354151 shutdown_connections 2024-05-09T08:41:34.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.221+0000 7fc9f3148640 1 -- 172.21.15.107:0/3405354151 wait complete. 2024-05-09T08:41:34.232 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:34 smithi107 ceph-mon[30122]: pgmap v253: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:41:34.232 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:34 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:41:34.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:41:34.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:34 smithi157 ceph-mon[33614]: pgmap v253: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:41:34.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:34 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:41:34.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.353+0000 7f6508909640 1 -- 172.21.15.107:0/29029703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6504106710 msgr2=0x7f6504106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.353+0000 7f6508909640 1 --2- 172.21.15.107:0/29029703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6504106710 0x7f6504106af0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f64f4009930 tx=0x7f64f402a420 comp rx=0 tx=0).stop 2024-05-09T08:41:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.353+0000 7f6508909640 1 -- 172.21.15.107:0/29029703 shutdown_connections 2024-05-09T08:41:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.353+0000 7f6508909640 1 --2- 172.21.15.107:0/29029703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65041010c0 0x7f650410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.353+0000 7f6508909640 1 --2- 172.21.15.107:0/29029703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6504100700 0x7f6504100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.353+0000 7f6508909640 1 --2- 172.21.15.107:0/29029703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6504106710 0x7f6504106af0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.353+0000 7f6508909640 1 -- 172.21.15.107:0/29029703 >> 172.21.15.107:0/29029703 conn(0x7f6504077830 msgr2=0x7f6504077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.353+0000 7f6508909640 1 -- 172.21.15.107:0/29029703 shutdown_connections 2024-05-09T08:41:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.353+0000 7f6508909640 1 -- 172.21.15.107:0/29029703 wait complete. 2024-05-09T08:41:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.354+0000 7f6508909640 1 Processor -- start 2024-05-09T08:41:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.354+0000 7f6508909640 1 -- start start 2024-05-09T08:41:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.354+0000 7f6508909640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6504100700 0x7f6504198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.354+0000 7f6508909640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65041010c0 0x7f6504199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.354+0000 7f6508909640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6504106710 0x7f65041a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.354+0000 7f6508909640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6504110990 con 0x7f6504100700 2024-05-09T08:41:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.354+0000 7f6508909640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6504110810 con 0x7f65041010c0 2024-05-09T08:41:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.354+0000 7f6508909640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6504110b10 con 0x7f6504106710 2024-05-09T08:41:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.354+0000 7f650259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6504100700 0x7f6504198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.354+0000 7f650259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6504100700 0x7f6504198ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57906/0 (socket says 172.21.15.107:57906) 2024-05-09T08:41:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.354+0000 7f650259c640 1 -- 172.21.15.107:0/735584852 learned_addr learned my addr 172.21.15.107:0/735584852 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.355+0000 7f6502d9d640 1 --2- 172.21.15.107:0/735584852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6504106710 0x7f65041a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.355+0000 7f6501d9b640 1 --2- 172.21.15.107:0/735584852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65041010c0 0x7f6504199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.355+0000 7f650259c640 1 -- 172.21.15.107:0/735584852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6504106710 msgr2=0x7f65041a3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.355+0000 7f650259c640 1 --2- 172.21.15.107:0/735584852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6504106710 0x7f65041a3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.355+0000 7f650259c640 1 -- 172.21.15.107:0/735584852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65041010c0 msgr2=0x7f6504199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.355+0000 7f650259c640 1 --2- 172.21.15.107:0/735584852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65041010c0 0x7f6504199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.355+0000 7f650259c640 1 -- 172.21.15.107:0/735584852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65041a4490 con 0x7f6504100700 2024-05-09T08:41:34.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.355+0000 7f6501d9b640 1 --2- 172.21.15.107:0/735584852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65041010c0 0x7f6504199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:34.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.355+0000 7f650259c640 1 --2- 172.21.15.107:0/735584852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6504100700 0x7f6504198ed0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f64f4004770 tx=0x7f64f40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.355+0000 7f64f37fe640 1 -- 172.21.15.107:0/735584852 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f403f070 con 0x7f6504100700 2024-05-09T08:41:34.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.355+0000 7f64f37fe640 1 -- 172.21.15.107:0/735584852 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64f403b460 con 0x7f6504100700 2024-05-09T08:41:34.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.355+0000 7f6508909640 1 -- 172.21.15.107:0/735584852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65041a4660 con 0x7f6504100700 2024-05-09T08:41:34.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.355+0000 7f64f37fe640 1 -- 172.21.15.107:0/735584852 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f4035070 con 0x7f6504100700 2024-05-09T08:41:34.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.355+0000 7f6508909640 1 -- 172.21.15.107:0/735584852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65041a4b20 con 0x7f6504100700 2024-05-09T08:41:34.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.357+0000 7f64f37fe640 1 -- 172.21.15.107:0/735584852 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f64f403b610 con 0x7f6504100700 2024-05-09T08:41:34.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.358+0000 7f64f37fe640 1 --2- 172.21.15.107:0/735584852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f64ec0a2d00 0x7f64ec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.358+0000 7f650b60f740 1 Processor -- start 2024-05-09T08:41:34.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.358+0000 7f64f37fe640 1 -- 172.21.15.107:0/735584852 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f64f40efb70 con 0x7f6504100700 2024-05-09T08:41:34.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.358+0000 7f6501d9b640 1 --2- 172.21.15.107:0/735584852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f64ec0a2d00 0x7f64ec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.358+0000 7f650b60f740 1 -- start start 2024-05-09T08:41:34.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f650b60f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e6461eb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f650b60f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461ebc50 0x55e6461f5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f650b60f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461f6320 0x55e6461f8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f650b60f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e646150c40 con 0x55e6461eb330 2024-05-09T08:41:34.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f650b60f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e6460f8580 con 0x55e6461f6320 2024-05-09T08:41:34.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f650b60f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55e64614cc00 con 0x55e6461ebc50 2024-05-09T08:41:34.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f650259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e6461eb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f650259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e6461eb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57918/0 (socket says 172.21.15.107:57918) 2024-05-09T08:41:34.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f650259c640 1 -- 172.21.15.107:0/3582623027 learned_addr learned my addr 172.21.15.107:0/3582623027 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f6502d9d640 1 --2- 172.21.15.107:0/3582623027 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461f6320 0x55e6461f8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f6501d9b640 1 --2- 172.21.15.107:0/735584852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f64ec0a2d00 0x7f64ec0a51c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f64f80058c0 tx=0x7f64f8005850 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f6501d9b640 1 --2- 172.21.15.107:0/3582623027 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461ebc50 0x55e6461f5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f650259c640 1 -- 172.21.15.107:0/3582623027 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461ebc50 msgr2=0x55e6461f5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f650259c640 1 --2- 172.21.15.107:0/3582623027 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461ebc50 0x55e6461f5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f650259c640 1 -- 172.21.15.107:0/3582623027 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461f6320 msgr2=0x55e6461f8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f650259c640 1 --2- 172.21.15.107:0/3582623027 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461f6320 0x55e6461f8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f650259c640 1 -- 172.21.15.107:0/3582623027 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e645e643f0 con 0x55e6461eb330 2024-05-09T08:41:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.359+0000 7f650259c640 1 --2- 172.21.15.107:0/3582623027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e6461eb710 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f64f4005d00 tx=0x7f64f40044e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ca181924c31878 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.360+0000 7f64f0ff9640 1 -- 172.21.15.107:0/3582623027 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f4048040 con 0x55e6461eb330 2024-05-09T08:41:34.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.360+0000 7f64f0ff9640 1 -- 172.21.15.107:0/3582623027 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64f40c0090 con 0x55e6461eb330 2024-05-09T08:41:34.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.360+0000 7f64f0ff9640 1 -- 172.21.15.107:0/3582623027 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f40c44c0 con 0x55e6461eb330 2024-05-09T08:41:34.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.360+0000 7f650b60f740 1 -- 172.21.15.107:0/3582623027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 msgr2=0x55e6461eb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.360+0000 7f650b60f740 1 --2- 172.21.15.107:0/3582623027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e6461eb710 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f64f4005d00 tx=0x7f64f40044e0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.360+0000 7f650b60f740 1 -- 172.21.15.107:0/3582623027 shutdown_connections 2024-05-09T08:41:34.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.360+0000 7f650b60f740 1 --2- 172.21.15.107:0/3582623027 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461f6320 0x55e6461f8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.360+0000 7f650b60f740 1 --2- 172.21.15.107:0/3582623027 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461ebc50 0x55e6461f5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.360+0000 7f650b60f740 1 --2- 172.21.15.107:0/3582623027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e6461eb710 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.360+0000 7f650b60f740 1 -- 172.21.15.107:0/3582623027 >> 172.21.15.107:0/3582623027 conn(0x55e6461e7650 msgr2=0x55e6461e7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.361+0000 7f650b60f740 1 -- 172.21.15.107:0/3582623027 shutdown_connections 2024-05-09T08:41:34.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.361+0000 7f650b60f740 1 -- 172.21.15.107:0/3582623027 wait complete. 2024-05-09T08:41:34.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.361+0000 7f650b60f740 1 -- start start 2024-05-09T08:41:34.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.361+0000 7f650b60f740 1 Processor -- start 2024-05-09T08:41:34.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.361+0000 7f650b60f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e6461f3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.361+0000 7f650b60f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461ebc50 0x55e6461f37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.361+0000 7f650b60f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461f6320 0x55e6461f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.361+0000 7f650b60f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e6461f8ed0 con 0x55e6461eb330 2024-05-09T08:41:34.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.361+0000 7f650b60f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e64610bc40 con 0x55e6461f6320 2024-05-09T08:41:34.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.361+0000 7f650b60f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55e6461f9050 con 0x55e6461ebc50 2024-05-09T08:41:34.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.361+0000 7f650b60f740 10 client.0 _ll_register_callbacks cb 0x55e645e63be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:34.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.361+0000 7f650259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e6461f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.361+0000 7f650259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e6461f3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57932/0 (socket says 172.21.15.107:57932) 2024-05-09T08:41:34.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.361+0000 7f650259c640 1 -- 172.21.15.107:0/2332270960 learned_addr learned my addr 172.21.15.107:0/2332270960 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:34.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.361+0000 7f6501d9b640 1 --2- 172.21.15.107:0/2332270960 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461ebc50 0x55e6461f37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f6502d9d640 1 --2- 172.21.15.107:0/2332270960 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461f6320 0x55e6461f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f650259c640 1 -- 172.21.15.107:0/2332270960 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461ebc50 msgr2=0x55e6461f37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f650259c640 1 --2- 172.21.15.107:0/2332270960 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461ebc50 0x55e6461f37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f650259c640 1 -- 172.21.15.107:0/2332270960 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461f6320 msgr2=0x55e6461f3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f650259c640 1 --2- 172.21.15.107:0/2332270960 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461f6320 0x55e6461f3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f650259c640 1 -- 172.21.15.107:0/2332270960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e645f2e7d0 con 0x55e6461eb330 2024-05-09T08:41:34.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f6502d9d640 1 --2- 172.21.15.107:0/2332270960 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461f6320 0x55e6461f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:34.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f650259c640 1 --2- 172.21.15.107:0/2332270960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e6461f3290 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f64f40bb9e0 tx=0x7f64f402a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b956ebb00dbdfc2d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f64e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:34.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f64e9ffb640 1 -- 172.21.15.107:0/2332270960 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f4048070 con 0x55e6461eb330 2024-05-09T08:41:34.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f64e9ffb640 1 -- 172.21.15.107:0/2332270960 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f64f40b9fe0 con 0x55e6461eb330 2024-05-09T08:41:34.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f650b60f740 1 -- 172.21.15.107:0/2332270960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64f40c0090 con 0x55e6461eb330 2024-05-09T08:41:34.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f64e9ffb640 1 -- 172.21.15.107:0/2332270960 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f64f40c4950 con 0x55e6461eb330 2024-05-09T08:41:34.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f650b60f740 10 client.18036 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:34.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f650b60f740 1 -- 172.21.15.107:0/2332270960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e6461f8ed0 con 0x55e6461eb330 2024-05-09T08:41:34.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f64e9ffb640 1 -- 172.21.15.107:0/2332270960 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64f40c0090 con 0x55e6461eb330 2024-05-09T08:41:34.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.362+0000 7f64e9ffb640 1 -- 172.21.15.107:0/2332270960 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f64f40d3070 con 0x55e6461eb330 2024-05-09T08:41:34.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.363+0000 7f650b60f740 10 client.18036 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:34.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.363+0000 7f650b60f740 10 client.18036 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:41:34.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.363+0000 7f650b60f740 20 client.18036 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:34.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.363+0000 7f650b60f740 1 --2- 172.21.15.107:0/2332270960 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55e646206250 0x55e646208680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.363+0000 7f650b60f740 4 client.18036 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:34.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.363+0000 7f650b60f740 1 -- 172.21.15.107:0/2332270960 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e64615e370 con 0x55e646206250 2024-05-09T08:41:34.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.363+0000 7f6501d9b640 1 --2- 172.21.15.107:0/2332270960 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55e646206250 0x55e646208680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.363+0000 7f64e9ffb640 10 client.18036 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:41:34.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.363+0000 7f6501d9b640 1 --2- 172.21.15.107:0/2332270960 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55e646206250 0x55e646208680 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=998a5cf7af9430ba server_cookie=27a4417782fbbf01 in_seq=0 out_seq=0 2024-05-09T08:41:34.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.364+0000 7f64e9ffb640 1 -- 172.21.15.107:0/2332270960 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e64615e370 con 0x55e646206250 2024-05-09T08:41:34.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.364+0000 7f64e9ffb640 10 client.18036 handle_command_reply: tid=0 2024-05-09T08:41:34.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.364+0000 7f650b60f740 1 client.18036 shutdown 2024-05-09T08:41:34.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.364+0000 7f650b60f740 1 -- 172.21.15.107:0/2332270960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 msgr2=0x55e6461f3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.364+0000 7f650b60f740 1 --2- 172.21.15.107:0/2332270960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e6461f3290 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f64f40bb9e0 tx=0x7f64f402a420 comp rx=0 tx=0).stop 2024-05-09T08:41:34.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.364+0000 7f650b60f740 1 -- 172.21.15.107:0/2332270960 shutdown_connections 2024-05-09T08:41:34.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.364+0000 7f650b60f740 1 --2- 172.21.15.107:0/2332270960 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55e646206250 0x55e646208680 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.364+0000 7f64e9ffb640 0 client.18036 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:41:34.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.364+0000 7f650b60f740 1 --2- 172.21.15.107:0/2332270960 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461f6320 0x55e6461f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.364+0000 7f650b60f740 1 --2- 172.21.15.107:0/2332270960 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461ebc50 0x55e6461f37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.364+0000 7f650b60f740 1 --2- 172.21.15.107:0/2332270960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e6461f3290 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.364+0000 7f650b60f740 1 -- 172.21.15.107:0/2332270960 >> 172.21.15.107:0/2332270960 conn(0x55e6461e7650 msgr2=0x55e6461ed450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.364+0000 7f650b60f740 1 -- 172.21.15.107:0/2332270960 shutdown_connections 2024-05-09T08:41:34.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.364+0000 7f650b60f740 1 -- 172.21.15.107:0/2332270960 wait complete. 2024-05-09T08:41:34.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.364+0000 7f650b60f740 20 client.18036 trim_cache size 0 max 16384 2024-05-09T08:41:34.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:34 smithi110 ceph-mon[33622]: pgmap v253: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:41:34.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:34 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:41:34.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.380+0000 7f650b60f740 1 Processor -- start 2024-05-09T08:41:34.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.380+0000 7f650b60f740 1 -- start start 2024-05-09T08:41:34.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.380+0000 7f650b60f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461eb330 0x55e646201d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.380+0000 7f650b60f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461ebc50 0x55e6461f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.380+0000 7f650b60f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461f6320 0x55e6461ed2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.380+0000 7f650b60f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f65041b15d0 con 0x55e6461f6320 2024-05-09T08:41:34.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.380+0000 7f650b60f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f65041aeea0 con 0x55e6461ebc50 2024-05-09T08:41:34.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.380+0000 7f650b60f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f65041acd10 con 0x55e6461eb330 2024-05-09T08:41:34.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.381+0000 7f6502d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461f6320 0x55e6461ed2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.381+0000 7f6502d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461f6320 0x55e6461ed2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57944/0 (socket says 172.21.15.107:57944) 2024-05-09T08:41:34.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.381+0000 7f6502d9d640 1 -- 172.21.15.107:0/1285150728 learned_addr learned my addr 172.21.15.107:0/1285150728 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:34.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.381+0000 7f6501d9b640 1 --2- 172.21.15.107:0/1285150728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461ebc50 0x55e6461f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.381+0000 7f6502d9d640 1 -- 172.21.15.107:0/1285150728 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461eb330 msgr2=0x55e646201d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.381+0000 7f6502d9d640 1 --2- 172.21.15.107:0/1285150728 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461eb330 0x55e646201d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.381+0000 7f6502d9d640 1 -- 172.21.15.107:0/1285150728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461ebc50 msgr2=0x55e6461f23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.381+0000 7f6502d9d640 1 --2- 172.21.15.107:0/1285150728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461ebc50 0x55e6461f23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.381+0000 7f6502d9d640 1 -- 172.21.15.107:0/1285150728 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6461eba80 con 0x55e6461f6320 2024-05-09T08:41:34.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.382+0000 7f6502d9d640 1 --2- 172.21.15.107:0/1285150728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461f6320 0x55e6461ed2a0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f64fc011470 tx=0x7f64fc011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=127c7b0a8af24526 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.382+0000 7f64ebfff640 1 -- 172.21.15.107:0/1285150728 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc004d20 con 0x55e6461f6320 2024-05-09T08:41:34.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.382+0000 7f64ebfff640 1 -- 172.21.15.107:0/1285150728 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64fc010ac0 con 0x55e6461f6320 2024-05-09T08:41:34.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.382+0000 7f64ebfff640 1 -- 172.21.15.107:0/1285150728 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc005680 con 0x55e6461f6320 2024-05-09T08:41:34.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.383+0000 7f650b60f740 1 -- 172.21.15.107:0/1285150728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461f6320 msgr2=0x55e6461ed2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.383+0000 7f650b60f740 1 --2- 172.21.15.107:0/1285150728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461f6320 0x55e6461ed2a0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f64fc011470 tx=0x7f64fc011930 comp rx=0 tx=0).stop 2024-05-09T08:41:34.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.383+0000 7f650b60f740 1 -- 172.21.15.107:0/1285150728 shutdown_connections 2024-05-09T08:41:34.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.383+0000 7f650b60f740 1 --2- 172.21.15.107:0/1285150728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461f6320 0x55e6461ed2a0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.383+0000 7f650b60f740 1 --2- 172.21.15.107:0/1285150728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461ebc50 0x55e6461f23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.383+0000 7f650b60f740 1 --2- 172.21.15.107:0/1285150728 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461eb330 0x55e646201d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.383+0000 7f650b60f740 1 -- 172.21.15.107:0/1285150728 >> 172.21.15.107:0/1285150728 conn(0x55e6461e7650 msgr2=0x55e6461ecb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.383+0000 7f650b60f740 1 -- 172.21.15.107:0/1285150728 shutdown_connections 2024-05-09T08:41:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.383+0000 7f650b60f740 1 -- 172.21.15.107:0/1285150728 wait complete. 2024-05-09T08:41:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.383+0000 7f650b60f740 1 -- start start 2024-05-09T08:41:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.384+0000 7f650b60f740 1 Processor -- start 2024-05-09T08:41:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.384+0000 7f650b60f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e646267df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.384+0000 7f650b60f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461ebc50 0x55e6461fe140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.384+0000 7f650b60f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461f6320 0x55e6461fe680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.384+0000 7f650b60f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f65041ad360 con 0x55e6461eb330 2024-05-09T08:41:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.384+0000 7f650b60f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f65041ae7d0 con 0x55e6461ebc50 2024-05-09T08:41:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.384+0000 7f650b60f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f65041ada30 con 0x55e6461f6320 2024-05-09T08:41:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.384+0000 7f650259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e646267df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.384+0000 7f650259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e646267df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57954/0 (socket says 172.21.15.107:57954) 2024-05-09T08:41:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.384+0000 7f650259c640 1 -- 172.21.15.107:0/3506423210 learned_addr learned my addr 172.21.15.107:0/3506423210 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.384+0000 7f650b60f740 10 client.0 _ll_register_callbacks cb 0x7f64fc009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.384+0000 7f650259c640 1 -- 172.21.15.107:0/3506423210 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461f6320 msgr2=0x55e6461fe680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.384+0000 7f650259c640 1 --2- 172.21.15.107:0/3506423210 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461f6320 0x55e6461fe680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f650259c640 1 -- 172.21.15.107:0/3506423210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461ebc50 msgr2=0x55e6461fe140 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:41:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f650259c640 1 --2- 172.21.15.107:0/3506423210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461ebc50 0x55e6461fe140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f650259c640 1 -- 172.21.15.107:0/3506423210 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6461eff30 con 0x55e6461eb330 2024-05-09T08:41:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f650259c640 1 --2- 172.21.15.107:0/3506423210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e646267df0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f64f40b8140 tx=0x7f64f40095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f6c5d0928f03a23 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f64e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f64e9ffb640 1 -- 172.21.15.107:0/3506423210 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f4048030 con 0x55e6461eb330 2024-05-09T08:41:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f64e9ffb640 1 -- 172.21.15.107:0/3506423210 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f64f40b7cb0 con 0x55e6461eb330 2024-05-09T08:41:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f650b60f740 1 -- 172.21.15.107:0/3506423210 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64fc010ac0 con 0x55e6461eb330 2024-05-09T08:41:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f64e9ffb640 1 -- 172.21.15.107:0/3506423210 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f64f40b7a30 con 0x55e6461eb330 2024-05-09T08:41:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f650b60f740 10 client.18042 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f650b60f740 1 -- 172.21.15.107:0/3506423210 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e6461eff30 con 0x55e6461eb330 2024-05-09T08:41:34.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f64e9ffb640 1 -- 172.21.15.107:0/3506423210 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64fc010ac0 con 0x55e6461eb330 2024-05-09T08:41:34.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f64e9ffb640 1 -- 172.21.15.107:0/3506423210 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f64f40d3050 con 0x55e6461eb330 2024-05-09T08:41:34.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f650b60f740 10 client.18042 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:34.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f650b60f740 10 client.18042 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:41:34.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f650b60f740 20 client.18042 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:34.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f650b60f740 1 --2- 172.21.15.107:0/3506423210 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55e646206250 0x55e646273160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f650b60f740 4 client.18042 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:41:34.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.385+0000 7f650b60f740 1 -- 172.21.15.107:0/3506423210 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e6461586c0 con 0x55e646206250 2024-05-09T08:41:34.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.386+0000 7f6501d9b640 1 --2- 172.21.15.107:0/3506423210 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55e646206250 0x55e646273160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.386+0000 7f6501d9b640 1 --2- 172.21.15.107:0/3506423210 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55e646206250 0x55e646273160 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d88e640d765f4306 server_cookie=51500899f58aa485 in_seq=0 out_seq=0 2024-05-09T08:41:34.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.386+0000 7f64e9ffb640 10 client.18042 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:41:34.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.386+0000 7f64e9ffb640 1 -- 172.21.15.107:0/3506423210 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e6461586c0 con 0x55e646206250 2024-05-09T08:41:34.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.386+0000 7f64e9ffb640 10 client.18042 handle_command_reply: tid=0 2024-05-09T08:41:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.386+0000 7f650b60f740 1 client.18042 shutdown 2024-05-09T08:41:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.386+0000 7f650b60f740 1 -- 172.21.15.107:0/3506423210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 msgr2=0x55e646267df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.386+0000 7f650b60f740 1 --2- 172.21.15.107:0/3506423210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e646267df0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f64f40b8140 tx=0x7f64f40095f0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f650b60f740 1 -- 172.21.15.107:0/3506423210 shutdown_connections 2024-05-09T08:41:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f650b60f740 1 --2- 172.21.15.107:0/3506423210 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55e646206250 0x55e646273160 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f650b60f740 1 --2- 172.21.15.107:0/3506423210 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6461f6320 0x55e6461fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f650b60f740 1 --2- 172.21.15.107:0/3506423210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e6461ebc50 0x55e6461fe140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f64e9ffb640 0 client.18042 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:41:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f650b60f740 1 --2- 172.21.15.107:0/3506423210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e6461eb330 0x55e646267df0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f650b60f740 1 -- 172.21.15.107:0/3506423210 >> 172.21.15.107:0/3506423210 conn(0x55e6461e7650 msgr2=0x55e6461e67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f650b60f740 1 -- 172.21.15.107:0/3506423210 shutdown_connections 2024-05-09T08:41:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f650b60f740 1 -- 172.21.15.107:0/3506423210 wait complete. 2024-05-09T08:41:34.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f650b60f740 20 client.18042 trim_cache size 0 max 16384 2024-05-09T08:41:34.392 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:41:34.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f6508909640 1 -- 172.21.15.107:0/735584852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f64ec0a2d00 msgr2=0x7f64ec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f6508909640 1 --2- 172.21.15.107:0/735584852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f64ec0a2d00 0x7f64ec0a51c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f64f80058c0 tx=0x7f64f8005850 comp rx=0 tx=0).stop 2024-05-09T08:41:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f6508909640 1 -- 172.21.15.107:0/735584852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6504100700 msgr2=0x7f6504198ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f6508909640 1 --2- 172.21.15.107:0/735584852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6504100700 0x7f6504198ed0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f64f4004770 tx=0x7f64f40042f0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f6508909640 1 -- 172.21.15.107:0/735584852 shutdown_connections 2024-05-09T08:41:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f6508909640 1 --2- 172.21.15.107:0/735584852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f64ec0a2d00 0x7f64ec0a51c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f6508909640 1 --2- 172.21.15.107:0/735584852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6504106710 0x7f65041a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f6508909640 1 --2- 172.21.15.107:0/735584852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65041010c0 0x7f6504199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f6508909640 1 --2- 172.21.15.107:0/735584852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6504100700 0x7f6504198ed0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f6508909640 1 -- 172.21.15.107:0/735584852 >> 172.21.15.107:0/735584852 conn(0x7f6504077830 msgr2=0x7f6504103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f6508909640 1 -- 172.21.15.107:0/735584852 shutdown_connections 2024-05-09T08:41:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.387+0000 7f6508909640 1 -- 172.21.15.107:0/735584852 wait complete. 2024-05-09T08:41:34.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:41:34.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.511+0000 7f257609e640 1 -- 172.21.15.107:0/1643741637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2570106710 msgr2=0x7f2570106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.511+0000 7f257609e640 1 --2- 172.21.15.107:0/1643741637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2570106710 0x7f2570106af0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f2560009930 tx=0x7f256002a420 comp rx=0 tx=0).stop 2024-05-09T08:41:34.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.511+0000 7f257609e640 1 -- 172.21.15.107:0/1643741637 shutdown_connections 2024-05-09T08:41:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.511+0000 7f257609e640 1 --2- 172.21.15.107:0/1643741637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25701010c0 0x7f257010d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.511+0000 7f257609e640 1 --2- 172.21.15.107:0/1643741637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2570100700 0x7f2570100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.511+0000 7f257609e640 1 --2- 172.21.15.107:0/1643741637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2570106710 0x7f2570106af0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.511+0000 7f257609e640 1 -- 172.21.15.107:0/1643741637 >> 172.21.15.107:0/1643741637 conn(0x7f2570077830 msgr2=0x7f2570077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.511+0000 7f257609e640 1 -- 172.21.15.107:0/1643741637 shutdown_connections 2024-05-09T08:41:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.511+0000 7f257609e640 1 -- 172.21.15.107:0/1643741637 wait complete. 2024-05-09T08:41:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.512+0000 7f257609e640 1 Processor -- start 2024-05-09T08:41:34.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.512+0000 7f257609e640 1 -- start start 2024-05-09T08:41:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f257609e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2570100700 0x7f2570196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f257609e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25701010c0 0x7f25701974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f257609e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2570106710 0x7f257019b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f257609e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f25701109d0 con 0x7f2570100700 2024-05-09T08:41:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f257609e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2570110850 con 0x7f25701010c0 2024-05-09T08:41:34.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f257609e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2570110b50 con 0x7f2570106710 2024-05-09T08:41:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f256f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2570100700 0x7f2570196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f256f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2570100700 0x7f2570196f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57974/0 (socket says 172.21.15.107:57974) 2024-05-09T08:41:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f256f7fe640 1 -- 172.21.15.107:0/4169423460 learned_addr learned my addr 172.21.15.107:0/4169423460 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f256effd640 1 --2- 172.21.15.107:0/4169423460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25701010c0 0x7f25701974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f256ffff640 1 --2- 172.21.15.107:0/4169423460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2570106710 0x7f257019b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f256f7fe640 1 -- 172.21.15.107:0/4169423460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2570106710 msgr2=0x7f257019b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f256f7fe640 1 --2- 172.21.15.107:0/4169423460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2570106710 0x7f257019b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f256f7fe640 1 -- 172.21.15.107:0/4169423460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25701010c0 msgr2=0x7f25701974b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f256f7fe640 1 --2- 172.21.15.107:0/4169423460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25701010c0 0x7f25701974b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f256f7fe640 1 -- 172.21.15.107:0/4169423460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f257019bf10 con 0x7f2570100700 2024-05-09T08:41:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f256effd640 1 --2- 172.21.15.107:0/4169423460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25701010c0 0x7f25701974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.513+0000 7f256f7fe640 1 --2- 172.21.15.107:0/4169423460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2570100700 0x7f2570196f70 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f2560005940 tx=0x7f25600040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.514+0000 7f256cff9640 1 -- 172.21.15.107:0/4169423460 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560035050 con 0x7f2570100700 2024-05-09T08:41:34.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.514+0000 7f257609e640 1 -- 172.21.15.107:0/4169423460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f257019c1a0 con 0x7f2570100700 2024-05-09T08:41:34.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.514+0000 7f256cff9640 1 -- 172.21.15.107:0/4169423460 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25600044c0 con 0x7f2570100700 2024-05-09T08:41:34.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.514+0000 7f256cff9640 1 -- 172.21.15.107:0/4169423460 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560039c70 con 0x7f2570100700 2024-05-09T08:41:34.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.514+0000 7f257609e640 1 -- 172.21.15.107:0/4169423460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25700689c0 con 0x7f2570100700 2024-05-09T08:41:34.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.516+0000 7f256cff9640 1 -- 172.21.15.107:0/4169423460 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f25600394e0 con 0x7f2570100700 2024-05-09T08:41:34.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.516+0000 7f2578dac740 1 Processor -- start 2024-05-09T08:41:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.516+0000 7f2578dac740 1 -- start start 2024-05-09T08:41:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f2578dac740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x55563360b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f2578dac740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 0x555633615de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f2578dac740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555633616320 0x555633618710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f2578dac740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555633570c40 con 0x55563360bc50 2024-05-09T08:41:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f2578dac740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555633518580 con 0x555633616320 2024-05-09T08:41:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f2578dac740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55563356cc00 con 0x55563360b330 2024-05-09T08:41:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f256f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 0x555633615de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f256effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555633616320 0x555633618710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f256ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x55563360b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f256effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555633616320 0x555633618710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:59252/0 (socket says 172.21.15.107:59252) 2024-05-09T08:41:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f256effd640 1 -- 172.21.15.107:0/1928575463 learned_addr learned my addr 172.21.15.107:0/1928575463 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f256cff9640 1 --2- 172.21.15.107:0/4169423460 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f25580a2d00 0x7f25580a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f256ffff640 1 -- 172.21.15.107:0/1928575463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555633616320 msgr2=0x555633618710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f256ffff640 1 --2- 172.21.15.107:0/1928575463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555633616320 0x555633618710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f256ffff640 1 -- 172.21.15.107:0/1928575463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 msgr2=0x555633615de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f256ffff640 1 --2- 172.21.15.107:0/1928575463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 0x555633615de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f256ffff640 1 -- 172.21.15.107:0/1928575463 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556332843f0 con 0x55563360b330 2024-05-09T08:41:34.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f256ffff640 1 --2- 172.21.15.107:0/4169423460 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f25580a2d00 0x7f25580a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.517+0000 7f256cff9640 1 -- 172.21.15.107:0/4169423460 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f25600f1c50 con 0x7f2570100700 2024-05-09T08:41:34.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.518+0000 7f256ffff640 1 --2- 172.21.15.107:0/1928575463 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x55563360b710 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2568009f90 tx=0x7f2568010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=854f31f30f761d87 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.518+0000 7f2555ffb640 1 -- 172.21.15.107:0/1928575463 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2568024070 con 0x55563360b330 2024-05-09T08:41:34.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.518+0000 7f2555ffb640 1 -- 172.21.15.107:0/1928575463 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f256800ed40 con 0x55563360b330 2024-05-09T08:41:34.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.518+0000 7f2555ffb640 1 -- 172.21.15.107:0/1928575463 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256801ec70 con 0x55563360b330 2024-05-09T08:41:34.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.518+0000 7f256ffff640 1 --2- 172.21.15.107:0/4169423460 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f25580a2d00 0x7f25580a51c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2568021000 tx=0x7f256801e000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.518+0000 7f2578dac740 1 -- 172.21.15.107:0/1928575463 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 msgr2=0x55563360b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.518+0000 7f2578dac740 1 --2- 172.21.15.107:0/1928575463 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x55563360b710 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2568009f90 tx=0x7f2568010860 comp rx=0 tx=0).stop 2024-05-09T08:41:34.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.519+0000 7f2578dac740 1 -- 172.21.15.107:0/1928575463 shutdown_connections 2024-05-09T08:41:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.519+0000 7f2578dac740 1 --2- 172.21.15.107:0/1928575463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555633616320 0x555633618710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.519+0000 7f2578dac740 1 --2- 172.21.15.107:0/1928575463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 0x555633615de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.519+0000 7f2578dac740 1 --2- 172.21.15.107:0/1928575463 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x55563360b710 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.519+0000 7f2578dac740 1 -- 172.21.15.107:0/1928575463 >> 172.21.15.107:0/1928575463 conn(0x555633607650 msgr2=0x555633607a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.519+0000 7f2578dac740 1 -- 172.21.15.107:0/1928575463 shutdown_connections 2024-05-09T08:41:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.519+0000 7f2578dac740 1 -- 172.21.15.107:0/1928575463 wait complete. 2024-05-09T08:41:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.519+0000 7f2578dac740 1 -- start start 2024-05-09T08:41:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.519+0000 7f2578dac740 1 Processor -- start 2024-05-09T08:41:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.519+0000 7f2578dac740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x555633613230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.519+0000 7f2578dac740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55563360bc50 0x555633613770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f2578dac740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555633616320 0x555633613cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f2578dac740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555633619110 con 0x555633616320 2024-05-09T08:41:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f2578dac740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55563352bc40 con 0x55563360bc50 2024-05-09T08:41:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f2578dac740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x555633619290 con 0x55563360b330 2024-05-09T08:41:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f2578dac740 10 client.0 _ll_register_callbacks cb 0x555633283be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f256ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555633616320 0x555633613cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f256ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555633616320 0x555633613cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58004/0 (socket says 172.21.15.107:58004) 2024-05-09T08:41:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f256ffff640 1 -- 172.21.15.107:0/3358489299 learned_addr learned my addr 172.21.15.107:0/3358489299 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f256effd640 1 --2- 172.21.15.107:0/3358489299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55563360bc50 0x555633613770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f256f7fe640 1 --2- 172.21.15.107:0/3358489299 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x555633613230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f256effd640 1 -- 172.21.15.107:0/3358489299 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 msgr2=0x555633613230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f256effd640 1 --2- 172.21.15.107:0/3358489299 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x555633613230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f256effd640 1 -- 172.21.15.107:0/3358489299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555633616320 msgr2=0x555633613cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f256effd640 1 --2- 172.21.15.107:0/3358489299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555633616320 0x555633613cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f256effd640 1 -- 172.21.15.107:0/3358489299 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55563334e7d0 con 0x55563360bc50 2024-05-09T08:41:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f256ffff640 1 --2- 172.21.15.107:0/3358489299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555633616320 0x555633613cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f256effd640 1 --2- 172.21.15.107:0/3358489299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55563360bc50 0x555633613770 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2564012d20 tx=0x7f256400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=467c8ca79bfbadd1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f254effd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:41:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f254effd640 1 -- 172.21.15.107:0/3358489299 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564012110 con 0x55563360bc50 2024-05-09T08:41:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f2578dac740 1 -- 172.21.15.107:0/3358489299 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f256800ed40 con 0x55563360bc50 2024-05-09T08:41:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.520+0000 7f254effd640 1 -- 172.21.15.107:0/3358489299 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f2564004050 con 0x55563360bc50 2024-05-09T08:41:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.521+0000 7f254effd640 1 -- 172.21.15.107:0/3358489299 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f25640122b0 con 0x55563360bc50 2024-05-09T08:41:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.521+0000 7f2578dac740 10 client.27029 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.521+0000 7f2578dac740 1 -- 172.21.15.107:0/3358489299 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555633625350 con 0x55563360bc50 2024-05-09T08:41:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.521+0000 7f254effd640 1 -- 172.21.15.107:0/3358489299 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f256800ed40 con 0x55563360bc50 2024-05-09T08:41:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.521+0000 7f254effd640 1 -- 172.21.15.107:0/3358489299 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f2564010430 con 0x55563360bc50 2024-05-09T08:41:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.521+0000 7f2578dac740 10 client.27029 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.521+0000 7f2578dac740 10 client.27029 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:41:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.521+0000 7f2578dac740 20 client.27029 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:34.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.521+0000 7f2578dac740 1 --2- 172.21.15.107:0/3358489299 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5556336266f0 0x555633628b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.521+0000 7f2578dac740 4 client.27029 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:34.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.521+0000 7f2578dac740 1 -- 172.21.15.107:0/3358489299 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55563357e370 con 0x5556336266f0 2024-05-09T08:41:34.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.522+0000 7f256f7fe640 1 --2- 172.21.15.107:0/3358489299 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5556336266f0 0x555633628b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.522+0000 7f254effd640 10 client.27029 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:41:34.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.522+0000 7f256f7fe640 1 --2- 172.21.15.107:0/3358489299 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5556336266f0 0x555633628b50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c35f07e59a8611bb server_cookie=7acecf7353f2dec in_seq=0 out_seq=0 2024-05-09T08:41:34.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.522+0000 7f254effd640 1 -- 172.21.15.107:0/3358489299 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55563357e370 con 0x5556336266f0 2024-05-09T08:41:34.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.522+0000 7f254effd640 10 client.27029 handle_command_reply: tid=0 2024-05-09T08:41:34.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.523+0000 7f2578dac740 1 client.27029 shutdown 2024-05-09T08:41:34.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.523+0000 7f2578dac740 1 -- 172.21.15.107:0/3358489299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55563360bc50 msgr2=0x555633613770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.523+0000 7f2578dac740 1 --2- 172.21.15.107:0/3358489299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55563360bc50 0x555633613770 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2564012d20 tx=0x7f256400f790 comp rx=0 tx=0).stop 2024-05-09T08:41:34.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.523+0000 7f2578dac740 1 -- 172.21.15.107:0/3358489299 shutdown_connections 2024-05-09T08:41:34.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.523+0000 7f2578dac740 1 --2- 172.21.15.107:0/3358489299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555633616320 0x555633613cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.523+0000 7f2578dac740 1 --2- 172.21.15.107:0/3358489299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55563360bc50 0x555633613770 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.523+0000 7f2578dac740 1 --2- 172.21.15.107:0/3358489299 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5556336266f0 0x555633628b50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.523+0000 7f2578dac740 1 --2- 172.21.15.107:0/3358489299 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x555633613230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.523+0000 7f254effd640 0 client.27029 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:41:34.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.523+0000 7f2578dac740 1 -- 172.21.15.107:0/3358489299 >> 172.21.15.107:0/3358489299 conn(0x555633607650 msgr2=0x55563360d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.523+0000 7f2578dac740 1 -- 172.21.15.107:0/3358489299 shutdown_connections 2024-05-09T08:41:34.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.523+0000 7f2578dac740 1 -- 172.21.15.107:0/3358489299 wait complete. 2024-05-09T08:41:34.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.523+0000 7f2578dac740 20 client.27029 trim_cache size 0 max 16384 2024-05-09T08:41:34.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.541+0000 7f2578dac740 1 Processor -- start 2024-05-09T08:41:34.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.541+0000 7f2578dac740 1 -- start start 2024-05-09T08:41:34.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.541+0000 7f2578dac740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x555633621fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.541+0000 7f2578dac740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 0x55563362e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.541+0000 7f2578dac740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555633616320 0x55563362e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.541+0000 7f2578dac740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f25701af2d0 con 0x55563360bc50 2024-05-09T08:41:34.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.541+0000 7f2578dac740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f25701acba0 con 0x555633616320 2024-05-09T08:41:34.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.541+0000 7f2578dac740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f25701aaa10 con 0x55563360b330 2024-05-09T08:41:34.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.541+0000 7f256effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 0x55563362e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.541+0000 7f256effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 0x55563362e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58012/0 (socket says 172.21.15.107:58012) 2024-05-09T08:41:34.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.541+0000 7f256effd640 1 -- 172.21.15.107:0/3945024098 learned_addr learned my addr 172.21.15.107:0/3945024098 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:34.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.541+0000 7f256f7fe640 1 --2- 172.21.15.107:0/3945024098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x555633621fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.541+0000 7f256ffff640 1 --2- 172.21.15.107:0/3945024098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555633616320 0x55563362e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.542+0000 7f256effd640 1 -- 172.21.15.107:0/3945024098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 msgr2=0x555633621fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.542+0000 7f256effd640 1 --2- 172.21.15.107:0/3945024098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x555633621fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.542+0000 7f256effd640 1 -- 172.21.15.107:0/3945024098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555633616320 msgr2=0x55563362e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.542+0000 7f256effd640 1 --2- 172.21.15.107:0/3945024098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555633616320 0x55563362e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.542+0000 7f256effd640 1 -- 172.21.15.107:0/3945024098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555633607480 con 0x55563360bc50 2024-05-09T08:41:34.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.542+0000 7f256f7fe640 1 --2- 172.21.15.107:0/3945024098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x555633621fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:34.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.542+0000 7f256effd640 1 --2- 172.21.15.107:0/3945024098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 0x55563362e420 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f2564001eb0 tx=0x7f25640057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67e407b7b81324b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.542+0000 7f25557fa640 1 -- 172.21.15.107:0/3945024098 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256402e070 con 0x55563360bc50 2024-05-09T08:41:34.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.542+0000 7f25557fa640 1 -- 172.21.15.107:0/3945024098 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2564005590 con 0x55563360bc50 2024-05-09T08:41:34.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.542+0000 7f25557fa640 1 -- 172.21.15.107:0/3945024098 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564010e60 con 0x55563360bc50 2024-05-09T08:41:34.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.543+0000 7f2578dac740 1 -- 172.21.15.107:0/3945024098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 msgr2=0x55563362e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.543+0000 7f2578dac740 1 --2- 172.21.15.107:0/3945024098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 0x55563362e420 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f2564001eb0 tx=0x7f25640057d0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.543+0000 7f2578dac740 1 -- 172.21.15.107:0/3945024098 shutdown_connections 2024-05-09T08:41:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.543+0000 7f2578dac740 1 --2- 172.21.15.107:0/3945024098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555633616320 0x55563362e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.543+0000 7f2578dac740 1 --2- 172.21.15.107:0/3945024098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 0x55563362e420 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.543+0000 7f2578dac740 1 --2- 172.21.15.107:0/3945024098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x555633621fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.543+0000 7f2578dac740 1 -- 172.21.15.107:0/3945024098 >> 172.21.15.107:0/3945024098 conn(0x555633607650 msgr2=0x55563360e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.543+0000 7f2578dac740 1 -- 172.21.15.107:0/3945024098 shutdown_connections 2024-05-09T08:41:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.543+0000 7f2578dac740 1 -- 172.21.15.107:0/3945024098 wait complete. 2024-05-09T08:41:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.543+0000 7f2578dac740 1 -- start start 2024-05-09T08:41:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.544+0000 7f2578dac740 1 Processor -- start 2024-05-09T08:41:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.544+0000 7f2578dac740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x55563361ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.544+0000 7f2578dac740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 0x555633610930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.544+0000 7f2578dac740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555633616320 0x55563360f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.544+0000 7f2578dac740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f25701ab060 con 0x55563360bc50 2024-05-09T08:41:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.544+0000 7f2578dac740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f25701ac4d0 con 0x555633616320 2024-05-09T08:41:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.544+0000 7f2578dac740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f25701ab730 con 0x55563360b330 2024-05-09T08:41:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.544+0000 7f2578dac740 10 client.0 _ll_register_callbacks cb 0x7f2564009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.544+0000 7f256effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 0x555633610930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.544+0000 7f256effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 0x555633610930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58014/0 (socket says 172.21.15.107:58014) 2024-05-09T08:41:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.544+0000 7f256effd640 1 -- 172.21.15.107:0/2426266809 learned_addr learned my addr 172.21.15.107:0/2426266809 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.544+0000 7f256f7fe640 1 --2- 172.21.15.107:0/2426266809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x55563361ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.545+0000 7f256effd640 1 -- 172.21.15.107:0/2426266809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 msgr2=0x55563361ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.545+0000 7f256effd640 1 --2- 172.21.15.107:0/2426266809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x55563361ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.545+0000 7f256effd640 1 -- 172.21.15.107:0/2426266809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555633616320 msgr2=0x55563360f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.545+0000 7f256effd640 1 --2- 172.21.15.107:0/2426266809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555633616320 0x55563360f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.545+0000 7f256effd640 1 -- 172.21.15.107:0/2426266809 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55563360f9e0 con 0x55563360bc50 2024-05-09T08:41:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.545+0000 7f256f7fe640 1 --2- 172.21.15.107:0/2426266809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x55563361ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.545+0000 7f256effd640 1 --2- 172.21.15.107:0/2426266809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 0x555633610930 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f2564010940 tx=0x7f25640106e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20910a9da293713 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.545+0000 7f254effd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.545+0000 7f254effd640 1 -- 172.21.15.107:0/2426266809 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256402e070 con 0x55563360bc50 2024-05-09T08:41:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.545+0000 7f2578dac740 1 -- 172.21.15.107:0/2426266809 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2564005590 con 0x55563360bc50 2024-05-09T08:41:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.545+0000 7f254effd640 1 -- 172.21.15.107:0/2426266809 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f25640035f0 con 0x55563360bc50 2024-05-09T08:41:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.545+0000 7f254effd640 1 -- 172.21.15.107:0/2426266809 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2564010e60 con 0x55563360bc50 2024-05-09T08:41:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.545+0000 7f2578dac740 10 client.18060 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.545+0000 7f2578dac740 1 -- 172.21.15.107:0/2426266809 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55563368c2b0 con 0x55563360bc50 2024-05-09T08:41:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.545+0000 7f254effd640 1 -- 172.21.15.107:0/2426266809 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2564005590 con 0x55563360bc50 2024-05-09T08:41:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.546+0000 7f254effd640 1 -- 172.21.15.107:0/2426266809 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f2564029850 con 0x55563360bc50 2024-05-09T08:41:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.546+0000 7f2578dac740 10 client.18060 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.546+0000 7f2578dac740 10 client.18060 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:41:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.546+0000 7f2578dac740 20 client.18060 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.546+0000 7f2578dac740 1 --2- 172.21.15.107:0/2426266809 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55563368c990 0x55563368edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.546+0000 7f2578dac740 4 client.18060 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:41:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.546+0000 7f2578dac740 1 -- 172.21.15.107:0/2426266809 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5556335786c0 con 0x55563368c990 2024-05-09T08:41:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.546+0000 7f256f7fe640 1 --2- 172.21.15.107:0/2426266809 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55563368c990 0x55563368edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.547+0000 7f254effd640 10 client.18060 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:41:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.547+0000 7f256f7fe640 1 --2- 172.21.15.107:0/2426266809 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55563368c990 0x55563368edb0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=63a1a7bb0590db7f server_cookie=938b2a3cdcd9cf85 in_seq=0 out_seq=0 2024-05-09T08:41:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.547+0000 7f254effd640 1 -- 172.21.15.107:0/2426266809 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5556335786c0 con 0x55563368c990 2024-05-09T08:41:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.547+0000 7f254effd640 10 client.18060 handle_command_reply: tid=0 2024-05-09T08:41:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.547+0000 7f2578dac740 1 client.18060 shutdown 2024-05-09T08:41:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.547+0000 7f2578dac740 1 -- 172.21.15.107:0/2426266809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 msgr2=0x555633610930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.547+0000 7f2578dac740 1 --2- 172.21.15.107:0/2426266809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 0x555633610930 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f2564010940 tx=0x7f25640106e0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.547+0000 7f2578dac740 1 -- 172.21.15.107:0/2426266809 shutdown_connections 2024-05-09T08:41:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.547+0000 7f2578dac740 1 --2- 172.21.15.107:0/2426266809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555633616320 0x55563360f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.547+0000 7f2578dac740 1 --2- 172.21.15.107:0/2426266809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55563360bc50 0x555633610930 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.547+0000 7f2578dac740 1 --2- 172.21.15.107:0/2426266809 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55563368c990 0x55563368edb0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.547+0000 7f2578dac740 1 --2- 172.21.15.107:0/2426266809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55563360b330 0x55563361ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.547+0000 7f2578dac740 1 -- 172.21.15.107:0/2426266809 >> 172.21.15.107:0/2426266809 conn(0x555633607650 msgr2=0x555633611b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.547+0000 7f254effd640 0 client.18060 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:41:34.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.547+0000 7f2578dac740 1 -- 172.21.15.107:0/2426266809 shutdown_connections 2024-05-09T08:41:34.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.547+0000 7f2578dac740 1 -- 172.21.15.107:0/2426266809 wait complete. 2024-05-09T08:41:34.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.548+0000 7f2578dac740 20 client.18060 trim_cache size 0 max 16384 2024-05-09T08:41:34.552 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:41:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.548+0000 7f257609e640 1 -- 172.21.15.107:0/4169423460 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f25580a2d00 msgr2=0x7f25580a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.548+0000 7f257609e640 1 --2- 172.21.15.107:0/4169423460 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f25580a2d00 0x7f25580a51c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2568021000 tx=0x7f256801e000 comp rx=0 tx=0).stop 2024-05-09T08:41:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.548+0000 7f257609e640 1 -- 172.21.15.107:0/4169423460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2570100700 msgr2=0x7f2570196f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.548+0000 7f257609e640 1 --2- 172.21.15.107:0/4169423460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2570100700 0x7f2570196f70 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f2560005940 tx=0x7f25600040c0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.548+0000 7f257609e640 1 -- 172.21.15.107:0/4169423460 shutdown_connections 2024-05-09T08:41:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.548+0000 7f257609e640 1 --2- 172.21.15.107:0/4169423460 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f25580a2d00 0x7f25580a51c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.548+0000 7f257609e640 1 --2- 172.21.15.107:0/4169423460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2570106710 0x7f257019b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.548+0000 7f257609e640 1 --2- 172.21.15.107:0/4169423460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25701010c0 0x7f25701974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.548+0000 7f257609e640 1 --2- 172.21.15.107:0/4169423460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2570100700 0x7f2570196f70 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.548+0000 7f257609e640 1 -- 172.21.15.107:0/4169423460 >> 172.21.15.107:0/4169423460 conn(0x7f2570077830 msgr2=0x7f2570103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.548+0000 7f257609e640 1 -- 172.21.15.107:0/4169423460 shutdown_connections 2024-05-09T08:41:34.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.548+0000 7f257609e640 1 -- 172.21.15.107:0/4169423460 wait complete. 2024-05-09T08:41:34.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:41:34.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.677+0000 7ff1bad9e640 1 -- 172.21.15.107:0/4171759212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1b4102890 msgr2=0x7ff1b4102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.677+0000 7ff1bad9e640 1 --2- 172.21.15.107:0/4171759212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1b4102890 0x7ff1b4102c70 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0009900 tx=0x7ff1b002a480 comp rx=0 tx=0).stop 2024-05-09T08:41:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.677+0000 7ff1bad9e640 1 -- 172.21.15.107:0/4171759212 shutdown_connections 2024-05-09T08:41:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.677+0000 7ff1bad9e640 1 --2- 172.21.15.107:0/4171759212 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1b410bcb0 0x7ff1b410e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.677+0000 7ff1bad9e640 1 --2- 172.21.15.107:0/4171759212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1b41031b0 0x7ff1b410b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.677+0000 7ff1bad9e640 1 --2- 172.21.15.107:0/4171759212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1b4102890 0x7ff1b4102c70 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.677+0000 7ff1bad9e640 1 -- 172.21.15.107:0/4171759212 >> 172.21.15.107:0/4171759212 conn(0x7ff1b40fc0b0 msgr2=0x7ff1b40fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.677+0000 7ff1bad9e640 1 -- 172.21.15.107:0/4171759212 shutdown_connections 2024-05-09T08:41:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.677+0000 7ff1bad9e640 1 -- 172.21.15.107:0/4171759212 wait complete. 2024-05-09T08:41:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1bad9e640 1 Processor -- start 2024-05-09T08:41:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1bad9e640 1 -- start start 2024-05-09T08:41:34.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1bad9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1b4102890 0x7ff1b4197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1bad9e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1b41031b0 0x7ff1b4198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1bad9e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1b410bcb0 0x7ff1b419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1bad9e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b4111960 con 0x7ff1b4102890 2024-05-09T08:41:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1bad9e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b41117e0 con 0x7ff1b410bcb0 2024-05-09T08:41:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1bad9e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b4111ae0 con 0x7ff1b41031b0 2024-05-09T08:41:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1b9d9c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1b4102890 0x7ff1b4197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1b9d9c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1b4102890 0x7ff1b4197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58034/0 (socket says 172.21.15.107:58034) 2024-05-09T08:41:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1b9d9c640 1 -- 172.21.15.107:0/936504868 learned_addr learned my addr 172.21.15.107:0/936504868 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1b959b640 1 --2- 172.21.15.107:0/936504868 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1b41031b0 0x7ff1b4198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1ba59d640 1 --2- 172.21.15.107:0/936504868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1b410bcb0 0x7ff1b419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1b9d9c640 1 -- 172.21.15.107:0/936504868 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1b41031b0 msgr2=0x7ff1b4198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1b9d9c640 1 --2- 172.21.15.107:0/936504868 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1b41031b0 0x7ff1b4198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1b9d9c640 1 -- 172.21.15.107:0/936504868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1b410bcb0 msgr2=0x7ff1b419c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1b9d9c640 1 --2- 172.21.15.107:0/936504868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1b410bcb0 0x7ff1b419c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.678+0000 7ff1b9d9c640 1 -- 172.21.15.107:0/936504868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b419ceb0 con 0x7ff1b4102890 2024-05-09T08:41:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.679+0000 7ff1b9d9c640 1 --2- 172.21.15.107:0/936504868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1b4102890 0x7ff1b4197ec0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0005940 tx=0x7ff1b00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.679+0000 7ff1aaffd640 1 -- 172.21.15.107:0/936504868 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b003f030 con 0x7ff1b4102890 2024-05-09T08:41:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.679+0000 7ff1aaffd640 1 -- 172.21.15.107:0/936504868 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff1b0003cb0 con 0x7ff1b4102890 2024-05-09T08:41:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.679+0000 7ff1bad9e640 1 -- 172.21.15.107:0/936504868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b419d140 con 0x7ff1b4102890 2024-05-09T08:41:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.679+0000 7ff1aaffd640 1 -- 172.21.15.107:0/936504868 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b0035070 con 0x7ff1b4102890 2024-05-09T08:41:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.679+0000 7ff1bad9e640 1 -- 172.21.15.107:0/936504868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1b41a4a30 con 0x7ff1b4102890 2024-05-09T08:41:34.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.681+0000 7ff1aaffd640 1 -- 172.21.15.107:0/936504868 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ff1b0042030 con 0x7ff1b4102890 2024-05-09T08:41:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.682+0000 7ff1bedee740 1 Processor -- start 2024-05-09T08:41:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.682+0000 7ff1bedee740 1 -- start start 2024-05-09T08:41:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.682+0000 7ff1aaffd640 1 --2- 172.21.15.107:0/936504868 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff1a00a2d00 0x7ff1a00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.682+0000 7ff1bedee740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cda3d0 0x55c9a6cda7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.682+0000 7ff1bedee740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6cdacf0 0x55c9a6ce93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.682+0000 7ff1bedee740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6ce98f0 0x55c9a6cebce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.682+0000 7ff1bedee740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c9a6c3fc40 con 0x55c9a6cda3d0 2024-05-09T08:41:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.682+0000 7ff1bedee740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c9a6be7580 con 0x55c9a6cdacf0 2024-05-09T08:41:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1bedee740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c9a6c3bc00 con 0x55c9a6ce98f0 2024-05-09T08:41:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1ba59d640 1 --2- 172.21.15.107:0/936504868 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff1a00a2d00 0x7ff1a00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1b9d9c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cda3d0 0x55c9a6cda7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1b9d9c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cda3d0 0x55c9a6cda7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58042/0 (socket says 172.21.15.107:58042) 2024-05-09T08:41:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1b9d9c640 1 -- 172.21.15.107:0/827077949 learned_addr learned my addr 172.21.15.107:0/827077949 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1aaffd640 1 -- 172.21.15.107:0/936504868 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7ff1b00efbe0 con 0x7ff1b4102890 2024-05-09T08:41:34.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1ba59d640 1 --2- 172.21.15.107:0/827077949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6ce98f0 0x55c9a6cebce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1b959b640 1 --2- 172.21.15.107:0/827077949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6cdacf0 0x55c9a6ce93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1b9d9c640 1 -- 172.21.15.107:0/827077949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6ce98f0 msgr2=0x55c9a6cebce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1b9d9c640 1 --2- 172.21.15.107:0/827077949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6ce98f0 0x55c9a6cebce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1b9d9c640 1 -- 172.21.15.107:0/827077949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6cdacf0 msgr2=0x55c9a6ce93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1b9d9c640 1 --2- 172.21.15.107:0/827077949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6cdacf0 0x55c9a6ce93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1b9d9c640 1 -- 172.21.15.107:0/827077949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9a69533f0 con 0x55c9a6cda3d0 2024-05-09T08:41:34.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1b959b640 1 --2- 172.21.15.107:0/827077949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6cdacf0 0x55c9a6ce93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:34.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1ba59d640 1 --2- 172.21.15.107:0/827077949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6ce98f0 0x55c9a6cebce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:34.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1ba59d640 1 --2- 172.21.15.107:0/936504868 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff1a00a2d00 0x7ff1a00a51c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac005fd0 tx=0x7ff1ac014910 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff1b9d9c640 1 --2- 172.21.15.107:0/827077949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cda3d0 0x55c9a6cda7b0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0006010 tx=0x7ff1b00bbc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71b20e03faa5d606 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff19ffff640 1 -- 172.21.15.107:0/827077949 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b0049040 con 0x55c9a6cda3d0 2024-05-09T08:41:34.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff19ffff640 1 -- 172.21.15.107:0/827077949 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff1b003b460 con 0x55c9a6cda3d0 2024-05-09T08:41:34.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.683+0000 7ff19ffff640 1 -- 172.21.15.107:0/827077949 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b00ba1d0 con 0x55c9a6cda3d0 2024-05-09T08:41:34.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.684+0000 7ff1bedee740 1 -- 172.21.15.107:0/827077949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cda3d0 msgr2=0x55c9a6cda7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.684+0000 7ff1bedee740 1 --2- 172.21.15.107:0/827077949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cda3d0 0x55c9a6cda7b0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0006010 tx=0x7ff1b00bbc90 comp rx=0 tx=0).stop 2024-05-09T08:41:34.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.684+0000 7ff1bedee740 1 -- 172.21.15.107:0/827077949 shutdown_connections 2024-05-09T08:41:34.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.684+0000 7ff1bedee740 1 --2- 172.21.15.107:0/827077949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6ce98f0 0x55c9a6cebce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.684+0000 7ff1bedee740 1 --2- 172.21.15.107:0/827077949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6cdacf0 0x55c9a6ce93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.684+0000 7ff1bedee740 1 --2- 172.21.15.107:0/827077949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cda3d0 0x55c9a6cda7b0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.684+0000 7ff1bedee740 1 -- 172.21.15.107:0/827077949 >> 172.21.15.107:0/827077949 conn(0x55c9a6cd6810 msgr2=0x55c9a6cd6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.684+0000 7ff1bedee740 1 -- 172.21.15.107:0/827077949 shutdown_connections 2024-05-09T08:41:34.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.684+0000 7ff1bedee740 1 -- 172.21.15.107:0/827077949 wait complete. 2024-05-09T08:41:34.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.685+0000 7ff1bedee740 1 -- start start 2024-05-09T08:41:34.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.685+0000 7ff1bedee740 1 Processor -- start 2024-05-09T08:41:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.685+0000 7ff1bedee740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6cda3d0 0x55c9a6ce67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.685+0000 7ff1bedee740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cdacf0 0x55c9a6ce6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.685+0000 7ff1bedee740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6ce98f0 0x55c9a6ce7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.685+0000 7ff1bedee740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c9a6cec4f0 con 0x55c9a6cdacf0 2024-05-09T08:41:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.685+0000 7ff1bedee740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c9a6bfac40 con 0x55c9a6ce98f0 2024-05-09T08:41:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.685+0000 7ff1bedee740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c9a6cec670 con 0x55c9a6cda3d0 2024-05-09T08:41:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1bedee740 10 client.0 _ll_register_callbacks cb 0x55c9a6952be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1b959b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cdacf0 0x55c9a6ce6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1b959b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cdacf0 0x55c9a6ce6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58046/0 (socket says 172.21.15.107:58046) 2024-05-09T08:41:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1b959b640 1 -- 172.21.15.107:0/1863449509 learned_addr learned my addr 172.21.15.107:0/1863449509 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1ba59d640 1 --2- 172.21.15.107:0/1863449509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6ce98f0 0x55c9a6ce7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1b9d9c640 1 --2- 172.21.15.107:0/1863449509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6cda3d0 0x55c9a6ce67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1b959b640 1 -- 172.21.15.107:0/1863449509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6cda3d0 msgr2=0x55c9a6ce67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1b959b640 1 --2- 172.21.15.107:0/1863449509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6cda3d0 0x55c9a6ce67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1b959b640 1 -- 172.21.15.107:0/1863449509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6ce98f0 msgr2=0x55c9a6ce7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1b959b640 1 --2- 172.21.15.107:0/1863449509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6ce98f0 0x55c9a6ce7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1b959b640 1 -- 172.21.15.107:0/1863449509 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9a6a1d7d0 con 0x55c9a6cdacf0 2024-05-09T08:41:34.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1ba59d640 1 --2- 172.21.15.107:0/1863449509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6ce98f0 0x55c9a6ce7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:34.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1b9d9c640 1 --2- 172.21.15.107:0/1863449509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6cda3d0 0x55c9a6ce67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:34.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1b959b640 1 --2- 172.21.15.107:0/1863449509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cdacf0 0x55c9a6ce6d00 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7ff1a40136e0 tx=0x7ff1a4013ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=496a67db2be86835 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff19d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:34.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff19d7fa640 1 -- 172.21.15.107:0/1863449509 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4020060 con 0x55c9a6cdacf0 2024-05-09T08:41:34.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff19d7fa640 1 -- 172.21.15.107:0/1863449509 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7ff1a4011bc0 con 0x55c9a6cdacf0 2024-05-09T08:41:34.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1bedee740 1 -- 172.21.15.107:0/1863449509 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c9a6c4d370 con 0x55c9a6cdacf0 2024-05-09T08:41:34.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff19d7fa640 1 -- 172.21.15.107:0/1863449509 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff1a4011dc0 con 0x55c9a6cdacf0 2024-05-09T08:41:34.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1bedee740 10 client.18072 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:34.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.686+0000 7ff1bedee740 1 -- 172.21.15.107:0/1863449509 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c9a6cf85f0 con 0x55c9a6cdacf0 2024-05-09T08:41:34.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.687+0000 7ff19d7fa640 1 -- 172.21.15.107:0/1863449509 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55c9a6c4d370 con 0x55c9a6cdacf0 2024-05-09T08:41:34.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.687+0000 7ff19d7fa640 1 -- 172.21.15.107:0/1863449509 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff1a400f3c0 con 0x55c9a6cdacf0 2024-05-09T08:41:34.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.687+0000 7ff1bedee740 10 client.18072 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:34.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.687+0000 7ff1bedee740 10 client.18072 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:34.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.687+0000 7ff1bedee740 20 client.18072 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:34.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.687+0000 7ff1bedee740 1 --2- 172.21.15.107:0/1863449509 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c9a6cf9ab0 0x55c9a6cfbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.687+0000 7ff1bedee740 4 client.18072 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:34.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.687+0000 7ff1bedee740 1 -- 172.21.15.107:0/1863449509 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9a6c476c0 con 0x55c9a6cf9ab0 2024-05-09T08:41:34.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.687+0000 7ff1b9d9c640 1 --2- 172.21.15.107:0/1863449509 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c9a6cf9ab0 0x55c9a6cfbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.688+0000 7ff19d7fa640 10 client.18072 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:34.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.688+0000 7ff1b9d9c640 1 --2- 172.21.15.107:0/1863449509 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c9a6cf9ab0 0x55c9a6cfbee0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4de8721d77976b8a server_cookie=5c46552129e6256f in_seq=0 out_seq=0 2024-05-09T08:41:34.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.688+0000 7ff19d7fa640 1 -- 172.21.15.107:0/1863449509 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c9a6c476c0 con 0x55c9a6cf9ab0 2024-05-09T08:41:34.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.688+0000 7ff19d7fa640 10 client.18072 handle_command_reply: tid=0 2024-05-09T08:41:34.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.688+0000 7ff1bedee740 1 client.18072 shutdown 2024-05-09T08:41:34.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.688+0000 7ff1bedee740 1 -- 172.21.15.107:0/1863449509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cdacf0 msgr2=0x55c9a6ce6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.688+0000 7ff1bedee740 1 --2- 172.21.15.107:0/1863449509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cdacf0 0x55c9a6ce6d00 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7ff1a40136e0 tx=0x7ff1a4013ba0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.689+0000 7ff1bedee740 1 -- 172.21.15.107:0/1863449509 shutdown_connections 2024-05-09T08:41:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.689+0000 7ff1bedee740 1 --2- 172.21.15.107:0/1863449509 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c9a6cf9ab0 0x55c9a6cfbee0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.689+0000 7ff1bedee740 1 --2- 172.21.15.107:0/1863449509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6ce98f0 0x55c9a6ce7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.689+0000 7ff1bedee740 1 --2- 172.21.15.107:0/1863449509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cdacf0 0x55c9a6ce6d00 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.689+0000 7ff19d7fa640 0 client.18072 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.689+0000 7ff1bedee740 1 --2- 172.21.15.107:0/1863449509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6cda3d0 0x55c9a6ce67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.689+0000 7ff1bedee740 1 -- 172.21.15.107:0/1863449509 >> 172.21.15.107:0/1863449509 conn(0x55c9a6cd6810 msgr2=0x55c9a6cdc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.689+0000 7ff1bedee740 1 -- 172.21.15.107:0/1863449509 shutdown_connections 2024-05-09T08:41:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.689+0000 7ff1bedee740 1 -- 172.21.15.107:0/1863449509 wait complete. 2024-05-09T08:41:34.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.689+0000 7ff1bedee740 20 client.18072 trim_cache size 0 max 16384 2024-05-09T08:41:34.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.707+0000 7ff1bedee740 1 Processor -- start 2024-05-09T08:41:34.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.707+0000 7ff1bedee740 1 -- start start 2024-05-09T08:41:34.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.707+0000 7ff1bedee740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cda3d0 0x55c9a6cf53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.707+0000 7ff1bedee740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6cdacf0 0x55c9a6cd6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.707+0000 7ff1bedee740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6ce98f0 0x55c9a6d017f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.707+0000 7ff1bedee740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b41af140 con 0x55c9a6cda3d0 2024-05-09T08:41:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.707+0000 7ff1bedee740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b41aca10 con 0x55c9a6ce98f0 2024-05-09T08:41:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.707+0000 7ff1bedee740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b41aa880 con 0x55c9a6cdacf0 2024-05-09T08:41:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.707+0000 7ff1b9d9c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cda3d0 0x55c9a6cf53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.708+0000 7ff1b9d9c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cda3d0 0x55c9a6cf53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58054/0 (socket says 172.21.15.107:58054) 2024-05-09T08:41:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.708+0000 7ff1ba59d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6ce98f0 0x55c9a6d017f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.708+0000 7ff1b9d9c640 1 -- 172.21.15.107:0/1372930958 learned_addr learned my addr 172.21.15.107:0/1372930958 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.708+0000 7ff1b959b640 1 --2- 172.21.15.107:0/1372930958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6cdacf0 0x55c9a6cd6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.708+0000 7ff1b9d9c640 1 -- 172.21.15.107:0/1372930958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6cdacf0 msgr2=0x55c9a6cd6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.708+0000 7ff1b9d9c640 1 --2- 172.21.15.107:0/1372930958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6cdacf0 0x55c9a6cd6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.708+0000 7ff1b9d9c640 1 -- 172.21.15.107:0/1372930958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6ce98f0 msgr2=0x55c9a6d017f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.708+0000 7ff1b9d9c640 1 --2- 172.21.15.107:0/1372930958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6ce98f0 0x55c9a6d017f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.708+0000 7ff1b9d9c640 1 -- 172.21.15.107:0/1372930958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9a6cdba60 con 0x55c9a6cda3d0 2024-05-09T08:41:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.708+0000 7ff1ba59d640 1 --2- 172.21.15.107:0/1372930958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6ce98f0 0x55c9a6d017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.708+0000 7ff1b9d9c640 1 --2- 172.21.15.107:0/1372930958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cda3d0 0x55c9a6cf53b0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7ff1b00ed030 tx=0x7ff1b002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e94009e531b1366 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.708+0000 7ff19f7fe640 1 -- 172.21.15.107:0/1372930958 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b0049040 con 0x55c9a6cda3d0 2024-05-09T08:41:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.708+0000 7ff19f7fe640 1 -- 172.21.15.107:0/1372930958 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff1b00bbe00 con 0x55c9a6cda3d0 2024-05-09T08:41:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.708+0000 7ff19f7fe640 1 -- 172.21.15.107:0/1372930958 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b00ba190 con 0x55c9a6cda3d0 2024-05-09T08:41:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.709+0000 7ff1bedee740 1 -- 172.21.15.107:0/1372930958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cda3d0 msgr2=0x55c9a6cf53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.709+0000 7ff1bedee740 1 --2- 172.21.15.107:0/1372930958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cda3d0 0x55c9a6cf53b0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7ff1b00ed030 tx=0x7ff1b002a480 comp rx=0 tx=0).stop 2024-05-09T08:41:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.709+0000 7ff1bedee740 1 -- 172.21.15.107:0/1372930958 shutdown_connections 2024-05-09T08:41:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.709+0000 7ff1bedee740 1 --2- 172.21.15.107:0/1372930958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6ce98f0 0x55c9a6d017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.709+0000 7ff1bedee740 1 --2- 172.21.15.107:0/1372930958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6cdacf0 0x55c9a6cd6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.709+0000 7ff1bedee740 1 --2- 172.21.15.107:0/1372930958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cda3d0 0x55c9a6cf53b0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.709+0000 7ff1bedee740 1 -- 172.21.15.107:0/1372930958 >> 172.21.15.107:0/1372930958 conn(0x55c9a6cd6810 msgr2=0x55c9a6ce7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.709+0000 7ff1bedee740 1 -- 172.21.15.107:0/1372930958 shutdown_connections 2024-05-09T08:41:34.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.709+0000 7ff1bedee740 1 -- 172.21.15.107:0/1372930958 wait complete. 2024-05-09T08:41:34.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.709+0000 7ff1bedee740 1 -- start start 2024-05-09T08:41:34.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.709+0000 7ff1bedee740 1 Processor -- start 2024-05-09T08:41:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1bedee740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6cda3d0 0x55c9a6ce3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1bedee740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cdacf0 0x55c9a6ce3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1bedee740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6ce98f0 0x55c9a6ce40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1bedee740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b41aaed0 con 0x55c9a6cdacf0 2024-05-09T08:41:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1bedee740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b41ac340 con 0x55c9a6cda3d0 2024-05-09T08:41:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1bedee740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b41ab5a0 con 0x55c9a6ce98f0 2024-05-09T08:41:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1bedee740 10 client.0 _ll_register_callbacks cb 0x7ff1a4009790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1b959b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cdacf0 0x55c9a6ce3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1b9d9c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6cda3d0 0x55c9a6ce3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1b959b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cdacf0 0x55c9a6ce3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58064/0 (socket says 172.21.15.107:58064) 2024-05-09T08:41:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1b959b640 1 -- 172.21.15.107:0/2796941831 learned_addr learned my addr 172.21.15.107:0/2796941831 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1ba59d640 1 --2- 172.21.15.107:0/2796941831 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6ce98f0 0x55c9a6ce40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1b9d9c640 1 -- 172.21.15.107:0/2796941831 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6ce98f0 msgr2=0x55c9a6ce40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1b9d9c640 1 --2- 172.21.15.107:0/2796941831 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6ce98f0 0x55c9a6ce40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1b9d9c640 1 -- 172.21.15.107:0/2796941831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cdacf0 msgr2=0x55c9a6ce3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1b9d9c640 1 --2- 172.21.15.107:0/2796941831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cdacf0 0x55c9a6ce3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1b9d9c640 1 -- 172.21.15.107:0/2796941831 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9a6ce1580 con 0x55c9a6cda3d0 2024-05-09T08:41:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff1b9d9c640 1 --2- 172.21.15.107:0/2796941831 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6cda3d0 0x55c9a6ce3650 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff1b00bbd90 tx=0x7ff1b0005bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56fa7ee25357f4ca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.710+0000 7ff19d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:41:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.711+0000 7ff19d7fa640 1 -- 172.21.15.107:0/2796941831 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b0049070 con 0x55c9a6cda3d0 2024-05-09T08:41:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.711+0000 7ff19d7fa640 1 -- 172.21.15.107:0/2796941831 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7ff1b00ed200 con 0x55c9a6cda3d0 2024-05-09T08:41:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.711+0000 7ff1bedee740 1 -- 172.21.15.107:0/2796941831 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1b00bbe00 con 0x55c9a6cda3d0 2024-05-09T08:41:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.711+0000 7ff19d7fa640 1 -- 172.21.15.107:0/2796941831 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff1b00b9ea0 con 0x55c9a6cda3d0 2024-05-09T08:41:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.711+0000 7ff1bedee740 10 client.27047 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.711+0000 7ff1bedee740 1 -- 172.21.15.107:0/2796941831 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c9a6d60350 con 0x55c9a6cda3d0 2024-05-09T08:41:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.711+0000 7ff19d7fa640 1 -- 172.21.15.107:0/2796941831 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff1b00bbe00 con 0x55c9a6cda3d0 2024-05-09T08:41:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.711+0000 7ff19d7fa640 1 -- 172.21.15.107:0/2796941831 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff1b00d4050 con 0x55c9a6cda3d0 2024-05-09T08:41:34.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.711+0000 7ff1bedee740 10 client.27047 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.711+0000 7ff1bedee740 10 client.27047 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.711+0000 7ff1bedee740 20 client.27047 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.711+0000 7ff1bedee740 1 --2- 172.21.15.107:0/2796941831 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c9a6d60740 0x55c9a6d62b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.711+0000 7ff1bedee740 4 client.27047 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:41:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.711+0000 7ff1bedee740 1 -- 172.21.15.107:0/2796941831 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c9a6bc4820 con 0x55c9a6d60740 2024-05-09T08:41:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.711+0000 7ff1b959b640 1 --2- 172.21.15.107:0/2796941831 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c9a6d60740 0x55c9a6d62b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff1b959b640 1 --2- 172.21.15.107:0/2796941831 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c9a6d60740 0x55c9a6d62b60 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c690bab5f56476c7 server_cookie=4d8850f6aad6e0e1 in_seq=0 out_seq=0 2024-05-09T08:41:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff19d7fa640 10 client.27047 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff19d7fa640 1 -- 172.21.15.107:0/2796941831 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c9a6bc4820 con 0x55c9a6d60740 2024-05-09T08:41:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff19d7fa640 10 client.27047 handle_command_reply: tid=0 2024-05-09T08:41:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff1bedee740 1 client.27047 shutdown 2024-05-09T08:41:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff1bedee740 1 -- 172.21.15.107:0/2796941831 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6cda3d0 msgr2=0x55c9a6ce3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff1bedee740 1 --2- 172.21.15.107:0/2796941831 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6cda3d0 0x55c9a6ce3650 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff1b00bbd90 tx=0x7ff1b0005bc0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff1bedee740 1 -- 172.21.15.107:0/2796941831 shutdown_connections 2024-05-09T08:41:34.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff1bedee740 1 --2- 172.21.15.107:0/2796941831 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c9a6d60740 0x55c9a6d62b60 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff1bedee740 1 --2- 172.21.15.107:0/2796941831 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c9a6ce98f0 0x55c9a6ce40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff19d7fa640 0 client.27047 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff1bedee740 1 --2- 172.21.15.107:0/2796941831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c9a6cdacf0 0x55c9a6ce3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff1bedee740 1 --2- 172.21.15.107:0/2796941831 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9a6cda3d0 0x55c9a6ce3650 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff1bedee740 1 -- 172.21.15.107:0/2796941831 >> 172.21.15.107:0/2796941831 conn(0x55c9a6cd6810 msgr2=0x55c9a6cdbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff1bedee740 1 -- 172.21.15.107:0/2796941831 shutdown_connections 2024-05-09T08:41:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff1bedee740 1 -- 172.21.15.107:0/2796941831 wait complete. 2024-05-09T08:41:34.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.712+0000 7ff1bedee740 20 client.27047 trim_cache size 0 max 16384 2024-05-09T08:41:34.718 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:41:34.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.713+0000 7ff1bad9e640 1 -- 172.21.15.107:0/936504868 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff1a00a2d00 msgr2=0x7ff1a00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.713+0000 7ff1bad9e640 1 --2- 172.21.15.107:0/936504868 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff1a00a2d00 0x7ff1a00a51c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac005fd0 tx=0x7ff1ac014910 comp rx=0 tx=0).stop 2024-05-09T08:41:34.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.713+0000 7ff1bad9e640 1 -- 172.21.15.107:0/936504868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1b4102890 msgr2=0x7ff1b4197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:34.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.713+0000 7ff1bad9e640 1 --2- 172.21.15.107:0/936504868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1b4102890 0x7ff1b4197ec0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0005940 tx=0x7ff1b00042f0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.713+0000 7ff1bad9e640 1 -- 172.21.15.107:0/936504868 shutdown_connections 2024-05-09T08:41:34.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.713+0000 7ff1bad9e640 1 --2- 172.21.15.107:0/936504868 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff1a00a2d00 0x7ff1a00a51c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.713+0000 7ff1bad9e640 1 --2- 172.21.15.107:0/936504868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1b410bcb0 0x7ff1b419c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.713+0000 7ff1bad9e640 1 --2- 172.21.15.107:0/936504868 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1b41031b0 0x7ff1b4198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.713+0000 7ff1bad9e640 1 --2- 172.21.15.107:0/936504868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1b4102890 0x7ff1b4197ec0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:34.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.713+0000 7ff1bad9e640 1 -- 172.21.15.107:0/936504868 >> 172.21.15.107:0/936504868 conn(0x7ff1b40fc0b0 msgr2=0x7ff1b40fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:34.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.713+0000 7ff1bad9e640 1 -- 172.21.15.107:0/936504868 shutdown_connections 2024-05-09T08:41:34.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:34.713+0000 7ff1bad9e640 1 -- 172.21.15.107:0/936504868 wait complete. 2024-05-09T08:41:35.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:35 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3405354151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:41:35.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:35 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3405354151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:41:35.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:35 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3405354151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:41:35.726 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:41:35.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:41:35.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:41:35.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.849+0000 7f8abb95a640 1 -- 172.21.15.107:0/1371629748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ab4106710 msgr2=0x7f8ab4106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:35.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.849+0000 7f8abb95a640 1 --2- 172.21.15.107:0/1371629748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ab4106710 0x7f8ab4106af0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0009930 tx=0x7f8ab002a420 comp rx=0 tx=0).stop 2024-05-09T08:41:35.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.849+0000 7f8abb95a640 1 -- 172.21.15.107:0/1371629748 shutdown_connections 2024-05-09T08:41:35.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.849+0000 7f8abb95a640 1 --2- 172.21.15.107:0/1371629748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ab41010c0 0x7f8ab410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.849+0000 7f8abb95a640 1 --2- 172.21.15.107:0/1371629748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ab4100700 0x7f8ab4100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.849+0000 7f8abb95a640 1 --2- 172.21.15.107:0/1371629748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ab4106710 0x7f8ab4106af0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.849+0000 7f8abb95a640 1 -- 172.21.15.107:0/1371629748 >> 172.21.15.107:0/1371629748 conn(0x7f8ab4077830 msgr2=0x7f8ab4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:35.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.849+0000 7f8abb95a640 1 -- 172.21.15.107:0/1371629748 shutdown_connections 2024-05-09T08:41:35.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.849+0000 7f8abb95a640 1 -- 172.21.15.107:0/1371629748 wait complete. 2024-05-09T08:41:35.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.850+0000 7f8abb95a640 1 Processor -- start 2024-05-09T08:41:35.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.850+0000 7f8abb95a640 1 -- start start 2024-05-09T08:41:35.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.850+0000 7f8abb95a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ab4100700 0x7f8ab4198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.850+0000 7f8abb95a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ab41010c0 0x7f8ab4199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.850+0000 7f8abb95a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ab4106710 0x7f8ab41a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.850+0000 7f8abb95a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab41109d0 con 0x7f8ab4100700 2024-05-09T08:41:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.850+0000 7f8abb95a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab4110850 con 0x7f8ab41010c0 2024-05-09T08:41:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.850+0000 7f8abb95a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab4110b50 con 0x7f8ab4106710 2024-05-09T08:41:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.850+0000 7f8ab96f6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ab4100700 0x7f8ab4198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.850+0000 7f8ab96f6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ab4100700 0x7f8ab4198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58080/0 (socket says 172.21.15.107:58080) 2024-05-09T08:41:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.850+0000 7f8ab96f6640 1 -- 172.21.15.107:0/1818839291 learned_addr learned my addr 172.21.15.107:0/1818839291 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.850+0000 7f8ab9ef7640 1 --2- 172.21.15.107:0/1818839291 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ab4106710 0x7f8ab41a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.851+0000 7f8ab8ef5640 1 --2- 172.21.15.107:0/1818839291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ab41010c0 0x7f8ab4199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.851+0000 7f8ab96f6640 1 -- 172.21.15.107:0/1818839291 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ab4106710 msgr2=0x7f8ab41a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.851+0000 7f8ab96f6640 1 --2- 172.21.15.107:0/1818839291 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ab4106710 0x7f8ab41a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.851+0000 7f8ab96f6640 1 -- 172.21.15.107:0/1818839291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ab41010c0 msgr2=0x7f8ab4199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.851+0000 7f8ab96f6640 1 --2- 172.21.15.107:0/1818839291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ab41010c0 0x7f8ab4199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.851+0000 7f8ab96f6640 1 -- 172.21.15.107:0/1818839291 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ab41a43b0 con 0x7f8ab4100700 2024-05-09T08:41:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.851+0000 7f8ab8ef5640 1 --2- 172.21.15.107:0/1818839291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ab41010c0 0x7f8ab4199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.851+0000 7f8ab96f6640 1 --2- 172.21.15.107:0/1818839291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ab4100700 0x7f8ab4198df0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0005940 tx=0x7f8ab00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.851+0000 7f8aaa7fc640 1 -- 172.21.15.107:0/1818839291 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab003f030 con 0x7f8ab4100700 2024-05-09T08:41:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.851+0000 7f8abb95a640 1 -- 172.21.15.107:0/1818839291 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ab41a4580 con 0x7f8ab4100700 2024-05-09T08:41:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.851+0000 7f8aaa7fc640 1 -- 172.21.15.107:0/1818839291 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ab00318a0 con 0x7f8ab4100700 2024-05-09T08:41:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.851+0000 7f8aaa7fc640 1 -- 172.21.15.107:0/1818839291 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab0035070 con 0x7f8ab4100700 2024-05-09T08:41:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.851+0000 7f8abb95a640 1 -- 172.21.15.107:0/1818839291 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ab41a4a40 con 0x7f8ab4100700 2024-05-09T08:41:35.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.854+0000 7f8aaa7fc640 1 -- 172.21.15.107:0/1818839291 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8ab0042030 con 0x7f8ab4100700 2024-05-09T08:41:35.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.854+0000 7f8abe662740 1 Processor -- start 2024-05-09T08:41:35.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.854+0000 7f8abe662740 1 -- start start 2024-05-09T08:41:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.854+0000 7f8abe662740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 0x562bfa026800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.854+0000 7f8abe662740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa026dd0 0x562bfa027250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.854+0000 7f8abe662740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa031b50 0x562bfa033f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.854+0000 7f8abe662740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562bf9f87990 con 0x562bfa026dd0 2024-05-09T08:41:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.854+0000 7f8abe662740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562bf9f2f5b0 con 0x562bfa031b50 2024-05-09T08:41:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.854+0000 7f8abe662740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x562bf9f83950 con 0x562bfa0284c0 2024-05-09T08:41:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.855+0000 7f8ab96f6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa026dd0 0x562bfa027250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.855+0000 7f8ab96f6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa026dd0 0x562bfa027250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58082/0 (socket says 172.21.15.107:58082) 2024-05-09T08:41:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.855+0000 7f8ab96f6640 1 -- 172.21.15.107:0/2305990154 learned_addr learned my addr 172.21.15.107:0/2305990154 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.855+0000 7f8ab9ef7640 1 --2- 172.21.15.107:0/2305990154 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 0x562bfa026800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.855+0000 7f8aaa7fc640 1 --2- 172.21.15.107:0/1818839291 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8aa00a2d00 0x7f8aa00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.855+0000 7f8ab8ef5640 1 --2- 172.21.15.107:0/2305990154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa031b50 0x562bfa033f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.855+0000 7f8ab9ef7640 1 --2- 172.21.15.107:0/1818839291 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8aa00a2d00 0x7f8aa00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.855+0000 7f8ab96f6640 1 -- 172.21.15.107:0/2305990154 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 msgr2=0x562bfa026800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.855+0000 7f8ab96f6640 1 --2- 172.21.15.107:0/2305990154 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 0x562bfa026800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.855+0000 7f8ab96f6640 1 -- 172.21.15.107:0/2305990154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa031b50 msgr2=0x562bfa033f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.855+0000 7f8ab96f6640 1 --2- 172.21.15.107:0/2305990154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa031b50 0x562bfa033f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.855+0000 7f8aaa7fc640 1 -- 172.21.15.107:0/1818839291 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f8ab00efff0 con 0x7f8ab4100700 2024-05-09T08:41:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.855+0000 7f8ab96f6640 1 -- 172.21.15.107:0/2305990154 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bf9d637e0 con 0x562bfa026dd0 2024-05-09T08:41:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.855+0000 7f8ab8ef5640 1 --2- 172.21.15.107:0/2305990154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa031b50 0x562bfa033f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.855+0000 7f8ab96f6640 1 --2- 172.21.15.107:0/2305990154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa026dd0 0x562bfa027250 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00ed300 tx=0x7f8ab002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=157b6c5ade203afe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.855+0000 7f8ab9ef7640 1 --2- 172.21.15.107:0/1818839291 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8aa00a2d00 0x7f8aa00a51c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8aac009fd0 tx=0x7f8aac004ab0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.856+0000 7f8a9f7fe640 1 -- 172.21.15.107:0/2305990154 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab0049070 con 0x562bfa026dd0 2024-05-09T08:41:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.856+0000 7f8a9f7fe640 1 -- 172.21.15.107:0/2305990154 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ab0003e20 con 0x562bfa026dd0 2024-05-09T08:41:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.856+0000 7f8a9f7fe640 1 -- 172.21.15.107:0/2305990154 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab00bc860 con 0x562bfa026dd0 2024-05-09T08:41:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.856+0000 7f8abe662740 1 -- 172.21.15.107:0/2305990154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa026dd0 msgr2=0x562bfa027250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.856+0000 7f8abe662740 1 --2- 172.21.15.107:0/2305990154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa026dd0 0x562bfa027250 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00ed300 tx=0x7f8ab002a420 comp rx=0 tx=0).stop 2024-05-09T08:41:35.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.856+0000 7f8abe662740 1 -- 172.21.15.107:0/2305990154 shutdown_connections 2024-05-09T08:41:35.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.856+0000 7f8abe662740 1 --2- 172.21.15.107:0/2305990154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa031b50 0x562bfa033f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.856+0000 7f8abe662740 1 --2- 172.21.15.107:0/2305990154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa026dd0 0x562bfa027250 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.856+0000 7f8abe662740 1 --2- 172.21.15.107:0/2305990154 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 0x562bfa026800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.856+0000 7f8abe662740 1 -- 172.21.15.107:0/2305990154 >> 172.21.15.107:0/2305990154 conn(0x562bf9feb0a0 msgr2=0x562bfa023330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:35.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.856+0000 7f8abe662740 1 -- 172.21.15.107:0/2305990154 shutdown_connections 2024-05-09T08:41:35.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.856+0000 7f8abe662740 1 -- 172.21.15.107:0/2305990154 wait complete. 2024-05-09T08:41:35.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.857+0000 7f8abe662740 1 -- start start 2024-05-09T08:41:35.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.857+0000 7f8abe662740 1 Processor -- start 2024-05-09T08:41:35.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.857+0000 7f8abe662740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa026dd0 0x562bfa02ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.857+0000 7f8abe662740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 0x562bfa02efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.857+0000 7f8abe662740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa031b50 0x562bfa02f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.857+0000 7f8abe662740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562bf9f87990 con 0x562bfa031b50 2024-05-09T08:41:35.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.857+0000 7f8abe662740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562bf9f42b40 con 0x562bfa026dd0 2024-05-09T08:41:35.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.857+0000 7f8abe662740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x562bfa0347a0 con 0x562bfa0284c0 2024-05-09T08:41:35.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.857+0000 7f8abe662740 10 client.0 _ll_register_callbacks cb 0x562bf9f370f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:35.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.857+0000 7f8ab9ef7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa031b50 0x562bfa02f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.857+0000 7f8ab9ef7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa031b50 0x562bfa02f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58098/0 (socket says 172.21.15.107:58098) 2024-05-09T08:41:35.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.858+0000 7f8ab9ef7640 1 -- 172.21.15.107:0/3727368191 learned_addr learned my addr 172.21.15.107:0/3727368191 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:35.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.858+0000 7f8ab96f6640 1 --2- 172.21.15.107:0/3727368191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa026dd0 0x562bfa02ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.858+0000 7f8ab8ef5640 1 --2- 172.21.15.107:0/3727368191 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 0x562bfa02efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.858+0000 7f8ab9ef7640 1 -- 172.21.15.107:0/3727368191 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 msgr2=0x562bfa02efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:35.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.858+0000 7f8ab9ef7640 1 --2- 172.21.15.107:0/3727368191 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 0x562bfa02efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.858+0000 7f8ab9ef7640 1 -- 172.21.15.107:0/3727368191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa026dd0 msgr2=0x562bfa02ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.858+0000 7f8ab9ef7640 1 --2- 172.21.15.107:0/3727368191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa026dd0 0x562bfa02ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.858+0000 7f8ab9ef7640 1 -- 172.21.15.107:0/3727368191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bf9dc3f70 con 0x562bfa031b50 2024-05-09T08:41:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.858+0000 7f8ab9ef7640 1 --2- 172.21.15.107:0/3727368191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa031b50 0x562bfa02f4f0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00fab0 tx=0x7f8aac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b879e1e131d47a1a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.858+0000 7f8a9cff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.858+0000 7f8a9cff9640 1 -- 172.21.15.107:0/3727368191 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac025070 con 0x562bfa031b50 2024-05-09T08:41:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.858+0000 7f8abe662740 1 -- 172.21.15.107:0/3727368191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562bf9f950c0 con 0x562bfa031b50 2024-05-09T08:41:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.858+0000 7f8a9cff9640 1 -- 172.21.15.107:0/3727368191 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f8aac02a460 con 0x562bfa031b50 2024-05-09T08:41:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.858+0000 7f8a9cff9640 1 -- 172.21.15.107:0/3727368191 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8aac033050 con 0x562bfa031b50 2024-05-09T08:41:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.858+0000 7f8abe662740 10 client.18090 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.858+0000 7f8abe662740 1 -- 172.21.15.107:0/3727368191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562bf9c5b170 con 0x562bfa031b50 2024-05-09T08:41:35.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.859+0000 7f8a9cff9640 1 -- 172.21.15.107:0/3727368191 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x562bf9f950c0 con 0x562bfa031b50 2024-05-09T08:41:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.859+0000 7f8a9cff9640 1 -- 172.21.15.107:0/3727368191 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8aac0203d0 con 0x562bfa031b50 2024-05-09T08:41:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.859+0000 7f8abe662740 10 client.18090 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.859+0000 7f8abe662740 10 client.18090 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.859+0000 7f8abe662740 20 client.18090 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.859+0000 7f8abe662740 1 --2- 172.21.15.107:0/3727368191 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562bfa041980 0x562bfa043d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.859+0000 7f8abe662740 4 client.18090 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.859+0000 7f8abe662740 1 -- 172.21.15.107:0/3727368191 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562bf9f8f410 con 0x562bfa041980 2024-05-09T08:41:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.859+0000 7f8ab96f6640 1 --2- 172.21.15.107:0/3727368191 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562bfa041980 0x562bfa043d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.860+0000 7f8a9cff9640 10 client.18090 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.860+0000 7f8ab96f6640 1 --2- 172.21.15.107:0/3727368191 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562bfa041980 0x562bfa043d70 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0b3978eda7a6cb8 server_cookie=9b2c96f0ac235f61 in_seq=0 out_seq=0 2024-05-09T08:41:35.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.860+0000 7f8a9cff9640 1 -- 172.21.15.107:0/3727368191 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562bf9f8f410 con 0x562bfa041980 2024-05-09T08:41:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.860+0000 7f8a9cff9640 10 client.18090 handle_command_reply: tid=0 2024-05-09T08:41:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.860+0000 7f8abe662740 1 client.18090 shutdown 2024-05-09T08:41:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.860+0000 7f8abe662740 1 -- 172.21.15.107:0/3727368191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa031b50 msgr2=0x562bfa02f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.860+0000 7f8abe662740 1 --2- 172.21.15.107:0/3727368191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa031b50 0x562bfa02f4f0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00fab0 tx=0x7f8aac004a40 comp rx=0 tx=0).stop 2024-05-09T08:41:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.860+0000 7f8abe662740 1 -- 172.21.15.107:0/3727368191 shutdown_connections 2024-05-09T08:41:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.860+0000 7f8abe662740 1 --2- 172.21.15.107:0/3727368191 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562bfa041980 0x562bfa043d70 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.860+0000 7f8abe662740 1 --2- 172.21.15.107:0/3727368191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa031b50 0x562bfa02f4f0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.860+0000 7f8abe662740 1 --2- 172.21.15.107:0/3727368191 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 0x562bfa02efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.860+0000 7f8a9cff9640 0 client.18090 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.860+0000 7f8abe662740 1 --2- 172.21.15.107:0/3727368191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa026dd0 0x562bfa02ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.860+0000 7f8abe662740 1 -- 172.21.15.107:0/3727368191 >> 172.21.15.107:0/3727368191 conn(0x562bf9feb0a0 msgr2=0x562bfa01e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:35.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.861+0000 7f8abe662740 1 -- 172.21.15.107:0/3727368191 shutdown_connections 2024-05-09T08:41:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.861+0000 7f8abe662740 1 -- 172.21.15.107:0/3727368191 wait complete. 2024-05-09T08:41:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.861+0000 7f8abe662740 20 client.18090 trim_cache size 0 max 16384 2024-05-09T08:41:35.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8abe662740 1 Processor -- start 2024-05-09T08:41:35.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8abe662740 1 -- start start 2024-05-09T08:41:35.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8abe662740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa026dd0 0x562bfa03d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8abe662740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 0x562bfa02dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8abe662740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa031b50 0x562bfa0309b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8abe662740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab41af380 con 0x562bfa026dd0 2024-05-09T08:41:35.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8abe662740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab41acc50 con 0x562bfa031b50 2024-05-09T08:41:35.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8abe662740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab41aaac0 con 0x562bfa0284c0 2024-05-09T08:41:35.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8ab96f6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa026dd0 0x562bfa03d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8ab96f6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa026dd0 0x562bfa03d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58106/0 (socket says 172.21.15.107:58106) 2024-05-09T08:41:35.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8ab96f6640 1 -- 172.21.15.107:0/3459243747 learned_addr learned my addr 172.21.15.107:0/3459243747 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:35.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8ab8ef5640 1 --2- 172.21.15.107:0/3459243747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 0x562bfa02dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8ab9ef7640 1 --2- 172.21.15.107:0/3459243747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa031b50 0x562bfa0309b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8ab96f6640 1 -- 172.21.15.107:0/3459243747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 msgr2=0x562bfa02dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:35.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8ab96f6640 1 --2- 172.21.15.107:0/3459243747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 0x562bfa02dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8ab96f6640 1 -- 172.21.15.107:0/3459243747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa031b50 msgr2=0x562bfa0309b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:35.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8ab96f6640 1 --2- 172.21.15.107:0/3459243747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa031b50 0x562bfa0309b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8ab96f6640 1 -- 172.21.15.107:0/3459243747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bfa026a40 con 0x562bfa026dd0 2024-05-09T08:41:35.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.883+0000 7f8ab9ef7640 1 --2- 172.21.15.107:0/3459243747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa031b50 0x562bfa0309b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:35.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.884+0000 7f8ab96f6640 1 --2- 172.21.15.107:0/3459243747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa026dd0 0x562bfa03d520 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00ba410 tx=0x7f8ab003b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca9418525770fcd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:35.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.884+0000 7f8a9effd640 1 -- 172.21.15.107:0/3459243747 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab0049040 con 0x562bfa026dd0 2024-05-09T08:41:35.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.884+0000 7f8a9effd640 1 -- 172.21.15.107:0/3459243747 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ab00bc860 con 0x562bfa026dd0 2024-05-09T08:41:35.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.884+0000 7f8a9effd640 1 -- 172.21.15.107:0/3459243747 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab00c4dd0 con 0x562bfa026dd0 2024-05-09T08:41:35.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.884+0000 7f8abe662740 1 -- 172.21.15.107:0/3459243747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa026dd0 msgr2=0x562bfa03d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:35.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.884+0000 7f8abe662740 1 --2- 172.21.15.107:0/3459243747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa026dd0 0x562bfa03d520 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00ba410 tx=0x7f8ab003b460 comp rx=0 tx=0).stop 2024-05-09T08:41:35.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.885+0000 7f8abe662740 1 -- 172.21.15.107:0/3459243747 shutdown_connections 2024-05-09T08:41:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.885+0000 7f8abe662740 1 --2- 172.21.15.107:0/3459243747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa031b50 0x562bfa0309b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.885+0000 7f8abe662740 1 --2- 172.21.15.107:0/3459243747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 0x562bfa02dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.885+0000 7f8abe662740 1 --2- 172.21.15.107:0/3459243747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa026dd0 0x562bfa03d520 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.885+0000 7f8abe662740 1 -- 172.21.15.107:0/3459243747 >> 172.21.15.107:0/3459243747 conn(0x562bf9feb0a0 msgr2=0x562bfa0302a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.885+0000 7f8abe662740 1 -- 172.21.15.107:0/3459243747 shutdown_connections 2024-05-09T08:41:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.885+0000 7f8abe662740 1 -- 172.21.15.107:0/3459243747 wait complete. 2024-05-09T08:41:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.885+0000 7f8abe662740 1 -- start start 2024-05-09T08:41:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.885+0000 7f8abe662740 1 Processor -- start 2024-05-09T08:41:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.885+0000 7f8abe662740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa026dd0 0x562bfa02b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.885+0000 7f8abe662740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 0x562bfa02be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.885+0000 7f8abe662740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa031b50 0x562bfa02c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.885+0000 7f8abe662740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab41ab110 con 0x562bfa031b50 2024-05-09T08:41:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.885+0000 7f8abe662740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab41ac580 con 0x562bfa026dd0 2024-05-09T08:41:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.885+0000 7f8abe662740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab41ab7e0 con 0x562bfa0284c0 2024-05-09T08:41:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8abe662740 10 client.0 _ll_register_callbacks cb 0x7f8aac009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8ab9ef7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa031b50 0x562bfa02c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8ab8ef5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 0x562bfa02be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8ab9ef7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa031b50 0x562bfa02c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58122/0 (socket says 172.21.15.107:58122) 2024-05-09T08:41:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8ab9ef7640 1 -- 172.21.15.107:0/3680468618 learned_addr learned my addr 172.21.15.107:0/3680468618 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8ab96f6640 1 --2- 172.21.15.107:0/3680468618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa026dd0 0x562bfa02b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8ab9ef7640 1 -- 172.21.15.107:0/3680468618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 msgr2=0x562bfa02be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8ab9ef7640 1 --2- 172.21.15.107:0/3680468618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 0x562bfa02be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8ab9ef7640 1 -- 172.21.15.107:0/3680468618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa026dd0 msgr2=0x562bfa02b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8ab9ef7640 1 --2- 172.21.15.107:0/3680468618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa026dd0 0x562bfa02b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8ab9ef7640 1 -- 172.21.15.107:0/3680468618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bfa0a70f0 con 0x562bfa031b50 2024-05-09T08:41:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8ab9ef7640 1 --2- 172.21.15.107:0/3680468618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa031b50 0x562bfa02c390 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00f800 tx=0x7f8aac0133d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75b74b74d2c6cd0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8a9cff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8a9cff9640 1 -- 172.21.15.107:0/3680468618 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac03b070 con 0x562bfa031b50 2024-05-09T08:41:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8a9cff9640 1 -- 172.21.15.107:0/3680468618 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f8aac020a10 con 0x562bfa031b50 2024-05-09T08:41:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8abe662740 1 -- 172.21.15.107:0/3680468618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ab00bc860 con 0x562bfa031b50 2024-05-09T08:41:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8a9cff9640 1 -- 172.21.15.107:0/3680468618 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8aac044050 con 0x562bfa031b50 2024-05-09T08:41:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8abe662740 10 client.18096 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.886+0000 7f8abe662740 1 -- 172.21.15.107:0/3680468618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562bfa0a83e0 con 0x562bfa031b50 2024-05-09T08:41:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.887+0000 7f8a9cff9640 1 -- 172.21.15.107:0/3680468618 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ab00bc860 con 0x562bfa031b50 2024-05-09T08:41:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.887+0000 7f8a9cff9640 1 -- 172.21.15.107:0/3680468618 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8aac03e020 con 0x562bfa031b50 2024-05-09T08:41:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.887+0000 7f8abe662740 10 client.18096 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.887+0000 7f8abe662740 10 client.18096 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.887+0000 7f8abe662740 20 client.18096 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.887+0000 7f8abe662740 1 --2- 172.21.15.107:0/3680468618 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562bfa041980 0x562bfa0aabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.887+0000 7f8abe662740 4 client.18096 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:41:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.887+0000 7f8abe662740 1 -- 172.21.15.107:0/3680468618 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562bf9f0cda0 con 0x562bfa041980 2024-05-09T08:41:35.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.887+0000 7f8ab96f6640 1 --2- 172.21.15.107:0/3680468618 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562bfa041980 0x562bfa0aabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:35.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.887+0000 7f8a9cff9640 10 client.18096 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:35.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.887+0000 7f8ab96f6640 1 --2- 172.21.15.107:0/3680468618 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562bfa041980 0x562bfa0aabf0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db59196dffc81548 server_cookie=9e6e5143bf7bb7b6 in_seq=0 out_seq=0 2024-05-09T08:41:35.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.888+0000 7f8a9cff9640 1 -- 172.21.15.107:0/3680468618 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562bf9f0cda0 con 0x562bfa041980 2024-05-09T08:41:35.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.888+0000 7f8a9cff9640 10 client.18096 handle_command_reply: tid=0 2024-05-09T08:41:35.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.888+0000 7f8abe662740 1 client.18096 shutdown 2024-05-09T08:41:35.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.889+0000 7f8abe662740 1 -- 172.21.15.107:0/3680468618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa031b50 msgr2=0x562bfa02c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:35.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.889+0000 7f8abe662740 1 --2- 172.21.15.107:0/3680468618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa031b50 0x562bfa02c390 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00f800 tx=0x7f8aac0133d0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.889+0000 7f8abe662740 1 -- 172.21.15.107:0/3680468618 shutdown_connections 2024-05-09T08:41:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.889+0000 7f8abe662740 1 --2- 172.21.15.107:0/3680468618 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562bfa041980 0x562bfa0aabf0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.889+0000 7f8abe662740 1 --2- 172.21.15.107:0/3680468618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562bfa031b50 0x562bfa02c390 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.889+0000 7f8a9cff9640 0 client.18096 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.889+0000 7f8abe662740 1 --2- 172.21.15.107:0/3680468618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562bfa0284c0 0x562bfa02be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.889+0000 7f8abe662740 1 --2- 172.21.15.107:0/3680468618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562bfa026dd0 0x562bfa02b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.889+0000 7f8abe662740 1 -- 172.21.15.107:0/3680468618 >> 172.21.15.107:0/3680468618 conn(0x562bf9feb0a0 msgr2=0x562bfa030b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.889+0000 7f8abe662740 1 -- 172.21.15.107:0/3680468618 shutdown_connections 2024-05-09T08:41:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.889+0000 7f8abe662740 1 -- 172.21.15.107:0/3680468618 wait complete. 2024-05-09T08:41:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.889+0000 7f8abe662740 20 client.18096 trim_cache size 0 max 16384 2024-05-09T08:41:35.895 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:41:35.895 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:41:35.895 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "8de51e0f-d92c-4e96-8a29-70553747beb1", 2024-05-09T08:41:35.895 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:41:35.895 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:41:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.890+0000 7f8abb95a640 1 -- 172.21.15.107:0/1818839291 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8aa00a2d00 msgr2=0x7f8aa00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.890+0000 7f8abb95a640 1 --2- 172.21.15.107:0/1818839291 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8aa00a2d00 0x7f8aa00a51c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8aac009fd0 tx=0x7f8aac004ab0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.890+0000 7f8abb95a640 1 -- 172.21.15.107:0/1818839291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ab4100700 msgr2=0x7f8ab4198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:35.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.890+0000 7f8abb95a640 1 --2- 172.21.15.107:0/1818839291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ab4100700 0x7f8ab4198df0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0005940 tx=0x7f8ab00042f0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.890+0000 7f8abb95a640 1 -- 172.21.15.107:0/1818839291 shutdown_connections 2024-05-09T08:41:35.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.890+0000 7f8abb95a640 1 --2- 172.21.15.107:0/1818839291 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8aa00a2d00 0x7f8aa00a51c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.890+0000 7f8abb95a640 1 --2- 172.21.15.107:0/1818839291 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ab4106710 0x7f8ab41a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.890+0000 7f8abb95a640 1 --2- 172.21.15.107:0/1818839291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ab41010c0 0x7f8ab4199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.890+0000 7f8abb95a640 1 --2- 172.21.15.107:0/1818839291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ab4100700 0x7f8ab4198df0 secure :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0005940 tx=0x7f8ab00042f0 comp rx=0 tx=0).stop 2024-05-09T08:41:35.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.890+0000 7f8abb95a640 1 -- 172.21.15.107:0/1818839291 >> 172.21.15.107:0/1818839291 conn(0x7f8ab4077830 msgr2=0x7f8ab4103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:35.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.890+0000 7f8abb95a640 1 -- 172.21.15.107:0/1818839291 shutdown_connections 2024-05-09T08:41:35.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:35.890+0000 7f8abb95a640 1 -- 172.21.15.107:0/1818839291 wait complete. 2024-05-09T08:41:35.902 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:41:35.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:41:36.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.024+0000 7f07ead7e640 1 -- 172.21.15.107:0/4168152589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07e4106bd0 msgr2=0x7f07e410d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:36.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.024+0000 7f07ead7e640 1 --2- 172.21.15.107:0/4168152589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07e4106bd0 0x7f07e410d600 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f07dc009ca0 tx=0x7f07dc02a8c0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.024+0000 7f07ead7e640 1 -- 172.21.15.107:0/4168152589 shutdown_connections 2024-05-09T08:41:36.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.024+0000 7f07ead7e640 1 --2- 172.21.15.107:0/4168152589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07e4106bd0 0x7f07e410d600 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.024+0000 7f07ead7e640 1 --2- 172.21.15.107:0/4168152589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e41020c0 0x7f07e41064f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.024+0000 7f07ead7e640 1 --2- 172.21.15.107:0/4168152589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07e4101710 0x7f07e4101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.024+0000 7f07ead7e640 1 -- 172.21.15.107:0/4168152589 >> 172.21.15.107:0/4168152589 conn(0x7f07e4075a30 msgr2=0x7f07e4075e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:36.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.024+0000 7f07ead7e640 1 -- 172.21.15.107:0/4168152589 shutdown_connections 2024-05-09T08:41:36.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.024+0000 7f07ead7e640 1 -- 172.21.15.107:0/4168152589 wait complete. 2024-05-09T08:41:36.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.024+0000 7f07ead7e640 1 Processor -- start 2024-05-09T08:41:36.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.024+0000 7f07ead7e640 1 -- start start 2024-05-09T08:41:36.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07ead7e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07e4101710 0x7f07e4196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07ead7e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e4106bd0 0x7f07e4197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07ead7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07e419b850 0x7f07e419bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07ead7e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f07e4110a40 con 0x7f07e419b850 2024-05-09T08:41:36.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07ead7e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f07e41108c0 con 0x7f07e4106bd0 2024-05-09T08:41:36.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07ead7e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f07e4110bc0 con 0x7f07e4101710 2024-05-09T08:41:36.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07e931b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07e419b850 0x7f07e419bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07e931b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07e419b850 0x7f07e419bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58140/0 (socket says 172.21.15.107:58140) 2024-05-09T08:41:36.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07e931b640 1 -- 172.21.15.107:0/1889188087 learned_addr learned my addr 172.21.15.107:0/1889188087 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:36.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07e3fff640 1 --2- 172.21.15.107:0/1889188087 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e4106bd0 0x7f07e4197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07e8b1a640 1 --2- 172.21.15.107:0/1889188087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07e4101710 0x7f07e4196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07e931b640 1 -- 172.21.15.107:0/1889188087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07e4101710 msgr2=0x7f07e4196fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:36.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07e931b640 1 --2- 172.21.15.107:0/1889188087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07e4101710 0x7f07e4196fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07e931b640 1 -- 172.21.15.107:0/1889188087 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e4106bd0 msgr2=0x7f07e4197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:36.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07e931b640 1 --2- 172.21.15.107:0/1889188087 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e4106bd0 0x7f07e4197520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07e931b640 1 -- 172.21.15.107:0/1889188087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e419c340 con 0x7f07e419b850 2024-05-09T08:41:36.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07e3fff640 1 --2- 172.21.15.107:0/1889188087 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e4106bd0 0x7f07e4197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:36.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.025+0000 7f07e931b640 1 --2- 172.21.15.107:0/1889188087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07e419b850 0x7f07e419bd00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f07dc031040 tx=0x7f07dc003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:36.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.026+0000 7f07e1ffb640 1 -- 172.21.15.107:0/1889188087 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc003ce0 con 0x7f07e419b850 2024-05-09T08:41:36.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.026+0000 7f07ead7e640 1 -- 172.21.15.107:0/1889188087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07e419c5d0 con 0x7f07e419b850 2024-05-09T08:41:36.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.026+0000 7f07e1ffb640 1 -- 172.21.15.107:0/1889188087 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07dc0044f0 con 0x7f07e419b850 2024-05-09T08:41:36.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.026+0000 7f07e1ffb640 1 -- 172.21.15.107:0/1889188087 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc030bf0 con 0x7f07e419b850 2024-05-09T08:41:36.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.026+0000 7f07ead7e640 1 -- 172.21.15.107:0/1889188087 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07e41006b0 con 0x7f07e419b850 2024-05-09T08:41:36.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.029+0000 7f07edb69740 1 Processor -- start 2024-05-09T08:41:36.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.029+0000 7f07e1ffb640 1 -- 172.21.15.107:0/1889188087 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f07dc0046a0 con 0x7f07e419b850 2024-05-09T08:41:36.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.029+0000 7f07edb69740 1 -- start start 2024-05-09T08:41:36.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.029+0000 7f07edb69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8ed280 0x561e9f8ed660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.029+0000 7f07edb69740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8edba0 0x561e9f8f7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.029+0000 7f07edb69740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8f8270 0x561e9f8fa660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.029+0000 7f07edb69740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561e9f852d00 con 0x561e9f8edba0 2024-05-09T08:41:36.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.029+0000 7f07edb69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561e9f7fa4c0 con 0x561e9f8ed280 2024-05-09T08:41:36.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.029+0000 7f07edb69740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x561e9f84ecc0 con 0x561e9f8f8270 2024-05-09T08:41:36.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.029+0000 7f07e8b1a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8edba0 0x561e9f8f7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.029+0000 7f07e8b1a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8edba0 0x561e9f8f7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58154/0 (socket says 172.21.15.107:58154) 2024-05-09T08:41:36.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.029+0000 7f07e8b1a640 1 -- 172.21.15.107:0/3682619575 learned_addr learned my addr 172.21.15.107:0/3682619575 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:36.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.030+0000 7f07e3fff640 1 --2- 172.21.15.107:0/3682619575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8f8270 0x561e9f8fa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.030+0000 7f07e3fff640 1 --2- 172.21.15.107:0/3682619575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8ed280 0x561e9f8ed660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.030+0000 7f07e8b1a640 1 -- 172.21.15.107:0/3682619575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8f8270 msgr2=0x561e9f8fa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:36.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.030+0000 7f07e8b1a640 1 --2- 172.21.15.107:0/3682619575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8f8270 0x561e9f8fa660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.030+0000 7f07e8b1a640 1 -- 172.21.15.107:0/3682619575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8ed280 msgr2=0x561e9f8ed660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:36.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.030+0000 7f07e8b1a640 1 --2- 172.21.15.107:0/3682619575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8ed280 0x561e9f8ed660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.030+0000 7f07e8b1a640 1 -- 172.21.15.107:0/3682619575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e9f6307d0 con 0x561e9f8edba0 2024-05-09T08:41:36.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.030+0000 7f07e3fff640 1 --2- 172.21.15.107:0/3682619575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8ed280 0x561e9f8ed660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:36.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.030+0000 7f07e1ffb640 1 --2- 172.21.15.107:0/1889188087 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07d40a2d00 0x7f07d40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.030+0000 7f07e8b1a640 1 --2- 172.21.15.107:0/3682619575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8edba0 0x561e9f8f7d30 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f07d8009a70 tx=0x7f07d8009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b64934022a2260 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:36.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.030+0000 7f07e3fff640 1 --2- 172.21.15.107:0/1889188087 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07d40a2d00 0x7f07d40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.030+0000 7f07e1ffb640 1 -- 172.21.15.107:0/1889188087 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f07dc0f0b00 con 0x7f07e419b850 2024-05-09T08:41:36.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.030+0000 7f07ceffd640 1 -- 172.21.15.107:0/3682619575 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d8017070 con 0x561e9f8edba0 2024-05-09T08:41:36.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.030+0000 7f07ceffd640 1 -- 172.21.15.107:0/3682619575 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07d8013470 con 0x561e9f8edba0 2024-05-09T08:41:36.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.030+0000 7f07ceffd640 1 -- 172.21.15.107:0/3682619575 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d8012630 con 0x561e9f8edba0 2024-05-09T08:41:36.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.031+0000 7f07edb69740 1 -- 172.21.15.107:0/3682619575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8edba0 msgr2=0x561e9f8f7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:36.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.031+0000 7f07e3fff640 1 --2- 172.21.15.107:0/1889188087 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07d40a2d00 0x7f07d40a51c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f07d0009fd0 tx=0x7f07d0009500 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:36.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.031+0000 7f07edb69740 1 --2- 172.21.15.107:0/3682619575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8edba0 0x561e9f8f7d30 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f07d8009a70 tx=0x7f07d8009f30 comp rx=0 tx=0).stop 2024-05-09T08:41:36.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.031+0000 7f07edb69740 1 -- 172.21.15.107:0/3682619575 shutdown_connections 2024-05-09T08:41:36.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.031+0000 7f07edb69740 1 --2- 172.21.15.107:0/3682619575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8f8270 0x561e9f8fa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.031+0000 7f07edb69740 1 --2- 172.21.15.107:0/3682619575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8edba0 0x561e9f8f7d30 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.031+0000 7f07edb69740 1 --2- 172.21.15.107:0/3682619575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8ed280 0x561e9f8ed660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.031+0000 7f07edb69740 1 -- 172.21.15.107:0/3682619575 >> 172.21.15.107:0/3682619575 conn(0x561e9f8e95a0 msgr2=0x561e9f8e9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:36.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.031+0000 7f07edb69740 1 -- 172.21.15.107:0/3682619575 shutdown_connections 2024-05-09T08:41:36.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.031+0000 7f07edb69740 1 -- 172.21.15.107:0/3682619575 wait complete. 2024-05-09T08:41:36.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.031+0000 7f07edb69740 1 -- start start 2024-05-09T08:41:36.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07edb69740 1 Processor -- start 2024-05-09T08:41:36.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07edb69740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8ed280 0x561e9f8f5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07edb69740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8edba0 0x561e9f8f57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07edb69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8f8270 0x561e9f8f5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07edb69740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561e9f8ff470 con 0x561e9f8ed280 2024-05-09T08:41:36.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07edb69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561e9f80db90 con 0x561e9f8f8270 2024-05-09T08:41:36.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07edb69740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x561e9f8ff5f0 con 0x561e9f8edba0 2024-05-09T08:41:36.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07e8b1a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8ed280 0x561e9f8f5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07edb69740 10 client.0 _ll_register_callbacks cb 0x561e9f849da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:36.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07e8b1a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8ed280 0x561e9f8f5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58156/0 (socket says 172.21.15.107:58156) 2024-05-09T08:41:36.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07e8b1a640 1 -- 172.21.15.107:0/1003282983 learned_addr learned my addr 172.21.15.107:0/1003282983 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:36.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07e3fff640 1 --2- 172.21.15.107:0/1003282983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8edba0 0x561e9f8f57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07e931b640 1 --2- 172.21.15.107:0/1003282983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8f8270 0x561e9f8f5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07e8b1a640 1 -- 172.21.15.107:0/1003282983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8edba0 msgr2=0x561e9f8f57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:36.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07e8b1a640 1 --2- 172.21.15.107:0/1003282983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8edba0 0x561e9f8f57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07e8b1a640 1 -- 172.21.15.107:0/1003282983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8f8270 msgr2=0x561e9f8f5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:36.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07e8b1a640 1 --2- 172.21.15.107:0/1003282983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8f8270 0x561e9f8f5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07e8b1a640 1 -- 172.21.15.107:0/1003282983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e9f5663f0 con 0x561e9f8ed280 2024-05-09T08:41:36.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.032+0000 7f07e931b640 1 --2- 172.21.15.107:0/1003282983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8f8270 0x561e9f8f5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:36.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.033+0000 7f07c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:36.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.033+0000 7f07e8b1a640 1 --2- 172.21.15.107:0/1003282983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8ed280 0x561e9f8f5270 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f07d8013e00 tx=0x7f07d800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd45071d889f941 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:36.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.033+0000 7f07c3fff640 1 -- 172.21.15.107:0/1003282983 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d8017050 con 0x561e9f8ed280 2024-05-09T08:41:36.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.033+0000 7f07c3fff640 1 -- 172.21.15.107:0/1003282983 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f07d801c450 con 0x561e9f8ed280 2024-05-09T08:41:36.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.033+0000 7f07edb69740 1 -- 172.21.15.107:0/1003282983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07d8013470 con 0x561e9f8ed280 2024-05-09T08:41:36.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.033+0000 7f07c3fff640 1 -- 172.21.15.107:0/1003282983 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f07d802f050 con 0x561e9f8ed280 2024-05-09T08:41:36.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.033+0000 7f07edb69740 10 client.18108 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:36.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.033+0000 7f07edb69740 1 -- 172.21.15.107:0/1003282983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561e9f5260f0 con 0x561e9f8ed280 2024-05-09T08:41:36.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.033+0000 7f07c3fff640 1 -- 172.21.15.107:0/1003282983 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07d8013470 con 0x561e9f8ed280 2024-05-09T08:41:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.033+0000 7f07c3fff640 1 -- 172.21.15.107:0/1003282983 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f07d80267f0 con 0x561e9f8ed280 2024-05-09T08:41:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.033+0000 7f07edb69740 10 client.18108 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.033+0000 7f07edb69740 10 client.18108 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.033+0000 7f07edb69740 20 client.18108 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.033+0000 7f07edb69740 1 --2- 172.21.15.107:0/1003282983 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561e9f90c930 0x561e9f90ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.033+0000 7f07edb69740 4 client.18108 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.033+0000 7f07edb69740 1 -- 172.21.15.107:0/1003282983 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e9f860430 con 0x561e9f90c930 2024-05-09T08:41:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.034+0000 7f07e3fff640 1 --2- 172.21.15.107:0/1003282983 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561e9f90c930 0x561e9f90ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.034+0000 7f07c3fff640 10 client.18108 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.034+0000 7f07e3fff640 1 --2- 172.21.15.107:0/1003282983 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561e9f90c930 0x561e9f90ed20 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3e868cc4ed26ae1 server_cookie=8bf184368a7a7658 in_seq=0 out_seq=0 2024-05-09T08:41:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.035+0000 7f07c3fff640 1 -- 172.21.15.107:0/1003282983 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561e9f860430 con 0x561e9f90c930 2024-05-09T08:41:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.035+0000 7f07c3fff640 10 client.18108 handle_command_reply: tid=0 2024-05-09T08:41:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.035+0000 7f07edb69740 1 client.18108 shutdown 2024-05-09T08:41:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.035+0000 7f07edb69740 1 -- 172.21.15.107:0/1003282983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8ed280 msgr2=0x561e9f8f5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.035+0000 7f07edb69740 1 --2- 172.21.15.107:0/1003282983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8ed280 0x561e9f8f5270 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f07d8013e00 tx=0x7f07d800ee60 comp rx=0 tx=0).stop 2024-05-09T08:41:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.035+0000 7f07edb69740 1 -- 172.21.15.107:0/1003282983 shutdown_connections 2024-05-09T08:41:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.035+0000 7f07edb69740 1 --2- 172.21.15.107:0/1003282983 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561e9f90c930 0x561e9f90ed20 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.035+0000 7f07edb69740 1 --2- 172.21.15.107:0/1003282983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8f8270 0x561e9f8f5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.035+0000 7f07c3fff640 0 client.18108 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.035+0000 7f07edb69740 1 --2- 172.21.15.107:0/1003282983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8edba0 0x561e9f8f57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.035+0000 7f07edb69740 1 --2- 172.21.15.107:0/1003282983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8ed280 0x561e9f8f5270 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.035+0000 7f07edb69740 1 -- 172.21.15.107:0/1003282983 >> 172.21.15.107:0/1003282983 conn(0x561e9f8e95a0 msgr2=0x561e9f8e87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.035+0000 7f07edb69740 1 -- 172.21.15.107:0/1003282983 shutdown_connections 2024-05-09T08:41:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.035+0000 7f07edb69740 1 -- 172.21.15.107:0/1003282983 wait complete. 2024-05-09T08:41:36.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.035+0000 7f07edb69740 20 client.18108 trim_cache size 0 max 16384 2024-05-09T08:41:36.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.054+0000 7f07edb69740 1 Processor -- start 2024-05-09T08:41:36.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.054+0000 7f07edb69740 1 -- start start 2024-05-09T08:41:36.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.054+0000 7f07edb69740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8ed280 0x561e9f8ef520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.054+0000 7f07edb69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8edba0 0x561e9f908330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.054+0000 7f07edb69740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8f8270 0x561e9f914380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.054+0000 7f07edb69740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f07e41af2f0 con 0x561e9f8f8270 2024-05-09T08:41:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.054+0000 7f07edb69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f07e41acbc0 con 0x561e9f8edba0 2024-05-09T08:41:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.054+0000 7f07edb69740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f07e41aaa30 con 0x561e9f8ed280 2024-05-09T08:41:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.054+0000 7f07e931b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8f8270 0x561e9f914380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.054+0000 7f07e931b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8f8270 0x561e9f914380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58170/0 (socket says 172.21.15.107:58170) 2024-05-09T08:41:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.054+0000 7f07e931b640 1 -- 172.21.15.107:0/3912846514 learned_addr learned my addr 172.21.15.107:0/3912846514 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.054+0000 7f07e8b1a640 1 --2- 172.21.15.107:0/3912846514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8ed280 0x561e9f8ef520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.054+0000 7f07e3fff640 1 --2- 172.21.15.107:0/3912846514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8edba0 0x561e9f908330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.055+0000 7f07e931b640 1 -- 172.21.15.107:0/3912846514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8ed280 msgr2=0x561e9f8ef520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.055+0000 7f07e931b640 1 --2- 172.21.15.107:0/3912846514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8ed280 0x561e9f8ef520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.055+0000 7f07e931b640 1 -- 172.21.15.107:0/3912846514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8edba0 msgr2=0x561e9f908330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.055+0000 7f07e931b640 1 --2- 172.21.15.107:0/3912846514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8edba0 0x561e9f908330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.055+0000 7f07e931b640 1 -- 172.21.15.107:0/3912846514 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e9f8f37c0 con 0x561e9f8f8270 2024-05-09T08:41:36.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.055+0000 7f07e931b640 1 --2- 172.21.15.107:0/3912846514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8f8270 0x561e9f914380 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f07dc003fa0 tx=0x7f07dc0b9020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c377268cf92ea4bf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:36.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.055+0000 7f07ce7fc640 1 -- 172.21.15.107:0/3912846514 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc0098b0 con 0x561e9f8f8270 2024-05-09T08:41:36.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.055+0000 7f07ce7fc640 1 -- 172.21.15.107:0/3912846514 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07dc036030 con 0x561e9f8f8270 2024-05-09T08:41:36.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.055+0000 7f07ce7fc640 1 -- 172.21.15.107:0/3912846514 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc0ee280 con 0x561e9f8f8270 2024-05-09T08:41:36.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.056+0000 7f07edb69740 1 -- 172.21.15.107:0/3912846514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8f8270 msgr2=0x561e9f914380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:36.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.056+0000 7f07edb69740 1 --2- 172.21.15.107:0/3912846514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8f8270 0x561e9f914380 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f07dc003fa0 tx=0x7f07dc0b9020 comp rx=0 tx=0).stop 2024-05-09T08:41:36.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.056+0000 7f07edb69740 1 -- 172.21.15.107:0/3912846514 shutdown_connections 2024-05-09T08:41:36.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.056+0000 7f07edb69740 1 --2- 172.21.15.107:0/3912846514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8f8270 0x561e9f914380 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.056+0000 7f07edb69740 1 --2- 172.21.15.107:0/3912846514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8edba0 0x561e9f908330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.056+0000 7f07edb69740 1 --2- 172.21.15.107:0/3912846514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8ed280 0x561e9f8ef520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.056+0000 7f07edb69740 1 -- 172.21.15.107:0/3912846514 >> 172.21.15.107:0/3912846514 conn(0x561e9f8e95a0 msgr2=0x561e9f8f3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:36.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.056+0000 7f07edb69740 1 -- 172.21.15.107:0/3912846514 shutdown_connections 2024-05-09T08:41:36.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.056+0000 7f07edb69740 1 -- 172.21.15.107:0/3912846514 wait complete. 2024-05-09T08:41:36.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.056+0000 7f07edb69740 1 -- start start 2024-05-09T08:41:36.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.056+0000 7f07edb69740 1 Processor -- start 2024-05-09T08:41:36.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07edb69740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8ed280 0x561e9f904d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07edb69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8edba0 0x561e9f9000f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07edb69740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8f8270 0x561e9f900830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07edb69740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f07e41ab080 con 0x561e9f8f8270 2024-05-09T08:41:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07edb69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f07e41ac4f0 con 0x561e9f8edba0 2024-05-09T08:41:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07edb69740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f07e41ab750 con 0x561e9f8ed280 2024-05-09T08:41:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07edb69740 10 client.0 _ll_register_callbacks cb 0x7f07d80098f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07e931b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8f8270 0x561e9f900830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07e931b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8f8270 0x561e9f900830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58184/0 (socket says 172.21.15.107:58184) 2024-05-09T08:41:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07e931b640 1 -- 172.21.15.107:0/3530176698 learned_addr learned my addr 172.21.15.107:0/3530176698 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07e3fff640 1 --2- 172.21.15.107:0/3530176698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8edba0 0x561e9f9000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07e8b1a640 1 --2- 172.21.15.107:0/3530176698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8ed280 0x561e9f904d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07e931b640 1 -- 172.21.15.107:0/3530176698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8ed280 msgr2=0x561e9f904d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07e931b640 1 --2- 172.21.15.107:0/3530176698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8ed280 0x561e9f904d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07e931b640 1 -- 172.21.15.107:0/3530176698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8edba0 msgr2=0x561e9f9000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07e931b640 1 --2- 172.21.15.107:0/3530176698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8edba0 0x561e9f9000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07e931b640 1 -- 172.21.15.107:0/3530176698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e9f971bf0 con 0x561e9f8f8270 2024-05-09T08:41:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07e3fff640 1 --2- 172.21.15.107:0/3530176698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8edba0 0x561e9f9000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07e8b1a640 1 --2- 172.21.15.107:0/3530176698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8ed280 0x561e9f904d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07e931b640 1 --2- 172.21.15.107:0/3530176698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8f8270 0x561e9f900830 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f07dc0b91e0 tx=0x7f07dc0bcdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd92de88f4799b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.057+0000 7f07c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.058+0000 7f07c3fff640 1 -- 172.21.15.107:0/3530176698 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc0d9070 con 0x561e9f8f8270 2024-05-09T08:41:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.058+0000 7f07c3fff640 1 -- 172.21.15.107:0/3530176698 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f07dc0b93f0 con 0x561e9f8f8270 2024-05-09T08:41:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.058+0000 7f07edb69740 1 -- 172.21.15.107:0/3530176698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07dc036030 con 0x561e9f8f8270 2024-05-09T08:41:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.058+0000 7f07c3fff640 1 -- 172.21.15.107:0/3530176698 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f07dc0c59f0 con 0x561e9f8f8270 2024-05-09T08:41:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.058+0000 7f07edb69740 10 client.18114 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.058+0000 7f07edb69740 1 -- 172.21.15.107:0/3530176698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561e9f972da0 con 0x561e9f8f8270 2024-05-09T08:41:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.058+0000 7f07c3fff640 1 -- 172.21.15.107:0/3530176698 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07dc036030 con 0x561e9f8f8270 2024-05-09T08:41:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.058+0000 7f07c3fff640 1 -- 172.21.15.107:0/3530176698 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f07dc0c5560 con 0x561e9f8f8270 2024-05-09T08:41:36.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.058+0000 7f07edb69740 10 client.18114 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:36.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.058+0000 7f07edb69740 10 client.18114 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:36.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.058+0000 7f07edb69740 20 client.18114 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:36.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.058+0000 7f07edb69740 1 --2- 172.21.15.107:0/3530176698 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561e9f973400 0x561e9f9757f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:36.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.058+0000 7f07edb69740 4 client.18114 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:41:36.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.058+0000 7f07edb69740 1 -- 172.21.15.107:0/3530176698 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561e9f85a780 con 0x561e9f973400 2024-05-09T08:41:36.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.058+0000 7f07e8b1a640 1 --2- 172.21.15.107:0/3530176698 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561e9f973400 0x561e9f9757f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:36.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.059+0000 7f07e8b1a640 1 --2- 172.21.15.107:0/3530176698 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561e9f973400 0x561e9f9757f0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e83f277b7399ac5 server_cookie=d3c4009482f79222 in_seq=0 out_seq=0 2024-05-09T08:41:36.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.059+0000 7f07c3fff640 10 client.18114 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:36.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.059+0000 7f07c3fff640 1 -- 172.21.15.107:0/3530176698 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561e9f85a780 con 0x561e9f973400 2024-05-09T08:41:36.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.059+0000 7f07c3fff640 10 client.18114 handle_command_reply: tid=0 2024-05-09T08:41:36.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.059+0000 7f07edb69740 1 client.18114 shutdown 2024-05-09T08:41:36.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.059+0000 7f07edb69740 1 -- 172.21.15.107:0/3530176698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8f8270 msgr2=0x561e9f900830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:36.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.059+0000 7f07edb69740 1 --2- 172.21.15.107:0/3530176698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8f8270 0x561e9f900830 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f07dc0b91e0 tx=0x7f07dc0bcdc0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.059+0000 7f07edb69740 1 -- 172.21.15.107:0/3530176698 shutdown_connections 2024-05-09T08:41:36.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.059+0000 7f07edb69740 1 --2- 172.21.15.107:0/3530176698 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561e9f973400 0x561e9f9757f0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.059+0000 7f07edb69740 1 --2- 172.21.15.107:0/3530176698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561e9f8f8270 0x561e9f900830 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.059+0000 7f07edb69740 1 --2- 172.21.15.107:0/3530176698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561e9f8edba0 0x561e9f9000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.059+0000 7f07edb69740 1 --2- 172.21.15.107:0/3530176698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e9f8ed280 0x561e9f904d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.059+0000 7f07edb69740 1 -- 172.21.15.107:0/3530176698 >> 172.21.15.107:0/3530176698 conn(0x561e9f8e95a0 msgr2=0x561e9f8e7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:36.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.059+0000 7f07c3fff640 0 client.18114 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:36.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.059+0000 7f07edb69740 1 -- 172.21.15.107:0/3530176698 shutdown_connections 2024-05-09T08:41:36.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.059+0000 7f07edb69740 1 -- 172.21.15.107:0/3530176698 wait complete. 2024-05-09T08:41:36.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.060+0000 7f07edb69740 20 client.18114 trim_cache size 0 max 16384 2024-05-09T08:41:36.065 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:41:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:41:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:41:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: "8de51e0f-d92c-4e96-8a29-70553747beb1": { 2024-05-09T08:41:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:41:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "8de51e0f-d92c-4e96-8a29-70553747beb1", 2024-05-09T08:41:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:41:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:41:36.066 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:41:36.066 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:41:36.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.060+0000 7f07ead7e640 1 -- 172.21.15.107:0/1889188087 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07d40a2d00 msgr2=0x7f07d40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:36.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.060+0000 7f07ead7e640 1 --2- 172.21.15.107:0/1889188087 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07d40a2d00 0x7f07d40a51c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f07d0009fd0 tx=0x7f07d0009500 comp rx=0 tx=0).stop 2024-05-09T08:41:36.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.060+0000 7f07ead7e640 1 -- 172.21.15.107:0/1889188087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07e419b850 msgr2=0x7f07e419bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:36.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.060+0000 7f07ead7e640 1 --2- 172.21.15.107:0/1889188087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07e419b850 0x7f07e419bd00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f07dc031040 tx=0x7f07dc003970 comp rx=0 tx=0).stop 2024-05-09T08:41:36.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.060+0000 7f07ead7e640 1 -- 172.21.15.107:0/1889188087 shutdown_connections 2024-05-09T08:41:36.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.060+0000 7f07ead7e640 1 --2- 172.21.15.107:0/1889188087 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07d40a2d00 0x7f07d40a51c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.060+0000 7f07ead7e640 1 --2- 172.21.15.107:0/1889188087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07e419b850 0x7f07e419bd00 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.060+0000 7f07ead7e640 1 --2- 172.21.15.107:0/1889188087 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e4106bd0 0x7f07e4197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.060+0000 7f07ead7e640 1 --2- 172.21.15.107:0/1889188087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07e4101710 0x7f07e4196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:36.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.060+0000 7f07ead7e640 1 -- 172.21.15.107:0/1889188087 >> 172.21.15.107:0/1889188087 conn(0x7f07e4075a30 msgr2=0x7f07e4104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:36.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.060+0000 7f07ead7e640 1 -- 172.21.15.107:0/1889188087 shutdown_connections 2024-05-09T08:41:36.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:41:36.060+0000 7f07ead7e640 1 -- 172.21.15.107:0/1889188087 wait complete. 2024-05-09T08:41:36.071 INFO:tasks.cephfs.filesystem:scrub status for tag:8de51e0f-d92c-4e96-8a29-70553747beb1 - {'path': '/', 'tag': '8de51e0f-d92c-4e96-8a29-70553747beb1', 'options': 'recursive,force'} 2024-05-09T08:41:36.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:36 smithi157 ceph-mon[33614]: pgmap v254: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 20 op/s 2024-05-09T08:41:36.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:36 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:41:36.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:36 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:36.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:36 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:41:36.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:36 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:36.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:36 smithi110 ceph-mon[33622]: pgmap v254: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 20 op/s 2024-05-09T08:41:36.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:36 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:41:36.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:36 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:36.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:36 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:41:36.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:36 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:36.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:36 smithi107 ceph-mon[30122]: pgmap v254: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 24 MiB/s wr, 20 op/s 2024-05-09T08:41:36.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:36 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:41:36.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:36 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:36.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:36 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:41:36.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:36 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:37.345 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:37 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:41:37.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:37 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:37.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:37 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:41:37.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:37 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:37.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:37 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:41:37.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:37 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:37.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:37 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:41:37.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:37 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:37.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:37 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:41:37.346 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:37 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:37.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:37 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:41:37.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:37 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:37.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:37 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:41:37.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:37 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:37.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:37 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:41:37.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:37 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:37.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:37 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:41:37.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:37 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:37.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:37 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:41:37.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:37 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:37.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:37 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:41:37.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:37 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:37.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:37 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:41:37.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:37 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:37.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:37 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:41:37.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:37 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:37.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:37 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:41:37.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:37 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:37.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:37 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:41:37.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:37 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:41:38.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:38 smithi157 ceph-mon[33614]: pgmap v255: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 36 MiB/s wr, 61 op/s 2024-05-09T08:41:38.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:38 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:41:38.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:38 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:41:38.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:38 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:41:38.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:38 smithi110 ceph-mon[33622]: pgmap v255: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 36 MiB/s wr, 61 op/s 2024-05-09T08:41:38.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:38 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:41:38.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:38 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:41:38.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:38 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:41:38.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:38 smithi107 ceph-mon[30122]: pgmap v255: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 36 MiB/s wr, 61 op/s 2024-05-09T08:41:38.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:38 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:41:38.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:38 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:41:38.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:38 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:41:40.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:40 smithi157 ceph-mon[33614]: pgmap v256: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 24 MiB/s wr, 53 op/s 2024-05-09T08:41:40.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:40 smithi110 ceph-mon[33622]: pgmap v256: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 24 MiB/s wr, 53 op/s 2024-05-09T08:41:40.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:40 smithi107 ceph-mon[30122]: pgmap v256: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 24 MiB/s wr, 53 op/s 2024-05-09T08:41:42.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:42 smithi157 ceph-mon[33614]: pgmap v257: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 24 MiB/s wr, 53 op/s 2024-05-09T08:41:42.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:42 smithi110 ceph-mon[33622]: pgmap v257: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 24 MiB/s wr, 53 op/s 2024-05-09T08:41:42.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:42 smithi107 ceph-mon[30122]: pgmap v257: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 24 MiB/s wr, 53 op/s 2024-05-09T08:41:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.152+0000 7fa326645640 1 -- 172.21.15.107:0/3449250443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa320100760 msgr2=0x7fa320100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.152+0000 7fa326645640 1 --2- 172.21.15.107:0/3449250443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa320100760 0x7fa320100be0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fa314009930 tx=0x7fa31402a420 comp rx=0 tx=0).stop 2024-05-09T08:41:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.152+0000 7fa326645640 1 -- 172.21.15.107:0/3449250443 shutdown_connections 2024-05-09T08:41:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.152+0000 7fa326645640 1 --2- 172.21.15.107:0/3449250443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa320101120 0x7fa32010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.152+0000 7fa326645640 1 --2- 172.21.15.107:0/3449250443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa320100760 0x7fa320100be0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.152+0000 7fa326645640 1 --2- 172.21.15.107:0/3449250443 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa320106720 0x7fa320106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.152+0000 7fa326645640 1 -- 172.21.15.107:0/3449250443 >> 172.21.15.107:0/3449250443 conn(0x7fa3200778b0 msgr2=0x7fa320077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.152+0000 7fa326645640 1 -- 172.21.15.107:0/3449250443 shutdown_connections 2024-05-09T08:41:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.152+0000 7fa326645640 1 -- 172.21.15.107:0/3449250443 wait complete. 2024-05-09T08:41:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.153+0000 7fa326645640 1 Processor -- start 2024-05-09T08:41:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.153+0000 7fa326645640 1 -- start start 2024-05-09T08:41:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.153+0000 7fa326645640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa320100760 0x7fa32019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.153+0000 7fa326645640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa320101120 0x7fa32019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.153+0000 7fa326645640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa320106720 0x7fa32019ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.153+0000 7fa326645640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa320113d00 con 0x7fa320100760 2024-05-09T08:41:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.153+0000 7fa326645640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa320113b80 con 0x7fa320106720 2024-05-09T08:41:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.153+0000 7fa326645640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa320113e80 con 0x7fa320101120 2024-05-09T08:41:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.153+0000 7fa31ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa320100760 0x7fa32019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.153+0000 7fa31ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa320100760 0x7fa32019a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59526/0 (socket says 172.21.15.107:59526) 2024-05-09T08:41:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.153+0000 7fa31ffff640 1 -- 172.21.15.107:0/2201468893 learned_addr learned my addr 172.21.15.107:0/2201468893 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.153+0000 7fa31f7fe640 1 --2- 172.21.15.107:0/2201468893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa320101120 0x7fa32019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.154+0000 7fa31ffff640 1 -- 172.21.15.107:0/2201468893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa320101120 msgr2=0x7fa32019a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.154+0000 7fa324be2640 1 --2- 172.21.15.107:0/2201468893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa320106720 0x7fa32019ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.154+0000 7fa31ffff640 1 --2- 172.21.15.107:0/2201468893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa320101120 0x7fa32019a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.154+0000 7fa31ffff640 1 -- 172.21.15.107:0/2201468893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa320106720 msgr2=0x7fa32019ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:43.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.154+0000 7fa31ffff640 1 --2- 172.21.15.107:0/2201468893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa320106720 0x7fa32019ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.154+0000 7fa31ffff640 1 -- 172.21.15.107:0/2201468893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa32019f1b0 con 0x7fa320100760 2024-05-09T08:41:43.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.154+0000 7fa31ffff640 1 --2- 172.21.15.107:0/2201468893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa320100760 0x7fa32019a1c0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fa31000d770 tx=0x7fa31000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:43.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.154+0000 7fa31d7fa640 1 -- 172.21.15.107:0/2201468893 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa310017070 con 0x7fa320100760 2024-05-09T08:41:43.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.154+0000 7fa31d7fa640 1 -- 172.21.15.107:0/2201468893 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3100054d0 con 0x7fa320100760 2024-05-09T08:41:43.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.154+0000 7fa326645640 1 -- 172.21.15.107:0/2201468893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa32019f4a0 con 0x7fa320100760 2024-05-09T08:41:43.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.154+0000 7fa31d7fa640 1 -- 172.21.15.107:0/2201468893 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa310004cf0 con 0x7fa320100760 2024-05-09T08:41:43.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.154+0000 7fa326645640 1 -- 172.21.15.107:0/2201468893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3201a6d30 con 0x7fa320100760 2024-05-09T08:41:43.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.156+0000 7fa31d7fa640 1 -- 172.21.15.107:0/2201468893 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fa310004e90 con 0x7fa320100760 2024-05-09T08:41:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.157+0000 7fa32942a740 1 Processor -- start 2024-05-09T08:41:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.157+0000 7fa31d7fa640 1 --2- 172.21.15.107:0/2201468893 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa3080a2da0 0x7fa3080a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.157+0000 7fa32942a740 1 -- start start 2024-05-09T08:41:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.157+0000 7fa32942a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a245450 0x55929a245830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.157+0000 7fa324be2640 1 --2- 172.21.15.107:0/2201468893 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa3080a2da0 0x7fa3080a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.157+0000 7fa31d7fa640 1 -- 172.21.15.107:0/2201468893 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fa3100cf280 con 0x7fa320100760 2024-05-09T08:41:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.157+0000 7fa32942a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a245d70 0x55929a2500f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.157+0000 7fa32942a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a250630 0x55929a252a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.157+0000 7fa32942a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55929a1aa340 con 0x55929a245d70 2024-05-09T08:41:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.157+0000 7fa32942a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55929a152290 con 0x55929a250630 2024-05-09T08:41:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.157+0000 7fa32942a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55929a1a6300 con 0x55929a245450 2024-05-09T08:41:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.157+0000 7fa31f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a245d70 0x55929a2500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.157+0000 7fa31f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a245d70 0x55929a2500f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59542/0 (socket says 172.21.15.107:59542) 2024-05-09T08:41:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.157+0000 7fa31f7fe640 1 -- 172.21.15.107:0/3832939925 learned_addr learned my addr 172.21.15.107:0/3832939925 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.157+0000 7fa31ffff640 1 --2- 172.21.15.107:0/3832939925 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a245450 0x55929a245830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.158+0000 7fa324be2640 1 --2- 172.21.15.107:0/3832939925 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a250630 0x55929a252a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.158+0000 7fa31f7fe640 1 -- 172.21.15.107:0/3832939925 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a245450 msgr2=0x55929a245830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.158+0000 7fa31f7fe640 1 --2- 172.21.15.107:0/3832939925 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a245450 0x55929a245830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.158+0000 7fa31f7fe640 1 -- 172.21.15.107:0/3832939925 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a250630 msgr2=0x55929a252a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.158+0000 7fa31f7fe640 1 --2- 172.21.15.107:0/3832939925 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a250630 0x55929a252a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.158+0000 7fa31f7fe640 1 -- 172.21.15.107:0/3832939925 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55929a1768e0 con 0x55929a245d70 2024-05-09T08:41:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.158+0000 7fa324be2640 1 --2- 172.21.15.107:0/3832939925 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a250630 0x55929a252a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:41:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.158+0000 7fa324be2640 1 --2- 172.21.15.107:0/2201468893 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa3080a2da0 0x7fa3080a5260 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa318005fd0 tx=0x7fa31800f730 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.158+0000 7fa31f7fe640 1 --2- 172.21.15.107:0/3832939925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a245d70 0x55929a2500f0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fa314031440 tx=0x7fa314032b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c774ff50879eef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.158+0000 7fa3067fc640 1 -- 172.21.15.107:0/3832939925 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31403f070 con 0x55929a245d70 2024-05-09T08:41:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.158+0000 7fa3067fc640 1 -- 172.21.15.107:0/3832939925 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa314004030 con 0x55929a245d70 2024-05-09T08:41:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.158+0000 7fa3067fc640 1 -- 172.21.15.107:0/3832939925 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa314035070 con 0x55929a245d70 2024-05-09T08:41:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.159+0000 7fa32942a740 1 -- 172.21.15.107:0/3832939925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a245d70 msgr2=0x55929a2500f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.159+0000 7fa32942a740 1 --2- 172.21.15.107:0/3832939925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a245d70 0x55929a2500f0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fa314031440 tx=0x7fa314032b80 comp rx=0 tx=0).stop 2024-05-09T08:41:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.159+0000 7fa32942a740 1 -- 172.21.15.107:0/3832939925 shutdown_connections 2024-05-09T08:41:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.159+0000 7fa32942a740 1 --2- 172.21.15.107:0/3832939925 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a250630 0x55929a252a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.159+0000 7fa32942a740 1 --2- 172.21.15.107:0/3832939925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a245d70 0x55929a2500f0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.159+0000 7fa32942a740 1 --2- 172.21.15.107:0/3832939925 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a245450 0x55929a245830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.159+0000 7fa32942a740 1 -- 172.21.15.107:0/3832939925 >> 172.21.15.107:0/3832939925 conn(0x55929a241960 msgr2=0x55929a241d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.159+0000 7fa32942a740 1 -- 172.21.15.107:0/3832939925 shutdown_connections 2024-05-09T08:41:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.159+0000 7fa32942a740 1 -- 172.21.15.107:0/3832939925 wait complete. 2024-05-09T08:41:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.159+0000 7fa32942a740 1 -- start start 2024-05-09T08:41:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.159+0000 7fa32942a740 1 Processor -- start 2024-05-09T08:41:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa32942a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a245450 0x55929a24d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa32942a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a245d70 0x55929a24daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa32942a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a250630 0x55929a24e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa32942a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55929a253200 con 0x55929a250630 2024-05-09T08:41:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa32942a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55929a165790 con 0x55929a245d70 2024-05-09T08:41:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa32942a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55929a253380 con 0x55929a245450 2024-05-09T08:41:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa32942a740 10 client.0 _ll_register_callbacks cb 0x55929a1a0a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa324be2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a250630 0x55929a24e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa324be2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a250630 0x55929a24e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59546/0 (socket says 172.21.15.107:59546) 2024-05-09T08:41:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa324be2640 1 -- 172.21.15.107:0/2748105677 learned_addr learned my addr 172.21.15.107:0/2748105677 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa31f7fe640 1 --2- 172.21.15.107:0/2748105677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a245d70 0x55929a24daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa31ffff640 1 --2- 172.21.15.107:0/2748105677 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a245450 0x55929a24d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa324be2640 1 -- 172.21.15.107:0/2748105677 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a245450 msgr2=0x55929a24d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa324be2640 1 --2- 172.21.15.107:0/2748105677 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a245450 0x55929a24d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa324be2640 1 -- 172.21.15.107:0/2748105677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a245d70 msgr2=0x55929a24daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa324be2640 1 --2- 172.21.15.107:0/2748105677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a245d70 0x55929a24daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa324be2640 1 -- 172.21.15.107:0/2748105677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559299ebe210 con 0x55929a250630 2024-05-09T08:41:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa31ffff640 1 --2- 172.21.15.107:0/2748105677 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a245450 0x55929a24d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa324be2640 1 --2- 172.21.15.107:0/2748105677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a250630 0x55929a24e030 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fa318012870 tx=0x7fa318004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be50c8b25a215a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa2ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.160+0000 7fa2ff7fe640 1 -- 172.21.15.107:0/2748105677 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31801dd50 con 0x55929a250630 2024-05-09T08:41:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.161+0000 7fa2ff7fe640 1 -- 172.21.15.107:0/2748105677 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fa318026430 con 0x55929a250630 2024-05-09T08:41:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.161+0000 7fa32942a740 1 -- 172.21.15.107:0/2748105677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa314004030 con 0x55929a250630 2024-05-09T08:41:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.161+0000 7fa2ff7fe640 1 -- 172.21.15.107:0/2748105677 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa318031050 con 0x55929a250630 2024-05-09T08:41:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.161+0000 7fa32942a740 10 client.18126 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.161+0000 7fa32942a740 1 -- 172.21.15.107:0/2748105677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559299f8a620 con 0x55929a250630 2024-05-09T08:41:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.161+0000 7fa2ff7fe640 1 -- 172.21.15.107:0/2748105677 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa314004030 con 0x55929a250630 2024-05-09T08:41:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.161+0000 7fa2ff7fe640 1 -- 172.21.15.107:0/2748105677 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa31801e630 con 0x55929a250630 2024-05-09T08:41:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.161+0000 7fa32942a740 10 client.18126 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.161+0000 7fa32942a740 10 client.18126 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.161+0000 7fa32942a740 20 client.18126 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.161+0000 7fa32942a740 1 --2- 172.21.15.107:0/2748105677 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55929a2603f0 0x55929a262800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.161+0000 7fa32942a740 4 client.18126 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.161+0000 7fa32942a740 1 -- 172.21.15.107:0/2748105677 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55929a1b7a70 con 0x55929a2603f0 2024-05-09T08:41:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.161+0000 7fa31ffff640 1 --2- 172.21.15.107:0/2748105677 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55929a2603f0 0x55929a262800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.162+0000 7fa2ff7fe640 10 client.18126 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.162+0000 7fa31ffff640 1 --2- 172.21.15.107:0/2748105677 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55929a2603f0 0x55929a262800 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72c3e6a16ba70d26 server_cookie=3a29e965428d26cd in_seq=0 out_seq=0 2024-05-09T08:41:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.163+0000 7fa2ff7fe640 1 -- 172.21.15.107:0/2748105677 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55929a1b7a70 con 0x55929a2603f0 2024-05-09T08:41:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.163+0000 7fa2ff7fe640 10 client.18126 handle_command_reply: tid=0 2024-05-09T08:41:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.163+0000 7fa32942a740 1 client.18126 shutdown 2024-05-09T08:41:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.163+0000 7fa32942a740 1 -- 172.21.15.107:0/2748105677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a250630 msgr2=0x55929a24e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.163+0000 7fa32942a740 1 --2- 172.21.15.107:0/2748105677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a250630 0x55929a24e030 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fa318012870 tx=0x7fa318004ab0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.163+0000 7fa32942a740 1 -- 172.21.15.107:0/2748105677 shutdown_connections 2024-05-09T08:41:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.163+0000 7fa32942a740 1 --2- 172.21.15.107:0/2748105677 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55929a2603f0 0x55929a262800 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.163+0000 7fa2ff7fe640 0 client.18126 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.163+0000 7fa32942a740 1 --2- 172.21.15.107:0/2748105677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a250630 0x55929a24e030 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.163+0000 7fa32942a740 1 --2- 172.21.15.107:0/2748105677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a245d70 0x55929a24daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.163+0000 7fa32942a740 1 --2- 172.21.15.107:0/2748105677 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a245450 0x55929a24d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.163+0000 7fa32942a740 1 -- 172.21.15.107:0/2748105677 >> 172.21.15.107:0/2748105677 conn(0x55929a241960 msgr2=0x55929a247690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.163+0000 7fa32942a740 1 -- 172.21.15.107:0/2748105677 shutdown_connections 2024-05-09T08:41:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.163+0000 7fa32942a740 1 -- 172.21.15.107:0/2748105677 wait complete. 2024-05-09T08:41:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.163+0000 7fa32942a740 20 client.18126 trim_cache size 0 max 16384 2024-05-09T08:41:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.179+0000 7fa32942a740 1 Processor -- start 2024-05-09T08:41:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.179+0000 7fa32942a740 1 -- start start 2024-05-09T08:41:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.179+0000 7fa32942a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a245450 0x55929a25c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.179+0000 7fa32942a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a245d70 0x55929a247e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.179+0000 7fa32942a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a250630 0x55929a24dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.179+0000 7fa32942a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa3201b35f0 con 0x55929a250630 2024-05-09T08:41:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.179+0000 7fa32942a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa3201b0ec0 con 0x55929a245d70 2024-05-09T08:41:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.179+0000 7fa32942a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa3201aed30 con 0x55929a245450 2024-05-09T08:41:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.179+0000 7fa324be2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a250630 0x55929a24dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.179+0000 7fa324be2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a250630 0x55929a24dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59552/0 (socket says 172.21.15.107:59552) 2024-05-09T08:41:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.179+0000 7fa324be2640 1 -- 172.21.15.107:0/2590611877 learned_addr learned my addr 172.21.15.107:0/2590611877 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.180+0000 7fa31ffff640 1 --2- 172.21.15.107:0/2590611877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a245450 0x55929a25c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.180+0000 7fa31f7fe640 1 --2- 172.21.15.107:0/2590611877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a245d70 0x55929a247e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.180+0000 7fa324be2640 1 -- 172.21.15.107:0/2590611877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a245450 msgr2=0x55929a25c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.180+0000 7fa324be2640 1 --2- 172.21.15.107:0/2590611877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a245450 0x55929a25c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.180+0000 7fa324be2640 1 -- 172.21.15.107:0/2590611877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a245d70 msgr2=0x55929a247e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.180+0000 7fa324be2640 1 --2- 172.21.15.107:0/2590611877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a245d70 0x55929a247e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.180+0000 7fa324be2640 1 -- 172.21.15.107:0/2590611877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55929a248350 con 0x55929a250630 2024-05-09T08:41:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.180+0000 7fa324be2640 1 --2- 172.21.15.107:0/2590611877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a250630 0x55929a24dc80 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fa318009970 tx=0x7fa318009340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f8e0b5b4edf7cf8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.180+0000 7fa305ffb640 1 -- 172.21.15.107:0/2590611877 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa318009650 con 0x55929a250630 2024-05-09T08:41:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.180+0000 7fa305ffb640 1 -- 172.21.15.107:0/2590611877 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa31803a070 con 0x55929a250630 2024-05-09T08:41:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.180+0000 7fa305ffb640 1 -- 172.21.15.107:0/2590611877 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31801e770 con 0x55929a250630 2024-05-09T08:41:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.181+0000 7fa32942a740 1 -- 172.21.15.107:0/2590611877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a250630 msgr2=0x55929a24dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.181+0000 7fa32942a740 1 --2- 172.21.15.107:0/2590611877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a250630 0x55929a24dc80 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fa318009970 tx=0x7fa318009340 comp rx=0 tx=0).stop 2024-05-09T08:41:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.181+0000 7fa32942a740 1 -- 172.21.15.107:0/2590611877 shutdown_connections 2024-05-09T08:41:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.181+0000 7fa32942a740 1 --2- 172.21.15.107:0/2590611877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a250630 0x55929a24dc80 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.181+0000 7fa32942a740 1 --2- 172.21.15.107:0/2590611877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a245d70 0x55929a247e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.181+0000 7fa32942a740 1 --2- 172.21.15.107:0/2590611877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a245450 0x55929a25c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.181+0000 7fa32942a740 1 -- 172.21.15.107:0/2590611877 >> 172.21.15.107:0/2590611877 conn(0x55929a241960 msgr2=0x55929a24d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.181+0000 7fa32942a740 1 -- 172.21.15.107:0/2590611877 shutdown_connections 2024-05-09T08:41:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.181+0000 7fa32942a740 1 -- 172.21.15.107:0/2590611877 wait complete. 2024-05-09T08:41:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.181+0000 7fa32942a740 1 -- start start 2024-05-09T08:41:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa32942a740 1 Processor -- start 2024-05-09T08:41:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa32942a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a245450 0x55929a2587c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa32942a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a245d70 0x55929a249e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa32942a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a250630 0x55929a24a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa32942a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa3201af380 con 0x55929a245450 2024-05-09T08:41:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa32942a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa3201b07f0 con 0x55929a245d70 2024-05-09T08:41:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa32942a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa3201afa50 con 0x55929a250630 2024-05-09T08:41:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa32942a740 10 client.0 _ll_register_callbacks cb 0x7fa31800fee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa31ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a245450 0x55929a2587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa31ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a245450 0x55929a2587c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59564/0 (socket says 172.21.15.107:59564) 2024-05-09T08:41:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa31ffff640 1 -- 172.21.15.107:0/959520546 learned_addr learned my addr 172.21.15.107:0/959520546 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa324be2640 1 --2- 172.21.15.107:0/959520546 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a250630 0x55929a24a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa31f7fe640 1 --2- 172.21.15.107:0/959520546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a245d70 0x55929a249e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa31ffff640 1 -- 172.21.15.107:0/959520546 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a250630 msgr2=0x55929a24a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa31ffff640 1 --2- 172.21.15.107:0/959520546 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a250630 0x55929a24a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa31ffff640 1 -- 172.21.15.107:0/959520546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a245d70 msgr2=0x55929a249e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa31ffff640 1 --2- 172.21.15.107:0/959520546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a245d70 0x55929a249e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.182+0000 7fa31ffff640 1 -- 172.21.15.107:0/959520546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55929a24a9c0 con 0x55929a245450 2024-05-09T08:41:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.183+0000 7fa31ffff640 1 --2- 172.21.15.107:0/959520546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a245450 0x55929a2587c0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fa310097dc0 tx=0x7fa310097720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4b57a627a1e641 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.183+0000 7fa2ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.183+0000 7fa2ff7fe640 1 -- 172.21.15.107:0/959520546 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3100b4060 con 0x55929a245450 2024-05-09T08:41:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.183+0000 7fa2ff7fe640 1 -- 172.21.15.107:0/959520546 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fa310097a60 con 0x55929a245450 2024-05-09T08:41:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.183+0000 7fa32942a740 1 -- 172.21.15.107:0/959520546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa31803a070 con 0x55929a245450 2024-05-09T08:41:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.183+0000 7fa2ff7fe640 1 -- 172.21.15.107:0/959520546 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa310084030 con 0x55929a245450 2024-05-09T08:41:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.183+0000 7fa32942a740 10 client.18132 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.183+0000 7fa32942a740 1 -- 172.21.15.107:0/959520546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55929a2c67b0 con 0x55929a245450 2024-05-09T08:41:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.183+0000 7fa2ff7fe640 1 -- 172.21.15.107:0/959520546 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa31803a070 con 0x55929a245450 2024-05-09T08:41:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.183+0000 7fa2ff7fe640 1 -- 172.21.15.107:0/959520546 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa3100b4200 con 0x55929a245450 2024-05-09T08:41:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.183+0000 7fa32942a740 10 client.18132 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.183+0000 7fa32942a740 10 client.18132 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.183+0000 7fa32942a740 20 client.18132 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.183+0000 7fa32942a740 1 --2- 172.21.15.107:0/959520546 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55929a2603f0 0x55929a2c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.183+0000 7fa32942a740 4 client.18132 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:41:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.183+0000 7fa32942a740 1 -- 172.21.15.107:0/959520546 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55929a1b1dc0 con 0x55929a2603f0 2024-05-09T08:41:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.184+0000 7fa31f7fe640 1 --2- 172.21.15.107:0/959520546 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55929a2603f0 0x55929a2c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.184+0000 7fa31f7fe640 1 --2- 172.21.15.107:0/959520546 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55929a2603f0 0x55929a2c8f70 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe6430014d8a7c03 server_cookie=2a901b650bad4927 in_seq=0 out_seq=0 2024-05-09T08:41:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.184+0000 7fa2ff7fe640 10 client.18132 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.200+0000 7fa2ff7fe640 1 -- 172.21.15.107:0/959520546 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55929a1b1dc0 con 0x55929a2603f0 2024-05-09T08:41:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.200+0000 7fa2ff7fe640 10 client.18132 handle_command_reply: tid=0 2024-05-09T08:41:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.200+0000 7fa32942a740 1 client.18132 shutdown 2024-05-09T08:41:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.200+0000 7fa32942a740 1 -- 172.21.15.107:0/959520546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a245450 msgr2=0x55929a2587c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.200+0000 7fa32942a740 1 --2- 172.21.15.107:0/959520546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a245450 0x55929a2587c0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fa310097dc0 tx=0x7fa310097720 comp rx=0 tx=0).stop 2024-05-09T08:41:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.200+0000 7fa32942a740 1 -- 172.21.15.107:0/959520546 shutdown_connections 2024-05-09T08:41:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.200+0000 7fa32942a740 1 --2- 172.21.15.107:0/959520546 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55929a2603f0 0x55929a2c8f70 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.200+0000 7fa32942a740 1 --2- 172.21.15.107:0/959520546 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55929a250630 0x55929a24a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.200+0000 7fa32942a740 1 --2- 172.21.15.107:0/959520546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55929a245d70 0x55929a249e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.200+0000 7fa2ff7fe640 0 client.18132 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.200+0000 7fa32942a740 1 --2- 172.21.15.107:0/959520546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55929a245450 0x55929a2587c0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.200+0000 7fa32942a740 1 -- 172.21.15.107:0/959520546 >> 172.21.15.107:0/959520546 conn(0x55929a241960 msgr2=0x55929a24de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.200+0000 7fa32942a740 1 -- 172.21.15.107:0/959520546 shutdown_connections 2024-05-09T08:41:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.200+0000 7fa32942a740 1 -- 172.21.15.107:0/959520546 wait complete. 2024-05-09T08:41:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.200+0000 7fa32942a740 20 client.18132 trim_cache size 0 max 16384 2024-05-09T08:41:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:41:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:41:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:41:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:41:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.201+0000 7fa326645640 1 -- 172.21.15.107:0/2201468893 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa3080a2da0 msgr2=0x7fa3080a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.201+0000 7fa326645640 1 --2- 172.21.15.107:0/2201468893 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa3080a2da0 0x7fa3080a5260 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa318005fd0 tx=0x7fa31800f730 comp rx=0 tx=0).stop 2024-05-09T08:41:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.201+0000 7fa326645640 1 -- 172.21.15.107:0/2201468893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa320100760 msgr2=0x7fa32019a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.201+0000 7fa326645640 1 --2- 172.21.15.107:0/2201468893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa320100760 0x7fa32019a1c0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fa31000d770 tx=0x7fa31000dc30 comp rx=0 tx=0).stop 2024-05-09T08:41:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.201+0000 7fa326645640 1 -- 172.21.15.107:0/2201468893 shutdown_connections 2024-05-09T08:41:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.201+0000 7fa326645640 1 --2- 172.21.15.107:0/2201468893 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa3080a2da0 0x7fa3080a5260 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.201+0000 7fa326645640 1 --2- 172.21.15.107:0/2201468893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa320106720 0x7fa32019ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.201+0000 7fa326645640 1 --2- 172.21.15.107:0/2201468893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa320101120 0x7fa32019a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.201+0000 7fa326645640 1 --2- 172.21.15.107:0/2201468893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa320100760 0x7fa32019a1c0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.201+0000 7fa326645640 1 -- 172.21.15.107:0/2201468893 >> 172.21.15.107:0/2201468893 conn(0x7fa3200778b0 msgr2=0x7fa3201032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.201+0000 7fa326645640 1 -- 172.21.15.107:0/2201468893 shutdown_connections 2024-05-09T08:41:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:43.201+0000 7fa326645640 1 -- 172.21.15.107:0/2201468893 wait complete. 2024-05-09T08:41:44.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:44 smithi157 ceph-mon[33614]: pgmap v258: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 36 MiB/s wr, 61 op/s 2024-05-09T08:41:44.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:44 smithi110 ceph-mon[33622]: pgmap v258: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 36 MiB/s wr, 61 op/s 2024-05-09T08:41:44.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:44 smithi107 ceph-mon[30122]: pgmap v258: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 36 MiB/s wr, 61 op/s 2024-05-09T08:41:46.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:46 smithi107 ceph-mon[30122]: pgmap v259: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 24 MiB/s wr, 48 op/s 2024-05-09T08:41:46.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:46 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:41:46.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:46 smithi157 ceph-mon[33614]: pgmap v259: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 24 MiB/s wr, 48 op/s 2024-05-09T08:41:46.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:46 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:41:46.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:46 smithi110 ceph-mon[33622]: pgmap v259: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 24 MiB/s wr, 48 op/s 2024-05-09T08:41:46.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:46 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:41:47.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:47 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:41:47.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:47 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:41:47.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:47 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:41:47.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:47 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:41:47.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:47 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:41:47.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:47 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:41:48.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:48 smithi157 ceph-mon[33614]: pgmap v260: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 37 MiB/s wr, 61 op/s 2024-05-09T08:41:48.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:48 smithi110 ceph-mon[33622]: pgmap v260: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 37 MiB/s wr, 61 op/s 2024-05-09T08:41:48.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:48 smithi107 ceph-mon[30122]: pgmap v260: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 37 MiB/s wr, 61 op/s 2024-05-09T08:41:50.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:50 smithi157 ceph-mon[33614]: pgmap v261: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:41:50.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:50 smithi110 ceph-mon[33622]: pgmap v261: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:41:50.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:50 smithi107 ceph-mon[30122]: pgmap v261: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:41:52.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:52 smithi157 ceph-mon[33614]: pgmap v262: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:41:52.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:52 smithi110 ceph-mon[33622]: pgmap v262: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:41:52.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:52 smithi107 ceph-mon[30122]: pgmap v262: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:41:54.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:54 smithi157 ceph-mon[33614]: pgmap v263: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:41:54.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:54 smithi110 ceph-mon[33622]: pgmap v263: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:41:54.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:54 smithi107 ceph-mon[30122]: pgmap v263: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 37 MiB/s wr, 27 op/s 2024-05-09T08:41:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.297+0000 7f2d7d110640 1 -- 172.21.15.107:0/3977580262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d78107850 msgr2=0x7f2d78107c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.297+0000 7f2d7d110640 1 --2- 172.21.15.107:0/3977580262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d78107850 0x7f2d78107c30 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f2d68009930 tx=0x7f2d6802a420 comp rx=0 tx=0).stop 2024-05-09T08:41:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.297+0000 7f2d7d110640 1 -- 172.21.15.107:0/3977580262 shutdown_connections 2024-05-09T08:41:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.297+0000 7f2d7d110640 1 --2- 172.21.15.107:0/3977580262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d78102200 0x7f2d7810e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.297+0000 7f2d7d110640 1 --2- 172.21.15.107:0/3977580262 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d78101840 0x7f2d78101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.297+0000 7f2d7d110640 1 --2- 172.21.15.107:0/3977580262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d78107850 0x7f2d78107c30 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.297+0000 7f2d7d110640 1 -- 172.21.15.107:0/3977580262 >> 172.21.15.107:0/3977580262 conn(0x7f2d780fd330 msgr2=0x7f2d780ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.297+0000 7f2d7d110640 1 -- 172.21.15.107:0/3977580262 shutdown_connections 2024-05-09T08:41:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.297+0000 7f2d7d110640 1 -- 172.21.15.107:0/3977580262 wait complete. 2024-05-09T08:41:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.298+0000 7f2d7d110640 1 Processor -- start 2024-05-09T08:41:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.298+0000 7f2d7d110640 1 -- start start 2024-05-09T08:41:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.298+0000 7f2d7d110640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d78101840 0x7f2d7819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.298+0000 7f2d7d110640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d78102200 0x7f2d7819a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.298+0000 7f2d7d110640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d78107850 0x7f2d7819eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.298+0000 7f2d7d110640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2d78111b30 con 0x7f2d78101840 2024-05-09T08:41:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.298+0000 7f2d7d110640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2d781119b0 con 0x7f2d78107850 2024-05-09T08:41:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.298+0000 7f2d7d110640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2d78111cb0 con 0x7f2d78102200 2024-05-09T08:41:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.298+0000 7f2d76d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d78101840 0x7f2d7819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.298+0000 7f2d76d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d78101840 0x7f2d7819a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54562/0 (socket says 172.21.15.107:54562) 2024-05-09T08:41:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.298+0000 7f2d76d9d640 1 -- 172.21.15.107:0/291794555 learned_addr learned my addr 172.21.15.107:0/291794555 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.298+0000 7f2d7659c640 1 --2- 172.21.15.107:0/291794555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d78102200 0x7f2d7819a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.298+0000 7f2d7759e640 1 --2- 172.21.15.107:0/291794555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d78107850 0x7f2d7819eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.299+0000 7f2d7659c640 1 -- 172.21.15.107:0/291794555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d78107850 msgr2=0x7f2d7819eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.299+0000 7f2d7659c640 1 --2- 172.21.15.107:0/291794555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d78107850 0x7f2d7819eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.299+0000 7f2d7659c640 1 -- 172.21.15.107:0/291794555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d78101840 msgr2=0x7f2d7819a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.299+0000 7f2d7659c640 1 --2- 172.21.15.107:0/291794555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d78101840 0x7f2d7819a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.299+0000 7f2d7659c640 1 -- 172.21.15.107:0/291794555 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d7819f250 con 0x7f2d78102200 2024-05-09T08:41:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.299+0000 7f2d76d9d640 1 --2- 172.21.15.107:0/291794555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d78101840 0x7f2d7819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:41:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.299+0000 7f2d7659c640 1 --2- 172.21.15.107:0/291794555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d78102200 0x7f2d7819a7a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00d720 tx=0x7f2d6c00dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.299+0000 7f2d5ffff640 1 -- 172.21.15.107:0/291794555 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c017070 con 0x7f2d78102200 2024-05-09T08:41:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.299+0000 7f2d7d110640 1 -- 172.21.15.107:0/291794555 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d7819f540 con 0x7f2d78102200 2024-05-09T08:41:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.299+0000 7f2d5ffff640 1 -- 172.21.15.107:0/291794555 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d6c00de30 con 0x7f2d78102200 2024-05-09T08:41:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.299+0000 7f2d5ffff640 1 -- 172.21.15.107:0/291794555 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c0047a0 con 0x7f2d78102200 2024-05-09T08:41:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.299+0000 7f2d7d110640 1 -- 172.21.15.107:0/291794555 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d781a6dd0 con 0x7f2d78102200 2024-05-09T08:41:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.301+0000 7f2d5ffff640 1 -- 172.21.15.107:0/291794555 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f2d6c0099d0 con 0x7f2d78102200 2024-05-09T08:41:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d7fe18740 1 Processor -- start 2024-05-09T08:41:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d7fe18740 1 -- start start 2024-05-09T08:41:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d7fe18740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314450 0x556d85314830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d7fe18740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d85314d70 0x556d8531f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d7fe18740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d8531f630 0x556d85321a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d7fe18740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556d85279340 con 0x556d85314450 2024-05-09T08:41:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d7fe18740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556d85221290 con 0x556d8531f630 2024-05-09T08:41:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d7fe18740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x556d85275300 con 0x556d85314d70 2024-05-09T08:41:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d5ffff640 1 --2- 172.21.15.107:0/291794555 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2d600a2da0 0x7f2d600a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d7759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314450 0x556d85314830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d76d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d85314d70 0x556d8531f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d7759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314450 0x556d85314830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54572/0 (socket says 172.21.15.107:54572) 2024-05-09T08:41:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d7759e640 1 -- 172.21.15.107:0/1519741141 learned_addr learned my addr 172.21.15.107:0/1519741141 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d7759e640 1 --2- 172.21.15.107:0/291794555 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2d600a2da0 0x7f2d600a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d7659c640 1 --2- 172.21.15.107:0/1519741141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d8531f630 0x556d85321a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d5ffff640 1 -- 172.21.15.107:0/291794555 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f2d6c0cf420 con 0x7f2d78102200 2024-05-09T08:41:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d76d9d640 1 -- 172.21.15.107:0/1519741141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d8531f630 msgr2=0x556d85321a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d76d9d640 1 --2- 172.21.15.107:0/1519741141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d8531f630 0x556d85321a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d76d9d640 1 -- 172.21.15.107:0/1519741141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314450 msgr2=0x556d85314830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.302+0000 7f2d76d9d640 1 --2- 172.21.15.107:0/1519741141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314450 0x556d85314830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.303+0000 7f2d76d9d640 1 -- 172.21.15.107:0/1519741141 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d852458e0 con 0x556d85314d70 2024-05-09T08:41:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.303+0000 7f2d7759e640 1 --2- 172.21.15.107:0/291794555 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2d600a2da0 0x7f2d600a5260 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2d70009900 tx=0x7f2d70009650 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.303+0000 7f2d76d9d640 1 --2- 172.21.15.107:0/1519741141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d85314d70 0x556d8531f0f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2d68004770 tx=0x7f2d68031570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ec3f1a6894d6505 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.303+0000 7f2d5d7fa640 1 -- 172.21.15.107:0/1519741141 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d68035040 con 0x556d85314d70 2024-05-09T08:41:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.303+0000 7f2d5d7fa640 1 -- 172.21.15.107:0/1519741141 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d68005020 con 0x556d85314d70 2024-05-09T08:41:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.303+0000 7f2d5d7fa640 1 -- 172.21.15.107:0/1519741141 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d68004a00 con 0x556d85314d70 2024-05-09T08:41:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.304+0000 7f2d7fe18740 1 -- 172.21.15.107:0/1519741141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d85314d70 msgr2=0x556d8531f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.304+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/1519741141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d85314d70 0x556d8531f0f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2d68004770 tx=0x7f2d68031570 comp rx=0 tx=0).stop 2024-05-09T08:41:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.304+0000 7f2d7fe18740 1 -- 172.21.15.107:0/1519741141 shutdown_connections 2024-05-09T08:41:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.304+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/1519741141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d8531f630 0x556d85321a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.304+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/1519741141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d85314d70 0x556d8531f0f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.304+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/1519741141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314450 0x556d85314830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.304+0000 7f2d7fe18740 1 -- 172.21.15.107:0/1519741141 >> 172.21.15.107:0/1519741141 conn(0x556d85310960 msgr2=0x556d85310d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.304+0000 7f2d7fe18740 1 -- 172.21.15.107:0/1519741141 shutdown_connections 2024-05-09T08:41:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.304+0000 7f2d7fe18740 1 -- 172.21.15.107:0/1519741141 wait complete. 2024-05-09T08:41:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.304+0000 7f2d7fe18740 1 -- start start 2024-05-09T08:41:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.304+0000 7f2d7fe18740 1 Processor -- start 2024-05-09T08:41:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7fe18740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d85314450 0x556d8531c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7fe18740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314d70 0x556d8531ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7fe18740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d8531f630 0x556d8531cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7fe18740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556d85322200 con 0x556d85314d70 2024-05-09T08:41:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7fe18740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556d85234790 con 0x556d85314450 2024-05-09T08:41:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7fe18740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x556d85322380 con 0x556d8531f630 2024-05-09T08:41:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7fe18740 10 client.0 _ll_register_callbacks cb 0x556d8526fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314d70 0x556d8531ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314d70 0x556d8531ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54586/0 (socket says 172.21.15.107:54586) 2024-05-09T08:41:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7659c640 1 -- 172.21.15.107:0/2596555053 learned_addr learned my addr 172.21.15.107:0/2596555053 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d76d9d640 1 --2- 172.21.15.107:0/2596555053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d85314450 0x556d8531c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7659c640 1 -- 172.21.15.107:0/2596555053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d8531f630 msgr2=0x556d8531cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7759e640 1 --2- 172.21.15.107:0/2596555053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d8531f630 0x556d8531cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7659c640 1 --2- 172.21.15.107:0/2596555053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d8531f630 0x556d8531cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7659c640 1 -- 172.21.15.107:0/2596555053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d85314450 msgr2=0x556d8531c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7659c640 1 --2- 172.21.15.107:0/2596555053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d85314450 0x556d8531c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7659c640 1 -- 172.21.15.107:0/2596555053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d84f8d210 con 0x556d85314d70 2024-05-09T08:41:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7759e640 1 --2- 172.21.15.107:0/2596555053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d8531f630 0x556d8531cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:41:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d7659c640 1 --2- 172.21.15.107:0/2596555053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314d70 0x556d8531ca70 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c0a3c70 tx=0x7f2d6c004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563972f0d26debb5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d5aff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.305+0000 7f2d5aff5640 1 -- 172.21.15.107:0/2596555053 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c0bb040 con 0x556d85314d70 2024-05-09T08:41:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.306+0000 7f2d5aff5640 1 -- 172.21.15.107:0/2596555053 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f2d6c009cc0 con 0x556d85314d70 2024-05-09T08:41:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.306+0000 7f2d7fe18740 1 -- 172.21.15.107:0/2596555053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d68005020 con 0x556d85314d70 2024-05-09T08:41:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.306+0000 7f2d5aff5640 1 -- 172.21.15.107:0/2596555053 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2d6c09ae90 con 0x556d85314d70 2024-05-09T08:41:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.306+0000 7f2d7fe18740 10 client.18144 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.306+0000 7f2d7fe18740 1 -- 172.21.15.107:0/2596555053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556d85059620 con 0x556d85314d70 2024-05-09T08:41:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.306+0000 7f2d5aff5640 1 -- 172.21.15.107:0/2596555053 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d68005020 con 0x556d85314d70 2024-05-09T08:41:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.306+0000 7f2d5aff5640 1 -- 172.21.15.107:0/2596555053 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f2d6c097ba0 con 0x556d85314d70 2024-05-09T08:41:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.306+0000 7f2d7fe18740 10 client.18144 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.306+0000 7f2d7fe18740 10 client.18144 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.306+0000 7f2d7fe18740 20 client.18144 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.306+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/2596555053 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556d8532f2f0 0x556d853316e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.306+0000 7f2d7fe18740 4 client.18144 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:41:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.306+0000 7f2d7fe18740 1 -- 172.21.15.107:0/2596555053 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d85286a70 con 0x556d8532f2f0 2024-05-09T08:41:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.307+0000 7f2d76d9d640 1 --2- 172.21.15.107:0/2596555053 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556d8532f2f0 0x556d853316e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.307+0000 7f2d76d9d640 1 --2- 172.21.15.107:0/2596555053 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556d8532f2f0 0x556d853316e0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=461bc8be7bc2d114 server_cookie=1ca2e982a495e7e5 in_seq=0 out_seq=0 2024-05-09T08:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.307+0000 7f2d5aff5640 10 client.18144 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.308+0000 7f2d5aff5640 1 -- 172.21.15.107:0/2596555053 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556d85286a70 con 0x556d8532f2f0 2024-05-09T08:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.308+0000 7f2d5aff5640 10 client.18144 handle_command_reply: tid=0 2024-05-09T08:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.308+0000 7f2d7fe18740 1 client.18144 shutdown 2024-05-09T08:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.308+0000 7f2d7fe18740 1 -- 172.21.15.107:0/2596555053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314d70 msgr2=0x556d8531ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.308+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/2596555053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314d70 0x556d8531ca70 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c0a3c70 tx=0x7f2d6c004f90 comp rx=0 tx=0).stop 2024-05-09T08:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.308+0000 7f2d7fe18740 1 -- 172.21.15.107:0/2596555053 shutdown_connections 2024-05-09T08:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.308+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/2596555053 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556d8532f2f0 0x556d853316e0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.308+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/2596555053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d8531f630 0x556d8531cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.308+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/2596555053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314d70 0x556d8531ca70 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.308+0000 7f2d5aff5640 0 client.18144 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.308+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/2596555053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d85314450 0x556d8531c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.308+0000 7f2d7fe18740 1 -- 172.21.15.107:0/2596555053 >> 172.21.15.107:0/2596555053 conn(0x556d85310960 msgr2=0x556d853166e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.308+0000 7f2d7fe18740 1 -- 172.21.15.107:0/2596555053 shutdown_connections 2024-05-09T08:41:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.308+0000 7f2d7fe18740 1 -- 172.21.15.107:0/2596555053 wait complete. 2024-05-09T08:41:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.308+0000 7f2d7fe18740 20 client.18144 trim_cache size 0 max 16384 2024-05-09T08:41:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.325+0000 7f2d7fe18740 1 Processor -- start 2024-05-09T08:41:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.325+0000 7f2d7fe18740 1 -- start start 2024-05-09T08:41:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d7fe18740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d85314450 0x556d8531ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d7fe18740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d85314d70 0x556d8532b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d7fe18740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d8531f630 0x556d8531b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d7fe18740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2d781b3670 con 0x556d8531f630 2024-05-09T08:41:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d7fe18740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2d781b0f40 con 0x556d85314450 2024-05-09T08:41:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d7fe18740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2d781aedb0 con 0x556d85314d70 2024-05-09T08:41:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d7759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d8531f630 0x556d8531b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d7759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d8531f630 0x556d8531b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54592/0 (socket says 172.21.15.107:54592) 2024-05-09T08:41:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d7759e640 1 -- 172.21.15.107:0/1387377456 learned_addr learned my addr 172.21.15.107:0/1387377456 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d7659c640 1 --2- 172.21.15.107:0/1387377456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d85314d70 0x556d8532b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d76d9d640 1 --2- 172.21.15.107:0/1387377456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d85314450 0x556d8531ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d7759e640 1 -- 172.21.15.107:0/1387377456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d85314d70 msgr2=0x556d8532b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d7759e640 1 --2- 172.21.15.107:0/1387377456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d85314d70 0x556d8532b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d7759e640 1 -- 172.21.15.107:0/1387377456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d85314450 msgr2=0x556d8531ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d7759e640 1 --2- 172.21.15.107:0/1387377456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d85314450 0x556d8531ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d7759e640 1 -- 172.21.15.107:0/1387377456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d85321d90 con 0x556d8531f630 2024-05-09T08:41:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.326+0000 7f2d7759e640 1 --2- 172.21.15.107:0/1387377456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d8531f630 0x556d8531b690 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f2d70009f90 tx=0x7f2d70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa8d043915f07843 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.327+0000 7f2d5cff9640 1 -- 172.21.15.107:0/1387377456 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d70014040 con 0x556d8531f630 2024-05-09T08:41:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.327+0000 7f2d5cff9640 1 -- 172.21.15.107:0/1387377456 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d7001cbb0 con 0x556d8531f630 2024-05-09T08:41:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.327+0000 7f2d5cff9640 1 -- 172.21.15.107:0/1387377456 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d70026be0 con 0x556d8531f630 2024-05-09T08:41:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.327+0000 7f2d7fe18740 1 -- 172.21.15.107:0/1387377456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d8531f630 msgr2=0x556d8531b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.327+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/1387377456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d8531f630 0x556d8531b690 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f2d70009f90 tx=0x7f2d70009bb0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.327+0000 7f2d7fe18740 1 -- 172.21.15.107:0/1387377456 shutdown_connections 2024-05-09T08:41:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.327+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/1387377456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d8531f630 0x556d8531b690 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.327+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/1387377456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d85314d70 0x556d8532b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.327+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/1387377456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d85314450 0x556d8531ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.327+0000 7f2d7fe18740 1 -- 172.21.15.107:0/1387377456 >> 172.21.15.107:0/1387377456 conn(0x556d85310960 msgr2=0x556d8531c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.327+0000 7f2d7fe18740 1 -- 172.21.15.107:0/1387377456 shutdown_connections 2024-05-09T08:41:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.327+0000 7f2d7fe18740 1 -- 172.21.15.107:0/1387377456 wait complete. 2024-05-09T08:41:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.328+0000 7f2d7fe18740 1 -- start start 2024-05-09T08:41:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.328+0000 7f2d7fe18740 1 Processor -- start 2024-05-09T08:41:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.328+0000 7f2d7fe18740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d85314450 0x556d85327940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.328+0000 7f2d7fe18740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314d70 0x556d85322cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.328+0000 7f2d7fe18740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d8531f630 0x556d853231f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.328+0000 7f2d7fe18740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2d781af400 con 0x556d85314d70 2024-05-09T08:41:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.328+0000 7f2d7fe18740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2d781b0870 con 0x556d85314450 2024-05-09T08:41:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.328+0000 7f2d7fe18740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2d781afad0 con 0x556d8531f630 2024-05-09T08:41:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.328+0000 7f2d7fe18740 10 client.0 _ll_register_callbacks cb 0x7f2d6c097110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:41:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.328+0000 7f2d7659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314d70 0x556d85322cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.328+0000 7f2d7659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314d70 0x556d85322cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54596/0 (socket says 172.21.15.107:54596) 2024-05-09T08:41:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.328+0000 7f2d7659c640 1 -- 172.21.15.107:0/2439100959 learned_addr learned my addr 172.21.15.107:0/2439100959 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:41:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.328+0000 7f2d76d9d640 1 --2- 172.21.15.107:0/2439100959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d85314450 0x556d85327940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.328+0000 7f2d7759e640 1 --2- 172.21.15.107:0/2439100959 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d8531f630 0x556d853231f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d7659c640 1 -- 172.21.15.107:0/2439100959 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d8531f630 msgr2=0x556d853231f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d7659c640 1 --2- 172.21.15.107:0/2439100959 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d8531f630 0x556d853231f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d7659c640 1 -- 172.21.15.107:0/2439100959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d85314450 msgr2=0x556d85327940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:41:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d7659c640 1 --2- 172.21.15.107:0/2439100959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d85314450 0x556d85327940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d7659c640 1 -- 172.21.15.107:0/2439100959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d85323a20 con 0x556d85314d70 2024-05-09T08:41:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d7659c640 1 --2- 172.21.15.107:0/2439100959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314d70 0x556d85322cb0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c09b4c0 tx=0x7f2d6c09b300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6157eb0c20b2c190 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:41:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d5aff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:41:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d5aff5640 1 -- 172.21.15.107:0/2439100959 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c0bb040 con 0x556d85314d70 2024-05-09T08:41:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d5aff5640 1 -- 172.21.15.107:0/2439100959 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f2d6c0b42a0 con 0x556d85314d70 2024-05-09T08:41:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d7fe18740 1 -- 172.21.15.107:0/2439100959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d7001cbb0 con 0x556d85314d70 2024-05-09T08:41:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d5aff5640 1 -- 172.21.15.107:0/2439100959 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2d6c01fe30 con 0x556d85314d70 2024-05-09T08:41:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d7fe18740 10 client.18150 fetch_fsmap learned FSMap version 27 2024-05-09T08:41:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d7fe18740 1 -- 172.21.15.107:0/2439100959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556d85395800 con 0x556d85314d70 2024-05-09T08:41:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d5aff5640 1 -- 172.21.15.107:0/2439100959 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d7001cbb0 con 0x556d85314d70 2024-05-09T08:41:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d5aff5640 1 -- 172.21.15.107:0/2439100959 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f2d6c097280 con 0x556d85314d70 2024-05-09T08:41:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d7fe18740 10 client.18150 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:41:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d7fe18740 10 client.18150 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:41:56.336 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.329+0000 7f2d7fe18740 20 client.18150 populate_metadata read hostname 'smithi107' 2024-05-09T08:41:56.336 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.330+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/2439100959 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556d85395d30 0x556d85398120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:41:56.336 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.330+0000 7f2d7fe18740 4 client.18150 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:41:56.336 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.330+0000 7f2d7fe18740 1 -- 172.21.15.107:0/2439100959 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556d85280dc0 con 0x556d85395d30 2024-05-09T08:41:56.336 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.330+0000 7f2d76d9d640 1 --2- 172.21.15.107:0/2439100959 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556d85395d30 0x556d85398120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:41:56.336 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.330+0000 7f2d76d9d640 1 --2- 172.21.15.107:0/2439100959 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556d85395d30 0x556d85398120 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c085986a039fa29c server_cookie=4da0112cd910543b in_seq=0 out_seq=0 2024-05-09T08:41:56.336 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.330+0000 7f2d5aff5640 10 client.18150 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:56.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:56 smithi157 ceph-mon[33614]: pgmap v264: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:41:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.346+0000 7f2d5aff5640 1 -- 172.21.15.107:0/2439100959 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556d85280dc0 con 0x556d85395d30 2024-05-09T08:41:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.346+0000 7f2d5aff5640 10 client.18150 handle_command_reply: tid=0 2024-05-09T08:41:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.346+0000 7f2d7fe18740 1 client.18150 shutdown 2024-05-09T08:41:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.346+0000 7f2d7fe18740 1 -- 172.21.15.107:0/2439100959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314d70 msgr2=0x556d85322cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.346+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/2439100959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314d70 0x556d85322cb0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c09b4c0 tx=0x7f2d6c09b300 comp rx=0 tx=0).stop 2024-05-09T08:41:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.346+0000 7f2d7fe18740 1 -- 172.21.15.107:0/2439100959 shutdown_connections 2024-05-09T08:41:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.346+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/2439100959 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556d85395d30 0x556d85398120 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.346+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/2439100959 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556d8531f630 0x556d853231f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.346+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/2439100959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d85314d70 0x556d85322cb0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.346+0000 7f2d5aff5640 0 client.18150 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:41:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.346+0000 7f2d7fe18740 1 --2- 172.21.15.107:0/2439100959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556d85314450 0x556d85327940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.346+0000 7f2d7fe18740 1 -- 172.21.15.107:0/2439100959 >> 172.21.15.107:0/2439100959 conn(0x556d85310960 msgr2=0x556d8531cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.346+0000 7f2d7fe18740 1 -- 172.21.15.107:0/2439100959 shutdown_connections 2024-05-09T08:41:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.346+0000 7f2d7fe18740 1 -- 172.21.15.107:0/2439100959 wait complete. 2024-05-09T08:41:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.346+0000 7f2d7fe18740 20 client.18150 trim_cache size 0 max 16384 2024-05-09T08:41:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:41:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:41:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:41:56.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:41:56.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.347+0000 7f2d7d110640 1 -- 172.21.15.107:0/291794555 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2d600a2da0 msgr2=0x7f2d600a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:56.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.347+0000 7f2d7d110640 1 --2- 172.21.15.107:0/291794555 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2d600a2da0 0x7f2d600a5260 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2d70009900 tx=0x7f2d70009650 comp rx=0 tx=0).stop 2024-05-09T08:41:56.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.347+0000 7f2d7d110640 1 -- 172.21.15.107:0/291794555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d78102200 msgr2=0x7f2d7819a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:41:56.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.347+0000 7f2d7d110640 1 --2- 172.21.15.107:0/291794555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d78102200 0x7f2d7819a7a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00d720 tx=0x7f2d6c00dbe0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.347+0000 7f2d7d110640 1 -- 172.21.15.107:0/291794555 shutdown_connections 2024-05-09T08:41:56.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.347+0000 7f2d7d110640 1 --2- 172.21.15.107:0/291794555 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2d600a2da0 0x7f2d600a5260 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.347+0000 7f2d7d110640 1 --2- 172.21.15.107:0/291794555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d78107850 0x7f2d7819eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.347+0000 7f2d7d110640 1 --2- 172.21.15.107:0/291794555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d78102200 0x7f2d7819a7a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.347+0000 7f2d7d110640 1 --2- 172.21.15.107:0/291794555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d78101840 0x7f2d7819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:41:56.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.347+0000 7f2d7d110640 1 -- 172.21.15.107:0/291794555 >> 172.21.15.107:0/291794555 conn(0x7f2d780fd330 msgr2=0x7f2d7810e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:41:56.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.347+0000 7f2d7d110640 1 -- 172.21.15.107:0/291794555 shutdown_connections 2024-05-09T08:41:56.351 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:41:56.347+0000 7f2d7d110640 1 -- 172.21.15.107:0/291794555 wait complete. 2024-05-09T08:41:56.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:56 smithi110 ceph-mon[33622]: pgmap v264: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:41:56.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:56 smithi107 ceph-mon[30122]: pgmap v264: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:41:58.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:41:58 smithi157 ceph-mon[33614]: pgmap v265: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 37 MiB/s wr, 32 op/s 2024-05-09T08:41:58.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:41:58 smithi110 ceph-mon[33622]: pgmap v265: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 37 MiB/s wr, 32 op/s 2024-05-09T08:41:58.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:41:58 smithi107 ceph-mon[30122]: pgmap v265: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 37 MiB/s wr, 32 op/s 2024-05-09T08:42:00.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:00 smithi157 ceph-mon[33614]: pgmap v266: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:42:00.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:00 smithi110 ceph-mon[33622]: pgmap v266: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:42:00.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:00 smithi107 ceph-mon[30122]: pgmap v266: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:42:02.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:02 smithi157 ceph-mon[33614]: pgmap v267: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:42:02.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:02 smithi157 ceph-mon[33614]: osdmap e99: 12 total, 12 up, 12 in 2024-05-09T08:42:02.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:02 smithi110 ceph-mon[33622]: pgmap v267: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:42:02.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:02 smithi110 ceph-mon[33622]: osdmap e99: 12 total, 12 up, 12 in 2024-05-09T08:42:02.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:02 smithi107 ceph-mon[30122]: pgmap v267: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-05-09T08:42:02.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:02 smithi107 ceph-mon[30122]: osdmap e99: 12 total, 12 up, 12 in 2024-05-09T08:42:04.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:04 smithi110 ceph-mon[33622]: pgmap v269: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 30 MiB/s wr, 44 op/s 2024-05-09T08:42:04.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:04 smithi107 ceph-mon[30122]: pgmap v269: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 30 MiB/s wr, 44 op/s 2024-05-09T08:42:04.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:04 smithi157 ceph-mon[33614]: pgmap v269: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 30 MiB/s wr, 44 op/s 2024-05-09T08:42:06.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:42:06.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:42:06.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.187+0000 7f8a76b1b640 1 -- 172.21.15.107:0/564012961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a70106be0 msgr2=0x7f8a7010e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.187+0000 7f8a76b1b640 1 --2- 172.21.15.107:0/564012961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a70106be0 0x7f8a7010e740 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f8a68009cd0 tx=0x7f8a6802a8d0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.187+0000 7f8a76b1b640 1 -- 172.21.15.107:0/564012961 shutdown_connections 2024-05-09T08:42:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.187+0000 7f8a76b1b640 1 --2- 172.21.15.107:0/564012961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a70106be0 0x7f8a7010e740 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.187+0000 7f8a76b1b640 1 --2- 172.21.15.107:0/564012961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a70100980 0x7f8a701064b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.187+0000 7f8a76b1b640 1 --2- 172.21.15.107:0/564012961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a700fffd0 0x7f8a701003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.187+0000 7f8a76b1b640 1 -- 172.21.15.107:0/564012961 >> 172.21.15.107:0/564012961 conn(0x7f8a70076cd0 msgr2=0x7f8a700fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.187+0000 7f8a76b1b640 1 -- 172.21.15.107:0/564012961 shutdown_connections 2024-05-09T08:42:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.187+0000 7f8a76b1b640 1 -- 172.21.15.107:0/564012961 wait complete. 2024-05-09T08:42:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a76b1b640 1 Processor -- start 2024-05-09T08:42:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a76b1b640 1 -- start start 2024-05-09T08:42:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a76b1b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a700fffd0 0x7f8a7019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a76b1b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a70106be0 0x7f8a7019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a76b1b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a7019eb10 0x7f8a7019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a76b1b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a70111aa0 con 0x7f8a70106be0 2024-05-09T08:42:06.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a76b1b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a70111920 con 0x7f8a700fffd0 2024-05-09T08:42:06.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a76b1b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8a70111c20 con 0x7f8a7019eb10 2024-05-09T08:42:06.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a6ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a70106be0 0x7f8a7019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a6ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a70106be0 0x7f8a7019a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41484/0 (socket says 172.21.15.107:41484) 2024-05-09T08:42:06.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a748b7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a700fffd0 0x7f8a7019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a6ffff640 1 -- 172.21.15.107:0/3355500084 learned_addr learned my addr 172.21.15.107:0/3355500084 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:06.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a750b8640 1 --2- 172.21.15.107:0/3355500084 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a7019eb10 0x7f8a7019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a6ffff640 1 -- 172.21.15.107:0/3355500084 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a7019eb10 msgr2=0x7f8a7019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a6ffff640 1 --2- 172.21.15.107:0/3355500084 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a7019eb10 0x7f8a7019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a6ffff640 1 -- 172.21.15.107:0/3355500084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a700fffd0 msgr2=0x7f8a7019a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a6ffff640 1 --2- 172.21.15.107:0/3355500084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a700fffd0 0x7f8a7019a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a6ffff640 1 -- 172.21.15.107:0/3355500084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7019f600 con 0x7f8a70106be0 2024-05-09T08:42:06.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.188+0000 7f8a748b7640 1 --2- 172.21.15.107:0/3355500084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a700fffd0 0x7f8a7019a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:42:06.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.189+0000 7f8a6ffff640 1 --2- 172.21.15.107:0/3355500084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a70106be0 0x7f8a7019a790 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c00e9c0 tx=0x7f8a5c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.189+0000 7f8a6dffb640 1 -- 172.21.15.107:0/3355500084 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c00cd80 con 0x7f8a70106be0 2024-05-09T08:42:06.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.189+0000 7f8a76b1b640 1 -- 172.21.15.107:0/3355500084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a7019f8f0 con 0x7f8a70106be0 2024-05-09T08:42:06.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.189+0000 7f8a6dffb640 1 -- 172.21.15.107:0/3355500084 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a5c0054d0 con 0x7f8a70106be0 2024-05-09T08:42:06.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.189+0000 7f8a6dffb640 1 -- 172.21.15.107:0/3355500084 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c010430 con 0x7f8a70106be0 2024-05-09T08:42:06.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.189+0000 7f8a76b1b640 1 -- 172.21.15.107:0/3355500084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a701a7130 con 0x7f8a70106be0 2024-05-09T08:42:06.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.192+0000 7f8a6dffb640 1 -- 172.21.15.107:0/3355500084 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8a5c004fd0 con 0x7f8a70106be0 2024-05-09T08:42:06.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.192+0000 7f8a7982b740 1 Processor -- start 2024-05-09T08:42:06.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.192+0000 7f8a7982b740 1 -- start start 2024-05-09T08:42:06.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.192+0000 7f8a7982b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dc280 0x56200b2dc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.192+0000 7f8a7982b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dcba0 0x56200b2e6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.192+0000 7f8a7982b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 0x56200b2e9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a7982b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56200b241d00 con 0x56200b2dcba0 2024-05-09T08:42:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a7982b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56200b1e94c0 con 0x56200b2e7270 2024-05-09T08:42:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a7982b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56200b23dcc0 con 0x56200b2dc280 2024-05-09T08:42:06.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a748b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dcba0 0x56200b2e6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a748b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dcba0 0x56200b2e6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41494/0 (socket says 172.21.15.107:41494) 2024-05-09T08:42:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a748b7640 1 -- 172.21.15.107:0/3709435426 learned_addr learned my addr 172.21.15.107:0/3709435426 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a6dffb640 1 --2- 172.21.15.107:0/3355500084 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a600a2d00 0x7f8a600a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a750b8640 1 --2- 172.21.15.107:0/3709435426 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dc280 0x56200b2dc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a750b8640 1 --2- 172.21.15.107:0/3355500084 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a600a2d00 0x7f8a600a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a6ffff640 1 --2- 172.21.15.107:0/3709435426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 0x56200b2e9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a6dffb640 1 -- 172.21.15.107:0/3355500084 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9555+0+0 (secure 0 0 0) 0x7f8a5c0cfcd0 con 0x7f8a70106be0 2024-05-09T08:42:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a748b7640 1 -- 172.21.15.107:0/3709435426 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dc280 msgr2=0x56200b2dc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a748b7640 1 --2- 172.21.15.107:0/3709435426 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dc280 0x56200b2dc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a748b7640 1 -- 172.21.15.107:0/3709435426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 msgr2=0x56200b2e9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a748b7640 1 --2- 172.21.15.107:0/3709435426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 0x56200b2e9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a748b7640 1 -- 172.21.15.107:0/3709435426 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56200b01f7d0 con 0x56200b2dcba0 2024-05-09T08:42:06.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a750b8640 1 --2- 172.21.15.107:0/3709435426 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dc280 0x56200b2dc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:42:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a748b7640 1 --2- 172.21.15.107:0/3709435426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dcba0 0x56200b2e6d30 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f8a640104d0 tx=0x7f8a64010990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=396fd63cd4f5e2d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a750b8640 1 --2- 172.21.15.107:0/3355500084 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a600a2d00 0x7f8a600a51c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8a68009430 tx=0x7f8a680095c0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.193+0000 7f8a5affd640 1 -- 172.21.15.107:0/3709435426 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a64011760 con 0x56200b2dcba0 2024-05-09T08:42:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.194+0000 7f8a5affd640 1 -- 172.21.15.107:0/3709435426 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a64011db0 con 0x56200b2dcba0 2024-05-09T08:42:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.194+0000 7f8a5affd640 1 -- 172.21.15.107:0/3709435426 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6401cdf0 con 0x56200b2dcba0 2024-05-09T08:42:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.194+0000 7f8a7982b740 1 -- 172.21.15.107:0/3709435426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dcba0 msgr2=0x56200b2e6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.194+0000 7f8a7982b740 1 --2- 172.21.15.107:0/3709435426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dcba0 0x56200b2e6d30 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f8a640104d0 tx=0x7f8a64010990 comp rx=0 tx=0).stop 2024-05-09T08:42:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.194+0000 7f8a7982b740 1 -- 172.21.15.107:0/3709435426 shutdown_connections 2024-05-09T08:42:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.194+0000 7f8a7982b740 1 --2- 172.21.15.107:0/3709435426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 0x56200b2e9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.194+0000 7f8a7982b740 1 --2- 172.21.15.107:0/3709435426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dcba0 0x56200b2e6d30 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.194+0000 7f8a7982b740 1 --2- 172.21.15.107:0/3709435426 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dc280 0x56200b2dc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.194+0000 7f8a7982b740 1 -- 172.21.15.107:0/3709435426 >> 172.21.15.107:0/3709435426 conn(0x56200b2d85a0 msgr2=0x56200b2d8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.194+0000 7f8a7982b740 1 -- 172.21.15.107:0/3709435426 shutdown_connections 2024-05-09T08:42:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.194+0000 7f8a7982b740 1 -- 172.21.15.107:0/3709435426 wait complete. 2024-05-09T08:42:06.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.195+0000 7f8a7982b740 1 -- start start 2024-05-09T08:42:06.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.195+0000 7f8a7982b740 1 Processor -- start 2024-05-09T08:42:06.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.195+0000 7f8a7982b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dc280 0x56200b2e41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.195+0000 7f8a7982b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dcba0 0x56200b2e4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.195+0000 7f8a7982b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 0x56200b2e4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.195+0000 7f8a7982b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56200b2ea060 con 0x56200b2dc280 2024-05-09T08:42:06.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.195+0000 7f8a7982b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56200b1fcb90 con 0x56200b2e7270 2024-05-09T08:42:06.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.195+0000 7f8a7982b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56200b2ea1e0 con 0x56200b2dcba0 2024-05-09T08:42:06.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.195+0000 7f8a7982b740 10 client.0 _ll_register_callbacks cb 0x56200b238da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:06.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.195+0000 7f8a748b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dc280 0x56200b2e41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.195+0000 7f8a748b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dc280 0x56200b2e41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41496/0 (socket says 172.21.15.107:41496) 2024-05-09T08:42:06.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.196+0000 7f8a748b7640 1 -- 172.21.15.107:0/3534598793 learned_addr learned my addr 172.21.15.107:0/3534598793 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:06.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.196+0000 7f8a6ffff640 1 --2- 172.21.15.107:0/3534598793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dcba0 0x56200b2e4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.196+0000 7f8a750b8640 1 --2- 172.21.15.107:0/3534598793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 0x56200b2e4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.196+0000 7f8a748b7640 1 -- 172.21.15.107:0/3534598793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dcba0 msgr2=0x56200b2e4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.196+0000 7f8a748b7640 1 --2- 172.21.15.107:0/3534598793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dcba0 0x56200b2e4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.196+0000 7f8a748b7640 1 -- 172.21.15.107:0/3534598793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 msgr2=0x56200b2e4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.196+0000 7f8a748b7640 1 --2- 172.21.15.107:0/3534598793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 0x56200b2e4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.196+0000 7f8a748b7640 1 -- 172.21.15.107:0/3534598793 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56200af553f0 con 0x56200b2dc280 2024-05-09T08:42:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.196+0000 7f8a748b7640 1 --2- 172.21.15.107:0/3534598793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dc280 0x56200b2e41f0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400c010 tx=0x7f8a6401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf2aa02a126c751 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.196+0000 7f8a4ffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.196+0000 7f8a4ffff640 1 -- 172.21.15.107:0/3534598793 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6402a060 con 0x56200b2dc280 2024-05-09T08:42:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.196+0000 7f8a7982b740 1 -- 172.21.15.107:0/3534598793 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a64011db0 con 0x56200b2dc280 2024-05-09T08:42:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.196+0000 7f8a4ffff640 1 -- 172.21.15.107:0/3534598793 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9555+0+0 (secure 0 0 0) 0x7f8a6401cd80 con 0x56200b2dc280 2024-05-09T08:42:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.196+0000 7f8a4ffff640 1 -- 172.21.15.107:0/3534598793 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8a64031050 con 0x56200b2dc280 2024-05-09T08:42:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.196+0000 7f8a7982b740 10 client.18162 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.196+0000 7f8a7982b740 1 -- 172.21.15.107:0/3534598793 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56200af150f0 con 0x56200b2dc280 2024-05-09T08:42:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.197+0000 7f8a4ffff640 1 -- 172.21.15.107:0/3534598793 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a64011db0 con 0x56200b2dc280 2024-05-09T08:42:06.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.197+0000 7f8a4ffff640 1 -- 172.21.15.107:0/3534598793 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8a64011560 con 0x56200b2dc280 2024-05-09T08:42:06.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.197+0000 7f8a7982b740 10 client.18162 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:06.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.197+0000 7f8a7982b740 10 client.18162 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:06.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.197+0000 7f8a7982b740 20 client.18162 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:06.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.197+0000 7f8a7982b740 1 --2- 172.21.15.107:0/3534598793 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56200b2f7720 0x56200b2f9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.197+0000 7f8a7982b740 4 client.18162 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:06.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.197+0000 7f8a7982b740 1 -- 172.21.15.107:0/3534598793 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56200b24f430 con 0x56200b2f7720 2024-05-09T08:42:06.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.197+0000 7f8a6ffff640 1 --2- 172.21.15.107:0/3534598793 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56200b2f7720 0x56200b2f9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.198+0000 7f8a6ffff640 1 --2- 172.21.15.107:0/3534598793 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56200b2f7720 0x56200b2f9b90 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cc1798b91208817 server_cookie=fb7a955e6be25ce8 in_seq=0 out_seq=0 2024-05-09T08:42:06.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.198+0000 7f8a4ffff640 10 client.18162 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:06.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.198+0000 7f8a4ffff640 1 -- 172.21.15.107:0/3534598793 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56200b24f430 con 0x56200b2f7720 2024-05-09T08:42:06.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.198+0000 7f8a4ffff640 10 client.18162 handle_command_reply: tid=0 2024-05-09T08:42:06.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.198+0000 7f8a7982b740 1 client.18162 shutdown 2024-05-09T08:42:06.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.199+0000 7f8a7982b740 1 -- 172.21.15.107:0/3534598793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dc280 msgr2=0x56200b2e41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.199+0000 7f8a7982b740 1 --2- 172.21.15.107:0/3534598793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dc280 0x56200b2e41f0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400c010 tx=0x7f8a6401c480 comp rx=0 tx=0).stop 2024-05-09T08:42:06.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.199+0000 7f8a7982b740 1 -- 172.21.15.107:0/3534598793 shutdown_connections 2024-05-09T08:42:06.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.199+0000 7f8a7982b740 1 --2- 172.21.15.107:0/3534598793 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56200b2f7720 0x56200b2f9b90 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.199+0000 7f8a7982b740 1 --2- 172.21.15.107:0/3534598793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 0x56200b2e4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.199+0000 7f8a7982b740 1 --2- 172.21.15.107:0/3534598793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dcba0 0x56200b2e4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.199+0000 7f8a7982b740 1 --2- 172.21.15.107:0/3534598793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dc280 0x56200b2e41f0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.199+0000 7f8a7982b740 1 -- 172.21.15.107:0/3534598793 >> 172.21.15.107:0/3534598793 conn(0x56200b2d85a0 msgr2=0x56200b2de3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.199+0000 7f8a4ffff640 0 client.18162 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:06.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.199+0000 7f8a7982b740 1 -- 172.21.15.107:0/3534598793 shutdown_connections 2024-05-09T08:42:06.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.199+0000 7f8a7982b740 1 -- 172.21.15.107:0/3534598793 wait complete. 2024-05-09T08:42:06.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.199+0000 7f8a7982b740 20 client.18162 trim_cache size 0 max 16384 2024-05-09T08:42:06.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.218+0000 7f8a7982b740 1 Processor -- start 2024-05-09T08:42:06.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.218+0000 7f8a7982b740 1 -- start start 2024-05-09T08:42:06.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a7982b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dc280 0x56200b2f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a7982b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dcba0 0x56200b2d3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a7982b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 0x56200b2e3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a7982b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a701b37f0 con 0x56200b2dcba0 2024-05-09T08:42:06.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a7982b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a701b10c0 con 0x56200b2e7270 2024-05-09T08:42:06.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a7982b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8a701aef30 con 0x56200b2dc280 2024-05-09T08:42:06.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a6ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dcba0 0x56200b2d3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a6ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dcba0 0x56200b2d3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41498/0 (socket says 172.21.15.107:41498) 2024-05-09T08:42:06.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a6ffff640 1 -- 172.21.15.107:0/3324572242 learned_addr learned my addr 172.21.15.107:0/3324572242 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:06.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a750b8640 1 --2- 172.21.15.107:0/3324572242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 0x56200b2e3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a748b7640 1 --2- 172.21.15.107:0/3324572242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dc280 0x56200b2f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a6ffff640 1 -- 172.21.15.107:0/3324572242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dc280 msgr2=0x56200b2f2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a6ffff640 1 --2- 172.21.15.107:0/3324572242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dc280 0x56200b2f2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a6ffff640 1 -- 172.21.15.107:0/3324572242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 msgr2=0x56200b2e3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a6ffff640 1 --2- 172.21.15.107:0/3324572242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 0x56200b2e3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a6ffff640 1 -- 172.21.15.107:0/3324572242 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56200b2d2bf0 con 0x56200b2dcba0 2024-05-09T08:42:06.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a6ffff640 1 --2- 172.21.15.107:0/3324572242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dcba0 0x56200b2d3670 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c0cd070 tx=0x7f8a5c013330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179c4371c9af2ed3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.219+0000 7f8a5a7fc640 1 -- 172.21.15.107:0/3324572242 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c09b740 con 0x56200b2dcba0 2024-05-09T08:42:06.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.220+0000 7f8a5a7fc640 1 -- 172.21.15.107:0/3324572242 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a5c0610b0 con 0x56200b2dcba0 2024-05-09T08:42:06.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.220+0000 7f8a5a7fc640 1 -- 172.21.15.107:0/3324572242 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c013190 con 0x56200b2dcba0 2024-05-09T08:42:06.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.220+0000 7f8a7982b740 1 -- 172.21.15.107:0/3324572242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dcba0 msgr2=0x56200b2d3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.220+0000 7f8a7982b740 1 --2- 172.21.15.107:0/3324572242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dcba0 0x56200b2d3670 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c0cd070 tx=0x7f8a5c013330 comp rx=0 tx=0).stop 2024-05-09T08:42:06.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.220+0000 7f8a7982b740 1 -- 172.21.15.107:0/3324572242 shutdown_connections 2024-05-09T08:42:06.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.220+0000 7f8a7982b740 1 --2- 172.21.15.107:0/3324572242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 0x56200b2e3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.220+0000 7f8a7982b740 1 --2- 172.21.15.107:0/3324572242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dcba0 0x56200b2d3670 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.220+0000 7f8a7982b740 1 --2- 172.21.15.107:0/3324572242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dc280 0x56200b2f2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.220+0000 7f8a7982b740 1 -- 172.21.15.107:0/3324572242 >> 172.21.15.107:0/3324572242 conn(0x56200b2d85a0 msgr2=0x56200b2d6f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.220+0000 7f8a7982b740 1 -- 172.21.15.107:0/3324572242 shutdown_connections 2024-05-09T08:42:06.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.220+0000 7f8a7982b740 1 -- 172.21.15.107:0/3324572242 wait complete. 2024-05-09T08:42:06.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a7982b740 1 -- start start 2024-05-09T08:42:06.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a7982b740 1 Processor -- start 2024-05-09T08:42:06.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a7982b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dc280 0x56200b2ef430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a7982b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dcba0 0x56200b2ef970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a7982b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 0x56200b2e0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a7982b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a701af580 con 0x56200b2dc280 2024-05-09T08:42:06.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a7982b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a701b09f0 con 0x56200b2e7270 2024-05-09T08:42:06.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a7982b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8a701afc50 con 0x56200b2dcba0 2024-05-09T08:42:06.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a7982b740 10 client.0 _ll_register_callbacks cb 0x7f8a6400bf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:06.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a748b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dc280 0x56200b2ef430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a748b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dc280 0x56200b2ef430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41504/0 (socket says 172.21.15.107:41504) 2024-05-09T08:42:06.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a748b7640 1 -- 172.21.15.107:0/117365760 learned_addr learned my addr 172.21.15.107:0/117365760 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:06.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a6ffff640 1 --2- 172.21.15.107:0/117365760 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dcba0 0x56200b2ef970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a750b8640 1 --2- 172.21.15.107:0/117365760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 0x56200b2e0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a748b7640 1 -- 172.21.15.107:0/117365760 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dcba0 msgr2=0x56200b2ef970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a748b7640 1 --2- 172.21.15.107:0/117365760 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dcba0 0x56200b2ef970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a748b7640 1 -- 172.21.15.107:0/117365760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 msgr2=0x56200b2e0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a748b7640 1 --2- 172.21.15.107:0/117365760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 0x56200b2e0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.221+0000 7f8a748b7640 1 -- 172.21.15.107:0/117365760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56200b2e0b40 con 0x56200b2dc280 2024-05-09T08:42:06.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.222+0000 7f8a748b7640 1 --2- 172.21.15.107:0/117365760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dc280 0x56200b2ef430 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400b3d0 tx=0x7f8a6401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5381d2cf9292cac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.222+0000 7f8a4ffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:06.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.222+0000 7f8a4ffff640 1 -- 172.21.15.107:0/117365760 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6402a070 con 0x56200b2dc280 2024-05-09T08:42:06.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.222+0000 7f8a4ffff640 1 -- 172.21.15.107:0/117365760 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9555+0+0 (secure 0 0 0) 0x7f8a64011b50 con 0x56200b2dc280 2024-05-09T08:42:06.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.222+0000 7f8a7982b740 1 -- 172.21.15.107:0/117365760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a5c0610b0 con 0x56200b2dc280 2024-05-09T08:42:06.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.222+0000 7f8a4ffff640 1 -- 172.21.15.107:0/117365760 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8a64011770 con 0x56200b2dc280 2024-05-09T08:42:06.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.222+0000 7f8a7982b740 10 client.18168 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:06.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.222+0000 7f8a7982b740 1 -- 172.21.15.107:0/117365760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56200b35d5e0 con 0x56200b2dc280 2024-05-09T08:42:06.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.222+0000 7f8a4ffff640 1 -- 172.21.15.107:0/117365760 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a5c0610b0 con 0x56200b2dc280 2024-05-09T08:42:06.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.222+0000 7f8a4ffff640 1 -- 172.21.15.107:0/117365760 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8a64003c30 con 0x56200b2dc280 2024-05-09T08:42:06.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.222+0000 7f8a7982b740 10 client.18168 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:06.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.222+0000 7f8a7982b740 10 client.18168 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:06.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.222+0000 7f8a7982b740 20 client.18168 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:06.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.222+0000 7f8a7982b740 1 --2- 172.21.15.107:0/117365760 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56200b35dd50 0x56200b360170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.222+0000 7f8a7982b740 4 client.18168 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:42:06.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.222+0000 7f8a7982b740 1 -- 172.21.15.107:0/117365760 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56200b249780 con 0x56200b35dd50 2024-05-09T08:42:06.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.223+0000 7f8a6ffff640 1 --2- 172.21.15.107:0/117365760 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56200b35dd50 0x56200b360170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.223+0000 7f8a6ffff640 1 --2- 172.21.15.107:0/117365760 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56200b35dd50 0x56200b360170 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=635ea4e027bbf6ec server_cookie=fc89ea8b92e680c in_seq=0 out_seq=0 2024-05-09T08:42:06.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.223+0000 7f8a4ffff640 10 client.18168 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:06.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.223+0000 7f8a4ffff640 1 -- 172.21.15.107:0/117365760 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56200b249780 con 0x56200b35dd50 2024-05-09T08:42:06.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.223+0000 7f8a4ffff640 10 client.18168 handle_command_reply: tid=0 2024-05-09T08:42:06.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.223+0000 7f8a7982b740 1 client.18168 shutdown 2024-05-09T08:42:06.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.223+0000 7f8a7982b740 1 -- 172.21.15.107:0/117365760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dc280 msgr2=0x56200b2ef430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.223+0000 7f8a7982b740 1 --2- 172.21.15.107:0/117365760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dc280 0x56200b2ef430 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400b3d0 tx=0x7f8a6401c9c0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.223+0000 7f8a7982b740 1 -- 172.21.15.107:0/117365760 shutdown_connections 2024-05-09T08:42:06.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.223+0000 7f8a7982b740 1 --2- 172.21.15.107:0/117365760 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56200b35dd50 0x56200b360170 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.223+0000 7f8a7982b740 1 --2- 172.21.15.107:0/117365760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200b2e7270 0x56200b2e0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.223+0000 7f8a7982b740 1 --2- 172.21.15.107:0/117365760 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56200b2dcba0 0x56200b2ef970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.223+0000 7f8a7982b740 1 --2- 172.21.15.107:0/117365760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56200b2dc280 0x56200b2ef430 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.223+0000 7f8a7982b740 1 -- 172.21.15.107:0/117365760 >> 172.21.15.107:0/117365760 conn(0x56200b2d85a0 msgr2=0x56200b2d6f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.224+0000 7f8a4ffff640 0 client.18168 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:06.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.224+0000 7f8a7982b740 1 -- 172.21.15.107:0/117365760 shutdown_connections 2024-05-09T08:42:06.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.224+0000 7f8a7982b740 1 -- 172.21.15.107:0/117365760 wait complete. 2024-05-09T08:42:06.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.224+0000 7f8a7982b740 20 client.18168 trim_cache size 0 max 16384 2024-05-09T08:42:06.231 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:42:06.231 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:42:06.231 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:42:06.231 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:42:06.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.224+0000 7f8a76b1b640 1 -- 172.21.15.107:0/3355500084 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a600a2d00 msgr2=0x7f8a600a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.224+0000 7f8a76b1b640 1 --2- 172.21.15.107:0/3355500084 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a600a2d00 0x7f8a600a51c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8a68009430 tx=0x7f8a680095c0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.224+0000 7f8a76b1b640 1 -- 172.21.15.107:0/3355500084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a70106be0 msgr2=0x7f8a7019a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.224+0000 7f8a76b1b640 1 --2- 172.21.15.107:0/3355500084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a70106be0 0x7f8a7019a790 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c00e9c0 tx=0x7f8a5c00ee80 comp rx=0 tx=0).stop 2024-05-09T08:42:06.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.224+0000 7f8a76b1b640 1 -- 172.21.15.107:0/3355500084 shutdown_connections 2024-05-09T08:42:06.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.224+0000 7f8a76b1b640 1 --2- 172.21.15.107:0/3355500084 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a600a2d00 0x7f8a600a51c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.224+0000 7f8a76b1b640 1 --2- 172.21.15.107:0/3355500084 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a7019eb10 0x7f8a7019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.224+0000 7f8a76b1b640 1 --2- 172.21.15.107:0/3355500084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a70106be0 0x7f8a7019a790 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.224+0000 7f8a76b1b640 1 --2- 172.21.15.107:0/3355500084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a700fffd0 0x7f8a7019a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.224+0000 7f8a76b1b640 1 -- 172.21.15.107:0/3355500084 >> 172.21.15.107:0/3355500084 conn(0x7f8a70076cd0 msgr2=0x7f8a70102550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.224+0000 7f8a76b1b640 1 -- 172.21.15.107:0/3355500084 shutdown_connections 2024-05-09T08:42:06.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.225+0000 7f8a76b1b640 1 -- 172.21.15.107:0/3355500084 wait complete. 2024-05-09T08:42:06.235 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:42:06.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:42:06.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.346+0000 7f44b3010640 1 -- 172.21.15.107:0/799382521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44ac106720 msgr2=0x7f44ac106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.346+0000 7f44b3010640 1 --2- 172.21.15.107:0/799382521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44ac106720 0x7f44ac106b00 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f44a4009cd0 tx=0x7f44a402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.346+0000 7f44b3010640 1 -- 172.21.15.107:0/799382521 shutdown_connections 2024-05-09T08:42:06.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.346+0000 7f44b3010640 1 --2- 172.21.15.107:0/799382521 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44ac1010d0 0x7f44ac10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.346+0000 7f44b3010640 1 --2- 172.21.15.107:0/799382521 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44ac100710 0x7f44ac100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.346+0000 7f44b3010640 1 --2- 172.21.15.107:0/799382521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44ac106720 0x7f44ac106b00 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.346+0000 7f44b3010640 1 -- 172.21.15.107:0/799382521 >> 172.21.15.107:0/799382521 conn(0x7f44ac077830 msgr2=0x7f44ac077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.346+0000 7f44b3010640 1 -- 172.21.15.107:0/799382521 shutdown_connections 2024-05-09T08:42:06.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.346+0000 7f44b3010640 1 -- 172.21.15.107:0/799382521 wait complete. 2024-05-09T08:42:06.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.347+0000 7f44b3010640 1 Processor -- start 2024-05-09T08:42:06.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.347+0000 7f44b3010640 1 -- start start 2024-05-09T08:42:06.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.347+0000 7f44b3010640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44ac100710 0x7f44ac198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.347+0000 7f44b3010640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44ac1010d0 0x7f44ac198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.347+0000 7f44b3010640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44ac106720 0x7f44ac19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.347+0000 7f44b3010640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac1109a0 con 0x7f44ac1010d0 2024-05-09T08:42:06.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.347+0000 7f44b3010640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac110820 con 0x7f44ac106720 2024-05-09T08:42:06.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.347+0000 7f44b3010640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac110b20 con 0x7f44ac100710 2024-05-09T08:42:06.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.347+0000 7f44abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44ac1010d0 0x7f44ac198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.347+0000 7f44abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44ac1010d0 0x7f44ac198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41528/0 (socket says 172.21.15.107:41528) 2024-05-09T08:42:06.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.347+0000 7f44abfff640 1 -- 172.21.15.107:0/3921460355 learned_addr learned my addr 172.21.15.107:0/3921460355 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:06.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.347+0000 7f44b15ad640 1 --2- 172.21.15.107:0/3921460355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44ac106720 0x7f44ac19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.347+0000 7f44b0dac640 1 --2- 172.21.15.107:0/3921460355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44ac100710 0x7f44ac198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.348+0000 7f44abfff640 1 -- 172.21.15.107:0/3921460355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44ac100710 msgr2=0x7f44ac198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.348+0000 7f44abfff640 1 --2- 172.21.15.107:0/3921460355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44ac100710 0x7f44ac198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.348+0000 7f44abfff640 1 -- 172.21.15.107:0/3921460355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44ac106720 msgr2=0x7f44ac19c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.348+0000 7f44abfff640 1 --2- 172.21.15.107:0/3921460355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44ac106720 0x7f44ac19c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.348+0000 7f44abfff640 1 -- 172.21.15.107:0/3921460355 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44ac19d020 con 0x7f44ac1010d0 2024-05-09T08:42:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.348+0000 7f44b15ad640 1 --2- 172.21.15.107:0/3921460355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44ac106720 0x7f44ac19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:42:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.348+0000 7f44abfff640 1 --2- 172.21.15.107:0/3921460355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44ac1010d0 0x7f44ac198570 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f449c00d950 tx=0x7f449c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.348+0000 7f44a9ffb640 1 -- 172.21.15.107:0/3921460355 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449c017030 con 0x7f44ac1010d0 2024-05-09T08:42:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.348+0000 7f44a9ffb640 1 -- 172.21.15.107:0/3921460355 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f449c004020 con 0x7f44ac1010d0 2024-05-09T08:42:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.348+0000 7f44b3010640 1 -- 172.21.15.107:0/3921460355 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44ac19d310 con 0x7f44ac1010d0 2024-05-09T08:42:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.348+0000 7f44a9ffb640 1 -- 172.21.15.107:0/3921460355 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449c0047a0 con 0x7f44ac1010d0 2024-05-09T08:42:06.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.348+0000 7f44b3010640 1 -- 172.21.15.107:0/3921460355 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44ac1a4ba0 con 0x7f44ac1010d0 2024-05-09T08:42:06.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.350+0000 7f44b3010640 1 -- 172.21.15.107:0/3921460355 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44ac068650 con 0x7f44ac1010d0 2024-05-09T08:42:06.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.351+0000 7f44a9ffb640 1 -- 172.21.15.107:0/3921460355 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f449c0041d0 con 0x7f44ac1010d0 2024-05-09T08:42:06.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.351+0000 7f44a9ffb640 1 --2- 172.21.15.107:0/3921460355 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f44980a2d80 0x7f44980a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.352+0000 7f44b0dac640 1 --2- 172.21.15.107:0/3921460355 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f44980a2d80 0x7f44980a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.352+0000 7f44a9ffb640 1 -- 172.21.15.107:0/3921460355 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9555+0+0 (secure 0 0 0) 0x7f449c0cf1a0 con 0x7f44ac1010d0 2024-05-09T08:42:06.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.352+0000 7f44b0dac640 1 --2- 172.21.15.107:0/3921460355 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f44980a2d80 0x7f44980a5240 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f44a4006010 tx=0x7f44a40099c0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.356+0000 7f44a9ffb640 1 -- 172.21.15.107:0/3921460355 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f449c09b9b0 con 0x7f44ac1010d0 2024-05-09T08:42:06.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:06 smithi110 ceph-mon[33622]: pgmap v270: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 30 MiB/s wr, 44 op/s 2024-05-09T08:42:06.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:06 smithi107 ceph-mon[30122]: pgmap v270: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 30 MiB/s wr, 44 op/s 2024-05-09T08:42:06.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:06 smithi157 ceph-mon[33614]: pgmap v270: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 30 MiB/s wr, 44 op/s 2024-05-09T08:42:06.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.602+0000 7f44b3010640 1 -- 172.21.15.107:0/3921460355 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44ac109cf0 con 0x7f44ac1010d0 2024-05-09T08:42:06.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.603+0000 7f44a9ffb640 1 -- 172.21.15.107:0/3921460355 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11887 (secure 0 0 0) 0x7f449c095e20 con 0x7f44ac1010d0 2024-05-09T08:42:06.603 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:42:06.603 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:37:10.930317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:42:06.604 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2024-05-09T08:42:06.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.606+0000 7f44b3010640 1 -- 172.21.15.107:0/3921460355 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f44980a2d80 msgr2=0x7f44980a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.606+0000 7f44b3010640 1 --2- 172.21.15.107:0/3921460355 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f44980a2d80 0x7f44980a5240 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f44a4006010 tx=0x7f44a40099c0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.606+0000 7f44b3010640 1 -- 172.21.15.107:0/3921460355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44ac1010d0 msgr2=0x7f44ac198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.606+0000 7f44b3010640 1 --2- 172.21.15.107:0/3921460355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44ac1010d0 0x7f44ac198570 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f449c00d950 tx=0x7f449c00de10 comp rx=0 tx=0).stop 2024-05-09T08:42:06.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.606+0000 7f44b3010640 1 -- 172.21.15.107:0/3921460355 shutdown_connections 2024-05-09T08:42:06.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.606+0000 7f44b3010640 1 --2- 172.21.15.107:0/3921460355 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f44980a2d80 0x7f44980a5240 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.606+0000 7f44b3010640 1 --2- 172.21.15.107:0/3921460355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44ac106720 0x7f44ac19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.606+0000 7f44b3010640 1 --2- 172.21.15.107:0/3921460355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44ac1010d0 0x7f44ac198570 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.606+0000 7f44b3010640 1 --2- 172.21.15.107:0/3921460355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44ac100710 0x7f44ac198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.606+0000 7f44b3010640 1 -- 172.21.15.107:0/3921460355 >> 172.21.15.107:0/3921460355 conn(0x7f44ac077830 msgr2=0x7f44ac068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.606+0000 7f44b3010640 1 -- 172.21.15.107:0/3921460355 shutdown_connections 2024-05-09T08:42:06.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.606+0000 7f44b3010640 1 -- 172.21.15.107:0/3921460355 wait complete. 2024-05-09T08:42:06.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:42:06.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.733+0000 7f76bb8bd640 1 -- 172.21.15.107:0/900592524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b410bdf0 msgr2=0x7f76b410e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.733+0000 7f76bb8bd640 1 --2- 172.21.15.107:0/900592524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b410bdf0 0x7f76b410e290 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f76ac009ca0 tx=0x7f76ac02a9e0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.733+0000 7f76bb8bd640 1 -- 172.21.15.107:0/900592524 shutdown_connections 2024-05-09T08:42:06.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.733+0000 7f76bb8bd640 1 --2- 172.21.15.107:0/900592524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b410bdf0 0x7f76b410e290 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.733+0000 7f76bb8bd640 1 --2- 172.21.15.107:0/900592524 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76b41033d0 0x7f76b410b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.733+0000 7f76bb8bd640 1 --2- 172.21.15.107:0/900592524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b4102ab0 0x7f76b4102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.733+0000 7f76bb8bd640 1 -- 172.21.15.107:0/900592524 >> 172.21.15.107:0/900592524 conn(0x7f76b4077830 msgr2=0x7f76b4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.733+0000 7f76bb8bd640 1 -- 172.21.15.107:0/900592524 shutdown_connections 2024-05-09T08:42:06.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.733+0000 7f76bb8bd640 1 -- 172.21.15.107:0/900592524 wait complete. 2024-05-09T08:42:06.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76bb8bd640 1 Processor -- start 2024-05-09T08:42:06.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76bb8bd640 1 -- start start 2024-05-09T08:42:06.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76bb8bd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b4102ab0 0x7f76b4198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76bb8bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b41033d0 0x7f76b4198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76bb8bd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76b410bdf0 0x7f76b419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76bb8bd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f76b4111aa0 con 0x7f76b41033d0 2024-05-09T08:42:06.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76bb8bd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f76b4111920 con 0x7f76b4102ab0 2024-05-09T08:42:06.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76bb8bd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f76b4111c20 con 0x7f76b410bdf0 2024-05-09T08:42:06.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76b8e58640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b41033d0 0x7f76b4198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76b8e58640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b41033d0 0x7f76b4198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41548/0 (socket says 172.21.15.107:41548) 2024-05-09T08:42:06.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76b8e58640 1 -- 172.21.15.107:0/2104530184 learned_addr learned my addr 172.21.15.107:0/2104530184 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:06.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76b9659640 1 --2- 172.21.15.107:0/2104530184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b4102ab0 0x7f76b4198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/2104530184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76b410bdf0 0x7f76b419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76b8e58640 1 -- 172.21.15.107:0/2104530184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76b410bdf0 msgr2=0x7f76b419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76b8e58640 1 --2- 172.21.15.107:0/2104530184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76b410bdf0 0x7f76b419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76b8e58640 1 -- 172.21.15.107:0/2104530184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b4102ab0 msgr2=0x7f76b4198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76b8e58640 1 --2- 172.21.15.107:0/2104530184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b4102ab0 0x7f76b4198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76b8e58640 1 -- 172.21.15.107:0/2104530184 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76b419cfa0 con 0x7f76b41033d0 2024-05-09T08:42:06.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/2104530184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76b410bdf0 0x7f76b419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:42:06.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.734+0000 7f76b9659640 1 --2- 172.21.15.107:0/2104530184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b4102ab0 0x7f76b4198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:42:06.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.735+0000 7f76b8e58640 1 --2- 172.21.15.107:0/2104530184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b41033d0 0x7f76b4198540 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f76a400d770 tx=0x7f76a400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.735+0000 7f76aa7fc640 1 -- 172.21.15.107:0/2104530184 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a4017070 con 0x7f76b41033d0 2024-05-09T08:42:06.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.735+0000 7f76aa7fc640 1 -- 172.21.15.107:0/2104530184 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76a40054d0 con 0x7f76b41033d0 2024-05-09T08:42:06.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.735+0000 7f76bb8bd640 1 -- 172.21.15.107:0/2104530184 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76b419d290 con 0x7f76b41033d0 2024-05-09T08:42:06.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.735+0000 7f76aa7fc640 1 -- 172.21.15.107:0/2104530184 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a4004cf0 con 0x7f76b41033d0 2024-05-09T08:42:06.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.735+0000 7f76bb8bd640 1 -- 172.21.15.107:0/2104530184 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76b41a4b20 con 0x7f76b41033d0 2024-05-09T08:42:06.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.738+0000 7f76aa7fc640 1 -- 172.21.15.107:0/2104530184 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f76a4004e90 con 0x7f76b41033d0 2024-05-09T08:42:06.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.738+0000 7f76be5d2740 1 Processor -- start 2024-05-09T08:42:06.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.738+0000 7f76be5d2740 1 -- start start 2024-05-09T08:42:06.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76be5d2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012c330 0x55c97012c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76be5d2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c97012cc50 0x55c970136de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76aa7fc640 1 --2- 172.21.15.107:0/2104530184 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f76a00a2d80 0x7f76a00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76be5d2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c970137320 0x55c970139710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76be5d2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c970091c40 con 0x55c970137320 2024-05-09T08:42:06.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76be5d2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c970039580 con 0x55c97012c330 2024-05-09T08:42:06.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76be5d2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c97008dc00 con 0x55c97012cc50 2024-05-09T08:42:06.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76b9e5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c970137320 0x55c970139710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76b8e58640 1 --2- 172.21.15.107:0/2104530184 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f76a00a2d80 0x7f76a00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76b9e5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c970137320 0x55c970139710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41562/0 (socket says 172.21.15.107:41562) 2024-05-09T08:42:06.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76b9e5a640 1 -- 172.21.15.107:0/3606268134 learned_addr learned my addr 172.21.15.107:0/3606268134 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:06.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76b9659640 1 --2- 172.21.15.107:0/3606268134 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c97012cc50 0x55c970136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/3606268134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012c330 0x55c97012c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76aa7fc640 1 -- 172.21.15.107:0/2104530184 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9555+0+0 (secure 0 0 0) 0x7f76a40cf260 con 0x7f76b41033d0 2024-05-09T08:42:06.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76b9e5a640 1 -- 172.21.15.107:0/3606268134 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c97012cc50 msgr2=0x55c970136de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/3606268134 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c97012cc50 0x55c970136de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76b9e5a640 1 -- 172.21.15.107:0/3606268134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012c330 msgr2=0x55c97012c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/3606268134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012c330 0x55c97012c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76b9e5a640 1 -- 172.21.15.107:0/3606268134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c96fda53f0 con 0x55c970137320 2024-05-09T08:42:06.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76b8e58640 1 --2- 172.21.15.107:0/2104530184 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f76a00a2d80 0x7f76a00a5240 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f76a4021000 tx=0x7f76a4005bb0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.739+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/3606268134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c970137320 0x55c970139710 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f76ac004af0 tx=0x7f76ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea3cd370d5cc791 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.740+0000 7f769f7fe640 1 -- 172.21.15.107:0/3606268134 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac04d070 con 0x55c970137320 2024-05-09T08:42:06.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.740+0000 7f769f7fe640 1 -- 172.21.15.107:0/3606268134 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76ac02f640 con 0x55c970137320 2024-05-09T08:42:06.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.740+0000 7f769f7fe640 1 -- 172.21.15.107:0/3606268134 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac033040 con 0x55c970137320 2024-05-09T08:42:06.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.740+0000 7f76be5d2740 1 -- 172.21.15.107:0/3606268134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c970137320 msgr2=0x55c970139710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.740+0000 7f76be5d2740 1 --2- 172.21.15.107:0/3606268134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c970137320 0x55c970139710 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f76ac004af0 tx=0x7f76ac004a40 comp rx=0 tx=0).stop 2024-05-09T08:42:06.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.740+0000 7f76be5d2740 1 -- 172.21.15.107:0/3606268134 shutdown_connections 2024-05-09T08:42:06.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.740+0000 7f76be5d2740 1 --2- 172.21.15.107:0/3606268134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c970137320 0x55c970139710 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.740+0000 7f76be5d2740 1 --2- 172.21.15.107:0/3606268134 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c97012cc50 0x55c970136de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.740+0000 7f76be5d2740 1 --2- 172.21.15.107:0/3606268134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012c330 0x55c97012c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.740+0000 7f76be5d2740 1 -- 172.21.15.107:0/3606268134 >> 172.21.15.107:0/3606268134 conn(0x55c970128650 msgr2=0x55c970128a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.741+0000 7f76be5d2740 1 -- 172.21.15.107:0/3606268134 shutdown_connections 2024-05-09T08:42:06.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.741+0000 7f76be5d2740 1 -- 172.21.15.107:0/3606268134 wait complete. 2024-05-09T08:42:06.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.741+0000 7f76be5d2740 1 -- start start 2024-05-09T08:42:06.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.741+0000 7f76be5d2740 1 Processor -- start 2024-05-09T08:42:06.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.741+0000 7f76be5d2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012c330 0x55c9701342f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.741+0000 7f76be5d2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c97012cc50 0x55c970134830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.741+0000 7f76be5d2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c970137320 0x55c970134d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.741+0000 7f76be5d2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c97013a0b0 con 0x55c97012cc50 2024-05-09T08:42:06.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.741+0000 7f76be5d2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c97004cc40 con 0x55c97012c330 2024-05-09T08:42:06.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.741+0000 7f76be5d2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c97013a230 con 0x55c970137320 2024-05-09T08:42:06.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.741+0000 7f76be5d2740 10 client.0 _ll_register_callbacks cb 0x55c96fda4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:06.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.741+0000 7f76b9659640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c97012cc50 0x55c970134830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.741+0000 7f76b9659640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c97012cc50 0x55c970134830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41566/0 (socket says 172.21.15.107:41566) 2024-05-09T08:42:06.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.741+0000 7f76b9659640 1 -- 172.21.15.107:0/98645301 learned_addr learned my addr 172.21.15.107:0/98645301 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:06.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.742+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/98645301 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012c330 0x55c9701342f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.742+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/98645301 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c970137320 0x55c970134d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.742+0000 7f76b9659640 1 -- 172.21.15.107:0/98645301 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c970137320 msgr2=0x55c970134d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.742+0000 7f76b9659640 1 --2- 172.21.15.107:0/98645301 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c970137320 0x55c970134d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.742+0000 7f76b9659640 1 -- 172.21.15.107:0/98645301 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012c330 msgr2=0x55c9701342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.742+0000 7f76b9659640 1 --2- 172.21.15.107:0/98645301 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012c330 0x55c9701342f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.742+0000 7f76b9659640 1 -- 172.21.15.107:0/98645301 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c96fe6f7d0 con 0x55c97012cc50 2024-05-09T08:42:06.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.742+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/98645301 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c970137320 0x55c970134d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:42:06.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.742+0000 7f76b9659640 1 --2- 172.21.15.107:0/98645301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c97012cc50 0x55c970134830 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f76b000bf00 tx=0x7f76b0012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2eacad91daf1690 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.742+0000 7f769cff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:06.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.742+0000 7f769cff9640 1 -- 172.21.15.107:0/98645301 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0010960 con 0x55c97012cc50 2024-05-09T08:42:06.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.742+0000 7f76be5d2740 1 -- 172.21.15.107:0/98645301 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76ac02f640 con 0x55c97012cc50 2024-05-09T08:42:06.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.742+0000 7f769cff9640 1 -- 172.21.15.107:0/98645301 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9555+0+0 (secure 0 0 0) 0x7f76b0010b00 con 0x55c97012cc50 2024-05-09T08:42:06.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.742+0000 7f769cff9640 1 -- 172.21.15.107:0/98645301 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f76b0010d40 con 0x55c97012cc50 2024-05-09T08:42:06.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.742+0000 7f76be5d2740 10 client.18186 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:06.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.742+0000 7f76be5d2740 1 -- 172.21.15.107:0/98645301 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c9701463d0 con 0x55c97012cc50 2024-05-09T08:42:06.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.743+0000 7f769cff9640 1 -- 172.21.15.107:0/98645301 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76ac02f640 con 0x55c97012cc50 2024-05-09T08:42:06.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.743+0000 7f769cff9640 1 -- 172.21.15.107:0/98645301 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f76b0019070 con 0x55c97012cc50 2024-05-09T08:42:06.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.743+0000 7f76be5d2740 10 client.18186 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:06.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.743+0000 7f76be5d2740 10 client.18186 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:42:06.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.743+0000 7f76be5d2740 20 client.18186 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:06.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.743+0000 7f76be5d2740 1 --2- 172.21.15.107:0/98645301 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55c970147890 0x55c970149d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.743+0000 7f76be5d2740 4 client.18186 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:06.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.743+0000 7f76be5d2740 1 -- 172.21.15.107:0/98645301 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c97009f370 con 0x55c970147890 2024-05-09T08:42:06.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.743+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/98645301 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55c970147890 0x55c970149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.743+0000 7f769cff9640 10 client.18186 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:42:06.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.743+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/98645301 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55c970147890 0x55c970149d30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c47699e2fb0afea1 server_cookie=d794d0e477ecbd2c in_seq=0 out_seq=0 2024-05-09T08:42:06.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.744+0000 7f769cff9640 1 -- 172.21.15.107:0/98645301 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c97009f370 con 0x55c970147890 2024-05-09T08:42:06.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.744+0000 7f769cff9640 10 client.18186 handle_command_reply: tid=0 2024-05-09T08:42:06.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.744+0000 7f76be5d2740 1 client.18186 shutdown 2024-05-09T08:42:06.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.744+0000 7f76be5d2740 1 -- 172.21.15.107:0/98645301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c97012cc50 msgr2=0x55c970134830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.744+0000 7f76be5d2740 1 --2- 172.21.15.107:0/98645301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c97012cc50 0x55c970134830 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f76b000bf00 tx=0x7f76b0012820 comp rx=0 tx=0).stop 2024-05-09T08:42:06.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.744+0000 7f76be5d2740 1 -- 172.21.15.107:0/98645301 shutdown_connections 2024-05-09T08:42:06.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.744+0000 7f76be5d2740 1 --2- 172.21.15.107:0/98645301 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55c970147890 0x55c970149d30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.744+0000 7f76be5d2740 1 --2- 172.21.15.107:0/98645301 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c970137320 0x55c970134d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.744+0000 7f76be5d2740 1 --2- 172.21.15.107:0/98645301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c97012cc50 0x55c970134830 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.744+0000 7f769cff9640 0 client.18186 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:42:06.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.744+0000 7f76be5d2740 1 --2- 172.21.15.107:0/98645301 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012c330 0x55c9701342f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.744+0000 7f76be5d2740 1 -- 172.21.15.107:0/98645301 >> 172.21.15.107:0/98645301 conn(0x55c970128650 msgr2=0x55c97012e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.744+0000 7f76be5d2740 1 -- 172.21.15.107:0/98645301 shutdown_connections 2024-05-09T08:42:06.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.744+0000 7f76be5d2740 1 -- 172.21.15.107:0/98645301 wait complete. 2024-05-09T08:42:06.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.745+0000 7f76be5d2740 20 client.18186 trim_cache size 0 max 16384 2024-05-09T08:42:06.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76be5d2740 1 Processor -- start 2024-05-09T08:42:06.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76be5d2740 1 -- start start 2024-05-09T08:42:06.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76be5d2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c97012c330 0x55c970136a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76be5d2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012cc50 0x55c970142fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76be5d2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c970137320 0x55c970127630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76be5d2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f76b41af330 con 0x55c970137320 2024-05-09T08:42:06.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76be5d2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f76b41acc00 con 0x55c97012cc50 2024-05-09T08:42:06.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76be5d2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f76b41aaa70 con 0x55c97012c330 2024-05-09T08:42:06.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76b9e5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c970137320 0x55c970127630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76b9e5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c970137320 0x55c970127630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41578/0 (socket says 172.21.15.107:41578) 2024-05-09T08:42:06.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76b9e5a640 1 -- 172.21.15.107:0/3656212809 learned_addr learned my addr 172.21.15.107:0/3656212809 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:06.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/3656212809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c97012c330 0x55c970136a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76b9659640 1 --2- 172.21.15.107:0/3656212809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012cc50 0x55c970142fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76b9e5a640 1 -- 172.21.15.107:0/3656212809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c97012c330 msgr2=0x55c970136a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/3656212809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c97012c330 0x55c970136a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76b9e5a640 1 -- 172.21.15.107:0/3656212809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012cc50 msgr2=0x55c970142fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/3656212809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012cc50 0x55c970142fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.761+0000 7f76b9e5a640 1 -- 172.21.15.107:0/3656212809 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9701241a0 con 0x55c970137320 2024-05-09T08:42:06.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.762+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/3656212809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c970137320 0x55c970127630 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x55c9701297e0 tx=0x7f76ac02f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf15373e1769c4c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.762+0000 7f769effd640 1 -- 172.21.15.107:0/3656212809 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac05b070 con 0x55c970137320 2024-05-09T08:42:06.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.762+0000 7f769effd640 1 -- 172.21.15.107:0/3656212809 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76ac04e040 con 0x55c970137320 2024-05-09T08:42:06.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.762+0000 7f769effd640 1 -- 172.21.15.107:0/3656212809 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac04d040 con 0x55c970137320 2024-05-09T08:42:06.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 -- 172.21.15.107:0/3656212809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c970137320 msgr2=0x55c970127630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 --2- 172.21.15.107:0/3656212809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c970137320 0x55c970127630 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x55c9701297e0 tx=0x7f76ac02f000 comp rx=0 tx=0).stop 2024-05-09T08:42:06.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 -- 172.21.15.107:0/3656212809 shutdown_connections 2024-05-09T08:42:06.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 --2- 172.21.15.107:0/3656212809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c970137320 0x55c970127630 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 --2- 172.21.15.107:0/3656212809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012cc50 0x55c970142fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 --2- 172.21.15.107:0/3656212809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c97012c330 0x55c970136a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 -- 172.21.15.107:0/3656212809 >> 172.21.15.107:0/3656212809 conn(0x55c970128650 msgr2=0x55c970126f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 -- 172.21.15.107:0/3656212809 shutdown_connections 2024-05-09T08:42:06.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 -- 172.21.15.107:0/3656212809 wait complete. 2024-05-09T08:42:06.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 -- start start 2024-05-09T08:42:06.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 Processor -- start 2024-05-09T08:42:06.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c97012c330 0x55c97013f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012cc50 0x55c97013b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c970137320 0x55c97013bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f76b41ab0c0 con 0x55c97012c330 2024-05-09T08:42:06.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f76b41ac530 con 0x55c97012cc50 2024-05-09T08:42:06.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f76b41ab790 con 0x55c970137320 2024-05-09T08:42:06.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.763+0000 7f76be5d2740 10 client.0 _ll_register_callbacks cb 0x7f76b000b800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:06.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f76b9e5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c97012c330 0x55c97013f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f76b9e5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c97012c330 0x55c97013f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41592/0 (socket says 172.21.15.107:41592) 2024-05-09T08:42:06.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f76b9e5a640 1 -- 172.21.15.107:0/1775258199 learned_addr learned my addr 172.21.15.107:0/1775258199 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:06.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f76b9659640 1 --2- 172.21.15.107:0/1775258199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012cc50 0x55c97013b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f76b9e5a640 1 -- 172.21.15.107:0/1775258199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c970137320 msgr2=0x55c97013bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/1775258199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c970137320 0x55c97013bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f76b9e5a640 1 -- 172.21.15.107:0/1775258199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012cc50 msgr2=0x55c97013b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/1775258199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012cc50 0x55c97013b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f76b9e5a640 1 -- 172.21.15.107:0/1775258199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c97013c310 con 0x55c97012c330 2024-05-09T08:42:06.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f76b9659640 1 --2- 172.21.15.107:0/1775258199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012cc50 0x55c97013b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:42:06.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f76b9e5a640 1 --2- 172.21.15.107:0/1775258199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c97012c330 0x55c97013f510 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f76ac0098c0 tx=0x7f76ac0093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5507aca1818fe900 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f769cff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:06.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f769cff9640 1 -- 172.21.15.107:0/1775258199 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac04d070 con 0x55c97012c330 2024-05-09T08:42:06.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f76be5d2740 1 -- 172.21.15.107:0/1775258199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76ac04e040 con 0x55c97012c330 2024-05-09T08:42:06.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f769cff9640 1 -- 172.21.15.107:0/1775258199 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9555+0+0 (secure 0 0 0) 0x7f76ac02fa70 con 0x55c97012c330 2024-05-09T08:42:06.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f769cff9640 1 -- 172.21.15.107:0/1775258199 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f76ac069050 con 0x55c97012c330 2024-05-09T08:42:06.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f76be5d2740 10 client.18192 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:06.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.764+0000 7f76be5d2740 1 -- 172.21.15.107:0/1775258199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c9701aeb70 con 0x55c97012c330 2024-05-09T08:42:06.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.765+0000 7f769cff9640 1 -- 172.21.15.107:0/1775258199 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76ac04e040 con 0x55c97012c330 2024-05-09T08:42:06.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.765+0000 7f769cff9640 1 -- 172.21.15.107:0/1775258199 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f76ac054370 con 0x55c97012c330 2024-05-09T08:42:06.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.765+0000 7f76be5d2740 10 client.18192 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:06.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.765+0000 7f76be5d2740 10 client.18192 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:42:06.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.765+0000 7f76be5d2740 20 client.18192 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:06.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.765+0000 7f76be5d2740 1 --2- 172.21.15.107:0/1775258199 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55c9701aed40 0x55c9701b1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.765+0000 7f76be5d2740 4 client.18192 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:42:06.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.765+0000 7f76be5d2740 1 -- 172.21.15.107:0/1775258199 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c9700996c0 con 0x55c9701aed40 2024-05-09T08:42:06.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.765+0000 7f76b9659640 1 --2- 172.21.15.107:0/1775258199 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55c9701aed40 0x55c9701b1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.765+0000 7f769cff9640 10 client.18192 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:42:06.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.765+0000 7f76b9659640 1 --2- 172.21.15.107:0/1775258199 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55c9701aed40 0x55c9701b1130 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6d63ded1b9e55582 server_cookie=3516e7da1146dd5 in_seq=0 out_seq=0 2024-05-09T08:42:06.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f769cff9640 1 -- 172.21.15.107:0/1775258199 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c9700996c0 con 0x55c9701aed40 2024-05-09T08:42:06.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f769cff9640 10 client.18192 handle_command_reply: tid=0 2024-05-09T08:42:06.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f76be5d2740 1 client.18192 shutdown 2024-05-09T08:42:06.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f76be5d2740 1 -- 172.21.15.107:0/1775258199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c97012c330 msgr2=0x55c97013f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f76be5d2740 1 --2- 172.21.15.107:0/1775258199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c97012c330 0x55c97013f510 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f76ac0098c0 tx=0x7f76ac0093f0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f76be5d2740 1 -- 172.21.15.107:0/1775258199 shutdown_connections 2024-05-09T08:42:06.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f76be5d2740 1 --2- 172.21.15.107:0/1775258199 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55c9701aed40 0x55c9701b1130 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f76be5d2740 1 --2- 172.21.15.107:0/1775258199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c970137320 0x55c97013bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f76be5d2740 1 --2- 172.21.15.107:0/1775258199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c97012cc50 0x55c97013b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f76be5d2740 1 --2- 172.21.15.107:0/1775258199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c97012c330 0x55c97013f510 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f769cff9640 0 client.18192 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:42:06.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f76be5d2740 1 -- 172.21.15.107:0/1775258199 >> 172.21.15.107:0/1775258199 conn(0x55c970128650 msgr2=0x55c9701277b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f76be5d2740 1 -- 172.21.15.107:0/1775258199 shutdown_connections 2024-05-09T08:42:06.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f76be5d2740 1 -- 172.21.15.107:0/1775258199 wait complete. 2024-05-09T08:42:06.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f76be5d2740 20 client.18192 trim_cache size 0 max 16384 2024-05-09T08:42:06.773 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:42:06.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f76bb8bd640 1 -- 172.21.15.107:0/2104530184 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f76a00a2d80 msgr2=0x7f76a00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f76bb8bd640 1 --2- 172.21.15.107:0/2104530184 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f76a00a2d80 0x7f76a00a5240 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f76a4021000 tx=0x7f76a4005bb0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f76bb8bd640 1 -- 172.21.15.107:0/2104530184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b41033d0 msgr2=0x7f76b4198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.766+0000 7f76bb8bd640 1 --2- 172.21.15.107:0/2104530184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b41033d0 0x7f76b4198540 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f76a400d770 tx=0x7f76a400dc30 comp rx=0 tx=0).stop 2024-05-09T08:42:06.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.767+0000 7f76bb8bd640 1 -- 172.21.15.107:0/2104530184 shutdown_connections 2024-05-09T08:42:06.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.767+0000 7f76bb8bd640 1 --2- 172.21.15.107:0/2104530184 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f76a00a2d80 0x7f76a00a5240 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.767+0000 7f76bb8bd640 1 --2- 172.21.15.107:0/2104530184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76b410bdf0 0x7f76b419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.767+0000 7f76bb8bd640 1 --2- 172.21.15.107:0/2104530184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b41033d0 0x7f76b4198540 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.767+0000 7f76bb8bd640 1 --2- 172.21.15.107:0/2104530184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b4102ab0 0x7f76b4198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.767+0000 7f76bb8bd640 1 -- 172.21.15.107:0/2104530184 >> 172.21.15.107:0/2104530184 conn(0x7f76b4077830 msgr2=0x7f76b40fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.767+0000 7f76bb8bd640 1 -- 172.21.15.107:0/2104530184 shutdown_connections 2024-05-09T08:42:06.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.767+0000 7f76bb8bd640 1 -- 172.21.15.107:0/2104530184 wait complete. 2024-05-09T08:42:06.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:42:06.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.896+0000 7fc2c6cd4640 1 -- 172.21.15.107:0/95648181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2c0100f50 msgr2=0x7fc2c010dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.896+0000 7fc2c6cd4640 1 --2- 172.21.15.107:0/95648181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2c0100f50 0x7fc2c010dca0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac009900 tx=0x7fc2ac02a410 comp rx=0 tx=0).stop 2024-05-09T08:42:06.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.896+0000 7fc2c6cd4640 1 -- 172.21.15.107:0/95648181 shutdown_connections 2024-05-09T08:42:06.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.896+0000 7fc2c6cd4640 1 --2- 172.21.15.107:0/95648181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2c010e1e0 0x7fc2c01105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.896+0000 7fc2c6cd4640 1 --2- 172.21.15.107:0/95648181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2c0100f50 0x7fc2c010dca0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.896+0000 7fc2c6cd4640 1 --2- 172.21.15.107:0/95648181 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2c0100630 0x7fc2c0100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.896+0000 7fc2c6cd4640 1 -- 172.21.15.107:0/95648181 >> 172.21.15.107:0/95648181 conn(0x7fc2c0077830 msgr2=0x7fc2c0077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.896+0000 7fc2c6cd4640 1 -- 172.21.15.107:0/95648181 shutdown_connections 2024-05-09T08:42:06.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.896+0000 7fc2c6cd4640 1 -- 172.21.15.107:0/95648181 wait complete. 2024-05-09T08:42:06.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.896+0000 7fc2c6cd4640 1 Processor -- start 2024-05-09T08:42:06.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2c6cd4640 1 -- start start 2024-05-09T08:42:06.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2c6cd4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2c0100630 0x7fc2c019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2c6cd4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2c0100f50 0x7fc2c019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2bffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2c0100f50 0x7fc2c019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2c6cd4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2c010e1e0 0x7fc2c019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2c6cd4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c0113cd0 con 0x7fc2c0100f50 2024-05-09T08:42:06.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2c6cd4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c0113b50 con 0x7fc2c010e1e0 2024-05-09T08:42:06.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2bffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2c0100f50 0x7fc2c019a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41606/0 (socket says 172.21.15.107:41606) 2024-05-09T08:42:06.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2c6cd4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c0113e50 con 0x7fc2c0100630 2024-05-09T08:42:06.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2bffff640 1 -- 172.21.15.107:0/2385131093 learned_addr learned my addr 172.21.15.107:0/2385131093 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:06.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2c4a70640 1 --2- 172.21.15.107:0/2385131093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2c0100630 0x7fc2c019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2c5271640 1 --2- 172.21.15.107:0/2385131093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2c010e1e0 0x7fc2c019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2bffff640 1 -- 172.21.15.107:0/2385131093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2c0100630 msgr2=0x7fc2c019a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2bffff640 1 --2- 172.21.15.107:0/2385131093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2c0100630 0x7fc2c019a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2bffff640 1 -- 172.21.15.107:0/2385131093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2c010e1e0 msgr2=0x7fc2c019ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2bffff640 1 --2- 172.21.15.107:0/2385131093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2c010e1e0 0x7fc2c019ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.897+0000 7fc2bffff640 1 -- 172.21.15.107:0/2385131093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2c019f200 con 0x7fc2c0100f50 2024-05-09T08:42:06.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.898+0000 7fc2bffff640 1 --2- 172.21.15.107:0/2385131093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2c0100f50 0x7fc2c019a750 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac005ab0 tx=0x7fc2ac003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.898+0000 7fc2bdffb640 1 -- 172.21.15.107:0/2385131093 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac03f070 con 0x7fc2c0100f50 2024-05-09T08:42:06.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.898+0000 7fc2bdffb640 1 -- 172.21.15.107:0/2385131093 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2ac031d60 con 0x7fc2c0100f50 2024-05-09T08:42:06.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.898+0000 7fc2bdffb640 1 -- 172.21.15.107:0/2385131093 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac035030 con 0x7fc2c0100f50 2024-05-09T08:42:06.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.898+0000 7fc2c6cd4640 1 -- 172.21.15.107:0/2385131093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2c019f490 con 0x7fc2c0100f50 2024-05-09T08:42:06.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.898+0000 7fc2c6cd4640 1 -- 172.21.15.107:0/2385131093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2c01a6d80 con 0x7fc2c0100f50 2024-05-09T08:42:06.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.901+0000 7fc2bdffb640 1 -- 172.21.15.107:0/2385131093 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fc2ac042070 con 0x7fc2c0100f50 2024-05-09T08:42:06.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.901+0000 7fc2c99e8740 1 Processor -- start 2024-05-09T08:42:06.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.901+0000 7fc2c99e8740 1 -- start start 2024-05-09T08:42:06.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.901+0000 7fc2c99e8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26d330 0x5616dc26d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.901+0000 7fc2c99e8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc26dc50 0x5616dc277de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.901+0000 7fc2c99e8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc278320 0x5616dc27a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.901+0000 7fc2c99e8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5616dc1d2c40 con 0x5616dc26dc50 2024-05-09T08:42:06.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.901+0000 7fc2c99e8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5616dc17a580 con 0x5616dc26d330 2024-05-09T08:42:06.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.901+0000 7fc2c99e8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5616dc1cec00 con 0x5616dc278320 2024-05-09T08:42:06.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.902+0000 7fc2c4a70640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc26dc50 0x5616dc277de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.902+0000 7fc2c4a70640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc26dc50 0x5616dc277de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41622/0 (socket says 172.21.15.107:41622) 2024-05-09T08:42:06.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.902+0000 7fc2c4a70640 1 -- 172.21.15.107:0/3132500793 learned_addr learned my addr 172.21.15.107:0/3132500793 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:06.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.902+0000 7fc2bffff640 1 --2- 172.21.15.107:0/3132500793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc278320 0x5616dc27a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.902+0000 7fc2c5271640 1 --2- 172.21.15.107:0/3132500793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26d330 0x5616dc26d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.902+0000 7fc2bdffb640 1 --2- 172.21.15.107:0/2385131093 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc2b00a2d80 0x7fc2b00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.902+0000 7fc2c5271640 1 -- 172.21.15.107:0/3132500793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc278320 msgr2=0x5616dc27a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.902+0000 7fc2c5271640 1 --2- 172.21.15.107:0/3132500793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc278320 0x5616dc27a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.902+0000 7fc2c5271640 1 -- 172.21.15.107:0/3132500793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc26dc50 msgr2=0x5616dc277de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.902+0000 7fc2c5271640 1 --2- 172.21.15.107:0/3132500793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc26dc50 0x5616dc277de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.902+0000 7fc2c5271640 1 -- 172.21.15.107:0/3132500793 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616dbee63f0 con 0x5616dc26d330 2024-05-09T08:42:06.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.902+0000 7fc2bdffb640 1 -- 172.21.15.107:0/2385131093 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9555+0+0 (secure 0 0 0) 0x7fc2ac0f0040 con 0x7fc2c0100f50 2024-05-09T08:42:06.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.902+0000 7fc2c5271640 1 --2- 172.21.15.107:0/2385131093 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc2b00a2d80 0x7fc2b00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.902+0000 7fc2c5271640 1 --2- 172.21.15.107:0/3132500793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26d330 0x5616dc26d710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8005fd0 tx=0x7fc2b80107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d91ba55b679f0e23 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.903+0000 7fc2aaffd640 1 -- 172.21.15.107:0/3132500793 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8011920 con 0x5616dc26d330 2024-05-09T08:42:06.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.903+0000 7fc2aaffd640 1 -- 172.21.15.107:0/3132500793 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2b8015070 con 0x5616dc26d330 2024-05-09T08:42:06.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.903+0000 7fc2aaffd640 1 -- 172.21.15.107:0/3132500793 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8004680 con 0x5616dc26d330 2024-05-09T08:42:06.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.903+0000 7fc2c99e8740 1 -- 172.21.15.107:0/3132500793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26d330 msgr2=0x5616dc26d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.903+0000 7fc2c5271640 1 --2- 172.21.15.107:0/2385131093 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc2b00a2d80 0x7fc2b00a5240 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc2b801f000 tx=0x7fc2b8020000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.903+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/3132500793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26d330 0x5616dc26d710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8005fd0 tx=0x7fc2b80107a0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.903+0000 7fc2c99e8740 1 -- 172.21.15.107:0/3132500793 shutdown_connections 2024-05-09T08:42:06.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.903+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/3132500793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc278320 0x5616dc27a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.903+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/3132500793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc26dc50 0x5616dc277de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.903+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/3132500793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26d330 0x5616dc26d710 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.904+0000 7fc2c99e8740 1 -- 172.21.15.107:0/3132500793 >> 172.21.15.107:0/3132500793 conn(0x5616dc269650 msgr2=0x5616dc269a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.904+0000 7fc2c99e8740 1 -- 172.21.15.107:0/3132500793 shutdown_connections 2024-05-09T08:42:06.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.904+0000 7fc2c99e8740 1 -- 172.21.15.107:0/3132500793 wait complete. 2024-05-09T08:42:06.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.904+0000 7fc2c99e8740 1 -- start start 2024-05-09T08:42:06.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.904+0000 7fc2c99e8740 1 Processor -- start 2024-05-09T08:42:06.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.904+0000 7fc2c99e8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26d330 0x5616dc275240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.904+0000 7fc2c99e8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc26dc50 0x5616dc275780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2c99e8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc278320 0x5616dc275cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2c99e8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5616dc27b110 con 0x5616dc278320 2024-05-09T08:42:06.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2c99e8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5616dc18dc40 con 0x5616dc26d330 2024-05-09T08:42:06.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2c99e8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5616dc27b290 con 0x5616dc26dc50 2024-05-09T08:42:06.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2c99e8740 10 client.0 _ll_register_callbacks cb 0x5616dbee5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:06.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2c5271640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc278320 0x5616dc275cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2c5271640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc278320 0x5616dc275cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41624/0 (socket says 172.21.15.107:41624) 2024-05-09T08:42:06.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2c5271640 1 -- 172.21.15.107:0/3659625937 learned_addr learned my addr 172.21.15.107:0/3659625937 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:06.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2c4a70640 1 --2- 172.21.15.107:0/3659625937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26d330 0x5616dc275240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2bffff640 1 --2- 172.21.15.107:0/3659625937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc26dc50 0x5616dc275780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2c5271640 1 -- 172.21.15.107:0/3659625937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc26dc50 msgr2=0x5616dc275780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2c5271640 1 --2- 172.21.15.107:0/3659625937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc26dc50 0x5616dc275780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2c5271640 1 -- 172.21.15.107:0/3659625937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26d330 msgr2=0x5616dc275240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2c5271640 1 --2- 172.21.15.107:0/3659625937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26d330 0x5616dc275240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2c5271640 1 -- 172.21.15.107:0/3659625937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616dbfb07d0 con 0x5616dc278320 2024-05-09T08:42:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2c4a70640 1 --2- 172.21.15.107:0/3659625937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26d330 0x5616dc275240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:42:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc2c5271640 1 --2- 172.21.15.107:0/3659625937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc278320 0x5616dc275cc0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8005bb0 tx=0x7fc2b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a0cf301685f48 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc29ffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc29ffff640 1 -- 172.21.15.107:0/3659625937 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8036060 con 0x5616dc278320 2024-05-09T08:42:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.905+0000 7fc29ffff640 1 -- 172.21.15.107:0/3659625937 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9555+0+0 (secure 0 0 0) 0x7fc2b8011920 con 0x5616dc278320 2024-05-09T08:42:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.906+0000 7fc2c99e8740 1 -- 172.21.15.107:0/3659625937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2b8015070 con 0x5616dc278320 2024-05-09T08:42:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.906+0000 7fc29ffff640 1 -- 172.21.15.107:0/3659625937 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc2b803d050 con 0x5616dc278320 2024-05-09T08:42:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.906+0000 7fc2c99e8740 10 client.18204 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.906+0000 7fc2c99e8740 1 -- 172.21.15.107:0/3659625937 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5616dc2872d0 con 0x5616dc278320 2024-05-09T08:42:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.906+0000 7fc29ffff640 1 -- 172.21.15.107:0/3659625937 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2b8015070 con 0x5616dc278320 2024-05-09T08:42:06.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.906+0000 7fc29ffff640 1 -- 172.21.15.107:0/3659625937 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc2b8030070 con 0x5616dc278320 2024-05-09T08:42:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.906+0000 7fc2c99e8740 10 client.18204 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.906+0000 7fc2c99e8740 10 client.18204 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:42:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.906+0000 7fc2c99e8740 20 client.18204 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.906+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/3659625937 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5616dc288680 0x5616dc28aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.906+0000 7fc2c99e8740 4 client.18204 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.906+0000 7fc2c99e8740 1 -- 172.21.15.107:0/3659625937 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616dc1e0370 con 0x5616dc288680 2024-05-09T08:42:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.906+0000 7fc2c4a70640 1 --2- 172.21.15.107:0/3659625937 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5616dc288680 0x5616dc28aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.907+0000 7fc2c4a70640 1 --2- 172.21.15.107:0/3659625937 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5616dc288680 0x5616dc28aa70 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3f5057ab6a16de51 server_cookie=a4f1be8c62656ed9 in_seq=0 out_seq=0 2024-05-09T08:42:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.907+0000 7fc29ffff640 10 client.18204 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:42:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.907+0000 7fc29ffff640 1 -- 172.21.15.107:0/3659625937 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5616dc1e0370 con 0x5616dc288680 2024-05-09T08:42:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.908+0000 7fc29ffff640 10 client.18204 handle_command_reply: tid=0 2024-05-09T08:42:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.908+0000 7fc2c99e8740 1 client.18204 shutdown 2024-05-09T08:42:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.908+0000 7fc2c99e8740 1 -- 172.21.15.107:0/3659625937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc278320 msgr2=0x5616dc275cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.908+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/3659625937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc278320 0x5616dc275cc0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8005bb0 tx=0x7fc2b8004a40 comp rx=0 tx=0).stop 2024-05-09T08:42:06.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.908+0000 7fc2c99e8740 1 -- 172.21.15.107:0/3659625937 shutdown_connections 2024-05-09T08:42:06.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.908+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/3659625937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc278320 0x5616dc275cc0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.908+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/3659625937 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5616dc288680 0x5616dc28aa70 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.908+0000 7fc29ffff640 0 client.18204 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:42:06.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.908+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/3659625937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc26dc50 0x5616dc275780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.908+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/3659625937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26d330 0x5616dc275240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.908+0000 7fc2c99e8740 1 -- 172.21.15.107:0/3659625937 >> 172.21.15.107:0/3659625937 conn(0x5616dc269650 msgr2=0x5616dc2687b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.908+0000 7fc2c99e8740 1 -- 172.21.15.107:0/3659625937 shutdown_connections 2024-05-09T08:42:06.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.908+0000 7fc2c99e8740 1 -- 172.21.15.107:0/3659625937 wait complete. 2024-05-09T08:42:06.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.908+0000 7fc2c99e8740 20 client.18204 trim_cache size 0 max 16384 2024-05-09T08:42:06.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.924+0000 7fc2c99e8740 1 Processor -- start 2024-05-09T08:42:06.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.924+0000 7fc2c99e8740 1 -- start start 2024-05-09T08:42:06.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.924+0000 7fc2c99e8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc26d330 0x5616dc277970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2c99e8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26dc50 0x5616dc284000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2c99e8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc278320 0x5616dc26f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2c99e8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c01b37d0 con 0x5616dc278320 2024-05-09T08:42:06.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2c99e8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c01b10a0 con 0x5616dc26dc50 2024-05-09T08:42:06.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2c99e8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c01aef10 con 0x5616dc26d330 2024-05-09T08:42:06.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2c5271640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc278320 0x5616dc26f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2c5271640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc278320 0x5616dc26f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41626/0 (socket says 172.21.15.107:41626) 2024-05-09T08:42:06.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2c5271640 1 -- 172.21.15.107:0/2470680401 learned_addr learned my addr 172.21.15.107:0/2470680401 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:06.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2bffff640 1 --2- 172.21.15.107:0/2470680401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26dc50 0x5616dc284000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2c4a70640 1 --2- 172.21.15.107:0/2470680401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc26d330 0x5616dc277970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2c5271640 1 -- 172.21.15.107:0/2470680401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc26d330 msgr2=0x5616dc277970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2c5271640 1 --2- 172.21.15.107:0/2470680401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc26d330 0x5616dc277970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2c5271640 1 -- 172.21.15.107:0/2470680401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26dc50 msgr2=0x5616dc284000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2c5271640 1 --2- 172.21.15.107:0/2470680401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26dc50 0x5616dc284000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2c5271640 1 -- 172.21.15.107:0/2470680401 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616dc270e10 con 0x5616dc278320 2024-05-09T08:42:06.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2c5271640 1 --2- 172.21.15.107:0/2470680401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc278320 0x5616dc26f1d0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fc2b80107e0 tx=0x7fc2b801ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d73816ade9ef2db server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2aa7fc640 1 -- 172.21.15.107:0/2470680401 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8036070 con 0x5616dc278320 2024-05-09T08:42:06.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2aa7fc640 1 -- 172.21.15.107:0/2470680401 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2b8015070 con 0x5616dc278320 2024-05-09T08:42:06.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.925+0000 7fc2aa7fc640 1 -- 172.21.15.107:0/2470680401 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8003970 con 0x5616dc278320 2024-05-09T08:42:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.926+0000 7fc2c99e8740 1 -- 172.21.15.107:0/2470680401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc278320 msgr2=0x5616dc26f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.926+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/2470680401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc278320 0x5616dc26f1d0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fc2b80107e0 tx=0x7fc2b801ef30 comp rx=0 tx=0).stop 2024-05-09T08:42:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.926+0000 7fc2c99e8740 1 -- 172.21.15.107:0/2470680401 shutdown_connections 2024-05-09T08:42:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.926+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/2470680401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc278320 0x5616dc26f1d0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.926+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/2470680401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26dc50 0x5616dc284000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.926+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/2470680401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc26d330 0x5616dc277970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.926+0000 7fc2c99e8740 1 -- 172.21.15.107:0/2470680401 >> 172.21.15.107:0/2470680401 conn(0x5616dc269650 msgr2=0x5616dc26eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.926+0000 7fc2c99e8740 1 -- 172.21.15.107:0/2470680401 shutdown_connections 2024-05-09T08:42:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.927+0000 7fc2c99e8740 1 -- 172.21.15.107:0/2470680401 wait complete. 2024-05-09T08:42:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.927+0000 7fc2c99e8740 1 -- start start 2024-05-09T08:42:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.927+0000 7fc2c99e8740 1 Processor -- start 2024-05-09T08:42:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2c99e8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc26d330 0x5616dc280800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2c99e8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26dc50 0x5616dc27bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2c99e8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc278320 0x5616dc27c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2c99e8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c01af560 con 0x5616dc26d330 2024-05-09T08:42:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2c99e8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c01b09d0 con 0x5616dc26dc50 2024-05-09T08:42:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2c99e8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c01afc30 con 0x5616dc278320 2024-05-09T08:42:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2c99e8740 10 client.0 _ll_register_callbacks cb 0x7fc2b400b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2bffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26dc50 0x5616dc27bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2c4a70640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc26d330 0x5616dc280800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2bffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26dc50 0x5616dc27bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:52488/0 (socket says 172.21.15.107:52488) 2024-05-09T08:42:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2bffff640 1 -- 172.21.15.107:0/4138948856 learned_addr learned my addr 172.21.15.107:0/4138948856 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2c5271640 1 --2- 172.21.15.107:0/4138948856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc278320 0x5616dc27c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2c4a70640 1 -- 172.21.15.107:0/4138948856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc278320 msgr2=0x5616dc27c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2c4a70640 1 --2- 172.21.15.107:0/4138948856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc278320 0x5616dc27c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2c4a70640 1 -- 172.21.15.107:0/4138948856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26dc50 msgr2=0x5616dc27bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2c4a70640 1 --2- 172.21.15.107:0/4138948856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26dc50 0x5616dc27bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2c4a70640 1 -- 172.21.15.107:0/4138948856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616dc27c750 con 0x5616dc26d330 2024-05-09T08:42:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2bffff640 1 --2- 172.21.15.107:0/4138948856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26dc50 0x5616dc27bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:42:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc2c4a70640 1 --2- 172.21.15.107:0/4138948856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc26d330 0x5616dc280800 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x5616dc27b260 tx=0x7fc2b400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b555aea9433ed37 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc29ffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc29ffff640 1 -- 172.21.15.107:0/4138948856 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b4003dd0 con 0x5616dc26d330 2024-05-09T08:42:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.928+0000 7fc29ffff640 1 -- 172.21.15.107:0/4138948856 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9555+0+0 (secure 0 0 0) 0x7fc2b4020160 con 0x5616dc26d330 2024-05-09T08:42:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.929+0000 7fc2c99e8740 1 -- 172.21.15.107:0/4138948856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2b8015070 con 0x5616dc26d330 2024-05-09T08:42:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.929+0000 7fc29ffff640 1 -- 172.21.15.107:0/4138948856 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc2b402f050 con 0x5616dc26d330 2024-05-09T08:42:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.929+0000 7fc2c99e8740 10 client.18210 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.929+0000 7fc2c99e8740 1 -- 172.21.15.107:0/4138948856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5616dc2ee8a0 con 0x5616dc26d330 2024-05-09T08:42:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.929+0000 7fc29ffff640 1 -- 172.21.15.107:0/4138948856 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2b8015070 con 0x5616dc26d330 2024-05-09T08:42:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.929+0000 7fc29ffff640 1 -- 172.21.15.107:0/4138948856 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc2b4011a90 con 0x5616dc26d330 2024-05-09T08:42:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.929+0000 7fc2c99e8740 10 client.18210 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.929+0000 7fc2c99e8740 10 client.18210 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:42:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.929+0000 7fc2c99e8740 20 client.18210 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.929+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/4138948856 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5616dc2eefc0 0x5616dc2f13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.929+0000 7fc2c99e8740 4 client.18210 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:42:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.929+0000 7fc2c99e8740 1 -- 172.21.15.107:0/4138948856 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5616dc1da6c0 con 0x5616dc2eefc0 2024-05-09T08:42:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.929+0000 7fc2bffff640 1 --2- 172.21.15.107:0/4138948856 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5616dc2eefc0 0x5616dc2f13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc29ffff640 10 client.18210 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:42:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc2bffff640 1 --2- 172.21.15.107:0/4138948856 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5616dc2eefc0 0x5616dc2f13e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=be1f56f8fe94e92 server_cookie=f28547504e0899ac in_seq=0 out_seq=0 2024-05-09T08:42:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc29ffff640 1 -- 172.21.15.107:0/4138948856 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5616dc1da6c0 con 0x5616dc2eefc0 2024-05-09T08:42:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc29ffff640 10 client.18210 handle_command_reply: tid=0 2024-05-09T08:42:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc2c99e8740 1 client.18210 shutdown 2024-05-09T08:42:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc2c99e8740 1 -- 172.21.15.107:0/4138948856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc26d330 msgr2=0x5616dc280800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/4138948856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc26d330 0x5616dc280800 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x5616dc27b260 tx=0x7fc2b400be50 comp rx=0 tx=0).stop 2024-05-09T08:42:06.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc2c99e8740 1 -- 172.21.15.107:0/4138948856 shutdown_connections 2024-05-09T08:42:06.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/4138948856 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5616dc2eefc0 0x5616dc2f13e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc29ffff640 0 client.18210 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:42:06.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/4138948856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5616dc278320 0x5616dc27c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/4138948856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5616dc26dc50 0x5616dc27bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc2c99e8740 1 --2- 172.21.15.107:0/4138948856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5616dc26d330 0x5616dc280800 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc2c99e8740 1 -- 172.21.15.107:0/4138948856 >> 172.21.15.107:0/4138948856 conn(0x5616dc269650 msgr2=0x5616dc26f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc2c99e8740 1 -- 172.21.15.107:0/4138948856 shutdown_connections 2024-05-09T08:42:06.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc2c99e8740 1 -- 172.21.15.107:0/4138948856 wait complete. 2024-05-09T08:42:06.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.930+0000 7fc2c99e8740 20 client.18210 trim_cache size 0 max 16384 2024-05-09T08:42:06.936 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:42:06.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.931+0000 7fc2c6cd4640 1 -- 172.21.15.107:0/2385131093 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc2b00a2d80 msgr2=0x7fc2b00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.931+0000 7fc2c6cd4640 1 --2- 172.21.15.107:0/2385131093 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc2b00a2d80 0x7fc2b00a5240 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc2b801f000 tx=0x7fc2b8020000 comp rx=0 tx=0).stop 2024-05-09T08:42:06.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.931+0000 7fc2c6cd4640 1 -- 172.21.15.107:0/2385131093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2c0100f50 msgr2=0x7fc2c019a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:06.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.931+0000 7fc2c6cd4640 1 --2- 172.21.15.107:0/2385131093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2c0100f50 0x7fc2c019a750 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac005ab0 tx=0x7fc2ac003c80 comp rx=0 tx=0).stop 2024-05-09T08:42:06.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.931+0000 7fc2c6cd4640 1 -- 172.21.15.107:0/2385131093 shutdown_connections 2024-05-09T08:42:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.931+0000 7fc2c6cd4640 1 --2- 172.21.15.107:0/2385131093 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc2b00a2d80 0x7fc2b00a5240 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.931+0000 7fc2c6cd4640 1 --2- 172.21.15.107:0/2385131093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2c010e1e0 0x7fc2c019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.931+0000 7fc2c6cd4640 1 --2- 172.21.15.107:0/2385131093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2c0100f50 0x7fc2c019a750 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.931+0000 7fc2c6cd4640 1 --2- 172.21.15.107:0/2385131093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2c0100630 0x7fc2c019a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.931+0000 7fc2c6cd4640 1 -- 172.21.15.107:0/2385131093 >> 172.21.15.107:0/2385131093 conn(0x7fc2c0077830 msgr2=0x7fc2c010f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.931+0000 7fc2c6cd4640 1 -- 172.21.15.107:0/2385131093 shutdown_connections 2024-05-09T08:42:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:06.931+0000 7fc2c6cd4640 1 -- 172.21.15.107:0/2385131093 wait complete. 2024-05-09T08:42:06.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:42:07.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.064+0000 7f3f2af12640 1 -- 172.21.15.107:0/4112850333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24101810 msgr2=0x7f3f24101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:07.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.064+0000 7f3f2af12640 1 --2- 172.21.15.107:0/4112850333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24101810 0x7f3f24101c90 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009930 tx=0x7f3f1002a420 comp rx=0 tx=0).stop 2024-05-09T08:42:07.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.064+0000 7f3f2af12640 1 -- 172.21.15.107:0/4112850333 shutdown_connections 2024-05-09T08:42:07.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.064+0000 7f3f2af12640 1 --2- 172.21.15.107:0/4112850333 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f241021d0 0x7f3f2410e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.064+0000 7f3f2af12640 1 --2- 172.21.15.107:0/4112850333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24101810 0x7f3f24101c90 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.064+0000 7f3f2af12640 1 --2- 172.21.15.107:0/4112850333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f24107820 0x7f3f24107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.064+0000 7f3f2af12640 1 -- 172.21.15.107:0/4112850333 >> 172.21.15.107:0/4112850333 conn(0x7f3f24078c30 msgr2=0x7f3f24079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:07.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.065+0000 7f3f2af12640 1 -- 172.21.15.107:0/4112850333 shutdown_connections 2024-05-09T08:42:07.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.065+0000 7f3f2af12640 1 -- 172.21.15.107:0/4112850333 wait complete. 2024-05-09T08:42:07.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.065+0000 7f3f2af12640 1 Processor -- start 2024-05-09T08:42:07.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.065+0000 7f3f2af12640 1 -- start start 2024-05-09T08:42:07.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.065+0000 7f3f2af12640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24101810 0x7f3f2419a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.065+0000 7f3f2af12640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f241021d0 0x7f3f2419a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.065+0000 7f3f2af12640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f24107820 0x7f3f2419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.065+0000 7f3f2af12640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3f24111ae0 con 0x7f3f24101810 2024-05-09T08:42:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.065+0000 7f3f2af12640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3f24111960 con 0x7f3f24107820 2024-05-09T08:42:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.065+0000 7f3f2af12640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3f24111c60 con 0x7f3f241021d0 2024-05-09T08:42:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.065+0000 7f3f28cae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24101810 0x7f3f2419a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.065+0000 7f3f28cae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24101810 0x7f3f2419a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41650/0 (socket says 172.21.15.107:41650) 2024-05-09T08:42:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.065+0000 7f3f28cae640 1 -- 172.21.15.107:0/1491977484 learned_addr learned my addr 172.21.15.107:0/1491977484 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.066+0000 7f3f294af640 1 --2- 172.21.15.107:0/1491977484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f24107820 0x7f3f2419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.066+0000 7f3f23fff640 1 --2- 172.21.15.107:0/1491977484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f241021d0 0x7f3f2419a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.066+0000 7f3f28cae640 1 -- 172.21.15.107:0/1491977484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f241021d0 msgr2=0x7f3f2419a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.066+0000 7f3f28cae640 1 --2- 172.21.15.107:0/1491977484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f241021d0 0x7f3f2419a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.066+0000 7f3f28cae640 1 -- 172.21.15.107:0/1491977484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f24107820 msgr2=0x7f3f2419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.066+0000 7f3f28cae640 1 --2- 172.21.15.107:0/1491977484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f24107820 0x7f3f2419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.066+0000 7f3f28cae640 1 -- 172.21.15.107:0/1491977484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f2419f2a0 con 0x7f3f24101810 2024-05-09T08:42:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.066+0000 7f3f294af640 1 --2- 172.21.15.107:0/1491977484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f24107820 0x7f3f2419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:42:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.066+0000 7f3f28cae640 1 --2- 172.21.15.107:0/1491977484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24101810 0x7f3f2419a300 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f3f1800d950 tx=0x7f3f1800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.066+0000 7f3f21ffb640 1 -- 172.21.15.107:0/1491977484 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f18017070 con 0x7f3f24101810 2024-05-09T08:42:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.066+0000 7f3f2af12640 1 -- 172.21.15.107:0/1491977484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f2419f590 con 0x7f3f24101810 2024-05-09T08:42:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.066+0000 7f3f21ffb640 1 -- 172.21.15.107:0/1491977484 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f18009c30 con 0x7f3f24101810 2024-05-09T08:42:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.066+0000 7f3f21ffb640 1 -- 172.21.15.107:0/1491977484 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1800f620 con 0x7f3f24101810 2024-05-09T08:42:07.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.066+0000 7f3f2af12640 1 -- 172.21.15.107:0/1491977484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f241a6e20 con 0x7f3f24101810 2024-05-09T08:42:07.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.069+0000 7f3f21ffb640 1 -- 172.21.15.107:0/1491977484 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3f180109a0 con 0x7f3f24101810 2024-05-09T08:42:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.069+0000 7f3f2dc16740 1 Processor -- start 2024-05-09T08:42:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.069+0000 7f3f2dc16740 1 -- start start 2024-05-09T08:42:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.069+0000 7f3f2dc16740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383330 0x563ac1383710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.069+0000 7f3f2dc16740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac1383c50 0x563ac138dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.069+0000 7f3f2dc16740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac138e320 0x563ac1390710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.069+0000 7f3f2dc16740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563ac12e8c40 con 0x563ac1383330 2024-05-09T08:42:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.069+0000 7f3f2dc16740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563ac1290580 con 0x563ac138e320 2024-05-09T08:42:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.069+0000 7f3f2dc16740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563ac12e4c00 con 0x563ac1383c50 2024-05-09T08:42:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.069+0000 7f3f294af640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383330 0x563ac1383710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.069+0000 7f3f28cae640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac1383c50 0x563ac138dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.069+0000 7f3f294af640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383330 0x563ac1383710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41666/0 (socket says 172.21.15.107:41666) 2024-05-09T08:42:07.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.069+0000 7f3f294af640 1 -- 172.21.15.107:0/711723535 learned_addr learned my addr 172.21.15.107:0/711723535 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:07.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.070+0000 7f3f21ffb640 1 --2- 172.21.15.107:0/1491977484 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3f140a2d00 0x7f3f140a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.070+0000 7f3f23fff640 1 --2- 172.21.15.107:0/711723535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac138e320 0x563ac1390710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.070+0000 7f3f294af640 1 -- 172.21.15.107:0/711723535 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac1383c50 msgr2=0x563ac138dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:07.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.070+0000 7f3f294af640 1 --2- 172.21.15.107:0/711723535 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac1383c50 0x563ac138dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.070+0000 7f3f294af640 1 -- 172.21.15.107:0/711723535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac138e320 msgr2=0x563ac1390710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:07.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.070+0000 7f3f294af640 1 --2- 172.21.15.107:0/711723535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac138e320 0x563ac1390710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.070+0000 7f3f294af640 1 -- 172.21.15.107:0/711723535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ac0ffc3f0 con 0x563ac1383330 2024-05-09T08:42:07.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.070+0000 7f3f21ffb640 1 -- 172.21.15.107:0/1491977484 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f3f180cf480 con 0x7f3f24101810 2024-05-09T08:42:07.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.070+0000 7f3f294af640 1 --2- 172.21.15.107:0/1491977484 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3f140a2d00 0x7f3f140a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.070+0000 7f3f28cae640 1 --2- 172.21.15.107:0/711723535 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac1383c50 0x563ac138dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:42:07.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.070+0000 7f3f23fff640 1 --2- 172.21.15.107:0/711723535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac138e320 0x563ac1390710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:42:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.070+0000 7f3f294af640 1 --2- 172.21.15.107:0/711723535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383330 0x563ac1383710 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c010620 tx=0x7f3f1c010ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=636cf8c400cde2e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.070+0000 7f3f0effd640 1 -- 172.21.15.107:0/711723535 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c019070 con 0x563ac1383330 2024-05-09T08:42:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.070+0000 7f3f0effd640 1 -- 172.21.15.107:0/711723535 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f1c022070 con 0x563ac1383330 2024-05-09T08:42:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.070+0000 7f3f0effd640 1 -- 172.21.15.107:0/711723535 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c011b30 con 0x563ac1383330 2024-05-09T08:42:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.071+0000 7f3f2dc16740 1 -- 172.21.15.107:0/711723535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383330 msgr2=0x563ac1383710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.071+0000 7f3f294af640 1 --2- 172.21.15.107:0/1491977484 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3f140a2d00 0x7f3f140a51c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c021000 tx=0x7f3f1c01f000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.071+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/711723535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383330 0x563ac1383710 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c010620 tx=0x7f3f1c010ae0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.071+0000 7f3f2dc16740 1 -- 172.21.15.107:0/711723535 shutdown_connections 2024-05-09T08:42:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.071+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/711723535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac138e320 0x563ac1390710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.071+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/711723535 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac1383c50 0x563ac138dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.071+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/711723535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383330 0x563ac1383710 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.071+0000 7f3f2dc16740 1 -- 172.21.15.107:0/711723535 >> 172.21.15.107:0/711723535 conn(0x563ac137f650 msgr2=0x563ac137fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.071+0000 7f3f2dc16740 1 -- 172.21.15.107:0/711723535 shutdown_connections 2024-05-09T08:42:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.071+0000 7f3f2dc16740 1 -- 172.21.15.107:0/711723535 wait complete. 2024-05-09T08:42:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.071+0000 7f3f2dc16740 1 -- start start 2024-05-09T08:42:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f2dc16740 1 Processor -- start 2024-05-09T08:42:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f2dc16740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 0x563ac138b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f2dc16740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383c50 0x563ac138b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f2dc16740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac138e320 0x563ac138bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f2dc16740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563ac1391110 con 0x563ac1383c50 2024-05-09T08:42:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f2dc16740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563ac12a3c40 con 0x563ac1383330 2024-05-09T08:42:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f2dc16740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563ac1391290 con 0x563ac138e320 2024-05-09T08:42:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f2dc16740 10 client.0 _ll_register_callbacks cb 0x563ac0ffbbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f23fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383c50 0x563ac138b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f23fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383c50 0x563ac138b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41678/0 (socket says 172.21.15.107:41678) 2024-05-09T08:42:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f23fff640 1 -- 172.21.15.107:0/3052454277 learned_addr learned my addr 172.21.15.107:0/3052454277 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f294af640 1 --2- 172.21.15.107:0/3052454277 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac138e320 0x563ac138bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f28cae640 1 --2- 172.21.15.107:0/3052454277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 0x563ac138b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f294af640 1 -- 172.21.15.107:0/3052454277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 msgr2=0x563ac138b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f294af640 1 --2- 172.21.15.107:0/3052454277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 0x563ac138b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f294af640 1 -- 172.21.15.107:0/3052454277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383c50 msgr2=0x563ac138b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f294af640 1 --2- 172.21.15.107:0/3052454277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383c50 0x563ac138b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f294af640 1 -- 172.21.15.107:0/3052454277 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ac10c67d0 con 0x563ac138e320 2024-05-09T08:42:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.072+0000 7f3f23fff640 1 --2- 172.21.15.107:0/3052454277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383c50 0x563ac138b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:42:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.073+0000 7f3f294af640 1 --2- 172.21.15.107:0/3052454277 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac138e320 0x563ac138bd70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c005fd0 tx=0x7f3f1c011ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d42a3ee8f291cf1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.073+0000 7f3f03fff640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:42:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.073+0000 7f3f03fff640 1 -- 172.21.15.107:0/3052454277 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c019040 con 0x563ac138e320 2024-05-09T08:42:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.073+0000 7f3f2dc16740 1 -- 172.21.15.107:0/3052454277 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f1c022070 con 0x563ac138e320 2024-05-09T08:42:07.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.073+0000 7f3f03fff640 1 -- 172.21.15.107:0/3052454277 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f3f1c013ab0 con 0x563ac138e320 2024-05-09T08:42:07.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.073+0000 7f3f03fff640 1 -- 172.21.15.107:0/3052454277 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3f1c009d00 con 0x563ac138e320 2024-05-09T08:42:07.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.073+0000 7f3f2dc16740 10 client.27409 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:07.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.073+0000 7f3f2dc16740 1 -- 172.21.15.107:0/3052454277 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563ac139d190 con 0x563ac138e320 2024-05-09T08:42:07.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.073+0000 7f3f03fff640 1 -- 172.21.15.107:0/3052454277 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f1c022070 con 0x563ac138e320 2024-05-09T08:42:07.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.073+0000 7f3f03fff640 1 -- 172.21.15.107:0/3052454277 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3f1c02f920 con 0x563ac138e320 2024-05-09T08:42:07.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.073+0000 7f3f2dc16740 10 client.27409 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:07.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.073+0000 7f3f2dc16740 10 client.27409 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:07.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.073+0000 7f3f2dc16740 20 client.27409 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:07.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.073+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/3052454277 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563ac139e5a0 0x563ac13a0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.073+0000 7f3f2dc16740 4 client.27409 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:07.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.073+0000 7f3f2dc16740 1 -- 172.21.15.107:0/3052454277 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ac12f6370 con 0x563ac139e5a0 2024-05-09T08:42:07.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.074+0000 7f3f28cae640 1 --2- 172.21.15.107:0/3052454277 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563ac139e5a0 0x563ac13a0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.074+0000 7f3f28cae640 1 --2- 172.21.15.107:0/3052454277 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563ac139e5a0 0x563ac13a0990 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a76b8028e7aabfc6 server_cookie=2642662f097c413 in_seq=0 out_seq=0 2024-05-09T08:42:07.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.074+0000 7f3f03fff640 10 client.27409 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:07.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.075+0000 7f3f03fff640 1 -- 172.21.15.107:0/3052454277 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563ac12f6370 con 0x563ac139e5a0 2024-05-09T08:42:07.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.075+0000 7f3f03fff640 10 client.27409 handle_command_reply: tid=0 2024-05-09T08:42:07.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.075+0000 7f3f2dc16740 1 client.27409 shutdown 2024-05-09T08:42:07.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.075+0000 7f3f2dc16740 1 -- 172.21.15.107:0/3052454277 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac138e320 msgr2=0x563ac138bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:07.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.075+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/3052454277 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac138e320 0x563ac138bd70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c005fd0 tx=0x7f3f1c011ac0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.075+0000 7f3f2dc16740 1 -- 172.21.15.107:0/3052454277 shutdown_connections 2024-05-09T08:42:07.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.075+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/3052454277 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563ac139e5a0 0x563ac13a0990 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.075+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/3052454277 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac138e320 0x563ac138bd70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.075+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/3052454277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383c50 0x563ac138b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.075+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/3052454277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 0x563ac138b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.075+0000 7f3f03fff640 0 client.27409 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:07.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.075+0000 7f3f2dc16740 1 -- 172.21.15.107:0/3052454277 >> 172.21.15.107:0/3052454277 conn(0x563ac137f650 msgr2=0x563ac1385450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:07.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.075+0000 7f3f2dc16740 1 -- 172.21.15.107:0/3052454277 shutdown_connections 2024-05-09T08:42:07.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.075+0000 7f3f2dc16740 1 -- 172.21.15.107:0/3052454277 wait complete. 2024-05-09T08:42:07.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.075+0000 7f3f2dc16740 20 client.27409 trim_cache size 0 max 16384 2024-05-09T08:42:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.091+0000 7f3f2dc16740 1 Processor -- start 2024-05-09T08:42:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.091+0000 7f3f2dc16740 1 -- start start 2024-05-09T08:42:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.091+0000 7f3f2dc16740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 0x563ac1399fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.091+0000 7f3f2dc16740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383c50 0x563ac138bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.091+0000 7f3f2dc16740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac138e320 0x563ac13a6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.091+0000 7f3f2dc16740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3f241b37b0 con 0x563ac1383c50 2024-05-09T08:42:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.091+0000 7f3f2dc16740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3f241b1080 con 0x563ac1383330 2024-05-09T08:42:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.091+0000 7f3f2dc16740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3f241aeef0 con 0x563ac138e320 2024-05-09T08:42:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.091+0000 7f3f23fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383c50 0x563ac138bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.091+0000 7f3f23fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383c50 0x563ac138bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41694/0 (socket says 172.21.15.107:41694) 2024-05-09T08:42:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.091+0000 7f3f23fff640 1 -- 172.21.15.107:0/4132961721 learned_addr learned my addr 172.21.15.107:0/4132961721 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.091+0000 7f3f28cae640 1 --2- 172.21.15.107:0/4132961721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 0x563ac1399fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.092+0000 7f3f294af640 1 --2- 172.21.15.107:0/4132961721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac138e320 0x563ac13a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.092+0000 7f3f23fff640 1 -- 172.21.15.107:0/4132961721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac138e320 msgr2=0x563ac13a6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.092+0000 7f3f23fff640 1 --2- 172.21.15.107:0/4132961721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac138e320 0x563ac13a6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.092+0000 7f3f23fff640 1 -- 172.21.15.107:0/4132961721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 msgr2=0x563ac1399fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.092+0000 7f3f23fff640 1 --2- 172.21.15.107:0/4132961721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 0x563ac1399fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.092+0000 7f3f23fff640 1 -- 172.21.15.107:0/4132961721 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ac13a0d00 con 0x563ac1383c50 2024-05-09T08:42:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.092+0000 7f3f28cae640 1 --2- 172.21.15.107:0/4132961721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 0x563ac1399fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:42:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.092+0000 7f3f294af640 1 --2- 172.21.15.107:0/4132961721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac138e320 0x563ac13a6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:42:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.092+0000 7f3f23fff640 1 --2- 172.21.15.107:0/4132961721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383c50 0x563ac138bb90 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f3f10006010 tx=0x7f3f10005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f2c3f47275171e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.092+0000 7f3f0e7fc640 1 -- 172.21.15.107:0/4132961721 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f10035040 con 0x563ac1383c50 2024-05-09T08:42:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.092+0000 7f3f0e7fc640 1 -- 172.21.15.107:0/4132961721 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f10005040 con 0x563ac1383c50 2024-05-09T08:42:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.092+0000 7f3f0e7fc640 1 -- 172.21.15.107:0/4132961721 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f10004900 con 0x563ac1383c50 2024-05-09T08:42:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.093+0000 7f3f2dc16740 1 -- 172.21.15.107:0/4132961721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383c50 msgr2=0x563ac138bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.093+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/4132961721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383c50 0x563ac138bb90 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f3f10006010 tx=0x7f3f10005e80 comp rx=0 tx=0).stop 2024-05-09T08:42:07.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.093+0000 7f3f2dc16740 1 -- 172.21.15.107:0/4132961721 shutdown_connections 2024-05-09T08:42:07.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.093+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/4132961721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac138e320 0x563ac13a6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.093+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/4132961721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac1383c50 0x563ac138bb90 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.093+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/4132961721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 0x563ac1399fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.093+0000 7f3f2dc16740 1 -- 172.21.15.107:0/4132961721 >> 172.21.15.107:0/4132961721 conn(0x563ac137f650 msgr2=0x563ac137e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:07.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.093+0000 7f3f2dc16740 1 -- 172.21.15.107:0/4132961721 shutdown_connections 2024-05-09T08:42:07.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.093+0000 7f3f2dc16740 1 -- 172.21.15.107:0/4132961721 wait complete. 2024-05-09T08:42:07.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.093+0000 7f3f2dc16740 1 -- start start 2024-05-09T08:42:07.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.094+0000 7f3f2dc16740 1 Processor -- start 2024-05-09T08:42:07.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.094+0000 7f3f2dc16740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 0x563ac1387f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.094+0000 7f3f2dc16740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac1383c50 0x563ac13884a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.094+0000 7f3f2dc16740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac138e320 0x563ac13889e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.094+0000 7f3f2dc16740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3f241af540 con 0x563ac138e320 2024-05-09T08:42:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.094+0000 7f3f2dc16740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3f241b09b0 con 0x563ac1383330 2024-05-09T08:42:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.094+0000 7f3f2dc16740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3f241afc10 con 0x563ac1383c50 2024-05-09T08:42:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.094+0000 7f3f294af640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac138e320 0x563ac13889e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.094+0000 7f3f2dc16740 10 client.0 _ll_register_callbacks cb 0x7f3f18097910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.094+0000 7f3f294af640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac138e320 0x563ac13889e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41704/0 (socket says 172.21.15.107:41704) 2024-05-09T08:42:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.094+0000 7f3f23fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac1383c50 0x563ac13884a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.094+0000 7f3f294af640 1 -- 172.21.15.107:0/3182279121 learned_addr learned my addr 172.21.15.107:0/3182279121 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.094+0000 7f3f28cae640 1 --2- 172.21.15.107:0/3182279121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 0x563ac1387f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.095+0000 7f3f294af640 1 -- 172.21.15.107:0/3182279121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac1383c50 msgr2=0x563ac13884a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.095+0000 7f3f294af640 1 --2- 172.21.15.107:0/3182279121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac1383c50 0x563ac13884a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.095+0000 7f3f294af640 1 -- 172.21.15.107:0/3182279121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 msgr2=0x563ac1387f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.095+0000 7f3f294af640 1 --2- 172.21.15.107:0/3182279121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 0x563ac1387f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.095+0000 7f3f294af640 1 -- 172.21.15.107:0/3182279121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ac13860f0 con 0x563ac138e320 2024-05-09T08:42:07.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.095+0000 7f3f23fff640 1 --2- 172.21.15.107:0/3182279121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac1383c50 0x563ac13884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:42:07.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.095+0000 7f3f28cae640 1 --2- 172.21.15.107:0/3182279121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 0x563ac1387f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:42:07.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.095+0000 7f3f294af640 1 --2- 172.21.15.107:0/3182279121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac138e320 0x563ac13889e0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c010620 tx=0x7f3f1c02fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bca963733703de4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:07.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.095+0000 7f3f03fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:07.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.095+0000 7f3f03fff640 1 -- 172.21.15.107:0/3182279121 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c041060 con 0x563ac138e320 2024-05-09T08:42:07.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.095+0000 7f3f03fff640 1 -- 172.21.15.107:0/3182279121 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f3f1c013830 con 0x563ac138e320 2024-05-09T08:42:07.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.095+0000 7f3f2dc16740 1 -- 172.21.15.107:0/3182279121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f10005040 con 0x563ac138e320 2024-05-09T08:42:07.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.095+0000 7f3f03fff640 1 -- 172.21.15.107:0/3182279121 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3f1c048050 con 0x563ac138e320 2024-05-09T08:42:07.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.095+0000 7f3f2dc16740 10 client.18228 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:07.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.095+0000 7f3f2dc16740 1 -- 172.21.15.107:0/3182279121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563ac1404a70 con 0x563ac138e320 2024-05-09T08:42:07.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.095+0000 7f3f03fff640 1 -- 172.21.15.107:0/3182279121 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f10005040 con 0x563ac138e320 2024-05-09T08:42:07.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.096+0000 7f3f03fff640 1 -- 172.21.15.107:0/3182279121 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f3f1c013c40 con 0x563ac138e320 2024-05-09T08:42:07.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.096+0000 7f3f2dc16740 10 client.18228 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:07.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.096+0000 7f3f2dc16740 10 client.18228 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:07.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.096+0000 7f3f2dc16740 20 client.18228 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:07.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.096+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/3182279121 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563ac1405160 0x563ac1407580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:07.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.096+0000 7f3f2dc16740 4 client.18228 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:42:07.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.096+0000 7f3f2dc16740 1 -- 172.21.15.107:0/3182279121 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563ac12f06c0 con 0x563ac1405160 2024-05-09T08:42:07.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.096+0000 7f3f28cae640 1 --2- 172.21.15.107:0/3182279121 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563ac1405160 0x563ac1407580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:07.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.096+0000 7f3f03fff640 10 client.18228 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:07.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.096+0000 7f3f28cae640 1 --2- 172.21.15.107:0/3182279121 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563ac1405160 0x563ac1407580 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8d8f05dd1b91481 server_cookie=f48565677177135f in_seq=0 out_seq=0 2024-05-09T08:42:07.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.096+0000 7f3f03fff640 1 -- 172.21.15.107:0/3182279121 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563ac12f06c0 con 0x563ac1405160 2024-05-09T08:42:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.096+0000 7f3f03fff640 10 client.18228 handle_command_reply: tid=0 2024-05-09T08:42:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.096+0000 7f3f2dc16740 1 client.18228 shutdown 2024-05-09T08:42:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.097+0000 7f3f2dc16740 1 -- 172.21.15.107:0/3182279121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac138e320 msgr2=0x563ac13889e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.097+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/3182279121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac138e320 0x563ac13889e0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c010620 tx=0x7f3f1c02fb30 comp rx=0 tx=0).stop 2024-05-09T08:42:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.097+0000 7f3f2dc16740 1 -- 172.21.15.107:0/3182279121 shutdown_connections 2024-05-09T08:42:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.097+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/3182279121 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563ac1405160 0x563ac1407580 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.097+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/3182279121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ac138e320 0x563ac13889e0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.097+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/3182279121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563ac1383c50 0x563ac13884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.097+0000 7f3f03fff640 0 client.18228 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.097+0000 7f3f2dc16740 1 --2- 172.21.15.107:0/3182279121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563ac1383330 0x563ac1387f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.097+0000 7f3f2dc16740 1 -- 172.21.15.107:0/3182279121 >> 172.21.15.107:0/3182279121 conn(0x563ac137f650 msgr2=0x563ac138c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.097+0000 7f3f2dc16740 1 -- 172.21.15.107:0/3182279121 shutdown_connections 2024-05-09T08:42:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.097+0000 7f3f2dc16740 1 -- 172.21.15.107:0/3182279121 wait complete. 2024-05-09T08:42:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.097+0000 7f3f2dc16740 20 client.18228 trim_cache size 0 max 16384 2024-05-09T08:42:07.105 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:42:07.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.098+0000 7f3f2af12640 1 -- 172.21.15.107:0/1491977484 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3f140a2d00 msgr2=0x7f3f140a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:07.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.098+0000 7f3f2af12640 1 --2- 172.21.15.107:0/1491977484 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3f140a2d00 0x7f3f140a51c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c021000 tx=0x7f3f1c01f000 comp rx=0 tx=0).stop 2024-05-09T08:42:07.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.098+0000 7f3f2af12640 1 -- 172.21.15.107:0/1491977484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24101810 msgr2=0x7f3f2419a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:07.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.098+0000 7f3f2af12640 1 --2- 172.21.15.107:0/1491977484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24101810 0x7f3f2419a300 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f3f1800d950 tx=0x7f3f1800de10 comp rx=0 tx=0).stop 2024-05-09T08:42:07.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.098+0000 7f3f2af12640 1 -- 172.21.15.107:0/1491977484 shutdown_connections 2024-05-09T08:42:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.098+0000 7f3f2af12640 1 --2- 172.21.15.107:0/1491977484 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f3f140a2d00 0x7f3f140a51c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.098+0000 7f3f2af12640 1 --2- 172.21.15.107:0/1491977484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f24107820 0x7f3f2419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.098+0000 7f3f2af12640 1 --2- 172.21.15.107:0/1491977484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f241021d0 0x7f3f2419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.098+0000 7f3f2af12640 1 --2- 172.21.15.107:0/1491977484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f24101810 0x7f3f2419a300 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.098+0000 7f3f2af12640 1 -- 172.21.15.107:0/1491977484 >> 172.21.15.107:0/1491977484 conn(0x7f3f24078c30 msgr2=0x7f3f2410cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.098+0000 7f3f2af12640 1 -- 172.21.15.107:0/1491977484 shutdown_connections 2024-05-09T08:42:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:07.098+0000 7f3f2af12640 1 -- 172.21.15.107:0/1491977484 wait complete. 2024-05-09T08:42:07.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:07 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3921460355' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:42:07.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:07 smithi107 ceph-mon[30122]: osdmap e100: 12 total, 12 up, 12 in 2024-05-09T08:42:07.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:07 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3921460355' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:42:07.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:07 smithi110 ceph-mon[33622]: osdmap e100: 12 total, 12 up, 12 in 2024-05-09T08:42:07.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:07 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3921460355' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:42:07.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:07 smithi157 ceph-mon[33614]: osdmap e100: 12 total, 12 up, 12 in 2024-05-09T08:42:08.114 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:42:08.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:42:08.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:42:08.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.235+0000 7f755b85d640 1 -- 172.21.15.107:0/2160511945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7554107890 msgr2=0x7f7554107c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.235+0000 7f755b85d640 1 --2- 172.21.15.107:0/2160511945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7554107890 0x7f7554107c70 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f7550009930 tx=0x7f755002a420 comp rx=0 tx=0).stop 2024-05-09T08:42:08.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.235+0000 7f755b85d640 1 -- 172.21.15.107:0/2160511945 shutdown_connections 2024-05-09T08:42:08.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.235+0000 7f755b85d640 1 --2- 172.21.15.107:0/2160511945 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7554102240 0x7f755410e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.235+0000 7f755b85d640 1 --2- 172.21.15.107:0/2160511945 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7554101880 0x7f7554101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.235+0000 7f755b85d640 1 --2- 172.21.15.107:0/2160511945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7554107890 0x7f7554107c70 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.235+0000 7f755b85d640 1 -- 172.21.15.107:0/2160511945 >> 172.21.15.107:0/2160511945 conn(0x7f75540fd390 msgr2=0x7f75540ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:08.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.235+0000 7f755b85d640 1 -- 172.21.15.107:0/2160511945 shutdown_connections 2024-05-09T08:42:08.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.235+0000 7f755b85d640 1 -- 172.21.15.107:0/2160511945 wait complete. 2024-05-09T08:42:08.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.236+0000 7f755b85d640 1 Processor -- start 2024-05-09T08:42:08.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.236+0000 7f755b85d640 1 -- start start 2024-05-09T08:42:08.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.236+0000 7f755b85d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7554101880 0x7f755419a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.236+0000 7f755b85d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7554102240 0x7f755419a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.236+0000 7f755b85d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7554107890 0x7f755419ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.236+0000 7f755b85d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7554111b70 con 0x7f7554107890 2024-05-09T08:42:08.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.236+0000 7f755b85d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f75541119f0 con 0x7f7554102240 2024-05-09T08:42:08.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.237+0000 7f755b85d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7554111cf0 con 0x7f7554101880 2024-05-09T08:42:08.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.237+0000 7f75595f9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7554101880 0x7f755419a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.237+0000 7f7559dfa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7554107890 0x7f755419ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.237+0000 7f75595f9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7554101880 0x7f755419a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:43126/0 (socket says 172.21.15.107:43126) 2024-05-09T08:42:08.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.237+0000 7f75595f9640 1 -- 172.21.15.107:0/2882210679 learned_addr learned my addr 172.21.15.107:0/2882210679 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:08.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.237+0000 7f7559dfa640 1 -- 172.21.15.107:0/2882210679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7554101880 msgr2=0x7f755419a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.237+0000 7f7559dfa640 1 --2- 172.21.15.107:0/2882210679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7554101880 0x7f755419a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.237+0000 7f7559dfa640 1 -- 172.21.15.107:0/2882210679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7554102240 msgr2=0x7f755419a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:42:08.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.237+0000 7f7559dfa640 1 --2- 172.21.15.107:0/2882210679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7554102240 0x7f755419a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.237+0000 7f7559dfa640 1 -- 172.21.15.107:0/2882210679 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f755419f380 con 0x7f7554107890 2024-05-09T08:42:08.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.237+0000 7f75595f9640 1 --2- 172.21.15.107:0/2882210679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7554101880 0x7f755419a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:42:08.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.237+0000 7f7559dfa640 1 --2- 172.21.15.107:0/2882210679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7554107890 0x7f755419ec50 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f754c009fd0 tx=0x7f754c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:08.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.237+0000 7f754a7fc640 1 -- 172.21.15.107:0/2882210679 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754c00ce20 con 0x7f7554107890 2024-05-09T08:42:08.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.238+0000 7f754a7fc640 1 -- 172.21.15.107:0/2882210679 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f754c005020 con 0x7f7554107890 2024-05-09T08:42:08.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.238+0000 7f754a7fc640 1 -- 172.21.15.107:0/2882210679 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754c010690 con 0x7f7554107890 2024-05-09T08:42:08.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.238+0000 7f755b85d640 1 -- 172.21.15.107:0/2882210679 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f755419f670 con 0x7f7554107890 2024-05-09T08:42:08.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.238+0000 7f755b85d640 1 -- 172.21.15.107:0/2882210679 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75541a6f00 con 0x7f7554107890 2024-05-09T08:42:08.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.241+0000 7f755e56a740 1 Processor -- start 2024-05-09T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.241+0000 7f754a7fc640 1 -- 172.21.15.107:0/2882210679 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f754c0051d0 con 0x7f7554107890 2024-05-09T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.241+0000 7f755e56a740 1 -- start start 2024-05-09T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.241+0000 7f755e56a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad80b0 0x55d4ccad63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.241+0000 7f755e56a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccad69c0 0x55d4ccad6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.241+0000 7f755e56a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccae1740 0x55d4ccae3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f755e56a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d4cca3b990 con 0x55d4ccae1740 2024-05-09T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f755e56a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d4cc9e35b0 con 0x55d4ccad69c0 2024-05-09T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f755e56a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d4cca37950 con 0x55d4ccad80b0 2024-05-09T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f7558df8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad80b0 0x55d4ccad63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f7558df8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccae1740 0x55d4ccae3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f7558df8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccae1740 0x55d4ccae3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41736/0 (socket says 172.21.15.107:41736) 2024-05-09T08:42:08.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f7558df8640 1 -- 172.21.15.107:0/1643791755 learned_addr learned my addr 172.21.15.107:0/1643791755 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f75595f9640 1 --2- 172.21.15.107:0/1643791755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccad69c0 0x55d4ccad6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f754a7fc640 1 --2- 172.21.15.107:0/2882210679 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f75400a2d00 0x7f75400a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f7558df8640 1 -- 172.21.15.107:0/1643791755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad80b0 msgr2=0x55d4ccad63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f7558df8640 1 --2- 172.21.15.107:0/1643791755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad80b0 0x55d4ccad63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f7558df8640 1 -- 172.21.15.107:0/1643791755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccad69c0 msgr2=0x55d4ccad6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f7558df8640 1 --2- 172.21.15.107:0/1643791755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccad69c0 0x55d4ccad6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f7558df8640 1 -- 172.21.15.107:0/1643791755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4cc8177e0 con 0x55d4ccae1740 2024-05-09T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f754a7fc640 1 -- 172.21.15.107:0/2882210679 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f754c0cff40 con 0x7f7554107890 2024-05-09T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f7559dfa640 1 --2- 172.21.15.107:0/2882210679 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f75400a2d00 0x7f75400a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.242+0000 7f7558df8640 1 --2- 172.21.15.107:0/1643791755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccae1740 0x55d4ccae3b30 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f75440153a0 tx=0x7f7544000fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b02c94dbbd177e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.243+0000 7f753f7fe640 1 -- 172.21.15.107:0/1643791755 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7544012040 con 0x55d4ccae1740 2024-05-09T08:42:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.243+0000 7f753f7fe640 1 -- 172.21.15.107:0/1643791755 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7544008e30 con 0x55d4ccae1740 2024-05-09T08:42:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.243+0000 7f753f7fe640 1 -- 172.21.15.107:0/1643791755 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7544004a30 con 0x55d4ccae1740 2024-05-09T08:42:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.244+0000 7f755e56a740 1 -- 172.21.15.107:0/1643791755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccae1740 msgr2=0x55d4ccae3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.244+0000 7f7559dfa640 1 --2- 172.21.15.107:0/2882210679 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f75400a2d00 0x7f75400a51c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f754c000f80 tx=0x7f754c005f70 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.244+0000 7f755e56a740 1 --2- 172.21.15.107:0/1643791755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccae1740 0x55d4ccae3b30 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f75440153a0 tx=0x7f7544000fd0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.244+0000 7f755e56a740 1 -- 172.21.15.107:0/1643791755 shutdown_connections 2024-05-09T08:42:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.244+0000 7f755e56a740 1 --2- 172.21.15.107:0/1643791755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccae1740 0x55d4ccae3b30 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.244+0000 7f755e56a740 1 --2- 172.21.15.107:0/1643791755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccad69c0 0x55d4ccad6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.244+0000 7f755e56a740 1 --2- 172.21.15.107:0/1643791755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad80b0 0x55d4ccad63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.244+0000 7f755e56a740 1 -- 172.21.15.107:0/1643791755 >> 172.21.15.107:0/1643791755 conn(0x55d4cca9f0a0 msgr2=0x55d4ccad2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.244+0000 7f755e56a740 1 -- 172.21.15.107:0/1643791755 shutdown_connections 2024-05-09T08:42:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.244+0000 7f755e56a740 1 -- 172.21.15.107:0/1643791755 wait complete. 2024-05-09T08:42:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.244+0000 7f755e56a740 1 -- start start 2024-05-09T08:42:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f755e56a740 1 Processor -- start 2024-05-09T08:42:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f755e56a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad69c0 0x55d4ccade710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f755e56a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad80b0 0x55d4ccadec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f755e56a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccae1740 0x55d4ccadf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f755e56a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d4ccae45d0 con 0x55d4ccad69c0 2024-05-09T08:42:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f755e56a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d4cc9f6b40 con 0x55d4ccae1740 2024-05-09T08:42:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f755e56a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d4ccae4750 con 0x55d4ccad80b0 2024-05-09T08:42:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f755e56a740 10 client.0 _ll_register_callbacks cb 0x55d4cc9eb0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f7558df8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad69c0 0x55d4ccade710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f75595f9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad80b0 0x55d4ccadec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f7558df8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad69c0 0x55d4ccade710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41746/0 (socket says 172.21.15.107:41746) 2024-05-09T08:42:08.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f75595f9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad80b0 0x55d4ccadec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:43140/0 (socket says 172.21.15.107:43140) 2024-05-09T08:42:08.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f7558df8640 1 -- 172.21.15.107:0/744437184 learned_addr learned my addr 172.21.15.107:0/744437184 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:08.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f7558df8640 1 --2- 172.21.15.107:0/744437184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccae1740 0x55d4ccadf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f75595f9640 1 -- 172.21.15.107:0/744437184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccae1740 msgr2=0x55d4ccadf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:08.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f75595f9640 1 --2- 172.21.15.107:0/744437184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccae1740 0x55d4ccadf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f75595f9640 1 -- 172.21.15.107:0/744437184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad69c0 msgr2=0x55d4ccade710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:08.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.245+0000 7f75595f9640 1 --2- 172.21.15.107:0/744437184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad69c0 0x55d4ccade710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.246+0000 7f75595f9640 1 -- 172.21.15.107:0/744437184 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4cc877f70 con 0x55d4ccad80b0 2024-05-09T08:42:08.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.246+0000 7f75595f9640 1 --2- 172.21.15.107:0/744437184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad80b0 0x55d4ccadec50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7550004770 tx=0x7f75500318a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd3fd966e0b862ae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:08.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.246+0000 7f753cff9640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:42:08.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.246+0000 7f753cff9640 1 -- 172.21.15.107:0/744437184 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7550035040 con 0x55d4ccad80b0 2024-05-09T08:42:08.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.246+0000 7f753cff9640 1 -- 172.21.15.107:0/744437184 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f7550005020 con 0x55d4ccad80b0 2024-05-09T08:42:08.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.246+0000 7f755e56a740 1 -- 172.21.15.107:0/744437184 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7544008e30 con 0x55d4ccad80b0 2024-05-09T08:42:08.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.246+0000 7f753cff9640 1 -- 172.21.15.107:0/744437184 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7550050050 con 0x55d4ccad80b0 2024-05-09T08:42:08.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.246+0000 7f755e56a740 10 client.27427 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:08.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.246+0000 7f755e56a740 1 -- 172.21.15.107:0/744437184 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d4cc70f170 con 0x55d4ccad80b0 2024-05-09T08:42:08.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.246+0000 7f753cff9640 1 -- 172.21.15.107:0/744437184 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7544008e30 con 0x55d4ccad80b0 2024-05-09T08:42:08.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.246+0000 7f753cff9640 1 -- 172.21.15.107:0/744437184 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f75500035f0 con 0x55d4ccad80b0 2024-05-09T08:42:08.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.247+0000 7f755e56a740 10 client.27427 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:08.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.247+0000 7f755e56a740 10 client.27427 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:08.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.247+0000 7f755e56a740 20 client.27427 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:08.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.247+0000 7f755e56a740 1 --2- 172.21.15.107:0/744437184 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d4ccaf18f0 0x55d4ccaf3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.247+0000 7f755e56a740 4 client.27427 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:08.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.247+0000 7f755e56a740 1 -- 172.21.15.107:0/744437184 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4cca490c0 con 0x55d4ccaf18f0 2024-05-09T08:42:08.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.247+0000 7f7558df8640 1 --2- 172.21.15.107:0/744437184 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d4ccaf18f0 0x55d4ccaf3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.247+0000 7f753cff9640 10 client.27427 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:08.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.247+0000 7f7558df8640 1 --2- 172.21.15.107:0/744437184 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d4ccaf18f0 0x55d4ccaf3ce0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db8cc98901d6ce4b server_cookie=bee6225b721d91f4 in_seq=0 out_seq=0 2024-05-09T08:42:08.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.248+0000 7f753cff9640 1 -- 172.21.15.107:0/744437184 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d4cca490c0 con 0x55d4ccaf18f0 2024-05-09T08:42:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.248+0000 7f753cff9640 10 client.27427 handle_command_reply: tid=0 2024-05-09T08:42:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.248+0000 7f755e56a740 1 client.27427 shutdown 2024-05-09T08:42:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.248+0000 7f755e56a740 1 -- 172.21.15.107:0/744437184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad80b0 msgr2=0x55d4ccadec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.248+0000 7f755e56a740 1 --2- 172.21.15.107:0/744437184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad80b0 0x55d4ccadec50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7550004770 tx=0x7f75500318a0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.248+0000 7f755e56a740 1 -- 172.21.15.107:0/744437184 shutdown_connections 2024-05-09T08:42:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.248+0000 7f755e56a740 1 --2- 172.21.15.107:0/744437184 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d4ccaf18f0 0x55d4ccaf3ce0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.248+0000 7f755e56a740 1 --2- 172.21.15.107:0/744437184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccae1740 0x55d4ccadf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.248+0000 7f753cff9640 0 client.27427 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.248+0000 7f755e56a740 1 --2- 172.21.15.107:0/744437184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad80b0 0x55d4ccadec50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.248+0000 7f755e56a740 1 --2- 172.21.15.107:0/744437184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad69c0 0x55d4ccade710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.248+0000 7f755e56a740 1 -- 172.21.15.107:0/744437184 >> 172.21.15.107:0/744437184 conn(0x55d4cca9f0a0 msgr2=0x55d4ccad8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.248+0000 7f755e56a740 1 -- 172.21.15.107:0/744437184 shutdown_connections 2024-05-09T08:42:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.248+0000 7f755e56a740 1 -- 172.21.15.107:0/744437184 wait complete. 2024-05-09T08:42:08.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.248+0000 7f755e56a740 20 client.27427 trim_cache size 0 max 16384 2024-05-09T08:42:08.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.268+0000 7f755e56a740 1 Processor -- start 2024-05-09T08:42:08.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.268+0000 7f755e56a740 1 -- start start 2024-05-09T08:42:08.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.268+0000 7f755e56a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccad69c0 0x55d4ccaed4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.268+0000 7f755e56a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad80b0 0x55d4ccad1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.268+0000 7f755e56a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccae1740 0x55d4ccadede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.268+0000 7f755e56a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f75541b3850 con 0x55d4ccad80b0 2024-05-09T08:42:08.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.268+0000 7f755e56a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f75541b1120 con 0x55d4ccad69c0 2024-05-09T08:42:08.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.268+0000 7f755e56a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f75541aef90 con 0x55d4ccae1740 2024-05-09T08:42:08.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.268+0000 7f75595f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad80b0 0x55d4ccad1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.268+0000 7f75595f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad80b0 0x55d4ccad1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41760/0 (socket says 172.21.15.107:41760) 2024-05-09T08:42:08.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.268+0000 7f75595f9640 1 -- 172.21.15.107:0/4039503142 learned_addr learned my addr 172.21.15.107:0/4039503142 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:08.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.268+0000 7f7558df8640 1 --2- 172.21.15.107:0/4039503142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccad69c0 0x55d4ccaed4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.268+0000 7f7558df8640 1 --2- 172.21.15.107:0/4039503142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccae1740 0x55d4ccadede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.268+0000 7f75595f9640 1 -- 172.21.15.107:0/4039503142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccae1740 msgr2=0x55d4ccadede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:08.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.269+0000 7f75595f9640 1 --2- 172.21.15.107:0/4039503142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccae1740 0x55d4ccadede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.269+0000 7f75595f9640 1 -- 172.21.15.107:0/4039503142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccad69c0 msgr2=0x55d4ccaed4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:08.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.269+0000 7f75595f9640 1 --2- 172.21.15.107:0/4039503142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccad69c0 0x55d4ccaed4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.269+0000 7f75595f9640 1 -- 172.21.15.107:0/4039503142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4ccad2eb0 con 0x55d4ccad80b0 2024-05-09T08:42:08.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.269+0000 7f7558df8640 1 --2- 172.21.15.107:0/4039503142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccad69c0 0x55d4ccaed4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:42:08.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.269+0000 7f75595f9640 1 --2- 172.21.15.107:0/4039503142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad80b0 0x55d4ccad1e60 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f7550000c00 tx=0x7f755002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f91835ff685a83 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:08.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.269+0000 7f753effd640 1 -- 172.21.15.107:0/4039503142 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7550035030 con 0x55d4ccad80b0 2024-05-09T08:42:08.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.269+0000 7f753effd640 1 -- 172.21.15.107:0/4039503142 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7550004310 con 0x55d4ccad80b0 2024-05-09T08:42:08.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.269+0000 7f753effd640 1 -- 172.21.15.107:0/4039503142 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75500035f0 con 0x55d4ccad80b0 2024-05-09T08:42:08.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.270+0000 7f755e56a740 1 -- 172.21.15.107:0/4039503142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad80b0 msgr2=0x55d4ccad1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.270+0000 7f755e56a740 1 --2- 172.21.15.107:0/4039503142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad80b0 0x55d4ccad1e60 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f7550000c00 tx=0x7f755002a420 comp rx=0 tx=0).stop 2024-05-09T08:42:08.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.270+0000 7f755e56a740 1 -- 172.21.15.107:0/4039503142 shutdown_connections 2024-05-09T08:42:08.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.270+0000 7f755e56a740 1 --2- 172.21.15.107:0/4039503142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccae1740 0x55d4ccadede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.270+0000 7f755e56a740 1 --2- 172.21.15.107:0/4039503142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad80b0 0x55d4ccad1e60 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.270+0000 7f755e56a740 1 --2- 172.21.15.107:0/4039503142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccad69c0 0x55d4ccaed4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.270+0000 7f755e56a740 1 -- 172.21.15.107:0/4039503142 >> 172.21.15.107:0/4039503142 conn(0x55d4cca9f0a0 msgr2=0x55d4ccade6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:08.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.270+0000 7f755e56a740 1 -- 172.21.15.107:0/4039503142 shutdown_connections 2024-05-09T08:42:08.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.270+0000 7f755e56a740 1 -- 172.21.15.107:0/4039503142 wait complete. 2024-05-09T08:42:08.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.270+0000 7f755e56a740 1 -- start start 2024-05-09T08:42:08.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.270+0000 7f755e56a740 1 Processor -- start 2024-05-09T08:42:08.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f755e56a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad69c0 0x55d4ccae9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f755e56a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad80b0 0x55d4ccae69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f755e56a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccae1740 0x55d4ccae70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f755e56a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f75541af5e0 con 0x55d4ccad80b0 2024-05-09T08:42:08.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f755e56a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f75541b0a50 con 0x55d4ccae1740 2024-05-09T08:42:08.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f755e56a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f75541afcb0 con 0x55d4ccad69c0 2024-05-09T08:42:08.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f755e56a740 10 client.0 _ll_register_callbacks cb 0x7f7550032eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:08.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f75595f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad80b0 0x55d4ccae69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f75595f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad80b0 0x55d4ccae69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41772/0 (socket says 172.21.15.107:41772) 2024-05-09T08:42:08.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f75595f9640 1 -- 172.21.15.107:0/3832569467 learned_addr learned my addr 172.21.15.107:0/3832569467 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:08.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f7558df8640 1 --2- 172.21.15.107:0/3832569467 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad69c0 0x55d4ccae9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f7558df8640 1 --2- 172.21.15.107:0/3832569467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccae1740 0x55d4ccae70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f75595f9640 1 -- 172.21.15.107:0/3832569467 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad69c0 msgr2=0x55d4ccae9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:08.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f75595f9640 1 --2- 172.21.15.107:0/3832569467 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad69c0 0x55d4ccae9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f75595f9640 1 -- 172.21.15.107:0/3832569467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccae1740 msgr2=0x55d4ccae70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:08.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f75595f9640 1 --2- 172.21.15.107:0/3832569467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccae1740 0x55d4ccae70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f75595f9640 1 -- 172.21.15.107:0/3832569467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4ccb57e30 con 0x55d4ccad80b0 2024-05-09T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f75595f9640 1 --2- 172.21.15.107:0/3832569467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad80b0 0x55d4ccae69b0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f7550032fa0 tx=0x7f755002aae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbf38e01f614e08f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f753cff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f753cff9640 1 -- 172.21.15.107:0/3832569467 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7550061060 con 0x55d4ccad80b0 2024-05-09T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f753cff9640 1 -- 172.21.15.107:0/3832569467 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f75500035f0 con 0x55d4ccad80b0 2024-05-09T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.271+0000 7f755e56a740 1 -- 172.21.15.107:0/3832569467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7550004310 con 0x55d4ccad80b0 2024-05-09T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.272+0000 7f753cff9640 1 -- 172.21.15.107:0/3832569467 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7550068050 con 0x55d4ccad80b0 2024-05-09T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.272+0000 7f755e56a740 10 client.18246 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.272+0000 7f755e56a740 1 -- 172.21.15.107:0/3832569467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d4ccb58f20 con 0x55d4ccad80b0 2024-05-09T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.272+0000 7f753cff9640 1 -- 172.21.15.107:0/3832569467 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7550004310 con 0x55d4ccad80b0 2024-05-09T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.272+0000 7f753cff9640 1 -- 172.21.15.107:0/3832569467 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7550005160 con 0x55d4ccad80b0 2024-05-09T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.272+0000 7f755e56a740 10 client.18246 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.272+0000 7f755e56a740 10 client.18246 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:08.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.272+0000 7f755e56a740 20 client.18246 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:08.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.272+0000 7f755e56a740 1 --2- 172.21.15.107:0/3832569467 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d4ccb595b0 0x55d4ccb5b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.272+0000 7f755e56a740 4 client.18246 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:42:08.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.272+0000 7f755e56a740 1 -- 172.21.15.107:0/3832569467 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d4cca43410 con 0x55d4ccb595b0 2024-05-09T08:42:08.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.272+0000 7f7558df8640 1 --2- 172.21.15.107:0/3832569467 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d4ccb595b0 0x55d4ccb5b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.273+0000 7f753cff9640 10 client.18246 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:08.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.273+0000 7f7558df8640 1 --2- 172.21.15.107:0/3832569467 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d4ccb595b0 0x55d4ccb5b9d0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c28e44830a48c00 server_cookie=e2ae55833c2a7b85 in_seq=0 out_seq=0 2024-05-09T08:42:08.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.274+0000 7f753cff9640 1 -- 172.21.15.107:0/3832569467 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d4cca43410 con 0x55d4ccb595b0 2024-05-09T08:42:08.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.274+0000 7f753cff9640 10 client.18246 handle_command_reply: tid=0 2024-05-09T08:42:08.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.274+0000 7f755e56a740 1 client.18246 shutdown 2024-05-09T08:42:08.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.274+0000 7f755e56a740 1 -- 172.21.15.107:0/3832569467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad80b0 msgr2=0x55d4ccae69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.274+0000 7f755e56a740 1 --2- 172.21.15.107:0/3832569467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad80b0 0x55d4ccae69b0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f7550032fa0 tx=0x7f755002aae0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.274+0000 7f755e56a740 1 -- 172.21.15.107:0/3832569467 shutdown_connections 2024-05-09T08:42:08.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.274+0000 7f755e56a740 1 --2- 172.21.15.107:0/3832569467 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d4ccb595b0 0x55d4ccb5b9d0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.274+0000 7f755e56a740 1 --2- 172.21.15.107:0/3832569467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d4ccae1740 0x55d4ccae70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.274+0000 7f755e56a740 1 --2- 172.21.15.107:0/3832569467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d4ccad80b0 0x55d4ccae69b0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.274+0000 7f755e56a740 1 --2- 172.21.15.107:0/3832569467 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4ccad69c0 0x55d4ccae9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.274+0000 7f755e56a740 1 -- 172.21.15.107:0/3832569467 >> 172.21.15.107:0/3832569467 conn(0x55d4cca9f0a0 msgr2=0x55d4ccadef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.274+0000 7f753cff9640 0 client.18246 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.274+0000 7f755e56a740 1 -- 172.21.15.107:0/3832569467 shutdown_connections 2024-05-09T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.274+0000 7f755e56a740 1 -- 172.21.15.107:0/3832569467 wait complete. 2024-05-09T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.274+0000 7f755e56a740 20 client.18246 trim_cache size 0 max 16384 2024-05-09T08:42:08.280 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:42:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:42:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "5c91f14c-f432-4763-af5b-f30c806455d2", 2024-05-09T08:42:08.281 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:42:08.281 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.275+0000 7f755b85d640 1 -- 172.21.15.107:0/2882210679 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f75400a2d00 msgr2=0x7f75400a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.275+0000 7f755b85d640 1 --2- 172.21.15.107:0/2882210679 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f75400a2d00 0x7f75400a51c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f754c000f80 tx=0x7f754c005f70 comp rx=0 tx=0).stop 2024-05-09T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.275+0000 7f755b85d640 1 -- 172.21.15.107:0/2882210679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7554107890 msgr2=0x7f755419ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.275+0000 7f755b85d640 1 --2- 172.21.15.107:0/2882210679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7554107890 0x7f755419ec50 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f754c009fd0 tx=0x7f754c00ef10 comp rx=0 tx=0).stop 2024-05-09T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.275+0000 7f755b85d640 1 -- 172.21.15.107:0/2882210679 shutdown_connections 2024-05-09T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.275+0000 7f755b85d640 1 --2- 172.21.15.107:0/2882210679 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f75400a2d00 0x7f75400a51c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.275+0000 7f755b85d640 1 --2- 172.21.15.107:0/2882210679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7554107890 0x7f755419ec50 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.275+0000 7f755b85d640 1 --2- 172.21.15.107:0/2882210679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7554102240 0x7f755419a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.275+0000 7f755b85d640 1 --2- 172.21.15.107:0/2882210679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7554101880 0x7f755419a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.275+0000 7f755b85d640 1 -- 172.21.15.107:0/2882210679 >> 172.21.15.107:0/2882210679 conn(0x7f75540fd390 msgr2=0x7f755410d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.275+0000 7f755b85d640 1 -- 172.21.15.107:0/2882210679 shutdown_connections 2024-05-09T08:42:08.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.275+0000 7f755b85d640 1 -- 172.21.15.107:0/2882210679 wait complete. 2024-05-09T08:42:08.287 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:42:08.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:42:08.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:08 smithi110 ceph-mon[33622]: pgmap v272: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 37 MiB/s wr, 48 op/s 2024-05-09T08:42:08.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.403+0000 7fd4bed3b640 1 -- 172.21.15.107:0/2748988471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4b8101800 msgr2=0x7fd4b8101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.403+0000 7fd4bed3b640 1 --2- 172.21.15.107:0/2748988471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4b8101800 0x7fd4b8101c80 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4009930 tx=0x7fd4a402a420 comp rx=0 tx=0).stop 2024-05-09T08:42:08.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.403+0000 7fd4bed3b640 1 -- 172.21.15.107:0/2748988471 shutdown_connections 2024-05-09T08:42:08.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.403+0000 7fd4bed3b640 1 --2- 172.21.15.107:0/2748988471 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4b81021c0 0x7fd4b810e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.403+0000 7fd4bed3b640 1 --2- 172.21.15.107:0/2748988471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4b8101800 0x7fd4b8101c80 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.403+0000 7fd4bed3b640 1 --2- 172.21.15.107:0/2748988471 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4b8107810 0x7fd4b8107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.403+0000 7fd4bed3b640 1 -- 172.21.15.107:0/2748988471 >> 172.21.15.107:0/2748988471 conn(0x7fd4b8078c80 msgr2=0x7fd4b8079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:08.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.403+0000 7fd4bed3b640 1 -- 172.21.15.107:0/2748988471 shutdown_connections 2024-05-09T08:42:08.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.403+0000 7fd4bed3b640 1 -- 172.21.15.107:0/2748988471 wait complete. 2024-05-09T08:42:08.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.404+0000 7fd4bed3b640 1 Processor -- start 2024-05-09T08:42:08.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.404+0000 7fd4bed3b640 1 -- start start 2024-05-09T08:42:08.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.404+0000 7fd4bed3b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4b8101800 0x7fd4b8198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.404+0000 7fd4bed3b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4b81021c0 0x7fd4b8198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.404+0000 7fd4bed3b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4b8107810 0x7fd4b819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.404+0000 7fd4bed3b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b8111ae0 con 0x7fd4b8107810 2024-05-09T08:42:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.404+0000 7fd4bed3b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b8111960 con 0x7fd4b81021c0 2024-05-09T08:42:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.404+0000 7fd4bed3b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b8111c60 con 0x7fd4b8101800 2024-05-09T08:42:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.404+0000 7fd4bd2d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4b8107810 0x7fd4b819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.404+0000 7fd4bd2d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4b8107810 0x7fd4b819c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41784/0 (socket says 172.21.15.107:41784) 2024-05-09T08:42:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.404+0000 7fd4bd2d8640 1 -- 172.21.15.107:0/634676302 learned_addr learned my addr 172.21.15.107:0/634676302 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.405+0000 7fd4bcad7640 1 --2- 172.21.15.107:0/634676302 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4b8101800 0x7fd4b8198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.405+0000 7fd4b7fff640 1 --2- 172.21.15.107:0/634676302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4b81021c0 0x7fd4b8198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.405+0000 7fd4bd2d8640 1 -- 172.21.15.107:0/634676302 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4b8101800 msgr2=0x7fd4b8198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.405+0000 7fd4bd2d8640 1 --2- 172.21.15.107:0/634676302 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4b8101800 0x7fd4b8198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.405+0000 7fd4bd2d8640 1 -- 172.21.15.107:0/634676302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4b81021c0 msgr2=0x7fd4b8198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.405+0000 7fd4bd2d8640 1 --2- 172.21.15.107:0/634676302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4b81021c0 0x7fd4b8198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.405+0000 7fd4bd2d8640 1 -- 172.21.15.107:0/634676302 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4b819d010 con 0x7fd4b8107810 2024-05-09T08:42:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.405+0000 7fd4b7fff640 1 --2- 172.21.15.107:0/634676302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4b81021c0 0x7fd4b8198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:42:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.405+0000 7fd4bcad7640 1 --2- 172.21.15.107:0/634676302 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4b8101800 0x7fd4b8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:42:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.405+0000 7fd4bd2d8640 1 --2- 172.21.15.107:0/634676302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4b8107810 0x7fd4b819c8e0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fd4b000eb10 tx=0x7fd4b000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.405+0000 7fd4b5ffb640 1 -- 172.21.15.107:0/634676302 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0004b20 con 0x7fd4b8107810 2024-05-09T08:42:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.405+0000 7fd4bed3b640 1 -- 172.21.15.107:0/634676302 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4b819d300 con 0x7fd4b8107810 2024-05-09T08:42:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.405+0000 7fd4b5ffb640 1 -- 172.21.15.107:0/634676302 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4b0004cc0 con 0x7fd4b8107810 2024-05-09T08:42:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.405+0000 7fd4b5ffb640 1 -- 172.21.15.107:0/634676302 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0010690 con 0x7fd4b8107810 2024-05-09T08:42:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.405+0000 7fd4bed3b640 1 -- 172.21.15.107:0/634676302 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4b81a4b90 con 0x7fd4b8107810 2024-05-09T08:42:08.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.407+0000 7fd4b5ffb640 1 -- 172.21.15.107:0/634676302 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd4b0005020 con 0x7fd4b8107810 2024-05-09T08:42:08.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.408+0000 7fd4b5ffb640 1 --2- 172.21.15.107:0/634676302 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4a80a2d80 0x7fd4a80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.408+0000 7fd4c1a49740 1 Processor -- start 2024-05-09T08:42:08.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.408+0000 7fd4c1a49740 1 -- start start 2024-05-09T08:42:08.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.408+0000 7fd4bcad7640 1 --2- 172.21.15.107:0/634676302 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4a80a2d80 0x7fd4a80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.408+0000 7fd4b5ffb640 1 -- 172.21.15.107:0/634676302 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fd4b00cfe80 con 0x7fd4b8107810 2024-05-09T08:42:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.408+0000 7fd4c1a49740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f23fe280 0x5607f23fe660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4c1a49740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f23feba0 0x5607f2408d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4c1a49740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f2409270 0x5607f240b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4c1a49740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5607f2363d00 con 0x5607f23feba0 2024-05-09T08:42:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4c1a49740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5607f230b4c0 con 0x5607f23fe280 2024-05-09T08:42:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4c1a49740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5607f235fcc0 con 0x5607f2409270 2024-05-09T08:42:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4b7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f23feba0 0x5607f2408d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4b7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f23feba0 0x5607f2408d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41798/0 (socket says 172.21.15.107:41798) 2024-05-09T08:42:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4b7fff640 1 -- 172.21.15.107:0/3549166638 learned_addr learned my addr 172.21.15.107:0/3549166638 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4bcad7640 1 --2- 172.21.15.107:0/3549166638 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f23fe280 0x5607f23fe660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4bcad7640 1 --2- 172.21.15.107:0/634676302 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4a80a2d80 0x7fd4a80a5240 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac005e10 tx=0x7fd4ac005da0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:08.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4bd2d8640 1 --2- 172.21.15.107:0/3549166638 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f2409270 0x5607f240b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4b7fff640 1 -- 172.21.15.107:0/3549166638 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f2409270 msgr2=0x5607f240b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:08.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4b7fff640 1 --2- 172.21.15.107:0/3549166638 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f2409270 0x5607f240b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4b7fff640 1 -- 172.21.15.107:0/3549166638 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f23fe280 msgr2=0x5607f23fe660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:08.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4b7fff640 1 --2- 172.21.15.107:0/3549166638 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f23fe280 0x5607f23fe660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4b7fff640 1 -- 172.21.15.107:0/3549166638 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607f21417d0 con 0x5607f23feba0 2024-05-09T08:42:08.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4b7fff640 1 --2- 172.21.15.107:0/3549166638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f23feba0 0x5607f2408d30 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fd4a402aa10 tx=0x7fd4a40042d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359149f154d6ae8e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:08.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4a2ffd640 1 -- 172.21.15.107:0/3549166638 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4035030 con 0x5607f23feba0 2024-05-09T08:42:08.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.409+0000 7fd4a2ffd640 1 -- 172.21.15.107:0/3549166638 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4a40044c0 con 0x5607f23feba0 2024-05-09T08:42:08.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.410+0000 7fd4a2ffd640 1 -- 172.21.15.107:0/3549166638 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4003850 con 0x5607f23feba0 2024-05-09T08:42:08.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.410+0000 7fd4c1a49740 1 -- 172.21.15.107:0/3549166638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f23feba0 msgr2=0x5607f2408d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.410+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/3549166638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f23feba0 0x5607f2408d30 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fd4a402aa10 tx=0x7fd4a40042d0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.410+0000 7fd4c1a49740 1 -- 172.21.15.107:0/3549166638 shutdown_connections 2024-05-09T08:42:08.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.410+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/3549166638 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f2409270 0x5607f240b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.410+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/3549166638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f23feba0 0x5607f2408d30 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.410+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/3549166638 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f23fe280 0x5607f23fe660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.410+0000 7fd4c1a49740 1 -- 172.21.15.107:0/3549166638 >> 172.21.15.107:0/3549166638 conn(0x5607f23fa5a0 msgr2=0x5607f23fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.410+0000 7fd4c1a49740 1 -- 172.21.15.107:0/3549166638 shutdown_connections 2024-05-09T08:42:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.410+0000 7fd4c1a49740 1 -- 172.21.15.107:0/3549166638 wait complete. 2024-05-09T08:42:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.411+0000 7fd4c1a49740 1 -- start start 2024-05-09T08:42:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.411+0000 7fd4c1a49740 1 Processor -- start 2024-05-09T08:42:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.411+0000 7fd4c1a49740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f23fe280 0x5607f24061b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.411+0000 7fd4c1a49740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f23feba0 0x5607f24066f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.411+0000 7fd4c1a49740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f2409270 0x5607f2406c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.411+0000 7fd4c1a49740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5607f240c000 con 0x5607f2409270 2024-05-09T08:42:08.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.411+0000 7fd4c1a49740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5607f231eb90 con 0x5607f23feba0 2024-05-09T08:42:08.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.411+0000 7fd4c1a49740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5607f240c180 con 0x5607f23fe280 2024-05-09T08:42:08.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.411+0000 7fd4c1a49740 10 client.0 _ll_register_callbacks cb 0x5607f235ada0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:08.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.411+0000 7fd4bd2d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f2409270 0x5607f2406c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.411+0000 7fd4bd2d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f2409270 0x5607f2406c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41802/0 (socket says 172.21.15.107:41802) 2024-05-09T08:42:08.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.411+0000 7fd4bd2d8640 1 -- 172.21.15.107:0/2067144127 learned_addr learned my addr 172.21.15.107:0/2067144127 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:08.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.411+0000 7fd4bcad7640 1 --2- 172.21.15.107:0/2067144127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f23fe280 0x5607f24061b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.412+0000 7fd4b7fff640 1 --2- 172.21.15.107:0/2067144127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f23feba0 0x5607f24066f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.412+0000 7fd4bd2d8640 1 -- 172.21.15.107:0/2067144127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f23fe280 msgr2=0x5607f24061b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:08.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.412+0000 7fd4bd2d8640 1 --2- 172.21.15.107:0/2067144127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f23fe280 0x5607f24061b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.412+0000 7fd4bd2d8640 1 -- 172.21.15.107:0/2067144127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f23feba0 msgr2=0x5607f24066f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:08.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.412+0000 7fd4bd2d8640 1 --2- 172.21.15.107:0/2067144127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f23feba0 0x5607f24066f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.412+0000 7fd4bd2d8640 1 -- 172.21.15.107:0/2067144127 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607f20773f0 con 0x5607f2409270 2024-05-09T08:42:08.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.412+0000 7fd4b7fff640 1 --2- 172.21.15.107:0/2067144127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f23feba0 0x5607f24066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:42:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.412+0000 7fd4bd2d8640 1 --2- 172.21.15.107:0/2067144127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f2409270 0x5607f2406c30 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00d01a0 tx=0x7fd4b009bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c7a0e1db0b767 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.412+0000 7fd497fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.412+0000 7fd497fff640 1 -- 172.21.15.107:0/2067144127 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b00a03a0 con 0x5607f2409270 2024-05-09T08:42:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.412+0000 7fd497fff640 1 -- 172.21.15.107:0/2067144127 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fd4b0098660 con 0x5607f2409270 2024-05-09T08:42:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.412+0000 7fd4c1a49740 1 -- 172.21.15.107:0/2067144127 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5607f2371430 con 0x5607f2409270 2024-05-09T08:42:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.412+0000 7fd497fff640 1 -- 172.21.15.107:0/2067144127 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd4b00a4c60 con 0x5607f2409270 2024-05-09T08:42:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.412+0000 7fd4c1a49740 10 client.18258 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.412+0000 7fd4c1a49740 1 -- 172.21.15.107:0/2067144127 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5607f20370f0 con 0x5607f2409270 2024-05-09T08:42:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.412+0000 7fd497fff640 1 -- 172.21.15.107:0/2067144127 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5607f2371430 con 0x5607f2409270 2024-05-09T08:42:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.413+0000 7fd497fff640 1 -- 172.21.15.107:0/2067144127 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd4b009c250 con 0x5607f2409270 2024-05-09T08:42:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.413+0000 7fd4c1a49740 10 client.18258 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.413+0000 7fd4c1a49740 10 client.18258 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.413+0000 7fd4c1a49740 20 client.18258 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.413+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/2067144127 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607f2419520 0x5607f241b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.413+0000 7fd4c1a49740 4 client.18258 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.413+0000 7fd4c1a49740 1 -- 172.21.15.107:0/2067144127 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607f236b780 con 0x5607f2419520 2024-05-09T08:42:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.413+0000 7fd4bcad7640 1 --2- 172.21.15.107:0/2067144127 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607f2419520 0x5607f241b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.413+0000 7fd497fff640 10 client.18258 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.413+0000 7fd4bcad7640 1 --2- 172.21.15.107:0/2067144127 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607f2419520 0x5607f241b960 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43761e0cf18660e8 server_cookie=719be34b9c6219c in_seq=0 out_seq=0 2024-05-09T08:42:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.414+0000 7fd497fff640 1 -- 172.21.15.107:0/2067144127 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5607f236b780 con 0x5607f2419520 2024-05-09T08:42:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.414+0000 7fd497fff640 10 client.18258 handle_command_reply: tid=0 2024-05-09T08:42:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.414+0000 7fd4c1a49740 1 client.18258 shutdown 2024-05-09T08:42:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.414+0000 7fd4c1a49740 1 -- 172.21.15.107:0/2067144127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f2409270 msgr2=0x5607f2406c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.414+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/2067144127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f2409270 0x5607f2406c30 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00d01a0 tx=0x7fd4b009bc20 comp rx=0 tx=0).stop 2024-05-09T08:42:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.414+0000 7fd4c1a49740 1 -- 172.21.15.107:0/2067144127 shutdown_connections 2024-05-09T08:42:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.414+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/2067144127 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607f2419520 0x5607f241b960 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.414+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/2067144127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f2409270 0x5607f2406c30 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.414+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/2067144127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f23feba0 0x5607f24066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.414+0000 7fd497fff640 0 client.18258 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:08.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.414+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/2067144127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f23fe280 0x5607f24061b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.414+0000 7fd4c1a49740 1 -- 172.21.15.107:0/2067144127 >> 172.21.15.107:0/2067144127 conn(0x5607f23fa5a0 msgr2=0x5607f23f97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:08.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.414+0000 7fd4c1a49740 1 -- 172.21.15.107:0/2067144127 shutdown_connections 2024-05-09T08:42:08.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.414+0000 7fd4c1a49740 1 -- 172.21.15.107:0/2067144127 wait complete. 2024-05-09T08:42:08.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.414+0000 7fd4c1a49740 20 client.18258 trim_cache size 0 max 16384 2024-05-09T08:42:08.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.434+0000 7fd4c1a49740 1 Processor -- start 2024-05-09T08:42:08.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.434+0000 7fd4c1a49740 1 -- start start 2024-05-09T08:42:08.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.434+0000 7fd4c1a49740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f23fe280 0x5607f2414ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.434+0000 7fd4c1a49740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f23feba0 0x5607f2406a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.434+0000 7fd4c1a49740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f2409270 0x5607f2421280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.434+0000 7fd4c1a49740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b81b1400 con 0x5607f2409270 2024-05-09T08:42:08.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.434+0000 7fd4c1a49740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b81aecd0 con 0x5607f23fe280 2024-05-09T08:42:08.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.434+0000 7fd4c1a49740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b81acb40 con 0x5607f23feba0 2024-05-09T08:42:08.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.435+0000 7fd4bd2d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f2409270 0x5607f2421280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.435+0000 7fd4bd2d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f2409270 0x5607f2421280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41804/0 (socket says 172.21.15.107:41804) 2024-05-09T08:42:08.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.435+0000 7fd4bd2d8640 1 -- 172.21.15.107:0/1227874782 learned_addr learned my addr 172.21.15.107:0/1227874782 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.435+0000 7fd4b7fff640 1 --2- 172.21.15.107:0/1227874782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f23feba0 0x5607f2406a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.435+0000 7fd4bd2d8640 1 -- 172.21.15.107:0/1227874782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f23feba0 msgr2=0x5607f2406a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.435+0000 7fd4bd2d8640 1 --2- 172.21.15.107:0/1227874782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f23feba0 0x5607f2406a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.435+0000 7fd4bd2d8640 1 -- 172.21.15.107:0/1227874782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f23fe280 msgr2=0x5607f2414ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.435+0000 7fd4bcad7640 1 --2- 172.21.15.107:0/1227874782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f23fe280 0x5607f2414ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.435+0000 7fd4bd2d8640 1 --2- 172.21.15.107:0/1227874782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f23fe280 0x5607f2414ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.435+0000 7fd4bd2d8640 1 -- 172.21.15.107:0/1227874782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607f23ff0a0 con 0x5607f2409270 2024-05-09T08:42:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.435+0000 7fd4bd2d8640 1 --2- 172.21.15.107:0/1227874782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f2409270 0x5607f2421280 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00989c0 tx=0x7fd4b009a500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55120828d19881c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.435+0000 7fd4a27fc640 1 -- 172.21.15.107:0/1227874782 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b00a03a0 con 0x5607f2409270 2024-05-09T08:42:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.435+0000 7fd4a27fc640 1 -- 172.21.15.107:0/1227874782 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4b00850e0 con 0x5607f2409270 2024-05-09T08:42:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.435+0000 7fd4a27fc640 1 -- 172.21.15.107:0/1227874782 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b009c250 con 0x5607f2409270 2024-05-09T08:42:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.436+0000 7fd4c1a49740 1 -- 172.21.15.107:0/1227874782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f2409270 msgr2=0x5607f2421280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.436+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/1227874782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f2409270 0x5607f2421280 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00989c0 tx=0x7fd4b009a500 comp rx=0 tx=0).stop 2024-05-09T08:42:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.436+0000 7fd4c1a49740 1 -- 172.21.15.107:0/1227874782 shutdown_connections 2024-05-09T08:42:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.436+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/1227874782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f2409270 0x5607f2421280 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.436+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/1227874782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f23feba0 0x5607f2406a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.436+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/1227874782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f23fe280 0x5607f2414ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.436+0000 7fd4c1a49740 1 -- 172.21.15.107:0/1227874782 >> 172.21.15.107:0/1227874782 conn(0x5607f23fa5a0 msgr2=0x5607f2408270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:08.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.436+0000 7fd4c1a49740 1 -- 172.21.15.107:0/1227874782 shutdown_connections 2024-05-09T08:42:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.436+0000 7fd4c1a49740 1 -- 172.21.15.107:0/1227874782 wait complete. 2024-05-09T08:42:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.436+0000 7fd4c1a49740 1 -- start start 2024-05-09T08:42:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.436+0000 7fd4c1a49740 1 Processor -- start 2024-05-09T08:42:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4c1a49740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f23fe280 0x5607f2402f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4c1a49740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f23feba0 0x5607f2403490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4c1a49740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f2409270 0x5607f24039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4c1a49740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b81ad190 con 0x5607f23feba0 2024-05-09T08:42:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4c1a49740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b81ae600 con 0x5607f2409270 2024-05-09T08:42:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4c1a49740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b81ad860 con 0x5607f23fe280 2024-05-09T08:42:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4c1a49740 10 client.0 _ll_register_callbacks cb 0x7fd4b0004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4b7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f23feba0 0x5607f2403490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4b7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f23feba0 0x5607f2403490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41820/0 (socket says 172.21.15.107:41820) 2024-05-09T08:42:08.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4b7fff640 1 -- 172.21.15.107:0/3773248052 learned_addr learned my addr 172.21.15.107:0/3773248052 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4bd2d8640 1 --2- 172.21.15.107:0/3773248052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f2409270 0x5607f24039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4bcad7640 1 --2- 172.21.15.107:0/3773248052 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f23fe280 0x5607f2402f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4b7fff640 1 -- 172.21.15.107:0/3773248052 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f23fe280 msgr2=0x5607f2402f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4b7fff640 1 --2- 172.21.15.107:0/3773248052 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f23fe280 0x5607f2402f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4b7fff640 1 -- 172.21.15.107:0/3773248052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f2409270 msgr2=0x5607f24039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4b7fff640 1 --2- 172.21.15.107:0/3773248052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f2409270 0x5607f24039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4b7fff640 1 -- 172.21.15.107:0/3773248052 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607f247ebf0 con 0x5607f23feba0 2024-05-09T08:42:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd4b7fff640 1 --2- 172.21.15.107:0/3773248052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f23feba0 0x5607f2403490 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fd4a402aa10 tx=0x7fd4a40056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30adce0f02e12f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.437+0000 7fd497fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.438+0000 7fd497fff640 1 -- 172.21.15.107:0/3773248052 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4035070 con 0x5607f23feba0 2024-05-09T08:42:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.438+0000 7fd497fff640 1 -- 172.21.15.107:0/3773248052 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fd4a40048d0 con 0x5607f23feba0 2024-05-09T08:42:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.438+0000 7fd4c1a49740 1 -- 172.21.15.107:0/3773248052 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4b00850e0 con 0x5607f23feba0 2024-05-09T08:42:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.438+0000 7fd497fff640 1 -- 172.21.15.107:0/3773248052 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd4a4004ad0 con 0x5607f23feba0 2024-05-09T08:42:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.438+0000 7fd4c1a49740 10 client.18264 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.438+0000 7fd4c1a49740 1 -- 172.21.15.107:0/3773248052 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5607f247fea0 con 0x5607f23feba0 2024-05-09T08:42:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.438+0000 7fd497fff640 1 -- 172.21.15.107:0/3773248052 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4b00850e0 con 0x5607f23feba0 2024-05-09T08:42:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.438+0000 7fd497fff640 1 -- 172.21.15.107:0/3773248052 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd4a403b430 con 0x5607f23feba0 2024-05-09T08:42:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.438+0000 7fd4c1a49740 10 client.18264 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.438+0000 7fd4c1a49740 10 client.18264 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.438+0000 7fd4c1a49740 20 client.18264 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.438+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/3773248052 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607f2480260 0x5607f2482650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.438+0000 7fd4c1a49740 4 client.18264 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:42:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.438+0000 7fd4c1a49740 1 -- 172.21.15.107:0/3773248052 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5607f22e86f0 con 0x5607f2480260 2024-05-09T08:42:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.438+0000 7fd4bcad7640 1 --2- 172.21.15.107:0/3773248052 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607f2480260 0x5607f2482650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd497fff640 10 client.18264 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd4bcad7640 1 --2- 172.21.15.107:0/3773248052 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607f2480260 0x5607f2482650 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2fc47b835a39e27 server_cookie=6dde55f60843740d in_seq=0 out_seq=0 2024-05-09T08:42:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd497fff640 1 -- 172.21.15.107:0/3773248052 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5607f22e86f0 con 0x5607f2480260 2024-05-09T08:42:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd497fff640 10 client.18264 handle_command_reply: tid=0 2024-05-09T08:42:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd4c1a49740 1 client.18264 shutdown 2024-05-09T08:42:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd4c1a49740 1 -- 172.21.15.107:0/3773248052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f23feba0 msgr2=0x5607f2403490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/3773248052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f23feba0 0x5607f2403490 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fd4a402aa10 tx=0x7fd4a40056e0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd4c1a49740 1 -- 172.21.15.107:0/3773248052 shutdown_connections 2024-05-09T08:42:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/3773248052 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607f2480260 0x5607f2482650 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/3773248052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607f2409270 0x5607f24039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd497fff640 0 client.18264 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/3773248052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607f23feba0 0x5607f2403490 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd4c1a49740 1 --2- 172.21.15.107:0/3773248052 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607f23fe280 0x5607f2402f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd4c1a49740 1 -- 172.21.15.107:0/3773248052 >> 172.21.15.107:0/3773248052 conn(0x5607f23fa5a0 msgr2=0x5607f23f8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd4c1a49740 1 -- 172.21.15.107:0/3773248052 shutdown_connections 2024-05-09T08:42:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd4c1a49740 1 -- 172.21.15.107:0/3773248052 wait complete. 2024-05-09T08:42:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.439+0000 7fd4c1a49740 20 client.18264 trim_cache size 0 max 16384 2024-05-09T08:42:08.448 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:42:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:42:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:42:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "5c91f14c-f432-4763-af5b-f30c806455d2": { 2024-05-09T08:42:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:42:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "5c91f14c-f432-4763-af5b-f30c806455d2", 2024-05-09T08:42:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:42:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:42:08.448 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:42:08.449 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:42:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.440+0000 7fd4bed3b640 1 -- 172.21.15.107:0/634676302 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4a80a2d80 msgr2=0x7fd4a80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.440+0000 7fd4bed3b640 1 --2- 172.21.15.107:0/634676302 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4a80a2d80 0x7fd4a80a5240 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac005e10 tx=0x7fd4ac005da0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.440+0000 7fd4bed3b640 1 -- 172.21.15.107:0/634676302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4b8107810 msgr2=0x7fd4b819c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.440+0000 7fd4bed3b640 1 --2- 172.21.15.107:0/634676302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4b8107810 0x7fd4b819c8e0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fd4b000eb10 tx=0x7fd4b000c2d0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.440+0000 7fd4bed3b640 1 -- 172.21.15.107:0/634676302 shutdown_connections 2024-05-09T08:42:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.440+0000 7fd4bed3b640 1 --2- 172.21.15.107:0/634676302 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd4a80a2d80 0x7fd4a80a5240 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.440+0000 7fd4bed3b640 1 --2- 172.21.15.107:0/634676302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4b8107810 0x7fd4b819c8e0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.440+0000 7fd4bed3b640 1 --2- 172.21.15.107:0/634676302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4b81021c0 0x7fd4b8198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.440+0000 7fd4bed3b640 1 --2- 172.21.15.107:0/634676302 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4b8101800 0x7fd4b8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.440+0000 7fd4bed3b640 1 -- 172.21.15.107:0/634676302 >> 172.21.15.107:0/634676302 conn(0x7fd4b8078c80 msgr2=0x7fd4b80ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:08.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.440+0000 7fd4bed3b640 1 -- 172.21.15.107:0/634676302 shutdown_connections 2024-05-09T08:42:08.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:08.440+0000 7fd4bed3b640 1 -- 172.21.15.107:0/634676302 wait complete. 2024-05-09T08:42:08.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:08 smithi107 ceph-mon[30122]: pgmap v272: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 37 MiB/s wr, 48 op/s 2024-05-09T08:42:08.453 INFO:tasks.cephfs.filesystem:scrub status for tag:5c91f14c-f432-4763-af5b-f30c806455d2 - {'path': '/', 'tag': '5c91f14c-f432-4763-af5b-f30c806455d2', 'options': 'recursive,force'} 2024-05-09T08:42:08.494 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:08 smithi157 ceph-mon[33614]: pgmap v272: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 37 MiB/s wr, 48 op/s 2024-05-09T08:42:09.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:09 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:42:09.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:09 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:09 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:42:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:09 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:42:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:09 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:09 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:42:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:09 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:09 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:42:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:09 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:09 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:42:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:09 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:09 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:42:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:09 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:09 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.437+0000 7f58d62da640 1 -- 172.21.15.107:0/3163205341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58d0101120 msgr2=0x7f58d010d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.437+0000 7f58d62da640 1 --2- 172.21.15.107:0/3163205341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58d0101120 0x7f58d010d5e0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f58c8009cd0 tx=0x7f58c802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.437+0000 7f58d62da640 1 -- 172.21.15.107:0/3163205341 shutdown_connections 2024-05-09T08:42:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.437+0000 7f58d62da640 1 --2- 172.21.15.107:0/3163205341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58d0101120 0x7f58d010d5e0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.437+0000 7f58d62da640 1 --2- 172.21.15.107:0/3163205341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f58d0100760 0x7f58d0100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.437+0000 7f58d62da640 1 --2- 172.21.15.107:0/3163205341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58d0106720 0x7f58d0106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.437+0000 7f58d62da640 1 -- 172.21.15.107:0/3163205341 >> 172.21.15.107:0/3163205341 conn(0x7f58d00778b0 msgr2=0x7f58d0077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.437+0000 7f58d62da640 1 -- 172.21.15.107:0/3163205341 shutdown_connections 2024-05-09T08:42:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.437+0000 7f58d62da640 1 -- 172.21.15.107:0/3163205341 wait complete. 2024-05-09T08:42:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.438+0000 7f58d62da640 1 Processor -- start 2024-05-09T08:42:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.438+0000 7f58d62da640 1 -- start start 2024-05-09T08:42:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.438+0000 7f58d62da640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58d0100760 0x7f58d0196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.438+0000 7f58d62da640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f58d0101120 0x7f58d01973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.438+0000 7f58d62da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58d0106720 0x7f58d019b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.438+0000 7f58d62da640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f58d01109a0 con 0x7f58d0106720 2024-05-09T08:42:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.438+0000 7f58d62da640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f58d0110820 con 0x7f58d0101120 2024-05-09T08:42:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.438+0000 7f58d62da640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f58d0110b20 con 0x7f58d0100760 2024-05-09T08:42:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.438+0000 7f58d4877640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58d0106720 0x7f58d019b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.438+0000 7f58d4877640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58d0106720 0x7f58d019b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41828/0 (socket says 172.21.15.107:41828) 2024-05-09T08:42:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.438+0000 7f58d4877640 1 -- 172.21.15.107:0/3411361491 learned_addr learned my addr 172.21.15.107:0/3411361491 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.438+0000 7f58cf7fe640 1 --2- 172.21.15.107:0/3411361491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f58d0101120 0x7f58d01973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.439+0000 7f58cffff640 1 --2- 172.21.15.107:0/3411361491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58d0100760 0x7f58d0196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.439+0000 7f58d4877640 1 -- 172.21.15.107:0/3411361491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58d0100760 msgr2=0x7f58d0196e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.439+0000 7f58d4877640 1 --2- 172.21.15.107:0/3411361491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58d0100760 0x7f58d0196e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.439+0000 7f58d4877640 1 -- 172.21.15.107:0/3411361491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f58d0101120 msgr2=0x7f58d01973a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.439+0000 7f58d4877640 1 --2- 172.21.15.107:0/3411361491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f58d0101120 0x7f58d01973a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.439+0000 7f58d4877640 1 -- 172.21.15.107:0/3411361491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d019be50 con 0x7f58d0106720 2024-05-09T08:42:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.439+0000 7f58d4877640 1 --2- 172.21.15.107:0/3411361491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58d0106720 0x7f58d019b720 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f58c8033040 tx=0x7f58c8031dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.439+0000 7f58cd7fa640 1 -- 172.21.15.107:0/3411361491 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c804a070 con 0x7f58d0106720 2024-05-09T08:42:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.439+0000 7f58d62da640 1 -- 172.21.15.107:0/3411361491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d019c0e0 con 0x7f58d0106720 2024-05-09T08:42:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.439+0000 7f58cd7fa640 1 -- 172.21.15.107:0/3411361491 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58c8004040 con 0x7f58d0106720 2024-05-09T08:42:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.439+0000 7f58cd7fa640 1 -- 172.21.15.107:0/3411361491 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c8030800 con 0x7f58d0106720 2024-05-09T08:42:09.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.439+0000 7f58d62da640 1 -- 172.21.15.107:0/3411361491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d0068ee0 con 0x7f58d0106720 2024-05-09T08:42:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.442+0000 7f58d8fed740 1 Processor -- start 2024-05-09T08:42:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.442+0000 7f58cd7fa640 1 -- 172.21.15.107:0/3411361491 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f58c80041f0 con 0x7f58d0106720 2024-05-09T08:42:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.442+0000 7f58d8fed740 1 -- start start 2024-05-09T08:42:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.442+0000 7f58d8fed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa4e330 0x55775fa4e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.442+0000 7f58d8fed740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4ec50 0x55775fa5d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.442+0000 7f58d8fed740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa5da40 0x55775fa5fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.442+0000 7f58d8fed740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55775f9b3340 con 0x55775fa4ec50 2024-05-09T08:42:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.442+0000 7f58d8fed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55775f95b290 con 0x55775fa4e330 2024-05-09T08:42:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.442+0000 7f58d8fed740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55775f9af300 con 0x55775fa5da40 2024-05-09T08:42:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.442+0000 7f58cffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4ec50 0x55775fa5d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.442+0000 7f58cffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4ec50 0x55775fa5d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41844/0 (socket says 172.21.15.107:41844) 2024-05-09T08:42:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.442+0000 7f58cffff640 1 -- 172.21.15.107:0/3237982679 learned_addr learned my addr 172.21.15.107:0/3237982679 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.443+0000 7f58cf7fe640 1 --2- 172.21.15.107:0/3237982679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa5da40 0x55775fa5fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.443+0000 7f58cf7fe640 1 --2- 172.21.15.107:0/3237982679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa4e330 0x55775fa4e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.443+0000 7f58cd7fa640 1 --2- 172.21.15.107:0/3411361491 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f58b80a2da0 0x7f58b80a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.443+0000 7f58cffff640 1 -- 172.21.15.107:0/3237982679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa5da40 msgr2=0x55775fa5fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.443+0000 7f58cffff640 1 --2- 172.21.15.107:0/3237982679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa5da40 0x55775fa5fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.443+0000 7f58cffff640 1 -- 172.21.15.107:0/3237982679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa4e330 msgr2=0x55775fa4e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.443+0000 7f58cffff640 1 --2- 172.21.15.107:0/3237982679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa4e330 0x55775fa4e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.443+0000 7f58cffff640 1 -- 172.21.15.107:0/3237982679 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55775f97f8e0 con 0x55775fa4ec50 2024-05-09T08:42:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.443+0000 7f58d4877640 1 --2- 172.21.15.107:0/3411361491 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f58b80a2da0 0x7f58b80a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.443+0000 7f58cf7fe640 1 --2- 172.21.15.107:0/3237982679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa4e330 0x55775fa4e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:42:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.443+0000 7f58cd7fa640 1 -- 172.21.15.107:0/3411361491 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f58c802e020 con 0x7f58d0106720 2024-05-09T08:42:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.443+0000 7f58cffff640 1 --2- 172.21.15.107:0/3237982679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4ec50 0x55775fa5d500 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f58c000bc90 tx=0x7f58c000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8056e2ed0f3f2eb2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.443+0000 7f58b67fc640 1 -- 172.21.15.107:0/3237982679 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c0015070 con 0x55775fa4ec50 2024-05-09T08:42:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.443+0000 7f58b67fc640 1 -- 172.21.15.107:0/3237982679 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58c00118f0 con 0x55775fa4ec50 2024-05-09T08:42:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.443+0000 7f58b67fc640 1 -- 172.21.15.107:0/3237982679 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c0010750 con 0x55775fa4ec50 2024-05-09T08:42:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.444+0000 7f58d8fed740 1 -- 172.21.15.107:0/3237982679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4ec50 msgr2=0x55775fa5d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.444+0000 7f58d4877640 1 --2- 172.21.15.107:0/3411361491 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f58b80a2da0 0x7f58b80a5260 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f58c80ba150 tx=0x7f58c808cc20 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.444+0000 7f58d8fed740 1 --2- 172.21.15.107:0/3237982679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4ec50 0x55775fa5d500 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f58c000bc90 tx=0x7f58c000f730 comp rx=0 tx=0).stop 2024-05-09T08:42:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.444+0000 7f58d8fed740 1 -- 172.21.15.107:0/3237982679 shutdown_connections 2024-05-09T08:42:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.444+0000 7f58d8fed740 1 --2- 172.21.15.107:0/3237982679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa5da40 0x55775fa5fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.444+0000 7f58d8fed740 1 --2- 172.21.15.107:0/3237982679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4ec50 0x55775fa5d500 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.444+0000 7f58d8fed740 1 --2- 172.21.15.107:0/3237982679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa4e330 0x55775fa4e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.444+0000 7f58d8fed740 1 -- 172.21.15.107:0/3237982679 >> 172.21.15.107:0/3237982679 conn(0x55775fa4a960 msgr2=0x55775fa4ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.444+0000 7f58d8fed740 1 -- 172.21.15.107:0/3237982679 shutdown_connections 2024-05-09T08:42:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.444+0000 7f58d8fed740 1 -- 172.21.15.107:0/3237982679 wait complete. 2024-05-09T08:42:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.445+0000 7f58d8fed740 1 -- start start 2024-05-09T08:42:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.445+0000 7f58d8fed740 1 Processor -- start 2024-05-09T08:42:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.445+0000 7f58d8fed740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4e330 0x55775fa5a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.445+0000 7f58d8fed740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa4ec50 0x55775fa5aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.445+0000 7f58d8fed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa5da40 0x55775fa5b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.445+0000 7f58d8fed740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55775fa60670 con 0x55775fa4e330 2024-05-09T08:42:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.445+0000 7f58d8fed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55775f96e790 con 0x55775fa5da40 2024-05-09T08:42:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.445+0000 7f58d8fed740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55775fa607f0 con 0x55775fa4ec50 2024-05-09T08:42:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.445+0000 7f58d8fed740 10 client.0 _ll_register_callbacks cb 0x55775f9a9a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.445+0000 7f58cf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4e330 0x55775fa5a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.445+0000 7f58cf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4e330 0x55775fa5a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41856/0 (socket says 172.21.15.107:41856) 2024-05-09T08:42:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.445+0000 7f58cf7fe640 1 -- 172.21.15.107:0/755081772 learned_addr learned my addr 172.21.15.107:0/755081772 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.445+0000 7f58cffff640 1 --2- 172.21.15.107:0/755081772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa4ec50 0x55775fa5aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.445+0000 7f58cf7fe640 1 --2- 172.21.15.107:0/755081772 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa5da40 0x55775fa5b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.446+0000 7f58cf7fe640 1 -- 172.21.15.107:0/755081772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa4ec50 msgr2=0x55775fa5aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.446+0000 7f58cf7fe640 1 --2- 172.21.15.107:0/755081772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa4ec50 0x55775fa5aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.446+0000 7f58cf7fe640 1 -- 172.21.15.107:0/755081772 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa5da40 msgr2=0x55775fa5b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.446+0000 7f58cf7fe640 1 --2- 172.21.15.107:0/755081772 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa5da40 0x55775fa5b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.446+0000 7f58cf7fe640 1 -- 172.21.15.107:0/755081772 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55775f6c7210 con 0x55775fa4e330 2024-05-09T08:42:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.446+0000 7f58cf7fe640 1 --2- 172.21.15.107:0/755081772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4e330 0x55775fa5a990 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009ef0 tx=0x7f58c4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df56a24de4356bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.446+0000 7f58af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.446+0000 7f58af7fe640 1 -- 172.21.15.107:0/755081772 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c400fb90 con 0x55775fa4e330 2024-05-09T08:42:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.446+0000 7f58af7fe640 1 -- 172.21.15.107:0/755081772 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f58c400fd30 con 0x55775fa4e330 2024-05-09T08:42:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.446+0000 7f58d8fed740 1 -- 172.21.15.107:0/755081772 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58c00118f0 con 0x55775fa4e330 2024-05-09T08:42:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.446+0000 7f58af7fe640 1 -- 172.21.15.107:0/755081772 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f58c4029050 con 0x55775fa4e330 2024-05-09T08:42:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.446+0000 7f58d8fed740 10 client.18276 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.446+0000 7f58d8fed740 1 -- 172.21.15.107:0/755081772 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55775f793620 con 0x55775fa4e330 2024-05-09T08:42:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.446+0000 7f58af7fe640 1 -- 172.21.15.107:0/755081772 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58c00118f0 con 0x55775fa4e330 2024-05-09T08:42:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.447+0000 7f58af7fe640 1 -- 172.21.15.107:0/755081772 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f58c4003bc0 con 0x55775fa4e330 2024-05-09T08:42:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.447+0000 7f58d8fed740 10 client.18276 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.447+0000 7f58d8fed740 10 client.18276 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.447+0000 7f58d8fed740 20 client.18276 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.447+0000 7f58d8fed740 1 --2- 172.21.15.107:0/755081772 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55775fa6d830 0x55775fa6fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.447+0000 7f58d8fed740 4 client.18276 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.447+0000 7f58d8fed740 1 -- 172.21.15.107:0/755081772 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55775f9c0a70 con 0x55775fa6d830 2024-05-09T08:42:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.447+0000 7f58cffff640 1 --2- 172.21.15.107:0/755081772 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55775fa6d830 0x55775fa6fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.447+0000 7f58af7fe640 10 client.18276 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.447+0000 7f58cffff640 1 --2- 172.21.15.107:0/755081772 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55775fa6d830 0x55775fa6fc20 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcdef9f2325e2686 server_cookie=3ae1be5a09a9d3b6 in_seq=0 out_seq=0 2024-05-09T08:42:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.448+0000 7f58af7fe640 1 -- 172.21.15.107:0/755081772 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55775f9c0a70 con 0x55775fa6d830 2024-05-09T08:42:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.448+0000 7f58af7fe640 10 client.18276 handle_command_reply: tid=0 2024-05-09T08:42:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.448+0000 7f58d8fed740 1 client.18276 shutdown 2024-05-09T08:42:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.448+0000 7f58d8fed740 1 -- 172.21.15.107:0/755081772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4e330 msgr2=0x55775fa5a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.448+0000 7f58d8fed740 1 --2- 172.21.15.107:0/755081772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4e330 0x55775fa5a990 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009ef0 tx=0x7f58c4005f90 comp rx=0 tx=0).stop 2024-05-09T08:42:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.448+0000 7f58d8fed740 1 -- 172.21.15.107:0/755081772 shutdown_connections 2024-05-09T08:42:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.449+0000 7f58d8fed740 1 --2- 172.21.15.107:0/755081772 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55775fa6d830 0x55775fa6fc20 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.449+0000 7f58d8fed740 1 --2- 172.21.15.107:0/755081772 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa5da40 0x55775fa5b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.449+0000 7f58d8fed740 1 --2- 172.21.15.107:0/755081772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa4ec50 0x55775fa5aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.449+0000 7f58d8fed740 1 --2- 172.21.15.107:0/755081772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4e330 0x55775fa5a990 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.449+0000 7f58d8fed740 1 -- 172.21.15.107:0/755081772 >> 172.21.15.107:0/755081772 conn(0x55775fa4a960 msgr2=0x55775fa50770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.449+0000 7f58af7fe640 0 client.18276 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.449+0000 7f58d8fed740 1 -- 172.21.15.107:0/755081772 shutdown_connections 2024-05-09T08:42:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.449+0000 7f58d8fed740 1 -- 172.21.15.107:0/755081772 wait complete. 2024-05-09T08:42:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.449+0000 7f58d8fed740 20 client.18276 trim_cache size 0 max 16384 2024-05-09T08:42:09.469 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.465+0000 7f58d8fed740 1 Processor -- start 2024-05-09T08:42:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.465+0000 7f58d8fed740 1 -- start start 2024-05-09T08:42:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58d8fed740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4e330 0x55775fa5d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58d8fed740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa4ec50 0x55775fa69570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58d8fed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa5da40 0x55775fa59970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58d8fed740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f58d01af230 con 0x55775fa4e330 2024-05-09T08:42:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58d8fed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f58d01acb00 con 0x55775fa5da40 2024-05-09T08:42:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58d8fed740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f58d01aa970 con 0x55775fa4ec50 2024-05-09T08:42:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58cf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4e330 0x55775fa5d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58cf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4e330 0x55775fa5d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41866/0 (socket says 172.21.15.107:41866) 2024-05-09T08:42:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58cf7fe640 1 -- 172.21.15.107:0/3898131042 learned_addr learned my addr 172.21.15.107:0/3898131042 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58cffff640 1 --2- 172.21.15.107:0/3898131042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa4ec50 0x55775fa69570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58cf7fe640 1 --2- 172.21.15.107:0/3898131042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa5da40 0x55775fa59970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58cf7fe640 1 -- 172.21.15.107:0/3898131042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa4ec50 msgr2=0x55775fa69570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58cf7fe640 1 --2- 172.21.15.107:0/3898131042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa4ec50 0x55775fa69570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58cf7fe640 1 -- 172.21.15.107:0/3898131042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa5da40 msgr2=0x55775fa59970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58cf7fe640 1 --2- 172.21.15.107:0/3898131042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa5da40 0x55775fa59970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58cf7fe640 1 -- 172.21.15.107:0/3898131042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55775fa4f8d0 con 0x55775fa4e330 2024-05-09T08:42:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58cf7fe640 1 --2- 172.21.15.107:0/3898131042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4e330 0x55775fa5d0c0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f58c40108d0 tx=0x7f58c40109b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ced754cdfe2d7a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.466+0000 7f58b5ffb640 1 -- 172.21.15.107:0/3898131042 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c400fb90 con 0x55775fa4e330 2024-05-09T08:42:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.467+0000 7f58b5ffb640 1 -- 172.21.15.107:0/3898131042 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58c400fd30 con 0x55775fa4e330 2024-05-09T08:42:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.467+0000 7f58b5ffb640 1 -- 172.21.15.107:0/3898131042 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c40035f0 con 0x55775fa4e330 2024-05-09T08:42:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.467+0000 7f58d8fed740 1 -- 172.21.15.107:0/3898131042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4e330 msgr2=0x55775fa5d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.467+0000 7f58d8fed740 1 --2- 172.21.15.107:0/3898131042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4e330 0x55775fa5d0c0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f58c40108d0 tx=0x7f58c40109b0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.467+0000 7f58d8fed740 1 -- 172.21.15.107:0/3898131042 shutdown_connections 2024-05-09T08:42:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.467+0000 7f58d8fed740 1 --2- 172.21.15.107:0/3898131042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa5da40 0x55775fa59970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.467+0000 7f58d8fed740 1 --2- 172.21.15.107:0/3898131042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa4ec50 0x55775fa69570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.467+0000 7f58d8fed740 1 --2- 172.21.15.107:0/3898131042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa4e330 0x55775fa5d0c0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.467+0000 7f58d8fed740 1 -- 172.21.15.107:0/3898131042 >> 172.21.15.107:0/3898131042 conn(0x55775fa4a960 msgr2=0x55775fa59260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.467+0000 7f58d8fed740 1 -- 172.21.15.107:0/3898131042 shutdown_connections 2024-05-09T08:42:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.467+0000 7f58d8fed740 1 -- 172.21.15.107:0/3898131042 wait complete. 2024-05-09T08:42:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.468+0000 7f58d8fed740 1 -- start start 2024-05-09T08:42:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.468+0000 7f58d8fed740 1 Processor -- start 2024-05-09T08:42:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.468+0000 7f58d8fed740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa4e330 0x55775fa60a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.468+0000 7f58d8fed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa4ec50 0x55775fa60f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.468+0000 7f58d8fed740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa5da40 0x55775fad1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.468+0000 7f58d8fed740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f58d01aafc0 con 0x55775fa5da40 2024-05-09T08:42:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.468+0000 7f58d8fed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f58d01ac430 con 0x55775fa4ec50 2024-05-09T08:42:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.468+0000 7f58d8fed740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f58d01ab690 con 0x55775fa4e330 2024-05-09T08:42:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.468+0000 7f58d8fed740 10 client.0 _ll_register_callbacks cb 0x7f58c4010900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.468+0000 7f58cf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa5da40 0x55775fad1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.468+0000 7f58cf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa5da40 0x55775fad1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41870/0 (socket says 172.21.15.107:41870) 2024-05-09T08:42:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.468+0000 7f58cf7fe640 1 -- 172.21.15.107:0/939867601 learned_addr learned my addr 172.21.15.107:0/939867601 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.468+0000 7f58cf7fe640 1 --2- 172.21.15.107:0/939867601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa4e330 0x55775fa60a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.468+0000 7f58cffff640 1 --2- 172.21.15.107:0/939867601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa4ec50 0x55775fa60f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.469+0000 7f58cf7fe640 1 -- 172.21.15.107:0/939867601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa4e330 msgr2=0x55775fa60a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.469+0000 7f58cf7fe640 1 --2- 172.21.15.107:0/939867601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa4e330 0x55775fa60a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.469+0000 7f58cf7fe640 1 -- 172.21.15.107:0/939867601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa4ec50 msgr2=0x55775fa60f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.469+0000 7f58cf7fe640 1 --2- 172.21.15.107:0/939867601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa4ec50 0x55775fa60f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.469+0000 7f58cf7fe640 1 -- 172.21.15.107:0/939867601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55775fa50e50 con 0x55775fa5da40 2024-05-09T08:42:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.469+0000 7f58cf7fe640 1 --2- 172.21.15.107:0/939867601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa5da40 0x55775fad1d70 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009730 tx=0x7f58c400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=750eaa54d7687be3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.469+0000 7f58af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.469+0000 7f58af7fe640 1 -- 172.21.15.107:0/939867601 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c40217d0 con 0x55775fa5da40 2024-05-09T08:42:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.469+0000 7f58af7fe640 1 -- 172.21.15.107:0/939867601 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f58c4021a70 con 0x55775fa5da40 2024-05-09T08:42:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.469+0000 7f58d8fed740 1 -- 172.21.15.107:0/939867601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58c400fd30 con 0x55775fa5da40 2024-05-09T08:42:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.469+0000 7f58af7fe640 1 -- 172.21.15.107:0/939867601 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f58c4021c70 con 0x55775fa5da40 2024-05-09T08:42:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.469+0000 7f58d8fed740 10 client.18282 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.469+0000 7f58d8fed740 1 -- 172.21.15.107:0/939867601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55775fad32b0 con 0x55775fa5da40 2024-05-09T08:42:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.469+0000 7f58af7fe640 1 -- 172.21.15.107:0/939867601 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58c400fd30 con 0x55775fa5da40 2024-05-09T08:42:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.470+0000 7f58af7fe640 1 -- 172.21.15.107:0/939867601 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f58c403f040 con 0x55775fa5da40 2024-05-09T08:42:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.470+0000 7f58d8fed740 10 client.18282 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.470+0000 7f58d8fed740 10 client.18282 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.470+0000 7f58d8fed740 20 client.18282 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.470+0000 7f58d8fed740 1 --2- 172.21.15.107:0/939867601 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55775fad3910 0x55775fad5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.470+0000 7f58d8fed740 4 client.18282 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:42:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.470+0000 7f58d8fed740 1 -- 172.21.15.107:0/939867601 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55775f9badc0 con 0x55775fad3910 2024-05-09T08:42:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.470+0000 7f58cffff640 1 --2- 172.21.15.107:0/939867601 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55775fad3910 0x55775fad5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.470+0000 7f58cffff640 1 --2- 172.21.15.107:0/939867601 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55775fad3910 0x55775fad5d30 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b287d1fc1bb96b29 server_cookie=f4ca4c214772f36a in_seq=0 out_seq=0 2024-05-09T08:42:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.470+0000 7f58af7fe640 10 client.18282 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.490+0000 7f58af7fe640 1 -- 172.21.15.107:0/939867601 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55775f9badc0 con 0x55775fad3910 2024-05-09T08:42:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.490+0000 7f58af7fe640 10 client.18282 handle_command_reply: tid=0 2024-05-09T08:42:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.490+0000 7f58d8fed740 1 client.18282 shutdown 2024-05-09T08:42:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.490+0000 7f58d8fed740 1 -- 172.21.15.107:0/939867601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa5da40 msgr2=0x55775fad1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.490+0000 7f58d8fed740 1 --2- 172.21.15.107:0/939867601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa5da40 0x55775fad1d70 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009730 tx=0x7f58c400f790 comp rx=0 tx=0).stop 2024-05-09T08:42:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.490+0000 7f58d8fed740 1 -- 172.21.15.107:0/939867601 shutdown_connections 2024-05-09T08:42:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.490+0000 7f58d8fed740 1 --2- 172.21.15.107:0/939867601 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55775fad3910 0x55775fad5d30 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.490+0000 7f58af7fe640 0 client.18282 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.490+0000 7f58d8fed740 1 --2- 172.21.15.107:0/939867601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55775fa5da40 0x55775fad1d70 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.490+0000 7f58d8fed740 1 --2- 172.21.15.107:0/939867601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55775fa4ec50 0x55775fa60f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.490+0000 7f58d8fed740 1 --2- 172.21.15.107:0/939867601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55775fa4e330 0x55775fa60a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.490+0000 7f58d8fed740 1 -- 172.21.15.107:0/939867601 >> 172.21.15.107:0/939867601 conn(0x55775fa4a960 msgr2=0x55775fa5a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.490+0000 7f58d8fed740 1 -- 172.21.15.107:0/939867601 shutdown_connections 2024-05-09T08:42:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.490+0000 7f58d8fed740 1 -- 172.21.15.107:0/939867601 wait complete. 2024-05-09T08:42:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.490+0000 7f58d8fed740 20 client.18282 trim_cache size 0 max 16384 2024-05-09T08:42:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:42:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:42:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:42:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:42:09.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.491+0000 7f58d62da640 1 -- 172.21.15.107:0/3411361491 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f58b80a2da0 msgr2=0x7f58b80a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.491+0000 7f58d62da640 1 --2- 172.21.15.107:0/3411361491 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f58b80a2da0 0x7f58b80a5260 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f58c80ba150 tx=0x7f58c808cc20 comp rx=0 tx=0).stop 2024-05-09T08:42:09.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.491+0000 7f58d62da640 1 -- 172.21.15.107:0/3411361491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58d0106720 msgr2=0x7f58d019b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.491+0000 7f58d62da640 1 --2- 172.21.15.107:0/3411361491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58d0106720 0x7f58d019b720 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f58c8033040 tx=0x7f58c8031dd0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.491+0000 7f58d62da640 1 -- 172.21.15.107:0/3411361491 shutdown_connections 2024-05-09T08:42:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.491+0000 7f58d62da640 1 --2- 172.21.15.107:0/3411361491 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f58b80a2da0 0x7f58b80a5260 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.491+0000 7f58d62da640 1 --2- 172.21.15.107:0/3411361491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58d0106720 0x7f58d019b720 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.491+0000 7f58d62da640 1 --2- 172.21.15.107:0/3411361491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f58d0101120 0x7f58d01973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.491+0000 7f58d62da640 1 --2- 172.21.15.107:0/3411361491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58d0100760 0x7f58d0196e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.491+0000 7f58d62da640 1 -- 172.21.15.107:0/3411361491 >> 172.21.15.107:0/3411361491 conn(0x7f58d00778b0 msgr2=0x7f58d01031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.491+0000 7f58d62da640 1 -- 172.21.15.107:0/3411361491 shutdown_connections 2024-05-09T08:42:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:09.491+0000 7f58d62da640 1 -- 172.21.15.107:0/3411361491 wait complete. 2024-05-09T08:42:09.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:09 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:42:09.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:09 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:09 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:42:09.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:09 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:42:09.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:09 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:09 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:42:09.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:09 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:09 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:42:09.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:09 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:09 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:42:09.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:09 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:09 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:42:09.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:09 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:09 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:09 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:42:09.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:09 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:09 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:42:09.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:09 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:42:09.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:09 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:09 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:42:09.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:09 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:09 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:42:09.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:09 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:09 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:42:09.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:09 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:09 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:42:09.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:09 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:09.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:09 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:10.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:10 smithi110 ceph-mon[33622]: pgmap v273: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 37 MiB/s wr, 48 op/s 2024-05-09T08:42:10.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:10 smithi107 ceph-mon[30122]: pgmap v273: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 37 MiB/s wr, 48 op/s 2024-05-09T08:42:10.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:10 smithi157 ceph-mon[33614]: pgmap v273: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 37 MiB/s wr, 48 op/s 2024-05-09T08:42:12.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:12 smithi110 ceph-mon[33622]: pgmap v274: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 30 MiB/s wr, 39 op/s 2024-05-09T08:42:12.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:12 smithi107 ceph-mon[30122]: pgmap v274: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 30 MiB/s wr, 39 op/s 2024-05-09T08:42:12.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:12 smithi157 ceph-mon[33614]: pgmap v274: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 30 MiB/s wr, 39 op/s 2024-05-09T08:42:14.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:14 smithi110 ceph-mon[33622]: pgmap v275: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 29 MiB/s wr, 65 op/s 2024-05-09T08:42:14.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:14 smithi107 ceph-mon[30122]: pgmap v275: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 29 MiB/s wr, 65 op/s 2024-05-09T08:42:14.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:14 smithi157 ceph-mon[33614]: pgmap v275: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 29 MiB/s wr, 65 op/s 2024-05-09T08:42:16.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:16 smithi110 ceph-mon[33622]: pgmap v276: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 29 MiB/s wr, 65 op/s 2024-05-09T08:42:16.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:16 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:42:16.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:16 smithi107 ceph-mon[30122]: pgmap v276: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 29 MiB/s wr, 65 op/s 2024-05-09T08:42:16.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:16 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:42:16.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:16 smithi157 ceph-mon[33614]: pgmap v276: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 29 MiB/s wr, 65 op/s 2024-05-09T08:42:16.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:16 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:42:17.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:17 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:42:17.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:17 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:42:17.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:17 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:42:17.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:17 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:42:17.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:17 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:42:17.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:17 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:42:18.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:18 smithi110 ceph-mon[33622]: pgmap v277: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 30 MiB/s wr, 65 op/s 2024-05-09T08:42:18.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:18 smithi107 ceph-mon[30122]: pgmap v277: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 30 MiB/s wr, 65 op/s 2024-05-09T08:42:18.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:18 smithi157 ceph-mon[33614]: pgmap v277: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 30 MiB/s wr, 65 op/s 2024-05-09T08:42:20.209 INFO:tasks.workunit.client.0.smithi107.stdout:fs setup took 441 secs 2024-05-09T08:42:20.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:20 smithi110 ceph-mon[33622]: pgmap v278: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 25 MiB/s wr, 54 op/s 2024-05-09T08:42:20.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:20 smithi107 ceph-mon[30122]: pgmap v278: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 25 MiB/s wr, 54 op/s 2024-05-09T08:42:20.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:20 smithi157 ceph-mon[33614]: pgmap v278: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 25 MiB/s wr, 54 op/s 2024-05-09T08:42:22.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:22 smithi107 ceph-mon[30122]: pgmap v279: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 25 MiB/s wr, 54 op/s 2024-05-09T08:42:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.592+0000 7f6581b28640 1 -- 172.21.15.107:0/3843006179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f657c101150 msgr2=0x7f657c10d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.592+0000 7f6581b28640 1 --2- 172.21.15.107:0/3843006179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f657c101150 0x7f657c10d610 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f656c009930 tx=0x7f656c02a420 comp rx=0 tx=0).stop 2024-05-09T08:42:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.592+0000 7f6581b28640 1 -- 172.21.15.107:0/3843006179 shutdown_connections 2024-05-09T08:42:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.592+0000 7f6581b28640 1 --2- 172.21.15.107:0/3843006179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f657c101150 0x7f657c10d610 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.592+0000 7f6581b28640 1 --2- 172.21.15.107:0/3843006179 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f657c100790 0x7f657c100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.592+0000 7f6581b28640 1 --2- 172.21.15.107:0/3843006179 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f657c106750 0x7f657c106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.592+0000 7f6581b28640 1 -- 172.21.15.107:0/3843006179 >> 172.21.15.107:0/3843006179 conn(0x7f657c077940 msgr2=0x7f657c077d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.592+0000 7f6581b28640 1 -- 172.21.15.107:0/3843006179 shutdown_connections 2024-05-09T08:42:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.592+0000 7f6581b28640 1 -- 172.21.15.107:0/3843006179 wait complete. 2024-05-09T08:42:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.592+0000 7f6581b28640 1 Processor -- start 2024-05-09T08:42:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f6581b28640 1 -- start start 2024-05-09T08:42:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f6581b28640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f657c100790 0x7f657c196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f6581b28640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f657c101150 0x7f657c197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f6581b28640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f657c106750 0x7f657c19b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f6581b28640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f657c110a10 con 0x7f657c106750 2024-05-09T08:42:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f6581b28640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f657c110890 con 0x7f657c101150 2024-05-09T08:42:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f6581b28640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f657c110b90 con 0x7f657c100790 2024-05-09T08:42:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f657bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f657c106750 0x7f657c19b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f657bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f657c106750 0x7f657c19b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41962/0 (socket says 172.21.15.107:41962) 2024-05-09T08:42:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f657bfff640 1 -- 172.21.15.107:0/103594427 learned_addr learned my addr 172.21.15.107:0/103594427 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f657affd640 1 --2- 172.21.15.107:0/103594427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f657c101150 0x7f657c197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f657b7fe640 1 --2- 172.21.15.107:0/103594427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f657c100790 0x7f657c196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f657bfff640 1 -- 172.21.15.107:0/103594427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f657c100790 msgr2=0x7f657c196e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f657bfff640 1 --2- 172.21.15.107:0/103594427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f657c100790 0x7f657c196e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f657bfff640 1 -- 172.21.15.107:0/103594427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f657c101150 msgr2=0x7f657c197350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f657bfff640 1 --2- 172.21.15.107:0/103594427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f657c101150 0x7f657c197350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f657bfff640 1 -- 172.21.15.107:0/103594427 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657c19be00 con 0x7f657c106750 2024-05-09T08:42:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f657bfff640 1 --2- 172.21.15.107:0/103594427 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f657c106750 0x7f657c19b6d0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f656c005940 tx=0x7f656c031a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.593+0000 7f6578ff9640 1 -- 172.21.15.107:0/103594427 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656c03f030 con 0x7f657c106750 2024-05-09T08:42:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.594+0000 7f6581b28640 1 -- 172.21.15.107:0/103594427 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f657c19c090 con 0x7f657c106750 2024-05-09T08:42:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.594+0000 7f6578ff9640 1 -- 172.21.15.107:0/103594427 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f656c0043d0 con 0x7f657c106750 2024-05-09T08:42:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.594+0000 7f6578ff9640 1 -- 172.21.15.107:0/103594427 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656c035070 con 0x7f657c106750 2024-05-09T08:42:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.594+0000 7f6581b28640 1 -- 172.21.15.107:0/103594427 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f657c1a3980 con 0x7f657c106750 2024-05-09T08:42:22.597 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:22 smithi157 ceph-mon[33614]: pgmap v279: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 25 MiB/s wr, 54 op/s 2024-05-09T08:42:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.596+0000 7f6578ff9640 1 -- 172.21.15.107:0/103594427 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f656c042030 con 0x7f657c106750 2024-05-09T08:42:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.596+0000 7f6584831740 1 Processor -- start 2024-05-09T08:42:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f6584831740 1 -- start start 2024-05-09T08:42:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f6584831740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56e3390 0x560bd56e3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f6584831740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3cb0 0x560bd56f0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f6584831740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56f0840 0x560bd56f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f6584831740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560bd5648340 con 0x560bd56e3390 2024-05-09T08:42:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f6584831740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560bd55f0290 con 0x560bd56e3cb0 2024-05-09T08:42:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f6584831740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560bd5644300 con 0x560bd56f0840 2024-05-09T08:42:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f6578ff9640 1 --2- 172.21.15.107:0/103594427 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65640a2da0 0x7f65640a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f657affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56e3390 0x560bd56e3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f657affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56e3390 0x560bd56e3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41978/0 (socket says 172.21.15.107:41978) 2024-05-09T08:42:22.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f657affd640 1 -- 172.21.15.107:0/3512556659 learned_addr learned my addr 172.21.15.107:0/3512556659 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:22.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f657bfff640 1 --2- 172.21.15.107:0/103594427 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65640a2da0 0x7f65640a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f657b7fe640 1 --2- 172.21.15.107:0/3512556659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3cb0 0x560bd56f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f657affd640 1 --2- 172.21.15.107:0/3512556659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56f0840 0x560bd56f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f6578ff9640 1 -- 172.21.15.107:0/103594427 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f656c0efb20 con 0x7f657c106750 2024-05-09T08:42:22.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f657affd640 1 -- 172.21.15.107:0/3512556659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56f0840 msgr2=0x560bd56f2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:22.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f657affd640 1 --2- 172.21.15.107:0/3512556659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56f0840 0x560bd56f2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f657affd640 1 -- 172.21.15.107:0/3512556659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3cb0 msgr2=0x560bd56f0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f657affd640 1 --2- 172.21.15.107:0/3512556659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3cb0 0x560bd56f0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f657affd640 1 -- 172.21.15.107:0/3512556659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bd56148e0 con 0x560bd56e3390 2024-05-09T08:42:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.597+0000 7f657b7fe640 1 --2- 172.21.15.107:0/3512556659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3cb0 0x560bd56f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:42:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.598+0000 7f657affd640 1 --2- 172.21.15.107:0/3512556659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56e3390 0x560bd56e3770 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f6570013580 tx=0x7f6570009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9882ccb784937b86 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.598+0000 7f657bfff640 1 --2- 172.21.15.107:0/103594427 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65640a2da0 0x7f65640a5260 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f656c0bb9e0 tx=0x7f656c0e67c0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.598+0000 7f6561ffb640 1 -- 172.21.15.107:0/3512556659 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6570011720 con 0x560bd56e3390 2024-05-09T08:42:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.598+0000 7f6561ffb640 1 -- 172.21.15.107:0/3512556659 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6570011d70 con 0x560bd56e3390 2024-05-09T08:42:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.598+0000 7f6561ffb640 1 -- 172.21.15.107:0/3512556659 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657001ed10 con 0x560bd56e3390 2024-05-09T08:42:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 -- 172.21.15.107:0/3512556659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56e3390 msgr2=0x560bd56e3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 --2- 172.21.15.107:0/3512556659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56e3390 0x560bd56e3770 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f6570013580 tx=0x7f6570009830 comp rx=0 tx=0).stop 2024-05-09T08:42:22.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 -- 172.21.15.107:0/3512556659 shutdown_connections 2024-05-09T08:42:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 --2- 172.21.15.107:0/3512556659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56f0840 0x560bd56f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 --2- 172.21.15.107:0/3512556659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3cb0 0x560bd56f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 --2- 172.21.15.107:0/3512556659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56e3390 0x560bd56e3770 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 -- 172.21.15.107:0/3512556659 >> 172.21.15.107:0/3512556659 conn(0x560bd56df960 msgr2=0x560bd56dfd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 -- 172.21.15.107:0/3512556659 shutdown_connections 2024-05-09T08:42:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 -- 172.21.15.107:0/3512556659 wait complete. 2024-05-09T08:42:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 -- start start 2024-05-09T08:42:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 Processor -- start 2024-05-09T08:42:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3390 0x560bd56ed7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56e3cb0 0x560bd56edcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56f0840 0x560bd56ee230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560bd56f3410 con 0x560bd56f0840 2024-05-09T08:42:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560bd5603790 con 0x560bd56e3390 2024-05-09T08:42:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.599+0000 7f6584831740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560bd56f3590 con 0x560bd56e3cb0 2024-05-09T08:42:22.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f6584831740 10 client.0 _ll_register_callbacks cb 0x560bd563ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:22.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f657affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56f0840 0x560bd56ee230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f657affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56f0840 0x560bd56ee230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41994/0 (socket says 172.21.15.107:41994) 2024-05-09T08:42:22.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f657affd640 1 -- 172.21.15.107:0/3785513747 learned_addr learned my addr 172.21.15.107:0/3785513747 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:22.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f657b7fe640 1 --2- 172.21.15.107:0/3785513747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56e3cb0 0x560bd56edcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f657affd640 1 --2- 172.21.15.107:0/3785513747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3390 0x560bd56ed7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f657affd640 1 -- 172.21.15.107:0/3785513747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56e3cb0 msgr2=0x560bd56edcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:22.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f657affd640 1 --2- 172.21.15.107:0/3785513747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56e3cb0 0x560bd56edcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f657affd640 1 -- 172.21.15.107:0/3785513747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3390 msgr2=0x560bd56ed7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:22.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f657affd640 1 --2- 172.21.15.107:0/3785513747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3390 0x560bd56ed7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f657affd640 1 -- 172.21.15.107:0/3785513747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bd535c210 con 0x560bd56f0840 2024-05-09T08:42:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f657affd640 1 --2- 172.21.15.107:0/3785513747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56f0840 0x560bd56ee230 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f6570005bc0 tx=0x7f6570011f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4df7aaa04892aa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f655affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f655affd640 1 -- 172.21.15.107:0/3785513747 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6570030060 con 0x560bd56f0840 2024-05-09T08:42:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f655affd640 1 -- 172.21.15.107:0/3785513747 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f6570005030 con 0x560bd56f0840 2024-05-09T08:42:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f6584831740 1 -- 172.21.15.107:0/3785513747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6570011d70 con 0x560bd56f0840 2024-05-09T08:42:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f655affd640 1 -- 172.21.15.107:0/3785513747 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6570005230 con 0x560bd56f0840 2024-05-09T08:42:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f6584831740 10 client.18294 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.600+0000 7f6584831740 1 -- 172.21.15.107:0/3785513747 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560bd5428620 con 0x560bd56f0840 2024-05-09T08:42:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.601+0000 7f655affd640 1 -- 172.21.15.107:0/3785513747 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6570011d70 con 0x560bd56f0840 2024-05-09T08:42:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.601+0000 7f655affd640 1 -- 172.21.15.107:0/3785513747 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f657002a040 con 0x560bd56f0840 2024-05-09T08:42:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.601+0000 7f6584831740 10 client.18294 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.601+0000 7f6584831740 10 client.18294 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.601+0000 7f6584831740 20 client.18294 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.601+0000 7f6584831740 1 --2- 172.21.15.107:0/3785513747 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560bd57004f0 0x560bd57028e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.601+0000 7f6584831740 4 client.18294 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:22.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.601+0000 7f6584831740 1 -- 172.21.15.107:0/3785513747 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bd5655a70 con 0x560bd57004f0 2024-05-09T08:42:22.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.601+0000 7f657b7fe640 1 --2- 172.21.15.107:0/3785513747 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560bd57004f0 0x560bd57028e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.602+0000 7f655affd640 10 client.18294 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:22.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.602+0000 7f657b7fe640 1 --2- 172.21.15.107:0/3785513747 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560bd57004f0 0x560bd57028e0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=408338f04c128e25 server_cookie=337cbdf3b7437ca5 in_seq=0 out_seq=0 2024-05-09T08:42:22.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.602+0000 7f655affd640 1 -- 172.21.15.107:0/3785513747 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560bd5655a70 con 0x560bd57004f0 2024-05-09T08:42:22.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.602+0000 7f655affd640 10 client.18294 handle_command_reply: tid=0 2024-05-09T08:42:22.614 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.602+0000 7f6584831740 1 client.18294 shutdown 2024-05-09T08:42:22.614 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.602+0000 7f6584831740 1 -- 172.21.15.107:0/3785513747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56f0840 msgr2=0x560bd56ee230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:22.614 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.602+0000 7f6584831740 1 --2- 172.21.15.107:0/3785513747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56f0840 0x560bd56ee230 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f6570005bc0 tx=0x7f6570011f20 comp rx=0 tx=0).stop 2024-05-09T08:42:22.614 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.603+0000 7f6584831740 1 -- 172.21.15.107:0/3785513747 shutdown_connections 2024-05-09T08:42:22.614 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.603+0000 7f6584831740 1 --2- 172.21.15.107:0/3785513747 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560bd57004f0 0x560bd57028e0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.614 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.603+0000 7f6584831740 1 --2- 172.21.15.107:0/3785513747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56f0840 0x560bd56ee230 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.614 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.603+0000 7f6584831740 1 --2- 172.21.15.107:0/3785513747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56e3cb0 0x560bd56edcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.615 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.603+0000 7f655affd640 0 client.18294 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:22.615 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.603+0000 7f6584831740 1 --2- 172.21.15.107:0/3785513747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3390 0x560bd56ed7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.615 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.603+0000 7f6584831740 1 -- 172.21.15.107:0/3785513747 >> 172.21.15.107:0/3785513747 conn(0x560bd56df960 msgr2=0x560bd56e56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:22.615 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.603+0000 7f6584831740 1 -- 172.21.15.107:0/3785513747 shutdown_connections 2024-05-09T08:42:22.615 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.603+0000 7f6584831740 1 -- 172.21.15.107:0/3785513747 wait complete. 2024-05-09T08:42:22.616 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.603+0000 7f6584831740 20 client.18294 trim_cache size 0 max 16384 2024-05-09T08:42:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f6584831740 1 Processor -- start 2024-05-09T08:42:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f6584831740 1 -- start start 2024-05-09T08:42:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f6584831740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56e3390 0x560bd56efee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f6584831740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3cb0 0x560bd56fc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f6584831740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56f0840 0x560bd56dfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f6584831740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f657c1af2a0 con 0x560bd56f0840 2024-05-09T08:42:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f6584831740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f657c1acb70 con 0x560bd56e3cb0 2024-05-09T08:42:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f6584831740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f657c1aa9e0 con 0x560bd56e3390 2024-05-09T08:42:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f657affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56f0840 0x560bd56dfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f657affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56f0840 0x560bd56dfd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42006/0 (socket says 172.21.15.107:42006) 2024-05-09T08:42:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f657affd640 1 -- 172.21.15.107:0/973328095 learned_addr learned my addr 172.21.15.107:0/973328095 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f657affd640 1 -- 172.21.15.107:0/973328095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56e3390 msgr2=0x560bd56efee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f657affd640 1 --2- 172.21.15.107:0/973328095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56e3390 0x560bd56efee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f657affd640 1 -- 172.21.15.107:0/973328095 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3cb0 msgr2=0x560bd56fc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f657b7fe640 1 --2- 172.21.15.107:0/973328095 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3cb0 0x560bd56fc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f657affd640 1 --2- 172.21.15.107:0/973328095 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3cb0 0x560bd56fc340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.618+0000 7f657affd640 1 -- 172.21.15.107:0/973328095 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bd56ec7a0 con 0x560bd56f0840 2024-05-09T08:42:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.619+0000 7f657affd640 1 --2- 172.21.15.107:0/973328095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56f0840 0x560bd56dfd40 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f6570001eb0 tx=0x7f6570005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b35463d60598ce7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.619+0000 7f65617fa640 1 -- 172.21.15.107:0/973328095 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6570040070 con 0x560bd56f0840 2024-05-09T08:42:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.619+0000 7f65617fa640 1 -- 172.21.15.107:0/973328095 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6570009780 con 0x560bd56f0840 2024-05-09T08:42:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.619+0000 7f65617fa640 1 -- 172.21.15.107:0/973328095 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6570030070 con 0x560bd56f0840 2024-05-09T08:42:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.619+0000 7f6584831740 1 -- 172.21.15.107:0/973328095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56f0840 msgr2=0x560bd56dfd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.619+0000 7f6584831740 1 --2- 172.21.15.107:0/973328095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56f0840 0x560bd56dfd40 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f6570001eb0 tx=0x7f6570005f90 comp rx=0 tx=0).stop 2024-05-09T08:42:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.619+0000 7f6584831740 1 -- 172.21.15.107:0/973328095 shutdown_connections 2024-05-09T08:42:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.619+0000 7f6584831740 1 --2- 172.21.15.107:0/973328095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56f0840 0x560bd56dfd40 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.619+0000 7f6584831740 1 --2- 172.21.15.107:0/973328095 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3cb0 0x560bd56fc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.619+0000 7f6584831740 1 --2- 172.21.15.107:0/973328095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56e3390 0x560bd56efee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.619+0000 7f6584831740 1 -- 172.21.15.107:0/973328095 >> 172.21.15.107:0/973328095 conn(0x560bd56df960 msgr2=0x560bd56dea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.619+0000 7f6584831740 1 -- 172.21.15.107:0/973328095 shutdown_connections 2024-05-09T08:42:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.619+0000 7f6584831740 1 -- 172.21.15.107:0/973328095 wait complete. 2024-05-09T08:42:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.620+0000 7f6584831740 1 -- start start 2024-05-09T08:42:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.620+0000 7f6584831740 1 Processor -- start 2024-05-09T08:42:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.620+0000 7f6584831740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56e3390 0x560bd56ea540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.620+0000 7f6584831740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3cb0 0x560bd56eaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.620+0000 7f6584831740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56f0840 0x560bd56eafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.620+0000 7f6584831740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f657c1ab030 con 0x560bd56e3390 2024-05-09T08:42:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.620+0000 7f6584831740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f657c1ac4a0 con 0x560bd56e3cb0 2024-05-09T08:42:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.620+0000 7f6584831740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f657c1ab700 con 0x560bd56f0840 2024-05-09T08:42:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.620+0000 7f6584831740 10 client.0 _ll_register_callbacks cb 0x7f6570013460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.620+0000 7f657affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56e3390 0x560bd56ea540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.620+0000 7f657affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56e3390 0x560bd56ea540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42022/0 (socket says 172.21.15.107:42022) 2024-05-09T08:42:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.620+0000 7f657affd640 1 -- 172.21.15.107:0/2149810934 learned_addr learned my addr 172.21.15.107:0/2149810934 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.620+0000 7f657b7fe640 1 --2- 172.21.15.107:0/2149810934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3cb0 0x560bd56eaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.620+0000 7f657affd640 1 --2- 172.21.15.107:0/2149810934 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56f0840 0x560bd56eafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f657affd640 1 -- 172.21.15.107:0/2149810934 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56f0840 msgr2=0x560bd56eafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f657affd640 1 --2- 172.21.15.107:0/2149810934 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56f0840 0x560bd56eafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f657affd640 1 -- 172.21.15.107:0/2149810934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3cb0 msgr2=0x560bd56eaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f657affd640 1 --2- 172.21.15.107:0/2149810934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3cb0 0x560bd56eaa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f657affd640 1 -- 172.21.15.107:0/2149810934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bd5765870 con 0x560bd56e3390 2024-05-09T08:42:22.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f657affd640 1 --2- 172.21.15.107:0/2149810934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56e3390 0x560bd56ea540 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f6570004a00 tx=0x7f6570005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9feba26fce3eb3f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:22.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f655affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:22.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f655affd640 1 -- 172.21.15.107:0/2149810934 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6570040070 con 0x560bd56e3390 2024-05-09T08:42:22.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f655affd640 1 -- 172.21.15.107:0/2149810934 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f6570004f80 con 0x560bd56e3390 2024-05-09T08:42:22.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f6584831740 1 -- 172.21.15.107:0/2149810934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6570009780 con 0x560bd56e3390 2024-05-09T08:42:22.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f655affd640 1 -- 172.21.15.107:0/2149810934 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6570030040 con 0x560bd56e3390 2024-05-09T08:42:22.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f6584831740 10 client.18300 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:22.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f6584831740 1 -- 172.21.15.107:0/2149810934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560bd5766b30 con 0x560bd56e3390 2024-05-09T08:42:22.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f655affd640 1 -- 172.21.15.107:0/2149810934 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6570009780 con 0x560bd56e3390 2024-05-09T08:42:22.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f655affd640 1 -- 172.21.15.107:0/2149810934 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f6570043020 con 0x560bd56e3390 2024-05-09T08:42:22.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f6584831740 10 client.18300 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:22.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f6584831740 10 client.18300 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:22.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f6584831740 20 client.18300 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:22.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.621+0000 7f6584831740 1 --2- 172.21.15.107:0/2149810934 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560bd5767140 0x560bd5769560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:22.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.622+0000 7f6584831740 4 client.18300 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:42:22.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.622+0000 7f6584831740 1 -- 172.21.15.107:0/2149810934 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560bd564fdc0 con 0x560bd5767140 2024-05-09T08:42:22.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.622+0000 7f657b7fe640 1 --2- 172.21.15.107:0/2149810934 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560bd5767140 0x560bd5769560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:22.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.622+0000 7f655affd640 10 client.18300 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:22.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.622+0000 7f657b7fe640 1 --2- 172.21.15.107:0/2149810934 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560bd5767140 0x560bd5769560 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7789c2be0c7b063 server_cookie=856810ff50c9bb18 in_seq=0 out_seq=0 2024-05-09T08:42:22.627 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:22 smithi110 ceph-mon[33622]: pgmap v279: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 25 MiB/s wr, 54 op/s 2024-05-09T08:42:22.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.640+0000 7f655affd640 1 -- 172.21.15.107:0/2149810934 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560bd564fdc0 con 0x560bd5767140 2024-05-09T08:42:22.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.640+0000 7f655affd640 10 client.18300 handle_command_reply: tid=0 2024-05-09T08:42:22.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.640+0000 7f6584831740 1 client.18300 shutdown 2024-05-09T08:42:22.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.640+0000 7f6584831740 1 -- 172.21.15.107:0/2149810934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56e3390 msgr2=0x560bd56ea540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:22.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.640+0000 7f6584831740 1 --2- 172.21.15.107:0/2149810934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56e3390 0x560bd56ea540 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f6570004a00 tx=0x7f6570005850 comp rx=0 tx=0).stop 2024-05-09T08:42:22.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.640+0000 7f6584831740 1 -- 172.21.15.107:0/2149810934 shutdown_connections 2024-05-09T08:42:22.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.640+0000 7f6584831740 1 --2- 172.21.15.107:0/2149810934 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560bd5767140 0x560bd5769560 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.640+0000 7f6584831740 1 --2- 172.21.15.107:0/2149810934 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bd56f0840 0x560bd56eafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.640+0000 7f655affd640 0 client.18300 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:22.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.640+0000 7f6584831740 1 --2- 172.21.15.107:0/2149810934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560bd56e3cb0 0x560bd56eaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.640+0000 7f6584831740 1 --2- 172.21.15.107:0/2149810934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560bd56e3390 0x560bd56ea540 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.640+0000 7f6584831740 1 -- 172.21.15.107:0/2149810934 >> 172.21.15.107:0/2149810934 conn(0x560bd56df960 msgr2=0x560bd56dea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:22.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.640+0000 7f6584831740 1 -- 172.21.15.107:0/2149810934 shutdown_connections 2024-05-09T08:42:22.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.640+0000 7f6584831740 1 -- 172.21.15.107:0/2149810934 wait complete. 2024-05-09T08:42:22.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.640+0000 7f6584831740 20 client.18300 trim_cache size 0 max 16384 2024-05-09T08:42:22.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:42:22.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:42:22.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:42:22.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:42:22.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.641+0000 7f6581b28640 1 -- 172.21.15.107:0/103594427 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65640a2da0 msgr2=0x7f65640a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:22.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.641+0000 7f6581b28640 1 --2- 172.21.15.107:0/103594427 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65640a2da0 0x7f65640a5260 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f656c0bb9e0 tx=0x7f656c0e67c0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.641+0000 7f6581b28640 1 -- 172.21.15.107:0/103594427 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f657c106750 msgr2=0x7f657c19b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:22.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.641+0000 7f6581b28640 1 --2- 172.21.15.107:0/103594427 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f657c106750 0x7f657c19b6d0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f656c005940 tx=0x7f656c031a10 comp rx=0 tx=0).stop 2024-05-09T08:42:22.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.641+0000 7f6581b28640 1 -- 172.21.15.107:0/103594427 shutdown_connections 2024-05-09T08:42:22.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.641+0000 7f6581b28640 1 --2- 172.21.15.107:0/103594427 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65640a2da0 0x7f65640a5260 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.641+0000 7f6581b28640 1 --2- 172.21.15.107:0/103594427 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f657c106750 0x7f657c19b6d0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.641+0000 7f6581b28640 1 --2- 172.21.15.107:0/103594427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f657c101150 0x7f657c197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.641+0000 7f6581b28640 1 --2- 172.21.15.107:0/103594427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f657c100790 0x7f657c196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:22.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.641+0000 7f6581b28640 1 -- 172.21.15.107:0/103594427 >> 172.21.15.107:0/103594427 conn(0x7f657c077940 msgr2=0x7f657c103270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:22.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.641+0000 7f6581b28640 1 -- 172.21.15.107:0/103594427 shutdown_connections 2024-05-09T08:42:22.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:22.641+0000 7f6581b28640 1 -- 172.21.15.107:0/103594427 wait complete. 2024-05-09T08:42:22.656 INFO:tasks.workunit.client.0.smithi107.stdout:Syncing()...2 sec 2024-05-09T08:42:22.657 INFO:tasks.workunit.client.0.smithi107.stdout:Starting Actual Benchmark At: Thu May 9 08:42:22 2024 2024-05-09T08:42:22.657 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:42:24.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:24 smithi107 ceph-mon[30122]: pgmap v280: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 33 MiB/s wr, 60 op/s 2024-05-09T08:42:24.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:24 smithi157 ceph-mon[33614]: pgmap v280: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 33 MiB/s wr, 60 op/s 2024-05-09T08:42:24.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:24 smithi110 ceph-mon[33622]: pgmap v280: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 33 MiB/s wr, 60 op/s 2024-05-09T08:42:26.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:26 smithi107 ceph-mon[30122]: pgmap v281: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 21 MiB/s wr, 14 op/s 2024-05-09T08:42:26.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:26 smithi157 ceph-mon[33614]: pgmap v281: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 21 MiB/s wr, 14 op/s 2024-05-09T08:42:26.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:26 smithi110 ceph-mon[33622]: pgmap v281: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 21 MiB/s wr, 14 op/s 2024-05-09T08:42:28.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:28 smithi107 ceph-mon[30122]: pgmap v282: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 27 MiB/s wr, 1.49k op/s 2024-05-09T08:42:28.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:28 smithi157 ceph-mon[33614]: pgmap v282: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 27 MiB/s wr, 1.49k op/s 2024-05-09T08:42:28.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:28 smithi110 ceph-mon[33622]: pgmap v282: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 27 MiB/s wr, 1.49k op/s 2024-05-09T08:42:30.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:30 smithi107 ceph-mon[30122]: pgmap v283: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 15 MiB/s wr, 1.61k op/s 2024-05-09T08:42:30.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:30 smithi157 ceph-mon[33614]: pgmap v283: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 15 MiB/s wr, 1.61k op/s 2024-05-09T08:42:30.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:30 smithi110 ceph-mon[33622]: pgmap v283: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 15 MiB/s wr, 1.61k op/s 2024-05-09T08:42:32.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:32 smithi107 ceph-mon[30122]: pgmap v284: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 15 MiB/s wr, 1.61k op/s 2024-05-09T08:42:32.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:32 smithi157 ceph-mon[33614]: pgmap v284: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 15 MiB/s wr, 1.61k op/s 2024-05-09T08:42:32.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:32 smithi110 ceph-mon[33622]: pgmap v284: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 15 MiB/s wr, 1.61k op/s 2024-05-09T08:42:34.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:34 smithi107 ceph-mon[30122]: pgmap v285: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 23 MiB/s wr, 3.63k op/s 2024-05-09T08:42:34.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:34 smithi157 ceph-mon[33614]: pgmap v285: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 23 MiB/s wr, 3.63k op/s 2024-05-09T08:42:34.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:34 smithi110 ceph-mon[33622]: pgmap v285: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 23 MiB/s wr, 3.63k op/s 2024-05-09T08:42:35.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.849+0000 7faf3111d640 1 -- 172.21.15.107:0/915581961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf2c11b010 msgr2=0x7faf2c11d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:35.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.849+0000 7faf3111d640 1 --2- 172.21.15.107:0/915581961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf2c11b010 0x7faf2c11d400 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faf2401acb0 tx=0x7faf2403b990 comp rx=0 tx=0).stop 2024-05-09T08:42:35.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.849+0000 7faf3111d640 1 -- 172.21.15.107:0/915581961 shutdown_connections 2024-05-09T08:42:35.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.849+0000 7faf3111d640 1 --2- 172.21.15.107:0/915581961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf2c11b010 0x7faf2c11d400 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.849+0000 7faf3111d640 1 --2- 172.21.15.107:0/915581961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf2c110400 0x7faf2c108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.849+0000 7faf3111d640 1 --2- 172.21.15.107:0/915581961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c110020 0x7faf2c108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.849+0000 7faf3111d640 1 -- 172.21.15.107:0/915581961 >> 172.21.15.107:0/915581961 conn(0x7faf2c06c900 msgr2=0x7faf2c06cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:35.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.849+0000 7faf3111d640 1 -- 172.21.15.107:0/915581961 shutdown_connections 2024-05-09T08:42:35.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.849+0000 7faf3111d640 1 -- 172.21.15.107:0/915581961 wait complete. 2024-05-09T08:42:35.851 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.850+0000 7faf3111d640 1 Processor -- start 2024-05-09T08:42:35.851 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.851+0000 7faf3111d640 1 -- start start 2024-05-09T08:42:35.851 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.851+0000 7faf3111d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf2c110020 0x7faf2c11a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.851 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.851+0000 7faf3111d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c11b010 0x7faf2c116810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.851 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.851+0000 7faf3111d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf2c116d50 0x7faf2c117200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.851 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.851+0000 7faf3111d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c113110 con 0x7faf2c11b010 2024-05-09T08:42:35.851 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.851+0000 7faf3111d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c112f90 con 0x7faf2c110020 2024-05-09T08:42:35.851 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.851+0000 7faf3111d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c113290 con 0x7faf2c116d50 2024-05-09T08:42:35.851 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.851+0000 7faf2ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf2c110020 0x7faf2c11a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:35.851 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.851+0000 7faf2ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf2c110020 0x7faf2c11a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60662/0 (socket says 172.21.15.107:60662) 2024-05-09T08:42:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.851+0000 7faf2ad9d640 1 -- 172.21.15.107:0/174616468 learned_addr learned my addr 172.21.15.107:0/174616468 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.852+0000 7faf2b59e640 1 --2- 172.21.15.107:0/174616468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf2c116d50 0x7faf2c117200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:35.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.854+0000 7faf2b59e640 1 -- 172.21.15.107:0/174616468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf2c110020 msgr2=0x7faf2c11a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:35.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.854+0000 7faf2b59e640 1 --2- 172.21.15.107:0/174616468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf2c110020 0x7faf2c11a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.854+0000 7faf2b59e640 1 -- 172.21.15.107:0/174616468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c11b010 msgr2=0x7faf2c116810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:35.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.854+0000 7faf2b59e640 1 --2- 172.21.15.107:0/174616468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c11b010 0x7faf2c116810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.854+0000 7faf2b59e640 1 -- 172.21.15.107:0/174616468 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf2c1aae00 con 0x7faf2c116d50 2024-05-09T08:42:35.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.855+0000 7faf2b59e640 1 --2- 172.21.15.107:0/174616468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf2c116d50 0x7faf2c117200 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faf2403bf80 tx=0x7faf24003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:35.858 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.855+0000 7faf13fff640 1 -- 172.21.15.107:0/174616468 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf24045030 con 0x7faf2c116d50 2024-05-09T08:42:35.858 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.855+0000 7faf3111d640 1 -- 172.21.15.107:0/174616468 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf2c1ab090 con 0x7faf2c116d50 2024-05-09T08:42:35.858 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.855+0000 7faf13fff640 1 -- 172.21.15.107:0/174616468 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf2404f070 con 0x7faf2c116d50 2024-05-09T08:42:35.858 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.855+0000 7faf13fff640 1 -- 172.21.15.107:0/174616468 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf24004760 con 0x7faf2c116d50 2024-05-09T08:42:35.858 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.855+0000 7faf3111d640 1 -- 172.21.15.107:0/174616468 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf2c1ab5a0 con 0x7faf2c116d50 2024-05-09T08:42:35.860 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.861+0000 7faf33e22740 1 Processor -- start 2024-05-09T08:42:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.861+0000 7faf33e22740 1 -- start start 2024-05-09T08:42:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.861+0000 7faf33e22740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac737450 0x5578ac737830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.862+0000 7faf33e22740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac737d70 0x5578ac7420f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.862+0000 7faf33e22740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac742630 0x5578ac744a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.862+0000 7faf33e22740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5578ac69c340 con 0x5578ac742630 2024-05-09T08:42:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.862+0000 7faf33e22740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5578ac644290 con 0x5578ac737d70 2024-05-09T08:42:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.862+0000 7faf33e22740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5578ac698300 con 0x5578ac737450 2024-05-09T08:42:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.862+0000 7faf13fff640 1 -- 172.21.15.107:0/174616468 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7faf24048020 con 0x7faf2c116d50 2024-05-09T08:42:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.862+0000 7faf2ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac737d70 0x5578ac7420f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.862+0000 7faf2ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac737d70 0x5578ac7420f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60668/0 (socket says 172.21.15.107:60668) 2024-05-09T08:42:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.862+0000 7faf2ad9d640 1 -- 172.21.15.107:0/3149355436 learned_addr learned my addr 172.21.15.107:0/3149355436 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.863+0000 7faf2ad9d640 1 -- 172.21.15.107:0/3149355436 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac737450 msgr2=0x5578ac737830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.863+0000 7faf2ad9d640 1 --2- 172.21.15.107:0/3149355436 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac737450 0x5578ac737830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.863+0000 7faf2ad9d640 1 -- 172.21.15.107:0/3149355436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac742630 msgr2=0x5578ac744a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.863+0000 7faf2ad9d640 1 --2- 172.21.15.107:0/3149355436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac742630 0x5578ac744a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.863+0000 7faf2ad9d640 1 -- 172.21.15.107:0/3149355436 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578ac6688e0 con 0x5578ac737d70 2024-05-09T08:42:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.868+0000 7faf13fff640 1 --2- 172.21.15.107:0/174616468 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf140a2ef0 0x7faf140a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.868+0000 7faf13fff640 1 -- 172.21.15.107:0/174616468 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7faf24102910 con 0x7faf2c116d50 2024-05-09T08:42:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.868+0000 7faf2a59c640 1 --2- 172.21.15.107:0/174616468 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf140a2ef0 0x7faf140a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.869+0000 7faf2ad9d640 1 --2- 172.21.15.107:0/3149355436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac737d70 0x5578ac7420f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7faf1c009b20 tx=0x7faf1c00e1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=532d0b9397f32f0d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.869+0000 7faf117fa640 1 -- 172.21.15.107:0/3149355436 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c017070 con 0x5578ac737d70 2024-05-09T08:42:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.870+0000 7faf2a59c640 1 --2- 172.21.15.107:0/174616468 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf140a2ef0 0x7faf140a53b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x5578ac734d80 tx=0x7faf20007b00 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.873+0000 7faf117fa640 1 -- 172.21.15.107:0/3149355436 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf1c00ec10 con 0x5578ac737d70 2024-05-09T08:42:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.873+0000 7faf117fa640 1 -- 172.21.15.107:0/3149355436 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c012630 con 0x5578ac737d70 2024-05-09T08:42:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.875+0000 7faf33e22740 1 -- 172.21.15.107:0/3149355436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac737d70 msgr2=0x5578ac7420f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.875+0000 7faf33e22740 1 --2- 172.21.15.107:0/3149355436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac737d70 0x5578ac7420f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7faf1c009b20 tx=0x7faf1c00e1d0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.881+0000 7faf33e22740 1 -- 172.21.15.107:0/3149355436 shutdown_connections 2024-05-09T08:42:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.881+0000 7faf33e22740 1 --2- 172.21.15.107:0/3149355436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac742630 0x5578ac744a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.881+0000 7faf33e22740 1 --2- 172.21.15.107:0/3149355436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac737d70 0x5578ac7420f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.881+0000 7faf33e22740 1 --2- 172.21.15.107:0/3149355436 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac737450 0x5578ac737830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.881+0000 7faf33e22740 1 -- 172.21.15.107:0/3149355436 >> 172.21.15.107:0/3149355436 conn(0x5578ac733960 msgr2=0x5578ac733d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.883+0000 7faf33e22740 1 -- 172.21.15.107:0/3149355436 shutdown_connections 2024-05-09T08:42:35.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.885+0000 7faf33e22740 1 -- 172.21.15.107:0/3149355436 wait complete. 2024-05-09T08:42:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.886+0000 7faf33e22740 1 -- start start 2024-05-09T08:42:35.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.886+0000 7faf33e22740 1 Processor -- start 2024-05-09T08:42:35.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.888+0000 7faf33e22740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac737450 0x5578ac73f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.888+0000 7faf33e22740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac737d70 0x5578ac73fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.888+0000 7faf33e22740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac742630 0x5578ac740070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.888+0000 7faf33e22740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5578ac745260 con 0x5578ac737d70 2024-05-09T08:42:35.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.888+0000 7faf33e22740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5578ac657790 con 0x5578ac742630 2024-05-09T08:42:35.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.888+0000 7faf33e22740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5578ac7453e0 con 0x5578ac737450 2024-05-09T08:42:35.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.888+0000 7faf33e22740 10 client.0 _ll_register_callbacks cb 0x5578ac692a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.891+0000 7faf2b59e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac742630 0x5578ac740070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.891+0000 7faf2b59e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac742630 0x5578ac740070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60680/0 (socket says 172.21.15.107:60680) 2024-05-09T08:42:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.891+0000 7faf2b59e640 1 -- 172.21.15.107:0/3561909545 learned_addr learned my addr 172.21.15.107:0/3561909545 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.891+0000 7faf2b59e640 1 -- 172.21.15.107:0/3561909545 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac737450 msgr2=0x5578ac73f5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:42:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.891+0000 7faf2b59e640 1 --2- 172.21.15.107:0/3561909545 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac737450 0x5578ac73f5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.891+0000 7faf2b59e640 1 -- 172.21.15.107:0/3561909545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac737d70 msgr2=0x5578ac73fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.891+0000 7faf2b59e640 1 --2- 172.21.15.107:0/3561909545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac737d70 0x5578ac73fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.891+0000 7faf2b59e640 1 -- 172.21.15.107:0/3561909545 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578ac3b0210 con 0x5578ac742630 2024-05-09T08:42:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.892+0000 7faf2b59e640 1 --2- 172.21.15.107:0/3561909545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac742630 0x5578ac740070 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7faf240d7270 tx=0x7faf240ce490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2dd377f675c149c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.892+0000 7faf0eff5640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:42:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.892+0000 7faf0eff5640 1 -- 172.21.15.107:0/3561909545 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf240cfe60 con 0x5578ac742630 2024-05-09T08:42:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.892+0000 7faf0eff5640 1 -- 172.21.15.107:0/3561909545 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7faf240ffed0 con 0x5578ac742630 2024-05-09T08:42:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.892+0000 7faf33e22740 1 -- 172.21.15.107:0/3561909545 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf1c00ec10 con 0x5578ac742630 2024-05-09T08:42:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.893+0000 7faf0eff5640 1 -- 172.21.15.107:0/3561909545 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faf241000d0 con 0x5578ac742630 2024-05-09T08:42:35.895 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.897+0000 7faf33e22740 10 client.27230 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:35.896 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.897+0000 7faf33e22740 1 -- 172.21.15.107:0/3561909545 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5578ac47c620 con 0x5578ac742630 2024-05-09T08:42:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.900+0000 7faf0eff5640 1 -- 172.21.15.107:0/3561909545 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf1c00ec10 con 0x5578ac742630 2024-05-09T08:42:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.900+0000 7faf0eff5640 1 -- 172.21.15.107:0/3561909545 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7faf240ed2a0 con 0x5578ac742630 2024-05-09T08:42:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.900+0000 7faf33e22740 10 client.27230 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.900+0000 7faf33e22740 10 client.27230 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.900+0000 7faf33e22740 20 client.27230 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.900+0000 7faf33e22740 1 --2- 172.21.15.107:0/3561909545 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5578ac752310 0x5578ac7547b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.900+0000 7faf33e22740 4 client.27230 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.900+0000 7faf33e22740 1 -- 172.21.15.107:0/3561909545 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578ac6a9a70 con 0x5578ac752310 2024-05-09T08:42:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.900+0000 7faf2ad9d640 1 --2- 172.21.15.107:0/3561909545 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5578ac752310 0x5578ac7547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.902+0000 7faf2ad9d640 1 --2- 172.21.15.107:0/3561909545 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5578ac752310 0x5578ac7547b0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5baee71f80d5523f server_cookie=be3b14c9b8ba30b6 in_seq=0 out_seq=0 2024-05-09T08:42:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.903+0000 7faf0eff5640 10 client.27230 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.905+0000 7faf0eff5640 1 -- 172.21.15.107:0/3561909545 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5578ac6a9a70 con 0x5578ac752310 2024-05-09T08:42:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.905+0000 7faf0eff5640 10 client.27230 handle_command_reply: tid=0 2024-05-09T08:42:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.905+0000 7faf33e22740 1 client.27230 shutdown 2024-05-09T08:42:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.908+0000 7faf33e22740 1 -- 172.21.15.107:0/3561909545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac742630 msgr2=0x5578ac740070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.908+0000 7faf33e22740 1 --2- 172.21.15.107:0/3561909545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac742630 0x5578ac740070 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7faf240d7270 tx=0x7faf240ce490 comp rx=0 tx=0).stop 2024-05-09T08:42:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.908+0000 7faf33e22740 1 -- 172.21.15.107:0/3561909545 shutdown_connections 2024-05-09T08:42:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.908+0000 7faf33e22740 1 --2- 172.21.15.107:0/3561909545 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5578ac752310 0x5578ac7547b0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.908+0000 7faf33e22740 1 --2- 172.21.15.107:0/3561909545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac742630 0x5578ac740070 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.908+0000 7faf33e22740 1 --2- 172.21.15.107:0/3561909545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac737d70 0x5578ac73fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.908 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.908+0000 7faf33e22740 1 --2- 172.21.15.107:0/3561909545 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac737450 0x5578ac73f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.908 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.908+0000 7faf33e22740 1 -- 172.21.15.107:0/3561909545 >> 172.21.15.107:0/3561909545 conn(0x5578ac733960 msgr2=0x5578ac739690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:35.908 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.908+0000 7faf0eff5640 0 client.27230 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:35.912 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.912+0000 7faf33e22740 1 -- 172.21.15.107:0/3561909545 shutdown_connections 2024-05-09T08:42:35.912 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.913+0000 7faf33e22740 1 -- 172.21.15.107:0/3561909545 wait complete. 2024-05-09T08:42:35.912 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.913+0000 7faf33e22740 20 client.27230 trim_cache size 0 max 16384 2024-05-09T08:42:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.962+0000 7faf33e22740 1 Processor -- start 2024-05-09T08:42:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.962+0000 7faf33e22740 1 -- start start 2024-05-09T08:42:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.962+0000 7faf33e22740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac737450 0x5578ac74e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.962+0000 7faf33e22740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac737d70 0x5578ac739e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.962+0000 7faf33e22740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac742630 0x5578ac732880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.962+0000 7faf33e22740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c1b7c70 con 0x5578ac742630 2024-05-09T08:42:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.962+0000 7faf33e22740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c1b5540 con 0x5578ac737450 2024-05-09T08:42:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.962+0000 7faf33e22740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c1b33d0 con 0x5578ac737d70 2024-05-09T08:42:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.962+0000 7faf2ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac737450 0x5578ac74e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.962+0000 7faf2ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac737450 0x5578ac74e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60696/0 (socket says 172.21.15.107:60696) 2024-05-09T08:42:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.962+0000 7faf2ad9d640 1 -- 172.21.15.107:0/145235745 learned_addr learned my addr 172.21.15.107:0/145235745 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.962+0000 7faf2a59c640 1 --2- 172.21.15.107:0/145235745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac737d70 0x5578ac739e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:35.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.962+0000 7faf2b59e640 1 --2- 172.21.15.107:0/145235745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac742630 0x5578ac732880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:35.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.963+0000 7faf2ad9d640 1 -- 172.21.15.107:0/145235745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac737d70 msgr2=0x5578ac739e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:35.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.963+0000 7faf2ad9d640 1 --2- 172.21.15.107:0/145235745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac737d70 0x5578ac739e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.963+0000 7faf2ad9d640 1 -- 172.21.15.107:0/145235745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac742630 msgr2=0x5578ac732880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:35.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.963+0000 7faf2ad9d640 1 --2- 172.21.15.107:0/145235745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac742630 0x5578ac732880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.963+0000 7faf2ad9d640 1 -- 172.21.15.107:0/145235745 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578ac73e5a0 con 0x5578ac737450 2024-05-09T08:42:35.964 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.965+0000 7faf2ad9d640 1 --2- 172.21.15.107:0/145235745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac737450 0x5578ac74e120 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faf1c013dc0 tx=0x7faf1c00e1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa53449bf8317c00 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:35.967 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.968+0000 7faf10ff9640 1 -- 172.21.15.107:0/145235745 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c017040 con 0x5578ac737450 2024-05-09T08:42:35.967 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.968+0000 7faf10ff9640 1 -- 172.21.15.107:0/145235745 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf1c01c450 con 0x5578ac737450 2024-05-09T08:42:35.967 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.968+0000 7faf10ff9640 1 -- 172.21.15.107:0/145235745 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c012630 con 0x5578ac737450 2024-05-09T08:42:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.973+0000 7faf33e22740 1 -- 172.21.15.107:0/145235745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac737450 msgr2=0x5578ac74e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.973+0000 7faf33e22740 1 --2- 172.21.15.107:0/145235745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac737450 0x5578ac74e120 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faf1c013dc0 tx=0x7faf1c00e1d0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.975+0000 7faf33e22740 1 -- 172.21.15.107:0/145235745 shutdown_connections 2024-05-09T08:42:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.975+0000 7faf33e22740 1 --2- 172.21.15.107:0/145235745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac742630 0x5578ac732880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.975+0000 7faf33e22740 1 --2- 172.21.15.107:0/145235745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac737d70 0x5578ac739e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.975+0000 7faf33e22740 1 --2- 172.21.15.107:0/145235745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac737450 0x5578ac74e120 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faf1c013dc0 tx=0x7faf1c00e1d0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.975+0000 7faf33e22740 1 -- 172.21.15.107:0/145235745 >> 172.21.15.107:0/145235745 conn(0x5578ac733960 msgr2=0x5578ac732170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:35.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.985+0000 7faf33e22740 1 -- 172.21.15.107:0/145235745 shutdown_connections 2024-05-09T08:42:35.988 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.989+0000 7faf33e22740 1 -- 172.21.15.107:0/145235745 wait complete. 2024-05-09T08:42:35.990 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.990+0000 7faf33e22740 1 -- start start 2024-05-09T08:42:35.990 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.990+0000 7faf33e22740 1 Processor -- start 2024-05-09T08:42:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.991+0000 7faf33e22740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac737d70 0x5578ac745fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.991+0000 7faf33e22740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac742630 0x5578ac7464e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.991+0000 7faf33e22740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac746a20 0x5578ac7b7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.991+0000 7faf33e22740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c1b3a20 con 0x5578ac737d70 2024-05-09T08:42:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.991+0000 7faf33e22740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c1b4e70 con 0x5578ac746a20 2024-05-09T08:42:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.991+0000 7faf33e22740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c1b40d0 con 0x5578ac742630 2024-05-09T08:42:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.991+0000 7faf33e22740 10 client.0 _ll_register_callbacks cb 0x7faf24004900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.991+0000 7faf2b59e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac746a20 0x5578ac7b7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.991+0000 7faf2b59e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac746a20 0x5578ac7b7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60700/0 (socket says 172.21.15.107:60700) 2024-05-09T08:42:35.991 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.991+0000 7faf2b59e640 1 -- 172.21.15.107:0/2875281895 learned_addr learned my addr 172.21.15.107:0/2875281895 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:35.992 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.992+0000 7faf2a59c640 1 --2- 172.21.15.107:0/2875281895 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac742630 0x5578ac7464e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:35.992 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.992+0000 7faf2ad9d640 1 --2- 172.21.15.107:0/2875281895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac737d70 0x5578ac745fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:35.992 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.993+0000 7faf2a59c640 1 -- 172.21.15.107:0/2875281895 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac746a20 msgr2=0x5578ac7b7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:35.992 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.993+0000 7faf2a59c640 1 --2- 172.21.15.107:0/2875281895 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac746a20 0x5578ac7b7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.992 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.993+0000 7faf2a59c640 1 -- 172.21.15.107:0/2875281895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac737d70 msgr2=0x5578ac745fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:35.992 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.993+0000 7faf2a59c640 1 --2- 172.21.15.107:0/2875281895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac737d70 0x5578ac745fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:35.992 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.993+0000 7faf2a59c640 1 -- 172.21.15.107:0/2875281895 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578ac7b7e70 con 0x5578ac742630 2024-05-09T08:42:35.992 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.993+0000 7faf2ad9d640 1 --2- 172.21.15.107:0/2875281895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac737d70 0x5578ac745fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:42:35.993 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.993+0000 7faf2a59c640 1 --2- 172.21.15.107:0/2875281895 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac742630 0x5578ac7464e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faf20007fd0 tx=0x7faf200011b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f03078483b936552 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:35.994 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.993+0000 7faf0eff5640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:42:35.994 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.993+0000 7faf0eff5640 1 -- 172.21.15.107:0/2875281895 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf20006900 con 0x5578ac742630 2024-05-09T08:42:35.994 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.993+0000 7faf33e22740 1 -- 172.21.15.107:0/2875281895 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf1c01c450 con 0x5578ac742630 2024-05-09T08:42:35.994 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.994+0000 7faf0eff5640 1 -- 172.21.15.107:0/2875281895 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7faf20006aa0 con 0x5578ac742630 2024-05-09T08:42:35.994 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:35.994+0000 7faf0eff5640 1 -- 172.21.15.107:0/2875281895 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faf20006ca0 con 0x5578ac742630 2024-05-09T08:42:36.001 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.002+0000 7faf33e22740 10 client.27493 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:36.001 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.002+0000 7faf33e22740 1 -- 172.21.15.107:0/2875281895 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5578ac7b90b0 con 0x5578ac742630 2024-05-09T08:42:36.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.007+0000 7faf0eff5640 1 -- 172.21.15.107:0/2875281895 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf1c01c450 con 0x5578ac742630 2024-05-09T08:42:36.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.007+0000 7faf0eff5640 1 -- 172.21.15.107:0/2875281895 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7faf200146d0 con 0x5578ac742630 2024-05-09T08:42:36.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.007+0000 7faf33e22740 10 client.27493 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:36.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.007+0000 7faf33e22740 10 client.27493 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:36.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.007+0000 7faf33e22740 20 client.27493 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:36.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.007+0000 7faf33e22740 1 --2- 172.21.15.107:0/2875281895 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5578ac7b9880 0x5578ac7bbca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:36.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.007+0000 7faf33e22740 4 client.27493 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:42:36.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.007+0000 7faf33e22740 1 -- 172.21.15.107:0/2875281895 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5578ac6a3dc0 con 0x5578ac7b9880 2024-05-09T08:42:36.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.009+0000 7faf2ad9d640 1 --2- 172.21.15.107:0/2875281895 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5578ac7b9880 0x5578ac7bbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.018+0000 7faf2ad9d640 1 --2- 172.21.15.107:0/2875281895 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5578ac7b9880 0x5578ac7bbca0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f617c3ec4c24766e server_cookie=610e539062e8836c in_seq=0 out_seq=0 2024-05-09T08:42:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.018+0000 7faf0eff5640 10 client.27493 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.095+0000 7faf0eff5640 1 -- 172.21.15.107:0/2875281895 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5578ac6a3dc0 con 0x5578ac7b9880 2024-05-09T08:42:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.095+0000 7faf0eff5640 10 client.27493 handle_command_reply: tid=0 2024-05-09T08:42:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.095+0000 7faf33e22740 1 client.27493 shutdown 2024-05-09T08:42:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.095+0000 7faf33e22740 1 -- 172.21.15.107:0/2875281895 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac742630 msgr2=0x5578ac7464e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.095+0000 7faf33e22740 1 --2- 172.21.15.107:0/2875281895 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac742630 0x5578ac7464e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faf20007fd0 tx=0x7faf200011b0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.096+0000 7faf33e22740 1 -- 172.21.15.107:0/2875281895 shutdown_connections 2024-05-09T08:42:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.096+0000 7faf33e22740 1 --2- 172.21.15.107:0/2875281895 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5578ac7b9880 0x5578ac7bbca0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.096+0000 7faf33e22740 1 --2- 172.21.15.107:0/2875281895 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5578ac746a20 0x5578ac7b7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.096+0000 7faf33e22740 1 --2- 172.21.15.107:0/2875281895 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5578ac742630 0x5578ac7464e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.096+0000 7faf33e22740 1 --2- 172.21.15.107:0/2875281895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578ac737d70 0x5578ac745fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.096+0000 7faf33e22740 1 -- 172.21.15.107:0/2875281895 >> 172.21.15.107:0/2875281895 conn(0x5578ac733960 msgr2=0x5578ac732a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.096+0000 7faf0eff5640 0 client.27493 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:42:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:42:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:42:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:42:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.098+0000 7faf33e22740 1 -- 172.21.15.107:0/2875281895 shutdown_connections 2024-05-09T08:42:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.098+0000 7faf33e22740 1 -- 172.21.15.107:0/2875281895 wait complete. 2024-05-09T08:42:36.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.098+0000 7faf33e22740 20 client.27493 trim_cache size 0 max 16384 2024-05-09T08:42:36.113 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.113+0000 7faf3111d640 1 -- 172.21.15.107:0/174616468 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf140a2ef0 msgr2=0x7faf140a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.113+0000 7faf3111d640 1 --2- 172.21.15.107:0/174616468 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf140a2ef0 0x7faf140a53b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x5578ac734d80 tx=0x7faf20007b00 comp rx=0 tx=0).stop 2024-05-09T08:42:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.113+0000 7faf3111d640 1 -- 172.21.15.107:0/174616468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf2c116d50 msgr2=0x7faf2c117200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.113+0000 7faf3111d640 1 --2- 172.21.15.107:0/174616468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf2c116d50 0x7faf2c117200 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faf2403bf80 tx=0x7faf24003ba0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.113+0000 7faf3111d640 1 -- 172.21.15.107:0/174616468 shutdown_connections 2024-05-09T08:42:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.113+0000 7faf3111d640 1 --2- 172.21.15.107:0/174616468 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf140a2ef0 0x7faf140a53b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.113+0000 7faf3111d640 1 --2- 172.21.15.107:0/174616468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf2c116d50 0x7faf2c117200 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.113+0000 7faf3111d640 1 --2- 172.21.15.107:0/174616468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf2c11b010 0x7faf2c116810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.114+0000 7faf3111d640 1 --2- 172.21.15.107:0/174616468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf2c110020 0x7faf2c11a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.114+0000 7faf3111d640 1 -- 172.21.15.107:0/174616468 >> 172.21.15.107:0/174616468 conn(0x7faf2c06c900 msgr2=0x7faf2c11b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.114+0000 7faf3111d640 1 -- 172.21.15.107:0/174616468 shutdown_connections 2024-05-09T08:42:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:36.114+0000 7faf3111d640 1 -- 172.21.15.107:0/174616468 wait complete. 2024-05-09T08:42:36.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:36 smithi107 ceph-mon[30122]: pgmap v286: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 14 MiB/s wr, 3.62k op/s 2024-05-09T08:42:36.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:36 smithi157 ceph-mon[33614]: pgmap v286: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 14 MiB/s wr, 3.62k op/s 2024-05-09T08:42:36.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:36 smithi110 ceph-mon[33622]: pgmap v286: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 14 MiB/s wr, 3.62k op/s 2024-05-09T08:42:38.454 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:42:38.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:42:38.475 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:38 smithi107 ceph-mon[30122]: pgmap v287: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 21 MiB/s wr, 5.30k op/s 2024-05-09T08:42:38.475 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:38 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:38.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:38 smithi157 ceph-mon[33614]: pgmap v287: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 21 MiB/s wr, 5.30k op/s 2024-05-09T08:42:38.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:38 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:38.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:38 smithi110 ceph-mon[33622]: pgmap v287: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 21 MiB/s wr, 5.30k op/s 2024-05-09T08:42:38.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:38 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:42:38.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.662+0000 7fad3208c640 1 -- 172.21.15.107:0/3608707101 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2c11b010 msgr2=0x7fad2c11d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:38.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.662+0000 7fad3208c640 1 --2- 172.21.15.107:0/3608707101 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2c11b010 0x7fad2c11d400 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fad24009ca0 tx=0x7fad2402a8d0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.664+0000 7fad3208c640 1 -- 172.21.15.107:0/3608707101 shutdown_connections 2024-05-09T08:42:38.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.664+0000 7fad3208c640 1 --2- 172.21.15.107:0/3608707101 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2c11b010 0x7fad2c11d400 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.664+0000 7fad3208c640 1 --2- 172.21.15.107:0/3608707101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad2c10e0b0 0x7fad2c10e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.664+0000 7fad3208c640 1 --2- 172.21.15.107:0/3608707101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad2c110080 0x7fad2c10db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.664+0000 7fad3208c640 1 -- 172.21.15.107:0/3608707101 >> 172.21.15.107:0/3608707101 conn(0x7fad2c06c940 msgr2=0x7fad2c06cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:38.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.670+0000 7fad3208c640 1 -- 172.21.15.107:0/3608707101 shutdown_connections 2024-05-09T08:42:38.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.670+0000 7fad3208c640 1 -- 172.21.15.107:0/3608707101 wait complete. 2024-05-09T08:42:38.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.671+0000 7fad3208c640 1 Processor -- start 2024-05-09T08:42:38.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.671+0000 7fad3208c640 1 -- start start 2024-05-09T08:42:38.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.671+0000 7fad3208c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad2c10e0b0 0x7fad2c11a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.671+0000 7fad3208c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2c110080 0x7fad2c114790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.671+0000 7fad3208c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad2c11b010 0x7fad2c114cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.671+0000 7fad3208c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1110f0 con 0x7fad2c11b010 2024-05-09T08:42:38.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.671+0000 7fad3208c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c110f70 con 0x7fad2c10e0b0 2024-05-09T08:42:38.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.671+0000 7fad3208c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c111270 con 0x7fad2c110080 2024-05-09T08:42:38.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.672+0000 7fad2b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad2c10e0b0 0x7fad2c11a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:38.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.672+0000 7fad2b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad2c10e0b0 0x7fad2c11a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60714/0 (socket says 172.21.15.107:60714) 2024-05-09T08:42:38.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.672+0000 7fad2b7fe640 1 -- 172.21.15.107:0/386836558 learned_addr learned my addr 172.21.15.107:0/386836558 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:38.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.672+0000 7fad2affd640 1 --2- 172.21.15.107:0/386836558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2c110080 0x7fad2c114790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:38.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.672+0000 7fad2bfff640 1 --2- 172.21.15.107:0/386836558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad2c11b010 0x7fad2c114cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:38.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.673+0000 7fad2affd640 1 -- 172.21.15.107:0/386836558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad2c10e0b0 msgr2=0x7fad2c11a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:38.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.673+0000 7fad2affd640 1 --2- 172.21.15.107:0/386836558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad2c10e0b0 0x7fad2c11a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.673+0000 7fad2affd640 1 -- 172.21.15.107:0/386836558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad2c11b010 msgr2=0x7fad2c114cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:38.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.673+0000 7fad2affd640 1 --2- 172.21.15.107:0/386836558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad2c11b010 0x7fad2c114cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.673+0000 7fad2affd640 1 -- 172.21.15.107:0/386836558 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2c115400 con 0x7fad2c110080 2024-05-09T08:42:38.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.673+0000 7fad2affd640 1 --2- 172.21.15.107:0/386836558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2c110080 0x7fad2c114790 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fad2000b3f0 tx=0x7fad2000b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:38.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.682+0000 7fad28ff9640 1 -- 172.21.15.107:0/386836558 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20015020 con 0x7fad2c110080 2024-05-09T08:42:38.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.682+0000 7fad28ff9640 1 -- 172.21.15.107:0/386836558 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad20011ce0 con 0x7fad2c110080 2024-05-09T08:42:38.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.682+0000 7fad28ff9640 1 -- 172.21.15.107:0/386836558 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20010d30 con 0x7fad2c110080 2024-05-09T08:42:38.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.682+0000 7fad3208c640 1 -- 172.21.15.107:0/386836558 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2c1bbd90 con 0x7fad2c110080 2024-05-09T08:42:38.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.682+0000 7fad3208c640 1 -- 172.21.15.107:0/386836558 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad2c1bc1a0 con 0x7fad2c110080 2024-05-09T08:42:38.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.687+0000 7fad28ff9640 1 -- 172.21.15.107:0/386836558 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fad20018020 con 0x7fad2c110080 2024-05-09T08:42:38.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.687+0000 7fad34d98740 1 Processor -- start 2024-05-09T08:42:38.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.687+0000 7fad34d98740 1 -- start start 2024-05-09T08:42:38.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.688+0000 7fad34d98740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974280 0x56496a974660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.688+0000 7fad34d98740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a974ba0 0x56496a97ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.688+0000 7fad34d98740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a97f270 0x56496a981660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.688+0000 7fad34d98740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56496a8d9d00 con 0x56496a974280 2024-05-09T08:42:38.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.688+0000 7fad34d98740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56496a8814c0 con 0x56496a97f270 2024-05-09T08:42:38.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.688+0000 7fad34d98740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56496a8d5cc0 con 0x56496a974ba0 2024-05-09T08:42:38.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.688+0000 7fad2b7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a974ba0 0x56496a97ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:38.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.688+0000 7fad2b7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a974ba0 0x56496a97ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:58532/0 (socket says 172.21.15.107:58532) 2024-05-09T08:42:38.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.688+0000 7fad2b7fe640 1 -- 172.21.15.107:0/2678390247 learned_addr learned my addr 172.21.15.107:0/2678390247 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:38.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.688+0000 7fad2affd640 1 --2- 172.21.15.107:0/2678390247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a97f270 0x56496a981660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:38.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.689+0000 7fad2bfff640 1 --2- 172.21.15.107:0/2678390247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974280 0x56496a974660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:38.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.690+0000 7fad2bfff640 1 -- 172.21.15.107:0/2678390247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a974ba0 msgr2=0x56496a97ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:38.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.693+0000 7fad28ff9640 1 --2- 172.21.15.107:0/386836558 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fad140a2f00 0x7fad140a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.693+0000 7fad28ff9640 1 -- 172.21.15.107:0/386836558 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fad200cf1f0 con 0x7fad2c110080 2024-05-09T08:42:38.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.694+0000 7fad2bfff640 1 --2- 172.21.15.107:0/2678390247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a974ba0 0x56496a97ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.694+0000 7fad2bfff640 1 -- 172.21.15.107:0/2678390247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a97f270 msgr2=0x56496a981660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:38.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.694+0000 7fad2bfff640 1 --2- 172.21.15.107:0/2678390247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a97f270 0x56496a981660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.694+0000 7fad2bfff640 1 -- 172.21.15.107:0/2678390247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56496a6b77d0 con 0x56496a974280 2024-05-09T08:42:38.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.695+0000 7fad2bfff640 1 --2- 172.21.15.107:0/386836558 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fad140a2f00 0x7fad140a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:38.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.695+0000 7fad2b7fe640 1 --2- 172.21.15.107:0/2678390247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a974ba0 0x56496a97ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:42:38.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.695+0000 7fad2affd640 1 --2- 172.21.15.107:0/2678390247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a97f270 0x56496a981660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:42:38.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.695+0000 7fad2bfff640 1 --2- 172.21.15.107:0/2678390247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974280 0x56496a974660 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fad2402f3b0 tx=0x7fad24031510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcfe21292c71a824 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:38.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.697+0000 7fad11ffb640 1 -- 172.21.15.107:0/2678390247 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad24003c30 con 0x56496a974280 2024-05-09T08:42:38.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.697+0000 7fad11ffb640 1 -- 172.21.15.107:0/2678390247 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad24036040 con 0x56496a974280 2024-05-09T08:42:38.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.698+0000 7fad34d98740 1 -- 172.21.15.107:0/2678390247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974280 msgr2=0x56496a974660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:38.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.698+0000 7fad34d98740 1 --2- 172.21.15.107:0/2678390247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974280 0x56496a974660 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fad2402f3b0 tx=0x7fad24031510 comp rx=0 tx=0).stop 2024-05-09T08:42:38.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.698+0000 7fad2bfff640 1 --2- 172.21.15.107:0/386836558 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fad140a2f00 0x7fad140a53c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fad24006010 tx=0x7fad2404d000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:38.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.698+0000 7fad11ffb640 1 -- 172.21.15.107:0/2678390247 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad24030830 con 0x56496a974280 2024-05-09T08:42:38.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.698+0000 7fad34d98740 1 -- 172.21.15.107:0/2678390247 shutdown_connections 2024-05-09T08:42:38.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.698+0000 7fad34d98740 1 --2- 172.21.15.107:0/2678390247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a97f270 0x56496a981660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.698+0000 7fad34d98740 1 --2- 172.21.15.107:0/2678390247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a974ba0 0x56496a97ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.698+0000 7fad34d98740 1 --2- 172.21.15.107:0/2678390247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974280 0x56496a974660 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.698+0000 7fad34d98740 1 -- 172.21.15.107:0/2678390247 >> 172.21.15.107:0/2678390247 conn(0x56496a9705a0 msgr2=0x56496a970980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:38.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.702+0000 7fad34d98740 1 -- 172.21.15.107:0/2678390247 shutdown_connections 2024-05-09T08:42:38.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.702+0000 7fad34d98740 1 -- 172.21.15.107:0/2678390247 wait complete. 2024-05-09T08:42:38.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.702+0000 7fad34d98740 1 -- start start 2024-05-09T08:42:38.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.703+0000 7fad34d98740 1 Processor -- start 2024-05-09T08:42:38.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.703+0000 7fad34d98740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a974280 0x56496a97c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.703+0000 7fad34d98740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 0x56496a97c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.703+0000 7fad34d98740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a97f270 0x56496a97cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.703+0000 7fad34d98740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56496a9820b0 con 0x56496a974ba0 2024-05-09T08:42:38.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.703+0000 7fad34d98740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56496a894b90 con 0x56496a974280 2024-05-09T08:42:38.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.703+0000 7fad34d98740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56496a982230 con 0x56496a97f270 2024-05-09T08:42:38.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.703+0000 7fad34d98740 10 client.0 _ll_register_callbacks cb 0x56496a8d0da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:38.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.703+0000 7fad2b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a974280 0x56496a97c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:38.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.703+0000 7fad2b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a974280 0x56496a97c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60724/0 (socket says 172.21.15.107:60724) 2024-05-09T08:42:38.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.703+0000 7fad2b7fe640 1 -- 172.21.15.107:0/604846633 learned_addr learned my addr 172.21.15.107:0/604846633 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:38.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.704+0000 7fad2bfff640 1 --2- 172.21.15.107:0/604846633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a97f270 0x56496a97cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:38.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.704+0000 7fad2affd640 1 --2- 172.21.15.107:0/604846633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 0x56496a97c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:38.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.704+0000 7fad2bfff640 1 -- 172.21.15.107:0/604846633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a974280 msgr2=0x56496a97c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:38.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.704+0000 7fad2bfff640 1 --2- 172.21.15.107:0/604846633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a974280 0x56496a97c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.704+0000 7fad2bfff640 1 -- 172.21.15.107:0/604846633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 msgr2=0x56496a97c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:38.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.704+0000 7fad2bfff640 1 --2- 172.21.15.107:0/604846633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 0x56496a97c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.704+0000 7fad2bfff640 1 -- 172.21.15.107:0/604846633 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56496a5ed3f0 con 0x56496a97f270 2024-05-09T08:42:38.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.705+0000 7fad2affd640 1 --2- 172.21.15.107:0/604846633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 0x56496a97c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:42:38.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.705+0000 7fad2bfff640 1 --2- 172.21.15.107:0/604846633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a97f270 0x56496a97cbd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fad24049040 tx=0x7fad240095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58892816472a7b6b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:38.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.706+0000 7fad0affd640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:42:38.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.706+0000 7fad0affd640 1 -- 172.21.15.107:0/604846633 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2405f060 con 0x56496a97f270 2024-05-09T08:42:38.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.706+0000 7fad0affd640 1 -- 172.21.15.107:0/604846633 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fad24030aa0 con 0x56496a97f270 2024-05-09T08:42:38.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.706+0000 7fad34d98740 1 -- 172.21.15.107:0/604846633 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad24036040 con 0x56496a97f270 2024-05-09T08:42:38.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.706+0000 7fad0affd640 1 -- 172.21.15.107:0/604846633 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fad24030ca0 con 0x56496a97f270 2024-05-09T08:42:38.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.707+0000 7fad34d98740 10 client.27505 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:38.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.707+0000 7fad34d98740 1 -- 172.21.15.107:0/604846633 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56496a5ad0f0 con 0x56496a97f270 2024-05-09T08:42:38.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.707+0000 7fad0affd640 1 -- 172.21.15.107:0/604846633 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad24036040 con 0x56496a97f270 2024-05-09T08:42:38.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.708+0000 7fad0affd640 1 -- 172.21.15.107:0/604846633 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fad2403c410 con 0x56496a97f270 2024-05-09T08:42:38.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.708+0000 7fad34d98740 10 client.27505 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:38.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.708+0000 7fad34d98740 10 client.27505 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:38.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.708+0000 7fad34d98740 20 client.27505 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:38.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.708+0000 7fad34d98740 1 --2- 172.21.15.107:0/604846633 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56496a98f580 0x56496a9919c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.708+0000 7fad34d98740 4 client.27505 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:38.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.708+0000 7fad34d98740 1 -- 172.21.15.107:0/604846633 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56496a8e7430 con 0x56496a98f580 2024-05-09T08:42:38.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.708+0000 7fad2b7fe640 1 --2- 172.21.15.107:0/604846633 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56496a98f580 0x56496a9919c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:38.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.712+0000 7fad2b7fe640 1 --2- 172.21.15.107:0/604846633 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56496a98f580 0x56496a9919c0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80958fc9b3d3862 server_cookie=dc5ac961274e8d34 in_seq=0 out_seq=0 2024-05-09T08:42:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.717+0000 7fad0affd640 10 client.27505 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.717+0000 7fad0affd640 1 -- 172.21.15.107:0/604846633 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56496a8e7430 con 0x56496a98f580 2024-05-09T08:42:38.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.718+0000 7fad0affd640 10 client.27505 handle_command_reply: tid=0 2024-05-09T08:42:38.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.719+0000 7fad34d98740 1 client.27505 shutdown 2024-05-09T08:42:38.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.720+0000 7fad34d98740 1 -- 172.21.15.107:0/604846633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a97f270 msgr2=0x56496a97cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:38.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.720+0000 7fad34d98740 1 --2- 172.21.15.107:0/604846633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a97f270 0x56496a97cbd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fad24049040 tx=0x7fad240095c0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.720+0000 7fad34d98740 1 -- 172.21.15.107:0/604846633 shutdown_connections 2024-05-09T08:42:38.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.720+0000 7fad34d98740 1 --2- 172.21.15.107:0/604846633 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56496a98f580 0x56496a9919c0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.720+0000 7fad34d98740 1 --2- 172.21.15.107:0/604846633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a97f270 0x56496a97cbd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.720+0000 7fad34d98740 1 --2- 172.21.15.107:0/604846633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 0x56496a97c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.720+0000 7fad34d98740 1 --2- 172.21.15.107:0/604846633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a974280 0x56496a97c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.720+0000 7fad34d98740 1 -- 172.21.15.107:0/604846633 >> 172.21.15.107:0/604846633 conn(0x56496a9705a0 msgr2=0x56496a96f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:38.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.720+0000 7fad0affd640 0 client.27505 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:38.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.721+0000 7fad34d98740 1 -- 172.21.15.107:0/604846633 shutdown_connections 2024-05-09T08:42:38.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.721+0000 7fad34d98740 1 -- 172.21.15.107:0/604846633 wait complete. 2024-05-09T08:42:38.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.721+0000 7fad34d98740 20 client.27505 trim_cache size 0 max 16384 2024-05-09T08:42:38.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.776+0000 7fad34d98740 1 Processor -- start 2024-05-09T08:42:38.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.776+0000 7fad34d98740 1 -- start start 2024-05-09T08:42:38.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.777+0000 7fad34d98740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a974280 0x56496a98b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.777+0000 7fad34d98740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 0x56496a96ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.777+0000 7fad34d98740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a97f270 0x56496a97aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.778+0000 7fad34d98740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fad18012910 con 0x56496a974ba0 2024-05-09T08:42:38.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.778+0000 7fad34d98740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fad180101e0 con 0x56496a974280 2024-05-09T08:42:38.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.778+0000 7fad34d98740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fad1800e050 con 0x56496a97f270 2024-05-09T08:42:38.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.778+0000 7fad2affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 0x56496a96ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:38.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.778+0000 7fad2affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 0x56496a96ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40264/0 (socket says 172.21.15.107:40264) 2024-05-09T08:42:38.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.778+0000 7fad2affd640 1 -- 172.21.15.107:0/1063724623 learned_addr learned my addr 172.21.15.107:0/1063724623 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:38.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.779+0000 7fad2affd640 1 -- 172.21.15.107:0/1063724623 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a97f270 msgr2=0x56496a97aa80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:42:38.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.779+0000 7fad2affd640 1 --2- 172.21.15.107:0/1063724623 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a97f270 0x56496a97aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.779+0000 7fad2affd640 1 -- 172.21.15.107:0/1063724623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a974280 msgr2=0x56496a98b020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:42:38.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.779+0000 7fad2affd640 1 --2- 172.21.15.107:0/1063724623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a974280 0x56496a98b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.779+0000 7fad2affd640 1 -- 172.21.15.107:0/1063724623 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56496a9972d0 con 0x56496a974ba0 2024-05-09T08:42:38.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.779+0000 7fad2affd640 1 --2- 172.21.15.107:0/1063724623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 0x56496a96ff60 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fad2000bdb0 tx=0x7fad200d2780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7adf030c23080d65 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:38.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.781+0000 7fad117fa640 1 -- 172.21.15.107:0/1063724623 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20010ed0 con 0x56496a974ba0 2024-05-09T08:42:38.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.781+0000 7fad117fa640 1 -- 172.21.15.107:0/1063724623 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad2009bb10 con 0x56496a974ba0 2024-05-09T08:42:38.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.781+0000 7fad117fa640 1 -- 172.21.15.107:0/1063724623 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20097a80 con 0x56496a974ba0 2024-05-09T08:42:38.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.784+0000 7fad34d98740 1 -- 172.21.15.107:0/1063724623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 msgr2=0x56496a96ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:38.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.784+0000 7fad34d98740 1 --2- 172.21.15.107:0/1063724623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 0x56496a96ff60 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fad2000bdb0 tx=0x7fad200d2780 comp rx=0 tx=0).stop 2024-05-09T08:42:38.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.788+0000 7fad34d98740 1 -- 172.21.15.107:0/1063724623 shutdown_connections 2024-05-09T08:42:38.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.788+0000 7fad34d98740 1 --2- 172.21.15.107:0/1063724623 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a97f270 0x56496a97aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.788+0000 7fad34d98740 1 --2- 172.21.15.107:0/1063724623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 0x56496a96ff60 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.788+0000 7fad34d98740 1 --2- 172.21.15.107:0/1063724623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a974280 0x56496a98b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.788+0000 7fad34d98740 1 -- 172.21.15.107:0/1063724623 >> 172.21.15.107:0/1063724623 conn(0x56496a9705a0 msgr2=0x56496a9761d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:38.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.794+0000 7fad34d98740 1 -- 172.21.15.107:0/1063724623 shutdown_connections 2024-05-09T08:42:38.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.795+0000 7fad34d98740 1 -- 172.21.15.107:0/1063724623 wait complete. 2024-05-09T08:42:38.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.795+0000 7fad34d98740 1 -- start start 2024-05-09T08:42:38.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.795+0000 7fad34d98740 1 Processor -- start 2024-05-09T08:42:38.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.800+0000 7fad34d98740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a974280 0x56496a979590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.800+0000 7fad34d98740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 0x56496a979ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.800+0000 7fad34d98740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a97f270 0x56496a9f4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.800+0000 7fad34d98740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fad1800e6a0 con 0x56496a974ba0 2024-05-09T08:42:38.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.800+0000 7fad34d98740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fad1800fb10 con 0x56496a97f270 2024-05-09T08:42:38.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.800+0000 7fad34d98740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fad1800ed70 con 0x56496a974280 2024-05-09T08:42:38.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.800+0000 7fad34d98740 10 client.0 _ll_register_callbacks cb 0x7fad200d2650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:38.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.800+0000 7fad2bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a97f270 0x56496a9f4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:38.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.800+0000 7fad2bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a97f270 0x56496a9f4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60728/0 (socket says 172.21.15.107:60728) 2024-05-09T08:42:38.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.800+0000 7fad2bfff640 1 -- 172.21.15.107:0/380204912 learned_addr learned my addr 172.21.15.107:0/380204912 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:38.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.800+0000 7fad2b7fe640 1 --2- 172.21.15.107:0/380204912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a974280 0x56496a979590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:38.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.801+0000 7fad2b7fe640 1 -- 172.21.15.107:0/380204912 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a97f270 msgr2=0x56496a9f4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:38.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.801+0000 7fad2b7fe640 1 --2- 172.21.15.107:0/380204912 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a97f270 0x56496a9f4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.801+0000 7fad2b7fe640 1 -- 172.21.15.107:0/380204912 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 msgr2=0x56496a979ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:38.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.801+0000 7fad2b7fe640 1 --2- 172.21.15.107:0/380204912 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 0x56496a979ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.801+0000 7fad2b7fe640 1 -- 172.21.15.107:0/380204912 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56496a9f5080 con 0x56496a974280 2024-05-09T08:42:38.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.803+0000 7fad2b7fe640 1 --2- 172.21.15.107:0/380204912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a974280 0x56496a979590 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fad1c000c00 tx=0x7fad1c01cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5714bc20ca4c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:38.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.803+0000 7fad0affd640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:42:38.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.803+0000 7fad0affd640 1 -- 172.21.15.107:0/380204912 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c043070 con 0x56496a974280 2024-05-09T08:42:38.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.803+0000 7fad0affd640 1 -- 172.21.15.107:0/380204912 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fad1c00f040 con 0x56496a974280 2024-05-09T08:42:38.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.803+0000 7fad34d98740 1 -- 172.21.15.107:0/380204912 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad2009bb10 con 0x56496a974280 2024-05-09T08:42:38.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.808+0000 7fad0affd640 1 -- 172.21.15.107:0/380204912 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fad1c04f050 con 0x56496a974280 2024-05-09T08:42:38.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.808+0000 7fad34d98740 10 client.27508 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:38.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.808+0000 7fad34d98740 1 -- 172.21.15.107:0/380204912 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56496a9f63f0 con 0x56496a974280 2024-05-09T08:42:38.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.810+0000 7fad0affd640 1 -- 172.21.15.107:0/380204912 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad2009bb10 con 0x56496a974280 2024-05-09T08:42:38.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.810+0000 7fad0affd640 1 -- 172.21.15.107:0/380204912 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fad1c0473b0 con 0x56496a974280 2024-05-09T08:42:38.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.810+0000 7fad34d98740 10 client.27508 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:38.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.810+0000 7fad34d98740 10 client.27508 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:38.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.810+0000 7fad34d98740 20 client.27508 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:38.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.810+0000 7fad34d98740 1 --2- 172.21.15.107:0/380204912 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56496a9f6ac0 0x56496a9f8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:38.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.810+0000 7fad34d98740 4 client.27508 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:42:38.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.810+0000 7fad34d98740 1 -- 172.21.15.107:0/380204912 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56496a8e1780 con 0x56496a9f6ac0 2024-05-09T08:42:38.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.813+0000 7fad2affd640 1 --2- 172.21.15.107:0/380204912 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56496a9f6ac0 0x56496a9f8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:38.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.814+0000 7fad2affd640 1 --2- 172.21.15.107:0/380204912 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56496a9f6ac0 0x56496a9f8ee0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29a83ec422c11eaf server_cookie=970b60477b90d1a5 in_seq=0 out_seq=0 2024-05-09T08:42:38.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.814+0000 7fad0affd640 10 client.27508 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:38.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.816+0000 7fad0affd640 1 -- 172.21.15.107:0/380204912 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56496a8e1780 con 0x56496a9f6ac0 2024-05-09T08:42:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.816+0000 7fad0affd640 10 client.27508 handle_command_reply: tid=0 2024-05-09T08:42:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.816+0000 7fad34d98740 1 client.27508 shutdown 2024-05-09T08:42:38.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.827+0000 7fad34d98740 1 -- 172.21.15.107:0/380204912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a974280 msgr2=0x56496a979590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:38.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.827+0000 7fad34d98740 1 --2- 172.21.15.107:0/380204912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a974280 0x56496a979590 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fad1c000c00 tx=0x7fad1c01cf20 comp rx=0 tx=0).stop 2024-05-09T08:42:38.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.832+0000 7fad34d98740 1 -- 172.21.15.107:0/380204912 shutdown_connections 2024-05-09T08:42:38.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.832+0000 7fad34d98740 1 --2- 172.21.15.107:0/380204912 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56496a9f6ac0 0x56496a9f8ee0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.832+0000 7fad34d98740 1 --2- 172.21.15.107:0/380204912 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56496a97f270 0x56496a9f4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.832+0000 7fad34d98740 1 --2- 172.21.15.107:0/380204912 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56496a974ba0 0x56496a979ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.832+0000 7fad34d98740 1 --2- 172.21.15.107:0/380204912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56496a974280 0x56496a979590 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.832+0000 7fad34d98740 1 -- 172.21.15.107:0/380204912 >> 172.21.15.107:0/380204912 conn(0x56496a9705a0 msgr2=0x56496a97c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:38.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.834+0000 7fad0affd640 0 client.27508 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:38.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.839+0000 7fad34d98740 1 -- 172.21.15.107:0/380204912 shutdown_connections 2024-05-09T08:42:38.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.839+0000 7fad34d98740 1 -- 172.21.15.107:0/380204912 wait complete. 2024-05-09T08:42:38.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.842+0000 7fad34d98740 20 client.27508 trim_cache size 0 max 16384 2024-05-09T08:42:38.844 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:42:38.844 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:42:38.844 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:42:38.844 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:42:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.848+0000 7fad3208c640 1 -- 172.21.15.107:0/386836558 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fad140a2f00 msgr2=0x7fad140a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.848+0000 7fad3208c640 1 --2- 172.21.15.107:0/386836558 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fad140a2f00 0x7fad140a53c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fad24006010 tx=0x7fad2404d000 comp rx=0 tx=0).stop 2024-05-09T08:42:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.848+0000 7fad3208c640 1 -- 172.21.15.107:0/386836558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2c110080 msgr2=0x7fad2c114790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.849+0000 7fad3208c640 1 --2- 172.21.15.107:0/386836558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2c110080 0x7fad2c114790 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fad2000b3f0 tx=0x7fad2000b8b0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.857+0000 7fad3208c640 1 -- 172.21.15.107:0/386836558 shutdown_connections 2024-05-09T08:42:38.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.857+0000 7fad3208c640 1 --2- 172.21.15.107:0/386836558 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fad140a2f00 0x7fad140a53c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.857+0000 7fad3208c640 1 --2- 172.21.15.107:0/386836558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad2c11b010 0x7fad2c114cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.857+0000 7fad3208c640 1 --2- 172.21.15.107:0/386836558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2c110080 0x7fad2c114790 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.857+0000 7fad3208c640 1 --2- 172.21.15.107:0/386836558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad2c10e0b0 0x7fad2c11a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:38.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.857+0000 7fad3208c640 1 -- 172.21.15.107:0/386836558 >> 172.21.15.107:0/386836558 conn(0x7fad2c06c940 msgr2=0x7fad2c11c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.860+0000 7fad3208c640 1 -- 172.21.15.107:0/386836558 shutdown_connections 2024-05-09T08:42:38.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:38.860+0000 7fad3208c640 1 -- 172.21.15.107:0/386836558 wait complete. 2024-05-09T08:42:38.904 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:42:38.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:42:39.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.248+0000 7fcd02a72640 1 -- 172.21.15.107:0/719985765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc10e430 msgr2=0x7fccfc11d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:39.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.248+0000 7fcd02a72640 1 --2- 172.21.15.107:0/719985765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc10e430 0x7fccfc11d020 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fccf4009be0 tx=0x7fccf402a710 comp rx=0 tx=0).stop 2024-05-09T08:42:39.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.252+0000 7fcd02a72640 1 -- 172.21.15.107:0/719985765 shutdown_connections 2024-05-09T08:42:39.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.252+0000 7fcd02a72640 1 --2- 172.21.15.107:0/719985765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc10e430 0x7fccfc11d020 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:39.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.252+0000 7fcd02a72640 1 --2- 172.21.15.107:0/719985765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccfc10d910 0x7fccfc10dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:39.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.252+0000 7fcd02a72640 1 --2- 172.21.15.107:0/719985765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc10fdb0 0x7fccfc110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:39.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.252+0000 7fcd02a72640 1 -- 172.21.15.107:0/719985765 >> 172.21.15.107:0/719985765 conn(0x7fccfc06c740 msgr2=0x7fccfc06cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:39.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.257+0000 7fcd02a72640 1 -- 172.21.15.107:0/719985765 shutdown_connections 2024-05-09T08:42:39.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.259+0000 7fcd02a72640 1 -- 172.21.15.107:0/719985765 wait complete. 2024-05-09T08:42:39.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.259+0000 7fcd02a72640 1 Processor -- start 2024-05-09T08:42:39.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.262+0000 7fcd02a72640 1 -- start start 2024-05-09T08:42:39.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.262+0000 7fcd02a72640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc10d910 0x7fccfc1ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:39.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.262+0000 7fcd02a72640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccfc10e430 0x7fccfc1ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:39.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.262+0000 7fcd02a72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc10fdb0 0x7fccfc1a6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:39.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.262+0000 7fcd02a72640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc120710 con 0x7fccfc10fdb0 2024-05-09T08:42:39.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.262+0000 7fcd02a72640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc120590 con 0x7fccfc10d910 2024-05-09T08:42:39.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.262+0000 7fcd02a72640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc120890 con 0x7fccfc10e430 2024-05-09T08:42:39.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.262+0000 7fcd01a70640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc10d910 0x7fccfc1ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:39.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.262+0000 7fcd01a70640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc10d910 0x7fccfc1ac940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60740/0 (socket says 172.21.15.107:60740) 2024-05-09T08:42:39.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.262+0000 7fcd01a70640 1 -- 172.21.15.107:0/2540595904 learned_addr learned my addr 172.21.15.107:0/2540595904 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:39.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.263+0000 7fcd0126f640 1 --2- 172.21.15.107:0/2540595904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccfc10e430 0x7fccfc1ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:39.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.263+0000 7fcd0126f640 1 -- 172.21.15.107:0/2540595904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc10d910 msgr2=0x7fccfc1ac940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:39.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.263+0000 7fcd0126f640 1 --2- 172.21.15.107:0/2540595904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc10d910 0x7fccfc1ac940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:39.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.263+0000 7fcd0126f640 1 -- 172.21.15.107:0/2540595904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc10fdb0 msgr2=0x7fccfc1a6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:39.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.263+0000 7fcd0126f640 1 --2- 172.21.15.107:0/2540595904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc10fdb0 0x7fccfc1a6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:39.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.263+0000 7fcd0126f640 1 -- 172.21.15.107:0/2540595904 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccfc1a7240 con 0x7fccfc10e430 2024-05-09T08:42:39.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.268+0000 7fcd0126f640 1 --2- 172.21.15.107:0/2540595904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccfc10e430 0x7fccfc1ace80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fccec00b3e0 tx=0x7fccec00b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:39.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.270+0000 7fccf2ffd640 1 -- 172.21.15.107:0/2540595904 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec015020 con 0x7fccfc10e430 2024-05-09T08:42:39.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.270+0000 7fcd02a72640 1 -- 172.21.15.107:0/2540595904 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccfc1a7530 con 0x7fccfc10e430 2024-05-09T08:42:39.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.270+0000 7fcd02a72640 1 -- 172.21.15.107:0/2540595904 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccfc1107f0 con 0x7fccfc10e430 2024-05-09T08:42:39.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.270+0000 7fccf2ffd640 1 -- 172.21.15.107:0/2540595904 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fccec011ce0 con 0x7fccfc10e430 2024-05-09T08:42:39.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.270+0000 7fccf2ffd640 1 -- 172.21.15.107:0/2540595904 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec010d80 con 0x7fccfc10e430 2024-05-09T08:42:39.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.279+0000 7fccf2ffd640 1 -- 172.21.15.107:0/2540595904 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fccec018020 con 0x7fccfc10e430 2024-05-09T08:42:39.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.280+0000 7fccf2ffd640 1 --2- 172.21.15.107:0/2540595904 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcce80a2f00 0x7fcce80a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:39.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.281+0000 7fcd01a70640 1 --2- 172.21.15.107:0/2540595904 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcce80a2f00 0x7fcce80a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:39.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.281+0000 7fccf2ffd640 1 -- 172.21.15.107:0/2540595904 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fccec0cee40 con 0x7fccfc10e430 2024-05-09T08:42:39.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.281+0000 7fcd02a72640 1 -- 172.21.15.107:0/2540595904 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcce0005290 con 0x7fccfc10e430 2024-05-09T08:42:39.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.285+0000 7fcd01a70640 1 --2- 172.21.15.107:0/2540595904 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcce80a2f00 0x7fcce80a53c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fccf80058c0 tx=0x7fccf8009260 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:39.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.296+0000 7fccf2ffd640 1 -- 172.21.15.107:0/2540595904 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fccec01f020 con 0x7fccfc10e430 2024-05-09T08:42:39.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.861+0000 7fcd02a72640 1 -- 172.21.15.107:0/2540595904 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcce0005850 con 0x7fccfc10e430 2024-05-09T08:42:39.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.863+0000 7fccf2ffd640 1 -- 172.21.15.107:0/2540595904 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11887 (secure 0 0 0) 0x7fccec09b760 con 0x7fccfc10e430 2024-05-09T08:42:39.864 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:42:39.864 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:37:10.930317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:42:39.864 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2024-05-09T08:42:39.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.871+0000 7fcd02a72640 1 -- 172.21.15.107:0/2540595904 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcce80a2f00 msgr2=0x7fcce80a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:39.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.872+0000 7fcd02a72640 1 --2- 172.21.15.107:0/2540595904 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcce80a2f00 0x7fcce80a53c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fccf80058c0 tx=0x7fccf8009260 comp rx=0 tx=0).stop 2024-05-09T08:42:39.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.872+0000 7fcd02a72640 1 -- 172.21.15.107:0/2540595904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccfc10e430 msgr2=0x7fccfc1ace80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:39.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.872+0000 7fcd02a72640 1 --2- 172.21.15.107:0/2540595904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccfc10e430 0x7fccfc1ace80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fccec00b3e0 tx=0x7fccec00b8a0 comp rx=0 tx=0).stop 2024-05-09T08:42:39.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.872+0000 7fcd02a72640 1 -- 172.21.15.107:0/2540595904 shutdown_connections 2024-05-09T08:42:39.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.872+0000 7fcd02a72640 1 --2- 172.21.15.107:0/2540595904 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcce80a2f00 0x7fcce80a53c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:39.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.872+0000 7fcd02a72640 1 --2- 172.21.15.107:0/2540595904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc10fdb0 0x7fccfc1a6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:39.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.872+0000 7fcd02a72640 1 --2- 172.21.15.107:0/2540595904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccfc10e430 0x7fccfc1ace80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:39.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.872+0000 7fcd02a72640 1 --2- 172.21.15.107:0/2540595904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc10d910 0x7fccfc1ac940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:39.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.872+0000 7fcd02a72640 1 -- 172.21.15.107:0/2540595904 >> 172.21.15.107:0/2540595904 conn(0x7fccfc06c740 msgr2=0x7fccfc070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:39.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.874+0000 7fcd02a72640 1 -- 172.21.15.107:0/2540595904 shutdown_connections 2024-05-09T08:42:39.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:39.874+0000 7fcd02a72640 1 -- 172.21.15.107:0/2540595904 wait complete. 2024-05-09T08:42:39.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:42:40.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.146+0000 7f7d8ffff640 1 -- 172.21.15.107:0/3589615722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d901092b0 msgr2=0x7f7d9011d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.146+0000 7f7d8ffff640 1 --2- 172.21.15.107:0/3589615722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d901092b0 0x7f7d9011d020 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7d88009ca0 tx=0x7f7d8802a980 comp rx=0 tx=0).stop 2024-05-09T08:42:40.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.147+0000 7f7d8ffff640 1 -- 172.21.15.107:0/3589615722 shutdown_connections 2024-05-09T08:42:40.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.147+0000 7f7d8ffff640 1 --2- 172.21.15.107:0/3589615722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d901092b0 0x7f7d9011d020 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.147+0000 7f7d8ffff640 1 --2- 172.21.15.107:0/3589615722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d90108830 0x7f7d90108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.147+0000 7f7d8ffff640 1 --2- 172.21.15.107:0/3589615722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d9010fed0 0x7f7d901102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.147+0000 7f7d8ffff640 1 -- 172.21.15.107:0/3589615722 >> 172.21.15.107:0/3589615722 conn(0x7f7d9006c7b0 msgr2=0x7f7d9006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.147+0000 7f7d8ffff640 1 -- 172.21.15.107:0/3589615722 shutdown_connections 2024-05-09T08:42:40.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.147+0000 7f7d8ffff640 1 -- 172.21.15.107:0/3589615722 wait complete. 2024-05-09T08:42:40.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.148+0000 7f7d8ffff640 1 Processor -- start 2024-05-09T08:42:40.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.148+0000 7f7d8ffff640 1 -- start start 2024-05-09T08:42:40.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.149+0000 7f7d8ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d90108830 0x7f7d9011cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.149+0000 7f7d8ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d901092b0 0x7f7d9011d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.149+0000 7f7d8ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d9010fed0 0x7f7d90115db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.149+0000 7f7d8ffff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7d901207e0 con 0x7f7d90108830 2024-05-09T08:42:40.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.149+0000 7f7d8ffff640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7d90120660 con 0x7f7d901092b0 2024-05-09T08:42:40.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.149+0000 7f7d8ffff640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7d90120960 con 0x7f7d9010fed0 2024-05-09T08:42:40.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.149+0000 7f7d8e7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d901092b0 0x7f7d9011d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.150+0000 7f7d8f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d9010fed0 0x7f7d90115db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.150+0000 7f7d8f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d9010fed0 0x7f7d90115db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:58602/0 (socket says 172.21.15.107:58602) 2024-05-09T08:42:40.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.150+0000 7f7d8f7fe640 1 -- 172.21.15.107:0/2385645943 learned_addr learned my addr 172.21.15.107:0/2385645943 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:40.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.150+0000 7f7d8effd640 1 --2- 172.21.15.107:0/2385645943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d90108830 0x7f7d9011cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.150+0000 7f7d8f7fe640 1 -- 172.21.15.107:0/2385645943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d901092b0 msgr2=0x7f7d9011d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.150+0000 7f7d8f7fe640 1 --2- 172.21.15.107:0/2385645943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d901092b0 0x7f7d9011d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.150+0000 7f7d8f7fe640 1 -- 172.21.15.107:0/2385645943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d90108830 msgr2=0x7f7d9011cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.150+0000 7f7d8f7fe640 1 --2- 172.21.15.107:0/2385645943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d90108830 0x7f7d9011cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.150+0000 7f7d8f7fe640 1 -- 172.21.15.107:0/2385645943 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d90116670 con 0x7f7d9010fed0 2024-05-09T08:42:40.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.151+0000 7f7d8f7fe640 1 --2- 172.21.15.107:0/2385645943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d9010fed0 0x7f7d90115db0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7d8802af70 tx=0x7f7d880308b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.156+0000 7f7d948ec640 1 -- 172.21.15.107:0/2385645943 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d88034050 con 0x7f7d9010fed0 2024-05-09T08:42:40.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.157+0000 7f7d948ec640 1 -- 172.21.15.107:0/2385645943 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d88004040 con 0x7f7d9010fed0 2024-05-09T08:42:40.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.157+0000 7f7d948ec640 1 -- 172.21.15.107:0/2385645943 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8802e7f0 con 0x7f7d9010fed0 2024-05-09T08:42:40.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.157+0000 7f7d8ffff640 1 -- 172.21.15.107:0/2385645943 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d901168a0 con 0x7f7d9010fed0 2024-05-09T08:42:40.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.157+0000 7f7d8ffff640 1 -- 172.21.15.107:0/2385645943 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d9011a850 con 0x7f7d9010fed0 2024-05-09T08:42:40.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.162+0000 7f7d985fc740 1 Processor -- start 2024-05-09T08:42:40.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.162+0000 7f7d985fc740 1 -- start start 2024-05-09T08:42:40.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.162+0000 7f7d985fc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785cb520 0x560e785cb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.162+0000 7f7d985fc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 0x560e785d5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.162+0000 7f7d985fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785d6510 0x560e785d8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.162+0000 7f7d985fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560e78530c40 con 0x560e785d6510 2024-05-09T08:42:40.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.162+0000 7f7d985fc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560e784d8580 con 0x560e785cbe40 2024-05-09T08:42:40.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.162+0000 7f7d985fc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560e7852cc00 con 0x560e785cb520 2024-05-09T08:42:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.163+0000 7f7d8e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785d6510 0x560e785d8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.163+0000 7f7d8e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785d6510 0x560e785d8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40306/0 (socket says 172.21.15.107:40306) 2024-05-09T08:42:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.163+0000 7f7d8e7fc640 1 -- 172.21.15.107:0/2056727580 learned_addr learned my addr 172.21.15.107:0/2056727580 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.163+0000 7f7d8e7fc640 1 --2- 172.21.15.107:0/2056727580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785cb520 0x560e785cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.165+0000 7f7d8effd640 1 --2- 172.21.15.107:0/2056727580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 0x560e785d5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.165+0000 7f7d948ec640 1 -- 172.21.15.107:0/2385645943 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7d88037020 con 0x7f7d9010fed0 2024-05-09T08:42:40.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.166+0000 7f7d8e7fc640 1 -- 172.21.15.107:0/2056727580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 msgr2=0x560e785d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.166+0000 7f7d8e7fc640 1 --2- 172.21.15.107:0/2056727580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 0x560e785d5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.166+0000 7f7d8e7fc640 1 -- 172.21.15.107:0/2056727580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785d6510 msgr2=0x560e785d8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.166+0000 7f7d8e7fc640 1 --2- 172.21.15.107:0/2056727580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785d6510 0x560e785d8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.166+0000 7f7d8e7fc640 1 -- 172.21.15.107:0/2056727580 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e782443f0 con 0x560e785cb520 2024-05-09T08:42:40.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.166+0000 7f7d8e7fc640 1 --2- 172.21.15.107:0/2056727580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785cb520 0x560e785cb900 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000ced0 tx=0x7f7d80006b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be07f0c7fe84c1ed server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.170+0000 7f7d948ec640 1 --2- 172.21.15.107:0/2385645943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7d780a2f10 0x7f7d780a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.168+0000 7f7d75ffb640 1 -- 172.21.15.107:0/2056727580 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8001b070 con 0x560e785cb520 2024-05-09T08:42:40.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.170+0000 7f7d948ec640 1 -- 172.21.15.107:0/2385645943 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f7d880f08a0 con 0x7f7d9010fed0 2024-05-09T08:42:40.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.170+0000 7f7d75ffb640 1 -- 172.21.15.107:0/2056727580 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d80016aa0 con 0x560e785cb520 2024-05-09T08:42:40.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.170+0000 7f7d75ffb640 1 -- 172.21.15.107:0/2056727580 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8000f040 con 0x560e785cb520 2024-05-09T08:42:40.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.171+0000 7f7d8effd640 1 --2- 172.21.15.107:0/2385645943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7d780a2f10 0x7f7d780a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.171+0000 7f7d985fc740 1 -- 172.21.15.107:0/2056727580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785cb520 msgr2=0x560e785cb900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.171+0000 7f7d985fc740 1 --2- 172.21.15.107:0/2056727580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785cb520 0x560e785cb900 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000ced0 tx=0x7f7d80006b20 comp rx=0 tx=0).stop 2024-05-09T08:42:40.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.172+0000 7f7d985fc740 1 -- 172.21.15.107:0/2056727580 shutdown_connections 2024-05-09T08:42:40.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.172+0000 7f7d985fc740 1 --2- 172.21.15.107:0/2056727580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785d6510 0x560e785d8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.172+0000 7f7d985fc740 1 --2- 172.21.15.107:0/2056727580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 0x560e785d5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.172+0000 7f7d985fc740 1 --2- 172.21.15.107:0/2056727580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785cb520 0x560e785cb900 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.172+0000 7f7d985fc740 1 -- 172.21.15.107:0/2056727580 >> 172.21.15.107:0/2056727580 conn(0x560e785c7840 msgr2=0x560e785c7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.174+0000 7f7d985fc740 1 -- 172.21.15.107:0/2056727580 shutdown_connections 2024-05-09T08:42:40.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.174+0000 7f7d985fc740 1 -- 172.21.15.107:0/2056727580 wait complete. 2024-05-09T08:42:40.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.174+0000 7f7d985fc740 1 -- start start 2024-05-09T08:42:40.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.175+0000 7f7d985fc740 1 Processor -- start 2024-05-09T08:42:40.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.175+0000 7f7d985fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 0x560e785d34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.175+0000 7f7d985fc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 0x560e785d3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.175+0000 7f7d985fc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785d6510 0x560e785d3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.175+0000 7f7d985fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560e785d9350 con 0x560e785cb520 2024-05-09T08:42:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.175+0000 7f7d985fc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560e784ebc40 con 0x560e785cbe40 2024-05-09T08:42:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.175+0000 7f7d985fc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560e785d94d0 con 0x560e785d6510 2024-05-09T08:42:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.175+0000 7f7d8effd640 1 --2- 172.21.15.107:0/2385645943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7d780a2f10 0x7f7d780a53d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7d84005fd0 tx=0x7f7d8400b650 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.175+0000 7f7d985fc740 10 client.0 _ll_register_callbacks cb 0x560e78243be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.175+0000 7f7d8e7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 0x560e785d3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.175+0000 7f7d8e7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 0x560e785d3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60782/0 (socket says 172.21.15.107:60782) 2024-05-09T08:42:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.175+0000 7f7d8e7fc640 1 -- 172.21.15.107:0/2526827199 learned_addr learned my addr 172.21.15.107:0/2526827199 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.175+0000 7f7d8effd640 1 --2- 172.21.15.107:0/2526827199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 0x560e785d34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.175+0000 7f7d8f7fe640 1 --2- 172.21.15.107:0/2526827199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785d6510 0x560e785d3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.176+0000 7f7d8effd640 1 -- 172.21.15.107:0/2526827199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785d6510 msgr2=0x560e785d3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.176+0000 7f7d8effd640 1 --2- 172.21.15.107:0/2526827199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785d6510 0x560e785d3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.176+0000 7f7d8effd640 1 -- 172.21.15.107:0/2526827199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 msgr2=0x560e785d3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.176+0000 7f7d8effd640 1 --2- 172.21.15.107:0/2526827199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 0x560e785d3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.176+0000 7f7d8effd640 1 -- 172.21.15.107:0/2526827199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e7830e7d0 con 0x560e785cb520 2024-05-09T08:42:40.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.176+0000 7f7d8effd640 1 --2- 172.21.15.107:0/2526827199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 0x560e785d34f0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f7d840126d0 tx=0x7f7d840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1a55d01786f1bd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.176+0000 7f7d6effd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:40.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.177+0000 7f7d6effd640 1 -- 172.21.15.107:0/2526827199 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8400fa60 con 0x560e785cb520 2024-05-09T08:42:40.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.177+0000 7f7d6effd640 1 -- 172.21.15.107:0/2526827199 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f7d8400fc00 con 0x560e785cb520 2024-05-09T08:42:40.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.177+0000 7f7d985fc740 1 -- 172.21.15.107:0/2526827199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d80016aa0 con 0x560e785cb520 2024-05-09T08:42:40.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.177+0000 7f7d6effd640 1 -- 172.21.15.107:0/2526827199 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7d84030050 con 0x560e785cb520 2024-05-09T08:42:40.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.177+0000 7f7d985fc740 10 client.18327 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:40.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.177+0000 7f7d985fc740 1 -- 172.21.15.107:0/2526827199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560e785e56c0 con 0x560e785cb520 2024-05-09T08:42:40.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.179+0000 7f7d6effd640 1 -- 172.21.15.107:0/2526827199 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d80016aa0 con 0x560e785cb520 2024-05-09T08:42:40.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.179+0000 7f7d6effd640 1 -- 172.21.15.107:0/2526827199 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7d84027400 con 0x560e785cb520 2024-05-09T08:42:40.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.179+0000 7f7d985fc740 10 client.18327 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:40.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.179+0000 7f7d985fc740 10 client.18327 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:42:40.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.179+0000 7f7d985fc740 20 client.18327 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:40.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.179+0000 7f7d985fc740 1 --2- 172.21.15.107:0/2526827199 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x560e785e6c20 0x560e785e90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.179+0000 7f7d985fc740 4 client.18327 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:40.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.179+0000 7f7d985fc740 1 -- 172.21.15.107:0/2526827199 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e7853e370 con 0x560e785e6c20 2024-05-09T08:42:40.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.180+0000 7f7d8e7fc640 1 --2- 172.21.15.107:0/2526827199 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x560e785e6c20 0x560e785e90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.182+0000 7f7d8e7fc640 1 --2- 172.21.15.107:0/2526827199 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x560e785e6c20 0x560e785e90a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=57a5c137a7cae2b1 server_cookie=44d72690feeea620 in_seq=0 out_seq=0 2024-05-09T08:42:40.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.182+0000 7f7d6effd640 10 client.18327 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:42:40.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.184+0000 7f7d6effd640 1 -- 172.21.15.107:0/2526827199 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e7853e370 con 0x560e785e6c20 2024-05-09T08:42:40.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.184+0000 7f7d6effd640 10 client.18327 handle_command_reply: tid=0 2024-05-09T08:42:40.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.184+0000 7f7d985fc740 1 client.18327 shutdown 2024-05-09T08:42:40.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.187+0000 7f7d985fc740 1 -- 172.21.15.107:0/2526827199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 msgr2=0x560e785d34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.187+0000 7f7d985fc740 1 --2- 172.21.15.107:0/2526827199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 0x560e785d34f0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f7d840126d0 tx=0x7f7d840046c0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.188+0000 7f7d985fc740 1 -- 172.21.15.107:0/2526827199 shutdown_connections 2024-05-09T08:42:40.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.188+0000 7f7d985fc740 1 --2- 172.21.15.107:0/2526827199 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x560e785e6c20 0x560e785e90a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.188+0000 7f7d985fc740 1 --2- 172.21.15.107:0/2526827199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785d6510 0x560e785d3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.188+0000 7f7d985fc740 1 --2- 172.21.15.107:0/2526827199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 0x560e785d3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.188+0000 7f7d985fc740 1 --2- 172.21.15.107:0/2526827199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 0x560e785d34f0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.188+0000 7f7d985fc740 1 -- 172.21.15.107:0/2526827199 >> 172.21.15.107:0/2526827199 conn(0x560e785c7840 msgr2=0x560e785cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.190+0000 7f7d6effd640 0 client.18327 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:42:40.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.191+0000 7f7d985fc740 1 -- 172.21.15.107:0/2526827199 shutdown_connections 2024-05-09T08:42:40.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.191+0000 7f7d985fc740 1 -- 172.21.15.107:0/2526827199 wait complete. 2024-05-09T08:42:40.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.191+0000 7f7d985fc740 20 client.18327 trim_cache size 0 max 16384 2024-05-09T08:42:40.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.233+0000 7f7d985fc740 1 Processor -- start 2024-05-09T08:42:40.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.233+0000 7f7d985fc740 1 -- start start 2024-05-09T08:42:40.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.233+0000 7f7d985fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 0x560e785e22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.233+0000 7f7d985fc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 0x560e785cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.234+0000 7f7d985fc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785d6510 0x560e785ee9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.234+0000 7f7d8e7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 0x560e785cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.234+0000 7f7d8e7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 0x560e785cdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60796/0 (socket says 172.21.15.107:60796) 2024-05-09T08:42:40.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.234+0000 7f7d8e7fc640 1 -- 172.21.15.107:0/791820061 learned_addr learned my addr 172.21.15.107:0/791820061 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:40.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.234+0000 7f7d8effd640 1 --2- 172.21.15.107:0/791820061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 0x560e785e22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.234+0000 7f7d985fc740 1 -- 172.21.15.107:0/791820061 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7c012820 con 0x560e785cb520 2024-05-09T08:42:40.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.234+0000 7f7d8f7fe640 1 --2- 172.21.15.107:0/791820061 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785d6510 0x560e785ee9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.234+0000 7f7d985fc740 1 -- 172.21.15.107:0/791820061 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7c0100f0 con 0x560e785cbe40 2024-05-09T08:42:40.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.234+0000 7f7d985fc740 1 -- 172.21.15.107:0/791820061 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7c00df60 con 0x560e785d6510 2024-05-09T08:42:40.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.235+0000 7f7d8e7fc640 1 -- 172.21.15.107:0/791820061 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785d6510 msgr2=0x560e785ee9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.235+0000 7f7d8e7fc640 1 --2- 172.21.15.107:0/791820061 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785d6510 0x560e785ee9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.235+0000 7f7d8e7fc640 1 -- 172.21.15.107:0/791820061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 msgr2=0x560e785e22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.235+0000 7f7d8e7fc640 1 --2- 172.21.15.107:0/791820061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 0x560e785e22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.235+0000 7f7d8e7fc640 1 -- 172.21.15.107:0/791820061 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e785c1da0 con 0x560e785cbe40 2024-05-09T08:42:40.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.235+0000 7f7d8e7fc640 1 --2- 172.21.15.107:0/791820061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 0x560e785cdc80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7d80013f50 tx=0x7f7d80006ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52974d264c385bf2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.237+0000 7f7d757fa640 1 -- 172.21.15.107:0/791820061 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8001b040 con 0x560e785cbe40 2024-05-09T08:42:40.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.237+0000 7f7d757fa640 1 -- 172.21.15.107:0/791820061 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d8002a030 con 0x560e785cbe40 2024-05-09T08:42:40.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.237+0000 7f7d757fa640 1 -- 172.21.15.107:0/791820061 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8000c040 con 0x560e785cbe40 2024-05-09T08:42:40.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.238+0000 7f7d985fc740 1 -- 172.21.15.107:0/791820061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 msgr2=0x560e785cdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.238+0000 7f7d985fc740 1 --2- 172.21.15.107:0/791820061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 0x560e785cdc80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7d80013f50 tx=0x7f7d80006ab0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.238+0000 7f7d985fc740 1 -- 172.21.15.107:0/791820061 shutdown_connections 2024-05-09T08:42:40.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.238+0000 7f7d985fc740 1 --2- 172.21.15.107:0/791820061 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785d6510 0x560e785ee9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.238+0000 7f7d985fc740 1 --2- 172.21.15.107:0/791820061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785cbe40 0x560e785cdc80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.238+0000 7f7d985fc740 1 --2- 172.21.15.107:0/791820061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 0x560e785e22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.238+0000 7f7d985fc740 1 -- 172.21.15.107:0/791820061 >> 172.21.15.107:0/791820061 conn(0x560e785c7840 msgr2=0x560e785c69c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.238+0000 7f7d985fc740 1 -- 172.21.15.107:0/791820061 shutdown_connections 2024-05-09T08:42:40.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.238+0000 7f7d985fc740 1 -- 172.21.15.107:0/791820061 wait complete. 2024-05-09T08:42:40.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.239+0000 7f7d985fc740 1 -- start start 2024-05-09T08:42:40.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.239+0000 7f7d985fc740 1 Processor -- start 2024-05-09T08:42:40.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.240+0000 7f7d985fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 0x560e785deba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.240+0000 7f7d985fc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785cbe40 0x560e785cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.240+0000 7f7d985fc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785d6510 0x560e785d0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.240+0000 7f7d985fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7c00e5b0 con 0x560e785cb520 2024-05-09T08:42:40.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.241+0000 7f7d985fc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7c00fa20 con 0x560e785d6510 2024-05-09T08:42:40.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.241+0000 7f7d8effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 0x560e785deba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.241+0000 7f7d8effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 0x560e785deba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40330/0 (socket says 172.21.15.107:40330) 2024-05-09T08:42:40.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.241+0000 7f7d8effd640 1 -- 172.21.15.107:0/1195698593 learned_addr learned my addr 172.21.15.107:0/1195698593 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:40.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.241+0000 7f7d985fc740 1 -- 172.21.15.107:0/1195698593 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7c00ec80 con 0x560e785cbe40 2024-05-09T08:42:40.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.241+0000 7f7d985fc740 10 client.0 _ll_register_callbacks cb 0x7f7d8400bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:40.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.241+0000 7f7d8f7fe640 1 --2- 172.21.15.107:0/1195698593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785d6510 0x560e785d0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.241+0000 7f7d8e7fc640 1 --2- 172.21.15.107:0/1195698593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785cbe40 0x560e785cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.241+0000 7f7d8f7fe640 1 -- 172.21.15.107:0/1195698593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785cbe40 msgr2=0x560e785cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.241+0000 7f7d8f7fe640 1 --2- 172.21.15.107:0/1195698593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785cbe40 0x560e785cff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.241+0000 7f7d8f7fe640 1 -- 172.21.15.107:0/1195698593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 msgr2=0x560e785deba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.241+0000 7f7d8f7fe640 1 --2- 172.21.15.107:0/1195698593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 0x560e785deba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.242+0000 7f7d8f7fe640 1 -- 172.21.15.107:0/1195698593 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e785d0c90 con 0x560e785d6510 2024-05-09T08:42:40.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.242+0000 7f7d8effd640 1 --2- 172.21.15.107:0/1195698593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 0x560e785deba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:42:40.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.242+0000 7f7d8f7fe640 1 --2- 172.21.15.107:0/1195698593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785d6510 0x560e785d0460 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7d880edbe0 tx=0x7f7d880309a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ed70bab9522d56b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.242+0000 7f7d6effd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:42:40.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.242+0000 7f7d6effd640 1 -- 172.21.15.107:0/1195698593 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d880c0d60 con 0x560e785d6510 2024-05-09T08:42:40.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.243+0000 7f7d6effd640 1 -- 172.21.15.107:0/1195698593 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f7d880b9020 con 0x560e785d6510 2024-05-09T08:42:40.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.243+0000 7f7d985fc740 1 -- 172.21.15.107:0/1195698593 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d8002a030 con 0x560e785d6510 2024-05-09T08:42:40.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.246+0000 7f7d6effd640 1 -- 172.21.15.107:0/1195698593 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7d880c5620 con 0x560e785d6510 2024-05-09T08:42:40.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.247+0000 7f7d985fc740 10 client.27269 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:40.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.247+0000 7f7d985fc740 1 -- 172.21.15.107:0/1195698593 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560e7864d2e0 con 0x560e785d6510 2024-05-09T08:42:40.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.252+0000 7f7d6effd640 1 -- 172.21.15.107:0/1195698593 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d8002a030 con 0x560e785d6510 2024-05-09T08:42:40.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.252+0000 7f7d6effd640 1 -- 172.21.15.107:0/1195698593 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f7d880bdce0 con 0x560e785d6510 2024-05-09T08:42:40.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.253+0000 7f7d985fc740 10 client.27269 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:40.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.253+0000 7f7d985fc740 10 client.27269 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:42:40.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.253+0000 7f7d985fc740 20 client.27269 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:40.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.253+0000 7f7d985fc740 1 --2- 172.21.15.107:0/1195698593 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x560e7864d870 0x560e7864fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.253+0000 7f7d985fc740 4 client.27269 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:42:40.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.253+0000 7f7d985fc740 1 -- 172.21.15.107:0/1195698593 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560e785386c0 con 0x560e7864d870 2024-05-09T08:42:40.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.254+0000 7f7d8effd640 1 --2- 172.21.15.107:0/1195698593 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x560e7864d870 0x560e7864fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.257+0000 7f7d8effd640 1 --2- 172.21.15.107:0/1195698593 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x560e7864d870 0x560e7864fc60 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=350803bd6e3bc6f9 server_cookie=f639c17ebabb860 in_seq=0 out_seq=0 2024-05-09T08:42:40.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.259+0000 7f7d6effd640 10 client.27269 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:42:40.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.259+0000 7f7d6effd640 1 -- 172.21.15.107:0/1195698593 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560e785386c0 con 0x560e7864d870 2024-05-09T08:42:40.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.259+0000 7f7d6effd640 10 client.27269 handle_command_reply: tid=0 2024-05-09T08:42:40.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.259+0000 7f7d985fc740 1 client.27269 shutdown 2024-05-09T08:42:40.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.262+0000 7f7d985fc740 1 -- 172.21.15.107:0/1195698593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785d6510 msgr2=0x560e785d0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.262+0000 7f7d985fc740 1 --2- 172.21.15.107:0/1195698593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785d6510 0x560e785d0460 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7d880edbe0 tx=0x7f7d880309a0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.268+0000 7f7d985fc740 1 -- 172.21.15.107:0/1195698593 shutdown_connections 2024-05-09T08:42:40.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.268+0000 7f7d985fc740 1 --2- 172.21.15.107:0/1195698593 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x560e7864d870 0x560e7864fc60 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.268+0000 7f7d985fc740 1 --2- 172.21.15.107:0/1195698593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560e785d6510 0x560e785d0460 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.268+0000 7f7d6effd640 0 client.27269 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:42:40.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.268+0000 7f7d985fc740 1 --2- 172.21.15.107:0/1195698593 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560e785cbe40 0x560e785cff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.268+0000 7f7d985fc740 1 --2- 172.21.15.107:0/1195698593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e785cb520 0x560e785deba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.268+0000 7f7d985fc740 1 -- 172.21.15.107:0/1195698593 >> 172.21.15.107:0/1195698593 conn(0x560e785c7840 msgr2=0x560e785d4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.270+0000 7f7d985fc740 1 -- 172.21.15.107:0/1195698593 shutdown_connections 2024-05-09T08:42:40.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.270+0000 7f7d985fc740 1 -- 172.21.15.107:0/1195698593 wait complete. 2024-05-09T08:42:40.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.270+0000 7f7d985fc740 20 client.27269 trim_cache size 0 max 16384 2024-05-09T08:42:40.271 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:42:40.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.274+0000 7f7d8ffff640 1 -- 172.21.15.107:0/2385645943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7d780a2f10 msgr2=0x7f7d780a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.274+0000 7f7d8ffff640 1 --2- 172.21.15.107:0/2385645943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7d780a2f10 0x7f7d780a53d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7d84005fd0 tx=0x7f7d8400b650 comp rx=0 tx=0).stop 2024-05-09T08:42:40.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.274+0000 7f7d8ffff640 1 -- 172.21.15.107:0/2385645943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d9010fed0 msgr2=0x7f7d90115db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.274+0000 7f7d8ffff640 1 --2- 172.21.15.107:0/2385645943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d9010fed0 0x7f7d90115db0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7d8802af70 tx=0x7f7d880308b0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.274+0000 7f7d8ffff640 1 -- 172.21.15.107:0/2385645943 shutdown_connections 2024-05-09T08:42:40.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.275+0000 7f7d8ffff640 1 --2- 172.21.15.107:0/2385645943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7d780a2f10 0x7f7d780a53d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.275+0000 7f7d8ffff640 1 --2- 172.21.15.107:0/2385645943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d9010fed0 0x7f7d90115db0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.275+0000 7f7d8ffff640 1 --2- 172.21.15.107:0/2385645943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d901092b0 0x7f7d9011d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.275+0000 7f7d8ffff640 1 --2- 172.21.15.107:0/2385645943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d90108830 0x7f7d9011cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.275+0000 7f7d8ffff640 1 -- 172.21.15.107:0/2385645943 >> 172.21.15.107:0/2385645943 conn(0x7f7d9006c7b0 msgr2=0x7f7d90070390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.275+0000 7f7d8ffff640 1 -- 172.21.15.107:0/2385645943 shutdown_connections 2024-05-09T08:42:40.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.275+0000 7f7d8ffff640 1 -- 172.21.15.107:0/2385645943 wait complete. 2024-05-09T08:42:40.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:42:40.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:40 smithi107 ceph-mon[30122]: pgmap v288: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 16 MiB/s wr, 4.17k op/s 2024-05-09T08:42:40.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:40 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:40.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:40 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:40.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:40 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:42:40.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:40 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2540595904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:42:40.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.555+0000 7fcf00ec4640 1 -- 172.21.15.107:0/1077804531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcef40a4e10 msgr2=0x7fcef40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.555+0000 7fcf00ec4640 1 --2- 172.21.15.107:0/1077804531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcef40a4e10 0x7fcef40a51f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fcefc067530 tx=0x7fcefc09ede0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.556+0000 7fcf00ec4640 1 -- 172.21.15.107:0/1077804531 shutdown_connections 2024-05-09T08:42:40.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.556+0000 7fcf00ec4640 1 --2- 172.21.15.107:0/1077804531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcef40b7aa0 0x7fcef40b9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.556+0000 7fcf00ec4640 1 --2- 172.21.15.107:0/1077804531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcef40a5730 0x7fcef40b7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.556+0000 7fcf00ec4640 1 --2- 172.21.15.107:0/1077804531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcef40a4e10 0x7fcef40a51f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.556+0000 7fcf00ec4640 1 -- 172.21.15.107:0/1077804531 >> 172.21.15.107:0/1077804531 conn(0x7fcef401a670 msgr2=0x7fcef401aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.556+0000 7fcf00ec4640 1 -- 172.21.15.107:0/1077804531 shutdown_connections 2024-05-09T08:42:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.556+0000 7fcf00ec4640 1 -- 172.21.15.107:0/1077804531 wait complete. 2024-05-09T08:42:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.557+0000 7fcf00ec4640 1 Processor -- start 2024-05-09T08:42:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.558+0000 7fcf00ec4640 1 -- start start 2024-05-09T08:42:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.558+0000 7fcf00ec4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcef40a4e10 0x7fcef40b7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.558+0000 7fcf00ec4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcef40a5730 0x7fcef40b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.558+0000 7fcf00ec4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcef40b7aa0 0x7fcef40b3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.558+0000 7fcf00ec4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcef40af530 con 0x7fcef40b7aa0 2024-05-09T08:42:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.558+0000 7fcf00ec4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcef40af3b0 con 0x7fcef40a4e10 2024-05-09T08:42:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.558+0000 7fcf00ec4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcef40af6b0 con 0x7fcef40a5730 2024-05-09T08:42:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.558+0000 7fcefa59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcef40a4e10 0x7fcef40b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.558+0000 7fcefa59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcef40a4e10 0x7fcef40b7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60824/0 (socket says 172.21.15.107:60824) 2024-05-09T08:42:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.558+0000 7fcefa59c640 1 -- 172.21.15.107:0/330532870 learned_addr learned my addr 172.21.15.107:0/330532870 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.559+0000 7fcef9d9b640 1 --2- 172.21.15.107:0/330532870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcef40a5730 0x7fcef40b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.559+0000 7fcefad9d640 1 --2- 172.21.15.107:0/330532870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcef40b7aa0 0x7fcef40b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.559+0000 7fcefa59c640 1 -- 172.21.15.107:0/330532870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcef40a5730 msgr2=0x7fcef40b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.559+0000 7fcefa59c640 1 --2- 172.21.15.107:0/330532870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcef40a5730 0x7fcef40b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.559+0000 7fcefa59c640 1 -- 172.21.15.107:0/330532870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcef40b7aa0 msgr2=0x7fcef40b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.559+0000 7fcefa59c640 1 --2- 172.21.15.107:0/330532870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcef40b7aa0 0x7fcef40b3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.559+0000 7fcefa59c640 1 -- 172.21.15.107:0/330532870 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef40b3b40 con 0x7fcef40a4e10 2024-05-09T08:42:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.560+0000 7fcefa59c640 1 --2- 172.21.15.107:0/330532870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcef40a4e10 0x7fcef40b7030 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcefc02d750 tx=0x7fcefc0a8a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.560+0000 7fceeb7fe640 1 -- 172.21.15.107:0/330532870 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0a8d90 con 0x7fcef40a4e10 2024-05-09T08:42:40.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.560+0000 7fcf00ec4640 1 -- 172.21.15.107:0/330532870 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef40b3dd0 con 0x7fcef40a4e10 2024-05-09T08:42:40.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.560+0000 7fcf00ec4640 1 -- 172.21.15.107:0/330532870 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef4147d00 con 0x7fcef40a4e10 2024-05-09T08:42:40.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.561+0000 7fceeb7fe640 1 -- 172.21.15.107:0/330532870 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcefc09fda0 con 0x7fcef40a4e10 2024-05-09T08:42:40.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.561+0000 7fceeb7fe640 1 -- 172.21.15.107:0/330532870 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0b2420 con 0x7fcef40a4e10 2024-05-09T08:42:40.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.566+0000 7fcf03bd8740 1 Processor -- start 2024-05-09T08:42:40.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.568+0000 7fcf03bd8740 1 -- start start 2024-05-09T08:42:40.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.568+0000 7fcf03bd8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36d520 0x55896e36d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.568+0000 7fcf03bd8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36de40 0x55896e377fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.568+0000 7fcf03bd8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e378510 0x55896e37a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.568+0000 7fcf03bd8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55896e2d2c40 con 0x55896e36de40 2024-05-09T08:42:40.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.568+0000 7fcf03bd8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55896e27a580 con 0x55896e36d520 2024-05-09T08:42:40.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.568+0000 7fcf03bd8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55896e2cec00 con 0x55896e378510 2024-05-09T08:42:40.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.569+0000 7fcefad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36d520 0x55896e36d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.569+0000 7fcefad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36d520 0x55896e36d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60840/0 (socket says 172.21.15.107:60840) 2024-05-09T08:42:40.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.569+0000 7fcefad9d640 1 -- 172.21.15.107:0/3185432878 learned_addr learned my addr 172.21.15.107:0/3185432878 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:40.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.569+0000 7fceeb7fe640 1 -- 172.21.15.107:0/330532870 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fcefc0b25c0 con 0x7fcef40a4e10 2024-05-09T08:42:40.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.570+0000 7fcefa59c640 1 --2- 172.21.15.107:0/3185432878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36de40 0x55896e377fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.570+0000 7fcef9d9b640 1 --2- 172.21.15.107:0/3185432878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e378510 0x55896e37a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.571+0000 7fceeb7fe640 1 --2- 172.21.15.107:0/330532870 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcee40a2f00 0x7fcee40a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.571+0000 7fceeb7fe640 1 -- 172.21.15.107:0/330532870 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fcefc1640c0 con 0x7fcef40a4e10 2024-05-09T08:42:40.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.574+0000 7fcefa59c640 1 -- 172.21.15.107:0/3185432878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e378510 msgr2=0x55896e37a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.574+0000 7fcefa59c640 1 --2- 172.21.15.107:0/3185432878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e378510 0x55896e37a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.574+0000 7fcefa59c640 1 -- 172.21.15.107:0/3185432878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36d520 msgr2=0x55896e36d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.574+0000 7fcefa59c640 1 --2- 172.21.15.107:0/3185432878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36d520 0x55896e36d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.574+0000 7fcefa59c640 1 -- 172.21.15.107:0/3185432878 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55896dfe63f0 con 0x55896e36de40 2024-05-09T08:42:40.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.574+0000 7fcefad9d640 1 --2- 172.21.15.107:0/330532870 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcee40a2f00 0x7fcee40a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.575+0000 7fcefa59c640 1 --2- 172.21.15.107:0/3185432878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36de40 0x55896e377fd0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fcefc161510 tx=0x7fcefc09ede0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed86988d88fc472e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.577+0000 7fcefad9d640 1 --2- 172.21.15.107:0/330532870 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcee40a2f00 0x7fcee40a53c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fceec015720 tx=0x7fceec004730 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.577+0000 7fcee8ff9640 1 -- 172.21.15.107:0/3185432878 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc12fc40 con 0x55896e36de40 2024-05-09T08:42:40.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.578+0000 7fcee8ff9640 1 -- 172.21.15.107:0/3185432878 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcefc0a3030 con 0x55896e36de40 2024-05-09T08:42:40.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.578+0000 7fcee8ff9640 1 -- 172.21.15.107:0/3185432878 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc12e420 con 0x55896e36de40 2024-05-09T08:42:40.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.581+0000 7fcf03bd8740 1 -- 172.21.15.107:0/3185432878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36de40 msgr2=0x55896e377fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.581+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/3185432878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36de40 0x55896e377fd0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fcefc161510 tx=0x7fcefc09ede0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.586+0000 7fcf03bd8740 1 -- 172.21.15.107:0/3185432878 shutdown_connections 2024-05-09T08:42:40.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.586+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/3185432878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e378510 0x55896e37a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.586+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/3185432878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36de40 0x55896e377fd0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.586+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/3185432878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36d520 0x55896e36d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.586+0000 7fcf03bd8740 1 -- 172.21.15.107:0/3185432878 >> 172.21.15.107:0/3185432878 conn(0x55896e369840 msgr2=0x55896e369c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.588+0000 7fcf03bd8740 1 -- 172.21.15.107:0/3185432878 shutdown_connections 2024-05-09T08:42:40.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.589+0000 7fcf03bd8740 1 -- 172.21.15.107:0/3185432878 wait complete. 2024-05-09T08:42:40.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.589+0000 7fcf03bd8740 1 -- start start 2024-05-09T08:42:40.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.589+0000 7fcf03bd8740 1 Processor -- start 2024-05-09T08:42:40.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.590+0000 7fcf03bd8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36d520 0x55896e375400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.590+0000 7fcf03bd8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36de40 0x55896e375940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.590+0000 7fcf03bd8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e378510 0x55896e375e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.590+0000 7fcf03bd8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55896e37b0c0 con 0x55896e36d520 2024-05-09T08:42:40.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.590+0000 7fcf03bd8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55896e28dc40 con 0x55896e36de40 2024-05-09T08:42:40.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.590+0000 7fcf03bd8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55896e37b240 con 0x55896e378510 2024-05-09T08:42:40.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.590+0000 7fcf03bd8740 10 client.0 _ll_register_callbacks cb 0x55896dfe5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:40.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.591+0000 7fcef9d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36de40 0x55896e375940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.591+0000 7fcefad9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e378510 0x55896e375e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.592+0000 7fcef9d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36de40 0x55896e375940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60854/0 (socket says 172.21.15.107:60854) 2024-05-09T08:42:40.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.592+0000 7fcef9d9b640 1 -- 172.21.15.107:0/1289845961 learned_addr learned my addr 172.21.15.107:0/1289845961 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:40.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:40 smithi157 ceph-mon[33614]: pgmap v288: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 16 MiB/s wr, 4.17k op/s 2024-05-09T08:42:40.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:40 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:40.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:40 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:40.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:40 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:42:40.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:40 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2540595904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:42:40.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.593+0000 7fcefa59c640 1 --2- 172.21.15.107:0/1289845961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36d520 0x55896e375400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.594+0000 7fcefa59c640 1 -- 172.21.15.107:0/1289845961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e378510 msgr2=0x55896e375e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.594+0000 7fcefa59c640 1 --2- 172.21.15.107:0/1289845961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e378510 0x55896e375e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.594+0000 7fcefa59c640 1 -- 172.21.15.107:0/1289845961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36de40 msgr2=0x55896e375940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.594+0000 7fcefa59c640 1 --2- 172.21.15.107:0/1289845961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36de40 0x55896e375940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.594+0000 7fcefa59c640 1 -- 172.21.15.107:0/1289845961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55896e0b07d0 con 0x55896e36d520 2024-05-09T08:42:40.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.594+0000 7fcefa59c640 1 --2- 172.21.15.107:0/1289845961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36d520 0x55896e375400 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fcefc000b60 tx=0x7fcefc12cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a700b4d7a33b98 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.594+0000 7fcee1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:40.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.595+0000 7fcee1ffb640 1 -- 172.21.15.107:0/1289845961 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc12e420 con 0x55896e36d520 2024-05-09T08:42:40.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.595+0000 7fcf03bd8740 1 -- 172.21.15.107:0/1289845961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcefc0a3030 con 0x55896e36d520 2024-05-09T08:42:40.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.595+0000 7fcee1ffb640 1 -- 172.21.15.107:0/1289845961 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fcefc167140 con 0x55896e36d520 2024-05-09T08:42:40.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.595+0000 7fcee1ffb640 1 -- 172.21.15.107:0/1289845961 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcefc12fc40 con 0x55896e36d520 2024-05-09T08:42:40.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.595+0000 7fcf03bd8740 10 client.18345 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:40.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.595+0000 7fcf03bd8740 1 -- 172.21.15.107:0/1289845961 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55896e37b0c0 con 0x55896e36d520 2024-05-09T08:42:40.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.596+0000 7fcee1ffb640 1 -- 172.21.15.107:0/1289845961 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcefc0a3030 con 0x55896e36d520 2024-05-09T08:42:40.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.597+0000 7fcee1ffb640 1 -- 172.21.15.107:0/1289845961 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fcefc0ba070 con 0x55896e36d520 2024-05-09T08:42:40.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.597+0000 7fcf03bd8740 10 client.18345 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:40.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.597+0000 7fcf03bd8740 10 client.18345 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:42:40.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.597+0000 7fcf03bd8740 20 client.18345 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:40.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.598+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/1289845961 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55896e388590 0x55896e38aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.598+0000 7fcf03bd8740 4 client.18345 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:40.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.598+0000 7fcf03bd8740 1 -- 172.21.15.107:0/1289845961 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55896e2e0370 con 0x55896e388590 2024-05-09T08:42:40.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.606+0000 7fcef9d9b640 1 --2- 172.21.15.107:0/1289845961 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55896e388590 0x55896e38aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.617+0000 7fcef9d9b640 1 --2- 172.21.15.107:0/1289845961 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55896e388590 0x55896e38aa50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8510acaeff31cb9f server_cookie=94a1376d75f97227 in_seq=0 out_seq=0 2024-05-09T08:42:40.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.617+0000 7fcee1ffb640 10 client.18345 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:42:40.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:40 smithi110 ceph-mon[33622]: pgmap v288: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 16 MiB/s wr, 4.17k op/s 2024-05-09T08:42:40.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:40 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:42:40.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:40 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:42:40.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:40 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:42:40.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:40 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2540595904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:42:40.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.622+0000 7fcee1ffb640 1 -- 172.21.15.107:0/1289845961 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55896e2e0370 con 0x55896e388590 2024-05-09T08:42:40.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.622+0000 7fcee1ffb640 10 client.18345 handle_command_reply: tid=0 2024-05-09T08:42:40.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.623+0000 7fcf03bd8740 1 client.18345 shutdown 2024-05-09T08:42:40.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.625+0000 7fcf03bd8740 1 -- 172.21.15.107:0/1289845961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36d520 msgr2=0x55896e375400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.625+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/1289845961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36d520 0x55896e375400 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fcefc000b60 tx=0x7fcefc12cb60 comp rx=0 tx=0).stop 2024-05-09T08:42:40.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.626+0000 7fcf03bd8740 1 -- 172.21.15.107:0/1289845961 shutdown_connections 2024-05-09T08:42:40.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.626+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/1289845961 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55896e388590 0x55896e38aa50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.626+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/1289845961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e378510 0x55896e375e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.626+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/1289845961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36de40 0x55896e375940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.626+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/1289845961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36d520 0x55896e375400 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.626+0000 7fcf03bd8740 1 -- 172.21.15.107:0/1289845961 >> 172.21.15.107:0/1289845961 conn(0x55896e369840 msgr2=0x55896e3689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.628+0000 7fcee1ffb640 0 client.18345 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:42:40.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.628+0000 7fcf03bd8740 1 -- 172.21.15.107:0/1289845961 shutdown_connections 2024-05-09T08:42:40.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.628+0000 7fcf03bd8740 1 -- 172.21.15.107:0/1289845961 wait complete. 2024-05-09T08:42:40.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.631+0000 7fcf03bd8740 20 client.18345 trim_cache size 0 max 16384 2024-05-09T08:42:40.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.667+0000 7fcf03bd8740 1 Processor -- start 2024-05-09T08:42:40.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.667+0000 7fcf03bd8740 1 -- start start 2024-05-09T08:42:40.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.668+0000 7fcf03bd8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36d520 0x55896e377b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.668+0000 7fcf03bd8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e36de40 0x55896e383ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.668+0000 7fcf03bd8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e378510 0x55896e368840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.668+0000 7fcef9d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e36de40 0x55896e383ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.668+0000 7fcef9d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e36de40 0x55896e383ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47552/0 (socket says 172.21.15.107:47552) 2024-05-09T08:42:40.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.668+0000 7fcef9d9b640 1 -- 172.21.15.107:0/1006686553 learned_addr learned my addr 172.21.15.107:0/1006686553 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:40.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.668+0000 7fcefad9d640 1 --2- 172.21.15.107:0/1006686553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e378510 0x55896e368840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.669+0000 7fcf03bd8740 1 -- 172.21.15.107:0/1006686553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc012700 con 0x55896e36d520 2024-05-09T08:42:40.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.669+0000 7fcf03bd8740 1 -- 172.21.15.107:0/1006686553 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc00ffd0 con 0x55896e378510 2024-05-09T08:42:40.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.669+0000 7fcf03bd8740 1 -- 172.21.15.107:0/1006686553 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc00ded0 con 0x55896e36de40 2024-05-09T08:42:40.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.669+0000 7fcefa59c640 1 --2- 172.21.15.107:0/1006686553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36d520 0x55896e377b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.670+0000 7fcef9d9b640 1 -- 172.21.15.107:0/1006686553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e378510 msgr2=0x55896e368840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.670+0000 7fcef9d9b640 1 --2- 172.21.15.107:0/1006686553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e378510 0x55896e368840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.670+0000 7fcef9d9b640 1 -- 172.21.15.107:0/1006686553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36d520 msgr2=0x55896e377b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.670+0000 7fcef9d9b640 1 --2- 172.21.15.107:0/1006686553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36d520 0x55896e377b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.670+0000 7fcef9d9b640 1 -- 172.21.15.107:0/1006686553 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55896e36d320 con 0x55896e36de40 2024-05-09T08:42:40.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.670+0000 7fcef9d9b640 1 --2- 172.21.15.107:0/1006686553 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e36de40 0x55896e383ef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcef0005e00 tx=0x7fcef0005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a99f6dd69d4d3eaf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.671+0000 7fcee3fff640 1 -- 172.21.15.107:0/1006686553 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef0004050 con 0x55896e36de40 2024-05-09T08:42:40.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.671+0000 7fcee3fff640 1 -- 172.21.15.107:0/1006686553 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcef00054d0 con 0x55896e36de40 2024-05-09T08:42:40.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.671+0000 7fcee3fff640 1 -- 172.21.15.107:0/1006686553 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef0004cc0 con 0x55896e36de40 2024-05-09T08:42:40.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.673+0000 7fcf03bd8740 1 -- 172.21.15.107:0/1006686553 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e36de40 msgr2=0x55896e383ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.674+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/1006686553 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e36de40 0x55896e383ef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcef0005e00 tx=0x7fcef0005f90 comp rx=0 tx=0).stop 2024-05-09T08:42:40.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.674+0000 7fcf03bd8740 1 -- 172.21.15.107:0/1006686553 shutdown_connections 2024-05-09T08:42:40.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.674+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/1006686553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e378510 0x55896e368840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.674+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/1006686553 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e36de40 0x55896e383ef0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.674+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/1006686553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e36d520 0x55896e377b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.674+0000 7fcf03bd8740 1 -- 172.21.15.107:0/1006686553 >> 172.21.15.107:0/1006686553 conn(0x55896e369840 msgr2=0x55896e368130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.680+0000 7fcf03bd8740 1 -- 172.21.15.107:0/1006686553 shutdown_connections 2024-05-09T08:42:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.680+0000 7fcf03bd8740 1 -- 172.21.15.107:0/1006686553 wait complete. 2024-05-09T08:42:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.680+0000 7fcf03bd8740 1 -- start start 2024-05-09T08:42:40.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.681+0000 7fcf03bd8740 1 Processor -- start 2024-05-09T08:42:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.681+0000 7fcf03bd8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36d520 0x55896e380790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.681+0000 7fcefa59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36d520 0x55896e380790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.682+0000 7fcefa59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36d520 0x55896e380790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:37676/0 (socket says 172.21.15.107:37676) 2024-05-09T08:42:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.682+0000 7fcefa59c640 1 -- 172.21.15.107:0/3544417224 learned_addr learned my addr 172.21.15.107:0/3544417224 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.682+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/3544417224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e36de40 0x55896e371e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.682+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/3544417224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e378510 0x55896e372380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.682+0000 7fcf03bd8740 1 -- 172.21.15.107:0/3544417224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc00e490 con 0x55896e378510 2024-05-09T08:42:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.683+0000 7fcef9d9b640 1 --2- 172.21.15.107:0/3544417224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e36de40 0x55896e371e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.683+0000 7fcf03bd8740 1 -- 172.21.15.107:0/3544417224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc00f900 con 0x55896e36d520 2024-05-09T08:42:40.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.683+0000 7fcf03bd8740 1 -- 172.21.15.107:0/3544417224 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc00eb60 con 0x55896e36de40 2024-05-09T08:42:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.684+0000 7fcefad9d640 1 --2- 172.21.15.107:0/3544417224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e378510 0x55896e372380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.684+0000 7fcefa59c640 1 -- 172.21.15.107:0/3544417224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e36de40 msgr2=0x55896e371e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.684+0000 7fcefa59c640 1 --2- 172.21.15.107:0/3544417224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e36de40 0x55896e371e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.684+0000 7fcefa59c640 1 -- 172.21.15.107:0/3544417224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e378510 msgr2=0x55896e372380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.684+0000 7fcefa59c640 1 --2- 172.21.15.107:0/3544417224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e378510 0x55896e372380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.684+0000 7fcefa59c640 1 -- 172.21.15.107:0/3544417224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55896e372bb0 con 0x55896e36d520 2024-05-09T08:42:40.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.684+0000 7fcefa59c640 1 --2- 172.21.15.107:0/3544417224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36d520 0x55896e380790 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcefc067120 tx=0x7fcefc09ff50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efa429bf46d4d7ce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.685+0000 7fcee1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:42:40.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.685+0000 7fcf03bd8740 10 client.0 _ll_register_callbacks cb 0x7fcefc050610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:40.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.685+0000 7fcee1ffb640 1 -- 172.21.15.107:0/3544417224 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc12e420 con 0x55896e36d520 2024-05-09T08:42:40.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.685+0000 7fcf03bd8740 1 -- 172.21.15.107:0/3544417224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcef00054d0 con 0x55896e36d520 2024-05-09T08:42:40.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.687+0000 7fcee1ffb640 1 -- 172.21.15.107:0/3544417224 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fcefc161b50 con 0x55896e36d520 2024-05-09T08:42:40.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.688+0000 7fcee1ffb640 1 -- 172.21.15.107:0/3544417224 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcefc1309e0 con 0x55896e36d520 2024-05-09T08:42:40.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.688+0000 7fcf03bd8740 10 client.27287 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:40.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.688+0000 7fcf03bd8740 1 -- 172.21.15.107:0/3544417224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55896e3ee640 con 0x55896e36d520 2024-05-09T08:42:40.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.692+0000 7fcee1ffb640 1 -- 172.21.15.107:0/3544417224 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fcefc0ba070 con 0x55896e36d520 2024-05-09T08:42:40.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.692+0000 7fcf03bd8740 10 client.27287 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:40.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.692+0000 7fcf03bd8740 10 client.27287 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:42:40.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.692+0000 7fcf03bd8740 20 client.27287 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:40.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.692+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/3544417224 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55896e3ee990 0x55896e3f0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.692+0000 7fcf03bd8740 4 client.27287 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:42:40.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.692+0000 7fcf03bd8740 1 -- 172.21.15.107:0/3544417224 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fcef00054d0 con 0x55896e3ee990 2024-05-09T08:42:40.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.696+0000 7fcef9d9b640 1 --2- 172.21.15.107:0/3544417224 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55896e3ee990 0x55896e3f0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.703+0000 7fcef9d9b640 1 --2- 172.21.15.107:0/3544417224 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55896e3ee990 0x55896e3f0e10 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=265cd92099f30f8b server_cookie=fffd1b888444e952 in_seq=0 out_seq=0 2024-05-09T08:42:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.703+0000 7fcee1ffb640 10 client.27287 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:42:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.703+0000 7fcee1ffb640 1 -- 172.21.15.107:0/3544417224 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fcef00054d0 con 0x55896e3ee990 2024-05-09T08:42:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.703+0000 7fcee1ffb640 10 client.27287 handle_command_reply: tid=0 2024-05-09T08:42:40.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.703+0000 7fcf03bd8740 1 client.27287 shutdown 2024-05-09T08:42:40.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.703+0000 7fcf03bd8740 1 -- 172.21.15.107:0/3544417224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36d520 msgr2=0x55896e380790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.703+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/3544417224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36d520 0x55896e380790 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcefc067120 tx=0x7fcefc09ff50 comp rx=0 tx=0).stop 2024-05-09T08:42:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.705+0000 7fcf03bd8740 1 -- 172.21.15.107:0/3544417224 shutdown_connections 2024-05-09T08:42:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.705+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/3544417224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55896e378510 0x55896e372380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.705+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/3544417224 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55896e3ee990 0x55896e3f0e10 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.705+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/3544417224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55896e36de40 0x55896e371e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.705+0000 7fcf03bd8740 1 --2- 172.21.15.107:0/3544417224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55896e36d520 0x55896e380790 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.705+0000 7fcf03bd8740 1 -- 172.21.15.107:0/3544417224 >> 172.21.15.107:0/3544417224 conn(0x55896e369840 msgr2=0x55896e3689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.705+0000 7fcee1ffb640 0 client.27287 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:42:40.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.708+0000 7fcf03bd8740 1 -- 172.21.15.107:0/3544417224 shutdown_connections 2024-05-09T08:42:40.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.709+0000 7fcf03bd8740 1 -- 172.21.15.107:0/3544417224 wait complete. 2024-05-09T08:42:40.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.714+0000 7fcf03bd8740 20 client.27287 trim_cache size 0 max 16384 2024-05-09T08:42:40.719 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:42:40.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.723+0000 7fcf00ec4640 1 -- 172.21.15.107:0/330532870 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcee40a2f00 msgr2=0x7fcee40a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.723+0000 7fcf00ec4640 1 --2- 172.21.15.107:0/330532870 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcee40a2f00 0x7fcee40a53c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fceec015720 tx=0x7fceec004730 comp rx=0 tx=0).stop 2024-05-09T08:42:40.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.723+0000 7fcf00ec4640 1 -- 172.21.15.107:0/330532870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcef40a4e10 msgr2=0x7fcef40b7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.723+0000 7fcf00ec4640 1 --2- 172.21.15.107:0/330532870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcef40a4e10 0x7fcef40b7030 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcefc02d750 tx=0x7fcefc0a8a10 comp rx=0 tx=0).stop 2024-05-09T08:42:40.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.726+0000 7fcf00ec4640 1 -- 172.21.15.107:0/330532870 shutdown_connections 2024-05-09T08:42:40.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.726+0000 7fcf00ec4640 1 --2- 172.21.15.107:0/330532870 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcee40a2f00 0x7fcee40a53c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.726+0000 7fcf00ec4640 1 --2- 172.21.15.107:0/330532870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcef40b7aa0 0x7fcef40b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.726+0000 7fcf00ec4640 1 --2- 172.21.15.107:0/330532870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcef40a5730 0x7fcef40b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.727+0000 7fcf00ec4640 1 --2- 172.21.15.107:0/330532870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcef40a4e10 0x7fcef40b7030 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.727+0000 7fcf00ec4640 1 -- 172.21.15.107:0/330532870 >> 172.21.15.107:0/330532870 conn(0x7fcef401a670 msgr2=0x7fcef40b80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.730+0000 7fcf00ec4640 1 -- 172.21.15.107:0/330532870 shutdown_connections 2024-05-09T08:42:40.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.730+0000 7fcf00ec4640 1 -- 172.21.15.107:0/330532870 wait complete. 2024-05-09T08:42:40.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:42:40.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.918+0000 7fcfb7eae640 1 -- 172.21.15.107:0/3336563033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfb006a7f0 msgr2=0x7fcfb010e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.918+0000 7fcfb7eae640 1 --2- 172.21.15.107:0/3336563033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfb006a7f0 0x7fcfb010e710 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009900 tx=0x7fcfa002a480 comp rx=0 tx=0).stop 2024-05-09T08:42:40.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.918+0000 7fcfb7eae640 1 -- 172.21.15.107:0/3336563033 shutdown_connections 2024-05-09T08:42:40.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.918+0000 7fcfb7eae640 1 --2- 172.21.15.107:0/3336563033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfb006a7f0 0x7fcfb010e710 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.918+0000 7fcfb7eae640 1 --2- 172.21.15.107:0/3336563033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfb0069e50 0x7fcfb006a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.918+0000 7fcfb7eae640 1 --2- 172.21.15.107:0/3336563033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcfb0107910 0x7fcfb0107cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.918+0000 7fcfb7eae640 1 -- 172.21.15.107:0/3336563033 >> 172.21.15.107:0/3336563033 conn(0x7fcfb0077710 msgr2=0x7fcfb0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.918+0000 7fcfb7eae640 1 -- 172.21.15.107:0/3336563033 shutdown_connections 2024-05-09T08:42:40.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.918+0000 7fcfb7eae640 1 -- 172.21.15.107:0/3336563033 wait complete. 2024-05-09T08:42:40.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.918+0000 7fcfb7eae640 1 Processor -- start 2024-05-09T08:42:40.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.918+0000 7fcfb7eae640 1 -- start start 2024-05-09T08:42:40.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb7eae640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfb0069e50 0x7fcfb0198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb7eae640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcfb006a7f0 0x7fcfb01985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb7eae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfb0107910 0x7fcfb019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb7eae640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb0111b40 con 0x7fcfb0107910 2024-05-09T08:42:40.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb7eae640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb01119c0 con 0x7fcfb0069e50 2024-05-09T08:42:40.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb7eae640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb0111cc0 con 0x7fcfb006a7f0 2024-05-09T08:42:40.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb644b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfb0107910 0x7fcfb019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb644b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfb0107910 0x7fcfb019c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37540/0 (socket says 172.21.15.107:37540) 2024-05-09T08:42:40.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb644b640 1 -- 172.21.15.107:0/2759011900 learned_addr learned my addr 172.21.15.107:0/2759011900 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:40.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb5c4a640 1 --2- 172.21.15.107:0/2759011900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfb0069e50 0x7fcfb0198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb5449640 1 --2- 172.21.15.107:0/2759011900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcfb006a7f0 0x7fcfb01985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb644b640 1 -- 172.21.15.107:0/2759011900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcfb006a7f0 msgr2=0x7fcfb01985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb644b640 1 --2- 172.21.15.107:0/2759011900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcfb006a7f0 0x7fcfb01985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb644b640 1 -- 172.21.15.107:0/2759011900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfb0069e50 msgr2=0x7fcfb0198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb644b640 1 --2- 172.21.15.107:0/2759011900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfb0069e50 0x7fcfb0198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb644b640 1 -- 172.21.15.107:0/2759011900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb019d000 con 0x7fcfb0107910 2024-05-09T08:42:40.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb644b640 1 --2- 172.21.15.107:0/2759011900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfb0107910 0x7fcfb019c8d0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0005ab0 tx=0x7fcfa00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfaeffd640 1 -- 172.21.15.107:0/2759011900 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa003f030 con 0x7fcfb0107910 2024-05-09T08:42:40.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfaeffd640 1 -- 172.21.15.107:0/2759011900 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfa0003cb0 con 0x7fcfb0107910 2024-05-09T08:42:40.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb7eae640 1 -- 172.21.15.107:0/2759011900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfb019d290 con 0x7fcfb0107910 2024-05-09T08:42:40.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfaeffd640 1 -- 172.21.15.107:0/2759011900 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0035070 con 0x7fcfb0107910 2024-05-09T08:42:40.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.919+0000 7fcfb7eae640 1 -- 172.21.15.107:0/2759011900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb01a4b80 con 0x7fcfb0107910 2024-05-09T08:42:40.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.922+0000 7fcfaeffd640 1 -- 172.21.15.107:0/2759011900 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fcfa0042030 con 0x7fcfb0107910 2024-05-09T08:42:40.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.922+0000 7fcfbabb6740 1 Processor -- start 2024-05-09T08:42:40.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.922+0000 7fcfbabb6740 1 -- start start 2024-05-09T08:42:40.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.922+0000 7fcfbabb6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299ba330 0x55c5299ba710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.922+0000 7fcfbabb6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299bac50 0x55c5299c4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.922+0000 7fcfbabb6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299c5320 0x55c5299c7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.922+0000 7fcfbabb6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c52991fc40 con 0x55c5299c5320 2024-05-09T08:42:40.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.922+0000 7fcfbabb6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c5298c7580 con 0x55c5299ba330 2024-05-09T08:42:40.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.922+0000 7fcfbabb6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c52991bc00 con 0x55c5299bac50 2024-05-09T08:42:40.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.922+0000 7fcfaeffd640 1 --2- 172.21.15.107:0/2759011900 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcf980a2d80 0x7fcf980a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.922+0000 7fcfb5449640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299c5320 0x55c5299c7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.922+0000 7fcfb644b640 1 --2- 172.21.15.107:0/2759011900 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcf980a2d80 0x7fcf980a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.922+0000 7fcfb5449640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299c5320 0x55c5299c7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37552/0 (socket says 172.21.15.107:37552) 2024-05-09T08:42:40.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.923+0000 7fcfb5449640 1 -- 172.21.15.107:0/4047707771 learned_addr learned my addr 172.21.15.107:0/4047707771 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:40.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.923+0000 7fcfb5449640 1 --2- 172.21.15.107:0/4047707771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299ba330 0x55c5299ba710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.923+0000 7fcfb5c4a640 1 --2- 172.21.15.107:0/4047707771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299bac50 0x55c5299c4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.923+0000 7fcfaeffd640 1 -- 172.21.15.107:0/2759011900 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fcfa00effc0 con 0x7fcfb0107910 2024-05-09T08:42:40.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.923+0000 7fcfb5449640 1 -- 172.21.15.107:0/4047707771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299bac50 msgr2=0x55c5299c4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.923+0000 7fcfb5449640 1 --2- 172.21.15.107:0/4047707771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299bac50 0x55c5299c4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.923+0000 7fcfb5449640 1 -- 172.21.15.107:0/4047707771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299ba330 msgr2=0x55c5299ba710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.923+0000 7fcfb5449640 1 --2- 172.21.15.107:0/4047707771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299ba330 0x55c5299ba710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.923+0000 7fcfb5449640 1 -- 172.21.15.107:0/4047707771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5296333f0 con 0x55c5299c5320 2024-05-09T08:42:40.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.923+0000 7fcfb644b640 1 --2- 172.21.15.107:0/2759011900 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcf980a2d80 0x7fcf980a5240 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcfa00ed410 tx=0x7fcfa008e0f0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.923+0000 7fcfb5449640 1 --2- 172.21.15.107:0/4047707771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299c5320 0x55c5299c7710 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fcfa40149c0 tx=0x7fcfa400c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4574dfcd624fe619 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.923+0000 7fcf97fff640 1 -- 172.21.15.107:0/4047707771 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa40142d0 con 0x55c5299c5320 2024-05-09T08:42:40.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.923+0000 7fcf97fff640 1 -- 172.21.15.107:0/4047707771 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfa400fc10 con 0x55c5299c5320 2024-05-09T08:42:40.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.923+0000 7fcf97fff640 1 -- 172.21.15.107:0/4047707771 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa4012ba0 con 0x55c5299c5320 2024-05-09T08:42:40.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.924+0000 7fcfbabb6740 1 -- 172.21.15.107:0/4047707771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299c5320 msgr2=0x55c5299c7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.924+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/4047707771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299c5320 0x55c5299c7710 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fcfa40149c0 tx=0x7fcfa400c4d0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.924+0000 7fcfbabb6740 1 -- 172.21.15.107:0/4047707771 shutdown_connections 2024-05-09T08:42:40.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.924+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/4047707771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299c5320 0x55c5299c7710 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.924+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/4047707771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299bac50 0x55c5299c4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.924+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/4047707771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299ba330 0x55c5299ba710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.924+0000 7fcfbabb6740 1 -- 172.21.15.107:0/4047707771 >> 172.21.15.107:0/4047707771 conn(0x55c5299b6650 msgr2=0x55c5299b6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.924+0000 7fcfbabb6740 1 -- 172.21.15.107:0/4047707771 shutdown_connections 2024-05-09T08:42:40.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.924+0000 7fcfbabb6740 1 -- 172.21.15.107:0/4047707771 wait complete. 2024-05-09T08:42:40.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.924+0000 7fcfbabb6740 1 -- start start 2024-05-09T08:42:40.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.924+0000 7fcfbabb6740 1 Processor -- start 2024-05-09T08:42:40.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfbabb6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299ba330 0x55c5299c2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfbabb6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299bac50 0x55c5299c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfbabb6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299c5320 0x55c5299c2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfbabb6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c5299c8110 con 0x55c5299bac50 2024-05-09T08:42:40.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfbabb6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c5298dac40 con 0x55c5299c5320 2024-05-09T08:42:40.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfbabb6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c5299c8290 con 0x55c5299ba330 2024-05-09T08:42:40.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfbabb6740 10 client.0 _ll_register_callbacks cb 0x55c529632be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:40.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfb5c4a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299bac50 0x55c5299c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfb5449640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299ba330 0x55c5299c2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfb5c4a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299bac50 0x55c5299c27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37564/0 (socket says 172.21.15.107:37564) 2024-05-09T08:42:40.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfb5c4a640 1 -- 172.21.15.107:0/3026099898 learned_addr learned my addr 172.21.15.107:0/3026099898 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:40.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfb5449640 1 --2- 172.21.15.107:0/3026099898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299c5320 0x55c5299c2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfb5c4a640 1 -- 172.21.15.107:0/3026099898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299ba330 msgr2=0x55c5299c2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfb5c4a640 1 --2- 172.21.15.107:0/3026099898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299ba330 0x55c5299c2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfb5c4a640 1 -- 172.21.15.107:0/3026099898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299c5320 msgr2=0x55c5299c2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfb5c4a640 1 --2- 172.21.15.107:0/3026099898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299c5320 0x55c5299c2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfb5c4a640 1 -- 172.21.15.107:0/3026099898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5296fd7d0 con 0x55c5299bac50 2024-05-09T08:42:40.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfb5449640 1 --2- 172.21.15.107:0/3026099898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299c5320 0x55c5299c2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:42:40.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfb5c4a640 1 --2- 172.21.15.107:0/3026099898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299bac50 0x55c5299c27b0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fcfa800ab60 tx=0x7fcfa8011580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f4c241b027ca28f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcf957fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:40.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcf957fa640 1 -- 172.21.15.107:0/3026099898 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa801a070 con 0x55c5299bac50 2024-05-09T08:42:40.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcf957fa640 1 -- 172.21.15.107:0/3026099898 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fcfa8016450 con 0x55c5299bac50 2024-05-09T08:42:40.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.925+0000 7fcfbabb6740 1 -- 172.21.15.107:0/3026099898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfa400fc10 con 0x55c5299bac50 2024-05-09T08:42:40.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.926+0000 7fcf957fa640 1 -- 172.21.15.107:0/3026099898 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcfa8026050 con 0x55c5299bac50 2024-05-09T08:42:40.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.926+0000 7fcfbabb6740 10 client.18363 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:40.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.926+0000 7fcfbabb6740 1 -- 172.21.15.107:0/3026099898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c5299d4340 con 0x55c5299bac50 2024-05-09T08:42:40.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.926+0000 7fcf957fa640 1 -- 172.21.15.107:0/3026099898 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfa400fc10 con 0x55c5299bac50 2024-05-09T08:42:40.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.926+0000 7fcf957fa640 1 -- 172.21.15.107:0/3026099898 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fcfa801e6e0 con 0x55c5299bac50 2024-05-09T08:42:40.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.926+0000 7fcfbabb6740 10 client.18363 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:40.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.926+0000 7fcfbabb6740 10 client.18363 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:40.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.926+0000 7fcfbabb6740 20 client.18363 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:40.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.926+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/3026099898 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c5299d57f0 0x55c5299d7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.926+0000 7fcfbabb6740 4 client.18363 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:40.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.926+0000 7fcfbabb6740 1 -- 172.21.15.107:0/3026099898 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c52992d370 con 0x55c5299d57f0 2024-05-09T08:42:40.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.926+0000 7fcfb5449640 1 --2- 172.21.15.107:0/3026099898 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c5299d57f0 0x55c5299d7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.927+0000 7fcf957fa640 10 client.18363 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:40.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.927+0000 7fcfb5449640 1 --2- 172.21.15.107:0/3026099898 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c5299d57f0 0x55c5299d7c30 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bf468dde650aaf8 server_cookie=15154a067d0910a8 in_seq=0 out_seq=0 2024-05-09T08:42:40.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.927+0000 7fcf957fa640 1 -- 172.21.15.107:0/3026099898 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c52992d370 con 0x55c5299d57f0 2024-05-09T08:42:40.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.927+0000 7fcf957fa640 10 client.18363 handle_command_reply: tid=0 2024-05-09T08:42:40.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.927+0000 7fcfbabb6740 1 client.18363 shutdown 2024-05-09T08:42:40.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.927+0000 7fcfbabb6740 1 -- 172.21.15.107:0/3026099898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299bac50 msgr2=0x55c5299c27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.927+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/3026099898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299bac50 0x55c5299c27b0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fcfa800ab60 tx=0x7fcfa8011580 comp rx=0 tx=0).stop 2024-05-09T08:42:40.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.928+0000 7fcfbabb6740 1 -- 172.21.15.107:0/3026099898 shutdown_connections 2024-05-09T08:42:40.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.928+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/3026099898 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c5299d57f0 0x55c5299d7c30 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.928+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/3026099898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299c5320 0x55c5299c2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.928+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/3026099898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299bac50 0x55c5299c27b0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.928+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/3026099898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299ba330 0x55c5299c2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.928+0000 7fcf957fa640 0 client.18363 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:40.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.928+0000 7fcfbabb6740 1 -- 172.21.15.107:0/3026099898 >> 172.21.15.107:0/3026099898 conn(0x55c5299b6650 msgr2=0x55c5299bc450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.928+0000 7fcfbabb6740 1 -- 172.21.15.107:0/3026099898 shutdown_connections 2024-05-09T08:42:40.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.928+0000 7fcfbabb6740 1 -- 172.21.15.107:0/3026099898 wait complete. 2024-05-09T08:42:40.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.928+0000 7fcfbabb6740 20 client.18363 trim_cache size 0 max 16384 2024-05-09T08:42:40.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.941+0000 7fcfbabb6740 1 Processor -- start 2024-05-09T08:42:40.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.941+0000 7fcfbabb6740 1 -- start start 2024-05-09T08:42:40.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.941+0000 7fcfbabb6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299ba330 0x55c5299d1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.941+0000 7fcfbabb6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299bac50 0x55c5299c13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.941+0000 7fcfbabb6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299c5320 0x55c5299bca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.941+0000 7fcfbabb6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb01b1470 con 0x55c5299c5320 2024-05-09T08:42:40.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.941+0000 7fcfbabb6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb01aed40 con 0x55c5299bac50 2024-05-09T08:42:40.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.941+0000 7fcfbabb6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb01acbb0 con 0x55c5299ba330 2024-05-09T08:42:40.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.941+0000 7fcfb5449640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299c5320 0x55c5299bca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.941+0000 7fcfb5449640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299ba330 0x55c5299d1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.941+0000 7fcfb5449640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299ba330 0x55c5299d1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47628/0 (socket says 172.21.15.107:47628) 2024-05-09T08:42:40.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.941+0000 7fcfb5449640 1 -- 172.21.15.107:0/4056629366 learned_addr learned my addr 172.21.15.107:0/4056629366 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:40.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.941+0000 7fcfb5c4a640 1 --2- 172.21.15.107:0/4056629366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299bac50 0x55c5299c13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.941+0000 7fcfb5449640 1 -- 172.21.15.107:0/4056629366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299bac50 msgr2=0x55c5299c13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.942+0000 7fcfb5449640 1 --2- 172.21.15.107:0/4056629366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299bac50 0x55c5299c13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.942+0000 7fcfb5449640 1 -- 172.21.15.107:0/4056629366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299c5320 msgr2=0x55c5299bca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.942+0000 7fcfb5449640 1 --2- 172.21.15.107:0/4056629366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299c5320 0x55c5299bca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.942+0000 7fcfb5449640 1 -- 172.21.15.107:0/4056629366 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5299a7d20 con 0x55c5299ba330 2024-05-09T08:42:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.942+0000 7fcfb5449640 1 --2- 172.21.15.107:0/4056629366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299ba330 0x55c5299d1000 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x55c5299b77e0 tx=0x7fcfa400cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c08ad202b19b5236 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.942+0000 7fcf977fe640 1 -- 172.21.15.107:0/4056629366 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa4012450 con 0x55c5299ba330 2024-05-09T08:42:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.942+0000 7fcf977fe640 1 -- 172.21.15.107:0/4056629366 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfa4014600 con 0x55c5299ba330 2024-05-09T08:42:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.942+0000 7fcf977fe640 1 -- 172.21.15.107:0/4056629366 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa402f440 con 0x55c5299ba330 2024-05-09T08:42:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.942+0000 7fcfbabb6740 1 -- 172.21.15.107:0/4056629366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299ba330 msgr2=0x55c5299d1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.942+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/4056629366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299ba330 0x55c5299d1000 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x55c5299b77e0 tx=0x7fcfa400cc10 comp rx=0 tx=0).stop 2024-05-09T08:42:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.942+0000 7fcfbabb6740 1 -- 172.21.15.107:0/4056629366 shutdown_connections 2024-05-09T08:42:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.942+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/4056629366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299c5320 0x55c5299bca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.942+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/4056629366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299bac50 0x55c5299c13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.942+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/4056629366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299ba330 0x55c5299d1000 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.942+0000 7fcfbabb6740 1 -- 172.21.15.107:0/4056629366 >> 172.21.15.107:0/4056629366 conn(0x55c5299b6650 msgr2=0x55c5299b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.943+0000 7fcfbabb6740 1 -- 172.21.15.107:0/4056629366 shutdown_connections 2024-05-09T08:42:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.943+0000 7fcfbabb6740 1 -- 172.21.15.107:0/4056629366 wait complete. 2024-05-09T08:42:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.943+0000 7fcfbabb6740 1 -- start start 2024-05-09T08:42:40.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.943+0000 7fcfbabb6740 1 Processor -- start 2024-05-09T08:42:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.943+0000 7fcfbabb6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299ba330 0x55c5299bf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.943+0000 7fcfbabb6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299bac50 0x55c5299bf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.943+0000 7fcfbabb6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299c5320 0x55c5299bfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.943+0000 7fcfbabb6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb01ad200 con 0x55c5299ba330 2024-05-09T08:42:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.943+0000 7fcfbabb6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb01ae670 con 0x55c5299c5320 2024-05-09T08:42:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.943+0000 7fcfbabb6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb01ad8d0 con 0x55c5299bac50 2024-05-09T08:42:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.943+0000 7fcfbabb6740 10 client.0 _ll_register_callbacks cb 0x7fcfa8009b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.943+0000 7fcfb5449640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299ba330 0x55c5299bf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.943+0000 7fcfb5449640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299ba330 0x55c5299bf110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37590/0 (socket says 172.21.15.107:37590) 2024-05-09T08:42:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.943+0000 7fcfb5449640 1 -- 172.21.15.107:0/3001818276 learned_addr learned my addr 172.21.15.107:0/3001818276 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.943+0000 7fcfb5c4a640 1 --2- 172.21.15.107:0/3001818276 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299bac50 0x55c5299bf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.943+0000 7fcfb5449640 1 --2- 172.21.15.107:0/3001818276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299c5320 0x55c5299bfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcfb5449640 1 -- 172.21.15.107:0/3001818276 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299bac50 msgr2=0x55c5299bf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcfb5449640 1 --2- 172.21.15.107:0/3001818276 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299bac50 0x55c5299bf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcfb5449640 1 -- 172.21.15.107:0/3001818276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299c5320 msgr2=0x55c5299bfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcfb5449640 1 --2- 172.21.15.107:0/3001818276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299c5320 0x55c5299bfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcfb5449640 1 -- 172.21.15.107:0/3001818276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c529a3af20 con 0x55c5299ba330 2024-05-09T08:42:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcfb5449640 1 --2- 172.21.15.107:0/3001818276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299ba330 0x55c5299bf110 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4005780 tx=0x7fcfa400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba426645d1ae155 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcf957fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcf957fa640 1 -- 172.21.15.107:0/3001818276 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa4004170 con 0x55c5299ba330 2024-05-09T08:42:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcf957fa640 1 -- 172.21.15.107:0/3001818276 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fcfa4004310 con 0x55c5299ba330 2024-05-09T08:42:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcfbabb6740 1 -- 172.21.15.107:0/3001818276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfa4014600 con 0x55c5299ba330 2024-05-09T08:42:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcf957fa640 1 -- 172.21.15.107:0/3001818276 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcfa4012840 con 0x55c5299ba330 2024-05-09T08:42:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcfbabb6740 10 client.18369 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcfbabb6740 1 -- 172.21.15.107:0/3001818276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c529a3c200 con 0x55c5299ba330 2024-05-09T08:42:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcf957fa640 1 -- 172.21.15.107:0/3001818276 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfa4014600 con 0x55c5299ba330 2024-05-09T08:42:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcf957fa640 1 -- 172.21.15.107:0/3001818276 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fcfa402fce0 con 0x55c5299ba330 2024-05-09T08:42:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcfbabb6740 10 client.18369 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcfbabb6740 10 client.18369 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcfbabb6740 20 client.18369 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/3001818276 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c529a3c8c0 0x55c529a3ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcfbabb6740 4 client.18369 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:42:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.944+0000 7fcfbabb6740 1 -- 172.21.15.107:0/3001818276 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c5299276c0 con 0x55c529a3c8c0 2024-05-09T08:42:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.945+0000 7fcfb5c4a640 1 --2- 172.21.15.107:0/3001818276 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c529a3c8c0 0x55c529a3ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.945+0000 7fcfb5c4a640 1 --2- 172.21.15.107:0/3001818276 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c529a3c8c0 0x55c529a3ece0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abbaf56d4fa475d9 server_cookie=7b7ac5c384b3029 in_seq=0 out_seq=0 2024-05-09T08:42:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.945+0000 7fcf957fa640 10 client.18369 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.945+0000 7fcf957fa640 1 -- 172.21.15.107:0/3001818276 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c5299276c0 con 0x55c529a3c8c0 2024-05-09T08:42:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.945+0000 7fcf957fa640 10 client.18369 handle_command_reply: tid=0 2024-05-09T08:42:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.945+0000 7fcfbabb6740 1 client.18369 shutdown 2024-05-09T08:42:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.945+0000 7fcfbabb6740 1 -- 172.21.15.107:0/3001818276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299ba330 msgr2=0x55c5299bf110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.945+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/3001818276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299ba330 0x55c5299bf110 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4005780 tx=0x7fcfa400c2d0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.945+0000 7fcfbabb6740 1 -- 172.21.15.107:0/3001818276 shutdown_connections 2024-05-09T08:42:40.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.945+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/3001818276 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c529a3c8c0 0x55c529a3ece0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/3001818276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c5299c5320 0x55c5299bfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/3001818276 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c5299bac50 0x55c5299bf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfbabb6740 1 --2- 172.21.15.107:0/3001818276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c5299ba330 0x55c5299bf110 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcf957fa640 0 client.18369 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:40.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfbabb6740 1 -- 172.21.15.107:0/3001818276 >> 172.21.15.107:0/3001818276 conn(0x55c5299b6650 msgr2=0x55c5299b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfbabb6740 1 -- 172.21.15.107:0/3001818276 shutdown_connections 2024-05-09T08:42:40.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfbabb6740 1 -- 172.21.15.107:0/3001818276 wait complete. 2024-05-09T08:42:40.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfbabb6740 20 client.18369 trim_cache size 0 max 16384 2024-05-09T08:42:40.952 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:42:40.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfb7eae640 1 -- 172.21.15.107:0/2759011900 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcf980a2d80 msgr2=0x7fcf980a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfb7eae640 1 --2- 172.21.15.107:0/2759011900 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcf980a2d80 0x7fcf980a5240 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcfa00ed410 tx=0x7fcfa008e0f0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfb7eae640 1 -- 172.21.15.107:0/2759011900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfb0107910 msgr2=0x7fcfb019c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:40.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfb7eae640 1 --2- 172.21.15.107:0/2759011900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfb0107910 0x7fcfb019c8d0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0005ab0 tx=0x7fcfa00042f0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfb7eae640 1 -- 172.21.15.107:0/2759011900 shutdown_connections 2024-05-09T08:42:40.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfb7eae640 1 --2- 172.21.15.107:0/2759011900 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcf980a2d80 0x7fcf980a5240 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfb7eae640 1 --2- 172.21.15.107:0/2759011900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfb0107910 0x7fcfb019c8d0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfb7eae640 1 --2- 172.21.15.107:0/2759011900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcfb006a7f0 0x7fcfb01985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfb7eae640 1 --2- 172.21.15.107:0/2759011900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfb0069e50 0x7fcfb0198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:40.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfb7eae640 1 -- 172.21.15.107:0/2759011900 >> 172.21.15.107:0/2759011900 conn(0x7fcfb0077710 msgr2=0x7fcfb00ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:40.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfb7eae640 1 -- 172.21.15.107:0/2759011900 shutdown_connections 2024-05-09T08:42:40.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:40.946+0000 7fcfb7eae640 1 -- 172.21.15.107:0/2759011900 wait complete. 2024-05-09T08:42:41.956 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:42:41.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:42:41.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:42:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.270+0000 7ff6db59e640 1 -- 172.21.15.107:0/3068561571 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6dc1092b0 msgr2=0x7ff6dc11d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.270+0000 7ff6db59e640 1 --2- 172.21.15.107:0/3068561571 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6dc1092b0 0x7ff6dc11d020 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009ca0 tx=0x7ff6d402a980 comp rx=0 tx=0).stop 2024-05-09T08:42:42.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.274+0000 7ff6db59e640 1 -- 172.21.15.107:0/3068561571 shutdown_connections 2024-05-09T08:42:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.274+0000 7ff6db59e640 1 --2- 172.21.15.107:0/3068561571 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6dc1092b0 0x7ff6dc11d020 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.274+0000 7ff6db59e640 1 --2- 172.21.15.107:0/3068561571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6dc108830 0x7ff6dc108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.274+0000 7ff6db59e640 1 --2- 172.21.15.107:0/3068561571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6dc10fed0 0x7ff6dc1102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.274+0000 7ff6db59e640 1 -- 172.21.15.107:0/3068561571 >> 172.21.15.107:0/3068561571 conn(0x7ff6dc06c7b0 msgr2=0x7ff6dc06cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:42.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.277+0000 7ff6db59e640 1 -- 172.21.15.107:0/3068561571 shutdown_connections 2024-05-09T08:42:42.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.281+0000 7ff6db59e640 1 -- 172.21.15.107:0/3068561571 wait complete. 2024-05-09T08:42:42.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.282+0000 7ff6db59e640 1 Processor -- start 2024-05-09T08:42:42.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.282+0000 7ff6db59e640 1 -- start start 2024-05-09T08:42:42.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.282+0000 7ff6db59e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6dc108830 0x7ff6dc1aca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.282+0000 7ff6db59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6dc1092b0 0x7ff6dc1acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.282+0000 7ff6db59e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6dc10fed0 0x7ff6dc1a6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.282+0000 7ff6db59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc1207b0 con 0x7ff6dc1092b0 2024-05-09T08:42:42.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.283+0000 7ff6db59e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc120630 con 0x7ff6dc10fed0 2024-05-09T08:42:42.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.283+0000 7ff6db59e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc120930 con 0x7ff6dc108830 2024-05-09T08:42:42.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.283+0000 7ff6da59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6dc108830 0x7ff6dc1aca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.283+0000 7ff6da59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6dc108830 0x7ff6dc1aca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47648/0 (socket says 172.21.15.107:47648) 2024-05-09T08:42:42.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.283+0000 7ff6da59c640 1 -- 172.21.15.107:0/2617983980 learned_addr learned my addr 172.21.15.107:0/2617983980 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:42.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.283+0000 7ff6d9d9b640 1 --2- 172.21.15.107:0/2617983980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6dc1092b0 0x7ff6dc1acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.283+0000 7ff6d9d9b640 1 -- 172.21.15.107:0/2617983980 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6dc108830 msgr2=0x7ff6dc1aca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.283+0000 7ff6d9d9b640 1 --2- 172.21.15.107:0/2617983980 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6dc108830 0x7ff6dc1aca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.283+0000 7ff6d9d9b640 1 -- 172.21.15.107:0/2617983980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6dc10fed0 msgr2=0x7ff6dc1a6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.283+0000 7ff6d9d9b640 1 --2- 172.21.15.107:0/2617983980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6dc10fed0 0x7ff6dc1a6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.283+0000 7ff6d9d9b640 1 -- 172.21.15.107:0/2617983980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc1a7340 con 0x7ff6dc1092b0 2024-05-09T08:42:42.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.284+0000 7ff6d9d9b640 1 --2- 172.21.15.107:0/2617983980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6dc1092b0 0x7ff6dc1acfb0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000d6d0 tx=0x7ff6d000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:42.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.284+0000 7ff6cb7fe640 1 -- 172.21.15.107:0/2617983980 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0017070 con 0x7ff6dc1092b0 2024-05-09T08:42:42.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.284+0000 7ff6db59e640 1 -- 172.21.15.107:0/2617983980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6dc1a7630 con 0x7ff6dc1092b0 2024-05-09T08:42:42.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.284+0000 7ff6cb7fe640 1 -- 172.21.15.107:0/2617983980 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff6d0009820 con 0x7ff6dc1092b0 2024-05-09T08:42:42.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.284+0000 7ff6cb7fe640 1 -- 172.21.15.107:0/2617983980 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0004ba0 con 0x7ff6dc1092b0 2024-05-09T08:42:42.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.284+0000 7ff6db59e640 1 -- 172.21.15.107:0/2617983980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6dc1107f0 con 0x7ff6dc1092b0 2024-05-09T08:42:42.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.289+0000 7ff6e3978740 1 Processor -- start 2024-05-09T08:42:42.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.289+0000 7ff6cb7fe640 1 -- 172.21.15.107:0/2617983980 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ff6d0010460 con 0x7ff6dc1092b0 2024-05-09T08:42:42.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.289+0000 7ff6e3978740 1 -- start start 2024-05-09T08:42:42.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.289+0000 7ff6e3978740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b3c2a0 0x55aac0b3a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.289+0000 7ff6e3978740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b3abb0 0x55aac0b3b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.290+0000 7ff6e3978740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b45930 0x55aac0b47d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.290+0000 7ff6e3978740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55aac0a9f990 con 0x55aac0b3abb0 2024-05-09T08:42:42.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.290+0000 7ff6e3978740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55aac0a475b0 con 0x55aac0b45930 2024-05-09T08:42:42.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.290+0000 7ff6e3978740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55aac0a9b950 con 0x55aac0b3c2a0 2024-05-09T08:42:42.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.290+0000 7ff6dad9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b3c2a0 0x55aac0b3a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.290+0000 7ff6dad9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b3c2a0 0x55aac0b3a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47650/0 (socket says 172.21.15.107:47650) 2024-05-09T08:42:42.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.290+0000 7ff6dad9d640 1 -- 172.21.15.107:0/3627484447 learned_addr learned my addr 172.21.15.107:0/3627484447 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:42.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.290+0000 7ff6d9d9b640 1 --2- 172.21.15.107:0/3627484447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b45930 0x55aac0b47d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.291+0000 7ff6cb7fe640 1 --2- 172.21.15.107:0/2617983980 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff6c40a2e80 0x7ff6c40a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.291+0000 7ff6cb7fe640 1 -- 172.21.15.107:0/2617983980 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7ff6d00cf090 con 0x7ff6dc1092b0 2024-05-09T08:42:42.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.297+0000 7ff6da59c640 1 --2- 172.21.15.107:0/3627484447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b3abb0 0x55aac0b3b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.297+0000 7ff6d9d9b640 1 -- 172.21.15.107:0/3627484447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b3c2a0 msgr2=0x55aac0b3a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:42.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.297+0000 7ff6d9d9b640 1 --2- 172.21.15.107:0/3627484447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b3c2a0 0x55aac0b3a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.297+0000 7ff6d9d9b640 1 -- 172.21.15.107:0/3627484447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b3abb0 msgr2=0x55aac0b3b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:42.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.297+0000 7ff6d9d9b640 1 --2- 172.21.15.107:0/3627484447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b3abb0 0x55aac0b3b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.297+0000 7ff6d9d9b640 1 -- 172.21.15.107:0/3627484447 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aac087b7e0 con 0x55aac0b45930 2024-05-09T08:42:42.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.299+0000 7ff6d9d9b640 1 --2- 172.21.15.107:0/3627484447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b45930 0x55aac0b47d20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff6d00cc550 tx=0x7ff6d009bca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4907ed2e2ae0c85 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:42.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.302+0000 7ff6c8ff9640 1 -- 172.21.15.107:0/3627484447 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0020050 con 0x55aac0b45930 2024-05-09T08:42:42.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.302+0000 7ff6c8ff9640 1 -- 172.21.15.107:0/3627484447 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff6d009c860 con 0x55aac0b45930 2024-05-09T08:42:42.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.302+0000 7ff6c8ff9640 1 -- 172.21.15.107:0/3627484447 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d00bc0e0 con 0x55aac0b45930 2024-05-09T08:42:42.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.302+0000 7ff6dad9d640 1 --2- 172.21.15.107:0/2617983980 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff6c40a2e80 0x7ff6c40a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.304+0000 7ff6dad9d640 1 --2- 172.21.15.107:0/2617983980 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff6c40a2e80 0x7ff6c40a5340 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4005e30 tx=0x7ff6d402ae80 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:42.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.307+0000 7ff6e3978740 1 -- 172.21.15.107:0/3627484447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b45930 msgr2=0x55aac0b47d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.307+0000 7ff6e3978740 1 --2- 172.21.15.107:0/3627484447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b45930 0x55aac0b47d20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff6d00cc550 tx=0x7ff6d009bca0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.307+0000 7ff6e3978740 1 -- 172.21.15.107:0/3627484447 shutdown_connections 2024-05-09T08:42:42.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.307+0000 7ff6e3978740 1 --2- 172.21.15.107:0/3627484447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b45930 0x55aac0b47d20 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff6d00cc550 tx=0x7ff6d009bca0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.307+0000 7ff6e3978740 1 --2- 172.21.15.107:0/3627484447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b3abb0 0x55aac0b3b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.307+0000 7ff6e3978740 1 --2- 172.21.15.107:0/3627484447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b3c2a0 0x55aac0b3a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.307+0000 7ff6e3978740 1 -- 172.21.15.107:0/3627484447 >> 172.21.15.107:0/3627484447 conn(0x55aac0b030a0 msgr2=0x55aac0b37110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:42.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.309+0000 7ff6e3978740 1 -- 172.21.15.107:0/3627484447 shutdown_connections 2024-05-09T08:42:42.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.309+0000 7ff6e3978740 1 -- 172.21.15.107:0/3627484447 wait complete. 2024-05-09T08:42:42.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.310+0000 7ff6e3978740 1 -- start start 2024-05-09T08:42:42.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.310+0000 7ff6e3978740 1 Processor -- start 2024-05-09T08:42:42.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.311+0000 7ff6e3978740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b3abb0 0x55aac0b429d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.311+0000 7ff6e3978740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b3c2a0 0x55aac0b42f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.311+0000 7ff6e3978740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b43450 0x55aac0b51c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.311+0000 7ff6e3978740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55aac0b48760 con 0x55aac0b43450 2024-05-09T08:42:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.311+0000 7ff6e3978740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55aac0a5ab40 con 0x55aac0b3abb0 2024-05-09T08:42:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.311+0000 7ff6d9d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b3c2a0 0x55aac0b42f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.311+0000 7ff6e3978740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55aac0b488e0 con 0x55aac0b3c2a0 2024-05-09T08:42:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.311+0000 7ff6d9d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b3c2a0 0x55aac0b42f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47662/0 (socket says 172.21.15.107:47662) 2024-05-09T08:42:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.311+0000 7ff6d9d9b640 1 -- 172.21.15.107:0/2814246499 learned_addr learned my addr 172.21.15.107:0/2814246499 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.311+0000 7ff6e3978740 10 client.0 _ll_register_callbacks cb 0x55aac0a4f0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.311+0000 7ff6dad9d640 1 --2- 172.21.15.107:0/2814246499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b43450 0x55aac0b51c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.312+0000 7ff6d9d9b640 1 -- 172.21.15.107:0/2814246499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b3abb0 msgr2=0x55aac0b429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:42.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.312+0000 7ff6d9d9b640 1 --2- 172.21.15.107:0/2814246499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b3abb0 0x55aac0b429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.312+0000 7ff6d9d9b640 1 -- 172.21.15.107:0/2814246499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b43450 msgr2=0x55aac0b51c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:42.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.312+0000 7ff6d9d9b640 1 --2- 172.21.15.107:0/2814246499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b43450 0x55aac0b51c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.312+0000 7ff6d9d9b640 1 -- 172.21.15.107:0/2814246499 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aac08dbf70 con 0x55aac0b3c2a0 2024-05-09T08:42:42.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.312+0000 7ff6d9d9b640 1 --2- 172.21.15.107:0/2814246499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b3c2a0 0x55aac0b42f10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009610 tx=0x7ff6d0004f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5358e2a293ac0690 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:42.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.312+0000 7ff6c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:42:42.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.312+0000 7ff6c1ffb640 1 -- 172.21.15.107:0/2814246499 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0020070 con 0x55aac0b3c2a0 2024-05-09T08:42:42.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.312+0000 7ff6e3978740 1 -- 172.21.15.107:0/2814246499 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6d009c860 con 0x55aac0b3c2a0 2024-05-09T08:42:42.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.312+0000 7ff6c1ffb640 1 -- 172.21.15.107:0/2814246499 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7ff6d00987d0 con 0x55aac0b3c2a0 2024-05-09T08:42:42.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.313+0000 7ff6c1ffb640 1 -- 172.21.15.107:0/2814246499 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff6d00a0510 con 0x55aac0b3c2a0 2024-05-09T08:42:42.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.315+0000 7ff6e3978740 10 client.27574 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:42.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.315+0000 7ff6e3978740 1 -- 172.21.15.107:0/2814246499 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55aac0773170 con 0x55aac0b3c2a0 2024-05-09T08:42:42.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.317+0000 7ff6c1ffb640 1 -- 172.21.15.107:0/2814246499 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff6d009c860 con 0x55aac0b3c2a0 2024-05-09T08:42:42.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.317+0000 7ff6c1ffb640 1 -- 172.21.15.107:0/2814246499 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff6d00bc0b0 con 0x55aac0b3c2a0 2024-05-09T08:42:42.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.322+0000 7ff6e3978740 10 client.27574 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.322+0000 7ff6e3978740 10 client.27574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.322+0000 7ff6e3978740 20 client.27574 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.322+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2814246499 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55aac0b55df0 0x55aac0b581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.322+0000 7ff6e3978740 4 client.27574 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:42.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.322+0000 7ff6e3978740 1 -- 172.21.15.107:0/2814246499 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aac0aad0c0 con 0x55aac0b55df0 2024-05-09T08:42:42.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.326+0000 7ff6da59c640 1 --2- 172.21.15.107:0/2814246499 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55aac0b55df0 0x55aac0b581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.342+0000 7ff6da59c640 1 --2- 172.21.15.107:0/2814246499 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55aac0b55df0 0x55aac0b581e0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3070ed13c9f678c5 server_cookie=adc02ee4918f3af7 in_seq=0 out_seq=0 2024-05-09T08:42:42.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.343+0000 7ff6c1ffb640 10 client.27574 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:42.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.344+0000 7ff6c1ffb640 1 -- 172.21.15.107:0/2814246499 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aac0aad0c0 con 0x55aac0b55df0 2024-05-09T08:42:42.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.344+0000 7ff6c1ffb640 10 client.27574 handle_command_reply: tid=0 2024-05-09T08:42:42.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.344+0000 7ff6e3978740 1 client.27574 shutdown 2024-05-09T08:42:42.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.345+0000 7ff6e3978740 1 -- 172.21.15.107:0/2814246499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b3c2a0 msgr2=0x55aac0b42f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.345+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2814246499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b3c2a0 0x55aac0b42f10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009610 tx=0x7ff6d0004f70 comp rx=0 tx=0).stop 2024-05-09T08:42:42.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.347+0000 7ff6e3978740 1 -- 172.21.15.107:0/2814246499 shutdown_connections 2024-05-09T08:42:42.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.347+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2814246499 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55aac0b55df0 0x55aac0b581e0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.347+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2814246499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b43450 0x55aac0b51c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.347+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2814246499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b3c2a0 0x55aac0b42f10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.347+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2814246499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b3abb0 0x55aac0b429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.347+0000 7ff6e3978740 1 -- 172.21.15.107:0/2814246499 >> 172.21.15.107:0/2814246499 conn(0x55aac0b030a0 msgr2=0x55aac0b3cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:42.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.347+0000 7ff6c1ffb640 0 client.27574 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:42.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.350+0000 7ff6e3978740 1 -- 172.21.15.107:0/2814246499 shutdown_connections 2024-05-09T08:42:42.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.350+0000 7ff6e3978740 1 -- 172.21.15.107:0/2814246499 wait complete. 2024-05-09T08:42:42.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.350+0000 7ff6e3978740 20 client.27574 trim_cache size 0 max 16384 2024-05-09T08:42:42.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.393+0000 7ff6e3978740 1 Processor -- start 2024-05-09T08:42:42.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.393+0000 7ff6e3978740 1 -- start start 2024-05-09T08:42:42.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.393+0000 7ff6e3978740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b3abb0 0x55aac0b37240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.393+0000 7ff6e3978740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b3c2a0 0x55aac0b51bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.394+0000 7ff6da59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b3abb0 0x55aac0b37240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.394+0000 7ff6da59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b3abb0 0x55aac0b37240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:37776/0 (socket says 172.21.15.107:37776) 2024-05-09T08:42:42.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.394+0000 7ff6da59c640 1 -- 172.21.15.107:0/2270074975 learned_addr learned my addr 172.21.15.107:0/2270074975 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:42.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.394+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2270074975 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b43450 0x55aac0b5ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.394+0000 7ff6e3978740 1 -- 172.21.15.107:0/2270074975 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc0128f0 con 0x55aac0b3c2a0 2024-05-09T08:42:42.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.394+0000 7ff6e3978740 1 -- 172.21.15.107:0/2270074975 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc0101c0 con 0x55aac0b3abb0 2024-05-09T08:42:42.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.394+0000 7ff6e3978740 1 -- 172.21.15.107:0/2270074975 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc00e030 con 0x55aac0b43450 2024-05-09T08:42:42.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.394+0000 7ff6d9d9b640 1 --2- 172.21.15.107:0/2270074975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b3c2a0 0x55aac0b51bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.394+0000 7ff6dad9d640 1 --2- 172.21.15.107:0/2270074975 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b43450 0x55aac0b5ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.394+0000 7ff6da59c640 1 -- 172.21.15.107:0/2270074975 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b43450 msgr2=0x55aac0b5ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:42.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.394+0000 7ff6da59c640 1 --2- 172.21.15.107:0/2270074975 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b43450 0x55aac0b5ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.394+0000 7ff6da59c640 1 -- 172.21.15.107:0/2270074975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b3c2a0 msgr2=0x55aac0b51bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:42.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.394+0000 7ff6da59c640 1 --2- 172.21.15.107:0/2270074975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b3c2a0 0x55aac0b51bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.394+0000 7ff6da59c640 1 -- 172.21.15.107:0/2270074975 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aac0b41980 con 0x55aac0b3abb0 2024-05-09T08:42:42.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.395+0000 7ff6d9d9b640 1 --2- 172.21.15.107:0/2270074975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b3c2a0 0x55aac0b51bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:42:42.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.395+0000 7ff6da59c640 1 --2- 172.21.15.107:0/2270074975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b3abb0 0x55aac0b37240 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc000c00 tx=0x7ff6cc00b750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a766b205f0cb40a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:42.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.396+0000 7ff6c3fff640 1 -- 172.21.15.107:0/2270074975 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc0192d0 con 0x55aac0b3abb0 2024-05-09T08:42:42.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.396+0000 7ff6c3fff640 1 -- 172.21.15.107:0/2270074975 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff6cc01d030 con 0x55aac0b3abb0 2024-05-09T08:42:42.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.396+0000 7ff6c3fff640 1 -- 172.21.15.107:0/2270074975 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc016d80 con 0x55aac0b3abb0 2024-05-09T08:42:42.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.397+0000 7ff6e3978740 1 -- 172.21.15.107:0/2270074975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b3abb0 msgr2=0x55aac0b37240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.397+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2270074975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b3abb0 0x55aac0b37240 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc000c00 tx=0x7ff6cc00b750 comp rx=0 tx=0).stop 2024-05-09T08:42:42.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.399+0000 7ff6e3978740 1 -- 172.21.15.107:0/2270074975 shutdown_connections 2024-05-09T08:42:42.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.399+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2270074975 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b43450 0x55aac0b5ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.399+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2270074975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b3c2a0 0x55aac0b51bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.399+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2270074975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b3abb0 0x55aac0b37240 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc000c00 tx=0x7ff6cc00b750 comp rx=0 tx=0).stop 2024-05-09T08:42:42.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.399+0000 7ff6e3978740 1 -- 172.21.15.107:0/2270074975 >> 172.21.15.107:0/2270074975 conn(0x55aac0b030a0 msgr2=0x55aac0b36b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:42.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.400+0000 7ff6e3978740 1 -- 172.21.15.107:0/2270074975 shutdown_connections 2024-05-09T08:42:42.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.401+0000 7ff6e3978740 1 -- 172.21.15.107:0/2270074975 wait complete. 2024-05-09T08:42:42.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.401+0000 7ff6e3978740 1 -- start start 2024-05-09T08:42:42.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.401+0000 7ff6e3978740 1 Processor -- start 2024-05-09T08:42:42.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.402+0000 7ff6e3978740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b3c2a0 0x55aac0b58ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.402+0000 7ff6e3978740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b43450 0x55aac0b59010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.402+0000 7ff6e3978740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b59550 0x55aac0bbafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.402+0000 7ff6e3978740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc00e680 con 0x55aac0b3c2a0 2024-05-09T08:42:42.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.403+0000 7ff6e3978740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc00faf0 con 0x55aac0b43450 2024-05-09T08:42:42.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.403+0000 7ff6e3978740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc00ed50 con 0x55aac0b59550 2024-05-09T08:42:42.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.403+0000 7ff6dad9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b59550 0x55aac0bbafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.403+0000 7ff6dad9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b59550 0x55aac0bbafd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47678/0 (socket says 172.21.15.107:47678) 2024-05-09T08:42:42.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.403+0000 7ff6dad9d640 1 -- 172.21.15.107:0/2916551374 learned_addr learned my addr 172.21.15.107:0/2916551374 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:42.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.403+0000 7ff6e3978740 10 client.0 _ll_register_callbacks cb 0x7ff6d00984b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:42.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.405+0000 7ff6d9d9b640 1 --2- 172.21.15.107:0/2916551374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b43450 0x55aac0b59010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.405+0000 7ff6dad9d640 1 -- 172.21.15.107:0/2916551374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b43450 msgr2=0x55aac0b59010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:42.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.405+0000 7ff6dad9d640 1 --2- 172.21.15.107:0/2916551374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b43450 0x55aac0b59010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.405+0000 7ff6dad9d640 1 -- 172.21.15.107:0/2916551374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b3c2a0 msgr2=0x55aac0b58ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:42.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.405+0000 7ff6dad9d640 1 --2- 172.21.15.107:0/2916551374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b3c2a0 0x55aac0b58ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.405+0000 7ff6dad9d640 1 -- 172.21.15.107:0/2916551374 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aac0bbb6c0 con 0x55aac0b59550 2024-05-09T08:42:42.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.406+0000 7ff6dad9d640 1 --2- 172.21.15.107:0/2916551374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b59550 0x55aac0bbafd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff6d402aef0 tx=0x7ff6d40095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=772cd7be6562323 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:42.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.409+0000 7ff6c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:42:42.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.409+0000 7ff6c1ffb640 1 -- 172.21.15.107:0/2916551374 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4034040 con 0x55aac0b59550 2024-05-09T08:42:42.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.409+0000 7ff6c1ffb640 1 -- 172.21.15.107:0/2916551374 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7ff6d4030680 con 0x55aac0b59550 2024-05-09T08:42:42.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.409+0000 7ff6e3978740 1 -- 172.21.15.107:0/2916551374 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6cc01d030 con 0x55aac0b59550 2024-05-09T08:42:42.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.416+0000 7ff6c1ffb640 1 -- 172.21.15.107:0/2916551374 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff6d4030880 con 0x55aac0b59550 2024-05-09T08:42:42.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.419+0000 7ff6e3978740 10 client.27580 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:42.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.419+0000 7ff6e3978740 1 -- 172.21.15.107:0/2916551374 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55aac0bbc980 con 0x55aac0b59550 2024-05-09T08:42:42.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.426+0000 7ff6c1ffb640 1 -- 172.21.15.107:0/2916551374 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff6cc01d030 con 0x55aac0b59550 2024-05-09T08:42:42.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.426+0000 7ff6c1ffb640 1 -- 172.21.15.107:0/2916551374 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ff6d40045f0 con 0x55aac0b59550 2024-05-09T08:42:42.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.427+0000 7ff6e3978740 10 client.27580 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:42.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.427+0000 7ff6e3978740 10 client.27580 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:42.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.427+0000 7ff6e3978740 20 client.27580 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:42.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.427+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2916551374 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55aac0bbd080 0x55aac0bbf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.427+0000 7ff6e3978740 4 client.27580 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:42:42.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.427+0000 7ff6e3978740 1 -- 172.21.15.107:0/2916551374 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55aac0aa7410 con 0x55aac0bbd080 2024-05-09T08:42:42.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.429+0000 7ff6da59c640 1 --2- 172.21.15.107:0/2916551374 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55aac0bbd080 0x55aac0bbf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.430+0000 7ff6da59c640 1 --2- 172.21.15.107:0/2916551374 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55aac0bbd080 0x55aac0bbf500 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a94ededdccda7f62 server_cookie=e2e8efa0c9f820c8 in_seq=0 out_seq=0 2024-05-09T08:42:42.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.436+0000 7ff6c1ffb640 10 client.27580 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:42.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.436+0000 7ff6c1ffb640 1 -- 172.21.15.107:0/2916551374 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55aac0aa7410 con 0x55aac0bbd080 2024-05-09T08:42:42.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.436+0000 7ff6c1ffb640 10 client.27580 handle_command_reply: tid=0 2024-05-09T08:42:42.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.437+0000 7ff6e3978740 1 client.27580 shutdown 2024-05-09T08:42:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.441+0000 7ff6e3978740 1 -- 172.21.15.107:0/2916551374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b59550 msgr2=0x55aac0bbafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.441+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2916551374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b59550 0x55aac0bbafd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff6d402aef0 tx=0x7ff6d40095c0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.442+0000 7ff6e3978740 1 -- 172.21.15.107:0/2916551374 shutdown_connections 2024-05-09T08:42:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.442+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2916551374 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55aac0bbd080 0x55aac0bbf500 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.442+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2916551374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55aac0b59550 0x55aac0bbafd0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.442+0000 7ff6c1ffb640 0 client.27580 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.442+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2916551374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aac0b43450 0x55aac0b59010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.442+0000 7ff6e3978740 1 --2- 172.21.15.107:0/2916551374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aac0b3c2a0 0x55aac0b58ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.442+0000 7ff6e3978740 1 -- 172.21.15.107:0/2916551374 >> 172.21.15.107:0/2916551374 conn(0x55aac0b030a0 msgr2=0x55aac0b359e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:42.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.444+0000 7ff6e3978740 1 -- 172.21.15.107:0/2916551374 shutdown_connections 2024-05-09T08:42:42.447 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:42:42.447 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:42:42.447 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "0b13f178-f01a-43e0-a356-fe8e69742920", 2024-05-09T08:42:42.447 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:42:42.447 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:42:42.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.447+0000 7ff6e3978740 1 -- 172.21.15.107:0/2916551374 wait complete. 2024-05-09T08:42:42.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.448+0000 7ff6e3978740 20 client.27580 trim_cache size 0 max 16384 2024-05-09T08:42:42.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.450+0000 7ff6db59e640 1 -- 172.21.15.107:0/2617983980 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff6c40a2e80 msgr2=0x7ff6c40a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.451+0000 7ff6db59e640 1 --2- 172.21.15.107:0/2617983980 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff6c40a2e80 0x7ff6c40a5340 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4005e30 tx=0x7ff6d402ae80 comp rx=0 tx=0).stop 2024-05-09T08:42:42.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.451+0000 7ff6db59e640 1 -- 172.21.15.107:0/2617983980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6dc1092b0 msgr2=0x7ff6dc1acfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.451+0000 7ff6db59e640 1 --2- 172.21.15.107:0/2617983980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6dc1092b0 0x7ff6dc1acfb0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000d6d0 tx=0x7ff6d000db90 comp rx=0 tx=0).stop 2024-05-09T08:42:42.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.452+0000 7ff6db59e640 1 -- 172.21.15.107:0/2617983980 shutdown_connections 2024-05-09T08:42:42.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.452+0000 7ff6db59e640 1 --2- 172.21.15.107:0/2617983980 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ff6c40a2e80 0x7ff6c40a5340 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.452+0000 7ff6db59e640 1 --2- 172.21.15.107:0/2617983980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6dc10fed0 0x7ff6dc1a6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.452+0000 7ff6db59e640 1 --2- 172.21.15.107:0/2617983980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6dc1092b0 0x7ff6dc1acfb0 secure :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000d6d0 tx=0x7ff6d000db90 comp rx=0 tx=0).stop 2024-05-09T08:42:42.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.452+0000 7ff6db59e640 1 --2- 172.21.15.107:0/2617983980 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6dc108830 0x7ff6dc1aca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.452+0000 7ff6db59e640 1 -- 172.21.15.107:0/2617983980 >> 172.21.15.107:0/2617983980 conn(0x7ff6dc06c7b0 msgr2=0x7ff6dc070490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:42.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.456+0000 7ff6db59e640 1 -- 172.21.15.107:0/2617983980 shutdown_connections 2024-05-09T08:42:42.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.458+0000 7ff6db59e640 1 -- 172.21.15.107:0/2617983980 wait complete. 2024-05-09T08:42:42.491 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:42:42.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:42:42.492 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:42 smithi107 ceph-mon[30122]: pgmap v289: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 16 MiB/s wr, 4.04k op/s 2024-05-09T08:42:42.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:42 smithi157 ceph-mon[33614]: pgmap v289: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 16 MiB/s wr, 4.04k op/s 2024-05-09T08:42:42.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:42 smithi110 ceph-mon[33622]: pgmap v289: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 16 MiB/s wr, 4.04k op/s 2024-05-09T08:42:42.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.774+0000 7f12ab5ce640 1 -- 172.21.15.107:0/3518655118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12a410e0b0 msgr2=0x7f12a410e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.774+0000 7f12ab5ce640 1 --2- 172.21.15.107:0/3518655118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12a410e0b0 0x7f12a410e530 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1294009900 tx=0x7f129402a440 comp rx=0 tx=0).stop 2024-05-09T08:42:42.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.776+0000 7f12ab5ce640 1 -- 172.21.15.107:0/3518655118 shutdown_connections 2024-05-09T08:42:42.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.776+0000 7f12ab5ce640 1 --2- 172.21.15.107:0/3518655118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12a411b010 0x7f12a411d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.776+0000 7f12ab5ce640 1 --2- 172.21.15.107:0/3518655118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12a410e0b0 0x7f12a410e530 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.776+0000 7f12ab5ce640 1 --2- 172.21.15.107:0/3518655118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a4110080 0x7f12a410db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.776+0000 7f12ab5ce640 1 -- 172.21.15.107:0/3518655118 >> 172.21.15.107:0/3518655118 conn(0x7f12a406c940 msgr2=0x7f12a406cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:42.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.776+0000 7f12ab5ce640 1 -- 172.21.15.107:0/3518655118 shutdown_connections 2024-05-09T08:42:42.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.777+0000 7f12ab5ce640 1 -- 172.21.15.107:0/3518655118 wait complete. 2024-05-09T08:42:42.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.778+0000 7f12ab5ce640 1 Processor -- start 2024-05-09T08:42:42.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.778+0000 7f12ab5ce640 1 -- start start 2024-05-09T08:42:42.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.778+0000 7f12ab5ce640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12a410e0b0 0x7f12a41a6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.778+0000 7f12ab5ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12a4110080 0x7f12a41a73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.778+0000 7f12ab5ce640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a411b010 0x7f12a41ab720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.778+0000 7f12ab5ce640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f12a41208f0 con 0x7f12a4110080 2024-05-09T08:42:42.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.778+0000 7f12ab5ce640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f12a4120770 con 0x7f12a411b010 2024-05-09T08:42:42.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.778+0000 7f12ab5ce640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f12a4120a70 con 0x7f12a410e0b0 2024-05-09T08:42:42.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.780+0000 7f12a9b6b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a411b010 0x7f12a41ab720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.780+0000 7f12a9b6b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a411b010 0x7f12a41ab720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:37802/0 (socket says 172.21.15.107:37802) 2024-05-09T08:42:42.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.780+0000 7f12a9b6b640 1 -- 172.21.15.107:0/181999419 learned_addr learned my addr 172.21.15.107:0/181999419 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:42.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.780+0000 7f12a936a640 1 --2- 172.21.15.107:0/181999419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12a410e0b0 0x7f12a41a6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.782+0000 7f12a9b6b640 1 -- 172.21.15.107:0/181999419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12a410e0b0 msgr2=0x7f12a41a6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.782+0000 7f12a9b6b640 1 --2- 172.21.15.107:0/181999419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12a410e0b0 0x7f12a41a6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.782+0000 7f12a9b6b640 1 -- 172.21.15.107:0/181999419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12a4110080 msgr2=0x7f12a41a73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.782+0000 7f12a8b69640 1 --2- 172.21.15.107:0/181999419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12a4110080 0x7f12a41a73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.783+0000 7f12a9b6b640 1 --2- 172.21.15.107:0/181999419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12a4110080 0x7f12a41a73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.783+0000 7f12a9b6b640 1 -- 172.21.15.107:0/181999419 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a41abe20 con 0x7f12a411b010 2024-05-09T08:42:42.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.785+0000 7f12a9b6b640 1 --2- 172.21.15.107:0/181999419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a411b010 0x7f12a41ab720 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f12a000b450 tx=0x7f12a000b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:42.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.785+0000 7f12a936a640 1 --2- 172.21.15.107:0/181999419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12a410e0b0 0x7f12a41a6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:42:42.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.786+0000 7f12a8b69640 1 --2- 172.21.15.107:0/181999419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12a4110080 0x7f12a41a73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:42:42.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.790+0000 7f129a7fc640 1 -- 172.21.15.107:0/181999419 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a0015020 con 0x7f12a411b010 2024-05-09T08:42:42.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.790+0000 7f129a7fc640 1 -- 172.21.15.107:0/181999419 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12a0011d40 con 0x7f12a411b010 2024-05-09T08:42:42.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.790+0000 7f129a7fc640 1 -- 172.21.15.107:0/181999419 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a000f340 con 0x7f12a411b010 2024-05-09T08:42:42.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.791+0000 7f12ab5ce640 1 -- 172.21.15.107:0/181999419 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12a41ac110 con 0x7f12a411b010 2024-05-09T08:42:42.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.792+0000 7f12ab5ce640 1 -- 172.21.15.107:0/181999419 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12a41107f0 con 0x7f12a411b010 2024-05-09T08:42:42.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.795+0000 7f12ae3ba740 1 Processor -- start 2024-05-09T08:42:42.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.798+0000 7f129a7fc640 1 -- 172.21.15.107:0/181999419 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f12a0018020 con 0x7f12a411b010 2024-05-09T08:42:42.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.799+0000 7f129a7fc640 1 --2- 172.21.15.107:0/181999419 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f12900a2f00 0x7f12900a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.799+0000 7f12ae3ba740 1 -- start start 2024-05-09T08:42:42.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.799+0000 7f129a7fc640 1 -- 172.21.15.107:0/181999419 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f12a00cf0d0 con 0x7f12a411b010 2024-05-09T08:42:42.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.799+0000 7f12ae3ba740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c69694280 0x563c69694660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.799+0000 7f12ae3ba740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c69694ba0 0x563c6969ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.800+0000 7f12ae3ba740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6969f270 0x563c696a1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.800+0000 7f12ae3ba740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563c695f9d00 con 0x563c69694ba0 2024-05-09T08:42:42.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.800+0000 7f12ae3ba740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563c695a14c0 con 0x563c6969f270 2024-05-09T08:42:42.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.800+0000 7f12ae3ba740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563c695f5cc0 con 0x563c69694280 2024-05-09T08:42:42.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.800+0000 7f12a936a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c69694280 0x563c69694660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.800+0000 7f12a936a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c69694280 0x563c69694660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47698/0 (socket says 172.21.15.107:47698) 2024-05-09T08:42:42.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.800+0000 7f12a936a640 1 -- 172.21.15.107:0/1168492163 learned_addr learned my addr 172.21.15.107:0/1168492163 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:42.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.800+0000 7f12a936a640 1 -- 172.21.15.107:0/1168492163 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6969f270 msgr2=0x563c696a1660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:42:42.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.800+0000 7f12a936a640 1 --2- 172.21.15.107:0/1168492163 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6969f270 0x563c696a1660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.800+0000 7f12a936a640 1 -- 172.21.15.107:0/1168492163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c69694ba0 msgr2=0x563c6969ed30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:42:42.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.800+0000 7f12a936a640 1 --2- 172.21.15.107:0/1168492163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c69694ba0 0x563c6969ed30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.800+0000 7f12a936a640 1 -- 172.21.15.107:0/1168492163 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c693d77d0 con 0x563c69694280 2024-05-09T08:42:42.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.801+0000 7f12a936a640 1 --2- 172.21.15.107:0/1168492163 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c69694280 0x563c69694660 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f129c00f390 tx=0x7f129c00f850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41cfce485ca4d6c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:42.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.801+0000 7f128f7fe640 1 -- 172.21.15.107:0/1168492163 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129c003b70 con 0x563c69694280 2024-05-09T08:42:42.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.801+0000 7f128f7fe640 1 -- 172.21.15.107:0/1168492163 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f129c0119b0 con 0x563c69694280 2024-05-09T08:42:42.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.802+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1168492163 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c69694280 msgr2=0x563c69694660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.802+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/1168492163 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c69694280 0x563c69694660 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f129c00f390 tx=0x7f129c00f850 comp rx=0 tx=0).stop 2024-05-09T08:42:42.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.805+0000 7f12a8b69640 1 --2- 172.21.15.107:0/181999419 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f12900a2f00 0x7f12900a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.805+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1168492163 shutdown_connections 2024-05-09T08:42:42.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.805+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/1168492163 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6969f270 0x563c696a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.805+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/1168492163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c69694ba0 0x563c6969ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.805+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/1168492163 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c69694280 0x563c69694660 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.805+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1168492163 >> 172.21.15.107:0/1168492163 conn(0x563c696905a0 msgr2=0x563c69690980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:42.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.805+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1168492163 shutdown_connections 2024-05-09T08:42:42.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.805+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1168492163 wait complete. 2024-05-09T08:42:42.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.806+0000 7f12ae3ba740 1 -- start start 2024-05-09T08:42:42.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.806+0000 7f12ae3ba740 1 Processor -- start 2024-05-09T08:42:42.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.806+0000 7f12ae3ba740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c69694280 0x563c6969c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.806+0000 7f12ae3ba740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c69694ba0 0x563c6969c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.806+0000 7f12ae3ba740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c6969f270 0x563c6969cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.806+0000 7f12ae3ba740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563c696a20b0 con 0x563c69694280 2024-05-09T08:42:42.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.806+0000 7f12ae3ba740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563c695b4b90 con 0x563c69694ba0 2024-05-09T08:42:42.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.806+0000 7f12ae3ba740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563c696a2230 con 0x563c6969f270 2024-05-09T08:42:42.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.806+0000 7f12ae3ba740 10 client.0 _ll_register_callbacks cb 0x563c695f0da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:42.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.807+0000 7f12a936a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c69694280 0x563c6969c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.807+0000 7f12a936a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c69694280 0x563c6969c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37678/0 (socket says 172.21.15.107:37678) 2024-05-09T08:42:42.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.807+0000 7f12a936a640 1 -- 172.21.15.107:0/1610468988 learned_addr learned my addr 172.21.15.107:0/1610468988 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:42.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.809+0000 7f12a936a640 1 -- 172.21.15.107:0/1610468988 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c6969f270 msgr2=0x563c6969cbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:42:42.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.809+0000 7f12a936a640 1 --2- 172.21.15.107:0/1610468988 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c6969f270 0x563c6969cbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.809+0000 7f12a936a640 1 -- 172.21.15.107:0/1610468988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c69694ba0 msgr2=0x563c6969c690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:42:42.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.809+0000 7f12a936a640 1 --2- 172.21.15.107:0/1610468988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c69694ba0 0x563c6969c690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.809+0000 7f12a936a640 1 -- 172.21.15.107:0/1610468988 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c6930d3f0 con 0x563c69694280 2024-05-09T08:42:42.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.810+0000 7f12a936a640 1 --2- 172.21.15.107:0/1610468988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c69694280 0x563c6969c150 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f129c010410 tx=0x7f129c010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6045249ab5b94f27 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:42.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.810+0000 7f128cff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:42.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.811+0000 7f128cff9640 1 -- 172.21.15.107:0/1610468988 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129c011d70 con 0x563c69694280 2024-05-09T08:42:42.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.811+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1610468988 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f129c0119b0 con 0x563c69694280 2024-05-09T08:42:42.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.811+0000 7f128cff9640 1 -- 172.21.15.107:0/1610468988 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f129c011500 con 0x563c69694280 2024-05-09T08:42:42.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.811+0000 7f128cff9640 1 -- 172.21.15.107:0/1610468988 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f129c02f050 con 0x563c69694280 2024-05-09T08:42:42.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.812+0000 7f12ae3ba740 10 client.18387 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:42.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.812+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1610468988 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563c692cd0f0 con 0x563c69694280 2024-05-09T08:42:42.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.813+0000 7f128cff9640 1 -- 172.21.15.107:0/1610468988 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f129c0119b0 con 0x563c69694280 2024-05-09T08:42:42.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.813+0000 7f128cff9640 1 -- 172.21.15.107:0/1610468988 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f129c0055a0 con 0x563c69694280 2024-05-09T08:42:42.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.814+0000 7f12ae3ba740 10 client.18387 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:42.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.814+0000 7f12ae3ba740 10 client.18387 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:42.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.814+0000 7f12ae3ba740 20 client.18387 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:42.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.814+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/1610468988 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563c696af900 0x563c696b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.814+0000 7f12ae3ba740 4 client.18387 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:42.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.814+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1610468988 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c69607430 con 0x563c696af900 2024-05-09T08:42:42.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.815+0000 7f12a8b69640 1 --2- 172.21.15.107:0/1610468988 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563c696af900 0x563c696b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.816+0000 7f12a8b69640 1 --2- 172.21.15.107:0/181999419 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f12900a2f00 0x7f12900a53c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1294003820 tx=0x7f1294005fd0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:42.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.817+0000 7f12a8b69640 1 --2- 172.21.15.107:0/1610468988 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563c696af900 0x563c696b1d80 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b30a596c8cc323b4 server_cookie=ad433e12fe366968 in_seq=0 out_seq=0 2024-05-09T08:42:42.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.817+0000 7f128cff9640 10 client.18387 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:42.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.820+0000 7f128cff9640 1 -- 172.21.15.107:0/1610468988 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563c69607430 con 0x563c696af900 2024-05-09T08:42:42.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.821+0000 7f128cff9640 10 client.18387 handle_command_reply: tid=0 2024-05-09T08:42:42.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.821+0000 7f12ae3ba740 1 client.18387 shutdown 2024-05-09T08:42:42.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.821+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1610468988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c69694280 msgr2=0x563c6969c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.821+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/1610468988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c69694280 0x563c6969c150 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f129c010410 tx=0x7f129c010f00 comp rx=0 tx=0).stop 2024-05-09T08:42:42.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.821+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1610468988 shutdown_connections 2024-05-09T08:42:42.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.821+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/1610468988 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563c696af900 0x563c696b1d80 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.821+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/1610468988 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c6969f270 0x563c6969cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.821+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/1610468988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c69694ba0 0x563c6969c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.821+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/1610468988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c69694280 0x563c6969c150 secure :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f129c010410 tx=0x7f129c010f00 comp rx=0 tx=0).stop 2024-05-09T08:42:42.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.821+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1610468988 >> 172.21.15.107:0/1610468988 conn(0x563c696905a0 msgr2=0x563c6968f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:42.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.823+0000 7f128cff9640 0 client.18387 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:42.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.824+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1610468988 shutdown_connections 2024-05-09T08:42:42.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.824+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1610468988 wait complete. 2024-05-09T08:42:42.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.824+0000 7f12ae3ba740 20 client.18387 trim_cache size 0 max 16384 2024-05-09T08:42:42.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.876+0000 7f12ae3ba740 1 Processor -- start 2024-05-09T08:42:42.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.879+0000 7f12ae3ba740 1 -- start start 2024-05-09T08:42:42.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.879+0000 7f12ae3ba740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c69694ba0 0x563c6969e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.879+0000 7f12ae3ba740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c6969f270 0x563c696ab070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.879+0000 7f12a936a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c69694ba0 0x563c6969e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.879+0000 7f12a936a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c69694ba0 0x563c6969e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:37808/0 (socket says 172.21.15.107:37808) 2024-05-09T08:42:42.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.880+0000 7f12a936a640 1 -- 172.21.15.107:0/2258376388 learned_addr learned my addr 172.21.15.107:0/2258376388 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:42.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.880+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/2258376388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c6969ca20 0x563c696b7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.880+0000 7f12a8b69640 1 --2- 172.21.15.107:0/2258376388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c6969f270 0x563c696ab070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.880+0000 7f12ae3ba740 1 -- 172.21.15.107:0/2258376388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1288012930 con 0x563c6969ca20 2024-05-09T08:42:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.880+0000 7f12ae3ba740 1 -- 172.21.15.107:0/2258376388 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1288010200 con 0x563c69694ba0 2024-05-09T08:42:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.880+0000 7f12ae3ba740 1 -- 172.21.15.107:0/2258376388 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f128800e070 con 0x563c6969f270 2024-05-09T08:42:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.880+0000 7f12a9b6b640 1 --2- 172.21.15.107:0/2258376388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c6969ca20 0x563c696b7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.881+0000 7f12a9b6b640 1 -- 172.21.15.107:0/2258376388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c6969f270 msgr2=0x563c696ab070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.881+0000 7f12a9b6b640 1 --2- 172.21.15.107:0/2258376388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c6969f270 0x563c696ab070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.881+0000 7f12a9b6b640 1 -- 172.21.15.107:0/2258376388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c69694ba0 msgr2=0x563c6969e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.881+0000 7f12a9b6b640 1 --2- 172.21.15.107:0/2258376388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c69694ba0 0x563c6969e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.881+0000 7f12a9b6b640 1 -- 172.21.15.107:0/2258376388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c6969a6a0 con 0x563c6969ca20 2024-05-09T08:42:42.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.882+0000 7f12a9b6b640 1 --2- 172.21.15.107:0/2258376388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c6969ca20 0x563c696b7690 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f12a00cc520 tx=0x7f12a009b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e23a9f8764bb916 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:42.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.882+0000 7f12a936a640 1 --2- 172.21.15.107:0/2258376388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c69694ba0 0x563c6969e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:42:42.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.884+0000 7f12a8b69640 1 --2- 172.21.15.107:0/2258376388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c6969f270 0x563c696ab070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:42:42.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.886+0000 7f128effd640 1 -- 172.21.15.107:0/2258376388 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a0095e60 con 0x563c6969ca20 2024-05-09T08:42:42.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.886+0000 7f128effd640 1 -- 172.21.15.107:0/2258376388 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12a00a3f60 con 0x563c6969ca20 2024-05-09T08:42:42.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.886+0000 7f128effd640 1 -- 172.21.15.107:0/2258376388 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a00ccdc0 con 0x563c6969ca20 2024-05-09T08:42:42.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.891+0000 7f12ae3ba740 1 -- 172.21.15.107:0/2258376388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c6969ca20 msgr2=0x563c696b7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.891+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/2258376388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c6969ca20 0x563c696b7690 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f12a00cc520 tx=0x7f12a009b9f0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.894+0000 7f12ae3ba740 1 -- 172.21.15.107:0/2258376388 shutdown_connections 2024-05-09T08:42:42.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.894+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/2258376388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c6969ca20 0x563c696b7690 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.894+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/2258376388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c6969f270 0x563c696ab070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.894+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/2258376388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c69694ba0 0x563c6969e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.894+0000 7f12ae3ba740 1 -- 172.21.15.107:0/2258376388 >> 172.21.15.107:0/2258376388 conn(0x563c696905a0 msgr2=0x563c696974b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:42.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.895+0000 7f12ae3ba740 1 -- 172.21.15.107:0/2258376388 shutdown_connections 2024-05-09T08:42:42.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.895+0000 7f12ae3ba740 1 -- 172.21.15.107:0/2258376388 wait complete. 2024-05-09T08:42:42.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.895+0000 7f12ae3ba740 1 -- start start 2024-05-09T08:42:42.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.895+0000 7f12ae3ba740 1 Processor -- start 2024-05-09T08:42:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.896+0000 7f12ae3ba740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c69694ba0 0x563c696a7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.896+0000 7f12ae3ba740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c6969ca20 0x563c696a7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.896+0000 7f12ae3ba740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6969f270 0x563c69698400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.896+0000 7f12ae3ba740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f128800e6c0 con 0x563c6969ca20 2024-05-09T08:42:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.896+0000 7f12ae3ba740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f128800fb30 con 0x563c6969f270 2024-05-09T08:42:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.896+0000 7f12ae3ba740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f128800ed90 con 0x563c69694ba0 2024-05-09T08:42:42.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.896+0000 7f12ae3ba740 10 client.0 _ll_register_callbacks cb 0x7f129c015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:42.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.897+0000 7f12a936a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c69694ba0 0x563c696a7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.900+0000 7f12a8b69640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c6969ca20 0x563c696a7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.900+0000 7f12a8b69640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c6969ca20 0x563c696a7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37692/0 (socket says 172.21.15.107:37692) 2024-05-09T08:42:42.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.900+0000 7f12a8b69640 1 -- 172.21.15.107:0/1677581431 learned_addr learned my addr 172.21.15.107:0/1677581431 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:42.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.901+0000 7f12a8b69640 1 -- 172.21.15.107:0/1677581431 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c69694ba0 msgr2=0x563c696a7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:42.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.901+0000 7f12a8b69640 1 --2- 172.21.15.107:0/1677581431 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c69694ba0 0x563c696a7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.901+0000 7f12a8b69640 1 -- 172.21.15.107:0/1677581431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6969f270 msgr2=0x563c69698400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:42:42.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.901+0000 7f12a8b69640 1 --2- 172.21.15.107:0/1677581431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6969f270 0x563c69698400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.901+0000 7f12a8b69640 1 -- 172.21.15.107:0/1677581431 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c69698a10 con 0x563c6969ca20 2024-05-09T08:42:42.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.901+0000 7f12a8b69640 1 --2- 172.21.15.107:0/1677581431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c6969ca20 0x563c696a7ac0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f1294033040 tx=0x7f1294009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1686eef99324da server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:42.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.901+0000 7f128cff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:42:42.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.902+0000 7f128cff9640 1 -- 172.21.15.107:0/1677581431 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12940503d0 con 0x563c6969ca20 2024-05-09T08:42:42.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.902+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1677581431 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12a00a3f60 con 0x563c6969ca20 2024-05-09T08:42:42.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.902+0000 7f128cff9640 1 -- 172.21.15.107:0/1677581431 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f1294050570 con 0x563c6969ca20 2024-05-09T08:42:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.902+0000 7f128cff9640 1 -- 172.21.15.107:0/1677581431 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1294060050 con 0x563c6969ca20 2024-05-09T08:42:42.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.902+0000 7f12ae3ba740 10 client.18393 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.902+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1677581431 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563c697157b0 con 0x563c6969ca20 2024-05-09T08:42:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.903+0000 7f128cff9640 1 -- 172.21.15.107:0/1677581431 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12a00a3f60 con 0x563c6969ca20 2024-05-09T08:42:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.904+0000 7f128cff9640 1 -- 172.21.15.107:0/1677581431 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f1294049420 con 0x563c6969ca20 2024-05-09T08:42:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.904+0000 7f12ae3ba740 10 client.18393 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.904+0000 7f12ae3ba740 10 client.18393 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.904+0000 7f12ae3ba740 20 client.18393 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.904+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/1677581431 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563c69715ee0 0x563c69718300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:42.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.904+0000 7f12ae3ba740 4 client.18393 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:42:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.904+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1677581431 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563c69601780 con 0x563c69715ee0 2024-05-09T08:42:42.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.906+0000 7f12a936a640 1 --2- 172.21.15.107:0/1677581431 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563c69715ee0 0x563c69718300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.908+0000 7f12a936a640 1 --2- 172.21.15.107:0/1677581431 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563c69715ee0 0x563c69718300 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=242e8564c3b5a4f6 server_cookie=1113242bbb0b0a3d in_seq=0 out_seq=0 2024-05-09T08:42:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.910+0000 7f128cff9640 10 client.18393 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:42.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.912+0000 7f128cff9640 1 -- 172.21.15.107:0/1677581431 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x563c69601780 con 0x563c69715ee0 2024-05-09T08:42:42.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.912+0000 7f128cff9640 10 client.18393 handle_command_reply: tid=0 2024-05-09T08:42:42.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.912+0000 7f12ae3ba740 1 client.18393 shutdown 2024-05-09T08:42:42.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.917+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1677581431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c6969ca20 msgr2=0x563c696a7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.917+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/1677581431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c6969ca20 0x563c696a7ac0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f1294033040 tx=0x7f1294009260 comp rx=0 tx=0).stop 2024-05-09T08:42:42.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.920+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1677581431 shutdown_connections 2024-05-09T08:42:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.920+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/1677581431 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563c69715ee0 0x563c69718300 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.920+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/1677581431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6969f270 0x563c69698400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.920+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/1677581431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c6969ca20 0x563c696a7ac0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.920+0000 7f12ae3ba740 1 --2- 172.21.15.107:0/1677581431 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c69694ba0 0x563c696a7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.920+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1677581431 >> 172.21.15.107:0/1677581431 conn(0x563c696905a0 msgr2=0x563c6968ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.922+0000 7f128cff9640 0 client.18393 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:42.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.922+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1677581431 shutdown_connections 2024-05-09T08:42:42.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.922+0000 7f12ae3ba740 1 -- 172.21.15.107:0/1677581431 wait complete. 2024-05-09T08:42:42.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.924+0000 7f12ae3ba740 20 client.18393 trim_cache size 0 max 16384 2024-05-09T08:42:42.927 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:42:42.927 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "scrub active (125 inodes in the stack)", 2024-05-09T08:42:42.927 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:42:42.927 INFO:teuthology.orchestra.run.smithi107.stdout: "0b13f178-f01a-43e0-a356-fe8e69742920": { 2024-05-09T08:42:42.928 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:42:42.928 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "0b13f178-f01a-43e0-a356-fe8e69742920", 2024-05-09T08:42:42.928 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:42:42.928 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:42:42.928 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:42:42.928 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:42:42.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.930+0000 7f12ab5ce640 1 -- 172.21.15.107:0/181999419 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f12900a2f00 msgr2=0x7f12900a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.930+0000 7f12ab5ce640 1 --2- 172.21.15.107:0/181999419 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f12900a2f00 0x7f12900a53c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1294003820 tx=0x7f1294005fd0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.930+0000 7f12ab5ce640 1 -- 172.21.15.107:0/181999419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a411b010 msgr2=0x7f12a41ab720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:42.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.930+0000 7f12ab5ce640 1 --2- 172.21.15.107:0/181999419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a411b010 0x7f12a41ab720 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f12a000b450 tx=0x7f12a000b910 comp rx=0 tx=0).stop 2024-05-09T08:42:42.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.930+0000 7f12ab5ce640 1 -- 172.21.15.107:0/181999419 shutdown_connections 2024-05-09T08:42:42.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.930+0000 7f12ab5ce640 1 --2- 172.21.15.107:0/181999419 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f12900a2f00 0x7f12900a53c0 secure :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1294003820 tx=0x7f1294005fd0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.930+0000 7f12ab5ce640 1 --2- 172.21.15.107:0/181999419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a411b010 0x7f12a41ab720 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.930+0000 7f12ab5ce640 1 --2- 172.21.15.107:0/181999419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12a4110080 0x7f12a41a73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.930+0000 7f12ab5ce640 1 --2- 172.21.15.107:0/181999419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12a410e0b0 0x7f12a41a6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:42.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.930+0000 7f12ab5ce640 1 -- 172.21.15.107:0/181999419 >> 172.21.15.107:0/181999419 conn(0x7f12a406c940 msgr2=0x7f12a4072330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:42.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.930+0000 7f12ab5ce640 1 -- 172.21.15.107:0/181999419 shutdown_connections 2024-05-09T08:42:42.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:42:42.930+0000 7f12ab5ce640 1 -- 172.21.15.107:0/181999419 wait complete. 2024-05-09T08:42:42.959 INFO:tasks.cephfs.filesystem:scrub status for tag:0b13f178-f01a-43e0-a356-fe8e69742920 - {'path': '/', 'tag': '0b13f178-f01a-43e0-a356-fe8e69742920', 'options': 'recursive,force'} 2024-05-09T08:42:43.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:43 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:42:43.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:43 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:43.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:43 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:42:43.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:43 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:42:43.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:43 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:43.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:43 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:42:43.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:43 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:42:43.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:43 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:43.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:43 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:42:44.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:44 smithi107 ceph-mon[30122]: pgmap v290: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 23 MiB/s wr, 5.86k op/s 2024-05-09T08:42:44.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:44 smithi157 ceph-mon[33614]: pgmap v290: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 23 MiB/s wr, 5.86k op/s 2024-05-09T08:42:44.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:44 smithi110 ceph-mon[33622]: pgmap v290: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 23 MiB/s wr, 5.86k op/s 2024-05-09T08:42:45.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:45 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:42:45.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:45 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:45.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:45 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:42:45.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:45 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:45.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:45 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:42:45.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:45 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:45.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:45 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:42:45.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:45 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:45.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:45 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:42:45.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:45 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:45.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:45 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:42:45.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:45 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:45.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:45 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:42:45.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:45 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:45.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:45 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:42:45.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:45 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:45.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:45 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:42:45.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:45 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:45.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:45 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:42:45.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:45 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:45.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:45 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:42:45.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:45 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:45.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:45 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:42:45.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:45 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:45.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:45 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:42:45.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:45 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:45.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:45 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:42:45.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:45 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:45.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:45 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:42:45.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:45 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:46.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:46 smithi107 ceph-mon[30122]: pgmap v291: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 15 MiB/s wr, 3.83k op/s 2024-05-09T08:42:46.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:46 smithi157 ceph-mon[33614]: pgmap v291: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 15 MiB/s wr, 3.83k op/s 2024-05-09T08:42:46.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:46 smithi110 ceph-mon[33622]: pgmap v291: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 15 MiB/s wr, 3.83k op/s 2024-05-09T08:42:48.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:48 smithi107 ceph-mon[30122]: pgmap v292: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 20 MiB/s wr, 5.12k op/s 2024-05-09T08:42:48.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:48 smithi157 ceph-mon[33614]: pgmap v292: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 20 MiB/s wr, 5.12k op/s 2024-05-09T08:42:48.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:48 smithi110 ceph-mon[33622]: pgmap v292: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 20 MiB/s wr, 5.12k op/s 2024-05-09T08:42:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.340+0000 7fc583fff640 1 -- 172.21.15.107:0/2106672421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc58410fef0 msgr2=0x7fc58410d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.340+0000 7fc583fff640 1 --2- 172.21.15.107:0/2106672421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc58410fef0 0x7fc58410d9a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc574009900 tx=0x7fc57402a480 comp rx=0 tx=0).stop 2024-05-09T08:42:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.340+0000 7fc583fff640 1 -- 172.21.15.107:0/2106672421 shutdown_connections 2024-05-09T08:42:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.340+0000 7fc583fff640 1 --2- 172.21.15.107:0/2106672421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc58410e500 0x7fc58411d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.340+0000 7fc583fff640 1 --2- 172.21.15.107:0/2106672421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5841102d0 0x7fc58410dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.340+0000 7fc583fff640 1 --2- 172.21.15.107:0/2106672421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc58410fef0 0x7fc58410d9a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.340+0000 7fc583fff640 1 -- 172.21.15.107:0/2106672421 >> 172.21.15.107:0/2106672421 conn(0x7fc58406c770 msgr2=0x7fc58406cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.340+0000 7fc583fff640 1 -- 172.21.15.107:0/2106672421 shutdown_connections 2024-05-09T08:42:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.340+0000 7fc583fff640 1 -- 172.21.15.107:0/2106672421 wait complete. 2024-05-09T08:42:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.341+0000 7fc583fff640 1 Processor -- start 2024-05-09T08:42:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.341+0000 7fc583fff640 1 -- start start 2024-05-09T08:42:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.341+0000 7fc583fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc58410e500 0x7fc58411c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.341+0000 7fc583fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc58410fef0 0x7fc584115710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.341+0000 7fc583fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5841102d0 0x7fc584115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.341+0000 7fc583fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc584120800 con 0x7fc58410fef0 2024-05-09T08:42:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.341+0000 7fc583fff640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc584120680 con 0x7fc5841102d0 2024-05-09T08:42:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.341+0000 7fc583fff640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc584120980 con 0x7fc58410e500 2024-05-09T08:42:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.342+0000 7fc582ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc58410e500 0x7fc58411c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.342+0000 7fc582ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc58410e500 0x7fc58411c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47728/0 (socket says 172.21.15.107:47728) 2024-05-09T08:42:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.342+0000 7fc582ffd640 1 -- 172.21.15.107:0/2495976699 learned_addr learned my addr 172.21.15.107:0/2495976699 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.342+0000 7fc5837fe640 1 --2- 172.21.15.107:0/2495976699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5841102d0 0x7fc584115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.342+0000 7fc5827fc640 1 --2- 172.21.15.107:0/2495976699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc58410fef0 0x7fc584115710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.342+0000 7fc582ffd640 1 -- 172.21.15.107:0/2495976699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5841102d0 msgr2=0x7fc584115c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.342+0000 7fc582ffd640 1 --2- 172.21.15.107:0/2495976699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5841102d0 0x7fc584115c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.342+0000 7fc582ffd640 1 -- 172.21.15.107:0/2495976699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc58410fef0 msgr2=0x7fc584115710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.342+0000 7fc582ffd640 1 --2- 172.21.15.107:0/2495976699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc58410fef0 0x7fc584115710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.342+0000 7fc582ffd640 1 -- 172.21.15.107:0/2495976699 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc584116210 con 0x7fc58410e500 2024-05-09T08:42:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.343+0000 7fc582ffd640 1 --2- 172.21.15.107:0/2495976699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc58410e500 0x7fc58411c690 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc57402aa70 tx=0x7fc574030940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.343+0000 7fc56bfff640 1 -- 172.21.15.107:0/2495976699 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574040070 con 0x7fc58410e500 2024-05-09T08:42:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.343+0000 7fc56bfff640 1 -- 172.21.15.107:0/2495976699 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5740035f0 con 0x7fc58410e500 2024-05-09T08:42:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.343+0000 7fc56bfff640 1 -- 172.21.15.107:0/2495976699 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574035030 con 0x7fc58410e500 2024-05-09T08:42:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.343+0000 7fc583fff640 1 -- 172.21.15.107:0/2495976699 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5841164a0 con 0x7fc58410e500 2024-05-09T08:42:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.343+0000 7fc583fff640 1 -- 172.21.15.107:0/2495976699 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc58411a3f0 con 0x7fc58410e500 2024-05-09T08:42:49.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.348+0000 7fc56bfff640 1 -- 172.21.15.107:0/2495976699 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fc5740037a0 con 0x7fc58410e500 2024-05-09T08:42:49.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.349+0000 7fc56bfff640 1 --2- 172.21.15.107:0/2495976699 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc56c0a2f00 0x7fc56c0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.350+0000 7fc56bfff640 1 -- 172.21.15.107:0/2495976699 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fc5740f02b0 con 0x7fc58410e500 2024-05-09T08:42:49.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.350+0000 7fc5827fc640 1 --2- 172.21.15.107:0/2495976699 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc56c0a2f00 0x7fc56c0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.356+0000 7fc5827fc640 1 --2- 172.21.15.107:0/2495976699 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc56c0a2f00 0x7fc56c0a53c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc584116ed0 tx=0x7fc57c0094f0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.358+0000 7fc58c01b740 1 Processor -- start 2024-05-09T08:42:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.360+0000 7fc58c01b740 1 -- start start 2024-05-09T08:42:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.360+0000 7fc58c01b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 0x563e18b419b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.360+0000 7fc58c01b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b41ef0 0x563e18b4c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.360+0000 7fc58c01b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b4c7b0 0x563e18b4eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.360+0000 7fc58c01b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563e18aa6340 con 0x563e18b41ef0 2024-05-09T08:42:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.360+0000 7fc58c01b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563e18a4e290 con 0x563e18b4c7b0 2024-05-09T08:42:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.360+0000 7fc58c01b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563e18aa2300 con 0x563e18b415d0 2024-05-09T08:42:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.361+0000 7fc582ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 0x563e18b419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.361+0000 7fc582ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 0x563e18b419b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47738/0 (socket says 172.21.15.107:47738) 2024-05-09T08:42:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.361+0000 7fc582ffd640 1 -- 172.21.15.107:0/3610743241 learned_addr learned my addr 172.21.15.107:0/3610743241 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:49.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.363+0000 7fc5837fe640 1 --2- 172.21.15.107:0/3610743241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b4c7b0 0x563e18b4eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.364+0000 7fc5827fc640 1 --2- 172.21.15.107:0/3610743241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b41ef0 0x563e18b4c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.367+0000 7fc5827fc640 1 -- 172.21.15.107:0/3610743241 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 msgr2=0x563e18b419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.367+0000 7fc5827fc640 1 --2- 172.21.15.107:0/3610743241 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 0x563e18b419b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.367+0000 7fc5827fc640 1 -- 172.21.15.107:0/3610743241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b4c7b0 msgr2=0x563e18b4eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.367+0000 7fc5827fc640 1 --2- 172.21.15.107:0/3610743241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b4c7b0 0x563e18b4eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.367+0000 7fc5827fc640 1 -- 172.21.15.107:0/3610743241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e18a728e0 con 0x563e18b41ef0 2024-05-09T08:42:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.367+0000 7fc5827fc640 1 --2- 172.21.15.107:0/3610743241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b41ef0 0x563e18b4c270 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fc57c017730 tx=0x7fc57c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74512da729d566d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.370+0000 7fc5697fa640 1 -- 172.21.15.107:0/3610743241 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57c027070 con 0x563e18b41ef0 2024-05-09T08:42:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.370+0000 7fc5697fa640 1 -- 172.21.15.107:0/3610743241 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc57c017de0 con 0x563e18b41ef0 2024-05-09T08:42:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.371+0000 7fc58c01b740 1 -- 172.21.15.107:0/3610743241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b41ef0 msgr2=0x563e18b4c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.371+0000 7fc58c01b740 1 --2- 172.21.15.107:0/3610743241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b41ef0 0x563e18b4c270 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fc57c017730 tx=0x7fc57c011000 comp rx=0 tx=0).stop 2024-05-09T08:42:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.373+0000 7fc58c01b740 1 -- 172.21.15.107:0/3610743241 shutdown_connections 2024-05-09T08:42:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.373+0000 7fc58c01b740 1 --2- 172.21.15.107:0/3610743241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b4c7b0 0x563e18b4eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.373+0000 7fc58c01b740 1 --2- 172.21.15.107:0/3610743241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b41ef0 0x563e18b4c270 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.373+0000 7fc58c01b740 1 --2- 172.21.15.107:0/3610743241 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 0x563e18b419b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.373+0000 7fc58c01b740 1 -- 172.21.15.107:0/3610743241 >> 172.21.15.107:0/3610743241 conn(0x563e18b3dae0 msgr2=0x563e18b3dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.373+0000 7fc58c01b740 1 -- 172.21.15.107:0/3610743241 shutdown_connections 2024-05-09T08:42:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.374+0000 7fc58c01b740 1 -- 172.21.15.107:0/3610743241 wait complete. 2024-05-09T08:42:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.374+0000 7fc58c01b740 1 -- start start 2024-05-09T08:42:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.374+0000 7fc58c01b740 1 Processor -- start 2024-05-09T08:42:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.375+0000 7fc58c01b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b415d0 0x563e18b498d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.375+0000 7fc58c01b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b41ef0 0x563e18b49e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.375+0000 7fc58c01b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b4c7b0 0x563e18b4a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.375+0000 7fc58c01b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563e18b4f3e0 con 0x563e18b41ef0 2024-05-09T08:42:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.375+0000 7fc58c01b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563e18a61790 con 0x563e18b415d0 2024-05-09T08:42:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.375+0000 7fc58c01b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563e18b4f560 con 0x563e18b4c7b0 2024-05-09T08:42:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.375+0000 7fc58c01b740 10 client.0 _ll_register_callbacks cb 0x563e18a9ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:49.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.375+0000 7fc582ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b415d0 0x563e18b498d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:49.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.375+0000 7fc582ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b415d0 0x563e18b498d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:37868/0 (socket says 172.21.15.107:37868) 2024-05-09T08:42:49.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.375+0000 7fc582ffd640 1 -- 172.21.15.107:0/1481991899 learned_addr learned my addr 172.21.15.107:0/1481991899 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:49.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.375+0000 7fc5837fe640 1 --2- 172.21.15.107:0/1481991899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b4c7b0 0x563e18b4a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:49.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.376+0000 7fc5837fe640 1 -- 172.21.15.107:0/1481991899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b415d0 msgr2=0x563e18b498d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:49.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.376+0000 7fc5837fe640 1 --2- 172.21.15.107:0/1481991899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b415d0 0x563e18b498d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.376+0000 7fc5837fe640 1 -- 172.21.15.107:0/1481991899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b41ef0 msgr2=0x563e18b49e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.376+0000 7fc5837fe640 1 --2- 172.21.15.107:0/1481991899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b41ef0 0x563e18b49e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.376+0000 7fc5837fe640 1 -- 172.21.15.107:0/1481991899 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e187ba210 con 0x563e18b4c7b0 2024-05-09T08:42:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.376+0000 7fc5837fe640 1 --2- 172.21.15.107:0/1481991899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b4c7b0 0x563e18b4a350 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x563e18b3ef00 tx=0x7fc57800bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88dcb60980f5d32d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.378+0000 7fc566ff5640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:42:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.378+0000 7fc566ff5640 1 -- 172.21.15.107:0/1481991899 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57801b440 con 0x563e18b4c7b0 2024-05-09T08:42:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.378+0000 7fc566ff5640 1 -- 172.21.15.107:0/1481991899 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fc57801b5e0 con 0x563e18b4c7b0 2024-05-09T08:42:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.378+0000 7fc58c01b740 1 -- 172.21.15.107:0/1481991899 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563e18ab3a70 con 0x563e18b4c7b0 2024-05-09T08:42:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.379+0000 7fc566ff5640 1 -- 172.21.15.107:0/1481991899 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc57801b7e0 con 0x563e18b4c7b0 2024-05-09T08:42:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.380+0000 7fc58c01b740 10 client.27598 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.380+0000 7fc58c01b740 1 -- 172.21.15.107:0/1481991899 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563e18886620 con 0x563e18b4c7b0 2024-05-09T08:42:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.381+0000 7fc566ff5640 1 -- 172.21.15.107:0/1481991899 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x563e18ab3a70 con 0x563e18b4c7b0 2024-05-09T08:42:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.382+0000 7fc566ff5640 1 -- 172.21.15.107:0/1481991899 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc578019020 con 0x563e18b4c7b0 2024-05-09T08:42:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.383+0000 7fc58c01b740 10 client.27598 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.383+0000 7fc58c01b740 10 client.27598 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.383+0000 7fc58c01b740 20 client.27598 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.383+0000 7fc58c01b740 1 --2- 172.21.15.107:0/1481991899 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563e18b5cc50 0x563e18b5f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.383+0000 7fc58c01b740 4 client.27598 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:42:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.383+0000 7fc58c01b740 1 -- 172.21.15.107:0/1481991899 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e18aaddc0 con 0x563e18b5cc50 2024-05-09T08:42:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.384+0000 7fc582ffd640 1 --2- 172.21.15.107:0/1481991899 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563e18b5cc50 0x563e18b5f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.385+0000 7fc582ffd640 1 --2- 172.21.15.107:0/1481991899 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563e18b5cc50 0x563e18b5f0d0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2db8ad33b8041d95 server_cookie=fde214d194055143 in_seq=0 out_seq=0 2024-05-09T08:42:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.385+0000 7fc566ff5640 10 client.27598 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:49.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.387+0000 7fc566ff5640 1 -- 172.21.15.107:0/1481991899 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563e18aaddc0 con 0x563e18b5cc50 2024-05-09T08:42:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.387+0000 7fc566ff5640 10 client.27598 handle_command_reply: tid=0 2024-05-09T08:42:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.388+0000 7fc58c01b740 1 client.27598 shutdown 2024-05-09T08:42:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.388+0000 7fc58c01b740 1 -- 172.21.15.107:0/1481991899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b4c7b0 msgr2=0x563e18b4a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:49.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.388+0000 7fc58c01b740 1 --2- 172.21.15.107:0/1481991899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b4c7b0 0x563e18b4a350 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x563e18b3ef00 tx=0x7fc57800bd80 comp rx=0 tx=0).stop 2024-05-09T08:42:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.390+0000 7fc58c01b740 1 -- 172.21.15.107:0/1481991899 shutdown_connections 2024-05-09T08:42:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.390+0000 7fc58c01b740 1 --2- 172.21.15.107:0/1481991899 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563e18b5cc50 0x563e18b5f0d0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.390+0000 7fc58c01b740 1 --2- 172.21.15.107:0/1481991899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b4c7b0 0x563e18b4a350 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.390+0000 7fc58c01b740 1 --2- 172.21.15.107:0/1481991899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b41ef0 0x563e18b49e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.390+0000 7fc58c01b740 1 --2- 172.21.15.107:0/1481991899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b415d0 0x563e18b498d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.390+0000 7fc58c01b740 1 -- 172.21.15.107:0/1481991899 >> 172.21.15.107:0/1481991899 conn(0x563e18b3dae0 msgr2=0x563e18b43890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.391+0000 7fc566ff5640 0 client.27598 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.392+0000 7fc58c01b740 1 -- 172.21.15.107:0/1481991899 shutdown_connections 2024-05-09T08:42:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.392+0000 7fc58c01b740 1 -- 172.21.15.107:0/1481991899 wait complete. 2024-05-09T08:42:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.396+0000 7fc58c01b740 20 client.27598 trim_cache size 0 max 16384 2024-05-09T08:42:49.434 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.435+0000 7fc58c01b740 1 Processor -- start 2024-05-09T08:42:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.436+0000 7fc58c01b740 1 -- start start 2024-05-09T08:42:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.437+0000 7fc58c01b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 0x563e18b4c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.437+0000 7fc58c01b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b41ef0 0x563e18b3d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.437+0000 7fc58c01b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b4c7b0 0x563e18b44c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.437+0000 7fc58c01b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc5841c88a0 con 0x563e18b41ef0 2024-05-09T08:42:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.437+0000 7fc58c01b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc5841c6170 con 0x563e18b4c7b0 2024-05-09T08:42:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.437+0000 7fc58c01b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc5841c4070 con 0x563e18b415d0 2024-05-09T08:42:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.438+0000 7fc5837fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b4c7b0 0x563e18b44c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.438+0000 7fc5837fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b4c7b0 0x563e18b44c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:37874/0 (socket says 172.21.15.107:37874) 2024-05-09T08:42:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.438+0000 7fc5837fe640 1 -- 172.21.15.107:0/1194639854 learned_addr learned my addr 172.21.15.107:0/1194639854 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.438+0000 7fc582ffd640 1 --2- 172.21.15.107:0/1194639854 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 0x563e18b4c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.439+0000 7fc5827fc640 1 --2- 172.21.15.107:0/1194639854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b41ef0 0x563e18b3d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.439+0000 7fc582ffd640 1 -- 172.21.15.107:0/1194639854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b4c7b0 msgr2=0x563e18b44c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.439+0000 7fc582ffd640 1 --2- 172.21.15.107:0/1194639854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b4c7b0 0x563e18b44c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.439+0000 7fc582ffd640 1 -- 172.21.15.107:0/1194639854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b41ef0 msgr2=0x563e18b3d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.439+0000 7fc582ffd640 1 --2- 172.21.15.107:0/1194639854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b41ef0 0x563e18b3d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.439+0000 7fc582ffd640 1 -- 172.21.15.107:0/1194639854 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e18b5b400 con 0x563e18b415d0 2024-05-09T08:42:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.439+0000 7fc582ffd640 1 --2- 172.21.15.107:0/1194639854 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 0x563e18b4c000 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc574009a20 tx=0x7fc5740dc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d5316e95ec07881 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.440+0000 7fc568ff9640 1 -- 172.21.15.107:0/1194639854 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5740cf070 con 0x563e18b415d0 2024-05-09T08:42:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.440+0000 7fc568ff9640 1 -- 172.21.15.107:0/1194639854 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5740bcac0 con 0x563e18b415d0 2024-05-09T08:42:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.440+0000 7fc568ff9640 1 -- 172.21.15.107:0/1194639854 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574003b30 con 0x563e18b415d0 2024-05-09T08:42:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.442+0000 7fc58c01b740 1 -- 172.21.15.107:0/1194639854 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 msgr2=0x563e18b4c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.442+0000 7fc58c01b740 1 --2- 172.21.15.107:0/1194639854 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 0x563e18b4c000 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc574009a20 tx=0x7fc5740dc040 comp rx=0 tx=0).stop 2024-05-09T08:42:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.443+0000 7fc58c01b740 1 -- 172.21.15.107:0/1194639854 shutdown_connections 2024-05-09T08:42:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.443+0000 7fc58c01b740 1 --2- 172.21.15.107:0/1194639854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b4c7b0 0x563e18b44c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.443+0000 7fc58c01b740 1 --2- 172.21.15.107:0/1194639854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b41ef0 0x563e18b3d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.443+0000 7fc58c01b740 1 --2- 172.21.15.107:0/1194639854 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 0x563e18b4c000 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.443+0000 7fc58c01b740 1 -- 172.21.15.107:0/1194639854 >> 172.21.15.107:0/1194639854 conn(0x563e18b3dae0 msgr2=0x563e18b4a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.443+0000 7fc58c01b740 1 -- 172.21.15.107:0/1194639854 shutdown_connections 2024-05-09T08:42:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.443+0000 7fc58c01b740 1 -- 172.21.15.107:0/1194639854 wait complete. 2024-05-09T08:42:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.444+0000 7fc58c01b740 1 -- start start 2024-05-09T08:42:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.444+0000 7fc58c01b740 1 Processor -- start 2024-05-09T08:42:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.444+0000 7fc58c01b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 0x563e18b54800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.444+0000 7fc58c01b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b41ef0 0x563e18b54d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.444+0000 7fc58c01b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b4c7b0 0x563e18b45b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.444+0000 7fc58c01b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc5841c4630 con 0x563e18b4c7b0 2024-05-09T08:42:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.444+0000 7fc58c01b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc5841c5aa0 con 0x563e18b41ef0 2024-05-09T08:42:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.444+0000 7fc58c01b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc5841c4d00 con 0x563e18b415d0 2024-05-09T08:42:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.444+0000 7fc58c01b740 10 client.0 _ll_register_callbacks cb 0x7fc5780041c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:42:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.445+0000 7fc5827fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b41ef0 0x563e18b54d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.445+0000 7fc5827fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b41ef0 0x563e18b54d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:37876/0 (socket says 172.21.15.107:37876) 2024-05-09T08:42:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.445+0000 7fc5827fc640 1 -- 172.21.15.107:0/1119813409 learned_addr learned my addr 172.21.15.107:0/1119813409 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:42:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.445+0000 7fc582ffd640 1 --2- 172.21.15.107:0/1119813409 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 0x563e18b54800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.445+0000 7fc5827fc640 1 -- 172.21.15.107:0/1119813409 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 msgr2=0x563e18b54800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.445+0000 7fc5827fc640 1 --2- 172.21.15.107:0/1119813409 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 0x563e18b54800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.445+0000 7fc5827fc640 1 -- 172.21.15.107:0/1119813409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b4c7b0 msgr2=0x563e18b45b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:42:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.445+0000 7fc5827fc640 1 --2- 172.21.15.107:0/1119813409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b4c7b0 0x563e18b45b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.445+0000 7fc5827fc640 1 -- 172.21.15.107:0/1119813409 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e18b46100 con 0x563e18b41ef0 2024-05-09T08:42:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.449+0000 7fc5827fc640 1 --2- 172.21.15.107:0/1119813409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b41ef0 0x563e18b54d40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc57c017730 tx=0x7fc57c016c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7d631513f2ac66a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:42:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.449+0000 7fc566ff5640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:42:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.450+0000 7fc566ff5640 1 -- 172.21.15.107:0/1119813409 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57c032070 con 0x563e18b41ef0 2024-05-09T08:42:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.450+0000 7fc566ff5640 1 -- 172.21.15.107:0/1119813409 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fc57c017de0 con 0x563e18b41ef0 2024-05-09T08:42:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.450+0000 7fc58c01b740 1 -- 172.21.15.107:0/1119813409 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5740bcac0 con 0x563e18b41ef0 2024-05-09T08:42:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.450+0000 7fc566ff5640 1 -- 172.21.15.107:0/1119813409 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc57c027040 con 0x563e18b41ef0 2024-05-09T08:42:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.450+0000 7fc58c01b740 10 client.27335 fetch_fsmap learned FSMap version 27 2024-05-09T08:42:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.450+0000 7fc58c01b740 1 -- 172.21.15.107:0/1119813409 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563e18bc2ae0 con 0x563e18b41ef0 2024-05-09T08:42:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.451+0000 7fc566ff5640 1 -- 172.21.15.107:0/1119813409 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5740bcac0 con 0x563e18b41ef0 2024-05-09T08:42:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.451+0000 7fc566ff5640 1 -- 172.21.15.107:0/1119813409 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc57c02eaa0 con 0x563e18b41ef0 2024-05-09T08:42:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.452+0000 7fc58c01b740 10 client.27335 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:42:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.452+0000 7fc58c01b740 10 client.27335 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:42:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.452+0000 7fc58c01b740 20 client.27335 populate_metadata read hostname 'smithi107' 2024-05-09T08:42:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.452+0000 7fc58c01b740 1 --2- 172.21.15.107:0/1119813409 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563e18bc2eb0 0x563e18bc52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:42:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.452+0000 7fc58c01b740 4 client.27335 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:42:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.452+0000 7fc58c01b740 1 -- 172.21.15.107:0/1119813409 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563e18a2b8e0 con 0x563e18bc2eb0 2024-05-09T08:42:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.454+0000 7fc582ffd640 1 --2- 172.21.15.107:0/1119813409 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563e18bc2eb0 0x563e18bc52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:42:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.455+0000 7fc582ffd640 1 --2- 172.21.15.107:0/1119813409 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563e18bc2eb0 0x563e18bc52a0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd09c7b34607b58c server_cookie=1086d45dc5a5cfc7 in_seq=0 out_seq=0 2024-05-09T08:42:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.455+0000 7fc566ff5640 10 client.27335 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:49.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:49 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.555+0000 7fc566ff5640 1 -- 172.21.15.107:0/1119813409 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563e18a2b8e0 con 0x563e18bc2eb0 2024-05-09T08:42:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.555+0000 7fc566ff5640 10 client.27335 handle_command_reply: tid=0 2024-05-09T08:42:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.555+0000 7fc58c01b740 1 client.27335 shutdown 2024-05-09T08:42:49.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.562+0000 7fc58c01b740 1 -- 172.21.15.107:0/1119813409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b41ef0 msgr2=0x563e18b54d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:49.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.562+0000 7fc58c01b740 1 --2- 172.21.15.107:0/1119813409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b41ef0 0x563e18b54d40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc57c017730 tx=0x7fc57c016c60 comp rx=0 tx=0).stop 2024-05-09T08:42:49.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.562+0000 7fc58c01b740 1 -- 172.21.15.107:0/1119813409 shutdown_connections 2024-05-09T08:42:49.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.562+0000 7fc58c01b740 1 --2- 172.21.15.107:0/1119813409 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563e18bc2eb0 0x563e18bc52a0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.562+0000 7fc58c01b740 1 --2- 172.21.15.107:0/1119813409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563e18b4c7b0 0x563e18b45b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.562+0000 7fc58c01b740 1 --2- 172.21.15.107:0/1119813409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563e18b41ef0 0x563e18b54d40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.562+0000 7fc58c01b740 1 --2- 172.21.15.107:0/1119813409 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563e18b415d0 0x563e18b54800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.562+0000 7fc58c01b740 1 -- 172.21.15.107:0/1119813409 >> 172.21.15.107:0/1119813409 conn(0x563e18b3dae0 msgr2=0x563e18b3c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:49.565 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.562+0000 7fc566ff5640 0 client.27335 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:42:49.566 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.567+0000 7fc58c01b740 1 -- 172.21.15.107:0/1119813409 shutdown_connections 2024-05-09T08:42:49.571 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:42:49.571 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:42:49.571 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:42:49.571 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:42:49.572 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.568+0000 7fc58c01b740 1 -- 172.21.15.107:0/1119813409 wait complete. 2024-05-09T08:42:49.572 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.568+0000 7fc58c01b740 20 client.27335 trim_cache size 0 max 16384 2024-05-09T08:42:49.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.578+0000 7fc583fff640 1 -- 172.21.15.107:0/2495976699 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc56c0a2f00 msgr2=0x7fc56c0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:49.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.578+0000 7fc583fff640 1 --2- 172.21.15.107:0/2495976699 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc56c0a2f00 0x7fc56c0a53c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc584116ed0 tx=0x7fc57c0094f0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.578+0000 7fc583fff640 1 -- 172.21.15.107:0/2495976699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc58410e500 msgr2=0x7fc58411c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:42:49.577 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.578+0000 7fc583fff640 1 --2- 172.21.15.107:0/2495976699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc58410e500 0x7fc58411c690 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc57402aa70 tx=0x7fc574030940 comp rx=0 tx=0).stop 2024-05-09T08:42:49.588 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.586+0000 7fc583fff640 1 -- 172.21.15.107:0/2495976699 shutdown_connections 2024-05-09T08:42:49.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.586+0000 7fc583fff640 1 --2- 172.21.15.107:0/2495976699 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc56c0a2f00 0x7fc56c0a53c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.586+0000 7fc583fff640 1 --2- 172.21.15.107:0/2495976699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5841102d0 0x7fc584115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.586+0000 7fc583fff640 1 --2- 172.21.15.107:0/2495976699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc58410fef0 0x7fc584115710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.586+0000 7fc583fff640 1 --2- 172.21.15.107:0/2495976699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc58410e500 0x7fc58411c690 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:42:49.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.586+0000 7fc583fff640 1 -- 172.21.15.107:0/2495976699 >> 172.21.15.107:0/2495976699 conn(0x7fc58406c770 msgr2=0x7fc584070c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:42:49.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.588+0000 7fc583fff640 1 -- 172.21.15.107:0/2495976699 shutdown_connections 2024-05-09T08:42:49.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:42:49.588+0000 7fc583fff640 1 -- 172.21.15.107:0/2495976699 wait complete. 2024-05-09T08:42:49.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:49 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:49.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:49 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:42:50.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:50 smithi107 ceph-mon[30122]: pgmap v293: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 16 MiB/s wr, 4.15k op/s 2024-05-09T08:42:50.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:50 smithi157 ceph-mon[33614]: pgmap v293: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 16 MiB/s wr, 4.15k op/s 2024-05-09T08:42:50.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:50 smithi110 ceph-mon[33622]: pgmap v293: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 16 MiB/s wr, 4.15k op/s 2024-05-09T08:42:51.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:51 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:42:51.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:51 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:42:51.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:51 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:42:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:52 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:42:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:52 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:42:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:52 smithi107 ceph-mon[30122]: pgmap v294: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 15 MiB/s wr, 3.80k op/s 2024-05-09T08:42:52.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:52 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:42:52.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:52 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:42:52.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:52 smithi157 ceph-mon[33614]: pgmap v294: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 15 MiB/s wr, 3.80k op/s 2024-05-09T08:42:52.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:52 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:42:52.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:52 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:42:52.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:52 smithi110 ceph-mon[33622]: pgmap v294: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 15 MiB/s wr, 3.80k op/s 2024-05-09T08:42:54.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:54 smithi107 ceph-mon[30122]: pgmap v295: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 22 MiB/s wr, 5.70k op/s 2024-05-09T08:42:54.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:54 smithi157 ceph-mon[33614]: pgmap v295: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 22 MiB/s wr, 5.70k op/s 2024-05-09T08:42:54.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:54 smithi110 ceph-mon[33622]: pgmap v295: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 22 MiB/s wr, 5.70k op/s 2024-05-09T08:42:56.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:56 smithi107 ceph-mon[30122]: pgmap v296: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 15 MiB/s wr, 3.89k op/s 2024-05-09T08:42:56.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:56 smithi157 ceph-mon[33614]: pgmap v296: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 15 MiB/s wr, 3.89k op/s 2024-05-09T08:42:56.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:56 smithi110 ceph-mon[33622]: pgmap v296: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 15 MiB/s wr, 3.89k op/s 2024-05-09T08:42:58.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:42:58 smithi107 ceph-mon[30122]: pgmap v297: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 20 MiB/s wr, 5.20k op/s 2024-05-09T08:42:58.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:42:58 smithi157 ceph-mon[33614]: pgmap v297: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 20 MiB/s wr, 5.20k op/s 2024-05-09T08:42:58.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:42:58 smithi110 ceph-mon[33622]: pgmap v297: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 20 MiB/s wr, 5.20k op/s 2024-05-09T08:43:00.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:00 smithi107 ceph-mon[30122]: pgmap v298: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 18 MiB/s wr, 4.54k op/s 2024-05-09T08:43:00.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:00 smithi157 ceph-mon[33614]: pgmap v298: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 18 MiB/s wr, 4.54k op/s 2024-05-09T08:43:00.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:00 smithi110 ceph-mon[33622]: pgmap v298: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 18 MiB/s wr, 4.54k op/s 2024-05-09T08:43:02.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:02 smithi157 ceph-mon[33614]: pgmap v299: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 15 MiB/s wr, 3.83k op/s 2024-05-09T08:43:02.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:02 smithi110 ceph-mon[33622]: pgmap v299: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 15 MiB/s wr, 3.83k op/s 2024-05-09T08:43:02.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.776+0000 7f9318b4f640 1 -- 172.21.15.107:0/3294354111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f931410e5a0 msgr2=0x7f9314108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:02.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.776+0000 7f9318b4f640 1 --2- 172.21.15.107:0/3294354111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f931410e5a0 0x7f9314108830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f930c009ca0 tx=0x7f930c02a8d0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:02 smithi107 ceph-mon[30122]: pgmap v299: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 15 MiB/s wr, 3.83k op/s 2024-05-09T08:43:02.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.780+0000 7f9318b4f640 1 -- 172.21.15.107:0/3294354111 shutdown_connections 2024-05-09T08:43:02.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.780+0000 7f9318b4f640 1 --2- 172.21.15.107:0/3294354111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f931410e5a0 0x7f9314108830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.780+0000 7f9318b4f640 1 --2- 172.21.15.107:0/3294354111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f931410db00 0x7f931410df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.780+0000 7f9318b4f640 1 --2- 172.21.15.107:0/3294354111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f931410fda0 0x7f9314110180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.780+0000 7f9318b4f640 1 -- 172.21.15.107:0/3294354111 >> 172.21.15.107:0/3294354111 conn(0x7f931406c7e0 msgr2=0x7f931406cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:02.782 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.782+0000 7f9318b4f640 1 -- 172.21.15.107:0/3294354111 shutdown_connections 2024-05-09T08:43:02.784 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.783+0000 7f9318b4f640 1 -- 172.21.15.107:0/3294354111 wait complete. 2024-05-09T08:43:02.784 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.784+0000 7f9318b4f640 1 Processor -- start 2024-05-09T08:43:02.784 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.784+0000 7f9318b4f640 1 -- start start 2024-05-09T08:43:02.784 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.784+0000 7f9318b4f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f931410db00 0x7f93141ac800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.784+0000 7f9318b4f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f931410e5a0 0x7f93141acd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.784+0000 7f9318b4f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f931410fda0 0x7f93141a6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.784+0000 7f9318b4f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9314120760 con 0x7f931410e5a0 2024-05-09T08:43:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.784+0000 7f9318b4f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f93141205e0 con 0x7f931410fda0 2024-05-09T08:43:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.784+0000 7f9318b4f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f93141208e0 con 0x7f931410db00 2024-05-09T08:43:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.785+0000 7f9312d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f931410fda0 0x7f93141a6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.785+0000 7f9312d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f931410fda0 0x7f93141a6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:42804/0 (socket says 172.21.15.107:42804) 2024-05-09T08:43:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.785+0000 7f9312d9d640 1 -- 172.21.15.107:0/3005238851 learned_addr learned my addr 172.21.15.107:0/3005238851 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:02.786 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.786+0000 7f9312d9d640 1 -- 172.21.15.107:0/3005238851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f931410db00 msgr2=0x7f93141ac800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:43:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.786+0000 7f9312d9d640 1 --2- 172.21.15.107:0/3005238851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f931410db00 0x7f93141ac800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.786+0000 7f9312d9d640 1 -- 172.21.15.107:0/3005238851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f931410e5a0 msgr2=0x7f93141acd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:43:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.786+0000 7f9312d9d640 1 --2- 172.21.15.107:0/3005238851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f931410e5a0 0x7f93141acd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.786+0000 7f9312d9d640 1 -- 172.21.15.107:0/3005238851 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93141a7150 con 0x7f931410fda0 2024-05-09T08:43:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.788+0000 7f9312d9d640 1 --2- 172.21.15.107:0/3005238851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f931410fda0 0x7f93141a6a00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f930c003ba0 tx=0x7f930c003bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:02.789 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.789+0000 7f93037fe640 1 -- 172.21.15.107:0/3005238851 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c004040 con 0x7f931410fda0 2024-05-09T08:43:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.790+0000 7f9318b4f640 1 -- 172.21.15.107:0/3005238851 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93141a73e0 con 0x7f931410fda0 2024-05-09T08:43:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.790+0000 7f9318b4f640 1 -- 172.21.15.107:0/3005238851 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93141107f0 con 0x7f931410fda0 2024-05-09T08:43:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.790+0000 7f93037fe640 1 -- 172.21.15.107:0/3005238851 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f930c009800 con 0x7f931410fda0 2024-05-09T08:43:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.790+0000 7f93037fe640 1 -- 172.21.15.107:0/3005238851 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c030b30 con 0x7f931410fda0 2024-05-09T08:43:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.794+0000 7f931b857740 1 Processor -- start 2024-05-09T08:43:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.798+0000 7f931b857740 1 -- start start 2024-05-09T08:43:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.803+0000 7f93037fe640 1 -- 172.21.15.107:0/3005238851 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f930c0041e0 con 0x7f931410fda0 2024-05-09T08:43:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.804+0000 7f931b857740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c304b600 0x5579c304b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.804+0000 7f931b857740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c304bf20 0x5579c30562a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.804+0000 7f931b857740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c30567e0 0x5579c3058bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.804+0000 7f931b857740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5579c2fb0340 con 0x5579c30567e0 2024-05-09T08:43:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.804+0000 7f931b857740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5579c2f58290 con 0x5579c304bf20 2024-05-09T08:43:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.804+0000 7f931b857740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5579c2fac300 con 0x5579c304b600 2024-05-09T08:43:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.805+0000 7f931259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c304bf20 0x5579c30562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.805+0000 7f931259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c304bf20 0x5579c30562a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:42814/0 (socket says 172.21.15.107:42814) 2024-05-09T08:43:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.805+0000 7f931259c640 1 -- 172.21.15.107:0/2899279288 learned_addr learned my addr 172.21.15.107:0/2899279288 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.805+0000 7f931259c640 1 -- 172.21.15.107:0/2899279288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c304b600 msgr2=0x5579c304b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.805+0000 7f931259c640 1 --2- 172.21.15.107:0/2899279288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c304b600 0x5579c304b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.805+0000 7f931259c640 1 -- 172.21.15.107:0/2899279288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c30567e0 msgr2=0x5579c3058bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.805+0000 7f931259c640 1 --2- 172.21.15.107:0/2899279288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c30567e0 0x5579c3058bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.805+0000 7f931259c640 1 -- 172.21.15.107:0/2899279288 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579c2f7c8e0 con 0x5579c304bf20 2024-05-09T08:43:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.805+0000 7f931259c640 1 --2- 172.21.15.107:0/2899279288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c304bf20 0x5579c30562a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f930800a9f0 tx=0x7f930800aeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a1b05483f84b16c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.807+0000 7f9300ff9640 1 -- 172.21.15.107:0/2899279288 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930800cde0 con 0x5579c304bf20 2024-05-09T08:43:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.807+0000 7f9300ff9640 1 -- 172.21.15.107:0/2899279288 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f930800f900 con 0x5579c304bf20 2024-05-09T08:43:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.807+0000 7f9300ff9640 1 -- 172.21.15.107:0/2899279288 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930800e980 con 0x5579c304bf20 2024-05-09T08:43:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.808+0000 7f93037fe640 1 --2- 172.21.15.107:0/3005238851 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f92fc0a2e40 0x7f92fc0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.808+0000 7f93037fe640 1 -- 172.21.15.107:0/3005238851 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f930c0f0fa0 con 0x7f931410fda0 2024-05-09T08:43:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.809+0000 7f931b857740 1 -- 172.21.15.107:0/2899279288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c304bf20 msgr2=0x5579c30562a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.809+0000 7f931b857740 1 --2- 172.21.15.107:0/2899279288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c304bf20 0x5579c30562a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f930800a9f0 tx=0x7f930800aeb0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.811+0000 7f931b857740 1 -- 172.21.15.107:0/2899279288 shutdown_connections 2024-05-09T08:43:02.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.811+0000 7f931b857740 1 --2- 172.21.15.107:0/2899279288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c30567e0 0x5579c3058bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.811+0000 7f931b857740 1 --2- 172.21.15.107:0/2899279288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c304bf20 0x5579c30562a0 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f930800a9f0 tx=0x7f930800aeb0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.811+0000 7f931b857740 1 --2- 172.21.15.107:0/2899279288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c304b600 0x5579c304b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.811+0000 7f931b857740 1 -- 172.21.15.107:0/2899279288 >> 172.21.15.107:0/2899279288 conn(0x5579c3047b10 msgr2=0x5579c3047ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:02.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.813+0000 7f931b857740 1 -- 172.21.15.107:0/2899279288 shutdown_connections 2024-05-09T08:43:02.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.815+0000 7f9311d9b640 1 --2- 172.21.15.107:0/3005238851 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f92fc0a2e40 0x7f92fc0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:02.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.817+0000 7f931b857740 1 -- 172.21.15.107:0/2899279288 wait complete. 2024-05-09T08:43:02.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.817+0000 7f931b857740 1 -- start start 2024-05-09T08:43:02.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.817+0000 7f931b857740 1 Processor -- start 2024-05-09T08:43:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.818+0000 7f931b857740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c304b600 0x5579c30538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.818+0000 7f931b857740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c30567e0 0x5579c3053e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.818+0000 7f931b857740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c3054360 0x5579c30628e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.818+0000 7f931b857740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5579c3059410 con 0x5579c304b600 2024-05-09T08:43:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.818+0000 7f931b857740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5579c2f6b790 con 0x5579c3054360 2024-05-09T08:43:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.818+0000 7f931b857740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5579c3059590 con 0x5579c30567e0 2024-05-09T08:43:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.818+0000 7f931b857740 10 client.0 _ll_register_callbacks cb 0x5579c2fa6a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.818+0000 7f9311d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c30567e0 0x5579c3053e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.818+0000 7f9311d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c30567e0 0x5579c3053e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:52612/0 (socket says 172.21.15.107:52612) 2024-05-09T08:43:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.818+0000 7f9311d9b640 1 -- 172.21.15.107:0/2739782329 learned_addr learned my addr 172.21.15.107:0/2739782329 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.818+0000 7f9311d9b640 1 -- 172.21.15.107:0/2739782329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c3054360 msgr2=0x5579c30628e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:43:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.818+0000 7f9311d9b640 1 --2- 172.21.15.107:0/2739782329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c3054360 0x5579c30628e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.818+0000 7f9311d9b640 1 -- 172.21.15.107:0/2739782329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c304b600 msgr2=0x5579c30538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.818+0000 7f9311d9b640 1 --2- 172.21.15.107:0/2739782329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c304b600 0x5579c30538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.818+0000 7f9311d9b640 1 -- 172.21.15.107:0/2739782329 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579c2cc4210 con 0x5579c30567e0 2024-05-09T08:43:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.819+0000 7f9311d9b640 1 --2- 172.21.15.107:0/2739782329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c30567e0 0x5579c3053e20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f931406d4b0 tx=0x7f9304004840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c0f148453de276c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:02.824 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.822+0000 7f92f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:43:02.824 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.822+0000 7f92f9ffb640 1 -- 172.21.15.107:0/2739782329 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93040160d0 con 0x5579c30567e0 2024-05-09T08:43:02.824 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.822+0000 7f92f9ffb640 1 -- 172.21.15.107:0/2739782329 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f9304016270 con 0x5579c30567e0 2024-05-09T08:43:02.824 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.823+0000 7f931b857740 1 -- 172.21.15.107:0/2739782329 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f930800f900 con 0x5579c30567e0 2024-05-09T08:43:02.826 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.825+0000 7f9311d9b640 1 --2- 172.21.15.107:0/3005238851 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f92fc0a2e40 0x7f92fc0a5300 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9304007fd0 tx=0x7f93040047d0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:02.826 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.825+0000 7f92f9ffb640 1 -- 172.21.15.107:0/2739782329 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f93040068d0 con 0x5579c30567e0 2024-05-09T08:43:02.826 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.826+0000 7f931b857740 10 client.27607 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:02.826 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.826+0000 7f931b857740 1 -- 172.21.15.107:0/2739782329 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5579c2d90620 con 0x5579c30567e0 2024-05-09T08:43:02.829 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.829+0000 7f92f9ffb640 1 -- 172.21.15.107:0/2739782329 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f930800f900 con 0x5579c30567e0 2024-05-09T08:43:02.830 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.830+0000 7f92f9ffb640 1 -- 172.21.15.107:0/2739782329 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9304020460 con 0x5579c30567e0 2024-05-09T08:43:02.830 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.830+0000 7f931b857740 10 client.27607 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:02.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.830+0000 7f931b857740 10 client.27607 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:02.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.830+0000 7f931b857740 20 client.27607 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:02.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.830+0000 7f931b857740 1 --2- 172.21.15.107:0/2739782329 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5579c30671d0 0x5579c3069690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.830+0000 7f931b857740 4 client.27607 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:02.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.830+0000 7f931b857740 1 -- 172.21.15.107:0/2739782329 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579c2fbda70 con 0x5579c30671d0 2024-05-09T08:43:02.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.832+0000 7f931259c640 1 --2- 172.21.15.107:0/2739782329 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5579c30671d0 0x5579c3069690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:02.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.834+0000 7f931259c640 1 --2- 172.21.15.107:0/2739782329 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5579c30671d0 0x5579c3069690 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4525b8f74a8f29ec server_cookie=2503d8752ae11a16 in_seq=0 out_seq=0 2024-05-09T08:43:02.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.834+0000 7f92f9ffb640 10 client.27607 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.837+0000 7f92f9ffb640 1 -- 172.21.15.107:0/2739782329 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5579c2fbda70 con 0x5579c30671d0 2024-05-09T08:43:02.837 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.837+0000 7f92f9ffb640 10 client.27607 handle_command_reply: tid=0 2024-05-09T08:43:02.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.838+0000 7f931b857740 1 client.27607 shutdown 2024-05-09T08:43:02.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.838+0000 7f931b857740 1 -- 172.21.15.107:0/2739782329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c30567e0 msgr2=0x5579c3053e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:02.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.838+0000 7f931b857740 1 --2- 172.21.15.107:0/2739782329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c30567e0 0x5579c3053e20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f931406d4b0 tx=0x7f9304004840 comp rx=0 tx=0).stop 2024-05-09T08:43:02.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.842+0000 7f931b857740 1 -- 172.21.15.107:0/2739782329 shutdown_connections 2024-05-09T08:43:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.842+0000 7f931b857740 1 --2- 172.21.15.107:0/2739782329 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5579c30671d0 0x5579c3069690 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.842+0000 7f931b857740 1 --2- 172.21.15.107:0/2739782329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c3054360 0x5579c30628e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.842+0000 7f931b857740 1 --2- 172.21.15.107:0/2739782329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c30567e0 0x5579c3053e20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.842+0000 7f931b857740 1 --2- 172.21.15.107:0/2739782329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c304b600 0x5579c30538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.842+0000 7f931b857740 1 -- 172.21.15.107:0/2739782329 >> 172.21.15.107:0/2739782329 conn(0x5579c3047b10 msgr2=0x5579c304d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:02.843 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.843+0000 7f92f9ffb640 0 client.27607 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:02.846 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.846+0000 7f931b857740 1 -- 172.21.15.107:0/2739782329 shutdown_connections 2024-05-09T08:43:02.846 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.846+0000 7f931b857740 1 -- 172.21.15.107:0/2739782329 wait complete. 2024-05-09T08:43:02.846 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.846+0000 7f931b857740 20 client.27607 trim_cache size 0 max 16384 2024-05-09T08:43:02.894 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.894+0000 7f931b857740 1 Processor -- start 2024-05-09T08:43:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.895+0000 7f931b857740 1 -- start start 2024-05-09T08:43:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.895+0000 7f931b857740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c3047b10 0x5579c306efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.895+0000 7f931b857740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c304b600 0x5579c306f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.895+0000 7f931b857740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c3054360 0x5579c3062330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.896+0000 7f9311d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c304b600 0x5579c306f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.896+0000 7f931259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c3047b10 0x5579c306efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.896+0000 7f931259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c3047b10 0x5579c306efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:52618/0 (socket says 172.21.15.107:52618) 2024-05-09T08:43:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.896+0000 7f9311d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c304b600 0x5579c306f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:42822/0 (socket says 172.21.15.107:42822) 2024-05-09T08:43:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.896+0000 7f931259c640 1 -- 172.21.15.107:0/2005120967 learned_addr learned my addr 172.21.15.107:0/2005120967 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.896+0000 7f931b857740 1 -- 172.21.15.107:0/2005120967 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f92f4012780 con 0x5579c3054360 2024-05-09T08:43:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.896+0000 7f931b857740 1 -- 172.21.15.107:0/2005120967 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f92f4010050 con 0x5579c304b600 2024-05-09T08:43:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.896+0000 7f931b857740 1 -- 172.21.15.107:0/2005120967 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f92f400df50 con 0x5579c3047b10 2024-05-09T08:43:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.897+0000 7f9312d9d640 1 --2- 172.21.15.107:0/2005120967 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c3054360 0x5579c3062330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.899+0000 7f9311d9b640 1 -- 172.21.15.107:0/2005120967 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c3047b10 msgr2=0x5579c306efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.899+0000 7f9311d9b640 1 --2- 172.21.15.107:0/2005120967 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c3047b10 0x5579c306efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.899+0000 7f9311d9b640 1 -- 172.21.15.107:0/2005120967 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c3054360 msgr2=0x5579c3062330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.899+0000 7f9311d9b640 1 --2- 172.21.15.107:0/2005120967 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c3054360 0x5579c3062330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.899+0000 7f9311d9b640 1 -- 172.21.15.107:0/2005120967 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579c3058f00 con 0x5579c304b600 2024-05-09T08:43:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.900+0000 7f9311d9b640 1 --2- 172.21.15.107:0/2005120967 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c304b600 0x5579c306f4e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9304013c10 tx=0x7f9304016ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c09f822a4dba7436 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:02.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.902+0000 7f92fbfff640 1 -- 172.21.15.107:0/2005120967 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9304032070 con 0x5579c304b600 2024-05-09T08:43:02.906 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.902+0000 7f92fbfff640 1 -- 172.21.15.107:0/2005120967 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9304016190 con 0x5579c304b600 2024-05-09T08:43:02.906 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.902+0000 7f92fbfff640 1 -- 172.21.15.107:0/2005120967 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930402db90 con 0x5579c304b600 2024-05-09T08:43:02.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.907+0000 7f931b857740 1 -- 172.21.15.107:0/2005120967 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c304b600 msgr2=0x5579c306f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:02.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.907+0000 7f931b857740 1 --2- 172.21.15.107:0/2005120967 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c304b600 0x5579c306f4e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9304013c10 tx=0x7f9304016ae0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.914+0000 7f931b857740 1 -- 172.21.15.107:0/2005120967 shutdown_connections 2024-05-09T08:43:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.914+0000 7f931b857740 1 --2- 172.21.15.107:0/2005120967 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c3054360 0x5579c3062330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.914+0000 7f931b857740 1 --2- 172.21.15.107:0/2005120967 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c304b600 0x5579c306f4e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.914+0000 7f931b857740 1 --2- 172.21.15.107:0/2005120967 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c3047b10 0x5579c306efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.914+0000 7f931b857740 1 -- 172.21.15.107:0/2005120967 >> 172.21.15.107:0/2005120967 conn(0x5579c3053f80 msgr2=0x5579c3047ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.918+0000 7f931b857740 1 -- 172.21.15.107:0/2005120967 shutdown_connections 2024-05-09T08:43:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.918+0000 7f931b857740 1 -- 172.21.15.107:0/2005120967 wait complete. 2024-05-09T08:43:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.919+0000 7f931b857740 1 -- start start 2024-05-09T08:43:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.919+0000 7f931b857740 1 Processor -- start 2024-05-09T08:43:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.919+0000 7f931b857740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c3047b10 0x5579c305e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.919+0000 7f931b857740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c304b600 0x5579c305ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.920+0000 7f931b857740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c3054360 0x5579c30500e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.920+0000 7f931259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c3047b10 0x5579c305e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.920+0000 7f931259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c3047b10 0x5579c305e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:42830/0 (socket says 172.21.15.107:42830) 2024-05-09T08:43:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.920+0000 7f9311d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c304b600 0x5579c305ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.920+0000 7f9311d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c304b600 0x5579c305ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:52632/0 (socket says 172.21.15.107:52632) 2024-05-09T08:43:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.920+0000 7f9312d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c3054360 0x5579c30500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.920+0000 7f9312d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c3054360 0x5579c30500e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45418/0 (socket says 172.21.15.107:45418) 2024-05-09T08:43:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.920+0000 7f931259c640 1 -- 172.21.15.107:0/2997151474 learned_addr learned my addr 172.21.15.107:0/2997151474 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.920+0000 7f931b857740 1 -- 172.21.15.107:0/2997151474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f92f400e510 con 0x5579c3054360 2024-05-09T08:43:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.920+0000 7f931b857740 1 -- 172.21.15.107:0/2997151474 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f92f400f980 con 0x5579c3047b10 2024-05-09T08:43:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.920+0000 7f931b857740 1 -- 172.21.15.107:0/2997151474 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f92f400ebe0 con 0x5579c304b600 2024-05-09T08:43:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.920+0000 7f931b857740 10 client.0 _ll_register_callbacks cb 0x7f9304005b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.922+0000 7f9312d9d640 1 -- 172.21.15.107:0/2997151474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c304b600 msgr2=0x5579c305ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.922+0000 7f9312d9d640 1 --2- 172.21.15.107:0/2997151474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c304b600 0x5579c305ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.922+0000 7f9312d9d640 1 -- 172.21.15.107:0/2997151474 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c3047b10 msgr2=0x5579c305e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.922+0000 7f9312d9d640 1 --2- 172.21.15.107:0/2997151474 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c3047b10 0x5579c305e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.922+0000 7f9312d9d640 1 -- 172.21.15.107:0/2997151474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579c3050910 con 0x5579c3054360 2024-05-09T08:43:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.922+0000 7f9312d9d640 1 --2- 172.21.15.107:0/2997151474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c3054360 0x5579c30500e0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f930c0ee3f0 tx=0x7f930c0bda40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b52c1937efb2f49 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.923+0000 7f92f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:43:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.923+0000 7f92f9ffb640 1 -- 172.21.15.107:0/2997151474 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c0c5ef0 con 0x5579c3054360 2024-05-09T08:43:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.923+0000 7f92f9ffb640 1 -- 172.21.15.107:0/2997151474 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f930c030d90 con 0x5579c3054360 2024-05-09T08:43:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.925+0000 7f931b857740 1 -- 172.21.15.107:0/2997151474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9304016190 con 0x5579c3054360 2024-05-09T08:43:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.927+0000 7f92f9ffb640 1 -- 172.21.15.107:0/2997151474 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f930c0ad050 con 0x5579c3054360 2024-05-09T08:43:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.927+0000 7f931b857740 10 client.18411 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.927+0000 7f931b857740 1 -- 172.21.15.107:0/2997151474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5579c30cd480 con 0x5579c3054360 2024-05-09T08:43:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.929+0000 7f92f9ffb640 1 -- 172.21.15.107:0/2997151474 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f930c036030 con 0x5579c3054360 2024-05-09T08:43:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.929+0000 7f92f9ffb640 1 -- 172.21.15.107:0/2997151474 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f930c0bd570 con 0x5579c3054360 2024-05-09T08:43:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.930+0000 7f931b857740 10 client.18411 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.930+0000 7f931b857740 10 client.18411 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.930+0000 7f931b857740 20 client.18411 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.930+0000 7f931b857740 1 --2- 172.21.15.107:0/2997151474 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5579c30567e0 0x5579c30cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.930+0000 7f931b857740 4 client.18411 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:43:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.930+0000 7f931b857740 1 -- 172.21.15.107:0/2997151474 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f9304016190 con 0x5579c30567e0 2024-05-09T08:43:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.930+0000 7f931259c640 1 --2- 172.21.15.107:0/2997151474 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5579c30567e0 0x5579c30cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.932+0000 7f931259c640 1 --2- 172.21.15.107:0/2997151474 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5579c30567e0 0x5579c30cfb90 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f680f35d3de702 server_cookie=4e03c3d77a4f396a in_seq=0 out_seq=0 2024-05-09T08:43:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:02.932+0000 7f92f9ffb640 10 client.18411 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:03.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.040+0000 7f92f9ffb640 1 -- 172.21.15.107:0/2997151474 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f9304016190 con 0x5579c30567e0 2024-05-09T08:43:03.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.041+0000 7f92f9ffb640 10 client.18411 handle_command_reply: tid=0 2024-05-09T08:43:03.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.041+0000 7f931b857740 1 client.18411 shutdown 2024-05-09T08:43:03.043 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.043+0000 7f931b857740 1 -- 172.21.15.107:0/2997151474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c3054360 msgr2=0x5579c30500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:03.043 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.043+0000 7f931b857740 1 --2- 172.21.15.107:0/2997151474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c3054360 0x5579c30500e0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f930c0ee3f0 tx=0x7f930c0bda40 comp rx=0 tx=0).stop 2024-05-09T08:43:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.049+0000 7f931b857740 1 -- 172.21.15.107:0/2997151474 shutdown_connections 2024-05-09T08:43:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.049+0000 7f931b857740 1 --2- 172.21.15.107:0/2997151474 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5579c30567e0 0x5579c30cfb90 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.049+0000 7f931b857740 1 --2- 172.21.15.107:0/2997151474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579c3054360 0x5579c30500e0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.049+0000 7f931b857740 1 --2- 172.21.15.107:0/2997151474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5579c304b600 0x5579c305ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.049+0000 7f931b857740 1 --2- 172.21.15.107:0/2997151474 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5579c3047b10 0x5579c305e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.049+0000 7f931b857740 1 -- 172.21.15.107:0/2997151474 >> 172.21.15.107:0/2997151474 conn(0x5579c3053f80 msgr2=0x5579c30520a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.049+0000 7f92f9ffb640 0 client.18411 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.049+0000 7f931b857740 1 -- 172.21.15.107:0/2997151474 shutdown_connections 2024-05-09T08:43:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.049+0000 7f931b857740 1 -- 172.21.15.107:0/2997151474 wait complete. 2024-05-09T08:43:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.050+0000 7f931b857740 20 client.18411 trim_cache size 0 max 16384 2024-05-09T08:43:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:43:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:43:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:43:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:43:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.052+0000 7f9318b4f640 1 -- 172.21.15.107:0/3005238851 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f92fc0a2e40 msgr2=0x7f92fc0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.052+0000 7f9318b4f640 1 --2- 172.21.15.107:0/3005238851 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f92fc0a2e40 0x7f92fc0a5300 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9304007fd0 tx=0x7f93040047d0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.052+0000 7f9318b4f640 1 -- 172.21.15.107:0/3005238851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f931410fda0 msgr2=0x7f93141a6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.052+0000 7f9318b4f640 1 --2- 172.21.15.107:0/3005238851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f931410fda0 0x7f93141a6a00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f930c003ba0 tx=0x7f930c003bd0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.053+0000 7f9318b4f640 1 -- 172.21.15.107:0/3005238851 shutdown_connections 2024-05-09T08:43:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.053+0000 7f9318b4f640 1 --2- 172.21.15.107:0/3005238851 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f92fc0a2e40 0x7f92fc0a5300 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.053+0000 7f9318b4f640 1 --2- 172.21.15.107:0/3005238851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f931410fda0 0x7f93141a6a00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.053+0000 7f9318b4f640 1 --2- 172.21.15.107:0/3005238851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f931410e5a0 0x7f93141acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.053+0000 7f9318b4f640 1 --2- 172.21.15.107:0/3005238851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f931410db00 0x7f93141ac800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.053+0000 7f9318b4f640 1 -- 172.21.15.107:0/3005238851 >> 172.21.15.107:0/3005238851 conn(0x7f931406c7e0 msgr2=0x7f9314109db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.054+0000 7f9318b4f640 1 -- 172.21.15.107:0/3005238851 shutdown_connections 2024-05-09T08:43:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:03.054+0000 7f9318b4f640 1 -- 172.21.15.107:0/3005238851 wait complete. 2024-05-09T08:43:04.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:04 smithi157 ceph-mon[33614]: pgmap v300: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 23 MiB/s wr, 5.76k op/s 2024-05-09T08:43:04.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:04 smithi110 ceph-mon[33622]: pgmap v300: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 23 MiB/s wr, 5.76k op/s 2024-05-09T08:43:04.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:04 smithi107 ceph-mon[30122]: pgmap v300: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 23 MiB/s wr, 5.76k op/s 2024-05-09T08:43:06.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:06 smithi157 ceph-mon[33614]: pgmap v301: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 15 MiB/s wr, 3.87k op/s 2024-05-09T08:43:06.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:06 smithi110 ceph-mon[33622]: pgmap v301: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 15 MiB/s wr, 3.87k op/s 2024-05-09T08:43:06.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:06 smithi107 ceph-mon[30122]: pgmap v301: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 15 MiB/s wr, 3.87k op/s 2024-05-09T08:43:07.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:07 smithi157 ceph-mon[33614]: osdmap e101: 12 total, 12 up, 12 in 2024-05-09T08:43:07.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:07 smithi110 ceph-mon[33622]: osdmap e101: 12 total, 12 up, 12 in 2024-05-09T08:43:07.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:07 smithi107 ceph-mon[30122]: osdmap e101: 12 total, 12 up, 12 in 2024-05-09T08:43:08.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:08 smithi157 ceph-mon[33614]: pgmap v303: 129 pgs: 9 active+clean+snaptrim, 100 active+clean, 20 active+clean+snaptrim_wait; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 18 MiB/s wr, 4.46k op/s 2024-05-09T08:43:08.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:08 smithi110 ceph-mon[33622]: pgmap v303: 129 pgs: 9 active+clean+snaptrim, 100 active+clean, 20 active+clean+snaptrim_wait; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 18 MiB/s wr, 4.46k op/s 2024-05-09T08:43:08.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:08 smithi107 ceph-mon[30122]: pgmap v303: 129 pgs: 9 active+clean+snaptrim, 100 active+clean, 20 active+clean+snaptrim_wait; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 18 MiB/s wr, 4.46k op/s 2024-05-09T08:43:10.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:10 smithi157 ceph-mon[33614]: pgmap v304: 129 pgs: 17 active+clean+snaptrim, 74 active+clean, 38 active+clean+snaptrim_wait; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 18 MiB/s wr, 4.42k op/s 2024-05-09T08:43:10.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:10 smithi110 ceph-mon[33622]: pgmap v304: 129 pgs: 17 active+clean+snaptrim, 74 active+clean, 38 active+clean+snaptrim_wait; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 18 MiB/s wr, 4.42k op/s 2024-05-09T08:43:10.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:10 smithi107 ceph-mon[30122]: pgmap v304: 129 pgs: 17 active+clean+snaptrim, 74 active+clean, 38 active+clean+snaptrim_wait; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 18 MiB/s wr, 4.42k op/s 2024-05-09T08:43:12.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:12 smithi157 ceph-mon[33614]: pgmap v305: 129 pgs: 17 active+clean+snaptrim, 74 active+clean, 38 active+clean+snaptrim_wait; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 18 MiB/s wr, 4.42k op/s 2024-05-09T08:43:12.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:12 smithi110 ceph-mon[33622]: pgmap v305: 129 pgs: 17 active+clean+snaptrim, 74 active+clean, 38 active+clean+snaptrim_wait; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 18 MiB/s wr, 4.42k op/s 2024-05-09T08:43:12.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:12 smithi107 ceph-mon[30122]: pgmap v305: 129 pgs: 17 active+clean+snaptrim, 74 active+clean, 38 active+clean+snaptrim_wait; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 18 MiB/s wr, 4.42k op/s 2024-05-09T08:43:12.960 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:43:12.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:43:13.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.306+0000 7fdcfea8c640 1 -- 172.21.15.107:0/355632120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcf8108830 msgr2=0x7fdcf8108c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.306+0000 7fdcfea8c640 1 --2- 172.21.15.107:0/355632120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcf8108830 0x7fdcf8108c90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdce801a910 tx=0x7fdce803b500 comp rx=0 tx=0).stop 2024-05-09T08:43:13.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.308+0000 7fdcfea8c640 1 -- 172.21.15.107:0/355632120 shutdown_connections 2024-05-09T08:43:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.308+0000 7fdcfea8c640 1 --2- 172.21.15.107:0/355632120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcf81092b0 0x7fdcf811d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.308+0000 7fdcfea8c640 1 --2- 172.21.15.107:0/355632120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcf8108830 0x7fdcf8108c90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.308+0000 7fdcfea8c640 1 --2- 172.21.15.107:0/355632120 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcf810fed0 0x7fdcf81102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.308+0000 7fdcfea8c640 1 -- 172.21.15.107:0/355632120 >> 172.21.15.107:0/355632120 conn(0x7fdcf806c7b0 msgr2=0x7fdcf806cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.312+0000 7fdcfea8c640 1 -- 172.21.15.107:0/355632120 shutdown_connections 2024-05-09T08:43:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.313+0000 7fdcfea8c640 1 -- 172.21.15.107:0/355632120 wait complete. 2024-05-09T08:43:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.314+0000 7fdcfea8c640 1 Processor -- start 2024-05-09T08:43:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.318+0000 7fdcfea8c640 1 -- start start 2024-05-09T08:43:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.318+0000 7fdcfea8c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcf8108830 0x7fdcf811cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.318+0000 7fdcfea8c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcf81092b0 0x7fdcf8118d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.318+0000 7fdcfea8c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcf810fed0 0x7fdcf81192b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.318+0000 7fdcfea8c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf8113140 con 0x7fdcf810fed0 2024-05-09T08:43:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.318+0000 7fdcfea8c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf8112fc0 con 0x7fdcf81092b0 2024-05-09T08:43:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.318+0000 7fdcfea8c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf81132c0 con 0x7fdcf8108830 2024-05-09T08:43:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.319+0000 7fdcfd289640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcf81092b0 0x7fdcf8118d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.319+0000 7fdcfd289640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcf81092b0 0x7fdcf8118d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:45766/0 (socket says 172.21.15.107:45766) 2024-05-09T08:43:13.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.319+0000 7fdcfd289640 1 -- 172.21.15.107:0/127666839 learned_addr learned my addr 172.21.15.107:0/127666839 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:13.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.319+0000 7fdcfe28b640 1 --2- 172.21.15.107:0/127666839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcf810fed0 0x7fdcf81192b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.319+0000 7fdcfe28b640 1 -- 172.21.15.107:0/127666839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcf8108830 msgr2=0x7fdcf811cce0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:43:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.319+0000 7fdcfe28b640 1 --2- 172.21.15.107:0/127666839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcf8108830 0x7fdcf811cce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.319+0000 7fdcfe28b640 1 -- 172.21.15.107:0/127666839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcf81092b0 msgr2=0x7fdcf8118d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.319+0000 7fdcfe28b640 1 --2- 172.21.15.107:0/127666839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcf81092b0 0x7fdcf8118d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.319+0000 7fdcfe28b640 1 -- 172.21.15.107:0/127666839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf8119990 con 0x7fdcf810fed0 2024-05-09T08:43:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.320+0000 7fdcfe28b640 1 --2- 172.21.15.107:0/127666839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcf810fed0 0x7fdcf81192b0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fdcf000e940 tx=0x7fdcf000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.321+0000 7fdceeffd640 1 -- 172.21.15.107:0/127666839 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0004d20 con 0x7fdcf810fed0 2024-05-09T08:43:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.321+0000 7fdcfea8c640 1 -- 172.21.15.107:0/127666839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcf81ccca0 con 0x7fdcf810fed0 2024-05-09T08:43:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.321+0000 7fdcfea8c640 1 -- 172.21.15.107:0/127666839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcf81cd0a0 con 0x7fdcf810fed0 2024-05-09T08:43:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.325+0000 7fdd029fa740 1 Processor -- start 2024-05-09T08:43:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.325+0000 7fdceeffd640 1 -- 172.21.15.107:0/127666839 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcf0005020 con 0x7fdcf810fed0 2024-05-09T08:43:13.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.325+0000 7fdceeffd640 1 -- 172.21.15.107:0/127666839 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf00116d0 con 0x7fdcf810fed0 2024-05-09T08:43:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.327+0000 7fdd029fa740 1 -- start start 2024-05-09T08:43:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.327+0000 7fdd029fa740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a4470 0x5613a89a4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.327+0000 7fdd029fa740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4d90 0x5613a89aef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.327+0000 7fdd029fa740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89af460 0x5613a89b1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.327+0000 7fdd029fa740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5613a8909d00 con 0x5613a89a4470 2024-05-09T08:43:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.327+0000 7fdd029fa740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5613a88b14c0 con 0x5613a89a4d90 2024-05-09T08:43:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.327+0000 7fdd029fa740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5613a8905cc0 con 0x5613a89af460 2024-05-09T08:43:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.329+0000 7fdceeffd640 1 -- 172.21.15.107:0/127666839 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fdcf000cb20 con 0x7fdcf810fed0 2024-05-09T08:43:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.330+0000 7fdceeffd640 1 --2- 172.21.15.107:0/127666839 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdce40a2ed0 0x7fdce40a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.330+0000 7fdceeffd640 1 -- 172.21.15.107:0/127666839 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fdcf00cfa00 con 0x7fdcf810fed0 2024-05-09T08:43:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.331+0000 7fdcfda8a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4d90 0x5613a89aef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.331+0000 7fdcfda8a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4d90 0x5613a89aef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:45772/0 (socket says 172.21.15.107:45772) 2024-05-09T08:43:13.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.331+0000 7fdcfda8a640 1 -- 172.21.15.107:0/2905205348 learned_addr learned my addr 172.21.15.107:0/2905205348 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:13.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.332+0000 7fdcfda8a640 1 -- 172.21.15.107:0/2905205348 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89af460 msgr2=0x5613a89b1850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:43:13.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.332+0000 7fdcfda8a640 1 --2- 172.21.15.107:0/2905205348 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89af460 0x5613a89b1850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.332+0000 7fdcfda8a640 1 -- 172.21.15.107:0/2905205348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a4470 msgr2=0x5613a89a4850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:43:13.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.332+0000 7fdcfda8a640 1 --2- 172.21.15.107:0/2905205348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a4470 0x5613a89a4850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.332+0000 7fdcfda8a640 1 -- 172.21.15.107:0/2905205348 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613a86e77d0 con 0x5613a89a4d90 2024-05-09T08:43:13.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.333+0000 7fdcfda8a640 1 --2- 172.21.15.107:0/2905205348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4d90 0x5613a89aef20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009780 tx=0x7fdcf4009c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e822a840d638fc1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:13.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.334+0000 7fdce3fff640 1 -- 172.21.15.107:0/2905205348 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4015020 con 0x5613a89a4d90 2024-05-09T08:43:13.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.334+0000 7fdce3fff640 1 -- 172.21.15.107:0/2905205348 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcf4011420 con 0x5613a89a4d90 2024-05-09T08:43:13.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.334+0000 7fdce3fff640 1 -- 172.21.15.107:0/2905205348 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf40105e0 con 0x5613a89a4d90 2024-05-09T08:43:13.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.336+0000 7fdd029fa740 1 -- 172.21.15.107:0/2905205348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4d90 msgr2=0x5613a89aef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:13.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.336+0000 7fdd029fa740 1 --2- 172.21.15.107:0/2905205348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4d90 0x5613a89aef20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009780 tx=0x7fdcf4009c40 comp rx=0 tx=0).stop 2024-05-09T08:43:13.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.337+0000 7fdcfe28b640 1 --2- 172.21.15.107:0/127666839 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdce40a2ed0 0x7fdce40a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.338+0000 7fdcfe28b640 1 --2- 172.21.15.107:0/127666839 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdce40a2ed0 0x7fdce40a5390 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0098480 tx=0x7fdcf0005b40 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:13.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.338+0000 7fdd029fa740 1 -- 172.21.15.107:0/2905205348 shutdown_connections 2024-05-09T08:43:13.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.338+0000 7fdd029fa740 1 --2- 172.21.15.107:0/2905205348 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89af460 0x5613a89b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.338+0000 7fdd029fa740 1 --2- 172.21.15.107:0/2905205348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4d90 0x5613a89aef20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.338+0000 7fdd029fa740 1 --2- 172.21.15.107:0/2905205348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a4470 0x5613a89a4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.338+0000 7fdd029fa740 1 -- 172.21.15.107:0/2905205348 >> 172.21.15.107:0/2905205348 conn(0x5613a89a0790 msgr2=0x5613a89a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:13.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.342+0000 7fdd029fa740 1 -- 172.21.15.107:0/2905205348 shutdown_connections 2024-05-09T08:43:13.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.343+0000 7fdd029fa740 1 -- 172.21.15.107:0/2905205348 wait complete. 2024-05-09T08:43:13.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.343+0000 7fdd029fa740 1 -- start start 2024-05-09T08:43:13.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.343+0000 7fdd029fa740 1 Processor -- start 2024-05-09T08:43:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.344+0000 7fdd029fa740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a4470 0x5613a89ac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.344+0000 7fdd029fa740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a4d90 0x5613a89ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.344+0000 7fdd029fa740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89af460 0x5613a89ace10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.344+0000 7fdd029fa740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5613a89b2250 con 0x5613a89a4470 2024-05-09T08:43:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.344+0000 7fdd029fa740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5613a88c4b90 con 0x5613a89af460 2024-05-09T08:43:13.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.344+0000 7fdd029fa740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5613a89b23d0 con 0x5613a89a4d90 2024-05-09T08:43:13.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.344+0000 7fdd029fa740 10 client.0 _ll_register_callbacks cb 0x5613a8900da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:13.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.346+0000 7fdcfda8a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a4d90 0x5613a89ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.346+0000 7fdcfda8a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a4d90 0x5613a89ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:59926/0 (socket says 172.21.15.107:59926) 2024-05-09T08:43:13.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.346+0000 7fdcfda8a640 1 -- 172.21.15.107:0/4237910926 learned_addr learned my addr 172.21.15.107:0/4237910926 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:13.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.346+0000 7fdcfd289640 1 --2- 172.21.15.107:0/4237910926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a4470 0x5613a89ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.346+0000 7fdcfd289640 1 --2- 172.21.15.107:0/4237910926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89af460 0x5613a89ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.346+0000 7fdcfd289640 1 -- 172.21.15.107:0/4237910926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a4d90 msgr2=0x5613a89ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:13.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.346+0000 7fdcfd289640 1 --2- 172.21.15.107:0/4237910926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a4d90 0x5613a89ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.346+0000 7fdcfd289640 1 -- 172.21.15.107:0/4237910926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89af460 msgr2=0x5613a89ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:13.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.346+0000 7fdcfd289640 1 --2- 172.21.15.107:0/4237910926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89af460 0x5613a89ace10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.346+0000 7fdcfd289640 1 -- 172.21.15.107:0/4237910926 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613a861d3f0 con 0x5613a89a4470 2024-05-09T08:43:13.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.346+0000 7fdcfd289640 1 --2- 172.21.15.107:0/4237910926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a4470 0x5613a89ac390 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fdcf811a5c0 tx=0x7fdce8005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b214cd1a1efa5b5f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:13.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.347+0000 7fdce17fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:43:13.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.347+0000 7fdce17fa640 1 -- 172.21.15.107:0/4237910926 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce8046070 con 0x5613a89a4470 2024-05-09T08:43:13.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.347+0000 7fdd029fa740 1 -- 172.21.15.107:0/4237910926 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcf4011420 con 0x5613a89a4470 2024-05-09T08:43:13.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.347+0000 7fdce17fa640 1 -- 172.21.15.107:0/4237910926 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fdce80052d0 con 0x5613a89a4470 2024-05-09T08:43:13.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.349+0000 7fdce17fa640 1 -- 172.21.15.107:0/4237910926 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdce804e050 con 0x5613a89a4470 2024-05-09T08:43:13.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.350+0000 7fdd029fa740 10 client.18420 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:13.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.350+0000 7fdd029fa740 1 -- 172.21.15.107:0/4237910926 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5613a85dd0f0 con 0x5613a89a4470 2024-05-09T08:43:13.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.354+0000 7fdce17fa640 1 -- 172.21.15.107:0/4237910926 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcf4011420 con 0x5613a89a4470 2024-05-09T08:43:13.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.354+0000 7fdce17fa640 1 -- 172.21.15.107:0/4237910926 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fdce80124c0 con 0x5613a89a4470 2024-05-09T08:43:13.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.354+0000 7fdd029fa740 10 client.18420 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:13.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.354+0000 7fdd029fa740 10 client.18420 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:13.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.354+0000 7fdd029fa740 20 client.18420 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:13.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.354+0000 7fdd029fa740 1 --2- 172.21.15.107:0/4237910926 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5613a89bfd20 0x5613a89c21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.354+0000 7fdd029fa740 4 client.18420 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:13.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.354+0000 7fdd029fa740 1 -- 172.21.15.107:0/4237910926 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613a8917430 con 0x5613a89bfd20 2024-05-09T08:43:13.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.355+0000 7fdcfda8a640 1 --2- 172.21.15.107:0/4237910926 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5613a89bfd20 0x5613a89c21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.363+0000 7fdcfda8a640 1 --2- 172.21.15.107:0/4237910926 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5613a89bfd20 0x5613a89c21a0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b41265c77772c51 server_cookie=b4346af6e9e6166f in_seq=0 out_seq=0 2024-05-09T08:43:13.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.363+0000 7fdce17fa640 10 client.18420 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:13.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.364+0000 7fdce17fa640 1 -- 172.21.15.107:0/4237910926 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5613a8917430 con 0x5613a89bfd20 2024-05-09T08:43:13.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.364+0000 7fdce17fa640 10 client.18420 handle_command_reply: tid=0 2024-05-09T08:43:13.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.364+0000 7fdd029fa740 1 client.18420 shutdown 2024-05-09T08:43:13.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.367+0000 7fdd029fa740 1 -- 172.21.15.107:0/4237910926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a4470 msgr2=0x5613a89ac390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:13.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.367+0000 7fdd029fa740 1 --2- 172.21.15.107:0/4237910926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a4470 0x5613a89ac390 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fdcf811a5c0 tx=0x7fdce8005e60 comp rx=0 tx=0).stop 2024-05-09T08:43:13.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.369+0000 7fdd029fa740 1 -- 172.21.15.107:0/4237910926 shutdown_connections 2024-05-09T08:43:13.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.369+0000 7fdd029fa740 1 --2- 172.21.15.107:0/4237910926 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5613a89bfd20 0x5613a89c21a0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.369+0000 7fdd029fa740 1 --2- 172.21.15.107:0/4237910926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89af460 0x5613a89ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.369+0000 7fdd029fa740 1 --2- 172.21.15.107:0/4237910926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a4d90 0x5613a89ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.369+0000 7fdd029fa740 1 --2- 172.21.15.107:0/4237910926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a4470 0x5613a89ac390 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.369+0000 7fdd029fa740 1 -- 172.21.15.107:0/4237910926 >> 172.21.15.107:0/4237910926 conn(0x5613a89a0790 msgr2=0x5613a899fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:13.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.375+0000 7fdce17fa640 0 client.18420 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:13.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.377+0000 7fdd029fa740 1 -- 172.21.15.107:0/4237910926 shutdown_connections 2024-05-09T08:43:13.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.383+0000 7fdd029fa740 1 -- 172.21.15.107:0/4237910926 wait complete. 2024-05-09T08:43:13.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.383+0000 7fdd029fa740 20 client.18420 trim_cache size 0 max 16384 2024-05-09T08:43:13.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.447+0000 7fdd029fa740 1 Processor -- start 2024-05-09T08:43:13.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.447+0000 7fdd029fa740 1 -- start start 2024-05-09T08:43:13.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.447+0000 7fdd029fa740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a0790 0x5613a89bb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.447+0000 7fdd029fa740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4470 0x5613a89a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.447+0000 7fdcfd289640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a0790 0x5613a89bb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.447+0000 7fdcfd289640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a0790 0x5613a89bb1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54886/0 (socket says 172.21.15.107:54886) 2024-05-09T08:43:13.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.447+0000 7fdcfd289640 1 -- 172.21.15.107:0/2925781275 learned_addr learned my addr 172.21.15.107:0/2925781275 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:13.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.447+0000 7fdd029fa740 1 --2- 172.21.15.107:0/2925781275 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a4d90 0x5613a89c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.447+0000 7fdd029fa740 1 -- 172.21.15.107:0/2925781275 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc012a10 con 0x5613a89a0790 2024-05-09T08:43:13.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.447+0000 7fdd029fa740 1 -- 172.21.15.107:0/2925781275 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc0102e0 con 0x5613a89a4470 2024-05-09T08:43:13.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.447+0000 7fdd029fa740 1 -- 172.21.15.107:0/2925781275 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc00e150 con 0x5613a89a4d90 2024-05-09T08:43:13.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.450+0000 7fdcfda8a640 1 --2- 172.21.15.107:0/2925781275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4470 0x5613a89a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.450+0000 7fdcfd289640 1 --2- 172.21.15.107:0/2925781275 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a4d90 0x5613a89c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.452+0000 7fdcfda8a640 1 -- 172.21.15.107:0/2925781275 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a4d90 msgr2=0x5613a89c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:13.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.452+0000 7fdcfda8a640 1 --2- 172.21.15.107:0/2925781275 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a4d90 0x5613a89c7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.452+0000 7fdcfda8a640 1 -- 172.21.15.107:0/2925781275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a0790 msgr2=0x5613a89bb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:13.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.452+0000 7fdcfda8a640 1 --2- 172.21.15.107:0/2925781275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a0790 0x5613a89bb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.452+0000 7fdcfda8a640 1 -- 172.21.15.107:0/2925781275 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613a89aa8e0 con 0x5613a89a4470 2024-05-09T08:43:13.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.456+0000 7fdcfda8a640 1 --2- 172.21.15.107:0/2925781275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4470 0x5613a89a62d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x5613a89c2a50 tx=0x7fdcf4009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77daa73788c77fba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:13.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.457+0000 7fdce37fe640 1 -- 172.21.15.107:0/2925781275 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4015040 con 0x5613a89a4470 2024-05-09T08:43:13.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.457+0000 7fdce37fe640 1 -- 172.21.15.107:0/2925781275 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcf4024860 con 0x5613a89a4470 2024-05-09T08:43:13.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.457+0000 7fdce37fe640 1 -- 172.21.15.107:0/2925781275 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4011950 con 0x5613a89a4470 2024-05-09T08:43:13.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.458+0000 7fdd029fa740 1 -- 172.21.15.107:0/2925781275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4470 msgr2=0x5613a89a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:13.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.458+0000 7fdd029fa740 1 --2- 172.21.15.107:0/2925781275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4470 0x5613a89a62d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x5613a89c2a50 tx=0x7fdcf4009040 comp rx=0 tx=0).stop 2024-05-09T08:43:13.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.460+0000 7fdd029fa740 1 -- 172.21.15.107:0/2925781275 shutdown_connections 2024-05-09T08:43:13.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.460+0000 7fdd029fa740 1 --2- 172.21.15.107:0/2925781275 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a4d90 0x5613a89c7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.460+0000 7fdd029fa740 1 --2- 172.21.15.107:0/2925781275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4470 0x5613a89a62d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.460+0000 7fdd029fa740 1 --2- 172.21.15.107:0/2925781275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a0790 0x5613a89bb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.460+0000 7fdd029fa740 1 -- 172.21.15.107:0/2925781275 >> 172.21.15.107:0/2925781275 conn(0x5613a89aafd0 msgr2=0x5613a89a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:13.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.460+0000 7fdd029fa740 1 -- 172.21.15.107:0/2925781275 shutdown_connections 2024-05-09T08:43:13.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.460+0000 7fdd029fa740 1 -- 172.21.15.107:0/2925781275 wait complete. 2024-05-09T08:43:13.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.460+0000 7fdd029fa740 1 -- start start 2024-05-09T08:43:13.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.461+0000 7fdd029fa740 1 Processor -- start 2024-05-09T08:43:13.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.461+0000 7fdd029fa740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a0790 0x5613a89b7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.461+0000 7fdd029fa740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a4470 0x5613a89b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.461+0000 7fdd029fa740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4d90 0x5613a89b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.461+0000 7fdd029fa740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc00e7a0 con 0x5613a89a4470 2024-05-09T08:43:13.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.461+0000 7fdd029fa740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc00fc10 con 0x5613a89a4d90 2024-05-09T08:43:13.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.461+0000 7fdd029fa740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc00ee70 con 0x5613a89a0790 2024-05-09T08:43:13.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.461+0000 7fdd029fa740 10 client.0 _ll_register_callbacks cb 0x7fdce8005db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:13.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.461+0000 7fdcfd289640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4d90 0x5613a89b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.461+0000 7fdcfda8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a4470 0x5613a89b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.461+0000 7fdcfd289640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4d90 0x5613a89b32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:45802/0 (socket says 172.21.15.107:45802) 2024-05-09T08:43:13.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.461+0000 7fdcfd289640 1 -- 172.21.15.107:0/3059377157 learned_addr learned my addr 172.21.15.107:0/3059377157 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:13.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.461+0000 7fdcfd289640 1 --2- 172.21.15.107:0/3059377157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a0790 0x5613a89b7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.462+0000 7fdcfd289640 1 -- 172.21.15.107:0/3059377157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4d90 msgr2=0x5613a89b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:13.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.462+0000 7fdcfd289640 1 --2- 172.21.15.107:0/3059377157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4d90 0x5613a89b32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.462+0000 7fdcfd289640 1 -- 172.21.15.107:0/3059377157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a4470 msgr2=0x5613a89b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:13.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.462+0000 7fdcfd289640 1 --2- 172.21.15.107:0/3059377157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a4470 0x5613a89b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.462+0000 7fdcfd289640 1 -- 172.21.15.107:0/3059377157 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613a89b3af0 con 0x5613a89a0790 2024-05-09T08:43:13.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.465+0000 7fdcfd289640 1 --2- 172.21.15.107:0/3059377157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a0790 0x5613a89b7a40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdce803baf0 tx=0x7fdce803b500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=723f010643be5cd0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:13.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.465+0000 7fdce17fa640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:43:13.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.465+0000 7fdce17fa640 1 -- 172.21.15.107:0/3059377157 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce8056070 con 0x5613a89a0790 2024-05-09T08:43:13.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.465+0000 7fdce17fa640 1 -- 172.21.15.107:0/3059377157 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fdce8012cc0 con 0x5613a89a0790 2024-05-09T08:43:13.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.465+0000 7fdd029fa740 1 -- 172.21.15.107:0/3059377157 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcf4024860 con 0x5613a89a0790 2024-05-09T08:43:13.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.466+0000 7fdce17fa640 1 -- 172.21.15.107:0/3059377157 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdce8066050 con 0x5613a89a0790 2024-05-09T08:43:13.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.470+0000 7fdd029fa740 10 client.27622 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:13.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.470+0000 7fdd029fa740 1 -- 172.21.15.107:0/3059377157 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5613a8a262c0 con 0x5613a89a0790 2024-05-09T08:43:13.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.471+0000 7fdce17fa640 1 -- 172.21.15.107:0/3059377157 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcf4024860 con 0x5613a89a0790 2024-05-09T08:43:13.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.471+0000 7fdce17fa640 1 -- 172.21.15.107:0/3059377157 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fdce805f070 con 0x5613a89a0790 2024-05-09T08:43:13.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.471+0000 7fdd029fa740 10 client.27622 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:13.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.471+0000 7fdd029fa740 10 client.27622 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:13.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.471+0000 7fdd029fa740 20 client.27622 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:13.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.471+0000 7fdd029fa740 1 --2- 172.21.15.107:0/3059377157 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5613a89af460 0x5613a8a28a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.471+0000 7fdd029fa740 4 client.27622 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:43:13.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.471+0000 7fdd029fa740 1 -- 172.21.15.107:0/3059377157 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5613a8911780 con 0x5613a89af460 2024-05-09T08:43:13.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.471+0000 7fdcfda8a640 1 --2- 172.21.15.107:0/3059377157 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5613a89af460 0x5613a8a28a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.472+0000 7fdce17fa640 10 client.27622 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:13.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.473+0000 7fdcfda8a640 1 --2- 172.21.15.107:0/3059377157 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5613a89af460 0x5613a8a28a50 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72d448fcd2b84e2a server_cookie=812497b28e36811c in_seq=0 out_seq=0 2024-05-09T08:43:13.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.474+0000 7fdce17fa640 1 -- 172.21.15.107:0/3059377157 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5613a8911780 con 0x5613a89af460 2024-05-09T08:43:13.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.474+0000 7fdce17fa640 10 client.27622 handle_command_reply: tid=0 2024-05-09T08:43:13.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.474+0000 7fdd029fa740 1 client.27622 shutdown 2024-05-09T08:43:13.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.474+0000 7fdd029fa740 1 -- 172.21.15.107:0/3059377157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a0790 msgr2=0x5613a89b7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:13.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.474+0000 7fdd029fa740 1 --2- 172.21.15.107:0/3059377157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a0790 0x5613a89b7a40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdce803baf0 tx=0x7fdce803b500 comp rx=0 tx=0).stop 2024-05-09T08:43:13.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.474+0000 7fdd029fa740 1 -- 172.21.15.107:0/3059377157 shutdown_connections 2024-05-09T08:43:13.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.474+0000 7fdd029fa740 1 --2- 172.21.15.107:0/3059377157 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5613a89af460 0x5613a8a28a50 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.474+0000 7fdd029fa740 1 --2- 172.21.15.107:0/3059377157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5613a89a4d90 0x5613a89b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.474+0000 7fdd029fa740 1 --2- 172.21.15.107:0/3059377157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5613a89a4470 0x5613a89b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.474+0000 7fdd029fa740 1 --2- 172.21.15.107:0/3059377157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5613a89a0790 0x5613a89b7a40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.474+0000 7fdd029fa740 1 -- 172.21.15.107:0/3059377157 >> 172.21.15.107:0/3059377157 conn(0x5613a89aafd0 msgr2=0x5613a89a6450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.474+0000 7fdce17fa640 0 client.27622 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.475+0000 7fdd029fa740 1 -- 172.21.15.107:0/3059377157 shutdown_connections 2024-05-09T08:43:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.475+0000 7fdd029fa740 1 -- 172.21.15.107:0/3059377157 wait complete. 2024-05-09T08:43:13.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.475+0000 7fdd029fa740 20 client.27622 trim_cache size 0 max 16384 2024-05-09T08:43:13.478 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:43:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:43:13.479 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:43:13.479 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:43:13.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.484+0000 7fdcfea8c640 1 -- 172.21.15.107:0/127666839 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdce40a2ed0 msgr2=0x7fdce40a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:13.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.484+0000 7fdcfea8c640 1 --2- 172.21.15.107:0/127666839 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdce40a2ed0 0x7fdce40a5390 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0098480 tx=0x7fdcf0005b40 comp rx=0 tx=0).stop 2024-05-09T08:43:13.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.485+0000 7fdcfea8c640 1 -- 172.21.15.107:0/127666839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcf810fed0 msgr2=0x7fdcf81192b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:13.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.485+0000 7fdcfea8c640 1 --2- 172.21.15.107:0/127666839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcf810fed0 0x7fdcf81192b0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fdcf000e940 tx=0x7fdcf000ee00 comp rx=0 tx=0).stop 2024-05-09T08:43:13.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.487+0000 7fdcfea8c640 1 -- 172.21.15.107:0/127666839 shutdown_connections 2024-05-09T08:43:13.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.487+0000 7fdcfea8c640 1 --2- 172.21.15.107:0/127666839 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdce40a2ed0 0x7fdce40a5390 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.487+0000 7fdcfea8c640 1 --2- 172.21.15.107:0/127666839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdcf810fed0 0x7fdcf81192b0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.487+0000 7fdcfea8c640 1 --2- 172.21.15.107:0/127666839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcf81092b0 0x7fdcf8118d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.487+0000 7fdcfea8c640 1 --2- 172.21.15.107:0/127666839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcf8108830 0x7fdcf811cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.487+0000 7fdcfea8c640 1 -- 172.21.15.107:0/127666839 >> 172.21.15.107:0/127666839 conn(0x7fdcf806c7b0 msgr2=0x7fdcf8070430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:13.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.491+0000 7fdcfea8c640 1 -- 172.21.15.107:0/127666839 shutdown_connections 2024-05-09T08:43:13.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.491+0000 7fdcfea8c640 1 -- 172.21.15.107:0/127666839 wait complete. 2024-05-09T08:43:13.514 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:43:13.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:43:13.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.855+0000 7f6ab96cd640 1 -- 172.21.15.107:0/3573254009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab410e790 msgr2=0x7f6ab411d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:13.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.855+0000 7f6ab96cd640 1 --2- 172.21.15.107:0/3573254009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab410e790 0x7f6ab411d080 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6aac009be0 tx=0x7f6aac02a710 comp rx=0 tx=0).stop 2024-05-09T08:43:13.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.857+0000 7f6ab96cd640 1 -- 172.21.15.107:0/3573254009 shutdown_connections 2024-05-09T08:43:13.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.857+0000 7f6ab96cd640 1 --2- 172.21.15.107:0/3573254009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab410e790 0x7f6ab411d080 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.857+0000 7f6ab96cd640 1 --2- 172.21.15.107:0/3573254009 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ab4110400 0x7f6ab410e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.857+0000 7f6ab96cd640 1 --2- 172.21.15.107:0/3573254009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ab410ff90 0x7f6ab410db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.857+0000 7f6ab96cd640 1 -- 172.21.15.107:0/3573254009 >> 172.21.15.107:0/3573254009 conn(0x7f6ab406c950 msgr2=0x7f6ab406cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:13.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.878+0000 7f6ab96cd640 1 -- 172.21.15.107:0/3573254009 shutdown_connections 2024-05-09T08:43:13.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.879+0000 7f6ab96cd640 1 -- 172.21.15.107:0/3573254009 wait complete. 2024-05-09T08:43:13.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.880+0000 7f6ab96cd640 1 Processor -- start 2024-05-09T08:43:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.881+0000 7f6ab96cd640 1 -- start start 2024-05-09T08:43:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.881+0000 7f6ab96cd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab410ff90 0x7f6ab41a6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.881+0000 7f6ab96cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ab41a73b0 0x7f6ab41ab730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.881+0000 7f6ab96cd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ab41abc70 0x7f6ab41ac120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.881+0000 7f6ab96cd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab41208f0 con 0x7f6ab41a73b0 2024-05-09T08:43:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.881+0000 7f6ab96cd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab4120770 con 0x7f6ab410ff90 2024-05-09T08:43:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.881+0000 7f6ab96cd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab4120a70 con 0x7f6ab41abc70 2024-05-09T08:43:13.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.884+0000 7f6ab27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ab41a73b0 0x7f6ab41ab730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.884+0000 7f6ab27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ab41a73b0 0x7f6ab41ab730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54916/0 (socket says 172.21.15.107:54916) 2024-05-09T08:43:13.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.884+0000 7f6ab27fc640 1 -- 172.21.15.107:0/4199146356 learned_addr learned my addr 172.21.15.107:0/4199146356 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:13.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.885+0000 7f6ab2ffd640 1 --2- 172.21.15.107:0/4199146356 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab410ff90 0x7f6ab41a6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.885+0000 7f6ab27fc640 1 -- 172.21.15.107:0/4199146356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ab41abc70 msgr2=0x7f6ab41ac120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:43:13.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.885+0000 7f6ab27fc640 1 --2- 172.21.15.107:0/4199146356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ab41abc70 0x7f6ab41ac120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.885+0000 7f6ab27fc640 1 -- 172.21.15.107:0/4199146356 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab410ff90 msgr2=0x7f6ab41a6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:13.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.885+0000 7f6ab27fc640 1 --2- 172.21.15.107:0/4199146356 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab410ff90 0x7f6ab41a6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:13.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.885+0000 7f6ab27fc640 1 -- 172.21.15.107:0/4199146356 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ab41107f0 con 0x7f6ab41a73b0 2024-05-09T08:43:13.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.886+0000 7f6ab2ffd640 1 --2- 172.21.15.107:0/4199146356 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab410ff90 0x7f6ab41a6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:43:13.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.886+0000 7f6ab27fc640 1 --2- 172.21.15.107:0/4199146356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ab41a73b0 0x7f6ab41ab730 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400b450 tx=0x7f6aa400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:13.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.886+0000 7f6a9bfff640 1 -- 172.21.15.107:0/4199146356 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4015020 con 0x7f6ab41a73b0 2024-05-09T08:43:13.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.886+0000 7f6ab96cd640 1 -- 172.21.15.107:0/4199146356 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ab4110ae0 con 0x7f6ab41a73b0 2024-05-09T08:43:13.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.886+0000 7f6a9bfff640 1 -- 172.21.15.107:0/4199146356 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6aa4011d40 con 0x7f6ab41a73b0 2024-05-09T08:43:13.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.886+0000 7f6a9bfff640 1 -- 172.21.15.107:0/4199146356 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa400f340 con 0x7f6ab41a73b0 2024-05-09T08:43:13.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.887+0000 7f6ab96cd640 1 -- 172.21.15.107:0/4199146356 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ab4111030 con 0x7f6ab41a73b0 2024-05-09T08:43:13.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.890+0000 7f6ab96cd640 1 -- 172.21.15.107:0/4199146356 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ab406dbe0 con 0x7f6ab41a73b0 2024-05-09T08:43:13.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.904+0000 7f6a9bfff640 1 -- 172.21.15.107:0/4199146356 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6aa4018020 con 0x7f6ab41a73b0 2024-05-09T08:43:13.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.905+0000 7f6a9bfff640 1 --2- 172.21.15.107:0/4199146356 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6a9c0a2e50 0x7f6a9c0a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:13.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.910+0000 7f6ab2ffd640 1 --2- 172.21.15.107:0/4199146356 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6a9c0a2e50 0x7f6a9c0a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:13.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.910+0000 7f6a9bfff640 1 -- 172.21.15.107:0/4199146356 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f6aa40d2060 con 0x7f6ab41a73b0 2024-05-09T08:43:13.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.913+0000 7f6ab2ffd640 1 --2- 172.21.15.107:0/4199146356 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6a9c0a2e50 0x7f6a9c0a5310 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8005fd0 tx=0x7f6aa8005a20 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:13.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:13.915+0000 7f6a9bfff640 1 -- 172.21.15.107:0/4199146356 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6aa404c740 con 0x7f6ab41a73b0 2024-05-09T08:43:14.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.454+0000 7f6ab96cd640 1 -- 172.21.15.107:0/4199146356 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ab4068650 con 0x7f6ab41a73b0 2024-05-09T08:43:14.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.459+0000 7f6a9bfff640 1 -- 172.21.15.107:0/4199146356 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11887 (secure 0 0 0) 0x7f6aa400d040 con 0x7f6ab41a73b0 2024-05-09T08:43:14.459 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:43:14.460 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:37:10.930317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:43:14.460 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2024-05-09T08:43:14.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.465+0000 7f6ab96cd640 1 -- 172.21.15.107:0/4199146356 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6a9c0a2e50 msgr2=0x7f6a9c0a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:14.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.465+0000 7f6ab96cd640 1 --2- 172.21.15.107:0/4199146356 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6a9c0a2e50 0x7f6a9c0a5310 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8005fd0 tx=0x7f6aa8005a20 comp rx=0 tx=0).stop 2024-05-09T08:43:14.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.465+0000 7f6ab96cd640 1 -- 172.21.15.107:0/4199146356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ab41a73b0 msgr2=0x7f6ab41ab730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:14.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.465+0000 7f6ab96cd640 1 --2- 172.21.15.107:0/4199146356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ab41a73b0 0x7f6ab41ab730 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400b450 tx=0x7f6aa400b910 comp rx=0 tx=0).stop 2024-05-09T08:43:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.470+0000 7f6ab96cd640 1 -- 172.21.15.107:0/4199146356 shutdown_connections 2024-05-09T08:43:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.471+0000 7f6ab96cd640 1 --2- 172.21.15.107:0/4199146356 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6a9c0a2e50 0x7f6a9c0a5310 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.471+0000 7f6ab96cd640 1 --2- 172.21.15.107:0/4199146356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ab41abc70 0x7f6ab41ac120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.471+0000 7f6ab96cd640 1 --2- 172.21.15.107:0/4199146356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ab41a73b0 0x7f6ab41ab730 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.471+0000 7f6ab96cd640 1 --2- 172.21.15.107:0/4199146356 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab410ff90 0x7f6ab41a6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.471+0000 7f6ab96cd640 1 -- 172.21.15.107:0/4199146356 >> 172.21.15.107:0/4199146356 conn(0x7f6ab406c950 msgr2=0x7f6ab4070e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:14.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.472+0000 7f6ab96cd640 1 -- 172.21.15.107:0/4199146356 shutdown_connections 2024-05-09T08:43:14.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.473+0000 7f6ab96cd640 1 -- 172.21.15.107:0/4199146356 wait complete. 2024-05-09T08:43:14.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:43:14.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:14 smithi157 ceph-mon[33614]: pgmap v306: 129 pgs: 8 active+clean+snaptrim, 118 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 16 MiB/s wr, 4.09k op/s 2024-05-09T08:43:14.594 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:14 smithi107 ceph-mon[30122]: pgmap v306: 129 pgs: 8 active+clean+snaptrim, 118 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 16 MiB/s wr, 4.09k op/s 2024-05-09T08:43:14.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:14 smithi110 ceph-mon[33622]: pgmap v306: 129 pgs: 8 active+clean+snaptrim, 118 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 16 MiB/s wr, 4.09k op/s 2024-05-09T08:43:14.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.813+0000 7f1c24bb0640 1 -- 172.21.15.107:0/3627004909 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c20108830 msgr2=0x7f1c20108c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:14.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.813+0000 7f1c24bb0640 1 --2- 172.21.15.107:0/3627004909 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c20108830 0x7f1c20108c90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1c10009be0 tx=0x7f1c1002a790 comp rx=0 tx=0).stop 2024-05-09T08:43:14.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.814+0000 7f1c24bb0640 1 -- 172.21.15.107:0/3627004909 shutdown_connections 2024-05-09T08:43:14.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.814+0000 7f1c24bb0640 1 --2- 172.21.15.107:0/3627004909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c201092b0 0x7f1c2011d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.814+0000 7f1c24bb0640 1 --2- 172.21.15.107:0/3627004909 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c20108830 0x7f1c20108c90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.814+0000 7f1c24bb0640 1 --2- 172.21.15.107:0/3627004909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c2010fed0 0x7f1c201102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.814+0000 7f1c24bb0640 1 -- 172.21.15.107:0/3627004909 >> 172.21.15.107:0/3627004909 conn(0x7f1c2006c7b0 msgr2=0x7f1c2006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.818+0000 7f1c24bb0640 1 -- 172.21.15.107:0/3627004909 shutdown_connections 2024-05-09T08:43:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.818+0000 7f1c24bb0640 1 -- 172.21.15.107:0/3627004909 wait complete. 2024-05-09T08:43:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.819+0000 7f1c24bb0640 1 Processor -- start 2024-05-09T08:43:14.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.819+0000 7f1c24bb0640 1 -- start start 2024-05-09T08:43:14.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.819+0000 7f1c24bb0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c20108830 0x7f1c2011ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.819+0000 7f1c24bb0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c201092b0 0x7f1c20115d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.819+0000 7f1c24bb0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c2010fed0 0x7f1c201162c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.819+0000 7f1c24bb0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1c20120760 con 0x7f1c20108830 2024-05-09T08:43:14.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.819+0000 7f1c24bb0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1c201205e0 con 0x7f1c2010fed0 2024-05-09T08:43:14.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.819+0000 7f1c24bb0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1c201208e0 con 0x7f1c201092b0 2024-05-09T08:43:14.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.820+0000 7f1c1f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c20108830 0x7f1c2011ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:14.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.820+0000 7f1c1f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c20108830 0x7f1c2011ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54938/0 (socket says 172.21.15.107:54938) 2024-05-09T08:43:14.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.820+0000 7f1c1f7fe640 1 -- 172.21.15.107:0/2865344993 learned_addr learned my addr 172.21.15.107:0/2865344993 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:14.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.823+0000 7f1c1f7fe640 1 -- 172.21.15.107:0/2865344993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c201092b0 msgr2=0x7f1c20115d80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:43:14.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.823+0000 7f1c1f7fe640 1 --2- 172.21.15.107:0/2865344993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c201092b0 0x7f1c20115d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.823+0000 7f1c1f7fe640 1 -- 172.21.15.107:0/2865344993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c2010fed0 msgr2=0x7f1c201162c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:43:14.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.823+0000 7f1c1f7fe640 1 --2- 172.21.15.107:0/2865344993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c2010fed0 0x7f1c201162c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.823+0000 7f1c1f7fe640 1 -- 172.21.15.107:0/2865344993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c20116930 con 0x7f1c20108830 2024-05-09T08:43:14.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.828+0000 7f1c1f7fe640 1 --2- 172.21.15.107:0/2865344993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c20108830 0x7f1c2011ccb0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800e9c0 tx=0x7f1c0800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:14.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.829+0000 7f1c1cff9640 1 -- 172.21.15.107:0/2865344993 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0800ccc0 con 0x7f1c20108830 2024-05-09T08:43:14.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.829+0000 7f1c1cff9640 1 -- 172.21.15.107:0/2865344993 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c080054d0 con 0x7f1c20108830 2024-05-09T08:43:14.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.829+0000 7f1c1cff9640 1 -- 172.21.15.107:0/2865344993 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c08010430 con 0x7f1c20108830 2024-05-09T08:43:14.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.829+0000 7f1c24bb0640 1 -- 172.21.15.107:0/2865344993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c2011a7d0 con 0x7f1c20108830 2024-05-09T08:43:14.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.829+0000 7f1c24bb0640 1 -- 172.21.15.107:0/2865344993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c2011acd0 con 0x7f1c20108830 2024-05-09T08:43:14.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.835+0000 7f1c1cff9640 1 -- 172.21.15.107:0/2865344993 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1c080046c0 con 0x7f1c20108830 2024-05-09T08:43:14.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.836+0000 7f1c1cff9640 1 --2- 172.21.15.107:0/2865344993 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c0c0a2f30 0x7f1c0c0a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.836+0000 7f1c1cff9640 1 -- 172.21.15.107:0/2865344993 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f1c080cf9c0 con 0x7f1c20108830 2024-05-09T08:43:14.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.838+0000 7f1c1effd640 1 --2- 172.21.15.107:0/2865344993 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c0c0a2f30 0x7f1c0c0a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:14.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.839+0000 7f1c28bfb740 1 Processor -- start 2024-05-09T08:43:14.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.840+0000 7f1c28bfb740 1 -- start start 2024-05-09T08:43:14.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.840+0000 7f1c28bfb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f180004f0 0x561f180008d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.840+0000 7f1c28bfb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f18000e10 0x561f1800afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.840+0000 7f1c28bfb740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f1800b4e0 0x561f1800d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.840+0000 7f1c28bfb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561f17f65c40 con 0x561f18000e10 2024-05-09T08:43:14.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.840+0000 7f1c28bfb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561f17f0d580 con 0x561f180004f0 2024-05-09T08:43:14.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.840+0000 7f1c28bfb740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x561f17f61c00 con 0x561f1800b4e0 2024-05-09T08:43:14.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.841+0000 7f1c1f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f180004f0 0x561f180008d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:14.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.841+0000 7f1c1f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f180004f0 0x561f180008d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:45834/0 (socket says 172.21.15.107:45834) 2024-05-09T08:43:14.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.841+0000 7f1c1f7fe640 1 -- 172.21.15.107:0/1806107240 learned_addr learned my addr 172.21.15.107:0/1806107240 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:14.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.844+0000 7f1c1f7fe640 1 -- 172.21.15.107:0/1806107240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f1800b4e0 msgr2=0x561f1800d8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:43:14.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.844+0000 7f1c1f7fe640 1 --2- 172.21.15.107:0/1806107240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f1800b4e0 0x561f1800d8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.844+0000 7f1c1f7fe640 1 -- 172.21.15.107:0/1806107240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f18000e10 msgr2=0x561f1800afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:14.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.844+0000 7f1c1f7fe640 1 --2- 172.21.15.107:0/1806107240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f18000e10 0x561f1800afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.844+0000 7f1c1f7fe640 1 -- 172.21.15.107:0/1806107240 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f17c793f0 con 0x561f180004f0 2024-05-09T08:43:14.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.845+0000 7f1c1effd640 1 --2- 172.21.15.107:0/2865344993 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c0c0a2f30 0x7f1c0c0a53f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1c10009b30 tx=0x7f1c10000f40 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:14.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.852+0000 7f1c1f7fe640 1 --2- 172.21.15.107:0/1806107240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f180004f0 0x561f180008d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1c08009230 tx=0x7f1c0809c170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39cbb2f27174902e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:14.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.858+0000 7f1c15ffb640 1 -- 172.21.15.107:0/1806107240 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c080b3020 con 0x561f180004f0 2024-05-09T08:43:14.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.866+0000 7f1c15ffb640 1 -- 172.21.15.107:0/1806107240 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c080a46e0 con 0x561f180004f0 2024-05-09T08:43:14.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.867+0000 7f1c28bfb740 1 -- 172.21.15.107:0/1806107240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f180004f0 msgr2=0x561f180008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:14.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.867+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/1806107240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f180004f0 0x561f180008d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1c08009230 tx=0x7f1c0809c170 comp rx=0 tx=0).stop 2024-05-09T08:43:14.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.867+0000 7f1c28bfb740 1 -- 172.21.15.107:0/1806107240 shutdown_connections 2024-05-09T08:43:14.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.867+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/1806107240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f1800b4e0 0x561f1800d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.867+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/1806107240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f18000e10 0x561f1800afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.867+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/1806107240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f180004f0 0x561f180008d0 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1c08009230 tx=0x7f1c0809c170 comp rx=0 tx=0).stop 2024-05-09T08:43:14.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.867+0000 7f1c28bfb740 1 -- 172.21.15.107:0/1806107240 >> 172.21.15.107:0/1806107240 conn(0x561f17ffc810 msgr2=0x561f17ffcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:14.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.873+0000 7f1c28bfb740 1 -- 172.21.15.107:0/1806107240 shutdown_connections 2024-05-09T08:43:14.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.873+0000 7f1c28bfb740 1 -- 172.21.15.107:0/1806107240 wait complete. 2024-05-09T08:43:14.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.873+0000 7f1c28bfb740 1 -- start start 2024-05-09T08:43:14.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.874+0000 7f1c28bfb740 1 Processor -- start 2024-05-09T08:43:14.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.874+0000 7f1c28bfb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f18000e10 0x561f18008550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.874+0000 7f1c28bfb740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f1800b4e0 0x561f18008a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.874+0000 7f1c28bfb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f18008fd0 0x561f18017800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.874+0000 7f1c28bfb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561f1800e310 con 0x561f18000e10 2024-05-09T08:43:14.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.874+0000 7f1c28bfb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561f17f20c40 con 0x561f18008fd0 2024-05-09T08:43:14.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.874+0000 7f1c28bfb740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x561f1800e490 con 0x561f1800b4e0 2024-05-09T08:43:14.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.874+0000 7f1c28bfb740 10 client.0 _ll_register_callbacks cb 0x561f17c78be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:14.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.875+0000 7f1c1effd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f1800b4e0 0x561f18008a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:14.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.875+0000 7f1c1effd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f1800b4e0 0x561f18008a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:59988/0 (socket says 172.21.15.107:59988) 2024-05-09T08:43:14.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.875+0000 7f1c1effd640 1 -- 172.21.15.107:0/3719798683 learned_addr learned my addr 172.21.15.107:0/3719798683 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:14.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.875+0000 7f1c1ffff640 1 --2- 172.21.15.107:0/3719798683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f18008fd0 0x561f18017800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:14.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.875+0000 7f1c1effd640 1 -- 172.21.15.107:0/3719798683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f18008fd0 msgr2=0x561f18017800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:14.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.875+0000 7f1c1effd640 1 --2- 172.21.15.107:0/3719798683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f18008fd0 0x561f18017800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.875+0000 7f1c1effd640 1 -- 172.21.15.107:0/3719798683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f18000e10 msgr2=0x561f18008550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:14.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.875+0000 7f1c1effd640 1 --2- 172.21.15.107:0/3719798683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f18000e10 0x561f18008550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.875+0000 7f1c1effd640 1 -- 172.21.15.107:0/3719798683 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f17d437d0 con 0x561f1800b4e0 2024-05-09T08:43:14.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.876+0000 7f1c1effd640 1 --2- 172.21.15.107:0/3719798683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f1800b4e0 0x561f18008a90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1c10033000 tx=0x7f1c1002a790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fe61c56990528ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:14.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.876+0000 7f1bfeffd640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:43:14.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.876+0000 7f1bfeffd640 1 -- 172.21.15.107:0/3719798683 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1004c020 con 0x561f1800b4e0 2024-05-09T08:43:14.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.876+0000 7f1c28bfb740 1 -- 172.21.15.107:0/3719798683 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c080a46e0 con 0x561f1800b4e0 2024-05-09T08:43:14.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.876+0000 7f1bfeffd640 1 -- 172.21.15.107:0/3719798683 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f1c10031850 con 0x561f1800b4e0 2024-05-09T08:43:14.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.877+0000 7f1bfeffd640 1 -- 172.21.15.107:0/3719798683 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1c10057050 con 0x561f1800b4e0 2024-05-09T08:43:14.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.877+0000 7f1c28bfb740 10 client.27628 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:14.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.877+0000 7f1c28bfb740 1 -- 172.21.15.107:0/3719798683 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561f1801ab20 con 0x561f1800b4e0 2024-05-09T08:43:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.879+0000 7f1bfeffd640 1 -- 172.21.15.107:0/3719798683 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c080a46e0 con 0x561f1800b4e0 2024-05-09T08:43:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.880+0000 7f1bfeffd640 1 -- 172.21.15.107:0/3719798683 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f1c1003c070 con 0x561f1800b4e0 2024-05-09T08:43:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.881+0000 7f1c28bfb740 10 client.27628 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.881+0000 7f1c28bfb740 10 client.27628 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:43:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.881+0000 7f1c28bfb740 20 client.27628 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.881+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/3719798683 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x561f1801c030 0x561f1801e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.881+0000 7f1c28bfb740 4 client.27628 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:14.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.881+0000 7f1c28bfb740 1 -- 172.21.15.107:0/3719798683 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f17f73370 con 0x561f1801c030 2024-05-09T08:43:14.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.884+0000 7f1c1f7fe640 1 --2- 172.21.15.107:0/3719798683 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x561f1801c030 0x561f1801e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:14.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.890+0000 7f1c1f7fe640 1 --2- 172.21.15.107:0/3719798683 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x561f1801c030 0x561f1801e4f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c1a27fdda06aa5a3 server_cookie=49741105a0440e08 in_seq=0 out_seq=0 2024-05-09T08:43:14.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.891+0000 7f1bfeffd640 10 client.27628 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:43:14.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.892+0000 7f1bfeffd640 1 -- 172.21.15.107:0/3719798683 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561f17f73370 con 0x561f1801c030 2024-05-09T08:43:14.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.892+0000 7f1bfeffd640 10 client.27628 handle_command_reply: tid=0 2024-05-09T08:43:14.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.894+0000 7f1c28bfb740 1 client.27628 shutdown 2024-05-09T08:43:14.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.897+0000 7f1c28bfb740 1 -- 172.21.15.107:0/3719798683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f1800b4e0 msgr2=0x561f18008a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:14.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.897+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/3719798683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f1800b4e0 0x561f18008a90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1c10033000 tx=0x7f1c1002a790 comp rx=0 tx=0).stop 2024-05-09T08:43:14.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.900+0000 7f1c28bfb740 1 -- 172.21.15.107:0/3719798683 shutdown_connections 2024-05-09T08:43:14.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.900+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/3719798683 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x561f1801c030 0x561f1801e4f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.900+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/3719798683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f18008fd0 0x561f18017800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.900+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/3719798683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f1800b4e0 0x561f18008a90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.900+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/3719798683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f18000e10 0x561f18008550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.900+0000 7f1c28bfb740 1 -- 172.21.15.107:0/3719798683 >> 172.21.15.107:0/3719798683 conn(0x561f17ffc810 msgr2=0x561f180024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:14.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.900+0000 7f1bfeffd640 0 client.27628 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:43:14.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.902+0000 7f1c28bfb740 1 -- 172.21.15.107:0/3719798683 shutdown_connections 2024-05-09T08:43:14.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.904+0000 7f1c28bfb740 1 -- 172.21.15.107:0/3719798683 wait complete. 2024-05-09T08:43:14.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.905+0000 7f1c28bfb740 20 client.27628 trim_cache size 0 max 16384 2024-05-09T08:43:14.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.952+0000 7f1c28bfb740 1 Processor -- start 2024-05-09T08:43:14.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.956+0000 7f1c28bfb740 1 -- start start 2024-05-09T08:43:14.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.956+0000 7f1c28bfb740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f18000e10 0x561f18007530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.956+0000 7f1c28bfb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f18008fd0 0x561f18023e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.956+0000 7f1c28bfb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f1800b4e0 0x561f18024340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.956+0000 7f1c28bfb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1c000128e0 con 0x561f1800b4e0 2024-05-09T08:43:14.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.956+0000 7f1c28bfb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1c000101b0 con 0x561f18008fd0 2024-05-09T08:43:14.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.956+0000 7f1c28bfb740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0000e0b0 con 0x561f18000e10 2024-05-09T08:43:14.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.956+0000 7f1c1effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f18008fd0 0x561f18023e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:14.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.956+0000 7f1c1effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f18008fd0 0x561f18023e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:45846/0 (socket says 172.21.15.107:45846) 2024-05-09T08:43:14.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.956+0000 7f1c1effd640 1 -- 172.21.15.107:0/2222206281 learned_addr learned my addr 172.21.15.107:0/2222206281 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:14.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.956+0000 7f1c1f7fe640 1 --2- 172.21.15.107:0/2222206281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f18000e10 0x561f18007530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:14.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.958+0000 7f1c1ffff640 1 --2- 172.21.15.107:0/2222206281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f1800b4e0 0x561f18024340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:14.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.959+0000 7f1c1ffff640 1 -- 172.21.15.107:0/2222206281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f18000e10 msgr2=0x561f18007530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:14.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.959+0000 7f1c1ffff640 1 --2- 172.21.15.107:0/2222206281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f18000e10 0x561f18007530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.959+0000 7f1c1ffff640 1 -- 172.21.15.107:0/2222206281 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f18008fd0 msgr2=0x561f18023e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:14.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.959+0000 7f1c1ffff640 1 --2- 172.21.15.107:0/2222206281 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f18008fd0 0x561f18023e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.959+0000 7f1c1ffff640 1 -- 172.21.15.107:0/2222206281 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f18006aa0 con 0x561f1800b4e0 2024-05-09T08:43:14.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.959+0000 7f1c1ffff640 1 --2- 172.21.15.107:0/2222206281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f1800b4e0 0x561f18024340 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f1c18004530 tx=0x7f1c180039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac7e6bbc8dcac88 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.961+0000 7f1c157fa640 1 -- 172.21.15.107:0/2222206281 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1800b760 con 0x561f1800b4e0 2024-05-09T08:43:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.961+0000 7f1c157fa640 1 -- 172.21.15.107:0/2222206281 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c18015070 con 0x561f1800b4e0 2024-05-09T08:43:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.961+0000 7f1c157fa640 1 -- 172.21.15.107:0/2222206281 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c18011d40 con 0x561f1800b4e0 2024-05-09T08:43:14.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.964+0000 7f1c28bfb740 1 -- 172.21.15.107:0/2222206281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f1800b4e0 msgr2=0x561f18024340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:14.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.964+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/2222206281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f1800b4e0 0x561f18024340 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f1c18004530 tx=0x7f1c180039c0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.965+0000 7f1c28bfb740 1 -- 172.21.15.107:0/2222206281 shutdown_connections 2024-05-09T08:43:14.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.965+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/2222206281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f1800b4e0 0x561f18024340 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.965+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/2222206281 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f18008fd0 0x561f18023e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.965+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/2222206281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f18000e10 0x561f18007530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.965+0000 7f1c28bfb740 1 -- 172.21.15.107:0/2222206281 >> 172.21.15.107:0/2222206281 conn(0x561f17ffc810 msgr2=0x561f18006e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:14.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.969+0000 7f1c28bfb740 1 -- 172.21.15.107:0/2222206281 shutdown_connections 2024-05-09T08:43:14.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.974+0000 7f1c28bfb740 1 -- 172.21.15.107:0/2222206281 wait complete. 2024-05-09T08:43:14.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.974+0000 7f1c28bfb740 1 -- start start 2024-05-09T08:43:14.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.974+0000 7f1c28bfb740 1 Processor -- start 2024-05-09T08:43:14.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.978+0000 7f1c28bfb740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f18000e10 0x561f1800f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.978+0000 7f1c28bfb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f18008fd0 0x561f1800f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.978+0000 7f1c28bfb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f1800b4e0 0x561f1800fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.978+0000 7f1c28bfb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0000e670 con 0x561f18008fd0 2024-05-09T08:43:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.978+0000 7f1c28bfb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0000fae0 con 0x561f1800b4e0 2024-05-09T08:43:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.978+0000 7f1c28bfb740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0000ed40 con 0x561f18000e10 2024-05-09T08:43:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.978+0000 7f1c28bfb740 10 client.0 _ll_register_callbacks cb 0x7f1c10034070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.979+0000 7f1c1f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f18000e10 0x561f1800f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.980+0000 7f1c1f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f18000e10 0x561f1800f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:60016/0 (socket says 172.21.15.107:60016) 2024-05-09T08:43:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.980+0000 7f1c1f7fe640 1 -- 172.21.15.107:0/3542476772 learned_addr learned my addr 172.21.15.107:0/3542476772 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:14.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.980+0000 7f1c1f7fe640 1 -- 172.21.15.107:0/3542476772 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f1800b4e0 msgr2=0x561f1800fd50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:43:14.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.980+0000 7f1c1f7fe640 1 --2- 172.21.15.107:0/3542476772 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f1800b4e0 0x561f1800fd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.980+0000 7f1c1f7fe640 1 -- 172.21.15.107:0/3542476772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f18008fd0 msgr2=0x561f1800f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:14.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.980+0000 7f1c1f7fe640 1 --2- 172.21.15.107:0/3542476772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f18008fd0 0x561f1800f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:14.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.980+0000 7f1c1f7fe640 1 -- 172.21.15.107:0/3542476772 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f180817d0 con 0x561f18000e10 2024-05-09T08:43:14.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.980+0000 7f1c1f7fe640 1 --2- 172.21.15.107:0/3542476772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f18000e10 0x561f1800f2d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1c08004ad0 tx=0x7f1c0800e940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=178fb48258e94f0b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:14.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.980+0000 7f1bfeffd640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:43:14.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.983+0000 7f1bfeffd640 1 -- 172.21.15.107:0/3542476772 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c080b3040 con 0x561f18000e10 2024-05-09T08:43:14.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.983+0000 7f1bfeffd640 1 -- 172.21.15.107:0/3542476772 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f1c080ba340 con 0x561f18000e10 2024-05-09T08:43:14.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.986+0000 7f1c28bfb740 1 -- 172.21.15.107:0/3542476772 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c18015070 con 0x561f18000e10 2024-05-09T08:43:14.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.989+0000 7f1bfeffd640 1 -- 172.21.15.107:0/3542476772 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1c0809b840 con 0x561f18000e10 2024-05-09T08:43:14.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.991+0000 7f1c28bfb740 10 client.27634 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:14.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.991+0000 7f1c28bfb740 1 -- 172.21.15.107:0/3542476772 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561f18082b10 con 0x561f18000e10 2024-05-09T08:43:14.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.996+0000 7f1bfeffd640 1 -- 172.21.15.107:0/3542476772 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c18015070 con 0x561f18000e10 2024-05-09T08:43:14.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:14.996+0000 7f1bfeffd640 1 -- 172.21.15.107:0/3542476772 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f1c0809a930 con 0x561f18000e10 2024-05-09T08:43:15.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.003+0000 7f1c28bfb740 10 client.27634 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:15.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.003+0000 7f1c28bfb740 10 client.27634 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:43:15.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.003+0000 7f1c28bfb740 20 client.27634 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:15.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.003+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/3542476772 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x561f18082e00 0x561f180851f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.003+0000 7f1c28bfb740 4 client.27634 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:43:15.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.003+0000 7f1c28bfb740 1 -- 172.21.15.107:0/3542476772 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561f17f6d6c0 con 0x561f18082e00 2024-05-09T08:43:15.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.006+0000 7f1c1effd640 1 --2- 172.21.15.107:0/3542476772 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x561f18082e00 0x561f180851f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.012+0000 7f1c1effd640 1 --2- 172.21.15.107:0/3542476772 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x561f18082e00 0x561f180851f0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ebd6a111007a05f6 server_cookie=f2b06d3aeee88f21 in_seq=0 out_seq=0 2024-05-09T08:43:15.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.012+0000 7f1bfeffd640 10 client.27634 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:43:15.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.012+0000 7f1bfeffd640 1 -- 172.21.15.107:0/3542476772 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561f17f6d6c0 con 0x561f18082e00 2024-05-09T08:43:15.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.013+0000 7f1bfeffd640 10 client.27634 handle_command_reply: tid=0 2024-05-09T08:43:15.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.013+0000 7f1c28bfb740 1 client.27634 shutdown 2024-05-09T08:43:15.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.029+0000 7f1c28bfb740 1 -- 172.21.15.107:0/3542476772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f18000e10 msgr2=0x561f1800f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.029+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/3542476772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f18000e10 0x561f1800f2d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1c08004ad0 tx=0x7f1c0800e940 comp rx=0 tx=0).stop 2024-05-09T08:43:15.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.029+0000 7f1c28bfb740 1 -- 172.21.15.107:0/3542476772 shutdown_connections 2024-05-09T08:43:15.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.029+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/3542476772 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x561f18082e00 0x561f180851f0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.029+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/3542476772 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f1800b4e0 0x561f1800fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.029+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/3542476772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f18008fd0 0x561f1800f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.029+0000 7f1c28bfb740 1 --2- 172.21.15.107:0/3542476772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f18000e10 0x561f1800f2d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.029+0000 7f1c28bfb740 1 -- 172.21.15.107:0/3542476772 >> 172.21.15.107:0/3542476772 conn(0x561f17ffc810 msgr2=0x561f17ffb990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:15.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.029+0000 7f1bfeffd640 0 client.27634 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:43:15.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.029+0000 7f1c28bfb740 1 -- 172.21.15.107:0/3542476772 shutdown_connections 2024-05-09T08:43:15.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.029+0000 7f1c28bfb740 1 -- 172.21.15.107:0/3542476772 wait complete. 2024-05-09T08:43:15.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.030+0000 7f1c28bfb740 20 client.27634 trim_cache size 0 max 16384 2024-05-09T08:43:15.034 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:43:15.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.037+0000 7f1c24bb0640 1 -- 172.21.15.107:0/2865344993 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c0c0a2f30 msgr2=0x7f1c0c0a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.037+0000 7f1c24bb0640 1 --2- 172.21.15.107:0/2865344993 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c0c0a2f30 0x7f1c0c0a53f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1c10009b30 tx=0x7f1c10000f40 comp rx=0 tx=0).stop 2024-05-09T08:43:15.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.037+0000 7f1c24bb0640 1 -- 172.21.15.107:0/2865344993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c20108830 msgr2=0x7f1c2011ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.037+0000 7f1c24bb0640 1 --2- 172.21.15.107:0/2865344993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c20108830 0x7f1c2011ccb0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800e9c0 tx=0x7f1c0800ee80 comp rx=0 tx=0).stop 2024-05-09T08:43:15.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.038+0000 7f1c24bb0640 1 -- 172.21.15.107:0/2865344993 shutdown_connections 2024-05-09T08:43:15.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.038+0000 7f1c24bb0640 1 --2- 172.21.15.107:0/2865344993 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c0c0a2f30 0x7f1c0c0a53f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.038+0000 7f1c24bb0640 1 --2- 172.21.15.107:0/2865344993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c2010fed0 0x7f1c201162c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.038+0000 7f1c24bb0640 1 --2- 172.21.15.107:0/2865344993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c201092b0 0x7f1c20115d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.038+0000 7f1c24bb0640 1 --2- 172.21.15.107:0/2865344993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c20108830 0x7f1c2011ccb0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.038+0000 7f1c24bb0640 1 -- 172.21.15.107:0/2865344993 >> 172.21.15.107:0/2865344993 conn(0x7f1c2006c7b0 msgr2=0x7f1c20071950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:15.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.042+0000 7f1c24bb0640 1 -- 172.21.15.107:0/2865344993 shutdown_connections 2024-05-09T08:43:15.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.044+0000 7f1c24bb0640 1 -- 172.21.15.107:0/2865344993 wait complete. 2024-05-09T08:43:15.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:43:15.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:15 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/4199146356' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:43:15.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.364+0000 7f1c309c5640 1 -- 172.21.15.107:0/4107091953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c2c100120 msgr2=0x7f1c2c100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.364+0000 7f1c309c5640 1 --2- 172.21.15.107:0/4107091953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c2c100120 0x7f1c2c100580 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1c24009900 tx=0x7f1c2402a4d0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.370+0000 7f1c309c5640 1 -- 172.21.15.107:0/4107091953 shutdown_connections 2024-05-09T08:43:15.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.370+0000 7f1c309c5640 1 --2- 172.21.15.107:0/4107091953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c2c1123c0 0x7f1c2c1147b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.370+0000 7f1c309c5640 1 --2- 172.21.15.107:0/4107091953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c2c100120 0x7f1c2c100580 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.370+0000 7f1c309c5640 1 --2- 172.21.15.107:0/4107091953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c2c1076d0 0x7f1c2c0ffbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.370+0000 7f1c309c5640 1 -- 172.21.15.107:0/4107091953 >> 172.21.15.107:0/4107091953 conn(0x7f1c2c075530 msgr2=0x7f1c2c075940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:15.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.371+0000 7f1c309c5640 1 -- 172.21.15.107:0/4107091953 shutdown_connections 2024-05-09T08:43:15.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.371+0000 7f1c309c5640 1 -- 172.21.15.107:0/4107091953 wait complete. 2024-05-09T08:43:15.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.372+0000 7f1c309c5640 1 Processor -- start 2024-05-09T08:43:15.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.373+0000 7f1c309c5640 1 -- start start 2024-05-09T08:43:15.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.373+0000 7f1c309c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c2c100120 0x7f1c2c10f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.373+0000 7f1c309c5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c2c1076d0 0x7f1c2c10fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.373+0000 7f1c309c5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c2c1123c0 0x7f1c2c1089c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.373+0000 7f1c309c5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c1180b0 con 0x7f1c2c100120 2024-05-09T08:43:15.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.373+0000 7f1c309c5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c117f30 con 0x7f1c2c1076d0 2024-05-09T08:43:15.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.373+0000 7f1c309c5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c118230 con 0x7f1c2c1123c0 2024-05-09T08:43:15.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.377+0000 7f1c2affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c2c1076d0 0x7f1c2c10fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.377+0000 7f1c2affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c2c1076d0 0x7f1c2c10fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:45860/0 (socket says 172.21.15.107:45860) 2024-05-09T08:43:15.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.377+0000 7f1c2affd640 1 -- 172.21.15.107:0/2977165435 learned_addr learned my addr 172.21.15.107:0/2977165435 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:15.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.377+0000 7f1c2bfff640 1 --2- 172.21.15.107:0/2977165435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c2c1123c0 0x7f1c2c1089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.377+0000 7f1c2affd640 1 -- 172.21.15.107:0/2977165435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c2c1123c0 msgr2=0x7f1c2c1089c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.377+0000 7f1c2affd640 1 --2- 172.21.15.107:0/2977165435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c2c1123c0 0x7f1c2c1089c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.377+0000 7f1c2affd640 1 -- 172.21.15.107:0/2977165435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c2c100120 msgr2=0x7f1c2c10f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.377+0000 7f1c2affd640 1 --2- 172.21.15.107:0/2977165435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c2c100120 0x7f1c2c10f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.377+0000 7f1c2affd640 1 -- 172.21.15.107:0/2977165435 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c2c109280 con 0x7f1c2c1076d0 2024-05-09T08:43:15.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.378+0000 7f1c2affd640 1 --2- 172.21.15.107:0/2977165435 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c2c1076d0 0x7f1c2c10fd60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1c2402aac0 tx=0x7f1c240306c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:15.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.379+0000 7f1c28ff9640 1 -- 172.21.15.107:0/2977165435 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2403f070 con 0x7f1c2c1076d0 2024-05-09T08:43:15.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.379+0000 7f1c28ff9640 1 -- 172.21.15.107:0/2977165435 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c24004440 con 0x7f1c2c1076d0 2024-05-09T08:43:15.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.379+0000 7f1c28ff9640 1 -- 172.21.15.107:0/2977165435 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c24035070 con 0x7f1c2c1076d0 2024-05-09T08:43:15.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.379+0000 7f1c309c5640 1 -- 172.21.15.107:0/2977165435 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c2c109510 con 0x7f1c2c1076d0 2024-05-09T08:43:15.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.379+0000 7f1c309c5640 1 -- 172.21.15.107:0/2977165435 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c2c10d520 con 0x7f1c2c1076d0 2024-05-09T08:43:15.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.384+0000 7f1c34a0d740 1 Processor -- start 2024-05-09T08:43:15.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.384+0000 7f1c34a0d740 1 -- start start 2024-05-09T08:43:15.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.384+0000 7f1c34a0d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6ca9f8520 0x55a6ca9f8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.384+0000 7f1c34a0d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8e40 0x55a6caa02fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.384+0000 7f1c34a0d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6caa03510 0x55a6caa05900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.384+0000 7f1c34a0d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a6ca95dc40 con 0x55a6caa03510 2024-05-09T08:43:15.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.384+0000 7f1c34a0d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a6ca905580 con 0x55a6ca9f8520 2024-05-09T08:43:15.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.384+0000 7f1c34a0d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a6ca959c00 con 0x55a6ca9f8e40 2024-05-09T08:43:15.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.386+0000 7f1c2bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6ca9f8520 0x55a6ca9f8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.386+0000 7f1c2bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6ca9f8520 0x55a6ca9f8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:45862/0 (socket says 172.21.15.107:45862) 2024-05-09T08:43:15.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.386+0000 7f1c2bfff640 1 -- 172.21.15.107:0/1083797169 learned_addr learned my addr 172.21.15.107:0/1083797169 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:15.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.387+0000 7f1c2bfff640 1 -- 172.21.15.107:0/1083797169 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8e40 msgr2=0x55a6caa02fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:43:15.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.387+0000 7f1c2bfff640 1 --2- 172.21.15.107:0/1083797169 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8e40 0x55a6caa02fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.387+0000 7f1c2bfff640 1 -- 172.21.15.107:0/1083797169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6caa03510 msgr2=0x55a6caa05900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:15.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.387+0000 7f1c2bfff640 1 --2- 172.21.15.107:0/1083797169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6caa03510 0x55a6caa05900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.387+0000 7f1c2bfff640 1 -- 172.21.15.107:0/1083797169 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6ca6713f0 con 0x55a6ca9f8520 2024-05-09T08:43:15.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.387+0000 7f1c28ff9640 1 -- 172.21.15.107:0/2977165435 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1c24042070 con 0x7f1c2c1076d0 2024-05-09T08:43:15.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.388+0000 7f1c2bfff640 1 --2- 172.21.15.107:0/1083797169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6ca9f8520 0x55a6ca9f8900 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400a840 tx=0x7f1c1400ad00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3bb994af3af7cd3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:15.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.388+0000 7f1c28ff9640 1 --2- 172.21.15.107:0/2977165435 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c180a2ed0 0x7f1c180a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.389+0000 7f1c28ff9640 1 -- 172.21.15.107:0/2977165435 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f1c240f2440 con 0x7f1c2c1076d0 2024-05-09T08:43:15.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.395+0000 7f1c2b7fe640 1 --2- 172.21.15.107:0/2977165435 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c180a2ed0 0x7f1c180a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.396+0000 7f1c21ffb640 1 -- 172.21.15.107:0/1083797169 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14015070 con 0x55a6ca9f8520 2024-05-09T08:43:15.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.396+0000 7f1c21ffb640 1 -- 172.21.15.107:0/1083797169 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c140113f0 con 0x55a6ca9f8520 2024-05-09T08:43:15.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.396+0000 7f1c2b7fe640 1 --2- 172.21.15.107:0/2977165435 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c180a2ed0 0x7f1c180a5390 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x55a6ca9f9ae0 tx=0x7f1c1c009230 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:15.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.397+0000 7f1c34a0d740 1 -- 172.21.15.107:0/1083797169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6ca9f8520 msgr2=0x55a6ca9f8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.397+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/1083797169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6ca9f8520 0x55a6ca9f8900 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400a840 tx=0x7f1c1400ad00 comp rx=0 tx=0).stop 2024-05-09T08:43:15.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.401+0000 7f1c34a0d740 1 -- 172.21.15.107:0/1083797169 shutdown_connections 2024-05-09T08:43:15.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.401+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/1083797169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6caa03510 0x55a6caa05900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.401+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/1083797169 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8e40 0x55a6caa02fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.401+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/1083797169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6ca9f8520 0x55a6ca9f8900 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.401+0000 7f1c34a0d740 1 -- 172.21.15.107:0/1083797169 >> 172.21.15.107:0/1083797169 conn(0x55a6ca9f4840 msgr2=0x55a6ca9f4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:15.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.401+0000 7f1c34a0d740 1 -- 172.21.15.107:0/1083797169 shutdown_connections 2024-05-09T08:43:15.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.402+0000 7f1c34a0d740 1 -- 172.21.15.107:0/1083797169 wait complete. 2024-05-09T08:43:15.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.402+0000 7f1c34a0d740 1 -- start start 2024-05-09T08:43:15.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.403+0000 7f1c34a0d740 1 Processor -- start 2024-05-09T08:43:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.403+0000 7f1c34a0d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6ca9f8520 0x55a6caa003a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.403+0000 7f1c34a0d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8e40 0x55a6caa008e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.403+0000 7f1c34a0d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6caa03510 0x55a6caa00e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.403+0000 7f1c34a0d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a6caa06350 con 0x55a6ca9f8520 2024-05-09T08:43:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.403+0000 7f1c34a0d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a6ca918c40 con 0x55a6caa03510 2024-05-09T08:43:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.403+0000 7f1c34a0d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a6caa064d0 con 0x55a6ca9f8e40 2024-05-09T08:43:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.403+0000 7f1c34a0d740 10 client.0 _ll_register_callbacks cb 0x55a6ca670be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.405+0000 7f1c2b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6ca9f8520 0x55a6caa003a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.405+0000 7f1c2b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6ca9f8520 0x55a6caa003a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55000/0 (socket says 172.21.15.107:55000) 2024-05-09T08:43:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.405+0000 7f1c2b7fe640 1 -- 172.21.15.107:0/1165392013 learned_addr learned my addr 172.21.15.107:0/1165392013 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.406+0000 7f1c2b7fe640 1 -- 172.21.15.107:0/1165392013 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8e40 msgr2=0x55a6caa008e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.406+0000 7f1c2b7fe640 1 --2- 172.21.15.107:0/1165392013 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8e40 0x55a6caa008e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.406+0000 7f1c2b7fe640 1 -- 172.21.15.107:0/1165392013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6caa03510 msgr2=0x55a6caa00e20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:43:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.406+0000 7f1c2b7fe640 1 --2- 172.21.15.107:0/1165392013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6caa03510 0x55a6caa00e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.406+0000 7f1c2b7fe640 1 -- 172.21.15.107:0/1165392013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6ca73b7d0 con 0x55a6ca9f8520 2024-05-09T08:43:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.406+0000 7f1c2b7fe640 1 --2- 172.21.15.107:0/1165392013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6ca9f8520 0x55a6caa003a0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c00de00 tx=0x7f1c1c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b01444f7454d805 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.406+0000 7f1c0affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:43:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.406+0000 7f1c0affd640 1 -- 172.21.15.107:0/1165392013 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c00ce10 con 0x55a6ca9f8520 2024-05-09T08:43:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.406+0000 7f1c34a0d740 1 -- 172.21.15.107:0/1165392013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c140113f0 con 0x55a6ca9f8520 2024-05-09T08:43:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.407+0000 7f1c0affd640 1 -- 172.21.15.107:0/1165392013 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f1c1c023480 con 0x55a6ca9f8520 2024-05-09T08:43:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.407+0000 7f1c0affd640 1 -- 172.21.15.107:0/1165392013 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1c1c0236c0 con 0x55a6ca9f8520 2024-05-09T08:43:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.410+0000 7f1c34a0d740 10 client.18444 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.410+0000 7f1c34a0d740 1 -- 172.21.15.107:0/1165392013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a6caa12550 con 0x55a6ca9f8520 2024-05-09T08:43:15.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.411+0000 7f1c0affd640 1 -- 172.21.15.107:0/1165392013 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c140113f0 con 0x55a6ca9f8520 2024-05-09T08:43:15.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.411+0000 7f1c0affd640 1 -- 172.21.15.107:0/1165392013 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f1c1c02db30 con 0x55a6ca9f8520 2024-05-09T08:43:15.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.411+0000 7f1c34a0d740 10 client.18444 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:15.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.411+0000 7f1c34a0d740 10 client.18444 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:43:15.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.411+0000 7f1c34a0d740 20 client.18444 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:15.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.412+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/1165392013 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a6caa13e70 0x55a6caa162e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.412+0000 7f1c34a0d740 4 client.18444 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:15.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.412+0000 7f1c34a0d740 1 -- 172.21.15.107:0/1165392013 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6ca96b370 con 0x55a6caa13e70 2024-05-09T08:43:15.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.413+0000 7f1c2affd640 1 --2- 172.21.15.107:0/1165392013 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a6caa13e70 0x55a6caa162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.414+0000 7f1c2affd640 1 --2- 172.21.15.107:0/1165392013 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a6caa13e70 0x55a6caa162e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d52aa5eff1413285 server_cookie=f05ac3faec89e03c in_seq=0 out_seq=0 2024-05-09T08:43:15.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.415+0000 7f1c0affd640 10 client.18444 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:43:15.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.417+0000 7f1c0affd640 1 -- 172.21.15.107:0/1165392013 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a6ca96b370 con 0x55a6caa13e70 2024-05-09T08:43:15.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.417+0000 7f1c0affd640 10 client.18444 handle_command_reply: tid=0 2024-05-09T08:43:15.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.417+0000 7f1c34a0d740 1 client.18444 shutdown 2024-05-09T08:43:15.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.418+0000 7f1c34a0d740 1 -- 172.21.15.107:0/1165392013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6ca9f8520 msgr2=0x55a6caa003a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.418+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/1165392013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6ca9f8520 0x55a6caa003a0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c00de00 tx=0x7f1c1c004b40 comp rx=0 tx=0).stop 2024-05-09T08:43:15.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.418+0000 7f1c34a0d740 1 -- 172.21.15.107:0/1165392013 shutdown_connections 2024-05-09T08:43:15.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.418+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/1165392013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6caa03510 0x55a6caa00e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.418+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/1165392013 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a6caa13e70 0x55a6caa162e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.418+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/1165392013 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8e40 0x55a6caa008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.418+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/1165392013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6ca9f8520 0x55a6caa003a0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.418+0000 7f1c34a0d740 1 -- 172.21.15.107:0/1165392013 >> 172.21.15.107:0/1165392013 conn(0x55a6ca9f4840 msgr2=0x55a6ca9fa640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:15.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.419+0000 7f1c0affd640 0 client.18444 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:43:15.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.419+0000 7f1c34a0d740 1 -- 172.21.15.107:0/1165392013 shutdown_connections 2024-05-09T08:43:15.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.419+0000 7f1c34a0d740 1 -- 172.21.15.107:0/1165392013 wait complete. 2024-05-09T08:43:15.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.423+0000 7f1c34a0d740 20 client.18444 trim_cache size 0 max 16384 2024-05-09T08:43:15.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.462+0000 7f1c34a0d740 1 Processor -- start 2024-05-09T08:43:15.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.464+0000 7f1c34a0d740 1 -- start start 2024-05-09T08:43:15.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.464+0000 7f1c34a0d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6ca9f8520 0x55a6caa0f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.464+0000 7f1c34a0d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8e40 0x55a6ca9ff380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.464+0000 7f1c34a0d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6caa03510 0x55a6caa1bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.464+0000 7f1c34a0d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c06bda0 con 0x55a6caa03510 2024-05-09T08:43:15.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.464+0000 7f1c34a0d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c1ac9b0 con 0x55a6ca9f8520 2024-05-09T08:43:15.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.464+0000 7f1c34a0d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c1aa820 con 0x55a6ca9f8e40 2024-05-09T08:43:15.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.467+0000 7f1c2b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6ca9f8520 0x55a6caa0f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.467+0000 7f1c2b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6ca9f8520 0x55a6caa0f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:45886/0 (socket says 172.21.15.107:45886) 2024-05-09T08:43:15.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.467+0000 7f1c2b7fe640 1 -- 172.21.15.107:0/2051217779 learned_addr learned my addr 172.21.15.107:0/2051217779 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:15.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.467+0000 7f1c2affd640 1 --2- 172.21.15.107:0/2051217779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8e40 0x55a6ca9ff380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.467+0000 7f1c2affd640 1 -- 172.21.15.107:0/2051217779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6ca9f8520 msgr2=0x55a6caa0f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:15.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.467+0000 7f1c2affd640 1 --2- 172.21.15.107:0/2051217779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6ca9f8520 0x55a6caa0f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.467+0000 7f1c2affd640 1 -- 172.21.15.107:0/2051217779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6caa03510 msgr2=0x55a6caa1bbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:43:15.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.467+0000 7f1c2affd640 1 --2- 172.21.15.107:0/2051217779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6caa03510 0x55a6caa1bbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.467+0000 7f1c2affd640 1 -- 172.21.15.107:0/2051217779 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6caa16650 con 0x55a6ca9f8e40 2024-05-09T08:43:15.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.472+0000 7f1c2affd640 1 --2- 172.21.15.107:0/2051217779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8e40 0x55a6ca9ff380 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1c240ef720 tx=0x7f1c24005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13055c87bde6ac7a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:15.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.473+0000 7f1c217fa640 1 -- 172.21.15.107:0/2051217779 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c240035f0 con 0x55a6ca9f8e40 2024-05-09T08:43:15.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.473+0000 7f1c217fa640 1 -- 172.21.15.107:0/2051217779 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c240d1070 con 0x55a6ca9f8e40 2024-05-09T08:43:15.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.473+0000 7f1c217fa640 1 -- 172.21.15.107:0/2051217779 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c240bc7f0 con 0x55a6ca9f8e40 2024-05-09T08:43:15.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.474+0000 7f1c34a0d740 1 -- 172.21.15.107:0/2051217779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8e40 msgr2=0x55a6ca9ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.475+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/2051217779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8e40 0x55a6ca9ff380 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1c240ef720 tx=0x7f1c24005b50 comp rx=0 tx=0).stop 2024-05-09T08:43:15.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.475+0000 7f1c34a0d740 1 -- 172.21.15.107:0/2051217779 shutdown_connections 2024-05-09T08:43:15.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.475+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/2051217779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6caa03510 0x55a6caa1bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.475+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/2051217779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8e40 0x55a6ca9ff380 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.475+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/2051217779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6ca9f8520 0x55a6caa0f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.475+0000 7f1c34a0d740 1 -- 172.21.15.107:0/2051217779 >> 172.21.15.107:0/2051217779 conn(0x55a6ca9f4840 msgr2=0x55a6ca9fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:15.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.475+0000 7f1c34a0d740 1 -- 172.21.15.107:0/2051217779 shutdown_connections 2024-05-09T08:43:15.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.488+0000 7f1c34a0d740 1 -- 172.21.15.107:0/2051217779 wait complete. 2024-05-09T08:43:15.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.488+0000 7f1c34a0d740 1 -- start start 2024-05-09T08:43:15.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.488+0000 7f1c34a0d740 1 Processor -- start 2024-05-09T08:43:15.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.489+0000 7f1c34a0d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8520 0x55a6caa06920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.489+0000 7f1c34a0d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6ca9f8e40 0x55a6caa06e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.489+0000 7f1c34a0d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6caa03510 0x55a6caa78450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.489+0000 7f1c34a0d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c1aae70 con 0x55a6ca9f8e40 2024-05-09T08:43:15.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.489+0000 7f1c34a0d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c1ac2e0 con 0x55a6caa03510 2024-05-09T08:43:15.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.489+0000 7f1c34a0d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c1ab540 con 0x55a6ca9f8520 2024-05-09T08:43:15.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.489+0000 7f1c34a0d740 10 client.0 _ll_register_callbacks cb 0x7f1c1c00cb30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:15.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.491+0000 7f1c2bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6caa03510 0x55a6caa78450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.492+0000 7f1c2bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6caa03510 0x55a6caa78450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:45892/0 (socket says 172.21.15.107:45892) 2024-05-09T08:43:15.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.492+0000 7f1c2bfff640 1 -- 172.21.15.107:0/3182177612 learned_addr learned my addr 172.21.15.107:0/3182177612 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:15.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.493+0000 7f1c2affd640 1 --2- 172.21.15.107:0/3182177612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6ca9f8e40 0x55a6caa06e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.493+0000 7f1c2b7fe640 1 --2- 172.21.15.107:0/3182177612 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8520 0x55a6caa06920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.493+0000 7f1c2affd640 1 -- 172.21.15.107:0/3182177612 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8520 msgr2=0x55a6caa06920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:15.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.493+0000 7f1c2affd640 1 --2- 172.21.15.107:0/3182177612 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8520 0x55a6caa06920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.493+0000 7f1c2affd640 1 -- 172.21.15.107:0/3182177612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6caa03510 msgr2=0x55a6caa78450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:15.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.493+0000 7f1c2affd640 1 --2- 172.21.15.107:0/3182177612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6caa03510 0x55a6caa78450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.494+0000 7f1c2affd640 1 -- 172.21.15.107:0/3182177612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6caa78990 con 0x55a6ca9f8e40 2024-05-09T08:43:15.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.495+0000 7f1c2bfff640 1 --2- 172.21.15.107:0/3182177612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6caa03510 0x55a6caa78450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:15.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.498+0000 7f1c2affd640 1 --2- 172.21.15.107:0/3182177612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6ca9f8e40 0x55a6caa06e60 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f1c240be1a0 tx=0x7f1c24009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59b8a4676bd952 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:15.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.499+0000 7f1c0affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:43:15.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.499+0000 7f1c0affd640 1 -- 172.21.15.107:0/3182177612 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c240035f0 con 0x55a6ca9f8e40 2024-05-09T08:43:15.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.499+0000 7f1c34a0d740 1 -- 172.21.15.107:0/3182177612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c240d1070 con 0x55a6ca9f8e40 2024-05-09T08:43:15.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.499+0000 7f1c0affd640 1 -- 172.21.15.107:0/3182177612 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f1c240bab60 con 0x55a6ca9f8e40 2024-05-09T08:43:15.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.499+0000 7f1c0affd640 1 -- 172.21.15.107:0/3182177612 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1c240be4f0 con 0x55a6ca9f8e40 2024-05-09T08:43:15.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.500+0000 7f1c34a0d740 10 client.18447 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:15.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.500+0000 7f1c34a0d740 1 -- 172.21.15.107:0/3182177612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a6caa79c70 con 0x55a6ca9f8e40 2024-05-09T08:43:15.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.501+0000 7f1c0affd640 1 -- 172.21.15.107:0/3182177612 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c240d1070 con 0x55a6ca9f8e40 2024-05-09T08:43:15.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.501+0000 7f1c0affd640 1 -- 172.21.15.107:0/3182177612 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f1c240ef9a0 con 0x55a6ca9f8e40 2024-05-09T08:43:15.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.501+0000 7f1c34a0d740 10 client.18447 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:15.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.501+0000 7f1c34a0d740 10 client.18447 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:43:15.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.501+0000 7f1c34a0d740 20 client.18447 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:15.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.501+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/3182177612 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a6caa7a1c0 0x55a6caa7c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.501+0000 7f1c34a0d740 4 client.18447 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:43:15.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.501+0000 7f1c34a0d740 1 -- 172.21.15.107:0/3182177612 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a6ca9656c0 con 0x55a6caa7a1c0 2024-05-09T08:43:15.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.501+0000 7f1c2b7fe640 1 --2- 172.21.15.107:0/3182177612 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a6caa7a1c0 0x55a6caa7c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.502+0000 7f1c2b7fe640 1 --2- 172.21.15.107:0/3182177612 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a6caa7a1c0 0x55a6caa7c5e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7d0ea2c623886967 server_cookie=7a98f3c034f3bd25 in_seq=0 out_seq=0 2024-05-09T08:43:15.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.502+0000 7f1c0affd640 10 client.18447 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:43:15.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.503+0000 7f1c0affd640 1 -- 172.21.15.107:0/3182177612 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a6ca9656c0 con 0x55a6caa7a1c0 2024-05-09T08:43:15.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.503+0000 7f1c0affd640 10 client.18447 handle_command_reply: tid=0 2024-05-09T08:43:15.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.503+0000 7f1c34a0d740 1 client.18447 shutdown 2024-05-09T08:43:15.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.509+0000 7f1c34a0d740 1 -- 172.21.15.107:0/3182177612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6ca9f8e40 msgr2=0x55a6caa06e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.509+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/3182177612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6ca9f8e40 0x55a6caa06e60 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f1c240be1a0 tx=0x7f1c24009680 comp rx=0 tx=0).stop 2024-05-09T08:43:15.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.512+0000 7f1c34a0d740 1 -- 172.21.15.107:0/3182177612 shutdown_connections 2024-05-09T08:43:15.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.512+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/3182177612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a6caa03510 0x55a6caa78450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.512+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/3182177612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6ca9f8e40 0x55a6caa06e60 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.512+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/3182177612 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a6caa7a1c0 0x55a6caa7c5e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.512+0000 7f1c34a0d740 1 --2- 172.21.15.107:0/3182177612 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a6ca9f8520 0x55a6caa06920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.512+0000 7f1c34a0d740 1 -- 172.21.15.107:0/3182177612 >> 172.21.15.107:0/3182177612 conn(0x55a6ca9f4840 msgr2=0x55a6ca9ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:15.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.512+0000 7f1c0affd640 0 client.18447 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:43:15.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.513+0000 7f1c34a0d740 1 -- 172.21.15.107:0/3182177612 shutdown_connections 2024-05-09T08:43:15.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.513+0000 7f1c34a0d740 1 -- 172.21.15.107:0/3182177612 wait complete. 2024-05-09T08:43:15.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.514+0000 7f1c34a0d740 20 client.18447 trim_cache size 0 max 16384 2024-05-09T08:43:15.522 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:43:15.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.519+0000 7f1c309c5640 1 -- 172.21.15.107:0/2977165435 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c180a2ed0 msgr2=0x7f1c180a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.519+0000 7f1c309c5640 1 --2- 172.21.15.107:0/2977165435 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c180a2ed0 0x7f1c180a5390 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x55a6ca9f9ae0 tx=0x7f1c1c009230 comp rx=0 tx=0).stop 2024-05-09T08:43:15.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.519+0000 7f1c309c5640 1 -- 172.21.15.107:0/2977165435 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c2c1076d0 msgr2=0x7f1c2c10fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.519+0000 7f1c309c5640 1 --2- 172.21.15.107:0/2977165435 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c2c1076d0 0x7f1c2c10fd60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1c2402aac0 tx=0x7f1c240306c0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.520+0000 7f1c309c5640 1 -- 172.21.15.107:0/2977165435 shutdown_connections 2024-05-09T08:43:15.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.520+0000 7f1c309c5640 1 --2- 172.21.15.107:0/2977165435 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1c180a2ed0 0x7f1c180a5390 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.520+0000 7f1c309c5640 1 --2- 172.21.15.107:0/2977165435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c2c1123c0 0x7f1c2c1089c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.520+0000 7f1c309c5640 1 --2- 172.21.15.107:0/2977165435 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c2c1076d0 0x7f1c2c10fd60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.520+0000 7f1c309c5640 1 --2- 172.21.15.107:0/2977165435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c2c100120 0x7f1c2c10f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.520+0000 7f1c309c5640 1 -- 172.21.15.107:0/2977165435 >> 172.21.15.107:0/2977165435 conn(0x7f1c2c075530 msgr2=0x7f1c2c105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:15.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.522+0000 7f1c309c5640 1 -- 172.21.15.107:0/2977165435 shutdown_connections 2024-05-09T08:43:15.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.522+0000 7f1c309c5640 1 -- 172.21.15.107:0/2977165435 wait complete. 2024-05-09T08:43:15.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:43:15.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:15 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/4199146356' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:43:15.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.716+0000 7fc9b8eda640 1 -- 172.21.15.107:0/149399462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9b406a7f0 msgr2=0x7fc9b410e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.716+0000 7fc9b8eda640 1 --2- 172.21.15.107:0/149399462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9b406a7f0 0x7fc9b410e6d0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac009cd0 tx=0x7fc9ac02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.716+0000 7fc9b8eda640 1 -- 172.21.15.107:0/149399462 shutdown_connections 2024-05-09T08:43:15.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.716+0000 7fc9b8eda640 1 --2- 172.21.15.107:0/149399462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9b406a7f0 0x7fc9b410e6d0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.716+0000 7fc9b8eda640 1 --2- 172.21.15.107:0/149399462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9b4069e30 0x7fc9b406a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.716+0000 7fc9b8eda640 1 --2- 172.21.15.107:0/149399462 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9b4107810 0x7fc9b4107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.716+0000 7fc9b8eda640 1 -- 172.21.15.107:0/149399462 >> 172.21.15.107:0/149399462 conn(0x7fc9b4076530 msgr2=0x7fc9b4076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:15.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.716+0000 7fc9b8eda640 1 -- 172.21.15.107:0/149399462 shutdown_connections 2024-05-09T08:43:15.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.716+0000 7fc9b8eda640 1 -- 172.21.15.107:0/149399462 wait complete. 2024-05-09T08:43:15.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.717+0000 7fc9b8eda640 1 Processor -- start 2024-05-09T08:43:15.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.717+0000 7fc9b8eda640 1 -- start start 2024-05-09T08:43:15.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.717+0000 7fc9b8eda640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9b4069e30 0x7fc9b4197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.717+0000 7fc9b8eda640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9b406a7f0 0x7fc9b41984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.717+0000 7fc9b8eda640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9b4107810 0x7fc9b419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.717+0000 7fc9b8eda640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b4111ad0 con 0x7fc9b406a7f0 2024-05-09T08:43:15.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.717+0000 7fc9b8eda640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b4111950 con 0x7fc9b4069e30 2024-05-09T08:43:15.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.717+0000 7fc9b8eda640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b4111c50 con 0x7fc9b4107810 2024-05-09T08:43:15.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9b1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9b406a7f0 0x7fc9b41984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9b1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9b406a7f0 0x7fc9b41984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55042/0 (socket says 172.21.15.107:55042) 2024-05-09T08:43:15.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9b1d9b640 1 -- 172.21.15.107:0/4004346847 learned_addr learned my addr 172.21.15.107:0/4004346847 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:15.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9b259c640 1 --2- 172.21.15.107:0/4004346847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9b4069e30 0x7fc9b4197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9b2d9d640 1 --2- 172.21.15.107:0/4004346847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9b4107810 0x7fc9b419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9b1d9b640 1 -- 172.21.15.107:0/4004346847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9b4107810 msgr2=0x7fc9b419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9b1d9b640 1 --2- 172.21.15.107:0/4004346847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9b4107810 0x7fc9b419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9b1d9b640 1 -- 172.21.15.107:0/4004346847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9b4069e30 msgr2=0x7fc9b4197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9b1d9b640 1 --2- 172.21.15.107:0/4004346847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9b4069e30 0x7fc9b4197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9b1d9b640 1 -- 172.21.15.107:0/4004346847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b419cf50 con 0x7fc9b406a7f0 2024-05-09T08:43:15.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9b2d9d640 1 --2- 172.21.15.107:0/4004346847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9b4107810 0x7fc9b419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:43:15.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9b1d9b640 1 --2- 172.21.15.107:0/4004346847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9b406a7f0 0x7fc9b41984f0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00c920 tx=0x7fc99c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:15.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9ab7fe640 1 -- 172.21.15.107:0/4004346847 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c007c70 con 0x7fc9b406a7f0 2024-05-09T08:43:15.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9ab7fe640 1 -- 172.21.15.107:0/4004346847 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc99c007e10 con 0x7fc9b406a7f0 2024-05-09T08:43:15.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9b8eda640 1 -- 172.21.15.107:0/4004346847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9b419d240 con 0x7fc9b406a7f0 2024-05-09T08:43:15.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9ab7fe640 1 -- 172.21.15.107:0/4004346847 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c018820 con 0x7fc9b406a7f0 2024-05-09T08:43:15.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.718+0000 7fc9b8eda640 1 -- 172.21.15.107:0/4004346847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9b41a4ad0 con 0x7fc9b406a7f0 2024-05-09T08:43:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.720+0000 7fc9ab7fe640 1 -- 172.21.15.107:0/4004346847 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fc99c0189c0 con 0x7fc9b406a7f0 2024-05-09T08:43:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.721+0000 7fc9bbbec740 1 Processor -- start 2024-05-09T08:43:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.721+0000 7fc9bbbec740 1 -- start start 2024-05-09T08:43:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.721+0000 7fc9bbbec740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80330 0x558ee4e80710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.721+0000 7fc9bbbec740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e80c50 0x558ee4e8ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.721+0000 7fc9bbbec740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e8b320 0x558ee4e8d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.721+0000 7fc9bbbec740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558ee4de5c40 con 0x558ee4e80330 2024-05-09T08:43:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.721+0000 7fc9bbbec740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558ee4d8d580 con 0x558ee4e80c50 2024-05-09T08:43:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.721+0000 7fc9bbbec740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x558ee4de1c00 con 0x558ee4e8b320 2024-05-09T08:43:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.721+0000 7fc9b2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80330 0x558ee4e80710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.721+0000 7fc9ab7fe640 1 --2- 172.21.15.107:0/4004346847 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9a00a2d00 0x7fc9a00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.721+0000 7fc9b2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80330 0x558ee4e80710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55056/0 (socket says 172.21.15.107:55056) 2024-05-09T08:43:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.721+0000 7fc9b2d9d640 1 -- 172.21.15.107:0/1172703542 learned_addr learned my addr 172.21.15.107:0/1172703542 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.721+0000 7fc9b259c640 1 --2- 172.21.15.107:0/1172703542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e80c50 0x558ee4e8ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.722+0000 7fc9b1d9b640 1 --2- 172.21.15.107:0/1172703542 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e8b320 0x558ee4e8d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.722+0000 7fc9b2d9d640 1 --2- 172.21.15.107:0/4004346847 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9a00a2d00 0x7fc9a00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.722+0000 7fc9ab7fe640 1 -- 172.21.15.107:0/4004346847 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fc99c0cf9f0 con 0x7fc9b406a7f0 2024-05-09T08:43:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.722+0000 7fc9b259c640 1 -- 172.21.15.107:0/1172703542 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e8b320 msgr2=0x558ee4e8d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.722+0000 7fc9b259c640 1 --2- 172.21.15.107:0/1172703542 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e8b320 0x558ee4e8d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.722+0000 7fc9b259c640 1 -- 172.21.15.107:0/1172703542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80330 msgr2=0x558ee4e80710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:15.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.722+0000 7fc9b259c640 1 --2- 172.21.15.107:0/1172703542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80330 0x558ee4e80710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.722+0000 7fc9b259c640 1 -- 172.21.15.107:0/1172703542 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ee4af93f0 con 0x558ee4e80c50 2024-05-09T08:43:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.722+0000 7fc9b2d9d640 1 --2- 172.21.15.107:0/1172703542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80330 0x558ee4e80710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.722+0000 7fc9b259c640 1 --2- 172.21.15.107:0/1172703542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e80c50 0x558ee4e8ade0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400aac0 tx=0x7fc9a400af80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7756fdcf03080d7e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.722+0000 7fc9b2d9d640 1 --2- 172.21.15.107:0/4004346847 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9a00a2d00 0x7fc9a00a51c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0317f0 tx=0x7fc9ac02a8e0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.722+0000 7fc9a8ff9640 1 -- 172.21.15.107:0/1172703542 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4004020 con 0x558ee4e80c50 2024-05-09T08:43:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.722+0000 7fc9a8ff9640 1 -- 172.21.15.107:0/1172703542 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc9a40041c0 con 0x558ee4e80c50 2024-05-09T08:43:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.722+0000 7fc9a8ff9640 1 -- 172.21.15.107:0/1172703542 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a40186a0 con 0x558ee4e80c50 2024-05-09T08:43:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.723+0000 7fc9bbbec740 1 -- 172.21.15.107:0/1172703542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e80c50 msgr2=0x558ee4e8ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.723+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/1172703542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e80c50 0x558ee4e8ade0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400aac0 tx=0x7fc9a400af80 comp rx=0 tx=0).stop 2024-05-09T08:43:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.723+0000 7fc9bbbec740 1 -- 172.21.15.107:0/1172703542 shutdown_connections 2024-05-09T08:43:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.723+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/1172703542 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e8b320 0x558ee4e8d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.723+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/1172703542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e80c50 0x558ee4e8ade0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.723+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/1172703542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80330 0x558ee4e80710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.723+0000 7fc9bbbec740 1 -- 172.21.15.107:0/1172703542 >> 172.21.15.107:0/1172703542 conn(0x558ee4e7c650 msgr2=0x558ee4e7ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.723+0000 7fc9bbbec740 1 -- 172.21.15.107:0/1172703542 shutdown_connections 2024-05-09T08:43:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.723+0000 7fc9bbbec740 1 -- 172.21.15.107:0/1172703542 wait complete. 2024-05-09T08:43:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.723+0000 7fc9bbbec740 1 -- start start 2024-05-09T08:43:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.723+0000 7fc9bbbec740 1 Processor -- start 2024-05-09T08:43:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9bbbec740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80330 0x558ee4e88230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9bbbec740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e80c50 0x558ee4e88770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9bbbec740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 0x558ee4e88cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9bbbec740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558ee4e8e110 con 0x558ee4e80330 2024-05-09T08:43:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9bbbec740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558ee4da0c40 con 0x558ee4e8b320 2024-05-09T08:43:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9bbbec740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x558ee4e8e290 con 0x558ee4e80c50 2024-05-09T08:43:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9b259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80330 0x558ee4e88230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9bbbec740 10 client.0 _ll_register_callbacks cb 0x558ee4af8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9b259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80330 0x558ee4e88230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55060/0 (socket says 172.21.15.107:55060) 2024-05-09T08:43:15.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9b259c640 1 -- 172.21.15.107:0/2391635724 learned_addr learned my addr 172.21.15.107:0/2391635724 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9b1d9b640 1 --2- 172.21.15.107:0/2391635724 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e80c50 0x558ee4e88770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9b2d9d640 1 --2- 172.21.15.107:0/2391635724 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 0x558ee4e88cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9b259c640 1 -- 172.21.15.107:0/2391635724 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e80c50 msgr2=0x558ee4e88770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9b259c640 1 --2- 172.21.15.107:0/2391635724 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e80c50 0x558ee4e88770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9b259c640 1 -- 172.21.15.107:0/2391635724 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 msgr2=0x558ee4e88cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9b259c640 1 --2- 172.21.15.107:0/2391635724 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 0x558ee4e88cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9b259c640 1 -- 172.21.15.107:0/2391635724 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ee4bc37d0 con 0x558ee4e80330 2024-05-09T08:43:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9b2d9d640 1 --2- 172.21.15.107:0/2391635724 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 0x558ee4e88cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:43:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9b259c640 1 --2- 172.21.15.107:0/2391635724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80330 0x558ee4e88230 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4003f40 tx=0x7fc9a400ffc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7251e400f35c77b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc999ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:43:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc999ffb640 1 -- 172.21.15.107:0/2391635724 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4027020 con 0x558ee4e80330 2024-05-09T08:43:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc999ffb640 1 -- 172.21.15.107:0/2391635724 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fc9a4005040 con 0x558ee4e80330 2024-05-09T08:43:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9bbbec740 1 -- 172.21.15.107:0/2391635724 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9a40041c0 con 0x558ee4e80330 2024-05-09T08:43:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc999ffb640 1 -- 172.21.15.107:0/2391635724 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc9a4032050 con 0x558ee4e80330 2024-05-09T08:43:15.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9bbbec740 10 client.18459 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:15.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.724+0000 7fc9bbbec740 1 -- 172.21.15.107:0/2391635724 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558ee4e9a2a0 con 0x558ee4e80330 2024-05-09T08:43:15.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.725+0000 7fc999ffb640 1 -- 172.21.15.107:0/2391635724 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc9a40041c0 con 0x558ee4e80330 2024-05-09T08:43:15.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.725+0000 7fc999ffb640 1 -- 172.21.15.107:0/2391635724 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc9a4004a10 con 0x558ee4e80330 2024-05-09T08:43:15.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.725+0000 7fc9bbbec740 10 client.18459 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:15.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.725+0000 7fc9bbbec740 10 client.18459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:15.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.725+0000 7fc9bbbec740 20 client.18459 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:15.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.725+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/2391635724 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558ee4e9b760 0x558ee4e9db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.725+0000 7fc9bbbec740 4 client.18459 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:15.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.725+0000 7fc9bbbec740 1 -- 172.21.15.107:0/2391635724 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ee4df3370 con 0x558ee4e9b760 2024-05-09T08:43:15.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.725+0000 7fc9b1d9b640 1 --2- 172.21.15.107:0/2391635724 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558ee4e9b760 0x558ee4e9db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.725+0000 7fc999ffb640 10 client.18459 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:15.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.725+0000 7fc9b1d9b640 1 --2- 172.21.15.107:0/2391635724 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558ee4e9b760 0x558ee4e9db50 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd48b6d30c96350d server_cookie=a2a1a3bf3986c638 in_seq=0 out_seq=0 2024-05-09T08:43:15.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.726+0000 7fc999ffb640 1 -- 172.21.15.107:0/2391635724 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558ee4df3370 con 0x558ee4e9b760 2024-05-09T08:43:15.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.726+0000 7fc999ffb640 10 client.18459 handle_command_reply: tid=0 2024-05-09T08:43:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.726+0000 7fc9bbbec740 1 client.18459 shutdown 2024-05-09T08:43:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.726+0000 7fc9bbbec740 1 -- 172.21.15.107:0/2391635724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80330 msgr2=0x558ee4e88230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.726+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/2391635724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80330 0x558ee4e88230 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4003f40 tx=0x7fc9a400ffc0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.726+0000 7fc9bbbec740 1 -- 172.21.15.107:0/2391635724 shutdown_connections 2024-05-09T08:43:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.726+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/2391635724 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558ee4e9b760 0x558ee4e9db50 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.726+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/2391635724 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 0x558ee4e88cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.726+0000 7fc999ffb640 0 client.18459 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.726+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/2391635724 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e80c50 0x558ee4e88770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.726+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/2391635724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80330 0x558ee4e88230 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.726+0000 7fc9bbbec740 1 -- 172.21.15.107:0/2391635724 >> 172.21.15.107:0/2391635724 conn(0x558ee4e7c650 msgr2=0x558ee4e82350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.726+0000 7fc9bbbec740 1 -- 172.21.15.107:0/2391635724 shutdown_connections 2024-05-09T08:43:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.726+0000 7fc9bbbec740 1 -- 172.21.15.107:0/2391635724 wait complete. 2024-05-09T08:43:15.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.726+0000 7fc9bbbec740 20 client.18459 trim_cache size 0 max 16384 2024-05-09T08:43:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9bbbec740 1 Processor -- start 2024-05-09T08:43:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9bbbec740 1 -- start start 2024-05-09T08:43:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9bbbec740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e80330 0x558ee4e96fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9bbbec740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80c50 0x558ee4e89950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9bbbec740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 0x558ee4e82870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9bbbec740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b41b15b0 con 0x558ee4e80c50 2024-05-09T08:43:15.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9bbbec740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b41aee80 con 0x558ee4e8b320 2024-05-09T08:43:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9bbbec740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b41accf0 con 0x558ee4e80330 2024-05-09T08:43:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9b1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80c50 0x558ee4e89950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9b1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80c50 0x558ee4e89950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55068/0 (socket says 172.21.15.107:55068) 2024-05-09T08:43:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9b1d9b640 1 -- 172.21.15.107:0/2780907400 learned_addr learned my addr 172.21.15.107:0/2780907400 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9b259c640 1 --2- 172.21.15.107:0/2780907400 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e80330 0x558ee4e96fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9b2d9d640 1 --2- 172.21.15.107:0/2780907400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 0x558ee4e82870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9b1d9b640 1 -- 172.21.15.107:0/2780907400 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e80330 msgr2=0x558ee4e96fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9b1d9b640 1 --2- 172.21.15.107:0/2780907400 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e80330 0x558ee4e96fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9b1d9b640 1 -- 172.21.15.107:0/2780907400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 msgr2=0x558ee4e82870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9b1d9b640 1 --2- 172.21.15.107:0/2780907400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 0x558ee4e82870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9b1d9b640 1 -- 172.21.15.107:0/2780907400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ee4e8a800 con 0x558ee4e80c50 2024-05-09T08:43:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc9b1d9b640 1 --2- 172.21.15.107:0/2780907400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80c50 0x558ee4e89950 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00c8f0 tx=0x7fc99c013b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2285af07e3361560 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc99bfff640 1 -- 172.21.15.107:0/2780907400 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c09fe50 con 0x558ee4e80c50 2024-05-09T08:43:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc99bfff640 1 -- 172.21.15.107:0/2780907400 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc99c0a4710 con 0x558ee4e80c50 2024-05-09T08:43:15.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.740+0000 7fc99bfff640 1 -- 172.21.15.107:0/2780907400 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c0a4280 con 0x558ee4e80c50 2024-05-09T08:43:15.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.741+0000 7fc9bbbec740 1 -- 172.21.15.107:0/2780907400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80c50 msgr2=0x558ee4e89950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.741+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/2780907400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80c50 0x558ee4e89950 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00c8f0 tx=0x7fc99c013b80 comp rx=0 tx=0).stop 2024-05-09T08:43:15.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.741+0000 7fc9bbbec740 1 -- 172.21.15.107:0/2780907400 shutdown_connections 2024-05-09T08:43:15.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.741+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/2780907400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 0x558ee4e82870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.741+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/2780907400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80c50 0x558ee4e89950 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.741+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/2780907400 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e80330 0x558ee4e96fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.741+0000 7fc9bbbec740 1 -- 172.21.15.107:0/2780907400 >> 172.21.15.107:0/2780907400 conn(0x558ee4e7c650 msgr2=0x558ee4e88a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:15.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.741+0000 7fc9bbbec740 1 -- 172.21.15.107:0/2780907400 shutdown_connections 2024-05-09T08:43:15.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.741+0000 7fc9bbbec740 1 -- 172.21.15.107:0/2780907400 wait complete. 2024-05-09T08:43:15.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.741+0000 7fc9bbbec740 1 -- start start 2024-05-09T08:43:15.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.741+0000 7fc9bbbec740 1 Processor -- start 2024-05-09T08:43:15.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9bbbec740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e80330 0x558ee4e934e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9bbbec740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80c50 0x558ee4e93a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9bbbec740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 0x558ee4e904b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9bbbec740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b41ad340 con 0x558ee4e80c50 2024-05-09T08:43:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9bbbec740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b41ae7b0 con 0x558ee4e8b320 2024-05-09T08:43:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9bbbec740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b41ada10 con 0x558ee4e80330 2024-05-09T08:43:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9bbbec740 10 client.0 _ll_register_callbacks cb 0x7fc9ac009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9b1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80c50 0x558ee4e93a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9b1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80c50 0x558ee4e93a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55072/0 (socket says 172.21.15.107:55072) 2024-05-09T08:43:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9b1d9b640 1 -- 172.21.15.107:0/574854702 learned_addr learned my addr 172.21.15.107:0/574854702 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9b2d9d640 1 --2- 172.21.15.107:0/574854702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 0x558ee4e904b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9b259c640 1 --2- 172.21.15.107:0/574854702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e80330 0x558ee4e934e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9b1d9b640 1 -- 172.21.15.107:0/574854702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e80330 msgr2=0x558ee4e934e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9b1d9b640 1 --2- 172.21.15.107:0/574854702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e80330 0x558ee4e934e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9b1d9b640 1 -- 172.21.15.107:0/574854702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 msgr2=0x558ee4e904b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9b1d9b640 1 --2- 172.21.15.107:0/574854702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 0x558ee4e904b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9b1d9b640 1 -- 172.21.15.107:0/574854702 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ee4e90af0 con 0x558ee4e80c50 2024-05-09T08:43:15.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9b2d9d640 1 --2- 172.21.15.107:0/574854702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 0x558ee4e904b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:43:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9b1d9b640 1 --2- 172.21.15.107:0/574854702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80c50 0x558ee4e93a20 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0a41e0 tx=0x7fc99c0ccca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df40132cd259077f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc999ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:43:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc999ffb640 1 -- 172.21.15.107:0/574854702 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c099da0 con 0x558ee4e80c50 2024-05-09T08:43:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc999ffb640 1 -- 172.21.15.107:0/574854702 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fc99c0ccf50 con 0x558ee4e80c50 2024-05-09T08:43:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9bbbec740 1 -- 172.21.15.107:0/574854702 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc99c0a4710 con 0x558ee4e80c50 2024-05-09T08:43:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc999ffb640 1 -- 172.21.15.107:0/574854702 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc99c0cd150 con 0x558ee4e80c50 2024-05-09T08:43:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9bbbec740 10 client.18465 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc9bbbec740 1 -- 172.21.15.107:0/574854702 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558ee4f023f0 con 0x558ee4e80c50 2024-05-09T08:43:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.742+0000 7fc999ffb640 1 -- 172.21.15.107:0/574854702 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc99c0a4710 con 0x558ee4e80c50 2024-05-09T08:43:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.743+0000 7fc999ffb640 1 -- 172.21.15.107:0/574854702 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fc99c013bf0 con 0x558ee4e80c50 2024-05-09T08:43:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.743+0000 7fc9bbbec740 10 client.18465 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:15.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.743+0000 7fc9bbbec740 10 client.18465 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:15.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.743+0000 7fc9bbbec740 20 client.18465 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:15.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.743+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/574854702 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558ee4f02af0 0x558ee4f04f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:15.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.743+0000 7fc9bbbec740 4 client.18465 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:43:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.743+0000 7fc9bbbec740 1 -- 172.21.15.107:0/574854702 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558ee4ded6c0 con 0x558ee4f02af0 2024-05-09T08:43:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.743+0000 7fc9b259c640 1 --2- 172.21.15.107:0/574854702 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558ee4f02af0 0x558ee4f04f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.743+0000 7fc999ffb640 10 client.18465 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.743+0000 7fc9b259c640 1 --2- 172.21.15.107:0/574854702 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558ee4f02af0 0x558ee4f04f10 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e22c55001066048 server_cookie=a0526dce0606d930 in_seq=0 out_seq=0 2024-05-09T08:43:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.743+0000 7fc999ffb640 1 -- 172.21.15.107:0/574854702 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558ee4ded6c0 con 0x558ee4f02af0 2024-05-09T08:43:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.743+0000 7fc999ffb640 10 client.18465 handle_command_reply: tid=0 2024-05-09T08:43:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.743+0000 7fc9bbbec740 1 client.18465 shutdown 2024-05-09T08:43:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.743+0000 7fc9bbbec740 1 -- 172.21.15.107:0/574854702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80c50 msgr2=0x558ee4e93a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.743+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/574854702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80c50 0x558ee4e93a20 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0a41e0 tx=0x7fc99c0ccca0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9bbbec740 1 -- 172.21.15.107:0/574854702 shutdown_connections 2024-05-09T08:43:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/574854702 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558ee4f02af0 0x558ee4f04f10 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/574854702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558ee4e8b320 0x558ee4e904b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/574854702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ee4e80c50 0x558ee4e93a20 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc999ffb640 0 client.18465 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:15.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9bbbec740 1 --2- 172.21.15.107:0/574854702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ee4e80330 0x558ee4e934e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9bbbec740 1 -- 172.21.15.107:0/574854702 >> 172.21.15.107:0/574854702 conn(0x558ee4e7c650 msgr2=0x558ee4e88a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:15.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9bbbec740 1 -- 172.21.15.107:0/574854702 shutdown_connections 2024-05-09T08:43:15.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9bbbec740 1 -- 172.21.15.107:0/574854702 wait complete. 2024-05-09T08:43:15.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9bbbec740 20 client.18465 trim_cache size 0 max 16384 2024-05-09T08:43:15.755 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:43:15.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9b8eda640 1 -- 172.21.15.107:0/4004346847 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9a00a2d00 msgr2=0x7fc9a00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9b8eda640 1 --2- 172.21.15.107:0/4004346847 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9a00a2d00 0x7fc9a00a51c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0317f0 tx=0x7fc9ac02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9b8eda640 1 -- 172.21.15.107:0/4004346847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9b406a7f0 msgr2=0x7fc9b41984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:15.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9b8eda640 1 --2- 172.21.15.107:0/4004346847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9b406a7f0 0x7fc9b41984f0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00c920 tx=0x7fc99c00cde0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9b8eda640 1 -- 172.21.15.107:0/4004346847 shutdown_connections 2024-05-09T08:43:15.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9b8eda640 1 --2- 172.21.15.107:0/4004346847 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fc9a00a2d00 0x7fc9a00a51c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9b8eda640 1 --2- 172.21.15.107:0/4004346847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9b4107810 0x7fc9b419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9b8eda640 1 --2- 172.21.15.107:0/4004346847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9b406a7f0 0x7fc9b41984f0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9b8eda640 1 --2- 172.21.15.107:0/4004346847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9b4069e30 0x7fc9b4197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:15.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9b8eda640 1 -- 172.21.15.107:0/4004346847 >> 172.21.15.107:0/4004346847 conn(0x7fc9b4076530 msgr2=0x7fc9b40fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:15.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9b8eda640 1 -- 172.21.15.107:0/4004346847 shutdown_connections 2024-05-09T08:43:15.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:15.744+0000 7fc9b8eda640 1 -- 172.21.15.107:0/4004346847 wait complete. 2024-05-09T08:43:15.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:15 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/4199146356' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:43:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.261+0000 7fa465f3c640 1 -- 172.21.15.107:0/1510662335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa4601102d0 msgr2=0x7fa46010dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.261+0000 7fa465f3c640 1 --2- 172.21.15.107:0/1510662335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa4601102d0 0x7fa46010dee0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa45801acb0 tx=0x7fa45803b990 comp rx=0 tx=0).stop 2024-05-09T08:43:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.267+0000 7fa465f3c640 1 -- 172.21.15.107:0/1510662335 shutdown_connections 2024-05-09T08:43:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.267+0000 7fa465f3c640 1 --2- 172.21.15.107:0/1510662335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa46010e500 0x7fa46011d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.267+0000 7fa465f3c640 1 --2- 172.21.15.107:0/1510662335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa4601102d0 0x7fa46010dee0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.267+0000 7fa465f3c640 1 --2- 172.21.15.107:0/1510662335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa46010fef0 0x7fa46010d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.267+0000 7fa465f3c640 1 -- 172.21.15.107:0/1510662335 >> 172.21.15.107:0/1510662335 conn(0x7fa46006c770 msgr2=0x7fa46006cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:16.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.272+0000 7fa465f3c640 1 -- 172.21.15.107:0/1510662335 shutdown_connections 2024-05-09T08:43:16.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.272+0000 7fa465f3c640 1 -- 172.21.15.107:0/1510662335 wait complete. 2024-05-09T08:43:16.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.273+0000 7fa465f3c640 1 Processor -- start 2024-05-09T08:43:16.274 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.274+0000 7fa465f3c640 1 -- start start 2024-05-09T08:43:16.274 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.274+0000 7fa465f3c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa46010e500 0x7fa46011cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.274+0000 7fa465f3c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa46010fef0 0x7fa460118d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.274+0000 7fa465f3c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4601102d0 0x7fa4601192d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.274+0000 7fa465f3c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa460113140 con 0x7fa46010e500 2024-05-09T08:43:16.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.274+0000 7fa465f3c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa460112fc0 con 0x7fa46010fef0 2024-05-09T08:43:16.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.274+0000 7fa465f3c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa4601132c0 con 0x7fa4601102d0 2024-05-09T08:43:16.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.274+0000 7fa464f3a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa46010e500 0x7fa46011cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.274+0000 7fa464f3a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa46010e500 0x7fa46011cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55084/0 (socket says 172.21.15.107:55084) 2024-05-09T08:43:16.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.274+0000 7fa464f3a640 1 -- 172.21.15.107:0/1690288053 learned_addr learned my addr 172.21.15.107:0/1690288053 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:16.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.274+0000 7fa45ffff640 1 --2- 172.21.15.107:0/1690288053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa46010fef0 0x7fa460118d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.274+0000 7fa46573b640 1 --2- 172.21.15.107:0/1690288053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4601102d0 0x7fa4601192d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.275+0000 7fa45ffff640 1 -- 172.21.15.107:0/1690288053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4601102d0 msgr2=0x7fa4601192d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:16.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.275+0000 7fa45ffff640 1 --2- 172.21.15.107:0/1690288053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4601102d0 0x7fa4601192d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.275+0000 7fa45ffff640 1 -- 172.21.15.107:0/1690288053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa46010e500 msgr2=0x7fa46011cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:16.276 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.275+0000 7fa45ffff640 1 --2- 172.21.15.107:0/1690288053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa46010e500 0x7fa46011cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.276 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.275+0000 7fa45ffff640 1 -- 172.21.15.107:0/1690288053 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa460119a00 con 0x7fa46010fef0 2024-05-09T08:43:16.276 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.277+0000 7fa45ffff640 1 --2- 172.21.15.107:0/1690288053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa46010fef0 0x7fa460118d90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa45801ac00 tx=0x7fa458003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.279+0000 7fa46573b640 1 --2- 172.21.15.107:0/1690288053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4601102d0 0x7fa4601192d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:43:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.279+0000 7fa464f3a640 1 --2- 172.21.15.107:0/1690288053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa46010e500 0x7fa46011cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:43:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.280+0000 7fa45dffb640 1 -- 172.21.15.107:0/1690288053 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458045070 con 0x7fa46010fef0 2024-05-09T08:43:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.280+0000 7fa45dffb640 1 -- 172.21.15.107:0/1690288053 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa458040dc0 con 0x7fa46010fef0 2024-05-09T08:43:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.280+0000 7fa465f3c640 1 -- 172.21.15.107:0/1690288053 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4601aacb0 con 0x7fa46010fef0 2024-05-09T08:43:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.281+0000 7fa465f3c640 1 -- 172.21.15.107:0/1690288053 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4601ab0b0 con 0x7fa46010fef0 2024-05-09T08:43:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.281+0000 7fa45dffb640 1 -- 172.21.15.107:0/1690288053 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458004680 con 0x7fa46010fef0 2024-05-09T08:43:16.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.284+0000 7fa45dffb640 1 -- 172.21.15.107:0/1690288053 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fa458048020 con 0x7fa46010fef0 2024-05-09T08:43:16.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.285+0000 7fa45dffb640 1 --2- 172.21.15.107:0/1690288053 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa4480a2f00 0x7fa4480a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.286+0000 7fa45dffb640 1 -- 172.21.15.107:0/1690288053 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fa458101830 con 0x7fa46010fef0 2024-05-09T08:43:16.290 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.287+0000 7fa464f3a640 1 --2- 172.21.15.107:0/1690288053 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa4480a2f00 0x7fa4480a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.290 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.287+0000 7fa464f3a640 1 --2- 172.21.15.107:0/1690288053 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa4480a2f00 0x7fa4480a53c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa4500099f0 tx=0x7fa450008040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:16.292 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.291+0000 7fa469f87740 1 Processor -- start 2024-05-09T08:43:16.292 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.291+0000 7fa469f87740 1 -- start start 2024-05-09T08:43:16.292 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.291+0000 7fa469f87740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560144290600 0x5601442909e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.292 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.291+0000 7fa469f87740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560144290f20 0x56014429b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.292 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.291+0000 7fa469f87740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 0x56014429dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.292 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.291+0000 7fa469f87740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5601441f5340 con 0x560144290600 2024-05-09T08:43:16.292 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.291+0000 7fa469f87740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56014419d290 con 0x560144290f20 2024-05-09T08:43:16.292 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.291+0000 7fa469f87740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5601441f1300 con 0x56014429b7e0 2024-05-09T08:43:16.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.292+0000 7fa464f3a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560144290600 0x5601442909e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.292+0000 7fa464f3a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560144290600 0x5601442909e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55100/0 (socket says 172.21.15.107:55100) 2024-05-09T08:43:16.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.292+0000 7fa464f3a640 1 -- 172.21.15.107:0/1912229049 learned_addr learned my addr 172.21.15.107:0/1912229049 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:16.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.293+0000 7fa46573b640 1 --2- 172.21.15.107:0/1912229049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 0x56014429dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.295+0000 7fa464f3a640 1 -- 172.21.15.107:0/1912229049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 msgr2=0x56014429dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.295+0000 7fa464f3a640 1 --2- 172.21.15.107:0/1912229049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 0x56014429dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.295+0000 7fa464f3a640 1 -- 172.21.15.107:0/1912229049 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560144290f20 msgr2=0x56014429b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.295+0000 7fa464f3a640 1 --2- 172.21.15.107:0/1912229049 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560144290f20 0x56014429b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.295+0000 7fa464f3a640 1 -- 172.21.15.107:0/1912229049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601441c18e0 con 0x560144290600 2024-05-09T08:43:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.295+0000 7fa464f3a640 1 --2- 172.21.15.107:0/1912229049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560144290600 0x5601442909e0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fa45000d9a0 tx=0x7fa450010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c28a6a1a1051518e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.296+0000 7fa446ffd640 1 -- 172.21.15.107:0/1912229049 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa450024070 con 0x560144290600 2024-05-09T08:43:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.296+0000 7fa446ffd640 1 -- 172.21.15.107:0/1912229049 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa450014bc0 con 0x560144290600 2024-05-09T08:43:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.297+0000 7fa446ffd640 1 -- 172.21.15.107:0/1912229049 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa450012640 con 0x560144290600 2024-05-09T08:43:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.297+0000 7fa469f87740 1 -- 172.21.15.107:0/1912229049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560144290600 msgr2=0x5601442909e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.297+0000 7fa469f87740 1 --2- 172.21.15.107:0/1912229049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560144290600 0x5601442909e0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fa45000d9a0 tx=0x7fa450010000 comp rx=0 tx=0).stop 2024-05-09T08:43:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.298+0000 7fa469f87740 1 -- 172.21.15.107:0/1912229049 shutdown_connections 2024-05-09T08:43:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.298+0000 7fa469f87740 1 --2- 172.21.15.107:0/1912229049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 0x56014429dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.298+0000 7fa469f87740 1 --2- 172.21.15.107:0/1912229049 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560144290f20 0x56014429b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.298+0000 7fa469f87740 1 --2- 172.21.15.107:0/1912229049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560144290600 0x5601442909e0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.298+0000 7fa469f87740 1 -- 172.21.15.107:0/1912229049 >> 172.21.15.107:0/1912229049 conn(0x56014428cb10 msgr2=0x56014428cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.298+0000 7fa469f87740 1 -- 172.21.15.107:0/1912229049 shutdown_connections 2024-05-09T08:43:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.298+0000 7fa469f87740 1 -- 172.21.15.107:0/1912229049 wait complete. 2024-05-09T08:43:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.298+0000 7fa469f87740 1 -- start start 2024-05-09T08:43:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.298+0000 7fa469f87740 1 Processor -- start 2024-05-09T08:43:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.299+0000 7fa469f87740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560144290600 0x560144298810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.299+0000 7fa469f87740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560144290f20 0x560144298d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.299+0000 7fa469f87740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 0x560144299290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.299+0000 7fa469f87740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56014429e410 con 0x560144290600 2024-05-09T08:43:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.299+0000 7fa469f87740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5601441b0790 con 0x560144290f20 2024-05-09T08:43:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.299+0000 7fa469f87740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56014429e590 con 0x56014429b7e0 2024-05-09T08:43:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.299+0000 7fa469f87740 10 client.0 _ll_register_callbacks cb 0x5601441eba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.299+0000 7fa46573b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 0x560144299290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.299+0000 7fa46573b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 0x560144299290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:60150/0 (socket says 172.21.15.107:60150) 2024-05-09T08:43:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.299+0000 7fa46573b640 1 -- 172.21.15.107:0/218769465 learned_addr learned my addr 172.21.15.107:0/218769465 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.299+0000 7fa45ffff640 1 --2- 172.21.15.107:0/218769465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560144290f20 0x560144298d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.300+0000 7fa464f3a640 1 --2- 172.21.15.107:0/218769465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560144290600 0x560144298810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.300+0000 7fa45ffff640 1 -- 172.21.15.107:0/218769465 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 msgr2=0x560144299290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.300+0000 7fa45ffff640 1 --2- 172.21.15.107:0/218769465 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 0x560144299290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.300+0000 7fa45ffff640 1 -- 172.21.15.107:0/218769465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560144290600 msgr2=0x560144298810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.300+0000 7fa45ffff640 1 --2- 172.21.15.107:0/218769465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560144290600 0x560144298810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.300+0000 7fa45ffff640 1 -- 172.21.15.107:0/218769465 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560143f09210 con 0x560144290f20 2024-05-09T08:43:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.303+0000 7fa45ffff640 1 --2- 172.21.15.107:0/218769465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560144290f20 0x560144298d50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa4580fec20 tx=0x7fa4580cd350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=292dffdcf13aba6c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:16.308 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.305+0000 7fa43ffff640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:43:16.308 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.306+0000 7fa43ffff640 1 -- 172.21.15.107:0/218769465 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4580e5070 con 0x560144290f20 2024-05-09T08:43:16.308 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.306+0000 7fa43ffff640 1 -- 172.21.15.107:0/218769465 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fa4580fed90 con 0x560144290f20 2024-05-09T08:43:16.308 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.306+0000 7fa469f87740 1 -- 172.21.15.107:0/218769465 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa450014bc0 con 0x560144290f20 2024-05-09T08:43:16.308 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.309+0000 7fa43ffff640 1 -- 172.21.15.107:0/218769465 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa4580fef90 con 0x560144290f20 2024-05-09T08:43:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.309+0000 7fa469f87740 10 client.27416 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.309+0000 7fa469f87740 1 -- 172.21.15.107:0/218769465 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560143fd5620 con 0x560144290f20 2024-05-09T08:43:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.310+0000 7fa43ffff640 1 -- 172.21.15.107:0/218769465 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa450014bc0 con 0x560144290f20 2024-05-09T08:43:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.310+0000 7fa43ffff640 1 -- 172.21.15.107:0/218769465 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa45805c030 con 0x560144290f20 2024-05-09T08:43:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.310+0000 7fa469f87740 10 client.27416 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.310+0000 7fa469f87740 10 client.27416 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.310+0000 7fa469f87740 20 client.27416 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.310+0000 7fa469f87740 1 --2- 172.21.15.107:0/218769465 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5601442ab770 0x5601442adb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.310+0000 7fa469f87740 4 client.27416 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:16.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.310+0000 7fa469f87740 1 -- 172.21.15.107:0/218769465 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560144202a70 con 0x5601442ab770 2024-05-09T08:43:16.314 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.314+0000 7fa464f3a640 1 --2- 172.21.15.107:0/218769465 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5601442ab770 0x5601442adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.330+0000 7fa464f3a640 1 --2- 172.21.15.107:0/218769465 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5601442ab770 0x5601442adb90 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3edc7bdfff203287 server_cookie=81ea0fb3d3b3b530 in_seq=0 out_seq=0 2024-05-09T08:43:16.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.336+0000 7fa43ffff640 10 client.27416 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:16.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.336+0000 7fa43ffff640 1 -- 172.21.15.107:0/218769465 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560144202a70 con 0x5601442ab770 2024-05-09T08:43:16.335 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.336+0000 7fa43ffff640 10 client.27416 handle_command_reply: tid=0 2024-05-09T08:43:16.336 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.337+0000 7fa469f87740 1 client.27416 shutdown 2024-05-09T08:43:16.338 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.338+0000 7fa469f87740 1 -- 172.21.15.107:0/218769465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560144290f20 msgr2=0x560144298d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:16.338 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.338+0000 7fa469f87740 1 --2- 172.21.15.107:0/218769465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560144290f20 0x560144298d50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa4580fec20 tx=0x7fa4580cd350 comp rx=0 tx=0).stop 2024-05-09T08:43:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.338+0000 7fa469f87740 1 -- 172.21.15.107:0/218769465 shutdown_connections 2024-05-09T08:43:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.338+0000 7fa469f87740 1 --2- 172.21.15.107:0/218769465 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5601442ab770 0x5601442adb90 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.338+0000 7fa469f87740 1 --2- 172.21.15.107:0/218769465 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 0x560144299290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.338+0000 7fa469f87740 1 --2- 172.21.15.107:0/218769465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560144290f20 0x560144298d50 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa4580fec20 tx=0x7fa4580cd350 comp rx=0 tx=0).stop 2024-05-09T08:43:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.338+0000 7fa469f87740 1 --2- 172.21.15.107:0/218769465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560144290600 0x560144298810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.338+0000 7fa469f87740 1 -- 172.21.15.107:0/218769465 >> 172.21.15.107:0/218769465 conn(0x56014428cb10 msgr2=0x560144292890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:16.343 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.340+0000 7fa43ffff640 0 client.27416 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:16.344 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.345+0000 7fa469f87740 1 -- 172.21.15.107:0/218769465 shutdown_connections 2024-05-09T08:43:16.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.345+0000 7fa469f87740 1 -- 172.21.15.107:0/218769465 wait complete. 2024-05-09T08:43:16.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.345+0000 7fa469f87740 20 client.27416 trim_cache size 0 max 16384 2024-05-09T08:43:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.398+0000 7fa469f87740 1 Processor -- start 2024-05-09T08:43:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.399+0000 7fa469f87740 1 -- start start 2024-05-09T08:43:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.400+0000 7fa469f87740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560144290600 0x5601442a7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.400+0000 7fa469f87740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 0x560144297100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.400+0000 7fa469f87740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5601442990b0 0x5601442b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.400+0000 7fa469f87740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa4601b7be0 con 0x5601442990b0 2024-05-09T08:43:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.400+0000 7fa469f87740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa4601b54b0 con 0x560144290600 2024-05-09T08:43:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.400+0000 7fa469f87740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa4601b3320 con 0x56014429b7e0 2024-05-09T08:43:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.401+0000 7fa45ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 0x560144297100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.401+0000 7fa46573b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5601442990b0 0x5601442b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.401+0000 7fa45ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 0x560144297100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:60164/0 (socket says 172.21.15.107:60164) 2024-05-09T08:43:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.401+0000 7fa45ffff640 1 -- 172.21.15.107:0/3385614067 learned_addr learned my addr 172.21.15.107:0/3385614067 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.401+0000 7fa464f3a640 1 --2- 172.21.15.107:0/3385614067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560144290600 0x5601442a7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.402+0000 7fa45ffff640 1 -- 172.21.15.107:0/3385614067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560144290600 msgr2=0x5601442a7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.402+0000 7fa45ffff640 1 --2- 172.21.15.107:0/3385614067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560144290600 0x5601442a7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.402+0000 7fa45ffff640 1 -- 172.21.15.107:0/3385614067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5601442990b0 msgr2=0x5601442b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.402+0000 7fa45ffff640 1 --2- 172.21.15.107:0/3385614067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5601442990b0 0x5601442b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.402+0000 7fa45ffff640 1 -- 172.21.15.107:0/3385614067 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601442977e0 con 0x56014429b7e0 2024-05-09T08:43:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.403+0000 7fa45ffff640 1 --2- 172.21.15.107:0/3385614067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 0x560144297100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa458000f80 tx=0x7fa4580cd480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d8af5f7a60961a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.404+0000 7fa4467fc640 1 -- 172.21.15.107:0/3385614067 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4580ce0f0 con 0x56014429b7e0 2024-05-09T08:43:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.404+0000 7fa4467fc640 1 -- 172.21.15.107:0/3385614067 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa458004820 con 0x56014429b7e0 2024-05-09T08:43:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.404+0000 7fa4467fc640 1 -- 172.21.15.107:0/3385614067 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4580ec120 con 0x56014429b7e0 2024-05-09T08:43:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.405+0000 7fa469f87740 1 -- 172.21.15.107:0/3385614067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 msgr2=0x560144297100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.405+0000 7fa469f87740 1 --2- 172.21.15.107:0/3385614067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 0x560144297100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa458000f80 tx=0x7fa4580cd480 comp rx=0 tx=0).stop 2024-05-09T08:43:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.406+0000 7fa469f87740 1 -- 172.21.15.107:0/3385614067 shutdown_connections 2024-05-09T08:43:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.406+0000 7fa469f87740 1 --2- 172.21.15.107:0/3385614067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5601442990b0 0x5601442b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.406+0000 7fa469f87740 1 --2- 172.21.15.107:0/3385614067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56014429b7e0 0x560144297100 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.406+0000 7fa469f87740 1 --2- 172.21.15.107:0/3385614067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560144290600 0x5601442a7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.406+0000 7fa469f87740 1 -- 172.21.15.107:0/3385614067 >> 172.21.15.107:0/3385614067 conn(0x56014428cb10 msgr2=0x56014428bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.406+0000 7fa469f87740 1 -- 172.21.15.107:0/3385614067 shutdown_connections 2024-05-09T08:43:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.406+0000 7fa469f87740 1 -- 172.21.15.107:0/3385614067 wait complete. 2024-05-09T08:43:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.406+0000 7fa469f87740 1 -- start start 2024-05-09T08:43:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.406+0000 7fa469f87740 1 Processor -- start 2024-05-09T08:43:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.407+0000 7fa469f87740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560144290600 0x5601442a3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.407+0000 7fa469f87740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601442990b0 0x5601442a3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.407+0000 7fa469f87740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56014429b7e0 0x5601442a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.407+0000 7fa469f87740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa4601b3970 con 0x560144290600 2024-05-09T08:43:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.407+0000 7fa469f87740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa4601b4de0 con 0x56014429b7e0 2024-05-09T08:43:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.407+0000 7fa469f87740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa4601b4040 con 0x5601442990b0 2024-05-09T08:43:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.407+0000 7fa469f87740 10 client.0 _ll_register_callbacks cb 0x7fa458005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.407+0000 7fa46573b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56014429b7e0 0x5601442a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.407+0000 7fa46573b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56014429b7e0 0x5601442a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:45992/0 (socket says 172.21.15.107:45992) 2024-05-09T08:43:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.407+0000 7fa46573b640 1 -- 172.21.15.107:0/1314073740 learned_addr learned my addr 172.21.15.107:0/1314073740 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.408+0000 7fa45ffff640 1 --2- 172.21.15.107:0/1314073740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601442990b0 0x5601442a3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.413+0000 7fa46573b640 1 -- 172.21.15.107:0/1314073740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601442990b0 msgr2=0x5601442a3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.413+0000 7fa46573b640 1 --2- 172.21.15.107:0/1314073740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601442990b0 0x5601442a3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.413+0000 7fa46573b640 1 -- 172.21.15.107:0/1314073740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560144290600 msgr2=0x5601442a3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.413+0000 7fa46573b640 1 --2- 172.21.15.107:0/1314073740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560144290600 0x5601442a3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.413+0000 7fa46573b640 1 -- 172.21.15.107:0/1314073740 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601442a0fe0 con 0x56014429b7e0 2024-05-09T08:43:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.419+0000 7fa46573b640 1 --2- 172.21.15.107:0/1314073740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56014429b7e0 0x5601442a07e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa454015640 tx=0x7fa454015b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6f338f0b8743f75 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.420+0000 7fa43ffff640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:43:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.420+0000 7fa43ffff640 1 -- 172.21.15.107:0/1314073740 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa454013560 con 0x56014429b7e0 2024-05-09T08:43:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.420+0000 7fa43ffff640 1 -- 172.21.15.107:0/1314073740 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fa454013700 con 0x56014429b7e0 2024-05-09T08:43:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.421+0000 7fa469f87740 1 -- 172.21.15.107:0/1314073740 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa458004820 con 0x56014429b7e0 2024-05-09T08:43:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.421+0000 7fa43ffff640 1 -- 172.21.15.107:0/1314073740 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa454013900 con 0x56014429b7e0 2024-05-09T08:43:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.422+0000 7fa469f87740 10 client.27422 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.422+0000 7fa469f87740 1 -- 172.21.15.107:0/1314073740 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560144334780 con 0x56014429b7e0 2024-05-09T08:43:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.424+0000 7fa43ffff640 1 -- 172.21.15.107:0/1314073740 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa454029470 con 0x56014429b7e0 2024-05-09T08:43:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.425+0000 7fa43ffff640 1 -- 172.21.15.107:0/1314073740 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa454016430 con 0x56014429b7e0 2024-05-09T08:43:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.425+0000 7fa469f87740 10 client.27422 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.425+0000 7fa469f87740 10 client.27422 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.425+0000 7fa469f87740 20 client.27422 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.425+0000 7fa469f87740 1 --2- 172.21.15.107:0/1314073740 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560144334ee0 0x560144337300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.425+0000 7fa469f87740 4 client.27422 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:43:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.425+0000 7fa469f87740 1 -- 172.21.15.107:0/1314073740 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fa458004820 con 0x560144334ee0 2024-05-09T08:43:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.426+0000 7fa464f3a640 1 --2- 172.21.15.107:0/1314073740 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560144334ee0 0x560144337300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.430+0000 7fa464f3a640 1 --2- 172.21.15.107:0/1314073740 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560144334ee0 0x560144337300 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aec7faa35b12de3c server_cookie=6d559a1ee89e225b in_seq=0 out_seq=0 2024-05-09T08:43:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.430+0000 7fa43ffff640 10 client.27422 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:16.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.504+0000 7fa43ffff640 1 -- 172.21.15.107:0/1314073740 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fa458004820 con 0x560144334ee0 2024-05-09T08:43:16.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.504+0000 7fa43ffff640 10 client.27422 handle_command_reply: tid=0 2024-05-09T08:43:16.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.505+0000 7fa469f87740 1 client.27422 shutdown 2024-05-09T08:43:16.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.505+0000 7fa469f87740 1 -- 172.21.15.107:0/1314073740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56014429b7e0 msgr2=0x5601442a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:16.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.505+0000 7fa469f87740 1 --2- 172.21.15.107:0/1314073740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56014429b7e0 0x5601442a07e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa454015640 tx=0x7fa454015b00 comp rx=0 tx=0).stop 2024-05-09T08:43:16.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.506+0000 7fa469f87740 1 -- 172.21.15.107:0/1314073740 shutdown_connections 2024-05-09T08:43:16.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.506+0000 7fa469f87740 1 --2- 172.21.15.107:0/1314073740 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x560144334ee0 0x560144337300 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.506+0000 7fa469f87740 1 --2- 172.21.15.107:0/1314073740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56014429b7e0 0x5601442a07e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.506+0000 7fa469f87740 1 --2- 172.21.15.107:0/1314073740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601442990b0 0x5601442a3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.506+0000 7fa469f87740 1 --2- 172.21.15.107:0/1314073740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560144290600 0x5601442a3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.506+0000 7fa469f87740 1 -- 172.21.15.107:0/1314073740 >> 172.21.15.107:0/1314073740 conn(0x56014428cb10 msgr2=0x5601442980a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:16.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.506+0000 7fa43ffff640 0 client.27422 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:16.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.507+0000 7fa469f87740 1 -- 172.21.15.107:0/1314073740 shutdown_connections 2024-05-09T08:43:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.507+0000 7fa469f87740 1 -- 172.21.15.107:0/1314073740 wait complete. 2024-05-09T08:43:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.507+0000 7fa469f87740 20 client.27422 trim_cache size 0 max 16384 2024-05-09T08:43:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:43:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:43:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:43:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:43:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.509+0000 7fa465f3c640 1 -- 172.21.15.107:0/1690288053 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa4480a2f00 msgr2=0x7fa4480a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.509+0000 7fa465f3c640 1 --2- 172.21.15.107:0/1690288053 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa4480a2f00 0x7fa4480a53c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa4500099f0 tx=0x7fa450008040 comp rx=0 tx=0).stop 2024-05-09T08:43:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.509+0000 7fa465f3c640 1 -- 172.21.15.107:0/1690288053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa46010fef0 msgr2=0x7fa460118d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.509+0000 7fa465f3c640 1 --2- 172.21.15.107:0/1690288053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa46010fef0 0x7fa460118d90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa45801ac00 tx=0x7fa458003ba0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.510+0000 7fa465f3c640 1 -- 172.21.15.107:0/1690288053 shutdown_connections 2024-05-09T08:43:16.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.510+0000 7fa465f3c640 1 --2- 172.21.15.107:0/1690288053 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa4480a2f00 0x7fa4480a53c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.510+0000 7fa465f3c640 1 --2- 172.21.15.107:0/1690288053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4601102d0 0x7fa4601192d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.510+0000 7fa465f3c640 1 --2- 172.21.15.107:0/1690288053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa46010fef0 0x7fa460118d90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.510+0000 7fa465f3c640 1 --2- 172.21.15.107:0/1690288053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa46010e500 0x7fa46011cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.510+0000 7fa465f3c640 1 -- 172.21.15.107:0/1690288053 >> 172.21.15.107:0/1690288053 conn(0x7fa46006c770 msgr2=0x7fa46010cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.511+0000 7fa465f3c640 1 -- 172.21.15.107:0/1690288053 shutdown_connections 2024-05-09T08:43:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:16.511+0000 7fa465f3c640 1 -- 172.21.15.107:0/1690288053 wait complete. 2024-05-09T08:43:16.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:16 smithi110 ceph-mon[33622]: pgmap v307: 129 pgs: 8 active+clean+snaptrim, 118 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 17 MiB/s wr, 4.25k op/s 2024-05-09T08:43:16.760 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:43:16.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:43:16.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:43:16.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:16 smithi107 ceph-mon[30122]: pgmap v307: 129 pgs: 8 active+clean+snaptrim, 118 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 17 MiB/s wr, 4.25k op/s 2024-05-09T08:43:16.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:16 smithi157 ceph-mon[33614]: pgmap v307: 129 pgs: 8 active+clean+snaptrim, 118 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 17 MiB/s wr, 4.25k op/s 2024-05-09T08:43:16.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:16.997+0000 7f21dd638640 1 -- 172.21.15.107:0/3587536166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810fe40 msgr2=0x7f21d8110220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:16.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:16.997+0000 7f21dd638640 1 --2- 172.21.15.107:0/3587536166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810fe40 0x7f21d8110220 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f21c8009930 tx=0x7f21c802a480 comp rx=0 tx=0).stop 2024-05-09T08:43:16.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:16.997+0000 7f21dd638640 1 -- 172.21.15.107:0/3587536166 shutdown_connections 2024-05-09T08:43:16.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:16.997+0000 7f21dd638640 1 --2- 172.21.15.107:0/3587536166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21d810e4c0 0x7f21d811d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:16.997+0000 7f21dd638640 1 --2- 172.21.15.107:0/3587536166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21d810d910 0x7f21d810dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:16.997+0000 7f21dd638640 1 --2- 172.21.15.107:0/3587536166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810fe40 0x7f21d8110220 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:16.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:16.997+0000 7f21dd638640 1 -- 172.21.15.107:0/3587536166 >> 172.21.15.107:0/3587536166 conn(0x7f21d806c710 msgr2=0x7f21d806cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.002+0000 7f21dd638640 1 -- 172.21.15.107:0/3587536166 shutdown_connections 2024-05-09T08:43:17.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.002+0000 7f21dd638640 1 -- 172.21.15.107:0/3587536166 wait complete. 2024-05-09T08:43:17.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.003+0000 7f21dd638640 1 Processor -- start 2024-05-09T08:43:17.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.003+0000 7f21dd638640 1 -- start start 2024-05-09T08:43:17.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.003+0000 7f21dd638640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21d810d910 0x7f21d811c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.003+0000 7f21dd638640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21d810e4c0 0x7f21d8115780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.003+0000 7f21dd638640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810fe40 0x7f21d8115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.003+0000 7f21dd638640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f21d81207c0 con 0x7f21d810fe40 2024-05-09T08:43:17.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.003+0000 7f21dd638640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f21d8120640 con 0x7f21d810e4c0 2024-05-09T08:43:17.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.003+0000 7f21dd638640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f21d8120940 con 0x7f21d810d910 2024-05-09T08:43:17.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.004+0000 7f21dce37640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810fe40 0x7f21d8115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.004+0000 7f21dce37640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810fe40 0x7f21d8115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55152/0 (socket says 172.21.15.107:55152) 2024-05-09T08:43:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.004+0000 7f21dce37640 1 -- 172.21.15.107:0/660409108 learned_addr learned my addr 172.21.15.107:0/660409108 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.004+0000 7f21d77fe640 1 --2- 172.21.15.107:0/660409108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21d810e4c0 0x7f21d8115780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.004+0000 7f21d7fff640 1 --2- 172.21.15.107:0/660409108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21d810d910 0x7f21d811c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.005+0000 7f21d77fe640 1 -- 172.21.15.107:0/660409108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21d810d910 msgr2=0x7f21d811c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.005+0000 7f21d77fe640 1 --2- 172.21.15.107:0/660409108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21d810d910 0x7f21d811c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.005+0000 7f21d77fe640 1 -- 172.21.15.107:0/660409108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810fe40 msgr2=0x7f21d8115cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.005+0000 7f21d77fe640 1 --2- 172.21.15.107:0/660409108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810fe40 0x7f21d8115cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.005+0000 7f21d77fe640 1 -- 172.21.15.107:0/660409108 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d8116300 con 0x7f21d810e4c0 2024-05-09T08:43:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.006+0000 7f21dce37640 1 --2- 172.21.15.107:0/660409108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810fe40 0x7f21d8115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:17.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.006+0000 7f21d77fe640 1 --2- 172.21.15.107:0/660409108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21d810e4c0 0x7f21d8115780 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f21cc00b420 tx=0x7f21cc00b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.006+0000 7f21d57fa640 1 -- 172.21.15.107:0/660409108 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc011830 con 0x7f21d810e4c0 2024-05-09T08:43:17.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.007+0000 7f21d57fa640 1 -- 172.21.15.107:0/660409108 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21cc016070 con 0x7f21d810e4c0 2024-05-09T08:43:17.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.007+0000 7f21dd638640 1 -- 172.21.15.107:0/660409108 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d811a220 con 0x7f21d810e4c0 2024-05-09T08:43:17.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.007+0000 7f21d57fa640 1 -- 172.21.15.107:0/660409108 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc010dd0 con 0x7f21d810e4c0 2024-05-09T08:43:17.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.007+0000 7f21dd638640 1 -- 172.21.15.107:0/660409108 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d811a770 con 0x7f21d810e4c0 2024-05-09T08:43:17.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.010+0000 7f21e15a5740 1 Processor -- start 2024-05-09T08:43:17.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.013+0000 7f21d57fa640 1 -- 172.21.15.107:0/660409108 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f21cc0119d0 con 0x7f21d810e4c0 2024-05-09T08:43:17.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.014+0000 7f21d57fa640 1 --2- 172.21.15.107:0/660409108 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f21c00a2e80 0x7f21c00a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.015+0000 7f21d57fa640 1 -- 172.21.15.107:0/660409108 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f21cc0ceaf0 con 0x7f21d810e4c0 2024-05-09T08:43:17.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.016+0000 7f21e15a5740 1 -- start start 2024-05-09T08:43:17.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.016+0000 7f21e15a5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3ba2c0 0x55644f3b8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.016+0000 7f21e15a5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3b8bd0 0x55644f3b9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.016+0000 7f21e15a5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3c3950 0x55644f3c5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.016+0000 7f21e15a5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55644f31d990 con 0x55644f3c3950 2024-05-09T08:43:17.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.016+0000 7f21e15a5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55644f2c55b0 con 0x55644f3ba2c0 2024-05-09T08:43:17.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.016+0000 7f21e15a5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55644f319950 con 0x55644f3b8bd0 2024-05-09T08:43:17.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.017+0000 7f21d77fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3b8bd0 0x55644f3b9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.018+0000 7f21d77fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3b8bd0 0x55644f3b9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:60198/0 (socket says 172.21.15.107:60198) 2024-05-09T08:43:17.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.018+0000 7f21d77fe640 1 -- 172.21.15.107:0/4016435084 learned_addr learned my addr 172.21.15.107:0/4016435084 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:17.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.019+0000 7f21d77fe640 1 -- 172.21.15.107:0/4016435084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3ba2c0 msgr2=0x55644f3b8600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:43:17.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.019+0000 7f21d77fe640 1 --2- 172.21.15.107:0/4016435084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3ba2c0 0x55644f3b8600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.019+0000 7f21d77fe640 1 -- 172.21.15.107:0/4016435084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3c3950 msgr2=0x55644f3c5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.019+0000 7f21d77fe640 1 --2- 172.21.15.107:0/4016435084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3c3950 0x55644f3c5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.019+0000 7f21d77fe640 1 -- 172.21.15.107:0/4016435084 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55644f0f97e0 con 0x55644f3b8bd0 2024-05-09T08:43:17.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.019+0000 7f21d77fe640 1 --2- 172.21.15.107:0/4016435084 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3b8bd0 0x55644f3b9050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f21cc00b3f0 tx=0x7f21cc09b4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9c1829317c3b2cf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.025+0000 7f21be7fc640 1 -- 172.21.15.107:0/4016435084 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc0a3980 con 0x55644f3b8bd0 2024-05-09T08:43:17.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.025+0000 7f21be7fc640 1 -- 172.21.15.107:0/4016435084 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21cc0cc0b0 con 0x55644f3b8bd0 2024-05-09T08:43:17.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.025+0000 7f21be7fc640 1 -- 172.21.15.107:0/4016435084 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc096c80 con 0x55644f3b8bd0 2024-05-09T08:43:17.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.027+0000 7f21e15a5740 1 -- 172.21.15.107:0/4016435084 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3b8bd0 msgr2=0x55644f3b9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.027+0000 7f21e15a5740 1 --2- 172.21.15.107:0/4016435084 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3b8bd0 0x55644f3b9050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f21cc00b3f0 tx=0x7f21cc09b4d0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.028+0000 7f21dce37640 1 --2- 172.21.15.107:0/660409108 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f21c00a2e80 0x7f21c00a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.031+0000 7f21e15a5740 1 -- 172.21.15.107:0/4016435084 shutdown_connections 2024-05-09T08:43:17.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.031+0000 7f21e15a5740 1 --2- 172.21.15.107:0/4016435084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3c3950 0x55644f3c5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.031+0000 7f21e15a5740 1 --2- 172.21.15.107:0/4016435084 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3b8bd0 0x55644f3b9050 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.031+0000 7f21e15a5740 1 --2- 172.21.15.107:0/4016435084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3ba2c0 0x55644f3b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.031+0000 7f21e15a5740 1 -- 172.21.15.107:0/4016435084 >> 172.21.15.107:0/4016435084 conn(0x55644f3810a0 msgr2=0x55644f3b5130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.033+0000 7f21dce37640 1 --2- 172.21.15.107:0/660409108 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f21c00a2e80 0x7f21c00a5340 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x55644f3b62c0 tx=0x7f21d000b040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.038+0000 7f21e15a5740 1 -- 172.21.15.107:0/4016435084 shutdown_connections 2024-05-09T08:43:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.038+0000 7f21e15a5740 1 -- 172.21.15.107:0/4016435084 wait complete. 2024-05-09T08:43:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.039+0000 7f21e15a5740 1 -- start start 2024-05-09T08:43:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.039+0000 7f21e15a5740 1 Processor -- start 2024-05-09T08:43:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.039+0000 7f21e15a5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3b8bd0 0x55644f3c07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.040+0000 7f21e15a5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3ba2c0 0x55644f3c0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.040+0000 7f21e15a5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3c3950 0x55644f3c1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.040+0000 7f21e15a5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55644f3c67e0 con 0x55644f3ba2c0 2024-05-09T08:43:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.040+0000 7f21e15a5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55644f2d8b40 con 0x55644f3b8bd0 2024-05-09T08:43:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.040+0000 7f21e15a5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55644f3c6960 con 0x55644f3c3950 2024-05-09T08:43:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.040+0000 7f21e15a5740 10 client.0 _ll_register_callbacks cb 0x55644f2cd0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:17.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.041+0000 7f21dce37640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3c3950 0x55644f3c1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.042+0000 7f21dce37640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3c3950 0x55644f3c1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:60202/0 (socket says 172.21.15.107:60202) 2024-05-09T08:43:17.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.042+0000 7f21dce37640 1 -- 172.21.15.107:0/1356286667 learned_addr learned my addr 172.21.15.107:0/1356286667 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:17.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.042+0000 7f21d7fff640 1 --2- 172.21.15.107:0/1356286667 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3b8bd0 0x55644f3c07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.043+0000 7f21dce37640 1 -- 172.21.15.107:0/1356286667 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3b8bd0 msgr2=0x55644f3c07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.043+0000 7f21dce37640 1 --2- 172.21.15.107:0/1356286667 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3b8bd0 0x55644f3c07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.043+0000 7f21dce37640 1 -- 172.21.15.107:0/1356286667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3ba2c0 msgr2=0x55644f3c0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.043+0000 7f21dce37640 1 --2- 172.21.15.107:0/1356286667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3ba2c0 0x55644f3c0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.044+0000 7f21dce37640 1 -- 172.21.15.107:0/1356286667 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55644f159f70 con 0x55644f3c3950 2024-05-09T08:43:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.047+0000 7f21dce37640 1 --2- 172.21.15.107:0/1356286667 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3c3950 0x55644f3c1220 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f21d0008410 tx=0x7f21d0004b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d1358028da2c269 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.047+0000 7f21b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:43:17.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.048+0000 7f21b77fe640 1 -- 172.21.15.107:0/1356286667 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d0026060 con 0x55644f3c3950 2024-05-09T08:43:17.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.048+0000 7f21e15a5740 1 -- 172.21.15.107:0/1356286667 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21cc0cc0b0 con 0x55644f3c3950 2024-05-09T08:43:17.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.049+0000 7f21b77fe640 1 -- 172.21.15.107:0/1356286667 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f21d0016420 con 0x55644f3c3950 2024-05-09T08:43:17.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.049+0000 7f21b77fe640 1 -- 172.21.15.107:0/1356286667 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f21d002d050 con 0x55644f3c3950 2024-05-09T08:43:17.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.053+0000 7f21e15a5740 10 client.27676 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:17.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.053+0000 7f21e15a5740 1 -- 172.21.15.107:0/1356286667 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55644eff1170 con 0x55644f3c3950 2024-05-09T08:43:17.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.055+0000 7f21b77fe640 1 -- 172.21.15.107:0/1356286667 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21cc0cc0b0 con 0x55644f3c3950 2024-05-09T08:43:17.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.055+0000 7f21b77fe640 1 -- 172.21.15.107:0/1356286667 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f21d0021680 con 0x55644f3c3950 2024-05-09T08:43:17.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.056+0000 7f21e15a5740 10 client.27676 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:17.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.056+0000 7f21e15a5740 10 client.27676 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:17.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.056+0000 7f21e15a5740 20 client.27676 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:17.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.056+0000 7f21e15a5740 1 --2- 172.21.15.107:0/1356286667 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55644f3d3f40 0x55644f3d6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.056+0000 7f21e15a5740 4 client.27676 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:17.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.056+0000 7f21e15a5740 1 -- 172.21.15.107:0/1356286667 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55644f32b0c0 con 0x55644f3d3f40 2024-05-09T08:43:17.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.059+0000 7f21d7fff640 1 --2- 172.21.15.107:0/1356286667 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55644f3d3f40 0x55644f3d6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.065+0000 7f21d7fff640 1 --2- 172.21.15.107:0/1356286667 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55644f3d3f40 0x55644f3d6400 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a0461ac63402fc3 server_cookie=41adc003e57c957a in_seq=0 out_seq=0 2024-05-09T08:43:17.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.066+0000 7f21b77fe640 10 client.27676 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:17.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.068+0000 7f21b77fe640 1 -- 172.21.15.107:0/1356286667 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55644f32b0c0 con 0x55644f3d3f40 2024-05-09T08:43:17.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.068+0000 7f21b77fe640 10 client.27676 handle_command_reply: tid=0 2024-05-09T08:43:17.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.068+0000 7f21e15a5740 1 client.27676 shutdown 2024-05-09T08:43:17.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.074+0000 7f21e15a5740 1 -- 172.21.15.107:0/1356286667 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3c3950 msgr2=0x55644f3c1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.074+0000 7f21e15a5740 1 --2- 172.21.15.107:0/1356286667 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3c3950 0x55644f3c1220 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f21d0008410 tx=0x7f21d0004b50 comp rx=0 tx=0).stop 2024-05-09T08:43:17.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.074+0000 7f21e15a5740 1 -- 172.21.15.107:0/1356286667 shutdown_connections 2024-05-09T08:43:17.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.074+0000 7f21e15a5740 1 --2- 172.21.15.107:0/1356286667 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55644f3d3f40 0x55644f3d6400 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.074+0000 7f21e15a5740 1 --2- 172.21.15.107:0/1356286667 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3c3950 0x55644f3c1220 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.074+0000 7f21e15a5740 1 --2- 172.21.15.107:0/1356286667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3ba2c0 0x55644f3c0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.074+0000 7f21e15a5740 1 --2- 172.21.15.107:0/1356286667 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3b8bd0 0x55644f3c07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.074+0000 7f21e15a5740 1 -- 172.21.15.107:0/1356286667 >> 172.21.15.107:0/1356286667 conn(0x55644f3810a0 msgr2=0x55644f3ba880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.077+0000 7f21b77fe640 0 client.27676 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:17.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.079+0000 7f21e15a5740 1 -- 172.21.15.107:0/1356286667 shutdown_connections 2024-05-09T08:43:17.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.080+0000 7f21e15a5740 1 -- 172.21.15.107:0/1356286667 wait complete. 2024-05-09T08:43:17.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.080+0000 7f21e15a5740 20 client.27676 trim_cache size 0 max 16384 2024-05-09T08:43:17.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.126+0000 7f21e15a5740 1 Processor -- start 2024-05-09T08:43:17.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.130+0000 7f21e15a5740 1 -- start start 2024-05-09T08:43:17.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.130+0000 7f21e15a5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3b8bd0 0x55644f3cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.130+0000 7f21e15a5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3ba2c0 0x55644f3dbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.130+0000 7f21e15a5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3c3950 0x55644f3dc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.130+0000 7f21e15a5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f21c4012890 con 0x55644f3b8bd0 2024-05-09T08:43:17.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.130+0000 7f21e15a5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f21c4010160 con 0x55644f3ba2c0 2024-05-09T08:43:17.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.130+0000 7f21e15a5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f21c400dfd0 con 0x55644f3c3950 2024-05-09T08:43:17.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.131+0000 7f21d77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3ba2c0 0x55644f3dbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.131+0000 7f21d77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3ba2c0 0x55644f3dbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:46044/0 (socket says 172.21.15.107:46044) 2024-05-09T08:43:17.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.131+0000 7f21d77fe640 1 -- 172.21.15.107:0/921030472 learned_addr learned my addr 172.21.15.107:0/921030472 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:17.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.134+0000 7f21dce37640 1 --2- 172.21.15.107:0/921030472 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3c3950 0x55644f3dc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.135+0000 7f21d77fe640 1 -- 172.21.15.107:0/921030472 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3c3950 msgr2=0x55644f3dc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.135+0000 7f21d77fe640 1 --2- 172.21.15.107:0/921030472 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3c3950 0x55644f3dc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.135+0000 7f21d77fe640 1 -- 172.21.15.107:0/921030472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3b8bd0 msgr2=0x55644f3cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.135+0000 7f21d77fe640 1 --2- 172.21.15.107:0/921030472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3b8bd0 0x55644f3cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.135+0000 7f21d77fe640 1 -- 172.21.15.107:0/921030472 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55644f3d2630 con 0x55644f3ba2c0 2024-05-09T08:43:17.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.135+0000 7f21d77fe640 1 --2- 172.21.15.107:0/921030472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3ba2c0 0x55644f3dbcd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f21cc097170 tx=0x7f21cc09ac10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9e7a5fcc6ea449c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.135+0000 7f21bdffb640 1 -- 172.21.15.107:0/921030472 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc0a3980 con 0x55644f3ba2c0 2024-05-09T08:43:17.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.135+0000 7f21bdffb640 1 -- 172.21.15.107:0/921030472 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21cc0cc0b0 con 0x55644f3ba2c0 2024-05-09T08:43:17.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.136+0000 7f21bdffb640 1 -- 172.21.15.107:0/921030472 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc099e30 con 0x55644f3ba2c0 2024-05-09T08:43:17.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.137+0000 7f21e15a5740 1 -- 172.21.15.107:0/921030472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3ba2c0 msgr2=0x55644f3dbcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.137+0000 7f21e15a5740 1 --2- 172.21.15.107:0/921030472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3ba2c0 0x55644f3dbcd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f21cc097170 tx=0x7f21cc09ac10 comp rx=0 tx=0).stop 2024-05-09T08:43:17.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.146+0000 7f21e15a5740 1 -- 172.21.15.107:0/921030472 shutdown_connections 2024-05-09T08:43:17.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.146+0000 7f21e15a5740 1 --2- 172.21.15.107:0/921030472 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3c3950 0x55644f3dc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.146+0000 7f21e15a5740 1 --2- 172.21.15.107:0/921030472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3ba2c0 0x55644f3dbcd0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.146+0000 7f21e15a5740 1 --2- 172.21.15.107:0/921030472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3b8bd0 0x55644f3cf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.146+0000 7f21e15a5740 1 -- 172.21.15.107:0/921030472 >> 172.21.15.107:0/921030472 conn(0x55644f3810a0 msgr2=0x55644f3b4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.165+0000 7f21e15a5740 1 -- 172.21.15.107:0/921030472 shutdown_connections 2024-05-09T08:43:17.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.168+0000 7f21e15a5740 1 -- 172.21.15.107:0/921030472 wait complete. 2024-05-09T08:43:17.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.168+0000 7f21e15a5740 1 -- start start 2024-05-09T08:43:17.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.168+0000 7f21e15a5740 1 Processor -- start 2024-05-09T08:43:17.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.169+0000 7f21e15a5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3b8bd0 0x55644f3cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.169+0000 7f21e15a5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3ba2c0 0x55644f3c4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.169+0000 7f21e15a5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3c3950 0x55644f3c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.169+0000 7f21e15a5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f21c400e620 con 0x55644f3ba2c0 2024-05-09T08:43:17.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.169+0000 7f21e15a5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f21c400fa90 con 0x55644f3c3950 2024-05-09T08:43:17.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.169+0000 7f21e15a5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f21c400ecf0 con 0x55644f3b8bd0 2024-05-09T08:43:17.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.169+0000 7f21e15a5740 10 client.0 _ll_register_callbacks cb 0x7f21d0008600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:17.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.169+0000 7f21d77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3ba2c0 0x55644f3c4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.169+0000 7f21d77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3ba2c0 0x55644f3c4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55174/0 (socket says 172.21.15.107:55174) 2024-05-09T08:43:17.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.169+0000 7f21d77fe640 1 -- 172.21.15.107:0/2079900232 learned_addr learned my addr 172.21.15.107:0/2079900232 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:17.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.175+0000 7f21dce37640 1 --2- 172.21.15.107:0/2079900232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3c3950 0x55644f3c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.175+0000 7f21d7fff640 1 --2- 172.21.15.107:0/2079900232 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3b8bd0 0x55644f3cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.176+0000 7f21d7fff640 1 -- 172.21.15.107:0/2079900232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3c3950 msgr2=0x55644f3c46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.176+0000 7f21d7fff640 1 --2- 172.21.15.107:0/2079900232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3c3950 0x55644f3c46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.176+0000 7f21d7fff640 1 -- 172.21.15.107:0/2079900232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3ba2c0 msgr2=0x55644f3c4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.176+0000 7f21d7fff640 1 --2- 172.21.15.107:0/2079900232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3ba2c0 0x55644f3c4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.176+0000 7f21d7fff640 1 -- 172.21.15.107:0/2079900232 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55644f3c0570 con 0x55644f3b8bd0 2024-05-09T08:43:17.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.177+0000 7f21d7fff640 1 --2- 172.21.15.107:0/2079900232 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3b8bd0 0x55644f3cbf10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f21c802aa70 tx=0x7f21c8004310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bde43eccb034e57 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.177+0000 7f21b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:43:17.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.177+0000 7f21b77fe640 1 -- 172.21.15.107:0/2079900232 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c8035040 con 0x55644f3b8bd0 2024-05-09T08:43:17.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.177+0000 7f21b77fe640 1 -- 172.21.15.107:0/2079900232 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f21c8041070 con 0x55644f3b8bd0 2024-05-09T08:43:17.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.177+0000 7f21e15a5740 1 -- 172.21.15.107:0/2079900232 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21cc0cc0b0 con 0x55644f3b8bd0 2024-05-09T08:43:17.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.178+0000 7f21b77fe640 1 -- 172.21.15.107:0/2079900232 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f21c8004490 con 0x55644f3b8bd0 2024-05-09T08:43:17.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.179+0000 7f21e15a5740 10 client.27679 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:17.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.179+0000 7f21e15a5740 1 -- 172.21.15.107:0/2079900232 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55644f3cb9d0 con 0x55644f3b8bd0 2024-05-09T08:43:17.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.181+0000 7f21b77fe640 1 -- 172.21.15.107:0/2079900232 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21cc0cc0b0 con 0x55644f3b8bd0 2024-05-09T08:43:17.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.181+0000 7f21b77fe640 1 -- 172.21.15.107:0/2079900232 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f21c8003680 con 0x55644f3b8bd0 2024-05-09T08:43:17.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.181+0000 7f21e15a5740 10 client.27679 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:17.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.181+0000 7f21e15a5740 10 client.27679 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:17.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.181+0000 7f21e15a5740 20 client.27679 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:17.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.181+0000 7f21e15a5740 1 --2- 172.21.15.107:0/2079900232 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55644f43a940 0x55644f43cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.181+0000 7f21e15a5740 4 client.27679 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:43:17.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.181+0000 7f21e15a5740 1 -- 172.21.15.107:0/2079900232 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55644f325410 con 0x55644f43a940 2024-05-09T08:43:17.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.183+0000 7f21d77fe640 1 --2- 172.21.15.107:0/2079900232 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55644f43a940 0x55644f43cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.186+0000 7f21d77fe640 1 --2- 172.21.15.107:0/2079900232 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55644f43a940 0x55644f43cd60 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9901c70223939ad server_cookie=6eda85c3dd9fb09c in_seq=0 out_seq=0 2024-05-09T08:43:17.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.186+0000 7f21b77fe640 10 client.27679 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:17.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.189+0000 7f21b77fe640 1 -- 172.21.15.107:0/2079900232 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55644f325410 con 0x55644f43a940 2024-05-09T08:43:17.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.189+0000 7f21b77fe640 10 client.27679 handle_command_reply: tid=0 2024-05-09T08:43:17.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.189+0000 7f21e15a5740 1 client.27679 shutdown 2024-05-09T08:43:17.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.193+0000 7f21e15a5740 1 -- 172.21.15.107:0/2079900232 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3b8bd0 msgr2=0x55644f3cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.193+0000 7f21e15a5740 1 --2- 172.21.15.107:0/2079900232 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3b8bd0 0x55644f3cbf10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f21c802aa70 tx=0x7f21c8004310 comp rx=0 tx=0).stop 2024-05-09T08:43:17.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.193+0000 7f21e15a5740 1 -- 172.21.15.107:0/2079900232 shutdown_connections 2024-05-09T08:43:17.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.193+0000 7f21e15a5740 1 --2- 172.21.15.107:0/2079900232 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55644f43a940 0x55644f43cd60 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.193+0000 7f21e15a5740 1 --2- 172.21.15.107:0/2079900232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55644f3c3950 0x55644f3c46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.193+0000 7f21e15a5740 1 --2- 172.21.15.107:0/2079900232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55644f3ba2c0 0x55644f3c4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.193+0000 7f21e15a5740 1 --2- 172.21.15.107:0/2079900232 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55644f3b8bd0 0x55644f3cbf10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.193+0000 7f21e15a5740 1 -- 172.21.15.107:0/2079900232 >> 172.21.15.107:0/2079900232 conn(0x55644f3810a0 msgr2=0x55644f3bf030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.193+0000 7f21b77fe640 0 client.27679 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:17.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.193+0000 7f21e15a5740 1 -- 172.21.15.107:0/2079900232 shutdown_connections 2024-05-09T08:43:17.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.193+0000 7f21e15a5740 1 -- 172.21.15.107:0/2079900232 wait complete. 2024-05-09T08:43:17.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.194+0000 7f21e15a5740 20 client.27679 trim_cache size 0 max 16384 2024-05-09T08:43:17.194 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:43:17.194 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:43:17.194 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "ca1e0a3f-d43f-4ef6-ac55-56a6bb6eab3b", 2024-05-09T08:43:17.194 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:43:17.194 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:43:17.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.199+0000 7f21dd638640 1 -- 172.21.15.107:0/660409108 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f21c00a2e80 msgr2=0x7f21c00a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.199+0000 7f21dd638640 1 --2- 172.21.15.107:0/660409108 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f21c00a2e80 0x7f21c00a5340 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x55644f3b62c0 tx=0x7f21d000b040 comp rx=0 tx=0).stop 2024-05-09T08:43:17.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.199+0000 7f21dd638640 1 -- 172.21.15.107:0/660409108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21d810e4c0 msgr2=0x7f21d8115780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.199+0000 7f21dd638640 1 --2- 172.21.15.107:0/660409108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21d810e4c0 0x7f21d8115780 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f21cc00b420 tx=0x7f21cc00b8e0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.199+0000 7f21dd638640 1 -- 172.21.15.107:0/660409108 shutdown_connections 2024-05-09T08:43:17.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.199+0000 7f21dd638640 1 --2- 172.21.15.107:0/660409108 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f21c00a2e80 0x7f21c00a5340 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.199+0000 7f21dd638640 1 --2- 172.21.15.107:0/660409108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810fe40 0x7f21d8115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.199+0000 7f21dd638640 1 --2- 172.21.15.107:0/660409108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21d810e4c0 0x7f21d8115780 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.199+0000 7f21dd638640 1 --2- 172.21.15.107:0/660409108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21d810d910 0x7f21d811c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.200+0000 7f21dd638640 1 -- 172.21.15.107:0/660409108 >> 172.21.15.107:0/660409108 conn(0x7f21d806c710 msgr2=0x7f21d8070cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.203+0000 7f21dd638640 1 -- 172.21.15.107:0/660409108 shutdown_connections 2024-05-09T08:43:17.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.203+0000 7f21dd638640 1 -- 172.21.15.107:0/660409108 wait complete. 2024-05-09T08:43:17.233 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:43:17.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:43:17.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.538+0000 7fec5e6a5640 1 -- 172.21.15.107:0/2506866844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec58100a90 msgr2=0x7fec58104ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.538+0000 7fec5e6a5640 1 --2- 172.21.15.107:0/2506866844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec58100a90 0x7fec58104ad0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fec40005fd0 tx=0x7fec4002c3c0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.539+0000 7fec5e6a5640 1 -- 172.21.15.107:0/2506866844 shutdown_connections 2024-05-09T08:43:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.539+0000 7fec5e6a5640 1 --2- 172.21.15.107:0/2506866844 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec581051b0 0x7fec5810cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.539+0000 7fec5e6a5640 1 --2- 172.21.15.107:0/2506866844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec58100a90 0x7fec58104ad0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.539+0000 7fec5e6a5640 1 --2- 172.21.15.107:0/2506866844 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fec581000e0 0x7fec581004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.539+0000 7fec5e6a5640 1 -- 172.21.15.107:0/2506866844 >> 172.21.15.107:0/2506866844 conn(0x7fec58076cd0 msgr2=0x7fec580fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.539+0000 7fec5e6a5640 1 -- 172.21.15.107:0/2506866844 shutdown_connections 2024-05-09T08:43:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.539+0000 7fec5e6a5640 1 -- 172.21.15.107:0/2506866844 wait complete. 2024-05-09T08:43:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.540+0000 7fec5e6a5640 1 Processor -- start 2024-05-09T08:43:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.540+0000 7fec5e6a5640 1 -- start start 2024-05-09T08:43:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.541+0000 7fec5e6a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec581000e0 0x7fec58073150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.541+0000 7fec5e6a5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec581051b0 0x7fec5806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.541+0000 7fec5e6a5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fec5806d750 0x7fec5806dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.541+0000 7fec5e6a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fec5806bb10 con 0x7fec581000e0 2024-05-09T08:43:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.541+0000 7fec5e6a5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fec5806b990 con 0x7fec581051b0 2024-05-09T08:43:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.541+0000 7fec5e6a5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fec5806bc90 con 0x7fec5806d750 2024-05-09T08:43:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.541+0000 7fec57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec581000e0 0x7fec58073150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.541+0000 7fec5cc42640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fec5806d750 0x7fec5806dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.541+0000 7fec57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec581000e0 0x7fec58073150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55186/0 (socket says 172.21.15.107:55186) 2024-05-09T08:43:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.541+0000 7fec57fff640 1 -- 172.21.15.107:0/201541633 learned_addr learned my addr 172.21.15.107:0/201541633 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.541+0000 7fec577fe640 1 --2- 172.21.15.107:0/201541633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec581051b0 0x7fec5806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.542+0000 7fec5cc42640 1 -- 172.21.15.107:0/201541633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec581051b0 msgr2=0x7fec5806d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.542+0000 7fec5cc42640 1 --2- 172.21.15.107:0/201541633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec581051b0 0x7fec5806d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.542+0000 7fec5cc42640 1 -- 172.21.15.107:0/201541633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec581000e0 msgr2=0x7fec58073150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.542+0000 7fec5cc42640 1 --2- 172.21.15.107:0/201541633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec581000e0 0x7fec58073150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.542+0000 7fec5cc42640 1 -- 172.21.15.107:0/201541633 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec581b3580 con 0x7fec5806d750 2024-05-09T08:43:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.542+0000 7fec5cc42640 1 --2- 172.21.15.107:0/201541633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fec5806d750 0x7fec5806dc00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fec5000ef00 tx=0x7fec5000c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.542+0000 7fec557fa640 1 -- 172.21.15.107:0/201541633 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50004b20 con 0x7fec5806d750 2024-05-09T08:43:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.542+0000 7fec557fa640 1 -- 172.21.15.107:0/201541633 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec50003e20 con 0x7fec5806d750 2024-05-09T08:43:17.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.542+0000 7fec557fa640 1 -- 172.21.15.107:0/201541633 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5000b570 con 0x7fec5806d750 2024-05-09T08:43:17.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.543+0000 7fec5e6a5640 1 -- 172.21.15.107:0/201541633 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec581b3810 con 0x7fec5806d750 2024-05-09T08:43:17.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.543+0000 7fec5e6a5640 1 -- 172.21.15.107:0/201541633 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec581b3c80 con 0x7fec5806d750 2024-05-09T08:43:17.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.546+0000 7fec613b2740 1 Processor -- start 2024-05-09T08:43:17.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.547+0000 7fec557fa640 1 -- 172.21.15.107:0/201541633 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fec5001d020 con 0x7fec5806d750 2024-05-09T08:43:17.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.547+0000 7fec613b2740 1 -- start start 2024-05-09T08:43:17.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.547+0000 7fec613b2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714280 0x55ab2b714660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.547+0000 7fec613b2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714ba0 0x55ab2b71ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.547+0000 7fec613b2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b71f270 0x55ab2b721660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.547+0000 7fec613b2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ab2b679d00 con 0x55ab2b71f270 2024-05-09T08:43:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.547+0000 7fec613b2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ab2b6214c0 con 0x55ab2b714ba0 2024-05-09T08:43:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.547+0000 7fec613b2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ab2b675cc0 con 0x55ab2b714280 2024-05-09T08:43:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.547+0000 7fec57fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714ba0 0x55ab2b71ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.547+0000 7fec57fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714ba0 0x55ab2b71ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:46076/0 (socket says 172.21.15.107:46076) 2024-05-09T08:43:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.547+0000 7fec57fff640 1 -- 172.21.15.107:0/3901312494 learned_addr learned my addr 172.21.15.107:0/3901312494 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.547+0000 7fec577fe640 1 --2- 172.21.15.107:0/3901312494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714280 0x55ab2b714660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.547+0000 7fec577fe640 1 --2- 172.21.15.107:0/3901312494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b71f270 0x55ab2b721660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.548+0000 7fec57fff640 1 -- 172.21.15.107:0/3901312494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714280 msgr2=0x55ab2b714660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.548+0000 7fec57fff640 1 --2- 172.21.15.107:0/3901312494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714280 0x55ab2b714660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.548+0000 7fec57fff640 1 -- 172.21.15.107:0/3901312494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b71f270 msgr2=0x55ab2b721660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.548+0000 7fec57fff640 1 --2- 172.21.15.107:0/3901312494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b71f270 0x55ab2b721660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.548+0000 7fec57fff640 1 -- 172.21.15.107:0/3901312494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab2b4577d0 con 0x55ab2b714ba0 2024-05-09T08:43:17.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.548+0000 7fec577fe640 1 --2- 172.21.15.107:0/3901312494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b71f270 0x55ab2b721660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:43:17.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.548+0000 7fec557fa640 1 --2- 172.21.15.107:0/201541633 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fec440a2e40 0x7fec440a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.548+0000 7fec57fff640 1 --2- 172.21.15.107:0/3901312494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714ba0 0x55ab2b71ed30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fec4800bdd0 tx=0x7fec480107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d4a0b769b45f7a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.548+0000 7fec557fa640 1 -- 172.21.15.107:0/201541633 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fec500cf5b0 con 0x7fec5806d750 2024-05-09T08:43:17.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.549+0000 7fec4e7fc640 1 -- 172.21.15.107:0/3901312494 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec48005220 con 0x55ab2b714ba0 2024-05-09T08:43:17.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.549+0000 7fec4e7fc640 1 -- 172.21.15.107:0/3901312494 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec48004c50 con 0x55ab2b714ba0 2024-05-09T08:43:17.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.549+0000 7fec4e7fc640 1 -- 172.21.15.107:0/3901312494 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec48011800 con 0x55ab2b714ba0 2024-05-09T08:43:17.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.549+0000 7fec577fe640 1 --2- 172.21.15.107:0/201541633 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fec440a2e40 0x7fec440a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.549+0000 7fec613b2740 1 -- 172.21.15.107:0/3901312494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714ba0 msgr2=0x55ab2b71ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.549+0000 7fec613b2740 1 --2- 172.21.15.107:0/3901312494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714ba0 0x55ab2b71ed30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fec4800bdd0 tx=0x7fec480107a0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.550+0000 7fec613b2740 1 -- 172.21.15.107:0/3901312494 shutdown_connections 2024-05-09T08:43:17.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.550+0000 7fec577fe640 1 --2- 172.21.15.107:0/201541633 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fec440a2e40 0x7fec440a5300 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fec5806ea10 tx=0x7fec400045c0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.550+0000 7fec613b2740 1 --2- 172.21.15.107:0/3901312494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b71f270 0x55ab2b721660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.550+0000 7fec613b2740 1 --2- 172.21.15.107:0/3901312494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714ba0 0x55ab2b71ed30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.550+0000 7fec613b2740 1 --2- 172.21.15.107:0/3901312494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714280 0x55ab2b714660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.550+0000 7fec613b2740 1 -- 172.21.15.107:0/3901312494 >> 172.21.15.107:0/3901312494 conn(0x55ab2b7105a0 msgr2=0x55ab2b710980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.551+0000 7fec613b2740 1 -- 172.21.15.107:0/3901312494 shutdown_connections 2024-05-09T08:43:17.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.551+0000 7fec613b2740 1 -- 172.21.15.107:0/3901312494 wait complete. 2024-05-09T08:43:17.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.551+0000 7fec613b2740 1 -- start start 2024-05-09T08:43:17.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.552+0000 7fec613b2740 1 Processor -- start 2024-05-09T08:43:17.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.552+0000 7fec613b2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714280 0x55ab2b71c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.553+0000 7fec613b2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714ba0 0x55ab2b71c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.553+0000 7fec57fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714280 0x55ab2b71c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.553+0000 7fec57fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714280 0x55ab2b71c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:46086/0 (socket says 172.21.15.107:46086) 2024-05-09T08:43:17.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.553+0000 7fec57fff640 1 -- 172.21.15.107:0/4225251522 learned_addr learned my addr 172.21.15.107:0/4225251522 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:17.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.553+0000 7fec613b2740 1 --2- 172.21.15.107:0/4225251522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b71f270 0x55ab2b71cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.553+0000 7fec613b2740 1 -- 172.21.15.107:0/4225251522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ab2b722060 con 0x55ab2b71f270 2024-05-09T08:43:17.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.553+0000 7fec613b2740 1 -- 172.21.15.107:0/4225251522 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ab2b634b90 con 0x55ab2b714280 2024-05-09T08:43:17.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.553+0000 7fec613b2740 1 -- 172.21.15.107:0/4225251522 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ab2b7221e0 con 0x55ab2b714ba0 2024-05-09T08:43:17.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.553+0000 7fec577fe640 1 --2- 172.21.15.107:0/4225251522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714ba0 0x55ab2b71c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.553+0000 7fec613b2740 10 client.0 _ll_register_callbacks cb 0x55ab2b670da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:17.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.553+0000 7fec5cc42640 1 --2- 172.21.15.107:0/4225251522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b71f270 0x55ab2b71cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.553+0000 7fec57fff640 1 -- 172.21.15.107:0/4225251522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714ba0 msgr2=0x55ab2b71c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.553+0000 7fec57fff640 1 --2- 172.21.15.107:0/4225251522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714ba0 0x55ab2b71c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.553+0000 7fec57fff640 1 -- 172.21.15.107:0/4225251522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b71f270 msgr2=0x55ab2b71cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.553+0000 7fec57fff640 1 --2- 172.21.15.107:0/4225251522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b71f270 0x55ab2b71cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.553+0000 7fec57fff640 1 -- 172.21.15.107:0/4225251522 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab2b38d3f0 con 0x55ab2b714280 2024-05-09T08:43:17.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.554+0000 7fec57fff640 1 --2- 172.21.15.107:0/4225251522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714280 0x55ab2b71c220 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fec48010ca0 tx=0x7fec480048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b25d203b366c4633 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.554+0000 7fec377fe640 10 client.27446 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:43:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.554+0000 7fec613b2740 1 -- 172.21.15.107:0/4225251522 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec48004c50 con 0x55ab2b714280 2024-05-09T08:43:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.554+0000 7fec377fe640 1 -- 172.21.15.107:0/4225251522 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec48005000 con 0x55ab2b714280 2024-05-09T08:43:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.554+0000 7fec377fe640 1 -- 172.21.15.107:0/4225251522 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fec480051a0 con 0x55ab2b714280 2024-05-09T08:43:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.554+0000 7fec377fe640 1 -- 172.21.15.107:0/4225251522 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fec480053e0 con 0x55ab2b714280 2024-05-09T08:43:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.554+0000 7fec613b2740 10 client.27446 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.554+0000 7fec613b2740 1 -- 172.21.15.107:0/4225251522 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55ab2b34d0f0 con 0x55ab2b714280 2024-05-09T08:43:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.555+0000 7fec377fe640 1 -- 172.21.15.107:0/4225251522 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fec48011540 con 0x55ab2b714280 2024-05-09T08:43:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.555+0000 7fec613b2740 10 client.27446 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.555+0000 7fec613b2740 10 client.27446 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.555+0000 7fec613b2740 20 client.27446 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.555+0000 7fec613b2740 1 --2- 172.21.15.107:0/4225251522 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ab2b72f3b0 0x55ab2b7317a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.555+0000 7fec613b2740 4 client.27446 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.555+0000 7fec613b2740 1 -- 172.21.15.107:0/4225251522 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fec48004c50 con 0x55ab2b72f3b0 2024-05-09T08:43:17.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.555+0000 7fec577fe640 1 --2- 172.21.15.107:0/4225251522 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ab2b72f3b0 0x55ab2b7317a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.556+0000 7fec377fe640 10 client.27446 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:17.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.556+0000 7fec577fe640 1 --2- 172.21.15.107:0/4225251522 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ab2b72f3b0 0x55ab2b7317a0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e256d0d1da1380b server_cookie=edc539ada78a3aae in_seq=0 out_seq=0 2024-05-09T08:43:17.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.557+0000 7fec377fe640 1 -- 172.21.15.107:0/4225251522 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fec48004c50 con 0x55ab2b72f3b0 2024-05-09T08:43:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.557+0000 7fec377fe640 10 client.27446 handle_command_reply: tid=0 2024-05-09T08:43:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.557+0000 7fec613b2740 1 client.27446 shutdown 2024-05-09T08:43:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.557+0000 7fec613b2740 1 -- 172.21.15.107:0/4225251522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714280 msgr2=0x55ab2b71c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.557+0000 7fec613b2740 1 --2- 172.21.15.107:0/4225251522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714280 0x55ab2b71c220 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fec48010ca0 tx=0x7fec480048f0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.557+0000 7fec613b2740 1 -- 172.21.15.107:0/4225251522 shutdown_connections 2024-05-09T08:43:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.557+0000 7fec613b2740 1 --2- 172.21.15.107:0/4225251522 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ab2b72f3b0 0x55ab2b7317a0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.557+0000 7fec613b2740 1 --2- 172.21.15.107:0/4225251522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b71f270 0x55ab2b71cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.557+0000 7fec613b2740 1 --2- 172.21.15.107:0/4225251522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714ba0 0x55ab2b71c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.557+0000 7fec613b2740 1 --2- 172.21.15.107:0/4225251522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714280 0x55ab2b71c220 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.558+0000 7fec613b2740 1 -- 172.21.15.107:0/4225251522 >> 172.21.15.107:0/4225251522 conn(0x55ab2b7105a0 msgr2=0x55ab2b7163a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.558+0000 7fec377fe640 0 client.27446 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.559+0000 7fec613b2740 1 -- 172.21.15.107:0/4225251522 shutdown_connections 2024-05-09T08:43:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.559+0000 7fec613b2740 1 -- 172.21.15.107:0/4225251522 wait complete. 2024-05-09T08:43:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.560+0000 7fec613b2740 20 client.27446 trim_cache size 0 max 16384 2024-05-09T08:43:17.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.592+0000 7fec613b2740 1 Processor -- start 2024-05-09T08:43:17.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.592+0000 7fec613b2740 1 -- start start 2024-05-09T08:43:17.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.593+0000 7fec613b2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714280 0x55ab2b72af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.593+0000 7fec613b2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b714ba0 0x55ab2b71b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.593+0000 7fec57fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714280 0x55ab2b72af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.593+0000 7fec613b2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b71f270 0x55ab2b716a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.593+0000 7fec57fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714280 0x55ab2b72af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:60262/0 (socket says 172.21.15.107:60262) 2024-05-09T08:43:17.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.593+0000 7fec57fff640 1 -- 172.21.15.107:0/1172911273 learned_addr learned my addr 172.21.15.107:0/1172911273 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:17.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.593+0000 7fec613b2740 1 -- 172.21.15.107:0/1172911273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fec581c04d0 con 0x55ab2b714ba0 2024-05-09T08:43:17.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.593+0000 7fec613b2740 1 -- 172.21.15.107:0/1172911273 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fec581bdda0 con 0x55ab2b71f270 2024-05-09T08:43:17.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.593+0000 7fec613b2740 1 -- 172.21.15.107:0/1172911273 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fec581bbc10 con 0x55ab2b714280 2024-05-09T08:43:17.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.593+0000 7fec577fe640 1 --2- 172.21.15.107:0/1172911273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b714ba0 0x55ab2b71b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.594+0000 7fec5cc42640 1 --2- 172.21.15.107:0/1172911273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b71f270 0x55ab2b716a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.594+0000 7fec57fff640 1 -- 172.21.15.107:0/1172911273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b71f270 msgr2=0x55ab2b716a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.594+0000 7fec57fff640 1 --2- 172.21.15.107:0/1172911273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b71f270 0x55ab2b716a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.594+0000 7fec57fff640 1 -- 172.21.15.107:0/1172911273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b714ba0 msgr2=0x55ab2b71b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.594+0000 7fec57fff640 1 --2- 172.21.15.107:0/1172911273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b714ba0 0x55ab2b71b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.594+0000 7fec57fff640 1 -- 172.21.15.107:0/1172911273 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab2b717e00 con 0x55ab2b714280 2024-05-09T08:43:17.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.594+0000 7fec57fff640 1 --2- 172.21.15.107:0/1172911273 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714280 0x55ab2b72af20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fec48011a90 tx=0x7fec48022000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39f8efb3a49f24cc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.595+0000 7fec4dffb640 1 -- 172.21.15.107:0/1172911273 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec48032060 con 0x55ab2b714280 2024-05-09T08:43:17.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.595+0000 7fec4dffb640 1 -- 172.21.15.107:0/1172911273 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec48028070 con 0x55ab2b714280 2024-05-09T08:43:17.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.595+0000 7fec4dffb640 1 -- 172.21.15.107:0/1172911273 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4802d450 con 0x55ab2b714280 2024-05-09T08:43:17.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.596+0000 7fec613b2740 1 -- 172.21.15.107:0/1172911273 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714280 msgr2=0x55ab2b72af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.596+0000 7fec613b2740 1 --2- 172.21.15.107:0/1172911273 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714280 0x55ab2b72af20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fec48011a90 tx=0x7fec48022000 comp rx=0 tx=0).stop 2024-05-09T08:43:17.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.596+0000 7fec613b2740 1 -- 172.21.15.107:0/1172911273 shutdown_connections 2024-05-09T08:43:17.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.596+0000 7fec613b2740 1 --2- 172.21.15.107:0/1172911273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b71f270 0x55ab2b716a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.596+0000 7fec613b2740 1 --2- 172.21.15.107:0/1172911273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b714ba0 0x55ab2b71b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.596+0000 7fec613b2740 1 --2- 172.21.15.107:0/1172911273 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b714280 0x55ab2b72af20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.596+0000 7fec613b2740 1 -- 172.21.15.107:0/1172911273 >> 172.21.15.107:0/1172911273 conn(0x55ab2b7105a0 msgr2=0x55ab2b70ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.596+0000 7fec613b2740 1 -- 172.21.15.107:0/1172911273 shutdown_connections 2024-05-09T08:43:17.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.596+0000 7fec613b2740 1 -- 172.21.15.107:0/1172911273 wait complete. 2024-05-09T08:43:17.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.596+0000 7fec613b2740 1 -- start start 2024-05-09T08:43:17.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.596+0000 7fec613b2740 1 Processor -- start 2024-05-09T08:43:17.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec613b2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714280 0x55ab2b727620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec613b2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b714ba0 0x55ab2b718ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec613b2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b71f270 0x55ab2b718ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec613b2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fec581bc260 con 0x55ab2b714ba0 2024-05-09T08:43:17.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec613b2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fec581bd6d0 con 0x55ab2b714280 2024-05-09T08:43:17.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec613b2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fec581bc930 con 0x55ab2b71f270 2024-05-09T08:43:17.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec57fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714280 0x55ab2b727620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec57fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714280 0x55ab2b727620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:46090/0 (socket says 172.21.15.107:46090) 2024-05-09T08:43:17.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec57fff640 1 -- 172.21.15.107:0/1686415377 learned_addr learned my addr 172.21.15.107:0/1686415377 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:17.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec613b2740 10 client.0 _ll_register_callbacks cb 0x7fec4000d870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:17.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec5cc42640 1 --2- 172.21.15.107:0/1686415377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b71f270 0x55ab2b718ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec577fe640 1 --2- 172.21.15.107:0/1686415377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b714ba0 0x55ab2b718ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec57fff640 1 -- 172.21.15.107:0/1686415377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b71f270 msgr2=0x55ab2b718ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec57fff640 1 --2- 172.21.15.107:0/1686415377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b71f270 0x55ab2b718ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec57fff640 1 -- 172.21.15.107:0/1686415377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b714ba0 msgr2=0x55ab2b718ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec57fff640 1 --2- 172.21.15.107:0/1686415377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b714ba0 0x55ab2b718ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec57fff640 1 -- 172.21.15.107:0/1686415377 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab2b7195b0 con 0x55ab2b714280 2024-05-09T08:43:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.597+0000 7fec5cc42640 1 --2- 172.21.15.107:0/1686415377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b71f270 0x55ab2b718ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:43:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.599+0000 7fec57fff640 1 --2- 172.21.15.107:0/1686415377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714280 0x55ab2b727620 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fec48005bc0 tx=0x7fec4800b500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7af2f801df5fac9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.599+0000 7fec377fe640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:43:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.599+0000 7fec377fe640 1 -- 172.21.15.107:0/1686415377 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4803c070 con 0x55ab2b714280 2024-05-09T08:43:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.599+0000 7fec377fe640 1 -- 172.21.15.107:0/1686415377 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fec48005000 con 0x55ab2b714280 2024-05-09T08:43:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.599+0000 7fec613b2740 1 -- 172.21.15.107:0/1686415377 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec48028070 con 0x55ab2b714280 2024-05-09T08:43:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.599+0000 7fec377fe640 1 -- 172.21.15.107:0/1686415377 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fec48005200 con 0x55ab2b714280 2024-05-09T08:43:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.599+0000 7fec613b2740 10 client.27452 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:17.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.599+0000 7fec613b2740 1 -- 172.21.15.107:0/1686415377 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ab2b795530 con 0x55ab2b714280 2024-05-09T08:43:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.600+0000 7fec377fe640 1 -- 172.21.15.107:0/1686415377 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec4802d900 con 0x55ab2b714280 2024-05-09T08:43:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.600+0000 7fec377fe640 1 -- 172.21.15.107:0/1686415377 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fec48036660 con 0x55ab2b714280 2024-05-09T08:43:17.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.600+0000 7fec613b2740 10 client.27452 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.600+0000 7fec613b2740 10 client.27452 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.600+0000 7fec613b2740 20 client.27452 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.600+0000 7fec613b2740 1 --2- 172.21.15.107:0/1686415377 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ab2b795a50 0x55ab2b797e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.600+0000 7fec613b2740 4 client.27452 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:43:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.600+0000 7fec613b2740 1 -- 172.21.15.107:0/1686415377 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fec48028070 con 0x55ab2b795a50 2024-05-09T08:43:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.600+0000 7fec577fe640 1 --2- 172.21.15.107:0/1686415377 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ab2b795a50 0x55ab2b797e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.601+0000 7fec577fe640 1 --2- 172.21.15.107:0/1686415377 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ab2b795a50 0x55ab2b797e40 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b1e7d6250fbbe92 server_cookie=6190490c0917edca in_seq=0 out_seq=0 2024-05-09T08:43:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.601+0000 7fec377fe640 10 client.27452 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.601+0000 7fec377fe640 1 -- 172.21.15.107:0/1686415377 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x7fec48028070 con 0x55ab2b795a50 2024-05-09T08:43:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.601+0000 7fec377fe640 10 client.27452 handle_command_reply: tid=0 2024-05-09T08:43:17.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.601+0000 7fec613b2740 1 client.27452 shutdown 2024-05-09T08:43:17.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.603+0000 7fec613b2740 1 -- 172.21.15.107:0/1686415377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714280 msgr2=0x55ab2b727620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.603+0000 7fec613b2740 1 --2- 172.21.15.107:0/1686415377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714280 0x55ab2b727620 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fec48005bc0 tx=0x7fec4800b500 comp rx=0 tx=0).stop 2024-05-09T08:43:17.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.603+0000 7fec613b2740 1 -- 172.21.15.107:0/1686415377 shutdown_connections 2024-05-09T08:43:17.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.603+0000 7fec613b2740 1 --2- 172.21.15.107:0/1686415377 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ab2b795a50 0x55ab2b797e40 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.603+0000 7fec613b2740 1 --2- 172.21.15.107:0/1686415377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ab2b71f270 0x55ab2b718ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.603+0000 7fec377fe640 0 client.27452 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:17.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.603+0000 7fec613b2740 1 --2- 172.21.15.107:0/1686415377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab2b714ba0 0x55ab2b718ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.603+0000 7fec613b2740 1 --2- 172.21.15.107:0/1686415377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ab2b714280 0x55ab2b727620 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.603+0000 7fec613b2740 1 -- 172.21.15.107:0/1686415377 >> 172.21.15.107:0/1686415377 conn(0x55ab2b7105a0 msgr2=0x55ab2b71b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.604+0000 7fec613b2740 1 -- 172.21.15.107:0/1686415377 shutdown_connections 2024-05-09T08:43:17.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.604+0000 7fec613b2740 1 -- 172.21.15.107:0/1686415377 wait complete. 2024-05-09T08:43:17.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.604+0000 7fec613b2740 20 client.27452 trim_cache size 0 max 16384 2024-05-09T08:43:17.605 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:43:17.605 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "scrub active (155 inodes in the stack)", 2024-05-09T08:43:17.605 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:43:17.605 INFO:teuthology.orchestra.run.smithi107.stdout: "ca1e0a3f-d43f-4ef6-ac55-56a6bb6eab3b": { 2024-05-09T08:43:17.605 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:43:17.605 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "ca1e0a3f-d43f-4ef6-ac55-56a6bb6eab3b", 2024-05-09T08:43:17.605 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:43:17.606 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:43:17.606 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:43:17.606 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:43:17.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.605+0000 7fec5e6a5640 1 -- 172.21.15.107:0/201541633 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fec440a2e40 msgr2=0x7fec440a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.605+0000 7fec5e6a5640 1 --2- 172.21.15.107:0/201541633 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fec440a2e40 0x7fec440a5300 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fec5806ea10 tx=0x7fec400045c0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.605+0000 7fec5e6a5640 1 -- 172.21.15.107:0/201541633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fec5806d750 msgr2=0x7fec5806dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:17.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.605+0000 7fec5e6a5640 1 --2- 172.21.15.107:0/201541633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fec5806d750 0x7fec5806dc00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fec5000ef00 tx=0x7fec5000c740 comp rx=0 tx=0).stop 2024-05-09T08:43:17.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.605+0000 7fec5e6a5640 1 -- 172.21.15.107:0/201541633 shutdown_connections 2024-05-09T08:43:17.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.605+0000 7fec5e6a5640 1 --2- 172.21.15.107:0/201541633 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fec440a2e40 0x7fec440a5300 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.605+0000 7fec5e6a5640 1 --2- 172.21.15.107:0/201541633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fec5806d750 0x7fec5806dc00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.605+0000 7fec5e6a5640 1 --2- 172.21.15.107:0/201541633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec581051b0 0x7fec5806d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.605+0000 7fec5e6a5640 1 --2- 172.21.15.107:0/201541633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec581000e0 0x7fec58073150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:17.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.605+0000 7fec5e6a5640 1 -- 172.21.15.107:0/201541633 >> 172.21.15.107:0/201541633 conn(0x7fec58076cd0 msgr2=0x7fec58103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:17.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.606+0000 7fec5e6a5640 1 -- 172.21.15.107:0/201541633 shutdown_connections 2024-05-09T08:43:17.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:17.606+0000 7fec5e6a5640 1 -- 172.21.15.107:0/201541633 wait complete. 2024-05-09T08:43:17.624 INFO:tasks.cephfs.filesystem:scrub status for tag:ca1e0a3f-d43f-4ef6-ac55-56a6bb6eab3b - {'path': '/', 'tag': 'ca1e0a3f-d43f-4ef6-ac55-56a6bb6eab3b', 'options': 'recursive,force'} 2024-05-09T08:43:18.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:18 smithi110 ceph-mon[33622]: pgmap v308: 129 pgs: 5 active+clean+snaptrim, 121 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 20 MiB/s wr, 5.02k op/s 2024-05-09T08:43:18.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:18 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:43:18.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:18 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:18.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:18 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:43:18.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:18 smithi107 ceph-mon[30122]: pgmap v308: 129 pgs: 5 active+clean+snaptrim, 121 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 20 MiB/s wr, 5.02k op/s 2024-05-09T08:43:18.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:18 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:43:18.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:18 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:18.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:18 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:43:18.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:18 smithi157 ceph-mon[33614]: pgmap v308: 129 pgs: 5 active+clean+snaptrim, 121 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 20 MiB/s wr, 5.02k op/s 2024-05-09T08:43:18.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:18 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:43:18.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:18 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:18.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:18 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:43:20.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:20 smithi110 ceph-mon[33622]: pgmap v309: 129 pgs: 4 active+clean+snaptrim, 122 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 16 MiB/s wr, 4.10k op/s 2024-05-09T08:43:20.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:20 smithi107 ceph-mon[30122]: pgmap v309: 129 pgs: 4 active+clean+snaptrim, 122 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 16 MiB/s wr, 4.10k op/s 2024-05-09T08:43:20.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:20 smithi157 ceph-mon[33614]: pgmap v309: 129 pgs: 4 active+clean+snaptrim, 122 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 16 MiB/s wr, 4.10k op/s 2024-05-09T08:43:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:21 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:43:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:21 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:21 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:43:21.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:21 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:21 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:43:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:21 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:21 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:43:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:21 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:21 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:43:21.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:21 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:21.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:21 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:43:21.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:21 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:21 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:43:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:21 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:21 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:43:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:21 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:21 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:43:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:21 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:21 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:43:21.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:21 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:21.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:21 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:43:21.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:21 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:21.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:21 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:43:21.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:21 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:21 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:43:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:21 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:21 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:43:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:21 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:21 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:43:21.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:21 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:22.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:22 smithi107 ceph-mon[30122]: pgmap v310: 129 pgs: 4 active+clean+snaptrim, 122 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 14 MiB/s wr, 3.51k op/s 2024-05-09T08:43:22.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:22 smithi157 ceph-mon[33614]: pgmap v310: 129 pgs: 4 active+clean+snaptrim, 122 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 14 MiB/s wr, 3.51k op/s 2024-05-09T08:43:22.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:22 smithi110 ceph-mon[33622]: pgmap v310: 129 pgs: 4 active+clean+snaptrim, 122 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 14 MiB/s wr, 3.51k op/s 2024-05-09T08:43:23.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:23 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:43:23.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:23 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:23.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:23 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:43:23.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:23 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:23.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:23 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:43:23.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:23 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:24.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:24 smithi107 ceph-mon[30122]: pgmap v311: 129 pgs: 3 active+clean+snaptrim, 123 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 20 MiB/s wr, 5.19k op/s 2024-05-09T08:43:24.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:24 smithi157 ceph-mon[33614]: pgmap v311: 129 pgs: 3 active+clean+snaptrim, 123 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 20 MiB/s wr, 5.19k op/s 2024-05-09T08:43:24.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:24 smithi110 ceph-mon[33622]: pgmap v311: 129 pgs: 3 active+clean+snaptrim, 123 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 20 MiB/s wr, 5.19k op/s 2024-05-09T08:43:26.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:26 smithi107 ceph-mon[30122]: pgmap v312: 129 pgs: 3 active+clean+snaptrim, 123 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 14 MiB/s wr, 3.70k op/s 2024-05-09T08:43:26.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:26 smithi157 ceph-mon[33614]: pgmap v312: 129 pgs: 3 active+clean+snaptrim, 123 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 14 MiB/s wr, 3.70k op/s 2024-05-09T08:43:26.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:26 smithi110 ceph-mon[33622]: pgmap v312: 129 pgs: 3 active+clean+snaptrim, 123 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 14 MiB/s wr, 3.70k op/s 2024-05-09T08:43:28.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:28 smithi107 ceph-mon[30122]: pgmap v313: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 17 MiB/s wr, 4.45k op/s 2024-05-09T08:43:28.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:28 smithi157 ceph-mon[33614]: pgmap v313: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 17 MiB/s wr, 4.45k op/s 2024-05-09T08:43:28.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:28 smithi110 ceph-mon[33622]: pgmap v313: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 17 MiB/s wr, 4.45k op/s 2024-05-09T08:43:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.726+0000 7f9be4ba7640 1 -- 172.21.15.107:0/2004052599 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9be01102d0 msgr2=0x7f9be010dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.726+0000 7f9be4ba7640 1 --2- 172.21.15.107:0/2004052599 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9be01102d0 0x7f9be010dee0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009ca0 tx=0x7f9bd802a980 comp rx=0 tx=0).stop 2024-05-09T08:43:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.730+0000 7f9be4ba7640 1 -- 172.21.15.107:0/2004052599 shutdown_connections 2024-05-09T08:43:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.730+0000 7f9be4ba7640 1 --2- 172.21.15.107:0/2004052599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9be010e500 0x7f9be011d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.730+0000 7f9be4ba7640 1 --2- 172.21.15.107:0/2004052599 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9be01102d0 0x7f9be010dee0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.730+0000 7f9be4ba7640 1 --2- 172.21.15.107:0/2004052599 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9be010fef0 0x7f9be010d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.730+0000 7f9be4ba7640 1 -- 172.21.15.107:0/2004052599 >> 172.21.15.107:0/2004052599 conn(0x7f9be006c770 msgr2=0x7f9be006cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.733+0000 7f9be4ba7640 1 -- 172.21.15.107:0/2004052599 shutdown_connections 2024-05-09T08:43:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.734+0000 7f9be4ba7640 1 -- 172.21.15.107:0/2004052599 wait complete. 2024-05-09T08:43:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.735+0000 7f9be4ba7640 1 Processor -- start 2024-05-09T08:43:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.735+0000 7f9be4ba7640 1 -- start start 2024-05-09T08:43:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.735+0000 7f9be4ba7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9be010e500 0x7f9be01a6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.736+0000 7f9be4ba7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9be010fef0 0x7f9be01a72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.736+0000 7f9be4ba7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9be01102d0 0x7f9be01ab640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.736+0000 7f9be4ba7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9be0120820 con 0x7f9be01102d0 2024-05-09T08:43:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.736+0000 7f9be4ba7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9be01206a0 con 0x7f9be010fef0 2024-05-09T08:43:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.736+0000 7f9be4ba7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9be01209a0 con 0x7f9be010e500 2024-05-09T08:43:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.736+0000 7f9bdeffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9be010fef0 0x7f9be01a72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.736+0000 7f9bdeffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9be010fef0 0x7f9be01a72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:42226/0 (socket says 172.21.15.107:42226) 2024-05-09T08:43:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.736+0000 7f9bdeffd640 1 -- 172.21.15.107:0/3157306465 learned_addr learned my addr 172.21.15.107:0/3157306465 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.736+0000 7f9bdeffd640 1 -- 172.21.15.107:0/3157306465 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9be010e500 msgr2=0x7f9be01a6d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:43:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.736+0000 7f9bdeffd640 1 --2- 172.21.15.107:0/3157306465 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9be010e500 0x7f9be01a6d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.736+0000 7f9bdeffd640 1 -- 172.21.15.107:0/3157306465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9be01102d0 msgr2=0x7f9be01ab640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.736+0000 7f9bdeffd640 1 --2- 172.21.15.107:0/3157306465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9be01102d0 0x7f9be01ab640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.736+0000 7f9bdeffd640 1 -- 172.21.15.107:0/3157306465 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be01abd70 con 0x7f9be010fef0 2024-05-09T08:43:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.737+0000 7f9bdeffd640 1 --2- 172.21.15.107:0/3157306465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9be010fef0 0x7f9be01a72c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9bd802af70 tx=0x7f9bd8030c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:29.737 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.738+0000 7f9bdcff9640 1 -- 172.21.15.107:0/3157306465 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8034050 con 0x7f9be010fef0 2024-05-09T08:43:29.737 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.738+0000 7f9bdcff9640 1 -- 172.21.15.107:0/3157306465 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bd8039ce0 con 0x7f9be010fef0 2024-05-09T08:43:29.737 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.738+0000 7f9bdcff9640 1 -- 172.21.15.107:0/3157306465 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8042d20 con 0x7f9be010fef0 2024-05-09T08:43:29.737 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.738+0000 7f9be4ba7640 1 -- 172.21.15.107:0/3157306465 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9be01ac000 con 0x7f9be010fef0 2024-05-09T08:43:29.737 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.738+0000 7f9be4ba7640 1 -- 172.21.15.107:0/3157306465 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9be01b38f0 con 0x7f9be010fef0 2024-05-09T08:43:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.742+0000 7f9be8bf1740 1 Processor -- start 2024-05-09T08:43:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.742+0000 7f9bdcff9640 1 -- 172.21.15.107:0/3157306465 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f9bd8037020 con 0x7f9be010fef0 2024-05-09T08:43:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.742+0000 7f9be8bf1740 1 -- start start 2024-05-09T08:43:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.743+0000 7f9be8bf1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128600 0x55b0981289e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.743+0000 7f9be8bf1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128f20 0x55b0981332a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.743+0000 7f9be8bf1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 0x55b098135bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.743+0000 7f9be8bf1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b09808d340 con 0x55b098128f20 2024-05-09T08:43:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.743+0000 7f9be8bf1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b098035290 con 0x55b0981337e0 2024-05-09T08:43:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.743+0000 7f9be8bf1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55b098089300 con 0x55b098128600 2024-05-09T08:43:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.744+0000 7f9bdcff9640 1 --2- 172.21.15.107:0/3157306465 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9bc80a2f00 0x7f9bc80a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.744+0000 7f9bdeffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 0x55b098135bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.744+0000 7f9bdffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128600 0x55b0981289e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.744+0000 7f9bdf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128f20 0x55b0981332a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.744+0000 7f9bdffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128600 0x55b0981289e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:56460/0 (socket says 172.21.15.107:56460) 2024-05-09T08:43:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.744+0000 7f9bdffff640 1 -- 172.21.15.107:0/2124819767 learned_addr learned my addr 172.21.15.107:0/2124819767 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.744+0000 7f9bdffff640 1 --2- 172.21.15.107:0/3157306465 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9bc80a2f00 0x7f9bc80a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.745+0000 7f9bdeffd640 1 -- 172.21.15.107:0/2124819767 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128600 msgr2=0x55b0981289e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.745+0000 7f9bdeffd640 1 --2- 172.21.15.107:0/2124819767 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128600 0x55b0981289e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.745+0000 7f9bdeffd640 1 -- 172.21.15.107:0/2124819767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128f20 msgr2=0x55b0981332a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.745+0000 7f9bdeffd640 1 --2- 172.21.15.107:0/2124819767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128f20 0x55b0981332a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.745+0000 7f9bdeffd640 1 -- 172.21.15.107:0/2124819767 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0980598e0 con 0x55b0981337e0 2024-05-09T08:43:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.745+0000 7f9bdf7fe640 1 --2- 172.21.15.107:0/2124819767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128f20 0x55b0981332a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:43:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.746+0000 7f9bdcff9640 1 -- 172.21.15.107:0/3157306465 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f9bd80f0740 con 0x7f9be010fef0 2024-05-09T08:43:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.746+0000 7f9bdffff640 1 --2- 172.21.15.107:0/3157306465 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9bc80a2f00 0x7f9bc80a53c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00058c0 tx=0x7f9bd0005850 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.746+0000 7f9bdeffd640 1 --2- 172.21.15.107:0/2124819767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 0x55b098135bd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9bd80eda20 tx=0x7f9bd8087400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb24167c8ff302a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.746+0000 7f9bc5ffb640 1 -- 172.21.15.107:0/2124819767 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8090040 con 0x55b0981337e0 2024-05-09T08:43:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.746+0000 7f9bc5ffb640 1 -- 172.21.15.107:0/2124819767 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bd8087470 con 0x55b0981337e0 2024-05-09T08:43:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.746+0000 7f9bc5ffb640 1 -- 172.21.15.107:0/2124819767 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd802fe40 con 0x55b0981337e0 2024-05-09T08:43:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.747+0000 7f9be8bf1740 1 -- 172.21.15.107:0/2124819767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 msgr2=0x55b098135bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.747+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/2124819767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 0x55b098135bd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9bd80eda20 tx=0x7f9bd8087400 comp rx=0 tx=0).stop 2024-05-09T08:43:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.747+0000 7f9be8bf1740 1 -- 172.21.15.107:0/2124819767 shutdown_connections 2024-05-09T08:43:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.747+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/2124819767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 0x55b098135bd0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.747+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/2124819767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128f20 0x55b0981332a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.747+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/2124819767 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128600 0x55b0981289e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.748+0000 7f9be8bf1740 1 -- 172.21.15.107:0/2124819767 >> 172.21.15.107:0/2124819767 conn(0x55b098124b10 msgr2=0x55b098124ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.748+0000 7f9be8bf1740 1 -- 172.21.15.107:0/2124819767 shutdown_connections 2024-05-09T08:43:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.748+0000 7f9be8bf1740 1 -- 172.21.15.107:0/2124819767 wait complete. 2024-05-09T08:43:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.748+0000 7f9be8bf1740 1 -- start start 2024-05-09T08:43:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.748+0000 7f9be8bf1740 1 Processor -- start 2024-05-09T08:43:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.749+0000 7f9be8bf1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 0x55b098130720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.749+0000 7f9be8bf1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128f20 0x55b098130c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.749+0000 7f9be8bf1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 0x55b0981311a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.749+0000 7f9be8bf1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b098035290 con 0x55b098128600 2024-05-09T08:43:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.749+0000 7f9be8bf1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b098048790 con 0x55b0981337e0 2024-05-09T08:43:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.749+0000 7f9be8bf1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55b0981361d0 con 0x55b098128f20 2024-05-09T08:43:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.749+0000 7f9be8bf1740 10 client.0 _ll_register_callbacks cb 0x55b098083a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.749+0000 7f9bdf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 0x55b098130720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.749+0000 7f9bdeffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128f20 0x55b098130c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.749+0000 7f9bdf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 0x55b098130720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38054/0 (socket says 172.21.15.107:38054) 2024-05-09T08:43:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.749+0000 7f9bdf7fe640 1 -- 172.21.15.107:0/298929265 learned_addr learned my addr 172.21.15.107:0/298929265 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.749+0000 7f9bdffff640 1 --2- 172.21.15.107:0/298929265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 0x55b0981311a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.749+0000 7f9bdf7fe640 1 -- 172.21.15.107:0/298929265 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128f20 msgr2=0x55b098130c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.750+0000 7f9bdf7fe640 1 --2- 172.21.15.107:0/298929265 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128f20 0x55b098130c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.750+0000 7f9bdf7fe640 1 -- 172.21.15.107:0/298929265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 msgr2=0x55b0981311a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.750+0000 7f9bdf7fe640 1 --2- 172.21.15.107:0/298929265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 0x55b0981311a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.750+0000 7f9bdf7fe640 1 -- 172.21.15.107:0/298929265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b097da1210 con 0x55b098128600 2024-05-09T08:43:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.750+0000 7f9bdffff640 1 --2- 172.21.15.107:0/298929265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 0x55b0981311a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.750+0000 7f9bdeffd640 1 --2- 172.21.15.107:0/298929265 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128f20 0x55b098130c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.750+0000 7f9bdf7fe640 1 --2- 172.21.15.107:0/298929265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 0x55b098130720 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400afb0 tx=0x7f9bd400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af59ea95b30e664 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.750+0000 7f9bbeffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:43:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.750+0000 7f9bbeffd640 1 -- 172.21.15.107:0/298929265 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4015070 con 0x55b098128600 2024-05-09T08:43:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.750+0000 7f9be8bf1740 1 -- 172.21.15.107:0/298929265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b09809aa70 con 0x55b098128600 2024-05-09T08:43:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.750+0000 7f9bbeffd640 1 -- 172.21.15.107:0/298929265 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f9bd40115f0 con 0x55b098128600 2024-05-09T08:43:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.750+0000 7f9bbeffd640 1 -- 172.21.15.107:0/298929265 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9bd4021050 con 0x55b098128600 2024-05-09T08:43:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.750+0000 7f9be8bf1740 10 client.18510 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.750+0000 7f9be8bf1740 1 -- 172.21.15.107:0/298929265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b097e6d620 con 0x55b098128600 2024-05-09T08:43:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.751+0000 7f9bbeffd640 1 -- 172.21.15.107:0/298929265 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b09809aa70 con 0x55b098128600 2024-05-09T08:43:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.751+0000 7f9bbeffd640 1 -- 172.21.15.107:0/298929265 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9bd4019940 con 0x55b098128600 2024-05-09T08:43:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.752+0000 7f9be8bf1740 10 client.18510 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.752+0000 7f9be8bf1740 10 client.18510 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.752+0000 7f9be8bf1740 20 client.18510 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.752+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/298929265 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b098142ef0 0x55b0981452e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.752+0000 7f9be8bf1740 4 client.18510 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.752+0000 7f9be8bf1740 1 -- 172.21.15.107:0/298929265 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b098094dc0 con 0x55b098142ef0 2024-05-09T08:43:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.752+0000 7f9bdeffd640 1 --2- 172.21.15.107:0/298929265 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b098142ef0 0x55b0981452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.753+0000 7f9bbeffd640 10 client.18510 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.753+0000 7f9bdeffd640 1 --2- 172.21.15.107:0/298929265 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b098142ef0 0x55b0981452e0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed57ebd8191d7a07 server_cookie=5779621ae64b8d21 in_seq=0 out_seq=0 2024-05-09T08:43:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.754+0000 7f9bbeffd640 1 -- 172.21.15.107:0/298929265 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b098094dc0 con 0x55b098142ef0 2024-05-09T08:43:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.754+0000 7f9bbeffd640 10 client.18510 handle_command_reply: tid=0 2024-05-09T08:43:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.754+0000 7f9be8bf1740 1 client.18510 shutdown 2024-05-09T08:43:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.754+0000 7f9be8bf1740 1 -- 172.21.15.107:0/298929265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 msgr2=0x55b098130720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.754+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/298929265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 0x55b098130720 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400afb0 tx=0x7f9bd400c520 comp rx=0 tx=0).stop 2024-05-09T08:43:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.754+0000 7f9be8bf1740 1 -- 172.21.15.107:0/298929265 shutdown_connections 2024-05-09T08:43:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.754+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/298929265 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b098142ef0 0x55b0981452e0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.754+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/298929265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 0x55b0981311a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.754+0000 7f9bbeffd640 0 client.18510 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.754+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/298929265 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128f20 0x55b098130c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.754+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/298929265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 0x55b098130720 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.754+0000 7f9be8bf1740 1 -- 172.21.15.107:0/298929265 >> 172.21.15.107:0/298929265 conn(0x55b098124b10 msgr2=0x55b09812a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.755+0000 7f9be8bf1740 1 -- 172.21.15.107:0/298929265 shutdown_connections 2024-05-09T08:43:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.755+0000 7f9be8bf1740 1 -- 172.21.15.107:0/298929265 wait complete. 2024-05-09T08:43:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.755+0000 7f9be8bf1740 20 client.18510 trim_cache size 0 max 16384 2024-05-09T08:43:29.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.793+0000 7f9be8bf1740 1 Processor -- start 2024-05-09T08:43:29.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.793+0000 7f9be8bf1740 1 -- start start 2024-05-09T08:43:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.795+0000 7f9be8bf1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 0x55b09813ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.795+0000 7f9be8bf1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128f20 0x55b09812afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.795+0000 7f9be8bf1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 0x55b09812f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.795+0000 7f9be8bf1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9be01c0260 con 0x55b098128600 2024-05-09T08:43:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.795+0000 7f9be8bf1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9be01bdb30 con 0x55b0981337e0 2024-05-09T08:43:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.795+0000 7f9be8bf1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9be01bb9a0 con 0x55b098128f20 2024-05-09T08:43:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.795+0000 7f9bdf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 0x55b09813ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.795+0000 7f9bdf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 0x55b09813ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38056/0 (socket says 172.21.15.107:38056) 2024-05-09T08:43:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.795+0000 7f9bdf7fe640 1 -- 172.21.15.107:0/349895976 learned_addr learned my addr 172.21.15.107:0/349895976 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.795+0000 7f9bdffff640 1 --2- 172.21.15.107:0/349895976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 0x55b09812f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.795+0000 7f9bdeffd640 1 --2- 172.21.15.107:0/349895976 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128f20 0x55b09812afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:29.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.797+0000 7f9bdffff640 1 -- 172.21.15.107:0/349895976 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128f20 msgr2=0x55b09812afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:29.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.797+0000 7f9bdffff640 1 --2- 172.21.15.107:0/349895976 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128f20 0x55b09812afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.797+0000 7f9bdffff640 1 -- 172.21.15.107:0/349895976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 msgr2=0x55b09813ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:29.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.797+0000 7f9bdffff640 1 --2- 172.21.15.107:0/349895976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 0x55b09813ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.797+0000 7f9bdffff640 1 -- 172.21.15.107:0/349895976 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b098145650 con 0x55b0981337e0 2024-05-09T08:43:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.797+0000 7f9bdffff640 1 --2- 172.21.15.107:0/349895976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 0x55b09812f8a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0005d00 tx=0x7f9bd0014070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdbf3f9ed6da247f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.798+0000 7f9bc57fa640 1 -- 172.21.15.107:0/349895976 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0025070 con 0x55b0981337e0 2024-05-09T08:43:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.798+0000 7f9bc57fa640 1 -- 172.21.15.107:0/349895976 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bd001ed40 con 0x55b0981337e0 2024-05-09T08:43:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.798+0000 7f9bc57fa640 1 -- 172.21.15.107:0/349895976 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd00047a0 con 0x55b0981337e0 2024-05-09T08:43:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.799+0000 7f9be8bf1740 1 -- 172.21.15.107:0/349895976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 msgr2=0x55b09812f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.799+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/349895976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 0x55b09812f8a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0005d00 tx=0x7f9bd0014070 comp rx=0 tx=0).stop 2024-05-09T08:43:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.799+0000 7f9be8bf1740 1 -- 172.21.15.107:0/349895976 shutdown_connections 2024-05-09T08:43:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.799+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/349895976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0981337e0 0x55b09812f8a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.799+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/349895976 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b098128f20 0x55b09812afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.799+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/349895976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 0x55b09813ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.799+0000 7f9be8bf1740 1 -- 172.21.15.107:0/349895976 >> 172.21.15.107:0/349895976 conn(0x55b098124b10 msgr2=0x55b098123320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:29.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.800+0000 7f9be8bf1740 1 -- 172.21.15.107:0/349895976 shutdown_connections 2024-05-09T08:43:29.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.800+0000 7f9be8bf1740 1 -- 172.21.15.107:0/349895976 wait complete. 2024-05-09T08:43:29.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.800+0000 7f9be8bf1740 1 -- start start 2024-05-09T08:43:29.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.801+0000 7f9be8bf1740 1 Processor -- start 2024-05-09T08:43:29.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.801+0000 7f9be8bf1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 0x55b09813b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.804+0000 7f9be8bf1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b098128f20 0x55b09813b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.804+0000 7f9be8bf1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0981337e0 0x55b09812c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.804+0000 7f9be8bf1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9be01bbff0 con 0x55b098128600 2024-05-09T08:43:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.804+0000 7f9be8bf1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9be01bd460 con 0x55b098128f20 2024-05-09T08:43:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.804+0000 7f9be8bf1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9be01bc6c0 con 0x55b0981337e0 2024-05-09T08:43:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.804+0000 7f9be8bf1740 10 client.0 _ll_register_callbacks cb 0x7f9bd4006f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.804+0000 7f9bdf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 0x55b09813b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.805+0000 7f9bdeffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b098128f20 0x55b09813b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.805+0000 7f9bdeffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b098128f20 0x55b09813b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:42266/0 (socket says 172.21.15.107:42266) 2024-05-09T08:43:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.805+0000 7f9bdeffd640 1 -- 172.21.15.107:0/362956897 learned_addr learned my addr 172.21.15.107:0/362956897 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.805+0000 7f9bdffff640 1 --2- 172.21.15.107:0/362956897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0981337e0 0x55b09812c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:29.805 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.805+0000 7f9bdffff640 1 -- 172.21.15.107:0/362956897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b098128f20 msgr2=0x55b09813b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:29.806 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.805+0000 7f9bdffff640 1 --2- 172.21.15.107:0/362956897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b098128f20 0x55b09813b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.806 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.805+0000 7f9bdffff640 1 -- 172.21.15.107:0/362956897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 msgr2=0x55b09813b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:29.806 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.805+0000 7f9bdffff640 1 --2- 172.21.15.107:0/362956897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 0x55b09813b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:29.806 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.805+0000 7f9bdffff640 1 -- 172.21.15.107:0/362956897 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b098135f40 con 0x55b0981337e0 2024-05-09T08:43:29.806 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.806+0000 7f9bdffff640 1 --2- 172.21.15.107:0/362956897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0981337e0 0x55b09812c9b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0005d00 tx=0x7f9bd0012620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4803aa8cb183631 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:29.806 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.806+0000 7f9bbeffd640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:43:29.806 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.806+0000 7f9bbeffd640 1 -- 172.21.15.107:0/362956897 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0038060 con 0x55b0981337e0 2024-05-09T08:43:29.806 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.807+0000 7f9bbeffd640 1 -- 172.21.15.107:0/362956897 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f9bd00047a0 con 0x55b0981337e0 2024-05-09T08:43:29.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.808+0000 7f9be8bf1740 1 -- 172.21.15.107:0/362956897 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bd001ed40 con 0x55b0981337e0 2024-05-09T08:43:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.809+0000 7f9bbeffd640 1 -- 172.21.15.107:0/362956897 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9bd0025050 con 0x55b0981337e0 2024-05-09T08:43:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.809+0000 7f9be8bf1740 10 client.27706 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.809+0000 7f9be8bf1740 1 -- 172.21.15.107:0/362956897 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b0981a8ab0 con 0x55b0981337e0 2024-05-09T08:43:29.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.809+0000 7f9bbeffd640 1 -- 172.21.15.107:0/362956897 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bd000a3d0 con 0x55b0981337e0 2024-05-09T08:43:29.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.810+0000 7f9bbeffd640 1 -- 172.21.15.107:0/362956897 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9bd001f430 con 0x55b0981337e0 2024-05-09T08:43:29.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.810+0000 7f9be8bf1740 10 client.27706 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:29.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.810+0000 7f9be8bf1740 10 client.27706 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:29.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.810+0000 7f9be8bf1740 20 client.27706 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:29.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.810+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/362956897 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b0981a90b0 0x55b0981ab4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:29.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.810+0000 7f9be8bf1740 4 client.27706 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:43:29.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.810+0000 7f9be8bf1740 1 -- 172.21.15.107:0/362956897 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f9bd001ed40 con 0x55b0981a90b0 2024-05-09T08:43:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.810+0000 7f9bdf7fe640 1 --2- 172.21.15.107:0/362956897 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b0981a90b0 0x55b0981ab4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.812+0000 7f9bdf7fe640 1 --2- 172.21.15.107:0/362956897 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b0981a90b0 0x55b0981ab4d0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3186d659d0aaba1e server_cookie=9aee03b494b51bc9 in_seq=0 out_seq=0 2024-05-09T08:43:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:29.812+0000 7f9bbeffd640 10 client.27706 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:30.314 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.315+0000 7f9bbeffd640 1 -- 172.21.15.107:0/362956897 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f9bd001ed40 con 0x55b0981a90b0 2024-05-09T08:43:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.315+0000 7f9bbeffd640 10 client.27706 handle_command_reply: tid=0 2024-05-09T08:43:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.315+0000 7f9be8bf1740 1 client.27706 shutdown 2024-05-09T08:43:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.315+0000 7f9be8bf1740 1 -- 172.21.15.107:0/362956897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0981337e0 msgr2=0x55b09812c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.315+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/362956897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0981337e0 0x55b09812c9b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0005d00 tx=0x7f9bd0012620 comp rx=0 tx=0).stop 2024-05-09T08:43:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.315+0000 7f9be8bf1740 1 -- 172.21.15.107:0/362956897 shutdown_connections 2024-05-09T08:43:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.315+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/362956897 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b0981a90b0 0x55b0981ab4d0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.315+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/362956897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0981337e0 0x55b09812c9b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.315+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/362956897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b098128f20 0x55b09813b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.315+0000 7f9be8bf1740 1 --2- 172.21.15.107:0/362956897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b098128600 0x55b09813b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.315+0000 7f9be8bf1740 1 -- 172.21.15.107:0/362956897 >> 172.21.15.107:0/362956897 conn(0x55b098124b10 msgr2=0x55b098123bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.315+0000 7f9bbeffd640 0 client.27706 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:30.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.315+0000 7f9be8bf1740 1 -- 172.21.15.107:0/362956897 shutdown_connections 2024-05-09T08:43:30.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.315+0000 7f9be8bf1740 1 -- 172.21.15.107:0/362956897 wait complete. 2024-05-09T08:43:30.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.317+0000 7f9be8bf1740 20 client.27706 trim_cache size 0 max 16384 2024-05-09T08:43:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:43:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:43:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:43:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:43:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.318+0000 7f9be4ba7640 1 -- 172.21.15.107:0/3157306465 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9bc80a2f00 msgr2=0x7f9bc80a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.318+0000 7f9be4ba7640 1 --2- 172.21.15.107:0/3157306465 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9bc80a2f00 0x7f9bc80a53c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00058c0 tx=0x7f9bd0005850 comp rx=0 tx=0).stop 2024-05-09T08:43:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.319+0000 7f9be4ba7640 1 -- 172.21.15.107:0/3157306465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9be010fef0 msgr2=0x7f9be01a72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.319+0000 7f9be4ba7640 1 --2- 172.21.15.107:0/3157306465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9be010fef0 0x7f9be01a72c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9bd802af70 tx=0x7f9bd8030c40 comp rx=0 tx=0).stop 2024-05-09T08:43:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.319+0000 7f9be4ba7640 1 -- 172.21.15.107:0/3157306465 shutdown_connections 2024-05-09T08:43:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.319+0000 7f9be4ba7640 1 --2- 172.21.15.107:0/3157306465 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9bc80a2f00 0x7f9bc80a53c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.319+0000 7f9be4ba7640 1 --2- 172.21.15.107:0/3157306465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9be01102d0 0x7f9be01ab640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.319+0000 7f9be4ba7640 1 --2- 172.21.15.107:0/3157306465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9be010fef0 0x7f9be01a72c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.319+0000 7f9be4ba7640 1 --2- 172.21.15.107:0/3157306465 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9be010e500 0x7f9be01a6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.319+0000 7f9be4ba7640 1 -- 172.21.15.107:0/3157306465 >> 172.21.15.107:0/3157306465 conn(0x7f9be006c770 msgr2=0x7f9be0070c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.320+0000 7f9be4ba7640 1 -- 172.21.15.107:0/3157306465 shutdown_connections 2024-05-09T08:43:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:30.320+0000 7f9be4ba7640 1 -- 172.21.15.107:0/3157306465 wait complete. 2024-05-09T08:43:30.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:30 smithi107 ceph-mon[30122]: pgmap v314: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 16 MiB/s wr, 4.13k op/s 2024-05-09T08:43:30.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:30 smithi157 ceph-mon[33614]: pgmap v314: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 16 MiB/s wr, 4.13k op/s 2024-05-09T08:43:30.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:30 smithi110 ceph-mon[33622]: pgmap v314: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 16 MiB/s wr, 4.13k op/s 2024-05-09T08:43:32.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:32 smithi107 ceph-mon[30122]: pgmap v315: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 13 MiB/s wr, 3.41k op/s 2024-05-09T08:43:32.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:32 smithi157 ceph-mon[33614]: pgmap v315: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 13 MiB/s wr, 3.41k op/s 2024-05-09T08:43:32.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:32 smithi110 ceph-mon[33622]: pgmap v315: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 13 MiB/s wr, 3.41k op/s 2024-05-09T08:43:34.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:34 smithi107 ceph-mon[30122]: pgmap v316: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 19 MiB/s wr, 4.86k op/s 2024-05-09T08:43:34.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:34 smithi157 ceph-mon[33614]: pgmap v316: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 19 MiB/s wr, 4.86k op/s 2024-05-09T08:43:34.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:34 smithi110 ceph-mon[33622]: pgmap v316: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 19 MiB/s wr, 4.86k op/s 2024-05-09T08:43:36.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:36 smithi107 ceph-mon[30122]: pgmap v317: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 13 MiB/s wr, 3.25k op/s 2024-05-09T08:43:36.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:36 smithi157 ceph-mon[33614]: pgmap v317: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 13 MiB/s wr, 3.25k op/s 2024-05-09T08:43:36.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:36 smithi110 ceph-mon[33622]: pgmap v317: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 13 MiB/s wr, 3.25k op/s 2024-05-09T08:43:38.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:38 smithi107 ceph-mon[30122]: pgmap v318: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 16 MiB/s wr, 4.05k op/s 2024-05-09T08:43:38.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:38 smithi157 ceph-mon[33614]: pgmap v318: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 16 MiB/s wr, 4.05k op/s 2024-05-09T08:43:38.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:38 smithi110 ceph-mon[33622]: pgmap v318: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 16 MiB/s wr, 4.05k op/s 2024-05-09T08:43:40.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:40 smithi107 ceph-mon[30122]: pgmap v319: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 15 MiB/s wr, 3.71k op/s 2024-05-09T08:43:40.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:40 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:40.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:40 smithi157 ceph-mon[33614]: pgmap v319: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 15 MiB/s wr, 3.71k op/s 2024-05-09T08:43:40.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:40 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:40.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:40 smithi110 ceph-mon[33622]: pgmap v319: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 15 MiB/s wr, 3.71k op/s 2024-05-09T08:43:40.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:40 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:43:41.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:41 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:41.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:41 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:43:41.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:41 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:43:41.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:41 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:41.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:41 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:43:41.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:41 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:43:41.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:41 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:41.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:41 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:43:41.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:41 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:43:42.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:42 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:43:42.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:42 smithi107 ceph-mon[30122]: pgmap v320: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 13 MiB/s wr, 3.18k op/s 2024-05-09T08:43:42.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:42 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:43:42.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:42 smithi157 ceph-mon[33614]: pgmap v320: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 13 MiB/s wr, 3.18k op/s 2024-05-09T08:43:42.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:42 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:43:42.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:42 smithi110 ceph-mon[33622]: pgmap v320: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 13 MiB/s wr, 3.18k op/s 2024-05-09T08:43:43.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.426+0000 7fe27d948640 1 -- 172.21.15.107:0/150724661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe278101120 msgr2=0x7fe27810d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:43.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.426+0000 7fe27d948640 1 --2- 172.21.15.107:0/150724661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe278101120 0x7fe27810d5e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe270009c10 tx=0x7fe27002a6e0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.426+0000 7fe27d948640 1 -- 172.21.15.107:0/150724661 shutdown_connections 2024-05-09T08:43:43.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.426+0000 7fe27d948640 1 --2- 172.21.15.107:0/150724661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe278101120 0x7fe27810d5e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.426+0000 7fe27d948640 1 --2- 172.21.15.107:0/150724661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe278100760 0x7fe278100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.426+0000 7fe27d948640 1 --2- 172.21.15.107:0/150724661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe278106720 0x7fe278106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.426+0000 7fe27d948640 1 -- 172.21.15.107:0/150724661 >> 172.21.15.107:0/150724661 conn(0x7fe2780778b0 msgr2=0x7fe278077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:43.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.426+0000 7fe27d948640 1 -- 172.21.15.107:0/150724661 shutdown_connections 2024-05-09T08:43:43.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.426+0000 7fe27d948640 1 -- 172.21.15.107:0/150724661 wait complete. 2024-05-09T08:43:43.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.426+0000 7fe27d948640 1 Processor -- start 2024-05-09T08:43:43.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.426+0000 7fe27d948640 1 -- start start 2024-05-09T08:43:43.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe27d948640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe278100760 0x7fe278196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe27d948640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe278101120 0x7fe278197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe27d948640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe278106720 0x7fe27819b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe27d948640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe2781109e0 con 0x7fe278101120 2024-05-09T08:43:43.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe27d948640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe278110860 con 0x7fe278106720 2024-05-09T08:43:43.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe27d948640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe278110b60 con 0x7fe278100760 2024-05-09T08:43:43.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe278101120 0x7fe278197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe278101120 0x7fe278197340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57552/0 (socket says 172.21.15.107:57552) 2024-05-09T08:43:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe26ffff640 1 -- 172.21.15.107:0/3181688025 learned_addr learned my addr 172.21.15.107:0/3181688025 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe276ffd640 1 --2- 172.21.15.107:0/3181688025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe278100760 0x7fe278196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe2777fe640 1 --2- 172.21.15.107:0/3181688025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe278106720 0x7fe27819b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe26ffff640 1 -- 172.21.15.107:0/3181688025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe278100760 msgr2=0x7fe278196e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe26ffff640 1 --2- 172.21.15.107:0/3181688025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe278100760 0x7fe278196e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe26ffff640 1 -- 172.21.15.107:0/3181688025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe278106720 msgr2=0x7fe27819b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe26ffff640 1 --2- 172.21.15.107:0/3181688025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe278106720 0x7fe27819b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe26ffff640 1 -- 172.21.15.107:0/3181688025 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe27819bdf0 con 0x7fe278101120 2024-05-09T08:43:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe2777fe640 1 --2- 172.21.15.107:0/3181688025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe278106720 0x7fe27819b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:43:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe26ffff640 1 --2- 172.21.15.107:0/3181688025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe278101120 0x7fe278197340 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fe26000e9f0 tx=0x7fe26000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe27c946640 1 -- 172.21.15.107:0/3181688025 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26000cd80 con 0x7fe278101120 2024-05-09T08:43:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe27d948640 1 -- 172.21.15.107:0/3181688025 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe27819c0e0 con 0x7fe278101120 2024-05-09T08:43:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe27d948640 1 -- 172.21.15.107:0/3181688025 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2781a3970 con 0x7fe278101120 2024-05-09T08:43:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe27c946640 1 -- 172.21.15.107:0/3181688025 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2600054d0 con 0x7fe278101120 2024-05-09T08:43:43.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.427+0000 7fe27c946640 1 -- 172.21.15.107:0/3181688025 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe260010430 con 0x7fe278101120 2024-05-09T08:43:43.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.430+0000 7fe28064b740 1 Processor -- start 2024-05-09T08:43:43.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.430+0000 7fe27c946640 1 -- 172.21.15.107:0/3181688025 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe2600046c0 con 0x7fe278101120 2024-05-09T08:43:43.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.430+0000 7fe28064b740 1 -- start start 2024-05-09T08:43:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.430+0000 7fe28064b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689bd7450 0x564689bd7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.430+0000 7fe28064b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 0x564689be20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.430+0000 7fe28064b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689be2630 0x564689be4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.430+0000 7fe28064b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564689b3c340 con 0x564689bd7d70 2024-05-09T08:43:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.430+0000 7fe28064b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564689ae4290 con 0x564689bd7450 2024-05-09T08:43:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.430+0000 7fe28064b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564689b38300 con 0x564689be2630 2024-05-09T08:43:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.430+0000 7fe276ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 0x564689be20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.430+0000 7fe2777fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689bd7450 0x564689bd7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.430+0000 7fe276ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 0x564689be20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57568/0 (socket says 172.21.15.107:57568) 2024-05-09T08:43:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.430+0000 7fe276ffd640 1 -- 172.21.15.107:0/2249774795 learned_addr learned my addr 172.21.15.107:0/2249774795 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.430+0000 7fe26ffff640 1 --2- 172.21.15.107:0/2249774795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689be2630 0x564689be4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.431+0000 7fe2777fe640 1 -- 172.21.15.107:0/2249774795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689be2630 msgr2=0x564689be4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.431+0000 7fe2777fe640 1 --2- 172.21.15.107:0/2249774795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689be2630 0x564689be4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.431+0000 7fe2777fe640 1 -- 172.21.15.107:0/2249774795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 msgr2=0x564689be20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.431+0000 7fe2777fe640 1 --2- 172.21.15.107:0/2249774795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 0x564689be20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.431+0000 7fe2777fe640 1 -- 172.21.15.107:0/2249774795 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564689b088e0 con 0x564689bd7450 2024-05-09T08:43:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.431+0000 7fe2777fe640 1 --2- 172.21.15.107:0/2249774795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689bd7450 0x564689bd7830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe270006010 tx=0x7fe2700304b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a237964b5612941 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.431+0000 7fe26dffb640 1 -- 172.21.15.107:0/2249774795 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe270041020 con 0x564689bd7450 2024-05-09T08:43:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.431+0000 7fe27c946640 1 --2- 172.21.15.107:0/3181688025 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2640a2da0 0x7fe2640a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.431+0000 7fe26dffb640 1 -- 172.21.15.107:0/2249774795 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe270037040 con 0x564689bd7450 2024-05-09T08:43:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.431+0000 7fe26dffb640 1 -- 172.21.15.107:0/2249774795 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe270036070 con 0x564689bd7450 2024-05-09T08:43:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.431+0000 7fe276ffd640 1 --2- 172.21.15.107:0/3181688025 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2640a2da0 0x7fe2640a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.431+0000 7fe27c946640 1 -- 172.21.15.107:0/3181688025 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fe2600cfd00 con 0x7fe278101120 2024-05-09T08:43:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.432+0000 7fe28064b740 1 -- 172.21.15.107:0/2249774795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689bd7450 msgr2=0x564689bd7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.432+0000 7fe276ffd640 1 --2- 172.21.15.107:0/3181688025 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2640a2da0 0x7fe2640a5260 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x564689bd89f0 tx=0x7fe268009260 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.432+0000 7fe28064b740 1 --2- 172.21.15.107:0/2249774795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689bd7450 0x564689bd7830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe270006010 tx=0x7fe2700304b0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.432+0000 7fe28064b740 1 -- 172.21.15.107:0/2249774795 shutdown_connections 2024-05-09T08:43:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.432+0000 7fe28064b740 1 --2- 172.21.15.107:0/2249774795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689be2630 0x564689be4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.432+0000 7fe28064b740 1 --2- 172.21.15.107:0/2249774795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 0x564689be20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.432+0000 7fe28064b740 1 --2- 172.21.15.107:0/2249774795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689bd7450 0x564689bd7830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.432+0000 7fe28064b740 1 -- 172.21.15.107:0/2249774795 >> 172.21.15.107:0/2249774795 conn(0x564689bd3960 msgr2=0x564689bd3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.432+0000 7fe28064b740 1 -- 172.21.15.107:0/2249774795 shutdown_connections 2024-05-09T08:43:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.432+0000 7fe28064b740 1 -- 172.21.15.107:0/2249774795 wait complete. 2024-05-09T08:43:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.432+0000 7fe28064b740 1 -- start start 2024-05-09T08:43:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe28064b740 1 Processor -- start 2024-05-09T08:43:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe28064b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689bd7450 0x564689bdf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe28064b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689bd7d70 0x564689bdfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe28064b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689be2630 0x564689be0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe28064b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564689be5200 con 0x564689be2630 2024-05-09T08:43:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe28064b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564689af7790 con 0x564689bd7d70 2024-05-09T08:43:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe28064b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564689be5380 con 0x564689bd7450 2024-05-09T08:43:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe28064b740 10 client.0 _ll_register_callbacks cb 0x564689b32a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe2777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689be2630 0x564689be0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe2777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689be2630 0x564689be0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57570/0 (socket says 172.21.15.107:57570) 2024-05-09T08:43:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe276ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689bd7450 0x564689bdf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689bd7d70 0x564689bdfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe2777fe640 1 -- 172.21.15.107:0/2719025393 learned_addr learned my addr 172.21.15.107:0/2719025393 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe2777fe640 1 -- 172.21.15.107:0/2719025393 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689bd7450 msgr2=0x564689bdf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe2777fe640 1 --2- 172.21.15.107:0/2719025393 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689bd7450 0x564689bdf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe2777fe640 1 -- 172.21.15.107:0/2719025393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689bd7d70 msgr2=0x564689bdfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe2777fe640 1 --2- 172.21.15.107:0/2719025393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689bd7d70 0x564689bdfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe2777fe640 1 -- 172.21.15.107:0/2719025393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564689850210 con 0x564689be2630 2024-05-09T08:43:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe276ffd640 1 --2- 172.21.15.107:0/2719025393 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689bd7450 0x564689bdf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe2777fe640 1 --2- 172.21.15.107:0/2719025393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689be2630 0x564689be0010 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fe270004af0 tx=0x7fe2700309a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d13f24bacc0fb62e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe256ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:43:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.433+0000 7fe256ffd640 1 -- 172.21.15.107:0/2719025393 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27004a070 con 0x564689be2630 2024-05-09T08:43:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.434+0000 7fe256ffd640 1 -- 172.21.15.107:0/2719025393 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fe270003b70 con 0x564689be2630 2024-05-09T08:43:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.434+0000 7fe28064b740 1 -- 172.21.15.107:0/2719025393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe270037040 con 0x564689be2630 2024-05-09T08:43:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.434+0000 7fe256ffd640 1 -- 172.21.15.107:0/2719025393 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe270056050 con 0x564689be2630 2024-05-09T08:43:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.434+0000 7fe28064b740 10 client.18528 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.434+0000 7fe28064b740 1 -- 172.21.15.107:0/2719025393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56468991c620 con 0x564689be2630 2024-05-09T08:43:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.434+0000 7fe256ffd640 1 -- 172.21.15.107:0/2719025393 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe270037040 con 0x564689be2630 2024-05-09T08:43:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.434+0000 7fe256ffd640 1 -- 172.21.15.107:0/2719025393 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe27003d510 con 0x564689be2630 2024-05-09T08:43:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.434+0000 7fe28064b740 10 client.18528 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.434+0000 7fe28064b740 10 client.18528 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.434+0000 7fe28064b740 20 client.18528 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.434+0000 7fe28064b740 1 --2- 172.21.15.107:0/2719025393 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564689bf6590 0x564689bf8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.434+0000 7fe28064b740 4 client.18528 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.434+0000 7fe28064b740 1 -- 172.21.15.107:0/2719025393 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564689b49a70 con 0x564689bf6590 2024-05-09T08:43:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.434+0000 7fe276ffd640 1 --2- 172.21.15.107:0/2719025393 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564689bf6590 0x564689bf8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.435+0000 7fe256ffd640 10 client.18528 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.435+0000 7fe276ffd640 1 --2- 172.21.15.107:0/2719025393 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564689bf6590 0x564689bf8980 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dabdb2ad4d47df84 server_cookie=1c61e665cd332a40 in_seq=0 out_seq=0 2024-05-09T08:43:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.435+0000 7fe256ffd640 1 -- 172.21.15.107:0/2719025393 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564689b49a70 con 0x564689bf6590 2024-05-09T08:43:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.435+0000 7fe256ffd640 10 client.18528 handle_command_reply: tid=0 2024-05-09T08:43:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.435+0000 7fe28064b740 1 client.18528 shutdown 2024-05-09T08:43:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.435+0000 7fe28064b740 1 -- 172.21.15.107:0/2719025393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689be2630 msgr2=0x564689be0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.435+0000 7fe28064b740 1 --2- 172.21.15.107:0/2719025393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689be2630 0x564689be0010 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fe270004af0 tx=0x7fe2700309a0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.436+0000 7fe28064b740 1 -- 172.21.15.107:0/2719025393 shutdown_connections 2024-05-09T08:43:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.436+0000 7fe28064b740 1 --2- 172.21.15.107:0/2719025393 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564689bf6590 0x564689bf8980 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.436+0000 7fe28064b740 1 --2- 172.21.15.107:0/2719025393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689be2630 0x564689be0010 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.436+0000 7fe28064b740 1 --2- 172.21.15.107:0/2719025393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689bd7d70 0x564689bdfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.436+0000 7fe28064b740 1 --2- 172.21.15.107:0/2719025393 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689bd7450 0x564689bdf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.436+0000 7fe256ffd640 0 client.18528 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.436+0000 7fe28064b740 1 -- 172.21.15.107:0/2719025393 >> 172.21.15.107:0/2719025393 conn(0x564689bd3960 msgr2=0x564689bd96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.436+0000 7fe28064b740 1 -- 172.21.15.107:0/2719025393 shutdown_connections 2024-05-09T08:43:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.436+0000 7fe28064b740 1 -- 172.21.15.107:0/2719025393 wait complete. 2024-05-09T08:43:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.436+0000 7fe28064b740 20 client.18528 trim_cache size 0 max 16384 2024-05-09T08:43:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.459+0000 7fe28064b740 1 Processor -- start 2024-05-09T08:43:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.459+0000 7fe28064b740 1 -- start start 2024-05-09T08:43:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.459+0000 7fe28064b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689bd7450 0x564689bd3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.459+0000 7fe28064b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 0x564689bfdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.459+0000 7fe28064b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689be2630 0x564689bfe4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.459+0000 7fe28064b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe2781af190 con 0x564689bd7d70 2024-05-09T08:43:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.459+0000 7fe28064b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe2781aca60 con 0x564689bd7450 2024-05-09T08:43:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.459+0000 7fe28064b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe2781aa8d0 con 0x564689be2630 2024-05-09T08:43:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.459+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 0x564689bfdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.459+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 0x564689bfdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57578/0 (socket says 172.21.15.107:57578) 2024-05-09T08:43:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.459+0000 7fe26ffff640 1 -- 172.21.15.107:0/1862782887 learned_addr learned my addr 172.21.15.107:0/1862782887 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.459+0000 7fe2777fe640 1 --2- 172.21.15.107:0/1862782887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689be2630 0x564689bfe4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.459+0000 7fe276ffd640 1 --2- 172.21.15.107:0/1862782887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689bd7450 0x564689bd3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.460+0000 7fe26ffff640 1 -- 172.21.15.107:0/1862782887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689be2630 msgr2=0x564689bfe4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.460+0000 7fe26ffff640 1 --2- 172.21.15.107:0/1862782887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689be2630 0x564689bfe4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.460+0000 7fe26ffff640 1 -- 172.21.15.107:0/1862782887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689bd7450 msgr2=0x564689bd3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.460+0000 7fe26ffff640 1 --2- 172.21.15.107:0/1862782887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689bd7450 0x564689bd3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.460+0000 7fe26ffff640 1 -- 172.21.15.107:0/1862782887 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564689bf2f40 con 0x564689bd7d70 2024-05-09T08:43:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.460+0000 7fe26ffff640 1 --2- 172.21.15.107:0/1862782887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 0x564689bfdfb0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fe2600987e0 tx=0x7fe2600057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c505b736606af231 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.460+0000 7fe26d7fa640 1 -- 172.21.15.107:0/1862782887 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26000b2c0 con 0x564689bd7d70 2024-05-09T08:43:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.460+0000 7fe26d7fa640 1 -- 172.21.15.107:0/1862782887 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe26009c510 con 0x564689bd7d70 2024-05-09T08:43:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.460+0000 7fe26d7fa640 1 -- 172.21.15.107:0/1862782887 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2600cd450 con 0x564689bd7d70 2024-05-09T08:43:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 -- 172.21.15.107:0/1862782887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 msgr2=0x564689bfdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 --2- 172.21.15.107:0/1862782887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 0x564689bfdfb0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fe2600987e0 tx=0x7fe2600057a0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 -- 172.21.15.107:0/1862782887 shutdown_connections 2024-05-09T08:43:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 --2- 172.21.15.107:0/1862782887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689be2630 0x564689bfe4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 --2- 172.21.15.107:0/1862782887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 0x564689bfdfb0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 --2- 172.21.15.107:0/1862782887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689bd7450 0x564689bd3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 -- 172.21.15.107:0/1862782887 >> 172.21.15.107:0/1862782887 conn(0x564689bd3960 msgr2=0x564689bd2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 -- 172.21.15.107:0/1862782887 shutdown_connections 2024-05-09T08:43:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 -- 172.21.15.107:0/1862782887 wait complete. 2024-05-09T08:43:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 -- start start 2024-05-09T08:43:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 Processor -- start 2024-05-09T08:43:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689bd7450 0x564689beef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 0x564689bdc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689be2630 0x564689bdc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe2781aaf20 con 0x564689bd7d70 2024-05-09T08:43:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe2781ac390 con 0x564689be2630 2024-05-09T08:43:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.461+0000 7fe28064b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe2781ab5f0 con 0x564689bd7450 2024-05-09T08:43:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe28064b740 10 client.0 _ll_register_callbacks cb 0x7fe2680095a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe276ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689bd7450 0x564689beef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe26ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 0x564689bdc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe2777fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689be2630 0x564689bdc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe276ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689bd7450 0x564689beef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47674/0 (socket says 172.21.15.107:47674) 2024-05-09T08:43:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe276ffd640 1 -- 172.21.15.107:0/1159899982 learned_addr learned my addr 172.21.15.107:0/1159899982 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe26ffff640 1 -- 172.21.15.107:0/1159899982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689bd7450 msgr2=0x564689beef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe26ffff640 1 --2- 172.21.15.107:0/1159899982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689bd7450 0x564689beef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe26ffff640 1 -- 172.21.15.107:0/1159899982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689be2630 msgr2=0x564689bdc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe26ffff640 1 --2- 172.21.15.107:0/1159899982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689be2630 0x564689bdc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe26ffff640 1 -- 172.21.15.107:0/1159899982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564689bd9d80 con 0x564689bd7d70 2024-05-09T08:43:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe2777fe640 1 --2- 172.21.15.107:0/1159899982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689be2630 0x564689bdc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe276ffd640 1 --2- 172.21.15.107:0/1159899982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689bd7450 0x564689beef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe26ffff640 1 --2- 172.21.15.107:0/1159899982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 0x564689bdc080 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fe26009bac0 tx=0x7fe2600d00b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea4423a8962db55 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe256ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:43:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe256ffd640 1 -- 172.21.15.107:0/1159899982 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26000b2c0 con 0x564689bd7d70 2024-05-09T08:43:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe256ffd640 1 -- 172.21.15.107:0/1159899982 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fe2600a45f0 con 0x564689bd7d70 2024-05-09T08:43:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe28064b740 1 -- 172.21.15.107:0/1159899982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe26009c510 con 0x564689bd7d70 2024-05-09T08:43:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe256ffd640 1 -- 172.21.15.107:0/1159899982 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe26009be10 con 0x564689bd7d70 2024-05-09T08:43:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe28064b740 10 client.18534 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.462+0000 7fe28064b740 1 -- 172.21.15.107:0/1159899982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564689c5c780 con 0x564689bd7d70 2024-05-09T08:43:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.463+0000 7fe256ffd640 1 -- 172.21.15.107:0/1159899982 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe26009c510 con 0x564689bd7d70 2024-05-09T08:43:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.463+0000 7fe256ffd640 1 -- 172.21.15.107:0/1159899982 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe260097b60 con 0x564689bd7d70 2024-05-09T08:43:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.463+0000 7fe28064b740 10 client.18534 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.463+0000 7fe28064b740 10 client.18534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.463+0000 7fe28064b740 20 client.18534 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.463+0000 7fe28064b740 1 --2- 172.21.15.107:0/1159899982 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564689c5ce70 0x564689c5f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.463+0000 7fe28064b740 4 client.18534 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:43:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.463+0000 7fe28064b740 1 -- 172.21.15.107:0/1159899982 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564689b43dc0 con 0x564689c5ce70 2024-05-09T08:43:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.463+0000 7fe276ffd640 1 --2- 172.21.15.107:0/1159899982 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564689c5ce70 0x564689c5f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.463+0000 7fe256ffd640 10 client.18534 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:43.463+0000 7fe276ffd640 1 --2- 172.21.15.107:0/1159899982 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564689c5ce70 0x564689c5f290 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e7b132a0a6413c5 server_cookie=ce80d1f41caff0d7 in_seq=0 out_seq=0 2024-05-09T08:43:44.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:44 smithi107 ceph-mon[30122]: pgmap v321: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 17 MiB/s wr, 4.35k op/s 2024-05-09T08:43:44.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:44 smithi157 ceph-mon[33614]: pgmap v321: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 17 MiB/s wr, 4.35k op/s 2024-05-09T08:43:44.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:44 smithi110 ceph-mon[33622]: pgmap v321: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 17 MiB/s wr, 4.35k op/s 2024-05-09T08:43:46.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:46 smithi107 ceph-mon[30122]: pgmap v322: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 11 MiB/s wr, 2.90k op/s 2024-05-09T08:43:46.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:46 smithi157 ceph-mon[33614]: pgmap v322: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 11 MiB/s wr, 2.90k op/s 2024-05-09T08:43:46.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:46 smithi110 ceph-mon[33622]: pgmap v322: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 11 MiB/s wr, 2.90k op/s 2024-05-09T08:43:47.402 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.402+0000 7fe256ffd640 1 -- 172.21.15.107:0/1159899982 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564689b43dc0 con 0x564689c5ce70 2024-05-09T08:43:47.402 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.402+0000 7fe256ffd640 10 client.18534 handle_command_reply: tid=0 2024-05-09T08:43:47.402 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.402+0000 7fe28064b740 1 client.18534 shutdown 2024-05-09T08:43:47.402 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.402+0000 7fe28064b740 1 -- 172.21.15.107:0/1159899982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 msgr2=0x564689bdc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.402+0000 7fe28064b740 1 --2- 172.21.15.107:0/1159899982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 0x564689bdc080 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fe26009bac0 tx=0x7fe2600d00b0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.402+0000 7fe28064b740 1 -- 172.21.15.107:0/1159899982 shutdown_connections 2024-05-09T08:43:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.402+0000 7fe28064b740 1 --2- 172.21.15.107:0/1159899982 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564689c5ce70 0x564689c5f290 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.402+0000 7fe28064b740 1 --2- 172.21.15.107:0/1159899982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564689be2630 0x564689bdc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.402+0000 7fe28064b740 1 --2- 172.21.15.107:0/1159899982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564689bd7d70 0x564689bdc080 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.402+0000 7fe256ffd640 0 client.18534 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.402+0000 7fe28064b740 1 --2- 172.21.15.107:0/1159899982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564689bd7450 0x564689beef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.402+0000 7fe28064b740 1 -- 172.21.15.107:0/1159899982 >> 172.21.15.107:0/1159899982 conn(0x564689bd3960 msgr2=0x564689bd2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.403+0000 7fe28064b740 1 -- 172.21.15.107:0/1159899982 shutdown_connections 2024-05-09T08:43:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.403+0000 7fe28064b740 1 -- 172.21.15.107:0/1159899982 wait complete. 2024-05-09T08:43:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.403+0000 7fe28064b740 20 client.18534 trim_cache size 0 max 16384 2024-05-09T08:43:47.404 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:43:47.404 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:43:47.404 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:43:47.404 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:43:47.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.403+0000 7fe27d948640 1 -- 172.21.15.107:0/3181688025 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2640a2da0 msgr2=0x7fe2640a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:47.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.403+0000 7fe27d948640 1 --2- 172.21.15.107:0/3181688025 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2640a2da0 0x7fe2640a5260 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x564689bd89f0 tx=0x7fe268009260 comp rx=0 tx=0).stop 2024-05-09T08:43:47.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.403+0000 7fe27d948640 1 -- 172.21.15.107:0/3181688025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe278101120 msgr2=0x7fe278197340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:47.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.403+0000 7fe27d948640 1 --2- 172.21.15.107:0/3181688025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe278101120 0x7fe278197340 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fe26000e9f0 tx=0x7fe26000edf0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.403+0000 7fe27d948640 1 -- 172.21.15.107:0/3181688025 shutdown_connections 2024-05-09T08:43:47.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.403+0000 7fe27d948640 1 --2- 172.21.15.107:0/3181688025 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2640a2da0 0x7fe2640a5260 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.403+0000 7fe27d948640 1 --2- 172.21.15.107:0/3181688025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe278106720 0x7fe27819b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.403+0000 7fe27d948640 1 --2- 172.21.15.107:0/3181688025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe278101120 0x7fe278197340 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.403+0000 7fe27d948640 1 --2- 172.21.15.107:0/3181688025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe278100760 0x7fe278196e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.403+0000 7fe27d948640 1 -- 172.21.15.107:0/3181688025 >> 172.21.15.107:0/3181688025 conn(0x7fe2780778b0 msgr2=0x7fe278103240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.404+0000 7fe27d948640 1 -- 172.21.15.107:0/3181688025 shutdown_connections 2024-05-09T08:43:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:43:47.404+0000 7fe27d948640 1 -- 172.21.15.107:0/3181688025 wait complete. 2024-05-09T08:43:47.625 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:43:47.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:43:47.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.776+0000 7fe21a851640 1 -- 172.21.15.107:0/3278780918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe20c09acf0 msgr2=0x7fe20c004740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:47.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.776+0000 7fe21a851640 1 --2- 172.21.15.107:0/3278780918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe20c09acf0 0x7fe20c004740 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fe210009900 tx=0x7fe21002a410 comp rx=0 tx=0).stop 2024-05-09T08:43:47.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.776+0000 7fe21a851640 1 -- 172.21.15.107:0/3278780918 shutdown_connections 2024-05-09T08:43:47.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.776+0000 7fe21a851640 1 --2- 172.21.15.107:0/3278780918 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe20c0986e0 0x7fe20c098b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.776+0000 7fe21a851640 1 --2- 172.21.15.107:0/3278780918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe20c004c80 0x7fe20c005100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.776+0000 7fe21a851640 1 --2- 172.21.15.107:0/3278780918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe20c09acf0 0x7fe20c004740 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.776+0000 7fe21a851640 1 -- 172.21.15.107:0/3278780918 >> 172.21.15.107:0/3278780918 conn(0x7fe20c00a740 msgr2=0x7fe20c00ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:47.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.776+0000 7fe21a851640 1 -- 172.21.15.107:0/3278780918 shutdown_connections 2024-05-09T08:43:47.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.776+0000 7fe21a851640 1 -- 172.21.15.107:0/3278780918 wait complete. 2024-05-09T08:43:47.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.777+0000 7fe21a851640 1 Processor -- start 2024-05-09T08:43:47.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.777+0000 7fe21a851640 1 -- start start 2024-05-09T08:43:47.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.777+0000 7fe21a851640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe20c004c80 0x7fe20c138690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.777+0000 7fe21a851640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe20c0986e0 0x7fe20c138bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.777+0000 7fe21a851640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe20c09acf0 0x7fe20c132750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.777+0000 7fe21a851640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c0ac4d0 con 0x7fe20c0986e0 2024-05-09T08:43:47.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.777+0000 7fe21a851640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c0ac350 con 0x7fe20c004c80 2024-05-09T08:43:47.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.777+0000 7fe21a851640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c0ac650 con 0x7fe20c09acf0 2024-05-09T08:43:47.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.777+0000 7fe21904e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe20c0986e0 0x7fe20c138bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.777+0000 7fe21984f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe20c004c80 0x7fe20c138690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.777+0000 7fe21984f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe20c004c80 0x7fe20c138690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34388/0 (socket says 172.21.15.107:34388) 2024-05-09T08:43:47.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.777+0000 7fe21904e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe20c0986e0 0x7fe20c138bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57618/0 (socket says 172.21.15.107:57618) 2024-05-09T08:43:47.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.777+0000 7fe21984f640 1 -- 172.21.15.107:0/3619585527 learned_addr learned my addr 172.21.15.107:0/3619585527 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:47.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.778+0000 7fe21a050640 1 --2- 172.21.15.107:0/3619585527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe20c09acf0 0x7fe20c132750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.778+0000 7fe21984f640 1 -- 172.21.15.107:0/3619585527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe20c09acf0 msgr2=0x7fe20c132750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:47.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.778+0000 7fe21984f640 1 --2- 172.21.15.107:0/3619585527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe20c09acf0 0x7fe20c132750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.778+0000 7fe21984f640 1 -- 172.21.15.107:0/3619585527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe20c0986e0 msgr2=0x7fe20c138bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:47.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.778+0000 7fe21984f640 1 --2- 172.21.15.107:0/3619585527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe20c0986e0 0x7fe20c138bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.778+0000 7fe21984f640 1 -- 172.21.15.107:0/3619585527 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20c133010 con 0x7fe20c004c80 2024-05-09T08:43:47.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.778+0000 7fe21984f640 1 --2- 172.21.15.107:0/3619585527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe20c004c80 0x7fe20c138690 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe210031440 tx=0x7fe2100045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:47.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.778+0000 7fe20affd640 1 -- 172.21.15.107:0/3619585527 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe210035030 con 0x7fe20c004c80 2024-05-09T08:43:47.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.778+0000 7fe21a851640 1 -- 172.21.15.107:0/3619585527 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe20c133300 con 0x7fe20c004c80 2024-05-09T08:43:47.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.778+0000 7fe20affd640 1 -- 172.21.15.107:0/3619585527 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe210003e30 con 0x7fe20c004c80 2024-05-09T08:43:47.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.778+0000 7fe20affd640 1 -- 172.21.15.107:0/3619585527 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe210031880 con 0x7fe20c004c80 2024-05-09T08:43:47.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.778+0000 7fe21a851640 1 -- 172.21.15.107:0/3619585527 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe20c13f610 con 0x7fe20c004c80 2024-05-09T08:43:47.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.780+0000 7fe20affd640 1 -- 172.21.15.107:0/3619585527 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe21003b740 con 0x7fe20c004c80 2024-05-09T08:43:47.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.781+0000 7fe20affd640 1 --2- 172.21.15.107:0/3619585527 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2000a2d00 0x7fe2000a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.781+0000 7fe21e7bf740 1 Processor -- start 2024-05-09T08:43:47.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.781+0000 7fe21904e640 1 --2- 172.21.15.107:0/3619585527 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2000a2d00 0x7fe2000a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.781+0000 7fe20affd640 1 -- 172.21.15.107:0/3619585527 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fe2100f0120 con 0x7fe20c004c80 2024-05-09T08:43:47.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.781+0000 7fe21e7bf740 1 -- start start 2024-05-09T08:43:47.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21e7bf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623af430 0x55b5623af810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21e7bf740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623afd50 0x55b5623b9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21e7bf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623ba420 0x55b5623bc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21e7bf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b562314d00 con 0x55b5623ba420 2024-05-09T08:43:47.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21e7bf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b5622bc4c0 con 0x55b5623af430 2024-05-09T08:43:47.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21e7bf740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55b562310cc0 con 0x55b5623afd50 2024-05-09T08:43:47.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21904e640 1 --2- 172.21.15.107:0/3619585527 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2000a2d00 0x7fe2000a51c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe20c133fa0 tx=0x7fe20400b410 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:47.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21a050640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623ba420 0x55b5623bc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21984f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623af430 0x55b5623af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21a050640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623ba420 0x55b5623bc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57622/0 (socket says 172.21.15.107:57622) 2024-05-09T08:43:47.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21984f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623af430 0x55b5623af810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34392/0 (socket says 172.21.15.107:34392) 2024-05-09T08:43:47.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21a050640 1 -- 172.21.15.107:0/2996066837 learned_addr learned my addr 172.21.15.107:0/2996066837 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:47.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21984f640 1 -- 172.21.15.107:0/2996066837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623afd50 msgr2=0x55b5623b9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:47.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21984f640 1 --2- 172.21.15.107:0/2996066837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623afd50 0x55b5623b9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21984f640 1 -- 172.21.15.107:0/2996066837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623ba420 msgr2=0x55b5623bc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:47.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21984f640 1 --2- 172.21.15.107:0/2996066837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623ba420 0x55b5623bc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21984f640 1 -- 172.21.15.107:0/2996066837 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5620f27d0 con 0x55b5623af430 2024-05-09T08:43:47.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe21984f640 1 --2- 172.21.15.107:0/2996066837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623af430 0x55b5623af810 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe2100ed400 tx=0x7fe2100b8650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a452c80bdc6a5f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:47.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe1fffff640 1 -- 172.21.15.107:0/2996066837 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21004a040 con 0x55b5623af430 2024-05-09T08:43:47.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe1fffff640 1 -- 172.21.15.107:0/2996066837 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2100c4e40 con 0x55b5623af430 2024-05-09T08:43:47.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.782+0000 7fe1fffff640 1 -- 172.21.15.107:0/2996066837 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2100dc0f0 con 0x55b5623af430 2024-05-09T08:43:47.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.783+0000 7fe21e7bf740 1 -- 172.21.15.107:0/2996066837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623af430 msgr2=0x55b5623af810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:47.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.783+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/2996066837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623af430 0x55b5623af810 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe2100ed400 tx=0x7fe2100b8650 comp rx=0 tx=0).stop 2024-05-09T08:43:47.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.783+0000 7fe21e7bf740 1 -- 172.21.15.107:0/2996066837 shutdown_connections 2024-05-09T08:43:47.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.783+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/2996066837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623ba420 0x55b5623bc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.783+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/2996066837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623afd50 0x55b5623b9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.783+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/2996066837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623af430 0x55b5623af810 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.783+0000 7fe21e7bf740 1 -- 172.21.15.107:0/2996066837 >> 172.21.15.107:0/2996066837 conn(0x55b5623ab750 msgr2=0x55b5623abb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:47.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.783+0000 7fe21e7bf740 1 -- 172.21.15.107:0/2996066837 shutdown_connections 2024-05-09T08:43:47.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.783+0000 7fe21e7bf740 1 -- 172.21.15.107:0/2996066837 wait complete. 2024-05-09T08:43:47.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.784+0000 7fe21e7bf740 1 -- start start 2024-05-09T08:43:47.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.784+0000 7fe21e7bf740 1 Processor -- start 2024-05-09T08:43:47.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21e7bf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623b7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21e7bf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623afd50 0x55b5623b7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21984f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623b7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21e7bf740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623ba420 0x55b5623b7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21984f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623b7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57628/0 (socket says 172.21.15.107:57628) 2024-05-09T08:43:47.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21e7bf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b5623bcfd0 con 0x55b5623af430 2024-05-09T08:43:47.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21984f640 1 -- 172.21.15.107:0/1404674381 learned_addr learned my addr 172.21.15.107:0/1404674381 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:47.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21e7bf740 1 -- 172.21.15.107:0/1404674381 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b5622cfb90 con 0x55b5623afd50 2024-05-09T08:43:47.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21e7bf740 1 -- 172.21.15.107:0/1404674381 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55b5623bd150 con 0x55b5623ba420 2024-05-09T08:43:47.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21984f640 1 -- 172.21.15.107:0/1404674381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623ba420 msgr2=0x55b5623b7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:47.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21a050640 1 --2- 172.21.15.107:0/1404674381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623ba420 0x55b5623b7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21984f640 1 --2- 172.21.15.107:0/1404674381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623ba420 0x55b5623b7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21984f640 1 -- 172.21.15.107:0/1404674381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623afd50 msgr2=0x55b5623b7940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:43:47.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21984f640 1 --2- 172.21.15.107:0/1404674381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623afd50 0x55b5623b7940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21984f640 1 -- 172.21.15.107:0/1404674381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5620283f0 con 0x55b5623af430 2024-05-09T08:43:47.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21a050640 1 --2- 172.21.15.107:0/1404674381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623ba420 0x55b5623b7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:43:47.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.785+0000 7fe21e7bf740 10 client.0 _ll_register_callbacks cb 0x55b56230bda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:47.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.786+0000 7fe21e7bf740 1 -- 172.21.15.107:0/1404674381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2100c4e40 con 0x55b5623af430 2024-05-09T08:43:47.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.786+0000 7fe21984f640 1 --2- 172.21.15.107:0/1404674381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623b7400 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fe210005940 tx=0x7fe2100ba600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d58d2ea4f9eddc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:47.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.786+0000 7fe1fd7fa640 10 client.18546 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:43:47.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.786+0000 7fe1fd7fa640 1 -- 172.21.15.107:0/1404674381 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21004a070 con 0x55b5623af430 2024-05-09T08:43:47.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.786+0000 7fe1fd7fa640 1 -- 172.21.15.107:0/1404674381 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fe2100b88f0 con 0x55b5623af430 2024-05-09T08:43:47.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.786+0000 7fe1fd7fa640 1 -- 172.21.15.107:0/1404674381 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe2100dc0f0 con 0x55b5623af430 2024-05-09T08:43:47.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.786+0000 7fe21e7bf740 10 client.18546 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:47.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.786+0000 7fe21e7bf740 1 -- 172.21.15.107:0/1404674381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55b561fe80f0 con 0x55b5623af430 2024-05-09T08:43:47.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.787+0000 7fe1fd7fa640 1 -- 172.21.15.107:0/1404674381 <== mon.0 v2:172.21.15.107:3300/0 4 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe2100c0120 con 0x55b5623af430 2024-05-09T08:43:47.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.787+0000 7fe21e7bf740 10 client.18546 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:47.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.787+0000 7fe21e7bf740 10 client.18546 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:47.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.787+0000 7fe21e7bf740 20 client.18546 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:47.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.787+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/1404674381 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b5623ca290 0x55b5623cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.787+0000 7fe21e7bf740 4 client.18546 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:47.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.787+0000 7fe21e7bf740 1 -- 172.21.15.107:0/1404674381 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b562322430 con 0x55b5623ca290 2024-05-09T08:43:47.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.787+0000 7fe21904e640 1 --2- 172.21.15.107:0/1404674381 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b5623ca290 0x55b5623cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.789+0000 7fe21904e640 1 --2- 172.21.15.107:0/1404674381 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b5623ca290 0x55b5623cc6e0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1f8e5d88f93a1f3 server_cookie=2d4d38931c464542 in_seq=0 out_seq=0 2024-05-09T08:43:47.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.789+0000 7fe1fd7fa640 10 client.18546 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:47.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.790+0000 7fe1fd7fa640 1 -- 172.21.15.107:0/1404674381 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b562322430 con 0x55b5623ca290 2024-05-09T08:43:47.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.790+0000 7fe1fd7fa640 10 client.18546 handle_command_reply: tid=0 2024-05-09T08:43:47.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.790+0000 7fe21e7bf740 1 client.18546 shutdown 2024-05-09T08:43:47.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.790+0000 7fe21e7bf740 1 -- 172.21.15.107:0/1404674381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 msgr2=0x55b5623b7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:47.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.790+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/1404674381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623b7400 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fe210005940 tx=0x7fe2100ba600 comp rx=0 tx=0).stop 2024-05-09T08:43:47.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.791+0000 7fe21e7bf740 1 -- 172.21.15.107:0/1404674381 shutdown_connections 2024-05-09T08:43:47.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.791+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/1404674381 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b5623ca290 0x55b5623cc6e0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.791+0000 7fe1fd7fa640 0 client.18546 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:47.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.791+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/1404674381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623ba420 0x55b5623b7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.791+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/1404674381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623afd50 0x55b5623b7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.791+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/1404674381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623b7400 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.791+0000 7fe21e7bf740 1 -- 172.21.15.107:0/1404674381 >> 172.21.15.107:0/1404674381 conn(0x55b5623ab750 msgr2=0x55b5623aaa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:47.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.791+0000 7fe21e7bf740 1 -- 172.21.15.107:0/1404674381 shutdown_connections 2024-05-09T08:43:47.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.791+0000 7fe21e7bf740 1 -- 172.21.15.107:0/1404674381 wait complete. 2024-05-09T08:43:47.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.791+0000 7fe21e7bf740 20 client.18546 trim_cache size 0 max 16384 2024-05-09T08:43:47.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21e7bf740 1 Processor -- start 2024-05-09T08:43:47.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21e7bf740 1 -- start start 2024-05-09T08:43:47.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21e7bf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623c5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21e7bf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623afd50 0x55b5623b7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21e7bf740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623ba420 0x55b5623d1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21e7bf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c14c010 con 0x55b5623af430 2024-05-09T08:43:47.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21e7bf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c1498e0 con 0x55b5623afd50 2024-05-09T08:43:47.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21e7bf740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c147750 con 0x55b5623ba420 2024-05-09T08:43:47.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21984f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623c5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21984f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623c5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57632/0 (socket says 172.21.15.107:57632) 2024-05-09T08:43:47.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21984f640 1 -- 172.21.15.107:0/4177664453 learned_addr learned my addr 172.21.15.107:0/4177664453 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:47.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21904e640 1 --2- 172.21.15.107:0/4177664453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623afd50 0x55b5623b7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21a050640 1 --2- 172.21.15.107:0/4177664453 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623ba420 0x55b5623d1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21984f640 1 -- 172.21.15.107:0/4177664453 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623ba420 msgr2=0x55b5623d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21984f640 1 --2- 172.21.15.107:0/4177664453 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623ba420 0x55b5623d1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21984f640 1 -- 172.21.15.107:0/4177664453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623afd50 msgr2=0x55b5623b7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21984f640 1 --2- 172.21.15.107:0/4177664453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623afd50 0x55b5623b7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21984f640 1 -- 172.21.15.107:0/4177664453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5623b1b20 con 0x55b5623af430 2024-05-09T08:43:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.810+0000 7fe21984f640 1 --2- 172.21.15.107:0/4177664453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623c5c80 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fe2100bbf80 tx=0x7fe2100dc000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=165155dc1272be23 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.811+0000 7fe1ff7fe640 1 -- 172.21.15.107:0/4177664453 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21009e030 con 0x55b5623af430 2024-05-09T08:43:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.811+0000 7fe1ff7fe640 1 -- 172.21.15.107:0/4177664453 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2100c4e40 con 0x55b5623af430 2024-05-09T08:43:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.811+0000 7fe1ff7fe640 1 -- 172.21.15.107:0/4177664453 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2100c0120 con 0x55b5623af430 2024-05-09T08:43:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.811+0000 7fe21e7bf740 1 -- 172.21.15.107:0/4177664453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 msgr2=0x55b5623c5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.811+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/4177664453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623c5c80 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fe2100bbf80 tx=0x7fe2100dc000 comp rx=0 tx=0).stop 2024-05-09T08:43:47.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.811+0000 7fe21e7bf740 1 -- 172.21.15.107:0/4177664453 shutdown_connections 2024-05-09T08:43:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.811+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/4177664453 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623ba420 0x55b5623d1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.811+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/4177664453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623afd50 0x55b5623b7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.811+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/4177664453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623c5c80 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.811+0000 7fe21e7bf740 1 -- 172.21.15.107:0/4177664453 >> 172.21.15.107:0/4177664453 conn(0x55b5623ab750 msgr2=0x55b5623b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.811+0000 7fe21e7bf740 1 -- 172.21.15.107:0/4177664453 shutdown_connections 2024-05-09T08:43:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.811+0000 7fe21e7bf740 1 -- 172.21.15.107:0/4177664453 wait complete. 2024-05-09T08:43:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.812+0000 7fe21e7bf740 1 -- start start 2024-05-09T08:43:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.812+0000 7fe21e7bf740 1 Processor -- start 2024-05-09T08:43:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.812+0000 7fe21e7bf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623c2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.812+0000 7fe21e7bf740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623afd50 0x55b5623b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.812+0000 7fe21e7bf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623ba420 0x55b5623b4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.812+0000 7fe21e7bf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c147da0 con 0x55b5623af430 2024-05-09T08:43:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.812+0000 7fe21e7bf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c149210 con 0x55b5623ba420 2024-05-09T08:43:47.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.812+0000 7fe21e7bf740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c148470 con 0x55b5623afd50 2024-05-09T08:43:47.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.812+0000 7fe21e7bf740 10 client.0 _ll_register_callbacks cb 0x7fe2100bbdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:47.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.812+0000 7fe21984f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623c2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.812+0000 7fe21984f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623c2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57648/0 (socket says 172.21.15.107:57648) 2024-05-09T08:43:47.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.812+0000 7fe21984f640 1 -- 172.21.15.107:0/2572026496 learned_addr learned my addr 172.21.15.107:0/2572026496 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:47.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.812+0000 7fe21904e640 1 --2- 172.21.15.107:0/2572026496 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623afd50 0x55b5623b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.812+0000 7fe21a050640 1 --2- 172.21.15.107:0/2572026496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623ba420 0x55b5623b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe21984f640 1 -- 172.21.15.107:0/2572026496 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623afd50 msgr2=0x55b5623b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:47.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe21984f640 1 --2- 172.21.15.107:0/2572026496 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623afd50 0x55b5623b3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe21984f640 1 -- 172.21.15.107:0/2572026496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623ba420 msgr2=0x55b5623b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:47.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe21984f640 1 --2- 172.21.15.107:0/2572026496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623ba420 0x55b5623b4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe21984f640 1 -- 172.21.15.107:0/2572026496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5623b4a90 con 0x55b5623af430 2024-05-09T08:43:47.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe21984f640 1 --2- 172.21.15.107:0/2572026496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623c2490 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fe2100c4910 tx=0x7fe2100bc400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18b6d600b45cba4b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:47.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe1fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:43:47.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe1fd7fa640 1 -- 172.21.15.107:0/2572026496 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21009e070 con 0x55b5623af430 2024-05-09T08:43:47.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe1fd7fa640 1 -- 172.21.15.107:0/2572026496 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fe2100c4bb0 con 0x55b5623af430 2024-05-09T08:43:47.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe21e7bf740 1 -- 172.21.15.107:0/2572026496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2100c4e40 con 0x55b5623af430 2024-05-09T08:43:47.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe1fd7fa640 1 -- 172.21.15.107:0/2572026496 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe2100ce050 con 0x55b5623af430 2024-05-09T08:43:47.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe21e7bf740 10 client.18552 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:47.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe21e7bf740 1 -- 172.21.15.107:0/2572026496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b5623b4a90 con 0x55b5623af430 2024-05-09T08:43:47.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe1fd7fa640 1 -- 172.21.15.107:0/2572026496 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2100c4e40 con 0x55b5623af430 2024-05-09T08:43:47.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe1fd7fa640 1 -- 172.21.15.107:0/2572026496 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe2100bc570 con 0x55b5623af430 2024-05-09T08:43:47.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe21e7bf740 10 client.18552 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:47.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe21e7bf740 10 client.18552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:47.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe21e7bf740 20 client.18552 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:47.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/2572026496 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b5623ca290 0x55b562432930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe21e7bf740 4 client.18552 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:43:47.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.813+0000 7fe21e7bf740 1 -- 172.21.15.107:0/2572026496 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b56231c780 con 0x55b5623ca290 2024-05-09T08:43:47.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe21904e640 1 --2- 172.21.15.107:0/2572026496 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b5623ca290 0x55b562432930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe1fd7fa640 10 client.18552 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:47.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe21904e640 1 --2- 172.21.15.107:0/2572026496 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b5623ca290 0x55b562432930 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf3a79ef535032c9 server_cookie=f11f25b1a34e8346 in_seq=0 out_seq=0 2024-05-09T08:43:47.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe1fd7fa640 1 -- 172.21.15.107:0/2572026496 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b56231c780 con 0x55b5623ca290 2024-05-09T08:43:47.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe1fd7fa640 10 client.18552 handle_command_reply: tid=0 2024-05-09T08:43:47.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe21e7bf740 1 client.18552 shutdown 2024-05-09T08:43:47.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe21e7bf740 1 -- 172.21.15.107:0/2572026496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 msgr2=0x55b5623c2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:47.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/2572026496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623c2490 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fe2100c4910 tx=0x7fe2100bc400 comp rx=0 tx=0).stop 2024-05-09T08:43:47.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe21e7bf740 1 -- 172.21.15.107:0/2572026496 shutdown_connections 2024-05-09T08:43:47.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/2572026496 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b5623ca290 0x55b562432930 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/2572026496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b5623ba420 0x55b5623b4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/2572026496 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5623afd50 0x55b5623b3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe21e7bf740 1 --2- 172.21.15.107:0/2572026496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5623af430 0x55b5623c2490 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe1fd7fa640 0 client.18552 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:47.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe21e7bf740 1 -- 172.21.15.107:0/2572026496 >> 172.21.15.107:0/2572026496 conn(0x55b5623ab750 msgr2=0x55b5623b8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:47.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe21e7bf740 1 -- 172.21.15.107:0/2572026496 shutdown_connections 2024-05-09T08:43:47.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.814+0000 7fe21e7bf740 1 -- 172.21.15.107:0/2572026496 wait complete. 2024-05-09T08:43:47.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.815+0000 7fe21e7bf740 20 client.18552 trim_cache size 0 max 16384 2024-05-09T08:43:47.821 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:43:47.821 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:43:47.821 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:43:47.821 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:43:47.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.815+0000 7fe21a851640 1 -- 172.21.15.107:0/3619585527 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2000a2d00 msgr2=0x7fe2000a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:47.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.815+0000 7fe21a851640 1 --2- 172.21.15.107:0/3619585527 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2000a2d00 0x7fe2000a51c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe20c133fa0 tx=0x7fe20400b410 comp rx=0 tx=0).stop 2024-05-09T08:43:47.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.815+0000 7fe21a851640 1 -- 172.21.15.107:0/3619585527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe20c004c80 msgr2=0x7fe20c138690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:47.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.815+0000 7fe21a851640 1 --2- 172.21.15.107:0/3619585527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe20c004c80 0x7fe20c138690 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe210031440 tx=0x7fe2100045e0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.815+0000 7fe21a851640 1 -- 172.21.15.107:0/3619585527 shutdown_connections 2024-05-09T08:43:47.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.815+0000 7fe21a851640 1 --2- 172.21.15.107:0/3619585527 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2000a2d00 0x7fe2000a51c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.815+0000 7fe21a851640 1 --2- 172.21.15.107:0/3619585527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe20c09acf0 0x7fe20c132750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.815+0000 7fe21a851640 1 --2- 172.21.15.107:0/3619585527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe20c0986e0 0x7fe20c138bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.815+0000 7fe21a851640 1 --2- 172.21.15.107:0/3619585527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe20c004c80 0x7fe20c138690 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.815+0000 7fe21a851640 1 -- 172.21.15.107:0/3619585527 >> 172.21.15.107:0/3619585527 conn(0x7fe20c00a740 msgr2=0x7fe20c092270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:47.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.815+0000 7fe21a851640 1 -- 172.21.15.107:0/3619585527 shutdown_connections 2024-05-09T08:43:47.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.815+0000 7fe21a851640 1 -- 172.21.15.107:0/3619585527 wait complete. 2024-05-09T08:43:47.825 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:43:47.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:43:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.948+0000 7f7bb59e9640 1 -- 172.21.15.107:0/2602457204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bb0104200 msgr2=0x7f7bb0110710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.948+0000 7f7bb59e9640 1 --2- 172.21.15.107:0/2602457204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bb0104200 0x7f7bb0110710 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8009c10 tx=0x7f7ba802a6f0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.951+0000 7f7bb59e9640 1 -- 172.21.15.107:0/2602457204 shutdown_connections 2024-05-09T08:43:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.951+0000 7f7bb59e9640 1 --2- 172.21.15.107:0/2602457204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bb0104200 0x7f7bb0110710 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.951+0000 7f7bb59e9640 1 --2- 172.21.15.107:0/2602457204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7bb0103840 0x7f7bb0103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.951+0000 7f7bb59e9640 1 --2- 172.21.15.107:0/2602457204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7bb0109850 0x7f7bb0109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.951+0000 7f7bb59e9640 1 -- 172.21.15.107:0/2602457204 >> 172.21.15.107:0/2602457204 conn(0x7f7bb00ff310 msgr2=0x7f7bb0101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:47.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.952+0000 7f7bb59e9640 1 -- 172.21.15.107:0/2602457204 shutdown_connections 2024-05-09T08:43:47.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.952+0000 7f7bb59e9640 1 -- 172.21.15.107:0/2602457204 wait complete. 2024-05-09T08:43:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.953+0000 7f7bb59e9640 1 Processor -- start 2024-05-09T08:43:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.953+0000 7f7bb59e9640 1 -- start start 2024-05-09T08:43:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.953+0000 7f7bb59e9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7bb0103840 0x7f7bb0073360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.953+0000 7f7bb59e9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bb0104200 0x7f7bb006b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.953+0000 7f7bb59e9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7bb0109850 0x7f7bb006b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.953+0000 7f7bb59e9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb0113b10 con 0x7f7bb0104200 2024-05-09T08:43:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.953+0000 7f7bb59e9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb0113990 con 0x7f7bb0109850 2024-05-09T08:43:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.953+0000 7f7bb59e9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb0113c90 con 0x7f7bb0103840 2024-05-09T08:43:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.954+0000 7f7baf7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7bb0109850 0x7f7bb006b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.954+0000 7f7baf7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7bb0109850 0x7f7bb006b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34446/0 (socket says 172.21.15.107:34446) 2024-05-09T08:43:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.954+0000 7f7baf7fe640 1 -- 172.21.15.107:0/3898857003 learned_addr learned my addr 172.21.15.107:0/3898857003 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.954+0000 7f7bae7fc640 1 --2- 172.21.15.107:0/3898857003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bb0104200 0x7f7bb006b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.955+0000 7f7baf7fe640 1 -- 172.21.15.107:0/3898857003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7bb0103840 msgr2=0x7f7bb0073360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.955+0000 7f7baf7fe640 1 --2- 172.21.15.107:0/3898857003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7bb0103840 0x7f7bb0073360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.955+0000 7f7baf7fe640 1 -- 172.21.15.107:0/3898857003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bb0104200 msgr2=0x7f7bb006b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.955+0000 7f7baf7fe640 1 --2- 172.21.15.107:0/3898857003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bb0104200 0x7f7bb006b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.955+0000 7f7baf7fe640 1 -- 172.21.15.107:0/3898857003 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb006c080 con 0x7f7bb0109850 2024-05-09T08:43:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.955+0000 7f7bae7fc640 1 --2- 172.21.15.107:0/3898857003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bb0104200 0x7f7bb006b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:43:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.955+0000 7f7baf7fe640 1 --2- 172.21.15.107:0/3898857003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7bb0109850 0x7f7bb006b860 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8004af0 tx=0x7f7ba8003c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.956+0000 7f7bb49e7640 1 -- 172.21.15.107:0/3898857003 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8003e60 con 0x7f7bb0109850 2024-05-09T08:43:47.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.956+0000 7f7bb49e7640 1 -- 172.21.15.107:0/3898857003 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ba8058070 con 0x7f7bb0109850 2024-05-09T08:43:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.957+0000 7f7bb59e9640 1 -- 172.21.15.107:0/3898857003 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bb0075c70 con 0x7f7bb0109850 2024-05-09T08:43:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.957+0000 7f7bb59e9640 1 -- 172.21.15.107:0/3898857003 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bb0076160 con 0x7f7bb0109850 2024-05-09T08:43:47.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.957+0000 7f7bb49e7640 1 -- 172.21.15.107:0/3898857003 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8004040 con 0x7f7bb0109850 2024-05-09T08:43:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.959+0000 7f7bb59e9640 1 -- 172.21.15.107:0/3898857003 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bb0104f80 con 0x7f7bb0109850 2024-05-09T08:43:47.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.962+0000 7f7bb49e7640 1 -- 172.21.15.107:0/3898857003 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7ba8063c90 con 0x7f7bb0109850 2024-05-09T08:43:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.963+0000 7f7bb49e7640 1 --2- 172.21.15.107:0/3898857003 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b980a2d30 0x7f7b980a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.963+0000 7f7bb49e7640 1 -- 172.21.15.107:0/3898857003 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f7ba811a780 con 0x7f7bb0109850 2024-05-09T08:43:47.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.964+0000 7f7baeffd640 1 --2- 172.21.15.107:0/3898857003 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b980a2d30 0x7f7b980a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:47.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.968+0000 7f7bb49e7640 1 -- 172.21.15.107:0/3898857003 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7ba80e7140 con 0x7f7bb0109850 2024-05-09T08:43:47.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:47.972+0000 7f7baeffd640 1 --2- 172.21.15.107:0/3898857003 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b980a2d30 0x7f7b980a51f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0009ad0 tx=0x7f7ba0009590 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:48.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.286+0000 7f7bb59e9640 1 -- 172.21.15.107:0/3898857003 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bb006c850 con 0x7f7bb0109850 2024-05-09T08:43:48.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.287+0000 7f7bb49e7640 1 -- 172.21.15.107:0/3898857003 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11887 (secure 0 0 0) 0x7f7ba80e6a40 con 0x7f7bb0109850 2024-05-09T08:43:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:43:48.288 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:37:10.930317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:43:48.289 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2024-05-09T08:43:48.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.296+0000 7f7bb59e9640 1 -- 172.21.15.107:0/3898857003 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b980a2d30 msgr2=0x7f7b980a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.296+0000 7f7bb59e9640 1 --2- 172.21.15.107:0/3898857003 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b980a2d30 0x7f7b980a51f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0009ad0 tx=0x7f7ba0009590 comp rx=0 tx=0).stop 2024-05-09T08:43:48.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.296+0000 7f7bb59e9640 1 -- 172.21.15.107:0/3898857003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7bb0109850 msgr2=0x7f7bb006b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.296+0000 7f7bb59e9640 1 --2- 172.21.15.107:0/3898857003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7bb0109850 0x7f7bb006b860 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8004af0 tx=0x7f7ba8003c70 comp rx=0 tx=0).stop 2024-05-09T08:43:48.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.299+0000 7f7bb59e9640 1 -- 172.21.15.107:0/3898857003 shutdown_connections 2024-05-09T08:43:48.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.299+0000 7f7bb59e9640 1 --2- 172.21.15.107:0/3898857003 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b980a2d30 0x7f7b980a51f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.299+0000 7f7bb59e9640 1 --2- 172.21.15.107:0/3898857003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7bb0109850 0x7f7bb006b860 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.299+0000 7f7bb59e9640 1 --2- 172.21.15.107:0/3898857003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bb0104200 0x7f7bb006b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.299+0000 7f7bb59e9640 1 --2- 172.21.15.107:0/3898857003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7bb0103840 0x7f7bb0073360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.299+0000 7f7bb59e9640 1 -- 172.21.15.107:0/3898857003 >> 172.21.15.107:0/3898857003 conn(0x7f7bb00ff310 msgr2=0x7f7bb00ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:48.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.305+0000 7f7bb59e9640 1 -- 172.21.15.107:0/3898857003 shutdown_connections 2024-05-09T08:43:48.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.307+0000 7f7bb59e9640 1 -- 172.21.15.107:0/3898857003 wait complete. 2024-05-09T08:43:48.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:43:48.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.568+0000 7f1912dfb640 1 -- 172.21.15.107:0/3298642072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19040a51c0 msgr2=0x7f19040a55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.568+0000 7f1910df7640 1 -- 172.21.15.107:0/3298642072 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908037070 con 0x7f19040a51c0 2024-05-09T08:43:48.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.568+0000 7f1912dfb640 1 --2- 172.21.15.107:0/3298642072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19040a51c0 0x7f19040a55a0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f1908009860 tx=0x7f190802a360 comp rx=0 tx=0).stop 2024-05-09T08:43:48.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.569+0000 7f1912dfb640 1 -- 172.21.15.107:0/3298642072 shutdown_connections 2024-05-09T08:43:48.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.569+0000 7f1912dfb640 1 --2- 172.21.15.107:0/3298642072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19040aa290 0x7f19040b1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.569+0000 7f1912dfb640 1 --2- 172.21.15.107:0/3298642072 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19040a5b70 0x7f19040a9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.569+0000 7f1912dfb640 1 --2- 172.21.15.107:0/3298642072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19040a51c0 0x7f19040a55a0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.569+0000 7f1912dfb640 1 -- 172.21.15.107:0/3298642072 >> 172.21.15.107:0/3298642072 conn(0x7f190401aa60 msgr2=0x7f190401ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:48.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.569+0000 7f1912dfb640 1 -- 172.21.15.107:0/3298642072 shutdown_connections 2024-05-09T08:43:48.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.570+0000 7f1912dfb640 1 -- 172.21.15.107:0/3298642072 wait complete. 2024-05-09T08:43:48.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.570+0000 7f1912dfb640 1 Processor -- start 2024-05-09T08:43:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.570+0000 7f1912dfb640 1 -- start start 2024-05-09T08:43:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.571+0000 7f1912dfb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19040a51c0 0x7f1904149880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.571+0000 7f1912dfb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19040aa290 0x7f1904149dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.571+0000 7f1912dfb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1904143a30 0x7f1904143ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.571+0000 7f1912dfb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f19040b4df0 con 0x7f19040a51c0 2024-05-09T08:43:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.571+0000 7f1912dfb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f19040b4c70 con 0x7f19040aa290 2024-05-09T08:43:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.571+0000 7f1912dfb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f19040b4f70 con 0x7f1904143a30 2024-05-09T08:43:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.571+0000 7f19125fa640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1904143a30 0x7f1904143ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.571+0000 7f1911df9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19040a51c0 0x7f1904149880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.571+0000 7f19125fa640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1904143a30 0x7f1904143ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47758/0 (socket says 172.21.15.107:47758) 2024-05-09T08:43:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.571+0000 7f19125fa640 1 -- 172.21.15.107:0/2905193247 learned_addr learned my addr 172.21.15.107:0/2905193247 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.571+0000 7f19115f8640 1 --2- 172.21.15.107:0/2905193247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19040aa290 0x7f1904149dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.572+0000 7f1911df9640 1 -- 172.21.15.107:0/2905193247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1904143a30 msgr2=0x7f1904143ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.572+0000 7f1911df9640 1 --2- 172.21.15.107:0/2905193247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1904143a30 0x7f1904143ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.572+0000 7f1911df9640 1 -- 172.21.15.107:0/2905193247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19040aa290 msgr2=0x7f1904149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.572+0000 7f1911df9640 1 --2- 172.21.15.107:0/2905193247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19040aa290 0x7f1904149dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.572+0000 7f1911df9640 1 -- 172.21.15.107:0/2905193247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1904144550 con 0x7f19040a51c0 2024-05-09T08:43:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.572+0000 7f19125fa640 1 --2- 172.21.15.107:0/2905193247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1904143a30 0x7f1904143ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.572+0000 7f19115f8640 1 --2- 172.21.15.107:0/2905193247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19040aa290 0x7f1904149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.572+0000 7f1911df9640 1 --2- 172.21.15.107:0/2905193247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19040a51c0 0x7f1904149880 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f1908009860 tx=0x7f1908005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.572+0000 7f1902ffd640 1 -- 172.21.15.107:0/2905193247 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908037040 con 0x7f19040a51c0 2024-05-09T08:43:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.573+0000 7f1912dfb640 1 -- 172.21.15.107:0/2905193247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19041447e0 con 0x7f19040a51c0 2024-05-09T08:43:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.573+0000 7f1912dfb640 1 -- 172.21.15.107:0/2905193247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19041509d0 con 0x7f19040a51c0 2024-05-09T08:43:48.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.573+0000 7f1902ffd640 1 -- 172.21.15.107:0/2905193247 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1908004040 con 0x7f19040a51c0 2024-05-09T08:43:48.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.573+0000 7f1902ffd640 1 -- 172.21.15.107:0/2905193247 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19080307e0 con 0x7f19040a51c0 2024-05-09T08:43:48.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.575+0000 7f1916d6a740 1 Processor -- start 2024-05-09T08:43:48.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.575+0000 7f1916d6a740 1 -- start start 2024-05-09T08:43:48.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f1916d6a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 0x55f6177878d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f1916d6a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 0x55f617791fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f1916d6a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6177924e0 0x55f6177948d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f1916d6a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f6176ecc40 con 0x55f617787e10 2024-05-09T08:43:48.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f1916d6a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f617694580 con 0x55f6177924e0 2024-05-09T08:43:48.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f1916d6a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f6176e8c00 con 0x55f6177874f0 2024-05-09T08:43:48.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f1902ffd640 1 -- 172.21.15.107:0/2905193247 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1908044030 con 0x7f19040a51c0 2024-05-09T08:43:48.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f19125fa640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 0x55f6177878d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f19125fa640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 0x55f6177878d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47762/0 (socket says 172.21.15.107:47762) 2024-05-09T08:43:48.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f19125fa640 1 -- 172.21.15.107:0/3685446199 learned_addr learned my addr 172.21.15.107:0/3685446199 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:48.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f19115f8640 1 --2- 172.21.15.107:0/3685446199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6177924e0 0x55f6177948d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f1911df9640 1 --2- 172.21.15.107:0/3685446199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 0x55f617791fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f19125fa640 1 -- 172.21.15.107:0/3685446199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6177924e0 msgr2=0x55f6177948d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:48.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f19125fa640 1 --2- 172.21.15.107:0/3685446199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6177924e0 0x55f6177948d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f19125fa640 1 -- 172.21.15.107:0/3685446199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 msgr2=0x55f617791fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:48.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f19125fa640 1 --2- 172.21.15.107:0/3685446199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 0x55f617791fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.576+0000 7f19125fa640 1 -- 172.21.15.107:0/3685446199 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6174003f0 con 0x55f6177874f0 2024-05-09T08:43:48.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.577+0000 7f19125fa640 1 --2- 172.21.15.107:0/3685446199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 0x55f6177878d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f190c000b60 tx=0x7f190c050610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15afcd15bac3e242 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.577+0000 7f18f7fff640 1 -- 172.21.15.107:0/3685446199 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190c080070 con 0x55f6177874f0 2024-05-09T08:43:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.577+0000 7f18f7fff640 1 -- 172.21.15.107:0/3685446199 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f190c06bdc0 con 0x55f6177874f0 2024-05-09T08:43:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.577+0000 7f1902ffd640 1 --2- 172.21.15.107:0/2905193247 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f18f80a2f00 0x7f18f80a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.577+0000 7f1902ffd640 1 -- 172.21.15.107:0/2905193247 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f19080eeb50 con 0x7f19040a51c0 2024-05-09T08:43:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.577+0000 7f18f7fff640 1 -- 172.21.15.107:0/3685446199 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190c078630 con 0x55f6177874f0 2024-05-09T08:43:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.577+0000 7f1911df9640 1 --2- 172.21.15.107:0/2905193247 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f18f80a2f00 0x7f18f80a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.578+0000 7f1916d6a740 1 -- 172.21.15.107:0/3685446199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 msgr2=0x55f6177878d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.578+0000 7f1916d6a740 1 --2- 172.21.15.107:0/3685446199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 0x55f6177878d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f190c000b60 tx=0x7f190c050610 comp rx=0 tx=0).stop 2024-05-09T08:43:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.578+0000 7f1911df9640 1 --2- 172.21.15.107:0/2905193247 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f18f80a2f00 0x7f18f80a53c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x55f617788ab0 tx=0x7f1908005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.578+0000 7f1916d6a740 1 -- 172.21.15.107:0/3685446199 shutdown_connections 2024-05-09T08:43:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.578+0000 7f1916d6a740 1 --2- 172.21.15.107:0/3685446199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6177924e0 0x55f6177948d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.578+0000 7f1916d6a740 1 --2- 172.21.15.107:0/3685446199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 0x55f617791fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.578+0000 7f1916d6a740 1 --2- 172.21.15.107:0/3685446199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 0x55f6177878d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.578+0000 7f1916d6a740 1 -- 172.21.15.107:0/3685446199 >> 172.21.15.107:0/3685446199 conn(0x55f617783810 msgr2=0x55f617783bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.578+0000 7f1916d6a740 1 -- 172.21.15.107:0/3685446199 shutdown_connections 2024-05-09T08:43:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.578+0000 7f1916d6a740 1 -- 172.21.15.107:0/3685446199 wait complete. 2024-05-09T08:43:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.578+0000 7f1916d6a740 1 -- start start 2024-05-09T08:43:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.578+0000 7f1916d6a740 1 Processor -- start 2024-05-09T08:43:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f1916d6a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 0x55f617791f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f1916d6a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 0x55f61778f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f1916d6a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6177924e0 0x55f61778f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f1916d6a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f6177952d0 con 0x55f617787e10 2024-05-09T08:43:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f1916d6a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f6176a7c40 con 0x55f6177924e0 2024-05-09T08:43:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f1916d6a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f617795450 con 0x55f6177874f0 2024-05-09T08:43:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f1916d6a740 10 client.0 _ll_register_callbacks cb 0x55f6173ffbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f19115f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 0x55f61778f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f19115f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 0x55f61778f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57712/0 (socket says 172.21.15.107:57712) 2024-05-09T08:43:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f19115f8640 1 -- 172.21.15.107:0/250493020 learned_addr learned my addr 172.21.15.107:0/250493020 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f19125fa640 1 --2- 172.21.15.107:0/250493020 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 0x55f617791f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f19125fa640 1 --2- 172.21.15.107:0/250493020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6177924e0 0x55f61778f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f19115f8640 1 -- 172.21.15.107:0/250493020 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 msgr2=0x55f617791f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f19115f8640 1 --2- 172.21.15.107:0/250493020 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 0x55f617791f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f19115f8640 1 -- 172.21.15.107:0/250493020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6177924e0 msgr2=0x55f61778f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f19115f8640 1 --2- 172.21.15.107:0/250493020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6177924e0 0x55f61778f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f19115f8640 1 -- 172.21.15.107:0/250493020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6174ca7d0 con 0x55f617787e10 2024-05-09T08:43:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f19115f8640 1 --2- 172.21.15.107:0/250493020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 0x55f61778f300 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f18fc012d20 tx=0x7f18fc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b010be42f6360a28 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.579+0000 7f18f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:43:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.580+0000 7f18f57fa640 1 -- 172.21.15.107:0/250493020 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18fc012110 con 0x55f617787e10 2024-05-09T08:43:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.580+0000 7f18f57fa640 1 -- 172.21.15.107:0/250493020 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f18fc004050 con 0x55f617787e10 2024-05-09T08:43:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.580+0000 7f1916d6a740 1 -- 172.21.15.107:0/250493020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f190c06bdc0 con 0x55f617787e10 2024-05-09T08:43:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.580+0000 7f18f57fa640 1 -- 172.21.15.107:0/250493020 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f18fc025050 con 0x55f617787e10 2024-05-09T08:43:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.580+0000 7f1916d6a740 10 client.18570 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.580+0000 7f1916d6a740 1 -- 172.21.15.107:0/250493020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f6177a1460 con 0x55f617787e10 2024-05-09T08:43:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.580+0000 7f18f57fa640 1 -- 172.21.15.107:0/250493020 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f190c06bdc0 con 0x55f617787e10 2024-05-09T08:43:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.580+0000 7f18f57fa640 1 -- 172.21.15.107:0/250493020 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f18fc010430 con 0x55f617787e10 2024-05-09T08:43:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.580+0000 7f1916d6a740 10 client.18570 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.580+0000 7f1916d6a740 10 client.18570 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:43:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.580+0000 7f1916d6a740 20 client.18570 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.580+0000 7f1916d6a740 1 --2- 172.21.15.107:0/250493020 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55f6177a29c0 0x55f6177a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.580+0000 7f1916d6a740 4 client.18570 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.580+0000 7f1916d6a740 1 -- 172.21.15.107:0/250493020 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6176fa370 con 0x55f6177a29c0 2024-05-09T08:43:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.581+0000 7f19125fa640 1 --2- 172.21.15.107:0/250493020 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55f6177a29c0 0x55f6177a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.581+0000 7f18f57fa640 10 client.18570 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:43:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.581+0000 7f19125fa640 1 --2- 172.21.15.107:0/250493020 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55f6177a29c0 0x55f6177a4e00 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f8e741e8c99719cd server_cookie=c170038cbc354305 in_seq=0 out_seq=0 2024-05-09T08:43:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.582+0000 7f18f57fa640 1 -- 172.21.15.107:0/250493020 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f6176fa370 con 0x55f6177a29c0 2024-05-09T08:43:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.582+0000 7f18f57fa640 10 client.18570 handle_command_reply: tid=0 2024-05-09T08:43:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.582+0000 7f1916d6a740 1 client.18570 shutdown 2024-05-09T08:43:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.582+0000 7f1916d6a740 1 -- 172.21.15.107:0/250493020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 msgr2=0x55f61778f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.582+0000 7f1916d6a740 1 --2- 172.21.15.107:0/250493020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 0x55f61778f300 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f18fc012d20 tx=0x7f18fc00f790 comp rx=0 tx=0).stop 2024-05-09T08:43:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.582+0000 7f1916d6a740 1 -- 172.21.15.107:0/250493020 shutdown_connections 2024-05-09T08:43:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.582+0000 7f1916d6a740 1 --2- 172.21.15.107:0/250493020 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55f6177a29c0 0x55f6177a4e00 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.582+0000 7f1916d6a740 1 --2- 172.21.15.107:0/250493020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6177924e0 0x55f61778f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.582+0000 7f1916d6a740 1 --2- 172.21.15.107:0/250493020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 0x55f61778f300 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.582+0000 7f18f57fa640 0 client.18570 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:43:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.582+0000 7f1916d6a740 1 --2- 172.21.15.107:0/250493020 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 0x55f617791f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.582+0000 7f1916d6a740 1 -- 172.21.15.107:0/250493020 >> 172.21.15.107:0/250493020 conn(0x55f617783810 msgr2=0x55f6177894f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.582+0000 7f1916d6a740 1 -- 172.21.15.107:0/250493020 shutdown_connections 2024-05-09T08:43:48.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.582+0000 7f1916d6a740 1 -- 172.21.15.107:0/250493020 wait complete. 2024-05-09T08:43:48.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.582+0000 7f1916d6a740 20 client.18570 trim_cache size 0 max 16384 2024-05-09T08:43:48.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.596+0000 7f1916d6a740 1 Processor -- start 2024-05-09T08:43:48.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.596+0000 7f1916d6a740 1 -- start start 2024-05-09T08:43:48.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.596+0000 7f1916d6a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 0x55f61779e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.596+0000 7f1916d6a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f617787e10 0x55f617789c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.596+0000 7f1916d6a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6177924e0 0x55f6177aa710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.596+0000 7f1916d6a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f18f00128a0 con 0x55f6177924e0 2024-05-09T08:43:48.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.596+0000 7f1916d6a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f18f0010170 con 0x55f617787e10 2024-05-09T08:43:48.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.596+0000 7f1916d6a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f18f000dfe0 con 0x55f6177874f0 2024-05-09T08:43:48.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.596+0000 7f19125fa640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 0x55f61779e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.596+0000 7f19125fa640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 0x55f61779e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47772/0 (socket says 172.21.15.107:47772) 2024-05-09T08:43:48.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.596+0000 7f19125fa640 1 -- 172.21.15.107:0/2275962936 learned_addr learned my addr 172.21.15.107:0/2275962936 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:48.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.596+0000 7f19125fa640 1 --2- 172.21.15.107:0/2275962936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6177924e0 0x55f6177aa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.596+0000 7f19115f8640 1 --2- 172.21.15.107:0/2275962936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f617787e10 0x55f617789c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.597+0000 7f19125fa640 1 -- 172.21.15.107:0/2275962936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f617787e10 msgr2=0x55f617789c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:48.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.597+0000 7f19125fa640 1 --2- 172.21.15.107:0/2275962936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f617787e10 0x55f617789c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.597+0000 7f19125fa640 1 -- 172.21.15.107:0/2275962936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6177924e0 msgr2=0x55f6177aa710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:48.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.597+0000 7f19125fa640 1 --2- 172.21.15.107:0/2275962936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6177924e0 0x55f6177aa710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.597+0000 7f19125fa640 1 -- 172.21.15.107:0/2275962936 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f61778ffd0 con 0x55f6177874f0 2024-05-09T08:43:48.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.597+0000 7f19125fa640 1 --2- 172.21.15.107:0/2275962936 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 0x55f61779e220 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f190c067530 tx=0x7f190c050610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7718ce241ea9c4d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:48.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.597+0000 7f18f77fe640 1 -- 172.21.15.107:0/2275962936 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190c080040 con 0x55f6177874f0 2024-05-09T08:43:48.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.597+0000 7f18f77fe640 1 -- 172.21.15.107:0/2275962936 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f190c06bd80 con 0x55f6177874f0 2024-05-09T08:43:48.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.597+0000 7f18f77fe640 1 -- 172.21.15.107:0/2275962936 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190c07a7c0 con 0x55f6177874f0 2024-05-09T08:43:48.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.597+0000 7f1916d6a740 1 -- 172.21.15.107:0/2275962936 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 msgr2=0x55f61779e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.597+0000 7f1916d6a740 1 --2- 172.21.15.107:0/2275962936 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 0x55f61779e220 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f190c067530 tx=0x7f190c050610 comp rx=0 tx=0).stop 2024-05-09T08:43:48.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f1916d6a740 1 -- 172.21.15.107:0/2275962936 shutdown_connections 2024-05-09T08:43:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f1916d6a740 1 --2- 172.21.15.107:0/2275962936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f6177924e0 0x55f6177aa710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f1916d6a740 1 --2- 172.21.15.107:0/2275962936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f617787e10 0x55f617789c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f1916d6a740 1 --2- 172.21.15.107:0/2275962936 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177874f0 0x55f61779e220 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f1916d6a740 1 -- 172.21.15.107:0/2275962936 >> 172.21.15.107:0/2275962936 conn(0x55f617783810 msgr2=0x55f61778a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f1916d6a740 1 -- 172.21.15.107:0/2275962936 shutdown_connections 2024-05-09T08:43:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f1916d6a740 1 -- 172.21.15.107:0/2275962936 wait complete. 2024-05-09T08:43:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f1916d6a740 1 -- start start 2024-05-09T08:43:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f1916d6a740 1 Processor -- start 2024-05-09T08:43:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f1916d6a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6177874f0 0x55f61778bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f1916d6a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 0x55f61778c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f1916d6a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177924e0 0x55f61778c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f1916d6a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f18f000e630 con 0x55f617787e10 2024-05-09T08:43:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f1916d6a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f18f000faa0 con 0x55f6177874f0 2024-05-09T08:43:48.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f1916d6a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f18f000ed00 con 0x55f6177924e0 2024-05-09T08:43:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f1916d6a740 10 client.0 _ll_register_callbacks cb 0x7f18fc009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f19125fa640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177924e0 0x55f61778c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f19125fa640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177924e0 0x55f61778c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47778/0 (socket says 172.21.15.107:47778) 2024-05-09T08:43:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.598+0000 7f19115f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 0x55f61778c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f19125fa640 1 -- 172.21.15.107:0/2204610699 learned_addr learned my addr 172.21.15.107:0/2204610699 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f19125fa640 1 --2- 172.21.15.107:0/2204610699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6177874f0 0x55f61778bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f19115f8640 1 -- 172.21.15.107:0/2204610699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177924e0 msgr2=0x55f61778c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f19115f8640 1 --2- 172.21.15.107:0/2204610699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177924e0 0x55f61778c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f19115f8640 1 -- 172.21.15.107:0/2204610699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6177874f0 msgr2=0x55f61778bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f19115f8640 1 --2- 172.21.15.107:0/2204610699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6177874f0 0x55f61778bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f19115f8640 1 -- 172.21.15.107:0/2204610699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f617798800 con 0x55f617787e10 2024-05-09T08:43:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f19125fa640 1 --2- 172.21.15.107:0/2204610699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177924e0 0x55f61778c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f19115f8640 1 --2- 172.21.15.107:0/2204610699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 0x55f61778c410 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f18fc012cf0 tx=0x7f18fc009c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=564f96362ce0ae63 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:48.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f18f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:43:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f18f57fa640 1 -- 172.21.15.107:0/2204610699 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18fc022040 con 0x55f617787e10 2024-05-09T08:43:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f18f57fa640 1 -- 172.21.15.107:0/2204610699 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f18fc012250 con 0x55f617787e10 2024-05-09T08:43:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f1916d6a740 1 -- 172.21.15.107:0/2204610699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f190c06bd80 con 0x55f617787e10 2024-05-09T08:43:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f18f57fa640 1 -- 172.21.15.107:0/2204610699 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f18fc03a050 con 0x55f617787e10 2024-05-09T08:43:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f1916d6a740 10 client.18576 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f1916d6a740 1 -- 172.21.15.107:0/2204610699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f6178091c0 con 0x55f617787e10 2024-05-09T08:43:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.599+0000 7f18f57fa640 1 -- 172.21.15.107:0/2204610699 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f190c06bd80 con 0x55f617787e10 2024-05-09T08:43:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.600+0000 7f18f57fa640 1 -- 172.21.15.107:0/2204610699 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f18fc0035f0 con 0x55f617787e10 2024-05-09T08:43:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.600+0000 7f1916d6a740 10 client.18576 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.600+0000 7f1916d6a740 10 client.18576 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:43:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.600+0000 7f1916d6a740 20 client.18576 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.600+0000 7f1916d6a740 1 --2- 172.21.15.107:0/2204610699 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55f617809970 0x55f61780bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.600+0000 7f1916d6a740 4 client.18576 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:43:48.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.600+0000 7f1916d6a740 1 -- 172.21.15.107:0/2204610699 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f6176f46c0 con 0x55f617809970 2024-05-09T08:43:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.600+0000 7f19125fa640 1 --2- 172.21.15.107:0/2204610699 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55f617809970 0x55f61780bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.600+0000 7f18f57fa640 10 client.18576 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:43:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.600+0000 7f19125fa640 1 --2- 172.21.15.107:0/2204610699 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55f617809970 0x55f61780bd90 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e7c9751f972eb97f server_cookie=e984de57fe330f1b in_seq=0 out_seq=0 2024-05-09T08:43:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.600+0000 7f18f57fa640 1 -- 172.21.15.107:0/2204610699 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f6176f46c0 con 0x55f617809970 2024-05-09T08:43:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.600+0000 7f18f57fa640 10 client.18576 handle_command_reply: tid=0 2024-05-09T08:43:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.600+0000 7f1916d6a740 1 client.18576 shutdown 2024-05-09T08:43:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.600+0000 7f1916d6a740 1 -- 172.21.15.107:0/2204610699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 msgr2=0x55f61778c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.600+0000 7f1916d6a740 1 --2- 172.21.15.107:0/2204610699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 0x55f61778c410 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f18fc012cf0 tx=0x7f18fc009c70 comp rx=0 tx=0).stop 2024-05-09T08:43:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1916d6a740 1 -- 172.21.15.107:0/2204610699 shutdown_connections 2024-05-09T08:43:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1916d6a740 1 --2- 172.21.15.107:0/2204610699 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55f617809970 0x55f61780bd90 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f18f57fa640 0 client.18576 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:43:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1916d6a740 1 --2- 172.21.15.107:0/2204610699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6177924e0 0x55f61778c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1916d6a740 1 --2- 172.21.15.107:0/2204610699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f617787e10 0x55f61778c410 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1916d6a740 1 --2- 172.21.15.107:0/2204610699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f6177874f0 0x55f61778bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1916d6a740 1 -- 172.21.15.107:0/2204610699 >> 172.21.15.107:0/2204610699 conn(0x55f617783810 msgr2=0x55f61778db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:48.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1916d6a740 1 -- 172.21.15.107:0/2204610699 shutdown_connections 2024-05-09T08:43:48.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1916d6a740 1 -- 172.21.15.107:0/2204610699 wait complete. 2024-05-09T08:43:48.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1916d6a740 20 client.18576 trim_cache size 0 max 16384 2024-05-09T08:43:48.608 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:43:48.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1912dfb640 1 -- 172.21.15.107:0/2905193247 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f18f80a2f00 msgr2=0x7f18f80a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1912dfb640 1 --2- 172.21.15.107:0/2905193247 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f18f80a2f00 0x7f18f80a53c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x55f617788ab0 tx=0x7f1908005f90 comp rx=0 tx=0).stop 2024-05-09T08:43:48.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1912dfb640 1 -- 172.21.15.107:0/2905193247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19040a51c0 msgr2=0x7f1904149880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1912dfb640 1 --2- 172.21.15.107:0/2905193247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19040a51c0 0x7f1904149880 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f1908009860 tx=0x7f1908005c70 comp rx=0 tx=0).stop 2024-05-09T08:43:48.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1912dfb640 1 -- 172.21.15.107:0/2905193247 shutdown_connections 2024-05-09T08:43:48.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1912dfb640 1 --2- 172.21.15.107:0/2905193247 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f18f80a2f00 0x7f18f80a53c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1912dfb640 1 --2- 172.21.15.107:0/2905193247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1904143a30 0x7f1904143ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1912dfb640 1 --2- 172.21.15.107:0/2905193247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19040aa290 0x7f1904149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1912dfb640 1 --2- 172.21.15.107:0/2905193247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19040a51c0 0x7f1904149880 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1912dfb640 1 -- 172.21.15.107:0/2905193247 >> 172.21.15.107:0/2905193247 conn(0x7f190401aa60 msgr2=0x7f19040a3100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:48.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1912dfb640 1 -- 172.21.15.107:0/2905193247 shutdown_connections 2024-05-09T08:43:48.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.601+0000 7f1912dfb640 1 -- 172.21.15.107:0/2905193247 wait complete. 2024-05-09T08:43:48.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:48 smithi107 ceph-mon[30122]: pgmap v323: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 12 MiB/s wr, 3.04k op/s 2024-05-09T08:43:48.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:48 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3898857003' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:43:48.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:43:48.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.757+0000 7fcdcb22c640 1 -- 172.21.15.107:0/3469402871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdc410e0b0 msgr2=0x7fcdc410e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.757+0000 7fcdcb22c640 1 --2- 172.21.15.107:0/3469402871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdc410e0b0 0x7fcdc410e530 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009cd0 tx=0x7fcdbc02a910 comp rx=0 tx=0).stop 2024-05-09T08:43:48.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.760+0000 7fcdcb22c640 1 -- 172.21.15.107:0/3469402871 shutdown_connections 2024-05-09T08:43:48.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.760+0000 7fcdcb22c640 1 --2- 172.21.15.107:0/3469402871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdc411b010 0x7fcdc411d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.760+0000 7fcdcb22c640 1 --2- 172.21.15.107:0/3469402871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdc410e0b0 0x7fcdc410e530 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.760+0000 7fcdcb22c640 1 --2- 172.21.15.107:0/3469402871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc4110080 0x7fcdc410db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.760+0000 7fcdcb22c640 1 -- 172.21.15.107:0/3469402871 >> 172.21.15.107:0/3469402871 conn(0x7fcdc406c940 msgr2=0x7fcdc406cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:48.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.761+0000 7fcdcb22c640 1 -- 172.21.15.107:0/3469402871 shutdown_connections 2024-05-09T08:43:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.761+0000 7fcdcb22c640 1 -- 172.21.15.107:0/3469402871 wait complete. 2024-05-09T08:43:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.762+0000 7fcdcb22c640 1 Processor -- start 2024-05-09T08:43:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.762+0000 7fcdcb22c640 1 -- start start 2024-05-09T08:43:48.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.762+0000 7fcdcb22c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdc410e0b0 0x7fcdc411a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.762+0000 7fcdcb22c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdc4110080 0x7fcdc4114790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.762+0000 7fcdcb22c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc411b010 0x7fcdc4114cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.762+0000 7fcdcb22c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc41110f0 con 0x7fcdc410e0b0 2024-05-09T08:43:48.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.762+0000 7fcdcb22c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc4110f70 con 0x7fcdc411b010 2024-05-09T08:43:48.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.762+0000 7fcdcb22c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc4111270 con 0x7fcdc4110080 2024-05-09T08:43:48.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.763+0000 7fcdc3fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdc4110080 0x7fcdc4114790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.764+0000 7fcdc3fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdc4110080 0x7fcdc4114790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47782/0 (socket says 172.21.15.107:47782) 2024-05-09T08:43:48.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.764+0000 7fcdc3fff640 1 -- 172.21.15.107:0/3934723629 learned_addr learned my addr 172.21.15.107:0/3934723629 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:48.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.765+0000 7fcdc97c9640 1 --2- 172.21.15.107:0/3934723629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc411b010 0x7fcdc4114cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.766+0000 7fcdc97c9640 1 -- 172.21.15.107:0/3934723629 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdc4110080 msgr2=0x7fcdc4114790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.766+0000 7fcdc97c9640 1 --2- 172.21.15.107:0/3934723629 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdc4110080 0x7fcdc4114790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.766+0000 7fcdc97c9640 1 -- 172.21.15.107:0/3934723629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdc410e0b0 msgr2=0x7fcdc411a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.766+0000 7fcdc97c9640 1 --2- 172.21.15.107:0/3934723629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdc410e0b0 0x7fcdc411a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.766+0000 7fcdc97c9640 1 -- 172.21.15.107:0/3934723629 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc4115400 con 0x7fcdc411b010 2024-05-09T08:43:48.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.766+0000 7fcdc97c9640 1 --2- 172.21.15.107:0/3934723629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc411b010 0x7fcdc4114cd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcdac00b3e0 tx=0x7fcdac00b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:48.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.768+0000 7fcdc1ffb640 1 -- 172.21.15.107:0/3934723629 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac015020 con 0x7fcdc411b010 2024-05-09T08:43:48.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.768+0000 7fcdc1ffb640 1 -- 172.21.15.107:0/3934723629 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdac011ce0 con 0x7fcdc411b010 2024-05-09T08:43:48.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.768+0000 7fcdc1ffb640 1 -- 172.21.15.107:0/3934723629 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac010e00 con 0x7fcdc411b010 2024-05-09T08:43:48.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.768+0000 7fcdcb22c640 1 -- 172.21.15.107:0/3934723629 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc41bbd60 con 0x7fcdc411b010 2024-05-09T08:43:48.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.768+0000 7fcdcb22c640 1 -- 172.21.15.107:0/3934723629 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc41bc1d0 con 0x7fcdc411b010 2024-05-09T08:43:48.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.773+0000 7fcdce00f740 1 Processor -- start 2024-05-09T08:43:48.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.774+0000 7fcdc1ffb640 1 -- 172.21.15.107:0/3934723629 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fcdac018020 con 0x7fcdc411b010 2024-05-09T08:43:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.775+0000 7fcdc1ffb640 1 --2- 172.21.15.107:0/3934723629 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcdb00a2e80 0x7fcdb00a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.776+0000 7fcdce00f740 1 -- start start 2024-05-09T08:43:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.776+0000 7fcdce00f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 0x55a11e285710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.776+0000 7fcdce00f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285c50 0x55a11e28fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.776+0000 7fcdce00f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e290320 0x55a11e292710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.776+0000 7fcdce00f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a11e1eac40 con 0x55a11e285330 2024-05-09T08:43:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.776+0000 7fcdce00f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a11e192580 con 0x55a11e285c50 2024-05-09T08:43:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.776+0000 7fcdce00f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a11e1e6c00 con 0x55a11e290320 2024-05-09T08:43:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.776+0000 7fcdc3fff640 1 --2- 172.21.15.107:0/3934723629 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcdb00a2e80 0x7fcdb00a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.776+0000 7fcdc1ffb640 1 -- 172.21.15.107:0/3934723629 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fcdac098f90 con 0x7fcdc411b010 2024-05-09T08:43:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.777+0000 7fcdc8fc8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 0x55a11e285710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.777+0000 7fcdc3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285c50 0x55a11e28fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.777+0000 7fcdc8fc8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 0x55a11e285710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57764/0 (socket says 172.21.15.107:57764) 2024-05-09T08:43:48.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.777+0000 7fcdc8fc8640 1 -- 172.21.15.107:0/3510106523 learned_addr learned my addr 172.21.15.107:0/3510106523 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:48.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.777+0000 7fcdc3fff640 1 -- 172.21.15.107:0/3510106523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e290320 msgr2=0x55a11e292710 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:43:48.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.777+0000 7fcdc3fff640 1 --2- 172.21.15.107:0/3510106523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e290320 0x55a11e292710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.777+0000 7fcdc3fff640 1 -- 172.21.15.107:0/3510106523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 msgr2=0x55a11e285710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:48.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.777+0000 7fcdc3fff640 1 --2- 172.21.15.107:0/3510106523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 0x55a11e285710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.777+0000 7fcdc3fff640 1 -- 172.21.15.107:0/3510106523 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a11defe3f0 con 0x55a11e285c50 2024-05-09T08:43:48.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.777+0000 7fcdc8fc8640 1 --2- 172.21.15.107:0/3510106523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 0x55a11e285710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:43:48.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.778+0000 7fcdc3fff640 1 --2- 172.21.15.107:0/3510106523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285c50 0x55a11e28fde0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4115fe0 tx=0x7fcdbc031560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9708d3122c91e151 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:48.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.778+0000 7fcdc3fff640 1 --2- 172.21.15.107:0/3934723629 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcdb00a2e80 0x7fcdb00a5340 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc005fe0 tx=0x7fcdbc045000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:48.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.779+0000 7fcdbaffd640 1 -- 172.21.15.107:0/3510106523 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc04f070 con 0x55a11e285c50 2024-05-09T08:43:48.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.780+0000 7fcdbaffd640 1 -- 172.21.15.107:0/3510106523 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdbc036050 con 0x55a11e285c50 2024-05-09T08:43:48.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.780+0000 7fcdbaffd640 1 -- 172.21.15.107:0/3510106523 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc030c80 con 0x55a11e285c50 2024-05-09T08:43:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.781+0000 7fcdce00f740 1 -- 172.21.15.107:0/3510106523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285c50 msgr2=0x55a11e28fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.781+0000 7fcdce00f740 1 --2- 172.21.15.107:0/3510106523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285c50 0x55a11e28fde0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4115fe0 tx=0x7fcdbc031560 comp rx=0 tx=0).stop 2024-05-09T08:43:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.781+0000 7fcdce00f740 1 -- 172.21.15.107:0/3510106523 shutdown_connections 2024-05-09T08:43:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.781+0000 7fcdce00f740 1 --2- 172.21.15.107:0/3510106523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e290320 0x55a11e292710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.781+0000 7fcdce00f740 1 --2- 172.21.15.107:0/3510106523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285c50 0x55a11e28fde0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.781+0000 7fcdce00f740 1 --2- 172.21.15.107:0/3510106523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 0x55a11e285710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.781+0000 7fcdce00f740 1 -- 172.21.15.107:0/3510106523 >> 172.21.15.107:0/3510106523 conn(0x55a11e281650 msgr2=0x55a11e281a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.781+0000 7fcdce00f740 1 -- 172.21.15.107:0/3510106523 shutdown_connections 2024-05-09T08:43:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.781+0000 7fcdce00f740 1 -- 172.21.15.107:0/3510106523 wait complete. 2024-05-09T08:43:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.781+0000 7fcdce00f740 1 -- start start 2024-05-09T08:43:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.781+0000 7fcdce00f740 1 Processor -- start 2024-05-09T08:43:48.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.782+0000 7fcdce00f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 0x55a11e28d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.782+0000 7fcdce00f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e285c50 0x55a11e28d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.782+0000 7fcdce00f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e290320 0x55a11e28dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.783+0000 7fcdce00f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a11e2930b0 con 0x55a11e285330 2024-05-09T08:43:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.783+0000 7fcdc8fc8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 0x55a11e28d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.783+0000 7fcdc8fc8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 0x55a11e28d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57772/0 (socket says 172.21.15.107:57772) 2024-05-09T08:43:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.783+0000 7fcdc8fc8640 1 -- 172.21.15.107:0/2063872570 learned_addr learned my addr 172.21.15.107:0/2063872570 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.783+0000 7fcdce00f740 1 -- 172.21.15.107:0/2063872570 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a11e1a5c40 con 0x55a11e290320 2024-05-09T08:43:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.783+0000 7fcdce00f740 1 -- 172.21.15.107:0/2063872570 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a11e293230 con 0x55a11e285c50 2024-05-09T08:43:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.783+0000 7fcdc97c9640 1 --2- 172.21.15.107:0/2063872570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e290320 0x55a11e28dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.783+0000 7fcdce00f740 10 client.0 _ll_register_callbacks cb 0x55a11defdbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.783+0000 7fcdc97c9640 1 -- 172.21.15.107:0/2063872570 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e285c50 msgr2=0x55a11e28d840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:43:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.783+0000 7fcdc97c9640 1 --2- 172.21.15.107:0/2063872570 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e285c50 0x55a11e28d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.783+0000 7fcdc97c9640 1 -- 172.21.15.107:0/2063872570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 msgr2=0x55a11e28d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:48.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.783+0000 7fcdc97c9640 1 --2- 172.21.15.107:0/2063872570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 0x55a11e28d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.783+0000 7fcdc97c9640 1 -- 172.21.15.107:0/2063872570 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a11dfc87d0 con 0x55a11e290320 2024-05-09T08:43:48.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.784+0000 7fcdc97c9640 1 --2- 172.21.15.107:0/2063872570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e290320 0x55a11e28dd80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcdac0cc120 tx=0x7fcdac0a3660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1e44d227f7a5b42 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:48.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.784+0000 7fcda3fff640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:43:48.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.784+0000 7fcda3fff640 1 -- 172.21.15.107:0/2063872570 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac097920 con 0x55a11e290320 2024-05-09T08:43:48.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.784+0000 7fcdce00f740 1 -- 172.21.15.107:0/2063872570 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdbc036050 con 0x55a11e290320 2024-05-09T08:43:48.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.784+0000 7fcda3fff640 1 -- 172.21.15.107:0/2063872570 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fcdac0cc290 con 0x55a11e290320 2024-05-09T08:43:48.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.784+0000 7fcda3fff640 1 -- 172.21.15.107:0/2063872570 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcdac097ac0 con 0x55a11e290320 2024-05-09T08:43:48.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.784+0000 7fcdce00f740 10 client.27536 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:48.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.784+0000 7fcdce00f740 1 -- 172.21.15.107:0/2063872570 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a11e29f3a0 con 0x55a11e290320 2024-05-09T08:43:48.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.785+0000 7fcda3fff640 1 -- 172.21.15.107:0/2063872570 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdac0b6020 con 0x55a11e290320 2024-05-09T08:43:48.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.785+0000 7fcda3fff640 1 -- 172.21.15.107:0/2063872570 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fcdac097560 con 0x55a11e290320 2024-05-09T08:43:48.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.785+0000 7fcdce00f740 10 client.27536 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:48.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.785+0000 7fcdce00f740 10 client.27536 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:43:48.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.785+0000 7fcdce00f740 20 client.27536 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:48.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.785+0000 7fcdce00f740 1 --2- 172.21.15.107:0/2063872570 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a11e2a0cf0 0x55a11e2a3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.786+0000 7fcdce00f740 4 client.27536 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:48.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.786+0000 7fcdce00f740 1 -- 172.21.15.107:0/2063872570 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a11e1f8370 con 0x55a11e2a0cf0 2024-05-09T08:43:48.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.786+0000 7fcdc8fc8640 1 --2- 172.21.15.107:0/2063872570 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a11e2a0cf0 0x55a11e2a3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.786+0000 7fcdc8fc8640 1 --2- 172.21.15.107:0/2063872570 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a11e2a0cf0 0x55a11e2a3150 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=97f22be9d87bd70b server_cookie=6c7547545d68936c in_seq=0 out_seq=0 2024-05-09T08:43:48.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.786+0000 7fcda3fff640 10 client.27536 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:43:48.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.788+0000 7fcda3fff640 1 -- 172.21.15.107:0/2063872570 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a11e1f8370 con 0x55a11e2a0cf0 2024-05-09T08:43:48.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.788+0000 7fcda3fff640 10 client.27536 handle_command_reply: tid=0 2024-05-09T08:43:48.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.788+0000 7fcdce00f740 1 client.27536 shutdown 2024-05-09T08:43:48.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.788+0000 7fcdce00f740 1 -- 172.21.15.107:0/2063872570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e290320 msgr2=0x55a11e28dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.788+0000 7fcdce00f740 1 --2- 172.21.15.107:0/2063872570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e290320 0x55a11e28dd80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcdac0cc120 tx=0x7fcdac0a3660 comp rx=0 tx=0).stop 2024-05-09T08:43:48.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.788+0000 7fcdce00f740 1 -- 172.21.15.107:0/2063872570 shutdown_connections 2024-05-09T08:43:48.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.788+0000 7fcdce00f740 1 --2- 172.21.15.107:0/2063872570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e290320 0x55a11e28dd80 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.788+0000 7fcdce00f740 1 --2- 172.21.15.107:0/2063872570 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a11e2a0cf0 0x55a11e2a3150 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.788+0000 7fcdce00f740 1 --2- 172.21.15.107:0/2063872570 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e285c50 0x55a11e28d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.788+0000 7fcdce00f740 1 --2- 172.21.15.107:0/2063872570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 0x55a11e28d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.788+0000 7fcdce00f740 1 -- 172.21.15.107:0/2063872570 >> 172.21.15.107:0/2063872570 conn(0x55a11e281650 msgr2=0x55a11e287350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:48.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.788+0000 7fcda3fff640 0 client.27536 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:43:48.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.788+0000 7fcdce00f740 1 -- 172.21.15.107:0/2063872570 shutdown_connections 2024-05-09T08:43:48.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.788+0000 7fcdce00f740 1 -- 172.21.15.107:0/2063872570 wait complete. 2024-05-09T08:43:48.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.789+0000 7fcdce00f740 20 client.27536 trim_cache size 0 max 16384 2024-05-09T08:43:48.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.818+0000 7fcdce00f740 1 Processor -- start 2024-05-09T08:43:48.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.818+0000 7fcdce00f740 1 -- start start 2024-05-09T08:43:48.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.819+0000 7fcdce00f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 0x55a11e29bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.819+0000 7fcdce00f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285c50 0x55a11e2a8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.819+0000 7fcdce00f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e290320 0x55a11e2a8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.819+0000 7fcdce00f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcda40128d0 con 0x55a11e285330 2024-05-09T08:43:48.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.819+0000 7fcdc3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285c50 0x55a11e2a8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.819+0000 7fcdc8fc8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 0x55a11e29bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.819+0000 7fcdc3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285c50 0x55a11e2a8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34548/0 (socket says 172.21.15.107:34548) 2024-05-09T08:43:48.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.819+0000 7fcdc8fc8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 0x55a11e29bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57778/0 (socket says 172.21.15.107:57778) 2024-05-09T08:43:48.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.819+0000 7fcdc3fff640 1 -- 172.21.15.107:0/3582621313 learned_addr learned my addr 172.21.15.107:0/3582621313 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:48.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.819+0000 7fcdc97c9640 1 --2- 172.21.15.107:0/3582621313 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e290320 0x55a11e2a8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.819+0000 7fcdce00f740 1 -- 172.21.15.107:0/3582621313 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcda40101a0 con 0x55a11e285c50 2024-05-09T08:43:48.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.819+0000 7fcdce00f740 1 -- 172.21.15.107:0/3582621313 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcda400e010 con 0x55a11e290320 2024-05-09T08:43:48.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.820+0000 7fcdc97c9640 1 -- 172.21.15.107:0/3582621313 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285c50 msgr2=0x55a11e2a8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:48.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.820+0000 7fcdc97c9640 1 --2- 172.21.15.107:0/3582621313 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285c50 0x55a11e2a8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.820+0000 7fcdc97c9640 1 -- 172.21.15.107:0/3582621313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 msgr2=0x55a11e29bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:48.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.820+0000 7fcdc97c9640 1 --2- 172.21.15.107:0/3582621313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 0x55a11e29bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.820+0000 7fcdc97c9640 1 -- 172.21.15.107:0/3582621313 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a11e272d20 con 0x55a11e290320 2024-05-09T08:43:48.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.820+0000 7fcdc97c9640 1 --2- 172.21.15.107:0/3582621313 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e290320 0x55a11e2a8fa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcdac0b93e0 tx=0x7fcdac006b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0dd0627da0d543c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:48.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.821+0000 7fcdba7fc640 1 -- 172.21.15.107:0/3582621313 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac097c60 con 0x55a11e290320 2024-05-09T08:43:48.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.821+0000 7fcdba7fc640 1 -- 172.21.15.107:0/3582621313 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdac0b6040 con 0x55a11e290320 2024-05-09T08:43:48.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.821+0000 7fcdba7fc640 1 -- 172.21.15.107:0/3582621313 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac097e30 con 0x55a11e290320 2024-05-09T08:43:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.823+0000 7fcdce00f740 1 -- 172.21.15.107:0/3582621313 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e290320 msgr2=0x55a11e2a8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.823+0000 7fcdce00f740 1 --2- 172.21.15.107:0/3582621313 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e290320 0x55a11e2a8fa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcdac0b93e0 tx=0x7fcdac006b00 comp rx=0 tx=0).stop 2024-05-09T08:43:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.823+0000 7fcdce00f740 1 -- 172.21.15.107:0/3582621313 shutdown_connections 2024-05-09T08:43:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.823+0000 7fcdce00f740 1 --2- 172.21.15.107:0/3582621313 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e290320 0x55a11e2a8fa0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.823+0000 7fcdce00f740 1 --2- 172.21.15.107:0/3582621313 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285c50 0x55a11e2a8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.823+0000 7fcdce00f740 1 --2- 172.21.15.107:0/3582621313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e285330 0x55a11e29bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.823+0000 7fcdce00f740 1 -- 172.21.15.107:0/3582621313 >> 172.21.15.107:0/3582621313 conn(0x55a11e281650 msgr2=0x55a11e281a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.823+0000 7fcdce00f740 1 -- 172.21.15.107:0/3582621313 shutdown_connections 2024-05-09T08:43:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.824+0000 7fcdce00f740 1 -- 172.21.15.107:0/3582621313 wait complete. 2024-05-09T08:43:48.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.824+0000 7fcdce00f740 1 -- start start 2024-05-09T08:43:48.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.824+0000 7fcdce00f740 1 Processor -- start 2024-05-09T08:43:48.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.825+0000 7fcdce00f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285330 0x55a11e2984f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.825+0000 7fcdce00f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e285c50 0x55a11e298a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.825+0000 7fcdce00f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e290320 0x55a11e293a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.825+0000 7fcdce00f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcda400e660 con 0x55a11e290320 2024-05-09T08:43:48.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.825+0000 7fcdce00f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcda400fad0 con 0x55a11e285330 2024-05-09T08:43:48.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.825+0000 7fcdce00f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcda400ed30 con 0x55a11e285c50 2024-05-09T08:43:48.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.825+0000 7fcdce00f740 10 client.0 _ll_register_callbacks cb 0x7fcdbc009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:48.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.825+0000 7fcdc8fc8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285330 0x55a11e2984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.825+0000 7fcdc8fc8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285330 0x55a11e2984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34558/0 (socket says 172.21.15.107:34558) 2024-05-09T08:43:48.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.825+0000 7fcdc8fc8640 1 -- 172.21.15.107:0/907253494 learned_addr learned my addr 172.21.15.107:0/907253494 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:48.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.825+0000 7fcdc3fff640 1 --2- 172.21.15.107:0/907253494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e285c50 0x55a11e298a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.825+0000 7fcdc97c9640 1 --2- 172.21.15.107:0/907253494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e290320 0x55a11e293a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.826+0000 7fcdc8fc8640 1 -- 172.21.15.107:0/907253494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e285c50 msgr2=0x55a11e298a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.826+0000 7fcdc8fc8640 1 --2- 172.21.15.107:0/907253494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e285c50 0x55a11e298a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.826+0000 7fcdc8fc8640 1 -- 172.21.15.107:0/907253494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e290320 msgr2=0x55a11e293a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.826+0000 7fcdc8fc8640 1 --2- 172.21.15.107:0/907253494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e290320 0x55a11e293a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.826+0000 7fcdc8fc8640 1 -- 172.21.15.107:0/907253494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a11e294260 con 0x55a11e285330 2024-05-09T08:43:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.826+0000 7fcdc8fc8640 1 --2- 172.21.15.107:0/907253494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285330 0x55a11e2984f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55a11e2a3a00 tx=0x7fcdb401af90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d19fd05b5fd953d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.826+0000 7fcda3fff640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:43:48.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.827+0000 7fcda3fff640 1 -- 172.21.15.107:0/907253494 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb400db00 con 0x55a11e285330 2024-05-09T08:43:48.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.827+0000 7fcdce00f740 1 -- 172.21.15.107:0/907253494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdac0b6040 con 0x55a11e285330 2024-05-09T08:43:48.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.827+0000 7fcda3fff640 1 -- 172.21.15.107:0/907253494 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7fcdb401a6b0 con 0x55a11e285330 2024-05-09T08:43:48.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.828+0000 7fcda3fff640 1 -- 172.21.15.107:0/907253494 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcdb401a8b0 con 0x55a11e285330 2024-05-09T08:43:48.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.828+0000 7fcdce00f740 10 client.27542 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:48.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.828+0000 7fcdce00f740 1 -- 172.21.15.107:0/907253494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a11e306fc0 con 0x55a11e285330 2024-05-09T08:43:48.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.829+0000 7fcda3fff640 1 -- 172.21.15.107:0/907253494 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdac0b6040 con 0x55a11e285330 2024-05-09T08:43:48.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.829+0000 7fcda3fff640 1 -- 172.21.15.107:0/907253494 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fcdb4022030 con 0x55a11e285330 2024-05-09T08:43:48.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.829+0000 7fcdce00f740 10 client.27542 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:48.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.829+0000 7fcdce00f740 10 client.27542 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:43:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.829+0000 7fcdce00f740 20 client.27542 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.829+0000 7fcdce00f740 1 --2- 172.21.15.107:0/907253494 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a11e3076d0 0x55a11e309af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.829+0000 7fcdce00f740 4 client.27542 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:43:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.829+0000 7fcdce00f740 1 -- 172.21.15.107:0/907253494 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a11e1f26c0 con 0x55a11e3076d0 2024-05-09T08:43:48.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.829+0000 7fcdc3fff640 1 --2- 172.21.15.107:0/907253494 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a11e3076d0 0x55a11e309af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.831+0000 7fcda3fff640 10 client.27542 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:43:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.831+0000 7fcdc3fff640 1 --2- 172.21.15.107:0/907253494 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a11e3076d0 0x55a11e309af0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e8b2919ea428016 server_cookie=6c7c845f13bc97fc in_seq=0 out_seq=0 2024-05-09T08:43:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.831+0000 7fcda3fff640 1 -- 172.21.15.107:0/907253494 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a11e1f26c0 con 0x55a11e3076d0 2024-05-09T08:43:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.831+0000 7fcda3fff640 10 client.27542 handle_command_reply: tid=0 2024-05-09T08:43:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.831+0000 7fcdce00f740 1 client.27542 shutdown 2024-05-09T08:43:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.832+0000 7fcdce00f740 1 -- 172.21.15.107:0/907253494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285330 msgr2=0x55a11e2984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.832+0000 7fcdce00f740 1 --2- 172.21.15.107:0/907253494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285330 0x55a11e2984f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55a11e2a3a00 tx=0x7fcdb401af90 comp rx=0 tx=0).stop 2024-05-09T08:43:48.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.832+0000 7fcdce00f740 1 -- 172.21.15.107:0/907253494 shutdown_connections 2024-05-09T08:43:48.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.832+0000 7fcdce00f740 1 --2- 172.21.15.107:0/907253494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a11e290320 0x55a11e293a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.832+0000 7fcdce00f740 1 --2- 172.21.15.107:0/907253494 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55a11e3076d0 0x55a11e309af0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.832+0000 7fcdce00f740 1 --2- 172.21.15.107:0/907253494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a11e285c50 0x55a11e298a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.832+0000 7fcdce00f740 1 --2- 172.21.15.107:0/907253494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a11e285330 0x55a11e2984f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.832+0000 7fcdce00f740 1 -- 172.21.15.107:0/907253494 >> 172.21.15.107:0/907253494 conn(0x55a11e281650 msgr2=0x55a11e28b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:48.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.833+0000 7fcda3fff640 0 client.27542 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:43:48.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.833+0000 7fcdce00f740 1 -- 172.21.15.107:0/907253494 shutdown_connections 2024-05-09T08:43:48.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.833+0000 7fcdce00f740 1 -- 172.21.15.107:0/907253494 wait complete. 2024-05-09T08:43:48.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.833+0000 7fcdce00f740 20 client.27542 trim_cache size 0 max 16384 2024-05-09T08:43:48.834 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:43:48.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.836+0000 7fcdcb22c640 1 -- 172.21.15.107:0/3934723629 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcdb00a2e80 msgr2=0x7fcdb00a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.836+0000 7fcdcb22c640 1 --2- 172.21.15.107:0/3934723629 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcdb00a2e80 0x7fcdb00a5340 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc005fe0 tx=0x7fcdbc045000 comp rx=0 tx=0).stop 2024-05-09T08:43:48.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.836+0000 7fcdcb22c640 1 -- 172.21.15.107:0/3934723629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc411b010 msgr2=0x7fcdc4114cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:48.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.836+0000 7fcdcb22c640 1 --2- 172.21.15.107:0/3934723629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc411b010 0x7fcdc4114cd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcdac00b3e0 tx=0x7fcdac00b8a0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.836+0000 7fcdcb22c640 1 -- 172.21.15.107:0/3934723629 shutdown_connections 2024-05-09T08:43:48.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.836+0000 7fcdcb22c640 1 --2- 172.21.15.107:0/3934723629 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcdb00a2e80 0x7fcdb00a5340 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.836+0000 7fcdcb22c640 1 --2- 172.21.15.107:0/3934723629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc411b010 0x7fcdc4114cd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.836+0000 7fcdcb22c640 1 --2- 172.21.15.107:0/3934723629 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcdc4110080 0x7fcdc4114790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.836+0000 7fcdcb22c640 1 --2- 172.21.15.107:0/3934723629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdc410e0b0 0x7fcdc411a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:48.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.836+0000 7fcdcb22c640 1 -- 172.21.15.107:0/3934723629 >> 172.21.15.107:0/3934723629 conn(0x7fcdc406c940 msgr2=0x7fcdc411c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:48.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.836+0000 7fcdcb22c640 1 -- 172.21.15.107:0/3934723629 shutdown_connections 2024-05-09T08:43:48.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:48.836+0000 7fcdcb22c640 1 -- 172.21.15.107:0/3934723629 wait complete. 2024-05-09T08:43:48.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:48 smithi157 ceph-mon[33614]: pgmap v323: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 12 MiB/s wr, 3.04k op/s 2024-05-09T08:43:48.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:48 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3898857003' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:43:48.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:43:48.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:48 smithi110 ceph-mon[33622]: pgmap v323: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 12 MiB/s wr, 3.04k op/s 2024-05-09T08:43:48.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:48 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3898857003' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:43:49.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.000+0000 7ffbd1d09640 1 -- 172.21.15.107:0/3985166228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc10bdd0 msgr2=0x7ffbcc10e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:49.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.000+0000 7ffbd1d09640 1 --2- 172.21.15.107:0/3985166228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc10bdd0 0x7ffbcc10e290 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4009ca0 tx=0x7ffbc402a8c0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.001+0000 7ffbd1d09640 1 -- 172.21.15.107:0/3985166228 shutdown_connections 2024-05-09T08:43:49.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.001+0000 7ffbd1d09640 1 --2- 172.21.15.107:0/3985166228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc10bdd0 0x7ffbcc10e290 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.001+0000 7ffbd1d09640 1 --2- 172.21.15.107:0/3985166228 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffbcc100fb0 0x7ffbcc10b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.001+0000 7ffbd1d09640 1 --2- 172.21.15.107:0/3985166228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbcc100600 0x7ffbcc1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.001+0000 7ffbd1d09640 1 -- 172.21.15.107:0/3985166228 >> 172.21.15.107:0/3985166228 conn(0x7ffbcc075a30 msgr2=0x7ffbcc075e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:49.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.001+0000 7ffbd1d09640 1 -- 172.21.15.107:0/3985166228 shutdown_connections 2024-05-09T08:43:49.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.001+0000 7ffbd1d09640 1 -- 172.21.15.107:0/3985166228 wait complete. 2024-05-09T08:43:49.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.001+0000 7ffbd1d09640 1 Processor -- start 2024-05-09T08:43:49.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.001+0000 7ffbd1d09640 1 -- start start 2024-05-09T08:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbd1d09640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbcc100600 0x7ffbcc197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbd1d09640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc10bdd0 0x7ffbcc198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbd1d09640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffbcc19c8a0 0x7ffbcc19cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbd1d09640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc111ac0 con 0x7ffbcc10bdd0 2024-05-09T08:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbd1d09640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc111940 con 0x7ffbcc100600 2024-05-09T08:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbd1d09640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc111c40 con 0x7ffbcc19c8a0 2024-05-09T08:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbcaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc10bdd0 0x7ffbcc198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbcb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbcc100600 0x7ffbcc197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbcb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbcc100600 0x7ffbcc197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34564/0 (socket says 172.21.15.107:34564) 2024-05-09T08:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbcaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc10bdd0 0x7ffbcc198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57788/0 (socket says 172.21.15.107:57788) 2024-05-09T08:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbcb7fe640 1 -- 172.21.15.107:0/1412393543 learned_addr learned my addr 172.21.15.107:0/1412393543 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbcbfff640 1 --2- 172.21.15.107:0/1412393543 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffbcc19c8a0 0x7ffbcc19cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbcb7fe640 1 -- 172.21.15.107:0/1412393543 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffbcc19c8a0 msgr2=0x7ffbcc19cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbcb7fe640 1 --2- 172.21.15.107:0/1412393543 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffbcc19c8a0 0x7ffbcc19cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbcb7fe640 1 -- 172.21.15.107:0/1412393543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc10bdd0 msgr2=0x7ffbcc198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbcb7fe640 1 --2- 172.21.15.107:0/1412393543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc10bdd0 0x7ffbcc198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbcb7fe640 1 -- 172.21.15.107:0/1412393543 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbcc19d390 con 0x7ffbcc100600 2024-05-09T08:43:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbcaffd640 1 --2- 172.21.15.107:0/1412393543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc10bdd0 0x7ffbcc198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbcb7fe640 1 --2- 172.21.15.107:0/1412393543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbcc100600 0x7ffbcc197fe0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc00d950 tx=0x7ffbbc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbc8ff9640 1 -- 172.21.15.107:0/1412393543 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc017030 con 0x7ffbcc100600 2024-05-09T08:43:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbc8ff9640 1 -- 172.21.15.107:0/1412393543 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffbbc004020 con 0x7ffbcc100600 2024-05-09T08:43:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbd1d09640 1 -- 172.21.15.107:0/1412393543 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbcc19d680 con 0x7ffbcc100600 2024-05-09T08:43:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.002+0000 7ffbc8ff9640 1 -- 172.21.15.107:0/1412393543 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc0047a0 con 0x7ffbcc100600 2024-05-09T08:43:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.003+0000 7ffbd1d09640 1 -- 172.21.15.107:0/1412393543 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbcc1a4e70 con 0x7ffbcc100600 2024-05-09T08:43:49.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.005+0000 7ffbc8ff9640 1 -- 172.21.15.107:0/1412393543 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ffbbc0041d0 con 0x7ffbcc100600 2024-05-09T08:43:49.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.008+0000 7ffbc8ff9640 1 --2- 172.21.15.107:0/1412393543 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffbb40a2d80 0x7ffbb40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.009+0000 7ffbc8ff9640 1 -- 172.21.15.107:0/1412393543 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7ffbbc0cf2f0 con 0x7ffbcc100600 2024-05-09T08:43:49.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.009+0000 7ffbd4a14740 1 Processor -- start 2024-05-09T08:43:49.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.009+0000 7ffbd4a14740 1 -- start start 2024-05-09T08:43:49.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.009+0000 7ffbd4a14740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952c1fe0 0x557a952c23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.009+0000 7ffbd4a14740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c2900 0x557a952d59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.009+0000 7ffbd4a14740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952d5f00 0x557a952d82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.010+0000 7ffbcaffd640 1 --2- 172.21.15.107:0/1412393543 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffbb40a2d80 0x7ffbb40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.010+0000 7ffbcb7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952c1fe0 0x557a952c23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.010+0000 7ffbcb7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952c1fe0 0x557a952c23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47874/0 (socket says 172.21.15.107:47874) 2024-05-09T08:43:49.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.010+0000 7ffbcb7fe640 1 -- 172.21.15.107:0/4102379878 learned_addr learned my addr 172.21.15.107:0/4102379878 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:49.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.010+0000 7ffbd4a14740 1 -- 172.21.15.107:0/4102379878 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557a95227c40 con 0x557a952d5f00 2024-05-09T08:43:49.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.010+0000 7ffbcbfff640 1 --2- 172.21.15.107:0/4102379878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952d5f00 0x557a952d82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.010+0000 7ffbd4a14740 1 -- 172.21.15.107:0/4102379878 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557a951cf580 con 0x557a952c2900 2024-05-09T08:43:49.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.010+0000 7ffbd4a14740 1 -- 172.21.15.107:0/4102379878 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x557a95223c00 con 0x557a952c1fe0 2024-05-09T08:43:49.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.010+0000 7ffbcaffd640 1 --2- 172.21.15.107:0/4102379878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c2900 0x557a952d59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.011+0000 7ffbcaffd640 1 -- 172.21.15.107:0/4102379878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952c1fe0 msgr2=0x557a952c23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:49.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.011+0000 7ffbcaffd640 1 --2- 172.21.15.107:0/4102379878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952c1fe0 0x557a952c23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.011+0000 7ffbcaffd640 1 -- 172.21.15.107:0/4102379878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952d5f00 msgr2=0x557a952d82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:49.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.011+0000 7ffbcaffd640 1 --2- 172.21.15.107:0/4102379878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952d5f00 0x557a952d82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.011+0000 7ffbcaffd640 1 -- 172.21.15.107:0/4102379878 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a94f3b3f0 con 0x557a952c2900 2024-05-09T08:43:49.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.011+0000 7ffbcaffd640 1 --2- 172.21.15.107:0/1412393543 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffbb40a2d80 0x7ffbb40a5240 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc10da90 tx=0x7ffbc0009c10 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:49.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.011+0000 7ffbcaffd640 1 --2- 172.21.15.107:0/4102379878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c2900 0x557a952d59c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc077160 tx=0x7ffbc001d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd4664cc3bf097f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:49.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.011+0000 7ffbb1ffb640 1 -- 172.21.15.107:0/4102379878 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0025070 con 0x557a952c2900 2024-05-09T08:43:49.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.011+0000 7ffbb1ffb640 1 -- 172.21.15.107:0/4102379878 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffbc0013170 con 0x557a952c2900 2024-05-09T08:43:49.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.011+0000 7ffbb1ffb640 1 -- 172.21.15.107:0/4102379878 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc00035f0 con 0x557a952c2900 2024-05-09T08:43:49.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.013+0000 7ffbd4a14740 1 -- 172.21.15.107:0/4102379878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c2900 msgr2=0x557a952d59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:49.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.013+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/4102379878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c2900 0x557a952d59c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc077160 tx=0x7ffbc001d000 comp rx=0 tx=0).stop 2024-05-09T08:43:49.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.013+0000 7ffbd4a14740 1 -- 172.21.15.107:0/4102379878 shutdown_connections 2024-05-09T08:43:49.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.013+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/4102379878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952d5f00 0x557a952d82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.013+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/4102379878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c2900 0x557a952d59c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.013+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/4102379878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952c1fe0 0x557a952c23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.013+0000 7ffbd4a14740 1 -- 172.21.15.107:0/4102379878 >> 172.21.15.107:0/4102379878 conn(0x557a952be650 msgr2=0x557a952bea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:49.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.013+0000 7ffbd4a14740 1 -- 172.21.15.107:0/4102379878 shutdown_connections 2024-05-09T08:43:49.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.013+0000 7ffbd4a14740 1 -- 172.21.15.107:0/4102379878 wait complete. 2024-05-09T08:43:49.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.014+0000 7ffbd4a14740 1 -- start start 2024-05-09T08:43:49.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.014+0000 7ffbd4a14740 1 Processor -- start 2024-05-09T08:43:49.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.015+0000 7ffbd4a14740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952cd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.015+0000 7ffbd4a14740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952c2900 0x557a952cd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.015+0000 7ffbcb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.015+0000 7ffbcb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952cd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34586/0 (socket says 172.21.15.107:34586) 2024-05-09T08:43:49.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.015+0000 7ffbcb7fe640 1 -- 172.21.15.107:0/3906337380 learned_addr learned my addr 172.21.15.107:0/3906337380 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.015+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/3906337380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952d5f00 0x557a952cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.016+0000 7ffbcaffd640 1 --2- 172.21.15.107:0/3906337380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952c2900 0x557a952cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.016+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3906337380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557a952c9250 con 0x557a952c2900 2024-05-09T08:43:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.016+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3906337380 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557a951e2c40 con 0x557a952c1fe0 2024-05-09T08:43:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.016+0000 7ffbcbfff640 1 --2- 172.21.15.107:0/3906337380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952d5f00 0x557a952cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.016+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3906337380 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x557a952c93d0 con 0x557a952d5f00 2024-05-09T08:43:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.016+0000 7ffbd4a14740 10 client.0 _ll_register_callbacks cb 0x557a94f3abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.016+0000 7ffbcb7fe640 1 -- 172.21.15.107:0/3906337380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952d5f00 msgr2=0x557a952cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.016+0000 7ffbcb7fe640 1 --2- 172.21.15.107:0/3906337380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952d5f00 0x557a952cdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.016+0000 7ffbcb7fe640 1 -- 172.21.15.107:0/3906337380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952c2900 msgr2=0x557a952cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.016+0000 7ffbcb7fe640 1 --2- 172.21.15.107:0/3906337380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952c2900 0x557a952cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.016+0000 7ffbcb7fe640 1 -- 172.21.15.107:0/3906337380 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a950057d0 con 0x557a952c1fe0 2024-05-09T08:43:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.017+0000 7ffbcb7fe640 1 --2- 172.21.15.107:0/3906337380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952cd050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc09b0a0 tx=0x7ffbbc00d8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12f9766da3b69944 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.017+0000 7ffbaaffd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:43:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.017+0000 7ffbaaffd640 1 -- 172.21.15.107:0/3906337380 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc0cca50 con 0x557a952c1fe0 2024-05-09T08:43:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.017+0000 7ffbaaffd640 1 -- 172.21.15.107:0/3906337380 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7ffbbc09a7f0 con 0x557a952c1fe0 2024-05-09T08:43:49.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.017+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3906337380 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbc0013170 con 0x557a952c1fe0 2024-05-09T08:43:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.018+0000 7ffbaaffd640 1 -- 172.21.15.107:0/3906337380 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffbbc0a4010 con 0x557a952c1fe0 2024-05-09T08:43:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.018+0000 7ffbd4a14740 10 client.27554 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.018+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3906337380 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=27}) -- 0x557a952dbed0 con 0x557a952c1fe0 2024-05-09T08:43:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.018+0000 7ffbaaffd640 1 -- 172.21.15.107:0/3906337380 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ffbbc0b4200 con 0x557a952c1fe0 2024-05-09T08:43:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.018+0000 7ffbd4a14740 10 client.27554 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.018+0000 7ffbd4a14740 10 client.27554 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.018+0000 7ffbd4a14740 20 client.27554 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.018+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/3906337380 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557a952dd2d0 0x557a952df760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.018+0000 7ffbd4a14740 4 client.27554 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.018+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3906337380 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ffbc0013170 con 0x557a952dd2d0 2024-05-09T08:43:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.019+0000 7ffbcaffd640 1 --2- 172.21.15.107:0/3906337380 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557a952dd2d0 0x557a952df760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.020+0000 7ffbaaffd640 10 client.27554 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.020+0000 7ffbcaffd640 1 --2- 172.21.15.107:0/3906337380 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557a952dd2d0 0x557a952df760 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=252cd8c131e21438 server_cookie=9cc0488734b4ba00 in_seq=0 out_seq=0 2024-05-09T08:43:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.021+0000 7ffbaaffd640 1 -- 172.21.15.107:0/3906337380 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7ffbc0013170 con 0x557a952dd2d0 2024-05-09T08:43:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.021+0000 7ffbaaffd640 10 client.27554 handle_command_reply: tid=0 2024-05-09T08:43:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.021+0000 7ffbd4a14740 1 client.27554 shutdown 2024-05-09T08:43:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.021+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3906337380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 msgr2=0x557a952cd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.021+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/3906337380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952cd050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc09b0a0 tx=0x7ffbbc00d8d0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.022+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3906337380 shutdown_connections 2024-05-09T08:43:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.022+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/3906337380 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557a952dd2d0 0x557a952df760 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.022+0000 7ffbaaffd640 0 client.27554 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.022+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/3906337380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952d5f00 0x557a952cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.022+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/3906337380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952c2900 0x557a952cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.022+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/3906337380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952cd050 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.022+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3906337380 >> 172.21.15.107:0/3906337380 conn(0x557a952be650 msgr2=0x557a952c4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.022+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3906337380 shutdown_connections 2024-05-09T08:43:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.022+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3906337380 wait complete. 2024-05-09T08:43:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.022+0000 7ffbd4a14740 20 client.27554 trim_cache size 0 max 16384 2024-05-09T08:43:49.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.045+0000 7ffbd4a14740 1 Processor -- start 2024-05-09T08:43:49.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.046+0000 7ffbd4a14740 1 -- start start 2024-05-09T08:43:49.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.046+0000 7ffbd4a14740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.046+0000 7ffbd4a14740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952c2900 0x557a952bd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.046+0000 7ffbd4a14740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952d5f00 0x557a952e5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.046+0000 7ffbd4a14740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1af430 con 0x557a952d5f00 2024-05-09T08:43:49.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.046+0000 7ffbd4a14740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1acd00 con 0x557a952c1fe0 2024-05-09T08:43:49.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.046+0000 7ffbd4a14740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1aab70 con 0x557a952c2900 2024-05-09T08:43:49.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.046+0000 7ffbcb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.046+0000 7ffbcb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952cc1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34588/0 (socket says 172.21.15.107:34588) 2024-05-09T08:43:49.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.046+0000 7ffbcb7fe640 1 -- 172.21.15.107:0/1316666110 learned_addr learned my addr 172.21.15.107:0/1316666110 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:49.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.046+0000 7ffbcaffd640 1 --2- 172.21.15.107:0/1316666110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952c2900 0x557a952bd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.046+0000 7ffbcbfff640 1 --2- 172.21.15.107:0/1316666110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952d5f00 0x557a952e5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.047+0000 7ffbcaffd640 1 -- 172.21.15.107:0/1316666110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 msgr2=0x557a952cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:49.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.047+0000 7ffbcaffd640 1 --2- 172.21.15.107:0/1316666110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.047+0000 7ffbcaffd640 1 -- 172.21.15.107:0/1316666110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952d5f00 msgr2=0x557a952e5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:49.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.047+0000 7ffbcaffd640 1 --2- 172.21.15.107:0/1316666110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952d5f00 0x557a952e5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.047+0000 7ffbcaffd640 1 -- 172.21.15.107:0/1316666110 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a952d26e0 con 0x557a952c2900 2024-05-09T08:43:49.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.047+0000 7ffbcb7fe640 1 --2- 172.21.15.107:0/1316666110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:49.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.048+0000 7ffbcaffd640 1 --2- 172.21.15.107:0/1316666110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952c2900 0x557a952bd630 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x557a952cc8b0 tx=0x7ffbc0013ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ff8a5249a621e8e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.050+0000 7ffbb17fa640 1 -- 172.21.15.107:0/1316666110 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0025040 con 0x557a952c2900 2024-05-09T08:43:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.050+0000 7ffbb17fa640 1 -- 172.21.15.107:0/1316666110 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffbc0013170 con 0x557a952c2900 2024-05-09T08:43:49.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.050+0000 7ffbb17fa640 1 -- 172.21.15.107:0/1316666110 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc00035f0 con 0x557a952c2900 2024-05-09T08:43:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.051+0000 7ffbd4a14740 1 -- 172.21.15.107:0/1316666110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952c2900 msgr2=0x557a952bd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.051+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/1316666110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952c2900 0x557a952bd630 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x557a952cc8b0 tx=0x7ffbc0013ee0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.051+0000 7ffbd4a14740 1 -- 172.21.15.107:0/1316666110 shutdown_connections 2024-05-09T08:43:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.051+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/1316666110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952d5f00 0x557a952e5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.051+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/1316666110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952c2900 0x557a952bd630 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.051+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/1316666110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.051+0000 7ffbd4a14740 1 -- 172.21.15.107:0/1316666110 >> 172.21.15.107:0/1316666110 conn(0x557a952be650 msgr2=0x557a952bcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.052+0000 7ffbd4a14740 1 -- 172.21.15.107:0/1316666110 shutdown_connections 2024-05-09T08:43:49.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.052+0000 7ffbd4a14740 1 -- 172.21.15.107:0/1316666110 wait complete. 2024-05-09T08:43:49.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.052+0000 7ffbd4a14740 1 -- start start 2024-05-09T08:43:49.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.053+0000 7ffbd4a14740 1 Processor -- start 2024-05-09T08:43:49.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.053+0000 7ffbd4a14740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952ed5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.053+0000 7ffbd4a14740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952c2900 0x557a952edb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.053+0000 7ffbd4a14740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952d5f00 0x557a952ee070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.053+0000 7ffbcb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952ed5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.053+0000 7ffbcb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952ed5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34590/0 (socket says 172.21.15.107:34590) 2024-05-09T08:43:49.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.053+0000 7ffbcb7fe640 1 -- 172.21.15.107:0/3680309887 learned_addr learned my addr 172.21.15.107:0/3680309887 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:49.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.053+0000 7ffbd4a14740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1ab1c0 con 0x557a952c2900 2024-05-09T08:43:49.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.053+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3680309887 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1ac630 con 0x557a952c1fe0 2024-05-09T08:43:49.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.053+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3680309887 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1ab890 con 0x557a952d5f00 2024-05-09T08:43:49.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.053+0000 7ffbcbfff640 1 --2- 172.21.15.107:0/3680309887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952d5f00 0x557a952ee070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.054+0000 7ffbd4a14740 10 client.0 _ll_register_callbacks cb 0x7ffbbc0cc570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:49.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.054+0000 7ffbcbfff640 1 -- 172.21.15.107:0/3680309887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 msgr2=0x557a952ed5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:49.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.054+0000 7ffbcbfff640 1 --2- 172.21.15.107:0/3680309887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952ed5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.054+0000 7ffbcbfff640 1 -- 172.21.15.107:0/3680309887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952c2900 msgr2=0x557a952edb30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:43:49.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.054+0000 7ffbcbfff640 1 --2- 172.21.15.107:0/3680309887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952c2900 0x557a952edb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.054+0000 7ffbcbfff640 1 -- 172.21.15.107:0/3680309887 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a953543a0 con 0x557a952d5f00 2024-05-09T08:43:49.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.055+0000 7ffbcb7fe640 1 --2- 172.21.15.107:0/3680309887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952ed5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:43:49.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.056+0000 7ffbcbfff640 1 --2- 172.21.15.107:0/3680309887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952d5f00 0x557a952ee070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4031040 tx=0x7ffbc40317c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8215181e6575f92 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:49.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.056+0000 7ffbaaffd640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:43:49.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.058+0000 7ffbaaffd640 1 -- 172.21.15.107:0/3680309887 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4003d80 con 0x557a952d5f00 2024-05-09T08:43:49.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.058+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3680309887 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbc0013170 con 0x557a952d5f00 2024-05-09T08:43:49.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.058+0000 7ffbaaffd640 1 -- 172.21.15.107:0/3680309887 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7ffbc4004040 con 0x557a952d5f00 2024-05-09T08:43:49.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.060+0000 7ffbaaffd640 1 -- 172.21.15.107:0/3680309887 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffbc4077050 con 0x557a952d5f00 2024-05-09T08:43:49.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.061+0000 7ffbd4a14740 10 client.27781 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:49.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.061+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3680309887 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557a9533b830 con 0x557a952d5f00 2024-05-09T08:43:49.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.062+0000 7ffbaaffd640 1 -- 172.21.15.107:0/3680309887 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffbc0013170 con 0x557a952d5f00 2024-05-09T08:43:49.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.062+0000 7ffbaaffd640 1 -- 172.21.15.107:0/3680309887 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7ffbc4072030 con 0x557a952d5f00 2024-05-09T08:43:49.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.062+0000 7ffbd4a14740 10 client.27781 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:49.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.062+0000 7ffbd4a14740 10 client.27781 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:49.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.062+0000 7ffbd4a14740 20 client.27781 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:49.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.062+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/3680309887 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557a952dd2d0 0x557a9533dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:49.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.062+0000 7ffbd4a14740 4 client.27781 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:43:49.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.062+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3680309887 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557a95235370 con 0x557a952dd2d0 2024-05-09T08:43:49.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.064+0000 7ffbcb7fe640 1 --2- 172.21.15.107:0/3680309887 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557a952dd2d0 0x557a9533dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:49.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.065+0000 7ffbcb7fe640 1 --2- 172.21.15.107:0/3680309887 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557a952dd2d0 0x557a9533dfb0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9479c7f33c87cabb server_cookie=18906cf901869b5 in_seq=0 out_seq=0 2024-05-09T08:43:49.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.065+0000 7ffbaaffd640 10 client.27781 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:49.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.066+0000 7ffbaaffd640 1 -- 172.21.15.107:0/3680309887 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557a95235370 con 0x557a952dd2d0 2024-05-09T08:43:49.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.066+0000 7ffbaaffd640 10 client.27781 handle_command_reply: tid=0 2024-05-09T08:43:49.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.066+0000 7ffbd4a14740 1 client.27781 shutdown 2024-05-09T08:43:49.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.066+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3680309887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952d5f00 msgr2=0x557a952ee070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:49.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.066+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/3680309887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952d5f00 0x557a952ee070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4031040 tx=0x7ffbc40317c0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.067+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3680309887 shutdown_connections 2024-05-09T08:43:49.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.067+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/3680309887 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557a952dd2d0 0x557a9533dfb0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.067+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/3680309887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557a952d5f00 0x557a952ee070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.067+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/3680309887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a952c2900 0x557a952edb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.067+0000 7ffbd4a14740 1 --2- 172.21.15.107:0/3680309887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557a952c1fe0 0x557a952ed5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.067+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3680309887 >> 172.21.15.107:0/3680309887 conn(0x557a952be650 msgr2=0x557a952cc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:49.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.067+0000 7ffbaaffd640 0 client.27781 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:49.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.067+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3680309887 shutdown_connections 2024-05-09T08:43:49.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.067+0000 7ffbd4a14740 1 -- 172.21.15.107:0/3680309887 wait complete. 2024-05-09T08:43:49.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.067+0000 7ffbd4a14740 20 client.27781 trim_cache size 0 max 16384 2024-05-09T08:43:49.067 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:43:49.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.069+0000 7ffbd1d09640 1 -- 172.21.15.107:0/1412393543 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffbb40a2d80 msgr2=0x7ffbb40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:49.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.069+0000 7ffbd1d09640 1 --2- 172.21.15.107:0/1412393543 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffbb40a2d80 0x7ffbb40a5240 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc10da90 tx=0x7ffbc0009c10 comp rx=0 tx=0).stop 2024-05-09T08:43:49.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.069+0000 7ffbd1d09640 1 -- 172.21.15.107:0/1412393543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbcc100600 msgr2=0x7ffbcc197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:49.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.069+0000 7ffbd1d09640 1 --2- 172.21.15.107:0/1412393543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbcc100600 0x7ffbcc197fe0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc00d950 tx=0x7ffbbc00de10 comp rx=0 tx=0).stop 2024-05-09T08:43:49.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.070+0000 7ffbd1d09640 1 -- 172.21.15.107:0/1412393543 shutdown_connections 2024-05-09T08:43:49.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.070+0000 7ffbd1d09640 1 --2- 172.21.15.107:0/1412393543 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7ffbb40a2d80 0x7ffbb40a5240 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.070+0000 7ffbd1d09640 1 --2- 172.21.15.107:0/1412393543 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffbcc19c8a0 0x7ffbcc19cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.070+0000 7ffbd1d09640 1 --2- 172.21.15.107:0/1412393543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbcc10bdd0 0x7ffbcc198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.071+0000 7ffbd1d09640 1 --2- 172.21.15.107:0/1412393543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbcc100600 0x7ffbcc197fe0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:49.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.071+0000 7ffbd1d09640 1 -- 172.21.15.107:0/1412393543 >> 172.21.15.107:0/1412393543 conn(0x7ffbcc075a30 msgr2=0x7ffbcc0fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:49.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.072+0000 7ffbd1d09640 1 -- 172.21.15.107:0/1412393543 shutdown_connections 2024-05-09T08:43:49.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:49.073+0000 7ffbd1d09640 1 -- 172.21.15.107:0/1412393543 wait complete. 2024-05-09T08:43:50.090 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:43:50.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:43:50.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:43:50.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.242+0000 7faf8434f640 1 -- 172.21.15.107:0/2592840893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf7c106710 msgr2=0x7faf7c106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.242+0000 7faf8434f640 1 --2- 172.21.15.107:0/2592840893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf7c106710 0x7faf7c106af0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7faf78009930 tx=0x7faf7802a420 comp rx=0 tx=0).stop 2024-05-09T08:43:50.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.243+0000 7faf8434f640 1 -- 172.21.15.107:0/2592840893 shutdown_connections 2024-05-09T08:43:50.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.243+0000 7faf8434f640 1 --2- 172.21.15.107:0/2592840893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf7c1010c0 0x7faf7c10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.243+0000 7faf8434f640 1 --2- 172.21.15.107:0/2592840893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf7c100700 0x7faf7c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.243+0000 7faf8434f640 1 --2- 172.21.15.107:0/2592840893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf7c106710 0x7faf7c106af0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.243+0000 7faf8434f640 1 -- 172.21.15.107:0/2592840893 >> 172.21.15.107:0/2592840893 conn(0x7faf7c077830 msgr2=0x7faf7c077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:50.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.243+0000 7faf8434f640 1 -- 172.21.15.107:0/2592840893 shutdown_connections 2024-05-09T08:43:50.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.243+0000 7faf8434f640 1 -- 172.21.15.107:0/2592840893 wait complete. 2024-05-09T08:43:50.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.243+0000 7faf8434f640 1 Processor -- start 2024-05-09T08:43:50.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.244+0000 7faf8434f640 1 -- start start 2024-05-09T08:43:50.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.244+0000 7faf8434f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf7c100700 0x7faf7c10e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.244+0000 7faf8434f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf7c1010c0 0x7faf7c10e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf8434f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf7c106710 0x7faf7c10eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf8434f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c111ad0 con 0x7faf7c100700 2024-05-09T08:43:50.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf8434f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c111950 con 0x7faf7c106710 2024-05-09T08:43:50.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf8434f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c111c50 con 0x7faf7c1010c0 2024-05-09T08:43:50.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf820eb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf7c100700 0x7faf7c10e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf818ea640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf7c1010c0 0x7faf7c10e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf818ea640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf7c1010c0 0x7faf7c10e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47924/0 (socket says 172.21.15.107:47924) 2024-05-09T08:43:50.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf820eb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf7c100700 0x7faf7c10e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57816/0 (socket says 172.21.15.107:57816) 2024-05-09T08:43:50.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf818ea640 1 -- 172.21.15.107:0/2760030762 learned_addr learned my addr 172.21.15.107:0/2760030762 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:50.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf828ec640 1 --2- 172.21.15.107:0/2760030762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf7c106710 0x7faf7c10eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf818ea640 1 -- 172.21.15.107:0/2760030762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf7c106710 msgr2=0x7faf7c10eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf818ea640 1 --2- 172.21.15.107:0/2760030762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf7c106710 0x7faf7c10eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf818ea640 1 -- 172.21.15.107:0/2760030762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf7c100700 msgr2=0x7faf7c10e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf818ea640 1 --2- 172.21.15.107:0/2760030762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf7c100700 0x7faf7c10e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf818ea640 1 -- 172.21.15.107:0/2760030762 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf7c1a64f0 con 0x7faf7c1010c0 2024-05-09T08:43:50.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf820eb640 1 --2- 172.21.15.107:0/2760030762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf7c100700 0x7faf7c10e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:50.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf818ea640 1 --2- 172.21.15.107:0/2760030762 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf7c1010c0 0x7faf7c10e5a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00e9f0 tx=0x7faf6c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:50.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf737fe640 1 -- 172.21.15.107:0/2760030762 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c00cd80 con 0x7faf7c1010c0 2024-05-09T08:43:50.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf737fe640 1 -- 172.21.15.107:0/2760030762 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf6c0054d0 con 0x7faf7c1010c0 2024-05-09T08:43:50.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf8434f640 1 -- 172.21.15.107:0/2760030762 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf7c1a67e0 con 0x7faf7c1010c0 2024-05-09T08:43:50.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.245+0000 7faf737fe640 1 -- 172.21.15.107:0/2760030762 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c010490 con 0x7faf7c1010c0 2024-05-09T08:43:50.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.246+0000 7faf8434f640 1 -- 172.21.15.107:0/2760030762 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf7c1a6ce0 con 0x7faf7c1010c0 2024-05-09T08:43:50.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.248+0000 7faf737fe640 1 -- 172.21.15.107:0/2760030762 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7faf6c005020 con 0x7faf7c1010c0 2024-05-09T08:43:50.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.248+0000 7faf8705c740 1 Processor -- start 2024-05-09T08:43:50.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf8705c740 1 -- start start 2024-05-09T08:43:50.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf737fe640 1 --2- 172.21.15.107:0/2760030762 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf680a2d30 0x7faf680a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf8705c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 0x5560063bf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf828ec640 1 --2- 172.21.15.107:0/2760030762 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf680a2d30 0x7faf680a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf8705c740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063bf9c0 0x5560063bfe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf8705c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 0x5560063ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf737fe640 1 -- 172.21.15.107:0/2760030762 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7faf6c0cfbc0 con 0x7faf7c1010c0 2024-05-09T08:43:50.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf8705c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556006324990 con 0x5560063c10b0 2024-05-09T08:43:50.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf8705c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5560062cc5b0 con 0x5560063ca740 2024-05-09T08:43:50.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf8705c740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x556006320950 con 0x5560063bf9c0 2024-05-09T08:43:50.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf820eb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 0x5560063bf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf820eb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 0x5560063bf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57818/0 (socket says 172.21.15.107:57818) 2024-05-09T08:43:50.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf820eb640 1 -- 172.21.15.107:0/2616283075 learned_addr learned my addr 172.21.15.107:0/2616283075 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:50.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf818ea640 1 --2- 172.21.15.107:0/2616283075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063bf9c0 0x5560063bfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf828ec640 1 --2- 172.21.15.107:0/2616283075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 0x5560063ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf820eb640 1 -- 172.21.15.107:0/2616283075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063bf9c0 msgr2=0x5560063bfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:50.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf820eb640 1 --2- 172.21.15.107:0/2616283075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063bf9c0 0x5560063bfe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf820eb640 1 -- 172.21.15.107:0/2616283075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 msgr2=0x5560063ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:50.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf820eb640 1 --2- 172.21.15.107:0/2616283075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 0x5560063ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf820eb640 1 -- 172.21.15.107:0/2616283075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560061007e0 con 0x5560063c10b0 2024-05-09T08:43:50.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf818ea640 1 --2- 172.21.15.107:0/2616283075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063bf9c0 0x5560063bfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:50.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf828ec640 1 --2- 172.21.15.107:0/2616283075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 0x5560063ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:43:50.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.249+0000 7faf828ec640 1 --2- 172.21.15.107:0/2760030762 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf680a2d30 0x7faf680a51f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faf74005fd0 tx=0x7faf74011b40 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:50.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.250+0000 7faf820eb640 1 --2- 172.21.15.107:0/2616283075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 0x5560063bf3f0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7faf78006010 tx=0x7faf78032eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced4e773bbfe6688 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:50.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.250+0000 7faf70bfe640 1 -- 172.21.15.107:0/2616283075 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf78035040 con 0x5560063c10b0 2024-05-09T08:43:50.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.250+0000 7faf70bfe640 1 -- 172.21.15.107:0/2616283075 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf780317a0 con 0x5560063c10b0 2024-05-09T08:43:50.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.250+0000 7faf70bfe640 1 -- 172.21.15.107:0/2616283075 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf780047a0 con 0x5560063c10b0 2024-05-09T08:43:50.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.250+0000 7faf8705c740 1 -- 172.21.15.107:0/2616283075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 msgr2=0x5560063bf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.250+0000 7faf8705c740 1 --2- 172.21.15.107:0/2616283075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 0x5560063bf3f0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7faf78006010 tx=0x7faf78032eb0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.250+0000 7faf8705c740 1 -- 172.21.15.107:0/2616283075 shutdown_connections 2024-05-09T08:43:50.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.250+0000 7faf8705c740 1 --2- 172.21.15.107:0/2616283075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 0x5560063ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.250+0000 7faf8705c740 1 --2- 172.21.15.107:0/2616283075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063bf9c0 0x5560063bfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.250+0000 7faf8705c740 1 --2- 172.21.15.107:0/2616283075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 0x5560063bf3f0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.250+0000 7faf8705c740 1 -- 172.21.15.107:0/2616283075 >> 172.21.15.107:0/2616283075 conn(0x5560063880a0 msgr2=0x5560063bbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.251+0000 7faf8705c740 1 -- 172.21.15.107:0/2616283075 shutdown_connections 2024-05-09T08:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.251+0000 7faf8705c740 1 -- 172.21.15.107:0/2616283075 wait complete. 2024-05-09T08:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.251+0000 7faf8705c740 1 -- start start 2024-05-09T08:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.251+0000 7faf8705c740 1 Processor -- start 2024-05-09T08:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.251+0000 7faf8705c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063bf9c0 0x5560063c7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.251+0000 7faf8705c740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063c10b0 0x5560063c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.251+0000 7faf8705c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 0x5560063c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.251+0000 7faf8705c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5560063cd570 con 0x5560063bf9c0 2024-05-09T08:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.251+0000 7faf8705c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5560062dfb40 con 0x5560063ca740 2024-05-09T08:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.251+0000 7faf8705c740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5560063cd6f0 con 0x5560063c10b0 2024-05-09T08:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.251+0000 7faf8705c740 10 client.0 _ll_register_callbacks cb 0x5560062d40f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.251+0000 7faf820eb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063bf9c0 0x5560063c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.251+0000 7faf820eb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063bf9c0 0x5560063c7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57820/0 (socket says 172.21.15.107:57820) 2024-05-09T08:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.251+0000 7faf820eb640 1 -- 172.21.15.107:0/2782119882 learned_addr learned my addr 172.21.15.107:0/2782119882 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.251+0000 7faf818ea640 1 --2- 172.21.15.107:0/2782119882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063c10b0 0x5560063c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.251+0000 7faf828ec640 1 --2- 172.21.15.107:0/2782119882 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 0x5560063c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf820eb640 1 -- 172.21.15.107:0/2782119882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063c10b0 msgr2=0x5560063c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf820eb640 1 --2- 172.21.15.107:0/2782119882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063c10b0 0x5560063c7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf820eb640 1 -- 172.21.15.107:0/2782119882 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 msgr2=0x5560063c80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf820eb640 1 --2- 172.21.15.107:0/2782119882 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 0x5560063c80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf820eb640 1 -- 172.21.15.107:0/2782119882 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556006160f70 con 0x5560063bf9c0 2024-05-09T08:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf818ea640 1 --2- 172.21.15.107:0/2782119882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063c10b0 0x5560063c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf828ec640 1 --2- 172.21.15.107:0/2782119882 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 0x5560063c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf820eb640 1 --2- 172.21.15.107:0/2782119882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063bf9c0 0x5560063c7620 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7faf78000c00 tx=0x7faf78009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed54a15ae90bfae2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:43:50.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf65ffb640 1 -- 172.21.15.107:0/2782119882 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7804e070 con 0x5560063bf9c0 2024-05-09T08:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf65ffb640 1 -- 172.21.15.107:0/2782119882 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7faf78046860 con 0x5560063bf9c0 2024-05-09T08:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf8705c740 1 -- 172.21.15.107:0/2782119882 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf780317a0 con 0x5560063bf9c0 2024-05-09T08:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf65ffb640 1 -- 172.21.15.107:0/2782119882 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faf7805a050 con 0x5560063bf9c0 2024-05-09T08:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf8705c740 10 client.18618 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf8705c740 1 -- 172.21.15.107:0/2782119882 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556005ff8170 con 0x5560063bf9c0 2024-05-09T08:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf65ffb640 1 -- 172.21.15.107:0/2782119882 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf780317a0 con 0x5560063bf9c0 2024-05-09T08:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf65ffb640 1 -- 172.21.15.107:0/2782119882 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7faf7803cbe0 con 0x5560063bf9c0 2024-05-09T08:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf8705c740 10 client.18618 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf8705c740 10 client.18618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf8705c740 20 client.18618 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf8705c740 1 --2- 172.21.15.107:0/2782119882 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5560063da850 0x5560063dcc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf8705c740 4 client.18618 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.252+0000 7faf8705c740 1 -- 172.21.15.107:0/2782119882 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560063320c0 con 0x5560063da850 2024-05-09T08:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.253+0000 7faf818ea640 1 --2- 172.21.15.107:0/2782119882 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5560063da850 0x5560063dcc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.253+0000 7faf65ffb640 10 client.18618 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.253+0000 7faf818ea640 1 --2- 172.21.15.107:0/2782119882 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5560063da850 0x5560063dcc40 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c164fc129431abb6 server_cookie=fd937f5ec6d444b4 in_seq=0 out_seq=0 2024-05-09T08:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.254+0000 7faf65ffb640 1 -- 172.21.15.107:0/2782119882 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5560063320c0 con 0x5560063da850 2024-05-09T08:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.254+0000 7faf65ffb640 10 client.18618 handle_command_reply: tid=0 2024-05-09T08:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.254+0000 7faf8705c740 1 client.18618 shutdown 2024-05-09T08:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.254+0000 7faf8705c740 1 -- 172.21.15.107:0/2782119882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063bf9c0 msgr2=0x5560063c7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.254+0000 7faf8705c740 1 --2- 172.21.15.107:0/2782119882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063bf9c0 0x5560063c7620 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7faf78000c00 tx=0x7faf78009590 comp rx=0 tx=0).stop 2024-05-09T08:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.254+0000 7faf8705c740 1 -- 172.21.15.107:0/2782119882 shutdown_connections 2024-05-09T08:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.254+0000 7faf8705c740 1 --2- 172.21.15.107:0/2782119882 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5560063da850 0x5560063dcc40 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.254+0000 7faf8705c740 1 --2- 172.21.15.107:0/2782119882 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 0x5560063c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.254+0000 7faf65ffb640 0 client.18618 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.254+0000 7faf8705c740 1 --2- 172.21.15.107:0/2782119882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063c10b0 0x5560063c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.254+0000 7faf8705c740 1 --2- 172.21.15.107:0/2782119882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063bf9c0 0x5560063c7620 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.254+0000 7faf8705c740 1 -- 172.21.15.107:0/2782119882 >> 172.21.15.107:0/2782119882 conn(0x5560063880a0 msgr2=0x5560063c1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:50.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.254+0000 7faf8705c740 1 -- 172.21.15.107:0/2782119882 shutdown_connections 2024-05-09T08:43:50.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.254+0000 7faf8705c740 1 -- 172.21.15.107:0/2782119882 wait complete. 2024-05-09T08:43:50.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.254+0000 7faf8705c740 20 client.18618 trim_cache size 0 max 16384 2024-05-09T08:43:50.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.278+0000 7faf8705c740 1 Processor -- start 2024-05-09T08:43:50.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf8705c740 1 -- start start 2024-05-09T08:43:50.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf8705c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063bf9c0 0x5560063bae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf8705c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 0x5560063d6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf8705c740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063ca740 0x5560063c48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf8705c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c1b37e0 con 0x5560063c10b0 2024-05-09T08:43:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf8705c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c1b10b0 con 0x5560063bf9c0 2024-05-09T08:43:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf8705c740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c1aef20 con 0x5560063ca740 2024-05-09T08:43:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf818ea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 0x5560063d6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf818ea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 0x5560063d6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57822/0 (socket says 172.21.15.107:57822) 2024-05-09T08:43:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf818ea640 1 -- 172.21.15.107:0/3199187983 learned_addr learned my addr 172.21.15.107:0/3199187983 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf820eb640 1 --2- 172.21.15.107:0/3199187983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063bf9c0 0x5560063bae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf828ec640 1 --2- 172.21.15.107:0/3199187983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063ca740 0x5560063c48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf820eb640 1 -- 172.21.15.107:0/3199187983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063ca740 msgr2=0x5560063c48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf820eb640 1 --2- 172.21.15.107:0/3199187983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063ca740 0x5560063c48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf820eb640 1 -- 172.21.15.107:0/3199187983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 msgr2=0x5560063d6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf820eb640 1 --2- 172.21.15.107:0/3199187983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 0x5560063d6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf820eb640 1 -- 172.21.15.107:0/3199187983 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560063c0c50 con 0x5560063bf9c0 2024-05-09T08:43:50.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf818ea640 1 --2- 172.21.15.107:0/3199187983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 0x5560063d6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:50.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.279+0000 7faf820eb640 1 --2- 172.21.15.107:0/3199187983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063bf9c0 0x5560063bae60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faf780057c0 tx=0x7faf78009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c4794fb4df070c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:50.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.280+0000 7faf67fff640 1 -- 172.21.15.107:0/3199187983 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7804e070 con 0x5560063bf9c0 2024-05-09T08:43:50.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.280+0000 7faf67fff640 1 -- 172.21.15.107:0/3199187983 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf7803bd80 con 0x5560063bf9c0 2024-05-09T08:43:50.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.280+0000 7faf67fff640 1 -- 172.21.15.107:0/3199187983 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf78035040 con 0x5560063bf9c0 2024-05-09T08:43:50.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:50 smithi107 ceph-mon[30122]: pgmap v324: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 8.8 MiB/s wr, 2.24k op/s 2024-05-09T08:43:50.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:50 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:43:50.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:50 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:43:50.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:50 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:43:50.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:50 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:43:50.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:50 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:50 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:50 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:50 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:50 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:50 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:50 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:50 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.280+0000 7faf8705c740 1 -- 172.21.15.107:0/3199187983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063bf9c0 msgr2=0x5560063bae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.280+0000 7faf8705c740 1 --2- 172.21.15.107:0/3199187983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063bf9c0 0x5560063bae60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faf780057c0 tx=0x7faf78009520 comp rx=0 tx=0).stop 2024-05-09T08:43:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.280+0000 7faf8705c740 1 -- 172.21.15.107:0/3199187983 shutdown_connections 2024-05-09T08:43:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.280+0000 7faf8705c740 1 --2- 172.21.15.107:0/3199187983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063ca740 0x5560063c48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.280+0000 7faf8705c740 1 --2- 172.21.15.107:0/3199187983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 0x5560063d6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.280+0000 7faf8705c740 1 --2- 172.21.15.107:0/3199187983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063bf9c0 0x5560063bae60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.280+0000 7faf8705c740 1 -- 172.21.15.107:0/3199187983 >> 172.21.15.107:0/3199187983 conn(0x5560063880a0 msgr2=0x5560063d5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.280+0000 7faf8705c740 1 -- 172.21.15.107:0/3199187983 shutdown_connections 2024-05-09T08:43:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.280+0000 7faf8705c740 1 -- 172.21.15.107:0/3199187983 wait complete. 2024-05-09T08:43:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.281+0000 7faf8705c740 1 -- start start 2024-05-09T08:43:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.281+0000 7faf8705c740 1 Processor -- start 2024-05-09T08:43:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.281+0000 7faf8705c740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063bf9c0 0x5560063ddb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.281+0000 7faf8705c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 0x5560063de040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.281+0000 7faf8705c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 0x55600643fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.281+0000 7faf8705c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c1af570 con 0x5560063c10b0 2024-05-09T08:43:50.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.281+0000 7faf8705c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c1b09e0 con 0x5560063ca740 2024-05-09T08:43:50.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.281+0000 7faf8705c740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c1afc40 con 0x5560063bf9c0 2024-05-09T08:43:50.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.281+0000 7faf8705c740 10 client.0 _ll_register_callbacks cb 0x7faf78040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:50.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.281+0000 7faf820eb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063bf9c0 0x5560063ddb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.281+0000 7faf818ea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 0x5560063de040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.281+0000 7faf828ec640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 0x55600643fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.281+0000 7faf820eb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063bf9c0 0x5560063ddb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47952/0 (socket says 172.21.15.107:47952) 2024-05-09T08:43:50.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.281+0000 7faf820eb640 1 -- 172.21.15.107:0/1728037528 learned_addr learned my addr 172.21.15.107:0/1728037528 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:50.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.281+0000 7faf820eb640 1 -- 172.21.15.107:0/1728037528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 msgr2=0x55600643fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:50.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.281+0000 7faf820eb640 1 --2- 172.21.15.107:0/1728037528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 0x55600643fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf820eb640 1 -- 172.21.15.107:0/1728037528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 msgr2=0x5560063de040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:50.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf820eb640 1 --2- 172.21.15.107:0/1728037528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 0x5560063de040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf820eb640 1 -- 172.21.15.107:0/1728037528 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55600643ff70 con 0x5560063bf9c0 2024-05-09T08:43:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf828ec640 1 --2- 172.21.15.107:0/1728037528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 0x55600643fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf820eb640 1 --2- 172.21.15.107:0/1728037528 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063bf9c0 0x5560063ddb00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faf7803b690 tx=0x7faf78005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9f03015880ba238 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:43:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf65ffb640 1 -- 172.21.15.107:0/1728037528 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7804e070 con 0x5560063bf9c0 2024-05-09T08:43:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf8705c740 1 -- 172.21.15.107:0/1728037528 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf7803bd80 con 0x5560063bf9c0 2024-05-09T08:43:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf65ffb640 1 -- 172.21.15.107:0/1728037528 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7faf7803bad0 con 0x5560063bf9c0 2024-05-09T08:43:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf65ffb640 1 -- 172.21.15.107:0/1728037528 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faf78035040 con 0x5560063bf9c0 2024-05-09T08:43:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf8705c740 10 client.27799 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf8705c740 1 -- 172.21.15.107:0/1728037528 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556006441130 con 0x5560063bf9c0 2024-05-09T08:43:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf65ffb640 1 -- 172.21.15.107:0/1728037528 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf7803bd80 con 0x5560063bf9c0 2024-05-09T08:43:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf65ffb640 1 -- 172.21.15.107:0/1728037528 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7faf780694b0 con 0x5560063bf9c0 2024-05-09T08:43:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf8705c740 10 client.27799 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:50.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf8705c740 10 client.27799 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf8705c740 20 client.27799 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf8705c740 1 --2- 172.21.15.107:0/1728037528 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556006441630 0x556006443a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf8705c740 4 client.27799 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:43:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.282+0000 7faf8705c740 1 -- 172.21.15.107:0/1728037528 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55600632c410 con 0x556006441630 2024-05-09T08:43:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.283+0000 7faf818ea640 1 --2- 172.21.15.107:0/1728037528 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556006441630 0x556006443a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.283+0000 7faf65ffb640 10 client.27799 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.283+0000 7faf818ea640 1 --2- 172.21.15.107:0/1728037528 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556006441630 0x556006443a50 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd3143fd060b8119 server_cookie=c108108eecd35a1 in_seq=0 out_seq=0 2024-05-09T08:43:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.284+0000 7faf65ffb640 1 -- 172.21.15.107:0/1728037528 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55600632c410 con 0x556006441630 2024-05-09T08:43:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.284+0000 7faf65ffb640 10 client.27799 handle_command_reply: tid=0 2024-05-09T08:43:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.284+0000 7faf8705c740 1 client.27799 shutdown 2024-05-09T08:43:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.284+0000 7faf8705c740 1 -- 172.21.15.107:0/1728037528 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063bf9c0 msgr2=0x5560063ddb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.284+0000 7faf8705c740 1 --2- 172.21.15.107:0/1728037528 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063bf9c0 0x5560063ddb00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faf7803b690 tx=0x7faf78005e50 comp rx=0 tx=0).stop 2024-05-09T08:43:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.284+0000 7faf8705c740 1 -- 172.21.15.107:0/1728037528 shutdown_connections 2024-05-09T08:43:50.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.284+0000 7faf8705c740 1 --2- 172.21.15.107:0/1728037528 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x556006441630 0x556006443a50 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.284+0000 7faf8705c740 1 --2- 172.21.15.107:0/1728037528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5560063ca740 0x55600643fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.284+0000 7faf65ffb640 0 client.27799 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:50.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.284+0000 7faf8705c740 1 --2- 172.21.15.107:0/1728037528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560063c10b0 0x5560063de040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.284+0000 7faf8705c740 1 --2- 172.21.15.107:0/1728037528 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5560063bf9c0 0x5560063ddb00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.284+0000 7faf8705c740 1 -- 172.21.15.107:0/1728037528 >> 172.21.15.107:0/1728037528 conn(0x5560063880a0 msgr2=0x5560063c5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:50.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.284+0000 7faf8705c740 1 -- 172.21.15.107:0/1728037528 shutdown_connections 2024-05-09T08:43:50.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.284+0000 7faf8705c740 1 -- 172.21.15.107:0/1728037528 wait complete. 2024-05-09T08:43:50.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.284+0000 7faf8705c740 20 client.27799 trim_cache size 0 max 16384 2024-05-09T08:43:50.291 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:43:50.291 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:43:50.292 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "3b933174-fae6-43ad-ab39-1a56bec70927", 2024-05-09T08:43:50.292 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:43:50.292 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:43:50.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.285+0000 7faf8434f640 1 -- 172.21.15.107:0/2760030762 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf680a2d30 msgr2=0x7faf680a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.285+0000 7faf8434f640 1 --2- 172.21.15.107:0/2760030762 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf680a2d30 0x7faf680a51f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faf74005fd0 tx=0x7faf74011b40 comp rx=0 tx=0).stop 2024-05-09T08:43:50.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.285+0000 7faf8434f640 1 -- 172.21.15.107:0/2760030762 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf7c1010c0 msgr2=0x7faf7c10e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.285+0000 7faf8434f640 1 --2- 172.21.15.107:0/2760030762 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf7c1010c0 0x7faf7c10e5a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00e9f0 tx=0x7faf6c00edf0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.285+0000 7faf8434f640 1 -- 172.21.15.107:0/2760030762 shutdown_connections 2024-05-09T08:43:50.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.285+0000 7faf8434f640 1 --2- 172.21.15.107:0/2760030762 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faf680a2d30 0x7faf680a51f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.285+0000 7faf8434f640 1 --2- 172.21.15.107:0/2760030762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf7c106710 0x7faf7c10eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.285+0000 7faf8434f640 1 --2- 172.21.15.107:0/2760030762 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf7c1010c0 0x7faf7c10e5a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.285+0000 7faf8434f640 1 --2- 172.21.15.107:0/2760030762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf7c100700 0x7faf7c10e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.285+0000 7faf8434f640 1 -- 172.21.15.107:0/2760030762 >> 172.21.15.107:0/2760030762 conn(0x7faf7c077830 msgr2=0x7faf7c103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:50.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.285+0000 7faf8434f640 1 -- 172.21.15.107:0/2760030762 shutdown_connections 2024-05-09T08:43:50.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.285+0000 7faf8434f640 1 -- 172.21.15.107:0/2760030762 wait complete. 2024-05-09T08:43:50.296 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:43:50.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:43:50.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:50 smithi157 ceph-mon[33614]: pgmap v324: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 8.8 MiB/s wr, 2.24k op/s 2024-05-09T08:43:50.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:50 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:43:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:50 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:43:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:50 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:43:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:50 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:43:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:50 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:50 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:50 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:50 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:50 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:50 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:50 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:50 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:50 smithi110 ceph-mon[33622]: pgmap v324: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 8.8 MiB/s wr, 2.24k op/s 2024-05-09T08:43:50.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:50 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:43:50.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:50 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:43:50.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:50 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:43:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:50 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:43:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:50 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:50 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:50 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:50 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:50 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:50 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:50 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:50 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:43:50.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.451+0000 7f54cf8da640 1 -- 172.21.15.107:0/800970242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c81006f0 msgr2=0x7f54c8100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.451+0000 7f54cf8da640 1 --2- 172.21.15.107:0/800970242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c81006f0 0x7f54c8100b70 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009930 tx=0x7f54b802a420 comp rx=0 tx=0).stop 2024-05-09T08:43:50.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.451+0000 7f54cf8da640 1 -- 172.21.15.107:0/800970242 shutdown_connections 2024-05-09T08:43:50.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.451+0000 7f54cf8da640 1 --2- 172.21.15.107:0/800970242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c81010b0 0x7f54c810d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.451+0000 7f54cf8da640 1 --2- 172.21.15.107:0/800970242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c81006f0 0x7f54c8100b70 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.451+0000 7f54cf8da640 1 --2- 172.21.15.107:0/800970242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54c8106700 0x7f54c8106ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.451+0000 7f54cf8da640 1 -- 172.21.15.107:0/800970242 >> 172.21.15.107:0/800970242 conn(0x7f54c80777d0 msgr2=0x7f54c8077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:50.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.451+0000 7f54cf8da640 1 -- 172.21.15.107:0/800970242 shutdown_connections 2024-05-09T08:43:50.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.452+0000 7f54cf8da640 1 -- 172.21.15.107:0/800970242 wait complete. 2024-05-09T08:43:50.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.452+0000 7f54cf8da640 1 Processor -- start 2024-05-09T08:43:50.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.452+0000 7f54cf8da640 1 -- start start 2024-05-09T08:43:50.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cf8da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c81006f0 0x7f54c819b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cf8da640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54c81010b0 0x7f54c819be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cd676640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c81006f0 0x7f54c819b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cd676640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c81006f0 0x7f54c819b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57842/0 (socket says 172.21.15.107:57842) 2024-05-09T08:43:50.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cf8da640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c8106700 0x7f54c8195b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cd676640 1 -- 172.21.15.107:0/2141077985 learned_addr learned my addr 172.21.15.107:0/2141077985 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:50.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cf8da640 1 -- 172.21.15.107:0/2141077985 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f54c810f8b0 con 0x7f54c81006f0 2024-05-09T08:43:50.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cf8da640 1 -- 172.21.15.107:0/2141077985 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f54c810f730 con 0x7f54c8106700 2024-05-09T08:43:50.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cf8da640 1 -- 172.21.15.107:0/2141077985 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f54c810fa30 con 0x7f54c81010b0 2024-05-09T08:43:50.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cce75640 1 --2- 172.21.15.107:0/2141077985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54c81010b0 0x7f54c819be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cde77640 1 --2- 172.21.15.107:0/2141077985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c8106700 0x7f54c8195b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cce75640 1 -- 172.21.15.107:0/2141077985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c8106700 msgr2=0x7f54c8195b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cce75640 1 --2- 172.21.15.107:0/2141077985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c8106700 0x7f54c8195b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cce75640 1 -- 172.21.15.107:0/2141077985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c81006f0 msgr2=0x7f54c819b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cce75640 1 --2- 172.21.15.107:0/2141077985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c81006f0 0x7f54c819b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cce75640 1 -- 172.21.15.107:0/2141077985 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54c8196290 con 0x7f54c81010b0 2024-05-09T08:43:50.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cde77640 1 --2- 172.21.15.107:0/2141077985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c8106700 0x7f54c8195b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:50.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cd676640 1 --2- 172.21.15.107:0/2141077985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c81006f0 0x7f54c819b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:50.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.453+0000 7f54cce75640 1 --2- 172.21.15.107:0/2141077985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54c81010b0 0x7f54c819be50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009900 tx=0x7f54b8030650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:50.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.454+0000 7f54be7fc640 1 -- 172.21.15.107:0/2141077985 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b803f070 con 0x7f54c81010b0 2024-05-09T08:43:50.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.454+0000 7f54cf8da640 1 -- 172.21.15.107:0/2141077985 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54c8196520 con 0x7f54c81010b0 2024-05-09T08:43:50.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.454+0000 7f54be7fc640 1 -- 172.21.15.107:0/2141077985 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54b8004460 con 0x7f54c81010b0 2024-05-09T08:43:50.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.454+0000 7f54be7fc640 1 -- 172.21.15.107:0/2141077985 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b8035030 con 0x7f54c81010b0 2024-05-09T08:43:50.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.454+0000 7f54cf8da640 1 -- 172.21.15.107:0/2141077985 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54c81a27d0 con 0x7f54c81010b0 2024-05-09T08:43:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.456+0000 7f54be7fc640 1 -- 172.21.15.107:0/2141077985 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f54b8042070 con 0x7f54c81010b0 2024-05-09T08:43:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.456+0000 7f54d25ec740 1 Processor -- start 2024-05-09T08:43:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.456+0000 7f54d25ec740 1 -- start start 2024-05-09T08:43:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54d25ec740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 0x55668b8fa660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54d25ec740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8faba0 0x55668b904d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54d25ec740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 0x55668b907660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54d25ec740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55668b85fd00 con 0x55668b905270 2024-05-09T08:43:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54d25ec740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55668b8074c0 con 0x55668b8faba0 2024-05-09T08:43:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54d25ec740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55668b85bcc0 con 0x55668b8fa280 2024-05-09T08:43:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54cce75640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 0x55668b907660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54cde77640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 0x55668b8fa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54cd676640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8faba0 0x55668b904d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54cde77640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 0x55668b8fa660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47976/0 (socket says 172.21.15.107:47976) 2024-05-09T08:43:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54cde77640 1 -- 172.21.15.107:0/2577113242 learned_addr learned my addr 172.21.15.107:0/2577113242 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54be7fc640 1 --2- 172.21.15.107:0/2141077985 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f54b40a2d00 0x7f54b40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54cde77640 1 --2- 172.21.15.107:0/2141077985 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f54b40a2d00 0x7f54b40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54cde77640 1 -- 172.21.15.107:0/2577113242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8faba0 msgr2=0x55668b904d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54cde77640 1 --2- 172.21.15.107:0/2577113242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8faba0 0x55668b904d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54cde77640 1 -- 172.21.15.107:0/2577113242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 msgr2=0x55668b907660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54be7fc640 1 -- 172.21.15.107:0/2141077985 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f54b80f0240 con 0x7f54c81010b0 2024-05-09T08:43:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54cde77640 1 --2- 172.21.15.107:0/2577113242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 0x55668b907660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.457+0000 7f54cde77640 1 -- 172.21.15.107:0/2577113242 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55668b63d7d0 con 0x55668b8fa280 2024-05-09T08:43:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.458+0000 7f54cde77640 1 --2- 172.21.15.107:0/2577113242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 0x55668b8fa660 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f54c000e780 tx=0x7f54c000ec40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee189fc76ce50c88 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.458+0000 7f54b37fe640 1 -- 172.21.15.107:0/2577113242 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c0025040 con 0x55668b8fa280 2024-05-09T08:43:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.458+0000 7f54b37fe640 1 -- 172.21.15.107:0/2577113242 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54c0026020 con 0x55668b8fa280 2024-05-09T08:43:50.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.458+0000 7f54b37fe640 1 -- 172.21.15.107:0/2577113242 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c001c070 con 0x55668b8fa280 2024-05-09T08:43:50.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.458+0000 7f54cde77640 1 --2- 172.21.15.107:0/2141077985 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f54b40a2d00 0x7f54b40a51c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f54c0027000 tx=0x7f54c0024000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:50.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.460+0000 7f54d25ec740 1 -- 172.21.15.107:0/2577113242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 msgr2=0x55668b8fa660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.460+0000 7f54d25ec740 1 --2- 172.21.15.107:0/2577113242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 0x55668b8fa660 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f54c000e780 tx=0x7f54c000ec40 comp rx=0 tx=0).stop 2024-05-09T08:43:50.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.461+0000 7f54d25ec740 1 -- 172.21.15.107:0/2577113242 shutdown_connections 2024-05-09T08:43:50.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.461+0000 7f54d25ec740 1 --2- 172.21.15.107:0/2577113242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 0x55668b907660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.461+0000 7f54d25ec740 1 --2- 172.21.15.107:0/2577113242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8faba0 0x55668b904d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.461+0000 7f54d25ec740 1 --2- 172.21.15.107:0/2577113242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 0x55668b8fa660 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.461+0000 7f54d25ec740 1 -- 172.21.15.107:0/2577113242 >> 172.21.15.107:0/2577113242 conn(0x55668b8f65a0 msgr2=0x55668b8f6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:50.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.461+0000 7f54d25ec740 1 -- 172.21.15.107:0/2577113242 shutdown_connections 2024-05-09T08:43:50.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.461+0000 7f54d25ec740 1 -- 172.21.15.107:0/2577113242 wait complete. 2024-05-09T08:43:50.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.461+0000 7f54d25ec740 1 -- start start 2024-05-09T08:43:50.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.462+0000 7f54d25ec740 1 Processor -- start 2024-05-09T08:43:50.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.462+0000 7f54d25ec740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 0x55668b9021f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.462+0000 7f54d25ec740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8faba0 0x55668b902730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.462+0000 7f54d25ec740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 0x55668b902c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.463+0000 7f54cd676640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 0x55668b9021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.463+0000 7f54cd676640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 0x55668b9021f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:47984/0 (socket says 172.21.15.107:47984) 2024-05-09T08:43:50.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.463+0000 7f54cd676640 1 -- 172.21.15.107:0/2717123846 learned_addr learned my addr 172.21.15.107:0/2717123846 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:50.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.463+0000 7f54d25ec740 1 -- 172.21.15.107:0/2717123846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55668b908060 con 0x55668b905270 2024-05-09T08:43:50.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.463+0000 7f54d25ec740 1 -- 172.21.15.107:0/2717123846 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55668b81ab90 con 0x55668b8faba0 2024-05-09T08:43:50.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.463+0000 7f54d25ec740 1 -- 172.21.15.107:0/2717123846 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55668b9081e0 con 0x55668b8fa280 2024-05-09T08:43:50.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.463+0000 7f54d25ec740 10 client.0 _ll_register_callbacks cb 0x55668b856da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:50.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.463+0000 7f54cce75640 1 --2- 172.21.15.107:0/2717123846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8faba0 0x55668b902730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.463+0000 7f54cde77640 1 --2- 172.21.15.107:0/2717123846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 0x55668b902c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.463+0000 7f54cd676640 1 -- 172.21.15.107:0/2717123846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8faba0 msgr2=0x55668b902730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:50.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.463+0000 7f54cd676640 1 --2- 172.21.15.107:0/2717123846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8faba0 0x55668b902730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.464+0000 7f54cd676640 1 -- 172.21.15.107:0/2717123846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 msgr2=0x55668b902c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:50.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.464+0000 7f54cd676640 1 --2- 172.21.15.107:0/2717123846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 0x55668b902c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.464+0000 7f54cd676640 1 -- 172.21.15.107:0/2717123846 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55668b5733f0 con 0x55668b8fa280 2024-05-09T08:43:50.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.464+0000 7f54cd676640 1 --2- 172.21.15.107:0/2717123846 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 0x55668b9021f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f54c400aab0 tx=0x7f54c400af70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de449bd95b98cdda server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:50.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.464+0000 7f54b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:43:50.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.464+0000 7f54b0ff9640 1 -- 172.21.15.107:0/2717123846 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c401b070 con 0x55668b8fa280 2024-05-09T08:43:50.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.464+0000 7f54b0ff9640 1 -- 172.21.15.107:0/2717123846 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f54c4013d20 con 0x55668b8fa280 2024-05-09T08:43:50.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.464+0000 7f54d25ec740 1 -- 172.21.15.107:0/2717123846 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54c0026020 con 0x55668b8fa280 2024-05-09T08:43:50.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.464+0000 7f54b0ff9640 1 -- 172.21.15.107:0/2717123846 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f54c4027050 con 0x55668b8fa280 2024-05-09T08:43:50.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.465+0000 7f54d25ec740 10 client.27811 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:50.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.465+0000 7f54d25ec740 1 -- 172.21.15.107:0/2717123846 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55668b5330f0 con 0x55668b8fa280 2024-05-09T08:43:50.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.465+0000 7f54b0ff9640 1 -- 172.21.15.107:0/2717123846 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54c40178e0 con 0x55668b8fa280 2024-05-09T08:43:50.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.466+0000 7f54b0ff9640 1 -- 172.21.15.107:0/2717123846 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f54c401f5f0 con 0x55668b8fa280 2024-05-09T08:43:50.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.466+0000 7f54d25ec740 10 client.27811 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:50.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.466+0000 7f54d25ec740 10 client.27811 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:50.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.466+0000 7f54d25ec740 20 client.27811 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:50.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.466+0000 7f54d25ec740 1 --2- 172.21.15.107:0/2717123846 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55668b91df90 0x55668b920430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.466+0000 7f54d25ec740 4 client.27811 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:43:50.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.466+0000 7f54d25ec740 1 -- 172.21.15.107:0/2717123846 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f54c0026020 con 0x55668b91df90 2024-05-09T08:43:50.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.466+0000 7f54cce75640 1 --2- 172.21.15.107:0/2717123846 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55668b91df90 0x55668b920430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.466+0000 7f54b0ff9640 10 client.27811 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:50.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.467+0000 7f54cce75640 1 --2- 172.21.15.107:0/2717123846 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55668b91df90 0x55668b920430 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4aa415448a518676 server_cookie=6797f3b4b3a3822e in_seq=0 out_seq=0 2024-05-09T08:43:50.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.468+0000 7f54b0ff9640 1 -- 172.21.15.107:0/2717123846 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f54c0026020 con 0x55668b91df90 2024-05-09T08:43:50.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.468+0000 7f54b0ff9640 10 client.27811 handle_command_reply: tid=0 2024-05-09T08:43:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.469+0000 7f54d25ec740 1 client.27811 shutdown 2024-05-09T08:43:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.469+0000 7f54d25ec740 1 -- 172.21.15.107:0/2717123846 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 msgr2=0x55668b9021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.469+0000 7f54d25ec740 1 --2- 172.21.15.107:0/2717123846 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 0x55668b9021f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f54c400aab0 tx=0x7f54c400af70 comp rx=0 tx=0).stop 2024-05-09T08:43:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.469+0000 7f54d25ec740 1 -- 172.21.15.107:0/2717123846 shutdown_connections 2024-05-09T08:43:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.469+0000 7f54d25ec740 1 --2- 172.21.15.107:0/2717123846 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55668b91df90 0x55668b920430 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.469+0000 7f54d25ec740 1 --2- 172.21.15.107:0/2717123846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 0x55668b902c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.469+0000 7f54d25ec740 1 --2- 172.21.15.107:0/2717123846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8faba0 0x55668b902730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.469+0000 7f54b0ff9640 0 client.27811 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.469+0000 7f54d25ec740 1 --2- 172.21.15.107:0/2717123846 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 0x55668b9021f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.469+0000 7f54d25ec740 1 -- 172.21.15.107:0/2717123846 >> 172.21.15.107:0/2717123846 conn(0x55668b8f65a0 msgr2=0x55668b8fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.469+0000 7f54d25ec740 1 -- 172.21.15.107:0/2717123846 shutdown_connections 2024-05-09T08:43:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.469+0000 7f54d25ec740 1 -- 172.21.15.107:0/2717123846 wait complete. 2024-05-09T08:43:50.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.469+0000 7f54d25ec740 20 client.27811 trim_cache size 0 max 16384 2024-05-09T08:43:50.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54d25ec740 1 Processor -- start 2024-05-09T08:43:50.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54d25ec740 1 -- start start 2024-05-09T08:43:50.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54d25ec740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 0x55668b9197b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54d25ec740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8faba0 0x55668b925d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54d25ec740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 0x55668b926240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54d25ec740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f54c81af290 con 0x55668b905270 2024-05-09T08:43:50.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54d25ec740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f54c81acb60 con 0x55668b8faba0 2024-05-09T08:43:50.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54d25ec740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f54c81aa9d0 con 0x55668b8fa280 2024-05-09T08:43:50.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54cde77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 0x55668b926240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54cde77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 0x55668b926240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57886/0 (socket says 172.21.15.107:57886) 2024-05-09T08:43:50.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54cd676640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 0x55668b9197b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54cde77640 1 -- 172.21.15.107:0/2171004379 learned_addr learned my addr 172.21.15.107:0/2171004379 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:50.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54cce75640 1 --2- 172.21.15.107:0/2171004379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8faba0 0x55668b925d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54cde77640 1 -- 172.21.15.107:0/2171004379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 msgr2=0x55668b9197b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:50.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54cde77640 1 --2- 172.21.15.107:0/2171004379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 0x55668b9197b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54cde77640 1 -- 172.21.15.107:0/2171004379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8faba0 msgr2=0x55668b925d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:50.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54cde77640 1 --2- 172.21.15.107:0/2171004379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8faba0 0x55668b925d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54cde77640 1 -- 172.21.15.107:0/2171004379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55668b91ca30 con 0x55668b905270 2024-05-09T08:43:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.493+0000 7f54cde77640 1 --2- 172.21.15.107:0/2171004379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 0x55668b926240 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f54c000efd0 tx=0x7f54c000e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c10586859f755bc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.494+0000 7f54b2ffd640 1 -- 172.21.15.107:0/2171004379 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c001c040 con 0x55668b905270 2024-05-09T08:43:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.494+0000 7f54b2ffd640 1 -- 172.21.15.107:0/2171004379 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54c0030070 con 0x55668b905270 2024-05-09T08:43:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.494+0000 7f54b2ffd640 1 -- 172.21.15.107:0/2171004379 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c0025040 con 0x55668b905270 2024-05-09T08:43:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.494+0000 7f54d25ec740 1 -- 172.21.15.107:0/2171004379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 msgr2=0x55668b926240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.494+0000 7f54d25ec740 1 --2- 172.21.15.107:0/2171004379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 0x55668b926240 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f54c000efd0 tx=0x7f54c000e680 comp rx=0 tx=0).stop 2024-05-09T08:43:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.494+0000 7f54d25ec740 1 -- 172.21.15.107:0/2171004379 shutdown_connections 2024-05-09T08:43:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.494+0000 7f54d25ec740 1 --2- 172.21.15.107:0/2171004379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b905270 0x55668b926240 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.494+0000 7f54d25ec740 1 --2- 172.21.15.107:0/2171004379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8faba0 0x55668b925d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.494+0000 7f54d25ec740 1 --2- 172.21.15.107:0/2171004379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b8fa280 0x55668b9197b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.494+0000 7f54d25ec740 1 -- 172.21.15.107:0/2171004379 >> 172.21.15.107:0/2171004379 conn(0x55668b8f65a0 msgr2=0x55668b902a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.494+0000 7f54d25ec740 1 -- 172.21.15.107:0/2171004379 shutdown_connections 2024-05-09T08:43:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.495+0000 7f54d25ec740 1 -- 172.21.15.107:0/2171004379 wait complete. 2024-05-09T08:43:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.495+0000 7f54d25ec740 1 -- start start 2024-05-09T08:43:50.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.495+0000 7f54d25ec740 1 Processor -- start 2024-05-09T08:43:50.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.495+0000 7f54d25ec740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8fa280 0x55668b8ff110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.495+0000 7f54d25ec740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b8faba0 0x55668b8ff650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.495+0000 7f54d25ec740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b905270 0x55668b8ffb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.495+0000 7f54d25ec740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f54c81ab020 con 0x55668b8faba0 2024-05-09T08:43:50.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.495+0000 7f54d25ec740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f54c81ac490 con 0x55668b8fa280 2024-05-09T08:43:50.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.495+0000 7f54d25ec740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f54c81ab6f0 con 0x55668b905270 2024-05-09T08:43:50.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.495+0000 7f54d25ec740 10 client.0 _ll_register_callbacks cb 0x7f54c400b2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:43:50.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.495+0000 7f54cce75640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b8faba0 0x55668b8ff650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.495+0000 7f54cce75640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b8faba0 0x55668b8ff650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57892/0 (socket says 172.21.15.107:57892) 2024-05-09T08:43:50.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.495+0000 7f54cce75640 1 -- 172.21.15.107:0/511070466 learned_addr learned my addr 172.21.15.107:0/511070466 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:43:50.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.495+0000 7f54cd676640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8fa280 0x55668b8ff110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.495+0000 7f54cde77640 1 --2- 172.21.15.107:0/511070466 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b905270 0x55668b8ffb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.496+0000 7f54cce75640 1 -- 172.21.15.107:0/511070466 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b905270 msgr2=0x55668b8ffb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:50.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.496+0000 7f54cce75640 1 --2- 172.21.15.107:0/511070466 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b905270 0x55668b8ffb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.496+0000 7f54cce75640 1 -- 172.21.15.107:0/511070466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8fa280 msgr2=0x55668b8ff110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:43:50.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.496+0000 7f54cce75640 1 --2- 172.21.15.107:0/511070466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8fa280 0x55668b8ff110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.496+0000 7f54cce75640 1 -- 172.21.15.107:0/511070466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55668b9837c0 con 0x55668b8faba0 2024-05-09T08:43:50.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.496+0000 7f54cd676640 1 --2- 172.21.15.107:0/511070466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8fa280 0x55668b8ff110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:43:50.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.496+0000 7f54cce75640 1 --2- 172.21.15.107:0/511070466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b8faba0 0x55668b8ff650 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f54b80bc290 tx=0x7f54b8004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed3d7b2909cf5a56 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:43:50.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.496+0000 7f54b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:43:50.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.496+0000 7f54b0ff9640 1 -- 172.21.15.107:0/511070466 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b80a1ae0 con 0x55668b8faba0 2024-05-09T08:43:50.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.496+0000 7f54b0ff9640 1 -- 172.21.15.107:0/511070466 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f54b80b89c0 con 0x55668b8faba0 2024-05-09T08:43:50.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.496+0000 7f54d25ec740 1 -- 172.21.15.107:0/511070466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54c0030070 con 0x55668b8faba0 2024-05-09T08:43:50.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.496+0000 7f54b0ff9640 1 -- 172.21.15.107:0/511070466 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f54b80bc350 con 0x55668b8faba0 2024-05-09T08:43:50.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.496+0000 7f54d25ec740 10 client.18642 fetch_fsmap learned FSMap version 27 2024-05-09T08:43:50.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.496+0000 7f54d25ec740 1 -- 172.21.15.107:0/511070466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55668b984b30 con 0x55668b8faba0 2024-05-09T08:43:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.496+0000 7f54b0ff9640 1 -- 172.21.15.107:0/511070466 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54c0030070 con 0x55668b8faba0 2024-05-09T08:43:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.496+0000 7f54b0ff9640 1 -- 172.21.15.107:0/511070466 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f54b80ed890 con 0x55668b8faba0 2024-05-09T08:43:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.497+0000 7f54d25ec740 10 client.18642 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:43:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.497+0000 7f54d25ec740 10 client.18642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:43:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.497+0000 7f54d25ec740 20 client.18642 populate_metadata read hostname 'smithi107' 2024-05-09T08:43:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.497+0000 7f54d25ec740 1 --2- 172.21.15.107:0/511070466 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55668b9852d0 0x55668b9876f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:43:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.497+0000 7f54d25ec740 4 client.18642 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:43:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.497+0000 7f54d25ec740 1 -- 172.21.15.107:0/511070466 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55668b86d430 con 0x55668b9852d0 2024-05-09T08:43:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.497+0000 7f54cd676640 1 --2- 172.21.15.107:0/511070466 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55668b9852d0 0x55668b9876f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:43:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.497+0000 7f54cd676640 1 --2- 172.21.15.107:0/511070466 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55668b9852d0 0x55668b9876f0 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=625288bca7d62c7 server_cookie=cda4280471f1423a in_seq=0 out_seq=0 2024-05-09T08:43:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.497+0000 7f54b0ff9640 10 client.18642 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.497+0000 7f54b0ff9640 1 -- 172.21.15.107:0/511070466 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55668b86d430 con 0x55668b9852d0 2024-05-09T08:43:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.497+0000 7f54b0ff9640 10 client.18642 handle_command_reply: tid=0 2024-05-09T08:43:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.497+0000 7f54d25ec740 1 client.18642 shutdown 2024-05-09T08:43:50.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.497+0000 7f54d25ec740 1 -- 172.21.15.107:0/511070466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b8faba0 msgr2=0x55668b8ff650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.497+0000 7f54d25ec740 1 --2- 172.21.15.107:0/511070466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b8faba0 0x55668b8ff650 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f54b80bc290 tx=0x7f54b8004610 comp rx=0 tx=0).stop 2024-05-09T08:43:50.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.498+0000 7f54d25ec740 1 -- 172.21.15.107:0/511070466 shutdown_connections 2024-05-09T08:43:50.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.498+0000 7f54d25ec740 1 --2- 172.21.15.107:0/511070466 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55668b9852d0 0x55668b9876f0 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.498+0000 7f54d25ec740 1 --2- 172.21.15.107:0/511070466 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55668b905270 0x55668b8ffb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.498+0000 7f54b0ff9640 0 client.18642 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:43:50.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.498+0000 7f54d25ec740 1 --2- 172.21.15.107:0/511070466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55668b8faba0 0x55668b8ff650 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.498+0000 7f54d25ec740 1 --2- 172.21.15.107:0/511070466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55668b8fa280 0x55668b8ff110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.498+0000 7f54d25ec740 1 -- 172.21.15.107:0/511070466 >> 172.21.15.107:0/511070466 conn(0x55668b8f65a0 msgr2=0x55668b902190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:50.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.498+0000 7f54d25ec740 1 -- 172.21.15.107:0/511070466 shutdown_connections 2024-05-09T08:43:50.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.498+0000 7f54d25ec740 1 -- 172.21.15.107:0/511070466 wait complete. 2024-05-09T08:43:50.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.498+0000 7f54d25ec740 20 client.18642 trim_cache size 0 max 16384 2024-05-09T08:43:50.505 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:43:50.505 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "scrub active (1 inodes in the stack)", 2024-05-09T08:43:50.505 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:43:50.505 INFO:teuthology.orchestra.run.smithi107.stdout: "3b933174-fae6-43ad-ab39-1a56bec70927": { 2024-05-09T08:43:50.505 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:43:50.505 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "3b933174-fae6-43ad-ab39-1a56bec70927", 2024-05-09T08:43:50.505 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:43:50.505 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:43:50.505 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:43:50.505 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:43:50.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.499+0000 7f54cf8da640 1 -- 172.21.15.107:0/2141077985 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f54b40a2d00 msgr2=0x7f54b40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.499+0000 7f54cf8da640 1 --2- 172.21.15.107:0/2141077985 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f54b40a2d00 0x7f54b40a51c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f54c0027000 tx=0x7f54c0024000 comp rx=0 tx=0).stop 2024-05-09T08:43:50.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.499+0000 7f54cf8da640 1 -- 172.21.15.107:0/2141077985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54c81010b0 msgr2=0x7f54c819be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:43:50.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.499+0000 7f54cf8da640 1 --2- 172.21.15.107:0/2141077985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54c81010b0 0x7f54c819be50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009900 tx=0x7f54b8030650 comp rx=0 tx=0).stop 2024-05-09T08:43:50.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.499+0000 7f54cf8da640 1 -- 172.21.15.107:0/2141077985 shutdown_connections 2024-05-09T08:43:50.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.499+0000 7f54cf8da640 1 --2- 172.21.15.107:0/2141077985 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f54b40a2d00 0x7f54b40a51c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.499+0000 7f54cf8da640 1 --2- 172.21.15.107:0/2141077985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c8106700 0x7f54c8195b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.499+0000 7f54cf8da640 1 --2- 172.21.15.107:0/2141077985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54c81010b0 0x7f54c819be50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.499+0000 7f54cf8da640 1 --2- 172.21.15.107:0/2141077985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c81006f0 0x7f54c819b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:43:50.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.499+0000 7f54cf8da640 1 -- 172.21.15.107:0/2141077985 >> 172.21.15.107:0/2141077985 conn(0x7f54c80777d0 msgr2=0x7f54c80fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:43:50.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.500+0000 7f54cf8da640 1 -- 172.21.15.107:0/2141077985 shutdown_connections 2024-05-09T08:43:50.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:43:50.500+0000 7f54cf8da640 1 -- 172.21.15.107:0/2141077985 wait complete. 2024-05-09T08:43:50.512 INFO:tasks.cephfs.filesystem:scrub status for tag:3b933174-fae6-43ad-ab39-1a56bec70927 - {'path': '/', 'tag': '3b933174-fae6-43ad-ab39-1a56bec70927', 'options': 'recursive,force'} 2024-05-09T08:43:51.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:51 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi110 to 176.1M 2024-05-09T08:43:51.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:51 smithi107 ceph-mon[30122]: Unable to set osd_memory_target on smithi110 to 184745881: error parsing value: Value '184745881' is below minimum 939524096 2024-05-09T08:43:51.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:51 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi157 to 176.2M 2024-05-09T08:43:51.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:51 smithi107 ceph-mon[30122]: Unable to set osd_memory_target on smithi157 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-05-09T08:43:51.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:51 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi110 to 176.1M 2024-05-09T08:43:51.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:51 smithi157 ceph-mon[33614]: Unable to set osd_memory_target on smithi110 to 184745881: error parsing value: Value '184745881' is below minimum 939524096 2024-05-09T08:43:51.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:51 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi157 to 176.2M 2024-05-09T08:43:51.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:51 smithi157 ceph-mon[33614]: Unable to set osd_memory_target on smithi157 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-05-09T08:43:51.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:51 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi110 to 176.1M 2024-05-09T08:43:51.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:51 smithi110 ceph-mon[33622]: Unable to set osd_memory_target on smithi110 to 184745881: error parsing value: Value '184745881' is below minimum 939524096 2024-05-09T08:43:51.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:51 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi157 to 176.2M 2024-05-09T08:43:51.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:51 smithi110 ceph-mon[33622]: Unable to set osd_memory_target on smithi157 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-05-09T08:43:52.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:52 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:43:52.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:52 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:52.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:52 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:43:52.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:52 smithi107 ceph-mon[30122]: pgmap v325: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 6.7 MiB/s wr, 1.69k op/s 2024-05-09T08:43:52.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:52 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:43:52.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:52 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:52.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:52 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:43:52.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:52 smithi157 ceph-mon[33614]: pgmap v325: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 6.7 MiB/s wr, 1.69k op/s 2024-05-09T08:43:52.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:52 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:43:52.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:52 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:43:52.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:52 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:43:52.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:52 smithi110 ceph-mon[33622]: pgmap v325: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 6.7 MiB/s wr, 1.69k op/s 2024-05-09T08:43:54.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:54 smithi107 ceph-mon[30122]: pgmap v326: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 9.1 MiB/s wr, 2.31k op/s 2024-05-09T08:43:54.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:54 smithi157 ceph-mon[33614]: pgmap v326: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 9.1 MiB/s wr, 2.31k op/s 2024-05-09T08:43:54.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:54 smithi110 ceph-mon[33622]: pgmap v326: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 9.1 MiB/s wr, 2.31k op/s 2024-05-09T08:43:56.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:56 smithi107 ceph-mon[30122]: pgmap v327: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 4.5 MiB/s wr, 1.16k op/s 2024-05-09T08:43:56.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:56 smithi157 ceph-mon[33614]: pgmap v327: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 4.5 MiB/s wr, 1.16k op/s 2024-05-09T08:43:56.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:56 smithi110 ceph-mon[33622]: pgmap v327: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 4.5 MiB/s wr, 1.16k op/s 2024-05-09T08:43:58.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:43:58 smithi107 ceph-mon[30122]: pgmap v328: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 6.9 MiB/s wr, 1.75k op/s 2024-05-09T08:43:58.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:43:58 smithi157 ceph-mon[33614]: pgmap v328: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 6.9 MiB/s wr, 1.75k op/s 2024-05-09T08:43:58.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:43:58 smithi110 ceph-mon[33622]: pgmap v328: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 6.9 MiB/s wr, 1.75k op/s 2024-05-09T08:44:00.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.614+0000 7f0573abd640 1 -- 172.21.15.107:0/3872635502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f056c1091f0 msgr2=0x7f056c115120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:00.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.614+0000 7f0573abd640 1 --2- 172.21.15.107:0/3872635502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f056c1091f0 0x7f056c115120 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0564009c10 tx=0x7f056402a720 comp rx=0 tx=0).stop 2024-05-09T08:44:00.614 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.615+0000 7f0573abd640 1 -- 172.21.15.107:0/3872635502 shutdown_connections 2024-05-09T08:44:00.614 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.615+0000 7f0573abd640 1 --2- 172.21.15.107:0/3872635502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f056c1091f0 0x7f056c115120 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.614 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.615+0000 7f0573abd640 1 --2- 172.21.15.107:0/3872635502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f056c108830 0x7f056c108cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.614 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.615+0000 7f0573abd640 1 --2- 172.21.15.107:0/3872635502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c10d290 0x7f056c10d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.614 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.615+0000 7f0573abd640 1 -- 172.21.15.107:0/3872635502 >> 172.21.15.107:0/3872635502 conn(0x7f056c06cd90 msgr2=0x7f056c06d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:00.616 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.617+0000 7f0573abd640 1 -- 172.21.15.107:0/3872635502 shutdown_connections 2024-05-09T08:44:00.616 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.617+0000 7f0573abd640 1 -- 172.21.15.107:0/3872635502 wait complete. 2024-05-09T08:44:00.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.618+0000 7f0573abd640 1 Processor -- start 2024-05-09T08:44:00.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.618+0000 7f0573abd640 1 -- start start 2024-05-09T08:44:00.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.618+0000 7f0573abd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f056c108830 0x7f056c1a7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.618+0000 7f0573abd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f056c1091f0 0x7f056c1a8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.618+0000 7f0573abd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c10d290 0x7f056c1b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.618+0000 7f0573abd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f056c1184a0 con 0x7f056c10d290 2024-05-09T08:44:00.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.618+0000 7f0573abd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f056c118320 con 0x7f056c1091f0 2024-05-09T08:44:00.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.618+0000 7f0573abd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f056c118620 con 0x7f056c108830 2024-05-09T08:44:00.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.619+0000 7f0571859640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f056c108830 0x7f056c1a7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.619+0000 7f0571058640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f056c1091f0 0x7f056c1a8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.619+0000 7f0571859640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f056c108830 0x7f056c1a7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:55430/0 (socket says 172.21.15.107:55430) 2024-05-09T08:44:00.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.619+0000 7f057205a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c10d290 0x7f056c1b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.619+0000 7f057205a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c10d290 0x7f056c1b2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51834/0 (socket says 172.21.15.107:51834) 2024-05-09T08:44:00.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.619+0000 7f057205a640 1 -- 172.21.15.107:0/1380168911 learned_addr learned my addr 172.21.15.107:0/1380168911 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:00.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.620+0000 7f057205a640 1 -- 172.21.15.107:0/1380168911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f056c108830 msgr2=0x7f056c1a7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:00.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.620+0000 7f057205a640 1 --2- 172.21.15.107:0/1380168911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f056c108830 0x7f056c1a7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.620+0000 7f057205a640 1 -- 172.21.15.107:0/1380168911 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f056c1091f0 msgr2=0x7f056c1a8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:00.620 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.620+0000 7f057205a640 1 --2- 172.21.15.107:0/1380168911 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f056c1091f0 0x7f056c1a8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.620 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.620+0000 7f057205a640 1 -- 172.21.15.107:0/1380168911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f056c1b30e0 con 0x7f056c10d290 2024-05-09T08:44:00.620 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.620+0000 7f057205a640 1 --2- 172.21.15.107:0/1380168911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c10d290 0x7f056c1b2ba0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f0564009b30 tx=0x7f0564003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:00.622 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.623+0000 7f055effd640 1 -- 172.21.15.107:0/1380168911 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0564036070 con 0x7f056c10d290 2024-05-09T08:44:00.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.624+0000 7f0573abd640 1 -- 172.21.15.107:0/1380168911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f056c1b32b0 con 0x7f056c10d290 2024-05-09T08:44:00.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.624+0000 7f0573abd640 1 -- 172.21.15.107:0/1380168911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f056c1b37a0 con 0x7f056c10d290 2024-05-09T08:44:00.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.624+0000 7f055effd640 1 -- 172.21.15.107:0/1380168911 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0564037040 con 0x7f056c10d290 2024-05-09T08:44:00.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.624+0000 7f055effd640 1 -- 172.21.15.107:0/1380168911 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0564004470 con 0x7f056c10d290 2024-05-09T08:44:00.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.628+0000 7f05767d5740 1 Processor -- start 2024-05-09T08:44:00.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.628+0000 7f055effd640 1 -- 172.21.15.107:0/1380168911 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0564004610 con 0x7f056c10d290 2024-05-09T08:44:00.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.629+0000 7f05767d5740 1 -- start start 2024-05-09T08:44:00.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.629+0000 7f05767d5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8dff450 0x55cfb8dff830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.629+0000 7f05767d5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8dffd70 0x55cfb8e0a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.629+0000 7f05767d5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8e0a630 0x55cfb8e0ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.629+0000 7f05767d5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cfb8d64340 con 0x55cfb8dff450 2024-05-09T08:44:00.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.629+0000 7f05767d5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cfb8d0c290 con 0x55cfb8e0a630 2024-05-09T08:44:00.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.629+0000 7f05767d5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55cfb8d60300 con 0x55cfb8dffd70 2024-05-09T08:44:00.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.629+0000 7f0571058640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8dff450 0x55cfb8dff830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.630+0000 7f0571058640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8dff450 0x55cfb8dff830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35848/0 (socket says 172.21.15.107:35848) 2024-05-09T08:44:00.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.630+0000 7f0571058640 1 -- 172.21.15.107:0/2142976655 learned_addr learned my addr 172.21.15.107:0/2142976655 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:00.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.630+0000 7f0571058640 1 --2- 172.21.15.107:0/2142976655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8e0a630 0x55cfb8e0ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.630+0000 7f0571859640 1 --2- 172.21.15.107:0/2142976655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8dffd70 0x55cfb8e0a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.631+0000 7f055effd640 1 --2- 172.21.15.107:0/1380168911 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f05580a2fd0 0x7f05580a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.631+0000 7f055effd640 1 -- 172.21.15.107:0/1380168911 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f05640f05e0 con 0x7f056c10d290 2024-05-09T08:44:00.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.631+0000 7f0571058640 1 -- 172.21.15.107:0/2142976655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8dffd70 msgr2=0x55cfb8e0a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:00.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.631+0000 7f0571058640 1 --2- 172.21.15.107:0/2142976655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8dffd70 0x55cfb8e0a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.632+0000 7f0571058640 1 -- 172.21.15.107:0/2142976655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8e0a630 msgr2=0x55cfb8e0ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:00.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.632+0000 7f0571058640 1 --2- 172.21.15.107:0/2142976655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8e0a630 0x55cfb8e0ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.632+0000 7f0571058640 1 -- 172.21.15.107:0/2142976655 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfb8d308e0 con 0x55cfb8dff450 2024-05-09T08:44:00.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.632+0000 7f057205a640 1 --2- 172.21.15.107:0/1380168911 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f05580a2fd0 0x7f05580a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.632+0000 7f0571058640 1 --2- 172.21.15.107:0/2142976655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8dff450 0x55cfb8dff830 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f05680058c0 tx=0x7f0568009350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef24452ab014bc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:00.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.633+0000 7f057205a640 1 --2- 172.21.15.107:0/1380168911 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f05580a2fd0 0x7f05580a5490 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f05640bc620 tx=0x7f05640098d0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:00.633 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.634+0000 7f0557fff640 1 -- 172.21.15.107:0/2142976655 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0568005220 con 0x55cfb8dff450 2024-05-09T08:44:00.633 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.634+0000 7f0557fff640 1 -- 172.21.15.107:0/2142976655 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0568015e50 con 0x55cfb8dff450 2024-05-09T08:44:00.634 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.635+0000 7f05767d5740 1 -- 172.21.15.107:0/2142976655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8dff450 msgr2=0x55cfb8dff830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:00.634 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.635+0000 7f05767d5740 1 --2- 172.21.15.107:0/2142976655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8dff450 0x55cfb8dff830 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f05680058c0 tx=0x7f0568009350 comp rx=0 tx=0).stop 2024-05-09T08:44:00.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.640+0000 7f05767d5740 1 -- 172.21.15.107:0/2142976655 shutdown_connections 2024-05-09T08:44:00.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.640+0000 7f05767d5740 1 --2- 172.21.15.107:0/2142976655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8e0a630 0x55cfb8e0ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.640+0000 7f05767d5740 1 --2- 172.21.15.107:0/2142976655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8dffd70 0x55cfb8e0a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.640+0000 7f05767d5740 1 --2- 172.21.15.107:0/2142976655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8dff450 0x55cfb8dff830 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.640+0000 7f05767d5740 1 -- 172.21.15.107:0/2142976655 >> 172.21.15.107:0/2142976655 conn(0x55cfb8dfb960 msgr2=0x55cfb8dfbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:00.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.640+0000 7f05767d5740 1 -- 172.21.15.107:0/2142976655 shutdown_connections 2024-05-09T08:44:00.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.640+0000 7f05767d5740 1 -- 172.21.15.107:0/2142976655 wait complete. 2024-05-09T08:44:00.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.640+0000 7f05767d5740 1 -- start start 2024-05-09T08:44:00.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.641+0000 7f05767d5740 1 Processor -- start 2024-05-09T08:44:00.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.641+0000 7f05767d5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dff450 0x55cfb8e0a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.641+0000 7f05767d5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8dffd70 0x55cfb8e07440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.641+0000 7f05767d5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8e0a630 0x55cfb8e07b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.641+0000 7f05767d5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cfb8e0d200 con 0x55cfb8e0a630 2024-05-09T08:44:00.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.641+0000 7f05767d5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cfb8d1f790 con 0x55cfb8dff450 2024-05-09T08:44:00.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.641+0000 7f05767d5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55cfb8e0d380 con 0x55cfb8dffd70 2024-05-09T08:44:00.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.641+0000 7f05767d5740 10 client.0 _ll_register_callbacks cb 0x55cfb8d5aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:00.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.641+0000 7f0571058640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dff450 0x55cfb8e0a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.641+0000 7f0571058640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dff450 0x55cfb8e0a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:56726/0 (socket says 172.21.15.107:56726) 2024-05-09T08:44:00.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.641+0000 7f0571058640 1 -- 172.21.15.107:0/2795285713 learned_addr learned my addr 172.21.15.107:0/2795285713 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:00.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.641+0000 7f0571058640 1 --2- 172.21.15.107:0/2795285713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8e0a630 0x55cfb8e07b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.642+0000 7f0571859640 1 --2- 172.21.15.107:0/2795285713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8dffd70 0x55cfb8e07440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.642+0000 7f0571058640 1 -- 172.21.15.107:0/2795285713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8dffd70 msgr2=0x55cfb8e07440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:00.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.642+0000 7f0571058640 1 --2- 172.21.15.107:0/2795285713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8dffd70 0x55cfb8e07440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.642+0000 7f0571058640 1 -- 172.21.15.107:0/2795285713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8e0a630 msgr2=0x55cfb8e07b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:00.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.642+0000 7f0571058640 1 --2- 172.21.15.107:0/2795285713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8e0a630 0x55cfb8e07b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.642+0000 7f0571058640 1 -- 172.21.15.107:0/2795285713 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfb8a78210 con 0x55cfb8dff450 2024-05-09T08:44:00.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.642+0000 7f0571058640 1 --2- 172.21.15.107:0/2795285713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dff450 0x55cfb8e0a0b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f05680057b0 tx=0x7f05680056b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=479b7ff9c570faab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:00.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.642+0000 7f05557fa640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:44:00.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.642+0000 7f05557fa640 1 -- 172.21.15.107:0/2795285713 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0568010880 con 0x55cfb8dff450 2024-05-09T08:44:00.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.642+0000 7f05557fa640 1 -- 172.21.15.107:0/2795285713 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f0568010a20 con 0x55cfb8dff450 2024-05-09T08:44:00.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.643+0000 7f05767d5740 1 -- 172.21.15.107:0/2795285713 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0568015e50 con 0x55cfb8dff450 2024-05-09T08:44:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.643+0000 7f05557fa640 1 -- 172.21.15.107:0/2795285713 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0568010c20 con 0x55cfb8dff450 2024-05-09T08:44:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.643+0000 7f05767d5740 10 client.27602 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.643+0000 7f05767d5740 1 -- 172.21.15.107:0/2795285713 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cfb8b44620 con 0x55cfb8dff450 2024-05-09T08:44:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.643+0000 7f05557fa640 1 -- 172.21.15.107:0/2795285713 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0568015e50 con 0x55cfb8dff450 2024-05-09T08:44:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.643+0000 7f05557fa640 1 -- 172.21.15.107:0/2795285713 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f0568023440 con 0x55cfb8dff450 2024-05-09T08:44:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.643+0000 7f05767d5740 10 client.27602 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.643+0000 7f05767d5740 10 client.27602 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.644+0000 7f05767d5740 20 client.27602 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.644+0000 7f05767d5740 1 --2- 172.21.15.107:0/2795285713 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55cfb8e1a1c0 0x55cfb8e1c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.644+0000 7f05767d5740 4 client.27602 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:44:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.644+0000 7f05767d5740 1 -- 172.21.15.107:0/2795285713 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfb8d71a70 con 0x55cfb8e1a1c0 2024-05-09T08:44:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.644+0000 7f0571859640 1 --2- 172.21.15.107:0/2795285713 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55cfb8e1a1c0 0x55cfb8e1c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.644+0000 7f05557fa640 10 client.27602 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.644+0000 7f0571859640 1 --2- 172.21.15.107:0/2795285713 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55cfb8e1a1c0 0x55cfb8e1c5b0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa0fac29ada3bc20 server_cookie=235bb440a5bfc3b6 in_seq=0 out_seq=0 2024-05-09T08:44:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.645+0000 7f05557fa640 1 -- 172.21.15.107:0/2795285713 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cfb8d71a70 con 0x55cfb8e1a1c0 2024-05-09T08:44:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.645+0000 7f05557fa640 10 client.27602 handle_command_reply: tid=0 2024-05-09T08:44:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.645+0000 7f05767d5740 1 client.27602 shutdown 2024-05-09T08:44:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.646+0000 7f05767d5740 1 -- 172.21.15.107:0/2795285713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dff450 msgr2=0x55cfb8e0a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.646+0000 7f05767d5740 1 --2- 172.21.15.107:0/2795285713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dff450 0x55cfb8e0a0b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f05680057b0 tx=0x7f05680056b0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.648+0000 7f05767d5740 1 -- 172.21.15.107:0/2795285713 shutdown_connections 2024-05-09T08:44:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.648+0000 7f05767d5740 1 --2- 172.21.15.107:0/2795285713 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55cfb8e1a1c0 0x55cfb8e1c5b0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.648+0000 7f05767d5740 1 --2- 172.21.15.107:0/2795285713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8e0a630 0x55cfb8e07b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.648+0000 7f05767d5740 1 --2- 172.21.15.107:0/2795285713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8dffd70 0x55cfb8e07440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.648+0000 7f05767d5740 1 --2- 172.21.15.107:0/2795285713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dff450 0x55cfb8e0a0b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.648+0000 7f05767d5740 1 -- 172.21.15.107:0/2795285713 >> 172.21.15.107:0/2795285713 conn(0x55cfb8dfb960 msgr2=0x55cfb8e016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.648+0000 7f05557fa640 0 client.27602 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.648+0000 7f05767d5740 1 -- 172.21.15.107:0/2795285713 shutdown_connections 2024-05-09T08:44:00.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.648+0000 7f05767d5740 1 -- 172.21.15.107:0/2795285713 wait complete. 2024-05-09T08:44:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.650+0000 7f05767d5740 20 client.27602 trim_cache size 0 max 16384 2024-05-09T08:44:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.684+0000 7f05767d5740 1 Processor -- start 2024-05-09T08:44:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.684+0000 7f05767d5740 1 -- start start 2024-05-09T08:44:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.684+0000 7f05767d5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8dff450 0x55cfb8e21dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.684+0000 7f05767d5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dffd70 0x55cfb8e22300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.685+0000 7f05767d5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8e0a630 0x55cfb8e09b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.685+0000 7f05767d5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0550012a50 con 0x55cfb8dff450 2024-05-09T08:44:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.685+0000 7f05767d5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0550010320 con 0x55cfb8dffd70 2024-05-09T08:44:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.685+0000 7f0571859640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dffd70 0x55cfb8e22300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.685+0000 7f0571859640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dffd70 0x55cfb8e22300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:56740/0 (socket says 172.21.15.107:56740) 2024-05-09T08:44:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.685+0000 7f0571859640 1 -- 172.21.15.107:0/3241199060 learned_addr learned my addr 172.21.15.107:0/3241199060 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.685+0000 7f05767d5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f055000e190 con 0x55cfb8e0a630 2024-05-09T08:44:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.685+0000 7f0571058640 1 --2- 172.21.15.107:0/3241199060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8e0a630 0x55cfb8e09b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.685+0000 7f0571058640 1 --2- 172.21.15.107:0/3241199060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8dff450 0x55cfb8e21dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.685+0000 7f0571859640 1 -- 172.21.15.107:0/3241199060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8e0a630 msgr2=0x55cfb8e09b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.685+0000 7f0571859640 1 --2- 172.21.15.107:0/3241199060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8e0a630 0x55cfb8e09b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.685+0000 7f0571859640 1 -- 172.21.15.107:0/3241199060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8dff450 msgr2=0x55cfb8e21dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.685+0000 7f0571859640 1 --2- 172.21.15.107:0/3241199060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8dff450 0x55cfb8e21dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.685+0000 7f0571859640 1 -- 172.21.15.107:0/3241199060 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfb8e07ef0 con 0x55cfb8dffd70 2024-05-09T08:44:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.686+0000 7f0571859640 1 --2- 172.21.15.107:0/3241199060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dffd70 0x55cfb8e22300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0560005940 tx=0x7f0560009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89c0c3a075d606bd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.687+0000 7f05577fe640 1 -- 172.21.15.107:0/3241199060 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0560003ee0 con 0x55cfb8dffd70 2024-05-09T08:44:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.687+0000 7f05577fe640 1 -- 172.21.15.107:0/3241199060 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0560020660 con 0x55cfb8dffd70 2024-05-09T08:44:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.687+0000 7f05577fe640 1 -- 172.21.15.107:0/3241199060 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0560011c00 con 0x55cfb8dffd70 2024-05-09T08:44:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.688+0000 7f05767d5740 1 -- 172.21.15.107:0/3241199060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dffd70 msgr2=0x55cfb8e22300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.688+0000 7f05767d5740 1 --2- 172.21.15.107:0/3241199060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dffd70 0x55cfb8e22300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0560005940 tx=0x7f0560009260 comp rx=0 tx=0).stop 2024-05-09T08:44:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.688+0000 7f05767d5740 1 -- 172.21.15.107:0/3241199060 shutdown_connections 2024-05-09T08:44:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.688+0000 7f05767d5740 1 --2- 172.21.15.107:0/3241199060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8e0a630 0x55cfb8e09b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.688+0000 7f05767d5740 1 --2- 172.21.15.107:0/3241199060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dffd70 0x55cfb8e22300 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.688+0000 7f05767d5740 1 --2- 172.21.15.107:0/3241199060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8dff450 0x55cfb8e21dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.688+0000 7f05767d5740 1 -- 172.21.15.107:0/3241199060 >> 172.21.15.107:0/3241199060 conn(0x55cfb8dfb960 msgr2=0x55cfb8dfaa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.691+0000 7f05767d5740 1 -- 172.21.15.107:0/3241199060 shutdown_connections 2024-05-09T08:44:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.691+0000 7f05767d5740 1 -- 172.21.15.107:0/3241199060 wait complete. 2024-05-09T08:44:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.691+0000 7f05767d5740 1 -- start start 2024-05-09T08:44:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.691+0000 7f05767d5740 1 Processor -- start 2024-05-09T08:44:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.692+0000 7f05767d5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dff450 0x55cfb8e129a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.692+0000 7f05767d5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8dffd70 0x55cfb8e0dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.692+0000 7f0571058640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dff450 0x55cfb8e129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.692+0000 7f0571058640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dff450 0x55cfb8e129a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:56748/0 (socket says 172.21.15.107:56748) 2024-05-09T08:44:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.692+0000 7f0571058640 1 -- 172.21.15.107:0/3737269194 learned_addr learned my addr 172.21.15.107:0/3737269194 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.692+0000 7f05767d5740 1 --2- 172.21.15.107:0/3737269194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8e0a630 0x55cfb8e0e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.692+0000 7f0571859640 1 --2- 172.21.15.107:0/3737269194 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8dffd70 0x55cfb8e0dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.692+0000 7f05767d5740 1 -- 172.21.15.107:0/3737269194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f055000e7e0 con 0x55cfb8e0a630 2024-05-09T08:44:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.693+0000 7f05767d5740 1 -- 172.21.15.107:0/3737269194 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f055000fc50 con 0x55cfb8dff450 2024-05-09T08:44:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.693+0000 7f05767d5740 1 -- 172.21.15.107:0/3737269194 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f055000eeb0 con 0x55cfb8dffd70 2024-05-09T08:44:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.693+0000 7f0571058640 1 --2- 172.21.15.107:0/3737269194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8e0a630 0x55cfb8e0e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.693+0000 7f0571058640 1 -- 172.21.15.107:0/3737269194 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8dffd70 msgr2=0x55cfb8e0dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.693+0000 7f0571058640 1 --2- 172.21.15.107:0/3737269194 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8dffd70 0x55cfb8e0dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.693+0000 7f0571058640 1 -- 172.21.15.107:0/3737269194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8e0a630 msgr2=0x55cfb8e0e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.693+0000 7f0571058640 1 --2- 172.21.15.107:0/3737269194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8e0a630 0x55cfb8e0e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.693+0000 7f0571058640 1 -- 172.21.15.107:0/3737269194 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfb8e0e940 con 0x55cfb8dff450 2024-05-09T08:44:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.693+0000 7f05767d5740 10 client.0 _ll_register_callbacks cb 0x7f0568009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.693+0000 7f0571058640 1 --2- 172.21.15.107:0/3737269194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dff450 0x55cfb8e129a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0568005780 tx=0x7f05680092a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96e47c6589b64c29 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.693+0000 7f05767d5740 1 -- 172.21.15.107:0/3737269194 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0560020660 con 0x55cfb8dff450 2024-05-09T08:44:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.694+0000 7f05557fa640 10 client.27608 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:44:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.694+0000 7f05557fa640 1 -- 172.21.15.107:0/3737269194 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0568042020 con 0x55cfb8dff450 2024-05-09T08:44:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.694+0000 7f05557fa640 1 -- 172.21.15.107:0/3737269194 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f0568023b00 con 0x55cfb8dff450 2024-05-09T08:44:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.694+0000 7f05557fa640 1 -- 172.21.15.107:0/3737269194 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0568010880 con 0x55cfb8dff450 2024-05-09T08:44:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.694+0000 7f05767d5740 10 client.27608 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.695+0000 7f05767d5740 1 -- 172.21.15.107:0/3737269194 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55cfb8e80320 con 0x55cfb8dff450 2024-05-09T08:44:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.696+0000 7f05557fa640 1 -- 172.21.15.107:0/3737269194 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f0568045020 con 0x55cfb8dff450 2024-05-09T08:44:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.696+0000 7f05767d5740 10 client.27608 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.696+0000 7f05767d5740 10 client.27608 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.696+0000 7f05767d5740 20 client.27608 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.696+0000 7f05767d5740 1 --2- 172.21.15.107:0/3737269194 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55cfb8e809f0 0x55cfb8e82e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.696+0000 7f05767d5740 4 client.27608 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:44:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.696+0000 7f05767d5740 1 -- 172.21.15.107:0/3737269194 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f0560020660 con 0x55cfb8e809f0 2024-05-09T08:44:00.698 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.698+0000 7f0571859640 1 --2- 172.21.15.107:0/3737269194 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55cfb8e809f0 0x55cfb8e82e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.699+0000 7f0571859640 1 --2- 172.21.15.107:0/3737269194 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55cfb8e809f0 0x55cfb8e82e30 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b5ae1c689cf8d3e server_cookie=9b61d18cf88424af in_seq=0 out_seq=0 2024-05-09T08:44:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.700+0000 7f05557fa640 10 client.27608 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:00.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:00 smithi110 ceph-mon[33622]: pgmap v329: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.6 MiB/s wr, 1.94k op/s 2024-05-09T08:44:00.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:00 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:44:00.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:00 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:44:00.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:00 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:00.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:00 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:00.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:00 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:00.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:00 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:00.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:00 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:00.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:00 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.953+0000 7f05557fa640 1 -- 172.21.15.107:0/3737269194 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f0560020660 con 0x55cfb8e809f0 2024-05-09T08:44:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.953+0000 7f05557fa640 10 client.27608 handle_command_reply: tid=0 2024-05-09T08:44:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.953+0000 7f05767d5740 1 client.27608 shutdown 2024-05-09T08:44:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.953+0000 7f05767d5740 1 -- 172.21.15.107:0/3737269194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dff450 msgr2=0x55cfb8e129a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.953+0000 7f05767d5740 1 --2- 172.21.15.107:0/3737269194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dff450 0x55cfb8e129a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0568005780 tx=0x7f05680092a0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.954+0000 7f05767d5740 1 -- 172.21.15.107:0/3737269194 shutdown_connections 2024-05-09T08:44:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.954+0000 7f05767d5740 1 --2- 172.21.15.107:0/3737269194 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55cfb8e809f0 0x55cfb8e82e30 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.954+0000 7f05767d5740 1 --2- 172.21.15.107:0/3737269194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfb8e0a630 0x55cfb8e0e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.954+0000 7f05767d5740 1 --2- 172.21.15.107:0/3737269194 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cfb8dffd70 0x55cfb8e0dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.954+0000 7f05767d5740 1 --2- 172.21.15.107:0/3737269194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cfb8dff450 0x55cfb8e129a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.954+0000 7f05767d5740 1 -- 172.21.15.107:0/3737269194 >> 172.21.15.107:0/3737269194 conn(0x55cfb8dfb960 msgr2=0x55cfb8e00e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.954+0000 7f05557fa640 0 client.27608 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.954+0000 7f05767d5740 1 -- 172.21.15.107:0/3737269194 shutdown_connections 2024-05-09T08:44:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.954+0000 7f05767d5740 1 -- 172.21.15.107:0/3737269194 wait complete. 2024-05-09T08:44:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.954+0000 7f05767d5740 20 client.27608 trim_cache size 0 max 16384 2024-05-09T08:44:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:44:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:44:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:44:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:44:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.955+0000 7f0573abd640 1 -- 172.21.15.107:0/1380168911 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f05580a2fd0 msgr2=0x7f05580a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.955+0000 7f0573abd640 1 --2- 172.21.15.107:0/1380168911 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f05580a2fd0 0x7f05580a5490 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f05640bc620 tx=0x7f05640098d0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.955+0000 7f0573abd640 1 -- 172.21.15.107:0/1380168911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c10d290 msgr2=0x7f056c1b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.955+0000 7f0573abd640 1 --2- 172.21.15.107:0/1380168911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c10d290 0x7f056c1b2ba0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f0564009b30 tx=0x7f0564003ce0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.955+0000 7f0573abd640 1 -- 172.21.15.107:0/1380168911 shutdown_connections 2024-05-09T08:44:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.955+0000 7f0573abd640 1 --2- 172.21.15.107:0/1380168911 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f05580a2fd0 0x7f05580a5490 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.955+0000 7f0573abd640 1 --2- 172.21.15.107:0/1380168911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c10d290 0x7f056c1b2ba0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.955+0000 7f0573abd640 1 --2- 172.21.15.107:0/1380168911 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f056c1091f0 0x7f056c1a8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.955+0000 7f0573abd640 1 --2- 172.21.15.107:0/1380168911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f056c108830 0x7f056c1a7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.955+0000 7f0573abd640 1 -- 172.21.15.107:0/1380168911 >> 172.21.15.107:0/1380168911 conn(0x7f056c06cd90 msgr2=0x7f056c119890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.955+0000 7f0573abd640 1 -- 172.21.15.107:0/1380168911 shutdown_connections 2024-05-09T08:44:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:00.955+0000 7f0573abd640 1 -- 172.21.15.107:0/1380168911 wait complete. 2024-05-09T08:44:01.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:00 smithi107 ceph-mon[30122]: pgmap v329: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.6 MiB/s wr, 1.94k op/s 2024-05-09T08:44:01.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:00 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:44:01.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:00 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:44:01.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:00 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:01.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:00 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:01.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:00 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:01.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:00 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:01.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:00 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:01.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:00 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:01.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:00 smithi157 ceph-mon[33614]: pgmap v329: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.6 MiB/s wr, 1.94k op/s 2024-05-09T08:44:01.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:00 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:44:01.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:00 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:44:01.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:00 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:01.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:00 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:01.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:00 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:01.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:00 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T08:44:01.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:00 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:44:01.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:00 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:44:01.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:01 smithi110 ceph-mon[33622]: Adjusting osd_memory_target on smithi107 to 176.1M 2024-05-09T08:44:01.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:01 smithi110 ceph-mon[33622]: Unable to set osd_memory_target on smithi107 to 184708608: error parsing value: Value '184708608' is below minimum 939524096 2024-05-09T08:44:01.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:01 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:44:02.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:01 smithi107 ceph-mon[30122]: Adjusting osd_memory_target on smithi107 to 176.1M 2024-05-09T08:44:02.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:01 smithi107 ceph-mon[30122]: Unable to set osd_memory_target on smithi107 to 184708608: error parsing value: Value '184708608' is below minimum 939524096 2024-05-09T08:44:02.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:01 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:44:02.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:01 smithi157 ceph-mon[33614]: Adjusting osd_memory_target on smithi107 to 176.1M 2024-05-09T08:44:02.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:01 smithi157 ceph-mon[33614]: Unable to set osd_memory_target on smithi107 to 184708608: error parsing value: Value '184708608' is below minimum 939524096 2024-05-09T08:44:02.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:01 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:44:02.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:02 smithi110 ceph-mon[33622]: pgmap v330: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.9 MiB/s wr, 1.77k op/s 2024-05-09T08:44:03.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:02 smithi107 ceph-mon[30122]: pgmap v330: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.9 MiB/s wr, 1.77k op/s 2024-05-09T08:44:03.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:02 smithi157 ceph-mon[33614]: pgmap v330: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.9 MiB/s wr, 1.77k op/s 2024-05-09T08:44:05.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:04 smithi107 ceph-mon[30122]: pgmap v331: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 9.8 MiB/s wr, 2.50k op/s 2024-05-09T08:44:05.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:04 smithi157 ceph-mon[33614]: pgmap v331: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 9.8 MiB/s wr, 2.50k op/s 2024-05-09T08:44:05.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:04 smithi110 ceph-mon[33622]: pgmap v331: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 9.8 MiB/s wr, 2.50k op/s 2024-05-09T08:44:07.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:06 smithi107 ceph-mon[30122]: pgmap v332: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 7.5 MiB/s wr, 1.92k op/s 2024-05-09T08:44:07.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:06 smithi157 ceph-mon[33614]: pgmap v332: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 7.5 MiB/s wr, 1.92k op/s 2024-05-09T08:44:07.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:06 smithi110 ceph-mon[33622]: pgmap v332: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 7.5 MiB/s wr, 1.92k op/s 2024-05-09T08:44:08.895 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:08 smithi157 ceph-mon[33614]: pgmap v333: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 10 MiB/s wr, 2.61k op/s 2024-05-09T08:44:08.895 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:08 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:44:08.895 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:08 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:08.895 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:08 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:44:08.895 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:08 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:08.895 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:08 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:44:09.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:08 smithi107 ceph-mon[30122]: pgmap v333: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 10 MiB/s wr, 2.61k op/s 2024-05-09T08:44:09.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:08 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:44:09.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:08 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:09.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:08 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:44:09.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:08 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:09.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:08 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:44:09.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:08 smithi110 ceph-mon[33622]: pgmap v333: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 10 MiB/s wr, 2.61k op/s 2024-05-09T08:44:09.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:08 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:44:09.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:08 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:09.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:08 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:44:09.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:08 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:09.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:08 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:44:09.905 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:09 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:44:09.962 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:09 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:09.962 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:09 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:44:09.962 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:09 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:09.962 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:09 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:09.962 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:09 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:44:09.962 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:09 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:10.021 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:09 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:44:10.022 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:09 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:10.022 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:09 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:44:10.022 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:09 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:10.022 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:09 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:10.022 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:09 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:44:10.022 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:09 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:10.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:09 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:44:10.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:09 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:10.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:09 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:44:10.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:09 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:10.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:09 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:10.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:09 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:44:10.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:09 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:11.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:10 smithi107 ceph-mon[30122]: pgmap v334: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.3 MiB/s wr, 2.38k op/s 2024-05-09T08:44:11.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:10 smithi157 ceph-mon[33614]: pgmap v334: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.3 MiB/s wr, 2.38k op/s 2024-05-09T08:44:11.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:10 smithi110 ceph-mon[33622]: pgmap v334: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.3 MiB/s wr, 2.38k op/s 2024-05-09T08:44:12.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:11 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:12.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:11 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:12.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:11 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:13.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:12 smithi107 ceph-mon[30122]: pgmap v335: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.8 MiB/s wr, 1.99k op/s 2024-05-09T08:44:13.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:12 smithi157 ceph-mon[33614]: pgmap v335: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.8 MiB/s wr, 1.99k op/s 2024-05-09T08:44:13.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:12 smithi110 ceph-mon[33622]: pgmap v335: 129 pgs: 2 active+clean+snaptrim, 124 active+clean, 3 active+clean+snaptrim_wait; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.8 MiB/s wr, 1.99k op/s 2024-05-09T08:44:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.088+0000 7f81464b3640 1 -- 172.21.15.107:0/4025273726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140103400 msgr2=0x7f814010b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.088+0000 7f81464b3640 1 --2- 172.21.15.107:0/4025273726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140103400 0x7f814010b8e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8134009860 tx=0x7f813402a3d0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.088+0000 7f81464b3640 1 -- 172.21.15.107:0/4025273726 shutdown_connections 2024-05-09T08:44:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.088+0000 7f81464b3640 1 --2- 172.21.15.107:0/4025273726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814010be20 0x7f814010e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.088+0000 7f81464b3640 1 --2- 172.21.15.107:0/4025273726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140103400 0x7f814010b8e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.088+0000 7f81464b3640 1 --2- 172.21.15.107:0/4025273726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8140102ae0 0x7f8140102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.088+0000 7f81464b3640 1 -- 172.21.15.107:0/4025273726 >> 172.21.15.107:0/4025273726 conn(0x7f81400778b0 msgr2=0x7f8140077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.089+0000 7f81464b3640 1 -- 172.21.15.107:0/4025273726 shutdown_connections 2024-05-09T08:44:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.089+0000 7f81464b3640 1 -- 172.21.15.107:0/4025273726 wait complete. 2024-05-09T08:44:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.089+0000 7f81464b3640 1 Processor -- start 2024-05-09T08:44:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.089+0000 7f81464b3640 1 -- start start 2024-05-09T08:44:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.090+0000 7f81464b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8140102ae0 0x7f8140197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.090+0000 7f81464b3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140103400 0x7f81401984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.090+0000 7f81464b3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f814010be20 0x7f814019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.090+0000 7f813ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8140102ae0 0x7f8140197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.090+0000 7f813ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8140102ae0 0x7f8140197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58760/0 (socket says 172.21.15.107:58760) 2024-05-09T08:44:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.090+0000 7f81464b3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8140111aa0 con 0x7f8140102ae0 2024-05-09T08:44:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.090+0000 7f81464b3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8140111920 con 0x7f8140103400 2024-05-09T08:44:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.090+0000 7f81464b3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8140111c20 con 0x7f814010be20 2024-05-09T08:44:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.091+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140103400 0x7f81401984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.091+0000 7f8144a50640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f814010be20 0x7f814019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.091+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140103400 0x7f81401984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:38518/0 (socket says 172.21.15.107:38518) 2024-05-09T08:44:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.091+0000 7f813f7fe640 1 -- 172.21.15.107:0/1219226859 learned_addr learned my addr 172.21.15.107:0/1219226859 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:14.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.091+0000 7f813f7fe640 1 -- 172.21.15.107:0/1219226859 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f814010be20 msgr2=0x7f814019c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.091+0000 7f813f7fe640 1 --2- 172.21.15.107:0/1219226859 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f814010be20 0x7f814019c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.091+0000 7f813f7fe640 1 -- 172.21.15.107:0/1219226859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8140102ae0 msgr2=0x7f8140197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.091+0000 7f813f7fe640 1 --2- 172.21.15.107:0/1219226859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8140102ae0 0x7f8140197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.091+0000 7f813f7fe640 1 -- 172.21.15.107:0/1219226859 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f814019cf90 con 0x7f8140103400 2024-05-09T08:44:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.091+0000 7f8144a50640 1 --2- 172.21.15.107:0/1219226859 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f814010be20 0x7f814019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:44:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.091+0000 7f813f7fe640 1 --2- 172.21.15.107:0/1219226859 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140103400 0x7f81401984e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8134009980 tx=0x7f8134030ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.091+0000 7f813d7fa640 1 -- 172.21.15.107:0/1219226859 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8134040070 con 0x7f8140103400 2024-05-09T08:44:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.092+0000 7f813d7fa640 1 -- 172.21.15.107:0/1219226859 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8134031400 con 0x7f8140103400 2024-05-09T08:44:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.092+0000 7f813d7fa640 1 -- 172.21.15.107:0/1219226859 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8134004b20 con 0x7f8140103400 2024-05-09T08:44:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.092+0000 7f81464b3640 1 -- 172.21.15.107:0/1219226859 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f814019d220 con 0x7f8140103400 2024-05-09T08:44:14.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.092+0000 7f81464b3640 1 -- 172.21.15.107:0/1219226859 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81401a4b10 con 0x7f8140103400 2024-05-09T08:44:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.094+0000 7f81491be740 1 Processor -- start 2024-05-09T08:44:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f81491be740 1 -- start start 2024-05-09T08:44:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f813d7fa640 1 -- 172.21.15.107:0/1219226859 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8134039040 con 0x7f8140103400 2024-05-09T08:44:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f81491be740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120330 0x564fe0120710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f81491be740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120c50 0x564fe012f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f81491be740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe0131e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f81491be740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564fe0085340 con 0x564fe0120c50 2024-05-09T08:44:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f81491be740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564fe002d290 con 0x564fe012fa40 2024-05-09T08:44:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f81491be740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564fe0081300 con 0x564fe0120330 2024-05-09T08:44:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f8144a50640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120330 0x564fe0120710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f8144a50640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120330 0x564fe0120710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:45366/0 (socket says 172.21.15.107:45366) 2024-05-09T08:44:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f8144a50640 1 -- 172.21.15.107:0/38166599 learned_addr learned my addr 172.21.15.107:0/38166599 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f813ffff640 1 --2- 172.21.15.107:0/38166599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120c50 0x564fe012f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f8144a50640 1 -- 172.21.15.107:0/38166599 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 msgr2=0x564fe0131e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f8144a50640 1 --2- 172.21.15.107:0/38166599 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe0131e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f8144a50640 1 -- 172.21.15.107:0/38166599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120c50 msgr2=0x564fe012f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f8144a50640 1 --2- 172.21.15.107:0/38166599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120c50 0x564fe012f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.095+0000 7f8144a50640 1 -- 172.21.15.107:0/38166599 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fe00518e0 con 0x564fe0120330 2024-05-09T08:44:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.096+0000 7f8144a50640 1 --2- 172.21.15.107:0/38166599 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120330 0x564fe0120710 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f813800e870 tx=0x7f81380095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ce21623eab027de server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.096+0000 7f813d7fa640 1 --2- 172.21.15.107:0/1219226859 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f81280a2e20 0x7f81280a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.096+0000 7f81267fc640 1 -- 172.21.15.107:0/38166599 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813800ddd0 con 0x564fe0120330 2024-05-09T08:44:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.096+0000 7f81267fc640 1 -- 172.21.15.107:0/38166599 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f813801c070 con 0x564fe0120330 2024-05-09T08:44:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.096+0000 7f81267fc640 1 -- 172.21.15.107:0/38166599 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81380165e0 con 0x564fe0120330 2024-05-09T08:44:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.096+0000 7f813d7fa640 1 -- 172.21.15.107:0/1219226859 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f81340f02e0 con 0x7f8140103400 2024-05-09T08:44:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.096+0000 7f813ffff640 1 --2- 172.21.15.107:0/1219226859 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f81280a2e20 0x7f81280a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.097+0000 7f81491be740 1 -- 172.21.15.107:0/38166599 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120330 msgr2=0x564fe0120710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:14.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.097+0000 7f813ffff640 1 --2- 172.21.15.107:0/1219226859 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f81280a2e20 0x7f81280a52e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8130009da0 tx=0x7f8130005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:14.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.097+0000 7f81491be740 1 --2- 172.21.15.107:0/38166599 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120330 0x564fe0120710 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f813800e870 tx=0x7f81380095c0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.097+0000 7f81491be740 1 -- 172.21.15.107:0/38166599 shutdown_connections 2024-05-09T08:44:14.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.097+0000 7f81491be740 1 --2- 172.21.15.107:0/38166599 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe0131e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.097+0000 7f81491be740 1 --2- 172.21.15.107:0/38166599 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120c50 0x564fe012f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.097+0000 7f81491be740 1 --2- 172.21.15.107:0/38166599 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120330 0x564fe0120710 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.097+0000 7f81491be740 1 -- 172.21.15.107:0/38166599 >> 172.21.15.107:0/38166599 conn(0x564fe011c960 msgr2=0x564fe011cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.097+0000 7f81491be740 1 -- 172.21.15.107:0/38166599 shutdown_connections 2024-05-09T08:44:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.097+0000 7f81491be740 1 -- 172.21.15.107:0/38166599 wait complete. 2024-05-09T08:44:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.097+0000 7f81491be740 1 -- start start 2024-05-09T08:44:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.098+0000 7f81491be740 1 Processor -- start 2024-05-09T08:44:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.098+0000 7f81491be740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120330 0x564fe012ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.098+0000 7f81491be740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120c50 0x564fe012cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.098+0000 7f81491be740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe012d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.098+0000 7f81491be740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564fe0132670 con 0x564fe0120330 2024-05-09T08:44:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.098+0000 7f81491be740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564fe0040790 con 0x564fe012fa40 2024-05-09T08:44:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.098+0000 7f81491be740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564fe01327f0 con 0x564fe0120c50 2024-05-09T08:44:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.098+0000 7f81491be740 10 client.0 _ll_register_callbacks cb 0x564fe007ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.098+0000 7f8144a50640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe012d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.098+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120c50 0x564fe012cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.098+0000 7f8144a50640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe012d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:38544/0 (socket says 172.21.15.107:38544) 2024-05-09T08:44:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.098+0000 7f8144a50640 1 -- 172.21.15.107:0/3390615365 learned_addr learned my addr 172.21.15.107:0/3390615365 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.098+0000 7f813ffff640 1 --2- 172.21.15.107:0/3390615365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120330 0x564fe012ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.099+0000 7f813f7fe640 1 -- 172.21.15.107:0/3390615365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 msgr2=0x564fe012d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.099+0000 7f813f7fe640 1 --2- 172.21.15.107:0/3390615365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe012d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.099+0000 7f813f7fe640 1 -- 172.21.15.107:0/3390615365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120330 msgr2=0x564fe012ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.099+0000 7f813f7fe640 1 --2- 172.21.15.107:0/3390615365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120330 0x564fe012ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.099+0000 7f813f7fe640 1 -- 172.21.15.107:0/3390615365 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fdfd99210 con 0x564fe0120c50 2024-05-09T08:44:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.099+0000 7f813f7fe640 1 --2- 172.21.15.107:0/3390615365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120c50 0x564fe012cf60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f81340ed620 tx=0x7f81340bbd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d7699a0f4e7947a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.099+0000 7f811f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:44:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.099+0000 7f811f7fe640 1 -- 172.21.15.107:0/3390615365 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81340f0110 con 0x564fe0120c50 2024-05-09T08:44:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.099+0000 7f811f7fe640 1 -- 172.21.15.107:0/3390615365 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f8134004cc0 con 0x564fe0120c50 2024-05-09T08:44:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.099+0000 7f81491be740 1 -- 172.21.15.107:0/3390615365 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f813801c070 con 0x564fe0120c50 2024-05-09T08:44:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.100+0000 7f811f7fe640 1 -- 172.21.15.107:0/3390615365 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f81340c07a0 con 0x564fe0120c50 2024-05-09T08:44:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.100+0000 7f81491be740 10 client.27847 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.100+0000 7f81491be740 1 -- 172.21.15.107:0/3390615365 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564fdfe65620 con 0x564fe0120c50 2024-05-09T08:44:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.100+0000 7f811f7fe640 1 -- 172.21.15.107:0/3390615365 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f813801c070 con 0x564fe0120c50 2024-05-09T08:44:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.101+0000 7f811f7fe640 1 -- 172.21.15.107:0/3390615365 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f813404c030 con 0x564fe0120c50 2024-05-09T08:44:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.101+0000 7f81491be740 10 client.27847 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.101+0000 7f81491be740 10 client.27847 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.101+0000 7f81491be740 20 client.27847 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.101+0000 7f81491be740 1 --2- 172.21.15.107:0/3390615365 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564fe013f860 0x564fe0141c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.104 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.101+0000 7f81491be740 4 client.27847 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:44:14.105 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.101+0000 7f81491be740 1 -- 172.21.15.107:0/3390615365 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564fe0092a70 con 0x564fe013f860 2024-05-09T08:44:14.105 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.101+0000 7f813ffff640 1 --2- 172.21.15.107:0/3390615365 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564fe013f860 0x564fe0141c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.106 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.101+0000 7f813ffff640 1 --2- 172.21.15.107:0/3390615365 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564fe013f860 0x564fe0141c80 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e19e2c19284d43ca server_cookie=25eb886afce4da20 in_seq=0 out_seq=0 2024-05-09T08:44:14.106 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.101+0000 7f811f7fe640 10 client.27847 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:14.106 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.102+0000 7f811f7fe640 1 -- 172.21.15.107:0/3390615365 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564fe0092a70 con 0x564fe013f860 2024-05-09T08:44:14.106 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.102+0000 7f811f7fe640 10 client.27847 handle_command_reply: tid=0 2024-05-09T08:44:14.106 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.103+0000 7f81491be740 1 client.27847 shutdown 2024-05-09T08:44:14.106 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.103+0000 7f81491be740 1 -- 172.21.15.107:0/3390615365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120c50 msgr2=0x564fe012cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:14.106 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.103+0000 7f81491be740 1 --2- 172.21.15.107:0/3390615365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120c50 0x564fe012cf60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f81340ed620 tx=0x7f81340bbd50 comp rx=0 tx=0).stop 2024-05-09T08:44:14.106 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.103+0000 7f81491be740 1 -- 172.21.15.107:0/3390615365 shutdown_connections 2024-05-09T08:44:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.103+0000 7f81491be740 1 --2- 172.21.15.107:0/3390615365 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564fe013f860 0x564fe0141c80 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.103+0000 7f81491be740 1 --2- 172.21.15.107:0/3390615365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe012d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.103+0000 7f81491be740 1 --2- 172.21.15.107:0/3390615365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120c50 0x564fe012cf60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.103+0000 7f81491be740 1 --2- 172.21.15.107:0/3390615365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120330 0x564fe012ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.103+0000 7f81491be740 1 -- 172.21.15.107:0/3390615365 >> 172.21.15.107:0/3390615365 conn(0x564fe011c960 msgr2=0x564fe0122770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.103+0000 7f811f7fe640 0 client.27847 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.103+0000 7f81491be740 1 -- 172.21.15.107:0/3390615365 shutdown_connections 2024-05-09T08:44:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.103+0000 7f81491be740 1 -- 172.21.15.107:0/3390615365 wait complete. 2024-05-09T08:44:14.107 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.103+0000 7f81491be740 20 client.27847 trim_cache size 0 max 16384 2024-05-09T08:44:14.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.122+0000 7f81491be740 1 Processor -- start 2024-05-09T08:44:14.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.122+0000 7f81491be740 1 -- start start 2024-05-09T08:44:14.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.122+0000 7f81491be740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120330 0x564fe013b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.122 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.122+0000 7f81491be740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120c50 0x564fe012d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.122+0000 7f81491be740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe0147570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.122+0000 7f81491be740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f81401af270 con 0x564fe0120330 2024-05-09T08:44:14.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.122+0000 7f81491be740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f81401acb40 con 0x564fe012fa40 2024-05-09T08:44:14.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.122+0000 7f813ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120330 0x564fe013b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.122+0000 7f81491be740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f81401aa9b0 con 0x564fe0120c50 2024-05-09T08:44:14.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.122+0000 7f813ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120330 0x564fe013b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58798/0 (socket says 172.21.15.107:58798) 2024-05-09T08:44:14.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.122+0000 7f813ffff640 1 -- 172.21.15.107:0/521819901 learned_addr learned my addr 172.21.15.107:0/521819901 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:14.125 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.122+0000 7f8144a50640 1 --2- 172.21.15.107:0/521819901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe0147570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.125 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.123+0000 7f813f7fe640 1 --2- 172.21.15.107:0/521819901 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120c50 0x564fe012d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.125 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.123+0000 7f8144a50640 1 -- 172.21.15.107:0/521819901 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120c50 msgr2=0x564fe012d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:14.125 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.123+0000 7f8144a50640 1 --2- 172.21.15.107:0/521819901 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120c50 0x564fe012d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.125 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.123+0000 7f8144a50640 1 -- 172.21.15.107:0/521819901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120330 msgr2=0x564fe013b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:14.125 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.123+0000 7f8144a50640 1 --2- 172.21.15.107:0/521819901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120330 0x564fe013b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.125 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.123+0000 7f8144a50640 1 -- 172.21.15.107:0/521819901 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fe0132550 con 0x564fe012fa40 2024-05-09T08:44:14.125 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.123+0000 7f813f7fe640 1 --2- 172.21.15.107:0/521819901 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120c50 0x564fe012d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:44:14.125 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.123+0000 7f813ffff640 1 --2- 172.21.15.107:0/521819901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120330 0x564fe013b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:44:14.125 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.123+0000 7f8144a50640 1 --2- 172.21.15.107:0/521819901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe0147570 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8138005fd0 tx=0x7f813800e600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f88be6c16f8f2dd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:14.126 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.123+0000 7f8125ffb640 1 -- 172.21.15.107:0/521819901 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8138004a00 con 0x564fe012fa40 2024-05-09T08:44:14.126 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.123+0000 7f8125ffb640 1 -- 172.21.15.107:0/521819901 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f813801c040 con 0x564fe012fa40 2024-05-09T08:44:14.126 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.123+0000 7f8125ffb640 1 -- 172.21.15.107:0/521819901 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8138005040 con 0x564fe012fa40 2024-05-09T08:44:14.126 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.124+0000 7f81491be740 1 -- 172.21.15.107:0/521819901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 msgr2=0x564fe0147570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:14.126 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.124+0000 7f81491be740 1 --2- 172.21.15.107:0/521819901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe0147570 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8138005fd0 tx=0x7f813800e600 comp rx=0 tx=0).stop 2024-05-09T08:44:14.126 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.124+0000 7f81491be740 1 -- 172.21.15.107:0/521819901 shutdown_connections 2024-05-09T08:44:14.126 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.124+0000 7f81491be740 1 --2- 172.21.15.107:0/521819901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe0147570 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.126 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.124+0000 7f81491be740 1 --2- 172.21.15.107:0/521819901 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120c50 0x564fe012d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.126 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.124+0000 7f81491be740 1 --2- 172.21.15.107:0/521819901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120330 0x564fe013b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.126 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.124+0000 7f81491be740 1 -- 172.21.15.107:0/521819901 >> 172.21.15.107:0/521819901 conn(0x564fe011c960 msgr2=0x564fe011ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:14.126 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.124+0000 7f81491be740 1 -- 172.21.15.107:0/521819901 shutdown_connections 2024-05-09T08:44:14.126 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.124+0000 7f81491be740 1 -- 172.21.15.107:0/521819901 wait complete. 2024-05-09T08:44:14.127 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.125+0000 7f81491be740 1 -- start start 2024-05-09T08:44:14.127 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.125+0000 7f81491be740 1 Processor -- start 2024-05-09T08:44:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.125+0000 7f81491be740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120330 0x564fe0137ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.125+0000 7f81491be740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120c50 0x564fe0129410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.125+0000 7f81491be740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe0129950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.125+0000 7f81491be740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f81401ab000 con 0x564fe0120c50 2024-05-09T08:44:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.125+0000 7f81491be740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f81401ac470 con 0x564fe012fa40 2024-05-09T08:44:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.125+0000 7f81491be740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f81401ab6d0 con 0x564fe0120330 2024-05-09T08:44:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.125+0000 7f81491be740 10 client.0 _ll_register_callbacks cb 0x7f8134005ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.125+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120c50 0x564fe0129410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.125+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120c50 0x564fe0129410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58810/0 (socket says 172.21.15.107:58810) 2024-05-09T08:44:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.125+0000 7f813f7fe640 1 -- 172.21.15.107:0/3960842104 learned_addr learned my addr 172.21.15.107:0/3960842104 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.125+0000 7f813ffff640 1 --2- 172.21.15.107:0/3960842104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120330 0x564fe0137ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.126+0000 7f8144a50640 1 --2- 172.21.15.107:0/3960842104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe0129950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.126+0000 7f813ffff640 1 -- 172.21.15.107:0/3960842104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 msgr2=0x564fe0129950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.126+0000 7f813ffff640 1 --2- 172.21.15.107:0/3960842104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe0129950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.126+0000 7f813ffff640 1 -- 172.21.15.107:0/3960842104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120c50 msgr2=0x564fe0129410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.126+0000 7f813ffff640 1 --2- 172.21.15.107:0/3960842104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120c50 0x564fe0129410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.126+0000 7f813ffff640 1 -- 172.21.15.107:0/3960842104 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fe012a180 con 0x564fe0120330 2024-05-09T08:44:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.126+0000 7f813f7fe640 1 --2- 172.21.15.107:0/3960842104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120c50 0x564fe0129410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:44:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.126+0000 7f813ffff640 1 --2- 172.21.15.107:0/3960842104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120330 0x564fe0137ed0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8130009b80 tx=0x7f81300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57c1af24fa9df04f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.126+0000 7f811f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:44:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.126+0000 7f811f7fe640 1 -- 172.21.15.107:0/3960842104 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8130004460 con 0x564fe0120330 2024-05-09T08:44:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.126+0000 7f811f7fe640 1 -- 172.21.15.107:0/3960842104 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f813002b380 con 0x564fe0120330 2024-05-09T08:44:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.127+0000 7f81491be740 1 -- 172.21.15.107:0/3960842104 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f813801c040 con 0x564fe0120330 2024-05-09T08:44:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.127+0000 7f811f7fe640 1 -- 172.21.15.107:0/3960842104 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f813003b050 con 0x564fe0120330 2024-05-09T08:44:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.127+0000 7f81491be740 10 client.27850 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.127+0000 7f81491be740 1 -- 172.21.15.107:0/3960842104 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564fe01a5dc0 con 0x564fe0120330 2024-05-09T08:44:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.127+0000 7f811f7fe640 1 -- 172.21.15.107:0/3960842104 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8130005020 con 0x564fe0120330 2024-05-09T08:44:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.127+0000 7f811f7fe640 1 -- 172.21.15.107:0/3960842104 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8130004960 con 0x564fe0120330 2024-05-09T08:44:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.127+0000 7f81491be740 10 client.27850 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.127+0000 7f81491be740 10 client.27850 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.127+0000 7f81491be740 20 client.27850 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.128+0000 7f81491be740 1 --2- 172.21.15.107:0/3960842104 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564fe01a6570 0x564fe01a8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.128+0000 7f81491be740 4 client.27850 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:44:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.128+0000 7f81491be740 1 -- 172.21.15.107:0/3960842104 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f813801c040 con 0x564fe01a6570 2024-05-09T08:44:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.128+0000 7f813f7fe640 1 --2- 172.21.15.107:0/3960842104 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564fe01a6570 0x564fe01a8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.128+0000 7f813f7fe640 1 --2- 172.21.15.107:0/3960842104 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564fe01a6570 0x564fe01a8990 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5acaec561599f83c server_cookie=862a43ca68ea8d33 in_seq=0 out_seq=0 2024-05-09T08:44:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.128+0000 7f811f7fe640 10 client.27850 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:14.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.164+0000 7f811f7fe640 1 -- 172.21.15.107:0/3960842104 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f813801c040 con 0x564fe01a6570 2024-05-09T08:44:14.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.164+0000 7f811f7fe640 10 client.27850 handle_command_reply: tid=0 2024-05-09T08:44:14.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.164+0000 7f81491be740 1 client.27850 shutdown 2024-05-09T08:44:14.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.164+0000 7f81491be740 1 -- 172.21.15.107:0/3960842104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120330 msgr2=0x564fe0137ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:14.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.164+0000 7f81491be740 1 --2- 172.21.15.107:0/3960842104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120330 0x564fe0137ed0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8130009b80 tx=0x7f81300046c0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.164+0000 7f81491be740 1 -- 172.21.15.107:0/3960842104 shutdown_connections 2024-05-09T08:44:14.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.165+0000 7f81491be740 1 --2- 172.21.15.107:0/3960842104 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564fe01a6570 0x564fe01a8990 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.165+0000 7f81491be740 1 --2- 172.21.15.107:0/3960842104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564fe012fa40 0x564fe0129950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.165+0000 7f81491be740 1 --2- 172.21.15.107:0/3960842104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564fe0120c50 0x564fe0129410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.165+0000 7f81491be740 1 --2- 172.21.15.107:0/3960842104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564fe0120330 0x564fe0137ed0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.165+0000 7f81491be740 1 -- 172.21.15.107:0/3960842104 >> 172.21.15.107:0/3960842104 conn(0x564fe011c960 msgr2=0x564fe0121ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:14.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.165+0000 7f811f7fe640 0 client.27850 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:14.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.165+0000 7f81491be740 1 -- 172.21.15.107:0/3960842104 shutdown_connections 2024-05-09T08:44:14.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.165+0000 7f81491be740 1 -- 172.21.15.107:0/3960842104 wait complete. 2024-05-09T08:44:14.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.165+0000 7f81491be740 20 client.27850 trim_cache size 0 max 16384 2024-05-09T08:44:14.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:44:14.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:44:14.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:44:14.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:44:14.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.165+0000 7f81464b3640 1 -- 172.21.15.107:0/1219226859 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f81280a2e20 msgr2=0x7f81280a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:14.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.165+0000 7f81464b3640 1 --2- 172.21.15.107:0/1219226859 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f81280a2e20 0x7f81280a52e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8130009da0 tx=0x7f8130005f90 comp rx=0 tx=0).stop 2024-05-09T08:44:14.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.165+0000 7f81464b3640 1 -- 172.21.15.107:0/1219226859 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140103400 msgr2=0x7f81401984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:14.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.166+0000 7f81464b3640 1 --2- 172.21.15.107:0/1219226859 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140103400 0x7f81401984e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8134009980 tx=0x7f8134030ef0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.166+0000 7f81464b3640 1 -- 172.21.15.107:0/1219226859 shutdown_connections 2024-05-09T08:44:14.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.166+0000 7f81464b3640 1 --2- 172.21.15.107:0/1219226859 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f81280a2e20 0x7f81280a52e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.166+0000 7f81464b3640 1 --2- 172.21.15.107:0/1219226859 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f814010be20 0x7f814019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.166+0000 7f81464b3640 1 --2- 172.21.15.107:0/1219226859 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140103400 0x7f81401984e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.166+0000 7f81464b3640 1 --2- 172.21.15.107:0/1219226859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8140102ae0 0x7f8140197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:14.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.166+0000 7f81464b3640 1 -- 172.21.15.107:0/1219226859 >> 172.21.15.107:0/1219226859 conn(0x7f81400778b0 msgr2=0x7f81400fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:14.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.166+0000 7f81464b3640 1 -- 172.21.15.107:0/1219226859 shutdown_connections 2024-05-09T08:44:14.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:14.166+0000 7f81464b3640 1 -- 172.21.15.107:0/1219226859 wait complete. 2024-05-09T08:44:15.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:14 smithi107 ceph-mon[30122]: pgmap v336: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 12 MiB/s wr, 3.02k op/s 2024-05-09T08:44:15.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:14 smithi157 ceph-mon[33614]: pgmap v336: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 12 MiB/s wr, 3.02k op/s 2024-05-09T08:44:15.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:14 smithi110 ceph-mon[33622]: pgmap v336: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 12 MiB/s wr, 3.02k op/s 2024-05-09T08:44:17.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:16 smithi107 ceph-mon[30122]: pgmap v337: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 9.3 MiB/s wr, 2.36k op/s 2024-05-09T08:44:17.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:16 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:44:17.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:16 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:44:17.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:16 smithi157 ceph-mon[33614]: pgmap v337: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 9.3 MiB/s wr, 2.36k op/s 2024-05-09T08:44:17.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:16 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:44:17.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:16 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:44:17.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:16 smithi110 ceph-mon[33622]: pgmap v337: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 9.3 MiB/s wr, 2.36k op/s 2024-05-09T08:44:17.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:16 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:44:17.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:16 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:44:18.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:17 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:44:18.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:17 smithi107 ceph-mon[30122]: osdmap e102: 12 total, 12 up, 12 in 2024-05-09T08:44:18.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:17 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:44:18.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:17 smithi157 ceph-mon[33614]: osdmap e102: 12 total, 12 up, 12 in 2024-05-09T08:44:18.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:17 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:44:18.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:17 smithi110 ceph-mon[33622]: osdmap e102: 12 total, 12 up, 12 in 2024-05-09T08:44:19.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:18 smithi107 ceph-mon[30122]: pgmap v339: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.5 MiB/s wr, 2.13k op/s 2024-05-09T08:44:19.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:18 smithi107 ceph-mon[30122]: osdmap e103: 12 total, 12 up, 12 in 2024-05-09T08:44:19.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:18 smithi157 ceph-mon[33614]: pgmap v339: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.5 MiB/s wr, 2.13k op/s 2024-05-09T08:44:19.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:18 smithi157 ceph-mon[33614]: osdmap e103: 12 total, 12 up, 12 in 2024-05-09T08:44:19.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:18 smithi110 ceph-mon[33622]: pgmap v339: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.5 MiB/s wr, 2.13k op/s 2024-05-09T08:44:19.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:18 smithi110 ceph-mon[33622]: osdmap e103: 12 total, 12 up, 12 in 2024-05-09T08:44:20.513 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:44:20.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:44:20.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.756+0000 7f042cb66640 1 -- 172.21.15.107:0/3393152854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04200aa2a0 msgr2=0x7f04200b1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:20.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.756+0000 7f042cb66640 1 --2- 172.21.15.107:0/3393152854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04200aa2a0 0x7f04200b1e70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0428067530 tx=0x7f042808dc50 comp rx=0 tx=0).stop 2024-05-09T08:44:20.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.756+0000 7f042cb66640 1 -- 172.21.15.107:0/3393152854 shutdown_connections 2024-05-09T08:44:20.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.756+0000 7f042cb66640 1 --2- 172.21.15.107:0/3393152854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04200aa2a0 0x7f04200b1e70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.756+0000 7f042cb66640 1 --2- 172.21.15.107:0/3393152854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04200a5b80 0x7f04200a9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.756+0000 7f042cb66640 1 --2- 172.21.15.107:0/3393152854 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04200a51d0 0x7f04200a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.756+0000 7f042cb66640 1 -- 172.21.15.107:0/3393152854 >> 172.21.15.107:0/3393152854 conn(0x7f042001aa70 msgr2=0x7f042001ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:20.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.761+0000 7f042cb66640 1 -- 172.21.15.107:0/3393152854 shutdown_connections 2024-05-09T08:44:20.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.767+0000 7f042cb66640 1 -- 172.21.15.107:0/3393152854 wait complete. 2024-05-09T08:44:20.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.768+0000 7f042cb66640 1 Processor -- start 2024-05-09T08:44:20.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.768+0000 7f042cb66640 1 -- start start 2024-05-09T08:44:20.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.769+0000 7f042cb66640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04200a51d0 0x7f0420142190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.769 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.769+0000 7f042cb66640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04200aa2a0 0x7f042013b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.769+0000 7f042cb66640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f042013b750 0x7f042013bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.769+0000 7f042cb66640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f04200b4d40 con 0x7f042013b750 2024-05-09T08:44:20.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.769+0000 7f042cb66640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f04200b4bc0 con 0x7f04200a51d0 2024-05-09T08:44:20.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.769+0000 7f042cb66640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f04200b4ec0 con 0x7f04200aa2a0 2024-05-09T08:44:20.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.771+0000 7f0426ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04200aa2a0 0x7f042013b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:20.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.773+0000 7f04277fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04200a51d0 0x7f0420142190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:20.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.773+0000 7f0426ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04200aa2a0 0x7f042013b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:57282/0 (socket says 172.21.15.107:57282) 2024-05-09T08:44:20.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.773+0000 7f0426ffd640 1 -- 172.21.15.107:0/2559817691 learned_addr learned my addr 172.21.15.107:0/2559817691 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:20.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.775+0000 7f0427fff640 1 --2- 172.21.15.107:0/2559817691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f042013b750 0x7f042013bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:20.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.777+0000 7f0427fff640 1 -- 172.21.15.107:0/2559817691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04200aa2a0 msgr2=0x7f042013b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:20.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.777+0000 7f0427fff640 1 --2- 172.21.15.107:0/2559817691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04200aa2a0 0x7f042013b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.777+0000 7f0427fff640 1 -- 172.21.15.107:0/2559817691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04200a51d0 msgr2=0x7f0420142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:20.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.777+0000 7f0427fff640 1 --2- 172.21.15.107:0/2559817691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04200a51d0 0x7f0420142190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.777+0000 7f0427fff640 1 -- 172.21.15.107:0/2559817691 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f042013fd30 con 0x7f042013b750 2024-05-09T08:44:20.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.777+0000 7f0427fff640 1 --2- 172.21.15.107:0/2559817691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f042013b750 0x7f042013bc00 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f0428050860 tx=0x7f04280976a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:20.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.778+0000 7f0424ff9640 1 -- 172.21.15.107:0/2559817691 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0428097a60 con 0x7f042013b750 2024-05-09T08:44:20.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.778+0000 7f0424ff9640 1 -- 172.21.15.107:0/2559817691 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0428097c00 con 0x7f042013b750 2024-05-09T08:44:20.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.778+0000 7f042cb66640 1 -- 172.21.15.107:0/2559817691 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f042013ffc0 con 0x7f042013b750 2024-05-09T08:44:20.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.778+0000 7f042cb66640 1 -- 172.21.15.107:0/2559817691 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0420140480 con 0x7f042013b750 2024-05-09T08:44:20.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.783+0000 7f0424ff9640 1 -- 172.21.15.107:0/2559817691 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0428096670 con 0x7f042013b750 2024-05-09T08:44:20.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.783+0000 7f0430bb5740 1 Processor -- start 2024-05-09T08:44:20.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.797+0000 7f0430bb5740 1 -- start start 2024-05-09T08:44:20.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.797+0000 7f0430bb5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d20470 0x55d577d20850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.797+0000 7f0430bb5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d20d90 0x55d577d2af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.797+0000 7f0430bb5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d2b460 0x55d577d2d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.797+0000 7f0430bb5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d577c85d00 con 0x55d577d2b460 2024-05-09T08:44:20.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.797+0000 7f0430bb5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d577c2d4c0 con 0x55d577d20470 2024-05-09T08:44:20.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.797+0000 7f0430bb5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d577c81cc0 con 0x55d577d20d90 2024-05-09T08:44:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.798+0000 7f0424ff9640 1 -- 172.21.15.107:0/2559817691 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f04280968d0 con 0x7f042013b750 2024-05-09T08:44:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.799+0000 7f0424ff9640 1 --2- 172.21.15.107:0/2559817691 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f04180a2e90 0x7f04180a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.799+0000 7f0424ff9640 1 -- 172.21.15.107:0/2559817691 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7f0428152f50 con 0x7f042013b750 2024-05-09T08:44:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.800+0000 7f0426ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d20470 0x55d577d20850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.800+0000 7f0426ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d20470 0x55d577d20850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:57632/0 (socket says 172.21.15.107:57632) 2024-05-09T08:44:20.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.800+0000 7f0426ffd640 1 -- 172.21.15.107:0/856297230 learned_addr learned my addr 172.21.15.107:0/856297230 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:20.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.800+0000 7f0427fff640 1 --2- 172.21.15.107:0/2559817691 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f04180a2e90 0x7f04180a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:20.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.801+0000 7f0427fff640 1 --2- 172.21.15.107:0/2559817691 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f04180a2e90 0x7f04180a5350 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0428150390 tx=0x7f0428090040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:20.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.802+0000 7f04277fe640 1 --2- 172.21.15.107:0/856297230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d20d90 0x55d577d2af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:20.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.803+0000 7f0426ffd640 1 --2- 172.21.15.107:0/856297230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d2b460 0x55d577d2d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:20.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.806+0000 7f0426ffd640 1 -- 172.21.15.107:0/856297230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d20d90 msgr2=0x55d577d2af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:20.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.806+0000 7f0426ffd640 1 --2- 172.21.15.107:0/856297230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d20d90 0x55d577d2af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.806+0000 7f0426ffd640 1 -- 172.21.15.107:0/856297230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d2b460 msgr2=0x55d577d2d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:20.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.806+0000 7f0426ffd640 1 --2- 172.21.15.107:0/856297230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d2b460 0x55d577d2d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.806+0000 7f0426ffd640 1 -- 172.21.15.107:0/856297230 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d577a637d0 con 0x55d577d20470 2024-05-09T08:44:20.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.813+0000 7f0426ffd640 1 --2- 172.21.15.107:0/856297230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d20470 0x55d577d20850 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0414017870 tx=0x7f041400a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9d8ef173b11f5b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:20.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.822+0000 7f0411ffb640 1 -- 172.21.15.107:0/856297230 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041401e020 con 0x55d577d20470 2024-05-09T08:44:20.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.822+0000 7f0411ffb640 1 -- 172.21.15.107:0/856297230 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f041401a430 con 0x55d577d20470 2024-05-09T08:44:20.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.822+0000 7f0411ffb640 1 -- 172.21.15.107:0/856297230 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0414019680 con 0x55d577d20470 2024-05-09T08:44:20.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.824+0000 7f0430bb5740 1 -- 172.21.15.107:0/856297230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d20470 msgr2=0x55d577d20850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:20.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.824+0000 7f0430bb5740 1 --2- 172.21.15.107:0/856297230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d20470 0x55d577d20850 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0414017870 tx=0x7f041400a760 comp rx=0 tx=0).stop 2024-05-09T08:44:20.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.832+0000 7f0430bb5740 1 -- 172.21.15.107:0/856297230 shutdown_connections 2024-05-09T08:44:20.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.832+0000 7f0430bb5740 1 --2- 172.21.15.107:0/856297230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d2b460 0x55d577d2d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.832+0000 7f0430bb5740 1 --2- 172.21.15.107:0/856297230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d20d90 0x55d577d2af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.832+0000 7f0430bb5740 1 --2- 172.21.15.107:0/856297230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d20470 0x55d577d20850 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0414017870 tx=0x7f041400a760 comp rx=0 tx=0).stop 2024-05-09T08:44:20.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.832+0000 7f0430bb5740 1 -- 172.21.15.107:0/856297230 >> 172.21.15.107:0/856297230 conn(0x55d577d1c790 msgr2=0x55d577d1cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:20.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.835+0000 7f0430bb5740 1 -- 172.21.15.107:0/856297230 shutdown_connections 2024-05-09T08:44:20.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.835+0000 7f0430bb5740 1 -- 172.21.15.107:0/856297230 wait complete. 2024-05-09T08:44:20.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.836+0000 7f0430bb5740 1 -- start start 2024-05-09T08:44:20.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.836+0000 7f0430bb5740 1 Processor -- start 2024-05-09T08:44:20.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.837+0000 7f0430bb5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d20d90 0x55d577d286c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.837+0000 7f0430bb5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d2b460 0x55d577d28c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.837+0000 7f0430bb5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d292a0 0x55d577d376c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.837+0000 7f0430bb5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d577d2e250 con 0x55d577d292a0 2024-05-09T08:44:20.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.837+0000 7f0430bb5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d577c40b90 con 0x55d577d2b460 2024-05-09T08:44:20.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.837+0000 7f04277fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d2b460 0x55d577d28c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:20.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.838+0000 7f04277fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d2b460 0x55d577d28c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:57646/0 (socket says 172.21.15.107:57646) 2024-05-09T08:44:20.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.838+0000 7f04277fe640 1 -- 172.21.15.107:0/74699446 learned_addr learned my addr 172.21.15.107:0/74699446 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:20.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.838+0000 7f0430bb5740 1 -- 172.21.15.107:0/74699446 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d577d2e3d0 con 0x55d577d20d90 2024-05-09T08:44:20.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.838+0000 7f0430bb5740 10 client.0 _ll_register_callbacks cb 0x55d577c7cda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:20.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.838+0000 7f04277fe640 1 -- 172.21.15.107:0/74699446 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d20d90 msgr2=0x55d577d286c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:44:20.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.838+0000 7f04277fe640 1 --2- 172.21.15.107:0/74699446 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d20d90 0x55d577d286c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.838+0000 7f04277fe640 1 -- 172.21.15.107:0/74699446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d292a0 msgr2=0x55d577d376c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:44:20.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.838+0000 7f04277fe640 1 --2- 172.21.15.107:0/74699446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d292a0 0x55d577d376c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.839+0000 7f04277fe640 1 -- 172.21.15.107:0/74699446 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5779993f0 con 0x55d577d2b460 2024-05-09T08:44:20.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.844+0000 7f04277fe640 1 --2- 172.21.15.107:0/74699446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d2b460 0x55d577d28c00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f041c0095f0 tx=0x7f041c009ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6f15b384374be9e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:20.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.844+0000 7f0406ffd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:44:20.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.848+0000 7f0406ffd640 1 -- 172.21.15.107:0/74699446 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c01d060 con 0x55d577d2b460 2024-05-09T08:44:20.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.848+0000 7f0406ffd640 1 -- 172.21.15.107:0/74699446 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7f041c004230 con 0x55d577d2b460 2024-05-09T08:44:20.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.849+0000 7f0430bb5740 1 -- 172.21.15.107:0/74699446 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f041401a430 con 0x55d577d2b460 2024-05-09T08:44:20.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.854+0000 7f0406ffd640 1 -- 172.21.15.107:0/74699446 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f041c004430 con 0x55d577d2b460 2024-05-09T08:44:20.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.854+0000 7f0430bb5740 10 client.27635 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:20.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.854+0000 7f0430bb5740 1 -- 172.21.15.107:0/74699446 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d5779590f0 con 0x55d577d2b460 2024-05-09T08:44:20.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.859+0000 7f0406ffd640 1 -- 172.21.15.107:0/74699446 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f041c003aa0 con 0x55d577d2b460 2024-05-09T08:44:20.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.859+0000 7f0406ffd640 1 -- 172.21.15.107:0/74699446 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f041c00cc10 con 0x55d577d2b460 2024-05-09T08:44:20.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.859+0000 7f0430bb5740 10 client.27635 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:20.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.859+0000 7f0430bb5740 10 client.27635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:20.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.859+0000 7f0430bb5740 20 client.27635 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:20.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.859+0000 7f0430bb5740 1 --2- 172.21.15.107:0/74699446 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d577d3c2b0 0x55d577d3e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.859+0000 7f0430bb5740 4 client.27635 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:44:20.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.859+0000 7f0430bb5740 1 -- 172.21.15.107:0/74699446 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f041401a430 con 0x55d577d3c2b0 2024-05-09T08:44:20.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.861+0000 7f0426ffd640 1 --2- 172.21.15.107:0/74699446 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d577d3c2b0 0x55d577d3e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:20.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.861+0000 7f0426ffd640 1 --2- 172.21.15.107:0/74699446 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d577d3c2b0 0x55d577d3e770 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d40e646ea990fd72 server_cookie=3d041a10bc86fdcd in_seq=0 out_seq=0 2024-05-09T08:44:20.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.863+0000 7f0406ffd640 10 client.27635 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:20.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.865+0000 7f0406ffd640 1 -- 172.21.15.107:0/74699446 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f041401a430 con 0x55d577d3c2b0 2024-05-09T08:44:20.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.865+0000 7f0406ffd640 10 client.27635 handle_command_reply: tid=0 2024-05-09T08:44:20.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.865+0000 7f0430bb5740 1 client.27635 shutdown 2024-05-09T08:44:20.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.865+0000 7f0430bb5740 1 -- 172.21.15.107:0/74699446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d2b460 msgr2=0x55d577d28c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:20.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.865+0000 7f0430bb5740 1 --2- 172.21.15.107:0/74699446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d2b460 0x55d577d28c00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f041c0095f0 tx=0x7f041c009ab0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.865+0000 7f0430bb5740 1 -- 172.21.15.107:0/74699446 shutdown_connections 2024-05-09T08:44:20.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.865+0000 7f0430bb5740 1 --2- 172.21.15.107:0/74699446 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d577d3c2b0 0x55d577d3e770 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.866+0000 7f0430bb5740 1 --2- 172.21.15.107:0/74699446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d292a0 0x55d577d376c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.866+0000 7f0430bb5740 1 --2- 172.21.15.107:0/74699446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d2b460 0x55d577d28c00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.866+0000 7f0430bb5740 1 --2- 172.21.15.107:0/74699446 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d20d90 0x55d577d286c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.866+0000 7f0430bb5740 1 -- 172.21.15.107:0/74699446 >> 172.21.15.107:0/74699446 conn(0x55d577d1c790 msgr2=0x55d577d22590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:20.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.866+0000 7f0406ffd640 0 client.27635 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.870+0000 7f0430bb5740 1 -- 172.21.15.107:0/74699446 shutdown_connections 2024-05-09T08:44:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.870+0000 7f0430bb5740 1 -- 172.21.15.107:0/74699446 wait complete. 2024-05-09T08:44:20.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.870+0000 7f0430bb5740 20 client.27635 trim_cache size 0 max 16384 2024-05-09T08:44:20.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.923+0000 7f0430bb5740 1 Processor -- start 2024-05-09T08:44:20.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.923+0000 7f0430bb5740 1 -- start start 2024-05-09T08:44:20.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.923+0000 7f0430bb5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d20d90 0x55d577d2af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.923+0000 7f0430bb5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d292a0 0x55d577d178c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.923+0000 7f0430bb5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d2b460 0x55d577d1b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.923+0000 7f0430bb5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0420165e00 con 0x55d577d20d90 2024-05-09T08:44:20.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.923+0000 7f0430bb5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f04201636d0 con 0x55d577d292a0 2024-05-09T08:44:20.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.923+0000 7f0430bb5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0420161540 con 0x55d577d2b460 2024-05-09T08:44:20.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.924+0000 7f0426ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d2b460 0x55d577d1b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:20.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.924+0000 7f0426ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d2b460 0x55d577d1b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:57308/0 (socket says 172.21.15.107:57308) 2024-05-09T08:44:20.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.924+0000 7f0426ffd640 1 -- 172.21.15.107:0/3041062427 learned_addr learned my addr 172.21.15.107:0/3041062427 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:20.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.924+0000 7f0426ffd640 1 --2- 172.21.15.107:0/3041062427 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d20d90 0x55d577d2af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:20.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.924+0000 7f0426ffd640 1 -- 172.21.15.107:0/3041062427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d292a0 msgr2=0x55d577d178c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:44:20.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.924+0000 7f0426ffd640 1 --2- 172.21.15.107:0/3041062427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d292a0 0x55d577d178c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.924+0000 7f0426ffd640 1 -- 172.21.15.107:0/3041062427 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d20d90 msgr2=0x55d577d2af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:20.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.924+0000 7f0426ffd640 1 --2- 172.21.15.107:0/3041062427 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d20d90 0x55d577d2af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.924+0000 7f0426ffd640 1 -- 172.21.15.107:0/3041062427 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d577d27670 con 0x55d577d2b460 2024-05-09T08:44:20.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.926+0000 7f0426ffd640 1 --2- 172.21.15.107:0/3041062427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d2b460 0x55d577d1b8b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0414005e40 tx=0x7f0414005d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57ec56535a7f0f6c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:20.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.927+0000 7f04117fa640 1 -- 172.21.15.107:0/3041062427 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041401e040 con 0x55d577d2b460 2024-05-09T08:44:20.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.927+0000 7f04117fa640 1 -- 172.21.15.107:0/3041062427 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0414005040 con 0x55d577d2b460 2024-05-09T08:44:20.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.928+0000 7f04117fa640 1 -- 172.21.15.107:0/3041062427 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0414012aa0 con 0x55d577d2b460 2024-05-09T08:44:20.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.928+0000 7f0430bb5740 1 -- 172.21.15.107:0/3041062427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d2b460 msgr2=0x55d577d1b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:20.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.928+0000 7f0430bb5740 1 --2- 172.21.15.107:0/3041062427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d2b460 0x55d577d1b8b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0414005e40 tx=0x7f0414005d90 comp rx=0 tx=0).stop 2024-05-09T08:44:20.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.932+0000 7f0430bb5740 1 -- 172.21.15.107:0/3041062427 shutdown_connections 2024-05-09T08:44:20.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.932+0000 7f0430bb5740 1 --2- 172.21.15.107:0/3041062427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d2b460 0x55d577d1b8b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.932+0000 7f0430bb5740 1 --2- 172.21.15.107:0/3041062427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d292a0 0x55d577d178c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.932+0000 7f0430bb5740 1 --2- 172.21.15.107:0/3041062427 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d20d90 0x55d577d2af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.932+0000 7f0430bb5740 1 -- 172.21.15.107:0/3041062427 >> 172.21.15.107:0/3041062427 conn(0x55d577d1c790 msgr2=0x55d577d1b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:20.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.937+0000 7f0430bb5740 1 -- 172.21.15.107:0/3041062427 shutdown_connections 2024-05-09T08:44:20.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.937+0000 7f0430bb5740 1 -- 172.21.15.107:0/3041062427 wait complete. 2024-05-09T08:44:20.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.937+0000 7f0430bb5740 1 -- start start 2024-05-09T08:44:20.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.938+0000 7f0430bb5740 1 Processor -- start 2024-05-09T08:44:20.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.938+0000 7f0430bb5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d20d90 0x55d577d33810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.938+0000 7f0430bb5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d292a0 0x55d577d24f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.938+0000 7f0430bb5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d2b460 0x55d577d25450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.938+0000 7f0430bb5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0420161b90 con 0x55d577d292a0 2024-05-09T08:44:20.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.938+0000 7f0430bb5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0420163000 con 0x55d577d2b460 2024-05-09T08:44:20.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.938+0000 7f0430bb5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0420162260 con 0x55d577d20d90 2024-05-09T08:44:20.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.938+0000 7f0430bb5740 10 client.0 _ll_register_callbacks cb 0x7f041c00d650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:20.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.939+0000 7f0426ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d20d90 0x55d577d33810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:20.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.939+0000 7f04277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d292a0 0x55d577d24f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:20.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.939+0000 7f04277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d292a0 0x55d577d24f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58264/0 (socket says 172.21.15.107:58264) 2024-05-09T08:44:20.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.939+0000 7f0426ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d2b460 0x55d577d25450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:20.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.939+0000 7f04277fe640 1 -- 172.21.15.107:0/4082969531 learned_addr learned my addr 172.21.15.107:0/4082969531 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:20.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.939+0000 7f04277fe640 1 -- 172.21.15.107:0/4082969531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d20d90 msgr2=0x55d577d33810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:20.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.939+0000 7f04277fe640 1 --2- 172.21.15.107:0/4082969531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d20d90 0x55d577d33810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.939+0000 7f04277fe640 1 -- 172.21.15.107:0/4082969531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d2b460 msgr2=0x55d577d25450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:20.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.939+0000 7f04277fe640 1 --2- 172.21.15.107:0/4082969531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d2b460 0x55d577d25450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.939+0000 7f04277fe640 1 -- 172.21.15.107:0/4082969531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d577d25a90 con 0x55d577d292a0 2024-05-09T08:44:20.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.939+0000 7f04277fe640 1 --2- 172.21.15.107:0/4082969531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d292a0 0x55d577d24f10 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x55d577d29c90 tx=0x7f041c00d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c361b7b37b53753 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:20.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.939+0000 7f0406ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:44:20.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.940+0000 7f0406ffd640 1 -- 172.21.15.107:0/4082969531 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c027080 con 0x55d577d292a0 2024-05-09T08:44:20.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.940+0000 7f0406ffd640 1 -- 172.21.15.107:0/4082969531 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7f041c005240 con 0x55d577d292a0 2024-05-09T08:44:20.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.940+0000 7f0430bb5740 1 -- 172.21.15.107:0/4082969531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0414005040 con 0x55d577d292a0 2024-05-09T08:44:20.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.940+0000 7f0406ffd640 1 -- 172.21.15.107:0/4082969531 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f041c01d030 con 0x55d577d292a0 2024-05-09T08:44:20.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.940+0000 7f0430bb5740 10 client.18693 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:20.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.940+0000 7f0430bb5740 1 -- 172.21.15.107:0/4082969531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d577da2730 con 0x55d577d292a0 2024-05-09T08:44:20.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.941+0000 7f0406ffd640 1 -- 172.21.15.107:0/4082969531 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0414005040 con 0x55d577d292a0 2024-05-09T08:44:20.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.942+0000 7f0406ffd640 1 -- 172.21.15.107:0/4082969531 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f041c0045d0 con 0x55d577d292a0 2024-05-09T08:44:20.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.942+0000 7f0430bb5740 10 client.18693 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:20.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.942+0000 7f0430bb5740 10 client.18693 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:20.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.942+0000 7f0430bb5740 20 client.18693 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:20.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.942+0000 7f0430bb5740 1 --2- 172.21.15.107:0/4082969531 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d577da2e30 0x55d577da5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:20.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.942+0000 7f0430bb5740 4 client.18693 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:44:20.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.942+0000 7f0430bb5740 1 -- 172.21.15.107:0/4082969531 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d577c93430 con 0x55d577da2e30 2024-05-09T08:44:20.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.943+0000 7f0426ffd640 1 --2- 172.21.15.107:0/4082969531 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d577da2e30 0x55d577da5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:20.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.944+0000 7f0426ffd640 1 --2- 172.21.15.107:0/4082969531 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d577da2e30 0x55d577da5250 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3227053f76f7b999 server_cookie=4af6d30f795b4479 in_seq=0 out_seq=0 2024-05-09T08:44:20.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.944+0000 7f0406ffd640 10 client.18693 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:20.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.944+0000 7f0406ffd640 1 -- 172.21.15.107:0/4082969531 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d577c93430 con 0x55d577da2e30 2024-05-09T08:44:20.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.944+0000 7f0406ffd640 10 client.18693 handle_command_reply: tid=0 2024-05-09T08:44:20.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.944+0000 7f0430bb5740 1 client.18693 shutdown 2024-05-09T08:44:20.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.953+0000 7f0430bb5740 1 -- 172.21.15.107:0/4082969531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d292a0 msgr2=0x55d577d24f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:20.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.953+0000 7f0430bb5740 1 --2- 172.21.15.107:0/4082969531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d292a0 0x55d577d24f10 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x55d577d29c90 tx=0x7f041c00d2e0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.953+0000 7f0430bb5740 1 -- 172.21.15.107:0/4082969531 shutdown_connections 2024-05-09T08:44:20.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.953+0000 7f0430bb5740 1 --2- 172.21.15.107:0/4082969531 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d577da2e30 0x55d577da5250 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.953+0000 7f0430bb5740 1 --2- 172.21.15.107:0/4082969531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d577d2b460 0x55d577d25450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.953+0000 7f0430bb5740 1 --2- 172.21.15.107:0/4082969531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d577d292a0 0x55d577d24f10 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.953+0000 7f0430bb5740 1 --2- 172.21.15.107:0/4082969531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d577d20d90 0x55d577d33810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.953+0000 7f0430bb5740 1 -- 172.21.15.107:0/4082969531 >> 172.21.15.107:0/4082969531 conn(0x55d577d1c790 msgr2=0x55d577d1ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:20.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.953+0000 7f0406ffd640 0 client.18693 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:20.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.955+0000 7f0430bb5740 1 -- 172.21.15.107:0/4082969531 shutdown_connections 2024-05-09T08:44:20.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.955+0000 7f0430bb5740 1 -- 172.21.15.107:0/4082969531 wait complete. 2024-05-09T08:44:20.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.955+0000 7f0430bb5740 20 client.18693 trim_cache size 0 max 16384 2024-05-09T08:44:20.955 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:44:20.955 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:44:20.956 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:44:20.956 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:44:20.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.963+0000 7f042cb66640 1 -- 172.21.15.107:0/2559817691 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f04180a2e90 msgr2=0x7f04180a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:20.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.963+0000 7f042cb66640 1 --2- 172.21.15.107:0/2559817691 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f04180a2e90 0x7f04180a5350 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0428150390 tx=0x7f0428090040 comp rx=0 tx=0).stop 2024-05-09T08:44:20.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.964+0000 7f042cb66640 1 -- 172.21.15.107:0/2559817691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f042013b750 msgr2=0x7f042013bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:20.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.964+0000 7f042cb66640 1 --2- 172.21.15.107:0/2559817691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f042013b750 0x7f042013bc00 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f0428050860 tx=0x7f04280976a0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.964+0000 7f042cb66640 1 -- 172.21.15.107:0/2559817691 shutdown_connections 2024-05-09T08:44:20.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.964+0000 7f042cb66640 1 --2- 172.21.15.107:0/2559817691 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f04180a2e90 0x7f04180a5350 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.964+0000 7f042cb66640 1 --2- 172.21.15.107:0/2559817691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f042013b750 0x7f042013bc00 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.964+0000 7f042cb66640 1 --2- 172.21.15.107:0/2559817691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04200aa2a0 0x7f042013b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.964+0000 7f042cb66640 1 --2- 172.21.15.107:0/2559817691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04200a51d0 0x7f0420142190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:20.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.964+0000 7f042cb66640 1 -- 172.21.15.107:0/2559817691 >> 172.21.15.107:0/2559817691 conn(0x7f042001aa70 msgr2=0x7f04200a7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:20.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.980+0000 7f042cb66640 1 -- 172.21.15.107:0/2559817691 shutdown_connections 2024-05-09T08:44:20.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:20.980+0000 7f042cb66640 1 -- 172.21.15.107:0/2559817691 wait complete. 2024-05-09T08:44:21.022 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:20 smithi107 ceph-mon[30122]: pgmap v341: 129 pgs: 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 120 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 13 MiB/s wr, 3.32k op/s 2024-05-09T08:44:21.023 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:44:21.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:44:21.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:20 smithi157 ceph-mon[33614]: pgmap v341: 129 pgs: 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 120 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 13 MiB/s wr, 3.32k op/s 2024-05-09T08:44:21.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:20 smithi110 ceph-mon[33622]: pgmap v341: 129 pgs: 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 120 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 13 MiB/s wr, 3.32k op/s 2024-05-09T08:44:21.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.330+0000 7f15d0dd5640 1 -- 172.21.15.107:0/2929018684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15cc108a60 msgr2=0x7f15cc10caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:21.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.330+0000 7f15d0dd5640 1 --2- 172.21.15.107:0/2929018684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15cc108a60 0x7f15cc10caa0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f15c0009860 tx=0x7f15c002a340 comp rx=0 tx=0).stop 2024-05-09T08:44:21.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.330+0000 7f15d0dd5640 1 -- 172.21.15.107:0/2929018684 shutdown_connections 2024-05-09T08:44:21.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.330+0000 7f15d0dd5640 1 --2- 172.21.15.107:0/2929018684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc10d180 0x7f15cc114d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:21.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.330+0000 7f15d0dd5640 1 --2- 172.21.15.107:0/2929018684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15cc108a60 0x7f15cc10caa0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:21.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.330+0000 7f15d0dd5640 1 --2- 172.21.15.107:0/2929018684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15cc1080b0 0x7f15cc108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:21.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.330+0000 7f15d0dd5640 1 -- 172.21.15.107:0/2929018684 >> 172.21.15.107:0/2929018684 conn(0x7f15cc075070 msgr2=0x7f15cc106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:21.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.331+0000 7f15d0dd5640 1 -- 172.21.15.107:0/2929018684 shutdown_connections 2024-05-09T08:44:21.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.335+0000 7f15d0dd5640 1 -- 172.21.15.107:0/2929018684 wait complete. 2024-05-09T08:44:21.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.336+0000 7f15d0dd5640 1 Processor -- start 2024-05-09T08:44:21.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.336+0000 7f15d0dd5640 1 -- start start 2024-05-09T08:44:21.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.336+0000 7f15d0dd5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15cc1080b0 0x7f15cc081f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:21.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.336+0000 7f15d0dd5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15cc10d180 0x7f15cc07afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:21.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.336+0000 7f15d0dd5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc07b510 0x7f15cc07b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:21.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.336+0000 7f15d0dd5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc118170 con 0x7f15cc07b510 2024-05-09T08:44:21.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.336+0000 7f15d0dd5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc117ff0 con 0x7f15cc1080b0 2024-05-09T08:44:21.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.336+0000 7f15d0dd5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc1182f0 con 0x7f15cc10d180 2024-05-09T08:44:21.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.338+0000 7f15cad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc07b510 0x7f15cc07b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:21.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.338+0000 7f15cad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc07b510 0x7f15cc07b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58282/0 (socket says 172.21.15.107:58282) 2024-05-09T08:44:21.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.338+0000 7f15cad9d640 1 -- 172.21.15.107:0/4091944798 learned_addr learned my addr 172.21.15.107:0/4091944798 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:21.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.339+0000 7f15ca59c640 1 --2- 172.21.15.107:0/4091944798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15cc1080b0 0x7f15cc081f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:21.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.339+0000 7f15c9d9b640 1 --2- 172.21.15.107:0/4091944798 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15cc10d180 0x7f15cc07afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:21.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.340+0000 7f15ca59c640 1 -- 172.21.15.107:0/4091944798 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15cc10d180 msgr2=0x7f15cc07afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:21.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.340+0000 7f15ca59c640 1 --2- 172.21.15.107:0/4091944798 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15cc10d180 0x7f15cc07afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:21.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.340+0000 7f15ca59c640 1 -- 172.21.15.107:0/4091944798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc07b510 msgr2=0x7f15cc07b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:21.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.340+0000 7f15ca59c640 1 --2- 172.21.15.107:0/4091944798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc07b510 0x7f15cc07b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:21.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.340+0000 7f15ca59c640 1 -- 172.21.15.107:0/4091944798 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15cc07fa20 con 0x7f15cc1080b0 2024-05-09T08:44:21.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.341+0000 7f15ca59c640 1 --2- 172.21.15.107:0/4091944798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15cc1080b0 0x7f15cc081f00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f15c400ef60 tx=0x7f15c400ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:21.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.341+0000 7f15bb7fe640 1 -- 172.21.15.107:0/4091944798 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c401b070 con 0x7f15cc1080b0 2024-05-09T08:44:21.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.342+0000 7f15bb7fe640 1 -- 172.21.15.107:0/4091944798 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15c400ba20 con 0x7f15cc1080b0 2024-05-09T08:44:21.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.344+0000 7f15bb7fe640 1 -- 172.21.15.107:0/4091944798 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c4016bf0 con 0x7f15cc1080b0 2024-05-09T08:44:21.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.344+0000 7f15d0dd5640 1 -- 172.21.15.107:0/4091944798 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15cc07fd10 con 0x7f15cc1080b0 2024-05-09T08:44:21.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.344+0000 7f15d0dd5640 1 -- 172.21.15.107:0/4091944798 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15cc0801e0 con 0x7f15cc1080b0 2024-05-09T08:44:21.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.350+0000 7f15d0dd5640 1 -- 172.21.15.107:0/4091944798 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15ac005290 con 0x7f15cc1080b0 2024-05-09T08:44:21.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.353+0000 7f15bb7fe640 1 -- 172.21.15.107:0/4091944798 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f15c401e070 con 0x7f15cc1080b0 2024-05-09T08:44:21.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.358+0000 7f15bb7fe640 1 --2- 172.21.15.107:0/4091944798 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f15b40a2f00 0x7f15b40a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:21.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.359+0000 7f15c9d9b640 1 --2- 172.21.15.107:0/4091944798 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f15b40a2f00 0x7f15b40a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:21.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.359+0000 7f15bb7fe640 1 -- 172.21.15.107:0/4091944798 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7f15c4016410 con 0x7f15cc1080b0 2024-05-09T08:44:21.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.362+0000 7f15bb7fe640 1 -- 172.21.15.107:0/4091944798 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f15c409c7d0 con 0x7f15cc1080b0 2024-05-09T08:44:21.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.363+0000 7f15c9d9b640 1 --2- 172.21.15.107:0/4091944798 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f15b40a2f00 0x7f15b40a53c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f15c002a930 tx=0x7f15c0009490 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.833+0000 7f15d0dd5640 1 -- 172.21.15.107:0/4091944798 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15ac005850 con 0x7f15cc1080b0 2024-05-09T08:44:21.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.835+0000 7f15bb7fe640 1 -- 172.21.15.107:0/4091944798 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11887 (secure 0 0 0) 0x7f15c40a0ad0 con 0x7f15cc1080b0 2024-05-09T08:44:21.841 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:44:21.842 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:37:10.930317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:44:21.842 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2024-05-09T08:44:21.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.846+0000 7f15d0dd5640 1 -- 172.21.15.107:0/4091944798 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f15b40a2f00 msgr2=0x7f15b40a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:21.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.846+0000 7f15d0dd5640 1 --2- 172.21.15.107:0/4091944798 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f15b40a2f00 0x7f15b40a53c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f15c002a930 tx=0x7f15c0009490 comp rx=0 tx=0).stop 2024-05-09T08:44:21.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.846+0000 7f15d0dd5640 1 -- 172.21.15.107:0/4091944798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15cc1080b0 msgr2=0x7f15cc081f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:21.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.846+0000 7f15d0dd5640 1 --2- 172.21.15.107:0/4091944798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15cc1080b0 0x7f15cc081f00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f15c400ef60 tx=0x7f15c400ef90 comp rx=0 tx=0).stop 2024-05-09T08:44:21.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.850+0000 7f15d0dd5640 1 -- 172.21.15.107:0/4091944798 shutdown_connections 2024-05-09T08:44:21.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.850+0000 7f15d0dd5640 1 --2- 172.21.15.107:0/4091944798 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f15b40a2f00 0x7f15b40a53c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:21.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.850+0000 7f15d0dd5640 1 --2- 172.21.15.107:0/4091944798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15cc07b510 0x7f15cc07b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:21.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.850+0000 7f15d0dd5640 1 --2- 172.21.15.107:0/4091944798 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15cc10d180 0x7f15cc07afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:21.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.850+0000 7f15d0dd5640 1 --2- 172.21.15.107:0/4091944798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15cc1080b0 0x7f15cc081f00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:21.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.850+0000 7f15d0dd5640 1 -- 172.21.15.107:0/4091944798 >> 172.21.15.107:0/4091944798 conn(0x7f15cc075070 msgr2=0x7f15cc1074e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:21.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.850+0000 7f15d0dd5640 1 -- 172.21.15.107:0/4091944798 shutdown_connections 2024-05-09T08:44:21.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:21.850+0000 7f15d0dd5640 1 -- 172.21.15.107:0/4091944798 wait complete. 2024-05-09T08:44:21.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:44:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.081+0000 7fa71bda2640 1 -- 172.21.15.107:0/2010413007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7141010c0 msgr2=0x7fa71410d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.081+0000 7fa71bda2640 1 --2- 172.21.15.107:0/2010413007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7141010c0 0x7fa71410d5d0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fa70c009930 tx=0x7fa70c02a420 comp rx=0 tx=0).stop 2024-05-09T08:44:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.082+0000 7fa71bda2640 1 -- 172.21.15.107:0/2010413007 shutdown_connections 2024-05-09T08:44:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.082+0000 7fa71bda2640 1 --2- 172.21.15.107:0/2010413007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7141010c0 0x7fa71410d5d0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.082+0000 7fa71bda2640 1 --2- 172.21.15.107:0/2010413007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa714100700 0x7fa714100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.082+0000 7fa71bda2640 1 --2- 172.21.15.107:0/2010413007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa714106710 0x7fa714106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.082+0000 7fa71bda2640 1 -- 172.21.15.107:0/2010413007 >> 172.21.15.107:0/2010413007 conn(0x7fa714077830 msgr2=0x7fa714077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.082+0000 7fa71bda2640 1 -- 172.21.15.107:0/2010413007 shutdown_connections 2024-05-09T08:44:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.082+0000 7fa71bda2640 1 -- 172.21.15.107:0/2010413007 wait complete. 2024-05-09T08:44:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.082+0000 7fa71bda2640 1 Processor -- start 2024-05-09T08:44:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.082+0000 7fa71bda2640 1 -- start start 2024-05-09T08:44:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.082+0000 7fa71bda2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa714100700 0x7fa714198f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.082+0000 7fa71bda2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7141010c0 0x7fa714199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.082+0000 7fa71bda2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa714106710 0x7fa7141a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.082+0000 7fa71bda2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa714110a20 con 0x7fa714106710 2024-05-09T08:44:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.082+0000 7fa71bda2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa7141108a0 con 0x7fa714100700 2024-05-09T08:44:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.082+0000 7fa71bda2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa714110ba0 con 0x7fa7141010c0 2024-05-09T08:44:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.082+0000 7fa71a33f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa714106710 0x7fa7141a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.083+0000 7fa71a33f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa714106710 0x7fa7141a3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58304/0 (socket says 172.21.15.107:58304) 2024-05-09T08:44:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.083+0000 7fa71a33f640 1 -- 172.21.15.107:0/737161067 learned_addr learned my addr 172.21.15.107:0/737161067 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.083+0000 7fa719b3e640 1 --2- 172.21.15.107:0/737161067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa714100700 0x7fa714198f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.083+0000 7fa71933d640 1 --2- 172.21.15.107:0/737161067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7141010c0 0x7fa714199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.083+0000 7fa71a33f640 1 -- 172.21.15.107:0/737161067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7141010c0 msgr2=0x7fa714199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.083+0000 7fa71a33f640 1 --2- 172.21.15.107:0/737161067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7141010c0 0x7fa714199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.083+0000 7fa71a33f640 1 -- 172.21.15.107:0/737161067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa714100700 msgr2=0x7fa714198f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.083+0000 7fa71a33f640 1 --2- 172.21.15.107:0/737161067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa714100700 0x7fa714198f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.083+0000 7fa71a33f640 1 -- 172.21.15.107:0/737161067 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7141a4500 con 0x7fa714106710 2024-05-09T08:44:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.083+0000 7fa71a33f640 1 --2- 172.21.15.107:0/737161067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa714106710 0x7fa7141a3fc0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fa70c02a940 tx=0x7fa70c003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.083+0000 7fa712ffd640 1 -- 172.21.15.107:0/737161067 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70c03f070 con 0x7fa714106710 2024-05-09T08:44:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.083+0000 7fa712ffd640 1 -- 172.21.15.107:0/737161067 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa70c003e40 con 0x7fa714106710 2024-05-09T08:44:22.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.083+0000 7fa71bda2640 1 -- 172.21.15.107:0/737161067 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7141a46d0 con 0x7fa714106710 2024-05-09T08:44:22.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.083+0000 7fa712ffd640 1 -- 172.21.15.107:0/737161067 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70c035030 con 0x7fa714106710 2024-05-09T08:44:22.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.083+0000 7fa71bda2640 1 -- 172.21.15.107:0/737161067 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7141a4b30 con 0x7fa714106710 2024-05-09T08:44:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.085+0000 7fa712ffd640 1 -- 172.21.15.107:0/737161067 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fa70c042070 con 0x7fa714106710 2024-05-09T08:44:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.086+0000 7fa71eb92740 1 Processor -- start 2024-05-09T08:44:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.086+0000 7fa71eb92740 1 -- start start 2024-05-09T08:44:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.086+0000 7fa71eb92740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840ea330 0x5615840ea710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.086+0000 7fa712ffd640 1 --2- 172.21.15.107:0/737161067 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa6fc0a2d00 0x7fa6fc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.086+0000 7fa71eb92740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x5615840f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.086+0000 7fa71eb92740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840f5320 0x5615840f7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.086+0000 7fa71eb92740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56158404fc40 con 0x5615840eac50 2024-05-09T08:44:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.086+0000 7fa71eb92740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561583ff7580 con 0x5615840ea330 2024-05-09T08:44:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.086+0000 7fa71eb92740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56158404bc00 con 0x5615840f5320 2024-05-09T08:44:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.086+0000 7fa71933d640 1 --2- 172.21.15.107:0/737161067 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa6fc0a2d00 0x7fa6fc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.086+0000 7fa712ffd640 1 -- 172.21.15.107:0/737161067 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7fa70c0f00b0 con 0x7fa714106710 2024-05-09T08:44:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.086+0000 7fa71933d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x5615840f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.086+0000 7fa719b3e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840ea330 0x5615840ea710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.087+0000 7fa719b3e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840ea330 0x5615840ea710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:57708/0 (socket says 172.21.15.107:57708) 2024-05-09T08:44:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.087+0000 7fa71933d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x5615840f4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58314/0 (socket says 172.21.15.107:58314) 2024-05-09T08:44:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.087+0000 7fa719b3e640 1 -- 172.21.15.107:0/1909608988 learned_addr learned my addr 172.21.15.107:0/1909608988 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.087+0000 7fa71a33f640 1 --2- 172.21.15.107:0/1909608988 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840f5320 0x5615840f7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.087+0000 7fa719b3e640 1 -- 172.21.15.107:0/1909608988 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840f5320 msgr2=0x5615840f7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.087+0000 7fa719b3e640 1 --2- 172.21.15.107:0/1909608988 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840f5320 0x5615840f7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.087+0000 7fa719b3e640 1 -- 172.21.15.107:0/1909608988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 msgr2=0x5615840f4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.087+0000 7fa719b3e640 1 --2- 172.21.15.107:0/1909608988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x5615840f4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.087+0000 7fa719b3e640 1 -- 172.21.15.107:0/1909608988 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561583d633f0 con 0x5615840ea330 2024-05-09T08:44:22.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.087+0000 7fa71933d640 1 --2- 172.21.15.107:0/737161067 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa6fc0a2d00 0x7fa6fc0a51c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa708005d00 tx=0x7fa70800cba0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.087+0000 7fa719b3e640 1 --2- 172.21.15.107:0/1909608988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840ea330 0x5615840ea710 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa70400aec0 tx=0x7fa70400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=610ee28c2c3ccc45 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.087+0000 7fa6fbfff640 1 -- 172.21.15.107:0/1909608988 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa704010840 con 0x5615840ea330 2024-05-09T08:44:22.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.087+0000 7fa6fbfff640 1 -- 172.21.15.107:0/1909608988 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7040183f0 con 0x5615840ea330 2024-05-09T08:44:22.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.087+0000 7fa6fbfff640 1 -- 172.21.15.107:0/1909608988 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa704017600 con 0x5615840ea330 2024-05-09T08:44:22.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.088+0000 7fa71eb92740 1 -- 172.21.15.107:0/1909608988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840ea330 msgr2=0x5615840ea710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.088+0000 7fa71eb92740 1 --2- 172.21.15.107:0/1909608988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840ea330 0x5615840ea710 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa70400aec0 tx=0x7fa70400f790 comp rx=0 tx=0).stop 2024-05-09T08:44:22.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.088+0000 7fa71eb92740 1 -- 172.21.15.107:0/1909608988 shutdown_connections 2024-05-09T08:44:22.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.088+0000 7fa71eb92740 1 --2- 172.21.15.107:0/1909608988 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840f5320 0x5615840f7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.088+0000 7fa71eb92740 1 --2- 172.21.15.107:0/1909608988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x5615840f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.088+0000 7fa71eb92740 1 --2- 172.21.15.107:0/1909608988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840ea330 0x5615840ea710 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.088+0000 7fa71eb92740 1 -- 172.21.15.107:0/1909608988 >> 172.21.15.107:0/1909608988 conn(0x5615840e6650 msgr2=0x5615840e6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.088+0000 7fa71eb92740 1 -- 172.21.15.107:0/1909608988 shutdown_connections 2024-05-09T08:44:22.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.088+0000 7fa71eb92740 1 -- 172.21.15.107:0/1909608988 wait complete. 2024-05-09T08:44:22.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.088+0000 7fa71eb92740 1 -- start start 2024-05-09T08:44:22.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.088+0000 7fa71eb92740 1 Processor -- start 2024-05-09T08:44:22.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa71eb92740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840ea330 0x5615840f2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa71eb92740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840eac50 0x5615840f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa71eb92740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840f5320 0x5615840f2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa71eb92740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5615840f8110 con 0x5615840ea330 2024-05-09T08:44:22.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa71eb92740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56158400ac40 con 0x5615840f5320 2024-05-09T08:44:22.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa71eb92740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5615840f8290 con 0x5615840eac50 2024-05-09T08:44:22.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa71eb92740 10 client.0 _ll_register_callbacks cb 0x561583d62be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:22.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa719b3e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840ea330 0x5615840f2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa719b3e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840ea330 0x5615840f2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58326/0 (socket says 172.21.15.107:58326) 2024-05-09T08:44:22.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa719b3e640 1 -- 172.21.15.107:0/2813614851 learned_addr learned my addr 172.21.15.107:0/2813614851 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:22.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa71a33f640 1 --2- 172.21.15.107:0/2813614851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840f5320 0x5615840f2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa71933d640 1 --2- 172.21.15.107:0/2813614851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840eac50 0x5615840f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa719b3e640 1 -- 172.21.15.107:0/2813614851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840eac50 msgr2=0x5615840f27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa719b3e640 1 --2- 172.21.15.107:0/2813614851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840eac50 0x5615840f27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa719b3e640 1 -- 172.21.15.107:0/2813614851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840f5320 msgr2=0x5615840f2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa719b3e640 1 --2- 172.21.15.107:0/2813614851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840f5320 0x5615840f2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa719b3e640 1 -- 172.21.15.107:0/2813614851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561583e2d7d0 con 0x5615840ea330 2024-05-09T08:44:22.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa71a33f640 1 --2- 172.21.15.107:0/2813614851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840f5320 0x5615840f2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:44:22.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa719b3e640 1 --2- 172.21.15.107:0/2813614851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840ea330 0x5615840f2280 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fa704005e90 tx=0x7fa704004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b25cc777090079b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.089+0000 7fa6f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:44:22.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.090+0000 7fa6f97fa640 1 -- 172.21.15.107:0/2813614851 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa704026070 con 0x5615840ea330 2024-05-09T08:44:22.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.090+0000 7fa6f97fa640 1 -- 172.21.15.107:0/2813614851 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7fa704010a50 con 0x5615840ea330 2024-05-09T08:44:22.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.090+0000 7fa71eb92740 1 -- 172.21.15.107:0/2813614851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7040183f0 con 0x5615840ea330 2024-05-09T08:44:22.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.090+0000 7fa6f97fa640 1 -- 172.21.15.107:0/2813614851 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa704032050 con 0x5615840ea330 2024-05-09T08:44:22.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.090+0000 7fa71eb92740 10 client.18708 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:22.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.090+0000 7fa71eb92740 1 -- 172.21.15.107:0/2813614851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5615841042e0 con 0x5615840ea330 2024-05-09T08:44:22.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.090+0000 7fa6f97fa640 1 -- 172.21.15.107:0/2813614851 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7040183f0 con 0x5615840ea330 2024-05-09T08:44:22.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.090+0000 7fa6f97fa640 1 -- 172.21.15.107:0/2813614851 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa7040054f0 con 0x5615840ea330 2024-05-09T08:44:22.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.090+0000 7fa71eb92740 10 client.18708 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:22.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.090+0000 7fa71eb92740 10 client.18708 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:44:22.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.090+0000 7fa71eb92740 20 client.18708 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:22.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.090+0000 7fa71eb92740 1 --2- 172.21.15.107:0/2813614851 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5615841056f0 0x561584107ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.090+0000 7fa71eb92740 4 client.18708 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:44:22.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.090+0000 7fa71eb92740 1 -- 172.21.15.107:0/2813614851 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56158405d370 con 0x5615841056f0 2024-05-09T08:44:22.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.090+0000 7fa71933d640 1 --2- 172.21.15.107:0/2813614851 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5615841056f0 0x561584107ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.091+0000 7fa6f97fa640 10 client.18708 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:44:22.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.091+0000 7fa71933d640 1 --2- 172.21.15.107:0/2813614851 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5615841056f0 0x561584107ae0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e8cd497157efce98 server_cookie=2c508d54079b8709 in_seq=0 out_seq=0 2024-05-09T08:44:22.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.091+0000 7fa6f97fa640 1 -- 172.21.15.107:0/2813614851 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56158405d370 con 0x5615841056f0 2024-05-09T08:44:22.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.091+0000 7fa6f97fa640 10 client.18708 handle_command_reply: tid=0 2024-05-09T08:44:22.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.091+0000 7fa71eb92740 1 client.18708 shutdown 2024-05-09T08:44:22.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.092+0000 7fa71eb92740 1 -- 172.21.15.107:0/2813614851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840ea330 msgr2=0x5615840f2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.092+0000 7fa71eb92740 1 --2- 172.21.15.107:0/2813614851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840ea330 0x5615840f2280 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fa704005e90 tx=0x7fa704004c70 comp rx=0 tx=0).stop 2024-05-09T08:44:22.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.092+0000 7fa71eb92740 1 -- 172.21.15.107:0/2813614851 shutdown_connections 2024-05-09T08:44:22.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.092+0000 7fa71eb92740 1 --2- 172.21.15.107:0/2813614851 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5615841056f0 0x561584107ae0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.092+0000 7fa71eb92740 1 --2- 172.21.15.107:0/2813614851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840f5320 0x5615840f2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.092+0000 7fa6f97fa640 0 client.18708 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:44:22.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.092+0000 7fa71eb92740 1 --2- 172.21.15.107:0/2813614851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840eac50 0x5615840f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.092+0000 7fa71eb92740 1 --2- 172.21.15.107:0/2813614851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840ea330 0x5615840f2280 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.092+0000 7fa71eb92740 1 -- 172.21.15.107:0/2813614851 >> 172.21.15.107:0/2813614851 conn(0x5615840e6650 msgr2=0x5615840ec450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.092+0000 7fa71eb92740 1 -- 172.21.15.107:0/2813614851 shutdown_connections 2024-05-09T08:44:22.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.092+0000 7fa71eb92740 1 -- 172.21.15.107:0/2813614851 wait complete. 2024-05-09T08:44:22.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.092+0000 7fa71eb92740 20 client.18708 trim_cache size 0 max 16384 2024-05-09T08:44:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71eb92740 1 Processor -- start 2024-05-09T08:44:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71eb92740 1 -- start start 2024-05-09T08:44:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71eb92740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840ea330 0x5615840f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71eb92740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x561584100fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71eb92740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840f5320 0x5615840ec420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71eb92740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa7141b1600 con 0x5615840eac50 2024-05-09T08:44:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71eb92740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa7141aeed0 con 0x5615840ea330 2024-05-09T08:44:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71eb92740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa7141acd40 con 0x5615840f5320 2024-05-09T08:44:22.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71933d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x561584100fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71933d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x561584100fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58334/0 (socket says 172.21.15.107:58334) 2024-05-09T08:44:22.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71933d640 1 -- 172.21.15.107:0/3832928309 learned_addr learned my addr 172.21.15.107:0/3832928309 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:22.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa719b3e640 1 --2- 172.21.15.107:0/3832928309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840ea330 0x5615840f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71a33f640 1 --2- 172.21.15.107:0/3832928309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840f5320 0x5615840ec420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71933d640 1 -- 172.21.15.107:0/3832928309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840f5320 msgr2=0x5615840ec420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71933d640 1 --2- 172.21.15.107:0/3832928309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840f5320 0x5615840ec420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71933d640 1 -- 172.21.15.107:0/3832928309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840ea330 msgr2=0x5615840f49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71933d640 1 --2- 172.21.15.107:0/3832928309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840ea330 0x5615840f49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71933d640 1 -- 172.21.15.107:0/3832928309 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615840e6480 con 0x5615840eac50 2024-05-09T08:44:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa71a33f640 1 --2- 172.21.15.107:0/3832928309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840f5320 0x5615840ec420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:44:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.105+0000 7fa719b3e640 1 --2- 172.21.15.107:0/3832928309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840ea330 0x5615840f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:44:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.106+0000 7fa71933d640 1 --2- 172.21.15.107:0/3832928309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x561584100fd0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fa714069510 tx=0x7fa708004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574f0cdeb071359e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.106+0000 7fa6fb7fe640 1 -- 172.21.15.107:0/3832928309 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708012920 con 0x5615840eac50 2024-05-09T08:44:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.106+0000 7fa6fb7fe640 1 -- 172.21.15.107:0/3832928309 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa70802a330 con 0x5615840eac50 2024-05-09T08:44:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.106+0000 7fa6fb7fe640 1 -- 172.21.15.107:0/3832928309 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708005260 con 0x5615840eac50 2024-05-09T08:44:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.106+0000 7fa71eb92740 1 -- 172.21.15.107:0/3832928309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 msgr2=0x561584100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.106+0000 7fa71eb92740 1 --2- 172.21.15.107:0/3832928309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x561584100fd0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fa714069510 tx=0x7fa708004730 comp rx=0 tx=0).stop 2024-05-09T08:44:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.106+0000 7fa71eb92740 1 -- 172.21.15.107:0/3832928309 shutdown_connections 2024-05-09T08:44:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.106+0000 7fa71eb92740 1 --2- 172.21.15.107:0/3832928309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840f5320 0x5615840ec420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.106+0000 7fa71eb92740 1 --2- 172.21.15.107:0/3832928309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x561584100fd0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.106+0000 7fa71eb92740 1 --2- 172.21.15.107:0/3832928309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840ea330 0x5615840f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.106+0000 7fa71eb92740 1 -- 172.21.15.107:0/3832928309 >> 172.21.15.107:0/3832928309 conn(0x5615840e6650 msgr2=0x5615840e6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.106+0000 7fa71eb92740 1 -- 172.21.15.107:0/3832928309 shutdown_connections 2024-05-09T08:44:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.106+0000 7fa71eb92740 1 -- 172.21.15.107:0/3832928309 wait complete. 2024-05-09T08:44:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71eb92740 1 -- start start 2024-05-09T08:44:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71eb92740 1 Processor -- start 2024-05-09T08:44:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71eb92740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840ea330 0x5615840f8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71eb92740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x5615840f8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71eb92740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840f5320 0x561584167e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71eb92740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa7141ad390 con 0x5615840eac50 2024-05-09T08:44:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71eb92740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa7141ae800 con 0x5615840f5320 2024-05-09T08:44:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71eb92740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa7141ada60 con 0x5615840ea330 2024-05-09T08:44:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71eb92740 10 client.0 _ll_register_callbacks cb 0x7fa7080144f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71933d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x5615840f8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71933d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x5615840f8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58338/0 (socket says 172.21.15.107:58338) 2024-05-09T08:44:22.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71933d640 1 -- 172.21.15.107:0/727517953 learned_addr learned my addr 172.21.15.107:0/727517953 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:22.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa719b3e640 1 --2- 172.21.15.107:0/727517953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840ea330 0x5615840f8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71a33f640 1 --2- 172.21.15.107:0/727517953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840f5320 0x561584167e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71933d640 1 -- 172.21.15.107:0/727517953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840ea330 msgr2=0x5615840f8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71933d640 1 --2- 172.21.15.107:0/727517953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840ea330 0x5615840f8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71933d640 1 -- 172.21.15.107:0/727517953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840f5320 msgr2=0x561584167e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71933d640 1 --2- 172.21.15.107:0/727517953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840f5320 0x561584167e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.107+0000 7fa71933d640 1 -- 172.21.15.107:0/727517953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561584168440 con 0x5615840eac50 2024-05-09T08:44:22.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa71933d640 1 --2- 172.21.15.107:0/727517953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x5615840f8d50 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fa70802c040 tx=0x7fa7080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e6d071fd62b392d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa6f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:44:22.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa6f97fa640 1 -- 172.21.15.107:0/727517953 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708038020 con 0x5615840eac50 2024-05-09T08:44:22.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa6f97fa640 1 -- 172.21.15.107:0/727517953 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7fa70802a6f0 con 0x5615840eac50 2024-05-09T08:44:22.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa71eb92740 1 -- 172.21.15.107:0/727517953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa70802a330 con 0x5615840eac50 2024-05-09T08:44:22.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa6f97fa640 1 -- 172.21.15.107:0/727517953 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa708012920 con 0x5615840eac50 2024-05-09T08:44:22.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa71eb92740 10 client.18714 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa71eb92740 1 -- 172.21.15.107:0/727517953 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56158416f760 con 0x5615840eac50 2024-05-09T08:44:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa6f97fa640 1 -- 172.21.15.107:0/727517953 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa70802a330 con 0x5615840eac50 2024-05-09T08:44:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa6f97fa640 1 -- 172.21.15.107:0/727517953 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa708033070 con 0x5615840eac50 2024-05-09T08:44:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa71eb92740 10 client.18714 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa71eb92740 10 client.18714 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:44:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa71eb92740 20 client.18714 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa71eb92740 1 --2- 172.21.15.107:0/727517953 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x56158416fea0 0x5615841722c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa71eb92740 4 client.18714 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:44:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa71eb92740 1 -- 172.21.15.107:0/727517953 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5615840576c0 con 0x56158416fea0 2024-05-09T08:44:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.108+0000 7fa719b3e640 1 --2- 172.21.15.107:0/727517953 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x56158416fea0 0x5615841722c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa719b3e640 1 --2- 172.21.15.107:0/727517953 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x56158416fea0 0x5615841722c0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a37a80487110a1b0 server_cookie=f87bea06bc970c0b in_seq=0 out_seq=0 2024-05-09T08:44:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa6f97fa640 10 client.18714 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:44:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa6f97fa640 1 -- 172.21.15.107:0/727517953 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5615840576c0 con 0x56158416fea0 2024-05-09T08:44:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa6f97fa640 10 client.18714 handle_command_reply: tid=0 2024-05-09T08:44:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa71eb92740 1 client.18714 shutdown 2024-05-09T08:44:22.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa71eb92740 1 -- 172.21.15.107:0/727517953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 msgr2=0x5615840f8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa71eb92740 1 --2- 172.21.15.107:0/727517953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x5615840f8d50 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fa70802c040 tx=0x7fa7080046c0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa71eb92740 1 -- 172.21.15.107:0/727517953 shutdown_connections 2024-05-09T08:44:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa71eb92740 1 --2- 172.21.15.107:0/727517953 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x56158416fea0 0x5615841722c0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa71eb92740 1 --2- 172.21.15.107:0/727517953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5615840f5320 0x561584167e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa71eb92740 1 --2- 172.21.15.107:0/727517953 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615840eac50 0x5615840f8d50 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa71eb92740 1 --2- 172.21.15.107:0/727517953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615840ea330 0x5615840f8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa6f97fa640 0 client.18714 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:44:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa71eb92740 1 -- 172.21.15.107:0/727517953 >> 172.21.15.107:0/727517953 conn(0x5615840e6650 msgr2=0x5615840f12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa71eb92740 1 -- 172.21.15.107:0/727517953 shutdown_connections 2024-05-09T08:44:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa71eb92740 1 -- 172.21.15.107:0/727517953 wait complete. 2024-05-09T08:44:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.109+0000 7fa71eb92740 20 client.18714 trim_cache size 0 max 16384 2024-05-09T08:44:22.119 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:44:22.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.110+0000 7fa71bda2640 1 -- 172.21.15.107:0/737161067 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa6fc0a2d00 msgr2=0x7fa6fc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.110+0000 7fa71bda2640 1 --2- 172.21.15.107:0/737161067 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa6fc0a2d00 0x7fa6fc0a51c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa708005d00 tx=0x7fa70800cba0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.110+0000 7fa71bda2640 1 -- 172.21.15.107:0/737161067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa714106710 msgr2=0x7fa7141a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.110+0000 7fa71bda2640 1 --2- 172.21.15.107:0/737161067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa714106710 0x7fa7141a3fc0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fa70c02a940 tx=0x7fa70c003cd0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.110+0000 7fa71bda2640 1 -- 172.21.15.107:0/737161067 shutdown_connections 2024-05-09T08:44:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.110+0000 7fa71bda2640 1 --2- 172.21.15.107:0/737161067 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa6fc0a2d00 0x7fa6fc0a51c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.110+0000 7fa71bda2640 1 --2- 172.21.15.107:0/737161067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa714106710 0x7fa7141a3fc0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.110+0000 7fa71bda2640 1 --2- 172.21.15.107:0/737161067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7141010c0 0x7fa714199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.110+0000 7fa71bda2640 1 --2- 172.21.15.107:0/737161067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa714100700 0x7fa714198f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.110+0000 7fa71bda2640 1 -- 172.21.15.107:0/737161067 >> 172.21.15.107:0/737161067 conn(0x7fa714077830 msgr2=0x7fa714102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.110+0000 7fa71bda2640 1 -- 172.21.15.107:0/737161067 shutdown_connections 2024-05-09T08:44:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.110+0000 7fa71bda2640 1 -- 172.21.15.107:0/737161067 wait complete. 2024-05-09T08:44:22.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:44:22.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.250+0000 7f06d0b7a640 1 -- 172.21.15.107:0/1621173066 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06cc108830 msgr2=0x7f06cc108c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.250+0000 7f06d0b7a640 1 --2- 172.21.15.107:0/1621173066 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06cc108830 0x7f06cc108c90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f06c4009ca0 tx=0x7f06c402a980 comp rx=0 tx=0).stop 2024-05-09T08:44:22.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.251+0000 7f06d0b7a640 1 -- 172.21.15.107:0/1621173066 shutdown_connections 2024-05-09T08:44:22.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.251+0000 7f06d0b7a640 1 --2- 172.21.15.107:0/1621173066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06cc1092b0 0x7f06cc11d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.251+0000 7f06d0b7a640 1 --2- 172.21.15.107:0/1621173066 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06cc108830 0x7f06cc108c90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.251+0000 7f06d0b7a640 1 --2- 172.21.15.107:0/1621173066 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06cc10fed0 0x7f06cc1102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.251+0000 7f06d0b7a640 1 -- 172.21.15.107:0/1621173066 >> 172.21.15.107:0/1621173066 conn(0x7f06cc06c7b0 msgr2=0x7f06cc06cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.252+0000 7f06d0b7a640 1 -- 172.21.15.107:0/1621173066 shutdown_connections 2024-05-09T08:44:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.254+0000 7f06d0b7a640 1 -- 172.21.15.107:0/1621173066 wait complete. 2024-05-09T08:44:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.255+0000 7f06d0b7a640 1 Processor -- start 2024-05-09T08:44:22.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.257+0000 7f06d0b7a640 1 -- start start 2024-05-09T08:44:22.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.257+0000 7f06d0b7a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06cc108830 0x7f06cc1ac930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.257+0000 7f06d0b7a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06cc1092b0 0x7f06cc1ace70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.257+0000 7f06d0b7a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06cc10fed0 0x7f06cc1a6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.257+0000 7f06d0b7a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc120760 con 0x7f06cc10fed0 2024-05-09T08:44:22.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.257+0000 7f06d0b7a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc1205e0 con 0x7f06cc108830 2024-05-09T08:44:22.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.257+0000 7f06d0b7a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc1208e0 con 0x7f06cc1092b0 2024-05-09T08:44:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.258+0000 7f06caffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06cc1092b0 0x7f06cc1ace70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.258+0000 7f06caffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06cc1092b0 0x7f06cc1ace70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:57392/0 (socket says 172.21.15.107:57392) 2024-05-09T08:44:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.258+0000 7f06caffd640 1 -- 172.21.15.107:0/1244687133 learned_addr learned my addr 172.21.15.107:0/1244687133 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.258+0000 7f06cb7fe640 1 --2- 172.21.15.107:0/1244687133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06cc108830 0x7f06cc1ac930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.259+0000 7f06caffd640 1 -- 172.21.15.107:0/1244687133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06cc108830 msgr2=0x7f06cc1ac930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.259+0000 7f06caffd640 1 --2- 172.21.15.107:0/1244687133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06cc108830 0x7f06cc1ac930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.259+0000 7f06caffd640 1 -- 172.21.15.107:0/1244687133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06cc10fed0 msgr2=0x7f06cc1a6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.259+0000 7f06caffd640 1 --2- 172.21.15.107:0/1244687133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06cc10fed0 0x7f06cc1a6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.259+0000 7f06caffd640 1 -- 172.21.15.107:0/1244687133 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06cc1a72b0 con 0x7f06cc1092b0 2024-05-09T08:44:22.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.259+0000 7f06caffd640 1 --2- 172.21.15.107:0/1244687133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06cc1092b0 0x7f06cc1ace70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f06c402fe10 tx=0x7f06c402f970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.259+0000 7f06c8ff9640 1 -- 172.21.15.107:0/1244687133 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c4034070 con 0x7f06cc1092b0 2024-05-09T08:44:22.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.259+0000 7f06d0b7a640 1 -- 172.21.15.107:0/1244687133 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06cc1a7540 con 0x7f06cc1092b0 2024-05-09T08:44:22.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.259+0000 7f06d0b7a640 1 -- 172.21.15.107:0/1244687133 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06cc1107f0 con 0x7f06cc1092b0 2024-05-09T08:44:22.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.260+0000 7f06c8ff9640 1 -- 172.21.15.107:0/1244687133 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06c4004040 con 0x7f06cc1092b0 2024-05-09T08:44:22.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.260+0000 7f06c8ff9640 1 -- 172.21.15.107:0/1244687133 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c4039680 con 0x7f06cc1092b0 2024-05-09T08:44:22.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.268+0000 7f06d4bc6740 1 Processor -- start 2024-05-09T08:44:22.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.268+0000 7f06d4bc6740 1 -- start start 2024-05-09T08:44:22.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.268+0000 7f06d4bc6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b254f0 0x55cc29b258d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.268+0000 7f06d4bc6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 0x55cc29b2ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.268+0000 7f06d4bc6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b304e0 0x55cc29b328d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.268+0000 7f06d4bc6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cc29a8ac40 con 0x55cc29b304e0 2024-05-09T08:44:22.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.268+0000 7f06d4bc6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cc29a32580 con 0x55cc29b254f0 2024-05-09T08:44:22.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.268+0000 7f06d4bc6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55cc29a86c00 con 0x55cc29b25e10 2024-05-09T08:44:22.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.269+0000 7f06cbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b254f0 0x55cc29b258d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.269+0000 7f06cbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b254f0 0x55cc29b258d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:57758/0 (socket says 172.21.15.107:57758) 2024-05-09T08:44:22.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.269+0000 7f06cbfff640 1 -- 172.21.15.107:0/2838292847 learned_addr learned my addr 172.21.15.107:0/2838292847 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:22.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.269+0000 7f06cb7fe640 1 --2- 172.21.15.107:0/2838292847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 0x55cc29b2ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.269+0000 7f06cb7fe640 1 -- 172.21.15.107:0/2838292847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b254f0 msgr2=0x55cc29b258d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.269+0000 7f06cb7fe640 1 --2- 172.21.15.107:0/2838292847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b254f0 0x55cc29b258d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.269+0000 7f06cb7fe640 1 -- 172.21.15.107:0/2838292847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b304e0 msgr2=0x55cc29b328d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:44:22.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.269+0000 7f06cb7fe640 1 --2- 172.21.15.107:0/2838292847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b304e0 0x55cc29b328d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.269+0000 7f06cb7fe640 1 -- 172.21.15.107:0/2838292847 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc2979e3f0 con 0x55cc29b25e10 2024-05-09T08:44:22.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.270+0000 7f06cb7fe640 1 --2- 172.21.15.107:0/2838292847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 0x55cc29b2ffa0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f06bc00bb60 tx=0x7f06bc00f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f7ffd3af995bc33 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.270+0000 7f06b1ffb640 1 -- 172.21.15.107:0/2838292847 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc015070 con 0x55cc29b25e10 2024-05-09T08:44:22.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.270+0000 7f06b1ffb640 1 -- 172.21.15.107:0/2838292847 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06bc00be50 con 0x55cc29b25e10 2024-05-09T08:44:22.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.271+0000 7f06d4bc6740 1 -- 172.21.15.107:0/2838292847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 msgr2=0x55cc29b2ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.271+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/2838292847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 0x55cc29b2ffa0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f06bc00bb60 tx=0x7f06bc00f730 comp rx=0 tx=0).stop 2024-05-09T08:44:22.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.275+0000 7f06d4bc6740 1 -- 172.21.15.107:0/2838292847 shutdown_connections 2024-05-09T08:44:22.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.275+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/2838292847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b304e0 0x55cc29b328d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.275+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/2838292847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 0x55cc29b2ffa0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.275+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/2838292847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b254f0 0x55cc29b258d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.275+0000 7f06d4bc6740 1 -- 172.21.15.107:0/2838292847 >> 172.21.15.107:0/2838292847 conn(0x55cc29b21810 msgr2=0x55cc29b21bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.275+0000 7f06d4bc6740 1 -- 172.21.15.107:0/2838292847 shutdown_connections 2024-05-09T08:44:22.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.275+0000 7f06d4bc6740 1 -- 172.21.15.107:0/2838292847 wait complete. 2024-05-09T08:44:22.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.275+0000 7f06c8ff9640 1 -- 172.21.15.107:0/1244687133 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f06c4037020 con 0x7f06cc1092b0 2024-05-09T08:44:22.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.275+0000 7f06d4bc6740 1 -- start start 2024-05-09T08:44:22.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.276+0000 7f06d4bc6740 1 Processor -- start 2024-05-09T08:44:22.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.276+0000 7f06d4bc6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b254f0 0x55cc29b2d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.276+0000 7f06d4bc6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 0x55cc29b2d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.276+0000 7f06d4bc6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b304e0 0x55cc29b2de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.276+0000 7f06d4bc6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cc29b33370 con 0x55cc29b254f0 2024-05-09T08:44:22.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.276+0000 7f06d4bc6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cc29a45c40 con 0x55cc29b304e0 2024-05-09T08:44:22.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.276+0000 7f06d4bc6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55cc29b334f0 con 0x55cc29b25e10 2024-05-09T08:44:22.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.276+0000 7f06d4bc6740 10 client.0 _ll_register_callbacks cb 0x55cc2979dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:22.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.277+0000 7f06c8ff9640 1 --2- 172.21.15.107:0/1244687133 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f06b40a2ad0 0x7f06b40a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.278+0000 7f06c8ff9640 1 -- 172.21.15.107:0/1244687133 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7f06c40f05d0 con 0x7f06cc1092b0 2024-05-09T08:44:22.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.279+0000 7f06cbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b254f0 0x55cc29b2d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.279+0000 7f06cbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b254f0 0x55cc29b2d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58368/0 (socket says 172.21.15.107:58368) 2024-05-09T08:44:22.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.279+0000 7f06cbfff640 1 -- 172.21.15.107:0/1877925457 learned_addr learned my addr 172.21.15.107:0/1877925457 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:22.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.279+0000 7f06cb7fe640 1 --2- 172.21.15.107:0/1877925457 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 0x55cc29b2d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.279+0000 7f06caffd640 1 --2- 172.21.15.107:0/1877925457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b304e0 0x55cc29b2de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.282+0000 7f06cbfff640 1 --2- 172.21.15.107:0/1244687133 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f06b40a2ad0 0x7f06b40a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.282+0000 7f06cb7fe640 1 -- 172.21.15.107:0/1877925457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b304e0 msgr2=0x55cc29b2de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.282+0000 7f06cb7fe640 1 --2- 172.21.15.107:0/1877925457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b304e0 0x55cc29b2de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.282+0000 7f06cb7fe640 1 -- 172.21.15.107:0/1877925457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b254f0 msgr2=0x55cc29b2d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.282+0000 7f06cb7fe640 1 --2- 172.21.15.107:0/1877925457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b254f0 0x55cc29b2d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.282+0000 7f06cb7fe640 1 -- 172.21.15.107:0/1877925457 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc298687d0 con 0x55cc29b25e10 2024-05-09T08:44:22.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.285+0000 7f06cb7fe640 1 --2- 172.21.15.107:0/1877925457 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 0x55cc29b2d940 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f06bc000c00 tx=0x7f06bc00b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89d6fb6bdacf4a27 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.285+0000 7f06af7f6640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:44:22.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.285+0000 7f06af7f6640 1 -- 172.21.15.107:0/1877925457 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc015070 con 0x55cc29b25e10 2024-05-09T08:44:22.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.285+0000 7f06d4bc6740 1 -- 172.21.15.107:0/1877925457 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06bc00be50 con 0x55cc29b25e10 2024-05-09T08:44:22.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.285+0000 7f06af7f6640 1 -- 172.21.15.107:0/1877925457 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7f06bc00bc50 con 0x55cc29b25e10 2024-05-09T08:44:22.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.286+0000 7f06af7f6640 1 -- 172.21.15.107:0/1877925457 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f06bc003fe0 con 0x55cc29b25e10 2024-05-09T08:44:22.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.286+0000 7f06d4bc6740 10 client.27898 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.286+0000 7f06d4bc6740 1 -- 172.21.15.107:0/1877925457 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cc29b615e0 con 0x55cc29b25e10 2024-05-09T08:44:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.287+0000 7f06cbfff640 1 --2- 172.21.15.107:0/1244687133 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f06b40a2ad0 0x7f06b40a4f90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f06c0013540 tx=0x7f06c000a2d0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.287+0000 7f06af7f6640 1 -- 172.21.15.107:0/1877925457 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06bc00be50 con 0x55cc29b25e10 2024-05-09T08:44:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.287+0000 7f06af7f6640 1 -- 172.21.15.107:0/1877925457 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f06bc024420 con 0x55cc29b25e10 2024-05-09T08:44:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.287+0000 7f06d4bc6740 10 client.27898 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.287+0000 7f06d4bc6740 10 client.27898 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:44:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.287+0000 7f06d4bc6740 20 client.27898 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.287+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/1877925457 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cc29b62a40 0x55cc29b64e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.287+0000 7f06d4bc6740 4 client.27898 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:44:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.287+0000 7f06d4bc6740 1 -- 172.21.15.107:0/1877925457 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc29a98370 con 0x55cc29b62a40 2024-05-09T08:44:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.287+0000 7f06caffd640 1 --2- 172.21.15.107:0/1877925457 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cc29b62a40 0x55cc29b64e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.298+0000 7f06caffd640 1 --2- 172.21.15.107:0/1877925457 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cc29b62a40 0x55cc29b64e30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fdb273262685d2ff server_cookie=e229486c80022efc in_seq=0 out_seq=0 2024-05-09T08:44:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.299+0000 7f06af7f6640 10 client.27898 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:44:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.300+0000 7f06af7f6640 1 -- 172.21.15.107:0/1877925457 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cc29a98370 con 0x55cc29b62a40 2024-05-09T08:44:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.300+0000 7f06af7f6640 10 client.27898 handle_command_reply: tid=0 2024-05-09T08:44:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.301+0000 7f06d4bc6740 1 client.27898 shutdown 2024-05-09T08:44:22.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.310+0000 7f06d4bc6740 1 -- 172.21.15.107:0/1877925457 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 msgr2=0x55cc29b2d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.310+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/1877925457 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 0x55cc29b2d940 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f06bc000c00 tx=0x7f06bc00b8a0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.310+0000 7f06d4bc6740 1 -- 172.21.15.107:0/1877925457 shutdown_connections 2024-05-09T08:44:22.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.310+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/1877925457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b304e0 0x55cc29b2de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.310+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/1877925457 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cc29b62a40 0x55cc29b64e30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.310+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/1877925457 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 0x55cc29b2d940 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.310+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/1877925457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b254f0 0x55cc29b2d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.310+0000 7f06d4bc6740 1 -- 172.21.15.107:0/1877925457 >> 172.21.15.107:0/1877925457 conn(0x55cc29b21810 msgr2=0x55cc29b27610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.310+0000 7f06af7f6640 0 client.27898 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:44:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.319+0000 7f06d4bc6740 1 -- 172.21.15.107:0/1877925457 shutdown_connections 2024-05-09T08:44:22.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.322+0000 7f06d4bc6740 1 -- 172.21.15.107:0/1877925457 wait complete. 2024-05-09T08:44:22.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.328+0000 7f06d4bc6740 20 client.27898 trim_cache size 0 max 16384 2024-05-09T08:44:22.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.361+0000 7f06d4bc6740 1 Processor -- start 2024-05-09T08:44:22.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.361+0000 7f06d4bc6740 1 -- start start 2024-05-09T08:44:22.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.361+0000 7f06d4bc6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b254f0 0x55cc29b3c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.361+0000 7f06d4bc6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 0x55cc29b2dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.361+0000 7f06d4bc6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b304e0 0x55cc29b6a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.361+0000 7f06d4bc6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f06b8012a20 con 0x55cc29b254f0 2024-05-09T08:44:22.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.361+0000 7f06d4bc6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f06b80102f0 con 0x55cc29b304e0 2024-05-09T08:44:22.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.361+0000 7f06d4bc6740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f06b800e160 con 0x55cc29b25e10 2024-05-09T08:44:22.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.362+0000 7f06cbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b304e0 0x55cc29b6a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.362+0000 7f06cbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b304e0 0x55cc29b6a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:57780/0 (socket says 172.21.15.107:57780) 2024-05-09T08:44:22.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.362+0000 7f06cb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b254f0 0x55cc29b3c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.362+0000 7f06cbfff640 1 -- 172.21.15.107:0/3614389954 learned_addr learned my addr 172.21.15.107:0/3614389954 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:22.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.362+0000 7f06caffd640 1 --2- 172.21.15.107:0/3614389954 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 0x55cc29b2dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.362+0000 7f06cb7fe640 1 -- 172.21.15.107:0/3614389954 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 msgr2=0x55cc29b2dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.362+0000 7f06cb7fe640 1 --2- 172.21.15.107:0/3614389954 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 0x55cc29b2dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.363+0000 7f06cb7fe640 1 -- 172.21.15.107:0/3614389954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b304e0 msgr2=0x55cc29b6a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.363+0000 7f06cb7fe640 1 --2- 172.21.15.107:0/3614389954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b304e0 0x55cc29b6a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.363+0000 7f06cb7fe640 1 -- 172.21.15.107:0/3614389954 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc29b21640 con 0x55cc29b254f0 2024-05-09T08:44:22.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.365+0000 7f06cb7fe640 1 --2- 172.21.15.107:0/3614389954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b254f0 0x55cc29b3c310 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x55cc29b2e9d0 tx=0x7f06bc00fda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f74fa6e53f8478b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.366+0000 7f06b17fa640 1 -- 172.21.15.107:0/3614389954 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc015040 con 0x55cc29b254f0 2024-05-09T08:44:22.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.366+0000 7f06b17fa640 1 -- 172.21.15.107:0/3614389954 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06bc01d030 con 0x55cc29b254f0 2024-05-09T08:44:22.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.366+0000 7f06b17fa640 1 -- 172.21.15.107:0/3614389954 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc0039e0 con 0x55cc29b254f0 2024-05-09T08:44:22.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.367+0000 7f06d4bc6740 1 -- 172.21.15.107:0/3614389954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b254f0 msgr2=0x55cc29b3c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.367+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/3614389954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b254f0 0x55cc29b3c310 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x55cc29b2e9d0 tx=0x7f06bc00fda0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.374+0000 7f06d4bc6740 1 -- 172.21.15.107:0/3614389954 shutdown_connections 2024-05-09T08:44:22.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.374+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/3614389954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b304e0 0x55cc29b6a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.374+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/3614389954 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b25e10 0x55cc29b2dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.374+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/3614389954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b254f0 0x55cc29b3c310 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.374+0000 7f06d4bc6740 1 -- 172.21.15.107:0/3614389954 >> 172.21.15.107:0/3614389954 conn(0x55cc29b21810 msgr2=0x55cc29b28680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.374+0000 7f06d4bc6740 1 -- 172.21.15.107:0/3614389954 shutdown_connections 2024-05-09T08:44:22.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.374+0000 7f06d4bc6740 1 -- 172.21.15.107:0/3614389954 wait complete. 2024-05-09T08:44:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.375+0000 7f06d4bc6740 1 -- start start 2024-05-09T08:44:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.375+0000 7f06d4bc6740 1 Processor -- start 2024-05-09T08:44:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.375+0000 7f06d4bc6740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b254f0 0x55cc29b38840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.375+0000 7f06d4bc6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b25e10 0x55cc29b38d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.375+0000 7f06d4bc6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b304e0 0x55cc29b33db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.376+0000 7f06caffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b25e10 0x55cc29b38d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.376+0000 7f06d4bc6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f06b800e7b0 con 0x55cc29b25e10 2024-05-09T08:44:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.376+0000 7f06caffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b25e10 0x55cc29b38d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58384/0 (socket says 172.21.15.107:58384) 2024-05-09T08:44:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.376+0000 7f06caffd640 1 -- 172.21.15.107:0/3523969502 learned_addr learned my addr 172.21.15.107:0/3523969502 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.376+0000 7f06d4bc6740 1 -- 172.21.15.107:0/3523969502 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f06b800fc20 con 0x55cc29b304e0 2024-05-09T08:44:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.376+0000 7f06d4bc6740 1 -- 172.21.15.107:0/3523969502 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f06b800ee80 con 0x55cc29b254f0 2024-05-09T08:44:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.376+0000 7f06d4bc6740 10 client.0 _ll_register_callbacks cb 0x7f06c0005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.377+0000 7f06caffd640 1 -- 172.21.15.107:0/3523969502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b254f0 msgr2=0x55cc29b38840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.377+0000 7f06caffd640 1 --2- 172.21.15.107:0/3523969502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b254f0 0x55cc29b38840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.377+0000 7f06caffd640 1 -- 172.21.15.107:0/3523969502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b304e0 msgr2=0x55cc29b33db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.377+0000 7f06caffd640 1 --2- 172.21.15.107:0/3523969502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b304e0 0x55cc29b33db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.377+0000 7f06caffd640 1 -- 172.21.15.107:0/3523969502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc29b28170 con 0x55cc29b25e10 2024-05-09T08:44:22.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.384+0000 7f06caffd640 1 --2- 172.21.15.107:0/3523969502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b25e10 0x55cc29b38d80 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f06c40bc370 tx=0x7f06c402fc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4121188ac35a709 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.386+0000 7f06af7f6640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:44:22.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.386+0000 7f06af7f6640 1 -- 172.21.15.107:0/3523969502 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c40da060 con 0x55cc29b25e10 2024-05-09T08:44:22.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.386+0000 7f06af7f6640 1 -- 172.21.15.107:0/3523969502 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7f06c40edb20 con 0x55cc29b25e10 2024-05-09T08:44:22.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.391+0000 7f06d4bc6740 1 -- 172.21.15.107:0/3523969502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06bc01d030 con 0x55cc29b25e10 2024-05-09T08:44:22.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.404+0000 7f06af7f6640 1 -- 172.21.15.107:0/3523969502 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f06c40a0050 con 0x55cc29b25e10 2024-05-09T08:44:22.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.410+0000 7f06d4bc6740 10 client.18726 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:22.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.410+0000 7f06d4bc6740 1 -- 172.21.15.107:0/3523969502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cc29bc8910 con 0x55cc29b25e10 2024-05-09T08:44:22.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.416+0000 7f06af7f6640 1 -- 172.21.15.107:0/3523969502 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06bc01d030 con 0x55cc29b25e10 2024-05-09T08:44:22.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.418+0000 7f06af7f6640 1 -- 172.21.15.107:0/3523969502 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f06c4041020 con 0x55cc29b25e10 2024-05-09T08:44:22.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.418+0000 7f06d4bc6740 10 client.18726 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:22.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.418+0000 7f06d4bc6740 10 client.18726 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:44:22.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.418+0000 7f06d4bc6740 20 client.18726 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:22.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.418+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/3523969502 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cc29bc9110 0x55cc29bcb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.418+0000 7f06d4bc6740 4 client.18726 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:44:22.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.418+0000 7f06d4bc6740 1 -- 172.21.15.107:0/3523969502 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cc29a926c0 con 0x55cc29bc9110 2024-05-09T08:44:22.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.421+0000 7f06cb7fe640 1 --2- 172.21.15.107:0/3523969502 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cc29bc9110 0x55cc29bcb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.426+0000 7f06cb7fe640 1 --2- 172.21.15.107:0/3523969502 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cc29bc9110 0x55cc29bcb590 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f88d24df92aed831 server_cookie=660c7f6264cda93f in_seq=0 out_seq=0 2024-05-09T08:44:22.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.426+0000 7f06af7f6640 10 client.18726 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:44:22.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.426+0000 7f06af7f6640 1 -- 172.21.15.107:0/3523969502 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cc29a926c0 con 0x55cc29bc9110 2024-05-09T08:44:22.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.427+0000 7f06af7f6640 10 client.18726 handle_command_reply: tid=0 2024-05-09T08:44:22.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.427+0000 7f06d4bc6740 1 client.18726 shutdown 2024-05-09T08:44:22.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.427+0000 7f06d4bc6740 1 -- 172.21.15.107:0/3523969502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b25e10 msgr2=0x55cc29b38d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.427+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/3523969502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b25e10 0x55cc29b38d80 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f06c40bc370 tx=0x7f06c402fc50 comp rx=0 tx=0).stop 2024-05-09T08:44:22.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.432+0000 7f06d4bc6740 1 -- 172.21.15.107:0/3523969502 shutdown_connections 2024-05-09T08:44:22.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.432+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/3523969502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cc29b304e0 0x55cc29b33db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.432+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/3523969502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cc29b25e10 0x55cc29b38d80 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.432+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/3523969502 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cc29bc9110 0x55cc29bcb590 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.432+0000 7f06d4bc6740 1 --2- 172.21.15.107:0/3523969502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc29b254f0 0x55cc29b38840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.432+0000 7f06d4bc6740 1 -- 172.21.15.107:0/3523969502 >> 172.21.15.107:0/3523969502 conn(0x55cc29b21810 msgr2=0x55cc29b2ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.432+0000 7f06af7f6640 0 client.18726 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:44:22.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.437+0000 7f06d4bc6740 1 -- 172.21.15.107:0/3523969502 shutdown_connections 2024-05-09T08:44:22.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.437+0000 7f06d4bc6740 1 -- 172.21.15.107:0/3523969502 wait complete. 2024-05-09T08:44:22.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.439+0000 7f06d4bc6740 20 client.18726 trim_cache size 0 max 16384 2024-05-09T08:44:22.443 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:44:22.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.445+0000 7f06d0b7a640 1 -- 172.21.15.107:0/1244687133 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f06b40a2ad0 msgr2=0x7f06b40a4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.445+0000 7f06d0b7a640 1 --2- 172.21.15.107:0/1244687133 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f06b40a2ad0 0x7f06b40a4f90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f06c0013540 tx=0x7f06c000a2d0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.445+0000 7f06d0b7a640 1 -- 172.21.15.107:0/1244687133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06cc1092b0 msgr2=0x7f06cc1ace70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.445+0000 7f06d0b7a640 1 --2- 172.21.15.107:0/1244687133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06cc1092b0 0x7f06cc1ace70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f06c402fe10 tx=0x7f06c402f970 comp rx=0 tx=0).stop 2024-05-09T08:44:22.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.446+0000 7f06d0b7a640 1 -- 172.21.15.107:0/1244687133 shutdown_connections 2024-05-09T08:44:22.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.446+0000 7f06d0b7a640 1 --2- 172.21.15.107:0/1244687133 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f06b40a2ad0 0x7f06b40a4f90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.446+0000 7f06d0b7a640 1 --2- 172.21.15.107:0/1244687133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06cc10fed0 0x7f06cc1a6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.446+0000 7f06d0b7a640 1 --2- 172.21.15.107:0/1244687133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06cc1092b0 0x7f06cc1ace70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.446+0000 7f06d0b7a640 1 --2- 172.21.15.107:0/1244687133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06cc108830 0x7f06cc1ac930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.446+0000 7f06d0b7a640 1 -- 172.21.15.107:0/1244687133 >> 172.21.15.107:0/1244687133 conn(0x7f06cc06c7b0 msgr2=0x7f06cc0704b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.448+0000 7f06d0b7a640 1 -- 172.21.15.107:0/1244687133 shutdown_connections 2024-05-09T08:44:22.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.449+0000 7f06d0b7a640 1 -- 172.21.15.107:0/1244687133 wait complete. 2024-05-09T08:44:22.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:44:22.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.717+0000 7fd2c9347640 1 -- 172.21.15.107:0/2831487890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c4100680 msgr2=0x7fd2c4100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.717+0000 7fd2c9347640 1 --2- 172.21.15.107:0/2831487890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c4100680 0x7fd2c4100a60 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009be0 tx=0x7fd2bc02a780 comp rx=0 tx=0).stop 2024-05-09T08:44:22.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.717+0000 7fd2c9347640 1 -- 172.21.15.107:0/2831487890 shutdown_connections 2024-05-09T08:44:22.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.717+0000 7fd2c9347640 1 --2- 172.21.15.107:0/2831487890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2c4105a80 0x7fd2c4105f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.717+0000 7fd2c9347640 1 --2- 172.21.15.107:0/2831487890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2c4100fa0 0x7fd2c4105460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.717+0000 7fd2c9347640 1 --2- 172.21.15.107:0/2831487890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c4100680 0x7fd2c4100a60 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.717+0000 7fd2c9347640 1 -- 172.21.15.107:0/2831487890 >> 172.21.15.107:0/2831487890 conn(0x7fd2c40fc0b0 msgr2=0x7fd2c40fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.717+0000 7fd2c9347640 1 -- 172.21.15.107:0/2831487890 shutdown_connections 2024-05-09T08:44:22.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.720+0000 7fd2c9347640 1 -- 172.21.15.107:0/2831487890 wait complete. 2024-05-09T08:44:22.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.720+0000 7fd2c9347640 1 Processor -- start 2024-05-09T08:44:22.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.720+0000 7fd2c9347640 1 -- start start 2024-05-09T08:44:22.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.721+0000 7fd2c9347640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2c4100680 0x7fd2c4073260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.721+0000 7fd2c9347640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2c4105a80 0x7fd2c406c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.721+0000 7fd2c3fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2c4100680 0x7fd2c4073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.721+0000 7fd2c3fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2c4100680 0x7fd2c4073260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:57442/0 (socket says 172.21.15.107:57442) 2024-05-09T08:44:22.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.721+0000 7fd2c3fff640 1 -- 172.21.15.107:0/2417094963 learned_addr learned my addr 172.21.15.107:0/2417094963 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:22.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.721+0000 7fd2c9347640 1 --2- 172.21.15.107:0/2417094963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c406c820 0x7fd2c406ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.722+0000 7fd2c9347640 1 -- 172.21.15.107:0/2417094963 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c4117fd0 con 0x7fd2c406c820 2024-05-09T08:44:22.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.722+0000 7fd2c37fe640 1 --2- 172.21.15.107:0/2417094963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2c4105a80 0x7fd2c406c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.722+0000 7fd2c8b46640 1 --2- 172.21.15.107:0/2417094963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c406c820 0x7fd2c406ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.722+0000 7fd2c9347640 1 -- 172.21.15.107:0/2417094963 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c4117e50 con 0x7fd2c4105a80 2024-05-09T08:44:22.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.722+0000 7fd2c9347640 1 -- 172.21.15.107:0/2417094963 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c4118150 con 0x7fd2c4100680 2024-05-09T08:44:22.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.722+0000 7fd2c3fff640 1 -- 172.21.15.107:0/2417094963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2c4105a80 msgr2=0x7fd2c406c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.722+0000 7fd2c3fff640 1 --2- 172.21.15.107:0/2417094963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2c4105a80 0x7fd2c406c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.722+0000 7fd2c3fff640 1 -- 172.21.15.107:0/2417094963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c406c820 msgr2=0x7fd2c406ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.722+0000 7fd2c3fff640 1 --2- 172.21.15.107:0/2417094963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c406c820 0x7fd2c406ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.722+0000 7fd2c3fff640 1 -- 172.21.15.107:0/2417094963 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c4070eb0 con 0x7fd2c4100680 2024-05-09T08:44:22.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.723+0000 7fd2c3fff640 1 --2- 172.21.15.107:0/2417094963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2c4100680 0x7fd2c4073260 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc003f10 tx=0x7fd2bc003f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.723+0000 7fd2c17fa640 1 -- 172.21.15.107:0/2417094963 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc040070 con 0x7fd2c4100680 2024-05-09T08:44:22.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.723+0000 7fd2c9347640 1 -- 172.21.15.107:0/2417094963 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c4071140 con 0x7fd2c4100680 2024-05-09T08:44:22.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.723+0000 7fd2c9347640 1 -- 172.21.15.107:0/2417094963 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c40716e0 con 0x7fd2c4100680 2024-05-09T08:44:22.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.723+0000 7fd2c17fa640 1 -- 172.21.15.107:0/2417094963 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2bc031870 con 0x7fd2c4100680 2024-05-09T08:44:22.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.723+0000 7fd2c17fa640 1 -- 172.21.15.107:0/2417094963 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc0046f0 con 0x7fd2c4100680 2024-05-09T08:44:22.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.728+0000 7fd2c17fa640 1 -- 172.21.15.107:0/2417094963 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd2bc043070 con 0x7fd2c4100680 2024-05-09T08:44:22.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.732+0000 7fd2cd396740 1 Processor -- start 2024-05-09T08:44:22.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.733+0000 7fd2c17fa640 1 --2- 172.21.15.107:0/2417094963 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd2ac0a2e00 0x7fd2ac0a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.733+0000 7fd2c17fa640 1 -- 172.21.15.107:0/2417094963 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7fd2bc0f0250 con 0x7fd2c4100680 2024-05-09T08:44:22.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.733+0000 7fd2cd396740 1 -- start start 2024-05-09T08:44:22.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.734+0000 7fd2c8b46640 1 --2- 172.21.15.107:0/2417094963 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd2ac0a2e00 0x7fd2ac0a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.734+0000 7fd2cd396740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda66794f0 0x55fda66798d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.734+0000 7fd2cd396740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda6679e10 0x55fda6683fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.735+0000 7fd2cd396740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66954f0 0x55fda66978e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.735+0000 7fd2cd396740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fda65dec40 con 0x55fda66794f0 2024-05-09T08:44:22.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.735+0000 7fd2cd396740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55fda6586580 con 0x55fda66954f0 2024-05-09T08:44:22.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.735+0000 7fd2cd396740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55fda65dac00 con 0x55fda6679e10 2024-05-09T08:44:22.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.735+0000 7fd2c3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda66794f0 0x55fda66798d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.735+0000 7fd2c3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda66794f0 0x55fda66798d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58402/0 (socket says 172.21.15.107:58402) 2024-05-09T08:44:22.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.735+0000 7fd2c3fff640 1 -- 172.21.15.107:0/1017375652 learned_addr learned my addr 172.21.15.107:0/1017375652 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:22.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.735+0000 7fd2c8b46640 1 --2- 172.21.15.107:0/1017375652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66954f0 0x55fda66978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.735+0000 7fd2c37fe640 1 --2- 172.21.15.107:0/1017375652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda6679e10 0x55fda6683fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.735+0000 7fd2c8b46640 1 --2- 172.21.15.107:0/2417094963 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd2ac0a2e00 0x7fd2ac0a52c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd2b400bbc0 tx=0x7fd2b40103d0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.736+0000 7fd2c37fe640 1 -- 172.21.15.107:0/1017375652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66954f0 msgr2=0x55fda66978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.736+0000 7fd2c37fe640 1 --2- 172.21.15.107:0/1017375652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66954f0 0x55fda66978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.736+0000 7fd2c37fe640 1 -- 172.21.15.107:0/1017375652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda66794f0 msgr2=0x55fda66798d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.736+0000 7fd2c37fe640 1 --2- 172.21.15.107:0/1017375652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda66794f0 0x55fda66798d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.736+0000 7fd2c37fe640 1 -- 172.21.15.107:0/1017375652 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fda62f23f0 con 0x55fda6679e10 2024-05-09T08:44:22.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.736+0000 7fd2c37fe640 1 --2- 172.21.15.107:0/1017375652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda6679e10 0x55fda6683fa0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd2c406db30 tx=0x7fd2b8009ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61fd317be3842cf4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.737+0000 7fd2c8b46640 1 --2- 172.21.15.107:0/1017375652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66954f0 0x55fda66978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:44:22.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.738+0000 7fd2aa7fc640 1 -- 172.21.15.107:0/1017375652 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8017070 con 0x55fda6679e10 2024-05-09T08:44:22.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.738+0000 7fd2aa7fc640 1 -- 172.21.15.107:0/1017375652 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2b800ee40 con 0x55fda6679e10 2024-05-09T08:44:22.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.738+0000 7fd2aa7fc640 1 -- 172.21.15.107:0/1017375652 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b80123d0 con 0x55fda6679e10 2024-05-09T08:44:22.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.739+0000 7fd2cd396740 1 -- 172.21.15.107:0/1017375652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda6679e10 msgr2=0x55fda6683fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.739+0000 7fd2cd396740 1 --2- 172.21.15.107:0/1017375652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda6679e10 0x55fda6683fa0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd2c406db30 tx=0x7fd2b8009ce0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.740+0000 7fd2cd396740 1 -- 172.21.15.107:0/1017375652 shutdown_connections 2024-05-09T08:44:22.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.740+0000 7fd2cd396740 1 --2- 172.21.15.107:0/1017375652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66954f0 0x55fda66978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.740+0000 7fd2cd396740 1 --2- 172.21.15.107:0/1017375652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda6679e10 0x55fda6683fa0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.740+0000 7fd2cd396740 1 --2- 172.21.15.107:0/1017375652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda66794f0 0x55fda66798d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.740+0000 7fd2cd396740 1 -- 172.21.15.107:0/1017375652 >> 172.21.15.107:0/1017375652 conn(0x55fda6675810 msgr2=0x55fda6675bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.745+0000 7fd2cd396740 1 -- 172.21.15.107:0/1017375652 shutdown_connections 2024-05-09T08:44:22.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.749+0000 7fd2cd396740 1 -- 172.21.15.107:0/1017375652 wait complete. 2024-05-09T08:44:22.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.749+0000 7fd2cd396740 1 -- start start 2024-05-09T08:44:22.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.749+0000 7fd2cd396740 1 Processor -- start 2024-05-09T08:44:22.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.750+0000 7fd2cd396740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66794f0 0x55fda6681410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.750+0000 7fd2cd396740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda6679e10 0x55fda6681950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.750+0000 7fd2cd396740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda66954f0 0x55fda6681e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.750+0000 7fd2cd396740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fda6698330 con 0x55fda66954f0 2024-05-09T08:44:22.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.750+0000 7fd2cd396740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55fda6599c40 con 0x55fda66794f0 2024-05-09T08:44:22.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.750+0000 7fd2cd396740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55fda66984b0 con 0x55fda6679e10 2024-05-09T08:44:22.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.750+0000 7fd2cd396740 10 client.0 _ll_register_callbacks cb 0x55fda62f1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:22.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.751+0000 7fd2c3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66794f0 0x55fda6681410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.751+0000 7fd2c3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66794f0 0x55fda6681410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:57812/0 (socket says 172.21.15.107:57812) 2024-05-09T08:44:22.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.751+0000 7fd2c3fff640 1 -- 172.21.15.107:0/708328218 learned_addr learned my addr 172.21.15.107:0/708328218 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:22.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.752+0000 7fd2c8b46640 1 --2- 172.21.15.107:0/708328218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda66954f0 0x55fda6681e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.753+0000 7fd2c3fff640 1 -- 172.21.15.107:0/708328218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda6679e10 msgr2=0x55fda6681950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:44:22.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.753+0000 7fd2c3fff640 1 --2- 172.21.15.107:0/708328218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda6679e10 0x55fda6681950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.753+0000 7fd2c3fff640 1 -- 172.21.15.107:0/708328218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda66954f0 msgr2=0x55fda6681e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.753+0000 7fd2c3fff640 1 --2- 172.21.15.107:0/708328218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda66954f0 0x55fda6681e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.753+0000 7fd2c3fff640 1 -- 172.21.15.107:0/708328218 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fda63bc7d0 con 0x55fda66794f0 2024-05-09T08:44:22.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.753+0000 7fd2c8b46640 1 --2- 172.21.15.107:0/708328218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda66954f0 0x55fda6681e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:44:22.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.755+0000 7fd2c3fff640 1 --2- 172.21.15.107:0/708328218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66794f0 0x55fda6681410 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0c4b40 tx=0x7fd2bc0bbe10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6afcb8edbfb0b323 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.755+0000 7fd2a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:44:22.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.755+0000 7fd2a37fe640 1 -- 172.21.15.107:0/708328218 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc0d5060 con 0x55fda66794f0 2024-05-09T08:44:22.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.755+0000 7fd2a37fe640 1 -- 172.21.15.107:0/708328218 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7fd2bc0ed7a0 con 0x55fda66794f0 2024-05-09T08:44:22.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.757+0000 7fd2cd396740 1 -- 172.21.15.107:0/708328218 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2b800ee40 con 0x55fda66794f0 2024-05-09T08:44:22.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.760+0000 7fd2a37fe640 1 -- 172.21.15.107:0/708328218 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd2bc0da050 con 0x55fda66794f0 2024-05-09T08:44:22.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.760+0000 7fd2cd396740 10 client.27680 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:22.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.760+0000 7fd2cd396740 1 -- 172.21.15.107:0/708328218 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fda66a4560 con 0x55fda66794f0 2024-05-09T08:44:22.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.760+0000 7fd2a37fe640 1 -- 172.21.15.107:0/708328218 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2b800ee40 con 0x55fda66794f0 2024-05-09T08:44:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.760+0000 7fd2a37fe640 1 -- 172.21.15.107:0/708328218 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd2bc0bba20 con 0x55fda66794f0 2024-05-09T08:44:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.760+0000 7fd2cd396740 10 client.27680 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.760+0000 7fd2cd396740 10 client.27680 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.760+0000 7fd2cd396740 20 client.27680 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.760+0000 7fd2cd396740 1 --2- 172.21.15.107:0/708328218 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fda66a5cf0 0x55fda66a8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.760+0000 7fd2cd396740 4 client.27680 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:44:22.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.760+0000 7fd2cd396740 1 -- 172.21.15.107:0/708328218 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fda65ec370 con 0x55fda66a5cf0 2024-05-09T08:44:22.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.761+0000 7fd2c37fe640 1 --2- 172.21.15.107:0/708328218 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fda66a5cf0 0x55fda66a8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.761+0000 7fd2c37fe640 1 --2- 172.21.15.107:0/708328218 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fda66a5cf0 0x55fda66a8170 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7fc1d3571d64000 server_cookie=4b2fdc3d398a345a in_seq=0 out_seq=0 2024-05-09T08:44:22.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.762+0000 7fd2a37fe640 10 client.27680 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:22.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.762+0000 7fd2a37fe640 1 -- 172.21.15.107:0/708328218 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fda65ec370 con 0x55fda66a5cf0 2024-05-09T08:44:22.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.762+0000 7fd2a37fe640 10 client.27680 handle_command_reply: tid=0 2024-05-09T08:44:22.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.762+0000 7fd2cd396740 1 client.27680 shutdown 2024-05-09T08:44:22.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.763+0000 7fd2cd396740 1 -- 172.21.15.107:0/708328218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66794f0 msgr2=0x55fda6681410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.763+0000 7fd2cd396740 1 --2- 172.21.15.107:0/708328218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66794f0 0x55fda6681410 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0c4b40 tx=0x7fd2bc0bbe10 comp rx=0 tx=0).stop 2024-05-09T08:44:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.763+0000 7fd2cd396740 1 -- 172.21.15.107:0/708328218 shutdown_connections 2024-05-09T08:44:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.763+0000 7fd2cd396740 1 --2- 172.21.15.107:0/708328218 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fda66a5cf0 0x55fda66a8170 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.763+0000 7fd2cd396740 1 --2- 172.21.15.107:0/708328218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda66954f0 0x55fda6681e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.763+0000 7fd2cd396740 1 --2- 172.21.15.107:0/708328218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda6679e10 0x55fda6681950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.763+0000 7fd2cd396740 1 --2- 172.21.15.107:0/708328218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66794f0 0x55fda6681410 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.763+0000 7fd2cd396740 1 -- 172.21.15.107:0/708328218 >> 172.21.15.107:0/708328218 conn(0x55fda6675810 msgr2=0x55fda667b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.763+0000 7fd2a37fe640 0 client.27680 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.763+0000 7fd2cd396740 1 -- 172.21.15.107:0/708328218 shutdown_connections 2024-05-09T08:44:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.763+0000 7fd2cd396740 1 -- 172.21.15.107:0/708328218 wait complete. 2024-05-09T08:44:22.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.764+0000 7fd2cd396740 20 client.27680 trim_cache size 0 max 16384 2024-05-09T08:44:22.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:22 smithi107 ceph-mon[30122]: pgmap v342: 129 pgs: 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 120 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.1 MiB/s wr, 1.76k op/s 2024-05-09T08:44:22.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:22 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/4091944798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:44:22.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.799+0000 7fd2cd396740 1 Processor -- start 2024-05-09T08:44:22.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.800+0000 7fd2cd396740 1 -- start start 2024-05-09T08:44:22.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.800+0000 7fd2cd396740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda66794f0 0x55fda66a1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.800+0000 7fd2cd396740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda6679e10 0x55fda667bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.800+0000 7fd2cd396740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66954f0 0x55fda6674810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.800+0000 7fd2cd396740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b0012990 con 0x55fda6679e10 2024-05-09T08:44:22.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.800+0000 7fd2cd396740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b0010260 con 0x55fda66954f0 2024-05-09T08:44:22.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.800+0000 7fd2cd396740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b000e0d0 con 0x55fda66794f0 2024-05-09T08:44:22.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.801+0000 7fd2c37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda6679e10 0x55fda667bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.801+0000 7fd2c37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda6679e10 0x55fda667bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58416/0 (socket says 172.21.15.107:58416) 2024-05-09T08:44:22.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.801+0000 7fd2c37fe640 1 -- 172.21.15.107:0/4016896279 learned_addr learned my addr 172.21.15.107:0/4016896279 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.801+0000 7fd2c8b46640 1 --2- 172.21.15.107:0/4016896279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66954f0 0x55fda6674810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.801+0000 7fd2c37fe640 1 -- 172.21.15.107:0/4016896279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda66794f0 msgr2=0x55fda66a1280 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:44:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.801+0000 7fd2c37fe640 1 --2- 172.21.15.107:0/4016896279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda66794f0 0x55fda66a1280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.801+0000 7fd2c37fe640 1 -- 172.21.15.107:0/4016896279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66954f0 msgr2=0x55fda6674810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.802+0000 7fd2c37fe640 1 --2- 172.21.15.107:0/4016896279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66954f0 0x55fda6674810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.802+0000 7fd2c37fe640 1 -- 172.21.15.107:0/4016896279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fda667a360 con 0x55fda6679e10 2024-05-09T08:44:22.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.802+0000 7fd2c8b46640 1 --2- 172.21.15.107:0/4016896279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66954f0 0x55fda6674810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:44:22.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.804+0000 7fd2c37fe640 1 --2- 172.21.15.107:0/4016896279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda6679e10 0x55fda667bc50 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x55fda667aab0 tx=0x7fd2b801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48206c1570af8f4c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.808+0000 7fd2a9ffb640 1 -- 172.21.15.107:0/4016896279 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8017040 con 0x55fda6679e10 2024-05-09T08:44:22.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.808+0000 7fd2a9ffb640 1 -- 172.21.15.107:0/4016896279 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2b800e990 con 0x55fda6679e10 2024-05-09T08:44:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.809+0000 7fd2cd396740 1 -- 172.21.15.107:0/4016896279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda6679e10 msgr2=0x55fda667bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.809+0000 7fd2cd396740 1 --2- 172.21.15.107:0/4016896279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda6679e10 0x55fda667bc50 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x55fda667aab0 tx=0x7fd2b801e040 comp rx=0 tx=0).stop 2024-05-09T08:44:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.809+0000 7fd2cd396740 1 -- 172.21.15.107:0/4016896279 shutdown_connections 2024-05-09T08:44:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.809+0000 7fd2cd396740 1 --2- 172.21.15.107:0/4016896279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66954f0 0x55fda6674810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.809+0000 7fd2cd396740 1 --2- 172.21.15.107:0/4016896279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda6679e10 0x55fda667bc50 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.809+0000 7fd2cd396740 1 --2- 172.21.15.107:0/4016896279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda66794f0 0x55fda66a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.809+0000 7fd2cd396740 1 -- 172.21.15.107:0/4016896279 >> 172.21.15.107:0/4016896279 conn(0x55fda6675810 msgr2=0x55fda6674100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.810+0000 7fd2cd396740 1 -- 172.21.15.107:0/4016896279 shutdown_connections 2024-05-09T08:44:22.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.810+0000 7fd2cd396740 1 -- 172.21.15.107:0/4016896279 wait complete. 2024-05-09T08:44:22.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.810+0000 7fd2cd396740 1 -- start start 2024-05-09T08:44:22.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.811+0000 7fd2cd396740 1 Processor -- start 2024-05-09T08:44:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.811+0000 7fd2cd396740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66794f0 0x55fda6698940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.811+0000 7fd2cd396740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda66954f0 0x55fda6698e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.811+0000 7fd2cd396740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda67082d0 0x55fda670a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.811+0000 7fd2cd396740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b000e720 con 0x55fda66954f0 2024-05-09T08:44:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.811+0000 7fd2cd396740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b000fb90 con 0x55fda66794f0 2024-05-09T08:44:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.811+0000 7fd2cd396740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b000edf0 con 0x55fda67082d0 2024-05-09T08:44:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.811+0000 7fd2cd396740 10 client.0 _ll_register_callbacks cb 0x7fd2bc0b8840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.812+0000 7fd2c8b46640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda67082d0 0x55fda670a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.812+0000 7fd2c8b46640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda67082d0 0x55fda670a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:57484/0 (socket says 172.21.15.107:57484) 2024-05-09T08:44:22.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.812+0000 7fd2c8b46640 1 -- 172.21.15.107:0/1779378823 learned_addr learned my addr 172.21.15.107:0/1779378823 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:22.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.812+0000 7fd2c3fff640 1 --2- 172.21.15.107:0/1779378823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66794f0 0x55fda6698940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.812+0000 7fd2c8b46640 1 -- 172.21.15.107:0/1779378823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66794f0 msgr2=0x55fda6698940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.812+0000 7fd2c8b46640 1 --2- 172.21.15.107:0/1779378823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66794f0 0x55fda6698940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.812+0000 7fd2c8b46640 1 -- 172.21.15.107:0/1779378823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda66954f0 msgr2=0x55fda6698e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:22.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.812+0000 7fd2c8b46640 1 --2- 172.21.15.107:0/1779378823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda66954f0 0x55fda6698e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.812+0000 7fd2c8b46640 1 -- 172.21.15.107:0/1779378823 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fda670ac00 con 0x55fda67082d0 2024-05-09T08:44:22.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.813+0000 7fd2c8b46640 1 --2- 172.21.15.107:0/1779378823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda67082d0 0x55fda670a6c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd2b400b950 tx=0x7fd2b4015070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5fb54c9b70ca840 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:22.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.813+0000 7fd2a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:44:22.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.813+0000 7fd2a37fe640 1 -- 172.21.15.107:0/1779378823 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b402e060 con 0x55fda67082d0 2024-05-09T08:44:22.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.813+0000 7fd2cd396740 1 -- 172.21.15.107:0/1779378823 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fda65e66c0 con 0x55fda67082d0 2024-05-09T08:44:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.814+0000 7fd2a37fe640 1 -- 172.21.15.107:0/1779378823 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7fd2b4029620 con 0x55fda67082d0 2024-05-09T08:44:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.814+0000 7fd2a37fe640 1 -- 172.21.15.107:0/1779378823 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd2b4035050 con 0x55fda67082d0 2024-05-09T08:44:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.814+0000 7fd2cd396740 10 client.27913 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.814+0000 7fd2cd396740 1 -- 172.21.15.107:0/1779378823 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fda670c000 con 0x55fda67082d0 2024-05-09T08:44:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.815+0000 7fd2a37fe640 1 -- 172.21.15.107:0/1779378823 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55fda65e66c0 con 0x55fda67082d0 2024-05-09T08:44:22.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.815+0000 7fd2a37fe640 1 -- 172.21.15.107:0/1779378823 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fd2b401f760 con 0x55fda67082d0 2024-05-09T08:44:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.815+0000 7fd2cd396740 10 client.27913 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.815+0000 7fd2cd396740 10 client.27913 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.815+0000 7fd2cd396740 20 client.27913 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.815+0000 7fd2cd396740 1 --2- 172.21.15.107:0/1779378823 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fda670c7e0 0x55fda670ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.815+0000 7fd2cd396740 4 client.27913 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:44:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.815+0000 7fd2cd396740 1 -- 172.21.15.107:0/1779378823 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fda6563820 con 0x55fda670c7e0 2024-05-09T08:44:22.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.817+0000 7fd2c3fff640 1 --2- 172.21.15.107:0/1779378823 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fda670c7e0 0x55fda670ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:22.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.820+0000 7fd2c3fff640 1 --2- 172.21.15.107:0/1779378823 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fda670c7e0 0x55fda670ec00 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7f9dea58b2bd63e server_cookie=8751493d35fbb8fa in_seq=0 out_seq=0 2024-05-09T08:44:22.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.820+0000 7fd2a37fe640 10 client.27913 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:22.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.823+0000 7fd2a37fe640 1 -- 172.21.15.107:0/1779378823 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fda6563820 con 0x55fda670c7e0 2024-05-09T08:44:22.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.823+0000 7fd2a37fe640 10 client.27913 handle_command_reply: tid=0 2024-05-09T08:44:22.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.823+0000 7fd2cd396740 1 client.27913 shutdown 2024-05-09T08:44:22.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.824+0000 7fd2cd396740 1 -- 172.21.15.107:0/1779378823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda67082d0 msgr2=0x55fda670a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.824+0000 7fd2cd396740 1 --2- 172.21.15.107:0/1779378823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda67082d0 0x55fda670a6c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd2b400b950 tx=0x7fd2b4015070 comp rx=0 tx=0).stop 2024-05-09T08:44:22.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.825+0000 7fd2cd396740 1 -- 172.21.15.107:0/1779378823 shutdown_connections 2024-05-09T08:44:22.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.825+0000 7fd2cd396740 1 --2- 172.21.15.107:0/1779378823 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fda670c7e0 0x55fda670ec00 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.825+0000 7fd2cd396740 1 --2- 172.21.15.107:0/1779378823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fda67082d0 0x55fda670a6c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.825+0000 7fd2cd396740 1 --2- 172.21.15.107:0/1779378823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fda66954f0 0x55fda6698e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.825+0000 7fd2cd396740 1 --2- 172.21.15.107:0/1779378823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fda66794f0 0x55fda6698940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.825+0000 7fd2cd396740 1 -- 172.21.15.107:0/1779378823 >> 172.21.15.107:0/1779378823 conn(0x55fda6675810 msgr2=0x55fda6674990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.825+0000 7fd2a37fe640 0 client.27913 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:22.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.825+0000 7fd2cd396740 1 -- 172.21.15.107:0/1779378823 shutdown_connections 2024-05-09T08:44:22.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.825+0000 7fd2cd396740 1 -- 172.21.15.107:0/1779378823 wait complete. 2024-05-09T08:44:22.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.825+0000 7fd2cd396740 20 client.27913 trim_cache size 0 max 16384 2024-05-09T08:44:22.826 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:44:22.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.829+0000 7fd2c9347640 1 -- 172.21.15.107:0/2417094963 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd2ac0a2e00 msgr2=0x7fd2ac0a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.829+0000 7fd2c9347640 1 --2- 172.21.15.107:0/2417094963 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd2ac0a2e00 0x7fd2ac0a52c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd2b400bbc0 tx=0x7fd2b40103d0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.829+0000 7fd2c9347640 1 -- 172.21.15.107:0/2417094963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2c4100680 msgr2=0x7fd2c4073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:22.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.830+0000 7fd2c9347640 1 --2- 172.21.15.107:0/2417094963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2c4100680 0x7fd2c4073260 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc003f10 tx=0x7fd2bc003f40 comp rx=0 tx=0).stop 2024-05-09T08:44:22.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.832+0000 7fd2c9347640 1 -- 172.21.15.107:0/2417094963 shutdown_connections 2024-05-09T08:44:22.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.832+0000 7fd2c9347640 1 --2- 172.21.15.107:0/2417094963 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fd2ac0a2e00 0x7fd2ac0a52c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.832+0000 7fd2c9347640 1 --2- 172.21.15.107:0/2417094963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c406c820 0x7fd2c406ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.832+0000 7fd2c9347640 1 --2- 172.21.15.107:0/2417094963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2c4105a80 0x7fd2c406c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.832+0000 7fd2c9347640 1 --2- 172.21.15.107:0/2417094963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2c4100680 0x7fd2c4073260 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:22.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.832+0000 7fd2c9347640 1 -- 172.21.15.107:0/2417094963 >> 172.21.15.107:0/2417094963 conn(0x7fd2c40fc0b0 msgr2=0x7fd2c40fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:22.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.835+0000 7fd2c9347640 1 -- 172.21.15.107:0/2417094963 shutdown_connections 2024-05-09T08:44:22.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:22.836+0000 7fd2c9347640 1 -- 172.21.15.107:0/2417094963 wait complete. 2024-05-09T08:44:23.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:22 smithi157 ceph-mon[33614]: pgmap v342: 129 pgs: 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 120 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.1 MiB/s wr, 1.76k op/s 2024-05-09T08:44:23.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:22 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/4091944798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:44:23.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:22 smithi110 ceph-mon[33622]: pgmap v342: 129 pgs: 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 120 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.1 MiB/s wr, 1.76k op/s 2024-05-09T08:44:23.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:22 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/4091944798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:44:23.861 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:44:23.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:44:23.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:44:24.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.104+0000 7f8fd8adf640 1 -- 172.21.15.107:0/1543184058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fd4106730 msgr2=0x7f8fd4106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.104+0000 7f8fd8adf640 1 --2- 172.21.15.107:0/1543184058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fd4106730 0x7f8fd4106b10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009900 tx=0x7f8fc402a480 comp rx=0 tx=0).stop 2024-05-09T08:44:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.105+0000 7f8fd8adf640 1 -- 172.21.15.107:0/1543184058 shutdown_connections 2024-05-09T08:44:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.105+0000 7f8fd8adf640 1 --2- 172.21.15.107:0/1543184058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fd41017e0 0x7f8fd410d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.105+0000 7f8fd8adf640 1 --2- 172.21.15.107:0/1543184058 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fd4100e40 0x7f8fd41012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.105+0000 7f8fd8adf640 1 --2- 172.21.15.107:0/1543184058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fd4106730 0x7f8fd4106b10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.105+0000 7f8fd8adf640 1 -- 172.21.15.107:0/1543184058 >> 172.21.15.107:0/1543184058 conn(0x7f8fd4076cf0 msgr2=0x7f8fd40fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.105+0000 7f8fd8adf640 1 -- 172.21.15.107:0/1543184058 shutdown_connections 2024-05-09T08:44:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.105+0000 7f8fd8adf640 1 -- 172.21.15.107:0/1543184058 wait complete. 2024-05-09T08:44:24.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.106+0000 7f8fd8adf640 1 Processor -- start 2024-05-09T08:44:24.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.106+0000 7f8fd8adf640 1 -- start start 2024-05-09T08:44:24.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.107+0000 7f8fd8adf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fd4100e40 0x7f8fd4198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.107+0000 7f8fd8adf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fd41017e0 0x7f8fd4198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.107+0000 7f8fd259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fd4100e40 0x7f8fd4198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.107+0000 7f8fd259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fd4100e40 0x7f8fd4198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58442/0 (socket says 172.21.15.107:58442) 2024-05-09T08:44:24.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.107+0000 7f8fd259c640 1 -- 172.21.15.107:0/2900808047 learned_addr learned my addr 172.21.15.107:0/2900808047 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:24.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.107+0000 7f8fd1d9b640 1 --2- 172.21.15.107:0/2900808047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fd41017e0 0x7f8fd4198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.107+0000 7f8fd8adf640 1 --2- 172.21.15.107:0/2900808047 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fd4106730 0x7f8fd419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.107+0000 7f8fd8adf640 1 -- 172.21.15.107:0/2900808047 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd4111a90 con 0x7f8fd4100e40 2024-05-09T08:44:24.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.107+0000 7f8fd8adf640 1 -- 172.21.15.107:0/2900808047 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd4111910 con 0x7f8fd41017e0 2024-05-09T08:44:24.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.107+0000 7f8fd8adf640 1 -- 172.21.15.107:0/2900808047 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd4111c10 con 0x7f8fd4106730 2024-05-09T08:44:24.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.107+0000 7f8fd2d9d640 1 --2- 172.21.15.107:0/2900808047 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fd4106730 0x7f8fd419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.108+0000 7f8fd1d9b640 1 -- 172.21.15.107:0/2900808047 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fd4106730 msgr2=0x7f8fd419c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.108+0000 7f8fd1d9b640 1 --2- 172.21.15.107:0/2900808047 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fd4106730 0x7f8fd419c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.108+0000 7f8fd1d9b640 1 -- 172.21.15.107:0/2900808047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fd4100e40 msgr2=0x7f8fd4198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.108+0000 7f8fd1d9b640 1 --2- 172.21.15.107:0/2900808047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fd4100e40 0x7f8fd4198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.108+0000 7f8fd1d9b640 1 -- 172.21.15.107:0/2900808047 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd419d040 con 0x7f8fd41017e0 2024-05-09T08:44:24.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.108+0000 7f8fd1d9b640 1 --2- 172.21.15.107:0/2900808047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fd41017e0 0x7f8fd4198590 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800e9f0 tx=0x7f8fc800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.108+0000 7f8fc37fe640 1 -- 172.21.15.107:0/2900808047 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc800ccc0 con 0x7f8fd41017e0 2024-05-09T08:44:24.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.109+0000 7f8fd8adf640 1 -- 172.21.15.107:0/2900808047 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd419d330 con 0x7f8fd41017e0 2024-05-09T08:44:24.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.109+0000 7f8fc37fe640 1 -- 172.21.15.107:0/2900808047 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fc8005020 con 0x7f8fd41017e0 2024-05-09T08:44:24.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.109+0000 7f8fc37fe640 1 -- 172.21.15.107:0/2900808047 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8004a00 con 0x7f8fd41017e0 2024-05-09T08:44:24.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.109+0000 7f8fd8adf640 1 -- 172.21.15.107:0/2900808047 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd41a4bc0 con 0x7f8fd41017e0 2024-05-09T08:44:24.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.112+0000 7f8fc37fe640 1 -- 172.21.15.107:0/2900808047 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8fc8004ba0 con 0x7f8fd41017e0 2024-05-09T08:44:24.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.112+0000 7f8fdb7e0740 1 Processor -- start 2024-05-09T08:44:24.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.112+0000 7f8fdb7e0740 1 -- start start 2024-05-09T08:44:24.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fdb7e0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c730b0 0x558f19c713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fdb7e0740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c719c0 0x558f19c71e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fdb7e0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c7c740 0x558f19c7eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fdb7e0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558f19bd6990 con 0x558f19c730b0 2024-05-09T08:44:24.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fdb7e0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558f19b7e5b0 con 0x558f19c7c740 2024-05-09T08:44:24.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fdb7e0740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x558f19bd2950 con 0x558f19c719c0 2024-05-09T08:44:24.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fd259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c719c0 0x558f19c71e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fc37fe640 1 --2- 172.21.15.107:0/2900808047 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fbc0a2e00 0x7f8fbc0a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fd259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c719c0 0x558f19c71e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:57510/0 (socket says 172.21.15.107:57510) 2024-05-09T08:44:24.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fd259c640 1 -- 172.21.15.107:0/2701327984 learned_addr learned my addr 172.21.15.107:0/2701327984 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:24.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fd2d9d640 1 --2- 172.21.15.107:0/2701327984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c730b0 0x558f19c713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fd1d9b640 1 --2- 172.21.15.107:0/2701327984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c7c740 0x558f19c7eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fd2d9d640 1 --2- 172.21.15.107:0/2900808047 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fbc0a2e00 0x7f8fbc0a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fd259c640 1 -- 172.21.15.107:0/2701327984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c7c740 msgr2=0x558f19c7eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fc37fe640 1 -- 172.21.15.107:0/2900808047 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7f8fc80cf690 con 0x7f8fd41017e0 2024-05-09T08:44:24.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fd259c640 1 --2- 172.21.15.107:0/2701327984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c7c740 0x558f19c7eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fd259c640 1 -- 172.21.15.107:0/2701327984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c730b0 msgr2=0x558f19c713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fd259c640 1 --2- 172.21.15.107:0/2701327984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c730b0 0x558f19c713f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.113+0000 7f8fd259c640 1 -- 172.21.15.107:0/2701327984 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f199b27e0 con 0x558f19c719c0 2024-05-09T08:44:24.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.114+0000 7f8fd259c640 1 --2- 172.21.15.107:0/2701327984 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c719c0 0x558f19c71e40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009a20 tx=0x7f8fc4031480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=daa47629dc0516e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.114+0000 7f8fd2d9d640 1 --2- 172.21.15.107:0/2900808047 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fbc0a2e00 0x7f8fbc0a52c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc0094d0 tx=0x7f8fcc004ab0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.114+0000 7f8fc0ff9640 1 -- 172.21.15.107:0/2701327984 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4035040 con 0x558f19c719c0 2024-05-09T08:44:24.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.114+0000 7f8fc0ff9640 1 -- 172.21.15.107:0/2701327984 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fc4004380 con 0x558f19c719c0 2024-05-09T08:44:24.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.114+0000 7f8fc0ff9640 1 -- 172.21.15.107:0/2701327984 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4004e30 con 0x558f19c719c0 2024-05-09T08:44:24.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.115+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2701327984 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c719c0 msgr2=0x558f19c71e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.115+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/2701327984 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c719c0 0x558f19c71e40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009a20 tx=0x7f8fc4031480 comp rx=0 tx=0).stop 2024-05-09T08:44:24.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.115+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2701327984 shutdown_connections 2024-05-09T08:44:24.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.115+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/2701327984 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c7c740 0x558f19c7eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.115+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/2701327984 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c719c0 0x558f19c71e40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.115+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/2701327984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c730b0 0x558f19c713f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.115+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2701327984 >> 172.21.15.107:0/2701327984 conn(0x558f19c3a0a0 msgr2=0x558f19c6df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.115+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2701327984 shutdown_connections 2024-05-09T08:44:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.115+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2701327984 wait complete. 2024-05-09T08:44:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.115+0000 7f8fdb7e0740 1 -- start start 2024-05-09T08:44:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fdb7e0740 1 Processor -- start 2024-05-09T08:44:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fdb7e0740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c719c0 0x558f19c795f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fdb7e0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c730b0 0x558f19c79b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fdb7e0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c7c740 0x558f19c7a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fdb7e0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558f19c7f570 con 0x558f19c7c740 2024-05-09T08:44:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fdb7e0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558f19b91b40 con 0x558f19c730b0 2024-05-09T08:44:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fdb7e0740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x558f19c7f6f0 con 0x558f19c719c0 2024-05-09T08:44:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fdb7e0740 10 client.0 _ll_register_callbacks cb 0x558f19b860f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fd2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c7c740 0x558f19c7a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fd2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c7c740 0x558f19c7a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58468/0 (socket says 172.21.15.107:58468) 2024-05-09T08:44:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fd2d9d640 1 -- 172.21.15.107:0/2943555264 learned_addr learned my addr 172.21.15.107:0/2943555264 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fd1d9b640 1 --2- 172.21.15.107:0/2943555264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c730b0 0x558f19c79b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fd2d9d640 1 -- 172.21.15.107:0/2943555264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c719c0 msgr2=0x558f19c795f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:44:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fd259c640 1 --2- 172.21.15.107:0/2943555264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c719c0 0x558f19c795f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fd2d9d640 1 --2- 172.21.15.107:0/2943555264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c719c0 0x558f19c795f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fd2d9d640 1 -- 172.21.15.107:0/2943555264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c730b0 msgr2=0x558f19c79b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fd2d9d640 1 --2- 172.21.15.107:0/2943555264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c730b0 0x558f19c79b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fd2d9d640 1 -- 172.21.15.107:0/2943555264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f19a12f70 con 0x558f19c7c740 2024-05-09T08:44:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.116+0000 7f8fd1d9b640 1 --2- 172.21.15.107:0/2943555264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c730b0 0x558f19c79b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:44:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.117+0000 7f8fd2d9d640 1 --2- 172.21.15.107:0/2943555264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c7c740 0x558f19c7a070 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00ba90 tx=0x7f8fcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a58cceb0e87d99b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.117+0000 7f8fb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:44:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.117+0000 7f8fb9ffb640 1 -- 172.21.15.107:0/2943555264 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc027060 con 0x558f19c7c740 2024-05-09T08:44:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.117+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2943555264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fc4004380 con 0x558f19c7c740 2024-05-09T08:44:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.117+0000 7f8fb9ffb640 1 -- 172.21.15.107:0/2943555264 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7f8fcc003e20 con 0x558f19c7c740 2024-05-09T08:44:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.117+0000 7f8fb9ffb640 1 -- 172.21.15.107:0/2943555264 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8fcc02e050 con 0x558f19c7c740 2024-05-09T08:44:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.117+0000 7f8fdb7e0740 10 client.18753 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.117+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2943555264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558f198aa170 con 0x558f19c7c740 2024-05-09T08:44:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.118+0000 7f8fb9ffb640 1 -- 172.21.15.107:0/2943555264 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fc4004380 con 0x558f19c7c740 2024-05-09T08:44:24.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.118+0000 7f8fb9ffb640 1 -- 172.21.15.107:0/2943555264 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8fcc010a30 con 0x558f19c7c740 2024-05-09T08:44:24.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.118+0000 7f8fdb7e0740 10 client.18753 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:24.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.118+0000 7f8fdb7e0740 10 client.18753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:24.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.118+0000 7f8fdb7e0740 20 client.18753 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:24.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.118+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/2943555264 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558f19c95040 0x558f19c974e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.118+0000 7f8fdb7e0740 4 client.18753 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:44:24.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.118+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2943555264 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f19be40c0 con 0x558f19c95040 2024-05-09T08:44:24.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.118+0000 7f8fd259c640 1 --2- 172.21.15.107:0/2943555264 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558f19c95040 0x558f19c974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.119+0000 7f8fb9ffb640 10 client.18753 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:24.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.119+0000 7f8fd259c640 1 --2- 172.21.15.107:0/2943555264 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558f19c95040 0x558f19c974e0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d59dd8d5859959d2 server_cookie=8479118abefb44f6 in_seq=0 out_seq=0 2024-05-09T08:44:24.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.119+0000 7f8fb9ffb640 1 -- 172.21.15.107:0/2943555264 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558f19be40c0 con 0x558f19c95040 2024-05-09T08:44:24.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.120+0000 7f8fb9ffb640 10 client.18753 handle_command_reply: tid=0 2024-05-09T08:44:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.120+0000 7f8fdb7e0740 1 client.18753 shutdown 2024-05-09T08:44:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.120+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2943555264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c7c740 msgr2=0x558f19c7a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.120+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/2943555264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c7c740 0x558f19c7a070 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00ba90 tx=0x7f8fcc004a40 comp rx=0 tx=0).stop 2024-05-09T08:44:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.120+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2943555264 shutdown_connections 2024-05-09T08:44:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.120+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/2943555264 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558f19c95040 0x558f19c974e0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.120+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/2943555264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c7c740 0x558f19c7a070 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.120+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/2943555264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c730b0 0x558f19c79b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.120+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/2943555264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c719c0 0x558f19c795f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.120+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2943555264 >> 172.21.15.107:0/2943555264 conn(0x558f19c3a0a0 msgr2=0x558f19c73d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.120+0000 7f8fb9ffb640 0 client.18753 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.120+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2943555264 shutdown_connections 2024-05-09T08:44:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.120+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2943555264 wait complete. 2024-05-09T08:44:24.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.120+0000 7f8fdb7e0740 20 client.18753 trim_cache size 0 max 16384 2024-05-09T08:44:24.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.163+0000 7f8fdb7e0740 1 Processor -- start 2024-05-09T08:44:24.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.168+0000 7f8fdb7e0740 1 -- start start 2024-05-09T08:44:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.168+0000 7f8fdb7e0740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c719c0 0x558f19c90d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.168+0000 7f8fdb7e0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c730b0 0x558f19c6cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.168+0000 7f8fdb7e0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c7c740 0x558f19c9ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.168+0000 7f8fdb7e0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb40128b0 con 0x558f19c7c740 2024-05-09T08:44:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.168+0000 7f8fdb7e0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb4010180 con 0x558f19c730b0 2024-05-09T08:44:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.168+0000 7f8fdb7e0740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb400dff0 con 0x558f19c719c0 2024-05-09T08:44:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.169+0000 7f8fd1d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c730b0 0x558f19c6cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.169+0000 7f8fd1d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c730b0 0x558f19c6cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:57880/0 (socket says 172.21.15.107:57880) 2024-05-09T08:44:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.169+0000 7f8fd1d9b640 1 -- 172.21.15.107:0/2356976840 learned_addr learned my addr 172.21.15.107:0/2356976840 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.170+0000 7f8fd2d9d640 1 --2- 172.21.15.107:0/2356976840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c7c740 0x558f19c9ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.170+0000 7f8fd1d9b640 1 -- 172.21.15.107:0/2356976840 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c719c0 msgr2=0x558f19c90d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.170+0000 7f8fd1d9b640 1 --2- 172.21.15.107:0/2356976840 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c719c0 0x558f19c90d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.170+0000 7f8fd1d9b640 1 -- 172.21.15.107:0/2356976840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c7c740 msgr2=0x558f19c9ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.170+0000 7f8fd1d9b640 1 --2- 172.21.15.107:0/2356976840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c7c740 0x558f19c9ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.170+0000 7f8fd1d9b640 1 -- 172.21.15.107:0/2356976840 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f19c785a0 con 0x558f19c730b0 2024-05-09T08:44:24.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.171+0000 7f8fd1d9b640 1 --2- 172.21.15.107:0/2356976840 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c730b0 0x558f19c6cf00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8097b90 tx=0x7f8fc8086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebcba541de724f15 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.171+0000 7f8fbbfff640 1 -- 172.21.15.107:0/2356976840 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80cb070 con 0x558f19c730b0 2024-05-09T08:44:24.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.172+0000 7f8fbbfff640 1 -- 172.21.15.107:0/2356976840 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fc809fae0 con 0x558f19c730b0 2024-05-09T08:44:24.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.172+0000 7f8fbbfff640 1 -- 172.21.15.107:0/2356976840 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80b3030 con 0x558f19c730b0 2024-05-09T08:44:24.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.176+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2356976840 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c730b0 msgr2=0x558f19c6cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.176+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/2356976840 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c730b0 0x558f19c6cf00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8097b90 tx=0x7f8fc8086040 comp rx=0 tx=0).stop 2024-05-09T08:44:24.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.176+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2356976840 shutdown_connections 2024-05-09T08:44:24.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.176+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/2356976840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c7c740 0x558f19c9ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.176+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/2356976840 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c730b0 0x558f19c6cf00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.176+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/2356976840 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c719c0 0x558f19c90d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.176+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2356976840 >> 172.21.15.107:0/2356976840 conn(0x558f19c3a0a0 msgr2=0x558f19c6c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.180+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2356976840 shutdown_connections 2024-05-09T08:44:24.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.180+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/2356976840 wait complete. 2024-05-09T08:44:24.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.180+0000 7f8fdb7e0740 1 -- start start 2024-05-09T08:44:24.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.181+0000 7f8fdb7e0740 1 Processor -- start 2024-05-09T08:44:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.181+0000 7f8fdb7e0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c719c0 0x558f19c8d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.181+0000 7f8fdb7e0740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c730b0 0x558f19c8d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.182+0000 7f8fdb7e0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c7c740 0x558f19c8a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.182+0000 7f8fd1d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c730b0 0x558f19c8d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.182+0000 7f8fd1d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c730b0 0x558f19c8d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:57546/0 (socket says 172.21.15.107:57546) 2024-05-09T08:44:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.182+0000 7f8fd1d9b640 1 -- 172.21.15.107:0/703548528 learned_addr learned my addr 172.21.15.107:0/703548528 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:24.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.182+0000 7f8fd2d9d640 1 --2- 172.21.15.107:0/703548528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c7c740 0x558f19c8a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.182+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/703548528 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb400e640 con 0x558f19c719c0 2024-05-09T08:44:24.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.182+0000 7f8fd259c640 1 --2- 172.21.15.107:0/703548528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c719c0 0x558f19c8d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.182+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/703548528 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb400fab0 con 0x558f19c7c740 2024-05-09T08:44:24.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.182+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/703548528 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb400ed10 con 0x558f19c730b0 2024-05-09T08:44:24.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.182+0000 7f8fdb7e0740 10 client.0 _ll_register_callbacks cb 0x7f8fcc003d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:24.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.183+0000 7f8fd2d9d640 1 -- 172.21.15.107:0/703548528 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c730b0 msgr2=0x558f19c8d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.183+0000 7f8fd2d9d640 1 --2- 172.21.15.107:0/703548528 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c730b0 0x558f19c8d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.183+0000 7f8fd2d9d640 1 -- 172.21.15.107:0/703548528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c719c0 msgr2=0x558f19c8d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.183+0000 7f8fd2d9d640 1 --2- 172.21.15.107:0/703548528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c719c0 0x558f19c8d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.183+0000 7f8fd2d9d640 1 -- 172.21.15.107:0/703548528 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f19c8aa40 con 0x558f19c7c740 2024-05-09T08:44:24.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.183+0000 7f8fd2d9d640 1 --2- 172.21.15.107:0/703548528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c7c740 0x558f19c8a240 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009d40 tx=0x7f8fcc0122e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbe7c4472a8d30 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.183+0000 7f8fb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:44:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.183+0000 7f8fb9ffb640 1 -- 172.21.15.107:0/703548528 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc00be90 con 0x558f19c7c740 2024-05-09T08:44:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.184+0000 7f8fb9ffb640 1 -- 172.21.15.107:0/703548528 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7f8fcc01b040 con 0x558f19c7c740 2024-05-09T08:44:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.185+0000 7f8fd1d9b640 1 --2- 172.21.15.107:0/703548528 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c730b0 0x558f19c8d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:44:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.185+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/703548528 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fc809fae0 con 0x558f19c7c740 2024-05-09T08:44:24.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.187+0000 7f8fb9ffb640 1 -- 172.21.15.107:0/703548528 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8fcc009e80 con 0x558f19c7c740 2024-05-09T08:44:24.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.187+0000 7f8fdb7e0740 10 client.27704 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:24.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.187+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/703548528 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558f19cfc100 con 0x558f19c7c740 2024-05-09T08:44:24.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.193+0000 7f8fb9ffb640 1 -- 172.21.15.107:0/703548528 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fc809fae0 con 0x558f19c7c740 2024-05-09T08:44:24.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.193+0000 7f8fb9ffb640 1 -- 172.21.15.107:0/703548528 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8fcc010460 con 0x558f19c7c740 2024-05-09T08:44:24.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.193+0000 7f8fdb7e0740 10 client.27704 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:24.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.193+0000 7f8fdb7e0740 10 client.27704 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:24.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.193+0000 7f8fdb7e0740 20 client.27704 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:24.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.193+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/703548528 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558f19cfc7c0 0x558f19cfebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.194+0000 7f8fdb7e0740 4 client.27704 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:44:24.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.194+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/703548528 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558f19bde410 con 0x558f19cfc7c0 2024-05-09T08:44:24.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.198+0000 7f8fd259c640 1 --2- 172.21.15.107:0/703548528 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558f19cfc7c0 0x558f19cfebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.202+0000 7f8fd259c640 1 --2- 172.21.15.107:0/703548528 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558f19cfc7c0 0x558f19cfebe0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77e073a58210e18c server_cookie=13062e832996d008 in_seq=0 out_seq=0 2024-05-09T08:44:24.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.202+0000 7f8fb9ffb640 10 client.27704 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:24.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.207+0000 7f8fb9ffb640 1 -- 172.21.15.107:0/703548528 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558f19bde410 con 0x558f19cfc7c0 2024-05-09T08:44:24.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.207+0000 7f8fb9ffb640 10 client.27704 handle_command_reply: tid=0 2024-05-09T08:44:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.208+0000 7f8fdb7e0740 1 client.27704 shutdown 2024-05-09T08:44:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.208+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/703548528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c7c740 msgr2=0x558f19c8a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.208+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/703548528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c7c740 0x558f19c8a240 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009d40 tx=0x7f8fcc0122e0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.210+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/703548528 shutdown_connections 2024-05-09T08:44:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.211+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/703548528 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558f19cfc7c0 0x558f19cfebe0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.211+0000 7f8fb9ffb640 0 client.27704 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.211+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/703548528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f19c7c740 0x558f19c8a240 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.211+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/703548528 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558f19c730b0 0x558f19c8d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.211+0000 7f8fdb7e0740 1 --2- 172.21.15.107:0/703548528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f19c719c0 0x558f19c8d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.211+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/703548528 >> 172.21.15.107:0/703548528 conn(0x558f19c3a0a0 msgr2=0x558f19c6d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.216+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/703548528 shutdown_connections 2024-05-09T08:44:24.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.217+0000 7f8fdb7e0740 1 -- 172.21.15.107:0/703548528 wait complete. 2024-05-09T08:44:24.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.218+0000 7f8fdb7e0740 20 client.27704 trim_cache size 0 max 16384 2024-05-09T08:44:24.217 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:44:24.217 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:44:24.217 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "196cdfca-7dfe-4c61-8a89-763c753e759f", 2024-05-09T08:44:24.217 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:44:24.217 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:44:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.219+0000 7f8fd8adf640 1 -- 172.21.15.107:0/2900808047 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fbc0a2e00 msgr2=0x7f8fbc0a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.219+0000 7f8fd8adf640 1 --2- 172.21.15.107:0/2900808047 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fbc0a2e00 0x7f8fbc0a52c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc0094d0 tx=0x7f8fcc004ab0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.219+0000 7f8fd8adf640 1 -- 172.21.15.107:0/2900808047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fd41017e0 msgr2=0x7f8fd4198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.219+0000 7f8fd8adf640 1 --2- 172.21.15.107:0/2900808047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fd41017e0 0x7f8fd4198590 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800e9f0 tx=0x7f8fc800eeb0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.219+0000 7f8fd8adf640 1 -- 172.21.15.107:0/2900808047 shutdown_connections 2024-05-09T08:44:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.219+0000 7f8fd8adf640 1 --2- 172.21.15.107:0/2900808047 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fbc0a2e00 0x7f8fbc0a52c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.219+0000 7f8fd8adf640 1 --2- 172.21.15.107:0/2900808047 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fd4106730 0x7f8fd419c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.219+0000 7f8fd8adf640 1 --2- 172.21.15.107:0/2900808047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fd41017e0 0x7f8fd4198590 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.219+0000 7f8fd8adf640 1 --2- 172.21.15.107:0/2900808047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fd4100e40 0x7f8fd4198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.219+0000 7f8fd8adf640 1 -- 172.21.15.107:0/2900808047 >> 172.21.15.107:0/2900808047 conn(0x7f8fd4076cf0 msgr2=0x7f8fd40fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.220+0000 7f8fd8adf640 1 -- 172.21.15.107:0/2900808047 shutdown_connections 2024-05-09T08:44:24.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.220+0000 7f8fd8adf640 1 -- 172.21.15.107:0/2900808047 wait complete. 2024-05-09T08:44:24.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:44:24.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:44:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.462+0000 7fbd3f59d640 1 -- 172.21.15.107:0/2405672476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd380fffb0 msgr2=0x7fbd38100390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.462+0000 7fbd3f59d640 1 --2- 172.21.15.107:0/2405672476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd380fffb0 0x7fbd38100390 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fbd34009900 tx=0x7fbd3402a410 comp rx=0 tx=0).stop 2024-05-09T08:44:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.462+0000 7fbd3f59d640 1 -- 172.21.15.107:0/2405672476 shutdown_connections 2024-05-09T08:44:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.462+0000 7fbd3f59d640 1 --2- 172.21.15.107:0/2405672476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd38106bc0 0x7fbd3810e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.462+0000 7fbd3f59d640 1 --2- 172.21.15.107:0/2405672476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd38100960 0x7fbd38106490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.462+0000 7fbd3f59d640 1 --2- 172.21.15.107:0/2405672476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd380fffb0 0x7fbd38100390 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.462+0000 7fbd3f59d640 1 -- 172.21.15.107:0/2405672476 >> 172.21.15.107:0/2405672476 conn(0x7fbd38076cd0 msgr2=0x7fbd380fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.462+0000 7fbd3f59d640 1 -- 172.21.15.107:0/2405672476 shutdown_connections 2024-05-09T08:44:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.462+0000 7fbd3f59d640 1 -- 172.21.15.107:0/2405672476 wait complete. 2024-05-09T08:44:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.463+0000 7fbd3f59d640 1 Processor -- start 2024-05-09T08:44:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.463+0000 7fbd3f59d640 1 -- start start 2024-05-09T08:44:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.463+0000 7fbd3f59d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd380fffb0 0x7fbd38100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.463+0000 7fbd3f59d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38106bc0 0x7fbd38101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.463+0000 7fbd3f59d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd381019a0 0x7fbd381a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.463+0000 7fbd3f59d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbd38111ad0 con 0x7fbd38106bc0 2024-05-09T08:44:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.463+0000 7fbd3f59d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbd38111950 con 0x7fbd380fffb0 2024-05-09T08:44:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.463+0000 7fbd3f59d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbd38111c50 con 0x7fbd381019a0 2024-05-09T08:44:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.463+0000 7fbd3d339640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd380fffb0 0x7fbd38100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.463+0000 7fbd3d339640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd380fffb0 0x7fbd38100f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:57900/0 (socket says 172.21.15.107:57900) 2024-05-09T08:44:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.463+0000 7fbd3d339640 1 -- 172.21.15.107:0/2173349648 learned_addr learned my addr 172.21.15.107:0/2173349648 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.463+0000 7fbd3cb38640 1 --2- 172.21.15.107:0/2173349648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38106bc0 0x7fbd38101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.463+0000 7fbd3db3a640 1 --2- 172.21.15.107:0/2173349648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd381019a0 0x7fbd381a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.464+0000 7fbd3cb38640 1 -- 172.21.15.107:0/2173349648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd381019a0 msgr2=0x7fbd381a6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.464+0000 7fbd3cb38640 1 --2- 172.21.15.107:0/2173349648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd381019a0 0x7fbd381a6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.464+0000 7fbd3cb38640 1 -- 172.21.15.107:0/2173349648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd380fffb0 msgr2=0x7fbd38100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.464+0000 7fbd3cb38640 1 --2- 172.21.15.107:0/2173349648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd380fffb0 0x7fbd38100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.464+0000 7fbd3cb38640 1 -- 172.21.15.107:0/2173349648 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd381a6960 con 0x7fbd38106bc0 2024-05-09T08:44:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.464+0000 7fbd3d339640 1 --2- 172.21.15.107:0/2173349648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd380fffb0 0x7fbd38100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:44:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.464+0000 7fbd3cb38640 1 --2- 172.21.15.107:0/2173349648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38106bc0 0x7fbd38101460 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fbd38073fc0 tx=0x7fbd2800daf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.464+0000 7fbd2e7fc640 1 -- 172.21.15.107:0/2173349648 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd28017070 con 0x7fbd38106bc0 2024-05-09T08:44:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.464+0000 7fbd3f59d640 1 -- 172.21.15.107:0/2173349648 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd381a6c50 con 0x7fbd38106bc0 2024-05-09T08:44:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.464+0000 7fbd3f59d640 1 -- 172.21.15.107:0/2173349648 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd381a71a0 con 0x7fbd38106bc0 2024-05-09T08:44:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.464+0000 7fbd2e7fc640 1 -- 172.21.15.107:0/2173349648 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd28004c50 con 0x7fbd38106bc0 2024-05-09T08:44:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.464+0000 7fbd2e7fc640 1 -- 172.21.15.107:0/2173349648 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd280056d0 con 0x7fbd38106bc0 2024-05-09T08:44:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.468+0000 7fbd42382740 1 Processor -- start 2024-05-09T08:44:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.468+0000 7fbd42382740 1 -- start start 2024-05-09T08:44:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.468+0000 7fbd42382740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67ef4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.468+0000 7fbd42382740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67ef4ba0 0x559a67efed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.468+0000 7fbd42382740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67eff270 0x559a67f01660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.468+0000 7fbd42382740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559a67e59d00 con 0x559a67ef4ba0 2024-05-09T08:44:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.468+0000 7fbd42382740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559a67e014c0 con 0x559a67ef4280 2024-05-09T08:44:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.468+0000 7fbd42382740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x559a67e55cc0 con 0x559a67eff270 2024-05-09T08:44:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.468+0000 7fbd3d339640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67ef4ba0 0x559a67efed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.468+0000 7fbd3d339640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67ef4ba0 0x559a67efed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58512/0 (socket says 172.21.15.107:58512) 2024-05-09T08:44:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.468+0000 7fbd3d339640 1 -- 172.21.15.107:0/1626267382 learned_addr learned my addr 172.21.15.107:0/1626267382 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.468+0000 7fbd3db3a640 1 --2- 172.21.15.107:0/1626267382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67ef4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.468+0000 7fbd2e7fc640 1 -- 172.21.15.107:0/2173349648 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fbd28004220 con 0x7fbd38106bc0 2024-05-09T08:44:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.469+0000 7fbd3cb38640 1 --2- 172.21.15.107:0/1626267382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67eff270 0x559a67f01660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.469+0000 7fbd3db3a640 1 -- 172.21.15.107:0/1626267382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67eff270 msgr2=0x559a67f01660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.469+0000 7fbd3db3a640 1 --2- 172.21.15.107:0/1626267382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67eff270 0x559a67f01660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.469+0000 7fbd3db3a640 1 -- 172.21.15.107:0/1626267382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67ef4ba0 msgr2=0x559a67efed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.469+0000 7fbd3db3a640 1 --2- 172.21.15.107:0/1626267382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67ef4ba0 0x559a67efed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.469+0000 7fbd3db3a640 1 -- 172.21.15.107:0/1626267382 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a67c377d0 con 0x559a67ef4280 2024-05-09T08:44:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.469+0000 7fbd3db3a640 1 --2- 172.21.15.107:0/1626267382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67ef4660 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbd30009f10 tx=0x7fbd30011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75e7ec5c42ff696d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.469+0000 7fbd2e7fc640 1 --2- 172.21.15.107:0/2173349648 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbd240a2d30 0x7fbd240a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.469+0000 7fbd237fe640 1 -- 172.21.15.107:0/1626267382 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd30004d20 con 0x559a67ef4280 2024-05-09T08:44:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.469+0000 7fbd2e7fc640 1 -- 172.21.15.107:0/2173349648 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7fbd280cf2d0 con 0x7fbd38106bc0 2024-05-09T08:44:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.470+0000 7fbd237fe640 1 -- 172.21.15.107:0/1626267382 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd30004050 con 0x559a67ef4280 2024-05-09T08:44:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.470+0000 7fbd237fe640 1 -- 172.21.15.107:0/1626267382 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd30005260 con 0x559a67ef4280 2024-05-09T08:44:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.470+0000 7fbd3d339640 1 --2- 172.21.15.107:0/2173349648 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbd240a2d30 0x7fbd240a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.470+0000 7fbd42382740 1 -- 172.21.15.107:0/1626267382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 msgr2=0x559a67ef4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.470+0000 7fbd42382740 1 --2- 172.21.15.107:0/1626267382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67ef4660 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbd30009f10 tx=0x7fbd30011810 comp rx=0 tx=0).stop 2024-05-09T08:44:24.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.470+0000 7fbd42382740 1 -- 172.21.15.107:0/1626267382 shutdown_connections 2024-05-09T08:44:24.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.470+0000 7fbd42382740 1 --2- 172.21.15.107:0/1626267382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67eff270 0x559a67f01660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.470+0000 7fbd42382740 1 --2- 172.21.15.107:0/1626267382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67ef4ba0 0x559a67efed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.470+0000 7fbd42382740 1 --2- 172.21.15.107:0/1626267382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67ef4660 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.470+0000 7fbd42382740 1 -- 172.21.15.107:0/1626267382 >> 172.21.15.107:0/1626267382 conn(0x559a67ef05a0 msgr2=0x559a67ef0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.471+0000 7fbd3d339640 1 --2- 172.21.15.107:0/2173349648 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbd240a2d30 0x7fbd240a51f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbd34032fd0 tx=0x7fbd340058d0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.471+0000 7fbd42382740 1 -- 172.21.15.107:0/1626267382 shutdown_connections 2024-05-09T08:44:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.471+0000 7fbd42382740 1 -- 172.21.15.107:0/1626267382 wait complete. 2024-05-09T08:44:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.471+0000 7fbd42382740 1 -- start start 2024-05-09T08:44:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.471+0000 7fbd42382740 1 Processor -- start 2024-05-09T08:44:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd42382740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67efc100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd42382740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 0x559a67efc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd42382740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 0x559a67efcb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd42382740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559a67f02060 con 0x559a67eff270 2024-05-09T08:44:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd42382740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559a67e14b90 con 0x559a67ef4280 2024-05-09T08:44:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd42382740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x559a67f021e0 con 0x559a67ef4ba0 2024-05-09T08:44:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd42382740 10 client.0 _ll_register_callbacks cb 0x559a67e50da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd3db3a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 0x559a67efcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd3db3a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 0x559a67efcb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58526/0 (socket says 172.21.15.107:58526) 2024-05-09T08:44:24.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd3db3a640 1 -- 172.21.15.107:0/2958976791 learned_addr learned my addr 172.21.15.107:0/2958976791 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:24.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd3d339640 1 --2- 172.21.15.107:0/2958976791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67efc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd3cb38640 1 --2- 172.21.15.107:0/2958976791 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 0x559a67efc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd3db3a640 1 -- 172.21.15.107:0/2958976791 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 msgr2=0x559a67efc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd3db3a640 1 --2- 172.21.15.107:0/2958976791 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 0x559a67efc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd3db3a640 1 -- 172.21.15.107:0/2958976791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 msgr2=0x559a67efc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd3db3a640 1 --2- 172.21.15.107:0/2958976791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67efc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd3db3a640 1 -- 172.21.15.107:0/2958976791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a67b6d3f0 con 0x559a67eff270 2024-05-09T08:44:24.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd3d339640 1 --2- 172.21.15.107:0/2958976791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67efc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:44:24.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd3db3a640 1 --2- 172.21.15.107:0/2958976791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 0x559a67efcb80 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fbd3001b040 tx=0x7fbd30004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4849c4e6a53ce7d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.472+0000 7fbd20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:44:24.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.473+0000 7fbd20ff9640 1 -- 172.21.15.107:0/2958976791 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd30029020 con 0x559a67eff270 2024-05-09T08:44:24.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.473+0000 7fbd20ff9640 1 -- 172.21.15.107:0/2958976791 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7fbd30005040 con 0x559a67eff270 2024-05-09T08:44:24.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.473+0000 7fbd42382740 1 -- 172.21.15.107:0/2958976791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd30004050 con 0x559a67eff270 2024-05-09T08:44:24.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.473+0000 7fbd20ff9640 1 -- 172.21.15.107:0/2958976791 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbd30034050 con 0x559a67eff270 2024-05-09T08:44:24.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.473+0000 7fbd42382740 10 client.18771 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:24.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.473+0000 7fbd42382740 1 -- 172.21.15.107:0/2958976791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559a67b2d0f0 con 0x559a67eff270 2024-05-09T08:44:24.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.473+0000 7fbd20ff9640 1 -- 172.21.15.107:0/2958976791 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd30004050 con 0x559a67eff270 2024-05-09T08:44:24.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.473+0000 7fbd20ff9640 1 -- 172.21.15.107:0/2958976791 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fbd30025590 con 0x559a67eff270 2024-05-09T08:44:24.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.473+0000 7fbd42382740 10 client.18771 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:24.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.473+0000 7fbd42382740 10 client.18771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:24.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.473+0000 7fbd42382740 20 client.18771 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:24.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.473+0000 7fbd42382740 1 --2- 172.21.15.107:0/2958976791 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x559a67f0f550 0x559a67f11940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.473+0000 7fbd42382740 4 client.18771 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:44:24.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.473+0000 7fbd42382740 1 -- 172.21.15.107:0/2958976791 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a67e67430 con 0x559a67f0f550 2024-05-09T08:44:24.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.474+0000 7fbd3d339640 1 --2- 172.21.15.107:0/2958976791 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x559a67f0f550 0x559a67f11940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.474+0000 7fbd20ff9640 10 client.18771 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:24.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.474+0000 7fbd3d339640 1 --2- 172.21.15.107:0/2958976791 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x559a67f0f550 0x559a67f11940 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8037ccee9e019781 server_cookie=e5de36da285126f0 in_seq=0 out_seq=0 2024-05-09T08:44:24.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.475+0000 7fbd20ff9640 1 -- 172.21.15.107:0/2958976791 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559a67e67430 con 0x559a67f0f550 2024-05-09T08:44:24.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.475+0000 7fbd20ff9640 10 client.18771 handle_command_reply: tid=0 2024-05-09T08:44:24.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.475+0000 7fbd42382740 1 client.18771 shutdown 2024-05-09T08:44:24.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.475+0000 7fbd42382740 1 -- 172.21.15.107:0/2958976791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 msgr2=0x559a67efcb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.475+0000 7fbd42382740 1 --2- 172.21.15.107:0/2958976791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 0x559a67efcb80 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fbd3001b040 tx=0x7fbd30004c70 comp rx=0 tx=0).stop 2024-05-09T08:44:24.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.475+0000 7fbd42382740 1 -- 172.21.15.107:0/2958976791 shutdown_connections 2024-05-09T08:44:24.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.475+0000 7fbd42382740 1 --2- 172.21.15.107:0/2958976791 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x559a67f0f550 0x559a67f11940 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.475+0000 7fbd42382740 1 --2- 172.21.15.107:0/2958976791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 0x559a67efcb80 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.475+0000 7fbd42382740 1 --2- 172.21.15.107:0/2958976791 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 0x559a67efc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.475+0000 7fbd42382740 1 --2- 172.21.15.107:0/2958976791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67efc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.475+0000 7fbd42382740 1 -- 172.21.15.107:0/2958976791 >> 172.21.15.107:0/2958976791 conn(0x559a67ef05a0 msgr2=0x559a67eef7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.475+0000 7fbd20ff9640 0 client.18771 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:24.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.475+0000 7fbd42382740 1 -- 172.21.15.107:0/2958976791 shutdown_connections 2024-05-09T08:44:24.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.475+0000 7fbd42382740 1 -- 172.21.15.107:0/2958976791 wait complete. 2024-05-09T08:44:24.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.475+0000 7fbd42382740 20 client.18771 trim_cache size 0 max 16384 2024-05-09T08:44:24.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.506+0000 7fbd42382740 1 Processor -- start 2024-05-09T08:44:24.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.507+0000 7fbd42382740 1 -- start start 2024-05-09T08:44:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.507+0000 7fbd42382740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67f0af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.507+0000 7fbd42382740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 0x559a67f171f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.507+0000 7fbd42382740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 0x559a67f17730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.507+0000 7fbd42382740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c0128c0 con 0x559a67eff270 2024-05-09T08:44:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.507+0000 7fbd42382740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c010190 con 0x559a67ef4280 2024-05-09T08:44:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.507+0000 7fbd42382740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c00e000 con 0x559a67ef4ba0 2024-05-09T08:44:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.507+0000 7fbd3cb38640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 0x559a67f171f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.507+0000 7fbd3cb38640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 0x559a67f171f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:57590/0 (socket says 172.21.15.107:57590) 2024-05-09T08:44:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.507+0000 7fbd3cb38640 1 -- 172.21.15.107:0/3240035726 learned_addr learned my addr 172.21.15.107:0/3240035726 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.508+0000 7fbd3cb38640 1 -- 172.21.15.107:0/3240035726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 msgr2=0x559a67f0af80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:44:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.508+0000 7fbd3cb38640 1 --2- 172.21.15.107:0/3240035726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67f0af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.508+0000 7fbd3cb38640 1 -- 172.21.15.107:0/3240035726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 msgr2=0x559a67f17730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.508+0000 7fbd3cb38640 1 --2- 172.21.15.107:0/3240035726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 0x559a67f17730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.508+0000 7fbd3cb38640 1 -- 172.21.15.107:0/3240035726 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a67efa650 con 0x559a67ef4ba0 2024-05-09T08:44:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.508+0000 7fbd3cb38640 1 --2- 172.21.15.107:0/3240035726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 0x559a67f171f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbd2809b360 tx=0x7fbd28005120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cbe06016c02dcd8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.509+0000 7fbd22ffd640 1 -- 172.21.15.107:0/3240035726 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd28004e00 con 0x559a67ef4ba0 2024-05-09T08:44:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.509+0000 7fbd22ffd640 1 -- 172.21.15.107:0/3240035726 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd2809bad0 con 0x559a67ef4ba0 2024-05-09T08:44:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.509+0000 7fbd22ffd640 1 -- 172.21.15.107:0/3240035726 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd28097a40 con 0x559a67ef4ba0 2024-05-09T08:44:24.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.510+0000 7fbd42382740 1 -- 172.21.15.107:0/3240035726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 msgr2=0x559a67f171f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.510+0000 7fbd42382740 1 --2- 172.21.15.107:0/3240035726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 0x559a67f171f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbd2809b360 tx=0x7fbd28005120 comp rx=0 tx=0).stop 2024-05-09T08:44:24.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.510+0000 7fbd42382740 1 -- 172.21.15.107:0/3240035726 shutdown_connections 2024-05-09T08:44:24.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.510+0000 7fbd42382740 1 --2- 172.21.15.107:0/3240035726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 0x559a67f17730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.510+0000 7fbd42382740 1 --2- 172.21.15.107:0/3240035726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 0x559a67f171f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.510+0000 7fbd42382740 1 --2- 172.21.15.107:0/3240035726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67f0af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.510+0000 7fbd42382740 1 -- 172.21.15.107:0/3240035726 >> 172.21.15.107:0/3240035726 conn(0x559a67ef05a0 msgr2=0x559a67efb130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.511+0000 7fbd42382740 1 -- 172.21.15.107:0/3240035726 shutdown_connections 2024-05-09T08:44:24.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.511+0000 7fbd42382740 1 -- 172.21.15.107:0/3240035726 wait complete. 2024-05-09T08:44:24.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.511+0000 7fbd42382740 1 -- start start 2024-05-09T08:44:24.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.511+0000 7fbd42382740 1 Processor -- start 2024-05-09T08:44:24.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.512+0000 7fbd42382740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67f070f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.512+0000 7fbd42382740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 0x559a67f07630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.512+0000 7fbd42382740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 0x559a67f02630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.513+0000 7fbd42382740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c00e650 con 0x559a67eff270 2024-05-09T08:44:24.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.513+0000 7fbd42382740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c00fac0 con 0x559a67ef4280 2024-05-09T08:44:24.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.513+0000 7fbd42382740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c00ed20 con 0x559a67ef4ba0 2024-05-09T08:44:24.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.513+0000 7fbd42382740 10 client.0 _ll_register_callbacks cb 0x7fbd34009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:24.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.515+0000 7fbd3d339640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67f070f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.515+0000 7fbd3db3a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 0x559a67f02630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.515+0000 7fbd3db3a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 0x559a67f02630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58546/0 (socket says 172.21.15.107:58546) 2024-05-09T08:44:24.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.515+0000 7fbd3db3a640 1 -- 172.21.15.107:0/1385614666 learned_addr learned my addr 172.21.15.107:0/1385614666 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:24.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.516+0000 7fbd3cb38640 1 --2- 172.21.15.107:0/1385614666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 0x559a67f07630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.516+0000 7fbd3db3a640 1 -- 172.21.15.107:0/1385614666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 msgr2=0x559a67f07630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.516+0000 7fbd3db3a640 1 --2- 172.21.15.107:0/1385614666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 0x559a67f07630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.516+0000 7fbd3db3a640 1 -- 172.21.15.107:0/1385614666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 msgr2=0x559a67f070f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:24.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.516+0000 7fbd3db3a640 1 --2- 172.21.15.107:0/1385614666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67f070f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.516+0000 7fbd3db3a640 1 -- 172.21.15.107:0/1385614666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a67f02d20 con 0x559a67eff270 2024-05-09T08:44:24.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.516+0000 7fbd3db3a640 1 --2- 172.21.15.107:0/1385614666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 0x559a67f02630 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fbd30009c50 tx=0x7fbd30005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa9c5a6d53a63ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:24.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.516+0000 7fbd20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:44:24.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.518+0000 7fbd3d339640 1 --2- 172.21.15.107:0/1385614666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67f070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:44:24.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.521+0000 7fbd20ff9640 1 -- 172.21.15.107:0/1385614666 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd30029040 con 0x559a67eff270 2024-05-09T08:44:24.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.521+0000 7fbd20ff9640 1 -- 172.21.15.107:0/1385614666 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7fbd30025de0 con 0x559a67eff270 2024-05-09T08:44:24.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.521+0000 7fbd42382740 1 -- 172.21.15.107:0/1385614666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd2809bad0 con 0x559a67eff270 2024-05-09T08:44:24.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.523+0000 7fbd20ff9640 1 -- 172.21.15.107:0/1385614666 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbd30004e50 con 0x559a67eff270 2024-05-09T08:44:24.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.523+0000 7fbd42382740 10 client.18774 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:24.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.523+0000 7fbd42382740 1 -- 172.21.15.107:0/1385614666 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559a67f75460 con 0x559a67eff270 2024-05-09T08:44:24.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.524+0000 7fbd20ff9640 1 -- 172.21.15.107:0/1385614666 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd2809bad0 con 0x559a67eff270 2024-05-09T08:44:24.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.524+0000 7fbd20ff9640 1 -- 172.21.15.107:0/1385614666 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fbd300052f0 con 0x559a67eff270 2024-05-09T08:44:24.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.524+0000 7fbd42382740 10 client.18774 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:24.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.524+0000 7fbd42382740 10 client.18774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:24.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.524+0000 7fbd42382740 20 client.18774 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:24.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.524+0000 7fbd42382740 1 --2- 172.21.15.107:0/1385614666 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x559a67f75b30 0x559a67f89250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:24.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.524+0000 7fbd42382740 4 client.18774 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:44:24.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.524+0000 7fbd42382740 1 -- 172.21.15.107:0/1385614666 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559a67e61780 con 0x559a67f75b30 2024-05-09T08:44:24.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.525+0000 7fbd3d339640 1 --2- 172.21.15.107:0/1385614666 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x559a67f75b30 0x559a67f89250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:24.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.525+0000 7fbd20ff9640 10 client.18774 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:24.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.526+0000 7fbd3d339640 1 --2- 172.21.15.107:0/1385614666 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x559a67f75b30 0x559a67f89250 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8edf27befdda984e server_cookie=fc87ea928c569b04 in_seq=0 out_seq=0 2024-05-09T08:44:24.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.528+0000 7fbd20ff9640 1 -- 172.21.15.107:0/1385614666 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x559a67e61780 con 0x559a67f75b30 2024-05-09T08:44:24.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.528+0000 7fbd20ff9640 10 client.18774 handle_command_reply: tid=0 2024-05-09T08:44:24.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.529+0000 7fbd42382740 1 client.18774 shutdown 2024-05-09T08:44:24.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.529+0000 7fbd42382740 1 -- 172.21.15.107:0/1385614666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 msgr2=0x559a67f02630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.529+0000 7fbd42382740 1 --2- 172.21.15.107:0/1385614666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 0x559a67f02630 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fbd30009c50 tx=0x7fbd30005f90 comp rx=0 tx=0).stop 2024-05-09T08:44:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.532+0000 7fbd42382740 1 -- 172.21.15.107:0/1385614666 shutdown_connections 2024-05-09T08:44:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.532+0000 7fbd42382740 1 --2- 172.21.15.107:0/1385614666 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x559a67f75b30 0x559a67f89250 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.532+0000 7fbd42382740 1 --2- 172.21.15.107:0/1385614666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559a67eff270 0x559a67f02630 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.532+0000 7fbd42382740 1 --2- 172.21.15.107:0/1385614666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559a67ef4ba0 0x559a67f07630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.532+0000 7fbd42382740 1 --2- 172.21.15.107:0/1385614666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a67ef4280 0x559a67f070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.532+0000 7fbd42382740 1 -- 172.21.15.107:0/1385614666 >> 172.21.15.107:0/1385614666 conn(0x559a67ef05a0 msgr2=0x559a67efc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.532+0000 7fbd20ff9640 0 client.18774 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.532+0000 7fbd42382740 1 -- 172.21.15.107:0/1385614666 shutdown_connections 2024-05-09T08:44:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.532+0000 7fbd42382740 1 -- 172.21.15.107:0/1385614666 wait complete. 2024-05-09T08:44:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.533+0000 7fbd42382740 20 client.18774 trim_cache size 0 max 16384 2024-05-09T08:44:24.533 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:44:24.533 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "scrub active (222 inodes in the stack)", 2024-05-09T08:44:24.533 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:44:24.533 INFO:teuthology.orchestra.run.smithi107.stdout: "196cdfca-7dfe-4c61-8a89-763c753e759f": { 2024-05-09T08:44:24.533 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:44:24.533 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "196cdfca-7dfe-4c61-8a89-763c753e759f", 2024-05-09T08:44:24.533 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:44:24.534 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:44:24.534 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:44:24.534 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:44:24.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.534+0000 7fbd3f59d640 1 -- 172.21.15.107:0/2173349648 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbd240a2d30 msgr2=0x7fbd240a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.534+0000 7fbd3f59d640 1 --2- 172.21.15.107:0/2173349648 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbd240a2d30 0x7fbd240a51f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbd34032fd0 tx=0x7fbd340058d0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.534+0000 7fbd3f59d640 1 -- 172.21.15.107:0/2173349648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38106bc0 msgr2=0x7fbd38101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.534+0000 7fbd3f59d640 1 --2- 172.21.15.107:0/2173349648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38106bc0 0x7fbd38101460 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fbd38073fc0 tx=0x7fbd2800daf0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.535+0000 7fbd3f59d640 1 -- 172.21.15.107:0/2173349648 shutdown_connections 2024-05-09T08:44:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.535+0000 7fbd3f59d640 1 --2- 172.21.15.107:0/2173349648 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbd240a2d30 0x7fbd240a51f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.535+0000 7fbd3f59d640 1 --2- 172.21.15.107:0/2173349648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd381019a0 0x7fbd381a6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.535+0000 7fbd3f59d640 1 --2- 172.21.15.107:0/2173349648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38106bc0 0x7fbd38101460 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.535+0000 7fbd3f59d640 1 --2- 172.21.15.107:0/2173349648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd380fffb0 0x7fbd38100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.535+0000 7fbd3f59d640 1 -- 172.21.15.107:0/2173349648 >> 172.21.15.107:0/2173349648 conn(0x7fbd38076cd0 msgr2=0x7fbd38104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:24.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.539+0000 7fbd3f59d640 1 -- 172.21.15.107:0/2173349648 shutdown_connections 2024-05-09T08:44:24.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:24.539+0000 7fbd3f59d640 1 -- 172.21.15.107:0/2173349648 wait complete. 2024-05-09T08:44:24.558 INFO:tasks.cephfs.filesystem:scrub status for tag:196cdfca-7dfe-4c61-8a89-763c753e759f - {'path': '/', 'tag': '196cdfca-7dfe-4c61-8a89-763c753e759f', 'options': 'recursive,force'} 2024-05-09T08:44:25.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:24 smithi107 ceph-mon[30122]: pgmap v343: 129 pgs: 38 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 75 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 14 MiB/s wr, 3.48k op/s 2024-05-09T08:44:25.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:24 smithi157 ceph-mon[33614]: pgmap v343: 129 pgs: 38 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 75 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 14 MiB/s wr, 3.48k op/s 2024-05-09T08:44:25.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:24 smithi110 ceph-mon[33622]: pgmap v343: 129 pgs: 38 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 75 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 14 MiB/s wr, 3.48k op/s 2024-05-09T08:44:25.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:25 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:44:25.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:25 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:25.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:25 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:44:26.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:25 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:44:26.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:25 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:26.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:25 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:44:26.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:25 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:44:26.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:25 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:44:26.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:25 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:44:27.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:26 smithi107 ceph-mon[30122]: pgmap v344: 129 pgs: 38 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 75 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 14 MiB/s wr, 3.38k op/s 2024-05-09T08:44:27.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:26 smithi157 ceph-mon[33614]: pgmap v344: 129 pgs: 38 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 75 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 14 MiB/s wr, 3.38k op/s 2024-05-09T08:44:27.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:26 smithi110 ceph-mon[33622]: pgmap v344: 129 pgs: 38 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 75 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 14 MiB/s wr, 3.38k op/s 2024-05-09T08:44:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.263+0000 7f82a3064640 1 -- 172.21.15.107:0/1032600152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f829c100760 msgr2=0x7f829c100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.263+0000 7f82a3064640 1 --2- 172.21.15.107:0/1032600152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f829c100760 0x7f829c100be0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f828c009930 tx=0x7f828c02a420 comp rx=0 tx=0).stop 2024-05-09T08:44:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.263+0000 7f82a3064640 1 -- 172.21.15.107:0/1032600152 shutdown_connections 2024-05-09T08:44:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.263+0000 7f82a3064640 1 --2- 172.21.15.107:0/1032600152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f829c101120 0x7f829c10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.263+0000 7f82a3064640 1 --2- 172.21.15.107:0/1032600152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f829c100760 0x7f829c100be0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.263+0000 7f82a3064640 1 --2- 172.21.15.107:0/1032600152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f829c106720 0x7f829c106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.263+0000 7f82a3064640 1 -- 172.21.15.107:0/1032600152 >> 172.21.15.107:0/1032600152 conn(0x7f829c0778b0 msgr2=0x7f829c077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.264+0000 7f82a3064640 1 -- 172.21.15.107:0/1032600152 shutdown_connections 2024-05-09T08:44:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.264+0000 7f82a3064640 1 -- 172.21.15.107:0/1032600152 wait complete. 2024-05-09T08:44:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.264+0000 7f82a3064640 1 Processor -- start 2024-05-09T08:44:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.264+0000 7f82a3064640 1 -- start start 2024-05-09T08:44:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.264+0000 7f82a3064640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f829c100760 0x7f829c196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.264+0000 7f82a3064640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f829c101120 0x7f829c197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.264+0000 7f82a3064640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f829c106720 0x7f829c19b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.264+0000 7f82a3064640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f829c1109e0 con 0x7f829c101120 2024-05-09T08:44:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.264+0000 7f82a3064640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f829c110860 con 0x7f829c106720 2024-05-09T08:44:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.264+0000 7f82a3064640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f829c110b60 con 0x7f829c100760 2024-05-09T08:44:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.264+0000 7f829bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f829c101120 0x7f829c197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.264+0000 7f829bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f829c101120 0x7f829c197440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58570/0 (socket says 172.21.15.107:58570) 2024-05-09T08:44:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.265+0000 7f829bfff640 1 -- 172.21.15.107:0/4023707265 learned_addr learned my addr 172.21.15.107:0/4023707265 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.265+0000 7f82a1601640 1 --2- 172.21.15.107:0/4023707265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f829c106720 0x7f829c19b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.265+0000 7f82a0e00640 1 --2- 172.21.15.107:0/4023707265 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f829c100760 0x7f829c196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.265+0000 7f829bfff640 1 -- 172.21.15.107:0/4023707265 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f829c100760 msgr2=0x7f829c196f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.265+0000 7f829bfff640 1 --2- 172.21.15.107:0/4023707265 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f829c100760 0x7f829c196f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.265+0000 7f829bfff640 1 -- 172.21.15.107:0/4023707265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f829c106720 msgr2=0x7f829c19b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.265+0000 7f829bfff640 1 --2- 172.21.15.107:0/4023707265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f829c106720 0x7f829c19b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.265+0000 7f829bfff640 1 -- 172.21.15.107:0/4023707265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829c19bea0 con 0x7f829c101120 2024-05-09T08:44:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.265+0000 7f82a1601640 1 --2- 172.21.15.107:0/4023707265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f829c106720 0x7f829c19b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:44:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.265+0000 7f829bfff640 1 --2- 172.21.15.107:0/4023707265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f829c101120 0x7f829c197440 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f828c005940 tx=0x7f828c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.265+0000 7f8299ffb640 1 -- 172.21.15.107:0/4023707265 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c03f030 con 0x7f829c101120 2024-05-09T08:44:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.265+0000 7f8299ffb640 1 -- 172.21.15.107:0/4023707265 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f828c0318a0 con 0x7f829c101120 2024-05-09T08:44:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.265+0000 7f82a3064640 1 -- 172.21.15.107:0/4023707265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f829c19c130 con 0x7f829c101120 2024-05-09T08:44:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.265+0000 7f8299ffb640 1 -- 172.21.15.107:0/4023707265 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c035070 con 0x7f829c101120 2024-05-09T08:44:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.265+0000 7f82a3064640 1 -- 172.21.15.107:0/4023707265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f829c1a3a20 con 0x7f829c101120 2024-05-09T08:44:27.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.268+0000 7f8299ffb640 1 -- 172.21.15.107:0/4023707265 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f828c042030 con 0x7f829c101120 2024-05-09T08:44:27.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.268+0000 7f82a5d73740 1 Processor -- start 2024-05-09T08:44:27.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.268+0000 7f82a5d73740 1 -- start start 2024-05-09T08:44:27.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f82a5d73740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a62b0 0x55fae82a6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f82a5d73740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a6bd0 0x55fae82b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f82a5d73740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 0x55fae82b5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f82a5d73740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fae820b340 con 0x55fae82b3840 2024-05-09T08:44:27.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f82a5d73740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55fae81b3290 con 0x55fae82a62b0 2024-05-09T08:44:27.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f82a5d73740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55fae8207300 con 0x55fae82a6bd0 2024-05-09T08:44:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f82a1601640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a62b0 0x55fae82a6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f82a1601640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a62b0 0x55fae82a6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:57954/0 (socket says 172.21.15.107:57954) 2024-05-09T08:44:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f82a1601640 1 -- 172.21.15.107:0/261346605 learned_addr learned my addr 172.21.15.107:0/261346605 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f829bfff640 1 --2- 172.21.15.107:0/261346605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 0x55fae82b5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f8299ffb640 1 --2- 172.21.15.107:0/4023707265 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f82840a2da0 0x7f82840a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f82a0e00640 1 --2- 172.21.15.107:0/261346605 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a6bd0 0x55fae82b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f82a1601640 1 --2- 172.21.15.107:0/4023707265 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f82840a2da0 0x7f82840a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f82a1601640 1 -- 172.21.15.107:0/261346605 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a6bd0 msgr2=0x55fae82b3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f82a1601640 1 --2- 172.21.15.107:0/261346605 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a6bd0 0x55fae82b3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f82a1601640 1 -- 172.21.15.107:0/261346605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 msgr2=0x55fae82b5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f8299ffb640 1 -- 172.21.15.107:0/4023707265 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7f828c0f0020 con 0x7f829c101120 2024-05-09T08:44:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f82a1601640 1 --2- 172.21.15.107:0/261346605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 0x55fae82b5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.269+0000 7f82a1601640 1 -- 172.21.15.107:0/261346605 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fae81d78e0 con 0x55fae82a62b0 2024-05-09T08:44:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.270+0000 7f82a1601640 1 --2- 172.21.15.107:0/261346605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a62b0 0x55fae82a6690 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8294010e40 tx=0x7f82940147e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=920a863799bb93e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.270+0000 7f82a1601640 1 --2- 172.21.15.107:0/4023707265 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f82840a2da0 0x7f82840a5260 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f829401c000 tx=0x7f829401e000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.270+0000 7f8282ffd640 1 -- 172.21.15.107:0/261346605 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8294014080 con 0x55fae82a62b0 2024-05-09T08:44:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.270+0000 7f8282ffd640 1 -- 172.21.15.107:0/261346605 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8294014220 con 0x55fae82a62b0 2024-05-09T08:44:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.270+0000 7f8282ffd640 1 -- 172.21.15.107:0/261346605 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8294011620 con 0x55fae82a62b0 2024-05-09T08:44:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.270+0000 7f82a5d73740 1 -- 172.21.15.107:0/261346605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a62b0 msgr2=0x55fae82a6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.270+0000 7f82a5d73740 1 --2- 172.21.15.107:0/261346605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a62b0 0x55fae82a6690 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8294010e40 tx=0x7f82940147e0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.271+0000 7f82a5d73740 1 -- 172.21.15.107:0/261346605 shutdown_connections 2024-05-09T08:44:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.271+0000 7f82a5d73740 1 --2- 172.21.15.107:0/261346605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 0x55fae82b5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.271+0000 7f82a5d73740 1 --2- 172.21.15.107:0/261346605 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a6bd0 0x55fae82b3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.271+0000 7f82a5d73740 1 --2- 172.21.15.107:0/261346605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a62b0 0x55fae82a6690 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.271+0000 7f82a5d73740 1 -- 172.21.15.107:0/261346605 >> 172.21.15.107:0/261346605 conn(0x55fae82a2960 msgr2=0x55fae82a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.271+0000 7f82a5d73740 1 -- 172.21.15.107:0/261346605 shutdown_connections 2024-05-09T08:44:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.271+0000 7f82a5d73740 1 -- 172.21.15.107:0/261346605 wait complete. 2024-05-09T08:44:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.271+0000 7f82a5d73740 1 -- start start 2024-05-09T08:44:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.271+0000 7f82a5d73740 1 Processor -- start 2024-05-09T08:44:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.271+0000 7f82a5d73740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a62b0 0x55fae82b0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.271+0000 7f82a5d73740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a6bd0 0x55fae82b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.271+0000 7f82a5d73740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 0x55fae82b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.271+0000 7f82a1601640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 0x55fae82b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.271+0000 7f82a1601640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 0x55fae82b11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58586/0 (socket says 172.21.15.107:58586) 2024-05-09T08:44:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.271+0000 7f82a1601640 1 -- 172.21.15.107:0/2404331048 learned_addr learned my addr 172.21.15.107:0/2404331048 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f82a5d73740 1 -- 172.21.15.107:0/2404331048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fae82b6410 con 0x55fae82b3840 2024-05-09T08:44:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f82a5d73740 1 -- 172.21.15.107:0/2404331048 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55fae81c6790 con 0x55fae82a62b0 2024-05-09T08:44:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f82a5d73740 1 -- 172.21.15.107:0/2404331048 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55fae82b6590 con 0x55fae82a6bd0 2024-05-09T08:44:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f82a0e00640 1 --2- 172.21.15.107:0/2404331048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a62b0 0x55fae82b0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f829bfff640 1 --2- 172.21.15.107:0/2404331048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a6bd0 0x55fae82b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f82a5d73740 10 client.0 _ll_register_callbacks cb 0x55fae8201a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f82a0e00640 1 -- 172.21.15.107:0/2404331048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a6bd0 msgr2=0x55fae82b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f82a0e00640 1 --2- 172.21.15.107:0/2404331048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a6bd0 0x55fae82b0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f82a0e00640 1 -- 172.21.15.107:0/2404331048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 msgr2=0x55fae82b11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f82a0e00640 1 --2- 172.21.15.107:0/2404331048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 0x55fae82b11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f82a0e00640 1 -- 172.21.15.107:0/2404331048 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fae7f1f210 con 0x55fae82a62b0 2024-05-09T08:44:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f829bfff640 1 --2- 172.21.15.107:0/2404331048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a6bd0 0x55fae82b0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:44:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f82a1601640 1 --2- 172.21.15.107:0/2404331048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 0x55fae82b11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:44:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f82a0e00640 1 --2- 172.21.15.107:0/2404331048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a62b0 0x55fae82b0720 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f82900124f0 tx=0x7f82900129b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bee940538b5a654d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f827bfff640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:44:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f827bfff640 1 -- 172.21.15.107:0/2404331048 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8290011890 con 0x55fae82a62b0 2024-05-09T08:44:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f82a5d73740 1 -- 172.21.15.107:0/2404331048 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8294014220 con 0x55fae82a62b0 2024-05-09T08:44:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f827bfff640 1 -- 172.21.15.107:0/2404331048 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7f8290011a30 con 0x55fae82a62b0 2024-05-09T08:44:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f827bfff640 1 -- 172.21.15.107:0/2404331048 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8290011c30 con 0x55fae82a62b0 2024-05-09T08:44:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f82a5d73740 10 client.27728 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f82a5d73740 1 -- 172.21.15.107:0/2404331048 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fae7feb620 con 0x55fae82a62b0 2024-05-09T08:44:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.272+0000 7f827bfff640 1 -- 172.21.15.107:0/2404331048 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8294014220 con 0x55fae82a62b0 2024-05-09T08:44:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.273+0000 7f827bfff640 1 -- 172.21.15.107:0/2404331048 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f829001c420 con 0x55fae82a62b0 2024-05-09T08:44:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.273+0000 7f82a5d73740 10 client.27728 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.273+0000 7f82a5d73740 10 client.27728 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.273+0000 7f82a5d73740 20 client.27728 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.273+0000 7f82a5d73740 1 --2- 172.21.15.107:0/2404331048 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fae82c3570 0x55fae82c5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.273+0000 7f82a5d73740 4 client.27728 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:44:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.273+0000 7f82a5d73740 1 -- 172.21.15.107:0/2404331048 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fae8218a70 con 0x55fae82c3570 2024-05-09T08:44:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.273+0000 7f829bfff640 1 --2- 172.21.15.107:0/2404331048 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fae82c3570 0x55fae82c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.273+0000 7f827bfff640 10 client.27728 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.273+0000 7f829bfff640 1 --2- 172.21.15.107:0/2404331048 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fae82c3570 0x55fae82c5980 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3b52087a20489c9 server_cookie=124c4a1ed3fb9e62 in_seq=0 out_seq=0 2024-05-09T08:44:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.274+0000 7f827bfff640 1 -- 172.21.15.107:0/2404331048 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fae8218a70 con 0x55fae82c3570 2024-05-09T08:44:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.274+0000 7f827bfff640 10 client.27728 handle_command_reply: tid=0 2024-05-09T08:44:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.274+0000 7f82a5d73740 1 client.27728 shutdown 2024-05-09T08:44:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.274+0000 7f82a5d73740 1 -- 172.21.15.107:0/2404331048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a62b0 msgr2=0x55fae82b0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.274+0000 7f82a5d73740 1 --2- 172.21.15.107:0/2404331048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a62b0 0x55fae82b0720 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f82900124f0 tx=0x7f82900129b0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.274+0000 7f82a5d73740 1 -- 172.21.15.107:0/2404331048 shutdown_connections 2024-05-09T08:44:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.274+0000 7f82a5d73740 1 --2- 172.21.15.107:0/2404331048 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fae82c3570 0x55fae82c5980 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.274+0000 7f82a5d73740 1 --2- 172.21.15.107:0/2404331048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 0x55fae82b11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.274+0000 7f82a5d73740 1 --2- 172.21.15.107:0/2404331048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a6bd0 0x55fae82b0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.274+0000 7f827bfff640 0 client.27728 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.274+0000 7f82a5d73740 1 --2- 172.21.15.107:0/2404331048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a62b0 0x55fae82b0720 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.274+0000 7f82a5d73740 1 -- 172.21.15.107:0/2404331048 >> 172.21.15.107:0/2404331048 conn(0x55fae82a2960 msgr2=0x55fae82a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.274+0000 7f82a5d73740 1 -- 172.21.15.107:0/2404331048 shutdown_connections 2024-05-09T08:44:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.274+0000 7f82a5d73740 1 -- 172.21.15.107:0/2404331048 wait complete. 2024-05-09T08:44:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.275+0000 7f82a5d73740 20 client.27728 trim_cache size 0 max 16384 2024-05-09T08:44:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.289+0000 7f82a5d73740 1 Processor -- start 2024-05-09T08:44:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.289+0000 7f82a5d73740 1 -- start start 2024-05-09T08:44:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.289+0000 7f82a5d73740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a62b0 0x55fae82b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.289+0000 7f82a5d73740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a6bd0 0x55fae82bf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.289+0000 7f82a5d73740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 0x55fae82af880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.289+0000 7f82a5d73740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f829c1af310 con 0x55fae82b3840 2024-05-09T08:44:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.289+0000 7f82a5d73740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f829c1acbe0 con 0x55fae82a6bd0 2024-05-09T08:44:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.289+0000 7f82a5d73740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f829c1aaa50 con 0x55fae82a62b0 2024-05-09T08:44:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.289+0000 7f82a1601640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 0x55fae82af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.289+0000 7f82a1601640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 0x55fae82af880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58594/0 (socket says 172.21.15.107:58594) 2024-05-09T08:44:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.289+0000 7f82a1601640 1 -- 172.21.15.107:0/4124770488 learned_addr learned my addr 172.21.15.107:0/4124770488 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.289+0000 7f82a0e00640 1 --2- 172.21.15.107:0/4124770488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a62b0 0x55fae82b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.290+0000 7f829bfff640 1 --2- 172.21.15.107:0/4124770488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a6bd0 0x55fae82bf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.290+0000 7f82a1601640 1 -- 172.21.15.107:0/4124770488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a62b0 msgr2=0x55fae82b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:27.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.290+0000 7f82a1601640 1 --2- 172.21.15.107:0/4124770488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a62b0 0x55fae82b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.290+0000 7f82a1601640 1 -- 172.21.15.107:0/4124770488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a6bd0 msgr2=0x55fae82bf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:27.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.290+0000 7f82a1601640 1 --2- 172.21.15.107:0/4124770488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a6bd0 0x55fae82bf2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.290+0000 7f82a1601640 1 -- 172.21.15.107:0/4124770488 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fae829cba0 con 0x55fae82b3840 2024-05-09T08:44:27.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.290+0000 7f82a0e00640 1 --2- 172.21.15.107:0/4124770488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a62b0 0x55fae82b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:44:27.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.290+0000 7f82a1601640 1 --2- 172.21.15.107:0/4124770488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 0x55fae82af880 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f8294014ce0 tx=0x7f8294010620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=469b7e85fab273c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:27.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.290+0000 7f82827fc640 1 -- 172.21.15.107:0/4124770488 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8294010400 con 0x55fae82b3840 2024-05-09T08:44:27.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.290+0000 7f82827fc640 1 -- 172.21.15.107:0/4124770488 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8294010a70 con 0x55fae82b3840 2024-05-09T08:44:27.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.290+0000 7f82827fc640 1 -- 172.21.15.107:0/4124770488 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8294012be0 con 0x55fae82b3840 2024-05-09T08:44:27.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 -- 172.21.15.107:0/4124770488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 msgr2=0x55fae82af880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:27.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 --2- 172.21.15.107:0/4124770488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 0x55fae82af880 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f8294014ce0 tx=0x7f8294010620 comp rx=0 tx=0).stop 2024-05-09T08:44:27.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 -- 172.21.15.107:0/4124770488 shutdown_connections 2024-05-09T08:44:27.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 --2- 172.21.15.107:0/4124770488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82b3840 0x55fae82af880 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 --2- 172.21.15.107:0/4124770488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82a6bd0 0x55fae82bf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 --2- 172.21.15.107:0/4124770488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a62b0 0x55fae82b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 -- 172.21.15.107:0/4124770488 >> 172.21.15.107:0/4124770488 conn(0x55fae82a2960 msgr2=0x55fae82a2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:27.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 -- 172.21.15.107:0/4124770488 shutdown_connections 2024-05-09T08:44:27.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 -- 172.21.15.107:0/4124770488 wait complete. 2024-05-09T08:44:27.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 -- start start 2024-05-09T08:44:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 Processor -- start 2024-05-09T08:44:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82a62b0 0x55fae82bb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a6bd0 0x55fae82bbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82b3840 0x55fae82acb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f829c1ab0a0 con 0x55fae82a62b0 2024-05-09T08:44:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f829c1ac510 con 0x55fae82b3840 2024-05-09T08:44:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f829c1ab770 con 0x55fae82a6bd0 2024-05-09T08:44:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.291+0000 7f82a5d73740 10 client.0 _ll_register_callbacks cb 0x7f82900096c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f82a0e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82a62b0 0x55fae82bb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f82a0e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82a62b0 0x55fae82bb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58602/0 (socket says 172.21.15.107:58602) 2024-05-09T08:44:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f82a0e00640 1 -- 172.21.15.107:0/593923508 learned_addr learned my addr 172.21.15.107:0/593923508 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f82a1601640 1 --2- 172.21.15.107:0/593923508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82b3840 0x55fae82acb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f829bfff640 1 --2- 172.21.15.107:0/593923508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a6bd0 0x55fae82bbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f82a0e00640 1 -- 172.21.15.107:0/593923508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a6bd0 msgr2=0x55fae82bbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f82a0e00640 1 --2- 172.21.15.107:0/593923508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a6bd0 0x55fae82bbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f82a0e00640 1 -- 172.21.15.107:0/593923508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82b3840 msgr2=0x55fae82acb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f82a0e00640 1 --2- 172.21.15.107:0/593923508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82b3840 0x55fae82acb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f82a0e00640 1 -- 172.21.15.107:0/593923508 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fae82ad370 con 0x55fae82a62b0 2024-05-09T08:44:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f82a0e00640 1 --2- 172.21.15.107:0/593923508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82a62b0 0x55fae82bb7e0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f82900124f0 tx=0x7f8290009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8179728c92bb5277 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f827bfff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:44:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f827bfff640 1 -- 172.21.15.107:0/593923508 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829002e070 con 0x55fae82a62b0 2024-05-09T08:44:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f827bfff640 1 -- 172.21.15.107:0/593923508 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9571+0+0 (secure 0 0 0) 0x7f829001c420 con 0x55fae82a62b0 2024-05-09T08:44:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f82a5d73740 1 -- 172.21.15.107:0/593923508 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8294010a70 con 0x55fae82a62b0 2024-05-09T08:44:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f827bfff640 1 -- 172.21.15.107:0/593923508 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f829001ccc0 con 0x55fae82a62b0 2024-05-09T08:44:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f82a5d73740 10 client.18792 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.292+0000 7f82a5d73740 1 -- 172.21.15.107:0/593923508 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fae8329630 con 0x55fae82a62b0 2024-05-09T08:44:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.293+0000 7f827bfff640 1 -- 172.21.15.107:0/593923508 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8294010a70 con 0x55fae82a62b0 2024-05-09T08:44:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.293+0000 7f827bfff640 1 -- 172.21.15.107:0/593923508 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f82900054c0 con 0x55fae82a62b0 2024-05-09T08:44:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.293+0000 7f82a5d73740 10 client.18792 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.293+0000 7f82a5d73740 10 client.18792 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.293+0000 7f82a5d73740 20 client.18792 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.293+0000 7f82a5d73740 1 --2- 172.21.15.107:0/593923508 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fae82c3570 0x55fae832bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.293+0000 7f82a5d73740 4 client.18792 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:44:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.293+0000 7f82a5d73740 1 -- 172.21.15.107:0/593923508 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fae8212dc0 con 0x55fae82c3570 2024-05-09T08:44:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.293+0000 7f829bfff640 1 --2- 172.21.15.107:0/593923508 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fae82c3570 0x55fae832bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.293+0000 7f829bfff640 1 --2- 172.21.15.107:0/593923508 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fae82c3570 0x55fae832bdf0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72c51d9b8b22df6b server_cookie=eb814b48bce9a029 in_seq=0 out_seq=0 2024-05-09T08:44:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:27.293+0000 7f827bfff640 10 client.18792 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:29.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:28 smithi107 ceph-mon[30122]: pgmap v345: 129 pgs: 32 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 82 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 13 MiB/s wr, 3.30k op/s 2024-05-09T08:44:29.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:28 smithi157 ceph-mon[33614]: pgmap v345: 129 pgs: 32 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 82 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 13 MiB/s wr, 3.30k op/s 2024-05-09T08:44:29.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:28 smithi110 ceph-mon[33622]: pgmap v345: 129 pgs: 32 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 82 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 13 MiB/s wr, 3.30k op/s 2024-05-09T08:44:31.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:30 smithi107 ceph-mon[30122]: pgmap v346: 129 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 110 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 11 MiB/s wr, 2.86k op/s 2024-05-09T08:44:31.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:30 smithi157 ceph-mon[33614]: pgmap v346: 129 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 110 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 11 MiB/s wr, 2.86k op/s 2024-05-09T08:44:31.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:30 smithi110 ceph-mon[33622]: pgmap v346: 129 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 110 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 11 MiB/s wr, 2.86k op/s 2024-05-09T08:44:33.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:32 smithi107 ceph-mon[30122]: pgmap v347: 129 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 110 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 10 MiB/s wr, 2.53k op/s 2024-05-09T08:44:33.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:32 smithi107 ceph-mon[30122]: osdmap e104: 12 total, 12 up, 12 in 2024-05-09T08:44:33.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:32 smithi157 ceph-mon[33614]: pgmap v347: 129 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 110 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 10 MiB/s wr, 2.53k op/s 2024-05-09T08:44:33.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:32 smithi157 ceph-mon[33614]: osdmap e104: 12 total, 12 up, 12 in 2024-05-09T08:44:33.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:32 smithi110 ceph-mon[33622]: pgmap v347: 129 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 110 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 10 MiB/s wr, 2.53k op/s 2024-05-09T08:44:33.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:32 smithi110 ceph-mon[33622]: osdmap e104: 12 total, 12 up, 12 in 2024-05-09T08:44:35.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:34 smithi107 ceph-mon[30122]: pgmap v349: 129 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-05-09T08:44:35.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:34 smithi157 ceph-mon[33614]: pgmap v349: 129 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-05-09T08:44:35.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:34 smithi110 ceph-mon[33622]: pgmap v349: 129 pgs: 10 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-05-09T08:44:37.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:36 smithi107 ceph-mon[30122]: pgmap v350: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 MiB/s wr, 2.84k op/s 2024-05-09T08:44:37.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:36 smithi157 ceph-mon[33614]: pgmap v350: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 MiB/s wr, 2.84k op/s 2024-05-09T08:44:37.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:36 smithi110 ceph-mon[33622]: pgmap v350: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 MiB/s wr, 2.84k op/s 2024-05-09T08:44:37.210 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.209+0000 7f827bfff640 1 -- 172.21.15.107:0/593923508 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fae8212dc0 con 0x55fae82c3570 2024-05-09T08:44:37.210 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.209+0000 7f827bfff640 10 client.18792 handle_command_reply: tid=0 2024-05-09T08:44:37.210 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.209+0000 7f82a5d73740 1 client.18792 shutdown 2024-05-09T08:44:37.210 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.209+0000 7f82a5d73740 1 -- 172.21.15.107:0/593923508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82a62b0 msgr2=0x55fae82bb7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:37.210 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.209+0000 7f82a5d73740 1 --2- 172.21.15.107:0/593923508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82a62b0 0x55fae82bb7e0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f82900124f0 tx=0x7f8290009d10 comp rx=0 tx=0).stop 2024-05-09T08:44:37.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.209+0000 7f82a5d73740 1 -- 172.21.15.107:0/593923508 shutdown_connections 2024-05-09T08:44:37.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.209+0000 7f82a5d73740 1 --2- 172.21.15.107:0/593923508 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fae82c3570 0x55fae832bdf0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:37.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.209+0000 7f82a5d73740 1 --2- 172.21.15.107:0/593923508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fae82b3840 0x55fae82acb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:37.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.209+0000 7f82a5d73740 1 --2- 172.21.15.107:0/593923508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fae82a6bd0 0x55fae82bbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:37.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.209+0000 7f82a5d73740 1 --2- 172.21.15.107:0/593923508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fae82a62b0 0x55fae82bb7e0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:37.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.209+0000 7f82a5d73740 1 -- 172.21.15.107:0/593923508 >> 172.21.15.107:0/593923508 conn(0x55fae82a2960 msgr2=0x55fae82a1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:37.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.209+0000 7f827bfff640 0 client.18792 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:37.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.210+0000 7f82a5d73740 1 -- 172.21.15.107:0/593923508 shutdown_connections 2024-05-09T08:44:37.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.210+0000 7f82a5d73740 1 -- 172.21.15.107:0/593923508 wait complete. 2024-05-09T08:44:37.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.210+0000 7f82a5d73740 20 client.18792 trim_cache size 0 max 16384 2024-05-09T08:44:37.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:44:37.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:44:37.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:44:37.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:44:37.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.212+0000 7f82a3064640 1 -- 172.21.15.107:0/4023707265 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f82840a2da0 msgr2=0x7f82840a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:37.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.212+0000 7f82a3064640 1 --2- 172.21.15.107:0/4023707265 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f82840a2da0 0x7f82840a5260 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f829401c000 tx=0x7f829401e000 comp rx=0 tx=0).stop 2024-05-09T08:44:37.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.212+0000 7f82a3064640 1 -- 172.21.15.107:0/4023707265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f829c101120 msgr2=0x7f829c197440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:37.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.212+0000 7f82a3064640 1 --2- 172.21.15.107:0/4023707265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f829c101120 0x7f829c197440 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f828c005940 tx=0x7f828c0042f0 comp rx=0 tx=0).stop 2024-05-09T08:44:37.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.212+0000 7f82a3064640 1 -- 172.21.15.107:0/4023707265 shutdown_connections 2024-05-09T08:44:37.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.212+0000 7f82a3064640 1 --2- 172.21.15.107:0/4023707265 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f82840a2da0 0x7f82840a5260 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:37.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.212+0000 7f82a3064640 1 --2- 172.21.15.107:0/4023707265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f829c106720 0x7f829c19b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:37.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.212+0000 7f82a3064640 1 --2- 172.21.15.107:0/4023707265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f829c101120 0x7f829c197440 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:37.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.212+0000 7f82a3064640 1 --2- 172.21.15.107:0/4023707265 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f829c100760 0x7f829c196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:37.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.212+0000 7f82a3064640 1 -- 172.21.15.107:0/4023707265 >> 172.21.15.107:0/4023707265 conn(0x7f829c0778b0 msgr2=0x7f829c103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:37.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.214+0000 7f82a3064640 1 -- 172.21.15.107:0/4023707265 shutdown_connections 2024-05-09T08:44:37.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:37.214+0000 7f82a3064640 1 -- 172.21.15.107:0/4023707265 wait complete. 2024-05-09T08:44:39.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:38 smithi107 ceph-mon[30122]: pgmap v351: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 10 MiB/s wr, 2.58k op/s 2024-05-09T08:44:39.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:38 smithi157 ceph-mon[33614]: pgmap v351: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 10 MiB/s wr, 2.58k op/s 2024-05-09T08:44:39.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:38 smithi110 ceph-mon[33622]: pgmap v351: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 10 MiB/s wr, 2.58k op/s 2024-05-09T08:44:41.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:40 smithi107 ceph-mon[30122]: pgmap v352: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 8.8 MiB/s wr, 2.21k op/s 2024-05-09T08:44:41.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:40 smithi157 ceph-mon[33614]: pgmap v352: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 8.8 MiB/s wr, 2.21k op/s 2024-05-09T08:44:41.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:40 smithi110 ceph-mon[33622]: pgmap v352: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 8.8 MiB/s wr, 2.21k op/s 2024-05-09T08:44:43.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:42 smithi107 ceph-mon[30122]: pgmap v353: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.1 MiB/s wr, 2.31k op/s 2024-05-09T08:44:43.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:42 smithi157 ceph-mon[33614]: pgmap v353: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.1 MiB/s wr, 2.31k op/s 2024-05-09T08:44:43.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:42 smithi110 ceph-mon[33622]: pgmap v353: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.1 MiB/s wr, 2.31k op/s 2024-05-09T08:44:45.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:44 smithi107 ceph-mon[30122]: pgmap v354: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.1 MiB/s wr, 2.32k op/s 2024-05-09T08:44:45.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:44 smithi157 ceph-mon[33614]: pgmap v354: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.1 MiB/s wr, 2.32k op/s 2024-05-09T08:44:45.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:44 smithi110 ceph-mon[33622]: pgmap v354: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.1 MiB/s wr, 2.32k op/s 2024-05-09T08:44:46.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:45 smithi157 ceph-mon[33614]: pgmap v355: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 1.84k op/s 2024-05-09T08:44:46.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:45 smithi110 ceph-mon[33622]: pgmap v355: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 1.84k op/s 2024-05-09T08:44:46.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:45 smithi107 ceph-mon[30122]: pgmap v355: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 1.84k op/s 2024-05-09T08:44:48.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:48 smithi157 ceph-mon[33614]: pgmap v356: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 2.39k op/s 2024-05-09T08:44:48.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:48 smithi110 ceph-mon[33622]: pgmap v356: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 2.39k op/s 2024-05-09T08:44:48.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:48 smithi107 ceph-mon[30122]: pgmap v356: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 2.39k op/s 2024-05-09T08:44:50.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:50 smithi157 ceph-mon[33614]: pgmap v357: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 9.3 MiB/s wr, 2.36k op/s 2024-05-09T08:44:50.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:50 smithi110 ceph-mon[33622]: pgmap v357: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 9.3 MiB/s wr, 2.36k op/s 2024-05-09T08:44:50.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.455+0000 7fb17ffe2640 1 -- 172.21.15.107:0/2370220800 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb17811b010 msgr2=0x7fb17811d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:50.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.455+0000 7fb17ffe2640 1 --2- 172.21.15.107:0/2370220800 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb17811b010 0x7fb17811d400 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb170009be0 tx=0x7fb17002a750 comp rx=0 tx=0).stop 2024-05-09T08:44:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.460+0000 7fb17ffe2640 1 -- 172.21.15.107:0/2370220800 shutdown_connections 2024-05-09T08:44:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.460+0000 7fb17ffe2640 1 --2- 172.21.15.107:0/2370220800 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb17811b010 0x7fb17811d400 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.460+0000 7fb17ffe2640 1 --2- 172.21.15.107:0/2370220800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb178110400 0x7fb178108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.460+0000 7fb17ffe2640 1 --2- 172.21.15.107:0/2370220800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb178110020 0x7fb178108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.460+0000 7fb17ffe2640 1 -- 172.21.15.107:0/2370220800 >> 172.21.15.107:0/2370220800 conn(0x7fb17806c900 msgr2=0x7fb17806cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:50.462 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.463+0000 7fb17ffe2640 1 -- 172.21.15.107:0/2370220800 shutdown_connections 2024-05-09T08:44:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.463+0000 7fb17ffe2640 1 -- 172.21.15.107:0/2370220800 wait complete. 2024-05-09T08:44:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.464+0000 7fb17ffe2640 1 Processor -- start 2024-05-09T08:44:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.464+0000 7fb17ffe2640 1 -- start start 2024-05-09T08:44:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.464+0000 7fb17ffe2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb178110020 0x7fb17811a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.464+0000 7fb17ffe2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb17811b010 0x7fb1781146e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.464+0000 7fb17ffe2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb178114c20 0x7fb1781150d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.464+0000 7fb17ffe2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb1781110b0 con 0x7fb178114c20 2024-05-09T08:44:50.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.464+0000 7fb17ffe2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb178110f30 con 0x7fb17811b010 2024-05-09T08:44:50.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.464+0000 7fb17ffe2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb178111230 con 0x7fb178110020 2024-05-09T08:44:50.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.464+0000 7fb17e57f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb178114c20 0x7fb1781150d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:50.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.465+0000 7fb17e57f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb178114c20 0x7fb1781150d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46374/0 (socket says 172.21.15.107:46374) 2024-05-09T08:44:50.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.465+0000 7fb17e57f640 1 -- 172.21.15.107:0/1744663073 learned_addr learned my addr 172.21.15.107:0/1744663073 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:50.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.465+0000 7fb17d57d640 1 --2- 172.21.15.107:0/1744663073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb17811b010 0x7fb1781146e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:50.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.465+0000 7fb17dd7e640 1 --2- 172.21.15.107:0/1744663073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb178110020 0x7fb17811a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:50.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.465+0000 7fb17e57f640 1 -- 172.21.15.107:0/1744663073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb178110020 msgr2=0x7fb17811a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:50.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.465+0000 7fb17e57f640 1 --2- 172.21.15.107:0/1744663073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb178110020 0x7fb17811a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.465+0000 7fb17e57f640 1 -- 172.21.15.107:0/1744663073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb17811b010 msgr2=0x7fb1781146e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:50.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.465+0000 7fb17e57f640 1 --2- 172.21.15.107:0/1744663073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb17811b010 0x7fb1781146e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.465+0000 7fb17e57f640 1 -- 172.21.15.107:0/1744663073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb178126050 con 0x7fb178114c20 2024-05-09T08:44:50.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.466+0000 7fb17e57f640 1 --2- 172.21.15.107:0/1744663073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb178114c20 0x7fb1781150d0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fb170034040 tx=0x7fb170003bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:50.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.468+0000 7fb16effd640 1 -- 172.21.15.107:0/1744663073 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb170041020 con 0x7fb178114c20 2024-05-09T08:44:50.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.468+0000 7fb17ffe2640 1 -- 172.21.15.107:0/1744663073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1781262e0 con 0x7fb178114c20 2024-05-09T08:44:50.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.468+0000 7fb17ffe2640 1 -- 172.21.15.107:0/1744663073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1781267d0 con 0x7fb178114c20 2024-05-09T08:44:50.471 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.472+0000 7fb182dc5740 1 Processor -- start 2024-05-09T08:44:50.473 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.472+0000 7fb182dc5740 1 -- start start 2024-05-09T08:44:50.473 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.472+0000 7fb182dc5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2450 0x564e82dc2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.473 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.472+0000 7fb182dc5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2d70 0x564e82dcd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.473 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.472+0000 7fb182dc5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dcd630 0x564e82dcfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.473 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.472+0000 7fb182dc5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564e82d27340 con 0x564e82dc2d70 2024-05-09T08:44:50.473 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.472+0000 7fb182dc5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564e82ccf290 con 0x564e82dcd630 2024-05-09T08:44:50.473 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.473+0000 7fb182dc5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564e82d23300 con 0x564e82dc2450 2024-05-09T08:44:50.473 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.473+0000 7fb17d57d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2450 0x564e82dc2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:50.473 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.474+0000 7fb17d57d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2450 0x564e82dc2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:36182/0 (socket says 172.21.15.107:36182) 2024-05-09T08:44:50.473 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.474+0000 7fb17d57d640 1 -- 172.21.15.107:0/2641321886 learned_addr learned my addr 172.21.15.107:0/2641321886 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:50.474 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.474+0000 7fb17d57d640 1 --2- 172.21.15.107:0/2641321886 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dcd630 0x564e82dcfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:50.474 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.474+0000 7fb16effd640 1 -- 172.21.15.107:0/1744663073 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb170037040 con 0x7fb178114c20 2024-05-09T08:44:50.474 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.474+0000 7fb16effd640 1 -- 172.21.15.107:0/1744663073 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb170036070 con 0x7fb178114c20 2024-05-09T08:44:50.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.477+0000 7fb17dd7e640 1 --2- 172.21.15.107:0/2641321886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2d70 0x564e82dcd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:50.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.479+0000 7fb17d57d640 1 -- 172.21.15.107:0/2641321886 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dcd630 msgr2=0x564e82dcfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:50.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.479+0000 7fb17d57d640 1 --2- 172.21.15.107:0/2641321886 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dcd630 0x564e82dcfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.479+0000 7fb17d57d640 1 -- 172.21.15.107:0/2641321886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2d70 msgr2=0x564e82dcd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:50.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.479+0000 7fb17d57d640 1 --2- 172.21.15.107:0/2641321886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2d70 0x564e82dcd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.479+0000 7fb17d57d640 1 -- 172.21.15.107:0/2641321886 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e82cf38e0 con 0x564e82dc2450 2024-05-09T08:44:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.479+0000 7fb16effd640 1 -- 172.21.15.107:0/1744663073 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb170044020 con 0x7fb178114c20 2024-05-09T08:44:50.482 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.481+0000 7fb16effd640 1 --2- 172.21.15.107:0/1744663073 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb1640a2fd0 0x7fb1640a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.482 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.482+0000 7fb16effd640 1 -- 172.21.15.107:0/1744663073 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7fb1700f0e60 con 0x7fb178114c20 2024-05-09T08:44:50.485 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.486+0000 7fb17d57d640 1 --2- 172.21.15.107:0/2641321886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2450 0x564e82dc2830 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb168012680 tx=0x7fb168009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb5a174d41fb38dd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.486+0000 7fb163fff640 1 -- 172.21.15.107:0/2641321886 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb168012040 con 0x564e82dc2450 2024-05-09T08:44:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.486+0000 7fb163fff640 1 -- 172.21.15.107:0/2641321886 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1680121e0 con 0x564e82dc2450 2024-05-09T08:44:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.486+0000 7fb163fff640 1 -- 172.21.15.107:0/2641321886 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb168012040 con 0x564e82dc2450 2024-05-09T08:44:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.486+0000 7fb17dd7e640 1 --2- 172.21.15.107:0/1744663073 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb1640a2fd0 0x7fb1640a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.487+0000 7fb182dc5740 1 -- 172.21.15.107:0/2641321886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2450 msgr2=0x564e82dc2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.487+0000 7fb182dc5740 1 --2- 172.21.15.107:0/2641321886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2450 0x564e82dc2830 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb168012680 tx=0x7fb168009f90 comp rx=0 tx=0).stop 2024-05-09T08:44:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.489+0000 7fb17dd7e640 1 --2- 172.21.15.107:0/1744663073 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb1640a2fd0 0x7fb1640a5490 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x564e82dc39f0 tx=0x7fb174009260 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.492+0000 7fb182dc5740 1 -- 172.21.15.107:0/2641321886 shutdown_connections 2024-05-09T08:44:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.492+0000 7fb182dc5740 1 --2- 172.21.15.107:0/2641321886 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dcd630 0x564e82dcfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.492+0000 7fb182dc5740 1 --2- 172.21.15.107:0/2641321886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2d70 0x564e82dcd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.492+0000 7fb182dc5740 1 --2- 172.21.15.107:0/2641321886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2450 0x564e82dc2830 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.492+0000 7fb182dc5740 1 -- 172.21.15.107:0/2641321886 >> 172.21.15.107:0/2641321886 conn(0x564e82dbe960 msgr2=0x564e82dbed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.494+0000 7fb182dc5740 1 -- 172.21.15.107:0/2641321886 shutdown_connections 2024-05-09T08:44:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.494+0000 7fb182dc5740 1 -- 172.21.15.107:0/2641321886 wait complete. 2024-05-09T08:44:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.495+0000 7fb182dc5740 1 -- start start 2024-05-09T08:44:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.495+0000 7fb182dc5740 1 Processor -- start 2024-05-09T08:44:50.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.496+0000 7fb182dc5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2450 0x564e82dca550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.495 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.496+0000 7fb182dc5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2d70 0x564e82dcaa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.496+0000 7fb182dc5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dcd630 0x564e82dcafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.496+0000 7fb182dc5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564e82dd0260 con 0x564e82dc2450 2024-05-09T08:44:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.496+0000 7fb182dc5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564e82ce2790 con 0x564e82dcd630 2024-05-09T08:44:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.496+0000 7fb182dc5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564e82dd03e0 con 0x564e82dc2d70 2024-05-09T08:44:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.496+0000 7fb182dc5740 10 client.0 _ll_register_callbacks cb 0x564e82d1da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.496+0000 7fb17dd7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2450 0x564e82dca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.497+0000 7fb17dd7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2450 0x564e82dca550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46386/0 (socket says 172.21.15.107:46386) 2024-05-09T08:44:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.497+0000 7fb17dd7e640 1 -- 172.21.15.107:0/1269484765 learned_addr learned my addr 172.21.15.107:0/1269484765 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.497+0000 7fb17dd7e640 1 -- 172.21.15.107:0/1269484765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2d70 msgr2=0x564e82dcaa90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:44:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.497+0000 7fb17dd7e640 1 --2- 172.21.15.107:0/1269484765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2d70 0x564e82dcaa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.497+0000 7fb17dd7e640 1 -- 172.21.15.107:0/1269484765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dcd630 msgr2=0x564e82dcafd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:44:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.497+0000 7fb17dd7e640 1 --2- 172.21.15.107:0/1269484765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dcd630 0x564e82dcafd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.497+0000 7fb17dd7e640 1 -- 172.21.15.107:0/1269484765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e82a3b210 con 0x564e82dc2450 2024-05-09T08:44:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.497+0000 7fb17dd7e640 1 --2- 172.21.15.107:0/1269484765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2450 0x564e82dca550 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fb174004650 tx=0x7fb1740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa58c0a7c8417b7e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:50.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.498+0000 7fb1617fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:44:50.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.498+0000 7fb1617fa640 1 -- 172.21.15.107:0/1269484765 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb174010ce0 con 0x564e82dc2450 2024-05-09T08:44:50.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.498+0000 7fb182dc5740 1 -- 172.21.15.107:0/1269484765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1680121e0 con 0x564e82dc2450 2024-05-09T08:44:50.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.498+0000 7fb1617fa640 1 -- 172.21.15.107:0/1269484765 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7fb1740203f0 con 0x564e82dc2450 2024-05-09T08:44:50.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.499+0000 7fb1617fa640 1 -- 172.21.15.107:0/1269484765 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb1740205f0 con 0x564e82dc2450 2024-05-09T08:44:50.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.499+0000 7fb182dc5740 10 client.18804 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:50.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.499+0000 7fb182dc5740 1 -- 172.21.15.107:0/1269484765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564e82b07620 con 0x564e82dc2450 2024-05-09T08:44:50.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.499+0000 7fb1617fa640 1 -- 172.21.15.107:0/1269484765 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1680121e0 con 0x564e82dc2450 2024-05-09T08:44:50.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.499+0000 7fb1617fa640 1 -- 172.21.15.107:0/1269484765 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fb174028950 con 0x564e82dc2450 2024-05-09T08:44:50.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.500+0000 7fb182dc5740 10 client.18804 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:50.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.500+0000 7fb182dc5740 10 client.18804 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:50.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.500+0000 7fb182dc5740 20 client.18804 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:50.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.500+0000 7fb182dc5740 1 --2- 172.21.15.107:0/1269484765 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564e82ddd550 0x564e82ddf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.500+0000 7fb182dc5740 4 client.18804 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:44:50.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.500+0000 7fb182dc5740 1 -- 172.21.15.107:0/1269484765 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e82d34a70 con 0x564e82ddd550 2024-05-09T08:44:50.500 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.500+0000 7fb17d57d640 1 --2- 172.21.15.107:0/1269484765 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564e82ddd550 0x564e82ddf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:50.505 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.505+0000 7fb17d57d640 1 --2- 172.21.15.107:0/1269484765 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564e82ddd550 0x564e82ddf980 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=614ac9106482faaa server_cookie=c5e62671fa36ce70 in_seq=0 out_seq=0 2024-05-09T08:44:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.507+0000 7fb1617fa640 10 client.18804 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.507+0000 7fb1617fa640 1 -- 172.21.15.107:0/1269484765 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e82d34a70 con 0x564e82ddd550 2024-05-09T08:44:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.507+0000 7fb1617fa640 10 client.18804 handle_command_reply: tid=0 2024-05-09T08:44:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.507+0000 7fb182dc5740 1 client.18804 shutdown 2024-05-09T08:44:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.509+0000 7fb182dc5740 1 -- 172.21.15.107:0/1269484765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2450 msgr2=0x564e82dca550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.509+0000 7fb182dc5740 1 --2- 172.21.15.107:0/1269484765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2450 0x564e82dca550 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fb174004650 tx=0x7fb1740046c0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.511+0000 7fb182dc5740 1 -- 172.21.15.107:0/1269484765 shutdown_connections 2024-05-09T08:44:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.511+0000 7fb182dc5740 1 --2- 172.21.15.107:0/1269484765 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564e82ddd550 0x564e82ddf980 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.511+0000 7fb182dc5740 1 --2- 172.21.15.107:0/1269484765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dcd630 0x564e82dcafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.511+0000 7fb182dc5740 1 --2- 172.21.15.107:0/1269484765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2d70 0x564e82dcaa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.511+0000 7fb182dc5740 1 --2- 172.21.15.107:0/1269484765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2450 0x564e82dca550 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.511+0000 7fb182dc5740 1 -- 172.21.15.107:0/1269484765 >> 172.21.15.107:0/1269484765 conn(0x564e82dbe960 msgr2=0x564e82dc46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.511+0000 7fb1617fa640 0 client.18804 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.519+0000 7fb182dc5740 1 -- 172.21.15.107:0/1269484765 shutdown_connections 2024-05-09T08:44:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.520+0000 7fb182dc5740 1 -- 172.21.15.107:0/1269484765 wait complete. 2024-05-09T08:44:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.524+0000 7fb182dc5740 20 client.18804 trim_cache size 0 max 16384 2024-05-09T08:44:50.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:50 smithi107 ceph-mon[30122]: pgmap v357: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 9.3 MiB/s wr, 2.36k op/s 2024-05-09T08:44:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.581+0000 7fb182dc5740 1 Processor -- start 2024-05-09T08:44:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.581+0000 7fb182dc5740 1 -- start start 2024-05-09T08:44:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.582+0000 7fb182dc5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2450 0x564e82dd9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.582+0000 7fb182dc5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dc2d70 0x564e82de5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.582+0000 7fb182dc5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dcd630 0x564e82de57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.582+0000 7fb182dc5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb15c012940 con 0x564e82dcd630 2024-05-09T08:44:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.582+0000 7fb182dc5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb15c010210 con 0x564e82dc2d70 2024-05-09T08:44:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.582+0000 7fb182dc5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb15c00e060 con 0x564e82dc2450 2024-05-09T08:44:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.582+0000 7fb17dd7e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2450 0x564e82dd9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.582+0000 7fb17dd7e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2450 0x564e82dd9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:36190/0 (socket says 172.21.15.107:36190) 2024-05-09T08:44:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.582+0000 7fb17dd7e640 1 -- 172.21.15.107:0/3318788261 learned_addr learned my addr 172.21.15.107:0/3318788261 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:50.583 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.583+0000 7fb17e57f640 1 --2- 172.21.15.107:0/3318788261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dcd630 0x564e82de57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:50.584 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.583+0000 7fb17e57f640 1 -- 172.21.15.107:0/3318788261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2450 msgr2=0x564e82dd9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:50.584 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.583+0000 7fb17e57f640 1 --2- 172.21.15.107:0/3318788261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2450 0x564e82dd9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.584 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.583+0000 7fb17e57f640 1 -- 172.21.15.107:0/3318788261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dc2d70 msgr2=0x564e82de5270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:44:50.584 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.583+0000 7fb17e57f640 1 --2- 172.21.15.107:0/3318788261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dc2d70 0x564e82de5270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.584 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.583+0000 7fb17e57f640 1 -- 172.21.15.107:0/3318788261 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e82dcfd90 con 0x564e82dcd630 2024-05-09T08:44:50.584 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.583+0000 7fb17e57f640 1 --2- 172.21.15.107:0/3318788261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dcd630 0x564e82de57b0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fb1700ee140 tx=0x7fb1700bce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c8d3f95e86a7f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.588+0000 7fb1637fe640 1 -- 172.21.15.107:0/3318788261 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1700d5030 con 0x564e82dcd630 2024-05-09T08:44:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.588+0000 7fb1637fe640 1 -- 172.21.15.107:0/3318788261 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1700bcfc0 con 0x564e82dcd630 2024-05-09T08:44:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.588+0000 7fb1637fe640 1 -- 172.21.15.107:0/3318788261 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1700ee3f0 con 0x564e82dcd630 2024-05-09T08:44:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.589+0000 7fb182dc5740 1 -- 172.21.15.107:0/3318788261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dcd630 msgr2=0x564e82de57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.589+0000 7fb182dc5740 1 --2- 172.21.15.107:0/3318788261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dcd630 0x564e82de57b0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fb1700ee140 tx=0x7fb1700bce50 comp rx=0 tx=0).stop 2024-05-09T08:44:50.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.590+0000 7fb182dc5740 1 -- 172.21.15.107:0/3318788261 shutdown_connections 2024-05-09T08:44:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.590+0000 7fb182dc5740 1 --2- 172.21.15.107:0/3318788261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dcd630 0x564e82de57b0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.590+0000 7fb182dc5740 1 --2- 172.21.15.107:0/3318788261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dc2d70 0x564e82de5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.590+0000 7fb182dc5740 1 --2- 172.21.15.107:0/3318788261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dc2450 0x564e82dd9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.590 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.590+0000 7fb182dc5740 1 -- 172.21.15.107:0/3318788261 >> 172.21.15.107:0/3318788261 conn(0x564e82dbe960 msgr2=0x564e82dc8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:50.591 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.592+0000 7fb182dc5740 1 -- 172.21.15.107:0/3318788261 shutdown_connections 2024-05-09T08:44:50.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.592+0000 7fb182dc5740 1 -- 172.21.15.107:0/3318788261 wait complete. 2024-05-09T08:44:50.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.593+0000 7fb182dc5740 1 -- start start 2024-05-09T08:44:50.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.593+0000 7fb182dc5740 1 Processor -- start 2024-05-09T08:44:50.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.593+0000 7fb182dc5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2450 0x564e82dd5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.593+0000 7fb182dc5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dc2d70 0x564e82dc6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.593+0000 7fb182dc5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dcd630 0x564e82dc7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.593+0000 7fb182dc5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb15c00e6d0 con 0x564e82dc2450 2024-05-09T08:44:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.593+0000 7fb182dc5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb15c00fb40 con 0x564e82dc2d70 2024-05-09T08:44:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.593+0000 7fb182dc5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb15c00eda0 con 0x564e82dcd630 2024-05-09T08:44:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.593+0000 7fb182dc5740 10 client.0 _ll_register_callbacks cb 0x7fb174009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:50.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.597+0000 7fb17dd7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2450 0x564e82dd5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:50.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.597+0000 7fb17dd7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2450 0x564e82dd5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46402/0 (socket says 172.21.15.107:46402) 2024-05-09T08:44:50.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.597+0000 7fb17dd7e640 1 -- 172.21.15.107:0/810457479 learned_addr learned my addr 172.21.15.107:0/810457479 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.599+0000 7fb17e57f640 1 --2- 172.21.15.107:0/810457479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dcd630 0x564e82dc7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.599+0000 7fb17d57d640 1 --2- 172.21.15.107:0/810457479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dc2d70 0x564e82dc6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.600+0000 7fb17d57d640 1 -- 172.21.15.107:0/810457479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dcd630 msgr2=0x564e82dc7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.600+0000 7fb17d57d640 1 --2- 172.21.15.107:0/810457479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dcd630 0x564e82dc7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.600+0000 7fb17d57d640 1 -- 172.21.15.107:0/810457479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2450 msgr2=0x564e82dd5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.600+0000 7fb17d57d640 1 --2- 172.21.15.107:0/810457479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2450 0x564e82dd5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.600+0000 7fb17d57d640 1 -- 172.21.15.107:0/810457479 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e82dc7a80 con 0x564e82dc2d70 2024-05-09T08:44:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.600+0000 7fb17d57d640 1 --2- 172.21.15.107:0/810457479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dc2d70 0x564e82dc6e50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb168009250 tx=0x7fb168005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32ecf6957aae7f40 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.601+0000 7fb1617fa640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:44:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.601+0000 7fb1617fa640 1 -- 172.21.15.107:0/810457479 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb168012040 con 0x564e82dc2d70 2024-05-09T08:44:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.601+0000 7fb1617fa640 1 -- 172.21.15.107:0/810457479 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7fb1680051b0 con 0x564e82dc2d70 2024-05-09T08:44:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.602+0000 7fb182dc5740 1 -- 172.21.15.107:0/810457479 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1700bcfc0 con 0x564e82dc2d70 2024-05-09T08:44:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.602+0000 7fb1617fa640 1 -- 172.21.15.107:0/810457479 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb1680053d0 con 0x564e82dc2d70 2024-05-09T08:44:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.602+0000 7fb182dc5740 10 client.27746 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.602+0000 7fb182dc5740 1 -- 172.21.15.107:0/810457479 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564e82e43a10 con 0x564e82dc2d70 2024-05-09T08:44:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.603+0000 7fb1617fa640 1 -- 172.21.15.107:0/810457479 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1700bcfc0 con 0x564e82dc2d70 2024-05-09T08:44:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.603+0000 7fb1617fa640 1 -- 172.21.15.107:0/810457479 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fb168026020 con 0x564e82dc2d70 2024-05-09T08:44:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.603+0000 7fb182dc5740 10 client.27746 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.603+0000 7fb182dc5740 10 client.27746 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.603+0000 7fb182dc5740 20 client.27746 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.603+0000 7fb182dc5740 1 --2- 172.21.15.107:0/810457479 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564e82e43de0 0x564e82e461d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.603+0000 7fb182dc5740 4 client.27746 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:44:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.603+0000 7fb182dc5740 1 -- 172.21.15.107:0/810457479 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564e82d2edc0 con 0x564e82e43de0 2024-05-09T08:44:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.603+0000 7fb17dd7e640 1 --2- 172.21.15.107:0/810457479 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564e82e43de0 0x564e82e461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.606+0000 7fb17dd7e640 1 --2- 172.21.15.107:0/810457479 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564e82e43de0 0x564e82e461d0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=685a0a48bd4bc5bd server_cookie=968825fc2ecd9671 in_seq=0 out_seq=0 2024-05-09T08:44:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:50.606+0000 7fb1617fa640 10 client.27746 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.147+0000 7fb1617fa640 1 -- 172.21.15.107:0/810457479 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564e82d2edc0 con 0x564e82e43de0 2024-05-09T08:44:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.147+0000 7fb1617fa640 10 client.27746 handle_command_reply: tid=0 2024-05-09T08:44:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.147+0000 7fb182dc5740 1 client.27746 shutdown 2024-05-09T08:44:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.148+0000 7fb182dc5740 1 -- 172.21.15.107:0/810457479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dc2d70 msgr2=0x564e82dc6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.148+0000 7fb182dc5740 1 --2- 172.21.15.107:0/810457479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dc2d70 0x564e82dc6e50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb168009250 tx=0x7fb168005850 comp rx=0 tx=0).stop 2024-05-09T08:44:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.148+0000 7fb182dc5740 1 -- 172.21.15.107:0/810457479 shutdown_connections 2024-05-09T08:44:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.148+0000 7fb182dc5740 1 --2- 172.21.15.107:0/810457479 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564e82e43de0 0x564e82e461d0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.148+0000 7fb182dc5740 1 --2- 172.21.15.107:0/810457479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564e82dcd630 0x564e82dc7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.148+0000 7fb182dc5740 1 --2- 172.21.15.107:0/810457479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e82dc2d70 0x564e82dc6e50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.148+0000 7fb182dc5740 1 --2- 172.21.15.107:0/810457479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564e82dc2450 0x564e82dd5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.148+0000 7fb182dc5740 1 -- 172.21.15.107:0/810457479 >> 172.21.15.107:0/810457479 conn(0x564e82dbe960 msgr2=0x564e82dbd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.148+0000 7fb1617fa640 0 client.27746 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.148+0000 7fb182dc5740 1 -- 172.21.15.107:0/810457479 shutdown_connections 2024-05-09T08:44:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.148+0000 7fb182dc5740 1 -- 172.21.15.107:0/810457479 wait complete. 2024-05-09T08:44:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.149+0000 7fb182dc5740 20 client.27746 trim_cache size 0 max 16384 2024-05-09T08:44:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:44:51.150 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:44:51.150 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:44:51.150 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:44:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.151+0000 7fb17ffe2640 1 -- 172.21.15.107:0/1744663073 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb1640a2fd0 msgr2=0x7fb1640a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.151+0000 7fb17ffe2640 1 --2- 172.21.15.107:0/1744663073 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb1640a2fd0 0x7fb1640a5490 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x564e82dc39f0 tx=0x7fb174009260 comp rx=0 tx=0).stop 2024-05-09T08:44:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.151+0000 7fb17ffe2640 1 -- 172.21.15.107:0/1744663073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb178114c20 msgr2=0x7fb1781150d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.151+0000 7fb17ffe2640 1 --2- 172.21.15.107:0/1744663073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb178114c20 0x7fb1781150d0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fb170034040 tx=0x7fb170003bf0 comp rx=0 tx=0).stop 2024-05-09T08:44:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.151+0000 7fb17ffe2640 1 -- 172.21.15.107:0/1744663073 shutdown_connections 2024-05-09T08:44:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.151+0000 7fb17ffe2640 1 --2- 172.21.15.107:0/1744663073 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb1640a2fd0 0x7fb1640a5490 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.152+0000 7fb17ffe2640 1 --2- 172.21.15.107:0/1744663073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb178114c20 0x7fb1781150d0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.152+0000 7fb17ffe2640 1 --2- 172.21.15.107:0/1744663073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb17811b010 0x7fb1781146e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.152+0000 7fb17ffe2640 1 --2- 172.21.15.107:0/1744663073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb178110020 0x7fb17811a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.152+0000 7fb17ffe2640 1 -- 172.21.15.107:0/1744663073 >> 172.21.15.107:0/1744663073 conn(0x7fb17806c900 msgr2=0x7fb17811b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.152+0000 7fb17ffe2640 1 -- 172.21.15.107:0/1744663073 shutdown_connections 2024-05-09T08:44:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:44:51.152+0000 7fb17ffe2640 1 -- 172.21.15.107:0/1744663073 wait complete. 2024-05-09T08:44:52.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:52 smithi157 ceph-mon[33614]: pgmap v358: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.6 MiB/s wr, 2.19k op/s 2024-05-09T08:44:52.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:52 smithi110 ceph-mon[33622]: pgmap v358: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.6 MiB/s wr, 2.19k op/s 2024-05-09T08:44:52.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:52 smithi107 ceph-mon[30122]: pgmap v358: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.6 MiB/s wr, 2.19k op/s 2024-05-09T08:44:54.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:54 smithi157 ceph-mon[33614]: pgmap v359: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 11 MiB/s wr, 2.86k op/s 2024-05-09T08:44:54.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:54 smithi110 ceph-mon[33622]: pgmap v359: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 11 MiB/s wr, 2.86k op/s 2024-05-09T08:44:54.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:54 smithi107 ceph-mon[30122]: pgmap v359: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 11 MiB/s wr, 2.86k op/s 2024-05-09T08:44:54.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:44:54.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:44:54.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.803+0000 7f854359e640 1 -- 172.21.15.107:0/2966665510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f854410fed0 msgr2=0x7f85441102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:54.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.803+0000 7f854359e640 1 --2- 172.21.15.107:0/2966665510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f854410fed0 0x7f85441102b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8534009860 tx=0x7f853402a3f0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.807+0000 7f854359e640 1 -- 172.21.15.107:0/2966665510 shutdown_connections 2024-05-09T08:44:54.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.807+0000 7f854359e640 1 --2- 172.21.15.107:0/2966665510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85441092b0 0x7f854411d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.807+0000 7f854359e640 1 --2- 172.21.15.107:0/2966665510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8544108830 0x7f8544108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.807+0000 7f854359e640 1 --2- 172.21.15.107:0/2966665510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f854410fed0 0x7f85441102b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.807+0000 7f854359e640 1 -- 172.21.15.107:0/2966665510 >> 172.21.15.107:0/2966665510 conn(0x7f854406c7b0 msgr2=0x7f854406cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:54.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.808+0000 7f854359e640 1 -- 172.21.15.107:0/2966665510 shutdown_connections 2024-05-09T08:44:54.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.808+0000 7f854359e640 1 -- 172.21.15.107:0/2966665510 wait complete. 2024-05-09T08:44:54.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.809+0000 7f854359e640 1 Processor -- start 2024-05-09T08:44:54.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.814+0000 7f854359e640 1 -- start start 2024-05-09T08:44:54.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.814+0000 7f854359e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8544108830 0x7f854411cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.814+0000 7f854359e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85441092b0 0x7f8544118cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.814+0000 7f854359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f854410fed0 0x7f8544119230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.814+0000 7f854359e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8544113080 con 0x7f854410fed0 2024-05-09T08:44:54.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.814+0000 7f854359e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8544112f00 con 0x7f85441092b0 2024-05-09T08:44:54.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.814+0000 7f854359e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8544113200 con 0x7f8544108830 2024-05-09T08:44:54.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.815+0000 7f8541d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85441092b0 0x7f8544118cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:54.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.815+0000 7f8541d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85441092b0 0x7f8544118cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60272/0 (socket says 172.21.15.107:60272) 2024-05-09T08:44:54.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.815+0000 7f8541d9b640 1 -- 172.21.15.107:0/4283823222 learned_addr learned my addr 172.21.15.107:0/4283823222 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:54.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.816+0000 7f8542d9d640 1 --2- 172.21.15.107:0/4283823222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f854410fed0 0x7f8544119230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:54.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.816+0000 7f854259c640 1 --2- 172.21.15.107:0/4283823222 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8544108830 0x7f854411cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:54.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.816+0000 7f8541d9b640 1 -- 172.21.15.107:0/4283823222 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8544108830 msgr2=0x7f854411cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:54.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.816+0000 7f8541d9b640 1 --2- 172.21.15.107:0/4283823222 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8544108830 0x7f854411cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.816+0000 7f8541d9b640 1 -- 172.21.15.107:0/4283823222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f854410fed0 msgr2=0x7f8544119230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:54.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.816+0000 7f8541d9b640 1 --2- 172.21.15.107:0/4283823222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f854410fed0 0x7f8544119230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.816+0000 7f8541d9b640 1 -- 172.21.15.107:0/4283823222 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85441197f0 con 0x7f85441092b0 2024-05-09T08:44:54.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.816+0000 7f8541d9b640 1 --2- 172.21.15.107:0/4283823222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85441092b0 0x7f8544118cf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f853c00eab0 tx=0x7f853c00ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:54.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.818+0000 7f85337fe640 1 -- 172.21.15.107:0/4283823222 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c00ce40 con 0x7f85441092b0 2024-05-09T08:44:54.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.818+0000 7f85337fe640 1 -- 172.21.15.107:0/4283823222 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f853c011920 con 0x7f85441092b0 2024-05-09T08:44:54.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.818+0000 7f85337fe640 1 -- 172.21.15.107:0/4283823222 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c010aa0 con 0x7f85441092b0 2024-05-09T08:44:54.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.819+0000 7f854359e640 1 -- 172.21.15.107:0/4283823222 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8544119ae0 con 0x7f85441092b0 2024-05-09T08:44:54.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.819+0000 7f854359e640 1 -- 172.21.15.107:0/4283823222 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85441aaf90 con 0x7f85441092b0 2024-05-09T08:44:54.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.823+0000 7f854bc3b740 1 Processor -- start 2024-05-09T08:44:54.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.823+0000 7f854bc3b740 1 -- start start 2024-05-09T08:44:54.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.824+0000 7f854bc3b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906a9430 0x561f906a9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.824+0000 7f854bc3b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9d50 0x561f906b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.824+0000 7f854bc3b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906b4420 0x561f906b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.824+0000 7f854bc3b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561f9060ed00 con 0x561f906a9430 2024-05-09T08:44:54.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.824+0000 7f854bc3b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561f905b64c0 con 0x561f906a9d50 2024-05-09T08:44:54.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.824+0000 7f854bc3b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x561f9060acc0 con 0x561f906b4420 2024-05-09T08:44:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.824+0000 7f854259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9d50 0x561f906b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.824+0000 7f8542d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906a9430 0x561f906a9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.824+0000 7f8542d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906a9430 0x561f906a9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44080/0 (socket says 172.21.15.107:44080) 2024-05-09T08:44:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.824+0000 7f8542d9d640 1 -- 172.21.15.107:0/3561969330 learned_addr learned my addr 172.21.15.107:0/3561969330 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:54.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.825+0000 7f8541d9b640 1 --2- 172.21.15.107:0/3561969330 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906b4420 0x561f906b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:54.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.826+0000 7f8542d9d640 1 -- 172.21.15.107:0/3561969330 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906b4420 msgr2=0x561f906b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:54.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.826+0000 7f8542d9d640 1 --2- 172.21.15.107:0/3561969330 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906b4420 0x561f906b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.826+0000 7f8542d9d640 1 -- 172.21.15.107:0/3561969330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9d50 msgr2=0x561f906b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:54.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.826+0000 7f8542d9d640 1 --2- 172.21.15.107:0/3561969330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9d50 0x561f906b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.826+0000 7f8542d9d640 1 -- 172.21.15.107:0/3561969330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f903ec7d0 con 0x561f906a9430 2024-05-09T08:44:54.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.826+0000 7f8542d9d640 1 --2- 172.21.15.107:0/3561969330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906a9430 0x561f906a9810 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f853801abc0 tx=0x7f85380217b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8215ce8042fb81fa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:54.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.827+0000 7f85337fe640 1 -- 172.21.15.107:0/4283823222 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f853c011ad0 con 0x7f85441092b0 2024-05-09T08:44:54.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.827+0000 7f8530ff9640 1 -- 172.21.15.107:0/3561969330 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853802d480 con 0x561f906a9430 2024-05-09T08:44:54.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.827+0000 7f8530ff9640 1 -- 172.21.15.107:0/3561969330 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f853802dad0 con 0x561f906a9430 2024-05-09T08:44:54.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.828+0000 7f85337fe640 1 --2- 172.21.15.107:0/4283823222 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f852c0a2f30 0x7f852c0a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.828+0000 7f854bc3b740 1 -- 172.21.15.107:0/3561969330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906a9430 msgr2=0x561f906a9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:54.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.828+0000 7f854bc3b740 1 --2- 172.21.15.107:0/3561969330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906a9430 0x561f906a9810 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f853801abc0 tx=0x7f85380217b0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.828+0000 7f85337fe640 1 -- 172.21.15.107:0/4283823222 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7f853c0cf600 con 0x7f85441092b0 2024-05-09T08:44:54.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.829+0000 7f854259c640 1 --2- 172.21.15.107:0/4283823222 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f852c0a2f30 0x7f852c0a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:54.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.829+0000 7f854bc3b740 1 -- 172.21.15.107:0/3561969330 shutdown_connections 2024-05-09T08:44:54.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.829+0000 7f854bc3b740 1 --2- 172.21.15.107:0/3561969330 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906b4420 0x561f906b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.829+0000 7f854bc3b740 1 --2- 172.21.15.107:0/3561969330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9d50 0x561f906b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.829+0000 7f854bc3b740 1 --2- 172.21.15.107:0/3561969330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906a9430 0x561f906a9810 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.829+0000 7f854bc3b740 1 -- 172.21.15.107:0/3561969330 >> 172.21.15.107:0/3561969330 conn(0x561f906a5750 msgr2=0x561f906a5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:54.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.829+0000 7f854bc3b740 1 -- 172.21.15.107:0/3561969330 shutdown_connections 2024-05-09T08:44:54.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.829+0000 7f854bc3b740 1 -- 172.21.15.107:0/3561969330 wait complete. 2024-05-09T08:44:54.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.830+0000 7f854bc3b740 1 -- start start 2024-05-09T08:44:54.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.830+0000 7f854bc3b740 1 Processor -- start 2024-05-09T08:44:54.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.830+0000 7f854bc3b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906a9430 0x561f906b1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.830+0000 7f854bc3b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9d50 0x561f906b1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.830+0000 7f854bc3b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906b4420 0x561f906b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.830+0000 7f854bc3b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561f906b7260 con 0x561f906b4420 2024-05-09T08:44:54.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.830+0000 7f854bc3b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561f905c9b90 con 0x561f906a9d50 2024-05-09T08:44:54.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.830+0000 7f854bc3b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x561f906b73e0 con 0x561f906a9430 2024-05-09T08:44:54.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.830+0000 7f854bc3b740 10 client.0 _ll_register_callbacks cb 0x561f90605da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:54.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.831+0000 7f8542d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906b4420 0x561f906b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:54.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.831+0000 7f8542d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906b4420 0x561f906b1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44092/0 (socket says 172.21.15.107:44092) 2024-05-09T08:44:54.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.831+0000 7f8542d9d640 1 -- 172.21.15.107:0/4201522637 learned_addr learned my addr 172.21.15.107:0/4201522637 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:54.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.831+0000 7f8541d9b640 1 --2- 172.21.15.107:0/4201522637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9d50 0x561f906b1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:54.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.831+0000 7f8542d9d640 1 -- 172.21.15.107:0/4201522637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906a9430 msgr2=0x561f906b1350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:44:54.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.831+0000 7f8542d9d640 1 --2- 172.21.15.107:0/4201522637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906a9430 0x561f906b1350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.831+0000 7f8542d9d640 1 -- 172.21.15.107:0/4201522637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9d50 msgr2=0x561f906b1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:54.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.831+0000 7f8542d9d640 1 --2- 172.21.15.107:0/4201522637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9d50 0x561f906b1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.831+0000 7f8542d9d640 1 -- 172.21.15.107:0/4201522637 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f903223f0 con 0x561f906b4420 2024-05-09T08:44:54.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.832+0000 7f8542d9d640 1 --2- 172.21.15.107:0/4201522637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906b4420 0x561f906b1dd0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f8538022500 tx=0x7f85380225d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88760e42db6e7e3d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:54.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.832+0000 7f854259c640 1 --2- 172.21.15.107:0/4283823222 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f852c0a2f30 0x7f852c0a53f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x561f906aa9f0 tx=0x7f8534030620 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:54.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.832+0000 7f8529ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:44:54.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.833+0000 7f8529ffb640 1 -- 172.21.15.107:0/4201522637 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853802d480 con 0x561f906b4420 2024-05-09T08:44:54.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.833+0000 7f854bc3b740 1 -- 172.21.15.107:0/4201522637 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f853802dad0 con 0x561f906b4420 2024-05-09T08:44:54.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.833+0000 7f8529ffb640 1 -- 172.21.15.107:0/4201522637 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7f853802d620 con 0x561f906b4420 2024-05-09T08:44:54.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.833+0000 7f8529ffb640 1 -- 172.21.15.107:0/4201522637 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8538040050 con 0x561f906b4420 2024-05-09T08:44:54.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.833+0000 7f854bc3b740 10 client.18822 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:54.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.833+0000 7f854bc3b740 1 -- 172.21.15.107:0/4201522637 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561f902e20f0 con 0x561f906b4420 2024-05-09T08:44:54.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.834+0000 7f8529ffb640 1 -- 172.21.15.107:0/4201522637 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f853802dad0 con 0x561f906b4420 2024-05-09T08:44:54.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.836+0000 7f8529ffb640 1 -- 172.21.15.107:0/4201522637 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8538003780 con 0x561f906b4420 2024-05-09T08:44:54.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.836+0000 7f854bc3b740 10 client.18822 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:54.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.836+0000 7f854bc3b740 10 client.18822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:54.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.836+0000 7f854bc3b740 20 client.18822 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:54.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.836+0000 7f854bc3b740 1 --2- 172.21.15.107:0/4201522637 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561f906c4b40 0x561f906c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.836+0000 7f854bc3b740 4 client.18822 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:44:54.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.836+0000 7f854bc3b740 1 -- 172.21.15.107:0/4201522637 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f9061c430 con 0x561f906c4b40 2024-05-09T08:44:54.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.836+0000 7f854259c640 1 --2- 172.21.15.107:0/4201522637 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561f906c4b40 0x561f906c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:54.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.839+0000 7f8529ffb640 10 client.18822 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.839+0000 7f854259c640 1 --2- 172.21.15.107:0/4201522637 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561f906c4b40 0x561f906c6fc0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e706a6a52f3659ce server_cookie=eeae546633985e72 in_seq=0 out_seq=0 2024-05-09T08:44:54.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.841+0000 7f8529ffb640 1 -- 172.21.15.107:0/4201522637 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561f9061c430 con 0x561f906c4b40 2024-05-09T08:44:54.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.841+0000 7f8529ffb640 10 client.18822 handle_command_reply: tid=0 2024-05-09T08:44:54.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.841+0000 7f854bc3b740 1 client.18822 shutdown 2024-05-09T08:44:54.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.853+0000 7f854bc3b740 1 -- 172.21.15.107:0/4201522637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906b4420 msgr2=0x561f906b1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:54.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.853+0000 7f854bc3b740 1 --2- 172.21.15.107:0/4201522637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906b4420 0x561f906b1dd0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f8538022500 tx=0x7f85380225d0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.854+0000 7f854bc3b740 1 -- 172.21.15.107:0/4201522637 shutdown_connections 2024-05-09T08:44:54.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.854+0000 7f854bc3b740 1 --2- 172.21.15.107:0/4201522637 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561f906c4b40 0x561f906c6fc0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.854+0000 7f854bc3b740 1 --2- 172.21.15.107:0/4201522637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906b4420 0x561f906b1dd0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.854+0000 7f854bc3b740 1 --2- 172.21.15.107:0/4201522637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9d50 0x561f906b1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.854+0000 7f854bc3b740 1 --2- 172.21.15.107:0/4201522637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906a9430 0x561f906b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.854+0000 7f854bc3b740 1 -- 172.21.15.107:0/4201522637 >> 172.21.15.107:0/4201522637 conn(0x561f906a5750 msgr2=0x561f906ab550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:54.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.854+0000 7f8529ffb640 0 client.18822 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:54.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.855+0000 7f854bc3b740 1 -- 172.21.15.107:0/4201522637 shutdown_connections 2024-05-09T08:44:54.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.855+0000 7f854bc3b740 1 -- 172.21.15.107:0/4201522637 wait complete. 2024-05-09T08:44:54.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.855+0000 7f854bc3b740 20 client.18822 trim_cache size 0 max 16384 2024-05-09T08:44:54.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.916+0000 7f854bc3b740 1 Processor -- start 2024-05-09T08:44:54.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.916+0000 7f854bc3b740 1 -- start start 2024-05-09T08:44:54.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.917+0000 7f854bc3b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9430 0x561f906c01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.924+0000 7f854bc3b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906a9d50 0x561f906b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.924+0000 7f854259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9430 0x561f906c01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:54.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.924+0000 7f854259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9430 0x561f906c01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60302/0 (socket says 172.21.15.107:60302) 2024-05-09T08:44:54.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.924+0000 7f854259c640 1 -- 172.21.15.107:0/1086692870 learned_addr learned my addr 172.21.15.107:0/1086692870 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:54.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.924+0000 7f854bc3b740 1 --2- 172.21.15.107:0/1086692870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906b4420 0x561f906ab3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.924+0000 7f854bc3b740 1 -- 172.21.15.107:0/1086692870 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f85441b7940 con 0x561f906a9d50 2024-05-09T08:44:54.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.924+0000 7f854bc3b740 1 -- 172.21.15.107:0/1086692870 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f85441b5210 con 0x561f906a9430 2024-05-09T08:44:54.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.924+0000 7f854bc3b740 1 -- 172.21.15.107:0/1086692870 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f85441b3110 con 0x561f906b4420 2024-05-09T08:44:54.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.924+0000 7f8542d9d640 1 --2- 172.21.15.107:0/1086692870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906b4420 0x561f906ab3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:54.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.925+0000 7f8542d9d640 1 -- 172.21.15.107:0/1086692870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9430 msgr2=0x561f906c01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:54.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.925+0000 7f8542d9d640 1 --2- 172.21.15.107:0/1086692870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9430 0x561f906c01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.925+0000 7f8542d9d640 1 -- 172.21.15.107:0/1086692870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906a9d50 msgr2=0x561f906b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:54.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.925+0000 7f8542d9d640 1 --2- 172.21.15.107:0/1086692870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906a9d50 0x561f906b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.925+0000 7f8542d9d640 1 -- 172.21.15.107:0/1086692870 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f906acf90 con 0x561f906b4420 2024-05-09T08:44:54.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.928+0000 7f8542d9d640 1 --2- 172.21.15.107:0/1086692870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906b4420 0x561f906ab3a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8538000c00 tx=0x7f853801a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fe7e494a30116ec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:54.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.929+0000 7f852bfff640 1 -- 172.21.15.107:0/1086692870 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8538049070 con 0x561f906b4420 2024-05-09T08:44:54.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.929+0000 7f852bfff640 1 -- 172.21.15.107:0/1086692870 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f853803a040 con 0x561f906b4420 2024-05-09T08:44:54.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.929+0000 7f852bfff640 1 -- 172.21.15.107:0/1086692870 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85380052d0 con 0x561f906b4420 2024-05-09T08:44:54.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.931+0000 7f854bc3b740 1 -- 172.21.15.107:0/1086692870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906b4420 msgr2=0x561f906ab3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:54.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.931+0000 7f854bc3b740 1 --2- 172.21.15.107:0/1086692870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906b4420 0x561f906ab3a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8538000c00 tx=0x7f853801a040 comp rx=0 tx=0).stop 2024-05-09T08:44:54.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.936+0000 7f854bc3b740 1 -- 172.21.15.107:0/1086692870 shutdown_connections 2024-05-09T08:44:54.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.936+0000 7f854bc3b740 1 --2- 172.21.15.107:0/1086692870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906b4420 0x561f906ab3a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.936+0000 7f854bc3b740 1 --2- 172.21.15.107:0/1086692870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906a9d50 0x561f906b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.936+0000 7f854bc3b740 1 --2- 172.21.15.107:0/1086692870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9430 0x561f906c01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.936+0000 7f854bc3b740 1 -- 172.21.15.107:0/1086692870 >> 172.21.15.107:0/1086692870 conn(0x561f906a5750 msgr2=0x561f906aac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:54.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.940+0000 7f854bc3b740 1 -- 172.21.15.107:0/1086692870 shutdown_connections 2024-05-09T08:44:54.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.940+0000 7f854bc3b740 1 -- 172.21.15.107:0/1086692870 wait complete. 2024-05-09T08:44:54.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.940+0000 7f854bc3b740 1 -- start start 2024-05-09T08:44:54.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.940+0000 7f854bc3b740 1 Processor -- start 2024-05-09T08:44:54.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.941+0000 7f854bc3b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9430 0x561f906b7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.941+0000 7f854bc3b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906a9d50 0x561f906b7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.941+0000 7f854bc3b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906b4420 0x561f906b80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.941+0000 7f854259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9430 0x561f906b7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:54.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.941+0000 7f854259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9430 0x561f906b7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60304/0 (socket says 172.21.15.107:60304) 2024-05-09T08:44:54.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.941+0000 7f8541d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906a9d50 0x561f906b7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:54.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.941+0000 7f8541d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906a9d50 0x561f906b7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:35906/0 (socket says 172.21.15.107:35906) 2024-05-09T08:44:54.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.941+0000 7f854259c640 1 -- 172.21.15.107:0/1400556021 learned_addr learned my addr 172.21.15.107:0/1400556021 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:44:54.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.941+0000 7f854bc3b740 1 -- 172.21.15.107:0/1400556021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f85441b36d0 con 0x561f906b4420 2024-05-09T08:44:54.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.941+0000 7f854bc3b740 1 -- 172.21.15.107:0/1400556021 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f85441b4b40 con 0x561f906a9430 2024-05-09T08:44:54.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.941+0000 7f854bc3b740 1 -- 172.21.15.107:0/1400556021 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f85441b3da0 con 0x561f906a9d50 2024-05-09T08:44:54.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.942+0000 7f854bc3b740 10 client.0 _ll_register_callbacks cb 0x7f8538005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:44:54.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.942+0000 7f854259c640 1 -- 172.21.15.107:0/1400556021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906a9d50 msgr2=0x561f906b7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:54.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.942+0000 7f854259c640 1 --2- 172.21.15.107:0/1400556021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906a9d50 0x561f906b7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.942+0000 7f854259c640 1 -- 172.21.15.107:0/1400556021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906b4420 msgr2=0x561f906b80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:44:54.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.942+0000 7f854259c640 1 --2- 172.21.15.107:0/1400556021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906b4420 0x561f906b80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.942+0000 7f854259c640 1 -- 172.21.15.107:0/1400556021 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f90729130 con 0x561f906a9430 2024-05-09T08:44:54.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.942+0000 7f854259c640 1 --2- 172.21.15.107:0/1400556021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9430 0x561f906b7670 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f85340305f0 tx=0x7f8534005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eecfb78e52fcc1ca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:44:54.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.942+0000 7f854bc3b740 1 -- 172.21.15.107:0/1400556021 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f853803a040 con 0x561f906a9430 2024-05-09T08:44:54.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.942+0000 7f8529ffb640 10 client.27764 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:44:54.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.942+0000 7f8529ffb640 1 -- 172.21.15.107:0/1400556021 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534030d20 con 0x561f906a9430 2024-05-09T08:44:54.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.943+0000 7f8529ffb640 1 -- 172.21.15.107:0/1400556021 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7f8534004030 con 0x561f906a9430 2024-05-09T08:44:54.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.943+0000 7f8529ffb640 1 -- 172.21.15.107:0/1400556021 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8534061050 con 0x561f906a9430 2024-05-09T08:44:54.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.946+0000 7f854bc3b740 10 client.27764 fetch_fsmap learned FSMap version 27 2024-05-09T08:44:54.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.946+0000 7f854bc3b740 1 -- 172.21.15.107:0/1400556021 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=27}) -- 0x561f9072a390 con 0x561f906a9430 2024-05-09T08:44:54.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.947+0000 7f8529ffb640 1 -- 172.21.15.107:0/1400556021 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8534004230 con 0x561f906a9430 2024-05-09T08:44:54.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.947+0000 7f854bc3b740 10 client.27764 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:44:54.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.947+0000 7f854bc3b740 10 client.27764 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:44:54.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.947+0000 7f854bc3b740 20 client.27764 populate_metadata read hostname 'smithi107' 2024-05-09T08:44:54.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.947+0000 7f854bc3b740 1 --2- 172.21.15.107:0/1400556021 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561f9072aab0 0x561f9072cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:44:54.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.947+0000 7f854bc3b740 4 client.27764 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:44:54.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.947+0000 7f854bc3b740 1 -- 172.21.15.107:0/1400556021 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f853803a040 con 0x561f9072aab0 2024-05-09T08:44:54.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.948+0000 7f8541d9b640 1 --2- 172.21.15.107:0/1400556021 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561f9072aab0 0x561f9072cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:44:54.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.949+0000 7f8541d9b640 1 --2- 172.21.15.107:0/1400556021 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561f9072aab0 0x561f9072cf10 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=521434ba10d3c68e server_cookie=8faf87f85b53d643 in_seq=0 out_seq=0 2024-05-09T08:44:54.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.949+0000 7f8529ffb640 10 client.27764 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:54.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.949+0000 7f8529ffb640 1 -- 172.21.15.107:0/1400556021 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x7f853803a040 con 0x561f9072aab0 2024-05-09T08:44:54.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.949+0000 7f8529ffb640 10 client.27764 handle_command_reply: tid=0 2024-05-09T08:44:54.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.949+0000 7f854bc3b740 1 client.27764 shutdown 2024-05-09T08:44:54.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.955+0000 7f854bc3b740 1 -- 172.21.15.107:0/1400556021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9430 msgr2=0x561f906b7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:54.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.955+0000 7f854bc3b740 1 --2- 172.21.15.107:0/1400556021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9430 0x561f906b7670 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f85340305f0 tx=0x7f8534005f90 comp rx=0 tx=0).stop 2024-05-09T08:44:54.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.955+0000 7f854bc3b740 1 -- 172.21.15.107:0/1400556021 shutdown_connections 2024-05-09T08:44:54.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.955+0000 7f854bc3b740 1 --2- 172.21.15.107:0/1400556021 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x561f9072aab0 0x561f9072cf10 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.955+0000 7f854bc3b740 1 --2- 172.21.15.107:0/1400556021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561f906b4420 0x561f906b80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.955+0000 7f854bc3b740 1 --2- 172.21.15.107:0/1400556021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561f906a9d50 0x561f906b7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.955+0000 7f854bc3b740 1 --2- 172.21.15.107:0/1400556021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f906a9430 0x561f906b7670 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.955+0000 7f854bc3b740 1 -- 172.21.15.107:0/1400556021 >> 172.21.15.107:0/1400556021 conn(0x561f906a5750 msgr2=0x561f906a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:54.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.955+0000 7f8529ffb640 0 client.27764 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:44:54.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.956+0000 7f854bc3b740 1 -- 172.21.15.107:0/1400556021 shutdown_connections 2024-05-09T08:44:54.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.956+0000 7f854bc3b740 1 -- 172.21.15.107:0/1400556021 wait complete. 2024-05-09T08:44:54.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.957+0000 7f854bc3b740 20 client.27764 trim_cache size 0 max 16384 2024-05-09T08:44:54.972 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:44:54.972 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "scrub active (61 inodes in the stack)", 2024-05-09T08:44:54.972 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:44:54.972 INFO:teuthology.orchestra.run.smithi107.stdout: "196cdfca-7dfe-4c61-8a89-763c753e759f": { 2024-05-09T08:44:54.972 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:44:54.972 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "196cdfca-7dfe-4c61-8a89-763c753e759f", 2024-05-09T08:44:54.972 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:44:54.972 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:44:54.972 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:44:54.972 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:44:54.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.976+0000 7f854359e640 1 -- 172.21.15.107:0/4283823222 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f852c0a2f30 msgr2=0x7f852c0a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:54.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.976+0000 7f854359e640 1 --2- 172.21.15.107:0/4283823222 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f852c0a2f30 0x7f852c0a53f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x561f906aa9f0 tx=0x7f8534030620 comp rx=0 tx=0).stop 2024-05-09T08:44:54.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.976+0000 7f854359e640 1 -- 172.21.15.107:0/4283823222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85441092b0 msgr2=0x7f8544118cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:44:54.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.976+0000 7f854359e640 1 --2- 172.21.15.107:0/4283823222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85441092b0 0x7f8544118cf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f853c00eab0 tx=0x7f853c00ef70 comp rx=0 tx=0).stop 2024-05-09T08:44:54.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.976+0000 7f854359e640 1 -- 172.21.15.107:0/4283823222 shutdown_connections 2024-05-09T08:44:54.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.976+0000 7f854359e640 1 --2- 172.21.15.107:0/4283823222 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f852c0a2f30 0x7f852c0a53f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.976+0000 7f854359e640 1 --2- 172.21.15.107:0/4283823222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f854410fed0 0x7f8544119230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.976+0000 7f854359e640 1 --2- 172.21.15.107:0/4283823222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85441092b0 0x7f8544118cf0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.977+0000 7f854359e640 1 --2- 172.21.15.107:0/4283823222 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8544108830 0x7f854411cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:44:54.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.977+0000 7f854359e640 1 -- 172.21.15.107:0/4283823222 >> 172.21.15.107:0/4283823222 conn(0x7f854406c7b0 msgr2=0x7f85440704b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:44:54.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.981+0000 7f854359e640 1 -- 172.21.15.107:0/4283823222 shutdown_connections 2024-05-09T08:44:54.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:44:54.981+0000 7f854359e640 1 -- 172.21.15.107:0/4283823222 wait complete. 2024-05-09T08:44:55.012 INFO:tasks.cephfs.filesystem:scrub status for tag:196cdfca-7dfe-4c61-8a89-763c753e759f - {'path': '/', 'tag': '196cdfca-7dfe-4c61-8a89-763c753e759f', 'options': 'recursive,force'} 2024-05-09T08:44:56.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:56 smithi110 ceph-mon[33622]: pgmap v360: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 9.1 MiB/s wr, 2.30k op/s 2024-05-09T08:44:56.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:56 smithi107 ceph-mon[30122]: pgmap v360: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 9.1 MiB/s wr, 2.30k op/s 2024-05-09T08:44:56.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:56 smithi157 ceph-mon[33614]: pgmap v360: 129 pgs: 9 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 110 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 9.1 MiB/s wr, 2.30k op/s 2024-05-09T08:44:58.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:44:58 smithi110 ceph-mon[33622]: pgmap v361: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 11 MiB/s wr, 2.77k op/s 2024-05-09T08:44:58.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:44:58 smithi107 ceph-mon[30122]: pgmap v361: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 11 MiB/s wr, 2.77k op/s 2024-05-09T08:44:58.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:44:58 smithi157 ceph-mon[33614]: pgmap v361: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 11 MiB/s wr, 2.77k op/s 2024-05-09T08:45:00.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:00 smithi110 ceph-mon[33622]: pgmap v362: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 10 MiB/s wr, 2.59k op/s 2024-05-09T08:45:00.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:00 smithi107 ceph-mon[30122]: pgmap v362: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 10 MiB/s wr, 2.59k op/s 2024-05-09T08:45:00.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:00 smithi157 ceph-mon[33614]: pgmap v362: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 10 MiB/s wr, 2.59k op/s 2024-05-09T08:45:01.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:01 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:01.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:01 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:01.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:01 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:45:02.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:02 smithi157 ceph-mon[33614]: pgmap v363: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 9.1 MiB/s wr, 2.31k op/s 2024-05-09T08:45:02.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:02 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:45:02.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:02 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:45:02.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:02 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:45:02.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:02 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:45:02.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:02 smithi110 ceph-mon[33622]: pgmap v363: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 9.1 MiB/s wr, 2.31k op/s 2024-05-09T08:45:02.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:02 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:45:02.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:02 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:45:02.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:02 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:45:02.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:02 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:45:03.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:02 smithi107 ceph-mon[30122]: pgmap v363: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 9.1 MiB/s wr, 2.31k op/s 2024-05-09T08:45:03.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:02 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:45:03.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:02 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:45:03.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:02 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:45:03.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:02 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:45:03.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:03 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:45:03.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:03 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:03.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:03 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:03.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:03 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:03.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:03 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:45:03.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:03 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:45:03.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:03 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:03.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:03 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:03.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:03 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:03.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:03 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:45:04.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:03 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:45:04.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:03 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:04.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:03 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:45:04.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:03 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:45:04.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:03 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:45:04.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.331+0000 7f5b14d61640 1 -- 172.21.15.107:0/1794600595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b10108830 msgr2=0x7f5b10108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:04.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.331+0000 7f5b14d61640 1 --2- 172.21.15.107:0/1794600595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b10108830 0x7f5b10108cb0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f5b00008570 tx=0x7f5b0002a1f0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.332+0000 7f5b14d61640 1 -- 172.21.15.107:0/1794600595 shutdown_connections 2024-05-09T08:45:04.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.332+0000 7f5b14d61640 1 --2- 172.21.15.107:0/1794600595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b101091f0 0x7f5b10115120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.332+0000 7f5b14d61640 1 --2- 172.21.15.107:0/1794600595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b10108830 0x7f5b10108cb0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.332+0000 7f5b14d61640 1 --2- 172.21.15.107:0/1794600595 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b1010d290 0x7f5b1010d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.332+0000 7f5b14d61640 1 -- 172.21.15.107:0/1794600595 >> 172.21.15.107:0/1794600595 conn(0x7f5b1006cd90 msgr2=0x7f5b1006d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.337+0000 7f5b14d61640 1 -- 172.21.15.107:0/1794600595 shutdown_connections 2024-05-09T08:45:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.340+0000 7f5b14d61640 1 -- 172.21.15.107:0/1794600595 wait complete. 2024-05-09T08:45:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.341+0000 7f5b14d61640 1 Processor -- start 2024-05-09T08:45:04.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.345+0000 7f5b14d61640 1 -- start start 2024-05-09T08:45:04.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.345+0000 7f5b14d61640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b10108830 0x7f5b1019e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.345+0000 7f5b14d61640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b101091f0 0x7f5b1019e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.345+0000 7f5b14d61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1010d290 0x7f5b101a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.345+0000 7f5b14d61640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101180f0 con 0x7f5b1010d290 2024-05-09T08:45:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.345+0000 7f5b14d61640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5b10117f70 con 0x7f5b101091f0 2024-05-09T08:45:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.345+0000 7f5b14d61640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5b10118270 con 0x7f5b10108830 2024-05-09T08:45:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.345+0000 7f5b0dd9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b101091f0 0x7f5b1019e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.346+0000 7f5b0dd9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b101091f0 0x7f5b1019e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:39444/0 (socket says 172.21.15.107:39444) 2024-05-09T08:45:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.346+0000 7f5b0dd9b640 1 -- 172.21.15.107:0/467934176 learned_addr learned my addr 172.21.15.107:0/467934176 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.347+0000 7f5b0e59c640 1 --2- 172.21.15.107:0/467934176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b10108830 0x7f5b1019e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.347+0000 7f5b0dd9b640 1 -- 172.21.15.107:0/467934176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b10108830 msgr2=0x7f5b1019e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.347+0000 7f5b0dd9b640 1 --2- 172.21.15.107:0/467934176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b10108830 0x7f5b1019e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.347+0000 7f5b0dd9b640 1 -- 172.21.15.107:0/467934176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1010d290 msgr2=0x7f5b101a2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.347+0000 7f5b0dd9b640 1 --2- 172.21.15.107:0/467934176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1010d290 0x7f5b101a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.347+0000 7f5b0dd9b640 1 -- 172.21.15.107:0/467934176 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b101a3480 con 0x7f5b101091f0 2024-05-09T08:45:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.347+0000 7f5b0e59c640 1 --2- 172.21.15.107:0/467934176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b10108830 0x7f5b1019e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:45:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.348+0000 7f5b0dd9b640 1 --2- 172.21.15.107:0/467934176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b101091f0 0x7f5b1019e9d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5b0002a710 tx=0x7f5b000048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.348+0000 7f5aff7fe640 1 -- 172.21.15.107:0/467934176 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0002ab30 con 0x7f5b101091f0 2024-05-09T08:45:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.348+0000 7f5b14d61640 1 -- 172.21.15.107:0/467934176 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b101a3710 con 0x7f5b101091f0 2024-05-09T08:45:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.348+0000 7f5b14d61640 1 -- 172.21.15.107:0/467934176 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b101ab000 con 0x7f5b101091f0 2024-05-09T08:45:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.348+0000 7f5aff7fe640 1 -- 172.21.15.107:0/467934176 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b00040070 con 0x7f5b101091f0 2024-05-09T08:45:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.350+0000 7f5aff7fe640 1 -- 172.21.15.107:0/467934176 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0003b710 con 0x7f5b101091f0 2024-05-09T08:45:04.351 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.352+0000 7f5aff7fe640 1 -- 172.21.15.107:0/467934176 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f5b0000a2e0 con 0x7f5b101091f0 2024-05-09T08:45:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.356+0000 7f5aff7fe640 1 --2- 172.21.15.107:0/467934176 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5af80b3de0 0x7f5af80b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.357+0000 7f5aff7fe640 1 -- 172.21.15.107:0/467934176 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7f5b000ef650 con 0x7f5b101091f0 2024-05-09T08:45:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.357+0000 7f5b0e59c640 1 --2- 172.21.15.107:0/467934176 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5af80b3de0 0x7f5af80b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.360+0000 7f5b17a6d740 1 Processor -- start 2024-05-09T08:45:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.360+0000 7f5b17a6d740 1 -- start start 2024-05-09T08:45:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.360+0000 7f5b17a6d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225baf450 0x55a225baf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.360+0000 7f5b17a6d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bafd70 0x55a225bba0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.360+0000 7f5b17a6d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225bba630 0x55a225bbca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.360+0000 7f5b17a6d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a225b14340 con 0x55a225baf450 2024-05-09T08:45:04.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.360+0000 7f5b17a6d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a225abc290 con 0x55a225bba630 2024-05-09T08:45:04.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.360+0000 7f5b17a6d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a225b10300 con 0x55a225bafd70 2024-05-09T08:45:04.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.360+0000 7f5b0e59c640 1 --2- 172.21.15.107:0/467934176 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5af80b3de0 0x7f5af80b62a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5b04009c20 tx=0x7f5b04005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.362+0000 7f5b0e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225baf450 0x55a225baf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.362+0000 7f5b0e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225baf450 0x55a225baf830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56150/0 (socket says 172.21.15.107:56150) 2024-05-09T08:45:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.362+0000 7f5b0e59c640 1 -- 172.21.15.107:0/372748395 learned_addr learned my addr 172.21.15.107:0/372748395 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.362+0000 7f5b0dd9b640 1 --2- 172.21.15.107:0/372748395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bafd70 0x55a225bba0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.364+0000 7f5b0ed9d640 1 --2- 172.21.15.107:0/372748395 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225bba630 0x55a225bbca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.364+0000 7f5b0e59c640 1 -- 172.21.15.107:0/372748395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bafd70 msgr2=0x55a225bba0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.364+0000 7f5b0e59c640 1 --2- 172.21.15.107:0/372748395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bafd70 0x55a225bba0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.364+0000 7f5b0e59c640 1 -- 172.21.15.107:0/372748395 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225bba630 msgr2=0x55a225bbca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.364+0000 7f5b0e59c640 1 --2- 172.21.15.107:0/372748395 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225bba630 0x55a225bbca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.364+0000 7f5b0e59c640 1 -- 172.21.15.107:0/372748395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a225ae08e0 con 0x55a225baf450 2024-05-09T08:45:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.366+0000 7f5b0e59c640 1 --2- 172.21.15.107:0/372748395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225baf450 0x55a225baf830 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f5b04004160 tx=0x7f5b04014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cee781ba465cde5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.367+0000 7f5afcff9640 1 -- 172.21.15.107:0/372748395 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b04003e30 con 0x55a225baf450 2024-05-09T08:45:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.367+0000 7f5afcff9640 1 -- 172.21.15.107:0/372748395 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b04025070 con 0x55a225baf450 2024-05-09T08:45:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.367+0000 7f5afcff9640 1 -- 172.21.15.107:0/372748395 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b04020780 con 0x55a225baf450 2024-05-09T08:45:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.372+0000 7f5b17a6d740 1 -- 172.21.15.107:0/372748395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225baf450 msgr2=0x55a225baf830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.372+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/372748395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225baf450 0x55a225baf830 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f5b04004160 tx=0x7f5b04014000 comp rx=0 tx=0).stop 2024-05-09T08:45:04.372 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.373+0000 7f5b17a6d740 1 -- 172.21.15.107:0/372748395 shutdown_connections 2024-05-09T08:45:04.372 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.373+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/372748395 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225bba630 0x55a225bbca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.372 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.373+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/372748395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bafd70 0x55a225bba0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.372 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.373+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/372748395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225baf450 0x55a225baf830 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.372 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.373+0000 7f5b17a6d740 1 -- 172.21.15.107:0/372748395 >> 172.21.15.107:0/372748395 conn(0x55a225bab960 msgr2=0x55a225babd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:04.373 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.374+0000 7f5b17a6d740 1 -- 172.21.15.107:0/372748395 shutdown_connections 2024-05-09T08:45:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.374+0000 7f5b17a6d740 1 -- 172.21.15.107:0/372748395 wait complete. 2024-05-09T08:45:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.374+0000 7f5b17a6d740 1 -- start start 2024-05-09T08:45:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.375+0000 7f5b17a6d740 1 Processor -- start 2024-05-09T08:45:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.375+0000 7f5b17a6d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225baf450 0x55a225bb75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.375+0000 7f5b17a6d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225bafd70 0x55a225bb7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.375+0000 7f5b17a6d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bba630 0x55a225bb8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.375+0000 7f5b17a6d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a225bbd260 con 0x55a225baf450 2024-05-09T08:45:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.376+0000 7f5b0dd9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225bafd70 0x55a225bb7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.376+0000 7f5b17a6d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a225acf790 con 0x55a225bafd70 2024-05-09T08:45:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.376+0000 7f5b0dd9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225bafd70 0x55a225bb7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:39456/0 (socket says 172.21.15.107:39456) 2024-05-09T08:45:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.376+0000 7f5b0dd9b640 1 -- 172.21.15.107:0/694989926 learned_addr learned my addr 172.21.15.107:0/694989926 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.376+0000 7f5b17a6d740 1 -- 172.21.15.107:0/694989926 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55a225bbd3e0 con 0x55a225bba630 2024-05-09T08:45:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.376+0000 7f5b0e59c640 1 --2- 172.21.15.107:0/694989926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225baf450 0x55a225bb75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.376+0000 7f5b17a6d740 10 client.0 _ll_register_callbacks cb 0x55a225b0aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.376+0000 7f5b0ed9d640 1 --2- 172.21.15.107:0/694989926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bba630 0x55a225bb8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.377+0000 7f5b0e59c640 1 -- 172.21.15.107:0/694989926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bba630 msgr2=0x55a225bb8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.377+0000 7f5b0e59c640 1 --2- 172.21.15.107:0/694989926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bba630 0x55a225bb8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.377+0000 7f5b0e59c640 1 -- 172.21.15.107:0/694989926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225bafd70 msgr2=0x55a225bb7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.377+0000 7f5b0e59c640 1 --2- 172.21.15.107:0/694989926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225bafd70 0x55a225bb7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.377+0000 7f5b0e59c640 1 -- 172.21.15.107:0/694989926 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a225828210 con 0x55a225baf450 2024-05-09T08:45:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.377+0000 7f5b0e59c640 1 --2- 172.21.15.107:0/694989926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225baf450 0x55a225bb75b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f5b04011c40 tx=0x7f5b04020660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4225bf526ffb9c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.377+0000 7f5af5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:45:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.378+0000 7f5af5ffb640 1 -- 172.21.15.107:0/694989926 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b04009d10 con 0x55a225baf450 2024-05-09T08:45:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.378+0000 7f5af5ffb640 1 -- 172.21.15.107:0/694989926 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7f5b04012450 con 0x55a225baf450 2024-05-09T08:45:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.378+0000 7f5b17a6d740 1 -- 172.21.15.107:0/694989926 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b04025070 con 0x55a225baf450 2024-05-09T08:45:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.379+0000 7f5af5ffb640 1 -- 172.21.15.107:0/694989926 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5b0403b050 con 0x55a225baf450 2024-05-09T08:45:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.380+0000 7f5b17a6d740 10 client.18831 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:04.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.381+0000 7f5b17a6d740 1 -- 172.21.15.107:0/694989926 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a2258f4620 con 0x55a225baf450 2024-05-09T08:45:04.381 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.382+0000 7f5af5ffb640 1 -- 172.21.15.107:0/694989926 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b04016070 con 0x55a225baf450 2024-05-09T08:45:04.381 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.382+0000 7f5af5ffb640 1 -- 172.21.15.107:0/694989926 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5b040298c0 con 0x55a225baf450 2024-05-09T08:45:04.383 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.384+0000 7f5b17a6d740 10 client.18831 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.384+0000 7f5b17a6d740 10 client.18831 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:45:04.385 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.384+0000 7f5b17a6d740 20 client.18831 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.384+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/694989926 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a225bca110 0x55a225bcc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.385+0000 7f5b17a6d740 4 client.18831 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:45:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.385+0000 7f5b17a6d740 1 -- 172.21.15.107:0/694989926 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5b04025070 con 0x55a225bca110 2024-05-09T08:45:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.385+0000 7f5b0dd9b640 1 --2- 172.21.15.107:0/694989926 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a225bca110 0x55a225bcc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.385+0000 7f5af5ffb640 10 client.18831 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.385+0000 7f5b0dd9b640 1 --2- 172.21.15.107:0/694989926 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a225bca110 0x55a225bcc500 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9806c0e4e6ab58ae server_cookie=1cdd6994a5082355 in_seq=0 out_seq=0 2024-05-09T08:45:04.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.388+0000 7f5af5ffb640 1 -- 172.21.15.107:0/694989926 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f5b04025070 con 0x55a225bca110 2024-05-09T08:45:04.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.388+0000 7f5af5ffb640 10 client.18831 handle_command_reply: tid=0 2024-05-09T08:45:04.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.388+0000 7f5b17a6d740 1 client.18831 shutdown 2024-05-09T08:45:04.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.388+0000 7f5b17a6d740 1 -- 172.21.15.107:0/694989926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225baf450 msgr2=0x55a225bb75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:04.388 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.388+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/694989926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225baf450 0x55a225bb75b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f5b04011c40 tx=0x7f5b04020660 comp rx=0 tx=0).stop 2024-05-09T08:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.390+0000 7f5b17a6d740 1 -- 172.21.15.107:0/694989926 shutdown_connections 2024-05-09T08:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.390+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/694989926 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a225bca110 0x55a225bcc500 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.390+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/694989926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bba630 0x55a225bb8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.390+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/694989926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225bafd70 0x55a225bb7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.390+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/694989926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225baf450 0x55a225bb75b0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.390+0000 7f5b17a6d740 1 -- 172.21.15.107:0/694989926 >> 172.21.15.107:0/694989926 conn(0x55a225bab960 msgr2=0x55a225bb16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.390+0000 7f5af5ffb640 0 client.18831 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.392+0000 7f5b17a6d740 1 -- 172.21.15.107:0/694989926 shutdown_connections 2024-05-09T08:45:04.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.396+0000 7f5b17a6d740 1 -- 172.21.15.107:0/694989926 wait complete. 2024-05-09T08:45:04.396 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.396+0000 7f5b17a6d740 20 client.18831 trim_cache size 0 max 16384 2024-05-09T08:45:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.429+0000 7f5b17a6d740 1 Processor -- start 2024-05-09T08:45:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.430+0000 7f5b17a6d740 1 -- start start 2024-05-09T08:45:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.430+0000 7f5b17a6d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225baf450 0x55a225bc6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.430+0000 7f5b17a6d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225bafd70 0x55a225bb5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.430+0000 7f5b17a6d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bba630 0x55a225bb7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.430+0000 7f5b0e59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225baf450 0x55a225bc6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.430+0000 7f5b0e59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225baf450 0x55a225bc6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:39470/0 (socket says 172.21.15.107:39470) 2024-05-09T08:45:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.431+0000 7f5b0e59c640 1 -- 172.21.15.107:0/2532969194 learned_addr learned my addr 172.21.15.107:0/2532969194 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.431+0000 7f5b17a6d740 1 -- 172.21.15.107:0/2532969194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5af0012940 con 0x55a225bafd70 2024-05-09T08:45:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.431+0000 7f5b17a6d740 1 -- 172.21.15.107:0/2532969194 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5af0010210 con 0x55a225baf450 2024-05-09T08:45:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.431+0000 7f5b17a6d740 1 -- 172.21.15.107:0/2532969194 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5af000e060 con 0x55a225bba630 2024-05-09T08:45:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.431+0000 7f5b0ed9d640 1 --2- 172.21.15.107:0/2532969194 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bba630 0x55a225bb7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.432+0000 7f5b0e59c640 1 -- 172.21.15.107:0/2532969194 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bba630 msgr2=0x55a225bb7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.432+0000 7f5b0e59c640 1 --2- 172.21.15.107:0/2532969194 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bba630 0x55a225bb7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.432+0000 7f5b0e59c640 1 -- 172.21.15.107:0/2532969194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225bafd70 msgr2=0x55a225bb5e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:45:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.432+0000 7f5b0dd9b640 1 --2- 172.21.15.107:0/2532969194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225bafd70 0x55a225bb5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.433+0000 7f5b0e59c640 1 --2- 172.21.15.107:0/2532969194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225bafd70 0x55a225bb5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.433+0000 7f5b0e59c640 1 -- 172.21.15.107:0/2532969194 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a225bb6560 con 0x55a225baf450 2024-05-09T08:45:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.433+0000 7f5b0dd9b640 1 --2- 172.21.15.107:0/2532969194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225bafd70 0x55a225bb5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:45:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.433+0000 7f5b0e59c640 1 --2- 172.21.15.107:0/2532969194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225baf450 0x55a225bc6150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5b040095c0 tx=0x7f5b04009550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8710e20caec14fc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.433+0000 7f5af7fff640 1 -- 172.21.15.107:0/2532969194 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b040207d0 con 0x55a225baf450 2024-05-09T08:45:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.433+0000 7f5af7fff640 1 -- 172.21.15.107:0/2532969194 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b04025070 con 0x55a225baf450 2024-05-09T08:45:04.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.433+0000 7f5af7fff640 1 -- 172.21.15.107:0/2532969194 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b040377f0 con 0x55a225baf450 2024-05-09T08:45:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.434+0000 7f5b17a6d740 1 -- 172.21.15.107:0/2532969194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225baf450 msgr2=0x55a225bc6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.434+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/2532969194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225baf450 0x55a225bc6150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5b040095c0 tx=0x7f5b04009550 comp rx=0 tx=0).stop 2024-05-09T08:45:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.435+0000 7f5b17a6d740 1 -- 172.21.15.107:0/2532969194 shutdown_connections 2024-05-09T08:45:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.435+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/2532969194 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bba630 0x55a225bb7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.435+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/2532969194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225bafd70 0x55a225bb5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.435+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/2532969194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225baf450 0x55a225bc6150 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.435+0000 7f5b17a6d740 1 -- 172.21.15.107:0/2532969194 >> 172.21.15.107:0/2532969194 conn(0x55a225bab960 msgr2=0x55a225baaa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.435+0000 7f5b17a6d740 1 -- 172.21.15.107:0/2532969194 shutdown_connections 2024-05-09T08:45:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.435+0000 7f5b17a6d740 1 -- 172.21.15.107:0/2532969194 wait complete. 2024-05-09T08:45:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.435+0000 7f5b17a6d740 1 -- start start 2024-05-09T08:45:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.435+0000 7f5b17a6d740 1 Processor -- start 2024-05-09T08:45:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.435+0000 7f5b17a6d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bafd70 0x55a225bbe300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.435+0000 7f5b17a6d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225bba630 0x55a225bbe840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.435+0000 7f5b17a6d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225bbed80 0x55a225c2f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.435+0000 7f5b17a6d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5af000e6d0 con 0x55a225bbed80 2024-05-09T08:45:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.435+0000 7f5b17a6d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5af000fb40 con 0x55a225bba630 2024-05-09T08:45:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.435+0000 7f5b17a6d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5af000eda0 con 0x55a225bafd70 2024-05-09T08:45:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.435+0000 7f5b17a6d740 10 client.0 _ll_register_callbacks cb 0x7f5b00005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.436+0000 7f5b0ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225bbed80 0x55a225c2f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.436+0000 7f5b0e59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bafd70 0x55a225bbe300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.436+0000 7f5b0ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225bbed80 0x55a225c2f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56172/0 (socket says 172.21.15.107:56172) 2024-05-09T08:45:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.436+0000 7f5b0e59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bafd70 0x55a225bbe300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:58802/0 (socket says 172.21.15.107:58802) 2024-05-09T08:45:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.436+0000 7f5b0ed9d640 1 -- 172.21.15.107:0/3453977898 learned_addr learned my addr 172.21.15.107:0/3453977898 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.436+0000 7f5b0dd9b640 1 --2- 172.21.15.107:0/3453977898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225bba630 0x55a225bbe840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.436+0000 7f5b0ed9d640 1 -- 172.21.15.107:0/3453977898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bafd70 msgr2=0x55a225bbe300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.436+0000 7f5b0ed9d640 1 --2- 172.21.15.107:0/3453977898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bafd70 0x55a225bbe300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.436+0000 7f5b0ed9d640 1 -- 172.21.15.107:0/3453977898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225bba630 msgr2=0x55a225bbe840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.436+0000 7f5b0ed9d640 1 --2- 172.21.15.107:0/3453977898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225bba630 0x55a225bbe840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.436+0000 7f5b0ed9d640 1 -- 172.21.15.107:0/3453977898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a225c2fb70 con 0x55a225bbed80 2024-05-09T08:45:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.436+0000 7f5b0ed9d640 1 --2- 172.21.15.107:0/3453977898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225bbed80 0x55a225c2f4b0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f5b08016d50 tx=0x7f5b0800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc65378e00b32b9b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.436+0000 7f5af5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:45:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.437+0000 7f5af5ffb640 1 -- 172.21.15.107:0/3453977898 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0801e030 con 0x55a225bbed80 2024-05-09T08:45:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.437+0000 7f5af5ffb640 1 -- 172.21.15.107:0/3453977898 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9511+0+0 (secure 0 0 0) 0x7f5b08009040 con 0x55a225bbed80 2024-05-09T08:45:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.437+0000 7f5b17a6d740 1 -- 172.21.15.107:0/3453977898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b04025070 con 0x55a225bbed80 2024-05-09T08:45:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.438+0000 7f5af5ffb640 1 -- 172.21.15.107:0/3453977898 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5b08028050 con 0x55a225bbed80 2024-05-09T08:45:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.438+0000 7f5b17a6d740 10 client.18834 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.438+0000 7f5b17a6d740 1 -- 172.21.15.107:0/3453977898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a225c30e30 con 0x55a225bbed80 2024-05-09T08:45:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.439+0000 7f5af5ffb640 1 -- 172.21.15.107:0/3453977898 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b04025070 con 0x55a225bbed80 2024-05-09T08:45:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.439+0000 7f5af5ffb640 1 -- 172.21.15.107:0/3453977898 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5b080043c0 con 0x55a225bbed80 2024-05-09T08:45:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.439+0000 7f5b17a6d740 10 client.18834 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.439+0000 7f5b17a6d740 10 client.18834 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:45:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.439+0000 7f5b17a6d740 20 client.18834 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.439+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/3453977898 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a225c315e0 0x55a225c33a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.439+0000 7f5b17a6d740 4 client.18834 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:45:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.439+0000 7f5b17a6d740 1 -- 172.21.15.107:0/3453977898 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a225b21a70 con 0x55a225c315e0 2024-05-09T08:45:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.439+0000 7f5b0e59c640 1 --2- 172.21.15.107:0/3453977898 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a225c315e0 0x55a225c33a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.440+0000 7f5af5ffb640 10 client.18834 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:04.440+0000 7f5b0e59c640 1 --2- 172.21.15.107:0/3453977898 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a225c315e0 0x55a225c33a00 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9237343283eadb4 server_cookie=e9e7d29fbdce42ab in_seq=0 out_seq=0 2024-05-09T08:45:04.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:04 smithi157 ceph-mon[33614]: pgmap v364: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 12 MiB/s wr, 3.04k op/s 2024-05-09T08:45:04.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:04 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:45:04.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:04 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:04.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:04 smithi110 ceph-mon[33622]: pgmap v364: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 12 MiB/s wr, 3.04k op/s 2024-05-09T08:45:04.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:04 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:45:04.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:04 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:05.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:04 smithi107 ceph-mon[30122]: pgmap v364: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 12 MiB/s wr, 3.04k op/s 2024-05-09T08:45:05.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:04 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:45:05.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:04 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:05.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:05 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:45:05.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:05 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:06.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:05 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:45:06.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:05 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:06.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:05 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:45:06.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:05 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:06.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:06 smithi110 ceph-mon[33622]: pgmap v365: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 9.4 MiB/s wr, 2.39k op/s 2024-05-09T08:45:07.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:06 smithi107 ceph-mon[30122]: pgmap v365: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 9.4 MiB/s wr, 2.39k op/s 2024-05-09T08:45:07.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:06 smithi157 ceph-mon[33614]: pgmap v365: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 9.4 MiB/s wr, 2.39k op/s 2024-05-09T08:45:08.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:08 smithi110 ceph-mon[33622]: pgmap v366: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 12 MiB/s wr, 3.05k op/s 2024-05-09T08:45:08.894 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:08 smithi157 ceph-mon[33614]: pgmap v366: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 12 MiB/s wr, 3.05k op/s 2024-05-09T08:45:09.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:08 smithi107 ceph-mon[30122]: pgmap v366: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 12 MiB/s wr, 3.05k op/s 2024-05-09T08:45:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.078+0000 7f5af5ffb640 1 -- 172.21.15.107:0/3453977898 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a225b21a70 con 0x55a225c315e0 2024-05-09T08:45:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.078+0000 7f5af5ffb640 10 client.18834 handle_command_reply: tid=0 2024-05-09T08:45:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.078+0000 7f5b17a6d740 1 client.18834 shutdown 2024-05-09T08:45:09.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.078+0000 7f5b17a6d740 1 -- 172.21.15.107:0/3453977898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225bbed80 msgr2=0x55a225c2f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:09.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.078+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/3453977898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225bbed80 0x55a225c2f4b0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f5b08016d50 tx=0x7f5b0800c6a0 comp rx=0 tx=0).stop 2024-05-09T08:45:09.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.079+0000 7f5b17a6d740 1 -- 172.21.15.107:0/3453977898 shutdown_connections 2024-05-09T08:45:09.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.079+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/3453977898 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55a225c315e0 0x55a225c33a00 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:09.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.079+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/3453977898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a225bbed80 0x55a225c2f4b0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:09.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.079+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/3453977898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a225bba630 0x55a225bbe840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:09.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.079+0000 7f5b17a6d740 1 --2- 172.21.15.107:0/3453977898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a225bafd70 0x55a225bbe300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:09.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.079+0000 7f5b17a6d740 1 -- 172.21.15.107:0/3453977898 >> 172.21.15.107:0/3453977898 conn(0x55a225bab960 msgr2=0x55a225bb0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:09.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.079+0000 7f5af5ffb640 0 client.18834 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:09.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.079+0000 7f5b17a6d740 1 -- 172.21.15.107:0/3453977898 shutdown_connections 2024-05-09T08:45:09.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.079+0000 7f5b17a6d740 1 -- 172.21.15.107:0/3453977898 wait complete. 2024-05-09T08:45:09.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.079+0000 7f5b17a6d740 20 client.18834 trim_cache size 0 max 16384 2024-05-09T08:45:09.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:45:09.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:45:09.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:45:09.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:45:09.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.080+0000 7f5b14d61640 1 -- 172.21.15.107:0/467934176 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5af80b3de0 msgr2=0x7f5af80b62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:09.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.080+0000 7f5b14d61640 1 --2- 172.21.15.107:0/467934176 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5af80b3de0 0x7f5af80b62a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5b04009c20 tx=0x7f5b04005f90 comp rx=0 tx=0).stop 2024-05-09T08:45:09.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.080+0000 7f5b14d61640 1 -- 172.21.15.107:0/467934176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b101091f0 msgr2=0x7f5b1019e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:09.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.080+0000 7f5b14d61640 1 --2- 172.21.15.107:0/467934176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b101091f0 0x7f5b1019e9d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5b0002a710 tx=0x7f5b000048f0 comp rx=0 tx=0).stop 2024-05-09T08:45:09.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.080+0000 7f5b14d61640 1 -- 172.21.15.107:0/467934176 shutdown_connections 2024-05-09T08:45:09.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.080+0000 7f5b14d61640 1 --2- 172.21.15.107:0/467934176 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5af80b3de0 0x7f5af80b62a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:09.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.080+0000 7f5b14d61640 1 --2- 172.21.15.107:0/467934176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1010d290 0x7f5b101a2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:09.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.080+0000 7f5b14d61640 1 --2- 172.21.15.107:0/467934176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b101091f0 0x7f5b1019e9d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:09.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.080+0000 7f5b14d61640 1 --2- 172.21.15.107:0/467934176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b10108830 0x7f5b1019e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:09.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.080+0000 7f5b14d61640 1 -- 172.21.15.107:0/467934176 >> 172.21.15.107:0/467934176 conn(0x7f5b1006cd90 msgr2=0x7f5b10070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:09.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.080+0000 7f5b14d61640 1 -- 172.21.15.107:0/467934176 shutdown_connections 2024-05-09T08:45:09.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:09.080+0000 7f5b14d61640 1 -- 172.21.15.107:0/467934176 wait complete. 2024-05-09T08:45:09.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:09 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:45:09.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:09 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:10.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:09 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:45:10.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:09 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:10.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:09 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:45:10.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:09 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:10.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:10 smithi110 ceph-mon[33622]: pgmap v367: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 11 MiB/s wr, 2.85k op/s 2024-05-09T08:45:10.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:10 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:45:10.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:10 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:11.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:10 smithi107 ceph-mon[30122]: pgmap v367: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 11 MiB/s wr, 2.85k op/s 2024-05-09T08:45:11.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:10 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:45:11.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:10 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:11.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:10 smithi157 ceph-mon[33614]: pgmap v367: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 11 MiB/s wr, 2.85k op/s 2024-05-09T08:45:11.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:10 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:45:11.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:10 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:12.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:12 smithi110 ceph-mon[33622]: pgmap v368: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 9.9 MiB/s wr, 2.51k op/s 2024-05-09T08:45:13.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:12 smithi107 ceph-mon[30122]: pgmap v368: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 9.9 MiB/s wr, 2.51k op/s 2024-05-09T08:45:13.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:12 smithi157 ceph-mon[33614]: pgmap v368: 129 pgs: 9 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 111 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 9.9 MiB/s wr, 2.51k op/s 2024-05-09T08:45:14.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:13 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:14.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:13 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:14.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:13 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:15.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:14 smithi107 ceph-mon[30122]: pgmap v369: 129 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 120 active+clean; 15 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 13 MiB/s wr, 3.27k op/s 2024-05-09T08:45:15.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:14 smithi157 ceph-mon[33614]: pgmap v369: 129 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 120 active+clean; 15 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 13 MiB/s wr, 3.27k op/s 2024-05-09T08:45:15.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:14 smithi110 ceph-mon[33622]: pgmap v369: 129 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 120 active+clean; 15 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 13 MiB/s wr, 3.27k op/s 2024-05-09T08:45:17.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:16 smithi107 ceph-mon[30122]: pgmap v370: 129 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 121 active+clean; 15 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 9.8 MiB/s wr, 2.49k op/s 2024-05-09T08:45:17.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:16 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:45:17.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:16 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:45:17.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:16 smithi157 ceph-mon[33614]: pgmap v370: 129 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 121 active+clean; 15 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 9.8 MiB/s wr, 2.49k op/s 2024-05-09T08:45:17.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:16 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:45:17.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:16 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:45:17.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:16 smithi110 ceph-mon[33622]: pgmap v370: 129 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 121 active+clean; 15 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 9.8 MiB/s wr, 2.49k op/s 2024-05-09T08:45:17.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:16 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:45:17.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:16 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:45:18.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:17 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:45:18.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:17 smithi107 ceph-mon[30122]: osdmap e105: 12 total, 12 up, 12 in 2024-05-09T08:45:18.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:17 smithi107 ceph-mon[30122]: osdmap e106: 12 total, 12 up, 12 in 2024-05-09T08:45:18.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:17 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:45:18.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:17 smithi157 ceph-mon[33614]: osdmap e105: 12 total, 12 up, 12 in 2024-05-09T08:45:18.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:17 smithi157 ceph-mon[33614]: osdmap e106: 12 total, 12 up, 12 in 2024-05-09T08:45:18.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:17 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:45:18.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:17 smithi110 ceph-mon[33622]: osdmap e105: 12 total, 12 up, 12 in 2024-05-09T08:45:18.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:17 smithi110 ceph-mon[33622]: osdmap e106: 12 total, 12 up, 12 in 2024-05-09T08:45:18.929 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:18 smithi107 ceph-mon[30122]: pgmap v373: 129 pgs: 27 active+clean+snaptrim_wait, 13 active+clean+snaptrim, 89 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 11 MiB/s wr, 2.74k op/s 2024-05-09T08:45:19.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:18 smithi157 ceph-mon[33614]: pgmap v373: 129 pgs: 27 active+clean+snaptrim_wait, 13 active+clean+snaptrim, 89 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 11 MiB/s wr, 2.74k op/s 2024-05-09T08:45:19.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:18 smithi110 ceph-mon[33622]: pgmap v373: 129 pgs: 27 active+clean+snaptrim_wait, 13 active+clean+snaptrim, 89 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 11 MiB/s wr, 2.74k op/s 2024-05-09T08:45:21.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:20 smithi107 ceph-mon[30122]: pgmap v374: 129 pgs: 51 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 58 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 13 MiB/s wr, 3.23k op/s 2024-05-09T08:45:21.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:20 smithi157 ceph-mon[33614]: pgmap v374: 129 pgs: 51 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 58 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 13 MiB/s wr, 3.23k op/s 2024-05-09T08:45:21.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:20 smithi110 ceph-mon[33622]: pgmap v374: 129 pgs: 51 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 58 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 13 MiB/s wr, 3.23k op/s 2024-05-09T08:45:21.529 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:45:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:45:21.139+0000 7fe274b72700 -1 calc_pg_upmaps abort due to max <= 0 2024-05-09T08:45:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.199+0000 7f2724363640 1 -- 172.21.15.107:0/3026718112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f271c100760 msgr2=0x7f271c100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.199+0000 7f2724363640 1 --2- 172.21.15.107:0/3026718112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f271c100760 0x7f271c100be0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f270c009930 tx=0x7f270c02a420 comp rx=0 tx=0).stop 2024-05-09T08:45:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.199+0000 7f2724363640 1 -- 172.21.15.107:0/3026718112 shutdown_connections 2024-05-09T08:45:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.199+0000 7f2724363640 1 --2- 172.21.15.107:0/3026718112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f271c101120 0x7f271c10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.199+0000 7f2724363640 1 --2- 172.21.15.107:0/3026718112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f271c100760 0x7f271c100be0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.199+0000 7f2724363640 1 --2- 172.21.15.107:0/3026718112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f271c106720 0x7f271c106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.199+0000 7f2724363640 1 -- 172.21.15.107:0/3026718112 >> 172.21.15.107:0/3026718112 conn(0x7f271c0778b0 msgr2=0x7f271c077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.199+0000 7f2724363640 1 -- 172.21.15.107:0/3026718112 shutdown_connections 2024-05-09T08:45:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.199+0000 7f2724363640 1 -- 172.21.15.107:0/3026718112 wait complete. 2024-05-09T08:45:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.200+0000 7f2724363640 1 Processor -- start 2024-05-09T08:45:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.200+0000 7f2724363640 1 -- start start 2024-05-09T08:45:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.200+0000 7f2724363640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f271c100760 0x7f271c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.200+0000 7f2724363640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f271c101120 0x7f271c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.203 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.200+0000 7f2724363640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f271c106720 0x7f271c19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.200+0000 7f2724363640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f271c1109e0 con 0x7f271c106720 2024-05-09T08:45:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.200+0000 7f2724363640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f271c110860 con 0x7f271c101120 2024-05-09T08:45:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.200+0000 7f2724363640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f271c110b60 con 0x7f271c100760 2024-05-09T08:45:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.201+0000 7f2722900640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f271c106720 0x7f271c19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.201+0000 7f27220ff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f271c100760 0x7f271c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.201+0000 7f2722900640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f271c106720 0x7f271c19c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56428/0 (socket says 172.21.15.107:56428) 2024-05-09T08:45:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.201+0000 7f2722900640 1 -- 172.21.15.107:0/3650919816 learned_addr learned my addr 172.21.15.107:0/3650919816 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.201+0000 7f27218fe640 1 --2- 172.21.15.107:0/3650919816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f271c101120 0x7f271c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.201+0000 7f27220ff640 1 -- 172.21.15.107:0/3650919816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f271c101120 msgr2=0x7f271c198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.201+0000 7f27220ff640 1 --2- 172.21.15.107:0/3650919816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f271c101120 0x7f271c198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.201+0000 7f27220ff640 1 -- 172.21.15.107:0/3650919816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f271c106720 msgr2=0x7f271c19c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.201+0000 7f27220ff640 1 --2- 172.21.15.107:0/3650919816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f271c106720 0x7f271c19c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.201+0000 7f27220ff640 1 -- 172.21.15.107:0/3650919816 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271c19cfc0 con 0x7f271c100760 2024-05-09T08:45:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.201+0000 7f2722900640 1 --2- 172.21.15.107:0/3650919816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f271c106720 0x7f271c19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:45:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.201+0000 7f27220ff640 1 --2- 172.21.15.107:0/3650919816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f271c100760 0x7f271c197fd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f271800e9f0 tx=0x7f271800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.202+0000 7f27137fe640 1 -- 172.21.15.107:0/3650919816 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271800ccc0 con 0x7f271c100760 2024-05-09T08:45:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.202+0000 7f2724363640 1 -- 172.21.15.107:0/3650919816 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f271c19d2b0 con 0x7f271c100760 2024-05-09T08:45:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.202+0000 7f27137fe640 1 -- 172.21.15.107:0/3650919816 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27180108e0 con 0x7f271c100760 2024-05-09T08:45:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.202+0000 7f27137fe640 1 -- 172.21.15.107:0/3650919816 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2718018880 con 0x7f271c100760 2024-05-09T08:45:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.202+0000 7f2724363640 1 -- 172.21.15.107:0/3650919816 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f271c1a4b40 con 0x7f271c100760 2024-05-09T08:45:22.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.205+0000 7f27137fe640 1 -- 172.21.15.107:0/3650919816 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f2718010a90 con 0x7f271c100760 2024-05-09T08:45:22.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.206+0000 7f272706a740 1 Processor -- start 2024-05-09T08:45:22.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.206+0000 7f272706a740 1 -- start start 2024-05-09T08:45:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f272706a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318acc450 0x55b318acc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f272706a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318accd70 0x55b318ad70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f272706a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318ad7630 0x55b318ad9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f272706a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b318a31340 con 0x55b318accd70 2024-05-09T08:45:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f272706a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b3189d9290 con 0x55b318ad7630 2024-05-09T08:45:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f272706a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55b318a2d300 con 0x55b318acc450 2024-05-09T08:45:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f27137fe640 1 --2- 172.21.15.107:0/3650919816 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f27080a2da0 0x7f27080a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f27220ff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318accd70 0x55b318ad70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f27218fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318ad7630 0x55b318ad9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f27218fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318ad7630 0x55b318ad9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44120/0 (socket says 172.21.15.107:44120) 2024-05-09T08:45:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f27220ff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318accd70 0x55b318ad70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56434/0 (socket says 172.21.15.107:56434) 2024-05-09T08:45:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f27218fe640 1 -- 172.21.15.107:0/1357910418 learned_addr learned my addr 172.21.15.107:0/1357910418 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f2722900640 1 --2- 172.21.15.107:0/3650919816 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f27080a2da0 0x7f27080a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f27137fe640 1 -- 172.21.15.107:0/3650919816 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7f27180cfb70 con 0x7f271c100760 2024-05-09T08:45:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f27218fe640 1 -- 172.21.15.107:0/1357910418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318acc450 msgr2=0x55b318acc830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:45:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f27218fe640 1 --2- 172.21.15.107:0/1357910418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318acc450 0x55b318acc830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f27218fe640 1 -- 172.21.15.107:0/1357910418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318accd70 msgr2=0x55b318ad70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f27218fe640 1 --2- 172.21.15.107:0/1357910418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318accd70 0x55b318ad70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f27218fe640 1 -- 172.21.15.107:0/1357910418 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3189fd8e0 con 0x55b318ad7630 2024-05-09T08:45:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.207+0000 7f27218fe640 1 --2- 172.21.15.107:0/1357910418 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318ad7630 0x55b318ad9a20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f270c02a940 tx=0x7f270c031ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ffe359f14ece38c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.208+0000 7f2722900640 1 --2- 172.21.15.107:0/3650919816 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f27080a2da0 0x7f27080a5260 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2714009270 tx=0x7f2714009680 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.208+0000 7f2710bfe640 1 -- 172.21.15.107:0/1357910418 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270c040070 con 0x55b318ad7630 2024-05-09T08:45:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.208+0000 7f2710bfe640 1 -- 172.21.15.107:0/1357910418 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f270c031e50 con 0x55b318ad7630 2024-05-09T08:45:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.208+0000 7f2710bfe640 1 -- 172.21.15.107:0/1357910418 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270c035040 con 0x55b318ad7630 2024-05-09T08:45:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 -- 172.21.15.107:0/1357910418 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318ad7630 msgr2=0x55b318ad9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 --2- 172.21.15.107:0/1357910418 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318ad7630 0x55b318ad9a20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f270c02a940 tx=0x7f270c031ba0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 -- 172.21.15.107:0/1357910418 shutdown_connections 2024-05-09T08:45:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 --2- 172.21.15.107:0/1357910418 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318ad7630 0x55b318ad9a20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 --2- 172.21.15.107:0/1357910418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318accd70 0x55b318ad70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 --2- 172.21.15.107:0/1357910418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318acc450 0x55b318acc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 -- 172.21.15.107:0/1357910418 >> 172.21.15.107:0/1357910418 conn(0x55b318ac8960 msgr2=0x55b318ac8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 -- 172.21.15.107:0/1357910418 shutdown_connections 2024-05-09T08:45:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 -- 172.21.15.107:0/1357910418 wait complete. 2024-05-09T08:45:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 -- start start 2024-05-09T08:45:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 Processor -- start 2024-05-09T08:45:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318acc450 0x55b318ad4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318accd70 0x55b318ad4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318ad7630 0x55b318ad4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b318ada200 con 0x55b318ad7630 2024-05-09T08:45:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b3189ec790 con 0x55b318acc450 2024-05-09T08:45:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55b318ada380 con 0x55b318accd70 2024-05-09T08:45:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.209+0000 7f272706a740 10 client.0 _ll_register_callbacks cb 0x55b318a27a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f2722900640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318ad7630 0x55b318ad4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f2722900640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318ad7630 0x55b318ad4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56436/0 (socket says 172.21.15.107:56436) 2024-05-09T08:45:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f2722900640 1 -- 172.21.15.107:0/700955290 learned_addr learned my addr 172.21.15.107:0/700955290 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f27220ff640 1 --2- 172.21.15.107:0/700955290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318acc450 0x55b318ad4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f27218fe640 1 --2- 172.21.15.107:0/700955290 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318accd70 0x55b318ad4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f2722900640 1 -- 172.21.15.107:0/700955290 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318accd70 msgr2=0x55b318ad4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f2722900640 1 --2- 172.21.15.107:0/700955290 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318accd70 0x55b318ad4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f2722900640 1 -- 172.21.15.107:0/700955290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318acc450 msgr2=0x55b318ad4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f2722900640 1 --2- 172.21.15.107:0/700955290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318acc450 0x55b318ad4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f2722900640 1 -- 172.21.15.107:0/700955290 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b318745210 con 0x55b318ad7630 2024-05-09T08:45:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f27218fe640 1 --2- 172.21.15.107:0/700955290 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318accd70 0x55b318ad4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:45:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f27220ff640 1 --2- 172.21.15.107:0/700955290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318acc450 0x55b318ad4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:45:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f2722900640 1 --2- 172.21.15.107:0/700955290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318ad7630 0x55b318ad4fd0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f271400fd20 tx=0x7f2714004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5055812ef9d3bd96 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f2705ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:45:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f2705ffb640 1 -- 172.21.15.107:0/700955290 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271401f620 con 0x55b318ad7630 2024-05-09T08:45:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f2705ffb640 1 -- 172.21.15.107:0/700955290 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7f271401f7c0 con 0x55b318ad7630 2024-05-09T08:45:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.210+0000 7f272706a740 1 -- 172.21.15.107:0/700955290 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f270c031e50 con 0x55b318ad7630 2024-05-09T08:45:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.211+0000 7f2705ffb640 1 -- 172.21.15.107:0/700955290 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f271401f9c0 con 0x55b318ad7630 2024-05-09T08:45:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.211+0000 7f272706a740 10 client.18846 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.211+0000 7f272706a740 1 -- 172.21.15.107:0/700955290 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b318811620 con 0x55b318ad7630 2024-05-09T08:45:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.211+0000 7f2705ffb640 1 -- 172.21.15.107:0/700955290 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f270c031e50 con 0x55b318ad7630 2024-05-09T08:45:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.211+0000 7f2705ffb640 1 -- 172.21.15.107:0/700955290 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f271400f040 con 0x55b318ad7630 2024-05-09T08:45:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.211+0000 7f272706a740 10 client.18846 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.211+0000 7f272706a740 10 client.18846 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:45:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.211+0000 7f272706a740 20 client.18846 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.211+0000 7f272706a740 1 --2- 172.21.15.107:0/700955290 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b318aeb710 0x55b318aedb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.211+0000 7f272706a740 4 client.18846 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:45:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.211+0000 7f272706a740 1 -- 172.21.15.107:0/700955290 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b318a3ea70 con 0x55b318aeb710 2024-05-09T08:45:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.211+0000 7f27220ff640 1 --2- 172.21.15.107:0/700955290 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b318aeb710 0x55b318aedb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.212+0000 7f2705ffb640 10 client.18846 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.212+0000 7f27220ff640 1 --2- 172.21.15.107:0/700955290 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b318aeb710 0x55b318aedb00 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=330131315a85446d server_cookie=78ab472a86ef15ce in_seq=0 out_seq=0 2024-05-09T08:45:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.212+0000 7f2705ffb640 1 -- 172.21.15.107:0/700955290 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b318a3ea70 con 0x55b318aeb710 2024-05-09T08:45:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.212+0000 7f2705ffb640 10 client.18846 handle_command_reply: tid=0 2024-05-09T08:45:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.212+0000 7f272706a740 1 client.18846 shutdown 2024-05-09T08:45:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.212+0000 7f272706a740 1 -- 172.21.15.107:0/700955290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318ad7630 msgr2=0x55b318ad4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.212+0000 7f272706a740 1 --2- 172.21.15.107:0/700955290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318ad7630 0x55b318ad4fd0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f271400fd20 tx=0x7f2714004ab0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.212+0000 7f272706a740 1 -- 172.21.15.107:0/700955290 shutdown_connections 2024-05-09T08:45:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.212+0000 7f272706a740 1 --2- 172.21.15.107:0/700955290 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b318aeb710 0x55b318aedb00 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.212+0000 7f272706a740 1 --2- 172.21.15.107:0/700955290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318ad7630 0x55b318ad4fd0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.212+0000 7f2705ffb640 0 client.18846 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.212+0000 7f272706a740 1 --2- 172.21.15.107:0/700955290 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318accd70 0x55b318ad4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.212+0000 7f272706a740 1 --2- 172.21.15.107:0/700955290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318acc450 0x55b318ad4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.212+0000 7f272706a740 1 -- 172.21.15.107:0/700955290 >> 172.21.15.107:0/700955290 conn(0x55b318ac8960 msgr2=0x55b318ace6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.213+0000 7f272706a740 1 -- 172.21.15.107:0/700955290 shutdown_connections 2024-05-09T08:45:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.213+0000 7f272706a740 1 -- 172.21.15.107:0/700955290 wait complete. 2024-05-09T08:45:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.213+0000 7f272706a740 20 client.18846 trim_cache size 0 max 16384 2024-05-09T08:45:22.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f272706a740 1 Processor -- start 2024-05-09T08:45:22.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f272706a740 1 -- start start 2024-05-09T08:45:22.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f272706a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318acc450 0x55b318ae30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f272706a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318accd70 0x55b318acee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f272706a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318ad7630 0x55b318af3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f272706a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f271c1af390 con 0x55b318accd70 2024-05-09T08:45:22.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f272706a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f271c1acc60 con 0x55b318ad7630 2024-05-09T08:45:22.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f272706a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f271c1aaab0 con 0x55b318acc450 2024-05-09T08:45:22.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f27218fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318accd70 0x55b318acee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f27218fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318accd70 0x55b318acee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56442/0 (socket says 172.21.15.107:56442) 2024-05-09T08:45:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f27218fe640 1 -- 172.21.15.107:0/357101711 learned_addr learned my addr 172.21.15.107:0/357101711 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f27220ff640 1 --2- 172.21.15.107:0/357101711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318acc450 0x55b318ae30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f2722900640 1 --2- 172.21.15.107:0/357101711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318ad7630 0x55b318af3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f27218fe640 1 -- 172.21.15.107:0/357101711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318acc450 msgr2=0x55b318ae30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f27218fe640 1 --2- 172.21.15.107:0/357101711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318acc450 0x55b318ae30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f27218fe640 1 -- 172.21.15.107:0/357101711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318ad7630 msgr2=0x55b318af3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f27218fe640 1 --2- 172.21.15.107:0/357101711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318ad7630 0x55b318af3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f27218fe640 1 -- 172.21.15.107:0/357101711 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b318ac6f00 con 0x55b318accd70 2024-05-09T08:45:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f27220ff640 1 --2- 172.21.15.107:0/357101711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318acc450 0x55b318ae30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:45:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.231+0000 7f2722900640 1 --2- 172.21.15.107:0/357101711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318ad7630 0x55b318af3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:45:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.232+0000 7f27218fe640 1 --2- 172.21.15.107:0/357101711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318accd70 0x55b318acee10 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f270c02a940 tx=0x7f270c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d1fa78e5de62348 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.232+0000 7f2707fff640 1 -- 172.21.15.107:0/357101711 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270c04c070 con 0x55b318accd70 2024-05-09T08:45:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.232+0000 7f2707fff640 1 -- 172.21.15.107:0/357101711 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f270c031e50 con 0x55b318accd70 2024-05-09T08:45:22.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.232+0000 7f2707fff640 1 -- 172.21.15.107:0/357101711 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270c040070 con 0x55b318accd70 2024-05-09T08:45:22.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 -- 172.21.15.107:0/357101711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318accd70 msgr2=0x55b318acee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:22.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 --2- 172.21.15.107:0/357101711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318accd70 0x55b318acee10 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f270c02a940 tx=0x7f270c005d50 comp rx=0 tx=0).stop 2024-05-09T08:45:22.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 -- 172.21.15.107:0/357101711 shutdown_connections 2024-05-09T08:45:22.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 --2- 172.21.15.107:0/357101711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318ad7630 0x55b318af3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 --2- 172.21.15.107:0/357101711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318accd70 0x55b318acee10 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 --2- 172.21.15.107:0/357101711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318acc450 0x55b318ae30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 -- 172.21.15.107:0/357101711 >> 172.21.15.107:0/357101711 conn(0x55b318ac8960 msgr2=0x55b318ac7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:22.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 -- 172.21.15.107:0/357101711 shutdown_connections 2024-05-09T08:45:22.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 -- 172.21.15.107:0/357101711 wait complete. 2024-05-09T08:45:22.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 -- start start 2024-05-09T08:45:22.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 Processor -- start 2024-05-09T08:45:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318acc450 0x55b318ad1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318accd70 0x55b318ad1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318ad7630 0x55b318b50c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f271c1ab120 con 0x55b318ad7630 2024-05-09T08:45:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f271c1ac590 con 0x55b318accd70 2024-05-09T08:45:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f271c1ab7f0 con 0x55b318acc450 2024-05-09T08:45:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f272706a740 10 client.0 _ll_register_callbacks cb 0x7f271400f9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f2722900640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318ad7630 0x55b318b50c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f2722900640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318ad7630 0x55b318b50c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56452/0 (socket says 172.21.15.107:56452) 2024-05-09T08:45:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f2722900640 1 -- 172.21.15.107:0/4048741122 learned_addr learned my addr 172.21.15.107:0/4048741122 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.233+0000 7f27218fe640 1 --2- 172.21.15.107:0/4048741122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318accd70 0x55b318ad1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.234+0000 7f27220ff640 1 --2- 172.21.15.107:0/4048741122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318acc450 0x55b318ad1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.234+0000 7f2722900640 1 -- 172.21.15.107:0/4048741122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318acc450 msgr2=0x55b318ad1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.234+0000 7f2722900640 1 --2- 172.21.15.107:0/4048741122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318acc450 0x55b318ad1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.234+0000 7f2722900640 1 -- 172.21.15.107:0/4048741122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318accd70 msgr2=0x55b318ad1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.234+0000 7f2722900640 1 --2- 172.21.15.107:0/4048741122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318accd70 0x55b318ad1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.234+0000 7f2722900640 1 -- 172.21.15.107:0/4048741122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b318b511c0 con 0x55b318ad7630 2024-05-09T08:45:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.234+0000 7f2722900640 1 --2- 172.21.15.107:0/4048741122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318ad7630 0x55b318b50c80 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f2714027f70 tx=0x7f271401fe10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=517eb70166480520 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.234+0000 7f2705ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:45:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.234+0000 7f2705ffb640 1 -- 172.21.15.107:0/4048741122 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714012cb0 con 0x55b318ad7630 2024-05-09T08:45:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.234+0000 7f2705ffb640 1 -- 172.21.15.107:0/4048741122 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7f2714027400 con 0x55b318ad7630 2024-05-09T08:45:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.234+0000 7f272706a740 1 -- 172.21.15.107:0/4048741122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f270c031e50 con 0x55b318ad7630 2024-05-09T08:45:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.234+0000 7f2705ffb640 1 -- 172.21.15.107:0/4048741122 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2714027600 con 0x55b318ad7630 2024-05-09T08:45:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.234+0000 7f272706a740 10 client.18852 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.234+0000 7f272706a740 1 -- 172.21.15.107:0/4048741122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b318b52480 con 0x55b318ad7630 2024-05-09T08:45:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.234+0000 7f2705ffb640 1 -- 172.21.15.107:0/4048741122 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f270c031e50 con 0x55b318ad7630 2024-05-09T08:45:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.235+0000 7f2705ffb640 1 -- 172.21.15.107:0/4048741122 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f271401dd20 con 0x55b318ad7630 2024-05-09T08:45:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.235+0000 7f272706a740 10 client.18852 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.235+0000 7f272706a740 10 client.18852 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:45:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.235+0000 7f272706a740 20 client.18852 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.235+0000 7f272706a740 1 --2- 172.21.15.107:0/4048741122 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b318b52c00 0x55b318b55020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.235+0000 7f272706a740 4 client.18852 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:45:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.235+0000 7f272706a740 1 -- 172.21.15.107:0/4048741122 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b318a38dc0 con 0x55b318b52c00 2024-05-09T08:45:22.245 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.235+0000 7f27220ff640 1 --2- 172.21.15.107:0/4048741122 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b318b52c00 0x55b318b55020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:22.246 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.235+0000 7f27220ff640 1 --2- 172.21.15.107:0/4048741122 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b318b52c00 0x55b318b55020 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d52d6e6c5ea6dc05 server_cookie=395a2e5839c96d36 in_seq=0 out_seq=0 2024-05-09T08:45:22.246 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:22.235+0000 7f2705ffb640 10 client.18852 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:23.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:22 smithi107 ceph-mon[30122]: pgmap v375: 129 pgs: 51 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 58 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 7.9 MiB/s wr, 1.96k op/s 2024-05-09T08:45:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.054+0000 7f2705ffb640 1 -- 172.21.15.107:0/4048741122 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b318a38dc0 con 0x55b318b52c00 2024-05-09T08:45:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.054+0000 7f2705ffb640 10 client.18852 handle_command_reply: tid=0 2024-05-09T08:45:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.054+0000 7f272706a740 1 client.18852 shutdown 2024-05-09T08:45:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.054+0000 7f272706a740 1 -- 172.21.15.107:0/4048741122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318ad7630 msgr2=0x55b318b50c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.054+0000 7f272706a740 1 --2- 172.21.15.107:0/4048741122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318ad7630 0x55b318b50c80 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f2714027f70 tx=0x7f271401fe10 comp rx=0 tx=0).stop 2024-05-09T08:45:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.055+0000 7f272706a740 1 -- 172.21.15.107:0/4048741122 shutdown_connections 2024-05-09T08:45:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.055+0000 7f272706a740 1 --2- 172.21.15.107:0/4048741122 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55b318b52c00 0x55b318b55020 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.055+0000 7f2705ffb640 0 client.18852 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.055+0000 7f272706a740 1 --2- 172.21.15.107:0/4048741122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b318ad7630 0x55b318b50c80 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.055+0000 7f272706a740 1 --2- 172.21.15.107:0/4048741122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b318accd70 0x55b318ad1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.055+0000 7f272706a740 1 --2- 172.21.15.107:0/4048741122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b318acc450 0x55b318ad1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.055+0000 7f272706a740 1 -- 172.21.15.107:0/4048741122 >> 172.21.15.107:0/4048741122 conn(0x55b318ac8960 msgr2=0x55b318ad44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.055+0000 7f272706a740 1 -- 172.21.15.107:0/4048741122 shutdown_connections 2024-05-09T08:45:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.055+0000 7f272706a740 1 -- 172.21.15.107:0/4048741122 wait complete. 2024-05-09T08:45:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.055+0000 7f272706a740 20 client.18852 trim_cache size 0 max 16384 2024-05-09T08:45:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:45:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:45:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:45:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:45:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.055+0000 7f2724363640 1 -- 172.21.15.107:0/3650919816 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f27080a2da0 msgr2=0x7f27080a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.055+0000 7f2724363640 1 --2- 172.21.15.107:0/3650919816 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f27080a2da0 0x7f27080a5260 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2714009270 tx=0x7f2714009680 comp rx=0 tx=0).stop 2024-05-09T08:45:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.056+0000 7f2724363640 1 -- 172.21.15.107:0/3650919816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f271c100760 msgr2=0x7f271c197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.056+0000 7f2724363640 1 --2- 172.21.15.107:0/3650919816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f271c100760 0x7f271c197fd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f271800e9f0 tx=0x7f271800eeb0 comp rx=0 tx=0).stop 2024-05-09T08:45:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.056+0000 7f2724363640 1 -- 172.21.15.107:0/3650919816 shutdown_connections 2024-05-09T08:45:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.056+0000 7f2724363640 1 --2- 172.21.15.107:0/3650919816 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f27080a2da0 0x7f27080a5260 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.056+0000 7f2724363640 1 --2- 172.21.15.107:0/3650919816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f271c106720 0x7f271c19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.056+0000 7f2724363640 1 --2- 172.21.15.107:0/3650919816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f271c101120 0x7f271c198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.056+0000 7f2724363640 1 --2- 172.21.15.107:0/3650919816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f271c100760 0x7f271c197fd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:23.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.056+0000 7f2724363640 1 -- 172.21.15.107:0/3650919816 >> 172.21.15.107:0/3650919816 conn(0x7f271c0778b0 msgr2=0x7f271c069280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.056+0000 7f2724363640 1 -- 172.21.15.107:0/3650919816 shutdown_connections 2024-05-09T08:45:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:23.056+0000 7f2724363640 1 -- 172.21.15.107:0/3650919816 wait complete. 2024-05-09T08:45:23.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:22 smithi157 ceph-mon[33614]: pgmap v375: 129 pgs: 51 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 58 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 7.9 MiB/s wr, 1.96k op/s 2024-05-09T08:45:23.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:22 smithi110 ceph-mon[33622]: pgmap v375: 129 pgs: 51 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 58 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 7.9 MiB/s wr, 1.96k op/s 2024-05-09T08:45:25.013 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:45:25.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:45:25.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:24 smithi107 ceph-mon[30122]: pgmap v376: 129 pgs: 17 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 102 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 13 MiB/s wr, 3.18k op/s 2024-05-09T08:45:25.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:24 smithi157 ceph-mon[33614]: pgmap v376: 129 pgs: 17 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 102 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 13 MiB/s wr, 3.18k op/s 2024-05-09T08:45:25.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:24 smithi110 ceph-mon[33622]: pgmap v376: 129 pgs: 17 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 102 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 13 MiB/s wr, 3.18k op/s 2024-05-09T08:45:25.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.326+0000 7fdad5d84640 1 -- 172.21.15.107:0/834375690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdad010e0b0 msgr2=0x7fdad010e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:25.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.326+0000 7fdad5d84640 1 --2- 172.21.15.107:0/834375690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdad010e0b0 0x7fdad010e530 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009cd0 tx=0x7fdac802a930 comp rx=0 tx=0).stop 2024-05-09T08:45:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.331+0000 7fdad5d84640 1 -- 172.21.15.107:0/834375690 shutdown_connections 2024-05-09T08:45:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.331+0000 7fdad5d84640 1 --2- 172.21.15.107:0/834375690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdad011b010 0x7fdad011d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.331+0000 7fdad5d84640 1 --2- 172.21.15.107:0/834375690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdad010e0b0 0x7fdad010e530 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.331+0000 7fdad5d84640 1 --2- 172.21.15.107:0/834375690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdad0110080 0x7fdad010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.331+0000 7fdad5d84640 1 -- 172.21.15.107:0/834375690 >> 172.21.15.107:0/834375690 conn(0x7fdad006c940 msgr2=0x7fdad006cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:25.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.338+0000 7fdad5d84640 1 -- 172.21.15.107:0/834375690 shutdown_connections 2024-05-09T08:45:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.342+0000 7fdad5d84640 1 -- 172.21.15.107:0/834375690 wait complete. 2024-05-09T08:45:25.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.342+0000 7fdad5d84640 1 Processor -- start 2024-05-09T08:45:25.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.345+0000 7fdad5d84640 1 -- start start 2024-05-09T08:45:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.345+0000 7fdad5d84640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdad010e0b0 0x7fdad01a6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.345+0000 7fdad5d84640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdad0110080 0x7fdad01a7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.345+0000 7fdad5d84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdad011b010 0x7fdad01ab710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.345+0000 7fdad5d84640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdad0120990 con 0x7fdad011b010 2024-05-09T08:45:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.345+0000 7fdad5d84640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdad0120810 con 0x7fdad010e0b0 2024-05-09T08:45:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.345+0000 7fdad5d84640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdad0120b10 con 0x7fdad0110080 2024-05-09T08:45:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.346+0000 7fdacffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdad011b010 0x7fdad01ab710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.346+0000 7fdacffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdad011b010 0x7fdad01ab710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56468/0 (socket says 172.21.15.107:56468) 2024-05-09T08:45:25.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.346+0000 7fdacffff640 1 -- 172.21.15.107:0/1733743438 learned_addr learned my addr 172.21.15.107:0/1733743438 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.347+0000 7fdacffff640 1 -- 172.21.15.107:0/1733743438 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdad0110080 msgr2=0x7fdad01a7390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:45:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.347+0000 7fdacffff640 1 --2- 172.21.15.107:0/1733743438 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdad0110080 0x7fdad01a7390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.347+0000 7fdacffff640 1 -- 172.21.15.107:0/1733743438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdad010e0b0 msgr2=0x7fdad01a6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.347+0000 7fdacffff640 1 --2- 172.21.15.107:0/1733743438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdad010e0b0 0x7fdad01a6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.347+0000 7fdacffff640 1 -- 172.21.15.107:0/1733743438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad01abe10 con 0x7fdad011b010 2024-05-09T08:45:25.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.347+0000 7fdacffff640 1 --2- 172.21.15.107:0/1733743438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdad011b010 0x7fdad01ab710 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fdac400b3f0 tx=0x7fdac400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.355+0000 7fdaccff9640 1 -- 172.21.15.107:0/1733743438 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac4015020 con 0x7fdad011b010 2024-05-09T08:45:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.355+0000 7fdad5d84640 1 -- 172.21.15.107:0/1733743438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdad01ac100 con 0x7fdad011b010 2024-05-09T08:45:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.355+0000 7fdad5d84640 1 -- 172.21.15.107:0/1733743438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdad01107f0 con 0x7fdad011b010 2024-05-09T08:45:25.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.355+0000 7fdaccff9640 1 -- 172.21.15.107:0/1733743438 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdac4011ce0 con 0x7fdad011b010 2024-05-09T08:45:25.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.356+0000 7fdaccff9640 1 -- 172.21.15.107:0/1733743438 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac4010dd0 con 0x7fdad011b010 2024-05-09T08:45:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.363+0000 7fdaccff9640 1 -- 172.21.15.107:0/1733743438 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fdac4018020 con 0x7fdad011b010 2024-05-09T08:45:25.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.367+0000 7fdad8b68740 1 Processor -- start 2024-05-09T08:45:25.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.373+0000 7fdad8b68740 1 -- start start 2024-05-09T08:45:25.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.373+0000 7fdad8b68740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a0a280 0x563f72a0a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.373+0000 7fdad8b68740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a0aba0 0x563f72a14d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.373+0000 7fdad8b68740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a15270 0x563f72a17660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.373+0000 7fdad8b68740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563f7296fd00 con 0x563f72a0a280 2024-05-09T08:45:25.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.373+0000 7fdad8b68740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563f729174c0 con 0x563f72a0aba0 2024-05-09T08:45:25.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.373+0000 7fdad8b68740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563f7296bcc0 con 0x563f72a15270 2024-05-09T08:45:25.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.373+0000 7fdacf7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a0aba0 0x563f72a14d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:25.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.373+0000 7fdacf7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a0aba0 0x563f72a14d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44176/0 (socket says 172.21.15.107:44176) 2024-05-09T08:45:25.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.373+0000 7fdacf7fe640 1 -- 172.21.15.107:0/3969840917 learned_addr learned my addr 172.21.15.107:0/3969840917 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:25.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.374+0000 7fdacf7fe640 1 -- 172.21.15.107:0/3969840917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a15270 msgr2=0x563f72a17660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:45:25.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.374+0000 7fdacf7fe640 1 --2- 172.21.15.107:0/3969840917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a15270 0x563f72a17660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.374+0000 7fdacf7fe640 1 -- 172.21.15.107:0/3969840917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a0a280 msgr2=0x563f72a0a660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:45:25.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.374+0000 7fdacf7fe640 1 --2- 172.21.15.107:0/3969840917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a0a280 0x563f72a0a660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.374+0000 7fdacf7fe640 1 -- 172.21.15.107:0/3969840917 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f7274d7d0 con 0x563f72a0aba0 2024-05-09T08:45:25.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.374+0000 7fdacf7fe640 1 --2- 172.21.15.107:0/3969840917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a0aba0 0x563f72a14d30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdac0009820 tx=0x7fdac0009ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6644c3a2c0312c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:25.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.382+0000 7fdaccff9640 1 --2- 172.21.15.107:0/1733743438 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdab80a2de0 0x7fdab80a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.382+0000 7fdaccff9640 1 -- 172.21.15.107:0/1733743438 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7fdac40cef30 con 0x7fdad011b010 2024-05-09T08:45:25.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.380+0000 7fdab5ffb640 1 -- 172.21.15.107:0/3969840917 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac0015020 con 0x563f72a0aba0 2024-05-09T08:45:25.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.383+0000 7fdab5ffb640 1 -- 172.21.15.107:0/3969840917 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdac0011420 con 0x563f72a0aba0 2024-05-09T08:45:25.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.385+0000 7fdad8b68740 1 -- 172.21.15.107:0/3969840917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a0aba0 msgr2=0x563f72a14d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:25.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.385+0000 7fdad8b68740 1 --2- 172.21.15.107:0/3969840917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a0aba0 0x563f72a14d30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdac0009820 tx=0x7fdac0009ce0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.388+0000 7fdaceffd640 1 --2- 172.21.15.107:0/1733743438 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdab80a2de0 0x7fdab80a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:25.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.390+0000 7fdad8b68740 1 -- 172.21.15.107:0/3969840917 shutdown_connections 2024-05-09T08:45:25.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.390+0000 7fdad8b68740 1 --2- 172.21.15.107:0/3969840917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a15270 0x563f72a17660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.390+0000 7fdad8b68740 1 --2- 172.21.15.107:0/3969840917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a0aba0 0x563f72a14d30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.390+0000 7fdad8b68740 1 --2- 172.21.15.107:0/3969840917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a0a280 0x563f72a0a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.390+0000 7fdad8b68740 1 -- 172.21.15.107:0/3969840917 >> 172.21.15.107:0/3969840917 conn(0x563f72a065a0 msgr2=0x563f72a06980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:25.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.391+0000 7fdad8b68740 1 -- 172.21.15.107:0/3969840917 shutdown_connections 2024-05-09T08:45:25.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.391+0000 7fdad8b68740 1 -- 172.21.15.107:0/3969840917 wait complete. 2024-05-09T08:45:25.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.391+0000 7fdad8b68740 1 -- start start 2024-05-09T08:45:25.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.391+0000 7fdad8b68740 1 Processor -- start 2024-05-09T08:45:25.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.393+0000 7fdad8b68740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a0a280 0x563f72a120b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.393+0000 7fdad8b68740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a0aba0 0x563f72a125f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.393+0000 7fdad8b68740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a15270 0x563f72a12b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.393+0000 7fdad8b68740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563f72a18100 con 0x563f72a0aba0 2024-05-09T08:45:25.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.393+0000 7fdad8b68740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563f7292ab90 con 0x563f72a15270 2024-05-09T08:45:25.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.393+0000 7fdad8b68740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563f72a18280 con 0x563f72a0a280 2024-05-09T08:45:25.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.393+0000 7fdad8b68740 10 client.0 _ll_register_callbacks cb 0x563f72966da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:25.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.394+0000 7fdacffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a15270 0x563f72a12b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:25.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.394+0000 7fdacffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a15270 0x563f72a12b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44178/0 (socket says 172.21.15.107:44178) 2024-05-09T08:45:25.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.394+0000 7fdacffff640 1 -- 172.21.15.107:0/1806361492 learned_addr learned my addr 172.21.15.107:0/1806361492 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:25.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.396+0000 7fdacf7fe640 1 --2- 172.21.15.107:0/1806361492 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a0a280 0x563f72a120b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.400+0000 7fdacffff640 1 -- 172.21.15.107:0/1806361492 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a0a280 msgr2=0x563f72a120b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.400+0000 7fdacffff640 1 --2- 172.21.15.107:0/1806361492 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a0a280 0x563f72a120b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.400+0000 7fdacffff640 1 -- 172.21.15.107:0/1806361492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a0aba0 msgr2=0x563f72a125f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.400+0000 7fdacffff640 1 --2- 172.21.15.107:0/1806361492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a0aba0 0x563f72a125f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.400+0000 7fdacffff640 1 -- 172.21.15.107:0/1806361492 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f726833f0 con 0x563f72a15270 2024-05-09T08:45:25.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.404+0000 7fdacffff640 1 --2- 172.21.15.107:0/1806361492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a15270 0x563f72a12b30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdac4097ad0 tx=0x7fdac409b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb93eaa8c9d9d1fc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:25.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.405+0000 7fdaceffd640 1 --2- 172.21.15.107:0/1733743438 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdab80a2de0 0x7fdab80a52a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdac8006010 tx=0x7fdac8009990 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:25.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.408+0000 7fdaaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:45:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.408+0000 7fdaaeffd640 1 -- 172.21.15.107:0/1806361492 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac40b4060 con 0x563f72a15270 2024-05-09T08:45:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.408+0000 7fdaaeffd640 1 -- 172.21.15.107:0/1806361492 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7fdac4097770 con 0x563f72a15270 2024-05-09T08:45:25.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.409+0000 7fdad8b68740 1 -- 172.21.15.107:0/1806361492 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdac0011420 con 0x563f72a15270 2024-05-09T08:45:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.414+0000 7fdaaeffd640 1 -- 172.21.15.107:0/1806361492 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdac40a3d70 con 0x563f72a15270 2024-05-09T08:45:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.414+0000 7fdad8b68740 10 client.27806 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.414+0000 7fdad8b68740 1 -- 172.21.15.107:0/1806361492 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563f726430f0 con 0x563f72a15270 2024-05-09T08:45:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.415+0000 7fdaaeffd640 1 -- 172.21.15.107:0/1806361492 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdac0011420 con 0x563f72a15270 2024-05-09T08:45:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.415+0000 7fdaaeffd640 1 -- 172.21.15.107:0/1806361492 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fdac4010380 con 0x563f72a15270 2024-05-09T08:45:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.415+0000 7fdad8b68740 10 client.27806 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.415+0000 7fdad8b68740 10 client.27806 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:45:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.415+0000 7fdad8b68740 20 client.27806 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.415+0000 7fdad8b68740 1 --2- 172.21.15.107:0/1806361492 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563f72a25af0 0x563f72a27f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.415+0000 7fdad8b68740 4 client.27806 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:45:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.415+0000 7fdad8b68740 1 -- 172.21.15.107:0/1806361492 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f7297d430 con 0x563f72a25af0 2024-05-09T08:45:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.416+0000 7fdacf7fe640 1 --2- 172.21.15.107:0/1806361492 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563f72a25af0 0x563f72a27f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:25.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.419+0000 7fdaaeffd640 10 client.27806 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:25.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.420+0000 7fdacf7fe640 1 --2- 172.21.15.107:0/1806361492 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563f72a25af0 0x563f72a27f90 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52fdde2fe619397 server_cookie=11e5db00cb83254d in_seq=0 out_seq=0 2024-05-09T08:45:25.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.421+0000 7fdaaeffd640 1 -- 172.21.15.107:0/1806361492 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563f7297d430 con 0x563f72a25af0 2024-05-09T08:45:25.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.421+0000 7fdaaeffd640 10 client.27806 handle_command_reply: tid=0 2024-05-09T08:45:25.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.421+0000 7fdad8b68740 1 client.27806 shutdown 2024-05-09T08:45:25.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.424+0000 7fdad8b68740 1 -- 172.21.15.107:0/1806361492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a15270 msgr2=0x563f72a12b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:25.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.424+0000 7fdad8b68740 1 --2- 172.21.15.107:0/1806361492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a15270 0x563f72a12b30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdac4097ad0 tx=0x7fdac409b940 comp rx=0 tx=0).stop 2024-05-09T08:45:25.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.426+0000 7fdad8b68740 1 -- 172.21.15.107:0/1806361492 shutdown_connections 2024-05-09T08:45:25.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.426+0000 7fdad8b68740 1 --2- 172.21.15.107:0/1806361492 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563f72a25af0 0x563f72a27f90 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.426+0000 7fdad8b68740 1 --2- 172.21.15.107:0/1806361492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a15270 0x563f72a12b30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.426+0000 7fdad8b68740 1 --2- 172.21.15.107:0/1806361492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a0aba0 0x563f72a125f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.426+0000 7fdad8b68740 1 --2- 172.21.15.107:0/1806361492 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a0a280 0x563f72a120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.426+0000 7fdad8b68740 1 -- 172.21.15.107:0/1806361492 >> 172.21.15.107:0/1806361492 conn(0x563f72a065a0 msgr2=0x563f72a057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:25.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.426+0000 7fdaaeffd640 0 client.27806 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:25.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.427+0000 7fdad8b68740 1 -- 172.21.15.107:0/1806361492 shutdown_connections 2024-05-09T08:45:25.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.427+0000 7fdad8b68740 1 -- 172.21.15.107:0/1806361492 wait complete. 2024-05-09T08:45:25.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.427+0000 7fdad8b68740 20 client.27806 trim_cache size 0 max 16384 2024-05-09T08:45:25.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.476+0000 7fdad8b68740 1 Processor -- start 2024-05-09T08:45:25.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.476+0000 7fdad8b68740 1 -- start start 2024-05-09T08:45:25.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.476+0000 7fdad8b68740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a0a280 0x563f72a210a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.476+0000 7fdad8b68740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a0aba0 0x563f72a11230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.476+0000 7fdad8b68740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a15270 0x563f72a12950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.476+0000 7fdad8b68740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc012950 con 0x563f72a0a280 2024-05-09T08:45:25.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.476+0000 7fdad8b68740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc010220 con 0x563f72a15270 2024-05-09T08:45:25.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.476+0000 7fdad8b68740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc00e070 con 0x563f72a0aba0 2024-05-09T08:45:25.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.476+0000 7fdacffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a15270 0x563f72a12950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:25.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.476+0000 7fdacffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a15270 0x563f72a12950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44188/0 (socket says 172.21.15.107:44188) 2024-05-09T08:45:25.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.476+0000 7fdacffff640 1 -- 172.21.15.107:0/1425844937 learned_addr learned my addr 172.21.15.107:0/1425844937 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:25.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.477+0000 7fdacffff640 1 -- 172.21.15.107:0/1425844937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a0aba0 msgr2=0x563f72a11230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:45:25.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.477+0000 7fdacffff640 1 --2- 172.21.15.107:0/1425844937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a0aba0 0x563f72a11230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.477+0000 7fdacffff640 1 -- 172.21.15.107:0/1425844937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a0a280 msgr2=0x563f72a210a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:45:25.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.477+0000 7fdacffff640 1 --2- 172.21.15.107:0/1425844937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a0a280 0x563f72a210a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.477+0000 7fdacffff640 1 -- 172.21.15.107:0/1425844937 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f72a063d0 con 0x563f72a15270 2024-05-09T08:45:25.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.477+0000 7fdacffff640 1 --2- 172.21.15.107:0/1425844937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a15270 0x563f72a12950 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdac409b9b0 tx=0x7fdac40a3b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3195e15687a31caa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:25.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.477+0000 7fdab57fa640 1 -- 172.21.15.107:0/1425844937 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac409a7f0 con 0x563f72a15270 2024-05-09T08:45:25.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.477+0000 7fdab57fa640 1 -- 172.21.15.107:0/1425844937 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdac40b4060 con 0x563f72a15270 2024-05-09T08:45:25.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.477+0000 7fdab57fa640 1 -- 172.21.15.107:0/1425844937 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac4096e50 con 0x563f72a15270 2024-05-09T08:45:25.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.478+0000 7fdad8b68740 1 -- 172.21.15.107:0/1425844937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a15270 msgr2=0x563f72a12950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:25.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.478+0000 7fdad8b68740 1 --2- 172.21.15.107:0/1425844937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a15270 0x563f72a12950 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdac409b9b0 tx=0x7fdac40a3b40 comp rx=0 tx=0).stop 2024-05-09T08:45:25.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.479+0000 7fdad8b68740 1 -- 172.21.15.107:0/1425844937 shutdown_connections 2024-05-09T08:45:25.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.479+0000 7fdad8b68740 1 --2- 172.21.15.107:0/1425844937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a15270 0x563f72a12950 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.479+0000 7fdad8b68740 1 --2- 172.21.15.107:0/1425844937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a0aba0 0x563f72a11230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.479+0000 7fdad8b68740 1 --2- 172.21.15.107:0/1425844937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a0a280 0x563f72a210a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.479+0000 7fdad8b68740 1 -- 172.21.15.107:0/1425844937 >> 172.21.15.107:0/1425844937 conn(0x563f72a065a0 msgr2=0x563f72a0b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:25.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.480+0000 7fdad8b68740 1 -- 172.21.15.107:0/1425844937 shutdown_connections 2024-05-09T08:45:25.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.480+0000 7fdad8b68740 1 -- 172.21.15.107:0/1425844937 wait complete. 2024-05-09T08:45:25.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.480+0000 7fdad8b68740 1 -- start start 2024-05-09T08:45:25.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.480+0000 7fdad8b68740 1 Processor -- start 2024-05-09T08:45:25.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.481+0000 7fdad8b68740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a0a280 0x563f72a1d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.481+0000 7fdad8b68740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a0aba0 0x563f72a1daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.481+0000 7fdad8b68740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a15270 0x563f72a0e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.481+0000 7fdad8b68740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc00e6e0 con 0x563f72a15270 2024-05-09T08:45:25.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.481+0000 7fdad8b68740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc00fb50 con 0x563f72a0aba0 2024-05-09T08:45:25.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.481+0000 7fdad8b68740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc00edb0 con 0x563f72a0a280 2024-05-09T08:45:25.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.481+0000 7fdad8b68740 10 client.0 _ll_register_callbacks cb 0x7fdac40973a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:25.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.481+0000 7fdacffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a15270 0x563f72a0e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:25.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.481+0000 7fdacffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a15270 0x563f72a0e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56480/0 (socket says 172.21.15.107:56480) 2024-05-09T08:45:25.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.481+0000 7fdacffff640 1 -- 172.21.15.107:0/3146743772 learned_addr learned my addr 172.21.15.107:0/3146743772 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:25.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.482+0000 7fdaceffd640 1 --2- 172.21.15.107:0/3146743772 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a0aba0 0x563f72a1daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:25.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.482+0000 7fdacf7fe640 1 --2- 172.21.15.107:0/3146743772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a0a280 0x563f72a1d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:25.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.482+0000 7fdacffff640 1 -- 172.21.15.107:0/3146743772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a0a280 msgr2=0x563f72a1d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.482+0000 7fdacffff640 1 --2- 172.21.15.107:0/3146743772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a0a280 0x563f72a1d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.482+0000 7fdacffff640 1 -- 172.21.15.107:0/3146743772 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a0aba0 msgr2=0x563f72a1daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.482+0000 7fdacffff640 1 --2- 172.21.15.107:0/3146743772 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a0aba0 0x563f72a1daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.482+0000 7fdacffff640 1 -- 172.21.15.107:0/3146743772 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f72a0ecc0 con 0x563f72a15270 2024-05-09T08:45:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.482+0000 7fdacffff640 1 --2- 172.21.15.107:0/3146743772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a15270 0x563f72a0e4c0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fdac40a3c60 tx=0x7fdac400bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5006e4f6aaf71 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.482+0000 7fdaaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:45:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.483+0000 7fdaaeffd640 1 -- 172.21.15.107:0/3146743772 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac409a550 con 0x563f72a15270 2024-05-09T08:45:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.483+0000 7fdaaeffd640 1 -- 172.21.15.107:0/3146743772 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7fdac40bb110 con 0x563f72a15270 2024-05-09T08:45:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.483+0000 7fdad8b68740 1 -- 172.21.15.107:0/3146743772 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdac40b4060 con 0x563f72a15270 2024-05-09T08:45:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.483+0000 7fdaaeffd640 1 -- 172.21.15.107:0/3146743772 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdac40af120 con 0x563f72a15270 2024-05-09T08:45:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.483+0000 7fdad8b68740 10 client.18861 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:25.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.483+0000 7fdad8b68740 1 -- 172.21.15.107:0/3146743772 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563f72a8bb40 con 0x563f72a15270 2024-05-09T08:45:25.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.484+0000 7fdaaeffd640 1 -- 172.21.15.107:0/3146743772 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdac40b4060 con 0x563f72a15270 2024-05-09T08:45:25.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.484+0000 7fdaaeffd640 1 -- 172.21.15.107:0/3146743772 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fdac409a160 con 0x563f72a15270 2024-05-09T08:45:25.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.484+0000 7fdad8b68740 10 client.18861 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:25.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.484+0000 7fdad8b68740 10 client.18861 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:45:25.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.484+0000 7fdad8b68740 20 client.18861 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:25.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.484+0000 7fdad8b68740 1 --2- 172.21.15.107:0/3146743772 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563f72a8bfc0 0x563f72a8e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.484+0000 7fdad8b68740 4 client.18861 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:45:25.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.484+0000 7fdad8b68740 1 -- 172.21.15.107:0/3146743772 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563f72977780 con 0x563f72a8bfc0 2024-05-09T08:45:25.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.489+0000 7fdacf7fe640 1 --2- 172.21.15.107:0/3146743772 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563f72a8bfc0 0x563f72a8e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:25.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.496+0000 7fdacf7fe640 1 --2- 172.21.15.107:0/3146743772 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563f72a8bfc0 0x563f72a8e3b0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c033c7ac0e28115c server_cookie=8694394cc02d0b33 in_seq=0 out_seq=0 2024-05-09T08:45:25.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.496+0000 7fdaaeffd640 10 client.18861 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:25.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.503+0000 7fdaaeffd640 1 -- 172.21.15.107:0/3146743772 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563f72977780 con 0x563f72a8bfc0 2024-05-09T08:45:25.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.503+0000 7fdaaeffd640 10 client.18861 handle_command_reply: tid=0 2024-05-09T08:45:25.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.503+0000 7fdad8b68740 1 client.18861 shutdown 2024-05-09T08:45:25.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.510+0000 7fdad8b68740 1 -- 172.21.15.107:0/3146743772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a15270 msgr2=0x563f72a0e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:25.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.510+0000 7fdad8b68740 1 --2- 172.21.15.107:0/3146743772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a15270 0x563f72a0e4c0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fdac40a3c60 tx=0x7fdac400bf60 comp rx=0 tx=0).stop 2024-05-09T08:45:25.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.510+0000 7fdad8b68740 1 -- 172.21.15.107:0/3146743772 shutdown_connections 2024-05-09T08:45:25.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.510+0000 7fdad8b68740 1 --2- 172.21.15.107:0/3146743772 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563f72a8bfc0 0x563f72a8e3b0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.510+0000 7fdad8b68740 1 --2- 172.21.15.107:0/3146743772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563f72a15270 0x563f72a0e4c0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.510+0000 7fdad8b68740 1 --2- 172.21.15.107:0/3146743772 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563f72a0aba0 0x563f72a1daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.510+0000 7fdad8b68740 1 --2- 172.21.15.107:0/3146743772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563f72a0a280 0x563f72a1d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.510+0000 7fdad8b68740 1 -- 172.21.15.107:0/3146743772 >> 172.21.15.107:0/3146743772 conn(0x563f72a065a0 msgr2=0x563f72a0c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:25.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.510+0000 7fdaaeffd640 0 client.18861 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:25.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.515+0000 7fdad8b68740 1 -- 172.21.15.107:0/3146743772 shutdown_connections 2024-05-09T08:45:25.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.519+0000 7fdad8b68740 1 -- 172.21.15.107:0/3146743772 wait complete. 2024-05-09T08:45:25.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.519+0000 7fdad8b68740 20 client.18861 trim_cache size 0 max 16384 2024-05-09T08:45:25.523 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:45:25.523 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:45:25.523 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:45:25.523 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:45:25.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.523+0000 7fdad5d84640 1 -- 172.21.15.107:0/1733743438 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdab80a2de0 msgr2=0x7fdab80a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:25.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.523+0000 7fdad5d84640 1 --2- 172.21.15.107:0/1733743438 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdab80a2de0 0x7fdab80a52a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdac8006010 tx=0x7fdac8009990 comp rx=0 tx=0).stop 2024-05-09T08:45:25.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.523+0000 7fdad5d84640 1 -- 172.21.15.107:0/1733743438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdad011b010 msgr2=0x7fdad01ab710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:25.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.523+0000 7fdad5d84640 1 --2- 172.21.15.107:0/1733743438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdad011b010 0x7fdad01ab710 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fdac400b3f0 tx=0x7fdac400b8b0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.526+0000 7fdad5d84640 1 -- 172.21.15.107:0/1733743438 shutdown_connections 2024-05-09T08:45:25.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.526+0000 7fdad5d84640 1 --2- 172.21.15.107:0/1733743438 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdab80a2de0 0x7fdab80a52a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.526+0000 7fdad5d84640 1 --2- 172.21.15.107:0/1733743438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdad011b010 0x7fdad01ab710 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.526+0000 7fdad5d84640 1 --2- 172.21.15.107:0/1733743438 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdad0110080 0x7fdad01a7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.526+0000 7fdad5d84640 1 --2- 172.21.15.107:0/1733743438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdad010e0b0 0x7fdad01a6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.526+0000 7fdad5d84640 1 -- 172.21.15.107:0/1733743438 >> 172.21.15.107:0/1733743438 conn(0x7fdad006c940 msgr2=0x7fdad011bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:25.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.528+0000 7fdad5d84640 1 -- 172.21.15.107:0/1733743438 shutdown_connections 2024-05-09T08:45:25.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.528+0000 7fdad5d84640 1 -- 172.21.15.107:0/1733743438 wait complete. 2024-05-09T08:45:25.550 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:45:25.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:45:25.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.851+0000 7efccf1f9640 1 -- 172.21.15.107:0/2921465997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcc810ce60 msgr2=0x7efcc810d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:25.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.851+0000 7efccf1f9640 1 --2- 172.21.15.107:0/2921465997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcc810ce60 0x7efcc810d240 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7efcbc009860 tx=0x7efcbc02a340 comp rx=0 tx=0).stop 2024-05-09T08:45:25.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.851+0000 7efccf1f9640 1 -- 172.21.15.107:0/2921465997 shutdown_connections 2024-05-09T08:45:25.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.852+0000 7efccf1f9640 1 --2- 172.21.15.107:0/2921465997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efcc8109030 0x7efcc8114cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.852+0000 7efccf1f9640 1 --2- 172.21.15.107:0/2921465997 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efcc8108670 0x7efcc8108af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.852+0000 7efccf1f9640 1 --2- 172.21.15.107:0/2921465997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcc810ce60 0x7efcc810d240 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.852+0000 7efccf1f9640 1 -- 172.21.15.107:0/2921465997 >> 172.21.15.107:0/2921465997 conn(0x7efcc8077430 msgr2=0x7efcc8079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:25.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.852+0000 7efccf1f9640 1 -- 172.21.15.107:0/2921465997 shutdown_connections 2024-05-09T08:45:25.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.852+0000 7efccf1f9640 1 -- 172.21.15.107:0/2921465997 wait complete. 2024-05-09T08:45:25.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.853+0000 7efccf1f9640 1 Processor -- start 2024-05-09T08:45:25.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.853+0000 7efccf1f9640 1 -- start start 2024-05-09T08:45:25.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.853+0000 7efccf1f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcc8108670 0x7efcc806dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.853+0000 7efccf1f9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efcc8109030 0x7efcc806e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.854+0000 7efccf1f9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efcc810ce60 0x7efcc81b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.854+0000 7efcc7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efcc8109030 0x7efcc806e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:25.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.854+0000 7efcc7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efcc8109030 0x7efcc806e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44218/0 (socket says 172.21.15.107:44218) 2024-05-09T08:45:25.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.854+0000 7efcc7fff640 1 -- 172.21.15.107:0/3459465481 learned_addr learned my addr 172.21.15.107:0/3459465481 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:25.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.854+0000 7efccf1f9640 1 -- 172.21.15.107:0/3459465481 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efcc806c5b0 con 0x7efcc8108670 2024-05-09T08:45:25.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.854+0000 7efccf1f9640 1 -- 172.21.15.107:0/3459465481 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7efcc806c430 con 0x7efcc8109030 2024-05-09T08:45:25.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.854+0000 7efccf1f9640 1 -- 172.21.15.107:0/3459465481 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efcc806c730 con 0x7efcc810ce60 2024-05-09T08:45:25.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.854+0000 7efccd796640 1 --2- 172.21.15.107:0/3459465481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efcc810ce60 0x7efcc81b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:25.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.854+0000 7efcccf95640 1 --2- 172.21.15.107:0/3459465481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcc8108670 0x7efcc806dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:25.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.854+0000 7efcc7fff640 1 -- 172.21.15.107:0/3459465481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efcc810ce60 msgr2=0x7efcc81b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:25.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.854+0000 7efcc7fff640 1 --2- 172.21.15.107:0/3459465481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efcc810ce60 0x7efcc81b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.854+0000 7efcc7fff640 1 -- 172.21.15.107:0/3459465481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcc8108670 msgr2=0x7efcc806dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:25.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.854+0000 7efcc7fff640 1 --2- 172.21.15.107:0/3459465481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcc8108670 0x7efcc806dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:25.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.854+0000 7efcc7fff640 1 -- 172.21.15.107:0/3459465481 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc81b3020 con 0x7efcc8109030 2024-05-09T08:45:25.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.855+0000 7efcc7fff640 1 --2- 172.21.15.107:0/3459465481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efcc8109030 0x7efcc806e2c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7efcb800e940 tx=0x7efcb800ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:25.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.855+0000 7efcc5ffb640 1 -- 172.21.15.107:0/3459465481 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb800cd80 con 0x7efcc8109030 2024-05-09T08:45:25.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.855+0000 7efcc5ffb640 1 -- 172.21.15.107:0/3459465481 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcb80108e0 con 0x7efcc8109030 2024-05-09T08:45:25.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.855+0000 7efcc5ffb640 1 -- 172.21.15.107:0/3459465481 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb8018860 con 0x7efcc8109030 2024-05-09T08:45:25.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.855+0000 7efccf1f9640 1 -- 172.21.15.107:0/3459465481 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcc81b32b0 con 0x7efcc8109030 2024-05-09T08:45:25.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.855+0000 7efccf1f9640 1 -- 172.21.15.107:0/3459465481 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcc81b37b0 con 0x7efcc8109030 2024-05-09T08:45:25.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.860+0000 7efcc5ffb640 1 -- 172.21.15.107:0/3459465481 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7efcb8010a90 con 0x7efcc8109030 2024-05-09T08:45:25.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.861+0000 7efcc5ffb640 1 --2- 172.21.15.107:0/3459465481 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7efcb00a2e40 0x7efcb00a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:25.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.861+0000 7efcc5ffb640 1 -- 172.21.15.107:0/3459465481 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7efcb80cfd00 con 0x7efcc8109030 2024-05-09T08:45:25.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.861+0000 7efcccf95640 1 --2- 172.21.15.107:0/3459465481 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7efcb00a2e40 0x7efcb00a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:25.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.862+0000 7efccf1f9640 1 -- 172.21.15.107:0/3459465481 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcc8109680 con 0x7efcc8109030 2024-05-09T08:45:25.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.862+0000 7efcccf95640 1 --2- 172.21.15.107:0/3459465481 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7efcb00a2e40 0x7efcb00a5300 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7efcbc000c00 tx=0x7efcbc009490 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:25.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:25.868+0000 7efcc5ffb640 1 -- 172.21.15.107:0/3459465481 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efcb809c4c0 con 0x7efcc8109030 2024-05-09T08:45:26.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.347+0000 7efccf1f9640 1 -- 172.21.15.107:0/3459465481 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efcc81103c0 con 0x7efcc8109030 2024-05-09T08:45:26.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.349+0000 7efcc5ffb640 1 -- 172.21.15.107:0/3459465481 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11887 (secure 0 0 0) 0x7efcb809bdc0 con 0x7efcc8109030 2024-05-09T08:45:26.354 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:45:26.354 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:37:10.930317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:45:26.354 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2024-05-09T08:45:26.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.361+0000 7efccf1f9640 1 -- 172.21.15.107:0/3459465481 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7efcb00a2e40 msgr2=0x7efcb00a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.361+0000 7efccf1f9640 1 --2- 172.21.15.107:0/3459465481 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7efcb00a2e40 0x7efcb00a5300 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7efcbc000c00 tx=0x7efcbc009490 comp rx=0 tx=0).stop 2024-05-09T08:45:26.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.361+0000 7efccf1f9640 1 -- 172.21.15.107:0/3459465481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efcc8109030 msgr2=0x7efcc806e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.361+0000 7efccf1f9640 1 --2- 172.21.15.107:0/3459465481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efcc8109030 0x7efcc806e2c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7efcb800e940 tx=0x7efcb800ee00 comp rx=0 tx=0).stop 2024-05-09T08:45:26.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.364+0000 7efccf1f9640 1 -- 172.21.15.107:0/3459465481 shutdown_connections 2024-05-09T08:45:26.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.364+0000 7efccf1f9640 1 --2- 172.21.15.107:0/3459465481 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7efcb00a2e40 0x7efcb00a5300 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.364+0000 7efccf1f9640 1 --2- 172.21.15.107:0/3459465481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efcc810ce60 0x7efcc81b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.364+0000 7efccf1f9640 1 --2- 172.21.15.107:0/3459465481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efcc8109030 0x7efcc806e2c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.364+0000 7efccf1f9640 1 --2- 172.21.15.107:0/3459465481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcc8108670 0x7efcc806dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.364+0000 7efccf1f9640 1 -- 172.21.15.107:0/3459465481 >> 172.21.15.107:0/3459465481 conn(0x7efcc8077430 msgr2=0x7efcc810aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:26.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.369+0000 7efccf1f9640 1 -- 172.21.15.107:0/3459465481 shutdown_connections 2024-05-09T08:45:26.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.369+0000 7efccf1f9640 1 -- 172.21.15.107:0/3459465481 wait complete. 2024-05-09T08:45:26.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:45:26.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.565+0000 7f55437a5640 1 -- 172.21.15.107:0/1430692704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f553c100710 msgr2=0x7f553c100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.565+0000 7f55437a5640 1 --2- 172.21.15.107:0/1430692704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f553c100710 0x7f553c100b90 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f5538009930 tx=0x7f553802a420 comp rx=0 tx=0).stop 2024-05-09T08:45:26.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.565+0000 7f55437a5640 1 -- 172.21.15.107:0/1430692704 shutdown_connections 2024-05-09T08:45:26.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.565+0000 7f55437a5640 1 --2- 172.21.15.107:0/1430692704 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f553c1010d0 0x7f553c10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.565+0000 7f55437a5640 1 --2- 172.21.15.107:0/1430692704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f553c100710 0x7f553c100b90 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.565+0000 7f55437a5640 1 --2- 172.21.15.107:0/1430692704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f553c106720 0x7f553c106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.565+0000 7f55437a5640 1 -- 172.21.15.107:0/1430692704 >> 172.21.15.107:0/1430692704 conn(0x7f553c077790 msgr2=0x7f553c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:26.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.566+0000 7f55437a5640 1 -- 172.21.15.107:0/1430692704 shutdown_connections 2024-05-09T08:45:26.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.566+0000 7f55437a5640 1 -- 172.21.15.107:0/1430692704 wait complete. 2024-05-09T08:45:26.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.566+0000 7f55437a5640 1 Processor -- start 2024-05-09T08:45:26.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.566+0000 7f55437a5640 1 -- start start 2024-05-09T08:45:26.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.566+0000 7f55437a5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f553c100710 0x7f553c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.566+0000 7f55437a5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f553c1010d0 0x7f553c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.566+0000 7f55437a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f553c106720 0x7f553c19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.566+0000 7f55437a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f553c111ae0 con 0x7f553c106720 2024-05-09T08:45:26.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.566+0000 7f55437a5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f553c111960 con 0x7f553c100710 2024-05-09T08:45:26.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.566+0000 7f55437a5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f553c111c60 con 0x7f553c1010d0 2024-05-09T08:45:26.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.566+0000 7f5541d42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f553c106720 0x7f553c19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.566+0000 7f5541d42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f553c106720 0x7f553c19c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56518/0 (socket says 172.21.15.107:56518) 2024-05-09T08:45:26.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.566+0000 7f5541d42640 1 -- 172.21.15.107:0/502346829 learned_addr learned my addr 172.21.15.107:0/502346829 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:26.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.566+0000 7f5540d40640 1 --2- 172.21.15.107:0/502346829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f553c1010d0 0x7f553c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.566+0000 7f5541541640 1 --2- 172.21.15.107:0/502346829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f553c100710 0x7f553c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.567+0000 7f5541d42640 1 -- 172.21.15.107:0/502346829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f553c1010d0 msgr2=0x7f553c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.567+0000 7f5541d42640 1 --2- 172.21.15.107:0/502346829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f553c1010d0 0x7f553c198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.567+0000 7f5541d42640 1 -- 172.21.15.107:0/502346829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f553c100710 msgr2=0x7f553c198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.567+0000 7f5541d42640 1 --2- 172.21.15.107:0/502346829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f553c100710 0x7f553c198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.567+0000 7f5541d42640 1 -- 172.21.15.107:0/502346829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f553c19cff0 con 0x7f553c106720 2024-05-09T08:45:26.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.567+0000 7f5541d42640 1 --2- 172.21.15.107:0/502346829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f553c106720 0x7f553c19c8c0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f553400eaa0 tx=0x7f553400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:26.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.567+0000 7f552e7fc640 1 -- 172.21.15.107:0/502346829 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553400cdb0 con 0x7f553c106720 2024-05-09T08:45:26.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.567+0000 7f552e7fc640 1 -- 172.21.15.107:0/502346829 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5534005020 con 0x7f553c106720 2024-05-09T08:45:26.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.567+0000 7f55437a5640 1 -- 172.21.15.107:0/502346829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f553c19d2e0 con 0x7f553c106720 2024-05-09T08:45:26.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.567+0000 7f552e7fc640 1 -- 172.21.15.107:0/502346829 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55340116d0 con 0x7f553c106720 2024-05-09T08:45:26.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.567+0000 7f55437a5640 1 -- 172.21.15.107:0/502346829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f553c1a4b70 con 0x7f553c106720 2024-05-09T08:45:26.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.570+0000 7f552e7fc640 1 -- 172.21.15.107:0/502346829 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f55340051d0 con 0x7f553c106720 2024-05-09T08:45:26.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.570+0000 7f554658d740 1 Processor -- start 2024-05-09T08:45:26.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.570+0000 7f554658d740 1 -- start start 2024-05-09T08:45:26.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.570+0000 7f554658d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eee9330 0x56517eee9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.570+0000 7f554658d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9c50 0x56517eef3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.570+0000 7f554658d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eef4320 0x56517eef6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.570+0000 7f554658d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56517ee4ec40 con 0x56517eef4320 2024-05-09T08:45:26.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.570+0000 7f554658d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56517edf6580 con 0x56517eee9330 2024-05-09T08:45:26.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.570+0000 7f554658d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56517ee4ac00 con 0x56517eee9c50 2024-05-09T08:45:26.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f5540d40640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eee9330 0x56517eee9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f5540d40640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eee9330 0x56517eee9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44258/0 (socket says 172.21.15.107:44258) 2024-05-09T08:45:26.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f5540d40640 1 -- 172.21.15.107:0/1648232823 learned_addr learned my addr 172.21.15.107:0/1648232823 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:26.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f5541541640 1 --2- 172.21.15.107:0/1648232823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9c50 0x56517eef3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f552e7fc640 1 --2- 172.21.15.107:0/502346829 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f55280a2d00 0x7f55280a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f5540d40640 1 --2- 172.21.15.107:0/1648232823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eef4320 0x56517eef6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f5541d42640 1 --2- 172.21.15.107:0/502346829 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f55280a2d00 0x7f55280a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f5541541640 1 -- 172.21.15.107:0/1648232823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eee9330 msgr2=0x56517eee9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:26.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f5541541640 1 --2- 172.21.15.107:0/1648232823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eee9330 0x56517eee9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f5541541640 1 -- 172.21.15.107:0/1648232823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eef4320 msgr2=0x56517eef6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:26.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f5541541640 1 --2- 172.21.15.107:0/1648232823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eef4320 0x56517eef6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f5541541640 1 -- 172.21.15.107:0/1648232823 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56517eb623f0 con 0x56517eee9c50 2024-05-09T08:45:26.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f5540d40640 1 --2- 172.21.15.107:0/1648232823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eee9330 0x56517eee9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:45:26.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f552e7fc640 1 -- 172.21.15.107:0/502346829 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7f55340cf970 con 0x7f553c106720 2024-05-09T08:45:26.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f5541541640 1 --2- 172.21.15.107:0/1648232823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9c50 0x56517eef3de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f553000bb70 tx=0x7f55300107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe43369964622bd1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:26.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f5541d42640 1 --2- 172.21.15.107:0/502346829 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f55280a2d00 0x7f55280a51c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f55340ccd70 tx=0x7f553401eb40 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:26.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f55277fe640 1 -- 172.21.15.107:0/1648232823 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5530004030 con 0x56517eee9c50 2024-05-09T08:45:26.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f55277fe640 1 -- 172.21.15.107:0/1648232823 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55300035f0 con 0x56517eee9c50 2024-05-09T08:45:26.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.571+0000 7f55277fe640 1 -- 172.21.15.107:0/1648232823 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55300116a0 con 0x56517eee9c50 2024-05-09T08:45:26.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.572+0000 7f554658d740 1 -- 172.21.15.107:0/1648232823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9c50 msgr2=0x56517eef3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.572+0000 7f554658d740 1 --2- 172.21.15.107:0/1648232823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9c50 0x56517eef3de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f553000bb70 tx=0x7f55300107a0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.572+0000 7f554658d740 1 -- 172.21.15.107:0/1648232823 shutdown_connections 2024-05-09T08:45:26.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.572+0000 7f554658d740 1 --2- 172.21.15.107:0/1648232823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eef4320 0x56517eef6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.572+0000 7f554658d740 1 --2- 172.21.15.107:0/1648232823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9c50 0x56517eef3de0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.572+0000 7f554658d740 1 --2- 172.21.15.107:0/1648232823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eee9330 0x56517eee9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.572+0000 7f554658d740 1 -- 172.21.15.107:0/1648232823 >> 172.21.15.107:0/1648232823 conn(0x56517eee5650 msgr2=0x56517eee5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:26.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.572+0000 7f554658d740 1 -- 172.21.15.107:0/1648232823 shutdown_connections 2024-05-09T08:45:26.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.572+0000 7f554658d740 1 -- 172.21.15.107:0/1648232823 wait complete. 2024-05-09T08:45:26.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.572+0000 7f554658d740 1 -- start start 2024-05-09T08:45:26.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.572+0000 7f554658d740 1 Processor -- start 2024-05-09T08:45:26.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f554658d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9330 0x56517eef11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f554658d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9c50 0x56517eef1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f554658d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eef4320 0x56517eef1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f554658d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56517eef7110 con 0x56517eee9c50 2024-05-09T08:45:26.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f554658d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56517ee09c40 con 0x56517eef4320 2024-05-09T08:45:26.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f554658d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56517eef7290 con 0x56517eee9330 2024-05-09T08:45:26.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f554658d740 10 client.0 _ll_register_callbacks cb 0x56517eb61be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:26.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f5540d40640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9330 0x56517eef11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f5541541640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9c50 0x56517eef1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f5540d40640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9330 0x56517eef11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:50426/0 (socket says 172.21.15.107:50426) 2024-05-09T08:45:26.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f5540d40640 1 -- 172.21.15.107:0/2412631700 learned_addr learned my addr 172.21.15.107:0/2412631700 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:26.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f5540d40640 1 --2- 172.21.15.107:0/2412631700 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eef4320 0x56517eef1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f5541541640 1 -- 172.21.15.107:0/2412631700 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9330 msgr2=0x56517eef11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:26.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f5541541640 1 --2- 172.21.15.107:0/2412631700 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9330 0x56517eef11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f5541541640 1 -- 172.21.15.107:0/2412631700 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eef4320 msgr2=0x56517eef1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:26.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f5541541640 1 --2- 172.21.15.107:0/2412631700 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eef4320 0x56517eef1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f5541541640 1 -- 172.21.15.107:0/2412631700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56517ec2c7d0 con 0x56517eee9c50 2024-05-09T08:45:26.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f5541541640 1 --2- 172.21.15.107:0/2412631700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9c50 0x56517eef1730 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f553000b620 tx=0x7f553000bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846c63e53e756292 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:26.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.573+0000 7f5524ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:45:26.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.574+0000 7f5524ff9640 1 -- 172.21.15.107:0/2412631700 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553002a060 con 0x56517eee9c50 2024-05-09T08:45:26.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.574+0000 7f5524ff9640 1 -- 172.21.15.107:0/2412631700 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7f55300039b0 con 0x56517eee9c50 2024-05-09T08:45:26.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.574+0000 7f554658d740 1 -- 172.21.15.107:0/2412631700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55300035f0 con 0x56517eee9c50 2024-05-09T08:45:26.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.574+0000 7f5524ff9640 1 -- 172.21.15.107:0/2412631700 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5530031050 con 0x56517eee9c50 2024-05-09T08:45:26.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.574+0000 7f554658d740 10 client.18879 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:26.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.574+0000 7f554658d740 1 -- 172.21.15.107:0/2412631700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56517ef032a0 con 0x56517eee9c50 2024-05-09T08:45:26.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.574+0000 7f5524ff9640 1 -- 172.21.15.107:0/2412631700 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55300035f0 con 0x56517eee9c50 2024-05-09T08:45:26.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.574+0000 7f5524ff9640 1 -- 172.21.15.107:0/2412631700 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5530005420 con 0x56517eee9c50 2024-05-09T08:45:26.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.574+0000 7f554658d740 10 client.18879 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:26.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.574+0000 7f554658d740 10 client.18879 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:45:26.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.574+0000 7f554658d740 20 client.18879 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:26.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.574+0000 7f554658d740 1 --2- 172.21.15.107:0/2412631700 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x56517ef04700 0x56517ef06af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.574+0000 7f554658d740 4 client.18879 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:45:26.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.574+0000 7f554658d740 1 -- 172.21.15.107:0/2412631700 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56517ee5c370 con 0x56517ef04700 2024-05-09T08:45:26.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.574+0000 7f5540d40640 1 --2- 172.21.15.107:0/2412631700 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x56517ef04700 0x56517ef06af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.575+0000 7f5524ff9640 10 client.18879 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:45:26.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.575+0000 7f5540d40640 1 --2- 172.21.15.107:0/2412631700 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x56517ef04700 0x56517ef06af0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ed288683ca8ff1e9 server_cookie=5cfacad4b64b068b in_seq=0 out_seq=0 2024-05-09T08:45:26.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.575+0000 7f5524ff9640 1 -- 172.21.15.107:0/2412631700 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56517ee5c370 con 0x56517ef04700 2024-05-09T08:45:26.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.575+0000 7f5524ff9640 10 client.18879 handle_command_reply: tid=0 2024-05-09T08:45:26.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.575+0000 7f554658d740 1 client.18879 shutdown 2024-05-09T08:45:26.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.575+0000 7f554658d740 1 -- 172.21.15.107:0/2412631700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9c50 msgr2=0x56517eef1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.575+0000 7f554658d740 1 --2- 172.21.15.107:0/2412631700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9c50 0x56517eef1730 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f553000b620 tx=0x7f553000bf50 comp rx=0 tx=0).stop 2024-05-09T08:45:26.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.576+0000 7f554658d740 1 -- 172.21.15.107:0/2412631700 shutdown_connections 2024-05-09T08:45:26.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.576+0000 7f554658d740 1 --2- 172.21.15.107:0/2412631700 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x56517ef04700 0x56517ef06af0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.576+0000 7f554658d740 1 --2- 172.21.15.107:0/2412631700 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eef4320 0x56517eef1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.576+0000 7f554658d740 1 --2- 172.21.15.107:0/2412631700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9c50 0x56517eef1730 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.576+0000 7f554658d740 1 --2- 172.21.15.107:0/2412631700 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9330 0x56517eef11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.576+0000 7f5524ff9640 0 client.18879 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:45:26.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.576+0000 7f554658d740 1 -- 172.21.15.107:0/2412631700 >> 172.21.15.107:0/2412631700 conn(0x56517eee5650 msgr2=0x56517eeeb450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:26.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.576+0000 7f554658d740 1 -- 172.21.15.107:0/2412631700 shutdown_connections 2024-05-09T08:45:26.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.576+0000 7f554658d740 1 -- 172.21.15.107:0/2412631700 wait complete. 2024-05-09T08:45:26.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.576+0000 7f554658d740 20 client.18879 trim_cache size 0 max 16384 2024-05-09T08:45:26.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.589+0000 7f554658d740 1 Processor -- start 2024-05-09T08:45:26.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.589+0000 7f554658d740 1 -- start start 2024-05-09T08:45:26.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.589+0000 7f554658d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9330 0x56517ef00030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.589+0000 7f554658d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9c50 0x56517eef1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.589+0000 7f554658d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eef4320 0x56517eeeb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.589+0000 7f554658d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f553c1b1670 con 0x56517eee9330 2024-05-09T08:45:26.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.589+0000 7f554658d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f553c1aef40 con 0x56517eef4320 2024-05-09T08:45:26.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.589+0000 7f554658d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f553c1acd90 con 0x56517eee9c50 2024-05-09T08:45:26.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.589+0000 7f5540d40640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9330 0x56517ef00030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.589+0000 7f5540d40640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9330 0x56517ef00030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56556/0 (socket says 172.21.15.107:56556) 2024-05-09T08:45:26.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.589+0000 7f5540d40640 1 -- 172.21.15.107:0/1465380380 learned_addr learned my addr 172.21.15.107:0/1465380380 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:26.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.589+0000 7f5541541640 1 --2- 172.21.15.107:0/1465380380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9c50 0x56517eef1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.590+0000 7f5540d40640 1 -- 172.21.15.107:0/1465380380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9c50 msgr2=0x56517eef1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:26.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.590+0000 7f5540d40640 1 --2- 172.21.15.107:0/1465380380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9c50 0x56517eef1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.590+0000 7f5540d40640 1 -- 172.21.15.107:0/1465380380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eef4320 msgr2=0x56517eeeb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:26.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.590+0000 7f5540d40640 1 --2- 172.21.15.107:0/1465380380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eef4320 0x56517eeeb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.590+0000 7f5540d40640 1 -- 172.21.15.107:0/1465380380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56517eee9a80 con 0x56517eee9330 2024-05-09T08:45:26.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.590+0000 7f5540d40640 1 --2- 172.21.15.107:0/1465380380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9330 0x56517ef00030 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f5538009930 tx=0x7f5538032b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=215d55b767d8116b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:26.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.590+0000 7f5526ffd640 1 -- 172.21.15.107:0/1465380380 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5538035040 con 0x56517eee9330 2024-05-09T08:45:26.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.590+0000 7f5526ffd640 1 -- 172.21.15.107:0/1465380380 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5538031dd0 con 0x56517eee9330 2024-05-09T08:45:26.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.590+0000 7f5526ffd640 1 -- 172.21.15.107:0/1465380380 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5538005310 con 0x56517eee9330 2024-05-09T08:45:26.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.591+0000 7f554658d740 1 -- 172.21.15.107:0/1465380380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9330 msgr2=0x56517ef00030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.591+0000 7f554658d740 1 --2- 172.21.15.107:0/1465380380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9330 0x56517ef00030 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f5538009930 tx=0x7f5538032b00 comp rx=0 tx=0).stop 2024-05-09T08:45:26.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.591+0000 7f554658d740 1 -- 172.21.15.107:0/1465380380 shutdown_connections 2024-05-09T08:45:26.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.591+0000 7f554658d740 1 --2- 172.21.15.107:0/1465380380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eef4320 0x56517eeeb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.591+0000 7f554658d740 1 --2- 172.21.15.107:0/1465380380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eee9c50 0x56517eef1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.591+0000 7f554658d740 1 --2- 172.21.15.107:0/1465380380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9330 0x56517ef00030 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.591+0000 7f554658d740 1 -- 172.21.15.107:0/1465380380 >> 172.21.15.107:0/1465380380 conn(0x56517eee5650 msgr2=0x56517eeeab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:26.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.591+0000 7f554658d740 1 -- 172.21.15.107:0/1465380380 shutdown_connections 2024-05-09T08:45:26.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.591+0000 7f554658d740 1 -- 172.21.15.107:0/1465380380 wait complete. 2024-05-09T08:45:26.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.591+0000 7f554658d740 1 -- start start 2024-05-09T08:45:26.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.591+0000 7f554658d740 1 Processor -- start 2024-05-09T08:45:26.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f554658d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9330 0x56517eefc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f554658d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eee9c50 0x56517eeedbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f554658d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eef4320 0x56517eeee130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f554658d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f553c1ad400 con 0x56517eee9330 2024-05-09T08:45:26.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f554658d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f553c1ae870 con 0x56517eee9c50 2024-05-09T08:45:26.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f554658d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f553c1adad0 con 0x56517eef4320 2024-05-09T08:45:26.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f554658d740 10 client.0 _ll_register_callbacks cb 0x7f5538032d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:26.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f5540d40640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9330 0x56517eefc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f5540d40640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9330 0x56517eefc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56572/0 (socket says 172.21.15.107:56572) 2024-05-09T08:45:26.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f5540d40640 1 -- 172.21.15.107:0/1410831440 learned_addr learned my addr 172.21.15.107:0/1410831440 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:26.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f5540d40640 1 --2- 172.21.15.107:0/1410831440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eef4320 0x56517eeee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f5540d40640 1 -- 172.21.15.107:0/1410831440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eef4320 msgr2=0x56517eeee130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:26.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f5540d40640 1 --2- 172.21.15.107:0/1410831440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eef4320 0x56517eeee130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f5540d40640 1 -- 172.21.15.107:0/1410831440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eee9c50 msgr2=0x56517eeedbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:26.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f5540d40640 1 --2- 172.21.15.107:0/1410831440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eee9c50 0x56517eeedbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f5540d40640 1 -- 172.21.15.107:0/1410831440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56517eeee960 con 0x56517eee9330 2024-05-09T08:45:26.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f5540d40640 1 --2- 172.21.15.107:0/1410831440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9330 0x56517eefc8f0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f55380042b0 tx=0x7f5538004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec51a6c317f3c1e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:26.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f5524ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:45:26.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f5524ff9640 1 -- 172.21.15.107:0/1410831440 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5538035030 con 0x56517eee9330 2024-05-09T08:45:26.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f5524ff9640 1 -- 172.21.15.107:0/1410831440 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7f5538005020 con 0x56517eee9330 2024-05-09T08:45:26.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f554658d740 1 -- 172.21.15.107:0/1410831440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5538031dd0 con 0x56517eee9330 2024-05-09T08:45:26.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f5524ff9640 1 -- 172.21.15.107:0/1410831440 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5538065050 con 0x56517eee9330 2024-05-09T08:45:26.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f554658d740 10 client.18885 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:26.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f554658d740 1 -- 172.21.15.107:0/1410831440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56517ef6a8e0 con 0x56517eee9330 2024-05-09T08:45:26.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.592+0000 7f5524ff9640 1 -- 172.21.15.107:0/1410831440 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5538031dd0 con 0x56517eee9330 2024-05-09T08:45:26.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.593+0000 7f5524ff9640 1 -- 172.21.15.107:0/1410831440 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f553804e030 con 0x56517eee9330 2024-05-09T08:45:26.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.593+0000 7f554658d740 10 client.18885 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.593+0000 7f554658d740 10 client.18885 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:45:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.593+0000 7f554658d740 20 client.18885 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.593+0000 7f554658d740 1 --2- 172.21.15.107:0/1410831440 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x56517ef6af40 0x56517ef6d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.593+0000 7f554658d740 4 client.18885 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:45:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.593+0000 7f554658d740 1 -- 172.21.15.107:0/1410831440 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56517ee566c0 con 0x56517ef6af40 2024-05-09T08:45:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.593+0000 7f5541541640 1 --2- 172.21.15.107:0/1410831440 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x56517ef6af40 0x56517ef6d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.593+0000 7f5541541640 1 --2- 172.21.15.107:0/1410831440 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x56517ef6af40 0x56517ef6d380 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a433d7e5fe79d01b server_cookie=b6d0e877bf5b6367 in_seq=0 out_seq=0 2024-05-09T08:45:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.593+0000 7f5524ff9640 10 client.18885 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:45:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.593+0000 7f5524ff9640 1 -- 172.21.15.107:0/1410831440 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56517ee566c0 con 0x56517ef6af40 2024-05-09T08:45:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.593+0000 7f5524ff9640 10 client.18885 handle_command_reply: tid=0 2024-05-09T08:45:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.593+0000 7f554658d740 1 client.18885 shutdown 2024-05-09T08:45:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f554658d740 1 -- 172.21.15.107:0/1410831440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9330 msgr2=0x56517eefc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f554658d740 1 --2- 172.21.15.107:0/1410831440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9330 0x56517eefc8f0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f55380042b0 tx=0x7f5538004550 comp rx=0 tx=0).stop 2024-05-09T08:45:26.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f554658d740 1 -- 172.21.15.107:0/1410831440 shutdown_connections 2024-05-09T08:45:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f554658d740 1 --2- 172.21.15.107:0/1410831440 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x56517ef6af40 0x56517ef6d380 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f554658d740 1 --2- 172.21.15.107:0/1410831440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56517eef4320 0x56517eeee130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f5524ff9640 0 client.18885 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:45:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f554658d740 1 --2- 172.21.15.107:0/1410831440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56517eee9c50 0x56517eeedbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f554658d740 1 --2- 172.21.15.107:0/1410831440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56517eee9330 0x56517eefc8f0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f554658d740 1 -- 172.21.15.107:0/1410831440 >> 172.21.15.107:0/1410831440 conn(0x56517eee5650 msgr2=0x56517eee47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f554658d740 1 -- 172.21.15.107:0/1410831440 shutdown_connections 2024-05-09T08:45:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f554658d740 1 -- 172.21.15.107:0/1410831440 wait complete. 2024-05-09T08:45:26.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f554658d740 20 client.18885 trim_cache size 0 max 16384 2024-05-09T08:45:26.601 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:45:26.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f55437a5640 1 -- 172.21.15.107:0/502346829 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f55280a2d00 msgr2=0x7f55280a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f55437a5640 1 --2- 172.21.15.107:0/502346829 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f55280a2d00 0x7f55280a51c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f55340ccd70 tx=0x7f553401eb40 comp rx=0 tx=0).stop 2024-05-09T08:45:26.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f55437a5640 1 -- 172.21.15.107:0/502346829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f553c106720 msgr2=0x7f553c19c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f55437a5640 1 --2- 172.21.15.107:0/502346829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f553c106720 0x7f553c19c8c0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f553400eaa0 tx=0x7f553400ef60 comp rx=0 tx=0).stop 2024-05-09T08:45:26.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f55437a5640 1 -- 172.21.15.107:0/502346829 shutdown_connections 2024-05-09T08:45:26.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f55437a5640 1 --2- 172.21.15.107:0/502346829 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f55280a2d00 0x7f55280a51c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f55437a5640 1 --2- 172.21.15.107:0/502346829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f553c106720 0x7f553c19c8c0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f55437a5640 1 --2- 172.21.15.107:0/502346829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f553c1010d0 0x7f553c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f55437a5640 1 --2- 172.21.15.107:0/502346829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f553c100710 0x7f553c198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.594+0000 7f55437a5640 1 -- 172.21.15.107:0/502346829 >> 172.21.15.107:0/502346829 conn(0x7f553c077790 msgr2=0x7f553c1033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:26.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.595+0000 7f55437a5640 1 -- 172.21.15.107:0/502346829 shutdown_connections 2024-05-09T08:45:26.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.595+0000 7f55437a5640 1 -- 172.21.15.107:0/502346829 wait complete. 2024-05-09T08:45:26.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:45:26.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.866+0000 7f503c88c640 1 -- 172.21.15.107:0/1780000274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50381088c0 msgr2=0x7f5038108d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.866+0000 7f503c88c640 1 --2- 172.21.15.107:0/1780000274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50381088c0 0x7f5038108d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f502c009890 tx=0x7f502c02a380 comp rx=0 tx=0).stop 2024-05-09T08:45:26.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.868+0000 7f503c88c640 1 -- 172.21.15.107:0/1780000274 shutdown_connections 2024-05-09T08:45:26.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.868+0000 7f503c88c640 1 --2- 172.21.15.107:0/1780000274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f503811b010 0x7f503811d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.868+0000 7f503c88c640 1 --2- 172.21.15.107:0/1780000274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50381088c0 0x7f5038108d40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.868+0000 7f503c88c640 1 --2- 172.21.15.107:0/1780000274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f503810fea0 0x7f5038110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.868+0000 7f503c88c640 1 -- 172.21.15.107:0/1780000274 >> 172.21.15.107:0/1780000274 conn(0x7f503806c810 msgr2=0x7f503806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:26.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.872+0000 7f503c88c640 1 -- 172.21.15.107:0/1780000274 shutdown_connections 2024-05-09T08:45:26.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.874+0000 7f503c88c640 1 -- 172.21.15.107:0/1780000274 wait complete. 2024-05-09T08:45:26.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.875+0000 7f503c88c640 1 Processor -- start 2024-05-09T08:45:26.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.876+0000 7f503c88c640 1 -- start start 2024-05-09T08:45:26.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.876+0000 7f503c88c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50381088c0 0x7f50381a6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.876+0000 7f503c88c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f503810fea0 0x7f50381a7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.876+0000 7f503c88c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f503811b010 0x7f50381ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.876+0000 7f503c88c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5038120740 con 0x7f503810fea0 2024-05-09T08:45:26.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.876+0000 7f503c88c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f50381205c0 con 0x7f503811b010 2024-05-09T08:45:26.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.876+0000 7f503c88c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f50381208c0 con 0x7f50381088c0 2024-05-09T08:45:26.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.876+0000 7f503659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50381088c0 0x7f50381a6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.876+0000 7f503659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50381088c0 0x7f50381a6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:50454/0 (socket says 172.21.15.107:50454) 2024-05-09T08:45:26.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.876+0000 7f503659c640 1 -- 172.21.15.107:0/3838438278 learned_addr learned my addr 172.21.15.107:0/3838438278 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:26.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.877+0000 7f5036d9d640 1 --2- 172.21.15.107:0/3838438278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f503811b010 0x7f50381ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.877+0000 7f5035d9b640 1 --2- 172.21.15.107:0/3838438278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f503810fea0 0x7f50381a7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.877+0000 7f503659c640 1 -- 172.21.15.107:0/3838438278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f503811b010 msgr2=0x7f50381ab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.877+0000 7f503659c640 1 --2- 172.21.15.107:0/3838438278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f503811b010 0x7f50381ab5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.877+0000 7f503659c640 1 -- 172.21.15.107:0/3838438278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f503810fea0 msgr2=0x7f50381a7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.877+0000 7f503659c640 1 --2- 172.21.15.107:0/3838438278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f503810fea0 0x7f50381a7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.877+0000 7f503659c640 1 -- 172.21.15.107:0/3838438278 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50381abd10 con 0x7f50381088c0 2024-05-09T08:45:26.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.879+0000 7f503659c640 1 --2- 172.21.15.107:0/3838438278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50381088c0 0x7f50381a6d20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f503000c420 tx=0x7f5030009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:26.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.880+0000 7f50277fe640 1 -- 172.21.15.107:0/3838438278 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5030003970 con 0x7f50381088c0 2024-05-09T08:45:26.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.880+0000 7f503c88c640 1 -- 172.21.15.107:0/3838438278 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50381ac000 con 0x7f50381088c0 2024-05-09T08:45:26.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.880+0000 7f503c88c640 1 -- 172.21.15.107:0/3838438278 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50381b3890 con 0x7f50381088c0 2024-05-09T08:45:26.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.880+0000 7f50277fe640 1 -- 172.21.15.107:0/3838438278 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5030004020 con 0x7f50381088c0 2024-05-09T08:45:26.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.880+0000 7f50277fe640 1 -- 172.21.15.107:0/3838438278 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503000b9f0 con 0x7f50381088c0 2024-05-09T08:45:26.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.883+0000 7f503f596740 1 Processor -- start 2024-05-09T08:45:26.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.884+0000 7f503f596740 1 -- start start 2024-05-09T08:45:26.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.884+0000 7f503f596740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c50520 0x564165c50900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.884+0000 7f503f596740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c50e40 0x564165c5afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.884+0000 7f503f596740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c5b510 0x564165c5d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.884+0000 7f503f596740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564165bb5c40 con 0x564165c50e40 2024-05-09T08:45:26.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.884+0000 7f503f596740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564165b5d580 con 0x564165c5b510 2024-05-09T08:45:26.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.884+0000 7f503f596740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564165bb1c00 con 0x564165c50520 2024-05-09T08:45:26.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.887+0000 7f5036d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c50520 0x564165c50900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.887+0000 7f5036d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c50520 0x564165c50900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:50468/0 (socket says 172.21.15.107:50468) 2024-05-09T08:45:26.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.888+0000 7f5036d9d640 1 -- 172.21.15.107:0/3918395620 learned_addr learned my addr 172.21.15.107:0/3918395620 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:26.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.888+0000 7f503659c640 1 --2- 172.21.15.107:0/3918395620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c50e40 0x564165c5afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.888+0000 7f5036d9d640 1 -- 172.21.15.107:0/3918395620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c5b510 msgr2=0x564165c5d900 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:45:26.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.888+0000 7f5036d9d640 1 --2- 172.21.15.107:0/3918395620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c5b510 0x564165c5d900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.888+0000 7f5036d9d640 1 -- 172.21.15.107:0/3918395620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c50e40 msgr2=0x564165c5afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:26.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.888+0000 7f5036d9d640 1 --2- 172.21.15.107:0/3918395620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c50e40 0x564165c5afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.888+0000 7f5036d9d640 1 -- 172.21.15.107:0/3918395620 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641658c93f0 con 0x564165c50520 2024-05-09T08:45:26.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.889+0000 7f50277fe640 1 -- 172.21.15.107:0/3838438278 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f503001d020 con 0x7f50381088c0 2024-05-09T08:45:26.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.889+0000 7f5036d9d640 1 --2- 172.21.15.107:0/3918395620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c50520 0x564165c50900 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5028009b50 tx=0x7f5028010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e21a9ae5f8f8328c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:26.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.889+0000 7f5024ff9640 1 -- 172.21.15.107:0/3918395620 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028009e40 con 0x564165c50520 2024-05-09T08:45:26.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.889+0000 7f5024ff9640 1 -- 172.21.15.107:0/3918395620 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f502801b930 con 0x564165c50520 2024-05-09T08:45:26.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.889+0000 7f5024ff9640 1 -- 172.21.15.107:0/3918395620 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50280129e0 con 0x564165c50520 2024-05-09T08:45:26.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.890+0000 7f50277fe640 1 --2- 172.21.15.107:0/3838438278 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f50200a2ed0 0x7f50200a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.890+0000 7f503f596740 1 -- 172.21.15.107:0/3918395620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c50520 msgr2=0x564165c50900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.890+0000 7f503f596740 1 --2- 172.21.15.107:0/3918395620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c50520 0x564165c50900 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5028009b50 tx=0x7f5028010740 comp rx=0 tx=0).stop 2024-05-09T08:45:26.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.890+0000 7f50277fe640 1 -- 172.21.15.107:0/3838438278 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7f50300d1760 con 0x7f50381088c0 2024-05-09T08:45:26.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.891+0000 7f503659c640 1 --2- 172.21.15.107:0/3838438278 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f50200a2ed0 0x7f50200a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.891+0000 7f503f596740 1 -- 172.21.15.107:0/3918395620 shutdown_connections 2024-05-09T08:45:26.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.891+0000 7f503f596740 1 --2- 172.21.15.107:0/3918395620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c5b510 0x564165c5d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.891+0000 7f503f596740 1 --2- 172.21.15.107:0/3918395620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c50e40 0x564165c5afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.891+0000 7f503f596740 1 --2- 172.21.15.107:0/3918395620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c50520 0x564165c50900 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.891+0000 7f503f596740 1 -- 172.21.15.107:0/3918395620 >> 172.21.15.107:0/3918395620 conn(0x564165c4c840 msgr2=0x564165c4cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:26.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.892+0000 7f503659c640 1 --2- 172.21.15.107:0/3838438278 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f50200a2ed0 0x7f50200a5390 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x564165c51ae0 tx=0x7f503000cf20 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:26.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.895+0000 7f503f596740 1 -- 172.21.15.107:0/3918395620 shutdown_connections 2024-05-09T08:45:26.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.895+0000 7f503f596740 1 -- 172.21.15.107:0/3918395620 wait complete. 2024-05-09T08:45:26.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.896+0000 7f503f596740 1 -- start start 2024-05-09T08:45:26.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.896+0000 7f503f596740 1 Processor -- start 2024-05-09T08:45:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.896+0000 7f503f596740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c50520 0x564165c584a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.896+0000 7f503f596740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c50e40 0x564165c589e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.896+0000 7f503f596740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c5b510 0x564165c58f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.896+0000 7f503f596740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564165c6f310 con 0x564165c50520 2024-05-09T08:45:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.896+0000 7f503f596740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564165b70c40 con 0x564165c50e40 2024-05-09T08:45:26.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.896+0000 7f503f596740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564165c6f490 con 0x564165c5b510 2024-05-09T08:45:26.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.896+0000 7f503f596740 10 client.0 _ll_register_callbacks cb 0x5641658c8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:26.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.897+0000 7f5035d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c50e40 0x564165c589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.897+0000 7f5035d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c50e40 0x564165c589e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44336/0 (socket says 172.21.15.107:44336) 2024-05-09T08:45:26.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.897+0000 7f5035d9b640 1 -- 172.21.15.107:0/3443763093 learned_addr learned my addr 172.21.15.107:0/3443763093 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:26.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.897+0000 7f5036d9d640 1 --2- 172.21.15.107:0/3443763093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c5b510 0x564165c58f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.898+0000 7f5036d9d640 1 --2- 172.21.15.107:0/3443763093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c50520 0x564165c584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.898+0000 7f5035d9b640 1 -- 172.21.15.107:0/3443763093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c5b510 msgr2=0x564165c58f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:26.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.898+0000 7f5035d9b640 1 --2- 172.21.15.107:0/3443763093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c5b510 0x564165c58f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.898+0000 7f5035d9b640 1 -- 172.21.15.107:0/3443763093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c50520 msgr2=0x564165c584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:26.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.898+0000 7f5035d9b640 1 --2- 172.21.15.107:0/3443763093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c50520 0x564165c584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.898+0000 7f5035d9b640 1 -- 172.21.15.107:0/3443763093 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641659937d0 con 0x564165c50e40 2024-05-09T08:45:26.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.899+0000 7f5035d9b640 1 --2- 172.21.15.107:0/3443763093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c50e40 0x564165c589e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f502c02a8a0 tx=0x7f502c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5853ac975982cb7e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:26.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.899+0000 7f501dffb640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:45:26.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.899+0000 7f501dffb640 1 -- 172.21.15.107:0/3443763093 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502c043020 con 0x564165c50e40 2024-05-09T08:45:26.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.899+0000 7f503f596740 1 -- 172.21.15.107:0/3443763093 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f502801b930 con 0x564165c50e40 2024-05-09T08:45:26.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.899+0000 7f501dffb640 1 -- 172.21.15.107:0/3443763093 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7f502c030680 con 0x564165c50e40 2024-05-09T08:45:26.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.900+0000 7f501dffb640 1 -- 172.21.15.107:0/3443763093 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f502c04e050 con 0x564165c50e40 2024-05-09T08:45:26.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.905+0000 7f503f596740 10 client.27833 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:26.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.905+0000 7f503f596740 1 -- 172.21.15.107:0/3443763093 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564165c7b5a0 con 0x564165c50e40 2024-05-09T08:45:26.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.906+0000 7f501dffb640 1 -- 172.21.15.107:0/3443763093 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f502801b930 con 0x564165c50e40 2024-05-09T08:45:26.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.906+0000 7f501dffb640 1 -- 172.21.15.107:0/3443763093 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f502c04a4b0 con 0x564165c50e40 2024-05-09T08:45:26.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.906+0000 7f503f596740 10 client.27833 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:26.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.906+0000 7f503f596740 10 client.27833 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:45:26.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.906+0000 7f503f596740 20 client.27833 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:26.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.906+0000 7f503f596740 1 --2- 172.21.15.107:0/3443763093 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x564165c7ce90 0x564165c7f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.906+0000 7f503f596740 4 client.27833 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:45:26.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.906+0000 7f503f596740 1 -- 172.21.15.107:0/3443763093 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564165bc3370 con 0x564165c7ce90 2024-05-09T08:45:26.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.908+0000 7f5036d9d640 1 --2- 172.21.15.107:0/3443763093 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x564165c7ce90 0x564165c7f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.910+0000 7f501dffb640 10 client.27833 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:45:26.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.910+0000 7f5036d9d640 1 --2- 172.21.15.107:0/3443763093 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x564165c7ce90 0x564165c7f350 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=90fff27b6313aa7a server_cookie=8c044f48add35f9f in_seq=0 out_seq=0 2024-05-09T08:45:26.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.914+0000 7f501dffb640 1 -- 172.21.15.107:0/3443763093 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564165bc3370 con 0x564165c7ce90 2024-05-09T08:45:26.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.914+0000 7f501dffb640 10 client.27833 handle_command_reply: tid=0 2024-05-09T08:45:26.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.914+0000 7f503f596740 1 client.27833 shutdown 2024-05-09T08:45:26.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.916+0000 7f503f596740 1 -- 172.21.15.107:0/3443763093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c50e40 msgr2=0x564165c589e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.916+0000 7f503f596740 1 --2- 172.21.15.107:0/3443763093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c50e40 0x564165c589e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f502c02a8a0 tx=0x7f502c004270 comp rx=0 tx=0).stop 2024-05-09T08:45:26.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.916+0000 7f503f596740 1 -- 172.21.15.107:0/3443763093 shutdown_connections 2024-05-09T08:45:26.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.916+0000 7f503f596740 1 --2- 172.21.15.107:0/3443763093 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x564165c7ce90 0x564165c7f350 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.916+0000 7f503f596740 1 --2- 172.21.15.107:0/3443763093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c5b510 0x564165c58f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.916+0000 7f503f596740 1 --2- 172.21.15.107:0/3443763093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c50e40 0x564165c589e0 secure :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f502c02a8a0 tx=0x7f502c004270 comp rx=0 tx=0).stop 2024-05-09T08:45:26.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.916+0000 7f503f596740 1 --2- 172.21.15.107:0/3443763093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c50520 0x564165c584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.916+0000 7f503f596740 1 -- 172.21.15.107:0/3443763093 >> 172.21.15.107:0/3443763093 conn(0x564165c4c840 msgr2=0x564165c52500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:26.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.916+0000 7f501dffb640 0 client.27833 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:45:26.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.917+0000 7f503f596740 1 -- 172.21.15.107:0/3443763093 shutdown_connections 2024-05-09T08:45:26.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.918+0000 7f503f596740 1 -- 172.21.15.107:0/3443763093 wait complete. 2024-05-09T08:45:26.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.918+0000 7f503f596740 20 client.27833 trim_cache size 0 max 16384 2024-05-09T08:45:26.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.958+0000 7f503f596740 1 Processor -- start 2024-05-09T08:45:26.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.959+0000 7f503f596740 1 -- start start 2024-05-09T08:45:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.960+0000 7f503f596740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c50520 0x564165c78230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.960+0000 7f503f596740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c5b510 0x564165c84c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.960+0000 7f5036d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c50520 0x564165c78230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.960+0000 7f5036d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c50520 0x564165c78230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44348/0 (socket says 172.21.15.107:44348) 2024-05-09T08:45:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.960+0000 7f5036d9d640 1 -- 172.21.15.107:0/1029205809 learned_addr learned my addr 172.21.15.107:0/1029205809 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.960+0000 7f503f596740 1 --2- 172.21.15.107:0/1029205809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c85160 0x564165c59b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.961+0000 7f503f596740 1 -- 172.21.15.107:0/1029205809 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5018012960 con 0x564165c85160 2024-05-09T08:45:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.961+0000 7f503f596740 1 -- 172.21.15.107:0/1029205809 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5018010230 con 0x564165c50520 2024-05-09T08:45:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.961+0000 7f503f596740 1 -- 172.21.15.107:0/1029205809 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f501800e080 con 0x564165c5b510 2024-05-09T08:45:26.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.962+0000 7f5035d9b640 1 --2- 172.21.15.107:0/1029205809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c5b510 0x564165c84c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.963+0000 7f5036d9d640 1 -- 172.21.15.107:0/1029205809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c5b510 msgr2=0x564165c84c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:26.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.963+0000 7f5036d9d640 1 --2- 172.21.15.107:0/1029205809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c5b510 0x564165c84c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.963+0000 7f5036d9d640 1 -- 172.21.15.107:0/1029205809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c85160 msgr2=0x564165c59b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:26.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.963+0000 7f5036d9d640 1 --2- 172.21.15.107:0/1029205809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c85160 0x564165c59b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.963+0000 7f5036d9d640 1 -- 172.21.15.107:0/1029205809 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564165c5dc70 con 0x564165c50520 2024-05-09T08:45:26.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.963+0000 7f5036d9d640 1 --2- 172.21.15.107:0/1029205809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c50520 0x564165c78230 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5028010dc0 tx=0x7f5028005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55837d27010e8096 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:26.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.965+0000 7f501ffff640 1 -- 172.21.15.107:0/1029205809 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028005d30 con 0x564165c50520 2024-05-09T08:45:26.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.965+0000 7f501ffff640 1 -- 172.21.15.107:0/1029205809 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f50280054f0 con 0x564165c50520 2024-05-09T08:45:26.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.966+0000 7f503f596740 1 -- 172.21.15.107:0/1029205809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c50520 msgr2=0x564165c78230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.966+0000 7f503f596740 1 --2- 172.21.15.107:0/1029205809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c50520 0x564165c78230 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5028010dc0 tx=0x7f5028005f90 comp rx=0 tx=0).stop 2024-05-09T08:45:26.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.969+0000 7f503f596740 1 -- 172.21.15.107:0/1029205809 shutdown_connections 2024-05-09T08:45:26.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.969+0000 7f503f596740 1 --2- 172.21.15.107:0/1029205809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c85160 0x564165c59b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.969+0000 7f503f596740 1 --2- 172.21.15.107:0/1029205809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c5b510 0x564165c84c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.969+0000 7f503f596740 1 --2- 172.21.15.107:0/1029205809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c50520 0x564165c78230 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.969+0000 7f503f596740 1 -- 172.21.15.107:0/1029205809 >> 172.21.15.107:0/1029205809 conn(0x564165c4c840 msgr2=0x564165c537f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:26.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.969+0000 7f503f596740 1 -- 172.21.15.107:0/1029205809 shutdown_connections 2024-05-09T08:45:26.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.969+0000 7f503f596740 1 -- 172.21.15.107:0/1029205809 wait complete. 2024-05-09T08:45:26.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.970+0000 7f503f596740 1 -- start start 2024-05-09T08:45:26.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.970+0000 7f503f596740 1 Processor -- start 2024-05-09T08:45:26.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.970+0000 7f503f596740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c50520 0x564165c74740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.970+0000 7f503f596740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c5b510 0x564165c74c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.970+0000 7f503f596740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c85160 0x564165c71710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.970+0000 7f503f596740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f501800e6f0 con 0x564165c85160 2024-05-09T08:45:26.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.970+0000 7f503f596740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f501800fb60 con 0x564165c5b510 2024-05-09T08:45:26.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.970+0000 7f503f596740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f501800edc0 con 0x564165c50520 2024-05-09T08:45:26.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.970+0000 7f503f596740 10 client.0 _ll_register_callbacks cb 0x7f5030009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:26.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.971+0000 7f5035d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c5b510 0x564165c74c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.971+0000 7f5035d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c5b510 0x564165c74c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44358/0 (socket says 172.21.15.107:44358) 2024-05-09T08:45:26.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.971+0000 7f5035d9b640 1 -- 172.21.15.107:0/2392290198 learned_addr learned my addr 172.21.15.107:0/2392290198 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:26.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.971+0000 7f5036d9d640 1 --2- 172.21.15.107:0/2392290198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c85160 0x564165c71710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.971+0000 7f5036d9d640 1 --2- 172.21.15.107:0/2392290198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c50520 0x564165c74740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.971+0000 7f5035d9b640 1 -- 172.21.15.107:0/2392290198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c50520 msgr2=0x564165c74740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:26.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.971+0000 7f5035d9b640 1 --2- 172.21.15.107:0/2392290198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c50520 0x564165c74740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.971+0000 7f5035d9b640 1 -- 172.21.15.107:0/2392290198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c85160 msgr2=0x564165c71710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:26.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.971+0000 7f5035d9b640 1 --2- 172.21.15.107:0/2392290198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c85160 0x564165c71710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.971+0000 7f5035d9b640 1 -- 172.21.15.107:0/2392290198 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564165c71f20 con 0x564165c5b510 2024-05-09T08:45:26.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.972+0000 7f5035d9b640 1 --2- 172.21.15.107:0/2392290198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c5b510 0x564165c74c80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f502c04a880 tx=0x7f502c04ad20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d4ae219c9590937 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:26.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.972+0000 7f501dffb640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:45:26.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.972+0000 7f501dffb640 1 -- 172.21.15.107:0/2392290198 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502c057060 con 0x564165c5b510 2024-05-09T08:45:26.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.972+0000 7f503f596740 1 -- 172.21.15.107:0/2392290198 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50280054f0 con 0x564165c5b510 2024-05-09T08:45:26.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.972+0000 7f501dffb640 1 -- 172.21.15.107:0/2392290198 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7f502c030890 con 0x564165c5b510 2024-05-09T08:45:26.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.972+0000 7f501dffb640 1 -- 172.21.15.107:0/2392290198 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f502c05e050 con 0x564165c5b510 2024-05-09T08:45:26.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.972+0000 7f503f596740 10 client.27839 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:26.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.972+0000 7f503f596740 1 -- 172.21.15.107:0/2392290198 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564165ce3f20 con 0x564165c5b510 2024-05-09T08:45:26.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.974+0000 7f501dffb640 1 -- 172.21.15.107:0/2392290198 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f50280054f0 con 0x564165c5b510 2024-05-09T08:45:26.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.974+0000 7f501dffb640 1 -- 172.21.15.107:0/2392290198 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f502c051070 con 0x564165c5b510 2024-05-09T08:45:26.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.974+0000 7f503f596740 10 client.27839 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:26.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.974+0000 7f503f596740 10 client.27839 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:45:26.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.974+0000 7f503f596740 20 client.27839 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:26.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.974+0000 7f503f596740 1 --2- 172.21.15.107:0/2392290198 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x564165ce45a0 0x564165ce6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:26.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.974+0000 7f503f596740 4 client.27839 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:45:26.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.974+0000 7f503f596740 1 -- 172.21.15.107:0/2392290198 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564165bbd6c0 con 0x564165ce45a0 2024-05-09T08:45:26.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.975+0000 7f5036d9d640 1 --2- 172.21.15.107:0/2392290198 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x564165ce45a0 0x564165ce6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:26.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.982+0000 7f5036d9d640 1 --2- 172.21.15.107:0/2392290198 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x564165ce45a0 0x564165ce6a00 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c4fe09b5f06a4b2a server_cookie=ae0e6e1d32876b1c in_seq=0 out_seq=0 2024-05-09T08:45:26.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.982+0000 7f501dffb640 10 client.27839 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:45:26.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.982+0000 7f501dffb640 1 -- 172.21.15.107:0/2392290198 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564165bbd6c0 con 0x564165ce45a0 2024-05-09T08:45:26.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.982+0000 7f501dffb640 10 client.27839 handle_command_reply: tid=0 2024-05-09T08:45:26.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.983+0000 7f503f596740 1 client.27839 shutdown 2024-05-09T08:45:26.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.987+0000 7f503f596740 1 -- 172.21.15.107:0/2392290198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c5b510 msgr2=0x564165c74c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:26.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.987+0000 7f503f596740 1 --2- 172.21.15.107:0/2392290198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c5b510 0x564165c74c80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f502c04a880 tx=0x7f502c04ad20 comp rx=0 tx=0).stop 2024-05-09T08:45:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.995+0000 7f503f596740 1 -- 172.21.15.107:0/2392290198 shutdown_connections 2024-05-09T08:45:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.995+0000 7f503f596740 1 --2- 172.21.15.107:0/2392290198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564165c85160 0x564165c71710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.995+0000 7f503f596740 1 --2- 172.21.15.107:0/2392290198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564165c5b510 0x564165c74c80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.995+0000 7f503f596740 1 --2- 172.21.15.107:0/2392290198 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x564165ce45a0 0x564165ce6a00 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.995+0000 7f503f596740 1 --2- 172.21.15.107:0/2392290198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564165c50520 0x564165c74740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.995+0000 7f503f596740 1 -- 172.21.15.107:0/2392290198 >> 172.21.15.107:0/2392290198 conn(0x564165c4c840 msgr2=0x564165c4b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:26.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.995+0000 7f501dffb640 0 client.27839 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:45:26.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.996+0000 7f503f596740 1 -- 172.21.15.107:0/2392290198 shutdown_connections 2024-05-09T08:45:26.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.996+0000 7f503f596740 1 -- 172.21.15.107:0/2392290198 wait complete. 2024-05-09T08:45:26.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:26.996+0000 7f503f596740 20 client.27839 trim_cache size 0 max 16384 2024-05-09T08:45:27.000 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:45:27.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.001+0000 7f503c88c640 1 -- 172.21.15.107:0/3838438278 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f50200a2ed0 msgr2=0x7f50200a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:27.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.001+0000 7f503c88c640 1 --2- 172.21.15.107:0/3838438278 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f50200a2ed0 0x7f50200a5390 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x564165c51ae0 tx=0x7f503000cf20 comp rx=0 tx=0).stop 2024-05-09T08:45:27.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.001+0000 7f503c88c640 1 -- 172.21.15.107:0/3838438278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50381088c0 msgr2=0x7f50381a6d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:27.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.001+0000 7f503c88c640 1 --2- 172.21.15.107:0/3838438278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50381088c0 0x7f50381a6d20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f503000c420 tx=0x7f5030009f90 comp rx=0 tx=0).stop 2024-05-09T08:45:27.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.002+0000 7f503c88c640 1 -- 172.21.15.107:0/3838438278 shutdown_connections 2024-05-09T08:45:27.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.002+0000 7f503c88c640 1 --2- 172.21.15.107:0/3838438278 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f50200a2ed0 0x7f50200a5390 secure :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0x564165c51ae0 tx=0x7f503000cf20 comp rx=0 tx=0).stop 2024-05-09T08:45:27.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.002+0000 7f503c88c640 1 --2- 172.21.15.107:0/3838438278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f503811b010 0x7f50381ab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.002+0000 7f503c88c640 1 --2- 172.21.15.107:0/3838438278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f503810fea0 0x7f50381a7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.002+0000 7f503c88c640 1 --2- 172.21.15.107:0/3838438278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50381088c0 0x7f50381a6d20 secure :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f503000c420 tx=0x7f5030009f90 comp rx=0 tx=0).stop 2024-05-09T08:45:27.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.002+0000 7f503c88c640 1 -- 172.21.15.107:0/3838438278 >> 172.21.15.107:0/3838438278 conn(0x7f503806c810 msgr2=0x7f5038119000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.005+0000 7f503c88c640 1 -- 172.21.15.107:0/3838438278 shutdown_connections 2024-05-09T08:45:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.006+0000 7f503c88c640 1 -- 172.21.15.107:0/3838438278 wait complete. 2024-05-09T08:45:27.036 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:26 smithi107 ceph-mon[30122]: pgmap v377: 129 pgs: 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 111 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 12 MiB/s wr, 2.95k op/s 2024-05-09T08:45:27.036 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:26 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3459465481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:45:27.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:45:27.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:26 smithi157 ceph-mon[33614]: pgmap v377: 129 pgs: 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 111 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 12 MiB/s wr, 2.95k op/s 2024-05-09T08:45:27.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:26 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3459465481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:45:27.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:26 smithi110 ceph-mon[33622]: pgmap v377: 129 pgs: 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 111 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 12 MiB/s wr, 2.95k op/s 2024-05-09T08:45:27.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:26 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3459465481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:45:27.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.274+0000 7fac7ffff640 1 -- 172.21.15.107:0/2362582580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac801092b0 msgr2=0x7fac8011d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:27.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.274+0000 7fac7ffff640 1 --2- 172.21.15.107:0/2362582580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac801092b0 0x7fac8011d020 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fac70009860 tx=0x7fac7002a410 comp rx=0 tx=0).stop 2024-05-09T08:45:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.275+0000 7fac7ffff640 1 -- 172.21.15.107:0/2362582580 shutdown_connections 2024-05-09T08:45:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.275+0000 7fac7ffff640 1 --2- 172.21.15.107:0/2362582580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac801092b0 0x7fac8011d020 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.275+0000 7fac7ffff640 1 --2- 172.21.15.107:0/2362582580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac80108830 0x7fac80108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.275+0000 7fac7ffff640 1 --2- 172.21.15.107:0/2362582580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac8010fed0 0x7fac801102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.275+0000 7fac7ffff640 1 -- 172.21.15.107:0/2362582580 >> 172.21.15.107:0/2362582580 conn(0x7fac8006c7b0 msgr2=0x7fac8006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.276+0000 7fac7ffff640 1 -- 172.21.15.107:0/2362582580 shutdown_connections 2024-05-09T08:45:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.276+0000 7fac7ffff640 1 -- 172.21.15.107:0/2362582580 wait complete. 2024-05-09T08:45:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.277+0000 7fac7ffff640 1 Processor -- start 2024-05-09T08:45:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.277+0000 7fac7ffff640 1 -- start start 2024-05-09T08:45:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.278+0000 7fac7ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80108830 0x7fac801ac920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.278+0000 7fac7ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac801092b0 0x7fac801ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.278+0000 7fac7ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8010fed0 0x7fac801a6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.278+0000 7fac7ffff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fac80120790 con 0x7fac8010fed0 2024-05-09T08:45:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.278+0000 7fac7ffff640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fac80120610 con 0x7fac80108830 2024-05-09T08:45:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.278+0000 7fac7ffff640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fac80120910 con 0x7fac801092b0 2024-05-09T08:45:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.278+0000 7fac7f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8010fed0 0x7fac801a6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.278+0000 7fac7f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8010fed0 0x7fac801a6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56650/0 (socket says 172.21.15.107:56650) 2024-05-09T08:45:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.278+0000 7fac7f7fe640 1 -- 172.21.15.107:0/85128262 learned_addr learned my addr 172.21.15.107:0/85128262 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:27.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.279+0000 7fac7f7fe640 1 -- 172.21.15.107:0/85128262 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac801092b0 msgr2=0x7fac801ace60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:45:27.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.279+0000 7fac7f7fe640 1 --2- 172.21.15.107:0/85128262 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac801092b0 0x7fac801ace60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.279+0000 7fac7f7fe640 1 -- 172.21.15.107:0/85128262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80108830 msgr2=0x7fac801ac920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:27.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.279+0000 7fac7f7fe640 1 --2- 172.21.15.107:0/85128262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80108830 0x7fac801ac920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.279+0000 7fac7f7fe640 1 -- 172.21.15.107:0/85128262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac801a7250 con 0x7fac8010fed0 2024-05-09T08:45:27.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.279+0000 7fac7f7fe640 1 --2- 172.21.15.107:0/85128262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8010fed0 0x7fac801a6ad0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fac70009980 tx=0x7fac70030ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:27.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.279+0000 7fac67fff640 1 -- 172.21.15.107:0/85128262 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac70004260 con 0x7fac8010fed0 2024-05-09T08:45:27.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.280+0000 7fac7ffff640 1 -- 172.21.15.107:0/85128262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac801a74e0 con 0x7fac8010fed0 2024-05-09T08:45:27.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.280+0000 7fac7ffff640 1 -- 172.21.15.107:0/85128262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac80112800 con 0x7fac8010fed0 2024-05-09T08:45:27.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.280+0000 7fac67fff640 1 -- 172.21.15.107:0/85128262 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac70004400 con 0x7fac8010fed0 2024-05-09T08:45:27.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.281+0000 7fac67fff640 1 -- 172.21.15.107:0/85128262 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac70004c80 con 0x7fac8010fed0 2024-05-09T08:45:27.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.283+0000 7fac8839b740 1 Processor -- start 2024-05-09T08:45:27.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.285+0000 7fac67fff640 1 -- 172.21.15.107:0/85128262 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fac70039040 con 0x7fac8010fed0 2024-05-09T08:45:27.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.286+0000 7fac67fff640 1 --2- 172.21.15.107:0/85128262 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fac680a2d20 0x7fac680a51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.286+0000 7fac67fff640 1 -- 172.21.15.107:0/85128262 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7fac700f0270 con 0x7fac8010fed0 2024-05-09T08:45:27.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.287+0000 7fac8839b740 1 -- start start 2024-05-09T08:45:27.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.287+0000 7fac8839b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fbb520 0x55de09fbb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.287+0000 7fac8839b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fbbe40 0x55de09fc5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.287+0000 7fac8839b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fc6510 0x55de09fc8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.287+0000 7fac8839b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55de09f20c40 con 0x55de09fbbe40 2024-05-09T08:45:27.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.287+0000 7fac8839b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55de09ec8580 con 0x55de09fc6510 2024-05-09T08:45:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.287+0000 7fac8839b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55de09f1cc00 con 0x55de09fbb520 2024-05-09T08:45:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.287+0000 7fac7effd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fbb520 0x55de09fbb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.287+0000 7fac7effd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fbb520 0x55de09fbb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:50526/0 (socket says 172.21.15.107:50526) 2024-05-09T08:45:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.287+0000 7fac7effd640 1 -- 172.21.15.107:0/2259948685 learned_addr learned my addr 172.21.15.107:0/2259948685 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.287+0000 7fac7f7fe640 1 --2- 172.21.15.107:0/2259948685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fc6510 0x55de09fc8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.288+0000 7fac7e7fc640 1 --2- 172.21.15.107:0/85128262 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fac680a2d20 0x7fac680a51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.288+0000 7fac7f7fe640 1 -- 172.21.15.107:0/2259948685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fbb520 msgr2=0x55de09fbb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.288+0000 7fac7f7fe640 1 --2- 172.21.15.107:0/2259948685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fbb520 0x55de09fbb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.288+0000 7fac7f7fe640 1 -- 172.21.15.107:0/2259948685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fbbe40 msgr2=0x55de09fc5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.288+0000 7fac7f7fe640 1 --2- 172.21.15.107:0/2259948685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fbbe40 0x55de09fc5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.288+0000 7fac7f7fe640 1 -- 172.21.15.107:0/2259948685 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de09c343f0 con 0x55de09fc6510 2024-05-09T08:45:27.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.289+0000 7fac7f7fe640 1 --2- 172.21.15.107:0/2259948685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fc6510 0x55de09fc8900 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fac700ed560 tx=0x7fac700bbe70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3688cba743c25028 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:27.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.289+0000 7fac657fa640 1 -- 172.21.15.107:0/2259948685 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac70049040 con 0x55de09fc6510 2024-05-09T08:45:27.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.289+0000 7fac657fa640 1 -- 172.21.15.107:0/2259948685 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac700bca30 con 0x55de09fc6510 2024-05-09T08:45:27.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.289+0000 7fac657fa640 1 -- 172.21.15.107:0/2259948685 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac700ed9e0 con 0x55de09fc6510 2024-05-09T08:45:27.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.290+0000 7fac7e7fc640 1 --2- 172.21.15.107:0/85128262 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fac680a2d20 0x7fac680a51e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55de09fbcae0 tx=0x7fac74008dd0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:27.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.291+0000 7fac8839b740 1 -- 172.21.15.107:0/2259948685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fc6510 msgr2=0x55de09fc8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.291+0000 7fac8839b740 1 --2- 172.21.15.107:0/2259948685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fc6510 0x55de09fc8900 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fac700ed560 tx=0x7fac700bbe70 comp rx=0 tx=0).stop 2024-05-09T08:45:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.291+0000 7fac8839b740 1 -- 172.21.15.107:0/2259948685 shutdown_connections 2024-05-09T08:45:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.292+0000 7fac8839b740 1 --2- 172.21.15.107:0/2259948685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fc6510 0x55de09fc8900 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.292+0000 7fac8839b740 1 --2- 172.21.15.107:0/2259948685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fbbe40 0x55de09fc5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.292+0000 7fac8839b740 1 --2- 172.21.15.107:0/2259948685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fbb520 0x55de09fbb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.292+0000 7fac8839b740 1 -- 172.21.15.107:0/2259948685 >> 172.21.15.107:0/2259948685 conn(0x55de09fb7840 msgr2=0x55de09fb7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.294+0000 7fac8839b740 1 -- 172.21.15.107:0/2259948685 shutdown_connections 2024-05-09T08:45:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.294+0000 7fac8839b740 1 -- 172.21.15.107:0/2259948685 wait complete. 2024-05-09T08:45:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.294+0000 7fac8839b740 1 -- start start 2024-05-09T08:45:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.294+0000 7fac8839b740 1 Processor -- start 2024-05-09T08:45:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.295+0000 7fac8839b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fbb520 0x55de09fc34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.295+0000 7fac8839b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fbbe40 0x55de09fc39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.295+0000 7fac8839b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fc6510 0x55de09fc3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.295+0000 7fac8839b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55de09fc90c0 con 0x55de09fbbe40 2024-05-09T08:45:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.295+0000 7fac8839b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55de09edbc40 con 0x55de09fbb520 2024-05-09T08:45:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.295+0000 7fac8839b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55de09fc9240 con 0x55de09fc6510 2024-05-09T08:45:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.295+0000 7fac7f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fc6510 0x55de09fc3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.296+0000 7fac7f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fc6510 0x55de09fc3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:50532/0 (socket says 172.21.15.107:50532) 2024-05-09T08:45:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.296+0000 7fac7f7fe640 1 -- 172.21.15.107:0/3736696127 learned_addr learned my addr 172.21.15.107:0/3736696127 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.296+0000 7fac8839b740 10 client.0 _ll_register_callbacks cb 0x55de09c33be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.297+0000 7fac7e7fc640 1 --2- 172.21.15.107:0/3736696127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fbbe40 0x55de09fc39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.297+0000 7fac7e7fc640 1 -- 172.21.15.107:0/3736696127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fc6510 msgr2=0x55de09fc3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.297+0000 7fac7e7fc640 1 --2- 172.21.15.107:0/3736696127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fc6510 0x55de09fc3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.297+0000 7fac7e7fc640 1 -- 172.21.15.107:0/3736696127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fbb520 msgr2=0x55de09fc34b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:45:27.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.298+0000 7fac7e7fc640 1 --2- 172.21.15.107:0/3736696127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fbb520 0x55de09fc34b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.298+0000 7fac7e7fc640 1 -- 172.21.15.107:0/3736696127 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de09cfe7d0 con 0x55de09fbbe40 2024-05-09T08:45:27.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.298+0000 7fac7e7fc640 1 --2- 172.21.15.107:0/3736696127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fbbe40 0x55de09fc39f0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fac8010f230 tx=0x7fac740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=862f581383c3c552 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:27.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.299+0000 7fac62ff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:45:27.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.302+0000 7fac62ff5640 1 -- 172.21.15.107:0/3736696127 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7400d040 con 0x55de09fbbe40 2024-05-09T08:45:27.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.302+0000 7fac8839b740 1 -- 172.21.15.107:0/3736696127 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac700bca30 con 0x55de09fbbe40 2024-05-09T08:45:27.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.302+0000 7fac62ff5640 1 -- 172.21.15.107:0/3736696127 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7fac7401e430 con 0x55de09fbbe40 2024-05-09T08:45:27.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.304+0000 7fac62ff5640 1 -- 172.21.15.107:0/3736696127 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fac7402c050 con 0x55de09fbbe40 2024-05-09T08:45:27.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.304+0000 7fac8839b740 10 client.18906 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:27.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.304+0000 7fac8839b740 1 -- 172.21.15.107:0/3736696127 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55de09fd5090 con 0x55de09fbbe40 2024-05-09T08:45:27.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.304+0000 7fac62ff5640 1 -- 172.21.15.107:0/3736696127 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac700bca30 con 0x55de09fbbe40 2024-05-09T08:45:27.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.305+0000 7fac62ff5640 1 -- 172.21.15.107:0/3736696127 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fac7401d070 con 0x55de09fbbe40 2024-05-09T08:45:27.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.305+0000 7fac8839b740 10 client.18906 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:27.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.305+0000 7fac8839b740 10 client.18906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:45:27.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.305+0000 7fac8839b740 20 client.18906 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:27.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.305+0000 7fac8839b740 1 --2- 172.21.15.107:0/3736696127 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55de09fd6a40 0x55de09fd8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.305+0000 7fac8839b740 4 client.18906 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:45:27.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.305+0000 7fac8839b740 1 -- 172.21.15.107:0/3736696127 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de09f2e370 con 0x55de09fd6a40 2024-05-09T08:45:27.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.306+0000 7fac7effd640 1 --2- 172.21.15.107:0/3736696127 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55de09fd6a40 0x55de09fd8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:27.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.307+0000 7fac7effd640 1 --2- 172.21.15.107:0/3736696127 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55de09fd6a40 0x55de09fd8ee0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4f99aacaa5055e6 server_cookie=8069d71b47ae131 in_seq=0 out_seq=0 2024-05-09T08:45:27.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.307+0000 7fac62ff5640 10 client.18906 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:27.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.310+0000 7fac62ff5640 1 -- 172.21.15.107:0/3736696127 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55de09f2e370 con 0x55de09fd6a40 2024-05-09T08:45:27.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.310+0000 7fac62ff5640 10 client.18906 handle_command_reply: tid=0 2024-05-09T08:45:27.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.310+0000 7fac8839b740 1 client.18906 shutdown 2024-05-09T08:45:27.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.313+0000 7fac8839b740 1 -- 172.21.15.107:0/3736696127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fbbe40 msgr2=0x55de09fc39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:27.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.313+0000 7fac8839b740 1 --2- 172.21.15.107:0/3736696127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fbbe40 0x55de09fc39f0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fac8010f230 tx=0x7fac740046c0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.315+0000 7fac8839b740 1 -- 172.21.15.107:0/3736696127 shutdown_connections 2024-05-09T08:45:27.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.315+0000 7fac8839b740 1 --2- 172.21.15.107:0/3736696127 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55de09fd6a40 0x55de09fd8ee0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.315+0000 7fac8839b740 1 --2- 172.21.15.107:0/3736696127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fc6510 0x55de09fc3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.315+0000 7fac8839b740 1 --2- 172.21.15.107:0/3736696127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fbbe40 0x55de09fc39f0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.315+0000 7fac8839b740 1 --2- 172.21.15.107:0/3736696127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fbb520 0x55de09fc34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.315+0000 7fac8839b740 1 -- 172.21.15.107:0/3736696127 >> 172.21.15.107:0/3736696127 conn(0x55de09fb7840 msgr2=0x55de09fbd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:27.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.315+0000 7fac62ff5640 0 client.18906 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:27.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.317+0000 7fac8839b740 1 -- 172.21.15.107:0/3736696127 shutdown_connections 2024-05-09T08:45:27.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.318+0000 7fac8839b740 1 -- 172.21.15.107:0/3736696127 wait complete. 2024-05-09T08:45:27.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.318+0000 7fac8839b740 20 client.18906 trim_cache size 0 max 16384 2024-05-09T08:45:27.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.345+0000 7fac8839b740 1 Processor -- start 2024-05-09T08:45:27.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.347+0000 7fac8839b740 1 -- start start 2024-05-09T08:45:27.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.347+0000 7fac8839b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fbb520 0x55de09fc3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.347+0000 7fac8839b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fbbe40 0x55de09fd1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.347+0000 7fac8839b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fc6510 0x55de09fc52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.347+0000 7fac8839b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fac6c012910 con 0x55de09fbbe40 2024-05-09T08:45:27.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.347+0000 7fac8839b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fac6c0101e0 con 0x55de09fc6510 2024-05-09T08:45:27.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.347+0000 7fac8839b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fac6c00e030 con 0x55de09fbb520 2024-05-09T08:45:27.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.348+0000 7fac7e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fbbe40 0x55de09fd1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:27.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.348+0000 7fac7e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fbbe40 0x55de09fd1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56664/0 (socket says 172.21.15.107:56664) 2024-05-09T08:45:27.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.348+0000 7fac7e7fc640 1 -- 172.21.15.107:0/1926470312 learned_addr learned my addr 172.21.15.107:0/1926470312 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:27.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.348+0000 7fac7effd640 1 --2- 172.21.15.107:0/1926470312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fbb520 0x55de09fc3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:27.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.348+0000 7fac7f7fe640 1 --2- 172.21.15.107:0/1926470312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fc6510 0x55de09fc52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:27.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.348+0000 7fac7effd640 1 -- 172.21.15.107:0/1926470312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fc6510 msgr2=0x55de09fc52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:27.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.348+0000 7fac7effd640 1 --2- 172.21.15.107:0/1926470312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fc6510 0x55de09fc52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.348+0000 7fac7effd640 1 -- 172.21.15.107:0/1926470312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fbbe40 msgr2=0x55de09fd1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:27.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.348+0000 7fac7effd640 1 --2- 172.21.15.107:0/1926470312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fbbe40 0x55de09fd1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.348+0000 7fac7effd640 1 -- 172.21.15.107:0/1926470312 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de09fc1a00 con 0x55de09fbb520 2024-05-09T08:45:27.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.349+0000 7fac7effd640 1 --2- 172.21.15.107:0/1926470312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fbb520 0x55de09fc3d50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55de09fd9760 tx=0x7fac78009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9d0c19d096e1a2c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:27.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.354+0000 7fac64ff9640 1 -- 172.21.15.107:0/1926470312 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac78020020 con 0x55de09fbb520 2024-05-09T08:45:27.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.354+0000 7fac64ff9640 1 -- 172.21.15.107:0/1926470312 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac7801c840 con 0x55de09fbb520 2024-05-09T08:45:27.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.354+0000 7fac64ff9640 1 -- 172.21.15.107:0/1926470312 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7801a410 con 0x55de09fbb520 2024-05-09T08:45:27.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.357+0000 7fac8839b740 1 -- 172.21.15.107:0/1926470312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fbb520 msgr2=0x55de09fc3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:27.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.357+0000 7fac8839b740 1 --2- 172.21.15.107:0/1926470312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fbb520 0x55de09fc3d50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55de09fd9760 tx=0x7fac78009b50 comp rx=0 tx=0).stop 2024-05-09T08:45:27.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.358+0000 7fac8839b740 1 -- 172.21.15.107:0/1926470312 shutdown_connections 2024-05-09T08:45:27.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.358+0000 7fac8839b740 1 --2- 172.21.15.107:0/1926470312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fc6510 0x55de09fc52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.358+0000 7fac8839b740 1 --2- 172.21.15.107:0/1926470312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fbbe40 0x55de09fd1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.358+0000 7fac8839b740 1 --2- 172.21.15.107:0/1926470312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fbb520 0x55de09fc3d50 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.358+0000 7fac8839b740 1 -- 172.21.15.107:0/1926470312 >> 172.21.15.107:0/1926470312 conn(0x55de09fb7840 msgr2=0x55de09fc4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:27.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.360+0000 7fac8839b740 1 -- 172.21.15.107:0/1926470312 shutdown_connections 2024-05-09T08:45:27.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.360+0000 7fac8839b740 1 -- 172.21.15.107:0/1926470312 wait complete. 2024-05-09T08:45:27.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.360+0000 7fac8839b740 1 -- start start 2024-05-09T08:45:27.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.360+0000 7fac8839b740 1 Processor -- start 2024-05-09T08:45:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.361+0000 7fac8839b740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fbb520 0x55de09fce520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.361+0000 7fac8839b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fbbe40 0x55de09fc9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.361+0000 7fac8839b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fc6510 0x55de09fc9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.361+0000 7fac8839b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fac6c00e6a0 con 0x55de09fc6510 2024-05-09T08:45:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.361+0000 7fac8839b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fac6c00fb10 con 0x55de09fbbe40 2024-05-09T08:45:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.361+0000 7fac8839b740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fac6c00ed70 con 0x55de09fbb520 2024-05-09T08:45:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.361+0000 7fac8839b740 10 client.0 _ll_register_callbacks cb 0x7fac7400be60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.361+0000 7fac7e7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fbbe40 0x55de09fc9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.361+0000 7fac7e7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fbbe40 0x55de09fc9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44424/0 (socket says 172.21.15.107:44424) 2024-05-09T08:45:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.361+0000 7fac7e7fc640 1 -- 172.21.15.107:0/2190217661 learned_addr learned my addr 172.21.15.107:0/2190217661 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.362+0000 7fac7e7fc640 1 -- 172.21.15.107:0/2190217661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fbb520 msgr2=0x55de09fce520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:45:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.362+0000 7fac7e7fc640 1 --2- 172.21.15.107:0/2190217661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fbb520 0x55de09fce520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.362+0000 7fac7e7fc640 1 -- 172.21.15.107:0/2190217661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fc6510 msgr2=0x55de09fc9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:27.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.362+0000 7fac7e7fc640 1 --2- 172.21.15.107:0/2190217661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fc6510 0x55de09fc9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.362+0000 7fac7e7fc640 1 -- 172.21.15.107:0/2190217661 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de09fca670 con 0x55de09fbbe40 2024-05-09T08:45:27.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.362+0000 7fac7e7fc640 1 --2- 172.21.15.107:0/2190217661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fbbe40 0x55de09fc9890 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fac7400cb80 tx=0x7fac7400cbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bf1915dd137cf8c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:27.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.363+0000 7fac62ff5640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:45:27.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.363+0000 7fac62ff5640 1 -- 172.21.15.107:0/2190217661 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7401c950 con 0x55de09fbbe40 2024-05-09T08:45:27.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.363+0000 7fac62ff5640 1 -- 172.21.15.107:0/2190217661 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7fac7401caf0 con 0x55de09fbbe40 2024-05-09T08:45:27.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.366+0000 7fac8839b740 1 -- 172.21.15.107:0/2190217661 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac7801c840 con 0x55de09fbbe40 2024-05-09T08:45:27.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.375+0000 7fac62ff5640 1 -- 172.21.15.107:0/2190217661 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fac74004040 con 0x55de09fbbe40 2024-05-09T08:45:27.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.376+0000 7fac8839b740 10 client.27851 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:27.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.376+0000 7fac8839b740 1 -- 172.21.15.107:0/2190217661 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55de0a03d0d0 con 0x55de09fbbe40 2024-05-09T08:45:27.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.377+0000 7fac62ff5640 1 -- 172.21.15.107:0/2190217661 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac7801c840 con 0x55de09fbbe40 2024-05-09T08:45:27.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.377+0000 7fac62ff5640 1 -- 172.21.15.107:0/2190217661 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fac74010030 con 0x55de09fbbe40 2024-05-09T08:45:27.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.378+0000 7fac8839b740 10 client.27851 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:27.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.378+0000 7fac8839b740 10 client.27851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:45:27.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.378+0000 7fac8839b740 20 client.27851 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:27.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.378+0000 7fac8839b740 1 --2- 172.21.15.107:0/2190217661 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55de0a03d880 0x55de0a03fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:27.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.378+0000 7fac8839b740 4 client.27851 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:45:27.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.378+0000 7fac8839b740 1 -- 172.21.15.107:0/2190217661 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55de09f286c0 con 0x55de0a03d880 2024-05-09T08:45:27.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.378+0000 7fac7effd640 1 --2- 172.21.15.107:0/2190217661 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55de0a03d880 0x55de0a03fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:27.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.379+0000 7fac62ff5640 10 client.27851 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:27.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.379+0000 7fac7effd640 1 --2- 172.21.15.107:0/2190217661 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55de0a03d880 0x55de0a03fd00 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27ac32fe95dfa50f server_cookie=11f288024bb4ec49 in_seq=0 out_seq=0 2024-05-09T08:45:27.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.380+0000 7fac62ff5640 1 -- 172.21.15.107:0/2190217661 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55de09f286c0 con 0x55de0a03d880 2024-05-09T08:45:27.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.380+0000 7fac62ff5640 10 client.27851 handle_command_reply: tid=0 2024-05-09T08:45:27.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.383+0000 7fac8839b740 1 client.27851 shutdown 2024-05-09T08:45:27.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.384+0000 7fac8839b740 1 -- 172.21.15.107:0/2190217661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fbbe40 msgr2=0x55de09fc9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:27.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.384+0000 7fac8839b740 1 --2- 172.21.15.107:0/2190217661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fbbe40 0x55de09fc9890 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fac7400cb80 tx=0x7fac7400cbb0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.384+0000 7fac8839b740 1 -- 172.21.15.107:0/2190217661 shutdown_connections 2024-05-09T08:45:27.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.384+0000 7fac8839b740 1 --2- 172.21.15.107:0/2190217661 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55de0a03d880 0x55de0a03fd00 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.384+0000 7fac8839b740 1 --2- 172.21.15.107:0/2190217661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55de09fc6510 0x55de09fc9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.384+0000 7fac8839b740 1 --2- 172.21.15.107:0/2190217661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55de09fbbe40 0x55de09fc9890 secure :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fac7400cb80 tx=0x7fac7400cbb0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.384+0000 7fac8839b740 1 --2- 172.21.15.107:0/2190217661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55de09fbb520 0x55de09fce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.384+0000 7fac8839b740 1 -- 172.21.15.107:0/2190217661 >> 172.21.15.107:0/2190217661 conn(0x55de09fb7840 msgr2=0x55de09fb69c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:27.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.385+0000 7fac62ff5640 0 client.27851 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:27.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.386+0000 7fac8839b740 1 -- 172.21.15.107:0/2190217661 shutdown_connections 2024-05-09T08:45:27.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.387+0000 7fac8839b740 1 -- 172.21.15.107:0/2190217661 wait complete. 2024-05-09T08:45:27.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.387+0000 7fac8839b740 20 client.27851 trim_cache size 0 max 16384 2024-05-09T08:45:27.388 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:45:27.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.391+0000 7fac7ffff640 1 -- 172.21.15.107:0/85128262 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fac680a2d20 msgr2=0x7fac680a51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:27.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.391+0000 7fac7ffff640 1 --2- 172.21.15.107:0/85128262 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fac680a2d20 0x7fac680a51e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55de09fbcae0 tx=0x7fac74008dd0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.391+0000 7fac7ffff640 1 -- 172.21.15.107:0/85128262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8010fed0 msgr2=0x7fac801a6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:27.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.391+0000 7fac7ffff640 1 --2- 172.21.15.107:0/85128262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8010fed0 0x7fac801a6ad0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fac70009980 tx=0x7fac70030ef0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.392+0000 7fac7ffff640 1 -- 172.21.15.107:0/85128262 shutdown_connections 2024-05-09T08:45:27.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.392+0000 7fac7ffff640 1 --2- 172.21.15.107:0/85128262 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fac680a2d20 0x7fac680a51e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.392+0000 7fac7ffff640 1 --2- 172.21.15.107:0/85128262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8010fed0 0x7fac801a6ad0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.392+0000 7fac7ffff640 1 --2- 172.21.15.107:0/85128262 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac801092b0 0x7fac801ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.392+0000 7fac7ffff640 1 --2- 172.21.15.107:0/85128262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80108830 0x7fac801ac920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:27.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.392+0000 7fac7ffff640 1 -- 172.21.15.107:0/85128262 >> 172.21.15.107:0/85128262 conn(0x7fac8006c7b0 msgr2=0x7fac80070490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:27.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.392+0000 7fac7ffff640 1 -- 172.21.15.107:0/85128262 shutdown_connections 2024-05-09T08:45:27.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:27.392+0000 7fac7ffff640 1 -- 172.21.15.107:0/85128262 wait complete. 2024-05-09T08:45:28.412 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:45:28.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:45:28.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:45:28.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.662+0000 7fe2de980640 1 -- 172.21.15.107:0/2065387601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2d810e4c0 msgr2=0x7fe2d811d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:28.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.662+0000 7fe2de980640 1 --2- 172.21.15.107:0/2065387601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2d810e4c0 0x7fe2d811d020 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009cd0 tx=0x7fe2d002a930 comp rx=0 tx=0).stop 2024-05-09T08:45:28.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.663+0000 7fe2de980640 1 -- 172.21.15.107:0/2065387601 shutdown_connections 2024-05-09T08:45:28.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.663+0000 7fe2de980640 1 --2- 172.21.15.107:0/2065387601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2d810e4c0 0x7fe2d811d020 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.663+0000 7fe2de980640 1 --2- 172.21.15.107:0/2065387601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2d810d910 0x7fe2d810dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.663+0000 7fe2de980640 1 --2- 172.21.15.107:0/2065387601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2d810fe40 0x7fe2d8110220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.663+0000 7fe2de980640 1 -- 172.21.15.107:0/2065387601 >> 172.21.15.107:0/2065387601 conn(0x7fe2d806c710 msgr2=0x7fe2d806cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:28.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.663+0000 7fe2de980640 1 -- 172.21.15.107:0/2065387601 shutdown_connections 2024-05-09T08:45:28.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.663+0000 7fe2de980640 1 -- 172.21.15.107:0/2065387601 wait complete. 2024-05-09T08:45:28.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.664+0000 7fe2de980640 1 Processor -- start 2024-05-09T08:45:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.664+0000 7fe2de980640 1 -- start start 2024-05-09T08:45:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.665+0000 7fe2de980640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2d810d910 0x7fe2d81ac830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.665+0000 7fe2de980640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2d810e4c0 0x7fe2d81acd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.665+0000 7fe2de980640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2d810fe40 0x7fe2d81a69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.665+0000 7fe2de980640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d81206a0 con 0x7fe2d810d910 2024-05-09T08:45:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.665+0000 7fe2de980640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d8120520 con 0x7fe2d810fe40 2024-05-09T08:45:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.665+0000 7fe2de980640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d8120820 con 0x7fe2d810e4c0 2024-05-09T08:45:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.665+0000 7fe2dd17d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2d810e4c0 0x7fe2d81acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.665+0000 7fe2dd97e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2d810d910 0x7fe2d81ac830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.665+0000 7fe2dd17d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2d810e4c0 0x7fe2d81acd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:50574/0 (socket says 172.21.15.107:50574) 2024-05-09T08:45:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.665+0000 7fe2dd17d640 1 -- 172.21.15.107:0/372618943 learned_addr learned my addr 172.21.15.107:0/372618943 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.665+0000 7fe2de17f640 1 --2- 172.21.15.107:0/372618943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2d810fe40 0x7fe2d81a69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.665+0000 7fe2dd97e640 1 -- 172.21.15.107:0/372618943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2d810e4c0 msgr2=0x7fe2d81acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.665+0000 7fe2dd97e640 1 --2- 172.21.15.107:0/372618943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2d810e4c0 0x7fe2d81acd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.665+0000 7fe2dd97e640 1 -- 172.21.15.107:0/372618943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2d810fe40 msgr2=0x7fe2d81a69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.665+0000 7fe2dd97e640 1 --2- 172.21.15.107:0/372618943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2d810fe40 0x7fe2d81a69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.665+0000 7fe2dd97e640 1 -- 172.21.15.107:0/372618943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d81a7160 con 0x7fe2d810d910 2024-05-09T08:45:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.666+0000 7fe2dd17d640 1 --2- 172.21.15.107:0/372618943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2d810e4c0 0x7fe2d81acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:45:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.666+0000 7fe2dd97e640 1 --2- 172.21.15.107:0/372618943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2d810d910 0x7fe2d81ac830 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400d6d0 tx=0x7fe2d400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.666+0000 7fe2ceffd640 1 -- 172.21.15.107:0/372618943 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4017070 con 0x7fe2d810d910 2024-05-09T08:45:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.666+0000 7fe2de980640 1 -- 172.21.15.107:0/372618943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2d81a7450 con 0x7fe2d810d910 2024-05-09T08:45:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.666+0000 7fe2ceffd640 1 -- 172.21.15.107:0/372618943 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2d4009820 con 0x7fe2d810d910 2024-05-09T08:45:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.666+0000 7fe2ceffd640 1 -- 172.21.15.107:0/372618943 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4004c40 con 0x7fe2d810d910 2024-05-09T08:45:28.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.666+0000 7fe2de980640 1 -- 172.21.15.107:0/372618943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2d8079a90 con 0x7fe2d810d910 2024-05-09T08:45:28.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.671+0000 7fe2e29c5740 1 Processor -- start 2024-05-09T08:45:28.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.675+0000 7fe2e29c5740 1 -- start start 2024-05-09T08:45:28.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.676+0000 7fe2e29c5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6e2c0 0x55fa69a6c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.676+0000 7fe2e29c5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6cbd0 0x55fa69a6d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.676+0000 7fe2e29c5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a77950 0x55fa69a79d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.676+0000 7fe2e29c5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fa699d1990 con 0x55fa69a77950 2024-05-09T08:45:28.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.676+0000 7fe2e29c5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55fa699795b0 con 0x55fa69a6e2c0 2024-05-09T08:45:28.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.676+0000 7fe2e29c5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55fa699cd950 con 0x55fa69a6cbd0 2024-05-09T08:45:28.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.676+0000 7fe2de17f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6e2c0 0x55fa69a6c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.676+0000 7fe2de17f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6e2c0 0x55fa69a6c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44456/0 (socket says 172.21.15.107:44456) 2024-05-09T08:45:28.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.676+0000 7fe2de17f640 1 -- 172.21.15.107:0/367698780 learned_addr learned my addr 172.21.15.107:0/367698780 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:28.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.677+0000 7fe2dd17d640 1 --2- 172.21.15.107:0/367698780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a77950 0x55fa69a79d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.677+0000 7fe2dd97e640 1 --2- 172.21.15.107:0/367698780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6cbd0 0x55fa69a6d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.677+0000 7fe2ceffd640 1 -- 172.21.15.107:0/372618943 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe2d40099d0 con 0x7fe2d810d910 2024-05-09T08:45:28.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.677+0000 7fe2dd97e640 1 -- 172.21.15.107:0/367698780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6e2c0 msgr2=0x55fa69a6c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:28.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.677+0000 7fe2dd97e640 1 --2- 172.21.15.107:0/367698780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6e2c0 0x55fa69a6c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.677+0000 7fe2dd97e640 1 -- 172.21.15.107:0/367698780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a77950 msgr2=0x55fa69a79d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:28.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.677+0000 7fe2dd97e640 1 --2- 172.21.15.107:0/367698780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a77950 0x55fa69a79d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.677+0000 7fe2dd97e640 1 -- 172.21.15.107:0/367698780 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa697ad7e0 con 0x55fa69a6cbd0 2024-05-09T08:45:28.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.678+0000 7fe2ceffd640 1 --2- 172.21.15.107:0/372618943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2c40a2e00 0x7fe2c40a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.678+0000 7fe2ceffd640 1 -- 172.21.15.107:0/372618943 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7fe2d40ceff0 con 0x7fe2d810d910 2024-05-09T08:45:28.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.679+0000 7fe2dd97e640 1 --2- 172.21.15.107:0/367698780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6cbd0 0x55fa69a6d050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe2d40cc2e0 tx=0x7fe2d40d7890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa58d4f852ac8c43 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:28.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.679+0000 7fe2c3fff640 1 -- 172.21.15.107:0/367698780 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4020070 con 0x55fa69a6cbd0 2024-05-09T08:45:28.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.680+0000 7fe2de17f640 1 --2- 172.21.15.107:0/372618943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2c40a2e00 0x7fe2c40a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.680+0000 7fe2c3fff640 1 -- 172.21.15.107:0/367698780 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2d4009d60 con 0x55fa69a6cbd0 2024-05-09T08:45:28.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.680+0000 7fe2c3fff640 1 -- 172.21.15.107:0/367698780 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d409aca0 con 0x55fa69a6cbd0 2024-05-09T08:45:28.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.681+0000 7fe2e29c5740 1 -- 172.21.15.107:0/367698780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6cbd0 msgr2=0x55fa69a6d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:28.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.681+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/367698780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6cbd0 0x55fa69a6d050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe2d40cc2e0 tx=0x7fe2d40d7890 comp rx=0 tx=0).stop 2024-05-09T08:45:28.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.681+0000 7fe2de17f640 1 --2- 172.21.15.107:0/372618943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2c40a2e00 0x7fe2c40a52c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0006010 tx=0x7fe2d0005da0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:28.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.681+0000 7fe2e29c5740 1 -- 172.21.15.107:0/367698780 shutdown_connections 2024-05-09T08:45:28.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.681+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/367698780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a77950 0x55fa69a79d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.681+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/367698780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6cbd0 0x55fa69a6d050 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.681+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/367698780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6e2c0 0x55fa69a6c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.681+0000 7fe2e29c5740 1 -- 172.21.15.107:0/367698780 >> 172.21.15.107:0/367698780 conn(0x55fa69a350a0 msgr2=0x55fa69a69130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:28.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.682+0000 7fe2e29c5740 1 -- 172.21.15.107:0/367698780 shutdown_connections 2024-05-09T08:45:28.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.682+0000 7fe2e29c5740 1 -- 172.21.15.107:0/367698780 wait complete. 2024-05-09T08:45:28.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.682+0000 7fe2e29c5740 1 -- start start 2024-05-09T08:45:28.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.682+0000 7fe2e29c5740 1 Processor -- start 2024-05-09T08:45:28.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.683+0000 7fe2e29c5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6cbd0 0x55fa69a74880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.683+0000 7fe2e29c5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6e2c0 0x55fa69a74dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.683+0000 7fe2e29c5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a77950 0x55fa69a75300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.683+0000 7fe2dd97e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6cbd0 0x55fa69a74880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.683+0000 7fe2dd97e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6cbd0 0x55fa69a74880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44470/0 (socket says 172.21.15.107:44470) 2024-05-09T08:45:28.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.683+0000 7fe2dd17d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6e2c0 0x55fa69a74dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.683+0000 7fe2dd17d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6e2c0 0x55fa69a74dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:50594/0 (socket says 172.21.15.107:50594) 2024-05-09T08:45:28.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.683+0000 7fe2dd97e640 1 -- 172.21.15.107:0/1195265343 learned_addr learned my addr 172.21.15.107:0/1195265343 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:28.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.683+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1195265343 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fa69a7a7e0 con 0x55fa69a77950 2024-05-09T08:45:28.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.683+0000 7fe2de17f640 1 --2- 172.21.15.107:0/1195265343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a77950 0x55fa69a75300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.683+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1195265343 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55fa6998cb40 con 0x55fa69a6cbd0 2024-05-09T08:45:28.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.683+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1195265343 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55fa69a7a960 con 0x55fa69a6e2c0 2024-05-09T08:45:28.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.683+0000 7fe2e29c5740 10 client.0 _ll_register_callbacks cb 0x55fa699810f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:28.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.684+0000 7fe2dd97e640 1 -- 172.21.15.107:0/1195265343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6e2c0 msgr2=0x55fa69a74dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:28.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.684+0000 7fe2dd97e640 1 --2- 172.21.15.107:0/1195265343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6e2c0 0x55fa69a74dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.684+0000 7fe2dd97e640 1 -- 172.21.15.107:0/1195265343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a77950 msgr2=0x55fa69a75300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:28.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.684+0000 7fe2dd97e640 1 --2- 172.21.15.107:0/1195265343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a77950 0x55fa69a75300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.684+0000 7fe2dd97e640 1 -- 172.21.15.107:0/1195265343 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa6980df70 con 0x55fa69a6cbd0 2024-05-09T08:45:28.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.685+0000 7fe2dd97e640 1 --2- 172.21.15.107:0/1195265343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6cbd0 0x55fa69a74880 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe2d409ae60 tx=0x7fe2d4005bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b78b8912f57d21f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:28.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.685+0000 7fe2c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:45:28.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.688+0000 7fe2c17fa640 1 -- 172.21.15.107:0/1195265343 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4020070 con 0x55fa69a6cbd0 2024-05-09T08:45:28.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.688+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1195265343 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fa699df0c0 con 0x55fa69a6cbd0 2024-05-09T08:45:28.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.688+0000 7fe2c17fa640 1 -- 172.21.15.107:0/1195265343 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7fe2d40cc560 con 0x55fa69a6cbd0 2024-05-09T08:45:28.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.690+0000 7fe2c17fa640 1 -- 172.21.15.107:0/1195265343 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe2d409aac0 con 0x55fa69a6cbd0 2024-05-09T08:45:28.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.690+0000 7fe2e29c5740 10 client.27863 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:28.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.690+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1195265343 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fa696a5170 con 0x55fa69a6cbd0 2024-05-09T08:45:28.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.690+0000 7fe2c17fa640 1 -- 172.21.15.107:0/1195265343 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55fa699df0c0 con 0x55fa69a6cbd0 2024-05-09T08:45:28.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.691+0000 7fe2c17fa640 1 -- 172.21.15.107:0/1195265343 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe2d40977d0 con 0x55fa69a6cbd0 2024-05-09T08:45:28.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.691+0000 7fe2e29c5740 10 client.27863 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:28.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.691+0000 7fe2e29c5740 10 client.27863 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:45:28.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.691+0000 7fe2e29c5740 20 client.27863 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:28.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.691+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/1195265343 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fa69a878c0 0x55fa69a89d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.691+0000 7fe2e29c5740 4 client.27863 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:45:28.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.691+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1195265343 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa699d9410 con 0x55fa69a878c0 2024-05-09T08:45:28.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.692+0000 7fe2dd17d640 1 --2- 172.21.15.107:0/1195265343 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fa69a878c0 0x55fa69a89d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.692+0000 7fe2dd17d640 1 --2- 172.21.15.107:0/1195265343 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fa69a878c0 0x55fa69a89d30 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2186de5e34c706dc server_cookie=da860753c0818aec in_seq=0 out_seq=0 2024-05-09T08:45:28.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.692+0000 7fe2c17fa640 10 client.27863 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:28.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.693+0000 7fe2c17fa640 1 -- 172.21.15.107:0/1195265343 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fa699d9410 con 0x55fa69a878c0 2024-05-09T08:45:28.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.693+0000 7fe2c17fa640 10 client.27863 handle_command_reply: tid=0 2024-05-09T08:45:28.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.693+0000 7fe2e29c5740 1 client.27863 shutdown 2024-05-09T08:45:28.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.693+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1195265343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6cbd0 msgr2=0x55fa69a74880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:28.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.693+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/1195265343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6cbd0 0x55fa69a74880 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe2d409ae60 tx=0x7fe2d4005bf0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.693+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1195265343 shutdown_connections 2024-05-09T08:45:28.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.693+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/1195265343 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fa69a878c0 0x55fa69a89d30 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.693+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/1195265343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a77950 0x55fa69a75300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.693+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/1195265343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6e2c0 0x55fa69a74dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.693+0000 7fe2c17fa640 0 client.27863 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:28.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.693+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/1195265343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6cbd0 0x55fa69a74880 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.693+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1195265343 >> 172.21.15.107:0/1195265343 conn(0x55fa69a350a0 msgr2=0x55fa69a6ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:28.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.694+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1195265343 shutdown_connections 2024-05-09T08:45:28.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.694+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1195265343 wait complete. 2024-05-09T08:45:28.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.694+0000 7fe2e29c5740 20 client.27863 trim_cache size 0 max 16384 2024-05-09T08:45:28.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.713+0000 7fe2e29c5740 1 Processor -- start 2024-05-09T08:45:28.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.713+0000 7fe2e29c5740 1 -- start start 2024-05-09T08:45:28.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2e29c5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a6cbd0 0x55fa69a68070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2e29c5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6e2c0 0x55fa69a836e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2e29c5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a77950 0x55fa69a74f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2dd97e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a6cbd0 0x55fa69a68070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2dd97e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a6cbd0 0x55fa69a68070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56712/0 (socket says 172.21.15.107:56712) 2024-05-09T08:45:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2dd97e640 1 -- 172.21.15.107:0/1681915028 learned_addr learned my addr 172.21.15.107:0/1681915028 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1681915028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d81d1360 con 0x55fa69a6cbd0 2024-05-09T08:45:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1681915028 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d8084000 con 0x55fa69a77950 2024-05-09T08:45:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2dd17d640 1 --2- 172.21.15.107:0/1681915028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6e2c0 0x55fa69a836e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1681915028 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d8068570 con 0x55fa69a6e2c0 2024-05-09T08:45:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2de17f640 1 --2- 172.21.15.107:0/1681915028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a77950 0x55fa69a74f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2dd17d640 1 -- 172.21.15.107:0/1681915028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a77950 msgr2=0x55fa69a74f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2dd17d640 1 --2- 172.21.15.107:0/1681915028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a77950 0x55fa69a74f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2dd17d640 1 -- 172.21.15.107:0/1681915028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a6cbd0 msgr2=0x55fa69a68070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2dd17d640 1 --2- 172.21.15.107:0/1681915028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a6cbd0 0x55fa69a68070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2dd17d640 1 -- 172.21.15.107:0/1681915028 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa69a690c0 con 0x55fa69a6e2c0 2024-05-09T08:45:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.714+0000 7fe2dd97e640 1 --2- 172.21.15.107:0/1681915028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a6cbd0 0x55fa69a68070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:45:28.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.715+0000 7fe2dd17d640 1 --2- 172.21.15.107:0/1681915028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6e2c0 0x55fa69a836e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x55fa69a6a2c0 tx=0x7fe2c80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed5f948a86a79528 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:28.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.715+0000 7fe2c37fe640 1 -- 172.21.15.107:0/1681915028 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8023030 con 0x55fa69a6e2c0 2024-05-09T08:45:28.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.715+0000 7fe2c37fe640 1 -- 172.21.15.107:0/1681915028 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2c8012020 con 0x55fa69a6e2c0 2024-05-09T08:45:28.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.715+0000 7fe2c37fe640 1 -- 172.21.15.107:0/1681915028 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c801f920 con 0x55fa69a6e2c0 2024-05-09T08:45:28.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.717+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1681915028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6e2c0 msgr2=0x55fa69a836e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:28.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.717+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/1681915028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6e2c0 0x55fa69a836e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x55fa69a6a2c0 tx=0x7fe2c80046c0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.717+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1681915028 shutdown_connections 2024-05-09T08:45:28.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.717+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/1681915028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a77950 0x55fa69a74f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.717+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/1681915028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6e2c0 0x55fa69a836e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.717+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/1681915028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a6cbd0 0x55fa69a68070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.717+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1681915028 >> 172.21.15.107:0/1681915028 conn(0x55fa69a350a0 msgr2=0x55fa69a74840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.717+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1681915028 shutdown_connections 2024-05-09T08:45:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.717+0000 7fe2e29c5740 1 -- 172.21.15.107:0/1681915028 wait complete. 2024-05-09T08:45:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.717+0000 7fe2e29c5740 1 -- start start 2024-05-09T08:45:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.717+0000 7fe2e29c5740 1 Processor -- start 2024-05-09T08:45:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2e29c5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6cbd0 0x55fa69a7fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2e29c5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6e2c0 0x55fa69a80130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2e29c5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a77950 0x55fa69a7cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2e29c5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d80824c0 con 0x55fa69a77950 2024-05-09T08:45:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2e29c5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d8083930 con 0x55fa69a6e2c0 2024-05-09T08:45:28.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2e29c5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d8082b90 con 0x55fa69a6cbd0 2024-05-09T08:45:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2e29c5740 10 client.0 _ll_register_callbacks cb 0x7fe2d0032f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2dd17d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6e2c0 0x55fa69a80130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2dd17d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6e2c0 0x55fa69a80130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44488/0 (socket says 172.21.15.107:44488) 2024-05-09T08:45:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2dd17d640 1 -- 172.21.15.107:0/221140142 learned_addr learned my addr 172.21.15.107:0/221140142 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2de17f640 1 --2- 172.21.15.107:0/221140142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a77950 0x55fa69a7cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2dd97e640 1 --2- 172.21.15.107:0/221140142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6cbd0 0x55fa69a7fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2dd17d640 1 -- 172.21.15.107:0/221140142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6cbd0 msgr2=0x55fa69a7fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2dd17d640 1 --2- 172.21.15.107:0/221140142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6cbd0 0x55fa69a7fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2dd17d640 1 -- 172.21.15.107:0/221140142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a77950 msgr2=0x55fa69a7cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2dd17d640 1 --2- 172.21.15.107:0/221140142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a77950 0x55fa69a7cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2dd17d640 1 -- 172.21.15.107:0/221140142 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa69a7d450 con 0x55fa69a6e2c0 2024-05-09T08:45:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2de17f640 1 --2- 172.21.15.107:0/221140142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a77950 0x55fa69a7cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:45:28.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.718+0000 7fe2dd97e640 1 --2- 172.21.15.107:0/221140142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6cbd0 0x55fa69a7fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:45:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.719+0000 7fe2dd17d640 1 --2- 172.21.15.107:0/221140142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6e2c0 0x55fa69a80130 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800b740 tx=0x7fe2c80045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6884f3423eec962 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.719+0000 7fe2c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:45:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.719+0000 7fe2c17fa640 1 -- 172.21.15.107:0/221140142 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8023070 con 0x55fa69a6e2c0 2024-05-09T08:45:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.719+0000 7fe2e29c5740 1 -- 172.21.15.107:0/221140142 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2c8012020 con 0x55fa69a6e2c0 2024-05-09T08:45:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.719+0000 7fe2c17fa640 1 -- 172.21.15.107:0/221140142 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7fe2c801f430 con 0x55fa69a6e2c0 2024-05-09T08:45:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.719+0000 7fe2c17fa640 1 -- 172.21.15.107:0/221140142 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe2c803a050 con 0x55fa69a6e2c0 2024-05-09T08:45:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.719+0000 7fe2e29c5740 10 client.27869 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.719+0000 7fe2e29c5740 1 -- 172.21.15.107:0/221140142 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fa69aeea10 con 0x55fa69a6e2c0 2024-05-09T08:45:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.719+0000 7fe2c17fa640 1 -- 172.21.15.107:0/221140142 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2c8012020 con 0x55fa69a6e2c0 2024-05-09T08:45:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.719+0000 7fe2c17fa640 1 -- 172.21.15.107:0/221140142 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe2c8004730 con 0x55fa69a6e2c0 2024-05-09T08:45:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.719+0000 7fe2e29c5740 10 client.27869 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.719+0000 7fe2e29c5740 10 client.27869 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:45:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.719+0000 7fe2e29c5740 20 client.27869 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:28.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.719+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/221140142 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fa69a878c0 0x55fa69af1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:28.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.719+0000 7fe2e29c5740 4 client.27869 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:45:28.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.719+0000 7fe2e29c5740 1 -- 172.21.15.107:0/221140142 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fa69956da0 con 0x55fa69a878c0 2024-05-09T08:45:28.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.720+0000 7fe2dd97e640 1 --2- 172.21.15.107:0/221140142 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fa69a878c0 0x55fa69af1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:28.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.720+0000 7fe2c17fa640 10 client.27869 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:28.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.720+0000 7fe2dd97e640 1 --2- 172.21.15.107:0/221140142 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fa69a878c0 0x55fa69af1180 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da7b0a311dcbb768 server_cookie=ce491ea38bd3e9d9 in_seq=0 out_seq=0 2024-05-09T08:45:28.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.721+0000 7fe2c17fa640 1 -- 172.21.15.107:0/221140142 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fa69956da0 con 0x55fa69a878c0 2024-05-09T08:45:28.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.721+0000 7fe2c17fa640 10 client.27869 handle_command_reply: tid=0 2024-05-09T08:45:28.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.721+0000 7fe2e29c5740 1 client.27869 shutdown 2024-05-09T08:45:28.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.722+0000 7fe2e29c5740 1 -- 172.21.15.107:0/221140142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6e2c0 msgr2=0x55fa69a80130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:28.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.722+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/221140142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6e2c0 0x55fa69a80130 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800b740 tx=0x7fe2c80045b0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.722+0000 7fe2e29c5740 1 -- 172.21.15.107:0/221140142 shutdown_connections 2024-05-09T08:45:28.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.722+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/221140142 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55fa69a878c0 0x55fa69af1180 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.722+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/221140142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa69a77950 0x55fa69a7cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.722+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/221140142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fa69a6e2c0 0x55fa69a80130 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.722+0000 7fe2e29c5740 1 --2- 172.21.15.107:0/221140142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fa69a6cbd0 0x55fa69a7fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.722+0000 7fe2e29c5740 1 -- 172.21.15.107:0/221140142 >> 172.21.15.107:0/221140142 conn(0x55fa69a350a0 msgr2=0x55fa69a6ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:28.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.722+0000 7fe2c17fa640 0 client.27869 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:28.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.722+0000 7fe2e29c5740 1 -- 172.21.15.107:0/221140142 shutdown_connections 2024-05-09T08:45:28.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.722+0000 7fe2e29c5740 1 -- 172.21.15.107:0/221140142 wait complete. 2024-05-09T08:45:28.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.723+0000 7fe2e29c5740 20 client.27869 trim_cache size 0 max 16384 2024-05-09T08:45:28.726 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:45:28.727 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:45:28.727 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "9a69fb35-17df-462b-9df8-d3955393209f", 2024-05-09T08:45:28.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:45:28.727 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:45:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.723+0000 7fe2de980640 1 -- 172.21.15.107:0/372618943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2c40a2e00 msgr2=0x7fe2c40a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.723+0000 7fe2de980640 1 --2- 172.21.15.107:0/372618943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2c40a2e00 0x7fe2c40a52c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0006010 tx=0x7fe2d0005da0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.723+0000 7fe2de980640 1 -- 172.21.15.107:0/372618943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2d810d910 msgr2=0x7fe2d81ac830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.723+0000 7fe2de980640 1 --2- 172.21.15.107:0/372618943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2d810d910 0x7fe2d81ac830 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400d6d0 tx=0x7fe2d400db90 comp rx=0 tx=0).stop 2024-05-09T08:45:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.724+0000 7fe2de980640 1 -- 172.21.15.107:0/372618943 shutdown_connections 2024-05-09T08:45:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.724+0000 7fe2de980640 1 --2- 172.21.15.107:0/372618943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe2c40a2e00 0x7fe2c40a52c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.724+0000 7fe2de980640 1 --2- 172.21.15.107:0/372618943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2d810fe40 0x7fe2d81a69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.724+0000 7fe2de980640 1 --2- 172.21.15.107:0/372618943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2d810e4c0 0x7fe2d81acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.724+0000 7fe2de980640 1 --2- 172.21.15.107:0/372618943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2d810d910 0x7fe2d81ac830 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.724+0000 7fe2de980640 1 -- 172.21.15.107:0/372618943 >> 172.21.15.107:0/372618943 conn(0x7fe2d806c710 msgr2=0x7fe2d8070b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.724+0000 7fe2de980640 1 -- 172.21.15.107:0/372618943 shutdown_connections 2024-05-09T08:45:28.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.724+0000 7fe2de980640 1 -- 172.21.15.107:0/372618943 wait complete. 2024-05-09T08:45:28.744 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:45:28.744 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:45:28.745 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:28 smithi107 ceph-mon[30122]: pgmap v378: 129 pgs: 10 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 112 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 9.6 MiB/s wr, 2.35k op/s 2024-05-09T08:45:28.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.994+0000 7feedb59e640 1 -- 172.21.15.107:0/3345562991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feedc10cff0 msgr2=0x7feedc079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:28.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.994+0000 7feedb59e640 1 --2- 172.21.15.107:0/3345562991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feedc10cff0 0x7feedc079ab0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7feed0009860 tx=0x7feed002a340 comp rx=0 tx=0).stop 2024-05-09T08:45:28.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.997+0000 7feedb59e640 1 -- 172.21.15.107:0/3345562991 shutdown_connections 2024-05-09T08:45:28.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.997+0000 7feedb59e640 1 --2- 172.21.15.107:0/3345562991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feedc10cff0 0x7feedc079ab0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.997+0000 7feedb59e640 1 --2- 172.21.15.107:0/3345562991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feedc109450 0x7feedc10c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.997+0000 7feedb59e640 1 --2- 172.21.15.107:0/3345562991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feedc108aa0 0x7feedc108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:28.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:28.997+0000 7feedb59e640 1 -- 172.21.15.107:0/3345562991 >> 172.21.15.107:0/3345562991 conn(0x7feedc06c7b0 msgr2=0x7feedc06cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:29.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.000+0000 7feedb59e640 1 -- 172.21.15.107:0/3345562991 shutdown_connections 2024-05-09T08:45:29.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.001+0000 7feedb59e640 1 -- 172.21.15.107:0/3345562991 wait complete. 2024-05-09T08:45:29.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.002+0000 7feedb59e640 1 Processor -- start 2024-05-09T08:45:29.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.002+0000 7feedb59e640 1 -- start start 2024-05-09T08:45:29.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.002+0000 7feedb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feedc108aa0 0x7feedc0844a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.002+0000 7feedb59e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feedc10cff0 0x7feedc07e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.002+0000 7feedb59e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feedc07ec00 0x7feedc1bb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.002+0000 7feedb59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feedc07ce70 con 0x7feedc108aa0 2024-05-09T08:45:29.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.002+0000 7feedb59e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feedc07ccf0 con 0x7feedc07ec00 2024-05-09T08:45:29.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.002+0000 7feedb59e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feedc07cff0 con 0x7feedc10cff0 2024-05-09T08:45:29.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.003+0000 7feeda59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feedc108aa0 0x7feedc0844a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:29.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.003+0000 7feeda59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feedc108aa0 0x7feedc0844a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56736/0 (socket says 172.21.15.107:56736) 2024-05-09T08:45:29.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.003+0000 7feeda59c640 1 -- 172.21.15.107:0/3098871200 learned_addr learned my addr 172.21.15.107:0/3098871200 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:29.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.003+0000 7feedad9d640 1 --2- 172.21.15.107:0/3098871200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feedc07ec00 0x7feedc1bb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:29.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.004+0000 7feeda59c640 1 -- 172.21.15.107:0/3098871200 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feedc10cff0 msgr2=0x7feedc07e560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:45:29.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.004+0000 7feeda59c640 1 --2- 172.21.15.107:0/3098871200 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feedc10cff0 0x7feedc07e560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.004+0000 7feeda59c640 1 -- 172.21.15.107:0/3098871200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feedc07ec00 msgr2=0x7feedc1bb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:29.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.004+0000 7feeda59c640 1 --2- 172.21.15.107:0/3098871200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feedc07ec00 0x7feedc1bb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.004+0000 7feeda59c640 1 -- 172.21.15.107:0/3098871200 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feedc07f3a0 con 0x7feedc108aa0 2024-05-09T08:45:29.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.004+0000 7feeda59c640 1 --2- 172.21.15.107:0/3098871200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feedc108aa0 0x7feedc0844a0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7feecc0097d0 tx=0x7feecc009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:29.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.005+0000 7feecb7fe640 1 -- 172.21.15.107:0/3098871200 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feecc018020 con 0x7feedc108aa0 2024-05-09T08:45:29.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.005+0000 7feecb7fe640 1 -- 172.21.15.107:0/3098871200 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feecc00ed60 con 0x7feedc108aa0 2024-05-09T08:45:29.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.005+0000 7feecb7fe640 1 -- 172.21.15.107:0/3098871200 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feecc013370 con 0x7feedc108aa0 2024-05-09T08:45:29.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.005+0000 7feedb59e640 1 -- 172.21.15.107:0/3098871200 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feedc1bbec0 con 0x7feedc108aa0 2024-05-09T08:45:29.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.005+0000 7feedb59e640 1 -- 172.21.15.107:0/3098871200 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feedc1bc3c0 con 0x7feedc108aa0 2024-05-09T08:45:29.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.010+0000 7feecb7fe640 1 -- 172.21.15.107:0/3098871200 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7feecc01b020 con 0x7feedc108aa0 2024-05-09T08:45:29.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.011+0000 7feecb7fe640 1 --2- 172.21.15.107:0/3098871200 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feec40a2f00 0x7feec40a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.013+0000 7feecb7fe640 1 -- 172.21.15.107:0/3098871200 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7feecc0d0620 con 0x7feedc108aa0 2024-05-09T08:45:29.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.013+0000 7feee3630740 1 Processor -- start 2024-05-09T08:45:29.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.013+0000 7feee3630740 1 -- start start 2024-05-09T08:45:29.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.013+0000 7feee3630740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8470 0x5576de7d8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.013+0000 7feee3630740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8d90 0x5576de7e2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.013+0000 7feee3630740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7e3460 0x5576de7e5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.013+0000 7feee3630740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5576de73dd00 con 0x5576de7d8470 2024-05-09T08:45:29.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.013+0000 7feee3630740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5576de6e54c0 con 0x5576de7e3460 2024-05-09T08:45:29.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.013+0000 7feee3630740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5576de739cc0 con 0x5576de7d8d90 2024-05-09T08:45:29.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.013+0000 7feed9d9b640 1 --2- 172.21.15.107:0/3098871200 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feec40a2f00 0x7feec40a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.014+0000 7feedad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7e3460 0x5576de7e5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.014+0000 7feedad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7e3460 0x5576de7e5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44518/0 (socket says 172.21.15.107:44518) 2024-05-09T08:45:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.015+0000 7feedad9d640 1 -- 172.21.15.107:0/1249335993 learned_addr learned my addr 172.21.15.107:0/1249335993 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.015+0000 7feeda59c640 1 --2- 172.21.15.107:0/1249335993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8470 0x5576de7d8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.015+0000 7feed9d9b640 1 --2- 172.21.15.107:0/1249335993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8d90 0x5576de7e2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.015+0000 7feeda59c640 1 -- 172.21.15.107:0/1249335993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8d90 msgr2=0x5576de7e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.015+0000 7feeda59c640 1 --2- 172.21.15.107:0/1249335993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8d90 0x5576de7e2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.015+0000 7feeda59c640 1 -- 172.21.15.107:0/1249335993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7e3460 msgr2=0x5576de7e5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.015+0000 7feeda59c640 1 --2- 172.21.15.107:0/1249335993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7e3460 0x5576de7e5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.015+0000 7feeda59c640 1 -- 172.21.15.107:0/1249335993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576de51b7d0 con 0x5576de7d8470 2024-05-09T08:45:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.015+0000 7feed9d9b640 1 --2- 172.21.15.107:0/1249335993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8d90 0x5576de7e2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:45:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.016+0000 7feedad9d640 1 --2- 172.21.15.107:0/1249335993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7e3460 0x5576de7e5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:45:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.016+0000 7feeda59c640 1 --2- 172.21.15.107:0/1249335993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8470 0x5576de7d8850 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7feecc0cd910 tx=0x7feecc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8543ad1a276687 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.016+0000 7feed9d9b640 1 --2- 172.21.15.107:0/3098871200 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feec40a2f00 0x7feec40a53c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7feedc07fcd0 tx=0x7feed4010a30 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.016+0000 7feec8ff9640 1 -- 172.21.15.107:0/1249335993 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feecc09c6e0 con 0x5576de7d8470 2024-05-09T08:45:29.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.016+0000 7feec8ff9640 1 -- 172.21.15.107:0/1249335993 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feecc0a0a90 con 0x5576de7d8470 2024-05-09T08:45:29.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.017+0000 7feee3630740 1 -- 172.21.15.107:0/1249335993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8470 msgr2=0x5576de7d8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:29.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.017+0000 7feee3630740 1 --2- 172.21.15.107:0/1249335993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8470 0x5576de7d8850 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7feecc0cd910 tx=0x7feecc004730 comp rx=0 tx=0).stop 2024-05-09T08:45:29.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.018+0000 7feee3630740 1 -- 172.21.15.107:0/1249335993 shutdown_connections 2024-05-09T08:45:29.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.018+0000 7feee3630740 1 --2- 172.21.15.107:0/1249335993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7e3460 0x5576de7e5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.018+0000 7feee3630740 1 --2- 172.21.15.107:0/1249335993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8d90 0x5576de7e2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.018+0000 7feee3630740 1 --2- 172.21.15.107:0/1249335993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8470 0x5576de7d8850 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.018+0000 7feee3630740 1 -- 172.21.15.107:0/1249335993 >> 172.21.15.107:0/1249335993 conn(0x5576de7d4790 msgr2=0x5576de7d4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:29.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.020+0000 7feee3630740 1 -- 172.21.15.107:0/1249335993 shutdown_connections 2024-05-09T08:45:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.022+0000 7feee3630740 1 -- 172.21.15.107:0/1249335993 wait complete. 2024-05-09T08:45:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.022+0000 7feee3630740 1 -- start start 2024-05-09T08:45:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.022+0000 7feee3630740 1 Processor -- start 2024-05-09T08:45:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.023+0000 7feee3630740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8470 0x5576de7e0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.023+0000 7feee3630740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8d90 0x5576de7e0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.023+0000 7feee3630740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7e3460 0x5576de7e0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.023+0000 7feee3630740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5576de7e62a0 con 0x5576de7d8d90 2024-05-09T08:45:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.023+0000 7feee3630740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5576de6f8b90 con 0x5576de7e3460 2024-05-09T08:45:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.023+0000 7feee3630740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5576de7e6420 con 0x5576de7d8470 2024-05-09T08:45:29.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.023+0000 7feee3630740 10 client.0 _ll_register_callbacks cb 0x5576de734da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.023+0000 7feeda59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8470 0x5576de7e0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.023+0000 7feeda59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8470 0x5576de7e0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:50624/0 (socket says 172.21.15.107:50624) 2024-05-09T08:45:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.023+0000 7feeda59c640 1 -- 172.21.15.107:0/4170718767 learned_addr learned my addr 172.21.15.107:0/4170718767 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.023+0000 7feedad9d640 1 --2- 172.21.15.107:0/4170718767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7e3460 0x5576de7e0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:29.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.023+0000 7feed9d9b640 1 --2- 172.21.15.107:0/4170718767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8d90 0x5576de7e0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.024+0000 7feeda59c640 1 -- 172.21.15.107:0/4170718767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7e3460 msgr2=0x5576de7e0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.024+0000 7feeda59c640 1 --2- 172.21.15.107:0/4170718767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7e3460 0x5576de7e0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.024+0000 7feeda59c640 1 -- 172.21.15.107:0/4170718767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8d90 msgr2=0x5576de7e0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.024+0000 7feeda59c640 1 --2- 172.21.15.107:0/4170718767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8d90 0x5576de7e0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.024+0000 7feeda59c640 1 -- 172.21.15.107:0/4170718767 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576de4513f0 con 0x5576de7d8470 2024-05-09T08:45:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.025+0000 7feeda59c640 1 --2- 172.21.15.107:0/4170718767 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8470 0x5576de7e0320 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feecc014a80 tx=0x7feecc014ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dec57eae74ed5ef0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.027+0000 7feec1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:45:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.031+0000 7feec1ffb640 1 -- 172.21.15.107:0/4170718767 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feecc0aa060 con 0x5576de7d8470 2024-05-09T08:45:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.031+0000 7feec1ffb640 1 -- 172.21.15.107:0/4170718767 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7feecc09bb30 con 0x5576de7d8470 2024-05-09T08:45:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.031+0000 7feee3630740 1 -- 172.21.15.107:0/4170718767 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feecc0a0a90 con 0x5576de7d8470 2024-05-09T08:45:29.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.032+0000 7feec1ffb640 1 -- 172.21.15.107:0/4170718767 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feecc0bb070 con 0x5576de7d8470 2024-05-09T08:45:29.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.033+0000 7feee3630740 10 client.28102 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:29.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.033+0000 7feee3630740 1 -- 172.21.15.107:0/4170718767 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5576de4110f0 con 0x5576de7d8470 2024-05-09T08:45:29.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.038+0000 7feec1ffb640 1 -- 172.21.15.107:0/4170718767 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feecc0a0a90 con 0x5576de7d8470 2024-05-09T08:45:29.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.038+0000 7feec1ffb640 1 -- 172.21.15.107:0/4170718767 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7feecc09b5a0 con 0x5576de7d8470 2024-05-09T08:45:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.039+0000 7feee3630740 10 client.28102 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.039+0000 7feee3630740 10 client.28102 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:45:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.039+0000 7feee3630740 20 client.28102 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.039+0000 7feee3630740 1 --2- 172.21.15.107:0/4170718767 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5576de7f3950 0x5576de7f5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.039+0000 7feee3630740 4 client.28102 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:45:29.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.039+0000 7feee3630740 1 -- 172.21.15.107:0/4170718767 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576de74b430 con 0x5576de7f3950 2024-05-09T08:45:29.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.042+0000 7feed9d9b640 1 --2- 172.21.15.107:0/4170718767 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5576de7f3950 0x5576de7f5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:29.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.046+0000 7feed9d9b640 1 --2- 172.21.15.107:0/4170718767 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5576de7f3950 0x5576de7f5d70 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cb7ea384a05b17a server_cookie=32e68cecb0c2795f in_seq=0 out_seq=0 2024-05-09T08:45:29.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.050+0000 7feec1ffb640 10 client.28102 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:29.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.050+0000 7feec1ffb640 1 -- 172.21.15.107:0/4170718767 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5576de74b430 con 0x5576de7f3950 2024-05-09T08:45:29.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.050+0000 7feec1ffb640 10 client.28102 handle_command_reply: tid=0 2024-05-09T08:45:29.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.050+0000 7feee3630740 1 client.28102 shutdown 2024-05-09T08:45:29.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.057+0000 7feee3630740 1 -- 172.21.15.107:0/4170718767 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8470 msgr2=0x5576de7e0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:29.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.057+0000 7feee3630740 1 --2- 172.21.15.107:0/4170718767 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8470 0x5576de7e0320 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feecc014a80 tx=0x7feecc014ed0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.061+0000 7feee3630740 1 -- 172.21.15.107:0/4170718767 shutdown_connections 2024-05-09T08:45:29.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.061+0000 7feee3630740 1 --2- 172.21.15.107:0/4170718767 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5576de7f3950 0x5576de7f5d70 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.061+0000 7feee3630740 1 --2- 172.21.15.107:0/4170718767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7e3460 0x5576de7e0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.061+0000 7feee3630740 1 --2- 172.21.15.107:0/4170718767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8d90 0x5576de7e0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.061+0000 7feee3630740 1 --2- 172.21.15.107:0/4170718767 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8470 0x5576de7e0320 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.061+0000 7feee3630740 1 -- 172.21.15.107:0/4170718767 >> 172.21.15.107:0/4170718767 conn(0x5576de7d4790 msgr2=0x5576de7d3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:29.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.061+0000 7feec1ffb640 0 client.28102 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:29.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.061+0000 7feee3630740 1 -- 172.21.15.107:0/4170718767 shutdown_connections 2024-05-09T08:45:29.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.061+0000 7feee3630740 1 -- 172.21.15.107:0/4170718767 wait complete. 2024-05-09T08:45:29.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.062+0000 7feee3630740 20 client.28102 trim_cache size 0 max 16384 2024-05-09T08:45:29.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:28 smithi157 ceph-mon[33614]: pgmap v378: 129 pgs: 10 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 112 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 9.6 MiB/s wr, 2.35k op/s 2024-05-09T08:45:29.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.112+0000 7feee3630740 1 Processor -- start 2024-05-09T08:45:29.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.112+0000 7feee3630740 1 -- start start 2024-05-09T08:45:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.112+0000 7feee3630740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8470 0x5576de7e0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.112+0000 7feee3630740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7d8d90 0x5576de7fb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.112+0000 7feee3630740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7e3460 0x5576de7fbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.113+0000 7feee3630740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feebc012900 con 0x5576de7e3460 2024-05-09T08:45:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.113+0000 7feee3630740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feebc0101d0 con 0x5576de7d8d90 2024-05-09T08:45:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.113+0000 7feee3630740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feebc00e020 con 0x5576de7d8470 2024-05-09T08:45:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.113+0000 7feed9d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7d8d90 0x5576de7fb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.113+0000 7feed9d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7d8d90 0x5576de7fb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44538/0 (socket says 172.21.15.107:44538) 2024-05-09T08:45:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.113+0000 7feed9d9b640 1 -- 172.21.15.107:0/885296214 learned_addr learned my addr 172.21.15.107:0/885296214 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.113+0000 7feeda59c640 1 --2- 172.21.15.107:0/885296214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8470 0x5576de7e0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.113+0000 7feed9d9b640 1 -- 172.21.15.107:0/885296214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8470 msgr2=0x5576de7e0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.113+0000 7feed9d9b640 1 --2- 172.21.15.107:0/885296214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8470 0x5576de7e0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.113+0000 7feed9d9b640 1 -- 172.21.15.107:0/885296214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7e3460 msgr2=0x5576de7fbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.113+0000 7feed9d9b640 1 --2- 172.21.15.107:0/885296214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7e3460 0x5576de7fbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.113+0000 7feed9d9b640 1 -- 172.21.15.107:0/885296214 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576de7c5930 con 0x5576de7d8d90 2024-05-09T08:45:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.114+0000 7feed9d9b640 1 --2- 172.21.15.107:0/885296214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7d8d90 0x5576de7fb660 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7feed4005fa0 tx=0x7feed40095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=734cb849967ca042 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:29.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.117+0000 7feec3fff640 1 -- 172.21.15.107:0/885296214 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed4004b20 con 0x5576de7d8d90 2024-05-09T08:45:29.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.117+0000 7feec3fff640 1 -- 172.21.15.107:0/885296214 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feed4003e20 con 0x5576de7d8d90 2024-05-09T08:45:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.118+0000 7feee3630740 1 -- 172.21.15.107:0/885296214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7d8d90 msgr2=0x5576de7fb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.118+0000 7feee3630740 1 --2- 172.21.15.107:0/885296214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7d8d90 0x5576de7fb660 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7feed4005fa0 tx=0x7feed40095c0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.118+0000 7feee3630740 1 -- 172.21.15.107:0/885296214 shutdown_connections 2024-05-09T08:45:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.118+0000 7feee3630740 1 --2- 172.21.15.107:0/885296214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7e3460 0x5576de7fbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.118+0000 7feee3630740 1 --2- 172.21.15.107:0/885296214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7d8d90 0x5576de7fb660 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.118+0000 7feee3630740 1 --2- 172.21.15.107:0/885296214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7d8470 0x5576de7e0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.118+0000 7feee3630740 1 -- 172.21.15.107:0/885296214 >> 172.21.15.107:0/885296214 conn(0x5576de7d4790 msgr2=0x5576de7d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:29.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:28 smithi110 ceph-mon[33622]: pgmap v378: 129 pgs: 10 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 112 active+clean; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 9.6 MiB/s wr, 2.35k op/s 2024-05-09T08:45:29.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.122+0000 7feee3630740 1 -- 172.21.15.107:0/885296214 shutdown_connections 2024-05-09T08:45:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.122+0000 7feee3630740 1 -- 172.21.15.107:0/885296214 wait complete. 2024-05-09T08:45:29.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.123+0000 7feee3630740 1 -- start start 2024-05-09T08:45:29.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.123+0000 7feee3630740 1 Processor -- start 2024-05-09T08:45:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.126+0000 7feee3630740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8470 0x5576de7eb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.126+0000 7feee3630740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7d8d90 0x5576de7ebc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.126+0000 7feee3630740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7e3460 0x5576de7e86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.126+0000 7feee3630740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feebc00e690 con 0x5576de7d8470 2024-05-09T08:45:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.126+0000 7feee3630740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feebc00fb00 con 0x5576de7d8d90 2024-05-09T08:45:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.126+0000 7feee3630740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feebc00ed60 con 0x5576de7e3460 2024-05-09T08:45:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.126+0000 7feee3630740 10 client.0 _ll_register_callbacks cb 0x7feed0035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:29.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.128+0000 7feedad9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7e3460 0x5576de7e86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:29.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.128+0000 7feed9d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7d8d90 0x5576de7ebc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:29.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.128+0000 7feed9d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7d8d90 0x5576de7ebc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44542/0 (socket says 172.21.15.107:44542) 2024-05-09T08:45:29.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.128+0000 7feed9d9b640 1 -- 172.21.15.107:0/2613627600 learned_addr learned my addr 172.21.15.107:0/2613627600 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:29.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.128+0000 7feeda59c640 1 --2- 172.21.15.107:0/2613627600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8470 0x5576de7eb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:29.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.128+0000 7feeda59c640 1 -- 172.21.15.107:0/2613627600 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7e3460 msgr2=0x5576de7e86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:29.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.128+0000 7feeda59c640 1 --2- 172.21.15.107:0/2613627600 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7e3460 0x5576de7e86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.129+0000 7feeda59c640 1 -- 172.21.15.107:0/2613627600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7d8d90 msgr2=0x5576de7ebc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:29.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.129+0000 7feeda59c640 1 --2- 172.21.15.107:0/2613627600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7d8d90 0x5576de7ebc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.129+0000 7feeda59c640 1 -- 172.21.15.107:0/2613627600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576de7e8d60 con 0x5576de7d8470 2024-05-09T08:45:29.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.129+0000 7feeda59c640 1 --2- 172.21.15.107:0/2613627600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8470 0x5576de7eb720 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7feecc09a990 tx=0x7feecc0cdc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aed313d28c51ebd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:29.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.129+0000 7feec1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:45:29.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.132+0000 7feec1ffb640 1 -- 172.21.15.107:0/2613627600 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feecc0aa040 con 0x5576de7d8470 2024-05-09T08:45:29.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.133+0000 7feee3630740 1 -- 172.21.15.107:0/2613627600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5576de745780 con 0x5576de7d8470 2024-05-09T08:45:29.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.133+0000 7feec1ffb640 1 -- 172.21.15.107:0/2613627600 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7feecc09bb30 con 0x5576de7d8470 2024-05-09T08:45:29.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.133+0000 7feec1ffb640 1 -- 172.21.15.107:0/2613627600 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feecc0b4040 con 0x5576de7d8470 2024-05-09T08:45:29.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.133+0000 7feee3630740 10 client.18942 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:29.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.134+0000 7feee3630740 1 -- 172.21.15.107:0/2613627600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5576de85a8c0 con 0x5576de7d8470 2024-05-09T08:45:29.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.135+0000 7feec1ffb640 1 -- 172.21.15.107:0/2613627600 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5576de745780 con 0x5576de7d8470 2024-05-09T08:45:29.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.135+0000 7feec1ffb640 1 -- 172.21.15.107:0/2613627600 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7feecc0bc3c0 con 0x5576de7d8470 2024-05-09T08:45:29.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.135+0000 7feee3630740 10 client.18942 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:29.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.135+0000 7feee3630740 10 client.18942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:45:29.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.135+0000 7feee3630740 20 client.18942 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:29.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.135+0000 7feee3630740 1 --2- 172.21.15.107:0/2613627600 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5576de85ada0 0x5576de85d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:29.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.135+0000 7feee3630740 4 client.18942 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:45:29.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.135+0000 7feee3630740 1 -- 172.21.15.107:0/2613627600 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5576de6c26f0 con 0x5576de85ada0 2024-05-09T08:45:29.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.142+0000 7feed9d9b640 1 --2- 172.21.15.107:0/2613627600 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5576de85ada0 0x5576de85d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:29.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.143+0000 7feed9d9b640 1 --2- 172.21.15.107:0/2613627600 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5576de85ada0 0x5576de85d190 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d4e4607067632bc server_cookie=40597b5b63fe81e8 in_seq=0 out_seq=0 2024-05-09T08:45:29.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.143+0000 7feec1ffb640 10 client.18942 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:29.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.146+0000 7feec1ffb640 1 -- 172.21.15.107:0/2613627600 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5576de6c26f0 con 0x5576de85ada0 2024-05-09T08:45:29.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.146+0000 7feec1ffb640 10 client.18942 handle_command_reply: tid=0 2024-05-09T08:45:29.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.146+0000 7feee3630740 1 client.18942 shutdown 2024-05-09T08:45:29.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.149+0000 7feee3630740 1 -- 172.21.15.107:0/2613627600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8470 msgr2=0x5576de7eb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:29.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.149+0000 7feee3630740 1 --2- 172.21.15.107:0/2613627600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8470 0x5576de7eb720 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7feecc09a990 tx=0x7feecc0cdc20 comp rx=0 tx=0).stop 2024-05-09T08:45:29.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.153+0000 7feee3630740 1 -- 172.21.15.107:0/2613627600 shutdown_connections 2024-05-09T08:45:29.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.153+0000 7feee3630740 1 --2- 172.21.15.107:0/2613627600 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5576de85ada0 0x5576de85d190 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.153+0000 7feee3630740 1 --2- 172.21.15.107:0/2613627600 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5576de7e3460 0x5576de7e86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.153+0000 7feee3630740 1 --2- 172.21.15.107:0/2613627600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5576de7d8d90 0x5576de7ebc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.153+0000 7feee3630740 1 --2- 172.21.15.107:0/2613627600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5576de7d8470 0x5576de7eb720 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.153+0000 7feee3630740 1 -- 172.21.15.107:0/2613627600 >> 172.21.15.107:0/2613627600 conn(0x5576de7d4790 msgr2=0x5576de7d3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:29.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.153+0000 7feec1ffb640 0 client.18942 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:29.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.155+0000 7feee3630740 1 -- 172.21.15.107:0/2613627600 shutdown_connections 2024-05-09T08:45:29.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.155+0000 7feee3630740 1 -- 172.21.15.107:0/2613627600 wait complete. 2024-05-09T08:45:29.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.159+0000 7feee3630740 20 client.18942 trim_cache size 0 max 16384 2024-05-09T08:45:29.163 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:45:29.163 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "scrub active (222 inodes in the stack)", 2024-05-09T08:45:29.163 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:45:29.163 INFO:teuthology.orchestra.run.smithi107.stdout: "9a69fb35-17df-462b-9df8-d3955393209f": { 2024-05-09T08:45:29.163 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:45:29.163 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "9a69fb35-17df-462b-9df8-d3955393209f", 2024-05-09T08:45:29.163 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:45:29.163 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:45:29.163 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:45:29.163 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:45:29.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.163+0000 7feedb59e640 1 -- 172.21.15.107:0/3098871200 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feec40a2f00 msgr2=0x7feec40a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:29.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.163+0000 7feedb59e640 1 --2- 172.21.15.107:0/3098871200 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feec40a2f00 0x7feec40a53c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7feedc07fcd0 tx=0x7feed4010a30 comp rx=0 tx=0).stop 2024-05-09T08:45:29.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.163+0000 7feedb59e640 1 -- 172.21.15.107:0/3098871200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feedc108aa0 msgr2=0x7feedc0844a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:29.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.163+0000 7feedb59e640 1 --2- 172.21.15.107:0/3098871200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feedc108aa0 0x7feedc0844a0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7feecc0097d0 tx=0x7feecc009c90 comp rx=0 tx=0).stop 2024-05-09T08:45:29.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.163+0000 7feedb59e640 1 -- 172.21.15.107:0/3098871200 shutdown_connections 2024-05-09T08:45:29.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.164+0000 7feedb59e640 1 --2- 172.21.15.107:0/3098871200 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7feec40a2f00 0x7feec40a53c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.164+0000 7feedb59e640 1 --2- 172.21.15.107:0/3098871200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feedc07ec00 0x7feedc1bb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.164+0000 7feedb59e640 1 --2- 172.21.15.107:0/3098871200 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feedc10cff0 0x7feedc07e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.164+0000 7feedb59e640 1 --2- 172.21.15.107:0/3098871200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feedc108aa0 0x7feedc0844a0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:29.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.164+0000 7feedb59e640 1 -- 172.21.15.107:0/3098871200 >> 172.21.15.107:0/3098871200 conn(0x7feedc06c7b0 msgr2=0x7feedc071270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:29.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.168+0000 7feedb59e640 1 -- 172.21.15.107:0/3098871200 shutdown_connections 2024-05-09T08:45:29.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:29.168+0000 7feedb59e640 1 -- 172.21.15.107:0/3098871200 wait complete. 2024-05-09T08:45:29.191 INFO:tasks.cephfs.filesystem:scrub status for tag:9a69fb35-17df-462b-9df8-d3955393209f - {'path': '/', 'tag': '9a69fb35-17df-462b-9df8-d3955393209f', 'options': 'recursive,force'} 2024-05-09T08:45:30.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:29 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:45:30.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:29 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:30.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:29 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:45:30.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:29 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:45:30.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:29 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:30.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:29 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:45:30.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:29 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:45:30.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:29 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:45:30.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:29 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:45:31.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:30 smithi107 ceph-mon[30122]: pgmap v379: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-05-09T08:45:31.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:30 smithi157 ceph-mon[33614]: pgmap v379: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-05-09T08:45:31.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:30 smithi110 ceph-mon[33622]: pgmap v379: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-05-09T08:45:33.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:32 smithi107 ceph-mon[30122]: pgmap v380: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 8.3 MiB/s wr, 2.04k op/s 2024-05-09T08:45:33.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:32 smithi157 ceph-mon[33614]: pgmap v380: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 8.3 MiB/s wr, 2.04k op/s 2024-05-09T08:45:33.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:32 smithi110 ceph-mon[33622]: pgmap v380: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 8.3 MiB/s wr, 2.04k op/s 2024-05-09T08:45:35.032 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:34 smithi107 ceph-mon[30122]: pgmap v381: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 13 MiB/s wr, 3.10k op/s 2024-05-09T08:45:35.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:34 smithi157 ceph-mon[33614]: pgmap v381: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 13 MiB/s wr, 3.10k op/s 2024-05-09T08:45:35.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:34 smithi110 ceph-mon[33622]: pgmap v381: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 13 MiB/s wr, 3.10k op/s 2024-05-09T08:45:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.212+0000 7fadf487b640 1 -- 172.21.15.107:0/665558558 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade4030d70 con 0x7fadf01033e0 2024-05-09T08:45:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.213+0000 7fadf7ae1640 1 -- 172.21.15.107:0/665558558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadf01033e0 msgr2=0x7fadf010b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.213+0000 7fadf7ae1640 1 --2- 172.21.15.107:0/665558558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadf01033e0 0x7fadf010b8c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fade4009860 tx=0x7fade402a460 comp rx=0 tx=0).stop 2024-05-09T08:45:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.213+0000 7fadf7ae1640 1 -- 172.21.15.107:0/665558558 shutdown_connections 2024-05-09T08:45:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.213+0000 7fadf7ae1640 1 --2- 172.21.15.107:0/665558558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadf010be00 0x7fadf010e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.213+0000 7fadf7ae1640 1 --2- 172.21.15.107:0/665558558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadf01033e0 0x7fadf010b8c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.213+0000 7fadf7ae1640 1 --2- 172.21.15.107:0/665558558 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadf0102ac0 0x7fadf0102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.213+0000 7fadf7ae1640 1 -- 172.21.15.107:0/665558558 >> 172.21.15.107:0/665558558 conn(0x7fadf00778b0 msgr2=0x7fadf0077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.213+0000 7fadf7ae1640 1 -- 172.21.15.107:0/665558558 shutdown_connections 2024-05-09T08:45:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.213+0000 7fadf7ae1640 1 -- 172.21.15.107:0/665558558 wait complete. 2024-05-09T08:45:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.214+0000 7fadf7ae1640 1 Processor -- start 2024-05-09T08:45:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.214+0000 7fadf7ae1640 1 -- start start 2024-05-09T08:45:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.214+0000 7fadf7ae1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadf0102ac0 0x7fadf0107b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.214+0000 7fadf7ae1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadf01033e0 0x7fadf01080b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.214+0000 7fadf7ae1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadf010be00 0x7fadf01085f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.214+0000 7fadf7ae1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fadf01013f0 con 0x7fadf010be00 2024-05-09T08:45:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.214+0000 7fadf7ae1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fadf0101270 con 0x7fadf01033e0 2024-05-09T08:45:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.214+0000 7fadf7ae1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fadf0101570 con 0x7fadf0102ac0 2024-05-09T08:45:36.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.214+0000 7fadf507c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadf01033e0 0x7fadf01080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:36.214 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.214+0000 7fadf507c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadf01033e0 0x7fadf01080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:49438/0 (socket says 172.21.15.107:49438) 2024-05-09T08:45:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.214+0000 7fadf507c640 1 -- 172.21.15.107:0/265471461 learned_addr learned my addr 172.21.15.107:0/265471461 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.214+0000 7fadf587d640 1 --2- 172.21.15.107:0/265471461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadf0102ac0 0x7fadf0107b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.215+0000 7fadf607e640 1 --2- 172.21.15.107:0/265471461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadf010be00 0x7fadf01085f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.215+0000 7fadf587d640 1 -- 172.21.15.107:0/265471461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadf01033e0 msgr2=0x7fadf01080b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.215+0000 7fadf587d640 1 --2- 172.21.15.107:0/265471461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadf01033e0 0x7fadf01080b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.215+0000 7fadf587d640 1 -- 172.21.15.107:0/265471461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadf010be00 msgr2=0x7fadf01085f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.215+0000 7fadf587d640 1 --2- 172.21.15.107:0/265471461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadf010be00 0x7fadf01085f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.215+0000 7fadf587d640 1 -- 172.21.15.107:0/265471461 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf01aa810 con 0x7fadf0102ac0 2024-05-09T08:45:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.215+0000 7fadf507c640 1 --2- 172.21.15.107:0/265471461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadf01033e0 0x7fadf01080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:45:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.215+0000 7fadf607e640 1 --2- 172.21.15.107:0/265471461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadf010be00 0x7fadf01085f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:45:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.215+0000 7fadf587d640 1 --2- 172.21.15.107:0/265471461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadf0102ac0 0x7fadf0107b70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fadec00e7c0 tx=0x7fadec00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.216+0000 7fade2ffd640 1 -- 172.21.15.107:0/265471461 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadec00ccc0 con 0x7fadf0102ac0 2024-05-09T08:45:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.216+0000 7fadf7ae1640 1 -- 172.21.15.107:0/265471461 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadf01aab00 con 0x7fadf0102ac0 2024-05-09T08:45:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.216+0000 7fadf7ae1640 1 -- 172.21.15.107:0/265471461 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadf01ab050 con 0x7fadf0102ac0 2024-05-09T08:45:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.218+0000 7fade2ffd640 1 -- 172.21.15.107:0/265471461 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadec00ce60 con 0x7fadf0102ac0 2024-05-09T08:45:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.218+0000 7fade2ffd640 1 -- 172.21.15.107:0/265471461 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadec018680 con 0x7fadf0102ac0 2024-05-09T08:45:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.220+0000 7fadfa7f4740 1 Processor -- start 2024-05-09T08:45:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.220+0000 7fadfa7f4740 1 -- start start 2024-05-09T08:45:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.221+0000 7fadfa7f4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4c450 0x5639f7c4c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.221+0000 7fadfa7f4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c4cd70 0x5639f7c570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.221+0000 7fadfa7f4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c57630 0x5639f7c59a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.221+0000 7fadfa7f4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5639f7bb1340 con 0x5639f7c4cd70 2024-05-09T08:45:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.221+0000 7fadfa7f4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5639f7b59290 con 0x5639f7c57630 2024-05-09T08:45:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.221+0000 7fadfa7f4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5639f7bad300 con 0x5639f7c4c450 2024-05-09T08:45:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.221+0000 7fadf507c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c57630 0x5639f7c59a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.221+0000 7fadf507c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c57630 0x5639f7c59a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:49440/0 (socket says 172.21.15.107:49440) 2024-05-09T08:45:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.221+0000 7fadf507c640 1 -- 172.21.15.107:0/3234754747 learned_addr learned my addr 172.21.15.107:0/3234754747 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.221+0000 7fadf607e640 1 --2- 172.21.15.107:0/3234754747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4c450 0x5639f7c4c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.221+0000 7fadf507c640 1 -- 172.21.15.107:0/3234754747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4c450 msgr2=0x5639f7c4c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.221+0000 7fadf507c640 1 --2- 172.21.15.107:0/3234754747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4c450 0x5639f7c4c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.221+0000 7fadf507c640 1 -- 172.21.15.107:0/3234754747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c4cd70 msgr2=0x5639f7c570f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:45:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.221+0000 7fadf507c640 1 --2- 172.21.15.107:0/3234754747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c4cd70 0x5639f7c570f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.221+0000 7fadf507c640 1 -- 172.21.15.107:0/3234754747 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639f7b7d8e0 con 0x5639f7c57630 2024-05-09T08:45:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.222+0000 7fadf607e640 1 --2- 172.21.15.107:0/3234754747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4c450 0x5639f7c4c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:45:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.222+0000 7fadf507c640 1 --2- 172.21.15.107:0/3234754747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c57630 0x5639f7c59a20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fade4009980 tx=0x7fade4032f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c53e4b50e48f5194 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.222+0000 7fade2ffd640 1 -- 172.21.15.107:0/265471461 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fadec018820 con 0x7fadf0102ac0 2024-05-09T08:45:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.222+0000 7faddbfff640 1 -- 172.21.15.107:0/3234754747 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade4030c70 con 0x5639f7c57630 2024-05-09T08:45:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.222+0000 7faddbfff640 1 -- 172.21.15.107:0/3234754747 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fade40054f0 con 0x5639f7c57630 2024-05-09T08:45:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.222+0000 7faddbfff640 1 -- 172.21.15.107:0/3234754747 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade4046440 con 0x5639f7c57630 2024-05-09T08:45:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.223+0000 7fade2ffd640 1 --2- 172.21.15.107:0/265471461 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faddc0a2f80 0x7faddc0a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.223+0000 7fade2ffd640 1 -- 172.21.15.107:0/265471461 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7fadec0cf900 con 0x7fadf0102ac0 2024-05-09T08:45:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.223+0000 7fadfa7f4740 1 -- 172.21.15.107:0/3234754747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c57630 msgr2=0x5639f7c59a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.223+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/3234754747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c57630 0x5639f7c59a20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fade4009980 tx=0x7fade4032f70 comp rx=0 tx=0).stop 2024-05-09T08:45:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.223+0000 7fadfa7f4740 1 -- 172.21.15.107:0/3234754747 shutdown_connections 2024-05-09T08:45:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.224+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/3234754747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c57630 0x5639f7c59a20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.224+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/3234754747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c4cd70 0x5639f7c570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.224+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/3234754747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4c450 0x5639f7c4c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.224+0000 7fadfa7f4740 1 -- 172.21.15.107:0/3234754747 >> 172.21.15.107:0/3234754747 conn(0x5639f7c48960 msgr2=0x5639f7c48d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.224+0000 7fadf607e640 1 --2- 172.21.15.107:0/265471461 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faddc0a2f80 0x7faddc0a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.224+0000 7fadfa7f4740 1 -- 172.21.15.107:0/3234754747 shutdown_connections 2024-05-09T08:45:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.224+0000 7fadfa7f4740 1 -- 172.21.15.107:0/3234754747 wait complete. 2024-05-09T08:45:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.224+0000 7fadfa7f4740 1 -- start start 2024-05-09T08:45:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.224+0000 7fadfa7f4740 1 Processor -- start 2024-05-09T08:45:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.225+0000 7fadf607e640 1 --2- 172.21.15.107:0/265471461 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faddc0a2f80 0x7faddc0a5440 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fade8009e60 tx=0x7fade80093b0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.225+0000 7fadfa7f4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4c450 0x5639f7c545e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.225+0000 7fadfa7f4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4cd70 0x5639f7c54b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.225+0000 7fadfa7f4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c57630 0x5639f7c55060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.225+0000 7fadfa7f4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5639f7c5a200 con 0x5639f7c57630 2024-05-09T08:45:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.225+0000 7fadfa7f4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5639f7b6c790 con 0x5639f7c4cd70 2024-05-09T08:45:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.225+0000 7fadfa7f4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5639f7c5a380 con 0x5639f7c4c450 2024-05-09T08:45:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.225+0000 7fadfa7f4740 10 client.0 _ll_register_callbacks cb 0x5639f7ba7a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.226+0000 7fadf507c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4cd70 0x5639f7c54b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.226+0000 7fadf507c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4cd70 0x5639f7c54b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:49446/0 (socket says 172.21.15.107:49446) 2024-05-09T08:45:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.226+0000 7fadf507c640 1 -- 172.21.15.107:0/2685157034 learned_addr learned my addr 172.21.15.107:0/2685157034 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.226+0000 7fadf587d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4c450 0x5639f7c545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.226+0000 7fadf507c640 1 -- 172.21.15.107:0/2685157034 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4c450 msgr2=0x5639f7c545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.226+0000 7fadf507c640 1 --2- 172.21.15.107:0/2685157034 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4c450 0x5639f7c545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.226+0000 7fadf507c640 1 -- 172.21.15.107:0/2685157034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c57630 msgr2=0x5639f7c55060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.226+0000 7fadf507c640 1 --2- 172.21.15.107:0/2685157034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c57630 0x5639f7c55060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.226+0000 7fadf507c640 1 -- 172.21.15.107:0/2685157034 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639f78c5210 con 0x5639f7c4cd70 2024-05-09T08:45:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.226+0000 7fadf507c640 1 --2- 172.21.15.107:0/2685157034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4cd70 0x5639f7c54b20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fade4031000 tx=0x7fade4030ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=468ca7315e37b620 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.227+0000 7fadd97fa640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:45:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.227+0000 7fadd97fa640 1 -- 172.21.15.107:0/2685157034 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade4030d90 con 0x5639f7c4cd70 2024-05-09T08:45:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.227+0000 7fadd97fa640 1 -- 172.21.15.107:0/2685157034 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7fade4046440 con 0x5639f7c4cd70 2024-05-09T08:45:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.227+0000 7fadfa7f4740 1 -- 172.21.15.107:0/2685157034 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fade40054f0 con 0x5639f7c4cd70 2024-05-09T08:45:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.228+0000 7fadd97fa640 1 -- 172.21.15.107:0/2685157034 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fade4030750 con 0x5639f7c4cd70 2024-05-09T08:45:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.228+0000 7fadfa7f4740 10 client.27899 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.228+0000 7fadfa7f4740 1 -- 172.21.15.107:0/2685157034 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5639f7991620 con 0x5639f7c4cd70 2024-05-09T08:45:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.228+0000 7fadd97fa640 1 -- 172.21.15.107:0/2685157034 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fade4046af0 con 0x5639f7c4cd70 2024-05-09T08:45:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.228+0000 7fadd97fa640 1 -- 172.21.15.107:0/2685157034 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fade405a020 con 0x5639f7c4cd70 2024-05-09T08:45:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.228+0000 7fadfa7f4740 10 client.27899 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.228+0000 7fadfa7f4740 10 client.27899 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:45:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.228+0000 7fadfa7f4740 20 client.27899 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.228+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/2685157034 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5639f7c66fd0 0x5639f7c693c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.228+0000 7fadfa7f4740 4 client.27899 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:45:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.228+0000 7fadfa7f4740 1 -- 172.21.15.107:0/2685157034 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fade40054f0 con 0x5639f7c66fd0 2024-05-09T08:45:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.229+0000 7fadf587d640 1 --2- 172.21.15.107:0/2685157034 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5639f7c66fd0 0x5639f7c693c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.229+0000 7fadf587d640 1 --2- 172.21.15.107:0/2685157034 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5639f7c66fd0 0x5639f7c693c0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51f0a814d594afb8 server_cookie=6ad6812ba61c4ee8 in_seq=0 out_seq=0 2024-05-09T08:45:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.229+0000 7fadd97fa640 10 client.27899 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.230+0000 7fadd97fa640 1 -- 172.21.15.107:0/2685157034 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fade40054f0 con 0x5639f7c66fd0 2024-05-09T08:45:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.230+0000 7fadd97fa640 10 client.27899 handle_command_reply: tid=0 2024-05-09T08:45:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.230+0000 7fadfa7f4740 1 client.27899 shutdown 2024-05-09T08:45:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.230+0000 7fadfa7f4740 1 -- 172.21.15.107:0/2685157034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4cd70 msgr2=0x5639f7c54b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.230+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/2685157034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4cd70 0x5639f7c54b20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fade4031000 tx=0x7fade4030ca0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.230+0000 7fadfa7f4740 1 -- 172.21.15.107:0/2685157034 shutdown_connections 2024-05-09T08:45:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.230+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/2685157034 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5639f7c66fd0 0x5639f7c693c0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.230+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/2685157034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c57630 0x5639f7c55060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.230+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/2685157034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4cd70 0x5639f7c54b20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.230+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/2685157034 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4c450 0x5639f7c545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.230+0000 7fadfa7f4740 1 -- 172.21.15.107:0/2685157034 >> 172.21.15.107:0/2685157034 conn(0x5639f7c48960 msgr2=0x5639f7c4e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.230+0000 7fadd97fa640 0 client.27899 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.230+0000 7fadfa7f4740 1 -- 172.21.15.107:0/2685157034 shutdown_connections 2024-05-09T08:45:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.230+0000 7fadfa7f4740 1 -- 172.21.15.107:0/2685157034 wait complete. 2024-05-09T08:45:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.231+0000 7fadfa7f4740 20 client.27899 trim_cache size 0 max 16384 2024-05-09T08:45:36.259 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.259+0000 7fadfa7f4740 1 Processor -- start 2024-05-09T08:45:36.259 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.259+0000 7fadfa7f4740 1 -- start start 2024-05-09T08:45:36.259 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.260+0000 7fadfa7f4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4c450 0x5639f7c56d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.259 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.260+0000 7fadfa7f4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4cd70 0x5639f7c537e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.260 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.260+0000 7fadf587d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4c450 0x5639f7c56d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:36.260 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.260+0000 7fadfa7f4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c57630 0x5639f7c4ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.261 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.260+0000 7fadfa7f4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fadd40128c0 con 0x5639f7c57630 2024-05-09T08:45:36.261 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.260+0000 7fadfa7f4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fadd4010190 con 0x5639f7c4c450 2024-05-09T08:45:36.261 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.260+0000 7fadfa7f4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fadd400dfe0 con 0x5639f7c4cd70 2024-05-09T08:45:36.261 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.260+0000 7fadf587d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4c450 0x5639f7c56d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:49448/0 (socket says 172.21.15.107:49448) 2024-05-09T08:45:36.261 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.260+0000 7fadf587d640 1 -- 172.21.15.107:0/2054067403 learned_addr learned my addr 172.21.15.107:0/2054067403 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:36.261 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.260+0000 7fadf607e640 1 --2- 172.21.15.107:0/2054067403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c57630 0x5639f7c4ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:36.261 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.260+0000 7fadf507c640 1 --2- 172.21.15.107:0/2054067403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4cd70 0x5639f7c537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:36.261 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.261+0000 7fadf587d640 1 -- 172.21.15.107:0/2054067403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4cd70 msgr2=0x5639f7c537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:36.261 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.261+0000 7fadf587d640 1 --2- 172.21.15.107:0/2054067403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4cd70 0x5639f7c537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.261 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.261+0000 7fadf587d640 1 -- 172.21.15.107:0/2054067403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c57630 msgr2=0x5639f7c4ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:36.261 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.261+0000 7fadf587d640 1 --2- 172.21.15.107:0/2054067403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c57630 0x5639f7c4ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.261 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.261+0000 7fadf587d640 1 -- 172.21.15.107:0/2054067403 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639f7c4d920 con 0x5639f7c4c450 2024-05-09T08:45:36.262 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.261+0000 7fadf587d640 1 --2- 172.21.15.107:0/2054067403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4c450 0x5639f7c56d10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fadec09b950 tx=0x7fadec00e710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79a1f00f86a917fd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:36.262 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.261+0000 7faddb7fe640 1 -- 172.21.15.107:0/2054067403 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadec099a60 con 0x5639f7c4c450 2024-05-09T08:45:36.262 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.261+0000 7faddb7fe640 1 -- 172.21.15.107:0/2054067403 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadec099cc0 con 0x5639f7c4c450 2024-05-09T08:45:36.262 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.261+0000 7faddb7fe640 1 -- 172.21.15.107:0/2054067403 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadec0b4bc0 con 0x5639f7c4c450 2024-05-09T08:45:36.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.262+0000 7fadfa7f4740 1 -- 172.21.15.107:0/2054067403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4c450 msgr2=0x5639f7c56d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:36.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.263+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/2054067403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4c450 0x5639f7c56d10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fadec09b950 tx=0x7fadec00e710 comp rx=0 tx=0).stop 2024-05-09T08:45:36.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.263+0000 7fadfa7f4740 1 -- 172.21.15.107:0/2054067403 shutdown_connections 2024-05-09T08:45:36.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.263+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/2054067403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c57630 0x5639f7c4ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.263+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/2054067403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4cd70 0x5639f7c537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.263+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/2054067403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4c450 0x5639f7c56d10 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.263+0000 7fadfa7f4740 1 -- 172.21.15.107:0/2054067403 >> 172.21.15.107:0/2054067403 conn(0x5639f7c48960 msgr2=0x5639f7c62630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:36.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.266+0000 7fadfa7f4740 1 -- 172.21.15.107:0/2054067403 shutdown_connections 2024-05-09T08:45:36.271 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.270+0000 7fadfa7f4740 1 -- 172.21.15.107:0/2054067403 wait complete. 2024-05-09T08:45:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.270+0000 7fadfa7f4740 1 -- start start 2024-05-09T08:45:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.271+0000 7fadfa7f4740 1 Processor -- start 2024-05-09T08:45:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.271+0000 7fadfa7f4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4c450 0x5639f7c6c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.271+0000 7fadfa7f4740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4cd70 0x5639f7c69790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.271+0000 7fadfa7f4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c57630 0x5639f7c69cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.271+0000 7fadfa7f4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fadd400e650 con 0x5639f7c57630 2024-05-09T08:45:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.271+0000 7fadfa7f4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fadd400fac0 con 0x5639f7c4c450 2024-05-09T08:45:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.271+0000 7fadfa7f4740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fadd400ed20 con 0x5639f7c4cd70 2024-05-09T08:45:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.271+0000 7fadf507c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4cd70 0x5639f7c69790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.272+0000 7fadf507c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4cd70 0x5639f7c69790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:43604/0 (socket says 172.21.15.107:43604) 2024-05-09T08:45:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.272+0000 7fadf507c640 1 -- 172.21.15.107:0/591966006 learned_addr learned my addr 172.21.15.107:0/591966006 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.272+0000 7fadfa7f4740 10 client.0 _ll_register_callbacks cb 0x7fade4032ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.272+0000 7fadf507c640 1 -- 172.21.15.107:0/591966006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4c450 msgr2=0x5639f7c6c400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:45:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.272+0000 7fadf507c640 1 --2- 172.21.15.107:0/591966006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4c450 0x5639f7c6c400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.272+0000 7fadf507c640 1 -- 172.21.15.107:0/591966006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c57630 msgr2=0x5639f7c69cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:36.273 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.272+0000 7fadf507c640 1 --2- 172.21.15.107:0/591966006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c57630 0x5639f7c69cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:36.273 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.272+0000 7fadf507c640 1 -- 172.21.15.107:0/591966006 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639f7c6a4d0 con 0x5639f7c4cd70 2024-05-09T08:45:36.273 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.272+0000 7fadf507c640 1 --2- 172.21.15.107:0/591966006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4cd70 0x5639f7c69790 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fade403cb60 tx=0x7fade4030f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50d7a8bf39b1ddbb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:36.273 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.272+0000 7fadd97fa640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:45:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.274+0000 7fadd97fa640 1 -- 172.21.15.107:0/591966006 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade4081070 con 0x5639f7c4cd70 2024-05-09T08:45:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.274+0000 7fadd97fa640 1 -- 172.21.15.107:0/591966006 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7fade403c420 con 0x5639f7c4cd70 2024-05-09T08:45:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.274+0000 7fadfa7f4740 1 -- 172.21.15.107:0/591966006 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadec099cc0 con 0x5639f7c4cd70 2024-05-09T08:45:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.274+0000 7fadd97fa640 1 -- 172.21.15.107:0/591966006 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fade403cde0 con 0x5639f7c4cd70 2024-05-09T08:45:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.275+0000 7fadfa7f4740 10 client.28117 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.275+0000 7fadfa7f4740 1 -- 172.21.15.107:0/591966006 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5639f7cc3300 con 0x5639f7c4cd70 2024-05-09T08:45:36.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.275+0000 7fadd97fa640 1 -- 172.21.15.107:0/591966006 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fade4040040 con 0x5639f7c4cd70 2024-05-09T08:45:36.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.275+0000 7fadd97fa640 1 -- 172.21.15.107:0/591966006 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fade405a050 con 0x5639f7c4cd70 2024-05-09T08:45:36.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.276+0000 7fadfa7f4740 10 client.28117 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:36.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.276+0000 7fadfa7f4740 10 client.28117 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:45:36.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.276+0000 7fadfa7f4740 20 client.28117 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:36.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.276+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/591966006 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5639f7cc3870 0x5639f7cc5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:36.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.276+0000 7fadfa7f4740 4 client.28117 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:45:36.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.276+0000 7fadfa7f4740 1 -- 172.21.15.107:0/591966006 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fadec099cc0 con 0x5639f7cc3870 2024-05-09T08:45:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.277+0000 7fadf587d640 1 --2- 172.21.15.107:0/591966006 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5639f7cc3870 0x5639f7cc5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.277+0000 7fadd97fa640 10 client.28117 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:36.277+0000 7fadf587d640 1 --2- 172.21.15.107:0/591966006 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5639f7cc3870 0x5639f7cc5cb0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=605695122750f844 server_cookie=573a97d032a0f3d8 in_seq=0 out_seq=0 2024-05-09T08:45:37.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:36 smithi107 ceph-mon[30122]: pgmap v382: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 9.6 MiB/s wr, 2.40k op/s 2024-05-09T08:45:37.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:36 smithi157 ceph-mon[33614]: pgmap v382: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 9.6 MiB/s wr, 2.40k op/s 2024-05-09T08:45:37.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:36 smithi110 ceph-mon[33622]: pgmap v382: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 9.6 MiB/s wr, 2.40k op/s 2024-05-09T08:45:39.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:38 smithi107 ceph-mon[30122]: pgmap v383: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-05-09T08:45:39.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:38 smithi157 ceph-mon[33614]: pgmap v383: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-05-09T08:45:39.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:38 smithi110 ceph-mon[33622]: pgmap v383: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-05-09T08:45:41.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:40 smithi107 ceph-mon[30122]: pgmap v384: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 13 MiB/s wr, 3.26k op/s 2024-05-09T08:45:41.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:40 smithi157 ceph-mon[33614]: pgmap v384: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 13 MiB/s wr, 3.26k op/s 2024-05-09T08:45:41.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:40 smithi110 ceph-mon[33622]: pgmap v384: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 13 MiB/s wr, 3.26k op/s 2024-05-09T08:45:43.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:42 smithi107 ceph-mon[30122]: pgmap v385: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 9.7 MiB/s wr, 2.44k op/s 2024-05-09T08:45:43.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:42 smithi157 ceph-mon[33614]: pgmap v385: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 9.7 MiB/s wr, 2.44k op/s 2024-05-09T08:45:43.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:42 smithi110 ceph-mon[33622]: pgmap v385: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 9.7 MiB/s wr, 2.44k op/s 2024-05-09T08:45:45.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:44 smithi107 ceph-mon[30122]: pgmap v386: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 13 MiB/s wr, 3.33k op/s 2024-05-09T08:45:45.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:44 smithi157 ceph-mon[33614]: pgmap v386: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 13 MiB/s wr, 3.33k op/s 2024-05-09T08:45:45.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:44 smithi110 ceph-mon[33622]: pgmap v386: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 13 MiB/s wr, 3.33k op/s 2024-05-09T08:45:47.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:46 smithi107 ceph-mon[30122]: pgmap v387: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 9.9 MiB/s wr, 2.48k op/s 2024-05-09T08:45:47.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:46 smithi157 ceph-mon[33614]: pgmap v387: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 9.9 MiB/s wr, 2.48k op/s 2024-05-09T08:45:47.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:46 smithi110 ceph-mon[33622]: pgmap v387: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 9.9 MiB/s wr, 2.48k op/s 2024-05-09T08:45:49.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:48 smithi107 ceph-mon[30122]: pgmap v388: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 10 MiB/s wr, 2.55k op/s 2024-05-09T08:45:49.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:48 smithi157 ceph-mon[33614]: pgmap v388: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 10 MiB/s wr, 2.55k op/s 2024-05-09T08:45:49.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:48 smithi110 ceph-mon[33622]: pgmap v388: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 10 MiB/s wr, 2.55k op/s 2024-05-09T08:45:51.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:50 smithi107 ceph-mon[30122]: pgmap v389: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 12 MiB/s wr, 3.11k op/s 2024-05-09T08:45:51.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:50 smithi157 ceph-mon[33614]: pgmap v389: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 12 MiB/s wr, 3.11k op/s 2024-05-09T08:45:51.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:50 smithi110 ceph-mon[33622]: pgmap v389: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 12 MiB/s wr, 3.11k op/s 2024-05-09T08:45:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.605+0000 7fadd97fa640 1 -- 172.21.15.107:0/591966006 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fadec099cc0 con 0x5639f7cc3870 2024-05-09T08:45:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.605+0000 7fadd97fa640 10 client.28117 handle_command_reply: tid=0 2024-05-09T08:45:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.605+0000 7fadfa7f4740 1 client.28117 shutdown 2024-05-09T08:45:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.605+0000 7fadfa7f4740 1 -- 172.21.15.107:0/591966006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4cd70 msgr2=0x5639f7c69790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.605+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/591966006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4cd70 0x5639f7c69790 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fade403cb60 tx=0x7fade4030f20 comp rx=0 tx=0).stop 2024-05-09T08:45:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.605+0000 7fadfa7f4740 1 -- 172.21.15.107:0/591966006 shutdown_connections 2024-05-09T08:45:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.605+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/591966006 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5639f7cc3870 0x5639f7cc5cb0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.605+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/591966006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639f7c57630 0x5639f7c69cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.605+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/591966006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639f7c4cd70 0x5639f7c69790 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.605+0000 7fadfa7f4740 1 --2- 172.21.15.107:0/591966006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639f7c4c450 0x5639f7c6c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.605+0000 7fadfa7f4740 1 -- 172.21.15.107:0/591966006 >> 172.21.15.107:0/591966006 conn(0x5639f7c48960 msgr2=0x5639f7c47a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:51.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.605+0000 7fadd97fa640 0 client.28117 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.608+0000 7fadfa7f4740 1 -- 172.21.15.107:0/591966006 shutdown_connections 2024-05-09T08:45:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.608+0000 7fadfa7f4740 1 -- 172.21.15.107:0/591966006 wait complete. 2024-05-09T08:45:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.610+0000 7fadfa7f4740 20 client.28117 trim_cache size 0 max 16384 2024-05-09T08:45:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:45:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:45:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:45:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:45:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.618+0000 7fadf7ae1640 1 -- 172.21.15.107:0/265471461 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faddc0a2f80 msgr2=0x7faddc0a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.618+0000 7fadf7ae1640 1 --2- 172.21.15.107:0/265471461 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faddc0a2f80 0x7faddc0a5440 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fade8009e60 tx=0x7fade80093b0 comp rx=0 tx=0).stop 2024-05-09T08:45:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.618+0000 7fadf7ae1640 1 -- 172.21.15.107:0/265471461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadf0102ac0 msgr2=0x7fadf0107b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.618+0000 7fadf7ae1640 1 --2- 172.21.15.107:0/265471461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadf0102ac0 0x7fadf0107b70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fadec00e7c0 tx=0x7fadec00ec80 comp rx=0 tx=0).stop 2024-05-09T08:45:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.618+0000 7fadf7ae1640 1 -- 172.21.15.107:0/265471461 shutdown_connections 2024-05-09T08:45:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.618+0000 7fadf7ae1640 1 --2- 172.21.15.107:0/265471461 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7faddc0a2f80 0x7faddc0a5440 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.618+0000 7fadf7ae1640 1 --2- 172.21.15.107:0/265471461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadf010be00 0x7fadf01085f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:51.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.618+0000 7fadf7ae1640 1 --2- 172.21.15.107:0/265471461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadf01033e0 0x7fadf01080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:51.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.618+0000 7fadf7ae1640 1 --2- 172.21.15.107:0/265471461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadf0102ac0 0x7fadf0107b70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:51.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.618+0000 7fadf7ae1640 1 -- 172.21.15.107:0/265471461 >> 172.21.15.107:0/265471461 conn(0x7fadf00778b0 msgr2=0x7fadf00ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:51.637 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.625+0000 7fadf7ae1640 1 -- 172.21.15.107:0/265471461 shutdown_connections 2024-05-09T08:45:51.637 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:45:51.625+0000 7fadf7ae1640 1 -- 172.21.15.107:0/265471461 wait complete. 2024-05-09T08:45:53.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:52 smithi107 ceph-mon[30122]: pgmap v390: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 9.4 MiB/s wr, 2.36k op/s 2024-05-09T08:45:53.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:52 smithi157 ceph-mon[33614]: pgmap v390: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 9.4 MiB/s wr, 2.36k op/s 2024-05-09T08:45:53.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:52 smithi110 ceph-mon[33622]: pgmap v390: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 9.4 MiB/s wr, 2.36k op/s 2024-05-09T08:45:55.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:54 smithi107 ceph-mon[30122]: pgmap v391: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 13 MiB/s wr, 3.39k op/s 2024-05-09T08:45:55.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:54 smithi157 ceph-mon[33614]: pgmap v391: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 13 MiB/s wr, 3.39k op/s 2024-05-09T08:45:55.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:54 smithi110 ceph-mon[33622]: pgmap v391: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 13 MiB/s wr, 3.39k op/s 2024-05-09T08:45:56.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:55 smithi157 ceph-mon[33614]: pgmap v392: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 10 MiB/s wr, 2.63k op/s 2024-05-09T08:45:56.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:55 smithi110 ceph-mon[33622]: pgmap v392: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 10 MiB/s wr, 2.63k op/s 2024-05-09T08:45:56.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:55 smithi107 ceph-mon[30122]: pgmap v392: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 10 MiB/s wr, 2.63k op/s 2024-05-09T08:45:58.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:45:58 smithi110 ceph-mon[33622]: pgmap v393: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 11 MiB/s wr, 2.74k op/s 2024-05-09T08:45:58.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:45:58 smithi107 ceph-mon[30122]: pgmap v393: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 11 MiB/s wr, 2.74k op/s 2024-05-09T08:45:58.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:45:58 smithi157 ceph-mon[33614]: pgmap v393: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 11 MiB/s wr, 2.74k op/s 2024-05-09T08:45:59.192 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:45:59.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:45:59.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.373+0000 7f71e9e64640 1 -- 172.21.15.107:0/2572056603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71e4100d80 msgr2=0x7f71e4104b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:59.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.373+0000 7f71e9e64640 1 --2- 172.21.15.107:0/2572056603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71e4100d80 0x7f71e4104b20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f71d4005fd0 tx=0x7f71d402a580 comp rx=0 tx=0).stop 2024-05-09T08:45:59.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.373+0000 7f71e9e64640 1 -- 172.21.15.107:0/2572056603 shutdown_connections 2024-05-09T08:45:59.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.373+0000 7f71e9e64640 1 --2- 172.21.15.107:0/2572056603 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71e4105250 0x7f71e410cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.373+0000 7f71e9e64640 1 --2- 172.21.15.107:0/2572056603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71e4100d80 0x7f71e4104b20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.373+0000 7f71e9e64640 1 --2- 172.21.15.107:0/2572056603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71e41003d0 0x7f71e41007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.373+0000 7f71e9e64640 1 -- 172.21.15.107:0/2572056603 >> 172.21.15.107:0/2572056603 conn(0x7f71e40759c0 msgr2=0x7f71e4075dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:59.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.374+0000 7f71e9e64640 1 -- 172.21.15.107:0/2572056603 shutdown_connections 2024-05-09T08:45:59.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.374+0000 7f71e9e64640 1 -- 172.21.15.107:0/2572056603 wait complete. 2024-05-09T08:45:59.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.375+0000 7f71e9e64640 1 Processor -- start 2024-05-09T08:45:59.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.375+0000 7f71e9e64640 1 -- start start 2024-05-09T08:45:59.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.376+0000 7f71e9e64640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71e41003d0 0x7f71e41a6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.376+0000 7f71e9e64640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71e4105250 0x7f71e41a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.376+0000 7f71e9e64640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71e41ab720 0x7f71e41abbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.376+0000 7f71e9e64640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f71e410f8f0 con 0x7f71e41ab720 2024-05-09T08:45:59.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.376+0000 7f71e9e64640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f71e410f770 con 0x7f71e4105250 2024-05-09T08:45:59.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.376+0000 7f71e9e64640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f71e410fa70 con 0x7f71e41003d0 2024-05-09T08:45:59.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.377+0000 7f71e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71e41ab720 0x7f71e41abbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:59.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.377+0000 7f71e37fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71e41003d0 0x7f71e41a6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:59.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.377+0000 7f71e37fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71e41003d0 0x7f71e41a6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:54458/0 (socket says 172.21.15.107:54458) 2024-05-09T08:45:59.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.377+0000 7f71e37fe640 1 -- 172.21.15.107:0/410719943 learned_addr learned my addr 172.21.15.107:0/410719943 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:59.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.377+0000 7f71e2ffd640 1 --2- 172.21.15.107:0/410719943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71e4105250 0x7f71e41a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:59.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.377+0000 7f71e3fff640 1 -- 172.21.15.107:0/410719943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71e41003d0 msgr2=0x7f71e41a6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:59.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.377+0000 7f71e3fff640 1 --2- 172.21.15.107:0/410719943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71e41003d0 0x7f71e41a6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.377+0000 7f71e3fff640 1 -- 172.21.15.107:0/410719943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71e4105250 msgr2=0x7f71e41a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:59.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.377+0000 7f71e3fff640 1 --2- 172.21.15.107:0/410719943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71e4105250 0x7f71e41a73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.377+0000 7f71e3fff640 1 -- 172.21.15.107:0/410719943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e41ac210 con 0x7f71e41ab720 2024-05-09T08:45:59.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.378+0000 7f71e3fff640 1 --2- 172.21.15.107:0/410719943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71e41ab720 0x7f71e41abbd0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f71d800be50 tx=0x7f71d8007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:59.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.378+0000 7f71e0ff9640 1 -- 172.21.15.107:0/410719943 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d8014620 con 0x7f71e41ab720 2024-05-09T08:45:59.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.378+0000 7f71e0ff9640 1 -- 172.21.15.107:0/410719943 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71d80054d0 con 0x7f71e41ab720 2024-05-09T08:45:59.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.378+0000 7f71e0ff9640 1 -- 172.21.15.107:0/410719943 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d8009390 con 0x7f71e41ab720 2024-05-09T08:45:59.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.379+0000 7f71e9e64640 1 -- 172.21.15.107:0/410719943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71e41ac500 con 0x7f71e41ab720 2024-05-09T08:45:59.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.379+0000 7f71e9e64640 1 -- 172.21.15.107:0/410719943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71e406b4f0 con 0x7f71e41ab720 2024-05-09T08:45:59.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.385+0000 7f71ecb73740 1 Processor -- start 2024-05-09T08:45:59.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.386+0000 7f71e0ff9640 1 -- 172.21.15.107:0/410719943 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f71d80046c0 con 0x7f71e41ab720 2024-05-09T08:45:59.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.386+0000 7f71e0ff9640 1 --2- 172.21.15.107:0/410719943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f71cc0a2f00 0x7f71cc0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.387+0000 7f71e0ff9640 1 -- 172.21.15.107:0/410719943 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7f71d80e0910 con 0x7f71e41ab720 2024-05-09T08:45:59.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.387+0000 7f71e2ffd640 1 --2- 172.21.15.107:0/410719943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f71cc0a2f00 0x7f71cc0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:59.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.388+0000 7f71ecb73740 1 -- start start 2024-05-09T08:45:59.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.388+0000 7f71ecb73740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 0x5607383ed660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.388+0000 7f71ecb73740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383edba0 0x5607383f7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.389+0000 7f71ecb73740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 0x5607383fa660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.389+0000 7f71ecb73740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560738352d00 con 0x5607383ed280 2024-05-09T08:45:59.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.389+0000 7f71ecb73740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5607382fa4c0 con 0x5607383f8270 2024-05-09T08:45:59.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.389+0000 7f71ecb73740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56073834ecc0 con 0x5607383edba0 2024-05-09T08:45:59.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.389+0000 7f71e37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 0x5607383ed660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:59.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.389+0000 7f71e37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 0x5607383ed660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52178/0 (socket says 172.21.15.107:52178) 2024-05-09T08:45:59.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.389+0000 7f71e37fe640 1 -- 172.21.15.107:0/3841106367 learned_addr learned my addr 172.21.15.107:0/3841106367 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:59.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.390+0000 7f71e37fe640 1 -- 172.21.15.107:0/3841106367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383edba0 msgr2=0x5607383f7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:59.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.390+0000 7f71e37fe640 1 --2- 172.21.15.107:0/3841106367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383edba0 0x5607383f7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.390+0000 7f71e37fe640 1 -- 172.21.15.107:0/3841106367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 msgr2=0x5607383fa660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:45:59.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.390+0000 7f71e37fe640 1 --2- 172.21.15.107:0/3841106367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 0x5607383fa660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.390+0000 7f71e37fe640 1 -- 172.21.15.107:0/3841106367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607381307d0 con 0x5607383ed280 2024-05-09T08:45:59.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.390+0000 7f71e2ffd640 1 --2- 172.21.15.107:0/410719943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f71cc0a2f00 0x7f71cc0a53c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f71d4000c00 tx=0x7f71d402aa80 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:59.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.390+0000 7f71e37fe640 1 --2- 172.21.15.107:0/3841106367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 0x5607383ed660 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f71dc00f7d0 tx=0x7f71dc0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9269280a21e5129d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:59.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.391+0000 7f71c9ffb640 1 -- 172.21.15.107:0/3841106367 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71dc00ebd0 con 0x5607383ed280 2024-05-09T08:45:59.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.391+0000 7f71c9ffb640 1 -- 172.21.15.107:0/3841106367 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71dc003e20 con 0x5607383ed280 2024-05-09T08:45:59.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.391+0000 7f71c9ffb640 1 -- 172.21.15.107:0/3841106367 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71dc004770 con 0x5607383ed280 2024-05-09T08:45:59.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.392+0000 7f71ecb73740 1 -- 172.21.15.107:0/3841106367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 msgr2=0x5607383ed660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:59.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.392+0000 7f71ecb73740 1 --2- 172.21.15.107:0/3841106367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 0x5607383ed660 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f71dc00f7d0 tx=0x7f71dc0095c0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.392+0000 7f71ecb73740 1 -- 172.21.15.107:0/3841106367 shutdown_connections 2024-05-09T08:45:59.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.392+0000 7f71ecb73740 1 --2- 172.21.15.107:0/3841106367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 0x5607383fa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.392+0000 7f71ecb73740 1 --2- 172.21.15.107:0/3841106367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383edba0 0x5607383f7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.392+0000 7f71ecb73740 1 --2- 172.21.15.107:0/3841106367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 0x5607383ed660 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.392+0000 7f71ecb73740 1 -- 172.21.15.107:0/3841106367 >> 172.21.15.107:0/3841106367 conn(0x5607383e95a0 msgr2=0x5607383e9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:59.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.393+0000 7f71ecb73740 1 -- 172.21.15.107:0/3841106367 shutdown_connections 2024-05-09T08:45:59.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.393+0000 7f71ecb73740 1 -- 172.21.15.107:0/3841106367 wait complete. 2024-05-09T08:45:59.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.394+0000 7f71ecb73740 1 -- start start 2024-05-09T08:45:59.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.394+0000 7f71ecb73740 1 Processor -- start 2024-05-09T08:45:59.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.394+0000 7f71ecb73740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 0x5607383f52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.394+0000 7f71ecb73740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383edba0 0x5607383f5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.394+0000 7f71ecb73740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 0x5607383f5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.394+0000 7f71ecb73740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5607383fb060 con 0x5607383ed280 2024-05-09T08:45:59.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.394+0000 7f71ecb73740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56073830db90 con 0x5607383f8270 2024-05-09T08:45:59.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.394+0000 7f71ecb73740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5607383fb1e0 con 0x5607383edba0 2024-05-09T08:45:59.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.395+0000 7f71ecb73740 10 client.0 _ll_register_callbacks cb 0x560738349da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:59.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.395+0000 7f71e37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 0x5607383f52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:59.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.395+0000 7f71e37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 0x5607383f52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52192/0 (socket says 172.21.15.107:52192) 2024-05-09T08:45:59.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.395+0000 7f71e37fe640 1 -- 172.21.15.107:0/2099349888 learned_addr learned my addr 172.21.15.107:0/2099349888 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:59.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.395+0000 7f71e2ffd640 1 --2- 172.21.15.107:0/2099349888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383edba0 0x5607383f5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:59.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.395+0000 7f71e37fe640 1 -- 172.21.15.107:0/2099349888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383edba0 msgr2=0x5607383f5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:59.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.395+0000 7f71e37fe640 1 --2- 172.21.15.107:0/2099349888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383edba0 0x5607383f5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.395+0000 7f71e37fe640 1 -- 172.21.15.107:0/2099349888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 msgr2=0x5607383f5d50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:45:59.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.395+0000 7f71e37fe640 1 --2- 172.21.15.107:0/2099349888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 0x5607383f5d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.395+0000 7f71e37fe640 1 -- 172.21.15.107:0/2099349888 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607380663f0 con 0x5607383ed280 2024-05-09T08:45:59.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.396+0000 7f71e37fe640 1 --2- 172.21.15.107:0/2099349888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 0x5607383f52d0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f71dc0043d0 tx=0x7f71dc00f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a2a17997a59d9b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:59.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.396+0000 7f71c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:45:59.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.396+0000 7f71c2ffd640 1 -- 172.21.15.107:0/2099349888 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71dc004470 con 0x5607383ed280 2024-05-09T08:45:59.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.396+0000 7f71c2ffd640 1 -- 172.21.15.107:0/2099349888 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7f71dc004610 con 0x5607383ed280 2024-05-09T08:45:59.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.396+0000 7f71ecb73740 1 -- 172.21.15.107:0/2099349888 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71dc003e20 con 0x5607383ed280 2024-05-09T08:45:59.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.396+0000 7f71c2ffd640 1 -- 172.21.15.107:0/2099349888 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f71dc004810 con 0x5607383ed280 2024-05-09T08:45:59.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.398+0000 7f71ecb73740 10 client.18960 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:59.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.398+0000 7f71ecb73740 1 -- 172.21.15.107:0/2099349888 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5607380260f0 con 0x5607383ed280 2024-05-09T08:45:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.398+0000 7f71c2ffd640 1 -- 172.21.15.107:0/2099349888 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71dc003e20 con 0x5607383ed280 2024-05-09T08:45:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.398+0000 7f71c2ffd640 1 -- 172.21.15.107:0/2099349888 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f71dc02d030 con 0x5607383ed280 2024-05-09T08:45:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.399+0000 7f71ecb73740 10 client.18960 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.399+0000 7f71ecb73740 10 client.18960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:45:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.399+0000 7f71ecb73740 20 client.18960 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.399+0000 7f71ecb73740 1 --2- 172.21.15.107:0/2099349888 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607384086d0 0x56073840aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.399+0000 7f71ecb73740 4 client.18960 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:45:59.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.399+0000 7f71ecb73740 1 -- 172.21.15.107:0/2099349888 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560738360430 con 0x5607384086d0 2024-05-09T08:45:59.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.399+0000 7f71e2ffd640 1 --2- 172.21.15.107:0/2099349888 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607384086d0 0x56073840aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:59.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.406+0000 7f71e2ffd640 1 --2- 172.21.15.107:0/2099349888 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607384086d0 0x56073840aac0 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c21798c42e0669f server_cookie=80ebcdc5ee77582 in_seq=0 out_seq=0 2024-05-09T08:45:59.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.407+0000 7f71c2ffd640 10 client.18960 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:59.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.408+0000 7f71c2ffd640 1 -- 172.21.15.107:0/2099349888 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560738360430 con 0x5607384086d0 2024-05-09T08:45:59.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.408+0000 7f71c2ffd640 10 client.18960 handle_command_reply: tid=0 2024-05-09T08:45:59.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.409+0000 7f71ecb73740 1 client.18960 shutdown 2024-05-09T08:45:59.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.409+0000 7f71ecb73740 1 -- 172.21.15.107:0/2099349888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 msgr2=0x5607383f52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:59.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.409+0000 7f71ecb73740 1 --2- 172.21.15.107:0/2099349888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 0x5607383f52d0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f71dc0043d0 tx=0x7f71dc00f490 comp rx=0 tx=0).stop 2024-05-09T08:45:59.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.409+0000 7f71ecb73740 1 -- 172.21.15.107:0/2099349888 shutdown_connections 2024-05-09T08:45:59.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.409+0000 7f71ecb73740 1 --2- 172.21.15.107:0/2099349888 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5607384086d0 0x56073840aac0 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.409+0000 7f71ecb73740 1 --2- 172.21.15.107:0/2099349888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 0x5607383f5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.409+0000 7f71ecb73740 1 --2- 172.21.15.107:0/2099349888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383edba0 0x5607383f5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.409+0000 7f71ecb73740 1 --2- 172.21.15.107:0/2099349888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 0x5607383f52d0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.409+0000 7f71ecb73740 1 -- 172.21.15.107:0/2099349888 >> 172.21.15.107:0/2099349888 conn(0x5607383e95a0 msgr2=0x5607383ef3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:59.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.409+0000 7f71c2ffd640 0 client.18960 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:59.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.411+0000 7f71ecb73740 1 -- 172.21.15.107:0/2099349888 shutdown_connections 2024-05-09T08:45:59.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.411+0000 7f71ecb73740 1 -- 172.21.15.107:0/2099349888 wait complete. 2024-05-09T08:45:59.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.411+0000 7f71ecb73740 20 client.18960 trim_cache size 0 max 16384 2024-05-09T08:45:59.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.445+0000 7f71ecb73740 1 Processor -- start 2024-05-09T08:45:59.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.445+0000 7f71ecb73740 1 -- start start 2024-05-09T08:45:59.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.446+0000 7f71ecb73740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 0x560738403f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.446+0000 7f71ecb73740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383edba0 0x5607383f4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.446+0000 7f71ecb73740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 0x5607383e8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.446+0000 7f71ecb73740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f71e41b7d30 con 0x5607383ed280 2024-05-09T08:45:59.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.446+0000 7f71ecb73740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f71e41b5600 con 0x5607383f8270 2024-05-09T08:45:59.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.446+0000 7f71ecb73740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f71e40734e0 con 0x5607383edba0 2024-05-09T08:45:59.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.448+0000 7f71e3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 0x5607383e8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:59.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.448+0000 7f71e2ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383edba0 0x5607383f4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:59.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.448+0000 7f71e2ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383edba0 0x5607383f4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:54492/0 (socket says 172.21.15.107:54492) 2024-05-09T08:45:59.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.448+0000 7f71e2ffd640 1 -- 172.21.15.107:0/2075099917 learned_addr learned my addr 172.21.15.107:0/2075099917 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:59.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.449+0000 7f71e2ffd640 1 -- 172.21.15.107:0/2075099917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 msgr2=0x5607383e8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:59.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.449+0000 7f71e2ffd640 1 --2- 172.21.15.107:0/2075099917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 0x5607383e8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.449+0000 7f71e2ffd640 1 -- 172.21.15.107:0/2075099917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 msgr2=0x560738403f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:45:59.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.449+0000 7f71e2ffd640 1 --2- 172.21.15.107:0/2075099917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 0x560738403f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.449+0000 7f71e2ffd640 1 -- 172.21.15.107:0/2075099917 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607383ed9d0 con 0x5607383edba0 2024-05-09T08:45:59.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.451+0000 7f71e2ffd640 1 --2- 172.21.15.107:0/2075099917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383edba0 0x5607383f4430 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f71d402ab70 tx=0x7f71d4040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f96ee3c1d8b9e1c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:59.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.453+0000 7f71c97fa640 1 -- 172.21.15.107:0/2075099917 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d4054bc0 con 0x5607383edba0 2024-05-09T08:45:59.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.453+0000 7f71c97fa640 1 -- 172.21.15.107:0/2075099917 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71d4059070 con 0x5607383edba0 2024-05-09T08:45:59.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.454+0000 7f71ecb73740 1 -- 172.21.15.107:0/2075099917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383edba0 msgr2=0x5607383f4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:59.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.454+0000 7f71ecb73740 1 --2- 172.21.15.107:0/2075099917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383edba0 0x5607383f4430 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f71d402ab70 tx=0x7f71d4040000 comp rx=0 tx=0).stop 2024-05-09T08:45:59.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.455+0000 7f71ecb73740 1 -- 172.21.15.107:0/2075099917 shutdown_connections 2024-05-09T08:45:59.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.455+0000 7f71ecb73740 1 --2- 172.21.15.107:0/2075099917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 0x5607383e8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.455+0000 7f71ecb73740 1 --2- 172.21.15.107:0/2075099917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383edba0 0x5607383f4430 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.455+0000 7f71ecb73740 1 --2- 172.21.15.107:0/2075099917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383ed280 0x560738403f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.455+0000 7f71ecb73740 1 -- 172.21.15.107:0/2075099917 >> 172.21.15.107:0/2075099917 conn(0x5607383e95a0 msgr2=0x5607383e7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:59.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.460+0000 7f71ecb73740 1 -- 172.21.15.107:0/2075099917 shutdown_connections 2024-05-09T08:45:59.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.464+0000 7f71ecb73740 1 -- 172.21.15.107:0/2075099917 wait complete. 2024-05-09T08:45:59.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.464+0000 7f71ecb73740 1 -- start start 2024-05-09T08:45:59.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.465+0000 7f71ecb73740 1 Processor -- start 2024-05-09T08:45:59.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.465+0000 7f71ecb73740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383ed280 0x5607384007e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.465+0000 7f71ecb73740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383edba0 0x5607383f1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.465+0000 7f71ecb73740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 0x5607383f21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.465+0000 7f71ecb73740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f71e41b3ac0 con 0x5607383edba0 2024-05-09T08:45:59.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.465+0000 7f71ecb73740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f71e41b4f30 con 0x5607383f8270 2024-05-09T08:45:59.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.465+0000 7f71ecb73740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f71e41b4190 con 0x5607383ed280 2024-05-09T08:45:59.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.465+0000 7f71ecb73740 10 client.0 _ll_register_callbacks cb 0x7f71dc00f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:45:59.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.466+0000 7f71e3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 0x5607383f21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:59.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.466+0000 7f71e3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 0x5607383f21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:59886/0 (socket says 172.21.15.107:59886) 2024-05-09T08:45:59.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.466+0000 7f71e3fff640 1 -- 172.21.15.107:0/225175990 learned_addr learned my addr 172.21.15.107:0/225175990 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:45:59.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.466+0000 7f71e3fff640 1 -- 172.21.15.107:0/225175990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383ed280 msgr2=0x5607384007e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:59.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.466+0000 7f71e3fff640 1 --2- 172.21.15.107:0/225175990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383ed280 0x5607384007e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.466+0000 7f71e3fff640 1 -- 172.21.15.107:0/225175990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383edba0 msgr2=0x5607383f1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:45:59.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.466+0000 7f71e3fff640 1 --2- 172.21.15.107:0/225175990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383edba0 0x5607383f1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.466+0000 7f71e3fff640 1 -- 172.21.15.107:0/225175990 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607383f29b0 con 0x5607383f8270 2024-05-09T08:45:59.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.471+0000 7f71e3fff640 1 --2- 172.21.15.107:0/225175990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 0x5607383f21b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f71d80ac750 tx=0x7f71d8005680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6bfa20d6354890c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:45:59.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.471+0000 7f71c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:45:59.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.477+0000 7f71c2ffd640 1 -- 172.21.15.107:0/225175990 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d80b0e00 con 0x5607383f8270 2024-05-09T08:45:59.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.477+0000 7f71c2ffd640 1 -- 172.21.15.107:0/225175990 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7f71d80dde80 con 0x5607383f8270 2024-05-09T08:45:59.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.478+0000 7f71ecb73740 1 -- 172.21.15.107:0/225175990 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56073835a780 con 0x5607383f8270 2024-05-09T08:45:59.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.481+0000 7f71c2ffd640 1 -- 172.21.15.107:0/225175990 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f71d80ba050 con 0x5607383f8270 2024-05-09T08:45:59.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.482+0000 7f71ecb73740 10 client.27914 fetch_fsmap learned FSMap version 27 2024-05-09T08:45:59.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.482+0000 7f71ecb73740 1 -- 172.21.15.107:0/225175990 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56073846eae0 con 0x5607383f8270 2024-05-09T08:45:59.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.483+0000 7f71c2ffd640 1 -- 172.21.15.107:0/225175990 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56073835a780 con 0x5607383f8270 2024-05-09T08:45:59.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.484+0000 7f71c2ffd640 1 -- 172.21.15.107:0/225175990 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f71d8036060 con 0x5607383f8270 2024-05-09T08:45:59.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.484+0000 7f71ecb73740 10 client.27914 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:45:59.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.484+0000 7f71ecb73740 10 client.27914 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:45:59.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.484+0000 7f71ecb73740 20 client.27914 populate_metadata read hostname 'smithi107' 2024-05-09T08:45:59.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.484+0000 7f71ecb73740 1 --2- 172.21.15.107:0/225175990 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56073846f300 0x560738471760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:45:59.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.484+0000 7f71ecb73740 4 client.27914 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:45:59.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.484+0000 7f71ecb73740 1 -- 172.21.15.107:0/225175990 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5607382d76f0 con 0x56073846f300 2024-05-09T08:45:59.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.486+0000 7f71e37fe640 1 --2- 172.21.15.107:0/225175990 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56073846f300 0x560738471760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:45:59.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.512+0000 7f71e37fe640 1 --2- 172.21.15.107:0/225175990 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56073846f300 0x560738471760 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d06e5f26babfc36 server_cookie=73185a7dbb5a60ba in_seq=0 out_seq=0 2024-05-09T08:45:59.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.513+0000 7f71c2ffd640 10 client.27914 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.518+0000 7f71c2ffd640 1 -- 172.21.15.107:0/225175990 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x5607382d76f0 con 0x56073846f300 2024-05-09T08:45:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.518+0000 7f71c2ffd640 10 client.27914 handle_command_reply: tid=0 2024-05-09T08:45:59.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.518+0000 7f71ecb73740 1 client.27914 shutdown 2024-05-09T08:45:59.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.522+0000 7f71ecb73740 1 -- 172.21.15.107:0/225175990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 msgr2=0x5607383f21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.522+0000 7f71ecb73740 1 --2- 172.21.15.107:0/225175990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 0x5607383f21b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f71d80ac750 tx=0x7f71d8005680 comp rx=0 tx=0).stop 2024-05-09T08:45:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.522+0000 7f71ecb73740 1 -- 172.21.15.107:0/225175990 shutdown_connections 2024-05-09T08:45:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.522+0000 7f71ecb73740 1 --2- 172.21.15.107:0/225175990 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56073846f300 0x560738471760 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.522+0000 7f71ecb73740 1 --2- 172.21.15.107:0/225175990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5607383f8270 0x5607383f21b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.522+0000 7f71ecb73740 1 --2- 172.21.15.107:0/225175990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5607383edba0 0x5607383f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.522+0000 7f71ecb73740 1 --2- 172.21.15.107:0/225175990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5607383ed280 0x5607384007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.522+0000 7f71ecb73740 1 -- 172.21.15.107:0/225175990 >> 172.21.15.107:0/225175990 conn(0x5607383e95a0 msgr2=0x5607383e7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.522+0000 7f71c2ffd640 0 client.27914 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:45:59.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.530+0000 7f71ecb73740 1 -- 172.21.15.107:0/225175990 shutdown_connections 2024-05-09T08:45:59.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.534+0000 7f71ecb73740 1 -- 172.21.15.107:0/225175990 wait complete. 2024-05-09T08:45:59.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.538+0000 7f71ecb73740 20 client.27914 trim_cache size 0 max 16384 2024-05-09T08:45:59.538 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:45:59.539 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "scrub active (65 inodes in the stack)", 2024-05-09T08:45:59.539 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:45:59.539 INFO:teuthology.orchestra.run.smithi107.stdout: "9a69fb35-17df-462b-9df8-d3955393209f": { 2024-05-09T08:45:59.539 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:45:59.539 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "9a69fb35-17df-462b-9df8-d3955393209f", 2024-05-09T08:45:59.539 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:45:59.539 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:45:59.539 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:45:59.539 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:45:59.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.547+0000 7f71e9e64640 1 -- 172.21.15.107:0/410719943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f71cc0a2f00 msgr2=0x7f71cc0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:59.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.547+0000 7f71e9e64640 1 --2- 172.21.15.107:0/410719943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f71cc0a2f00 0x7f71cc0a53c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f71d4000c00 tx=0x7f71d402aa80 comp rx=0 tx=0).stop 2024-05-09T08:45:59.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.548+0000 7f71e9e64640 1 -- 172.21.15.107:0/410719943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71e41ab720 msgr2=0x7f71e41abbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:45:59.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.548+0000 7f71e9e64640 1 --2- 172.21.15.107:0/410719943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71e41ab720 0x7f71e41abbd0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f71d800be50 tx=0x7f71d8007590 comp rx=0 tx=0).stop 2024-05-09T08:45:59.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.553+0000 7f71e9e64640 1 -- 172.21.15.107:0/410719943 shutdown_connections 2024-05-09T08:45:59.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.553+0000 7f71e9e64640 1 --2- 172.21.15.107:0/410719943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f71cc0a2f00 0x7f71cc0a53c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.553+0000 7f71e9e64640 1 --2- 172.21.15.107:0/410719943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f71e41ab720 0x7f71e41abbd0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.553+0000 7f71e9e64640 1 --2- 172.21.15.107:0/410719943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71e4105250 0x7f71e41a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.553+0000 7f71e9e64640 1 --2- 172.21.15.107:0/410719943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71e41003d0 0x7f71e41a6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:45:59.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.553+0000 7f71e9e64640 1 -- 172.21.15.107:0/410719943 >> 172.21.15.107:0/410719943 conn(0x7f71e40759c0 msgr2=0x7f71e40fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:45:59.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.557+0000 7f71e9e64640 1 -- 172.21.15.107:0/410719943 shutdown_connections 2024-05-09T08:45:59.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:45:59.557+0000 7f71e9e64640 1 -- 172.21.15.107:0/410719943 wait complete. 2024-05-09T08:45:59.586 INFO:tasks.cephfs.filesystem:scrub status for tag:9a69fb35-17df-462b-9df8-d3955393209f - {'path': '/', 'tag': '9a69fb35-17df-462b-9df8-d3955393209f', 'options': 'recursive,force'} 2024-05-09T08:46:00.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:00 smithi110 ceph-mon[33622]: pgmap v394: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 13 MiB/s wr, 3.28k op/s 2024-05-09T08:46:00.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:00 smithi107 ceph-mon[30122]: pgmap v394: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 13 MiB/s wr, 3.28k op/s 2024-05-09T08:46:00.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:00 smithi157 ceph-mon[33614]: pgmap v394: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 13 MiB/s wr, 3.28k op/s 2024-05-09T08:46:02.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:02 smithi107 ceph-mon[30122]: pgmap v395: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 10 MiB/s wr, 2.56k op/s 2024-05-09T08:46:02.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:02 smithi157 ceph-mon[33614]: pgmap v395: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 10 MiB/s wr, 2.56k op/s 2024-05-09T08:46:02.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:02 smithi110 ceph-mon[33622]: pgmap v395: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 10 MiB/s wr, 2.56k op/s 2024-05-09T08:46:04.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:04 smithi107 ceph-mon[30122]: pgmap v396: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 13 MiB/s wr, 3.35k op/s 2024-05-09T08:46:04.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:04 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:04.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:04 smithi157 ceph-mon[33614]: pgmap v396: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 13 MiB/s wr, 3.35k op/s 2024-05-09T08:46:04.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:04 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:04.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:04 smithi110 ceph-mon[33622]: pgmap v396: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 13 MiB/s wr, 3.35k op/s 2024-05-09T08:46:04.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:04 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:46:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.779+0000 7fa2f09c0640 1 -- 172.21.15.107:0/2771876004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2e4094e50 msgr2=0x7fa2e40952b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.779+0000 7fa2f09c0640 1 --2- 172.21.15.107:0/2771876004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2e4094e50 0x7fa2e40952b0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0009900 tx=0x7fa2e002a410 comp rx=0 tx=0).stop 2024-05-09T08:46:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.779+0000 7fa2f09c0640 1 -- 172.21.15.107:0/2771876004 shutdown_connections 2024-05-09T08:46:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.779+0000 7fa2f09c0640 1 --2- 172.21.15.107:0/2771876004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2e40957f0 0x7fa2e40a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.779+0000 7fa2f09c0640 1 --2- 172.21.15.107:0/2771876004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2e4094e50 0x7fa2e40952b0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.779+0000 7fa2f09c0640 1 --2- 172.21.15.107:0/2771876004 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2e4099bc0 0x7fa2e4099fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.779+0000 7fa2f09c0640 1 -- 172.21.15.107:0/2771876004 >> 172.21.15.107:0/2771876004 conn(0x7fa2e400a740 msgr2=0x7fa2e400ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:04.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.779+0000 7fa2f09c0640 1 -- 172.21.15.107:0/2771876004 shutdown_connections 2024-05-09T08:46:04.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.779+0000 7fa2f09c0640 1 -- 172.21.15.107:0/2771876004 wait complete. 2024-05-09T08:46:04.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.780+0000 7fa2f09c0640 1 Processor -- start 2024-05-09T08:46:04.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.780+0000 7fa2f09c0640 1 -- start start 2024-05-09T08:46:04.782 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.780+0000 7fa2f09c0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2e4094e50 0x7fa2e412c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.782 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.780+0000 7fa2f09c0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2e40957f0 0x7fa2e412ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.782 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.780+0000 7fa2f09c0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2e4099bc0 0x7fa2e4130da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.782 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.780+0000 7fa2f09c0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e40a5f80 con 0x7fa2e40957f0 2024-05-09T08:46:04.782 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.780+0000 7fa2f09c0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e40a5e00 con 0x7fa2e4099bc0 2024-05-09T08:46:04.782 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.780+0000 7fa2f09c0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e40a6100 con 0x7fa2e4094e50 2024-05-09T08:46:04.782 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.780+0000 7fa2ead9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2e4094e50 0x7fa2e412c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.780+0000 7fa2ead9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2e4094e50 0x7fa2e412c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:37070/0 (socket says 172.21.15.107:37070) 2024-05-09T08:46:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.780+0000 7fa2eb59e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2e4099bc0 0x7fa2e4130da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.780+0000 7fa2ead9d640 1 -- 172.21.15.107:0/3336049881 learned_addr learned my addr 172.21.15.107:0/3336049881 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.780+0000 7fa2ea59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2e40957f0 0x7fa2e412ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.781+0000 7fa2ead9d640 1 -- 172.21.15.107:0/3336049881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2e4099bc0 msgr2=0x7fa2e4130da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.781+0000 7fa2ead9d640 1 --2- 172.21.15.107:0/3336049881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2e4099bc0 0x7fa2e4130da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.781+0000 7fa2ead9d640 1 -- 172.21.15.107:0/3336049881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2e40957f0 msgr2=0x7fa2e412ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.781+0000 7fa2ead9d640 1 --2- 172.21.15.107:0/3336049881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2e40957f0 0x7fa2e412ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.781+0000 7fa2ead9d640 1 -- 172.21.15.107:0/3336049881 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e41314d0 con 0x7fa2e4094e50 2024-05-09T08:46:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.781+0000 7fa2ead9d640 1 --2- 172.21.15.107:0/3336049881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2e4094e50 0x7fa2e412c4e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc00e9f0 tx=0x7fa2dc00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.781+0000 7fa2d3fff640 1 -- 172.21.15.107:0/3336049881 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc00cd80 con 0x7fa2e4094e50 2024-05-09T08:46:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.781+0000 7fa2d3fff640 1 -- 172.21.15.107:0/3336049881 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa2dc0054d0 con 0x7fa2e4094e50 2024-05-09T08:46:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.781+0000 7fa2f09c0640 1 -- 172.21.15.107:0/3336049881 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2e41317c0 con 0x7fa2e4094e50 2024-05-09T08:46:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.781+0000 7fa2d3fff640 1 -- 172.21.15.107:0/3336049881 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc010490 con 0x7fa2e4094e50 2024-05-09T08:46:04.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.781+0000 7fa2f09c0640 1 -- 172.21.15.107:0/3336049881 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2e4139050 con 0x7fa2e4094e50 2024-05-09T08:46:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.783+0000 7fa2d3fff640 1 -- 172.21.15.107:0/3336049881 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fa2dc005020 con 0x7fa2e4094e50 2024-05-09T08:46:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2f3fac740 1 Processor -- start 2024-05-09T08:46:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2f3fac740 1 -- start start 2024-05-09T08:46:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2f3fac740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f4339c600 0x555f4339c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2f3fac740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339cf20 0x555f433a72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2f3fac740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f433a77e0 0x555f433a9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2f3fac740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555f43301340 con 0x555f4339c600 2024-05-09T08:46:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2f3fac740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555f432a9290 con 0x555f4339cf20 2024-05-09T08:46:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2f3fac740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x555f432fd300 con 0x555f433a77e0 2024-05-09T08:46:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2eb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f4339c600 0x555f4339c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2ead9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339cf20 0x555f433a72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2eb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f4339c600 0x555f4339c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45478/0 (socket says 172.21.15.107:45478) 2024-05-09T08:46:04.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2eb59e640 1 -- 172.21.15.107:0/322128323 learned_addr learned my addr 172.21.15.107:0/322128323 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:04.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2ea59c640 1 --2- 172.21.15.107:0/322128323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f433a77e0 0x555f433a9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2d3fff640 1 --2- 172.21.15.107:0/3336049881 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa2d40a2d80 0x7fa2d40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2eb59e640 1 --2- 172.21.15.107:0/3336049881 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa2d40a2d80 0x7fa2d40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2ead9d640 1 -- 172.21.15.107:0/322128323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f433a77e0 msgr2=0x555f433a9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:04.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2ead9d640 1 --2- 172.21.15.107:0/322128323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f433a77e0 0x555f433a9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2ead9d640 1 -- 172.21.15.107:0/322128323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f4339c600 msgr2=0x555f4339c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:04.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2ead9d640 1 --2- 172.21.15.107:0/322128323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f4339c600 0x555f4339c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2ead9d640 1 -- 172.21.15.107:0/322128323 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f432cd8e0 con 0x555f4339cf20 2024-05-09T08:46:04.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2ea59c640 1 --2- 172.21.15.107:0/322128323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f433a77e0 0x555f433a9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:46:04.797 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.784+0000 7fa2eb59e640 1 --2- 172.21.15.107:0/322128323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f4339c600 0x555f4339c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:46:04.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.785+0000 7fa2d3fff640 1 -- 172.21.15.107:0/3336049881 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7fa2dc0cfc10 con 0x7fa2e4094e50 2024-05-09T08:46:04.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.785+0000 7fa2eb59e640 1 --2- 172.21.15.107:0/3336049881 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa2d40a2d80 0x7fa2d40a5240 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec0503a0 tx=0x7fa2ec0677a0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:04.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.785+0000 7fa2ead9d640 1 --2- 172.21.15.107:0/322128323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339cf20 0x555f433a72a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc0ccf00 tx=0x7fa2dc021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac347d9e2376e780 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:04.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.785+0000 7fa2d17fa640 1 -- 172.21.15.107:0/322128323 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc0d4040 con 0x555f4339cf20 2024-05-09T08:46:04.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.785+0000 7fa2d17fa640 1 -- 172.21.15.107:0/322128323 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa2dc005310 con 0x555f4339cf20 2024-05-09T08:46:04.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.785+0000 7fa2d17fa640 1 -- 172.21.15.107:0/322128323 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc0a0080 con 0x555f4339cf20 2024-05-09T08:46:04.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.785+0000 7fa2f3fac740 1 -- 172.21.15.107:0/322128323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339cf20 msgr2=0x555f433a72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:04.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/322128323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339cf20 0x555f433a72a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc0ccf00 tx=0x7fa2dc021000 comp rx=0 tx=0).stop 2024-05-09T08:46:04.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 1 -- 172.21.15.107:0/322128323 shutdown_connections 2024-05-09T08:46:04.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/322128323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f433a77e0 0x555f433a9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/322128323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339cf20 0x555f433a72a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/322128323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f4339c600 0x555f4339c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.798 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 1 -- 172.21.15.107:0/322128323 >> 172.21.15.107:0/322128323 conn(0x555f43398b10 msgr2=0x555f43398ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 1 -- 172.21.15.107:0/322128323 shutdown_connections 2024-05-09T08:46:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 1 -- 172.21.15.107:0/322128323 wait complete. 2024-05-09T08:46:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 1 -- start start 2024-05-09T08:46:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 1 Processor -- start 2024-05-09T08:46:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f4339c600 0x555f433a4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f4339cf20 0x555f433a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f433a77e0 0x555f433a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555f433aa3b0 con 0x555f4339c600 2024-05-09T08:46:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555f432bc790 con 0x555f433a77e0 2024-05-09T08:46:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x555f433aa530 con 0x555f4339cf20 2024-05-09T08:46:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2f3fac740 10 client.0 _ll_register_callbacks cb 0x555f432f7a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2ead9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f4339c600 0x555f433a4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2ead9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f4339c600 0x555f433a4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45494/0 (socket says 172.21.15.107:45494) 2024-05-09T08:46:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2ead9d640 1 -- 172.21.15.107:0/2304440535 learned_addr learned my addr 172.21.15.107:0/2304440535 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2ea59c640 1 --2- 172.21.15.107:0/2304440535 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f4339cf20 0x555f433a4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.786+0000 7fa2eb59e640 1 --2- 172.21.15.107:0/2304440535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f433a77e0 0x555f433a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2ead9d640 1 -- 172.21.15.107:0/2304440535 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f4339cf20 msgr2=0x555f433a4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2ead9d640 1 --2- 172.21.15.107:0/2304440535 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f4339cf20 0x555f433a4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2ead9d640 1 -- 172.21.15.107:0/2304440535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f433a77e0 msgr2=0x555f433a51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2ead9d640 1 --2- 172.21.15.107:0/2304440535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f433a77e0 0x555f433a51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2ead9d640 1 -- 172.21.15.107:0/2304440535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f43015210 con 0x555f4339c600 2024-05-09T08:46:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2eb59e640 1 --2- 172.21.15.107:0/2304440535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f433a77e0 0x555f433a51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:46:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2ead9d640 1 --2- 172.21.15.107:0/2304440535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f4339c600 0x555f433a4750 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc09b9d0 tx=0x7fa2dc00c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f750c0784fd91c27 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2ceff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:46:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2ceff5640 1 -- 172.21.15.107:0/2304440535 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc09c3d0 con 0x555f4339c600 2024-05-09T08:46:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2ceff5640 1 -- 172.21.15.107:0/2304440535 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7fa2dc098340 con 0x555f4339c600 2024-05-09T08:46:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2304440535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555f4330ea70 con 0x555f4339c600 2024-05-09T08:46:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2ceff5640 1 -- 172.21.15.107:0/2304440535 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa2dc0a4940 con 0x555f4339c600 2024-05-09T08:46:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2f3fac740 10 client.18972 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2304440535 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555f430e1620 con 0x555f4339c600 2024-05-09T08:46:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2ceff5640 1 -- 172.21.15.107:0/2304440535 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x555f4330ea70 con 0x555f4339c600 2024-05-09T08:46:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2ceff5640 1 -- 172.21.15.107:0/2304440535 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa2dc097a20 con 0x555f4339c600 2024-05-09T08:46:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2f3fac740 10 client.18972 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2f3fac740 10 client.18972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:46:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2f3fac740 20 client.18972 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/2304440535 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555f433b7330 0x555f433b9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2f3fac740 4 client.18972 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:46:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.787+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2304440535 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f43308dc0 con 0x555f433b7330 2024-05-09T08:46:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.788+0000 7fa2ea59c640 1 --2- 172.21.15.107:0/2304440535 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555f433b7330 0x555f433b9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.788+0000 7fa2ceff5640 10 client.18972 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.788+0000 7fa2ea59c640 1 --2- 172.21.15.107:0/2304440535 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555f433b7330 0x555f433b9720 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=663dee7c3ef214f6 server_cookie=a545f90594aa82cf in_seq=0 out_seq=0 2024-05-09T08:46:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.789+0000 7fa2ceff5640 1 -- 172.21.15.107:0/2304440535 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f43308dc0 con 0x555f433b7330 2024-05-09T08:46:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.789+0000 7fa2ceff5640 10 client.18972 handle_command_reply: tid=0 2024-05-09T08:46:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.789+0000 7fa2f3fac740 1 client.18972 shutdown 2024-05-09T08:46:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.789+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2304440535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f4339c600 msgr2=0x555f433a4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.789+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/2304440535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f4339c600 0x555f433a4750 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc09b9d0 tx=0x7fa2dc00c7d0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.789+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2304440535 shutdown_connections 2024-05-09T08:46:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.789+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/2304440535 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555f433b7330 0x555f433b9720 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.789+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/2304440535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f433a77e0 0x555f433a51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.789+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/2304440535 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f4339cf20 0x555f433a4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.789+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/2304440535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f4339c600 0x555f433a4750 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.789+0000 7fa2ceff5640 0 client.18972 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.789+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2304440535 >> 172.21.15.107:0/2304440535 conn(0x555f43398b10 msgr2=0x555f4339e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.789+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2304440535 shutdown_connections 2024-05-09T08:46:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.789+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2304440535 wait complete. 2024-05-09T08:46:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.789+0000 7fa2f3fac740 20 client.18972 trim_cache size 0 max 16384 2024-05-09T08:46:04.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2f3fac740 1 Processor -- start 2024-05-09T08:46:04.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2f3fac740 1 -- start start 2024-05-09T08:46:04.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2f3fac740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339c600 0x555f4339efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2f3fac740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f4339cf20 0x555f43397a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2f3fac740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f433a77e0 0x555f433bc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2f3fac740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e4145960 con 0x555f433a77e0 2024-05-09T08:46:04.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2f3fac740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e4143230 con 0x555f4339c600 2024-05-09T08:46:04.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2f3fac740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e4141080 con 0x555f4339cf20 2024-05-09T08:46:04.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2eb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f433a77e0 0x555f433bc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2eb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f433a77e0 0x555f433bc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45498/0 (socket says 172.21.15.107:45498) 2024-05-09T08:46:04.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2eb59e640 1 -- 172.21.15.107:0/2994317655 learned_addr learned my addr 172.21.15.107:0/2994317655 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:04.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2ea59c640 1 --2- 172.21.15.107:0/2994317655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f4339cf20 0x555f43397a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2ead9d640 1 --2- 172.21.15.107:0/2994317655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339c600 0x555f4339efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2eb59e640 1 -- 172.21.15.107:0/2994317655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f4339cf20 msgr2=0x555f43397a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:04.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2eb59e640 1 --2- 172.21.15.107:0/2994317655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f4339cf20 0x555f43397a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2eb59e640 1 -- 172.21.15.107:0/2994317655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339c600 msgr2=0x555f4339efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:04.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2eb59e640 1 --2- 172.21.15.107:0/2994317655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339c600 0x555f4339efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2eb59e640 1 -- 172.21.15.107:0/2994317655 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f433a9ff0 con 0x555f433a77e0 2024-05-09T08:46:04.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2ead9d640 1 --2- 172.21.15.107:0/2994317655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339c600 0x555f4339efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:46:04.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2eb59e640 1 --2- 172.21.15.107:0/2994317655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f433a77e0 0x555f433bc740 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec0674e0 tx=0x7fa2ec068110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87271f00162e4c2d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:04.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2d0ff9640 1 -- 172.21.15.107:0/2994317655 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec08a040 con 0x555f433a77e0 2024-05-09T08:46:04.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2d0ff9640 1 -- 172.21.15.107:0/2994317655 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa2ec0908a0 con 0x555f433a77e0 2024-05-09T08:46:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.804+0000 7fa2d0ff9640 1 -- 172.21.15.107:0/2994317655 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec07c6a0 con 0x555f433a77e0 2024-05-09T08:46:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.805+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2994317655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f433a77e0 msgr2=0x555f433bc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.805+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/2994317655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f433a77e0 0x555f433bc740 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec0674e0 tx=0x7fa2ec068110 comp rx=0 tx=0).stop 2024-05-09T08:46:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.805+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2994317655 shutdown_connections 2024-05-09T08:46:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.805+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/2994317655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f433a77e0 0x555f433bc740 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.805+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/2994317655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f4339cf20 0x555f43397a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.805+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/2994317655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339c600 0x555f4339efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.805+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2994317655 >> 172.21.15.107:0/2994317655 conn(0x555f43398b10 msgr2=0x555f43397320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.805+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2994317655 shutdown_connections 2024-05-09T08:46:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.805+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2994317655 wait complete. 2024-05-09T08:46:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.805+0000 7fa2f3fac740 1 -- start start 2024-05-09T08:46:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.805+0000 7fa2f3fac740 1 Processor -- start 2024-05-09T08:46:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2f3fac740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f4339c600 0x555f433ba8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2f3fac740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339cf20 0x555f433bae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2f3fac740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f433a77e0 0x555f4341c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2f3fac740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e41416f0 con 0x555f433a77e0 2024-05-09T08:46:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2f3fac740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e4142b60 con 0x555f4339cf20 2024-05-09T08:46:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2f3fac740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e4141dc0 con 0x555f4339c600 2024-05-09T08:46:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2f3fac740 10 client.0 _ll_register_callbacks cb 0x7fa2ec068060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2eb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f433a77e0 0x555f4341c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2ea59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339cf20 0x555f433bae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2ead9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f4339c600 0x555f433ba8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2eb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f433a77e0 0x555f4341c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45506/0 (socket says 172.21.15.107:45506) 2024-05-09T08:46:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2eb59e640 1 -- 172.21.15.107:0/2515377865 learned_addr learned my addr 172.21.15.107:0/2515377865 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2eb59e640 1 -- 172.21.15.107:0/2515377865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f4339c600 msgr2=0x555f433ba8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:04.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2eb59e640 1 --2- 172.21.15.107:0/2515377865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f4339c600 0x555f433ba8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2eb59e640 1 -- 172.21.15.107:0/2515377865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339cf20 msgr2=0x555f433bae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:04.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2eb59e640 1 --2- 172.21.15.107:0/2515377865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339cf20 0x555f433bae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:04.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2eb59e640 1 -- 172.21.15.107:0/2515377865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f4341ca50 con 0x555f433a77e0 2024-05-09T08:46:04.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2eb59e640 1 --2- 172.21.15.107:0/2515377865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f433a77e0 0x555f4341c510 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec000b60 tx=0x7fa2ec0906a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffc984528efa9181 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:04.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2ceff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:46:04.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2ceff5640 1 -- 172.21.15.107:0/2515377865 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec07c620 con 0x555f433a77e0 2024-05-09T08:46:04.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2ceff5640 1 -- 172.21.15.107:0/2515377865 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9543+0+0 (secure 0 0 0) 0x7fa2ec07c7c0 con 0x555f433a77e0 2024-05-09T08:46:04.815 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2515377865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2ec0908a0 con 0x555f433a77e0 2024-05-09T08:46:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2ceff5640 1 -- 172.21.15.107:0/2515377865 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa2ec07c9c0 con 0x555f433a77e0 2024-05-09T08:46:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2f3fac740 10 client.18978 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.806+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2515377865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555f4341dd10 con 0x555f433a77e0 2024-05-09T08:46:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.807+0000 7fa2ceff5640 1 -- 172.21.15.107:0/2515377865 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa2ec0908a0 con 0x555f433a77e0 2024-05-09T08:46:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.807+0000 7fa2ceff5640 1 -- 172.21.15.107:0/2515377865 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fa2ec098b30 con 0x555f433a77e0 2024-05-09T08:46:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.807+0000 7fa2f3fac740 10 client.18978 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.807+0000 7fa2f3fac740 10 client.18978 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:46:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.807+0000 7fa2f3fac740 20 client.18978 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.807+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/2515377865 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555f4341e320 0x555f43420710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.807+0000 7fa2f3fac740 4 client.18978 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:46:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.807+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2515377865 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555f432868e0 con 0x555f4341e320 2024-05-09T08:46:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.807+0000 7fa2ead9d640 1 --2- 172.21.15.107:0/2515377865 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555f4341e320 0x555f43420710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.807+0000 7fa2ceff5640 10 client.18978 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:04.807+0000 7fa2ead9d640 1 --2- 172.21.15.107:0/2515377865 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555f4341e320 0x555f43420710 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc1c9707818f4cd0 server_cookie=6e4e9f0a9e1e11ed in_seq=0 out_seq=0 2024-05-09T08:46:06.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:06 smithi107 ceph-mon[30122]: pgmap v397: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 9.3 MiB/s wr, 2.37k op/s 2024-05-09T08:46:06.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:06 smithi157 ceph-mon[33614]: pgmap v397: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 9.3 MiB/s wr, 2.37k op/s 2024-05-09T08:46:06.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:06 smithi110 ceph-mon[33622]: pgmap v397: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 9.3 MiB/s wr, 2.37k op/s 2024-05-09T08:46:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.792+0000 7fa2ceff5640 1 -- 172.21.15.107:0/2515377865 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555f432868e0 con 0x555f4341e320 2024-05-09T08:46:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.792+0000 7fa2ceff5640 10 client.18978 handle_command_reply: tid=0 2024-05-09T08:46:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.792+0000 7fa2f3fac740 1 client.18978 shutdown 2024-05-09T08:46:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.793+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2515377865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f433a77e0 msgr2=0x555f4341c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.793+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/2515377865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f433a77e0 0x555f4341c510 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec000b60 tx=0x7fa2ec0906a0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.793+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2515377865 shutdown_connections 2024-05-09T08:46:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.793+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/2515377865 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555f4341e320 0x555f43420710 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.793+0000 7fa2ceff5640 0 client.18978 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.793+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/2515377865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555f433a77e0 0x555f4341c510 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.793+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/2515377865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555f4339cf20 0x555f433bae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.793+0000 7fa2f3fac740 1 --2- 172.21.15.107:0/2515377865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555f4339c600 0x555f433ba8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.793+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2515377865 >> 172.21.15.107:0/2515377865 conn(0x555f43398b10 msgr2=0x555f43397bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.794+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2515377865 shutdown_connections 2024-05-09T08:46:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.794+0000 7fa2f3fac740 1 -- 172.21.15.107:0/2515377865 wait complete. 2024-05-09T08:46:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.794+0000 7fa2f3fac740 20 client.18978 trim_cache size 0 max 16384 2024-05-09T08:46:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:46:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:46:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:46:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:46:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.795+0000 7fa2f09c0640 1 -- 172.21.15.107:0/3336049881 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa2d40a2d80 msgr2=0x7fa2d40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.795+0000 7fa2f09c0640 1 --2- 172.21.15.107:0/3336049881 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa2d40a2d80 0x7fa2d40a5240 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec0503a0 tx=0x7fa2ec0677a0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.795+0000 7fa2f09c0640 1 -- 172.21.15.107:0/3336049881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2e4094e50 msgr2=0x7fa2e412c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.795+0000 7fa2f09c0640 1 --2- 172.21.15.107:0/3336049881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2e4094e50 0x7fa2e412c4e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc00e9f0 tx=0x7fa2dc00edf0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.795+0000 7fa2f09c0640 1 -- 172.21.15.107:0/3336049881 shutdown_connections 2024-05-09T08:46:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.795+0000 7fa2f09c0640 1 --2- 172.21.15.107:0/3336049881 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa2d40a2d80 0x7fa2d40a5240 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.795+0000 7fa2f09c0640 1 --2- 172.21.15.107:0/3336049881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2e4099bc0 0x7fa2e4130da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.795+0000 7fa2f09c0640 1 --2- 172.21.15.107:0/3336049881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2e40957f0 0x7fa2e412ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.795+0000 7fa2f09c0640 1 --2- 172.21.15.107:0/3336049881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2e4094e50 0x7fa2e412c4e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.795+0000 7fa2f09c0640 1 -- 172.21.15.107:0/3336049881 >> 172.21.15.107:0/3336049881 conn(0x7fa2e400a740 msgr2=0x7fa2e4093060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.795+0000 7fa2f09c0640 1 -- 172.21.15.107:0/3336049881 shutdown_connections 2024-05-09T08:46:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:06.795+0000 7fa2f09c0640 1 -- 172.21.15.107:0/3336049881 wait complete. 2024-05-09T08:46:07.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:07 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:46:07.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:07 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:07.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:07 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:46:07.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:07 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:07.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:07 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:46:07.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:07 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:07.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:07 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:46:07.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:07 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:07.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:07 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:46:07.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:07 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:07.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:07 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:46:07.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:07 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:08.495 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:08 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:46:08.495 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:08 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:08.495 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:08 smithi157 ceph-mon[33614]: pgmap v398: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 10 MiB/s wr, 2.63k op/s 2024-05-09T08:46:08.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:08 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:46:08.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:08 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:08.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:08 smithi107 ceph-mon[30122]: pgmap v398: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 10 MiB/s wr, 2.63k op/s 2024-05-09T08:46:08.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:08 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:46:08.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:08 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:08.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:08 smithi110 ceph-mon[33622]: pgmap v398: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 10 MiB/s wr, 2.63k op/s 2024-05-09T08:46:09.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:09 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:46:09.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:09 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:09.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:09 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:46:09.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:09 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:09.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:09 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:46:09.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:09 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:46:09.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:09 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:46:09.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:09 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:09.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:09 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:46:09.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:09 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:09.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:09 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:46:09.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:09 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:46:09.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:09 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:46:09.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:09 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:09.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:09 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:46:09.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:09 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:09.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:09 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:46:09.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:09 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:46:10.522 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:10 smithi110 ceph-mon[33622]: pgmap v399: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-05-09T08:46:10.522 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:10 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:10.522 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:10 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:10.522 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:10 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:46:10.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:10 smithi107 ceph-mon[30122]: pgmap v399: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-05-09T08:46:10.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:10 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:10.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:10 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:10.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:10 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:46:10.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:10 smithi157 ceph-mon[33614]: pgmap v399: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-05-09T08:46:10.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:10 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:46:10.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:10 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:46:10.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:10 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:46:12.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:12 smithi107 ceph-mon[30122]: pgmap v400: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 8.6 MiB/s wr, 2.20k op/s 2024-05-09T08:46:12.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:12 smithi107 ceph-mon[30122]: osdmap e107: 12 total, 12 up, 12 in 2024-05-09T08:46:12.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:12 smithi157 ceph-mon[33614]: pgmap v400: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 8.6 MiB/s wr, 2.20k op/s 2024-05-09T08:46:12.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:12 smithi157 ceph-mon[33614]: osdmap e107: 12 total, 12 up, 12 in 2024-05-09T08:46:12.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:12 smithi110 ceph-mon[33622]: pgmap v400: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 8.6 MiB/s wr, 2.20k op/s 2024-05-09T08:46:12.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:12 smithi110 ceph-mon[33622]: osdmap e107: 12 total, 12 up, 12 in 2024-05-09T08:46:13.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:13 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:13.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:13 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:13.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:13 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:14.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:14 smithi107 ceph-mon[30122]: pgmap v402: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-05-09T08:46:14.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:14 smithi157 ceph-mon[33614]: pgmap v402: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-05-09T08:46:14.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:14 smithi110 ceph-mon[33622]: pgmap v402: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-05-09T08:46:16.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:16 smithi107 ceph-mon[30122]: pgmap v403: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 10 MiB/s wr, 2.62k op/s 2024-05-09T08:46:16.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:16 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:46:16.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:16 smithi157 ceph-mon[33614]: pgmap v403: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 10 MiB/s wr, 2.62k op/s 2024-05-09T08:46:16.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:16 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:46:16.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:16 smithi110 ceph-mon[33622]: pgmap v403: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 10 MiB/s wr, 2.62k op/s 2024-05-09T08:46:16.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:16 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:46:17.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:17 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:46:17.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:17 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:46:17.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:17 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:46:17.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:17 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:46:17.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:17 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:46:17.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:17 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:46:18.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:18 smithi107 ceph-mon[30122]: pgmap v404: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 9.2 MiB/s wr, 2.32k op/s 2024-05-09T08:46:18.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:18 smithi157 ceph-mon[33614]: pgmap v404: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 9.2 MiB/s wr, 2.32k op/s 2024-05-09T08:46:18.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:18 smithi110 ceph-mon[33622]: pgmap v404: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 9.2 MiB/s wr, 2.32k op/s 2024-05-09T08:46:19.934 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.934+0000 7fb135d42640 1 -- 172.21.15.107:0/456676872 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb130106720 msgr2=0x7fb130106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.934+0000 7fb135d42640 1 --2- 172.21.15.107:0/456676872 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb130106720 0x7fb130106b00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb118009900 tx=0x7fb11802a410 comp rx=0 tx=0).stop 2024-05-09T08:46:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.935+0000 7fb135d42640 1 -- 172.21.15.107:0/456676872 shutdown_connections 2024-05-09T08:46:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.935+0000 7fb135d42640 1 --2- 172.21.15.107:0/456676872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb130101120 0x7fb13010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.935+0000 7fb135d42640 1 --2- 172.21.15.107:0/456676872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb130100760 0x7fb130100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.935+0000 7fb135d42640 1 --2- 172.21.15.107:0/456676872 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb130106720 0x7fb130106b00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.935+0000 7fb135d42640 1 -- 172.21.15.107:0/456676872 >> 172.21.15.107:0/456676872 conn(0x7fb1300778b0 msgr2=0x7fb130077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.935+0000 7fb135d42640 1 -- 172.21.15.107:0/456676872 shutdown_connections 2024-05-09T08:46:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.935+0000 7fb135d42640 1 -- 172.21.15.107:0/456676872 wait complete. 2024-05-09T08:46:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.936+0000 7fb135d42640 1 Processor -- start 2024-05-09T08:46:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.936+0000 7fb135d42640 1 -- start start 2024-05-09T08:46:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.936+0000 7fb135d42640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb130100760 0x7fb13019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.936+0000 7fb135d42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb130101120 0x7fb13019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.936+0000 7fb135d42640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb130106720 0x7fb13019ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.936+0000 7fb135d42640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb13010f940 con 0x7fb130101120 2024-05-09T08:46:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.936+0000 7fb135d42640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb13010f7c0 con 0x7fb130100760 2024-05-09T08:46:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.936+0000 7fb135d42640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb13010fac0 con 0x7fb130106720 2024-05-09T08:46:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.936+0000 7fb127fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb130101120 0x7fb13019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.936+0000 7fb12ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb130106720 0x7fb13019ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.936+0000 7fb127fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb130101120 0x7fb13019a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41012/0 (socket says 172.21.15.107:41012) 2024-05-09T08:46:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.936+0000 7fb127fff640 1 -- 172.21.15.107:0/442665213 learned_addr learned my addr 172.21.15.107:0/442665213 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.936+0000 7fb12ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb130106720 0x7fb13019ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:52204/0 (socket says 172.21.15.107:52204) 2024-05-09T08:46:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.937+0000 7fb12f7fe640 1 --2- 172.21.15.107:0/442665213 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb130100760 0x7fb13019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.937+0000 7fb12f7fe640 1 -- 172.21.15.107:0/442665213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb130106720 msgr2=0x7fb13019ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.937+0000 7fb12f7fe640 1 --2- 172.21.15.107:0/442665213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb130106720 0x7fb13019ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.937+0000 7fb12f7fe640 1 -- 172.21.15.107:0/442665213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb130101120 msgr2=0x7fb13019a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.937+0000 7fb12f7fe640 1 --2- 172.21.15.107:0/442665213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb130101120 0x7fb13019a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.937+0000 7fb12f7fe640 1 -- 172.21.15.107:0/442665213 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13019f160 con 0x7fb130100760 2024-05-09T08:46:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.937+0000 7fb127fff640 1 --2- 172.21.15.107:0/442665213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb130101120 0x7fb13019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:46:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.937+0000 7fb12f7fe640 1 --2- 172.21.15.107:0/442665213 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb130100760 0x7fb13019a170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb11802aa00 tx=0x7fb118003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.937+0000 7fb12d7fa640 1 -- 172.21.15.107:0/442665213 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11803f070 con 0x7fb130100760 2024-05-09T08:46:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.937+0000 7fb135d42640 1 -- 172.21.15.107:0/442665213 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb13019f3f0 con 0x7fb130100760 2024-05-09T08:46:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.937+0000 7fb12d7fa640 1 -- 172.21.15.107:0/442665213 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb118031b50 con 0x7fb130100760 2024-05-09T08:46:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.937+0000 7fb12d7fa640 1 -- 172.21.15.107:0/442665213 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb118035030 con 0x7fb130100760 2024-05-09T08:46:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.937+0000 7fb135d42640 1 -- 172.21.15.107:0/442665213 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1301a6ce0 con 0x7fb130100760 2024-05-09T08:46:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.940+0000 7fb138a51740 1 Processor -- start 2024-05-09T08:46:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.940+0000 7fb138a51740 1 -- start start 2024-05-09T08:46:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.940+0000 7fb138a51740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5450 0x5646744c5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.940+0000 7fb138a51740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744c5d70 0x5646744d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.940+0000 7fb138a51740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744d0630 0x5646744d2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.940+0000 7fb138a51740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56467442a340 con 0x5646744c5450 2024-05-09T08:46:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.940+0000 7fb138a51740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5646743d2290 con 0x5646744d0630 2024-05-09T08:46:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.940+0000 7fb138a51740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x564674426300 con 0x5646744c5d70 2024-05-09T08:46:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.940+0000 7fb12d7fa640 1 -- 172.21.15.107:0/442665213 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb118042070 con 0x7fb130100760 2024-05-09T08:46:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.940+0000 7fb127fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744d0630 0x5646744d2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.941+0000 7fb127fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744d0630 0x5646744d2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:35270/0 (socket says 172.21.15.107:35270) 2024-05-09T08:46:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.941+0000 7fb127fff640 1 -- 172.21.15.107:0/2563815951 learned_addr learned my addr 172.21.15.107:0/2563815951 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.941+0000 7fb12ffff640 1 --2- 172.21.15.107:0/2563815951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5450 0x5646744c5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.941+0000 7fb12f7fe640 1 --2- 172.21.15.107:0/2563815951 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744c5d70 0x5646744d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.941+0000 7fb12ffff640 1 -- 172.21.15.107:0/2563815951 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744c5d70 msgr2=0x5646744d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.941+0000 7fb12ffff640 1 --2- 172.21.15.107:0/2563815951 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744c5d70 0x5646744d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.941+0000 7fb12ffff640 1 -- 172.21.15.107:0/2563815951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744d0630 msgr2=0x5646744d2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.941+0000 7fb12ffff640 1 --2- 172.21.15.107:0/2563815951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744d0630 0x5646744d2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.941+0000 7fb12ffff640 1 -- 172.21.15.107:0/2563815951 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646743f68e0 con 0x5646744c5450 2024-05-09T08:46:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.941+0000 7fb12ffff640 1 --2- 172.21.15.107:0/2563815951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5450 0x5646744c5830 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fb128009f00 tx=0x7fb128011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c59e38ad174ccc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.942+0000 7fb12d7fa640 1 --2- 172.21.15.107:0/442665213 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb11c0a2e20 0x7fb11c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.942+0000 7fb125ffb640 1 -- 172.21.15.107:0/2563815951 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb128004d20 con 0x5646744c5450 2024-05-09T08:46:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.942+0000 7fb125ffb640 1 -- 172.21.15.107:0/2563815951 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb128004050 con 0x5646744c5450 2024-05-09T08:46:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.942+0000 7fb125ffb640 1 -- 172.21.15.107:0/2563815951 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb128005280 con 0x5646744c5450 2024-05-09T08:46:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.942+0000 7fb12d7fa640 1 -- 172.21.15.107:0/442665213 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9527+0+0 (secure 0 0 0) 0x7fb1180ef9a0 con 0x7fb130100760 2024-05-09T08:46:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.942+0000 7fb12f7fe640 1 --2- 172.21.15.107:0/442665213 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb11c0a2e20 0x7fb11c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.943+0000 7fb138a51740 1 -- 172.21.15.107:0/2563815951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5450 msgr2=0x5646744c5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.943+0000 7fb12f7fe640 1 --2- 172.21.15.107:0/442665213 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb11c0a2e20 0x7fb11c0a52e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x5646744c69f0 tx=0x7fb118034000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.943+0000 7fb138a51740 1 --2- 172.21.15.107:0/2563815951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5450 0x5646744c5830 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fb128009f00 tx=0x7fb128011810 comp rx=0 tx=0).stop 2024-05-09T08:46:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.943+0000 7fb138a51740 1 -- 172.21.15.107:0/2563815951 shutdown_connections 2024-05-09T08:46:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.943+0000 7fb138a51740 1 --2- 172.21.15.107:0/2563815951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744d0630 0x5646744d2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.943+0000 7fb138a51740 1 --2- 172.21.15.107:0/2563815951 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744c5d70 0x5646744d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.943+0000 7fb138a51740 1 --2- 172.21.15.107:0/2563815951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5450 0x5646744c5830 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.943+0000 7fb138a51740 1 -- 172.21.15.107:0/2563815951 >> 172.21.15.107:0/2563815951 conn(0x5646744c1960 msgr2=0x5646744c1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.943+0000 7fb138a51740 1 -- 172.21.15.107:0/2563815951 shutdown_connections 2024-05-09T08:46:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.943+0000 7fb138a51740 1 -- 172.21.15.107:0/2563815951 wait complete. 2024-05-09T08:46:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.943+0000 7fb138a51740 1 -- start start 2024-05-09T08:46:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.944+0000 7fb138a51740 1 Processor -- start 2024-05-09T08:46:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.944+0000 7fb138a51740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744c5450 0x5646744cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.944+0000 7fb138a51740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5d70 0x5646744cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb138a51740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744d0630 0x5646744ce070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb127fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5d70 0x5646744cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb127fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5d70 0x5646744cdb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41038/0 (socket says 172.21.15.107:41038) 2024-05-09T08:46:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb127fff640 1 -- 172.21.15.107:0/2417306414 learned_addr learned my addr 172.21.15.107:0/2417306414 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb12ffff640 1 --2- 172.21.15.107:0/2417306414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744d0630 0x5646744ce070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb138a51740 1 -- 172.21.15.107:0/2417306414 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5646744d3260 con 0x5646744c5d70 2024-05-09T08:46:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb138a51740 1 -- 172.21.15.107:0/2417306414 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5646743e5790 con 0x5646744d0630 2024-05-09T08:46:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb138a51740 1 -- 172.21.15.107:0/2417306414 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5646744d33e0 con 0x5646744c5450 2024-05-09T08:46:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb138a51740 10 client.0 _ll_register_callbacks cb 0x564674420a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb12ffff640 1 --2- 172.21.15.107:0/2417306414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744c5450 0x5646744cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb127fff640 1 -- 172.21.15.107:0/2417306414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744c5450 msgr2=0x5646744cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb127fff640 1 --2- 172.21.15.107:0/2417306414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744c5450 0x5646744cd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb127fff640 1 -- 172.21.15.107:0/2417306414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744d0630 msgr2=0x5646744ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb127fff640 1 --2- 172.21.15.107:0/2417306414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744d0630 0x5646744ce070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb127fff640 1 -- 172.21.15.107:0/2417306414 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56467413e210 con 0x5646744c5d70 2024-05-09T08:46:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb12ffff640 1 --2- 172.21.15.107:0/2417306414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744d0630 0x5646744ce070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:46:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.945+0000 7fb138a51740 1 -- 172.21.15.107:0/2417306414 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb128004050 con 0x5646744c5d70 2024-05-09T08:46:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.946+0000 7fb127fff640 1 --2- 172.21.15.107:0/2417306414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5d70 0x5646744cdb30 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fb120012d30 tx=0x7fb12000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3fae3a7e90892f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.946+0000 7fb10effd640 10 client.18990 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:46:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.946+0000 7fb10effd640 1 -- 172.21.15.107:0/2417306414 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb120004050 con 0x5646744c5d70 2024-05-09T08:46:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.946+0000 7fb10effd640 1 -- 172.21.15.107:0/2417306414 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9527+0+0 (secure 0 0 0) 0x7fb1200041f0 con 0x5646744c5d70 2024-05-09T08:46:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.946+0000 7fb10effd640 1 -- 172.21.15.107:0/2417306414 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb120004470 con 0x5646744c5d70 2024-05-09T08:46:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.946+0000 7fb138a51740 10 client.18990 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.946+0000 7fb138a51740 1 -- 172.21.15.107:0/2417306414 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({fsmap=27}) -- 0x56467420a620 con 0x5646744c5d70 2024-05-09T08:46:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.947+0000 7fb10effd640 1 -- 172.21.15.107:0/2417306414 <== mon.0 v2:172.21.15.107:3300/0 4 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fb1200035f0 con 0x5646744c5d70 2024-05-09T08:46:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.947+0000 7fb138a51740 10 client.18990 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.947+0000 7fb138a51740 10 client.18990 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:46:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.947+0000 7fb138a51740 20 client.18990 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.947+0000 7fb138a51740 1 --2- 172.21.15.107:0/2417306414 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5646744e07e0 0x5646744e2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.947+0000 7fb138a51740 4 client.18990 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:46:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.947+0000 7fb138a51740 1 -- 172.21.15.107:0/2417306414 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564674437a70 con 0x5646744e07e0 2024-05-09T08:46:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.947+0000 7fb12ffff640 1 --2- 172.21.15.107:0/2417306414 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5646744e07e0 0x5646744e2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.947+0000 7fb12ffff640 1 --2- 172.21.15.107:0/2417306414 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5646744e07e0 0x5646744e2c00 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baf84dd69b9e5220 server_cookie=97e0ac51df222d09 in_seq=0 out_seq=0 2024-05-09T08:46:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.948+0000 7fb10effd640 10 client.18990 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.948+0000 7fb10effd640 1 -- 172.21.15.107:0/2417306414 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564674437a70 con 0x5646744e07e0 2024-05-09T08:46:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.948+0000 7fb10effd640 10 client.18990 handle_command_reply: tid=0 2024-05-09T08:46:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.948+0000 7fb138a51740 1 client.18990 shutdown 2024-05-09T08:46:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.949+0000 7fb138a51740 1 -- 172.21.15.107:0/2417306414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5d70 msgr2=0x5646744cdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.949+0000 7fb138a51740 1 --2- 172.21.15.107:0/2417306414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5d70 0x5646744cdb30 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fb120012d30 tx=0x7fb12000f790 comp rx=0 tx=0).stop 2024-05-09T08:46:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.949+0000 7fb138a51740 1 -- 172.21.15.107:0/2417306414 shutdown_connections 2024-05-09T08:46:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.949+0000 7fb138a51740 1 --2- 172.21.15.107:0/2417306414 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5646744e07e0 0x5646744e2c00 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.949+0000 7fb138a51740 1 --2- 172.21.15.107:0/2417306414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744d0630 0x5646744ce070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.949+0000 7fb138a51740 1 --2- 172.21.15.107:0/2417306414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5d70 0x5646744cdb30 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.949+0000 7fb138a51740 1 --2- 172.21.15.107:0/2417306414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744c5450 0x5646744cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.949+0000 7fb138a51740 1 -- 172.21.15.107:0/2417306414 >> 172.21.15.107:0/2417306414 conn(0x5646744c1960 msgr2=0x5646744c76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.949+0000 7fb10effd640 0 client.18990 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.950+0000 7fb138a51740 1 -- 172.21.15.107:0/2417306414 shutdown_connections 2024-05-09T08:46:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.950+0000 7fb138a51740 1 -- 172.21.15.107:0/2417306414 wait complete. 2024-05-09T08:46:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.950+0000 7fb138a51740 20 client.18990 trim_cache size 0 max 16384 2024-05-09T08:46:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.970+0000 7fb138a51740 1 Processor -- start 2024-05-09T08:46:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.971+0000 7fb138a51740 1 -- start start 2024-05-09T08:46:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.971+0000 7fb138a51740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5450 0x5646744cfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.971+0000 7fb138a51740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744c5d70 0x5646744dc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.971+0000 7fb138a51740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744d0630 0x5646744e84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.971+0000 7fb138a51740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb1301b36f0 con 0x5646744c5450 2024-05-09T08:46:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.971+0000 7fb138a51740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb1301b0fc0 con 0x5646744c5d70 2024-05-09T08:46:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.971+0000 7fb138a51740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb1301aee10 con 0x5646744d0630 2024-05-09T08:46:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.971+0000 7fb12ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744d0630 0x5646744e84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.971+0000 7fb12ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744d0630 0x5646744e84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:52234/0 (socket says 172.21.15.107:52234) 2024-05-09T08:46:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.972+0000 7fb12ffff640 1 -- 172.21.15.107:0/3136422529 learned_addr learned my addr 172.21.15.107:0/3136422529 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.972+0000 7fb12ffff640 1 --2- 172.21.15.107:0/3136422529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5450 0x5646744cfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.972+0000 7fb12ffff640 1 -- 172.21.15.107:0/3136422529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744c5d70 msgr2=0x5646744dc120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:46:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.972+0000 7fb12ffff640 1 --2- 172.21.15.107:0/3136422529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744c5d70 0x5646744dc120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.972+0000 7fb12ffff640 1 -- 172.21.15.107:0/3136422529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5450 msgr2=0x5646744cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.972+0000 7fb12ffff640 1 --2- 172.21.15.107:0/3136422529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5450 0x5646744cfd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.972+0000 7fb12ffff640 1 -- 172.21.15.107:0/3136422529 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646744c5ba0 con 0x5646744d0630 2024-05-09T08:46:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.972+0000 7fb12ffff640 1 --2- 172.21.15.107:0/3136422529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744d0630 0x5646744e84f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb128006010 tx=0x7fb128011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f45ba54f88194444 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.973+0000 7fb1257fa640 1 -- 172.21.15.107:0/3136422529 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb128003a30 con 0x5646744d0630 2024-05-09T08:46:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.973+0000 7fb1257fa640 1 -- 172.21.15.107:0/3136422529 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb128003bd0 con 0x5646744d0630 2024-05-09T08:46:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.973+0000 7fb1257fa640 1 -- 172.21.15.107:0/3136422529 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb128019670 con 0x5646744d0630 2024-05-09T08:46:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.974+0000 7fb138a51740 1 -- 172.21.15.107:0/3136422529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744d0630 msgr2=0x5646744e84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.974+0000 7fb138a51740 1 --2- 172.21.15.107:0/3136422529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744d0630 0x5646744e84f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb128006010 tx=0x7fb128011810 comp rx=0 tx=0).stop 2024-05-09T08:46:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.975+0000 7fb138a51740 1 -- 172.21.15.107:0/3136422529 shutdown_connections 2024-05-09T08:46:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.975+0000 7fb138a51740 1 --2- 172.21.15.107:0/3136422529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744d0630 0x5646744e84f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.975+0000 7fb138a51740 1 --2- 172.21.15.107:0/3136422529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744c5d70 0x5646744dc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.975+0000 7fb138a51740 1 --2- 172.21.15.107:0/3136422529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744c5450 0x5646744cfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.975+0000 7fb138a51740 1 -- 172.21.15.107:0/3136422529 >> 172.21.15.107:0/3136422529 conn(0x5646744c1960 msgr2=0x5646744c0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.978+0000 7fb138a51740 1 -- 172.21.15.107:0/3136422529 shutdown_connections 2024-05-09T08:46:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.980+0000 7fb138a51740 1 -- 172.21.15.107:0/3136422529 wait complete. 2024-05-09T08:46:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.981+0000 7fb138a51740 1 -- start start 2024-05-09T08:46:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.981+0000 7fb138a51740 1 Processor -- start 2024-05-09T08:46:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.981+0000 7fb138a51740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744c5450 0x5646744ca240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.982+0000 7fb138a51740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744d0630 0x5646744ca780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.982+0000 7fb138a51740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744cacc0 0x564674545d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.982+0000 7fb138a51740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb1301af480 con 0x5646744d0630 2024-05-09T08:46:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.982+0000 7fb138a51740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb1301b08f0 con 0x5646744cacc0 2024-05-09T08:46:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.982+0000 7fb138a51740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb1301afb50 con 0x5646744c5450 2024-05-09T08:46:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.982+0000 7fb138a51740 10 client.0 _ll_register_callbacks cb 0x7fb120009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.982+0000 7fb127fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744d0630 0x5646744ca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.982+0000 7fb12ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744c5450 0x5646744ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.982+0000 7fb127fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744d0630 0x5646744ca780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41056/0 (socket says 172.21.15.107:41056) 2024-05-09T08:46:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.982+0000 7fb127fff640 1 -- 172.21.15.107:0/2642964209 learned_addr learned my addr 172.21.15.107:0/2642964209 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.982+0000 7fb12ffff640 1 --2- 172.21.15.107:0/2642964209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744cacc0 0x564674545d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.982+0000 7fb127fff640 1 -- 172.21.15.107:0/2642964209 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744c5450 msgr2=0x5646744ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.982+0000 7fb127fff640 1 --2- 172.21.15.107:0/2642964209 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744c5450 0x5646744ca240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.982+0000 7fb127fff640 1 -- 172.21.15.107:0/2642964209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744cacc0 msgr2=0x564674545d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.982+0000 7fb127fff640 1 --2- 172.21.15.107:0/2642964209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744cacc0 0x564674545d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.982+0000 7fb127fff640 1 -- 172.21.15.107:0/2642964209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646745462a0 con 0x5646744d0630 2024-05-09T08:46:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.983+0000 7fb127fff640 1 --2- 172.21.15.107:0/2642964209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744d0630 0x5646744ca780 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fb120000c00 tx=0x7fb1200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c760911b0c2e1fce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.983+0000 7fb10effd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:46:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.983+0000 7fb10effd640 1 -- 172.21.15.107:0/2642964209 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12000fe80 con 0x5646744d0630 2024-05-09T08:46:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.983+0000 7fb138a51740 1 -- 172.21.15.107:0/2642964209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb128003bd0 con 0x5646744d0630 2024-05-09T08:46:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.983+0000 7fb10effd640 1 -- 172.21.15.107:0/2642964209 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9527+0+0 (secure 0 0 0) 0x7fb120003ef0 con 0x5646744d0630 2024-05-09T08:46:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.983+0000 7fb10effd640 1 -- 172.21.15.107:0/2642964209 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb12003a050 con 0x5646744d0630 2024-05-09T08:46:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.983+0000 7fb138a51740 10 client.18996 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.983+0000 7fb138a51740 1 -- 172.21.15.107:0/2642964209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564674547430 con 0x5646744d0630 2024-05-09T08:46:19.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.983+0000 7fb10effd640 1 -- 172.21.15.107:0/2642964209 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb128003bd0 con 0x5646744d0630 2024-05-09T08:46:19.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.984+0000 7fb10effd640 1 -- 172.21.15.107:0/2642964209 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fb1200054d0 con 0x5646744d0630 2024-05-09T08:46:19.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.984+0000 7fb138a51740 10 client.18996 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:19.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.984+0000 7fb138a51740 10 client.18996 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:46:19.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.984+0000 7fb138a51740 20 client.18996 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:19.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.984+0000 7fb138a51740 1 --2- 172.21.15.107:0/2642964209 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564674547b80 0x564674549fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:19.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.984+0000 7fb138a51740 4 client.18996 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:46:19.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.984+0000 7fb138a51740 1 -- 172.21.15.107:0/2642964209 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564674431dc0 con 0x564674547b80 2024-05-09T08:46:19.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.984+0000 7fb12ffff640 1 --2- 172.21.15.107:0/2642964209 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564674547b80 0x564674549fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:19.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.984+0000 7fb10effd640 10 client.18996 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:19.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:19.984+0000 7fb12ffff640 1 --2- 172.21.15.107:0/2642964209 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564674547b80 0x564674549fa0 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b03b38953f1f1ab server_cookie=a51c78034ba6f06b in_seq=0 out_seq=0 2024-05-09T08:46:20.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:20 smithi107 ceph-mon[30122]: pgmap v405: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-05-09T08:46:20.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:20 smithi157 ceph-mon[33614]: pgmap v405: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-05-09T08:46:20.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:20 smithi110 ceph-mon[33622]: pgmap v405: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-05-09T08:46:21.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.169+0000 7fb10effd640 1 -- 172.21.15.107:0/2642964209 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564674431dc0 con 0x564674547b80 2024-05-09T08:46:21.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.169+0000 7fb10effd640 10 client.18996 handle_command_reply: tid=0 2024-05-09T08:46:21.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.169+0000 7fb138a51740 1 client.18996 shutdown 2024-05-09T08:46:21.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.169+0000 7fb138a51740 1 -- 172.21.15.107:0/2642964209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744d0630 msgr2=0x5646744ca780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:21.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.169+0000 7fb138a51740 1 --2- 172.21.15.107:0/2642964209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744d0630 0x5646744ca780 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fb120000c00 tx=0x7fb1200046c0 comp rx=0 tx=0).stop 2024-05-09T08:46:21.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.169+0000 7fb138a51740 1 -- 172.21.15.107:0/2642964209 shutdown_connections 2024-05-09T08:46:21.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.169+0000 7fb138a51740 1 --2- 172.21.15.107:0/2642964209 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x564674547b80 0x564674549fa0 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:21.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.169+0000 7fb138a51740 1 --2- 172.21.15.107:0/2642964209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5646744cacc0 0x564674545d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:21.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.169+0000 7fb138a51740 1 --2- 172.21.15.107:0/2642964209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5646744d0630 0x5646744ca780 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:21.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.169+0000 7fb138a51740 1 --2- 172.21.15.107:0/2642964209 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5646744c5450 0x5646744ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:21.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.170+0000 7fb138a51740 1 -- 172.21.15.107:0/2642964209 >> 172.21.15.107:0/2642964209 conn(0x5646744c1960 msgr2=0x5646744c6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:21.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.170+0000 7fb10effd640 0 client.18996 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:21.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.170+0000 7fb138a51740 1 -- 172.21.15.107:0/2642964209 shutdown_connections 2024-05-09T08:46:21.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.170+0000 7fb138a51740 1 -- 172.21.15.107:0/2642964209 wait complete. 2024-05-09T08:46:21.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.170+0000 7fb138a51740 20 client.18996 trim_cache size 0 max 16384 2024-05-09T08:46:21.171 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:46:21.171 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:46:21.171 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:46:21.171 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:46:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.170+0000 7fb135d42640 1 -- 172.21.15.107:0/442665213 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb11c0a2e20 msgr2=0x7fb11c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.170+0000 7fb135d42640 1 --2- 172.21.15.107:0/442665213 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb11c0a2e20 0x7fb11c0a52e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x5646744c69f0 tx=0x7fb118034000 comp rx=0 tx=0).stop 2024-05-09T08:46:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.170+0000 7fb135d42640 1 -- 172.21.15.107:0/442665213 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb130100760 msgr2=0x7fb13019a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.170+0000 7fb135d42640 1 --2- 172.21.15.107:0/442665213 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb130100760 0x7fb13019a170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb11802aa00 tx=0x7fb118003b90 comp rx=0 tx=0).stop 2024-05-09T08:46:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.171+0000 7fb135d42640 1 -- 172.21.15.107:0/442665213 shutdown_connections 2024-05-09T08:46:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.171+0000 7fb135d42640 1 --2- 172.21.15.107:0/442665213 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb11c0a2e20 0x7fb11c0a52e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.171+0000 7fb135d42640 1 --2- 172.21.15.107:0/442665213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb130106720 0x7fb13019ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.171+0000 7fb135d42640 1 --2- 172.21.15.107:0/442665213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb130101120 0x7fb13019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.171+0000 7fb135d42640 1 --2- 172.21.15.107:0/442665213 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb130100760 0x7fb13019a170 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.171+0000 7fb135d42640 1 -- 172.21.15.107:0/442665213 >> 172.21.15.107:0/442665213 conn(0x7fb1300778b0 msgr2=0x7fb1301031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.171+0000 7fb135d42640 1 -- 172.21.15.107:0/442665213 shutdown_connections 2024-05-09T08:46:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:21.171+0000 7fb135d42640 1 -- 172.21.15.107:0/442665213 wait complete. 2024-05-09T08:46:22.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:22 smithi107 ceph-mon[30122]: pgmap v406: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-05-09T08:46:22.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:22 smithi107 ceph-mon[30122]: osdmap e108: 12 total, 12 up, 12 in 2024-05-09T08:46:22.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:22 smithi157 ceph-mon[33614]: pgmap v406: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-05-09T08:46:22.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:22 smithi157 ceph-mon[33614]: osdmap e108: 12 total, 12 up, 12 in 2024-05-09T08:46:22.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:22 smithi110 ceph-mon[33622]: pgmap v406: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-05-09T08:46:22.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:22 smithi110 ceph-mon[33622]: osdmap e108: 12 total, 12 up, 12 in 2024-05-09T08:46:24.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:24 smithi107 ceph-mon[30122]: pgmap v408: 129 pgs: 35 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 80 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 11 MiB/s wr, 2.64k op/s 2024-05-09T08:46:24.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:24 smithi157 ceph-mon[33614]: pgmap v408: 129 pgs: 35 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 80 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 11 MiB/s wr, 2.64k op/s 2024-05-09T08:46:24.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:24 smithi110 ceph-mon[33622]: pgmap v408: 129 pgs: 35 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 80 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 11 MiB/s wr, 2.64k op/s 2024-05-09T08:46:26.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:26 smithi107 ceph-mon[30122]: pgmap v409: 129 pgs: 35 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 78 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 11 MiB/s wr, 2.75k op/s 2024-05-09T08:46:26.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:26 smithi157 ceph-mon[33614]: pgmap v409: 129 pgs: 35 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 78 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 11 MiB/s wr, 2.75k op/s 2024-05-09T08:46:26.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:26 smithi110 ceph-mon[33622]: pgmap v409: 129 pgs: 35 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 78 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 11 MiB/s wr, 2.75k op/s 2024-05-09T08:46:28.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:28 smithi107 ceph-mon[30122]: pgmap v410: 129 pgs: 23 active+clean+snaptrim_wait, 13 active+clean+snaptrim, 93 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 11 MiB/s wr, 2.76k op/s 2024-05-09T08:46:28.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:28 smithi157 ceph-mon[33614]: pgmap v410: 129 pgs: 23 active+clean+snaptrim_wait, 13 active+clean+snaptrim, 93 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 11 MiB/s wr, 2.76k op/s 2024-05-09T08:46:28.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:28 smithi110 ceph-mon[33622]: pgmap v410: 129 pgs: 23 active+clean+snaptrim_wait, 13 active+clean+snaptrim, 93 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 11 MiB/s wr, 2.76k op/s 2024-05-09T08:46:29.587 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:46:29.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.699+0000 7f655bb53640 1 -- 172.21.15.107:0/43743168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65541010b0 msgr2=0x7f655410d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.699+0000 7f655bb53640 1 --2- 172.21.15.107:0/43743168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65541010b0 0x7f655410d5c0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f654c009cd0 tx=0x7f654c02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.699+0000 7f655bb53640 1 -- 172.21.15.107:0/43743168 shutdown_connections 2024-05-09T08:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.699+0000 7f655bb53640 1 --2- 172.21.15.107:0/43743168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65541010b0 0x7f655410d5c0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.699+0000 7f655bb53640 1 --2- 172.21.15.107:0/43743168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65541006f0 0x7f6554100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.699+0000 7f655bb53640 1 --2- 172.21.15.107:0/43743168 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6554106700 0x7f6554106ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.699+0000 7f655bb53640 1 -- 172.21.15.107:0/43743168 >> 172.21.15.107:0/43743168 conn(0x7f65540777d0 msgr2=0x7f6554077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.699+0000 7f655bb53640 1 -- 172.21.15.107:0/43743168 shutdown_connections 2024-05-09T08:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.699+0000 7f655bb53640 1 -- 172.21.15.107:0/43743168 wait complete. 2024-05-09T08:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.700+0000 7f655bb53640 1 Processor -- start 2024-05-09T08:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.700+0000 7f655bb53640 1 -- start start 2024-05-09T08:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.700+0000 7f655bb53640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65541006f0 0x7f65541980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.700+0000 7f655bb53640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65541010b0 0x7f65541985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.700+0000 7f655bb53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6554106700 0x7f655419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.700+0000 7f655bb53640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6554110980 con 0x7f6554106700 2024-05-09T08:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.700+0000 7f655bb53640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6554110800 con 0x7f65541010b0 2024-05-09T08:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.700+0000 7f655bb53640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6554110b00 con 0x7f65541006f0 2024-05-09T08:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.701+0000 7f655a0f0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6554106700 0x7f655419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.701+0000 7f65598ef640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65541006f0 0x7f65541980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.701+0000 7f65598ef640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65541006f0 0x7f65541980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:42958/0 (socket says 172.21.15.107:42958) 2024-05-09T08:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.701+0000 7f65598ef640 1 -- 172.21.15.107:0/2135496935 learned_addr learned my addr 172.21.15.107:0/2135496935 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.701+0000 7f65590ee640 1 --2- 172.21.15.107:0/2135496935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65541010b0 0x7f65541985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.701+0000 7f65598ef640 1 -- 172.21.15.107:0/2135496935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65541010b0 msgr2=0x7f65541985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.701+0000 7f65598ef640 1 --2- 172.21.15.107:0/2135496935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65541010b0 0x7f65541985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.701+0000 7f65598ef640 1 -- 172.21.15.107:0/2135496935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6554106700 msgr2=0x7f655419c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.701+0000 7f65598ef640 1 --2- 172.21.15.107:0/2135496935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6554106700 0x7f655419c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.701+0000 7f65598ef640 1 -- 172.21.15.107:0/2135496935 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655419d050 con 0x7f65541006f0 2024-05-09T08:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.701+0000 7f65598ef640 1 --2- 172.21.15.107:0/2135496935 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65541006f0 0x7f65541980b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f655000e9f0 tx=0x7f655000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.701+0000 7f654affd640 1 -- 172.21.15.107:0/2135496935 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655000cd80 con 0x7f65541006f0 2024-05-09T08:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.701+0000 7f655bb53640 1 -- 172.21.15.107:0/2135496935 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f655419d340 con 0x7f65541006f0 2024-05-09T08:46:29.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.701+0000 7f654affd640 1 -- 172.21.15.107:0/2135496935 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f65500108e0 con 0x7f65541006f0 2024-05-09T08:46:29.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.702+0000 7f654affd640 1 -- 172.21.15.107:0/2135496935 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6550018920 con 0x7f65541006f0 2024-05-09T08:46:29.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.702+0000 7f655bb53640 1 -- 172.21.15.107:0/2135496935 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65541a4bd0 con 0x7f65541006f0 2024-05-09T08:46:29.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.704+0000 7f654affd640 1 -- 172.21.15.107:0/2135496935 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6550010a90 con 0x7f65541006f0 2024-05-09T08:46:29.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.705+0000 7f655e85e740 1 Processor -- start 2024-05-09T08:46:29.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.706+0000 7f655e85e740 1 -- start start 2024-05-09T08:46:29.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.706+0000 7f654affd640 1 --2- 172.21.15.107:0/2135496935 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65400a2d80 0x7f65400a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.706+0000 7f655e85e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9b280 0x55556be9b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.706+0000 7f655e85e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9bba0 0x55556bea5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.706+0000 7f655e85e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 0x55556bea8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.706+0000 7f655e85e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55556be00d00 con 0x55556bea6270 2024-05-09T08:46:29.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.706+0000 7f655e85e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55556bda84c0 con 0x55556be9b280 2024-05-09T08:46:29.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.706+0000 7f655e85e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55556bdfccc0 con 0x55556be9bba0 2024-05-09T08:46:29.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.706+0000 7f655a0f0640 1 --2- 172.21.15.107:0/2135496935 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65400a2d80 0x7f65400a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.706+0000 7f65598ef640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9b280 0x55556be9b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.706+0000 7f655a0f0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 0x55556bea8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.706+0000 7f65598ef640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9b280 0x55556be9b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34382/0 (socket says 172.21.15.107:34382) 2024-05-09T08:46:29.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.706+0000 7f655a0f0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 0x55556bea8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35310/0 (socket says 172.21.15.107:35310) 2024-05-09T08:46:29.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.706+0000 7f65598ef640 1 -- 172.21.15.107:0/1663634421 learned_addr learned my addr 172.21.15.107:0/1663634421 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:29.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.707+0000 7f65598ef640 1 -- 172.21.15.107:0/1663634421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9bba0 msgr2=0x55556bea5d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:46:29.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.707+0000 7f65598ef640 1 --2- 172.21.15.107:0/1663634421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9bba0 0x55556bea5d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.707+0000 7f65598ef640 1 -- 172.21.15.107:0/1663634421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 msgr2=0x55556bea8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:29.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.707+0000 7f65598ef640 1 --2- 172.21.15.107:0/1663634421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 0x55556bea8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.707+0000 7f65598ef640 1 -- 172.21.15.107:0/1663634421 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55556bbde7d0 con 0x55556be9b280 2024-05-09T08:46:29.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.707+0000 7f654affd640 1 -- 172.21.15.107:0/2135496935 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f65500cfa90 con 0x7f65541006f0 2024-05-09T08:46:29.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.707+0000 7f655a0f0640 1 --2- 172.21.15.107:0/2135496935 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65400a2d80 0x7f65400a5240 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f654c004af0 tx=0x7f654c009910 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:29.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.707+0000 7f65598ef640 1 --2- 172.21.15.107:0/1663634421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9b280 0x55556be9b660 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f65500a4290 tx=0x7f6550010de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec3b174d0604e7b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:29.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.707+0000 7f653ffff640 1 -- 172.21.15.107:0/1663634421 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6550026040 con 0x55556be9b280 2024-05-09T08:46:29.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.707+0000 7f653ffff640 1 -- 172.21.15.107:0/1663634421 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6550067bb0 con 0x55556be9b280 2024-05-09T08:46:29.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.707+0000 7f653ffff640 1 -- 172.21.15.107:0/1663634421 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6550098270 con 0x55556be9b280 2024-05-09T08:46:29.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.708+0000 7f655e85e740 1 -- 172.21.15.107:0/1663634421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9b280 msgr2=0x55556be9b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:29.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.708+0000 7f655e85e740 1 --2- 172.21.15.107:0/1663634421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9b280 0x55556be9b660 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f65500a4290 tx=0x7f6550010de0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.708+0000 7f655e85e740 1 -- 172.21.15.107:0/1663634421 shutdown_connections 2024-05-09T08:46:29.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.708+0000 7f655e85e740 1 --2- 172.21.15.107:0/1663634421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 0x55556bea8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.708+0000 7f655e85e740 1 --2- 172.21.15.107:0/1663634421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9bba0 0x55556bea5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.708+0000 7f655e85e740 1 --2- 172.21.15.107:0/1663634421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9b280 0x55556be9b660 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.708+0000 7f655e85e740 1 -- 172.21.15.107:0/1663634421 >> 172.21.15.107:0/1663634421 conn(0x55556be975a0 msgr2=0x55556be97980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:29.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.709+0000 7f655e85e740 1 -- 172.21.15.107:0/1663634421 shutdown_connections 2024-05-09T08:46:29.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.709+0000 7f655e85e740 1 -- 172.21.15.107:0/1663634421 wait complete. 2024-05-09T08:46:29.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.711+0000 7f655e85e740 1 -- start start 2024-05-09T08:46:29.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.712+0000 7f655e85e740 1 Processor -- start 2024-05-09T08:46:29.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.713+0000 7f655e85e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9b280 0x55556bea32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.713+0000 7f655e85e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556be9bba0 0x55556bea37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.713+0000 7f655e85e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556bea6270 0x55556bea3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.713+0000 7f655e85e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55556bea8fa0 con 0x55556be9bba0 2024-05-09T08:46:29.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.713+0000 7f655e85e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55556bdbbb90 con 0x55556bea6270 2024-05-09T08:46:29.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.713+0000 7f655e85e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55556bea9120 con 0x55556be9b280 2024-05-09T08:46:29.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.713+0000 7f655e85e740 10 client.0 _ll_register_callbacks cb 0x55556bdf7da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:29.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.714+0000 7f655a0f0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556bea6270 0x55556bea3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.714+0000 7f655a0f0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556bea6270 0x55556bea3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34396/0 (socket says 172.21.15.107:34396) 2024-05-09T08:46:29.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.714+0000 7f655a0f0640 1 -- 172.21.15.107:0/2540992661 learned_addr learned my addr 172.21.15.107:0/2540992661 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:29.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.714+0000 7f65590ee640 1 --2- 172.21.15.107:0/2540992661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556be9bba0 0x55556bea37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.715+0000 7f655a0f0640 1 -- 172.21.15.107:0/2540992661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9b280 msgr2=0x55556bea32b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:46:29.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.715+0000 7f655a0f0640 1 --2- 172.21.15.107:0/2540992661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9b280 0x55556bea32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.715+0000 7f655a0f0640 1 -- 172.21.15.107:0/2540992661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556be9bba0 msgr2=0x55556bea37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:29.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.715+0000 7f655a0f0640 1 --2- 172.21.15.107:0/2540992661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556be9bba0 0x55556bea37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.715+0000 7f655a0f0640 1 -- 172.21.15.107:0/2540992661 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55556bb143f0 con 0x55556bea6270 2024-05-09T08:46:29.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.715+0000 7f655a0f0640 1 --2- 172.21.15.107:0/2540992661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556bea6270 0x55556bea3d30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f654c009cd0 tx=0x7f654c005da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7e141052b660a05 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:29.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.715+0000 7f653d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:46:29.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.716+0000 7f653d7fa640 1 -- 172.21.15.107:0/2540992661 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c051070 con 0x55556bea6270 2024-05-09T08:46:29.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.716+0000 7f655e85e740 1 -- 172.21.15.107:0/2540992661 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6550067bb0 con 0x55556bea6270 2024-05-09T08:46:29.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.716+0000 7f653d7fa640 1 -- 172.21.15.107:0/2540992661 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f654c031550 con 0x55556bea6270 2024-05-09T08:46:29.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.717+0000 7f653d7fa640 1 -- 172.21.15.107:0/2540992661 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f654c05d050 con 0x55556bea6270 2024-05-09T08:46:29.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.718+0000 7f655e85e740 10 client.27959 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:29.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.718+0000 7f655e85e740 1 -- 172.21.15.107:0/2540992661 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55556bad40f0 con 0x55556bea6270 2024-05-09T08:46:29.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.718+0000 7f653d7fa640 1 -- 172.21.15.107:0/2540992661 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6550067bb0 con 0x55556bea6270 2024-05-09T08:46:29.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.718+0000 7f653d7fa640 1 -- 172.21.15.107:0/2540992661 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f654c030070 con 0x55556bea6270 2024-05-09T08:46:29.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.719+0000 7f655e85e740 10 client.27959 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:29.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.719+0000 7f655e85e740 10 client.27959 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:46:29.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.719+0000 7f655e85e740 20 client.27959 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:29.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.719+0000 7f655e85e740 1 --2- 172.21.15.107:0/2540992661 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55556beb0860 0x55556beb2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.719+0000 7f655e85e740 4 client.27959 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:46:29.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.719+0000 7f655e85e740 1 -- 172.21.15.107:0/2540992661 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55556be0e430 con 0x55556beb0860 2024-05-09T08:46:29.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.719+0000 7f65598ef640 1 --2- 172.21.15.107:0/2540992661 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55556beb0860 0x55556beb2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.720+0000 7f653d7fa640 10 client.27959 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:29.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.720+0000 7f65598ef640 1 --2- 172.21.15.107:0/2540992661 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55556beb0860 0x55556beb2d20 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=403affe6c202349c server_cookie=df9035e7eb48c2df in_seq=0 out_seq=0 2024-05-09T08:46:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.720+0000 7f653d7fa640 1 -- 172.21.15.107:0/2540992661 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55556be0e430 con 0x55556beb0860 2024-05-09T08:46:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.720+0000 7f653d7fa640 10 client.27959 handle_command_reply: tid=0 2024-05-09T08:46:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.720+0000 7f655e85e740 1 client.27959 shutdown 2024-05-09T08:46:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.721+0000 7f655e85e740 1 -- 172.21.15.107:0/2540992661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556bea6270 msgr2=0x55556bea3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.721+0000 7f655e85e740 1 --2- 172.21.15.107:0/2540992661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556bea6270 0x55556bea3d30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f654c009cd0 tx=0x7f654c005da0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.721+0000 7f655e85e740 1 -- 172.21.15.107:0/2540992661 shutdown_connections 2024-05-09T08:46:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.721+0000 7f655e85e740 1 --2- 172.21.15.107:0/2540992661 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55556beb0860 0x55556beb2d20 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.721+0000 7f655e85e740 1 --2- 172.21.15.107:0/2540992661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556bea6270 0x55556bea3d30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.721+0000 7f655e85e740 1 --2- 172.21.15.107:0/2540992661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556be9bba0 0x55556bea37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.721+0000 7f655e85e740 1 --2- 172.21.15.107:0/2540992661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9b280 0x55556bea32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.721+0000 7f655e85e740 1 -- 172.21.15.107:0/2540992661 >> 172.21.15.107:0/2540992661 conn(0x55556be975a0 msgr2=0x55556be9d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:29.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.721+0000 7f653d7fa640 0 client.27959 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:29.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.721+0000 7f655e85e740 1 -- 172.21.15.107:0/2540992661 shutdown_connections 2024-05-09T08:46:29.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.721+0000 7f655e85e740 1 -- 172.21.15.107:0/2540992661 wait complete. 2024-05-09T08:46:29.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.721+0000 7f655e85e740 20 client.27959 trim_cache size 0 max 16384 2024-05-09T08:46:29.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.749+0000 7f655e85e740 1 Processor -- start 2024-05-09T08:46:29.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.749+0000 7f655e85e740 1 -- start start 2024-05-09T08:46:29.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.749+0000 7f655e85e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9b280 0x55556be92670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.750+0000 7f655e85e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9bba0 0x55556bebb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.750+0000 7f655e85e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 0x55556bed2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.750+0000 7f655e85e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f65541af4e0 con 0x55556bea6270 2024-05-09T08:46:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.750+0000 7f655e85e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f65541acdb0 con 0x55556be9bba0 2024-05-09T08:46:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.750+0000 7f655e85e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f65541aac20 con 0x55556be9b280 2024-05-09T08:46:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.750+0000 7f65598ef640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9b280 0x55556be92670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.750+0000 7f65598ef640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9b280 0x55556be92670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:42960/0 (socket says 172.21.15.107:42960) 2024-05-09T08:46:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.750+0000 7f65598ef640 1 -- 172.21.15.107:0/1775593817 learned_addr learned my addr 172.21.15.107:0/1775593817 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.750+0000 7f655a0f0640 1 --2- 172.21.15.107:0/1775593817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 0x55556bed2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.750+0000 7f655a0f0640 1 -- 172.21.15.107:0/1775593817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9b280 msgr2=0x55556be92670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.750+0000 7f655a0f0640 1 --2- 172.21.15.107:0/1775593817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9b280 0x55556be92670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.750+0000 7f655a0f0640 1 -- 172.21.15.107:0/1775593817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9bba0 msgr2=0x55556bebb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:29.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.750+0000 7f655a0f0640 1 --2- 172.21.15.107:0/1775593817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9bba0 0x55556bebb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.750+0000 7f655a0f0640 1 -- 172.21.15.107:0/1775593817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55556bea22a0 con 0x55556bea6270 2024-05-09T08:46:29.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.751+0000 7f655a0f0640 1 --2- 172.21.15.107:0/1775593817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 0x55556bed2040 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f654c009a60 tx=0x7f654c0319a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e196f7f977e95a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:29.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.751+0000 7f653f7fe640 1 -- 172.21.15.107:0/1775593817 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c051040 con 0x55556bea6270 2024-05-09T08:46:29.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.751+0000 7f653f7fe640 1 -- 172.21.15.107:0/1775593817 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f654c003c80 con 0x55556bea6270 2024-05-09T08:46:29.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.751+0000 7f653f7fe640 1 -- 172.21.15.107:0/1775593817 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c04b070 con 0x55556bea6270 2024-05-09T08:46:29.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.752+0000 7f655e85e740 1 -- 172.21.15.107:0/1775593817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 msgr2=0x55556bed2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:29.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.752+0000 7f655e85e740 1 --2- 172.21.15.107:0/1775593817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 0x55556bed2040 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f654c009a60 tx=0x7f654c0319a0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.752+0000 7f655e85e740 1 -- 172.21.15.107:0/1775593817 shutdown_connections 2024-05-09T08:46:29.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.752+0000 7f655e85e740 1 --2- 172.21.15.107:0/1775593817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 0x55556bed2040 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.752+0000 7f655e85e740 1 --2- 172.21.15.107:0/1775593817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9bba0 0x55556bebb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.752+0000 7f655e85e740 1 --2- 172.21.15.107:0/1775593817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9b280 0x55556be92670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.752+0000 7f655e85e740 1 -- 172.21.15.107:0/1775593817 >> 172.21.15.107:0/1775593817 conn(0x55556be975a0 msgr2=0x55556bebb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.752+0000 7f655e85e740 1 -- 172.21.15.107:0/1775593817 shutdown_connections 2024-05-09T08:46:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.752+0000 7f655e85e740 1 -- 172.21.15.107:0/1775593817 wait complete. 2024-05-09T08:46:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.752+0000 7f655e85e740 1 -- start start 2024-05-09T08:46:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.753+0000 7f655e85e740 1 Processor -- start 2024-05-09T08:46:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.753+0000 7f655e85e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9b280 0x55556beb8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.753+0000 7f655e85e740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9bba0 0x55556beb8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f655e85e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 0x55556be9f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f655e85e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f65541ab270 con 0x55556bea6270 2024-05-09T08:46:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f655e85e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f65541ac6e0 con 0x55556be9b280 2024-05-09T08:46:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f655e85e740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f65541ab940 con 0x55556be9bba0 2024-05-09T08:46:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f65598ef640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9b280 0x55556beb8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f655e85e740 10 client.0 _ll_register_callbacks cb 0x7f654c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f65598ef640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9b280 0x55556beb8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34406/0 (socket says 172.21.15.107:34406) 2024-05-09T08:46:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f65598ef640 1 -- 172.21.15.107:0/335198002 learned_addr learned my addr 172.21.15.107:0/335198002 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f655a0f0640 1 --2- 172.21.15.107:0/335198002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 0x55556be9f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f65590ee640 1 --2- 172.21.15.107:0/335198002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9bba0 0x55556beb8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f65598ef640 1 -- 172.21.15.107:0/335198002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9bba0 msgr2=0x55556beb8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f65598ef640 1 --2- 172.21.15.107:0/335198002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9bba0 0x55556beb8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f65598ef640 1 -- 172.21.15.107:0/335198002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 msgr2=0x55556be9f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:29.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f65598ef640 1 --2- 172.21.15.107:0/335198002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 0x55556be9f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f65598ef640 1 -- 172.21.15.107:0/335198002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55556beacae0 con 0x55556be9b280 2024-05-09T08:46:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f65590ee640 1 --2- 172.21.15.107:0/335198002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9bba0 0x55556beb8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:46:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f655a0f0640 1 --2- 172.21.15.107:0/335198002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 0x55556be9f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:46:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f65598ef640 1 --2- 172.21.15.107:0/335198002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9b280 0x55556beb8570 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6550009380 tx=0x7f655001f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8185c37ae3f20cf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.754+0000 7f653d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:46:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.755+0000 7f653d7fa640 1 -- 172.21.15.107:0/335198002 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6550097ea0 con 0x55556be9b280 2024-05-09T08:46:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.755+0000 7f653d7fa640 1 -- 172.21.15.107:0/335198002 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f65500a4330 con 0x55556be9b280 2024-05-09T08:46:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.755+0000 7f655e85e740 1 -- 172.21.15.107:0/335198002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55556be08780 con 0x55556be9b280 2024-05-09T08:46:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.755+0000 7f653d7fa640 1 -- 172.21.15.107:0/335198002 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f655009ff00 con 0x55556be9b280 2024-05-09T08:46:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.755+0000 7f655e85e740 10 client.27962 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.755+0000 7f655e85e740 1 -- 172.21.15.107:0/335198002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55556bf2ddd0 con 0x55556be9b280 2024-05-09T08:46:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.756+0000 7f653d7fa640 1 -- 172.21.15.107:0/335198002 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55556be08780 con 0x55556be9b280 2024-05-09T08:46:29.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.756+0000 7f653d7fa640 1 -- 172.21.15.107:0/335198002 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f655009b050 con 0x55556be9b280 2024-05-09T08:46:29.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.757+0000 7f655e85e740 10 client.27962 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:29.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.757+0000 7f655e85e740 10 client.27962 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:46:29.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.757+0000 7f655e85e740 20 client.27962 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:29.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.757+0000 7f655e85e740 1 --2- 172.21.15.107:0/335198002 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55556bf2e0c0 0x55556bf304b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.757+0000 7f655e85e740 4 client.27962 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:46:29.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.757+0000 7f655e85e740 1 -- 172.21.15.107:0/335198002 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55556bd856f0 con 0x55556bf2e0c0 2024-05-09T08:46:29.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.757+0000 7f65590ee640 1 --2- 172.21.15.107:0/335198002 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55556bf2e0c0 0x55556bf304b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.758+0000 7f65590ee640 1 --2- 172.21.15.107:0/335198002 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55556bf2e0c0 0x55556bf304b0 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb976855b42bff11 server_cookie=bad0e8646751675e in_seq=0 out_seq=0 2024-05-09T08:46:29.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.758+0000 7f653d7fa640 10 client.27962 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:29.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.759+0000 7f653d7fa640 1 -- 172.21.15.107:0/335198002 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55556bd856f0 con 0x55556bf2e0c0 2024-05-09T08:46:29.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.759+0000 7f653d7fa640 10 client.27962 handle_command_reply: tid=0 2024-05-09T08:46:29.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.759+0000 7f655e85e740 1 client.27962 shutdown 2024-05-09T08:46:29.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.762+0000 7f655e85e740 1 -- 172.21.15.107:0/335198002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9b280 msgr2=0x55556beb8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:29.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.762+0000 7f655e85e740 1 --2- 172.21.15.107:0/335198002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9b280 0x55556beb8570 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6550009380 tx=0x7f655001f000 comp rx=0 tx=0).stop 2024-05-09T08:46:29.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.764+0000 7f655e85e740 1 -- 172.21.15.107:0/335198002 shutdown_connections 2024-05-09T08:46:29.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.764+0000 7f655e85e740 1 --2- 172.21.15.107:0/335198002 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55556bf2e0c0 0x55556bf304b0 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.764+0000 7f655e85e740 1 --2- 172.21.15.107:0/335198002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55556bea6270 0x55556be9f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.764+0000 7f655e85e740 1 --2- 172.21.15.107:0/335198002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55556be9bba0 0x55556beb8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.764+0000 7f655e85e740 1 --2- 172.21.15.107:0/335198002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55556be9b280 0x55556beb8570 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.764+0000 7f655e85e740 1 -- 172.21.15.107:0/335198002 >> 172.21.15.107:0/335198002 conn(0x55556be975a0 msgr2=0x55556beab340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:29.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.764+0000 7f653d7fa640 0 client.27962 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:29.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.764+0000 7f655e85e740 1 -- 172.21.15.107:0/335198002 shutdown_connections 2024-05-09T08:46:29.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.764+0000 7f655e85e740 1 -- 172.21.15.107:0/335198002 wait complete. 2024-05-09T08:46:29.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.764+0000 7f655e85e740 20 client.27962 trim_cache size 0 max 16384 2024-05-09T08:46:29.768 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:46:29.768 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:46:29.768 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:46:29.768 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:46:29.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.769+0000 7f655bb53640 1 -- 172.21.15.107:0/2135496935 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65400a2d80 msgr2=0x7f65400a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:29.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.769+0000 7f655bb53640 1 --2- 172.21.15.107:0/2135496935 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65400a2d80 0x7f65400a5240 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f654c004af0 tx=0x7f654c009910 comp rx=0 tx=0).stop 2024-05-09T08:46:29.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.769+0000 7f655bb53640 1 -- 172.21.15.107:0/2135496935 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65541006f0 msgr2=0x7f65541980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:29.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.769+0000 7f655bb53640 1 --2- 172.21.15.107:0/2135496935 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65541006f0 0x7f65541980b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f655000e9f0 tx=0x7f655000eeb0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.769+0000 7f655bb53640 1 -- 172.21.15.107:0/2135496935 shutdown_connections 2024-05-09T08:46:29.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.769+0000 7f655bb53640 1 --2- 172.21.15.107:0/2135496935 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65400a2d80 0x7f65400a5240 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.769+0000 7f655bb53640 1 --2- 172.21.15.107:0/2135496935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6554106700 0x7f655419c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.769+0000 7f655bb53640 1 --2- 172.21.15.107:0/2135496935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65541010b0 0x7f65541985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.769+0000 7f655bb53640 1 --2- 172.21.15.107:0/2135496935 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65541006f0 0x7f65541980b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.769+0000 7f655bb53640 1 -- 172.21.15.107:0/2135496935 >> 172.21.15.107:0/2135496935 conn(0x7f65540777d0 msgr2=0x7f6554069ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:29.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.769+0000 7f655bb53640 1 -- 172.21.15.107:0/2135496935 shutdown_connections 2024-05-09T08:46:29.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.769+0000 7f655bb53640 1 -- 172.21.15.107:0/2135496935 wait complete. 2024-05-09T08:46:29.786 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:46:29.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:46:29.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.893+0000 7f2135370640 1 -- 172.21.15.107:0/995768739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21301010d0 msgr2=0x7f213010d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:29.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.893+0000 7f2135370640 1 --2- 172.21.15.107:0/995768739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21301010d0 0x7f213010d5e0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f2128009cd0 tx=0x7f212802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.893+0000 7f2135370640 1 -- 172.21.15.107:0/995768739 shutdown_connections 2024-05-09T08:46:29.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.893+0000 7f2135370640 1 --2- 172.21.15.107:0/995768739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21301010d0 0x7f213010d5e0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.893+0000 7f2135370640 1 --2- 172.21.15.107:0/995768739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2130100710 0x7f2130100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.893+0000 7f2135370640 1 --2- 172.21.15.107:0/995768739 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2130106720 0x7f2130106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.893+0000 7f2135370640 1 -- 172.21.15.107:0/995768739 >> 172.21.15.107:0/995768739 conn(0x7f2130077830 msgr2=0x7f2130077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:29.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.893+0000 7f2135370640 1 -- 172.21.15.107:0/995768739 shutdown_connections 2024-05-09T08:46:29.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.893+0000 7f2135370640 1 -- 172.21.15.107:0/995768739 wait complete. 2024-05-09T08:46:29.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.893+0000 7f2135370640 1 Processor -- start 2024-05-09T08:46:29.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f2135370640 1 -- start start 2024-05-09T08:46:29.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f2135370640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2130100710 0x7f2130198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f2135370640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21301010d0 0x7f2130199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f2135370640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2130106720 0x7f21301a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f2135370640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f21301109e0 con 0x7f21301010d0 2024-05-09T08:46:29.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f2135370640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2130110860 con 0x7f2130106720 2024-05-09T08:46:29.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f2135370640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2130110b60 con 0x7f2130100710 2024-05-09T08:46:29.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f212e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21301010d0 0x7f2130199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f212e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21301010d0 0x7f2130199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35346/0 (socket says 172.21.15.107:35346) 2024-05-09T08:46:29.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f212e7fc640 1 -- 172.21.15.107:0/757259231 learned_addr learned my addr 172.21.15.107:0/757259231 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:29.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f212f7fe640 1 --2- 172.21.15.107:0/757259231 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2130106720 0x7f21301a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f212effd640 1 --2- 172.21.15.107:0/757259231 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2130100710 0x7f2130198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f212e7fc640 1 -- 172.21.15.107:0/757259231 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2130100710 msgr2=0x7f2130198ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:29.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f212e7fc640 1 --2- 172.21.15.107:0/757259231 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2130100710 0x7f2130198ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f212e7fc640 1 -- 172.21.15.107:0/757259231 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2130106720 msgr2=0x7f21301a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:29.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f212e7fc640 1 --2- 172.21.15.107:0/757259231 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2130106720 0x7f21301a3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:29.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f212e7fc640 1 -- 172.21.15.107:0/757259231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21301a44a0 con 0x7f21301010d0 2024-05-09T08:46:29.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.894+0000 7f212e7fc640 1 --2- 172.21.15.107:0/757259231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21301010d0 0x7f2130199420 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f212400d950 tx=0x7f212400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:29.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.895+0000 7f2117fff640 1 -- 172.21.15.107:0/757259231 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2124017030 con 0x7f21301010d0 2024-05-09T08:46:29.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.895+0000 7f2135370640 1 -- 172.21.15.107:0/757259231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21301a4670 con 0x7f21301010d0 2024-05-09T08:46:29.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.895+0000 7f2117fff640 1 -- 172.21.15.107:0/757259231 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2124004020 con 0x7f21301010d0 2024-05-09T08:46:29.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.895+0000 7f2117fff640 1 -- 172.21.15.107:0/757259231 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21240047a0 con 0x7f21301010d0 2024-05-09T08:46:29.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.895+0000 7f2135370640 1 -- 172.21.15.107:0/757259231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21301a4be0 con 0x7f21301010d0 2024-05-09T08:46:29.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.896+0000 7f2135370640 1 -- 172.21.15.107:0/757259231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2130068650 con 0x7f21301010d0 2024-05-09T08:46:29.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.898+0000 7f2117fff640 1 -- 172.21.15.107:0/757259231 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f21240041d0 con 0x7f21301010d0 2024-05-09T08:46:29.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.899+0000 7f2117fff640 1 --2- 172.21.15.107:0/757259231 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f21180a2d80 0x7f21180a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:29.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.899+0000 7f2117fff640 1 -- 172.21.15.107:0/757259231 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f21240cf3b0 con 0x7f21301010d0 2024-05-09T08:46:29.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.899+0000 7f212effd640 1 --2- 172.21.15.107:0/757259231 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f21180a2d80 0x7f21180a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:29.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.899+0000 7f212effd640 1 --2- 172.21.15.107:0/757259231 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f21180a2d80 0x7f21180a5240 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f21200058c0 tx=0x7f2120005be0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:29.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:29.904+0000 7f2117fff640 1 -- 172.21.15.107:0/757259231 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f212409bb10 con 0x7f21301010d0 2024-05-09T08:46:30.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.135+0000 7f2135370640 1 -- 172.21.15.107:0/757259231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2130108ca0 con 0x7f21301010d0 2024-05-09T08:46:30.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.136+0000 7f2117fff640 1 -- 172.21.15.107:0/757259231 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11887 (secure 0 0 0) 0x7f212409b410 con 0x7f21301010d0 2024-05-09T08:46:30.136 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:46:30.136 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:37:10.930317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/843217781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":843217781},{"type":"v1","addr":"172.21.15.157:6835","nonce":843217781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:46:30.137 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2024-05-09T08:46:30.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.138+0000 7f2135370640 1 -- 172.21.15.107:0/757259231 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f21180a2d80 msgr2=0x7f21180a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.138+0000 7f2135370640 1 --2- 172.21.15.107:0/757259231 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f21180a2d80 0x7f21180a5240 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f21200058c0 tx=0x7f2120005be0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.138+0000 7f2135370640 1 -- 172.21.15.107:0/757259231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21301010d0 msgr2=0x7f2130199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.138+0000 7f2135370640 1 --2- 172.21.15.107:0/757259231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21301010d0 0x7f2130199420 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f212400d950 tx=0x7f212400de10 comp rx=0 tx=0).stop 2024-05-09T08:46:30.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.138+0000 7f2135370640 1 -- 172.21.15.107:0/757259231 shutdown_connections 2024-05-09T08:46:30.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.138+0000 7f2135370640 1 --2- 172.21.15.107:0/757259231 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f21180a2d80 0x7f21180a5240 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.138+0000 7f2135370640 1 --2- 172.21.15.107:0/757259231 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2130106720 0x7f21301a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.138+0000 7f2135370640 1 --2- 172.21.15.107:0/757259231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21301010d0 0x7f2130199420 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.138+0000 7f2135370640 1 --2- 172.21.15.107:0/757259231 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2130100710 0x7f2130198ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.138+0000 7f2135370640 1 -- 172.21.15.107:0/757259231 >> 172.21.15.107:0/757259231 conn(0x7f2130077830 msgr2=0x7f2130102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.138+0000 7f2135370640 1 -- 172.21.15.107:0/757259231 shutdown_connections 2024-05-09T08:46:30.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.138+0000 7f2135370640 1 -- 172.21.15.107:0/757259231 wait complete. 2024-05-09T08:46:30.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:46:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.261+0000 7f8a712a5640 1 -- 172.21.15.107:0/3042666081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a6c069eb0 msgr2=0x7f8a6c06a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.261+0000 7f8a712a5640 1 --2- 172.21.15.107:0/3042666081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a6c069eb0 0x7f8a6c06a330 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c009930 tx=0x7f8a5c02a440 comp rx=0 tx=0).stop 2024-05-09T08:46:30.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.261+0000 7f8a712a5640 1 -- 172.21.15.107:0/3042666081 shutdown_connections 2024-05-09T08:46:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.261+0000 7f8a712a5640 1 --2- 172.21.15.107:0/3042666081 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a6c06a870 0x7f8a6c10e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.261+0000 7f8a712a5640 1 --2- 172.21.15.107:0/3042666081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a6c069eb0 0x7f8a6c06a330 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.261+0000 7f8a712a5640 1 --2- 172.21.15.107:0/3042666081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a6c107820 0x7f8a6c107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.261+0000 7f8a712a5640 1 -- 172.21.15.107:0/3042666081 >> 172.21.15.107:0/3042666081 conn(0x7f8a6c077790 msgr2=0x7f8a6c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.261+0000 7f8a712a5640 1 -- 172.21.15.107:0/3042666081 shutdown_connections 2024-05-09T08:46:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.261+0000 7f8a712a5640 1 -- 172.21.15.107:0/3042666081 wait complete. 2024-05-09T08:46:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.261+0000 7f8a712a5640 1 Processor -- start 2024-05-09T08:46:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a712a5640 1 -- start start 2024-05-09T08:46:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a712a5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a6c069eb0 0x7f8a6c198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a712a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a6c06a870 0x7f8a6c198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a712a5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a6c107820 0x7f8a6c19c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a712a5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c111b20 con 0x7f8a6c06a870 2024-05-09T08:46:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a712a5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c1119a0 con 0x7f8a6c107820 2024-05-09T08:46:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a712a5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c111ca0 con 0x7f8a6c069eb0 2024-05-09T08:46:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a6a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a6c06a870 0x7f8a6c198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a6a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a6c06a870 0x7f8a6c198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35366/0 (socket says 172.21.15.107:35366) 2024-05-09T08:46:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a6a7fc640 1 -- 172.21.15.107:0/2430574045 learned_addr learned my addr 172.21.15.107:0/2430574045 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a6affd640 1 --2- 172.21.15.107:0/2430574045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a6c069eb0 0x7f8a6c198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a6b7fe640 1 --2- 172.21.15.107:0/2430574045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a6c107820 0x7f8a6c19c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a6a7fc640 1 -- 172.21.15.107:0/2430574045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a6c069eb0 msgr2=0x7f8a6c198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a6a7fc640 1 --2- 172.21.15.107:0/2430574045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a6c069eb0 0x7f8a6c198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a6a7fc640 1 -- 172.21.15.107:0/2430574045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a6c107820 msgr2=0x7f8a6c19c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a6a7fc640 1 --2- 172.21.15.107:0/2430574045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a6c107820 0x7f8a6c19c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a6a7fc640 1 -- 172.21.15.107:0/2430574045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6c19d0f0 con 0x7f8a6c06a870 2024-05-09T08:46:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a6b7fe640 1 --2- 172.21.15.107:0/2430574045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a6c107820 0x7f8a6c19c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:46:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.262+0000 7f8a6a7fc640 1 --2- 172.21.15.107:0/2430574045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a6c06a870 0x7f8a6c198640 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c004770 tx=0x7f8a5c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.263+0000 7f8a53fff640 1 -- 172.21.15.107:0/2430574045 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c03f070 con 0x7f8a6c06a870 2024-05-09T08:46:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.263+0000 7f8a53fff640 1 -- 172.21.15.107:0/2430574045 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a5c03b460 con 0x7f8a6c06a870 2024-05-09T08:46:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.263+0000 7f8a53fff640 1 -- 172.21.15.107:0/2430574045 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c035070 con 0x7f8a6c06a870 2024-05-09T08:46:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.263+0000 7f8a712a5640 1 -- 172.21.15.107:0/2430574045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a6c19d380 con 0x7f8a6c06a870 2024-05-09T08:46:30.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.263+0000 7f8a712a5640 1 -- 172.21.15.107:0/2430574045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a6c1a4c70 con 0x7f8a6c06a870 2024-05-09T08:46:30.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.265+0000 7f8a53fff640 1 -- 172.21.15.107:0/2430574045 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8a5c003a00 con 0x7f8a6c06a870 2024-05-09T08:46:30.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.265+0000 7f8a73fb1740 1 Processor -- start 2024-05-09T08:46:30.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.265+0000 7f8a73fb1740 1 -- start start 2024-05-09T08:46:30.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.265+0000 7f8a53fff640 1 --2- 172.21.15.107:0/2430574045 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a540a2d00 0x7f8a540a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a73fb1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72330 0x55d9e3f72710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a73fb1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72c50 0x55d9e3f7cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a73fb1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 0x55d9e3f7f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a73fb1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d9e3ed7c40 con 0x55d9e3f72c50 2024-05-09T08:46:30.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a73fb1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d9e3e7f580 con 0x55d9e3f7d320 2024-05-09T08:46:30.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a73fb1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d9e3ed3c00 con 0x55d9e3f72330 2024-05-09T08:46:30.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a6b7fe640 1 --2- 172.21.15.107:0/2430574045 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a540a2d00 0x7f8a540a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a6a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72c50 0x55d9e3f7cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a6a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72c50 0x55d9e3f7cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35374/0 (socket says 172.21.15.107:35374) 2024-05-09T08:46:30.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a6a7fc640 1 -- 172.21.15.107:0/32935996 learned_addr learned my addr 172.21.15.107:0/32935996 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:30.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a53fff640 1 -- 172.21.15.107:0/2430574045 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f8a5c0efb30 con 0x7f8a6c06a870 2024-05-09T08:46:30.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a6b7fe640 1 --2- 172.21.15.107:0/32935996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 0x55d9e3f7f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a6affd640 1 --2- 172.21.15.107:0/32935996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72330 0x55d9e3f72710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a6a7fc640 1 -- 172.21.15.107:0/32935996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72330 msgr2=0x55d9e3f72710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a6a7fc640 1 --2- 172.21.15.107:0/32935996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72330 0x55d9e3f72710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a6a7fc640 1 -- 172.21.15.107:0/32935996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 msgr2=0x55d9e3f7f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a6a7fc640 1 --2- 172.21.15.107:0/32935996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 0x55d9e3f7f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a6a7fc640 1 -- 172.21.15.107:0/32935996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9e3beb3f0 con 0x55d9e3f72c50 2024-05-09T08:46:30.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a6b7fe640 1 --2- 172.21.15.107:0/2430574045 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a540a2d00 0x7f8a540a51c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8a64005fd0 tx=0x7f8a64009f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a6a7fc640 1 --2- 172.21.15.107:0/32935996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72c50 0x55d9e3f7cde0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c005d00 tx=0x7f8a5c0bb900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2229921d25fbfce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a517fa640 1 -- 172.21.15.107:0/32935996 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c048070 con 0x55d9e3f72c50 2024-05-09T08:46:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a517fa640 1 -- 172.21.15.107:0/32935996 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a5c0bc400 con 0x55d9e3f72c50 2024-05-09T08:46:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.266+0000 7f8a517fa640 1 -- 172.21.15.107:0/32935996 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c0c4970 con 0x55d9e3f72c50 2024-05-09T08:46:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.267+0000 7f8a73fb1740 1 -- 172.21.15.107:0/32935996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72c50 msgr2=0x55d9e3f7cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.267+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/32935996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72c50 0x55d9e3f7cde0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c005d00 tx=0x7f8a5c0bb900 comp rx=0 tx=0).stop 2024-05-09T08:46:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.267+0000 7f8a73fb1740 1 -- 172.21.15.107:0/32935996 shutdown_connections 2024-05-09T08:46:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.267+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/32935996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 0x55d9e3f7f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.267+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/32935996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72c50 0x55d9e3f7cde0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.267+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/32935996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72330 0x55d9e3f72710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.267+0000 7f8a73fb1740 1 -- 172.21.15.107:0/32935996 >> 172.21.15.107:0/32935996 conn(0x55d9e3f6e650 msgr2=0x55d9e3f6ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.267+0000 7f8a73fb1740 1 -- 172.21.15.107:0/32935996 shutdown_connections 2024-05-09T08:46:30.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.267+0000 7f8a73fb1740 1 -- 172.21.15.107:0/32935996 wait complete. 2024-05-09T08:46:30.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.267+0000 7f8a73fb1740 1 -- start start 2024-05-09T08:46:30.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a73fb1740 1 Processor -- start 2024-05-09T08:46:30.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a73fb1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f72330 0x55d9e3f7a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a73fb1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72c50 0x55d9e3f7a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a73fb1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f7d320 0x55d9e3f7ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a73fb1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d9e3f7fed0 con 0x55d9e3f7d320 2024-05-09T08:46:30.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a73fb1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d9e3e92c40 con 0x55d9e3f72330 2024-05-09T08:46:30.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a73fb1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d9e3f80050 con 0x55d9e3f72c50 2024-05-09T08:46:30.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a73fb1740 10 client.0 _ll_register_callbacks cb 0x55d9e3beabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:30.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a6b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f7d320 0x55d9e3f7ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a6b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f7d320 0x55d9e3f7ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35376/0 (socket says 172.21.15.107:35376) 2024-05-09T08:46:30.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a6b7fe640 1 -- 172.21.15.107:0/2391843685 learned_addr learned my addr 172.21.15.107:0/2391843685 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:30.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a6affd640 1 --2- 172.21.15.107:0/2391843685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f72330 0x55d9e3f7a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a6b7fe640 1 -- 172.21.15.107:0/2391843685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72c50 msgr2=0x55d9e3f7a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a6b7fe640 1 --2- 172.21.15.107:0/2391843685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72c50 0x55d9e3f7a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a6b7fe640 1 -- 172.21.15.107:0/2391843685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f72330 msgr2=0x55d9e3f7a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a6b7fe640 1 --2- 172.21.15.107:0/2391843685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f72330 0x55d9e3f7a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a6b7fe640 1 -- 172.21.15.107:0/2391843685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9e3cb57d0 con 0x55d9e3f7d320 2024-05-09T08:46:30.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a6b7fe640 1 --2- 172.21.15.107:0/2391843685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f7d320 0x55d9e3f7ac70 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f8a640129a0 tx=0x7f8a64004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=803e61ce7e16fea2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a4eff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:46:30.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a4eff5640 1 -- 172.21.15.107:0/2391843685 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a64026070 con 0x55d9e3f7d320 2024-05-09T08:46:30.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a4eff5640 1 -- 172.21.15.107:0/2391843685 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f8a6400fcf0 con 0x55d9e3f7d320 2024-05-09T08:46:30.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.268+0000 7f8a73fb1740 1 -- 172.21.15.107:0/2391843685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a5c0bc400 con 0x55d9e3f7d320 2024-05-09T08:46:30.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.269+0000 7f8a4eff5640 1 -- 172.21.15.107:0/2391843685 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8a64032050 con 0x55d9e3f7d320 2024-05-09T08:46:30.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.269+0000 7f8a73fb1740 10 client.19032 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:30.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.269+0000 7f8a73fb1740 1 -- 172.21.15.107:0/2391843685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d9e3f8bf70 con 0x55d9e3f7d320 2024-05-09T08:46:30.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.269+0000 7f8a4eff5640 1 -- 172.21.15.107:0/2391843685 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a5c0bc400 con 0x55d9e3f7d320 2024-05-09T08:46:30.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.269+0000 7f8a4eff5640 1 -- 172.21.15.107:0/2391843685 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8a6401e040 con 0x55d9e3f7d320 2024-05-09T08:46:30.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.269+0000 7f8a73fb1740 10 client.19032 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:30.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.269+0000 7f8a73fb1740 10 client.19032 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:46:30.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.269+0000 7f8a73fb1740 20 client.19032 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:30.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.269+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/2391843685 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55d9e3f8d3d0 0x55d9e3f8f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.269+0000 7f8a73fb1740 4 client.19032 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:46:30.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.269+0000 7f8a73fb1740 1 -- 172.21.15.107:0/2391843685 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9e3ee5370 con 0x55d9e3f8d3d0 2024-05-09T08:46:30.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.269+0000 7f8a6affd640 1 --2- 172.21.15.107:0/2391843685 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55d9e3f8d3d0 0x55d9e3f8f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.270+0000 7f8a4eff5640 10 client.19032 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:46:30.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.270+0000 7f8a6affd640 1 --2- 172.21.15.107:0/2391843685 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55d9e3f8d3d0 0x55d9e3f8f840 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5aecff618f34697 server_cookie=8e0f65007124d0e8 in_seq=0 out_seq=0 2024-05-09T08:46:30.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.270+0000 7f8a4eff5640 1 -- 172.21.15.107:0/2391843685 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d9e3ee5370 con 0x55d9e3f8d3d0 2024-05-09T08:46:30.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.270+0000 7f8a4eff5640 10 client.19032 handle_command_reply: tid=0 2024-05-09T08:46:30.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.270+0000 7f8a73fb1740 1 client.19032 shutdown 2024-05-09T08:46:30.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.271+0000 7f8a73fb1740 1 -- 172.21.15.107:0/2391843685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f7d320 msgr2=0x55d9e3f7ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.271+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/2391843685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f7d320 0x55d9e3f7ac70 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f8a640129a0 tx=0x7f8a64004ab0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.271+0000 7f8a73fb1740 1 -- 172.21.15.107:0/2391843685 shutdown_connections 2024-05-09T08:46:30.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.271+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/2391843685 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55d9e3f8d3d0 0x55d9e3f8f840 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.271+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/2391843685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f7d320 0x55d9e3f7ac70 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.271+0000 7f8a4eff5640 0 client.19032 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:46:30.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.271+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/2391843685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72c50 0x55d9e3f7a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.271+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/2391843685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f72330 0x55d9e3f7a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.271+0000 7f8a73fb1740 1 -- 172.21.15.107:0/2391843685 >> 172.21.15.107:0/2391843685 conn(0x55d9e3f6e650 msgr2=0x55d9e3f6d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.271+0000 7f8a73fb1740 1 -- 172.21.15.107:0/2391843685 shutdown_connections 2024-05-09T08:46:30.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.271+0000 7f8a73fb1740 1 -- 172.21.15.107:0/2391843685 wait complete. 2024-05-09T08:46:30.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.271+0000 7f8a73fb1740 20 client.19032 trim_cache size 0 max 16384 2024-05-09T08:46:30.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.285+0000 7f8a73fb1740 1 Processor -- start 2024-05-09T08:46:30.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.285+0000 7f8a73fb1740 1 -- start start 2024-05-09T08:46:30.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.285+0000 7f8a73fb1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72330 0x55d9e3f7c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.285+0000 7f8a73fb1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72c50 0x55d9e3f88d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.285+0000 7f8a73fb1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 0x55d9e3f6ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.285+0000 7f8a73fb1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c1b1660 con 0x55d9e3f72c50 2024-05-09T08:46:30.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.285+0000 7f8a73fb1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c1aef30 con 0x55d9e3f7d320 2024-05-09T08:46:30.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.285+0000 7f8a73fb1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c1acda0 con 0x55d9e3f72330 2024-05-09T08:46:30.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.285+0000 7f8a6a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72c50 0x55d9e3f88d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.285+0000 7f8a6a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72c50 0x55d9e3f88d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35378/0 (socket says 172.21.15.107:35378) 2024-05-09T08:46:30.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.285+0000 7f8a6a7fc640 1 -- 172.21.15.107:0/3577935511 learned_addr learned my addr 172.21.15.107:0/3577935511 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:30.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.285+0000 7f8a6affd640 1 --2- 172.21.15.107:0/3577935511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72330 0x55d9e3f7c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.285+0000 7f8a6b7fe640 1 --2- 172.21.15.107:0/3577935511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 0x55d9e3f6ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.286+0000 7f8a6affd640 1 -- 172.21.15.107:0/3577935511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 msgr2=0x55d9e3f6ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.286+0000 7f8a6affd640 1 --2- 172.21.15.107:0/3577935511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 0x55d9e3f6ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.286+0000 7f8a6affd640 1 -- 172.21.15.107:0/3577935511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72c50 msgr2=0x55d9e3f88d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.286+0000 7f8a6affd640 1 --2- 172.21.15.107:0/3577935511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72c50 0x55d9e3f88d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.286+0000 7f8a6affd640 1 -- 172.21.15.107:0/3577935511 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9e3f791a0 con 0x55d9e3f72330 2024-05-09T08:46:30.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.286+0000 7f8a6b7fe640 1 --2- 172.21.15.107:0/3577935511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 0x55d9e3f6ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:46:30.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.286+0000 7f8a6a7fc640 1 --2- 172.21.15.107:0/3577935511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72c50 0x55d9e3f88d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:46:30.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.286+0000 7f8a6affd640 1 --2- 172.21.15.107:0/3577935511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72330 0x55d9e3f7c920 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x55d9e3f8fe40 tx=0x7f8a60008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a31e1ff911145d7c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.286+0000 7f8a50ff9640 1 -- 172.21.15.107:0/3577935511 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60021070 con 0x55d9e3f72330 2024-05-09T08:46:30.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.286+0000 7f8a50ff9640 1 -- 172.21.15.107:0/3577935511 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a60016db0 con 0x55d9e3f72330 2024-05-09T08:46:30.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.286+0000 7f8a50ff9640 1 -- 172.21.15.107:0/3577935511 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60015640 con 0x55d9e3f72330 2024-05-09T08:46:30.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 -- 172.21.15.107:0/3577935511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72330 msgr2=0x55d9e3f7c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/3577935511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72330 0x55d9e3f7c920 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x55d9e3f8fe40 tx=0x7f8a60008040 comp rx=0 tx=0).stop 2024-05-09T08:46:30.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 -- 172.21.15.107:0/3577935511 shutdown_connections 2024-05-09T08:46:30.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/3577935511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 0x55d9e3f6ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/3577935511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72c50 0x55d9e3f88d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/3577935511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72330 0x55d9e3f7c920 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 -- 172.21.15.107:0/3577935511 >> 172.21.15.107:0/3577935511 conn(0x55d9e3f6e650 msgr2=0x55d9e3f6d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 -- 172.21.15.107:0/3577935511 shutdown_connections 2024-05-09T08:46:30.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 -- 172.21.15.107:0/3577935511 wait complete. 2024-05-09T08:46:30.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 -- start start 2024-05-09T08:46:30.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 Processor -- start 2024-05-09T08:46:30.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72330 0x55d9e3f76f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72c50 0x55d9e3f774a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 0x55d9e3f779e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c1ad3f0 con 0x55d9e3f72330 2024-05-09T08:46:30.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c1ae860 con 0x55d9e3f7d320 2024-05-09T08:46:30.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c1adac0 con 0x55d9e3f72c50 2024-05-09T08:46:30.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a73fb1740 10 client.0 _ll_register_callbacks cb 0x7f8a640093c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:30.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a6affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72330 0x55d9e3f76f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a6affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72330 0x55d9e3f76f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35394/0 (socket says 172.21.15.107:35394) 2024-05-09T08:46:30.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a6a7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72c50 0x55d9e3f774a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.287+0000 7f8a6affd640 1 -- 172.21.15.107:0/56162345 learned_addr learned my addr 172.21.15.107:0/56162345 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:30.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a6b7fe640 1 --2- 172.21.15.107:0/56162345 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 0x55d9e3f779e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a6affd640 1 -- 172.21.15.107:0/56162345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72c50 msgr2=0x55d9e3f774a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a6affd640 1 --2- 172.21.15.107:0/56162345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72c50 0x55d9e3f774a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a6affd640 1 -- 172.21.15.107:0/56162345 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 msgr2=0x55d9e3f779e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a6affd640 1 --2- 172.21.15.107:0/56162345 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 0x55d9e3f779e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a6affd640 1 -- 172.21.15.107:0/56162345 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9e3ff29b0 con 0x55d9e3f72330 2024-05-09T08:46:30.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a6b7fe640 1 --2- 172.21.15.107:0/56162345 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 0x55d9e3f779e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:46:30.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a6a7fc640 1 --2- 172.21.15.107:0/56162345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72c50 0x55d9e3f774a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:46:30.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a6affd640 1 --2- 172.21.15.107:0/56162345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72330 0x55d9e3f76f60 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009a90 tx=0x7f8a60015450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=266bc2de3fde98e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a4eff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:46:30.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a4eff5640 1 -- 172.21.15.107:0/56162345 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6002c070 con 0x55d9e3f72330 2024-05-09T08:46:30.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a4eff5640 1 -- 172.21.15.107:0/56162345 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f8a6001d860 con 0x55d9e3f72330 2024-05-09T08:46:30.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a73fb1740 1 -- 172.21.15.107:0/56162345 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a60016db0 con 0x55d9e3f72330 2024-05-09T08:46:30.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a4eff5640 1 -- 172.21.15.107:0/56162345 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8a60038050 con 0x55d9e3f72330 2024-05-09T08:46:30.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a73fb1740 10 client.19038 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:30.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a73fb1740 1 -- 172.21.15.107:0/56162345 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d9e3ff3c70 con 0x55d9e3f72330 2024-05-09T08:46:30.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.288+0000 7f8a4eff5640 1 -- 172.21.15.107:0/56162345 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a60016db0 con 0x55d9e3f72330 2024-05-09T08:46:30.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a4eff5640 1 -- 172.21.15.107:0/56162345 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8a60015ce0 con 0x55d9e3f72330 2024-05-09T08:46:30.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a73fb1740 10 client.19038 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:30.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a73fb1740 10 client.19038 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:46:30.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a73fb1740 20 client.19038 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:30.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/56162345 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55d9e3ff42d0 0x55d9e3ff66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a73fb1740 4 client.19038 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:46:30.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a73fb1740 1 -- 172.21.15.107:0/56162345 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d9e3edf6c0 con 0x55d9e3ff42d0 2024-05-09T08:46:30.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a6a7fc640 1 --2- 172.21.15.107:0/56162345 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55d9e3ff42d0 0x55d9e3ff66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a4eff5640 10 client.19038 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:46:30.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a6a7fc640 1 --2- 172.21.15.107:0/56162345 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55d9e3ff42d0 0x55d9e3ff66f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3b97bad5de784449 server_cookie=967a833cb0948c88 in_seq=0 out_seq=0 2024-05-09T08:46:30.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a4eff5640 1 -- 172.21.15.107:0/56162345 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d9e3edf6c0 con 0x55d9e3ff42d0 2024-05-09T08:46:30.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a4eff5640 10 client.19038 handle_command_reply: tid=0 2024-05-09T08:46:30.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a73fb1740 1 client.19038 shutdown 2024-05-09T08:46:30.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a73fb1740 1 -- 172.21.15.107:0/56162345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72330 msgr2=0x55d9e3f76f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/56162345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72330 0x55d9e3f76f60 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009a90 tx=0x7f8a60015450 comp rx=0 tx=0).stop 2024-05-09T08:46:30.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a73fb1740 1 -- 172.21.15.107:0/56162345 shutdown_connections 2024-05-09T08:46:30.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/56162345 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x55d9e3ff42d0 0x55d9e3ff66f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/56162345 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d9e3f7d320 0x55d9e3f779e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/56162345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e3f72c50 0x55d9e3f774a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a4eff5640 0 client.19038 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:46:30.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a73fb1740 1 --2- 172.21.15.107:0/56162345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9e3f72330 0x55d9e3f76f60 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.289+0000 7f8a73fb1740 1 -- 172.21.15.107:0/56162345 >> 172.21.15.107:0/56162345 conn(0x55d9e3f6e650 msgr2=0x55d9e3f6d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.290+0000 7f8a73fb1740 1 -- 172.21.15.107:0/56162345 shutdown_connections 2024-05-09T08:46:30.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.290+0000 7f8a73fb1740 1 -- 172.21.15.107:0/56162345 wait complete. 2024-05-09T08:46:30.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.290+0000 7f8a73fb1740 20 client.19038 trim_cache size 0 max 16384 2024-05-09T08:46:30.306 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:46:30.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.290+0000 7f8a712a5640 1 -- 172.21.15.107:0/2430574045 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a540a2d00 msgr2=0x7f8a540a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.290+0000 7f8a712a5640 1 --2- 172.21.15.107:0/2430574045 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a540a2d00 0x7f8a540a51c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8a64005fd0 tx=0x7f8a64009f90 comp rx=0 tx=0).stop 2024-05-09T08:46:30.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.290+0000 7f8a712a5640 1 -- 172.21.15.107:0/2430574045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a6c06a870 msgr2=0x7f8a6c198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.290+0000 7f8a712a5640 1 --2- 172.21.15.107:0/2430574045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a6c06a870 0x7f8a6c198640 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c004770 tx=0x7f8a5c0042f0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.290+0000 7f8a712a5640 1 -- 172.21.15.107:0/2430574045 shutdown_connections 2024-05-09T08:46:30.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.290+0000 7f8a712a5640 1 --2- 172.21.15.107:0/2430574045 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8a540a2d00 0x7f8a540a51c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.290+0000 7f8a712a5640 1 --2- 172.21.15.107:0/2430574045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a6c107820 0x7f8a6c19c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.290+0000 7f8a712a5640 1 --2- 172.21.15.107:0/2430574045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a6c06a870 0x7f8a6c198640 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.290+0000 7f8a712a5640 1 --2- 172.21.15.107:0/2430574045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a6c069eb0 0x7f8a6c198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.290+0000 7f8a712a5640 1 -- 172.21.15.107:0/2430574045 >> 172.21.15.107:0/2430574045 conn(0x7f8a6c077790 msgr2=0x7f8a6c0fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.290+0000 7f8a712a5640 1 -- 172.21.15.107:0/2430574045 shutdown_connections 2024-05-09T08:46:30.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.290+0000 7f8a712a5640 1 -- 172.21.15.107:0/2430574045 wait complete. 2024-05-09T08:46:30.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:46:30.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.413+0000 7f2631ded640 1 -- 172.21.15.107:0/659635650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c107810 msgr2=0x7f262c107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.413+0000 7f2631ded640 1 --2- 172.21.15.107:0/659635650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c107810 0x7f262c107bf0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f261c009930 tx=0x7f261c02a420 comp rx=0 tx=0).stop 2024-05-09T08:46:30.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.413+0000 7f2631ded640 1 -- 172.21.15.107:0/659635650 shutdown_connections 2024-05-09T08:46:30.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.413+0000 7f2631ded640 1 --2- 172.21.15.107:0/659635650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f262c06a7f0 0x7f262c10e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.413+0000 7f2631ded640 1 --2- 172.21.15.107:0/659635650 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f262c069e30 0x7f262c06a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.413+0000 7f2631ded640 1 --2- 172.21.15.107:0/659635650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c107810 0x7f262c107bf0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.413+0000 7f2631ded640 1 -- 172.21.15.107:0/659635650 >> 172.21.15.107:0/659635650 conn(0x7f262c076530 msgr2=0x7f262c076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.413+0000 7f2631ded640 1 -- 172.21.15.107:0/659635650 shutdown_connections 2024-05-09T08:46:30.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.413+0000 7f2631ded640 1 -- 172.21.15.107:0/659635650 wait complete. 2024-05-09T08:46:30.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f2631ded640 1 Processor -- start 2024-05-09T08:46:30.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f2631ded640 1 -- start start 2024-05-09T08:46:30.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f2631ded640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f262c069e30 0x7f262c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f2631ded640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f262c06a7f0 0x7f262c1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f2631ded640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c107810 0x7f262c19c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f2631ded640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f262c111ad0 con 0x7f262c107810 2024-05-09T08:46:30.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f2631ded640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f262c111950 con 0x7f262c06a7f0 2024-05-09T08:46:30.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f2631ded640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f262c111c50 con 0x7f262c069e30 2024-05-09T08:46:30.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f262bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c107810 0x7f262c19c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f262bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c107810 0x7f262c19c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35412/0 (socket says 172.21.15.107:35412) 2024-05-09T08:46:30.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f262bfff640 1 -- 172.21.15.107:0/2825977334 learned_addr learned my addr 172.21.15.107:0/2825977334 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f262b7fe640 1 --2- 172.21.15.107:0/2825977334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f262c069e30 0x7f262c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f2623fff640 1 --2- 172.21.15.107:0/2825977334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f262c06a7f0 0x7f262c1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f262bfff640 1 -- 172.21.15.107:0/2825977334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f262c069e30 msgr2=0x7f262c198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f262bfff640 1 --2- 172.21.15.107:0/2825977334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f262c069e30 0x7f262c198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f262bfff640 1 -- 172.21.15.107:0/2825977334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f262c06a7f0 msgr2=0x7f262c1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f262bfff640 1 --2- 172.21.15.107:0/2825977334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f262c06a7f0 0x7f262c1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f262bfff640 1 -- 172.21.15.107:0/2825977334 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f262c19d080 con 0x7f262c107810 2024-05-09T08:46:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.414+0000 7f262bfff640 1 --2- 172.21.15.107:0/2825977334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c107810 0x7f262c19c950 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f262400cad0 tx=0x7f262400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.415+0000 7f26297fa640 1 -- 172.21.15.107:0/2825977334 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262400ae20 con 0x7f262c107810 2024-05-09T08:46:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.415+0000 7f26297fa640 1 -- 172.21.15.107:0/2825977334 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2624005020 con 0x7f262c107810 2024-05-09T08:46:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.415+0000 7f26297fa640 1 -- 172.21.15.107:0/2825977334 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262400f6b0 con 0x7f262c107810 2024-05-09T08:46:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.415+0000 7f2631ded640 1 -- 172.21.15.107:0/2825977334 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f262c19d370 con 0x7f262c107810 2024-05-09T08:46:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.415+0000 7f2631ded640 1 -- 172.21.15.107:0/2825977334 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f262c1a4c00 con 0x7f262c107810 2024-05-09T08:46:30.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.417+0000 7f26297fa640 1 -- 172.21.15.107:0/2825977334 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f26240051d0 con 0x7f262c107810 2024-05-09T08:46:30.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.417+0000 7f2634bcf740 1 Processor -- start 2024-05-09T08:46:30.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.417+0000 7f2634bcf740 1 -- start start 2024-05-09T08:46:30.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.417+0000 7f2634bcf740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34d8330 0x5641b34d8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.417+0000 7f2634bcf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8c50 0x5641b34e2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.417+0000 7f2634bcf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34e3320 0x5641b34e5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.417+0000 7f2634bcf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5641b343dc40 con 0x5641b34d8c50 2024-05-09T08:46:30.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.417+0000 7f2634bcf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5641b33e5580 con 0x5641b34e3320 2024-05-09T08:46:30.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.417+0000 7f2634bcf740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5641b3439c00 con 0x5641b34d8330 2024-05-09T08:46:30.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.417+0000 7f262b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8c50 0x5641b34e2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.417+0000 7f262b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8c50 0x5641b34e2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35428/0 (socket says 172.21.15.107:35428) 2024-05-09T08:46:30.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.417+0000 7f262b7fe640 1 -- 172.21.15.107:0/3422114096 learned_addr learned my addr 172.21.15.107:0/3422114096 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:30.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.417+0000 7f26297fa640 1 --2- 172.21.15.107:0/2825977334 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f26180a2d00 0x7f26180a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.418+0000 7f2623fff640 1 --2- 172.21.15.107:0/3422114096 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34d8330 0x5641b34d8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.418+0000 7f2623fff640 1 --2- 172.21.15.107:0/3422114096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34e3320 0x5641b34e5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.418+0000 7f262b7fe640 1 -- 172.21.15.107:0/3422114096 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34d8330 msgr2=0x5641b34d8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.418+0000 7f262b7fe640 1 --2- 172.21.15.107:0/3422114096 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34d8330 0x5641b34d8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.418+0000 7f262b7fe640 1 -- 172.21.15.107:0/3422114096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34e3320 msgr2=0x5641b34e5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.418+0000 7f262b7fe640 1 --2- 172.21.15.107:0/3422114096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34e3320 0x5641b34e5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.418+0000 7f262b7fe640 1 -- 172.21.15.107:0/3422114096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641b31513f0 con 0x5641b34d8c50 2024-05-09T08:46:30.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.418+0000 7f26297fa640 1 -- 172.21.15.107:0/2825977334 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f26240cf950 con 0x7f262c107810 2024-05-09T08:46:30.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.418+0000 7f262bfff640 1 --2- 172.21.15.107:0/2825977334 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f26180a2d00 0x7f26180a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.418+0000 7f262b7fe640 1 --2- 172.21.15.107:0/3422114096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8c50 0x5641b34e2de0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f261c009930 tx=0x7f261c032cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2853ced38b10b04d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.418+0000 7f262bfff640 1 --2- 172.21.15.107:0/2825977334 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f26180a2d00 0x7f26180a51c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f262400a6e0 tx=0x7f2624090ea0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.418+0000 7f2621ffb640 1 -- 172.21.15.107:0/3422114096 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c03f070 con 0x5641b34d8c50 2024-05-09T08:46:30.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.418+0000 7f2621ffb640 1 -- 172.21.15.107:0/3422114096 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f261c032e30 con 0x5641b34d8c50 2024-05-09T08:46:30.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.418+0000 7f2621ffb640 1 -- 172.21.15.107:0/3422114096 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c035070 con 0x5641b34d8c50 2024-05-09T08:46:30.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.419+0000 7f2634bcf740 1 -- 172.21.15.107:0/3422114096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8c50 msgr2=0x5641b34e2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.419+0000 7f2634bcf740 1 --2- 172.21.15.107:0/3422114096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8c50 0x5641b34e2de0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f261c009930 tx=0x7f261c032cc0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.419+0000 7f2634bcf740 1 -- 172.21.15.107:0/3422114096 shutdown_connections 2024-05-09T08:46:30.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.419+0000 7f2634bcf740 1 --2- 172.21.15.107:0/3422114096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34e3320 0x5641b34e5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.419+0000 7f2634bcf740 1 --2- 172.21.15.107:0/3422114096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8c50 0x5641b34e2de0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.419+0000 7f2634bcf740 1 --2- 172.21.15.107:0/3422114096 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34d8330 0x5641b34d8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.419+0000 7f2634bcf740 1 -- 172.21.15.107:0/3422114096 >> 172.21.15.107:0/3422114096 conn(0x5641b34d4650 msgr2=0x5641b34d4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.419+0000 7f2634bcf740 1 -- 172.21.15.107:0/3422114096 shutdown_connections 2024-05-09T08:46:30.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.419+0000 7f2634bcf740 1 -- 172.21.15.107:0/3422114096 wait complete. 2024-05-09T08:46:30.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.419+0000 7f2634bcf740 1 -- start start 2024-05-09T08:46:30.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.419+0000 7f2634bcf740 1 Processor -- start 2024-05-09T08:46:30.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2634bcf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34d8330 0x5641b34e0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2634bcf740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34d8c50 0x5641b34e07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2634bcf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34e3320 0x5641b34e0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2634bcf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5641b34e60b0 con 0x5641b34e3320 2024-05-09T08:46:30.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2634bcf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5641b33f8c40 con 0x5641b34d8330 2024-05-09T08:46:30.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2634bcf740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5641b34e6230 con 0x5641b34d8c50 2024-05-09T08:46:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2634bcf740 10 client.0 _ll_register_callbacks cb 0x5641b3150be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2623fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34e3320 0x5641b34e0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2623fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34e3320 0x5641b34e0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35434/0 (socket says 172.21.15.107:35434) 2024-05-09T08:46:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2623fff640 1 -- 172.21.15.107:0/3307536375 learned_addr learned my addr 172.21.15.107:0/3307536375 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2623fff640 1 --2- 172.21.15.107:0/3307536375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34d8330 0x5641b34e0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f262b7fe640 1 --2- 172.21.15.107:0/3307536375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34d8c50 0x5641b34e07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2623fff640 1 -- 172.21.15.107:0/3307536375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34d8c50 msgr2=0x5641b34e07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2623fff640 1 --2- 172.21.15.107:0/3307536375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34d8c50 0x5641b34e07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2623fff640 1 -- 172.21.15.107:0/3307536375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34d8330 msgr2=0x5641b34e0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2623fff640 1 --2- 172.21.15.107:0/3307536375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34d8330 0x5641b34e0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2623fff640 1 -- 172.21.15.107:0/3307536375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641b321b7d0 con 0x5641b34e3320 2024-05-09T08:46:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f262b7fe640 1 --2- 172.21.15.107:0/3307536375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34d8c50 0x5641b34e07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:46:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2623fff640 1 --2- 172.21.15.107:0/3307536375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34e3320 0x5641b34e0d00 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f2614012e60 tx=0x7f261400cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9daf201ab20f8618 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f260affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:46:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f260affd640 1 -- 172.21.15.107:0/3307536375 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26140096c0 con 0x5641b34e3320 2024-05-09T08:46:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f260affd640 1 -- 172.21.15.107:0/3307536375 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f2614009860 con 0x5641b34e3320 2024-05-09T08:46:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2634bcf740 1 -- 172.21.15.107:0/3307536375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f261c032e30 con 0x5641b34e3320 2024-05-09T08:46:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f260affd640 1 -- 172.21.15.107:0/3307536375 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2614028050 con 0x5641b34e3320 2024-05-09T08:46:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2634bcf740 10 client.19050 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.420+0000 7f2634bcf740 1 -- 172.21.15.107:0/3307536375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5641b34f2280 con 0x5641b34e3320 2024-05-09T08:46:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.421+0000 7f260affd640 1 -- 172.21.15.107:0/3307536375 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f261c032e30 con 0x5641b34e3320 2024-05-09T08:46:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.421+0000 7f260affd640 1 -- 172.21.15.107:0/3307536375 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f261400f940 con 0x5641b34e3320 2024-05-09T08:46:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.421+0000 7f2634bcf740 10 client.19050 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.421+0000 7f2634bcf740 10 client.19050 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:46:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.421+0000 7f2634bcf740 20 client.19050 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.421+0000 7f2634bcf740 1 --2- 172.21.15.107:0/3307536375 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5641b34f3740 0x5641b34f5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.421+0000 7f2634bcf740 4 client.19050 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:46:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.421+0000 7f2634bcf740 1 -- 172.21.15.107:0/3307536375 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641b344b370 con 0x5641b34f3740 2024-05-09T08:46:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.421+0000 7f262b7fe640 1 --2- 172.21.15.107:0/3307536375 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5641b34f3740 0x5641b34f5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.421+0000 7f260affd640 10 client.19050 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:46:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.421+0000 7f262b7fe640 1 --2- 172.21.15.107:0/3307536375 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5641b34f3740 0x5641b34f5b90 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e661517fe4d98bcd server_cookie=59551c3be00e3f93 in_seq=0 out_seq=0 2024-05-09T08:46:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.422+0000 7f260affd640 1 -- 172.21.15.107:0/3307536375 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5641b344b370 con 0x5641b34f3740 2024-05-09T08:46:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.422+0000 7f260affd640 10 client.19050 handle_command_reply: tid=0 2024-05-09T08:46:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.422+0000 7f2634bcf740 1 client.19050 shutdown 2024-05-09T08:46:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.422+0000 7f2634bcf740 1 -- 172.21.15.107:0/3307536375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34e3320 msgr2=0x5641b34e0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.422+0000 7f2634bcf740 1 --2- 172.21.15.107:0/3307536375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34e3320 0x5641b34e0d00 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f2614012e60 tx=0x7f261400cb50 comp rx=0 tx=0).stop 2024-05-09T08:46:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.422+0000 7f2634bcf740 1 -- 172.21.15.107:0/3307536375 shutdown_connections 2024-05-09T08:46:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.422+0000 7f2634bcf740 1 --2- 172.21.15.107:0/3307536375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34e3320 0x5641b34e0d00 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.422+0000 7f2634bcf740 1 --2- 172.21.15.107:0/3307536375 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5641b34f3740 0x5641b34f5b90 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.422+0000 7f2634bcf740 1 --2- 172.21.15.107:0/3307536375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34d8c50 0x5641b34e07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.422+0000 7f2634bcf740 1 --2- 172.21.15.107:0/3307536375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34d8330 0x5641b34e0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.422+0000 7f260affd640 0 client.19050 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:46:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.422+0000 7f2634bcf740 1 -- 172.21.15.107:0/3307536375 >> 172.21.15.107:0/3307536375 conn(0x5641b34d4650 msgr2=0x5641b34da450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.422+0000 7f2634bcf740 1 -- 172.21.15.107:0/3307536375 shutdown_connections 2024-05-09T08:46:30.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.422+0000 7f2634bcf740 1 -- 172.21.15.107:0/3307536375 wait complete. 2024-05-09T08:46:30.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.422+0000 7f2634bcf740 20 client.19050 trim_cache size 0 max 16384 2024-05-09T08:46:30.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2634bcf740 1 Processor -- start 2024-05-09T08:46:30.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2634bcf740 1 -- start start 2024-05-09T08:46:30.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2634bcf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8330 0x5641b34eef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2634bcf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34d8c50 0x5641b34df3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2634bcf740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34e3320 0x5641b34daa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2634bcf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f262c1b1670 con 0x5641b34d8330 2024-05-09T08:46:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2634bcf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f262c1aef40 con 0x5641b34d8c50 2024-05-09T08:46:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2634bcf740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f262c1acdb0 con 0x5641b34e3320 2024-05-09T08:46:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2623fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8330 0x5641b34eef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2623fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8330 0x5641b34eef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35450/0 (socket says 172.21.15.107:35450) 2024-05-09T08:46:30.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2623fff640 1 -- 172.21.15.107:0/1701787590 learned_addr learned my addr 172.21.15.107:0/1701787590 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2623fff640 1 --2- 172.21.15.107:0/1701787590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34e3320 0x5641b34daa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2623fff640 1 -- 172.21.15.107:0/1701787590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34e3320 msgr2=0x5641b34daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2623fff640 1 --2- 172.21.15.107:0/1701787590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34e3320 0x5641b34daa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2623fff640 1 -- 172.21.15.107:0/1701787590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34d8c50 msgr2=0x5641b34df3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f262b7fe640 1 --2- 172.21.15.107:0/1701787590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34d8c50 0x5641b34df3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2623fff640 1 --2- 172.21.15.107:0/1701787590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34d8c50 0x5641b34df3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.436+0000 7f2623fff640 1 -- 172.21.15.107:0/1701787590 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641b34d95c0 con 0x5641b34d8330 2024-05-09T08:46:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.437+0000 7f2623fff640 1 --2- 172.21.15.107:0/1701787590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8330 0x5641b34eef70 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f26140059f0 tx=0x7f2614012bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e9a00b3e6e21a23 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.437+0000 7f26217fa640 1 -- 172.21.15.107:0/1701787590 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2614033070 con 0x5641b34d8330 2024-05-09T08:46:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.437+0000 7f26217fa640 1 -- 172.21.15.107:0/1701787590 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2614005720 con 0x5641b34d8330 2024-05-09T08:46:30.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.437+0000 7f26217fa640 1 -- 172.21.15.107:0/1701787590 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261400f9d0 con 0x5641b34d8330 2024-05-09T08:46:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.437+0000 7f2634bcf740 1 -- 172.21.15.107:0/1701787590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8330 msgr2=0x5641b34eef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.437+0000 7f2634bcf740 1 --2- 172.21.15.107:0/1701787590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8330 0x5641b34eef70 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f26140059f0 tx=0x7f2614012bb0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.437+0000 7f2634bcf740 1 -- 172.21.15.107:0/1701787590 shutdown_connections 2024-05-09T08:46:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.437+0000 7f2634bcf740 1 --2- 172.21.15.107:0/1701787590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34e3320 0x5641b34daa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.437+0000 7f2634bcf740 1 --2- 172.21.15.107:0/1701787590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34d8c50 0x5641b34df3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.437+0000 7f2634bcf740 1 --2- 172.21.15.107:0/1701787590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8330 0x5641b34eef70 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.437+0000 7f2634bcf740 1 -- 172.21.15.107:0/1701787590 >> 172.21.15.107:0/1701787590 conn(0x5641b34d4650 msgr2=0x5641b34d4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.438+0000 7f2634bcf740 1 -- 172.21.15.107:0/1701787590 shutdown_connections 2024-05-09T08:46:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.438+0000 7f2634bcf740 1 -- 172.21.15.107:0/1701787590 wait complete. 2024-05-09T08:46:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.438+0000 7f2634bcf740 1 -- start start 2024-05-09T08:46:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.438+0000 7f2634bcf740 1 Processor -- start 2024-05-09T08:46:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.438+0000 7f2634bcf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8330 0x5641b34eb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.438+0000 7f2634bcf740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34d8c50 0x5641b34e6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.438+0000 7f2634bcf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34e3320 0x5641b34e6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.438+0000 7f2634bcf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f262c1ad400 con 0x5641b34d8330 2024-05-09T08:46:30.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.438+0000 7f2634bcf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f262c1ae870 con 0x5641b34e3320 2024-05-09T08:46:30.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.438+0000 7f2634bcf740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f262c1adad0 con 0x5641b34d8c50 2024-05-09T08:46:30.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.438+0000 7f2634bcf740 10 client.0 _ll_register_callbacks cb 0x7f261400ceb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:30.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.438+0000 7f2623fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8330 0x5641b34eb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.438+0000 7f2623fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8330 0x5641b34eb6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35456/0 (socket says 172.21.15.107:35456) 2024-05-09T08:46:30.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.438+0000 7f2623fff640 1 -- 172.21.15.107:0/2110210074 learned_addr learned my addr 172.21.15.107:0/2110210074 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:30.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.438+0000 7f262b7fe640 1 --2- 172.21.15.107:0/2110210074 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34d8c50 0x5641b34e6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.439+0000 7f2623fff640 1 --2- 172.21.15.107:0/2110210074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34e3320 0x5641b34e6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.439+0000 7f2623fff640 1 -- 172.21.15.107:0/2110210074 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34d8c50 msgr2=0x5641b34e6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.439+0000 7f2623fff640 1 --2- 172.21.15.107:0/2110210074 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34d8c50 0x5641b34e6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.439+0000 7f2623fff640 1 -- 172.21.15.107:0/2110210074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34e3320 msgr2=0x5641b34e6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.439+0000 7f2623fff640 1 --2- 172.21.15.107:0/2110210074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34e3320 0x5641b34e6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.439+0000 7f2623fff640 1 -- 172.21.15.107:0/2110210074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641b34e75f0 con 0x5641b34d8330 2024-05-09T08:46:30.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.439+0000 7f2623fff640 1 --2- 172.21.15.107:0/2110210074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8330 0x5641b34eb6d0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f2614012e60 tx=0x7f2614012dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22eb6f5712085b44 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.439+0000 7f260affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:46:30.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.439+0000 7f260affd640 1 -- 172.21.15.107:0/2110210074 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2614033040 con 0x5641b34d8330 2024-05-09T08:46:30.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.439+0000 7f260affd640 1 -- 172.21.15.107:0/2110210074 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f261400fa60 con 0x5641b34d8330 2024-05-09T08:46:30.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.439+0000 7f2634bcf740 1 -- 172.21.15.107:0/2110210074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2614005720 con 0x5641b34d8330 2024-05-09T08:46:30.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.439+0000 7f260affd640 1 -- 172.21.15.107:0/2110210074 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2614043050 con 0x5641b34d8330 2024-05-09T08:46:30.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.439+0000 7f2634bcf740 10 client.19056 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:30.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.439+0000 7f2634bcf740 1 -- 172.21.15.107:0/2110210074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5641b3559ad0 con 0x5641b34d8330 2024-05-09T08:46:30.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.439+0000 7f260affd640 1 -- 172.21.15.107:0/2110210074 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2614005720 con 0x5641b34d8330 2024-05-09T08:46:30.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.440+0000 7f260affd640 1 -- 172.21.15.107:0/2110210074 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f2614004c20 con 0x5641b34d8330 2024-05-09T08:46:30.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.440+0000 7f2634bcf740 10 client.19056 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:30.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.440+0000 7f2634bcf740 10 client.19056 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:46:30.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.440+0000 7f2634bcf740 20 client.19056 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:30.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.440+0000 7f2634bcf740 1 --2- 172.21.15.107:0/2110210074 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5641b355a080 0x5641b355c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.440+0000 7f2634bcf740 4 client.19056 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:46:30.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.440+0000 7f2634bcf740 1 -- 172.21.15.107:0/2110210074 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5641b34456c0 con 0x5641b355a080 2024-05-09T08:46:30.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.440+0000 7f262b7fe640 1 --2- 172.21.15.107:0/2110210074 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5641b355a080 0x5641b355c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.440+0000 7f260affd640 10 client.19056 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:46:30.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.440+0000 7f262b7fe640 1 --2- 172.21.15.107:0/2110210074 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5641b355a080 0x5641b355c470 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b6cb2dcecf2ef387 server_cookie=dbf09e06346e7379 in_seq=0 out_seq=0 2024-05-09T08:46:30.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.440+0000 7f260affd640 1 -- 172.21.15.107:0/2110210074 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5641b34456c0 con 0x5641b355a080 2024-05-09T08:46:30.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.440+0000 7f260affd640 10 client.19056 handle_command_reply: tid=0 2024-05-09T08:46:30.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.440+0000 7f2634bcf740 1 client.19056 shutdown 2024-05-09T08:46:30.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2634bcf740 1 -- 172.21.15.107:0/2110210074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8330 msgr2=0x5641b34eb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2634bcf740 1 --2- 172.21.15.107:0/2110210074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8330 0x5641b34eb6d0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f2614012e60 tx=0x7f2614012dd0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2634bcf740 1 -- 172.21.15.107:0/2110210074 shutdown_connections 2024-05-09T08:46:30.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2634bcf740 1 --2- 172.21.15.107:0/2110210074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5641b34e3320 0x5641b34e6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2634bcf740 1 --2- 172.21.15.107:0/2110210074 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5641b355a080 0x5641b355c470 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2634bcf740 1 --2- 172.21.15.107:0/2110210074 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5641b34d8c50 0x5641b34e6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2634bcf740 1 --2- 172.21.15.107:0/2110210074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5641b34d8330 0x5641b34eb6d0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f260affd640 0 client.19056 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:46:30.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2634bcf740 1 -- 172.21.15.107:0/2110210074 >> 172.21.15.107:0/2110210074 conn(0x5641b34d4650 msgr2=0x5641b34d37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2634bcf740 1 -- 172.21.15.107:0/2110210074 shutdown_connections 2024-05-09T08:46:30.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2634bcf740 1 -- 172.21.15.107:0/2110210074 wait complete. 2024-05-09T08:46:30.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2634bcf740 20 client.19056 trim_cache size 0 max 16384 2024-05-09T08:46:30.451 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:46:30.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2631ded640 1 -- 172.21.15.107:0/2825977334 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f26180a2d00 msgr2=0x7f26180a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2631ded640 1 --2- 172.21.15.107:0/2825977334 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f26180a2d00 0x7f26180a51c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f262400a6e0 tx=0x7f2624090ea0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2631ded640 1 -- 172.21.15.107:0/2825977334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c107810 msgr2=0x7f262c19c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2631ded640 1 --2- 172.21.15.107:0/2825977334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c107810 0x7f262c19c950 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f262400cad0 tx=0x7f262400cf90 comp rx=0 tx=0).stop 2024-05-09T08:46:30.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2631ded640 1 -- 172.21.15.107:0/2825977334 shutdown_connections 2024-05-09T08:46:30.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2631ded640 1 --2- 172.21.15.107:0/2825977334 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f26180a2d00 0x7f26180a51c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2631ded640 1 --2- 172.21.15.107:0/2825977334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c107810 0x7f262c19c950 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2631ded640 1 --2- 172.21.15.107:0/2825977334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f262c06a7f0 0x7f262c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2631ded640 1 --2- 172.21.15.107:0/2825977334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f262c069e30 0x7f262c198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2631ded640 1 -- 172.21.15.107:0/2825977334 >> 172.21.15.107:0/2825977334 conn(0x7f262c076530 msgr2=0x7f262c0fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2631ded640 1 -- 172.21.15.107:0/2825977334 shutdown_connections 2024-05-09T08:46:30.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.441+0000 7f2631ded640 1 -- 172.21.15.107:0/2825977334 wait complete. 2024-05-09T08:46:30.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:30 smithi107 ceph-mon[30122]: pgmap v411: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-05-09T08:46:30.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:30 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/757259231' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:46:30.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:46:30.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:30 smithi157 ceph-mon[33614]: pgmap v411: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-05-09T08:46:30.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:30 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/757259231' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:46:30.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:30 smithi110 ceph-mon[33622]: pgmap v411: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-05-09T08:46:30.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:30 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/757259231' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:46:30.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.620+0000 7fbecded6640 1 -- 172.21.15.107:0/3768889404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbec810bdf0 msgr2=0x7fbec810e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.620+0000 7fbecded6640 1 --2- 172.21.15.107:0/3768889404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbec810bdf0 0x7fbec810e290 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009900 tx=0x7fbeb802a510 comp rx=0 tx=0).stop 2024-05-09T08:46:30.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.620+0000 7fbecded6640 1 -- 172.21.15.107:0/3768889404 shutdown_connections 2024-05-09T08:46:30.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.620+0000 7fbecded6640 1 --2- 172.21.15.107:0/3768889404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbec810bdf0 0x7fbec810e290 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.620+0000 7fbecded6640 1 --2- 172.21.15.107:0/3768889404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbec81033d0 0x7fbec810b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.620+0000 7fbecded6640 1 --2- 172.21.15.107:0/3768889404 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbec8102ab0 0x7fbec8102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.620+0000 7fbecded6640 1 -- 172.21.15.107:0/3768889404 >> 172.21.15.107:0/3768889404 conn(0x7fbec8077830 msgr2=0x7fbec8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.620+0000 7fbecded6640 1 -- 172.21.15.107:0/3768889404 shutdown_connections 2024-05-09T08:46:30.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.621+0000 7fbecded6640 1 -- 172.21.15.107:0/3768889404 wait complete. 2024-05-09T08:46:30.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.621+0000 7fbecded6640 1 Processor -- start 2024-05-09T08:46:30.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.621+0000 7fbecded6640 1 -- start start 2024-05-09T08:46:30.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.622+0000 7fbecded6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbec8102ab0 0x7fbec8198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.622+0000 7fbecded6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbec81033d0 0x7fbec8198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.622+0000 7fbecded6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbec810bdf0 0x7fbec819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.622+0000 7fbecded6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbec8111aa0 con 0x7fbec81033d0 2024-05-09T08:46:30.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.622+0000 7fbecded6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbec8111920 con 0x7fbec810bdf0 2024-05-09T08:46:30.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.622+0000 7fbecded6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbec8111c20 con 0x7fbec8102ab0 2024-05-09T08:46:30.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.622+0000 7fbec7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbec810bdf0 0x7fbec819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.622+0000 7fbec77fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbec8102ab0 0x7fbec8198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.622+0000 7fbec7fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbec810bdf0 0x7fbec819c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34538/0 (socket says 172.21.15.107:34538) 2024-05-09T08:46:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.622+0000 7fbec7fff640 1 -- 172.21.15.107:0/1329161514 learned_addr learned my addr 172.21.15.107:0/1329161514 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.623+0000 7fbec7fff640 1 -- 172.21.15.107:0/1329161514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbec8102ab0 msgr2=0x7fbec8198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.623+0000 7fbec7fff640 1 --2- 172.21.15.107:0/1329161514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbec8102ab0 0x7fbec8198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.623+0000 7fbec7fff640 1 -- 172.21.15.107:0/1329161514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbec81033d0 msgr2=0x7fbec8198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.623+0000 7fbec7fff640 1 --2- 172.21.15.107:0/1329161514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbec81033d0 0x7fbec8198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.623+0000 7fbec7fff640 1 -- 172.21.15.107:0/1329161514 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec819cf90 con 0x7fbec810bdf0 2024-05-09T08:46:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.623+0000 7fbec7fff640 1 --2- 172.21.15.107:0/1329161514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbec810bdf0 0x7fbec819c890 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8004770 tx=0x7fbeb8004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.623+0000 7fbec4ff9640 1 -- 172.21.15.107:0/1329161514 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb803f030 con 0x7fbec810bdf0 2024-05-09T08:46:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.623+0000 7fbec4ff9640 1 -- 172.21.15.107:0/1329161514 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbeb8004440 con 0x7fbec810bdf0 2024-05-09T08:46:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.623+0000 7fbecded6640 1 -- 172.21.15.107:0/1329161514 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbec819d220 con 0x7fbec810bdf0 2024-05-09T08:46:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.623+0000 7fbec4ff9640 1 -- 172.21.15.107:0/1329161514 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8035070 con 0x7fbec810bdf0 2024-05-09T08:46:30.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.623+0000 7fbecded6640 1 -- 172.21.15.107:0/1329161514 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbec8100630 con 0x7fbec810bdf0 2024-05-09T08:46:30.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.627+0000 7fbed0be1740 1 Processor -- start 2024-05-09T08:46:30.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.628+0000 7fbec4ff9640 1 -- 172.21.15.107:0/1329161514 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fbeb8042030 con 0x7fbec810bdf0 2024-05-09T08:46:30.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.628+0000 7fbed0be1740 1 -- start start 2024-05-09T08:46:30.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.628+0000 7fbed0be1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 0x5605fe89b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.628+0000 7fbed0be1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89bc50 0x5605fe8a5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.628+0000 7fbed0be1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 0x5605fe8a8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.628+0000 7fbed0be1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5605fe800c40 con 0x5605fe8a6320 2024-05-09T08:46:30.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.628+0000 7fbed0be1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5605fe7a8580 con 0x5605fe89bc50 2024-05-09T08:46:30.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.628+0000 7fbed0be1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5605fe7fcc00 con 0x5605fe89b330 2024-05-09T08:46:30.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.629+0000 7fbec6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 0x5605fe8a8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.629+0000 7fbec6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 0x5605fe8a8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33076/0 (socket says 172.21.15.107:33076) 2024-05-09T08:46:30.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.629+0000 7fbec6ffd640 1 -- 172.21.15.107:0/3982477832 learned_addr learned my addr 172.21.15.107:0/3982477832 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.629+0000 7fbec4ff9640 1 --2- 172.21.15.107:0/1329161514 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbeb00a2db0 0x7fbeb00a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.629+0000 7fbec6ffd640 1 --2- 172.21.15.107:0/3982477832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 0x5605fe89b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.629+0000 7fbec77fe640 1 --2- 172.21.15.107:0/3982477832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89bc50 0x5605fe8a5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.629+0000 7fbec7fff640 1 --2- 172.21.15.107:0/1329161514 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbeb00a2db0 0x7fbeb00a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.630+0000 7fbec4ff9640 1 -- 172.21.15.107:0/1329161514 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7fbeb80ef960 con 0x7fbec810bdf0 2024-05-09T08:46:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.630+0000 7fbec77fe640 1 -- 172.21.15.107:0/3982477832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 msgr2=0x5605fe89b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.630+0000 7fbec77fe640 1 --2- 172.21.15.107:0/3982477832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 0x5605fe89b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.630+0000 7fbec77fe640 1 -- 172.21.15.107:0/3982477832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 msgr2=0x5605fe8a8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.630+0000 7fbec77fe640 1 --2- 172.21.15.107:0/3982477832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 0x5605fe8a8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.630+0000 7fbec77fe640 1 -- 172.21.15.107:0/3982477832 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605fe5143f0 con 0x5605fe89bc50 2024-05-09T08:46:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.630+0000 7fbec77fe640 1 --2- 172.21.15.107:0/3982477832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89bc50 0x5605fe8a5de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbec0009f80 tx=0x7fbec00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3248b3183caa0576 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.630+0000 7fbeadffb640 1 -- 172.21.15.107:0/3982477832 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec0004d20 con 0x5605fe89bc50 2024-05-09T08:46:30.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.630+0000 7fbec7fff640 1 --2- 172.21.15.107:0/1329161514 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbeb00a2db0 0x7fbeb00a5270 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80ecd00 tx=0x7fbeb80b7610 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.630+0000 7fbeadffb640 1 -- 172.21.15.107:0/3982477832 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbec0004050 con 0x5605fe89bc50 2024-05-09T08:46:30.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.630+0000 7fbeadffb640 1 -- 172.21.15.107:0/3982477832 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec0005260 con 0x5605fe89bc50 2024-05-09T08:46:30.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.631+0000 7fbed0be1740 1 -- 172.21.15.107:0/3982477832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89bc50 msgr2=0x5605fe8a5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.631+0000 7fbed0be1740 1 --2- 172.21.15.107:0/3982477832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89bc50 0x5605fe8a5de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbec0009f80 tx=0x7fbec00118d0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.632+0000 7fbed0be1740 1 -- 172.21.15.107:0/3982477832 shutdown_connections 2024-05-09T08:46:30.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.632+0000 7fbed0be1740 1 --2- 172.21.15.107:0/3982477832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 0x5605fe8a8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.632+0000 7fbed0be1740 1 --2- 172.21.15.107:0/3982477832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89bc50 0x5605fe8a5de0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.632+0000 7fbed0be1740 1 --2- 172.21.15.107:0/3982477832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 0x5605fe89b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.632+0000 7fbed0be1740 1 -- 172.21.15.107:0/3982477832 >> 172.21.15.107:0/3982477832 conn(0x5605fe897650 msgr2=0x5605fe897a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.632+0000 7fbed0be1740 1 -- 172.21.15.107:0/3982477832 shutdown_connections 2024-05-09T08:46:30.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.633+0000 7fbed0be1740 1 -- 172.21.15.107:0/3982477832 wait complete. 2024-05-09T08:46:30.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.633+0000 7fbed0be1740 1 -- start start 2024-05-09T08:46:30.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.633+0000 7fbed0be1740 1 Processor -- start 2024-05-09T08:46:30.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbed0be1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89b330 0x5605fe8a3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbed0be1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89bc50 0x5605fe8a3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbed0be1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 0x5605fe8a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbed0be1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5605fe8a9110 con 0x5605fe8a6320 2024-05-09T08:46:30.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbed0be1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5605fe7bbc40 con 0x5605fe89b330 2024-05-09T08:46:30.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbed0be1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5605fe8a9290 con 0x5605fe89bc50 2024-05-09T08:46:30.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbed0be1740 10 client.0 _ll_register_callbacks cb 0x5605fe513be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:30.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbec6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 0x5605fe8a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbec77fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89bc50 0x5605fe8a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbec6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 0x5605fe8a3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33082/0 (socket says 172.21.15.107:33082) 2024-05-09T08:46:30.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbec6ffd640 1 -- 172.21.15.107:0/2777238070 learned_addr learned my addr 172.21.15.107:0/2777238070 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:30.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbec6ffd640 1 --2- 172.21.15.107:0/2777238070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89b330 0x5605fe8a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbec6ffd640 1 -- 172.21.15.107:0/2777238070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89bc50 msgr2=0x5605fe8a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbec6ffd640 1 --2- 172.21.15.107:0/2777238070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89bc50 0x5605fe8a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbec6ffd640 1 -- 172.21.15.107:0/2777238070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 msgr2=0x5605fe8a3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbec6ffd640 1 --2- 172.21.15.107:0/2777238070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 0x5605fe8a3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.634+0000 7fbec6ffd640 1 -- 172.21.15.107:0/2777238070 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605fe5de7d0 con 0x5605fe89b330 2024-05-09T08:46:30.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.635+0000 7fbec6ffd640 1 --2- 172.21.15.107:0/2777238070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89b330 0x5605fe8a3250 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009430 tx=0x7fbebc005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bacadc99cd895f4f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.635+0000 7fbea6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:46:30.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.635+0000 7fbea6ffd640 1 -- 172.21.15.107:0/2777238070 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc004230 con 0x5605fe89b330 2024-05-09T08:46:30.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.635+0000 7fbea6ffd640 1 -- 172.21.15.107:0/2777238070 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7fbebc0043d0 con 0x5605fe89b330 2024-05-09T08:46:30.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.635+0000 7fbed0be1740 1 -- 172.21.15.107:0/2777238070 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbec0004050 con 0x5605fe89b330 2024-05-09T08:46:30.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.636+0000 7fbea6ffd640 1 -- 172.21.15.107:0/2777238070 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbebc038050 con 0x5605fe89b330 2024-05-09T08:46:30.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.636+0000 7fbed0be1740 10 client.28007 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:30.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.636+0000 7fbed0be1740 1 -- 172.21.15.107:0/2777238070 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5605fe8b5310 con 0x5605fe89b330 2024-05-09T08:46:30.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.636+0000 7fbea6ffd640 1 -- 172.21.15.107:0/2777238070 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbebc004c50 con 0x5605fe89b330 2024-05-09T08:46:30.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.636+0000 7fbea6ffd640 1 -- 172.21.15.107:0/2777238070 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fbebc0053e0 con 0x5605fe89b330 2024-05-09T08:46:30.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.636+0000 7fbed0be1740 10 client.28007 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:30.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.636+0000 7fbed0be1740 10 client.28007 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:46:30.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.636+0000 7fbed0be1740 20 client.28007 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:30.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.636+0000 7fbed0be1740 1 --2- 172.21.15.107:0/2777238070 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5605fe8b67d0 0x5605fe8b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.636+0000 7fbed0be1740 4 client.28007 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:46:30.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.636+0000 7fbed0be1740 1 -- 172.21.15.107:0/2777238070 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbec0004050 con 0x5605fe8b67d0 2024-05-09T08:46:30.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.636+0000 7fbec77fe640 1 --2- 172.21.15.107:0/2777238070 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5605fe8b67d0 0x5605fe8b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.638+0000 7fbec77fe640 1 --2- 172.21.15.107:0/2777238070 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5605fe8b67d0 0x5605fe8b8c00 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8eb3b6cd36bc5f9 server_cookie=971777d0af1afbf1 in_seq=0 out_seq=0 2024-05-09T08:46:30.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.638+0000 7fbea6ffd640 10 client.28007 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:30.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.638+0000 7fbea6ffd640 1 -- 172.21.15.107:0/2777238070 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fbec0004050 con 0x5605fe8b67d0 2024-05-09T08:46:30.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.638+0000 7fbea6ffd640 10 client.28007 handle_command_reply: tid=0 2024-05-09T08:46:30.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.638+0000 7fbed0be1740 1 client.28007 shutdown 2024-05-09T08:46:30.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.639+0000 7fbed0be1740 1 -- 172.21.15.107:0/2777238070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89b330 msgr2=0x5605fe8a3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.639+0000 7fbed0be1740 1 --2- 172.21.15.107:0/2777238070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89b330 0x5605fe8a3250 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009430 tx=0x7fbebc005f90 comp rx=0 tx=0).stop 2024-05-09T08:46:30.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.639+0000 7fbed0be1740 1 -- 172.21.15.107:0/2777238070 shutdown_connections 2024-05-09T08:46:30.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.639+0000 7fbed0be1740 1 --2- 172.21.15.107:0/2777238070 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5605fe8b67d0 0x5605fe8b8c00 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.639+0000 7fbea6ffd640 0 client.28007 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:30.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.639+0000 7fbed0be1740 1 --2- 172.21.15.107:0/2777238070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 0x5605fe8a3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.639+0000 7fbed0be1740 1 --2- 172.21.15.107:0/2777238070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89bc50 0x5605fe8a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.639+0000 7fbed0be1740 1 --2- 172.21.15.107:0/2777238070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89b330 0x5605fe8a3250 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.639+0000 7fbed0be1740 1 -- 172.21.15.107:0/2777238070 >> 172.21.15.107:0/2777238070 conn(0x5605fe897650 msgr2=0x5605fe89d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.639+0000 7fbed0be1740 1 -- 172.21.15.107:0/2777238070 shutdown_connections 2024-05-09T08:46:30.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.639+0000 7fbed0be1740 1 -- 172.21.15.107:0/2777238070 wait complete. 2024-05-09T08:46:30.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.639+0000 7fbed0be1740 20 client.28007 trim_cache size 0 max 16384 2024-05-09T08:46:30.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.659+0000 7fbed0be1740 1 Processor -- start 2024-05-09T08:46:30.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.659+0000 7fbed0be1740 1 -- start start 2024-05-09T08:46:30.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.659+0000 7fbed0be1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 0x5605fe8a5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.659+0000 7fbed0be1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89bc50 0x5605fe89da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.659+0000 7fbed0be1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 0x5605fe8a3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.659+0000 7fbed0be1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbec81af440 con 0x5605fe8a6320 2024-05-09T08:46:30.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.659+0000 7fbed0be1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbec81acd10 con 0x5605fe89bc50 2024-05-09T08:46:30.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.659+0000 7fbed0be1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbec81aab80 con 0x5605fe89b330 2024-05-09T08:46:30.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.660+0000 7fbec6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 0x5605fe8a3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.660+0000 7fbec6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 0x5605fe8a3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33092/0 (socket says 172.21.15.107:33092) 2024-05-09T08:46:30.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.660+0000 7fbec6ffd640 1 -- 172.21.15.107:0/345809964 learned_addr learned my addr 172.21.15.107:0/345809964 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:30.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.660+0000 7fbec77fe640 1 --2- 172.21.15.107:0/345809964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89bc50 0x5605fe89da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.660+0000 7fbec6ffd640 1 --2- 172.21.15.107:0/345809964 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 0x5605fe8a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.660+0000 7fbec6ffd640 1 -- 172.21.15.107:0/345809964 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 msgr2=0x5605fe8a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.660+0000 7fbec6ffd640 1 --2- 172.21.15.107:0/345809964 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 0x5605fe8a5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.660+0000 7fbec6ffd640 1 -- 172.21.15.107:0/345809964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89bc50 msgr2=0x5605fe89da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.660+0000 7fbec6ffd640 1 --2- 172.21.15.107:0/345809964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89bc50 0x5605fe89da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.660+0000 7fbec6ffd640 1 -- 172.21.15.107:0/345809964 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605fe897480 con 0x5605fe8a6320 2024-05-09T08:46:30.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.660+0000 7fbec77fe640 1 --2- 172.21.15.107:0/345809964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89bc50 0x5605fe89da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:46:30.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.660+0000 7fbec6ffd640 1 --2- 172.21.15.107:0/345809964 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 0x5605fe8a3af0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009dd0 tx=0x7fbebc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32cfc7bb037e95c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.661+0000 7fbead7fa640 1 -- 172.21.15.107:0/345809964 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc005020 con 0x5605fe8a6320 2024-05-09T08:46:30.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.661+0000 7fbead7fa640 1 -- 172.21.15.107:0/345809964 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbebc0047a0 con 0x5605fe8a6320 2024-05-09T08:46:30.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.661+0000 7fbead7fa640 1 -- 172.21.15.107:0/345809964 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc03c640 con 0x5605fe8a6320 2024-05-09T08:46:30.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.662+0000 7fbed0be1740 1 -- 172.21.15.107:0/345809964 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 msgr2=0x5605fe8a3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.662+0000 7fbed0be1740 1 --2- 172.21.15.107:0/345809964 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 0x5605fe8a3af0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009dd0 tx=0x7fbebc0046c0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.662+0000 7fbed0be1740 1 -- 172.21.15.107:0/345809964 shutdown_connections 2024-05-09T08:46:30.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.662+0000 7fbed0be1740 1 --2- 172.21.15.107:0/345809964 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe8a6320 0x5605fe8a3af0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.662+0000 7fbed0be1740 1 --2- 172.21.15.107:0/345809964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe89bc50 0x5605fe89da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.662+0000 7fbed0be1740 1 --2- 172.21.15.107:0/345809964 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 0x5605fe8a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.662+0000 7fbed0be1740 1 -- 172.21.15.107:0/345809964 >> 172.21.15.107:0/345809964 conn(0x5605fe897650 msgr2=0x5605fe8967b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.663+0000 7fbed0be1740 1 -- 172.21.15.107:0/345809964 shutdown_connections 2024-05-09T08:46:30.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.663+0000 7fbed0be1740 1 -- 172.21.15.107:0/345809964 wait complete. 2024-05-09T08:46:30.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.663+0000 7fbed0be1740 1 -- start start 2024-05-09T08:46:30.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.664+0000 7fbed0be1740 1 Processor -- start 2024-05-09T08:46:30.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.664+0000 7fbed0be1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 0x5605fe8ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.664+0000 7fbed0be1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe89bc50 0x5605fe8a9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.664+0000 7fbed0be1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe8a6320 0x5605fe8aa1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.664+0000 7fbec77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe89bc50 0x5605fe8a9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.664+0000 7fbec6ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 0x5605fe8ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.664+0000 7fbec6ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 0x5605fe8ae8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:55032/0 (socket says 172.21.15.107:55032) 2024-05-09T08:46:30.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.664+0000 7fbec6ffd640 1 -- 172.21.15.107:0/3721247520 learned_addr learned my addr 172.21.15.107:0/3721247520 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:30.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.664+0000 7fbed0be1740 1 -- 172.21.15.107:0/3721247520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbec81ab1d0 con 0x5605fe89bc50 2024-05-09T08:46:30.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.664+0000 7fbed0be1740 1 -- 172.21.15.107:0/3721247520 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbec81ac640 con 0x5605fe8a6320 2024-05-09T08:46:30.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.664+0000 7fbed0be1740 1 -- 172.21.15.107:0/3721247520 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbec81ab8a0 con 0x5605fe89b330 2024-05-09T08:46:30.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.665+0000 7fbec6ffd640 1 --2- 172.21.15.107:0/3721247520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe8a6320 0x5605fe8aa1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.665+0000 7fbed0be1740 10 client.0 _ll_register_callbacks cb 0x7fbebc011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:30.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.665+0000 7fbec6ffd640 1 -- 172.21.15.107:0/3721247520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe8a6320 msgr2=0x5605fe8aa1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.665+0000 7fbec6ffd640 1 --2- 172.21.15.107:0/3721247520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe8a6320 0x5605fe8aa1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.665+0000 7fbec6ffd640 1 -- 172.21.15.107:0/3721247520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe89bc50 msgr2=0x5605fe8a9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:30.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.665+0000 7fbec6ffd640 1 --2- 172.21.15.107:0/3721247520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe89bc50 0x5605fe8a9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.665+0000 7fbec6ffd640 1 -- 172.21.15.107:0/3721247520 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605fe8aa9d0 con 0x5605fe89b330 2024-05-09T08:46:30.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.665+0000 7fbec6ffd640 1 --2- 172.21.15.107:0/3721247520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 0x5605fe8ae8f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009da0 tx=0x7fbebc009d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15e699b2d54f5edc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:30.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.665+0000 7fbea6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:46:30.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.665+0000 7fbea6ffd640 1 -- 172.21.15.107:0/3721247520 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc005020 con 0x5605fe89b330 2024-05-09T08:46:30.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.666+0000 7fbea6ffd640 1 -- 172.21.15.107:0/3721247520 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7fbebc0051c0 con 0x5605fe89b330 2024-05-09T08:46:30.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.666+0000 7fbed0be1740 1 -- 172.21.15.107:0/3721247520 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbebc0047a0 con 0x5605fe89b330 2024-05-09T08:46:30.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.666+0000 7fbea6ffd640 1 -- 172.21.15.107:0/3721247520 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbebc0053c0 con 0x5605fe89b330 2024-05-09T08:46:30.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.666+0000 7fbed0be1740 10 client.28222 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:30.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.666+0000 7fbed0be1740 1 -- 172.21.15.107:0/3721247520 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5605fe91cd40 con 0x5605fe89b330 2024-05-09T08:46:30.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.666+0000 7fbea6ffd640 1 -- 172.21.15.107:0/3721247520 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbebc0047a0 con 0x5605fe89b330 2024-05-09T08:46:30.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.666+0000 7fbea6ffd640 1 -- 172.21.15.107:0/3721247520 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fbebc033410 con 0x5605fe89b330 2024-05-09T08:46:30.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.667+0000 7fbed0be1740 10 client.28222 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:30.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.667+0000 7fbed0be1740 10 client.28222 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:46:30.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.667+0000 7fbed0be1740 20 client.28222 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:30.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.667+0000 7fbed0be1740 1 --2- 172.21.15.107:0/3721247520 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5605fe8b67d0 0x5605fe91f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:30.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.667+0000 7fbed0be1740 4 client.28222 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:46:30.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.667+0000 7fbed0be1740 1 -- 172.21.15.107:0/3721247520 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5605fe80e370 con 0x5605fe8b67d0 2024-05-09T08:46:30.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.667+0000 7fbec77fe640 1 --2- 172.21.15.107:0/3721247520 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5605fe8b67d0 0x5605fe91f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:30.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.668+0000 7fbec77fe640 1 --2- 172.21.15.107:0/3721247520 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5605fe8b67d0 0x5605fe91f350 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a995bbeca4751301 server_cookie=62794292841fe31a in_seq=0 out_seq=0 2024-05-09T08:46:30.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.668+0000 7fbea6ffd640 10 client.28222 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:30.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.670+0000 7fbea6ffd640 1 -- 172.21.15.107:0/3721247520 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5605fe80e370 con 0x5605fe8b67d0 2024-05-09T08:46:30.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.670+0000 7fbea6ffd640 10 client.28222 handle_command_reply: tid=0 2024-05-09T08:46:30.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.670+0000 7fbed0be1740 1 client.28222 shutdown 2024-05-09T08:46:30.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.670+0000 7fbed0be1740 1 -- 172.21.15.107:0/3721247520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 msgr2=0x5605fe8ae8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.670+0000 7fbed0be1740 1 --2- 172.21.15.107:0/3721247520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 0x5605fe8ae8f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009da0 tx=0x7fbebc009d30 comp rx=0 tx=0).stop 2024-05-09T08:46:30.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.671+0000 7fbed0be1740 1 -- 172.21.15.107:0/3721247520 shutdown_connections 2024-05-09T08:46:30.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.671+0000 7fbed0be1740 1 --2- 172.21.15.107:0/3721247520 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5605fe8b67d0 0x5605fe91f350 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.671+0000 7fbea6ffd640 0 client.28222 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:30.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.672+0000 7fbed0be1740 1 --2- 172.21.15.107:0/3721247520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5605fe8a6320 0x5605fe8aa1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.672+0000 7fbed0be1740 1 --2- 172.21.15.107:0/3721247520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5605fe89bc50 0x5605fe8a9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.672+0000 7fbed0be1740 1 --2- 172.21.15.107:0/3721247520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5605fe89b330 0x5605fe8ae8f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.672+0000 7fbed0be1740 1 -- 172.21.15.107:0/3721247520 >> 172.21.15.107:0/3721247520 conn(0x5605fe897650 msgr2=0x5605fe8b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.672+0000 7fbed0be1740 1 -- 172.21.15.107:0/3721247520 shutdown_connections 2024-05-09T08:46:30.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.672+0000 7fbed0be1740 1 -- 172.21.15.107:0/3721247520 wait complete. 2024-05-09T08:46:30.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.672+0000 7fbed0be1740 20 client.28222 trim_cache size 0 max 16384 2024-05-09T08:46:30.686 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:46:30.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.673+0000 7fbecded6640 1 -- 172.21.15.107:0/1329161514 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbeb00a2db0 msgr2=0x7fbeb00a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.673+0000 7fbecded6640 1 --2- 172.21.15.107:0/1329161514 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbeb00a2db0 0x7fbeb00a5270 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80ecd00 tx=0x7fbeb80b7610 comp rx=0 tx=0).stop 2024-05-09T08:46:30.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.673+0000 7fbecded6640 1 -- 172.21.15.107:0/1329161514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbec810bdf0 msgr2=0x7fbec819c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:30.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.673+0000 7fbecded6640 1 --2- 172.21.15.107:0/1329161514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbec810bdf0 0x7fbec819c890 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8004770 tx=0x7fbeb8004040 comp rx=0 tx=0).stop 2024-05-09T08:46:30.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.673+0000 7fbecded6640 1 -- 172.21.15.107:0/1329161514 shutdown_connections 2024-05-09T08:46:30.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.673+0000 7fbecded6640 1 --2- 172.21.15.107:0/1329161514 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbeb00a2db0 0x7fbeb00a5270 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.673+0000 7fbecded6640 1 --2- 172.21.15.107:0/1329161514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbec810bdf0 0x7fbec819c890 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.673+0000 7fbecded6640 1 --2- 172.21.15.107:0/1329161514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbec81033d0 0x7fbec8198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.673+0000 7fbecded6640 1 --2- 172.21.15.107:0/1329161514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbec8102ab0 0x7fbec8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:30.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.673+0000 7fbecded6640 1 -- 172.21.15.107:0/1329161514 >> 172.21.15.107:0/1329161514 conn(0x7fbec8077830 msgr2=0x7fbec80fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:30.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.673+0000 7fbecded6640 1 -- 172.21.15.107:0/1329161514 shutdown_connections 2024-05-09T08:46:30.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:30.673+0000 7fbecded6640 1 -- 172.21.15.107:0/1329161514 wait complete. 2024-05-09T08:46:31.692 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:46:31.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:46:31.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:46:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.867+0000 7f03ca461640 1 -- 172.21.15.107:0/2895522014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c4107810 msgr2=0x7f03c4107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.867+0000 7f03ca461640 1 --2- 172.21.15.107:0/2895522014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c4107810 0x7f03c4107bf0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f03b4009930 tx=0x7f03b402a420 comp rx=0 tx=0).stop 2024-05-09T08:46:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.867+0000 7f03ca461640 1 -- 172.21.15.107:0/2895522014 shutdown_connections 2024-05-09T08:46:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.867+0000 7f03ca461640 1 --2- 172.21.15.107:0/2895522014 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03c41021c0 0x7f03c410e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.867+0000 7f03ca461640 1 --2- 172.21.15.107:0/2895522014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03c4101800 0x7f03c4101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.867+0000 7f03ca461640 1 --2- 172.21.15.107:0/2895522014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c4107810 0x7f03c4107bf0 secure :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f03b4009930 tx=0x7f03b402a420 comp rx=0 tx=0).stop 2024-05-09T08:46:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.867+0000 7f03ca461640 1 -- 172.21.15.107:0/2895522014 >> 172.21.15.107:0/2895522014 conn(0x7f03c4078c80 msgr2=0x7f03c4079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.867+0000 7f03ca461640 1 -- 172.21.15.107:0/2895522014 shutdown_connections 2024-05-09T08:46:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.868+0000 7f03ca461640 1 -- 172.21.15.107:0/2895522014 wait complete. 2024-05-09T08:46:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.869+0000 7f03ca461640 1 Processor -- start 2024-05-09T08:46:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.869+0000 7f03ca461640 1 -- start start 2024-05-09T08:46:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.869+0000 7f03ca461640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03c4101800 0x7f03c419a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.869+0000 7f03ca461640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03c41021c0 0x7f03c419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.869+0000 7f03ca461640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c419eda0 0x7f03c419f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.869+0000 7f03ca461640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f03c4111ae0 con 0x7f03c419eda0 2024-05-09T08:46:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.869+0000 7f03ca461640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f03c4111960 con 0x7f03c41021c0 2024-05-09T08:46:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.869+0000 7f03ca461640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f03c4111c60 con 0x7f03c4101800 2024-05-09T08:46:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.869+0000 7f03c37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03c41021c0 0x7f03c419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.870+0000 7f03c89fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c419eda0 0x7f03c419f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.870+0000 7f03c37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03c41021c0 0x7f03c419aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:36226/0 (socket says 172.21.15.107:36226) 2024-05-09T08:46:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.870+0000 7f03c37fe640 1 -- 172.21.15.107:0/3817473402 learned_addr learned my addr 172.21.15.107:0/3817473402 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:31.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.870+0000 7f03c3fff640 1 --2- 172.21.15.107:0/3817473402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03c4101800 0x7f03c419a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:31.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.870+0000 7f03c89fe640 1 -- 172.21.15.107:0/3817473402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03c4101800 msgr2=0x7f03c419a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:31.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.870+0000 7f03c89fe640 1 --2- 172.21.15.107:0/3817473402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03c4101800 0x7f03c419a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.870+0000 7f03c89fe640 1 -- 172.21.15.107:0/3817473402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03c41021c0 msgr2=0x7f03c419aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:31.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.870+0000 7f03c89fe640 1 --2- 172.21.15.107:0/3817473402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03c41021c0 0x7f03c419aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.870+0000 7f03c89fe640 1 -- 172.21.15.107:0/3817473402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c419f890 con 0x7f03c419eda0 2024-05-09T08:46:31.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.871+0000 7f03c89fe640 1 --2- 172.21.15.107:0/3817473402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c419eda0 0x7f03c419f250 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00ec50 tx=0x7f03bc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:31.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.871+0000 7f03c17fa640 1 -- 172.21.15.107:0/3817473402 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc003b70 con 0x7f03c419eda0 2024-05-09T08:46:31.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.871+0000 7f03c17fa640 1 -- 172.21.15.107:0/3817473402 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f03bc003d10 con 0x7f03c419eda0 2024-05-09T08:46:31.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.871+0000 7f03c17fa640 1 -- 172.21.15.107:0/3817473402 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc0107f0 con 0x7f03c419eda0 2024-05-09T08:46:31.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.871+0000 7f03ca461640 1 -- 172.21.15.107:0/3817473402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03c419fb80 con 0x7f03c419eda0 2024-05-09T08:46:31.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.872+0000 7f03ca461640 1 -- 172.21.15.107:0/3817473402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03c4071550 con 0x7f03c419eda0 2024-05-09T08:46:31.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.877+0000 7f03cd168740 1 Processor -- start 2024-05-09T08:46:31.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.878+0000 7f03cd168740 1 -- start start 2024-05-09T08:46:31.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.878+0000 7f03cd168740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e101560b0 0x555e101543f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.878+0000 7f03cd168740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e101549c0 0x555e10154e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.878+0000 7f03cd168740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e10167f50 0x555e1016a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.878+0000 7f03cd168740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555e100b9990 con 0x555e10167f50 2024-05-09T08:46:31.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.878+0000 7f03cd168740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555e100615b0 con 0x555e101560b0 2024-05-09T08:46:31.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.878+0000 7f03cd168740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x555e100b5950 con 0x555e101549c0 2024-05-09T08:46:31.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.878+0000 7f03c17fa640 1 -- 172.21.15.107:0/3817473402 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f03bc003ec0 con 0x7f03c419eda0 2024-05-09T08:46:31.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.879+0000 7f03c17fa640 1 --2- 172.21.15.107:0/3817473402 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f03ac0a2ed0 0x7f03ac0a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.879+0000 7f03c17fa640 1 -- 172.21.15.107:0/3817473402 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f03bc0cfc80 con 0x7f03c419eda0 2024-05-09T08:46:31.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.880+0000 7f03c37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e101560b0 0x555e101543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:31.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.880+0000 7f03c3fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e101549c0 0x555e10154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:31.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.880+0000 7f03c37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e101560b0 0x555e101543f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:36236/0 (socket says 172.21.15.107:36236) 2024-05-09T08:46:31.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.880+0000 7f03c37fe640 1 -- 172.21.15.107:0/1255779461 learned_addr learned my addr 172.21.15.107:0/1255779461 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:31.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.880+0000 7f03c37fe640 1 --2- 172.21.15.107:0/1255779461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e10167f50 0x555e1016a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:31.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.880+0000 7f03c37fe640 1 -- 172.21.15.107:0/1255779461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e101549c0 msgr2=0x555e10154e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:31.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.880+0000 7f03c37fe640 1 --2- 172.21.15.107:0/1255779461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e101549c0 0x555e10154e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.880+0000 7f03c37fe640 1 -- 172.21.15.107:0/1255779461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e10167f50 msgr2=0x555e1016a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:31.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.880+0000 7f03c37fe640 1 --2- 172.21.15.107:0/1255779461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e10167f50 0x555e1016a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.880+0000 7f03c37fe640 1 -- 172.21.15.107:0/1255779461 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e0fe957e0 con 0x555e101560b0 2024-05-09T08:46:31.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.880+0000 7f03c89fe640 1 --2- 172.21.15.107:0/3817473402 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f03ac0a2ed0 0x7f03ac0a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:31.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.881+0000 7f03c37fe640 1 --2- 172.21.15.107:0/1255779461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e101560b0 0x555e101543f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f03b80118d0 tx=0x7f03b80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddea14926e345df7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:31.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.881+0000 7f03c89fe640 1 --2- 172.21.15.107:0/3817473402 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f03ac0a2ed0 0x7f03ac0a5390 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f03bc09b9e0 tx=0x7f03bc00ef50 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:31.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.882+0000 7f03aa7fc640 1 -- 172.21.15.107:0/1255779461 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b8005020 con 0x555e101560b0 2024-05-09T08:46:31.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.882+0000 7f03aa7fc640 1 -- 172.21.15.107:0/1255779461 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f03b80047a0 con 0x555e101560b0 2024-05-09T08:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.883+0000 7f03cd168740 1 -- 172.21.15.107:0/1255779461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e101560b0 msgr2=0x555e101543f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.883+0000 7f03cd168740 1 --2- 172.21.15.107:0/1255779461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e101560b0 0x555e101543f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f03b80118d0 tx=0x7f03b80046c0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.884+0000 7f03cd168740 1 -- 172.21.15.107:0/1255779461 shutdown_connections 2024-05-09T08:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.884+0000 7f03cd168740 1 --2- 172.21.15.107:0/1255779461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e10167f50 0x555e1016a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.884+0000 7f03cd168740 1 --2- 172.21.15.107:0/1255779461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e101549c0 0x555e10154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.884+0000 7f03cd168740 1 --2- 172.21.15.107:0/1255779461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e101560b0 0x555e101543f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.884+0000 7f03cd168740 1 -- 172.21.15.107:0/1255779461 >> 172.21.15.107:0/1255779461 conn(0x555e1011d0a0 msgr2=0x555e10150f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.884+0000 7f03cd168740 1 -- 172.21.15.107:0/1255779461 shutdown_connections 2024-05-09T08:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.884+0000 7f03cd168740 1 -- 172.21.15.107:0/1255779461 wait complete. 2024-05-09T08:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.885+0000 7f03cd168740 1 -- start start 2024-05-09T08:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.885+0000 7f03cd168740 1 Processor -- start 2024-05-09T08:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.885+0000 7f03cd168740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e101549c0 0x555e10161bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.885+0000 7f03cd168740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101560b0 0x555e101678f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.885+0000 7f03cd168740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e10167f50 0x555e1015cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.885+0000 7f03cd168740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555e10157af0 con 0x555e101560b0 2024-05-09T08:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.885+0000 7f03cd168740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555e10074b40 con 0x555e10167f50 2024-05-09T08:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.885+0000 7f03cd168740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x555e10157c70 con 0x555e101549c0 2024-05-09T08:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.885+0000 7f03cd168740 10 client.0 _ll_register_callbacks cb 0x555e100690f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:31.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.886+0000 7f03c3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101560b0 0x555e101678f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:31.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.886+0000 7f03c3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101560b0 0x555e101678f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33142/0 (socket says 172.21.15.107:33142) 2024-05-09T08:46:31.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.886+0000 7f03c3fff640 1 -- 172.21.15.107:0/806322746 learned_addr learned my addr 172.21.15.107:0/806322746 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:31.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.886+0000 7f03c3fff640 1 -- 172.21.15.107:0/806322746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e101549c0 msgr2=0x555e10161bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:46:31.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.887+0000 7f03c3fff640 1 --2- 172.21.15.107:0/806322746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e101549c0 0x555e10161bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.887+0000 7f03c3fff640 1 -- 172.21.15.107:0/806322746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e10167f50 msgr2=0x555e1015cf30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:46:31.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.887+0000 7f03c3fff640 1 --2- 172.21.15.107:0/806322746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e10167f50 0x555e1015cf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.887+0000 7f03c3fff640 1 -- 172.21.15.107:0/806322746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e0fef5f70 con 0x555e101560b0 2024-05-09T08:46:31.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.887+0000 7f03c3fff640 1 --2- 172.21.15.107:0/806322746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101560b0 0x555e101678f0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f03c4079cf0 tx=0x7f03b4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf4f9139a454914 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:31.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.887+0000 7f03a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:46:31.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.888+0000 7f03a37fe640 1 -- 172.21.15.107:0/806322746 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b4035040 con 0x555e101560b0 2024-05-09T08:46:31.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.888+0000 7f03cd168740 1 -- 172.21.15.107:0/806322746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03b80047a0 con 0x555e101560b0 2024-05-09T08:46:31.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.888+0000 7f03a37fe640 1 -- 172.21.15.107:0/806322746 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f03b4005220 con 0x555e101560b0 2024-05-09T08:46:31.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.889+0000 7f03a37fe640 1 -- 172.21.15.107:0/806322746 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f03b404a050 con 0x555e101560b0 2024-05-09T08:46:31.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.890+0000 7f03cd168740 10 client.19083 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:31.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.890+0000 7f03cd168740 1 -- 172.21.15.107:0/806322746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555e0fd8d170 con 0x555e101560b0 2024-05-09T08:46:31.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.891+0000 7f03a37fe640 1 -- 172.21.15.107:0/806322746 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f03b80047a0 con 0x555e101560b0 2024-05-09T08:46:31.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.891+0000 7f03a37fe640 1 -- 172.21.15.107:0/806322746 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f03b40038d0 con 0x555e101560b0 2024-05-09T08:46:31.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.892+0000 7f03cd168740 10 client.19083 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:31.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.892+0000 7f03cd168740 10 client.19083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:46:31.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.892+0000 7f03cd168740 20 client.19083 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:31.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.892+0000 7f03cd168740 1 --2- 172.21.15.107:0/806322746 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555e10180cd0 0x555e10183110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.892+0000 7f03cd168740 4 client.19083 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:46:31.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.892+0000 7f03cd168740 1 -- 172.21.15.107:0/806322746 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e100c70c0 con 0x555e10180cd0 2024-05-09T08:46:31.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.892+0000 7f03c37fe640 1 --2- 172.21.15.107:0/806322746 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555e10180cd0 0x555e10183110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:31.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.894+0000 7f03a37fe640 10 client.19083 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:31.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.894+0000 7f03c37fe640 1 --2- 172.21.15.107:0/806322746 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555e10180cd0 0x555e10183110 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ead0ce1b39d727cd server_cookie=f470cc8ce14b14a1 in_seq=0 out_seq=0 2024-05-09T08:46:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.895+0000 7f03a37fe640 1 -- 172.21.15.107:0/806322746 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555e100c70c0 con 0x555e10180cd0 2024-05-09T08:46:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.895+0000 7f03a37fe640 10 client.19083 handle_command_reply: tid=0 2024-05-09T08:46:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.895+0000 7f03cd168740 1 client.19083 shutdown 2024-05-09T08:46:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.896+0000 7f03cd168740 1 -- 172.21.15.107:0/806322746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101560b0 msgr2=0x555e101678f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.896+0000 7f03cd168740 1 --2- 172.21.15.107:0/806322746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101560b0 0x555e101678f0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f03c4079cf0 tx=0x7f03b4004270 comp rx=0 tx=0).stop 2024-05-09T08:46:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.896+0000 7f03cd168740 1 -- 172.21.15.107:0/806322746 shutdown_connections 2024-05-09T08:46:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.896+0000 7f03cd168740 1 --2- 172.21.15.107:0/806322746 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555e10180cd0 0x555e10183110 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.896+0000 7f03cd168740 1 --2- 172.21.15.107:0/806322746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e10167f50 0x555e1015cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.896+0000 7f03cd168740 1 --2- 172.21.15.107:0/806322746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101560b0 0x555e101678f0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.896+0000 7f03cd168740 1 --2- 172.21.15.107:0/806322746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e101549c0 0x555e10161bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.896+0000 7f03a37fe640 0 client.19083 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:31.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.896+0000 7f03cd168740 1 -- 172.21.15.107:0/806322746 >> 172.21.15.107:0/806322746 conn(0x555e1011d0a0 msgr2=0x555e10156d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:31.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.896+0000 7f03cd168740 1 -- 172.21.15.107:0/806322746 shutdown_connections 2024-05-09T08:46:31.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.896+0000 7f03cd168740 1 -- 172.21.15.107:0/806322746 wait complete. 2024-05-09T08:46:31.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.897+0000 7f03cd168740 20 client.19083 trim_cache size 0 max 16384 2024-05-09T08:46:31.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.930+0000 7f03cd168740 1 Processor -- start 2024-05-09T08:46:31.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.931+0000 7f03cd168740 1 -- start start 2024-05-09T08:46:31.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.931+0000 7f03cd168740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101549c0 0x555e101676a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.931+0000 7f03cd168740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e101560b0 0x555e10161780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.932+0000 7f03c37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101549c0 0x555e101676a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.932+0000 7f03c37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101549c0 0x555e101676a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33150/0 (socket says 172.21.15.107:33150) 2024-05-09T08:46:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.932+0000 7f03cd168740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e10167f50 0x555e10188a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.932+0000 7f03cd168740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f03c41b8200 con 0x555e101549c0 2024-05-09T08:46:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.932+0000 7f03cd168740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f03c41b5ad0 con 0x555e101560b0 2024-05-09T08:46:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.932+0000 7f03cd168740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f03c410c8f0 con 0x555e10167f50 2024-05-09T08:46:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.932+0000 7f03c37fe640 1 -- 172.21.15.107:0/3808139838 learned_addr learned my addr 172.21.15.107:0/3808139838 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.932+0000 7f03c37fe640 1 --2- 172.21.15.107:0/3808139838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e10167f50 0x555e10188a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.932+0000 7f03c37fe640 1 -- 172.21.15.107:0/3808139838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e10167f50 msgr2=0x555e10188a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.932+0000 7f03c37fe640 1 --2- 172.21.15.107:0/3808139838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e10167f50 0x555e10188a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.932+0000 7f03c37fe640 1 -- 172.21.15.107:0/3808139838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e101560b0 msgr2=0x555e10161780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.932+0000 7f03c37fe640 1 --2- 172.21.15.107:0/3808139838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e101560b0 0x555e10161780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.932+0000 7f03c37fe640 1 -- 172.21.15.107:0/3808139838 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e10155cb0 con 0x555e101549c0 2024-05-09T08:46:31.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.932+0000 7f03c37fe640 1 --2- 172.21.15.107:0/3808139838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101549c0 0x555e101676a0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f03b8001eb0 tx=0x7f03b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d5f69cab0d021b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:31.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.933+0000 7f03a9ffb640 1 -- 172.21.15.107:0/3808139838 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b800fea0 con 0x555e101549c0 2024-05-09T08:46:31.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.933+0000 7f03a9ffb640 1 -- 172.21.15.107:0/3808139838 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f03b80044f0 con 0x555e101549c0 2024-05-09T08:46:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.933+0000 7f03a9ffb640 1 -- 172.21.15.107:0/3808139838 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b8004c60 con 0x555e101549c0 2024-05-09T08:46:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.934+0000 7f03cd168740 1 -- 172.21.15.107:0/3808139838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101549c0 msgr2=0x555e101676a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.934+0000 7f03cd168740 1 --2- 172.21.15.107:0/3808139838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101549c0 0x555e101676a0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f03b8001eb0 tx=0x7f03b8004730 comp rx=0 tx=0).stop 2024-05-09T08:46:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.934+0000 7f03cd168740 1 -- 172.21.15.107:0/3808139838 shutdown_connections 2024-05-09T08:46:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.934+0000 7f03cd168740 1 --2- 172.21.15.107:0/3808139838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e10167f50 0x555e10188a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.934+0000 7f03cd168740 1 --2- 172.21.15.107:0/3808139838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e101560b0 0x555e10161780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.934+0000 7f03cd168740 1 --2- 172.21.15.107:0/3808139838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101549c0 0x555e101676a0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.934+0000 7f03cd168740 1 -- 172.21.15.107:0/3808139838 >> 172.21.15.107:0/3808139838 conn(0x555e1011d0a0 msgr2=0x555e1015d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.934+0000 7f03cd168740 1 -- 172.21.15.107:0/3808139838 shutdown_connections 2024-05-09T08:46:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.934+0000 7f03cd168740 1 -- 172.21.15.107:0/3808139838 wait complete. 2024-05-09T08:46:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.935+0000 7f03cd168740 1 -- start start 2024-05-09T08:46:31.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.935+0000 7f03cd168740 1 Processor -- start 2024-05-09T08:46:31.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.936+0000 7f03cd168740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e101549c0 0x555e10194010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.936+0000 7f03cd168740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101560b0 0x555e10194550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.936+0000 7f03c37fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e101549c0 0x555e10194010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:31.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.936+0000 7f03c37fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e101549c0 0x555e10194010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:55084/0 (socket says 172.21.15.107:55084) 2024-05-09T08:46:31.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.936+0000 7f03cd168740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e10167f50 0x555e10191140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.936+0000 7f03cd168740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f03c41b3f90 con 0x555e101560b0 2024-05-09T08:46:31.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.936+0000 7f03cd168740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f03c41b5400 con 0x555e10167f50 2024-05-09T08:46:31.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.936+0000 7f03cd168740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f03c41b4660 con 0x555e101549c0 2024-05-09T08:46:31.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.936+0000 7f03cd168740 10 client.0 _ll_register_callbacks cb 0x7f03bc0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:31.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.936+0000 7f03c37fe640 1 -- 172.21.15.107:0/2813302424 learned_addr learned my addr 172.21.15.107:0/2813302424 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:31.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.937+0000 7f03c37fe640 1 --2- 172.21.15.107:0/2813302424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e10167f50 0x555e10191140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:31.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.937+0000 7f03c37fe640 1 -- 172.21.15.107:0/2813302424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e10167f50 msgr2=0x555e10191140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:31.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.937+0000 7f03c37fe640 1 --2- 172.21.15.107:0/2813302424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e10167f50 0x555e10191140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.937+0000 7f03c37fe640 1 -- 172.21.15.107:0/2813302424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101560b0 msgr2=0x555e10194550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.937+0000 7f03c37fe640 1 --2- 172.21.15.107:0/2813302424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101560b0 0x555e10194550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.937+0000 7f03c37fe640 1 -- 172.21.15.107:0/2813302424 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e10191970 con 0x555e101549c0 2024-05-09T08:46:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.937+0000 7f03c37fe640 1 --2- 172.21.15.107:0/2813302424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e101549c0 0x555e10194010 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f03b8010010 tx=0x7f03b8004840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec8000285d1889ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.938+0000 7f03a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:46:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.938+0000 7f03a37fe640 1 -- 172.21.15.107:0/2813302424 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b8009d50 con 0x555e101549c0 2024-05-09T08:46:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.938+0000 7f03a37fe640 1 -- 172.21.15.107:0/2813302424 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f03b8004040 con 0x555e101549c0 2024-05-09T08:46:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.938+0000 7f03cd168740 1 -- 172.21.15.107:0/2813302424 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03b80044f0 con 0x555e101549c0 2024-05-09T08:46:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.938+0000 7f03a37fe640 1 -- 172.21.15.107:0/2813302424 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f03b803a050 con 0x555e101549c0 2024-05-09T08:46:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.938+0000 7f03cd168740 10 client.28234 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.938+0000 7f03cd168740 1 -- 172.21.15.107:0/2813302424 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555e101e79c0 con 0x555e101549c0 2024-05-09T08:46:31.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.938+0000 7f03a37fe640 1 -- 172.21.15.107:0/2813302424 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f03b8004240 con 0x555e101549c0 2024-05-09T08:46:31.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.939+0000 7f03a37fe640 1 -- 172.21.15.107:0/2813302424 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f03b801d030 con 0x555e101549c0 2024-05-09T08:46:31.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.939+0000 7f03cd168740 10 client.28234 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:31.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.939+0000 7f03cd168740 10 client.28234 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:46:31.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.939+0000 7f03cd168740 20 client.28234 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:31.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.939+0000 7f03cd168740 1 --2- 172.21.15.107:0/2813302424 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555e101e7da0 0x555e101ea190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:31.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.939+0000 7f03cd168740 4 client.28234 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:46:31.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.939+0000 7f03cd168740 1 -- 172.21.15.107:0/2813302424 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f03b80044f0 con 0x555e101e7da0 2024-05-09T08:46:31.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.940+0000 7f03c3fff640 1 --2- 172.21.15.107:0/2813302424 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555e101e7da0 0x555e101ea190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:31.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.941+0000 7f03c3fff640 1 --2- 172.21.15.107:0/2813302424 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555e101e7da0 0x555e101ea190 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea16e476e18ab1a7 server_cookie=d7c9fa2cbb6ac5d8 in_seq=0 out_seq=0 2024-05-09T08:46:31.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.941+0000 7f03a37fe640 10 client.28234 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:31.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.943+0000 7f03a37fe640 1 -- 172.21.15.107:0/2813302424 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f03b80044f0 con 0x555e101e7da0 2024-05-09T08:46:31.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.943+0000 7f03a37fe640 10 client.28234 handle_command_reply: tid=0 2024-05-09T08:46:31.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.943+0000 7f03cd168740 1 client.28234 shutdown 2024-05-09T08:46:31.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.943+0000 7f03cd168740 1 -- 172.21.15.107:0/2813302424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e101549c0 msgr2=0x555e10194010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:31.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.943+0000 7f03cd168740 1 --2- 172.21.15.107:0/2813302424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e101549c0 0x555e10194010 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f03b8010010 tx=0x7f03b8004840 comp rx=0 tx=0).stop 2024-05-09T08:46:31.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.943+0000 7f03cd168740 1 -- 172.21.15.107:0/2813302424 shutdown_connections 2024-05-09T08:46:31.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.943+0000 7f03cd168740 1 --2- 172.21.15.107:0/2813302424 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x555e101e7da0 0x555e101ea190 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.943+0000 7f03cd168740 1 --2- 172.21.15.107:0/2813302424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e10167f50 0x555e10191140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.944+0000 7f03cd168740 1 --2- 172.21.15.107:0/2813302424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e101560b0 0x555e10194550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.944+0000 7f03cd168740 1 --2- 172.21.15.107:0/2813302424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e101549c0 0x555e10194010 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.944+0000 7f03cd168740 1 -- 172.21.15.107:0/2813302424 >> 172.21.15.107:0/2813302424 conn(0x555e1011d0a0 msgr2=0x555e1015c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:31.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.944+0000 7f03a37fe640 0 client.28234 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:31.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.945+0000 7f03cd168740 1 -- 172.21.15.107:0/2813302424 shutdown_connections 2024-05-09T08:46:31.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.945+0000 7f03cd168740 1 -- 172.21.15.107:0/2813302424 wait complete. 2024-05-09T08:46:31.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.946+0000 7f03cd168740 20 client.28234 trim_cache size 0 max 16384 2024-05-09T08:46:31.946 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:46:31.946 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:46:31.946 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "0b0c6d85-4670-442d-96d1-ff7f2c7953f3", 2024-05-09T08:46:31.946 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:46:31.947 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:46:31.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.947+0000 7f03ca461640 1 -- 172.21.15.107:0/3817473402 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f03ac0a2ed0 msgr2=0x7f03ac0a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:31.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.947+0000 7f03ca461640 1 --2- 172.21.15.107:0/3817473402 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f03ac0a2ed0 0x7f03ac0a5390 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f03bc09b9e0 tx=0x7f03bc00ef50 comp rx=0 tx=0).stop 2024-05-09T08:46:31.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.947+0000 7f03ca461640 1 -- 172.21.15.107:0/3817473402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c419eda0 msgr2=0x7f03c419f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:31.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.947+0000 7f03ca461640 1 --2- 172.21.15.107:0/3817473402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c419eda0 0x7f03c419f250 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00ec50 tx=0x7f03bc00c6a0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.948+0000 7f03ca461640 1 -- 172.21.15.107:0/3817473402 shutdown_connections 2024-05-09T08:46:31.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.948+0000 7f03ca461640 1 --2- 172.21.15.107:0/3817473402 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f03ac0a2ed0 0x7f03ac0a5390 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.948+0000 7f03ca461640 1 --2- 172.21.15.107:0/3817473402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c419eda0 0x7f03c419f250 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.948+0000 7f03ca461640 1 --2- 172.21.15.107:0/3817473402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03c41021c0 0x7f03c419aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.948+0000 7f03ca461640 1 --2- 172.21.15.107:0/3817473402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03c4101800 0x7f03c419a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:31.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.948+0000 7f03ca461640 1 -- 172.21.15.107:0/3817473402 >> 172.21.15.107:0/3817473402 conn(0x7f03c4078c80 msgr2=0x7f03c410cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:31.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.948+0000 7f03ca461640 1 -- 172.21.15.107:0/3817473402 shutdown_connections 2024-05-09T08:46:31.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:31.948+0000 7f03ca461640 1 -- 172.21.15.107:0/3817473402 wait complete. 2024-05-09T08:46:31.966 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:46:31.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:46:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.178+0000 7f5a48915640 1 -- 172.21.15.107:0/2853275856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a3c0aa290 msgr2=0x7f5a3c0b1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.178+0000 7f5a48915640 1 --2- 172.21.15.107:0/2853275856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a3c0aa290 0x7f5a3c0b1e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5a380074e0 tx=0x7f5a3802b060 comp rx=0 tx=0).stop 2024-05-09T08:46:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.178+0000 7f5a48915640 1 -- 172.21.15.107:0/2853275856 shutdown_connections 2024-05-09T08:46:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.178+0000 7f5a48915640 1 --2- 172.21.15.107:0/2853275856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a3c0aa290 0x7f5a3c0b1e60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.178+0000 7f5a48915640 1 --2- 172.21.15.107:0/2853275856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a3c0a5b70 0x7f5a3c0a9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.178+0000 7f5a48915640 1 --2- 172.21.15.107:0/2853275856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a3c0a51c0 0x7f5a3c0a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.178+0000 7f5a48915640 1 -- 172.21.15.107:0/2853275856 >> 172.21.15.107:0/2853275856 conn(0x7f5a3c01aa60 msgr2=0x7f5a3c01ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:32.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.179+0000 7f5a48915640 1 -- 172.21.15.107:0/2853275856 shutdown_connections 2024-05-09T08:46:32.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.179+0000 7f5a48915640 1 -- 172.21.15.107:0/2853275856 wait complete. 2024-05-09T08:46:32.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.180+0000 7f5a48915640 1 Processor -- start 2024-05-09T08:46:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.180+0000 7f5a48915640 1 -- start start 2024-05-09T08:46:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.181+0000 7f5a48915640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a3c0a51c0 0x7f5a3c149810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.181+0000 7f5a48915640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a3c0aa290 0x7f5a3c149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.181+0000 7f5a48915640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a3c1439c0 0x7f5a3c143e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.181+0000 7f5a48915640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c0b4e70 con 0x7f5a3c0aa290 2024-05-09T08:46:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.181+0000 7f5a48915640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c0b4cf0 con 0x7f5a3c0a51c0 2024-05-09T08:46:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.181+0000 7f5a48915640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c0b4ff0 con 0x7f5a3c1439c0 2024-05-09T08:46:32.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.181+0000 7f5a4359e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a3c1439c0 0x7f5a3c143e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.181+0000 7f5a4359e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a3c1439c0 0x7f5a3c143e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:55110/0 (socket says 172.21.15.107:55110) 2024-05-09T08:46:32.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.181+0000 7f5a4359e640 1 -- 172.21.15.107:0/847191051 learned_addr learned my addr 172.21.15.107:0/847191051 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:32.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.181+0000 7f5a42d9d640 1 --2- 172.21.15.107:0/847191051 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a3c0a51c0 0x7f5a3c149810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.182+0000 7f5a4359e640 1 -- 172.21.15.107:0/847191051 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a3c0a51c0 msgr2=0x7f5a3c149810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:32.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.182+0000 7f5a4359e640 1 --2- 172.21.15.107:0/847191051 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a3c0a51c0 0x7f5a3c149810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.182+0000 7f5a4359e640 1 -- 172.21.15.107:0/847191051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a3c0aa290 msgr2=0x7f5a3c149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:32.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.182+0000 7f5a4359e640 1 --2- 172.21.15.107:0/847191051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a3c0aa290 0x7f5a3c149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.182+0000 7f5a4359e640 1 -- 172.21.15.107:0/847191051 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a3c1444e0 con 0x7f5a3c1439c0 2024-05-09T08:46:32.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.183+0000 7f5a4359e640 1 --2- 172.21.15.107:0/847191051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a3c1439c0 0x7f5a3c143e70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5a3802b5d0 tx=0x7f5a3802bbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:32.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.184+0000 7f5a2bfff640 1 -- 172.21.15.107:0/847191051 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a38041070 con 0x7f5a3c1439c0 2024-05-09T08:46:32.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.184+0000 7f5a2bfff640 1 -- 172.21.15.107:0/847191051 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a38004f40 con 0x7f5a3c1439c0 2024-05-09T08:46:32.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.184+0000 7f5a2bfff640 1 -- 172.21.15.107:0/847191051 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a38003850 con 0x7f5a3c1439c0 2024-05-09T08:46:32.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.189+0000 7f5a48915640 1 -- 172.21.15.107:0/847191051 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a3c144770 con 0x7f5a3c1439c0 2024-05-09T08:46:32.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.189+0000 7f5a48915640 1 -- 172.21.15.107:0/847191051 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a3c13a5e0 con 0x7f5a3c1439c0 2024-05-09T08:46:32.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.195+0000 7f5a4be1a740 1 Processor -- start 2024-05-09T08:46:32.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.200+0000 7f5a4be1a740 1 -- start start 2024-05-09T08:46:32.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.200+0000 7f5a4be1a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e46430 0x55dc70e46810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.200+0000 7f5a4be1a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e46d50 0x55dc70e50ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.200+0000 7f5a4be1a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e51420 0x55dc70e53810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.200+0000 7f5a4be1a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55dc70dabd00 con 0x55dc70e46430 2024-05-09T08:46:32.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.200+0000 7f5a4be1a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55dc70d534c0 con 0x55dc70e46d50 2024-05-09T08:46:32.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.200+0000 7f5a4be1a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55dc70da7cc0 con 0x55dc70e51420 2024-05-09T08:46:32.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.202+0000 7f5a2bfff640 1 -- 172.21.15.107:0/847191051 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f5a380050f0 con 0x7f5a3c1439c0 2024-05-09T08:46:32.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.203+0000 7f5a2bfff640 1 --2- 172.21.15.107:0/847191051 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5a2c0a2f00 0x7f5a2c0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.204+0000 7f5a2bfff640 1 -- 172.21.15.107:0/847191051 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f5a380eea10 con 0x7f5a3c1439c0 2024-05-09T08:46:32.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.204+0000 7f5a42d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e46d50 0x55dc70e50ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.204+0000 7f5a42d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e46d50 0x55dc70e50ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:36288/0 (socket says 172.21.15.107:36288) 2024-05-09T08:46:32.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.204+0000 7f5a42d9d640 1 -- 172.21.15.107:0/3487726894 learned_addr learned my addr 172.21.15.107:0/3487726894 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:32.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.204+0000 7f5a4259c640 1 --2- 172.21.15.107:0/3487726894 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e51420 0x55dc70e53810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.204+0000 7f5a4259c640 1 --2- 172.21.15.107:0/3487726894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e46430 0x55dc70e46810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.208+0000 7f5a4359e640 1 --2- 172.21.15.107:0/847191051 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5a2c0a2f00 0x7f5a2c0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.208+0000 7f5a4259c640 1 -- 172.21.15.107:0/3487726894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e46d50 msgr2=0x55dc70e50ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:32.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.208+0000 7f5a4259c640 1 --2- 172.21.15.107:0/3487726894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e46d50 0x55dc70e50ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.208+0000 7f5a4259c640 1 -- 172.21.15.107:0/3487726894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e46430 msgr2=0x55dc70e46810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:32.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.208+0000 7f5a4259c640 1 --2- 172.21.15.107:0/3487726894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e46430 0x55dc70e46810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.208+0000 7f5a4259c640 1 -- 172.21.15.107:0/3487726894 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc70b897d0 con 0x55dc70e51420 2024-05-09T08:46:32.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.209+0000 7f5a4259c640 1 --2- 172.21.15.107:0/3487726894 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e51420 0x55dc70e53810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0a9bf0 tx=0x7f5a34005c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7197eb9b93b3ce5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:32.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.209+0000 7f5a297fa640 1 -- 172.21.15.107:0/3487726894 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a34011660 con 0x55dc70e51420 2024-05-09T08:46:32.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.209+0000 7f5a297fa640 1 -- 172.21.15.107:0/3487726894 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a34017030 con 0x55dc70e51420 2024-05-09T08:46:32.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.209+0000 7f5a297fa640 1 -- 172.21.15.107:0/3487726894 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a34010970 con 0x55dc70e51420 2024-05-09T08:46:32.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.210+0000 7f5a4be1a740 1 -- 172.21.15.107:0/3487726894 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e51420 msgr2=0x55dc70e53810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:32.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.210+0000 7f5a4359e640 1 --2- 172.21.15.107:0/847191051 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5a2c0a2f00 0x7f5a2c0a53c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5a380ebe10 tx=0x7f5a38005b70 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:32.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.210+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/3487726894 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e51420 0x55dc70e53810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0a9bf0 tx=0x7f5a34005c10 comp rx=0 tx=0).stop 2024-05-09T08:46:32.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.210+0000 7f5a4be1a740 1 -- 172.21.15.107:0/3487726894 shutdown_connections 2024-05-09T08:46:32.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.210+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/3487726894 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e51420 0x55dc70e53810 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.210+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/3487726894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e46d50 0x55dc70e50ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.210+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/3487726894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e46430 0x55dc70e46810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.210+0000 7f5a4be1a740 1 -- 172.21.15.107:0/3487726894 >> 172.21.15.107:0/3487726894 conn(0x55dc70e42750 msgr2=0x55dc70e42b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:32.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.210+0000 7f5a4be1a740 1 -- 172.21.15.107:0/3487726894 shutdown_connections 2024-05-09T08:46:32.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.211+0000 7f5a4be1a740 1 -- 172.21.15.107:0/3487726894 wait complete. 2024-05-09T08:46:32.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.211+0000 7f5a4be1a740 1 -- start start 2024-05-09T08:46:32.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.211+0000 7f5a4be1a740 1 Processor -- start 2024-05-09T08:46:32.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.211+0000 7f5a4be1a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e46430 0x55dc70e4e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.212+0000 7f5a4be1a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e46d50 0x55dc70e4e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.212+0000 7f5a4be1a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e51420 0x55dc70e4ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.212+0000 7f5a4be1a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55dc70e542b0 con 0x55dc70e46d50 2024-05-09T08:46:32.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.212+0000 7f5a4be1a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55dc70d66b90 con 0x55dc70e46430 2024-05-09T08:46:32.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.212+0000 7f5a4be1a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55dc70e54430 con 0x55dc70e51420 2024-05-09T08:46:32.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.212+0000 7f5a4be1a740 10 client.0 _ll_register_callbacks cb 0x55dc70da2da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:32.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.212+0000 7f5a4259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e46430 0x55dc70e4e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.212+0000 7f5a4259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e51420 0x55dc70e4ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.212+0000 7f5a4259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e46430 0x55dc70e4e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:36290/0 (socket says 172.21.15.107:36290) 2024-05-09T08:46:32.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.212+0000 7f5a4259c640 1 -- 172.21.15.107:0/911198624 learned_addr learned my addr 172.21.15.107:0/911198624 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:32.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.212+0000 7f5a42d9d640 1 --2- 172.21.15.107:0/911198624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e46d50 0x55dc70e4e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.213+0000 7f5a4259c640 1 -- 172.21.15.107:0/911198624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e51420 msgr2=0x55dc70e4ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:32.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.213+0000 7f5a4259c640 1 --2- 172.21.15.107:0/911198624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e51420 0x55dc70e4ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.213+0000 7f5a4259c640 1 -- 172.21.15.107:0/911198624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e46d50 msgr2=0x55dc70e4e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:32.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.213+0000 7f5a4259c640 1 --2- 172.21.15.107:0/911198624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e46d50 0x55dc70e4e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.213+0000 7f5a4259c640 1 -- 172.21.15.107:0/911198624 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc70abf3f0 con 0x55dc70e46430 2024-05-09T08:46:32.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.213+0000 7f5a42d9d640 1 --2- 172.21.15.107:0/911198624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e46d50 0x55dc70e4e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:46:32.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.213+0000 7f5a26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:46:32.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.213+0000 7f5a4259c640 1 --2- 172.21.15.107:0/911198624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e46430 0x55dc70e4e2c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a34004700 tx=0x7f5a34015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c55c07f2a4aebe87 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:32.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.214+0000 7f5a26ff5640 1 -- 172.21.15.107:0/911198624 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a34026040 con 0x55dc70e46430 2024-05-09T08:46:32.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.214+0000 7f5a26ff5640 1 -- 172.21.15.107:0/911198624 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f5a34004220 con 0x55dc70e46430 2024-05-09T08:46:32.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.214+0000 7f5a4be1a740 1 -- 172.21.15.107:0/911198624 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a34017030 con 0x55dc70e46430 2024-05-09T08:46:32.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.214+0000 7f5a26ff5640 1 -- 172.21.15.107:0/911198624 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5a34004420 con 0x55dc70e46430 2024-05-09T08:46:32.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.216+0000 7f5a4be1a740 10 client.28034 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:32.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.216+0000 7f5a4be1a740 1 -- 172.21.15.107:0/911198624 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dc70a7f0f0 con 0x55dc70e46430 2024-05-09T08:46:32.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.216+0000 7f5a26ff5640 1 -- 172.21.15.107:0/911198624 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a34017030 con 0x55dc70e46430 2024-05-09T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.216+0000 7f5a26ff5640 1 -- 172.21.15.107:0/911198624 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5a3400f040 con 0x55dc70e46430 2024-05-09T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.216+0000 7f5a4be1a740 10 client.28034 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.216+0000 7f5a4be1a740 10 client.28034 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.217+0000 7f5a4be1a740 20 client.28034 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.217+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/911198624 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dc70e61730 0x55dc70e63b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.217+0000 7f5a4be1a740 4 client.28034 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.217+0000 7f5a4be1a740 1 -- 172.21.15.107:0/911198624 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc70db9430 con 0x55dc70e61730 2024-05-09T08:46:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.217+0000 7f5a42d9d640 1 --2- 172.21.15.107:0/911198624 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dc70e61730 0x55dc70e63b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.219+0000 7f5a42d9d640 1 --2- 172.21.15.107:0/911198624 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dc70e61730 0x55dc70e63b20 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=803d735303c91e14 server_cookie=1b19ce3109f90a9d in_seq=0 out_seq=0 2024-05-09T08:46:32.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.220+0000 7f5a26ff5640 10 client.28034 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.220+0000 7f5a26ff5640 1 -- 172.21.15.107:0/911198624 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dc70db9430 con 0x55dc70e61730 2024-05-09T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.220+0000 7f5a26ff5640 10 client.28034 handle_command_reply: tid=0 2024-05-09T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.220+0000 7f5a4be1a740 1 client.28034 shutdown 2024-05-09T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.220+0000 7f5a4be1a740 1 -- 172.21.15.107:0/911198624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e46430 msgr2=0x55dc70e4e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.220+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/911198624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e46430 0x55dc70e4e2c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a34004700 tx=0x7f5a34015040 comp rx=0 tx=0).stop 2024-05-09T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.221+0000 7f5a4be1a740 1 -- 172.21.15.107:0/911198624 shutdown_connections 2024-05-09T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.221+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/911198624 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dc70e61730 0x55dc70e63b20 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.221+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/911198624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e51420 0x55dc70e4ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.221+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/911198624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e46d50 0x55dc70e4e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.221+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/911198624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e46430 0x55dc70e4e2c0 secure :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a34004700 tx=0x7f5a34015040 comp rx=0 tx=0).stop 2024-05-09T08:46:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.221+0000 7f5a4be1a740 1 -- 172.21.15.107:0/911198624 >> 172.21.15.107:0/911198624 conn(0x55dc70e42750 msgr2=0x55dc70e41a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.221+0000 7f5a26ff5640 0 client.28034 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.221+0000 7f5a4be1a740 1 -- 172.21.15.107:0/911198624 shutdown_connections 2024-05-09T08:46:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.221+0000 7f5a4be1a740 1 -- 172.21.15.107:0/911198624 wait complete. 2024-05-09T08:46:32.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.223+0000 7f5a4be1a740 20 client.28034 trim_cache size 0 max 16384 2024-05-09T08:46:32.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.259+0000 7f5a4be1a740 1 Processor -- start 2024-05-09T08:46:32.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.259+0000 7f5a4be1a740 1 -- start start 2024-05-09T08:46:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.259+0000 7f5a4be1a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e46d50 0x55dc70e5d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.260+0000 7f5a4be1a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e51420 0x55dc70e42b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.260+0000 7f5a4be1a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e50b00 0x55dc70e692e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.260+0000 7f5a4be1a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c165970 con 0x55dc70e50b00 2024-05-09T08:46:32.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.260+0000 7f5a4259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e46d50 0x55dc70e5d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.260+0000 7f5a4be1a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c163240 con 0x55dc70e51420 2024-05-09T08:46:32.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.260+0000 7f5a4259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e46d50 0x55dc70e5d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:55134/0 (socket says 172.21.15.107:55134) 2024-05-09T08:46:32.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.260+0000 7f5a4259c640 1 -- 172.21.15.107:0/200068871 learned_addr learned my addr 172.21.15.107:0/200068871 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:32.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.260+0000 7f5a4be1a740 1 -- 172.21.15.107:0/200068871 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c144ef0 con 0x55dc70e46d50 2024-05-09T08:46:32.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.260+0000 7f5a42d9d640 1 --2- 172.21.15.107:0/200068871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e51420 0x55dc70e42b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.260+0000 7f5a4259c640 1 --2- 172.21.15.107:0/200068871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e50b00 0x55dc70e692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.265+0000 7f5a4259c640 1 -- 172.21.15.107:0/200068871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e51420 msgr2=0x55dc70e42b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:32.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.265+0000 7f5a4259c640 1 --2- 172.21.15.107:0/200068871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e51420 0x55dc70e42b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.265+0000 7f5a4259c640 1 -- 172.21.15.107:0/200068871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e50b00 msgr2=0x55dc70e692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:32.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.265+0000 7f5a4259c640 1 --2- 172.21.15.107:0/200068871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e50b00 0x55dc70e692e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.265+0000 7f5a4259c640 1 -- 172.21.15.107:0/200068871 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc70e46b80 con 0x55dc70e46d50 2024-05-09T08:46:32.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.268+0000 7f5a4259c640 1 --2- 172.21.15.107:0/200068871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e46d50 0x55dc70e5d200 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400e470 tx=0x7f5a34005c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbb2afb831960926 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:32.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.268+0000 7f5a28ff9640 1 -- 172.21.15.107:0/200068871 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a34004420 con 0x55dc70e46d50 2024-05-09T08:46:32.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.269+0000 7f5a28ff9640 1 -- 172.21.15.107:0/200068871 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a34017070 con 0x55dc70e46d50 2024-05-09T08:46:32.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.269+0000 7f5a28ff9640 1 -- 172.21.15.107:0/200068871 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a34004a90 con 0x55dc70e46d50 2024-05-09T08:46:32.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.270+0000 7f5a4be1a740 1 -- 172.21.15.107:0/200068871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e46d50 msgr2=0x55dc70e5d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:32.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.270+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/200068871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e46d50 0x55dc70e5d200 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400e470 tx=0x7f5a34005c10 comp rx=0 tx=0).stop 2024-05-09T08:46:32.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.270+0000 7f5a4be1a740 1 -- 172.21.15.107:0/200068871 shutdown_connections 2024-05-09T08:46:32.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.270+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/200068871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e50b00 0x55dc70e692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.270+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/200068871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e51420 0x55dc70e42b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.270+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/200068871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e46d50 0x55dc70e5d200 secure :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400e470 tx=0x7f5a34005c10 comp rx=0 tx=0).stop 2024-05-09T08:46:32.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.270+0000 7f5a4be1a740 1 -- 172.21.15.107:0/200068871 >> 172.21.15.107:0/200068871 conn(0x55dc70e42750 msgr2=0x55dc70e4d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:32.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.272+0000 7f5a4be1a740 1 -- 172.21.15.107:0/200068871 shutdown_connections 2024-05-09T08:46:32.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.272+0000 7f5a4be1a740 1 -- 172.21.15.107:0/200068871 wait complete. 2024-05-09T08:46:32.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.272+0000 7f5a4be1a740 1 -- start start 2024-05-09T08:46:32.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.273+0000 7f5a4be1a740 1 Processor -- start 2024-05-09T08:46:32.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.273+0000 7f5a4be1a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e50b00 0x55dc70e59a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.273+0000 7f5a4be1a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e51420 0x55dc70e54d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.273+0000 7f5a4be1a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e55290 0x55dc70e55710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.273+0000 7f5a4be1a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c161730 con 0x55dc70e55290 2024-05-09T08:46:32.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.273+0000 7f5a4be1a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c162b70 con 0x55dc70e51420 2024-05-09T08:46:32.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.273+0000 7f5a4be1a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c161dd0 con 0x55dc70e50b00 2024-05-09T08:46:32.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.273+0000 7f5a4be1a740 10 client.0 _ll_register_callbacks cb 0x7f5a4404fe40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:32.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.273+0000 7f5a4259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e50b00 0x55dc70e59a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.273+0000 7f5a4259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e50b00 0x55dc70e59a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:55136/0 (socket says 172.21.15.107:55136) 2024-05-09T08:46:32.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.273+0000 7f5a4259c640 1 -- 172.21.15.107:0/428465093 learned_addr learned my addr 172.21.15.107:0/428465093 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:32.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.273+0000 7f5a42d9d640 1 --2- 172.21.15.107:0/428465093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e51420 0x55dc70e54d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.273+0000 7f5a4259c640 1 --2- 172.21.15.107:0/428465093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e55290 0x55dc70e55710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.274+0000 7f5a42d9d640 1 -- 172.21.15.107:0/428465093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e50b00 msgr2=0x55dc70e59a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:32.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.274+0000 7f5a42d9d640 1 --2- 172.21.15.107:0/428465093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e50b00 0x55dc70e59a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.274+0000 7f5a42d9d640 1 -- 172.21.15.107:0/428465093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e55290 msgr2=0x55dc70e55710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:32.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.274+0000 7f5a42d9d640 1 --2- 172.21.15.107:0/428465093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e55290 0x55dc70e55710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.274+0000 7f5a42d9d640 1 -- 172.21.15.107:0/428465093 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc70ec6ba0 con 0x55dc70e51420 2024-05-09T08:46:32.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.275+0000 7f5a42d9d640 1 --2- 172.21.15.107:0/428465093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e51420 0x55dc70e54d50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5a44067670 tx=0x7f5a440807c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a1500f2298f6e8a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:32.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.275+0000 7f5a26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:46:32.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.275+0000 7f5a26ff5640 1 -- 172.21.15.107:0/428465093 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4408b640 con 0x55dc70e51420 2024-05-09T08:46:32.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.275+0000 7f5a26ff5640 1 -- 172.21.15.107:0/428465093 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f5a44073420 con 0x55dc70e51420 2024-05-09T08:46:32.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.276+0000 7f5a4be1a740 1 -- 172.21.15.107:0/428465093 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a34017070 con 0x55dc70e51420 2024-05-09T08:46:32.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.279+0000 7f5a26ff5640 1 -- 172.21.15.107:0/428465093 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5a4408b7e0 con 0x55dc70e51420 2024-05-09T08:46:32.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:32 smithi107 ceph-mon[30122]: pgmap v412: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 11 MiB/s wr, 2.73k op/s 2024-05-09T08:46:32.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:32 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:46:32.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:32 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:32.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:32 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:46:32.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.281+0000 7f5a4be1a740 10 client.28040 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:32.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.281+0000 7f5a4be1a740 1 -- 172.21.15.107:0/428465093 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dc70ec7ef0 con 0x55dc70e51420 2024-05-09T08:46:32.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.283+0000 7f5a26ff5640 1 -- 172.21.15.107:0/428465093 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a34017070 con 0x55dc70e51420 2024-05-09T08:46:32.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.283+0000 7f5a26ff5640 1 -- 172.21.15.107:0/428465093 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f5a44089070 con 0x55dc70e51420 2024-05-09T08:46:32.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.283+0000 7f5a4be1a740 10 client.28040 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:32.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.283+0000 7f5a4be1a740 10 client.28040 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:46:32.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.283+0000 7f5a4be1a740 20 client.28040 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:32.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.283+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/428465093 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dc70ec85f0 0x55dc70ecaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:32.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.283+0000 7f5a4be1a740 4 client.28040 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:46:32.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.283+0000 7f5a4be1a740 1 -- 172.21.15.107:0/428465093 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dc70db3780 con 0x55dc70ec85f0 2024-05-09T08:46:32.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.283+0000 7f5a4259c640 1 --2- 172.21.15.107:0/428465093 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dc70ec85f0 0x55dc70ecaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:32.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.284+0000 7f5a4259c640 1 --2- 172.21.15.107:0/428465093 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dc70ec85f0 0x55dc70ecaa10 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=634bfed99762a578 server_cookie=5945c815839038e2 in_seq=0 out_seq=0 2024-05-09T08:46:32.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.286+0000 7f5a26ff5640 10 client.28040 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:32.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.286+0000 7f5a26ff5640 1 -- 172.21.15.107:0/428465093 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55dc70db3780 con 0x55dc70ec85f0 2024-05-09T08:46:32.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.286+0000 7f5a26ff5640 10 client.28040 handle_command_reply: tid=0 2024-05-09T08:46:32.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.287+0000 7f5a4be1a740 1 client.28040 shutdown 2024-05-09T08:46:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.288+0000 7f5a4be1a740 1 -- 172.21.15.107:0/428465093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e51420 msgr2=0x55dc70e54d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:32.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.288+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/428465093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e51420 0x55dc70e54d50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5a44067670 tx=0x7f5a440807c0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.295+0000 7f5a4be1a740 1 -- 172.21.15.107:0/428465093 shutdown_connections 2024-05-09T08:46:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.295+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/428465093 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dc70ec85f0 0x55dc70ecaa10 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.295+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/428465093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc70e55290 0x55dc70e55710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.295+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/428465093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dc70e51420 0x55dc70e54d50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.295+0000 7f5a4be1a740 1 --2- 172.21.15.107:0/428465093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dc70e50b00 0x55dc70e59a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.295+0000 7f5a4be1a740 1 -- 172.21.15.107:0/428465093 >> 172.21.15.107:0/428465093 conn(0x55dc70e42750 msgr2=0x55dc70e4e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:32.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.296+0000 7f5a26ff5640 0 client.28040 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:32.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.296+0000 7f5a4be1a740 1 -- 172.21.15.107:0/428465093 shutdown_connections 2024-05-09T08:46:32.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.296+0000 7f5a4be1a740 1 -- 172.21.15.107:0/428465093 wait complete. 2024-05-09T08:46:32.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.296+0000 7f5a4be1a740 20 client.28040 trim_cache size 0 max 16384 2024-05-09T08:46:32.299 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:46:32.300 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "scrub active (148 inodes in the stack)", 2024-05-09T08:46:32.300 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:46:32.300 INFO:teuthology.orchestra.run.smithi107.stdout: "0b0c6d85-4670-442d-96d1-ff7f2c7953f3": { 2024-05-09T08:46:32.300 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:46:32.300 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "0b0c6d85-4670-442d-96d1-ff7f2c7953f3", 2024-05-09T08:46:32.300 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:46:32.300 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:46:32.300 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:46:32.300 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:46:32.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.300+0000 7f5a48915640 1 -- 172.21.15.107:0/847191051 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5a2c0a2f00 msgr2=0x7f5a2c0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:32.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.300+0000 7f5a48915640 1 --2- 172.21.15.107:0/847191051 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5a2c0a2f00 0x7f5a2c0a53c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5a380ebe10 tx=0x7f5a38005b70 comp rx=0 tx=0).stop 2024-05-09T08:46:32.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.300+0000 7f5a48915640 1 -- 172.21.15.107:0/847191051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a3c1439c0 msgr2=0x7f5a3c143e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:32.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.301+0000 7f5a48915640 1 --2- 172.21.15.107:0/847191051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a3c1439c0 0x7f5a3c143e70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5a3802b5d0 tx=0x7f5a3802bbd0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.301+0000 7f5a48915640 1 -- 172.21.15.107:0/847191051 shutdown_connections 2024-05-09T08:46:32.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.302+0000 7f5a48915640 1 --2- 172.21.15.107:0/847191051 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f5a2c0a2f00 0x7f5a2c0a53c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.302+0000 7f5a48915640 1 --2- 172.21.15.107:0/847191051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a3c1439c0 0x7f5a3c143e70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.302+0000 7f5a48915640 1 --2- 172.21.15.107:0/847191051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a3c0aa290 0x7f5a3c149d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.302+0000 7f5a48915640 1 --2- 172.21.15.107:0/847191051 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a3c0a51c0 0x7f5a3c149810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.302+0000 7f5a48915640 1 -- 172.21.15.107:0/847191051 >> 172.21.15.107:0/847191051 conn(0x7f5a3c01aa60 msgr2=0x7f5a3c0a3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.302+0000 7f5a48915640 1 -- 172.21.15.107:0/847191051 shutdown_connections 2024-05-09T08:46:32.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:46:32.302+0000 7f5a48915640 1 -- 172.21.15.107:0/847191051 wait complete. 2024-05-09T08:46:32.322 INFO:tasks.cephfs.filesystem:scrub status for tag:0b0c6d85-4670-442d-96d1-ff7f2c7953f3 - {'path': '/', 'tag': '0b0c6d85-4670-442d-96d1-ff7f2c7953f3', 'options': 'recursive,force'} 2024-05-09T08:46:32.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:32 smithi157 ceph-mon[33614]: pgmap v412: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 11 MiB/s wr, 2.73k op/s 2024-05-09T08:46:32.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:32 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:46:32.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:32 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:32.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:32 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:46:32.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:32 smithi110 ceph-mon[33622]: pgmap v412: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 11 MiB/s wr, 2.73k op/s 2024-05-09T08:46:32.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:32 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:46:32.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:32 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:46:32.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:32 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:46:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.447+0000 7f652e409640 1 -- 172.21.15.107:0/1370991414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65281102d0 msgr2=0x7f652810dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.447+0000 7f652e409640 1 --2- 172.21.15.107:0/1370991414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65281102d0 0x7f652810dee0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6520009ca0 tx=0x7f652002a950 comp rx=0 tx=0).stop 2024-05-09T08:46:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.454+0000 7f652e409640 1 -- 172.21.15.107:0/1370991414 shutdown_connections 2024-05-09T08:46:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.454+0000 7f652e409640 1 --2- 172.21.15.107:0/1370991414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f652810e500 0x7f652811d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.454+0000 7f652e409640 1 --2- 172.21.15.107:0/1370991414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65281102d0 0x7f652810dee0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.454+0000 7f652e409640 1 --2- 172.21.15.107:0/1370991414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f652810fef0 0x7f652810d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.454+0000 7f652e409640 1 -- 172.21.15.107:0/1370991414 >> 172.21.15.107:0/1370991414 conn(0x7f652806c770 msgr2=0x7f652806cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.455+0000 7f652e409640 1 -- 172.21.15.107:0/1370991414 shutdown_connections 2024-05-09T08:46:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.455+0000 7f652e409640 1 -- 172.21.15.107:0/1370991414 wait complete. 2024-05-09T08:46:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.455+0000 7f652e409640 1 Processor -- start 2024-05-09T08:46:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.456+0000 7f652e409640 1 -- start start 2024-05-09T08:46:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.456+0000 7f652e409640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f652810e500 0x7f65281a6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.456+0000 7f652e409640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f652810fef0 0x7f65281a7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.456+0000 7f652e409640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65281102d0 0x7f65281ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.456+0000 7f652e409640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6528120800 con 0x7f652810fef0 2024-05-09T08:46:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.456+0000 7f652e409640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6528120680 con 0x7f65281102d0 2024-05-09T08:46:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.456+0000 7f652e409640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6528120980 con 0x7f652810e500 2024-05-09T08:46:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.456+0000 7f652d407640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f652810e500 0x7f65281a6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.456+0000 7f652d407640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f652810e500 0x7f65281a6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:55158/0 (socket says 172.21.15.107:55158) 2024-05-09T08:46:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.456+0000 7f652d407640 1 -- 172.21.15.107:0/695929837 learned_addr learned my addr 172.21.15.107:0/695929837 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.456+0000 7f652cc06640 1 --2- 172.21.15.107:0/695929837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f652810fef0 0x7f65281a7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.458+0000 7f652d407640 1 -- 172.21.15.107:0/695929837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65281102d0 msgr2=0x7f65281ab6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:46:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.458+0000 7f652d407640 1 --2- 172.21.15.107:0/695929837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65281102d0 0x7f65281ab6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.458+0000 7f652d407640 1 -- 172.21.15.107:0/695929837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f652810fef0 msgr2=0x7f65281a7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.458+0000 7f652d407640 1 --2- 172.21.15.107:0/695929837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f652810fef0 0x7f65281a7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.458+0000 7f652d407640 1 -- 172.21.15.107:0/695929837 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65281abdc0 con 0x7f652810e500 2024-05-09T08:46:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.463+0000 7f652d407640 1 --2- 172.21.15.107:0/695929837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f652810e500 0x7f65281a6e00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f651800d980 tx=0x7f651800de40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.464+0000 7f651e7fc640 1 -- 172.21.15.107:0/695929837 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6518017070 con 0x7f652810e500 2024-05-09T08:46:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.464+0000 7f652e409640 1 -- 172.21.15.107:0/695929837 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65281ac0b0 con 0x7f652810e500 2024-05-09T08:46:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.464+0000 7f652e409640 1 -- 172.21.15.107:0/695929837 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65281107f0 con 0x7f652810e500 2024-05-09T08:46:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.467+0000 7f653237a740 1 Processor -- start 2024-05-09T08:46:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.467+0000 7f653237a740 1 -- start start 2024-05-09T08:46:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.468+0000 7f653237a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851c600 0x5632f851c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.468+0000 7f653237a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f851cf20 0x5632f85272a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.468+0000 7f653237a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f85277e0 0x5632f8529bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.468+0000 7f653237a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5632f8481340 con 0x5632f851cf20 2024-05-09T08:46:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.468+0000 7f653237a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5632f8429290 con 0x5632f85277e0 2024-05-09T08:46:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.468+0000 7f653237a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5632f847d300 con 0x5632f851c600 2024-05-09T08:46:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.468+0000 7f651e7fc640 1 -- 172.21.15.107:0/695929837 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6518009c30 con 0x7f652810e500 2024-05-09T08:46:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.468+0000 7f652cc06640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f85277e0 0x5632f8529bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.468+0000 7f652cc06640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f85277e0 0x5632f8529bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:36338/0 (socket says 172.21.15.107:36338) 2024-05-09T08:46:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.468+0000 7f652cc06640 1 -- 172.21.15.107:0/508975323 learned_addr learned my addr 172.21.15.107:0/508975323 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.469+0000 7f652dc08640 1 --2- 172.21.15.107:0/508975323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851c600 0x5632f851c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.469+0000 7f652cc06640 1 -- 172.21.15.107:0/508975323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851c600 msgr2=0x5632f851c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.469+0000 7f652cc06640 1 --2- 172.21.15.107:0/508975323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851c600 0x5632f851c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.469+0000 7f652cc06640 1 -- 172.21.15.107:0/508975323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f851cf20 msgr2=0x5632f85272a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:46:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.469+0000 7f652cc06640 1 --2- 172.21.15.107:0/508975323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f851cf20 0x5632f85272a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.469+0000 7f651e7fc640 1 -- 172.21.15.107:0/695929837 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651800f620 con 0x7f652810e500 2024-05-09T08:46:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.469+0000 7f652cc06640 1 -- 172.21.15.107:0/508975323 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632f844d8e0 con 0x5632f85277e0 2024-05-09T08:46:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.470+0000 7f652cc06640 1 --2- 172.21.15.107:0/508975323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f85277e0 0x5632f8529bd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6520009bf0 tx=0x7f652002f670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=782cea2fb8d40564 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.470+0000 7f65137fe640 1 -- 172.21.15.107:0/508975323 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6520034040 con 0x5632f85277e0 2024-05-09T08:46:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.470+0000 7f65137fe640 1 -- 172.21.15.107:0/508975323 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f652002fd20 con 0x5632f85277e0 2024-05-09T08:46:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.471+0000 7f653237a740 1 -- 172.21.15.107:0/508975323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f85277e0 msgr2=0x5632f8529bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.471+0000 7f653237a740 1 --2- 172.21.15.107:0/508975323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f85277e0 0x5632f8529bd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6520009bf0 tx=0x7f652002f670 comp rx=0 tx=0).stop 2024-05-09T08:46:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.474+0000 7f653237a740 1 -- 172.21.15.107:0/508975323 shutdown_connections 2024-05-09T08:46:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.474+0000 7f653237a740 1 --2- 172.21.15.107:0/508975323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f85277e0 0x5632f8529bd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.474+0000 7f653237a740 1 --2- 172.21.15.107:0/508975323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f851cf20 0x5632f85272a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.474+0000 7f653237a740 1 --2- 172.21.15.107:0/508975323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851c600 0x5632f851c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.474+0000 7f653237a740 1 -- 172.21.15.107:0/508975323 >> 172.21.15.107:0/508975323 conn(0x5632f8518b10 msgr2=0x5632f8518ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:34.484 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.483+0000 7f651e7fc640 1 -- 172.21.15.107:0/695929837 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6518010460 con 0x7f652810e500 2024-05-09T08:46:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.486+0000 7f651e7fc640 1 --2- 172.21.15.107:0/695929837 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65140a2de0 0x7f65140a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.487+0000 7f652cc06640 1 --2- 172.21.15.107:0/695929837 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65140a2de0 0x7f65140a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.487+0000 7f651e7fc640 1 -- 172.21.15.107:0/695929837 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f65180cf350 con 0x7f652810e500 2024-05-09T08:46:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.488+0000 7f652cc06640 1 --2- 172.21.15.107:0/695929837 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65140a2de0 0x7f65140a52a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6520003fd0 tx=0x7f652002a950 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.494+0000 7f653237a740 1 -- 172.21.15.107:0/508975323 shutdown_connections 2024-05-09T08:46:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.497+0000 7f653237a740 1 -- 172.21.15.107:0/508975323 wait complete. 2024-05-09T08:46:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.497+0000 7f653237a740 1 -- start start 2024-05-09T08:46:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.498+0000 7f653237a740 1 Processor -- start 2024-05-09T08:46:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.498+0000 7f653237a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f851c600 0x5632f85247a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.498+0000 7f653237a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851cf20 0x5632f8524ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.498+0000 7f653237a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f85277e0 0x5632f8525220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.498+0000 7f653237a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5632f852a3b0 con 0x5632f85277e0 2024-05-09T08:46:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.498+0000 7f653237a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5632f843c790 con 0x5632f851c600 2024-05-09T08:46:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.498+0000 7f653237a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5632f852a530 con 0x5632f851cf20 2024-05-09T08:46:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.498+0000 7f653237a740 10 client.0 _ll_register_callbacks cb 0x5632f8477a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.507+0000 7f652cc06640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851cf20 0x5632f8524ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.508+0000 7f652cc06640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851cf20 0x5632f8524ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:55182/0 (socket says 172.21.15.107:55182) 2024-05-09T08:46:34.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.508+0000 7f652cc06640 1 -- 172.21.15.107:0/791740474 learned_addr learned my addr 172.21.15.107:0/791740474 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:34.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.510+0000 7f652dc08640 1 --2- 172.21.15.107:0/791740474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f85277e0 0x5632f8525220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:34.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.513+0000 7f652dc08640 1 -- 172.21.15.107:0/791740474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851cf20 msgr2=0x5632f8524ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:34.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.513+0000 7f652dc08640 1 --2- 172.21.15.107:0/791740474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851cf20 0x5632f8524ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.513+0000 7f652dc08640 1 -- 172.21.15.107:0/791740474 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f851c600 msgr2=0x5632f85247a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:34.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.513+0000 7f652dc08640 1 --2- 172.21.15.107:0/791740474 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f851c600 0x5632f85247a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.513+0000 7f652dc08640 1 -- 172.21.15.107:0/791740474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632f8195210 con 0x5632f85277e0 2024-05-09T08:46:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.518+0000 7f652cc06640 1 --2- 172.21.15.107:0/791740474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851cf20 0x5632f8524ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:46:34.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.523+0000 7f652dc08640 1 --2- 172.21.15.107:0/791740474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f85277e0 0x5632f8525220 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f652400bf80 tx=0x7f652400d5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=113fbee79b1df8c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:34.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.523+0000 7f6510ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:46:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.525+0000 7f6510ff9640 1 -- 172.21.15.107:0/791740474 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6524014430 con 0x5632f85277e0 2024-05-09T08:46:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.525+0000 7f653237a740 1 -- 172.21.15.107:0/791740474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5632f848ea70 con 0x5632f85277e0 2024-05-09T08:46:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.525+0000 7f6510ff9640 1 -- 172.21.15.107:0/791740474 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f65240145d0 con 0x5632f85277e0 2024-05-09T08:46:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.526+0000 7f6510ff9640 1 -- 172.21.15.107:0/791740474 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6524022050 con 0x5632f85277e0 2024-05-09T08:46:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.526+0000 7f653237a740 10 client.19104 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.526+0000 7f653237a740 1 -- 172.21.15.107:0/791740474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5632f8261620 con 0x5632f85277e0 2024-05-09T08:46:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.527+0000 7f6510ff9640 1 -- 172.21.15.107:0/791740474 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5632f848ea70 con 0x5632f85277e0 2024-05-09T08:46:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.528+0000 7f6510ff9640 1 -- 172.21.15.107:0/791740474 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f6524013070 con 0x5632f85277e0 2024-05-09T08:46:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.528+0000 7f653237a740 10 client.19104 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.528+0000 7f653237a740 10 client.19104 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:46:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.528+0000 7f653237a740 20 client.19104 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.529+0000 7f653237a740 1 --2- 172.21.15.107:0/791740474 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5632f8537c60 0x5632f853a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.529+0000 7f653237a740 4 client.19104 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:46:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.529+0000 7f653237a740 1 -- 172.21.15.107:0/791740474 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632f8488dc0 con 0x5632f8537c60 2024-05-09T08:46:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.530+0000 7f652d407640 1 --2- 172.21.15.107:0/791740474 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5632f8537c60 0x5632f853a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:34.532 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:34 smithi107 ceph-mon[30122]: pgmap v413: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 13 MiB/s wr, 3.18k op/s 2024-05-09T08:46:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.537+0000 7f652d407640 1 --2- 172.21.15.107:0/791740474 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5632f8537c60 0x5632f853a120 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71ff8025326fb6df server_cookie=aad7564993694a2d in_seq=0 out_seq=0 2024-05-09T08:46:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.537+0000 7f6510ff9640 10 client.19104 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:34.541 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.542+0000 7f6510ff9640 1 -- 172.21.15.107:0/791740474 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5632f8488dc0 con 0x5632f8537c60 2024-05-09T08:46:34.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.542+0000 7f6510ff9640 10 client.19104 handle_command_reply: tid=0 2024-05-09T08:46:34.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.543+0000 7f653237a740 1 client.19104 shutdown 2024-05-09T08:46:34.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.546+0000 7f653237a740 1 -- 172.21.15.107:0/791740474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f85277e0 msgr2=0x5632f8525220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:34.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.546+0000 7f653237a740 1 --2- 172.21.15.107:0/791740474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f85277e0 0x5632f8525220 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f652400bf80 tx=0x7f652400d5f0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.549+0000 7f653237a740 1 -- 172.21.15.107:0/791740474 shutdown_connections 2024-05-09T08:46:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.549+0000 7f653237a740 1 --2- 172.21.15.107:0/791740474 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5632f8537c60 0x5632f853a120 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.549+0000 7f653237a740 1 --2- 172.21.15.107:0/791740474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f85277e0 0x5632f8525220 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.549+0000 7f653237a740 1 --2- 172.21.15.107:0/791740474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851cf20 0x5632f8524ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.549+0000 7f653237a740 1 --2- 172.21.15.107:0/791740474 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f851c600 0x5632f85247a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.549+0000 7f653237a740 1 -- 172.21.15.107:0/791740474 >> 172.21.15.107:0/791740474 conn(0x5632f8518b10 msgr2=0x5632f851e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.549+0000 7f6510ff9640 0 client.19104 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.549+0000 7f653237a740 1 -- 172.21.15.107:0/791740474 shutdown_connections 2024-05-09T08:46:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.549+0000 7f653237a740 1 -- 172.21.15.107:0/791740474 wait complete. 2024-05-09T08:46:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.549+0000 7f653237a740 20 client.19104 trim_cache size 0 max 16384 2024-05-09T08:46:34.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:34 smithi157 ceph-mon[33614]: pgmap v413: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 13 MiB/s wr, 3.18k op/s 2024-05-09T08:46:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.593+0000 7f653237a740 1 Processor -- start 2024-05-09T08:46:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.596+0000 7f653237a740 1 -- start start 2024-05-09T08:46:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.596+0000 7f653237a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f851c600 0x5632f8526ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.596+0000 7f653237a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f851cf20 0x5632f8533350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.596+0000 7f653237a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f85277e0 0x5632f851fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.596+0000 7f653237a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f650c0128c0 con 0x5632f851cf20 2024-05-09T08:46:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.596+0000 7f653237a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f650c010190 con 0x5632f851c600 2024-05-09T08:46:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.596+0000 7f653237a740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f650c00e090 con 0x5632f85277e0 2024-05-09T08:46:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.598+0000 7f652d407640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f851c600 0x5632f8526ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.598+0000 7f652d407640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f851c600 0x5632f8526ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:36352/0 (socket says 172.21.15.107:36352) 2024-05-09T08:46:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.598+0000 7f652d407640 1 -- 172.21.15.107:0/2996953246 learned_addr learned my addr 172.21.15.107:0/2996953246 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.600+0000 7f652dc08640 1 --2- 172.21.15.107:0/2996953246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f85277e0 0x5632f851fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.601+0000 7f652d407640 1 -- 172.21.15.107:0/2996953246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f85277e0 msgr2=0x5632f851fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.601+0000 7f652d407640 1 --2- 172.21.15.107:0/2996953246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f85277e0 0x5632f851fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.601+0000 7f652d407640 1 -- 172.21.15.107:0/2996953246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f851cf20 msgr2=0x5632f8533350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.601+0000 7f652d407640 1 --2- 172.21.15.107:0/2996953246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f851cf20 0x5632f8533350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.601+0000 7f652d407640 1 -- 172.21.15.107:0/2996953246 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632f8525770 con 0x5632f851c600 2024-05-09T08:46:34.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.605+0000 7f652d407640 1 --2- 172.21.15.107:0/2996953246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f851c600 0x5632f8526ed0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f65180978b0 tx=0x7f651801d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea0c0b77d15ec33 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:34.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.608+0000 7f6512ffd640 1 -- 172.21.15.107:0/2996953246 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65180b3070 con 0x5632f851c600 2024-05-09T08:46:34.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.608+0000 7f6512ffd640 1 -- 172.21.15.107:0/2996953246 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f651809b450 con 0x5632f851c600 2024-05-09T08:46:34.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.609+0000 7f653237a740 1 -- 172.21.15.107:0/2996953246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f851c600 msgr2=0x5632f8526ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:34.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.609+0000 7f653237a740 1 --2- 172.21.15.107:0/2996953246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f851c600 0x5632f8526ed0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f65180978b0 tx=0x7f651801d040 comp rx=0 tx=0).stop 2024-05-09T08:46:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.616+0000 7f653237a740 1 -- 172.21.15.107:0/2996953246 shutdown_connections 2024-05-09T08:46:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.616+0000 7f653237a740 1 --2- 172.21.15.107:0/2996953246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f85277e0 0x5632f851fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.616+0000 7f653237a740 1 --2- 172.21.15.107:0/2996953246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f851cf20 0x5632f8533350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.616+0000 7f653237a740 1 --2- 172.21.15.107:0/2996953246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f851c600 0x5632f8526ed0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.616+0000 7f653237a740 1 -- 172.21.15.107:0/2996953246 >> 172.21.15.107:0/2996953246 conn(0x5632f8518b10 msgr2=0x5632f8518ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:34.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:34 smithi110 ceph-mon[33622]: pgmap v413: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 13 MiB/s wr, 3.18k op/s 2024-05-09T08:46:34.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.619+0000 7f653237a740 1 -- 172.21.15.107:0/2996953246 shutdown_connections 2024-05-09T08:46:34.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.621+0000 7f653237a740 1 -- 172.21.15.107:0/2996953246 wait complete. 2024-05-09T08:46:34.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.621+0000 7f653237a740 1 -- start start 2024-05-09T08:46:34.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.621+0000 7f653237a740 1 Processor -- start 2024-05-09T08:46:34.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.622+0000 7f653237a740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851c600 0x5632f852f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.622+0000 7f653237a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f851cf20 0x5632f852fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.622+0000 7f653237a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f85277e0 0x5632f8520a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.622+0000 7f652d407640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851c600 0x5632f852f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.622+0000 7f652d407640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851c600 0x5632f852f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:55194/0 (socket says 172.21.15.107:55194) 2024-05-09T08:46:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.622+0000 7f652d407640 1 -- 172.21.15.107:0/4137237795 learned_addr learned my addr 172.21.15.107:0/4137237795 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.622+0000 7f653237a740 1 -- 172.21.15.107:0/4137237795 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f650c00e650 con 0x5632f85277e0 2024-05-09T08:46:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.622+0000 7f653237a740 1 -- 172.21.15.107:0/4137237795 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f650c00fac0 con 0x5632f851cf20 2024-05-09T08:46:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.622+0000 7f653237a740 1 -- 172.21.15.107:0/4137237795 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f650c00ed20 con 0x5632f851c600 2024-05-09T08:46:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.622+0000 7f653237a740 10 client.0 _ll_register_callbacks cb 0x7f652400b0e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.623+0000 7f652dc08640 1 --2- 172.21.15.107:0/4137237795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f85277e0 0x5632f8520a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.625+0000 7f652d407640 1 -- 172.21.15.107:0/4137237795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f851cf20 msgr2=0x5632f852fda0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:46:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.625+0000 7f652d407640 1 --2- 172.21.15.107:0/4137237795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f851cf20 0x5632f852fda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.625+0000 7f652d407640 1 -- 172.21.15.107:0/4137237795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f85277e0 msgr2=0x5632f8520a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.625+0000 7f652d407640 1 --2- 172.21.15.107:0/4137237795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f85277e0 0x5632f8520a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.625+0000 7f652d407640 1 -- 172.21.15.107:0/4137237795 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632f8521050 con 0x5632f851c600 2024-05-09T08:46:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.625+0000 7f652d407640 1 --2- 172.21.15.107:0/4137237795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851c600 0x5632f852f860 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f651800d950 tx=0x7f651809f870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b937326c48c4e40a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.625+0000 7f6510ff9640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:46:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.628+0000 7f6510ff9640 1 -- 172.21.15.107:0/4137237795 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65180b8060 con 0x5632f851c600 2024-05-09T08:46:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.628+0000 7f653237a740 1 -- 172.21.15.107:0/4137237795 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5632f84068e0 con 0x5632f851c600 2024-05-09T08:46:34.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.628+0000 7f6510ff9640 1 -- 172.21.15.107:0/4137237795 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f65180cc860 con 0x5632f851c600 2024-05-09T08:46:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.630+0000 7f6510ff9640 1 -- 172.21.15.107:0/4137237795 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f65180b3040 con 0x5632f851c600 2024-05-09T08:46:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.634+0000 7f653237a740 10 client.28270 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.634+0000 7f653237a740 1 -- 172.21.15.107:0/4137237795 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5632f859dc40 con 0x5632f851c600 2024-05-09T08:46:34.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.637+0000 7f6510ff9640 1 -- 172.21.15.107:0/4137237795 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5632f84068e0 con 0x5632f851c600 2024-05-09T08:46:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.637+0000 7f6510ff9640 1 -- 172.21.15.107:0/4137237795 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f6518010750 con 0x5632f851c600 2024-05-09T08:46:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.637+0000 7f653237a740 10 client.28270 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.637+0000 7f653237a740 10 client.28270 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:46:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.637+0000 7f653237a740 20 client.28270 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.637+0000 7f653237a740 1 --2- 172.21.15.107:0/4137237795 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5632f859e3d0 0x5632f85a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.637+0000 7f653237a740 4 client.28270 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:46:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.637+0000 7f653237a740 1 -- 172.21.15.107:0/4137237795 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5632f8452870 con 0x5632f859e3d0 2024-05-09T08:46:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.642+0000 7f652cc06640 1 --2- 172.21.15.107:0/4137237795 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5632f859e3d0 0x5632f85a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.651+0000 7f652cc06640 1 --2- 172.21.15.107:0/4137237795 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5632f859e3d0 0x5632f85a07f0 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=405414f19f06eae4 server_cookie=c10160f67f078138 in_seq=0 out_seq=0 2024-05-09T08:46:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:34.653+0000 7f6510ff9640 10 client.28270 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.405+0000 7f6510ff9640 1 -- 172.21.15.107:0/4137237795 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5632f8452870 con 0x5632f859e3d0 2024-05-09T08:46:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.405+0000 7f6510ff9640 10 client.28270 handle_command_reply: tid=0 2024-05-09T08:46:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.405+0000 7f653237a740 1 client.28270 shutdown 2024-05-09T08:46:35.410 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.410+0000 7f653237a740 1 -- 172.21.15.107:0/4137237795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851c600 msgr2=0x5632f852f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:35.410 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.410+0000 7f653237a740 1 --2- 172.21.15.107:0/4137237795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851c600 0x5632f852f860 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f651800d950 tx=0x7f651809f870 comp rx=0 tx=0).stop 2024-05-09T08:46:35.410 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.410+0000 7f653237a740 1 -- 172.21.15.107:0/4137237795 shutdown_connections 2024-05-09T08:46:35.410 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.410+0000 7f653237a740 1 --2- 172.21.15.107:0/4137237795 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5632f859e3d0 0x5632f85a07f0 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:35.410 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.410+0000 7f653237a740 1 --2- 172.21.15.107:0/4137237795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f85277e0 0x5632f8520a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.410+0000 7f653237a740 1 --2- 172.21.15.107:0/4137237795 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5632f851cf20 0x5632f852fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.410+0000 7f653237a740 1 --2- 172.21.15.107:0/4137237795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5632f851c600 0x5632f852f860 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f651800d950 tx=0x7f651809f870 comp rx=0 tx=0).stop 2024-05-09T08:46:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.410+0000 7f653237a740 1 -- 172.21.15.107:0/4137237795 >> 172.21.15.107:0/4137237795 conn(0x5632f8518b10 msgr2=0x5632f851fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:35.413 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.411+0000 7f6510ff9640 0 client.28270 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:35.422 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.422+0000 7f653237a740 1 -- 172.21.15.107:0/4137237795 shutdown_connections 2024-05-09T08:46:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.426+0000 7f653237a740 1 -- 172.21.15.107:0/4137237795 wait complete. 2024-05-09T08:46:35.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.426+0000 7f653237a740 20 client.28270 trim_cache size 0 max 16384 2024-05-09T08:46:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:46:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:46:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:46:35.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:46:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.429+0000 7f652e409640 1 -- 172.21.15.107:0/695929837 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65140a2de0 msgr2=0x7f65140a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.429+0000 7f652e409640 1 --2- 172.21.15.107:0/695929837 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65140a2de0 0x7f65140a52a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6520003fd0 tx=0x7f652002a950 comp rx=0 tx=0).stop 2024-05-09T08:46:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.429+0000 7f652e409640 1 -- 172.21.15.107:0/695929837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f652810e500 msgr2=0x7f65281a6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.429+0000 7f652e409640 1 --2- 172.21.15.107:0/695929837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f652810e500 0x7f65281a6e00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f651800d980 tx=0x7f651800de40 comp rx=0 tx=0).stop 2024-05-09T08:46:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.429+0000 7f652e409640 1 -- 172.21.15.107:0/695929837 shutdown_connections 2024-05-09T08:46:35.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.430+0000 7f652e409640 1 --2- 172.21.15.107:0/695929837 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f65140a2de0 0x7f65140a52a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:35.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.430+0000 7f652e409640 1 --2- 172.21.15.107:0/695929837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65281102d0 0x7f65281ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:35.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.430+0000 7f652e409640 1 --2- 172.21.15.107:0/695929837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f652810fef0 0x7f65281a7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:35.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.430+0000 7f652e409640 1 --2- 172.21.15.107:0/695929837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f652810e500 0x7f65281a6e00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:35.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.430+0000 7f652e409640 1 -- 172.21.15.107:0/695929837 >> 172.21.15.107:0/695929837 conn(0x7f652806c770 msgr2=0x7f6528070b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:35.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.431+0000 7f652e409640 1 -- 172.21.15.107:0/695929837 shutdown_connections 2024-05-09T08:46:35.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:35.431+0000 7f652e409640 1 -- 172.21.15.107:0/695929837 wait complete. 2024-05-09T08:46:36.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:36 smithi107 ceph-mon[30122]: pgmap v414: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 10 MiB/s wr, 2.45k op/s 2024-05-09T08:46:36.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:36 smithi157 ceph-mon[33614]: pgmap v414: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 10 MiB/s wr, 2.45k op/s 2024-05-09T08:46:36.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:36 smithi110 ceph-mon[33622]: pgmap v414: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 10 MiB/s wr, 2.45k op/s 2024-05-09T08:46:38.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:38 smithi107 ceph-mon[30122]: pgmap v415: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 10 MiB/s wr, 2.53k op/s 2024-05-09T08:46:38.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:38 smithi157 ceph-mon[33614]: pgmap v415: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 10 MiB/s wr, 2.53k op/s 2024-05-09T08:46:38.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:38 smithi110 ceph-mon[33622]: pgmap v415: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 10 MiB/s wr, 2.53k op/s 2024-05-09T08:46:40.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:40 smithi107 ceph-mon[30122]: pgmap v416: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 13 MiB/s wr, 3.23k op/s 2024-05-09T08:46:40.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:40 smithi157 ceph-mon[33614]: pgmap v416: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 13 MiB/s wr, 3.23k op/s 2024-05-09T08:46:40.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:40 smithi110 ceph-mon[33622]: pgmap v416: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 13 MiB/s wr, 3.23k op/s 2024-05-09T08:46:42.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:42 smithi107 ceph-mon[30122]: pgmap v417: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 9.8 MiB/s wr, 2.43k op/s 2024-05-09T08:46:42.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:42 smithi157 ceph-mon[33614]: pgmap v417: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 9.8 MiB/s wr, 2.43k op/s 2024-05-09T08:46:42.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:42 smithi110 ceph-mon[33622]: pgmap v417: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 9.8 MiB/s wr, 2.43k op/s 2024-05-09T08:46:44.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:44 smithi107 ceph-mon[30122]: pgmap v418: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 13 MiB/s wr, 3.31k op/s 2024-05-09T08:46:44.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:44 smithi157 ceph-mon[33614]: pgmap v418: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 13 MiB/s wr, 3.31k op/s 2024-05-09T08:46:44.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:44 smithi110 ceph-mon[33622]: pgmap v418: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 13 MiB/s wr, 3.31k op/s 2024-05-09T08:46:46.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:46 smithi107 ceph-mon[30122]: pgmap v419: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 10 MiB/s wr, 2.65k op/s 2024-05-09T08:46:46.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:46 smithi157 ceph-mon[33614]: pgmap v419: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 10 MiB/s wr, 2.65k op/s 2024-05-09T08:46:46.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:46 smithi110 ceph-mon[33622]: pgmap v419: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 10 MiB/s wr, 2.65k op/s 2024-05-09T08:46:48.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:48 smithi107 ceph-mon[30122]: pgmap v420: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 10 MiB/s wr, 2.55k op/s 2024-05-09T08:46:48.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:48 smithi157 ceph-mon[33614]: pgmap v420: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 10 MiB/s wr, 2.55k op/s 2024-05-09T08:46:48.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.615+0000 7f9a5da2a640 1 -- 172.21.15.107:0/3123101374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a5810dd50 msgr2=0x7f9a5810e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:48.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.615+0000 7f9a5da2a640 1 --2- 172.21.15.107:0/3123101374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a5810dd50 0x7f9a5810e130 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9a50009be0 tx=0x7f9a5002a710 comp rx=0 tx=0).stop 2024-05-09T08:46:48.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.617+0000 7f9a5da2a640 1 -- 172.21.15.107:0/3123101374 shutdown_connections 2024-05-09T08:46:48.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.617+0000 7f9a5da2a640 1 --2- 172.21.15.107:0/3123101374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a5810d190 0x7f9a58079b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.617+0000 7f9a5da2a640 1 --2- 172.21.15.107:0/3123101374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a5810e700 0x7f9a5810ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.617+0000 7f9a5da2a640 1 --2- 172.21.15.107:0/3123101374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a5810dd50 0x7f9a5810e130 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.617+0000 7f9a5da2a640 1 -- 172.21.15.107:0/3123101374 >> 172.21.15.107:0/3123101374 conn(0x7f9a5806c900 msgr2=0x7f9a5806cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:48.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:48 smithi110 ceph-mon[33622]: pgmap v420: 129 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 115 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 10 MiB/s wr, 2.55k op/s 2024-05-09T08:46:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.626+0000 7f9a5da2a640 1 -- 172.21.15.107:0/3123101374 shutdown_connections 2024-05-09T08:46:48.636 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.635+0000 7f9a5da2a640 1 -- 172.21.15.107:0/3123101374 wait complete. 2024-05-09T08:46:48.636 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.635+0000 7f9a5da2a640 1 Processor -- start 2024-05-09T08:46:48.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.636+0000 7f9a5da2a640 1 -- start start 2024-05-09T08:46:48.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.636+0000 7f9a5da2a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a5810d190 0x7f9a58079a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.636+0000 7f9a5da2a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a5810dd50 0x7f9a58073cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.636+0000 7f9a5da2a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a58074250 0x7f9a580746b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.636+0000 7f9a5da2a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5807c090 con 0x7f9a5810dd50 2024-05-09T08:46:48.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.636+0000 7f9a5da2a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5807bf10 con 0x7f9a58074250 2024-05-09T08:46:48.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.636+0000 7f9a5da2a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5807c210 con 0x7f9a5810d190 2024-05-09T08:46:48.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.636+0000 7f9a56ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a5810d190 0x7f9a58079a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:48.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.636+0000 7f9a56ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a5810d190 0x7f9a58079a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:45194/0 (socket says 172.21.15.107:45194) 2024-05-09T08:46:48.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.636+0000 7f9a56ffd640 1 -- 172.21.15.107:0/3516209040 learned_addr learned my addr 172.21.15.107:0/3516209040 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:48.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.637+0000 7f9a577fe640 1 --2- 172.21.15.107:0/3516209040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a58074250 0x7f9a580746b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.637+0000 7f9a56ffd640 1 -- 172.21.15.107:0/3516209040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a58074250 msgr2=0x7f9a580746b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.637+0000 7f9a56ffd640 1 --2- 172.21.15.107:0/3516209040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a58074250 0x7f9a580746b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.637+0000 7f9a56ffd640 1 -- 172.21.15.107:0/3516209040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a5810dd50 msgr2=0x7f9a58073cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.637+0000 7f9a56ffd640 1 --2- 172.21.15.107:0/3516209040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a5810dd50 0x7f9a58073cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.637+0000 7f9a56ffd640 1 -- 172.21.15.107:0/3516209040 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a58136f50 con 0x7f9a5810d190 2024-05-09T08:46:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.637+0000 7f9a56ffd640 1 --2- 172.21.15.107:0/3516209040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a5810d190 0x7f9a58079a70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9a50006010 tx=0x7f9a50003b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.638+0000 7f9a5ca28640 1 -- 172.21.15.107:0/3516209040 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a50041020 con 0x7f9a5810d190 2024-05-09T08:46:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.638+0000 7f9a5ca28640 1 -- 172.21.15.107:0/3516209040 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a50037040 con 0x7f9a5810d190 2024-05-09T08:46:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.638+0000 7f9a5ca28640 1 -- 172.21.15.107:0/3516209040 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a50036070 con 0x7f9a5810d190 2024-05-09T08:46:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.638+0000 7f9a5da2a640 1 -- 172.21.15.107:0/3516209040 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a581371e0 con 0x7f9a5810d190 2024-05-09T08:46:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.638+0000 7f9a5da2a640 1 -- 172.21.15.107:0/3516209040 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a58137680 con 0x7f9a5810d190 2024-05-09T08:46:48.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.643+0000 7f9a60813740 1 Processor -- start 2024-05-09T08:46:48.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.643+0000 7f9a60813740 1 -- start start 2024-05-09T08:46:48.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.643+0000 7f9a60813740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 0x55d89e07f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.643+0000 7f9a60813740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e08a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.643+0000 7f9a60813740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e08a610 0x55d89e08ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.643+0000 7f9a60813740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d89dfe4340 con 0x55d89e08a610 2024-05-09T08:46:48.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.643+0000 7f9a60813740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d89df8c290 con 0x55d89e07fd50 2024-05-09T08:46:48.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.643+0000 7f9a60813740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d89dfe0300 con 0x55d89e07f430 2024-05-09T08:46:48.643 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.644+0000 7f9a5ca28640 1 -- 172.21.15.107:0/3516209040 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f9a50044020 con 0x7f9a5810d190 2024-05-09T08:46:48.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.644+0000 7f9a577fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 0x55d89e07f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:48.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.644+0000 7f9a577fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 0x55d89e07f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:45198/0 (socket says 172.21.15.107:45198) 2024-05-09T08:46:48.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.644+0000 7f9a577fe640 1 -- 172.21.15.107:0/984974133 learned_addr learned my addr 172.21.15.107:0/984974133 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:48.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.644+0000 7f9a56ffd640 1 --2- 172.21.15.107:0/984974133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e08a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.648+0000 7f9a56ffd640 1 -- 172.21.15.107:0/984974133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 msgr2=0x55d89e07f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.648+0000 7f9a56ffd640 1 --2- 172.21.15.107:0/984974133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 0x55d89e07f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.648+0000 7f9a56ffd640 1 -- 172.21.15.107:0/984974133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e08a610 msgr2=0x55d89e08ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.648+0000 7f9a56ffd640 1 --2- 172.21.15.107:0/984974133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e08a610 0x55d89e08ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.648+0000 7f9a56ffd640 1 -- 172.21.15.107:0/984974133 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d89dfb08e0 con 0x55d89e07fd50 2024-05-09T08:46:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.648+0000 7f9a56ffd640 1 --2- 172.21.15.107:0/984974133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e08a0d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9a500edda0 tx=0x7f9a500f67d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df43dad2429a6dda server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.649+0000 7f9a5ca28640 1 --2- 172.21.15.107:0/3516209040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9a400a2fe0 0x7f9a400a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.649+0000 7f9a5ca28640 1 -- 172.21.15.107:0/3516209040 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f9a500f0b10 con 0x7f9a5810d190 2024-05-09T08:46:48.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.650+0000 7f9a577fe640 1 --2- 172.21.15.107:0/984974133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 0x55d89e07f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:46:48.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.650+0000 7f9a3dffb640 1 -- 172.21.15.107:0/984974133 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a50102620 con 0x55d89e07fd50 2024-05-09T08:46:48.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.650+0000 7f9a3dffb640 1 -- 172.21.15.107:0/984974133 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a500fd030 con 0x55d89e07fd50 2024-05-09T08:46:48.656 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.656+0000 7f9a60813740 1 -- 172.21.15.107:0/984974133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 msgr2=0x55d89e08a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:48.656 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.657+0000 7f9a577fe640 1 --2- 172.21.15.107:0/3516209040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9a400a2fe0 0x7f9a400a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:48.656 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.657+0000 7f9a60813740 1 --2- 172.21.15.107:0/984974133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e08a0d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9a500edda0 tx=0x7f9a500f67d0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.657 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.658+0000 7f9a60813740 1 -- 172.21.15.107:0/984974133 shutdown_connections 2024-05-09T08:46:48.657 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.658+0000 7f9a60813740 1 --2- 172.21.15.107:0/984974133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e08a610 0x55d89e08ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.657 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.658+0000 7f9a60813740 1 --2- 172.21.15.107:0/984974133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e08a0d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.657 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.658+0000 7f9a60813740 1 --2- 172.21.15.107:0/984974133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 0x55d89e07f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.657 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.658+0000 7f9a60813740 1 -- 172.21.15.107:0/984974133 >> 172.21.15.107:0/984974133 conn(0x55d89e07b940 msgr2=0x55d89e07bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:48.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.660+0000 7f9a577fe640 1 --2- 172.21.15.107:0/3516209040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9a400a2fe0 0x7f9a400a54a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c00bb30 tx=0x7f9a4c008040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:48.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.662+0000 7f9a60813740 1 -- 172.21.15.107:0/984974133 shutdown_connections 2024-05-09T08:46:48.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.662+0000 7f9a60813740 1 -- 172.21.15.107:0/984974133 wait complete. 2024-05-09T08:46:48.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.662+0000 7f9a60813740 1 -- start start 2024-05-09T08:46:48.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.663+0000 7f9a60813740 1 Processor -- start 2024-05-09T08:46:48.664 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.663+0000 7f9a60813740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 0x55d89e0875c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.664 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.663+0000 7f9a60813740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e087b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.664 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.663+0000 7f9a60813740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e08a610 0x55d89e088040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.664 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.663+0000 7f9a60813740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d89e08d000 con 0x55d89e08a610 2024-05-09T08:46:48.664 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.663+0000 7f9a60813740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d89df9f790 con 0x55d89e07fd50 2024-05-09T08:46:48.664 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.663+0000 7f9a60813740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d89e08d180 con 0x55d89e07f430 2024-05-09T08:46:48.664 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.663+0000 7f9a60813740 10 client.0 _ll_register_callbacks cb 0x55d89dfdaa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:48.664 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.663+0000 7f9a567fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e087b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:48.664 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.664+0000 7f9a567fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e087b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44402/0 (socket says 172.21.15.107:44402) 2024-05-09T08:46:48.664 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.664+0000 7f9a567fc640 1 -- 172.21.15.107:0/636448272 learned_addr learned my addr 172.21.15.107:0/636448272 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:48.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.664+0000 7f9a577fe640 1 --2- 172.21.15.107:0/636448272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e08a610 0x55d89e088040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:48.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.664+0000 7f9a567fc640 1 -- 172.21.15.107:0/636448272 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 msgr2=0x55d89e0875c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:46:48.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.664+0000 7f9a567fc640 1 --2- 172.21.15.107:0/636448272 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 0x55d89e0875c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.664+0000 7f9a567fc640 1 -- 172.21.15.107:0/636448272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e08a610 msgr2=0x55d89e088040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:48.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.664+0000 7f9a567fc640 1 --2- 172.21.15.107:0/636448272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e08a610 0x55d89e088040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.664+0000 7f9a567fc640 1 -- 172.21.15.107:0/636448272 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d89dcf8210 con 0x55d89e07fd50 2024-05-09T08:46:48.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.664+0000 7f9a567fc640 1 --2- 172.21.15.107:0/636448272 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e087b00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9a5810eb50 tx=0x7f9a4800ea60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3400eb3480a09351 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:48.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.664+0000 7f9a36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:46:48.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.664+0000 7f9a36ffd640 1 -- 172.21.15.107:0/636448272 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a48004230 con 0x55d89e07fd50 2024-05-09T08:46:48.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.665+0000 7f9a60813740 1 -- 172.21.15.107:0/636448272 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d89dff1a70 con 0x55d89e07fd50 2024-05-09T08:46:48.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.665+0000 7f9a36ffd640 1 -- 172.21.15.107:0/636448272 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f9a480043d0 con 0x55d89e07fd50 2024-05-09T08:46:48.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.665+0000 7f9a36ffd640 1 -- 172.21.15.107:0/636448272 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9a480107c0 con 0x55d89e07fd50 2024-05-09T08:46:48.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.666+0000 7f9a60813740 10 client.28061 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:48.666 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.666+0000 7f9a60813740 1 -- 172.21.15.107:0/636448272 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d89ddc4620 con 0x55d89e07fd50 2024-05-09T08:46:48.669 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.670+0000 7f9a36ffd640 1 -- 172.21.15.107:0/636448272 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55d89dff1a70 con 0x55d89e07fd50 2024-05-09T08:46:48.669 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.670+0000 7f9a36ffd640 1 -- 172.21.15.107:0/636448272 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9a48003c50 con 0x55d89e07fd50 2024-05-09T08:46:48.671 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.671+0000 7f9a60813740 10 client.28061 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:48.671 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.671+0000 7f9a60813740 10 client.28061 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:46:48.671 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.671+0000 7f9a60813740 20 client.28061 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:48.671 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.671+0000 7f9a60813740 1 --2- 172.21.15.107:0/636448272 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d89e09a280 0x55d89e09c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.671 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.671+0000 7f9a60813740 4 client.28061 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:46:48.671 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.671+0000 7f9a60813740 1 -- 172.21.15.107:0/636448272 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d89dfebdc0 con 0x55d89e09a280 2024-05-09T08:46:48.672 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.673+0000 7f9a56ffd640 1 --2- 172.21.15.107:0/636448272 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d89e09a280 0x55d89e09c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:48.674 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.675+0000 7f9a56ffd640 1 --2- 172.21.15.107:0/636448272 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d89e09a280 0x55d89e09c6d0 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=404d0f4c851bda2b server_cookie=bcf2ac814c6ef0f0 in_seq=0 out_seq=0 2024-05-09T08:46:48.674 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.675+0000 7f9a36ffd640 10 client.28061 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.678+0000 7f9a36ffd640 1 -- 172.21.15.107:0/636448272 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d89dfebdc0 con 0x55d89e09a280 2024-05-09T08:46:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.678+0000 7f9a36ffd640 10 client.28061 handle_command_reply: tid=0 2024-05-09T08:46:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.682+0000 7f9a60813740 1 client.28061 shutdown 2024-05-09T08:46:48.690 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.687+0000 7f9a60813740 1 -- 172.21.15.107:0/636448272 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 msgr2=0x55d89e087b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:48.690 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.687+0000 7f9a60813740 1 --2- 172.21.15.107:0/636448272 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e087b00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9a5810eb50 tx=0x7f9a4800ea60 comp rx=0 tx=0).stop 2024-05-09T08:46:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.693+0000 7f9a60813740 1 -- 172.21.15.107:0/636448272 shutdown_connections 2024-05-09T08:46:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.693+0000 7f9a60813740 1 --2- 172.21.15.107:0/636448272 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d89e09a280 0x55d89e09c6d0 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.693+0000 7f9a60813740 1 --2- 172.21.15.107:0/636448272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e08a610 0x55d89e088040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.693+0000 7f9a60813740 1 --2- 172.21.15.107:0/636448272 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e087b00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.693+0000 7f9a60813740 1 --2- 172.21.15.107:0/636448272 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 0x55d89e0875c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.693+0000 7f9a60813740 1 -- 172.21.15.107:0/636448272 >> 172.21.15.107:0/636448272 conn(0x55d89e07b940 msgr2=0x55d89e0816f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.694+0000 7f9a36ffd640 0 client.28061 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.694+0000 7f9a60813740 1 -- 172.21.15.107:0/636448272 shutdown_connections 2024-05-09T08:46:48.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.697+0000 7f9a60813740 1 -- 172.21.15.107:0/636448272 wait complete. 2024-05-09T08:46:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.697+0000 7f9a60813740 20 client.28061 trim_cache size 0 max 16384 2024-05-09T08:46:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.745+0000 7f9a60813740 1 Processor -- start 2024-05-09T08:46:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.745+0000 7f9a60813740 1 -- start start 2024-05-09T08:46:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.745+0000 7f9a60813740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e07f430 0x55d89e089cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.745+0000 7f9a60813740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e081df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.746+0000 7f9a60813740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e08a610 0x55d89e087e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.746+0000 7f9a60813740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9a581d9d60 con 0x55d89e07f430 2024-05-09T08:46:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.746+0000 7f9a60813740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9a581d7630 con 0x55d89e07fd50 2024-05-09T08:46:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.746+0000 7f9a60813740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9a581d54a0 con 0x55d89e08a610 2024-05-09T08:46:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.746+0000 7f9a567fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e081df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.746+0000 7f9a567fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e081df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44408/0 (socket says 172.21.15.107:44408) 2024-05-09T08:46:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.746+0000 7f9a567fc640 1 -- 172.21.15.107:0/4014563537 learned_addr learned my addr 172.21.15.107:0/4014563537 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.746+0000 7f9a577fe640 1 --2- 172.21.15.107:0/4014563537 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e08a610 0x55d89e087e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.747+0000 7f9a567fc640 1 -- 172.21.15.107:0/4014563537 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e08a610 msgr2=0x55d89e087e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.747+0000 7f9a567fc640 1 --2- 172.21.15.107:0/4014563537 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e08a610 0x55d89e087e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.747+0000 7f9a567fc640 1 -- 172.21.15.107:0/4014563537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e07f430 msgr2=0x55d89e089cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.747+0000 7f9a567fc640 1 --2- 172.21.15.107:0/4014563537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e07f430 0x55d89e089cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.747+0000 7f9a567fc640 1 -- 172.21.15.107:0/4014563537 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d89e086570 con 0x55d89e07fd50 2024-05-09T08:46:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.748+0000 7f9a567fc640 1 --2- 172.21.15.107:0/4014563537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e081df0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9a48017490 tx=0x7f9a48009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1a74d2736daa9d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.748+0000 7f9a3d7fa640 1 -- 172.21.15.107:0/4014563537 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4802b070 con 0x55d89e07fd50 2024-05-09T08:46:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.748+0000 7f9a3d7fa640 1 -- 172.21.15.107:0/4014563537 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a480044b0 con 0x55d89e07fd50 2024-05-09T08:46:48.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.749+0000 7f9a60813740 1 -- 172.21.15.107:0/4014563537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 msgr2=0x55d89e081df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:48.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.749+0000 7f9a60813740 1 --2- 172.21.15.107:0/4014563537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e081df0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9a48017490 tx=0x7f9a48009680 comp rx=0 tx=0).stop 2024-05-09T08:46:48.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.749+0000 7f9a60813740 1 -- 172.21.15.107:0/4014563537 shutdown_connections 2024-05-09T08:46:48.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.749+0000 7f9a60813740 1 --2- 172.21.15.107:0/4014563537 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e08a610 0x55d89e087e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.749+0000 7f9a60813740 1 --2- 172.21.15.107:0/4014563537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e07fd50 0x55d89e081df0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.749+0000 7f9a60813740 1 --2- 172.21.15.107:0/4014563537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e07f430 0x55d89e089cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.749+0000 7f9a60813740 1 -- 172.21.15.107:0/4014563537 >> 172.21.15.107:0/4014563537 conn(0x55d89e07b940 msgr2=0x55d89e07aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:48.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.749+0000 7f9a60813740 1 -- 172.21.15.107:0/4014563537 shutdown_connections 2024-05-09T08:46:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.750+0000 7f9a60813740 1 -- 172.21.15.107:0/4014563537 wait complete. 2024-05-09T08:46:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.750+0000 7f9a60813740 1 -- start start 2024-05-09T08:46:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.750+0000 7f9a60813740 1 Processor -- start 2024-05-09T08:46:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.751+0000 7f9a60813740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 0x55d89e0923d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.751+0000 7f9a60813740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e08a610 0x55d89e092910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.751+0000 7f9a56ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 0x55d89e0923d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.751+0000 7f9a56ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 0x55d89e0923d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:45212/0 (socket says 172.21.15.107:45212) 2024-05-09T08:46:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.751+0000 7f9a56ffd640 1 -- 172.21.15.107:0/3345951201 learned_addr learned my addr 172.21.15.107:0/3345951201 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:46:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.751+0000 7f9a60813740 1 --2- 172.21.15.107:0/3345951201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e083c60 0x55d89e0840e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.751+0000 7f9a60813740 1 -- 172.21.15.107:0/3345951201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9a581d5af0 con 0x55d89e08a610 2024-05-09T08:46:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.752+0000 7f9a577fe640 1 --2- 172.21.15.107:0/3345951201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e083c60 0x55d89e0840e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.752+0000 7f9a60813740 1 -- 172.21.15.107:0/3345951201 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9a581d6f60 con 0x55d89e083c60 2024-05-09T08:46:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.752+0000 7f9a60813740 1 -- 172.21.15.107:0/3345951201 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9a581d61c0 con 0x55d89e07f430 2024-05-09T08:46:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.752+0000 7f9a60813740 10 client.0 _ll_register_callbacks cb 0x7f9a480094a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:46:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.755+0000 7f9a567fc640 1 --2- 172.21.15.107:0/3345951201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e08a610 0x55d89e092910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.760+0000 7f9a56ffd640 1 -- 172.21.15.107:0/3345951201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e083c60 msgr2=0x55d89e0840e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.760+0000 7f9a56ffd640 1 --2- 172.21.15.107:0/3345951201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e083c60 0x55d89e0840e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.760+0000 7f9a56ffd640 1 -- 172.21.15.107:0/3345951201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e08a610 msgr2=0x55d89e092910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:46:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.760+0000 7f9a56ffd640 1 --2- 172.21.15.107:0/3345951201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e08a610 0x55d89e092910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.760+0000 7f9a56ffd640 1 -- 172.21.15.107:0/3345951201 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d89e084820 con 0x55d89e07f430 2024-05-09T08:46:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.761+0000 7f9a56ffd640 1 --2- 172.21.15.107:0/3345951201 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 0x55d89e0923d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9a50034040 tx=0x7f9a5002ad90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdde4788a0de988d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:46:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.761+0000 7f9a36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:46:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.762+0000 7f9a36ffd640 1 -- 172.21.15.107:0/3345951201 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a500b8e10 con 0x55d89e07f430 2024-05-09T08:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.762+0000 7f9a60813740 1 -- 172.21.15.107:0/3345951201 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d89df698e0 con 0x55d89e07f430 2024-05-09T08:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.763+0000 7f9a36ffd640 1 -- 172.21.15.107:0/3345951201 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f9a50030a30 con 0x55d89e07f430 2024-05-09T08:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.763+0000 7f9a36ffd640 1 -- 172.21.15.107:0/3345951201 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9a50030c90 con 0x55d89e07f430 2024-05-09T08:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.763+0000 7f9a60813740 10 client.28285 fetch_fsmap learned FSMap version 27 2024-05-09T08:46:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.763+0000 7f9a60813740 1 -- 172.21.15.107:0/3345951201 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d89e1007e0 con 0x55d89e07f430 2024-05-09T08:46:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.763+0000 7f9a36ffd640 1 -- 172.21.15.107:0/3345951201 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55d89df698e0 con 0x55d89e07f430 2024-05-09T08:46:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.764+0000 7f9a36ffd640 1 -- 172.21.15.107:0/3345951201 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f9a500b91e0 con 0x55d89e07f430 2024-05-09T08:46:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.765+0000 7f9a60813740 10 client.28285 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:46:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.765+0000 7f9a60813740 10 client.28285 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:46:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.765+0000 7f9a60813740 20 client.28285 populate_metadata read hostname 'smithi107' 2024-05-09T08:46:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.765+0000 7f9a60813740 1 --2- 172.21.15.107:0/3345951201 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d89e100f80 0x55d89e1033a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:46:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.765+0000 7f9a60813740 4 client.28285 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:46:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.765+0000 7f9a60813740 1 -- 172.21.15.107:0/3345951201 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d89dfb5870 con 0x55d89e100f80 2024-05-09T08:46:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.765+0000 7f9a567fc640 1 --2- 172.21.15.107:0/3345951201 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d89e100f80 0x55d89e1033a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:46:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.768+0000 7f9a567fc640 1 --2- 172.21.15.107:0/3345951201 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d89e100f80 0x55d89e1033a0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7884c75ac88f380 server_cookie=27d192a6c0a043c5 in_seq=0 out_seq=0 2024-05-09T08:46:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:48.768+0000 7f9a36ffd640 10 client.28285 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:50.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:50 smithi107 ceph-mon[30122]: pgmap v421: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 12 MiB/s wr, 3.01k op/s 2024-05-09T08:46:50.552 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.551+0000 7f9a36ffd640 1 -- 172.21.15.107:0/3345951201 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d89dfb5870 con 0x55d89e100f80 2024-05-09T08:46:50.552 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.551+0000 7f9a36ffd640 10 client.28285 handle_command_reply: tid=0 2024-05-09T08:46:50.552 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.553+0000 7f9a60813740 1 client.28285 shutdown 2024-05-09T08:46:50.555 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.555+0000 7f9a60813740 1 -- 172.21.15.107:0/3345951201 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 msgr2=0x55d89e0923d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:50.556 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.555+0000 7f9a60813740 1 --2- 172.21.15.107:0/3345951201 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 0x55d89e0923d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9a50034040 tx=0x7f9a5002ad90 comp rx=0 tx=0).stop 2024-05-09T08:46:50.556 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.555+0000 7f9a60813740 1 -- 172.21.15.107:0/3345951201 shutdown_connections 2024-05-09T08:46:50.556 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.555+0000 7f9a60813740 1 --2- 172.21.15.107:0/3345951201 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d89e100f80 0x55d89e1033a0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:50.556 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.555+0000 7f9a60813740 1 --2- 172.21.15.107:0/3345951201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d89e083c60 0x55d89e0840e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:50.556 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.555+0000 7f9a60813740 1 --2- 172.21.15.107:0/3345951201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d89e08a610 0x55d89e092910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:50.556 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.555+0000 7f9a60813740 1 --2- 172.21.15.107:0/3345951201 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d89e07f430 0x55d89e0923d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:50.556 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.555+0000 7f9a60813740 1 -- 172.21.15.107:0/3345951201 >> 172.21.15.107:0/3345951201 conn(0x55d89e07b940 msgr2=0x55d89e088c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:50.556 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.556+0000 7f9a36ffd640 0 client.28285 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:46:50.556 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.557+0000 7f9a60813740 1 -- 172.21.15.107:0/3345951201 shutdown_connections 2024-05-09T08:46:50.556 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.557+0000 7f9a60813740 1 -- 172.21.15.107:0/3345951201 wait complete. 2024-05-09T08:46:50.557 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.557+0000 7f9a60813740 20 client.28285 trim_cache size 0 max 16384 2024-05-09T08:46:50.558 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:46:50.558 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:46:50.559 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:46:50.559 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:46:50.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.562+0000 7f9a5da2a640 1 -- 172.21.15.107:0/3516209040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9a400a2fe0 msgr2=0x7f9a400a54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:50.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.562+0000 7f9a5da2a640 1 --2- 172.21.15.107:0/3516209040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9a400a2fe0 0x7f9a400a54a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c00bb30 tx=0x7f9a4c008040 comp rx=0 tx=0).stop 2024-05-09T08:46:50.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.562+0000 7f9a5da2a640 1 -- 172.21.15.107:0/3516209040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a5810d190 msgr2=0x7f9a58079a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:46:50.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.562+0000 7f9a5da2a640 1 --2- 172.21.15.107:0/3516209040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a5810d190 0x7f9a58079a70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9a50006010 tx=0x7f9a50003b80 comp rx=0 tx=0).stop 2024-05-09T08:46:50.566 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.565+0000 7f9a5da2a640 1 -- 172.21.15.107:0/3516209040 shutdown_connections 2024-05-09T08:46:50.566 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.565+0000 7f9a5da2a640 1 --2- 172.21.15.107:0/3516209040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9a400a2fe0 0x7f9a400a54a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:50.566 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.565+0000 7f9a5da2a640 1 --2- 172.21.15.107:0/3516209040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a58074250 0x7f9a580746b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:50.566 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.565+0000 7f9a5da2a640 1 --2- 172.21.15.107:0/3516209040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a5810dd50 0x7f9a58073cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:50.566 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.565+0000 7f9a5da2a640 1 --2- 172.21.15.107:0/3516209040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a5810d190 0x7f9a58079a70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:46:50.566 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.565+0000 7f9a5da2a640 1 -- 172.21.15.107:0/3516209040 >> 172.21.15.107:0/3516209040 conn(0x7f9a5806c900 msgr2=0x7f9a5807d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:46:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.569+0000 7f9a5da2a640 1 -- 172.21.15.107:0/3516209040 shutdown_connections 2024-05-09T08:46:50.569 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:46:50.570+0000 7f9a5da2a640 1 -- 172.21.15.107:0/3516209040 wait complete. 2024-05-09T08:46:50.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:50 smithi157 ceph-mon[33614]: pgmap v421: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 12 MiB/s wr, 3.01k op/s 2024-05-09T08:46:50.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:50 smithi110 ceph-mon[33622]: pgmap v421: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 12 MiB/s wr, 3.01k op/s 2024-05-09T08:46:52.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:52 smithi107 ceph-mon[30122]: pgmap v422: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 8.6 MiB/s wr, 2.16k op/s 2024-05-09T08:46:52.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:52 smithi157 ceph-mon[33614]: pgmap v422: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 8.6 MiB/s wr, 2.16k op/s 2024-05-09T08:46:52.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:52 smithi110 ceph-mon[33622]: pgmap v422: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 8.6 MiB/s wr, 2.16k op/s 2024-05-09T08:46:54.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:54 smithi107 ceph-mon[30122]: pgmap v423: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 12 MiB/s wr, 2.93k op/s 2024-05-09T08:46:54.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:54 smithi157 ceph-mon[33614]: pgmap v423: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 12 MiB/s wr, 2.93k op/s 2024-05-09T08:46:54.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:54 smithi110 ceph-mon[33622]: pgmap v423: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 12 MiB/s wr, 2.93k op/s 2024-05-09T08:46:56.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:56 smithi107 ceph-mon[30122]: pgmap v424: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 8.5 MiB/s wr, 2.16k op/s 2024-05-09T08:46:56.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:56 smithi157 ceph-mon[33614]: pgmap v424: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 8.5 MiB/s wr, 2.16k op/s 2024-05-09T08:46:56.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:56 smithi110 ceph-mon[33622]: pgmap v424: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 8.5 MiB/s wr, 2.16k op/s 2024-05-09T08:46:58.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:46:58 smithi157 ceph-mon[33614]: pgmap v425: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.6 MiB/s wr, 2.16k op/s 2024-05-09T08:46:58.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:46:58 smithi110 ceph-mon[33622]: pgmap v425: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.6 MiB/s wr, 2.16k op/s 2024-05-09T08:46:58.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:46:58 smithi107 ceph-mon[30122]: pgmap v425: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.6 MiB/s wr, 2.16k op/s 2024-05-09T08:47:00.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:00 smithi157 ceph-mon[33614]: pgmap v426: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 12 MiB/s wr, 3.00k op/s 2024-05-09T08:47:00.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:00 smithi110 ceph-mon[33622]: pgmap v426: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 12 MiB/s wr, 3.00k op/s 2024-05-09T08:47:00.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:00 smithi107 ceph-mon[30122]: pgmap v426: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 12 MiB/s wr, 3.00k op/s 2024-05-09T08:47:02.324 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:47:02.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:47:02.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.500+0000 7fe48b9e2640 1 -- 172.21.15.107:0/3201728373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe48411b010 msgr2=0x7fe48411d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:02.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.500+0000 7fe48b9e2640 1 --2- 172.21.15.107:0/3201728373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe48411b010 0x7fe48411d400 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe480009ca0 tx=0x7fe48002a8b0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.503+0000 7fe48b9e2640 1 -- 172.21.15.107:0/3201728373 shutdown_connections 2024-05-09T08:47:02.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.503+0000 7fe48b9e2640 1 --2- 172.21.15.107:0/3201728373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe48411b010 0x7fe48411d400 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.503+0000 7fe48b9e2640 1 --2- 172.21.15.107:0/3201728373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48410e0b0 0x7fe48410e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.503+0000 7fe48b9e2640 1 --2- 172.21.15.107:0/3201728373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe484110080 0x7fe48410db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.503+0000 7fe48b9e2640 1 -- 172.21.15.107:0/3201728373 >> 172.21.15.107:0/3201728373 conn(0x7fe48406c940 msgr2=0x7fe48406cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:02.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.507+0000 7fe48b9e2640 1 -- 172.21.15.107:0/3201728373 shutdown_connections 2024-05-09T08:47:02.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.511+0000 7fe48b9e2640 1 -- 172.21.15.107:0/3201728373 wait complete. 2024-05-09T08:47:02.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.512+0000 7fe48b9e2640 1 Processor -- start 2024-05-09T08:47:02.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.514+0000 7fe48b9e2640 1 -- start start 2024-05-09T08:47:02.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.514+0000 7fe48b9e2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48410e0b0 0x7fe4841185e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.514+0000 7fe48b9e2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe484110080 0x7fe484111700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.514+0000 7fe48b9e2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe48411b010 0x7fe484111c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.514+0000 7fe48b9e2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe4841208f0 con 0x7fe48411b010 2024-05-09T08:47:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.514+0000 7fe48b9e2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe484120770 con 0x7fe48410e0b0 2024-05-09T08:47:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.514+0000 7fe48b9e2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe484120a70 con 0x7fe484110080 2024-05-09T08:47:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.518+0000 7fe489f7f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe48411b010 0x7fe484111c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.518+0000 7fe489f7f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe48411b010 0x7fe484111c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41906/0 (socket says 172.21.15.107:41906) 2024-05-09T08:47:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.518+0000 7fe489f7f640 1 -- 172.21.15.107:0/1802218186 learned_addr learned my addr 172.21.15.107:0/1802218186 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.518+0000 7fe489f7f640 1 -- 172.21.15.107:0/1802218186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe484110080 msgr2=0x7fe484111700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:47:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.518+0000 7fe489f7f640 1 --2- 172.21.15.107:0/1802218186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe484110080 0x7fe484111700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.518+0000 7fe489f7f640 1 -- 172.21.15.107:0/1802218186 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48410e0b0 msgr2=0x7fe4841185e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:47:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.518+0000 7fe489f7f640 1 --2- 172.21.15.107:0/1802218186 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48410e0b0 0x7fe4841185e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.518+0000 7fe489f7f640 1 -- 172.21.15.107:0/1802218186 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe484112200 con 0x7fe48411b010 2024-05-09T08:47:02.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.519+0000 7fe489f7f640 1 --2- 172.21.15.107:0/1802218186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe48411b010 0x7fe484111c40 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fe4800098a0 tx=0x7fe480030fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:02.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.520+0000 7fe47a7fc640 1 -- 172.21.15.107:0/1802218186 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe480004040 con 0x7fe48411b010 2024-05-09T08:47:02.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.520+0000 7fe47a7fc640 1 -- 172.21.15.107:0/1802218186 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe480040070 con 0x7fe48411b010 2024-05-09T08:47:02.520 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.520+0000 7fe47a7fc640 1 -- 172.21.15.107:0/1802218186 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe480045bb0 con 0x7fe48411b010 2024-05-09T08:47:02.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.521+0000 7fe48b9e2640 1 -- 172.21.15.107:0/1802218186 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe484112490 con 0x7fe48411b010 2024-05-09T08:47:02.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.521+0000 7fe48b9e2640 1 -- 172.21.15.107:0/1802218186 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4841163a0 con 0x7fe48411b010 2024-05-09T08:47:02.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.525+0000 7fe48e7c5740 1 Processor -- start 2024-05-09T08:47:02.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.525+0000 7fe48e7c5740 1 -- start start 2024-05-09T08:47:02.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.525+0000 7fe48e7c5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267dc280 0x563c267dc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.525+0000 7fe48e7c5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267e6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.526+0000 7fe48e7c5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 0x563c267e9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.526+0000 7fe48e7c5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563c26741d00 con 0x563c267dc280 2024-05-09T08:47:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.526+0000 7fe48e7c5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563c266e94c0 con 0x563c267dcba0 2024-05-09T08:47:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.526+0000 7fe48e7c5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563c2673dcc0 con 0x563c267e7270 2024-05-09T08:47:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.526+0000 7fe48977e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267e6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.526+0000 7fe48977e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267e6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:45576/0 (socket says 172.21.15.107:45576) 2024-05-09T08:47:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.526+0000 7fe48977e640 1 -- 172.21.15.107:0/561106240 learned_addr learned my addr 172.21.15.107:0/561106240 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.526+0000 7fe488f7d640 1 --2- 172.21.15.107:0/561106240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 0x563c267e9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:02.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.527+0000 7fe47a7fc640 1 -- 172.21.15.107:0/1802218186 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe4800041e0 con 0x7fe48411b010 2024-05-09T08:47:02.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.528+0000 7fe47a7fc640 1 --2- 172.21.15.107:0/1802218186 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4700a2ed0 0x7fe4700a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.529+0000 7fe47a7fc640 1 -- 172.21.15.107:0/1802218186 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7fe4800f0b10 con 0x7fe48411b010 2024-05-09T08:47:02.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.529+0000 7fe489f7f640 1 --2- 172.21.15.107:0/1802218186 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4700a2ed0 0x7fe4700a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:02.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.529+0000 7fe488f7d640 1 -- 172.21.15.107:0/561106240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 msgr2=0x563c267e6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:02.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.529+0000 7fe488f7d640 1 --2- 172.21.15.107:0/561106240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267e6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.529+0000 7fe488f7d640 1 -- 172.21.15.107:0/561106240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267dc280 msgr2=0x563c267dc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:02.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.529+0000 7fe488f7d640 1 --2- 172.21.15.107:0/561106240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267dc280 0x563c267dc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.529+0000 7fe488f7d640 1 -- 172.21.15.107:0/561106240 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c2651f7d0 con 0x563c267e7270 2024-05-09T08:47:02.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.530+0000 7fe488f7d640 1 --2- 172.21.15.107:0/561106240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 0x563c267e9660 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe484112eb0 tx=0x7fe47c00df50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=935953e54dc31387 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:02.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.530+0000 7fe489f7f640 1 --2- 172.21.15.107:0/1802218186 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4700a2ed0 0x7fe4700a5390 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe4800edf10 tx=0x7fe48003e040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:02.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.533+0000 7fe46f7fe640 1 -- 172.21.15.107:0/561106240 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c00fd20 con 0x563c267e7270 2024-05-09T08:47:02.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.533+0000 7fe46f7fe640 1 -- 172.21.15.107:0/561106240 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe47c0118f0 con 0x563c267e7270 2024-05-09T08:47:02.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.533+0000 7fe46f7fe640 1 -- 172.21.15.107:0/561106240 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c019920 con 0x563c267e7270 2024-05-09T08:47:02.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.534+0000 7fe48e7c5740 1 -- 172.21.15.107:0/561106240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 msgr2=0x563c267e9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:02.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.534+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/561106240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 0x563c267e9660 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe484112eb0 tx=0x7fe47c00df50 comp rx=0 tx=0).stop 2024-05-09T08:47:02.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.536+0000 7fe48e7c5740 1 -- 172.21.15.107:0/561106240 shutdown_connections 2024-05-09T08:47:02.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.536+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/561106240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 0x563c267e9660 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.536+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/561106240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267e6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.536+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/561106240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267dc280 0x563c267dc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.536+0000 7fe48e7c5740 1 -- 172.21.15.107:0/561106240 >> 172.21.15.107:0/561106240 conn(0x563c267d85a0 msgr2=0x563c267d8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:02.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.539+0000 7fe48e7c5740 1 -- 172.21.15.107:0/561106240 shutdown_connections 2024-05-09T08:47:02.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.539+0000 7fe48e7c5740 1 -- 172.21.15.107:0/561106240 wait complete. 2024-05-09T08:47:02.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.540+0000 7fe48e7c5740 1 -- start start 2024-05-09T08:47:02.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.540+0000 7fe48e7c5740 1 Processor -- start 2024-05-09T08:47:02.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.541+0000 7fe48e7c5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267dc280 0x563c267e4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.541+0000 7fe48e7c5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267e4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.541+0000 7fe48e7c5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267e7270 0x563c267e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.541+0000 7fe48e7c5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563c267ea060 con 0x563c267e7270 2024-05-09T08:47:02.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.541+0000 7fe48e7c5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563c266fcb90 con 0x563c267dcba0 2024-05-09T08:47:02.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.541+0000 7fe48e7c5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563c267ea1e0 con 0x563c267dc280 2024-05-09T08:47:02.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.541+0000 7fe48e7c5740 10 client.0 _ll_register_callbacks cb 0x563c26738da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:02.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.541+0000 7fe488f7d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267dc280 0x563c267e4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:02.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.541+0000 7fe488f7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267e7270 0x563c267e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:02.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.541+0000 7fe488f7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267e7270 0x563c267e4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41918/0 (socket says 172.21.15.107:41918) 2024-05-09T08:47:02.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.541+0000 7fe488f7d640 1 -- 172.21.15.107:0/2574537757 learned_addr learned my addr 172.21.15.107:0/2574537757 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:02.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.541+0000 7fe48977e640 1 --2- 172.21.15.107:0/2574537757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267e4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:02.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.542+0000 7fe488f7d640 1 -- 172.21.15.107:0/2574537757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267dc280 msgr2=0x563c267e4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:02.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.542+0000 7fe488f7d640 1 --2- 172.21.15.107:0/2574537757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267dc280 0x563c267e4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.542+0000 7fe488f7d640 1 -- 172.21.15.107:0/2574537757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 msgr2=0x563c267e4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:02.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.542+0000 7fe488f7d640 1 --2- 172.21.15.107:0/2574537757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267e4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.542+0000 7fe488f7d640 1 -- 172.21.15.107:0/2574537757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c264553f0 con 0x563c267e7270 2024-05-09T08:47:02.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.543+0000 7fe488f7d640 1 --2- 172.21.15.107:0/2574537757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267e7270 0x563c267e4d90 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0057c0 tx=0x7fe47c00df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd66325eab776fe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:02.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.543+0000 7fe46cff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:47:02.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.543+0000 7fe46cff9640 1 -- 172.21.15.107:0/2574537757 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c0195a0 con 0x563c267e7270 2024-05-09T08:47:02.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.543+0000 7fe46cff9640 1 -- 172.21.15.107:0/2574537757 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7fe47c019740 con 0x563c267e7270 2024-05-09T08:47:02.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.544+0000 7fe48e7c5740 1 -- 172.21.15.107:0/2574537757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe47c0118f0 con 0x563c267e7270 2024-05-09T08:47:02.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.544+0000 7fe46cff9640 1 -- 172.21.15.107:0/2574537757 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe47c019940 con 0x563c267e7270 2024-05-09T08:47:02.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.544+0000 7fe48e7c5740 10 client.19119 fetch_fsmap learned FSMap version 27 2024-05-09T08:47:02.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.544+0000 7fe48e7c5740 1 -- 172.21.15.107:0/2574537757 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563c264150f0 con 0x563c267e7270 2024-05-09T08:47:02.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.545+0000 7fe46cff9640 1 -- 172.21.15.107:0/2574537757 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe47c0118f0 con 0x563c267e7270 2024-05-09T08:47:02.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.545+0000 7fe46cff9640 1 -- 172.21.15.107:0/2574537757 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe47c027040 con 0x563c267e7270 2024-05-09T08:47:02.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.545+0000 7fe48e7c5740 10 client.19119 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:47:02.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.545+0000 7fe48e7c5740 10 client.19119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:47:02.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.545+0000 7fe48e7c5740 20 client.19119 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:02.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.545+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/2574537757 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563c267f73e0 0x563c267f97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.545+0000 7fe48e7c5740 4 client.19119 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:47:02.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.545+0000 7fe48e7c5740 1 -- 172.21.15.107:0/2574537757 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c2674f430 con 0x563c267f73e0 2024-05-09T08:47:02.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.549+0000 7fe48977e640 1 --2- 172.21.15.107:0/2574537757 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563c267f73e0 0x563c267f97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:02.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.553+0000 7fe48977e640 1 --2- 172.21.15.107:0/2574537757 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563c267f73e0 0x563c267f97d0 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a31de837748ef37 server_cookie=1300ab5d55e2b6ac in_seq=0 out_seq=0 2024-05-09T08:47:02.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.555+0000 7fe46cff9640 10 client.19119 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:02.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.555+0000 7fe46cff9640 1 -- 172.21.15.107:0/2574537757 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563c2674f430 con 0x563c267f73e0 2024-05-09T08:47:02.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.555+0000 7fe46cff9640 10 client.19119 handle_command_reply: tid=0 2024-05-09T08:47:02.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.555+0000 7fe48e7c5740 1 client.19119 shutdown 2024-05-09T08:47:02.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.556+0000 7fe48e7c5740 1 -- 172.21.15.107:0/2574537757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267e7270 msgr2=0x563c267e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:02.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.556+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/2574537757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267e7270 0x563c267e4d90 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0057c0 tx=0x7fe47c00df50 comp rx=0 tx=0).stop 2024-05-09T08:47:02.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.560+0000 7fe48e7c5740 1 -- 172.21.15.107:0/2574537757 shutdown_connections 2024-05-09T08:47:02.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.560+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/2574537757 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563c267f73e0 0x563c267f97d0 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.560+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/2574537757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267e7270 0x563c267e4d90 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.560+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/2574537757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267e4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.560+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/2574537757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267dc280 0x563c267e4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.560+0000 7fe48e7c5740 1 -- 172.21.15.107:0/2574537757 >> 172.21.15.107:0/2574537757 conn(0x563c267d85a0 msgr2=0x563c267d77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:02.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.561+0000 7fe46cff9640 0 client.19119 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:02.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.564+0000 7fe48e7c5740 1 -- 172.21.15.107:0/2574537757 shutdown_connections 2024-05-09T08:47:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.564+0000 7fe48e7c5740 1 -- 172.21.15.107:0/2574537757 wait complete. 2024-05-09T08:47:02.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.566+0000 7fe48e7c5740 20 client.19119 trim_cache size 0 max 16384 2024-05-09T08:47:02.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:02 smithi157 ceph-mon[33614]: pgmap v427: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.6 MiB/s wr, 2.18k op/s 2024-05-09T08:47:02.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.613+0000 7fe48e7c5740 1 Processor -- start 2024-05-09T08:47:02.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.613+0000 7fe48e7c5740 1 -- start start 2024-05-09T08:47:02.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.613+0000 7fe48e7c5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267dc280 0x563c267d7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.614+0000 7fe48e7c5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267e4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.614+0000 7fe48e7c5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 0x563c267e6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.614+0000 7fe48e7c5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe4841c8c40 con 0x563c267dc280 2024-05-09T08:47:02.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.614+0000 7fe48e7c5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe4841c6510 con 0x563c267dcba0 2024-05-09T08:47:02.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.614+0000 7fe488f7d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 0x563c267e6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:02.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.614+0000 7fe488f7d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 0x563c267e6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:41342/0 (socket says 172.21.15.107:41342) 2024-05-09T08:47:02.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.614+0000 7fe488f7d640 1 -- 172.21.15.107:0/1743565423 learned_addr learned my addr 172.21.15.107:0/1743565423 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:02.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.614+0000 7fe48e7c5740 1 -- 172.21.15.107:0/1743565423 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe4841c4380 con 0x563c267e7270 2024-05-09T08:47:02.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.614+0000 7fe488f7d640 1 --2- 172.21.15.107:0/1743565423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267dc280 0x563c267d7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:02.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.615+0000 7fe488f7d640 1 -- 172.21.15.107:0/1743565423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 msgr2=0x563c267e4bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:47:02.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.615+0000 7fe488f7d640 1 --2- 172.21.15.107:0/1743565423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267e4bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.615+0000 7fe488f7d640 1 -- 172.21.15.107:0/1743565423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267dc280 msgr2=0x563c267d7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:02.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.615+0000 7fe488f7d640 1 --2- 172.21.15.107:0/1743565423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267dc280 0x563c267d7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.615+0000 7fe488f7d640 1 -- 172.21.15.107:0/1743565423 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c267dd0f0 con 0x563c267e7270 2024-05-09T08:47:02.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.615+0000 7fe488f7d640 1 --2- 172.21.15.107:0/1743565423 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 0x563c267e6250 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe47c029d00 tx=0x7fe47c029d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caee65d32707c8a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:02.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.615+0000 7fe46effd640 1 -- 172.21.15.107:0/1743565423 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c011440 con 0x563c267e7270 2024-05-09T08:47:02.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.616+0000 7fe46effd640 1 -- 172.21.15.107:0/1743565423 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe47c000c30 con 0x563c267e7270 2024-05-09T08:47:02.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.616+0000 7fe46effd640 1 -- 172.21.15.107:0/1743565423 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c019720 con 0x563c267e7270 2024-05-09T08:47:02.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.617+0000 7fe48e7c5740 1 -- 172.21.15.107:0/1743565423 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 msgr2=0x563c267e6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:02.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.617+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/1743565423 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 0x563c267e6250 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe47c029d00 tx=0x7fe47c029d80 comp rx=0 tx=0).stop 2024-05-09T08:47:02.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.617+0000 7fe48e7c5740 1 -- 172.21.15.107:0/1743565423 shutdown_connections 2024-05-09T08:47:02.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.617+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/1743565423 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 0x563c267e6250 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.617+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/1743565423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267e4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.617+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/1743565423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267dc280 0x563c267d7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.617+0000 7fe48e7c5740 1 -- 172.21.15.107:0/1743565423 >> 172.21.15.107:0/1743565423 conn(0x563c267d85a0 msgr2=0x563c267e5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.617+0000 7fe48e7c5740 1 -- 172.21.15.107:0/1743565423 shutdown_connections 2024-05-09T08:47:02.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.617+0000 7fe48e7c5740 1 -- 172.21.15.107:0/1743565423 wait complete. 2024-05-09T08:47:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.617+0000 7fe48e7c5740 1 -- start start 2024-05-09T08:47:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.618+0000 7fe48e7c5740 1 Processor -- start 2024-05-09T08:47:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.618+0000 7fe48e7c5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267dc280 0x563c267fc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.618+0000 7fe48e7c5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267f9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.618+0000 7fe48e7c5740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 0x563c267f9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.618+0000 7fe48e7c5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe4841c49d0 con 0x563c267dc280 2024-05-09T08:47:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.618+0000 7fe48e7c5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe4841c5e40 con 0x563c267dcba0 2024-05-09T08:47:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.618+0000 7fe48e7c5740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe4841c50a0 con 0x563c267e7270 2024-05-09T08:47:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.618+0000 7fe48e7c5740 10 client.0 _ll_register_callbacks cb 0x7fe474009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.618+0000 7fe488f7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267dc280 0x563c267fc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.618+0000 7fe488f7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267dc280 0x563c267fc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41942/0 (socket says 172.21.15.107:41942) 2024-05-09T08:47:02.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.618+0000 7fe488f7d640 1 -- 172.21.15.107:0/936319606 learned_addr learned my addr 172.21.15.107:0/936319606 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:02.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.619+0000 7fe488f7d640 1 --2- 172.21.15.107:0/936319606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 0x563c267f9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.619+0000 7fe48977e640 1 --2- 172.21.15.107:0/936319606 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267f9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.619+0000 7fe488f7d640 1 -- 172.21.15.107:0/936319606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 msgr2=0x563c267f9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.619+0000 7fe488f7d640 1 --2- 172.21.15.107:0/936319606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 0x563c267f9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.619+0000 7fe488f7d640 1 -- 172.21.15.107:0/936319606 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 msgr2=0x563c267f9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.619+0000 7fe488f7d640 1 --2- 172.21.15.107:0/936319606 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267f9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.619+0000 7fe488f7d640 1 -- 172.21.15.107:0/936319606 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c267fa4b0 con 0x563c267dc280 2024-05-09T08:47:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.619+0000 7fe488f7d640 1 --2- 172.21.15.107:0/936319606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267dc280 0x563c267fc5f0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0118b0 tx=0x7fe47c02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f9077232cf3c02 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.619+0000 7fe46cff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:47:02.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.620+0000 7fe46cff9640 1 -- 172.21.15.107:0/936319606 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c00f860 con 0x563c267dc280 2024-05-09T08:47:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.620+0000 7fe46cff9640 1 -- 172.21.15.107:0/936319606 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7fe47c029460 con 0x563c267dc280 2024-05-09T08:47:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.620+0000 7fe48e7c5740 1 -- 172.21.15.107:0/936319606 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe47c000c30 con 0x563c267dc280 2024-05-09T08:47:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.620+0000 7fe46cff9640 1 -- 172.21.15.107:0/936319606 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe47c04b050 con 0x563c267dc280 2024-05-09T08:47:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.620+0000 7fe48e7c5740 10 client.19125 fetch_fsmap learned FSMap version 27 2024-05-09T08:47:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.620+0000 7fe48e7c5740 1 -- 172.21.15.107:0/936319606 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563c2685d3d0 con 0x563c267dc280 2024-05-09T08:47:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.620+0000 7fe48977e640 1 --2- 172.21.15.107:0/936319606 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267f9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:47:02.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.621+0000 7fe46cff9640 1 -- 172.21.15.107:0/936319606 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe47c000c30 con 0x563c267dc280 2024-05-09T08:47:02.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:02 smithi110 ceph-mon[33622]: pgmap v427: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.6 MiB/s wr, 2.18k op/s 2024-05-09T08:47:02.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.622+0000 7fe46cff9640 1 -- 172.21.15.107:0/936319606 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fe47c0053e0 con 0x563c267dc280 2024-05-09T08:47:02.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.622+0000 7fe48e7c5740 10 client.19125 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:47:02.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.622+0000 7fe48e7c5740 10 client.19125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:47:02.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.622+0000 7fe48e7c5740 20 client.19125 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:02.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.622+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/936319606 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563c2685d980 0x563c2685fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:02.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.622+0000 7fe48e7c5740 4 client.19125 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:47:02.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.622+0000 7fe48e7c5740 1 -- 172.21.15.107:0/936319606 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563c26749780 con 0x563c2685d980 2024-05-09T08:47:02.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.622+0000 7fe48977e640 1 --2- 172.21.15.107:0/936319606 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563c2685d980 0x563c2685fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:02.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.623+0000 7fe48977e640 1 --2- 172.21.15.107:0/936319606 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563c2685d980 0x563c2685fda0 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f45af8972f8de72 server_cookie=dd422fcbd39aa86a in_seq=0 out_seq=0 2024-05-09T08:47:02.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.623+0000 7fe46cff9640 10 client.19125 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:02.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.623+0000 7fe46cff9640 1 -- 172.21.15.107:0/936319606 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x563c26749780 con 0x563c2685d980 2024-05-09T08:47:02.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.624+0000 7fe46cff9640 10 client.19125 handle_command_reply: tid=0 2024-05-09T08:47:02.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.624+0000 7fe48e7c5740 1 client.19125 shutdown 2024-05-09T08:47:02.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.624+0000 7fe48e7c5740 1 -- 172.21.15.107:0/936319606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267dc280 msgr2=0x563c267fc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:02.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.624+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/936319606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267dc280 0x563c267fc5f0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0118b0 tx=0x7fe47c02c040 comp rx=0 tx=0).stop 2024-05-09T08:47:02.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.626+0000 7fe48e7c5740 1 -- 172.21.15.107:0/936319606 shutdown_connections 2024-05-09T08:47:02.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.626+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/936319606 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x563c2685d980 0x563c2685fda0 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.626+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/936319606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563c267e7270 0x563c267f9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.626+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/936319606 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c267dcba0 0x563c267f9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.626+0000 7fe48e7c5740 1 --2- 172.21.15.107:0/936319606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563c267dc280 0x563c267fc5f0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.626+0000 7fe48e7c5740 1 -- 172.21.15.107:0/936319606 >> 172.21.15.107:0/936319606 conn(0x563c267d85a0 msgr2=0x563c267e63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:02.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.626+0000 7fe46cff9640 0 client.19125 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:02.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.627+0000 7fe48e7c5740 1 -- 172.21.15.107:0/936319606 shutdown_connections 2024-05-09T08:47:02.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.628+0000 7fe48e7c5740 1 -- 172.21.15.107:0/936319606 wait complete. 2024-05-09T08:47:02.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.628+0000 7fe48e7c5740 20 client.19125 trim_cache size 0 max 16384 2024-05-09T08:47:02.627 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:47:02.627 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "scrub active (12 inodes in the stack)", 2024-05-09T08:47:02.627 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:47:02.628 INFO:teuthology.orchestra.run.smithi107.stdout: "0b0c6d85-4670-442d-96d1-ff7f2c7953f3": { 2024-05-09T08:47:02.628 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:47:02.628 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "0b0c6d85-4670-442d-96d1-ff7f2c7953f3", 2024-05-09T08:47:02.628 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:47:02.628 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:47:02.628 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:47:02.628 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:47:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.630+0000 7fe48b9e2640 1 -- 172.21.15.107:0/1802218186 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4700a2ed0 msgr2=0x7fe4700a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.630+0000 7fe48b9e2640 1 --2- 172.21.15.107:0/1802218186 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4700a2ed0 0x7fe4700a5390 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe4800edf10 tx=0x7fe48003e040 comp rx=0 tx=0).stop 2024-05-09T08:47:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.630+0000 7fe48b9e2640 1 -- 172.21.15.107:0/1802218186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe48411b010 msgr2=0x7fe484111c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.630+0000 7fe48b9e2640 1 --2- 172.21.15.107:0/1802218186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe48411b010 0x7fe484111c40 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fe4800098a0 tx=0x7fe480030fe0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.631+0000 7fe48b9e2640 1 -- 172.21.15.107:0/1802218186 shutdown_connections 2024-05-09T08:47:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.631+0000 7fe48b9e2640 1 --2- 172.21.15.107:0/1802218186 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe4700a2ed0 0x7fe4700a5390 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.631+0000 7fe48b9e2640 1 --2- 172.21.15.107:0/1802218186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe48411b010 0x7fe484111c40 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.631+0000 7fe48b9e2640 1 --2- 172.21.15.107:0/1802218186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe484110080 0x7fe484111700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.631+0000 7fe48b9e2640 1 --2- 172.21.15.107:0/1802218186 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48410e0b0 0x7fe4841185e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:02.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.631+0000 7fe48b9e2640 1 -- 172.21.15.107:0/1802218186 >> 172.21.15.107:0/1802218186 conn(0x7fe48406c940 msgr2=0x7fe484072330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:02.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.634+0000 7fe48b9e2640 1 -- 172.21.15.107:0/1802218186 shutdown_connections 2024-05-09T08:47:02.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:02.634+0000 7fe48b9e2640 1 -- 172.21.15.107:0/1802218186 wait complete. 2024-05-09T08:47:02.655 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:02 smithi107 ceph-mon[30122]: pgmap v427: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.6 MiB/s wr, 2.18k op/s 2024-05-09T08:47:02.656 INFO:tasks.cephfs.filesystem:scrub status for tag:0b0c6d85-4670-442d-96d1-ff7f2c7953f3 - {'path': '/', 'tag': '0b0c6d85-4670-442d-96d1-ff7f2c7953f3', 'options': 'recursive,force'} 2024-05-09T08:47:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.707+0000 7f0e98b10640 1 -- 172.21.15.107:0/1510211485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e94101120 msgr2=0x7f0e9410d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.707+0000 7f0e98b10640 1 --2- 172.21.15.107:0/1510211485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e94101120 0x7f0e9410d5e0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009cd0 tx=0x7f0e8c02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.708+0000 7f0e98b10640 1 -- 172.21.15.107:0/1510211485 shutdown_connections 2024-05-09T08:47:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.708+0000 7f0e98b10640 1 --2- 172.21.15.107:0/1510211485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e94101120 0x7f0e9410d5e0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.708+0000 7f0e98b10640 1 --2- 172.21.15.107:0/1510211485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e94100760 0x7f0e94100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.708+0000 7f0e98b10640 1 --2- 172.21.15.107:0/1510211485 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e94106720 0x7f0e94106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.708+0000 7f0e98b10640 1 -- 172.21.15.107:0/1510211485 >> 172.21.15.107:0/1510211485 conn(0x7f0e940778b0 msgr2=0x7f0e94077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.708+0000 7f0e98b10640 1 -- 172.21.15.107:0/1510211485 shutdown_connections 2024-05-09T08:47:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.708+0000 7f0e98b10640 1 -- 172.21.15.107:0/1510211485 wait complete. 2024-05-09T08:47:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.708+0000 7f0e98b10640 1 Processor -- start 2024-05-09T08:47:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.708+0000 7f0e98b10640 1 -- start start 2024-05-09T08:47:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e98b10640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e94100760 0x7f0e940fd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e98b10640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e94101120 0x7f0e940fdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e98b10640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e94106720 0x7f0e940fe170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e98b10640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0e941109a0 con 0x7f0e94101120 2024-05-09T08:47:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e98b10640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0e94110820 con 0x7f0e94106720 2024-05-09T08:47:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e98b10640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0e94110b20 con 0x7f0e94100760 2024-05-09T08:47:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e91d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e94101120 0x7f0e940fdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e91d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e94101120 0x7f0e940fdc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41956/0 (socket says 172.21.15.107:41956) 2024-05-09T08:47:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e9259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e94100760 0x7f0e940fd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e91d9b640 1 -- 172.21.15.107:0/4188579389 learned_addr learned my addr 172.21.15.107:0/4188579389 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e92d9d640 1 --2- 172.21.15.107:0/4188579389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e94106720 0x7f0e940fe170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e91d9b640 1 -- 172.21.15.107:0/4188579389 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e94100760 msgr2=0x7f0e940fd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e91d9b640 1 --2- 172.21.15.107:0/4188579389 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e94100760 0x7f0e940fd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e91d9b640 1 -- 172.21.15.107:0/4188579389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e94106720 msgr2=0x7f0e940fe170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e91d9b640 1 --2- 172.21.15.107:0/4188579389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e94106720 0x7f0e940fe170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e91d9b640 1 -- 172.21.15.107:0/4188579389 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e941a63a0 con 0x7f0e94101120 2024-05-09T08:47:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e9259c640 1 --2- 172.21.15.107:0/4188579389 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e94100760 0x7f0e940fd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e92d9d640 1 --2- 172.21.15.107:0/4188579389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e94106720 0x7f0e940fe170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:03.712 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.709+0000 7f0e91d9b640 1 --2- 172.21.15.107:0/4188579389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e94101120 0x7f0e940fdc30 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f0e8800e9f0 tx=0x7f0e8800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:03.712 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.710+0000 7f0e837fe640 1 -- 172.21.15.107:0/4188579389 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8800cd80 con 0x7f0e94101120 2024-05-09T08:47:03.712 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.710+0000 7f0e837fe640 1 -- 172.21.15.107:0/4188579389 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e880108e0 con 0x7f0e94101120 2024-05-09T08:47:03.712 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.710+0000 7f0e98b10640 1 -- 172.21.15.107:0/4188579389 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e941a6690 con 0x7f0e94101120 2024-05-09T08:47:03.712 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.710+0000 7f0e837fe640 1 -- 172.21.15.107:0/4188579389 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e88018920 con 0x7f0e94101120 2024-05-09T08:47:03.712 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.710+0000 7f0e98b10640 1 -- 172.21.15.107:0/4188579389 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e941a6be0 con 0x7f0e94101120 2024-05-09T08:47:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.712+0000 7f0e837fe640 1 -- 172.21.15.107:0/4188579389 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0e88018ac0 con 0x7f0e94101120 2024-05-09T08:47:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.713+0000 7f0e9b818740 1 Processor -- start 2024-05-09T08:47:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.713+0000 7f0e837fe640 1 --2- 172.21.15.107:0/4188579389 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f0e7c0a2e80 0x7f0e7c0a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.713+0000 7f0e92d9d640 1 --2- 172.21.15.107:0/4188579389 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f0e7c0a2e80 0x7f0e7c0a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.713+0000 7f0e837fe640 1 -- 172.21.15.107:0/4188579389 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f0e880cfb70 con 0x7f0e94101120 2024-05-09T08:47:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.713+0000 7f0e9b818740 1 -- start start 2024-05-09T08:47:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.713+0000 7f0e9b818740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c612370 0x56154c612750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.713+0000 7f0e9b818740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 0x56154c625900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.713+0000 7f0e9b818740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c625e40 0x56154c628230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.713+0000 7f0e9b818740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56154c577340 con 0x56154c612c90 2024-05-09T08:47:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.713+0000 7f0e9b818740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56154c51f290 con 0x56154c625e40 2024-05-09T08:47:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.713+0000 7f0e9b818740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56154c573300 con 0x56154c612370 2024-05-09T08:47:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.714+0000 7f0e91d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 0x56154c625900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.714+0000 7f0e91d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 0x56154c625900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41958/0 (socket says 172.21.15.107:41958) 2024-05-09T08:47:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.714+0000 7f0e91d9b640 1 -- 172.21.15.107:0/4032821761 learned_addr learned my addr 172.21.15.107:0/4032821761 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.714+0000 7f0e9259c640 1 --2- 172.21.15.107:0/4032821761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c612370 0x56154c612750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.714+0000 7f0e92d9d640 1 --2- 172.21.15.107:0/4032821761 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c625e40 0x56154c628230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.714+0000 7f0e92d9d640 1 --2- 172.21.15.107:0/4188579389 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f0e7c0a2e80 0x7f0e7c0a5340 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c004af0 tx=0x7f0e8c004a60 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.714+0000 7f0e91d9b640 1 -- 172.21.15.107:0/4032821761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c612370 msgr2=0x56154c612750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.714+0000 7f0e91d9b640 1 --2- 172.21.15.107:0/4032821761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c612370 0x56154c612750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.714+0000 7f0e91d9b640 1 -- 172.21.15.107:0/4032821761 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c625e40 msgr2=0x56154c628230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.714+0000 7f0e91d9b640 1 --2- 172.21.15.107:0/4032821761 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c625e40 0x56154c628230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.714+0000 7f0e91d9b640 1 -- 172.21.15.107:0/4032821761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56154c5438e0 con 0x56154c612c90 2024-05-09T08:47:03.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.714+0000 7f0e91d9b640 1 --2- 172.21.15.107:0/4032821761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 0x56154c625900 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f0e88098790 tx=0x7f0e8809ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65bde86016cffc5b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:03.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.714+0000 7f0e80ff9640 1 -- 172.21.15.107:0/4032821761 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e88010a90 con 0x56154c612c90 2024-05-09T08:47:03.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.714+0000 7f0e80ff9640 1 -- 172.21.15.107:0/4032821761 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e88098430 con 0x56154c612c90 2024-05-09T08:47:03.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.714+0000 7f0e80ff9640 1 -- 172.21.15.107:0/4032821761 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e88010a90 con 0x56154c612c90 2024-05-09T08:47:03.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.715+0000 7f0e9b818740 1 -- 172.21.15.107:0/4032821761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 msgr2=0x56154c625900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:03.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.715+0000 7f0e9b818740 1 --2- 172.21.15.107:0/4032821761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 0x56154c625900 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f0e88098790 tx=0x7f0e8809ba30 comp rx=0 tx=0).stop 2024-05-09T08:47:03.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.715+0000 7f0e9b818740 1 -- 172.21.15.107:0/4032821761 shutdown_connections 2024-05-09T08:47:03.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.715+0000 7f0e9b818740 1 --2- 172.21.15.107:0/4032821761 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c625e40 0x56154c628230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.715+0000 7f0e9b818740 1 --2- 172.21.15.107:0/4032821761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 0x56154c625900 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.715+0000 7f0e9b818740 1 --2- 172.21.15.107:0/4032821761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c612370 0x56154c612750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.715+0000 7f0e9b818740 1 -- 172.21.15.107:0/4032821761 >> 172.21.15.107:0/4032821761 conn(0x56154c60e960 msgr2=0x56154c60ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:03.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.715+0000 7f0e9b818740 1 -- 172.21.15.107:0/4032821761 shutdown_connections 2024-05-09T08:47:03.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.715+0000 7f0e9b818740 1 -- 172.21.15.107:0/4032821761 wait complete. 2024-05-09T08:47:03.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.716+0000 7f0e9b818740 1 -- start start 2024-05-09T08:47:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.716+0000 7f0e9b818740 1 Processor -- start 2024-05-09T08:47:03.722 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.716+0000 7f0e9b818740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c612370 0x56154c622de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.722 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.716+0000 7f0e9b818740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 0x56154c623320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.722 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.716+0000 7f0e9b818740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c625e40 0x56154c623860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.722 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.716+0000 7f0e9b818740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56154c628a10 con 0x56154c612c90 2024-05-09T08:47:03.722 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.716+0000 7f0e91d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 0x56154c623320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.722 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.716+0000 7f0e9b818740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56154c532790 con 0x56154c625e40 2024-05-09T08:47:03.722 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.716+0000 7f0e9b818740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56154c628b90 con 0x56154c612370 2024-05-09T08:47:03.722 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.716+0000 7f0e9259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c612370 0x56154c622de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.722 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e9259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c612370 0x56154c622de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:41368/0 (socket says 172.21.15.107:41368) 2024-05-09T08:47:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e9259c640 1 -- 172.21.15.107:0/1640436441 learned_addr learned my addr 172.21.15.107:0/1640436441 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e92d9d640 1 --2- 172.21.15.107:0/1640436441 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c625e40 0x56154c623860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e9b818740 10 client.0 _ll_register_callbacks cb 0x56154c56da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e9259c640 1 -- 172.21.15.107:0/1640436441 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c625e40 msgr2=0x56154c623860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e9259c640 1 --2- 172.21.15.107:0/1640436441 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c625e40 0x56154c623860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e9259c640 1 -- 172.21.15.107:0/1640436441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 msgr2=0x56154c623320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e9259c640 1 --2- 172.21.15.107:0/1640436441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 0x56154c623320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e9259c640 1 -- 172.21.15.107:0/1640436441 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56154c28b210 con 0x56154c612370 2024-05-09T08:47:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e92d9d640 1 --2- 172.21.15.107:0/1640436441 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c625e40 0x56154c623860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e91d9b640 1 --2- 172.21.15.107:0/1640436441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 0x56154c623320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e9259c640 1 --2- 172.21.15.107:0/1640436441 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c612370 0x56154c622de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0e84011830 tx=0x7f0e84011cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39cd79765a8d7e58 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:47:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e79ffb640 1 -- 172.21.15.107:0/1640436441 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e84010ba0 con 0x56154c612370 2024-05-09T08:47:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e9b818740 1 -- 172.21.15.107:0/1640436441 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e88098430 con 0x56154c612370 2024-05-09T08:47:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e79ffb640 1 -- 172.21.15.107:0/1640436441 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f0e84010d40 con 0x56154c612370 2024-05-09T08:47:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e79ffb640 1 -- 172.21.15.107:0/1640436441 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0e84025050 con 0x56154c612370 2024-05-09T08:47:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e9b818740 10 client.28312 fetch_fsmap learned FSMap version 27 2024-05-09T08:47:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.717+0000 7f0e9b818740 1 -- 172.21.15.107:0/1640436441 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56154c357620 con 0x56154c612370 2024-05-09T08:47:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.718+0000 7f0e79ffb640 1 -- 172.21.15.107:0/1640436441 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e88098430 con 0x56154c612370 2024-05-09T08:47:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.718+0000 7f0e79ffb640 1 -- 172.21.15.107:0/1640436441 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f0e8400f4e0 con 0x56154c612370 2024-05-09T08:47:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.718+0000 7f0e9b818740 10 client.28312 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:47:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.718+0000 7f0e9b818740 10 client.28312 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:47:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.718+0000 7f0e9b818740 20 client.28312 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.718+0000 7f0e9b818740 1 --2- 172.21.15.107:0/1640436441 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56154c635bb0 0x56154c638040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.718+0000 7f0e9b818740 4 client.28312 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:47:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.718+0000 7f0e9b818740 1 -- 172.21.15.107:0/1640436441 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56154c584a70 con 0x56154c635bb0 2024-05-09T08:47:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.718+0000 7f0e91d9b640 1 --2- 172.21.15.107:0/1640436441 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56154c635bb0 0x56154c638040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.718+0000 7f0e91d9b640 1 --2- 172.21.15.107:0/1640436441 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56154c635bb0 0x56154c638040 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d76b968e536192 server_cookie=fad11767ff6ecf79 in_seq=0 out_seq=0 2024-05-09T08:47:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.718+0000 7f0e79ffb640 10 client.28312 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.719+0000 7f0e79ffb640 1 -- 172.21.15.107:0/1640436441 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56154c584a70 con 0x56154c635bb0 2024-05-09T08:47:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.719+0000 7f0e79ffb640 10 client.28312 handle_command_reply: tid=0 2024-05-09T08:47:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.719+0000 7f0e9b818740 1 client.28312 shutdown 2024-05-09T08:47:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.719+0000 7f0e9b818740 1 -- 172.21.15.107:0/1640436441 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c612370 msgr2=0x56154c622de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.719+0000 7f0e9b818740 1 --2- 172.21.15.107:0/1640436441 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c612370 0x56154c622de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0e84011830 tx=0x7f0e84011cf0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.719+0000 7f0e9b818740 1 -- 172.21.15.107:0/1640436441 shutdown_connections 2024-05-09T08:47:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.719+0000 7f0e9b818740 1 --2- 172.21.15.107:0/1640436441 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56154c635bb0 0x56154c638040 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.719+0000 7f0e9b818740 1 --2- 172.21.15.107:0/1640436441 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c625e40 0x56154c623860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.719+0000 7f0e9b818740 1 --2- 172.21.15.107:0/1640436441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 0x56154c623320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.719+0000 7f0e79ffb640 0 client.28312 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.719+0000 7f0e9b818740 1 --2- 172.21.15.107:0/1640436441 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c612370 0x56154c622de0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.719+0000 7f0e9b818740 1 -- 172.21.15.107:0/1640436441 >> 172.21.15.107:0/1640436441 conn(0x56154c60e960 msgr2=0x56154c6147d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.719+0000 7f0e9b818740 1 -- 172.21.15.107:0/1640436441 shutdown_connections 2024-05-09T08:47:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.719+0000 7f0e9b818740 1 -- 172.21.15.107:0/1640436441 wait complete. 2024-05-09T08:47:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.720+0000 7f0e9b818740 20 client.28312 trim_cache size 0 max 16384 2024-05-09T08:47:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.745+0000 7f0e9b818740 1 Processor -- start 2024-05-09T08:47:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.745+0000 7f0e9b818740 1 -- start start 2024-05-09T08:47:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.745+0000 7f0e9b818740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c612370 0x56154c625510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.745+0000 7f0e9b818740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c612c90 0x56154c613510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.745+0000 7f0e9b818740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c625e40 0x56154c6318d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.745+0000 7f0e9b818740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0e941af2d0 con 0x56154c625e40 2024-05-09T08:47:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.745+0000 7f0e9b818740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0e941acba0 con 0x56154c612c90 2024-05-09T08:47:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.745+0000 7f0e9b818740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0e941aaa50 con 0x56154c612370 2024-05-09T08:47:03.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.747+0000 7f0e92d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c625e40 0x56154c6318d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.747+0000 7f0e92d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c625e40 0x56154c6318d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41966/0 (socket says 172.21.15.107:41966) 2024-05-09T08:47:03.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.747+0000 7f0e92d9d640 1 -- 172.21.15.107:0/2117902918 learned_addr learned my addr 172.21.15.107:0/2117902918 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:03.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.747+0000 7f0e92d9d640 1 -- 172.21.15.107:0/2117902918 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c612370 msgr2=0x56154c625510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:47:03.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.747+0000 7f0e92d9d640 1 --2- 172.21.15.107:0/2117902918 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c612370 0x56154c625510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.747+0000 7f0e92d9d640 1 -- 172.21.15.107:0/2117902918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c612c90 msgr2=0x56154c613510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:47:03.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.747+0000 7f0e92d9d640 1 --2- 172.21.15.107:0/2117902918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c612c90 0x56154c613510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.747+0000 7f0e92d9d640 1 -- 172.21.15.107:0/2117902918 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56154c632370 con 0x56154c625e40 2024-05-09T08:47:03.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.748+0000 7f0e92d9d640 1 --2- 172.21.15.107:0/2117902918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c625e40 0x56154c6318d0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009cd0 tx=0x7f0e8c041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ee1fb12910760c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:03.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.748+0000 7f0e7bfff640 1 -- 172.21.15.107:0/2117902918 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c058070 con 0x56154c625e40 2024-05-09T08:47:03.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.748+0000 7f0e7bfff640 1 -- 172.21.15.107:0/2117902918 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e8c036040 con 0x56154c625e40 2024-05-09T08:47:03.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.748+0000 7f0e7bfff640 1 -- 172.21.15.107:0/2117902918 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c053670 con 0x56154c625e40 2024-05-09T08:47:03.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.749+0000 7f0e9b818740 1 -- 172.21.15.107:0/2117902918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c625e40 msgr2=0x56154c6318d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:03.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.749+0000 7f0e9b818740 1 --2- 172.21.15.107:0/2117902918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c625e40 0x56154c6318d0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009cd0 tx=0x7f0e8c041000 comp rx=0 tx=0).stop 2024-05-09T08:47:03.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.749+0000 7f0e9b818740 1 -- 172.21.15.107:0/2117902918 shutdown_connections 2024-05-09T08:47:03.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.749+0000 7f0e9b818740 1 --2- 172.21.15.107:0/2117902918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c625e40 0x56154c6318d0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.749+0000 7f0e9b818740 1 --2- 172.21.15.107:0/2117902918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c612c90 0x56154c613510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.749+0000 7f0e9b818740 1 --2- 172.21.15.107:0/2117902918 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c612370 0x56154c625510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.749+0000 7f0e9b818740 1 -- 172.21.15.107:0/2117902918 >> 172.21.15.107:0/2117902918 conn(0x56154c60e960 msgr2=0x56154c60ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:03.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.751+0000 7f0e9b818740 1 -- 172.21.15.107:0/2117902918 shutdown_connections 2024-05-09T08:47:03.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.752+0000 7f0e9b818740 1 -- 172.21.15.107:0/2117902918 wait complete. 2024-05-09T08:47:03.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.752+0000 7f0e9b818740 1 -- start start 2024-05-09T08:47:03.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.752+0000 7f0e9b818740 1 Processor -- start 2024-05-09T08:47:03.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.752+0000 7f0e9b818740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c612370 0x56154c62a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.753+0000 7f0e9b818740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 0x56154c62ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.753+0000 7f0e9b818740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c625e40 0x56154c62b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.753+0000 7f0e9b818740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0e941ab0a0 con 0x56154c612c90 2024-05-09T08:47:03.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.753+0000 7f0e9b818740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0e941ac4d0 con 0x56154c612370 2024-05-09T08:47:03.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.753+0000 7f0e9b818740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0e941ab730 con 0x56154c625e40 2024-05-09T08:47:03.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.753+0000 7f0e9b818740 10 client.0 _ll_register_callbacks cb 0x7f0e8400bc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:03.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.754+0000 7f0e9259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c612370 0x56154c62a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.754+0000 7f0e9259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c612370 0x56154c62a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:45642/0 (socket says 172.21.15.107:45642) 2024-05-09T08:47:03.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.754+0000 7f0e9259c640 1 -- 172.21.15.107:0/2787918371 learned_addr learned my addr 172.21.15.107:0/2787918371 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:03.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.754+0000 7f0e91d9b640 1 --2- 172.21.15.107:0/2787918371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 0x56154c62ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.754+0000 7f0e92d9d640 1 --2- 172.21.15.107:0/2787918371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c625e40 0x56154c62b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.754+0000 7f0e91d9b640 1 -- 172.21.15.107:0/2787918371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c625e40 msgr2=0x56154c62b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:03.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.754+0000 7f0e91d9b640 1 --2- 172.21.15.107:0/2787918371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c625e40 0x56154c62b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.754+0000 7f0e91d9b640 1 -- 172.21.15.107:0/2787918371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c612370 msgr2=0x56154c62a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:03.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.754+0000 7f0e91d9b640 1 --2- 172.21.15.107:0/2787918371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c612370 0x56154c62a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:03.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.754+0000 7f0e91d9b640 1 -- 172.21.15.107:0/2787918371 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56154c61be80 con 0x56154c612c90 2024-05-09T08:47:03.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.755+0000 7f0e91d9b640 1 --2- 172.21.15.107:0/2787918371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 0x56154c62ad30 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f0e8809bac0 tx=0x7f0e880a48a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b413ea6fbebbda48 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:03.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.755+0000 7f0e79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:47:03.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.756+0000 7f0e79ffb640 1 -- 172.21.15.107:0/2787918371 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e880ba060 con 0x56154c612c90 2024-05-09T08:47:03.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.756+0000 7f0e79ffb640 1 -- 172.21.15.107:0/2787918371 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9635+0+0 (secure 0 0 0) 0x7f0e88097b10 con 0x56154c612c90 2024-05-09T08:47:03.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.757+0000 7f0e9b818740 1 -- 172.21.15.107:0/2787918371 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e8c036040 con 0x56154c612c90 2024-05-09T08:47:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.758+0000 7f0e79ffb640 1 -- 172.21.15.107:0/2787918371 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0e880a9050 con 0x56154c612c90 2024-05-09T08:47:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.758+0000 7f0e9b818740 10 client.19143 fetch_fsmap learned FSMap version 27 2024-05-09T08:47:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.758+0000 7f0e9b818740 1 -- 172.21.15.107:0/2787918371 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56154c61d150 con 0x56154c612c90 2024-05-09T08:47:03.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.759+0000 7f0e79ffb640 1 -- 172.21.15.107:0/2787918371 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e8c036040 con 0x56154c612c90 2024-05-09T08:47:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.760+0000 7f0e79ffb640 1 -- 172.21.15.107:0/2787918371 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f0e880ac020 con 0x56154c612c90 2024-05-09T08:47:03.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.761+0000 7f0e9b818740 10 client.19143 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:47:03.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.761+0000 7f0e9b818740 10 client.19143 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:47:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.761+0000 7f0e9b818740 20 client.19143 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.761+0000 7f0e9b818740 1 --2- 172.21.15.107:0/2787918371 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56154c61d440 0x56154c6a71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.761+0000 7f0e9b818740 4 client.19143 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:47:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.761+0000 7f0e9b818740 1 -- 172.21.15.107:0/2787918371 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56154c57edc0 con 0x56154c61d440 2024-05-09T08:47:03.763 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.764+0000 7f0e9259c640 1 --2- 172.21.15.107:0/2787918371 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56154c61d440 0x56154c6a71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.765+0000 7f0e9259c640 1 --2- 172.21.15.107:0/2787918371 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56154c61d440 0x56154c6a71a0 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=596bb35975ea06b1 server_cookie=23c5a9057ca15415 in_seq=0 out_seq=0 2024-05-09T08:47:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:03.765+0000 7f0e79ffb640 10 client.19143 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:04.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:04 smithi157 ceph-mon[33614]: pgmap v428: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 11 MiB/s wr, 2.87k op/s 2024-05-09T08:47:04.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:04 smithi110 ceph-mon[33622]: pgmap v428: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 11 MiB/s wr, 2.87k op/s 2024-05-09T08:47:04.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:04 smithi107 ceph-mon[30122]: pgmap v428: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 11 MiB/s wr, 2.87k op/s 2024-05-09T08:47:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.238+0000 7f0e79ffb640 1 -- 172.21.15.107:0/2787918371 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56154c57edc0 con 0x56154c61d440 2024-05-09T08:47:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.238+0000 7f0e79ffb640 10 client.19143 handle_command_reply: tid=0 2024-05-09T08:47:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.238+0000 7f0e9b818740 1 client.19143 shutdown 2024-05-09T08:47:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.238+0000 7f0e9b818740 1 -- 172.21.15.107:0/2787918371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 msgr2=0x56154c62ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.238+0000 7f0e9b818740 1 --2- 172.21.15.107:0/2787918371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 0x56154c62ad30 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f0e8809bac0 tx=0x7f0e880a48a0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.239+0000 7f0e9b818740 1 -- 172.21.15.107:0/2787918371 shutdown_connections 2024-05-09T08:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.239+0000 7f0e9b818740 1 --2- 172.21.15.107:0/2787918371 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56154c61d440 0x56154c6a71a0 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.239+0000 7f0e9b818740 1 --2- 172.21.15.107:0/2787918371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56154c625e40 0x56154c62b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.239+0000 7f0e9b818740 1 --2- 172.21.15.107:0/2787918371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56154c612c90 0x56154c62ad30 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.239+0000 7f0e9b818740 1 --2- 172.21.15.107:0/2787918371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56154c612370 0x56154c62a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.239+0000 7f0e9b818740 1 -- 172.21.15.107:0/2787918371 >> 172.21.15.107:0/2787918371 conn(0x56154c60e960 msgr2=0x56154c60d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.239+0000 7f0e79ffb640 0 client.19143 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.239+0000 7f0e9b818740 1 -- 172.21.15.107:0/2787918371 shutdown_connections 2024-05-09T08:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.239+0000 7f0e9b818740 1 -- 172.21.15.107:0/2787918371 wait complete. 2024-05-09T08:47:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.239+0000 7f0e9b818740 20 client.19143 trim_cache size 0 max 16384 2024-05-09T08:47:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:47:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:47:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:47:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:47:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.240+0000 7f0e98b10640 1 -- 172.21.15.107:0/4188579389 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f0e7c0a2e80 msgr2=0x7f0e7c0a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.240+0000 7f0e98b10640 1 --2- 172.21.15.107:0/4188579389 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f0e7c0a2e80 0x7f0e7c0a5340 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c004af0 tx=0x7f0e8c004a60 comp rx=0 tx=0).stop 2024-05-09T08:47:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.240+0000 7f0e98b10640 1 -- 172.21.15.107:0/4188579389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e94101120 msgr2=0x7f0e940fdc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.240+0000 7f0e98b10640 1 --2- 172.21.15.107:0/4188579389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e94101120 0x7f0e940fdc30 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f0e8800e9f0 tx=0x7f0e8800eeb0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.240+0000 7f0e98b10640 1 -- 172.21.15.107:0/4188579389 shutdown_connections 2024-05-09T08:47:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.240+0000 7f0e98b10640 1 --2- 172.21.15.107:0/4188579389 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f0e7c0a2e80 0x7f0e7c0a5340 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.240+0000 7f0e98b10640 1 --2- 172.21.15.107:0/4188579389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e94106720 0x7f0e940fe170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.240+0000 7f0e98b10640 1 --2- 172.21.15.107:0/4188579389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e94101120 0x7f0e940fdc30 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.240+0000 7f0e98b10640 1 --2- 172.21.15.107:0/4188579389 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e94100760 0x7f0e940fd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.240+0000 7f0e98b10640 1 -- 172.21.15.107:0/4188579389 >> 172.21.15.107:0/4188579389 conn(0x7f0e940778b0 msgr2=0x7f0e940ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.240+0000 7f0e98b10640 1 -- 172.21.15.107:0/4188579389 shutdown_connections 2024-05-09T08:47:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:06.240+0000 7f0e98b10640 1 -- 172.21.15.107:0/4188579389 wait complete. 2024-05-09T08:47:06.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:06 smithi157 ceph-mon[33614]: pgmap v429: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.6 MiB/s wr, 2.45k op/s 2024-05-09T08:47:06.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:06 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:47:06.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:06 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:06.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:06 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:47:06.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:06 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:06.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:06 smithi110 ceph-mon[33622]: pgmap v429: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.6 MiB/s wr, 2.45k op/s 2024-05-09T08:47:06.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:06 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:47:06.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:06 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:06.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:06 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:47:06.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:06 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:06.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:06 smithi107 ceph-mon[30122]: pgmap v429: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.6 MiB/s wr, 2.45k op/s 2024-05-09T08:47:06.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:06 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:47:06.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:06 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:06.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:06 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:47:06.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:06 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:07.094 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: *** Caught signal (Segmentation fault) ** 2024-05-09T08:47:07.094 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: in thread 7f4f66bd8700 thread_name:md_log_replay 2024-05-09T08:47:07.094 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:47:07.094 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4f77f9ad20] 2024-05-09T08:47:07.095 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 2: /usr/bin/ceph-mds(+0x549382) [0x561b2a478382] 2024-05-09T08:47:07.095 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x561b2a47bb33] 2024-05-09T08:47:07.095 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 4: (MDLog::_replay_thread()+0x617) [0x561b2a4d6617] 2024-05-09T08:47:07.095 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 5: (MDLog::ReplayThread::entry()+0x11) [0x561b2a4dfd01] 2024-05-09T08:47:07.095 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f4f77f901ca] 2024-05-09T08:47:07.095 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 7: clone() 2024-05-09T08:47:07.095 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 2024-05-09T08:47:06.717+0000 7f4f66bd8700 -1 *** Caught signal (Segmentation fault) ** 2024-05-09T08:47:07.095 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: in thread 7f4f66bd8700 thread_name:md_log_replay 2024-05-09T08:47:07.095 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 2024-05-09T08:47:07.095 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:47:07.095 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4f77f9ad20] 2024-05-09T08:47:07.095 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 2: /usr/bin/ceph-mds(+0x549382) [0x561b2a478382] 2024-05-09T08:47:07.095 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x561b2a47bb33] 2024-05-09T08:47:07.095 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 4: (MDLog::_replay_thread()+0x617) [0x561b2a4d6617] 2024-05-09T08:47:07.096 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 5: (MDLog::ReplayThread::entry()+0x11) [0x561b2a4dfd01] 2024-05-09T08:47:07.096 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f4f77f901ca] 2024-05-09T08:47:07.096 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 7: clone() 2024-05-09T08:47:07.096 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T08:47:07.096 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 2024-05-09T08:47:07.096 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 0> 2024-05-09T08:47:06.717+0000 7f4f66bd8700 -1 *** Caught signal (Segmentation fault) ** 2024-05-09T08:47:07.096 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: in thread 7f4f66bd8700 thread_name:md_log_replay 2024-05-09T08:47:07.096 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 2024-05-09T08:47:07.096 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:47:07.096 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4f77f9ad20] 2024-05-09T08:47:07.096 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 2: /usr/bin/ceph-mds(+0x549382) [0x561b2a478382] 2024-05-09T08:47:07.096 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x561b2a47bb33] 2024-05-09T08:47:07.096 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 4: (MDLog::_replay_thread()+0x617) [0x561b2a4d6617] 2024-05-09T08:47:07.096 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 5: (MDLog::ReplayThread::entry()+0x11) [0x561b2a4dfd01] 2024-05-09T08:47:07.097 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f4f77f901ca] 2024-05-09T08:47:07.097 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 7: clone() 2024-05-09T08:47:07.097 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T08:47:07.097 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 2024-05-09T08:47:07.097 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 0> 2024-05-09T08:47:06.717+0000 7f4f66bd8700 -1 *** Caught signal (Segmentation fault) ** 2024-05-09T08:47:07.097 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: in thread 7f4f66bd8700 thread_name:md_log_replay 2024-05-09T08:47:07.097 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 2024-05-09T08:47:07.097 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T08:47:07.097 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4f77f9ad20] 2024-05-09T08:47:07.097 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 2: /usr/bin/ceph-mds(+0x549382) [0x561b2a478382] 2024-05-09T08:47:07.097 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x561b2a47bb33] 2024-05-09T08:47:07.097 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 4: (MDLog::_replay_thread()+0x617) [0x561b2a4d6617] 2024-05-09T08:47:07.097 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 5: (MDLog::ReplayThread::entry()+0x11) [0x561b2a4dfd01] 2024-05-09T08:47:07.097 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f4f77f901ca] 2024-05-09T08:47:07.097 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 7: clone() 2024-05-09T08:47:07.098 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T08:47:07.098 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:06 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[63728]: 2024-05-09T08:47:07.593 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:07 smithi157 podman[69926]: 2024-05-09 08:47:07.155449625 +0000 UTC m=+0.306323492 container died 9a5a8cc830f29a1e50651c8d87fcaee1ef81375587d670cd36e2c3fc2756c002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-09T08:47:07.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:07 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:47:07.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:07 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:07.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:07 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:47:07.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:07 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:07.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:07 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:47:07.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:07 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:08.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:08 smithi157 ceph-mon[33614]: pgmap v430: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.4 MiB/s wr, 2.40k op/s 2024-05-09T08:47:08.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:08 smithi110 ceph-mon[33622]: pgmap v430: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.4 MiB/s wr, 2.40k op/s 2024-05-09T08:47:08.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:08 smithi107 ceph-mon[30122]: pgmap v430: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.4 MiB/s wr, 2.40k op/s 2024-05-09T08:47:09.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:09 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:47:09.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:09 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:09.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:09 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:47:09.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:09 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:09.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:09 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:47:09.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:09 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:10.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:10 smithi157 ceph-mon[33614]: pgmap v431: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 12 MiB/s wr, 2.93k op/s 2024-05-09T08:47:10.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:10 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:47:10.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:10 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:10.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:10 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:47:10.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:10 smithi110 ceph-mon[33622]: pgmap v431: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 12 MiB/s wr, 2.93k op/s 2024-05-09T08:47:10.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:10 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:47:10.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:10 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:10.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:10 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:47:10.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:10 smithi107 ceph-mon[30122]: pgmap v431: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 12 MiB/s wr, 2.93k op/s 2024-05-09T08:47:10.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:10 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:47:10.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:10 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:10.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:10 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:47:11.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:11 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:11.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:11 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:11.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:11 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:12.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:12 smithi110 ceph-mon[33622]: pgmap v432: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 8.5 MiB/s wr, 2.17k op/s 2024-05-09T08:47:12.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:12 smithi107 ceph-mon[30122]: pgmap v432: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 8.5 MiB/s wr, 2.17k op/s 2024-05-09T08:47:12.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:12 smithi157 ceph-mon[33614]: pgmap v432: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 8.5 MiB/s wr, 2.17k op/s 2024-05-09T08:47:14.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:14 smithi110 ceph-mon[33622]: pgmap v433: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-05-09T08:47:14.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:14 smithi107 ceph-mon[30122]: pgmap v433: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-05-09T08:47:14.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:14 smithi157 ceph-mon[33614]: pgmap v433: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-05-09T08:47:15.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:15 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:15.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:15 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:15.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:15 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:47:15.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:15 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:15.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:15 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:15.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:15 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:47:15.780 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:15 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:47:15.780 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:15 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:47:15.780 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:15 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:47:16.093 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:15 smithi157 podman[69926]: 2024-05-09 08:47:15.781575818 +0000 UTC m=+8.932449701 container remove 9a5a8cc830f29a1e50651c8d87fcaee1ef81375587d670cd36e2c3fc2756c002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-09T08:47:16.094 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:15 smithi157 systemd[1]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.f.service: Main process exited, code=exited, status=139/n/a 2024-05-09T08:47:16.843 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:16 smithi157 systemd[1]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-09T08:47:16.844 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:16 smithi157 systemd[1]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.f.service: Consumed 3.854s CPU time. 2024-05-09T08:47:16.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:16 smithi157 ceph-mon[33614]: pgmap v434: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 10 MiB/s wr, 2.56k op/s 2024-05-09T08:47:16.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:16 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:47:16.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:16 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:47:16.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:16 smithi110 ceph-mon[33622]: pgmap v434: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 10 MiB/s wr, 2.56k op/s 2024-05-09T08:47:16.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:16 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:47:16.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:16 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:47:17.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:16 smithi107 ceph-mon[30122]: pgmap v434: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 10 MiB/s wr, 2.56k op/s 2024-05-09T08:47:17.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:16 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:47:17.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:16 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:47:17.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:17 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:47:17.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:17 smithi157 ceph-mon[33614]: osdmap e109: 12 total, 12 up, 12 in 2024-05-09T08:47:17.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:17 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:47:17.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:17 smithi110 ceph-mon[33622]: osdmap e109: 12 total, 12 up, 12 in 2024-05-09T08:47:18.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:17 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:47:18.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:17 smithi107 ceph-mon[30122]: osdmap e109: 12 total, 12 up, 12 in 2024-05-09T08:47:18.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:18 smithi157 ceph-mon[33614]: pgmap v436: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 9.9 MiB/s wr, 2.48k op/s 2024-05-09T08:47:18.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:18 smithi110 ceph-mon[33622]: pgmap v436: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 9.9 MiB/s wr, 2.48k op/s 2024-05-09T08:47:18.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:18 smithi107 ceph-mon[30122]: pgmap v436: 129 pgs: 3 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 123 active+clean; 16 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 9.9 MiB/s wr, 2.48k op/s 2024-05-09T08:47:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.405+0000 7f8b02e37640 1 -- 172.21.15.107:0/4088009417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8afc109450 msgr2=0x7f8afc10cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.405+0000 7f8b02e37640 1 --2- 172.21.15.107:0/4088009417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8afc109450 0x7f8afc10cee0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8af40074e0 tx=0x7f8af402b0a0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.406+0000 7f8b02e37640 1 -- 172.21.15.107:0/4088009417 shutdown_connections 2024-05-09T08:47:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.406+0000 7f8b02e37640 1 --2- 172.21.15.107:0/4088009417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8afc10d610 0x7f8afc115190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.406+0000 7f8b02e37640 1 --2- 172.21.15.107:0/4088009417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8afc109450 0x7f8afc10cee0 secure :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8af40074e0 tx=0x7f8af402b0a0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.406+0000 7f8b02e37640 1 --2- 172.21.15.107:0/4088009417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8afc108aa0 0x7f8afc108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.405 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.406+0000 7f8b02e37640 1 -- 172.21.15.107:0/4088009417 >> 172.21.15.107:0/4088009417 conn(0x7f8afc06c900 msgr2=0x7f8afc06cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:19.409 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.410+0000 7f8b02e37640 1 -- 172.21.15.107:0/4088009417 shutdown_connections 2024-05-09T08:47:19.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.412+0000 7f8b02e37640 1 -- 172.21.15.107:0/4088009417 wait complete. 2024-05-09T08:47:19.412 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.413+0000 7f8b02e37640 1 Processor -- start 2024-05-09T08:47:19.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.415+0000 7f8b02e37640 1 -- start start 2024-05-09T08:47:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.415+0000 7f8b02e37640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8afc108aa0 0x7f8afc1a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.415+0000 7f8b02e37640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8afc10d610 0x7f8afc19e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.415+0000 7f8b02e37640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8afc19ed60 0x7f8afc19f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.415+0000 7f8b02e37640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc118200 con 0x7f8afc19ed60 2024-05-09T08:47:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.415+0000 7f8b02e37640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc118080 con 0x7f8afc108aa0 2024-05-09T08:47:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.415+0000 7f8b02e37640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc118380 con 0x7f8afc10d610 2024-05-09T08:47:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.415+0000 7f8b00bd3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8afc108aa0 0x7f8afc1a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.415+0000 7f8afbfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8afc10d610 0x7f8afc19e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.415+0000 7f8afbfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8afc10d610 0x7f8afc19e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:35826/0 (socket says 172.21.15.107:35826) 2024-05-09T08:47:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.415+0000 7f8afbfff640 1 -- 172.21.15.107:0/1243242256 learned_addr learned my addr 172.21.15.107:0/1243242256 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.416+0000 7f8b013d4640 1 --2- 172.21.15.107:0/1243242256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8afc19ed60 0x7f8afc19f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.416+0000 7f8afbfff640 1 -- 172.21.15.107:0/1243242256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8afc108aa0 msgr2=0x7f8afc1a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.416+0000 7f8afbfff640 1 --2- 172.21.15.107:0/1243242256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8afc108aa0 0x7f8afc1a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.416+0000 7f8afbfff640 1 -- 172.21.15.107:0/1243242256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8afc19ed60 msgr2=0x7f8afc19f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.416+0000 7f8afbfff640 1 --2- 172.21.15.107:0/1243242256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8afc19ed60 0x7f8afc19f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.416+0000 7f8afbfff640 1 -- 172.21.15.107:0/1243242256 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8afc1a3460 con 0x7f8afc10d610 2024-05-09T08:47:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.421+0000 7f8afbfff640 1 --2- 172.21.15.107:0/1243242256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8afc10d610 0x7f8afc19e820 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8af402b610 tx=0x7f8af402bd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.425+0000 7f8af9ffb640 1 -- 172.21.15.107:0/1243242256 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4004240 con 0x7f8afc10d610 2024-05-09T08:47:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.425+0000 7f8af9ffb640 1 -- 172.21.15.107:0/1243242256 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8af40043e0 con 0x7f8afc10d610 2024-05-09T08:47:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.425+0000 7f8af9ffb640 1 -- 172.21.15.107:0/1243242256 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af400a5f0 con 0x7f8afc10d610 2024-05-09T08:47:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.425+0000 7f8b02e37640 1 -- 172.21.15.107:0/1243242256 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8afc1a36f0 con 0x7f8afc10d610 2024-05-09T08:47:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.425+0000 7f8b02e37640 1 -- 172.21.15.107:0/1243242256 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8afc1a3b60 con 0x7f8afc10d610 2024-05-09T08:47:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.429+0000 7f8b05c17740 1 Processor -- start 2024-05-09T08:47:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.429+0000 7f8b05c17740 1 -- start start 2024-05-09T08:47:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.429+0000 7f8b05c17740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b6ca0 0x55f7f41b5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.429+0000 7f8b05c17740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b59e0 0x55f7f41b5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.429+0000 7f8b05c17740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41c02a0 0x55f7f41c26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.429+0000 7f8b05c17740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f7f411a340 con 0x55f7f41c02a0 2024-05-09T08:47:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.429+0000 7f8b05c17740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f7f40c2290 con 0x55f7f41b6ca0 2024-05-09T08:47:19.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.429+0000 7f8b05c17740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f7f4116300 con 0x55f7f41b59e0 2024-05-09T08:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.430+0000 7f8b00bd3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b59e0 0x55f7f41b5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.430+0000 7f8b00bd3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b59e0 0x55f7f41b5e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:35842/0 (socket says 172.21.15.107:35842) 2024-05-09T08:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.430+0000 7f8b00bd3640 1 -- 172.21.15.107:0/3467315917 learned_addr learned my addr 172.21.15.107:0/3467315917 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.431+0000 7f8afbfff640 1 --2- 172.21.15.107:0/3467315917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41c02a0 0x55f7f41c26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.432+0000 7f8b013d4640 1 --2- 172.21.15.107:0/3467315917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b6ca0 0x55f7f41b5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.432+0000 7f8b00bd3640 1 -- 172.21.15.107:0/3467315917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b6ca0 msgr2=0x55f7f41b5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.432+0000 7f8b00bd3640 1 --2- 172.21.15.107:0/3467315917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b6ca0 0x55f7f41b5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.432+0000 7f8b00bd3640 1 -- 172.21.15.107:0/3467315917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41c02a0 msgr2=0x55f7f41c26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.432+0000 7f8b00bd3640 1 --2- 172.21.15.107:0/3467315917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41c02a0 0x55f7f41c26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.432+0000 7f8b00bd3640 1 -- 172.21.15.107:0/3467315917 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7f40e68e0 con 0x55f7f41b59e0 2024-05-09T08:47:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.432+0000 7f8af9ffb640 1 -- 172.21.15.107:0/1243242256 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8af4043060 con 0x7f8afc10d610 2024-05-09T08:47:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.433+0000 7f8af9ffb640 1 --2- 172.21.15.107:0/1243242256 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8ae80a2fd0 0x7f8ae80a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.433+0000 7f8af9ffb640 1 -- 172.21.15.107:0/1243242256 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7f8af40f0950 con 0x7f8afc10d610 2024-05-09T08:47:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.433+0000 7f8b00bd3640 1 --2- 172.21.15.107:0/3467315917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b59e0 0x55f7f41b5e40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8ae400b820 tx=0x7f8ae400bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=659cac260b62e575 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.435+0000 7f8b013d4640 1 --2- 172.21.15.107:0/1243242256 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8ae80a2fd0 0x7f8ae80a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:19.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.437+0000 7f8b013d4640 1 --2- 172.21.15.107:0/1243242256 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8ae80a2fd0 0x7f8ae80a5490 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00c460 tx=0x7f8aec00e040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:19.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.439+0000 7f8b02e37640 1 -- 172.21.15.107:0/3467315917 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae400d950 con 0x55f7f41b59e0 2024-05-09T08:47:19.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.439+0000 7f8b02e37640 1 -- 172.21.15.107:0/3467315917 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ae4019070 con 0x55f7f41b59e0 2024-05-09T08:47:19.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.439+0000 7f8b02e37640 1 -- 172.21.15.107:0/3467315917 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae40123e0 con 0x55f7f41b59e0 2024-05-09T08:47:19.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.441+0000 7f8b05c17740 1 -- 172.21.15.107:0/3467315917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b59e0 msgr2=0x55f7f41b5e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:19.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.441+0000 7f8b05c17740 1 --2- 172.21.15.107:0/3467315917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b59e0 0x55f7f41b5e40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8ae400b820 tx=0x7f8ae400bce0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.444+0000 7f8b05c17740 1 -- 172.21.15.107:0/3467315917 shutdown_connections 2024-05-09T08:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.444+0000 7f8b05c17740 1 --2- 172.21.15.107:0/3467315917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41c02a0 0x55f7f41c26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.444+0000 7f8b05c17740 1 --2- 172.21.15.107:0/3467315917 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b59e0 0x55f7f41b5e40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.444+0000 7f8b05c17740 1 --2- 172.21.15.107:0/3467315917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b6ca0 0x55f7f41b5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.444+0000 7f8b05c17740 1 -- 172.21.15.107:0/3467315917 >> 172.21.15.107:0/3467315917 conn(0x55f7f41b1b10 msgr2=0x55f7f41b1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.448+0000 7f8b05c17740 1 -- 172.21.15.107:0/3467315917 shutdown_connections 2024-05-09T08:47:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.449+0000 7f8b05c17740 1 -- 172.21.15.107:0/3467315917 wait complete. 2024-05-09T08:47:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.450+0000 7f8b05c17740 1 -- start start 2024-05-09T08:47:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.450+0000 7f8b05c17740 1 Processor -- start 2024-05-09T08:47:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.450+0000 7f8b05c17740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b59e0 0x55f7f41bd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.450+0000 7f8b05c17740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b6ca0 0x55f7f41bd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.450+0000 7f8b05c17740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41c02a0 0x55f7f41bdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.450+0000 7f8b05c17740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f7f41c3140 con 0x55f7f41c02a0 2024-05-09T08:47:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.450+0000 7f8b05c17740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f7f40d5790 con 0x55f7f41b59e0 2024-05-09T08:47:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.450+0000 7f8b05c17740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f7f41c32c0 con 0x55f7f41b6ca0 2024-05-09T08:47:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.451+0000 7f8b05c17740 10 client.0 _ll_register_callbacks cb 0x55f7f4110a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.451+0000 7f8b00bd3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b59e0 0x55f7f41bd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.451+0000 7f8b00bd3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b59e0 0x55f7f41bd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:35946/0 (socket says 172.21.15.107:35946) 2024-05-09T08:47:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.451+0000 7f8b00bd3640 1 -- 172.21.15.107:0/738355715 learned_addr learned my addr 172.21.15.107:0/738355715 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.451+0000 7f8b00bd3640 1 -- 172.21.15.107:0/738355715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b6ca0 msgr2=0x55f7f41bd780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:47:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.451+0000 7f8b00bd3640 1 --2- 172.21.15.107:0/738355715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b6ca0 0x55f7f41bd780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.451+0000 7f8b00bd3640 1 -- 172.21.15.107:0/738355715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41c02a0 msgr2=0x55f7f41bdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.451+0000 7f8b00bd3640 1 --2- 172.21.15.107:0/738355715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41c02a0 0x55f7f41bdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.451+0000 7f8b00bd3640 1 -- 172.21.15.107:0/738355715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7f3e2e210 con 0x55f7f41b59e0 2024-05-09T08:47:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.452+0000 7f8b00bd3640 1 --2- 172.21.15.107:0/738355715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b59e0 0x55f7f41bd240 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8ae40047b0 tx=0x7f8ae4014660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a33b35207b5283b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.452+0000 7f8adbfff640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:47:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.452+0000 7f8adbfff640 1 -- 172.21.15.107:0/738355715 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae4023070 con 0x55f7f41b59e0 2024-05-09T08:47:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.452+0000 7f8b05c17740 1 -- 172.21.15.107:0/738355715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ae4019070 con 0x55f7f41b59e0 2024-05-09T08:47:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.452+0000 7f8adbfff640 1 -- 172.21.15.107:0/738355715 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7f8ae400dd60 con 0x55f7f41b59e0 2024-05-09T08:47:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.453+0000 7f8adbfff640 1 -- 172.21.15.107:0/738355715 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8ae402f050 con 0x55f7f41b59e0 2024-05-09T08:47:19.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.457+0000 7f8b05c17740 10 client.28097 fetch_fsmap learned FSMap version 27 2024-05-09T08:47:19.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.457+0000 7f8b05c17740 1 -- 172.21.15.107:0/738355715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f7f3efa620 con 0x55f7f41b59e0 2024-05-09T08:47:19.458 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.459+0000 7f8adbfff640 1 -- 172.21.15.107:0/738355715 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ae4019070 con 0x55f7f41b59e0 2024-05-09T08:47:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.459+0000 7f8adbfff640 1 -- 172.21.15.107:0/738355715 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8ae4013070 con 0x55f7f41b59e0 2024-05-09T08:47:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.460+0000 7f8b05c17740 10 client.28097 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:47:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.460+0000 7f8b05c17740 10 client.28097 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:47:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.460+0000 7f8b05c17740 20 client.28097 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.460+0000 7f8b05c17740 1 --2- 172.21.15.107:0/738355715 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f7f41d0760 0x55f7f41d2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.460+0000 7f8b05c17740 4 client.28097 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:47:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.460+0000 7f8b05c17740 1 -- 172.21.15.107:0/738355715 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7f4127a70 con 0x55f7f41d0760 2024-05-09T08:47:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.460+0000 7f8afbfff640 1 --2- 172.21.15.107:0/738355715 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f7f41d0760 0x55f7f41d2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.464+0000 7f8adbfff640 10 client.28097 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.464+0000 7f8afbfff640 1 --2- 172.21.15.107:0/738355715 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f7f41d0760 0x55f7f41d2bf0 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a96508812a91333c server_cookie=fec9b825df8e8478 in_seq=0 out_seq=0 2024-05-09T08:47:19.468 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.469+0000 7f8adbfff640 1 -- 172.21.15.107:0/738355715 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f7f4127a70 con 0x55f7f41d0760 2024-05-09T08:47:19.468 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.469+0000 7f8adbfff640 10 client.28097 handle_command_reply: tid=0 2024-05-09T08:47:19.468 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.469+0000 7f8b05c17740 1 client.28097 shutdown 2024-05-09T08:47:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.469+0000 7f8b05c17740 1 -- 172.21.15.107:0/738355715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b59e0 msgr2=0x55f7f41bd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.469+0000 7f8b05c17740 1 --2- 172.21.15.107:0/738355715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b59e0 0x55f7f41bd240 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8ae40047b0 tx=0x7f8ae4014660 comp rx=0 tx=0).stop 2024-05-09T08:47:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.469+0000 7f8b05c17740 1 -- 172.21.15.107:0/738355715 shutdown_connections 2024-05-09T08:47:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.469+0000 7f8b05c17740 1 --2- 172.21.15.107:0/738355715 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f7f41d0760 0x55f7f41d2bf0 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.469+0000 7f8b05c17740 1 --2- 172.21.15.107:0/738355715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41c02a0 0x55f7f41bdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.469+0000 7f8b05c17740 1 --2- 172.21.15.107:0/738355715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b6ca0 0x55f7f41bd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.469+0000 7f8b05c17740 1 --2- 172.21.15.107:0/738355715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b59e0 0x55f7f41bd240 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.469+0000 7f8b05c17740 1 -- 172.21.15.107:0/738355715 >> 172.21.15.107:0/738355715 conn(0x55f7f41b1b10 msgr2=0x55f7f41b0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.469+0000 7f8adbfff640 0 client.28097 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.476+0000 7f8b05c17740 1 -- 172.21.15.107:0/738355715 shutdown_connections 2024-05-09T08:47:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.476+0000 7f8b05c17740 1 -- 172.21.15.107:0/738355715 wait complete. 2024-05-09T08:47:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.476+0000 7f8b05c17740 20 client.28097 trim_cache size 0 max 16384 2024-05-09T08:47:19.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.509+0000 7f8b05c17740 1 Processor -- start 2024-05-09T08:47:19.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.509+0000 7f8b05c17740 1 -- start start 2024-05-09T08:47:19.509 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.509+0000 7f8b05c17740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41b59e0 0x55f7f41cc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.510+0000 7f8b05c17740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b6ca0 0x55f7f41bf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.510+0000 7f8b00bd3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41b59e0 0x55f7f41cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.510+0000 7f8b00bd3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41b59e0 0x55f7f41cc110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47266/0 (socket says 172.21.15.107:47266) 2024-05-09T08:47:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.510+0000 7f8b00bd3640 1 -- 172.21.15.107:0/1249970500 learned_addr learned my addr 172.21.15.107:0/1249970500 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.510+0000 7f8b05c17740 1 --2- 172.21.15.107:0/1249970500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41c02a0 0x55f7f41d84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.510+0000 7f8b05c17740 1 -- 172.21.15.107:0/1249970500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1c92f0 con 0x55f7f41b59e0 2024-05-09T08:47:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.510+0000 7f8b05c17740 1 -- 172.21.15.107:0/1249970500 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1c6bc0 con 0x55f7f41c02a0 2024-05-09T08:47:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.510+0000 7f8b05c17740 1 -- 172.21.15.107:0/1249970500 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1c4a50 con 0x55f7f41b6ca0 2024-05-09T08:47:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.510+0000 7f8afbfff640 1 --2- 172.21.15.107:0/1249970500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b6ca0 0x55f7f41bf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:19.510 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.510+0000 7f8b013d4640 1 --2- 172.21.15.107:0/1249970500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41c02a0 0x55f7f41d84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.511+0000 7f8b013d4640 1 -- 172.21.15.107:0/1249970500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b6ca0 msgr2=0x55f7f41bf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.511+0000 7f8b013d4640 1 --2- 172.21.15.107:0/1249970500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b6ca0 0x55f7f41bf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.511+0000 7f8b013d4640 1 -- 172.21.15.107:0/1249970500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41b59e0 msgr2=0x55f7f41cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.511+0000 7f8b013d4640 1 --2- 172.21.15.107:0/1249970500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41b59e0 0x55f7f41cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.511+0000 7f8b013d4640 1 -- 172.21.15.107:0/1249970500 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7f41b2d40 con 0x55f7f41c02a0 2024-05-09T08:47:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.511+0000 7f8b00bd3640 1 --2- 172.21.15.107:0/1249970500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41b59e0 0x55f7f41cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:47:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.512+0000 7f8b013d4640 1 --2- 172.21.15.107:0/1249970500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41c02a0 0x55f7f41d84d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00a820 tx=0x7f8aec004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e71ee7e39f9c3178 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:19.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.513+0000 7f8af27fc640 1 -- 172.21.15.107:0/1249970500 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec017a20 con 0x55f7f41c02a0 2024-05-09T08:47:19.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.513+0000 7f8af27fc640 1 -- 172.21.15.107:0/1249970500 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8aec026070 con 0x55f7f41c02a0 2024-05-09T08:47:19.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.513+0000 7f8af27fc640 1 -- 172.21.15.107:0/1249970500 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec016370 con 0x55f7f41c02a0 2024-05-09T08:47:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.514+0000 7f8b05c17740 1 -- 172.21.15.107:0/1249970500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41c02a0 msgr2=0x55f7f41d84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.514+0000 7f8b05c17740 1 --2- 172.21.15.107:0/1249970500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41c02a0 0x55f7f41d84d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00a820 tx=0x7f8aec004af0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.514+0000 7f8b05c17740 1 -- 172.21.15.107:0/1249970500 shutdown_connections 2024-05-09T08:47:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.514+0000 7f8b05c17740 1 --2- 172.21.15.107:0/1249970500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41c02a0 0x55f7f41d84d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.514+0000 7f8b05c17740 1 --2- 172.21.15.107:0/1249970500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b6ca0 0x55f7f41bf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.514+0000 7f8b05c17740 1 --2- 172.21.15.107:0/1249970500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41b59e0 0x55f7f41cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.514+0000 7f8b05c17740 1 -- 172.21.15.107:0/1249970500 >> 172.21.15.107:0/1249970500 conn(0x55f7f41b1b10 msgr2=0x55f7f41be960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.517+0000 7f8b05c17740 1 -- 172.21.15.107:0/1249970500 shutdown_connections 2024-05-09T08:47:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.520+0000 7f8b05c17740 1 -- 172.21.15.107:0/1249970500 wait complete. 2024-05-09T08:47:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.520+0000 7f8b05c17740 1 -- start start 2024-05-09T08:47:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.520+0000 7f8b05c17740 1 Processor -- start 2024-05-09T08:47:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.521+0000 7f8b05c17740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b59e0 0x55f7f41b9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.521+0000 7f8b05c17740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b6ca0 0x55f7f41ba320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.521+0000 7f8b05c17740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41c02a0 0x55f7f41ba860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.522+0000 7f8b00bd3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b59e0 0x55f7f41b9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.522+0000 7f8b00bd3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b59e0 0x55f7f41b9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:35860/0 (socket says 172.21.15.107:35860) 2024-05-09T08:47:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.522+0000 7f8afbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b6ca0 0x55f7f41ba320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.522+0000 7f8afbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b6ca0 0x55f7f41ba320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:35962/0 (socket says 172.21.15.107:35962) 2024-05-09T08:47:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.522+0000 7f8b00bd3640 1 -- 172.21.15.107:0/2361200534 learned_addr learned my addr 172.21.15.107:0/2361200534 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.522+0000 7f8b05c17740 1 -- 172.21.15.107:0/2361200534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1c50a0 con 0x55f7f41c02a0 2024-05-09T08:47:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.522+0000 7f8b05c17740 1 -- 172.21.15.107:0/2361200534 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1c64f0 con 0x55f7f41b6ca0 2024-05-09T08:47:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.522+0000 7f8b05c17740 1 -- 172.21.15.107:0/2361200534 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1c5750 con 0x55f7f41b59e0 2024-05-09T08:47:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.522+0000 7f8b05c17740 10 client.0 _ll_register_callbacks cb 0x7f8ae400b670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.523+0000 7f8b013d4640 1 --2- 172.21.15.107:0/2361200534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41c02a0 0x55f7f41ba860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.524+0000 7f8afbfff640 1 -- 172.21.15.107:0/2361200534 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b59e0 msgr2=0x55f7f41b9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.524+0000 7f8afbfff640 1 --2- 172.21.15.107:0/2361200534 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b59e0 0x55f7f41b9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.524+0000 7f8afbfff640 1 -- 172.21.15.107:0/2361200534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41c02a0 msgr2=0x55f7f41ba860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.524+0000 7f8afbfff640 1 --2- 172.21.15.107:0/2361200534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41c02a0 0x55f7f41ba860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.524+0000 7f8afbfff640 1 -- 172.21.15.107:0/2361200534 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7f4235d40 con 0x55f7f41b6ca0 2024-05-09T08:47:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.526+0000 7f8afbfff640 1 --2- 172.21.15.107:0/2361200534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b6ca0 0x55f7f41ba320 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8af40edcb0 tx=0x7f8af4004590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99525bde76d3d2a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.526+0000 7f8adbfff640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:47:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.527+0000 7f8adbfff640 1 -- 172.21.15.107:0/2361200534 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af40f4040 con 0x55f7f41b6ca0 2024-05-09T08:47:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.527+0000 7f8adbfff640 1 -- 172.21.15.107:0/2361200534 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7f8af4003ca0 con 0x55f7f41b6ca0 2024-05-09T08:47:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.527+0000 7f8b05c17740 1 -- 172.21.15.107:0/2361200534 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8aec026070 con 0x55f7f41b6ca0 2024-05-09T08:47:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.527+0000 7f8adbfff640 1 -- 172.21.15.107:0/2361200534 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8af40bca80 con 0x55f7f41b6ca0 2024-05-09T08:47:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.527+0000 7f8b05c17740 10 client.28103 fetch_fsmap learned FSMap version 27 2024-05-09T08:47:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.527+0000 7f8b05c17740 1 -- 172.21.15.107:0/2361200534 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f7f4236ec0 con 0x55f7f41b6ca0 2024-05-09T08:47:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.527+0000 7f8adbfff640 1 -- 172.21.15.107:0/2361200534 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8af40d5030 con 0x55f7f41b6ca0 2024-05-09T08:47:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.527+0000 7f8adbfff640 1 -- 172.21.15.107:0/2361200534 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f8af40ef040 con 0x55f7f41b6ca0 2024-05-09T08:47:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.527+0000 7f8b05c17740 10 client.28103 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T08:47:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.527+0000 7f8b05c17740 10 client.28103 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:47:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.527+0000 7f8b05c17740 20 client.28103 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.527+0000 7f8b05c17740 1 --2- 172.21.15.107:0/2361200534 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f7f42375f0 0x55f7f4239a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.528+0000 7f8b05c17740 4 client.28103 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:47:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.528+0000 7f8b05c17740 1 -- 172.21.15.107:0/2361200534 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f8aec026070 con 0x55f7f42375f0 2024-05-09T08:47:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.528+0000 7f8b00bd3640 1 --2- 172.21.15.107:0/2361200534 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f7f42375f0 0x55f7f4239a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:19.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.539+0000 7f8b00bd3640 1 --2- 172.21.15.107:0/2361200534 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f7f42375f0 0x55f7f4239a10 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dc1c88d23a67cb5 server_cookie=b857bd702f501e4d in_seq=0 out_seq=0 2024-05-09T08:47:19.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:19.540+0000 7f8adbfff640 10 client.28103 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:20.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:20 smithi157 ceph-mon[33614]: pgmap v437: 129 pgs: 35 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 78 active+clean; 16 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 2.77k op/s 2024-05-09T08:47:20.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:20 smithi110 ceph-mon[33622]: pgmap v437: 129 pgs: 35 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 78 active+clean; 16 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 2.77k op/s 2024-05-09T08:47:21.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:20 smithi107 ceph-mon[30122]: pgmap v437: 129 pgs: 35 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 78 active+clean; 16 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 2.77k op/s 2024-05-09T08:47:22.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:22 smithi157 ceph-mon[33614]: pgmap v438: 129 pgs: 42 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 69 active+clean; 16 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-05-09T08:47:22.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:22 smithi157 ceph-mon[33614]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-05-09T08:47:22.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:22 smithi157 ceph-mon[33614]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 5 up:standby 2024-05-09T08:47:22.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:22 smithi110 ceph-mon[33622]: pgmap v438: 129 pgs: 42 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 69 active+clean; 16 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-05-09T08:47:22.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:22 smithi110 ceph-mon[33622]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-05-09T08:47:22.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:22 smithi110 ceph-mon[33622]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 5 up:standby 2024-05-09T08:47:23.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:22 smithi107 ceph-mon[30122]: pgmap v438: 129 pgs: 42 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 69 active+clean; 16 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-05-09T08:47:23.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:22 smithi107 ceph-mon[30122]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-05-09T08:47:23.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:22 smithi107 ceph-mon[30122]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 5 up:standby 2024-05-09T08:47:24.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:24 smithi157 ceph-mon[33614]: pgmap v439: 129 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 100 active+clean; 16 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 MiB/s wr, 2.64k op/s 2024-05-09T08:47:24.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:24 smithi110 ceph-mon[33622]: pgmap v439: 129 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 100 active+clean; 16 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 MiB/s wr, 2.64k op/s 2024-05-09T08:47:25.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:24 smithi107 ceph-mon[30122]: pgmap v439: 129 pgs: 17 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 100 active+clean; 16 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 MiB/s wr, 2.64k op/s 2024-05-09T08:47:26.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:26 smithi157 ceph-mon[33614]: pgmap v440: 129 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 109 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 11 MiB/s wr, 2.65k op/s 2024-05-09T08:47:26.844 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:26 smithi157 systemd[1]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-05-09T08:47:26.844 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:26 smithi157 systemd[1]: Stopped Ceph mds.f for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:47:26.844 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:26 smithi157 systemd[1]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.f.service: Consumed 3.854s CPU time. 2024-05-09T08:47:26.844 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:26 smithi157 systemd[1]: Starting Ceph mds.f for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:47:26.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:26 smithi110 ceph-mon[33622]: pgmap v440: 129 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 109 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 11 MiB/s wr, 2.65k op/s 2024-05-09T08:47:27.036 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:26 smithi107 ceph-mon[30122]: pgmap v440: 129 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 109 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 11 MiB/s wr, 2.65k op/s 2024-05-09T08:47:27.189 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:26 smithi157 podman[70155]: 2024-05-09 08:47:26.777115572 +0000 UTC m=+0.020067332 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:47:27.453 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:27 smithi157 podman[70155]: 2024-05-09 08:47:27.190934023 +0000 UTC m=+0.433885773 container create 58e5bc088347613f3902d75d4513c5831081760ad57ae0d15501d15e81f9f040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-09T08:47:27.844 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:27 smithi157 podman[70155]: 2024-05-09 08:47:27.454602728 +0000 UTC m=+0.697554474 container init 58e5bc088347613f3902d75d4513c5831081760ad57ae0d15501d15e81f9f040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_CLEAN=True, RELEASE=squid-96805dc, io.buildah.version=1.33.5) 2024-05-09T08:47:27.844 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:27 smithi157 podman[70155]: 2024-05-09 08:47:27.457829244 +0000 UTC m=+0.700780998 container start 58e5bc088347613f3902d75d4513c5831081760ad57ae0d15501d15e81f9f040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-09T08:47:27.844 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:27 smithi157 ceph-mds[70173]: -- 172.21.15.157:0/180577178 <== mon.1 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ec43d03a40 con 0x55ec43cd6800 2024-05-09T08:47:27.844 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:27 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[70169]: starting mds.f at 2024-05-09T08:47:27.844 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:27 smithi157 bash[70155]: 58e5bc088347613f3902d75d4513c5831081760ad57ae0d15501d15e81f9f040 2024-05-09T08:47:27.844 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:47:27 smithi157 systemd[1]: Started Ceph mds.f for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:47:28.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:28 smithi157 ceph-mon[33614]: pgmap v441: 129 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 109 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 10 MiB/s wr, 2.51k op/s 2024-05-09T08:47:28.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:28 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.157:6834/2116846059,v1:172.21.15.157:6835/2116846059] up:boot 2024-05-09T08:47:28.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:28 smithi157 ceph-mon[33614]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:47:28.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:28 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:47:28.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:28 smithi110 ceph-mon[33622]: pgmap v441: 129 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 109 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 10 MiB/s wr, 2.51k op/s 2024-05-09T08:47:28.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:28 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.157:6834/2116846059,v1:172.21.15.157:6835/2116846059] up:boot 2024-05-09T08:47:28.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:28 smithi110 ceph-mon[33622]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:47:28.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:28 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:47:29.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:28 smithi107 ceph-mon[30122]: pgmap v441: 129 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 109 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 10 MiB/s wr, 2.51k op/s 2024-05-09T08:47:29.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:28 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.157:6834/2116846059,v1:172.21.15.157:6835/2116846059] up:boot 2024-05-09T08:47:29.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:28 smithi107 ceph-mon[30122]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:47:29.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:28 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T08:47:30.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:30 smithi157 ceph-mon[33614]: pgmap v442: 129 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 109 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 114 KiB/s rd, 12 MiB/s wr, 2.96k op/s 2024-05-09T08:47:30.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:30 smithi110 ceph-mon[33622]: pgmap v442: 129 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 109 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 114 KiB/s rd, 12 MiB/s wr, 2.96k op/s 2024-05-09T08:47:31.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:30 smithi107 ceph-mon[30122]: pgmap v442: 129 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 109 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 114 KiB/s rd, 12 MiB/s wr, 2.96k op/s 2024-05-09T08:47:32.657 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:47:32.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:47:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.837+0000 7f27a5d90640 1 -- 172.21.15.107:0/2080991491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27a0100aa0 msgr2=0x7f27a0104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.837+0000 7f27a5d90640 1 --2- 172.21.15.107:0/2080991491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27a0100aa0 0x7f27a0104ae0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2794009860 tx=0x7f279402a340 comp rx=0 tx=0).stop 2024-05-09T08:47:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.837+0000 7f27a5d90640 1 -- 172.21.15.107:0/2080991491 shutdown_connections 2024-05-09T08:47:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.837+0000 7f27a5d90640 1 --2- 172.21.15.107:0/2080991491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a01051c0 0x7f27a010cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.837+0000 7f27a5d90640 1 --2- 172.21.15.107:0/2080991491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27a0100aa0 0x7f27a0104ae0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.837+0000 7f27a5d90640 1 --2- 172.21.15.107:0/2080991491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a01000f0 0x7f27a01004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.837+0000 7f27a5d90640 1 -- 172.21.15.107:0/2080991491 >> 172.21.15.107:0/2080991491 conn(0x7f27a0076cf0 msgr2=0x7f27a00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.840+0000 7f27a5d90640 1 -- 172.21.15.107:0/2080991491 shutdown_connections 2024-05-09T08:47:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.840+0000 7f27a5d90640 1 -- 172.21.15.107:0/2080991491 wait complete. 2024-05-09T08:47:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.841+0000 7f27a5d90640 1 Processor -- start 2024-05-09T08:47:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.841+0000 7f27a5d90640 1 -- start start 2024-05-09T08:47:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.842+0000 7f27a5d90640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a01000f0 0x7f27a019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.842+0000 7f27a5d90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a01051c0 0x7f27a019b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.842+0000 7f27a5d90640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27a0195c10 0x7f27a01960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.842+0000 7f27a5d90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f27a010f8b0 con 0x7f27a01051c0 2024-05-09T08:47:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.842+0000 7f27a5d90640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f27a010f730 con 0x7f27a0195c10 2024-05-09T08:47:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.842+0000 7f27a5d90640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f27a010fa30 con 0x7f27a01000f0 2024-05-09T08:47:32.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:32 smithi157 ceph-mon[33614]: pgmap v443: 129 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 109 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 115 KiB/s rd, 8.4 MiB/s wr, 2.05k op/s 2024-05-09T08:47:32.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.842+0000 7f279f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a01000f0 0x7f27a019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:32.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.842+0000 7f279ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27a0195c10 0x7f27a01960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:32.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.842+0000 7f279f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a01000f0 0x7f27a019ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:50440/0 (socket says 172.21.15.107:50440) 2024-05-09T08:47:32.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.842+0000 7f279f7fe640 1 -- 172.21.15.107:0/1520628353 learned_addr learned my addr 172.21.15.107:0/1520628353 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:32.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.842+0000 7f279effd640 1 --2- 172.21.15.107:0/1520628353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a01051c0 0x7f27a019b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:32.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.842+0000 7f279f7fe640 1 -- 172.21.15.107:0/1520628353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27a0195c10 msgr2=0x7f27a01960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:32.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.842+0000 7f279f7fe640 1 --2- 172.21.15.107:0/1520628353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27a0195c10 0x7f27a01960c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:32.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.842+0000 7f279f7fe640 1 -- 172.21.15.107:0/1520628353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a01051c0 msgr2=0x7f27a019b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:32.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.842+0000 7f279f7fe640 1 --2- 172.21.15.107:0/1520628353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a01051c0 0x7f27a019b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:32.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.842+0000 7f279f7fe640 1 -- 172.21.15.107:0/1520628353 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a0196980 con 0x7f27a01000f0 2024-05-09T08:47:32.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.843+0000 7f279f7fe640 1 --2- 172.21.15.107:0/1520628353 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a01000f0 0x7f27a019ab00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f279000ebc0 tx=0x7f279000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:32.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.843+0000 7f279cff9640 1 -- 172.21.15.107:0/1520628353 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2790010630 con 0x7f27a01000f0 2024-05-09T08:47:32.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.843+0000 7f279cff9640 1 -- 172.21.15.107:0/1520628353 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2790010c80 con 0x7f27a01000f0 2024-05-09T08:47:32.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.843+0000 7f279cff9640 1 -- 172.21.15.107:0/1520628353 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2790018bd0 con 0x7f27a01000f0 2024-05-09T08:47:32.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.844+0000 7f27a5d90640 1 -- 172.21.15.107:0/1520628353 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27a01a0810 con 0x7f27a01000f0 2024-05-09T08:47:32.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.844+0000 7f27a5d90640 1 -- 172.21.15.107:0/1520628353 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27a01a0ce0 con 0x7f27a01000f0 2024-05-09T08:47:32.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.848+0000 7f27a8b7d740 1 Processor -- start 2024-05-09T08:47:32.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.848+0000 7f27a8b7d740 1 -- start start 2024-05-09T08:47:32.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.848+0000 7f279cff9640 1 -- 172.21.15.107:0/1520628353 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f27900107d0 con 0x7f27a01000f0 2024-05-09T08:47:32.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.848+0000 7f27a8b7d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a000280 0x55c40a000660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:32.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.849+0000 7f27a8b7d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 0x55c40a00ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:32.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.849+0000 7f27a8b7d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a00b270 0x55c40a00d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:32.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.849+0000 7f279ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a000280 0x55c40a000660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:32.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.849+0000 7f279ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a000280 0x55c40a000660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:50454/0 (socket says 172.21.15.107:50454) 2024-05-09T08:47:32.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.849+0000 7f279f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 0x55c40a00ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:32.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.849+0000 7f279f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 0x55c40a00ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:39768/0 (socket says 172.21.15.107:39768) 2024-05-09T08:47:32.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.849+0000 7f279ffff640 1 -- 172.21.15.107:0/3154841350 learned_addr learned my addr 172.21.15.107:0/3154841350 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:32.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.849+0000 7f27a8b7d740 1 -- 172.21.15.107:0/3154841350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c409f65d00 con 0x55c40a00b270 2024-05-09T08:47:32.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.849+0000 7f27a8b7d740 1 -- 172.21.15.107:0/3154841350 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c409f0d4c0 con 0x55c40a000ba0 2024-05-09T08:47:32.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.849+0000 7f27a8b7d740 1 -- 172.21.15.107:0/3154841350 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c409f61cc0 con 0x55c40a000280 2024-05-09T08:47:32.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.849+0000 7f279effd640 1 --2- 172.21.15.107:0/3154841350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a00b270 0x55c40a00d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:32.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.850+0000 7f279ffff640 1 -- 172.21.15.107:0/3154841350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 msgr2=0x55c40a00ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:32.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.850+0000 7f279ffff640 1 --2- 172.21.15.107:0/3154841350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 0x55c40a00ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:32.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.850+0000 7f279ffff640 1 -- 172.21.15.107:0/3154841350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a00b270 msgr2=0x55c40a00d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:32.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.850+0000 7f279ffff640 1 --2- 172.21.15.107:0/3154841350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a00b270 0x55c40a00d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:32.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.850+0000 7f279ffff640 1 -- 172.21.15.107:0/3154841350 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c409d437d0 con 0x55c40a000280 2024-05-09T08:47:32.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.850+0000 7f279ffff640 1 --2- 172.21.15.107:0/3154841350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a000280 0x55c40a000660 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f279800aad0 tx=0x7f279800af90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79beb8c6c9b103a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:32.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.850+0000 7f279cff9640 1 --2- 172.21.15.107:0/1520628353 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f27880a2dc0 0x7f27880a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:32.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.850+0000 7f2785ffb640 1 -- 172.21.15.107:0/3154841350 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279801c070 con 0x55c40a000280 2024-05-09T08:47:32.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.850+0000 7f2785ffb640 1 -- 172.21.15.107:0/3154841350 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f279800cc90 con 0x55c40a000280 2024-05-09T08:47:32.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.850+0000 7f2785ffb640 1 -- 172.21.15.107:0/3154841350 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2798017d20 con 0x55c40a000280 2024-05-09T08:47:32.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.851+0000 7f27a8b7d740 1 -- 172.21.15.107:0/3154841350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a000280 msgr2=0x55c40a000660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:32.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.851+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/3154841350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a000280 0x55c40a000660 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f279800aad0 tx=0x7f279800af90 comp rx=0 tx=0).stop 2024-05-09T08:47:32.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.852+0000 7f279cff9640 1 -- 172.21.15.107:0/1520628353 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7f27900d0380 con 0x7f27a01000f0 2024-05-09T08:47:32.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.852+0000 7f27a8b7d740 1 -- 172.21.15.107:0/3154841350 shutdown_connections 2024-05-09T08:47:32.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.852+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/3154841350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a00b270 0x55c40a00d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:32.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.852+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/3154841350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 0x55c40a00ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:32.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.852+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/3154841350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a000280 0x55c40a000660 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:32.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.852+0000 7f27a8b7d740 1 -- 172.21.15.107:0/3154841350 >> 172.21.15.107:0/3154841350 conn(0x55c409ffc5a0 msgr2=0x55c409ffc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:32.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.854+0000 7f279f7fe640 1 --2- 172.21.15.107:0/1520628353 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f27880a2dc0 0x7f27880a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:32.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.854+0000 7f27a8b7d740 1 -- 172.21.15.107:0/3154841350 shutdown_connections 2024-05-09T08:47:32.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.855+0000 7f27a8b7d740 1 -- 172.21.15.107:0/3154841350 wait complete. 2024-05-09T08:47:32.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.855+0000 7f279f7fe640 1 --2- 172.21.15.107:0/1520628353 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f27880a2dc0 0x7f27880a5280 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x55c40a001840 tx=0x7f2790098040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:32.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.855+0000 7f27a8b7d740 1 -- start start 2024-05-09T08:47:32.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.856+0000 7f27a8b7d740 1 Processor -- start 2024-05-09T08:47:32.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.856+0000 7f27a8b7d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a000280 0x55c40a008220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:32.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.856+0000 7f27a8b7d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a000ba0 0x55c40a008760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:32.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.856+0000 7f27a8b7d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a00b270 0x55c40a008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:32.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.856+0000 7f27a8b7d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c40a00e060 con 0x55c40a000ba0 2024-05-09T08:47:32.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.856+0000 7f27a8b7d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c409f20b90 con 0x55c40a00b270 2024-05-09T08:47:32.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.856+0000 7f27a8b7d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c40a00e1e0 con 0x55c40a000280 2024-05-09T08:47:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.856+0000 7f27a8b7d740 10 client.0 _ll_register_callbacks cb 0x55c409f5cda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.856+0000 7f279ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a00b270 0x55c40a008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.856+0000 7f279ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a00b270 0x55c40a008ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:39782/0 (socket says 172.21.15.107:39782) 2024-05-09T08:47:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.856+0000 7f279ffff640 1 -- 172.21.15.107:0/1494908492 learned_addr learned my addr 172.21.15.107:0/1494908492 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.856+0000 7f279ffff640 1 --2- 172.21.15.107:0/1494908492 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a000280 0x55c40a008220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:32.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.857+0000 7f279effd640 1 --2- 172.21.15.107:0/1494908492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a000ba0 0x55c40a008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:32.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.858+0000 7f279ffff640 1 -- 172.21.15.107:0/1494908492 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a000280 msgr2=0x55c40a008220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:32.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.858+0000 7f279ffff640 1 --2- 172.21.15.107:0/1494908492 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a000280 0x55c40a008220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.858+0000 7f279ffff640 1 -- 172.21.15.107:0/1494908492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a000ba0 msgr2=0x55c40a008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.858+0000 7f279ffff640 1 --2- 172.21.15.107:0/1494908492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a000ba0 0x55c40a008760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.858+0000 7f279ffff640 1 -- 172.21.15.107:0/1494908492 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c409c793f0 con 0x55c40a00b270 2024-05-09T08:47:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.858+0000 7f279ffff640 1 --2- 172.21.15.107:0/1494908492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a00b270 0x55c40a008ca0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f279800cb80 tx=0x7f2798005e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bcf4c687e06e411 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.858+0000 7f277effd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:47:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.859+0000 7f277effd640 1 -- 172.21.15.107:0/1494908492 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279801c030 con 0x55c40a00b270 2024-05-09T08:47:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.859+0000 7f277effd640 1 -- 172.21.15.107:0/1494908492 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7f2798004030 con 0x55c40a00b270 2024-05-09T08:47:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.859+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1494908492 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f279800cc90 con 0x55c40a00b270 2024-05-09T08:47:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.859+0000 7f277effd640 1 -- 172.21.15.107:0/1494908492 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2798004440 con 0x55c40a00b270 2024-05-09T08:47:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.859+0000 7f27a8b7d740 10 client.28118 fetch_fsmap learned FSMap version 29 2024-05-09T08:47:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.859+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1494908492 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c409c390f0 con 0x55c40a00b270 2024-05-09T08:47:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.859+0000 7f277effd640 1 -- 172.21.15.107:0/1494908492 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27980054d0 con 0x55c40a00b270 2024-05-09T08:47:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.860+0000 7f277effd640 1 -- 172.21.15.107:0/1494908492 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f279800b040 con 0x55c40a00b270 2024-05-09T08:47:32.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.860+0000 7f27a8b7d740 10 client.28118 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:47:32.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.860+0000 7f27a8b7d740 10 client.28118 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:47:32.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.860+0000 7f27a8b7d740 20 client.28118 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:32.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.860+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/1494908492 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c40a010fa0 0x55c40a013390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:32.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.860+0000 7f27a8b7d740 4 client.28118 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:47:32.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.860+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1494908492 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f279800cc90 con 0x55c40a010fa0 2024-05-09T08:47:32.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.865+0000 7f279effd640 1 --2- 172.21.15.107:0/1494908492 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c40a010fa0 0x55c40a013390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:32.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.866+0000 7f279effd640 1 --2- 172.21.15.107:0/1494908492 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c40a010fa0 0x55c40a013390 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e73b19421505d266 server_cookie=54294774f380e3bf in_seq=0 out_seq=0 2024-05-09T08:47:32.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:32.866+0000 7f277effd640 10 client.28118 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:32.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:32 smithi110 ceph-mon[33622]: pgmap v443: 129 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 109 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 115 KiB/s rd, 8.4 MiB/s wr, 2.05k op/s 2024-05-09T08:47:33.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:32 smithi107 ceph-mon[30122]: pgmap v443: 129 pgs: 10 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 109 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 115 KiB/s rd, 8.4 MiB/s wr, 2.05k op/s 2024-05-09T08:47:34.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:34 smithi157 ceph-mon[33614]: pgmap v444: 129 pgs: 10 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 112 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 130 KiB/s rd, 11 MiB/s wr, 2.65k op/s 2024-05-09T08:47:34.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:34 smithi110 ceph-mon[33622]: pgmap v444: 129 pgs: 10 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 112 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 130 KiB/s rd, 11 MiB/s wr, 2.65k op/s 2024-05-09T08:47:35.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:34 smithi107 ceph-mon[30122]: pgmap v444: 129 pgs: 10 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 112 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 130 KiB/s rd, 11 MiB/s wr, 2.65k op/s 2024-05-09T08:47:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f8adbfff640 1 -- 172.21.15.107:0/2361200534 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f8aec026070 con 0x55f7f42375f0 2024-05-09T08:47:36.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f8adbfff640 10 client.28103 handle_command_reply: tid=0 2024-05-09T08:47:36.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f8b05c17740 1 client.28103 shutdown 2024-05-09T08:47:36.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f277effd640 1 -- 172.21.15.107:0/1494908492 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f279800cc90 con 0x55c40a010fa0 2024-05-09T08:47:36.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f277effd640 10 client.28118 handle_command_reply: tid=0 2024-05-09T08:47:36.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f27a8b7d740 1 client.28118 shutdown 2024-05-09T08:47:36.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f8b05c17740 1 -- 172.21.15.107:0/2361200534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b6ca0 msgr2=0x55f7f41ba320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:36.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f8b05c17740 1 --2- 172.21.15.107:0/2361200534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b6ca0 0x55f7f41ba320 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8af40edcb0 tx=0x7f8af4004590 comp rx=0 tx=0).stop 2024-05-09T08:47:36.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f8b05c17740 1 -- 172.21.15.107:0/2361200534 shutdown_connections 2024-05-09T08:47:36.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f8b05c17740 1 --2- 172.21.15.107:0/2361200534 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f7f42375f0 0x55f7f4239a10 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f8b05c17740 1 --2- 172.21.15.107:0/2361200534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f7f41c02a0 0x55f7f41ba860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f8b05c17740 1 --2- 172.21.15.107:0/2361200534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7f41b6ca0 0x55f7f41ba320 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f8b05c17740 1 --2- 172.21.15.107:0/2361200534 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f7f41b59e0 0x55f7f41b9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f8b05c17740 1 -- 172.21.15.107:0/2361200534 >> 172.21.15.107:0/2361200534 conn(0x55f7f41b1b10 msgr2=0x55f7f41b0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f8adbfff640 0 client.28103 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:36.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1494908492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a00b270 msgr2=0x55c40a008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:36.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.050+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/1494908492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a00b270 0x55c40a008ca0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f279800cb80 tx=0x7f2798005e10 comp rx=0 tx=0).stop 2024-05-09T08:47:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.051+0000 7f8b05c17740 1 -- 172.21.15.107:0/2361200534 shutdown_connections 2024-05-09T08:47:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.051+0000 7f8b05c17740 1 -- 172.21.15.107:0/2361200534 wait complete. 2024-05-09T08:47:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.051+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1494908492 shutdown_connections 2024-05-09T08:47:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.051+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/1494908492 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c40a010fa0 0x55c40a013390 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.051+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/1494908492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a00b270 0x55c40a008ca0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.051+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/1494908492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a000ba0 0x55c40a008760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.051+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/1494908492 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a000280 0x55c40a008220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.051+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1494908492 >> 172.21.15.107:0/1494908492 conn(0x55c409ffc5a0 msgr2=0x55c40a0023a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:36.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.051+0000 7f277effd640 0 client.28118 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.051+0000 7f8b05c17740 20 client.28103 trim_cache size 0 max 16384 2024-05-09T08:47:36.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.052+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1494908492 shutdown_connections 2024-05-09T08:47:36.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.052+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1494908492 wait complete. 2024-05-09T08:47:36.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.052+0000 7f27a8b7d740 20 client.28118 trim_cache size 0 max 16384 2024-05-09T08:47:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:47:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:47:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:47:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:47:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.053+0000 7f8af37fe640 1 -- 172.21.15.107:0/1243242256 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8ae80a2fd0 msgr2=0x7f8ae80a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.053+0000 7f8af37fe640 1 --2- 172.21.15.107:0/1243242256 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8ae80a2fd0 0x7f8ae80a5490 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00c460 tx=0x7f8aec00e040 comp rx=0 tx=0).stop 2024-05-09T08:47:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.053+0000 7f8af37fe640 1 -- 172.21.15.107:0/1243242256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8afc10d610 msgr2=0x7f8afc19e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.053+0000 7f8af37fe640 1 --2- 172.21.15.107:0/1243242256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8afc10d610 0x7f8afc19e820 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8af402b610 tx=0x7f8af402bd50 comp rx=0 tx=0).stop 2024-05-09T08:47:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.054+0000 7f8af37fe640 1 -- 172.21.15.107:0/1243242256 shutdown_connections 2024-05-09T08:47:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.054+0000 7f8af37fe640 1 --2- 172.21.15.107:0/1243242256 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8ae80a2fd0 0x7f8ae80a5490 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.054+0000 7f8af37fe640 1 --2- 172.21.15.107:0/1243242256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8afc19ed60 0x7f8afc19f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.054+0000 7f8af37fe640 1 --2- 172.21.15.107:0/1243242256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8afc10d610 0x7f8afc19e820 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.054+0000 7f8af37fe640 1 --2- 172.21.15.107:0/1243242256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8afc108aa0 0x7f8afc1a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.054+0000 7f8af37fe640 1 -- 172.21.15.107:0/1243242256 >> 172.21.15.107:0/1243242256 conn(0x7f8afc06c900 msgr2=0x7f8afc071810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.054+0000 7f8af37fe640 1 -- 172.21.15.107:0/1243242256 shutdown_connections 2024-05-09T08:47:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:36.054+0000 7f8af37fe640 1 -- 172.21.15.107:0/1243242256 wait complete. 2024-05-09T08:47:36.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.092+0000 7f27a8b7d740 1 Processor -- start 2024-05-09T08:47:36.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.092+0000 7f27a8b7d740 1 -- start start 2024-05-09T08:47:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.093+0000 7f27a8b7d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a000280 0x55c40a007160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.093+0000 7f27a8b7d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 0x55c40a012fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.093+0000 7f27a8b7d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a00b270 0x55c40a0029e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.093+0000 7f27a8b7d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f27a01b7dd0 con 0x55c40a000280 2024-05-09T08:47:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.093+0000 7f27a8b7d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f27a01b56a0 con 0x55c40a000ba0 2024-05-09T08:47:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.093+0000 7f27a8b7d740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f27a00725b0 con 0x55c40a00b270 2024-05-09T08:47:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.093+0000 7f279ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a000280 0x55c40a007160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.093+0000 7f279ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a000280 0x55c40a007160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33434/0 (socket says 172.21.15.107:33434) 2024-05-09T08:47:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.093+0000 7f279ffff640 1 -- 172.21.15.107:0/2659745715 learned_addr learned my addr 172.21.15.107:0/2659745715 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.093+0000 7f279effd640 1 --2- 172.21.15.107:0/2659745715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 0x55c40a012fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.093+0000 7f279effd640 1 -- 172.21.15.107:0/2659745715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a00b270 msgr2=0x55c40a0029e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:36.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.093+0000 7f279ffff640 1 --2- 172.21.15.107:0/2659745715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a00b270 0x55c40a0029e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:36.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.093+0000 7f279effd640 1 --2- 172.21.15.107:0/2659745715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a00b270 0x55c40a0029e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.094+0000 7f279effd640 1 -- 172.21.15.107:0/2659745715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a000280 msgr2=0x55c40a007160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:36.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.094+0000 7f279effd640 1 --2- 172.21.15.107:0/2659745715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a000280 0x55c40a007160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.094+0000 7f279effd640 1 -- 172.21.15.107:0/2659745715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c40a0009d0 con 0x55c40a000ba0 2024-05-09T08:47:36.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.095+0000 7f279effd640 1 --2- 172.21.15.107:0/2659745715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 0x55c40a012fb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2794035040 tx=0x7f2794030b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f758f33004448ba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:36.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.097+0000 7f27857fa640 1 -- 172.21.15.107:0/2659745715 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2794046030 con 0x55c40a000ba0 2024-05-09T08:47:36.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.097+0000 7f27857fa640 1 -- 172.21.15.107:0/2659745715 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2794038040 con 0x55c40a000ba0 2024-05-09T08:47:36.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.097+0000 7f27857fa640 1 -- 172.21.15.107:0/2659745715 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2794037040 con 0x55c40a000ba0 2024-05-09T08:47:36.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.098+0000 7f27a8b7d740 1 -- 172.21.15.107:0/2659745715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 msgr2=0x55c40a012fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:36.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.098+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/2659745715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 0x55c40a012fb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2794035040 tx=0x7f2794030b50 comp rx=0 tx=0).stop 2024-05-09T08:47:36.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.098+0000 7f27a8b7d740 1 -- 172.21.15.107:0/2659745715 shutdown_connections 2024-05-09T08:47:36.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.099+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/2659745715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a00b270 0x55c40a0029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.099+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/2659745715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 0x55c40a012fb0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.099+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/2659745715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a000280 0x55c40a007160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.099+0000 7f27a8b7d740 1 -- 172.21.15.107:0/2659745715 >> 172.21.15.107:0/2659745715 conn(0x55c409ffc5a0 msgr2=0x55c409ffb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:36.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.100+0000 7f27a8b7d740 1 -- 172.21.15.107:0/2659745715 shutdown_connections 2024-05-09T08:47:36.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.100+0000 7f27a8b7d740 1 -- 172.21.15.107:0/2659745715 wait complete. 2024-05-09T08:47:36.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.100+0000 7f27a8b7d740 1 -- start start 2024-05-09T08:47:36.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.101+0000 7f27a8b7d740 1 Processor -- start 2024-05-09T08:47:36.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.101+0000 7f27a8b7d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a000280 0x55c40a0240a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:36.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.102+0000 7f27a8b7d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 0x55c40a0245e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:36.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.102+0000 7f27a8b7d740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a00b270 0x55c40a004490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:36.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.102+0000 7f27a8b7d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f27a0072c00 con 0x55c40a000280 2024-05-09T08:47:36.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.102+0000 7f279effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 0x55c40a0245e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:36.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.102+0000 7f279effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 0x55c40a0245e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:39800/0 (socket says 172.21.15.107:39800) 2024-05-09T08:47:36.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.102+0000 7f279effd640 1 -- 172.21.15.107:0/1299988223 learned_addr learned my addr 172.21.15.107:0/1299988223 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:36.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.102+0000 7f279ffff640 1 --2- 172.21.15.107:0/1299988223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a000280 0x55c40a0240a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:36.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.102+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1299988223 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f27a01b4fd0 con 0x55c40a000ba0 2024-05-09T08:47:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.102+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1299988223 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f27a00732d0 con 0x55c40a00b270 2024-05-09T08:47:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.102+0000 7f279ffff640 1 --2- 172.21.15.107:0/1299988223 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a00b270 0x55c40a004490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.102+0000 7f27a8b7d740 10 client.0 _ll_register_callbacks cb 0x7f279801a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.103+0000 7f279effd640 1 -- 172.21.15.107:0/1299988223 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a00b270 msgr2=0x55c40a004490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.103+0000 7f279effd640 1 --2- 172.21.15.107:0/1299988223 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a00b270 0x55c40a004490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.103+0000 7f279effd640 1 -- 172.21.15.107:0/1299988223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a000280 msgr2=0x55c40a0240a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.103+0000 7f279effd640 1 --2- 172.21.15.107:0/1299988223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a000280 0x55c40a0240a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.103+0000 7f279effd640 1 -- 172.21.15.107:0/1299988223 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c40a0049d0 con 0x55c40a000ba0 2024-05-09T08:47:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.103+0000 7f279effd640 1 --2- 172.21.15.107:0/1299988223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 0x55c40a0245e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2794005dc0 tx=0x7f27940307c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=430f8710d49bc7a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.103+0000 7f277effd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:47:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.103+0000 7f277effd640 1 -- 172.21.15.107:0/1299988223 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2794050070 con 0x55c40a000ba0 2024-05-09T08:47:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.103+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1299988223 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2794038040 con 0x55c40a000ba0 2024-05-09T08:47:36.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.103+0000 7f277effd640 1 -- 172.21.15.107:0/1299988223 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7f2794042490 con 0x55c40a000ba0 2024-05-09T08:47:36.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.105+0000 7f277effd640 1 -- 172.21.15.107:0/1299988223 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f279405c050 con 0x55c40a000ba0 2024-05-09T08:47:36.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.105+0000 7f27a8b7d740 10 client.28124 fetch_fsmap learned FSMap version 29 2024-05-09T08:47:36.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.105+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1299988223 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c40a0a3210 con 0x55c40a000ba0 2024-05-09T08:47:36.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.107+0000 7f277effd640 1 -- 172.21.15.107:0/1299988223 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2794042690 con 0x55c40a000ba0 2024-05-09T08:47:36.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.107+0000 7f277effd640 1 -- 172.21.15.107:0/1299988223 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f279403d420 con 0x55c40a000ba0 2024-05-09T08:47:36.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.108+0000 7f27a8b7d740 10 client.28124 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:47:36.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.108+0000 7f27a8b7d740 10 client.28124 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:47:36.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.108+0000 7f27a8b7d740 20 client.28124 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:36.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.108+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/1299988223 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c40a0a38c0 0x55c40a0a5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:36.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.108+0000 7f27a8b7d740 4 client.28124 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:47:36.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.108+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1299988223 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f2794038040 con 0x55c40a0a38c0 2024-05-09T08:47:36.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.108+0000 7f279ffff640 1 --2- 172.21.15.107:0/1299988223 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c40a0a38c0 0x55c40a0a5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:36.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.110+0000 7f279ffff640 1 --2- 172.21.15.107:0/1299988223 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c40a0a38c0 0x55c40a0a5ce0 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d379164e976e0799 server_cookie=df3bb23113744c48 in_seq=0 out_seq=0 2024-05-09T08:47:36.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.112+0000 7f277effd640 10 client.28124 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:36.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.112+0000 7f277effd640 1 -- 172.21.15.107:0/1299988223 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f2794038040 con 0x55c40a0a38c0 2024-05-09T08:47:36.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.112+0000 7f277effd640 10 client.28124 handle_command_reply: tid=0 2024-05-09T08:47:36.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.112+0000 7f27a8b7d740 1 client.28124 shutdown 2024-05-09T08:47:36.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.113+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1299988223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 msgr2=0x55c40a0245e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:36.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.113+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/1299988223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 0x55c40a0245e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2794005dc0 tx=0x7f27940307c0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.115+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1299988223 shutdown_connections 2024-05-09T08:47:36.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.115+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/1299988223 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55c40a0a38c0 0x55c40a0a5ce0 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.115+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/1299988223 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c40a00b270 0x55c40a004490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.115+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/1299988223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c40a000ba0 0x55c40a0245e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.115+0000 7f27a8b7d740 1 --2- 172.21.15.107:0/1299988223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c40a000280 0x55c40a0240a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.115+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1299988223 >> 172.21.15.107:0/1299988223 conn(0x55c409ffc5a0 msgr2=0x55c40a008010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:36.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.115+0000 7f277effd640 0 client.28124 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:36.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.116+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1299988223 shutdown_connections 2024-05-09T08:47:36.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.116+0000 7f27a8b7d740 1 -- 172.21.15.107:0/1299988223 wait complete. 2024-05-09T08:47:36.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.117+0000 7f27a8b7d740 20 client.28124 trim_cache size 0 max 16384 2024-05-09T08:47:36.117 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:47:36.117 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:47:36.117 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:47:36.117 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:47:36.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.118+0000 7f27a5d90640 1 -- 172.21.15.107:0/1520628353 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f27880a2dc0 msgr2=0x7f27880a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:36.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.118+0000 7f27a5d90640 1 --2- 172.21.15.107:0/1520628353 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f27880a2dc0 0x7f27880a5280 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x55c40a001840 tx=0x7f2790098040 comp rx=0 tx=0).stop 2024-05-09T08:47:36.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.118+0000 7f27a5d90640 1 -- 172.21.15.107:0/1520628353 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a01000f0 msgr2=0x7f27a019ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:36.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.118+0000 7f27a5d90640 1 --2- 172.21.15.107:0/1520628353 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a01000f0 0x7f27a019ab00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f279000ebc0 tx=0x7f279000c6a0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.118+0000 7f27a5d90640 1 -- 172.21.15.107:0/1520628353 shutdown_connections 2024-05-09T08:47:36.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.118+0000 7f27a5d90640 1 --2- 172.21.15.107:0/1520628353 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f27880a2dc0 0x7f27880a5280 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.118+0000 7f27a5d90640 1 --2- 172.21.15.107:0/1520628353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27a0195c10 0x7f27a01960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.118+0000 7f27a5d90640 1 --2- 172.21.15.107:0/1520628353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a01051c0 0x7f27a019b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.118+0000 7f27a5d90640 1 --2- 172.21.15.107:0/1520628353 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27a01000f0 0x7f27a019ab00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.118+0000 7f27a5d90640 1 -- 172.21.15.107:0/1520628353 >> 172.21.15.107:0/1520628353 conn(0x7f27a0076cf0 msgr2=0x7f27a00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.119+0000 7f27a5d90640 1 -- 172.21.15.107:0/1520628353 shutdown_connections 2024-05-09T08:47:36.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.119+0000 7f27a5d90640 1 -- 172.21.15.107:0/1520628353 wait complete. 2024-05-09T08:47:36.135 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:47:36.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:47:36.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.319+0000 7fb0fe53c640 1 -- 172.21.15.107:0/3667743401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0f810e430 msgr2=0x7fb0f811d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:36.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.319+0000 7fb0fe53c640 1 --2- 172.21.15.107:0/3667743401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0f810e430 0x7fb0f811d020 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009900 tx=0x7fb0f002a440 comp rx=0 tx=0).stop 2024-05-09T08:47:36.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.320+0000 7fb0fe53c640 1 -- 172.21.15.107:0/3667743401 shutdown_connections 2024-05-09T08:47:36.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.320+0000 7fb0fe53c640 1 --2- 172.21.15.107:0/3667743401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0f810e430 0x7fb0f811d020 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.320+0000 7fb0fe53c640 1 --2- 172.21.15.107:0/3667743401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0f810d910 0x7fb0f810dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.320+0000 7fb0fe53c640 1 --2- 172.21.15.107:0/3667743401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0f810fdb0 0x7fb0f8110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.320+0000 7fb0fe53c640 1 -- 172.21.15.107:0/3667743401 >> 172.21.15.107:0/3667743401 conn(0x7fb0f806c740 msgr2=0x7fb0f806cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:36.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.326+0000 7fb0fe53c640 1 -- 172.21.15.107:0/3667743401 shutdown_connections 2024-05-09T08:47:36.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.326+0000 7fb0fe53c640 1 -- 172.21.15.107:0/3667743401 wait complete. 2024-05-09T08:47:36.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.327+0000 7fb0fe53c640 1 Processor -- start 2024-05-09T08:47:36.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.327+0000 7fb0fe53c640 1 -- start start 2024-05-09T08:47:36.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.328+0000 7fb0fe53c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0f810d910 0x7fb0f811cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:36.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.328+0000 7fb0fe53c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0f810e430 0x7fb0f8116ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:36.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.328+0000 7fb0fe53c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0f810fdb0 0x7fb0f8117220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:36.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.328+0000 7fb0fe53c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f81110f0 con 0x7fb0f810d910 2024-05-09T08:47:36.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.328+0000 7fb0fe53c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f8110f70 con 0x7fb0f810e430 2024-05-09T08:47:36.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.328+0000 7fb0fe53c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f8111270 con 0x7fb0f810fdb0 2024-05-09T08:47:36.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.328+0000 7fb0fdd3b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0f810fdb0 0x7fb0f8117220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:36.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.328+0000 7fb0fdd3b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0f810fdb0 0x7fb0f8117220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:50520/0 (socket says 172.21.15.107:50520) 2024-05-09T08:47:36.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.328+0000 7fb0fdd3b640 1 -- 172.21.15.107:0/2092714199 learned_addr learned my addr 172.21.15.107:0/2092714199 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:36.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.329+0000 7fb0fcd39640 1 --2- 172.21.15.107:0/2092714199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0f810e430 0x7fb0f8116ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:36.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.329+0000 7fb0fdd3b640 1 -- 172.21.15.107:0/2092714199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0f810e430 msgr2=0x7fb0f8116ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:36.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.329+0000 7fb0fdd3b640 1 --2- 172.21.15.107:0/2092714199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0f810e430 0x7fb0f8116ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.329+0000 7fb0fdd3b640 1 -- 172.21.15.107:0/2092714199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0f810d910 msgr2=0x7fb0f811cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:36.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.329+0000 7fb0fdd3b640 1 --2- 172.21.15.107:0/2092714199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0f810d910 0x7fb0f811cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.329+0000 7fb0fdd3b640 1 -- 172.21.15.107:0/2092714199 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f8117950 con 0x7fb0f810fdb0 2024-05-09T08:47:36.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.330+0000 7fb0fdd3b640 1 --2- 172.21.15.107:0/2092714199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0f810fdb0 0x7fb0f8117220 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb0f002a9b0 tx=0x7fb0f0003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:36.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.335+0000 7fb0f67fc640 1 -- 172.21.15.107:0/2092714199 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f003f070 con 0x7fb0f810fdb0 2024-05-09T08:47:36.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.335+0000 7fb0f67fc640 1 -- 172.21.15.107:0/2092714199 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0f0003e40 con 0x7fb0f810fdb0 2024-05-09T08:47:36.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.335+0000 7fb0f67fc640 1 -- 172.21.15.107:0/2092714199 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0035030 con 0x7fb0f810fdb0 2024-05-09T08:47:36.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.336+0000 7fb0fe53c640 1 -- 172.21.15.107:0/2092714199 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0f81bbbe0 con 0x7fb0f810fdb0 2024-05-09T08:47:36.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.337+0000 7fb0fe53c640 1 -- 172.21.15.107:0/2092714199 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0f81bbf90 con 0x7fb0f810fdb0 2024-05-09T08:47:36.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.346+0000 7fb0f67fc640 1 -- 172.21.15.107:0/2092714199 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb0f0042070 con 0x7fb0f810fdb0 2024-05-09T08:47:36.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.349+0000 7fb0fe53c640 1 -- 172.21.15.107:0/2092714199 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0f806d9e0 con 0x7fb0f810fdb0 2024-05-09T08:47:36.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.355+0000 7fb0f67fc640 1 --2- 172.21.15.107:0/2092714199 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb0e80a2f00 0x7fb0e80a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:36.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.356+0000 7fb0f67fc640 1 -- 172.21.15.107:0/2092714199 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7fb0f00f01c0 con 0x7fb0f810fdb0 2024-05-09T08:47:36.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.356+0000 7fb0fd53a640 1 --2- 172.21.15.107:0/2092714199 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb0e80a2f00 0x7fb0e80a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:36.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.362+0000 7fb0f67fc640 1 -- 172.21.15.107:0/2092714199 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb0f0049030 con 0x7fb0f810fdb0 2024-05-09T08:47:36.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.362+0000 7fb0fd53a640 1 --2- 172.21.15.107:0/2092714199 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb0e80a2f00 0x7fb0e80a53c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009710 tx=0x7fb0e4006c90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:36.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.810+0000 7fb0fe53c640 1 -- 172.21.15.107:0/2092714199 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0f810cf60 con 0x7fb0f810fdb0 2024-05-09T08:47:36.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.814+0000 7fb0f67fc640 1 -- 172.21.15.107:0/2092714199 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11891 (secure 0 0 0) 0x7fb0f00dae20 con 0x7fb0f810fdb0 2024-05-09T08:47:36.817 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:47:36.817 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":28109,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/2116846059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2116846059},{"type":"v1","addr":"172.21.15.157:6835","nonce":2116846059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:47:22.023356+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24418":{"gid":24418,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:47:36.818 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 29 2024-05-09T08:47:36.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.817+0000 7fb0e3fff640 1 -- 172.21.15.107:0/2092714199 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb0e80a2f00 msgr2=0x7fb0e80a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:36.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.818+0000 7fb0e3fff640 1 --2- 172.21.15.107:0/2092714199 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb0e80a2f00 0x7fb0e80a53c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009710 tx=0x7fb0e4006c90 comp rx=0 tx=0).stop 2024-05-09T08:47:36.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.818+0000 7fb0e3fff640 1 -- 172.21.15.107:0/2092714199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0f810fdb0 msgr2=0x7fb0f8117220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:36.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.818+0000 7fb0e3fff640 1 --2- 172.21.15.107:0/2092714199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0f810fdb0 0x7fb0f8117220 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb0f002a9b0 tx=0x7fb0f0003cd0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.818+0000 7fb0e3fff640 1 -- 172.21.15.107:0/2092714199 shutdown_connections 2024-05-09T08:47:36.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.818+0000 7fb0e3fff640 1 --2- 172.21.15.107:0/2092714199 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb0e80a2f00 0x7fb0e80a53c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.819+0000 7fb0e3fff640 1 --2- 172.21.15.107:0/2092714199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0f810fdb0 0x7fb0f8117220 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.819+0000 7fb0e3fff640 1 --2- 172.21.15.107:0/2092714199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0f810e430 0x7fb0f8116ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.819+0000 7fb0e3fff640 1 --2- 172.21.15.107:0/2092714199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0f810d910 0x7fb0f811cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:36.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.819+0000 7fb0e3fff640 1 -- 172.21.15.107:0/2092714199 >> 172.21.15.107:0/2092714199 conn(0x7fb0f806c740 msgr2=0x7fb0f8070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:36.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.819+0000 7fb0e3fff640 1 -- 172.21.15.107:0/2092714199 shutdown_connections 2024-05-09T08:47:36.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:36.819+0000 7fb0e3fff640 1 -- 172.21.15.107:0/2092714199 wait complete. 2024-05-09T08:47:36.843 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:36 smithi157 ceph-mon[33614]: pgmap v445: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 129 KiB/s rd, 8.7 MiB/s wr, 2.13k op/s 2024-05-09T08:47:36.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:36 smithi107 ceph-mon[30122]: pgmap v445: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 129 KiB/s rd, 8.7 MiB/s wr, 2.13k op/s 2024-05-09T08:47:36.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:47:36.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:36 smithi110 ceph-mon[33622]: pgmap v445: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 129 KiB/s rd, 8.7 MiB/s wr, 2.13k op/s 2024-05-09T08:47:37.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.114+0000 7fae90a4d640 1 -- 172.21.15.107:0/751073868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae8c11b010 msgr2=0x7fae8c11d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.114+0000 7fae90a4d640 1 --2- 172.21.15.107:0/751073868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae8c11b010 0x7fae8c11d400 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fae84009cd0 tx=0x7fae8402a910 comp rx=0 tx=0).stop 2024-05-09T08:47:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.117+0000 7fae90a4d640 1 -- 172.21.15.107:0/751073868 shutdown_connections 2024-05-09T08:47:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.117+0000 7fae90a4d640 1 --2- 172.21.15.107:0/751073868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae8c11b010 0x7fae8c11d400 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.117+0000 7fae90a4d640 1 --2- 172.21.15.107:0/751073868 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae8c10e0b0 0x7fae8c10e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.117+0000 7fae90a4d640 1 --2- 172.21.15.107:0/751073868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae8c110080 0x7fae8c10db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.117+0000 7fae90a4d640 1 -- 172.21.15.107:0/751073868 >> 172.21.15.107:0/751073868 conn(0x7fae8c06c940 msgr2=0x7fae8c06cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.118+0000 7fae90a4d640 1 -- 172.21.15.107:0/751073868 shutdown_connections 2024-05-09T08:47:37.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.118+0000 7fae90a4d640 1 -- 172.21.15.107:0/751073868 wait complete. 2024-05-09T08:47:37.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.119+0000 7fae90a4d640 1 Processor -- start 2024-05-09T08:47:37.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.119+0000 7fae90a4d640 1 -- start start 2024-05-09T08:47:37.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.119+0000 7fae90a4d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae8c10e0b0 0x7fae8c118650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.119+0000 7fae90a4d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae8c110080 0x7fae8c111720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.119+0000 7fae90a4d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae8c11b010 0x7fae8c111c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.119+0000 7fae90a4d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c120840 con 0x7fae8c10e0b0 2024-05-09T08:47:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.119+0000 7fae90a4d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c1206c0 con 0x7fae8c11b010 2024-05-09T08:47:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.119+0000 7fae90a4d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fae8c1209c0 con 0x7fae8c110080 2024-05-09T08:47:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.119+0000 7fae89d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae8c110080 0x7fae8c111720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.120+0000 7fae89d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae8c110080 0x7fae8c111720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:50536/0 (socket says 172.21.15.107:50536) 2024-05-09T08:47:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.120+0000 7fae89d9b640 1 -- 172.21.15.107:0/3769528926 learned_addr learned my addr 172.21.15.107:0/3769528926 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.120+0000 7fae8ad9d640 1 --2- 172.21.15.107:0/3769528926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae8c11b010 0x7fae8c111c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.120+0000 7fae8a59c640 1 --2- 172.21.15.107:0/3769528926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae8c10e0b0 0x7fae8c118650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.120+0000 7fae89d9b640 1 -- 172.21.15.107:0/3769528926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae8c11b010 msgr2=0x7fae8c111c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.120+0000 7fae89d9b640 1 --2- 172.21.15.107:0/3769528926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae8c11b010 0x7fae8c111c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.120+0000 7fae89d9b640 1 -- 172.21.15.107:0/3769528926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae8c10e0b0 msgr2=0x7fae8c118650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.120+0000 7fae89d9b640 1 --2- 172.21.15.107:0/3769528926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae8c10e0b0 0x7fae8c118650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.120+0000 7fae89d9b640 1 -- 172.21.15.107:0/3769528926 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae8c1122a0 con 0x7fae8c110080 2024-05-09T08:47:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.120+0000 7fae8a59c640 1 --2- 172.21.15.107:0/3769528926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae8c10e0b0 0x7fae8c118650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.121+0000 7fae89d9b640 1 --2- 172.21.15.107:0/3769528926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae8c110080 0x7fae8c111720 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00c8c0 tx=0x7fae7c00cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.121+0000 7fae7b7fe640 1 -- 172.21.15.107:0/3769528926 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c004220 con 0x7fae8c110080 2024-05-09T08:47:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.121+0000 7fae90a4d640 1 -- 172.21.15.107:0/3769528926 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae8c116170 con 0x7fae8c110080 2024-05-09T08:47:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.121+0000 7fae90a4d640 1 -- 172.21.15.107:0/3769528926 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae8c116710 con 0x7fae8c110080 2024-05-09T08:47:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.121+0000 7fae7b7fe640 1 -- 172.21.15.107:0/3769528926 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae7c0043c0 con 0x7fae8c110080 2024-05-09T08:47:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.122+0000 7fae7b7fe640 1 -- 172.21.15.107:0/3769528926 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c004a00 con 0x7fae8c110080 2024-05-09T08:47:37.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.124+0000 7fae7b7fe640 1 -- 172.21.15.107:0/3769528926 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fae7c004ba0 con 0x7fae8c110080 2024-05-09T08:47:37.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.124+0000 7fae93837740 1 Processor -- start 2024-05-09T08:47:37.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.125+0000 7fae93837740 1 -- start start 2024-05-09T08:47:37.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.125+0000 7fae93837740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 0x556c05a38710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.125+0000 7fae93837740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a38c50 0x556c05a42de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.125+0000 7fae93837740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a43320 0x556c05a45710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.125+0000 7fae93837740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556c0599dc40 con 0x556c05a38330 2024-05-09T08:47:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.125+0000 7fae93837740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556c05945580 con 0x556c05a38c50 2024-05-09T08:47:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.125+0000 7fae93837740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x556c05999c00 con 0x556c05a43320 2024-05-09T08:47:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.125+0000 7fae8a59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a38c50 0x556c05a42de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.125+0000 7fae8a59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a38c50 0x556c05a42de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:39850/0 (socket says 172.21.15.107:39850) 2024-05-09T08:47:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.125+0000 7fae8a59c640 1 -- 172.21.15.107:0/1887992171 learned_addr learned my addr 172.21.15.107:0/1887992171 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.125+0000 7fae7b7fe640 1 --2- 172.21.15.107:0/3769528926 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae740a2dc0 0x7fae740a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.125+0000 7fae8ad9d640 1 --2- 172.21.15.107:0/1887992171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 0x556c05a38710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.126+0000 7fae89d9b640 1 --2- 172.21.15.107:0/1887992171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a43320 0x556c05a45710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.126+0000 7fae8ad9d640 1 --2- 172.21.15.107:0/3769528926 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae740a2dc0 0x7fae740a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.126+0000 7fae8a59c640 1 -- 172.21.15.107:0/1887992171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a43320 msgr2=0x556c05a45710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.126+0000 7fae8a59c640 1 --2- 172.21.15.107:0/1887992171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a43320 0x556c05a45710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.126+0000 7fae8a59c640 1 -- 172.21.15.107:0/1887992171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 msgr2=0x556c05a38710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.126+0000 7fae8a59c640 1 --2- 172.21.15.107:0/1887992171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 0x556c05a38710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.126+0000 7fae8a59c640 1 -- 172.21.15.107:0/1887992171 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c056b13f0 con 0x556c05a38c50 2024-05-09T08:47:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.126+0000 7fae7b7fe640 1 -- 172.21.15.107:0/3769528926 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7fae7c0cf3d0 con 0x7fae8c110080 2024-05-09T08:47:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.126+0000 7fae8a59c640 1 --2- 172.21.15.107:0/1887992171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a38c50 0x556c05a42de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fae80010450 tx=0x7fae80010910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9a2692244c1d714 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.126+0000 7fae8ad9d640 1 --2- 172.21.15.107:0/3769528926 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae740a2dc0 0x7fae740a5280 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fae84005e30 tx=0x7fae8402a910 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.126+0000 7fae78ff9640 1 -- 172.21.15.107:0/1887992171 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80011760 con 0x556c05a38c50 2024-05-09T08:47:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.126+0000 7fae78ff9640 1 -- 172.21.15.107:0/1887992171 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae80011db0 con 0x556c05a38c50 2024-05-09T08:47:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.126+0000 7fae78ff9640 1 -- 172.21.15.107:0/1887992171 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8001a380 con 0x556c05a38c50 2024-05-09T08:47:37.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.127+0000 7fae93837740 1 -- 172.21.15.107:0/1887992171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a38c50 msgr2=0x556c05a42de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.127+0000 7fae93837740 1 --2- 172.21.15.107:0/1887992171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a38c50 0x556c05a42de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fae80010450 tx=0x7fae80010910 comp rx=0 tx=0).stop 2024-05-09T08:47:37.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.127+0000 7fae93837740 1 -- 172.21.15.107:0/1887992171 shutdown_connections 2024-05-09T08:47:37.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.127+0000 7fae93837740 1 --2- 172.21.15.107:0/1887992171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a43320 0x556c05a45710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.127+0000 7fae93837740 1 --2- 172.21.15.107:0/1887992171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a38c50 0x556c05a42de0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.127+0000 7fae93837740 1 --2- 172.21.15.107:0/1887992171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 0x556c05a38710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.127+0000 7fae93837740 1 -- 172.21.15.107:0/1887992171 >> 172.21.15.107:0/1887992171 conn(0x556c05a34650 msgr2=0x556c05a34a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.127+0000 7fae93837740 1 -- 172.21.15.107:0/1887992171 shutdown_connections 2024-05-09T08:47:37.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.127+0000 7fae93837740 1 -- 172.21.15.107:0/1887992171 wait complete. 2024-05-09T08:47:37.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.127+0000 7fae93837740 1 -- start start 2024-05-09T08:47:37.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae93837740 1 Processor -- start 2024-05-09T08:47:37.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae93837740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 0x556c05a401e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae93837740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38c50 0x556c05a40720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae93837740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a43320 0x556c05a40c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae93837740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556c05a46110 con 0x556c05a38330 2024-05-09T08:47:37.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae93837740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556c05958c40 con 0x556c05a43320 2024-05-09T08:47:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae93837740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x556c05a46290 con 0x556c05a38c50 2024-05-09T08:47:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae8a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 0x556c05a401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae93837740 10 client.0 _ll_register_callbacks cb 0x556c056b0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae8ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a43320 0x556c05a40c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae8a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 0x556c05a401e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33484/0 (socket says 172.21.15.107:33484) 2024-05-09T08:47:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae8ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a43320 0x556c05a40c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:39860/0 (socket says 172.21.15.107:39860) 2024-05-09T08:47:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae8a59c640 1 -- 172.21.15.107:0/2917349503 learned_addr learned my addr 172.21.15.107:0/2917349503 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae89d9b640 1 --2- 172.21.15.107:0/2917349503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38c50 0x556c05a40720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae8a59c640 1 -- 172.21.15.107:0/2917349503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38c50 msgr2=0x556c05a40720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae8a59c640 1 --2- 172.21.15.107:0/2917349503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38c50 0x556c05a40720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae8a59c640 1 -- 172.21.15.107:0/2917349503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a43320 msgr2=0x556c05a40c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.128+0000 7fae8a59c640 1 --2- 172.21.15.107:0/2917349503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a43320 0x556c05a40c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.129+0000 7fae8a59c640 1 -- 172.21.15.107:0/2917349503 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c0577b7d0 con 0x556c05a38330 2024-05-09T08:47:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.129+0000 7fae89d9b640 1 --2- 172.21.15.107:0/2917349503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38c50 0x556c05a40720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.129+0000 7fae8ad9d640 1 --2- 172.21.15.107:0/2917349503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a43320 0x556c05a40c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.129+0000 7fae8a59c640 1 --2- 172.21.15.107:0/2917349503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 0x556c05a401e0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fae8000c010 tx=0x7fae8001c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f2b16a66f0e1e07 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.129+0000 7fae71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:47:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.129+0000 7fae71ffb640 1 -- 172.21.15.107:0/2917349503 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80005450 con 0x556c05a38330 2024-05-09T08:47:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.129+0000 7fae71ffb640 1 -- 172.21.15.107:0/2917349503 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7fae800055f0 con 0x556c05a38330 2024-05-09T08:47:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.129+0000 7fae93837740 1 -- 172.21.15.107:0/2917349503 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae80011db0 con 0x556c05a38330 2024-05-09T08:47:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.129+0000 7fae71ffb640 1 -- 172.21.15.107:0/2917349503 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fae80032050 con 0x556c05a38330 2024-05-09T08:47:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.129+0000 7fae93837740 10 client.19188 fetch_fsmap learned FSMap version 29 2024-05-09T08:47:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.129+0000 7fae93837740 1 -- 172.21.15.107:0/2917349503 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556c05a52280 con 0x556c05a38330 2024-05-09T08:47:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.129+0000 7fae71ffb640 1 -- 172.21.15.107:0/2917349503 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae80011db0 con 0x556c05a38330 2024-05-09T08:47:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.130+0000 7fae71ffb640 1 -- 172.21.15.107:0/2917349503 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fae80004a60 con 0x556c05a38330 2024-05-09T08:47:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.130+0000 7fae93837740 10 client.19188 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:47:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.130+0000 7fae93837740 10 client.19188 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:47:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.130+0000 7fae93837740 20 client.19188 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.130+0000 7fae93837740 1 --2- 172.21.15.107:0/2917349503 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x556c05a536e0 0x556c05a55ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.130+0000 7fae93837740 4 client.19188 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:47:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.130+0000 7fae93837740 1 -- 172.21.15.107:0/2917349503 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c059ab370 con 0x556c05a536e0 2024-05-09T08:47:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.130+0000 7fae89d9b640 1 --2- 172.21.15.107:0/2917349503 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x556c05a536e0 0x556c05a55ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.130+0000 7fae71ffb640 10 client.19188 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:47:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.130+0000 7fae89d9b640 1 --2- 172.21.15.107:0/2917349503 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x556c05a536e0 0x556c05a55ad0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=404d91f8481f623e server_cookie=8f8dca998dab7baa in_seq=0 out_seq=0 2024-05-09T08:47:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.131+0000 7fae71ffb640 1 -- 172.21.15.107:0/2917349503 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556c059ab370 con 0x556c05a536e0 2024-05-09T08:47:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.131+0000 7fae71ffb640 10 client.19188 handle_command_reply: tid=0 2024-05-09T08:47:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.131+0000 7fae93837740 1 client.19188 shutdown 2024-05-09T08:47:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.131+0000 7fae93837740 1 -- 172.21.15.107:0/2917349503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 msgr2=0x556c05a401e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.131+0000 7fae93837740 1 --2- 172.21.15.107:0/2917349503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 0x556c05a401e0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fae8000c010 tx=0x7fae8001c730 comp rx=0 tx=0).stop 2024-05-09T08:47:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.131+0000 7fae93837740 1 -- 172.21.15.107:0/2917349503 shutdown_connections 2024-05-09T08:47:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.131+0000 7fae93837740 1 --2- 172.21.15.107:0/2917349503 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x556c05a536e0 0x556c05a55ad0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.131+0000 7fae71ffb640 0 client.19188 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:47:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.132+0000 7fae93837740 1 --2- 172.21.15.107:0/2917349503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a43320 0x556c05a40c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.132+0000 7fae93837740 1 --2- 172.21.15.107:0/2917349503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38c50 0x556c05a40720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.132+0000 7fae93837740 1 --2- 172.21.15.107:0/2917349503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 0x556c05a401e0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.132+0000 7fae93837740 1 -- 172.21.15.107:0/2917349503 >> 172.21.15.107:0/2917349503 conn(0x556c05a34650 msgr2=0x556c05a3a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.132+0000 7fae93837740 1 -- 172.21.15.107:0/2917349503 shutdown_connections 2024-05-09T08:47:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.132+0000 7fae93837740 1 -- 172.21.15.107:0/2917349503 wait complete. 2024-05-09T08:47:37.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.132+0000 7fae93837740 20 client.19188 trim_cache size 0 max 16384 2024-05-09T08:47:37.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.148+0000 7fae93837740 1 Processor -- start 2024-05-09T08:47:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.148+0000 7fae93837740 1 -- start start 2024-05-09T08:47:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae93837740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38330 0x556c05a42910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae93837740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a38c50 0x556c05a4f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae93837740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a43320 0x556c05a3f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae8a59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38330 0x556c05a42910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae8a59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38330 0x556c05a42910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:50556/0 (socket says 172.21.15.107:50556) 2024-05-09T08:47:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae8a59c640 1 -- 172.21.15.107:0/2464052308 learned_addr learned my addr 172.21.15.107:0/2464052308 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae93837740 1 -- 172.21.15.107:0/2464052308 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fae6c012990 con 0x556c05a43320 2024-05-09T08:47:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae93837740 1 -- 172.21.15.107:0/2464052308 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fae6c010260 con 0x556c05a38c50 2024-05-09T08:47:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae93837740 1 -- 172.21.15.107:0/2464052308 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fae6c00e0d0 con 0x556c05a38330 2024-05-09T08:47:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae89d9b640 1 --2- 172.21.15.107:0/2464052308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a38c50 0x556c05a4f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae8ad9d640 1 --2- 172.21.15.107:0/2464052308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a43320 0x556c05a3f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae89d9b640 1 -- 172.21.15.107:0/2464052308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38330 msgr2=0x556c05a42910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae89d9b640 1 --2- 172.21.15.107:0/2464052308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38330 0x556c05a42910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae89d9b640 1 -- 172.21.15.107:0/2464052308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a43320 msgr2=0x556c05a3f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae89d9b640 1 --2- 172.21.15.107:0/2464052308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a43320 0x556c05a3f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae89d9b640 1 -- 172.21.15.107:0/2464052308 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c05a3afb0 con 0x556c05a38c50 2024-05-09T08:47:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.149+0000 7fae8a59c640 1 --2- 172.21.15.107:0/2464052308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38330 0x556c05a42910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.150+0000 7fae89d9b640 1 --2- 172.21.15.107:0/2464052308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a38c50 0x556c05a4f030 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fae7c0cc650 tx=0x7fae7c0ce040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cd958c70f432602 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.150+0000 7fae73fff640 1 -- 172.21.15.107:0/2464052308 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c01d040 con 0x556c05a38c50 2024-05-09T08:47:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.150+0000 7fae73fff640 1 -- 172.21.15.107:0/2464052308 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae7c0a40c0 con 0x556c05a38c50 2024-05-09T08:47:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.150+0000 7fae73fff640 1 -- 172.21.15.107:0/2464052308 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c097250 con 0x556c05a38c50 2024-05-09T08:47:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.150+0000 7fae93837740 1 -- 172.21.15.107:0/2464052308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a38c50 msgr2=0x556c05a4f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.150+0000 7fae93837740 1 --2- 172.21.15.107:0/2464052308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a38c50 0x556c05a4f030 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fae7c0cc650 tx=0x7fae7c0ce040 comp rx=0 tx=0).stop 2024-05-09T08:47:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.151+0000 7fae93837740 1 -- 172.21.15.107:0/2464052308 shutdown_connections 2024-05-09T08:47:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.151+0000 7fae93837740 1 --2- 172.21.15.107:0/2464052308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a43320 0x556c05a3f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.151+0000 7fae93837740 1 --2- 172.21.15.107:0/2464052308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a38c50 0x556c05a4f030 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.151+0000 7fae93837740 1 --2- 172.21.15.107:0/2464052308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38330 0x556c05a42910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.151+0000 7fae93837740 1 -- 172.21.15.107:0/2464052308 >> 172.21.15.107:0/2464052308 conn(0x556c05a34650 msgr2=0x556c05a3eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.151+0000 7fae93837740 1 -- 172.21.15.107:0/2464052308 shutdown_connections 2024-05-09T08:47:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.151+0000 7fae93837740 1 -- 172.21.15.107:0/2464052308 wait complete. 2024-05-09T08:47:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.151+0000 7fae93837740 1 -- start start 2024-05-09T08:47:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.151+0000 7fae93837740 1 Processor -- start 2024-05-09T08:47:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.151+0000 7fae93837740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 0x556c05a4b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.151+0000 7fae93837740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38c50 0x556c05a4ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.151+0000 7fae93837740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a43320 0x556c05a3c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.151+0000 7fae93837740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fae6c00e720 con 0x556c05a38330 2024-05-09T08:47:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.151+0000 7fae93837740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fae6c00fb90 con 0x556c05a43320 2024-05-09T08:47:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.151+0000 7fae93837740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fae6c00edf0 con 0x556c05a38c50 2024-05-09T08:47:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae93837740 10 client.0 _ll_register_callbacks cb 0x7fae84009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae89d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38c50 0x556c05a4ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae89d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38c50 0x556c05a4ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:50568/0 (socket says 172.21.15.107:50568) 2024-05-09T08:47:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae89d9b640 1 -- 172.21.15.107:0/1150411351 learned_addr learned my addr 172.21.15.107:0/1150411351 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae8ad9d640 1 --2- 172.21.15.107:0/1150411351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a43320 0x556c05a3c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae8a59c640 1 --2- 172.21.15.107:0/1150411351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 0x556c05a4b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae8a59c640 1 -- 172.21.15.107:0/1150411351 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38c50 msgr2=0x556c05a4ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae8a59c640 1 --2- 172.21.15.107:0/1150411351 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38c50 0x556c05a4ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae8a59c640 1 -- 172.21.15.107:0/1150411351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a43320 msgr2=0x556c05a3c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae8a59c640 1 --2- 172.21.15.107:0/1150411351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a43320 0x556c05a3c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae8a59c640 1 -- 172.21.15.107:0/1150411351 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c05a3c9d0 con 0x556c05a38330 2024-05-09T08:47:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae89d9b640 1 --2- 172.21.15.107:0/1150411351 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38c50 0x556c05a4ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae8a59c640 1 --2- 172.21.15.107:0/1150411351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 0x556c05a4b540 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fae8001cbb0 tx=0x7fae8000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=620f185f91d796ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:47:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae71ffb640 1 -- 172.21.15.107:0/1150411351 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8003a070 con 0x556c05a38330 2024-05-09T08:47:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae71ffb640 1 -- 172.21.15.107:0/1150411351 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7fae80004a70 con 0x556c05a38330 2024-05-09T08:47:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae93837740 1 -- 172.21.15.107:0/1150411351 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae7c0a40c0 con 0x556c05a38330 2024-05-09T08:47:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.152+0000 7fae71ffb640 1 -- 172.21.15.107:0/1150411351 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fae80046050 con 0x556c05a38330 2024-05-09T08:47:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.153+0000 7fae93837740 10 client.19194 fetch_fsmap learned FSMap version 29 2024-05-09T08:47:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.153+0000 7fae93837740 1 -- 172.21.15.107:0/1150411351 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556c05ab9270 con 0x556c05a38330 2024-05-09T08:47:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.153+0000 7fae71ffb640 1 -- 172.21.15.107:0/1150411351 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae7c0a40c0 con 0x556c05a38330 2024-05-09T08:47:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.153+0000 7fae71ffb640 1 -- 172.21.15.107:0/1150411351 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fae80005250 con 0x556c05a38330 2024-05-09T08:47:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.153+0000 7fae93837740 10 client.19194 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:47:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.153+0000 7fae93837740 10 client.19194 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:47:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.153+0000 7fae93837740 20 client.19194 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.153+0000 7fae93837740 1 --2- 172.21.15.107:0/1150411351 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x556c05a536e0 0x556c05abb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.153+0000 7fae93837740 4 client.19194 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:47:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.153+0000 7fae93837740 1 -- 172.21.15.107:0/1150411351 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556c059a56c0 con 0x556c05a536e0 2024-05-09T08:47:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.154+0000 7fae89d9b640 1 --2- 172.21.15.107:0/1150411351 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x556c05a536e0 0x556c05abb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.154+0000 7fae71ffb640 10 client.19194 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:47:37.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.154+0000 7fae89d9b640 1 --2- 172.21.15.107:0/1150411351 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x556c05a536e0 0x556c05abb8e0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a00cd92856d8eeec server_cookie=1605919c371f84fb in_seq=0 out_seq=0 2024-05-09T08:47:37.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.154+0000 7fae71ffb640 1 -- 172.21.15.107:0/1150411351 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556c059a56c0 con 0x556c05a536e0 2024-05-09T08:47:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.154+0000 7fae71ffb640 10 client.19194 handle_command_reply: tid=0 2024-05-09T08:47:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.154+0000 7fae93837740 1 client.19194 shutdown 2024-05-09T08:47:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.154+0000 7fae93837740 1 -- 172.21.15.107:0/1150411351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 msgr2=0x556c05a4b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.154+0000 7fae93837740 1 --2- 172.21.15.107:0/1150411351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 0x556c05a4b540 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fae8001cbb0 tx=0x7fae8000b390 comp rx=0 tx=0).stop 2024-05-09T08:47:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.154+0000 7fae93837740 1 -- 172.21.15.107:0/1150411351 shutdown_connections 2024-05-09T08:47:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.154+0000 7fae93837740 1 --2- 172.21.15.107:0/1150411351 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x556c05a536e0 0x556c05abb8e0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.154+0000 7fae93837740 1 --2- 172.21.15.107:0/1150411351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556c05a43320 0x556c05a3c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.154+0000 7fae71ffb640 0 client.19194 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:47:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.154+0000 7fae93837740 1 --2- 172.21.15.107:0/1150411351 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556c05a38c50 0x556c05a4ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.154+0000 7fae93837740 1 --2- 172.21.15.107:0/1150411351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556c05a38330 0x556c05a4b540 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.154+0000 7fae93837740 1 -- 172.21.15.107:0/1150411351 >> 172.21.15.107:0/1150411351 conn(0x556c05a34650 msgr2=0x556c05a32f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.155+0000 7fae93837740 1 -- 172.21.15.107:0/1150411351 shutdown_connections 2024-05-09T08:47:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.155+0000 7fae93837740 1 -- 172.21.15.107:0/1150411351 wait complete. 2024-05-09T08:47:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.155+0000 7fae93837740 20 client.19194 trim_cache size 0 max 16384 2024-05-09T08:47:37.161 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:47:37.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.155+0000 7fae90a4d640 1 -- 172.21.15.107:0/3769528926 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae740a2dc0 msgr2=0x7fae740a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.155+0000 7fae90a4d640 1 --2- 172.21.15.107:0/3769528926 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae740a2dc0 0x7fae740a5280 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fae84005e30 tx=0x7fae8402a910 comp rx=0 tx=0).stop 2024-05-09T08:47:37.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.155+0000 7fae90a4d640 1 -- 172.21.15.107:0/3769528926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae8c110080 msgr2=0x7fae8c111720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.155+0000 7fae90a4d640 1 --2- 172.21.15.107:0/3769528926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae8c110080 0x7fae8c111720 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00c8c0 tx=0x7fae7c00cd80 comp rx=0 tx=0).stop 2024-05-09T08:47:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.155+0000 7fae90a4d640 1 -- 172.21.15.107:0/3769528926 shutdown_connections 2024-05-09T08:47:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.155+0000 7fae90a4d640 1 --2- 172.21.15.107:0/3769528926 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae740a2dc0 0x7fae740a5280 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.155+0000 7fae90a4d640 1 --2- 172.21.15.107:0/3769528926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae8c11b010 0x7fae8c111c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.155+0000 7fae90a4d640 1 --2- 172.21.15.107:0/3769528926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae8c110080 0x7fae8c111720 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.156+0000 7fae90a4d640 1 --2- 172.21.15.107:0/3769528926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae8c10e0b0 0x7fae8c118650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.156+0000 7fae90a4d640 1 -- 172.21.15.107:0/3769528926 >> 172.21.15.107:0/3769528926 conn(0x7fae8c06c940 msgr2=0x7fae8c072330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.156+0000 7fae90a4d640 1 -- 172.21.15.107:0/3769528926 shutdown_connections 2024-05-09T08:47:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.156+0000 7fae90a4d640 1 -- 172.21.15.107:0/3769528926 wait complete. 2024-05-09T08:47:37.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:47:37.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.406+0000 7fdd3c6cd640 1 -- 172.21.15.107:0/4065237470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd3410e0b0 msgr2=0x7fdd3410e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.406+0000 7fdd3c6cd640 1 --2- 172.21.15.107:0/4065237470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd3410e0b0 0x7fdd3410e530 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009890 tx=0x7fdd3002a390 comp rx=0 tx=0).stop 2024-05-09T08:47:37.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.407+0000 7fdd3c6cd640 1 -- 172.21.15.107:0/4065237470 shutdown_connections 2024-05-09T08:47:37.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.407+0000 7fdd3c6cd640 1 --2- 172.21.15.107:0/4065237470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd3411b010 0x7fdd3411d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.407+0000 7fdd3c6cd640 1 --2- 172.21.15.107:0/4065237470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd3410e0b0 0x7fdd3410e530 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.407+0000 7fdd3c6cd640 1 --2- 172.21.15.107:0/4065237470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd34110080 0x7fdd3410db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.407+0000 7fdd3c6cd640 1 -- 172.21.15.107:0/4065237470 >> 172.21.15.107:0/4065237470 conn(0x7fdd3406c940 msgr2=0x7fdd3406cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.407+0000 7fdd3c6cd640 1 -- 172.21.15.107:0/4065237470 shutdown_connections 2024-05-09T08:47:37.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.407+0000 7fdd3c6cd640 1 -- 172.21.15.107:0/4065237470 wait complete. 2024-05-09T08:47:37.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.408+0000 7fdd3c6cd640 1 Processor -- start 2024-05-09T08:47:37.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.408+0000 7fdd3c6cd640 1 -- start start 2024-05-09T08:47:37.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.409+0000 7fdd3c6cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd3410e0b0 0x7fdd341a6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.409+0000 7fdd3c6cd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd34110080 0x7fdd341a7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.409+0000 7fdd3c6cd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd3411b010 0x7fdd341ab710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.409+0000 7fdd3c6cd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdd341208d0 con 0x7fdd3410e0b0 2024-05-09T08:47:37.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.409+0000 7fdd3c6cd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdd34120750 con 0x7fdd34110080 2024-05-09T08:47:37.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.409+0000 7fdd3c6cd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdd34120a50 con 0x7fdd3411b010 2024-05-09T08:47:37.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.409+0000 7fdd3a469640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd3410e0b0 0x7fdd341a6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.409+0000 7fdd3a469640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd3410e0b0 0x7fdd341a6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33506/0 (socket says 172.21.15.107:33506) 2024-05-09T08:47:37.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.409+0000 7fdd3a469640 1 -- 172.21.15.107:0/15060692 learned_addr learned my addr 172.21.15.107:0/15060692 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:37.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.410+0000 7fdd3a469640 1 -- 172.21.15.107:0/15060692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd3411b010 msgr2=0x7fdd341ab710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:47:37.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.410+0000 7fdd3a469640 1 --2- 172.21.15.107:0/15060692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd3411b010 0x7fdd341ab710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.410+0000 7fdd3a469640 1 -- 172.21.15.107:0/15060692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd34110080 msgr2=0x7fdd341a7390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:47:37.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.410+0000 7fdd3a469640 1 --2- 172.21.15.107:0/15060692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd34110080 0x7fdd341a7390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.410+0000 7fdd3a469640 1 -- 172.21.15.107:0/15060692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd341abe40 con 0x7fdd3410e0b0 2024-05-09T08:47:37.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.410+0000 7fdd3a469640 1 --2- 172.21.15.107:0/15060692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd3410e0b0 0x7fdd341a6e50 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c009d50 tx=0x7fdd2c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.411+0000 7fdd2b7fe640 1 -- 172.21.15.107:0/15060692 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c003b70 con 0x7fdd3410e0b0 2024-05-09T08:47:37.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.411+0000 7fdd2b7fe640 1 -- 172.21.15.107:0/15060692 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd2c003d10 con 0x7fdd3410e0b0 2024-05-09T08:47:37.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.411+0000 7fdd2b7fe640 1 -- 172.21.15.107:0/15060692 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c010960 con 0x7fdd3410e0b0 2024-05-09T08:47:37.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.411+0000 7fdd3c6cd640 1 -- 172.21.15.107:0/15060692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd341ac130 con 0x7fdd3410e0b0 2024-05-09T08:47:37.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.411+0000 7fdd3c6cd640 1 -- 172.21.15.107:0/15060692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd341b39c0 con 0x7fdd3410e0b0 2024-05-09T08:47:37.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.416+0000 7fdd2b7fe640 1 -- 172.21.15.107:0/15060692 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fdd2c010b00 con 0x7fdd3410e0b0 2024-05-09T08:47:37.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.417+0000 7fdd2b7fe640 1 --2- 172.21.15.107:0/15060692 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdd200a2e40 0x7fdd200a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.417+0000 7fdd2b7fe640 1 -- 172.21.15.107:0/15060692 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7fdd2c0d03c0 con 0x7fdd3410e0b0 2024-05-09T08:47:37.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.420+0000 7fdd3f3dc740 1 Processor -- start 2024-05-09T08:47:37.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.420+0000 7fdd3f3dc740 1 -- start start 2024-05-09T08:47:37.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.420+0000 7fdd3f3dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 0x556cecc97710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.420+0000 7fdd3f3dc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecc97c50 0x556cecca1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.420+0000 7fdd3f3dc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecca2320 0x556cecca4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.420+0000 7fdd3f3dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556cecbfcc40 con 0x556cecc97330 2024-05-09T08:47:37.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.420+0000 7fdd3f3dc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556cecba4580 con 0x556cecca2320 2024-05-09T08:47:37.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.420+0000 7fdd3f3dc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x556cecbf8c00 con 0x556cecc97c50 2024-05-09T08:47:37.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.421+0000 7fdd3ac6a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecca2320 0x556cecca4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.421+0000 7fdd3ac6a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecca2320 0x556cecca4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:39900/0 (socket says 172.21.15.107:39900) 2024-05-09T08:47:37.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.421+0000 7fdd3ac6a640 1 -- 172.21.15.107:0/1721036876 learned_addr learned my addr 172.21.15.107:0/1721036876 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:37.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.422+0000 7fdd3ac6a640 1 -- 172.21.15.107:0/1721036876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecc97c50 msgr2=0x556cecca1de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:47:37.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.422+0000 7fdd3ac6a640 1 --2- 172.21.15.107:0/1721036876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecc97c50 0x556cecca1de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.422+0000 7fdd3ac6a640 1 -- 172.21.15.107:0/1721036876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 msgr2=0x556cecc97710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.422+0000 7fdd3ac6a640 1 --2- 172.21.15.107:0/1721036876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 0x556cecc97710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.422+0000 7fdd3ac6a640 1 -- 172.21.15.107:0/1721036876 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cec9103f0 con 0x556cecca2320 2024-05-09T08:47:37.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.422+0000 7fdd3ac6a640 1 --2- 172.21.15.107:0/1721036876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecca2320 0x556cecca4710 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009a50 tx=0x7fdd24009f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18fcfe04af1e57bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.424+0000 7fdd28ff9640 1 -- 172.21.15.107:0/1721036876 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd24030c20 con 0x556cecca2320 2024-05-09T08:47:37.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.424+0000 7fdd28ff9640 1 -- 172.21.15.107:0/1721036876 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd24030dc0 con 0x556cecca2320 2024-05-09T08:47:37.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.424+0000 7fdd28ff9640 1 -- 172.21.15.107:0/1721036876 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd24032440 con 0x556cecca2320 2024-05-09T08:47:37.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.425+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/1721036876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecca2320 msgr2=0x556cecca4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.425+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/1721036876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecca2320 0x556cecca4710 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009a50 tx=0x7fdd24009f10 comp rx=0 tx=0).stop 2024-05-09T08:47:37.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.425+0000 7fdd39c68640 1 --2- 172.21.15.107:0/15060692 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdd200a2e40 0x7fdd200a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.425+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/1721036876 shutdown_connections 2024-05-09T08:47:37.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.425+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/1721036876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecca2320 0x556cecca4710 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.425+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/1721036876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecc97c50 0x556cecca1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.425+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/1721036876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 0x556cecc97710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.425+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/1721036876 >> 172.21.15.107:0/1721036876 conn(0x556cecc93650 msgr2=0x556cecc93a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.426+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/1721036876 shutdown_connections 2024-05-09T08:47:37.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.426+0000 7fdd39c68640 1 --2- 172.21.15.107:0/15060692 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdd200a2e40 0x7fdd200a5300 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdd300303f0 tx=0x7fdd30000bc0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.426+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/1721036876 wait complete. 2024-05-09T08:47:37.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.427+0000 7fdd3f3dc740 1 -- start start 2024-05-09T08:47:37.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.427+0000 7fdd3f3dc740 1 Processor -- start 2024-05-09T08:47:37.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.428+0000 7fdd3f3dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 0x556cecca1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.428+0000 7fdd3f3dc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecc97c50 0x556cecc9f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.428+0000 7fdd3f3dc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecca2320 0x556cecc9f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.428+0000 7fdd3f3dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556cecca51b0 con 0x556cecc97330 2024-05-09T08:47:37.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.428+0000 7fdd3f3dc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556cecbb7c40 con 0x556cecca2320 2024-05-09T08:47:37.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.428+0000 7fdd3f3dc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x556cecca5330 con 0x556cecc97c50 2024-05-09T08:47:37.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.428+0000 7fdd3f3dc740 10 client.0 _ll_register_callbacks cb 0x556cec90fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:37.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.428+0000 7fdd3a469640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 0x556cecca1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.428+0000 7fdd3a469640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 0x556cecca1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33524/0 (socket says 172.21.15.107:33524) 2024-05-09T08:47:37.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.428+0000 7fdd3a469640 1 -- 172.21.15.107:0/3004470388 learned_addr learned my addr 172.21.15.107:0/3004470388 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:37.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.429+0000 7fdd39c68640 1 --2- 172.21.15.107:0/3004470388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecc97c50 0x556cecc9f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.429+0000 7fdd3ac6a640 1 --2- 172.21.15.107:0/3004470388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecca2320 0x556cecc9f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.429+0000 7fdd3a469640 1 -- 172.21.15.107:0/3004470388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecc97c50 msgr2=0x556cecc9f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.429+0000 7fdd3a469640 1 --2- 172.21.15.107:0/3004470388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecc97c50 0x556cecc9f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.429+0000 7fdd3a469640 1 -- 172.21.15.107:0/3004470388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecca2320 msgr2=0x556cecc9f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.429+0000 7fdd3a469640 1 --2- 172.21.15.107:0/3004470388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecca2320 0x556cecc9f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.429+0000 7fdd3a469640 1 -- 172.21.15.107:0/3004470388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cec9da7d0 con 0x556cecc97330 2024-05-09T08:47:37.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.431+0000 7fdd3a469640 1 --2- 172.21.15.107:0/3004470388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 0x556cecca1d90 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c0a4b20 tx=0x7fdd2c09cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53bd8944b2095971 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.431+0000 7fdd1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:47:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.435+0000 7fdd1dffb640 1 -- 172.21.15.107:0/3004470388 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c09c3e0 con 0x556cecc97330 2024-05-09T08:47:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.435+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/3004470388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd24030dc0 con 0x556cecc97330 2024-05-09T08:47:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.435+0000 7fdd1dffb640 1 -- 172.21.15.107:0/3004470388 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7fdd2c004660 con 0x556cecc97330 2024-05-09T08:47:37.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.436+0000 7fdd1dffb640 1 -- 172.21.15.107:0/3004470388 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdd2c0a0790 con 0x556cecc97330 2024-05-09T08:47:37.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.438+0000 7fdd3f3dc740 10 client.19203 fetch_fsmap learned FSMap version 29 2024-05-09T08:47:37.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.438+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/3004470388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556ceccb1300 con 0x556cecc97330 2024-05-09T08:47:37.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.442+0000 7fdd1dffb640 1 -- 172.21.15.107:0/3004470388 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd24030dc0 con 0x556cecc97330 2024-05-09T08:47:37.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.442+0000 7fdd1dffb640 1 -- 172.21.15.107:0/3004470388 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fdd2c020070 con 0x556cecc97330 2024-05-09T08:47:37.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.442+0000 7fdd3f3dc740 10 client.19203 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:47:37.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.442+0000 7fdd3f3dc740 10 client.19203 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:47:37.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.442+0000 7fdd3f3dc740 20 client.19203 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:37.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.442+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/3004470388 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x556ceccb2b90 0x556ceccb4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.442+0000 7fdd3f3dc740 4 client.19203 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:47:37.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.442+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/3004470388 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cecc0a370 con 0x556ceccb2b90 2024-05-09T08:47:37.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.443+0000 7fdd39c68640 1 --2- 172.21.15.107:0/3004470388 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x556ceccb2b90 0x556ceccb4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.445+0000 7fdd39c68640 1 --2- 172.21.15.107:0/3004470388 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x556ceccb2b90 0x556ceccb4f80 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=54a5a9f5bddd6089 server_cookie=98e434b0f8f2022 in_seq=0 out_seq=0 2024-05-09T08:47:37.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.445+0000 7fdd1dffb640 10 client.19203 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:47:37.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.450+0000 7fdd1dffb640 1 -- 172.21.15.107:0/3004470388 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556cecc0a370 con 0x556ceccb2b90 2024-05-09T08:47:37.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.450+0000 7fdd1dffb640 10 client.19203 handle_command_reply: tid=0 2024-05-09T08:47:37.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.450+0000 7fdd3f3dc740 1 client.19203 shutdown 2024-05-09T08:47:37.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.457+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/3004470388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 msgr2=0x556cecca1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.457+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/3004470388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 0x556cecca1d90 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c0a4b20 tx=0x7fdd2c09cae0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.461+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/3004470388 shutdown_connections 2024-05-09T08:47:37.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.461+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/3004470388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecca2320 0x556cecc9f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.461+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/3004470388 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x556ceccb2b90 0x556ceccb4f80 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.461+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/3004470388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecc97c50 0x556cecc9f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.461+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/3004470388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 0x556cecca1d90 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.461+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/3004470388 >> 172.21.15.107:0/3004470388 conn(0x556cecc93650 msgr2=0x556cecc99450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.461+0000 7fdd1dffb640 0 client.19203 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:47:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.461+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/3004470388 shutdown_connections 2024-05-09T08:47:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.461+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/3004470388 wait complete. 2024-05-09T08:47:37.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.461+0000 7fdd3f3dc740 20 client.19203 trim_cache size 0 max 16384 2024-05-09T08:47:37.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.481+0000 7fdd3f3dc740 1 Processor -- start 2024-05-09T08:47:37.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.481+0000 7fdd3f3dc740 1 -- start start 2024-05-09T08:47:37.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.481+0000 7fdd3f3dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 0x556cecc92630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.481+0000 7fdd3f3dc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecc97c50 0x556cecca1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.481+0000 7fdd3f3dc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecca2320 0x556cecca1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.481+0000 7fdd3f3dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdd34080550 con 0x556cecc97330 2024-05-09T08:47:37.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.481+0000 7fdd3f3dc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3407de20 con 0x556cecc97c50 2024-05-09T08:47:37.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.481+0000 7fdd3f3dc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdd34068690 con 0x556cecca2320 2024-05-09T08:47:37.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.481+0000 7fdd3a469640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 0x556cecc92630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.481+0000 7fdd3a469640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 0x556cecc92630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33538/0 (socket says 172.21.15.107:33538) 2024-05-09T08:47:37.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.481+0000 7fdd3a469640 1 -- 172.21.15.107:0/83678521 learned_addr learned my addr 172.21.15.107:0/83678521 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:37.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.482+0000 7fdd3ac6a640 1 --2- 172.21.15.107:0/83678521 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecca2320 0x556cecca1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.482+0000 7fdd39c68640 1 --2- 172.21.15.107:0/83678521 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecc97c50 0x556cecca1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.482+0000 7fdd3a469640 1 -- 172.21.15.107:0/83678521 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecca2320 msgr2=0x556cecca1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.482+0000 7fdd3a469640 1 --2- 172.21.15.107:0/83678521 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecca2320 0x556cecca1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.482+0000 7fdd3a469640 1 -- 172.21.15.107:0/83678521 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecc97c50 msgr2=0x556cecca1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.482+0000 7fdd3a469640 1 --2- 172.21.15.107:0/83678521 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecc97c50 0x556cecca1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.482+0000 7fdd3a469640 1 -- 172.21.15.107:0/83678521 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cecc96f00 con 0x556cecc97330 2024-05-09T08:47:37.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.482+0000 7fdd39c68640 1 --2- 172.21.15.107:0/83678521 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecc97c50 0x556cecca1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:37.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.482+0000 7fdd3a469640 1 --2- 172.21.15.107:0/83678521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 0x556cecc92630 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00ffd0 tx=0x7fdd2c09a8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f9a780c8e2d85c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.482+0000 7fdd1ffff640 1 -- 172.21.15.107:0/83678521 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c09c3e0 con 0x556cecc97330 2024-05-09T08:47:37.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.482+0000 7fdd1ffff640 1 -- 172.21.15.107:0/83678521 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd2c0a5050 con 0x556cecc97330 2024-05-09T08:47:37.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.482+0000 7fdd1ffff640 1 -- 172.21.15.107:0/83678521 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c09b370 con 0x556cecc97330 2024-05-09T08:47:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.483+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/83678521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 msgr2=0x556cecc92630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.483+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/83678521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 0x556cecc92630 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00ffd0 tx=0x7fdd2c09a8f0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.483+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/83678521 shutdown_connections 2024-05-09T08:47:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.483+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/83678521 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecca2320 0x556cecca1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.483+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/83678521 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecc97c50 0x556cecca1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.483+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/83678521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97330 0x556cecc92630 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.484+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/83678521 >> 172.21.15.107:0/83678521 conn(0x556cecc93650 msgr2=0x556cecc91f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.484+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/83678521 shutdown_connections 2024-05-09T08:47:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.484+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/83678521 wait complete. 2024-05-09T08:47:37.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.484+0000 7fdd3f3dc740 1 -- start start 2024-05-09T08:47:37.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.484+0000 7fdd3f3dc740 1 Processor -- start 2024-05-09T08:47:37.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.484+0000 7fdd3f3dc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecc97330 0x556cecc9ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.484+0000 7fdd3f3dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97c50 0x556ceccaa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.484+0000 7fdd3f3dc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecca2320 0x556cecca3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.484+0000 7fdd3f3dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3407c2e0 con 0x556cecc97c50 2024-05-09T08:47:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.484+0000 7fdd3f3dc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3407d750 con 0x556cecc97330 2024-05-09T08:47:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.484+0000 7fdd3f3dc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3407c9b0 con 0x556cecca2320 2024-05-09T08:47:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.484+0000 7fdd39c68640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97c50 0x556ceccaa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.484+0000 7fdd39c68640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97c50 0x556ceccaa5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33540/0 (socket says 172.21.15.107:33540) 2024-05-09T08:47:37.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.484+0000 7fdd39c68640 1 -- 172.21.15.107:0/140453029 learned_addr learned my addr 172.21.15.107:0/140453029 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.485+0000 7fdd3f3dc740 10 client.0 _ll_register_callbacks cb 0x7fdd2c09bfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.485+0000 7fdd3a469640 1 --2- 172.21.15.107:0/140453029 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecc97330 0x556cecc9ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.485+0000 7fdd3ac6a640 1 --2- 172.21.15.107:0/140453029 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecca2320 0x556cecca3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.485+0000 7fdd39c68640 1 -- 172.21.15.107:0/140453029 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecca2320 msgr2=0x556cecca3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.485+0000 7fdd39c68640 1 --2- 172.21.15.107:0/140453029 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecca2320 0x556cecca3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.485+0000 7fdd39c68640 1 -- 172.21.15.107:0/140453029 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecc97330 msgr2=0x556cecc9ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.485+0000 7fdd39c68640 1 --2- 172.21.15.107:0/140453029 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecc97330 0x556cecc9ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.485+0000 7fdd39c68640 1 -- 172.21.15.107:0/140453029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cecca3a80 con 0x556cecc97c50 2024-05-09T08:47:37.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.485+0000 7fdd3a469640 1 --2- 172.21.15.107:0/140453029 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecc97330 0x556cecc9ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:37.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.485+0000 7fdd39c68640 1 --2- 172.21.15.107:0/140453029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97c50 0x556ceccaa5f0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009890 tx=0x7fdd30005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5fb20fdfc7c3ade server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.485+0000 7fdd1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:47:37.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.485+0000 7fdd1dffb640 1 -- 172.21.15.107:0/140453029 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30037040 con 0x556cecc97c50 2024-05-09T08:47:37.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.485+0000 7fdd1dffb640 1 -- 172.21.15.107:0/140453029 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7fdd300043c0 con 0x556cecc97c50 2024-05-09T08:47:37.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.485+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/140453029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd2c0a5050 con 0x556cecc97c50 2024-05-09T08:47:37.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.486+0000 7fdd1dffb640 1 -- 172.21.15.107:0/140453029 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdd30030450 con 0x556cecc97c50 2024-05-09T08:47:37.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.486+0000 7fdd3f3dc740 10 client.19209 fetch_fsmap learned FSMap version 29 2024-05-09T08:47:37.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.486+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/140453029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556cecd197b0 con 0x556cecc97c50 2024-05-09T08:47:37.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.486+0000 7fdd1dffb640 1 -- 172.21.15.107:0/140453029 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd2c0a5050 con 0x556cecc97c50 2024-05-09T08:47:37.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.486+0000 7fdd1dffb640 1 -- 172.21.15.107:0/140453029 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fdd3003a030 con 0x556cecc97c50 2024-05-09T08:47:37.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.486+0000 7fdd3f3dc740 10 client.19209 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:47:37.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.486+0000 7fdd3f3dc740 10 client.19209 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:47:37.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.486+0000 7fdd3f3dc740 20 client.19209 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:37.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.486+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/140453029 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x556cecd19fb0 0x556cecd1c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.486+0000 7fdd3f3dc740 4 client.19209 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:47:37.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.486+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/140453029 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556cecc046c0 con 0x556cecd19fb0 2024-05-09T08:47:37.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.487+0000 7fdd3a469640 1 --2- 172.21.15.107:0/140453029 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x556cecd19fb0 0x556cecd1c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.487+0000 7fdd1dffb640 10 client.19209 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:47:37.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.487+0000 7fdd3a469640 1 --2- 172.21.15.107:0/140453029 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x556cecd19fb0 0x556cecd1c3d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=31fe3a700b64eaf4 server_cookie=c0e3eeb9e50f4477 in_seq=0 out_seq=0 2024-05-09T08:47:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.487+0000 7fdd1dffb640 1 -- 172.21.15.107:0/140453029 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556cecc046c0 con 0x556cecd19fb0 2024-05-09T08:47:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.487+0000 7fdd1dffb640 10 client.19209 handle_command_reply: tid=0 2024-05-09T08:47:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.487+0000 7fdd3f3dc740 1 client.19209 shutdown 2024-05-09T08:47:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.487+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/140453029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97c50 msgr2=0x556ceccaa5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.487+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/140453029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97c50 0x556ceccaa5f0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009890 tx=0x7fdd30005f90 comp rx=0 tx=0).stop 2024-05-09T08:47:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.488+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/140453029 shutdown_connections 2024-05-09T08:47:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.488+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/140453029 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x556cecd19fb0 0x556cecd1c3d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.488+0000 7fdd1dffb640 0 client.19209 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:47:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.488+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/140453029 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556cecca2320 0x556cecca3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.488+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/140453029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cecc97c50 0x556ceccaa5f0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.488+0000 7fdd3f3dc740 1 --2- 172.21.15.107:0/140453029 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556cecc97330 0x556cecc9ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.488+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/140453029 >> 172.21.15.107:0/140453029 conn(0x556cecc93650 msgr2=0x556cecc927b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.488+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/140453029 shutdown_connections 2024-05-09T08:47:37.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.488+0000 7fdd3f3dc740 1 -- 172.21.15.107:0/140453029 wait complete. 2024-05-09T08:47:37.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.488+0000 7fdd3f3dc740 20 client.19209 trim_cache size 0 max 16384 2024-05-09T08:47:37.505 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:47:37.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.488+0000 7fdd3c6cd640 1 -- 172.21.15.107:0/15060692 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdd200a2e40 msgr2=0x7fdd200a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.488+0000 7fdd3c6cd640 1 --2- 172.21.15.107:0/15060692 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdd200a2e40 0x7fdd200a5300 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdd300303f0 tx=0x7fdd30000bc0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.489+0000 7fdd3c6cd640 1 -- 172.21.15.107:0/15060692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd3410e0b0 msgr2=0x7fdd341a6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.489+0000 7fdd3c6cd640 1 --2- 172.21.15.107:0/15060692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd3410e0b0 0x7fdd341a6e50 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c009d50 tx=0x7fdd2c00c6a0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.489+0000 7fdd3c6cd640 1 -- 172.21.15.107:0/15060692 shutdown_connections 2024-05-09T08:47:37.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.489+0000 7fdd3c6cd640 1 --2- 172.21.15.107:0/15060692 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdd200a2e40 0x7fdd200a5300 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.489+0000 7fdd3c6cd640 1 --2- 172.21.15.107:0/15060692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd3411b010 0x7fdd341ab710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.489+0000 7fdd3c6cd640 1 --2- 172.21.15.107:0/15060692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd34110080 0x7fdd341a7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.489+0000 7fdd3c6cd640 1 --2- 172.21.15.107:0/15060692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd3410e0b0 0x7fdd341a6e50 secure :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c009d50 tx=0x7fdd2c00c6a0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.489+0000 7fdd3c6cd640 1 -- 172.21.15.107:0/15060692 >> 172.21.15.107:0/15060692 conn(0x7fdd3406c940 msgr2=0x7fdd34072330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.490+0000 7fdd3c6cd640 1 -- 172.21.15.107:0/15060692 shutdown_connections 2024-05-09T08:47:37.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.490+0000 7fdd3c6cd640 1 -- 172.21.15.107:0/15060692 wait complete. 2024-05-09T08:47:37.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:47:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.698+0000 7f1e57dc8640 1 -- 172.21.15.107:0/3750977796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e501010c0 msgr2=0x7f1e5010d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.698+0000 7f1e57dc8640 1 --2- 172.21.15.107:0/3750977796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e501010c0 0x7f1e5010d5d0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f1e48009cd0 tx=0x7f1e4802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.698+0000 7f1e57dc8640 1 -- 172.21.15.107:0/3750977796 shutdown_connections 2024-05-09T08:47:37.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.698+0000 7f1e57dc8640 1 --2- 172.21.15.107:0/3750977796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e501010c0 0x7f1e5010d5d0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.698+0000 7f1e57dc8640 1 --2- 172.21.15.107:0/3750977796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e50100700 0x7f1e50100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.698+0000 7f1e57dc8640 1 --2- 172.21.15.107:0/3750977796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e50106710 0x7f1e50106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.698+0000 7f1e57dc8640 1 -- 172.21.15.107:0/3750977796 >> 172.21.15.107:0/3750977796 conn(0x7f1e50077830 msgr2=0x7f1e50077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.698+0000 7f1e57dc8640 1 -- 172.21.15.107:0/3750977796 shutdown_connections 2024-05-09T08:47:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.699+0000 7f1e57dc8640 1 -- 172.21.15.107:0/3750977796 wait complete. 2024-05-09T08:47:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.699+0000 7f1e57dc8640 1 Processor -- start 2024-05-09T08:47:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.699+0000 7f1e57dc8640 1 -- start start 2024-05-09T08:47:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.699+0000 7f1e57dc8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e50100700 0x7f1e50196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.699+0000 7f1e57dc8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e501010c0 0x7f1e50197410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.699+0000 7f1e57dc8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50106710 0x7f1e5019b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.699+0000 7f1e57dc8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1e501109d0 con 0x7f1e50106710 2024-05-09T08:47:37.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.699+0000 7f1e57dc8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1e50110850 con 0x7f1e50100700 2024-05-09T08:47:37.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.699+0000 7f1e57dc8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1e50110b50 con 0x7f1e501010c0 2024-05-09T08:47:37.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.699+0000 7f1e56365640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50106710 0x7f1e5019b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.699+0000 7f1e56365640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50106710 0x7f1e5019b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33558/0 (socket says 172.21.15.107:33558) 2024-05-09T08:47:37.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.699+0000 7f1e56365640 1 -- 172.21.15.107:0/2154592824 learned_addr learned my addr 172.21.15.107:0/2154592824 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:37.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.699+0000 7f1e55363640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e501010c0 0x7f1e50197410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.699+0000 7f1e55b64640 1 --2- 172.21.15.107:0/2154592824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e50100700 0x7f1e50196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.700+0000 7f1e56365640 1 -- 172.21.15.107:0/2154592824 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e501010c0 msgr2=0x7f1e50197410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.700+0000 7f1e56365640 1 --2- 172.21.15.107:0/2154592824 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e501010c0 0x7f1e50197410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.700+0000 7f1e56365640 1 -- 172.21.15.107:0/2154592824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e50100700 msgr2=0x7f1e50196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.700+0000 7f1e56365640 1 --2- 172.21.15.107:0/2154592824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e50100700 0x7f1e50196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.700+0000 7f1e56365640 1 -- 172.21.15.107:0/2154592824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5019bec0 con 0x7f1e50106710 2024-05-09T08:47:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.700+0000 7f1e55363640 1 --2- 172.21.15.107:0/2154592824 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e501010c0 0x7f1e50197410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.700+0000 7f1e56365640 1 --2- 172.21.15.107:0/2154592824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50106710 0x7f1e5019b790 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f1e48033040 tx=0x7f1e48003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.700+0000 7f1e46ffd640 1 -- 172.21.15.107:0/2154592824 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48004040 con 0x7f1e50106710 2024-05-09T08:47:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.700+0000 7f1e57dc8640 1 -- 172.21.15.107:0/2154592824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e5019c150 con 0x7f1e50106710 2024-05-09T08:47:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.700+0000 7f1e46ffd640 1 -- 172.21.15.107:0/2154592824 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e48004690 con 0x7f1e50106710 2024-05-09T08:47:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.700+0000 7f1e46ffd640 1 -- 172.21.15.107:0/2154592824 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48030e10 con 0x7f1e50106710 2024-05-09T08:47:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.700+0000 7f1e57dc8640 1 -- 172.21.15.107:0/2154592824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e501a3a40 con 0x7f1e50106710 2024-05-09T08:47:37.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.702+0000 7f1e46ffd640 1 -- 172.21.15.107:0/2154592824 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1e480305a0 con 0x7f1e50106710 2024-05-09T08:47:37.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.703+0000 7f1e5abb1740 1 Processor -- start 2024-05-09T08:47:37.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.703+0000 7f1e5abb1740 1 -- start start 2024-05-09T08:47:37.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.703+0000 7f1e5abb1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f09926c330 0x55f09926c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.703+0000 7f1e5abb1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f099276de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.703+0000 7f1e46ffd640 1 --2- 172.21.15.107:0/2154592824 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1e3c0a2d00 0x7f1e3c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.703+0000 7f1e5abb1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f099277320 0x55f099279710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e5abb1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f0991d1c40 con 0x55f09926cc50 2024-05-09T08:47:37.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e5abb1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f099179580 con 0x55f099277320 2024-05-09T08:47:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e5abb1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f0991cdc00 con 0x55f09926c330 2024-05-09T08:47:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e55363640 1 --2- 172.21.15.107:0/2154592824 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1e3c0a2d00 0x7f1e3c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e55b64640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f099276de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e55363640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f09926c330 0x55f09926c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e55363640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f09926c330 0x55f09926c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:50632/0 (socket says 172.21.15.107:50632) 2024-05-09T08:47:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e55363640 1 -- 172.21.15.107:0/1537831194 learned_addr learned my addr 172.21.15.107:0/1537831194 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e55b64640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f099276de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33574/0 (socket says 172.21.15.107:33574) 2024-05-09T08:47:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e46ffd640 1 -- 172.21.15.107:0/2154592824 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7f1e4802e030 con 0x7f1e50106710 2024-05-09T08:47:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e56365640 1 --2- 172.21.15.107:0/1537831194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f099277320 0x55f099279710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e55363640 1 -- 172.21.15.107:0/1537831194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f099277320 msgr2=0x55f099279710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e55363640 1 --2- 172.21.15.107:0/1537831194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f099277320 0x55f099279710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e55363640 1 -- 172.21.15.107:0/1537831194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 msgr2=0x55f099276de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e55363640 1 --2- 172.21.15.107:0/1537831194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f099276de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e55363640 1 -- 172.21.15.107:0/1537831194 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f098ee53f0 con 0x55f09926c330 2024-05-09T08:47:37.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e55363640 1 --2- 172.21.15.107:0/2154592824 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1e3c0a2d00 0x7f1e3c0a51c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1e4001ccd0 tx=0x7f1e4001a000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.704+0000 7f1e55363640 1 --2- 172.21.15.107:0/1537831194 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f09926c330 0x55f09926c710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1e40010f00 tx=0x7f1e4001c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72cd297cab8a717d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.750+0000 7f1e3bfff640 1 -- 172.21.15.107:0/1537831194 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e400108d0 con 0x55f09926c330 2024-05-09T08:47:37.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.750+0000 7f1e3bfff640 1 -- 172.21.15.107:0/1537831194 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e40009e10 con 0x55f09926c330 2024-05-09T08:47:37.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.750+0000 7f1e3bfff640 1 -- 172.21.15.107:0/1537831194 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e400283a0 con 0x55f09926c330 2024-05-09T08:47:37.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.751+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1537831194 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f09926c330 msgr2=0x55f09926c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.751+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/1537831194 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f09926c330 0x55f09926c710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1e40010f00 tx=0x7f1e4001c7d0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.752+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1537831194 shutdown_connections 2024-05-09T08:47:37.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.752+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/1537831194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f099277320 0x55f099279710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.752+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/1537831194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f099276de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.752+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/1537831194 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f09926c330 0x55f09926c710 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.752+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1537831194 >> 172.21.15.107:0/1537831194 conn(0x55f099268650 msgr2=0x55f099268a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.754+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1537831194 shutdown_connections 2024-05-09T08:47:37.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.754+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1537831194 wait complete. 2024-05-09T08:47:37.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.754+0000 7f1e5abb1740 1 -- start start 2024-05-09T08:47:37.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.755+0000 7f1e5abb1740 1 Processor -- start 2024-05-09T08:47:37.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.755+0000 7f1e5abb1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 0x55f099274260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.755+0000 7f1e5abb1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f0992747a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.755+0000 7f1e5abb1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f099277320 0x55f099274ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.755+0000 7f1e5abb1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f09927a0b0 con 0x55f09926cc50 2024-05-09T08:47:37.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.755+0000 7f1e5abb1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f09918cc40 con 0x55f09926c330 2024-05-09T08:47:37.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.755+0000 7f1e5abb1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55f09927a230 con 0x55f099277320 2024-05-09T08:47:37.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.755+0000 7f1e5abb1740 10 client.0 _ll_register_callbacks cb 0x55f098ee4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:37.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.755+0000 7f1e55363640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f0992747a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.755+0000 7f1e55363640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f0992747a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33578/0 (socket says 172.21.15.107:33578) 2024-05-09T08:47:37.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.755+0000 7f1e55363640 1 -- 172.21.15.107:0/1051972400 learned_addr learned my addr 172.21.15.107:0/1051972400 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:37.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.755+0000 7f1e56365640 1 --2- 172.21.15.107:0/1051972400 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f099277320 0x55f099274ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.755+0000 7f1e55363640 1 -- 172.21.15.107:0/1051972400 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f099277320 msgr2=0x55f099274ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.755+0000 7f1e55363640 1 --2- 172.21.15.107:0/1051972400 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f099277320 0x55f099274ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.755+0000 7f1e55363640 1 -- 172.21.15.107:0/1051972400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 msgr2=0x55f099274260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.755+0000 7f1e55b64640 1 --2- 172.21.15.107:0/1051972400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 0x55f099274260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.756+0000 7f1e55363640 1 --2- 172.21.15.107:0/1051972400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 0x55f099274260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.756+0000 7f1e55363640 1 -- 172.21.15.107:0/1051972400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f098faf7d0 con 0x55f09926cc50 2024-05-09T08:47:37.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.756+0000 7f1e55b64640 1 --2- 172.21.15.107:0/1051972400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 0x55f099274260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:47:37.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.756+0000 7f1e55363640 1 --2- 172.21.15.107:0/1051972400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f0992747a0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f1e40013040 tx=0x7f1e4000ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46054e7df4b478d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.756+0000 7f1e397fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:47:37.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.756+0000 7f1e397fa640 1 -- 172.21.15.107:0/1051972400 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e400217a0 con 0x55f09926cc50 2024-05-09T08:47:37.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.756+0000 7f1e397fa640 1 -- 172.21.15.107:0/1051972400 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7f1e40019040 con 0x55f09926cc50 2024-05-09T08:47:37.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.756+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1051972400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f0991df370 con 0x55f09926cc50 2024-05-09T08:47:37.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.756+0000 7f1e397fa640 1 -- 172.21.15.107:0/1051972400 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1e4003d050 con 0x55f09926cc50 2024-05-09T08:47:37.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.756+0000 7f1e5abb1740 10 client.19221 fetch_fsmap learned FSMap version 29 2024-05-09T08:47:37.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.756+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1051972400 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f099286030 con 0x55f09926cc50 2024-05-09T08:47:37.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.757+0000 7f1e397fa640 1 -- 172.21.15.107:0/1051972400 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f0991df370 con 0x55f09926cc50 2024-05-09T08:47:37.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.757+0000 7f1e397fa640 1 -- 172.21.15.107:0/1051972400 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f1e400039c0 con 0x55f09926cc50 2024-05-09T08:47:37.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.757+0000 7f1e5abb1740 10 client.19221 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:47:37.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.757+0000 7f1e5abb1740 10 client.19221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:47:37.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.757+0000 7f1e5abb1740 20 client.19221 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.757+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/1051972400 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f0992874f0 0x55f0992898e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.757+0000 7f1e5abb1740 4 client.19221 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:47:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.757+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1051972400 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0991d96c0 con 0x55f0992874f0 2024-05-09T08:47:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.757+0000 7f1e55b64640 1 --2- 172.21.15.107:0/1051972400 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f0992874f0 0x55f0992898e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.758+0000 7f1e397fa640 10 client.19221 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.758+0000 7f1e55b64640 1 --2- 172.21.15.107:0/1051972400 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f0992874f0 0x55f0992898e0 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d804dd4d6eaa0ef9 server_cookie=d2f948a18c3d0c14 in_seq=0 out_seq=0 2024-05-09T08:47:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.759+0000 7f1e397fa640 1 -- 172.21.15.107:0/1051972400 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f0991d96c0 con 0x55f0992874f0 2024-05-09T08:47:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.759+0000 7f1e397fa640 10 client.19221 handle_command_reply: tid=0 2024-05-09T08:47:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.759+0000 7f1e5abb1740 1 client.19221 shutdown 2024-05-09T08:47:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.759+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1051972400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 msgr2=0x55f0992747a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.759+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/1051972400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f0992747a0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f1e40013040 tx=0x7f1e4000ef80 comp rx=0 tx=0).stop 2024-05-09T08:47:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.759+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1051972400 shutdown_connections 2024-05-09T08:47:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.759+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/1051972400 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f0992874f0 0x55f0992898e0 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.759+0000 7f1e397fa640 0 client.19221 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.759+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/1051972400 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f099277320 0x55f099274ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.759+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/1051972400 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f0992747a0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.759+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/1051972400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 0x55f099274260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.759+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1051972400 >> 172.21.15.107:0/1051972400 conn(0x55f099268650 msgr2=0x55f09926e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.759+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1051972400 shutdown_connections 2024-05-09T08:47:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.759+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1051972400 wait complete. 2024-05-09T08:47:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.759+0000 7f1e5abb1740 20 client.19221 trim_cache size 0 max 16384 2024-05-09T08:47:37.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:37 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2092714199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:37.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.780+0000 7f1e5abb1740 1 Processor -- start 2024-05-09T08:47:37.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.780+0000 7f1e5abb1740 1 -- start start 2024-05-09T08:47:37.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.780+0000 7f1e5abb1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 0x55f099282fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.780+0000 7f1e5abb1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f09926cc50 0x55f099276090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.781+0000 7f1e5abb1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f099277320 0x55f09928ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.781+0000 7f1e5abb1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1e501af3d0 con 0x55f099277320 2024-05-09T08:47:37.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.781+0000 7f1e5abb1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1e501acca0 con 0x55f09926c330 2024-05-09T08:47:37.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.781+0000 7f1e5abb1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1e501aab10 con 0x55f09926cc50 2024-05-09T08:47:37.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.781+0000 7f1e56365640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f099277320 0x55f09928ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.781+0000 7f1e56365640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f099277320 0x55f09928ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33590/0 (socket says 172.21.15.107:33590) 2024-05-09T08:47:37.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.781+0000 7f1e56365640 1 -- 172.21.15.107:0/3573000181 learned_addr learned my addr 172.21.15.107:0/3573000181 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:37.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.781+0000 7f1e55363640 1 --2- 172.21.15.107:0/3573000181 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f09926cc50 0x55f099276090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.782+0000 7f1e56365640 1 -- 172.21.15.107:0/3573000181 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f09926cc50 msgr2=0x55f099276090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.782+0000 7f1e56365640 1 --2- 172.21.15.107:0/3573000181 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f09926cc50 0x55f099276090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.782+0000 7f1e56365640 1 -- 172.21.15.107:0/3573000181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 msgr2=0x55f099282fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.782+0000 7f1e56365640 1 --2- 172.21.15.107:0/3573000181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 0x55f099282fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.782+0000 7f1e56365640 1 -- 172.21.15.107:0/3573000181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0992641a0 con 0x55f099277320 2024-05-09T08:47:37.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.782+0000 7f1e56365640 1 --2- 172.21.15.107:0/3573000181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f099277320 0x55f09928ef40 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f1e48005cc0 tx=0x7f1e4802a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c61d21fa309af0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.782+0000 7f1e3b7fe640 1 -- 172.21.15.107:0/3573000181 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48004840 con 0x55f099277320 2024-05-09T08:47:37.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.783+0000 7f1e3b7fe640 1 -- 172.21.15.107:0/3573000181 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e48036070 con 0x55f099277320 2024-05-09T08:47:37.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.783+0000 7f1e3b7fe640 1 -- 172.21.15.107:0/3573000181 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e480ee3c0 con 0x55f099277320 2024-05-09T08:47:37.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.784+0000 7f1e5abb1740 1 -- 172.21.15.107:0/3573000181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f099277320 msgr2=0x55f09928ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.784+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/3573000181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f099277320 0x55f09928ef40 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f1e48005cc0 tx=0x7f1e4802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.784+0000 7f1e5abb1740 1 -- 172.21.15.107:0/3573000181 shutdown_connections 2024-05-09T08:47:37.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.784+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/3573000181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f099277320 0x55f09928ef40 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.784+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/3573000181 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f09926cc50 0x55f099276090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.784+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/3573000181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 0x55f099282fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.784+0000 7f1e5abb1740 1 -- 172.21.15.107:0/3573000181 >> 172.21.15.107:0/3573000181 conn(0x55f099268650 msgr2=0x55f099275980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.784+0000 7f1e5abb1740 1 -- 172.21.15.107:0/3573000181 shutdown_connections 2024-05-09T08:47:37.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.784+0000 7f1e5abb1740 1 -- 172.21.15.107:0/3573000181 wait complete. 2024-05-09T08:47:37.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.784+0000 7f1e5abb1740 1 -- start start 2024-05-09T08:47:37.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.784+0000 7f1e5abb1740 1 Processor -- start 2024-05-09T08:47:37.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.785+0000 7f1e5abb1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 0x55f099271580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.785+0000 7f1e5abb1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f099271ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.785+0000 7f1e5abb1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f099277320 0x55f0992ec7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.785+0000 7f1e5abb1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1e501ab160 con 0x55f09926cc50 2024-05-09T08:47:37.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.785+0000 7f1e5abb1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1e501ac5d0 con 0x55f09926c330 2024-05-09T08:47:37.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.785+0000 7f1e5abb1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1e501ab830 con 0x55f099277320 2024-05-09T08:47:37.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.785+0000 7f1e5abb1740 10 client.0 _ll_register_callbacks cb 0x7f1e4c00b690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:37.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.785+0000 7f1e55363640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f099271ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.785+0000 7f1e55b64640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 0x55f099271580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.785+0000 7f1e55363640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f099271ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33596/0 (socket says 172.21.15.107:33596) 2024-05-09T08:47:37.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.785+0000 7f1e55b64640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 0x55f099271580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:39972/0 (socket says 172.21.15.107:39972) 2024-05-09T08:47:37.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.785+0000 7f1e55363640 1 -- 172.21.15.107:0/1456379999 learned_addr learned my addr 172.21.15.107:0/1456379999 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:37.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.785+0000 7f1e56365640 1 --2- 172.21.15.107:0/1456379999 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f099277320 0x55f0992ec7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.786+0000 7f1e55363640 1 -- 172.21.15.107:0/1456379999 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f099277320 msgr2=0x55f0992ec7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.786+0000 7f1e55363640 1 --2- 172.21.15.107:0/1456379999 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f099277320 0x55f0992ec7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.786+0000 7f1e55363640 1 -- 172.21.15.107:0/1456379999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 msgr2=0x55f099271580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:37.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.786+0000 7f1e55363640 1 --2- 172.21.15.107:0/1456379999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 0x55f099271580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.786+0000 7f1e55363640 1 -- 172.21.15.107:0/1456379999 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0992eccf0 con 0x55f09926cc50 2024-05-09T08:47:37.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.786+0000 7f1e55b64640 1 --2- 172.21.15.107:0/1456379999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 0x55f099271580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:37.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.786+0000 7f1e55363640 1 --2- 172.21.15.107:0/1456379999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f099271ac0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f1e40009ce0 tx=0x7f1e4001cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fdda4286113f89c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:37.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.786+0000 7f1e397fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:47:37.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.786+0000 7f1e397fa640 1 -- 172.21.15.107:0/1456379999 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e40046060 con 0x55f09926cc50 2024-05-09T08:47:37.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.786+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1456379999 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e48036070 con 0x55f09926cc50 2024-05-09T08:47:37.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.786+0000 7f1e397fa640 1 -- 172.21.15.107:0/1456379999 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7f1e40019040 con 0x55f09926cc50 2024-05-09T08:47:37.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.786+0000 7f1e397fa640 1 -- 172.21.15.107:0/1456379999 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1e400217a0 con 0x55f09926cc50 2024-05-09T08:47:37.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.786+0000 7f1e5abb1740 10 client.19227 fetch_fsmap learned FSMap version 29 2024-05-09T08:47:37.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.786+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1456379999 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f0992edf20 con 0x55f09926cc50 2024-05-09T08:47:37.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.787+0000 7f1e397fa640 1 -- 172.21.15.107:0/1456379999 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e48036070 con 0x55f09926cc50 2024-05-09T08:47:37.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.787+0000 7f1e397fa640 1 -- 172.21.15.107:0/1456379999 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f1e400035f0 con 0x55f09926cc50 2024-05-09T08:47:37.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.787+0000 7f1e5abb1740 10 client.19227 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:47:37.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.787+0000 7f1e5abb1740 10 client.19227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:47:37.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.787+0000 7f1e5abb1740 20 client.19227 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:37.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.787+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/1456379999 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f0992874f0 0x55f0992f05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:37.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.787+0000 7f1e5abb1740 4 client.19227 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:47:37.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.787+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1456379999 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f099156820 con 0x55f0992874f0 2024-05-09T08:47:37.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.787+0000 7f1e55b64640 1 --2- 172.21.15.107:0/1456379999 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f0992874f0 0x55f0992f05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:37.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.788+0000 7f1e55b64640 1 --2- 172.21.15.107:0/1456379999 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f0992874f0 0x55f0992f05a0 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=792f2f0eacba0be3 server_cookie=508da6e8455c91ef in_seq=0 out_seq=0 2024-05-09T08:47:37.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.788+0000 7f1e397fa640 10 client.19227 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:37.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.788+0000 7f1e397fa640 1 -- 172.21.15.107:0/1456379999 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f099156820 con 0x55f0992874f0 2024-05-09T08:47:37.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.788+0000 7f1e397fa640 10 client.19227 handle_command_reply: tid=0 2024-05-09T08:47:37.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.788+0000 7f1e5abb1740 1 client.19227 shutdown 2024-05-09T08:47:37.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.789+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1456379999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 msgr2=0x55f099271ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.789+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/1456379999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f099271ac0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f1e40009ce0 tx=0x7f1e4001cd00 comp rx=0 tx=0).stop 2024-05-09T08:47:37.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.789+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1456379999 shutdown_connections 2024-05-09T08:47:37.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.789+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/1456379999 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55f0992874f0 0x55f0992f05a0 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.789+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/1456379999 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f099277320 0x55f0992ec7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.789+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/1456379999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f09926cc50 0x55f099271ac0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.789+0000 7f1e397fa640 0 client.19227 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:37.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.789+0000 7f1e5abb1740 1 --2- 172.21.15.107:0/1456379999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f09926c330 0x55f099271580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.789+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1456379999 >> 172.21.15.107:0/1456379999 conn(0x55f099268650 msgr2=0x55f09926e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.789+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1456379999 shutdown_connections 2024-05-09T08:47:37.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.789+0000 7f1e5abb1740 1 -- 172.21.15.107:0/1456379999 wait complete. 2024-05-09T08:47:37.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.789+0000 7f1e5abb1740 20 client.19227 trim_cache size 0 max 16384 2024-05-09T08:47:37.793 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:47:37.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.790+0000 7f1e57dc8640 1 -- 172.21.15.107:0/2154592824 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1e3c0a2d00 msgr2=0x7f1e3c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.790+0000 7f1e57dc8640 1 --2- 172.21.15.107:0/2154592824 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1e3c0a2d00 0x7f1e3c0a51c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1e4001ccd0 tx=0x7f1e4001a000 comp rx=0 tx=0).stop 2024-05-09T08:47:37.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.790+0000 7f1e57dc8640 1 -- 172.21.15.107:0/2154592824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50106710 msgr2=0x7f1e5019b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:37.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.790+0000 7f1e57dc8640 1 --2- 172.21.15.107:0/2154592824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50106710 0x7f1e5019b790 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f1e48033040 tx=0x7f1e48003970 comp rx=0 tx=0).stop 2024-05-09T08:47:37.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.790+0000 7f1e57dc8640 1 -- 172.21.15.107:0/2154592824 shutdown_connections 2024-05-09T08:47:37.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.790+0000 7f1e57dc8640 1 --2- 172.21.15.107:0/2154592824 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1e3c0a2d00 0x7f1e3c0a51c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.790+0000 7f1e57dc8640 1 --2- 172.21.15.107:0/2154592824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e50106710 0x7f1e5019b790 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.790+0000 7f1e57dc8640 1 --2- 172.21.15.107:0/2154592824 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e501010c0 0x7f1e50197410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.790+0000 7f1e57dc8640 1 --2- 172.21.15.107:0/2154592824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e50100700 0x7f1e50196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:37.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.790+0000 7f1e57dc8640 1 -- 172.21.15.107:0/2154592824 >> 172.21.15.107:0/2154592824 conn(0x7f1e50077830 msgr2=0x7f1e50102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:37.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.791+0000 7f1e57dc8640 1 -- 172.21.15.107:0/2154592824 shutdown_connections 2024-05-09T08:47:37.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:37.791+0000 7f1e57dc8640 1 -- 172.21.15.107:0/2154592824 wait complete. 2024-05-09T08:47:37.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:37 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2092714199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:38.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:37 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2092714199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:47:38.810 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:47:38.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:47:38.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:47:38.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:38 smithi107 ceph-mon[30122]: pgmap v446: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 130 KiB/s rd, 7.6 MiB/s wr, 1.85k op/s 2024-05-09T08:47:38.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:38 smithi110 ceph-mon[33622]: pgmap v446: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 130 KiB/s rd, 7.6 MiB/s wr, 1.85k op/s 2024-05-09T08:47:38.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.962+0000 7f6128577640 1 -- 172.21.15.107:0/4245375180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61201010c0 msgr2=0x7f612010d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.962+0000 7f6128577640 1 --2- 172.21.15.107:0/4245375180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61201010c0 0x7f612010d5d0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f6118009cd0 tx=0x7f611802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.962+0000 7f6128577640 1 -- 172.21.15.107:0/4245375180 shutdown_connections 2024-05-09T08:47:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.962+0000 7f6128577640 1 --2- 172.21.15.107:0/4245375180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61201010c0 0x7f612010d5d0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.962+0000 7f6128577640 1 --2- 172.21.15.107:0/4245375180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6120100700 0x7f6120100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.962+0000 7f6128577640 1 --2- 172.21.15.107:0/4245375180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6120106710 0x7f6120106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.962+0000 7f6128577640 1 -- 172.21.15.107:0/4245375180 >> 172.21.15.107:0/4245375180 conn(0x7f6120077830 msgr2=0x7f6120077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.962+0000 7f6128577640 1 -- 172.21.15.107:0/4245375180 shutdown_connections 2024-05-09T08:47:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.962+0000 7f6128577640 1 -- 172.21.15.107:0/4245375180 wait complete. 2024-05-09T08:47:38.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6128577640 1 Processor -- start 2024-05-09T08:47:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6128577640 1 -- start start 2024-05-09T08:47:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6128577640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6120100700 0x7f6120196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6128577640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f61201010c0 0x7f6120197450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6128577640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6120106710 0x7f612019b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6128577640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f61201109d0 con 0x7f6120100700 2024-05-09T08:47:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6128577640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6120110850 con 0x7f6120106710 2024-05-09T08:47:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6128577640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6120110b50 con 0x7f61201010c0 2024-05-09T08:47:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6126313640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6120100700 0x7f6120196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6126313640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6120100700 0x7f6120196f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33616/0 (socket says 172.21.15.107:33616) 2024-05-09T08:47:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6126313640 1 -- 172.21.15.107:0/3247240482 learned_addr learned my addr 172.21.15.107:0/3247240482 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6125b12640 1 --2- 172.21.15.107:0/3247240482 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f61201010c0 0x7f6120197450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6126b14640 1 --2- 172.21.15.107:0/3247240482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6120106710 0x7f612019b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6125b12640 1 -- 172.21.15.107:0/3247240482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6120106710 msgr2=0x7f612019b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6125b12640 1 --2- 172.21.15.107:0/3247240482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6120106710 0x7f612019b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6125b12640 1 -- 172.21.15.107:0/3247240482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6120100700 msgr2=0x7f6120196f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6125b12640 1 --2- 172.21.15.107:0/3247240482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6120100700 0x7f6120196f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6125b12640 1 -- 172.21.15.107:0/3247240482 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612019bf00 con 0x7f61201010c0 2024-05-09T08:47:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.963+0000 7f6126313640 1 --2- 172.21.15.107:0/3247240482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6120100700 0x7f6120196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.964+0000 7f6125b12640 1 --2- 172.21.15.107:0/3247240482 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f61201010c0 0x7f6120197450 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f611000e9f0 tx=0x7f611000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.964+0000 7f61177fe640 1 -- 172.21.15.107:0/3247240482 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611000ccc0 con 0x7f61201010c0 2024-05-09T08:47:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.964+0000 7f6128577640 1 -- 172.21.15.107:0/3247240482 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f612019c1f0 con 0x7f61201010c0 2024-05-09T08:47:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.964+0000 7f61177fe640 1 -- 172.21.15.107:0/3247240482 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6110005020 con 0x7f61201010c0 2024-05-09T08:47:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.964+0000 7f61177fe640 1 -- 172.21.15.107:0/3247240482 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6110004a00 con 0x7f61201010c0 2024-05-09T08:47:38.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.964+0000 7f6128577640 1 -- 172.21.15.107:0/3247240482 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61201a3a80 con 0x7f61201010c0 2024-05-09T08:47:38.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.967+0000 7f61177fe640 1 -- 172.21.15.107:0/3247240482 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6110004ba0 con 0x7f61201010c0 2024-05-09T08:47:38.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.967+0000 7f612b366740 1 Processor -- start 2024-05-09T08:47:38.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.967+0000 7f612b366740 1 -- start start 2024-05-09T08:47:38.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.967+0000 7f612b366740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef21e0b0 0x557aef21c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.967+0000 7f612b366740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21c9c0 0x557aef21ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f612b366740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef229950 0x557aef22bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f612b366740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557aef181990 con 0x557aef21e0b0 2024-05-09T08:47:38.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f612b366740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557aef1295b0 con 0x557aef229950 2024-05-09T08:47:38.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f612b366740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x557aef17d950 con 0x557aef21c9c0 2024-05-09T08:47:38.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f61177fe640 1 --2- 172.21.15.107:0/3247240482 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f610c0a2d00 0x7f610c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f6126b14640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef21e0b0 0x557aef21c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f6126b14640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef21e0b0 0x557aef21c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33620/0 (socket says 172.21.15.107:33620) 2024-05-09T08:47:38.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f6126313640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21c9c0 0x557aef21ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f6126b14640 1 -- 172.21.15.107:0/4114937205 learned_addr learned my addr 172.21.15.107:0/4114937205 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:38.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f6125b12640 1 --2- 172.21.15.107:0/4114937205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef229950 0x557aef22bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f6126b14640 1 --2- 172.21.15.107:0/3247240482 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f610c0a2d00 0x7f610c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f61177fe640 1 -- 172.21.15.107:0/3247240482 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7f61100cf850 con 0x7f61201010c0 2024-05-09T08:47:38.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f6126313640 1 -- 172.21.15.107:0/4114937205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef229950 msgr2=0x557aef22bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:38.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f6126313640 1 --2- 172.21.15.107:0/4114937205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef229950 0x557aef22bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f6126313640 1 -- 172.21.15.107:0/4114937205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef21e0b0 msgr2=0x557aef21c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:38.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f6126313640 1 --2- 172.21.15.107:0/4114937205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef21e0b0 0x557aef21c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f6126313640 1 -- 172.21.15.107:0/4114937205 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557aeef5d7e0 con 0x557aef21c9c0 2024-05-09T08:47:38.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f6126b14640 1 --2- 172.21.15.107:0/4114937205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef21e0b0 0x557aef21c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:38.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f6126313640 1 --2- 172.21.15.107:0/4114937205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21c9c0 0x557aef21ce40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f611c005d00 tx=0x7f611c011950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a4e1397ee9acb59 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:38.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.968+0000 7f6126b14640 1 --2- 172.21.15.107:0/3247240482 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f610c0a2d00 0x7f610c0a51c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6118005e10 tx=0x7f611802a8e0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:38.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.969+0000 7f6114bfe640 1 -- 172.21.15.107:0/4114937205 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c005020 con 0x557aef21c9c0 2024-05-09T08:47:38.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.969+0000 7f6114bfe640 1 -- 172.21.15.107:0/4114937205 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f611c0047a0 con 0x557aef21c9c0 2024-05-09T08:47:38.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.969+0000 7f6114bfe640 1 -- 172.21.15.107:0/4114937205 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c00f840 con 0x557aef21c9c0 2024-05-09T08:47:38.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.969+0000 7f612b366740 1 -- 172.21.15.107:0/4114937205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21c9c0 msgr2=0x557aef21ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.969+0000 7f612b366740 1 --2- 172.21.15.107:0/4114937205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21c9c0 0x557aef21ce40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f611c005d00 tx=0x7f611c011950 comp rx=0 tx=0).stop 2024-05-09T08:47:38.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.969+0000 7f612b366740 1 -- 172.21.15.107:0/4114937205 shutdown_connections 2024-05-09T08:47:38.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.969+0000 7f612b366740 1 --2- 172.21.15.107:0/4114937205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef229950 0x557aef22bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.969+0000 7f612b366740 1 --2- 172.21.15.107:0/4114937205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21c9c0 0x557aef21ce40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.969+0000 7f612b366740 1 --2- 172.21.15.107:0/4114937205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef21e0b0 0x557aef21c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.969+0000 7f612b366740 1 -- 172.21.15.107:0/4114937205 >> 172.21.15.107:0/4114937205 conn(0x557aef1e50a0 msgr2=0x557aef218f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:38.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.970+0000 7f612b366740 1 -- 172.21.15.107:0/4114937205 shutdown_connections 2024-05-09T08:47:38.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.970+0000 7f612b366740 1 -- 172.21.15.107:0/4114937205 wait complete. 2024-05-09T08:47:38.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.970+0000 7f612b366740 1 -- start start 2024-05-09T08:47:38.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.970+0000 7f612b366740 1 Processor -- start 2024-05-09T08:47:38.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.970+0000 7f612b366740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef21c9c0 0x557aef2268a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.970+0000 7f612b366740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21e0b0 0x557aef226de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.970+0000 7f612b366740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef229950 0x557aef227320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.970+0000 7f612b366740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557aef22c7e0 con 0x557aef21c9c0 2024-05-09T08:47:38.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.970+0000 7f612b366740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557aef13cb40 con 0x557aef229950 2024-05-09T08:47:38.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.970+0000 7f612b366740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x557aef22c960 con 0x557aef21e0b0 2024-05-09T08:47:38.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.970+0000 7f612b366740 10 client.0 _ll_register_callbacks cb 0x557aef1310f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:38.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.970+0000 7f6126313640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef21c9c0 0x557aef2268a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.970+0000 7f6126313640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef21c9c0 0x557aef2268a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33630/0 (socket says 172.21.15.107:33630) 2024-05-09T08:47:38.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.970+0000 7f6126313640 1 -- 172.21.15.107:0/2758863555 learned_addr learned my addr 172.21.15.107:0/2758863555 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:38.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.970+0000 7f6126b14640 1 --2- 172.21.15.107:0/2758863555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef229950 0x557aef227320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.970+0000 7f6125b12640 1 --2- 172.21.15.107:0/2758863555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21e0b0 0x557aef226de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.971+0000 7f6126313640 1 -- 172.21.15.107:0/2758863555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21e0b0 msgr2=0x557aef226de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:38.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.971+0000 7f6126313640 1 --2- 172.21.15.107:0/2758863555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21e0b0 0x557aef226de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.971+0000 7f6126313640 1 -- 172.21.15.107:0/2758863555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef229950 msgr2=0x557aef227320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:38.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.971+0000 7f6126313640 1 --2- 172.21.15.107:0/2758863555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef229950 0x557aef227320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.971+0000 7f6126313640 1 -- 172.21.15.107:0/2758863555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557aeefbdf70 con 0x557aef21c9c0 2024-05-09T08:47:38.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.971+0000 7f6126313640 1 --2- 172.21.15.107:0/2758863555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef21c9c0 0x557aef2268a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f611c005230 tx=0x7f611c004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9f38cf1e1b09c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:38.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.971+0000 7f6109ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:47:38.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.971+0000 7f6109ffb640 1 -- 172.21.15.107:0/2758863555 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c027070 con 0x557aef21c9c0 2024-05-09T08:47:38.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.971+0000 7f6109ffb640 1 -- 172.21.15.107:0/2758863555 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7f611c004040 con 0x557aef21c9c0 2024-05-09T08:47:38.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.971+0000 7f612b366740 1 -- 172.21.15.107:0/2758863555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f611c0047a0 con 0x557aef21c9c0 2024-05-09T08:47:38.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.971+0000 7f6109ffb640 1 -- 172.21.15.107:0/2758863555 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f611c033050 con 0x557aef21c9c0 2024-05-09T08:47:38.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.971+0000 7f612b366740 10 client.19239 fetch_fsmap learned FSMap version 29 2024-05-09T08:47:38.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.971+0000 7f612b366740 1 -- 172.21.15.107:0/2758863555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557aeee55170 con 0x557aef21c9c0 2024-05-09T08:47:38.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.971+0000 7f6109ffb640 1 -- 172.21.15.107:0/2758863555 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f611c0047a0 con 0x557aef21c9c0 2024-05-09T08:47:38.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.972+0000 7f6109ffb640 1 -- 172.21.15.107:0/2758863555 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f611c003af0 con 0x557aef21c9c0 2024-05-09T08:47:38.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.972+0000 7f612b366740 10 client.19239 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:47:38.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.972+0000 7f612b366740 10 client.19239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:47:38.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.972+0000 7f612b366740 20 client.19239 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:38.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.972+0000 7f612b366740 1 --2- 172.21.15.107:0/2758863555 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557aef239960 0x557aef23bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:38.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.972+0000 7f612b366740 4 client.19239 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:47:38.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.972+0000 7f612b366740 1 -- 172.21.15.107:0/2758863555 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557aef18f0c0 con 0x557aef239960 2024-05-09T08:47:38.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.972+0000 7f6125b12640 1 --2- 172.21.15.107:0/2758863555 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557aef239960 0x557aef23bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:38.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.972+0000 7f6109ffb640 10 client.19239 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:38.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.972+0000 7f6125b12640 1 --2- 172.21.15.107:0/2758863555 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557aef239960 0x557aef23bd50 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9298ff8f2b026e40 server_cookie=f2b75f3afec08328 in_seq=0 out_seq=0 2024-05-09T08:47:38.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.973+0000 7f6109ffb640 1 -- 172.21.15.107:0/2758863555 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557aef18f0c0 con 0x557aef239960 2024-05-09T08:47:38.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.973+0000 7f6109ffb640 10 client.19239 handle_command_reply: tid=0 2024-05-09T08:47:38.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.973+0000 7f612b366740 1 client.19239 shutdown 2024-05-09T08:47:38.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.973+0000 7f612b366740 1 -- 172.21.15.107:0/2758863555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef21c9c0 msgr2=0x557aef2268a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:38.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.973+0000 7f612b366740 1 --2- 172.21.15.107:0/2758863555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef21c9c0 0x557aef2268a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f611c005230 tx=0x7f611c004df0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.973+0000 7f612b366740 1 -- 172.21.15.107:0/2758863555 shutdown_connections 2024-05-09T08:47:38.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.973+0000 7f612b366740 1 --2- 172.21.15.107:0/2758863555 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557aef239960 0x557aef23bd50 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.973+0000 7f612b366740 1 --2- 172.21.15.107:0/2758863555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef229950 0x557aef227320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.973+0000 7f612b366740 1 --2- 172.21.15.107:0/2758863555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21e0b0 0x557aef226de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.973+0000 7f6109ffb640 0 client.19239 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:38.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.973+0000 7f612b366740 1 --2- 172.21.15.107:0/2758863555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef21c9c0 0x557aef2268a0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:38.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.973+0000 7f612b366740 1 -- 172.21.15.107:0/2758863555 >> 172.21.15.107:0/2758863555 conn(0x557aef1e50a0 msgr2=0x557aef21ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:38.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.973+0000 7f612b366740 1 -- 172.21.15.107:0/2758863555 shutdown_connections 2024-05-09T08:47:38.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.973+0000 7f612b366740 1 -- 172.21.15.107:0/2758863555 wait complete. 2024-05-09T08:47:38.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:38.974+0000 7f612b366740 20 client.19239 trim_cache size 0 max 16384 2024-05-09T08:47:38.993 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:38 smithi157 ceph-mon[33614]: pgmap v446: 129 pgs: 10 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 114 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 130 KiB/s rd, 7.6 MiB/s wr, 1.85k op/s 2024-05-09T08:47:39.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.003+0000 7f612b366740 1 Processor -- start 2024-05-09T08:47:39.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.003+0000 7f612b366740 1 -- start start 2024-05-09T08:47:39.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.004+0000 7f612b366740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef21c9c0 0x557aef2356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.004+0000 7f612b366740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21e0b0 0x557aef226090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.004+0000 7f612b366740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef229950 0x557aef2287e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.004+0000 7f612b366740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f61201af540 con 0x557aef229950 2024-05-09T08:47:39.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.004+0000 7f612b366740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f61201ace10 con 0x557aef21c9c0 2024-05-09T08:47:39.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.004+0000 7f612b366740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f61201aac80 con 0x557aef21e0b0 2024-05-09T08:47:39.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.004+0000 7f6126b14640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef229950 0x557aef2287e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.004+0000 7f6126b14640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef229950 0x557aef2287e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33646/0 (socket says 172.21.15.107:33646) 2024-05-09T08:47:39.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.004+0000 7f6126b14640 1 -- 172.21.15.107:0/2438950562 learned_addr learned my addr 172.21.15.107:0/2438950562 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:39.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.004+0000 7f6126b14640 1 -- 172.21.15.107:0/2438950562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21e0b0 msgr2=0x557aef226090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:39.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.004+0000 7f6126b14640 1 --2- 172.21.15.107:0/2438950562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21e0b0 0x557aef226090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.004+0000 7f6126b14640 1 -- 172.21.15.107:0/2438950562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef21c9c0 msgr2=0x557aef2356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:39.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.004+0000 7f6126b14640 1 --2- 172.21.15.107:0/2438950562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef21c9c0 0x557aef2356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.004+0000 7f6126b14640 1 -- 172.21.15.107:0/2438950562 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557aef21d320 con 0x557aef229950 2024-05-09T08:47:39.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.004+0000 7f6126b14640 1 --2- 172.21.15.107:0/2438950562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef229950 0x557aef2287e0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f6118035000 tx=0x7f6118004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6a9114fd0bec046 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:39.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.005+0000 7f610bfff640 1 -- 172.21.15.107:0/2438950562 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61180045c0 con 0x557aef229950 2024-05-09T08:47:39.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.005+0000 7f610bfff640 1 -- 172.21.15.107:0/2438950562 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6118036040 con 0x557aef229950 2024-05-09T08:47:39.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.005+0000 7f610bfff640 1 -- 172.21.15.107:0/2438950562 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6118057480 con 0x557aef229950 2024-05-09T08:47:39.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.005+0000 7f612b366740 1 -- 172.21.15.107:0/2438950562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef229950 msgr2=0x557aef2287e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:39.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.005+0000 7f612b366740 1 --2- 172.21.15.107:0/2438950562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef229950 0x557aef2287e0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f6118035000 tx=0x7f6118004a60 comp rx=0 tx=0).stop 2024-05-09T08:47:39.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.005+0000 7f612b366740 1 -- 172.21.15.107:0/2438950562 shutdown_connections 2024-05-09T08:47:39.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.005+0000 7f612b366740 1 --2- 172.21.15.107:0/2438950562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef229950 0x557aef2287e0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.005+0000 7f612b366740 1 --2- 172.21.15.107:0/2438950562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21e0b0 0x557aef226090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.005+0000 7f612b366740 1 --2- 172.21.15.107:0/2438950562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef21c9c0 0x557aef2356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.005+0000 7f612b366740 1 -- 172.21.15.107:0/2438950562 >> 172.21.15.107:0/2438950562 conn(0x557aef1e50a0 msgr2=0x557aef2280d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:39.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.005+0000 7f612b366740 1 -- 172.21.15.107:0/2438950562 shutdown_connections 2024-05-09T08:47:39.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.006+0000 7f612b366740 1 -- 172.21.15.107:0/2438950562 wait complete. 2024-05-09T08:47:39.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.006+0000 7f612b366740 1 -- start start 2024-05-09T08:47:39.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.006+0000 7f612b366740 1 Processor -- start 2024-05-09T08:47:39.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.006+0000 7f612b366740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21c9c0 0x557aef22dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.006+0000 7f612b366740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef21e0b0 0x557aef22e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.006+0000 7f612b366740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef229950 0x557aef29ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.006+0000 7f612b366740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f61201ab2d0 con 0x557aef229950 2024-05-09T08:47:39.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.006+0000 7f612b366740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f61201ac740 con 0x557aef21e0b0 2024-05-09T08:47:39.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.006+0000 7f612b366740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f61201ab9a0 con 0x557aef21c9c0 2024-05-09T08:47:39.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.006+0000 7f612b366740 10 client.0 _ll_register_callbacks cb 0x7f6110005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:39.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.006+0000 7f6126313640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21c9c0 0x557aef22dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.006+0000 7f6125b12640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef21e0b0 0x557aef22e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.006+0000 7f6126313640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21c9c0 0x557aef22dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:50704/0 (socket says 172.21.15.107:50704) 2024-05-09T08:47:39.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.006+0000 7f6126313640 1 -- 172.21.15.107:0/1372870961 learned_addr learned my addr 172.21.15.107:0/1372870961 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:39.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.006+0000 7f6126b14640 1 --2- 172.21.15.107:0/1372870961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef229950 0x557aef29ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.007+0000 7f6125b12640 1 -- 172.21.15.107:0/1372870961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21c9c0 msgr2=0x557aef22dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:39.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.007+0000 7f6125b12640 1 --2- 172.21.15.107:0/1372870961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21c9c0 0x557aef22dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.007+0000 7f6125b12640 1 -- 172.21.15.107:0/1372870961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef229950 msgr2=0x557aef29ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:39.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.007+0000 7f6125b12640 1 --2- 172.21.15.107:0/1372870961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef229950 0x557aef29ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.007+0000 7f6125b12640 1 -- 172.21.15.107:0/1372870961 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557aef29f2c0 con 0x557aef21e0b0 2024-05-09T08:47:39.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.007+0000 7f6125b12640 1 --2- 172.21.15.107:0/1372870961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef21e0b0 0x557aef22e2c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f61100ccb00 tx=0x7f611006d890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5b56bc99c9d757 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:39.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.007+0000 7f6109ffb640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:47:39.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.007+0000 7f6109ffb640 1 -- 172.21.15.107:0/1372870961 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611001c040 con 0x557aef21e0b0 2024-05-09T08:47:39.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.008+0000 7f6109ffb640 1 -- 172.21.15.107:0/1372870961 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7f61100051d0 con 0x557aef21e0b0 2024-05-09T08:47:39.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.008+0000 7f612b366740 1 -- 172.21.15.107:0/1372870961 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6118036040 con 0x557aef21e0b0 2024-05-09T08:47:39.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.008+0000 7f6109ffb640 1 -- 172.21.15.107:0/1372870961 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f611009fc80 con 0x557aef21e0b0 2024-05-09T08:47:39.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.008+0000 7f612b366740 10 client.28187 fetch_fsmap learned FSMap version 29 2024-05-09T08:47:39.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.008+0000 7f612b366740 1 -- 172.21.15.107:0/1372870961 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557aef2a0690 con 0x557aef21e0b0 2024-05-09T08:47:39.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.008+0000 7f6109ffb640 1 -- 172.21.15.107:0/1372870961 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6118036040 con 0x557aef21e0b0 2024-05-09T08:47:39.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.008+0000 7f6109ffb640 1 -- 172.21.15.107:0/1372870961 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f61100ccd80 con 0x557aef21e0b0 2024-05-09T08:47:39.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.008+0000 7f612b366740 10 client.28187 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:47:39.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.008+0000 7f612b366740 10 client.28187 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:47:39.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.008+0000 7f612b366740 20 client.28187 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:39.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.008+0000 7f612b366740 1 --2- 172.21.15.107:0/1372870961 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557aef2a0e90 0x557aef2a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.008+0000 7f612b366740 4 client.28187 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:47:39.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.008+0000 7f612b366740 1 -- 172.21.15.107:0/1372870961 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557aef189410 con 0x557aef2a0e90 2024-05-09T08:47:39.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.009+0000 7f6126313640 1 --2- 172.21.15.107:0/1372870961 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557aef2a0e90 0x557aef2a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.009+0000 7f6109ffb640 10 client.28187 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:39.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.009+0000 7f6126313640 1 --2- 172.21.15.107:0/1372870961 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557aef2a0e90 0x557aef2a32b0 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96ad738cd046f3dc server_cookie=6ff06506df318417 in_seq=0 out_seq=0 2024-05-09T08:47:39.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.010+0000 7f6109ffb640 1 -- 172.21.15.107:0/1372870961 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557aef189410 con 0x557aef2a0e90 2024-05-09T08:47:39.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.010+0000 7f6109ffb640 10 client.28187 handle_command_reply: tid=0 2024-05-09T08:47:39.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.010+0000 7f612b366740 1 client.28187 shutdown 2024-05-09T08:47:39.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.010+0000 7f612b366740 1 -- 172.21.15.107:0/1372870961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef21e0b0 msgr2=0x557aef22e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:39.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.010+0000 7f612b366740 1 --2- 172.21.15.107:0/1372870961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef21e0b0 0x557aef22e2c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f61100ccb00 tx=0x7f611006d890 comp rx=0 tx=0).stop 2024-05-09T08:47:39.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.010+0000 7f612b366740 1 -- 172.21.15.107:0/1372870961 shutdown_connections 2024-05-09T08:47:39.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.010+0000 7f612b366740 1 --2- 172.21.15.107:0/1372870961 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x557aef2a0e90 0x557aef2a32b0 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.010+0000 7f612b366740 1 --2- 172.21.15.107:0/1372870961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557aef229950 0x557aef29ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.010+0000 7f612b366740 1 --2- 172.21.15.107:0/1372870961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557aef21e0b0 0x557aef22e2c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.010+0000 7f612b366740 1 --2- 172.21.15.107:0/1372870961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557aef21c9c0 0x557aef22dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.010+0000 7f6109ffb640 0 client.28187 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:39.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.010+0000 7f612b366740 1 -- 172.21.15.107:0/1372870961 >> 172.21.15.107:0/1372870961 conn(0x557aef1e50a0 msgr2=0x557aef228960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:39.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.010+0000 7f612b366740 1 -- 172.21.15.107:0/1372870961 shutdown_connections 2024-05-09T08:47:39.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.011+0000 7f612b366740 1 -- 172.21.15.107:0/1372870961 wait complete. 2024-05-09T08:47:39.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.011+0000 7f612b366740 20 client.28187 trim_cache size 0 max 16384 2024-05-09T08:47:39.015 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:47:39.015 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:47:39.015 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "c14c5ad0-5179-4016-b076-796805108edf", 2024-05-09T08:47:39.015 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:47:39.015 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:47:39.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.011+0000 7f6128577640 1 -- 172.21.15.107:0/3247240482 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f610c0a2d00 msgr2=0x7f610c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:39.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.011+0000 7f6128577640 1 --2- 172.21.15.107:0/3247240482 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f610c0a2d00 0x7f610c0a51c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6118005e10 tx=0x7f611802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.011+0000 7f6128577640 1 -- 172.21.15.107:0/3247240482 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f61201010c0 msgr2=0x7f6120197450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:39.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.011+0000 7f6128577640 1 --2- 172.21.15.107:0/3247240482 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f61201010c0 0x7f6120197450 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f611000e9f0 tx=0x7f611000eeb0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.011+0000 7f6128577640 1 -- 172.21.15.107:0/3247240482 shutdown_connections 2024-05-09T08:47:39.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.011+0000 7f6128577640 1 --2- 172.21.15.107:0/3247240482 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f610c0a2d00 0x7f610c0a51c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.011+0000 7f6128577640 1 --2- 172.21.15.107:0/3247240482 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6120106710 0x7f612019b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.011+0000 7f6128577640 1 --2- 172.21.15.107:0/3247240482 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f61201010c0 0x7f6120197450 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.011+0000 7f6128577640 1 --2- 172.21.15.107:0/3247240482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6120100700 0x7f6120196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.011+0000 7f6128577640 1 -- 172.21.15.107:0/3247240482 >> 172.21.15.107:0/3247240482 conn(0x7f6120077830 msgr2=0x7f6120102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:39.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.011+0000 7f6128577640 1 -- 172.21.15.107:0/3247240482 shutdown_connections 2024-05-09T08:47:39.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.011+0000 7f6128577640 1 -- 172.21.15.107:0/3247240482 wait complete. 2024-05-09T08:47:39.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:47:39.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:47:39.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.173+0000 7f311bbf9640 1 -- 172.21.15.107:0/3846532405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31141033d0 msgr2=0x7f311410b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:39.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.173+0000 7f311bbf9640 1 --2- 172.21.15.107:0/3846532405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31141033d0 0x7f311410b8b0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f3104009900 tx=0x7f310402a510 comp rx=0 tx=0).stop 2024-05-09T08:47:39.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.173+0000 7f311bbf9640 1 -- 172.21.15.107:0/3846532405 shutdown_connections 2024-05-09T08:47:39.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.173+0000 7f311bbf9640 1 --2- 172.21.15.107:0/3846532405 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f311410bdf0 0x7f311410e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.173+0000 7f311bbf9640 1 --2- 172.21.15.107:0/3846532405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31141033d0 0x7f311410b8b0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.173+0000 7f311bbf9640 1 --2- 172.21.15.107:0/3846532405 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3114102ab0 0x7f3114102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.173+0000 7f311bbf9640 1 -- 172.21.15.107:0/3846532405 >> 172.21.15.107:0/3846532405 conn(0x7f3114077830 msgr2=0x7f3114077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:39.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.173+0000 7f311bbf9640 1 -- 172.21.15.107:0/3846532405 shutdown_connections 2024-05-09T08:47:39.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.173+0000 7f311bbf9640 1 -- 172.21.15.107:0/3846532405 wait complete. 2024-05-09T08:47:39.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311bbf9640 1 Processor -- start 2024-05-09T08:47:39.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311bbf9640 1 -- start start 2024-05-09T08:47:39.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311bbf9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3114102ab0 0x7f31141980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311bbf9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31141033d0 0x7f3114198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311bbf9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f311410bdf0 0x7f311419c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311bbf9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3114111aa0 con 0x7f311410bdf0 2024-05-09T08:47:39.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311bbf9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3114111920 con 0x7f31141033d0 2024-05-09T08:47:39.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311bbf9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3114111c20 con 0x7f3114102ab0 2024-05-09T08:47:39.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311a196640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f311410bdf0 0x7f311419c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311a196640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f311410bdf0 0x7f311419c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33672/0 (socket says 172.21.15.107:33672) 2024-05-09T08:47:39.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f3119995640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3114102ab0 0x7f31141980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311a196640 1 -- 172.21.15.107:0/1985184960 learned_addr learned my addr 172.21.15.107:0/1985184960 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:39.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f3119194640 1 --2- 172.21.15.107:0/1985184960 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31141033d0 0x7f3114198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311a196640 1 -- 172.21.15.107:0/1985184960 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3114102ab0 msgr2=0x7f31141980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:39.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311a196640 1 --2- 172.21.15.107:0/1985184960 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3114102ab0 0x7f31141980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311a196640 1 -- 172.21.15.107:0/1985184960 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31141033d0 msgr2=0x7f3114198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:39.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311a196640 1 --2- 172.21.15.107:0/1985184960 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31141033d0 0x7f3114198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311a196640 1 -- 172.21.15.107:0/1985184960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311419d0d0 con 0x7f311410bdf0 2024-05-09T08:47:39.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f3119995640 1 --2- 172.21.15.107:0/1985184960 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3114102ab0 0x7f31141980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:39.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.174+0000 7f311a196640 1 --2- 172.21.15.107:0/1985184960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f311410bdf0 0x7f311419c9a0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f310c00e970 tx=0x7f310c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:39.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.175+0000 7f310affd640 1 -- 172.21.15.107:0/1985184960 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c00ccc0 con 0x7f311410bdf0 2024-05-09T08:47:39.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.175+0000 7f310affd640 1 -- 172.21.15.107:0/1985184960 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f310c00ce60 con 0x7f311410bdf0 2024-05-09T08:47:39.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.175+0000 7f310affd640 1 -- 172.21.15.107:0/1985184960 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c010430 con 0x7f311410bdf0 2024-05-09T08:47:39.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.175+0000 7f311bbf9640 1 -- 172.21.15.107:0/1985184960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f311419d3c0 con 0x7f311410bdf0 2024-05-09T08:47:39.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.175+0000 7f311bbf9640 1 -- 172.21.15.107:0/1985184960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31141a4c50 con 0x7f311410bdf0 2024-05-09T08:47:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.177+0000 7f310affd640 1 -- 172.21.15.107:0/1985184960 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f310c004f90 con 0x7f311410bdf0 2024-05-09T08:47:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.177+0000 7f311e9e2740 1 Processor -- start 2024-05-09T08:47:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f311e9e2740 1 -- start start 2024-05-09T08:47:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f311e9e2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97af180 0x55dba97af560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f311e9e2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97afaa0 0x55dba97be140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f311e9e2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97be680 0x55dba97c0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f311e9e2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55dba9714d00 con 0x55dba97be680 2024-05-09T08:47:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f311e9e2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55dba96bc4c0 con 0x55dba97af180 2024-05-09T08:47:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f311e9e2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55dba9710cc0 con 0x55dba97afaa0 2024-05-09T08:47:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f3119194640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97be680 0x55dba97c0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f3119995640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97afaa0 0x55dba97be140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f3119194640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97be680 0x55dba97c0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33674/0 (socket says 172.21.15.107:33674) 2024-05-09T08:47:39.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f3119194640 1 -- 172.21.15.107:0/1663783507 learned_addr learned my addr 172.21.15.107:0/1663783507 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f3119194640 1 --2- 172.21.15.107:0/1663783507 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97af180 0x55dba97af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f310affd640 1 --2- 172.21.15.107:0/1985184960 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f31000a2d80 0x7f31000a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f3119995640 1 -- 172.21.15.107:0/1663783507 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97af180 msgr2=0x55dba97af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f3119995640 1 --2- 172.21.15.107:0/1663783507 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97af180 0x55dba97af560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f3119995640 1 -- 172.21.15.107:0/1663783507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97be680 msgr2=0x55dba97c0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f3119995640 1 --2- 172.21.15.107:0/1663783507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97be680 0x55dba97c0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f3119995640 1 -- 172.21.15.107:0/1663783507 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dba94f27d0 con 0x55dba97afaa0 2024-05-09T08:47:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f3119194640 1 --2- 172.21.15.107:0/1663783507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97be680 0x55dba97c0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:47:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f311a196640 1 --2- 172.21.15.107:0/1985184960 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f31000a2d80 0x7f31000a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.178+0000 7f310affd640 1 -- 172.21.15.107:0/1985184960 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7f310c0cfb50 con 0x7f311410bdf0 2024-05-09T08:47:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.179+0000 7f3119995640 1 --2- 172.21.15.107:0/1663783507 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97afaa0 0x55dba97be140 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f31100117e0 tx=0x7f3110011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b8968a9e648f5b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.179+0000 7f30fffff640 1 -- 172.21.15.107:0/1663783507 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110003c30 con 0x55dba97afaa0 2024-05-09T08:47:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.179+0000 7f30fffff640 1 -- 172.21.15.107:0/1663783507 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f311000fe50 con 0x55dba97afaa0 2024-05-09T08:47:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.179+0000 7f30fffff640 1 -- 172.21.15.107:0/1663783507 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110004850 con 0x55dba97afaa0 2024-05-09T08:47:39.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.179+0000 7f311e9e2740 1 -- 172.21.15.107:0/1663783507 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97afaa0 msgr2=0x55dba97be140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:39.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.179+0000 7f311a196640 1 --2- 172.21.15.107:0/1985184960 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f31000a2d80 0x7f31000a5240 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f310c0cceb0 tx=0x7f310c091060 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:39.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.179+0000 7f311e9e2740 1 --2- 172.21.15.107:0/1663783507 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97afaa0 0x55dba97be140 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f31100117e0 tx=0x7f3110011ca0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.179+0000 7f311e9e2740 1 -- 172.21.15.107:0/1663783507 shutdown_connections 2024-05-09T08:47:39.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.179+0000 7f311e9e2740 1 --2- 172.21.15.107:0/1663783507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97be680 0x55dba97c0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.179+0000 7f311e9e2740 1 --2- 172.21.15.107:0/1663783507 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97afaa0 0x55dba97be140 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.179+0000 7f311e9e2740 1 --2- 172.21.15.107:0/1663783507 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97af180 0x55dba97af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.179+0000 7f311e9e2740 1 -- 172.21.15.107:0/1663783507 >> 172.21.15.107:0/1663783507 conn(0x55dba97ab5a0 msgr2=0x55dba97ab980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:39.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.180+0000 7f311e9e2740 1 -- 172.21.15.107:0/1663783507 shutdown_connections 2024-05-09T08:47:39.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.180+0000 7f311e9e2740 1 -- 172.21.15.107:0/1663783507 wait complete. 2024-05-09T08:47:39.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.180+0000 7f311e9e2740 1 -- start start 2024-05-09T08:47:39.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.180+0000 7f311e9e2740 1 Processor -- start 2024-05-09T08:47:39.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.180+0000 7f311e9e2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97af180 0x55dba97bb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.180+0000 7f311e9e2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97afaa0 0x55dba97bbb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.180+0000 7f311e9e2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97be680 0x55dba97bc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.180+0000 7f311e9e2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55dba97c1440 con 0x55dba97af180 2024-05-09T08:47:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.180+0000 7f311e9e2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55dba96cfb90 con 0x55dba97afaa0 2024-05-09T08:47:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.180+0000 7f311e9e2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55dba97c15c0 con 0x55dba97be680 2024-05-09T08:47:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.180+0000 7f311e9e2740 10 client.0 _ll_register_callbacks cb 0x55dba970bda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.180+0000 7f3119194640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97af180 0x55dba97bb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.180+0000 7f3119194640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97af180 0x55dba97bb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33686/0 (socket says 172.21.15.107:33686) 2024-05-09T08:47:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.180+0000 7f3119194640 1 -- 172.21.15.107:0/2916941385 learned_addr learned my addr 172.21.15.107:0/2916941385 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.181+0000 7f3119995640 1 --2- 172.21.15.107:0/2916941385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97afaa0 0x55dba97bbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.181+0000 7f3119194640 1 --2- 172.21.15.107:0/2916941385 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97be680 0x55dba97bc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.181+0000 7f3119194640 1 -- 172.21.15.107:0/2916941385 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97be680 msgr2=0x55dba97bc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.181+0000 7f3119194640 1 --2- 172.21.15.107:0/2916941385 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97be680 0x55dba97bc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.181+0000 7f3119194640 1 -- 172.21.15.107:0/2916941385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97afaa0 msgr2=0x55dba97bbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:39.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.181+0000 7f3119194640 1 --2- 172.21.15.107:0/2916941385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97afaa0 0x55dba97bbb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.181+0000 7f3119194640 1 -- 172.21.15.107:0/2916941385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dba94283f0 con 0x55dba97af180 2024-05-09T08:47:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.181+0000 7f3119194640 1 --2- 172.21.15.107:0/2916941385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97af180 0x55dba97bb600 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f3104031440 tx=0x7f310402a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e40f9d35ea0c881 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.181+0000 7f30fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:47:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.181+0000 7f30fd7fa640 1 -- 172.21.15.107:0/2916941385 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104035070 con 0x55dba97af180 2024-05-09T08:47:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.181+0000 7f30fd7fa640 1 -- 172.21.15.107:0/2916941385 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7f3104004250 con 0x55dba97af180 2024-05-09T08:47:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.181+0000 7f311e9e2740 1 -- 172.21.15.107:0/2916941385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f311000fe50 con 0x55dba97af180 2024-05-09T08:47:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.181+0000 7f30fd7fa640 1 -- 172.21.15.107:0/2916941385 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3104054050 con 0x55dba97af180 2024-05-09T08:47:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.181+0000 7f311e9e2740 10 client.19257 fetch_fsmap learned FSMap version 29 2024-05-09T08:47:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.181+0000 7f311e9e2740 1 -- 172.21.15.107:0/2916941385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dba93e80f0 con 0x55dba97af180 2024-05-09T08:47:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.181+0000 7f30fd7fa640 1 -- 172.21.15.107:0/2916941385 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f311000fe50 con 0x55dba97af180 2024-05-09T08:47:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.182+0000 7f30fd7fa640 1 -- 172.21.15.107:0/2916941385 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f310403d570 con 0x55dba97af180 2024-05-09T08:47:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.182+0000 7f311e9e2740 10 client.19257 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:47:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.182+0000 7f311e9e2740 10 client.19257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:47:39.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.182+0000 7f311e9e2740 20 client.19257 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:39.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.182+0000 7f311e9e2740 1 --2- 172.21.15.107:0/2916941385 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dba97ce970 0x55dba97d0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.182+0000 7f311e9e2740 4 client.19257 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:47:39.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.182+0000 7f311e9e2740 1 -- 172.21.15.107:0/2916941385 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dba9722430 con 0x55dba97ce970 2024-05-09T08:47:39.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.182+0000 7f3119995640 1 --2- 172.21.15.107:0/2916941385 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dba97ce970 0x55dba97d0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.182+0000 7f30fd7fa640 10 client.19257 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:39.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.182+0000 7f3119995640 1 --2- 172.21.15.107:0/2916941385 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dba97ce970 0x55dba97d0d80 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65532a32d8a3ba46 server_cookie=c8c1c97c32221042 in_seq=0 out_seq=0 2024-05-09T08:47:39.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.183+0000 7f30fd7fa640 1 -- 172.21.15.107:0/2916941385 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dba9722430 con 0x55dba97ce970 2024-05-09T08:47:39.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.183+0000 7f30fd7fa640 10 client.19257 handle_command_reply: tid=0 2024-05-09T08:47:39.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.183+0000 7f311e9e2740 1 client.19257 shutdown 2024-05-09T08:47:39.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.183+0000 7f311e9e2740 1 -- 172.21.15.107:0/2916941385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97af180 msgr2=0x55dba97bb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:39.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.183+0000 7f311e9e2740 1 --2- 172.21.15.107:0/2916941385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97af180 0x55dba97bb600 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f3104031440 tx=0x7f310402a510 comp rx=0 tx=0).stop 2024-05-09T08:47:39.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.183+0000 7f311e9e2740 1 -- 172.21.15.107:0/2916941385 shutdown_connections 2024-05-09T08:47:39.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.183+0000 7f311e9e2740 1 --2- 172.21.15.107:0/2916941385 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dba97ce970 0x55dba97d0d80 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.183+0000 7f311e9e2740 1 --2- 172.21.15.107:0/2916941385 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97be680 0x55dba97bc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.183+0000 7f311e9e2740 1 --2- 172.21.15.107:0/2916941385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97afaa0 0x55dba97bbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.183+0000 7f30fd7fa640 0 client.19257 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:39.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.183+0000 7f311e9e2740 1 --2- 172.21.15.107:0/2916941385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97af180 0x55dba97bb600 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.183+0000 7f311e9e2740 1 -- 172.21.15.107:0/2916941385 >> 172.21.15.107:0/2916941385 conn(0x55dba97ab5a0 msgr2=0x55dba97b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:39.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.183+0000 7f311e9e2740 1 -- 172.21.15.107:0/2916941385 shutdown_connections 2024-05-09T08:47:39.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.183+0000 7f311e9e2740 1 -- 172.21.15.107:0/2916941385 wait complete. 2024-05-09T08:47:39.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.183+0000 7f311e9e2740 20 client.19257 trim_cache size 0 max 16384 2024-05-09T08:47:39.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.202+0000 7f311e9e2740 1 Processor -- start 2024-05-09T08:47:39.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.202+0000 7f311e9e2740 1 -- start start 2024-05-09T08:47:39.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f311e9e2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97af180 0x55dba97ca370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f311e9e2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97afaa0 0x55dba97d6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f311e9e2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97be680 0x55dba97d6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f311e9e2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f31141af600 con 0x55dba97afaa0 2024-05-09T08:47:39.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f311e9e2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f31141aced0 con 0x55dba97be680 2024-05-09T08:47:39.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f311e9e2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f31141aad40 con 0x55dba97af180 2024-05-09T08:47:39.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f3119995640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97afaa0 0x55dba97d6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f3119995640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97afaa0 0x55dba97d6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33696/0 (socket says 172.21.15.107:33696) 2024-05-09T08:47:39.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f3119995640 1 -- 172.21.15.107:0/111678446 learned_addr learned my addr 172.21.15.107:0/111678446 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:39.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f3119194640 1 --2- 172.21.15.107:0/111678446 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97af180 0x55dba97ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f3119995640 1 -- 172.21.15.107:0/111678446 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97af180 msgr2=0x55dba97ca370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f3119995640 1 --2- 172.21.15.107:0/111678446 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97af180 0x55dba97ca370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f3119995640 1 -- 172.21.15.107:0/111678446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97be680 msgr2=0x55dba97d6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f3119995640 1 --2- 172.21.15.107:0/111678446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97be680 0x55dba97d6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f3119995640 1 -- 172.21.15.107:0/111678446 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dba97ab3d0 con 0x55dba97afaa0 2024-05-09T08:47:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f3119995640 1 --2- 172.21.15.107:0/111678446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97afaa0 0x55dba97d6650 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f311000f790 tx=0x7f3110009fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a57b9b90165b726 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f30ff7fe640 1 -- 172.21.15.107:0/111678446 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110011040 con 0x55dba97afaa0 2024-05-09T08:47:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f30ff7fe640 1 -- 172.21.15.107:0/111678446 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f31100111e0 con 0x55dba97afaa0 2024-05-09T08:47:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.203+0000 7f30ff7fe640 1 -- 172.21.15.107:0/111678446 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311000fd40 con 0x55dba97afaa0 2024-05-09T08:47:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.204+0000 7f311e9e2740 1 -- 172.21.15.107:0/111678446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97afaa0 msgr2=0x55dba97d6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.204+0000 7f311e9e2740 1 --2- 172.21.15.107:0/111678446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97afaa0 0x55dba97d6650 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f311000f790 tx=0x7f3110009fb0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.204+0000 7f311e9e2740 1 -- 172.21.15.107:0/111678446 shutdown_connections 2024-05-09T08:47:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.204+0000 7f311e9e2740 1 --2- 172.21.15.107:0/111678446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97be680 0x55dba97d6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.204+0000 7f311e9e2740 1 --2- 172.21.15.107:0/111678446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97afaa0 0x55dba97d6650 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.204+0000 7f311e9e2740 1 --2- 172.21.15.107:0/111678446 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97af180 0x55dba97ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.204+0000 7f311e9e2740 1 -- 172.21.15.107:0/111678446 >> 172.21.15.107:0/111678446 conn(0x55dba97ab5a0 msgr2=0x55dba97b13a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.204+0000 7f311e9e2740 1 -- 172.21.15.107:0/111678446 shutdown_connections 2024-05-09T08:47:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.204+0000 7f311e9e2740 1 -- 172.21.15.107:0/111678446 wait complete. 2024-05-09T08:47:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.204+0000 7f311e9e2740 1 -- start start 2024-05-09T08:47:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.205+0000 7f311e9e2740 1 Processor -- start 2024-05-09T08:47:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.205+0000 7f311e9e2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97af180 0x55dba97b8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.205+0000 7f311e9e2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97afaa0 0x55dba97b88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.205+0000 7f311e9e2740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97be680 0x55dba97b8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.205+0000 7f311e9e2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f31141ab390 con 0x55dba97af180 2024-05-09T08:47:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.205+0000 7f311e9e2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f31141ac800 con 0x55dba97afaa0 2024-05-09T08:47:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.205+0000 7f311e9e2740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f31141aba60 con 0x55dba97be680 2024-05-09T08:47:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.205+0000 7f311e9e2740 10 client.0 _ll_register_callbacks cb 0x7f3104031610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.205+0000 7f3119194640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97af180 0x55dba97b8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.205+0000 7f3119194640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97af180 0x55dba97b8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33700/0 (socket says 172.21.15.107:33700) 2024-05-09T08:47:39.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.205+0000 7f3119194640 1 -- 172.21.15.107:0/3566669643 learned_addr learned my addr 172.21.15.107:0/3566669643 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:39.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.205+0000 7f3119194640 1 --2- 172.21.15.107:0/3566669643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97be680 0x55dba97b8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.206+0000 7f3119194640 1 -- 172.21.15.107:0/3566669643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97be680 msgr2=0x55dba97b8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:39.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.206+0000 7f3119194640 1 --2- 172.21.15.107:0/3566669643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97be680 0x55dba97b8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.206+0000 7f3119194640 1 -- 172.21.15.107:0/3566669643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97afaa0 msgr2=0x55dba97b88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:39.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.206+0000 7f3119194640 1 --2- 172.21.15.107:0/3566669643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97afaa0 0x55dba97b88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.206+0000 7f3119194640 1 -- 172.21.15.107:0/3566669643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dba97b1ab0 con 0x55dba97af180 2024-05-09T08:47:39.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.206+0000 7f3119194640 1 --2- 172.21.15.107:0/3566669643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97af180 0x55dba97b8390 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f310403d940 tx=0x7f31040095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29dcc808942eda32 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:39.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.206+0000 7f30fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:47:39.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.206+0000 7f30fd7fa640 1 -- 172.21.15.107:0/3566669643 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104041070 con 0x55dba97af180 2024-05-09T08:47:39.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.207+0000 7f30fd7fa640 1 -- 172.21.15.107:0/3566669643 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9699+0+0 (secure 0 0 0) 0x7f3104004460 con 0x55dba97af180 2024-05-09T08:47:39.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.207+0000 7f311e9e2740 1 -- 172.21.15.107:0/3566669643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31100111e0 con 0x55dba97af180 2024-05-09T08:47:39.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.207+0000 7f30fd7fa640 1 -- 172.21.15.107:0/3566669643 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3104035040 con 0x55dba97af180 2024-05-09T08:47:39.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.207+0000 7f311e9e2740 10 client.19263 fetch_fsmap learned FSMap version 29 2024-05-09T08:47:39.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.207+0000 7f311e9e2740 1 -- 172.21.15.107:0/3566669643 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dba97b1cd0 con 0x55dba97af180 2024-05-09T08:47:39.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.207+0000 7f30fd7fa640 1 -- 172.21.15.107:0/3566669643 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f31100111e0 con 0x55dba97af180 2024-05-09T08:47:39.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.207+0000 7f30fd7fa640 1 -- 172.21.15.107:0/3566669643 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f3104064b40 con 0x55dba97af180 2024-05-09T08:47:39.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.207+0000 7f311e9e2740 10 client.19263 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:47:39.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.207+0000 7f311e9e2740 10 client.19263 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:47:39.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.207+0000 7f311e9e2740 20 client.19263 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:39.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.207+0000 7f311e9e2740 1 --2- 172.21.15.107:0/3566669643 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dba97b23d0 0x55dba97b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:39.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.207+0000 7f311e9e2740 4 client.19263 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:47:39.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.207+0000 7f311e9e2740 1 -- 172.21.15.107:0/3566669643 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dba971c780 con 0x55dba97b23d0 2024-05-09T08:47:39.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.207+0000 7f3119995640 1 --2- 172.21.15.107:0/3566669643 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dba97b23d0 0x55dba97b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:39.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f30fd7fa640 10 client.19263 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:39.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f3119995640 1 --2- 172.21.15.107:0/3566669643 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dba97b23d0 0x55dba97b4810 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ca72885720eb831 server_cookie=228ee5a09cd028a4 in_seq=0 out_seq=0 2024-05-09T08:47:39.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f30fd7fa640 1 -- 172.21.15.107:0/3566669643 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55dba971c780 con 0x55dba97b23d0 2024-05-09T08:47:39.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f30fd7fa640 10 client.19263 handle_command_reply: tid=0 2024-05-09T08:47:39.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f311e9e2740 1 client.19263 shutdown 2024-05-09T08:47:39.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f311e9e2740 1 -- 172.21.15.107:0/3566669643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97af180 msgr2=0x55dba97b8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:39.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f311e9e2740 1 --2- 172.21.15.107:0/3566669643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97af180 0x55dba97b8390 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f310403d940 tx=0x7f31040095a0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f311e9e2740 1 -- 172.21.15.107:0/3566669643 shutdown_connections 2024-05-09T08:47:39.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f311e9e2740 1 --2- 172.21.15.107:0/3566669643 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55dba97b23d0 0x55dba97b4810 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f311e9e2740 1 --2- 172.21.15.107:0/3566669643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55dba97be680 0x55dba97b8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f311e9e2740 1 --2- 172.21.15.107:0/3566669643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55dba97afaa0 0x55dba97b88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f311e9e2740 1 --2- 172.21.15.107:0/3566669643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dba97af180 0x55dba97b8390 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f30fd7fa640 0 client.19263 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:39.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f311e9e2740 1 -- 172.21.15.107:0/3566669643 >> 172.21.15.107:0/3566669643 conn(0x55dba97ab5a0 msgr2=0x55dba97b0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:39.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f311e9e2740 1 -- 172.21.15.107:0/3566669643 shutdown_connections 2024-05-09T08:47:39.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f311e9e2740 1 -- 172.21.15.107:0/3566669643 wait complete. 2024-05-09T08:47:39.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.208+0000 7f311e9e2740 20 client.19263 trim_cache size 0 max 16384 2024-05-09T08:47:39.214 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:47:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "scrub active (222 inodes in the stack)", 2024-05-09T08:47:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:47:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "c14c5ad0-5179-4016-b076-796805108edf": { 2024-05-09T08:47:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:47:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "c14c5ad0-5179-4016-b076-796805108edf", 2024-05-09T08:47:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:47:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:47:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:47:39.215 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:47:39.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.209+0000 7f311bbf9640 1 -- 172.21.15.107:0/1985184960 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f31000a2d80 msgr2=0x7f31000a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.209+0000 7f311bbf9640 1 --2- 172.21.15.107:0/1985184960 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f31000a2d80 0x7f31000a5240 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f310c0cceb0 tx=0x7f310c091060 comp rx=0 tx=0).stop 2024-05-09T08:47:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.209+0000 7f311bbf9640 1 -- 172.21.15.107:0/1985184960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f311410bdf0 msgr2=0x7f311419c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.209+0000 7f311bbf9640 1 --2- 172.21.15.107:0/1985184960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f311410bdf0 0x7f311419c9a0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f310c00e970 tx=0x7f310c00ed70 comp rx=0 tx=0).stop 2024-05-09T08:47:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.209+0000 7f311bbf9640 1 -- 172.21.15.107:0/1985184960 shutdown_connections 2024-05-09T08:47:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.209+0000 7f311bbf9640 1 --2- 172.21.15.107:0/1985184960 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f31000a2d80 0x7f31000a5240 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.209+0000 7f311bbf9640 1 --2- 172.21.15.107:0/1985184960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f311410bdf0 0x7f311419c9a0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.209+0000 7f311bbf9640 1 --2- 172.21.15.107:0/1985184960 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31141033d0 0x7f3114198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.209+0000 7f311bbf9640 1 --2- 172.21.15.107:0/1985184960 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3114102ab0 0x7f31141980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.209+0000 7f311bbf9640 1 -- 172.21.15.107:0/1985184960 >> 172.21.15.107:0/1985184960 conn(0x7f3114077830 msgr2=0x7f31140fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.209+0000 7f311bbf9640 1 -- 172.21.15.107:0/1985184960 shutdown_connections 2024-05-09T08:47:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:47:39.209+0000 7f311bbf9640 1 -- 172.21.15.107:0/1985184960 wait complete. 2024-05-09T08:47:39.219 INFO:tasks.cephfs.filesystem:scrub status for tag:c14c5ad0-5179-4016-b076-796805108edf - {'path': '/', 'tag': 'c14c5ad0-5179-4016-b076-796805108edf', 'options': 'recursive,force'} 2024-05-09T08:47:39.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:39 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:47:39.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:39 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:39.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:39 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:47:40.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:39 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:47:40.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:39 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:40.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:39 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:47:40.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:39 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:47:40.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:39 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:40.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:39 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:47:40.870 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:40 smithi110 ceph-mon[33622]: pgmap v447: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 162 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-05-09T08:47:41.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:40 smithi107 ceph-mon[30122]: pgmap v447: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 162 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-05-09T08:47:41.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:40 smithi157 ceph-mon[33614]: pgmap v447: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 162 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-05-09T08:47:41.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:41 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:47:41.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:41 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:41.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:41 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:47:41.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:41 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:41.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:41 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:47:41.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:41 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:42.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:41 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:47:42.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:41 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:42.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:41 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:47:42.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:41 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:42.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:41 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:47:42.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:41 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:42.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:41 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:47:42.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:41 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:42.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:41 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:47:42.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:41 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:42.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:41 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:47:42.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:41 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:42.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:42 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:47:42.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:42 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:42.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:42 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:47:42.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:42 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:42.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:42 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:47:42.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:42 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:42.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:42 smithi110 ceph-mon[33622]: pgmap v448: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 51 KiB/s rd, 8.3 MiB/s wr, 2.04k op/s 2024-05-09T08:47:42.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:42 smithi110 ceph-mon[33622]: osdmap e110: 12 total, 12 up, 12 in 2024-05-09T08:47:43.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:42 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:47:43.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:42 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:43.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:42 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:47:43.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:42 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:43.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:42 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:47:43.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:42 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:43.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:42 smithi107 ceph-mon[30122]: pgmap v448: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 51 KiB/s rd, 8.3 MiB/s wr, 2.04k op/s 2024-05-09T08:47:43.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:42 smithi107 ceph-mon[30122]: osdmap e110: 12 total, 12 up, 12 in 2024-05-09T08:47:43.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:42 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:47:43.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:42 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:43.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:42 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:47:43.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:42 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:43.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:42 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:47:43.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:42 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:43.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:42 smithi157 ceph-mon[33614]: pgmap v448: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 51 KiB/s rd, 8.3 MiB/s wr, 2.04k op/s 2024-05-09T08:47:43.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:42 smithi157 ceph-mon[33614]: osdmap e110: 12 total, 12 up, 12 in 2024-05-09T08:47:44.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:43 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:44.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:43 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:44.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:43 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:47:45.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:44 smithi107 ceph-mon[30122]: pgmap v450: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 86 KiB/s rd, 8.2 MiB/s wr, 2.04k op/s 2024-05-09T08:47:45.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:44 smithi157 ceph-mon[33614]: pgmap v450: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 86 KiB/s rd, 8.2 MiB/s wr, 2.04k op/s 2024-05-09T08:47:45.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:44 smithi110 ceph-mon[33622]: pgmap v450: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 86 KiB/s rd, 8.2 MiB/s wr, 2.04k op/s 2024-05-09T08:47:47.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:46 smithi107 ceph-mon[30122]: pgmap v451: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 99 KiB/s rd, 9.8 MiB/s wr, 2.46k op/s 2024-05-09T08:47:47.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:46 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:47:47.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:46 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:47:47.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:46 smithi157 ceph-mon[33614]: pgmap v451: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 99 KiB/s rd, 9.8 MiB/s wr, 2.46k op/s 2024-05-09T08:47:47.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:46 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:47:47.095 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:46 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:47:47.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:46 smithi110 ceph-mon[33622]: pgmap v451: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 99 KiB/s rd, 9.8 MiB/s wr, 2.46k op/s 2024-05-09T08:47:47.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:46 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:47:47.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:46 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:47:48.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:47 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:47:48.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:47 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:47:48.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:47 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:47:48.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:48 smithi107 ceph-mon[30122]: pgmap v452: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 99 KiB/s rd, 9.8 MiB/s wr, 2.46k op/s 2024-05-09T08:47:49.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:48 smithi157 ceph-mon[33614]: pgmap v452: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 99 KiB/s rd, 9.8 MiB/s wr, 2.46k op/s 2024-05-09T08:47:49.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:48 smithi110 ceph-mon[33622]: pgmap v452: 129 pgs: 3 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 124 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 99 KiB/s rd, 9.8 MiB/s wr, 2.46k op/s 2024-05-09T08:47:49.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.163+0000 7f536b1dc640 1 -- 172.21.15.107:0/1328483998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5364100760 msgr2=0x7f5364100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.163+0000 7f536b1dc640 1 --2- 172.21.15.107:0/1328483998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5364100760 0x7f5364100be0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f5350009930 tx=0x7f535002a420 comp rx=0 tx=0).stop 2024-05-09T08:47:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.163+0000 7f536b1dc640 1 -- 172.21.15.107:0/1328483998 shutdown_connections 2024-05-09T08:47:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.163+0000 7f536b1dc640 1 --2- 172.21.15.107:0/1328483998 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5364101120 0x7f536410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.163+0000 7f536b1dc640 1 --2- 172.21.15.107:0/1328483998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5364100760 0x7f5364100be0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.163+0000 7f536b1dc640 1 --2- 172.21.15.107:0/1328483998 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5364106720 0x7f5364106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.163+0000 7f536b1dc640 1 -- 172.21.15.107:0/1328483998 >> 172.21.15.107:0/1328483998 conn(0x7f53640778b0 msgr2=0x7f5364077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.163+0000 7f536b1dc640 1 -- 172.21.15.107:0/1328483998 shutdown_connections 2024-05-09T08:47:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.163+0000 7f536b1dc640 1 -- 172.21.15.107:0/1328483998 wait complete. 2024-05-09T08:47:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.164+0000 7f536b1dc640 1 Processor -- start 2024-05-09T08:47:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.164+0000 7f536b1dc640 1 -- start start 2024-05-09T08:47:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.164+0000 7f536b1dc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5364100760 0x7f5364198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.164+0000 7f536b1dc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5364101120 0x7f5364198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.164+0000 7f536b1dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5364106720 0x7f536419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.164+0000 7f536b1dc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f53641109e0 con 0x7f5364106720 2024-05-09T08:47:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.164+0000 7f536b1dc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5364110860 con 0x7f5364101120 2024-05-09T08:47:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.164+0000 7f536b1dc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5364110b60 con 0x7f5364100760 2024-05-09T08:47:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.164+0000 7f5369779640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5364106720 0x7f536419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.164+0000 7f5363fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5364101120 0x7f5364198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.164+0000 7f5363fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5364101120 0x7f5364198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34850/0 (socket says 172.21.15.107:34850) 2024-05-09T08:47:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.164+0000 7f5363fff640 1 -- 172.21.15.107:0/1206261478 learned_addr learned my addr 172.21.15.107:0/1206261478 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.164+0000 7f5368f78640 1 --2- 172.21.15.107:0/1206261478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5364100760 0x7f5364198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.165+0000 7f5363fff640 1 -- 172.21.15.107:0/1206261478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5364100760 msgr2=0x7f5364198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.165+0000 7f5363fff640 1 --2- 172.21.15.107:0/1206261478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5364100760 0x7f5364198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.165+0000 7f5363fff640 1 -- 172.21.15.107:0/1206261478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5364106720 msgr2=0x7f536419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.165+0000 7f5363fff640 1 --2- 172.21.15.107:0/1206261478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5364106720 0x7f536419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.165+0000 7f5363fff640 1 -- 172.21.15.107:0/1206261478 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536419d030 con 0x7f5364101120 2024-05-09T08:47:49.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.165+0000 7f5363fff640 1 --2- 172.21.15.107:0/1206261478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5364101120 0x7f5364198580 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5350005940 tx=0x7f53500042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:49.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.165+0000 7f5361ffb640 1 -- 172.21.15.107:0/1206261478 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535003f030 con 0x7f5364101120 2024-05-09T08:47:49.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.165+0000 7f536b1dc640 1 -- 172.21.15.107:0/1206261478 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f536419d2c0 con 0x7f5364101120 2024-05-09T08:47:49.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.165+0000 7f5361ffb640 1 -- 172.21.15.107:0/1206261478 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53500318a0 con 0x7f5364101120 2024-05-09T08:47:49.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.165+0000 7f5361ffb640 1 -- 172.21.15.107:0/1206261478 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350035070 con 0x7f5364101120 2024-05-09T08:47:49.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.165+0000 7f536b1dc640 1 -- 172.21.15.107:0/1206261478 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53641a4bb0 con 0x7f5364101120 2024-05-09T08:47:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.168+0000 7f536dfc7740 1 Processor -- start 2024-05-09T08:47:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.168+0000 7f5361ffb640 1 -- 172.21.15.107:0/1206261478 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f5350042030 con 0x7f5364101120 2024-05-09T08:47:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.168+0000 7f536dfc7740 1 -- start start 2024-05-09T08:47:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.168+0000 7f536dfc7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3f2450 0x5634be3f2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.168+0000 7f536dfc7740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3f2d70 0x5634be3fd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.168+0000 7f536dfc7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3fd630 0x5634be3ffa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.168+0000 7f536dfc7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5634be357340 con 0x5634be3f2450 2024-05-09T08:47:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.168+0000 7f536dfc7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5634be2ff290 con 0x5634be3fd630 2024-05-09T08:47:49.174 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.168+0000 7f536dfc7740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5634be353300 con 0x5634be3f2d70 2024-05-09T08:47:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.168+0000 7f5369779640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3f2450 0x5634be3f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.168+0000 7f5369779640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3f2450 0x5634be3f2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39686/0 (socket says 172.21.15.107:39686) 2024-05-09T08:47:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.168+0000 7f5369779640 1 -- 172.21.15.107:0/225090142 learned_addr learned my addr 172.21.15.107:0/225090142 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.168+0000 7f5363fff640 1 --2- 172.21.15.107:0/225090142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3fd630 0x5634be3ffa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.168+0000 7f5368f78640 1 --2- 172.21.15.107:0/225090142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3f2d70 0x5634be3fd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.169+0000 7f5369779640 1 -- 172.21.15.107:0/225090142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3f2d70 msgr2=0x5634be3fd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.169+0000 7f5369779640 1 --2- 172.21.15.107:0/225090142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3f2d70 0x5634be3fd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.169+0000 7f5369779640 1 -- 172.21.15.107:0/225090142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3fd630 msgr2=0x5634be3ffa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.169+0000 7f5369779640 1 --2- 172.21.15.107:0/225090142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3fd630 0x5634be3ffa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.169+0000 7f5361ffb640 1 --2- 172.21.15.107:0/1206261478 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f53540a2da0 0x7f53540a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.169+0000 7f5369779640 1 -- 172.21.15.107:0/225090142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634be3238e0 con 0x5634be3f2450 2024-05-09T08:47:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.169+0000 7f5361ffb640 1 -- 172.21.15.107:0/1206261478 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9623+0+0 (secure 0 0 0) 0x7f53500eff90 con 0x7f5364101120 2024-05-09T08:47:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.169+0000 7f5369779640 1 --2- 172.21.15.107:0/225090142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3f2450 0x5634be3f2830 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f535c010370 tx=0x7f535c010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a38cd82059e337 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.169+0000 7f5369779640 1 --2- 172.21.15.107:0/1206261478 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f53540a2da0 0x7f53540a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.169+0000 7f534effd640 1 -- 172.21.15.107:0/225090142 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c016070 con 0x5634be3f2450 2024-05-09T08:47:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.169+0000 7f534effd640 1 -- 172.21.15.107:0/225090142 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f535c01b460 con 0x5634be3f2450 2024-05-09T08:47:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.169+0000 7f534effd640 1 -- 172.21.15.107:0/225090142 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c011750 con 0x5634be3f2450 2024-05-09T08:47:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.170+0000 7f536dfc7740 1 -- 172.21.15.107:0/225090142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3f2450 msgr2=0x5634be3f2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.170+0000 7f5369779640 1 --2- 172.21.15.107:0/1206261478 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f53540a2da0 0x7f53540a5260 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f535c01a000 tx=0x7f535c011e00 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.170+0000 7f536dfc7740 1 --2- 172.21.15.107:0/225090142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3f2450 0x5634be3f2830 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f535c010370 tx=0x7f535c010830 comp rx=0 tx=0).stop 2024-05-09T08:47:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.170+0000 7f536dfc7740 1 -- 172.21.15.107:0/225090142 shutdown_connections 2024-05-09T08:47:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.170+0000 7f536dfc7740 1 --2- 172.21.15.107:0/225090142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3fd630 0x5634be3ffa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.170+0000 7f536dfc7740 1 --2- 172.21.15.107:0/225090142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3f2d70 0x5634be3fd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.170+0000 7f536dfc7740 1 --2- 172.21.15.107:0/225090142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3f2450 0x5634be3f2830 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.170+0000 7f536dfc7740 1 -- 172.21.15.107:0/225090142 >> 172.21.15.107:0/225090142 conn(0x5634be3ee960 msgr2=0x5634be3eed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.170+0000 7f536dfc7740 1 -- 172.21.15.107:0/225090142 shutdown_connections 2024-05-09T08:47:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.170+0000 7f536dfc7740 1 -- 172.21.15.107:0/225090142 wait complete. 2024-05-09T08:47:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.170+0000 7f536dfc7740 1 -- start start 2024-05-09T08:47:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.170+0000 7f536dfc7740 1 Processor -- start 2024-05-09T08:47:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f536dfc7740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3f2450 0x5634be3fa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f536dfc7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2d70 0x5634be3faa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f536dfc7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3fd630 0x5634be3faf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f536dfc7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5634be400260 con 0x5634be3fd630 2024-05-09T08:47:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f536dfc7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5634be312790 con 0x5634be3f2d70 2024-05-09T08:47:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f536dfc7740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5634be4003e0 con 0x5634be3f2450 2024-05-09T08:47:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f536dfc7740 10 client.0 _ll_register_callbacks cb 0x5634be34da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f5369779640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3fd630 0x5634be3faf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f5369779640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3fd630 0x5634be3faf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39690/0 (socket says 172.21.15.107:39690) 2024-05-09T08:47:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f5369779640 1 -- 172.21.15.107:0/2280169132 learned_addr learned my addr 172.21.15.107:0/2280169132 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f5363fff640 1 --2- 172.21.15.107:0/2280169132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2d70 0x5634be3faa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f5368f78640 1 --2- 172.21.15.107:0/2280169132 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3f2450 0x5634be3fa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f5369779640 1 -- 172.21.15.107:0/2280169132 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3f2450 msgr2=0x5634be3fa4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f5369779640 1 --2- 172.21.15.107:0/2280169132 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3f2450 0x5634be3fa4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f5369779640 1 -- 172.21.15.107:0/2280169132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2d70 msgr2=0x5634be3faa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f5369779640 1 --2- 172.21.15.107:0/2280169132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2d70 0x5634be3faa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f5369779640 1 -- 172.21.15.107:0/2280169132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634be06b210 con 0x5634be3fd630 2024-05-09T08:47:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f5368f78640 1 --2- 172.21.15.107:0/2280169132 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3f2450 0x5634be3fa4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:47:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f5369779640 1 --2- 172.21.15.107:0/2280169132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3fd630 0x5634be3faf60 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f535c01bed0 tx=0x7f535c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f259c4dc012de4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f5343fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:47:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f5343fff640 1 -- 172.21.15.107:0/2280169132 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c035060 con 0x5634be3fd630 2024-05-09T08:47:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.171+0000 7f5343fff640 1 -- 172.21.15.107:0/2280169132 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9623+0+0 (secure 0 0 0) 0x7f535c0125d0 con 0x5634be3fd630 2024-05-09T08:47:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.172+0000 7f536dfc7740 1 -- 172.21.15.107:0/2280169132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f535c01b460 con 0x5634be3fd630 2024-05-09T08:47:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.172+0000 7f5343fff640 1 -- 172.21.15.107:0/2280169132 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f535c03c050 con 0x5634be3fd630 2024-05-09T08:47:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.172+0000 7f536dfc7740 10 client.19275 fetch_fsmap learned FSMap version 29 2024-05-09T08:47:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.172+0000 7f536dfc7740 1 -- 172.21.15.107:0/2280169132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5634be137620 con 0x5634be3fd630 2024-05-09T08:47:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.172+0000 7f5343fff640 1 -- 172.21.15.107:0/2280169132 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f535c01b460 con 0x5634be3fd630 2024-05-09T08:47:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.172+0000 7f5343fff640 1 -- 172.21.15.107:0/2280169132 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f535c00f040 con 0x5634be3fd630 2024-05-09T08:47:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.172+0000 7f536dfc7740 10 client.19275 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:47:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.172+0000 7f536dfc7740 10 client.19275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:47:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.172+0000 7f536dfc7740 20 client.19275 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.172+0000 7f536dfc7740 1 --2- 172.21.15.107:0/2280169132 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5634be40d250 0x5634be40f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.172+0000 7f536dfc7740 4 client.19275 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:47:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.172+0000 7f536dfc7740 1 -- 172.21.15.107:0/2280169132 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634be364a70 con 0x5634be40d250 2024-05-09T08:47:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.172+0000 7f5368f78640 1 --2- 172.21.15.107:0/2280169132 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5634be40d250 0x5634be40f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.173+0000 7f5343fff640 10 client.19275 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.173+0000 7f5368f78640 1 --2- 172.21.15.107:0/2280169132 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5634be40d250 0x5634be40f640 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1d8dd8f8213d14f server_cookie=9f09e4bb08ea7f61 in_seq=0 out_seq=0 2024-05-09T08:47:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.173+0000 7f5343fff640 1 -- 172.21.15.107:0/2280169132 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5634be364a70 con 0x5634be40d250 2024-05-09T08:47:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.173+0000 7f5343fff640 10 client.19275 handle_command_reply: tid=0 2024-05-09T08:47:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.173+0000 7f536dfc7740 1 client.19275 shutdown 2024-05-09T08:47:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.174+0000 7f536dfc7740 1 -- 172.21.15.107:0/2280169132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3fd630 msgr2=0x5634be3faf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.174+0000 7f536dfc7740 1 --2- 172.21.15.107:0/2280169132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3fd630 0x5634be3faf60 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f535c01bed0 tx=0x7f535c01b000 comp rx=0 tx=0).stop 2024-05-09T08:47:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.174+0000 7f536dfc7740 1 -- 172.21.15.107:0/2280169132 shutdown_connections 2024-05-09T08:47:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.174+0000 7f536dfc7740 1 --2- 172.21.15.107:0/2280169132 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5634be40d250 0x5634be40f640 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.174+0000 7f536dfc7740 1 --2- 172.21.15.107:0/2280169132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3fd630 0x5634be3faf60 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.174+0000 7f536dfc7740 1 --2- 172.21.15.107:0/2280169132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2d70 0x5634be3faa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.174+0000 7f5343fff640 0 client.19275 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.174+0000 7f536dfc7740 1 --2- 172.21.15.107:0/2280169132 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3f2450 0x5634be3fa4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.174+0000 7f536dfc7740 1 -- 172.21.15.107:0/2280169132 >> 172.21.15.107:0/2280169132 conn(0x5634be3ee960 msgr2=0x5634be3f4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.174+0000 7f536dfc7740 1 -- 172.21.15.107:0/2280169132 shutdown_connections 2024-05-09T08:47:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.174+0000 7f536dfc7740 1 -- 172.21.15.107:0/2280169132 wait complete. 2024-05-09T08:47:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.174+0000 7f536dfc7740 20 client.19275 trim_cache size 0 max 16384 2024-05-09T08:47:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f536dfc7740 1 Processor -- start 2024-05-09T08:47:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f536dfc7740 1 -- start start 2024-05-09T08:47:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f536dfc7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2450 0x5634be409120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f536dfc7740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3f2d70 0x5634be3f4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f536dfc7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3fd630 0x5634be3edf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f536dfc7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f53641b15d0 con 0x5634be3fd630 2024-05-09T08:47:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f536dfc7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f53641aeea0 con 0x5634be3f2450 2024-05-09T08:47:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f536dfc7740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f53641accf0 con 0x5634be3f2d70 2024-05-09T08:47:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f5369779640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3fd630 0x5634be3edf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f5369779640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3fd630 0x5634be3edf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39698/0 (socket says 172.21.15.107:39698) 2024-05-09T08:47:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f5369779640 1 -- 172.21.15.107:0/140777729 learned_addr learned my addr 172.21.15.107:0/140777729 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f5363fff640 1 --2- 172.21.15.107:0/140777729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3f2d70 0x5634be3f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f5369779640 1 -- 172.21.15.107:0/140777729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3f2d70 msgr2=0x5634be3f4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f5368f78640 1 --2- 172.21.15.107:0/140777729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2450 0x5634be409120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f5369779640 1 --2- 172.21.15.107:0/140777729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3f2d70 0x5634be3f4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f5369779640 1 -- 172.21.15.107:0/140777729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2450 msgr2=0x5634be409120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f5369779640 1 --2- 172.21.15.107:0/140777729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2450 0x5634be409120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f5369779640 1 -- 172.21.15.107:0/140777729 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634be3f3820 con 0x5634be3fd630 2024-05-09T08:47:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.189+0000 7f5368f78640 1 --2- 172.21.15.107:0/140777729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2450 0x5634be409120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:47:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.190+0000 7f5369779640 1 --2- 172.21.15.107:0/140777729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3fd630 0x5634be3edf10 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f535c01bf80 tx=0x7f535c009a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e760a278d5165fcd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.190+0000 7f534e7fc640 1 -- 172.21.15.107:0/140777729 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c035070 con 0x5634be3fd630 2024-05-09T08:47:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.190+0000 7f534e7fc640 1 -- 172.21.15.107:0/140777729 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f535c010d30 con 0x5634be3fd630 2024-05-09T08:47:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.190+0000 7f534e7fc640 1 -- 172.21.15.107:0/140777729 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c016040 con 0x5634be3fd630 2024-05-09T08:47:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.190+0000 7f536dfc7740 1 -- 172.21.15.107:0/140777729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3fd630 msgr2=0x5634be3edf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.190+0000 7f536dfc7740 1 --2- 172.21.15.107:0/140777729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3fd630 0x5634be3edf10 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f535c01bf80 tx=0x7f535c009a00 comp rx=0 tx=0).stop 2024-05-09T08:47:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.190+0000 7f536dfc7740 1 -- 172.21.15.107:0/140777729 shutdown_connections 2024-05-09T08:47:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.190+0000 7f536dfc7740 1 --2- 172.21.15.107:0/140777729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3fd630 0x5634be3edf10 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.190+0000 7f536dfc7740 1 --2- 172.21.15.107:0/140777729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3f2d70 0x5634be3f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.190+0000 7f536dfc7740 1 --2- 172.21.15.107:0/140777729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2450 0x5634be409120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.190+0000 7f536dfc7740 1 -- 172.21.15.107:0/140777729 >> 172.21.15.107:0/140777729 conn(0x5634be3ee960 msgr2=0x5634be3f3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.190+0000 7f536dfc7740 1 -- 172.21.15.107:0/140777729 shutdown_connections 2024-05-09T08:47:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.190+0000 7f536dfc7740 1 -- 172.21.15.107:0/140777729 wait complete. 2024-05-09T08:47:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.191+0000 7f536dfc7740 1 -- start start 2024-05-09T08:47:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.191+0000 7f536dfc7740 1 Processor -- start 2024-05-09T08:47:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.191+0000 7f536dfc7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3f2450 0x5634be405a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.191+0000 7f536dfc7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2d70 0x5634be400da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.191+0000 7f536dfc7740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3fd630 0x5634be4012e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.191+0000 7f536dfc7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f53641ad360 con 0x5634be3f2450 2024-05-09T08:47:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.191+0000 7f536dfc7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f53641ae7d0 con 0x5634be3f2d70 2024-05-09T08:47:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.191+0000 7f536dfc7740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f53641ada30 con 0x5634be3fd630 2024-05-09T08:47:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.191+0000 7f536dfc7740 10 client.0 _ll_register_callbacks cb 0x7f53500f39b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:47:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.191+0000 7f5368f78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3f2450 0x5634be405a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.191+0000 7f5368f78640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3f2450 0x5634be405a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39714/0 (socket says 172.21.15.107:39714) 2024-05-09T08:47:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.191+0000 7f5368f78640 1 -- 172.21.15.107:0/1648664931 learned_addr learned my addr 172.21.15.107:0/1648664931 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:47:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f5369779640 1 --2- 172.21.15.107:0/1648664931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3fd630 0x5634be4012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f5363fff640 1 --2- 172.21.15.107:0/1648664931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2d70 0x5634be400da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f5368f78640 1 -- 172.21.15.107:0/1648664931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3fd630 msgr2=0x5634be4012e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f5368f78640 1 --2- 172.21.15.107:0/1648664931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3fd630 0x5634be4012e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f5368f78640 1 -- 172.21.15.107:0/1648664931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2d70 msgr2=0x5634be400da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:47:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f5368f78640 1 --2- 172.21.15.107:0/1648664931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2d70 0x5634be400da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f5368f78640 1 -- 172.21.15.107:0/1648664931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634be401b10 con 0x5634be3f2450 2024-05-09T08:47:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f5363fff640 1 --2- 172.21.15.107:0/1648664931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2d70 0x5634be400da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:47:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f5368f78640 1 --2- 172.21.15.107:0/1648664931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3f2450 0x5634be405a30 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f5358009c40 tx=0x7f5358010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d16523cf4892efc2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:47:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f5343fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:47:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f5343fff640 1 -- 172.21.15.107:0/1648664931 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5358004220 con 0x5634be3f2450 2024-05-09T08:47:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f5343fff640 1 -- 172.21.15.107:0/1648664931 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9623+0+0 (secure 0 0 0) 0x7f53580043c0 con 0x5634be3f2450 2024-05-09T08:47:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f536dfc7740 1 -- 172.21.15.107:0/1648664931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f535c010d30 con 0x5634be3f2450 2024-05-09T08:47:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f5343fff640 1 -- 172.21.15.107:0/1648664931 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f535802e050 con 0x5634be3f2450 2024-05-09T08:47:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f536dfc7740 10 client.19281 fetch_fsmap learned FSMap version 29 2024-05-09T08:47:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f536dfc7740 1 -- 172.21.15.107:0/1648664931 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5634be4734f0 con 0x5634be3f2450 2024-05-09T08:47:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.192+0000 7f5343fff640 1 -- 172.21.15.107:0/1648664931 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f535c010d30 con 0x5634be3f2450 2024-05-09T08:47:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.193+0000 7f5343fff640 1 -- 172.21.15.107:0/1648664931 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f5358003ae0 con 0x5634be3f2450 2024-05-09T08:47:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.193+0000 7f536dfc7740 10 client.19281 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:47:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.193+0000 7f536dfc7740 10 client.19281 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:47:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.193+0000 7f536dfc7740 20 client.19281 populate_metadata read hostname 'smithi107' 2024-05-09T08:47:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.193+0000 7f536dfc7740 1 --2- 172.21.15.107:0/1648664931 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5634be473c10 0x5634be476020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:47:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.193+0000 7f536dfc7740 4 client.19281 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:47:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.193+0000 7f536dfc7740 1 -- 172.21.15.107:0/1648664931 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5634be35edc0 con 0x5634be473c10 2024-05-09T08:47:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.193+0000 7f5363fff640 1 --2- 172.21.15.107:0/1648664931 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5634be473c10 0x5634be476020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:47:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.193+0000 7f5343fff640 10 client.19281 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.193+0000 7f5363fff640 1 --2- 172.21.15.107:0/1648664931 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5634be473c10 0x5634be476020 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17940cb0134a14b6 server_cookie=d1c7dc58698a62c1 in_seq=0 out_seq=0 2024-05-09T08:47:49.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.215+0000 7f5343fff640 1 -- 172.21.15.107:0/1648664931 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5634be35edc0 con 0x5634be473c10 2024-05-09T08:47:49.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.215+0000 7f5343fff640 10 client.19281 handle_command_reply: tid=0 2024-05-09T08:47:49.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.215+0000 7f536dfc7740 1 client.19281 shutdown 2024-05-09T08:47:49.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.215+0000 7f536dfc7740 1 -- 172.21.15.107:0/1648664931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3f2450 msgr2=0x5634be405a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:49.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.215+0000 7f536dfc7740 1 --2- 172.21.15.107:0/1648664931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3f2450 0x5634be405a30 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f5358009c40 tx=0x7f5358010800 comp rx=0 tx=0).stop 2024-05-09T08:47:49.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.215+0000 7f536dfc7740 1 -- 172.21.15.107:0/1648664931 shutdown_connections 2024-05-09T08:47:49.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.215+0000 7f536dfc7740 1 --2- 172.21.15.107:0/1648664931 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5634be473c10 0x5634be476020 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.215+0000 7f536dfc7740 1 --2- 172.21.15.107:0/1648664931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5634be3fd630 0x5634be4012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.215+0000 7f5343fff640 0 client.19281 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:47:49.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.215+0000 7f536dfc7740 1 --2- 172.21.15.107:0/1648664931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634be3f2d70 0x5634be400da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.215+0000 7f536dfc7740 1 --2- 172.21.15.107:0/1648664931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5634be3f2450 0x5634be405a30 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.215+0000 7f536dfc7740 1 -- 172.21.15.107:0/1648664931 >> 172.21.15.107:0/1648664931 conn(0x5634be3ee960 msgr2=0x5634be3f4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:49.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.216+0000 7f536dfc7740 1 -- 172.21.15.107:0/1648664931 shutdown_connections 2024-05-09T08:47:49.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.216+0000 7f536dfc7740 1 -- 172.21.15.107:0/1648664931 wait complete. 2024-05-09T08:47:49.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.216+0000 7f536dfc7740 20 client.19281 trim_cache size 0 max 16384 2024-05-09T08:47:49.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:47:49.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:47:49.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:47:49.217 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:47:49.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.216+0000 7f536b1dc640 1 -- 172.21.15.107:0/1206261478 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f53540a2da0 msgr2=0x7f53540a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:49.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.216+0000 7f536b1dc640 1 --2- 172.21.15.107:0/1206261478 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f53540a2da0 0x7f53540a5260 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f535c01a000 tx=0x7f535c011e00 comp rx=0 tx=0).stop 2024-05-09T08:47:49.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.216+0000 7f536b1dc640 1 -- 172.21.15.107:0/1206261478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5364101120 msgr2=0x7f5364198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:47:49.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.216+0000 7f536b1dc640 1 --2- 172.21.15.107:0/1206261478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5364101120 0x7f5364198580 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5350005940 tx=0x7f53500042f0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.216+0000 7f536b1dc640 1 -- 172.21.15.107:0/1206261478 shutdown_connections 2024-05-09T08:47:49.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.216+0000 7f536b1dc640 1 --2- 172.21.15.107:0/1206261478 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f53540a2da0 0x7f53540a5260 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.216+0000 7f536b1dc640 1 --2- 172.21.15.107:0/1206261478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5364106720 0x7f536419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.216+0000 7f536b1dc640 1 --2- 172.21.15.107:0/1206261478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5364101120 0x7f5364198580 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.216+0000 7f536b1dc640 1 --2- 172.21.15.107:0/1206261478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5364100760 0x7f5364198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:47:49.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.216+0000 7f536b1dc640 1 -- 172.21.15.107:0/1206261478 >> 172.21.15.107:0/1206261478 conn(0x7f53640778b0 msgr2=0x7f5364069280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:47:49.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.216+0000 7f536b1dc640 1 -- 172.21.15.107:0/1206261478 shutdown_connections 2024-05-09T08:47:49.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:47:49.217+0000 7f536b1dc640 1 -- 172.21.15.107:0/1206261478 wait complete. 2024-05-09T08:47:51.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:50 smithi107 ceph-mon[30122]: pgmap v453: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 97 KiB/s rd, 5.4 MiB/s wr, 1.38k op/s 2024-05-09T08:47:51.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:50 smithi157 ceph-mon[33614]: pgmap v453: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 97 KiB/s rd, 5.4 MiB/s wr, 1.38k op/s 2024-05-09T08:47:51.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:50 smithi110 ceph-mon[33622]: pgmap v453: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 97 KiB/s rd, 5.4 MiB/s wr, 1.38k op/s 2024-05-09T08:47:53.031 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:52 smithi107 ceph-mon[30122]: pgmap v454: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 97 KiB/s rd, 5.1 MiB/s wr, 1.29k op/s 2024-05-09T08:47:53.031 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:52 smithi107 ceph-mon[30122]: osdmap e111: 12 total, 12 up, 12 in 2024-05-09T08:47:53.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:52 smithi157 ceph-mon[33614]: pgmap v454: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 97 KiB/s rd, 5.1 MiB/s wr, 1.29k op/s 2024-05-09T08:47:53.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:52 smithi157 ceph-mon[33614]: osdmap e111: 12 total, 12 up, 12 in 2024-05-09T08:47:53.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:52 smithi110 ceph-mon[33622]: pgmap v454: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 97 KiB/s rd, 5.1 MiB/s wr, 1.29k op/s 2024-05-09T08:47:53.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:52 smithi110 ceph-mon[33622]: osdmap e111: 12 total, 12 up, 12 in 2024-05-09T08:47:55.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:54 smithi107 ceph-mon[30122]: pgmap v456: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 66 KiB/s rd, 2.4 MiB/s wr, 615 op/s 2024-05-09T08:47:55.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:54 smithi157 ceph-mon[33614]: pgmap v456: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 66 KiB/s rd, 2.4 MiB/s wr, 615 op/s 2024-05-09T08:47:55.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:54 smithi110 ceph-mon[33622]: pgmap v456: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 66 KiB/s rd, 2.4 MiB/s wr, 615 op/s 2024-05-09T08:47:57.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:56 smithi107 ceph-mon[30122]: pgmap v457: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 53 KiB/s rd, 163 KiB/s wr, 41 op/s 2024-05-09T08:47:57.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:56 smithi157 ceph-mon[33614]: pgmap v457: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 53 KiB/s rd, 163 KiB/s wr, 41 op/s 2024-05-09T08:47:57.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:56 smithi110 ceph-mon[33622]: pgmap v457: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 53 KiB/s rd, 163 KiB/s wr, 41 op/s 2024-05-09T08:47:59.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:47:58 smithi107 ceph-mon[30122]: pgmap v458: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 53 KiB/s rd, 163 KiB/s wr, 41 op/s 2024-05-09T08:47:59.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:47:58 smithi157 ceph-mon[33614]: pgmap v458: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 53 KiB/s rd, 163 KiB/s wr, 41 op/s 2024-05-09T08:47:59.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:47:58 smithi110 ceph-mon[33622]: pgmap v458: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 53 KiB/s rd, 163 KiB/s wr, 41 op/s 2024-05-09T08:48:01.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:00 smithi107 ceph-mon[30122]: pgmap v459: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 22 KiB/s rd, 29 KiB/s wr, 16 op/s 2024-05-09T08:48:01.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:00 smithi157 ceph-mon[33614]: pgmap v459: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 22 KiB/s rd, 29 KiB/s wr, 16 op/s 2024-05-09T08:48:01.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:00 smithi110 ceph-mon[33622]: pgmap v459: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 22 KiB/s rd, 29 KiB/s wr, 16 op/s 2024-05-09T08:48:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.314+0000 7f1edc0d4640 1 -- 172.21.15.107:0/1052114608 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ed4100760 msgr2=0x7f1ed4100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.314+0000 7f1edc0d4640 1 --2- 172.21.15.107:0/1052114608 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ed4100760 0x7f1ed4100be0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4009930 tx=0x7f1ec402a420 comp rx=0 tx=0).stop 2024-05-09T08:48:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.314+0000 7f1edc0d4640 1 -- 172.21.15.107:0/1052114608 shutdown_connections 2024-05-09T08:48:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.314+0000 7f1edc0d4640 1 --2- 172.21.15.107:0/1052114608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ed4101120 0x7f1ed410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.314+0000 7f1edc0d4640 1 --2- 172.21.15.107:0/1052114608 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ed4100760 0x7f1ed4100be0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.314+0000 7f1edc0d4640 1 --2- 172.21.15.107:0/1052114608 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ed4106720 0x7f1ed4106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.314+0000 7f1edc0d4640 1 -- 172.21.15.107:0/1052114608 >> 172.21.15.107:0/1052114608 conn(0x7f1ed40778b0 msgr2=0x7f1ed4077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.315+0000 7f1edc0d4640 1 -- 172.21.15.107:0/1052114608 shutdown_connections 2024-05-09T08:48:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.315+0000 7f1edc0d4640 1 -- 172.21.15.107:0/1052114608 wait complete. 2024-05-09T08:48:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.315+0000 7f1edc0d4640 1 Processor -- start 2024-05-09T08:48:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.315+0000 7f1edc0d4640 1 -- start start 2024-05-09T08:48:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.315+0000 7f1edc0d4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ed4100760 0x7f1ed4198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.315+0000 7f1edc0d4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ed4101120 0x7f1ed4199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.315+0000 7f1edc0d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ed4106720 0x7f1ed41a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.315+0000 7f1edc0d4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed41109e0 con 0x7f1ed4106720 2024-05-09T08:48:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.315+0000 7f1edc0d4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed4110860 con 0x7f1ed4100760 2024-05-09T08:48:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.315+0000 7f1edc0d4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed4110b60 con 0x7f1ed4101120 2024-05-09T08:48:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.316+0000 7f1eda671640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ed4106720 0x7f1ed41a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.316+0000 7f1eda671640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ed4106720 0x7f1ed41a3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51768/0 (socket says 172.21.15.107:51768) 2024-05-09T08:48:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.316+0000 7f1eda671640 1 -- 172.21.15.107:0/3462559943 learned_addr learned my addr 172.21.15.107:0/3462559943 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.316+0000 7f1ed966f640 1 --2- 172.21.15.107:0/3462559943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ed4101120 0x7f1ed4199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.316+0000 7f1ed9e70640 1 --2- 172.21.15.107:0/3462559943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ed4100760 0x7f1ed4198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.316+0000 7f1eda671640 1 -- 172.21.15.107:0/3462559943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ed4101120 msgr2=0x7f1ed4199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.316+0000 7f1eda671640 1 --2- 172.21.15.107:0/3462559943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ed4101120 0x7f1ed4199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.316+0000 7f1eda671640 1 -- 172.21.15.107:0/3462559943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ed4100760 msgr2=0x7f1ed4198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.316+0000 7f1eda671640 1 --2- 172.21.15.107:0/3462559943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ed4100760 0x7f1ed4198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.316+0000 7f1eda671640 1 -- 172.21.15.107:0/3462559943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed41a43c0 con 0x7f1ed4106720 2024-05-09T08:48:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.316+0000 7f1eda671640 1 --2- 172.21.15.107:0/3462559943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ed4106720 0x7f1ed41a3e80 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00cad0 tx=0x7f1ecc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.316+0000 7f1ecaffd640 1 -- 172.21.15.107:0/3462559943 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc00ae20 con 0x7f1ed4106720 2024-05-09T08:48:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.316+0000 7f1edc0d4640 1 -- 172.21.15.107:0/3462559943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ed41a4590 con 0x7f1ed4106720 2024-05-09T08:48:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.316+0000 7f1ecaffd640 1 -- 172.21.15.107:0/3462559943 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ecc005020 con 0x7f1ed4106720 2024-05-09T08:48:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.316+0000 7f1ecaffd640 1 -- 172.21.15.107:0/3462559943 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc00f6b0 con 0x7f1ed4106720 2024-05-09T08:48:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.316+0000 7f1edc0d4640 1 -- 172.21.15.107:0/3462559943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ed41a4b00 con 0x7f1ed4106720 2024-05-09T08:48:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.319+0000 7f1ecaffd640 1 -- 172.21.15.107:0/3462559943 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1ecc0051d0 con 0x7f1ed4106720 2024-05-09T08:48:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.319+0000 7f1edede8740 1 Processor -- start 2024-05-09T08:48:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1edede8740 1 -- start start 2024-05-09T08:48:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1edede8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56292317d450 0x56292317d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1edede8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317dd70 0x5629231880f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1edede8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562923188630 0x56292318aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1edede8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5629230e2340 con 0x56292317dd70 2024-05-09T08:48:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1edede8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56292308a290 con 0x56292317d450 2024-05-09T08:48:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1edede8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5629230de300 con 0x562923188630 2024-05-09T08:48:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1ed9e70640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317dd70 0x5629231880f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1ed9e70640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317dd70 0x5629231880f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51774/0 (socket says 172.21.15.107:51774) 2024-05-09T08:48:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1ed9e70640 1 -- 172.21.15.107:0/2712527002 learned_addr learned my addr 172.21.15.107:0/2712527002 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1ed966f640 1 --2- 172.21.15.107:0/2712527002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56292317d450 0x56292317d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1ecaffd640 1 --2- 172.21.15.107:0/3462559943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1ec00a2da0 0x7f1ec00a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1ed966f640 1 --2- 172.21.15.107:0/2712527002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562923188630 0x56292318aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1ed9e70640 1 -- 172.21.15.107:0/2712527002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562923188630 msgr2=0x56292318aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1ed9e70640 1 --2- 172.21.15.107:0/2712527002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562923188630 0x56292318aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1ed9e70640 1 -- 172.21.15.107:0/2712527002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56292317d450 msgr2=0x56292317d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1ed9e70640 1 --2- 172.21.15.107:0/2712527002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56292317d450 0x56292317d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1ed9e70640 1 -- 172.21.15.107:0/2712527002 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629230ae8e0 con 0x56292317dd70 2024-05-09T08:48:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.320+0000 7f1ed966f640 1 --2- 172.21.15.107:0/2712527002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562923188630 0x56292318aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.321+0000 7f1eda671640 1 --2- 172.21.15.107:0/3462559943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1ec00a2da0 0x7f1ec00a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.321+0000 7f1ed9e70640 1 --2- 172.21.15.107:0/2712527002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317dd70 0x5629231880f0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f1ed000bc50 tx=0x7f1ed00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5468ee701895180 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.321+0000 7f1ecaffd640 1 -- 172.21.15.107:0/3462559943 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9595+0+0 (secure 0 0 0) 0x7f1ecc0cf810 con 0x7f1ed4106720 2024-05-09T08:48:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.321+0000 7f1ebffff640 1 -- 172.21.15.107:0/2712527002 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed001c480 con 0x56292317dd70 2024-05-09T08:48:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.321+0000 7f1ebffff640 1 -- 172.21.15.107:0/2712527002 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ed001cad0 con 0x56292317dd70 2024-05-09T08:48:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.321+0000 7f1ebffff640 1 -- 172.21.15.107:0/2712527002 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0011ac0 con 0x56292317dd70 2024-05-09T08:48:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.322+0000 7f1edede8740 1 -- 172.21.15.107:0/2712527002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317dd70 msgr2=0x5629231880f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.322+0000 7f1eda671640 1 --2- 172.21.15.107:0/3462559943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1ec00a2da0 0x7f1ec00a5260 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc09b6b0 tx=0x7f1ecc097590 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.322+0000 7f1edede8740 1 --2- 172.21.15.107:0/2712527002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317dd70 0x5629231880f0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f1ed000bc50 tx=0x7f1ed00107a0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.322+0000 7f1edede8740 1 -- 172.21.15.107:0/2712527002 shutdown_connections 2024-05-09T08:48:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.322+0000 7f1edede8740 1 --2- 172.21.15.107:0/2712527002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562923188630 0x56292318aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.322+0000 7f1edede8740 1 --2- 172.21.15.107:0/2712527002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317dd70 0x5629231880f0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.322+0000 7f1edede8740 1 --2- 172.21.15.107:0/2712527002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56292317d450 0x56292317d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.322+0000 7f1edede8740 1 -- 172.21.15.107:0/2712527002 >> 172.21.15.107:0/2712527002 conn(0x562923179960 msgr2=0x562923179d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.322+0000 7f1edede8740 1 -- 172.21.15.107:0/2712527002 shutdown_connections 2024-05-09T08:48:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.322+0000 7f1edede8740 1 -- 172.21.15.107:0/2712527002 wait complete. 2024-05-09T08:48:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.322+0000 7f1edede8740 1 -- start start 2024-05-09T08:48:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.322+0000 7f1edede8740 1 Processor -- start 2024-05-09T08:48:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1edede8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x562923185580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1edede8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56292317dd70 0x562923185ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1edede8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562923188630 0x562923186000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1edede8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56292318b260 con 0x56292317d450 2024-05-09T08:48:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1edede8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56292309d790 con 0x562923188630 2024-05-09T08:48:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1edede8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56292318b3e0 con 0x56292317dd70 2024-05-09T08:48:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1edede8740 10 client.0 _ll_register_callbacks cb 0x5629230d8a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1ed966f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x562923185580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1ed966f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x562923185580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51786/0 (socket says 172.21.15.107:51786) 2024-05-09T08:48:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1ed966f640 1 -- 172.21.15.107:0/4240974645 learned_addr learned my addr 172.21.15.107:0/4240974645 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1ed9e70640 1 --2- 172.21.15.107:0/4240974645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56292317dd70 0x562923185ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1ed966f640 1 -- 172.21.15.107:0/4240974645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56292317dd70 msgr2=0x562923185ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1ed966f640 1 --2- 172.21.15.107:0/4240974645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56292317dd70 0x562923185ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1ed966f640 1 -- 172.21.15.107:0/4240974645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562923188630 msgr2=0x562923186000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1ed966f640 1 --2- 172.21.15.107:0/4240974645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562923188630 0x562923186000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1ed966f640 1 -- 172.21.15.107:0/4240974645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562922df6210 con 0x56292317d450 2024-05-09T08:48:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1ed966f640 1 --2- 172.21.15.107:0/4240974645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x562923185580 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f1ec402a940 tx=0x7f1ec4032e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa19bd515753e815 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1ebd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:48:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1ebd7fa640 1 -- 172.21.15.107:0/4240974645 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec4045070 con 0x56292317d450 2024-05-09T08:48:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1ebd7fa640 1 -- 172.21.15.107:0/4240974645 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9595+0+0 (secure 0 0 0) 0x7f1ec4004240 con 0x56292317d450 2024-05-09T08:48:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.323+0000 7f1edede8740 1 -- 172.21.15.107:0/4240974645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ed001cad0 con 0x56292317d450 2024-05-09T08:48:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.324+0000 7f1ebd7fa640 1 -- 172.21.15.107:0/4240974645 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1ec4035070 con 0x56292317d450 2024-05-09T08:48:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.324+0000 7f1edede8740 10 client.19293 fetch_fsmap learned FSMap version 29 2024-05-09T08:48:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.324+0000 7f1edede8740 1 -- 172.21.15.107:0/4240974645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562922ec2620 con 0x56292317d450 2024-05-09T08:48:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.324+0000 7f1ebd7fa640 1 -- 172.21.15.107:0/4240974645 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ed001cad0 con 0x56292317d450 2024-05-09T08:48:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.324+0000 7f1ebd7fa640 1 -- 172.21.15.107:0/4240974645 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f1ec403d450 con 0x56292317d450 2024-05-09T08:48:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.324+0000 7f1edede8740 10 client.19293 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:48:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.324+0000 7f1edede8740 10 client.19293 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:48:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.324+0000 7f1edede8740 20 client.19293 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.324+0000 7f1edede8740 1 --2- 172.21.15.107:0/4240974645 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562923198260 0x56292319a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.324+0000 7f1edede8740 4 client.19293 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:48:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.324+0000 7f1edede8740 1 -- 172.21.15.107:0/4240974645 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629230efa70 con 0x562923198260 2024-05-09T08:48:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.324+0000 7f1ed9e70640 1 --2- 172.21.15.107:0/4240974645 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562923198260 0x56292319a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.325+0000 7f1ebd7fa640 10 client.19293 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.325+0000 7f1ed9e70640 1 --2- 172.21.15.107:0/4240974645 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562923198260 0x56292319a690 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=253a6dc1290e2646 server_cookie=cfd209878d9a7d54 in_seq=0 out_seq=0 2024-05-09T08:48:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.325+0000 7f1ebd7fa640 1 -- 172.21.15.107:0/4240974645 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5629230efa70 con 0x562923198260 2024-05-09T08:48:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.325+0000 7f1ebd7fa640 10 client.19293 handle_command_reply: tid=0 2024-05-09T08:48:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.325+0000 7f1edede8740 1 client.19293 shutdown 2024-05-09T08:48:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.325+0000 7f1edede8740 1 -- 172.21.15.107:0/4240974645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 msgr2=0x562923185580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.326+0000 7f1edede8740 1 --2- 172.21.15.107:0/4240974645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x562923185580 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f1ec402a940 tx=0x7f1ec4032e30 comp rx=0 tx=0).stop 2024-05-09T08:48:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.326+0000 7f1edede8740 1 -- 172.21.15.107:0/4240974645 shutdown_connections 2024-05-09T08:48:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.326+0000 7f1edede8740 1 --2- 172.21.15.107:0/4240974645 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x562923198260 0x56292319a690 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.326+0000 7f1edede8740 1 --2- 172.21.15.107:0/4240974645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562923188630 0x562923186000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.326+0000 7f1ebd7fa640 0 client.19293 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.326+0000 7f1edede8740 1 --2- 172.21.15.107:0/4240974645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56292317dd70 0x562923185ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.326+0000 7f1edede8740 1 --2- 172.21.15.107:0/4240974645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x562923185580 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.326+0000 7f1edede8740 1 -- 172.21.15.107:0/4240974645 >> 172.21.15.107:0/4240974645 conn(0x562923179960 msgr2=0x56292317f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.326+0000 7f1edede8740 1 -- 172.21.15.107:0/4240974645 shutdown_connections 2024-05-09T08:48:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.326+0000 7f1edede8740 1 -- 172.21.15.107:0/4240974645 wait complete. 2024-05-09T08:48:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.326+0000 7f1edede8740 20 client.19293 trim_cache size 0 max 16384 2024-05-09T08:48:02.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1edede8740 1 Processor -- start 2024-05-09T08:48:02.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1edede8740 1 -- start start 2024-05-09T08:48:02.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1edede8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x562923187cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1edede8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56292317dd70 0x562923194120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1edede8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562923188630 0x562923190a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1edede8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed41b1610 con 0x56292317d450 2024-05-09T08:48:02.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1edede8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed41aeee0 con 0x56292317dd70 2024-05-09T08:48:02.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1edede8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed41acd50 con 0x562923188630 2024-05-09T08:48:02.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1ed966f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x562923187cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1ed966f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x562923187cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51800/0 (socket says 172.21.15.107:51800) 2024-05-09T08:48:02.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1ed966f640 1 -- 172.21.15.107:0/2075556924 learned_addr learned my addr 172.21.15.107:0/2075556924 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:02.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1ed966f640 1 -- 172.21.15.107:0/2075556924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562923188630 msgr2=0x562923190a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:02.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1ed966f640 1 --2- 172.21.15.107:0/2075556924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562923188630 0x562923190a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1ed966f640 1 -- 172.21.15.107:0/2075556924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56292317dd70 msgr2=0x562923194120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:02.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1ed966f640 1 --2- 172.21.15.107:0/2075556924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56292317dd70 0x562923194120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1ed966f640 1 -- 172.21.15.107:0/2075556924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562923177f00 con 0x56292317d450 2024-05-09T08:48:02.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.341+0000 7f1ed966f640 1 --2- 172.21.15.107:0/2075556924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x562923187cb0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4033000 tx=0x7f1ec40058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb481dfc0a336c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:02.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.342+0000 7f1ebf7fe640 1 -- 172.21.15.107:0/2075556924 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec405a070 con 0x56292317d450 2024-05-09T08:48:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.342+0000 7f1ebf7fe640 1 -- 172.21.15.107:0/2075556924 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ec403da90 con 0x56292317d450 2024-05-09T08:48:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.342+0000 7f1ebf7fe640 1 -- 172.21.15.107:0/2075556924 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec403de60 con 0x56292317d450 2024-05-09T08:48:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.342+0000 7f1edede8740 1 -- 172.21.15.107:0/2075556924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 msgr2=0x562923187cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.342+0000 7f1edede8740 1 --2- 172.21.15.107:0/2075556924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x562923187cb0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4033000 tx=0x7f1ec40058d0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.342+0000 7f1edede8740 1 -- 172.21.15.107:0/2075556924 shutdown_connections 2024-05-09T08:48:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.342+0000 7f1edede8740 1 --2- 172.21.15.107:0/2075556924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562923188630 0x562923190a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.342+0000 7f1edede8740 1 --2- 172.21.15.107:0/2075556924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56292317dd70 0x562923194120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.342+0000 7f1edede8740 1 --2- 172.21.15.107:0/2075556924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x562923187cb0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.342+0000 7f1edede8740 1 -- 172.21.15.107:0/2075556924 >> 172.21.15.107:0/2075556924 conn(0x562923179960 msgr2=0x56292317ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.342+0000 7f1edede8740 1 -- 172.21.15.107:0/2075556924 shutdown_connections 2024-05-09T08:48:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.342+0000 7f1edede8740 1 -- 172.21.15.107:0/2075556924 wait complete. 2024-05-09T08:48:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.343+0000 7f1edede8740 1 -- start start 2024-05-09T08:48:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.343+0000 7f1edede8740 1 Processor -- start 2024-05-09T08:48:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.343+0000 7f1edede8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x5629231f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.343+0000 7f1edede8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56292317dd70 0x5629231f6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.343+0000 7f1edede8740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562923188630 0x5629231fb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.343+0000 7f1edede8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed41ad3a0 con 0x56292317d450 2024-05-09T08:48:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.343+0000 7f1edede8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed41ae810 con 0x56292317dd70 2024-05-09T08:48:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.343+0000 7f1edede8740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed41ada70 con 0x562923188630 2024-05-09T08:48:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.343+0000 7f1edede8740 10 client.0 _ll_register_callbacks cb 0x7f1ec4009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.343+0000 7f1ed966f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x5629231f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.343+0000 7f1ed966f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x5629231f69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51810/0 (socket says 172.21.15.107:51810) 2024-05-09T08:48:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.343+0000 7f1ed966f640 1 -- 172.21.15.107:0/2155890711 learned_addr learned my addr 172.21.15.107:0/2155890711 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.343+0000 7f1ed9e70640 1 --2- 172.21.15.107:0/2155890711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56292317dd70 0x5629231f6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.344+0000 7f1ed966f640 1 -- 172.21.15.107:0/2155890711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562923188630 msgr2=0x5629231fb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.344+0000 7f1ed966f640 1 --2- 172.21.15.107:0/2155890711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562923188630 0x5629231fb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.344+0000 7f1ed966f640 1 -- 172.21.15.107:0/2155890711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56292317dd70 msgr2=0x5629231f6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.344+0000 7f1ed966f640 1 --2- 172.21.15.107:0/2155890711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56292317dd70 0x5629231f6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.344+0000 7f1ed966f640 1 -- 172.21.15.107:0/2155890711 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629231fbe10 con 0x56292317d450 2024-05-09T08:48:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.344+0000 7f1ed9e70640 1 --2- 172.21.15.107:0/2155890711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56292317dd70 0x5629231f6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.344+0000 7f1ed966f640 1 --2- 172.21.15.107:0/2155890711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x5629231f69c0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f1ec403d940 tx=0x7f1ec4032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c360c805010c42 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.344+0000 7f1ebd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:48:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.344+0000 7f1ebd7fa640 1 -- 172.21.15.107:0/2155890711 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec405a040 con 0x56292317d450 2024-05-09T08:48:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.344+0000 7f1ebd7fa640 1 -- 172.21.15.107:0/2155890711 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9595+0+0 (secure 0 0 0) 0x7f1ec4004350 con 0x56292317d450 2024-05-09T08:48:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.344+0000 7f1edede8740 1 -- 172.21.15.107:0/2155890711 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ec403da90 con 0x56292317d450 2024-05-09T08:48:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.344+0000 7f1ebd7fa640 1 -- 172.21.15.107:0/2155890711 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1ec406c050 con 0x56292317d450 2024-05-09T08:48:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.344+0000 7f1edede8740 10 client.19299 fetch_fsmap learned FSMap version 29 2024-05-09T08:48:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.344+0000 7f1edede8740 1 -- 172.21.15.107:0/2155890711 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5629232031a0 con 0x56292317d450 2024-05-09T08:48:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.344+0000 7f1ebd7fa640 1 -- 172.21.15.107:0/2155890711 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ec403da90 con 0x56292317d450 2024-05-09T08:48:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.344+0000 7f1ebd7fa640 1 -- 172.21.15.107:0/2155890711 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f1ec405d050 con 0x56292317d450 2024-05-09T08:48:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.345+0000 7f1edede8740 10 client.19299 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:48:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.345+0000 7f1edede8740 10 client.19299 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-05-09T08:48:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.345+0000 7f1edede8740 20 client.19299 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.345+0000 7f1edede8740 1 --2- 172.21.15.107:0/2155890711 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5629232037a0 0x562923205b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.345+0000 7f1edede8740 4 client.19299 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T08:48:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.345+0000 7f1edede8740 1 -- 172.21.15.107:0/2155890711 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5629230e9dc0 con 0x5629232037a0 2024-05-09T08:48:02.351 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.345+0000 7f1ed9e70640 1 --2- 172.21.15.107:0/2155890711 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5629232037a0 0x562923205b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:02.351 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.345+0000 7f1ebd7fa640 10 client.19299 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:02.351 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.345+0000 7f1ed9e70640 1 --2- 172.21.15.107:0/2155890711 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5629232037a0 0x562923205b90 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2149f8b679df937 server_cookie=a7ebbc2a2b709b93 in_seq=0 out_seq=0 2024-05-09T08:48:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.358+0000 7f1ebd7fa640 1 -- 172.21.15.107:0/2155890711 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5629230e9dc0 con 0x5629232037a0 2024-05-09T08:48:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.358+0000 7f1ebd7fa640 10 client.19299 handle_command_reply: tid=0 2024-05-09T08:48:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.358+0000 7f1edede8740 1 client.19299 shutdown 2024-05-09T08:48:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.358+0000 7f1edede8740 1 -- 172.21.15.107:0/2155890711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 msgr2=0x5629231f69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.358+0000 7f1edede8740 1 --2- 172.21.15.107:0/2155890711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x5629231f69c0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f1ec403d940 tx=0x7f1ec4032b90 comp rx=0 tx=0).stop 2024-05-09T08:48:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.358+0000 7f1edede8740 1 -- 172.21.15.107:0/2155890711 shutdown_connections 2024-05-09T08:48:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.358+0000 7f1edede8740 1 --2- 172.21.15.107:0/2155890711 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x5629232037a0 0x562923205b90 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.358+0000 7f1edede8740 1 --2- 172.21.15.107:0/2155890711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562923188630 0x5629231fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.358+0000 7f1ebd7fa640 0 client.19299 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.358+0000 7f1edede8740 1 --2- 172.21.15.107:0/2155890711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56292317dd70 0x5629231f6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.358+0000 7f1edede8740 1 --2- 172.21.15.107:0/2155890711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56292317d450 0x5629231f69c0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.358+0000 7f1edede8740 1 -- 172.21.15.107:0/2155890711 >> 172.21.15.107:0/2155890711 conn(0x562923179960 msgr2=0x562923178a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.358+0000 7f1edede8740 1 -- 172.21.15.107:0/2155890711 shutdown_connections 2024-05-09T08:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.358+0000 7f1edede8740 1 -- 172.21.15.107:0/2155890711 wait complete. 2024-05-09T08:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.358+0000 7f1edede8740 20 client.19299 trim_cache size 0 max 16384 2024-05-09T08:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-05-09T08:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-05-09T08:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-05-09T08:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-05-09T08:48:02.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.359+0000 7f1edc0d4640 1 -- 172.21.15.107:0/3462559943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1ec00a2da0 msgr2=0x7f1ec00a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:02.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.359+0000 7f1edc0d4640 1 --2- 172.21.15.107:0/3462559943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1ec00a2da0 0x7f1ec00a5260 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc09b6b0 tx=0x7f1ecc097590 comp rx=0 tx=0).stop 2024-05-09T08:48:02.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.359+0000 7f1edc0d4640 1 -- 172.21.15.107:0/3462559943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ed4106720 msgr2=0x7f1ed41a3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:02.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.359+0000 7f1edc0d4640 1 --2- 172.21.15.107:0/3462559943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ed4106720 0x7f1ed41a3e80 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00cad0 tx=0x7f1ecc00cf90 comp rx=0 tx=0).stop 2024-05-09T08:48:02.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.359+0000 7f1edc0d4640 1 -- 172.21.15.107:0/3462559943 shutdown_connections 2024-05-09T08:48:02.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.359+0000 7f1edc0d4640 1 --2- 172.21.15.107:0/3462559943 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f1ec00a2da0 0x7f1ec00a5260 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.359+0000 7f1edc0d4640 1 --2- 172.21.15.107:0/3462559943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ed4106720 0x7f1ed41a3e80 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.359+0000 7f1edc0d4640 1 --2- 172.21.15.107:0/3462559943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ed4101120 0x7f1ed4199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.359+0000 7f1edc0d4640 1 --2- 172.21.15.107:0/3462559943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ed4100760 0x7f1ed4198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:02.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.359+0000 7f1edc0d4640 1 -- 172.21.15.107:0/3462559943 >> 172.21.15.107:0/3462559943 conn(0x7f1ed40778b0 msgr2=0x7f1ed4103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:02.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.359+0000 7f1edc0d4640 1 -- 172.21.15.107:0/3462559943 shutdown_connections 2024-05-09T08:48:02.361 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-05-09T08:48:02.359+0000 7f1edc0d4640 1 -- 172.21.15.107:0/3462559943 wait complete. 2024-05-09T08:48:03.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:02 smithi107 ceph-mon[30122]: pgmap v460: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 22 KiB/s rd, 29 KiB/s wr, 16 op/s 2024-05-09T08:48:03.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:02 smithi157 ceph-mon[33614]: pgmap v460: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 22 KiB/s rd, 29 KiB/s wr, 16 op/s 2024-05-09T08:48:03.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:02 smithi110 ceph-mon[33622]: pgmap v460: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 22 KiB/s rd, 29 KiB/s wr, 16 op/s 2024-05-09T08:48:05.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:04 smithi107 ceph-mon[30122]: pgmap v461: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 26 KiB/s rd, 56 KiB/s wr, 22 op/s 2024-05-09T08:48:05.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:04 smithi157 ceph-mon[33614]: pgmap v461: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 26 KiB/s rd, 56 KiB/s wr, 22 op/s 2024-05-09T08:48:05.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:04 smithi110 ceph-mon[33622]: pgmap v461: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 26 KiB/s rd, 56 KiB/s wr, 22 op/s 2024-05-09T08:48:05.628 INFO:tasks.workunit.client.0.smithi107.stdout:Syncing()...26 sec 2024-05-09T08:48:05.628 INFO:tasks.workunit.client.0.smithi107.stdout:FFSB benchmark finished at: Thu May 9 08:48:05 2024 2024-05-09T08:48:05.628 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:48:05.628 INFO:tasks.workunit.client.0.smithi107.stdout:Results: 2024-05-09T08:48:05.628 INFO:tasks.workunit.client.0.smithi107.stdout:Benchmark took 342.97 sec 2024-05-09T08:48:05.629 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:48:05.629 INFO:tasks.workunit.client.0.smithi107.stdout:Total Results 2024-05-09T08:48:05.629 INFO:tasks.workunit.client.0.smithi107.stdout:=============== 2024-05-09T08:48:05.629 INFO:tasks.workunit.client.0.smithi107.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-05-09T08:48:05.629 INFO:tasks.workunit.client.0.smithi107.stdout: ======= ============ ========= ======= =========== ========== 2024-05-09T08:48:05.629 INFO:tasks.workunit.client.0.smithi107.stdout: write : 974080 2840.12 100.000% 100.000% 11.1MB/sec 2024-05-09T08:48:05.629 INFO:tasks.workunit.client.0.smithi107.stdout:- 2024-05-09T08:48:05.629 INFO:tasks.workunit.client.0.smithi107.stdout:2840.12 Transactions per Second 2024-05-09T08:48:05.629 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:48:05.629 INFO:tasks.workunit.client.0.smithi107.stdout:Throughput Results 2024-05-09T08:48:05.629 INFO:tasks.workunit.client.0.smithi107.stdout:=================== 2024-05-09T08:48:05.629 INFO:tasks.workunit.client.0.smithi107.stdout:Write Throughput: 11.1MB/sec 2024-05-09T08:48:05.634 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:48:05.634 INFO:tasks.workunit.client.0.smithi107.stdout:System Call Latency statistics in millisecs 2024-05-09T08:48:05.634 INFO:tasks.workunit.client.0.smithi107.stdout:===== 2024-05-09T08:48:05.634 INFO:tasks.workunit.client.0.smithi107.stdout: Min Avg Max Total Calls 2024-05-09T08:48:05.634 INFO:tasks.workunit.client.0.smithi107.stdout: ======== ======== ======== ============ 2024-05-09T08:48:05.634 INFO:tasks.workunit.client.0.smithi107.stdout:[ open] 0.464000 175.131627 20268.416016 761 2024-05-09T08:48:05.634 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[0] 5000.000000 - 10000.000000 : 1 2024-05-09T08:48:05.634 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-05-09T08:48:05.634 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[2] 1000.000000 - 2000.000000 : 1 2024-05-09T08:48:05.634 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[3] 500.000000 - 1000.000000 : 1 2024-05-09T08:48:05.634 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[4] 200.000000 - 500.000000 : 4 2024-05-09T08:48:05.634 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[5] 100.000000 - 200.000000 : 5 2024-05-09T08:48:05.634 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[6] 50.000000 - 100.000000 : 41 2024-05-09T08:48:05.635 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[7] 20.000000 - 50.000000 : 173 2024-05-09T08:48:05.635 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[8] 10.000000 - 20.000000 : 203 2024-05-09T08:48:05.635 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[9] 5.000000 - 10.000000 : 198 2024-05-09T08:48:05.635 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[10] 2.000000 - 5.000000 : 97 2024-05-09T08:48:05.635 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[11] 1.000000 - 2.000000 : 22 2024-05-09T08:48:05.635 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[12] 0.500000 - 1.000000 : 6 2024-05-09T08:48:05.635 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[13] 0.200000 - 0.500000 : 2 2024-05-09T08:48:05.635 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-05-09T08:48:05.635 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-05-09T08:48:05.635 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-05-09T08:48:05.635 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-05-09T08:48:05.635 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-05-09T08:48:05.635 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:48:05.635 INFO:tasks.workunit.client.0.smithi107.stdout:[ write] 0.082000 10.107621 48573.386719 974080 2024-05-09T08:48:05.635 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[0] 5000.000000 - 10000.000000 : 20 2024-05-09T08:48:05.636 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[1] 2000.000000 - 5000.000000 : 38 2024-05-09T08:48:05.636 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[2] 1000.000000 - 2000.000000 : 4 2024-05-09T08:48:05.636 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[3] 500.000000 - 1000.000000 : 87 2024-05-09T08:48:05.636 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[4] 200.000000 - 500.000000 : 1292 2024-05-09T08:48:05.636 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[5] 100.000000 - 200.000000 : 4936 2024-05-09T08:48:05.636 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[6] 50.000000 - 100.000000 : 11924 2024-05-09T08:48:05.636 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[7] 20.000000 - 50.000000 : 52711 2024-05-09T08:48:05.636 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[8] 10.000000 - 20.000000 : 84714 2024-05-09T08:48:05.636 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[9] 5.000000 - 10.000000 : 98595 2024-05-09T08:48:05.636 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[10] 2.000000 - 5.000000 : 108269 2024-05-09T08:48:05.636 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[11] 1.000000 - 2.000000 : 54269 2024-05-09T08:48:05.636 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[12] 0.500000 - 1.000000 : 55307 2024-05-09T08:48:05.636 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[13] 0.200000 - 0.500000 : 231885 2024-05-09T08:48:05.636 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[14] 0.100000 - 0.200000 : 262436 2024-05-09T08:48:05.636 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[15] 0.050000 - 0.100000 : 7448 2024-05-09T08:48:05.637 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-05-09T08:48:05.637 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-05-09T08:48:05.637 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-05-09T08:48:05.637 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:48:05.637 INFO:tasks.workunit.client.0.smithi107.stdout:[ lseek] 0.000000 0.000847 2.647000 974080 2024-05-09T08:48:05.637 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-05-09T08:48:05.637 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-05-09T08:48:05.637 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-05-09T08:48:05.637 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-05-09T08:48:05.637 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-05-09T08:48:05.637 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-05-09T08:48:05.638 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-05-09T08:48:05.638 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-05-09T08:48:05.638 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-05-09T08:48:05.638 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2024-05-09T08:48:05.638 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[10] 2.000000 - 5.000000 : 2 2024-05-09T08:48:05.638 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[11] 1.000000 - 2.000000 : 1 2024-05-09T08:48:05.638 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[12] 0.500000 - 1.000000 : 1 2024-05-09T08:48:05.638 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[13] 0.200000 - 0.500000 : 4 2024-05-09T08:48:05.638 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[14] 0.100000 - 0.200000 : 11 2024-05-09T08:48:05.638 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[15] 0.050000 - 0.100000 : 10 2024-05-09T08:48:05.638 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[16] 0.020000 - 0.050000 : 153 2024-05-09T08:48:05.638 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[17] 0.010000 - 0.020000 : 605 2024-05-09T08:48:05.638 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[18] 0.000000 - 0.010000 : 973293 2024-05-09T08:48:05.639 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:48:05.639 INFO:tasks.workunit.client.0.smithi107.stdout:[ close] 0.020000 1.836750 99.834000 761 2024-05-09T08:48:05.639 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-05-09T08:48:05.639 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-05-09T08:48:05.639 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-05-09T08:48:05.639 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-05-09T08:48:05.639 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-05-09T08:48:05.639 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-05-09T08:48:05.639 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[6] 50.000000 - 100.000000 : 4 2024-05-09T08:48:05.639 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[7] 20.000000 - 50.000000 : 13 2024-05-09T08:48:05.639 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[8] 10.000000 - 20.000000 : 16 2024-05-09T08:48:05.639 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[9] 5.000000 - 10.000000 : 36 2024-05-09T08:48:05.639 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[10] 2.000000 - 5.000000 : 38 2024-05-09T08:48:05.639 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[11] 1.000000 - 2.000000 : 39 2024-05-09T08:48:05.640 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[12] 0.500000 - 1.000000 : 32 2024-05-09T08:48:05.640 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[13] 0.200000 - 0.500000 : 54 2024-05-09T08:48:05.640 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[14] 0.100000 - 0.200000 : 125 2024-05-09T08:48:05.640 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[15] 0.050000 - 0.100000 : 69 2024-05-09T08:48:05.640 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[16] 0.020000 - 0.050000 : 335 2024-05-09T08:48:05.640 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-05-09T08:48:05.640 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-05-09T08:48:05.640 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:48:05.640 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:48:05.640 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:48:05.640 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-05-09T08:48:05.640 INFO:tasks.workunit.client.0.smithi107.stdout:0.4% User Time 2024-05-09T08:48:05.640 INFO:tasks.workunit.client.0.smithi107.stdout:21.6% System Time 2024-05-09T08:48:05.640 INFO:tasks.workunit.client.0.smithi107.stdout:22.1% CPU Utilization 2024-05-09T08:48:05.641 INFO:tasks.workunit.client.0.smithi107.stderr:+ cd .. 2024-05-09T08:48:05.644 INFO:tasks.workunit.client.0.smithi107.stderr:+ rm -r tmp ffsb 2024-05-09T08:48:07.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:06 smithi107 ceph-mon[30122]: pgmap v462: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 22 KiB/s rd, 59 KiB/s wr, 28 op/s 2024-05-09T08:48:07.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:06 smithi157 ceph-mon[33614]: pgmap v462: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 22 KiB/s rd, 59 KiB/s wr, 28 op/s 2024-05-09T08:48:07.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:06 smithi110 ceph-mon[33622]: pgmap v462: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 22 KiB/s rd, 59 KiB/s wr, 28 op/s 2024-05-09T08:48:08.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:07 smithi107 ceph-mon[30122]: osdmap e112: 12 total, 12 up, 12 in 2024-05-09T08:48:08.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:07 smithi157 ceph-mon[33614]: osdmap e112: 12 total, 12 up, 12 in 2024-05-09T08:48:08.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:07 smithi110 ceph-mon[33622]: osdmap e112: 12 total, 12 up, 12 in 2024-05-09T08:48:08.995 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:08 smithi157 ceph-mon[33614]: pgmap v464: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 26 KiB/s rd, 65 KiB/s wr, 30 op/s 2024-05-09T08:48:09.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:08 smithi107 ceph-mon[30122]: pgmap v464: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 26 KiB/s rd, 65 KiB/s wr, 30 op/s 2024-05-09T08:48:09.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:08 smithi110 ceph-mon[33622]: pgmap v464: 129 pgs: 129 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 26 KiB/s rd, 65 KiB/s wr, 30 op/s 2024-05-09T08:48:09.221 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:48:09.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:48:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.417+0000 7f8d27fff640 1 -- 172.21.15.107:0/2564604048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2810cff0 msgr2=0x7f8d28079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.417+0000 7f8d27fff640 1 --2- 172.21.15.107:0/2564604048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2810cff0 0x7f8d28079ab0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000b4a0 tx=0x7f8d2002ba20 comp rx=0 tx=0).stop 2024-05-09T08:48:09.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.421+0000 7f8d27fff640 1 -- 172.21.15.107:0/2564604048 shutdown_connections 2024-05-09T08:48:09.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.421+0000 7f8d27fff640 1 --2- 172.21.15.107:0/2564604048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2810cff0 0x7f8d28079ab0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.421+0000 7f8d27fff640 1 --2- 172.21.15.107:0/2564604048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d28109450 0x7f8d2810c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.421+0000 7f8d27fff640 1 --2- 172.21.15.107:0/2564604048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d28108aa0 0x7f8d28108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.421+0000 7f8d27fff640 1 -- 172.21.15.107:0/2564604048 >> 172.21.15.107:0/2564604048 conn(0x7f8d2806c7b0 msgr2=0x7f8d2806cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:09.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.428+0000 7f8d27fff640 1 -- 172.21.15.107:0/2564604048 shutdown_connections 2024-05-09T08:48:09.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.431+0000 7f8d27fff640 1 -- 172.21.15.107:0/2564604048 wait complete. 2024-05-09T08:48:09.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.431+0000 7f8d27fff640 1 Processor -- start 2024-05-09T08:48:09.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.436+0000 7f8d27fff640 1 -- start start 2024-05-09T08:48:09.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.436+0000 7f8d27fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d28108aa0 0x7f8d2807e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.436+0000 7f8d27fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d2810cff0 0x7f8d2807eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.436+0000 7f8d27fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2807f070 0x7f8d281bb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.436+0000 7f8d27fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2807cfc0 con 0x7f8d2807f070 2024-05-09T08:48:09.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.436+0000 7f8d27fff640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2807ce40 con 0x7f8d2810cff0 2024-05-09T08:48:09.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.436+0000 7f8d27fff640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2807d140 con 0x7f8d28108aa0 2024-05-09T08:48:09.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.440+0000 7f8d277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2807f070 0x7f8d281bb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.440+0000 7f8d26ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d28108aa0 0x7f8d2807e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.440+0000 7f8d277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2807f070 0x7f8d281bb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51814/0 (socket says 172.21.15.107:51814) 2024-05-09T08:48:09.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.440+0000 7f8d277fe640 1 -- 172.21.15.107:0/1412285391 learned_addr learned my addr 172.21.15.107:0/1412285391 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:09.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.440+0000 7f8d1e7fc640 1 --2- 172.21.15.107:0/1412285391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d2810cff0 0x7f8d2807eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.442+0000 7f8d26ffd640 1 -- 172.21.15.107:0/1412285391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d2810cff0 msgr2=0x7f8d2807eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.442+0000 7f8d26ffd640 1 --2- 172.21.15.107:0/1412285391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d2810cff0 0x7f8d2807eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.442+0000 7f8d26ffd640 1 -- 172.21.15.107:0/1412285391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2807f070 msgr2=0x7f8d281bb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.442+0000 7f8d26ffd640 1 --2- 172.21.15.107:0/1412285391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2807f070 0x7f8d281bb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.443+0000 7f8d26ffd640 1 -- 172.21.15.107:0/1412285391 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d281bbe00 con 0x7f8d28108aa0 2024-05-09T08:48:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.444+0000 7f8d26ffd640 1 --2- 172.21.15.107:0/1412285391 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d28108aa0 0x7f8d2807e5f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8d18009fd0 tx=0x7f8d1800b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.447+0000 7f8d24ff9640 1 -- 172.21.15.107:0/1412285391 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d18015020 con 0x7f8d28108aa0 2024-05-09T08:48:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.447+0000 7f8d24ff9640 1 -- 172.21.15.107:0/1412285391 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d18011ce0 con 0x7f8d28108aa0 2024-05-09T08:48:09.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.447+0000 7f8d24ff9640 1 -- 172.21.15.107:0/1412285391 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d18010de0 con 0x7f8d28108aa0 2024-05-09T08:48:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.448+0000 7f8d27fff640 1 -- 172.21.15.107:0/1412285391 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d281bc0f0 con 0x7f8d28108aa0 2024-05-09T08:48:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.448+0000 7f8d27fff640 1 -- 172.21.15.107:0/1412285391 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d281bc5f0 con 0x7f8d28108aa0 2024-05-09T08:48:09.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.453+0000 7f8d2ff9f740 1 Processor -- start 2024-05-09T08:48:09.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.456+0000 7f8d2ff9f740 1 -- start start 2024-05-09T08:48:09.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.456+0000 7f8d2ff9f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdbc430 0x56017cdbc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.456+0000 7f8d2ff9f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdbcd50 0x56017cdc6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.456+0000 7f8d2ff9f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdc7420 0x56017cdc9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.456+0000 7f8d2ff9f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56017cd21d00 con 0x56017cdbcd50 2024-05-09T08:48:09.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.456+0000 7f8d2ff9f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56017ccc94c0 con 0x56017cdc7420 2024-05-09T08:48:09.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.456+0000 7f8d2ff9f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56017cd1dcc0 con 0x56017cdbc430 2024-05-09T08:48:09.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.458+0000 7f8d277fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdbc430 0x56017cdbc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.458+0000 7f8d277fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdbc430 0x56017cdbc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:46608/0 (socket says 172.21.15.107:46608) 2024-05-09T08:48:09.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.458+0000 7f8d277fe640 1 -- 172.21.15.107:0/3036275057 learned_addr learned my addr 172.21.15.107:0/3036275057 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:09.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.458+0000 7f8d24ff9640 1 -- 172.21.15.107:0/1412285391 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8d18018020 con 0x7f8d28108aa0 2024-05-09T08:48:09.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.459+0000 7f8d24ff9640 1 --2- 172.21.15.107:0/1412285391 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8d140a2f00 0x7f8d140a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.460+0000 7f8d24ff9640 1 -- 172.21.15.107:0/1412285391 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7f8d180cedc0 con 0x7f8d28108aa0 2024-05-09T08:48:09.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.460+0000 7f8d1e7fc640 1 --2- 172.21.15.107:0/3036275057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdc7420 0x56017cdc9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.460+0000 7f8d277fe640 1 --2- 172.21.15.107:0/1412285391 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8d140a2f00 0x7f8d140a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.460+0000 7f8d277fe640 1 -- 172.21.15.107:0/3036275057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdc7420 msgr2=0x56017cdc9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:09.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.460+0000 7f8d277fe640 1 --2- 172.21.15.107:0/3036275057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdc7420 0x56017cdc9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.460+0000 7f8d277fe640 1 -- 172.21.15.107:0/3036275057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdbcd50 msgr2=0x56017cdc6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:09.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.460+0000 7f8d277fe640 1 --2- 172.21.15.107:0/3036275057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdbcd50 0x56017cdc6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.460+0000 7f8d277fe640 1 -- 172.21.15.107:0/3036275057 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56017caff7d0 con 0x56017cdbc430 2024-05-09T08:48:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.465+0000 7f8d277fe640 1 --2- 172.21.15.107:0/3036275057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdbc430 0x56017cdbc810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8d2002fcc0 tx=0x7f8d2002fcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=480a4d409883a6a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.468+0000 7f8d1d7fa640 1 -- 172.21.15.107:0/3036275057 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20034040 con 0x56017cdbc430 2024-05-09T08:48:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.468+0000 7f8d1d7fa640 1 -- 172.21.15.107:0/3036275057 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d20030b30 con 0x56017cdbc430 2024-05-09T08:48:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.468+0000 7f8d1d7fa640 1 -- 172.21.15.107:0/3036275057 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20004540 con 0x56017cdbc430 2024-05-09T08:48:09.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.468+0000 7f8d277fe640 1 --2- 172.21.15.107:0/1412285391 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8d140a2f00 0x7f8d140a53c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8d2004c000 tx=0x7f8d20048030 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:09.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.472+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/3036275057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdbc430 msgr2=0x56017cdbc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.472+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/3036275057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdbc430 0x56017cdbc810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8d2002fcc0 tx=0x7f8d2002fcf0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.474+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/3036275057 shutdown_connections 2024-05-09T08:48:09.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.474+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/3036275057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdc7420 0x56017cdc9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.474+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/3036275057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdbcd50 0x56017cdc6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.474+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/3036275057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdbc430 0x56017cdbc810 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.474+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/3036275057 >> 172.21.15.107:0/3036275057 conn(0x56017cdb8750 msgr2=0x56017cdb8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:09.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.479+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/3036275057 shutdown_connections 2024-05-09T08:48:09.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.482+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/3036275057 wait complete. 2024-05-09T08:48:09.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.483+0000 7f8d2ff9f740 1 -- start start 2024-05-09T08:48:09.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.483+0000 7f8d2ff9f740 1 Processor -- start 2024-05-09T08:48:09.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.484+0000 7f8d2ff9f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdbc430 0x56017cdc4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.484+0000 7f8d2ff9f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdbcd50 0x56017cdc49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.484+0000 7f8d2ff9f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdc7420 0x56017cdc4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.484+0000 7f8d2ff9f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56017cdca210 con 0x56017cdbc430 2024-05-09T08:48:09.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.484+0000 7f8d2ff9f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56017ccdcb90 con 0x56017cdc7420 2024-05-09T08:48:09.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.484+0000 7f8d2ff9f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56017cdca390 con 0x56017cdbcd50 2024-05-09T08:48:09.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.484+0000 7f8d2ff9f740 10 client.0 _ll_register_callbacks cb 0x56017cd18da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:09.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.487+0000 7f8d277fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdc7420 0x56017cdc4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.487+0000 7f8d277fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdc7420 0x56017cdc4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:53344/0 (socket says 172.21.15.107:53344) 2024-05-09T08:48:09.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.487+0000 7f8d277fe640 1 -- 172.21.15.107:0/963940287 learned_addr learned my addr 172.21.15.107:0/963940287 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:09.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.489+0000 7f8d26ffd640 1 --2- 172.21.15.107:0/963940287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdbc430 0x56017cdc4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.491+0000 7f8d277fe640 1 -- 172.21.15.107:0/963940287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdbcd50 msgr2=0x56017cdc49b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T08:48:09.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.491+0000 7f8d277fe640 1 --2- 172.21.15.107:0/963940287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdbcd50 0x56017cdc49b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.491+0000 7f8d277fe640 1 -- 172.21.15.107:0/963940287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdbc430 msgr2=0x56017cdc4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:09.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.491+0000 7f8d277fe640 1 --2- 172.21.15.107:0/963940287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdbc430 0x56017cdc4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.491+0000 7f8d277fe640 1 -- 172.21.15.107:0/963940287 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56017ca353f0 con 0x56017cdc7420 2024-05-09T08:48:09.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.492+0000 7f8d277fe640 1 --2- 172.21.15.107:0/963940287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdc7420 0x56017cdc4ef0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8d200097b0 tx=0x7f8d200041c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=332c1976dbc7f930 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:09.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.492+0000 7f8d0e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:48:09.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.492+0000 7f8d0e7fc640 1 -- 172.21.15.107:0/963940287 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2005e060 con 0x56017cdc7420 2024-05-09T08:48:09.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.493+0000 7f8d0e7fc640 1 -- 172.21.15.107:0/963940287 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7f8d20030ce0 con 0x56017cdc7420 2024-05-09T08:48:09.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.494+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/963940287 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d20030b30 con 0x56017cdc7420 2024-05-09T08:48:09.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.496+0000 7f8d0e7fc640 1 -- 172.21.15.107:0/963940287 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8d20004340 con 0x56017cdc7420 2024-05-09T08:48:09.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.496+0000 7f8d2ff9f740 10 client.28235 fetch_fsmap learned FSMap version 29 2024-05-09T08:48:09.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.496+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/963940287 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56017c9f50f0 con 0x56017cdc7420 2024-05-09T08:48:09.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.501+0000 7f8d0e7fc640 1 -- 172.21.15.107:0/963940287 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d20030b30 con 0x56017cdc7420 2024-05-09T08:48:09.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.502+0000 7f8d0e7fc640 1 -- 172.21.15.107:0/963940287 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f8d2004f070 con 0x56017cdc7420 2024-05-09T08:48:09.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.502+0000 7f8d2ff9f740 10 client.28235 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:48:09.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.502+0000 7f8d2ff9f740 10 client.28235 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:48:09.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.502+0000 7f8d2ff9f740 20 client.28235 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:09.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.502+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/963940287 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56017cdd7770 0x56017cdd9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.502+0000 7f8d2ff9f740 4 client.28235 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:48:09.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.502+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/963940287 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56017cd2f430 con 0x56017cdd7770 2024-05-09T08:48:09.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.504+0000 7f8d26ffd640 1 --2- 172.21.15.107:0/963940287 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56017cdd7770 0x56017cdd9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.507+0000 7f8d26ffd640 1 --2- 172.21.15.107:0/963940287 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56017cdd7770 0x56017cdd9bd0 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93d2cbd1193d4096 server_cookie=3d052de849fe0099 in_seq=0 out_seq=0 2024-05-09T08:48:09.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.507+0000 7f8d0e7fc640 10 client.28235 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:09.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.509+0000 7f8d0e7fc640 1 -- 172.21.15.107:0/963940287 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56017cd2f430 con 0x56017cdd7770 2024-05-09T08:48:09.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.509+0000 7f8d0e7fc640 10 client.28235 handle_command_reply: tid=0 2024-05-09T08:48:09.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.510+0000 7f8d2ff9f740 1 client.28235 shutdown 2024-05-09T08:48:09.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.513+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/963940287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdc7420 msgr2=0x56017cdc4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.513+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/963940287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdc7420 0x56017cdc4ef0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8d200097b0 tx=0x7f8d200041c0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.514+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/963940287 shutdown_connections 2024-05-09T08:48:09.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.514+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/963940287 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56017cdd7770 0x56017cdd9bd0 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.514+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/963940287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdc7420 0x56017cdc4ef0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.514+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/963940287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdbcd50 0x56017cdc49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.514+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/963940287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdbc430 0x56017cdc4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.514+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/963940287 >> 172.21.15.107:0/963940287 conn(0x56017cdb8750 msgr2=0x56017cdb7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:09.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.517+0000 7f8d0e7fc640 0 client.28235 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.519+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/963940287 shutdown_connections 2024-05-09T08:48:09.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.521+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/963940287 wait complete. 2024-05-09T08:48:09.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.521+0000 7f8d2ff9f740 20 client.28235 trim_cache size 0 max 16384 2024-05-09T08:48:09.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.560+0000 7f8d2ff9f740 1 Processor -- start 2024-05-09T08:48:09.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.560+0000 7f8d2ff9f740 1 -- start start 2024-05-09T08:48:09.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.561+0000 7f8d2ff9f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdbc430 0x56017cdd3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.561+0000 7f8d2ff9f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdbcd50 0x56017cdbe290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.561+0000 7f8d2ff9f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdc7420 0x56017cddf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.561+0000 7f8d1e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdbcd50 0x56017cdbe290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.561+0000 7f8d1e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdbcd50 0x56017cdbe290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51842/0 (socket says 172.21.15.107:51842) 2024-05-09T08:48:09.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.561+0000 7f8d1e7fc640 1 -- 172.21.15.107:0/2300915663 learned_addr learned my addr 172.21.15.107:0/2300915663 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:09.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.562+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/2300915663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8d281d99f0 con 0x56017cdbcd50 2024-05-09T08:48:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.562+0000 7f8d277fe640 1 --2- 172.21.15.107:0/2300915663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdc7420 0x56017cddf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.562+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/2300915663 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8d281d72c0 con 0x56017cdbc430 2024-05-09T08:48:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.562+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/2300915663 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8d281d51c0 con 0x56017cdc7420 2024-05-09T08:48:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.564+0000 7f8d26ffd640 1 --2- 172.21.15.107:0/2300915663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdbc430 0x56017cdd3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.565+0000 7f8d277fe640 1 -- 172.21.15.107:0/2300915663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdbc430 msgr2=0x56017cdd3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.565+0000 7f8d277fe640 1 --2- 172.21.15.107:0/2300915663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdbc430 0x56017cdd3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.565+0000 7f8d277fe640 1 -- 172.21.15.107:0/2300915663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdbcd50 msgr2=0x56017cdbe290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.565+0000 7f8d277fe640 1 --2- 172.21.15.107:0/2300915663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdbcd50 0x56017cdbe290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.565+0000 7f8d277fe640 1 -- 172.21.15.107:0/2300915663 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56017cdc9c30 con 0x56017cdc7420 2024-05-09T08:48:09.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.565+0000 7f8d277fe640 1 --2- 172.21.15.107:0/2300915663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdc7420 0x56017cddf4e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8d20002230 tx=0x7f8d20005b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d5e1af9a504273b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:09.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.572+0000 7f8d1cff9640 1 -- 172.21.15.107:0/2300915663 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2005e020 con 0x56017cdc7420 2024-05-09T08:48:09.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.572+0000 7f8d1cff9640 1 -- 172.21.15.107:0/2300915663 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d200044e0 con 0x56017cdc7420 2024-05-09T08:48:09.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.572+0000 7f8d1cff9640 1 -- 172.21.15.107:0/2300915663 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20068080 con 0x56017cdc7420 2024-05-09T08:48:09.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.573+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/2300915663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdc7420 msgr2=0x56017cddf4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.573+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/2300915663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdc7420 0x56017cddf4e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8d20002230 tx=0x7f8d20005b70 comp rx=0 tx=0).stop 2024-05-09T08:48:09.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.578+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/2300915663 shutdown_connections 2024-05-09T08:48:09.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.578+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/2300915663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdc7420 0x56017cddf4e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.578+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/2300915663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdbcd50 0x56017cdbe290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.578+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/2300915663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdbc430 0x56017cdd3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.578+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/2300915663 >> 172.21.15.107:0/2300915663 conn(0x56017cdb8750 msgr2=0x56017cdbdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:09.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.580+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/2300915663 shutdown_connections 2024-05-09T08:48:09.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.581+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/2300915663 wait complete. 2024-05-09T08:48:09.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.581+0000 7f8d2ff9f740 1 -- start start 2024-05-09T08:48:09.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.581+0000 7f8d2ff9f740 1 Processor -- start 2024-05-09T08:48:09.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.582+0000 7f8d2ff9f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdbc430 0x56017cdcf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.582+0000 7f8d2ff9f740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdbcd50 0x56017cdcabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.582+0000 7f8d2ff9f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdc7420 0x56017cdcb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.582+0000 7f8d2ff9f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8d281d5780 con 0x56017cdc7420 2024-05-09T08:48:09.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.582+0000 7f8d2ff9f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8d281d6bf0 con 0x56017cdbc430 2024-05-09T08:48:09.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.582+0000 7f8d2ff9f740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8d281d5e50 con 0x56017cdbcd50 2024-05-09T08:48:09.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.582+0000 7f8d2ff9f740 10 client.0 _ll_register_callbacks cb 0x7f8d2002fb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:09.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.584+0000 7f8d26ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdbc430 0x56017cdcf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.584+0000 7f8d26ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdbc430 0x56017cdcf810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:53362/0 (socket says 172.21.15.107:53362) 2024-05-09T08:48:09.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.584+0000 7f8d26ffd640 1 -- 172.21.15.107:0/1326924835 learned_addr learned my addr 172.21.15.107:0/1326924835 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:09.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.584+0000 7f8d277fe640 1 --2- 172.21.15.107:0/1326924835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdc7420 0x56017cdcb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.584+0000 7f8d26ffd640 1 -- 172.21.15.107:0/1326924835 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdbcd50 msgr2=0x56017cdcabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:09.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.584+0000 7f8d26ffd640 1 --2- 172.21.15.107:0/1326924835 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdbcd50 0x56017cdcabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.585+0000 7f8d26ffd640 1 -- 172.21.15.107:0/1326924835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdc7420 msgr2=0x56017cdcb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:09.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.585+0000 7f8d26ffd640 1 --2- 172.21.15.107:0/1326924835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdc7420 0x56017cdcb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.585+0000 7f8d26ffd640 1 -- 172.21.15.107:0/1326924835 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56017cdcb750 con 0x56017cdbc430 2024-05-09T08:48:09.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.585+0000 7f8d26ffd640 1 --2- 172.21.15.107:0/1326924835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdbc430 0x56017cdcf810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8d18010740 tx=0x7f8d180bb0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=239b3a2539a06584 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:09.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.585+0000 7f8d0e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:48:09.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.586+0000 7f8d0e7fc640 1 -- 172.21.15.107:0/1326924835 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1809b680 con 0x56017cdbc430 2024-05-09T08:48:09.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.586+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/1326924835 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d200044e0 con 0x56017cdbc430 2024-05-09T08:48:09.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.586+0000 7f8d0e7fc640 1 -- 172.21.15.107:0/1326924835 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7f8d180cc320 con 0x56017cdbc430 2024-05-09T08:48:09.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.586+0000 7f8d0e7fc640 1 -- 172.21.15.107:0/1326924835 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8d180ac050 con 0x56017cdbc430 2024-05-09T08:48:09.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.587+0000 7f8d2ff9f740 10 client.28241 fetch_fsmap learned FSMap version 29 2024-05-09T08:48:09.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.587+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/1326924835 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56017ce3db90 con 0x56017cdbc430 2024-05-09T08:48:09.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.591+0000 7f8d0e7fc640 1 -- 172.21.15.107:0/1326924835 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d200044e0 con 0x56017cdbc430 2024-05-09T08:48:09.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.591+0000 7f8d0e7fc640 1 -- 172.21.15.107:0/1326924835 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f8d180b8030 con 0x56017cdbc430 2024-05-09T08:48:09.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.591+0000 7f8d2ff9f740 10 client.28241 fetch_fsmap finished waiting for FSMap version 29 2024-05-09T08:48:09.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.591+0000 7f8d2ff9f740 10 client.28241 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:48:09.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.591+0000 7f8d2ff9f740 20 client.28241 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:09.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.591+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/1326924835 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56017ce3e360 0x56017ce407a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.591+0000 7f8d2ff9f740 4 client.28241 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:48:09.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.591+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/1326924835 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56017cd29780 con 0x56017ce3e360 2024-05-09T08:48:09.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.594+0000 7f8d1e7fc640 1 --2- 172.21.15.107:0/1326924835 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56017ce3e360 0x56017ce407a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.597+0000 7f8d1e7fc640 1 --2- 172.21.15.107:0/1326924835 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56017ce3e360 0x56017ce407a0 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4bbd6be9d32f944 server_cookie=8442bf26e31e47d6 in_seq=0 out_seq=0 2024-05-09T08:48:09.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.598+0000 7f8d0e7fc640 10 client.28241 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:09.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.599+0000 7f8d0e7fc640 1 -- 172.21.15.107:0/1326924835 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56017cd29780 con 0x56017ce3e360 2024-05-09T08:48:09.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.599+0000 7f8d0e7fc640 10 client.28241 handle_command_reply: tid=0 2024-05-09T08:48:09.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.599+0000 7f8d2ff9f740 1 client.28241 shutdown 2024-05-09T08:48:09.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.607+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/1326924835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdbc430 msgr2=0x56017cdcf810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.607+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/1326924835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdbc430 0x56017cdcf810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8d18010740 tx=0x7f8d180bb0b0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.611+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/1326924835 shutdown_connections 2024-05-09T08:48:09.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.611+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/1326924835 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56017ce3e360 0x56017ce407a0 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.611+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/1326924835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56017cdc7420 0x56017cdcb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.611+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/1326924835 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56017cdbcd50 0x56017cdcabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.611+0000 7f8d2ff9f740 1 --2- 172.21.15.107:0/1326924835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56017cdbc430 0x56017cdcf810 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.611+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/1326924835 >> 172.21.15.107:0/1326924835 conn(0x56017cdb8750 msgr2=0x56017cdc4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:09.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.613+0000 7f8d0e7fc640 0 client.28241 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:09.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.614+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/1326924835 shutdown_connections 2024-05-09T08:48:09.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.618+0000 7f8d2ff9f740 1 -- 172.21.15.107:0/1326924835 wait complete. 2024-05-09T08:48:09.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.620+0000 7f8d2ff9f740 20 client.28241 trim_cache size 0 max 16384 2024-05-09T08:48:09.625 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:48:09.626 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:48:09.626 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:48:09.626 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:48:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.630+0000 7f8d27fff640 1 -- 172.21.15.107:0/1412285391 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8d140a2f00 msgr2=0x7f8d140a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.630+0000 7f8d27fff640 1 --2- 172.21.15.107:0/1412285391 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8d140a2f00 0x7f8d140a53c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8d2004c000 tx=0x7f8d20048030 comp rx=0 tx=0).stop 2024-05-09T08:48:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.630+0000 7f8d27fff640 1 -- 172.21.15.107:0/1412285391 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d28108aa0 msgr2=0x7f8d2807e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.630+0000 7f8d27fff640 1 --2- 172.21.15.107:0/1412285391 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d28108aa0 0x7f8d2807e5f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8d18009fd0 tx=0x7f8d1800b870 comp rx=0 tx=0).stop 2024-05-09T08:48:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.633+0000 7f8d27fff640 1 -- 172.21.15.107:0/1412285391 shutdown_connections 2024-05-09T08:48:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.633+0000 7f8d27fff640 1 --2- 172.21.15.107:0/1412285391 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8d140a2f00 0x7f8d140a53c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.633+0000 7f8d27fff640 1 --2- 172.21.15.107:0/1412285391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2807f070 0x7f8d281bb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.633+0000 7f8d27fff640 1 --2- 172.21.15.107:0/1412285391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d2810cff0 0x7f8d2807eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.633+0000 7f8d27fff640 1 --2- 172.21.15.107:0/1412285391 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d28108aa0 0x7f8d2807e5f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.633+0000 7f8d27fff640 1 -- 172.21.15.107:0/1412285391 >> 172.21.15.107:0/1412285391 conn(0x7f8d2806c7b0 msgr2=0x7f8d28071410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:09.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.638+0000 7f8d27fff640 1 -- 172.21.15.107:0/1412285391 shutdown_connections 2024-05-09T08:48:09.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.638+0000 7f8d27fff640 1 -- 172.21.15.107:0/1412285391 wait complete. 2024-05-09T08:48:09.666 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:48:09.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:48:09.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.872+0000 7f07e4a2b640 1 -- 172.21.15.107:0/1279007437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d809acc0 msgr2=0x7f07d8004710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.872+0000 7f07e4a2b640 1 --2- 172.21.15.107:0/1279007437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d809acc0 0x7f07d8004710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f07e0064d50 tx=0x7f07e0096080 comp rx=0 tx=0).stop 2024-05-09T08:48:09.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.878+0000 7f07e4a2b640 1 -- 172.21.15.107:0/1279007437 shutdown_connections 2024-05-09T08:48:09.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.878+0000 7f07e4a2b640 1 --2- 172.21.15.107:0/1279007437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d80986b0 0x7f07d8098b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.878+0000 7f07e4a2b640 1 --2- 172.21.15.107:0/1279007437 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07d8004c50 0x7f07d80050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.878+0000 7f07e4a2b640 1 --2- 172.21.15.107:0/1279007437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d809acc0 0x7f07d8004710 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.878+0000 7f07e4a2b640 1 -- 172.21.15.107:0/1279007437 >> 172.21.15.107:0/1279007437 conn(0x7f07d800a710 msgr2=0x7f07d800ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:09.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.883+0000 7f07e4a2b640 1 -- 172.21.15.107:0/1279007437 shutdown_connections 2024-05-09T08:48:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.885+0000 7f07e4a2b640 1 -- 172.21.15.107:0/1279007437 wait complete. 2024-05-09T08:48:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.886+0000 7f07e4a2b640 1 Processor -- start 2024-05-09T08:48:09.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.886+0000 7f07e4a2b640 1 -- start start 2024-05-09T08:48:09.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.887+0000 7f07e4a2b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07d8004c50 0x7f07d8149670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.887+0000 7f07e4a2b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d80986b0 0x7f07d8149bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.887+0000 7f07e4a2b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d809acc0 0x7f07d8143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.887+0000 7f07e4a2b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f07d80ac3e0 con 0x7f07d809acc0 2024-05-09T08:48:09.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.887+0000 7f07e4a2b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f07d80ac260 con 0x7f07d80986b0 2024-05-09T08:48:09.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.887+0000 7f07e4a2b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f07d80ac560 con 0x7f07d8004c50 2024-05-09T08:48:09.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.887+0000 7f07df59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d809acc0 0x7f07d8143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.887+0000 7f07df59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d809acc0 0x7f07d8143730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51860/0 (socket says 172.21.15.107:51860) 2024-05-09T08:48:09.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.887+0000 7f07df59e640 1 -- 172.21.15.107:0/4178475229 learned_addr learned my addr 172.21.15.107:0/4178475229 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:09.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.887+0000 7f07de59c640 1 --2- 172.21.15.107:0/4178475229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d80986b0 0x7f07d8149bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.889+0000 7f07ded9d640 1 --2- 172.21.15.107:0/4178475229 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07d8004c50 0x7f07d8149670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.889+0000 7f07df59e640 1 -- 172.21.15.107:0/4178475229 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07d8004c50 msgr2=0x7f07d8149670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.889+0000 7f07df59e640 1 --2- 172.21.15.107:0/4178475229 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07d8004c50 0x7f07d8149670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.889+0000 7f07df59e640 1 -- 172.21.15.107:0/4178475229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d80986b0 msgr2=0x7f07d8149bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:09.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.889+0000 7f07df59e640 1 --2- 172.21.15.107:0/4178475229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d80986b0 0x7f07d8149bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:09.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.889+0000 7f07df59e640 1 -- 172.21.15.107:0/4178475229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d8143f40 con 0x7f07d809acc0 2024-05-09T08:48:09.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.889+0000 7f07df59e640 1 --2- 172.21.15.107:0/4178475229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d809acc0 0x7f07d8143730 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f07d000e970 tx=0x7f07d000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:09.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.889+0000 7f07c7fff640 1 -- 172.21.15.107:0/4178475229 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d000ccc0 con 0x7f07d809acc0 2024-05-09T08:48:09.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.890+0000 7f07c7fff640 1 -- 172.21.15.107:0/4178475229 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07d000ce60 con 0x7f07d809acc0 2024-05-09T08:48:09.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.890+0000 7f07c7fff640 1 -- 172.21.15.107:0/4178475229 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d0010430 con 0x7f07d809acc0 2024-05-09T08:48:09.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.893+0000 7f07ded9d640 1 --2- 172.21.15.107:0/4178475229 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07d8004c50 0x7f07d8149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:09.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.893+0000 7f07e4a2b640 1 -- 172.21.15.107:0/4178475229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07d8144230 con 0x7f07d809acc0 2024-05-09T08:48:09.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.893+0000 7f07e4a2b640 1 -- 172.21.15.107:0/4178475229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07d809b440 con 0x7f07d809acc0 2024-05-09T08:48:09.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.897+0000 7f07e4a2b640 1 -- 172.21.15.107:0/4178475229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07d80951f0 con 0x7f07d809acc0 2024-05-09T08:48:09.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.897+0000 7f07c7fff640 1 -- 172.21.15.107:0/4178475229 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f07d00046c0 con 0x7f07d809acc0 2024-05-09T08:48:09.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.898+0000 7f07c7fff640 1 --2- 172.21.15.107:0/4178475229 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07c80a2f00 0x7f07c80a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:09.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.898+0000 7f07c7fff640 1 -- 172.21.15.107:0/4178475229 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7f07d00cfc60 con 0x7f07d809acc0 2024-05-09T08:48:09.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.899+0000 7f07ded9d640 1 --2- 172.21.15.107:0/4178475229 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07c80a2f00 0x7f07c80a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:09.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.909+0000 7f07c7fff640 1 -- 172.21.15.107:0/4178475229 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f07d009c410 con 0x7f07d809acc0 2024-05-09T08:48:09.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:09.909+0000 7f07ded9d640 1 --2- 172.21.15.107:0/4178475229 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07c80a2f00 0x7f07c80a53c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f07e0066f90 tx=0x7f07e0062db0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:10.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.344+0000 7f07e4a2b640 1 -- 172.21.15.107:0/4178475229 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07d8006190 con 0x7f07d809acc0 2024-05-09T08:48:10.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.346+0000 7f07c7fff640 1 -- 172.21.15.107:0/4178475229 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11892 (secure 0 0 0) 0x7f07d009bd10 con 0x7f07d809acc0 2024-05-09T08:48:10.346 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:48:10.346 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":28109,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/2116846059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2116846059},{"type":"v1","addr":"172.21.15.157:6835","nonce":2116846059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:48:09.609485+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24418":{"gid":24418,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24478,24499]},"id":1}]} 2024-05-09T08:48:10.346 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 30 2024-05-09T08:48:10.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.352+0000 7f07e4a2b640 1 -- 172.21.15.107:0/4178475229 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07c80a2f00 msgr2=0x7f07c80a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:10.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.352+0000 7f07e4a2b640 1 --2- 172.21.15.107:0/4178475229 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07c80a2f00 0x7f07c80a53c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f07e0066f90 tx=0x7f07e0062db0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.352+0000 7f07e4a2b640 1 -- 172.21.15.107:0/4178475229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d809acc0 msgr2=0x7f07d8143730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:10.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.352+0000 7f07e4a2b640 1 --2- 172.21.15.107:0/4178475229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d809acc0 0x7f07d8143730 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f07d000e970 tx=0x7f07d000ed70 comp rx=0 tx=0).stop 2024-05-09T08:48:10.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.353+0000 7f07e4a2b640 1 -- 172.21.15.107:0/4178475229 shutdown_connections 2024-05-09T08:48:10.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.353+0000 7f07e4a2b640 1 --2- 172.21.15.107:0/4178475229 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f07c80a2f00 0x7f07c80a53c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.353+0000 7f07e4a2b640 1 --2- 172.21.15.107:0/4178475229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07d809acc0 0x7f07d8143730 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.353+0000 7f07e4a2b640 1 --2- 172.21.15.107:0/4178475229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d80986b0 0x7f07d8149bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.353+0000 7f07e4a2b640 1 --2- 172.21.15.107:0/4178475229 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07d8004c50 0x7f07d8149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.353+0000 7f07e4a2b640 1 -- 172.21.15.107:0/4178475229 >> 172.21.15.107:0/4178475229 conn(0x7f07d800a710 msgr2=0x7f07d80974d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:10.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.354+0000 7f07e4a2b640 1 -- 172.21.15.107:0/4178475229 shutdown_connections 2024-05-09T08:48:10.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.355+0000 7f07e4a2b640 1 -- 172.21.15.107:0/4178475229 wait complete. 2024-05-09T08:48:10.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:48:10.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.588+0000 7fa05581c640 1 -- 172.21.15.107:0/992840880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa05010e0b0 msgr2=0x7fa05010e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:10.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.588+0000 7fa05581c640 1 --2- 172.21.15.107:0/992840880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa05010e0b0 0x7fa05010e530 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa044009930 tx=0x7fa04402a480 comp rx=0 tx=0).stop 2024-05-09T08:48:10.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.588+0000 7fa05581c640 1 -- 172.21.15.107:0/992840880 shutdown_connections 2024-05-09T08:48:10.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.588+0000 7fa05581c640 1 --2- 172.21.15.107:0/992840880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa05011b010 0x7fa05011d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.588+0000 7fa05581c640 1 --2- 172.21.15.107:0/992840880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa05010e0b0 0x7fa05010e530 secure :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa044009930 tx=0x7fa04402a480 comp rx=0 tx=0).stop 2024-05-09T08:48:10.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.588+0000 7fa05581c640 1 --2- 172.21.15.107:0/992840880 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa050110080 0x7fa05010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.588+0000 7fa05581c640 1 -- 172.21.15.107:0/992840880 >> 172.21.15.107:0/992840880 conn(0x7fa05006c940 msgr2=0x7fa05006cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:10.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.592+0000 7fa05581c640 1 -- 172.21.15.107:0/992840880 shutdown_connections 2024-05-09T08:48:10.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.595+0000 7fa05581c640 1 -- 172.21.15.107:0/992840880 wait complete. 2024-05-09T08:48:10.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.596+0000 7fa05581c640 1 Processor -- start 2024-05-09T08:48:10.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.596+0000 7fa05581c640 1 -- start start 2024-05-09T08:48:10.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.596+0000 7fa05581c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa050110080 0x7fa0501a7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.596+0000 7fa05581c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa05011b010 0x7fa0501a7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.596+0000 7fa05581c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0501ab9a0 0x7fa0501abe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.596+0000 7fa05581c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa050120910 con 0x7fa050110080 2024-05-09T08:48:10.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.596+0000 7fa05581c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa050120790 con 0x7fa0501ab9a0 2024-05-09T08:48:10.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.596+0000 7fa05581c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa050120a90 con 0x7fa05011b010 2024-05-09T08:48:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.599+0000 7fa04f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0501ab9a0 0x7fa0501abe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.599+0000 7fa04f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0501ab9a0 0x7fa0501abe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:53392/0 (socket says 172.21.15.107:53392) 2024-05-09T08:48:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.599+0000 7fa04f7fe640 1 -- 172.21.15.107:0/3622204676 learned_addr learned my addr 172.21.15.107:0/3622204676 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.599+0000 7fa04e7fc640 1 --2- 172.21.15.107:0/3622204676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa05011b010 0x7fa0501a7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.601+0000 7fa04f7fe640 1 -- 172.21.15.107:0/3622204676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa05011b010 msgr2=0x7fa0501a7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.601+0000 7fa04f7fe640 1 --2- 172.21.15.107:0/3622204676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa05011b010 0x7fa0501a7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.601+0000 7fa04f7fe640 1 -- 172.21.15.107:0/3622204676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa050110080 msgr2=0x7fa0501a7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.601+0000 7fa04f7fe640 1 --2- 172.21.15.107:0/3622204676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa050110080 0x7fa0501a7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.601+0000 7fa04f7fe640 1 -- 172.21.15.107:0/3622204676 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0501ac490 con 0x7fa0501ab9a0 2024-05-09T08:48:10.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.602+0000 7fa04e7fc640 1 --2- 172.21.15.107:0/3622204676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa05011b010 0x7fa0501a7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:10.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.604+0000 7fa04f7fe640 1 --2- 172.21.15.107:0/3622204676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0501ab9a0 0x7fa0501abe50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa04000b3f0 tx=0x7fa04000b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:10.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.605+0000 7fa05481a640 1 -- 172.21.15.107:0/3622204676 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040015020 con 0x7fa0501ab9a0 2024-05-09T08:48:10.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.605+0000 7fa05481a640 1 -- 172.21.15.107:0/3622204676 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa040011ce0 con 0x7fa0501ab9a0 2024-05-09T08:48:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.605+0000 7fa05481a640 1 -- 172.21.15.107:0/3622204676 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040010dc0 con 0x7fa0501ab9a0 2024-05-09T08:48:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.606+0000 7fa05581c640 1 -- 172.21.15.107:0/3622204676 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0501ac780 con 0x7fa0501ab9a0 2024-05-09T08:48:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.606+0000 7fa05581c640 1 -- 172.21.15.107:0/3622204676 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0501b3ef0 con 0x7fa0501ab9a0 2024-05-09T08:48:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.619+0000 7fa058609740 1 Processor -- start 2024-05-09T08:48:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.619+0000 7fa058609740 1 -- start start 2024-05-09T08:48:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.619+0000 7fa058609740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55d4210 0x5559c55d45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.619+0000 7fa058609740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 0x5559c55efdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.619+0000 7fa058609740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55f0330 0x5559c55f2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.619+0000 7fa058609740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5559c5539c40 con 0x5559c55f0330 2024-05-09T08:48:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.619+0000 7fa058609740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5559c54e1580 con 0x5559c55d4b30 2024-05-09T08:48:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.619+0000 7fa058609740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5559c5535c00 con 0x5559c55d4210 2024-05-09T08:48:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.619+0000 7fa05481a640 1 -- 172.21.15.107:0/3622204676 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fa040018020 con 0x7fa0501ab9a0 2024-05-09T08:48:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.620+0000 7fa04effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 0x5559c55efdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.623+0000 7fa04e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55f0330 0x5559c55f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.623+0000 7fa04e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55f0330 0x5559c55f2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51878/0 (socket says 172.21.15.107:51878) 2024-05-09T08:48:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.623+0000 7fa04effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 0x5559c55efdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:53400/0 (socket says 172.21.15.107:53400) 2024-05-09T08:48:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.623+0000 7fa04e7fc640 1 -- 172.21.15.107:0/1251930016 learned_addr learned my addr 172.21.15.107:0/1251930016 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:10.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.623+0000 7fa04e7fc640 1 --2- 172.21.15.107:0/1251930016 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55d4210 0x5559c55d45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:10.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.625+0000 7fa04effd640 1 -- 172.21.15.107:0/1251930016 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55d4210 msgr2=0x5559c55d45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:10.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.625+0000 7fa04effd640 1 --2- 172.21.15.107:0/1251930016 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55d4210 0x5559c55d45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.625+0000 7fa04effd640 1 -- 172.21.15.107:0/1251930016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55f0330 msgr2=0x5559c55f2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:10.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.625+0000 7fa04effd640 1 --2- 172.21.15.107:0/1251930016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55f0330 0x5559c55f2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.625+0000 7fa04effd640 1 -- 172.21.15.107:0/1251930016 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559c524d3f0 con 0x5559c55d4b30 2024-05-09T08:48:10.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.625+0000 7fa05481a640 1 --2- 172.21.15.107:0/3622204676 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa0380a2f00 0x7fa0380a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.626+0000 7fa05481a640 1 -- 172.21.15.107:0/3622204676 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7fa0400cff30 con 0x7fa0501ab9a0 2024-05-09T08:48:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.626+0000 7fa04e7fc640 1 --2- 172.21.15.107:0/1251930016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55f0330 0x5559c55f2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.628+0000 7fa04e7fc640 1 --2- 172.21.15.107:0/3622204676 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa0380a2f00 0x7fa0380a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.629+0000 7fa04effd640 1 --2- 172.21.15.107:0/1251930016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 0x5559c55efdf0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa048007c30 tx=0x7fa04800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=491ab2c34b32c6f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:10.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.632+0000 7fa04e7fc640 1 --2- 172.21.15.107:0/3622204676 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa0380a2f00 0x7fa0380a53c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa0440096c0 tx=0x7fa044000bc0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.633+0000 7fa035ffb640 1 -- 172.21.15.107:0/1251930016 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa048014070 con 0x5559c55d4b30 2024-05-09T08:48:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.634+0000 7fa035ffb640 1 -- 172.21.15.107:0/1251930016 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa04800fb00 con 0x5559c55d4b30 2024-05-09T08:48:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.634+0000 7fa035ffb640 1 -- 172.21.15.107:0/1251930016 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04800e9d0 con 0x5559c55d4b30 2024-05-09T08:48:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.635+0000 7fa058609740 1 -- 172.21.15.107:0/1251930016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 msgr2=0x5559c55efdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.635+0000 7fa058609740 1 --2- 172.21.15.107:0/1251930016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 0x5559c55efdf0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa048007c30 tx=0x7fa04800c6a0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.636+0000 7fa058609740 1 -- 172.21.15.107:0/1251930016 shutdown_connections 2024-05-09T08:48:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.636+0000 7fa058609740 1 --2- 172.21.15.107:0/1251930016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55f0330 0x5559c55f2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.636+0000 7fa058609740 1 --2- 172.21.15.107:0/1251930016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 0x5559c55efdf0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.636+0000 7fa058609740 1 --2- 172.21.15.107:0/1251930016 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55d4210 0x5559c55d45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.636+0000 7fa058609740 1 -- 172.21.15.107:0/1251930016 >> 172.21.15.107:0/1251930016 conn(0x5559c55d0650 msgr2=0x5559c55d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.638+0000 7fa058609740 1 -- 172.21.15.107:0/1251930016 shutdown_connections 2024-05-09T08:48:10.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.638+0000 7fa058609740 1 -- 172.21.15.107:0/1251930016 wait complete. 2024-05-09T08:48:10.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.638+0000 7fa058609740 1 -- start start 2024-05-09T08:48:10.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.638+0000 7fa058609740 1 Processor -- start 2024-05-09T08:48:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.639+0000 7fa058609740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4210 0x5559c55ed250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.639+0000 7fa058609740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55d4b30 0x5559c55ed790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.639+0000 7fa058609740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 0x5559c55edcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.639+0000 7fa058609740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5559c55f3170 con 0x5559c55d4b30 2024-05-09T08:48:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.639+0000 7fa058609740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5559c54f4c40 con 0x5559c55d4210 2024-05-09T08:48:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.639+0000 7fa058609740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5559c55f32f0 con 0x5559c55f0330 2024-05-09T08:48:10.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.639+0000 7fa058609740 10 client.0 _ll_register_callbacks cb 0x5559c524cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:10.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.639+0000 7fa04f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 0x5559c55edcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:10.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.640+0000 7fa04f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 0x5559c55edcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:45358/0 (socket says 172.21.15.107:45358) 2024-05-09T08:48:10.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.640+0000 7fa04f7fe640 1 -- 172.21.15.107:0/3375159625 learned_addr learned my addr 172.21.15.107:0/3375159625 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:10.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.640+0000 7fa04effd640 1 --2- 172.21.15.107:0/3375159625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4210 0x5559c55ed250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:10.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.640+0000 7fa04e7fc640 1 --2- 172.21.15.107:0/3375159625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55d4b30 0x5559c55ed790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:10.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.640+0000 7fa04effd640 1 -- 172.21.15.107:0/3375159625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 msgr2=0x5559c55edcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:10.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.640+0000 7fa04effd640 1 --2- 172.21.15.107:0/3375159625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 0x5559c55edcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.640+0000 7fa04effd640 1 -- 172.21.15.107:0/3375159625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55d4b30 msgr2=0x5559c55ed790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:10.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.640+0000 7fa04effd640 1 --2- 172.21.15.107:0/3375159625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55d4b30 0x5559c55ed790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.640+0000 7fa04effd640 1 -- 172.21.15.107:0/3375159625 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559c53177d0 con 0x5559c55d4210 2024-05-09T08:48:10.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.641+0000 7fa04effd640 1 --2- 172.21.15.107:0/3375159625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4210 0x5559c55ed250 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa04800e630 tx=0x7fa04800f680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5b72d3023165da4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:10.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.641+0000 7fa02effd640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:48:10.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.642+0000 7fa02effd640 1 -- 172.21.15.107:0/3375159625 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa048014040 con 0x5559c55d4210 2024-05-09T08:48:10.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.642+0000 7fa02effd640 1 -- 172.21.15.107:0/3375159625 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7fa04800e6d0 con 0x5559c55d4210 2024-05-09T08:48:10.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.642+0000 7fa058609740 1 -- 172.21.15.107:0/3375159625 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa04800fb00 con 0x5559c55d4210 2024-05-09T08:48:10.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.643+0000 7fa02effd640 1 -- 172.21.15.107:0/3375159625 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa04800e8d0 con 0x5559c55d4210 2024-05-09T08:48:10.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.643+0000 7fa058609740 10 client.28259 fetch_fsmap learned FSMap version 30 2024-05-09T08:48:10.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.643+0000 7fa058609740 1 -- 172.21.15.107:0/3375159625 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5559c55ff3a0 con 0x5559c55d4210 2024-05-09T08:48:10.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.644+0000 7fa02effd640 1 -- 172.21.15.107:0/3375159625 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa04800fb00 con 0x5559c55d4210 2024-05-09T08:48:10.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.644+0000 7fa02effd640 1 -- 172.21.15.107:0/3375159625 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 30) ==== 8350+0+0 (secure 0 0 0) 0x7fa048019920 con 0x5559c55d4210 2024-05-09T08:48:10.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.644+0000 7fa058609740 10 client.28259 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T08:48:10.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.644+0000 7fa058609740 10 client.28259 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:48:10.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.644+0000 7fa058609740 20 client.28259 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:10.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.644+0000 7fa058609740 1 --2- 172.21.15.107:0/3375159625 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5559c5600800 0x5559c5602bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.644+0000 7fa058609740 4 client.28259 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:48:10.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.644+0000 7fa058609740 1 -- 172.21.15.107:0/3375159625 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559c5547370 con 0x5559c5600800 2024-05-09T08:48:10.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.648+0000 7fa04e7fc640 1 --2- 172.21.15.107:0/3375159625 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5559c5600800 0x5559c5602bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:10.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.657+0000 7fa04e7fc640 1 --2- 172.21.15.107:0/3375159625 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5559c5600800 0x5559c5602bf0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=12b4c20ce78d1399 server_cookie=bb9b4b1e32571ff2 in_seq=0 out_seq=0 2024-05-09T08:48:10.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.658+0000 7fa02effd640 10 client.28259 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:48:10.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.661+0000 7fa02effd640 1 -- 172.21.15.107:0/3375159625 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5559c5547370 con 0x5559c5600800 2024-05-09T08:48:10.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.661+0000 7fa02effd640 10 client.28259 handle_command_reply: tid=0 2024-05-09T08:48:10.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.661+0000 7fa058609740 1 client.28259 shutdown 2024-05-09T08:48:10.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.666+0000 7fa058609740 1 -- 172.21.15.107:0/3375159625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4210 msgr2=0x5559c55ed250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:10.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.666+0000 7fa058609740 1 --2- 172.21.15.107:0/3375159625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4210 0x5559c55ed250 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa04800e630 tx=0x7fa04800f680 comp rx=0 tx=0).stop 2024-05-09T08:48:10.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.666+0000 7fa058609740 1 -- 172.21.15.107:0/3375159625 shutdown_connections 2024-05-09T08:48:10.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.666+0000 7fa058609740 1 --2- 172.21.15.107:0/3375159625 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5559c5600800 0x5559c5602bf0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.666+0000 7fa058609740 1 --2- 172.21.15.107:0/3375159625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 0x5559c55edcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.666+0000 7fa058609740 1 --2- 172.21.15.107:0/3375159625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55d4b30 0x5559c55ed790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.666+0000 7fa058609740 1 --2- 172.21.15.107:0/3375159625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4210 0x5559c55ed250 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.666+0000 7fa058609740 1 -- 172.21.15.107:0/3375159625 >> 172.21.15.107:0/3375159625 conn(0x5559c55d0650 msgr2=0x5559c55d6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:10.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.666+0000 7fa02effd640 0 client.28259 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:48:10.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.667+0000 7fa058609740 1 -- 172.21.15.107:0/3375159625 shutdown_connections 2024-05-09T08:48:10.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.668+0000 7fa058609740 1 -- 172.21.15.107:0/3375159625 wait complete. 2024-05-09T08:48:10.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.668+0000 7fa058609740 20 client.28259 trim_cache size 0 max 16384 2024-05-09T08:48:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.698+0000 7fa058609740 1 Processor -- start 2024-05-09T08:48:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.699+0000 7fa058609740 1 -- start start 2024-05-09T08:48:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.699+0000 7fa058609740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55d4210 0x5559c55fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.699+0000 7fa058609740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 0x5559c55cf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.699+0000 7fa058609740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 0x5559c5608380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.699+0000 7fa058609740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c012a20 con 0x5559c55d4210 2024-05-09T08:48:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.699+0000 7fa058609740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c0102c0 con 0x5559c55d4b30 2024-05-09T08:48:10.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.700+0000 7fa058609740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c00e130 con 0x5559c55f0330 2024-05-09T08:48:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.700+0000 7fa04e7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 0x5559c55cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.700+0000 7fa04e7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 0x5559c55cf630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:40362/0 (socket says 172.21.15.107:40362) 2024-05-09T08:48:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.700+0000 7fa04e7fc640 1 -- 172.21.15.107:0/2854688706 learned_addr learned my addr 172.21.15.107:0/2854688706 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.701+0000 7fa04e7fc640 1 -- 172.21.15.107:0/2854688706 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 msgr2=0x5559c5608380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.701+0000 7fa04e7fc640 1 --2- 172.21.15.107:0/2854688706 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 0x5559c5608380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.701+0000 7fa04e7fc640 1 -- 172.21.15.107:0/2854688706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55d4210 msgr2=0x5559c55fc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.701+0000 7fa04e7fc640 1 --2- 172.21.15.107:0/2854688706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55d4210 0x5559c55fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.701+0000 7fa04e7fc640 1 -- 172.21.15.107:0/2854688706 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559c55d5780 con 0x5559c55d4b30 2024-05-09T08:48:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.701+0000 7fa04e7fc640 1 --2- 172.21.15.107:0/2854688706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 0x5559c55cf630 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa044004ff0 tx=0x7fa044041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5f16e186323c886 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.701+0000 7fa0357fa640 1 -- 172.21.15.107:0/2854688706 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0440054c0 con 0x5559c55d4b30 2024-05-09T08:48:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.702+0000 7fa0357fa640 1 -- 172.21.15.107:0/2854688706 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa044005660 con 0x5559c55d4b30 2024-05-09T08:48:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.702+0000 7fa0357fa640 1 -- 172.21.15.107:0/2854688706 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa044040ce0 con 0x5559c55d4b30 2024-05-09T08:48:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.703+0000 7fa058609740 1 -- 172.21.15.107:0/2854688706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 msgr2=0x5559c55cf630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.703+0000 7fa058609740 1 --2- 172.21.15.107:0/2854688706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 0x5559c55cf630 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa044004ff0 tx=0x7fa044041000 comp rx=0 tx=0).stop 2024-05-09T08:48:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.703+0000 7fa058609740 1 -- 172.21.15.107:0/2854688706 shutdown_connections 2024-05-09T08:48:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.703+0000 7fa058609740 1 --2- 172.21.15.107:0/2854688706 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 0x5559c5608380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.703+0000 7fa058609740 1 --2- 172.21.15.107:0/2854688706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 0x5559c55cf630 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.703+0000 7fa058609740 1 --2- 172.21.15.107:0/2854688706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55d4210 0x5559c55fc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.703+0000 7fa058609740 1 -- 172.21.15.107:0/2854688706 >> 172.21.15.107:0/2854688706 conn(0x5559c55d0650 msgr2=0x5559c55cef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:10.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.707+0000 7fa058609740 1 -- 172.21.15.107:0/2854688706 shutdown_connections 2024-05-09T08:48:10.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.707+0000 7fa058609740 1 -- 172.21.15.107:0/2854688706 wait complete. 2024-05-09T08:48:10.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.708+0000 7fa058609740 1 -- start start 2024-05-09T08:48:10.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.708+0000 7fa058609740 1 Processor -- start 2024-05-09T08:48:10.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.713+0000 7fa058609740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 0x5559c55f8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.713+0000 7fa058609740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 0x5559c55f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.713+0000 7fa058609740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55f44b0 0x5559c5665bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.713+0000 7fa058609740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c00e780 con 0x5559c55f44b0 2024-05-09T08:48:10.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.713+0000 7fa058609740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c00fbf0 con 0x5559c55d4b30 2024-05-09T08:48:10.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.713+0000 7fa058609740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c00ee50 con 0x5559c55f0330 2024-05-09T08:48:10.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.713+0000 7fa058609740 10 client.0 _ll_register_callbacks cb 0x7fa048007a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:10.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.717+0000 7fa04effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 0x5559c55f8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:10.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.717+0000 7fa04f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55f44b0 0x5559c5665bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:10.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.718+0000 7fa04e7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 0x5559c55f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:10.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.718+0000 7fa04e7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 0x5559c55f3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:45380/0 (socket says 172.21.15.107:45380) 2024-05-09T08:48:10.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.718+0000 7fa04e7fc640 1 -- 172.21.15.107:0/326404504 learned_addr learned my addr 172.21.15.107:0/326404504 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:10.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.724+0000 7fa04e7fc640 1 -- 172.21.15.107:0/326404504 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 msgr2=0x5559c55f8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:10.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.724+0000 7fa04e7fc640 1 --2- 172.21.15.107:0/326404504 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 0x5559c55f8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.724+0000 7fa04e7fc640 1 -- 172.21.15.107:0/326404504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55f44b0 msgr2=0x5559c5665bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:10.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.724+0000 7fa04e7fc640 1 --2- 172.21.15.107:0/326404504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55f44b0 0x5559c5665bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.724+0000 7fa04e7fc640 1 -- 172.21.15.107:0/326404504 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559c55f4bf0 con 0x5559c55f0330 2024-05-09T08:48:10.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.726+0000 7fa04e7fc640 1 --2- 172.21.15.107:0/326404504 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 0x5559c55f3e10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa044059040 tx=0x7fa0440052c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79ec6b1db92915fb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:10.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.727+0000 7fa02effd640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:48:10.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.729+0000 7fa02effd640 1 -- 172.21.15.107:0/326404504 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa044066070 con 0x5559c55f0330 2024-05-09T08:48:10.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.729+0000 7fa058609740 1 -- 172.21.15.107:0/326404504 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa044005660 con 0x5559c55f0330 2024-05-09T08:48:10.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.729+0000 7fa02effd640 1 -- 172.21.15.107:0/326404504 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7fa044040450 con 0x5559c55f0330 2024-05-09T08:48:10.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.735+0000 7fa02effd640 1 -- 172.21.15.107:0/326404504 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa044070050 con 0x5559c55f0330 2024-05-09T08:48:10.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.735+0000 7fa058609740 10 client.28465 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:10.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.735+0000 7fa058609740 1 -- 172.21.15.107:0/326404504 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5559c5667120 con 0x5559c55f0330 2024-05-09T08:48:10.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.739+0000 7fa02effd640 1 -- 172.21.15.107:0/326404504 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa044005660 con 0x5559c55f0330 2024-05-09T08:48:10.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.739+0000 7fa02effd640 1 -- 172.21.15.107:0/326404504 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fa04406b440 con 0x5559c55f0330 2024-05-09T08:48:10.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.739+0000 7fa058609740 10 client.28465 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:10.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.739+0000 7fa058609740 10 client.28465 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:48:10.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.739+0000 7fa058609740 20 client.28465 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:10.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.739+0000 7fa058609740 1 --2- 172.21.15.107:0/326404504 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5559c56678b0 0x5559c5669cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:10.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.739+0000 7fa058609740 4 client.28465 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:48:10.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.739+0000 7fa058609740 1 -- 172.21.15.107:0/326404504 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5559c55416c0 con 0x5559c56678b0 2024-05-09T08:48:10.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.740+0000 7fa04effd640 1 --2- 172.21.15.107:0/326404504 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5559c56678b0 0x5559c5669cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:10.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.745+0000 7fa04effd640 1 --2- 172.21.15.107:0/326404504 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5559c56678b0 0x5559c5669cd0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1976c85824db8edb server_cookie=90890bf0e8bc2677 in_seq=0 out_seq=0 2024-05-09T08:48:10.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.746+0000 7fa02effd640 10 client.28465 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:48:10.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.748+0000 7fa02effd640 1 -- 172.21.15.107:0/326404504 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5559c55416c0 con 0x5559c56678b0 2024-05-09T08:48:10.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.748+0000 7fa02effd640 10 client.28465 handle_command_reply: tid=0 2024-05-09T08:48:10.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.748+0000 7fa058609740 1 client.28465 shutdown 2024-05-09T08:48:10.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.749+0000 7fa058609740 1 -- 172.21.15.107:0/326404504 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 msgr2=0x5559c55f3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:10.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.749+0000 7fa058609740 1 --2- 172.21.15.107:0/326404504 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 0x5559c55f3e10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa044059040 tx=0x7fa0440052c0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.750+0000 7fa058609740 1 -- 172.21.15.107:0/326404504 shutdown_connections 2024-05-09T08:48:10.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.750+0000 7fa058609740 1 --2- 172.21.15.107:0/326404504 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x5559c56678b0 0x5559c5669cd0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.750+0000 7fa058609740 1 --2- 172.21.15.107:0/326404504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5559c55f44b0 0x5559c5665bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.750+0000 7fa058609740 1 --2- 172.21.15.107:0/326404504 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5559c55f0330 0x5559c55f3e10 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.750+0000 7fa058609740 1 --2- 172.21.15.107:0/326404504 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5559c55d4b30 0x5559c55f8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.750+0000 7fa058609740 1 -- 172.21.15.107:0/326404504 >> 172.21.15.107:0/326404504 conn(0x5559c55d0650 msgr2=0x5559c55d5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:10.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.750+0000 7fa02effd640 0 client.28465 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:48:10.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.750+0000 7fa058609740 1 -- 172.21.15.107:0/326404504 shutdown_connections 2024-05-09T08:48:10.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.750+0000 7fa058609740 1 -- 172.21.15.107:0/326404504 wait complete. 2024-05-09T08:48:10.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.751+0000 7fa058609740 20 client.28465 trim_cache size 0 max 16384 2024-05-09T08:48:10.756 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:48:10.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.760+0000 7fa05581c640 1 -- 172.21.15.107:0/3622204676 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa0380a2f00 msgr2=0x7fa0380a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:10.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.760+0000 7fa05581c640 1 --2- 172.21.15.107:0/3622204676 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa0380a2f00 0x7fa0380a53c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa0440096c0 tx=0x7fa044000bc0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.760+0000 7fa05581c640 1 -- 172.21.15.107:0/3622204676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0501ab9a0 msgr2=0x7fa0501abe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:10.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.760+0000 7fa05581c640 1 --2- 172.21.15.107:0/3622204676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0501ab9a0 0x7fa0501abe50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa04000b3f0 tx=0x7fa04000b8b0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.763+0000 7fa05581c640 1 -- 172.21.15.107:0/3622204676 shutdown_connections 2024-05-09T08:48:10.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.763+0000 7fa05581c640 1 --2- 172.21.15.107:0/3622204676 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fa0380a2f00 0x7fa0380a53c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.763+0000 7fa05581c640 1 --2- 172.21.15.107:0/3622204676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0501ab9a0 0x7fa0501abe50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.763+0000 7fa05581c640 1 --2- 172.21.15.107:0/3622204676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa05011b010 0x7fa0501a7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.763+0000 7fa05581c640 1 --2- 172.21.15.107:0/3622204676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa050110080 0x7fa0501a7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:10.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.763+0000 7fa05581c640 1 -- 172.21.15.107:0/3622204676 >> 172.21.15.107:0/3622204676 conn(0x7fa05006c940 msgr2=0x7fa050072330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:10.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.766+0000 7fa05581c640 1 -- 172.21.15.107:0/3622204676 shutdown_connections 2024-05-09T08:48:10.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.766+0000 7fa05581c640 1 -- 172.21.15.107:0/3622204676 wait complete. 2024-05-09T08:48:10.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:48:10.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.997+0000 7f57531a9640 1 -- 172.21.15.107:0/204344693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c112690 msgr2=0x7f574c114a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:10.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:10.997+0000 7f57531a9640 1 --2- 172.21.15.107:0/204344693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c112690 0x7f574c114a80 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f573c009900 tx=0x7f573c02a4d0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.002+0000 7f57531a9640 1 -- 172.21.15.107:0/204344693 shutdown_connections 2024-05-09T08:48:11.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.002+0000 7f57531a9640 1 --2- 172.21.15.107:0/204344693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c112690 0x7f574c114a80 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.002+0000 7f57531a9640 1 --2- 172.21.15.107:0/204344693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f574c100290 0x7f574c112150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.002+0000 7f57531a9640 1 --2- 172.21.15.107:0/204344693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f574c0ff970 0x7f574c0ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.002+0000 7f57531a9640 1 -- 172.21.15.107:0/204344693 >> 172.21.15.107:0/204344693 conn(0x7f574c076890 msgr2=0x7f574c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:11.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.008+0000 7f57531a9640 1 -- 172.21.15.107:0/204344693 shutdown_connections 2024-05-09T08:48:11.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.012+0000 7f57531a9640 1 -- 172.21.15.107:0/204344693 wait complete. 2024-05-09T08:48:11.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.012+0000 7f57531a9640 1 Processor -- start 2024-05-09T08:48:11.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.017+0000 7f57531a9640 1 -- start start 2024-05-09T08:48:11.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.017+0000 7f57531a9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f574c0ff970 0x7f574c072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.017+0000 7f57531a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f574c100290 0x7f574c0734b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.017+0000 7f57531a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c112690 0x7f574c06c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.017+0000 7f57531a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f574c1181d0 con 0x7f574c112690 2024-05-09T08:48:11.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.017+0000 7f57531a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f574c118050 con 0x7f574c100290 2024-05-09T08:48:11.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.017+0000 7f57531a9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f574c118350 con 0x7f574c0ff970 2024-05-09T08:48:11.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.019+0000 7f5750f45640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f574c0ff970 0x7f574c072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.019+0000 7f574bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f574c100290 0x7f574c0734b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.019+0000 7f574bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f574c100290 0x7f574c0734b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:40402/0 (socket says 172.21.15.107:40402) 2024-05-09T08:48:11.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.019+0000 7f574bfff640 1 -- 172.21.15.107:0/2275212157 learned_addr learned my addr 172.21.15.107:0/2275212157 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:11.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.021+0000 7f5751746640 1 --2- 172.21.15.107:0/2275212157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c112690 0x7f574c06c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.022+0000 7f574bfff640 1 -- 172.21.15.107:0/2275212157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f574c0ff970 msgr2=0x7f574c072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.022+0000 7f574bfff640 1 --2- 172.21.15.107:0/2275212157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f574c0ff970 0x7f574c072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.022+0000 7f574bfff640 1 -- 172.21.15.107:0/2275212157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c112690 msgr2=0x7f574c06c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.022+0000 7f574bfff640 1 --2- 172.21.15.107:0/2275212157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c112690 0x7f574c06c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.022+0000 7f574bfff640 1 -- 172.21.15.107:0/2275212157 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f574c06ca90 con 0x7f574c100290 2024-05-09T08:48:11.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.027+0000 7f574bfff640 1 --2- 172.21.15.107:0/2275212157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f574c100290 0x7f574c0734b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f574400b9e0 tx=0x7f574400bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:11.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:10 smithi107 ceph-mon[30122]: pgmap v465: 129 pgs: 18 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 60 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 120 KiB/s rd, 190 KiB/s wr, 35 op/s 2024-05-09T08:48:11.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:10 smithi107 ceph-mon[30122]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:48:11.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:10 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/4178475229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:11.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.030+0000 7f5749ffb640 1 -- 172.21.15.107:0/2275212157 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5744019070 con 0x7f574c100290 2024-05-09T08:48:11.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.030+0000 7f57531a9640 1 -- 172.21.15.107:0/2275212157 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f574c06cd80 con 0x7f574c100290 2024-05-09T08:48:11.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.030+0000 7f57531a9640 1 -- 172.21.15.107:0/2275212157 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f574c070d20 con 0x7f574c100290 2024-05-09T08:48:11.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.030+0000 7f5749ffb640 1 -- 172.21.15.107:0/2275212157 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f574400cba0 con 0x7f574c100290 2024-05-09T08:48:11.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.030+0000 7f5749ffb640 1 -- 172.21.15.107:0/2275212157 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57440125c0 con 0x7f574c100290 2024-05-09T08:48:11.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.036+0000 7f5755f98740 1 Processor -- start 2024-05-09T08:48:11.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.038+0000 7f5749ffb640 1 -- 172.21.15.107:0/2275212157 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f574401c070 con 0x7f574c100290 2024-05-09T08:48:11.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.039+0000 7f5755f98740 1 -- start start 2024-05-09T08:48:11.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.039+0000 7f5755f98740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4330 0x5639fd5f4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.039+0000 7f5755f98740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4c50 0x5639fd5fede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.039+0000 7f5755f98740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5ff320 0x5639fd601710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.039+0000 7f5755f98740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5639fd559c40 con 0x5639fd5f4330 2024-05-09T08:48:11.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.039+0000 7f5755f98740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5639fd501580 con 0x5639fd5f4c50 2024-05-09T08:48:11.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.039+0000 7f5755f98740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5639fd555c00 con 0x5639fd5ff320 2024-05-09T08:48:11.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.039+0000 7f5749ffb640 1 --2- 172.21.15.107:0/2275212157 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f57380a2f10 0x7f57380a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.039+0000 7f5751746640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4330 0x5639fd5f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.040+0000 7f5751746640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4330 0x5639fd5f4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41530/0 (socket says 172.21.15.107:41530) 2024-05-09T08:48:11.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.040+0000 7f5751746640 1 -- 172.21.15.107:0/139701047 learned_addr learned my addr 172.21.15.107:0/139701047 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:11.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.040+0000 7f5749ffb640 1 -- 172.21.15.107:0/2275212157 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7f57440ceb90 con 0x7f574c100290 2024-05-09T08:48:11.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.040+0000 7f5751746640 1 --2- 172.21.15.107:0/2275212157 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f57380a2f10 0x7f57380a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.039+0000 7f574bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5ff320 0x5639fd601710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.041+0000 7f5750f45640 1 --2- 172.21.15.107:0/139701047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4c50 0x5639fd5fede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.041+0000 7f5751746640 1 -- 172.21.15.107:0/139701047 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5ff320 msgr2=0x5639fd601710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:11.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.041+0000 7f5751746640 1 --2- 172.21.15.107:0/139701047 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5ff320 0x5639fd601710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.041+0000 7f5751746640 1 -- 172.21.15.107:0/139701047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4c50 msgr2=0x5639fd5fede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:11.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.041+0000 7f5751746640 1 --2- 172.21.15.107:0/139701047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4c50 0x5639fd5fede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.041+0000 7f5751746640 1 -- 172.21.15.107:0/139701047 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639fd26d3f0 con 0x5639fd5f4330 2024-05-09T08:48:11.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.042+0000 7f5751746640 1 --2- 172.21.15.107:0/139701047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4330 0x5639fd5f4710 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f573c009230 tx=0x7f573c031bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7505775a1fa94e4b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:11.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.042+0000 7f5751746640 1 --2- 172.21.15.107:0/2275212157 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f57380a2f10 0x7f57380a53d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f573c045000 tx=0x7f573c047000 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:11.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.042+0000 7f5742ffd640 1 -- 172.21.15.107:0/139701047 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c035040 con 0x5639fd5f4330 2024-05-09T08:48:11.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.042+0000 7f5742ffd640 1 -- 172.21.15.107:0/139701047 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f573c0043c0 con 0x5639fd5f4330 2024-05-09T08:48:11.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.042+0000 7f5742ffd640 1 -- 172.21.15.107:0/139701047 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c0496d0 con 0x5639fd5f4330 2024-05-09T08:48:11.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.043+0000 7f5755f98740 1 -- 172.21.15.107:0/139701047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4330 msgr2=0x5639fd5f4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.043+0000 7f5755f98740 1 --2- 172.21.15.107:0/139701047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4330 0x5639fd5f4710 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f573c009230 tx=0x7f573c031bd0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.044+0000 7f5755f98740 1 -- 172.21.15.107:0/139701047 shutdown_connections 2024-05-09T08:48:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.044+0000 7f5755f98740 1 --2- 172.21.15.107:0/139701047 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5ff320 0x5639fd601710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.044+0000 7f5755f98740 1 --2- 172.21.15.107:0/139701047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4c50 0x5639fd5fede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.044+0000 7f5755f98740 1 --2- 172.21.15.107:0/139701047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4330 0x5639fd5f4710 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.044+0000 7f5755f98740 1 -- 172.21.15.107:0/139701047 >> 172.21.15.107:0/139701047 conn(0x5639fd5f0650 msgr2=0x5639fd5f0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:11.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.048+0000 7f5755f98740 1 -- 172.21.15.107:0/139701047 shutdown_connections 2024-05-09T08:48:11.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.050+0000 7f5755f98740 1 -- 172.21.15.107:0/139701047 wait complete. 2024-05-09T08:48:11.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.050+0000 7f5755f98740 1 -- start start 2024-05-09T08:48:11.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.051+0000 7f5755f98740 1 Processor -- start 2024-05-09T08:48:11.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.053+0000 7f5755f98740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5f4330 0x5639fd5fc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.053+0000 7f5755f98740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4c50 0x5639fd5fc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.053+0000 7f5755f98740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5ff320 0x5639fd5fcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.053+0000 7f5755f98740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5639fd6020b0 con 0x5639fd5f4c50 2024-05-09T08:48:11.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.053+0000 7f5755f98740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5639fd514c40 con 0x5639fd5ff320 2024-05-09T08:48:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.053+0000 7f5755f98740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5639fd602230 con 0x5639fd5f4330 2024-05-09T08:48:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.053+0000 7f5755f98740 10 client.0 _ll_register_callbacks cb 0x5639fd26cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.055+0000 7f5751746640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5ff320 0x5639fd5fcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.055+0000 7f5751746640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5ff320 0x5639fd5fcd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:40426/0 (socket says 172.21.15.107:40426) 2024-05-09T08:48:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.055+0000 7f5751746640 1 -- 172.21.15.107:0/3883172575 learned_addr learned my addr 172.21.15.107:0/3883172575 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.055+0000 7f5750f45640 1 --2- 172.21.15.107:0/3883172575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5f4330 0x5639fd5fc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.055+0000 7f574bfff640 1 --2- 172.21.15.107:0/3883172575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4c50 0x5639fd5fc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.056+0000 7f5751746640 1 -- 172.21.15.107:0/3883172575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5f4330 msgr2=0x5639fd5fc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.056+0000 7f5751746640 1 --2- 172.21.15.107:0/3883172575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5f4330 0x5639fd5fc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.056+0000 7f5751746640 1 -- 172.21.15.107:0/3883172575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4c50 msgr2=0x5639fd5fc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.056+0000 7f5751746640 1 --2- 172.21.15.107:0/3883172575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4c50 0x5639fd5fc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.056+0000 7f5751746640 1 -- 172.21.15.107:0/3883172575 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639fd3377d0 con 0x5639fd5ff320 2024-05-09T08:48:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.056+0000 7f5751746640 1 --2- 172.21.15.107:0/3883172575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5ff320 0x5639fd5fcd90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f573c02a510 tx=0x7f573c004570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34ce3f624e0e7370 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.056+0000 7f572bfff640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:48:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.056+0000 7f572bfff640 1 -- 172.21.15.107:0/3883172575 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c057070 con 0x5639fd5ff320 2024-05-09T08:48:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.056+0000 7f5755f98740 1 -- 172.21.15.107:0/3883172575 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f573c0043c0 con 0x5639fd5ff320 2024-05-09T08:48:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.057+0000 7f572bfff640 1 -- 172.21.15.107:0/3883172575 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7f573c003a00 con 0x5639fd5ff320 2024-05-09T08:48:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.057+0000 7f572bfff640 1 -- 172.21.15.107:0/3883172575 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f573c061050 con 0x5639fd5ff320 2024-05-09T08:48:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.057+0000 7f5755f98740 10 client.28271 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.057+0000 7f5755f98740 1 -- 172.21.15.107:0/3883172575 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5639fd6020b0 con 0x5639fd5ff320 2024-05-09T08:48:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.058+0000 7f572bfff640 1 -- 172.21.15.107:0/3883172575 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f573c0043c0 con 0x5639fd5ff320 2024-05-09T08:48:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.058+0000 7f572bfff640 1 -- 172.21.15.107:0/3883172575 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f573c064020 con 0x5639fd5ff320 2024-05-09T08:48:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.058+0000 7f5755f98740 10 client.28271 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.058+0000 7f5755f98740 10 client.28271 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:48:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.058+0000 7f5755f98740 20 client.28271 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.058+0000 7f5755f98740 1 --2- 172.21.15.107:0/3883172575 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5639fd620980 0x5639fd622e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.058+0000 7f5755f98740 4 client.28271 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:48:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.058+0000 7f5755f98740 1 -- 172.21.15.107:0/3883172575 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639fd567370 con 0x5639fd620980 2024-05-09T08:48:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.060+0000 7f5750f45640 1 --2- 172.21.15.107:0/3883172575 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5639fd620980 0x5639fd622e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.064+0000 7f5750f45640 1 --2- 172.21.15.107:0/3883172575 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5639fd620980 0x5639fd622e40 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=72ab6873dc014b66 server_cookie=d5698ee1b4a45eba in_seq=0 out_seq=0 2024-05-09T08:48:11.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.064+0000 7f572bfff640 10 client.28271 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:48:11.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.070+0000 7f572bfff640 1 -- 172.21.15.107:0/3883172575 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5639fd567370 con 0x5639fd620980 2024-05-09T08:48:11.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.070+0000 7f572bfff640 10 client.28271 handle_command_reply: tid=0 2024-05-09T08:48:11.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.070+0000 7f5755f98740 1 client.28271 shutdown 2024-05-09T08:48:11.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.074+0000 7f5755f98740 1 -- 172.21.15.107:0/3883172575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5ff320 msgr2=0x5639fd5fcd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.074+0000 7f5755f98740 1 --2- 172.21.15.107:0/3883172575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5ff320 0x5639fd5fcd90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f573c02a510 tx=0x7f573c004570 comp rx=0 tx=0).stop 2024-05-09T08:48:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.074+0000 7f5755f98740 1 -- 172.21.15.107:0/3883172575 shutdown_connections 2024-05-09T08:48:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.074+0000 7f5755f98740 1 --2- 172.21.15.107:0/3883172575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5ff320 0x5639fd5fcd90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.074+0000 7f5755f98740 1 --2- 172.21.15.107:0/3883172575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4c50 0x5639fd5fc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.074+0000 7f5755f98740 1 --2- 172.21.15.107:0/3883172575 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5639fd620980 0x5639fd622e40 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.074+0000 7f5755f98740 1 --2- 172.21.15.107:0/3883172575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5f4330 0x5639fd5fc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.074+0000 7f5755f98740 1 -- 172.21.15.107:0/3883172575 >> 172.21.15.107:0/3883172575 conn(0x5639fd5f0650 msgr2=0x5639fd5f6350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:11.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.075+0000 7f572bfff640 0 client.28271 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:48:11.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.079+0000 7f5755f98740 1 -- 172.21.15.107:0/3883172575 shutdown_connections 2024-05-09T08:48:11.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.081+0000 7f5755f98740 1 -- 172.21.15.107:0/3883172575 wait complete. 2024-05-09T08:48:11.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.082+0000 7f5755f98740 20 client.28271 trim_cache size 0 max 16384 2024-05-09T08:48:11.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:10 smithi157 ceph-mon[33614]: pgmap v465: 129 pgs: 18 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 60 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 120 KiB/s rd, 190 KiB/s wr, 35 op/s 2024-05-09T08:48:11.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:10 smithi157 ceph-mon[33614]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:48:11.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:10 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/4178475229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:11.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.108+0000 7f5755f98740 1 Processor -- start 2024-05-09T08:48:11.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.110+0000 7f5755f98740 1 -- start start 2024-05-09T08:48:11.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.110+0000 7f5755f98740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4330 0x5639fd60afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.110+0000 7f5755f98740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4c50 0x5639fd5eff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.110+0000 7f5755f98740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5ff320 0x5639fd5fe140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.110+0000 7f5755f98740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f572c012940 con 0x5639fd5f4c50 2024-05-09T08:48:11.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.110+0000 7f5755f98740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f572c0101b0 con 0x5639fd5f4330 2024-05-09T08:48:11.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.110+0000 7f5755f98740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f572c00e020 con 0x5639fd5ff320 2024-05-09T08:48:11.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.113+0000 7f5750f45640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4330 0x5639fd60afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.113+0000 7f5750f45640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4330 0x5639fd60afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:40432/0 (socket says 172.21.15.107:40432) 2024-05-09T08:48:11.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.113+0000 7f5750f45640 1 -- 172.21.15.107:0/2920863989 learned_addr learned my addr 172.21.15.107:0/2920863989 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:11.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.114+0000 7f574bfff640 1 --2- 172.21.15.107:0/2920863989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4c50 0x5639fd5eff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.115+0000 7f5751746640 1 --2- 172.21.15.107:0/2920863989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5ff320 0x5639fd5fe140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.115+0000 7f574bfff640 1 -- 172.21.15.107:0/2920863989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5ff320 msgr2=0x5639fd5fe140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:11.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.115+0000 7f574bfff640 1 --2- 172.21.15.107:0/2920863989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5ff320 0x5639fd5fe140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.115+0000 7f574bfff640 1 -- 172.21.15.107:0/2920863989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4330 msgr2=0x5639fd60afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:11.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.115+0000 7f574bfff640 1 --2- 172.21.15.107:0/2920863989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4330 0x5639fd60afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.115+0000 7f574bfff640 1 -- 172.21.15.107:0/2920863989 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639fd5e1d20 con 0x5639fd5f4c50 2024-05-09T08:48:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.116+0000 7f574bfff640 1 --2- 172.21.15.107:0/2920863989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4c50 0x5639fd5eff30 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f57440cbf80 tx=0x7f574400e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0e3cfff9570fab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.116+0000 7f57427fc640 1 -- 172.21.15.107:0/2920863989 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574409f100 con 0x5639fd5f4c50 2024-05-09T08:48:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.116+0000 7f57427fc640 1 -- 172.21.15.107:0/2920863989 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57440a39c0 con 0x5639fd5f4c50 2024-05-09T08:48:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.117+0000 7f5755f98740 1 -- 172.21.15.107:0/2920863989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4c50 msgr2=0x5639fd5eff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.117+0000 7f5755f98740 1 --2- 172.21.15.107:0/2920863989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4c50 0x5639fd5eff30 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f57440cbf80 tx=0x7f574400e040 comp rx=0 tx=0).stop 2024-05-09T08:48:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.118+0000 7f5755f98740 1 -- 172.21.15.107:0/2920863989 shutdown_connections 2024-05-09T08:48:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.118+0000 7f5755f98740 1 --2- 172.21.15.107:0/2920863989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5ff320 0x5639fd5fe140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.118+0000 7f5755f98740 1 --2- 172.21.15.107:0/2920863989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5f4c50 0x5639fd5eff30 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.118+0000 7f5755f98740 1 --2- 172.21.15.107:0/2920863989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4330 0x5639fd60afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.118+0000 7f5755f98740 1 -- 172.21.15.107:0/2920863989 >> 172.21.15.107:0/2920863989 conn(0x5639fd5f0650 msgr2=0x5639fd5fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.120+0000 7f5755f98740 1 -- 172.21.15.107:0/2920863989 shutdown_connections 2024-05-09T08:48:11.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.120+0000 7f5755f98740 1 -- 172.21.15.107:0/2920863989 wait complete. 2024-05-09T08:48:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.120+0000 7f5755f98740 1 -- start start 2024-05-09T08:48:11.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.120+0000 7f5755f98740 1 Processor -- start 2024-05-09T08:48:11.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:10 smithi110 ceph-mon[33622]: pgmap v465: 129 pgs: 18 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 60 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 120 KiB/s rd, 190 KiB/s wr, 35 op/s 2024-05-09T08:48:11.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:10 smithi110 ceph-mon[33622]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:48:11.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:10 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/4178475229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:11.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.121+0000 7f5755f98740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4330 0x5639fd607740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.121+0000 7f5755f98740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5f4c50 0x5639fd5f8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.121+0000 7f5755f98740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5ff320 0x5639fd5f90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.121+0000 7f5755f98740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f572c00e670 con 0x5639fd5ff320 2024-05-09T08:48:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.121+0000 7f5755f98740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f572c00fae0 con 0x5639fd5f4330 2024-05-09T08:48:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.121+0000 7f5755f98740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f572c00ed40 con 0x5639fd5f4c50 2024-05-09T08:48:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.121+0000 7f5755f98740 10 client.0 _ll_register_callbacks cb 0x7f5744014bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.122+0000 7f5750f45640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4330 0x5639fd607740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.122+0000 7f5750f45640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4330 0x5639fd607740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:40438/0 (socket says 172.21.15.107:40438) 2024-05-09T08:48:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.122+0000 7f5750f45640 1 -- 172.21.15.107:0/2599924428 learned_addr learned my addr 172.21.15.107:0/2599924428 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:11.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.122+0000 7f574bfff640 1 --2- 172.21.15.107:0/2599924428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5f4c50 0x5639fd5f8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.124+0000 7f5750f45640 1 -- 172.21.15.107:0/2599924428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5f4c50 msgr2=0x5639fd5f8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.124+0000 7f5750f45640 1 --2- 172.21.15.107:0/2599924428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5f4c50 0x5639fd5f8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.124+0000 7f5750f45640 1 -- 172.21.15.107:0/2599924428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5ff320 msgr2=0x5639fd5f90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.124+0000 7f5750f45640 1 --2- 172.21.15.107:0/2599924428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5ff320 0x5639fd5f90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.124+0000 7f5750f45640 1 -- 172.21.15.107:0/2599924428 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639fd5f9750 con 0x5639fd5f4330 2024-05-09T08:48:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.125+0000 7f5750f45640 1 --2- 172.21.15.107:0/2599924428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4330 0x5639fd607740 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f573400afa0 tx=0x7f573400afd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70753e8000e14c99 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.125+0000 7f572bfff640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-05-09T08:48:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.125+0000 7f572bfff640 1 -- 172.21.15.107:0/2599924428 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573401be60 con 0x5639fd5f4330 2024-05-09T08:48:11.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.125+0000 7f572bfff640 1 -- 172.21.15.107:0/2599924428 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7f5734004020 con 0x5639fd5f4330 2024-05-09T08:48:11.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.126+0000 7f5755f98740 1 -- 172.21.15.107:0/2599924428 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57440a39c0 con 0x5639fd5f4330 2024-05-09T08:48:11.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.126+0000 7f572bfff640 1 -- 172.21.15.107:0/2599924428 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f573401b920 con 0x5639fd5f4330 2024-05-09T08:48:11.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.127+0000 7f5755f98740 10 client.28277 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:11.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.127+0000 7f5755f98740 1 -- 172.21.15.107:0/2599924428 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5639fd686e70 con 0x5639fd5f4330 2024-05-09T08:48:11.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.127+0000 7f572bfff640 1 -- 172.21.15.107:0/2599924428 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57440a39c0 con 0x5639fd5f4330 2024-05-09T08:48:11.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.128+0000 7f572bfff640 1 -- 172.21.15.107:0/2599924428 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f5734022070 con 0x5639fd5f4330 2024-05-09T08:48:11.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.130+0000 7f5755f98740 10 client.28277 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:11.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.130+0000 7f5755f98740 10 client.28277 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:48:11.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.130+0000 7f5755f98740 20 client.28277 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:11.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.130+0000 7f5755f98740 1 --2- 172.21.15.107:0/2599924428 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5639fd687540 0x5639fd689a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.130+0000 7f5755f98740 4 client.28277 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:48:11.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.130+0000 7f5755f98740 1 -- 172.21.15.107:0/2599924428 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5639fd5616c0 con 0x5639fd687540 2024-05-09T08:48:11.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.130+0000 7f574bfff640 1 --2- 172.21.15.107:0/2599924428 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5639fd687540 0x5639fd689a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.135+0000 7f574bfff640 1 --2- 172.21.15.107:0/2599924428 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5639fd687540 0x5639fd689a00 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8dc539e267a3fff0 server_cookie=7adfbe1402f2f410 in_seq=0 out_seq=0 2024-05-09T08:48:11.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.135+0000 7f572bfff640 10 client.28277 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:48:11.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.136+0000 7f572bfff640 1 -- 172.21.15.107:0/2599924428 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5639fd5616c0 con 0x5639fd687540 2024-05-09T08:48:11.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.136+0000 7f572bfff640 10 client.28277 handle_command_reply: tid=0 2024-05-09T08:48:11.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.138+0000 7f5755f98740 1 client.28277 shutdown 2024-05-09T08:48:11.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.144+0000 7f5755f98740 1 -- 172.21.15.107:0/2599924428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4330 msgr2=0x5639fd607740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.144+0000 7f5755f98740 1 --2- 172.21.15.107:0/2599924428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4330 0x5639fd607740 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f573400afa0 tx=0x7f573400afd0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.147+0000 7f5755f98740 1 -- 172.21.15.107:0/2599924428 shutdown_connections 2024-05-09T08:48:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.147+0000 7f5755f98740 1 --2- 172.21.15.107:0/2599924428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639fd5ff320 0x5639fd5f90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.147+0000 7f5755f98740 1 --2- 172.21.15.107:0/2599924428 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x5639fd687540 0x5639fd689a00 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.147+0000 7f5755f98740 1 --2- 172.21.15.107:0/2599924428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5639fd5f4c50 0x5639fd5f8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.147+0000 7f5755f98740 1 --2- 172.21.15.107:0/2599924428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5639fd5f4330 0x5639fd607740 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.147+0000 7f5755f98740 1 -- 172.21.15.107:0/2599924428 >> 172.21.15.107:0/2599924428 conn(0x5639fd5f0650 msgr2=0x5639fd5fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.147+0000 7f572bfff640 0 client.28277 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:48:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.148+0000 7f5755f98740 1 -- 172.21.15.107:0/2599924428 shutdown_connections 2024-05-09T08:48:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.148+0000 7f5755f98740 1 -- 172.21.15.107:0/2599924428 wait complete. 2024-05-09T08:48:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.148+0000 7f5755f98740 20 client.28277 trim_cache size 0 max 16384 2024-05-09T08:48:11.150 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:48:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.153+0000 7f57531a9640 1 -- 172.21.15.107:0/2275212157 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f57380a2f10 msgr2=0x7f57380a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.153+0000 7f57531a9640 1 --2- 172.21.15.107:0/2275212157 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f57380a2f10 0x7f57380a53d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f573c045000 tx=0x7f573c047000 comp rx=0 tx=0).stop 2024-05-09T08:48:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.153+0000 7f57531a9640 1 -- 172.21.15.107:0/2275212157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f574c100290 msgr2=0x7f574c0734b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.153+0000 7f57531a9640 1 --2- 172.21.15.107:0/2275212157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f574c100290 0x7f574c0734b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f574400b9e0 tx=0x7f574400bea0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.153+0000 7f57531a9640 1 -- 172.21.15.107:0/2275212157 shutdown_connections 2024-05-09T08:48:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.153+0000 7f57531a9640 1 --2- 172.21.15.107:0/2275212157 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f57380a2f10 0x7f57380a53d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.153+0000 7f57531a9640 1 --2- 172.21.15.107:0/2275212157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f574c112690 0x7f574c06c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.153+0000 7f57531a9640 1 --2- 172.21.15.107:0/2275212157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f574c100290 0x7f574c0734b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.153+0000 7f57531a9640 1 --2- 172.21.15.107:0/2275212157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f574c0ff970 0x7f574c072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.153+0000 7f57531a9640 1 -- 172.21.15.107:0/2275212157 >> 172.21.15.107:0/2275212157 conn(0x7f574c076890 msgr2=0x7f574c111010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.154+0000 7f57531a9640 1 -- 172.21.15.107:0/2275212157 shutdown_connections 2024-05-09T08:48:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.154+0000 7f57531a9640 1 -- 172.21.15.107:0/2275212157 wait complete. 2024-05-09T08:48:11.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:48:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.374+0000 7f192d002640 1 -- 172.21.15.107:0/482787627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1920007f10 msgr2=0x7f1920008390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.374+0000 7f192d002640 1 --2- 172.21.15.107:0/482787627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1920007f10 0x7f1920008390 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f1914009900 tx=0x7f191402a420 comp rx=0 tx=0).stop 2024-05-09T08:48:11.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.381+0000 7f192d002640 1 -- 172.21.15.107:0/482787627 shutdown_connections 2024-05-09T08:48:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.381+0000 7f192d002640 1 --2- 172.21.15.107:0/482787627 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19200a6b80 0x7f19200a8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.381+0000 7f192d002640 1 --2- 172.21.15.107:0/482787627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1920007f10 0x7f1920008390 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.381+0000 7f192d002640 1 --2- 172.21.15.107:0/482787627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1920009640 0x7f19200079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.381+0000 7f192d002640 1 -- 172.21.15.107:0/482787627 >> 172.21.15.107:0/482787627 conn(0x7f192000b8d0 msgr2=0x7f192000bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:11.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.384+0000 7f192d002640 1 -- 172.21.15.107:0/482787627 shutdown_connections 2024-05-09T08:48:11.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.387+0000 7f192d002640 1 -- 172.21.15.107:0/482787627 wait complete. 2024-05-09T08:48:11.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.388+0000 7f192d002640 1 Processor -- start 2024-05-09T08:48:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.391+0000 7f192d002640 1 -- start start 2024-05-09T08:48:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.391+0000 7f192d002640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1920007f10 0x7f19200a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.391+0000 7f192d002640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1920009640 0x7f192009e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.391+0000 7f192d002640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19200a6b80 0x7f192009e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.391+0000 7f192d002640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f19200ac6e0 con 0x7f1920009640 2024-05-09T08:48:11.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.391+0000 7f192d002640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f19200ac560 con 0x7f1920007f10 2024-05-09T08:48:11.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.391+0000 7f192d002640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f19200ac860 con 0x7f19200a6b80 2024-05-09T08:48:11.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.392+0000 7f192c801640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19200a6b80 0x7f192009e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.393+0000 7f1927fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1920007f10 0x7f19200a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.393+0000 7f1927fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1920007f10 0x7f19200a52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:40442/0 (socket says 172.21.15.107:40442) 2024-05-09T08:48:11.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.393+0000 7f1927fff640 1 -- 172.21.15.107:0/4165903282 learned_addr learned my addr 172.21.15.107:0/4165903282 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:11.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.394+0000 7f1927fff640 1 -- 172.21.15.107:0/4165903282 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19200a6b80 msgr2=0x7f192009e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.394+0000 7f1927fff640 1 --2- 172.21.15.107:0/4165903282 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19200a6b80 0x7f192009e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.394+0000 7f1927fff640 1 -- 172.21.15.107:0/4165903282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1920009640 msgr2=0x7f192009e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.394+0000 7f19277fe640 1 --2- 172.21.15.107:0/4165903282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1920009640 0x7f192009e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.397+0000 7f1927fff640 1 --2- 172.21.15.107:0/4165903282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1920009640 0x7f192009e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.397+0000 7f1927fff640 1 -- 172.21.15.107:0/4165903282 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f192009ef50 con 0x7f1920007f10 2024-05-09T08:48:11.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.398+0000 7f19277fe640 1 --2- 172.21.15.107:0/4165903282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1920009640 0x7f192009e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:11.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.402+0000 7f1927fff640 1 --2- 172.21.15.107:0/4165903282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1920007f10 0x7f19200a52a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1928050ef0 tx=0x7f1928078890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:11.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.404+0000 7f19257fa640 1 -- 172.21.15.107:0/4165903282 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192807d790 con 0x7f1920007f10 2024-05-09T08:48:11.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.405+0000 7f19257fa640 1 -- 172.21.15.107:0/4165903282 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1928083070 con 0x7f1920007f10 2024-05-09T08:48:11.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.405+0000 7f19257fa640 1 -- 172.21.15.107:0/4165903282 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192807eda0 con 0x7f1920007f10 2024-05-09T08:48:11.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.406+0000 7f192d002640 1 -- 172.21.15.107:0/4165903282 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19200a2dc0 con 0x7f1920007f10 2024-05-09T08:48:11.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.409+0000 7f192d002640 1 -- 172.21.15.107:0/4165903282 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19200a32c0 con 0x7f1920007f10 2024-05-09T08:48:11.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.413+0000 7f1930f71740 1 Processor -- start 2024-05-09T08:48:11.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.414+0000 7f19257fa640 1 -- 172.21.15.107:0/4165903282 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f192807d930 con 0x7f1920007f10 2024-05-09T08:48:11.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.416+0000 7f19257fa640 1 --2- 172.21.15.107:0/4165903282 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f19180a2f00 0x7f19180a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.416+0000 7f19257fa640 1 -- 172.21.15.107:0/4165903282 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7f192813acd0 con 0x7f1920007f10 2024-05-09T08:48:11.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.418+0000 7f192c801640 1 --2- 172.21.15.107:0/4165903282 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f19180a2f00 0x7f19180a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.419+0000 7f192c801640 1 --2- 172.21.15.107:0/4165903282 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f19180a2f00 0x7f19180a53c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f191c006fd0 tx=0x7f191c008040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:11.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.421+0000 7f1930f71740 1 -- start start 2024-05-09T08:48:11.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.421+0000 7f1930f71740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038ce4f0 0x55eb038ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.421+0000 7f1930f71740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038cee10 0x55eb038d8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.421+0000 7f1930f71740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038d94e0 0x55eb038db8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.421+0000 7f1930f71740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55eb03833c40 con 0x55eb038ce4f0 2024-05-09T08:48:11.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.421+0000 7f1930f71740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55eb037db580 con 0x55eb038cee10 2024-05-09T08:48:11.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.421+0000 7f1930f71740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55eb0382fc00 con 0x55eb038d94e0 2024-05-09T08:48:11.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.421+0000 7f1927fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038ce4f0 0x55eb038ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.421+0000 7f192c801640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038d94e0 0x55eb038db8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.421+0000 7f1927fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038ce4f0 0x55eb038ce8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41576/0 (socket says 172.21.15.107:41576) 2024-05-09T08:48:11.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.421+0000 7f1927fff640 1 -- 172.21.15.107:0/1384420676 learned_addr learned my addr 172.21.15.107:0/1384420676 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:11.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.422+0000 7f19277fe640 1 --2- 172.21.15.107:0/1384420676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038cee10 0x55eb038d8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.422+0000 7f1927fff640 1 -- 172.21.15.107:0/1384420676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038d94e0 msgr2=0x55eb038db8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:11.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.422+0000 7f1927fff640 1 --2- 172.21.15.107:0/1384420676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038d94e0 0x55eb038db8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.422+0000 7f1927fff640 1 -- 172.21.15.107:0/1384420676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038cee10 msgr2=0x55eb038d8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:11.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.422+0000 7f1927fff640 1 --2- 172.21.15.107:0/1384420676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038cee10 0x55eb038d8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.422+0000 7f1927fff640 1 -- 172.21.15.107:0/1384420676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb035473f0 con 0x55eb038ce4f0 2024-05-09T08:48:11.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.423+0000 7f1927fff640 1 --2- 172.21.15.107:0/1384420676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038ce4f0 0x55eb038ce8d0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f19281032f0 tx=0x7f1928103260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad94ee0462dbda0b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:11.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.423+0000 7f19127fc640 1 -- 172.21.15.107:0/1384420676 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1928106e90 con 0x55eb038ce4f0 2024-05-09T08:48:11.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.423+0000 7f19127fc640 1 -- 172.21.15.107:0/1384420676 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f192810b240 con 0x55eb038ce4f0 2024-05-09T08:48:11.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.424+0000 7f1930f71740 1 -- 172.21.15.107:0/1384420676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038ce4f0 msgr2=0x55eb038ce8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.424+0000 7f1930f71740 1 --2- 172.21.15.107:0/1384420676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038ce4f0 0x55eb038ce8d0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f19281032f0 tx=0x7f1928103260 comp rx=0 tx=0).stop 2024-05-09T08:48:11.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.430+0000 7f1930f71740 1 -- 172.21.15.107:0/1384420676 shutdown_connections 2024-05-09T08:48:11.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.430+0000 7f1930f71740 1 --2- 172.21.15.107:0/1384420676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038d94e0 0x55eb038db8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.430+0000 7f1930f71740 1 --2- 172.21.15.107:0/1384420676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038cee10 0x55eb038d8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.430+0000 7f1930f71740 1 --2- 172.21.15.107:0/1384420676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038ce4f0 0x55eb038ce8d0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.430+0000 7f1930f71740 1 -- 172.21.15.107:0/1384420676 >> 172.21.15.107:0/1384420676 conn(0x55eb038ca810 msgr2=0x55eb038cabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:11.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.435+0000 7f1930f71740 1 -- 172.21.15.107:0/1384420676 shutdown_connections 2024-05-09T08:48:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.435+0000 7f1930f71740 1 -- 172.21.15.107:0/1384420676 wait complete. 2024-05-09T08:48:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.436+0000 7f1930f71740 1 -- start start 2024-05-09T08:48:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.436+0000 7f1930f71740 1 Processor -- start 2024-05-09T08:48:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.437+0000 7f1930f71740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038ce4f0 0x55eb038d6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.437+0000 7f1930f71740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038d68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.437+0000 7f1930f71740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038d94e0 0x55eb038d6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.437+0000 7f1930f71740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55eb038ed330 con 0x55eb038cee10 2024-05-09T08:48:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.437+0000 7f1930f71740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55eb037eec40 con 0x55eb038d94e0 2024-05-09T08:48:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.437+0000 7f1930f71740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55eb038ed4b0 con 0x55eb038ce4f0 2024-05-09T08:48:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.437+0000 7f1930f71740 10 client.0 _ll_register_callbacks cb 0x55eb03546be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.437+0000 7f19277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038d68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.437+0000 7f19277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038d68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41590/0 (socket says 172.21.15.107:41590) 2024-05-09T08:48:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.437+0000 7f19277fe640 1 -- 172.21.15.107:0/91984712 learned_addr learned my addr 172.21.15.107:0/91984712 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.437+0000 7f1927fff640 1 --2- 172.21.15.107:0/91984712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038ce4f0 0x55eb038d6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.438+0000 7f192c801640 1 --2- 172.21.15.107:0/91984712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038d94e0 0x55eb038d6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.438+0000 7f19277fe640 1 -- 172.21.15.107:0/91984712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038ce4f0 msgr2=0x55eb038d6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.438+0000 7f19277fe640 1 --2- 172.21.15.107:0/91984712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038ce4f0 0x55eb038d6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.438+0000 7f19277fe640 1 -- 172.21.15.107:0/91984712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038d94e0 msgr2=0x55eb038d6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.438+0000 7f19277fe640 1 --2- 172.21.15.107:0/91984712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038d94e0 0x55eb038d6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.438+0000 7f19277fe640 1 -- 172.21.15.107:0/91984712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb036117d0 con 0x55eb038cee10 2024-05-09T08:48:11.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.439+0000 7f19277fe640 1 --2- 172.21.15.107:0/91984712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038d68d0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f1914009850 tx=0x7f19140044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b25d3f12463d76ff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:11.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.439+0000 7f19077fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:48:11.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.440+0000 7f19077fe640 1 -- 172.21.15.107:0/91984712 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191403f070 con 0x55eb038cee10 2024-05-09T08:48:11.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.440+0000 7f1930f71740 1 -- 172.21.15.107:0/91984712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f192810b240 con 0x55eb038cee10 2024-05-09T08:48:11.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.440+0000 7f19077fe640 1 -- 172.21.15.107:0/91984712 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7f1914005510 con 0x55eb038cee10 2024-05-09T08:48:11.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.440+0000 7f19077fe640 1 -- 172.21.15.107:0/91984712 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f191404b050 con 0x55eb038cee10 2024-05-09T08:48:11.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.442+0000 7f1930f71740 10 client.19353 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:11.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.442+0000 7f1930f71740 1 -- 172.21.15.107:0/91984712 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55eb038f9560 con 0x55eb038cee10 2024-05-09T08:48:11.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.445+0000 7f19077fe640 1 -- 172.21.15.107:0/91984712 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f192810b240 con 0x55eb038cee10 2024-05-09T08:48:11.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.445+0000 7f19077fe640 1 -- 172.21.15.107:0/91984712 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f1914003950 con 0x55eb038cee10 2024-05-09T08:48:11.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.447+0000 7f1930f71740 10 client.19353 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:11.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.447+0000 7f1930f71740 10 client.19353 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:48:11.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.447+0000 7f1930f71740 20 client.19353 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:11.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.447+0000 7f1930f71740 1 --2- 172.21.15.107:0/91984712 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55eb038fa970 0x55eb038fcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.447+0000 7f1930f71740 4 client.19353 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:48:11.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.447+0000 7f1930f71740 1 -- 172.21.15.107:0/91984712 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb03841370 con 0x55eb038fa970 2024-05-09T08:48:11.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.451+0000 7f1927fff640 1 --2- 172.21.15.107:0/91984712 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55eb038fa970 0x55eb038fcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.465+0000 7f1927fff640 1 --2- 172.21.15.107:0/91984712 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55eb038fa970 0x55eb038fcd60 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e67c30e42cedc79 server_cookie=bffa1bfbfaa0dcf4 in_seq=0 out_seq=0 2024-05-09T08:48:11.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.465+0000 7f19077fe640 10 client.19353 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:11.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.469+0000 7f19077fe640 1 -- 172.21.15.107:0/91984712 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb03841370 con 0x55eb038fa970 2024-05-09T08:48:11.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.469+0000 7f19077fe640 10 client.19353 handle_command_reply: tid=0 2024-05-09T08:48:11.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.472+0000 7f1930f71740 1 client.19353 shutdown 2024-05-09T08:48:11.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.473+0000 7f1930f71740 1 -- 172.21.15.107:0/91984712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 msgr2=0x55eb038d68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.473+0000 7f1930f71740 1 --2- 172.21.15.107:0/91984712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038d68d0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f1914009850 tx=0x7f19140044f0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.473+0000 7f1930f71740 1 -- 172.21.15.107:0/91984712 shutdown_connections 2024-05-09T08:48:11.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.473+0000 7f1930f71740 1 --2- 172.21.15.107:0/91984712 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55eb038fa970 0x55eb038fcd60 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.473+0000 7f1930f71740 1 --2- 172.21.15.107:0/91984712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038d94e0 0x55eb038d6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.473+0000 7f1930f71740 1 --2- 172.21.15.107:0/91984712 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038d68d0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.473+0000 7f1930f71740 1 --2- 172.21.15.107:0/91984712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038ce4f0 0x55eb038d6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.473+0000 7f1930f71740 1 -- 172.21.15.107:0/91984712 >> 172.21.15.107:0/91984712 conn(0x55eb038ca810 msgr2=0x55eb038d0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:11.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.474+0000 7f19077fe640 0 client.19353 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:11.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.474+0000 7f1930f71740 1 -- 172.21.15.107:0/91984712 shutdown_connections 2024-05-09T08:48:11.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.475+0000 7f1930f71740 1 -- 172.21.15.107:0/91984712 wait complete. 2024-05-09T08:48:11.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.475+0000 7f1930f71740 20 client.19353 trim_cache size 0 max 16384 2024-05-09T08:48:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.500+0000 7f1930f71740 1 Processor -- start 2024-05-09T08:48:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.505+0000 7f1930f71740 1 -- start start 2024-05-09T08:48:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.505+0000 7f1930f71740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038ce4f0 0x55eb038f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.505+0000 7f1930f71740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038d0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.505+0000 7f1930f71740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038d94e0 0x55eb038d54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.505+0000 7f1930f71740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1920154bd0 con 0x55eb038cee10 2024-05-09T08:48:11.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.505+0000 7f1930f71740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1920152440 con 0x55eb038d94e0 2024-05-09T08:48:11.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.505+0000 7f1930f71740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f19201502b0 con 0x55eb038ce4f0 2024-05-09T08:48:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.508+0000 7f1927fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038ce4f0 0x55eb038f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.508+0000 7f1927fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038ce4f0 0x55eb038f62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:45508/0 (socket says 172.21.15.107:45508) 2024-05-09T08:48:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.508+0000 7f1927fff640 1 -- 172.21.15.107:0/3036854138 learned_addr learned my addr 172.21.15.107:0/3036854138 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.508+0000 7f19277fe640 1 --2- 172.21.15.107:0/3036854138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038d0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.508+0000 7f192c801640 1 --2- 172.21.15.107:0/3036854138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038d94e0 0x55eb038d54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.511+0000 7f19277fe640 1 -- 172.21.15.107:0/3036854138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038ce4f0 msgr2=0x55eb038f62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:11.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.511+0000 7f19277fe640 1 --2- 172.21.15.107:0/3036854138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038ce4f0 0x55eb038f62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.511+0000 7f19277fe640 1 -- 172.21.15.107:0/3036854138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038d94e0 msgr2=0x55eb038d54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:11.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.511+0000 7f19277fe640 1 --2- 172.21.15.107:0/3036854138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038d94e0 0x55eb038d54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.511+0000 7f19277fe640 1 -- 172.21.15.107:0/3036854138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb038cf9b0 con 0x55eb038cee10 2024-05-09T08:48:11.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.515+0000 7f19277fe640 1 --2- 172.21.15.107:0/3036854138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038d0c50 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f1914003d50 tx=0x7f1914009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8276596a46310dc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:11.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.515+0000 7f1911ffb640 1 -- 172.21.15.107:0/3036854138 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1914054070 con 0x55eb038cee10 2024-05-09T08:48:11.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.516+0000 7f1911ffb640 1 -- 172.21.15.107:0/3036854138 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19140035f0 con 0x55eb038cee10 2024-05-09T08:48:11.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.516+0000 7f1911ffb640 1 -- 172.21.15.107:0/3036854138 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191403f070 con 0x55eb038cee10 2024-05-09T08:48:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.519+0000 7f1930f71740 1 -- 172.21.15.107:0/3036854138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 msgr2=0x55eb038d0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.519+0000 7f1930f71740 1 --2- 172.21.15.107:0/3036854138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038d0c50 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f1914003d50 tx=0x7f1914009260 comp rx=0 tx=0).stop 2024-05-09T08:48:11.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.525+0000 7f1930f71740 1 -- 172.21.15.107:0/3036854138 shutdown_connections 2024-05-09T08:48:11.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.525+0000 7f1930f71740 1 --2- 172.21.15.107:0/3036854138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038d94e0 0x55eb038d54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.525+0000 7f1930f71740 1 --2- 172.21.15.107:0/3036854138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038d0c50 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.525+0000 7f1930f71740 1 --2- 172.21.15.107:0/3036854138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038ce4f0 0x55eb038f62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.525+0000 7f1930f71740 1 -- 172.21.15.107:0/3036854138 >> 172.21.15.107:0/3036854138 conn(0x55eb038ca810 msgr2=0x55eb038d7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:11.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.527+0000 7f1930f71740 1 -- 172.21.15.107:0/3036854138 shutdown_connections 2024-05-09T08:48:11.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.527+0000 7f1930f71740 1 -- 172.21.15.107:0/3036854138 wait complete. 2024-05-09T08:48:11.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.527+0000 7f1930f71740 1 -- start start 2024-05-09T08:48:11.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.528+0000 7f1930f71740 1 Processor -- start 2024-05-09T08:48:11.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.530+0000 7f1930f71740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038ce4f0 0x55eb038f27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.530+0000 7f1930f71740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.530+0000 7f1930f71740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038d94e0 0x55eb038edf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.530+0000 7f1930f71740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1920150900 con 0x55eb038cee10 2024-05-09T08:48:11.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.530+0000 7f1930f71740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1920151d70 con 0x55eb038ce4f0 2024-05-09T08:48:11.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.530+0000 7f1930f71740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1920150fd0 con 0x55eb038d94e0 2024-05-09T08:48:11.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.530+0000 7f1930f71740 10 client.0 _ll_register_callbacks cb 0x7f1914009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:11.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.531+0000 7f192c801640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038d94e0 0x55eb038edf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.532+0000 7f19277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.532+0000 7f19277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038f2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41616/0 (socket says 172.21.15.107:41616) 2024-05-09T08:48:11.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.532+0000 7f19277fe640 1 -- 172.21.15.107:0/2964361983 learned_addr learned my addr 172.21.15.107:0/2964361983 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:11.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.532+0000 7f19277fe640 1 -- 172.21.15.107:0/2964361983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038d94e0 msgr2=0x55eb038edf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:11.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.532+0000 7f19277fe640 1 --2- 172.21.15.107:0/2964361983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038d94e0 0x55eb038edf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.532+0000 7f19277fe640 1 -- 172.21.15.107:0/2964361983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038ce4f0 msgr2=0x55eb038f27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:11.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.532+0000 7f19277fe640 1 --2- 172.21.15.107:0/2964361983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038ce4f0 0x55eb038f27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.532+0000 7f19277fe640 1 -- 172.21.15.107:0/2964361983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb038ee740 con 0x55eb038cee10 2024-05-09T08:48:11.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.533+0000 7f19277fe640 1 --2- 172.21.15.107:0/2964361983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038f2d10 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f1914009900 tx=0x7f1914031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f28c6213f68bd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.537+0000 7f19077fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:48:11.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.537+0000 7f19077fe640 1 -- 172.21.15.107:0/2964361983 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191405d060 con 0x55eb038cee10 2024-05-09T08:48:11.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.537+0000 7f19077fe640 1 -- 172.21.15.107:0/2964361983 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7f1914005510 con 0x55eb038cee10 2024-05-09T08:48:11.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.537+0000 7f1930f71740 1 -- 172.21.15.107:0/2964361983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19140035f0 con 0x55eb038cee10 2024-05-09T08:48:11.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.541+0000 7f19077fe640 1 -- 172.21.15.107:0/2964361983 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f191403f070 con 0x55eb038cee10 2024-05-09T08:48:11.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.543+0000 7f1930f71740 10 client.19359 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.543+0000 7f1930f71740 1 -- 172.21.15.107:0/2964361983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55eb03960b70 con 0x55eb038cee10 2024-05-09T08:48:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.544+0000 7f19077fe640 1 -- 172.21.15.107:0/2964361983 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19140035f0 con 0x55eb038cee10 2024-05-09T08:48:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.545+0000 7f19077fe640 1 -- 172.21.15.107:0/2964361983 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f1914057030 con 0x55eb038cee10 2024-05-09T08:48:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.548+0000 7f1930f71740 10 client.19359 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.548+0000 7f1930f71740 10 client.19359 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:48:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.548+0000 7f1930f71740 20 client.19359 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.548+0000 7f1930f71740 1 --2- 172.21.15.107:0/2964361983 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55eb03961160 0x55eb03963590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.548+0000 7f1930f71740 4 client.19359 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:48:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.548+0000 7f1930f71740 1 -- 172.21.15.107:0/2964361983 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eb0383b6c0 con 0x55eb03961160 2024-05-09T08:48:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.550+0000 7f1927fff640 1 --2- 172.21.15.107:0/2964361983 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55eb03961160 0x55eb03963590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:11.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.557+0000 7f1927fff640 1 --2- 172.21.15.107:0/2964361983 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55eb03961160 0x55eb03963590 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=958f08aeeccc748a server_cookie=9ae0cbcdcd47bdf5 in_seq=0 out_seq=0 2024-05-09T08:48:11.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.558+0000 7f19077fe640 10 client.19359 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:11.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.560+0000 7f19077fe640 1 -- 172.21.15.107:0/2964361983 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eb0383b6c0 con 0x55eb03961160 2024-05-09T08:48:11.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.560+0000 7f19077fe640 10 client.19359 handle_command_reply: tid=0 2024-05-09T08:48:11.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.562+0000 7f1930f71740 1 client.19359 shutdown 2024-05-09T08:48:11.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.563+0000 7f1930f71740 1 -- 172.21.15.107:0/2964361983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 msgr2=0x55eb038f2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.563+0000 7f1930f71740 1 --2- 172.21.15.107:0/2964361983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038f2d10 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f1914009900 tx=0x7f1914031400 comp rx=0 tx=0).stop 2024-05-09T08:48:11.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.565+0000 7f1930f71740 1 -- 172.21.15.107:0/2964361983 shutdown_connections 2024-05-09T08:48:11.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.565+0000 7f1930f71740 1 --2- 172.21.15.107:0/2964361983 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55eb03961160 0x55eb03963590 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.565+0000 7f1930f71740 1 --2- 172.21.15.107:0/2964361983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eb038d94e0 0x55eb038edf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.565+0000 7f1930f71740 1 --2- 172.21.15.107:0/2964361983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eb038cee10 0x55eb038f2d10 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.565+0000 7f1930f71740 1 --2- 172.21.15.107:0/2964361983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb038ce4f0 0x55eb038f27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.565+0000 7f1930f71740 1 -- 172.21.15.107:0/2964361983 >> 172.21.15.107:0/2964361983 conn(0x55eb038ca810 msgr2=0x55eb038d5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:11.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.565+0000 7f19077fe640 0 client.19359 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:11.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.569+0000 7f1930f71740 1 -- 172.21.15.107:0/2964361983 shutdown_connections 2024-05-09T08:48:11.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.572+0000 7f1930f71740 1 -- 172.21.15.107:0/2964361983 wait complete. 2024-05-09T08:48:11.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.573+0000 7f1930f71740 20 client.19359 trim_cache size 0 max 16384 2024-05-09T08:48:11.574 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:48:11.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.576+0000 7f192d002640 1 -- 172.21.15.107:0/4165903282 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f19180a2f00 msgr2=0x7f19180a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.576+0000 7f192d002640 1 --2- 172.21.15.107:0/4165903282 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f19180a2f00 0x7f19180a53c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f191c006fd0 tx=0x7f191c008040 comp rx=0 tx=0).stop 2024-05-09T08:48:11.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.576+0000 7f192d002640 1 -- 172.21.15.107:0/4165903282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1920007f10 msgr2=0x7f19200a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:11.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.576+0000 7f192d002640 1 --2- 172.21.15.107:0/4165903282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1920007f10 0x7f19200a52a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1928050ef0 tx=0x7f1928078890 comp rx=0 tx=0).stop 2024-05-09T08:48:11.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.576+0000 7f192d002640 1 -- 172.21.15.107:0/4165903282 shutdown_connections 2024-05-09T08:48:11.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.576+0000 7f192d002640 1 --2- 172.21.15.107:0/4165903282 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f19180a2f00 0x7f19180a53c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.576+0000 7f192d002640 1 --2- 172.21.15.107:0/4165903282 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19200a6b80 0x7f192009e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.576+0000 7f192d002640 1 --2- 172.21.15.107:0/4165903282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1920009640 0x7f192009e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:11.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.576+0000 7f192d002640 1 --2- 172.21.15.107:0/4165903282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1920007f10 0x7f19200a52a0 secure :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1928050ef0 tx=0x7f1928078890 comp rx=0 tx=0).stop 2024-05-09T08:48:11.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.576+0000 7f192d002640 1 -- 172.21.15.107:0/4165903282 >> 172.21.15.107:0/4165903282 conn(0x7f192000b8d0 msgr2=0x7f1920094f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:11.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.578+0000 7f192d002640 1 -- 172.21.15.107:0/4165903282 shutdown_connections 2024-05-09T08:48:11.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:11.578+0000 7f192d002640 1 -- 172.21.15.107:0/4165903282 wait complete. 2024-05-09T08:48:12.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:11 smithi107 ceph-mon[30122]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:48:12.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:11 smithi157 ceph-mon[33614]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:48:12.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:11 smithi110 ceph-mon[33622]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T08:48:12.598 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T08:48:12.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:48:12.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T08:48:12.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.788+0000 7f73c2302640 1 -- 172.21.15.107:0/3825446538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73bc10e0b0 msgr2=0x7f73bc10e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:12.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.788+0000 7f73c2302640 1 --2- 172.21.15.107:0/3825446538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73bc10e0b0 0x7f73bc10e530 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f73b0009930 tx=0x7f73b002a460 comp rx=0 tx=0).stop 2024-05-09T08:48:12.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.790+0000 7f73c2302640 1 -- 172.21.15.107:0/3825446538 shutdown_connections 2024-05-09T08:48:12.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.790+0000 7f73c2302640 1 --2- 172.21.15.107:0/3825446538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73bc11b010 0x7f73bc11d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.790+0000 7f73c2302640 1 --2- 172.21.15.107:0/3825446538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73bc10e0b0 0x7f73bc10e530 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.790+0000 7f73c2302640 1 --2- 172.21.15.107:0/3825446538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73bc110080 0x7f73bc10db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.790+0000 7f73c2302640 1 -- 172.21.15.107:0/3825446538 >> 172.21.15.107:0/3825446538 conn(0x7f73bc06c940 msgr2=0x7f73bc06cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:12.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.791+0000 7f73c2302640 1 -- 172.21.15.107:0/3825446538 shutdown_connections 2024-05-09T08:48:12.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.792+0000 7f73c2302640 1 -- 172.21.15.107:0/3825446538 wait complete. 2024-05-09T08:48:12.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.793+0000 7f73c2302640 1 Processor -- start 2024-05-09T08:48:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.795+0000 7f73c2302640 1 -- start start 2024-05-09T08:48:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.795+0000 7f73c2302640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73bc10e0b0 0x7f73bc114480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.796+0000 7f73c2302640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73bc110080 0x7f73bc1149c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.796+0000 7f73c2302640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73bc11b010 0x7f73bc118db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.796+0000 7f73c2302640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc110ca0 con 0x7f73bc110080 2024-05-09T08:48:12.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.796+0000 7f73c2302640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc110b20 con 0x7f73bc10e0b0 2024-05-09T08:48:12.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.796+0000 7f73c2302640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc110e20 con 0x7f73bc11b010 2024-05-09T08:48:12.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.797+0000 7f73c089f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73bc11b010 0x7f73bc118db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.797+0000 7f73bbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73bc10e0b0 0x7f73bc114480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.798+0000 7f73c089f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73bc11b010 0x7f73bc118db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:45532/0 (socket says 172.21.15.107:45532) 2024-05-09T08:48:12.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.798+0000 7f73c089f640 1 -- 172.21.15.107:0/2834474645 learned_addr learned my addr 172.21.15.107:0/2834474645 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:12.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.801+0000 7f73bb7fe640 1 --2- 172.21.15.107:0/2834474645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73bc110080 0x7f73bc1149c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.801+0000 7f73c089f640 1 -- 172.21.15.107:0/2834474645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73bc10e0b0 msgr2=0x7f73bc114480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:12.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.801+0000 7f73c089f640 1 --2- 172.21.15.107:0/2834474645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73bc10e0b0 0x7f73bc114480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.801+0000 7f73c089f640 1 -- 172.21.15.107:0/2834474645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73bc110080 msgr2=0x7f73bc1149c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:12.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.801+0000 7f73c089f640 1 --2- 172.21.15.107:0/2834474645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73bc110080 0x7f73bc1149c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.801+0000 7f73c089f640 1 -- 172.21.15.107:0/2834474645 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73bc1194e0 con 0x7f73bc11b010 2024-05-09T08:48:12.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.801+0000 7f73c089f640 1 --2- 172.21.15.107:0/2834474645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73bc11b010 0x7f73bc118db0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f73b400eaa0 tx=0x7f73b400ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:12.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.802+0000 7f73b97fa640 1 -- 172.21.15.107:0/2834474645 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b400cdb0 con 0x7f73bc11b010 2024-05-09T08:48:12.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.802+0000 7f73b97fa640 1 -- 172.21.15.107:0/2834474645 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73b4005020 con 0x7f73bc11b010 2024-05-09T08:48:12.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.802+0000 7f73b97fa640 1 -- 172.21.15.107:0/2834474645 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b40116d0 con 0x7f73bc11b010 2024-05-09T08:48:12.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.804+0000 7f73c2302640 1 -- 172.21.15.107:0/2834474645 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73bc1197d0 con 0x7f73bc11b010 2024-05-09T08:48:12.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.804+0000 7f73c2302640 1 -- 172.21.15.107:0/2834474645 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73bc1bc1a0 con 0x7f73bc11b010 2024-05-09T08:48:12.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.808+0000 7f73c5005740 1 Processor -- start 2024-05-09T08:48:12.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.808+0000 7f73c5005740 1 -- start start 2024-05-09T08:48:12.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.808+0000 7f73c5005740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25de9d0b0 0x55d25de9b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.809+0000 7f73c5005740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25de9b9c0 0x55d25de9be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.809+0000 7f73c5005740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25dea6740 0x55d25dea8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.809+0000 7f73c5005740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d25de00990 con 0x55d25de9b9c0 2024-05-09T08:48:12.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.809+0000 7f73c5005740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d25dda85b0 con 0x55d25dea6740 2024-05-09T08:48:12.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.809+0000 7f73c5005740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d25ddfc950 con 0x55d25de9d0b0 2024-05-09T08:48:12.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.809+0000 7f73bb7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25de9d0b0 0x55d25de9b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.809+0000 7f73bb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25dea6740 0x55d25dea8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.809+0000 7f73bbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25de9b9c0 0x55d25de9be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.809+0000 7f73bbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25de9b9c0 0x55d25de9be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41654/0 (socket says 172.21.15.107:41654) 2024-05-09T08:48:12.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.809+0000 7f73bbfff640 1 -- 172.21.15.107:0/203577668 learned_addr learned my addr 172.21.15.107:0/203577668 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:12.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.812+0000 7f73bbfff640 1 -- 172.21.15.107:0/203577668 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25de9d0b0 msgr2=0x55d25de9b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:12.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.812+0000 7f73bbfff640 1 --2- 172.21.15.107:0/203577668 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25de9d0b0 0x55d25de9b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.812+0000 7f73bbfff640 1 -- 172.21.15.107:0/203577668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25dea6740 msgr2=0x55d25dea8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:12.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.812+0000 7f73bbfff640 1 --2- 172.21.15.107:0/203577668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25dea6740 0x55d25dea8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.812+0000 7f73bbfff640 1 -- 172.21.15.107:0/203577668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d25dbdc7e0 con 0x55d25de9b9c0 2024-05-09T08:48:12.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.812+0000 7f73bb7fe640 1 --2- 172.21.15.107:0/203577668 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25de9d0b0 0x55d25de9b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:12.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.813+0000 7f73b97fa640 1 -- 172.21.15.107:0/2834474645 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f73b4004a40 con 0x7f73bc11b010 2024-05-09T08:48:12.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.814+0000 7f73b97fa640 1 --2- 172.21.15.107:0/2834474645 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f73a40a2f10 0x7f73a40a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.814+0000 7f73bbfff640 1 --2- 172.21.15.107:0/203577668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25de9b9c0 0x55d25de9be40 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00bd10 tx=0x7f73ac0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f6fa1bf963feb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:12.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.814+0000 7f73b97fa640 1 -- 172.21.15.107:0/2834474645 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7f73b40cfba0 con 0x7f73bc11b010 2024-05-09T08:48:12.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.815+0000 7f73bb7fe640 1 --2- 172.21.15.107:0/2834474645 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f73a40a2f10 0x7f73a40a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.815+0000 7f73a27fc640 1 -- 172.21.15.107:0/203577668 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac004030 con 0x55d25de9b9c0 2024-05-09T08:48:12.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.815+0000 7f73a27fc640 1 -- 172.21.15.107:0/203577668 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73ac0035f0 con 0x55d25de9b9c0 2024-05-09T08:48:12.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.816+0000 7f73c5005740 1 -- 172.21.15.107:0/203577668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25de9b9c0 msgr2=0x55d25de9be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:12.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.816+0000 7f73c5005740 1 --2- 172.21.15.107:0/203577668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25de9b9c0 0x55d25de9be40 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00bd10 tx=0x7f73ac0107a0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.816+0000 7f73c5005740 1 -- 172.21.15.107:0/203577668 shutdown_connections 2024-05-09T08:48:12.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.816+0000 7f73c5005740 1 --2- 172.21.15.107:0/203577668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25dea6740 0x55d25dea8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.816+0000 7f73c5005740 1 --2- 172.21.15.107:0/203577668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25de9b9c0 0x55d25de9be40 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.816+0000 7f73c5005740 1 --2- 172.21.15.107:0/203577668 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25de9d0b0 0x55d25de9b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.816+0000 7f73c5005740 1 -- 172.21.15.107:0/203577668 >> 172.21.15.107:0/203577668 conn(0x55d25de640a0 msgr2=0x55d25de97f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:12.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.819+0000 7f73c5005740 1 -- 172.21.15.107:0/203577668 shutdown_connections 2024-05-09T08:48:12.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.819+0000 7f73bb7fe640 1 --2- 172.21.15.107:0/2834474645 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f73a40a2f10 0x7f73a40a53d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f73bc115af0 tx=0x7f73b0009260 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:12.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.819+0000 7f73c5005740 1 -- 172.21.15.107:0/203577668 wait complete. 2024-05-09T08:48:12.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.820+0000 7f73c5005740 1 -- start start 2024-05-09T08:48:12.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.820+0000 7f73c5005740 1 Processor -- start 2024-05-09T08:48:12.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.820+0000 7f73c5005740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25de9d0b0 0x55d25dea6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.820+0000 7f73c5005740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25dea6740 0x55d25dea3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.820+0000 7f73c5005740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25dea3a80 0x55d25dea3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.820+0000 7f73c5005740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d25dea9620 con 0x55d25dea3a80 2024-05-09T08:48:12.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.820+0000 7f73c5005740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d25ddbbb40 con 0x55d25de9d0b0 2024-05-09T08:48:12.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.820+0000 7f73c5005740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d25dea97a0 con 0x55d25dea6740 2024-05-09T08:48:12.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.820+0000 7f73c5005740 10 client.0 _ll_register_callbacks cb 0x55d25ddb00f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:12.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.821+0000 7f73c089f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25dea3a80 0x55d25dea3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.821+0000 7f73c089f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25dea3a80 0x55d25dea3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41660/0 (socket says 172.21.15.107:41660) 2024-05-09T08:48:12.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.821+0000 7f73c089f640 1 -- 172.21.15.107:0/688119135 learned_addr learned my addr 172.21.15.107:0/688119135 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:12.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.821+0000 7f73bb7fe640 1 --2- 172.21.15.107:0/688119135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25dea6740 0x55d25dea3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.821+0000 7f73bbfff640 1 --2- 172.21.15.107:0/688119135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25de9d0b0 0x55d25dea6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.821+0000 7f73c089f640 1 -- 172.21.15.107:0/688119135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25dea6740 msgr2=0x55d25dea3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:12.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.821+0000 7f73c089f640 1 --2- 172.21.15.107:0/688119135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25dea6740 0x55d25dea3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.821+0000 7f73c089f640 1 -- 172.21.15.107:0/688119135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25de9d0b0 msgr2=0x55d25dea6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:12.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.821+0000 7f73c089f640 1 --2- 172.21.15.107:0/688119135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25de9d0b0 0x55d25dea6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.821+0000 7f73c089f640 1 -- 172.21.15.107:0/688119135 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d25dc3cf70 con 0x55d25dea3a80 2024-05-09T08:48:12.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.821+0000 7f73bbfff640 1 --2- 172.21.15.107:0/688119135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25de9d0b0 0x55d25dea6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:12.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.822+0000 7f73c089f640 1 --2- 172.21.15.107:0/688119135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25dea3a80 0x55d25dea3f00 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f73b40986a0 tx=0x7f73b409bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f2c37f54382d73 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:12.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.823+0000 7f739b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:48:12.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.823+0000 7f739b7fe640 1 -- 172.21.15.107:0/688119135 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b40b9060 con 0x55d25dea3a80 2024-05-09T08:48:12.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.823+0000 7f73c5005740 1 -- 172.21.15.107:0/688119135 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d25de0e0c0 con 0x55d25dea3a80 2024-05-09T08:48:12.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.823+0000 7f739b7fe640 1 -- 172.21.15.107:0/688119135 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7f73b409b120 con 0x55d25dea3a80 2024-05-09T08:48:12.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.826+0000 7f739b7fe640 1 -- 172.21.15.107:0/688119135 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f73b40a9050 con 0x55d25dea3a80 2024-05-09T08:48:12.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.826+0000 7f73c5005740 10 client.19371 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:12.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.826+0000 7f73c5005740 1 -- 172.21.15.107:0/688119135 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d25dad4170 con 0x55d25dea3a80 2024-05-09T08:48:12.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.830+0000 7f739b7fe640 1 -- 172.21.15.107:0/688119135 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55d25de0e0c0 con 0x55d25dea3a80 2024-05-09T08:48:12.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.831+0000 7f739b7fe640 1 -- 172.21.15.107:0/688119135 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f73b400f040 con 0x55d25dea3a80 2024-05-09T08:48:12.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.831+0000 7f73c5005740 10 client.19371 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:12.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.831+0000 7f73c5005740 10 client.19371 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:48:12.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.831+0000 7f73c5005740 20 client.19371 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:12.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.831+0000 7f73c5005740 1 --2- 172.21.15.107:0/688119135 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d25ded9060 0x55d25dedb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.831+0000 7f73c5005740 4 client.19371 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:48:12.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.831+0000 7f73c5005740 1 -- 172.21.15.107:0/688119135 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d25de08410 con 0x55d25ded9060 2024-05-09T08:48:12.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.831+0000 7f73bbfff640 1 --2- 172.21.15.107:0/688119135 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d25ded9060 0x55d25dedb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.834+0000 7f73bbfff640 1 --2- 172.21.15.107:0/688119135 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d25ded9060 0x55d25dedb4e0 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ced8ce06bc1c460a server_cookie=946ddfaf213e6b9 in_seq=0 out_seq=0 2024-05-09T08:48:12.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.834+0000 7f739b7fe640 10 client.19371 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:12.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.835+0000 7f739b7fe640 1 -- 172.21.15.107:0/688119135 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d25de08410 con 0x55d25ded9060 2024-05-09T08:48:12.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.835+0000 7f739b7fe640 10 client.19371 handle_command_reply: tid=0 2024-05-09T08:48:12.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.835+0000 7f73c5005740 1 client.19371 shutdown 2024-05-09T08:48:12.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.836+0000 7f73c5005740 1 -- 172.21.15.107:0/688119135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25dea3a80 msgr2=0x55d25dea3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:12.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.836+0000 7f73c5005740 1 --2- 172.21.15.107:0/688119135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25dea3a80 0x55d25dea3f00 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f73b40986a0 tx=0x7f73b409bd90 comp rx=0 tx=0).stop 2024-05-09T08:48:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.836+0000 7f73c5005740 1 -- 172.21.15.107:0/688119135 shutdown_connections 2024-05-09T08:48:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.836+0000 7f73c5005740 1 --2- 172.21.15.107:0/688119135 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d25ded9060 0x55d25dedb4e0 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.836+0000 7f73c5005740 1 --2- 172.21.15.107:0/688119135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25dea3a80 0x55d25dea3f00 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.836+0000 7f73c5005740 1 --2- 172.21.15.107:0/688119135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25dea6740 0x55d25dea3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.836+0000 7f73c5005740 1 --2- 172.21.15.107:0/688119135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25de9d0b0 0x55d25dea6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.836+0000 7f73c5005740 1 -- 172.21.15.107:0/688119135 >> 172.21.15.107:0/688119135 conn(0x55d25de640a0 msgr2=0x55d25de9dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.836+0000 7f739b7fe640 0 client.19371 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.836+0000 7f73c5005740 1 -- 172.21.15.107:0/688119135 shutdown_connections 2024-05-09T08:48:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.836+0000 7f73c5005740 1 -- 172.21.15.107:0/688119135 wait complete. 2024-05-09T08:48:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.837+0000 7f73c5005740 20 client.19371 trim_cache size 0 max 16384 2024-05-09T08:48:12.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.879+0000 7f73c5005740 1 Processor -- start 2024-05-09T08:48:12.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.879+0000 7f73c5005740 1 -- start start 2024-05-09T08:48:12.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.879+0000 7f73c5005740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25de9d0b0 0x55d25deb25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.879+0000 7f73c5005740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25dea3a80 0x55d25dea7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.880+0000 7f73c5005740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25dea6740 0x55d25dee0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.880+0000 7f73bbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25de9d0b0 0x55d25deb25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.880+0000 7f73bbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25de9d0b0 0x55d25deb25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:40528/0 (socket says 172.21.15.107:40528) 2024-05-09T08:48:12.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.880+0000 7f73bb7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25dea3a80 0x55d25dea7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.880+0000 7f73bb7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25dea3a80 0x55d25dea7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:45566/0 (socket says 172.21.15.107:45566) 2024-05-09T08:48:12.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.880+0000 7f73bbfff640 1 -- 172.21.15.107:0/4222167198 learned_addr learned my addr 172.21.15.107:0/4222167198 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:12.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.880+0000 7f73c5005740 1 -- 172.21.15.107:0/4222167198 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f73a8012aa0 con 0x55d25dea6740 2024-05-09T08:48:12.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.880+0000 7f73c5005740 1 -- 172.21.15.107:0/4222167198 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f73a8010310 con 0x55d25de9d0b0 2024-05-09T08:48:12.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.880+0000 7f73c5005740 1 -- 172.21.15.107:0/4222167198 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f73a800e180 con 0x55d25dea3a80 2024-05-09T08:48:12.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.882+0000 7f73c089f640 1 --2- 172.21.15.107:0/4222167198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25dea6740 0x55d25dee0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.886+0000 7f73bbfff640 1 -- 172.21.15.107:0/4222167198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25dea3a80 msgr2=0x55d25dea7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:12.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.886+0000 7f73bbfff640 1 --2- 172.21.15.107:0/4222167198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25dea3a80 0x55d25dea7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.886+0000 7f73bbfff640 1 -- 172.21.15.107:0/4222167198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25dea6740 msgr2=0x55d25dee0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:12.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.886+0000 7f73bbfff640 1 --2- 172.21.15.107:0/4222167198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25dea6740 0x55d25dee0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.886+0000 7f73bbfff640 1 -- 172.21.15.107:0/4222167198 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d25de96490 con 0x55d25de9d0b0 2024-05-09T08:48:12.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.886+0000 7f73c089f640 1 --2- 172.21.15.107:0/4222167198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25dea6740 0x55d25dee0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:48:12.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.887+0000 7f73bbfff640 1 --2- 172.21.15.107:0/4222167198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25de9d0b0 0x55d25deb25d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00bc30 tx=0x7f73ac00be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc6644bd4ac0ac50 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:12.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.889+0000 7f73a1ffb640 1 -- 172.21.15.107:0/4222167198 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac011950 con 0x55d25de9d0b0 2024-05-09T08:48:12.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.889+0000 7f73a1ffb640 1 -- 172.21.15.107:0/4222167198 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73ac0050b0 con 0x55d25de9d0b0 2024-05-09T08:48:12.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.889+0000 7f73a1ffb640 1 -- 172.21.15.107:0/4222167198 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac027530 con 0x55d25de9d0b0 2024-05-09T08:48:12.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.890+0000 7f73c5005740 1 -- 172.21.15.107:0/4222167198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25de9d0b0 msgr2=0x55d25deb25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:12.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.890+0000 7f73c5005740 1 --2- 172.21.15.107:0/4222167198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25de9d0b0 0x55d25deb25d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00bc30 tx=0x7f73ac00be00 comp rx=0 tx=0).stop 2024-05-09T08:48:12.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.894+0000 7f73c5005740 1 -- 172.21.15.107:0/4222167198 shutdown_connections 2024-05-09T08:48:12.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.894+0000 7f73c5005740 1 --2- 172.21.15.107:0/4222167198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25dea6740 0x55d25dee0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.894+0000 7f73c5005740 1 --2- 172.21.15.107:0/4222167198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25dea3a80 0x55d25dea7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.894+0000 7f73c5005740 1 --2- 172.21.15.107:0/4222167198 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25de9d0b0 0x55d25deb25d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.894+0000 7f73c5005740 1 -- 172.21.15.107:0/4222167198 >> 172.21.15.107:0/4222167198 conn(0x55d25de640a0 msgr2=0x55d25dea1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:12.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.895+0000 7f73c5005740 1 -- 172.21.15.107:0/4222167198 shutdown_connections 2024-05-09T08:48:12.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.896+0000 7f73c5005740 1 -- 172.21.15.107:0/4222167198 wait complete. 2024-05-09T08:48:12.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.896+0000 7f73c5005740 1 -- start start 2024-05-09T08:48:12.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.897+0000 7f73c5005740 1 Processor -- start 2024-05-09T08:48:12.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.897+0000 7f73c5005740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25de9d0b0 0x55d25deaefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.898+0000 7f73c5005740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25dea3a80 0x55d25dea0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.898+0000 7f73c5005740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25dea6740 0x55d25dea0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.898+0000 7f73c5005740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f73a800e7d0 con 0x55d25de9d0b0 2024-05-09T08:48:12.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.898+0000 7f73c5005740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f73a800fc40 con 0x55d25dea6740 2024-05-09T08:48:12.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.898+0000 7f73c5005740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f73a800eea0 con 0x55d25dea3a80 2024-05-09T08:48:12.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.898+0000 7f73c5005740 10 client.0 _ll_register_callbacks cb 0x7f73b4097f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:12.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.899+0000 7f73bbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25de9d0b0 0x55d25deaefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.899+0000 7f73bbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25de9d0b0 0x55d25deaefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41686/0 (socket says 172.21.15.107:41686) 2024-05-09T08:48:12.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.899+0000 7f73bbfff640 1 -- 172.21.15.107:0/1539430958 learned_addr learned my addr 172.21.15.107:0/1539430958 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:12.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.900+0000 7f73c089f640 1 --2- 172.21.15.107:0/1539430958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25dea6740 0x55d25dea0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.900+0000 7f73bb7fe640 1 --2- 172.21.15.107:0/1539430958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25dea3a80 0x55d25dea0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.901+0000 7f73bbfff640 1 -- 172.21.15.107:0/1539430958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25dea3a80 msgr2=0x55d25dea0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:12.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.901+0000 7f73bbfff640 1 --2- 172.21.15.107:0/1539430958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25dea3a80 0x55d25dea0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.901+0000 7f73bbfff640 1 -- 172.21.15.107:0/1539430958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25dea6740 msgr2=0x55d25dea0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:12.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.901+0000 7f73bbfff640 1 --2- 172.21.15.107:0/1539430958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25dea6740 0x55d25dea0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.901+0000 7f73bbfff640 1 -- 172.21.15.107:0/1539430958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d25dee0460 con 0x55d25de9d0b0 2024-05-09T08:48:12.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.903+0000 7f73bbfff640 1 --2- 172.21.15.107:0/1539430958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25de9d0b0 0x55d25deaefc0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f73ac006010 tx=0x7f73ac00b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e49a9959313387e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:12.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.903+0000 7f739b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:48:12.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.904+0000 7f739b7fe640 1 -- 172.21.15.107:0/1539430958 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac034060 con 0x55d25de9d0b0 2024-05-09T08:48:12.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.904+0000 7f73c5005740 1 -- 172.21.15.107:0/1539430958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73ac0050b0 con 0x55d25de9d0b0 2024-05-09T08:48:12.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.904+0000 7f739b7fe640 1 -- 172.21.15.107:0/1539430958 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7f73ac011950 con 0x55d25de9d0b0 2024-05-09T08:48:12.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.904+0000 7f739b7fe640 1 -- 172.21.15.107:0/1539430958 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f73ac03b050 con 0x55d25de9d0b0 2024-05-09T08:48:12.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.905+0000 7f73c5005740 10 client.19377 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:12.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.905+0000 7f73c5005740 1 -- 172.21.15.107:0/1539430958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d25df3fa20 con 0x55d25de9d0b0 2024-05-09T08:48:12.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.905+0000 7f739b7fe640 1 -- 172.21.15.107:0/1539430958 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73ac0050b0 con 0x55d25de9d0b0 2024-05-09T08:48:12.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.905+0000 7f739b7fe640 1 -- 172.21.15.107:0/1539430958 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f73ac0270d0 con 0x55d25de9d0b0 2024-05-09T08:48:12.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.905+0000 7f73c5005740 10 client.19377 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:12.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.905+0000 7f73c5005740 10 client.19377 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:48:12.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.905+0000 7f73c5005740 20 client.19377 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:12.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.905+0000 7f73c5005740 1 --2- 172.21.15.107:0/1539430958 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d25df3fef0 0x55d25df422e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:12.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.905+0000 7f73c5005740 4 client.19377 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T08:48:12.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.905+0000 7f73c5005740 1 -- 172.21.15.107:0/1539430958 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d25dd85da0 con 0x55d25df3fef0 2024-05-09T08:48:12.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.907+0000 7f73bb7fe640 1 --2- 172.21.15.107:0/1539430958 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d25df3fef0 0x55d25df422e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:12.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.914+0000 7f73bb7fe640 1 --2- 172.21.15.107:0/1539430958 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d25df3fef0 0x55d25df422e0 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c88f1b0d3438d6f8 server_cookie=40021a82142d77d5 in_seq=0 out_seq=0 2024-05-09T08:48:12.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.915+0000 7f739b7fe640 10 client.19377 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:12.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.921+0000 7f739b7fe640 1 -- 172.21.15.107:0/1539430958 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d25dd85da0 con 0x55d25df3fef0 2024-05-09T08:48:12.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.921+0000 7f739b7fe640 10 client.19377 handle_command_reply: tid=0 2024-05-09T08:48:12.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.922+0000 7f73c5005740 1 client.19377 shutdown 2024-05-09T08:48:12.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.926+0000 7f73c5005740 1 -- 172.21.15.107:0/1539430958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25de9d0b0 msgr2=0x55d25deaefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:12.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.926+0000 7f73c5005740 1 --2- 172.21.15.107:0/1539430958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25de9d0b0 0x55d25deaefc0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f73ac006010 tx=0x7f73ac00b9a0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.928+0000 7f73c5005740 1 -- 172.21.15.107:0/1539430958 shutdown_connections 2024-05-09T08:48:12.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.928+0000 7f73c5005740 1 --2- 172.21.15.107:0/1539430958 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55d25df3fef0 0x55d25df422e0 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.928+0000 7f73c5005740 1 --2- 172.21.15.107:0/1539430958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d25dea6740 0x55d25dea0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.928+0000 7f73c5005740 1 --2- 172.21.15.107:0/1539430958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d25dea3a80 0x55d25dea0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.928+0000 7f73c5005740 1 --2- 172.21.15.107:0/1539430958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d25de9d0b0 0x55d25deaefc0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.928+0000 7f73c5005740 1 -- 172.21.15.107:0/1539430958 >> 172.21.15.107:0/1539430958 conn(0x55d25de640a0 msgr2=0x55d25dea1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:12.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.929+0000 7f739b7fe640 0 client.19377 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:12.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.930+0000 7f73c5005740 1 -- 172.21.15.107:0/1539430958 shutdown_connections 2024-05-09T08:48:12.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.931+0000 7f73c5005740 1 -- 172.21.15.107:0/1539430958 wait complete. 2024-05-09T08:48:12.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.931+0000 7f73c5005740 20 client.19377 trim_cache size 0 max 16384 2024-05-09T08:48:12.931 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-09T08:48:12.931 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-09T08:48:12.932 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:48:12.932 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-05-09T08:48:12.932 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "95f3b22d-e454-4b35-b06b-0a059ba031e5", 2024-05-09T08:48:12.932 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-05-09T08:48:12.932 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:48:12.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.933+0000 7f73c2302640 1 -- 172.21.15.107:0/2834474645 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f73a40a2f10 msgr2=0x7f73a40a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:12.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.933+0000 7f73c2302640 1 --2- 172.21.15.107:0/2834474645 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f73a40a2f10 0x7f73a40a53d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f73bc115af0 tx=0x7f73b0009260 comp rx=0 tx=0).stop 2024-05-09T08:48:12.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.933+0000 7f73c2302640 1 -- 172.21.15.107:0/2834474645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73bc11b010 msgr2=0x7f73bc118db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:12.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.933+0000 7f73c2302640 1 --2- 172.21.15.107:0/2834474645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73bc11b010 0x7f73bc118db0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f73b400eaa0 tx=0x7f73b400ef60 comp rx=0 tx=0).stop 2024-05-09T08:48:12.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.934+0000 7f73c2302640 1 -- 172.21.15.107:0/2834474645 shutdown_connections 2024-05-09T08:48:12.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.934+0000 7f73c2302640 1 --2- 172.21.15.107:0/2834474645 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f73a40a2f10 0x7f73a40a53d0 secure :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f73bc115af0 tx=0x7f73b0009260 comp rx=0 tx=0).stop 2024-05-09T08:48:12.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.934+0000 7f73c2302640 1 --2- 172.21.15.107:0/2834474645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73bc11b010 0x7f73bc118db0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.934+0000 7f73c2302640 1 --2- 172.21.15.107:0/2834474645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f73bc110080 0x7f73bc1149c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.934+0000 7f73c2302640 1 --2- 172.21.15.107:0/2834474645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73bc10e0b0 0x7f73bc114480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:12.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.934+0000 7f73c2302640 1 -- 172.21.15.107:0/2834474645 >> 172.21.15.107:0/2834474645 conn(0x7f73bc06c940 msgr2=0x7f73bc11c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:12.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.936+0000 7f73c2302640 1 -- 172.21.15.107:0/2834474645 shutdown_connections 2024-05-09T08:48:12.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:12.937+0000 7f73c2302640 1 -- 172.21.15.107:0/2834474645 wait complete. 2024-05-09T08:48:12.958 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:48:12.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:48:12.959 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:12 smithi107 ceph-mon[30122]: pgmap v466: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 121 KiB/s rd, 191 KiB/s wr, 36 op/s 2024-05-09T08:48:13.015 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-05-09T08:48:13.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-09T08:48:13.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:12 smithi157 ceph-mon[33614]: pgmap v466: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 121 KiB/s rd, 191 KiB/s wr, 36 op/s 2024-05-09T08:48:13.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:12 smithi110 ceph-mon[33622]: pgmap v466: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 16 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 121 KiB/s rd, 191 KiB/s wr, 36 op/s 2024-05-09T08:48:13.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.212+0000 7fbff0161640 1 -- 172.21.15.107:0/1715590127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe810aaf0 msgr2=0x7fbfe8109070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:13.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.212+0000 7fbff0161640 1 --2- 172.21.15.107:0/1715590127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe810aaf0 0x7fbfe8109070 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4009860 tx=0x7fbfe402a380 comp rx=0 tx=0).stop 2024-05-09T08:48:13.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.214+0000 7fbff0161640 1 -- 172.21.15.107:0/1715590127 shutdown_connections 2024-05-09T08:48:13.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.214+0000 7fbff0161640 1 --2- 172.21.15.107:0/1715590127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfe81095b0 0x7fbfe81099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.214+0000 7fbff0161640 1 --2- 172.21.15.107:0/1715590127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe810aaf0 0x7fbfe8109070 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.214+0000 7fbff0161640 1 --2- 172.21.15.107:0/1715590127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfe810a1d0 0x7fbfe810a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.214+0000 7fbff0161640 1 -- 172.21.15.107:0/1715590127 >> 172.21.15.107:0/1715590127 conn(0x7fbfe806c810 msgr2=0x7fbfe806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:13.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.215+0000 7fbff0161640 1 -- 172.21.15.107:0/1715590127 shutdown_connections 2024-05-09T08:48:13.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.215+0000 7fbff0161640 1 -- 172.21.15.107:0/1715590127 wait complete. 2024-05-09T08:48:13.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.216+0000 7fbff0161640 1 Processor -- start 2024-05-09T08:48:13.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.216+0000 7fbff0161640 1 -- start start 2024-05-09T08:48:13.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.216+0000 7fbff0161640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfe81095b0 0x7fbfe8117530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.216+0000 7fbff0161640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfe810a1d0 0x7fbfe8117a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.216+0000 7fbff0161640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe810aaf0 0x7fbfe8117fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.216+0000 7fbff0161640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe811c3f0 con 0x7fbfe810aaf0 2024-05-09T08:48:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.216+0000 7fbff0161640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe811c270 con 0x7fbfe81095b0 2024-05-09T08:48:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.216+0000 7fbff0161640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe811c570 con 0x7fbfe810a1d0 2024-05-09T08:48:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.217+0000 7fbfee6fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe810aaf0 0x7fbfe8117fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.217+0000 7fbfee6fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe810aaf0 0x7fbfe8117fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41714/0 (socket says 172.21.15.107:41714) 2024-05-09T08:48:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.217+0000 7fbfee6fe640 1 -- 172.21.15.107:0/761915244 learned_addr learned my addr 172.21.15.107:0/761915244 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:13.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.217+0000 7fbfee6fe640 1 -- 172.21.15.107:0/761915244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfe810a1d0 msgr2=0x7fbfe8117a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:13.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.217+0000 7fbfee6fe640 1 --2- 172.21.15.107:0/761915244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfe810a1d0 0x7fbfe8117a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.217+0000 7fbfee6fe640 1 -- 172.21.15.107:0/761915244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfe81095b0 msgr2=0x7fbfe8117530 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T08:48:13.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.217+0000 7fbfee6fe640 1 --2- 172.21.15.107:0/761915244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfe81095b0 0x7fbfe8117530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.217+0000 7fbfee6fe640 1 -- 172.21.15.107:0/761915244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe81a2700 con 0x7fbfe810aaf0 2024-05-09T08:48:13.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.222+0000 7fbfee6fe640 1 --2- 172.21.15.107:0/761915244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe810aaf0 0x7fbfe8117fb0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fbfe000eda0 tx=0x7fbfe000c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:13.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.223+0000 7fbfdeffd640 1 -- 172.21.15.107:0/761915244 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe0003b70 con 0x7fbfe810aaf0 2024-05-09T08:48:13.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.223+0000 7fbfdeffd640 1 -- 172.21.15.107:0/761915244 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfe0003d10 con 0x7fbfe810aaf0 2024-05-09T08:48:13.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.223+0000 7fbfdeffd640 1 -- 172.21.15.107:0/761915244 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe0017940 con 0x7fbfe810aaf0 2024-05-09T08:48:13.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.226+0000 7fbff0161640 1 -- 172.21.15.107:0/761915244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfe81a29f0 con 0x7fbfe810aaf0 2024-05-09T08:48:13.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.226+0000 7fbff0161640 1 -- 172.21.15.107:0/761915244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfe81a2ec0 con 0x7fbfe810aaf0 2024-05-09T08:48:13.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.230+0000 7fbff2e6c740 1 Processor -- start 2024-05-09T08:48:13.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.231+0000 7fbfdeffd640 1 -- 172.21.15.107:0/761915244 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fbfe0017ae0 con 0x7fbfe810aaf0 2024-05-09T08:48:13.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.231+0000 7fbff2e6c740 1 -- start start 2024-05-09T08:48:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.232+0000 7fbff2e6c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f195470 0x56237f195850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.232+0000 7fbff2e6c740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195d90 0x56237f19ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.232+0000 7fbff2e6c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f1a0460 0x56237f1a2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.232+0000 7fbff2e6c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56237f0fad00 con 0x56237f195470 2024-05-09T08:48:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.232+0000 7fbff2e6c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56237f0a24c0 con 0x56237f1a0460 2024-05-09T08:48:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.232+0000 7fbff2e6c740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56237f0f6cc0 con 0x56237f195d90 2024-05-09T08:48:13.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.233+0000 7fbfdeffd640 1 --2- 172.21.15.107:0/761915244 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbfd40a2f00 0x7fbfd40a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.233+0000 7fbfdeffd640 1 -- 172.21.15.107:0/761915244 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7fbfe00cfba0 con 0x7fbfe810aaf0 2024-05-09T08:48:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.234+0000 7fbfee6fe640 1 --2- 172.21.15.107:0/761915244 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbfd40a2f00 0x7fbfd40a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.234+0000 7fbfed6fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f1a0460 0x56237f1a2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.234+0000 7fbfed6fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f195470 0x56237f195850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.234+0000 7fbfed6fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f195470 0x56237f195850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41730/0 (socket says 172.21.15.107:41730) 2024-05-09T08:48:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.234+0000 7fbfed6fc640 1 -- 172.21.15.107:0/2619177254 learned_addr learned my addr 172.21.15.107:0/2619177254 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:13.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.234+0000 7fbfedefd640 1 --2- 172.21.15.107:0/2619177254 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195d90 0x56237f19ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.234+0000 7fbfed6fc640 1 -- 172.21.15.107:0/2619177254 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195d90 msgr2=0x56237f19ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.234+0000 7fbfed6fc640 1 --2- 172.21.15.107:0/2619177254 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195d90 0x56237f19ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.234+0000 7fbfed6fc640 1 -- 172.21.15.107:0/2619177254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f1a0460 msgr2=0x56237f1a2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.234+0000 7fbfed6fc640 1 --2- 172.21.15.107:0/2619177254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f1a0460 0x56237f1a2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.234+0000 7fbfed6fc640 1 -- 172.21.15.107:0/2619177254 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56237eed87d0 con 0x56237f195470 2024-05-09T08:48:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.235+0000 7fbfed6fc640 1 --2- 172.21.15.107:0/2619177254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f195470 0x56237f195850 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4009830 tx=0x7fbfe4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=462b3ee7e9b235a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.235+0000 7fbfd3fff640 1 -- 172.21.15.107:0/2619177254 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4037030 con 0x56237f195470 2024-05-09T08:48:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.235+0000 7fbfd3fff640 1 -- 172.21.15.107:0/2619177254 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfe4004030 con 0x56237f195470 2024-05-09T08:48:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.235+0000 7fbfd3fff640 1 -- 172.21.15.107:0/2619177254 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4003850 con 0x56237f195470 2024-05-09T08:48:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.235+0000 7fbfee6fe640 1 --2- 172.21.15.107:0/761915244 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbfd40a2f00 0x7fbfd40a53c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbfe00ccff0 tx=0x7fbfe0004750 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.236+0000 7fbff2e6c740 1 -- 172.21.15.107:0/2619177254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f195470 msgr2=0x56237f195850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.236+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/2619177254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f195470 0x56237f195850 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4009830 tx=0x7fbfe4000bc0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.236+0000 7fbff2e6c740 1 -- 172.21.15.107:0/2619177254 shutdown_connections 2024-05-09T08:48:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.236+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/2619177254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f1a0460 0x56237f1a2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.236+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/2619177254 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195d90 0x56237f19ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.236+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/2619177254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f195470 0x56237f195850 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.236+0000 7fbff2e6c740 1 -- 172.21.15.107:0/2619177254 >> 172.21.15.107:0/2619177254 conn(0x56237f191790 msgr2=0x56237f191b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.237+0000 7fbff2e6c740 1 -- 172.21.15.107:0/2619177254 shutdown_connections 2024-05-09T08:48:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.237+0000 7fbff2e6c740 1 -- 172.21.15.107:0/2619177254 wait complete. 2024-05-09T08:48:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.237+0000 7fbff2e6c740 1 -- start start 2024-05-09T08:48:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.237+0000 7fbff2e6c740 1 Processor -- start 2024-05-09T08:48:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.238+0000 7fbff2e6c740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195470 0x56237f19d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.238+0000 7fbff2e6c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f195d90 0x56237f19d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.238+0000 7fbff2e6c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f1a0460 0x56237f19df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.238+0000 7fbff2e6c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56237f1a31f0 con 0x56237f195d90 2024-05-09T08:48:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.238+0000 7fbff2e6c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56237f0b5b90 con 0x56237f1a0460 2024-05-09T08:48:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.238+0000 7fbff2e6c740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x56237f1a3370 con 0x56237f195470 2024-05-09T08:48:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.238+0000 7fbff2e6c740 10 client.0 _ll_register_callbacks cb 0x56237f0f1da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.238+0000 7fbfed6fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195470 0x56237f19d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.238+0000 7fbfed6fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195470 0x56237f19d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:45612/0 (socket says 172.21.15.107:45612) 2024-05-09T08:48:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.238+0000 7fbfed6fc640 1 -- 172.21.15.107:0/1251797936 learned_addr learned my addr 172.21.15.107:0/1251797936 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.238+0000 7fbfed6fc640 1 --2- 172.21.15.107:0/1251797936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f1a0460 0x56237f19df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.239+0000 7fbfed6fc640 1 -- 172.21.15.107:0/1251797936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f1a0460 msgr2=0x56237f19df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.239+0000 7fbfed6fc640 1 --2- 172.21.15.107:0/1251797936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f1a0460 0x56237f19df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.239+0000 7fbfed6fc640 1 -- 172.21.15.107:0/1251797936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f195d90 msgr2=0x56237f19d9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T08:48:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.239+0000 7fbfed6fc640 1 --2- 172.21.15.107:0/1251797936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f195d90 0x56237f19d9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.239+0000 7fbfed6fc640 1 -- 172.21.15.107:0/1251797936 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56237ee0e3f0 con 0x56237f195470 2024-05-09T08:48:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.239+0000 7fbfed6fc640 1 --2- 172.21.15.107:0/1251797936 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195470 0x56237f19d490 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4003830 tx=0x7fbfe402a880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a40d3c16319555d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.239+0000 7fbfd17fa640 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:48:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.239+0000 7fbfd17fa640 1 -- 172.21.15.107:0/1251797936 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4037070 con 0x56237f195470 2024-05-09T08:48:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.240+0000 7fbff2e6c740 1 -- 172.21.15.107:0/1251797936 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfe4004030 con 0x56237f195470 2024-05-09T08:48:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.240+0000 7fbfd17fa640 1 -- 172.21.15.107:0/1251797936 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7fbfe4003cb0 con 0x56237f195470 2024-05-09T08:48:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.240+0000 7fbfd17fa640 1 -- 172.21.15.107:0/1251797936 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbfe4059050 con 0x56237f195470 2024-05-09T08:48:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.240+0000 7fbff2e6c740 10 client.28516 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:13.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.240+0000 7fbff2e6c740 1 -- 172.21.15.107:0/1251797936 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56237edce0f0 con 0x56237f195470 2024-05-09T08:48:13.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.241+0000 7fbfd17fa640 1 -- 172.21.15.107:0/1251797936 <== mon.1 v2:172.21.15.157:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfe4004030 con 0x56237f195470 2024-05-09T08:48:13.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.241+0000 7fbfd17fa640 1 -- 172.21.15.107:0/1251797936 <== mon.1 v2:172.21.15.157:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fbfe404ec40 con 0x56237f195470 2024-05-09T08:48:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.241+0000 7fbff2e6c740 10 client.28516 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.241+0000 7fbff2e6c740 10 client.28516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:48:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.241+0000 7fbff2e6c740 20 client.28516 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.241+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/1251797936 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56237f1b08f0 0x56237f1b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.241+0000 7fbff2e6c740 4 client.28516 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:48:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.241+0000 7fbff2e6c740 1 -- 172.21.15.107:0/1251797936 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56237f108430 con 0x56237f1b08f0 2024-05-09T08:48:13.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.246+0000 7fbfedefd640 1 --2- 172.21.15.107:0/1251797936 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56237f1b08f0 0x56237f1b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.251+0000 7fbfedefd640 1 --2- 172.21.15.107:0/1251797936 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56237f1b08f0 0x56237f1b2d70 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bec5879a39801309 server_cookie=d0529731fb41b0a1 in_seq=0 out_seq=0 2024-05-09T08:48:13.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.251+0000 7fbfd17fa640 10 client.28516 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:13.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.255+0000 7fbfd17fa640 1 -- 172.21.15.107:0/1251797936 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56237f108430 con 0x56237f1b08f0 2024-05-09T08:48:13.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.255+0000 7fbfd17fa640 10 client.28516 handle_command_reply: tid=0 2024-05-09T08:48:13.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.256+0000 7fbff2e6c740 1 client.28516 shutdown 2024-05-09T08:48:13.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.258+0000 7fbff2e6c740 1 -- 172.21.15.107:0/1251797936 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195470 msgr2=0x56237f19d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:13.257 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.258+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/1251797936 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195470 0x56237f19d490 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4003830 tx=0x7fbfe402a880 comp rx=0 tx=0).stop 2024-05-09T08:48:13.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.258+0000 7fbff2e6c740 1 -- 172.21.15.107:0/1251797936 shutdown_connections 2024-05-09T08:48:13.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.258+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/1251797936 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56237f1b08f0 0x56237f1b2d70 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.258+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/1251797936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f1a0460 0x56237f19df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.258+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/1251797936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f195d90 0x56237f19d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.258+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/1251797936 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195470 0x56237f19d490 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.258+0000 7fbff2e6c740 1 -- 172.21.15.107:0/1251797936 >> 172.21.15.107:0/1251797936 conn(0x56237f191790 msgr2=0x56237f197590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:13.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.259+0000 7fbfd17fa640 0 client.28516 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:13.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.259+0000 7fbff2e6c740 1 -- 172.21.15.107:0/1251797936 shutdown_connections 2024-05-09T08:48:13.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.260+0000 7fbff2e6c740 1 -- 172.21.15.107:0/1251797936 wait complete. 2024-05-09T08:48:13.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.260+0000 7fbff2e6c740 20 client.28516 trim_cache size 0 max 16384 2024-05-09T08:48:13.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.296+0000 7fbff2e6c740 1 Processor -- start 2024-05-09T08:48:13.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.299+0000 7fbff2e6c740 1 -- start start 2024-05-09T08:48:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.300+0000 7fbff2e6c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f195470 0x56237f1ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.300+0000 7fbff2e6c740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195d90 0x56237f197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.300+0000 7fbff2e6c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f1a0460 0x56237f19db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.300+0000 7fbff2e6c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe81c8b40 con 0x56237f1a0460 2024-05-09T08:48:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.300+0000 7fbff2e6c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe81c63b0 con 0x56237f195470 2024-05-09T08:48:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.300+0000 7fbff2e6c740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe806d3d0 con 0x56237f195d90 2024-05-09T08:48:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.301+0000 7fbfedefd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195d90 0x56237f197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.301+0000 7fbfed6fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f195470 0x56237f1ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.301+0000 7fbfed6fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f195470 0x56237f1ac0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:40564/0 (socket says 172.21.15.107:40564) 2024-05-09T08:48:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.301+0000 7fbfed6fc640 1 -- 172.21.15.107:0/4244138455 learned_addr learned my addr 172.21.15.107:0/4244138455 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:13.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.303+0000 7fbfed6fc640 1 --2- 172.21.15.107:0/4244138455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f1a0460 0x56237f19db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.305+0000 7fbfedefd640 1 -- 172.21.15.107:0/4244138455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f195470 msgr2=0x56237f1ac0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:13.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.305+0000 7fbfedefd640 1 --2- 172.21.15.107:0/4244138455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f195470 0x56237f1ac0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.305+0000 7fbfedefd640 1 -- 172.21.15.107:0/4244138455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f1a0460 msgr2=0x56237f19db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.305+0000 7fbfedefd640 1 --2- 172.21.15.107:0/4244138455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f1a0460 0x56237f19db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.305+0000 7fbfedefd640 1 -- 172.21.15.107:0/4244138455 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56237f1a2bc0 con 0x56237f195d90 2024-05-09T08:48:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.306+0000 7fbfedefd640 1 --2- 172.21.15.107:0/4244138455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195d90 0x56237f197cb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x56237f1b3620 tx=0x7fbfd8005f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb0bb5252f17ac75 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.306+0000 7fbfd37fe640 1 -- 172.21.15.107:0/4244138455 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd8010b60 con 0x56237f195d90 2024-05-09T08:48:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.306+0000 7fbfd37fe640 1 -- 172.21.15.107:0/4244138455 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfd801a8d0 con 0x56237f195d90 2024-05-09T08:48:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.306+0000 7fbfd37fe640 1 -- 172.21.15.107:0/4244138455 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd80047a0 con 0x56237f195d90 2024-05-09T08:48:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.307+0000 7fbff2e6c740 1 -- 172.21.15.107:0/4244138455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195d90 msgr2=0x56237f197cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.307+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/4244138455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195d90 0x56237f197cb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x56237f1b3620 tx=0x7fbfd8005f00 comp rx=0 tx=0).stop 2024-05-09T08:48:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.314+0000 7fbff2e6c740 1 -- 172.21.15.107:0/4244138455 shutdown_connections 2024-05-09T08:48:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.314+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/4244138455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f1a0460 0x56237f19db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.314+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/4244138455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195d90 0x56237f197cb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.314+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/4244138455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f195470 0x56237f1ac0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.314+0000 7fbff2e6c740 1 -- 172.21.15.107:0/4244138455 >> 172.21.15.107:0/4244138455 conn(0x56237f191790 msgr2=0x56237f19d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.314+0000 7fbff2e6c740 1 -- 172.21.15.107:0/4244138455 shutdown_connections 2024-05-09T08:48:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.314+0000 7fbff2e6c740 1 -- 172.21.15.107:0/4244138455 wait complete. 2024-05-09T08:48:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.315+0000 7fbff2e6c740 1 -- start start 2024-05-09T08:48:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.315+0000 7fbff2e6c740 1 Processor -- start 2024-05-09T08:48:13.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.316+0000 7fbff2e6c740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195470 0x56237f1a85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.316+0000 7fbff2e6c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f195d90 0x56237f1a8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.316+0000 7fbfed6fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195470 0x56237f1a85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.316+0000 7fbfed6fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195470 0x56237f1a85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:45616/0 (socket says 172.21.15.107:45616) 2024-05-09T08:48:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.316+0000 7fbfed6fc640 1 -- 172.21.15.107:0/3047165453 learned_addr learned my addr 172.21.15.107:0/3047165453 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.316+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/3047165453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f1a0460 0x56237f199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.316+0000 7fbff2e6c740 1 -- 172.21.15.107:0/3047165453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe81c4890 con 0x56237f195d90 2024-05-09T08:48:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.316+0000 7fbff2e6c740 1 -- 172.21.15.107:0/3047165453 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe81c5ce0 con 0x56237f1a0460 2024-05-09T08:48:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.316+0000 7fbff2e6c740 1 -- 172.21.15.107:0/3047165453 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe81c4f40 con 0x56237f195470 2024-05-09T08:48:13.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.317+0000 7fbfed6fc640 1 --2- 172.21.15.107:0/3047165453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f1a0460 0x56237f199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.317+0000 7fbfed6fc640 1 -- 172.21.15.107:0/3047165453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f1a0460 msgr2=0x56237f199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.317+0000 7fbfed6fc640 1 --2- 172.21.15.107:0/3047165453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f1a0460 0x56237f199bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.317+0000 7fbfed6fc640 1 -- 172.21.15.107:0/3047165453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f195d90 msgr2=0x56237f1a8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.317+0000 7fbfed6fc640 1 --2- 172.21.15.107:0/3047165453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f195d90 0x56237f1a8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.317+0000 7fbfed6fc640 1 -- 172.21.15.107:0/3047165453 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56237f19a3e0 con 0x56237f195470 2024-05-09T08:48:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.317+0000 7fbff2e6c740 10 client.0 _ll_register_callbacks cb 0x7fbfe40096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.318+0000 7fbfed6fc640 1 --2- 172.21.15.107:0/3047165453 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195470 0x56237f1a85c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4030d30 tx=0x7fbfe4005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d59e12dca35bb4d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.318+0000 7fbff2e6c740 1 -- 172.21.15.107:0/3047165453 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfd801a8d0 con 0x56237f195470 2024-05-09T08:48:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.319+0000 7fbfd17fa640 10 client.28522 ms_handle_connect on v2:172.21.15.157:3300/0 2024-05-09T08:48:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.323+0000 7fbfd17fa640 1 -- 172.21.15.107:0/3047165453 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4037040 con 0x56237f195470 2024-05-09T08:48:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.323+0000 7fbfd17fa640 1 -- 172.21.15.107:0/3047165453 <== mon.1 v2:172.21.15.157:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7fbfe404e440 con 0x56237f195470 2024-05-09T08:48:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.323+0000 7fbfd17fa640 1 -- 172.21.15.107:0/3047165453 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbfe404e6a0 con 0x56237f195470 2024-05-09T08:48:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.326+0000 7fbff2e6c740 10 client.28522 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.326+0000 7fbff2e6c740 1 -- 172.21.15.107:0/3047165453 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({fsmap=31}) -- 0x56237f216940 con 0x56237f195470 2024-05-09T08:48:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.327+0000 7fbfd17fa640 1 -- 172.21.15.107:0/3047165453 <== mon.1 v2:172.21.15.157:3300/0 4 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fbfe404e840 con 0x56237f195470 2024-05-09T08:48:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.327+0000 7fbff2e6c740 10 client.28522 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.327+0000 7fbff2e6c740 10 client.28522 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:48:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.327+0000 7fbff2e6c740 20 client.28522 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.327+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/3047165453 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56237f216f70 0x56237f219400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.327+0000 7fbff2e6c740 4 client.28522 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:48:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.327+0000 7fbff2e6c740 1 -- 172.21.15.107:0/3047165453 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56237f102780 con 0x56237f216f70 2024-05-09T08:48:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.328+0000 7fbfedefd640 1 --2- 172.21.15.107:0/3047165453 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56237f216f70 0x56237f219400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.330+0000 7fbfedefd640 1 --2- 172.21.15.107:0/3047165453 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56237f216f70 0x56237f219400 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7baa1b23cb110b80 server_cookie=2955026c72d0e68f in_seq=0 out_seq=0 2024-05-09T08:48:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.330+0000 7fbfd17fa640 10 client.28522 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.334+0000 7fbfd17fa640 1 -- 172.21.15.107:0/3047165453 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56237f102780 con 0x56237f216f70 2024-05-09T08:48:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.334+0000 7fbfd17fa640 10 client.28522 handle_command_reply: tid=0 2024-05-09T08:48:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.334+0000 7fbff2e6c740 1 client.28522 shutdown 2024-05-09T08:48:13.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.336+0000 7fbff2e6c740 1 -- 172.21.15.107:0/3047165453 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195470 msgr2=0x56237f1a85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:13.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.336+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/3047165453 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195470 0x56237f1a85c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4030d30 tx=0x7fbfe4005850 comp rx=0 tx=0).stop 2024-05-09T08:48:13.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.337+0000 7fbff2e6c740 1 -- 172.21.15.107:0/3047165453 shutdown_connections 2024-05-09T08:48:13.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.337+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/3047165453 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x56237f216f70 0x56237f219400 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.337+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/3047165453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56237f1a0460 0x56237f199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.337+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/3047165453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56237f195d90 0x56237f1a8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.337+0000 7fbff2e6c740 1 --2- 172.21.15.107:0/3047165453 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56237f195470 0x56237f1a85c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.337+0000 7fbff2e6c740 1 -- 172.21.15.107:0/3047165453 >> 172.21.15.107:0/3047165453 conn(0x56237f191790 msgr2=0x56237f19dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:13.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.338+0000 7fbfd17fa640 0 client.28522 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:13.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.339+0000 7fbff2e6c740 1 -- 172.21.15.107:0/3047165453 shutdown_connections 2024-05-09T08:48:13.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.341+0000 7fbff2e6c740 1 -- 172.21.15.107:0/3047165453 wait complete. 2024-05-09T08:48:13.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.342+0000 7fbff2e6c740 20 client.28522 trim_cache size 0 max 16384 2024-05-09T08:48:13.347 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:48:13.347 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T08:48:13.347 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-05-09T08:48:13.348 INFO:teuthology.orchestra.run.smithi107.stdout: "95f3b22d-e454-4b35-b06b-0a059ba031e5": { 2024-05-09T08:48:13.348 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-05-09T08:48:13.348 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "95f3b22d-e454-4b35-b06b-0a059ba031e5", 2024-05-09T08:48:13.348 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-05-09T08:48:13.348 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:13.348 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:13.348 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:48:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.351+0000 7fbff0161640 1 -- 172.21.15.107:0/761915244 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbfd40a2f00 msgr2=0x7fbfd40a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.351+0000 7fbff0161640 1 --2- 172.21.15.107:0/761915244 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbfd40a2f00 0x7fbfd40a53c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbfe00ccff0 tx=0x7fbfe0004750 comp rx=0 tx=0).stop 2024-05-09T08:48:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.351+0000 7fbff0161640 1 -- 172.21.15.107:0/761915244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe810aaf0 msgr2=0x7fbfe8117fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.351+0000 7fbff0161640 1 --2- 172.21.15.107:0/761915244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe810aaf0 0x7fbfe8117fb0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fbfe000eda0 tx=0x7fbfe000c6e0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.352+0000 7fbff0161640 1 -- 172.21.15.107:0/761915244 shutdown_connections 2024-05-09T08:48:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.352+0000 7fbff0161640 1 --2- 172.21.15.107:0/761915244 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbfd40a2f00 0x7fbfd40a53c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.352+0000 7fbff0161640 1 --2- 172.21.15.107:0/761915244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe810aaf0 0x7fbfe8117fb0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.352+0000 7fbff0161640 1 --2- 172.21.15.107:0/761915244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfe810a1d0 0x7fbfe8117a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.352+0000 7fbff0161640 1 --2- 172.21.15.107:0/761915244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfe81095b0 0x7fbfe8117530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.352+0000 7fbff0161640 1 -- 172.21.15.107:0/761915244 >> 172.21.15.107:0/761915244 conn(0x7fbfe806c810 msgr2=0x7fbfe8115450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.352+0000 7fbff0161640 1 -- 172.21.15.107:0/761915244 shutdown_connections 2024-05-09T08:48:13.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.352+0000 7fbff0161640 1 -- 172.21.15.107:0/761915244 wait complete. 2024-05-09T08:48:13.371 INFO:tasks.cephfs.filesystem:scrub status for tag:95f3b22d-e454-4b35-b06b-0a059ba031e5 - {'path': '/', 'tag': '95f3b22d-e454-4b35-b06b-0a059ba031e5', 'options': 'recursive,force'} 2024-05-09T08:48:13.519 DEBUG:teuthology.parallel:result is None 2024-05-09T08:48:13.519 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-05-09T08:48:13.588 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-05-09T08:48:13.588 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-05-09T08:48:13.599 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi107.front.sepia.ceph.com 2024-05-09T08:48:13.611 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-05-09T08:48:13.623 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-05-09T08:48:13.623 INFO:teuthology.task.exec:Executing custom commands... 2024-05-09T08:48:13.623 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-05-09T08:48:13.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-05-09T08:48:13.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-05-09T08:48:13.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-09T08:48:13.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.941+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/4194954650 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf281010f0 msgr2=0x7fdf2810d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:13.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.941+0000 7fdf2ebb7640 1 --2- 172.21.15.107:0/4194954650 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf281010f0 0x7fdf2810d600 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdf20009cd0 tx=0x7fdf2002a930 comp rx=0 tx=0).stop 2024-05-09T08:48:13.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.942+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/4194954650 shutdown_connections 2024-05-09T08:48:13.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.942+0000 7fdf2ebb7640 1 --2- 172.21.15.107:0/4194954650 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf281010f0 0x7fdf2810d600 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.942+0000 7fdf2ebb7640 1 --2- 172.21.15.107:0/4194954650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf28100730 0x7fdf28100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.942+0000 7fdf2ebb7640 1 --2- 172.21.15.107:0/4194954650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf28106740 0x7fdf28106b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.942+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/4194954650 >> 172.21.15.107:0/4194954650 conn(0x7fdf28077850 msgr2=0x7fdf28077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:13.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.946+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/4194954650 shutdown_connections 2024-05-09T08:48:13.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.946+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/4194954650 wait complete. 2024-05-09T08:48:13.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.947+0000 7fdf2ebb7640 1 Processor -- start 2024-05-09T08:48:13.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.947+0000 7fdf2ebb7640 1 -- start start 2024-05-09T08:48:13.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.948+0000 7fdf2ebb7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf28100730 0x7fdf2810d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.948+0000 7fdf2ebb7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf281010f0 0x7fdf28109550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.948+0000 7fdf2ebb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf28106740 0x7fdf281a21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.948+0000 7fdf27fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf281010f0 0x7fdf28109550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.948+0000 7fdf27fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf281010f0 0x7fdf28109550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:40586/0 (socket says 172.21.15.107:40586) 2024-05-09T08:48:13.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.948+0000 7fdf27fff640 1 -- 172.21.15.107:0/3950379882 learned_addr learned my addr 172.21.15.107:0/3950379882 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:13.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.948+0000 7fdf2ebb7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdf2810f970 con 0x7fdf28106740 2024-05-09T08:48:13.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.948+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/3950379882 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdf2810f7f0 con 0x7fdf281010f0 2024-05-09T08:48:13.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.948+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/3950379882 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdf2810faf0 con 0x7fdf28100730 2024-05-09T08:48:13.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.949+0000 7fdf2c953640 1 --2- 172.21.15.107:0/3950379882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf28100730 0x7fdf2810d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.949+0000 7fdf2d154640 1 --2- 172.21.15.107:0/3950379882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf28106740 0x7fdf281a21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.950+0000 7fdf27fff640 1 -- 172.21.15.107:0/3950379882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf28100730 msgr2=0x7fdf2810d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:13.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.950+0000 7fdf27fff640 1 --2- 172.21.15.107:0/3950379882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf28100730 0x7fdf2810d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.950+0000 7fdf27fff640 1 -- 172.21.15.107:0/3950379882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf28106740 msgr2=0x7fdf281a21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:13.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.950+0000 7fdf27fff640 1 --2- 172.21.15.107:0/3950379882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf28106740 0x7fdf281a21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:13.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.950+0000 7fdf27fff640 1 -- 172.21.15.107:0/3950379882 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf28109b60 con 0x7fdf281010f0 2024-05-09T08:48:13.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.952+0000 7fdf27fff640 1 --2- 172.21.15.107:0/3950379882 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf281010f0 0x7fdf28109550 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00b4a0 tx=0x7fdf1c00b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.953+0000 7fdf25ffb640 1 -- 172.21.15.107:0/3950379882 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c015020 con 0x7fdf281010f0 2024-05-09T08:48:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.953+0000 7fdf25ffb640 1 -- 172.21.15.107:0/3950379882 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf1c011ce0 con 0x7fdf281010f0 2024-05-09T08:48:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.953+0000 7fdf25ffb640 1 -- 172.21.15.107:0/3950379882 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c010dc0 con 0x7fdf281010f0 2024-05-09T08:48:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.953+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/3950379882 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf28109e50 con 0x7fdf281010f0 2024-05-09T08:48:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.953+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/3950379882 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf2810a2f0 con 0x7fdf281010f0 2024-05-09T08:48:13.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.956+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/3950379882 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf0c005290 con 0x7fdf281010f0 2024-05-09T08:48:13.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.957+0000 7fdf25ffb640 1 -- 172.21.15.107:0/3950379882 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fdf1c018020 con 0x7fdf281010f0 2024-05-09T08:48:13.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.958+0000 7fdf25ffb640 1 --2- 172.21.15.107:0/3950379882 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf180a2fd0 0x7fdf180a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:13.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.958+0000 7fdf25ffb640 1 -- 172.21.15.107:0/3950379882 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7fdf1c0ced90 con 0x7fdf281010f0 2024-05-09T08:48:13.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.962+0000 7fdf2c953640 1 --2- 172.21.15.107:0/3950379882 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf180a2fd0 0x7fdf180a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:13.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.962+0000 7fdf2c953640 1 --2- 172.21.15.107:0/3950379882 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf180a2fd0 0x7fdf180a5490 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdf14009990 tx=0x7fdf14009110 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:13.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:13.966+0000 7fdf25ffb640 1 -- 172.21.15.107:0/3950379882 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdf1c01f020 con 0x7fdf281010f0 2024-05-09T08:48:14.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:13 smithi107 ceph-mon[30122]: scrub queued for path: / 2024-05-09T08:48:14.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:13 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:48:14.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:13 smithi107 ceph-mon[30122]: scrub summary: active paths [/] 2024-05-09T08:48:14.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:13 smithi107 ceph-mon[30122]: scrub summary: idle+waiting paths [/] 2024-05-09T08:48:14.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:13 smithi157 ceph-mon[33614]: scrub queued for path: / 2024-05-09T08:48:14.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:13 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:48:14.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:13 smithi157 ceph-mon[33614]: scrub summary: active paths [/] 2024-05-09T08:48:14.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:13 smithi157 ceph-mon[33614]: scrub summary: idle+waiting paths [/] 2024-05-09T08:48:14.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:13 smithi110 ceph-mon[33622]: scrub queued for path: / 2024-05-09T08:48:14.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:13 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:48:14.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:13 smithi110 ceph-mon[33622]: scrub summary: active paths [/] 2024-05-09T08:48:14.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:13 smithi110 ceph-mon[33622]: scrub summary: idle+waiting paths [/] 2024-05-09T08:48:14.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.362+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/3950379882 --> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fdf0c000bc0 con 0x7fdf180a2fd0 2024-05-09T08:48:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.419+0000 7fdf25ffb640 1 -- 172.21.15.107:0/3950379882 <== mgr.15483 v2:172.21.15.107:6800/165281804 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fdf0c000bc0 con 0x7fdf180a2fd0 2024-05-09T08:48:14.419 INFO:teuthology.orchestra.run.smithi107.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-09T00:00:00", "created": "2024-05-09T08:33:29", "first": "2024-05-09T08:34:00", "last": "2024-05-09T08:48:00", "last_pruned": "2024-05-09T08:48:00", "created_count": 15, "pruned_count": 9, "active": true} 2024-05-09T08:48:14.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.422+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/3950379882 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf180a2fd0 msgr2=0x7fdf180a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.423+0000 7fdf2ebb7640 1 --2- 172.21.15.107:0/3950379882 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf180a2fd0 0x7fdf180a5490 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdf14009990 tx=0x7fdf14009110 comp rx=0 tx=0).stop 2024-05-09T08:48:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.423+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/3950379882 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf281010f0 msgr2=0x7fdf28109550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.423+0000 7fdf2ebb7640 1 --2- 172.21.15.107:0/3950379882 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf281010f0 0x7fdf28109550 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00b4a0 tx=0x7fdf1c00b8a0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.423+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/3950379882 shutdown_connections 2024-05-09T08:48:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.423+0000 7fdf2ebb7640 1 --2- 172.21.15.107:0/3950379882 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdf180a2fd0 0x7fdf180a5490 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.423+0000 7fdf2ebb7640 1 --2- 172.21.15.107:0/3950379882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf28106740 0x7fdf281a21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.423+0000 7fdf2ebb7640 1 --2- 172.21.15.107:0/3950379882 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf281010f0 0x7fdf28109550 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.423+0000 7fdf2ebb7640 1 --2- 172.21.15.107:0/3950379882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf28100730 0x7fdf2810d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.423+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/3950379882 >> 172.21.15.107:0/3950379882 conn(0x7fdf28077850 msgr2=0x7fdf280ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.423+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/3950379882 shutdown_connections 2024-05-09T08:48:14.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.423+0000 7fdf2ebb7640 1 -- 172.21.15.107:0/3950379882 wait complete. 2024-05-09T08:48:14.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-09T08:48:14.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.615+0000 7fcaca963640 1 -- 172.21.15.107:0/3324638350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcac4110020 msgr2=0x7fcac4108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.615+0000 7fcaca963640 1 --2- 172.21.15.107:0/3324638350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcac4110020 0x7fcac4108830 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fcabc009930 tx=0x7fcabc02a510 comp rx=0 tx=0).stop 2024-05-09T08:48:14.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.617+0000 7fcaca963640 1 -- 172.21.15.107:0/3324638350 shutdown_connections 2024-05-09T08:48:14.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.617+0000 7fcaca963640 1 --2- 172.21.15.107:0/3324638350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcac411b010 0x7fcac411d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.617+0000 7fcaca963640 1 --2- 172.21.15.107:0/3324638350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcac4110400 0x7fcac4108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.617+0000 7fcaca963640 1 --2- 172.21.15.107:0/3324638350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcac4110020 0x7fcac4108830 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.617+0000 7fcaca963640 1 -- 172.21.15.107:0/3324638350 >> 172.21.15.107:0/3324638350 conn(0x7fcac406c900 msgr2=0x7fcac406cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:14.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.622+0000 7fcaca963640 1 -- 172.21.15.107:0/3324638350 shutdown_connections 2024-05-09T08:48:14.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.623+0000 7fcaca963640 1 -- 172.21.15.107:0/3324638350 wait complete. 2024-05-09T08:48:14.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.623+0000 7fcaca963640 1 Processor -- start 2024-05-09T08:48:14.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.625+0000 7fcaca963640 1 -- start start 2024-05-09T08:48:14.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.625+0000 7fcaca963640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcac4110020 0x7fcac41a7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:14.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.625+0000 7fcaca963640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcac411b010 0x7fcac41a7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:14.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.625+0000 7fcaca963640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcac41ab8c0 0x7fcac41abd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:14.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.625+0000 7fcaca963640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcac4120a30 con 0x7fcac41ab8c0 2024-05-09T08:48:14.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.625+0000 7fcaca963640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcac41208b0 con 0x7fcac4110020 2024-05-09T08:48:14.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.625+0000 7fcaca963640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcac4120bb0 con 0x7fcac411b010 2024-05-09T08:48:14.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.626+0000 7fcac37fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcac411b010 0x7fcac41a7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:14.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.626+0000 7fcac37fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcac411b010 0x7fcac41a7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.107:45650/0 (socket says 172.21.15.107:45650) 2024-05-09T08:48:14.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.626+0000 7fcac37fe640 1 -- 172.21.15.107:0/3672672981 learned_addr learned my addr 172.21.15.107:0/3672672981 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:14.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.626+0000 7fcac37fe640 1 -- 172.21.15.107:0/3672672981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcac4110020 msgr2=0x7fcac41a7000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T08:48:14.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.626+0000 7fcac37fe640 1 --2- 172.21.15.107:0/3672672981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcac4110020 0x7fcac41a7000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.626+0000 7fcac37fe640 1 -- 172.21.15.107:0/3672672981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcac41ab8c0 msgr2=0x7fcac41abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:14.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.626+0000 7fcac37fe640 1 --2- 172.21.15.107:0/3672672981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcac41ab8c0 0x7fcac41abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:14.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.626+0000 7fcac37fe640 1 -- 172.21.15.107:0/3672672981 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac41ac3b0 con 0x7fcac411b010 2024-05-09T08:48:14.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.626+0000 7fcac37fe640 1 --2- 172.21.15.107:0/3672672981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcac411b010 0x7fcac41a7540 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcab800b470 tx=0x7fcab800b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:14.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.631+0000 7fcac17fa640 1 -- 172.21.15.107:0/3672672981 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab8015020 con 0x7fcac411b010 2024-05-09T08:48:14.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.631+0000 7fcac17fa640 1 -- 172.21.15.107:0/3672672981 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcab8011d40 con 0x7fcac411b010 2024-05-09T08:48:14.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.631+0000 7fcac17fa640 1 -- 172.21.15.107:0/3672672981 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab800f340 con 0x7fcac411b010 2024-05-09T08:48:14.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.633+0000 7fcaca963640 1 -- 172.21.15.107:0/3672672981 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcac41ac6a0 con 0x7fcac411b010 2024-05-09T08:48:14.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.633+0000 7fcaca963640 1 -- 172.21.15.107:0/3672672981 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcac4110ac0 con 0x7fcac411b010 2024-05-09T08:48:14.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.635+0000 7fcaca963640 1 -- 172.21.15.107:0/3672672981 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcac404f5f0 con 0x7fcac411b010 2024-05-09T08:48:14.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.644+0000 7fcac17fa640 1 -- 172.21.15.107:0/3672672981 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fcab8018020 con 0x7fcac411b010 2024-05-09T08:48:14.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.645+0000 7fcac17fa640 1 --2- 172.21.15.107:0/3672672981 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcaac0a2fd0 0x7fcaac0a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:14.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.646+0000 7fcac17fa640 1 -- 172.21.15.107:0/3672672981 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9555+0+0 (secure 0 0 0) 0x7fcab80cf3b0 con 0x7fcac411b010 2024-05-09T08:48:14.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.646+0000 7fcac17fa640 1 -- 172.21.15.107:0/3672672981 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcab801f020 con 0x7fcac411b010 2024-05-09T08:48:14.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.646+0000 7fcac3fff640 1 --2- 172.21.15.107:0/3672672981 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcaac0a2fd0 0x7fcaac0a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:14.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:14.647+0000 7fcac3fff640 1 --2- 172.21.15.107:0/3672672981 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcaac0a2fd0 0x7fcaac0a5490 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcabc005c20 tx=0x7fcabc005fd0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:15.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:14 smithi107 ceph-mon[30122]: pgmap v467: 129 pgs: 11 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 85 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 318 KiB/s rd, 351 KiB/s wr, 75 op/s 2024-05-09T08:48:15.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:14 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:48:15.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:15.050+0000 7fcaca963640 1 -- 172.21.15.107:0/3672672981 --> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fcac4110650 con 0x7fcaac0a2fd0 2024-05-09T08:48:15.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:14 smithi157 ceph-mon[33614]: pgmap v467: 129 pgs: 11 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 85 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 318 KiB/s rd, 351 KiB/s wr, 75 op/s 2024-05-09T08:48:15.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:14 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:48:15.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:15.118+0000 7fcac17fa640 1 -- 172.21.15.107:0/3672672981 <== mgr.15483 v2:172.21.15.107:6800/165281804 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fcac4110650 con 0x7fcaac0a2fd0 2024-05-09T08:48:15.118 INFO:teuthology.orchestra.run.smithi107.stdout:/ 1m 6m3h 2024-05-09T08:48:15.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:14 smithi110 ceph-mon[33622]: pgmap v467: 129 pgs: 11 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 85 active+clean; 16 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 318 KiB/s rd, 351 KiB/s wr, 75 op/s 2024-05-09T08:48:15.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:14 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:48:15.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:15.132+0000 7fcaca963640 1 -- 172.21.15.107:0/3672672981 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcaac0a2fd0 msgr2=0x7fcaac0a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:15.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:15.132+0000 7fcaca963640 1 --2- 172.21.15.107:0/3672672981 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcaac0a2fd0 0x7fcaac0a5490 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcabc005c20 tx=0x7fcabc005fd0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:15.132+0000 7fcaca963640 1 -- 172.21.15.107:0/3672672981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcac411b010 msgr2=0x7fcac41a7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:15.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:15.132+0000 7fcaca963640 1 --2- 172.21.15.107:0/3672672981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcac411b010 0x7fcac41a7540 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcab800b470 tx=0x7fcab800b930 comp rx=0 tx=0).stop 2024-05-09T08:48:15.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:15.134+0000 7fcaca963640 1 -- 172.21.15.107:0/3672672981 shutdown_connections 2024-05-09T08:48:15.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:15.134+0000 7fcaca963640 1 --2- 172.21.15.107:0/3672672981 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fcaac0a2fd0 0x7fcaac0a5490 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:15.134+0000 7fcaca963640 1 --2- 172.21.15.107:0/3672672981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcac41ab8c0 0x7fcac41abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:15.134+0000 7fcaca963640 1 --2- 172.21.15.107:0/3672672981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcac411b010 0x7fcac41a7540 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:15.134+0000 7fcaca963640 1 --2- 172.21.15.107:0/3672672981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcac4110020 0x7fcac41a7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:15.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:15.134+0000 7fcaca963640 1 -- 172.21.15.107:0/3672672981 >> 172.21.15.107:0/3672672981 conn(0x7fcac406c900 msgr2=0x7fcac410dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:15.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:15.135+0000 7fcaca963640 1 -- 172.21.15.107:0/3672672981 shutdown_connections 2024-05-09T08:48:15.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:15.138+0000 7fcaca963640 1 -- 172.21.15.107:0/3672672981 wait complete. 2024-05-09T08:48:15.159 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-05-09T08:48:15.169 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-05-09T08:48:16.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:15 smithi107 ceph-mon[30122]: from='client.28319 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:16.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:15 smithi157 ceph-mon[33614]: from='client.28319 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:16.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:15 smithi110 ceph-mon[33622]: from='client.28319 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:17.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:16 smithi107 ceph-mon[30122]: from='client.28531 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:17.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:16 smithi107 ceph-mon[30122]: pgmap v468: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 15 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 311 KiB/s rd, 330 KiB/s wr, 80 op/s 2024-05-09T08:48:17.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:16 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:17.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:16 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:48:17.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:16 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:48:17.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:16 smithi107 ceph-mon[30122]: scrub summary: idle 2024-05-09T08:48:17.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:16 smithi157 ceph-mon[33614]: from='client.28531 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:17.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:16 smithi157 ceph-mon[33614]: pgmap v468: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 15 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 311 KiB/s rd, 330 KiB/s wr, 80 op/s 2024-05-09T08:48:17.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:16 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:17.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:16 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:48:17.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:16 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:48:17.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:16 smithi157 ceph-mon[33614]: scrub summary: idle 2024-05-09T08:48:17.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:16 smithi110 ceph-mon[33622]: from='client.28531 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T08:48:17.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:16 smithi110 ceph-mon[33622]: pgmap v468: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 15 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 311 KiB/s rd, 330 KiB/s wr, 80 op/s 2024-05-09T08:48:17.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:16 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:48:17.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:16 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:48:17.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:16 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:48:17.122 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:16 smithi110 ceph-mon[33622]: scrub summary: idle 2024-05-09T08:48:19.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:18 smithi107 ceph-mon[30122]: pgmap v469: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 15 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 300 KiB/s rd, 316 KiB/s wr, 78 op/s 2024-05-09T08:48:19.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:18 smithi157 ceph-mon[33614]: pgmap v469: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 15 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 300 KiB/s rd, 316 KiB/s wr, 78 op/s 2024-05-09T08:48:19.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:18 smithi110 ceph-mon[33622]: pgmap v469: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 15 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 300 KiB/s rd, 316 KiB/s wr, 78 op/s 2024-05-09T08:48:21.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:20 smithi107 ceph-mon[30122]: pgmap v470: 129 pgs: 129 active+clean; 15 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 320 KiB/s rd, 321 KiB/s wr, 168 op/s 2024-05-09T08:48:21.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:20 smithi157 ceph-mon[33614]: pgmap v470: 129 pgs: 129 active+clean; 15 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 320 KiB/s rd, 321 KiB/s wr, 168 op/s 2024-05-09T08:48:21.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:20 smithi110 ceph-mon[33622]: pgmap v470: 129 pgs: 129 active+clean; 15 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 320 KiB/s rd, 321 KiB/s wr, 168 op/s 2024-05-09T08:48:23.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:22 smithi107 ceph-mon[30122]: pgmap v471: 129 pgs: 129 active+clean; 15 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 236 KiB/s rd, 213 KiB/s wr, 174 op/s 2024-05-09T08:48:23.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:22 smithi107 ceph-mon[30122]: osdmap e113: 12 total, 12 up, 12 in 2024-05-09T08:48:23.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:22 smithi157 ceph-mon[33614]: pgmap v471: 129 pgs: 129 active+clean; 15 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 236 KiB/s rd, 213 KiB/s wr, 174 op/s 2024-05-09T08:48:23.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:22 smithi157 ceph-mon[33614]: osdmap e113: 12 total, 12 up, 12 in 2024-05-09T08:48:23.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:22 smithi110 ceph-mon[33622]: pgmap v471: 129 pgs: 129 active+clean; 15 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 236 KiB/s rd, 213 KiB/s wr, 174 op/s 2024-05-09T08:48:23.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:22 smithi110 ceph-mon[33622]: osdmap e113: 12 total, 12 up, 12 in 2024-05-09T08:48:24.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:23 smithi157 ceph-mon[33614]: pgmap v473: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 85 KiB/s rd, 67 KiB/s wr, 279 op/s 2024-05-09T08:48:24.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:23 smithi110 ceph-mon[33622]: pgmap v473: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 85 KiB/s rd, 67 KiB/s wr, 279 op/s 2024-05-09T08:48:24.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:23 smithi107 ceph-mon[30122]: pgmap v473: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 85 KiB/s rd, 67 KiB/s wr, 279 op/s 2024-05-09T08:48:26.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:26 smithi107 ceph-mon[30122]: pgmap v474: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 80 KiB/s rd, 62 KiB/s wr, 265 op/s 2024-05-09T08:48:26.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:26 smithi157 ceph-mon[33614]: pgmap v474: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 80 KiB/s rd, 62 KiB/s wr, 265 op/s 2024-05-09T08:48:26.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:26 smithi110 ceph-mon[33622]: pgmap v474: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 80 KiB/s rd, 62 KiB/s wr, 265 op/s 2024-05-09T08:48:28.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:28 smithi107 ceph-mon[30122]: pgmap v475: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 80 KiB/s rd, 62 KiB/s wr, 285 op/s 2024-05-09T08:48:28.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:28 smithi157 ceph-mon[33614]: pgmap v475: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 80 KiB/s rd, 62 KiB/s wr, 285 op/s 2024-05-09T08:48:28.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:28 smithi110 ceph-mon[33622]: pgmap v475: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 80 KiB/s rd, 62 KiB/s wr, 285 op/s 2024-05-09T08:48:30.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:30 smithi107 ceph-mon[30122]: pgmap v476: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.7 KiB/s wr, 214 op/s 2024-05-09T08:48:30.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:30 smithi157 ceph-mon[33614]: pgmap v476: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.7 KiB/s wr, 214 op/s 2024-05-09T08:48:30.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:30 smithi110 ceph-mon[33622]: pgmap v476: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.7 KiB/s wr, 214 op/s 2024-05-09T08:48:32.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:32 smithi107 ceph-mon[30122]: pgmap v477: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 6.0 KiB/s wr, 199 op/s 2024-05-09T08:48:32.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:32 smithi157 ceph-mon[33614]: pgmap v477: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 6.0 KiB/s wr, 199 op/s 2024-05-09T08:48:32.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:32 smithi110 ceph-mon[33622]: pgmap v477: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 6.0 KiB/s wr, 199 op/s 2024-05-09T08:48:34.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:34 smithi107 ceph-mon[30122]: pgmap v478: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 10 KiB/s rd, 7.4 KiB/s wr, 162 op/s 2024-05-09T08:48:34.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:34 smithi157 ceph-mon[33614]: pgmap v478: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 10 KiB/s rd, 7.4 KiB/s wr, 162 op/s 2024-05-09T08:48:34.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:34 smithi110 ceph-mon[33622]: pgmap v478: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 10 KiB/s rd, 7.4 KiB/s wr, 162 op/s 2024-05-09T08:48:36.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:36 smithi107 ceph-mon[30122]: pgmap v479: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.5 KiB/s wr, 70 op/s 2024-05-09T08:48:36.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:36 smithi157 ceph-mon[33614]: pgmap v479: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.5 KiB/s wr, 70 op/s 2024-05-09T08:48:36.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:36 smithi110 ceph-mon[33622]: pgmap v479: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.5 KiB/s wr, 70 op/s 2024-05-09T08:48:38.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:38 smithi107 ceph-mon[30122]: pgmap v480: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.5 KiB/s wr, 64 op/s 2024-05-09T08:48:38.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:38 smithi157 ceph-mon[33614]: pgmap v480: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.5 KiB/s wr, 64 op/s 2024-05-09T08:48:38.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:38 smithi110 ceph-mon[33622]: pgmap v480: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.5 KiB/s wr, 64 op/s 2024-05-09T08:48:40.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:40 smithi107 ceph-mon[30122]: pgmap v481: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.5 KiB/s wr, 47 op/s 2024-05-09T08:48:40.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:40 smithi157 ceph-mon[33614]: pgmap v481: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.5 KiB/s wr, 47 op/s 2024-05-09T08:48:40.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:40 smithi110 ceph-mon[33622]: pgmap v481: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.5 KiB/s wr, 47 op/s 2024-05-09T08:48:42.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:42 smithi107 ceph-mon[30122]: pgmap v482: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-05-09T08:48:42.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:42 smithi157 ceph-mon[33614]: pgmap v482: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-05-09T08:48:42.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:42 smithi110 ceph-mon[33622]: pgmap v482: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-05-09T08:48:43.372 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T08:48:43.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T08:48:43.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.488+0000 7f235d8cd640 1 -- 172.21.15.107:0/4163719027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2358100760 msgr2=0x7f2358100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.488+0000 7f235d8cd640 1 --2- 172.21.15.107:0/4163719027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2358100760 0x7f2358100be0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f2340009930 tx=0x7f234002a420 comp rx=0 tx=0).stop 2024-05-09T08:48:43.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.488+0000 7f235d8cd640 1 -- 172.21.15.107:0/4163719027 shutdown_connections 2024-05-09T08:48:43.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.488+0000 7f235d8cd640 1 --2- 172.21.15.107:0/4163719027 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2358101120 0x7f235810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.488+0000 7f235d8cd640 1 --2- 172.21.15.107:0/4163719027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2358100760 0x7f2358100be0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.488+0000 7f235d8cd640 1 --2- 172.21.15.107:0/4163719027 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2358106720 0x7f2358106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.488+0000 7f235d8cd640 1 -- 172.21.15.107:0/4163719027 >> 172.21.15.107:0/4163719027 conn(0x7f2358077890 msgr2=0x7f2358077ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:43.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.488+0000 7f235d8cd640 1 -- 172.21.15.107:0/4163719027 shutdown_connections 2024-05-09T08:48:43.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.488+0000 7f235d8cd640 1 -- 172.21.15.107:0/4163719027 wait complete. 2024-05-09T08:48:43.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f235d8cd640 1 Processor -- start 2024-05-09T08:48:43.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f235d8cd640 1 -- start start 2024-05-09T08:48:43.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f235d8cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2358100760 0x7f2358198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f235d8cd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2358101120 0x7f23581985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f235d8cd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2358106720 0x7f235819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f235d8cd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f23581109e0 con 0x7f2358100760 2024-05-09T08:48:43.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f235d8cd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2358110860 con 0x7f2358106720 2024-05-09T08:48:43.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f235d8cd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2358110b60 con 0x7f2358101120 2024-05-09T08:48:43.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f2356ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2358100760 0x7f2358198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f2356ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2358100760 0x7f2358198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56926/0 (socket says 172.21.15.107:56926) 2024-05-09T08:48:43.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f2356ffd640 1 -- 172.21.15.107:0/3514760718 learned_addr learned my addr 172.21.15.107:0/3514760718 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:43.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f23577fe640 1 --2- 172.21.15.107:0/3514760718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2358106720 0x7f235819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f234ffff640 1 --2- 172.21.15.107:0/3514760718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2358101120 0x7f23581985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f2356ffd640 1 -- 172.21.15.107:0/3514760718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2358101120 msgr2=0x7f23581985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f2356ffd640 1 --2- 172.21.15.107:0/3514760718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2358101120 0x7f23581985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f2356ffd640 1 -- 172.21.15.107:0/3514760718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2358106720 msgr2=0x7f235819c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f2356ffd640 1 --2- 172.21.15.107:0/3514760718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2358106720 0x7f235819c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f2356ffd640 1 -- 172.21.15.107:0/3514760718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f235819d080 con 0x7f2358100760 2024-05-09T08:48:43.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.489+0000 7f234ffff640 1 --2- 172.21.15.107:0/3514760718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2358101120 0x7f23581985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:43.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.490+0000 7f2356ffd640 1 --2- 172.21.15.107:0/3514760718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2358100760 0x7f2358198090 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f234800d770 tx=0x7f234800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:43.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.490+0000 7f235c8cb640 1 -- 172.21.15.107:0/3514760718 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348017070 con 0x7f2358100760 2024-05-09T08:48:43.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.490+0000 7f235c8cb640 1 -- 172.21.15.107:0/3514760718 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f23480054d0 con 0x7f2358100760 2024-05-09T08:48:43.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.490+0000 7f235d8cd640 1 -- 172.21.15.107:0/3514760718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f235819d370 con 0x7f2358100760 2024-05-09T08:48:43.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.490+0000 7f235c8cb640 1 -- 172.21.15.107:0/3514760718 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348004cf0 con 0x7f2358100760 2024-05-09T08:48:43.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.490+0000 7f235d8cd640 1 -- 172.21.15.107:0/3514760718 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23581a4c00 con 0x7f2358100760 2024-05-09T08:48:43.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.493+0000 7f235c8cb640 1 -- 172.21.15.107:0/3514760718 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f2348004e90 con 0x7f2358100760 2024-05-09T08:48:43.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.493+0000 7f23605dc740 1 Processor -- start 2024-05-09T08:48:43.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.493+0000 7f23605dc740 1 -- start start 2024-05-09T08:48:43.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.493+0000 7f23605dc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138160 0x558c4e138540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.493+0000 7f23605dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138a80 0x558c4e147140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.493+0000 7f23605dc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e147680 0x558c4e149a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.493+0000 7f23605dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558c4e09dd00 con 0x558c4e138a80 2024-05-09T08:48:43.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.493+0000 7f23605dc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558c4e0454c0 con 0x558c4e138160 2024-05-09T08:48:43.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.493+0000 7f23605dc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x558c4e099cc0 con 0x558c4e147680 2024-05-09T08:48:43.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.493+0000 7f2356ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138a80 0x558c4e147140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.493+0000 7f23577fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138160 0x558c4e138540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.493+0000 7f2356ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138a80 0x558c4e147140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56942/0 (socket says 172.21.15.107:56942) 2024-05-09T08:48:43.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.493+0000 7f2356ffd640 1 -- 172.21.15.107:0/4191646727 learned_addr learned my addr 172.21.15.107:0/4191646727 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:43.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.494+0000 7f234ffff640 1 --2- 172.21.15.107:0/4191646727 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e147680 0x558c4e149a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.494+0000 7f235c8cb640 1 --2- 172.21.15.107:0/3514760718 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f23440a2d00 0x7f23440a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.494+0000 7f2356ffd640 1 -- 172.21.15.107:0/4191646727 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e147680 msgr2=0x558c4e149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:43.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.494+0000 7f2356ffd640 1 --2- 172.21.15.107:0/4191646727 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e147680 0x558c4e149a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.494+0000 7f2356ffd640 1 -- 172.21.15.107:0/4191646727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138160 msgr2=0x558c4e138540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:43.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.494+0000 7f2356ffd640 1 --2- 172.21.15.107:0/4191646727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138160 0x558c4e138540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.494+0000 7f2356ffd640 1 -- 172.21.15.107:0/4191646727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c4de7b7d0 con 0x558c4e138a80 2024-05-09T08:48:43.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.494+0000 7f23577fe640 1 --2- 172.21.15.107:0/4191646727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138160 0x558c4e138540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T08:48:43.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.494+0000 7f23577fe640 1 --2- 172.21.15.107:0/3514760718 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f23440a2d00 0x7f23440a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.494+0000 7f235c8cb640 1 -- 172.21.15.107:0/3514760718 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7f23480cf280 con 0x7f2358100760 2024-05-09T08:48:43.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.494+0000 7f2356ffd640 1 --2- 172.21.15.107:0/4191646727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138a80 0x558c4e147140 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f23480cc590 tx=0x7f2348065f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee5fedd77e73cd71 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:43.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.494+0000 7f234dffb640 1 -- 172.21.15.107:0/4191646727 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348020020 con 0x558c4e138a80 2024-05-09T08:48:43.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.494+0000 7f234dffb640 1 -- 172.21.15.107:0/4191646727 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2348009c30 con 0x558c4e138a80 2024-05-09T08:48:43.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.494+0000 7f234dffb640 1 -- 172.21.15.107:0/4191646727 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234809af30 con 0x558c4e138a80 2024-05-09T08:48:43.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.495+0000 7f23605dc740 1 -- 172.21.15.107:0/4191646727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138a80 msgr2=0x558c4e147140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.495+0000 7f23605dc740 1 --2- 172.21.15.107:0/4191646727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138a80 0x558c4e147140 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f23480cc590 tx=0x7f2348065f70 comp rx=0 tx=0).stop 2024-05-09T08:48:43.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.495+0000 7f23577fe640 1 --2- 172.21.15.107:0/3514760718 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f23440a2d00 0x7f23440a51c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f23500114c0 tx=0x7f2350004a40 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:43.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.495+0000 7f23605dc740 1 -- 172.21.15.107:0/4191646727 shutdown_connections 2024-05-09T08:48:43.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.495+0000 7f23605dc740 1 --2- 172.21.15.107:0/4191646727 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e147680 0x558c4e149a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.495+0000 7f23605dc740 1 --2- 172.21.15.107:0/4191646727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138a80 0x558c4e147140 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.495+0000 7f23605dc740 1 --2- 172.21.15.107:0/4191646727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138160 0x558c4e138540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.495+0000 7f23605dc740 1 -- 172.21.15.107:0/4191646727 >> 172.21.15.107:0/4191646727 conn(0x558c4e1345a0 msgr2=0x558c4e134980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.495+0000 7f23605dc740 1 -- 172.21.15.107:0/4191646727 shutdown_connections 2024-05-09T08:48:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.495+0000 7f23605dc740 1 -- 172.21.15.107:0/4191646727 wait complete. 2024-05-09T08:48:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.495+0000 7f23605dc740 1 -- start start 2024-05-09T08:48:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.495+0000 7f23605dc740 1 Processor -- start 2024-05-09T08:48:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f23605dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138160 0x558c4e1445a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f23605dc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138a80 0x558c4e144ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f23605dc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e147680 0x558c4e145020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f23605dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558c4e14a400 con 0x558c4e138160 2024-05-09T08:48:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f23605dc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558c4e058b90 con 0x558c4e138a80 2024-05-09T08:48:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f23605dc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x558c4e14a580 con 0x558c4e147680 2024-05-09T08:48:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f23605dc740 10 client.0 _ll_register_callbacks cb 0x558c4e094da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f2356ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138160 0x558c4e1445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f2356ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138160 0x558c4e1445a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56954/0 (socket says 172.21.15.107:56954) 2024-05-09T08:48:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f2356ffd640 1 -- 172.21.15.107:0/1321406008 learned_addr learned my addr 172.21.15.107:0/1321406008 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f23577fe640 1 --2- 172.21.15.107:0/1321406008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e147680 0x558c4e145020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f234ffff640 1 --2- 172.21.15.107:0/1321406008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138a80 0x558c4e144ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f2356ffd640 1 -- 172.21.15.107:0/1321406008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e147680 msgr2=0x558c4e145020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f2356ffd640 1 --2- 172.21.15.107:0/1321406008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e147680 0x558c4e145020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f2356ffd640 1 -- 172.21.15.107:0/1321406008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138a80 msgr2=0x558c4e144ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f2356ffd640 1 --2- 172.21.15.107:0/1321406008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138a80 0x558c4e144ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f2356ffd640 1 -- 172.21.15.107:0/1321406008 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c4ddb13f0 con 0x558c4e138160 2024-05-09T08:48:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f234ffff640 1 --2- 172.21.15.107:0/1321406008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138a80 0x558c4e144ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f2356ffd640 1 --2- 172.21.15.107:0/1321406008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138160 0x558c4e1445a0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f2348097850 tx=0x7f23480cf5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9edfc69aef532850 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f2336ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:48:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f2336ffd640 1 -- 172.21.15.107:0/1321406008 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348020040 con 0x558c4e138160 2024-05-09T08:48:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f2336ffd640 1 -- 172.21.15.107:0/1321406008 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7f23480996f0 con 0x558c4e138160 2024-05-09T08:48:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f23605dc740 1 -- 172.21.15.107:0/1321406008 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558c4e0ab430 con 0x558c4e138160 2024-05-09T08:48:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f2336ffd640 1 -- 172.21.15.107:0/1321406008 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2348005180 con 0x558c4e138160 2024-05-09T08:48:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.496+0000 7f23605dc740 10 client.19407 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.497+0000 7f23605dc740 1 -- 172.21.15.107:0/1321406008 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558c4dd710f0 con 0x558c4e138160 2024-05-09T08:48:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.497+0000 7f2336ffd640 1 -- 172.21.15.107:0/1321406008 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x558c4e0ab430 con 0x558c4e138160 2024-05-09T08:48:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.497+0000 7f2336ffd640 1 -- 172.21.15.107:0/1321406008 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f23480cc8a0 con 0x558c4e138160 2024-05-09T08:48:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.497+0000 7f23605dc740 10 client.19407 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.497+0000 7f23605dc740 10 client.19407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:48:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.497+0000 7f23605dc740 20 client.19407 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.497+0000 7f23605dc740 1 --2- 172.21.15.107:0/1321406008 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558c4e157830 0x558c4e159c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.497+0000 7f23605dc740 4 client.19407 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:48:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.497+0000 7f23605dc740 1 -- 172.21.15.107:0/1321406008 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c4e0a5780 con 0x558c4e157830 2024-05-09T08:48:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.497+0000 7f234ffff640 1 --2- 172.21.15.107:0/1321406008 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558c4e157830 0x558c4e159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.498+0000 7f2336ffd640 10 client.19407 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.498+0000 7f234ffff640 1 --2- 172.21.15.107:0/1321406008 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558c4e157830 0x558c4e159c20 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29e1e501337c638c server_cookie=523231faf2173e77 in_seq=0 out_seq=0 2024-05-09T08:48:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.498+0000 7f2336ffd640 1 -- 172.21.15.107:0/1321406008 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c4e0a5780 con 0x558c4e157830 2024-05-09T08:48:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.498+0000 7f2336ffd640 10 client.19407 handle_command_reply: tid=0 2024-05-09T08:48:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.498+0000 7f23605dc740 1 client.19407 shutdown 2024-05-09T08:48:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.499+0000 7f23605dc740 1 -- 172.21.15.107:0/1321406008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138160 msgr2=0x558c4e1445a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.499+0000 7f23605dc740 1 --2- 172.21.15.107:0/1321406008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138160 0x558c4e1445a0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f2348097850 tx=0x7f23480cf5f0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.499+0000 7f23605dc740 1 -- 172.21.15.107:0/1321406008 shutdown_connections 2024-05-09T08:48:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.499+0000 7f23605dc740 1 --2- 172.21.15.107:0/1321406008 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558c4e157830 0x558c4e159c20 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.499+0000 7f23605dc740 1 --2- 172.21.15.107:0/1321406008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e147680 0x558c4e145020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.499+0000 7f23605dc740 1 --2- 172.21.15.107:0/1321406008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138a80 0x558c4e144ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.499+0000 7f2336ffd640 0 client.19407 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.499+0000 7f23605dc740 1 --2- 172.21.15.107:0/1321406008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138160 0x558c4e1445a0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.499+0000 7f23605dc740 1 -- 172.21.15.107:0/1321406008 >> 172.21.15.107:0/1321406008 conn(0x558c4e1345a0 msgr2=0x558c4e13a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:43.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.499+0000 7f23605dc740 1 -- 172.21.15.107:0/1321406008 shutdown_connections 2024-05-09T08:48:43.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.499+0000 7f23605dc740 1 -- 172.21.15.107:0/1321406008 wait complete. 2024-05-09T08:48:43.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.499+0000 7f23605dc740 20 client.19407 trim_cache size 0 max 16384 2024-05-09T08:48:43.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f23605dc740 1 Processor -- start 2024-05-09T08:48:43.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f23605dc740 1 -- start start 2024-05-09T08:48:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f23605dc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e138160 0x558c4e152f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f23605dc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138a80 0x558c4e15cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f23605dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e147680 0x558c4e15d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f23605dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f23581af3e0 con 0x558c4e147680 2024-05-09T08:48:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f23605dc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f23581acc50 con 0x558c4e138a80 2024-05-09T08:48:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f23605dc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f23581aaac0 con 0x558c4e138160 2024-05-09T08:48:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f23577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e147680 0x558c4e15d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f23577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e147680 0x558c4e15d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56968/0 (socket says 172.21.15.107:56968) 2024-05-09T08:48:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f23577fe640 1 -- 172.21.15.107:0/2343361700 learned_addr learned my addr 172.21.15.107:0/2343361700 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f23577fe640 1 -- 172.21.15.107:0/2343361700 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e138160 msgr2=0x558c4e152f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f234ffff640 1 --2- 172.21.15.107:0/2343361700 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138a80 0x558c4e15cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f23577fe640 1 --2- 172.21.15.107:0/2343361700 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e138160 0x558c4e152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f23577fe640 1 -- 172.21.15.107:0/2343361700 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138a80 msgr2=0x558c4e15cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f23577fe640 1 --2- 172.21.15.107:0/2343361700 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138a80 0x558c4e15cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.516+0000 7f23577fe640 1 -- 172.21.15.107:0/2343361700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c4e1397f0 con 0x558c4e147680 2024-05-09T08:48:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.517+0000 7f23577fe640 1 --2- 172.21.15.107:0/2343361700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e147680 0x558c4e15d180 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f2350012890 tx=0x7f2350014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1fc67499bd44219 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.517+0000 7f234d7fa640 1 -- 172.21.15.107:0/2343361700 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235001fdf0 con 0x558c4e147680 2024-05-09T08:48:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.517+0000 7f234d7fa640 1 -- 172.21.15.107:0/2343361700 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2350016070 con 0x558c4e147680 2024-05-09T08:48:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.517+0000 7f234d7fa640 1 -- 172.21.15.107:0/2343361700 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23500286a0 con 0x558c4e147680 2024-05-09T08:48:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.517+0000 7f23605dc740 1 -- 172.21.15.107:0/2343361700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e147680 msgr2=0x558c4e15d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.517+0000 7f23605dc740 1 --2- 172.21.15.107:0/2343361700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e147680 0x558c4e15d180 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f2350012890 tx=0x7f2350014000 comp rx=0 tx=0).stop 2024-05-09T08:48:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.517+0000 7f23605dc740 1 -- 172.21.15.107:0/2343361700 shutdown_connections 2024-05-09T08:48:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.517+0000 7f23605dc740 1 --2- 172.21.15.107:0/2343361700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e147680 0x558c4e15d180 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.517+0000 7f23605dc740 1 --2- 172.21.15.107:0/2343361700 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e138a80 0x558c4e15cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.517+0000 7f23605dc740 1 --2- 172.21.15.107:0/2343361700 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e138160 0x558c4e152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.517+0000 7f23605dc740 1 -- 172.21.15.107:0/2343361700 >> 172.21.15.107:0/2343361700 conn(0x558c4e1345a0 msgr2=0x558c4e152840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.517+0000 7f23605dc740 1 -- 172.21.15.107:0/2343361700 shutdown_connections 2024-05-09T08:48:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.517+0000 7f23605dc740 1 -- 172.21.15.107:0/2343361700 wait complete. 2024-05-09T08:48:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.518+0000 7f23605dc740 1 -- start start 2024-05-09T08:48:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.518+0000 7f23605dc740 1 Processor -- start 2024-05-09T08:48:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.518+0000 7f23605dc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e138160 0x558c4e15cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.518+0000 7f23605dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138a80 0x558c4e15d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.518+0000 7f23605dc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e147680 0x558c4e15a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.518+0000 7f23605dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f23581ab110 con 0x558c4e138a80 2024-05-09T08:48:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.518+0000 7f23605dc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f23581ac580 con 0x558c4e147680 2024-05-09T08:48:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.518+0000 7f23605dc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f23581ab7e0 con 0x558c4e138160 2024-05-09T08:48:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.518+0000 7f23605dc740 10 client.0 _ll_register_callbacks cb 0x7f23580688d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.518+0000 7f234ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138a80 0x558c4e15d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.518+0000 7f234ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138a80 0x558c4e15d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56980/0 (socket says 172.21.15.107:56980) 2024-05-09T08:48:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.518+0000 7f234ffff640 1 -- 172.21.15.107:0/78044391 learned_addr learned my addr 172.21.15.107:0/78044391 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.518+0000 7f23577fe640 1 --2- 172.21.15.107:0/78044391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e147680 0x558c4e15a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f2356ffd640 1 --2- 172.21.15.107:0/78044391 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e138160 0x558c4e15cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f234ffff640 1 -- 172.21.15.107:0/78044391 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e138160 msgr2=0x558c4e15cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f234ffff640 1 --2- 172.21.15.107:0/78044391 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e138160 0x558c4e15cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f234ffff640 1 -- 172.21.15.107:0/78044391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e147680 msgr2=0x558c4e15a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f234ffff640 1 --2- 172.21.15.107:0/78044391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e147680 0x558c4e15a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f234ffff640 1 -- 172.21.15.107:0/78044391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c4e15aa60 con 0x558c4e138a80 2024-05-09T08:48:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f23577fe640 1 --2- 172.21.15.107:0/78044391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e147680 0x558c4e15a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f234ffff640 1 --2- 172.21.15.107:0/78044391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138a80 0x558c4e15d170 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f2340031440 tx=0x7f2340009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b645ce026c64b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f2336ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:48:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f2336ffd640 1 -- 172.21.15.107:0/78044391 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2340035040 con 0x558c4e138a80 2024-05-09T08:48:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f2336ffd640 1 -- 172.21.15.107:0/78044391 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7f2340004310 con 0x558c4e138a80 2024-05-09T08:48:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f23605dc740 1 -- 172.21.15.107:0/78044391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2350016070 con 0x558c4e138a80 2024-05-09T08:48:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f2336ffd640 1 -- 172.21.15.107:0/78044391 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2340005040 con 0x558c4e138a80 2024-05-09T08:48:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f23605dc740 10 client.19413 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f23605dc740 1 -- 172.21.15.107:0/78044391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558c4e1bd680 con 0x558c4e138a80 2024-05-09T08:48:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f2336ffd640 1 -- 172.21.15.107:0/78044391 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2350016070 con 0x558c4e138a80 2024-05-09T08:48:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f2336ffd640 1 -- 172.21.15.107:0/78044391 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f2340004cc0 con 0x558c4e138a80 2024-05-09T08:48:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f23605dc740 10 client.19413 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f23605dc740 10 client.19413 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:48:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f23605dc740 20 client.19413 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f23605dc740 1 --2- 172.21.15.107:0/78044391 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558c4e1bde80 0x558c4e1c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f23605dc740 4 client.19413 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T08:48:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.519+0000 7f23605dc740 1 -- 172.21.15.107:0/78044391 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558c4e0226f0 con 0x558c4e1bde80 2024-05-09T08:48:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.520+0000 7f2356ffd640 1 --2- 172.21.15.107:0/78044391 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558c4e1bde80 0x558c4e1c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.520+0000 7f2336ffd640 10 client.19413 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.520+0000 7f2356ffd640 1 --2- 172.21.15.107:0/78044391 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558c4e1bde80 0x558c4e1c02a0 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=287b29c939fc9bec server_cookie=8018f3b14bec531a in_seq=0 out_seq=0 2024-05-09T08:48:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.520+0000 7f2336ffd640 1 -- 172.21.15.107:0/78044391 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558c4e0226f0 con 0x558c4e1bde80 2024-05-09T08:48:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.520+0000 7f2336ffd640 10 client.19413 handle_command_reply: tid=0 2024-05-09T08:48:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.520+0000 7f23605dc740 1 client.19413 shutdown 2024-05-09T08:48:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.520+0000 7f23605dc740 1 -- 172.21.15.107:0/78044391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138a80 msgr2=0x558c4e15d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.520+0000 7f23605dc740 1 --2- 172.21.15.107:0/78044391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138a80 0x558c4e15d170 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f2340031440 tx=0x7f2340009590 comp rx=0 tx=0).stop 2024-05-09T08:48:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.520+0000 7f23605dc740 1 -- 172.21.15.107:0/78044391 shutdown_connections 2024-05-09T08:48:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.520+0000 7f23605dc740 1 --2- 172.21.15.107:0/78044391 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x558c4e1bde80 0x558c4e1c02a0 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.520+0000 7f23605dc740 1 --2- 172.21.15.107:0/78044391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558c4e147680 0x558c4e15a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.520+0000 7f23605dc740 1 --2- 172.21.15.107:0/78044391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c4e138a80 0x558c4e15d170 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.520+0000 7f23605dc740 1 --2- 172.21.15.107:0/78044391 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558c4e138160 0x558c4e15cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.520+0000 7f2336ffd640 0 client.19413 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.520+0000 7f23605dc740 1 -- 172.21.15.107:0/78044391 >> 172.21.15.107:0/78044391 conn(0x558c4e1345a0 msgr2=0x558c4e1530d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.520+0000 7f23605dc740 1 -- 172.21.15.107:0/78044391 shutdown_connections 2024-05-09T08:48:43.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.521+0000 7f23605dc740 1 -- 172.21.15.107:0/78044391 wait complete. 2024-05-09T08:48:43.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.521+0000 7f23605dc740 20 client.19413 trim_cache size 0 max 16384 2024-05-09T08:48:43.528 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:48:43.528 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-05-09T08:48:43.528 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-05-09T08:48:43.528 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:48:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.521+0000 7f235d8cd640 1 -- 172.21.15.107:0/3514760718 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f23440a2d00 msgr2=0x7f23440a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.521+0000 7f235d8cd640 1 --2- 172.21.15.107:0/3514760718 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f23440a2d00 0x7f23440a51c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f23500114c0 tx=0x7f2350004a40 comp rx=0 tx=0).stop 2024-05-09T08:48:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.521+0000 7f235d8cd640 1 -- 172.21.15.107:0/3514760718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2358100760 msgr2=0x7f2358198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.521+0000 7f235d8cd640 1 --2- 172.21.15.107:0/3514760718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2358100760 0x7f2358198090 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f234800d770 tx=0x7f234800dc30 comp rx=0 tx=0).stop 2024-05-09T08:48:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.521+0000 7f235d8cd640 1 -- 172.21.15.107:0/3514760718 shutdown_connections 2024-05-09T08:48:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.521+0000 7f235d8cd640 1 --2- 172.21.15.107:0/3514760718 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f23440a2d00 0x7f23440a51c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.521+0000 7f235d8cd640 1 --2- 172.21.15.107:0/3514760718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2358106720 0x7f235819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.521+0000 7f235d8cd640 1 --2- 172.21.15.107:0/3514760718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2358101120 0x7f23581985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.521+0000 7f235d8cd640 1 --2- 172.21.15.107:0/3514760718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2358100760 0x7f2358198090 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.521+0000 7f235d8cd640 1 -- 172.21.15.107:0/3514760718 >> 172.21.15.107:0/3514760718 conn(0x7f2358077890 msgr2=0x7f2358069ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:43.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.521+0000 7f235d8cd640 1 -- 172.21.15.107:0/3514760718 shutdown_connections 2024-05-09T08:48:43.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.521+0000 7f235d8cd640 1 -- 172.21.15.107:0/3514760718 wait complete. 2024-05-09T08:48:43.532 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T08:48:43.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:48:43.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.633+0000 7f4b7f7b7640 1 -- 172.21.15.107:0/1877300893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b781010d0 msgr2=0x7f4b7810d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.633+0000 7f4b7f7b7640 1 --2- 172.21.15.107:0/1877300893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b781010d0 0x7f4b7810d5e0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f4b70009cd0 tx=0x7f4b7002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.633+0000 7f4b7f7b7640 1 -- 172.21.15.107:0/1877300893 shutdown_connections 2024-05-09T08:48:43.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.633+0000 7f4b7f7b7640 1 --2- 172.21.15.107:0/1877300893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b781010d0 0x7f4b7810d5e0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.633+0000 7f4b7f7b7640 1 --2- 172.21.15.107:0/1877300893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b78100710 0x7f4b78100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.633+0000 7f4b7f7b7640 1 --2- 172.21.15.107:0/1877300893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b78106720 0x7f4b78106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.633+0000 7f4b7f7b7640 1 -- 172.21.15.107:0/1877300893 >> 172.21.15.107:0/1877300893 conn(0x7f4b78077830 msgr2=0x7f4b78077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:43.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.634+0000 7f4b7f7b7640 1 -- 172.21.15.107:0/1877300893 shutdown_connections 2024-05-09T08:48:43.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.634+0000 7f4b7f7b7640 1 -- 172.21.15.107:0/1877300893 wait complete. 2024-05-09T08:48:43.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.634+0000 7f4b7f7b7640 1 Processor -- start 2024-05-09T08:48:43.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.634+0000 7f4b7f7b7640 1 -- start start 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.634+0000 7f4b7f7b7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b78100710 0x7f4b78197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.634+0000 7f4b7f7b7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b781010d0 0x7f4b781984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.634+0000 7f4b7f7b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b78106720 0x7f4b7819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.634+0000 7f4b7f7b7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4b781109a0 con 0x7f4b78106720 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.634+0000 7f4b7f7b7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4b78110820 con 0x7f4b78100710 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.634+0000 7f4b7f7b7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4b78110b20 con 0x7f4b781010d0 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.634+0000 7f4b7dd54640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b78106720 0x7f4b7819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.635+0000 7f4b7dd54640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b78106720 0x7f4b7819c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56986/0 (socket says 172.21.15.107:56986) 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.635+0000 7f4b7dd54640 1 -- 172.21.15.107:0/3390139377 learned_addr learned my addr 172.21.15.107:0/3390139377 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.635+0000 7f4b7d553640 1 --2- 172.21.15.107:0/3390139377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b78100710 0x7f4b78197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.635+0000 7f4b7cd52640 1 --2- 172.21.15.107:0/3390139377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b781010d0 0x7f4b781984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.635+0000 7f4b7dd54640 1 -- 172.21.15.107:0/3390139377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b781010d0 msgr2=0x7f4b781984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.635+0000 7f4b7dd54640 1 --2- 172.21.15.107:0/3390139377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b781010d0 0x7f4b781984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.635+0000 7f4b7dd54640 1 -- 172.21.15.107:0/3390139377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b78100710 msgr2=0x7f4b78197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.635+0000 7f4b7dd54640 1 --2- 172.21.15.107:0/3390139377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b78100710 0x7f4b78197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.635+0000 7f4b7dd54640 1 -- 172.21.15.107:0/3390139377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7819cf90 con 0x7f4b78106720 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.635+0000 7f4b7dd54640 1 --2- 172.21.15.107:0/3390139377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b78106720 0x7f4b7819c860 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f4b70005cc0 tx=0x7f4b70003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.635+0000 7f4b6e7fc640 1 -- 172.21.15.107:0/3390139377 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b70004450 con 0x7f4b78106720 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.635+0000 7f4b6e7fc640 1 -- 172.21.15.107:0/3390139377 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b70031c90 con 0x7f4b78106720 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.635+0000 7f4b7f7b7640 1 -- 172.21.15.107:0/3390139377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b7819d220 con 0x7f4b78106720 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.635+0000 7f4b6e7fc640 1 -- 172.21.15.107:0/3390139377 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7003b400 con 0x7f4b78106720 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.635+0000 7f4b7f7b7640 1 -- 172.21.15.107:0/3390139377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b781a4b10 con 0x7f4b78106720 2024-05-09T08:48:43.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.636+0000 7f4b7f7b7640 1 -- 172.21.15.107:0/3390139377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b78068650 con 0x7f4b78106720 2024-05-09T08:48:43.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.637+0000 7f4b6e7fc640 1 -- 172.21.15.107:0/3390139377 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f4b70004680 con 0x7f4b78106720 2024-05-09T08:48:43.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.638+0000 7f4b6e7fc640 1 --2- 172.21.15.107:0/3390139377 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4b640a2d00 0x7f4b640a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.638+0000 7f4b7d553640 1 --2- 172.21.15.107:0/3390139377 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4b640a2d00 0x7f4b640a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.638+0000 7f4b6e7fc640 1 -- 172.21.15.107:0/3390139377 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7f4b7002e030 con 0x7f4b78106720 2024-05-09T08:48:43.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.638+0000 7f4b7d553640 1 --2- 172.21.15.107:0/3390139377 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4b640a2d00 0x7f4b640a51c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4b7400bb60 tx=0x7f4b7400b390 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:43.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.642+0000 7f4b6e7fc640 1 -- 172.21.15.107:0/3390139377 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4b700bcf90 con 0x7f4b78106720 2024-05-09T08:48:43.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.860+0000 7f4b7f7b7640 1 -- 172.21.15.107:0/3390139377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b78109d10 con 0x7f4b78106720 2024-05-09T08:48:43.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.861+0000 7f4b6e7fc640 1 -- 172.21.15.107:0/3390139377 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11893 (secure 0 0 0) 0x7f4b700098b0 con 0x7f4b78106720 2024-05-09T08:48:43.861 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:48:43.861 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":28109,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/2116846059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2116846059},{"type":"v1","addr":"172.21.15.157:6835","nonce":2116846059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:48:10.313321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24418":{"gid":24418,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:48:43.862 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 31 2024-05-09T08:48:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.863+0000 7f4b7f7b7640 1 -- 172.21.15.107:0/3390139377 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4b640a2d00 msgr2=0x7f4b640a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.863+0000 7f4b7f7b7640 1 --2- 172.21.15.107:0/3390139377 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4b640a2d00 0x7f4b640a51c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4b7400bb60 tx=0x7f4b7400b390 comp rx=0 tx=0).stop 2024-05-09T08:48:43.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.863+0000 7f4b7f7b7640 1 -- 172.21.15.107:0/3390139377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b78106720 msgr2=0x7f4b7819c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.863+0000 7f4b7f7b7640 1 --2- 172.21.15.107:0/3390139377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b78106720 0x7f4b7819c860 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f4b70005cc0 tx=0x7f4b70003c20 comp rx=0 tx=0).stop 2024-05-09T08:48:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.863+0000 7f4b7f7b7640 1 -- 172.21.15.107:0/3390139377 shutdown_connections 2024-05-09T08:48:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.863+0000 7f4b7f7b7640 1 --2- 172.21.15.107:0/3390139377 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f4b640a2d00 0x7f4b640a51c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.863+0000 7f4b7f7b7640 1 --2- 172.21.15.107:0/3390139377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b78106720 0x7f4b7819c860 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.863+0000 7f4b7f7b7640 1 --2- 172.21.15.107:0/3390139377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b781010d0 0x7f4b781984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.863+0000 7f4b7f7b7640 1 --2- 172.21.15.107:0/3390139377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b78100710 0x7f4b78197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.863+0000 7f4b7f7b7640 1 -- 172.21.15.107:0/3390139377 >> 172.21.15.107:0/3390139377 conn(0x7f4b78077830 msgr2=0x7f4b78068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.863+0000 7f4b7f7b7640 1 -- 172.21.15.107:0/3390139377 shutdown_connections 2024-05-09T08:48:43.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.863+0000 7f4b7f7b7640 1 -- 172.21.15.107:0/3390139377 wait complete. 2024-05-09T08:48:43.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.977+0000 7fbffebb7640 1 -- 172.21.15.107:0/666207637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbff8100710 msgr2=0x7fbff8100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.977+0000 7fbffebb7640 1 --2- 172.21.15.107:0/666207637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbff8100710 0x7fbff8100b90 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009930 tx=0x7fbfec02a420 comp rx=0 tx=0).stop 2024-05-09T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.977+0000 7fbffebb7640 1 -- 172.21.15.107:0/666207637 shutdown_connections 2024-05-09T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.977+0000 7fbffebb7640 1 --2- 172.21.15.107:0/666207637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbff81010d0 0x7fbff810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.977+0000 7fbffebb7640 1 --2- 172.21.15.107:0/666207637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbff8100710 0x7fbff8100b90 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.977+0000 7fbffebb7640 1 --2- 172.21.15.107:0/666207637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbff8106720 0x7fbff8106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.977+0000 7fbffebb7640 1 -- 172.21.15.107:0/666207637 >> 172.21.15.107:0/666207637 conn(0x7fbff8077790 msgr2=0x7fbff8077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.977+0000 7fbffebb7640 1 -- 172.21.15.107:0/666207637 shutdown_connections 2024-05-09T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.977+0000 7fbffebb7640 1 -- 172.21.15.107:0/666207637 wait complete. 2024-05-09T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffebb7640 1 Processor -- start 2024-05-09T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffebb7640 1 -- start start 2024-05-09T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffebb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbff8100710 0x7fbff8198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffebb7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbff81010d0 0x7fbff81993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffebb7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbff8106720 0x7fbff81a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffebb7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81109e0 con 0x7fbff8100710 2024-05-09T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffebb7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbff8110860 con 0x7fbff81010d0 2024-05-09T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffebb7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbff8110b60 con 0x7fbff8106720 2024-05-09T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffc953640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbff8100710 0x7fbff8198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffc953640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbff8100710 0x7fbff8198e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57010/0 (socket says 172.21.15.107:57010) 2024-05-09T08:48:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffc953640 1 -- 172.21.15.107:0/915150447 learned_addr learned my addr 172.21.15.107:0/915150447 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffd154640 1 --2- 172.21.15.107:0/915150447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbff8106720 0x7fbff81a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbff7fff640 1 --2- 172.21.15.107:0/915150447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbff81010d0 0x7fbff81993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffc953640 1 -- 172.21.15.107:0/915150447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbff8106720 msgr2=0x7fbff81a3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffc953640 1 --2- 172.21.15.107:0/915150447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbff8106720 0x7fbff81a3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffc953640 1 -- 172.21.15.107:0/915150447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbff81010d0 msgr2=0x7fbff81993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffc953640 1 --2- 172.21.15.107:0/915150447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbff81010d0 0x7fbff81993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.978+0000 7fbffc953640 1 -- 172.21.15.107:0/915150447 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff81a4420 con 0x7fbff8100710 2024-05-09T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.979+0000 7fbffc953640 1 --2- 172.21.15.107:0/915150447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbff8100710 0x7fbff8198e60 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800d950 tx=0x7fbfe800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.979+0000 7fbff5ffb640 1 -- 172.21.15.107:0/915150447 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8017030 con 0x7fbff8100710 2024-05-09T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.979+0000 7fbff5ffb640 1 -- 172.21.15.107:0/915150447 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfe8004020 con 0x7fbff8100710 2024-05-09T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.979+0000 7fbff5ffb640 1 -- 172.21.15.107:0/915150447 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe80047a0 con 0x7fbff8100710 2024-05-09T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.979+0000 7fbffebb7640 1 -- 172.21.15.107:0/915150447 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff81a45f0 con 0x7fbff8100710 2024-05-09T08:48:43.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.979+0000 7fbffebb7640 1 -- 172.21.15.107:0/915150447 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff81a4b10 con 0x7fbff8100710 2024-05-09T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.981+0000 7fbff5ffb640 1 -- 172.21.15.107:0/915150447 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fbfe80041d0 con 0x7fbff8100710 2024-05-09T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.981+0000 7fc0019a1740 1 Processor -- start 2024-05-09T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.981+0000 7fc0019a1740 1 -- start start 2024-05-09T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fc0019a1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae10d330 0x563aae10d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fc0019a1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10dc50 0x563aae117de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbff5ffb640 1 --2- 172.21.15.107:0/915150447 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbfe00a2d00 0x7fbfe00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fc0019a1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae118320 0x563aae11a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fc0019a1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563aae072c40 con 0x563aae10dc50 2024-05-09T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fc0019a1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563aae01a580 con 0x563aae10d330 2024-05-09T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fc0019a1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563aae06ec00 con 0x563aae118320 2024-05-09T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbff7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10dc50 0x563aae117de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbffc953640 1 --2- 172.21.15.107:0/915150447 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbfe00a2d00 0x7fbfe00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbff7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10dc50 0x563aae117de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57022/0 (socket says 172.21.15.107:57022) 2024-05-09T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbff7fff640 1 -- 172.21.15.107:0/3785070043 learned_addr learned my addr 172.21.15.107:0/3785070043 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbff5ffb640 1 -- 172.21.15.107:0/915150447 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7fbfe80cf2e0 con 0x7fbff8100710 2024-05-09T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbffd154640 1 --2- 172.21.15.107:0/3785070043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae10d330 0x563aae10d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbffd154640 1 --2- 172.21.15.107:0/3785070043 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae118320 0x563aae11a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbff7fff640 1 -- 172.21.15.107:0/3785070043 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae118320 msgr2=0x563aae11a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:43.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbff7fff640 1 --2- 172.21.15.107:0/3785070043 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae118320 0x563aae11a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbff7fff640 1 -- 172.21.15.107:0/3785070043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae10d330 msgr2=0x563aae10d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:43.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbff7fff640 1 --2- 172.21.15.107:0/3785070043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae10d330 0x563aae10d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbff7fff640 1 -- 172.21.15.107:0/3785070043 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563aadd863f0 con 0x563aae10dc50 2024-05-09T08:48:43.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbffc953640 1 --2- 172.21.15.107:0/915150447 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbfe00a2d00 0x7fbfe00a51c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbfe80cc730 tx=0x7fbfe809e040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:43.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbff7fff640 1 --2- 172.21.15.107:0/3785070043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10dc50 0x563aae117de0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fbfec031440 tx=0x7fbfec032c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d1ffd3dca032df server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:43.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbfdeffd640 1 -- 172.21.15.107:0/3785070043 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec03f070 con 0x563aae10dc50 2024-05-09T08:48:43.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbfdeffd640 1 -- 172.21.15.107:0/3785070043 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfec032e20 con 0x563aae10dc50 2024-05-09T08:48:43.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.982+0000 7fbfdeffd640 1 -- 172.21.15.107:0/3785070043 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec035070 con 0x563aae10dc50 2024-05-09T08:48:43.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.983+0000 7fc0019a1740 1 -- 172.21.15.107:0/3785070043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10dc50 msgr2=0x563aae117de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.983+0000 7fc0019a1740 1 --2- 172.21.15.107:0/3785070043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10dc50 0x563aae117de0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fbfec031440 tx=0x7fbfec032c30 comp rx=0 tx=0).stop 2024-05-09T08:48:43.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.983+0000 7fc0019a1740 1 -- 172.21.15.107:0/3785070043 shutdown_connections 2024-05-09T08:48:43.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.983+0000 7fc0019a1740 1 --2- 172.21.15.107:0/3785070043 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae118320 0x563aae11a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.983+0000 7fc0019a1740 1 --2- 172.21.15.107:0/3785070043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10dc50 0x563aae117de0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.983+0000 7fc0019a1740 1 --2- 172.21.15.107:0/3785070043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae10d330 0x563aae10d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.983+0000 7fc0019a1740 1 -- 172.21.15.107:0/3785070043 >> 172.21.15.107:0/3785070043 conn(0x563aae109650 msgr2=0x563aae109a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.983+0000 7fc0019a1740 1 -- 172.21.15.107:0/3785070043 shutdown_connections 2024-05-09T08:48:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.983+0000 7fc0019a1740 1 -- 172.21.15.107:0/3785070043 wait complete. 2024-05-09T08:48:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.983+0000 7fc0019a1740 1 -- start start 2024-05-09T08:48:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fc0019a1740 1 Processor -- start 2024-05-09T08:48:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fc0019a1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10d330 0x563aae1152d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fc0019a1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae10dc50 0x563aae115810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fc0019a1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae118320 0x563aae115d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fc0019a1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563aae11b0b0 con 0x563aae10d330 2024-05-09T08:48:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fc0019a1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563aae02dc40 con 0x563aae10dc50 2024-05-09T08:48:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fc0019a1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x563aae11b230 con 0x563aae118320 2024-05-09T08:48:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fc0019a1740 10 client.0 _ll_register_callbacks cb 0x563aadd85be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fbffd154640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10d330 0x563aae1152d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fbffd154640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10d330 0x563aae1152d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57024/0 (socket says 172.21.15.107:57024) 2024-05-09T08:48:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fbffd154640 1 -- 172.21.15.107:0/2324749878 learned_addr learned my addr 172.21.15.107:0/2324749878 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fbff7fff640 1 --2- 172.21.15.107:0/2324749878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae10dc50 0x563aae115810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fbffd154640 1 -- 172.21.15.107:0/2324749878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae118320 msgr2=0x563aae115d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fbffd154640 1 --2- 172.21.15.107:0/2324749878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae118320 0x563aae115d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fbffd154640 1 -- 172.21.15.107:0/2324749878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae10dc50 msgr2=0x563aae115810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fbffd154640 1 --2- 172.21.15.107:0/2324749878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae10dc50 0x563aae115810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.984+0000 7fbffd154640 1 -- 172.21.15.107:0/2324749878 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563aade507d0 con 0x563aae10d330 2024-05-09T08:48:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fbffd154640 1 --2- 172.21.15.107:0/2324749878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10d330 0x563aae1152d0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009f70 tx=0x7fbff0012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b6e8762bd4e02a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fbfd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:48:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fbfd7fff640 1 -- 172.21.15.107:0/2324749878 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff0018070 con 0x563aae10d330 2024-05-09T08:48:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fbfd7fff640 1 -- 172.21.15.107:0/2324749878 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7fbff0004d20 con 0x563aae10d330 2024-05-09T08:48:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fc0019a1740 1 -- 172.21.15.107:0/2324749878 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfec032e20 con 0x563aae10d330 2024-05-09T08:48:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fbfd7fff640 1 -- 172.21.15.107:0/2324749878 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbff0025050 con 0x563aae10d330 2024-05-09T08:48:43.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fc0019a1740 10 client.19431 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:43.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fc0019a1740 1 -- 172.21.15.107:0/2324749878 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563aae127330 con 0x563aae10d330 2024-05-09T08:48:43.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fbfd7fff640 1 -- 172.21.15.107:0/2324749878 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfec032e20 con 0x563aae10d330 2024-05-09T08:48:43.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fbfd7fff640 1 -- 172.21.15.107:0/2324749878 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fbff00056a0 con 0x563aae10d330 2024-05-09T08:48:43.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fc0019a1740 10 client.19431 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:43.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fc0019a1740 10 client.19431 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:48:43.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fc0019a1740 20 client.19431 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:43.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fc0019a1740 1 --2- 172.21.15.107:0/2324749878 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563aae128790 0x563aae12abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:43.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fc0019a1740 4 client.19431 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:48:43.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fc0019a1740 1 -- 172.21.15.107:0/2324749878 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563aae080370 con 0x563aae128790 2024-05-09T08:48:43.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.985+0000 7fbff7fff640 1 --2- 172.21.15.107:0/2324749878 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563aae128790 0x563aae12abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:43.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.986+0000 7fbff7fff640 1 --2- 172.21.15.107:0/2324749878 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563aae128790 0x563aae12abf0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fcaba84ee0c2e926 server_cookie=35d6e8808919b12b in_seq=0 out_seq=0 2024-05-09T08:48:43.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.986+0000 7fbfd7fff640 10 client.19431 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:48:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.987+0000 7fbfd7fff640 1 -- 172.21.15.107:0/2324749878 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563aae080370 con 0x563aae128790 2024-05-09T08:48:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.987+0000 7fbfd7fff640 10 client.19431 handle_command_reply: tid=0 2024-05-09T08:48:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.987+0000 7fc0019a1740 1 client.19431 shutdown 2024-05-09T08:48:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.987+0000 7fc0019a1740 1 -- 172.21.15.107:0/2324749878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10d330 msgr2=0x563aae1152d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.987+0000 7fc0019a1740 1 --2- 172.21.15.107:0/2324749878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10d330 0x563aae1152d0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009f70 tx=0x7fbff0012b60 comp rx=0 tx=0).stop 2024-05-09T08:48:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.987+0000 7fc0019a1740 1 -- 172.21.15.107:0/2324749878 shutdown_connections 2024-05-09T08:48:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.987+0000 7fc0019a1740 1 --2- 172.21.15.107:0/2324749878 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563aae128790 0x563aae12abf0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.987+0000 7fc0019a1740 1 --2- 172.21.15.107:0/2324749878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae118320 0x563aae115d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.987+0000 7fc0019a1740 1 --2- 172.21.15.107:0/2324749878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae10dc50 0x563aae115810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.987+0000 7fbfd7fff640 0 client.19431 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:48:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.987+0000 7fc0019a1740 1 --2- 172.21.15.107:0/2324749878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10d330 0x563aae1152d0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.987+0000 7fc0019a1740 1 -- 172.21.15.107:0/2324749878 >> 172.21.15.107:0/2324749878 conn(0x563aae109650 msgr2=0x563aae10f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.987+0000 7fc0019a1740 1 -- 172.21.15.107:0/2324749878 shutdown_connections 2024-05-09T08:48:43.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.987+0000 7fc0019a1740 1 -- 172.21.15.107:0/2324749878 wait complete. 2024-05-09T08:48:43.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:43.987+0000 7fc0019a1740 20 client.19431 trim_cache size 0 max 16384 2024-05-09T08:48:44.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.000+0000 7fc0019a1740 1 Processor -- start 2024-05-09T08:48:44.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.000+0000 7fc0019a1740 1 -- start start 2024-05-09T08:48:44.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.000+0000 7fc0019a1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10d330 0x563aae117a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.000+0000 7fc0019a1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae10dc50 0x563aae114bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.000+0000 7fc0019a1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae118320 0x563aae10fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.000+0000 7fc0019a1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81b1550 con 0x563aae10d330 2024-05-09T08:48:44.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.000+0000 7fc0019a1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81aedc0 con 0x563aae118320 2024-05-09T08:48:44.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.000+0000 7fc0019a1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81acc30 con 0x563aae10dc50 2024-05-09T08:48:44.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.000+0000 7fbffd154640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10d330 0x563aae117a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.000+0000 7fbffd154640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10d330 0x563aae117a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57026/0 (socket says 172.21.15.107:57026) 2024-05-09T08:48:44.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.000+0000 7fbffd154640 1 -- 172.21.15.107:0/595702609 learned_addr learned my addr 172.21.15.107:0/595702609 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:44.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.001+0000 7fbffd154640 1 -- 172.21.15.107:0/595702609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae10dc50 msgr2=0x563aae114bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.001+0000 7fbffd154640 1 --2- 172.21.15.107:0/595702609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae10dc50 0x563aae114bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.001+0000 7fbffd154640 1 -- 172.21.15.107:0/595702609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae118320 msgr2=0x563aae10fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.001+0000 7fbffd154640 1 --2- 172.21.15.107:0/595702609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae118320 0x563aae10fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.001+0000 7fbffd154640 1 -- 172.21.15.107:0/595702609 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563aae11aa80 con 0x563aae10d330 2024-05-09T08:48:44.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.001+0000 7fbffd154640 1 --2- 172.21.15.107:0/595702609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10d330 0x563aae117a00 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009f40 tx=0x7fbff0005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a7f70e63f9eae17 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:44.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.001+0000 7fbfde7fc640 1 -- 172.21.15.107:0/595702609 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff0018040 con 0x563aae10d330 2024-05-09T08:48:44.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.001+0000 7fbfde7fc640 1 -- 172.21.15.107:0/595702609 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbff001a030 con 0x563aae10d330 2024-05-09T08:48:44.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.001+0000 7fbfde7fc640 1 -- 172.21.15.107:0/595702609 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff0011bc0 con 0x563aae10d330 2024-05-09T08:48:44.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.001+0000 7fc0019a1740 1 -- 172.21.15.107:0/595702609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10d330 msgr2=0x563aae117a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.001+0000 7fc0019a1740 1 --2- 172.21.15.107:0/595702609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10d330 0x563aae117a00 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009f40 tx=0x7fbff0005c50 comp rx=0 tx=0).stop 2024-05-09T08:48:44.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.001+0000 7fc0019a1740 1 -- 172.21.15.107:0/595702609 shutdown_connections 2024-05-09T08:48:44.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.001+0000 7fc0019a1740 1 --2- 172.21.15.107:0/595702609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae118320 0x563aae10fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.001+0000 7fc0019a1740 1 --2- 172.21.15.107:0/595702609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae10dc50 0x563aae114bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.001+0000 7fc0019a1740 1 --2- 172.21.15.107:0/595702609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae10d330 0x563aae117a00 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.001+0000 7fc0019a1740 1 -- 172.21.15.107:0/595702609 >> 172.21.15.107:0/595702609 conn(0x563aae109650 msgr2=0x563aae1087b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.002+0000 7fc0019a1740 1 -- 172.21.15.107:0/595702609 shutdown_connections 2024-05-09T08:48:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.002+0000 7fc0019a1740 1 -- 172.21.15.107:0/595702609 wait complete. 2024-05-09T08:48:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.002+0000 7fc0019a1740 1 -- start start 2024-05-09T08:48:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.002+0000 7fc0019a1740 1 Processor -- start 2024-05-09T08:48:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.002+0000 7fc0019a1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae10d330 0x563aae112210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.002+0000 7fc0019a1740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae10dc50 0x563aae112750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.002+0000 7fc0019a1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae118320 0x563aae112c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.002+0000 7fc0019a1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81ad280 con 0x563aae118320 2024-05-09T08:48:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.002+0000 7fc0019a1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81ae6f0 con 0x563aae10d330 2024-05-09T08:48:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.002+0000 7fc0019a1740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81ad950 con 0x563aae10dc50 2024-05-09T08:48:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.002+0000 7fc0019a1740 10 client.0 _ll_register_callbacks cb 0x7fbff0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.002+0000 7fbffd154640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae118320 0x563aae112c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.002+0000 7fbffd154640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae118320 0x563aae112c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57034/0 (socket says 172.21.15.107:57034) 2024-05-09T08:48:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.002+0000 7fbffd154640 1 -- 172.21.15.107:0/2685679860 learned_addr learned my addr 172.21.15.107:0/2685679860 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fbffd154640 1 --2- 172.21.15.107:0/2685679860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae10d330 0x563aae112210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fbff7fff640 1 --2- 172.21.15.107:0/2685679860 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae10dc50 0x563aae112750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fbffd154640 1 -- 172.21.15.107:0/2685679860 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae10dc50 msgr2=0x563aae112750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fbffd154640 1 --2- 172.21.15.107:0/2685679860 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae10dc50 0x563aae112750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fbffd154640 1 -- 172.21.15.107:0/2685679860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae10d330 msgr2=0x563aae112210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fbffd154640 1 --2- 172.21.15.107:0/2685679860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae10d330 0x563aae112210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fbffd154640 1 -- 172.21.15.107:0/2685679860 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563aae18dfd0 con 0x563aae118320 2024-05-09T08:48:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fbff7fff640 1 --2- 172.21.15.107:0/2685679860 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae10dc50 0x563aae112750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fbffd154640 1 --2- 172.21.15.107:0/2685679860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae118320 0x563aae112c90 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fbff0015040 tx=0x7fbff0020040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878358cd626b2a73 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fbfd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:48:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fbfd7fff640 1 -- 172.21.15.107:0/2685679860 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff0011b60 con 0x563aae118320 2024-05-09T08:48:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fc0019a1740 1 -- 172.21.15.107:0/2685679860 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbff001a030 con 0x563aae118320 2024-05-09T08:48:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fbfd7fff640 1 -- 172.21.15.107:0/2685679860 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7fbff001d940 con 0x563aae118320 2024-05-09T08:48:44.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fbfd7fff640 1 -- 172.21.15.107:0/2685679860 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbff001db60 con 0x563aae118320 2024-05-09T08:48:44.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fc0019a1740 10 client.19437 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:44.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fc0019a1740 1 -- 172.21.15.107:0/2685679860 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563aae18f2b0 con 0x563aae118320 2024-05-09T08:48:44.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fbfd7fff640 1 -- 172.21.15.107:0/2685679860 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbff001a030 con 0x563aae118320 2024-05-09T08:48:44.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fbfd7fff640 1 -- 172.21.15.107:0/2685679860 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fbff0033c60 con 0x563aae118320 2024-05-09T08:48:44.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fc0019a1740 10 client.19437 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:44.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fc0019a1740 10 client.19437 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-05-09T08:48:44.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fc0019a1740 20 client.19437 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:44.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.003+0000 7fc0019a1740 1 --2- 172.21.15.107:0/2685679860 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563aae18f8f0 0x563aae191d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.004+0000 7fc0019a1740 4 client.19437 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:48:44.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.004+0000 7fc0019a1740 1 -- 172.21.15.107:0/2685679860 --> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563aae07a6c0 con 0x563aae18f8f0 2024-05-09T08:48:44.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.004+0000 7fbff7fff640 1 --2- 172.21.15.107:0/2685679860 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563aae18f8f0 0x563aae191d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.004+0000 7fbfd7fff640 10 client.19437 ms_handle_connect on v2:172.21.15.157:6836/1973902800 2024-05-09T08:48:44.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.004+0000 7fbff7fff640 1 --2- 172.21.15.107:0/2685679860 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563aae18f8f0 0x563aae191d10 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=85ed467aa1a963d4 server_cookie=b4a3081624465ed6 in_seq=0 out_seq=0 2024-05-09T08:48:44.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.004+0000 7fbfd7fff640 1 -- 172.21.15.107:0/2685679860 <== mds.2 v2:172.21.15.157:6836/1973902800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563aae07a6c0 con 0x563aae18f8f0 2024-05-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.004+0000 7fbfd7fff640 10 client.19437 handle_command_reply: tid=0 2024-05-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fc0019a1740 1 client.19437 shutdown 2024-05-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fc0019a1740 1 -- 172.21.15.107:0/2685679860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae118320 msgr2=0x563aae112c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fc0019a1740 1 --2- 172.21.15.107:0/2685679860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae118320 0x563aae112c90 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fbff0015040 tx=0x7fbff0020040 comp rx=0 tx=0).stop 2024-05-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fc0019a1740 1 -- 172.21.15.107:0/2685679860 shutdown_connections 2024-05-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fc0019a1740 1 --2- 172.21.15.107:0/2685679860 >> [v2:172.21.15.157:6836/1973902800,v1:172.21.15.157:6837/1973902800] conn(0x563aae18f8f0 0x563aae191d10 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fc0019a1740 1 --2- 172.21.15.107:0/2685679860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563aae118320 0x563aae112c90 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fc0019a1740 1 --2- 172.21.15.107:0/2685679860 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563aae10dc50 0x563aae112750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fbfd7fff640 0 client.19437 ms_handle_reset on v2:172.21.15.157:6836/1973902800 2024-05-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fc0019a1740 1 --2- 172.21.15.107:0/2685679860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563aae10d330 0x563aae112210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fc0019a1740 1 -- 172.21.15.107:0/2685679860 >> 172.21.15.107:0/2685679860 conn(0x563aae109650 msgr2=0x563aae116980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fc0019a1740 1 -- 172.21.15.107:0/2685679860 shutdown_connections 2024-05-09T08:48:44.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fc0019a1740 1 -- 172.21.15.107:0/2685679860 wait complete. 2024-05-09T08:48:44.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fc0019a1740 20 client.19437 trim_cache size 0 max 16384 2024-05-09T08:48:44.012 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:48:44.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fbffebb7640 1 -- 172.21.15.107:0/915150447 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbfe00a2d00 msgr2=0x7fbfe00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fbffebb7640 1 --2- 172.21.15.107:0/915150447 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbfe00a2d00 0x7fbfe00a51c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbfe80cc730 tx=0x7fbfe809e040 comp rx=0 tx=0).stop 2024-05-09T08:48:44.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fbffebb7640 1 -- 172.21.15.107:0/915150447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbff8100710 msgr2=0x7fbff8198e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fbffebb7640 1 --2- 172.21.15.107:0/915150447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbff8100710 0x7fbff8198e60 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800d950 tx=0x7fbfe800de10 comp rx=0 tx=0).stop 2024-05-09T08:48:44.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fbffebb7640 1 -- 172.21.15.107:0/915150447 shutdown_connections 2024-05-09T08:48:44.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fbffebb7640 1 --2- 172.21.15.107:0/915150447 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbfe00a2d00 0x7fbfe00a51c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fbffebb7640 1 --2- 172.21.15.107:0/915150447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbff8106720 0x7fbff81a3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fbffebb7640 1 --2- 172.21.15.107:0/915150447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbff81010d0 0x7fbff81993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fbffebb7640 1 --2- 172.21.15.107:0/915150447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbff8100710 0x7fbff8198e60 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.005+0000 7fbffebb7640 1 -- 172.21.15.107:0/915150447 >> 172.21.15.107:0/915150447 conn(0x7fbff8077790 msgr2=0x7fbff8103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:44.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.006+0000 7fbffebb7640 1 -- 172.21.15.107:0/915150447 shutdown_connections 2024-05-09T08:48:44.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.006+0000 7fbffebb7640 1 -- 172.21.15.107:0/915150447 wait complete. 2024-05-09T08:48:44.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T08:48:44.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.119+0000 7f18b768c640 1 -- 172.21.15.107:0/2991357258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b01010d0 msgr2=0x7f18b010d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.119+0000 7f18b768c640 1 --2- 172.21.15.107:0/2991357258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b01010d0 0x7f18b010d5e0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009cd0 tx=0x7f18a802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.119+0000 7f18b768c640 1 -- 172.21.15.107:0/2991357258 shutdown_connections 2024-05-09T08:48:44.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.119+0000 7f18b768c640 1 --2- 172.21.15.107:0/2991357258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b01010d0 0x7f18b010d5e0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.119+0000 7f18b768c640 1 --2- 172.21.15.107:0/2991357258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b0100710 0x7f18b0100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.119+0000 7f18b768c640 1 --2- 172.21.15.107:0/2991357258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18b0106720 0x7f18b0106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.119+0000 7f18b768c640 1 -- 172.21.15.107:0/2991357258 >> 172.21.15.107:0/2991357258 conn(0x7f18b0077790 msgr2=0x7f18b0077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.119+0000 7f18b768c640 1 -- 172.21.15.107:0/2991357258 shutdown_connections 2024-05-09T08:48:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.119+0000 7f18b768c640 1 -- 172.21.15.107:0/2991357258 wait complete. 2024-05-09T08:48:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b768c640 1 Processor -- start 2024-05-09T08:48:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b768c640 1 -- start start 2024-05-09T08:48:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b768c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b0100710 0x7f18b0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b768c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b01010d0 0x7f18b0198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b768c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18b0106720 0x7f18b019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b768c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f18b0111ae0 con 0x7f18b01010d0 2024-05-09T08:48:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b768c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f18b0111960 con 0x7f18b0106720 2024-05-09T08:48:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b768c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f18b0111c60 con 0x7f18b0100710 2024-05-09T08:48:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b4c27640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b01010d0 0x7f18b0198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b4c27640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b01010d0 0x7f18b0198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57052/0 (socket says 172.21.15.107:57052) 2024-05-09T08:48:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b4c27640 1 -- 172.21.15.107:0/3145473363 learned_addr learned my addr 172.21.15.107:0/3145473363 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b5428640 1 --2- 172.21.15.107:0/3145473363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b0100710 0x7f18b0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b4c27640 1 -- 172.21.15.107:0/3145473363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b0100710 msgr2=0x7f18b0197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b4c27640 1 --2- 172.21.15.107:0/3145473363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b0100710 0x7f18b0197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b4c27640 1 -- 172.21.15.107:0/3145473363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18b0106720 msgr2=0x7f18b019c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b4c27640 1 --2- 172.21.15.107:0/3145473363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18b0106720 0x7f18b019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b4c27640 1 -- 172.21.15.107:0/3145473363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b019cfd0 con 0x7f18b01010d0 2024-05-09T08:48:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18b4c27640 1 --2- 172.21.15.107:0/3145473363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b01010d0 0x7f18b0198520 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f18a400e9f0 tx=0x7f18a400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.120+0000 7f18a27fc640 1 -- 172.21.15.107:0/3145473363 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a400cd80 con 0x7f18b01010d0 2024-05-09T08:48:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.121+0000 7f18a27fc640 1 -- 172.21.15.107:0/3145473363 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18a40054d0 con 0x7f18b01010d0 2024-05-09T08:48:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.121+0000 7f18b768c640 1 -- 172.21.15.107:0/3145473363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b019d2c0 con 0x7f18b01010d0 2024-05-09T08:48:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.121+0000 7f18a27fc640 1 -- 172.21.15.107:0/3145473363 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4010490 con 0x7f18b01010d0 2024-05-09T08:48:44.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.121+0000 7f18b768c640 1 -- 172.21.15.107:0/3145473363 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b01a4b50 con 0x7f18b01010d0 2024-05-09T08:48:44.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.124+0000 7f18a27fc640 1 -- 172.21.15.107:0/3145473363 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f18a4005020 con 0x7f18b01010d0 2024-05-09T08:48:44.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.124+0000 7f18ba397740 1 Processor -- start 2024-05-09T08:48:44.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.124+0000 7f18ba397740 1 -- start start 2024-05-09T08:48:44.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.124+0000 7f18ba397740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6f4330 0x55cf2e6f4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.124+0000 7f18ba397740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6f4c50 0x55cf2e6fede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.124+0000 7f18ba397740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6ff320 0x55cf2e701710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.124+0000 7f18ba397740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cf2e659c40 con 0x55cf2e6ff320 2024-05-09T08:48:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.124+0000 7f18ba397740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cf2e601580 con 0x55cf2e6f4330 2024-05-09T08:48:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.124+0000 7f18ba397740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55cf2e655c00 con 0x55cf2e6f4c50 2024-05-09T08:48:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.124+0000 7f18b4c27640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6ff320 0x55cf2e701710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.124+0000 7f18b4c27640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6ff320 0x55cf2e701710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57058/0 (socket says 172.21.15.107:57058) 2024-05-09T08:48:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.124+0000 7f18b4c27640 1 -- 172.21.15.107:0/3791544398 learned_addr learned my addr 172.21.15.107:0/3791544398 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.124+0000 7f18a27fc640 1 --2- 172.21.15.107:0/3145473363 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f189c0a2d00 0x7f189c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.124+0000 7f18b5c29640 1 --2- 172.21.15.107:0/3791544398 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6f4330 0x55cf2e6f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.124+0000 7f18b5428640 1 --2- 172.21.15.107:0/3791544398 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6f4c50 0x55cf2e6fede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.124+0000 7f18b5c29640 1 --2- 172.21.15.107:0/3145473363 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f189c0a2d00 0x7f189c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.125+0000 7f18b4c27640 1 -- 172.21.15.107:0/3791544398 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6f4c50 msgr2=0x55cf2e6fede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.125+0000 7f18a27fc640 1 -- 172.21.15.107:0/3145473363 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7f18a40cfce0 con 0x7f18b01010d0 2024-05-09T08:48:44.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.125+0000 7f18b4c27640 1 --2- 172.21.15.107:0/3791544398 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6f4c50 0x55cf2e6fede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.125+0000 7f18b4c27640 1 -- 172.21.15.107:0/3791544398 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6f4330 msgr2=0x55cf2e6f4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.125+0000 7f18b4c27640 1 --2- 172.21.15.107:0/3791544398 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6f4330 0x55cf2e6f4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.125+0000 7f18b4c27640 1 -- 172.21.15.107:0/3791544398 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf2e36d3f0 con 0x55cf2e6ff320 2024-05-09T08:48:44.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.125+0000 7f18b5428640 1 --2- 172.21.15.107:0/3791544398 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6f4c50 0x55cf2e6fede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:44.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.125+0000 7f18b5c29640 1 --2- 172.21.15.107:0/3791544398 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6f4330 0x55cf2e6f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:44.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.125+0000 7f18b4c27640 1 --2- 172.21.15.107:0/3791544398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6ff320 0x55cf2e701710 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f18a409bde0 tx=0x7f18a4004aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=979c83326a632b28 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:44.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.125+0000 7f189b7fe640 1 -- 172.21.15.107:0/3791544398 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a409bc20 con 0x55cf2e6ff320 2024-05-09T08:48:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.125+0000 7f18b5c29640 1 --2- 172.21.15.107:0/3145473363 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f189c0a2d00 0x7f189c0a51c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f18a8004af0 tx=0x7f18a80095c0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.125+0000 7f189b7fe640 1 -- 172.21.15.107:0/3791544398 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18a4005310 con 0x55cf2e6ff320 2024-05-09T08:48:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.125+0000 7f189b7fe640 1 -- 172.21.15.107:0/3791544398 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a409b7b0 con 0x55cf2e6ff320 2024-05-09T08:48:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.126+0000 7f18ba397740 1 -- 172.21.15.107:0/3791544398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6ff320 msgr2=0x55cf2e701710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.126+0000 7f18ba397740 1 --2- 172.21.15.107:0/3791544398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6ff320 0x55cf2e701710 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f18a409bde0 tx=0x7f18a4004aa0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.126+0000 7f18ba397740 1 -- 172.21.15.107:0/3791544398 shutdown_connections 2024-05-09T08:48:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.126+0000 7f18ba397740 1 --2- 172.21.15.107:0/3791544398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6ff320 0x55cf2e701710 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.126+0000 7f18ba397740 1 --2- 172.21.15.107:0/3791544398 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6f4c50 0x55cf2e6fede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.126+0000 7f18ba397740 1 --2- 172.21.15.107:0/3791544398 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6f4330 0x55cf2e6f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.126+0000 7f18ba397740 1 -- 172.21.15.107:0/3791544398 >> 172.21.15.107:0/3791544398 conn(0x55cf2e6f0650 msgr2=0x55cf2e6f0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:44.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.126+0000 7f18ba397740 1 -- 172.21.15.107:0/3791544398 shutdown_connections 2024-05-09T08:48:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.126+0000 7f18ba397740 1 -- 172.21.15.107:0/3791544398 wait complete. 2024-05-09T08:48:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.126+0000 7f18ba397740 1 -- start start 2024-05-09T08:48:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.126+0000 7f18ba397740 1 Processor -- start 2024-05-09T08:48:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18ba397740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6f4330 0x55cf2e6fc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18ba397740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6f4c50 0x55cf2e6fc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18ba397740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6ff320 0x55cf2e6fccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18ba397740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cf2e7020b0 con 0x55cf2e6ff320 2024-05-09T08:48:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18ba397740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cf2e614c40 con 0x55cf2e6f4c50 2024-05-09T08:48:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18ba397740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55cf2e702230 con 0x55cf2e6f4330 2024-05-09T08:48:44.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18ba397740 10 client.0 _ll_register_callbacks cb 0x55cf2e36cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18b5c29640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6ff320 0x55cf2e6fccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18b5c29640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6ff320 0x55cf2e6fccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57072/0 (socket says 172.21.15.107:57072) 2024-05-09T08:48:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18b5c29640 1 -- 172.21.15.107:0/382952266 learned_addr learned my addr 172.21.15.107:0/382952266 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18b5428640 1 --2- 172.21.15.107:0/382952266 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6f4330 0x55cf2e6fc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18b4c27640 1 --2- 172.21.15.107:0/382952266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6f4c50 0x55cf2e6fc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18b5c29640 1 -- 172.21.15.107:0/382952266 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6f4330 msgr2=0x55cf2e6fc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18b5c29640 1 --2- 172.21.15.107:0/382952266 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6f4330 0x55cf2e6fc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18b5c29640 1 -- 172.21.15.107:0/382952266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6f4c50 msgr2=0x55cf2e6fc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18b5c29640 1 --2- 172.21.15.107:0/382952266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6f4c50 0x55cf2e6fc790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18b5c29640 1 -- 172.21.15.107:0/382952266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf2e4377d0 con 0x55cf2e6ff320 2024-05-09T08:48:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18b5428640 1 --2- 172.21.15.107:0/382952266 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6f4330 0x55cf2e6fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18b5c29640 1 --2- 172.21.15.107:0/382952266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6ff320 0x55cf2e6fccd0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f18a8035000 tx=0x7f18a8005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e14d55c5b3b90f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f1898ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:48:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f1898ff9640 1 -- 172.21.15.107:0/382952266 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a804b070 con 0x55cf2e6ff320 2024-05-09T08:48:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f1898ff9640 1 -- 172.21.15.107:0/382952266 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7f18a80305a0 con 0x55cf2e6ff320 2024-05-09T08:48:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f18ba397740 1 -- 172.21.15.107:0/382952266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cf2e667370 con 0x55cf2e6ff320 2024-05-09T08:48:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.127+0000 7f1898ff9640 1 -- 172.21.15.107:0/382952266 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f18a8057050 con 0x55cf2e6ff320 2024-05-09T08:48:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.128+0000 7f18ba397740 10 client.19449 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.128+0000 7f18ba397740 1 -- 172.21.15.107:0/382952266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cf2e70e250 con 0x55cf2e6ff320 2024-05-09T08:48:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.128+0000 7f1898ff9640 1 -- 172.21.15.107:0/382952266 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55cf2e667370 con 0x55cf2e6ff320 2024-05-09T08:48:44.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.128+0000 7f1898ff9640 1 -- 172.21.15.107:0/382952266 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f18a8046070 con 0x55cf2e6ff320 2024-05-09T08:48:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.128+0000 7f18ba397740 10 client.19449 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.128+0000 7f18ba397740 10 client.19449 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:48:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.128+0000 7f18ba397740 20 client.19449 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.128+0000 7f18ba397740 1 --2- 172.21.15.107:0/382952266 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cf2e70f6b0 0x55cf2e711b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.128+0000 7f18ba397740 4 client.19449 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:48:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.128+0000 7f18ba397740 1 -- 172.21.15.107:0/382952266 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf2e6616c0 con 0x55cf2e70f6b0 2024-05-09T08:48:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.128+0000 7f18b5428640 1 --2- 172.21.15.107:0/382952266 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cf2e70f6b0 0x55cf2e711b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.129+0000 7f1898ff9640 10 client.19449 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:48:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.129+0000 7f18b5428640 1 --2- 172.21.15.107:0/382952266 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cf2e70f6b0 0x55cf2e711b50 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1856c7df715d7fb9 server_cookie=b445b3b3471d927e in_seq=0 out_seq=0 2024-05-09T08:48:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.129+0000 7f1898ff9640 1 -- 172.21.15.107:0/382952266 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf2e6616c0 con 0x55cf2e70f6b0 2024-05-09T08:48:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.129+0000 7f1898ff9640 10 client.19449 handle_command_reply: tid=0 2024-05-09T08:48:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.129+0000 7f18ba397740 1 client.19449 shutdown 2024-05-09T08:48:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.129+0000 7f18ba397740 1 -- 172.21.15.107:0/382952266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6ff320 msgr2=0x55cf2e6fccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.129+0000 7f18ba397740 1 --2- 172.21.15.107:0/382952266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6ff320 0x55cf2e6fccd0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f18a8035000 tx=0x7f18a8005d80 comp rx=0 tx=0).stop 2024-05-09T08:48:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.129+0000 7f18ba397740 1 -- 172.21.15.107:0/382952266 shutdown_connections 2024-05-09T08:48:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.129+0000 7f18ba397740 1 --2- 172.21.15.107:0/382952266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6ff320 0x55cf2e6fccd0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.129+0000 7f18ba397740 1 --2- 172.21.15.107:0/382952266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6f4c50 0x55cf2e6fc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.129+0000 7f18ba397740 1 --2- 172.21.15.107:0/382952266 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cf2e70f6b0 0x55cf2e711b50 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.129+0000 7f18ba397740 1 --2- 172.21.15.107:0/382952266 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6f4330 0x55cf2e6fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.129+0000 7f1898ff9640 0 client.19449 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:48:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.129+0000 7f18ba397740 1 -- 172.21.15.107:0/382952266 >> 172.21.15.107:0/382952266 conn(0x55cf2e6f0650 msgr2=0x55cf2e6f6450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.130+0000 7f18ba397740 1 -- 172.21.15.107:0/382952266 shutdown_connections 2024-05-09T08:48:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.130+0000 7f18ba397740 1 -- 172.21.15.107:0/382952266 wait complete. 2024-05-09T08:48:44.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.130+0000 7f18ba397740 20 client.19449 trim_cache size 0 max 16384 2024-05-09T08:48:44.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18ba397740 1 Processor -- start 2024-05-09T08:48:44.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18ba397740 1 -- start start 2024-05-09T08:48:44.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18ba397740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6f4330 0x55cf2e70af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18ba397740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6f4c50 0x55cf2e6f6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18ba397740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6ff320 0x55cf2e717420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18ba397740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f18b01b1630 con 0x55cf2e6f4330 2024-05-09T08:48:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18ba397740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f18b01aeea0 con 0x55cf2e6ff320 2024-05-09T08:48:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18ba397740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f18b01acd10 con 0x55cf2e6f4c50 2024-05-09T08:48:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18b5428640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6f4330 0x55cf2e70af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18b5428640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6f4330 0x55cf2e70af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57078/0 (socket says 172.21.15.107:57078) 2024-05-09T08:48:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18b5428640 1 -- 172.21.15.107:0/1052004580 learned_addr learned my addr 172.21.15.107:0/1052004580 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18b5428640 1 -- 172.21.15.107:0/1052004580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6f4c50 msgr2=0x55cf2e6f6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18b5428640 1 --2- 172.21.15.107:0/1052004580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6f4c50 0x55cf2e6f6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18b5428640 1 -- 172.21.15.107:0/1052004580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6ff320 msgr2=0x55cf2e717420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18b5428640 1 --2- 172.21.15.107:0/1052004580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6ff320 0x55cf2e717420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18b5428640 1 -- 172.21.15.107:0/1052004580 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf2e6e1d20 con 0x55cf2e6f4330 2024-05-09T08:48:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f18b5428640 1 --2- 172.21.15.107:0/1052004580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6f4330 0x55cf2e70af70 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x55cf2e7123c0 tx=0x7f18ac005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f070baff305c889 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.143+0000 7f189affd640 1 -- 172.21.15.107:0/1052004580 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac004030 con 0x55cf2e6f4330 2024-05-09T08:48:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.144+0000 7f189affd640 1 -- 172.21.15.107:0/1052004580 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18ac01b770 con 0x55cf2e6f4330 2024-05-09T08:48:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.144+0000 7f189affd640 1 -- 172.21.15.107:0/1052004580 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac003830 con 0x55cf2e6f4330 2024-05-09T08:48:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.144+0000 7f18ba397740 1 -- 172.21.15.107:0/1052004580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6f4330 msgr2=0x55cf2e70af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.144+0000 7f18ba397740 1 --2- 172.21.15.107:0/1052004580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6f4330 0x55cf2e70af70 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x55cf2e7123c0 tx=0x7f18ac005f30 comp rx=0 tx=0).stop 2024-05-09T08:48:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.144+0000 7f18ba397740 1 -- 172.21.15.107:0/1052004580 shutdown_connections 2024-05-09T08:48:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.144+0000 7f18ba397740 1 --2- 172.21.15.107:0/1052004580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6ff320 0x55cf2e717420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.144+0000 7f18ba397740 1 --2- 172.21.15.107:0/1052004580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6f4c50 0x55cf2e6f6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.144+0000 7f18ba397740 1 --2- 172.21.15.107:0/1052004580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6f4330 0x55cf2e70af70 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.144+0000 7f18ba397740 1 -- 172.21.15.107:0/1052004580 >> 172.21.15.107:0/1052004580 conn(0x55cf2e6f0650 msgr2=0x55cf2e6fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.144+0000 7f18ba397740 1 -- 172.21.15.107:0/1052004580 shutdown_connections 2024-05-09T08:48:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.144+0000 7f18ba397740 1 -- 172.21.15.107:0/1052004580 wait complete. 2024-05-09T08:48:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.144+0000 7f18ba397740 1 -- start start 2024-05-09T08:48:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.144+0000 7f18ba397740 1 Processor -- start 2024-05-09T08:48:44.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18ba397740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6f4330 0x55cf2e707580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18ba397740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6f4c50 0x55cf2e702a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18ba397740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6ff320 0x55cf2e702fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18ba397740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f18b01ad360 con 0x55cf2e6f4330 2024-05-09T08:48:44.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18ba397740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f18b01ae7d0 con 0x55cf2e6f4c50 2024-05-09T08:48:44.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18ba397740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f18b01ada30 con 0x55cf2e6ff320 2024-05-09T08:48:44.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18ba397740 10 client.0 _ll_register_callbacks cb 0x7f18a4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:44.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18b5428640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6f4330 0x55cf2e707580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18b5428640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6f4330 0x55cf2e707580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57090/0 (socket says 172.21.15.107:57090) 2024-05-09T08:48:44.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18b5428640 1 -- 172.21.15.107:0/2507644974 learned_addr learned my addr 172.21.15.107:0/2507644974 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:44.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18b4c27640 1 --2- 172.21.15.107:0/2507644974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6f4c50 0x55cf2e702a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18b5428640 1 -- 172.21.15.107:0/2507644974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6ff320 msgr2=0x55cf2e702fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18b5428640 1 --2- 172.21.15.107:0/2507644974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6ff320 0x55cf2e702fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18b5428640 1 -- 172.21.15.107:0/2507644974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6f4c50 msgr2=0x55cf2e702a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18b5428640 1 --2- 172.21.15.107:0/2507644974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6f4c50 0x55cf2e702a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18b5428640 1 -- 172.21.15.107:0/2507644974 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf2e703510 con 0x55cf2e6f4330 2024-05-09T08:48:44.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f18b5428640 1 --2- 172.21.15.107:0/2507644974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6f4330 0x55cf2e707580 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f18ac004500 tx=0x7f18ac00fe50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf07d024d7ec656 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:44.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.145+0000 7f1898ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:48:44.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.146+0000 7f1898ff9640 1 -- 172.21.15.107:0/2507644974 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac01b2c0 con 0x55cf2e6f4330 2024-05-09T08:48:44.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.146+0000 7f1898ff9640 1 -- 172.21.15.107:0/2507644974 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7f18ac01b460 con 0x55cf2e6f4330 2024-05-09T08:48:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.146+0000 7f18ba397740 1 -- 172.21.15.107:0/2507644974 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18ac01b770 con 0x55cf2e6f4330 2024-05-09T08:48:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.146+0000 7f1898ff9640 1 -- 172.21.15.107:0/2507644974 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f18ac037050 con 0x55cf2e6f4330 2024-05-09T08:48:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.146+0000 7f18ba397740 10 client.19455 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.146+0000 7f18ba397740 1 -- 172.21.15.107:0/2507644974 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cf2e7759f0 con 0x55cf2e6f4330 2024-05-09T08:48:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.146+0000 7f1898ff9640 1 -- 172.21.15.107:0/2507644974 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18ac01b770 con 0x55cf2e6f4330 2024-05-09T08:48:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.146+0000 7f1898ff9640 1 -- 172.21.15.107:0/2507644974 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f18ac011440 con 0x55cf2e6f4330 2024-05-09T08:48:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.146+0000 7f18ba397740 10 client.19455 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.146+0000 7f18ba397740 10 client.19455 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-05-09T08:48:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.146+0000 7f18ba397740 20 client.19455 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.146+0000 7f18ba397740 1 --2- 172.21.15.107:0/2507644974 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cf2e776140 0x55cf2e778560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.146+0000 7f18ba397740 4 client.19455 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:48:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.146+0000 7f18ba397740 1 -- 172.21.15.107:0/2507644974 --> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cf2e5de820 con 0x55cf2e776140 2024-05-09T08:48:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.146+0000 7f18b4c27640 1 --2- 172.21.15.107:0/2507644974 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cf2e776140 0x55cf2e778560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f1898ff9640 10 client.19455 ms_handle_connect on v2:172.21.15.110:6838/593143450 2024-05-09T08:48:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f18b4c27640 1 --2- 172.21.15.107:0/2507644974 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cf2e776140 0x55cf2e778560 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=69fc71ab51251163 server_cookie=3438166acbaff4cb in_seq=0 out_seq=0 2024-05-09T08:48:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f1898ff9640 1 -- 172.21.15.107:0/2507644974 <== mds.1 v2:172.21.15.110:6838/593143450 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f18a409e020 con 0x55cf2e776140 2024-05-09T08:48:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f1898ff9640 10 client.19455 handle_command_reply: tid=0 2024-05-09T08:48:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f18ba397740 1 client.19455 shutdown 2024-05-09T08:48:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f18ba397740 1 -- 172.21.15.107:0/2507644974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6f4330 msgr2=0x55cf2e707580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f18ba397740 1 --2- 172.21.15.107:0/2507644974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6f4330 0x55cf2e707580 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f18ac004500 tx=0x7f18ac00fe50 comp rx=0 tx=0).stop 2024-05-09T08:48:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f18ba397740 1 -- 172.21.15.107:0/2507644974 shutdown_connections 2024-05-09T08:48:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f18ba397740 1 --2- 172.21.15.107:0/2507644974 >> [v2:172.21.15.110:6838/593143450,v1:172.21.15.110:6839/593143450] conn(0x55cf2e776140 0x55cf2e778560 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f18ba397740 1 --2- 172.21.15.107:0/2507644974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cf2e6ff320 0x55cf2e702fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f18ba397740 1 --2- 172.21.15.107:0/2507644974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf2e6f4c50 0x55cf2e702a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f1898ff9640 0 client.19455 ms_handle_reset on v2:172.21.15.110:6838/593143450 2024-05-09T08:48:44.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f18ba397740 1 --2- 172.21.15.107:0/2507644974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cf2e6f4330 0x55cf2e707580 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f18ba397740 1 -- 172.21.15.107:0/2507644974 >> 172.21.15.107:0/2507644974 conn(0x55cf2e6f0650 msgr2=0x55cf2e6eef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:44.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f18ba397740 1 -- 172.21.15.107:0/2507644974 shutdown_connections 2024-05-09T08:48:44.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f18ba397740 1 -- 172.21.15.107:0/2507644974 wait complete. 2024-05-09T08:48:44.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.147+0000 7f18ba397740 20 client.19455 trim_cache size 0 max 16384 2024-05-09T08:48:44.157 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:48:44.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.148+0000 7f18b768c640 1 -- 172.21.15.107:0/3145473363 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f189c0a2d00 msgr2=0x7f189c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.148+0000 7f18b768c640 1 --2- 172.21.15.107:0/3145473363 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f189c0a2d00 0x7f189c0a51c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f18a8004af0 tx=0x7f18a80095c0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.148+0000 7f18b768c640 1 -- 172.21.15.107:0/3145473363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b01010d0 msgr2=0x7f18b0198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.148+0000 7f18b768c640 1 --2- 172.21.15.107:0/3145473363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b01010d0 0x7f18b0198520 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f18a400e9f0 tx=0x7f18a400edf0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.148+0000 7f18b768c640 1 -- 172.21.15.107:0/3145473363 shutdown_connections 2024-05-09T08:48:44.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.148+0000 7f18b768c640 1 --2- 172.21.15.107:0/3145473363 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f189c0a2d00 0x7f189c0a51c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.148+0000 7f18b768c640 1 --2- 172.21.15.107:0/3145473363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18b0106720 0x7f18b019c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.148+0000 7f18b768c640 1 --2- 172.21.15.107:0/3145473363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18b01010d0 0x7f18b0198520 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.148+0000 7f18b768c640 1 --2- 172.21.15.107:0/3145473363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b0100710 0x7f18b0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.148+0000 7f18b768c640 1 -- 172.21.15.107:0/3145473363 >> 172.21.15.107:0/3145473363 conn(0x7f18b0077790 msgr2=0x7f18b01032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:44.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.148+0000 7f18b768c640 1 -- 172.21.15.107:0/3145473363 shutdown_connections 2024-05-09T08:48:44.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.148+0000 7f18b768c640 1 -- 172.21.15.107:0/3145473363 wait complete. 2024-05-09T08:48:44.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T08:48:44.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.263+0000 7f5656363640 1 -- 172.21.15.107:0/1272556527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5650100700 msgr2=0x7f5650100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.263+0000 7f5656363640 1 --2- 172.21.15.107:0/1272556527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5650100700 0x7f5650100b80 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f5644009930 tx=0x7f564402a420 comp rx=0 tx=0).stop 2024-05-09T08:48:44.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.263+0000 7f5656363640 1 -- 172.21.15.107:0/1272556527 shutdown_connections 2024-05-09T08:48:44.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.263+0000 7f5656363640 1 --2- 172.21.15.107:0/1272556527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f56501010c0 0x7f565010d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.263+0000 7f5656363640 1 --2- 172.21.15.107:0/1272556527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5650100700 0x7f5650100b80 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.263+0000 7f5656363640 1 --2- 172.21.15.107:0/1272556527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5650106710 0x7f5650106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.263+0000 7f5656363640 1 -- 172.21.15.107:0/1272556527 >> 172.21.15.107:0/1272556527 conn(0x7f5650077830 msgr2=0x7f5650077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:44.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.263+0000 7f5656363640 1 -- 172.21.15.107:0/1272556527 shutdown_connections 2024-05-09T08:48:44.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.263+0000 7f5656363640 1 -- 172.21.15.107:0/1272556527 wait complete. 2024-05-09T08:48:44.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.263+0000 7f5656363640 1 Processor -- start 2024-05-09T08:48:44.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.263+0000 7f5656363640 1 -- start start 2024-05-09T08:48:44.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5656363640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5650100700 0x7f5650199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5656363640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56501010c0 0x7f5650199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5656363640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5650106710 0x7f56501a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5656363640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5650110a50 con 0x7f5650106710 2024-05-09T08:48:44.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5656363640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f56501108d0 con 0x7f5650100700 2024-05-09T08:48:44.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5656363640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5650110bd0 con 0x7f56501010c0 2024-05-09T08:48:44.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5654900640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5650106710 0x7f56501a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5654900640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5650106710 0x7f56501a4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57102/0 (socket says 172.21.15.107:57102) 2024-05-09T08:48:44.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5654900640 1 -- 172.21.15.107:0/2231609784 learned_addr learned my addr 172.21.15.107:0/2231609784 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:44.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f564f7fe640 1 --2- 172.21.15.107:0/2231609784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56501010c0 0x7f5650199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5654900640 1 -- 172.21.15.107:0/2231609784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56501010c0 msgr2=0x7f5650199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5654900640 1 --2- 172.21.15.107:0/2231609784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56501010c0 0x7f5650199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5654900640 1 -- 172.21.15.107:0/2231609784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5650100700 msgr2=0x7f5650199010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5654900640 1 --2- 172.21.15.107:0/2231609784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5650100700 0x7f5650199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5654900640 1 -- 172.21.15.107:0/2231609784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56501a45d0 con 0x7f5650106710 2024-05-09T08:48:44.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5654900640 1 --2- 172.21.15.107:0/2231609784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5650106710 0x7f56501a4090 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f564800b970 tx=0x7f564800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:44.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f564d7fa640 1 -- 172.21.15.107:0/2231609784 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5648019070 con 0x7f5650106710 2024-05-09T08:48:44.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5656363640 1 -- 172.21.15.107:0/2231609784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56501a47a0 con 0x7f5650106710 2024-05-09T08:48:44.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f564d7fa640 1 -- 172.21.15.107:0/2231609784 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5648014440 con 0x7f5650106710 2024-05-09T08:48:44.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f564d7fa640 1 -- 172.21.15.107:0/2231609784 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56480075d0 con 0x7f5650106710 2024-05-09T08:48:44.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.264+0000 7f5656363640 1 -- 172.21.15.107:0/2231609784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56501a4cc0 con 0x7f5650106710 2024-05-09T08:48:44.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.267+0000 7f564d7fa640 1 -- 172.21.15.107:0/2231609784 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f564801c070 con 0x7f5650106710 2024-05-09T08:48:44.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.267+0000 7f5659077740 1 Processor -- start 2024-05-09T08:48:44.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.267+0000 7f5659077740 1 -- start start 2024-05-09T08:48:44.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.267+0000 7f5659077740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc2fc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.267+0000 7f5659077740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 0x55ecdc306de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.267+0000 7f5659077740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc307320 0x55ecdc309710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.267+0000 7f5659077740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ecdc261c40 con 0x55ecdc2fc330 2024-05-09T08:48:44.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.267+0000 7f5659077740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ecdc209580 con 0x55ecdc307320 2024-05-09T08:48:44.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.267+0000 7f5659077740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ecdc25dc00 con 0x55ecdc2fcc50 2024-05-09T08:48:44.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.267+0000 7f564f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc2fc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.267+0000 7f564f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc2fc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57108/0 (socket says 172.21.15.107:57108) 2024-05-09T08:48:44.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.267+0000 7f564f7fe640 1 -- 172.21.15.107:0/1560295958 learned_addr learned my addr 172.21.15.107:0/1560295958 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:44.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.267+0000 7f564d7fa640 1 --2- 172.21.15.107:0/2231609784 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f56380a2d00 0x7f56380a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.267+0000 7f564ffff640 1 --2- 172.21.15.107:0/1560295958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 0x55ecdc306de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.267+0000 7f564f7fe640 1 --2- 172.21.15.107:0/1560295958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc307320 0x55ecdc309710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.268+0000 7f5654900640 1 --2- 172.21.15.107:0/2231609784 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f56380a2d00 0x7f56380a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.268+0000 7f564f7fe640 1 -- 172.21.15.107:0/1560295958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 msgr2=0x55ecdc306de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.268+0000 7f564f7fe640 1 --2- 172.21.15.107:0/1560295958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 0x55ecdc306de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.268+0000 7f564f7fe640 1 -- 172.21.15.107:0/1560295958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc307320 msgr2=0x55ecdc309710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.268+0000 7f564f7fe640 1 --2- 172.21.15.107:0/1560295958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc307320 0x55ecdc309710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.268+0000 7f564f7fe640 1 -- 172.21.15.107:0/1560295958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecdbf753f0 con 0x55ecdc2fc330 2024-05-09T08:48:44.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.268+0000 7f564ffff640 1 --2- 172.21.15.107:0/1560295958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 0x55ecdc306de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:44.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.268+0000 7f564d7fa640 1 -- 172.21.15.107:0/2231609784 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7f56480ced70 con 0x7f5650106710 2024-05-09T08:48:44.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.268+0000 7f564f7fe640 1 --2- 172.21.15.107:0/1560295958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc2fc710 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f5644005ab0 tx=0x7f56440058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44347d629358be2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:44.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.268+0000 7f56367fc640 1 -- 172.21.15.107:0/1560295958 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644035050 con 0x55ecdc2fc330 2024-05-09T08:48:44.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.268+0000 7f5654900640 1 --2- 172.21.15.107:0/2231609784 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f56380a2d00 0x7f56380a51c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f56480cc2d0 tx=0x7f56480660c0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:44.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.268+0000 7f56367fc640 1 -- 172.21.15.107:0/1560295958 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5644031840 con 0x55ecdc2fc330 2024-05-09T08:48:44.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.268+0000 7f56367fc640 1 -- 172.21.15.107:0/1560295958 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564403d450 con 0x55ecdc2fc330 2024-05-09T08:48:44.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.269+0000 7f5659077740 1 -- 172.21.15.107:0/1560295958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 msgr2=0x55ecdc2fc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.269+0000 7f5659077740 1 --2- 172.21.15.107:0/1560295958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc2fc710 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f5644005ab0 tx=0x7f56440058b0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.269+0000 7f5659077740 1 -- 172.21.15.107:0/1560295958 shutdown_connections 2024-05-09T08:48:44.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.269+0000 7f5659077740 1 --2- 172.21.15.107:0/1560295958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc307320 0x55ecdc309710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.269+0000 7f5659077740 1 --2- 172.21.15.107:0/1560295958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 0x55ecdc306de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.269+0000 7f5659077740 1 --2- 172.21.15.107:0/1560295958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc2fc710 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.269+0000 7f5659077740 1 -- 172.21.15.107:0/1560295958 >> 172.21.15.107:0/1560295958 conn(0x55ecdc2f8650 msgr2=0x55ecdc2f8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:44.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.269+0000 7f5659077740 1 -- 172.21.15.107:0/1560295958 shutdown_connections 2024-05-09T08:48:44.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.269+0000 7f5659077740 1 -- 172.21.15.107:0/1560295958 wait complete. 2024-05-09T08:48:44.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.269+0000 7f5659077740 1 -- start start 2024-05-09T08:48:44.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.269+0000 7f5659077740 1 Processor -- start 2024-05-09T08:48:44.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.269+0000 7f5659077740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc304270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.269+0000 7f5659077740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 0x55ecdc3047b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f5659077740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc307320 0x55ecdc304cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f5659077740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ecdc30a0b0 con 0x55ecdc2fc330 2024-05-09T08:48:44.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f5659077740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ecdc21cc40 con 0x55ecdc307320 2024-05-09T08:48:44.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f5659077740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55ecdc30a230 con 0x55ecdc2fcc50 2024-05-09T08:48:44.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f5659077740 10 client.0 _ll_register_callbacks cb 0x55ecdbf74be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:44.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f564f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc304270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f564f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc304270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57116/0 (socket says 172.21.15.107:57116) 2024-05-09T08:48:44.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f564f7fe640 1 -- 172.21.15.107:0/2163875801 learned_addr learned my addr 172.21.15.107:0/2163875801 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:44.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f564ffff640 1 --2- 172.21.15.107:0/2163875801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 0x55ecdc3047b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f564f7fe640 1 --2- 172.21.15.107:0/2163875801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc307320 0x55ecdc304cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f564f7fe640 1 -- 172.21.15.107:0/2163875801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 msgr2=0x55ecdc3047b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f564f7fe640 1 --2- 172.21.15.107:0/2163875801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 0x55ecdc3047b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f564f7fe640 1 -- 172.21.15.107:0/2163875801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc307320 msgr2=0x55ecdc304cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f564f7fe640 1 --2- 172.21.15.107:0/2163875801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc307320 0x55ecdc304cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f564f7fe640 1 -- 172.21.15.107:0/2163875801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecdc03f7d0 con 0x55ecdc2fc330 2024-05-09T08:48:44.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f564f7fe640 1 --2- 172.21.15.107:0/2163875801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc304270 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f5644000c00 tx=0x7f5644004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d0d2647c0655f45 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:44.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f562f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:48:44.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f562f7fe640 1 -- 172.21.15.107:0/2163875801 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644035030 con 0x55ecdc2fc330 2024-05-09T08:48:44.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f562f7fe640 1 -- 172.21.15.107:0/2163875801 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7f56440050b0 con 0x55ecdc2fc330 2024-05-09T08:48:44.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f5659077740 1 -- 172.21.15.107:0/2163875801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5644031840 con 0x55ecdc2fc330 2024-05-09T08:48:44.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f562f7fe640 1 -- 172.21.15.107:0/2163875801 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f56440052b0 con 0x55ecdc2fc330 2024-05-09T08:48:44.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f5659077740 10 client.19467 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.270+0000 7f5659077740 1 -- 172.21.15.107:0/2163875801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ecdc3162a0 con 0x55ecdc2fc330 2024-05-09T08:48:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.271+0000 7f562f7fe640 1 -- 172.21.15.107:0/2163875801 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5644031840 con 0x55ecdc2fc330 2024-05-09T08:48:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.271+0000 7f562f7fe640 1 -- 172.21.15.107:0/2163875801 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f564404b040 con 0x55ecdc2fc330 2024-05-09T08:48:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.271+0000 7f5659077740 10 client.19467 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.271+0000 7f5659077740 10 client.19467 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:48:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.271+0000 7f5659077740 20 client.19467 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.271+0000 7f5659077740 1 --2- 172.21.15.107:0/2163875801 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ecdc3175f0 0x55ecdc3199e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.271+0000 7f5659077740 4 client.19467 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T08:48:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.271+0000 7f5659077740 1 -- 172.21.15.107:0/2163875801 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecdc26f370 con 0x55ecdc3175f0 2024-05-09T08:48:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.271+0000 7f564ffff640 1 --2- 172.21.15.107:0/2163875801 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ecdc3175f0 0x55ecdc3199e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.271+0000 7f562f7fe640 10 client.19467 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:44.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.271+0000 7f564ffff640 1 --2- 172.21.15.107:0/2163875801 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ecdc3175f0 0x55ecdc3199e0 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1677ea0bdb6e242b server_cookie=5fb5e16cbcef5948 in_seq=0 out_seq=0 2024-05-09T08:48:44.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.272+0000 7f562f7fe640 1 -- 172.21.15.107:0/2163875801 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ecdc26f370 con 0x55ecdc3175f0 2024-05-09T08:48:44.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.272+0000 7f562f7fe640 10 client.19467 handle_command_reply: tid=0 2024-05-09T08:48:44.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.272+0000 7f5659077740 1 client.19467 shutdown 2024-05-09T08:48:44.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.272+0000 7f5659077740 1 -- 172.21.15.107:0/2163875801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 msgr2=0x55ecdc304270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.272+0000 7f5659077740 1 --2- 172.21.15.107:0/2163875801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc304270 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f5644000c00 tx=0x7f5644004b00 comp rx=0 tx=0).stop 2024-05-09T08:48:44.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.272+0000 7f5659077740 1 -- 172.21.15.107:0/2163875801 shutdown_connections 2024-05-09T08:48:44.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.272+0000 7f5659077740 1 --2- 172.21.15.107:0/2163875801 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ecdc3175f0 0x55ecdc3199e0 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.272+0000 7f5659077740 1 --2- 172.21.15.107:0/2163875801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc307320 0x55ecdc304cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.272+0000 7f5659077740 1 --2- 172.21.15.107:0/2163875801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 0x55ecdc3047b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.272+0000 7f5659077740 1 --2- 172.21.15.107:0/2163875801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc304270 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.272+0000 7f562f7fe640 0 client.19467 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:44.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.272+0000 7f5659077740 1 -- 172.21.15.107:0/2163875801 >> 172.21.15.107:0/2163875801 conn(0x55ecdc2f8650 msgr2=0x55ecdc2fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:44.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.272+0000 7f5659077740 1 -- 172.21.15.107:0/2163875801 shutdown_connections 2024-05-09T08:48:44.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.272+0000 7f5659077740 1 -- 172.21.15.107:0/2163875801 wait complete. 2024-05-09T08:48:44.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.273+0000 7f5659077740 20 client.19467 trim_cache size 0 max 16384 2024-05-09T08:48:44.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.285+0000 7f5659077740 1 Processor -- start 2024-05-09T08:48:44.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f5659077740 1 -- start start 2024-05-09T08:48:44.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f5659077740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc2fc330 0x55ecdc3069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f5659077740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 0x55ecdc301510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f5659077740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc307320 0x55ecdc301a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f5659077740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f56501b1560 con 0x55ecdc307320 2024-05-09T08:48:44.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f5659077740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f56501aedd0 con 0x55ecdc2fc330 2024-05-09T08:48:44.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f5659077740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f56501acc40 con 0x55ecdc2fcc50 2024-05-09T08:48:44.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f564f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc307320 0x55ecdc301a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f564f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc307320 0x55ecdc301a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57126/0 (socket says 172.21.15.107:57126) 2024-05-09T08:48:44.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f564f7fe640 1 -- 172.21.15.107:0/2454731692 learned_addr learned my addr 172.21.15.107:0/2454731692 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:44.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f564f7fe640 1 -- 172.21.15.107:0/2454731692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 msgr2=0x55ecdc301510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f564f7fe640 1 --2- 172.21.15.107:0/2454731692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 0x55ecdc301510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f564f7fe640 1 -- 172.21.15.107:0/2454731692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc2fc330 msgr2=0x55ecdc3069a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f564f7fe640 1 --2- 172.21.15.107:0/2454731692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc2fc330 0x55ecdc3069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f564f7fe640 1 -- 172.21.15.107:0/2454731692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecdc2f3610 con 0x55ecdc307320 2024-05-09T08:48:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f564f7fe640 1 --2- 172.21.15.107:0/2454731692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc307320 0x55ecdc301a50 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f5644004b70 tx=0x7f5644031ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f9e8f2da71b03eb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f5635ffb640 1 -- 172.21.15.107:0/2454731692 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644035030 con 0x55ecdc307320 2024-05-09T08:48:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f5635ffb640 1 -- 172.21.15.107:0/2454731692 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5644004eb0 con 0x55ecdc307320 2024-05-09T08:48:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.286+0000 7f5635ffb640 1 -- 172.21.15.107:0/2454731692 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644004290 con 0x55ecdc307320 2024-05-09T08:48:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.287+0000 7f5659077740 1 -- 172.21.15.107:0/2454731692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc307320 msgr2=0x55ecdc301a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.287+0000 7f5659077740 1 --2- 172.21.15.107:0/2454731692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc307320 0x55ecdc301a50 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f5644004b70 tx=0x7f5644031ea0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.287+0000 7f5659077740 1 -- 172.21.15.107:0/2454731692 shutdown_connections 2024-05-09T08:48:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.287+0000 7f5659077740 1 --2- 172.21.15.107:0/2454731692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc307320 0x55ecdc301a50 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.287+0000 7f5659077740 1 --2- 172.21.15.107:0/2454731692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 0x55ecdc301510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.287+0000 7f5659077740 1 --2- 172.21.15.107:0/2454731692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc2fc330 0x55ecdc3069a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.287+0000 7f5659077740 1 -- 172.21.15.107:0/2454731692 >> 172.21.15.107:0/2454731692 conn(0x55ecdc2f8650 msgr2=0x55ecdc2f77b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.287+0000 7f5659077740 1 -- 172.21.15.107:0/2454731692 shutdown_connections 2024-05-09T08:48:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.287+0000 7f5659077740 1 -- 172.21.15.107:0/2454731692 wait complete. 2024-05-09T08:48:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.287+0000 7f5659077740 1 -- start start 2024-05-09T08:48:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.287+0000 7f5659077740 1 Processor -- start 2024-05-09T08:48:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f5659077740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc31a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f5659077740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 0x55ecdc31aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f5659077740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc307320 0x55ecdc31aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f5659077740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f56501ad290 con 0x55ecdc2fc330 2024-05-09T08:48:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f5659077740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f56501ae700 con 0x55ecdc307320 2024-05-09T08:48:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f5659077740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f56501ad960 con 0x55ecdc2fcc50 2024-05-09T08:48:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f5659077740 10 client.0 _ll_register_callbacks cb 0x7f5644031f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T08:48:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f564f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc31a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f564f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc31a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57130/0 (socket says 172.21.15.107:57130) 2024-05-09T08:48:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f564f7fe640 1 -- 172.21.15.107:0/1030718811 learned_addr learned my addr 172.21.15.107:0/1030718811 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f564f7fe640 1 -- 172.21.15.107:0/1030718811 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 msgr2=0x55ecdc31aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f564f7fe640 1 --2- 172.21.15.107:0/1030718811 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 0x55ecdc31aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f564f7fe640 1 -- 172.21.15.107:0/1030718811 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc307320 msgr2=0x55ecdc31aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T08:48:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f564f7fe640 1 --2- 172.21.15.107:0/1030718811 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc307320 0x55ecdc31aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f564f7fe640 1 -- 172.21.15.107:0/1030718811 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecdc37c8a0 con 0x55ecdc2fc330 2024-05-09T08:48:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f564f7fe640 1 --2- 172.21.15.107:0/1030718811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc31a570 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f5644005790 tx=0x7f5644005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e53cb5c4560a3d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f562f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-05-09T08:48:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f562f7fe640 1 -- 172.21.15.107:0/1030718811 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644035070 con 0x55ecdc2fc330 2024-05-09T08:48:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f562f7fe640 1 -- 172.21.15.107:0/1030718811 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7f5644005060 con 0x55ecdc2fc330 2024-05-09T08:48:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f5659077740 1 -- 172.21.15.107:0/1030718811 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5644004eb0 con 0x55ecdc2fc330 2024-05-09T08:48:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f562f7fe640 1 -- 172.21.15.107:0/1030718811 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f564403d880 con 0x55ecdc2fc330 2024-05-09T08:48:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f5659077740 10 client.19473 fetch_fsmap learned FSMap version 31 2024-05-09T08:48:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.288+0000 7f5659077740 1 -- 172.21.15.107:0/1030718811 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ecdc37dba0 con 0x55ecdc2fc330 2024-05-09T08:48:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.289+0000 7f562f7fe640 1 -- 172.21.15.107:0/1030718811 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5644004eb0 con 0x55ecdc2fc330 2024-05-09T08:48:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.289+0000 7f562f7fe640 1 -- 172.21.15.107:0/1030718811 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f564403a7b0 con 0x55ecdc2fc330 2024-05-09T08:48:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.289+0000 7f5659077740 10 client.19473 fetch_fsmap finished waiting for FSMap version 31 2024-05-09T08:48:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.289+0000 7f5659077740 10 client.19473 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-05-09T08:48:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.289+0000 7f5659077740 20 client.19473 populate_metadata read hostname 'smithi107' 2024-05-09T08:48:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.289+0000 7f5659077740 1 --2- 172.21.15.107:0/1030718811 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ecdc37e060 0x55ecdc380450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.289+0000 7f5659077740 4 client.19473 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T08:48:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.289+0000 7f5659077740 1 -- 172.21.15.107:0/1030718811 --> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ecdc2696c0 con 0x55ecdc37e060 2024-05-09T08:48:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.289+0000 7f564ffff640 1 --2- 172.21.15.107:0/1030718811 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ecdc37e060 0x55ecdc380450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.289+0000 7f562f7fe640 10 client.19473 ms_handle_connect on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.289+0000 7f564ffff640 1 --2- 172.21.15.107:0/1030718811 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ecdc37e060 0x55ecdc380450 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ccf77e89e5553f8 server_cookie=f3bf9ff184409c78 in_seq=0 out_seq=0 2024-05-09T08:48:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f562f7fe640 1 -- 172.21.15.107:0/1030718811 <== mds.0 v2:172.21.15.157:6838/3003356919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ecdc2696c0 con 0x55ecdc37e060 2024-05-09T08:48:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f562f7fe640 10 client.19473 handle_command_reply: tid=0 2024-05-09T08:48:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f5659077740 1 client.19473 shutdown 2024-05-09T08:48:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f5659077740 1 -- 172.21.15.107:0/1030718811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 msgr2=0x55ecdc31a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f5659077740 1 --2- 172.21.15.107:0/1030718811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc31a570 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f5644005790 tx=0x7f5644005e80 comp rx=0 tx=0).stop 2024-05-09T08:48:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f5659077740 1 -- 172.21.15.107:0/1030718811 shutdown_connections 2024-05-09T08:48:44.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f5659077740 1 --2- 172.21.15.107:0/1030718811 >> [v2:172.21.15.157:6838/3003356919,v1:172.21.15.157:6839/3003356919] conn(0x55ecdc37e060 0x55ecdc380450 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f5659077740 1 --2- 172.21.15.107:0/1030718811 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ecdc307320 0x55ecdc31aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f5659077740 1 --2- 172.21.15.107:0/1030718811 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ecdc2fcc50 0x55ecdc31aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f562f7fe640 0 client.19473 ms_handle_reset on v2:172.21.15.157:6838/3003356919 2024-05-09T08:48:44.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f5659077740 1 --2- 172.21.15.107:0/1030718811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ecdc2fc330 0x55ecdc31a570 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f5659077740 1 -- 172.21.15.107:0/1030718811 >> 172.21.15.107:0/1030718811 conn(0x55ecdc2f8650 msgr2=0x55ecdc304230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:44.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f5659077740 1 -- 172.21.15.107:0/1030718811 shutdown_connections 2024-05-09T08:48:44.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f5659077740 1 -- 172.21.15.107:0/1030718811 wait complete. 2024-05-09T08:48:44.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f5659077740 20 client.19473 trim_cache size 0 max 16384 2024-05-09T08:48:44.300 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-05-09T08:48:44.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f5656363640 1 -- 172.21.15.107:0/2231609784 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f56380a2d00 msgr2=0x7f56380a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f5656363640 1 --2- 172.21.15.107:0/2231609784 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f56380a2d00 0x7f56380a51c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f56480cc2d0 tx=0x7f56480660c0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f5656363640 1 -- 172.21.15.107:0/2231609784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5650106710 msgr2=0x7f56501a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:44.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.290+0000 7f5656363640 1 --2- 172.21.15.107:0/2231609784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5650106710 0x7f56501a4090 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f564800b970 tx=0x7f564800be30 comp rx=0 tx=0).stop 2024-05-09T08:48:44.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.291+0000 7f5656363640 1 -- 172.21.15.107:0/2231609784 shutdown_connections 2024-05-09T08:48:44.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.291+0000 7f5656363640 1 --2- 172.21.15.107:0/2231609784 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f56380a2d00 0x7f56380a51c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.291+0000 7f5656363640 1 --2- 172.21.15.107:0/2231609784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5650106710 0x7f56501a4090 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.291+0000 7f5656363640 1 --2- 172.21.15.107:0/2231609784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56501010c0 0x7f5650199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.291+0000 7f5656363640 1 --2- 172.21.15.107:0/2231609784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5650100700 0x7f5650199010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:44.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.291+0000 7f5656363640 1 -- 172.21.15.107:0/2231609784 >> 172.21.15.107:0/2231609784 conn(0x7f5650077830 msgr2=0x7f5650103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:44.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.291+0000 7f5656363640 1 -- 172.21.15.107:0/2231609784 shutdown_connections 2024-05-09T08:48:44.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:44.291+0000 7f5656363640 1 -- 172.21.15.107:0/2231609784 wait complete. 2024-05-09T08:48:44.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:44 smithi107 ceph-mon[30122]: pgmap v483: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.9 KiB/s wr, 7 op/s 2024-05-09T08:48:44.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:44 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3390139377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:44.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:44 smithi157 ceph-mon[33614]: pgmap v483: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.9 KiB/s wr, 7 op/s 2024-05-09T08:48:44.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:44 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3390139377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:44.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:44 smithi110 ceph-mon[33622]: pgmap v483: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.9 KiB/s wr, 7 op/s 2024-05-09T08:48:44.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:44 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3390139377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:48:45.303 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-05-09T08:48:45.304 INFO:tasks.fwd_scrub:done joining 2024-05-09T08:48:45.304 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-05-09T08:48:45.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:48:45.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.426+0000 7f7366d90640 1 -- 172.21.15.107:0/3837252497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f736010be00 msgr2=0x7f736010e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:45.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.426+0000 7f7366d90640 1 --2- 172.21.15.107:0/3837252497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f736010be00 0x7f736010e2a0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f7358009900 tx=0x7f735802a510 comp rx=0 tx=0).stop 2024-05-09T08:48:45.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.426+0000 7f7366d90640 1 -- 172.21.15.107:0/3837252497 shutdown_connections 2024-05-09T08:48:45.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.426+0000 7f7366d90640 1 --2- 172.21.15.107:0/3837252497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f736010be00 0x7f736010e2a0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.426+0000 7f7366d90640 1 --2- 172.21.15.107:0/3837252497 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73601033e0 0x7f736010b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.426+0000 7f7366d90640 1 --2- 172.21.15.107:0/3837252497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7360102ac0 0x7f7360102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.426+0000 7f7366d90640 1 -- 172.21.15.107:0/3837252497 >> 172.21.15.107:0/3837252497 conn(0x7f7360077830 msgr2=0x7f7360077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:45.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.427+0000 7f7366d90640 1 -- 172.21.15.107:0/3837252497 shutdown_connections 2024-05-09T08:48:45.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.427+0000 7f7366d90640 1 -- 172.21.15.107:0/3837252497 wait complete. 2024-05-09T08:48:45.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.427+0000 7f7366d90640 1 Processor -- start 2024-05-09T08:48:45.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.427+0000 7f7366d90640 1 -- start start 2024-05-09T08:48:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.427+0000 7f7366d90640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7360102ac0 0x7f7360197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.427+0000 7f7366d90640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73601033e0 0x7f7360198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.427+0000 7f7366d90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f736010be00 0x7f736019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.427+0000 7f7366d90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7360111ab0 con 0x7f736010be00 2024-05-09T08:48:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.427+0000 7f7366d90640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7360111930 con 0x7f7360102ac0 2024-05-09T08:48:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.427+0000 7f7366d90640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7360111c30 con 0x7f73601033e0 2024-05-09T08:48:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.427+0000 7f736532d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f736010be00 0x7f736019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.427+0000 7f736532d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f736010be00 0x7f736019c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57150/0 (socket says 172.21.15.107:57150) 2024-05-09T08:48:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.427+0000 7f736532d640 1 -- 172.21.15.107:0/4184520522 learned_addr learned my addr 172.21.15.107:0/4184520522 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.428+0000 7f7364b2c640 1 --2- 172.21.15.107:0/4184520522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7360102ac0 0x7f7360197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.428+0000 7f735ffff640 1 --2- 172.21.15.107:0/4184520522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73601033e0 0x7f7360198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.428+0000 7f736532d640 1 -- 172.21.15.107:0/4184520522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73601033e0 msgr2=0x7f7360198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.428+0000 7f736532d640 1 --2- 172.21.15.107:0/4184520522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73601033e0 0x7f7360198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.428+0000 7f736532d640 1 -- 172.21.15.107:0/4184520522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7360102ac0 msgr2=0x7f7360197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:45.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.428+0000 7f736532d640 1 --2- 172.21.15.107:0/4184520522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7360102ac0 0x7f7360197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.428+0000 7f736532d640 1 -- 172.21.15.107:0/4184520522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736019cf70 con 0x7f736010be00 2024-05-09T08:48:45.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.428+0000 7f736532d640 1 --2- 172.21.15.107:0/4184520522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f736010be00 0x7f736019c840 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f7358009230 tx=0x7f7358004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:45.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.428+0000 7f735dffb640 1 -- 172.21.15.107:0/4184520522 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735803f030 con 0x7f736010be00 2024-05-09T08:48:45.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.428+0000 7f7366d90640 1 -- 172.21.15.107:0/4184520522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f736019d200 con 0x7f736010be00 2024-05-09T08:48:45.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.428+0000 7f735dffb640 1 -- 172.21.15.107:0/4184520522 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7358003aa0 con 0x7f736010be00 2024-05-09T08:48:45.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.428+0000 7f735dffb640 1 -- 172.21.15.107:0/4184520522 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7358035070 con 0x7f736010be00 2024-05-09T08:48:45.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.428+0000 7f7366d90640 1 -- 172.21.15.107:0/4184520522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73601a4af0 con 0x7f736010be00 2024-05-09T08:48:45.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.429+0000 7f7366d90640 1 -- 172.21.15.107:0/4184520522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7360068650 con 0x7f736010be00 2024-05-09T08:48:45.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.431+0000 7f735dffb640 1 -- 172.21.15.107:0/4184520522 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7358042030 con 0x7f736010be00 2024-05-09T08:48:45.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.431+0000 7f735dffb640 1 --2- 172.21.15.107:0/4184520522 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f734c0a2d00 0x7f734c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:45.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.431+0000 7f7364b2c640 1 --2- 172.21.15.107:0/4184520522 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f734c0a2d00 0x7f734c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:45.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.431+0000 7f735dffb640 1 -- 172.21.15.107:0/4184520522 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7f73580f03f0 con 0x7f736010be00 2024-05-09T08:48:45.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.432+0000 7f7364b2c640 1 --2- 172.21.15.107:0/4184520522 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f734c0a2d00 0x7f734c0a51c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7350005fd0 tx=0x7f73500098b0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:45.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.435+0000 7f735dffb640 1 -- 172.21.15.107:0/4184520522 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f73580bcc60 con 0x7f736010be00 2024-05-09T08:48:45.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.612+0000 7f7366d90640 1 -- 172.21.15.107:0/4184520522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7360106f80 con 0x7f736010be00 2024-05-09T08:48:45.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.614+0000 7f735dffb640 1 -- 172.21.15.107:0/4184520522 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+25331 (secure 0 0 0) 0x7f73580bc560 con 0x7f736010be00 2024-05-09T08:48:45.615 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:48:45.615 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":113,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:48:22.038911+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":25,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":25,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/2923766500":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/2417648229":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1642826815":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/2689615560":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/846948416":"2024-05-10T08:33:20.828572+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/846948416":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":25,"length":1}]},{"pool":4,"snaps":[{"begin":12,"length":1},{"begin":25,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:48:45.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.616+0000 7f7366d90640 1 -- 172.21.15.107:0/4184520522 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f734c0a2d00 msgr2=0x7f734c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:45.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.616+0000 7f7366d90640 1 --2- 172.21.15.107:0/4184520522 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f734c0a2d00 0x7f734c0a51c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7350005fd0 tx=0x7f73500098b0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.616+0000 7f7366d90640 1 -- 172.21.15.107:0/4184520522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f736010be00 msgr2=0x7f736019c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:45.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.616+0000 7f7366d90640 1 --2- 172.21.15.107:0/4184520522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f736010be00 0x7f736019c840 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f7358009230 tx=0x7f7358004270 comp rx=0 tx=0).stop 2024-05-09T08:48:45.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.616+0000 7f7366d90640 1 -- 172.21.15.107:0/4184520522 shutdown_connections 2024-05-09T08:48:45.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.616+0000 7f7366d90640 1 --2- 172.21.15.107:0/4184520522 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f734c0a2d00 0x7f734c0a51c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.616+0000 7f7366d90640 1 --2- 172.21.15.107:0/4184520522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f736010be00 0x7f736019c840 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.616+0000 7f7366d90640 1 --2- 172.21.15.107:0/4184520522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73601033e0 0x7f7360198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.616+0000 7f7366d90640 1 --2- 172.21.15.107:0/4184520522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7360102ac0 0x7f7360197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.616+0000 7f7366d90640 1 -- 172.21.15.107:0/4184520522 >> 172.21.15.107:0/4184520522 conn(0x7f7360077830 msgr2=0x7f73600fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:45.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.616+0000 7f7366d90640 1 -- 172.21.15.107:0/4184520522 shutdown_connections 2024-05-09T08:48:45.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.616+0000 7f7366d90640 1 -- 172.21.15.107:0/4184520522 wait complete. 2024-05-09T08:48:45.628 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:26:21.698338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:30:58.268633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:30:59.345579+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 25, 'snap_epoch': 112, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:31:03.065991+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 25, 'snap_epoch': 112, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:48:45.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:48:45.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.733+0000 7f4826941640 1 -- 172.21.15.107:0/188230843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4820069eb0 msgr2=0x7f482006a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:45.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.733+0000 7f4826941640 1 --2- 172.21.15.107:0/188230843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4820069eb0 0x7f482006a330 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f4808009930 tx=0x7f480802a420 comp rx=0 tx=0).stop 2024-05-09T08:48:45.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.734+0000 7f4826941640 1 -- 172.21.15.107:0/188230843 shutdown_connections 2024-05-09T08:48:45.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.734+0000 7f4826941640 1 --2- 172.21.15.107:0/188230843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f482006a870 0x7f482010e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.734+0000 7f4826941640 1 --2- 172.21.15.107:0/188230843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4820069eb0 0x7f482006a330 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.734+0000 7f4826941640 1 --2- 172.21.15.107:0/188230843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4820107820 0x7f4820107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.734+0000 7f4826941640 1 -- 172.21.15.107:0/188230843 >> 172.21.15.107:0/188230843 conn(0x7f4820077790 msgr2=0x7f4820077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:45.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.734+0000 7f4826941640 1 -- 172.21.15.107:0/188230843 shutdown_connections 2024-05-09T08:48:45.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.734+0000 7f4826941640 1 -- 172.21.15.107:0/188230843 wait complete. 2024-05-09T08:48:45.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.734+0000 7f4826941640 1 Processor -- start 2024-05-09T08:48:45.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.734+0000 7f4826941640 1 -- start start 2024-05-09T08:48:45.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.734+0000 7f4826941640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4820069eb0 0x7f4820198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:45.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.734+0000 7f4826941640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f482006a870 0x7f4820198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:45.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.734+0000 7f4826941640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4820107820 0x7f482019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:45.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.734+0000 7f4826941640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4820111ae0 con 0x7f4820069eb0 2024-05-09T08:48:45.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.734+0000 7f4826941640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4820111960 con 0x7f482006a870 2024-05-09T08:48:45.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.734+0000 7f4826941640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4820111c60 con 0x7f4820107820 2024-05-09T08:48:45.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.734+0000 7f481ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4820069eb0 0x7f4820198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:45.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.735+0000 7f481ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4820069eb0 0x7f4820198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57158/0 (socket says 172.21.15.107:57158) 2024-05-09T08:48:45.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.735+0000 7f481ffff640 1 -- 172.21.15.107:0/2038366852 learned_addr learned my addr 172.21.15.107:0/2038366852 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:45.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.735+0000 7f4817fff640 1 --2- 172.21.15.107:0/2038366852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f482006a870 0x7f4820198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:45.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.735+0000 7f4824ede640 1 --2- 172.21.15.107:0/2038366852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4820107820 0x7f482019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:45.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.735+0000 7f481ffff640 1 -- 172.21.15.107:0/2038366852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4820107820 msgr2=0x7f482019c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:45.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.735+0000 7f481ffff640 1 --2- 172.21.15.107:0/2038366852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4820107820 0x7f482019c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.735+0000 7f481ffff640 1 -- 172.21.15.107:0/2038366852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f482006a870 msgr2=0x7f4820198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:45.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.735+0000 7f481ffff640 1 --2- 172.21.15.107:0/2038366852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f482006a870 0x7f4820198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.735+0000 7f481ffff640 1 -- 172.21.15.107:0/2038366852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f482019d030 con 0x7f4820069eb0 2024-05-09T08:48:45.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.735+0000 7f481ffff640 1 --2- 172.21.15.107:0/2038366852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4820069eb0 0x7f4820198040 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f481000d770 tx=0x7f481000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:45.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.735+0000 7f481dffb640 1 -- 172.21.15.107:0/2038366852 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4810017070 con 0x7f4820069eb0 2024-05-09T08:48:45.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.735+0000 7f4826941640 1 -- 172.21.15.107:0/2038366852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f482019d320 con 0x7f4820069eb0 2024-05-09T08:48:45.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.735+0000 7f481dffb640 1 -- 172.21.15.107:0/2038366852 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48100054d0 con 0x7f4820069eb0 2024-05-09T08:48:45.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.735+0000 7f481dffb640 1 -- 172.21.15.107:0/2038366852 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4810004cf0 con 0x7f4820069eb0 2024-05-09T08:48:45.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.735+0000 7f4826941640 1 -- 172.21.15.107:0/2038366852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48201a4bb0 con 0x7f4820069eb0 2024-05-09T08:48:45.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.736+0000 7f4826941640 1 -- 172.21.15.107:0/2038366852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4820102f50 con 0x7f4820069eb0 2024-05-09T08:48:45.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.738+0000 7f481dffb640 1 -- 172.21.15.107:0/2038366852 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f4810004e90 con 0x7f4820069eb0 2024-05-09T08:48:45.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.739+0000 7f481dffb640 1 --2- 172.21.15.107:0/2038366852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f480c0a2d00 0x7f480c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:45.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.739+0000 7f4817fff640 1 --2- 172.21.15.107:0/2038366852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f480c0a2d00 0x7f480c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:45.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.739+0000 7f481dffb640 1 -- 172.21.15.107:0/2038366852 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7f48100cf690 con 0x7f4820069eb0 2024-05-09T08:48:45.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.739+0000 7f4817fff640 1 --2- 172.21.15.107:0/2038366852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f480c0a2d00 0x7f480c0a51c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4808000c00 tx=0x7f4808032ba0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:45.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.742+0000 7f481dffb640 1 -- 172.21.15.107:0/2038366852 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f481009bf00 con 0x7f4820069eb0 2024-05-09T08:48:45.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.917+0000 7f4826941640 1 -- 172.21.15.107:0/2038366852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4820068aa0 con 0x7f4820069eb0 2024-05-09T08:48:45.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.918+0000 7f481dffb640 1 -- 172.21.15.107:0/2038366852 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) ==== 93+0+10 (secure 0 0 0) 0x7f481009b800 con 0x7f4820069eb0 2024-05-09T08:48:45.918 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-05-09T08:48:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.920+0000 7f4826941640 1 -- 172.21.15.107:0/2038366852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f480c0a2d00 msgr2=0x7f480c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.920+0000 7f4826941640 1 --2- 172.21.15.107:0/2038366852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f480c0a2d00 0x7f480c0a51c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4808000c00 tx=0x7f4808032ba0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.920+0000 7f4826941640 1 -- 172.21.15.107:0/2038366852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4820069eb0 msgr2=0x7f4820198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.920+0000 7f4826941640 1 --2- 172.21.15.107:0/2038366852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4820069eb0 0x7f4820198040 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f481000d770 tx=0x7f481000dc30 comp rx=0 tx=0).stop 2024-05-09T08:48:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.920+0000 7f4826941640 1 -- 172.21.15.107:0/2038366852 shutdown_connections 2024-05-09T08:48:45.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.920+0000 7f4826941640 1 --2- 172.21.15.107:0/2038366852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f480c0a2d00 0x7f480c0a51c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.920+0000 7f4826941640 1 --2- 172.21.15.107:0/2038366852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4820107820 0x7f482019c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.920+0000 7f4826941640 1 --2- 172.21.15.107:0/2038366852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f482006a870 0x7f4820198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.920+0000 7f4826941640 1 --2- 172.21.15.107:0/2038366852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4820069eb0 0x7f4820198040 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:45.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.920+0000 7f4826941640 1 -- 172.21.15.107:0/2038366852 >> 172.21.15.107:0/2038366852 conn(0x7f4820077790 msgr2=0x7f48200fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:45.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.920+0000 7f4826941640 1 -- 172.21.15.107:0/2038366852 shutdown_connections 2024-05-09T08:48:45.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:45.920+0000 7f4826941640 1 -- 172.21.15.107:0/2038366852 wait complete. 2024-05-09T08:48:45.931 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:48:46.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.035+0000 7f2e58934640 1 -- 172.21.15.107:0/3331016385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e541006f0 msgr2=0x7f2e5410e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.035+0000 7f2e58934640 1 --2- 172.21.15.107:0/3331016385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e541006f0 0x7f2e5410e6c0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c009cd0 tx=0x7f2e4c02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.035+0000 7f2e58934640 1 -- 172.21.15.107:0/3331016385 shutdown_connections 2024-05-09T08:48:46.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.035+0000 7f2e58934640 1 --2- 172.21.15.107:0/3331016385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e541006f0 0x7f2e5410e6c0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.035+0000 7f2e58934640 1 --2- 172.21.15.107:0/3331016385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e540ffd30 0x7f2e541001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.035+0000 7f2e58934640 1 --2- 172.21.15.107:0/3331016385 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e54106810 0x7f2e54106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.035+0000 7f2e58934640 1 -- 172.21.15.107:0/3331016385 >> 172.21.15.107:0/3331016385 conn(0x7f2e54076cf0 msgr2=0x7f2e540fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:46.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.035+0000 7f2e58934640 1 -- 172.21.15.107:0/3331016385 shutdown_connections 2024-05-09T08:48:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.035+0000 7f2e58934640 1 -- 172.21.15.107:0/3331016385 wait complete. 2024-05-09T08:48:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.035+0000 7f2e58934640 1 Processor -- start 2024-05-09T08:48:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.035+0000 7f2e58934640 1 -- start start 2024-05-09T08:48:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e58934640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e540ffd30 0x7f2e54198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e58934640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e541006f0 0x7f2e541986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e58934640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e54106810 0x7f2e5419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e58934640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2e54111ca0 con 0x7f2e541006f0 2024-05-09T08:48:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e58934640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2e54111b20 con 0x7f2e540ffd30 2024-05-09T08:48:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e58934640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2e54111e20 con 0x7f2e54106810 2024-05-09T08:48:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e4bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e541006f0 0x7f2e541986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e4bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e541006f0 0x7f2e541986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57176/0 (socket says 172.21.15.107:57176) 2024-05-09T08:48:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e4bfff640 1 -- 172.21.15.107:0/2772890828 learned_addr learned my addr 172.21.15.107:0/2772890828 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e5259c640 1 --2- 172.21.15.107:0/2772890828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e540ffd30 0x7f2e54198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e52d9d640 1 --2- 172.21.15.107:0/2772890828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e54106810 0x7f2e5419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e4bfff640 1 -- 172.21.15.107:0/2772890828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e54106810 msgr2=0x7f2e5419ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e4bfff640 1 --2- 172.21.15.107:0/2772890828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e54106810 0x7f2e5419ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e4bfff640 1 -- 172.21.15.107:0/2772890828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e540ffd30 msgr2=0x7f2e54198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e4bfff640 1 --2- 172.21.15.107:0/2772890828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e540ffd30 0x7f2e54198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e4bfff640 1 -- 172.21.15.107:0/2772890828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e5419d170 con 0x7f2e541006f0 2024-05-09T08:48:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e5259c640 1 --2- 172.21.15.107:0/2772890828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e540ffd30 0x7f2e54198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e4bfff640 1 --2- 172.21.15.107:0/2772890828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e541006f0 0x7f2e541986c0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c00b4a0 tx=0x7f2e3c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e4b7fe640 1 -- 172.21.15.107:0/2772890828 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c014070 con 0x7f2e541006f0 2024-05-09T08:48:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e4b7fe640 1 -- 172.21.15.107:0/2772890828 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e3c010ac0 con 0x7f2e541006f0 2024-05-09T08:48:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e58934640 1 -- 172.21.15.107:0/2772890828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e5419d460 con 0x7f2e541006f0 2024-05-09T08:48:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e4b7fe640 1 -- 172.21.15.107:0/2772890828 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c017c30 con 0x7f2e541006f0 2024-05-09T08:48:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.036+0000 7f2e58934640 1 -- 172.21.15.107:0/2772890828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e541a4cf0 con 0x7f2e541006f0 2024-05-09T08:48:46.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.037+0000 7f2e58934640 1 -- 172.21.15.107:0/2772890828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e54068aa0 con 0x7f2e541006f0 2024-05-09T08:48:46.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.038+0000 7f2e4b7fe640 1 -- 172.21.15.107:0/2772890828 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f2e3c017070 con 0x7f2e541006f0 2024-05-09T08:48:46.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.039+0000 7f2e4b7fe640 1 --2- 172.21.15.107:0/2772890828 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2e400a2d00 0x7f2e400a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.039+0000 7f2e5259c640 1 --2- 172.21.15.107:0/2772890828 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2e400a2d00 0x7f2e400a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.039+0000 7f2e4b7fe640 1 -- 172.21.15.107:0/2772890828 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7f2e3c0cee70 con 0x7f2e541006f0 2024-05-09T08:48:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.040+0000 7f2e5259c640 1 --2- 172.21.15.107:0/2772890828 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2e400a2d00 0x7f2e400a51c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2e44006fd0 tx=0x7f2e44008040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.044+0000 7f2e4b7fe640 1 -- 172.21.15.107:0/2772890828 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2e3c09b7f0 con 0x7f2e541006f0 2024-05-09T08:48:46.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.218+0000 7f2e58934640 1 -- 172.21.15.107:0/2772890828 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2e5402d1c0 con 0x7f2e541006f0 2024-05-09T08:48:46.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.219+0000 7f2e4b7fe640 1 -- 172.21.15.107:0/2772890828 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) ==== 104+0+11 (secure 0 0 0) 0x7f2e3c09b0f0 con 0x7f2e541006f0 2024-05-09T08:48:46.219 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:48:46.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.221+0000 7f2e58934640 1 -- 172.21.15.107:0/2772890828 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2e400a2d00 msgr2=0x7f2e400a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.221+0000 7f2e58934640 1 --2- 172.21.15.107:0/2772890828 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2e400a2d00 0x7f2e400a51c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2e44006fd0 tx=0x7f2e44008040 comp rx=0 tx=0).stop 2024-05-09T08:48:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.221+0000 7f2e58934640 1 -- 172.21.15.107:0/2772890828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e541006f0 msgr2=0x7f2e541986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.221+0000 7f2e58934640 1 --2- 172.21.15.107:0/2772890828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e541006f0 0x7f2e541986c0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c00b4a0 tx=0x7f2e3c00b960 comp rx=0 tx=0).stop 2024-05-09T08:48:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.221+0000 7f2e58934640 1 -- 172.21.15.107:0/2772890828 shutdown_connections 2024-05-09T08:48:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.221+0000 7f2e58934640 1 --2- 172.21.15.107:0/2772890828 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f2e400a2d00 0x7f2e400a51c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.221+0000 7f2e58934640 1 --2- 172.21.15.107:0/2772890828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e54106810 0x7f2e5419ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.221+0000 7f2e58934640 1 --2- 172.21.15.107:0/2772890828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e541006f0 0x7f2e541986c0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.221+0000 7f2e58934640 1 --2- 172.21.15.107:0/2772890828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e540ffd30 0x7f2e54198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.221+0000 7f2e58934640 1 -- 172.21.15.107:0/2772890828 >> 172.21.15.107:0/2772890828 conn(0x7f2e54076cf0 msgr2=0x7f2e541048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.221+0000 7f2e58934640 1 -- 172.21.15.107:0/2772890828 shutdown_connections 2024-05-09T08:48:46.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.221+0000 7f2e58934640 1 -- 172.21.15.107:0/2772890828 wait complete. 2024-05-09T08:48:46.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:48:46.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.336+0000 7f0f9a130640 1 -- 172.21.15.107:0/646561659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f9406a870 msgr2=0x7f0f9410e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.336+0000 7f0f9a130640 1 --2- 172.21.15.107:0/646561659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f9406a870 0x7f0f9410e6e0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c009930 tx=0x7f0f8c02a420 comp rx=0 tx=0).stop 2024-05-09T08:48:46.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.336+0000 7f0f9a130640 1 -- 172.21.15.107:0/646561659 shutdown_connections 2024-05-09T08:48:46.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.336+0000 7f0f9a130640 1 --2- 172.21.15.107:0/646561659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f9406a870 0x7f0f9410e6e0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.336+0000 7f0f9a130640 1 --2- 172.21.15.107:0/646561659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f94069eb0 0x7f0f9406a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.336+0000 7f0f9a130640 1 --2- 172.21.15.107:0/646561659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f94107820 0x7f0f94107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.336+0000 7f0f9a130640 1 -- 172.21.15.107:0/646561659 >> 172.21.15.107:0/646561659 conn(0x7f0f94077790 msgr2=0x7f0f94077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:46.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.336+0000 7f0f9a130640 1 -- 172.21.15.107:0/646561659 shutdown_connections 2024-05-09T08:48:46.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.336+0000 7f0f9a130640 1 -- 172.21.15.107:0/646561659 wait complete. 2024-05-09T08:48:46.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.337+0000 7f0f9a130640 1 Processor -- start 2024-05-09T08:48:46.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.337+0000 7f0f9a130640 1 -- start start 2024-05-09T08:48:46.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.337+0000 7f0f9a130640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f94069eb0 0x7f0f94197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.337+0000 7f0f9a130640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f9406a870 0x7f0f94198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.337+0000 7f0f9a130640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f94107820 0x7f0f9419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.337+0000 7f0f9a130640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0f94111ae0 con 0x7f0f9406a870 2024-05-09T08:48:46.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.337+0000 7f0f9a130640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0f94111960 con 0x7f0f94069eb0 2024-05-09T08:48:46.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.337+0000 7f0f9a130640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0f94111c60 con 0x7f0f94107820 2024-05-09T08:48:46.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f92ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f9406a870 0x7f0f94198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f92ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f9406a870 0x7f0f94198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57188/0 (socket says 172.21.15.107:57188) 2024-05-09T08:48:46.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f92ffd640 1 -- 172.21.15.107:0/2483435048 learned_addr learned my addr 172.21.15.107:0/2483435048 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:46.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f93fff640 1 --2- 172.21.15.107:0/2483435048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f94107820 0x7f0f9419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f937fe640 1 --2- 172.21.15.107:0/2483435048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f94069eb0 0x7f0f94197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f92ffd640 1 -- 172.21.15.107:0/2483435048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f94107820 msgr2=0x7f0f9419c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f92ffd640 1 --2- 172.21.15.107:0/2483435048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f94107820 0x7f0f9419c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f92ffd640 1 -- 172.21.15.107:0/2483435048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f94069eb0 msgr2=0x7f0f94197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f92ffd640 1 --2- 172.21.15.107:0/2483435048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f94069eb0 0x7f0f94197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f92ffd640 1 -- 172.21.15.107:0/2483435048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f9419cef0 con 0x7f0f9406a870 2024-05-09T08:48:46.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f93fff640 1 --2- 172.21.15.107:0/2483435048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f94107820 0x7f0f9419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:46.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f92ffd640 1 --2- 172.21.15.107:0/2483435048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f9406a870 0x7f0f94198440 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800c920 tx=0x7f0f8800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:46.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f90ff9640 1 -- 172.21.15.107:0/2483435048 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f88007c70 con 0x7f0f9406a870 2024-05-09T08:48:46.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f90ff9640 1 -- 172.21.15.107:0/2483435048 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0f88007e10 con 0x7f0f9406a870 2024-05-09T08:48:46.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f90ff9640 1 -- 172.21.15.107:0/2483435048 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f88018820 con 0x7f0f9406a870 2024-05-09T08:48:46.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f9a130640 1 -- 172.21.15.107:0/2483435048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f9419d1e0 con 0x7f0f9406a870 2024-05-09T08:48:46.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.338+0000 7f0f9a130640 1 -- 172.21.15.107:0/2483435048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f941a4a70 con 0x7f0f9406a870 2024-05-09T08:48:46.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.339+0000 7f0f9a130640 1 -- 172.21.15.107:0/2483435048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f94068aa0 con 0x7f0f9406a870 2024-05-09T08:48:46.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.340+0000 7f0f90ff9640 1 -- 172.21.15.107:0/2483435048 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0f880189c0 con 0x7f0f9406a870 2024-05-09T08:48:46.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.341+0000 7f0f90ff9640 1 --2- 172.21.15.107:0/2483435048 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f0f7c0a2d00 0x7f0f7c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.341+0000 7f0f937fe640 1 --2- 172.21.15.107:0/2483435048 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f0f7c0a2d00 0x7f0f7c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.341+0000 7f0f90ff9640 1 -- 172.21.15.107:0/2483435048 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7f0f880cfa00 con 0x7f0f9406a870 2024-05-09T08:48:46.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.341+0000 7f0f937fe640 1 --2- 172.21.15.107:0/2483435048 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f0f7c0a2d00 0x7f0f7c0a51c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0f84005fd0 tx=0x7f0f84009500 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:46.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.345+0000 7f0f90ff9640 1 -- 172.21.15.107:0/2483435048 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0f88014030 con 0x7f0f9406a870 2024-05-09T08:48:46.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.521+0000 7f0f9a130640 1 -- 172.21.15.107:0/2483435048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0f941a4e00 con 0x7f0f9406a870 2024-05-09T08:48:46.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.521+0000 7f0f90ff9640 1 -- 172.21.15.107:0/2483435048 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) ==== 100+0+11 (secure 0 0 0) 0x7f0f8800de20 con 0x7f0f9406a870 2024-05-09T08:48:46.521 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-05-09T08:48:46.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.523+0000 7f0f9a130640 1 -- 172.21.15.107:0/2483435048 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f0f7c0a2d00 msgr2=0x7f0f7c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.523+0000 7f0f9a130640 1 --2- 172.21.15.107:0/2483435048 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f0f7c0a2d00 0x7f0f7c0a51c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0f84005fd0 tx=0x7f0f84009500 comp rx=0 tx=0).stop 2024-05-09T08:48:46.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.523+0000 7f0f9a130640 1 -- 172.21.15.107:0/2483435048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f9406a870 msgr2=0x7f0f94198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.523+0000 7f0f9a130640 1 --2- 172.21.15.107:0/2483435048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f9406a870 0x7f0f94198440 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800c920 tx=0x7f0f8800cde0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.523+0000 7f0f9a130640 1 -- 172.21.15.107:0/2483435048 shutdown_connections 2024-05-09T08:48:46.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.523+0000 7f0f9a130640 1 --2- 172.21.15.107:0/2483435048 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f0f7c0a2d00 0x7f0f7c0a51c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.523+0000 7f0f9a130640 1 --2- 172.21.15.107:0/2483435048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f94107820 0x7f0f9419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.523+0000 7f0f9a130640 1 --2- 172.21.15.107:0/2483435048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f9406a870 0x7f0f94198440 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.523+0000 7f0f9a130640 1 --2- 172.21.15.107:0/2483435048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f94069eb0 0x7f0f94197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.523+0000 7f0f9a130640 1 -- 172.21.15.107:0/2483435048 >> 172.21.15.107:0/2483435048 conn(0x7f0f94077790 msgr2=0x7f0f940fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:46.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.523+0000 7f0f9a130640 1 -- 172.21.15.107:0/2483435048 shutdown_connections 2024-05-09T08:48:46.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.523+0000 7f0f9a130640 1 -- 172.21.15.107:0/2483435048 wait complete. 2024-05-09T08:48:46.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:46 smithi107 ceph-mon[30122]: pgmap v484: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-05-09T08:48:46.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:46 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/4184520522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:46.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:46 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2038366852' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:48:46.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:48:46.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:46 smithi157 ceph-mon[33614]: pgmap v484: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-05-09T08:48:46.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:46 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/4184520522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:46.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:46 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2038366852' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:48:46.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:46 smithi110 ceph-mon[33622]: pgmap v484: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-05-09T08:48:46.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:46 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/4184520522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:48:46.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:46 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2038366852' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:48:46.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.637+0000 7faeab69e640 1 -- 172.21.15.107:0/965588008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faea4107810 msgr2=0x7faea4107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.638+0000 7faeab69e640 1 --2- 172.21.15.107:0/965588008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faea4107810 0x7faea4107bf0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7faea0009930 tx=0x7faea002a420 comp rx=0 tx=0).stop 2024-05-09T08:48:46.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.638+0000 7faeab69e640 1 -- 172.21.15.107:0/965588008 shutdown_connections 2024-05-09T08:48:46.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.638+0000 7faeab69e640 1 --2- 172.21.15.107:0/965588008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faea406a7f0 0x7faea410e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.638+0000 7faeab69e640 1 --2- 172.21.15.107:0/965588008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faea4069e30 0x7faea406a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.638+0000 7faeab69e640 1 --2- 172.21.15.107:0/965588008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faea4107810 0x7faea4107bf0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.638+0000 7faeab69e640 1 -- 172.21.15.107:0/965588008 >> 172.21.15.107:0/965588008 conn(0x7faea4076530 msgr2=0x7faea4076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:46.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.638+0000 7faeab69e640 1 -- 172.21.15.107:0/965588008 shutdown_connections 2024-05-09T08:48:46.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.638+0000 7faeab69e640 1 -- 172.21.15.107:0/965588008 wait complete. 2024-05-09T08:48:46.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.638+0000 7faeab69e640 1 Processor -- start 2024-05-09T08:48:46.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.638+0000 7faeab69e640 1 -- start start 2024-05-09T08:48:46.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.638+0000 7faeab69e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faea4069e30 0x7faea4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.638+0000 7faeab69e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faea406a7f0 0x7faea41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.638+0000 7faeab69e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faea4107810 0x7faea419c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.638+0000 7faeab69e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faea4111ad0 con 0x7faea4069e30 2024-05-09T08:48:46.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.638+0000 7faeab69e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faea4111950 con 0x7faea4107810 2024-05-09T08:48:46.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.638+0000 7faeab69e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faea4111c50 con 0x7faea406a7f0 2024-05-09T08:48:46.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7faea943a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faea4069e30 0x7faea4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7faea943a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faea4069e30 0x7faea4198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57194/0 (socket says 172.21.15.107:57194) 2024-05-09T08:48:46.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7faea943a640 1 -- 172.21.15.107:0/2821848947 learned_addr learned my addr 172.21.15.107:0/2821848947 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:46.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7faea8c39640 1 --2- 172.21.15.107:0/2821848947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faea406a7f0 0x7faea41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7faea9c3b640 1 --2- 172.21.15.107:0/2821848947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faea4107810 0x7faea419c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7faea943a640 1 -- 172.21.15.107:0/2821848947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faea406a7f0 msgr2=0x7faea41985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7faea943a640 1 --2- 172.21.15.107:0/2821848947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faea406a7f0 0x7faea41985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7faea943a640 1 -- 172.21.15.107:0/2821848947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faea4107810 msgr2=0x7faea419c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7faea943a640 1 --2- 172.21.15.107:0/2821848947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faea4107810 0x7faea419c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7faea943a640 1 -- 172.21.15.107:0/2821848947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea419d080 con 0x7faea4069e30 2024-05-09T08:48:46.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7faea8c39640 1 --2- 172.21.15.107:0/2821848947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faea406a7f0 0x7faea41985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:48:46.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7faea9c3b640 1 --2- 172.21.15.107:0/2821848947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faea4107810 0x7faea419c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:48:46.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7faea943a640 1 --2- 172.21.15.107:0/2821848947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faea4069e30 0x7faea4198090 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7faea0004770 tx=0x7faea00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:46.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7fae9a7fc640 1 -- 172.21.15.107:0/2821848947 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea003f070 con 0x7faea4069e30 2024-05-09T08:48:46.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7fae9a7fc640 1 -- 172.21.15.107:0/2821848947 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faea003b460 con 0x7faea4069e30 2024-05-09T08:48:46.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7faeab69e640 1 -- 172.21.15.107:0/2821848947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faea419d310 con 0x7faea4069e30 2024-05-09T08:48:46.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7fae9a7fc640 1 -- 172.21.15.107:0/2821848947 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea0035070 con 0x7faea4069e30 2024-05-09T08:48:46.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.639+0000 7faeab69e640 1 -- 172.21.15.107:0/2821848947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faea41a4c00 con 0x7faea4069e30 2024-05-09T08:48:46.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.640+0000 7faeab69e640 1 -- 172.21.15.107:0/2821848947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faea4102f60 con 0x7faea4069e30 2024-05-09T08:48:46.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.641+0000 7fae9a7fc640 1 -- 172.21.15.107:0/2821848947 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7faea003b610 con 0x7faea4069e30 2024-05-09T08:48:46.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.642+0000 7fae9a7fc640 1 --2- 172.21.15.107:0/2821848947 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae900a2d00 0x7fae900a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.642+0000 7faea8c39640 1 --2- 172.21.15.107:0/2821848947 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae900a2d00 0x7fae900a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.643+0000 7fae9a7fc640 1 -- 172.21.15.107:0/2821848947 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7faea00efc40 con 0x7faea4069e30 2024-05-09T08:48:46.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.643+0000 7faea8c39640 1 --2- 172.21.15.107:0/2821848947 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae900a2d00 0x7fae900a51c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7faea41006d0 tx=0x7fae94005ec0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:46.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.647+0000 7fae9a7fc640 1 -- 172.21.15.107:0/2821848947 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faea00bc4b0 con 0x7faea4069e30 2024-05-09T08:48:46.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.820+0000 7faeab69e640 1 -- 172.21.15.107:0/2821848947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7faea4109d70 con 0x7faea4069e30 2024-05-09T08:48:46.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.820+0000 7fae9a7fc640 1 -- 172.21.15.107:0/2821848947 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v113) ==== 103+0+11 (secure 0 0 0) 0x7faea00bbdb0 con 0x7faea4069e30 2024-05-09T08:48:46.821 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:48:46.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.822+0000 7faeab69e640 1 -- 172.21.15.107:0/2821848947 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae900a2d00 msgr2=0x7fae900a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.822+0000 7faeab69e640 1 --2- 172.21.15.107:0/2821848947 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae900a2d00 0x7fae900a51c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7faea41006d0 tx=0x7fae94005ec0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.822+0000 7faeab69e640 1 -- 172.21.15.107:0/2821848947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faea4069e30 msgr2=0x7faea4198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.822+0000 7faeab69e640 1 --2- 172.21.15.107:0/2821848947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faea4069e30 0x7faea4198090 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7faea0004770 tx=0x7faea00042f0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.822+0000 7faeab69e640 1 -- 172.21.15.107:0/2821848947 shutdown_connections 2024-05-09T08:48:46.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.822+0000 7faeab69e640 1 --2- 172.21.15.107:0/2821848947 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fae900a2d00 0x7fae900a51c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.822+0000 7faeab69e640 1 --2- 172.21.15.107:0/2821848947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faea4107810 0x7faea419c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.822+0000 7faeab69e640 1 --2- 172.21.15.107:0/2821848947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faea406a7f0 0x7faea41985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.822+0000 7faeab69e640 1 --2- 172.21.15.107:0/2821848947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faea4069e30 0x7faea4198090 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.822+0000 7faeab69e640 1 -- 172.21.15.107:0/2821848947 >> 172.21.15.107:0/2821848947 conn(0x7faea4076530 msgr2=0x7faea40fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:46.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.822+0000 7faeab69e640 1 -- 172.21.15.107:0/2821848947 shutdown_connections 2024-05-09T08:48:46.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.822+0000 7faeab69e640 1 -- 172.21.15.107:0/2821848947 wait complete. 2024-05-09T08:48:46.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-05-09T08:48:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.935+0000 7f86de29a640 1 -- 172.21.15.107:0/689161035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d81010d0 msgr2=0x7f86d810d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.935+0000 7f86de29a640 1 --2- 172.21.15.107:0/689161035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d81010d0 0x7f86d810d5e0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f86d0009cd0 tx=0x7f86d002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.936+0000 7f86de29a640 1 -- 172.21.15.107:0/689161035 shutdown_connections 2024-05-09T08:48:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.936+0000 7f86de29a640 1 --2- 172.21.15.107:0/689161035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d81010d0 0x7f86d810d5e0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.936+0000 7f86de29a640 1 --2- 172.21.15.107:0/689161035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86d8100710 0x7f86d8100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.936+0000 7f86de29a640 1 --2- 172.21.15.107:0/689161035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86d8106720 0x7f86d8106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.936+0000 7f86de29a640 1 -- 172.21.15.107:0/689161035 >> 172.21.15.107:0/689161035 conn(0x7f86d8077830 msgr2=0x7f86d8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.936+0000 7f86de29a640 1 -- 172.21.15.107:0/689161035 shutdown_connections 2024-05-09T08:48:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.936+0000 7f86de29a640 1 -- 172.21.15.107:0/689161035 wait complete. 2024-05-09T08:48:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.936+0000 7f86de29a640 1 Processor -- start 2024-05-09T08:48:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.936+0000 7f86de29a640 1 -- start start 2024-05-09T08:48:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86de29a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d8100710 0x7f86d8198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86de29a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86d81010d0 0x7f86d8199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86de29a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86d8106720 0x7f86d81a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86de29a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f86d81109e0 con 0x7f86d8100710 2024-05-09T08:48:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86de29a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f86d8110860 con 0x7f86d81010d0 2024-05-09T08:48:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86de29a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f86d8110b60 con 0x7f86d8106720 2024-05-09T08:48:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86d7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d8100710 0x7f86d8198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86d7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d8100710 0x7f86d8198f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57200/0 (socket says 172.21.15.107:57200) 2024-05-09T08:48:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86d7fff640 1 -- 172.21.15.107:0/2399621766 learned_addr learned my addr 172.21.15.107:0/2399621766 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:48:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86dc837640 1 --2- 172.21.15.107:0/2399621766 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86d8106720 0x7f86d81a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86d77fe640 1 --2- 172.21.15.107:0/2399621766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86d81010d0 0x7f86d8199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86d7fff640 1 -- 172.21.15.107:0/2399621766 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86d8106720 msgr2=0x7f86d81a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86d7fff640 1 --2- 172.21.15.107:0/2399621766 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86d8106720 0x7f86d81a3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86d7fff640 1 -- 172.21.15.107:0/2399621766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86d81010d0 msgr2=0x7f86d8199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:46.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86d7fff640 1 --2- 172.21.15.107:0/2399621766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86d81010d0 0x7f86d8199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:46.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86d7fff640 1 -- 172.21.15.107:0/2399621766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d81a44e0 con 0x7f86d8100710 2024-05-09T08:48:46.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86d7fff640 1 --2- 172.21.15.107:0/2399621766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d8100710 0x7f86d8198f20 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f86c800d950 tx=0x7f86c800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:46.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86d57fa640 1 -- 172.21.15.107:0/2399621766 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c8017070 con 0x7f86d8100710 2024-05-09T08:48:46.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86d57fa640 1 -- 172.21.15.107:0/2399621766 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86c8009c30 con 0x7f86d8100710 2024-05-09T08:48:46.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86de29a640 1 -- 172.21.15.107:0/2399621766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86d81a46b0 con 0x7f86d8100710 2024-05-09T08:48:46.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86d57fa640 1 -- 172.21.15.107:0/2399621766 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c800f620 con 0x7f86d8100710 2024-05-09T08:48:46.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.937+0000 7f86de29a640 1 -- 172.21.15.107:0/2399621766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86d81a4bd0 con 0x7f86d8100710 2024-05-09T08:48:46.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.938+0000 7f86de29a640 1 -- 172.21.15.107:0/2399621766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86d8068650 con 0x7f86d8100710 2024-05-09T08:48:46.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.939+0000 7f86d57fa640 1 -- 172.21.15.107:0/2399621766 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f86c80109a0 con 0x7f86d8100710 2024-05-09T08:48:46.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.940+0000 7f86d57fa640 1 --2- 172.21.15.107:0/2399621766 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f86c00a2d00 0x7f86c00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:48:46.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.940+0000 7f86d77fe640 1 --2- 172.21.15.107:0/2399621766 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f86c00a2d00 0x7f86c00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:48:46.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.940+0000 7f86d57fa640 1 -- 172.21.15.107:0/2399621766 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9467+0+0 (secure 0 0 0) 0x7f86c80cf480 con 0x7f86d8100710 2024-05-09T08:48:46.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.940+0000 7f86d77fe640 1 --2- 172.21.15.107:0/2399621766 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f86c00a2d00 0x7f86c00a51c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f86d8069510 tx=0x7f86cc00b590 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:48:46.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:46.944+0000 7f86d57fa640 1 -- 172.21.15.107:0/2399621766 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f86c8009de0 con 0x7f86d8100710 2024-05-09T08:48:47.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:47.171+0000 7f86de29a640 1 -- 172.21.15.107:0/2399621766 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f86d8108c80 con 0x7f86d8100710 2024-05-09T08:48:47.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:47.197+0000 7f86d57fa640 1 -- 172.21.15.107:0/2399621766 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+740909 (secure 0 0 0) 0x7f86c809bcf0 con 0x7f86d8100710 2024-05-09T08:48:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:48:47.208 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:48:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "epoch": 26, 2024-05-09T08:48:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "active_gid": 15483, 2024-05-09T08:48:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "active_name": "x", 2024-05-09T08:48:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "active_addrs": { 2024-05-09T08:48:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-05-09T08:48:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2024-05-09T08:48:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6800", 2024-05-09T08:48:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 165281804 2024-05-09T08:48:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-05-09T08:48:47.209 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:6801", 2024-05-09T08:48:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 165281804 2024-05-09T08:48:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-05-09T08:48:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "active_addr": "172.21.15.107:6801/165281804", 2024-05-09T08:48:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "active_change": "2024-05-09T08:33:20.828812+0000", 2024-05-09T08:48:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "active_mgr_features": 4540701547738038271, 2024-05-09T08:48:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "available": true, 2024-05-09T08:48:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: "standbys": [ 2024-05-09T08:48:47.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "gid": 24965, 2024-05-09T08:48:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "y", 2024-05-09T08:48:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_features": 4540701547738038271, 2024-05-09T08:48:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "available_modules": [ 2024-05-09T08:48:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "alerts", 2024-05-09T08:48:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-05-09T08:48:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-05-09T08:48:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.211 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-05-09T08:48:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How frequently to reexamine health status", 2024-05-09T08:48:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.212 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.213 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.214 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.215 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.216 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.217 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_destination": { 2024-05-09T08:48:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_destination", 2024-05-09T08:48:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Email address to send alerts to", 2024-05-09T08:48:47.218 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_from_name": { 2024-05-09T08:48:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_from_name", 2024-05-09T08:48:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "Ceph", 2024-05-09T08:48:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Email From: name", 2024-05-09T08:48:47.219 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_host": { 2024-05-09T08:48:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_host", 2024-05-09T08:48:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP server", 2024-05-09T08:48:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.220 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_password": { 2024-05-09T08:48:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_password", 2024-05-09T08:48:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Password to authenticate with", 2024-05-09T08:48:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.221 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.222 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.222 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_port": { 2024-05-09T08:48:47.222 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_port", 2024-05-09T08:48:47.222 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.222 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.222 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.222 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "465", 2024-05-09T08:48:47.222 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.222 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP port", 2024-05-09T08:48:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_sender": { 2024-05-09T08:48:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_sender", 2024-05-09T08:48:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.223 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP envelope sender", 2024-05-09T08:48:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_ssl": { 2024-05-09T08:48:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_ssl", 2024-05-09T08:48:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.224 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.225 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.225 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.225 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.225 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-09T08:48:47.225 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.225 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.225 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.225 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_user": { 2024-05-09T08:48:47.225 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_user", 2024-05-09T08:48:47.225 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.225 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.225 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.226 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.226 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.226 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.226 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.226 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "User to authenticate as", 2024-05-09T08:48:47.226 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.226 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.226 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.226 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.226 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.226 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.226 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "balancer", 2024-05-09T08:48:47.226 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.227 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.227 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.227 INFO:teuthology.orchestra.run.smithi107.stdout: "active": { 2024-05-09T08:48:47.227 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "active", 2024-05-09T08:48:47.227 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.227 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.227 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.227 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.227 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.227 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.227 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.227 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "automatically balance PGs across cluster", 2024-05-09T08:48:47.227 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.227 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: "begin_time": { 2024-05-09T08:48:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "begin_time", 2024-05-09T08:48:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0000", 2024-05-09T08:48:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "beginning time of day to automatically balance", 2024-05-09T08:48:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T08:48:47.228 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.229 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.229 INFO:teuthology.orchestra.run.smithi107.stdout: "begin_weekday": { 2024-05-09T08:48:47.229 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "begin_weekday", 2024-05-09T08:48:47.229 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-05-09T08:48:47.229 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.229 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.229 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-05-09T08:48:47.229 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0", 2024-05-09T08:48:47.229 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "6", 2024-05-09T08:48:47.229 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.229 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-09T08:48:47.229 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T08:48:47.229 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_max_iterations": { 2024-05-09T08:48:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_max_iterations", 2024-05-09T08:48:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-05-09T08:48:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "25", 2024-05-09T08:48:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-05-09T08:48:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "250", 2024-05-09T08:48:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-09T08:48:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.230 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_metrics": { 2024-05-09T08:48:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_metrics", 2024-05-09T08:48:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "pgs,objects,bytes", 2024-05-09T08:48:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-09T08:48:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-09T08:48:47.231 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_step": { 2024-05-09T08:48:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_step", 2024-05-09T08:48:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0.5", 2024-05-09T08:48:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0.001", 2024-05-09T08:48:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "0.999", 2024-05-09T08:48:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "aggressiveness of optimization", 2024-05-09T08:48:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-09T08:48:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.232 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: "end_time": { 2024-05-09T08:48:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "end_time", 2024-05-09T08:48:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "2359", 2024-05-09T08:48:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "ending time of day to automatically balance", 2024-05-09T08:48:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T08:48:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: "end_weekday": { 2024-05-09T08:48:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "end_weekday", 2024-05-09T08:48:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-05-09T08:48:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-05-09T08:48:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0", 2024-05-09T08:48:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "6", 2024-05-09T08:48:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-09T08:48:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T08:48:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.236 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.236 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.236 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.236 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.236 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.236 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.236 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.236 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.236 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.236 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.236 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.236 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.237 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.238 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: "min_score": { 2024-05-09T08:48:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "min_score", 2024-05-09T08:48:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-05-09T08:48:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-09T08:48:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.240 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": { 2024-05-09T08:48:47.241 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mode", 2024-05-09T08:48:47.241 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.241 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.241 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.241 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "upmap", 2024-05-09T08:48:47.241 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.241 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.241 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.241 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-compat", 2024-05-09T08:48:47.241 INFO:teuthology.orchestra.run.smithi107.stdout: "none", 2024-05-09T08:48:47.241 INFO:teuthology.orchestra.run.smithi107.stdout: "read", 2024-05-09T08:48:47.241 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap", 2024-05-09T08:48:47.241 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap-read" 2024-05-09T08:48:47.241 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Balancer mode", 2024-05-09T08:48:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_ids": { 2024-05-09T08:48:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pool_ids", 2024-05-09T08:48:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.242 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-09T08:48:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-05-09T08:48:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-05-09T08:48:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-05-09T08:48:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.243 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-09T08:48:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap_max_deviation": { 2024-05-09T08:48:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "upmap_max_deviation", 2024-05-09T08:48:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-05-09T08:48:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-05-09T08:48:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-09T08:48:47.244 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-09T08:48:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap_max_optimizations": { 2024-05-09T08:48:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "upmap_max_optimizations", 2024-05-09T08:48:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-05-09T08:48:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-09T08:48:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephadm", 2024-05-09T08:48:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_down_multiplier": { 2024-05-09T08:48:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_down_multiplier", 2024-05-09T08:48:47.246 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3.0", 2024-05-09T08:48:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-09T08:48:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_refresh_rate": { 2024-05-09T08:48:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_refresh_rate", 2024-05-09T08:48:47.247 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "20", 2024-05-09T08:48:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-09T08:48:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_starting_port": { 2024-05-09T08:48:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_starting_port", 2024-05-09T08:48:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.248 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "4721", 2024-05-09T08:48:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-09T08:48:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_ptrace": { 2024-05-09T08:48:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_ptrace", 2024-05-09T08:48:47.249 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-09T08:48:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-09T08:48:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: "autotune_interval": { 2024-05-09T08:48:47.250 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "autotune_interval", 2024-05-09T08:48:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-05-09T08:48:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-09T08:48:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "autotune_memory_target_ratio": { 2024-05-09T08:48:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "autotune_memory_target_ratio", 2024-05-09T08:48:47.251 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0.7", 2024-05-09T08:48:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-09T08:48:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm_log_destination": { 2024-05-09T08:48:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephadm_log_destination", 2024-05-09T08:48:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.252 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: "file", 2024-05-09T08:48:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: "file,syslog", 2024-05-09T08:48:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: "syslog" 2024-05-09T08:48:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-09T08:48:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.253 INFO:teuthology.orchestra.run.smithi107.stdout: "cgroups_split": { 2024-05-09T08:48:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cgroups_split", 2024-05-09T08:48:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-09T08:48:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.254 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "config_checks_enabled": { 2024-05-09T08:48:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_checks_enabled", 2024-05-09T08:48:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-09T08:48:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.255 INFO:teuthology.orchestra.run.smithi107.stdout: "config_dashboard": { 2024-05-09T08:48:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_dashboard", 2024-05-09T08:48:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-09T08:48:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.256 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_alertmanager": { 2024-05-09T08:48:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_alertmanager", 2024-05-09T08:48:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-09T08:48:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-05-09T08:48:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_base": { 2024-05-09T08:48:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_base", 2024-05-09T08:48:47.257 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-09T08:48:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Container image name, without the tag", 2024-05-09T08:48:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_elasticsearch": { 2024-05-09T08:48:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_elasticsearch", 2024-05-09T08:48:47.258 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-09T08:48:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "elasticsearch container image", 2024-05-09T08:48:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_grafana": { 2024-05-09T08:48:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_grafana", 2024-05-09T08:48:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.259 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-09T08:48:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-05-09T08:48:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_haproxy": { 2024-05-09T08:48:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_haproxy", 2024-05-09T08:48:47.260 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-09T08:48:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "HAproxy container image", 2024-05-09T08:48:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_agent": { 2024-05-09T08:48:47.261 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_agent", 2024-05-09T08:48:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-09T08:48:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger agent container image", 2024-05-09T08:48:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.262 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_collector": { 2024-05-09T08:48:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_collector", 2024-05-09T08:48:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-09T08:48:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger collector container image", 2024-05-09T08:48:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_query": { 2024-05-09T08:48:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_query", 2024-05-09T08:48:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-09T08:48:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger query container image", 2024-05-09T08:48:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.264 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_keepalived": { 2024-05-09T08:48:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_keepalived", 2024-05-09T08:48:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-09T08:48:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Keepalived container image", 2024-05-09T08:48:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_loki": { 2024-05-09T08:48:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_loki", 2024-05-09T08:48:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-09T08:48:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Loki container image", 2024-05-09T08:48:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.266 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_node_exporter": { 2024-05-09T08:48:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_node_exporter", 2024-05-09T08:48:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-09T08:48:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-05-09T08:48:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_nvmeof": { 2024-05-09T08:48:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_nvmeof", 2024-05-09T08:48:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-09T08:48:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Nvme-of container image", 2024-05-09T08:48:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.268 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_prometheus": { 2024-05-09T08:48:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_prometheus", 2024-05-09T08:48:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-09T08:48:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-05-09T08:48:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_promtail": { 2024-05-09T08:48:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_promtail", 2024-05-09T08:48:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-09T08:48:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Promtail container image", 2024-05-09T08:48:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_samba": { 2024-05-09T08:48:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_samba", 2024-05-09T08:48:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-09T08:48:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Samba/SMB container image", 2024-05-09T08:48:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.271 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_snmp_gateway": { 2024-05-09T08:48:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_snmp_gateway", 2024-05-09T08:48:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-09T08:48:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SNMP Gateway container image", 2024-05-09T08:48:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: "container_init": { 2024-05-09T08:48:47.272 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_init", 2024-05-09T08:48:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Run podman/docker with `--init`", 2024-05-09T08:48:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: "daemon_cache_timeout": { 2024-05-09T08:48:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "daemon_cache_timeout", 2024-05-09T08:48:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-05-09T08:48:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-09T08:48:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.274 INFO:teuthology.orchestra.run.smithi107.stdout: "default_cephadm_command_timeout": { 2024-05-09T08:48:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "default_cephadm_command_timeout", 2024-05-09T08:48:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "900", 2024-05-09T08:48:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-09T08:48:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.275 INFO:teuthology.orchestra.run.smithi107.stdout: "default_registry": { 2024-05-09T08:48:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "default_registry", 2024-05-09T08:48:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io", 2024-05-09T08:48:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-09T08:48:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "device_cache_timeout": { 2024-05-09T08:48:47.276 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_cache_timeout", 2024-05-09T08:48:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1800", 2024-05-09T08:48:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache device inventory", 2024-05-09T08:48:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: "device_enhanced_scan": { 2024-05-09T08:48:47.277 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_enhanced_scan", 2024-05-09T08:48:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-09T08:48:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.278 INFO:teuthology.orchestra.run.smithi107.stdout: "facts_cache_timeout": { 2024-05-09T08:48:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "facts_cache_timeout", 2024-05-09T08:48:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-05-09T08:48:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache host facts data", 2024-05-09T08:48:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: "grafana_dashboards_path": { 2024-05-09T08:48:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "grafana_dashboards_path", 2024-05-09T08:48:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-09T08:48:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-09T08:48:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "host_check_interval": { 2024-05-09T08:48:47.280 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "host_check_interval", 2024-05-09T08:48:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-05-09T08:48:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to perform a host check", 2024-05-09T08:48:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.281 INFO:teuthology.orchestra.run.smithi107.stdout: "hw_monitoring": { 2024-05-09T08:48:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "hw_monitoring", 2024-05-09T08:48:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-09T08:48:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.282 INFO:teuthology.orchestra.run.smithi107.stdout: "inventory_list_all": { 2024-05-09T08:48:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "inventory_list_all", 2024-05-09T08:48:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-09T08:48:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.283 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.284 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.284 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.284 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.284 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.284 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.284 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.284 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.284 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.284 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.284 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.284 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.284 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.284 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: "log_refresh_metadata": { 2024-05-09T08:48:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_refresh_metadata", 2024-05-09T08:48:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-09T08:48:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.286 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-09T08:48:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.287 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.288 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-09T08:48:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-09T08:48:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-09T08:48:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.290 INFO:teuthology.orchestra.run.smithi107.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-09T08:48:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-09T08:48:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "*", 2024-05-09T08:48:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-09T08:48:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.291 INFO:teuthology.orchestra.run.smithi107.stdout: "max_count_per_host": { 2024-05-09T08:48:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_count_per_host", 2024-05-09T08:48:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "max number of daemons per service per host", 2024-05-09T08:48:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: "max_osd_draining_count": { 2024-05-09T08:48:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_osd_draining_count", 2024-05-09T08:48:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-09T08:48:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "migration_current": { 2024-05-09T08:48:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "migration_current", 2024-05-09T08:48:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "internal - do not modify", 2024-05-09T08:48:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": { 2024-05-09T08:48:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mode", 2024-05-09T08:48:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "root", 2024-05-09T08:48:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm-package", 2024-05-09T08:48:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "root" 2024-05-09T08:48:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "mode for remote execution of cephadm", 2024-05-09T08:48:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.295 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "oob_default_addr": { 2024-05-09T08:48:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "oob_default_addr", 2024-05-09T08:48:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "169.254.1.1", 2024-05-09T08:48:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-09T08:48:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.296 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: "prometheus_alerts_path": { 2024-05-09T08:48:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prometheus_alerts_path", 2024-05-09T08:48:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-09T08:48:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-09T08:48:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.297 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_insecure": { 2024-05-09T08:48:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_insecure", 2024-05-09T08:48:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.298 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-09T08:48:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_password": { 2024-05-09T08:48:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_password", 2024-05-09T08:48:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.299 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-09T08:48:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_url": { 2024-05-09T08:48:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_url", 2024-05-09T08:48:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.300 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-09T08:48:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_username": { 2024-05-09T08:48:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_username", 2024-05-09T08:48:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.301 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.302 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.302 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-09T08:48:47.302 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.302 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.302 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.302 INFO:teuthology.orchestra.run.smithi107.stdout: "secure_monitoring_stack": { 2024-05-09T08:48:47.302 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "secure_monitoring_stack", 2024-05-09T08:48:47.302 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.302 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.302 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.302 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.302 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.302 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-09T08:48:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: "service_discovery_port": { 2024-05-09T08:48:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "service_discovery_port", 2024-05-09T08:48:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8765", 2024-05-09T08:48:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.303 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "cephadm service discovery port", 2024-05-09T08:48:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "ssh_config_file": { 2024-05-09T08:48:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssh_config_file", 2024-05-09T08:48:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.304 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-09T08:48:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "use_agent": { 2024-05-09T08:48:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "use_agent", 2024-05-09T08:48:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.305 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-09T08:48:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "use_repo_digest": { 2024-05-09T08:48:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "use_repo_digest", 2024-05-09T08:48:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.306 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-09T08:48:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_failed_host_check": { 2024-05-09T08:48:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_failed_host_check", 2024-05-09T08:48:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.307 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if the host check fails", 2024-05-09T08:48:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_stray_daemons": { 2024-05-09T08:48:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_stray_daemons", 2024-05-09T08:48:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.308 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-09T08:48:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_stray_hosts": { 2024-05-09T08:48:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_stray_hosts", 2024-05-09T08:48:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.309 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-09T08:48:47.310 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.310 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.310 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.310 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.310 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.310 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crash", 2024-05-09T08:48:47.310 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.310 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.310 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.310 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.310 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.311 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.311 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.311 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.311 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.311 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.311 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.311 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.311 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.311 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.311 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.311 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.311 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.311 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.311 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.312 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.312 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.312 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.312 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.312 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.312 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.312 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.312 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.312 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.312 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.312 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.312 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.312 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.313 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.313 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.313 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.313 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.313 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.313 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.313 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.313 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.313 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.313 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.313 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.313 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.313 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.313 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.314 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.314 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.314 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.314 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.314 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.314 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.314 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.314 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.314 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.314 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.314 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.314 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.315 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.315 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.315 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.315 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.315 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.315 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.315 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.315 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.315 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.315 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.315 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.315 INFO:teuthology.orchestra.run.smithi107.stdout: "retain_interval": { 2024-05-09T08:48:47.315 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "retain_interval", 2024-05-09T08:48:47.315 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.316 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.316 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.316 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "31536000", 2024-05-09T08:48:47.316 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.316 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.316 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.316 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-09T08:48:47.316 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.316 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.316 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.316 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_recent_interval": { 2024-05-09T08:48:47.316 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_recent_interval", 2024-05-09T08:48:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1209600", 2024-05-09T08:48:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-09T08:48:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "dashboard", 2024-05-09T08:48:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-09T08:48:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-09T08:48:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.318 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "ALERTMANAGER_API_HOST": { 2024-05-09T08:48:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-09T08:48:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.319 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-09T08:48:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-09T08:48:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.320 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: "AUDIT_API_ENABLED": { 2024-05-09T08:48:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "AUDIT_API_ENABLED", 2024-05-09T08:48:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.321 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-09T08:48:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-09T08:48:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.322 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: "ENABLE_BROWSABLE_API": { 2024-05-09T08:48:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-09T08:48:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.323 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-09T08:48:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-09T08:48:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.324 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-09T08:48:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-09T08:48:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.325 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.326 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.326 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.326 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.326 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-09T08:48:47.326 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-09T08:48:47.326 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.326 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.326 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.326 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.326 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.326 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.326 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.327 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.327 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.327 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.327 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.327 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-09T08:48:47.327 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-09T08:48:47.327 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.327 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.327 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.327 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.327 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.327 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.327 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-09T08:48:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-09T08:48:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.328 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.329 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.329 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.329 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.329 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-09T08:48:47.329 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-09T08:48:47.329 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.329 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.329 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.329 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.329 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.329 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.329 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.329 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.330 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.330 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.330 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.330 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-09T08:48:47.330 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-09T08:48:47.330 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.330 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.330 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.330 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.330 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.330 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.330 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-09T08:48:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-09T08:48:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.331 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_PASSWORD": { 2024-05-09T08:48:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-09T08:48:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-05-09T08:48:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.332 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-09T08:48:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-09T08:48:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.333 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_URL": { 2024-05-09T08:48:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_URL", 2024-05-09T08:48:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.334 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_USERNAME": { 2024-05-09T08:48:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-09T08:48:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-05-09T08:48:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.335 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-09T08:48:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-09T08:48:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.336 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.337 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-09T08:48:47.337 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-09T08:48:47.337 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.337 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.337 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.337 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.337 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.337 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.337 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.337 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.337 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-09T08:48:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-09T08:48:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.338 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-09T08:48:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-09T08:48:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.339 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: "PROMETHEUS_API_HOST": { 2024-05-09T08:48:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-09T08:48:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.340 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-09T08:48:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-09T08:48:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.341 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-09T08:48:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-09T08:48:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-09T08:48:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-09T08:48:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.343 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-09T08:48:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-09T08:48:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.344 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-09T08:48:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-09T08:48:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.345 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-09T08:48:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-09T08:48:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-09T08:48:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-09T08:48:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.347 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-09T08:48:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-09T08:48:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.348 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_ENABLED": { 2024-05-09T08:48:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-09T08:48:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.349 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-09T08:48:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-09T08:48:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-09T08:48:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.350 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-09T08:48:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-09T08:48:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.351 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-09T08:48:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-09T08:48:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8", 2024-05-09T08:48:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.352 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-09T08:48:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-09T08:48:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "45", 2024-05-09T08:48:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.353 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_ACCESS_KEY": { 2024-05-09T08:48:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-09T08:48:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.354 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-09T08:48:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-09T08:48:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-05-09T08:48:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.355 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_SECRET_KEY": { 2024-05-09T08:48:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-09T08:48:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.356 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_SSL_VERIFY": { 2024-05-09T08:48:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-09T08:48:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.357 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-09T08:48:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-09T08:48:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-05-09T08:48:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-09T08:48:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-09T08:48:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.359 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-09T08:48:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-09T08:48:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-05-09T08:48:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.360 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "cross_origin_url": { 2024-05-09T08:48:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cross_origin_url", 2024-05-09T08:48:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.361 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: "crt_file": { 2024-05-09T08:48:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crt_file", 2024-05-09T08:48:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.362 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: "debug": { 2024-05-09T08:48:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "debug", 2024-05-09T08:48:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable/disable debug options", 2024-05-09T08:48:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.363 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "jwt_token_ttl": { 2024-05-09T08:48:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "jwt_token_ttl", 2024-05-09T08:48:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "28800", 2024-05-09T08:48:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.364 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: "key_file": { 2024-05-09T08:48:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "key_file", 2024-05-09T08:48:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.365 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.366 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.367 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.368 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.369 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.370 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: "motd": { 2024-05-09T08:48:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "motd", 2024-05-09T08:48:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "The message of the day", 2024-05-09T08:48:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.371 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: "redirect_resolve_ip_addr": { 2024-05-09T08:48:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "redirect_resolve_ip_addr", 2024-05-09T08:48:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.372 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-05-09T08:48:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-05-09T08:48:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "::", 2024-05-09T08:48:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-05-09T08:48:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-05-09T08:48:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8080", 2024-05-09T08:48:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl": { 2024-05-09T08:48:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl", 2024-05-09T08:48:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl_server_port": { 2024-05-09T08:48:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl_server_port", 2024-05-09T08:48:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8443", 2024-05-09T08:48:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_behaviour": { 2024-05-09T08:48:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_behaviour", 2024-05-09T08:48:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "redirect", 2024-05-09T08:48:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: "redirect" 2024-05-09T08:48:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_error_status_code": { 2024-05-09T08:48:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_error_status_code", 2024-05-09T08:48:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "500", 2024-05-09T08:48:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "400", 2024-05-09T08:48:47.378 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "599", 2024-05-09T08:48:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "url_prefix": { 2024-05-09T08:48:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "url_prefix", 2024-05-09T08:48:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "devicehealth", 2024-05-09T08:48:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.380 INFO:teuthology.orchestra.run.smithi107.stdout: "enable_monitoring": { 2024-05-09T08:48:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enable_monitoring", 2024-05-09T08:48:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "monitor device health metrics", 2024-05-09T08:48:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.381 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.382 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.383 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.384 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.385 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "mark_out_threshold": { 2024-05-09T08:48:47.386 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mark_out_threshold", 2024-05-09T08:48:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "2419200", 2024-05-09T08:48:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-09T08:48:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.387 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": { 2024-05-09T08:48:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pool_name", 2024-05-09T08:48:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "device_health_metrics", 2024-05-09T08:48:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-09T08:48:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.388 INFO:teuthology.orchestra.run.smithi107.stdout: "retention_period": { 2024-05-09T08:48:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "retention_period", 2024-05-09T08:48:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15552000", 2024-05-09T08:48:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long to retain device health metrics", 2024-05-09T08:48:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "scrape_frequency": { 2024-05-09T08:48:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "scrape_frequency", 2024-05-09T08:48:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "86400", 2024-05-09T08:48:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-09T08:48:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.390 INFO:teuthology.orchestra.run.smithi107.stdout: "self_heal": { 2024-05-09T08:48:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "self_heal", 2024-05-09T08:48:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-09T08:48:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.392 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-05-09T08:48:47.392 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-05-09T08:48:47.392 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.392 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.392 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.392 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-05-09T08:48:47.392 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.392 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.392 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.392 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to wake up and check device health", 2024-05-09T08:48:47.392 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.392 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.392 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.392 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_threshold": { 2024-05-09T08:48:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_threshold", 2024-05-09T08:48:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "7257600", 2024-05-09T08:48:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-09T08:48:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "diskprediction_local", 2024-05-09T08:48:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.394 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.395 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.396 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.398 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "predict_interval": { 2024-05-09T08:48:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "predict_interval", 2024-05-09T08:48:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "86400", 2024-05-09T08:48:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.399 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "predictor_model": { 2024-05-09T08:48:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "predictor_model", 2024-05-09T08:48:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "prophetstor", 2024-05-09T08:48:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.400 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-05-09T08:48:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-05-09T08:48:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-05-09T08:48:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.401 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "influx", 2024-05-09T08:48:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": false, 2024-05-09T08:48:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "influxdb python module not found", 2024-05-09T08:48:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: "batch_size": { 2024-05-09T08:48:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "batch_size", 2024-05-09T08:48:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.402 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5000", 2024-05-09T08:48:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-09T08:48:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: "database": { 2024-05-09T08:48:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "database", 2024-05-09T08:48:47.403 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.404 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.404 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.404 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "ceph", 2024-05-09T08:48:47.404 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.404 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.404 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.404 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-09T08:48:47.404 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.404 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.404 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.404 INFO:teuthology.orchestra.run.smithi107.stdout: "hostname": { 2024-05-09T08:48:47.404 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "hostname", 2024-05-09T08:48:47.404 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.404 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB server hostname", 2024-05-09T08:48:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-05-09T08:48:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-05-09T08:48:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.405 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "30", 2024-05-09T08:48:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "5", 2024-05-09T08:48:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-09T08:48:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.406 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.408 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.409 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.410 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.411 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.411 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.411 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.411 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.413 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.413 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.413 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.413 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: "password": { 2024-05-09T08:48:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "password", 2024-05-09T08:48:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "password of InfluxDB server user", 2024-05-09T08:48:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.414 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "port": { 2024-05-09T08:48:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "port", 2024-05-09T08:48:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8086", 2024-05-09T08:48:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB server port", 2024-05-09T08:48:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.415 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl": { 2024-05-09T08:48:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl", 2024-05-09T08:48:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "false", 2024-05-09T08:48:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T08:48:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.416 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: "threads": { 2024-05-09T08:48:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "threads", 2024-05-09T08:48:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-05-09T08:48:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-05-09T08:48:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "32", 2024-05-09T08:48:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-09T08:48:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "username": { 2024-05-09T08:48:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "username", 2024-05-09T08:48:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "username of InfluxDB server user", 2024-05-09T08:48:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "verify_ssl": { 2024-05-09T08:48:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "verify_ssl", 2024-05-09T08:48:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "true", 2024-05-09T08:48:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T08:48:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.419 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "insights", 2024-05-09T08:48:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.420 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.421 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.422 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.423 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.424 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "iostat", 2024-05-09T08:48:47.425 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.426 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.428 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.429 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.430 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "k8sevents", 2024-05-09T08:48:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_event_retention_days": { 2024-05-09T08:48:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ceph_event_retention_days", 2024-05-09T08:48:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.431 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "7", 2024-05-09T08:48:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-09T08:48:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: "config_check_secs": { 2024-05-09T08:48:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_check_secs", 2024-05-09T08:48:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.432 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "10", 2024-05-09T08:48:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-09T08:48:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.433 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.434 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.435 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.436 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.437 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "localpool", 2024-05-09T08:48:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: "failure_domain": { 2024-05-09T08:48:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "failure_domain", 2024-05-09T08:48:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "host", 2024-05-09T08:48:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.439 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "failure domain for any created local pool", 2024-05-09T08:48:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-09T08:48:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.440 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.441 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.442 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.443 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.444 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": { 2024-05-09T08:48:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "min_size", 2024-05-09T08:48:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default min_size for any created local pool", 2024-05-09T08:48:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-09T08:48:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: "num_rep": { 2024-05-09T08:48:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "num_rep", 2024-05-09T08:48:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3", 2024-05-09T08:48:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.446 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default replica count for any created local pool", 2024-05-09T08:48:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": { 2024-05-09T08:48:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pg_num", 2024-05-09T08:48:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "128", 2024-05-09T08:48:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.447 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default pg_num for any created local pool", 2024-05-09T08:48:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "prefix": { 2024-05-09T08:48:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prefix", 2024-05-09T08:48:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "name prefix for any created local pool", 2024-05-09T08:48:47.448 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "subtree": { 2024-05-09T08:48:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "subtree", 2024-05-09T08:48:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "rack", 2024-05-09T08:48:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-09T08:48:47.449 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-09T08:48:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mds_autoscaler", 2024-05-09T08:48:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.450 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.451 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.452 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.453 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.454 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.455 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mirroring", 2024-05-09T08:48:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.456 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.457 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.458 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.459 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.460 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "nfs", 2024-05-09T08:48:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.461 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.462 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.463 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.464 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.465 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.466 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "orchestrator", 2024-05-09T08:48:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "fail_fs": { 2024-05-09T08:48:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "fail_fs", 2024-05-09T08:48:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-09T08:48:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.467 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.468 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.469 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.470 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.471 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: "orchestrator": { 2024-05-09T08:48:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "orchestrator", 2024-05-09T08:48:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm", 2024-05-09T08:48:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: "rook", 2024-05-09T08:48:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: "test_orchestrator" 2024-05-09T08:48:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.473 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Orchestrator backend", 2024-05-09T08:48:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "osd_perf_query", 2024-05-09T08:48:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.474 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.475 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.476 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.477 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.478 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.478 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.478 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.478 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.478 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.478 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.478 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.478 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.478 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.478 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.478 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.478 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.479 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "osd_support", 2024-05-09T08:48:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.480 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.481 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.482 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.483 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.484 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pg_autoscaler", 2024-05-09T08:48:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.485 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.486 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.487 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.488 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.489 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-05-09T08:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-05-09T08:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-05-09T08:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: "threshold": { 2024-05-09T08:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "threshold", 2024-05-09T08:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3.0", 2024-05-09T08:48:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1.0", 2024-05-09T08:48:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "scaling threshold", 2024-05-09T08:48:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-09T08:48:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.492 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "progress", 2024-05-09T08:48:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_pg_recovery_event": { 2024-05-09T08:48:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_pg_recovery_event", 2024-05-09T08:48:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-09T08:48:47.493 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: "enabled": { 2024-05-09T08:48:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enabled", 2024-05-09T08:48:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.494 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.495 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.496 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.497 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.498 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.499 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "max_completed_events": { 2024-05-09T08:48:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_completed_events", 2024-05-09T08:48:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "50", 2024-05-09T08:48:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "number of past completed events to remember", 2024-05-09T08:48:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-05-09T08:48:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-05-09T08:48:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-05-09T08:48:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long the module is going to sleep", 2024-05-09T08:48:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.501 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.502 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.502 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.502 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prometheus", 2024-05-09T08:48:47.502 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.502 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.502 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.502 INFO:teuthology.orchestra.run.smithi107.stdout: "cache": { 2024-05-09T08:48:47.502 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cache", 2024-05-09T08:48:47.502 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.502 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.502 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.502 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.503 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.503 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.503 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.503 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.503 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.503 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.503 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.503 INFO:teuthology.orchestra.run.smithi107.stdout: "exclude_perf_counters": { 2024-05-09T08:48:47.503 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "exclude_perf_counters", 2024-05-09T08:48:47.503 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.503 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.503 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.503 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-09T08:48:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-09T08:48:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.504 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.505 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.506 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.507 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.507 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.507 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.507 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.507 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.507 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.507 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.507 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.507 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.507 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.507 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.507 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.507 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.507 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.508 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.508 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.508 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.508 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.508 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.508 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.508 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.508 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.508 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.508 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.508 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.508 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.508 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.508 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_stats_pools": { 2024-05-09T08:48:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_stats_pools", 2024-05-09T08:48:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-09T08:48:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-09T08:48:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "300", 2024-05-09T08:48:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: "scrape_interval": { 2024-05-09T08:48:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "scrape_interval", 2024-05-09T08:48:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15.0", 2024-05-09T08:48:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.512 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.512 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.512 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.512 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-05-09T08:48:47.512 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-05-09T08:48:47.512 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.512 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.512 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.512 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "::", 2024-05-09T08:48:47.512 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.512 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.512 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.512 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-09T08:48:47.512 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.513 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.513 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.513 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-05-09T08:48:47.513 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-05-09T08:48:47.513 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.513 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.513 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.513 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "9283", 2024-05-09T08:48:47.513 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.513 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.513 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.513 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-09T08:48:47.513 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.513 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.514 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.514 INFO:teuthology.orchestra.run.smithi107.stdout: "stale_cache_strategy": { 2024-05-09T08:48:47.514 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "stale_cache_strategy", 2024-05-09T08:48:47.514 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.514 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.514 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.514 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "log", 2024-05-09T08:48:47.514 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.514 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.514 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.514 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.514 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.514 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.515 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.515 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_behaviour": { 2024-05-09T08:48:47.515 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_behaviour", 2024-05-09T08:48:47.515 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.515 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.515 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.515 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "default", 2024-05-09T08:48:47.515 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.515 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.515 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.515 INFO:teuthology.orchestra.run.smithi107.stdout: "default", 2024-05-09T08:48:47.515 INFO:teuthology.orchestra.run.smithi107.stdout: "error" 2024-05-09T08:48:47.515 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.515 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.516 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.516 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.516 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.516 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_error_status_code": { 2024-05-09T08:48:47.516 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_error_status_code", 2024-05-09T08:48:47.516 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.516 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.516 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.516 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "500", 2024-05-09T08:48:47.516 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "400", 2024-05-09T08:48:47.516 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "599", 2024-05-09T08:48:47.516 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.516 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.517 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.517 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.517 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.517 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.517 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.517 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_support", 2024-05-09T08:48:47.517 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.517 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.517 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.517 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.517 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.517 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.517 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.518 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.518 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.518 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.518 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.518 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.518 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.518 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.518 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.518 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.518 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.518 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.518 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.518 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.518 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.519 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.519 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.519 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.519 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.519 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.519 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.519 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.519 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.519 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.519 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.519 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.519 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.519 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.520 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.520 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.520 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.520 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.520 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.520 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.520 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.520 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.520 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.520 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.520 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.520 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.520 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.521 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.521 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.521 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.521 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.521 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.521 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.521 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.521 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.521 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.521 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.521 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.521 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.521 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.522 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.522 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.522 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.522 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.522 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.522 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.522 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.522 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.522 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.522 INFO:teuthology.orchestra.run.smithi107.stdout: "max_concurrent_snap_create": { 2024-05-09T08:48:47.522 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_concurrent_snap_create", 2024-05-09T08:48:47.522 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.522 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.523 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.523 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:47.523 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.523 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.523 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.523 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.523 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.523 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.523 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.523 INFO:teuthology.orchestra.run.smithi107.stdout: "mirror_snapshot_schedule": { 2024-05-09T08:48:47.523 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mirror_snapshot_schedule", 2024-05-09T08:48:47.523 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.524 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.524 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.524 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.524 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.524 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.524 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.524 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.524 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.524 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.524 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.524 INFO:teuthology.orchestra.run.smithi107.stdout: "trash_purge_schedule": { 2024-05-09T08:48:47.524 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "trash_purge_schedule", 2024-05-09T08:48:47.524 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.524 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.525 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.525 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.525 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.525 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.525 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.525 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.525 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.525 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.525 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.525 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.525 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.525 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "restful", 2024-05-09T08:48:47.526 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.526 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.526 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.526 INFO:teuthology.orchestra.run.smithi107.stdout: "enable_auth": { 2024-05-09T08:48:47.526 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enable_auth", 2024-05-09T08:48:47.526 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.526 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.526 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.526 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.526 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.526 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.526 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.526 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.526 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.527 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.527 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.527 INFO:teuthology.orchestra.run.smithi107.stdout: "key_file": { 2024-05-09T08:48:47.527 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "key_file", 2024-05-09T08:48:47.527 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.527 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.527 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.527 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.527 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.527 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.527 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.527 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.527 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.528 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.528 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.528 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.528 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.528 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.528 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.528 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.528 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.528 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.528 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.529 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.529 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.529 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.529 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.529 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.529 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.529 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.529 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.529 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.529 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.529 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.529 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.529 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.530 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.531 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.531 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.531 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.531 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.531 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.531 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.531 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.531 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.531 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.531 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.531 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.531 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.531 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.531 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.532 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.532 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.532 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.532 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.532 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.532 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.532 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.532 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.532 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.532 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.532 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.532 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.532 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.533 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.533 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.533 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.533 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.533 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.533 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-05-09T08:48:47.533 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-05-09T08:48:47.533 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.533 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.533 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.533 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.533 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.533 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.534 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.534 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.534 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.534 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.534 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.534 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-05-09T08:48:47.534 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-05-09T08:48:47.534 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.534 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.534 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.534 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.534 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.534 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rgw", 2024-05-09T08:48:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.536 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.536 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.536 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.536 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.536 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.536 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.536 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.536 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.536 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.536 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.536 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.536 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.536 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.536 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.536 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.537 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.537 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.537 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.537 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.537 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.537 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.537 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.537 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.537 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.537 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.537 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.537 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.537 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.538 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.538 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.538 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.538 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.538 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.538 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.538 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.538 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.538 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.538 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.538 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.538 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.538 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.539 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.540 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.540 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.540 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.540 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.540 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.540 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.540 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.540 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.540 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.540 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.540 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.540 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.540 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.541 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rook", 2024-05-09T08:48:47.541 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.541 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.541 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.541 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.541 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.541 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.541 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.541 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.541 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.541 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.541 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.541 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.541 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.542 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.542 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.542 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.542 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.542 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.542 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.542 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.542 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.542 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.542 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.542 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.542 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.543 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.543 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.543 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.543 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.543 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.543 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.543 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.543 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.543 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.543 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.543 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.543 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.543 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.544 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.544 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.544 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.544 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.544 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.544 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.544 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.544 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.544 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.544 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.544 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.544 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.544 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.544 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.545 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.545 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.545 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.545 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.545 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.545 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.545 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.545 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.545 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.545 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.545 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.545 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.545 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.546 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.546 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.546 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.546 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.546 INFO:teuthology.orchestra.run.smithi107.stdout: "storage_class": { 2024-05-09T08:48:47.546 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "storage_class", 2024-05-09T08:48:47.546 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.546 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.546 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.546 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "local", 2024-05-09T08:48:47.546 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.546 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.546 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.547 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-09T08:48:47.547 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.547 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.547 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.547 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.547 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.547 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "selftest", 2024-05-09T08:48:47.547 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.547 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.547 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.547 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.547 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.548 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.548 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.548 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.548 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.548 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.548 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.548 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.548 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.548 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.548 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.548 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.548 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.548 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.548 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.549 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.549 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.549 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.549 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.549 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.549 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.549 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.549 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.549 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.549 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.549 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.549 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.549 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.549 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.550 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.550 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.550 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.550 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.550 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.550 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.550 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.550 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.550 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.550 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.550 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.550 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.550 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.551 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.551 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.551 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.551 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.551 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.551 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.551 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.551 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.551 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.551 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.551 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.551 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.551 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.552 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.552 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.552 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.552 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.552 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.552 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.552 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.552 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.552 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.552 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.552 INFO:teuthology.orchestra.run.smithi107.stdout: "roption1": { 2024-05-09T08:48:47.552 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "roption1", 2024-05-09T08:48:47.552 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.553 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.553 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.553 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.553 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.553 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.553 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.553 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.553 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.553 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.553 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.553 INFO:teuthology.orchestra.run.smithi107.stdout: "roption2": { 2024-05-09T08:48:47.553 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "roption2", 2024-05-09T08:48:47.553 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.554 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.554 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.554 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "xyz", 2024-05-09T08:48:47.554 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.554 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.554 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.554 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.554 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.554 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.554 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.554 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption1": { 2024-05-09T08:48:47.554 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption1", 2024-05-09T08:48:47.554 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.555 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.555 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.555 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.555 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.555 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.555 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.555 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.555 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.555 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.555 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.555 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption2": { 2024-05-09T08:48:47.555 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption2", 2024-05-09T08:48:47.555 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.556 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.556 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.556 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.556 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.556 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.556 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.556 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.556 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.556 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.556 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.556 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption3": { 2024-05-09T08:48:47.556 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption3", 2024-05-09T08:48:47.557 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:47.557 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.557 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.557 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.557 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.557 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.557 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.557 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.557 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.557 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.557 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.557 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption4": { 2024-05-09T08:48:47.557 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption4", 2024-05-09T08:48:47.558 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.558 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.558 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.558 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.558 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.558 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.558 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.558 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.558 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.558 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.558 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.558 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption5": { 2024-05-09T08:48:47.559 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption5", 2024-05-09T08:48:47.559 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.559 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.559 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.559 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.559 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.559 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.559 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.559 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.559 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.559 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.559 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.559 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption6": { 2024-05-09T08:48:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption6", 2024-05-09T08:48:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.560 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption7": { 2024-05-09T08:48:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption7", 2024-05-09T08:48:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-05-09T08:48:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "42", 2024-05-09T08:48:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.561 INFO:teuthology.orchestra.run.smithi107.stdout: "testkey": { 2024-05-09T08:48:47.562 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testkey", 2024-05-09T08:48:47.562 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.562 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.562 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.562 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.562 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.562 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.562 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.562 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.562 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.562 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.562 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.563 INFO:teuthology.orchestra.run.smithi107.stdout: "testlkey": { 2024-05-09T08:48:47.563 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testlkey", 2024-05-09T08:48:47.563 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.563 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.563 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.563 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.563 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.563 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.563 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.563 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.563 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.563 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.563 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.564 INFO:teuthology.orchestra.run.smithi107.stdout: "testnewline": { 2024-05-09T08:48:47.564 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testnewline", 2024-05-09T08:48:47.564 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.564 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.564 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.564 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.564 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.564 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.564 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.564 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.564 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.564 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.564 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.564 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.565 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.565 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snap_schedule", 2024-05-09T08:48:47.565 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.565 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.565 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.565 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_m_granularity": { 2024-05-09T08:48:47.565 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_m_granularity", 2024-05-09T08:48:47.565 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.565 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.565 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.565 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.565 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.565 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.566 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.566 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow minute scheduled snapshots", 2024-05-09T08:48:47.566 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.566 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.566 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.566 INFO:teuthology.orchestra.run.smithi107.stdout: "dump_on_update": { 2024-05-09T08:48:47.566 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "dump_on_update", 2024-05-09T08:48:47.566 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.566 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.566 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.566 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.566 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.566 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.567 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.567 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "dump database to debug log on update", 2024-05-09T08:48:47.567 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.567 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.567 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.567 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.567 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.567 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.567 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.567 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.567 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.567 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.567 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.568 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.568 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.568 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.568 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.568 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.568 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.568 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.568 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.568 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.568 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.568 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.568 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.568 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.569 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.569 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.569 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.569 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.569 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.569 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.569 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.569 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.569 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.569 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.569 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.569 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.569 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.569 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.570 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.570 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.570 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.570 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.570 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.570 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.570 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.570 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.570 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.570 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.570 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.570 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.571 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.571 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.571 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.571 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.571 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.571 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.571 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.571 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.571 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.571 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.571 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.571 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.571 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.572 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.572 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.572 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.572 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.572 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.572 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.572 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.572 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.572 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "stats", 2024-05-09T08:48:47.572 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.572 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.572 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.572 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.573 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.573 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.573 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.573 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.573 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.573 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.573 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.573 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.573 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.573 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.573 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.573 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.573 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.573 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.574 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.574 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.574 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.574 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.574 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.574 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.574 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.574 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.574 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.574 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.574 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.574 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.575 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.575 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.575 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.575 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.575 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.575 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.575 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.575 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.575 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.575 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.575 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.575 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.575 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.576 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.576 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.576 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.576 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.576 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.576 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.576 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.576 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.576 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.576 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.576 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.576 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.576 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.577 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.577 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.577 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.577 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.577 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.577 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.577 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.577 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.577 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.577 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.577 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.577 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.577 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.577 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.577 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.578 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "status", 2024-05-09T08:48:47.578 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.578 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.578 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.578 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.578 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.578 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.578 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.578 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.578 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.578 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.578 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.579 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.579 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.579 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.579 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.579 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.579 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.579 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.579 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.579 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.579 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.579 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.579 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.579 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.579 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.579 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.580 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.580 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.580 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.580 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.580 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.580 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.580 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.580 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.580 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.580 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.580 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.580 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.580 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.581 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.581 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.581 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.581 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.581 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.581 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.581 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.581 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.581 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.581 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.581 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.581 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.581 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.581 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.581 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.581 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.582 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.582 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.582 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.582 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.582 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.582 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.582 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.582 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.582 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.582 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.582 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.582 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.583 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.583 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.583 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.583 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.583 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "telegraf", 2024-05-09T08:48:47.583 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.583 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.583 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.583 INFO:teuthology.orchestra.run.smithi107.stdout: "address": { 2024-05-09T08:48:47.583 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "address", 2024-05-09T08:48:47.583 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.583 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.583 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.583 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-09T08:48:47.584 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.584 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.584 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.584 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.584 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.584 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.584 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.584 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-05-09T08:48:47.584 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-05-09T08:48:47.584 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.584 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.584 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.584 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15", 2024-05-09T08:48:47.585 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.585 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.585 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.585 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.585 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.585 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.585 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.585 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.585 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.585 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.585 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.585 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.585 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.586 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.586 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.586 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.586 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.586 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.586 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.586 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.586 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.586 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.586 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.586 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.586 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.586 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.586 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.587 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.587 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.587 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.587 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.587 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.587 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.587 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.587 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.587 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.587 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.587 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.587 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.587 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.588 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.588 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.588 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.588 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.588 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.588 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.588 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.588 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.588 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.588 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.588 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.588 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.588 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.589 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.589 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.589 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.589 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.589 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.589 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.589 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.589 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.589 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.589 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.589 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.589 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.589 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.590 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.590 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.590 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.590 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.590 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.590 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.590 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.590 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.590 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.590 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "telemetry", 2024-05-09T08:48:47.590 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.590 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.591 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.591 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_basic": { 2024-05-09T08:48:47.591 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_basic", 2024-05-09T08:48:47.591 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.591 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.591 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.591 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.591 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.591 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.591 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.591 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-09T08:48:47.591 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.591 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.592 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.592 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_crash": { 2024-05-09T08:48:47.592 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_crash", 2024-05-09T08:48:47.592 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.592 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.592 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.592 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.592 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.592 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.592 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.592 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-09T08:48:47.592 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.592 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.593 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.593 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_device": { 2024-05-09T08:48:47.593 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_device", 2024-05-09T08:48:47.593 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.593 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.593 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.593 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.593 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.593 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.593 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.593 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-09T08:48:47.593 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.593 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.594 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.594 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_ident": { 2024-05-09T08:48:47.594 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_ident", 2024-05-09T08:48:47.594 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.594 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.594 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.594 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.594 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.594 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.594 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.594 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-09T08:48:47.594 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.594 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.595 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.595 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_perf": { 2024-05-09T08:48:47.595 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_perf", 2024-05-09T08:48:47.595 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.595 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.595 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.595 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.595 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.595 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.595 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.595 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-09T08:48:47.595 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.595 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.596 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.596 INFO:teuthology.orchestra.run.smithi107.stdout: "contact": { 2024-05-09T08:48:47.596 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "contact", 2024-05-09T08:48:47.596 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.596 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.596 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.596 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.596 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.596 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.596 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.596 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.596 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.596 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.597 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.597 INFO:teuthology.orchestra.run.smithi107.stdout: "description": { 2024-05-09T08:48:47.597 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "description", 2024-05-09T08:48:47.597 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.597 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.597 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.597 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.597 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.597 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.597 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.597 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.597 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.597 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.598 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.598 INFO:teuthology.orchestra.run.smithi107.stdout: "device_url": { 2024-05-09T08:48:47.598 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_url", 2024-05-09T08:48:47.598 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.598 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.598 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.598 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-09T08:48:47.598 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.598 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.598 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.598 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.598 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.598 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.599 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.599 INFO:teuthology.orchestra.run.smithi107.stdout: "enabled": { 2024-05-09T08:48:47.599 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enabled", 2024-05-09T08:48:47.599 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.599 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.599 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.599 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.599 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.599 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.599 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.599 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.599 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.600 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.600 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.600 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-05-09T08:48:47.600 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-05-09T08:48:47.600 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.600 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.600 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.600 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "24", 2024-05-09T08:48:47.600 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "8", 2024-05-09T08:48:47.600 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.600 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.600 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.600 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.600 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.601 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.601 INFO:teuthology.orchestra.run.smithi107.stdout: "last_opt_revision": { 2024-05-09T08:48:47.601 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "last_opt_revision", 2024-05-09T08:48:47.601 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.601 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.601 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.601 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1", 2024-05-09T08:48:47.601 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.601 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.601 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.601 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.601 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.601 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.602 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.602 INFO:teuthology.orchestra.run.smithi107.stdout: "leaderboard": { 2024-05-09T08:48:47.602 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "leaderboard", 2024-05-09T08:48:47.602 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.602 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.602 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.602 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.602 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.602 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.602 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.602 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.602 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.603 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.603 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.603 INFO:teuthology.orchestra.run.smithi107.stdout: "leaderboard_description": { 2024-05-09T08:48:47.603 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "leaderboard_description", 2024-05-09T08:48:47.603 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.603 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.603 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.603 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.603 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.603 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.603 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.603 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.603 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.604 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.604 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.604 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.604 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.604 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.604 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.604 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.604 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.604 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.604 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.604 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.604 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.604 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.604 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.605 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.605 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.605 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.605 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.605 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.605 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.605 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.605 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.605 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.605 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.605 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.605 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.605 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.606 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.606 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.606 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.606 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.606 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.606 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.606 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.606 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.606 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.606 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.606 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.606 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.607 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.607 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.607 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.607 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.607 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.607 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.607 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.607 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.607 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.607 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.607 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.607 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.607 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.607 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.608 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.608 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.608 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.608 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.608 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.608 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.608 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.608 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.608 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.608 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.608 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.608 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.608 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.608 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.609 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.609 INFO:teuthology.orchestra.run.smithi107.stdout: "organization": { 2024-05-09T08:48:47.609 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "organization", 2024-05-09T08:48:47.609 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.609 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.609 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.609 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.609 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.609 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.609 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.609 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.609 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.609 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.610 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.610 INFO:teuthology.orchestra.run.smithi107.stdout: "proxy": { 2024-05-09T08:48:47.610 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "proxy", 2024-05-09T08:48:47.610 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.610 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.610 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.610 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.610 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.610 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.610 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.610 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.610 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.610 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.611 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.611 INFO:teuthology.orchestra.run.smithi107.stdout: "url": { 2024-05-09T08:48:47.611 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "url", 2024-05-09T08:48:47.611 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.611 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.611 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.611 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-09T08:48:47.611 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.611 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.611 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.611 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.611 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.612 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.612 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.612 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.612 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.612 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "test_orchestrator", 2024-05-09T08:48:47.612 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.612 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.612 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.612 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.612 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.612 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.612 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.613 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.613 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.613 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.613 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.613 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.613 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.613 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.613 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.613 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.613 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.613 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.613 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.613 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.613 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.614 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.615 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.616 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "volumes", 2024-05-09T08:48:47.617 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.618 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.619 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.620 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.622 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "max_concurrent_clones": { 2024-05-09T08:48:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_concurrent_clones", 2024-05-09T08:48:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "4", 2024-05-09T08:48:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-09T08:48:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "periodic_async_work": { 2024-05-09T08:48:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "periodic_async_work", 2024-05-09T08:48:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Periodically check for async work", 2024-05-09T08:48:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.624 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot_clone_delay": { 2024-05-09T08:48:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snapshot_clone_delay", 2024-05-09T08:48:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-05-09T08:48:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-09T08:48:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.625 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot_clone_no_wait": { 2024-05-09T08:48:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snapshot_clone_no_wait", 2024-05-09T08:48:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-09T08:48:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.626 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix", 2024-05-09T08:48:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "discovery_interval": { 2024-05-09T08:48:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "discovery_interval", 2024-05-09T08:48:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-05-09T08:48:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "100", 2024-05-09T08:48:47.627 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "identifier": { 2024-05-09T08:48:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "identifier", 2024-05-09T08:48:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.628 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-05-09T08:48:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-05-09T08:48:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-05-09T08:48:47.629 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.630 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.631 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.632 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.633 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.634 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_host": { 2024-05-09T08:48:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_host", 2024-05-09T08:48:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.635 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_port": { 2024-05-09T08:48:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_port", 2024-05-09T08:48:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10051", 2024-05-09T08:48:47.636 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_sender": { 2024-05-09T08:48:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_sender", 2024-05-09T08:48:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-09T08:48:47.637 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-05-09T08:48:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "gid": 24968, 2024-05-09T08:48:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "z", 2024-05-09T08:48:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_features": 4540701547738038271, 2024-05-09T08:48:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "available_modules": [ 2024-05-09T08:48:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "alerts", 2024-05-09T08:48:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-05-09T08:48:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-05-09T08:48:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.639 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-05-09T08:48:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How frequently to reexamine health status", 2024-05-09T08:48:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.640 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.642 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.643 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.644 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_destination": { 2024-05-09T08:48:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_destination", 2024-05-09T08:48:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.645 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Email address to send alerts to", 2024-05-09T08:48:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_from_name": { 2024-05-09T08:48:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_from_name", 2024-05-09T08:48:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.646 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.647 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "Ceph", 2024-05-09T08:48:47.647 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.647 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.647 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.647 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Email From: name", 2024-05-09T08:48:47.647 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.647 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.647 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.647 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_host": { 2024-05-09T08:48:47.647 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_host", 2024-05-09T08:48:47.647 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.647 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.647 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.647 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP server", 2024-05-09T08:48:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_password": { 2024-05-09T08:48:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_password", 2024-05-09T08:48:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.648 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Password to authenticate with", 2024-05-09T08:48:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_port": { 2024-05-09T08:48:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_port", 2024-05-09T08:48:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "465", 2024-05-09T08:48:47.649 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP port", 2024-05-09T08:48:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_sender": { 2024-05-09T08:48:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_sender", 2024-05-09T08:48:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.650 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP envelope sender", 2024-05-09T08:48:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_ssl": { 2024-05-09T08:48:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_ssl", 2024-05-09T08:48:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.651 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-09T08:48:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_user": { 2024-05-09T08:48:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_user", 2024-05-09T08:48:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.652 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "User to authenticate as", 2024-05-09T08:48:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "balancer", 2024-05-09T08:48:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.653 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "active": { 2024-05-09T08:48:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "active", 2024-05-09T08:48:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "automatically balance PGs across cluster", 2024-05-09T08:48:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.654 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "begin_time": { 2024-05-09T08:48:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "begin_time", 2024-05-09T08:48:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0000", 2024-05-09T08:48:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "beginning time of day to automatically balance", 2024-05-09T08:48:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T08:48:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.655 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "begin_weekday": { 2024-05-09T08:48:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "begin_weekday", 2024-05-09T08:48:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-05-09T08:48:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-05-09T08:48:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0", 2024-05-09T08:48:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "6", 2024-05-09T08:48:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-09T08:48:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T08:48:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_max_iterations": { 2024-05-09T08:48:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_max_iterations", 2024-05-09T08:48:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-05-09T08:48:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "25", 2024-05-09T08:48:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-05-09T08:48:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "250", 2024-05-09T08:48:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-09T08:48:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_metrics": { 2024-05-09T08:48:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_metrics", 2024-05-09T08:48:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "pgs,objects,bytes", 2024-05-09T08:48:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-09T08:48:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-09T08:48:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_step": { 2024-05-09T08:48:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_step", 2024-05-09T08:48:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0.5", 2024-05-09T08:48:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0.001", 2024-05-09T08:48:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "0.999", 2024-05-09T08:48:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "aggressiveness of optimization", 2024-05-09T08:48:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-09T08:48:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "end_time": { 2024-05-09T08:48:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "end_time", 2024-05-09T08:48:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "2359", 2024-05-09T08:48:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "ending time of day to automatically balance", 2024-05-09T08:48:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T08:48:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.660 INFO:teuthology.orchestra.run.smithi107.stdout: "end_weekday": { 2024-05-09T08:48:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "end_weekday", 2024-05-09T08:48:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-05-09T08:48:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-05-09T08:48:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0", 2024-05-09T08:48:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "6", 2024-05-09T08:48:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-09T08:48:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T08:48:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.661 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.663 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.664 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.665 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.666 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.667 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.668 INFO:teuthology.orchestra.run.smithi107.stdout: "min_score": { 2024-05-09T08:48:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "min_score", 2024-05-09T08:48:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-05-09T08:48:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.669 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-09T08:48:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": { 2024-05-09T08:48:47.670 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mode", 2024-05-09T08:48:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "upmap", 2024-05-09T08:48:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.671 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-compat", 2024-05-09T08:48:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "none", 2024-05-09T08:48:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "read", 2024-05-09T08:48:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap", 2024-05-09T08:48:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap-read" 2024-05-09T08:48:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Balancer mode", 2024-05-09T08:48:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_ids": { 2024-05-09T08:48:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pool_ids", 2024-05-09T08:48:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.673 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-09T08:48:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-05-09T08:48:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-05-09T08:48:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-05-09T08:48:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-09T08:48:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap_max_deviation": { 2024-05-09T08:48:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "upmap_max_deviation", 2024-05-09T08:48:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-05-09T08:48:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-05-09T08:48:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-09T08:48:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-09T08:48:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap_max_optimizations": { 2024-05-09T08:48:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "upmap_max_optimizations", 2024-05-09T08:48:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-05-09T08:48:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.676 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-09T08:48:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephadm", 2024-05-09T08:48:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_down_multiplier": { 2024-05-09T08:48:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_down_multiplier", 2024-05-09T08:48:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:47.677 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3.0", 2024-05-09T08:48:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-09T08:48:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_refresh_rate": { 2024-05-09T08:48:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_refresh_rate", 2024-05-09T08:48:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.678 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "20", 2024-05-09T08:48:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-09T08:48:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_starting_port": { 2024-05-09T08:48:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_starting_port", 2024-05-09T08:48:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.679 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "4721", 2024-05-09T08:48:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-09T08:48:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_ptrace": { 2024-05-09T08:48:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_ptrace", 2024-05-09T08:48:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.680 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-09T08:48:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-09T08:48:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: "autotune_interval": { 2024-05-09T08:48:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "autotune_interval", 2024-05-09T08:48:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.681 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.682 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.682 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-05-09T08:48:47.682 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.682 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.682 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.682 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-09T08:48:47.682 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.682 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.682 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.682 INFO:teuthology.orchestra.run.smithi107.stdout: "autotune_memory_target_ratio": { 2024-05-09T08:48:47.682 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "autotune_memory_target_ratio", 2024-05-09T08:48:47.682 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:47.682 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.683 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.683 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0.7", 2024-05-09T08:48:47.683 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.683 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.683 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.683 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-09T08:48:47.683 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.683 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.683 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.683 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm_log_destination": { 2024-05-09T08:48:47.683 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephadm_log_destination", 2024-05-09T08:48:47.683 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.683 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "file", 2024-05-09T08:48:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "file,syslog", 2024-05-09T08:48:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "syslog" 2024-05-09T08:48:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-09T08:48:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "cgroups_split": { 2024-05-09T08:48:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cgroups_split", 2024-05-09T08:48:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-09T08:48:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "config_checks_enabled": { 2024-05-09T08:48:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_checks_enabled", 2024-05-09T08:48:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-09T08:48:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.686 INFO:teuthology.orchestra.run.smithi107.stdout: "config_dashboard": { 2024-05-09T08:48:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_dashboard", 2024-05-09T08:48:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-09T08:48:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.687 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_alertmanager": { 2024-05-09T08:48:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_alertmanager", 2024-05-09T08:48:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-09T08:48:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-05-09T08:48:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_base": { 2024-05-09T08:48:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_base", 2024-05-09T08:48:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-09T08:48:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Container image name, without the tag", 2024-05-09T08:48:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_elasticsearch": { 2024-05-09T08:48:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_elasticsearch", 2024-05-09T08:48:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-09T08:48:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "elasticsearch container image", 2024-05-09T08:48:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_grafana": { 2024-05-09T08:48:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_grafana", 2024-05-09T08:48:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-09T08:48:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-05-09T08:48:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.691 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_haproxy": { 2024-05-09T08:48:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_haproxy", 2024-05-09T08:48:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-09T08:48:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "HAproxy container image", 2024-05-09T08:48:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_agent": { 2024-05-09T08:48:47.692 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_agent", 2024-05-09T08:48:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-09T08:48:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger agent container image", 2024-05-09T08:48:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.693 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_collector": { 2024-05-09T08:48:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_collector", 2024-05-09T08:48:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-09T08:48:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger collector container image", 2024-05-09T08:48:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.694 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_query": { 2024-05-09T08:48:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_query", 2024-05-09T08:48:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-09T08:48:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger query container image", 2024-05-09T08:48:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.695 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_keepalived": { 2024-05-09T08:48:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_keepalived", 2024-05-09T08:48:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-09T08:48:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Keepalived container image", 2024-05-09T08:48:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_loki": { 2024-05-09T08:48:47.696 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_loki", 2024-05-09T08:48:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-09T08:48:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Loki container image", 2024-05-09T08:48:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_node_exporter": { 2024-05-09T08:48:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_node_exporter", 2024-05-09T08:48:47.697 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-09T08:48:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-05-09T08:48:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_nvmeof": { 2024-05-09T08:48:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_nvmeof", 2024-05-09T08:48:47.698 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-09T08:48:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Nvme-of container image", 2024-05-09T08:48:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_prometheus": { 2024-05-09T08:48:47.699 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_prometheus", 2024-05-09T08:48:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-09T08:48:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-05-09T08:48:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_promtail": { 2024-05-09T08:48:47.700 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_promtail", 2024-05-09T08:48:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-09T08:48:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Promtail container image", 2024-05-09T08:48:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_samba": { 2024-05-09T08:48:47.701 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_samba", 2024-05-09T08:48:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-09T08:48:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Samba/SMB container image", 2024-05-09T08:48:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_snmp_gateway": { 2024-05-09T08:48:47.702 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_snmp_gateway", 2024-05-09T08:48:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-09T08:48:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SNMP Gateway container image", 2024-05-09T08:48:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.703 INFO:teuthology.orchestra.run.smithi107.stdout: "container_init": { 2024-05-09T08:48:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_init", 2024-05-09T08:48:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Run podman/docker with `--init`", 2024-05-09T08:48:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.704 INFO:teuthology.orchestra.run.smithi107.stdout: "daemon_cache_timeout": { 2024-05-09T08:48:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "daemon_cache_timeout", 2024-05-09T08:48:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-05-09T08:48:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-09T08:48:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "default_cephadm_command_timeout": { 2024-05-09T08:48:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "default_cephadm_command_timeout", 2024-05-09T08:48:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "900", 2024-05-09T08:48:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-09T08:48:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "default_registry": { 2024-05-09T08:48:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "default_registry", 2024-05-09T08:48:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io", 2024-05-09T08:48:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-09T08:48:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "device_cache_timeout": { 2024-05-09T08:48:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_cache_timeout", 2024-05-09T08:48:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1800", 2024-05-09T08:48:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache device inventory", 2024-05-09T08:48:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.708 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "device_enhanced_scan": { 2024-05-09T08:48:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_enhanced_scan", 2024-05-09T08:48:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-09T08:48:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.709 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "facts_cache_timeout": { 2024-05-09T08:48:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "facts_cache_timeout", 2024-05-09T08:48:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-05-09T08:48:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache host facts data", 2024-05-09T08:48:47.710 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "grafana_dashboards_path": { 2024-05-09T08:48:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "grafana_dashboards_path", 2024-05-09T08:48:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-09T08:48:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-09T08:48:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.711 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "host_check_interval": { 2024-05-09T08:48:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "host_check_interval", 2024-05-09T08:48:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-05-09T08:48:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to perform a host check", 2024-05-09T08:48:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.712 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "hw_monitoring": { 2024-05-09T08:48:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "hw_monitoring", 2024-05-09T08:48:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-09T08:48:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "inventory_list_all": { 2024-05-09T08:48:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "inventory_list_all", 2024-05-09T08:48:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-09T08:48:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.715 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "log_refresh_metadata": { 2024-05-09T08:48:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_refresh_metadata", 2024-05-09T08:48:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.716 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-09T08:48:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.717 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-09T08:48:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.718 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.720 INFO:teuthology.orchestra.run.smithi107.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-09T08:48:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-09T08:48:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-09T08:48:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.721 INFO:teuthology.orchestra.run.smithi107.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-09T08:48:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-09T08:48:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "*", 2024-05-09T08:48:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-09T08:48:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.722 INFO:teuthology.orchestra.run.smithi107.stdout: "max_count_per_host": { 2024-05-09T08:48:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_count_per_host", 2024-05-09T08:48:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "max number of daemons per service per host", 2024-05-09T08:48:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.723 INFO:teuthology.orchestra.run.smithi107.stdout: "max_osd_draining_count": { 2024-05-09T08:48:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_osd_draining_count", 2024-05-09T08:48:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-09T08:48:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.724 INFO:teuthology.orchestra.run.smithi107.stdout: "migration_current": { 2024-05-09T08:48:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "migration_current", 2024-05-09T08:48:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "internal - do not modify", 2024-05-09T08:48:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": { 2024-05-09T08:48:47.725 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mode", 2024-05-09T08:48:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "root", 2024-05-09T08:48:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm-package", 2024-05-09T08:48:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "root" 2024-05-09T08:48:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "mode for remote execution of cephadm", 2024-05-09T08:48:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "oob_default_addr": { 2024-05-09T08:48:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "oob_default_addr", 2024-05-09T08:48:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "169.254.1.1", 2024-05-09T08:48:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-09T08:48:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "prometheus_alerts_path": { 2024-05-09T08:48:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prometheus_alerts_path", 2024-05-09T08:48:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-09T08:48:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-09T08:48:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_insecure": { 2024-05-09T08:48:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_insecure", 2024-05-09T08:48:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-09T08:48:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_password": { 2024-05-09T08:48:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_password", 2024-05-09T08:48:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-09T08:48:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_url": { 2024-05-09T08:48:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_url", 2024-05-09T08:48:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-09T08:48:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_username": { 2024-05-09T08:48:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_username", 2024-05-09T08:48:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-09T08:48:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "secure_monitoring_stack": { 2024-05-09T08:48:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "secure_monitoring_stack", 2024-05-09T08:48:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-09T08:48:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "service_discovery_port": { 2024-05-09T08:48:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "service_discovery_port", 2024-05-09T08:48:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8765", 2024-05-09T08:48:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "cephadm service discovery port", 2024-05-09T08:48:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ssh_config_file": { 2024-05-09T08:48:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssh_config_file", 2024-05-09T08:48:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-09T08:48:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "use_agent": { 2024-05-09T08:48:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "use_agent", 2024-05-09T08:48:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-09T08:48:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "use_repo_digest": { 2024-05-09T08:48:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "use_repo_digest", 2024-05-09T08:48:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-09T08:48:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_failed_host_check": { 2024-05-09T08:48:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_failed_host_check", 2024-05-09T08:48:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if the host check fails", 2024-05-09T08:48:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_stray_daemons": { 2024-05-09T08:48:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_stray_daemons", 2024-05-09T08:48:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-09T08:48:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_stray_hosts": { 2024-05-09T08:48:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_stray_hosts", 2024-05-09T08:48:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-09T08:48:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crash", 2024-05-09T08:48:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "retain_interval": { 2024-05-09T08:48:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "retain_interval", 2024-05-09T08:48:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "31536000", 2024-05-09T08:48:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-09T08:48:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_recent_interval": { 2024-05-09T08:48:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_recent_interval", 2024-05-09T08:48:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1209600", 2024-05-09T08:48:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-09T08:48:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "dashboard", 2024-05-09T08:48:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-09T08:48:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-09T08:48:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "ALERTMANAGER_API_HOST": { 2024-05-09T08:48:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-09T08:48:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-09T08:48:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-09T08:48:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "AUDIT_API_ENABLED": { 2024-05-09T08:48:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "AUDIT_API_ENABLED", 2024-05-09T08:48:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-09T08:48:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-09T08:48:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "ENABLE_BROWSABLE_API": { 2024-05-09T08:48:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-09T08:48:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-09T08:48:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-09T08:48:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-09T08:48:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-09T08:48:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-09T08:48:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-09T08:48:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-09T08:48:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-09T08:48:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-09T08:48:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-09T08:48:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-09T08:48:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-09T08:48:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-09T08:48:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-09T08:48:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-09T08:48:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-09T08:48:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_PASSWORD": { 2024-05-09T08:48:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-09T08:48:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-05-09T08:48:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-09T08:48:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-09T08:48:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_URL": { 2024-05-09T08:48:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_URL", 2024-05-09T08:48:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_USERNAME": { 2024-05-09T08:48:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-09T08:48:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-05-09T08:48:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-09T08:48:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-09T08:48:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-09T08:48:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-09T08:48:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-09T08:48:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-09T08:48:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-09T08:48:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-09T08:48:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: "PROMETHEUS_API_HOST": { 2024-05-09T08:48:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-09T08:48:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-09T08:48:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-09T08:48:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.773 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-09T08:48:47.773 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-09T08:48:47.773 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.773 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.773 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.773 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.773 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.773 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.773 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.773 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.773 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.773 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.773 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.774 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-09T08:48:47.774 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-09T08:48:47.774 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.774 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.774 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.774 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.774 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.774 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.774 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.774 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.774 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.774 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.775 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.775 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-09T08:48:47.775 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-09T08:48:47.775 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.775 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.775 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.775 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.775 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.775 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.775 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.775 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.775 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.775 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-09T08:48:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-09T08:48:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.776 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-09T08:48:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-09T08:48:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.777 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-09T08:48:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-09T08:48:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.778 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-09T08:48:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-09T08:48:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.779 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_ENABLED": { 2024-05-09T08:48:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-09T08:48:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.780 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-09T08:48:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-09T08:48:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-09T08:48:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.781 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-09T08:48:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-09T08:48:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.782 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-09T08:48:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-09T08:48:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8", 2024-05-09T08:48:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.783 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-09T08:48:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-09T08:48:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "45", 2024-05-09T08:48:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.785 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_ACCESS_KEY": { 2024-05-09T08:48:47.785 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-09T08:48:47.785 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.785 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.785 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.785 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.785 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.785 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.785 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.785 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.785 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.785 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.785 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-09T08:48:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-09T08:48:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-05-09T08:48:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_SECRET_KEY": { 2024-05-09T08:48:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-09T08:48:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.787 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_SSL_VERIFY": { 2024-05-09T08:48:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-09T08:48:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.788 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-09T08:48:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-09T08:48:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-05-09T08:48:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-09T08:48:47.789 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-09T08:48:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-09T08:48:47.790 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-09T08:48:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-05-09T08:48:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: "cross_origin_url": { 2024-05-09T08:48:47.791 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cross_origin_url", 2024-05-09T08:48:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: "crt_file": { 2024-05-09T08:48:47.792 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crt_file", 2024-05-09T08:48:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: "debug": { 2024-05-09T08:48:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "debug", 2024-05-09T08:48:47.793 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.794 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.794 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.794 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.794 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.794 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.794 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.794 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable/disable debug options", 2024-05-09T08:48:47.794 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.794 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.794 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.794 INFO:teuthology.orchestra.run.smithi107.stdout: "jwt_token_ttl": { 2024-05-09T08:48:47.794 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "jwt_token_ttl", 2024-05-09T08:48:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "28800", 2024-05-09T08:48:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: "key_file": { 2024-05-09T08:48:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "key_file", 2024-05-09T08:48:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.796 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.797 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.798 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.799 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.800 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "motd": { 2024-05-09T08:48:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "motd", 2024-05-09T08:48:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.801 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.802 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.802 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.802 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.802 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.802 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.802 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "The message of the day", 2024-05-09T08:48:47.802 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.802 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.802 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.802 INFO:teuthology.orchestra.run.smithi107.stdout: "redirect_resolve_ip_addr": { 2024-05-09T08:48:47.802 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "redirect_resolve_ip_addr", 2024-05-09T08:48:47.802 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.802 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-05-09T08:48:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-05-09T08:48:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.803 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "::", 2024-05-09T08:48:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-05-09T08:48:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-05-09T08:48:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.804 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8080", 2024-05-09T08:48:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl": { 2024-05-09T08:48:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl", 2024-05-09T08:48:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.805 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl_server_port": { 2024-05-09T08:48:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl_server_port", 2024-05-09T08:48:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8443", 2024-05-09T08:48:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_behaviour": { 2024-05-09T08:48:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_behaviour", 2024-05-09T08:48:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "redirect", 2024-05-09T08:48:47.808 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.808 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.808 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.808 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.808 INFO:teuthology.orchestra.run.smithi107.stdout: "redirect" 2024-05-09T08:48:47.808 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.808 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.808 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.808 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.808 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.808 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_error_status_code": { 2024-05-09T08:48:47.808 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_error_status_code", 2024-05-09T08:48:47.808 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "500", 2024-05-09T08:48:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "400", 2024-05-09T08:48:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "599", 2024-05-09T08:48:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: "url_prefix": { 2024-05-09T08:48:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "url_prefix", 2024-05-09T08:48:47.809 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "devicehealth", 2024-05-09T08:48:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "enable_monitoring": { 2024-05-09T08:48:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enable_monitoring", 2024-05-09T08:48:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.811 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "monitor device health metrics", 2024-05-09T08:48:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.812 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.814 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.815 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.816 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: "mark_out_threshold": { 2024-05-09T08:48:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mark_out_threshold", 2024-05-09T08:48:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "2419200", 2024-05-09T08:48:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-09T08:48:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.817 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": { 2024-05-09T08:48:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pool_name", 2024-05-09T08:48:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "device_health_metrics", 2024-05-09T08:48:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-09T08:48:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.818 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: "retention_period": { 2024-05-09T08:48:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "retention_period", 2024-05-09T08:48:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15552000", 2024-05-09T08:48:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long to retain device health metrics", 2024-05-09T08:48:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: "scrape_frequency": { 2024-05-09T08:48:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "scrape_frequency", 2024-05-09T08:48:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "86400", 2024-05-09T08:48:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-09T08:48:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.823 INFO:teuthology.orchestra.run.smithi107.stdout: "self_heal": { 2024-05-09T08:48:47.824 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "self_heal", 2024-05-09T08:48:47.824 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.824 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.824 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.824 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.824 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.824 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.824 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.824 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-09T08:48:47.824 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.824 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.824 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.825 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-05-09T08:48:47.825 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-05-09T08:48:47.825 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.825 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.825 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.825 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-05-09T08:48:47.825 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.825 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.825 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.825 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to wake up and check device health", 2024-05-09T08:48:47.825 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.825 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.825 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.826 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_threshold": { 2024-05-09T08:48:47.826 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_threshold", 2024-05-09T08:48:47.826 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.826 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.826 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.826 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "7257600", 2024-05-09T08:48:47.826 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.826 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.826 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.826 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-09T08:48:47.826 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.826 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.826 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.826 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.827 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.827 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "diskprediction_local", 2024-05-09T08:48:47.827 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.827 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.827 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.827 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.827 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.827 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.827 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.827 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.827 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.827 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.828 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.828 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.828 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.828 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.828 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.828 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.828 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.828 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.828 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.828 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.828 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.828 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.828 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.829 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.829 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.829 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.829 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.829 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.829 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.829 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.829 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.829 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.829 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.829 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.829 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.830 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.831 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: "predict_interval": { 2024-05-09T08:48:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "predict_interval", 2024-05-09T08:48:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.832 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "86400", 2024-05-09T08:48:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "predictor_model": { 2024-05-09T08:48:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "predictor_model", 2024-05-09T08:48:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.833 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "prophetstor", 2024-05-09T08:48:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-05-09T08:48:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-05-09T08:48:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.834 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-05-09T08:48:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "influx", 2024-05-09T08:48:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": false, 2024-05-09T08:48:47.835 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "influxdb python module not found", 2024-05-09T08:48:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "batch_size": { 2024-05-09T08:48:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "batch_size", 2024-05-09T08:48:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5000", 2024-05-09T08:48:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-09T08:48:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.837 INFO:teuthology.orchestra.run.smithi107.stdout: "database": { 2024-05-09T08:48:47.837 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "database", 2024-05-09T08:48:47.837 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.837 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.837 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.837 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "ceph", 2024-05-09T08:48:47.837 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.837 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.837 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.837 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-09T08:48:47.837 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.837 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.837 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: "hostname": { 2024-05-09T08:48:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "hostname", 2024-05-09T08:48:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB server hostname", 2024-05-09T08:48:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.838 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-05-09T08:48:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-05-09T08:48:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "30", 2024-05-09T08:48:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "5", 2024-05-09T08:48:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-09T08:48:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.839 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.840 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.840 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.840 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.840 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.840 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.840 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.840 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.840 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.840 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.840 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.840 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.840 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.840 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.840 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.841 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.842 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.843 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: "password": { 2024-05-09T08:48:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "password", 2024-05-09T08:48:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "password of InfluxDB server user", 2024-05-09T08:48:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.845 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: "port": { 2024-05-09T08:48:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "port", 2024-05-09T08:48:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8086", 2024-05-09T08:48:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB server port", 2024-05-09T08:48:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.846 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl": { 2024-05-09T08:48:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl", 2024-05-09T08:48:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "false", 2024-05-09T08:48:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T08:48:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.847 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: "threads": { 2024-05-09T08:48:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "threads", 2024-05-09T08:48:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-05-09T08:48:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-05-09T08:48:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "32", 2024-05-09T08:48:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-09T08:48:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "username": { 2024-05-09T08:48:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "username", 2024-05-09T08:48:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "username of InfluxDB server user", 2024-05-09T08:48:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.850 INFO:teuthology.orchestra.run.smithi107.stdout: "verify_ssl": { 2024-05-09T08:48:47.850 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "verify_ssl", 2024-05-09T08:48:47.850 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.850 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.850 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.850 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "true", 2024-05-09T08:48:47.850 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.850 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.850 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.850 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T08:48:47.850 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.850 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.850 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "insights", 2024-05-09T08:48:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.851 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.852 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.853 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.854 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.855 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "iostat", 2024-05-09T08:48:47.856 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.857 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.858 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.858 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.858 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.858 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.858 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.858 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.858 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.858 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.858 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.859 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.859 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.859 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.859 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.859 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.859 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.859 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.860 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.861 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.862 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "k8sevents", 2024-05-09T08:48:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_event_retention_days": { 2024-05-09T08:48:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ceph_event_retention_days", 2024-05-09T08:48:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.863 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "7", 2024-05-09T08:48:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-09T08:48:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "config_check_secs": { 2024-05-09T08:48:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_check_secs", 2024-05-09T08:48:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.864 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "10", 2024-05-09T08:48:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-09T08:48:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.865 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.866 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.867 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.868 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.869 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.870 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "localpool", 2024-05-09T08:48:47.871 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.871 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.871 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.871 INFO:teuthology.orchestra.run.smithi107.stdout: "failure_domain": { 2024-05-09T08:48:47.871 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "failure_domain", 2024-05-09T08:48:47.871 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.871 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.871 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.871 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "host", 2024-05-09T08:48:47.871 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.871 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.871 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.871 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "failure domain for any created local pool", 2024-05-09T08:48:47.871 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-09T08:48:47.871 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.872 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.873 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.874 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.875 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.875 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.875 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.875 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.875 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.875 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.875 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.875 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.875 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.875 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.875 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.875 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.875 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.875 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.876 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.876 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.876 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.876 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.876 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.876 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.876 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.876 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.876 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.876 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.876 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.876 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.876 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.877 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": { 2024-05-09T08:48:47.877 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "min_size", 2024-05-09T08:48:47.877 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.877 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.877 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.877 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.877 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.877 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.877 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.877 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default min_size for any created local pool", 2024-05-09T08:48:47.877 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-09T08:48:47.877 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.877 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: "num_rep": { 2024-05-09T08:48:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "num_rep", 2024-05-09T08:48:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3", 2024-05-09T08:48:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default replica count for any created local pool", 2024-05-09T08:48:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.879 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": { 2024-05-09T08:48:47.879 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pg_num", 2024-05-09T08:48:47.879 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.879 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.879 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.879 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "128", 2024-05-09T08:48:47.879 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.879 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.879 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.879 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default pg_num for any created local pool", 2024-05-09T08:48:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "prefix": { 2024-05-09T08:48:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prefix", 2024-05-09T08:48:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "name prefix for any created local pool", 2024-05-09T08:48:47.880 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: "subtree": { 2024-05-09T08:48:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "subtree", 2024-05-09T08:48:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "rack", 2024-05-09T08:48:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-09T08:48:47.881 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-09T08:48:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mds_autoscaler", 2024-05-09T08:48:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.882 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.883 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.884 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.884 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.884 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.884 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.884 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.884 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.884 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.884 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.884 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.884 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.884 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.884 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.884 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.885 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.886 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mirroring", 2024-05-09T08:48:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.888 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.889 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.890 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.891 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.892 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "nfs", 2024-05-09T08:48:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.893 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.894 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.895 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.896 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.897 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "orchestrator", 2024-05-09T08:48:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.898 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "fail_fs": { 2024-05-09T08:48:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "fail_fs", 2024-05-09T08:48:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-09T08:48:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.899 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.900 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.901 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.902 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.903 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "orchestrator": { 2024-05-09T08:48:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "orchestrator", 2024-05-09T08:48:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.904 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm", 2024-05-09T08:48:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: "rook", 2024-05-09T08:48:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: "test_orchestrator" 2024-05-09T08:48:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Orchestrator backend", 2024-05-09T08:48:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.905 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "osd_perf_query", 2024-05-09T08:48:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.906 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.908 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.909 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.910 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.910 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.910 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.910 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.910 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.910 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.910 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.910 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.910 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.910 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.910 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.910 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.910 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "osd_support", 2024-05-09T08:48:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.911 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.912 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.913 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.913 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.913 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.913 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.913 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.913 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.913 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.913 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.913 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.913 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.913 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.913 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.913 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.914 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.915 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.916 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pg_autoscaler", 2024-05-09T08:48:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.918 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.919 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.920 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.921 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-05-09T08:48:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-05-09T08:48:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-05-09T08:48:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.922 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.923 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.923 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.923 INFO:teuthology.orchestra.run.smithi107.stdout: "threshold": { 2024-05-09T08:48:47.923 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "threshold", 2024-05-09T08:48:47.923 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:47.923 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.923 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.923 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3.0", 2024-05-09T08:48:47.923 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1.0", 2024-05-09T08:48:47.923 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.923 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.923 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "scaling threshold", 2024-05-09T08:48:47.923 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-09T08:48:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "progress", 2024-05-09T08:48:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_pg_recovery_event": { 2024-05-09T08:48:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_pg_recovery_event", 2024-05-09T08:48:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.924 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-09T08:48:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "enabled": { 2024-05-09T08:48:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enabled", 2024-05-09T08:48:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.925 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.926 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.927 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.928 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.929 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "max_completed_events": { 2024-05-09T08:48:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_completed_events", 2024-05-09T08:48:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "50", 2024-05-09T08:48:47.931 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.932 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.932 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.932 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "number of past completed events to remember", 2024-05-09T08:48:47.932 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.932 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.932 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.932 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-05-09T08:48:47.932 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-05-09T08:48:47.932 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:47.932 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.932 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.932 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-05-09T08:48:47.932 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long the module is going to sleep", 2024-05-09T08:48:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prometheus", 2024-05-09T08:48:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.933 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "cache": { 2024-05-09T08:48:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cache", 2024-05-09T08:48:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.934 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: "exclude_perf_counters": { 2024-05-09T08:48:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "exclude_perf_counters", 2024-05-09T08:48:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-09T08:48:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-09T08:48:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.935 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.936 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.937 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.938 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.939 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_stats_pools": { 2024-05-09T08:48:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_stats_pools", 2024-05-09T08:48:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.941 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-09T08:48:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-09T08:48:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "300", 2024-05-09T08:48:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.942 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: "scrape_interval": { 2024-05-09T08:48:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "scrape_interval", 2024-05-09T08:48:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15.0", 2024-05-09T08:48:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.943 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-05-09T08:48:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-05-09T08:48:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "::", 2024-05-09T08:48:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-09T08:48:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.944 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-05-09T08:48:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-05-09T08:48:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "9283", 2024-05-09T08:48:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-09T08:48:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: "stale_cache_strategy": { 2024-05-09T08:48:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "stale_cache_strategy", 2024-05-09T08:48:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "log", 2024-05-09T08:48:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.946 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_behaviour": { 2024-05-09T08:48:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_behaviour", 2024-05-09T08:48:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "default", 2024-05-09T08:48:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: "default", 2024-05-09T08:48:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: "error" 2024-05-09T08:48:47.947 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_error_status_code": { 2024-05-09T08:48:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_error_status_code", 2024-05-09T08:48:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "500", 2024-05-09T08:48:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "400", 2024-05-09T08:48:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "599", 2024-05-09T08:48:47.948 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_support", 2024-05-09T08:48:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.949 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.950 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.950 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.950 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.950 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.950 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.950 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.950 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.950 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.950 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.950 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.950 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.950 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.950 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.950 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.951 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.951 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.951 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.951 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.951 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.951 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.951 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.951 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.951 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.951 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.951 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.951 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.951 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.952 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.952 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.952 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.952 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.952 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.952 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.952 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.952 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.952 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.952 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.952 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.952 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.953 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.953 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.953 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.953 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.953 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.953 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.953 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.953 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.953 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.953 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.953 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.953 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.953 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.953 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.954 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.954 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.954 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.954 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.954 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.954 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.954 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.954 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.954 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.954 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.954 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.954 INFO:teuthology.orchestra.run.smithi107.stdout: "max_concurrent_snap_create": { 2024-05-09T08:48:47.954 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_concurrent_snap_create", 2024-05-09T08:48:47.955 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.955 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.955 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.955 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:47.955 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.955 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.955 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.955 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.955 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.955 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.955 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.955 INFO:teuthology.orchestra.run.smithi107.stdout: "mirror_snapshot_schedule": { 2024-05-09T08:48:47.955 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mirror_snapshot_schedule", 2024-05-09T08:48:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: "trash_purge_schedule": { 2024-05-09T08:48:47.956 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "trash_purge_schedule", 2024-05-09T08:48:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "restful", 2024-05-09T08:48:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "enable_auth": { 2024-05-09T08:48:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enable_auth", 2024-05-09T08:48:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.958 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "key_file": { 2024-05-09T08:48:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "key_file", 2024-05-09T08:48:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.959 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.960 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.963 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.964 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-05-09T08:48:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-05-09T08:48:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.965 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-05-09T08:48:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-05-09T08:48:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.966 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.967 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.967 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.967 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.967 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.967 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.967 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.967 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.967 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.967 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.967 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.967 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rgw", 2024-05-09T08:48:47.967 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.968 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.969 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.970 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.971 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.972 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rook", 2024-05-09T08:48:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.973 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.975 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.976 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.977 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: "storage_class": { 2024-05-09T08:48:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "storage_class", 2024-05-09T08:48:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "local", 2024-05-09T08:48:47.978 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-09T08:48:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "selftest", 2024-05-09T08:48:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.979 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.980 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.981 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.982 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:47.983 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "roption1": { 2024-05-09T08:48:47.984 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "roption1", 2024-05-09T08:48:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "roption2": { 2024-05-09T08:48:47.985 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "roption2", 2024-05-09T08:48:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "xyz", 2024-05-09T08:48:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.986 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption1": { 2024-05-09T08:48:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption1", 2024-05-09T08:48:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption2": { 2024-05-09T08:48:47.987 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption2", 2024-05-09T08:48:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption3": { 2024-05-09T08:48:47.988 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption3", 2024-05-09T08:48:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.989 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption4": { 2024-05-09T08:48:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption4", 2024-05-09T08:48:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption5": { 2024-05-09T08:48:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption5", 2024-05-09T08:48:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption6": { 2024-05-09T08:48:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption6", 2024-05-09T08:48:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption7": { 2024-05-09T08:48:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption7", 2024-05-09T08:48:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-05-09T08:48:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "42", 2024-05-09T08:48:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "testkey": { 2024-05-09T08:48:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testkey", 2024-05-09T08:48:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.994 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: "testlkey": { 2024-05-09T08:48:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testlkey", 2024-05-09T08:48:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.995 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: "testnewline": { 2024-05-09T08:48:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testnewline", 2024-05-09T08:48:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.996 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snap_schedule", 2024-05-09T08:48:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_m_granularity": { 2024-05-09T08:48:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_m_granularity", 2024-05-09T08:48:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.997 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow minute scheduled snapshots", 2024-05-09T08:48:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: "dump_on_update": { 2024-05-09T08:48:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "dump_on_update", 2024-05-09T08:48:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.998 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "dump database to debug log on update", 2024-05-09T08:48:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:47.999 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.000 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.001 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.002 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.002 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.002 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.002 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.002 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.002 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.002 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.002 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.002 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.002 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.002 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.002 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.002 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.002 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.003 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "stats", 2024-05-09T08:48:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.004 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.005 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.006 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.007 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.009 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "status", 2024-05-09T08:48:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.010 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.011 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.013 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.014 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "telegraf", 2024-05-09T08:48:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.015 INFO:teuthology.orchestra.run.smithi107.stdout: "address": { 2024-05-09T08:48:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "address", 2024-05-09T08:48:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-09T08:48:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.016 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-05-09T08:48:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-05-09T08:48:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15", 2024-05-09T08:48:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.018 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.019 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.020 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.021 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.022 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "telemetry", 2024-05-09T08:48:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_basic": { 2024-05-09T08:48:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_basic", 2024-05-09T08:48:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.023 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-09T08:48:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_crash": { 2024-05-09T08:48:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_crash", 2024-05-09T08:48:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-09T08:48:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_device": { 2024-05-09T08:48:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_device", 2024-05-09T08:48:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.025 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.026 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-09T08:48:48.026 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.026 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.026 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.026 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.026 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_ident": { 2024-05-09T08:48:48.026 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_ident", 2024-05-09T08:48:48.026 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.026 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.026 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.026 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.026 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.026 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.027 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.027 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-09T08:48:48.027 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.027 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.027 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.027 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_perf": { 2024-05-09T08:48:48.027 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_perf", 2024-05-09T08:48:48.027 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.027 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.027 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.027 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.027 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.027 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-09T08:48:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: "contact": { 2024-05-09T08:48:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "contact", 2024-05-09T08:48:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "description": { 2024-05-09T08:48:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "description", 2024-05-09T08:48:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.029 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "device_url": { 2024-05-09T08:48:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_url", 2024-05-09T08:48:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-09T08:48:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.030 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "enabled": { 2024-05-09T08:48:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enabled", 2024-05-09T08:48:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.031 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-05-09T08:48:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-05-09T08:48:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "24", 2024-05-09T08:48:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "8", 2024-05-09T08:48:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.032 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "last_opt_revision": { 2024-05-09T08:48:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "last_opt_revision", 2024-05-09T08:48:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1", 2024-05-09T08:48:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.033 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: "leaderboard": { 2024-05-09T08:48:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "leaderboard", 2024-05-09T08:48:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.034 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: "leaderboard_description": { 2024-05-09T08:48:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "leaderboard_description", 2024-05-09T08:48:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.035 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.036 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.037 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.038 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.039 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.040 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.040 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.040 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.040 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.040 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.040 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.040 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.040 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.040 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.040 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.040 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.041 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.041 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.041 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.041 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.041 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.041 INFO:teuthology.orchestra.run.smithi107.stdout: "organization": { 2024-05-09T08:48:48.041 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "organization", 2024-05-09T08:48:48.041 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.041 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.041 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.041 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.042 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.042 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.042 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.042 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.042 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.042 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.042 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.042 INFO:teuthology.orchestra.run.smithi107.stdout: "proxy": { 2024-05-09T08:48:48.042 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "proxy", 2024-05-09T08:48:48.042 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.042 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.042 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.042 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.043 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.043 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.043 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.043 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.043 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.043 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.043 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.043 INFO:teuthology.orchestra.run.smithi107.stdout: "url": { 2024-05-09T08:48:48.043 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "url", 2024-05-09T08:48:48.043 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.043 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.043 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.043 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-09T08:48:48.044 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.044 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.044 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.044 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.044 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.044 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.044 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.044 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.044 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.044 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.044 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "test_orchestrator", 2024-05-09T08:48:48.044 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.045 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.045 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.045 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.045 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.045 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.045 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.045 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.045 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.045 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.045 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.045 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.045 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.045 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.046 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.046 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.046 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.046 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.046 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.046 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.046 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.046 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.046 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.046 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.046 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.046 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.046 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.047 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.047 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.047 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.047 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.047 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.047 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.047 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.047 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.047 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.047 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.047 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.047 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.047 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.048 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.048 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.048 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.048 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.048 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.048 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.048 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.048 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.048 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.048 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.048 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.048 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.048 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.048 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.048 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.049 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.049 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.049 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.049 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.049 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.049 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.049 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.049 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.049 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.049 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.049 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.049 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.049 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.050 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.050 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.050 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.050 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "volumes", 2024-05-09T08:48:48.050 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.050 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.050 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.050 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.050 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.050 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.050 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.050 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.051 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.051 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.051 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.051 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.051 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.051 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.051 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.051 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.051 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.051 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.051 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.051 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.051 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.051 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.051 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.052 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.052 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.052 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.052 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.052 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.052 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.052 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.052 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.052 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.052 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.052 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.052 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.052 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.053 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.053 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.053 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.053 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.053 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.053 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.053 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.053 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.053 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.053 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.053 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.053 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.053 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.053 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.054 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.054 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.054 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.054 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.054 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.054 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.054 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.054 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.054 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.054 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.054 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.054 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.054 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.055 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.055 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.055 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.055 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.055 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.055 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.055 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.055 INFO:teuthology.orchestra.run.smithi107.stdout: "max_concurrent_clones": { 2024-05-09T08:48:48.055 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_concurrent_clones", 2024-05-09T08:48:48.055 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.055 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.055 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.055 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "4", 2024-05-09T08:48:48.056 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.056 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.056 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.056 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-09T08:48:48.056 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.056 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.056 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.056 INFO:teuthology.orchestra.run.smithi107.stdout: "periodic_async_work": { 2024-05-09T08:48:48.056 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "periodic_async_work", 2024-05-09T08:48:48.056 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.056 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.056 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.056 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.057 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.057 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.057 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.057 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Periodically check for async work", 2024-05-09T08:48:48.057 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.057 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.057 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.057 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot_clone_delay": { 2024-05-09T08:48:48.057 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snapshot_clone_delay", 2024-05-09T08:48:48.057 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.057 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.057 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.057 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-05-09T08:48:48.057 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.058 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.058 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.058 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-09T08:48:48.058 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.058 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.058 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.058 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot_clone_no_wait": { 2024-05-09T08:48:48.058 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snapshot_clone_no_wait", 2024-05-09T08:48:48.058 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.058 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.058 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.058 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.059 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.059 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.059 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.059 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-09T08:48:48.059 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.059 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.059 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.059 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.059 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.059 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix", 2024-05-09T08:48:48.059 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.060 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.060 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.060 INFO:teuthology.orchestra.run.smithi107.stdout: "discovery_interval": { 2024-05-09T08:48:48.060 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "discovery_interval", 2024-05-09T08:48:48.060 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-05-09T08:48:48.060 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.060 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.060 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "100", 2024-05-09T08:48:48.060 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.060 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.060 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.060 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.060 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.060 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.060 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.061 INFO:teuthology.orchestra.run.smithi107.stdout: "identifier": { 2024-05-09T08:48:48.061 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "identifier", 2024-05-09T08:48:48.061 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.061 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.061 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.061 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.061 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.061 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.061 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.061 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.061 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.061 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.061 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.062 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-05-09T08:48:48.062 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-05-09T08:48:48.062 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.062 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.062 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.062 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-05-09T08:48:48.062 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.062 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.062 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.062 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.062 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.062 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.063 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.063 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.063 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.063 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.063 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.063 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.063 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.063 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.063 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.063 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.063 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.063 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.063 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.063 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.064 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.064 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.064 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.064 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.064 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.064 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.064 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.064 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.064 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.064 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.064 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.064 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.064 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.065 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.065 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.065 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.065 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.065 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.065 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.065 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.065 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.065 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.065 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.065 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.065 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.065 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.065 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.066 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.066 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.066 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.066 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.066 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.066 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.066 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.066 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.066 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.066 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.066 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.066 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.066 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.067 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.067 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.067 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.067 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.067 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.067 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.067 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.067 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.067 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.067 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.067 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.067 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.067 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.068 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_host": { 2024-05-09T08:48:48.068 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_host", 2024-05-09T08:48:48.068 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.068 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.068 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.068 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.068 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.068 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.068 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.068 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.068 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.068 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.068 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.068 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_port": { 2024-05-09T08:48:48.069 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_port", 2024-05-09T08:48:48.069 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.069 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.069 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.069 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10051", 2024-05-09T08:48:48.069 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.069 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.069 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.069 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.069 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.069 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.069 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.069 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_sender": { 2024-05-09T08:48:48.070 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_sender", 2024-05-09T08:48:48.070 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.070 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.070 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.070 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-09T08:48:48.070 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.070 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.070 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.070 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.070 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.070 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.070 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.070 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.070 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.071 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.071 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-05-09T08:48:48.071 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.071 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.071 INFO:teuthology.orchestra.run.smithi107.stdout: "modules": [ 2024-05-09T08:48:48.071 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm", 2024-05-09T08:48:48.071 INFO:teuthology.orchestra.run.smithi107.stdout: "dashboard", 2024-05-09T08:48:48.071 INFO:teuthology.orchestra.run.smithi107.stdout: "iostat", 2024-05-09T08:48:48.071 INFO:teuthology.orchestra.run.smithi107.stdout: "nfs", 2024-05-09T08:48:48.071 INFO:teuthology.orchestra.run.smithi107.stdout: "restful", 2024-05-09T08:48:48.071 INFO:teuthology.orchestra.run.smithi107.stdout: "snap_schedule" 2024-05-09T08:48:48.071 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.071 INFO:teuthology.orchestra.run.smithi107.stdout: "available_modules": [ 2024-05-09T08:48:48.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.071 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "alerts", 2024-05-09T08:48:48.072 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.072 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.072 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.072 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-05-09T08:48:48.072 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-05-09T08:48:48.072 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.072 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.072 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.072 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-05-09T08:48:48.072 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.072 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.072 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.072 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How frequently to reexamine health status", 2024-05-09T08:48:48.073 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.073 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.073 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.073 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.073 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.073 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.073 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.073 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.073 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.073 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.073 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.073 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.073 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.074 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.074 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.074 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.074 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.074 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.074 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.074 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.074 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.074 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.074 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.074 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.074 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.074 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.075 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.076 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.076 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.076 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.076 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.076 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.076 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.076 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.076 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.076 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.076 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.076 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.076 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.076 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.077 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.077 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.077 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.077 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.077 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.077 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.077 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.077 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.077 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.077 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.077 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.077 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.077 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_destination": { 2024-05-09T08:48:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_destination", 2024-05-09T08:48:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.078 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.079 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Email address to send alerts to", 2024-05-09T08:48:48.079 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.079 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.079 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.079 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_from_name": { 2024-05-09T08:48:48.079 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_from_name", 2024-05-09T08:48:48.079 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.079 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.079 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.079 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "Ceph", 2024-05-09T08:48:48.079 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.079 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.079 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.080 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Email From: name", 2024-05-09T08:48:48.080 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.080 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.080 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.080 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_host": { 2024-05-09T08:48:48.080 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_host", 2024-05-09T08:48:48.080 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.080 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.080 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.080 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.080 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.080 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.080 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.081 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP server", 2024-05-09T08:48:48.081 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.081 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.081 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.081 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_password": { 2024-05-09T08:48:48.081 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_password", 2024-05-09T08:48:48.081 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.081 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.081 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.081 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.081 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.081 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.081 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.082 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Password to authenticate with", 2024-05-09T08:48:48.082 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.082 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.082 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.082 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_port": { 2024-05-09T08:48:48.082 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_port", 2024-05-09T08:48:48.082 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.082 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.082 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.082 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "465", 2024-05-09T08:48:48.082 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.082 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.082 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.083 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP port", 2024-05-09T08:48:48.083 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.083 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.083 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.083 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_sender": { 2024-05-09T08:48:48.083 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_sender", 2024-05-09T08:48:48.083 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.083 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.083 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.083 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.083 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.083 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.083 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.084 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SMTP envelope sender", 2024-05-09T08:48:48.084 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.084 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.084 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.084 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_ssl": { 2024-05-09T08:48:48.084 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_ssl", 2024-05-09T08:48:48.084 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.084 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.084 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.084 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.084 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.084 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.084 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.085 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-09T08:48:48.085 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.085 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.085 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.085 INFO:teuthology.orchestra.run.smithi107.stdout: "smtp_user": { 2024-05-09T08:48:48.085 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "smtp_user", 2024-05-09T08:48:48.085 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.085 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.085 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.085 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.085 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.085 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.085 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.086 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "User to authenticate as", 2024-05-09T08:48:48.086 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.086 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.086 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.086 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.086 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.086 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.086 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.086 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "balancer", 2024-05-09T08:48:48.086 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.086 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.086 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.086 INFO:teuthology.orchestra.run.smithi107.stdout: "active": { 2024-05-09T08:48:48.086 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "active", 2024-05-09T08:48:48.087 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.087 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.087 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.087 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.087 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.087 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.087 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.087 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "automatically balance PGs across cluster", 2024-05-09T08:48:48.087 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.087 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.087 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.087 INFO:teuthology.orchestra.run.smithi107.stdout: "begin_time": { 2024-05-09T08:48:48.087 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "begin_time", 2024-05-09T08:48:48.088 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.088 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.088 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.088 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0000", 2024-05-09T08:48:48.088 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.088 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.088 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.088 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "beginning time of day to automatically balance", 2024-05-09T08:48:48.088 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T08:48:48.088 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.088 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.088 INFO:teuthology.orchestra.run.smithi107.stdout: "begin_weekday": { 2024-05-09T08:48:48.088 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "begin_weekday", 2024-05-09T08:48:48.088 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-05-09T08:48:48.089 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.089 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.089 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-05-09T08:48:48.089 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0", 2024-05-09T08:48:48.089 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "6", 2024-05-09T08:48:48.089 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.089 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-09T08:48:48.089 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T08:48:48.089 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.089 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.089 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_max_iterations": { 2024-05-09T08:48:48.089 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_max_iterations", 2024-05-09T08:48:48.089 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-05-09T08:48:48.089 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.090 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.090 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "25", 2024-05-09T08:48:48.090 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-05-09T08:48:48.090 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "250", 2024-05-09T08:48:48.090 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.090 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-09T08:48:48.090 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.090 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.090 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.090 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_metrics": { 2024-05-09T08:48:48.090 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_metrics", 2024-05-09T08:48:48.090 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.091 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.091 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.091 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "pgs,objects,bytes", 2024-05-09T08:48:48.091 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.091 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.091 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.091 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-09T08:48:48.091 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-09T08:48:48.091 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.091 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.091 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_compat_step": { 2024-05-09T08:48:48.091 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crush_compat_step", 2024-05-09T08:48:48.091 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:48.092 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.092 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.092 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0.5", 2024-05-09T08:48:48.092 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0.001", 2024-05-09T08:48:48.092 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "0.999", 2024-05-09T08:48:48.092 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.092 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "aggressiveness of optimization", 2024-05-09T08:48:48.092 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-09T08:48:48.092 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.092 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.092 INFO:teuthology.orchestra.run.smithi107.stdout: "end_time": { 2024-05-09T08:48:48.092 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "end_time", 2024-05-09T08:48:48.092 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.093 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.093 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.093 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "2359", 2024-05-09T08:48:48.093 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.093 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.093 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.093 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "ending time of day to automatically balance", 2024-05-09T08:48:48.093 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T08:48:48.093 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.093 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.093 INFO:teuthology.orchestra.run.smithi107.stdout: "end_weekday": { 2024-05-09T08:48:48.093 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "end_weekday", 2024-05-09T08:48:48.093 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-05-09T08:48:48.093 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.094 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.094 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-05-09T08:48:48.094 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "0", 2024-05-09T08:48:48.094 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "6", 2024-05-09T08:48:48.094 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.094 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-09T08:48:48.094 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T08:48:48.094 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.094 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.094 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.094 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.094 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.094 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.094 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.095 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.095 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.095 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.095 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.095 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.095 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.095 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.095 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.095 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.095 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.095 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.095 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.095 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.095 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.096 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.097 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.098 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.098 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.098 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.098 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.098 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.098 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.098 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.098 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.098 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.098 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.098 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.098 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.098 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.099 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.099 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.099 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.099 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.099 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.099 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.099 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.099 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.099 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.099 INFO:teuthology.orchestra.run.smithi107.stdout: "min_score": { 2024-05-09T08:48:48.099 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "min_score", 2024-05-09T08:48:48.099 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:48.099 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.099 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.100 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-05-09T08:48:48.100 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.100 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.100 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.100 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-09T08:48:48.100 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.100 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.100 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.100 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.100 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": { 2024-05-09T08:48:48.100 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mode", 2024-05-09T08:48:48.100 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.100 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.100 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.100 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "upmap", 2024-05-09T08:48:48.101 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.101 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.101 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.101 INFO:teuthology.orchestra.run.smithi107.stdout: "crush-compat", 2024-05-09T08:48:48.101 INFO:teuthology.orchestra.run.smithi107.stdout: "none", 2024-05-09T08:48:48.101 INFO:teuthology.orchestra.run.smithi107.stdout: "read", 2024-05-09T08:48:48.101 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap", 2024-05-09T08:48:48.101 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap-read" 2024-05-09T08:48:48.101 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.101 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Balancer mode", 2024-05-09T08:48:48.101 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.101 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.101 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.102 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_ids": { 2024-05-09T08:48:48.102 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pool_ids", 2024-05-09T08:48:48.102 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.102 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.102 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.102 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.102 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.102 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.102 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.102 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-09T08:48:48.102 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.102 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.102 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.102 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.103 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-05-09T08:48:48.103 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-05-09T08:48:48.103 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.103 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.103 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.103 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-05-09T08:48:48.103 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.103 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.103 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.103 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-09T08:48:48.103 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.103 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.103 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.103 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.103 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap_max_deviation": { 2024-05-09T08:48:48.104 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "upmap_max_deviation", 2024-05-09T08:48:48.104 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.104 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.104 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.104 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-05-09T08:48:48.104 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-05-09T08:48:48.104 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.104 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.104 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-09T08:48:48.104 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-09T08:48:48.104 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.104 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.104 INFO:teuthology.orchestra.run.smithi107.stdout: "upmap_max_optimizations": { 2024-05-09T08:48:48.105 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "upmap_max_optimizations", 2024-05-09T08:48:48.105 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-05-09T08:48:48.105 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.105 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.105 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:48.105 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.105 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.105 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.105 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-09T08:48:48.105 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.105 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.105 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.105 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.105 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.105 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.106 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.106 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephadm", 2024-05-09T08:48:48.106 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.106 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.106 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.106 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_down_multiplier": { 2024-05-09T08:48:48.106 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_down_multiplier", 2024-05-09T08:48:48.106 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:48.106 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.106 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.106 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3.0", 2024-05-09T08:48:48.106 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.106 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.106 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.107 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-09T08:48:48.107 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.107 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.107 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.107 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.107 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_refresh_rate": { 2024-05-09T08:48:48.107 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_refresh_rate", 2024-05-09T08:48:48.107 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.107 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.107 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.107 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "20", 2024-05-09T08:48:48.107 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.107 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.107 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.108 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-09T08:48:48.108 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.108 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.108 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.108 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.108 INFO:teuthology.orchestra.run.smithi107.stdout: "agent_starting_port": { 2024-05-09T08:48:48.108 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "agent_starting_port", 2024-05-09T08:48:48.108 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.108 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.108 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.108 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "4721", 2024-05-09T08:48:48.108 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.108 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.108 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.109 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-09T08:48:48.109 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.109 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.109 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.109 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_ptrace": { 2024-05-09T08:48:48.109 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_ptrace", 2024-05-09T08:48:48.109 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.109 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.109 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.109 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.109 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.109 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.109 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.110 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-09T08:48:48.110 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-09T08:48:48.110 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.110 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.110 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.110 INFO:teuthology.orchestra.run.smithi107.stdout: "autotune_interval": { 2024-05-09T08:48:48.110 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "autotune_interval", 2024-05-09T08:48:48.110 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.110 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.110 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.110 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-05-09T08:48:48.110 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.110 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.110 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.111 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-09T08:48:48.111 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.111 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.111 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.111 INFO:teuthology.orchestra.run.smithi107.stdout: "autotune_memory_target_ratio": { 2024-05-09T08:48:48.111 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "autotune_memory_target_ratio", 2024-05-09T08:48:48.111 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:48.111 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.111 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.111 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0.7", 2024-05-09T08:48:48.111 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.111 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.111 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.112 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-09T08:48:48.112 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.112 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.112 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.112 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm_log_destination": { 2024-05-09T08:48:48.112 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cephadm_log_destination", 2024-05-09T08:48:48.112 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.112 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.112 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.112 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.112 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.112 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.112 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.112 INFO:teuthology.orchestra.run.smithi107.stdout: "file", 2024-05-09T08:48:48.113 INFO:teuthology.orchestra.run.smithi107.stdout: "file,syslog", 2024-05-09T08:48:48.113 INFO:teuthology.orchestra.run.smithi107.stdout: "syslog" 2024-05-09T08:48:48.113 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.113 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-09T08:48:48.113 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.113 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.113 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.113 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.113 INFO:teuthology.orchestra.run.smithi107.stdout: "cgroups_split": { 2024-05-09T08:48:48.113 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cgroups_split", 2024-05-09T08:48:48.113 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.113 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.113 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.113 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.114 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.114 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.114 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.114 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-09T08:48:48.114 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.114 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.114 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.114 INFO:teuthology.orchestra.run.smithi107.stdout: "config_checks_enabled": { 2024-05-09T08:48:48.114 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_checks_enabled", 2024-05-09T08:48:48.114 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.114 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.114 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.114 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.115 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.115 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.115 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.115 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-09T08:48:48.115 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.115 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.115 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.115 INFO:teuthology.orchestra.run.smithi107.stdout: "config_dashboard": { 2024-05-09T08:48:48.115 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_dashboard", 2024-05-09T08:48:48.115 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.115 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.115 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.115 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.116 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.116 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.116 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.116 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-09T08:48:48.116 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.116 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.116 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.116 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_alertmanager": { 2024-05-09T08:48:48.116 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_alertmanager", 2024-05-09T08:48:48.116 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.116 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.116 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.116 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-09T08:48:48.117 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.117 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.117 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.117 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-05-09T08:48:48.117 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.117 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.117 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.117 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_base": { 2024-05-09T08:48:48.117 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_base", 2024-05-09T08:48:48.117 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.117 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.117 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.117 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-09T08:48:48.117 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.118 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.118 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.118 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Container image name, without the tag", 2024-05-09T08:48:48.118 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.118 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.118 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.118 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_elasticsearch": { 2024-05-09T08:48:48.118 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_elasticsearch", 2024-05-09T08:48:48.118 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.118 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.118 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.118 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-09T08:48:48.119 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.119 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.119 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.119 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "elasticsearch container image", 2024-05-09T08:48:48.119 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.119 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.119 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.119 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_grafana": { 2024-05-09T08:48:48.119 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_grafana", 2024-05-09T08:48:48.119 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.119 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.119 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.119 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-09T08:48:48.120 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.120 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.120 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.120 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-05-09T08:48:48.120 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.120 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.120 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.120 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_haproxy": { 2024-05-09T08:48:48.120 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_haproxy", 2024-05-09T08:48:48.120 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.120 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.120 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.120 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-09T08:48:48.121 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.121 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.121 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.121 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "HAproxy container image", 2024-05-09T08:48:48.121 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.121 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.121 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.121 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_agent": { 2024-05-09T08:48:48.121 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_agent", 2024-05-09T08:48:48.121 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.121 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.121 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.121 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-09T08:48:48.122 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.122 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.122 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.122 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger agent container image", 2024-05-09T08:48:48.122 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.122 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.122 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.122 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_collector": { 2024-05-09T08:48:48.122 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_collector", 2024-05-09T08:48:48.122 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.122 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.122 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.122 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-09T08:48:48.123 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.123 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.123 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.123 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger collector container image", 2024-05-09T08:48:48.123 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.123 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.123 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.123 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_jaeger_query": { 2024-05-09T08:48:48.123 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_jaeger_query", 2024-05-09T08:48:48.123 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.123 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.123 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.123 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-09T08:48:48.124 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.124 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.124 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.124 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Jaeger query container image", 2024-05-09T08:48:48.124 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.124 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.124 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.124 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_keepalived": { 2024-05-09T08:48:48.124 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_keepalived", 2024-05-09T08:48:48.124 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.124 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.124 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.125 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-09T08:48:48.125 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.125 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.125 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.125 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Keepalived container image", 2024-05-09T08:48:48.125 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.125 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.125 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.125 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_loki": { 2024-05-09T08:48:48.125 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_loki", 2024-05-09T08:48:48.125 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.125 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.125 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.125 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-09T08:48:48.126 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.126 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.126 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.126 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Loki container image", 2024-05-09T08:48:48.126 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.126 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.126 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.126 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_node_exporter": { 2024-05-09T08:48:48.126 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_node_exporter", 2024-05-09T08:48:48.126 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.126 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.126 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.126 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-09T08:48:48.127 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.127 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.127 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.127 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-05-09T08:48:48.127 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.127 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.127 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.127 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_nvmeof": { 2024-05-09T08:48:48.127 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_nvmeof", 2024-05-09T08:48:48.127 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.127 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.127 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.127 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-09T08:48:48.128 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.128 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.128 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.128 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Nvme-of container image", 2024-05-09T08:48:48.128 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.128 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.128 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.128 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_prometheus": { 2024-05-09T08:48:48.128 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_prometheus", 2024-05-09T08:48:48.128 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.128 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.128 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.128 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-09T08:48:48.129 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.129 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.129 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.129 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Prometheus container image", 2024-05-09T08:48:48.129 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.129 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.129 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.129 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_promtail": { 2024-05-09T08:48:48.129 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_promtail", 2024-05-09T08:48:48.129 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.129 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.129 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.129 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-09T08:48:48.130 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.130 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.130 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.130 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Promtail container image", 2024-05-09T08:48:48.130 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.130 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.130 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.130 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_samba": { 2024-05-09T08:48:48.130 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_samba", 2024-05-09T08:48:48.130 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.130 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.130 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.130 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-09T08:48:48.131 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.131 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.131 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.131 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Samba/SMB container image", 2024-05-09T08:48:48.131 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.131 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.131 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.131 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image_snmp_gateway": { 2024-05-09T08:48:48.131 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_image_snmp_gateway", 2024-05-09T08:48:48.131 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.131 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.131 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.131 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-09T08:48:48.131 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.132 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.132 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.132 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "SNMP Gateway container image", 2024-05-09T08:48:48.132 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.132 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.132 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.132 INFO:teuthology.orchestra.run.smithi107.stdout: "container_init": { 2024-05-09T08:48:48.132 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "container_init", 2024-05-09T08:48:48.132 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.132 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.132 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.132 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.132 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.133 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.133 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.133 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Run podman/docker with `--init`", 2024-05-09T08:48:48.133 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.133 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.133 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.133 INFO:teuthology.orchestra.run.smithi107.stdout: "daemon_cache_timeout": { 2024-05-09T08:48:48.133 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "daemon_cache_timeout", 2024-05-09T08:48:48.133 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.133 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.133 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.133 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-05-09T08:48:48.133 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.134 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.134 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.134 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-09T08:48:48.134 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.134 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.134 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.134 INFO:teuthology.orchestra.run.smithi107.stdout: "default_cephadm_command_timeout": { 2024-05-09T08:48:48.134 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "default_cephadm_command_timeout", 2024-05-09T08:48:48.134 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.134 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.134 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.134 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "900", 2024-05-09T08:48:48.135 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.135 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.135 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.135 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-09T08:48:48.135 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.135 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.135 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.135 INFO:teuthology.orchestra.run.smithi107.stdout: "default_registry": { 2024-05-09T08:48:48.135 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "default_registry", 2024-05-09T08:48:48.135 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.135 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.136 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.136 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "docker.io", 2024-05-09T08:48:48.136 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.136 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.136 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.136 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-09T08:48:48.136 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.136 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.136 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.136 INFO:teuthology.orchestra.run.smithi107.stdout: "device_cache_timeout": { 2024-05-09T08:48:48.136 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_cache_timeout", 2024-05-09T08:48:48.136 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.136 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.137 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.137 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1800", 2024-05-09T08:48:48.137 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.137 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.137 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.137 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache device inventory", 2024-05-09T08:48:48.137 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.137 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.137 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.137 INFO:teuthology.orchestra.run.smithi107.stdout: "device_enhanced_scan": { 2024-05-09T08:48:48.137 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_enhanced_scan", 2024-05-09T08:48:48.137 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.137 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.138 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.138 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.138 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.138 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.138 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.138 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-09T08:48:48.138 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.138 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.138 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.138 INFO:teuthology.orchestra.run.smithi107.stdout: "facts_cache_timeout": { 2024-05-09T08:48:48.138 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "facts_cache_timeout", 2024-05-09T08:48:48.138 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.138 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.139 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.139 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-05-09T08:48:48.139 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.139 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.139 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.139 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "seconds to cache host facts data", 2024-05-09T08:48:48.139 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.139 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.139 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.139 INFO:teuthology.orchestra.run.smithi107.stdout: "grafana_dashboards_path": { 2024-05-09T08:48:48.139 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "grafana_dashboards_path", 2024-05-09T08:48:48.139 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.140 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.140 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.140 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-09T08:48:48.140 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.140 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.140 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.140 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-09T08:48:48.140 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.140 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.140 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.140 INFO:teuthology.orchestra.run.smithi107.stdout: "host_check_interval": { 2024-05-09T08:48:48.140 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "host_check_interval", 2024-05-09T08:48:48.140 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.140 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.141 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.141 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-05-09T08:48:48.141 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.141 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.141 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.141 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to perform a host check", 2024-05-09T08:48:48.141 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.141 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.141 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.141 INFO:teuthology.orchestra.run.smithi107.stdout: "hw_monitoring": { 2024-05-09T08:48:48.141 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "hw_monitoring", 2024-05-09T08:48:48.141 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.141 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.141 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.142 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.142 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.142 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.142 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.142 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-09T08:48:48.142 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.142 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.142 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.142 INFO:teuthology.orchestra.run.smithi107.stdout: "inventory_list_all": { 2024-05-09T08:48:48.142 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "inventory_list_all", 2024-05-09T08:48:48.142 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.142 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.142 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.143 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.143 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.143 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.143 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.143 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-09T08:48:48.143 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.143 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.143 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.143 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.143 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.143 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.143 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.143 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.143 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.144 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.144 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.144 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.144 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.144 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.144 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.144 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.144 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.144 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.144 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.144 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.144 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.144 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.144 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.145 INFO:teuthology.orchestra.run.smithi107.stdout: "log_refresh_metadata": { 2024-05-09T08:48:48.145 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_refresh_metadata", 2024-05-09T08:48:48.145 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.145 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.145 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.145 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.145 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.145 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.145 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.145 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-09T08:48:48.145 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.145 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.145 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.146 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.146 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.146 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.146 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.146 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.146 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.146 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.146 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.146 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.146 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-09T08:48:48.146 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.146 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.146 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.147 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.147 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.147 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.147 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.147 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.147 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.147 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.147 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.147 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.147 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.147 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.147 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.147 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.147 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.148 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.148 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.148 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.148 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.148 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.148 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.148 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.148 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.148 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.148 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.148 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.148 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.149 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.149 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.149 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.149 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.149 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.149 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.149 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.149 INFO:teuthology.orchestra.run.smithi107.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-09T08:48:48.149 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-09T08:48:48.149 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.149 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.149 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.149 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.150 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.150 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.150 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.150 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-09T08:48:48.150 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.150 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.150 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.150 INFO:teuthology.orchestra.run.smithi107.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-09T08:48:48.150 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-09T08:48:48.150 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.150 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.150 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.150 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "*", 2024-05-09T08:48:48.151 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.151 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.151 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.151 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-09T08:48:48.151 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.151 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.151 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.151 INFO:teuthology.orchestra.run.smithi107.stdout: "max_count_per_host": { 2024-05-09T08:48:48.151 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_count_per_host", 2024-05-09T08:48:48.151 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.151 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.151 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.152 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:48.152 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.152 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.152 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.152 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "max number of daemons per service per host", 2024-05-09T08:48:48.152 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.152 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.152 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.152 INFO:teuthology.orchestra.run.smithi107.stdout: "max_osd_draining_count": { 2024-05-09T08:48:48.152 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_osd_draining_count", 2024-05-09T08:48:48.152 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.152 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.152 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.152 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-09T08:48:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "migration_current": { 2024-05-09T08:48:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "migration_current", 2024-05-09T08:48:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.153 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.154 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.154 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.154 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.154 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.154 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "internal - do not modify", 2024-05-09T08:48:48.154 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.154 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.154 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.154 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": { 2024-05-09T08:48:48.154 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mode", 2024-05-09T08:48:48.154 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.154 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.154 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.155 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "root", 2024-05-09T08:48:48.155 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.155 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.155 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.155 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm-package", 2024-05-09T08:48:48.155 INFO:teuthology.orchestra.run.smithi107.stdout: "root" 2024-05-09T08:48:48.155 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.155 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "mode for remote execution of cephadm", 2024-05-09T08:48:48.155 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.155 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.155 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.155 INFO:teuthology.orchestra.run.smithi107.stdout: "oob_default_addr": { 2024-05-09T08:48:48.155 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "oob_default_addr", 2024-05-09T08:48:48.155 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.156 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.156 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.156 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "169.254.1.1", 2024-05-09T08:48:48.156 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.156 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.156 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.156 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-09T08:48:48.156 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.156 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.156 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.156 INFO:teuthology.orchestra.run.smithi107.stdout: "prometheus_alerts_path": { 2024-05-09T08:48:48.156 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prometheus_alerts_path", 2024-05-09T08:48:48.156 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.157 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.157 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.157 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-09T08:48:48.157 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.157 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.157 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.157 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-09T08:48:48.157 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.157 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.157 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.157 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_insecure": { 2024-05-09T08:48:48.157 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_insecure", 2024-05-09T08:48:48.158 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.158 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.158 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.158 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.158 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.158 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.158 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.158 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-09T08:48:48.158 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.158 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.158 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.158 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_password": { 2024-05-09T08:48:48.158 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_password", 2024-05-09T08:48:48.159 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.159 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.159 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.159 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.159 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.159 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.159 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.161 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-09T08:48:48.161 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.161 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.161 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.162 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_url": { 2024-05-09T08:48:48.162 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_url", 2024-05-09T08:48:48.162 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.162 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.162 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.162 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.162 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.162 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.162 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.162 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-09T08:48:48.162 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.162 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.162 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "registry_username": { 2024-05-09T08:48:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "registry_username", 2024-05-09T08:48:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-09T08:48:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.163 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.164 INFO:teuthology.orchestra.run.smithi107.stdout: "secure_monitoring_stack": { 2024-05-09T08:48:48.164 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "secure_monitoring_stack", 2024-05-09T08:48:48.164 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.164 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.164 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.164 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.164 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.164 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.164 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.164 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-09T08:48:48.164 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.164 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.164 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.165 INFO:teuthology.orchestra.run.smithi107.stdout: "service_discovery_port": { 2024-05-09T08:48:48.165 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "service_discovery_port", 2024-05-09T08:48:48.165 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.165 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.165 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.165 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8765", 2024-05-09T08:48:48.165 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.165 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.165 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.165 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "cephadm service discovery port", 2024-05-09T08:48:48.165 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.165 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.165 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.166 INFO:teuthology.orchestra.run.smithi107.stdout: "ssh_config_file": { 2024-05-09T08:48:48.166 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssh_config_file", 2024-05-09T08:48:48.166 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.166 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.166 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.166 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.166 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.166 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.166 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.166 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-09T08:48:48.166 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.166 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.167 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.167 INFO:teuthology.orchestra.run.smithi107.stdout: "use_agent": { 2024-05-09T08:48:48.167 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "use_agent", 2024-05-09T08:48:48.167 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.167 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.167 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.167 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.167 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.167 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.167 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.167 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-09T08:48:48.167 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.167 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.167 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.168 INFO:teuthology.orchestra.run.smithi107.stdout: "use_repo_digest": { 2024-05-09T08:48:48.168 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "use_repo_digest", 2024-05-09T08:48:48.168 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.168 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.168 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.168 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.168 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.168 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.168 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.168 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-09T08:48:48.168 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.168 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.168 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.169 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_failed_host_check": { 2024-05-09T08:48:48.169 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_failed_host_check", 2024-05-09T08:48:48.169 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.169 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.169 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.169 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.169 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.169 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.169 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.169 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if the host check fails", 2024-05-09T08:48:48.169 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.169 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.169 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.170 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_stray_daemons": { 2024-05-09T08:48:48.170 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_stray_daemons", 2024-05-09T08:48:48.170 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.170 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.170 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.170 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.170 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.170 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.170 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.170 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-09T08:48:48.170 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.170 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.170 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.171 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_on_stray_hosts": { 2024-05-09T08:48:48.171 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_on_stray_hosts", 2024-05-09T08:48:48.171 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.171 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.171 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.171 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.171 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.171 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.171 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.171 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-09T08:48:48.171 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.171 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.171 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.172 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.172 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.172 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crash", 2024-05-09T08:48:48.172 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.172 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.172 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.172 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.172 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.172 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.172 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.172 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.172 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.173 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.173 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.173 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.173 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.173 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.173 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.173 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.173 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.173 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.173 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.173 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.173 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.173 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.173 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.174 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.174 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.174 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.174 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.174 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.174 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.174 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.174 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.174 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.174 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.174 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.174 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.174 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.175 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.175 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.175 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.175 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.175 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.175 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.175 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.175 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.175 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.175 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.175 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.175 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.175 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.175 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.176 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.176 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.176 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.176 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.176 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.176 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.176 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.176 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.176 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.176 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.176 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.176 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.176 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.177 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.177 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.177 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.177 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.177 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.177 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.177 INFO:teuthology.orchestra.run.smithi107.stdout: "retain_interval": { 2024-05-09T08:48:48.177 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "retain_interval", 2024-05-09T08:48:48.177 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.177 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.177 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.177 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "31536000", 2024-05-09T08:48:48.177 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.177 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.178 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.178 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-09T08:48:48.178 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.178 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.178 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.178 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_recent_interval": { 2024-05-09T08:48:48.178 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_recent_interval", 2024-05-09T08:48:48.178 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.178 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.178 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.178 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1209600", 2024-05-09T08:48:48.178 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-09T08:48:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "dashboard", 2024-05-09T08:48:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.179 INFO:teuthology.orchestra.run.smithi107.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-09T08:48:48.180 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-09T08:48:48.180 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.180 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.180 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.180 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:48.180 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.180 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.180 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.180 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.180 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.180 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.180 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.181 INFO:teuthology.orchestra.run.smithi107.stdout: "ALERTMANAGER_API_HOST": { 2024-05-09T08:48:48.181 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-09T08:48:48.181 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.181 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.181 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.181 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.181 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.181 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.181 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.181 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.181 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.181 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.181 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.182 INFO:teuthology.orchestra.run.smithi107.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-09T08:48:48.182 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-09T08:48:48.182 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.182 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.182 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.182 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.182 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.182 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.182 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.182 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.182 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.182 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.182 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.183 INFO:teuthology.orchestra.run.smithi107.stdout: "AUDIT_API_ENABLED": { 2024-05-09T08:48:48.183 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "AUDIT_API_ENABLED", 2024-05-09T08:48:48.183 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.183 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.183 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.183 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.183 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.183 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.183 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.183 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.183 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.183 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.183 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.184 INFO:teuthology.orchestra.run.smithi107.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-09T08:48:48.184 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-09T08:48:48.184 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.184 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.184 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.184 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.184 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.184 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.184 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.184 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.184 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.184 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.184 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.184 INFO:teuthology.orchestra.run.smithi107.stdout: "ENABLE_BROWSABLE_API": { 2024-05-09T08:48:48.185 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-09T08:48:48.185 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.185 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.185 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.185 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.185 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.185 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.185 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.185 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.185 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.185 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.185 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.185 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-09T08:48:48.186 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-09T08:48:48.186 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.186 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.186 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.186 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.186 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.186 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.186 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.186 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.186 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.186 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.186 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.186 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-09T08:48:48.187 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-09T08:48:48.187 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.187 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.187 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.187 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.187 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.187 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.187 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.187 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.187 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.187 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.187 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.187 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-09T08:48:48.187 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-09T08:48:48.188 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.188 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.188 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.188 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.188 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.188 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.188 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.188 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.188 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.188 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.188 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.188 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-09T08:48:48.189 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-09T08:48:48.189 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.189 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.189 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.189 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.189 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.189 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.189 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.189 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.189 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.189 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.189 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.189 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-09T08:48:48.189 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-09T08:48:48.190 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.190 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.190 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.190 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.190 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.190 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.190 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.190 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.190 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.190 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.190 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.190 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-09T08:48:48.190 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-09T08:48:48.191 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.191 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.191 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.191 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.191 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.191 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.191 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.191 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.191 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.191 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.191 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.191 INFO:teuthology.orchestra.run.smithi107.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-09T08:48:48.191 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-09T08:48:48.192 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.192 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.192 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.192 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.192 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.192 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.192 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.192 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.192 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.192 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.192 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.192 INFO:teuthology.orchestra.run.smithi107.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-09T08:48:48.192 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-09T08:48:48.193 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.193 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.193 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.193 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.193 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.193 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.193 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.193 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.193 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.193 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.193 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.193 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_PASSWORD": { 2024-05-09T08:48:48.193 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-09T08:48:48.194 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.194 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.194 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.194 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-05-09T08:48:48.194 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.194 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.194 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.194 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.194 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.194 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.194 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.194 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-09T08:48:48.194 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-09T08:48:48.194 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.195 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.195 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.195 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.195 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.195 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.195 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.195 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.195 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.195 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.195 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.195 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_URL": { 2024-05-09T08:48:48.195 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_URL", 2024-05-09T08:48:48.195 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.196 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.196 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.196 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.196 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.196 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.196 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.196 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.196 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.196 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.196 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.196 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_API_USERNAME": { 2024-05-09T08:48:48.196 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-09T08:48:48.196 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.197 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.197 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.197 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-05-09T08:48:48.197 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.197 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.197 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.197 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.197 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.197 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.197 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.197 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-09T08:48:48.197 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-09T08:48:48.197 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.198 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.198 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.198 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.198 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.198 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.198 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.198 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.198 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.198 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.198 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.198 INFO:teuthology.orchestra.run.smithi107.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-09T08:48:48.198 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-09T08:48:48.198 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.199 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.199 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.199 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.199 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.199 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.199 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.199 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.199 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.199 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.199 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.199 INFO:teuthology.orchestra.run.smithi107.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-09T08:48:48.199 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-09T08:48:48.199 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.199 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.200 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.200 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.200 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.200 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.200 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.200 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.200 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.200 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.200 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.200 INFO:teuthology.orchestra.run.smithi107.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-09T08:48:48.200 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-09T08:48:48.200 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.200 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.201 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.201 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.201 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.201 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.201 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.201 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.201 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.201 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.201 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.201 INFO:teuthology.orchestra.run.smithi107.stdout: "PROMETHEUS_API_HOST": { 2024-05-09T08:48:48.201 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-09T08:48:48.201 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.201 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.202 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.202 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.202 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.202 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.202 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.202 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.202 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.202 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.202 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.202 INFO:teuthology.orchestra.run.smithi107.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-09T08:48:48.202 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-09T08:48:48.202 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-09T08:48:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-09T08:48:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.203 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.204 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.204 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.204 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.204 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.204 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.204 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.204 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.204 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.204 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.204 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-09T08:48:48.204 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-09T08:48:48.204 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.204 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-09T08:48:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-09T08:48:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-09T08:48:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-09T08:48:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-09T08:48:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-09T08:48:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-09T08:48:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-09T08:48:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-09T08:48:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-09T08:48:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_ENABLED": { 2024-05-09T08:48:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-09T08:48:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.211 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.211 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.211 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.211 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.211 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.211 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.211 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.211 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.211 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.211 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-09T08:48:48.211 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-09T08:48:48.211 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.211 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.212 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.212 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-09T08:48:48.212 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.212 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.212 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.212 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.212 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.212 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.212 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.212 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-09T08:48:48.212 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-09T08:48:48.212 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.212 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.213 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.213 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:48.213 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.213 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.213 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.213 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.213 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.213 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.213 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.213 INFO:teuthology.orchestra.run.smithi107.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-09T08:48:48.213 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-09T08:48:48.213 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.213 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.213 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8", 2024-05-09T08:48:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-09T08:48:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-09T08:48:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "45", 2024-05-09T08:48:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_ACCESS_KEY": { 2024-05-09T08:48:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-09T08:48:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-09T08:48:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-09T08:48:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "admin", 2024-05-09T08:48:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_SECRET_KEY": { 2024-05-09T08:48:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-09T08:48:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: "RGW_API_SSL_VERIFY": { 2024-05-09T08:48:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-09T08:48:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-09T08:48:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-09T08:48:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-05-09T08:48:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-09T08:48:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-09T08:48:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-09T08:48:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-09T08:48:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-05-09T08:48:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "cross_origin_url": { 2024-05-09T08:48:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cross_origin_url", 2024-05-09T08:48:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.222 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "crt_file": { 2024-05-09T08:48:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crt_file", 2024-05-09T08:48:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.223 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "debug": { 2024-05-09T08:48:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "debug", 2024-05-09T08:48:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.224 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Enable/disable debug options", 2024-05-09T08:48:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: "jwt_token_ttl": { 2024-05-09T08:48:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "jwt_token_ttl", 2024-05-09T08:48:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "28800", 2024-05-09T08:48:48.225 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "key_file": { 2024-05-09T08:48:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "key_file", 2024-05-09T08:48:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.226 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.227 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.229 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.230 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.231 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "motd": { 2024-05-09T08:48:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "motd", 2024-05-09T08:48:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.232 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "The message of the day", 2024-05-09T08:48:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "redirect_resolve_ip_addr": { 2024-05-09T08:48:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "redirect_resolve_ip_addr", 2024-05-09T08:48:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.233 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-05-09T08:48:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-05-09T08:48:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "::", 2024-05-09T08:48:48.234 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-05-09T08:48:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-05-09T08:48:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8080", 2024-05-09T08:48:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.235 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl": { 2024-05-09T08:48:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl", 2024-05-09T08:48:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.236 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl_server_port": { 2024-05-09T08:48:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl_server_port", 2024-05-09T08:48:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8443", 2024-05-09T08:48:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.237 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_behaviour": { 2024-05-09T08:48:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_behaviour", 2024-05-09T08:48:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "redirect", 2024-05-09T08:48:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: "redirect" 2024-05-09T08:48:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_error_status_code": { 2024-05-09T08:48:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_error_status_code", 2024-05-09T08:48:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "500", 2024-05-09T08:48:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "400", 2024-05-09T08:48:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "599", 2024-05-09T08:48:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "url_prefix": { 2024-05-09T08:48:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "url_prefix", 2024-05-09T08:48:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.240 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "devicehealth", 2024-05-09T08:48:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.241 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "enable_monitoring": { 2024-05-09T08:48:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enable_monitoring", 2024-05-09T08:48:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "monitor device health metrics", 2024-05-09T08:48:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.243 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.244 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.245 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.246 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.247 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.248 INFO:teuthology.orchestra.run.smithi107.stdout: "mark_out_threshold": { 2024-05-09T08:48:48.248 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mark_out_threshold", 2024-05-09T08:48:48.248 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.248 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.248 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.248 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "2419200", 2024-05-09T08:48:48.248 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.248 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.248 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.248 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-09T08:48:48.248 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.248 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.248 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.248 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "pool_name": { 2024-05-09T08:48:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pool_name", 2024-05-09T08:48:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "device_health_metrics", 2024-05-09T08:48:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-09T08:48:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.249 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.250 INFO:teuthology.orchestra.run.smithi107.stdout: "retention_period": { 2024-05-09T08:48:48.250 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "retention_period", 2024-05-09T08:48:48.250 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.250 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.250 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.250 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15552000", 2024-05-09T08:48:48.250 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.250 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.250 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.250 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long to retain device health metrics", 2024-05-09T08:48:48.250 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.250 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.250 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.250 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: "scrape_frequency": { 2024-05-09T08:48:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "scrape_frequency", 2024-05-09T08:48:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "86400", 2024-05-09T08:48:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-09T08:48:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.251 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "self_heal": { 2024-05-09T08:48:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "self_heal", 2024-05-09T08:48:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-09T08:48:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.252 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-05-09T08:48:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-05-09T08:48:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-05-09T08:48:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how frequently to wake up and check device health", 2024-05-09T08:48:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.253 INFO:teuthology.orchestra.run.smithi107.stdout: "warn_threshold": { 2024-05-09T08:48:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "warn_threshold", 2024-05-09T08:48:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "7257600", 2024-05-09T08:48:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-09T08:48:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.254 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "diskprediction_local", 2024-05-09T08:48:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.255 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.256 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.257 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.258 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.259 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "predict_interval": { 2024-05-09T08:48:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "predict_interval", 2024-05-09T08:48:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "86400", 2024-05-09T08:48:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.260 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "predictor_model": { 2024-05-09T08:48:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "predictor_model", 2024-05-09T08:48:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "prophetstor", 2024-05-09T08:48:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.261 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-05-09T08:48:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-05-09T08:48:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "600", 2024-05-09T08:48:48.262 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "influx", 2024-05-09T08:48:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": false, 2024-05-09T08:48:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "influxdb python module not found", 2024-05-09T08:48:48.263 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "batch_size": { 2024-05-09T08:48:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "batch_size", 2024-05-09T08:48:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5000", 2024-05-09T08:48:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-09T08:48:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.264 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "database": { 2024-05-09T08:48:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "database", 2024-05-09T08:48:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "ceph", 2024-05-09T08:48:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-09T08:48:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: "hostname": { 2024-05-09T08:48:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "hostname", 2024-05-09T08:48:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB server hostname", 2024-05-09T08:48:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.266 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-05-09T08:48:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-05-09T08:48:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "30", 2024-05-09T08:48:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "5", 2024-05-09T08:48:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-09T08:48:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.267 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.268 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.269 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.270 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "password": { 2024-05-09T08:48:48.272 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "password", 2024-05-09T08:48:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "password of InfluxDB server user", 2024-05-09T08:48:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "port": { 2024-05-09T08:48:48.273 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "port", 2024-05-09T08:48:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "8086", 2024-05-09T08:48:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "InfluxDB server port", 2024-05-09T08:48:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "ssl": { 2024-05-09T08:48:48.274 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ssl", 2024-05-09T08:48:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "false", 2024-05-09T08:48:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T08:48:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "threads": { 2024-05-09T08:48:48.275 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "threads", 2024-05-09T08:48:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-05-09T08:48:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-05-09T08:48:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "32", 2024-05-09T08:48:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-09T08:48:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: "username": { 2024-05-09T08:48:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "username", 2024-05-09T08:48:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "username of InfluxDB server user", 2024-05-09T08:48:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "verify_ssl": { 2024-05-09T08:48:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "verify_ssl", 2024-05-09T08:48:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "true", 2024-05-09T08:48:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T08:48:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.278 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "insights", 2024-05-09T08:48:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.279 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.280 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.281 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.282 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.283 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "iostat", 2024-05-09T08:48:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.284 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.285 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.286 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.287 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.288 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.289 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "k8sevents", 2024-05-09T08:48:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_event_retention_days": { 2024-05-09T08:48:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "ceph_event_retention_days", 2024-05-09T08:48:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.290 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "7", 2024-05-09T08:48:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-09T08:48:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "config_check_secs": { 2024-05-09T08:48:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "config_check_secs", 2024-05-09T08:48:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:48.291 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "10", 2024-05-09T08:48:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-09T08:48:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.292 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.294 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.295 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.296 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "localpool", 2024-05-09T08:48:48.297 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "failure_domain": { 2024-05-09T08:48:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "failure_domain", 2024-05-09T08:48:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "host", 2024-05-09T08:48:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "failure domain for any created local pool", 2024-05-09T08:48:48.298 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-09T08:48:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.300 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.301 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.302 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.303 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "min_size": { 2024-05-09T08:48:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "min_size", 2024-05-09T08:48:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default min_size for any created local pool", 2024-05-09T08:48:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-09T08:48:48.304 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "num_rep": { 2024-05-09T08:48:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "num_rep", 2024-05-09T08:48:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3", 2024-05-09T08:48:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default replica count for any created local pool", 2024-05-09T08:48:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.305 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_num": { 2024-05-09T08:48:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pg_num", 2024-05-09T08:48:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "128", 2024-05-09T08:48:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "default pg_num for any created local pool", 2024-05-09T08:48:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.306 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "prefix": { 2024-05-09T08:48:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prefix", 2024-05-09T08:48:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "name prefix for any created local pool", 2024-05-09T08:48:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.307 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "subtree": { 2024-05-09T08:48:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "subtree", 2024-05-09T08:48:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "rack", 2024-05-09T08:48:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-09T08:48:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-09T08:48:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.308 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mds_autoscaler", 2024-05-09T08:48:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.309 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.310 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.312 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.313 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mirroring", 2024-05-09T08:48:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.314 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.315 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.316 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.317 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.318 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.319 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "nfs", 2024-05-09T08:48:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.320 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.322 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.323 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.324 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "orchestrator", 2024-05-09T08:48:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: "fail_fs": { 2024-05-09T08:48:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "fail_fs", 2024-05-09T08:48:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-09T08:48:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.326 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.327 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.328 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.329 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.330 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "orchestrator": { 2024-05-09T08:48:48.331 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "orchestrator", 2024-05-09T08:48:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm", 2024-05-09T08:48:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "rook", 2024-05-09T08:48:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "test_orchestrator" 2024-05-09T08:48:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Orchestrator backend", 2024-05-09T08:48:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.332 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "osd_perf_query", 2024-05-09T08:48:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.334 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.335 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.336 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.337 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "osd_support", 2024-05-09T08:48:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.338 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.339 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.340 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "pg_autoscaler", 2024-05-09T08:48:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.345 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.346 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.347 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.348 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-05-09T08:48:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-05-09T08:48:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.349 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-05-09T08:48:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "threshold": { 2024-05-09T08:48:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "threshold", 2024-05-09T08:48:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.350 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "3.0", 2024-05-09T08:48:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1.0", 2024-05-09T08:48:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "scaling threshold", 2024-05-09T08:48:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-09T08:48:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "progress", 2024-05-09T08:48:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.351 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_pg_recovery_event": { 2024-05-09T08:48:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_pg_recovery_event", 2024-05-09T08:48:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-09T08:48:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.352 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "enabled": { 2024-05-09T08:48:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enabled", 2024-05-09T08:48:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.353 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.354 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.355 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.356 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.357 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.358 INFO:teuthology.orchestra.run.smithi107.stdout: "max_completed_events": { 2024-05-09T08:48:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_completed_events", 2024-05-09T08:48:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "50", 2024-05-09T08:48:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "number of past completed events to remember", 2024-05-09T08:48:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "sleep_interval": { 2024-05-09T08:48:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "sleep_interval", 2024-05-09T08:48:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "5", 2024-05-09T08:48:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "how long the module is going to sleep", 2024-05-09T08:48:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "prometheus", 2024-05-09T08:48:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "cache": { 2024-05-09T08:48:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "cache", 2024-05-09T08:48:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.361 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "exclude_perf_counters": { 2024-05-09T08:48:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "exclude_perf_counters", 2024-05-09T08:48:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.362 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-09T08:48:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-09T08:48:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.364 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.365 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_stats_pools": { 2024-05-09T08:48:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_stats_pools", 2024-05-09T08:48:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.368 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-09T08:48:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-09T08:48:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "300", 2024-05-09T08:48:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "scrape_interval": { 2024-05-09T08:48:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "scrape_interval", 2024-05-09T08:48:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15.0", 2024-05-09T08:48:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.370 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-05-09T08:48:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-05-09T08:48:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "::", 2024-05-09T08:48:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-09T08:48:48.371 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-05-09T08:48:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-05-09T08:48:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "9283", 2024-05-09T08:48:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-09T08:48:48.372 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "stale_cache_strategy": { 2024-05-09T08:48:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "stale_cache_strategy", 2024-05-09T08:48:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "log", 2024-05-09T08:48:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.373 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_behaviour": { 2024-05-09T08:48:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_behaviour", 2024-05-09T08:48:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "default", 2024-05-09T08:48:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "default", 2024-05-09T08:48:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: "error" 2024-05-09T08:48:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "standby_error_status_code": { 2024-05-09T08:48:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "standby_error_status_code", 2024-05-09T08:48:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "500", 2024-05-09T08:48:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "400", 2024-05-09T08:48:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "599", 2024-05-09T08:48:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_support", 2024-05-09T08:48:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.376 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.377 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.378 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.379 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.380 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "max_concurrent_snap_create": { 2024-05-09T08:48:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_concurrent_snap_create", 2024-05-09T08:48:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.381 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10", 2024-05-09T08:48:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "mirror_snapshot_schedule": { 2024-05-09T08:48:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mirror_snapshot_schedule", 2024-05-09T08:48:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.382 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "trash_purge_schedule": { 2024-05-09T08:48:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "trash_purge_schedule", 2024-05-09T08:48:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.383 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.384 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "restful", 2024-05-09T08:48:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "enable_auth": { 2024-05-09T08:48:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enable_auth", 2024-05-09T08:48:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.385 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "key_file": { 2024-05-09T08:48:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "key_file", 2024-05-09T08:48:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.386 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.387 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.388 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.389 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.390 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.391 INFO:teuthology.orchestra.run.smithi107.stdout: "server_addr": { 2024-05-09T08:48:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_addr", 2024-05-09T08:48:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "server_port": { 2024-05-09T08:48:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "server_port", 2024-05-09T08:48:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rgw", 2024-05-09T08:48:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.394 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.395 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.396 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.397 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.398 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rook", 2024-05-09T08:48:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.399 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.400 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.401 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.401 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.401 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.401 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.401 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.401 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.401 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.401 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.401 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.401 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.401 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.401 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.402 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.402 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.402 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.402 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.402 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.402 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.402 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.402 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.402 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.402 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.402 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.402 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.403 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.403 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.403 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.403 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.403 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.403 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.403 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.403 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.403 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.403 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.403 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.403 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.403 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.403 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.404 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.404 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.404 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.404 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.404 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.404 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.404 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.404 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.404 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.404 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.404 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.404 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.404 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.405 INFO:teuthology.orchestra.run.smithi107.stdout: "storage_class": { 2024-05-09T08:48:48.405 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "storage_class", 2024-05-09T08:48:48.405 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.405 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.405 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.405 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "local", 2024-05-09T08:48:48.405 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.405 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.405 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.405 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-09T08:48:48.405 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.405 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.405 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.406 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.406 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.406 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "selftest", 2024-05-09T08:48:48.406 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.406 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.406 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.406 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.406 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.406 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.406 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.406 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.406 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.407 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.407 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.407 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.407 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.407 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.407 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.407 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.407 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.407 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.407 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.407 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.407 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.407 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.407 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.408 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.408 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.408 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.408 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.408 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.408 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.408 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.408 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.408 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.408 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.408 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.408 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.408 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.408 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.409 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.409 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.409 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.409 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.409 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.409 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.409 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.409 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.409 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.409 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.409 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.409 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.409 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.409 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.410 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.410 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.410 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.410 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.410 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.410 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.410 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.410 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.410 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.410 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.410 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.410 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.410 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.411 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.411 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.411 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.411 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.411 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.411 INFO:teuthology.orchestra.run.smithi107.stdout: "roption1": { 2024-05-09T08:48:48.411 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "roption1", 2024-05-09T08:48:48.411 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.411 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.411 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.411 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.411 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.411 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.412 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.412 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.412 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.412 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.412 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.412 INFO:teuthology.orchestra.run.smithi107.stdout: "roption2": { 2024-05-09T08:48:48.412 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "roption2", 2024-05-09T08:48:48.412 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.412 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.412 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.412 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "xyz", 2024-05-09T08:48:48.412 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.412 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.413 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.413 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.413 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.413 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.413 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.413 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption1": { 2024-05-09T08:48:48.413 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption1", 2024-05-09T08:48:48.413 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.413 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.413 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.413 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.413 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.413 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.414 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.414 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.414 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.414 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.414 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.414 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption2": { 2024-05-09T08:48:48.414 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption2", 2024-05-09T08:48:48.414 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.414 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.414 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.414 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.414 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.414 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.415 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.415 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.415 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.415 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.415 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.415 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption3": { 2024-05-09T08:48:48.415 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption3", 2024-05-09T08:48:48.415 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "float", 2024-05-09T08:48:48.415 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.415 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.415 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.415 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.416 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.416 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.416 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.416 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.416 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.416 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.416 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption4": { 2024-05-09T08:48:48.416 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption4", 2024-05-09T08:48:48.416 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.416 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.416 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.416 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.416 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.417 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.417 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.417 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.417 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.417 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.417 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.417 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption5": { 2024-05-09T08:48:48.417 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption5", 2024-05-09T08:48:48.417 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.417 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.417 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.417 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.418 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.418 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.418 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.418 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.418 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.418 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.418 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.418 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption6": { 2024-05-09T08:48:48.418 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption6", 2024-05-09T08:48:48.418 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.418 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.418 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.419 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.419 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.419 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.419 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.419 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.419 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.419 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.419 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.419 INFO:teuthology.orchestra.run.smithi107.stdout: "rwoption7": { 2024-05-09T08:48:48.419 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rwoption7", 2024-05-09T08:48:48.419 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.419 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.419 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.420 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.420 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "1", 2024-05-09T08:48:48.420 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "42", 2024-05-09T08:48:48.420 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.420 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.420 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.420 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.420 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.420 INFO:teuthology.orchestra.run.smithi107.stdout: "testkey": { 2024-05-09T08:48:48.420 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testkey", 2024-05-09T08:48:48.420 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.420 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.421 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.421 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.421 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.421 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.421 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.421 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.421 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.421 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.421 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.421 INFO:teuthology.orchestra.run.smithi107.stdout: "testlkey": { 2024-05-09T08:48:48.421 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testlkey", 2024-05-09T08:48:48.421 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.421 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.422 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.422 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.422 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.422 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.422 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.422 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.422 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.422 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.422 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.422 INFO:teuthology.orchestra.run.smithi107.stdout: "testnewline": { 2024-05-09T08:48:48.422 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "testnewline", 2024-05-09T08:48:48.422 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.422 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.423 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.423 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.423 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.423 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.423 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.423 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.423 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.423 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.423 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.423 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.423 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.423 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snap_schedule", 2024-05-09T08:48:48.424 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.424 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.424 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.424 INFO:teuthology.orchestra.run.smithi107.stdout: "allow_m_granularity": { 2024-05-09T08:48:48.424 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "allow_m_granularity", 2024-05-09T08:48:48.424 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.424 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.424 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.424 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.424 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.424 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.424 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.424 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "allow minute scheduled snapshots", 2024-05-09T08:48:48.424 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.425 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.425 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.425 INFO:teuthology.orchestra.run.smithi107.stdout: "dump_on_update": { 2024-05-09T08:48:48.425 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "dump_on_update", 2024-05-09T08:48:48.425 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.425 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.425 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.425 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.425 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.425 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.425 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.425 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "dump database to debug log on update", 2024-05-09T08:48:48.425 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.426 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.426 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.426 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.426 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.426 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.426 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.426 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.426 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.426 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.426 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.426 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.426 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.427 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.427 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.427 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.427 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.427 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.427 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.427 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.427 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.427 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.427 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.427 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.427 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.427 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.427 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.428 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.428 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.428 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.428 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.428 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.428 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.428 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.428 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.428 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.428 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.428 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.428 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.428 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.429 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.429 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.429 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.429 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.429 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.429 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.429 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.429 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.429 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.429 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.429 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.429 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.429 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.429 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.430 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.430 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.430 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.430 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.430 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.430 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.430 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.430 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.430 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.430 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.430 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.430 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.431 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.431 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.431 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.431 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.431 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.431 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "stats", 2024-05-09T08:48:48.431 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.431 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.431 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.431 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.431 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.431 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.431 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.432 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.432 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.432 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.432 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.432 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.432 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.432 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.432 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.432 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.432 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.432 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.432 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.432 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.432 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.433 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.433 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.433 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.433 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.433 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.433 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.433 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.433 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.433 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.433 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.433 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.433 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.433 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.434 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.434 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.434 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.434 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.434 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.434 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.434 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.434 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.434 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.434 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.434 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.434 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.434 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.435 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.435 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.435 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.435 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.435 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.435 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.435 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.435 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.435 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.435 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.435 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.435 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.436 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.436 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.436 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.436 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.436 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.436 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.436 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.436 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.436 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.436 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.436 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.436 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.436 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "status", 2024-05-09T08:48:48.436 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.437 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.437 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.437 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.437 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.437 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.437 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.437 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.437 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.437 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.437 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.437 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.437 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.437 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.437 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.438 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.438 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.438 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.438 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.438 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.438 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.438 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.438 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.438 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.438 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.438 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.438 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.438 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.439 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.439 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.439 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.439 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.439 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.439 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.439 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.439 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.439 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.439 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.439 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.439 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.439 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.440 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.440 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.440 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.440 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.440 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.440 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.440 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.440 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.440 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.440 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.440 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.441 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.441 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.441 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.441 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.441 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.441 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.441 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.441 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.442 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.442 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.442 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.442 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.442 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.442 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.442 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.442 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.442 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.442 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.442 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.442 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "telegraf", 2024-05-09T08:48:48.443 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.443 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.443 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.443 INFO:teuthology.orchestra.run.smithi107.stdout: "address": { 2024-05-09T08:48:48.443 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "address", 2024-05-09T08:48:48.443 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.443 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.443 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.443 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-09T08:48:48.443 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.443 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.443 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.443 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.443 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.443 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.444 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.444 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-05-09T08:48:48.444 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-05-09T08:48:48.444 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.444 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.444 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.444 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "15", 2024-05-09T08:48:48.444 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.444 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.444 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.444 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.444 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.444 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.445 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.445 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.445 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.445 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.445 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.445 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.445 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.445 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.445 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.445 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.445 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.445 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.445 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.446 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.446 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.446 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.446 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.446 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.446 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.446 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.446 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.446 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.446 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.446 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.446 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.447 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.447 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.447 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.447 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.447 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.447 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.447 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.447 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.447 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.447 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.447 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.447 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.447 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.448 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.448 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.448 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.448 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.448 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.448 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.448 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.448 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.448 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.448 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.448 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.448 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.448 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.448 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.449 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.449 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.449 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.449 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.449 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.449 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.449 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.449 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.449 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.449 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.449 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.449 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.449 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.450 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.450 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.450 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.450 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.450 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "telemetry", 2024-05-09T08:48:48.450 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.450 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.450 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.450 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_basic": { 2024-05-09T08:48:48.450 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_basic", 2024-05-09T08:48:48.450 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.451 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.451 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.451 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.451 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.451 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.451 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.451 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-09T08:48:48.451 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.451 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.451 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.451 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_crash": { 2024-05-09T08:48:48.451 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_crash", 2024-05-09T08:48:48.451 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.451 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.452 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.452 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.452 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.452 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.452 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.452 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-09T08:48:48.452 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.452 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.452 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.452 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_device": { 2024-05-09T08:48:48.452 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_device", 2024-05-09T08:48:48.452 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.452 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.453 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.453 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.453 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.453 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.453 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.453 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-09T08:48:48.453 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.453 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.453 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.453 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_ident": { 2024-05-09T08:48:48.453 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_ident", 2024-05-09T08:48:48.453 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.453 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.453 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.454 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.454 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.454 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.454 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.454 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-09T08:48:48.454 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.454 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.454 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.454 INFO:teuthology.orchestra.run.smithi107.stdout: "channel_perf": { 2024-05-09T08:48:48.454 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "channel_perf", 2024-05-09T08:48:48.454 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.454 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.454 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.455 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.455 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.455 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.455 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.455 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-09T08:48:48.455 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.455 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.455 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.455 INFO:teuthology.orchestra.run.smithi107.stdout: "contact": { 2024-05-09T08:48:48.455 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "contact", 2024-05-09T08:48:48.455 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.455 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.455 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.456 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.456 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.456 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.456 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.456 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.456 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.456 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.456 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.456 INFO:teuthology.orchestra.run.smithi107.stdout: "description": { 2024-05-09T08:48:48.456 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "description", 2024-05-09T08:48:48.456 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.456 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.456 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.457 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.457 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.457 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.457 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.457 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.457 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.457 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.457 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.457 INFO:teuthology.orchestra.run.smithi107.stdout: "device_url": { 2024-05-09T08:48:48.457 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "device_url", 2024-05-09T08:48:48.457 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.457 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.457 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.458 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-09T08:48:48.458 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.458 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.458 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.458 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.458 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.458 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.458 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.458 INFO:teuthology.orchestra.run.smithi107.stdout: "enabled": { 2024-05-09T08:48:48.458 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "enabled", 2024-05-09T08:48:48.458 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.458 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.458 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.458 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.459 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.459 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.459 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.459 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.459 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.459 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.459 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.459 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-05-09T08:48:48.459 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-05-09T08:48:48.459 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.459 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.459 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.459 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "24", 2024-05-09T08:48:48.459 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "8", 2024-05-09T08:48:48.460 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.460 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.460 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.460 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.460 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.460 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.460 INFO:teuthology.orchestra.run.smithi107.stdout: "last_opt_revision": { 2024-05-09T08:48:48.460 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "last_opt_revision", 2024-05-09T08:48:48.460 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.460 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.460 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.460 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "1", 2024-05-09T08:48:48.460 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.461 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.461 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.461 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.461 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.461 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.461 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.461 INFO:teuthology.orchestra.run.smithi107.stdout: "leaderboard": { 2024-05-09T08:48:48.461 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "leaderboard", 2024-05-09T08:48:48.461 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.461 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.461 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.461 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.461 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.461 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.462 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.462 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.462 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.462 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.462 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.462 INFO:teuthology.orchestra.run.smithi107.stdout: "leaderboard_description": { 2024-05-09T08:48:48.462 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "leaderboard_description", 2024-05-09T08:48:48.462 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.462 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.462 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.462 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.462 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.462 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.463 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.463 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.463 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.463 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.463 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.463 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.463 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.463 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.463 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.463 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.463 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.463 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.463 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.463 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.464 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.464 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.464 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.464 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.464 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.464 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.464 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.464 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.464 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.464 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.464 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.464 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.464 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.465 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.465 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.465 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.465 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.465 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.465 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.465 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.465 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.465 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.465 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.465 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.465 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.465 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.465 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.466 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.466 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.466 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.466 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.466 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.466 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.466 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.466 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.466 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.466 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.466 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.466 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.466 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.466 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.467 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.467 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.467 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.467 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.467 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.467 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.467 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.467 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.467 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.467 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.467 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.467 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.467 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.468 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.468 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.468 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.468 INFO:teuthology.orchestra.run.smithi107.stdout: "organization": { 2024-05-09T08:48:48.468 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "organization", 2024-05-09T08:48:48.468 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.468 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.468 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.468 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.468 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.468 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.468 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.468 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.469 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.469 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.469 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.469 INFO:teuthology.orchestra.run.smithi107.stdout: "proxy": { 2024-05-09T08:48:48.469 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "proxy", 2024-05-09T08:48:48.469 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.469 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.469 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.469 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.469 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.469 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.469 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.469 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.470 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.470 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.470 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.470 INFO:teuthology.orchestra.run.smithi107.stdout: "url": { 2024-05-09T08:48:48.470 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "url", 2024-05-09T08:48:48.470 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.470 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.470 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.470 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-09T08:48:48.470 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.470 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.470 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.471 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.471 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.471 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.471 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.471 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.471 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.471 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "test_orchestrator", 2024-05-09T08:48:48.471 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.471 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.471 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.471 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.471 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.472 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.472 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.472 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.472 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.472 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.472 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.472 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.472 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.472 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.472 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.472 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.472 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.472 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.472 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.473 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.473 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.473 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.473 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.473 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.473 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.473 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.473 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.473 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.473 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.473 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.473 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.473 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.474 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.474 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.474 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.474 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.474 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.474 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.474 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.474 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.474 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.474 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.474 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.474 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.474 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.475 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.475 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.475 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.475 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.475 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.475 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.475 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.475 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.475 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.475 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.475 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.475 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.475 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.477 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "volumes", 2024-05-09T08:48:48.477 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.477 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.477 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.477 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.477 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.477 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.477 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.477 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.477 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.477 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.477 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.477 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.477 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.478 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.478 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.478 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.478 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.478 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.478 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.478 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.478 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.478 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.478 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.478 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.478 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.478 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.478 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.479 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.479 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.479 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.479 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.479 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.479 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.479 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.479 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.479 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.479 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.479 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.479 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.479 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.480 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.481 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.481 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.481 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.481 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.481 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.481 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.481 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.481 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.481 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.481 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.481 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.481 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.481 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.482 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.482 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.482 INFO:teuthology.orchestra.run.smithi107.stdout: "max_concurrent_clones": { 2024-05-09T08:48:48.482 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "max_concurrent_clones", 2024-05-09T08:48:48.482 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.482 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.482 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.482 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "4", 2024-05-09T08:48:48.482 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.482 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.482 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.482 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-09T08:48:48.482 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.483 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.483 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.483 INFO:teuthology.orchestra.run.smithi107.stdout: "periodic_async_work": { 2024-05-09T08:48:48.483 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "periodic_async_work", 2024-05-09T08:48:48.483 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.483 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.483 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.483 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.483 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.483 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.483 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.483 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Periodically check for async work", 2024-05-09T08:48:48.483 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot_clone_delay": { 2024-05-09T08:48:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snapshot_clone_delay", 2024-05-09T08:48:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "0", 2024-05-09T08:48:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-09T08:48:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.485 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.485 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.485 INFO:teuthology.orchestra.run.smithi107.stdout: "snapshot_clone_no_wait": { 2024-05-09T08:48:48.485 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snapshot_clone_no_wait", 2024-05-09T08:48:48.485 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.485 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.485 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.485 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "True", 2024-05-09T08:48:48.485 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.485 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.485 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.485 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-09T08:48:48.485 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.485 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.486 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.486 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.486 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.486 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix", 2024-05-09T08:48:48.486 INFO:teuthology.orchestra.run.smithi107.stdout: "can_run": true, 2024-05-09T08:48:48.486 INFO:teuthology.orchestra.run.smithi107.stdout: "error_string": "", 2024-05-09T08:48:48.486 INFO:teuthology.orchestra.run.smithi107.stdout: "module_options": { 2024-05-09T08:48:48.486 INFO:teuthology.orchestra.run.smithi107.stdout: "discovery_interval": { 2024-05-09T08:48:48.486 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "discovery_interval", 2024-05-09T08:48:48.486 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "uint", 2024-05-09T08:48:48.486 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.486 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.487 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "100", 2024-05-09T08:48:48.487 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.487 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.487 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.487 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.487 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.487 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.487 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.487 INFO:teuthology.orchestra.run.smithi107.stdout: "identifier": { 2024-05-09T08:48:48.487 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "identifier", 2024-05-09T08:48:48.487 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.487 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.488 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.488 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.488 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.488 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.488 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.488 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.488 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.488 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.488 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.488 INFO:teuthology.orchestra.run.smithi107.stdout: "interval": { 2024-05-09T08:48:48.488 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "interval", 2024-05-09T08:48:48.488 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "secs", 2024-05-09T08:48:48.488 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.489 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.489 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "60", 2024-05-09T08:48:48.489 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.489 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.489 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.489 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.489 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.489 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.489 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.489 INFO:teuthology.orchestra.run.smithi107.stdout: "log_level": { 2024-05-09T08:48:48.489 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_level", 2024-05-09T08:48:48.489 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.489 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.490 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.490 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.490 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.490 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.490 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.490 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.490 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.490 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.490 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.490 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.490 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.490 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.490 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.490 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.491 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.491 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.491 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster": { 2024-05-09T08:48:48.491 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster", 2024-05-09T08:48:48.491 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.491 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.491 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.491 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.491 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.491 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.491 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.491 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.491 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.492 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.492 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.492 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_cluster_level": { 2024-05-09T08:48:48.492 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_cluster_level", 2024-05-09T08:48:48.492 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.492 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.492 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.492 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "info", 2024-05-09T08:48:48.492 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.492 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.492 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [ 2024-05-09T08:48:48.492 INFO:teuthology.orchestra.run.smithi107.stdout: "", 2024-05-09T08:48:48.492 INFO:teuthology.orchestra.run.smithi107.stdout: "critical", 2024-05-09T08:48:48.493 INFO:teuthology.orchestra.run.smithi107.stdout: "debug", 2024-05-09T08:48:48.493 INFO:teuthology.orchestra.run.smithi107.stdout: "error", 2024-05-09T08:48:48.493 INFO:teuthology.orchestra.run.smithi107.stdout: "info", 2024-05-09T08:48:48.493 INFO:teuthology.orchestra.run.smithi107.stdout: "warning" 2024-05-09T08:48:48.493 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.493 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.493 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.493 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.493 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.493 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": { 2024-05-09T08:48:48.493 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "log_to_file", 2024-05-09T08:48:48.493 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "bool", 2024-05-09T08:48:48.493 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.493 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 1, 2024-05-09T08:48:48.494 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "False", 2024-05-09T08:48:48.494 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.494 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.494 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.494 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.494 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.494 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.494 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.494 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_host": { 2024-05-09T08:48:48.494 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_host", 2024-05-09T08:48:48.494 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.494 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.494 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.495 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "", 2024-05-09T08:48:48.495 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.495 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.495 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.495 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.495 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.495 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.495 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.495 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_port": { 2024-05-09T08:48:48.495 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_port", 2024-05-09T08:48:48.495 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "int", 2024-05-09T08:48:48.495 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.495 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.496 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "10051", 2024-05-09T08:48:48.496 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.496 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.496 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.496 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.496 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.496 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.496 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.496 INFO:teuthology.orchestra.run.smithi107.stdout: "zabbix_sender": { 2024-05-09T08:48:48.496 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "zabbix_sender", 2024-05-09T08:48:48.496 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "str", 2024-05-09T08:48:48.496 INFO:teuthology.orchestra.run.smithi107.stdout: "level": "advanced", 2024-05-09T08:48:48.496 INFO:teuthology.orchestra.run.smithi107.stdout: "flags": 0, 2024-05-09T08:48:48.497 INFO:teuthology.orchestra.run.smithi107.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-09T08:48:48.497 INFO:teuthology.orchestra.run.smithi107.stdout: "min": "", 2024-05-09T08:48:48.497 INFO:teuthology.orchestra.run.smithi107.stdout: "max": "", 2024-05-09T08:48:48.497 INFO:teuthology.orchestra.run.smithi107.stdout: "enum_allowed": [], 2024-05-09T08:48:48.497 INFO:teuthology.orchestra.run.smithi107.stdout: "desc": "", 2024-05-09T08:48:48.497 INFO:teuthology.orchestra.run.smithi107.stdout: "long_desc": "", 2024-05-09T08:48:48.497 INFO:teuthology.orchestra.run.smithi107.stdout: "tags": [], 2024-05-09T08:48:48.497 INFO:teuthology.orchestra.run.smithi107.stdout: "see_also": [] 2024-05-09T08:48:48.497 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.497 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.497 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.497 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.497 INFO:teuthology.orchestra.run.smithi107.stdout: "services": { 2024-05-09T08:48:48.497 INFO:teuthology.orchestra.run.smithi107.stdout: "dashboard": "https://172.21.15.107:8443/" 2024-05-09T08:48:48.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.498 INFO:teuthology.orchestra.run.smithi107.stdout: "always_on_modules": { 2024-05-09T08:48:48.498 INFO:teuthology.orchestra.run.smithi107.stdout: "octopus": [ 2024-05-09T08:48:48.498 INFO:teuthology.orchestra.run.smithi107.stdout: "balancer", 2024-05-09T08:48:48.498 INFO:teuthology.orchestra.run.smithi107.stdout: "crash", 2024-05-09T08:48:48.498 INFO:teuthology.orchestra.run.smithi107.stdout: "devicehealth", 2024-05-09T08:48:48.498 INFO:teuthology.orchestra.run.smithi107.stdout: "orchestrator", 2024-05-09T08:48:48.498 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscaler", 2024-05-09T08:48:48.498 INFO:teuthology.orchestra.run.smithi107.stdout: "progress", 2024-05-09T08:48:48.498 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_support", 2024-05-09T08:48:48.498 INFO:teuthology.orchestra.run.smithi107.stdout: "status", 2024-05-09T08:48:48.498 INFO:teuthology.orchestra.run.smithi107.stdout: "telemetry", 2024-05-09T08:48:48.498 INFO:teuthology.orchestra.run.smithi107.stdout: "volumes" 2024-05-09T08:48:48.498 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.499 INFO:teuthology.orchestra.run.smithi107.stdout: "pacific": [ 2024-05-09T08:48:48.499 INFO:teuthology.orchestra.run.smithi107.stdout: "balancer", 2024-05-09T08:48:48.499 INFO:teuthology.orchestra.run.smithi107.stdout: "crash", 2024-05-09T08:48:48.499 INFO:teuthology.orchestra.run.smithi107.stdout: "devicehealth", 2024-05-09T08:48:48.499 INFO:teuthology.orchestra.run.smithi107.stdout: "orchestrator", 2024-05-09T08:48:48.499 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscaler", 2024-05-09T08:48:48.499 INFO:teuthology.orchestra.run.smithi107.stdout: "progress", 2024-05-09T08:48:48.499 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_support", 2024-05-09T08:48:48.499 INFO:teuthology.orchestra.run.smithi107.stdout: "status", 2024-05-09T08:48:48.499 INFO:teuthology.orchestra.run.smithi107.stdout: "telemetry", 2024-05-09T08:48:48.499 INFO:teuthology.orchestra.run.smithi107.stdout: "volumes" 2024-05-09T08:48:48.499 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.499 INFO:teuthology.orchestra.run.smithi107.stdout: "quincy": [ 2024-05-09T08:48:48.499 INFO:teuthology.orchestra.run.smithi107.stdout: "balancer", 2024-05-09T08:48:48.500 INFO:teuthology.orchestra.run.smithi107.stdout: "crash", 2024-05-09T08:48:48.500 INFO:teuthology.orchestra.run.smithi107.stdout: "devicehealth", 2024-05-09T08:48:48.500 INFO:teuthology.orchestra.run.smithi107.stdout: "orchestrator", 2024-05-09T08:48:48.500 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscaler", 2024-05-09T08:48:48.500 INFO:teuthology.orchestra.run.smithi107.stdout: "progress", 2024-05-09T08:48:48.500 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_support", 2024-05-09T08:48:48.500 INFO:teuthology.orchestra.run.smithi107.stdout: "status", 2024-05-09T08:48:48.500 INFO:teuthology.orchestra.run.smithi107.stdout: "telemetry", 2024-05-09T08:48:48.500 INFO:teuthology.orchestra.run.smithi107.stdout: "volumes" 2024-05-09T08:48:48.500 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.500 INFO:teuthology.orchestra.run.smithi107.stdout: "reef": [ 2024-05-09T08:48:48.500 INFO:teuthology.orchestra.run.smithi107.stdout: "balancer", 2024-05-09T08:48:48.500 INFO:teuthology.orchestra.run.smithi107.stdout: "crash", 2024-05-09T08:48:48.500 INFO:teuthology.orchestra.run.smithi107.stdout: "devicehealth", 2024-05-09T08:48:48.501 INFO:teuthology.orchestra.run.smithi107.stdout: "orchestrator", 2024-05-09T08:48:48.501 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscaler", 2024-05-09T08:48:48.501 INFO:teuthology.orchestra.run.smithi107.stdout: "progress", 2024-05-09T08:48:48.501 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_support", 2024-05-09T08:48:48.501 INFO:teuthology.orchestra.run.smithi107.stdout: "status", 2024-05-09T08:48:48.501 INFO:teuthology.orchestra.run.smithi107.stdout: "telemetry", 2024-05-09T08:48:48.501 INFO:teuthology.orchestra.run.smithi107.stdout: "volumes" 2024-05-09T08:48:48.501 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-05-09T08:48:48.501 INFO:teuthology.orchestra.run.smithi107.stdout: "squid": [ 2024-05-09T08:48:48.501 INFO:teuthology.orchestra.run.smithi107.stdout: "balancer", 2024-05-09T08:48:48.501 INFO:teuthology.orchestra.run.smithi107.stdout: "crash", 2024-05-09T08:48:48.501 INFO:teuthology.orchestra.run.smithi107.stdout: "devicehealth", 2024-05-09T08:48:48.501 INFO:teuthology.orchestra.run.smithi107.stdout: "orchestrator", 2024-05-09T08:48:48.501 INFO:teuthology.orchestra.run.smithi107.stdout: "pg_autoscaler", 2024-05-09T08:48:48.502 INFO:teuthology.orchestra.run.smithi107.stdout: "progress", 2024-05-09T08:48:48.502 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_support", 2024-05-09T08:48:48.502 INFO:teuthology.orchestra.run.smithi107.stdout: "status", 2024-05-09T08:48:48.502 INFO:teuthology.orchestra.run.smithi107.stdout: "telemetry", 2024-05-09T08:48:48.502 INFO:teuthology.orchestra.run.smithi107.stdout: "volumes" 2024-05-09T08:48:48.502 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-05-09T08:48:48.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.502 INFO:teuthology.orchestra.run.smithi107.stdout: "last_failure_osd_epoch": 94, 2024-05-09T08:48:48.502 INFO:teuthology.orchestra.run.smithi107.stdout: "active_clients": [ 2024-05-09T08:48:48.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.502 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "devicehealth", 2024-05-09T08:48:48.502 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-05-09T08:48:48.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.502 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2024-05-09T08:48:48.502 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:0", 2024-05-09T08:48:48.503 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1144963265 2024-05-09T08:48:48.503 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.503 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-05-09T08:48:48.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.503 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "libcephsqlite", 2024-05-09T08:48:48.503 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-05-09T08:48:48.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.503 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2024-05-09T08:48:48.503 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:0", 2024-05-09T08:48:48.503 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1891418862 2024-05-09T08:48:48.503 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.503 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-05-09T08:48:48.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.504 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "rbd_support", 2024-05-09T08:48:48.504 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-05-09T08:48:48.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.504 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2024-05-09T08:48:48.504 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:0", 2024-05-09T08:48:48.504 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4217000757 2024-05-09T08:48:48.504 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.504 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-05-09T08:48:48.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.504 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snap_schedule", 2024-05-09T08:48:48.504 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-05-09T08:48:48.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.505 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2024-05-09T08:48:48.505 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:0", 2024-05-09T08:48:48.505 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 191362281 2024-05-09T08:48:48.505 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.505 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-05-09T08:48:48.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.505 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "snap_schedule", 2024-05-09T08:48:48.505 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-05-09T08:48:48.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.505 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2024-05-09T08:48:48.505 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:0", 2024-05-09T08:48:48.505 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 1787701072 2024-05-09T08:48:48.505 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.506 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-05-09T08:48:48.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:48.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.506 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "volumes", 2024-05-09T08:48:48.506 INFO:teuthology.orchestra.run.smithi107.stdout: "addrvec": [ 2024-05-09T08:48:48.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-05-09T08:48:48.506 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v2", 2024-05-09T08:48:48.506 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "172.21.15.107:0", 2024-05-09T08:48:48.506 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 4048220078 2024-05-09T08:48:48.506 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.506 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-05-09T08:48:48.506 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:48.506 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-05-09T08:48:48.506 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:48:48.508 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:47 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2772890828' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:48:48.508 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:47 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2483435048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:48:48.508 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:47 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2821848947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:48:48.508 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:47 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2772890828' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:48:48.508 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:47 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2483435048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:48:48.508 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:47 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2821848947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:48:48.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:47.204+0000 7f86de29a640 1 -- 172.21.15.107:0/2399621766 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f86c00a2d00 msgr2=0x7f86c00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:48.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:47.204+0000 7f86de29a640 1 --2- 172.21.15.107:0/2399621766 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f86c00a2d00 0x7f86c00a51c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f86d8069510 tx=0x7f86cc00b590 comp rx=0 tx=0).stop 2024-05-09T08:48:48.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:47.204+0000 7f86de29a640 1 -- 172.21.15.107:0/2399621766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d8100710 msgr2=0x7f86d8198f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:48:48.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:47.204+0000 7f86de29a640 1 --2- 172.21.15.107:0/2399621766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d8100710 0x7f86d8198f20 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f86c800d950 tx=0x7f86c800de10 comp rx=0 tx=0).stop 2024-05-09T08:48:48.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:47.204+0000 7f86de29a640 1 -- 172.21.15.107:0/2399621766 shutdown_connections 2024-05-09T08:48:48.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:47.204+0000 7f86de29a640 1 --2- 172.21.15.107:0/2399621766 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f86c00a2d00 0x7f86c00a51c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:48.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:47.204+0000 7f86de29a640 1 --2- 172.21.15.107:0/2399621766 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86d8106720 0x7f86d81a3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:48.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:47.204+0000 7f86de29a640 1 --2- 172.21.15.107:0/2399621766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86d81010d0 0x7f86d8199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:48.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:47.204+0000 7f86de29a640 1 --2- 172.21.15.107:0/2399621766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d8100710 0x7f86d8198f20 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:48:48.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:47.204+0000 7f86de29a640 1 -- 172.21.15.107:0/2399621766 >> 172.21.15.107:0/2399621766 conn(0x7f86d8077830 msgr2=0x7f86d8103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:48:48.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:47.204+0000 7f86de29a640 1 -- 172.21.15.107:0/2399621766 shutdown_connections 2024-05-09T08:48:48.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:48:47.204+0000 7f86de29a640 1 -- 172.21.15.107:0/2399621766 wait complete. 2024-05-09T08:48:48.516 DEBUG:tasks.check_counter:Getting stats from a 2024-05-09T08:48:48.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph daemon mds.a perf dump 2024-05-09T08:48:48.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:47 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2772890828' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:48:48.517 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:47 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2483435048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:48:48.517 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:47 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2821848947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:48:48.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:48 smithi107 ceph-mon[30122]: pgmap v485: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-09T08:48:48.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:48 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2399621766' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-09T08:48:48.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:48 smithi157 ceph-mon[33614]: pgmap v485: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-09T08:48:48.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:48 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2399621766' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-09T08:48:48.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:48 smithi110 ceph-mon[33622]: pgmap v485: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-09T08:48:48.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:48 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2399621766' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-09T08:48:48.768 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:50.463 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:50 smithi107 ceph-mon[30122]: pgmap v486: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-09T08:48:50.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:50 smithi157 ceph-mon[33614]: pgmap v486: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-09T08:48:50.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:50 smithi110 ceph-mon[33622]: pgmap v486: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-09T08:48:50.758 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:48:50.758 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:48:50.758 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 0, 2024-05-09T08:48:50.758 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 2, 2024-05-09T08:48:50.758 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 0, 2024-05-09T08:48:50.759 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 0, 2024-05-09T08:48:50.759 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 4, 2024-05-09T08:48:50.759 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 0, 2024-05-09T08:48:50.759 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.001274113, 2024-05-09T08:48:50.759 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.000000000, 2024-05-09T08:48:50.759 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.000349391, 2024-05-09T08:48:50.759 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T08:48:50.759 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:48:50.759 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:50.759 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:50.759 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:50.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:50.759 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:48:50.759 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:50.760 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:50.760 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:50.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:50.760 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T08:48:50.760 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T08:48:50.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:50.760 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:48:50.760 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 299, 2024-05-09T08:48:50.760 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 288, 2024-05-09T08:48:50.760 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 1602736, 2024-05-09T08:48:50.760 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 177504, 2024-05-09T08:48:50.760 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 2, 2024-05-09T08:48:50.760 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 1, 2024-05-09T08:48:50.760 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.049169279, 2024-05-09T08:48:50.761 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.016669302, 2024-05-09T08:48:50.761 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.035150995, 2024-05-09T08:48:50.761 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.005576613, 2024-05-09T08:48:50.761 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:48:50.761 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 287, 2024-05-09T08:48:50.761 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.015683448, 2024-05-09T08:48:50.761 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000054646 2024-05-09T08:48:50.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:50.761 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:48:50.761 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:50.761 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:50.761 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:50.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:50.761 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 1602736, 2024-05-09T08:48:50.761 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 177504 2024-05-09T08:48:50.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:50.762 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:48:50.762 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 7, 2024-05-09T08:48:50.762 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 223, 2024-05-09T08:48:50.762 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 3520, 2024-05-09T08:48:50.762 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 272864, 2024-05-09T08:48:50.762 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 6, 2024-05-09T08:48:50.762 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 3, 2024-05-09T08:48:50.762 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.015538397, 2024-05-09T08:48:50.762 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.010238265, 2024-05-09T08:48:50.762 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.002662514, 2024-05-09T08:48:50.762 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T08:48:50.762 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:48:50.763 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 221, 2024-05-09T08:48:50.763 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.022659295, 2024-05-09T08:48:50.763 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000102530 2024-05-09T08:48:50.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:50.763 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:48:50.763 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:50.763 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:50.763 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:50.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:50.763 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 3520, 2024-05-09T08:48:50.763 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 272864 2024-05-09T08:48:50.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:50.763 INFO:teuthology.orchestra.run.smithi107.stdout: "cct": { 2024-05-09T08:48:50.763 INFO:teuthology.orchestra.run.smithi107.stdout: "total_workers": 0, 2024-05-09T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stdout: "unhealthy_workers": 0 2024-05-09T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stdout: "mempool": { 2024-05-09T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_items": 0, 2024-05-09T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:48:50.764 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_items": 0, 2024-05-09T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_items": 0, 2024-05-09T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:48:50.765 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_items": 0, 2024-05-09T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_items": 0, 2024-05-09T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_bytes": 0, 2024-05-09T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_items": 0, 2024-05-09T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_bytes": 147970, 2024-05-09T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_items": 29, 2024-05-09T08:48:50.766 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_items": 0, 2024-05-09T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bytes": 0, 2024-05-09T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_items": 0, 2024-05-09T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_items": 0, 2024-05-09T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_items": 0, 2024-05-09T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_bytes": 0, 2024-05-09T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_items": 0, 2024-05-09T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_bytes": 0, 2024-05-09T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_items": 0, 2024-05-09T08:48:50.767 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_bytes": 928, 2024-05-09T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_items": 2, 2024-05-09T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_bytes": 0, 2024-05-09T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_items": 0, 2024-05-09T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_bytes": 0, 2024-05-09T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_items": 0 2024-05-09T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-05-09T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-05-09T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-05-09T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 302, 2024-05-09T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 1560575, 2024-05-09T08:48:50.768 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-05-09T08:48:50.769 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 302, 2024-05-09T08:48:50.769 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-05-09T08:48:50.769 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-05-09T08:48:50.769 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 302, 2024-05-09T08:48:50.769 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 1560575, 2024-05-09T08:48:50.769 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-05-09T08:48:50.769 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:50.769 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:50.769 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:50.769 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:50.769 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:50.769 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:48:51.519 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-05-09T08:48:51.519 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-05-09T08:48:51.519 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-05-09T08:48:51.519 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-05-09T08:48:51.520 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-05-09T08:48:51.520 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-05-09T08:48:51.520 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-05-09T08:48:51.520 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-05-09T08:48:51.520 DEBUG:tasks.check_counter:Getting stats from d 2024-05-09T08:48:51.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph daemon mds.d perf dump 2024-05-09T08:48:51.800 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:52.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:52 smithi107 ceph-mon[30122]: pgmap v487: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-05-09T08:48:52.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:52 smithi157 ceph-mon[33614]: pgmap v487: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-05-09T08:48:52.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:52 smithi110 ceph-mon[33622]: pgmap v487: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-05-09T08:48:53.630 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:48:53.630 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:48:53.630 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 4, 2024-05-09T08:48:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 4, 2024-05-09T08:48:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 3040, 2024-05-09T08:48:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 448, 2024-05-09T08:48:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 4, 2024-05-09T08:48:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 0, 2024-05-09T08:48:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.001532242, 2024-05-09T08:48:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.000038221, 2024-05-09T08:48:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.000678195, 2024-05-09T08:48:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T08:48:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:48:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 3, 2024-05-09T08:48:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000662809, 2024-05-09T08:48:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000220936 2024-05-09T08:48:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:48:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 3040, 2024-05-09T08:48:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 448 2024-05-09T08:48:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:48:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 3, 2024-05-09T08:48:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 226, 2024-05-09T08:48:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 480, 2024-05-09T08:48:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 273296, 2024-05-09T08:48:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 6, 2024-05-09T08:48:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 3, 2024-05-09T08:48:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.015588018, 2024-05-09T08:48:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.010736946, 2024-05-09T08:48:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.002262090, 2024-05-09T08:48:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T08:48:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:48:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 223, 2024-05-09T08:48:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.021167512, 2024-05-09T08:48:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000094921 2024-05-09T08:48:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:48:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 480, 2024-05-09T08:48:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 273296 2024-05-09T08:48:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:48:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 306, 2024-05-09T08:48:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 295, 2024-05-09T08:48:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 1605264, 2024-05-09T08:48:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 181680, 2024-05-09T08:48:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 2, 2024-05-09T08:48:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 1, 2024-05-09T08:48:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.050139933, 2024-05-09T08:48:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.017296423, 2024-05-09T08:48:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.035766975, 2024-05-09T08:48:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.005769770, 2024-05-09T08:48:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:48:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 294, 2024-05-09T08:48:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.016221848, 2024-05-09T08:48:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000055176 2024-05-09T08:48:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:48:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 1605264, 2024-05-09T08:48:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 181680 2024-05-09T08:48:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "cct": { 2024-05-09T08:48:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "total_workers": 0, 2024-05-09T08:48:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "unhealthy_workers": 0 2024-05-09T08:48:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mempool": { 2024-05-09T08:48:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:48:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_items": 0, 2024-05-09T08:48:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:48:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:48:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:48:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:48:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:48:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:48:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:48:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:48:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:48:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:48:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:48:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:48:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:48:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_items": 0, 2024-05-09T08:48:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:48:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_items": 0, 2024-05-09T08:48:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:48:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:48:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:48:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:48:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:48:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:48:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:48:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_items": 0, 2024-05-09T08:48:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:48:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:48:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:48:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_items": 0, 2024-05-09T08:48:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_bytes": 0, 2024-05-09T08:48:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_items": 0, 2024-05-09T08:48:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:48:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:48:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:48:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:48:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_bytes": 148731, 2024-05-09T08:48:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_items": 31, 2024-05-09T08:48:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:48:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_items": 0, 2024-05-09T08:48:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bytes": 0, 2024-05-09T08:48:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_items": 0, 2024-05-09T08:48:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:48:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_items": 0, 2024-05-09T08:48:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:48:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_items": 0, 2024-05-09T08:48:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_bytes": 0, 2024-05-09T08:48:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_items": 0, 2024-05-09T08:48:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:48:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:48:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_bytes": 0, 2024-05-09T08:48:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_items": 0, 2024-05-09T08:48:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_bytes": 928, 2024-05-09T08:48:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_items": 2, 2024-05-09T08:48:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_bytes": 0, 2024-05-09T08:48:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_items": 0, 2024-05-09T08:48:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_bytes": 0, 2024-05-09T08:48:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_items": 0 2024-05-09T08:48:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:48:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-05-09T08:48:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-05-09T08:48:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-05-09T08:48:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 309, 2024-05-09T08:48:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 1562122, 2024-05-09T08:48:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-05-09T08:48:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 309, 2024-05-09T08:48:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-05-09T08:48:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-05-09T08:48:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 309, 2024-05-09T08:48:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 1562122, 2024-05-09T08:48:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-05-09T08:48:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:53.641 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:48:54.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:54 smithi107 ceph-mon[30122]: pgmap v488: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-05-09T08:48:54.334 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-05-09T08:48:54.335 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-05-09T08:48:54.335 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-05-09T08:48:54.335 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-05-09T08:48:54.335 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-05-09T08:48:54.335 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-05-09T08:48:54.335 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-05-09T08:48:54.335 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-05-09T08:48:54.335 DEBUG:tasks.check_counter:Getting stats from g 2024-05-09T08:48:54.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph daemon mds.g perf dump 2024-05-09T08:48:54.578 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:54.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:54 smithi157 ceph-mon[33614]: pgmap v488: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-05-09T08:48:54.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:54 smithi110 ceph-mon[33622]: pgmap v488: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-05-09T08:48:56.341 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:48:56.341 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:48:56.341 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 4, 2024-05-09T08:48:56.341 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 4, 2024-05-09T08:48:56.342 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 3040, 2024-05-09T08:48:56.342 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 448, 2024-05-09T08:48:56.342 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 4, 2024-05-09T08:48:56.342 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 0, 2024-05-09T08:48:56.342 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.001464877, 2024-05-09T08:48:56.342 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.000033761, 2024-05-09T08:48:56.342 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.000595817, 2024-05-09T08:48:56.342 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T08:48:56.342 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:48:56.342 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 3, 2024-05-09T08:48:56.342 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000659926, 2024-05-09T08:48:56.342 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000219975 2024-05-09T08:48:56.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:56.343 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:48:56.343 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:56.343 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:56.343 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:56.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:56.343 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 3040, 2024-05-09T08:48:56.343 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 448 2024-05-09T08:48:56.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:56.343 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:48:56.343 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 3, 2024-05-09T08:48:56.343 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 222, 2024-05-09T08:48:56.343 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 480, 2024-05-09T08:48:56.344 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 272592, 2024-05-09T08:48:56.344 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 6, 2024-05-09T08:48:56.344 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 3, 2024-05-09T08:48:56.344 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.016259433, 2024-05-09T08:48:56.344 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.011575850, 2024-05-09T08:48:56.344 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.002224372, 2024-05-09T08:48:56.344 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T08:48:56.344 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:48:56.344 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 219, 2024-05-09T08:48:56.344 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.021115057, 2024-05-09T08:48:56.344 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000096415 2024-05-09T08:48:56.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:56.345 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:48:56.345 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:56.345 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:56.345 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:56.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:56.345 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 480, 2024-05-09T08:48:56.345 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 272592 2024-05-09T08:48:56.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:56.345 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:48:56.345 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 302, 2024-05-09T08:48:56.345 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 291, 2024-05-09T08:48:56.346 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 1604176, 2024-05-09T08:48:56.346 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 179120, 2024-05-09T08:48:56.346 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 2, 2024-05-09T08:48:56.346 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 1, 2024-05-09T08:48:56.346 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.050443575, 2024-05-09T08:48:56.346 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.015651610, 2024-05-09T08:48:56.346 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.038239675, 2024-05-09T08:48:56.346 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.005831373, 2024-05-09T08:48:56.346 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:48:56.346 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 290, 2024-05-09T08:48:56.346 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.016183311, 2024-05-09T08:48:56.346 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000055804 2024-05-09T08:48:56.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:56.347 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:48:56.347 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:56.347 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:56.347 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:56.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:56.347 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 1604176, 2024-05-09T08:48:56.347 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 179120 2024-05-09T08:48:56.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:56.347 INFO:teuthology.orchestra.run.smithi107.stdout: "cct": { 2024-05-09T08:48:56.347 INFO:teuthology.orchestra.run.smithi107.stdout: "total_workers": 0, 2024-05-09T08:48:56.347 INFO:teuthology.orchestra.run.smithi107.stdout: "unhealthy_workers": 0 2024-05-09T08:48:56.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:56.347 INFO:teuthology.orchestra.run.smithi107.stdout: "mempool": { 2024-05-09T08:48:56.348 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:48:56.348 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_items": 0, 2024-05-09T08:48:56.348 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:48:56.348 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:48:56.348 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:48:56.348 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:48:56.349 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:48:56.349 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:48:56.349 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:48:56.349 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:48:56.349 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:48:56.349 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:48:56.349 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:48:56.349 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:48:56.349 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:48:56.349 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_items": 0, 2024-05-09T08:48:56.349 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:48:56.350 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_items": 0, 2024-05-09T08:48:56.350 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:48:56.350 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:48:56.350 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:48:56.350 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:48:56.350 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:48:56.350 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:48:56.350 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:48:56.350 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_items": 0, 2024-05-09T08:48:56.350 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:48:56.350 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:48:56.351 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:48:56.351 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_items": 0, 2024-05-09T08:48:56.351 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_bytes": 0, 2024-05-09T08:48:56.351 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_items": 0, 2024-05-09T08:48:56.351 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:48:56.351 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:48:56.351 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:48:56.352 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:48:56.352 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_bytes": 149745, 2024-05-09T08:48:56.352 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_items": 31, 2024-05-09T08:48:56.352 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:48:56.352 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_items": 0, 2024-05-09T08:48:56.352 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bytes": 0, 2024-05-09T08:48:56.352 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_items": 0, 2024-05-09T08:48:56.352 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:48:56.353 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_items": 0, 2024-05-09T08:48:56.353 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:48:56.353 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_items": 0, 2024-05-09T08:48:56.353 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_bytes": 0, 2024-05-09T08:48:56.353 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_items": 0, 2024-05-09T08:48:56.353 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:48:56.353 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:48:56.353 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_bytes": 0, 2024-05-09T08:48:56.353 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_items": 0, 2024-05-09T08:48:56.354 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_bytes": 928, 2024-05-09T08:48:56.355 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_items": 2, 2024-05-09T08:48:56.355 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_bytes": 0, 2024-05-09T08:48:56.355 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_items": 0, 2024-05-09T08:48:56.355 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_bytes": 0, 2024-05-09T08:48:56.355 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_items": 0 2024-05-09T08:48:56.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:56.355 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:48:56.356 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-05-09T08:48:56.356 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-05-09T08:48:56.356 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-05-09T08:48:56.356 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 305, 2024-05-09T08:48:56.356 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 1561602, 2024-05-09T08:48:56.356 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-05-09T08:48:56.356 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 305, 2024-05-09T08:48:56.356 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-05-09T08:48:56.356 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-05-09T08:48:56.356 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 305, 2024-05-09T08:48:56.356 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 1561602, 2024-05-09T08:48:56.356 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-05-09T08:48:56.357 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:56.357 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:56.357 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:56.357 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:56.357 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:56.357 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:48:56.357 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:56 smithi107 ceph-mon[30122]: pgmap v489: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-09T08:48:56.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:56 smithi157 ceph-mon[33614]: pgmap v489: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-09T08:48:56.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:56 smithi110 ceph-mon[33622]: pgmap v489: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-09T08:48:57.066 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-05-09T08:48:57.067 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-05-09T08:48:57.067 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-05-09T08:48:57.067 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-05-09T08:48:57.067 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-05-09T08:48:57.067 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-05-09T08:48:57.067 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-05-09T08:48:57.067 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-05-09T08:48:57.067 DEBUG:tasks.check_counter:Getting stats from j 2024-05-09T08:48:57.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph daemon mds.j perf dump 2024-05-09T08:48:57.316 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T08:48:58.503 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:48:58 smithi107 ceph-mon[30122]: pgmap v490: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-09T08:48:58.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:48:58 smithi157 ceph-mon[33614]: pgmap v490: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-09T08:48:58.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:48:58 smithi110 ceph-mon[33622]: pgmap v490: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-09T08:48:59.110 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-05-09T08:48:59.110 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:48:59.110 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 0, 2024-05-09T08:48:59.110 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 2, 2024-05-09T08:48:59.111 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 0, 2024-05-09T08:48:59.111 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 0, 2024-05-09T08:48:59.111 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 4, 2024-05-09T08:48:59.111 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 0, 2024-05-09T08:48:59.111 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.001291143, 2024-05-09T08:48:59.111 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.000000000, 2024-05-09T08:48:59.111 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.000459730, 2024-05-09T08:48:59.111 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T08:48:59.111 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:48:59.111 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:59.111 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:59.111 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:59.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:59.112 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:48:59.112 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:59.112 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:59.112 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:59.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:59.112 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T08:48:59.112 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T08:48:59.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:59.112 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:48:59.112 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 3, 2024-05-09T08:48:59.112 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 227, 2024-05-09T08:48:59.112 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 480, 2024-05-09T08:48:59.112 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 273472, 2024-05-09T08:48:59.113 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 6, 2024-05-09T08:48:59.113 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 3, 2024-05-09T08:48:59.113 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.015887767, 2024-05-09T08:48:59.113 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.010621795, 2024-05-09T08:48:59.113 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.002322167, 2024-05-09T08:48:59.113 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T08:48:59.113 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:48:59.113 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 224, 2024-05-09T08:48:59.113 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.026290139, 2024-05-09T08:48:59.113 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000117366 2024-05-09T08:48:59.113 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:59.113 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:48:59.113 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:59.114 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:59.114 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:59.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:59.114 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 480, 2024-05-09T08:48:59.114 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 273472 2024-05-09T08:48:59.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:59.114 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:48:59.114 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 312, 2024-05-09T08:48:59.114 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 299, 2024-05-09T08:48:59.114 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 1608848, 2024-05-09T08:48:59.114 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 183408, 2024-05-09T08:48:59.115 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 2, 2024-05-09T08:48:59.115 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 1, 2024-05-09T08:48:59.115 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.051966683, 2024-05-09T08:48:59.115 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.015565081, 2024-05-09T08:48:59.115 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.040543496, 2024-05-09T08:48:59.115 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.005854045, 2024-05-09T08:48:59.115 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:48:59.115 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 299, 2024-05-09T08:48:59.115 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.013321922, 2024-05-09T08:48:59.115 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000044554 2024-05-09T08:48:59.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:59.115 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:48:59.115 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:59.116 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:59.116 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:59.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:59.116 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 1608848, 2024-05-09T08:48:59.116 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 183408 2024-05-09T08:48:59.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:59.116 INFO:teuthology.orchestra.run.smithi107.stdout: "cct": { 2024-05-09T08:48:59.116 INFO:teuthology.orchestra.run.smithi107.stdout: "total_workers": 0, 2024-05-09T08:48:59.117 INFO:teuthology.orchestra.run.smithi107.stdout: "unhealthy_workers": 0 2024-05-09T08:48:59.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:59.117 INFO:teuthology.orchestra.run.smithi107.stdout: "mempool": { 2024-05-09T08:48:59.117 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:48:59.117 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_items": 0, 2024-05-09T08:48:59.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:48:59.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:48:59.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:48:59.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:48:59.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:48:59.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:48:59.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:48:59.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:48:59.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:48:59.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:48:59.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:48:59.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:48:59.118 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:48:59.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_items": 0, 2024-05-09T08:48:59.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:48:59.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_items": 0, 2024-05-09T08:48:59.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:48:59.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:48:59.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:48:59.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:48:59.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:48:59.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:48:59.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:48:59.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_items": 0, 2024-05-09T08:48:59.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:48:59.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:48:59.119 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:48:59.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_items": 0, 2024-05-09T08:48:59.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_bytes": 0, 2024-05-09T08:48:59.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_items": 0, 2024-05-09T08:48:59.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:48:59.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:48:59.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:48:59.120 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:48:59.120 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_bytes": 148794, 2024-05-09T08:48:59.120 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_items": 32, 2024-05-09T08:48:59.120 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:48:59.120 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_items": 0, 2024-05-09T08:48:59.120 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bytes": 0, 2024-05-09T08:48:59.120 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_items": 0, 2024-05-09T08:48:59.121 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:48:59.121 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_items": 0, 2024-05-09T08:48:59.121 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:48:59.121 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_items": 0, 2024-05-09T08:48:59.121 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_bytes": 0, 2024-05-09T08:48:59.121 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_items": 0, 2024-05-09T08:48:59.121 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:48:59.121 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:48:59.121 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_bytes": 0, 2024-05-09T08:48:59.121 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_items": 0, 2024-05-09T08:48:59.121 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_bytes": 928, 2024-05-09T08:48:59.121 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_items": 2, 2024-05-09T08:48:59.121 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_bytes": 0, 2024-05-09T08:48:59.121 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_items": 0, 2024-05-09T08:48:59.122 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_bytes": 0, 2024-05-09T08:48:59.122 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_items": 0 2024-05-09T08:48:59.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-05-09T08:48:59.122 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:48:59.122 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-05-09T08:48:59.122 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-05-09T08:48:59.122 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-05-09T08:48:59.122 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 311, 2024-05-09T08:48:59.122 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 1562382, 2024-05-09T08:48:59.122 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-05-09T08:48:59.122 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 311, 2024-05-09T08:48:59.122 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-05-09T08:48:59.122 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-05-09T08:48:59.122 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 311, 2024-05-09T08:48:59.123 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 1562382, 2024-05-09T08:48:59.123 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-05-09T08:48:59.123 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-05-09T08:48:59.123 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-05-09T08:48:59.123 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-05-09T08:48:59.123 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:59.123 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-05-09T08:48:59.123 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-05-09T08:48:59.851 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-05-09T08:48:59.851 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-05-09T08:48:59.851 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-05-09T08:48:59.851 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-05-09T08:48:59.851 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-05-09T08:48:59.851 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-05-09T08:48:59.851 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-05-09T08:48:59.851 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-05-09T08:48:59.851 DEBUG:tasks.check_counter:Getting stats from b 2024-05-09T08:48:59.852 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph daemon mds.b perf dump 2024-05-09T08:49:00.143 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:49:00.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:00 smithi107 ceph-mon[30122]: pgmap v491: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-05-09T08:49:00.567 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:00 smithi110 ceph-mon[33622]: pgmap v491: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-05-09T08:49:00.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:00 smithi157 ceph-mon[33614]: pgmap v491: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-05-09T08:49:02.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:02 smithi107 ceph-mon[30122]: pgmap v492: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:49:02.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:02 smithi157 ceph-mon[33614]: pgmap v492: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:49:02.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:02 smithi110 ceph-mon[33622]: pgmap v492: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:49:02.717 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-05-09T08:49:02.718 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:49:02.718 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 5557, 2024-05-09T08:49:02.718 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 3210, 2024-05-09T08:49:02.718 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 1990700, 2024-05-09T08:49:02.718 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 1386059, 2024-05-09T08:49:02.718 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 10, 2024-05-09T08:49:02.718 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 4, 2024-05-09T08:49:02.718 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.274100270, 2024-05-09T08:49:02.719 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.102014987, 2024-05-09T08:49:02.719 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.219717117, 2024-05-09T08:49:02.719 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.009432805, 2024-05-09T08:49:02.719 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:02.719 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 3208, 2024-05-09T08:49:02.719 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.117112665, 2024-05-09T08:49:02.719 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000036506 2024-05-09T08:49:02.719 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.720 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:02.720 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 8236, 2024-05-09T08:49:02.720 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.001300334, 2024-05-09T08:49:02.720 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000157 2024-05-09T08:49:02.720 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.720 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-09T08:49:02.720 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 514816 2024-05-09T08:49:02.721 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.721 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:49:02.721 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 475, 2024-05-09T08:49:02.721 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 519, 2024-05-09T08:49:02.721 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 1698466, 2024-05-09T08:49:02.721 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 350196, 2024-05-09T08:49:02.721 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 18, 2024-05-09T08:49:02.721 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 6, 2024-05-09T08:49:02.721 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.071066273, 2024-05-09T08:49:02.721 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.023671034, 2024-05-09T08:49:02.721 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.049911504, 2024-05-09T08:49:02.721 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.007159433, 2024-05-09T08:49:02.721 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:02.721 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 518, 2024-05-09T08:49:02.722 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.036133224, 2024-05-09T08:49:02.722 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000069755 2024-05-09T08:49:02.722 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.722 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:02.722 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 130, 2024-05-09T08:49:02.722 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000026533, 2024-05-09T08:49:02.722 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000204 2024-05-09T08:49:02.722 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.722 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 1686640, 2024-05-09T08:49:02.722 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 223280 2024-05-09T08:49:02.722 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.723 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:49:02.723 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 164, 2024-05-09T08:49:02.723 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 182, 2024-05-09T08:49:02.723 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 51727, 2024-05-09T08:49:02.723 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 716551, 2024-05-09T08:49:02.723 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 59, 2024-05-09T08:49:02.723 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 5, 2024-05-09T08:49:02.723 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.032031278, 2024-05-09T08:49:02.723 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.004670040, 2024-05-09T08:49:02.723 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.023140077, 2024-05-09T08:49:02.723 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.002168001, 2024-05-09T08:49:02.723 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:02.723 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 180, 2024-05-09T08:49:02.724 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.031654556, 2024-05-09T08:49:02.724 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000175858 2024-05-09T08:49:02.724 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.724 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:02.724 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 120, 2024-05-09T08:49:02.724 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000015891, 2024-05-09T08:49:02.724 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000132 2024-05-09T08:49:02.724 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.724 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 5056, 2024-05-09T08:49:02.724 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 131968 2024-05-09T08:49:02.724 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.724 INFO:teuthology.orchestra.run.smithi110.stdout: "cct": { 2024-05-09T08:49:02.724 INFO:teuthology.orchestra.run.smithi110.stdout: "total_workers": 1, 2024-05-09T08:49:02.725 INFO:teuthology.orchestra.run.smithi110.stdout: "unhealthy_workers": 0 2024-05-09T08:49:02.725 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.725 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-MDSRank": { 2024-05-09T08:49:02.725 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-05-09T08:49:02.725 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-05-09T08:49:02.725 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 160, 2024-05-09T08:49:02.725 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.016270634, 2024-05-09T08:49:02.725 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000101691 2024-05-09T08:49:02.725 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:02.725 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.725 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-PurgeQueue": { 2024-05-09T08:49:02.725 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-05-09T08:49:02.725 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-05-09T08:49:02.726 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 28, 2024-05-09T08:49:02.726 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000996281, 2024-05-09T08:49:02.726 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000035581 2024-05-09T08:49:02.726 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:02.726 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.726 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": { 2024-05-09T08:49:02.726 INFO:teuthology.orchestra.run.smithi110.stdout: "request": 10, 2024-05-09T08:49:02.726 INFO:teuthology.orchestra.run.smithi110.stdout: "reply": 9, 2024-05-09T08:49:02.726 INFO:teuthology.orchestra.run.smithi110.stdout: "reply_latency": { 2024-05-09T08:49:02.726 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 9, 2024-05-09T08:49:02.727 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.013514519, 2024-05-09T08:49:02.727 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.001501613 2024-05-09T08:49:02.727 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.727 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_reply": 0, 2024-05-09T08:49:02.727 INFO:teuthology.orchestra.run.smithi110.stdout: "forward": 1, 2024-05-09T08:49:02.727 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_complete": 0, 2024-05-09T08:49:02.727 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_keys": 0, 2024-05-09T08:49:02.727 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_commit": 11, 2024-05-09T08:49:02.727 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_split": 0, 2024-05-09T08:49:02.727 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_merge": 0, 2024-05-09T08:49:02.727 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes": 18, 2024-05-09T08:49:02.728 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_top": 2, 2024-05-09T08:49:02.728 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_bottom": 6, 2024-05-09T08:49:02.728 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pin_tail": 10, 2024-05-09T08:49:02.728 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pinned": 14, 2024-05-09T08:49:02.728 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_expired": 177, 2024-05-09T08:49:02.728 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_with_caps": 3, 2024-05-09T08:49:02.729 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": 3, 2024-05-09T08:49:02.729 INFO:teuthology.orchestra.run.smithi110.stdout: "subtrees": 4, 2024-05-09T08:49:02.729 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse": 213, 2024-05-09T08:49:02.729 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_hit": 121, 2024-05-09T08:49:02.729 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_forward": 1, 2024-05-09T08:49:02.729 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_discover": 82, 2024-05-09T08:49:02.729 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_dir_fetch": 0, 2024-05-09T08:49:02.729 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_remote_ino": 0, 2024-05-09T08:49:02.729 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_lock": 0, 2024-05-09T08:49:02.729 INFO:teuthology.orchestra.run.smithi110.stdout: "load_cent": 0, 2024-05-09T08:49:02.729 INFO:teuthology.orchestra.run.smithi110.stdout: "q": 0, 2024-05-09T08:49:02.729 INFO:teuthology.orchestra.run.smithi110.stdout: "exported": 3, 2024-05-09T08:49:02.729 INFO:teuthology.orchestra.run.smithi110.stdout: "exported_inodes": 8, 2024-05-09T08:49:02.730 INFO:teuthology.orchestra.run.smithi110.stdout: "imported": 3, 2024-05-09T08:49:02.730 INFO:teuthology.orchestra.run.smithi110.stdout: "imported_inodes": 5, 2024-05-09T08:49:02.730 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_dir_fetch": 0, 2024-05-09T08:49:02.730 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_backtrace_fetch": 63, 2024-05-09T08:49:02.730 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_peer_discover": 0, 2024-05-09T08:49:02.730 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rfiles": 46, 2024-05-09T08:49:02.730 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rbytes": 729438, 2024-05-09T08:49:02.730 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rsnaps": 6, 2024-05-09T08:49:02.730 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_fetch": 63, 2024-05-09T08:49:02.730 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_set_tag": 0, 2024-05-09T08:49:02.730 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T08:49:02.730 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_inotable_repaired": 0, 2024-05-09T08:49:02.730 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_inodes": 0, 2024-05-09T08:49:02.731 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T08:49:02.731 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T08:49:02.731 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_file_inodes": 63, 2024-05-09T08:49:02.731 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_inode_file_caps": 0, 2024-05-09T08:49:02.731 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_revoke": 3, 2024-05-09T08:49:02.731 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_grant": 8, 2024-05-09T08:49:02.731 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T08:49:02.731 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T08:49:02.731 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T08:49:02.731 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps": 5, 2024-05-09T08:49:02.731 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps_dirty": 1, 2024-05-09T08:49:02.731 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_cap_release": 2, 2024-05-09T08:49:02.731 INFO:teuthology.orchestra.run.smithi110.stdout: "process_request_cap_release": 6 2024-05-09T08:49:02.731 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.731 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_cache": { 2024-05-09T08:49:02.732 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update": 3, 2024-05-09T08:49:02.732 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update_receipt": 290, 2024-05-09T08:49:02.732 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_try_discover": 185, 2024-05-09T08:49:02.732 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_send_discover": 189, 2024-05-09T08:49:02.732 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_handle_discover": 3, 2024-05-09T08:49:02.732 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays": 0, 2024-05-09T08:49:02.732 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_delayed": 0, 2024-05-09T08:49:02.732 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_enqueuing": 0, 2024-05-09T08:49:02.732 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_created": 3, 2024-05-09T08:49:02.732 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_enqueued": 3, 2024-05-09T08:49:02.732 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_reintegrated": 0, 2024-05-09T08:49:02.732 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_migrated": 0, 2024-05-09T08:49:02.732 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_processing": 0, 2024-05-09T08:49:02.733 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_enqueued": 0, 2024-05-09T08:49:02.733 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_prioritized": 0, 2024-05-09T08:49:02.733 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_started": 0, 2024-05-09T08:49:02.733 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_completed": 0, 2024-05-09T08:49:02.733 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_path": 0, 2024-05-09T08:49:02.733 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_inode": 0, 2024-05-09T08:49:02.733 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T08:49:02.733 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_exportdir": 3, 2024-05-09T08:49:02.733 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_flush": 0, 2024-05-09T08:49:02.733 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragmentdir": 0, 2024-05-09T08:49:02.733 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragstats": 0, 2024-05-09T08:49:02.733 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_inodestats": 0 2024-05-09T08:49:02.733 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.733 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_log": { 2024-05-09T08:49:02.734 INFO:teuthology.orchestra.run.smithi110.stdout: "evlrg": 0, 2024-05-09T08:49:02.734 INFO:teuthology.orchestra.run.smithi110.stdout: "evadd": 42, 2024-05-09T08:49:02.734 INFO:teuthology.orchestra.run.smithi110.stdout: "evex": 0, 2024-05-09T08:49:02.734 INFO:teuthology.orchestra.run.smithi110.stdout: "evtrm": 0, 2024-05-09T08:49:02.734 INFO:teuthology.orchestra.run.smithi110.stdout: "ev": 42, 2024-05-09T08:49:02.734 INFO:teuthology.orchestra.run.smithi110.stdout: "evexg": 0, 2024-05-09T08:49:02.734 INFO:teuthology.orchestra.run.smithi110.stdout: "evexd": 0, 2024-05-09T08:49:02.734 INFO:teuthology.orchestra.run.smithi110.stdout: "segadd": 2, 2024-05-09T08:49:02.734 INFO:teuthology.orchestra.run.smithi110.stdout: "segex": 0, 2024-05-09T08:49:02.734 INFO:teuthology.orchestra.run.smithi110.stdout: "segtrm": 0, 2024-05-09T08:49:02.734 INFO:teuthology.orchestra.run.smithi110.stdout: "seg": 2, 2024-05-09T08:49:02.734 INFO:teuthology.orchestra.run.smithi110.stdout: "segmjr": 2, 2024-05-09T08:49:02.734 INFO:teuthology.orchestra.run.smithi110.stdout: "segexg": 0, 2024-05-09T08:49:02.735 INFO:teuthology.orchestra.run.smithi110.stdout: "segexd": 0, 2024-05-09T08:49:02.735 INFO:teuthology.orchestra.run.smithi110.stdout: "expos": 4194304, 2024-05-09T08:49:02.735 INFO:teuthology.orchestra.run.smithi110.stdout: "wrpos": 4273152, 2024-05-09T08:49:02.735 INFO:teuthology.orchestra.run.smithi110.stdout: "rdpos": 0, 2024-05-09T08:49:02.735 INFO:teuthology.orchestra.run.smithi110.stdout: "jlat": { 2024-05-09T08:49:02.735 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 27, 2024-05-09T08:49:02.735 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.214136559, 2024-05-09T08:49:02.735 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.007930983 2024-05-09T08:49:02.735 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.735 INFO:teuthology.orchestra.run.smithi110.stdout: "replayed": 0 2024-05-09T08:49:02.735 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.735 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_mem": { 2024-05-09T08:49:02.736 INFO:teuthology.orchestra.run.smithi110.stdout: "ino": 24, 2024-05-09T08:49:02.736 INFO:teuthology.orchestra.run.smithi110.stdout: "ino+": 117, 2024-05-09T08:49:02.736 INFO:teuthology.orchestra.run.smithi110.stdout: "ino-": 93, 2024-05-09T08:49:02.736 INFO:teuthology.orchestra.run.smithi110.stdout: "dir": 22, 2024-05-09T08:49:02.736 INFO:teuthology.orchestra.run.smithi110.stdout: "dir+": 134, 2024-05-09T08:49:02.736 INFO:teuthology.orchestra.run.smithi110.stdout: "dir-": 112, 2024-05-09T08:49:02.736 INFO:teuthology.orchestra.run.smithi110.stdout: "dn": 19, 2024-05-09T08:49:02.736 INFO:teuthology.orchestra.run.smithi110.stdout: "dn+": 199, 2024-05-09T08:49:02.736 INFO:teuthology.orchestra.run.smithi110.stdout: "dn-": 180, 2024-05-09T08:49:02.736 INFO:teuthology.orchestra.run.smithi110.stdout: "cap": 3, 2024-05-09T08:49:02.736 INFO:teuthology.orchestra.run.smithi110.stdout: "cap+": 9, 2024-05-09T08:49:02.736 INFO:teuthology.orchestra.run.smithi110.stdout: "cap-": 6, 2024-05-09T08:49:02.736 INFO:teuthology.orchestra.run.smithi110.stdout: "rss": 57280, 2024-05-09T08:49:02.736 INFO:teuthology.orchestra.run.smithi110.stdout: "heap": 223612 2024-05-09T08:49:02.737 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.737 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_server": { 2024-05-09T08:49:02.737 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_client_request": 19, 2024-05-09T08:49:02.737 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_server_request": 8, 2024-05-09T08:49:02.737 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_request": 10, 2024-05-09T08:49:02.737 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_session": 24, 2024-05-09T08:49:02.737 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_peer_request": 22, 2024-05-09T08:49:02.737 INFO:teuthology.orchestra.run.smithi110.stdout: "req_create_latency": { 2024-05-09T08:49:02.737 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.737 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.737 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.737 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.737 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getattr_latency": { 2024-05-09T08:49:02.738 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 3, 2024-05-09T08:49:02.738 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000669391, 2024-05-09T08:49:02.738 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000223130 2024-05-09T08:49:02.738 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.738 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getfilelock_latency": { 2024-05-09T08:49:02.738 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.738 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.738 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.738 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.738 INFO:teuthology.orchestra.run.smithi110.stdout: "req_link_latency": { 2024-05-09T08:49:02.738 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.738 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.738 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.738 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.739 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookup_latency": { 2024-05-09T08:49:02.739 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.739 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.739 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.739 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.739 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookuphash_latency": { 2024-05-09T08:49:02.739 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.739 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.739 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.739 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.739 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupino_latency": { 2024-05-09T08:49:02.739 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.739 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.739 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.740 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.740 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupname_latency": { 2024-05-09T08:49:02.740 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.740 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.740 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.740 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.740 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupparent_latency": { 2024-05-09T08:49:02.740 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.740 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.740 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.740 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.740 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupsnap_latency": { 2024-05-09T08:49:02.740 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.740 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.740 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.740 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.741 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lssnap_latency": { 2024-05-09T08:49:02.741 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.741 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.741 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.742 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.743 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mkdir_latency": { 2024-05-09T08:49:02.743 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.743 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.743 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.744 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.744 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mknod_latency": { 2024-05-09T08:49:02.744 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.744 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.744 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.744 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.744 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mksnap_latency": { 2024-05-09T08:49:02.744 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.744 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.744 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.744 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.744 INFO:teuthology.orchestra.run.smithi110.stdout: "req_open_latency": { 2024-05-09T08:49:02.744 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.745 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.745 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.745 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.745 INFO:teuthology.orchestra.run.smithi110.stdout: "req_readdir_latency": { 2024-05-09T08:49:02.745 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 3, 2024-05-09T08:49:02.745 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.001060438, 2024-05-09T08:49:02.745 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000353479 2024-05-09T08:49:02.745 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.745 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rename_latency": { 2024-05-09T08:49:02.746 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.746 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.746 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.746 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.746 INFO:teuthology.orchestra.run.smithi110.stdout: "req_renamesnap_latency": { 2024-05-09T08:49:02.746 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.746 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.746 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.746 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.746 INFO:teuthology.orchestra.run.smithi110.stdout: "req_snapdiff_latency": { 2024-05-09T08:49:02.746 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.746 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.746 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.747 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.747 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmdir_latency": { 2024-05-09T08:49:02.747 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.747 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.747 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.747 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.747 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmsnap_latency": { 2024-05-09T08:49:02.747 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.747 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.747 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.747 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.747 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmxattr_latency": { 2024-05-09T08:49:02.747 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.747 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.748 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.748 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.748 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setattr_latency": { 2024-05-09T08:49:02.748 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.748 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.748 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.748 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.748 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setdirlayout_latency": { 2024-05-09T08:49:02.748 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.748 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.748 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.748 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.748 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setfilelock_latency": { 2024-05-09T08:49:02.748 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.748 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.749 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.749 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.749 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setlayout_latency": { 2024-05-09T08:49:02.749 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.749 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.749 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.749 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.749 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setxattr_latency": { 2024-05-09T08:49:02.749 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.749 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.749 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.749 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.749 INFO:teuthology.orchestra.run.smithi110.stdout: "req_symlink_latency": { 2024-05-09T08:49:02.749 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.750 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.750 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.750 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.750 INFO:teuthology.orchestra.run.smithi110.stdout: "req_unlink_latency": { 2024-05-09T08:49:02.750 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 3, 2024-05-09T08:49:02.750 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.011784690, 2024-05-09T08:49:02.750 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.003928230 2024-05-09T08:49:02.750 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.750 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_revoke_eviction": 0, 2024-05-09T08:49:02.750 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_acquisition_throttle": 0, 2024-05-09T08:49:02.750 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getvxattr_latency": { 2024-05-09T08:49:02.750 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.750 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.751 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.751 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:02.751 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.751 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_sessions": { 2024-05-09T08:49:02.751 INFO:teuthology.orchestra.run.smithi110.stdout: "session_count": 1, 2024-05-09T08:49:02.751 INFO:teuthology.orchestra.run.smithi110.stdout: "session_add": 4, 2024-05-09T08:49:02.751 INFO:teuthology.orchestra.run.smithi110.stdout: "session_remove": 3, 2024-05-09T08:49:02.751 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_open": 1, 2024-05-09T08:49:02.751 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_stale": 0, 2024-05-09T08:49:02.751 INFO:teuthology.orchestra.run.smithi110.stdout: "total_load": 2, 2024-05-09T08:49:02.751 INFO:teuthology.orchestra.run.smithi110.stdout: "average_load": 2, 2024-05-09T08:49:02.751 INFO:teuthology.orchestra.run.smithi110.stdout: "avg_session_uptime": 941, 2024-05-09T08:49:02.751 INFO:teuthology.orchestra.run.smithi110.stdout: "mdthresh_evicted": 0 2024-05-09T08:49:02.751 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.752 INFO:teuthology.orchestra.run.smithi110.stdout: "mempool": { 2024-05-09T08:49:02.752 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:49:02.752 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_items": 0, 2024-05-09T08:49:02.752 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:49:02.752 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:49:02.752 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:49:02.752 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:49:02.752 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:49:02.752 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:49:02.752 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:49:02.752 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:49:02.752 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:49:02.752 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:49:02.752 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:49:02.752 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:49:02.753 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:49:02.753 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_items": 0, 2024-05-09T08:49:02.753 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:49:02.753 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_items": 0, 2024-05-09T08:49:02.753 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:49:02.753 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:49:02.753 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:49:02.753 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:49:02.753 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:49:02.753 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:49:02.753 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:49:02.753 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_items": 0, 2024-05-09T08:49:02.753 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:49:02.753 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:49:02.754 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:49:02.754 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_items": 0, 2024-05-09T08:49:02.754 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_bytes": 0, 2024-05-09T08:49:02.754 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_items": 0, 2024-05-09T08:49:02.754 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:49:02.754 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:49:02.754 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:49:02.754 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:49:02.754 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_bytes": 361081, 2024-05-09T08:49:02.754 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_items": 119, 2024-05-09T08:49:02.754 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:49:02.754 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_items": 0, 2024-05-09T08:49:02.755 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_bytes": 0, 2024-05-09T08:49:02.755 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_items": 0, 2024-05-09T08:49:02.755 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:49:02.755 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_items": 0, 2024-05-09T08:49:02.755 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:49:02.755 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_items": 0, 2024-05-09T08:49:02.755 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_bytes": 9416, 2024-05-09T08:49:02.755 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_items": 180, 2024-05-09T08:49:02.755 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:49:02.755 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:49:02.755 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_bytes": 0, 2024-05-09T08:49:02.755 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_items": 0, 2024-05-09T08:49:02.755 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_bytes": 102589, 2024-05-09T08:49:02.755 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_items": 429, 2024-05-09T08:49:02.756 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_bytes": 0, 2024-05-09T08:49:02.756 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_items": 0, 2024-05-09T08:49:02.756 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_bytes": 0, 2024-05-09T08:49:02.756 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_items": 0 2024-05-09T08:49:02.756 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.756 INFO:teuthology.orchestra.run.smithi110.stdout: "objecter": { 2024-05-09T08:49:02.756 INFO:teuthology.orchestra.run.smithi110.stdout: "op_active": 0, 2024-05-09T08:49:02.756 INFO:teuthology.orchestra.run.smithi110.stdout: "op_laggy": 0, 2024-05-09T08:49:02.756 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send": 156, 2024-05-09T08:49:02.757 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send_bytes": 92545, 2024-05-09T08:49:02.757 INFO:teuthology.orchestra.run.smithi110.stdout: "op_resend": 33, 2024-05-09T08:49:02.757 INFO:teuthology.orchestra.run.smithi110.stdout: "op_reply": 156, 2024-05-09T08:49:02.757 INFO:teuthology.orchestra.run.smithi110.stdout: "op_latency": { 2024-05-09T08:49:02.757 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 156, 2024-05-09T08:49:02.757 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 1.489976983, 2024-05-09T08:49:02.757 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.009551134 2024-05-09T08:49:02.757 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.757 INFO:teuthology.orchestra.run.smithi110.stdout: "op_inflight": 0, 2024-05-09T08:49:02.757 INFO:teuthology.orchestra.run.smithi110.stdout: "oplen_avg": { 2024-05-09T08:49:02.757 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 156, 2024-05-09T08:49:02.757 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 188 2024-05-09T08:49:02.757 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.758 INFO:teuthology.orchestra.run.smithi110.stdout: "op": 156, 2024-05-09T08:49:02.758 INFO:teuthology.orchestra.run.smithi110.stdout: "op_r": 66, 2024-05-09T08:49:02.758 INFO:teuthology.orchestra.run.smithi110.stdout: "op_w": 90, 2024-05-09T08:49:02.758 INFO:teuthology.orchestra.run.smithi110.stdout: "op_rmw": 0, 2024-05-09T08:49:02.758 INFO:teuthology.orchestra.run.smithi110.stdout: "op_pg": 0, 2024-05-09T08:49:02.758 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_stat": 0, 2024-05-09T08:49:02.758 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_create": 10, 2024-05-09T08:49:02.758 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_read": 3, 2024-05-09T08:49:02.758 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_write": 30, 2024-05-09T08:49:02.758 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writefull": 12, 2024-05-09T08:49:02.758 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writesame": 0, 2024-05-09T08:49:02.758 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_append": 0, 2024-05-09T08:49:02.758 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_zero": 0, 2024-05-09T08:49:02.759 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_truncate": 0, 2024-05-09T08:49:02.759 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_delete": 18, 2024-05-09T08:49:02.759 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_mapext": 0, 2024-05-09T08:49:02.759 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_sparse_read": 0, 2024-05-09T08:49:02.759 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_clonerange": 0, 2024-05-09T08:49:02.759 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_getxattr": 63, 2024-05-09T08:49:02.759 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_setxattr": 20, 2024-05-09T08:49:02.759 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_cmpxattr": 0, 2024-05-09T08:49:02.759 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_rmxattr": 0, 2024-05-09T08:49:02.759 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_resetxattrs": 0, 2024-05-09T08:49:02.759 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_call": 0, 2024-05-09T08:49:02.759 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_watch": 0, 2024-05-09T08:49:02.759 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_notify": 0, 2024-05-09T08:49:02.759 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T08:49:02.759 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls": 0, 2024-05-09T08:49:02.760 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls_filter": 0, 2024-05-09T08:49:02.760 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_other": 0, 2024-05-09T08:49:02.760 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_active": 0, 2024-05-09T08:49:02.760 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_send": 0, 2024-05-09T08:49:02.760 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_resend": 0, 2024-05-09T08:49:02.760 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_ping": 0, 2024-05-09T08:49:02.760 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_active": 0, 2024-05-09T08:49:02.760 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_send": 0, 2024-05-09T08:49:02.760 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_resend": 0, 2024-05-09T08:49:02.760 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_active": 0, 2024-05-09T08:49:02.760 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_send": 0, 2024-05-09T08:49:02.760 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_resend": 0, 2024-05-09T08:49:02.760 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_active": 0, 2024-05-09T08:49:02.761 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_send": 0, 2024-05-09T08:49:02.761 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_resend": 0, 2024-05-09T08:49:02.761 INFO:teuthology.orchestra.run.smithi110.stdout: "command_active": 0, 2024-05-09T08:49:02.761 INFO:teuthology.orchestra.run.smithi110.stdout: "command_send": 0, 2024-05-09T08:49:02.761 INFO:teuthology.orchestra.run.smithi110.stdout: "command_resend": 0, 2024-05-09T08:49:02.761 INFO:teuthology.orchestra.run.smithi110.stdout: "map_epoch": 113, 2024-05-09T08:49:02.761 INFO:teuthology.orchestra.run.smithi110.stdout: "map_full": 0, 2024-05-09T08:49:02.762 INFO:teuthology.orchestra.run.smithi110.stdout: "map_inc": 24, 2024-05-09T08:49:02.762 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_sessions": 10, 2024-05-09T08:49:02.762 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_open": 14, 2024-05-09T08:49:02.762 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_close": 4, 2024-05-09T08:49:02.762 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_laggy": 0, 2024-05-09T08:49:02.762 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_wr": 26, 2024-05-09T08:49:02.762 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_rd": 0, 2024-05-09T08:49:02.762 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_del": 6 2024-05-09T08:49:02.762 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.763 INFO:teuthology.orchestra.run.smithi110.stdout: "oft": { 2024-05-09T08:49:02.763 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_objs": 1, 2024-05-09T08:49:02.763 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_kv_pairs": 0, 2024-05-09T08:49:02.763 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_updates": 0, 2024-05-09T08:49:02.763 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_removes": 0 2024-05-09T08:49:02.763 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.764 INFO:teuthology.orchestra.run.smithi110.stdout: "osd-slow-ops": { 2024-05-09T08:49:02.764 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_ops_count": 0 2024-05-09T08:49:02.764 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.764 INFO:teuthology.orchestra.run.smithi110.stdout: "purge_queue": { 2024-05-09T08:49:02.764 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops": 0, 2024-05-09T08:49:02.764 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops_high_water": 4, 2024-05-09T08:49:02.764 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing": 0, 2024-05-09T08:49:02.764 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_high_water": 2, 2024-05-09T08:49:02.764 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed_ops": 6, 2024-05-09T08:49:02.764 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed": 3, 2024-05-09T08:49:02.764 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_item_in_journal": 0 2024-05-09T08:49:02.764 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.764 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:49:02.764 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-05-09T08:49:02.765 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-05-09T08:49:02.765 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-05-09T08:49:02.765 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 6192, 2024-05-09T08:49:02.765 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 3159141, 2024-05-09T08:49:02.765 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:02.765 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 6192, 2024-05-09T08:49:02.765 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-05-09T08:49:02.765 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-05-09T08:49:02.765 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 6192, 2024-05-09T08:49:02.765 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 3159141, 2024-05-09T08:49:02.765 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-05-09T08:49:02.765 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.766 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.766 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.766 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:02.766 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.766 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_bytes": { 2024-05-09T08:49:02.766 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-05-09T08:49:02.766 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-05-09T08:49:02.766 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-05-09T08:49:02.767 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-05-09T08:49:02.767 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-05-09T08:49:02.767 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:02.767 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:02.767 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 156, 2024-05-09T08:49:02.767 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 93016, 2024-05-09T08:49:02.767 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 156, 2024-05-09T08:49:02.767 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 93016, 2024-05-09T08:49:02.767 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-05-09T08:49:02.768 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.768 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.768 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.768 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:02.770 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.770 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_ops": { 2024-05-09T08:49:02.770 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-05-09T08:49:02.771 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 1024, 2024-05-09T08:49:02.771 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-05-09T08:49:02.771 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-05-09T08:49:02.771 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-05-09T08:49:02.771 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:02.771 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:02.771 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 156, 2024-05-09T08:49:02.771 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 156, 2024-05-09T08:49:02.771 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 156, 2024-05-09T08:49:02.771 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 156, 2024-05-09T08:49:02.772 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-05-09T08:49:02.772 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.772 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.772 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.772 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:02.772 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.772 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle": { 2024-05-09T08:49:02.773 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-05-09T08:49:02.773 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-05-09T08:49:02.773 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-05-09T08:49:02.773 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 3, 2024-05-09T08:49:02.773 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 471, 2024-05-09T08:49:02.773 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:02.773 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 3, 2024-05-09T08:49:02.773 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-05-09T08:49:02.773 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-05-09T08:49:02.773 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 3, 2024-05-09T08:49:02.773 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 471, 2024-05-09T08:49:02.773 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-05-09T08:49:02.774 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.774 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.774 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.774 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:02.774 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:02.774 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle-0x55eb04188500": { 2024-05-09T08:49:02.774 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-05-09T08:49:02.774 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-05-09T08:49:02.774 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-05-09T08:49:02.774 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 42, 2024-05-09T08:49:02.774 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 78848, 2024-05-09T08:49:02.774 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:02.774 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 42, 2024-05-09T08:49:02.774 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-05-09T08:49:02.775 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-05-09T08:49:02.775 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 27, 2024-05-09T08:49:02.775 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 78848, 2024-05-09T08:49:02.775 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-05-09T08:49:02.775 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:02.775 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:02.775 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:02.775 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:02.775 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:02.775 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-05-09T08:49:03.708 INFO:tasks.check_counter:Daemon mds.b mds.exported=3 2024-05-09T08:49:03.708 INFO:tasks.check_counter:Daemon mds.b mds.imported=3 2024-05-09T08:49:03.708 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=3 2024-05-09T08:49:03.708 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=290 2024-05-09T08:49:03.708 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-05-09T08:49:03.708 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=0 2024-05-09T08:49:03.708 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T08:49:03.708 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=0 2024-05-09T08:49:03.708 DEBUG:tasks.check_counter:Getting stats from e 2024-05-09T08:49:03.708 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph daemon mds.e perf dump 2024-05-09T08:49:03.960 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:49:04.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:04 smithi110 ceph-mon[33622]: pgmap v493: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-09T08:49:04.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:04 smithi107 ceph-mon[30122]: pgmap v493: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-09T08:49:04.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:04 smithi157 ceph-mon[33614]: pgmap v493: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-09T08:49:05.751 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-05-09T08:49:05.751 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:49:05.751 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 348, 2024-05-09T08:49:05.751 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 329, 2024-05-09T08:49:05.751 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 1669106, 2024-05-09T08:49:05.751 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 188788, 2024-05-09T08:49:05.751 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 6, 2024-05-09T08:49:05.751 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 1, 2024-05-09T08:49:05.751 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.058136663, 2024-05-09T08:49:05.751 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.018845004, 2024-05-09T08:49:05.752 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.040351761, 2024-05-09T08:49:05.752 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.007152235, 2024-05-09T08:49:05.752 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:05.752 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 329, 2024-05-09T08:49:05.752 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.015947468, 2024-05-09T08:49:05.752 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000048472 2024-05-09T08:49:05.752 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.752 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:05.752 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.752 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.752 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.752 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.752 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 1668272, 2024-05-09T08:49:05.753 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 188464 2024-05-09T08:49:05.753 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.753 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:49:05.753 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 985, 2024-05-09T08:49:05.753 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 1209, 2024-05-09T08:49:05.753 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 242160, 2024-05-09T08:49:05.753 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 867415, 2024-05-09T08:49:05.753 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 7, 2024-05-09T08:49:05.753 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 5, 2024-05-09T08:49:05.753 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.097605262, 2024-05-09T08:49:05.753 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.032025829, 2024-05-09T08:49:05.753 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.020471528, 2024-05-09T08:49:05.753 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.033462434, 2024-05-09T08:49:05.753 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:05.754 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 1206, 2024-05-09T08:49:05.754 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.059836216, 2024-05-09T08:49:05.754 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000049615 2024-05-09T08:49:05.754 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.754 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:05.754 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.754 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.754 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.754 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.754 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-09T08:49:05.754 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 554816 2024-05-09T08:49:05.754 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.754 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:49:05.754 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 2112, 2024-05-09T08:49:05.755 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 2113, 2024-05-09T08:49:05.755 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 726840, 2024-05-09T08:49:05.755 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 671356, 2024-05-09T08:49:05.755 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 4, 2024-05-09T08:49:05.755 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 3, 2024-05-09T08:49:05.755 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.202110408, 2024-05-09T08:49:05.755 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.069919635, 2024-05-09T08:49:05.755 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.051854343, 2024-05-09T08:49:05.755 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.063246538, 2024-05-09T08:49:05.755 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:05.755 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 2111, 2024-05-09T08:49:05.755 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.089704542, 2024-05-09T08:49:05.755 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000042493 2024-05-09T08:49:05.755 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.756 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:05.756 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.756 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.756 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.756 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.756 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T08:49:05.756 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T08:49:05.756 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.756 INFO:teuthology.orchestra.run.smithi110.stdout: "cct": { 2024-05-09T08:49:05.756 INFO:teuthology.orchestra.run.smithi110.stdout: "total_workers": 1, 2024-05-09T08:49:05.756 INFO:teuthology.orchestra.run.smithi110.stdout: "unhealthy_workers": 0 2024-05-09T08:49:05.756 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.756 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-MDSRank": { 2024-05-09T08:49:05.757 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-05-09T08:49:05.757 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-05-09T08:49:05.757 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 5134, 2024-05-09T08:49:05.757 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.146168642, 2024-05-09T08:49:05.757 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000028470 2024-05-09T08:49:05.757 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:05.757 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.757 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-PurgeQueue": { 2024-05-09T08:49:05.757 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-05-09T08:49:05.757 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-05-09T08:49:05.757 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.757 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.757 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.757 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:05.758 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.758 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": { 2024-05-09T08:49:05.758 INFO:teuthology.orchestra.run.smithi110.stdout: "request": 0, 2024-05-09T08:49:05.758 INFO:teuthology.orchestra.run.smithi110.stdout: "reply": 0, 2024-05-09T08:49:05.758 INFO:teuthology.orchestra.run.smithi110.stdout: "reply_latency": { 2024-05-09T08:49:05.758 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.758 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.758 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.758 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.758 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_reply": 0, 2024-05-09T08:49:05.758 INFO:teuthology.orchestra.run.smithi110.stdout: "forward": 0, 2024-05-09T08:49:05.759 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_complete": 0, 2024-05-09T08:49:05.759 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_keys": 0, 2024-05-09T08:49:05.759 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_commit": 0, 2024-05-09T08:49:05.759 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_split": 0, 2024-05-09T08:49:05.759 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_merge": 0, 2024-05-09T08:49:05.759 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes": 8, 2024-05-09T08:49:05.759 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_top": 0, 2024-05-09T08:49:05.759 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_bottom": 8, 2024-05-09T08:49:05.759 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pin_tail": 0, 2024-05-09T08:49:05.759 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pinned": 8, 2024-05-09T08:49:05.760 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_expired": 0, 2024-05-09T08:49:05.760 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_with_caps": 0, 2024-05-09T08:49:05.760 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": 0, 2024-05-09T08:49:05.760 INFO:teuthology.orchestra.run.smithi110.stdout: "subtrees": 1, 2024-05-09T08:49:05.760 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse": 0, 2024-05-09T08:49:05.760 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_hit": 0, 2024-05-09T08:49:05.760 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_forward": 0, 2024-05-09T08:49:05.760 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_discover": 0, 2024-05-09T08:49:05.760 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_dir_fetch": 0, 2024-05-09T08:49:05.760 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_remote_ino": 0, 2024-05-09T08:49:05.760 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_lock": 0, 2024-05-09T08:49:05.760 INFO:teuthology.orchestra.run.smithi110.stdout: "load_cent": 0, 2024-05-09T08:49:05.760 INFO:teuthology.orchestra.run.smithi110.stdout: "q": 0, 2024-05-09T08:49:05.761 INFO:teuthology.orchestra.run.smithi110.stdout: "exported": 0, 2024-05-09T08:49:05.761 INFO:teuthology.orchestra.run.smithi110.stdout: "exported_inodes": 0, 2024-05-09T08:49:05.761 INFO:teuthology.orchestra.run.smithi110.stdout: "imported": 0, 2024-05-09T08:49:05.761 INFO:teuthology.orchestra.run.smithi110.stdout: "imported_inodes": 0, 2024-05-09T08:49:05.761 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_dir_fetch": 0, 2024-05-09T08:49:05.761 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_backtrace_fetch": 0, 2024-05-09T08:49:05.761 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_peer_discover": 0, 2024-05-09T08:49:05.761 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rfiles": 0, 2024-05-09T08:49:05.761 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rbytes": 0, 2024-05-09T08:49:05.761 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rsnaps": 0, 2024-05-09T08:49:05.761 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T08:49:05.761 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_set_tag": 0, 2024-05-09T08:49:05.761 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T08:49:05.761 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_inotable_repaired": 0, 2024-05-09T08:49:05.762 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_inodes": 0, 2024-05-09T08:49:05.762 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T08:49:05.762 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T08:49:05.762 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_file_inodes": 0, 2024-05-09T08:49:05.762 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_inode_file_caps": 0, 2024-05-09T08:49:05.762 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T08:49:05.762 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_grant": 0, 2024-05-09T08:49:05.762 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T08:49:05.762 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T08:49:05.762 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T08:49:05.762 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps": 0, 2024-05-09T08:49:05.762 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps_dirty": 0, 2024-05-09T08:49:05.762 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_cap_release": 0, 2024-05-09T08:49:05.763 INFO:teuthology.orchestra.run.smithi110.stdout: "process_request_cap_release": 0 2024-05-09T08:49:05.763 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.763 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_cache": { 2024-05-09T08:49:05.763 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update": 0, 2024-05-09T08:49:05.763 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update_receipt": 0, 2024-05-09T08:49:05.763 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_try_discover": 0, 2024-05-09T08:49:05.763 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_send_discover": 0, 2024-05-09T08:49:05.763 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_handle_discover": 0, 2024-05-09T08:49:05.763 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays": 0, 2024-05-09T08:49:05.763 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_delayed": 0, 2024-05-09T08:49:05.763 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_enqueuing": 0, 2024-05-09T08:49:05.763 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_created": 0, 2024-05-09T08:49:05.763 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_enqueued": 0, 2024-05-09T08:49:05.763 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_reintegrated": 0, 2024-05-09T08:49:05.764 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_migrated": 0, 2024-05-09T08:49:05.764 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_processing": 0, 2024-05-09T08:49:05.764 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_enqueued": 0, 2024-05-09T08:49:05.764 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_prioritized": 0, 2024-05-09T08:49:05.764 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_started": 0, 2024-05-09T08:49:05.764 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_completed": 0, 2024-05-09T08:49:05.764 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_path": 0, 2024-05-09T08:49:05.764 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_inode": 0, 2024-05-09T08:49:05.764 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T08:49:05.764 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_exportdir": 0, 2024-05-09T08:49:05.764 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_flush": 0, 2024-05-09T08:49:05.764 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragmentdir": 0, 2024-05-09T08:49:05.764 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragstats": 0, 2024-05-09T08:49:05.765 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_inodestats": 0 2024-05-09T08:49:05.765 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.765 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_log": { 2024-05-09T08:49:05.765 INFO:teuthology.orchestra.run.smithi110.stdout: "evlrg": 0, 2024-05-09T08:49:05.765 INFO:teuthology.orchestra.run.smithi110.stdout: "evadd": 0, 2024-05-09T08:49:05.765 INFO:teuthology.orchestra.run.smithi110.stdout: "evex": 0, 2024-05-09T08:49:05.765 INFO:teuthology.orchestra.run.smithi110.stdout: "evtrm": 0, 2024-05-09T08:49:05.765 INFO:teuthology.orchestra.run.smithi110.stdout: "ev": 46, 2024-05-09T08:49:05.765 INFO:teuthology.orchestra.run.smithi110.stdout: "evexg": 0, 2024-05-09T08:49:05.765 INFO:teuthology.orchestra.run.smithi110.stdout: "evexd": 0, 2024-05-09T08:49:05.765 INFO:teuthology.orchestra.run.smithi110.stdout: "segadd": 0, 2024-05-09T08:49:05.765 INFO:teuthology.orchestra.run.smithi110.stdout: "segex": 0, 2024-05-09T08:49:05.765 INFO:teuthology.orchestra.run.smithi110.stdout: "segtrm": 0, 2024-05-09T08:49:05.766 INFO:teuthology.orchestra.run.smithi110.stdout: "seg": 2, 2024-05-09T08:49:05.766 INFO:teuthology.orchestra.run.smithi110.stdout: "segmjr": 2, 2024-05-09T08:49:05.766 INFO:teuthology.orchestra.run.smithi110.stdout: "segexg": 0, 2024-05-09T08:49:05.766 INFO:teuthology.orchestra.run.smithi110.stdout: "segexd": 0, 2024-05-09T08:49:05.766 INFO:teuthology.orchestra.run.smithi110.stdout: "expos": 4194304, 2024-05-09T08:49:05.766 INFO:teuthology.orchestra.run.smithi110.stdout: "wrpos": 4290457, 2024-05-09T08:49:05.766 INFO:teuthology.orchestra.run.smithi110.stdout: "rdpos": 4287516, 2024-05-09T08:49:05.766 INFO:teuthology.orchestra.run.smithi110.stdout: "jlat": { 2024-05-09T08:49:05.766 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.766 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.766 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.766 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.766 INFO:teuthology.orchestra.run.smithi110.stdout: "replayed": 46 2024-05-09T08:49:05.766 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.767 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_mem": { 2024-05-09T08:49:05.767 INFO:teuthology.orchestra.run.smithi110.stdout: "ino": 6, 2024-05-09T08:49:05.767 INFO:teuthology.orchestra.run.smithi110.stdout: "ino+": 28, 2024-05-09T08:49:05.767 INFO:teuthology.orchestra.run.smithi110.stdout: "ino-": 22, 2024-05-09T08:49:05.767 INFO:teuthology.orchestra.run.smithi110.stdout: "dir": 3, 2024-05-09T08:49:05.767 INFO:teuthology.orchestra.run.smithi110.stdout: "dir+": 21, 2024-05-09T08:49:05.767 INFO:teuthology.orchestra.run.smithi110.stdout: "dir-": 18, 2024-05-09T08:49:05.767 INFO:teuthology.orchestra.run.smithi110.stdout: "dn": 8, 2024-05-09T08:49:05.767 INFO:teuthology.orchestra.run.smithi110.stdout: "dn+": 34, 2024-05-09T08:49:05.767 INFO:teuthology.orchestra.run.smithi110.stdout: "dn-": 26, 2024-05-09T08:49:05.767 INFO:teuthology.orchestra.run.smithi110.stdout: "cap": 0, 2024-05-09T08:49:05.767 INFO:teuthology.orchestra.run.smithi110.stdout: "cap+": 0, 2024-05-09T08:49:05.768 INFO:teuthology.orchestra.run.smithi110.stdout: "cap-": 0, 2024-05-09T08:49:05.768 INFO:teuthology.orchestra.run.smithi110.stdout: "rss": 51024, 2024-05-09T08:49:05.768 INFO:teuthology.orchestra.run.smithi110.stdout: "heap": 215420 2024-05-09T08:49:05.768 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.768 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_server": { 2024-05-09T08:49:05.768 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_client_request": 0, 2024-05-09T08:49:05.768 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_server_request": 0, 2024-05-09T08:49:05.768 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_request": 0, 2024-05-09T08:49:05.768 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_session": 0, 2024-05-09T08:49:05.768 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_peer_request": 0, 2024-05-09T08:49:05.768 INFO:teuthology.orchestra.run.smithi110.stdout: "req_create_latency": { 2024-05-09T08:49:05.768 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.768 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.769 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.769 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.769 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getattr_latency": { 2024-05-09T08:49:05.769 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.769 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.769 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.769 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.769 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getfilelock_latency": { 2024-05-09T08:49:05.769 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.769 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.769 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.769 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.769 INFO:teuthology.orchestra.run.smithi110.stdout: "req_link_latency": { 2024-05-09T08:49:05.769 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.770 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.770 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.770 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.770 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookup_latency": { 2024-05-09T08:49:05.770 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.770 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.770 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.770 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.770 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookuphash_latency": { 2024-05-09T08:49:05.770 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.771 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.771 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.771 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.771 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupino_latency": { 2024-05-09T08:49:05.771 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.771 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.771 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.771 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.771 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupname_latency": { 2024-05-09T08:49:05.771 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.771 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.772 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.772 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.772 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupparent_latency": { 2024-05-09T08:49:05.772 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.772 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.772 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.772 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.772 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupsnap_latency": { 2024-05-09T08:49:05.772 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.772 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.772 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.772 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.773 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lssnap_latency": { 2024-05-09T08:49:05.773 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.773 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.773 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.773 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.773 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mkdir_latency": { 2024-05-09T08:49:05.773 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.774 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.774 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.774 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.774 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mknod_latency": { 2024-05-09T08:49:05.774 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.774 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.774 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.774 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.774 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mksnap_latency": { 2024-05-09T08:49:05.774 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.774 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.774 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.775 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.775 INFO:teuthology.orchestra.run.smithi110.stdout: "req_open_latency": { 2024-05-09T08:49:05.775 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.775 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.775 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.775 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.775 INFO:teuthology.orchestra.run.smithi110.stdout: "req_readdir_latency": { 2024-05-09T08:49:05.775 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.775 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.775 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.775 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.775 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rename_latency": { 2024-05-09T08:49:05.775 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.775 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.776 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.776 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.776 INFO:teuthology.orchestra.run.smithi110.stdout: "req_renamesnap_latency": { 2024-05-09T08:49:05.776 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.776 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.776 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.776 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.776 INFO:teuthology.orchestra.run.smithi110.stdout: "req_snapdiff_latency": { 2024-05-09T08:49:05.776 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.776 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.776 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.776 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.776 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmdir_latency": { 2024-05-09T08:49:05.777 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.777 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.777 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.777 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.777 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmsnap_latency": { 2024-05-09T08:49:05.777 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.777 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.777 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.777 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.777 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmxattr_latency": { 2024-05-09T08:49:05.778 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.778 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.778 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.778 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.778 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setattr_latency": { 2024-05-09T08:49:05.778 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.778 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.778 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.778 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.778 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setdirlayout_latency": { 2024-05-09T08:49:05.778 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.778 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.778 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.779 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.779 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setfilelock_latency": { 2024-05-09T08:49:05.779 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.779 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.779 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.779 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.779 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setlayout_latency": { 2024-05-09T08:49:05.779 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.779 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.779 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.779 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.779 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setxattr_latency": { 2024-05-09T08:49:05.779 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.779 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.780 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.780 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.780 INFO:teuthology.orchestra.run.smithi110.stdout: "req_symlink_latency": { 2024-05-09T08:49:05.780 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.780 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.780 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.780 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.780 INFO:teuthology.orchestra.run.smithi110.stdout: "req_unlink_latency": { 2024-05-09T08:49:05.780 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.780 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.780 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.780 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.780 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_revoke_eviction": 0, 2024-05-09T08:49:05.780 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_acquisition_throttle": 0, 2024-05-09T08:49:05.781 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getvxattr_latency": { 2024-05-09T08:49:05.781 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.781 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.781 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.781 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:05.781 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.781 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_sessions": { 2024-05-09T08:49:05.781 INFO:teuthology.orchestra.run.smithi110.stdout: "session_count": 2, 2024-05-09T08:49:05.781 INFO:teuthology.orchestra.run.smithi110.stdout: "session_add": 6, 2024-05-09T08:49:05.781 INFO:teuthology.orchestra.run.smithi110.stdout: "session_remove": 4, 2024-05-09T08:49:05.781 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_open": 5, 2024-05-09T08:49:05.781 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_stale": 0, 2024-05-09T08:49:05.781 INFO:teuthology.orchestra.run.smithi110.stdout: "total_load": 0, 2024-05-09T08:49:05.782 INFO:teuthology.orchestra.run.smithi110.stdout: "average_load": 0, 2024-05-09T08:49:05.782 INFO:teuthology.orchestra.run.smithi110.stdout: "avg_session_uptime": 3762, 2024-05-09T08:49:05.782 INFO:teuthology.orchestra.run.smithi110.stdout: "mdthresh_evicted": 0 2024-05-09T08:49:05.782 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.782 INFO:teuthology.orchestra.run.smithi110.stdout: "mempool": { 2024-05-09T08:49:05.782 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:49:05.782 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_items": 0, 2024-05-09T08:49:05.782 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:49:05.782 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:49:05.782 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:49:05.782 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:49:05.782 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:49:05.782 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:49:05.783 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:49:05.783 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:49:05.783 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:49:05.783 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:49:05.783 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:49:05.783 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:49:05.783 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:49:05.783 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_items": 0, 2024-05-09T08:49:05.783 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:49:05.783 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_items": 0, 2024-05-09T08:49:05.783 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:49:05.783 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:49:05.783 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:49:05.784 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:49:05.784 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:49:05.784 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:49:05.784 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:49:05.784 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_items": 0, 2024-05-09T08:49:05.784 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:49:05.784 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:49:05.784 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:49:05.784 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_items": 0, 2024-05-09T08:49:05.784 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_bytes": 0, 2024-05-09T08:49:05.784 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_items": 0, 2024-05-09T08:49:05.784 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:49:05.784 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:49:05.785 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:49:05.785 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:49:05.785 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_bytes": 190167, 2024-05-09T08:49:05.785 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_items": 52, 2024-05-09T08:49:05.785 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:49:05.785 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_items": 0, 2024-05-09T08:49:05.785 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_bytes": 0, 2024-05-09T08:49:05.785 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_items": 0, 2024-05-09T08:49:05.785 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:49:05.785 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_items": 0, 2024-05-09T08:49:05.785 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:49:05.785 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_items": 0, 2024-05-09T08:49:05.785 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_bytes": 9416, 2024-05-09T08:49:05.785 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_items": 180, 2024-05-09T08:49:05.785 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:49:05.786 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:49:05.786 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_bytes": 0, 2024-05-09T08:49:05.786 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_items": 0, 2024-05-09T08:49:05.786 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_bytes": 25857, 2024-05-09T08:49:05.786 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_items": 177, 2024-05-09T08:49:05.786 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_bytes": 0, 2024-05-09T08:49:05.786 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_items": 0, 2024-05-09T08:49:05.786 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_bytes": 0, 2024-05-09T08:49:05.786 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_items": 0 2024-05-09T08:49:05.786 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.786 INFO:teuthology.orchestra.run.smithi110.stdout: "objecter": { 2024-05-09T08:49:05.786 INFO:teuthology.orchestra.run.smithi110.stdout: "op_active": 0, 2024-05-09T08:49:05.786 INFO:teuthology.orchestra.run.smithi110.stdout: "op_laggy": 0, 2024-05-09T08:49:05.786 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send": 3095, 2024-05-09T08:49:05.787 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send_bytes": 21, 2024-05-09T08:49:05.787 INFO:teuthology.orchestra.run.smithi110.stdout: "op_resend": 3, 2024-05-09T08:49:05.787 INFO:teuthology.orchestra.run.smithi110.stdout: "op_reply": 3095, 2024-05-09T08:49:05.787 INFO:teuthology.orchestra.run.smithi110.stdout: "op_latency": { 2024-05-09T08:49:05.787 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 3095, 2024-05-09T08:49:05.787 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 19.378698391, 2024-05-09T08:49:05.787 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.006261291 2024-05-09T08:49:05.787 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.787 INFO:teuthology.orchestra.run.smithi110.stdout: "op_inflight": 0, 2024-05-09T08:49:05.787 INFO:teuthology.orchestra.run.smithi110.stdout: "oplen_avg": { 2024-05-09T08:49:05.787 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 3095, 2024-05-09T08:49:05.787 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 3096 2024-05-09T08:49:05.787 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.788 INFO:teuthology.orchestra.run.smithi110.stdout: "op": 3095, 2024-05-09T08:49:05.788 INFO:teuthology.orchestra.run.smithi110.stdout: "op_r": 3095, 2024-05-09T08:49:05.788 INFO:teuthology.orchestra.run.smithi110.stdout: "op_w": 0, 2024-05-09T08:49:05.788 INFO:teuthology.orchestra.run.smithi110.stdout: "op_rmw": 0, 2024-05-09T08:49:05.788 INFO:teuthology.orchestra.run.smithi110.stdout: "op_pg": 0, 2024-05-09T08:49:05.788 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_stat": 2027, 2024-05-09T08:49:05.788 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_create": 0, 2024-05-09T08:49:05.788 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_read": 1066, 2024-05-09T08:49:05.788 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_write": 0, 2024-05-09T08:49:05.788 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writefull": 0, 2024-05-09T08:49:05.788 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writesame": 0, 2024-05-09T08:49:05.788 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_append": 0, 2024-05-09T08:49:05.788 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_zero": 0, 2024-05-09T08:49:05.788 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_truncate": 0, 2024-05-09T08:49:05.789 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_delete": 0, 2024-05-09T08:49:05.789 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_mapext": 0, 2024-05-09T08:49:05.789 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_sparse_read": 0, 2024-05-09T08:49:05.789 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_clonerange": 0, 2024-05-09T08:49:05.789 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_getxattr": 1, 2024-05-09T08:49:05.789 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_setxattr": 0, 2024-05-09T08:49:05.789 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_cmpxattr": 0, 2024-05-09T08:49:05.789 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_rmxattr": 0, 2024-05-09T08:49:05.789 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_resetxattrs": 0, 2024-05-09T08:49:05.789 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_call": 0, 2024-05-09T08:49:05.789 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_watch": 0, 2024-05-09T08:49:05.789 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_notify": 0, 2024-05-09T08:49:05.789 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T08:49:05.789 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls": 0, 2024-05-09T08:49:05.790 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls_filter": 0, 2024-05-09T08:49:05.790 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_other": 0, 2024-05-09T08:49:05.790 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_active": 0, 2024-05-09T08:49:05.790 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_send": 0, 2024-05-09T08:49:05.790 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_resend": 0, 2024-05-09T08:49:05.790 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_ping": 0, 2024-05-09T08:49:05.790 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_active": 0, 2024-05-09T08:49:05.790 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_send": 0, 2024-05-09T08:49:05.790 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_resend": 0, 2024-05-09T08:49:05.790 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_active": 0, 2024-05-09T08:49:05.790 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_send": 0, 2024-05-09T08:49:05.790 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_resend": 0, 2024-05-09T08:49:05.790 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_active": 0, 2024-05-09T08:49:05.791 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_send": 0, 2024-05-09T08:49:05.791 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_resend": 0, 2024-05-09T08:49:05.791 INFO:teuthology.orchestra.run.smithi110.stdout: "command_active": 0, 2024-05-09T08:49:05.791 INFO:teuthology.orchestra.run.smithi110.stdout: "command_send": 0, 2024-05-09T08:49:05.791 INFO:teuthology.orchestra.run.smithi110.stdout: "command_resend": 0, 2024-05-09T08:49:05.791 INFO:teuthology.orchestra.run.smithi110.stdout: "map_epoch": 113, 2024-05-09T08:49:05.791 INFO:teuthology.orchestra.run.smithi110.stdout: "map_full": 0, 2024-05-09T08:49:05.791 INFO:teuthology.orchestra.run.smithi110.stdout: "map_inc": 24, 2024-05-09T08:49:05.791 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_sessions": 5, 2024-05-09T08:49:05.791 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_open": 6, 2024-05-09T08:49:05.791 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_close": 1, 2024-05-09T08:49:05.791 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_laggy": 0, 2024-05-09T08:49:05.791 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_wr": 0, 2024-05-09T08:49:05.791 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_rd": 2, 2024-05-09T08:49:05.791 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_del": 0 2024-05-09T08:49:05.792 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.792 INFO:teuthology.orchestra.run.smithi110.stdout: "oft": { 2024-05-09T08:49:05.792 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_objs": 0, 2024-05-09T08:49:05.792 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_kv_pairs": 0, 2024-05-09T08:49:05.792 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_updates": 0, 2024-05-09T08:49:05.792 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_removes": 0 2024-05-09T08:49:05.792 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.792 INFO:teuthology.orchestra.run.smithi110.stdout: "osd-slow-ops": { 2024-05-09T08:49:05.792 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_ops_count": 0 2024-05-09T08:49:05.792 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.792 INFO:teuthology.orchestra.run.smithi110.stdout: "purge_queue": { 2024-05-09T08:49:05.792 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops": 0, 2024-05-09T08:49:05.793 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T08:49:05.793 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing": 0, 2024-05-09T08:49:05.793 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_high_water": 0, 2024-05-09T08:49:05.793 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed_ops": 0, 2024-05-09T08:49:05.793 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed": 0, 2024-05-09T08:49:05.793 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_item_in_journal": 0 2024-05-09T08:49:05.793 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.793 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:49:05.793 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-05-09T08:49:05.793 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-05-09T08:49:05.793 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-05-09T08:49:05.793 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 3441, 2024-05-09T08:49:05.793 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 2307867, 2024-05-09T08:49:05.793 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:05.794 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 3441, 2024-05-09T08:49:05.794 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-05-09T08:49:05.794 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-05-09T08:49:05.794 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 3441, 2024-05-09T08:49:05.794 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 2307867, 2024-05-09T08:49:05.794 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-05-09T08:49:05.794 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.794 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.794 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.794 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:05.794 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.794 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_bytes": { 2024-05-09T08:49:05.794 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-05-09T08:49:05.794 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-05-09T08:49:05.795 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-05-09T08:49:05.795 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-05-09T08:49:05.795 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-05-09T08:49:05.795 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:05.795 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:05.795 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 3095, 2024-05-09T08:49:05.795 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 96158, 2024-05-09T08:49:05.795 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 3095, 2024-05-09T08:49:05.795 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 96158, 2024-05-09T08:49:05.795 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-05-09T08:49:05.795 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.795 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.795 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.795 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:05.796 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.796 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_ops": { 2024-05-09T08:49:05.796 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-05-09T08:49:05.796 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 1024, 2024-05-09T08:49:05.796 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-05-09T08:49:05.796 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-05-09T08:49:05.796 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-05-09T08:49:05.796 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:05.796 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:05.796 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 3095, 2024-05-09T08:49:05.796 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 3095, 2024-05-09T08:49:05.796 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 3095, 2024-05-09T08:49:05.796 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 3095, 2024-05-09T08:49:05.796 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-05-09T08:49:05.797 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.797 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.797 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.797 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:05.797 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.797 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle": { 2024-05-09T08:49:05.797 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-05-09T08:49:05.797 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-05-09T08:49:05.797 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-05-09T08:49:05.797 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-05-09T08:49:05.797 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-05-09T08:49:05.797 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:05.797 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:05.797 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-05-09T08:49:05.797 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-05-09T08:49:05.798 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 0, 2024-05-09T08:49:05.798 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 0, 2024-05-09T08:49:05.798 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-05-09T08:49:05.798 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.798 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.798 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.798 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:05.798 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:05.798 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle-0x56444cf9a6e0": { 2024-05-09T08:49:05.798 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-05-09T08:49:05.798 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-05-09T08:49:05.798 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-05-09T08:49:05.798 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-05-09T08:49:05.798 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-05-09T08:49:05.799 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:05.799 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:05.799 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-05-09T08:49:05.799 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-05-09T08:49:05.799 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 0, 2024-05-09T08:49:05.799 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 0, 2024-05-09T08:49:05.799 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-05-09T08:49:05.799 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:05.799 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:05.799 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:05.799 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:05.799 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:05.799 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-05-09T08:49:06.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:06 smithi107 ceph-mon[30122]: pgmap v494: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-09T08:49:06.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:06 smithi157 ceph-mon[33614]: pgmap v494: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-09T08:49:06.604 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:06 smithi110 ceph-mon[33622]: pgmap v494: 129 pgs: 129 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-09T08:49:06.646 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-05-09T08:49:06.646 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-05-09T08:49:06.646 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-05-09T08:49:06.646 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-05-09T08:49:06.646 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-05-09T08:49:06.646 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-05-09T08:49:06.646 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T08:49:06.646 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-05-09T08:49:06.646 DEBUG:tasks.check_counter:Getting stats from h 2024-05-09T08:49:06.646 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph daemon mds.h perf dump 2024-05-09T08:49:07.123 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:49:07.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:07 smithi107 ceph-mon[30122]: osdmap e114: 12 total, 12 up, 12 in 2024-05-09T08:49:07.585 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:07 smithi110 ceph-mon[33622]: osdmap e114: 12 total, 12 up, 12 in 2024-05-09T08:49:07.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:07 smithi157 ceph-mon[33614]: osdmap e114: 12 total, 12 up, 12 in 2024-05-09T08:49:08.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:08 smithi107 ceph-mon[30122]: pgmap v496: 129 pgs: 6 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 105 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-09T08:49:08.592 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:08 smithi110 ceph-mon[33622]: pgmap v496: 129 pgs: 6 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 105 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-09T08:49:08.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:08 smithi157 ceph-mon[33614]: pgmap v496: 129 pgs: 6 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 105 active+clean; 15 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-09T08:49:09.576 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-05-09T08:49:09.577 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:49:09.577 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 3048, 2024-05-09T08:49:09.577 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 3049, 2024-05-09T08:49:09.577 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 940739, 2024-05-09T08:49:09.577 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 968953, 2024-05-09T08:49:09.577 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 6, 2024-05-09T08:49:09.577 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 2, 2024-05-09T08:49:09.577 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.210801111, 2024-05-09T08:49:09.577 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.065628045, 2024-05-09T08:49:09.577 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.051983546, 2024-05-09T08:49:09.577 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.077771725, 2024-05-09T08:49:09.577 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:09.578 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 3047, 2024-05-09T08:49:09.578 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.081410808, 2024-05-09T08:49:09.578 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000026718 2024-05-09T08:49:09.578 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.578 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:09.578 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.578 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.578 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.578 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.579 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T08:49:09.579 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T08:49:09.579 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.579 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:49:09.579 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 345, 2024-05-09T08:49:09.579 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 328, 2024-05-09T08:49:09.579 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 1666828, 2024-05-09T08:49:09.579 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 188250, 2024-05-09T08:49:09.579 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 3, 2024-05-09T08:49:09.579 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 2, 2024-05-09T08:49:09.579 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.054155871, 2024-05-09T08:49:09.580 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.017873044, 2024-05-09T08:49:09.580 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.037800611, 2024-05-09T08:49:09.580 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.006818427, 2024-05-09T08:49:09.580 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:09.580 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 327, 2024-05-09T08:49:09.580 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.016832304, 2024-05-09T08:49:09.580 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000051474 2024-05-09T08:49:09.580 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.580 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:09.580 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.580 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.580 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.581 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.581 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 1666272, 2024-05-09T08:49:09.581 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 187552 2024-05-09T08:49:09.581 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.581 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:49:09.581 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 8, 2024-05-09T08:49:09.581 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 230, 2024-05-09T08:49:09.581 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 3788, 2024-05-09T08:49:09.582 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 646286, 2024-05-09T08:49:09.582 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 8, 2024-05-09T08:49:09.582 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 4, 2024-05-09T08:49:09.582 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.016519778, 2024-05-09T08:49:09.582 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.008370534, 2024-05-09T08:49:09.582 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.005832968, 2024-05-09T08:49:09.582 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.000029705, 2024-05-09T08:49:09.582 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:09.582 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 228, 2024-05-09T08:49:09.582 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.023533525, 2024-05-09T08:49:09.582 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000103217 2024-05-09T08:49:09.582 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.582 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:09.583 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.583 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.583 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.583 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.583 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 3520, 2024-05-09T08:49:09.583 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 645968 2024-05-09T08:49:09.583 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.583 INFO:teuthology.orchestra.run.smithi110.stdout: "cct": { 2024-05-09T08:49:09.583 INFO:teuthology.orchestra.run.smithi110.stdout: "total_workers": 1, 2024-05-09T08:49:09.583 INFO:teuthology.orchestra.run.smithi110.stdout: "unhealthy_workers": 0 2024-05-09T08:49:09.583 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.583 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-MDSRank": { 2024-05-09T08:49:09.583 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-05-09T08:49:09.584 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-05-09T08:49:09.584 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 5132, 2024-05-09T08:49:09.584 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.113239231, 2024-05-09T08:49:09.584 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000022065 2024-05-09T08:49:09.584 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:09.584 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.584 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-PurgeQueue": { 2024-05-09T08:49:09.584 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-05-09T08:49:09.584 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-05-09T08:49:09.584 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.584 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.584 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.585 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:09.585 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.585 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": { 2024-05-09T08:49:09.585 INFO:teuthology.orchestra.run.smithi110.stdout: "request": 0, 2024-05-09T08:49:09.585 INFO:teuthology.orchestra.run.smithi110.stdout: "reply": 0, 2024-05-09T08:49:09.585 INFO:teuthology.orchestra.run.smithi110.stdout: "reply_latency": { 2024-05-09T08:49:09.585 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.585 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.585 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.585 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.585 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_reply": 0, 2024-05-09T08:49:09.585 INFO:teuthology.orchestra.run.smithi110.stdout: "forward": 0, 2024-05-09T08:49:09.585 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_complete": 0, 2024-05-09T08:49:09.585 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_keys": 0, 2024-05-09T08:49:09.586 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_commit": 0, 2024-05-09T08:49:09.586 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_split": 0, 2024-05-09T08:49:09.586 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_merge": 0, 2024-05-09T08:49:09.586 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes": 4, 2024-05-09T08:49:09.586 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_top": 0, 2024-05-09T08:49:09.586 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_bottom": 4, 2024-05-09T08:49:09.586 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pin_tail": 0, 2024-05-09T08:49:09.586 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pinned": 4, 2024-05-09T08:49:09.586 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_expired": 0, 2024-05-09T08:49:09.586 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_with_caps": 0, 2024-05-09T08:49:09.586 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": 0, 2024-05-09T08:49:09.586 INFO:teuthology.orchestra.run.smithi110.stdout: "subtrees": 1, 2024-05-09T08:49:09.587 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse": 0, 2024-05-09T08:49:09.587 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_hit": 0, 2024-05-09T08:49:09.587 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_forward": 0, 2024-05-09T08:49:09.587 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_discover": 0, 2024-05-09T08:49:09.587 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_dir_fetch": 0, 2024-05-09T08:49:09.587 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_remote_ino": 0, 2024-05-09T08:49:09.587 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_lock": 0, 2024-05-09T08:49:09.587 INFO:teuthology.orchestra.run.smithi110.stdout: "load_cent": 0, 2024-05-09T08:49:09.587 INFO:teuthology.orchestra.run.smithi110.stdout: "q": 0, 2024-05-09T08:49:09.587 INFO:teuthology.orchestra.run.smithi110.stdout: "exported": 0, 2024-05-09T08:49:09.587 INFO:teuthology.orchestra.run.smithi110.stdout: "exported_inodes": 0, 2024-05-09T08:49:09.587 INFO:teuthology.orchestra.run.smithi110.stdout: "imported": 0, 2024-05-09T08:49:09.588 INFO:teuthology.orchestra.run.smithi110.stdout: "imported_inodes": 0, 2024-05-09T08:49:09.588 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_dir_fetch": 0, 2024-05-09T08:49:09.588 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_backtrace_fetch": 0, 2024-05-09T08:49:09.588 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_peer_discover": 0, 2024-05-09T08:49:09.588 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rfiles": 0, 2024-05-09T08:49:09.588 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rbytes": 0, 2024-05-09T08:49:09.588 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rsnaps": 0, 2024-05-09T08:49:09.588 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T08:49:09.588 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_set_tag": 0, 2024-05-09T08:49:09.588 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T08:49:09.588 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_inotable_repaired": 0, 2024-05-09T08:49:09.588 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_inodes": 0, 2024-05-09T08:49:09.588 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_file_inodes": 0, 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_inode_file_caps": 0, 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_grant": 0, 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T08:49:09.589 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps": 0, 2024-05-09T08:49:09.590 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps_dirty": 0, 2024-05-09T08:49:09.590 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_cap_release": 0, 2024-05-09T08:49:09.590 INFO:teuthology.orchestra.run.smithi110.stdout: "process_request_cap_release": 0 2024-05-09T08:49:09.590 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.590 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_cache": { 2024-05-09T08:49:09.590 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update": 0, 2024-05-09T08:49:09.590 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update_receipt": 0, 2024-05-09T08:49:09.590 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_try_discover": 0, 2024-05-09T08:49:09.590 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_send_discover": 0, 2024-05-09T08:49:09.590 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_handle_discover": 0, 2024-05-09T08:49:09.590 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays": 0, 2024-05-09T08:49:09.590 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_delayed": 0, 2024-05-09T08:49:09.590 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_enqueuing": 0, 2024-05-09T08:49:09.590 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_created": 0, 2024-05-09T08:49:09.591 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_enqueued": 0, 2024-05-09T08:49:09.591 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_reintegrated": 0, 2024-05-09T08:49:09.591 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_migrated": 0, 2024-05-09T08:49:09.591 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_processing": 0, 2024-05-09T08:49:09.591 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_enqueued": 0, 2024-05-09T08:49:09.591 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_prioritized": 0, 2024-05-09T08:49:09.591 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_started": 0, 2024-05-09T08:49:09.591 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_completed": 0, 2024-05-09T08:49:09.591 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_path": 0, 2024-05-09T08:49:09.591 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_inode": 0, 2024-05-09T08:49:09.592 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T08:49:09.592 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_exportdir": 0, 2024-05-09T08:49:09.592 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_flush": 0, 2024-05-09T08:49:09.592 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragmentdir": 0, 2024-05-09T08:49:09.592 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragstats": 0, 2024-05-09T08:49:09.592 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_inodestats": 0 2024-05-09T08:49:09.592 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.592 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_log": { 2024-05-09T08:49:09.592 INFO:teuthology.orchestra.run.smithi110.stdout: "evlrg": 0, 2024-05-09T08:49:09.592 INFO:teuthology.orchestra.run.smithi110.stdout: "evadd": 0, 2024-05-09T08:49:09.592 INFO:teuthology.orchestra.run.smithi110.stdout: "evex": 0, 2024-05-09T08:49:09.593 INFO:teuthology.orchestra.run.smithi110.stdout: "evtrm": 0, 2024-05-09T08:49:09.593 INFO:teuthology.orchestra.run.smithi110.stdout: "ev": 42, 2024-05-09T08:49:09.593 INFO:teuthology.orchestra.run.smithi110.stdout: "evexg": 0, 2024-05-09T08:49:09.593 INFO:teuthology.orchestra.run.smithi110.stdout: "evexd": 0, 2024-05-09T08:49:09.593 INFO:teuthology.orchestra.run.smithi110.stdout: "segadd": 0, 2024-05-09T08:49:09.593 INFO:teuthology.orchestra.run.smithi110.stdout: "segex": 0, 2024-05-09T08:49:09.593 INFO:teuthology.orchestra.run.smithi110.stdout: "segtrm": 0, 2024-05-09T08:49:09.593 INFO:teuthology.orchestra.run.smithi110.stdout: "seg": 2, 2024-05-09T08:49:09.593 INFO:teuthology.orchestra.run.smithi110.stdout: "segmjr": 2, 2024-05-09T08:49:09.593 INFO:teuthology.orchestra.run.smithi110.stdout: "segexg": 0, 2024-05-09T08:49:09.594 INFO:teuthology.orchestra.run.smithi110.stdout: "segexd": 0, 2024-05-09T08:49:09.594 INFO:teuthology.orchestra.run.smithi110.stdout: "expos": 4194304, 2024-05-09T08:49:09.594 INFO:teuthology.orchestra.run.smithi110.stdout: "wrpos": 4273152, 2024-05-09T08:49:09.594 INFO:teuthology.orchestra.run.smithi110.stdout: "rdpos": 4270326, 2024-05-09T08:49:09.594 INFO:teuthology.orchestra.run.smithi110.stdout: "jlat": { 2024-05-09T08:49:09.594 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.594 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.594 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.594 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.594 INFO:teuthology.orchestra.run.smithi110.stdout: "replayed": 42 2024-05-09T08:49:09.594 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.594 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_mem": { 2024-05-09T08:49:09.595 INFO:teuthology.orchestra.run.smithi110.stdout: "ino": 4, 2024-05-09T08:49:09.595 INFO:teuthology.orchestra.run.smithi110.stdout: "ino+": 30, 2024-05-09T08:49:09.595 INFO:teuthology.orchestra.run.smithi110.stdout: "ino-": 26, 2024-05-09T08:49:09.595 INFO:teuthology.orchestra.run.smithi110.stdout: "dir": 2, 2024-05-09T08:49:09.595 INFO:teuthology.orchestra.run.smithi110.stdout: "dir+": 23, 2024-05-09T08:49:09.595 INFO:teuthology.orchestra.run.smithi110.stdout: "dir-": 21, 2024-05-09T08:49:09.595 INFO:teuthology.orchestra.run.smithi110.stdout: "dn": 4, 2024-05-09T08:49:09.595 INFO:teuthology.orchestra.run.smithi110.stdout: "dn+": 32, 2024-05-09T08:49:09.595 INFO:teuthology.orchestra.run.smithi110.stdout: "dn-": 28, 2024-05-09T08:49:09.595 INFO:teuthology.orchestra.run.smithi110.stdout: "cap": 0, 2024-05-09T08:49:09.596 INFO:teuthology.orchestra.run.smithi110.stdout: "cap+": 0, 2024-05-09T08:49:09.596 INFO:teuthology.orchestra.run.smithi110.stdout: "cap-": 0, 2024-05-09T08:49:09.596 INFO:teuthology.orchestra.run.smithi110.stdout: "rss": 51028, 2024-05-09T08:49:09.596 INFO:teuthology.orchestra.run.smithi110.stdout: "heap": 215420 2024-05-09T08:49:09.596 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.596 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_server": { 2024-05-09T08:49:09.596 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_client_request": 0, 2024-05-09T08:49:09.596 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_server_request": 0, 2024-05-09T08:49:09.596 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_request": 0, 2024-05-09T08:49:09.596 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_session": 0, 2024-05-09T08:49:09.596 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_peer_request": 0, 2024-05-09T08:49:09.597 INFO:teuthology.orchestra.run.smithi110.stdout: "req_create_latency": { 2024-05-09T08:49:09.597 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.597 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.597 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.597 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.597 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getattr_latency": { 2024-05-09T08:49:09.597 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.597 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.597 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.597 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.597 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getfilelock_latency": { 2024-05-09T08:49:09.598 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.598 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.598 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.598 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.598 INFO:teuthology.orchestra.run.smithi110.stdout: "req_link_latency": { 2024-05-09T08:49:09.598 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.598 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.598 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.598 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.598 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookup_latency": { 2024-05-09T08:49:09.599 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.599 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.599 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.599 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.599 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookuphash_latency": { 2024-05-09T08:49:09.599 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.599 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.599 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.599 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.599 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupino_latency": { 2024-05-09T08:49:09.600 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.600 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.600 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.600 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.600 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupname_latency": { 2024-05-09T08:49:09.600 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.600 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.600 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.600 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.600 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupparent_latency": { 2024-05-09T08:49:09.600 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.600 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.600 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.601 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.601 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupsnap_latency": { 2024-05-09T08:49:09.601 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.601 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.601 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.601 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.601 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lssnap_latency": { 2024-05-09T08:49:09.601 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.601 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.601 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.601 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.601 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mkdir_latency": { 2024-05-09T08:49:09.601 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.602 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.602 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.602 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.603 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mknod_latency": { 2024-05-09T08:49:09.603 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.603 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.603 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.603 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.603 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mksnap_latency": { 2024-05-09T08:49:09.603 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.604 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.604 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.604 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.604 INFO:teuthology.orchestra.run.smithi110.stdout: "req_open_latency": { 2024-05-09T08:49:09.604 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.604 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.604 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.604 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.604 INFO:teuthology.orchestra.run.smithi110.stdout: "req_readdir_latency": { 2024-05-09T08:49:09.605 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.605 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.605 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.605 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.605 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rename_latency": { 2024-05-09T08:49:09.605 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.605 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.605 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.605 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.605 INFO:teuthology.orchestra.run.smithi110.stdout: "req_renamesnap_latency": { 2024-05-09T08:49:09.605 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.605 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.605 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.606 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.606 INFO:teuthology.orchestra.run.smithi110.stdout: "req_snapdiff_latency": { 2024-05-09T08:49:09.606 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.606 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.606 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.606 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.606 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmdir_latency": { 2024-05-09T08:49:09.606 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.606 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.606 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.606 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmsnap_latency": { 2024-05-09T08:49:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmxattr_latency": { 2024-05-09T08:49:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.608 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setattr_latency": { 2024-05-09T08:49:09.608 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.608 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.608 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.608 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.608 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setdirlayout_latency": { 2024-05-09T08:49:09.608 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.608 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.608 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.608 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.608 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setfilelock_latency": { 2024-05-09T08:49:09.608 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.608 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.608 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.609 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.609 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setlayout_latency": { 2024-05-09T08:49:09.609 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.609 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.609 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.609 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.609 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setxattr_latency": { 2024-05-09T08:49:09.609 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.609 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.609 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.609 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.609 INFO:teuthology.orchestra.run.smithi110.stdout: "req_symlink_latency": { 2024-05-09T08:49:09.609 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.609 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.610 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.610 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.610 INFO:teuthology.orchestra.run.smithi110.stdout: "req_unlink_latency": { 2024-05-09T08:49:09.610 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.610 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.610 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.610 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.610 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_revoke_eviction": 0, 2024-05-09T08:49:09.610 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_acquisition_throttle": 0, 2024-05-09T08:49:09.610 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getvxattr_latency": { 2024-05-09T08:49:09.610 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.610 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.611 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.611 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:09.611 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.611 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_sessions": { 2024-05-09T08:49:09.611 INFO:teuthology.orchestra.run.smithi110.stdout: "session_count": 1, 2024-05-09T08:49:09.611 INFO:teuthology.orchestra.run.smithi110.stdout: "session_add": 4, 2024-05-09T08:49:09.611 INFO:teuthology.orchestra.run.smithi110.stdout: "session_remove": 3, 2024-05-09T08:49:09.611 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_open": 4, 2024-05-09T08:49:09.611 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_stale": 0, 2024-05-09T08:49:09.611 INFO:teuthology.orchestra.run.smithi110.stdout: "total_load": 0, 2024-05-09T08:49:09.611 INFO:teuthology.orchestra.run.smithi110.stdout: "average_load": 0, 2024-05-09T08:49:09.611 INFO:teuthology.orchestra.run.smithi110.stdout: "avg_session_uptime": 5457, 2024-05-09T08:49:09.611 INFO:teuthology.orchestra.run.smithi110.stdout: "mdthresh_evicted": 0 2024-05-09T08:49:09.612 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.612 INFO:teuthology.orchestra.run.smithi110.stdout: "mempool": { 2024-05-09T08:49:09.612 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:49:09.612 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_items": 0, 2024-05-09T08:49:09.612 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:49:09.612 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:49:09.612 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:49:09.612 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:49:09.612 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:49:09.612 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:49:09.612 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:49:09.612 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:49:09.612 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:49:09.612 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:49:09.613 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:49:09.613 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:49:09.613 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:49:09.613 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_items": 0, 2024-05-09T08:49:09.613 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:49:09.613 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_items": 0, 2024-05-09T08:49:09.613 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:49:09.613 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:49:09.613 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:49:09.613 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:49:09.613 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:49:09.613 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:49:09.614 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:49:09.614 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_items": 0, 2024-05-09T08:49:09.614 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:49:09.614 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:49:09.614 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:49:09.614 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_items": 0, 2024-05-09T08:49:09.614 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_bytes": 0, 2024-05-09T08:49:09.614 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_items": 0, 2024-05-09T08:49:09.614 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:49:09.614 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:49:09.614 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:49:09.614 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:49:09.614 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_bytes": 182065, 2024-05-09T08:49:09.615 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_items": 48, 2024-05-09T08:49:09.615 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:49:09.615 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_items": 0, 2024-05-09T08:49:09.615 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_bytes": 0, 2024-05-09T08:49:09.615 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_items": 0, 2024-05-09T08:49:09.615 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:49:09.615 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_items": 0, 2024-05-09T08:49:09.615 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:49:09.615 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_items": 0, 2024-05-09T08:49:09.615 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_bytes": 9656, 2024-05-09T08:49:09.615 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_items": 186, 2024-05-09T08:49:09.615 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:49:09.615 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:49:09.616 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_bytes": 0, 2024-05-09T08:49:09.616 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_items": 0, 2024-05-09T08:49:09.616 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_bytes": 16480, 2024-05-09T08:49:09.616 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_items": 48, 2024-05-09T08:49:09.616 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_bytes": 0, 2024-05-09T08:49:09.616 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_items": 0, 2024-05-09T08:49:09.616 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_bytes": 0, 2024-05-09T08:49:09.616 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_items": 0 2024-05-09T08:49:09.616 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.616 INFO:teuthology.orchestra.run.smithi110.stdout: "objecter": { 2024-05-09T08:49:09.616 INFO:teuthology.orchestra.run.smithi110.stdout: "op_active": 0, 2024-05-09T08:49:09.616 INFO:teuthology.orchestra.run.smithi110.stdout: "op_laggy": 0, 2024-05-09T08:49:09.616 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send": 3050, 2024-05-09T08:49:09.616 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send_bytes": 21, 2024-05-09T08:49:09.617 INFO:teuthology.orchestra.run.smithi110.stdout: "op_resend": 3, 2024-05-09T08:49:09.617 INFO:teuthology.orchestra.run.smithi110.stdout: "op_reply": 3050, 2024-05-09T08:49:09.617 INFO:teuthology.orchestra.run.smithi110.stdout: "op_latency": { 2024-05-09T08:49:09.617 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 3050, 2024-05-09T08:49:09.617 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 33.313480677, 2024-05-09T08:49:09.617 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.010922452 2024-05-09T08:49:09.617 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.617 INFO:teuthology.orchestra.run.smithi110.stdout: "op_inflight": 0, 2024-05-09T08:49:09.617 INFO:teuthology.orchestra.run.smithi110.stdout: "oplen_avg": { 2024-05-09T08:49:09.617 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 3050, 2024-05-09T08:49:09.617 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 3051 2024-05-09T08:49:09.617 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.617 INFO:teuthology.orchestra.run.smithi110.stdout: "op": 3050, 2024-05-09T08:49:09.618 INFO:teuthology.orchestra.run.smithi110.stdout: "op_r": 3050, 2024-05-09T08:49:09.618 INFO:teuthology.orchestra.run.smithi110.stdout: "op_w": 0, 2024-05-09T08:49:09.618 INFO:teuthology.orchestra.run.smithi110.stdout: "op_rmw": 0, 2024-05-09T08:49:09.618 INFO:teuthology.orchestra.run.smithi110.stdout: "op_pg": 0, 2024-05-09T08:49:09.618 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_stat": 1982, 2024-05-09T08:49:09.618 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_create": 0, 2024-05-09T08:49:09.618 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_read": 1066, 2024-05-09T08:49:09.618 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_write": 0, 2024-05-09T08:49:09.618 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writefull": 0, 2024-05-09T08:49:09.618 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writesame": 0, 2024-05-09T08:49:09.618 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_append": 0, 2024-05-09T08:49:09.618 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_zero": 0, 2024-05-09T08:49:09.618 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_truncate": 0, 2024-05-09T08:49:09.619 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_delete": 0, 2024-05-09T08:49:09.619 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_mapext": 0, 2024-05-09T08:49:09.619 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_sparse_read": 0, 2024-05-09T08:49:09.619 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_clonerange": 0, 2024-05-09T08:49:09.619 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_getxattr": 1, 2024-05-09T08:49:09.619 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_setxattr": 0, 2024-05-09T08:49:09.619 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_cmpxattr": 0, 2024-05-09T08:49:09.620 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_rmxattr": 0, 2024-05-09T08:49:09.620 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_resetxattrs": 0, 2024-05-09T08:49:09.620 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_call": 0, 2024-05-09T08:49:09.620 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_watch": 0, 2024-05-09T08:49:09.620 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_notify": 0, 2024-05-09T08:49:09.620 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T08:49:09.620 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls": 0, 2024-05-09T08:49:09.620 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls_filter": 0, 2024-05-09T08:49:09.620 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_other": 0, 2024-05-09T08:49:09.620 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_active": 0, 2024-05-09T08:49:09.620 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_send": 0, 2024-05-09T08:49:09.620 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_resend": 0, 2024-05-09T08:49:09.620 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_ping": 0, 2024-05-09T08:49:09.621 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_active": 0, 2024-05-09T08:49:09.621 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_send": 0, 2024-05-09T08:49:09.621 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_resend": 0, 2024-05-09T08:49:09.621 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_active": 0, 2024-05-09T08:49:09.621 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_send": 0, 2024-05-09T08:49:09.621 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_resend": 0, 2024-05-09T08:49:09.621 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_active": 0, 2024-05-09T08:49:09.621 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_send": 0, 2024-05-09T08:49:09.621 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_resend": 0, 2024-05-09T08:49:09.621 INFO:teuthology.orchestra.run.smithi110.stdout: "command_active": 0, 2024-05-09T08:49:09.621 INFO:teuthology.orchestra.run.smithi110.stdout: "command_send": 0, 2024-05-09T08:49:09.621 INFO:teuthology.orchestra.run.smithi110.stdout: "command_resend": 0, 2024-05-09T08:49:09.622 INFO:teuthology.orchestra.run.smithi110.stdout: "map_epoch": 114, 2024-05-09T08:49:09.622 INFO:teuthology.orchestra.run.smithi110.stdout: "map_full": 0, 2024-05-09T08:49:09.622 INFO:teuthology.orchestra.run.smithi110.stdout: "map_inc": 25, 2024-05-09T08:49:09.622 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_sessions": 4, 2024-05-09T08:49:09.622 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_open": 5, 2024-05-09T08:49:09.622 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_close": 1, 2024-05-09T08:49:09.622 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_laggy": 0, 2024-05-09T08:49:09.622 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_wr": 0, 2024-05-09T08:49:09.622 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_rd": 2, 2024-05-09T08:49:09.622 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_del": 0 2024-05-09T08:49:09.622 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.622 INFO:teuthology.orchestra.run.smithi110.stdout: "oft": { 2024-05-09T08:49:09.622 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_objs": 0, 2024-05-09T08:49:09.623 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_kv_pairs": 0, 2024-05-09T08:49:09.623 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_updates": 0, 2024-05-09T08:49:09.623 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_removes": 0 2024-05-09T08:49:09.623 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.623 INFO:teuthology.orchestra.run.smithi110.stdout: "osd-slow-ops": { 2024-05-09T08:49:09.623 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_ops_count": 0 2024-05-09T08:49:09.623 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.623 INFO:teuthology.orchestra.run.smithi110.stdout: "purge_queue": { 2024-05-09T08:49:09.623 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops": 0, 2024-05-09T08:49:09.623 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T08:49:09.623 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing": 0, 2024-05-09T08:49:09.623 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_high_water": 0, 2024-05-09T08:49:09.623 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed_ops": 0, 2024-05-09T08:49:09.624 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed": 0, 2024-05-09T08:49:09.624 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_item_in_journal": 0 2024-05-09T08:49:09.624 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.624 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:49:09.624 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-05-09T08:49:09.624 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-05-09T08:49:09.624 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-05-09T08:49:09.624 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 3397, 2024-05-09T08:49:09.624 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 2285035, 2024-05-09T08:49:09.624 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:09.624 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 3397, 2024-05-09T08:49:09.624 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-05-09T08:49:09.625 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-05-09T08:49:09.625 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 3397, 2024-05-09T08:49:09.625 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 2285035, 2024-05-09T08:49:09.625 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-05-09T08:49:09.625 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.625 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.625 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.625 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:09.625 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.625 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_bytes": { 2024-05-09T08:49:09.625 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-05-09T08:49:09.626 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-05-09T08:49:09.626 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-05-09T08:49:09.626 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-05-09T08:49:09.626 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-05-09T08:49:09.626 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:09.626 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:09.626 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 3050, 2024-05-09T08:49:09.626 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 78853, 2024-05-09T08:49:09.626 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 3050, 2024-05-09T08:49:09.626 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 78853, 2024-05-09T08:49:09.626 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-05-09T08:49:09.626 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.627 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.627 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.627 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:09.627 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.627 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_ops": { 2024-05-09T08:49:09.627 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-05-09T08:49:09.627 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 1024, 2024-05-09T08:49:09.627 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-05-09T08:49:09.627 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-05-09T08:49:09.627 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-05-09T08:49:09.627 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:09.627 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:09.628 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 3050, 2024-05-09T08:49:09.628 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 3050, 2024-05-09T08:49:09.628 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 3050, 2024-05-09T08:49:09.628 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 3050, 2024-05-09T08:49:09.628 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-05-09T08:49:09.628 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.628 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.628 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.628 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:09.628 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.628 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle": { 2024-05-09T08:49:09.628 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-05-09T08:49:09.629 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-05-09T08:49:09.629 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-05-09T08:49:09.629 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-05-09T08:49:09.629 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-05-09T08:49:09.629 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:09.629 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:09.629 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-05-09T08:49:09.629 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-05-09T08:49:09.629 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 0, 2024-05-09T08:49:09.629 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 0, 2024-05-09T08:49:09.629 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-05-09T08:49:09.630 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.630 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.630 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.630 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:09.630 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:09.630 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle-0x55ec5e6ee6e0": { 2024-05-09T08:49:09.630 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-05-09T08:49:09.630 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-05-09T08:49:09.630 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-05-09T08:49:09.630 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-05-09T08:49:09.631 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-05-09T08:49:09.631 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:09.631 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:09.631 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-05-09T08:49:09.631 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-05-09T08:49:09.631 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 0, 2024-05-09T08:49:09.631 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 0, 2024-05-09T08:49:09.631 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-05-09T08:49:09.631 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:09.632 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:09.632 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:09.632 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:09.632 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:09.632 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-05-09T08:49:10.338 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:10 smithi110 ceph-mon[33622]: pgmap v497: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 15 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-09T08:49:10.376 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-05-09T08:49:10.376 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-05-09T08:49:10.376 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-05-09T08:49:10.376 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2024-05-09T08:49:10.376 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-05-09T08:49:10.376 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-05-09T08:49:10.376 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T08:49:10.376 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-05-09T08:49:10.376 DEBUG:tasks.check_counter:Getting stats from k 2024-05-09T08:49:10.377 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph daemon mds.k perf dump 2024-05-09T08:49:10.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:10 smithi107 ceph-mon[30122]: pgmap v497: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 15 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-09T08:49:10.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:10 smithi157 ceph-mon[33614]: pgmap v497: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 15 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-09T08:49:10.689 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T08:49:12.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:12 smithi107 ceph-mon[30122]: pgmap v498: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 15 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-09T08:49:12.568 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:12 smithi110 ceph-mon[33622]: pgmap v498: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 15 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-09T08:49:12.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:12 smithi157 ceph-mon[33614]: pgmap v498: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 15 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-09T08:49:13.149 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-05-09T08:49:13.149 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:49:13.149 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 4, 2024-05-09T08:49:13.149 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 4, 2024-05-09T08:49:13.149 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 3040, 2024-05-09T08:49:13.149 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 448, 2024-05-09T08:49:13.149 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 4, 2024-05-09T08:49:13.149 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 0, 2024-05-09T08:49:13.150 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.001385345, 2024-05-09T08:49:13.150 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.000033377, 2024-05-09T08:49:13.150 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.000633909, 2024-05-09T08:49:13.150 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T08:49:13.150 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:13.150 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 3, 2024-05-09T08:49:13.150 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000736069, 2024-05-09T08:49:13.150 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000245356 2024-05-09T08:49:13.150 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:13.150 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:13.150 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:13.150 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:13.151 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:13.151 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:13.151 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 3040, 2024-05-09T08:49:13.151 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 448 2024-05-09T08:49:13.151 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:13.151 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:49:13.151 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 3, 2024-05-09T08:49:13.151 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 231, 2024-05-09T08:49:13.151 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 480, 2024-05-09T08:49:13.151 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 274176, 2024-05-09T08:49:13.151 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 6, 2024-05-09T08:49:13.152 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 3, 2024-05-09T08:49:13.152 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.018676698, 2024-05-09T08:49:13.152 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.010044542, 2024-05-09T08:49:13.152 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.001932648, 2024-05-09T08:49:13.152 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T08:49:13.152 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:13.152 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 228, 2024-05-09T08:49:13.152 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.029236108, 2024-05-09T08:49:13.152 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000128228 2024-05-09T08:49:13.152 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:13.152 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:13.152 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:13.152 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:13.152 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:13.152 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:13.153 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 480, 2024-05-09T08:49:13.153 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 274176 2024-05-09T08:49:13.153 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:13.153 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:49:13.153 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 312, 2024-05-09T08:49:13.153 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 301, 2024-05-09T08:49:13.153 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 1606896, 2024-05-09T08:49:13.153 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 185520, 2024-05-09T08:49:13.153 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 2, 2024-05-09T08:49:13.153 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 1, 2024-05-09T08:49:13.153 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.057323332, 2024-05-09T08:49:13.154 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.017826258, 2024-05-09T08:49:13.154 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.041037534, 2024-05-09T08:49:13.154 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.006834494, 2024-05-09T08:49:13.154 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:13.154 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 300, 2024-05-09T08:49:13.154 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.017827767, 2024-05-09T08:49:13.154 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000059425 2024-05-09T08:49:13.154 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:13.154 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:13.154 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:13.154 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:13.154 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:13.155 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:13.155 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 1606896, 2024-05-09T08:49:13.155 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 185520 2024-05-09T08:49:13.155 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:13.155 INFO:teuthology.orchestra.run.smithi110.stdout: "cct": { 2024-05-09T08:49:13.155 INFO:teuthology.orchestra.run.smithi110.stdout: "total_workers": 0, 2024-05-09T08:49:13.155 INFO:teuthology.orchestra.run.smithi110.stdout: "unhealthy_workers": 0 2024-05-09T08:49:13.155 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:13.155 INFO:teuthology.orchestra.run.smithi110.stdout: "mempool": { 2024-05-09T08:49:13.155 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:49:13.156 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_items": 0, 2024-05-09T08:49:13.156 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:49:13.156 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:49:13.156 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:49:13.156 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:49:13.156 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:49:13.156 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:49:13.156 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:49:13.156 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:49:13.157 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:49:13.157 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:49:13.157 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:49:13.157 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:49:13.157 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:49:13.157 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_items": 0, 2024-05-09T08:49:13.157 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:49:13.158 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_items": 0, 2024-05-09T08:49:13.158 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:49:13.158 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:49:13.158 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:49:13.158 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:49:13.158 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:49:13.158 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:49:13.158 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:49:13.158 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_items": 0, 2024-05-09T08:49:13.158 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:49:13.158 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:49:13.158 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:49:13.159 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_items": 0, 2024-05-09T08:49:13.159 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_bytes": 0, 2024-05-09T08:49:13.159 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_items": 0, 2024-05-09T08:49:13.159 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:49:13.159 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:49:13.159 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:49:13.159 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:49:13.159 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_bytes": 148704, 2024-05-09T08:49:13.159 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_items": 29, 2024-05-09T08:49:13.159 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:49:13.159 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_items": 0, 2024-05-09T08:49:13.159 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_bytes": 0, 2024-05-09T08:49:13.159 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_items": 0, 2024-05-09T08:49:13.159 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:49:13.160 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_items": 0, 2024-05-09T08:49:13.160 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:49:13.160 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_items": 0, 2024-05-09T08:49:13.160 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_bytes": 0, 2024-05-09T08:49:13.160 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_items": 0, 2024-05-09T08:49:13.160 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:49:13.160 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:49:13.160 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_bytes": 0, 2024-05-09T08:49:13.160 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_items": 0, 2024-05-09T08:49:13.160 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_bytes": 928, 2024-05-09T08:49:13.160 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_items": 2, 2024-05-09T08:49:13.160 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_bytes": 0, 2024-05-09T08:49:13.160 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_items": 0, 2024-05-09T08:49:13.161 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_bytes": 0, 2024-05-09T08:49:13.161 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_items": 0 2024-05-09T08:49:13.161 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-05-09T08:49:13.161 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:49:13.161 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-05-09T08:49:13.161 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-05-09T08:49:13.161 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-05-09T08:49:13.161 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 315, 2024-05-09T08:49:13.161 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 1562902, 2024-05-09T08:49:13.161 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:13.161 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 315, 2024-05-09T08:49:13.161 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-05-09T08:49:13.161 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-05-09T08:49:13.162 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 315, 2024-05-09T08:49:13.162 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 1562902, 2024-05-09T08:49:13.162 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-05-09T08:49:13.162 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-05-09T08:49:13.162 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-05-09T08:49:13.162 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-05-09T08:49:13.162 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:13.162 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-05-09T08:49:13.162 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-05-09T08:49:14.412 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-05-09T08:49:14.412 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-05-09T08:49:14.412 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-05-09T08:49:14.412 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-05-09T08:49:14.412 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-05-09T08:49:14.412 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-05-09T08:49:14.412 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-05-09T08:49:14.412 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-05-09T08:49:14.413 DEBUG:tasks.check_counter:Getting stats from c 2024-05-09T08:49:14.413 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph daemon mds.c perf dump 2024-05-09T08:49:14.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:14 smithi107 ceph-mon[30122]: pgmap v499: 129 pgs: 5 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 109 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-09T08:49:14.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:14 smithi157 ceph-mon[33614]: pgmap v499: 129 pgs: 5 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 109 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-09T08:49:14.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:14 smithi110 ceph-mon[33622]: pgmap v499: 129 pgs: 5 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 109 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-09T08:49:14.718 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:49:16.446 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:16 smithi157 ceph-mon[33614]: pgmap v500: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-09T08:49:16.446 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:16 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:49:16.478 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-05-09T08:49:16.478 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:49:16.478 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 186, 2024-05-09T08:49:16.478 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 250, 2024-05-09T08:49:16.478 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 48495, 2024-05-09T08:49:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 293033, 2024-05-09T08:49:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 22, 2024-05-09T08:49:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 6, 2024-05-09T08:49:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.026286119, 2024-05-09T08:49:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.005272058, 2024-05-09T08:49:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.011387763, 2024-05-09T08:49:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.003690746, 2024-05-09T08:49:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 249, 2024-05-09T08:49:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.030766474, 2024-05-09T08:49:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000123560 2024-05-09T08:49:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 166, 2024-05-09T08:49:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000038793, 2024-05-09T08:49:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000233 2024-05-09T08:49:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 7488, 2024-05-09T08:49:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 5984 2024-05-09T08:49:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:49:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 499, 2024-05-09T08:49:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 535, 2024-05-09T08:49:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 1703537, 2024-05-09T08:49:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 291521, 2024-05-09T08:49:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 12, 2024-05-09T08:49:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 6, 2024-05-09T08:49:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.071710990, 2024-05-09T08:49:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.024233219, 2024-05-09T08:49:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.049260142, 2024-05-09T08:49:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.008513213, 2024-05-09T08:49:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 534, 2024-05-09T08:49:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.039060982, 2024-05-09T08:49:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000073147 2024-05-09T08:49:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 104, 2024-05-09T08:49:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000020661, 2024-05-09T08:49:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000198 2024-05-09T08:49:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 1690432, 2024-05-09T08:49:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 241680 2024-05-09T08:49:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:49:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 5569, 2024-05-09T08:49:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 3279, 2024-05-09T08:49:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 1991529, 2024-05-09T08:49:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 2015961, 2024-05-09T08:49:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 55, 2024-05-09T08:49:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 6, 2024-05-09T08:49:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.318612536, 2024-05-09T08:49:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.137310065, 2024-05-09T08:49:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.236869922, 2024-05-09T08:49:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.005579160, 2024-05-09T08:49:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 3276, 2024-05-09T08:49:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.126451321, 2024-05-09T08:49:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000038599 2024-05-09T08:49:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 8335, 2024-05-09T08:49:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.001454572, 2024-05-09T08:49:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000174 2024-05-09T08:49:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 1248, 2024-05-09T08:49:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 561840 2024-05-09T08:49:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: "cct": { 2024-05-09T08:49:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: "total_workers": 1, 2024-05-09T08:49:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: "unhealthy_workers": 0 2024-05-09T08:49:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: "finisher-MDSRank": { 2024-05-09T08:49:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: "queue_len": 0, 2024-05-09T08:49:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: "complete_latency": { 2024-05-09T08:49:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 199, 2024-05-09T08:49:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.022613629, 2024-05-09T08:49:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000113636 2024-05-09T08:49:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: "finisher-PurgeQueue": { 2024-05-09T08:49:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "queue_len": 0, 2024-05-09T08:49:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "complete_latency": { 2024-05-09T08:49:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 23, 2024-05-09T08:49:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000810604, 2024-05-09T08:49:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000035243 2024-05-09T08:49:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "mds": { 2024-05-09T08:49:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "request": 30, 2024-05-09T08:49:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "reply": 22, 2024-05-09T08:49:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "reply_latency": { 2024-05-09T08:49:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 22, 2024-05-09T08:49:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.041907495, 2024-05-09T08:49:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.001904886 2024-05-09T08:49:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "slow_reply": 0, 2024-05-09T08:49:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "forward": 8, 2024-05-09T08:49:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_fetch_complete": 0, 2024-05-09T08:49:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_fetch_keys": 0, 2024-05-09T08:49:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_commit": 11, 2024-05-09T08:49:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_split": 0, 2024-05-09T08:49:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_merge": 0, 2024-05-09T08:49:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes": 22, 2024-05-09T08:49:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_top": 2, 2024-05-09T08:49:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_bottom": 8, 2024-05-09T08:49:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_pin_tail": 12, 2024-05-09T08:49:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_pinned": 17, 2024-05-09T08:49:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_expired": 174, 2024-05-09T08:49:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_with_caps": 1, 2024-05-09T08:49:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "caps": 1, 2024-05-09T08:49:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "subtrees": 3, 2024-05-09T08:49:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse": 245, 2024-05-09T08:49:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_hit": 148, 2024-05-09T08:49:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_forward": 1, 2024-05-09T08:49:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_discover": 83, 2024-05-09T08:49:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_dir_fetch": 0, 2024-05-09T08:49:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_remote_ino": 0, 2024-05-09T08:49:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_lock": 0, 2024-05-09T08:49:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "load_cent": 0, 2024-05-09T08:49:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "q": 0, 2024-05-09T08:49:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "exported": 3, 2024-05-09T08:49:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "exported_inodes": 8, 2024-05-09T08:49:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "imported": 3, 2024-05-09T08:49:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "imported_inodes": 7, 2024-05-09T08:49:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_dir_fetch": 0, 2024-05-09T08:49:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_backtrace_fetch": 84, 2024-05-09T08:49:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_peer_discover": 0, 2024-05-09T08:49:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rfiles": 46, 2024-05-09T08:49:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rbytes": 729438, 2024-05-09T08:49:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rsnaps": 6, 2024-05-09T08:49:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_backtrace_fetch": 84, 2024-05-09T08:49:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_set_tag": 0, 2024-05-09T08:49:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T08:49:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_inotable_repaired": 0, 2024-05-09T08:49:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dir_inodes": 21, 2024-05-09T08:49:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T08:49:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dirfrag_rstats": 21, 2024-05-09T08:49:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_file_inodes": 63, 2024-05-09T08:49:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_inode_file_caps": 0, 2024-05-09T08:49:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_revoke": 6, 2024-05-09T08:49:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_grant": 8, 2024-05-09T08:49:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T08:49:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T08:49:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T08:49:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_caps": 6, 2024-05-09T08:49:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_caps_dirty": 0, 2024-05-09T08:49:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_cap_release": 3, 2024-05-09T08:49:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "process_request_cap_release": 10 2024-05-09T08:49:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_cache": { 2024-05-09T08:49:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_update": 5, 2024-05-09T08:49:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_update_receipt": 289, 2024-05-09T08:49:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_try_discover": 185, 2024-05-09T08:49:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_send_discover": 191, 2024-05-09T08:49:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_handle_discover": 7, 2024-05-09T08:49:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays": 1, 2024-05-09T08:49:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays_delayed": 0, 2024-05-09T08:49:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays_enqueuing": 0, 2024-05-09T08:49:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_created": 4, 2024-05-09T08:49:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_enqueued": 3, 2024-05-09T08:49:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_reintegrated": 0, 2024-05-09T08:49:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_migrated": 0, 2024-05-09T08:49:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_processing": 0, 2024-05-09T08:49:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_enqueued": 0, 2024-05-09T08:49:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_prioritized": 0, 2024-05-09T08:49:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "recovery_started": 0, 2024-05-09T08:49:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "recovery_completed": 0, 2024-05-09T08:49:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_quiesce_path": 0, 2024-05-09T08:49:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_quiesce_inode": 0, 2024-05-09T08:49:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T08:49:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_exportdir": 3, 2024-05-09T08:49:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_flush": 0, 2024-05-09T08:49:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_fragmentdir": 0, 2024-05-09T08:49:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_fragstats": 0, 2024-05-09T08:49:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_inodestats": 0 2024-05-09T08:49:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_log": { 2024-05-09T08:49:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "evlrg": 0, 2024-05-09T08:49:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "evadd": 46, 2024-05-09T08:49:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "evex": 0, 2024-05-09T08:49:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "evtrm": 0, 2024-05-09T08:49:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "ev": 46, 2024-05-09T08:49:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "evexg": 0, 2024-05-09T08:49:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "evexd": 0, 2024-05-09T08:49:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "segadd": 2, 2024-05-09T08:49:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "segex": 0, 2024-05-09T08:49:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "segtrm": 0, 2024-05-09T08:49:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "seg": 2, 2024-05-09T08:49:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "segmjr": 2, 2024-05-09T08:49:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "segexg": 0, 2024-05-09T08:49:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "segexd": 0, 2024-05-09T08:49:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "expos": 4194304, 2024-05-09T08:49:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "wrpos": 4290457, 2024-05-09T08:49:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "rdpos": 0, 2024-05-09T08:49:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "jlat": { 2024-05-09T08:49:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 33, 2024-05-09T08:49:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.233024934, 2024-05-09T08:49:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.007061361 2024-05-09T08:49:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "replayed": 0 2024-05-09T08:49:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_mem": { 2024-05-09T08:49:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "ino": 24, 2024-05-09T08:49:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "ino+": 115, 2024-05-09T08:49:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "ino-": 91, 2024-05-09T08:49:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "dir": 22, 2024-05-09T08:49:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "dir+": 132, 2024-05-09T08:49:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "dir-": 110, 2024-05-09T08:49:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "dn": 22, 2024-05-09T08:49:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "dn+": 199, 2024-05-09T08:49:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "dn-": 177, 2024-05-09T08:49:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "cap": 1, 2024-05-09T08:49:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "cap+": 14, 2024-05-09T08:49:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "cap-": 13, 2024-05-09T08:49:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "rss": 57668, 2024-05-09T08:49:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "heap": 223612 2024-05-09T08:49:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_server": { 2024-05-09T08:49:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "dispatch_client_request": 46, 2024-05-09T08:49:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "dispatch_server_request": 8, 2024-05-09T08:49:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_request": 30, 2024-05-09T08:49:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_session": 40, 2024-05-09T08:49:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_peer_request": 22, 2024-05-09T08:49:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "req_create_latency": { 2024-05-09T08:49:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getattr_latency": { 2024-05-09T08:49:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 11, 2024-05-09T08:49:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.003240989, 2024-05-09T08:49:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000294635 2024-05-09T08:49:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getfilelock_latency": { 2024-05-09T08:49:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "req_link_latency": { 2024-05-09T08:49:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookup_latency": { 2024-05-09T08:49:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 3, 2024-05-09T08:49:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000673446, 2024-05-09T08:49:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000224482 2024-05-09T08:49:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookuphash_latency": { 2024-05-09T08:49:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupino_latency": { 2024-05-09T08:49:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupname_latency": { 2024-05-09T08:49:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupparent_latency": { 2024-05-09T08:49:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupsnap_latency": { 2024-05-09T08:49:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lssnap_latency": { 2024-05-09T08:49:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mkdir_latency": { 2024-05-09T08:49:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mknod_latency": { 2024-05-09T08:49:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mksnap_latency": { 2024-05-09T08:49:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "req_open_latency": { 2024-05-09T08:49:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "req_readdir_latency": { 2024-05-09T08:49:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 4, 2024-05-09T08:49:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.001923914, 2024-05-09T08:49:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000480978 2024-05-09T08:49:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rename_latency": { 2024-05-09T08:49:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "req_renamesnap_latency": { 2024-05-09T08:49:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "req_snapdiff_latency": { 2024-05-09T08:49:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmdir_latency": { 2024-05-09T08:49:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 1, 2024-05-09T08:49:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.017748734, 2024-05-09T08:49:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.017748734 2024-05-09T08:49:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmsnap_latency": { 2024-05-09T08:49:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmxattr_latency": { 2024-05-09T08:49:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setattr_latency": { 2024-05-09T08:49:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setdirlayout_latency": { 2024-05-09T08:49:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setfilelock_latency": { 2024-05-09T08:49:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setlayout_latency": { 2024-05-09T08:49:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setxattr_latency": { 2024-05-09T08:49:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "req_symlink_latency": { 2024-05-09T08:49:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "req_unlink_latency": { 2024-05-09T08:49:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 3, 2024-05-09T08:49:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.018320412, 2024-05-09T08:49:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.006106804 2024-05-09T08:49:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "cap_revoke_eviction": 0, 2024-05-09T08:49:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "cap_acquisition_throttle": 0, 2024-05-09T08:49:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getvxattr_latency": { 2024-05-09T08:49:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_sessions": { 2024-05-09T08:49:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "session_count": 2, 2024-05-09T08:49:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "session_add": 6, 2024-05-09T08:49:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "session_remove": 4, 2024-05-09T08:49:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "sessions_open": 2, 2024-05-09T08:49:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "sessions_stale": 0, 2024-05-09T08:49:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "total_load": 7, 2024-05-09T08:49:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "average_load": 3, 2024-05-09T08:49:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "avg_session_uptime": 935, 2024-05-09T08:49:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "mdthresh_evicted": 0 2024-05-09T08:49:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "mempool": { 2024-05-09T08:49:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:49:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "bloom_filter_items": 0, 2024-05-09T08:49:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:49:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:49:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:49:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:49:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:49:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:49:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:49:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:49:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:49:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:49:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:49:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:49:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:49:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_extent_items": 0, 2024-05-09T08:49:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:49:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_blob_items": 0, 2024-05-09T08:49:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:49:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:49:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:49:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:49:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:49:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:49:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:49:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_txc_items": 0, 2024-05-09T08:49:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:49:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:49:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:49:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_items": 0, 2024-05-09T08:49:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_bytes": 0, 2024-05-09T08:49:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_items": 0, 2024-05-09T08:49:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:49:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:49:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:49:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:49:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_anon_bytes": 362701, 2024-05-09T08:49:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_anon_items": 132, 2024-05-09T08:49:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:49:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_meta_items": 0, 2024-05-09T08:49:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_bytes": 0, 2024-05-09T08:49:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_items": 0, 2024-05-09T08:49:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:49:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_mapbl_items": 0, 2024-05-09T08:49:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:49:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_pglog_items": 0, 2024-05-09T08:49:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_bytes": 9656, 2024-05-09T08:49:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_items": 186, 2024-05-09T08:49:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:49:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:49:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "pgmap_bytes": 0, 2024-05-09T08:49:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "pgmap_items": 0, 2024-05-09T08:49:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_co_bytes": 105502, 2024-05-09T08:49:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_co_items": 564, 2024-05-09T08:49:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_1_bytes": 0, 2024-05-09T08:49:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_1_items": 0, 2024-05-09T08:49:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_2_bytes": 0, 2024-05-09T08:49:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_2_items": 0 2024-05-09T08:49:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "objecter": { 2024-05-09T08:49:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "op_active": 0, 2024-05-09T08:49:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "op_laggy": 0, 2024-05-09T08:49:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "op_send": 183, 2024-05-09T08:49:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "op_send_bytes": 110436, 2024-05-09T08:49:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "op_resend": 33, 2024-05-09T08:49:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "op_reply": 183, 2024-05-09T08:49:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "op_latency": { 2024-05-09T08:49:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 183, 2024-05-09T08:49:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 2.129960454, 2024-05-09T08:49:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.011639128 2024-05-09T08:49:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "op_inflight": 0, 2024-05-09T08:49:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "oplen_avg": { 2024-05-09T08:49:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 183, 2024-05-09T08:49:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 216 2024-05-09T08:49:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "op": 183, 2024-05-09T08:49:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "op_r": 86, 2024-05-09T08:49:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "op_w": 97, 2024-05-09T08:49:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "op_rmw": 0, 2024-05-09T08:49:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "op_pg": 0, 2024-05-09T08:49:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_stat": 0, 2024-05-09T08:49:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_create": 10, 2024-05-09T08:49:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_read": 2, 2024-05-09T08:49:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_write": 35, 2024-05-09T08:49:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_writefull": 13, 2024-05-09T08:49:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_writesame": 0, 2024-05-09T08:49:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_append": 0, 2024-05-09T08:49:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_zero": 0, 2024-05-09T08:49:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_truncate": 0, 2024-05-09T08:49:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_delete": 18, 2024-05-09T08:49:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_mapext": 0, 2024-05-09T08:49:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_sparse_read": 0, 2024-05-09T08:49:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_clonerange": 0, 2024-05-09T08:49:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_getxattr": 84, 2024-05-09T08:49:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_setxattr": 20, 2024-05-09T08:49:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_cmpxattr": 0, 2024-05-09T08:49:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_rmxattr": 0, 2024-05-09T08:49:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_resetxattrs": 0, 2024-05-09T08:49:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_call": 0, 2024-05-09T08:49:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_watch": 0, 2024-05-09T08:49:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_notify": 0, 2024-05-09T08:49:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T08:49:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_pgls": 0, 2024-05-09T08:49:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_pgls_filter": 0, 2024-05-09T08:49:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_other": 0, 2024-05-09T08:49:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_active": 0, 2024-05-09T08:49:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_send": 0, 2024-05-09T08:49:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_resend": 0, 2024-05-09T08:49:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_ping": 0, 2024-05-09T08:49:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_active": 0, 2024-05-09T08:49:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_send": 0, 2024-05-09T08:49:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_resend": 0, 2024-05-09T08:49:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_active": 0, 2024-05-09T08:49:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_send": 0, 2024-05-09T08:49:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_resend": 0, 2024-05-09T08:49:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_active": 0, 2024-05-09T08:49:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_send": 0, 2024-05-09T08:49:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_resend": 0, 2024-05-09T08:49:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "command_active": 0, 2024-05-09T08:49:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "command_send": 0, 2024-05-09T08:49:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "command_resend": 0, 2024-05-09T08:49:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "map_epoch": 114, 2024-05-09T08:49:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "map_full": 0, 2024-05-09T08:49:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "map_inc": 25, 2024-05-09T08:49:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_sessions": 11, 2024-05-09T08:49:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_session_open": 15, 2024-05-09T08:49:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_session_close": 4, 2024-05-09T08:49:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_laggy": 0, 2024-05-09T08:49:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_wr": 28, 2024-05-09T08:49:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_rd": 0, 2024-05-09T08:49:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_del": 6 2024-05-09T08:49:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "oft": { 2024-05-09T08:49:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_objs": 1, 2024-05-09T08:49:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_kv_pairs": 0, 2024-05-09T08:49:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_updates": 0, 2024-05-09T08:49:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_removes": 0 2024-05-09T08:49:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "osd-slow-ops": { 2024-05-09T08:49:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: "slow_ops_count": 0 2024-05-09T08:49:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: "purge_queue": { 2024-05-09T08:49:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_ops": 0, 2024-05-09T08:49:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_ops_high_water": 4, 2024-05-09T08:49:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing": 0, 2024-05-09T08:49:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_high_water": 2, 2024-05-09T08:49:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executed_ops": 6, 2024-05-09T08:49:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executed": 3, 2024-05-09T08:49:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_item_in_journal": 0 2024-05-09T08:49:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:49:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2024-05-09T08:49:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 104857600, 2024-05-09T08:49:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2024-05-09T08:49:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 6250, 2024-05-09T08:49:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 3154972, 2024-05-09T08:49:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 6250, 2024-05-09T08:49:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2024-05-09T08:49:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2024-05-09T08:49:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 6250, 2024-05-09T08:49:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 3154972, 2024-05-09T08:49:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2024-05-09T08:49:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-objecter_bytes": { 2024-05-09T08:49:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2024-05-09T08:49:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 104857600, 2024-05-09T08:49:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2024-05-09T08:49:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2024-05-09T08:49:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2024-05-09T08:49:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 183, 2024-05-09T08:49:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 110907, 2024-05-09T08:49:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 183, 2024-05-09T08:49:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 110907, 2024-05-09T08:49:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2024-05-09T08:49:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-objecter_ops": { 2024-05-09T08:49:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2024-05-09T08:49:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 1024, 2024-05-09T08:49:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2024-05-09T08:49:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2024-05-09T08:49:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2024-05-09T08:49:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 183, 2024-05-09T08:49:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 183, 2024-05-09T08:49:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 183, 2024-05-09T08:49:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 183, 2024-05-09T08:49:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2024-05-09T08:49:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-write_buf_throttle": { 2024-05-09T08:49:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2024-05-09T08:49:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 3758096384, 2024-05-09T08:49:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2024-05-09T08:49:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 3, 2024-05-09T08:49:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 471, 2024-05-09T08:49:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 3, 2024-05-09T08:49:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2024-05-09T08:49:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2024-05-09T08:49:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 2, 2024-05-09T08:49:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 471, 2024-05-09T08:49:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2024-05-09T08:49:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-write_buf_throttle-0x562dce136500": { 2024-05-09T08:49:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2024-05-09T08:49:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 3758096384, 2024-05-09T08:49:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2024-05-09T08:49:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 46, 2024-05-09T08:49:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 96153, 2024-05-09T08:49:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 46, 2024-05-09T08:49:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2024-05-09T08:49:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2024-05-09T08:49:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 33, 2024-05-09T08:49:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 96153, 2024-05-09T08:49:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2024-05-09T08:49:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:16.525 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-05-09T08:49:16.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:16 smithi107 ceph-mon[30122]: pgmap v500: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-09T08:49:16.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:16 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:49:16.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:16 smithi110 ceph-mon[33622]: pgmap v500: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-09T08:49:16.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:16 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T08:49:18.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:18 smithi107 ceph-mon[30122]: osdmap e115: 12 total, 12 up, 12 in 2024-05-09T08:49:18.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:18 smithi107 ceph-mon[30122]: pgmap v502: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-09T08:49:18.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:18 smithi157 ceph-mon[33614]: osdmap e115: 12 total, 12 up, 12 in 2024-05-09T08:49:18.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:18 smithi157 ceph-mon[33614]: pgmap v502: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-09T08:49:18.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:18 smithi110 ceph-mon[33622]: osdmap e115: 12 total, 12 up, 12 in 2024-05-09T08:49:18.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:18 smithi110 ceph-mon[33622]: pgmap v502: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-09T08:49:20.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:20 smithi107 ceph-mon[30122]: pgmap v503: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-09T08:49:20.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:20 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:49:20.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:20 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:49:20.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:20 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:49:20.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:20 smithi157 ceph-mon[33614]: pgmap v503: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-09T08:49:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:20 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:49:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:20 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:49:20.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:20 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:49:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:20 smithi110 ceph-mon[33622]: pgmap v503: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-09T08:49:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:20 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T08:49:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:20 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T08:49:20.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:20 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' 2024-05-09T08:49:21.692 INFO:tasks.check_counter:Daemon mds.c mds.exported=3 2024-05-09T08:49:21.692 INFO:tasks.check_counter:Daemon mds.c mds.imported=3 2024-05-09T08:49:21.692 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=5 2024-05-09T08:49:21.693 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=289 2024-05-09T08:49:21.693 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-05-09T08:49:21.693 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-05-09T08:49:21.693 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T08:49:21.693 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-05-09T08:49:21.693 DEBUG:tasks.check_counter:Getting stats from f 2024-05-09T08:49:21.693 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph daemon mds.f perf dump 2024-05-09T08:49:21.952 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:49:22.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:22 smithi157 ceph-mon[33614]: pgmap v504: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-09T08:49:22.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:22 smithi157 ceph-mon[33614]: osdmap e116: 12 total, 12 up, 12 in 2024-05-09T08:49:22.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:22 smithi107 ceph-mon[30122]: pgmap v504: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-09T08:49:22.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:22 smithi107 ceph-mon[30122]: osdmap e116: 12 total, 12 up, 12 in 2024-05-09T08:49:22.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:22 smithi110 ceph-mon[33622]: pgmap v504: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-09T08:49:22.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:22 smithi110 ceph-mon[33622]: osdmap e116: 12 total, 12 up, 12 in 2024-05-09T08:49:23.708 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-05-09T08:49:23.708 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:49:23.708 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 0, 2024-05-09T08:49:23.708 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 2, 2024-05-09T08:49:23.709 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 0, 2024-05-09T08:49:23.709 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 0, 2024-05-09T08:49:23.709 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 4, 2024-05-09T08:49:23.709 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 0, 2024-05-09T08:49:23.709 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.001205236, 2024-05-09T08:49:23.709 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.000000000, 2024-05-09T08:49:23.709 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.000299630, 2024-05-09T08:49:23.709 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T08:49:23.709 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:23.709 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:23.709 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:23.709 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:23.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:23.710 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:23.710 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:23.710 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:23.710 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:23.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:23.710 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T08:49:23.710 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T08:49:23.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:23.710 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:49:23.710 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 41, 2024-05-09T08:49:23.710 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 38, 2024-05-09T08:49:23.710 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 153664, 2024-05-09T08:49:23.710 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 21264, 2024-05-09T08:49:23.711 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 2, 2024-05-09T08:49:23.711 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 1, 2024-05-09T08:49:23.711 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.007087071, 2024-05-09T08:49:23.711 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.001597340, 2024-05-09T08:49:23.711 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.005012855, 2024-05-09T08:49:23.712 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.001277636, 2024-05-09T08:49:23.712 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:23.712 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 38, 2024-05-09T08:49:23.712 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.003139455, 2024-05-09T08:49:23.712 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000082617 2024-05-09T08:49:23.712 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:23.712 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:23.712 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:23.712 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:23.712 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:23.713 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:23.713 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 153664, 2024-05-09T08:49:23.713 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 21264 2024-05-09T08:49:23.713 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:23.713 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:49:23.713 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 1, 2024-05-09T08:49:23.713 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 27, 2024-05-09T08:49:23.713 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 160, 2024-05-09T08:49:23.713 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 82416, 2024-05-09T08:49:23.713 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 3, 2024-05-09T08:49:23.714 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 2, 2024-05-09T08:49:23.714 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.002469681, 2024-05-09T08:49:23.714 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.001123010, 2024-05-09T08:49:23.714 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.000967658, 2024-05-09T08:49:23.714 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T08:49:23.714 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:23.714 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 25, 2024-05-09T08:49:23.714 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.001531421, 2024-05-09T08:49:23.714 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000061256 2024-05-09T08:49:23.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:23.714 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:23.715 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:23.715 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:23.715 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:23.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:23.715 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 160, 2024-05-09T08:49:23.715 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 82416 2024-05-09T08:49:23.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:23.715 INFO:teuthology.orchestra.run.smithi157.stdout: "cct": { 2024-05-09T08:49:23.715 INFO:teuthology.orchestra.run.smithi157.stdout: "total_workers": 0, 2024-05-09T08:49:23.715 INFO:teuthology.orchestra.run.smithi157.stdout: "unhealthy_workers": 0 2024-05-09T08:49:23.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:23.715 INFO:teuthology.orchestra.run.smithi157.stdout: "mempool": { 2024-05-09T08:49:23.715 INFO:teuthology.orchestra.run.smithi157.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:49:23.716 INFO:teuthology.orchestra.run.smithi157.stdout: "bloom_filter_items": 0, 2024-05-09T08:49:23.716 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:49:23.716 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:49:23.716 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:49:23.716 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:49:23.716 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:49:23.716 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:49:23.716 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:49:23.716 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:49:23.716 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:49:23.716 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:49:23.716 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:49:23.716 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:49:23.716 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:49:23.717 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_extent_items": 0, 2024-05-09T08:49:23.717 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:49:23.717 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_blob_items": 0, 2024-05-09T08:49:23.717 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:49:23.717 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:49:23.717 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:49:23.717 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:49:23.717 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:49:23.717 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:49:23.717 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:49:23.717 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_txc_items": 0, 2024-05-09T08:49:23.717 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:49:23.717 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:49:23.717 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:49:23.718 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_items": 0, 2024-05-09T08:49:23.718 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_bytes": 0, 2024-05-09T08:49:23.718 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_items": 0, 2024-05-09T08:49:23.718 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:49:23.718 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:49:23.718 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:49:23.718 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:49:23.718 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_anon_bytes": 149070, 2024-05-09T08:49:23.718 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_anon_items": 26, 2024-05-09T08:49:23.718 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:49:23.718 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_meta_items": 0, 2024-05-09T08:49:23.718 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_bytes": 0, 2024-05-09T08:49:23.718 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_items": 0, 2024-05-09T08:49:23.718 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:49:23.719 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_mapbl_items": 0, 2024-05-09T08:49:23.719 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:49:23.719 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_pglog_items": 0, 2024-05-09T08:49:23.719 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_bytes": 0, 2024-05-09T08:49:23.719 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_items": 0, 2024-05-09T08:49:23.719 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:49:23.719 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:49:23.719 INFO:teuthology.orchestra.run.smithi157.stdout: "pgmap_bytes": 0, 2024-05-09T08:49:23.719 INFO:teuthology.orchestra.run.smithi157.stdout: "pgmap_items": 0, 2024-05-09T08:49:23.719 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_co_bytes": 928, 2024-05-09T08:49:23.719 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_co_items": 2, 2024-05-09T08:49:23.719 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_1_bytes": 0, 2024-05-09T08:49:23.719 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_1_items": 0, 2024-05-09T08:49:23.720 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_2_bytes": 0, 2024-05-09T08:49:23.720 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_2_items": 0 2024-05-09T08:49:23.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:23.720 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:49:23.720 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2024-05-09T08:49:23.720 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 104857600, 2024-05-09T08:49:23.720 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2024-05-09T08:49:23.720 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 38, 2024-05-09T08:49:23.720 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 145448, 2024-05-09T08:49:23.720 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:23.720 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 38, 2024-05-09T08:49:23.720 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2024-05-09T08:49:23.720 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2024-05-09T08:49:23.720 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 38, 2024-05-09T08:49:23.721 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 145448, 2024-05-09T08:49:23.721 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2024-05-09T08:49:23.721 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:23.721 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:23.721 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:23.721 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:23.721 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:23.721 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-05-09T08:49:24.393 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:24 smithi157 ceph-mon[33614]: pgmap v506: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-09T08:49:24.395 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.f 2024-05-09T08:49:24.395 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.f 2024-05-09T08:49:24.395 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.f 2024-05-09T08:49:24.395 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.f 2024-05-09T08:49:24.395 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.f 2024-05-09T08:49:24.395 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.f 2024-05-09T08:49:24.395 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.f 2024-05-09T08:49:24.396 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.f 2024-05-09T08:49:24.396 DEBUG:tasks.check_counter:Getting stats from i 2024-05-09T08:49:24.396 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph daemon mds.i perf dump 2024-05-09T08:49:24.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:24 smithi107 ceph-mon[30122]: pgmap v506: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-09T08:49:24.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:24 smithi110 ceph-mon[33622]: pgmap v506: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-09T08:49:24.644 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:49:26.497 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:26 smithi157 ceph-mon[33614]: pgmap v507: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-05-09T08:49:26.512 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-05-09T08:49:26.513 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:49:26.513 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 10035, 2024-05-09T08:49:26.513 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 11602, 2024-05-09T08:49:26.513 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 3567435, 2024-05-09T08:49:26.513 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 29449459, 2024-05-09T08:49:26.513 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 23, 2024-05-09T08:49:26.514 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 4, 2024-05-09T08:49:26.514 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.922755683, 2024-05-09T08:49:26.514 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.305613767, 2024-05-09T08:49:26.514 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.297684220, 2024-05-09T08:49:26.514 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.233501608, 2024-05-09T08:49:26.514 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:26.514 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 11604, 2024-05-09T08:49:26.514 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 2.192559233, 2024-05-09T08:49:26.514 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000188948 2024-05-09T08:49:26.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.515 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:26.515 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 9381, 2024-05-09T08:49:26.515 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.002229873, 2024-05-09T08:49:26.515 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000237 2024-05-09T08:49:26.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.515 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 538544, 2024-05-09T08:49:26.515 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 158800 2024-05-09T08:49:26.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.516 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:49:26.516 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 9757, 2024-05-09T08:49:26.516 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 12419, 2024-05-09T08:49:26.516 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 3645064, 2024-05-09T08:49:26.516 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 26501909, 2024-05-09T08:49:26.516 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 326, 2024-05-09T08:49:26.516 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 9, 2024-05-09T08:49:26.516 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 1.032084299, 2024-05-09T08:49:26.516 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.409604563, 2024-05-09T08:49:26.516 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.359604803, 2024-05-09T08:49:26.516 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.198481221, 2024-05-09T08:49:26.516 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:26.516 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 12419, 2024-05-09T08:49:26.516 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 1.377850492, 2024-05-09T08:49:26.517 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000110946 2024-05-09T08:49:26.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.517 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:26.517 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 8243, 2024-05-09T08:49:26.517 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.001620889, 2024-05-09T08:49:26.517 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000196 2024-05-09T08:49:26.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.517 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 42848, 2024-05-09T08:49:26.517 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 577440 2024-05-09T08:49:26.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.517 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:49:26.517 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 19543, 2024-05-09T08:49:26.518 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 22459, 2024-05-09T08:49:26.518 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 8913276, 2024-05-09T08:49:26.518 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 16825719, 2024-05-09T08:49:26.518 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 19, 2024-05-09T08:49:26.518 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 8, 2024-05-09T08:49:26.518 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 1.566816969, 2024-05-09T08:49:26.518 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.497208350, 2024-05-09T08:49:26.518 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.781851320, 2024-05-09T08:49:26.518 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.297371658, 2024-05-09T08:49:26.518 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:26.518 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 22452, 2024-05-09T08:49:26.518 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 2.385748983, 2024-05-09T08:49:26.518 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000106259 2024-05-09T08:49:26.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.519 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:26.519 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 24016, 2024-05-09T08:49:26.519 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.004402212, 2024-05-09T08:49:26.519 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000183 2024-05-09T08:49:26.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.519 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 1793456, 2024-05-09T08:49:26.519 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 262176 2024-05-09T08:49:26.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.519 INFO:teuthology.orchestra.run.smithi157.stdout: "cct": { 2024-05-09T08:49:26.519 INFO:teuthology.orchestra.run.smithi157.stdout: "total_workers": 1, 2024-05-09T08:49:26.519 INFO:teuthology.orchestra.run.smithi157.stdout: "unhealthy_workers": 0 2024-05-09T08:49:26.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.519 INFO:teuthology.orchestra.run.smithi157.stdout: "finisher-MDSRank": { 2024-05-09T08:49:26.520 INFO:teuthology.orchestra.run.smithi157.stdout: "queue_len": 0, 2024-05-09T08:49:26.520 INFO:teuthology.orchestra.run.smithi157.stdout: "complete_latency": { 2024-05-09T08:49:26.520 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 12003, 2024-05-09T08:49:26.520 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 6.350849265, 2024-05-09T08:49:26.520 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000529105 2024-05-09T08:49:26.520 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:26.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.520 INFO:teuthology.orchestra.run.smithi157.stdout: "finisher-PurgeQueue": { 2024-05-09T08:49:26.520 INFO:teuthology.orchestra.run.smithi157.stdout: "queue_len": 0, 2024-05-09T08:49:26.520 INFO:teuthology.orchestra.run.smithi157.stdout: "complete_latency": { 2024-05-09T08:49:26.520 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 5911, 2024-05-09T08:49:26.520 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.278606118, 2024-05-09T08:49:26.521 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000047133 2024-05-09T08:49:26.521 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:26.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.521 INFO:teuthology.orchestra.run.smithi157.stdout: "mds": { 2024-05-09T08:49:26.521 INFO:teuthology.orchestra.run.smithi157.stdout: "request": 4425, 2024-05-09T08:49:26.521 INFO:teuthology.orchestra.run.smithi157.stdout: "reply": 4413, 2024-05-09T08:49:26.521 INFO:teuthology.orchestra.run.smithi157.stdout: "reply_latency": { 2024-05-09T08:49:26.521 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 4413, 2024-05-09T08:49:26.521 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 144.562682860, 2024-05-09T08:49:26.521 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.032758369 2024-05-09T08:49:26.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.522 INFO:teuthology.orchestra.run.smithi157.stdout: "slow_reply": 0, 2024-05-09T08:49:26.522 INFO:teuthology.orchestra.run.smithi157.stdout: "forward": 12, 2024-05-09T08:49:26.522 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_fetch_complete": 25, 2024-05-09T08:49:26.522 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_fetch_keys": 22, 2024-05-09T08:49:26.522 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_commit": 842, 2024-05-09T08:49:26.522 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_split": 3, 2024-05-09T08:49:26.522 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_merge": 0, 2024-05-09T08:49:26.523 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes": 1471, 2024-05-09T08:49:26.523 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_top": 394, 2024-05-09T08:49:26.523 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_bottom": 1066, 2024-05-09T08:49:26.523 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_pin_tail": 11, 2024-05-09T08:49:26.523 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_pinned": 907, 2024-05-09T08:49:26.523 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_expired": 19, 2024-05-09T08:49:26.524 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_with_caps": 2, 2024-05-09T08:49:26.524 INFO:teuthology.orchestra.run.smithi157.stdout: "caps": 3, 2024-05-09T08:49:26.524 INFO:teuthology.orchestra.run.smithi157.stdout: "subtrees": 5, 2024-05-09T08:49:26.524 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse": 6435, 2024-05-09T08:49:26.524 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_hit": 6112, 2024-05-09T08:49:26.524 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_forward": 0, 2024-05-09T08:49:26.525 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_discover": 1, 2024-05-09T08:49:26.525 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_dir_fetch": 0, 2024-05-09T08:49:26.525 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_remote_ino": 0, 2024-05-09T08:49:26.525 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_lock": 44, 2024-05-09T08:49:26.525 INFO:teuthology.orchestra.run.smithi157.stdout: "load_cent": 0, 2024-05-09T08:49:26.525 INFO:teuthology.orchestra.run.smithi157.stdout: "q": 0, 2024-05-09T08:49:26.526 INFO:teuthology.orchestra.run.smithi157.stdout: "exported": 6, 2024-05-09T08:49:26.526 INFO:teuthology.orchestra.run.smithi157.stdout: "exported_inodes": 12, 2024-05-09T08:49:26.526 INFO:teuthology.orchestra.run.smithi157.stdout: "imported": 6, 2024-05-09T08:49:26.526 INFO:teuthology.orchestra.run.smithi157.stdout: "imported_inodes": 16, 2024-05-09T08:49:26.526 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_dir_fetch": 0, 2024-05-09T08:49:26.527 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_backtrace_fetch": 7727, 2024-05-09T08:49:26.527 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_peer_discover": 0, 2024-05-09T08:49:26.527 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rfiles": 4, 2024-05-09T08:49:26.527 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rbytes": 234, 2024-05-09T08:49:26.527 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rsnaps": 6, 2024-05-09T08:49:26.527 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_backtrace_fetch": 7727, 2024-05-09T08:49:26.528 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_set_tag": 0, 2024-05-09T08:49:26.528 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T08:49:26.528 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_inotable_repaired": 0, 2024-05-09T08:49:26.528 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dir_inodes": 1902, 2024-05-09T08:49:26.528 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dir_base_inodes": 25, 2024-05-09T08:49:26.528 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dirfrag_rstats": 1877, 2024-05-09T08:49:26.529 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_file_inodes": 5825, 2024-05-09T08:49:26.529 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_inode_file_caps": 0, 2024-05-09T08:49:26.529 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_revoke": 1794, 2024-05-09T08:49:26.529 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_grant": 3317, 2024-05-09T08:49:26.529 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_trunc": 6, 2024-05-09T08:49:26.530 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T08:49:26.530 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T08:49:26.530 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_caps": 5953, 2024-05-09T08:49:26.530 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_caps_dirty": 2951, 2024-05-09T08:49:26.530 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_cap_release": 27, 2024-05-09T08:49:26.530 INFO:teuthology.orchestra.run.smithi157.stdout: "process_request_cap_release": 2870 2024-05-09T08:49:26.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.531 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_cache": { 2024-05-09T08:49:26.531 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_update": 202, 2024-05-09T08:49:26.531 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_update_receipt": 3, 2024-05-09T08:49:26.531 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_try_discover": 2, 2024-05-09T08:49:26.531 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_send_discover": 2, 2024-05-09T08:49:26.532 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_handle_discover": 372, 2024-05-09T08:49:26.532 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays": 82, 2024-05-09T08:49:26.532 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays_delayed": 0, 2024-05-09T08:49:26.532 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays_enqueuing": 0, 2024-05-09T08:49:26.532 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_created": 927, 2024-05-09T08:49:26.532 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_enqueued": 784, 2024-05-09T08:49:26.532 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_reintegrated": 61, 2024-05-09T08:49:26.533 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_migrated": 0, 2024-05-09T08:49:26.533 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_processing": 0, 2024-05-09T08:49:26.533 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_enqueued": 0, 2024-05-09T08:49:26.533 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_prioritized": 0, 2024-05-09T08:49:26.533 INFO:teuthology.orchestra.run.smithi157.stdout: "recovery_started": 0, 2024-05-09T08:49:26.533 INFO:teuthology.orchestra.run.smithi157.stdout: "recovery_completed": 0, 2024-05-09T08:49:26.534 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_quiesce_path": 0, 2024-05-09T08:49:26.534 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_quiesce_inode": 0, 2024-05-09T08:49:26.534 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_enqueue_scrub": 25, 2024-05-09T08:49:26.534 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_exportdir": 6, 2024-05-09T08:49:26.534 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_flush": 0, 2024-05-09T08:49:26.534 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_fragmentdir": 3, 2024-05-09T08:49:26.535 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_fragstats": 0, 2024-05-09T08:49:26.535 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_inodestats": 0 2024-05-09T08:49:26.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.535 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_log": { 2024-05-09T08:49:26.535 INFO:teuthology.orchestra.run.smithi157.stdout: "evlrg": 0, 2024-05-09T08:49:26.535 INFO:teuthology.orchestra.run.smithi157.stdout: "evadd": 8247, 2024-05-09T08:49:26.535 INFO:teuthology.orchestra.run.smithi157.stdout: "evex": 7229, 2024-05-09T08:49:26.535 INFO:teuthology.orchestra.run.smithi157.stdout: "evtrm": 7229, 2024-05-09T08:49:26.536 INFO:teuthology.orchestra.run.smithi157.stdout: "ev": 1018, 2024-05-09T08:49:26.536 INFO:teuthology.orchestra.run.smithi157.stdout: "evexg": 0, 2024-05-09T08:49:26.536 INFO:teuthology.orchestra.run.smithi157.stdout: "evexd": 28, 2024-05-09T08:49:26.536 INFO:teuthology.orchestra.run.smithi157.stdout: "segadd": 75, 2024-05-09T08:49:26.536 INFO:teuthology.orchestra.run.smithi157.stdout: "segex": 74, 2024-05-09T08:49:26.536 INFO:teuthology.orchestra.run.smithi157.stdout: "segtrm": 74, 2024-05-09T08:49:26.536 INFO:teuthology.orchestra.run.smithi157.stdout: "seg": 1, 2024-05-09T08:49:26.536 INFO:teuthology.orchestra.run.smithi157.stdout: "segmjr": 1, 2024-05-09T08:49:26.537 INFO:teuthology.orchestra.run.smithi157.stdout: "segexg": 0, 2024-05-09T08:49:26.537 INFO:teuthology.orchestra.run.smithi157.stdout: "segexd": 2, 2024-05-09T08:49:26.537 INFO:teuthology.orchestra.run.smithi157.stdout: "expos": 46230233, 2024-05-09T08:49:26.537 INFO:teuthology.orchestra.run.smithi157.stdout: "wrpos": 50024101, 2024-05-09T08:49:26.537 INFO:teuthology.orchestra.run.smithi157.stdout: "rdpos": 0, 2024-05-09T08:49:26.537 INFO:teuthology.orchestra.run.smithi157.stdout: "jlat": { 2024-05-09T08:49:26.537 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 2805, 2024-05-09T08:49:26.538 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 40.811160243, 2024-05-09T08:49:26.538 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.014549433 2024-05-09T08:49:26.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.538 INFO:teuthology.orchestra.run.smithi157.stdout: "replayed": 0 2024-05-09T08:49:26.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.539 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_mem": { 2024-05-09T08:49:26.539 INFO:teuthology.orchestra.run.smithi157.stdout: "ino": 981, 2024-05-09T08:49:26.539 INFO:teuthology.orchestra.run.smithi157.stdout: "ino+": 1802, 2024-05-09T08:49:26.539 INFO:teuthology.orchestra.run.smithi157.stdout: "ino-": 821, 2024-05-09T08:49:26.539 INFO:teuthology.orchestra.run.smithi157.stdout: "dir": 122, 2024-05-09T08:49:26.539 INFO:teuthology.orchestra.run.smithi157.stdout: "dir+": 166, 2024-05-09T08:49:26.540 INFO:teuthology.orchestra.run.smithi157.stdout: "dir-": 44, 2024-05-09T08:49:26.540 INFO:teuthology.orchestra.run.smithi157.stdout: "dn": 1471, 2024-05-09T08:49:26.540 INFO:teuthology.orchestra.run.smithi157.stdout: "dn+": 2520, 2024-05-09T08:49:26.540 INFO:teuthology.orchestra.run.smithi157.stdout: "dn-": 1049, 2024-05-09T08:49:26.540 INFO:teuthology.orchestra.run.smithi157.stdout: "cap": 3, 2024-05-09T08:49:26.540 INFO:teuthology.orchestra.run.smithi157.stdout: "cap+": 891, 2024-05-09T08:49:26.541 INFO:teuthology.orchestra.run.smithi157.stdout: "cap-": 888, 2024-05-09T08:49:26.541 INFO:teuthology.orchestra.run.smithi157.stdout: "rss": 72064, 2024-05-09T08:49:26.541 INFO:teuthology.orchestra.run.smithi157.stdout: "heap": 231804 2024-05-09T08:49:26.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.541 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_server": { 2024-05-09T08:49:26.541 INFO:teuthology.orchestra.run.smithi157.stdout: "dispatch_client_request": 5794, 2024-05-09T08:49:26.542 INFO:teuthology.orchestra.run.smithi157.stdout: "dispatch_server_request": 12, 2024-05-09T08:49:26.542 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_request": 4425, 2024-05-09T08:49:26.542 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_session": 52, 2024-05-09T08:49:26.542 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_peer_request": 56, 2024-05-09T08:49:26.542 INFO:teuthology.orchestra.run.smithi157.stdout: "req_create_latency": { 2024-05-09T08:49:26.542 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 765, 2024-05-09T08:49:26.542 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 9.639671901, 2024-05-09T08:49:26.543 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.012600878 2024-05-09T08:49:26.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.543 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getattr_latency": { 2024-05-09T08:49:26.543 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 1358, 2024-05-09T08:49:26.543 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.593904280, 2024-05-09T08:49:26.543 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000437337 2024-05-09T08:49:26.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.543 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getfilelock_latency": { 2024-05-09T08:49:26.544 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.544 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.544 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.544 INFO:teuthology.orchestra.run.smithi157.stdout: "req_link_latency": { 2024-05-09T08:49:26.544 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 61, 2024-05-09T08:49:26.544 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.143706550, 2024-05-09T08:49:26.544 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.002355845 2024-05-09T08:49:26.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.545 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookup_latency": { 2024-05-09T08:49:26.545 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 155, 2024-05-09T08:49:26.545 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.392933853, 2024-05-09T08:49:26.545 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.002535057 2024-05-09T08:49:26.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.545 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookuphash_latency": { 2024-05-09T08:49:26.545 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.546 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.546 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.546 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupino_latency": { 2024-05-09T08:49:26.546 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.546 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.547 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.547 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupname_latency": { 2024-05-09T08:49:26.547 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.547 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.547 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.548 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupparent_latency": { 2024-05-09T08:49:26.548 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.548 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.548 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.548 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupsnap_latency": { 2024-05-09T08:49:26.549 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.549 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.549 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.551 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:26 smithi107 ceph-mon[30122]: pgmap v507: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-05-09T08:49:26.551 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lssnap_latency": { 2024-05-09T08:49:26.551 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 16, 2024-05-09T08:49:26.551 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.002983845, 2024-05-09T08:49:26.551 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000186490 2024-05-09T08:49:26.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.551 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mkdir_latency": { 2024-05-09T08:49:26.551 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 105, 2024-05-09T08:49:26.552 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.127532717, 2024-05-09T08:49:26.552 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.001214597 2024-05-09T08:49:26.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.552 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mknod_latency": { 2024-05-09T08:49:26.552 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.552 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.552 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.552 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mksnap_latency": { 2024-05-09T08:49:26.553 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 16, 2024-05-09T08:49:26.553 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 1.270382416, 2024-05-09T08:49:26.553 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.079398901 2024-05-09T08:49:26.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.553 INFO:teuthology.orchestra.run.smithi157.stdout: "req_open_latency": { 2024-05-09T08:49:26.553 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 688, 2024-05-09T08:49:26.553 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 119.761340219, 2024-05-09T08:49:26.553 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.174071715 2024-05-09T08:49:26.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.553 INFO:teuthology.orchestra.run.smithi157.stdout: "req_readdir_latency": { 2024-05-09T08:49:26.553 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 201, 2024-05-09T08:49:26.553 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.802921554, 2024-05-09T08:49:26.553 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.003994634 2024-05-09T08:49:26.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.554 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rename_latency": { 2024-05-09T08:49:26.554 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 113, 2024-05-09T08:49:26.554 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 6.245004607, 2024-05-09T08:49:26.554 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.055265527 2024-05-09T08:49:26.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.554 INFO:teuthology.orchestra.run.smithi157.stdout: "req_renamesnap_latency": { 2024-05-09T08:49:26.554 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.554 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.554 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.554 INFO:teuthology.orchestra.run.smithi157.stdout: "req_snapdiff_latency": { 2024-05-09T08:49:26.554 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.555 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.555 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.555 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmdir_latency": { 2024-05-09T08:49:26.555 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 98, 2024-05-09T08:49:26.555 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 3.744826494, 2024-05-09T08:49:26.555 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.038212515 2024-05-09T08:49:26.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.555 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmsnap_latency": { 2024-05-09T08:49:26.555 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 10, 2024-05-09T08:49:26.555 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.092693155, 2024-05-09T08:49:26.555 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.009269315 2024-05-09T08:49:26.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.555 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmxattr_latency": { 2024-05-09T08:49:26.556 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.556 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.556 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.556 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setattr_latency": { 2024-05-09T08:49:26.556 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 5, 2024-05-09T08:49:26.556 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.006795339, 2024-05-09T08:49:26.556 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.001359067 2024-05-09T08:49:26.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.556 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setdirlayout_latency": { 2024-05-09T08:49:26.556 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.556 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.556 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.556 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setfilelock_latency": { 2024-05-09T08:49:26.557 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 4, 2024-05-09T08:49:26.557 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.002030881, 2024-05-09T08:49:26.557 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000507720 2024-05-09T08:49:26.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.557 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setlayout_latency": { 2024-05-09T08:49:26.557 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.557 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.557 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.557 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setxattr_latency": { 2024-05-09T08:49:26.557 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 9, 2024-05-09T08:49:26.557 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.022267429, 2024-05-09T08:49:26.557 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.002474158 2024-05-09T08:49:26.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.558 INFO:teuthology.orchestra.run.smithi157.stdout: "req_symlink_latency": { 2024-05-09T08:49:26.558 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 13, 2024-05-09T08:49:26.558 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.013191942, 2024-05-09T08:49:26.558 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.001014764 2024-05-09T08:49:26.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.558 INFO:teuthology.orchestra.run.smithi157.stdout: "req_unlink_latency": { 2024-05-09T08:49:26.558 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 796, 2024-05-09T08:49:26.558 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 1.700495678, 2024-05-09T08:49:26.558 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.002136301 2024-05-09T08:49:26.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.558 INFO:teuthology.orchestra.run.smithi157.stdout: "cap_revoke_eviction": 0, 2024-05-09T08:49:26.558 INFO:teuthology.orchestra.run.smithi157.stdout: "cap_acquisition_throttle": 0, 2024-05-09T08:49:26.558 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getvxattr_latency": { 2024-05-09T08:49:26.559 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.559 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.559 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.559 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:26.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.559 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_sessions": { 2024-05-09T08:49:26.559 INFO:teuthology.orchestra.run.smithi157.stdout: "session_count": 2, 2024-05-09T08:49:26.559 INFO:teuthology.orchestra.run.smithi157.stdout: "session_add": 7, 2024-05-09T08:49:26.559 INFO:teuthology.orchestra.run.smithi157.stdout: "session_remove": 5, 2024-05-09T08:49:26.559 INFO:teuthology.orchestra.run.smithi157.stdout: "sessions_open": 2, 2024-05-09T08:49:26.559 INFO:teuthology.orchestra.run.smithi157.stdout: "sessions_stale": 0, 2024-05-09T08:49:26.559 INFO:teuthology.orchestra.run.smithi157.stdout: "total_load": 589, 2024-05-09T08:49:26.559 INFO:teuthology.orchestra.run.smithi157.stdout: "average_load": 294, 2024-05-09T08:49:26.560 INFO:teuthology.orchestra.run.smithi157.stdout: "avg_session_uptime": 946, 2024-05-09T08:49:26.560 INFO:teuthology.orchestra.run.smithi157.stdout: "mdthresh_evicted": 0 2024-05-09T08:49:26.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.560 INFO:teuthology.orchestra.run.smithi157.stdout: "mempool": { 2024-05-09T08:49:26.560 INFO:teuthology.orchestra.run.smithi157.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:49:26.560 INFO:teuthology.orchestra.run.smithi157.stdout: "bloom_filter_items": 0, 2024-05-09T08:49:26.560 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:49:26.560 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:49:26.560 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:49:26.561 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:49:26.561 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:49:26.561 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:49:26.561 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:49:26.561 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:49:26.561 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:49:26.561 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:49:26.561 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:49:26.561 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:49:26.561 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:49:26.561 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_extent_items": 0, 2024-05-09T08:49:26.562 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:49:26.562 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_blob_items": 0, 2024-05-09T08:49:26.562 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:49:26.562 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:49:26.562 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:49:26.562 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:49:26.562 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:49:26.563 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:49:26.563 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:49:26.563 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_txc_items": 0, 2024-05-09T08:49:26.563 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:49:26.563 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:49:26.563 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:49:26.563 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_items": 0, 2024-05-09T08:49:26.563 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_bytes": 0, 2024-05-09T08:49:26.563 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_items": 0, 2024-05-09T08:49:26.563 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:49:26.563 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:49:26.563 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:49:26.563 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:49:26.563 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_anon_bytes": 960360, 2024-05-09T08:49:26.564 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_anon_items": 287, 2024-05-09T08:49:26.564 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_meta_bytes": 88, 2024-05-09T08:49:26.564 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_meta_items": 1, 2024-05-09T08:49:26.564 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_bytes": 0, 2024-05-09T08:49:26.564 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_items": 0, 2024-05-09T08:49:26.564 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:49:26.564 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_mapbl_items": 0, 2024-05-09T08:49:26.564 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:49:26.564 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_pglog_items": 0, 2024-05-09T08:49:26.564 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_bytes": 9400, 2024-05-09T08:49:26.564 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_items": 179, 2024-05-09T08:49:26.564 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:49:26.564 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:49:26.565 INFO:teuthology.orchestra.run.smithi157.stdout: "pgmap_bytes": 0, 2024-05-09T08:49:26.565 INFO:teuthology.orchestra.run.smithi157.stdout: "pgmap_items": 0, 2024-05-09T08:49:26.565 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_co_bytes": 3458179, 2024-05-09T08:49:26.565 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_co_items": 36855, 2024-05-09T08:49:26.565 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_1_bytes": 0, 2024-05-09T08:49:26.565 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_1_items": 0, 2024-05-09T08:49:26.565 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_2_bytes": 0, 2024-05-09T08:49:26.565 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_2_items": 0 2024-05-09T08:49:26.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.565 INFO:teuthology.orchestra.run.smithi157.stdout: "objecter": { 2024-05-09T08:49:26.565 INFO:teuthology.orchestra.run.smithi157.stdout: "op_active": 0, 2024-05-09T08:49:26.565 INFO:teuthology.orchestra.run.smithi157.stdout: "op_laggy": 0, 2024-05-09T08:49:26.565 INFO:teuthology.orchestra.run.smithi157.stdout: "op_send": 20019, 2024-05-09T08:49:26.565 INFO:teuthology.orchestra.run.smithi157.stdout: "op_send_bytes": 49614657, 2024-05-09T08:49:26.565 INFO:teuthology.orchestra.run.smithi157.stdout: "op_resend": 32, 2024-05-09T08:49:26.566 INFO:teuthology.orchestra.run.smithi157.stdout: "op_reply": 20019, 2024-05-09T08:49:26.566 INFO:teuthology.orchestra.run.smithi157.stdout: "op_latency": { 2024-05-09T08:49:26.566 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 20019, 2024-05-09T08:49:26.566 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 1142.299731160, 2024-05-09T08:49:26.566 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.057060778 2024-05-09T08:49:26.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.566 INFO:teuthology.orchestra.run.smithi157.stdout: "op_inflight": 0, 2024-05-09T08:49:26.566 INFO:teuthology.orchestra.run.smithi157.stdout: "oplen_avg": { 2024-05-09T08:49:26.566 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 20019, 2024-05-09T08:49:26.566 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 23796 2024-05-09T08:49:26.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.566 INFO:teuthology.orchestra.run.smithi157.stdout: "op": 20019, 2024-05-09T08:49:26.566 INFO:teuthology.orchestra.run.smithi157.stdout: "op_r": 7865, 2024-05-09T08:49:26.567 INFO:teuthology.orchestra.run.smithi157.stdout: "op_w": 12154, 2024-05-09T08:49:26.567 INFO:teuthology.orchestra.run.smithi157.stdout: "op_rmw": 0, 2024-05-09T08:49:26.567 INFO:teuthology.orchestra.run.smithi157.stdout: "op_pg": 0, 2024-05-09T08:49:26.567 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_stat": 724, 2024-05-09T08:49:26.567 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_create": 944, 2024-05-09T08:49:26.567 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_read": 66, 2024-05-09T08:49:26.567 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_write": 2846, 2024-05-09T08:49:26.567 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_writefull": 246, 2024-05-09T08:49:26.567 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_writesame": 0, 2024-05-09T08:49:26.567 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_append": 0, 2024-05-09T08:49:26.567 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_zero": 0, 2024-05-09T08:49:26.567 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_truncate": 1, 2024-05-09T08:49:26.567 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_delete": 6653, 2024-05-09T08:49:26.568 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_mapext": 0, 2024-05-09T08:49:26.568 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_sparse_read": 0, 2024-05-09T08:49:26.568 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_clonerange": 0, 2024-05-09T08:49:26.568 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_getxattr": 7799, 2024-05-09T08:49:26.568 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_setxattr": 1470, 2024-05-09T08:49:26.568 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_cmpxattr": 0, 2024-05-09T08:49:26.568 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_rmxattr": 0, 2024-05-09T08:49:26.568 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_resetxattrs": 0, 2024-05-09T08:49:26.568 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_call": 0, 2024-05-09T08:49:26.568 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_watch": 0, 2024-05-09T08:49:26.568 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_notify": 0, 2024-05-09T08:49:26.568 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T08:49:26.568 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_pgls": 0, 2024-05-09T08:49:26.568 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_pgls_filter": 0, 2024-05-09T08:49:26.568 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_other": 6, 2024-05-09T08:49:26.569 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_active": 0, 2024-05-09T08:49:26.569 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_send": 0, 2024-05-09T08:49:26.569 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_resend": 0, 2024-05-09T08:49:26.569 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_ping": 0, 2024-05-09T08:49:26.569 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_active": 0, 2024-05-09T08:49:26.569 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_send": 0, 2024-05-09T08:49:26.569 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_resend": 0, 2024-05-09T08:49:26.569 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_active": 0, 2024-05-09T08:49:26.569 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_send": 0, 2024-05-09T08:49:26.569 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_resend": 0, 2024-05-09T08:49:26.569 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_active": 0, 2024-05-09T08:49:26.569 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_send": 0, 2024-05-09T08:49:26.569 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_resend": 0, 2024-05-09T08:49:26.569 INFO:teuthology.orchestra.run.smithi157.stdout: "command_active": 0, 2024-05-09T08:49:26.570 INFO:teuthology.orchestra.run.smithi157.stdout: "command_send": 0, 2024-05-09T08:49:26.570 INFO:teuthology.orchestra.run.smithi157.stdout: "command_resend": 0, 2024-05-09T08:49:26.570 INFO:teuthology.orchestra.run.smithi157.stdout: "map_epoch": 116, 2024-05-09T08:49:26.570 INFO:teuthology.orchestra.run.smithi157.stdout: "map_full": 0, 2024-05-09T08:49:26.570 INFO:teuthology.orchestra.run.smithi157.stdout: "map_inc": 32, 2024-05-09T08:49:26.570 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_sessions": 12, 2024-05-09T08:49:26.570 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_session_open": 12, 2024-05-09T08:49:26.570 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_session_close": 0, 2024-05-09T08:49:26.570 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_laggy": 0, 2024-05-09T08:49:26.570 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_wr": 2610, 2024-05-09T08:49:26.570 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_rd": 94, 2024-05-09T08:49:26.570 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_del": 337 2024-05-09T08:49:26.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.571 INFO:teuthology.orchestra.run.smithi157.stdout: "oft": { 2024-05-09T08:49:26.571 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_objs": 1, 2024-05-09T08:49:26.571 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_kv_pairs": 0, 2024-05-09T08:49:26.571 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_updates": 0, 2024-05-09T08:49:26.571 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_removes": 0 2024-05-09T08:49:26.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.571 INFO:teuthology.orchestra.run.smithi157.stdout: "osd-slow-ops": { 2024-05-09T08:49:26.571 INFO:teuthology.orchestra.run.smithi157.stdout: "slow_ops_count": 10 2024-05-09T08:49:26.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.571 INFO:teuthology.orchestra.run.smithi157.stdout: "purge_queue": { 2024-05-09T08:49:26.571 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_ops": 0, 2024-05-09T08:49:26.571 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_ops_high_water": 58, 2024-05-09T08:49:26.571 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing": 0, 2024-05-09T08:49:26.571 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_high_water": 11, 2024-05-09T08:49:26.571 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executed_ops": 6119, 2024-05-09T08:49:26.572 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executed": 784, 2024-05-09T08:49:26.572 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_item_in_journal": 0 2024-05-09T08:49:26.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.572 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:49:26.572 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2024-05-09T08:49:26.572 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 104857600, 2024-05-09T08:49:26.572 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2024-05-09T08:49:26.572 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 39331, 2024-05-09T08:49:26.572 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 12490414, 2024-05-09T08:49:26.573 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:26.573 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 39331, 2024-05-09T08:49:26.573 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2024-05-09T08:49:26.573 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2024-05-09T08:49:26.573 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 39331, 2024-05-09T08:49:26.573 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 12490414, 2024-05-09T08:49:26.573 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2024-05-09T08:49:26.573 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.573 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.573 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.574 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:26.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.574 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-objecter_bytes": { 2024-05-09T08:49:26.574 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2024-05-09T08:49:26.574 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 104857600, 2024-05-09T08:49:26.574 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2024-05-09T08:49:26.574 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2024-05-09T08:49:26.574 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2024-05-09T08:49:26.574 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:26.574 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:26.574 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 20019, 2024-05-09T08:49:26.574 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 49714513, 2024-05-09T08:49:26.574 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 20019, 2024-05-09T08:49:26.575 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 49714513, 2024-05-09T08:49:26.575 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2024-05-09T08:49:26.575 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.575 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.575 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.575 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:26.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.575 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-objecter_ops": { 2024-05-09T08:49:26.575 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2024-05-09T08:49:26.575 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 1024, 2024-05-09T08:49:26.575 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2024-05-09T08:49:26.575 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2024-05-09T08:49:26.575 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2024-05-09T08:49:26.575 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:26.576 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:26.576 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 20019, 2024-05-09T08:49:26.576 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 20019, 2024-05-09T08:49:26.576 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 20019, 2024-05-09T08:49:26.576 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 20019, 2024-05-09T08:49:26.576 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2024-05-09T08:49:26.576 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.576 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.576 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.576 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:26.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.576 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-write_buf_throttle": { 2024-05-09T08:49:26.576 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2024-05-09T08:49:26.576 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 3758096384, 2024-05-09T08:49:26.577 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2024-05-09T08:49:26.577 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 784, 2024-05-09T08:49:26.577 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 100344, 2024-05-09T08:49:26.577 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:26.577 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 784, 2024-05-09T08:49:26.577 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2024-05-09T08:49:26.577 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2024-05-09T08:49:26.577 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 41, 2024-05-09T08:49:26.577 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 100344, 2024-05-09T08:49:26.577 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2024-05-09T08:49:26.577 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.577 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.577 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.577 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:26.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:26.578 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-write_buf_throttle-0x5601dbaa6500": { 2024-05-09T08:49:26.578 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2024-05-09T08:49:26.578 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 3758096384, 2024-05-09T08:49:26.578 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2024-05-09T08:49:26.578 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 8247, 2024-05-09T08:49:26.578 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 45829797, 2024-05-09T08:49:26.578 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:26.578 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 8247, 2024-05-09T08:49:26.578 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2024-05-09T08:49:26.578 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2024-05-09T08:49:26.578 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 2805, 2024-05-09T08:49:26.578 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 45829797, 2024-05-09T08:49:26.579 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2024-05-09T08:49:26.579 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:26.579 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:26.579 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:26.579 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:26.579 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:26.579 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-05-09T08:49:26.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:26 smithi110 ceph-mon[33622]: pgmap v507: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-05-09T08:49:27.288 INFO:tasks.check_counter:Daemon mds.i mds.exported=6 2024-05-09T08:49:27.288 INFO:tasks.check_counter:Daemon mds.i mds.imported=6 2024-05-09T08:49:27.288 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=202 2024-05-09T08:49:27.288 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=3 2024-05-09T08:49:27.288 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-05-09T08:49:27.288 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=16 2024-05-09T08:49:27.288 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=10 2024-05-09T08:49:27.289 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=3 2024-05-09T08:49:27.289 DEBUG:tasks.check_counter:Getting stats from l 2024-05-09T08:49:27.289 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 -- ceph daemon mds.l perf dump 2024-05-09T08:49:27.535 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T08:49:28.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:28 smithi107 ceph-mon[30122]: pgmap v508: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-09T08:49:28.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:28 smithi157 ceph-mon[33614]: pgmap v508: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-09T08:49:28.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:28 smithi110 ceph-mon[33622]: pgmap v508: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-09T08:49:29.228 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-05-09T08:49:29.228 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T08:49:29.228 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 187, 2024-05-09T08:49:29.228 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 187, 2024-05-09T08:49:29.228 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 2694904, 2024-05-09T08:49:29.229 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 58652, 2024-05-09T08:49:29.229 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 6, 2024-05-09T08:49:29.229 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 2, 2024-05-09T08:49:29.229 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.022586272, 2024-05-09T08:49:29.229 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.006171232, 2024-05-09T08:49:29.229 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.007422464, 2024-05-09T08:49:29.229 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.006474690, 2024-05-09T08:49:29.229 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:29.229 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 186, 2024-05-09T08:49:29.229 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.010105211, 2024-05-09T08:49:29.229 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000054329 2024-05-09T08:49:29.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.229 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:29.229 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.230 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.230 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.230 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 3040, 2024-05-09T08:49:29.230 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 448 2024-05-09T08:49:29.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.230 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T08:49:29.230 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 568, 2024-05-09T08:49:29.230 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 555, 2024-05-09T08:49:29.230 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 5584347, 2024-05-09T08:49:29.230 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 266049, 2024-05-09T08:49:29.230 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 4, 2024-05-09T08:49:29.230 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 3, 2024-05-09T08:49:29.231 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.078054589, 2024-05-09T08:49:29.231 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.022228391, 2024-05-09T08:49:29.231 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.051134316, 2024-05-09T08:49:29.231 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.013997401, 2024-05-09T08:49:29.231 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:29.231 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 554, 2024-05-09T08:49:29.231 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.024338474, 2024-05-09T08:49:29.231 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000043932 2024-05-09T08:49:29.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.231 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:29.231 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.231 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.231 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.232 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 1636976, 2024-05-09T08:49:29.232 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 190944 2024-05-09T08:49:29.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.232 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T08:49:29.232 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 6, 2024-05-09T08:49:29.232 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 238, 2024-05-09T08:49:29.232 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 2373, 2024-05-09T08:49:29.232 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 334804, 2024-05-09T08:49:29.232 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 8, 2024-05-09T08:49:29.232 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 5, 2024-05-09T08:49:29.232 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.015117652, 2024-05-09T08:49:29.232 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.009961510, 2024-05-09T08:49:29.232 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.002265200, 2024-05-09T08:49:29.232 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.000119334, 2024-05-09T08:49:29.233 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T08:49:29.233 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 235, 2024-05-09T08:49:29.233 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.049956175, 2024-05-09T08:49:29.233 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000212579 2024-05-09T08:49:29.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.233 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2024-05-09T08:49:29.233 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.233 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.233 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.233 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 480, 2024-05-09T08:49:29.233 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 333840 2024-05-09T08:49:29.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.233 INFO:teuthology.orchestra.run.smithi157.stdout: "cct": { 2024-05-09T08:49:29.234 INFO:teuthology.orchestra.run.smithi157.stdout: "total_workers": 1, 2024-05-09T08:49:29.234 INFO:teuthology.orchestra.run.smithi157.stdout: "unhealthy_workers": 0 2024-05-09T08:49:29.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.234 INFO:teuthology.orchestra.run.smithi157.stdout: "finisher-MDSRank": { 2024-05-09T08:49:29.234 INFO:teuthology.orchestra.run.smithi157.stdout: "queue_len": 0, 2024-05-09T08:49:29.234 INFO:teuthology.orchestra.run.smithi157.stdout: "complete_latency": { 2024-05-09T08:49:29.234 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 693, 2024-05-09T08:49:29.234 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.022220884, 2024-05-09T08:49:29.234 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000032064 2024-05-09T08:49:29.234 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:29.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.234 INFO:teuthology.orchestra.run.smithi157.stdout: "finisher-PurgeQueue": { 2024-05-09T08:49:29.234 INFO:teuthology.orchestra.run.smithi157.stdout: "queue_len": 0, 2024-05-09T08:49:29.235 INFO:teuthology.orchestra.run.smithi157.stdout: "complete_latency": { 2024-05-09T08:49:29.235 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.235 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.235 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.235 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:29.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.235 INFO:teuthology.orchestra.run.smithi157.stdout: "mds": { 2024-05-09T08:49:29.235 INFO:teuthology.orchestra.run.smithi157.stdout: "request": 0, 2024-05-09T08:49:29.235 INFO:teuthology.orchestra.run.smithi157.stdout: "reply": 0, 2024-05-09T08:49:29.235 INFO:teuthology.orchestra.run.smithi157.stdout: "reply_latency": { 2024-05-09T08:49:29.235 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.235 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.235 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.236 INFO:teuthology.orchestra.run.smithi157.stdout: "slow_reply": 0, 2024-05-09T08:49:29.236 INFO:teuthology.orchestra.run.smithi157.stdout: "forward": 0, 2024-05-09T08:49:29.236 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_fetch_complete": 0, 2024-05-09T08:49:29.236 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_fetch_keys": 0, 2024-05-09T08:49:29.236 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_commit": 0, 2024-05-09T08:49:29.236 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_split": 0, 2024-05-09T08:49:29.236 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_merge": 0, 2024-05-09T08:49:29.236 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes": 1293, 2024-05-09T08:49:29.236 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_top": 57, 2024-05-09T08:49:29.236 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_bottom": 1236, 2024-05-09T08:49:29.236 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_pin_tail": 0, 2024-05-09T08:49:29.236 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_pinned": 1212, 2024-05-09T08:49:29.236 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_expired": 0, 2024-05-09T08:49:29.237 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_with_caps": 0, 2024-05-09T08:49:29.237 INFO:teuthology.orchestra.run.smithi157.stdout: "caps": 0, 2024-05-09T08:49:29.237 INFO:teuthology.orchestra.run.smithi157.stdout: "subtrees": 2, 2024-05-09T08:49:29.237 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse": 0, 2024-05-09T08:49:29.237 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_hit": 0, 2024-05-09T08:49:29.237 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_forward": 0, 2024-05-09T08:49:29.237 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_discover": 0, 2024-05-09T08:49:29.237 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_dir_fetch": 0, 2024-05-09T08:49:29.237 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_remote_ino": 0, 2024-05-09T08:49:29.237 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_lock": 0, 2024-05-09T08:49:29.237 INFO:teuthology.orchestra.run.smithi157.stdout: "load_cent": 0, 2024-05-09T08:49:29.237 INFO:teuthology.orchestra.run.smithi157.stdout: "q": 0, 2024-05-09T08:49:29.237 INFO:teuthology.orchestra.run.smithi157.stdout: "exported": 0, 2024-05-09T08:49:29.237 INFO:teuthology.orchestra.run.smithi157.stdout: "exported_inodes": 0, 2024-05-09T08:49:29.237 INFO:teuthology.orchestra.run.smithi157.stdout: "imported": 0, 2024-05-09T08:49:29.238 INFO:teuthology.orchestra.run.smithi157.stdout: "imported_inodes": 0, 2024-05-09T08:49:29.238 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_dir_fetch": 0, 2024-05-09T08:49:29.238 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_backtrace_fetch": 0, 2024-05-09T08:49:29.238 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_peer_discover": 0, 2024-05-09T08:49:29.238 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rfiles": 4, 2024-05-09T08:49:29.238 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rbytes": 234, 2024-05-09T08:49:29.238 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rsnaps": 6, 2024-05-09T08:49:29.238 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T08:49:29.238 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_set_tag": 0, 2024-05-09T08:49:29.238 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T08:49:29.238 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_inotable_repaired": 0, 2024-05-09T08:49:29.238 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dir_inodes": 0, 2024-05-09T08:49:29.238 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T08:49:29.238 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T08:49:29.239 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_file_inodes": 0, 2024-05-09T08:49:29.239 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_inode_file_caps": 0, 2024-05-09T08:49:29.239 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T08:49:29.239 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_grant": 0, 2024-05-09T08:49:29.239 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T08:49:29.239 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T08:49:29.239 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T08:49:29.239 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_caps": 0, 2024-05-09T08:49:29.239 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_caps_dirty": 0, 2024-05-09T08:49:29.239 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_cap_release": 0, 2024-05-09T08:49:29.239 INFO:teuthology.orchestra.run.smithi157.stdout: "process_request_cap_release": 0 2024-05-09T08:49:29.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.239 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_cache": { 2024-05-09T08:49:29.239 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_update": 0, 2024-05-09T08:49:29.240 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_update_receipt": 0, 2024-05-09T08:49:29.240 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_try_discover": 0, 2024-05-09T08:49:29.240 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_send_discover": 0, 2024-05-09T08:49:29.240 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_handle_discover": 0, 2024-05-09T08:49:29.240 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays": 0, 2024-05-09T08:49:29.240 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays_delayed": 0, 2024-05-09T08:49:29.240 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays_enqueuing": 0, 2024-05-09T08:49:29.240 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_created": 0, 2024-05-09T08:49:29.240 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_enqueued": 0, 2024-05-09T08:49:29.240 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_reintegrated": 0, 2024-05-09T08:49:29.240 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_migrated": 0, 2024-05-09T08:49:29.240 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_processing": 0, 2024-05-09T08:49:29.240 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_enqueued": 0, 2024-05-09T08:49:29.241 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_prioritized": 0, 2024-05-09T08:49:29.241 INFO:teuthology.orchestra.run.smithi157.stdout: "recovery_started": 0, 2024-05-09T08:49:29.241 INFO:teuthology.orchestra.run.smithi157.stdout: "recovery_completed": 0, 2024-05-09T08:49:29.241 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_quiesce_path": 0, 2024-05-09T08:49:29.241 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_quiesce_inode": 0, 2024-05-09T08:49:29.241 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T08:49:29.241 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_exportdir": 0, 2024-05-09T08:49:29.241 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_flush": 0, 2024-05-09T08:49:29.241 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_fragmentdir": 0, 2024-05-09T08:49:29.241 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_fragstats": 0, 2024-05-09T08:49:29.241 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_inodestats": 0 2024-05-09T08:49:29.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.241 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_log": { 2024-05-09T08:49:29.241 INFO:teuthology.orchestra.run.smithi157.stdout: "evlrg": 0, 2024-05-09T08:49:29.241 INFO:teuthology.orchestra.run.smithi157.stdout: "evadd": 0, 2024-05-09T08:49:29.242 INFO:teuthology.orchestra.run.smithi157.stdout: "evex": 0, 2024-05-09T08:49:29.242 INFO:teuthology.orchestra.run.smithi157.stdout: "evtrm": 0, 2024-05-09T08:49:29.242 INFO:teuthology.orchestra.run.smithi157.stdout: "ev": 1323, 2024-05-09T08:49:29.242 INFO:teuthology.orchestra.run.smithi157.stdout: "evexg": 0, 2024-05-09T08:49:29.242 INFO:teuthology.orchestra.run.smithi157.stdout: "evexd": 0, 2024-05-09T08:49:29.242 INFO:teuthology.orchestra.run.smithi157.stdout: "segadd": 0, 2024-05-09T08:49:29.242 INFO:teuthology.orchestra.run.smithi157.stdout: "segex": 0, 2024-05-09T08:49:29.242 INFO:teuthology.orchestra.run.smithi157.stdout: "segtrm": 0, 2024-05-09T08:49:29.242 INFO:teuthology.orchestra.run.smithi157.stdout: "seg": 2, 2024-05-09T08:49:29.242 INFO:teuthology.orchestra.run.smithi157.stdout: "segmjr": 4, 2024-05-09T08:49:29.242 INFO:teuthology.orchestra.run.smithi157.stdout: "segexg": 0, 2024-05-09T08:49:29.242 INFO:teuthology.orchestra.run.smithi157.stdout: "segexd": 0, 2024-05-09T08:49:29.242 INFO:teuthology.orchestra.run.smithi157.stdout: "expos": 46230233, 2024-05-09T08:49:29.242 INFO:teuthology.orchestra.run.smithi157.stdout: "wrpos": 50024101, 2024-05-09T08:49:29.243 INFO:teuthology.orchestra.run.smithi157.stdout: "rdpos": 50023965, 2024-05-09T08:49:29.243 INFO:teuthology.orchestra.run.smithi157.stdout: "jlat": { 2024-05-09T08:49:29.243 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.243 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.243 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.243 INFO:teuthology.orchestra.run.smithi157.stdout: "replayed": 1323 2024-05-09T08:49:29.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.243 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_mem": { 2024-05-09T08:49:29.243 INFO:teuthology.orchestra.run.smithi157.stdout: "ino": 491, 2024-05-09T08:49:29.243 INFO:teuthology.orchestra.run.smithi157.stdout: "ino+": 806, 2024-05-09T08:49:29.243 INFO:teuthology.orchestra.run.smithi157.stdout: "ino-": 315, 2024-05-09T08:49:29.243 INFO:teuthology.orchestra.run.smithi157.stdout: "dir": 105, 2024-05-09T08:49:29.243 INFO:teuthology.orchestra.run.smithi157.stdout: "dir+": 106, 2024-05-09T08:49:29.244 INFO:teuthology.orchestra.run.smithi157.stdout: "dir-": 1, 2024-05-09T08:49:29.244 INFO:teuthology.orchestra.run.smithi157.stdout: "dn": 1293, 2024-05-09T08:49:29.244 INFO:teuthology.orchestra.run.smithi157.stdout: "dn+": 1293, 2024-05-09T08:49:29.244 INFO:teuthology.orchestra.run.smithi157.stdout: "dn-": 0, 2024-05-09T08:49:29.244 INFO:teuthology.orchestra.run.smithi157.stdout: "cap": 0, 2024-05-09T08:49:29.244 INFO:teuthology.orchestra.run.smithi157.stdout: "cap+": 0, 2024-05-09T08:49:29.244 INFO:teuthology.orchestra.run.smithi157.stdout: "cap-": 0, 2024-05-09T08:49:29.244 INFO:teuthology.orchestra.run.smithi157.stdout: "rss": 55764, 2024-05-09T08:49:29.244 INFO:teuthology.orchestra.run.smithi157.stdout: "heap": 215420 2024-05-09T08:49:29.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.244 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_server": { 2024-05-09T08:49:29.244 INFO:teuthology.orchestra.run.smithi157.stdout: "dispatch_client_request": 0, 2024-05-09T08:49:29.244 INFO:teuthology.orchestra.run.smithi157.stdout: "dispatch_server_request": 0, 2024-05-09T08:49:29.244 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_request": 0, 2024-05-09T08:49:29.245 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_session": 0, 2024-05-09T08:49:29.245 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_peer_request": 0, 2024-05-09T08:49:29.245 INFO:teuthology.orchestra.run.smithi157.stdout: "req_create_latency": { 2024-05-09T08:49:29.245 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.245 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.245 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.245 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getattr_latency": { 2024-05-09T08:49:29.245 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.245 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.245 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.245 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getfilelock_latency": { 2024-05-09T08:49:29.245 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.245 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.246 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.246 INFO:teuthology.orchestra.run.smithi157.stdout: "req_link_latency": { 2024-05-09T08:49:29.246 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.246 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.246 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.246 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookup_latency": { 2024-05-09T08:49:29.246 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.246 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.246 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.246 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookuphash_latency": { 2024-05-09T08:49:29.247 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.247 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.247 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.247 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupino_latency": { 2024-05-09T08:49:29.247 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.247 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.247 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.247 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupname_latency": { 2024-05-09T08:49:29.247 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.247 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.247 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.248 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupparent_latency": { 2024-05-09T08:49:29.248 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.248 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.248 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.248 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupsnap_latency": { 2024-05-09T08:49:29.248 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.248 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.248 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.248 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lssnap_latency": { 2024-05-09T08:49:29.248 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.249 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.249 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.249 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mkdir_latency": { 2024-05-09T08:49:29.249 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.250 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.250 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.250 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mknod_latency": { 2024-05-09T08:49:29.250 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.250 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.250 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.250 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mksnap_latency": { 2024-05-09T08:49:29.250 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.250 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.250 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.251 INFO:teuthology.orchestra.run.smithi157.stdout: "req_open_latency": { 2024-05-09T08:49:29.251 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.251 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.251 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.251 INFO:teuthology.orchestra.run.smithi157.stdout: "req_readdir_latency": { 2024-05-09T08:49:29.251 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.251 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.251 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.251 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rename_latency": { 2024-05-09T08:49:29.251 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.251 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.251 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.252 INFO:teuthology.orchestra.run.smithi157.stdout: "req_renamesnap_latency": { 2024-05-09T08:49:29.252 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.252 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.252 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.252 INFO:teuthology.orchestra.run.smithi157.stdout: "req_snapdiff_latency": { 2024-05-09T08:49:29.252 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.252 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.252 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.252 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmdir_latency": { 2024-05-09T08:49:29.252 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.252 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.252 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.253 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmsnap_latency": { 2024-05-09T08:49:29.253 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.253 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.253 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.253 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmxattr_latency": { 2024-05-09T08:49:29.253 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.253 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.253 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.253 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setattr_latency": { 2024-05-09T08:49:29.253 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.253 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.253 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.254 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setdirlayout_latency": { 2024-05-09T08:49:29.254 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.254 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.254 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.254 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setfilelock_latency": { 2024-05-09T08:49:29.254 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.254 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.254 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.254 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setlayout_latency": { 2024-05-09T08:49:29.254 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.255 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.255 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.255 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setxattr_latency": { 2024-05-09T08:49:29.255 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.255 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.255 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.255 INFO:teuthology.orchestra.run.smithi157.stdout: "req_symlink_latency": { 2024-05-09T08:49:29.255 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.255 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.255 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.255 INFO:teuthology.orchestra.run.smithi157.stdout: "req_unlink_latency": { 2024-05-09T08:49:29.256 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.256 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.256 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.256 INFO:teuthology.orchestra.run.smithi157.stdout: "cap_revoke_eviction": 0, 2024-05-09T08:49:29.256 INFO:teuthology.orchestra.run.smithi157.stdout: "cap_acquisition_throttle": 0, 2024-05-09T08:49:29.256 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getvxattr_latency": { 2024-05-09T08:49:29.256 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.256 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.256 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.256 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:29.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.256 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_sessions": { 2024-05-09T08:49:29.256 INFO:teuthology.orchestra.run.smithi157.stdout: "session_count": 2, 2024-05-09T08:49:29.256 INFO:teuthology.orchestra.run.smithi157.stdout: "session_add": 2, 2024-05-09T08:49:29.257 INFO:teuthology.orchestra.run.smithi157.stdout: "session_remove": 0, 2024-05-09T08:49:29.257 INFO:teuthology.orchestra.run.smithi157.stdout: "sessions_open": 0, 2024-05-09T08:49:29.257 INFO:teuthology.orchestra.run.smithi157.stdout: "sessions_stale": 0, 2024-05-09T08:49:29.257 INFO:teuthology.orchestra.run.smithi157.stdout: "total_load": 0, 2024-05-09T08:49:29.257 INFO:teuthology.orchestra.run.smithi157.stdout: "average_load": 0, 2024-05-09T08:49:29.257 INFO:teuthology.orchestra.run.smithi157.stdout: "avg_session_uptime": 2067, 2024-05-09T08:49:29.257 INFO:teuthology.orchestra.run.smithi157.stdout: "mdthresh_evicted": 0 2024-05-09T08:49:29.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.257 INFO:teuthology.orchestra.run.smithi157.stdout: "mempool": { 2024-05-09T08:49:29.257 INFO:teuthology.orchestra.run.smithi157.stdout: "bloom_filter_bytes": 0, 2024-05-09T08:49:29.257 INFO:teuthology.orchestra.run.smithi157.stdout: "bloom_filter_items": 0, 2024-05-09T08:49:29.257 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T08:49:29.257 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_alloc_items": 0, 2024-05-09T08:49:29.257 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T08:49:29.257 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_data_items": 0, 2024-05-09T08:49:29.258 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T08:49:29.258 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T08:49:29.258 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T08:49:29.258 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T08:49:29.258 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T08:49:29.258 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_other_items": 0, 2024-05-09T08:49:29.258 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T08:49:29.258 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T08:49:29.258 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_extent_bytes": 0, 2024-05-09T08:49:29.258 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_extent_items": 0, 2024-05-09T08:49:29.258 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_blob_bytes": 0, 2024-05-09T08:49:29.258 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_blob_items": 0, 2024-05-09T08:49:29.258 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T08:49:29.258 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T08:49:29.259 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T08:49:29.259 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T08:49:29.259 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T08:49:29.259 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_fsck_items": 0, 2024-05-09T08:49:29.259 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_txc_bytes": 0, 2024-05-09T08:49:29.259 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_txc_items": 0, 2024-05-09T08:49:29.259 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T08:49:29.259 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T08:49:29.259 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_bytes": 0, 2024-05-09T08:49:29.259 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_items": 0, 2024-05-09T08:49:29.259 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_bytes": 0, 2024-05-09T08:49:29.259 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_items": 0, 2024-05-09T08:49:29.259 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T08:49:29.259 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_reader_items": 0, 2024-05-09T08:49:29.260 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T08:49:29.260 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_writer_items": 0, 2024-05-09T08:49:29.260 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_anon_bytes": 197188, 2024-05-09T08:49:29.260 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_anon_items": 55, 2024-05-09T08:49:29.260 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_meta_bytes": 0, 2024-05-09T08:49:29.260 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_meta_items": 0, 2024-05-09T08:49:29.260 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_bytes": 0, 2024-05-09T08:49:29.260 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_items": 0, 2024-05-09T08:49:29.260 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_mapbl_bytes": 0, 2024-05-09T08:49:29.260 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_mapbl_items": 0, 2024-05-09T08:49:29.260 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_pglog_bytes": 0, 2024-05-09T08:49:29.260 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_pglog_items": 0, 2024-05-09T08:49:29.260 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_bytes": 9400, 2024-05-09T08:49:29.260 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_items": 179, 2024-05-09T08:49:29.260 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T08:49:29.261 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_mapping_items": 0, 2024-05-09T08:49:29.261 INFO:teuthology.orchestra.run.smithi157.stdout: "pgmap_bytes": 0, 2024-05-09T08:49:29.261 INFO:teuthology.orchestra.run.smithi157.stdout: "pgmap_items": 0, 2024-05-09T08:49:29.261 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_co_bytes": 2211126, 2024-05-09T08:49:29.261 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_co_items": 26502, 2024-05-09T08:49:29.261 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_1_bytes": 0, 2024-05-09T08:49:29.261 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_1_items": 0, 2024-05-09T08:49:29.261 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_2_bytes": 0, 2024-05-09T08:49:29.261 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_2_items": 0 2024-05-09T08:49:29.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.261 INFO:teuthology.orchestra.run.smithi157.stdout: "objecter": { 2024-05-09T08:49:29.261 INFO:teuthology.orchestra.run.smithi157.stdout: "op_active": 0, 2024-05-09T08:49:29.261 INFO:teuthology.orchestra.run.smithi157.stdout: "op_laggy": 0, 2024-05-09T08:49:29.261 INFO:teuthology.orchestra.run.smithi157.stdout: "op_send": 422, 2024-05-09T08:49:29.262 INFO:teuthology.orchestra.run.smithi157.stdout: "op_send_bytes": 26, 2024-05-09T08:49:29.262 INFO:teuthology.orchestra.run.smithi157.stdout: "op_resend": 4, 2024-05-09T08:49:29.262 INFO:teuthology.orchestra.run.smithi157.stdout: "op_reply": 422, 2024-05-09T08:49:29.262 INFO:teuthology.orchestra.run.smithi157.stdout: "op_latency": { 2024-05-09T08:49:29.262 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 422, 2024-05-09T08:49:29.262 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 3.280939129, 2024-05-09T08:49:29.262 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.007774737 2024-05-09T08:49:29.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.262 INFO:teuthology.orchestra.run.smithi157.stdout: "op_inflight": 0, 2024-05-09T08:49:29.262 INFO:teuthology.orchestra.run.smithi157.stdout: "oplen_avg": { 2024-05-09T08:49:29.262 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 422, 2024-05-09T08:49:29.262 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 423 2024-05-09T08:49:29.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.262 INFO:teuthology.orchestra.run.smithi157.stdout: "op": 422, 2024-05-09T08:49:29.263 INFO:teuthology.orchestra.run.smithi157.stdout: "op_r": 422, 2024-05-09T08:49:29.263 INFO:teuthology.orchestra.run.smithi157.stdout: "op_w": 0, 2024-05-09T08:49:29.263 INFO:teuthology.orchestra.run.smithi157.stdout: "op_rmw": 0, 2024-05-09T08:49:29.263 INFO:teuthology.orchestra.run.smithi157.stdout: "op_pg": 0, 2024-05-09T08:49:29.263 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_stat": 252, 2024-05-09T08:49:29.263 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_create": 0, 2024-05-09T08:49:29.263 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_read": 167, 2024-05-09T08:49:29.263 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_write": 0, 2024-05-09T08:49:29.263 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_writefull": 0, 2024-05-09T08:49:29.263 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_writesame": 0, 2024-05-09T08:49:29.263 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_append": 0, 2024-05-09T08:49:29.263 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_zero": 0, 2024-05-09T08:49:29.263 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_truncate": 0, 2024-05-09T08:49:29.263 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_delete": 0, 2024-05-09T08:49:29.263 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_mapext": 0, 2024-05-09T08:49:29.264 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_sparse_read": 0, 2024-05-09T08:49:29.264 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_clonerange": 0, 2024-05-09T08:49:29.264 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_getxattr": 2, 2024-05-09T08:49:29.264 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_setxattr": 0, 2024-05-09T08:49:29.264 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_cmpxattr": 0, 2024-05-09T08:49:29.264 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_rmxattr": 0, 2024-05-09T08:49:29.264 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_resetxattrs": 0, 2024-05-09T08:49:29.264 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_call": 0, 2024-05-09T08:49:29.264 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_watch": 0, 2024-05-09T08:49:29.264 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_notify": 0, 2024-05-09T08:49:29.264 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T08:49:29.264 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_pgls": 0, 2024-05-09T08:49:29.264 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_pgls_filter": 0, 2024-05-09T08:49:29.264 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_other": 0, 2024-05-09T08:49:29.265 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_active": 0, 2024-05-09T08:49:29.265 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_send": 0, 2024-05-09T08:49:29.265 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_resend": 0, 2024-05-09T08:49:29.265 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_ping": 0, 2024-05-09T08:49:29.265 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_active": 0, 2024-05-09T08:49:29.265 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_send": 0, 2024-05-09T08:49:29.265 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_resend": 0, 2024-05-09T08:49:29.265 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_active": 0, 2024-05-09T08:49:29.265 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_send": 0, 2024-05-09T08:49:29.265 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_resend": 0, 2024-05-09T08:49:29.265 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_active": 0, 2024-05-09T08:49:29.265 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_send": 0, 2024-05-09T08:49:29.265 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_resend": 0, 2024-05-09T08:49:29.265 INFO:teuthology.orchestra.run.smithi157.stdout: "command_active": 0, 2024-05-09T08:49:29.265 INFO:teuthology.orchestra.run.smithi157.stdout: "command_send": 0, 2024-05-09T08:49:29.266 INFO:teuthology.orchestra.run.smithi157.stdout: "command_resend": 0, 2024-05-09T08:49:29.266 INFO:teuthology.orchestra.run.smithi157.stdout: "map_epoch": 116, 2024-05-09T08:49:29.266 INFO:teuthology.orchestra.run.smithi157.stdout: "map_full": 0, 2024-05-09T08:49:29.266 INFO:teuthology.orchestra.run.smithi157.stdout: "map_inc": 7, 2024-05-09T08:49:29.266 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_sessions": 6, 2024-05-09T08:49:29.266 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_session_open": 6, 2024-05-09T08:49:29.266 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_session_close": 0, 2024-05-09T08:49:29.266 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_laggy": 0, 2024-05-09T08:49:29.266 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_wr": 0, 2024-05-09T08:49:29.266 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_rd": 2, 2024-05-09T08:49:29.266 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_del": 0 2024-05-09T08:49:29.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.266 INFO:teuthology.orchestra.run.smithi157.stdout: "oft": { 2024-05-09T08:49:29.267 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_objs": 0, 2024-05-09T08:49:29.267 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_kv_pairs": 0, 2024-05-09T08:49:29.267 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_updates": 0, 2024-05-09T08:49:29.267 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_removes": 0 2024-05-09T08:49:29.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.267 INFO:teuthology.orchestra.run.smithi157.stdout: "osd-slow-ops": { 2024-05-09T08:49:29.267 INFO:teuthology.orchestra.run.smithi157.stdout: "slow_ops_count": 0 2024-05-09T08:49:29.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.267 INFO:teuthology.orchestra.run.smithi157.stdout: "purge_queue": { 2024-05-09T08:49:29.267 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_ops": 0, 2024-05-09T08:49:29.267 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T08:49:29.267 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing": 0, 2024-05-09T08:49:29.267 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_high_water": 0, 2024-05-09T08:49:29.267 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executed_ops": 0, 2024-05-09T08:49:29.267 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executed": 0, 2024-05-09T08:49:29.267 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_item_in_journal": 0 2024-05-09T08:49:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T08:49:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2024-05-09T08:49:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 104857600, 2024-05-09T08:49:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2024-05-09T08:49:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 757, 2024-05-09T08:49:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 8193374, 2024-05-09T08:49:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 757, 2024-05-09T08:49:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2024-05-09T08:49:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2024-05-09T08:49:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 757, 2024-05-09T08:49:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 8193374, 2024-05-09T08:49:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2024-05-09T08:49:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.268 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.269 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.269 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:29.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.269 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-objecter_bytes": { 2024-05-09T08:49:29.269 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2024-05-09T08:49:29.269 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 104857600, 2024-05-09T08:49:29.269 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2024-05-09T08:49:29.269 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2024-05-09T08:49:29.269 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2024-05-09T08:49:29.269 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:29.269 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:29.269 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 422, 2024-05-09T08:49:29.269 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 6517692, 2024-05-09T08:49:29.269 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 422, 2024-05-09T08:49:29.269 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 6517692, 2024-05-09T08:49:29.270 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2024-05-09T08:49:29.270 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.270 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.270 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.270 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:29.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.270 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-objecter_ops": { 2024-05-09T08:49:29.270 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2024-05-09T08:49:29.270 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 1024, 2024-05-09T08:49:29.270 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2024-05-09T08:49:29.270 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2024-05-09T08:49:29.270 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2024-05-09T08:49:29.270 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:29.270 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:29.271 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 422, 2024-05-09T08:49:29.271 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 422, 2024-05-09T08:49:29.271 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 422, 2024-05-09T08:49:29.271 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 422, 2024-05-09T08:49:29.271 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2024-05-09T08:49:29.271 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.271 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.271 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.271 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:29.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.271 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-write_buf_throttle": { 2024-05-09T08:49:29.271 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2024-05-09T08:49:29.271 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 3758096384, 2024-05-09T08:49:29.271 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2024-05-09T08:49:29.271 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2024-05-09T08:49:29.271 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2024-05-09T08:49:29.272 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:29.272 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:29.272 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2024-05-09T08:49:29.272 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2024-05-09T08:49:29.272 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 0, 2024-05-09T08:49:29.272 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 0, 2024-05-09T08:49:29.272 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2024-05-09T08:49:29.272 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.272 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.272 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.272 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:29.272 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-05-09T08:49:29.272 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-write_buf_throttle-0x55927dd786e0": { 2024-05-09T08:49:29.272 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2024-05-09T08:49:29.273 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 3758096384, 2024-05-09T08:49:29.273 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2024-05-09T08:49:29.273 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2024-05-09T08:49:29.273 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2024-05-09T08:49:29.273 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2024-05-09T08:49:29.273 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2024-05-09T08:49:29.273 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2024-05-09T08:49:29.273 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2024-05-09T08:49:29.273 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 0, 2024-05-09T08:49:29.273 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 0, 2024-05-09T08:49:29.273 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2024-05-09T08:49:29.273 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2024-05-09T08:49:29.273 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2024-05-09T08:49:29.273 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2024-05-09T08:49:29.273 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:29.273 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-05-09T08:49:29.274 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-05-09T08:49:30.007 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-05-09T08:49:30.007 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-05-09T08:49:30.007 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-05-09T08:49:30.007 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-05-09T08:49:30.007 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=6 2024-05-09T08:49:30.008 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-05-09T08:49:30.008 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-05-09T08:49:30.008 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-05-09T08:49:30.008 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-05-09T08:49:30.020 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-05-09T08:49:30.020 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:49:30.021 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T08:49:30.050 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:49:30.051 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T08:49:30.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:49:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.266+0000 7f3490bf9640 1 -- 172.21.15.107:0/762895903 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f348c1021b0 msgr2=0x7f348c10e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.266+0000 7f3490bf9640 1 --2- 172.21.15.107:0/762895903 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f348c1021b0 0x7f348c10e6c0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f3484009cd0 tx=0x7f348402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.266+0000 7f3490bf9640 1 -- 172.21.15.107:0/762895903 shutdown_connections 2024-05-09T08:49:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.266+0000 7f3490bf9640 1 --2- 172.21.15.107:0/762895903 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f348c1021b0 0x7f348c10e6c0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.266+0000 7f3490bf9640 1 --2- 172.21.15.107:0/762895903 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f348c1017f0 0x7f348c101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.266+0000 7f3490bf9640 1 --2- 172.21.15.107:0/762895903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f348c107800 0x7f348c107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.266+0000 7f3490bf9640 1 -- 172.21.15.107:0/762895903 >> 172.21.15.107:0/762895903 conn(0x7f348c078c30 msgr2=0x7f348c079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.266+0000 7f3490bf9640 1 -- 172.21.15.107:0/762895903 shutdown_connections 2024-05-09T08:49:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.266+0000 7f3490bf9640 1 -- 172.21.15.107:0/762895903 wait complete. 2024-05-09T08:49:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f3490bf9640 1 Processor -- start 2024-05-09T08:49:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f3490bf9640 1 -- start start 2024-05-09T08:49:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f3490bf9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f348c1017f0 0x7f348c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f3490bf9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f348c1021b0 0x7f348c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f3490bf9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f348c107800 0x7f348c0745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f3490bf9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f348c111a80 con 0x7f348c107800 2024-05-09T08:49:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f3490bf9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f348c111900 con 0x7f348c1017f0 2024-05-09T08:49:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f3490bf9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f348c111c00 con 0x7f348c1021b0 2024-05-09T08:49:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f348ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f348c107800 0x7f348c0745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f348ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f348c107800 0x7f348c0745b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42892/0 (socket says 172.21.15.107:42892) 2024-05-09T08:49:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f348ad9d640 1 -- 172.21.15.107:0/2167050040 learned_addr learned my addr 172.21.15.107:0/2167050040 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:30.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f348a59c640 1 --2- 172.21.15.107:0/2167050040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f348c1017f0 0x7f348c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f3483fff640 1 --2- 172.21.15.107:0/2167050040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f348c1021b0 0x7f348c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f348ad9d640 1 -- 172.21.15.107:0/2167050040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f348c1021b0 msgr2=0x7f348c074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f348ad9d640 1 --2- 172.21.15.107:0/2167050040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f348c1021b0 0x7f348c074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f348ad9d640 1 -- 172.21.15.107:0/2167050040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f348c1017f0 msgr2=0x7f348c073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f348ad9d640 1 --2- 172.21.15.107:0/2167050040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f348c1017f0 0x7f348c073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f348ad9d640 1 -- 172.21.15.107:0/2167050040 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f348c1a6500 con 0x7f348c107800 2024-05-09T08:49:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f348a59c640 1 --2- 172.21.15.107:0/2167050040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f348c1017f0 0x7f348c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:49:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.267+0000 7f348ad9d640 1 --2- 172.21.15.107:0/2167050040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f348c107800 0x7f348c0745b0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f3484005cc0 tx=0x7f3484003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.268+0000 7f34837fe640 1 -- 172.21.15.107:0/2167050040 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3484004450 con 0x7f348c107800 2024-05-09T08:49:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.268+0000 7f34837fe640 1 -- 172.21.15.107:0/2167050040 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3484031c90 con 0x7f348c107800 2024-05-09T08:49:30.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.268+0000 7f3490bf9640 1 -- 172.21.15.107:0/2167050040 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f348c1a6790 con 0x7f348c107800 2024-05-09T08:49:30.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.268+0000 7f34837fe640 1 -- 172.21.15.107:0/2167050040 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348403b400 con 0x7f348c107800 2024-05-09T08:49:30.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.268+0000 7f3490bf9640 1 -- 172.21.15.107:0/2167050040 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f348c1a6ce0 con 0x7f348c107800 2024-05-09T08:49:30.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.269+0000 7f3490bf9640 1 -- 172.21.15.107:0/2167050040 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f348c068aa0 con 0x7f348c107800 2024-05-09T08:49:30.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.270+0000 7f34837fe640 1 -- 172.21.15.107:0/2167050040 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3484004680 con 0x7f348c107800 2024-05-09T08:49:30.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.270+0000 7f34837fe640 1 --2- 172.21.15.107:0/2167050040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f34780a2d00 0x7f34780a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:30.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.271+0000 7f348a59c640 1 --2- 172.21.15.107:0/2167050040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f34780a2d00 0x7f34780a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:30.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.271+0000 7f34837fe640 1 -- 172.21.15.107:0/2167050040 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9451+0+0 (secure 0 0 0) 0x7f348402e030 con 0x7f348c107800 2024-05-09T08:49:30.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.271+0000 7f348a59c640 1 --2- 172.21.15.107:0/2167050040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f34780a2d00 0x7f34780a51c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f347c005d90 tx=0x7f347c005d20 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:30.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.274+0000 7f34837fe640 1 -- 172.21.15.107:0/2167050040 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f34840bd8b0 con 0x7f348c107800 2024-05-09T08:49:30.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:30 smithi157 ceph-mon[33614]: pgmap v509: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-09T08:49:30.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.450+0000 7f3490bf9640 1 -- 172.21.15.107:0/2167050040 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f348c02d1c0 con 0x7f348c107800 2024-05-09T08:49:30.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.452+0000 7f34837fe640 1 -- 172.21.15.107:0/2167050040 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) ==== 74+0+25307 (secure 0 0 0) 0x7f34840bd1b0 con 0x7f348c107800 2024-05-09T08:49:30.451 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:49:30.451 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":116,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:49:22.053969+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/2923766500":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/2417648229":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1642826815":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/2689615560":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/846948416":"2024-05-10T08:33:20.828572+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/846948416":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":27,"length":1}]},{"pool":4,"snaps":[{"begin":14,"length":1},{"begin":27,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:49:30.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.454+0000 7f3490bf9640 1 -- 172.21.15.107:0/2167050040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f34780a2d00 msgr2=0x7f34780a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.454+0000 7f3490bf9640 1 --2- 172.21.15.107:0/2167050040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f34780a2d00 0x7f34780a51c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f347c005d90 tx=0x7f347c005d20 comp rx=0 tx=0).stop 2024-05-09T08:49:30.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.454+0000 7f3490bf9640 1 -- 172.21.15.107:0/2167050040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f348c107800 msgr2=0x7f348c0745b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.454+0000 7f3490bf9640 1 --2- 172.21.15.107:0/2167050040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f348c107800 0x7f348c0745b0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f3484005cc0 tx=0x7f3484003c20 comp rx=0 tx=0).stop 2024-05-09T08:49:30.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.454+0000 7f3490bf9640 1 -- 172.21.15.107:0/2167050040 shutdown_connections 2024-05-09T08:49:30.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.454+0000 7f3490bf9640 1 --2- 172.21.15.107:0/2167050040 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f34780a2d00 0x7f34780a51c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.454+0000 7f3490bf9640 1 --2- 172.21.15.107:0/2167050040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f348c107800 0x7f348c0745b0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.454+0000 7f3490bf9640 1 --2- 172.21.15.107:0/2167050040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f348c1021b0 0x7f348c074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.454+0000 7f3490bf9640 1 --2- 172.21.15.107:0/2167050040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f348c1017f0 0x7f348c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.454+0000 7f3490bf9640 1 -- 172.21.15.107:0/2167050040 >> 172.21.15.107:0/2167050040 conn(0x7f348c078c30 msgr2=0x7f348c104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:30.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.454+0000 7f3490bf9640 1 -- 172.21.15.107:0/2167050040 shutdown_connections 2024-05-09T08:49:30.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.454+0000 7f3490bf9640 1 -- 172.21.15.107:0/2167050040 wait complete. 2024-05-09T08:49:30.458 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:30 smithi107 ceph-mon[30122]: pgmap v509: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-09T08:49:30.465 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:26:21.698338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:30:58.268633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:30:59.345579+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 27, 'snap_epoch': 114, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:31:03.065991+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 27, 'snap_epoch': 114, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:49:30.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:49:30.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.571+0000 7f738b0c3640 1 -- 172.21.15.107:0/3501512628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7384106720 msgr2=0x7f7384106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.571+0000 7f738b0c3640 1 --2- 172.21.15.107:0/3501512628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7384106720 0x7f7384106b00 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f736c009930 tx=0x7f736c02a420 comp rx=0 tx=0).stop 2024-05-09T08:49:30.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.571+0000 7f738b0c3640 1 -- 172.21.15.107:0/3501512628 shutdown_connections 2024-05-09T08:49:30.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.571+0000 7f738b0c3640 1 --2- 172.21.15.107:0/3501512628 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73841010d0 0x7f738410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.571+0000 7f738b0c3640 1 --2- 172.21.15.107:0/3501512628 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7384100710 0x7f7384100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.571+0000 7f738b0c3640 1 --2- 172.21.15.107:0/3501512628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7384106720 0x7f7384106b00 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.571+0000 7f738b0c3640 1 -- 172.21.15.107:0/3501512628 >> 172.21.15.107:0/3501512628 conn(0x7f7384076ab0 msgr2=0x7f7384076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.571+0000 7f738b0c3640 1 -- 172.21.15.107:0/3501512628 shutdown_connections 2024-05-09T08:49:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.571+0000 7f738b0c3640 1 -- 172.21.15.107:0/3501512628 wait complete. 2024-05-09T08:49:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.571+0000 7f738b0c3640 1 Processor -- start 2024-05-09T08:49:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.571+0000 7f738b0c3640 1 -- start start 2024-05-09T08:49:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f738b0c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7384100710 0x7f7384198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f738b0c3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73841010d0 0x7f73841986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f738b0c3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7384106720 0x7f738419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f738b0c3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7384111b60 con 0x7f7384100710 2024-05-09T08:49:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f738b0c3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f73841119e0 con 0x7f7384106720 2024-05-09T08:49:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f738b0c3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7384111ce0 con 0x7f73841010d0 2024-05-09T08:49:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f7388e5f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7384100710 0x7f7384198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f7388e5f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7384100710 0x7f7384198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42914/0 (socket says 172.21.15.107:42914) 2024-05-09T08:49:30.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f7388e5f640 1 -- 172.21.15.107:0/2396243204 learned_addr learned my addr 172.21.15.107:0/2396243204 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:30.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f7388e5f640 1 -- 172.21.15.107:0/2396243204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73841010d0 msgr2=0x7f73841986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f7388e5f640 1 --2- 172.21.15.107:0/2396243204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73841010d0 0x7f73841986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f7388e5f640 1 -- 172.21.15.107:0/2396243204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7384106720 msgr2=0x7f738419ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f7389660640 1 --2- 172.21.15.107:0/2396243204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7384106720 0x7f738419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:30.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f7388e5f640 1 --2- 172.21.15.107:0/2396243204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7384106720 0x7f738419ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f7388e5f640 1 -- 172.21.15.107:0/2396243204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f738419d180 con 0x7f7384100710 2024-05-09T08:49:30.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f7388e5f640 1 --2- 172.21.15.107:0/2396243204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7384100710 0x7f7384198190 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f736c005940 tx=0x7f736c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:30.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f7381ffb640 1 -- 172.21.15.107:0/2396243204 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736c03f030 con 0x7f7384100710 2024-05-09T08:49:30.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f7381ffb640 1 -- 172.21.15.107:0/2396243204 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f736c003cb0 con 0x7f7384100710 2024-05-09T08:49:30.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f738b0c3640 1 -- 172.21.15.107:0/2396243204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f738419d410 con 0x7f7384100710 2024-05-09T08:49:30.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f7381ffb640 1 -- 172.21.15.107:0/2396243204 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736c035070 con 0x7f7384100710 2024-05-09T08:49:30.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.572+0000 7f738b0c3640 1 -- 172.21.15.107:0/2396243204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73841a4d00 con 0x7f7384100710 2024-05-09T08:49:30.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.573+0000 7f738b0c3640 1 -- 172.21.15.107:0/2396243204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7384068aa0 con 0x7f7384100710 2024-05-09T08:49:30.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.575+0000 7f7381ffb640 1 -- 172.21.15.107:0/2396243204 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f736c042030 con 0x7f7384100710 2024-05-09T08:49:30.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.575+0000 7f7381ffb640 1 --2- 172.21.15.107:0/2396243204 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f73700a2d00 0x7f73700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:30.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.575+0000 7f7383fff640 1 --2- 172.21.15.107:0/2396243204 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f73700a2d00 0x7f73700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:30.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.575+0000 7f7381ffb640 1 -- 172.21.15.107:0/2396243204 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9451+0+0 (secure 0 0 0) 0x7f736c0f0070 con 0x7f7384100710 2024-05-09T08:49:30.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.576+0000 7f7383fff640 1 --2- 172.21.15.107:0/2396243204 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f73700a2d00 0x7f73700a51c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f73841000d0 tx=0x7f7374009250 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:30.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.579+0000 7f7381ffb640 1 -- 172.21.15.107:0/2396243204 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f736c0bc8f0 con 0x7f7384100710 2024-05-09T08:49:30.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:30 smithi110 ceph-mon[33622]: pgmap v509: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-09T08:49:30.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.754+0000 7f738b0c3640 1 -- 172.21.15.107:0/2396243204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7384108c80 con 0x7f7384100710 2024-05-09T08:49:30.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.755+0000 7f7381ffb640 1 -- 172.21.15.107:0/2396243204 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) ==== 93+0+10 (secure 0 0 0) 0x7f736c0bc1f0 con 0x7f7384100710 2024-05-09T08:49:30.754 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-05-09T08:49:30.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.756+0000 7f738b0c3640 1 -- 172.21.15.107:0/2396243204 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f73700a2d00 msgr2=0x7f73700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.756+0000 7f738b0c3640 1 --2- 172.21.15.107:0/2396243204 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f73700a2d00 0x7f73700a51c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f73841000d0 tx=0x7f7374009250 comp rx=0 tx=0).stop 2024-05-09T08:49:30.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.756+0000 7f738b0c3640 1 -- 172.21.15.107:0/2396243204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7384100710 msgr2=0x7f7384198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.757+0000 7f738b0c3640 1 --2- 172.21.15.107:0/2396243204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7384100710 0x7f7384198190 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f736c005940 tx=0x7f736c0042f0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.757+0000 7f738b0c3640 1 -- 172.21.15.107:0/2396243204 shutdown_connections 2024-05-09T08:49:30.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.757+0000 7f738b0c3640 1 --2- 172.21.15.107:0/2396243204 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f73700a2d00 0x7f73700a51c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.757+0000 7f738b0c3640 1 --2- 172.21.15.107:0/2396243204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7384106720 0x7f738419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.757+0000 7f738b0c3640 1 --2- 172.21.15.107:0/2396243204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73841010d0 0x7f73841986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.757+0000 7f738b0c3640 1 --2- 172.21.15.107:0/2396243204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7384100710 0x7f7384198190 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.757+0000 7f738b0c3640 1 -- 172.21.15.107:0/2396243204 >> 172.21.15.107:0/2396243204 conn(0x7f7384076ab0 msgr2=0x7f73840fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:30.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.757+0000 7f738b0c3640 1 -- 172.21.15.107:0/2396243204 shutdown_connections 2024-05-09T08:49:30.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.757+0000 7f738b0c3640 1 -- 172.21.15.107:0/2396243204 wait complete. 2024-05-09T08:49:30.767 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:49:30.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.871+0000 7f1400314640 1 -- 172.21.15.107:0/479762666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13f8100700 msgr2=0x7f13f8100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.871+0000 7f1400314640 1 --2- 172.21.15.107:0/479762666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13f8100700 0x7f13f8100b80 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f13ec009930 tx=0x7f13ec02a420 comp rx=0 tx=0).stop 2024-05-09T08:49:30.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f1400314640 1 -- 172.21.15.107:0/479762666 shutdown_connections 2024-05-09T08:49:30.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f1400314640 1 --2- 172.21.15.107:0/479762666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13f81010c0 0x7f13f810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f1400314640 1 --2- 172.21.15.107:0/479762666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13f8100700 0x7f13f8100b80 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f1400314640 1 --2- 172.21.15.107:0/479762666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13f8106710 0x7f13f8106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f1400314640 1 -- 172.21.15.107:0/479762666 >> 172.21.15.107:0/479762666 conn(0x7f13f8077830 msgr2=0x7f13f8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f1400314640 1 -- 172.21.15.107:0/479762666 shutdown_connections 2024-05-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f1400314640 1 -- 172.21.15.107:0/479762666 wait complete. 2024-05-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f1400314640 1 Processor -- start 2024-05-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f1400314640 1 -- start start 2024-05-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f1400314640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13f8100700 0x7f13f81980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f1400314640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13f81010c0 0x7f13f8198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f1400314640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13f8106710 0x7f13f819c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f1400314640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f13f81109d0 con 0x7f13f8106710 2024-05-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f1400314640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f13f8110850 con 0x7f13f8100700 2024-05-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f1400314640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f13f8110b50 con 0x7f13f81010c0 2024-05-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f13fe8b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13f8106710 0x7f13f819c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f13fe8b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13f8106710 0x7f13f819c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36658/0 (socket says 172.21.15.107:36658) 2024-05-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f13fe8b1640 1 -- 172.21.15.107:0/1245573879 learned_addr learned my addr 172.21.15.107:0/1245573879 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.872+0000 7f13fe0b0640 1 --2- 172.21.15.107:0/1245573879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13f8100700 0x7f13f81980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.873+0000 7f13fd8af640 1 --2- 172.21.15.107:0/1245573879 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13f81010c0 0x7f13f8198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.873+0000 7f13fe8b1640 1 -- 172.21.15.107:0/1245573879 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13f81010c0 msgr2=0x7f13f8198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.873+0000 7f13fe8b1640 1 --2- 172.21.15.107:0/1245573879 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13f81010c0 0x7f13f8198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.873+0000 7f13fe8b1640 1 -- 172.21.15.107:0/1245573879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13f8100700 msgr2=0x7f13f81980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.873+0000 7f13fe8b1640 1 --2- 172.21.15.107:0/1245573879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13f8100700 0x7f13f81980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.873+0000 7f13fe8b1640 1 -- 172.21.15.107:0/1245573879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f819d0d0 con 0x7f13f8106710 2024-05-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.873+0000 7f13fe8b1640 1 --2- 172.21.15.107:0/1245573879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13f8106710 0x7f13f819c9a0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f13f000c970 tx=0x7f13f000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.873+0000 7f13f77fe640 1 -- 172.21.15.107:0/1245573879 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f0007bb0 con 0x7f13f8106710 2024-05-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.873+0000 7f13f77fe640 1 -- 172.21.15.107:0/1245573879 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13f0007d50 con 0x7f13f8106710 2024-05-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.873+0000 7f13f77fe640 1 -- 172.21.15.107:0/1245573879 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f0004e10 con 0x7f13f8106710 2024-05-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.873+0000 7f1400314640 1 -- 172.21.15.107:0/1245573879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13f819d3c0 con 0x7f13f8106710 2024-05-09T08:49:30.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.873+0000 7f1400314640 1 -- 172.21.15.107:0/1245573879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13f81a4c50 con 0x7f13f8106710 2024-05-09T08:49:30.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.874+0000 7f1400314640 1 -- 172.21.15.107:0/1245573879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13f8068650 con 0x7f13f8106710 2024-05-09T08:49:30.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.875+0000 7f13f77fe640 1 -- 172.21.15.107:0/1245573879 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f13f0005000 con 0x7f13f8106710 2024-05-09T08:49:30.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.876+0000 7f13f77fe640 1 --2- 172.21.15.107:0/1245573879 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13e00a2d80 0x7f13e00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:30.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.876+0000 7f13fe0b0640 1 --2- 172.21.15.107:0/1245573879 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13e00a2d80 0x7f13e00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:30.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.876+0000 7f13f77fe640 1 -- 172.21.15.107:0/1245573879 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9451+0+0 (secure 0 0 0) 0x7f13f00cf800 con 0x7f13f8106710 2024-05-09T08:49:30.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.876+0000 7f13fe0b0640 1 --2- 172.21.15.107:0/1245573879 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13e00a2d80 0x7f13e00a5240 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f13e800a400 tx=0x7f13e800d040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:30.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:30.879+0000 7f13f77fe640 1 -- 172.21.15.107:0/1245573879 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f13f0014030 con 0x7f13f8106710 2024-05-09T08:49:31.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.054+0000 7f1400314640 1 -- 172.21.15.107:0/1245573879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f13f802d1c0 con 0x7f13f8106710 2024-05-09T08:49:31.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.054+0000 7f13f77fe640 1 -- 172.21.15.107:0/1245573879 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) ==== 104+0+11 (secure 0 0 0) 0x7f13f009c100 con 0x7f13f8106710 2024-05-09T08:49:31.054 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:49:31.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.056+0000 7f1400314640 1 -- 172.21.15.107:0/1245573879 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13e00a2d80 msgr2=0x7f13e00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.056+0000 7f1400314640 1 --2- 172.21.15.107:0/1245573879 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13e00a2d80 0x7f13e00a5240 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f13e800a400 tx=0x7f13e800d040 comp rx=0 tx=0).stop 2024-05-09T08:49:31.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.056+0000 7f1400314640 1 -- 172.21.15.107:0/1245573879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13f8106710 msgr2=0x7f13f819c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.056+0000 7f1400314640 1 --2- 172.21.15.107:0/1245573879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13f8106710 0x7f13f819c9a0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f13f000c970 tx=0x7f13f000ce30 comp rx=0 tx=0).stop 2024-05-09T08:49:31.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.056+0000 7f1400314640 1 -- 172.21.15.107:0/1245573879 shutdown_connections 2024-05-09T08:49:31.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.056+0000 7f1400314640 1 --2- 172.21.15.107:0/1245573879 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f13e00a2d80 0x7f13e00a5240 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.056+0000 7f1400314640 1 --2- 172.21.15.107:0/1245573879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13f8106710 0x7f13f819c9a0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.056+0000 7f1400314640 1 --2- 172.21.15.107:0/1245573879 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13f81010c0 0x7f13f8198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.056+0000 7f1400314640 1 --2- 172.21.15.107:0/1245573879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13f8100700 0x7f13f81980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.056+0000 7f1400314640 1 -- 172.21.15.107:0/1245573879 >> 172.21.15.107:0/1245573879 conn(0x7f13f8077830 msgr2=0x7f13f8068dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:31.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.057+0000 7f1400314640 1 -- 172.21.15.107:0/1245573879 shutdown_connections 2024-05-09T08:49:31.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.057+0000 7f1400314640 1 -- 172.21.15.107:0/1245573879 wait complete. 2024-05-09T08:49:31.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:49:31.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.173+0000 7fb57f914640 1 -- 172.21.15.107:0/2137526388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5781021c0 msgr2=0x7fb57810e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.173+0000 7fb57f914640 1 --2- 172.21.15.107:0/2137526388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5781021c0 0x7fb57810e6d0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fb570009cd0 tx=0x7fb57002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.173+0000 7fb57f914640 1 -- 172.21.15.107:0/2137526388 shutdown_connections 2024-05-09T08:49:31.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.173+0000 7fb57f914640 1 --2- 172.21.15.107:0/2137526388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5781021c0 0x7fb57810e6d0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.173+0000 7fb57f914640 1 --2- 172.21.15.107:0/2137526388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb578101800 0x7fb578101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.173+0000 7fb57f914640 1 --2- 172.21.15.107:0/2137526388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb578107810 0x7fb578107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.173+0000 7fb57f914640 1 -- 172.21.15.107:0/2137526388 >> 172.21.15.107:0/2137526388 conn(0x7fb578078c80 msgr2=0x7fb578079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:31.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.173+0000 7fb57f914640 1 -- 172.21.15.107:0/2137526388 shutdown_connections 2024-05-09T08:49:31.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.173+0000 7fb57f914640 1 -- 172.21.15.107:0/2137526388 wait complete. 2024-05-09T08:49:31.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57f914640 1 Processor -- start 2024-05-09T08:49:31.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57f914640 1 -- start start 2024-05-09T08:49:31.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57f914640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb578101800 0x7fb578197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:31.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57f914640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5781021c0 0x7fb578198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:31.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57f914640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb578107810 0x7fb57819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:31.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57f914640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb578111a90 con 0x7fb578101800 2024-05-09T08:49:31.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57f914640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb578111910 con 0x7fb5781021c0 2024-05-09T08:49:31.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57f914640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb578111c10 con 0x7fb578107810 2024-05-09T08:49:31.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57d6b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb578101800 0x7fb578197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:31.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57d6b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb578101800 0x7fb578197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36680/0 (socket says 172.21.15.107:36680) 2024-05-09T08:49:31.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57d6b0640 1 -- 172.21.15.107:0/3278458267 learned_addr learned my addr 172.21.15.107:0/3278458267 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:31.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57ceaf640 1 --2- 172.21.15.107:0/3278458267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5781021c0 0x7fb578198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:31.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57d6b0640 1 -- 172.21.15.107:0/3278458267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb578107810 msgr2=0x7fb57819c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57d6b0640 1 --2- 172.21.15.107:0/3278458267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb578107810 0x7fb57819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57d6b0640 1 -- 172.21.15.107:0/3278458267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5781021c0 msgr2=0x7fb578198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57d6b0640 1 --2- 172.21.15.107:0/3278458267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5781021c0 0x7fb578198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57d6b0640 1 -- 172.21.15.107:0/3278458267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb57819cee0 con 0x7fb578101800 2024-05-09T08:49:31.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.174+0000 7fb57ceaf640 1 --2- 172.21.15.107:0/3278458267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5781021c0 0x7fb578198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:49:31.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.175+0000 7fb57d6b0640 1 --2- 172.21.15.107:0/3278458267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb578101800 0x7fb578197ef0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fb57400c920 tx=0x7fb57400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:31.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.175+0000 7fb56e7fc640 1 -- 172.21.15.107:0/3278458267 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574007c70 con 0x7fb578101800 2024-05-09T08:49:31.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.175+0000 7fb56e7fc640 1 -- 172.21.15.107:0/3278458267 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb574007e10 con 0x7fb578101800 2024-05-09T08:49:31.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.175+0000 7fb57f914640 1 -- 172.21.15.107:0/3278458267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb57819d1d0 con 0x7fb578101800 2024-05-09T08:49:31.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.175+0000 7fb56e7fc640 1 -- 172.21.15.107:0/3278458267 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574018820 con 0x7fb578101800 2024-05-09T08:49:31.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.175+0000 7fb57f914640 1 -- 172.21.15.107:0/3278458267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5781a4a60 con 0x7fb578101800 2024-05-09T08:49:31.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.176+0000 7fb57f914640 1 -- 172.21.15.107:0/3278458267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb578068aa0 con 0x7fb578101800 2024-05-09T08:49:31.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.177+0000 7fb56e7fc640 1 -- 172.21.15.107:0/3278458267 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb5740189c0 con 0x7fb578101800 2024-05-09T08:49:31.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.178+0000 7fb56e7fc640 1 --2- 172.21.15.107:0/3278458267 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb5640a2d00 0x7fb5640a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:31.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.178+0000 7fb57ceaf640 1 --2- 172.21.15.107:0/3278458267 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb5640a2d00 0x7fb5640a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:31.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.178+0000 7fb56e7fc640 1 -- 172.21.15.107:0/3278458267 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9451+0+0 (secure 0 0 0) 0x7fb5740cfc00 con 0x7fb578101800 2024-05-09T08:49:31.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.178+0000 7fb57ceaf640 1 --2- 172.21.15.107:0/3278458267 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb5640a2d00 0x7fb5640a51c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb5780fe860 tx=0x7fb56800a5c0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:31.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.181+0000 7fb56e7fc640 1 -- 172.21.15.107:0/3278458267 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb574014030 con 0x7fb578101800 2024-05-09T08:49:31.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.361+0000 7fb57f914640 1 -- 172.21.15.107:0/3278458267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb578199240 con 0x7fb578101800 2024-05-09T08:49:31.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.361+0000 7fb56e7fc640 1 -- 172.21.15.107:0/3278458267 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) ==== 100+0+11 (secure 0 0 0) 0x7fb57409c510 con 0x7fb578101800 2024-05-09T08:49:31.361 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-05-09T08:49:31.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.363+0000 7fb57f914640 1 -- 172.21.15.107:0/3278458267 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb5640a2d00 msgr2=0x7fb5640a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.363+0000 7fb57f914640 1 --2- 172.21.15.107:0/3278458267 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb5640a2d00 0x7fb5640a51c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb5780fe860 tx=0x7fb56800a5c0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.363+0000 7fb57f914640 1 -- 172.21.15.107:0/3278458267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb578101800 msgr2=0x7fb578197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.363+0000 7fb57f914640 1 --2- 172.21.15.107:0/3278458267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb578101800 0x7fb578197ef0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fb57400c920 tx=0x7fb57400cde0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.363+0000 7fb57f914640 1 -- 172.21.15.107:0/3278458267 shutdown_connections 2024-05-09T08:49:31.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.363+0000 7fb57f914640 1 --2- 172.21.15.107:0/3278458267 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb5640a2d00 0x7fb5640a51c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.363+0000 7fb57f914640 1 --2- 172.21.15.107:0/3278458267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb578107810 0x7fb57819c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.363+0000 7fb57f914640 1 --2- 172.21.15.107:0/3278458267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5781021c0 0x7fb578198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.363+0000 7fb57f914640 1 --2- 172.21.15.107:0/3278458267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb578101800 0x7fb578197ef0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.363+0000 7fb57f914640 1 -- 172.21.15.107:0/3278458267 >> 172.21.15.107:0/3278458267 conn(0x7fb578078c80 msgr2=0x7fb5780ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:31.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.364+0000 7fb57f914640 1 -- 172.21.15.107:0/3278458267 shutdown_connections 2024-05-09T08:49:31.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.364+0000 7fb57f914640 1 -- 172.21.15.107:0/3278458267 wait complete. 2024-05-09T08:49:31.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:31 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2167050040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:49:31.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:31 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2396243204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:49:31.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:31 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1245573879' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:49:31.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:49:31.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.479+0000 7f7b63db0640 1 -- 172.21.15.107:0/1524252297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b5c06a870 msgr2=0x7f7b5c10e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.479+0000 7f7b63db0640 1 --2- 172.21.15.107:0/1524252297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b5c06a870 0x7f7b5c10e6e0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f7b54009930 tx=0x7f7b5402a420 comp rx=0 tx=0).stop 2024-05-09T08:49:31.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.479+0000 7f7b63db0640 1 -- 172.21.15.107:0/1524252297 shutdown_connections 2024-05-09T08:49:31.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.479+0000 7f7b63db0640 1 --2- 172.21.15.107:0/1524252297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b5c06a870 0x7f7b5c10e6e0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.479+0000 7f7b63db0640 1 --2- 172.21.15.107:0/1524252297 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b5c069eb0 0x7f7b5c06a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.479+0000 7f7b63db0640 1 --2- 172.21.15.107:0/1524252297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b5c107820 0x7f7b5c107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.479+0000 7f7b63db0640 1 -- 172.21.15.107:0/1524252297 >> 172.21.15.107:0/1524252297 conn(0x7f7b5c077790 msgr2=0x7f7b5c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.480+0000 7f7b63db0640 1 -- 172.21.15.107:0/1524252297 shutdown_connections 2024-05-09T08:49:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.480+0000 7f7b63db0640 1 -- 172.21.15.107:0/1524252297 wait complete. 2024-05-09T08:49:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.480+0000 7f7b63db0640 1 Processor -- start 2024-05-09T08:49:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.480+0000 7f7b63db0640 1 -- start start 2024-05-09T08:49:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.480+0000 7f7b63db0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b5c069eb0 0x7f7b5c197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.480+0000 7f7b63db0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b5c06a870 0x7f7b5c198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.480+0000 7f7b63db0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b5c107820 0x7f7b5c19c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.480+0000 7f7b63db0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5c111aa0 con 0x7f7b5c069eb0 2024-05-09T08:49:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.480+0000 7f7b63db0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5c111920 con 0x7f7b5c06a870 2024-05-09T08:49:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.480+0000 7f7b63db0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5c111c20 con 0x7f7b5c107820 2024-05-09T08:49:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.480+0000 7f7b61b4c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b5c069eb0 0x7f7b5c197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.480+0000 7f7b61b4c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b5c069eb0 0x7f7b5c197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36698/0 (socket says 172.21.15.107:36698) 2024-05-09T08:49:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.480+0000 7f7b61b4c640 1 -- 172.21.15.107:0/3570132742 learned_addr learned my addr 172.21.15.107:0/3570132742 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.480+0000 7f7b6134b640 1 --2- 172.21.15.107:0/3570132742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b5c06a870 0x7f7b5c198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.481+0000 7f7b6234d640 1 --2- 172.21.15.107:0/3570132742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b5c107820 0x7f7b5c19c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.481+0000 7f7b61b4c640 1 -- 172.21.15.107:0/3570132742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b5c107820 msgr2=0x7f7b5c19c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.481+0000 7f7b61b4c640 1 --2- 172.21.15.107:0/3570132742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b5c107820 0x7f7b5c19c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.481+0000 7f7b61b4c640 1 -- 172.21.15.107:0/3570132742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b5c06a870 msgr2=0x7f7b5c198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.481+0000 7f7b61b4c640 1 --2- 172.21.15.107:0/3570132742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b5c06a870 0x7f7b5c198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.481+0000 7f7b61b4c640 1 -- 172.21.15.107:0/3570132742 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b5c19ced0 con 0x7f7b5c069eb0 2024-05-09T08:49:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.481+0000 7f7b61b4c640 1 --2- 172.21.15.107:0/3570132742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b5c069eb0 0x7f7b5c197ee0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c00db80 tx=0x7f7b4c00df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.481+0000 7f7b5affd640 1 -- 172.21.15.107:0/3570132742 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c017070 con 0x7f7b5c069eb0 2024-05-09T08:49:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.481+0000 7f7b5affd640 1 -- 172.21.15.107:0/3570132742 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b4c00be30 con 0x7f7b5c069eb0 2024-05-09T08:49:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.481+0000 7f7b63db0640 1 -- 172.21.15.107:0/3570132742 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b5c19d1c0 con 0x7f7b5c069eb0 2024-05-09T08:49:31.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.481+0000 7f7b5affd640 1 -- 172.21.15.107:0/3570132742 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c010460 con 0x7f7b5c069eb0 2024-05-09T08:49:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.481+0000 7f7b63db0640 1 -- 172.21.15.107:0/3570132742 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b5c1a4a50 con 0x7f7b5c069eb0 2024-05-09T08:49:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.482+0000 7f7b63db0640 1 -- 172.21.15.107:0/3570132742 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b5c068aa0 con 0x7f7b5c069eb0 2024-05-09T08:49:31.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.483+0000 7f7b5affd640 1 -- 172.21.15.107:0/3570132742 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7b4c003970 con 0x7f7b5c069eb0 2024-05-09T08:49:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.484+0000 7f7b5affd640 1 --2- 172.21.15.107:0/3570132742 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b440a2d00 0x7f7b440a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.484+0000 7f7b5affd640 1 -- 172.21.15.107:0/3570132742 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9451+0+0 (secure 0 0 0) 0x7f7b4c0cf4c0 con 0x7f7b5c069eb0 2024-05-09T08:49:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.484+0000 7f7b6134b640 1 --2- 172.21.15.107:0/3570132742 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b440a2d00 0x7f7b440a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:31.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.484+0000 7f7b6134b640 1 --2- 172.21.15.107:0/3570132742 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b440a2d00 0x7f7b440a51c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c077c20 tx=0x7f7b5000a2d0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:31.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.488+0000 7f7b5affd640 1 -- 172.21.15.107:0/3570132742 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7b4c020070 con 0x7f7b5c069eb0 2024-05-09T08:49:31.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:31 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2167050040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:49:31.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:31 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2396243204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:49:31.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:31 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1245573879' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:49:31.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:31 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2167050040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:49:31.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:31 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2396243204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:49:31.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:31 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1245573879' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:49:31.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.663+0000 7f7b63db0640 1 -- 172.21.15.107:0/3570132742 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7b5c1a4d80 con 0x7f7b5c069eb0 2024-05-09T08:49:31.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.663+0000 7f7b5affd640 1 -- 172.21.15.107:0/3570132742 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v116) ==== 103+0+11 (secure 0 0 0) 0x7f7b4c097020 con 0x7f7b5c069eb0 2024-05-09T08:49:31.662 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:49:31.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.665+0000 7f7b63db0640 1 -- 172.21.15.107:0/3570132742 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b440a2d00 msgr2=0x7f7b440a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.665+0000 7f7b63db0640 1 --2- 172.21.15.107:0/3570132742 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b440a2d00 0x7f7b440a51c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c077c20 tx=0x7f7b5000a2d0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.665+0000 7f7b63db0640 1 -- 172.21.15.107:0/3570132742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b5c069eb0 msgr2=0x7f7b5c197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.665+0000 7f7b63db0640 1 --2- 172.21.15.107:0/3570132742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b5c069eb0 0x7f7b5c197ee0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c00db80 tx=0x7f7b4c00df80 comp rx=0 tx=0).stop 2024-05-09T08:49:31.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.665+0000 7f7b63db0640 1 -- 172.21.15.107:0/3570132742 shutdown_connections 2024-05-09T08:49:31.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.665+0000 7f7b63db0640 1 --2- 172.21.15.107:0/3570132742 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f7b440a2d00 0x7f7b440a51c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.665+0000 7f7b63db0640 1 --2- 172.21.15.107:0/3570132742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b5c107820 0x7f7b5c19c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.665+0000 7f7b63db0640 1 --2- 172.21.15.107:0/3570132742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b5c06a870 0x7f7b5c198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.665+0000 7f7b63db0640 1 --2- 172.21.15.107:0/3570132742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b5c069eb0 0x7f7b5c197ee0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.665+0000 7f7b63db0640 1 -- 172.21.15.107:0/3570132742 >> 172.21.15.107:0/3570132742 conn(0x7f7b5c077790 msgr2=0x7f7b5c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:31.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.665+0000 7f7b63db0640 1 -- 172.21.15.107:0/3570132742 shutdown_connections 2024-05-09T08:49:31.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.665+0000 7f7b63db0640 1 -- 172.21.15.107:0/3570132742 wait complete. 2024-05-09T08:49:31.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T08:49:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.779+0000 7fe025aee640 1 -- 172.21.15.107:0/1542066378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe02010bdf0 msgr2=0x7fe02010e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.779+0000 7fe025aee640 1 --2- 172.21.15.107:0/1542066378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe02010bdf0 0x7fe02010e290 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fe018009ca0 tx=0x7fe01802a9e0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.780+0000 7fe025aee640 1 -- 172.21.15.107:0/1542066378 shutdown_connections 2024-05-09T08:49:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.780+0000 7fe025aee640 1 --2- 172.21.15.107:0/1542066378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe02010bdf0 0x7fe02010e290 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.780+0000 7fe025aee640 1 --2- 172.21.15.107:0/1542066378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0201033d0 0x7fe02010b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.780+0000 7fe025aee640 1 --2- 172.21.15.107:0/1542066378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe020102ab0 0x7fe020102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.780+0000 7fe025aee640 1 -- 172.21.15.107:0/1542066378 >> 172.21.15.107:0/1542066378 conn(0x7fe020077830 msgr2=0x7fe020077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:31.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.780+0000 7fe025aee640 1 -- 172.21.15.107:0/1542066378 shutdown_connections 2024-05-09T08:49:31.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.780+0000 7fe025aee640 1 -- 172.21.15.107:0/1542066378 wait complete. 2024-05-09T08:49:31.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.780+0000 7fe025aee640 1 Processor -- start 2024-05-09T08:49:31.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.780+0000 7fe025aee640 1 -- start start 2024-05-09T08:49:31.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.780+0000 7fe025aee640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe020102ab0 0x7fe020100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:31.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.780+0000 7fe025aee640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0201033d0 0x7fe020100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:31.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.780+0000 7fe025aee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe02010bdf0 0x7fe020101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:31.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.780+0000 7fe025aee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe020111aa0 con 0x7fe02010bdf0 2024-05-09T08:49:31.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.780+0000 7fe025aee640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe020111920 con 0x7fe0201033d0 2024-05-09T08:49:31.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.780+0000 7fe025aee640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe020111c20 con 0x7fe020102ab0 2024-05-09T08:49:31.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.781+0000 7fe01ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe02010bdf0 0x7fe020101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:31.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.781+0000 7fe01ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe02010bdf0 0x7fe020101180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36708/0 (socket says 172.21.15.107:36708) 2024-05-09T08:49:31.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.781+0000 7fe01ffff640 1 -- 172.21.15.107:0/2690389585 learned_addr learned my addr 172.21.15.107:0/2690389585 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:31.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.781+0000 7fe017fff640 1 --2- 172.21.15.107:0/2690389585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0201033d0 0x7fe020100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:31.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.781+0000 7fe01f7fe640 1 --2- 172.21.15.107:0/2690389585 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe020102ab0 0x7fe020100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:31.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.781+0000 7fe01ffff640 1 -- 172.21.15.107:0/2690389585 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe020102ab0 msgr2=0x7fe020100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.781+0000 7fe01ffff640 1 --2- 172.21.15.107:0/2690389585 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe020102ab0 0x7fe020100700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.781+0000 7fe01ffff640 1 -- 172.21.15.107:0/2690389585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0201033d0 msgr2=0x7fe020100c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.781+0000 7fe01ffff640 1 --2- 172.21.15.107:0/2690389585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0201033d0 0x7fe020100c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.781+0000 7fe01ffff640 1 -- 172.21.15.107:0/2690389585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0201a64c0 con 0x7fe02010bdf0 2024-05-09T08:49:31.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.781+0000 7fe01ffff640 1 --2- 172.21.15.107:0/2690389585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe02010bdf0 0x7fe020101180 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fe018009dc0 tx=0x7fe01802fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:31.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.781+0000 7fe01d7fa640 1 -- 172.21.15.107:0/2690389585 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe018033040 con 0x7fe02010bdf0 2024-05-09T08:49:31.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.781+0000 7fe01d7fa640 1 -- 172.21.15.107:0/2690389585 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe01802e7a0 con 0x7fe02010bdf0 2024-05-09T08:49:31.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.781+0000 7fe025aee640 1 -- 172.21.15.107:0/2690389585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0201a6750 con 0x7fe02010bdf0 2024-05-09T08:49:31.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.781+0000 7fe01d7fa640 1 -- 172.21.15.107:0/2690389585 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe018043650 con 0x7fe02010bdf0 2024-05-09T08:49:31.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.781+0000 7fe025aee640 1 -- 172.21.15.107:0/2690389585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0201a6be0 con 0x7fe02010bdf0 2024-05-09T08:49:31.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.782+0000 7fe025aee640 1 -- 172.21.15.107:0/2690389585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe020068650 con 0x7fe02010bdf0 2024-05-09T08:49:31.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.783+0000 7fe01d7fa640 1 -- 172.21.15.107:0/2690389585 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe018036070 con 0x7fe02010bdf0 2024-05-09T08:49:31.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.784+0000 7fe01d7fa640 1 --2- 172.21.15.107:0/2690389585 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe00c0a2d00 0x7fe00c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:31.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.784+0000 7fe01f7fe640 1 --2- 172.21.15.107:0/2690389585 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe00c0a2d00 0x7fe00c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:31.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.785+0000 7fe01d7fa640 1 -- 172.21.15.107:0/2690389585 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9451+0+0 (secure 0 0 0) 0x7fe0180f04f0 con 0x7fe02010bdf0 2024-05-09T08:49:31.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.785+0000 7fe01f7fe640 1 --2- 172.21.15.107:0/2690389585 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe00c0a2d00 0x7fe00c0a51c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe010009b60 tx=0x7fe010005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:31.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.789+0000 7fe01d7fa640 1 -- 172.21.15.107:0/2690389585 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe01803e070 con 0x7fe02010bdf0 2024-05-09T08:49:31.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.976+0000 7fe025aee640 1 -- 172.21.15.107:0/2690389585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe020106f70 con 0x7fe02010bdf0 2024-05-09T08:49:31.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.977+0000 7fe01d7fa640 1 -- 172.21.15.107:0/2690389585 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v116) ==== 81+0+959 (secure 0 0 0) 0x7fe0180bcd70 con 0x7fe02010bdf0 2024-05-09T08:49:31.976 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/2923766500 2024-05-10T08:33:20.828572+0000 2024-05-09T08:49:31.976 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/2417648229 2024-05-10T08:33:20.828572+0000 2024-05-09T08:49:31.976 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/1642826815 2024-05-10T08:33:20.828572+0000 2024-05-09T08:49:31.976 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/1064417772 2024-05-10T08:23:55.372959+0000 2024-05-09T08:49:31.976 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/3922375261 2024-05-10T08:23:55.372959+0000 2024-05-09T08:49:31.976 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/3922375261 2024-05-10T08:23:55.372959+0000 2024-05-09T08:49:31.976 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/3934964045 2024-05-10T08:23:55.372959+0000 2024-05-09T08:49:31.977 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/2689615560 2024-05-10T08:33:20.828572+0000 2024-05-09T08:49:31.977 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/1620469781 2024-05-10T08:23:29.066232+0000 2024-05-09T08:49:31.977 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4101303556 2024-05-10T08:23:29.066232+0000 2024-05-09T08:49:31.977 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/846948416 2024-05-10T08:33:20.828572+0000 2024-05-09T08:49:31.977 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/2360407889 2024-05-10T08:23:29.066232+0000 2024-05-09T08:49:31.977 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/846948416 2024-05-10T08:33:20.828572+0000 2024-05-09T08:49:31.977 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/164842293 2024-05-10T08:23:29.066232+0000 2024-05-09T08:49:31.977 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/2360407889 2024-05-10T08:23:29.066232+0000 2024-05-09T08:49:31.977 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4237993479 2024-05-10T08:23:55.372959+0000 2024-05-09T08:49:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:listed 16 entries 2024-05-09T08:49:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.979+0000 7fe025aee640 1 -- 172.21.15.107:0/2690389585 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe00c0a2d00 msgr2=0x7fe00c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.979+0000 7fe025aee640 1 --2- 172.21.15.107:0/2690389585 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe00c0a2d00 0x7fe00c0a51c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe010009b60 tx=0x7fe010005f90 comp rx=0 tx=0).stop 2024-05-09T08:49:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.979+0000 7fe025aee640 1 -- 172.21.15.107:0/2690389585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe02010bdf0 msgr2=0x7fe020101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.979+0000 7fe025aee640 1 --2- 172.21.15.107:0/2690389585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe02010bdf0 0x7fe020101180 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fe018009dc0 tx=0x7fe01802fbe0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.979+0000 7fe025aee640 1 -- 172.21.15.107:0/2690389585 shutdown_connections 2024-05-09T08:49:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.979+0000 7fe025aee640 1 --2- 172.21.15.107:0/2690389585 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe00c0a2d00 0x7fe00c0a51c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.979+0000 7fe025aee640 1 --2- 172.21.15.107:0/2690389585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe02010bdf0 0x7fe020101180 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.979+0000 7fe025aee640 1 --2- 172.21.15.107:0/2690389585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0201033d0 0x7fe020100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.979+0000 7fe025aee640 1 --2- 172.21.15.107:0/2690389585 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe020102ab0 0x7fe020100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.979+0000 7fe025aee640 1 -- 172.21.15.107:0/2690389585 >> 172.21.15.107:0/2690389585 conn(0x7fe020077830 msgr2=0x7fe0200fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.979+0000 7fe025aee640 1 -- 172.21.15.107:0/2690389585 shutdown_connections 2024-05-09T08:49:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:31.979+0000 7fe025aee640 1 -- 172.21.15.107:0/2690389585 wait complete. 2024-05-09T08:49:31.989 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-05-09T08:49:31.989 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T08:49:32.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:49:32.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.148+0000 7f87e9d09640 1 -- 172.21.15.107:0/265254994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e4106710 msgr2=0x7f87e4106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.148+0000 7f87e9d09640 1 --2- 172.21.15.107:0/265254994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e4106710 0x7f87e4106af0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009cd0 tx=0x7f87d402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.148+0000 7f87e9d09640 1 -- 172.21.15.107:0/265254994 shutdown_connections 2024-05-09T08:49:32.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.148+0000 7f87e9d09640 1 --2- 172.21.15.107:0/265254994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e41010c0 0x7f87e410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.148+0000 7f87e9d09640 1 --2- 172.21.15.107:0/265254994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87e4100700 0x7f87e4100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.148+0000 7f87e9d09640 1 --2- 172.21.15.107:0/265254994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e4106710 0x7f87e4106af0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.148+0000 7f87e9d09640 1 -- 172.21.15.107:0/265254994 >> 172.21.15.107:0/265254994 conn(0x7f87e4076ab0 msgr2=0x7f87e4076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:32.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.148+0000 7f87e9d09640 1 -- 172.21.15.107:0/265254994 shutdown_connections 2024-05-09T08:49:32.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.148+0000 7f87e9d09640 1 -- 172.21.15.107:0/265254994 wait complete. 2024-05-09T08:49:32.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.148+0000 7f87e9d09640 1 Processor -- start 2024-05-09T08:49:32.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e9d09640 1 -- start start 2024-05-09T08:49:32.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e9d09640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87e4100700 0x7f87e4199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e9d09640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e41010c0 0x7f87e4199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e9d09640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e4106710 0x7f87e41a40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e9d09640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f87e4110a50 con 0x7f87e4106710 2024-05-09T08:49:32.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e9d09640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f87e41108d0 con 0x7f87e41010c0 2024-05-09T08:49:32.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e9d09640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f87e4110bd0 con 0x7f87e4100700 2024-05-09T08:49:32.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e4106710 0x7f87e41a40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e4106710 0x7f87e41a40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36728/0 (socket says 172.21.15.107:36728) 2024-05-09T08:49:32.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e3fff640 1 -- 172.21.15.107:0/1394057019 learned_addr learned my addr 172.21.15.107:0/1394057019 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:32.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e37fe640 1 --2- 172.21.15.107:0/1394057019 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87e4100700 0x7f87e4199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e2ffd640 1 --2- 172.21.15.107:0/1394057019 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e41010c0 0x7f87e4199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e3fff640 1 -- 172.21.15.107:0/1394057019 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87e4100700 msgr2=0x7f87e4199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e3fff640 1 --2- 172.21.15.107:0/1394057019 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87e4100700 0x7f87e4199020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e3fff640 1 -- 172.21.15.107:0/1394057019 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e41010c0 msgr2=0x7f87e4199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e3fff640 1 --2- 172.21.15.107:0/1394057019 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e41010c0 0x7f87e4199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e3fff640 1 -- 172.21.15.107:0/1394057019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e4196e10 con 0x7f87e4106710 2024-05-09T08:49:32.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e2ffd640 1 --2- 172.21.15.107:0/1394057019 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e41010c0 0x7f87e4199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:49:32.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e3fff640 1 --2- 172.21.15.107:0/1394057019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e4106710 0x7f87e41a40a0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009fd0 tx=0x7f87dc00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:32.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.149+0000 7f87e0ff9640 1 -- 172.21.15.107:0/1394057019 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc011830 con 0x7f87e4106710 2024-05-09T08:49:32.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.150+0000 7f87e9d09640 1 -- 172.21.15.107:0/1394057019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e4197100 con 0x7f87e4106710 2024-05-09T08:49:32.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.150+0000 7f87e0ff9640 1 -- 172.21.15.107:0/1394057019 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87dc016070 con 0x7f87e4106710 2024-05-09T08:49:32.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.150+0000 7f87e0ff9640 1 -- 172.21.15.107:0/1394057019 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc010de0 con 0x7f87e4106710 2024-05-09T08:49:32.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.150+0000 7f87e9d09640 1 -- 172.21.15.107:0/1394057019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e4197600 con 0x7f87e4106710 2024-05-09T08:49:32.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.150+0000 7f87e9d09640 1 -- 172.21.15.107:0/1394057019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87e40696e0 con 0x7f87e4106710 2024-05-09T08:49:32.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.152+0000 7f87e0ff9640 1 -- 172.21.15.107:0/1394057019 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f87dc0119d0 con 0x7f87e4106710 2024-05-09T08:49:32.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.153+0000 7f87e0ff9640 1 --2- 172.21.15.107:0/1394057019 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f87d00a2d00 0x7f87d00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.153+0000 7f87e37fe640 1 --2- 172.21.15.107:0/1394057019 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f87d00a2d00 0x7f87d00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.153+0000 7f87e0ff9640 1 -- 172.21.15.107:0/1394057019 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9451+0+0 (secure 0 0 0) 0x7f87dc0cf470 con 0x7f87e4106710 2024-05-09T08:49:32.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.153+0000 7f87e37fe640 1 --2- 172.21.15.107:0/1394057019 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f87d00a2d00 0x7f87d00a51c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f87d4006010 tx=0x7f87d4005f70 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:32.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.156+0000 7f87e0ff9640 1 -- 172.21.15.107:0/1394057019 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f87dc09be00 con 0x7f87e4106710 2024-05-09T08:49:32.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:32 smithi107 ceph-mon[30122]: pgmap v510: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-09T08:49:32.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:32 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3278458267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:49:32.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:32 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3570132742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:49:32.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:32 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2690389585' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:49:32.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.334+0000 7f87e9d09640 1 -- 172.21.15.107:0/1394057019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f87e402d1c0 con 0x7f87e4106710 2024-05-09T08:49:32.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.336+0000 7f87e0ff9640 1 -- 172.21.15.107:0/1394057019 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) ==== 74+0+25307 (secure 0 0 0) 0x7f87dc09b700 con 0x7f87e4106710 2024-05-09T08:49:32.335 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:49:32.335 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":116,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:49:22.053969+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/2923766500":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/2417648229":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1642826815":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/2689615560":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/846948416":"2024-05-10T08:33:20.828572+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/846948416":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":27,"length":1}]},{"pool":4,"snaps":[{"begin":14,"length":1},{"begin":27,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:49:32.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.338+0000 7f87e9d09640 1 -- 172.21.15.107:0/1394057019 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f87d00a2d00 msgr2=0x7f87d00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.338+0000 7f87e9d09640 1 --2- 172.21.15.107:0/1394057019 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f87d00a2d00 0x7f87d00a51c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f87d4006010 tx=0x7f87d4005f70 comp rx=0 tx=0).stop 2024-05-09T08:49:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.338+0000 7f87e9d09640 1 -- 172.21.15.107:0/1394057019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e4106710 msgr2=0x7f87e41a40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.338+0000 7f87e9d09640 1 --2- 172.21.15.107:0/1394057019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e4106710 0x7f87e41a40a0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009fd0 tx=0x7f87dc00b8e0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.338+0000 7f87e9d09640 1 -- 172.21.15.107:0/1394057019 shutdown_connections 2024-05-09T08:49:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.338+0000 7f87e9d09640 1 --2- 172.21.15.107:0/1394057019 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f87d00a2d00 0x7f87d00a51c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.338+0000 7f87e9d09640 1 --2- 172.21.15.107:0/1394057019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e4106710 0x7f87e41a40a0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.338+0000 7f87e9d09640 1 --2- 172.21.15.107:0/1394057019 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e41010c0 0x7f87e4199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.338+0000 7f87e9d09640 1 --2- 172.21.15.107:0/1394057019 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87e4100700 0x7f87e4199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.338+0000 7f87e9d09640 1 -- 172.21.15.107:0/1394057019 >> 172.21.15.107:0/1394057019 conn(0x7f87e4076ab0 msgr2=0x7f87e40fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.338+0000 7f87e9d09640 1 -- 172.21.15.107:0/1394057019 shutdown_connections 2024-05-09T08:49:32.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.338+0000 7f87e9d09640 1 -- 172.21.15.107:0/1394057019 wait complete. 2024-05-09T08:49:32.349 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T08:26:21.698338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T08:30:58.268633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T08:30:59.345579+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 27, 'snap_epoch': 114, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T08:31:03.065991+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 27, 'snap_epoch': 114, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T08:49:32.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T08:49:32.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.454+0000 7f9358a12640 1 -- 172.21.15.107:0/2347295731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93541021c0 msgr2=0x7f935410e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.454+0000 7f9358a12640 1 --2- 172.21.15.107:0/2347295731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93541021c0 0x7f935410e6d0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f934c009cd0 tx=0x7f934c02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.454+0000 7f9358a12640 1 -- 172.21.15.107:0/2347295731 shutdown_connections 2024-05-09T08:49:32.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.454+0000 7f9358a12640 1 --2- 172.21.15.107:0/2347295731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93541021c0 0x7f935410e6d0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.454+0000 7f9358a12640 1 --2- 172.21.15.107:0/2347295731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9354101800 0x7f9354101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.454+0000 7f9358a12640 1 --2- 172.21.15.107:0/2347295731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9354107810 0x7f9354107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.454+0000 7f9358a12640 1 -- 172.21.15.107:0/2347295731 >> 172.21.15.107:0/2347295731 conn(0x7f9354078c80 msgr2=0x7f9354079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:32.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.454+0000 7f9358a12640 1 -- 172.21.15.107:0/2347295731 shutdown_connections 2024-05-09T08:49:32.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.454+0000 7f9358a12640 1 -- 172.21.15.107:0/2347295731 wait complete. 2024-05-09T08:49:32.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.454+0000 7f9358a12640 1 Processor -- start 2024-05-09T08:49:32.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9358a12640 1 -- start start 2024-05-09T08:49:32.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9358a12640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9354101800 0x7f9354198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9358a12640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93541021c0 0x7f9354198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9358a12640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9354107810 0x7f935419c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9358a12640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9354111ad0 con 0x7f93541021c0 2024-05-09T08:49:32.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9358a12640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9354111950 con 0x7f9354107810 2024-05-09T08:49:32.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9358a12640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9354111c50 con 0x7f9354101800 2024-05-09T08:49:32.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9351d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93541021c0 0x7f9354198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9351d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93541021c0 0x7f9354198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36740/0 (socket says 172.21.15.107:36740) 2024-05-09T08:49:32.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9351d9b640 1 -- 172.21.15.107:0/287351981 learned_addr learned my addr 172.21.15.107:0/287351981 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:32.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9352d9d640 1 --2- 172.21.15.107:0/287351981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9354107810 0x7f935419c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f935259c640 1 --2- 172.21.15.107:0/287351981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9354101800 0x7f9354198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9351d9b640 1 -- 172.21.15.107:0/287351981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9354101800 msgr2=0x7f9354198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9351d9b640 1 --2- 172.21.15.107:0/287351981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9354101800 0x7f9354198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9351d9b640 1 -- 172.21.15.107:0/287351981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9354107810 msgr2=0x7f935419c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9351d9b640 1 --2- 172.21.15.107:0/287351981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9354107810 0x7f935419c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9351d9b640 1 -- 172.21.15.107:0/287351981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f935419d020 con 0x7f93541021c0 2024-05-09T08:49:32.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f935259c640 1 --2- 172.21.15.107:0/287351981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9354101800 0x7f9354198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:49:32.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.455+0000 7f9351d9b640 1 --2- 172.21.15.107:0/287351981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93541021c0 0x7f9354198570 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f934400d950 tx=0x7f934400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:32.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.456+0000 7f934b7fe640 1 -- 172.21.15.107:0/287351981 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344017070 con 0x7f93541021c0 2024-05-09T08:49:32.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.456+0000 7f9358a12640 1 -- 172.21.15.107:0/287351981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f935419d310 con 0x7f93541021c0 2024-05-09T08:49:32.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.456+0000 7f934b7fe640 1 -- 172.21.15.107:0/287351981 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9344009c30 con 0x7f93541021c0 2024-05-09T08:49:32.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.456+0000 7f934b7fe640 1 -- 172.21.15.107:0/287351981 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934400f620 con 0x7f93541021c0 2024-05-09T08:49:32.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.456+0000 7f9358a12640 1 -- 172.21.15.107:0/287351981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93541a4ba0 con 0x7f93541021c0 2024-05-09T08:49:32.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.457+0000 7f9358a12640 1 -- 172.21.15.107:0/287351981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9354068aa0 con 0x7f93541021c0 2024-05-09T08:49:32.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.458+0000 7f934b7fe640 1 -- 172.21.15.107:0/287351981 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f93440109a0 con 0x7f93541021c0 2024-05-09T08:49:32.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.458+0000 7f934b7fe640 1 --2- 172.21.15.107:0/287351981 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f93400a2d00 0x7f93400a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.459+0000 7f935259c640 1 --2- 172.21.15.107:0/287351981 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f93400a2d00 0x7f93400a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.459+0000 7f934b7fe640 1 -- 172.21.15.107:0/287351981 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9451+0+0 (secure 0 0 0) 0x7f93440cf420 con 0x7f93541021c0 2024-05-09T08:49:32.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.459+0000 7f935259c640 1 --2- 172.21.15.107:0/287351981 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f93400a2d00 0x7f93400a51c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f933c005d70 tx=0x7f933c005d00 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:32.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.463+0000 7f934b7fe640 1 -- 172.21.15.107:0/287351981 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9344009de0 con 0x7f93541021c0 2024-05-09T08:49:32.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:32 smithi157 ceph-mon[33614]: pgmap v510: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-09T08:49:32.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:32 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3278458267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:49:32.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:32 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3570132742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:49:32.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:32 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2690389585' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:49:32.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:32 smithi110 ceph-mon[33622]: pgmap v510: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-05-09T08:49:32.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:32 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3278458267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:49:32.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:32 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3570132742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:49:32.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:32 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2690389585' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:49:32.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.637+0000 7f9358a12640 1 -- 172.21.15.107:0/287351981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f935402d1c0 con 0x7f93541021c0 2024-05-09T08:49:32.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.637+0000 7f934b7fe640 1 -- 172.21.15.107:0/287351981 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) ==== 93+0+10 (secure 0 0 0) 0x7f934409bca0 con 0x7f93541021c0 2024-05-09T08:49:32.637 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-05-09T08:49:32.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.639+0000 7f9358a12640 1 -- 172.21.15.107:0/287351981 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f93400a2d00 msgr2=0x7f93400a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.640+0000 7f9358a12640 1 --2- 172.21.15.107:0/287351981 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f93400a2d00 0x7f93400a51c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f933c005d70 tx=0x7f933c005d00 comp rx=0 tx=0).stop 2024-05-09T08:49:32.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.640+0000 7f9358a12640 1 -- 172.21.15.107:0/287351981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93541021c0 msgr2=0x7f9354198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.640+0000 7f9358a12640 1 --2- 172.21.15.107:0/287351981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93541021c0 0x7f9354198570 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f934400d950 tx=0x7f934400de10 comp rx=0 tx=0).stop 2024-05-09T08:49:32.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.640+0000 7f9358a12640 1 -- 172.21.15.107:0/287351981 shutdown_connections 2024-05-09T08:49:32.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.640+0000 7f9358a12640 1 --2- 172.21.15.107:0/287351981 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f93400a2d00 0x7f93400a51c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.640+0000 7f9358a12640 1 --2- 172.21.15.107:0/287351981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9354107810 0x7f935419c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.640+0000 7f9358a12640 1 --2- 172.21.15.107:0/287351981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93541021c0 0x7f9354198570 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.640+0000 7f9358a12640 1 --2- 172.21.15.107:0/287351981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9354101800 0x7f9354198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.640+0000 7f9358a12640 1 -- 172.21.15.107:0/287351981 >> 172.21.15.107:0/287351981 conn(0x7f9354078c80 msgr2=0x7f93540ff2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:32.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.640+0000 7f9358a12640 1 -- 172.21.15.107:0/287351981 shutdown_connections 2024-05-09T08:49:32.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.640+0000 7f9358a12640 1 -- 172.21.15.107:0/287351981 wait complete. 2024-05-09T08:49:32.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T08:49:32.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.755+0000 7f6d4d914640 1 -- 172.21.15.107:0/105910437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d4810bdf0 msgr2=0x7f6d4810e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.755+0000 7f6d4d914640 1 --2- 172.21.15.107:0/105910437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d4810bdf0 0x7f6d4810e290 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f6d40009900 tx=0x7f6d4002a510 comp rx=0 tx=0).stop 2024-05-09T08:49:32.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.755+0000 7f6d4d914640 1 -- 172.21.15.107:0/105910437 shutdown_connections 2024-05-09T08:49:32.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.755+0000 7f6d4d914640 1 --2- 172.21.15.107:0/105910437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d4810bdf0 0x7f6d4810e290 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.755+0000 7f6d4d914640 1 --2- 172.21.15.107:0/105910437 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d481033d0 0x7f6d4810b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.755+0000 7f6d4d914640 1 --2- 172.21.15.107:0/105910437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d48102ab0 0x7f6d48102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.755+0000 7f6d4d914640 1 -- 172.21.15.107:0/105910437 >> 172.21.15.107:0/105910437 conn(0x7f6d48077830 msgr2=0x7f6d48077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:32.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.755+0000 7f6d4d914640 1 -- 172.21.15.107:0/105910437 shutdown_connections 2024-05-09T08:49:32.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.755+0000 7f6d4d914640 1 -- 172.21.15.107:0/105910437 wait complete. 2024-05-09T08:49:32.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.756+0000 7f6d4d914640 1 Processor -- start 2024-05-09T08:49:32.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.756+0000 7f6d4d914640 1 -- start start 2024-05-09T08:49:32.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.756+0000 7f6d4d914640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d48102ab0 0x7f6d48198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.756+0000 7f6d4d914640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d481033d0 0x7f6d48198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.756+0000 7f6d4d914640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d4810bdf0 0x7f6d4819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.756+0000 7f6d4d914640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6d48111aa0 con 0x7f6d481033d0 2024-05-09T08:49:32.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.756+0000 7f6d4d914640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6d48111920 con 0x7f6d4810bdf0 2024-05-09T08:49:32.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.756+0000 7f6d4d914640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6d48111c20 con 0x7f6d48102ab0 2024-05-09T08:49:32.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d467fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d481033d0 0x7f6d48198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d467fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d481033d0 0x7f6d48198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36754/0 (socket says 172.21.15.107:36754) 2024-05-09T08:49:32.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d467fc640 1 -- 172.21.15.107:0/2445340611 learned_addr learned my addr 172.21.15.107:0/2445340611 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:32.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d46ffd640 1 --2- 172.21.15.107:0/2445340611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d48102ab0 0x7f6d48198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d477fe640 1 --2- 172.21.15.107:0/2445340611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d4810bdf0 0x7f6d4819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d467fc640 1 -- 172.21.15.107:0/2445340611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d48102ab0 msgr2=0x7f6d48198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d467fc640 1 --2- 172.21.15.107:0/2445340611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d48102ab0 0x7f6d48198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d467fc640 1 -- 172.21.15.107:0/2445340611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d4810bdf0 msgr2=0x7f6d4819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d467fc640 1 --2- 172.21.15.107:0/2445340611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d4810bdf0 0x7f6d4819c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d467fc640 1 -- 172.21.15.107:0/2445340611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4819cfc0 con 0x7f6d481033d0 2024-05-09T08:49:32.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d477fe640 1 --2- 172.21.15.107:0/2445340611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d4810bdf0 0x7f6d4819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:49:32.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d467fc640 1 --2- 172.21.15.107:0/2445340611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d481033d0 0x7f6d48198560 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f6d3000d770 tx=0x7f6d3000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:32.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d4c912640 1 -- 172.21.15.107:0/2445340611 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d30017070 con 0x7f6d481033d0 2024-05-09T08:49:32.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d4d914640 1 -- 172.21.15.107:0/2445340611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d4819d2b0 con 0x7f6d481033d0 2024-05-09T08:49:32.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d4c912640 1 -- 172.21.15.107:0/2445340611 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d300054d0 con 0x7f6d481033d0 2024-05-09T08:49:32.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d4c912640 1 -- 172.21.15.107:0/2445340611 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d30004cf0 con 0x7f6d481033d0 2024-05-09T08:49:32.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.757+0000 7f6d4d914640 1 -- 172.21.15.107:0/2445340611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d481a4b40 con 0x7f6d481033d0 2024-05-09T08:49:32.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.758+0000 7f6d4d914640 1 -- 172.21.15.107:0/2445340611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d48068650 con 0x7f6d481033d0 2024-05-09T08:49:32.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.759+0000 7f6d4c912640 1 -- 172.21.15.107:0/2445340611 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6d30004e90 con 0x7f6d481033d0 2024-05-09T08:49:32.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.760+0000 7f6d4c912640 1 --2- 172.21.15.107:0/2445340611 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6d340a2d00 0x7f6d340a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:32.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.760+0000 7f6d46ffd640 1 --2- 172.21.15.107:0/2445340611 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6d340a2d00 0x7f6d340a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:32.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.760+0000 7f6d4c912640 1 -- 172.21.15.107:0/2445340611 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9451+0+0 (secure 0 0 0) 0x7f6d300cf1c0 con 0x7f6d481033d0 2024-05-09T08:49:32.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.760+0000 7f6d46ffd640 1 --2- 172.21.15.107:0/2445340611 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6d340a2d00 0x7f6d340a51c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6d38005fd0 tx=0x7f6d38009680 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:32.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.764+0000 7f6d4c912640 1 -- 172.21.15.107:0/2445340611 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6d3009ba40 con 0x7f6d481033d0 2024-05-09T08:49:32.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.937+0000 7f6d4d914640 1 -- 172.21.15.107:0/2445340611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6d481069e0 con 0x7f6d481033d0 2024-05-09T08:49:32.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.938+0000 7f6d4c912640 1 -- 172.21.15.107:0/2445340611 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) ==== 104+0+11 (secure 0 0 0) 0x7f6d3009b340 con 0x7f6d481033d0 2024-05-09T08:49:32.937 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:49:32.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.940+0000 7f6d4d914640 1 -- 172.21.15.107:0/2445340611 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6d340a2d00 msgr2=0x7f6d340a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.940+0000 7f6d4d914640 1 --2- 172.21.15.107:0/2445340611 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6d340a2d00 0x7f6d340a51c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6d38005fd0 tx=0x7f6d38009680 comp rx=0 tx=0).stop 2024-05-09T08:49:32.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.940+0000 7f6d4d914640 1 -- 172.21.15.107:0/2445340611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d481033d0 msgr2=0x7f6d48198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:32.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.940+0000 7f6d4d914640 1 --2- 172.21.15.107:0/2445340611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d481033d0 0x7f6d48198560 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f6d3000d770 tx=0x7f6d3000dc30 comp rx=0 tx=0).stop 2024-05-09T08:49:32.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.940+0000 7f6d4d914640 1 -- 172.21.15.107:0/2445340611 shutdown_connections 2024-05-09T08:49:32.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.940+0000 7f6d4d914640 1 --2- 172.21.15.107:0/2445340611 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f6d340a2d00 0x7f6d340a51c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.940+0000 7f6d4d914640 1 --2- 172.21.15.107:0/2445340611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d4810bdf0 0x7f6d4819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.940+0000 7f6d4d914640 1 --2- 172.21.15.107:0/2445340611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d481033d0 0x7f6d48198560 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.940+0000 7f6d4d914640 1 --2- 172.21.15.107:0/2445340611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d48102ab0 0x7f6d48198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:32.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.940+0000 7f6d4d914640 1 -- 172.21.15.107:0/2445340611 >> 172.21.15.107:0/2445340611 conn(0x7f6d48077830 msgr2=0x7f6d480fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:32.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.940+0000 7f6d4d914640 1 -- 172.21.15.107:0/2445340611 shutdown_connections 2024-05-09T08:49:32.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:32.940+0000 7f6d4d914640 1 -- 172.21.15.107:0/2445340611 wait complete. 2024-05-09T08:49:32.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T08:49:33.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.054+0000 7fdc70b70640 1 -- 172.21.15.107:0/3814910247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc6c100a10 msgr2=0x7fdc6c100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:33.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.054+0000 7fdc70b70640 1 --2- 172.21.15.107:0/3814910247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc6c100a10 0x7fdc6c100e90 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fdc60009900 tx=0x7fdc6002a410 comp rx=0 tx=0).stop 2024-05-09T08:49:33.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.054+0000 7fdc70b70640 1 -- 172.21.15.107:0/3814910247 shutdown_connections 2024-05-09T08:49:33.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.054+0000 7fdc70b70640 1 --2- 172.21.15.107:0/3814910247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc6c1013d0 0x7fdc6c10d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.054+0000 7fdc70b70640 1 --2- 172.21.15.107:0/3814910247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc6c100a10 0x7fdc6c100e90 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.054+0000 7fdc70b70640 1 --2- 172.21.15.107:0/3814910247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc6c106900 0x7fdc6c106ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.055+0000 7fdc70b70640 1 -- 172.21.15.107:0/3814910247 >> 172.21.15.107:0/3814910247 conn(0x7fdc6c077710 msgr2=0x7fdc6c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:33.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.055+0000 7fdc70b70640 1 -- 172.21.15.107:0/3814910247 shutdown_connections 2024-05-09T08:49:33.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.055+0000 7fdc70b70640 1 -- 172.21.15.107:0/3814910247 wait complete. 2024-05-09T08:49:33.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.055+0000 7fdc70b70640 1 Processor -- start 2024-05-09T08:49:33.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.055+0000 7fdc70b70640 1 -- start start 2024-05-09T08:49:33.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.055+0000 7fdc70b70640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc6c100a10 0x7fdc6c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:33.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.055+0000 7fdc70b70640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc6c1013d0 0x7fdc6c1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:33.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.055+0000 7fdc70b70640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc6c106900 0x7fdc6c19c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:33.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.055+0000 7fdc70b70640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c111b80 con 0x7fdc6c100a10 2024-05-09T08:49:33.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.055+0000 7fdc70b70640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c111a00 con 0x7fdc6c1013d0 2024-05-09T08:49:33.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.055+0000 7fdc70b70640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c111d00 con 0x7fdc6c106900 2024-05-09T08:49:33.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.056+0000 7fdc6a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc6c100a10 0x7fdc6c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:33.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.056+0000 7fdc6a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc6c100a10 0x7fdc6c198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36772/0 (socket says 172.21.15.107:36772) 2024-05-09T08:49:33.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.056+0000 7fdc6a59c640 1 -- 172.21.15.107:0/86611327 learned_addr learned my addr 172.21.15.107:0/86611327 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:33.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.056+0000 7fdc6ad9d640 1 --2- 172.21.15.107:0/86611327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc6c106900 0x7fdc6c19c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:33.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.056+0000 7fdc6a59c640 1 -- 172.21.15.107:0/86611327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc6c106900 msgr2=0x7fdc6c19c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.056+0000 7fdc6a59c640 1 --2- 172.21.15.107:0/86611327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc6c106900 0x7fdc6c19c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.056+0000 7fdc6a59c640 1 -- 172.21.15.107:0/86611327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc6c1013d0 msgr2=0x7fdc6c1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.056+0000 7fdc6a59c640 1 --2- 172.21.15.107:0/86611327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc6c1013d0 0x7fdc6c1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.056+0000 7fdc6a59c640 1 -- 172.21.15.107:0/86611327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc6c19d070 con 0x7fdc6c100a10 2024-05-09T08:49:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.056+0000 7fdc6ad9d640 1 --2- 172.21.15.107:0/86611327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc6c106900 0x7fdc6c19c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:49:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.056+0000 7fdc6a59c640 1 --2- 172.21.15.107:0/86611327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc6c100a10 0x7fdc6c198080 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c00eb90 tx=0x7fdc5c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.056+0000 7fdc5b7fe640 1 -- 172.21.15.107:0/86611327 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c004d20 con 0x7fdc6c100a10 2024-05-09T08:49:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.056+0000 7fdc5b7fe640 1 -- 172.21.15.107:0/86611327 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc5c00ee90 con 0x7fdc6c100a10 2024-05-09T08:49:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.056+0000 7fdc70b70640 1 -- 172.21.15.107:0/86611327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc6c19d360 con 0x7fdc6c100a10 2024-05-09T08:49:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.056+0000 7fdc5b7fe640 1 -- 172.21.15.107:0/86611327 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c010670 con 0x7fdc6c100a10 2024-05-09T08:49:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.056+0000 7fdc70b70640 1 -- 172.21.15.107:0/86611327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc6c1a4bf0 con 0x7fdc6c100a10 2024-05-09T08:49:33.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.057+0000 7fdc70b70640 1 -- 172.21.15.107:0/86611327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc6c101ed0 con 0x7fdc6c100a10 2024-05-09T08:49:33.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.058+0000 7fdc5b7fe640 1 -- 172.21.15.107:0/86611327 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fdc5c005020 con 0x7fdc6c100a10 2024-05-09T08:49:33.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.059+0000 7fdc5b7fe640 1 --2- 172.21.15.107:0/86611327 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdc540a2d00 0x7fdc540a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:33.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.059+0000 7fdc69d9b640 1 --2- 172.21.15.107:0/86611327 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdc540a2d00 0x7fdc540a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:33.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.059+0000 7fdc5b7fe640 1 -- 172.21.15.107:0/86611327 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9451+0+0 (secure 0 0 0) 0x7fdc5c0cff60 con 0x7fdc6c100a10 2024-05-09T08:49:33.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.060+0000 7fdc69d9b640 1 --2- 172.21.15.107:0/86611327 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdc540a2d00 0x7fdc540a51c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fdc600096c0 tx=0x7fdc60031400 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:33.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.064+0000 7fdc5b7fe640 1 -- 172.21.15.107:0/86611327 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdc5c09c7e0 con 0x7fdc6c100a10 2024-05-09T08:49:33.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.243+0000 7fdc70b70640 1 -- 172.21.15.107:0/86611327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdc6c108bf0 con 0x7fdc6c100a10 2024-05-09T08:49:33.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.243+0000 7fdc5b7fe640 1 -- 172.21.15.107:0/86611327 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) ==== 100+0+11 (secure 0 0 0) 0x7fdc5c09c0e0 con 0x7fdc6c100a10 2024-05-09T08:49:33.242 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-05-09T08:49:33.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.245+0000 7fdc70b70640 1 -- 172.21.15.107:0/86611327 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdc540a2d00 msgr2=0x7fdc540a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.245+0000 7fdc70b70640 1 --2- 172.21.15.107:0/86611327 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdc540a2d00 0x7fdc540a51c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fdc600096c0 tx=0x7fdc60031400 comp rx=0 tx=0).stop 2024-05-09T08:49:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.245+0000 7fdc70b70640 1 -- 172.21.15.107:0/86611327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc6c100a10 msgr2=0x7fdc6c198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.245+0000 7fdc70b70640 1 --2- 172.21.15.107:0/86611327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc6c100a10 0x7fdc6c198080 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c00eb90 tx=0x7fdc5c00c6a0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.245+0000 7fdc70b70640 1 -- 172.21.15.107:0/86611327 shutdown_connections 2024-05-09T08:49:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.245+0000 7fdc70b70640 1 --2- 172.21.15.107:0/86611327 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fdc540a2d00 0x7fdc540a51c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.245+0000 7fdc70b70640 1 --2- 172.21.15.107:0/86611327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc6c106900 0x7fdc6c19c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.245+0000 7fdc70b70640 1 --2- 172.21.15.107:0/86611327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc6c1013d0 0x7fdc6c1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.245+0000 7fdc70b70640 1 --2- 172.21.15.107:0/86611327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc6c100a10 0x7fdc6c198080 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.245+0000 7fdc70b70640 1 -- 172.21.15.107:0/86611327 >> 172.21.15.107:0/86611327 conn(0x7fdc6c077710 msgr2=0x7fdc6c0ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.245+0000 7fdc70b70640 1 -- 172.21.15.107:0/86611327 shutdown_connections 2024-05-09T08:49:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.245+0000 7fdc70b70640 1 -- 172.21.15.107:0/86611327 wait complete. 2024-05-09T08:49:33.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T08:49:33.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:33 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1394057019' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:49:33.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:33 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/287351981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:49:33.344 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:33 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2445340611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:49:33.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.359+0000 7f8fe2838640 1 -- 172.21.15.107:0/3351392769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc100a80 msgr2=0x7f8fdc100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:33.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.359+0000 7f8fe2838640 1 --2- 172.21.15.107:0/3351392769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc100a80 0x7f8fdc100ee0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009900 tx=0x7f8fd002a410 comp rx=0 tx=0).stop 2024-05-09T08:49:33.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.359+0000 7f8fe2838640 1 -- 172.21.15.107:0/3351392769 shutdown_connections 2024-05-09T08:49:33.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.359+0000 7f8fe2838640 1 --2- 172.21.15.107:0/3351392769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fdc101420 0x7f8fdc10d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.359+0000 7f8fe2838640 1 --2- 172.21.15.107:0/3351392769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc100a80 0x7f8fdc100ee0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.359+0000 7f8fe2838640 1 --2- 172.21.15.107:0/3351392769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fdc106780 0x7f8fdc106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.360+0000 7f8fe2838640 1 -- 172.21.15.107:0/3351392769 >> 172.21.15.107:0/3351392769 conn(0x7f8fdc076530 msgr2=0x7f8fdc076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:33.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.360+0000 7f8fe2838640 1 -- 172.21.15.107:0/3351392769 shutdown_connections 2024-05-09T08:49:33.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.360+0000 7f8fe2838640 1 -- 172.21.15.107:0/3351392769 wait complete. 2024-05-09T08:49:33.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.360+0000 7f8fe2838640 1 Processor -- start 2024-05-09T08:49:33.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.360+0000 7f8fe2838640 1 -- start start 2024-05-09T08:49:33.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.360+0000 7f8fe2838640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fdc100a80 0x7f8fdc198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:33.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.360+0000 7f8fe2838640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc101420 0x7f8fdc199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:33.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.360+0000 7f8fe2838640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fdc106780 0x7f8fdc1a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:33.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fe2838640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8fdc110940 con 0x7f8fdc101420 2024-05-09T08:49:33.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fe2838640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8fdc1107c0 con 0x7f8fdc100a80 2024-05-09T08:49:33.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fe2838640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8fdc110ac0 con 0x7f8fdc106780 2024-05-09T08:49:33.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fdb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc101420 0x7f8fdc199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:33.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fdb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc101420 0x7f8fdc199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36778/0 (socket says 172.21.15.107:36778) 2024-05-09T08:49:33.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3721830360 learned_addr learned my addr 172.21.15.107:0/3721830360 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fdbfff640 1 --2- 172.21.15.107:0/3721830360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fdc100a80 0x7f8fdc198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fe0dd5640 1 --2- 172.21.15.107:0/3721830360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fdc106780 0x7f8fdc1a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3721830360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fdc106780 msgr2=0x7f8fdc1a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fdb7fe640 1 --2- 172.21.15.107:0/3721830360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fdc106780 0x7f8fdc1a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3721830360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fdc100a80 msgr2=0x7f8fdc198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fdb7fe640 1 --2- 172.21.15.107:0/3721830360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fdc100a80 0x7f8fdc198dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fdb7fe640 1 -- 172.21.15.107:0/3721830360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fdc1a4380 con 0x7f8fdc101420 2024-05-09T08:49:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fdb7fe640 1 --2- 172.21.15.107:0/3721830360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc101420 0x7f8fdc199300 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0005940 tx=0x7f8fd0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fd97fa640 1 -- 172.21.15.107:0/3721830360 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd003f030 con 0x7f8fdc101420 2024-05-09T08:49:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fd97fa640 1 -- 172.21.15.107:0/3721830360 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fd0003aa0 con 0x7f8fdc101420 2024-05-09T08:49:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fe2838640 1 -- 172.21.15.107:0/3721830360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fdc1a4550 con 0x7f8fdc101420 2024-05-09T08:49:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fd97fa640 1 -- 172.21.15.107:0/3721830360 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0035070 con 0x7f8fdc101420 2024-05-09T08:49:33.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.361+0000 7f8fe2838640 1 -- 172.21.15.107:0/3721830360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fdc1a49c0 con 0x7f8fdc101420 2024-05-09T08:49:33.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.362+0000 7f8fe2838640 1 -- 172.21.15.107:0/3721830360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fdc101e70 con 0x7f8fdc101420 2024-05-09T08:49:33.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.363+0000 7f8fd97fa640 1 -- 172.21.15.107:0/3721830360 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8fd0042030 con 0x7f8fdc101420 2024-05-09T08:49:33.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.364+0000 7f8fd97fa640 1 --2- 172.21.15.107:0/3721830360 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fc40a2d00 0x7f8fc40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:33.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.364+0000 7f8fdbfff640 1 --2- 172.21.15.107:0/3721830360 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fc40a2d00 0x7f8fc40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:33.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.364+0000 7f8fd97fa640 1 -- 172.21.15.107:0/3721830360 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9451+0+0 (secure 0 0 0) 0x7f8fd00efef0 con 0x7f8fdc101420 2024-05-09T08:49:33.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.364+0000 7f8fdbfff640 1 --2- 172.21.15.107:0/3721830360 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fc40a2d00 0x7f8fc40a51c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc005d00 tx=0x7f8fcc00b410 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.368+0000 7f8fd97fa640 1 -- 172.21.15.107:0/3721830360 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8fd00bc120 con 0x7f8fdc101420 2024-05-09T08:49:33.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:33 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1394057019' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:49:33.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:33 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/287351981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:49:33.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:33 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2445340611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:49:33.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.543+0000 7f8fe2838640 1 -- 172.21.15.107:0/3721830360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8fdc108bf0 con 0x7f8fdc101420 2024-05-09T08:49:33.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.543+0000 7f8fd97fa640 1 -- 172.21.15.107:0/3721830360 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v116) ==== 103+0+11 (secure 0 0 0) 0x7f8fd00c0420 con 0x7f8fdc101420 2024-05-09T08:49:33.542 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-05-09T08:49:33.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.545+0000 7f8fe2838640 1 -- 172.21.15.107:0/3721830360 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fc40a2d00 msgr2=0x7f8fc40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:33.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.545+0000 7f8fe2838640 1 --2- 172.21.15.107:0/3721830360 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fc40a2d00 0x7f8fc40a51c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc005d00 tx=0x7f8fcc00b410 comp rx=0 tx=0).stop 2024-05-09T08:49:33.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.545+0000 7f8fe2838640 1 -- 172.21.15.107:0/3721830360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc101420 msgr2=0x7f8fdc199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:33.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.545+0000 7f8fe2838640 1 --2- 172.21.15.107:0/3721830360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc101420 0x7f8fdc199300 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0005940 tx=0x7f8fd0004270 comp rx=0 tx=0).stop 2024-05-09T08:49:33.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.545+0000 7f8fe2838640 1 -- 172.21.15.107:0/3721830360 shutdown_connections 2024-05-09T08:49:33.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.545+0000 7f8fe2838640 1 --2- 172.21.15.107:0/3721830360 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fc40a2d00 0x7f8fc40a51c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.545+0000 7f8fe2838640 1 --2- 172.21.15.107:0/3721830360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fdc106780 0x7f8fdc1a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.545+0000 7f8fe2838640 1 --2- 172.21.15.107:0/3721830360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fdc101420 0x7f8fdc199300 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.545+0000 7f8fe2838640 1 --2- 172.21.15.107:0/3721830360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fdc100a80 0x7f8fdc198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.545+0000 7f8fe2838640 1 -- 172.21.15.107:0/3721830360 >> 172.21.15.107:0/3721830360 conn(0x7f8fdc076530 msgr2=0x7f8fdc0ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:33.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.545+0000 7f8fe2838640 1 -- 172.21.15.107:0/3721830360 shutdown_connections 2024-05-09T08:49:33.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.545+0000 7f8fe2838640 1 -- 172.21.15.107:0/3721830360 wait complete. 2024-05-09T08:49:33.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T08:49:33.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:33 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1394057019' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:49:33.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:33 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/287351981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T08:49:33.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:33 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2445340611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T08:49:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.659+0000 7fe7078b2640 1 -- 172.21.15.107:0/2913280458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe700069eb0 msgr2=0x7fe70006a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.659+0000 7fe7078b2640 1 --2- 172.21.15.107:0/2913280458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe700069eb0 0x7fe70006a330 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4009930 tx=0x7fe6f402a420 comp rx=0 tx=0).stop 2024-05-09T08:49:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.659+0000 7fe7078b2640 1 -- 172.21.15.107:0/2913280458 shutdown_connections 2024-05-09T08:49:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.659+0000 7fe7078b2640 1 --2- 172.21.15.107:0/2913280458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe70006a870 0x7fe70010e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.659+0000 7fe7078b2640 1 --2- 172.21.15.107:0/2913280458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe700069eb0 0x7fe70006a330 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.659+0000 7fe7078b2640 1 --2- 172.21.15.107:0/2913280458 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe700107820 0x7fe700107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.659+0000 7fe7078b2640 1 -- 172.21.15.107:0/2913280458 >> 172.21.15.107:0/2913280458 conn(0x7fe700077790 msgr2=0x7fe700077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.659+0000 7fe7078b2640 1 -- 172.21.15.107:0/2913280458 shutdown_connections 2024-05-09T08:49:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.659+0000 7fe7078b2640 1 -- 172.21.15.107:0/2913280458 wait complete. 2024-05-09T08:49:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe7078b2640 1 Processor -- start 2024-05-09T08:49:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe7078b2640 1 -- start start 2024-05-09T08:49:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe7078b2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe700069eb0 0x7fe700198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe7078b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe70006a870 0x7fe700198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe7078b2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe700107820 0x7fe70019c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe7078b2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe700111ae0 con 0x7fe70006a870 2024-05-09T08:49:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe7078b2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe700111960 con 0x7fe700069eb0 2024-05-09T08:49:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe7078b2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe700111c60 con 0x7fe700107820 2024-05-09T08:49:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe704e4d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe70006a870 0x7fe700198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe704e4d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe70006a870 0x7fe700198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36790/0 (socket says 172.21.15.107:36790) 2024-05-09T08:49:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe704e4d640 1 -- 172.21.15.107:0/4146014662 learned_addr learned my addr 172.21.15.107:0/4146014662 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe705e4f640 1 --2- 172.21.15.107:0/4146014662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe700107820 0x7fe70019c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe70564e640 1 --2- 172.21.15.107:0/4146014662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe700069eb0 0x7fe700198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe704e4d640 1 -- 172.21.15.107:0/4146014662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe700107820 msgr2=0x7fe70019c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe704e4d640 1 --2- 172.21.15.107:0/4146014662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe700107820 0x7fe70019c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe704e4d640 1 -- 172.21.15.107:0/4146014662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe700069eb0 msgr2=0x7fe700198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe704e4d640 1 --2- 172.21.15.107:0/4146014662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe700069eb0 0x7fe700198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe704e4d640 1 -- 172.21.15.107:0/4146014662 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe70019cff0 con 0x7fe70006a870 2024-05-09T08:49:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe704e4d640 1 --2- 172.21.15.107:0/4146014662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe70006a870 0x7fe700198540 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4004770 tx=0x7fe6f40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe6f27fc640 1 -- 172.21.15.107:0/4146014662 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f403f070 con 0x7fe70006a870 2024-05-09T08:49:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe6f27fc640 1 -- 172.21.15.107:0/4146014662 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6f403b460 con 0x7fe70006a870 2024-05-09T08:49:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe6f27fc640 1 -- 172.21.15.107:0/4146014662 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4035070 con 0x7fe70006a870 2024-05-09T08:49:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.660+0000 7fe7078b2640 1 -- 172.21.15.107:0/4146014662 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe70019d280 con 0x7fe70006a870 2024-05-09T08:49:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.661+0000 7fe7078b2640 1 -- 172.21.15.107:0/4146014662 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7001a4b70 con 0x7fe70006a870 2024-05-09T08:49:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.661+0000 7fe7078b2640 1 -- 172.21.15.107:0/4146014662 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe700068aa0 con 0x7fe70006a870 2024-05-09T08:49:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.663+0000 7fe6f27fc640 1 -- 172.21.15.107:0/4146014662 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe6f403b610 con 0x7fe70006a870 2024-05-09T08:49:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.663+0000 7fe6f27fc640 1 --2- 172.21.15.107:0/4146014662 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6ec0a2d80 0x7fe6ec0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.663+0000 7fe70564e640 1 --2- 172.21.15.107:0/4146014662 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6ec0a2d80 0x7fe6ec0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.664+0000 7fe6f27fc640 1 -- 172.21.15.107:0/4146014662 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9451+0+0 (secure 0 0 0) 0x7fe6f40efdc0 con 0x7fe70006a870 2024-05-09T08:49:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.664+0000 7fe70564e640 1 --2- 172.21.15.107:0/4146014662 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6ec0a2d80 0x7fe6ec0a5240 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc005d80 tx=0x7fe6fc005c90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.667+0000 7fe6f27fc640 1 -- 172.21.15.107:0/4146014662 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe6f40bc640 con 0x7fe70006a870 2024-05-09T08:49:33.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.853+0000 7fe7078b2640 1 -- 172.21.15.107:0/4146014662 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe700109db0 con 0x7fe70006a870 2024-05-09T08:49:33.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.853+0000 7fe6f27fc640 1 -- 172.21.15.107:0/4146014662 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v116) ==== 81+0+959 (secure 0 0 0) 0x7fe6f40bbf40 con 0x7fe70006a870 2024-05-09T08:49:33.853 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/2923766500 2024-05-10T08:33:20.828572+0000 2024-05-09T08:49:33.853 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/2417648229 2024-05-10T08:33:20.828572+0000 2024-05-09T08:49:33.853 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/1642826815 2024-05-10T08:33:20.828572+0000 2024-05-09T08:49:33.853 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/1064417772 2024-05-10T08:23:55.372959+0000 2024-05-09T08:49:33.853 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/3922375261 2024-05-10T08:23:55.372959+0000 2024-05-09T08:49:33.853 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/3922375261 2024-05-10T08:23:55.372959+0000 2024-05-09T08:49:33.854 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/3934964045 2024-05-10T08:23:55.372959+0000 2024-05-09T08:49:33.854 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/2689615560 2024-05-10T08:33:20.828572+0000 2024-05-09T08:49:33.854 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/1620469781 2024-05-10T08:23:29.066232+0000 2024-05-09T08:49:33.854 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4101303556 2024-05-10T08:23:29.066232+0000 2024-05-09T08:49:33.854 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/846948416 2024-05-10T08:33:20.828572+0000 2024-05-09T08:49:33.854 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/2360407889 2024-05-10T08:23:29.066232+0000 2024-05-09T08:49:33.854 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/846948416 2024-05-10T08:33:20.828572+0000 2024-05-09T08:49:33.854 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/164842293 2024-05-10T08:23:29.066232+0000 2024-05-09T08:49:33.854 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/2360407889 2024-05-10T08:23:29.066232+0000 2024-05-09T08:49:33.854 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4237993479 2024-05-10T08:23:55.372959+0000 2024-05-09T08:49:33.855 INFO:teuthology.orchestra.run.smithi107.stderr:listed 16 entries 2024-05-09T08:49:33.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.855+0000 7fe7078b2640 1 -- 172.21.15.107:0/4146014662 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6ec0a2d80 msgr2=0x7fe6ec0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:33.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.855+0000 7fe7078b2640 1 --2- 172.21.15.107:0/4146014662 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6ec0a2d80 0x7fe6ec0a5240 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc005d80 tx=0x7fe6fc005c90 comp rx=0 tx=0).stop 2024-05-09T08:49:33.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.855+0000 7fe7078b2640 1 -- 172.21.15.107:0/4146014662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe70006a870 msgr2=0x7fe700198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:33.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.855+0000 7fe7078b2640 1 --2- 172.21.15.107:0/4146014662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe70006a870 0x7fe700198540 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4004770 tx=0x7fe6f40042f0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.855+0000 7fe7078b2640 1 -- 172.21.15.107:0/4146014662 shutdown_connections 2024-05-09T08:49:33.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.855+0000 7fe7078b2640 1 --2- 172.21.15.107:0/4146014662 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fe6ec0a2d80 0x7fe6ec0a5240 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.855+0000 7fe7078b2640 1 --2- 172.21.15.107:0/4146014662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe700107820 0x7fe70019c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.855+0000 7fe7078b2640 1 --2- 172.21.15.107:0/4146014662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe70006a870 0x7fe700198540 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.855+0000 7fe7078b2640 1 --2- 172.21.15.107:0/4146014662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe700069eb0 0x7fe700198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:33.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.855+0000 7fe7078b2640 1 -- 172.21.15.107:0/4146014662 >> 172.21.15.107:0/4146014662 conn(0x7fe700077790 msgr2=0x7fe7000fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:33.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.855+0000 7fe7078b2640 1 -- 172.21.15.107:0/4146014662 shutdown_connections 2024-05-09T08:49:33.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:33.855+0000 7fe7078b2640 1 -- 172.21.15.107:0/4146014662 wait complete. 2024-05-09T08:49:33.865 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi107.front.sepia.ceph.com... 2024-05-09T08:49:33.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:49:33.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T08:49:33.925 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi107.stderr:ceph-fuse[81701]: fuse finished with error 0 and tester_r 0 2024-05-09T08:49:33.927 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T08:49:34.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:34 smithi107 ceph-mon[30122]: pgmap v511: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-05-09T08:49:34.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:34 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/86611327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:49:34.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:34 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3721830360' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:49:34.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:34 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/4146014662' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:49:34.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:34 smithi157 ceph-mon[33614]: pgmap v511: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-05-09T08:49:34.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:34 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/86611327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:49:34.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:34 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3721830360' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:49:34.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:34 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/4146014662' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:49:34.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:34 smithi110 ceph-mon[33622]: pgmap v511: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-05-09T08:49:34.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:34 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/86611327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T08:49:34.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:34 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3721830360' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T08:49:34.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:34 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/4146014662' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T08:49:36.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:36 smithi107 ceph-mon[30122]: pgmap v512: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:49:36.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:36 smithi157 ceph-mon[33614]: pgmap v512: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:49:36.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:36 smithi110 ceph-mon[33622]: pgmap v512: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T08:49:38.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:38 smithi107 ceph-mon[30122]: pgmap v513: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T08:49:38.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:38 smithi157 ceph-mon[33614]: pgmap v513: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T08:49:38.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:38 smithi110 ceph-mon[33622]: pgmap v513: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T08:49:40.028 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2024-05-09T08:49:40.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:49:40.029 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T08:49:40.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:49:40.049 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2024-05-09T08:49:40.122 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T08:49:40.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:49:40.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T08:49:40.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T08:49:40.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link delete ceph-brx 2024-05-09T08:49:40.292 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-09T08:49:40.302 INFO:tasks.cephadm:Teardown begin 2024-05-09T08:49:40.303 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T08:49:40.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:49:40.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.408+0000 7f31fdbf4640 1 -- 172.21.15.107:0/3849682040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f8106720 msgr2=0x7f31f8106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:40.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.408+0000 7f31fdbf4640 1 --2- 172.21.15.107:0/3849682040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f8106720 0x7f31f8106b00 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f31e8009930 tx=0x7f31e802a420 comp rx=0 tx=0).stop 2024-05-09T08:49:40.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.408+0000 7f31fdbf4640 1 -- 172.21.15.107:0/3849682040 shutdown_connections 2024-05-09T08:49:40.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.408+0000 7f31fdbf4640 1 --2- 172.21.15.107:0/3849682040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31f81010d0 0x7f31f810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.408+0000 7f31fdbf4640 1 --2- 172.21.15.107:0/3849682040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f31f8100710 0x7f31f8100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.408+0000 7f31fdbf4640 1 --2- 172.21.15.107:0/3849682040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f8106720 0x7f31f8106b00 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.408+0000 7f31fdbf4640 1 -- 172.21.15.107:0/3849682040 >> 172.21.15.107:0/3849682040 conn(0x7f31f8077830 msgr2=0x7f31f8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:40.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.408+0000 7f31fdbf4640 1 -- 172.21.15.107:0/3849682040 shutdown_connections 2024-05-09T08:49:40.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.408+0000 7f31fdbf4640 1 -- 172.21.15.107:0/3849682040 wait complete. 2024-05-09T08:49:40.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31fdbf4640 1 Processor -- start 2024-05-09T08:49:40.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31fdbf4640 1 -- start start 2024-05-09T08:49:40.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31fdbf4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f8100710 0x7f31f8198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:40.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31fdbf4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31f81010d0 0x7f31f81993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:40.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31fdbf4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f31f8106720 0x7f31f81a3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:40.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31fdbf4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f31f81109e0 con 0x7f31f8100710 2024-05-09T08:49:40.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31fdbf4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f31f8110860 con 0x7f31f81010d0 2024-05-09T08:49:40.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31fdbf4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f31f8110b60 con 0x7f31f8106720 2024-05-09T08:49:40.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31f77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f8100710 0x7f31f8198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:40.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31f77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f8100710 0x7f31f8198ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36812/0 (socket says 172.21.15.107:36812) 2024-05-09T08:49:40.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31f77fe640 1 -- 172.21.15.107:0/1870402956 learned_addr learned my addr 172.21.15.107:0/1870402956 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:40.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31f7fff640 1 --2- 172.21.15.107:0/1870402956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f31f8106720 0x7f31f81a3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:40.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31f6ffd640 1 --2- 172.21.15.107:0/1870402956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31f81010d0 0x7f31f81993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:40.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31f77fe640 1 -- 172.21.15.107:0/1870402956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f31f8106720 msgr2=0x7f31f81a3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:40.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31f77fe640 1 --2- 172.21.15.107:0/1870402956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f31f8106720 0x7f31f81a3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31f77fe640 1 -- 172.21.15.107:0/1870402956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31f81010d0 msgr2=0x7f31f81993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:40.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31f77fe640 1 --2- 172.21.15.107:0/1870402956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31f81010d0 0x7f31f81993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31f77fe640 1 -- 172.21.15.107:0/1870402956 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f81a4460 con 0x7f31f8100710 2024-05-09T08:49:40.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.409+0000 7f31f6ffd640 1 --2- 172.21.15.107:0/1870402956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31f81010d0 0x7f31f81993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T08:49:40.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.410+0000 7f31f77fe640 1 --2- 172.21.15.107:0/1870402956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f8100710 0x7f31f8198ea0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f31e8005940 tx=0x7f31e80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:40.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.410+0000 7f31f4ff9640 1 -- 172.21.15.107:0/1870402956 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e803f030 con 0x7f31f8100710 2024-05-09T08:49:40.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.410+0000 7f31fdbf4640 1 -- 172.21.15.107:0/1870402956 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31f81a4630 con 0x7f31f8100710 2024-05-09T08:49:40.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.410+0000 7f31f4ff9640 1 -- 172.21.15.107:0/1870402956 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f31e8003cb0 con 0x7f31f8100710 2024-05-09T08:49:40.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.410+0000 7f31f4ff9640 1 -- 172.21.15.107:0/1870402956 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e8035070 con 0x7f31f8100710 2024-05-09T08:49:40.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.410+0000 7f31fdbf4640 1 -- 172.21.15.107:0/1870402956 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31f81a4af0 con 0x7f31f8100710 2024-05-09T08:49:40.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.411+0000 7f31fdbf4640 1 -- 172.21.15.107:0/1870402956 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31f8068650 con 0x7f31f8100710 2024-05-09T08:49:40.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.412+0000 7f31f4ff9640 1 -- 172.21.15.107:0/1870402956 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f31e8042030 con 0x7f31f8100710 2024-05-09T08:49:40.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.413+0000 7f31f4ff9640 1 --2- 172.21.15.107:0/1870402956 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f31e00a2d00 0x7f31e00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:40.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.413+0000 7f31f6ffd640 1 --2- 172.21.15.107:0/1870402956 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f31e00a2d00 0x7f31e00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:40.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.413+0000 7f31f4ff9640 1 -- 172.21.15.107:0/1870402956 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9451+0+0 (secure 0 0 0) 0x7f31e80f0050 con 0x7f31f8100710 2024-05-09T08:49:40.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.413+0000 7f31f6ffd640 1 --2- 172.21.15.107:0/1870402956 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f31e00a2d00 0x7f31e00a51c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f31ec006fd0 tx=0x7f31ec008040 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:40.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.416+0000 7f31f4ff9640 1 -- 172.21.15.107:0/1870402956 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f31e80bc8d0 con 0x7f31f8100710 2024-05-09T08:49:40.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:40 smithi107 ceph-mon[30122]: pgmap v514: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-09T08:49:40.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:40 smithi157 ceph-mon[33614]: pgmap v514: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-09T08:49:40.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:40 smithi110 ceph-mon[33622]: pgmap v514: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-09T08:49:40.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.634+0000 7f31fdbf4640 1 -- 172.21.15.107:0/1870402956 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31f8108c80 con 0x7f31f8100710 2024-05-09T08:49:40.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.634+0000 7f31f4ff9640 1 -- 172.21.15.107:0/1870402956 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11893 (secure 0 0 0) 0x7f31e80bc1d0 con 0x7f31f8100710 2024-05-09T08:49:40.634 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:49:40.634 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":28109,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/2116846059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2116846059},{"type":"v1","addr":"172.21.15.157:6835","nonce":2116846059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:48:10.313321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24418":{"gid":24418,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:49:40.634 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 31 2024-05-09T08:49:40.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.637+0000 7f31fdbf4640 1 -- 172.21.15.107:0/1870402956 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f31e00a2d00 msgr2=0x7f31e00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:40.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.637+0000 7f31fdbf4640 1 --2- 172.21.15.107:0/1870402956 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f31e00a2d00 0x7f31e00a51c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f31ec006fd0 tx=0x7f31ec008040 comp rx=0 tx=0).stop 2024-05-09T08:49:40.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.637+0000 7f31fdbf4640 1 -- 172.21.15.107:0/1870402956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f8100710 msgr2=0x7f31f8198ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:40.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.637+0000 7f31fdbf4640 1 --2- 172.21.15.107:0/1870402956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f8100710 0x7f31f8198ea0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f31e8005940 tx=0x7f31e80042f0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.637+0000 7f31fdbf4640 1 -- 172.21.15.107:0/1870402956 shutdown_connections 2024-05-09T08:49:40.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.637+0000 7f31fdbf4640 1 --2- 172.21.15.107:0/1870402956 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f31e00a2d00 0x7f31e00a51c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.637+0000 7f31fdbf4640 1 --2- 172.21.15.107:0/1870402956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f31f8106720 0x7f31f81a3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.637+0000 7f31fdbf4640 1 --2- 172.21.15.107:0/1870402956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31f81010d0 0x7f31f81993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.637+0000 7f31fdbf4640 1 --2- 172.21.15.107:0/1870402956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f8100710 0x7f31f8198ea0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.637+0000 7f31fdbf4640 1 -- 172.21.15.107:0/1870402956 >> 172.21.15.107:0/1870402956 conn(0x7f31f8077830 msgr2=0x7f31f81030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:40.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.637+0000 7f31fdbf4640 1 -- 172.21.15.107:0/1870402956 shutdown_connections 2024-05-09T08:49:40.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.637+0000 7f31fdbf4640 1 -- 172.21.15.107:0/1870402956 wait complete. 2024-05-09T08:49:40.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T08:49:40.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.750+0000 7f9be65ce640 1 -- 172.21.15.107:0/108715615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9be0100720 msgr2=0x7f9be010e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:40.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.750+0000 7f9be65ce640 1 --2- 172.21.15.107:0/108715615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9be0100720 0x7f9be010e770 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009cd0 tx=0x7f9bd802a8e0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.750+0000 7f9be65ce640 1 -- 172.21.15.107:0/108715615 shutdown_connections 2024-05-09T08:49:40.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.750+0000 7f9be65ce640 1 --2- 172.21.15.107:0/108715615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9be0100720 0x7f9be010e770 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.750+0000 7f9be65ce640 1 --2- 172.21.15.107:0/108715615 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9be00ffda0 0x7f9be01001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.750+0000 7f9be65ce640 1 --2- 172.21.15.107:0/108715615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9be01079b0 0x7f9be0107d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.750+0000 7f9be65ce640 1 -- 172.21.15.107:0/108715615 >> 172.21.15.107:0/108715615 conn(0x7f9be0076d60 msgr2=0x7f9be00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:40.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.750+0000 7f9be65ce640 1 -- 172.21.15.107:0/108715615 shutdown_connections 2024-05-09T08:49:40.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.750+0000 7f9be65ce640 1 -- 172.21.15.107:0/108715615 wait complete. 2024-05-09T08:49:40.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9be65ce640 1 Processor -- start 2024-05-09T08:49:40.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9be65ce640 1 -- start start 2024-05-09T08:49:40.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9be65ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9be00ffda0 0x7f9be0198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:40.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9be65ce640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9be0100720 0x7f9be01986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:40.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9be65ce640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9be01079b0 0x7f9be019ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:40.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9be65ce640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9be0111bd0 con 0x7f9be00ffda0 2024-05-09T08:49:40.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9be65ce640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9be0111a50 con 0x7f9be01079b0 2024-05-09T08:49:40.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9be65ce640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9be0111d50 con 0x7f9be0100720 2024-05-09T08:49:40.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9bdffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9be00ffda0 0x7f9be0198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:40.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9bdffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9be00ffda0 0x7f9be0198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44454/0 (socket says 172.21.15.107:44454) 2024-05-09T08:49:40.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9bdffff640 1 -- 172.21.15.107:0/3744466232 learned_addr learned my addr 172.21.15.107:0/3744466232 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:40.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9bdffff640 1 -- 172.21.15.107:0/3744466232 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9be0100720 msgr2=0x7f9be01986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:40.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9bdffff640 1 --2- 172.21.15.107:0/3744466232 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9be0100720 0x7f9be01986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9bdffff640 1 -- 172.21.15.107:0/3744466232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9be01079b0 msgr2=0x7f9be019ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:40.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9bdffff640 1 --2- 172.21.15.107:0/3744466232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9be01079b0 0x7f9be019ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9bdffff640 1 -- 172.21.15.107:0/3744466232 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be019d160 con 0x7f9be00ffda0 2024-05-09T08:49:40.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9bdffff640 1 --2- 172.21.15.107:0/3744466232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9be00ffda0 0x7f9be0198170 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f9bd000d950 tx=0x7f9bd000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:40.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.751+0000 7f9bddffb640 1 -- 172.21.15.107:0/3744466232 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0017070 con 0x7f9be00ffda0 2024-05-09T08:49:40.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.752+0000 7f9bddffb640 1 -- 172.21.15.107:0/3744466232 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bd0009c30 con 0x7f9be00ffda0 2024-05-09T08:49:40.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.752+0000 7f9be65ce640 1 -- 172.21.15.107:0/3744466232 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9be019d450 con 0x7f9be00ffda0 2024-05-09T08:49:40.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.752+0000 7f9bddffb640 1 -- 172.21.15.107:0/3744466232 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd000f620 con 0x7f9be00ffda0 2024-05-09T08:49:40.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.752+0000 7f9be65ce640 1 -- 172.21.15.107:0/3744466232 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9be01a4ce0 con 0x7f9be00ffda0 2024-05-09T08:49:40.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.752+0000 7f9be65ce640 1 -- 172.21.15.107:0/3744466232 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9be0068aa0 con 0x7f9be00ffda0 2024-05-09T08:49:40.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.754+0000 7f9bddffb640 1 -- 172.21.15.107:0/3744466232 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f9bd00109a0 con 0x7f9be00ffda0 2024-05-09T08:49:40.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.754+0000 7f9bddffb640 1 --2- 172.21.15.107:0/3744466232 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9bcc0a2d80 0x7f9bcc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:40.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.754+0000 7f9bd7fff640 1 --2- 172.21.15.107:0/3744466232 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9bcc0a2d80 0x7f9bcc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:40.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.754+0000 7f9bddffb640 1 -- 172.21.15.107:0/3744466232 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9451+0+0 (secure 0 0 0) 0x7f9bd00cf460 con 0x7f9be00ffda0 2024-05-09T08:49:40.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.755+0000 7f9bd7fff640 1 --2- 172.21.15.107:0/3744466232 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9bcc0a2d80 0x7f9bcc0a5240 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9be00ff430 tx=0x7f9bc8006c90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:40.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.759+0000 7f9bddffb640 1 -- 172.21.15.107:0/3744466232 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9bd0009de0 con 0x7f9be00ffda0 2024-05-09T08:49:40.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.969+0000 7f9be65ce640 1 -- 172.21.15.107:0/3744466232 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9be0109e20 con 0x7f9be00ffda0 2024-05-09T08:49:40.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.970+0000 7f9bddffb640 1 -- 172.21.15.107:0/3744466232 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11893 (secure 0 0 0) 0x7f9bd009bce0 con 0x7f9be00ffda0 2024-05-09T08:49:40.969 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:49:40.969 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3401311866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3401311866},{"type":"v1","addr":"172.21.15.107:6835","nonce":3401311866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4170346807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4170346807},{"type":"v1","addr":"172.21.15.107:6837","nonce":4170346807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2088211251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2088211251},{"type":"v1","addr":"172.21.15.107:6839","nonce":2088211251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/2463167549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":2463167549},{"type":"v1","addr":"172.21.15.107:6841","nonce":2463167549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/844626352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":844626352},{"type":"v1","addr":"172.21.15.110:6833","nonce":844626352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":28109,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/2116846059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2116846059},{"type":"v1","addr":"172.21.15.157:6835","nonce":2116846059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T08:31:00.157448+0000","modified":"2024-05-09T08:48:10.313321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24491,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24418":{"gid":24418,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6833/2435727962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":2435727962},{"type":"v1","addr":"172.21.15.157:6833","nonce":2435727962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/2138165106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2138165106},{"type":"v1","addr":"172.21.15.110:6835","nonce":2138165106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6837/624105719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":624105719},{"type":"v1","addr":"172.21.15.110:6837","nonce":624105719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.157:6837/1973902800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1973902800},{"type":"v1","addr":"172.21.15.157:6837","nonce":1973902800}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6839/593143450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":593143450},{"type":"v1","addr":"172.21.15.110:6839","nonce":593143450}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6839/3003356919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3003356919},{"type":"v1","addr":"172.21.15.157:6839","nonce":3003356919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24491,24499,24478]},"id":1}]} 2024-05-09T08:49:40.969 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 31 2024-05-09T08:49:40.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.972+0000 7f9be65ce640 1 -- 172.21.15.107:0/3744466232 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9bcc0a2d80 msgr2=0x7f9bcc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:40.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.972+0000 7f9be65ce640 1 --2- 172.21.15.107:0/3744466232 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9bcc0a2d80 0x7f9bcc0a5240 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9be00ff430 tx=0x7f9bc8006c90 comp rx=0 tx=0).stop 2024-05-09T08:49:40.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.972+0000 7f9be65ce640 1 -- 172.21.15.107:0/3744466232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9be00ffda0 msgr2=0x7f9be0198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:40.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.972+0000 7f9be65ce640 1 --2- 172.21.15.107:0/3744466232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9be00ffda0 0x7f9be0198170 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f9bd000d950 tx=0x7f9bd000de10 comp rx=0 tx=0).stop 2024-05-09T08:49:40.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.972+0000 7f9be65ce640 1 -- 172.21.15.107:0/3744466232 shutdown_connections 2024-05-09T08:49:40.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.972+0000 7f9be65ce640 1 --2- 172.21.15.107:0/3744466232 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f9bcc0a2d80 0x7f9bcc0a5240 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.972+0000 7f9be65ce640 1 --2- 172.21.15.107:0/3744466232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9be01079b0 0x7f9be019ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.972+0000 7f9be65ce640 1 --2- 172.21.15.107:0/3744466232 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9be0100720 0x7f9be01986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.972+0000 7f9be65ce640 1 --2- 172.21.15.107:0/3744466232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9be00ffda0 0x7f9be0198170 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:40.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.972+0000 7f9be65ce640 1 -- 172.21.15.107:0/3744466232 >> 172.21.15.107:0/3744466232 conn(0x7f9be0076d60 msgr2=0x7f9be0104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:40.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.972+0000 7f9be65ce640 1 -- 172.21.15.107:0/3744466232 shutdown_connections 2024-05-09T08:49:40.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:40.972+0000 7f9be65ce640 1 -- 172.21.15.107:0/3744466232 wait complete. 2024-05-09T08:49:40.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T08:49:41.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.085+0000 7fb92bb63640 1 -- 172.21.15.107:0/2827180609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9241010d0 msgr2=0x7fb92410d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.085+0000 7fb92bb63640 1 --2- 172.21.15.107:0/2827180609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9241010d0 0x7fb92410d5e0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fb91c009cd0 tx=0x7fb91c02a8e0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.085+0000 7fb92bb63640 1 -- 172.21.15.107:0/2827180609 shutdown_connections 2024-05-09T08:49:41.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.085+0000 7fb92bb63640 1 --2- 172.21.15.107:0/2827180609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9241010d0 0x7fb92410d5e0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.085+0000 7fb92bb63640 1 --2- 172.21.15.107:0/2827180609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb924100710 0x7fb924100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.085+0000 7fb92bb63640 1 --2- 172.21.15.107:0/2827180609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb924106720 0x7fb924106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.085+0000 7fb92bb63640 1 -- 172.21.15.107:0/2827180609 >> 172.21.15.107:0/2827180609 conn(0x7fb924077830 msgr2=0x7fb924077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:41.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.086+0000 7fb92bb63640 1 -- 172.21.15.107:0/2827180609 shutdown_connections 2024-05-09T08:49:41.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.086+0000 7fb92bb63640 1 -- 172.21.15.107:0/2827180609 wait complete. 2024-05-09T08:49:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.086+0000 7fb92bb63640 1 Processor -- start 2024-05-09T08:49:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.086+0000 7fb92bb63640 1 -- start start 2024-05-09T08:49:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.086+0000 7fb92bb63640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb924100710 0x7fb924197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.086+0000 7fb92bb63640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9241010d0 0x7fb924198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.086+0000 7fb92bb63640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb924106720 0x7fb92419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.086+0000 7fb92bb63640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb9241109e0 con 0x7fb924106720 2024-05-09T08:49:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.086+0000 7fb92bb63640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb924110860 con 0x7fb9241010d0 2024-05-09T08:49:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.086+0000 7fb92bb63640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb924110b60 con 0x7fb924100710 2024-05-09T08:49:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.086+0000 7fb92a100640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb924106720 0x7fb92419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.087+0000 7fb92a100640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb924106720 0x7fb92419c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44464/0 (socket says 172.21.15.107:44464) 2024-05-09T08:49:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.087+0000 7fb92a100640 1 -- 172.21.15.107:0/847207616 learned_addr learned my addr 172.21.15.107:0/847207616 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.087+0000 7fb9298ff640 1 --2- 172.21.15.107:0/847207616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb924100710 0x7fb924197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.087+0000 7fb9290fe640 1 --2- 172.21.15.107:0/847207616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9241010d0 0x7fb924198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.087+0000 7fb92a100640 1 -- 172.21.15.107:0/847207616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb924100710 msgr2=0x7fb924197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.087+0000 7fb92a100640 1 --2- 172.21.15.107:0/847207616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb924100710 0x7fb924197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.087+0000 7fb92a100640 1 -- 172.21.15.107:0/847207616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9241010d0 msgr2=0x7fb924198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.087+0000 7fb92a100640 1 --2- 172.21.15.107:0/847207616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9241010d0 0x7fb924198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.087+0000 7fb92a100640 1 -- 172.21.15.107:0/847207616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92419cf30 con 0x7fb924106720 2024-05-09T08:49:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.087+0000 7fb92a100640 1 --2- 172.21.15.107:0/847207616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb924106720 0x7fb92419c800 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fb91c033040 tx=0x7fb91c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.087+0000 7fb91affd640 1 -- 172.21.15.107:0/847207616 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91c004040 con 0x7fb924106720 2024-05-09T08:49:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.087+0000 7fb91affd640 1 -- 172.21.15.107:0/847207616 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb91c004690 con 0x7fb924106720 2024-05-09T08:49:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.087+0000 7fb91affd640 1 -- 172.21.15.107:0/847207616 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91c030d80 con 0x7fb924106720 2024-05-09T08:49:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.087+0000 7fb92bb63640 1 -- 172.21.15.107:0/847207616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb92419d1c0 con 0x7fb924106720 2024-05-09T08:49:41.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.087+0000 7fb92bb63640 1 -- 172.21.15.107:0/847207616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9241a4ab0 con 0x7fb924106720 2024-05-09T08:49:41.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.088+0000 7fb92bb63640 1 -- 172.21.15.107:0/847207616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb924068650 con 0x7fb924106720 2024-05-09T08:49:41.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.089+0000 7fb91affd640 1 -- 172.21.15.107:0/847207616 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb91c0041e0 con 0x7fb924106720 2024-05-09T08:49:41.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.090+0000 7fb91affd640 1 --2- 172.21.15.107:0/847207616 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb9100a2d80 0x7fb9100a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:41.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.090+0000 7fb9298ff640 1 --2- 172.21.15.107:0/847207616 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb9100a2d80 0x7fb9100a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:41.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.090+0000 7fb91affd640 1 -- 172.21.15.107:0/847207616 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9451+0+0 (secure 0 0 0) 0x7fb91c02e030 con 0x7fb924106720 2024-05-09T08:49:41.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.090+0000 7fb9298ff640 1 --2- 172.21.15.107:0/847207616 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb9100a2d80 0x7fb9100a5240 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb92000ab60 tx=0x7fb92000a2d0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:41.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.094+0000 7fb91affd640 1 -- 172.21.15.107:0/847207616 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb91c0bd010 con 0x7fb924106720 2024-05-09T08:49:41.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.276+0000 7fb92bb63640 1 -- 172.21.15.107:0/847207616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb924109720 con 0x7fb924106720 2024-05-09T08:49:41.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.277+0000 7fb91affd640 1 -- 172.21.15.107:0/847207616 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) ==== 74+0+25307 (secure 0 0 0) 0x7fb91c0bc910 con 0x7fb924106720 2024-05-09T08:49:41.277 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-05-09T08:49:41.277 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":116,"fsid":"3954c88c-0ddd-11ef-bc98-c7b262605968","created":"2024-05-09T08:22:57.028352+0000","modified":"2024-05-09T08:49:22.053969+0000","last_up_change":"2024-05-09T08:30:09.954012+0000","last_in_change":"2024-05-09T08:29:49.849873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T08:26:21.698338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T08:30:58.268633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T08:30:59.345579+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T08:31:03.065991+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8ab0acea-9500-4d88-ab8e-2d148374937b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6803","nonce":1215261349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6805","nonce":1215261349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6809","nonce":1215261349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1215261349},{"type":"v1","addr":"172.21.15.107:6807","nonce":1215261349}]},"public_addr":"172.21.15.107:6803/1215261349","cluster_addr":"172.21.15.107:6805/1215261349","heartbeat_back_addr":"172.21.15.107:6809/1215261349","heartbeat_front_addr":"172.21.15.107:6807/1215261349","state":["exists","up"]},{"osd":1,"uuid":"93653939-b195-46f0-a8a9-b25ad2af5464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6801","nonce":370054671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6803","nonce":370054671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6807","nonce":370054671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":370054671},{"type":"v1","addr":"172.21.15.110:6805","nonce":370054671}]},"public_addr":"172.21.15.110:6801/370054671","cluster_addr":"172.21.15.110:6803/370054671","heartbeat_back_addr":"172.21.15.110:6807/370054671","heartbeat_front_addr":"172.21.15.110:6805/370054671","state":["exists","up"]},{"osd":2,"uuid":"dc55ab61-bfc9-48ad-92b3-37ad560a0023","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6801","nonce":3005990647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6803","nonce":3005990647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6807","nonce":3005990647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3005990647},{"type":"v1","addr":"172.21.15.157:6805","nonce":3005990647}]},"public_addr":"172.21.15.157:6801/3005990647","cluster_addr":"172.21.15.157:6803/3005990647","heartbeat_back_addr":"172.21.15.157:6807/3005990647","heartbeat_front_addr":"172.21.15.157:6805/3005990647","state":["exists","up"]},{"osd":3,"uuid":"8b4ef7e0-8679-44c3-ac27-170d0ab391aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6811","nonce":1056009754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6813","nonce":1056009754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6817","nonce":1056009754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1056009754},{"type":"v1","addr":"172.21.15.107:6815","nonce":1056009754}]},"public_addr":"172.21.15.107:6811/1056009754","cluster_addr":"172.21.15.107:6813/1056009754","heartbeat_back_addr":"172.21.15.107:6817/1056009754","heartbeat_front_addr":"172.21.15.107:6815/1056009754","state":["exists","up"]},{"osd":4,"uuid":"b67c4b46-7857-48cd-8737-d7cbb37aabaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6809","nonce":2317455505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6811","nonce":2317455505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6815","nonce":2317455505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2317455505},{"type":"v1","addr":"172.21.15.110:6813","nonce":2317455505}]},"public_addr":"172.21.15.110:6809/2317455505","cluster_addr":"172.21.15.110:6811/2317455505","heartbeat_back_addr":"172.21.15.110:6815/2317455505","heartbeat_front_addr":"172.21.15.110:6813/2317455505","state":["exists","up"]},{"osd":5,"uuid":"2f4cfd0f-4faa-4697-aa3b-ef48a801be37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6809","nonce":423586420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6811","nonce":423586420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6815","nonce":423586420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":423586420},{"type":"v1","addr":"172.21.15.157:6813","nonce":423586420}]},"public_addr":"172.21.15.157:6809/423586420","cluster_addr":"172.21.15.157:6811/423586420","heartbeat_back_addr":"172.21.15.157:6815/423586420","heartbeat_front_addr":"172.21.15.157:6813/423586420","state":["exists","up"]},{"osd":6,"uuid":"c0451766-461b-4d7e-8087-f2d066097c16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6819","nonce":3011505372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6821","nonce":3011505372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6825","nonce":3011505372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3011505372},{"type":"v1","addr":"172.21.15.107:6823","nonce":3011505372}]},"public_addr":"172.21.15.107:6819/3011505372","cluster_addr":"172.21.15.107:6821/3011505372","heartbeat_back_addr":"172.21.15.107:6825/3011505372","heartbeat_front_addr":"172.21.15.107:6823/3011505372","state":["exists","up"]},{"osd":7,"uuid":"b7258141-aef2-4d46-9825-8c51d817f7a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6817","nonce":204549355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6819","nonce":204549355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6823","nonce":204549355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":204549355},{"type":"v1","addr":"172.21.15.110:6821","nonce":204549355}]},"public_addr":"172.21.15.110:6817/204549355","cluster_addr":"172.21.15.110:6819/204549355","heartbeat_back_addr":"172.21.15.110:6823/204549355","heartbeat_front_addr":"172.21.15.110:6821/204549355","state":["exists","up"]},{"osd":8,"uuid":"bcc1105e-25be-4d89-bb7b-920287c3f6cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6817","nonce":4115873287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6819","nonce":4115873287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6823","nonce":4115873287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4115873287},{"type":"v1","addr":"172.21.15.157:6821","nonce":4115873287}]},"public_addr":"172.21.15.157:6817/4115873287","cluster_addr":"172.21.15.157:6819/4115873287","heartbeat_back_addr":"172.21.15.157:6823/4115873287","heartbeat_front_addr":"172.21.15.157:6821/4115873287","state":["exists","up"]},{"osd":9,"uuid":"56600856-ce7c-47d1-aff0-a57cf7ce75e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6827","nonce":2273353916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6829","nonce":2273353916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6833","nonce":2273353916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":2273353916},{"type":"v1","addr":"172.21.15.107:6831","nonce":2273353916}]},"public_addr":"172.21.15.107:6827/2273353916","cluster_addr":"172.21.15.107:6829/2273353916","heartbeat_back_addr":"172.21.15.107:6833/2273353916","heartbeat_front_addr":"172.21.15.107:6831/2273353916","state":["exists","up"]},{"osd":10,"uuid":"f4022636-ba24-4685-9678-d261e04df620","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6825","nonce":1566970764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6827","nonce":1566970764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6831","nonce":1566970764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":1566970764},{"type":"v1","addr":"172.21.15.110:6829","nonce":1566970764}]},"public_addr":"172.21.15.110:6825/1566970764","cluster_addr":"172.21.15.110:6827/1566970764","heartbeat_back_addr":"172.21.15.110:6831/1566970764","heartbeat_front_addr":"172.21.15.110:6829/1566970764","state":["exists","up"]},{"osd":11,"uuid":"fef75204-3f64-4365-8ece-384eb4598b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6825","nonce":3347498474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6827","nonce":3347498474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6831","nonce":3347498474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":3347498474},{"type":"v1","addr":"172.21.15.157:6829","nonce":3347498474}]},"public_addr":"172.21.15.157:6825/3347498474","cluster_addr":"172.21.15.157:6827/3347498474","heartbeat_back_addr":"172.21.15.157:6831/3347498474","heartbeat_front_addr":"172.21.15.157:6829/3347498474","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:30.470101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:25:53.823434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:18.305231+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:26:43.987491+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:08.774682+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:33.468693+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:27:59.342091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:24.857974+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:28:50.248409+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:16.528825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:29:42.364614+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T08:30:08.500826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/2923766500":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/2417648229":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1642826815":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1064417772":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6801/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:6800/3922375261":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/3934964045":"2024-05-10T08:23:55.372959+0000","172.21.15.107:0/2689615560":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/1620469781":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4101303556":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/846948416":"2024-05-10T08:33:20.828572+0000","172.21.15.107:6800/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6800/846948416":"2024-05-10T08:33:20.828572+0000","172.21.15.107:0/164842293":"2024-05-10T08:23:29.066232+0000","172.21.15.107:6801/2360407889":"2024-05-10T08:23:29.066232+0000","172.21.15.107:0/4237993479":"2024-05-10T08:23:55.372959+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":27,"length":1}]},{"pool":4,"snaps":[{"begin":14,"length":1},{"begin":27,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T08:49:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.279+0000 7fb92bb63640 1 -- 172.21.15.107:0/847207616 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb9100a2d80 msgr2=0x7fb9100a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.279+0000 7fb92bb63640 1 --2- 172.21.15.107:0/847207616 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb9100a2d80 0x7fb9100a5240 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb92000ab60 tx=0x7fb92000a2d0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.279+0000 7fb92bb63640 1 -- 172.21.15.107:0/847207616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb924106720 msgr2=0x7fb92419c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.279+0000 7fb92bb63640 1 --2- 172.21.15.107:0/847207616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb924106720 0x7fb92419c800 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fb91c033040 tx=0x7fb91c003970 comp rx=0 tx=0).stop 2024-05-09T08:49:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.279+0000 7fb92bb63640 1 -- 172.21.15.107:0/847207616 shutdown_connections 2024-05-09T08:49:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.279+0000 7fb92bb63640 1 --2- 172.21.15.107:0/847207616 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fb9100a2d80 0x7fb9100a5240 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.279+0000 7fb92bb63640 1 --2- 172.21.15.107:0/847207616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb924106720 0x7fb92419c800 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.279+0000 7fb92bb63640 1 --2- 172.21.15.107:0/847207616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9241010d0 0x7fb924198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.279+0000 7fb92bb63640 1 --2- 172.21.15.107:0/847207616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb924100710 0x7fb924197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.279+0000 7fb92bb63640 1 -- 172.21.15.107:0/847207616 >> 172.21.15.107:0/847207616 conn(0x7fb924077830 msgr2=0x7fb924068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:41.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.279+0000 7fb92bb63640 1 -- 172.21.15.107:0/847207616 shutdown_connections 2024-05-09T08:49:41.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.279+0000 7fb92bb63640 1 -- 172.21.15.107:0/847207616 wait complete. 2024-05-09T08:49:41.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:41 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1870402956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:49:41.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:41 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3744466232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:49:41.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T08:49:41.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.394+0000 7fbdd0ddc640 1 -- 172.21.15.107:0/53701796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdcc1010d0 msgr2=0x7fbdcc10d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.394+0000 7fbdd0ddc640 1 --2- 172.21.15.107:0/53701796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdcc1010d0 0x7fbdcc10d5e0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4009cd0 tx=0x7fbdc402a8e0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.394+0000 7fbdd0ddc640 1 -- 172.21.15.107:0/53701796 shutdown_connections 2024-05-09T08:49:41.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.394+0000 7fbdd0ddc640 1 --2- 172.21.15.107:0/53701796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdcc1010d0 0x7fbdcc10d5e0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.394+0000 7fbdd0ddc640 1 --2- 172.21.15.107:0/53701796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbdcc100710 0x7fbdcc100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.394+0000 7fbdd0ddc640 1 --2- 172.21.15.107:0/53701796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbdcc106720 0x7fbdcc106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.394+0000 7fbdd0ddc640 1 -- 172.21.15.107:0/53701796 >> 172.21.15.107:0/53701796 conn(0x7fbdcc077790 msgr2=0x7fbdcc077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:41.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.394+0000 7fbdd0ddc640 1 -- 172.21.15.107:0/53701796 shutdown_connections 2024-05-09T08:49:41.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.394+0000 7fbdd0ddc640 1 -- 172.21.15.107:0/53701796 wait complete. 2024-05-09T08:49:41.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.394+0000 7fbdd0ddc640 1 Processor -- start 2024-05-09T08:49:41.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.394+0000 7fbdd0ddc640 1 -- start start 2024-05-09T08:49:41.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdd0ddc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbdcc100710 0x7fbdcc197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:41.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdd0ddc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbdcc1010d0 0x7fbdcc1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:41.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdd0ddc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdcc106720 0x7fbdcc19c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:41.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdd0ddc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbdcc111ae0 con 0x7fbdcc106720 2024-05-09T08:49:41.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdd0ddc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbdcc111960 con 0x7fbdcc100710 2024-05-09T08:49:41.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdd0ddc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbdcc111c60 con 0x7fbdcc1010d0 2024-05-09T08:49:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdcad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdcc106720 0x7fbdcc19c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdcad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdcc106720 0x7fbdcc19c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44486/0 (socket says 172.21.15.107:44486) 2024-05-09T08:49:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdcad9d640 1 -- 172.21.15.107:0/2342916120 learned_addr learned my addr 172.21.15.107:0/2342916120 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdca59c640 1 --2- 172.21.15.107:0/2342916120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbdcc100710 0x7fbdcc197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdcad9d640 1 -- 172.21.15.107:0/2342916120 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbdcc1010d0 msgr2=0x7fbdcc1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdcad9d640 1 --2- 172.21.15.107:0/2342916120 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbdcc1010d0 0x7fbdcc1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdcad9d640 1 -- 172.21.15.107:0/2342916120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbdcc100710 msgr2=0x7fbdcc197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdcad9d640 1 --2- 172.21.15.107:0/2342916120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbdcc100710 0x7fbdcc197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdcad9d640 1 -- 172.21.15.107:0/2342916120 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdcc19cf20 con 0x7fbdcc106720 2024-05-09T08:49:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdcad9d640 1 --2- 172.21.15.107:0/2342916120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdcc106720 0x7fbdcc19c7f0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4005cc0 tx=0x7fbdc4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdc37fe640 1 -- 172.21.15.107:0/2342916120 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4004240 con 0x7fbdcc106720 2024-05-09T08:49:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdd0ddc640 1 -- 172.21.15.107:0/2342916120 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdcc19d1b0 con 0x7fbdcc106720 2024-05-09T08:49:41.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdc37fe640 1 -- 172.21.15.107:0/2342916120 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbdc40305a0 con 0x7fbdcc106720 2024-05-09T08:49:41.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdc37fe640 1 -- 172.21.15.107:0/2342916120 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc40466c0 con 0x7fbdcc106720 2024-05-09T08:49:41.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.395+0000 7fbdd0ddc640 1 -- 172.21.15.107:0/2342916120 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdcc1a4aa0 con 0x7fbdcc106720 2024-05-09T08:49:41.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.396+0000 7fbdd0ddc640 1 -- 172.21.15.107:0/2342916120 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdcc068aa0 con 0x7fbdcc106720 2024-05-09T08:49:41.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.397+0000 7fbdc37fe640 1 -- 172.21.15.107:0/2342916120 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fbdc4030ac0 con 0x7fbdcc106720 2024-05-09T08:49:41.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.398+0000 7fbdc37fe640 1 --2- 172.21.15.107:0/2342916120 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbdb80a2d00 0x7fbdb80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:41.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.398+0000 7fbdca59c640 1 --2- 172.21.15.107:0/2342916120 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbdb80a2d00 0x7fbdb80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:41.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.398+0000 7fbdc37fe640 1 -- 172.21.15.107:0/2342916120 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9451+0+0 (secure 0 0 0) 0x7fbdc402e030 con 0x7fbdcc106720 2024-05-09T08:49:41.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.399+0000 7fbdca59c640 1 --2- 172.21.15.107:0/2342916120 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbdb80a2d00 0x7fbdb80a51c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc00b9c0 tx=0x7fbdbc005c50 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:41.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.402+0000 7fbdc37fe640 1 -- 172.21.15.107:0/2342916120 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbdc404b060 con 0x7fbdcc106720 2024-05-09T08:49:41.593 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:41 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1870402956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:49:41.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:41 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3744466232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:49:41.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.617+0000 7fbdd0ddc640 1 -- 172.21.15.107:0/2342916120 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fbdcc02d1c0 con 0x7fbdcc106720 2024-05-09T08:49:41.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.619+0000 7fbdc37fe640 1 -- 172.21.15.107:0/2342916120 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) ==== 161+0+0 (secure 0 0 0) 0x7fbdc40bd700 con 0x7fbdcc106720 2024-05-09T08:49:41.618 INFO:teuthology.orchestra.run.smithi107.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T08:49:41.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:41 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1870402956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:49:41.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:41 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3744466232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T08:49:41.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.622+0000 7fbdd0ddc640 1 -- 172.21.15.107:0/2342916120 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbdb80a2d00 msgr2=0x7fbdb80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.622+0000 7fbdd0ddc640 1 --2- 172.21.15.107:0/2342916120 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbdb80a2d00 0x7fbdb80a51c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc00b9c0 tx=0x7fbdbc005c50 comp rx=0 tx=0).stop 2024-05-09T08:49:41.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.622+0000 7fbdd0ddc640 1 -- 172.21.15.107:0/2342916120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdcc106720 msgr2=0x7fbdcc19c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.622+0000 7fbdd0ddc640 1 --2- 172.21.15.107:0/2342916120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdcc106720 0x7fbdcc19c7f0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4005cc0 tx=0x7fbdc4003ba0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.622+0000 7fbdd0ddc640 1 -- 172.21.15.107:0/2342916120 shutdown_connections 2024-05-09T08:49:41.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.622+0000 7fbdd0ddc640 1 --2- 172.21.15.107:0/2342916120 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7fbdb80a2d00 0x7fbdb80a51c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.622+0000 7fbdd0ddc640 1 --2- 172.21.15.107:0/2342916120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdcc106720 0x7fbdcc19c7f0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.622+0000 7fbdd0ddc640 1 --2- 172.21.15.107:0/2342916120 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbdcc1010d0 0x7fbdcc1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.622+0000 7fbdd0ddc640 1 --2- 172.21.15.107:0/2342916120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbdcc100710 0x7fbdcc197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.622+0000 7fbdd0ddc640 1 -- 172.21.15.107:0/2342916120 >> 172.21.15.107:0/2342916120 conn(0x7fbdcc077790 msgr2=0x7fbdcc103280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:41.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.622+0000 7fbdd0ddc640 1 -- 172.21.15.107:0/2342916120 shutdown_connections 2024-05-09T08:49:41.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.622+0000 7fbdd0ddc640 1 -- 172.21.15.107:0/2342916120 wait complete. 2024-05-09T08:49:41.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T08:49:41.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.740+0000 7f8fe5a38640 1 -- 172.21.15.107:0/2694927020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe01003d0 msgr2=0x7f8fe01007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.740+0000 7f8fe5a38640 1 --2- 172.21.15.107:0/2694927020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe01003d0 0x7f8fe01007b0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009900 tx=0x7f8fd002a410 comp rx=0 tx=0).stop 2024-05-09T08:49:41.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.740+0000 7f8fe5a38640 1 -- 172.21.15.107:0/2694927020 shutdown_connections 2024-05-09T08:49:41.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.740+0000 7f8fe5a38640 1 --2- 172.21.15.107:0/2694927020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fe0106b90 0x7f8fe010e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.740+0000 7f8fe5a38640 1 --2- 172.21.15.107:0/2694927020 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fe0100d80 0x7f8fe01064b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.740+0000 7f8fe5a38640 1 --2- 172.21.15.107:0/2694927020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe01003d0 0x7f8fe01007b0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.740+0000 7f8fe5a38640 1 -- 172.21.15.107:0/2694927020 >> 172.21.15.107:0/2694927020 conn(0x7f8fe00759c0 msgr2=0x7f8fe0075dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:41.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.740+0000 7f8fe5a38640 1 -- 172.21.15.107:0/2694927020 shutdown_connections 2024-05-09T08:49:41.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.740+0000 7f8fe5a38640 1 -- 172.21.15.107:0/2694927020 wait complete. 2024-05-09T08:49:41.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.741+0000 7f8fe5a38640 1 Processor -- start 2024-05-09T08:49:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.741+0000 7f8fe5a38640 1 -- start start 2024-05-09T08:49:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.741+0000 7f8fe5a38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe01003d0 0x7f8fe0198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.741+0000 7f8fe5a38640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fe0106b90 0x7f8fe0198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.741+0000 7f8fe5a38640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fe019c9f0 0x7f8fe019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.741+0000 7f8fe5a38640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe0111b30 con 0x7f8fe01003d0 2024-05-09T08:49:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.741+0000 7f8fe5a38640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01119b0 con 0x7f8fe0106b90 2024-05-09T08:49:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.741+0000 7f8fe5a38640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe0111cb0 con 0x7f8fe019c9f0 2024-05-09T08:49:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.741+0000 7f8fdeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe01003d0 0x7f8fe0198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.741+0000 7f8fdeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe01003d0 0x7f8fe0198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44504/0 (socket says 172.21.15.107:44504) 2024-05-09T08:49:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.741+0000 7f8fdeffd640 1 -- 172.21.15.107:0/3744006852 learned_addr learned my addr 172.21.15.107:0/3744006852 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:41.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.741+0000 7f8fde7fc640 1 --2- 172.21.15.107:0/3744006852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fe0106b90 0x7f8fe0198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:41.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.741+0000 7f8fdf7fe640 1 --2- 172.21.15.107:0/3744006852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fe019c9f0 0x7f8fe019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:41.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.742+0000 7f8fdeffd640 1 -- 172.21.15.107:0/3744006852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fe019c9f0 msgr2=0x7f8fe019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.742+0000 7f8fdeffd640 1 --2- 172.21.15.107:0/3744006852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fe019c9f0 0x7f8fe019cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.742+0000 7f8fdeffd640 1 -- 172.21.15.107:0/3744006852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fe0106b90 msgr2=0x7f8fe0198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:41.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.742+0000 7f8fdeffd640 1 --2- 172.21.15.107:0/3744006852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fe0106b90 0x7f8fe0198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:41.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.742+0000 7f8fdeffd640 1 -- 172.21.15.107:0/3744006852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe019d4e0 con 0x7f8fe01003d0 2024-05-09T08:49:41.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.742+0000 7f8fdeffd640 1 --2- 172.21.15.107:0/3744006852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe01003d0 0x7f8fe0198130 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0005940 tx=0x7f8fd00040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:41.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.742+0000 7f8fe4a36640 1 -- 172.21.15.107:0/3744006852 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd003f030 con 0x7f8fe01003d0 2024-05-09T08:49:41.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.742+0000 7f8fe5a38640 1 -- 172.21.15.107:0/3744006852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fe019d770 con 0x7f8fe01003d0 2024-05-09T08:49:41.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.742+0000 7f8fe4a36640 1 -- 172.21.15.107:0/3744006852 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fd00044c0 con 0x7f8fe01003d0 2024-05-09T08:49:41.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.742+0000 7f8fe4a36640 1 -- 172.21.15.107:0/3744006852 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0035070 con 0x7f8fe01003d0 2024-05-09T08:49:41.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.742+0000 7f8fe5a38640 1 -- 172.21.15.107:0/3744006852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fe01a4f90 con 0x7f8fe01003d0 2024-05-09T08:49:41.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.743+0000 7f8fe5a38640 1 -- 172.21.15.107:0/3744006852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fe0068aa0 con 0x7f8fe01003d0 2024-05-09T08:49:41.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.744+0000 7f8fe4a36640 1 -- 172.21.15.107:0/3744006852 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8fd0042030 con 0x7f8fe01003d0 2024-05-09T08:49:41.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.745+0000 7f8fe4a36640 1 --2- 172.21.15.107:0/3744006852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fc80a2d00 0x7f8fc80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:41.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.745+0000 7f8fde7fc640 1 --2- 172.21.15.107:0/3744006852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fc80a2d00 0x7f8fc80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:41.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.745+0000 7f8fe4a36640 1 -- 172.21.15.107:0/3744006852 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9637+0+0 (secure 0 0 0) 0x7f8fd00efbd0 con 0x7f8fe01003d0 2024-05-09T08:49:41.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.745+0000 7f8fde7fc640 1 --2- 172.21.15.107:0/3744006852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fc80a2d00 0x7f8fc80a51c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00ffe50 tx=0x7f8fd4005f90 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:41.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.749+0000 7f8fe4a36640 1 -- 172.21.15.107:0/3744006852 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8fd00bbd40 con 0x7f8fe01003d0 2024-05-09T08:49:41.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:41.964+0000 7f8fe5a38640 1 -- 172.21.15.107:0/3744006852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f8fe0109db0 con 0x7f8fe01003d0 2024-05-09T08:49:42.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:41 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-a[30118]: 2024-05-09T08:49:41.618+0000 7f03434f5700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T08:49:42.094 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:49:41 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-c[64054]: ignoring --setuser ceph since I am not root 2024-05-09T08:49:42.094 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:49:41 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-c[64054]: ignoring --setgroup ceph since I am not root 2024-05-09T08:49:42.094 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:49:41 smithi157 ceph-mds[64058]: -- 172.21.15.157:0/327381565 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bc8a411a40 con 0x55bc8a3e4800 2024-05-09T08:49:42.094 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:49:41 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-c[64054]: starting mds.c at 2024-05-09T08:49:42.094 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:49:41 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-i[64360]: ignoring --setuser ceph since I am not root 2024-05-09T08:49:42.095 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:49:41 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-i[64360]: ignoring --setgroup ceph since I am not root 2024-05-09T08:49:42.095 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:49:41 smithi157 ceph-mds[64364]: -- 172.21.15.157:0/2144680800 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x555c0093e4e0 con 0x555c00914800 2024-05-09T08:49:42.095 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:49:41 smithi157 ceph-mds[64364]: -- 172.21.15.157:0/2144680800 <== mon.1 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555c00941a40 con 0x555c00914800 2024-05-09T08:49:42.095 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:49:41 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-i[64360]: starting mds.i at 2024-05-09T08:49:42.095 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:49:41 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-l[63415]: ignoring --setuser ceph since I am not root 2024-05-09T08:49:42.095 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:49:41 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-l[63415]: ignoring --setgroup ceph since I am not root 2024-05-09T08:49:42.095 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:49:41 smithi157 ceph-mds[63419]: -- 172.21.15.157:0/4159846458 <== mon.1 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55663bd95a40 con 0x55663bd68c00 2024-05-09T08:49:42.095 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:49:41 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-l[63415]: starting mds.l at 2024-05-09T08:49:42.120 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:49:41 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-h[63480]: ignoring --setuser ceph since I am not root 2024-05-09T08:49:42.121 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:49:41 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-h[63480]: ignoring --setgroup ceph since I am not root 2024-05-09T08:49:42.121 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:49:41 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-h[63480]: starting mds.h at 2024-05-09T08:49:42.121 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:49:41 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-e[63163]: ignoring --setuser ceph since I am not root 2024-05-09T08:49:42.121 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:49:41 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-e[63163]: ignoring --setgroup ceph since I am not root 2024-05-09T08:49:42.121 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:49:41 smithi110 ceph-mds[63167]: -- 172.21.15.110:0/4190521133 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cc0d933a40 con 0x55cc0d906c00 2024-05-09T08:49:42.121 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:49:41 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-e[63163]: starting mds.e at 2024-05-09T08:49:42.122 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:49:41 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-b[63783]: ignoring --setuser ceph since I am not root 2024-05-09T08:49:42.122 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:49:41 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-b[63783]: ignoring --setgroup ceph since I am not root 2024-05-09T08:49:42.122 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:49:41 smithi110 ceph-mds[63787]: -- 172.21.15.110:0/488049221 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56305ad49a40 con 0x56305ad1d000 2024-05-09T08:49:42.122 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:49:41 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-b[63783]: starting mds.b at 2024-05-09T08:49:42.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:42 smithi107 ceph-mon[30122]: pgmap v515: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-09T08:49:42.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:42 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/847207616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:49:42.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:42 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2342916120' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-09T08:49:42.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:42 smithi107 ceph-mon[30122]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-09T08:49:42.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:42 smithi107 ceph-mon[30122]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T08:49:42.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:42 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2342916120' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-09T08:49:42.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:42 smithi107 ceph-mon[30122]: osdmap e117: 12 total, 12 up, 12 in 2024-05-09T08:49:42.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:42 smithi107 ceph-mon[30122]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-09T08:49:42.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:42 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3744006852' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:49:42.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:42 smithi157 ceph-mon[33614]: pgmap v515: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-09T08:49:42.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:42 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/847207616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:49:42.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:42 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2342916120' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-09T08:49:42.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:42 smithi157 ceph-mon[33614]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-09T08:49:42.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:42 smithi157 ceph-mon[33614]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T08:49:42.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:42 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2342916120' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-09T08:49:42.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:42 smithi157 ceph-mon[33614]: osdmap e117: 12 total, 12 up, 12 in 2024-05-09T08:49:42.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:42 smithi157 ceph-mon[33614]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-09T08:49:42.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:42 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3744006852' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:49:42.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.621+0000 7f8fe4a36640 1 -- 172.21.15.107:0/3744006852 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) ==== 104+0+0 (secure 0 0 0) 0x7f8fd00c0040 con 0x7f8fe01003d0 2024-05-09T08:49:42.620 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:42 smithi110 ceph-mon[33622]: pgmap v515: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-09T08:49:42.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:42 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/847207616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T08:49:42.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:42 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2342916120' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-09T08:49:42.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:42 smithi110 ceph-mon[33622]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-09T08:49:42.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:42 smithi110 ceph-mon[33622]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T08:49:42.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:42 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2342916120' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-09T08:49:42.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:42 smithi110 ceph-mon[33622]: osdmap e117: 12 total, 12 up, 12 in 2024-05-09T08:49:42.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:42 smithi110 ceph-mon[33622]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-09T08:49:42.621 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:42 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3744006852' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T08:49:42.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.625+0000 7f8fe5a38640 1 -- 172.21.15.107:0/3744006852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fc80a2d00 msgr2=0x7f8fc80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:42.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.625+0000 7f8fe5a38640 1 --2- 172.21.15.107:0/3744006852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fc80a2d00 0x7f8fc80a51c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00ffe50 tx=0x7f8fd4005f90 comp rx=0 tx=0).stop 2024-05-09T08:49:42.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.625+0000 7f8fe5a38640 1 -- 172.21.15.107:0/3744006852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe01003d0 msgr2=0x7f8fe0198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:42.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.625+0000 7f8fe5a38640 1 --2- 172.21.15.107:0/3744006852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe01003d0 0x7f8fe0198130 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0005940 tx=0x7f8fd00040c0 comp rx=0 tx=0).stop 2024-05-09T08:49:42.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.625+0000 7f8fe5a38640 1 -- 172.21.15.107:0/3744006852 shutdown_connections 2024-05-09T08:49:42.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.625+0000 7f8fe5a38640 1 --2- 172.21.15.107:0/3744006852 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f8fc80a2d00 0x7f8fc80a51c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:42.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.625+0000 7f8fe5a38640 1 --2- 172.21.15.107:0/3744006852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fe019c9f0 0x7f8fe019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:42.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.625+0000 7f8fe5a38640 1 --2- 172.21.15.107:0/3744006852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fe0106b90 0x7f8fe0198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:42.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.625+0000 7f8fe5a38640 1 --2- 172.21.15.107:0/3744006852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8fe01003d0 0x7f8fe0198130 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:42.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.625+0000 7f8fe5a38640 1 -- 172.21.15.107:0/3744006852 >> 172.21.15.107:0/3744006852 conn(0x7f8fe00759c0 msgr2=0x7f8fe00fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:42.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.625+0000 7f8fe5a38640 1 -- 172.21.15.107:0/3744006852 shutdown_connections 2024-05-09T08:49:42.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.625+0000 7f8fe5a38640 1 -- 172.21.15.107:0/3744006852 wait complete. 2024-05-09T08:49:42.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T08:49:42.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.754+0000 7f94ed441640 1 -- 172.21.15.107:0/2814606508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94e81010d0 msgr2=0x7f94e810d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.754+0000 7f94ed441640 1 --2- 172.21.15.107:0/2814606508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94e81010d0 0x7f94e810d5e0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f94e0009cd0 tx=0x7f94e002a8e0 comp rx=0 tx=0).stop 2024-05-09T08:49:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.754+0000 7f94ed441640 1 -- 172.21.15.107:0/2814606508 shutdown_connections 2024-05-09T08:49:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.754+0000 7f94ed441640 1 --2- 172.21.15.107:0/2814606508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94e81010d0 0x7f94e810d5e0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.754+0000 7f94ed441640 1 --2- 172.21.15.107:0/2814606508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94e8100710 0x7f94e8100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.754+0000 7f94ed441640 1 --2- 172.21.15.107:0/2814606508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94e8106720 0x7f94e8106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.754+0000 7f94ed441640 1 -- 172.21.15.107:0/2814606508 >> 172.21.15.107:0/2814606508 conn(0x7f94e8077790 msgr2=0x7f94e8077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.754+0000 7f94ed441640 1 -- 172.21.15.107:0/2814606508 shutdown_connections 2024-05-09T08:49:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.754+0000 7f94ed441640 1 -- 172.21.15.107:0/2814606508 wait complete. 2024-05-09T08:49:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94ed441640 1 Processor -- start 2024-05-09T08:49:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94ed441640 1 -- start start 2024-05-09T08:49:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94ed441640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94e8100710 0x7f94e8197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94ed441640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94e81010d0 0x7f94e8198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94ed441640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94e8106720 0x7f94e819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94ed441640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f94e81109a0 con 0x7f94e81010d0 2024-05-09T08:49:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94ed441640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f94e8110820 con 0x7f94e8100710 2024-05-09T08:49:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94ed441640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f94e8110b20 con 0x7f94e8106720 2024-05-09T08:49:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94dffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94e81010d0 0x7f94e8198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94dffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94e81010d0 0x7f94e8198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44510/0 (socket says 172.21.15.107:44510) 2024-05-09T08:49:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94dffff640 1 -- 172.21.15.107:0/1661130461 learned_addr learned my addr 172.21.15.107:0/1661130461 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94e77fe640 1 --2- 172.21.15.107:0/1661130461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94e8106720 0x7f94e819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:42.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94e6ffd640 1 --2- 172.21.15.107:0/1661130461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94e8100710 0x7f94e8197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:42.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94dffff640 1 -- 172.21.15.107:0/1661130461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94e8106720 msgr2=0x7f94e819c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:42.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94dffff640 1 --2- 172.21.15.107:0/1661130461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94e8106720 0x7f94e819c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:42.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94dffff640 1 -- 172.21.15.107:0/1661130461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94e8100710 msgr2=0x7f94e8197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:42.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94dffff640 1 --2- 172.21.15.107:0/1661130461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94e8100710 0x7f94e8197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:42.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94dffff640 1 -- 172.21.15.107:0/1661130461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e819cf40 con 0x7f94e81010d0 2024-05-09T08:49:42.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94e77fe640 1 --2- 172.21.15.107:0/1661130461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94e8106720 0x7f94e819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T08:49:42.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.755+0000 7f94dffff640 1 --2- 172.21.15.107:0/1661130461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94e81010d0 0x7f94e8198490 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f94d000c890 tx=0x7f94d000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:42.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.756+0000 7f94e4ff9640 1 -- 172.21.15.107:0/1661130461 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d0004220 con 0x7f94e81010d0 2024-05-09T08:49:42.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.756+0000 7f94ed441640 1 -- 172.21.15.107:0/1661130461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94e819d230 con 0x7f94e81010d0 2024-05-09T08:49:42.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.756+0000 7f94e4ff9640 1 -- 172.21.15.107:0/1661130461 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94d00043c0 con 0x7f94e81010d0 2024-05-09T08:49:42.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.756+0000 7f94e4ff9640 1 -- 172.21.15.107:0/1661130461 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d000f450 con 0x7f94e81010d0 2024-05-09T08:49:42.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.756+0000 7f94ed441640 1 -- 172.21.15.107:0/1661130461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e80fd670 con 0x7f94e81010d0 2024-05-09T08:49:42.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.757+0000 7f94ed441640 1 -- 172.21.15.107:0/1661130461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94e8068aa0 con 0x7f94e81010d0 2024-05-09T08:49:42.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.759+0000 7f94e4ff9640 1 -- 172.21.15.107:0/1661130461 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f94d0003680 con 0x7f94e81010d0 2024-05-09T08:49:42.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.759+0000 7f94e4ff9640 1 --2- 172.21.15.107:0/1661130461 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f94d40a2d00 0x7f94d40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:42.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.759+0000 7f94e4ff9640 1 -- 172.21.15.107:0/1661130461 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9637+0+0 (secure 0 0 0) 0x7f94d00cfa80 con 0x7f94e81010d0 2024-05-09T08:49:42.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.759+0000 7f94e6ffd640 1 --2- 172.21.15.107:0/1661130461 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f94d40a2d00 0x7f94d40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:42.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.760+0000 7f94e6ffd640 1 --2- 172.21.15.107:0/1661130461 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f94d40a2d00 0x7f94d40a51c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f94d800cac0 tx=0x7f94d800c4a0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:42.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.763+0000 7f94e4ff9640 1 -- 172.21.15.107:0/1661130461 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f94d0014030 con 0x7f94e81010d0 2024-05-09T08:49:42.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:42.946+0000 7f94ed441640 1 -- 172.21.15.107:0/1661130461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f94e8109d10 con 0x7f94e81010d0 2024-05-09T08:49:43.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-09T08:49:43.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-09T08:49:43.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: Cluster is now healthy 2024-05-09T08:49:43.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/3744006852' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-09T08:49:43.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.157:6832/2414872439,v1:172.21.15.157:6833/2414872439] up:boot 2024-05-09T08:49:43.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.157:6836/716074987,v1:172.21.15.157:6838/716074987] up:boot 2024-05-09T08:49:43.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.157:6837/2387344458,v1:172.21.15.157:6839/2387344458] up:boot 2024-05-09T08:49:43.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.110:6834/2271760228,v1:172.21.15.110:6835/2271760228] up:boot 2024-05-09T08:49:43.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.110:6836/3288830095,v1:172.21.15.110:6837/3288830095] up:boot 2024-05-09T08:49:43.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: mds.? [v2:172.21.15.110:6838/1591696568,v1:172.21.15.110:6839/1591696568] up:boot 2024-05-09T08:49:43.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: fsmap 12 up:standby 2024-05-09T08:49:43.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:49:43.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:49:43.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:49:43.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:49:43.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:49:43.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:49:43.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:43 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1661130461' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:49:43.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-09T08:49:43.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-09T08:49:43.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: Cluster is now healthy 2024-05-09T08:49:43.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/3744006852' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-09T08:49:43.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.157:6832/2414872439,v1:172.21.15.157:6833/2414872439] up:boot 2024-05-09T08:49:43.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.157:6836/716074987,v1:172.21.15.157:6838/716074987] up:boot 2024-05-09T08:49:43.594 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.157:6837/2387344458,v1:172.21.15.157:6839/2387344458] up:boot 2024-05-09T08:49:43.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.110:6834/2271760228,v1:172.21.15.110:6835/2271760228] up:boot 2024-05-09T08:49:43.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.110:6836/3288830095,v1:172.21.15.110:6837/3288830095] up:boot 2024-05-09T08:49:43.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: mds.? [v2:172.21.15.110:6838/1591696568,v1:172.21.15.110:6839/1591696568] up:boot 2024-05-09T08:49:43.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: fsmap 12 up:standby 2024-05-09T08:49:43.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:49:43.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:49:43.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:49:43.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:49:43.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:49:43.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:49:43.595 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:43 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1661130461' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:49:43.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-09T08:49:43.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-09T08:49:43.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: Cluster is now healthy 2024-05-09T08:49:43.622 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/3744006852' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-09T08:49:43.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.157:6832/2414872439,v1:172.21.15.157:6833/2414872439] up:boot 2024-05-09T08:49:43.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.157:6836/716074987,v1:172.21.15.157:6838/716074987] up:boot 2024-05-09T08:49:43.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.157:6837/2387344458,v1:172.21.15.157:6839/2387344458] up:boot 2024-05-09T08:49:43.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.110:6834/2271760228,v1:172.21.15.110:6835/2271760228] up:boot 2024-05-09T08:49:43.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.110:6836/3288830095,v1:172.21.15.110:6837/3288830095] up:boot 2024-05-09T08:49:43.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: mds.? [v2:172.21.15.110:6838/1591696568,v1:172.21.15.110:6839/1591696568] up:boot 2024-05-09T08:49:43.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: fsmap 12 up:standby 2024-05-09T08:49:43.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T08:49:43.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T08:49:43.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T08:49:43.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T08:49:43.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T08:49:43.623 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: from='mgr.15483 172.21.15.107:0/1176825267' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T08:49:43.624 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:43 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1661130461' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:49:43.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.624+0000 7f94e4ff9640 1 -- 172.21.15.107:0/1661130461 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v118) ==== 181+0+0 (secure 0 0 0) 0x7f94d009c240 con 0x7f94e81010d0 2024-05-09T08:49:43.624 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' removed 2024-05-09T08:49:43.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.627+0000 7f94ed441640 1 -- 172.21.15.107:0/1661130461 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f94d40a2d00 msgr2=0x7f94d40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.627+0000 7f94ed441640 1 --2- 172.21.15.107:0/1661130461 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f94d40a2d00 0x7f94d40a51c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f94d800cac0 tx=0x7f94d800c4a0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.628+0000 7f94ed441640 1 -- 172.21.15.107:0/1661130461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94e81010d0 msgr2=0x7f94e8198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.628+0000 7f94ed441640 1 --2- 172.21.15.107:0/1661130461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94e81010d0 0x7f94e8198490 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f94d000c890 tx=0x7f94d000cd50 comp rx=0 tx=0).stop 2024-05-09T08:49:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.628+0000 7f94ed441640 1 -- 172.21.15.107:0/1661130461 shutdown_connections 2024-05-09T08:49:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.628+0000 7f94ed441640 1 --2- 172.21.15.107:0/1661130461 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f94d40a2d00 0x7f94d40a51c0 secure :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f94d800cac0 tx=0x7f94d800c4a0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.628+0000 7f94ed441640 1 --2- 172.21.15.107:0/1661130461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94e8106720 0x7f94e819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.628+0000 7f94ed441640 1 --2- 172.21.15.107:0/1661130461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94e81010d0 0x7f94e8198490 secure :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f94d000c890 tx=0x7f94d000cd50 comp rx=0 tx=0).stop 2024-05-09T08:49:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.628+0000 7f94ed441640 1 --2- 172.21.15.107:0/1661130461 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94e8100710 0x7f94e8197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.628+0000 7f94ed441640 1 -- 172.21.15.107:0/1661130461 >> 172.21.15.107:0/1661130461 conn(0x7f94e8077790 msgr2=0x7f94e806a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:43.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.628+0000 7f94ed441640 1 -- 172.21.15.107:0/1661130461 shutdown_connections 2024-05-09T08:49:43.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.628+0000 7f94ed441640 1 -- 172.21.15.107:0/1661130461 wait complete. 2024-05-09T08:49:43.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T08:49:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.750+0000 7f16ba7f9640 1 -- 172.21.15.107:0/2517071664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b4100700 msgr2=0x7f16b4100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.750+0000 7f16ba7f9640 1 --2- 172.21.15.107:0/2517071664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b4100700 0x7f16b4100b80 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f169c009930 tx=0x7f169c02a420 comp rx=0 tx=0).stop 2024-05-09T08:49:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.751+0000 7f16ba7f9640 1 -- 172.21.15.107:0/2517071664 shutdown_connections 2024-05-09T08:49:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.751+0000 7f16ba7f9640 1 --2- 172.21.15.107:0/2517071664 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16b41010c0 0x7f16b410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.751+0000 7f16ba7f9640 1 --2- 172.21.15.107:0/2517071664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b4100700 0x7f16b4100b80 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.751+0000 7f16ba7f9640 1 --2- 172.21.15.107:0/2517071664 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f16b4106710 0x7f16b4106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.751+0000 7f16ba7f9640 1 -- 172.21.15.107:0/2517071664 >> 172.21.15.107:0/2517071664 conn(0x7f16b4077830 msgr2=0x7f16b4077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.751+0000 7f16ba7f9640 1 -- 172.21.15.107:0/2517071664 shutdown_connections 2024-05-09T08:49:43.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.751+0000 7f16ba7f9640 1 -- 172.21.15.107:0/2517071664 wait complete. 2024-05-09T08:49:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.751+0000 7f16ba7f9640 1 Processor -- start 2024-05-09T08:49:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.751+0000 7f16ba7f9640 1 -- start start 2024-05-09T08:49:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.751+0000 7f16ba7f9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16b4100700 0x7f16b4198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.751+0000 7f16ba7f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b41010c0 0x7f16b4199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.751+0000 7f16ba7f9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f16b4106710 0x7f16b41a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.751+0000 7f16ba7f9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f16b4110990 con 0x7f16b41010c0 2024-05-09T08:49:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.751+0000 7f16ba7f9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f16b4110810 con 0x7f16b4100700 2024-05-09T08:49:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.751+0000 7f16ba7f9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f16b4110b10 con 0x7f16b4106710 2024-05-09T08:49:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.751+0000 7f16abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b41010c0 0x7f16b4199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:43.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.752+0000 7f16abfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b41010c0 0x7f16b4199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44530/0 (socket says 172.21.15.107:44530) 2024-05-09T08:49:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.752+0000 7f16abfff640 1 -- 172.21.15.107:0/2332669555 learned_addr learned my addr 172.21.15.107:0/2332669555 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.752+0000 7f16b8d96640 1 --2- 172.21.15.107:0/2332669555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f16b4106710 0x7f16b41a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.752+0000 7f16b3fff640 1 --2- 172.21.15.107:0/2332669555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16b4100700 0x7f16b4198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.752+0000 7f16abfff640 1 -- 172.21.15.107:0/2332669555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f16b4106710 msgr2=0x7f16b41a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.752+0000 7f16abfff640 1 --2- 172.21.15.107:0/2332669555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f16b4106710 0x7f16b41a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.752+0000 7f16abfff640 1 -- 172.21.15.107:0/2332669555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16b4100700 msgr2=0x7f16b4198e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.752+0000 7f16abfff640 1 --2- 172.21.15.107:0/2332669555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16b4100700 0x7f16b4198e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.752+0000 7f16abfff640 1 -- 172.21.15.107:0/2332669555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b41a43f0 con 0x7f16b41010c0 2024-05-09T08:49:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.752+0000 7f16abfff640 1 --2- 172.21.15.107:0/2332669555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b41010c0 0x7f16b4199370 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f169c004770 tx=0x7f169c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.752+0000 7f16b1ffb640 1 -- 172.21.15.107:0/2332669555 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169c03f070 con 0x7f16b41010c0 2024-05-09T08:49:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.752+0000 7f16b1ffb640 1 -- 172.21.15.107:0/2332669555 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f169c03b460 con 0x7f16b41010c0 2024-05-09T08:49:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.752+0000 7f16ba7f9640 1 -- 172.21.15.107:0/2332669555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b41a45c0 con 0x7f16b41010c0 2024-05-09T08:49:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.752+0000 7f16b1ffb640 1 -- 172.21.15.107:0/2332669555 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169c035070 con 0x7f16b41010c0 2024-05-09T08:49:43.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.752+0000 7f16ba7f9640 1 -- 172.21.15.107:0/2332669555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b41a4a30 con 0x7f16b41010c0 2024-05-09T08:49:43.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.753+0000 7f16ba7f9640 1 -- 172.21.15.107:0/2332669555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16b4068650 con 0x7f16b41010c0 2024-05-09T08:49:43.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.754+0000 7f16b1ffb640 1 -- 172.21.15.107:0/2332669555 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f169c03b610 con 0x7f16b41010c0 2024-05-09T08:49:43.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.755+0000 7f16b1ffb640 1 --2- 172.21.15.107:0/2332669555 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f16a00a2d00 0x7f16a00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:43.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.755+0000 7f16b3fff640 1 --2- 172.21.15.107:0/2332669555 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f16a00a2d00 0x7f16a00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:43.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.755+0000 7f16b1ffb640 1 -- 172.21.15.107:0/2332669555 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9083+0+0 (secure 0 0 0) 0x7f169c0ef950 con 0x7f16b41010c0 2024-05-09T08:49:43.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.756+0000 7f16b3fff640 1 --2- 172.21.15.107:0/2332669555 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f16a00a2d00 0x7f16a00a51c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f16a400ab60 tx=0x7f16a400a2d0 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.760+0000 7f16b1ffb640 1 -- 172.21.15.107:0/2332669555 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f169c0bc340 con 0x7f16b41010c0 2024-05-09T08:49:43.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:43.942+0000 7f16ba7f9640 1 -- 172.21.15.107:0/2332669555 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f16b4068fd0 con 0x7f16b41010c0 2024-05-09T08:49:44.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.626+0000 7f16b1ffb640 1 -- 172.21.15.107:0/2332669555 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v119) ==== 169+0+0 (secure 0 0 0) 0x7f169c0bbc40 con 0x7f16b41010c0 2024-05-09T08:49:44.627 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' removed 2024-05-09T08:49:44.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.630+0000 7f16ba7f9640 1 -- 172.21.15.107:0/2332669555 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f16a00a2d00 msgr2=0x7f16a00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:44.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.630+0000 7f16ba7f9640 1 --2- 172.21.15.107:0/2332669555 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f16a00a2d00 0x7f16a00a51c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f16a400ab60 tx=0x7f16a400a2d0 comp rx=0 tx=0).stop 2024-05-09T08:49:44.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.630+0000 7f16ba7f9640 1 -- 172.21.15.107:0/2332669555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b41010c0 msgr2=0x7f16b4199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:44.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.630+0000 7f16ba7f9640 1 --2- 172.21.15.107:0/2332669555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b41010c0 0x7f16b4199370 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f169c004770 tx=0x7f169c0042f0 comp rx=0 tx=0).stop 2024-05-09T08:49:44.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.630+0000 7f16ba7f9640 1 -- 172.21.15.107:0/2332669555 shutdown_connections 2024-05-09T08:49:44.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.630+0000 7f16ba7f9640 1 --2- 172.21.15.107:0/2332669555 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f16a00a2d00 0x7f16a00a51c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:44.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.630+0000 7f16ba7f9640 1 --2- 172.21.15.107:0/2332669555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f16b4106710 0x7f16b41a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:44.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.630+0000 7f16ba7f9640 1 --2- 172.21.15.107:0/2332669555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b41010c0 0x7f16b4199370 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:44.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.630+0000 7f16ba7f9640 1 --2- 172.21.15.107:0/2332669555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16b4100700 0x7f16b4198e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:44.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.630+0000 7f16ba7f9640 1 -- 172.21.15.107:0/2332669555 >> 172.21.15.107:0/2332669555 conn(0x7f16b4077830 msgr2=0x7f16b4102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:44.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.630+0000 7f16ba7f9640 1 -- 172.21.15.107:0/2332669555 shutdown_connections 2024-05-09T08:49:44.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.630+0000 7f16ba7f9640 1 -- 172.21.15.107:0/2332669555 wait complete. 2024-05-09T08:49:44.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-05-09T08:49:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.771+0000 7f494feb3640 1 -- 172.21.15.107:0/611032656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49481010c0 msgr2=0x7f494810d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.771+0000 7f494feb3640 1 --2- 172.21.15.107:0/611032656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49481010c0 0x7f494810d5d0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f4940009930 tx=0x7f494002a420 comp rx=0 tx=0).stop 2024-05-09T08:49:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.772+0000 7f494feb3640 1 -- 172.21.15.107:0/611032656 shutdown_connections 2024-05-09T08:49:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.772+0000 7f494feb3640 1 --2- 172.21.15.107:0/611032656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49481010c0 0x7f494810d5d0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.772+0000 7f494feb3640 1 --2- 172.21.15.107:0/611032656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4948100700 0x7f4948100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.772+0000 7f494feb3640 1 --2- 172.21.15.107:0/611032656 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4948106710 0x7f4948106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.772+0000 7f494feb3640 1 -- 172.21.15.107:0/611032656 >> 172.21.15.107:0/611032656 conn(0x7f4948077830 msgr2=0x7f4948077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.772+0000 7f494feb3640 1 -- 172.21.15.107:0/611032656 shutdown_connections 2024-05-09T08:49:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.772+0000 7f494feb3640 1 -- 172.21.15.107:0/611032656 wait complete. 2024-05-09T08:49:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.772+0000 7f494feb3640 1 Processor -- start 2024-05-09T08:49:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.772+0000 7f494feb3640 1 -- start start 2024-05-09T08:49:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.772+0000 7f494feb3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4948100700 0x7f4948198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.772+0000 7f494feb3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49481010c0 0x7f4948198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.772+0000 7f494feb3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4948106710 0x7f494819c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.772+0000 7f494feb3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4948111ad0 con 0x7f49481010c0 2024-05-09T08:49:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.772+0000 7f494feb3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4948111950 con 0x7f4948100700 2024-05-09T08:49:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.772+0000 7f494feb3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4948111c50 con 0x7f4948106710 2024-05-09T08:49:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.772+0000 7f494d44e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49481010c0 0x7f4948198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.773+0000 7f494d44e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49481010c0 0x7f4948198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44558/0 (socket says 172.21.15.107:44558) 2024-05-09T08:49:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.773+0000 7f494d44e640 1 -- 172.21.15.107:0/2904501026 learned_addr learned my addr 172.21.15.107:0/2904501026 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-05-09T08:49:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.773+0000 7f494e450640 1 --2- 172.21.15.107:0/2904501026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4948106710 0x7f494819c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.773+0000 7f494dc4f640 1 --2- 172.21.15.107:0/2904501026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4948100700 0x7f4948198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.773+0000 7f494d44e640 1 -- 172.21.15.107:0/2904501026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4948106710 msgr2=0x7f494819c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.773+0000 7f494d44e640 1 --2- 172.21.15.107:0/2904501026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4948106710 0x7f494819c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.773+0000 7f494d44e640 1 -- 172.21.15.107:0/2904501026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4948100700 msgr2=0x7f4948198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.773+0000 7f494d44e640 1 --2- 172.21.15.107:0/2904501026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4948100700 0x7f4948198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.773+0000 7f494d44e640 1 -- 172.21.15.107:0/2904501026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f494819d040 con 0x7f49481010c0 2024-05-09T08:49:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.773+0000 7f494d44e640 1 --2- 172.21.15.107:0/2904501026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49481010c0 0x7f4948198590 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f493800b4a0 tx=0x7f493800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:44.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.773+0000 7f493effd640 1 -- 172.21.15.107:0/2904501026 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4938014070 con 0x7f49481010c0 2024-05-09T08:49:44.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.773+0000 7f493effd640 1 -- 172.21.15.107:0/2904501026 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4938010ac0 con 0x7f49481010c0 2024-05-09T08:49:44.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.773+0000 7f494feb3640 1 -- 172.21.15.107:0/2904501026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f494819d330 con 0x7f49481010c0 2024-05-09T08:49:44.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.773+0000 7f493effd640 1 -- 172.21.15.107:0/2904501026 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4938017c30 con 0x7f49481010c0 2024-05-09T08:49:44.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.773+0000 7f494feb3640 1 -- 172.21.15.107:0/2904501026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49481a4bc0 con 0x7f49481010c0 2024-05-09T08:49:44.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.774+0000 7f494feb3640 1 -- 172.21.15.107:0/2904501026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4948068650 con 0x7f49481010c0 2024-05-09T08:49:44.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.775+0000 7f493effd640 1 -- 172.21.15.107:0/2904501026 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f4938017070 con 0x7f49481010c0 2024-05-09T08:49:44.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.776+0000 7f493effd640 1 --2- 172.21.15.107:0/2904501026 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f49340a2d00 0x7f49340a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T08:49:44.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.776+0000 7f494dc4f640 1 --2- 172.21.15.107:0/2904501026 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f49340a2d00 0x7f49340a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T08:49:44.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.776+0000 7f493effd640 1 -- 172.21.15.107:0/2904501026 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8582+0+0 (secure 0 0 0) 0x7f49380cea50 con 0x7f49481010c0 2024-05-09T08:49:44.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.776+0000 7f494dc4f640 1 --2- 172.21.15.107:0/2904501026 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f49340a2d00 0x7f49340a51c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4944005c40 tx=0x7f494400a780 comp rx=0 tx=0).ready entity=mgr.15483 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T08:49:44.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.780+0000 7f493effd640 1 -- 172.21.15.107:0/2904501026 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f493809b740 con 0x7f49481010c0 2024-05-09T08:49:44.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:44.959+0000 7f494feb3640 1 -- 172.21.15.107:0/2904501026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f49481993a0 con 0x7f49481010c0 2024-05-09T08:49:45.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:44 smithi107 ceph-mon[30122]: pgmap v517: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 7 op/s 2024-05-09T08:49:45.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:44 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/1661130461' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T08:49:45.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:44 smithi107 ceph-mon[30122]: osdmap e118: 12 total, 12 up, 12 in 2024-05-09T08:49:45.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:44 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2332669555' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:49:45.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:44 smithi157 ceph-mon[33614]: pgmap v517: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 7 op/s 2024-05-09T08:49:45.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:44 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/1661130461' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T08:49:45.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:44 smithi157 ceph-mon[33614]: osdmap e118: 12 total, 12 up, 12 in 2024-05-09T08:49:45.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:44 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2332669555' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:49:45.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:44 smithi110 ceph-mon[33622]: pgmap v517: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 7 op/s 2024-05-09T08:49:45.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:44 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/1661130461' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T08:49:45.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:44 smithi110 ceph-mon[33622]: osdmap e118: 12 total, 12 up, 12 in 2024-05-09T08:49:45.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:44 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2332669555' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:49:45.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:45.639+0000 7f493effd640 1 -- 172.21.15.107:0/2904501026 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v120) ==== 178+0+0 (secure 0 0 0) 0x7f49481993a0 con 0x7f49481010c0 2024-05-09T08:49:45.640 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data_ec' removed 2024-05-09T08:49:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:45.643+0000 7f494feb3640 1 -- 172.21.15.107:0/2904501026 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f49340a2d00 msgr2=0x7f49340a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:45.643+0000 7f494feb3640 1 --2- 172.21.15.107:0/2904501026 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f49340a2d00 0x7f49340a51c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4944005c40 tx=0x7f494400a780 comp rx=0 tx=0).stop 2024-05-09T08:49:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:45.643+0000 7f494feb3640 1 -- 172.21.15.107:0/2904501026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49481010c0 msgr2=0x7f4948198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T08:49:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:45.643+0000 7f494feb3640 1 --2- 172.21.15.107:0/2904501026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49481010c0 0x7f4948198590 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f493800b4a0 tx=0x7f493800b960 comp rx=0 tx=0).stop 2024-05-09T08:49:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:45.643+0000 7f494feb3640 1 -- 172.21.15.107:0/2904501026 shutdown_connections 2024-05-09T08:49:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:45.643+0000 7f494feb3640 1 --2- 172.21.15.107:0/2904501026 >> [v2:172.21.15.107:6800/165281804,v1:172.21.15.107:6801/165281804] conn(0x7f49340a2d00 0x7f49340a51c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:45.643+0000 7f494feb3640 1 --2- 172.21.15.107:0/2904501026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4948106710 0x7f494819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:45.643+0000 7f494feb3640 1 --2- 172.21.15.107:0/2904501026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49481010c0 0x7f4948198590 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:45.643+0000 7f494feb3640 1 --2- 172.21.15.107:0/2904501026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4948100700 0x7f4948198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T08:49:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:45.643+0000 7f494feb3640 1 -- 172.21.15.107:0/2904501026 >> 172.21.15.107:0/2904501026 conn(0x7f4948077830 msgr2=0x7f4948103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T08:49:45.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:45.644+0000 7f494feb3640 1 -- 172.21.15.107:0/2904501026 shutdown_connections 2024-05-09T08:49:45.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-05-09T08:49:45.644+0000 7f494feb3640 1 -- 172.21.15.107:0/2904501026 wait complete. 2024-05-09T08:49:45.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:49:45.714 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:49:45.758 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:49:45.789 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-09T08:49:45.789 DEBUG:teuthology.orchestra.run.smithi107:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-09T08:49:45.807 DEBUG:teuthology.orchestra.run.smithi110:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-09T08:49:45.829 DEBUG:teuthology.orchestra.run.smithi157:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-09T08:49:45.847 INFO:tasks.cephadm:Stopping all daemons... 2024-05-09T08:49:45.848 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-09T08:49:45.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.a 2024-05-09T08:49:45.896 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:45 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2332669555' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T08:49:45.896 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:45 smithi107 ceph-mon[30122]: osdmap e119: 12 total, 12 up, 12 in 2024-05-09T08:49:45.896 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:45 smithi107 ceph-mon[30122]: from='client.? 172.21.15.107:0/2904501026' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:49:46.093 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:45 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2332669555' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T08:49:46.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:45 smithi157 ceph-mon[33614]: osdmap e119: 12 total, 12 up, 12 in 2024-05-09T08:49:46.094 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:45 smithi157 ceph-mon[33614]: from='client.? 172.21.15.107:0/2904501026' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:49:46.120 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:45 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2332669555' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T08:49:46.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:45 smithi110 ceph-mon[33622]: osdmap e119: 12 total, 12 up, 12 in 2024-05-09T08:49:46.121 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:45 smithi110 ceph-mon[33622]: from='client.? 172.21.15.107:0/2904501026' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T08:49:46.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:45 smithi107 systemd[1]: Stopping Ceph mon.a for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:49:46.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:46 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-a[30118]: 2024-05-09T08:49:46.338+0000 7f0348d00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:46.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:46 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-a[30118]: 2024-05-09T08:49:46.338+0000 7f0348d00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-09T08:49:46.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:46 smithi107 podman[115345]: 2024-05-09 08:49:46.566219937 +0000 UTC m=+0.377932788 container died 754b572e3f5fb58e446791cb4e4edd20751a86603ed77774f7f0ca1378e22b32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-05-09T08:49:47.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:47 smithi107 podman[115345]: 2024-05-09 08:49:47.334670007 +0000 UTC m=+1.146382858 container cleanup 754b572e3f5fb58e446791cb4e4edd20751a86603ed77774f7f0ca1378e22b32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-09T08:49:47.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:47 smithi107 bash[115345]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-a 2024-05-09T08:49:47.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.a.service' 2024-05-09T08:49:47.946 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:47 smithi107 podman[115354]: 2024-05-09 08:49:47.678134495 +0000 UTC m=+1.104751797 container remove 754b572e3f5fb58e446791cb4e4edd20751a86603ed77774f7f0ca1378e22b32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-09T08:49:47.946 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:47 smithi107 systemd[1]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-09T08:49:47.947 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:47 smithi107 systemd[1]: Stopped Ceph mon.a for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:49:47.947 INFO:journalctl@ceph.mon.a.smithi107.stdout:May 09 08:49:47 smithi107 systemd[1]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.a.service: Consumed 25.646s CPU time. 2024-05-09T08:49:48.068 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:48.068 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-09T08:49:48.068 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-09T08:49:48.068 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.b 2024-05-09T08:49:48.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:48 smithi110 systemd[1]: Stopping Ceph mon.b for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:49:48.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:48 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-b[33618]: 2024-05-09T08:49:48.568+0000 7f3cbe012700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:48.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:48 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-b[33618]: 2024-05-09T08:49:48.568+0000 7f3cbe012700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-09T08:49:48.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:48 smithi110 podman[69819]: 2024-05-09 08:49:48.631788789 +0000 UTC m=+0.193766649 container died ca5207af62f5837d92f0d60f651018c8ca7d3ab294c81771dfc844555770d6d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-b, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-09T08:49:49.294 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:49 smithi110 podman[69819]: 2024-05-09 08:49:49.01887083 +0000 UTC m=+0.580848699 container cleanup ca5207af62f5837d92f0d60f651018c8ca7d3ab294c81771dfc844555770d6d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-b, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:49:49.294 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:49 smithi110 bash[69819]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-b 2024-05-09T08:49:49.548 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:49 smithi110 podman[69830]: 2024-05-09 08:49:49.295851528 +0000 UTC m=+0.656818469 container remove ca5207af62f5837d92f0d60f651018c8ca7d3ab294c81771dfc844555770d6d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-09T08:49:49.573 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.b.service' 2024-05-09T08:49:49.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:49 smithi110 systemd[1]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.b.service: Deactivated successfully. 2024-05-09T08:49:49.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:49 smithi110 systemd[1]: Stopped Ceph mon.b for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:49:49.871 INFO:journalctl@ceph.mon.b.smithi110.stdout:May 09 08:49:49 smithi110 systemd[1]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.b.service: Consumed 16.193s CPU time. 2024-05-09T08:49:49.927 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:49.928 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-09T08:49:49.928 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-09T08:49:49.928 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.c 2024-05-09T08:49:50.343 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:49 smithi157 systemd[1]: Stopping Ceph mon.c for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:49:50.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:50 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-c[33588]: 2024-05-09T08:49:50.524+0000 7fb126863700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:50.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:50 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-c[33588]: 2024-05-09T08:49:50.524+0000 7fb126863700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-09T08:49:50.844 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:50 smithi157 podman[71215]: 2024-05-09 08:49:50.586195298 +0000 UTC m=+0.354907389 container died 7cd67865c9ab2215318b780a03ba7562b86221ae5bd875a6535cfe82ec670815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-05-09T08:49:51.317 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:50 smithi157 podman[71215]: 2024-05-09 08:49:50.991535086 +0000 UTC m=+0.760247168 container cleanup 7cd67865c9ab2215318b780a03ba7562b86221ae5bd875a6535cfe82ec670815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-09T08:49:51.317 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:50 smithi157 bash[71215]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-c 2024-05-09T08:49:51.545 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.c.service' 2024-05-09T08:49:51.574 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:51 smithi157 podman[71226]: 2024-05-09 08:49:51.318247998 +0000 UTC m=+0.733248170 container remove 7cd67865c9ab2215318b780a03ba7562b86221ae5bd875a6535cfe82ec670815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mon-c, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:49:51.574 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:51 smithi157 systemd[1]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.c.service: Deactivated successfully. 2024-05-09T08:49:51.574 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:51 smithi157 systemd[1]: Stopped Ceph mon.c for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:49:51.574 INFO:journalctl@ceph.mon.c.smithi157.stdout:May 09 08:49:51 smithi157 systemd[1]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mon.c.service: Consumed 15.713s CPU time. 2024-05-09T08:49:51.606 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:51.606 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-09T08:49:51.607 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-09T08:49:51.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mgr.x 2024-05-09T08:49:51.939 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:49:51 smithi107 systemd[1]: Stopping Ceph mgr.x for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:49:52.279 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:49:51 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x[30348]: 2024-05-09T08:49:51.938+0000 7fe2d9d62700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:49:52.279 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:49:51 smithi107 podman[115456]: 2024-05-09 08:49:51.971395227 +0000 UTC m=+0.157113479 container died a3dc9c9b852ef2d6a256f19726fde4535b1a760f7a4e6e4bda481804cbb87452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:49:52.641 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:49:52 smithi107 podman[115456]: 2024-05-09 08:49:52.390272198 +0000 UTC m=+0.575990452 container cleanup a3dc9c9b852ef2d6a256f19726fde4535b1a760f7a4e6e4bda481804cbb87452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-09T08:49:52.641 INFO:journalctl@ceph.mgr.x.smithi107.stdout:May 09 08:49:52 smithi107 bash[115456]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-x 2024-05-09T08:49:52.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mgr.x.service' 2024-05-09T08:49:52.796 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:49:52.796 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-09T08:49:52.796 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-09T08:49:52.796 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mgr.y 2024-05-09T08:49:53.121 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:49:52 smithi110 systemd[1]: Stopping Ceph mgr.y for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:49:54.871 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:49:54 smithi110 podman[69930]: 2024-05-09 08:49:54.466830405 +0000 UTC m=+1.473743027 container died 4edfd29d7064e1bb1579856ce1bf9deebe63d502e0c4a154ad42f96502afd0ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.build-date=20240507) 2024-05-09T08:49:58.871 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:49:58 smithi110 podman[69930]: 2024-05-09 08:49:58.610950195 +0000 UTC m=+5.617862817 container cleanup 4edfd29d7064e1bb1579856ce1bf9deebe63d502e0c4a154ad42f96502afd0ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-96805dc) 2024-05-09T08:49:58.871 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:49:58 smithi110 bash[69930]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y 2024-05-09T08:50:00.647 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:50:00 smithi110 podman[69944]: 2024-05-09 08:50:00.554979062 +0000 UTC m=+5.933199363 container remove 4edfd29d7064e1bb1579856ce1bf9deebe63d502e0c4a154ad42f96502afd0ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-y, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-05-09T08:50:00.647 INFO:journalctl@ceph.mgr.y.smithi110.stdout:May 09 08:50:00 smithi110 systemd[1]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-05-09T08:50:00.661 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mgr.y.service' 2024-05-09T08:50:00.697 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:50:00.697 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-09T08:50:00.698 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-05-09T08:50:00.698 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mgr.z 2024-05-09T08:50:01.094 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:50:00 smithi157 systemd[1]: Stopping Ceph mgr.z for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:50:02.844 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:50:02 smithi157 podman[71336]: 2024-05-09 08:50:02.561808038 +0000 UTC m=+1.643629231 container died e1a8727f5c6ddc6a0b3290e089328b8b100f8a60eb7420658a0360da7287aa6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-96805dc, ceph=True) 2024-05-09T08:50:05.823 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:50:05 smithi157 podman[71336]: 2024-05-09 08:50:05.498613998 +0000 UTC m=+4.580435242 container cleanup e1a8727f5c6ddc6a0b3290e089328b8b100f8a60eb7420658a0360da7287aa6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-96805dc, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-09T08:50:05.823 INFO:journalctl@ceph.mgr.z.smithi157.stdout:May 09 08:50:05 smithi157 bash[71336]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mgr-z 2024-05-09T08:50:05.972 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mgr.z.service' 2024-05-09T08:50:06.010 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:50:06.010 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-05-09T08:50:06.010 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-09T08:50:06.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.0 2024-05-09T08:50:06.529 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:50:06 smithi107 systemd[1]: Stopping Ceph osd.0 for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:50:08.029 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:50:07 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-0[42154]: 2024-05-09T08:50:07.537+0000 7f82e490f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:50:08.029 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:50:07 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-0[42154]: 2024-05-09T08:50:07.537+0000 7f82e490f700 -1 osd.0 120 *** Got signal Terminated *** 2024-05-09T08:50:08.029 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:50:07 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-0[42154]: 2024-05-09T08:50:07.537+0000 7f82e490f700 -1 osd.0 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:50:13.485 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:50:13 smithi107 podman[115581]: 2024-05-09 08:50:13.186878464 +0000 UTC m=+6.980845346 container died 79add31993e694d314f7a584fea50fe61ae73f5a98b8f847ce95ab796e247b57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS) 2024-05-09T08:50:13.779 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:50:13 smithi107 podman[115581]: 2024-05-09 08:50:13.585983608 +0000 UTC m=+7.379950497 container cleanup 79add31993e694d314f7a584fea50fe61ae73f5a98b8f847ce95ab796e247b57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-0, RELEASE=squid-96805dc, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-09T08:50:13.779 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:50:13 smithi107 bash[115581]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-0 2024-05-09T08:50:14.202 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:50:13 smithi107 podman[115598]: 2024-05-09 08:50:13.879251411 +0000 UTC m=+0.675078617 container remove 79add31993e694d314f7a584fea50fe61ae73f5a98b8f847ce95ab796e247b57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:50:14.498 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:50:14 smithi107 podman[115688]: 2024-05-09 08:50:14.106314864 +0000 UTC m=+0.018123639 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:50:14.498 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:50:14 smithi107 podman[115688]: 2024-05-09 08:50:14.238507918 +0000 UTC m=+0.150316679 container create 3b40b2bf96e043532c8b343dd9e0284fc13174f271ed478a2ff0d63ba25bd737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507) 2024-05-09T08:50:14.498 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:50:14 smithi107 podman[115688]: 2024-05-09 08:50:14.499618245 +0000 UTC m=+0.411427006 container init 3b40b2bf96e043532c8b343dd9e0284fc13174f271ed478a2ff0d63ba25bd737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, ceph=True) 2024-05-09T08:50:14.779 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:50:14 smithi107 podman[115688]: 2024-05-09 08:50:14.502943613 +0000 UTC m=+0.414752375 container start 3b40b2bf96e043532c8b343dd9e0284fc13174f271ed478a2ff0d63ba25bd737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-0-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5) 2024-05-09T08:50:14.779 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:50:14 smithi107 podman[115688]: 2024-05-09 08:50:14.625003311 +0000 UTC m=+0.536812078 container attach 3b40b2bf96e043532c8b343dd9e0284fc13174f271ed478a2ff0d63ba25bd737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-09T08:50:15.322 INFO:journalctl@ceph.osd.0.smithi107.stdout:May 09 08:50:15 smithi107 podman[115688]: 2024-05-09 08:50:15.045887274 +0000 UTC m=+0.957696027 container died 3b40b2bf96e043532c8b343dd9e0284fc13174f271ed478a2ff0d63ba25bd737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=) 2024-05-09T08:50:15.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.0.service' 2024-05-09T08:50:15.713 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:50:15.714 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-09T08:50:15.714 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-09T08:50:15.714 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.1 2024-05-09T08:50:16.024 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:50:15 smithi110 systemd[1]: Stopping Ceph osd.1 for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:50:16.297 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:50:16 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-1[38212]: 2024-05-09T08:50:16.024+0000 7efd0f408700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:50:16.297 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:50:16 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-1[38212]: 2024-05-09T08:50:16.024+0000 7efd0f408700 -1 osd.1 120 *** Got signal Terminated *** 2024-05-09T08:50:16.297 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:50:16 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-1[38212]: 2024-05-09T08:50:16.024+0000 7efd0f408700 -1 osd.1 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:50:21.991 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:50:21 smithi110 podman[70057]: 2024-05-09 08:50:21.692890674 +0000 UTC m=+5.774817296 container died 051155b4c6ea730a40054293499f13ea231db5ca1053a84c421f63b80f9a60b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-1, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-09T08:50:22.250 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:50:21 smithi110 podman[70057]: 2024-05-09 08:50:21.991950811 +0000 UTC m=+6.073877438 container cleanup 051155b4c6ea730a40054293499f13ea231db5ca1053a84c421f63b80f9a60b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-1, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-09T08:50:22.251 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:50:21 smithi110 bash[70057]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-1 2024-05-09T08:50:22.585 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:50:22 smithi110 podman[70087]: 2024-05-09 08:50:22.25202822 +0000 UTC m=+0.553958857 container remove 051155b4c6ea730a40054293499f13ea231db5ca1053a84c421f63b80f9a60b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.build-date=20240507, GIT_CLEAN=True, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-09T08:50:22.871 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:50:22 smithi110 podman[70174]: 2024-05-09 08:50:22.586666114 +0000 UTC m=+0.094371515 container create 7b974a7dbb8bc7e89f71f516a352aecf5e8dfe08fa460ea904bccceb95454bf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-1-deactivate, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS) 2024-05-09T08:50:22.871 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:50:22 smithi110 podman[70174]: 2024-05-09 08:50:22.510552739 +0000 UTC m=+0.018258113 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:50:22.871 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:50:22 smithi110 podman[70174]: 2024-05-09 08:50:22.822926914 +0000 UTC m=+0.330632277 container init 7b974a7dbb8bc7e89f71f516a352aecf5e8dfe08fa460ea904bccceb95454bf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-96805dc) 2024-05-09T08:50:22.871 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:50:22 smithi110 podman[70174]: 2024-05-09 08:50:22.826454881 +0000 UTC m=+0.334160238 container start 7b974a7dbb8bc7e89f71f516a352aecf5e8dfe08fa460ea904bccceb95454bf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-09T08:50:23.370 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:50:22 smithi110 podman[70174]: 2024-05-09 08:50:22.914546896 +0000 UTC m=+0.422252271 container attach 7b974a7dbb8bc7e89f71f516a352aecf5e8dfe08fa460ea904bccceb95454bf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:50:23.692 INFO:journalctl@ceph.osd.1.smithi110.stdout:May 09 08:50:23 smithi110 podman[70174]: 2024-05-09 08:50:23.393378884 +0000 UTC m=+0.901084262 container died 7b974a7dbb8bc7e89f71f516a352aecf5e8dfe08fa460ea904bccceb95454bf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:50:23.997 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.1.service' 2024-05-09T08:50:24.036 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:50:24.037 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-09T08:50:24.037 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-09T08:50:24.037 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.2 2024-05-09T08:50:24.343 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:50:24 smithi157 systemd[1]: Stopping Ceph osd.2 for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:50:24.844 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:50:24 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-2[38759]: 2024-05-09T08:50:24.351+0000 7f6e24394700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:50:24.844 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:50:24 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-2[38759]: 2024-05-09T08:50:24.351+0000 7f6e24394700 -1 osd.2 120 *** Got signal Terminated *** 2024-05-09T08:50:24.844 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:50:24 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-2[38759]: 2024-05-09T08:50:24.351+0000 7f6e24394700 -1 osd.2 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:50:30.344 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:50:30 smithi157 podman[71460]: 2024-05-09 08:50:30.086656706 +0000 UTC m=+5.855879481 container died d91f6b187209c3ff2ffadd4ec5a6073b474c4d3bcc062ecac4ef3bdbcdf30581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-09T08:50:30.752 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:50:30 smithi157 podman[71460]: 2024-05-09 08:50:30.493810629 +0000 UTC m=+6.263033404 container cleanup d91f6b187209c3ff2ffadd4ec5a6073b474c4d3bcc062ecac4ef3bdbcdf30581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-96805dc, ceph=True, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-09T08:50:30.752 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:50:30 smithi157 bash[71460]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-2 2024-05-09T08:50:31.047 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:50:30 smithi157 podman[71490]: 2024-05-09 08:50:30.753670261 +0000 UTC m=+0.664473343 container remove d91f6b187209c3ff2ffadd4ec5a6073b474c4d3bcc062ecac4ef3bdbcdf30581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:50:31.297 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:50:31 smithi157 podman[71578]: 2024-05-09 08:50:30.95157681 +0000 UTC m=+0.018020568 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:50:31.297 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:50:31 smithi157 podman[71578]: 2024-05-09 08:50:31.062903604 +0000 UTC m=+0.129347361 container create 75146041322e1ab1a42abe79c0140427de0b8505f9cc4cb51ad9916d86f02fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-09T08:50:31.298 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:50:31 smithi157 podman[71578]: 2024-05-09 08:50:31.29872486 +0000 UTC m=+0.365168618 container init 75146041322e1ab1a42abe79c0140427de0b8505f9cc4cb51ad9916d86f02fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-09T08:50:31.593 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:50:31 smithi157 podman[71578]: 2024-05-09 08:50:31.302267155 +0000 UTC m=+0.368710913 container start 75146041322e1ab1a42abe79c0140427de0b8505f9cc4cb51ad9916d86f02fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-2-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, io.buildah.version=1.33.5) 2024-05-09T08:50:31.594 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:50:31 smithi157 podman[71578]: 2024-05-09 08:50:31.432275973 +0000 UTC m=+0.498719739 container attach 75146041322e1ab1a42abe79c0140427de0b8505f9cc4cb51ad9916d86f02fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-2-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:50:32.034 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:50:31 smithi157 podman[71578]: 2024-05-09 08:50:31.673298384 +0000 UTC m=+0.739742140 container died 75146041322e1ab1a42abe79c0140427de0b8505f9cc4cb51ad9916d86f02fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_CLEAN=True) 2024-05-09T08:50:32.291 INFO:journalctl@ceph.osd.2.smithi157.stdout:May 09 08:50:32 smithi157 podman[71604]: 2024-05-09 08:50:32.278301519 +0000 UTC m=+0.598904573 container remove 75146041322e1ab1a42abe79c0140427de0b8505f9cc4cb51ad9916d86f02fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-2-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-09T08:50:32.297 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.2.service' 2024-05-09T08:50:32.340 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:50:32.340 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-09T08:50:32.340 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-09T08:50:32.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.3 2024-05-09T08:50:32.655 INFO:journalctl@ceph.osd.3.smithi107.stdout:May 09 08:50:32 smithi107 systemd[1]: Stopping Ceph osd.3 for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:50:33.029 INFO:journalctl@ceph.osd.3.smithi107.stdout:May 09 08:50:32 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-3[48973]: 2024-05-09T08:50:32.655+0000 7f7cf0651700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:50:33.029 INFO:journalctl@ceph.osd.3.smithi107.stdout:May 09 08:50:32 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-3[48973]: 2024-05-09T08:50:32.655+0000 7f7cf0651700 -1 osd.3 120 *** Got signal Terminated *** 2024-05-09T08:50:33.029 INFO:journalctl@ceph.osd.3.smithi107.stdout:May 09 08:50:32 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-3[48973]: 2024-05-09T08:50:32.655+0000 7f7cf0651700 -1 osd.3 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:50:38.756 INFO:journalctl@ceph.osd.3.smithi107.stdout:May 09 08:50:38 smithi107 podman[115809]: 2024-05-09 08:50:38.425229372 +0000 UTC m=+5.890884336 container died 7b72a0dfa6d25fff6593d4b3162099e508f741969ac7aa24acb474671c96e54f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-3, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-09T08:50:39.029 INFO:journalctl@ceph.osd.3.smithi107.stdout:May 09 08:50:38 smithi107 podman[115809]: 2024-05-09 08:50:38.846965024 +0000 UTC m=+6.312620003 container cleanup 7b72a0dfa6d25fff6593d4b3162099e508f741969ac7aa24acb474671c96e54f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-3, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-09T08:50:39.029 INFO:journalctl@ceph.osd.3.smithi107.stdout:May 09 08:50:38 smithi107 bash[115809]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-3 2024-05-09T08:50:39.502 INFO:journalctl@ceph.osd.3.smithi107.stdout:May 09 08:50:39 smithi107 podman[115828]: 2024-05-09 08:50:39.09849981 +0000 UTC m=+0.668575947 container remove 7b72a0dfa6d25fff6593d4b3162099e508f741969ac7aa24acb474671c96e54f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-3, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:50:39.779 INFO:journalctl@ceph.osd.3.smithi107.stdout:May 09 08:50:39 smithi107 podman[115919]: 2024-05-09 08:50:39.406413843 +0000 UTC m=+0.017957286 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:50:39.779 INFO:journalctl@ceph.osd.3.smithi107.stdout:May 09 08:50:39 smithi107 podman[115919]: 2024-05-09 08:50:39.524237589 +0000 UTC m=+0.135781017 container create 4a37da67851e2281053c377df0810fc205ff48f333bb84b5b66251cb1f7f4805 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, ceph=True, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=) 2024-05-09T08:50:40.083 INFO:journalctl@ceph.osd.3.smithi107.stdout:May 09 08:50:39 smithi107 podman[115919]: 2024-05-09 08:50:39.785094278 +0000 UTC m=+0.396637706 container init 4a37da67851e2281053c377df0810fc205ff48f333bb84b5b66251cb1f7f4805 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, io.buildah.version=1.33.5) 2024-05-09T08:50:40.083 INFO:journalctl@ceph.osd.3.smithi107.stdout:May 09 08:50:39 smithi107 podman[115919]: 2024-05-09 08:50:39.788393551 +0000 UTC m=+0.399936977 container start 4a37da67851e2281053c377df0810fc205ff48f333bb84b5b66251cb1f7f4805 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:50:40.083 INFO:journalctl@ceph.osd.3.smithi107.stdout:May 09 08:50:39 smithi107 podman[115919]: 2024-05-09 08:50:39.902088479 +0000 UTC m=+0.513631955 container attach 4a37da67851e2281053c377df0810fc205ff48f333bb84b5b66251cb1f7f4805 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-3-deactivate, RELEASE=squid-96805dc, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507) 2024-05-09T08:50:40.446 INFO:journalctl@ceph.osd.3.smithi107.stdout:May 09 08:50:40 smithi107 podman[115919]: 2024-05-09 08:50:40.085344276 +0000 UTC m=+0.696887710 container died 4a37da67851e2281053c377df0810fc205ff48f333bb84b5b66251cb1f7f4805 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-3-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, io.buildah.version=1.33.5) 2024-05-09T08:50:40.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.3.service' 2024-05-09T08:50:40.801 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:50:40.801 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-09T08:50:40.801 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-09T08:50:40.802 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.4 2024-05-09T08:50:41.097 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:50:40 smithi110 systemd[1]: Stopping Ceph osd.4 for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:50:41.371 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:50:41 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-4[44837]: 2024-05-09T08:50:41.097+0000 7f822341a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:50:41.371 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:50:41 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-4[44837]: 2024-05-09T08:50:41.097+0000 7f822341a700 -1 osd.4 120 *** Got signal Terminated *** 2024-05-09T08:50:41.371 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:50:41 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-4[44837]: 2024-05-09T08:50:41.097+0000 7f822341a700 -1 osd.4 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:50:47.006 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:50:46 smithi110 podman[70280]: 2024-05-09 08:50:46.694103737 +0000 UTC m=+5.693186796 container died 5fa7ede6cd9856c44140d7c9ed759a65bfa0a8638f54cb3bc68ba8b3d2c8c453 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-4, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-05-09T08:50:47.348 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:50:47 smithi110 podman[70280]: 2024-05-09 08:50:47.106655112 +0000 UTC m=+6.105738171 container cleanup 5fa7ede6cd9856c44140d7c9ed759a65bfa0a8638f54cb3bc68ba8b3d2c8c453 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-4, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507) 2024-05-09T08:50:47.348 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:50:47 smithi110 bash[70280]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-4 2024-05-09T08:50:47.621 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:50:47 smithi110 podman[70298]: 2024-05-09 08:50:47.349909181 +0000 UTC m=+0.655730716 container remove 5fa7ede6cd9856c44140d7c9ed759a65bfa0a8638f54cb3bc68ba8b3d2c8c453 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-4, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0) 2024-05-09T08:50:48.072 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:50:47 smithi110 podman[70397]: 2024-05-09 08:50:47.720526631 +0000 UTC m=+0.018410384 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:50:48.072 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:50:47 smithi110 podman[70397]: 2024-05-09 08:50:47.842884581 +0000 UTC m=+0.140768331 container create 9e300ea87fdfab9e962bfe7f4effe1ae4dbbd52761bf2d065bd8836e8ec0950c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux ) 2024-05-09T08:50:48.371 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:50:48 smithi110 podman[70397]: 2024-05-09 08:50:48.087408617 +0000 UTC m=+0.385292362 container init 9e300ea87fdfab9e962bfe7f4effe1ae4dbbd52761bf2d065bd8836e8ec0950c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-09T08:50:48.371 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:50:48 smithi110 podman[70397]: 2024-05-09 08:50:48.09106396 +0000 UTC m=+0.388947712 container start 9e300ea87fdfab9e962bfe7f4effe1ae4dbbd52761bf2d065bd8836e8ec0950c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, ceph=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=) 2024-05-09T08:50:48.371 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:50:48 smithi110 podman[70397]: 2024-05-09 08:50:48.154108076 +0000 UTC m=+0.451991827 container attach 9e300ea87fdfab9e962bfe7f4effe1ae4dbbd52761bf2d065bd8836e8ec0950c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-4-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-09T08:50:48.664 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:50:48 smithi110 podman[70397]: 2024-05-09 08:50:48.383455129 +0000 UTC m=+0.681338863 container died 9e300ea87fdfab9e962bfe7f4effe1ae4dbbd52761bf2d065bd8836e8ec0950c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.build-date=20240507) 2024-05-09T08:50:48.922 INFO:journalctl@ceph.osd.4.smithi110.stdout:May 09 08:50:48 smithi110 podman[70423]: 2024-05-09 08:50:48.908051762 +0000 UTC m=+0.519308176 container remove 9e300ea87fdfab9e962bfe7f4effe1ae4dbbd52761bf2d065bd8836e8ec0950c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True) 2024-05-09T08:50:48.928 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.4.service' 2024-05-09T08:50:48.968 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:50:48.968 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-09T08:50:48.968 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-09T08:50:48.968 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.5 2024-05-09T08:50:49.296 INFO:journalctl@ceph.osd.5.smithi157.stdout:May 09 08:50:49 smithi157 systemd[1]: Stopping Ceph osd.5 for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:50:49.593 INFO:journalctl@ceph.osd.5.smithi157.stdout:May 09 08:50:49 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-5[45403]: 2024-05-09T08:50:49.295+0000 7fa9f7c46700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:50:49.593 INFO:journalctl@ceph.osd.5.smithi157.stdout:May 09 08:50:49 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-5[45403]: 2024-05-09T08:50:49.295+0000 7fa9f7c46700 -1 osd.5 120 *** Got signal Terminated *** 2024-05-09T08:50:49.593 INFO:journalctl@ceph.osd.5.smithi157.stdout:May 09 08:50:49 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-5[45403]: 2024-05-09T08:50:49.295+0000 7fa9f7c46700 -1 osd.5 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:50:55.437 INFO:journalctl@ceph.osd.5.smithi157.stdout:May 09 08:50:55 smithi157 podman[71684]: 2024-05-09 08:50:55.129296584 +0000 UTC m=+5.964549529 container died 9ba4a45ed654439e562dedc2b08b7c0335c59f67020cbc3acb619470bfe1b3b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-5, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240507) 2024-05-09T08:50:55.754 INFO:journalctl@ceph.osd.5.smithi157.stdout:May 09 08:50:55 smithi157 podman[71684]: 2024-05-09 08:50:55.537915602 +0000 UTC m=+6.373168549 container cleanup 9ba4a45ed654439e562dedc2b08b7c0335c59f67020cbc3acb619470bfe1b3b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-5, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2) 2024-05-09T08:50:55.755 INFO:journalctl@ceph.osd.5.smithi157.stdout:May 09 08:50:55 smithi157 bash[71684]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-5 2024-05-09T08:50:56.093 INFO:journalctl@ceph.osd.5.smithi157.stdout:May 09 08:50:55 smithi157 podman[71701]: 2024-05-09 08:50:55.756033459 +0000 UTC m=+0.621320738 container remove 9ba4a45ed654439e562dedc2b08b7c0335c59f67020cbc3acb619470bfe1b3b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-5, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-09T08:50:56.458 INFO:journalctl@ceph.osd.5.smithi157.stdout:May 09 08:50:56 smithi157 podman[71788]: 2024-05-09 08:50:56.102884608 +0000 UTC m=+0.018267965 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:50:56.458 INFO:journalctl@ceph.osd.5.smithi157.stdout:May 09 08:50:56 smithi157 podman[71788]: 2024-05-09 08:50:56.223641235 +0000 UTC m=+0.139024575 container create 548f8fd84b626d761ae54d0d545099f136beb59111a6b7599b7b15c75a524dcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507) 2024-05-09T08:50:56.759 INFO:journalctl@ceph.osd.5.smithi157.stdout:May 09 08:50:56 smithi157 podman[71788]: 2024-05-09 08:50:56.459259675 +0000 UTC m=+0.374643012 container init 548f8fd84b626d761ae54d0d545099f136beb59111a6b7599b7b15c75a524dcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-09T08:50:56.759 INFO:journalctl@ceph.osd.5.smithi157.stdout:May 09 08:50:56 smithi157 podman[71788]: 2024-05-09 08:50:56.462697592 +0000 UTC m=+0.378080938 container start 548f8fd84b626d761ae54d0d545099f136beb59111a6b7599b7b15c75a524dcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, ceph=True, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-09T08:50:56.759 INFO:journalctl@ceph.osd.5.smithi157.stdout:May 09 08:50:56 smithi157 podman[71788]: 2024-05-09 08:50:56.559388072 +0000 UTC m=+0.474771424 container attach 548f8fd84b626d761ae54d0d545099f136beb59111a6b7599b7b15c75a524dcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507) 2024-05-09T08:50:57.061 INFO:journalctl@ceph.osd.5.smithi157.stdout:May 09 08:50:56 smithi157 podman[71788]: 2024-05-09 08:50:56.760510369 +0000 UTC m=+0.675893708 container died 548f8fd84b626d761ae54d0d545099f136beb59111a6b7599b7b15c75a524dcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=) 2024-05-09T08:50:57.400 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.5.service' 2024-05-09T08:50:57.443 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:50:57.443 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-09T08:50:57.443 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-09T08:50:57.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.6 2024-05-09T08:50:57.764 INFO:journalctl@ceph.osd.6.smithi107.stdout:May 09 08:50:57 smithi107 systemd[1]: Stopping Ceph osd.6 for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:50:58.029 INFO:journalctl@ceph.osd.6.smithi107.stdout:May 09 08:50:57 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-6[55944]: 2024-05-09T08:50:57.763+0000 7f75d5304700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:50:58.029 INFO:journalctl@ceph.osd.6.smithi107.stdout:May 09 08:50:57 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-6[55944]: 2024-05-09T08:50:57.763+0000 7f75d5304700 -1 osd.6 120 *** Got signal Terminated *** 2024-05-09T08:50:58.029 INFO:journalctl@ceph.osd.6.smithi107.stdout:May 09 08:50:57 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-6[55944]: 2024-05-09T08:50:57.763+0000 7f75d5304700 -1 osd.6 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:51:03.714 INFO:journalctl@ceph.osd.6.smithi107.stdout:May 09 08:51:03 smithi107 podman[116025]: 2024-05-09 08:51:03.419974551 +0000 UTC m=+5.779558739 container died 966083d185f1375e26d91e667fe7c6fa81ece61a870d454db2dffaa7bd2b70f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-6, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-09T08:51:04.029 INFO:journalctl@ceph.osd.6.smithi107.stdout:May 09 08:51:03 smithi107 podman[116025]: 2024-05-09 08:51:03.806021639 +0000 UTC m=+6.165605824 container cleanup 966083d185f1375e26d91e667fe7c6fa81ece61a870d454db2dffaa7bd2b70f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-6, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-09T08:51:04.029 INFO:journalctl@ceph.osd.6.smithi107.stdout:May 09 08:51:03 smithi107 bash[116025]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-6 2024-05-09T08:51:04.406 INFO:journalctl@ceph.osd.6.smithi107.stdout:May 09 08:51:04 smithi107 podman[116054]: 2024-05-09 08:51:04.099374724 +0000 UTC m=+0.673911629 container remove 966083d185f1375e26d91e667fe7c6fa81ece61a870d454db2dffaa7bd2b70f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-6, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-09T08:51:04.684 INFO:journalctl@ceph.osd.6.smithi107.stdout:May 09 08:51:04 smithi107 podman[116142]: 2024-05-09 08:51:04.310868384 +0000 UTC m=+0.017212504 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:51:04.684 INFO:journalctl@ceph.osd.6.smithi107.stdout:May 09 08:51:04 smithi107 podman[116142]: 2024-05-09 08:51:04.433682505 +0000 UTC m=+0.140026648 container create 11eed162194a61326d4a72727f8deb82ea475011257dcb2f8623106031bc683f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:51:04.684 INFO:journalctl@ceph.osd.6.smithi107.stdout:May 09 08:51:04 smithi107 podman[116142]: 2024-05-09 08:51:04.685986302 +0000 UTC m=+0.392330410 container init 11eed162194a61326d4a72727f8deb82ea475011257dcb2f8623106031bc683f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:51:04.976 INFO:journalctl@ceph.osd.6.smithi107.stdout:May 09 08:51:04 smithi107 podman[116142]: 2024-05-09 08:51:04.689433009 +0000 UTC m=+0.395777109 container start 11eed162194a61326d4a72727f8deb82ea475011257dcb2f8623106031bc683f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:51:04.976 INFO:journalctl@ceph.osd.6.smithi107.stdout:May 09 08:51:04 smithi107 podman[116142]: 2024-05-09 08:51:04.77757372 +0000 UTC m=+0.483917829 container attach 11eed162194a61326d4a72727f8deb82ea475011257dcb2f8623106031bc683f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-09T08:51:05.279 INFO:journalctl@ceph.osd.6.smithi107.stdout:May 09 08:51:04 smithi107 podman[116142]: 2024-05-09 08:51:04.977693629 +0000 UTC m=+0.684037733 container died 11eed162194a61326d4a72727f8deb82ea475011257dcb2f8623106031bc683f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, ceph=True, org.label-schema.build-date=20240507) 2024-05-09T08:51:05.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.6.service' 2024-05-09T08:51:05.687 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:51:05.687 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-09T08:51:05.687 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-09T08:51:05.687 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.7 2024-05-09T08:51:05.985 INFO:journalctl@ceph.osd.7.smithi110.stdout:May 09 08:51:05 smithi110 systemd[1]: Stopping Ceph osd.7 for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:51:06.371 INFO:journalctl@ceph.osd.7.smithi110.stdout:May 09 08:51:05 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-7[51809]: 2024-05-09T08:51:05.985+0000 7fa3292f4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:51:06.371 INFO:journalctl@ceph.osd.7.smithi110.stdout:May 09 08:51:05 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-7[51809]: 2024-05-09T08:51:05.985+0000 7fa3292f4700 -1 osd.7 120 *** Got signal Terminated *** 2024-05-09T08:51:06.371 INFO:journalctl@ceph.osd.7.smithi110.stdout:May 09 08:51:05 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-7[51809]: 2024-05-09T08:51:05.985+0000 7fa3292f4700 -1 osd.7 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:51:11.871 INFO:journalctl@ceph.osd.7.smithi110.stdout:May 09 08:51:11 smithi110 podman[70503]: 2024-05-09 08:51:11.602259214 +0000 UTC m=+5.724961324 container died 225cf89c273230c9c2f151781c35f0b0f0bf143808b834d16faf996d8834308c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-09T08:51:12.244 INFO:journalctl@ceph.osd.7.smithi110.stdout:May 09 08:51:11 smithi110 podman[70503]: 2024-05-09 08:51:11.978001135 +0000 UTC m=+6.100703252 container cleanup 225cf89c273230c9c2f151781c35f0b0f0bf143808b834d16faf996d8834308c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-7, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-09T08:51:12.245 INFO:journalctl@ceph.osd.7.smithi110.stdout:May 09 08:51:11 smithi110 bash[70503]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-7 2024-05-09T08:51:12.537 INFO:journalctl@ceph.osd.7.smithi110.stdout:May 09 08:51:12 smithi110 podman[70520]: 2024-05-09 08:51:12.246225194 +0000 UTC m=+0.647746474 container remove 225cf89c273230c9c2f151781c35f0b0f0bf143808b834d16faf996d8834308c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240507) 2024-05-09T08:51:12.799 INFO:journalctl@ceph.osd.7.smithi110.stdout:May 09 08:51:12 smithi110 podman[70612]: 2024-05-09 08:51:12.538720293 +0000 UTC m=+0.108866338 container create 29106896d63e76cbf1a37edd7a03ff1f2c5ba79f606868f0bcb2d8893b3081fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:51:12.799 INFO:journalctl@ceph.osd.7.smithi110.stdout:May 09 08:51:12 smithi110 podman[70612]: 2024-05-09 08:51:12.447780873 +0000 UTC m=+0.017926932 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:51:12.799 INFO:journalctl@ceph.osd.7.smithi110.stdout:May 09 08:51:12 smithi110 podman[70612]: 2024-05-09 08:51:12.800464698 +0000 UTC m=+0.370610755 container init 29106896d63e76cbf1a37edd7a03ff1f2c5ba79f606868f0bcb2d8893b3081fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:51:13.092 INFO:journalctl@ceph.osd.7.smithi110.stdout:May 09 08:51:12 smithi110 podman[70612]: 2024-05-09 08:51:12.803841866 +0000 UTC m=+0.373987922 container start 29106896d63e76cbf1a37edd7a03ff1f2c5ba79f606868f0bcb2d8893b3081fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-7-deactivate, RELEASE=squid-96805dc, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-05-09T08:51:13.092 INFO:journalctl@ceph.osd.7.smithi110.stdout:May 09 08:51:12 smithi110 podman[70612]: 2024-05-09 08:51:12.884407807 +0000 UTC m=+0.454553862 container attach 29106896d63e76cbf1a37edd7a03ff1f2c5ba79f606868f0bcb2d8893b3081fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-09T08:51:13.363 INFO:journalctl@ceph.osd.7.smithi110.stdout:May 09 08:51:13 smithi110 podman[70612]: 2024-05-09 08:51:13.093582544 +0000 UTC m=+0.663728597 container died 29106896d63e76cbf1a37edd7a03ff1f2c5ba79f606868f0bcb2d8893b3081fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-09T08:51:13.715 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.7.service' 2024-05-09T08:51:13.753 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:51:13.753 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-09T08:51:13.753 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-09T08:51:13.753 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.8 2024-05-09T08:51:14.056 INFO:journalctl@ceph.osd.8.smithi157.stdout:May 09 08:51:13 smithi157 systemd[1]: Stopping Ceph osd.8 for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:51:14.344 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:14 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:14.242+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:14.344 INFO:journalctl@ceph.osd.8.smithi157.stdout:May 09 08:51:14 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-8[52376]: 2024-05-09T08:51:14.056+0000 7f50cad0b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:51:14.344 INFO:journalctl@ceph.osd.8.smithi157.stdout:May 09 08:51:14 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-8[52376]: 2024-05-09T08:51:14.056+0000 7f50cad0b700 -1 osd.8 120 *** Got signal Terminated *** 2024-05-09T08:51:14.344 INFO:journalctl@ceph.osd.8.smithi157.stdout:May 09 08:51:14 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-8[52376]: 2024-05-09T08:51:14.056+0000 7f50cad0b700 -1 osd.8 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:51:14.529 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:14 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:14.139+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:10.303026+0000 front 2024-05-09T08:50:10.302842+0000 (oldest deadline 2024-05-09T08:51:13.203005+0000) 2024-05-09T08:51:15.529 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:15 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:15.117+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:10.303026+0000 front 2024-05-09T08:50:10.302842+0000 (oldest deadline 2024-05-09T08:51:13.203005+0000) 2024-05-09T08:51:15.593 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:15 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:15.237+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:16.529 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:16 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:16.163+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:10.303026+0000 front 2024-05-09T08:50:10.302842+0000 (oldest deadline 2024-05-09T08:51:13.203005+0000) 2024-05-09T08:51:16.593 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:16 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:16.214+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:17.529 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:17 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:17.145+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:10.303026+0000 front 2024-05-09T08:50:10.302842+0000 (oldest deadline 2024-05-09T08:51:13.203005+0000) 2024-05-09T08:51:17.593 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:17 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:17.168+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:18.529 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:18 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:18.187+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:10.303026+0000 front 2024-05-09T08:50:10.302842+0000 (oldest deadline 2024-05-09T08:51:13.203005+0000) 2024-05-09T08:51:18.593 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:18 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:18.165+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:18.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:18 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:18.167+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:19.529 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:19 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:19.159+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:10.303026+0000 front 2024-05-09T08:50:10.302842+0000 (oldest deadline 2024-05-09T08:51:13.203005+0000) 2024-05-09T08:51:19.593 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:19 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:19.116+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:19.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:19 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:19.128+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:19.931 INFO:journalctl@ceph.osd.8.smithi157.stdout:May 09 08:51:19 smithi157 podman[71905]: 2024-05-09 08:51:19.656628585 +0000 UTC m=+5.699181482 container died 6de81f2e9117fb0607f3fc5a5e87c801e97ee551e2a88ae1db049b57f6427b1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-96805dc) 2024-05-09T08:51:20.314 INFO:journalctl@ceph.osd.8.smithi157.stdout:May 09 08:51:20 smithi157 podman[71905]: 2024-05-09 08:51:20.031328014 +0000 UTC m=+6.073880920 container cleanup 6de81f2e9117fb0607f3fc5a5e87c801e97ee551e2a88ae1db049b57f6427b1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-8, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc) 2024-05-09T08:51:20.315 INFO:journalctl@ceph.osd.8.smithi157.stdout:May 09 08:51:20 smithi157 bash[71905]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-8 2024-05-09T08:51:20.315 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:20 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:20.077+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:20.529 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:20 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:20.149+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:10.303026+0000 front 2024-05-09T08:50:10.302842+0000 (oldest deadline 2024-05-09T08:51:13.203005+0000) 2024-05-09T08:51:20.593 INFO:journalctl@ceph.osd.8.smithi157.stdout:May 09 08:51:20 smithi157 podman[71922]: 2024-05-09 08:51:20.315793876 +0000 UTC m=+0.660542607 container remove 6de81f2e9117fb0607f3fc5a5e87c801e97ee551e2a88ae1db049b57f6427b1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-8, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD) 2024-05-09T08:51:20.620 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:20 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:20.168+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:20.934 INFO:journalctl@ceph.osd.8.smithi157.stdout:May 09 08:51:20 smithi157 podman[72009]: 2024-05-09 08:51:20.552638109 +0000 UTC m=+0.018854466 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:51:20.934 INFO:journalctl@ceph.osd.8.smithi157.stdout:May 09 08:51:20 smithi157 podman[72009]: 2024-05-09 08:51:20.683411747 +0000 UTC m=+0.149628128 container create 87995739ef2729df220aa5126e5414f638dc199da603449d65e1338b2039d639 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-05-09T08:51:20.935 INFO:journalctl@ceph.osd.8.smithi157.stdout:May 09 08:51:20 smithi157 podman[72009]: 2024-05-09 08:51:20.936111389 +0000 UTC m=+0.402327722 container init 87995739ef2729df220aa5126e5414f638dc199da603449d65e1338b2039d639 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-09T08:51:21.239 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:21 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:21.082+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:21.239 INFO:journalctl@ceph.osd.8.smithi157.stdout:May 09 08:51:20 smithi157 podman[72009]: 2024-05-09 08:51:20.939695122 +0000 UTC m=+0.405911453 container start 87995739ef2729df220aa5126e5414f638dc199da603449d65e1338b2039d639 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0) 2024-05-09T08:51:21.239 INFO:journalctl@ceph.osd.8.smithi157.stdout:May 09 08:51:21 smithi157 podman[72009]: 2024-05-09 08:51:21.019317186 +0000 UTC m=+0.485533512 container attach 87995739ef2729df220aa5126e5414f638dc199da603449d65e1338b2039d639 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-8-deactivate, RELEASE=squid-96805dc, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-09T08:51:21.518 INFO:journalctl@ceph.osd.8.smithi157.stdout:May 09 08:51:21 smithi157 podman[72009]: 2024-05-09 08:51:21.240712055 +0000 UTC m=+0.706928392 container died 87995739ef2729df220aa5126e5414f638dc199da603449d65e1338b2039d639 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:51:21.528 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:21 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:21.118+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:10.303026+0000 front 2024-05-09T08:50:10.302842+0000 (oldest deadline 2024-05-09T08:51:13.203005+0000) 2024-05-09T08:51:21.620 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:21 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:21.156+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:21.890 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.8.service' 2024-05-09T08:51:21.930 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:51:21.930 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-09T08:51:21.930 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-09T08:51:21.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.9 2024-05-09T08:51:22.248 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:21 smithi107 systemd[1]: Stopping Ceph osd.9 for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:51:22.248 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:22 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:22.136+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:10.303026+0000 front 2024-05-09T08:50:10.302842+0000 (oldest deadline 2024-05-09T08:51:13.203005+0000) 2024-05-09T08:51:22.248 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:22 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:22.248+0000 7f7bded52700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:51:22.248 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:22 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:22.248+0000 7f7bded52700 -1 osd.9 120 *** Got signal Terminated *** 2024-05-09T08:51:22.248 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:22 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:22.248+0000 7f7bded52700 -1 osd.9 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:51:22.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:22 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:22.093+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:22.620 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:22 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:22.177+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:23.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:23 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:23.089+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:23.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:23 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:23.089+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:23.529 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:23 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:23.134+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:10.303026+0000 front 2024-05-09T08:50:10.302842+0000 (oldest deadline 2024-05-09T08:51:13.203005+0000) 2024-05-09T08:51:23.620 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:23 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:23.184+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:24.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:24 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:24.047+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:24.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:24 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:24.047+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:24.529 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:24 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:24.133+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:10.303026+0000 front 2024-05-09T08:50:10.302842+0000 (oldest deadline 2024-05-09T08:51:13.203005+0000) 2024-05-09T08:51:24.529 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:24 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:24.133+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.604887+0000 front 2024-05-09T08:50:19.604863+0000 (oldest deadline 2024-05-09T08:51:23.704910+0000) 2024-05-09T08:51:24.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:24 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:24.193+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:25.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:25 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:25.038+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:25.344 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:25 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:25.038+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:25.529 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:25 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:25.150+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:10.303026+0000 front 2024-05-09T08:50:10.302842+0000 (oldest deadline 2024-05-09T08:51:13.203005+0000) 2024-05-09T08:51:25.529 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:25 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:25.150+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.604887+0000 front 2024-05-09T08:50:19.604863+0000 (oldest deadline 2024-05-09T08:51:23.704910+0000) 2024-05-09T08:51:25.620 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:25 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:25.216+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:26.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:26 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:26.003+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:26.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:26 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:26.003+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:26.529 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:26 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:26.110+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:10.303026+0000 front 2024-05-09T08:50:10.302842+0000 (oldest deadline 2024-05-09T08:51:13.203005+0000) 2024-05-09T08:51:26.529 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:26 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:26.110+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.604887+0000 front 2024-05-09T08:50:19.604863+0000 (oldest deadline 2024-05-09T08:51:23.704910+0000) 2024-05-09T08:51:26.620 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:26 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:26.200+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:26.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:26 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:26.200+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:20.667624+0000 front 2024-05-09T08:50:20.667682+0000 (oldest deadline 2024-05-09T08:51:25.367831+0000) 2024-05-09T08:51:27.346 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:26 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:26.964+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:27.346 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:26 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:26.964+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:27.430 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:27 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:27.141+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:10.303026+0000 front 2024-05-09T08:50:10.302842+0000 (oldest deadline 2024-05-09T08:51:13.203005+0000) 2024-05-09T08:51:27.430 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:27 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9[62756]: 2024-05-09T08:51:27.141+0000 7f7bdb346700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.604887+0000 front 2024-05-09T08:50:19.604863+0000 (oldest deadline 2024-05-09T08:51:23.704910+0000) 2024-05-09T08:51:27.620 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:27 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:27.213+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:27.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:27 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:27.213+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:20.667624+0000 front 2024-05-09T08:50:20.667682+0000 (oldest deadline 2024-05-09T08:51:25.367831+0000) 2024-05-09T08:51:28.344 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:28 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:28.008+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:28.344 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:28 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:28.008+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:28.406 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:28 smithi107 podman[116249]: 2024-05-09 08:51:28.10528388 +0000 UTC m=+5.973516687 container died ae8968d530d6f698ca35408d94a39ea60388717ab349d862e634a6b70ed48c42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-09T08:51:28.620 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:28 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:28.207+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:28.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:28 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:28.207+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:20.667624+0000 front 2024-05-09T08:50:20.667682+0000 (oldest deadline 2024-05-09T08:51:25.367831+0000) 2024-05-09T08:51:28.731 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:28 smithi107 podman[116249]: 2024-05-09 08:51:28.50615524 +0000 UTC m=+6.374388048 container cleanup ae8968d530d6f698ca35408d94a39ea60388717ab349d862e634a6b70ed48c42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9, org.label-schema.build-date=20240507, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:51:28.731 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:28 smithi107 bash[116249]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9 2024-05-09T08:51:29.029 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:28 smithi107 podman[116266]: 2024-05-09 08:51:28.732270705 +0000 UTC m=+0.626578216 container remove ae8968d530d6f698ca35408d94a39ea60388717ab349d862e634a6b70ed48c42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507) 2024-05-09T08:51:29.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:29 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:29.028+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:29.344 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:29 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:29.028+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:29.359 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:29 smithi107 podman[116359]: 2024-05-09 08:51:28.950282087 +0000 UTC m=+0.018181933 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:51:29.359 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:29 smithi107 podman[116359]: 2024-05-09 08:51:29.074795458 +0000 UTC m=+0.142695285 container create b9a94f543ff6ba210599b6c904394f7777a6fcfd6459123cadee2a6886c23f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-09T08:51:29.359 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:29 smithi107 podman[116359]: 2024-05-09 08:51:29.36089138 +0000 UTC m=+0.428791205 container init b9a94f543ff6ba210599b6c904394f7777a6fcfd6459123cadee2a6886c23f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-09T08:51:29.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:29 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:29.211+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:29.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:29 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:29.211+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:20.667624+0000 front 2024-05-09T08:50:20.667682+0000 (oldest deadline 2024-05-09T08:51:25.367831+0000) 2024-05-09T08:51:29.650 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:29 smithi107 podman[116359]: 2024-05-09 08:51:29.364677253 +0000 UTC m=+0.432577075 container start b9a94f543ff6ba210599b6c904394f7777a6fcfd6459123cadee2a6886c23f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9-deactivate, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-09T08:51:29.651 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:29 smithi107 podman[116359]: 2024-05-09 08:51:29.435729812 +0000 UTC m=+0.503629631 container attach b9a94f543ff6ba210599b6c904394f7777a6fcfd6459123cadee2a6886c23f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-05-09T08:51:29.936 INFO:journalctl@ceph.osd.9.smithi107.stdout:May 09 08:51:29 smithi107 podman[116359]: 2024-05-09 08:51:29.652513853 +0000 UTC m=+0.720413674 container died b9a94f543ff6ba210599b6c904394f7777a6fcfd6459123cadee2a6886c23f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-09T08:51:30.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.9.service' 2024-05-09T08:51:30.251 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:51:30.251 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-09T08:51:30.251 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-09T08:51:30.251 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.10 2024-05-09T08:51:30.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:30 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:30.045+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:30.344 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:30 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:30.045+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:30.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:30 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:30.238+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:30.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:30 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:30.238+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:20.667624+0000 front 2024-05-09T08:50:20.667682+0000 (oldest deadline 2024-05-09T08:51:25.367831+0000) 2024-05-09T08:51:30.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:30 smithi110 systemd[1]: Stopping Ceph osd.10 for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:51:31.121 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:30 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:30.700+0000 7f75b9cbf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:51:31.121 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:30 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:30.700+0000 7f75b9cbf700 -1 osd.10 120 *** Got signal Terminated *** 2024-05-09T08:51:31.121 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:30 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:30.700+0000 7f75b9cbf700 -1 osd.10 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:51:31.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:31 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:31.040+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:31.344 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:31 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:31.040+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:31.344 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:31 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:31.040+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.157:6804 osd.2 since back 2024-05-09T08:50:27.460318+0000 front 2024-05-09T08:50:27.460393+0000 (oldest deadline 2024-05-09T08:51:30.360308+0000) 2024-05-09T08:51:31.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:31 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:31.206+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:31.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:31 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:31.206+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:20.667624+0000 front 2024-05-09T08:50:20.667682+0000 (oldest deadline 2024-05-09T08:51:25.367831+0000) 2024-05-09T08:51:32.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:32 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:32.041+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:32.344 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:32 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:32.041+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:32.344 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:32 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:32.041+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.157:6804 osd.2 since back 2024-05-09T08:50:27.460318+0000 front 2024-05-09T08:50:27.460393+0000 (oldest deadline 2024-05-09T08:51:30.360308+0000) 2024-05-09T08:51:32.620 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:32 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:32.221+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:32.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:32 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:32.221+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:20.667624+0000 front 2024-05-09T08:50:20.667682+0000 (oldest deadline 2024-05-09T08:51:25.367831+0000) 2024-05-09T08:51:33.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:32 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:32.992+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:33.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:32 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:32.992+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:33.344 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:32 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:32.992+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.157:6804 osd.2 since back 2024-05-09T08:50:27.460318+0000 front 2024-05-09T08:50:27.460393+0000 (oldest deadline 2024-05-09T08:51:30.360308+0000) 2024-05-09T08:51:33.620 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:33 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:33.232+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:33.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:33 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:33.232+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:20.667624+0000 front 2024-05-09T08:50:20.667682+0000 (oldest deadline 2024-05-09T08:51:25.367831+0000) 2024-05-09T08:51:34.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:33 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:33.997+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:34.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:33 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:33.997+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:34.344 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:33 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:33.997+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.157:6804 osd.2 since back 2024-05-09T08:50:27.460318+0000 front 2024-05-09T08:50:27.460393+0000 (oldest deadline 2024-05-09T08:51:30.360308+0000) 2024-05-09T08:51:34.620 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:34 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:34.268+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:34.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:34 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:34.268+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:20.667624+0000 front 2024-05-09T08:50:20.667682+0000 (oldest deadline 2024-05-09T08:51:25.367831+0000) 2024-05-09T08:51:35.308 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:34 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:34.984+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:35.308 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:34 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:34.984+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:35.308 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:34 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:34.984+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.157:6804 osd.2 since back 2024-05-09T08:50:27.460318+0000 front 2024-05-09T08:50:27.460393+0000 (oldest deadline 2024-05-09T08:51:30.360308+0000) 2024-05-09T08:51:35.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:35 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:35.257+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.665304+0000 front 2024-05-09T08:50:09.665343+0000 (oldest deadline 2024-05-09T08:51:17.866130+0000) 2024-05-09T08:51:35.621 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:35 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10[58595]: 2024-05-09T08:51:35.257+0000 7f75b62b3700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:20.667624+0000 front 2024-05-09T08:50:20.667682+0000 (oldest deadline 2024-05-09T08:51:25.367831+0000) 2024-05-09T08:51:36.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:35 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:35.963+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:36.344 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:35 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:35.963+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:36.344 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:35 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:35.963+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.157:6804 osd.2 since back 2024-05-09T08:50:27.460318+0000 front 2024-05-09T08:50:27.460393+0000 (oldest deadline 2024-05-09T08:51:30.360308+0000) 2024-05-09T08:51:36.567 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:36 smithi110 podman[70730]: 2024-05-09 08:51:36.317884834 +0000 UTC m=+5.869874136 container died 8f3dfac6caedeed36f47520579892b4a14470c28cac0565f102f487b860c7db2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, ceph=True, org.label-schema.license=GPLv2) 2024-05-09T08:51:36.842 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:36 smithi110 podman[70730]: 2024-05-09 08:51:36.642988382 +0000 UTC m=+6.194977727 container cleanup 8f3dfac6caedeed36f47520579892b4a14470c28cac0565f102f487b860c7db2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-09T08:51:36.843 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:36 smithi110 bash[70730]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10 2024-05-09T08:51:37.120 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:36 smithi110 podman[70746]: 2024-05-09 08:51:36.844071478 +0000 UTC m=+0.523113394 container remove 8f3dfac6caedeed36f47520579892b4a14470c28cac0565f102f487b860c7db2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-09T08:51:37.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:36 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:36.929+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:37.344 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:36 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:36.929+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:37.344 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:36 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:36.929+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.157:6804 osd.2 since back 2024-05-09T08:50:27.460318+0000 front 2024-05-09T08:50:27.460393+0000 (oldest deadline 2024-05-09T08:51:30.360308+0000) 2024-05-09T08:51:37.513 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:37 smithi110 podman[70842]: 2024-05-09 08:51:37.253601959 +0000 UTC m=+0.109435154 container create ae0f5b5d3640cd2dd223993825a18573c3ae8d0620409eb12f4feb77b91a96d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.license=GPLv2) 2024-05-09T08:51:37.513 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:37 smithi110 podman[70842]: 2024-05-09 08:51:37.162178847 +0000 UTC m=+0.018012042 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:51:37.513 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:37 smithi110 podman[70842]: 2024-05-09 08:51:37.439331745 +0000 UTC m=+0.295164941 container init ae0f5b5d3640cd2dd223993825a18573c3ae8d0620409eb12f4feb77b91a96d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10-deactivate, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240507, io.buildah.version=1.33.5) 2024-05-09T08:51:37.513 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:37 smithi110 podman[70842]: 2024-05-09 08:51:37.443029071 +0000 UTC m=+0.298862275 container start ae0f5b5d3640cd2dd223993825a18573c3ae8d0620409eb12f4feb77b91a96d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:51:37.513 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:37 smithi110 podman[70842]: 2024-05-09 08:51:37.514200378 +0000 UTC m=+0.370033582 container attach ae0f5b5d3640cd2dd223993825a18573c3ae8d0620409eb12f4feb77b91a96d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-09T08:51:37.871 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:37 smithi110 podman[70842]: 2024-05-09 08:51:37.734403748 +0000 UTC m=+0.590236944 container died ae0f5b5d3640cd2dd223993825a18573c3ae8d0620409eb12f4feb77b91a96d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:51:38.313 INFO:journalctl@ceph.osd.10.smithi110.stdout:May 09 08:51:38 smithi110 podman[70868]: 2024-05-09 08:51:38.292736343 +0000 UTC m=+0.553369916 container remove ae0f5b5d3640cd2dd223993825a18573c3ae8d0620409eb12f4feb77b91a96d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507) 2024-05-09T08:51:38.318 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.10.service' 2024-05-09T08:51:38.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:37 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:37.894+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:38.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:37 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:37.894+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:38.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:37 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:37.894+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.157:6804 osd.2 since back 2024-05-09T08:50:27.460318+0000 front 2024-05-09T08:50:27.460393+0000 (oldest deadline 2024-05-09T08:51:30.360308+0000) 2024-05-09T08:51:38.356 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:51:38.356 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-09T08:51:38.356 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-09T08:51:38.356 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.11 2024-05-09T08:51:38.674 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:38 smithi157 systemd[1]: Stopping Ceph osd.11 for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:51:39.093 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:38 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:38.675+0000 7fa845b97700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:51:39.094 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:38 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:38.675+0000 7fa845b97700 -1 osd.11 120 *** Got signal Terminated *** 2024-05-09T08:51:39.094 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:38 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:38.675+0000 7fa845b97700 -1 osd.11 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T08:51:39.094 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:38 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:38.858+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:39.094 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:38 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:38.858+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:39.094 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:38 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:38.858+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.157:6804 osd.2 since back 2024-05-09T08:50:27.460318+0000 front 2024-05-09T08:50:27.460393+0000 (oldest deadline 2024-05-09T08:51:30.360308+0000) 2024-05-09T08:51:40.093 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:39 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:39.832+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:40.094 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:39 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:39.832+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:40.094 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:39 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:39.832+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.157:6804 osd.2 since back 2024-05-09T08:50:27.460318+0000 front 2024-05-09T08:50:27.460393+0000 (oldest deadline 2024-05-09T08:51:30.360308+0000) 2024-05-09T08:51:41.093 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:40 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:40.794+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:41.094 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:40 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:40.794+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:41.094 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:40 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:40.794+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.157:6804 osd.2 since back 2024-05-09T08:50:27.460318+0000 front 2024-05-09T08:50:27.460393+0000 (oldest deadline 2024-05-09T08:51:30.360308+0000) 2024-05-09T08:51:42.093 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:41 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:41.801+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:42.094 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:41 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:41.801+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:42.094 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:41 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:41.801+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.157:6804 osd.2 since back 2024-05-09T08:50:27.460318+0000 front 2024-05-09T08:50:27.460393+0000 (oldest deadline 2024-05-09T08:51:30.360308+0000) 2024-05-09T08:51:43.093 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:42 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:42.785+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-05-09T08:50:09.456783+0000 front 2024-05-09T08:50:09.456744+0000 (oldest deadline 2024-05-09T08:51:14.057358+0000) 2024-05-09T08:51:43.094 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:42 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:42.785+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-05-09T08:50:19.358961+0000 front 2024-05-09T08:50:19.358721+0000 (oldest deadline 2024-05-09T08:51:22.759252+0000) 2024-05-09T08:51:43.094 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:42 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11[58937]: 2024-05-09T08:51:42.785+0000 7fa84218b700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.157:6804 osd.2 since back 2024-05-09T08:50:27.460318+0000 front 2024-05-09T08:50:27.460393+0000 (oldest deadline 2024-05-09T08:51:30.360308+0000) 2024-05-09T08:51:44.682 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:44 smithi157 podman[72128]: 2024-05-09 08:51:44.42943819 +0000 UTC m=+5.875346201 container died afafdccd53bb964161e5841c0c0b1b794b5d08dd4a5d8eec1c4109e05e255977 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-96805dc, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:51:45.065 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:44 smithi157 podman[72128]: 2024-05-09 08:51:44.790742508 +0000 UTC m=+6.236650519 container cleanup afafdccd53bb964161e5841c0c0b1b794b5d08dd4a5d8eec1c4109e05e255977 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:51:45.066 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:44 smithi157 bash[72128]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11 2024-05-09T08:51:45.343 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:45 smithi157 podman[72145]: 2024-05-09 08:51:45.066805939 +0000 UTC m=+0.640926203 container remove afafdccd53bb964161e5841c0c0b1b794b5d08dd4a5d8eec1c4109e05e255977 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-09T08:51:45.643 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:45 smithi157 podman[72236]: 2024-05-09 08:51:45.268909707 +0000 UTC m=+0.017960341 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T08:51:45.644 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:45 smithi157 podman[72236]: 2024-05-09 08:51:45.392608216 +0000 UTC m=+0.141658889 container create eaa1250d2fb64d3c35dda18831876e599cc07f295f275cd335a507abe7be1e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-96805dc, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507) 2024-05-09T08:51:45.644 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:45 smithi157 podman[72236]: 2024-05-09 08:51:45.570470292 +0000 UTC m=+0.319520929 container init eaa1250d2fb64d3c35dda18831876e599cc07f295f275cd335a507abe7be1e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-09T08:51:45.644 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:45 smithi157 podman[72236]: 2024-05-09 08:51:45.574061849 +0000 UTC m=+0.323112473 container start eaa1250d2fb64d3c35dda18831876e599cc07f295f275cd335a507abe7be1e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11-deactivate, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True) 2024-05-09T08:51:45.644 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:45 smithi157 podman[72236]: 2024-05-09 08:51:45.644607944 +0000 UTC m=+0.393658595 container attach eaa1250d2fb64d3c35dda18831876e599cc07f295f275cd335a507abe7be1e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11-deactivate, ceph=True, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:51:46.093 INFO:journalctl@ceph.osd.11.smithi157.stdout:May 09 08:51:45 smithi157 podman[72236]: 2024-05-09 08:51:45.87924063 +0000 UTC m=+0.628291258 container died eaa1250d2fb64d3c35dda18831876e599cc07f295f275cd335a507abe7be1e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-09T08:51:46.464 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@osd.11.service' 2024-05-09T08:51:46.502 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:51:46.502 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-09T08:51:46.502 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-05-09T08:51:46.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.a 2024-05-09T08:51:46.838 INFO:journalctl@ceph.mds.a.smithi107.stdout:May 09 08:51:46 smithi107 systemd[1]: Stopping Ceph mds.a for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:51:47.279 INFO:journalctl@ceph.mds.a.smithi107.stdout:May 09 08:51:46 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-a[69428]: 2024-05-09T08:51:46.838+0000 7f0e7d02b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:51:47.279 INFO:journalctl@ceph.mds.a.smithi107.stdout:May 09 08:51:46 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-a[69428]: 2024-05-09T08:51:46.838+0000 7f0e7d02b700 -1 mds.a *** got signal Terminated *** 2024-05-09T08:51:51.171 INFO:journalctl@ceph.mds.a.smithi107.stdout:May 09 08:51:50 smithi107 podman[116464]: 2024-05-09 08:51:50.883380229 +0000 UTC m=+4.177821837 container died de4ae376929bbf0b76f1a05a35d3935a0f3b362c1ab78bf64353332a50c1415b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-a, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-05-09T08:51:51.529 INFO:journalctl@ceph.mds.a.smithi107.stdout:May 09 08:51:51 smithi107 podman[116464]: 2024-05-09 08:51:51.27197056 +0000 UTC m=+4.566412159 container cleanup de4ae376929bbf0b76f1a05a35d3935a0f3b362c1ab78bf64353332a50c1415b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-09T08:51:51.529 INFO:journalctl@ceph.mds.a.smithi107.stdout:May 09 08:51:51 smithi107 bash[116464]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-a 2024-05-09T08:51:51.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.a.service' 2024-05-09T08:51:51.733 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:51:51.734 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-05-09T08:51:51.734 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-05-09T08:51:51.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.d 2024-05-09T08:51:52.049 INFO:journalctl@ceph.mds.d.smithi107.stdout:May 09 08:51:51 smithi107 systemd[1]: Stopping Ceph mds.d for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:51:52.529 INFO:journalctl@ceph.mds.d.smithi107.stdout:May 09 08:51:52 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-d[68473]: 2024-05-09T08:51:52.050+0000 7f8120a93700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:51:52.529 INFO:journalctl@ceph.mds.d.smithi107.stdout:May 09 08:51:52 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-d[68473]: 2024-05-09T08:51:52.050+0000 7f8120a93700 -1 mds.d *** got signal Terminated *** 2024-05-09T08:51:56.105 INFO:journalctl@ceph.mds.d.smithi107.stdout:May 09 08:51:55 smithi107 podman[116588]: 2024-05-09 08:51:55.852491888 +0000 UTC m=+3.902472931 container died 8dc3fa32b084a7c674d0136484916ea60cb3223abd39f9325eb947cfe0fa2422 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-d, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507) 2024-05-09T08:51:56.529 INFO:journalctl@ceph.mds.d.smithi107.stdout:May 09 08:51:56 smithi107 podman[116588]: 2024-05-09 08:51:56.183111802 +0000 UTC m=+4.233092844 container cleanup 8dc3fa32b084a7c674d0136484916ea60cb3223abd39f9325eb947cfe0fa2422 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-09T08:51:56.529 INFO:journalctl@ceph.mds.d.smithi107.stdout:May 09 08:51:56 smithi107 bash[116588]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-d 2024-05-09T08:51:56.529 INFO:journalctl@ceph.mds.d.smithi107.stdout:May 09 08:51:56 smithi107 podman[116602]: 2024-05-09 08:51:56.350948933 +0000 UTC m=+0.495429823 container remove 8dc3fa32b084a7c674d0136484916ea60cb3223abd39f9325eb947cfe0fa2422 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:51:56.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.d.service' 2024-05-09T08:51:56.591 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:51:56.591 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-05-09T08:51:56.591 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-05-09T08:51:56.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.g 2024-05-09T08:51:56.919 INFO:journalctl@ceph.mds.g.smithi107.stdout:May 09 08:51:56 smithi107 systemd[1]: Stopping Ceph mds.g for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:51:57.279 INFO:journalctl@ceph.mds.g.smithi107.stdout:May 09 08:51:56 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-g[69080]: 2024-05-09T08:51:56.919+0000 7fe811967700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:51:57.279 INFO:journalctl@ceph.mds.g.smithi107.stdout:May 09 08:51:56 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-g[69080]: 2024-05-09T08:51:56.919+0000 7fe811967700 -1 mds.g *** got signal Terminated *** 2024-05-09T08:52:01.211 INFO:journalctl@ceph.mds.g.smithi107.stdout:May 09 08:52:00 smithi107 podman[116709]: 2024-05-09 08:52:00.926630525 +0000 UTC m=+4.117013202 container died a3f4774b0a1b45071a05362363240508b0d148b2ac6517ebf923418e76a1ca39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-09T08:52:01.529 INFO:journalctl@ceph.mds.g.smithi107.stdout:May 09 08:52:01 smithi107 podman[116709]: 2024-05-09 08:52:01.311347584 +0000 UTC m=+4.501730250 container cleanup a3f4774b0a1b45071a05362363240508b0d148b2ac6517ebf923418e76a1ca39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-g, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, GIT_BRANCH=HEAD) 2024-05-09T08:52:01.529 INFO:journalctl@ceph.mds.g.smithi107.stdout:May 09 08:52:01 smithi107 bash[116709]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-g 2024-05-09T08:52:01.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.g.service' 2024-05-09T08:52:01.758 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:52:01.759 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-05-09T08:52:01.759 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-05-09T08:52:01.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.j 2024-05-09T08:52:02.072 INFO:journalctl@ceph.mds.j.smithi107.stdout:May 09 08:52:01 smithi107 systemd[1]: Stopping Ceph mds.j for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:52:02.529 INFO:journalctl@ceph.mds.j.smithi107.stdout:May 09 08:52:02 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-j[68123]: 2024-05-09T08:52:02.072+0000 7feee22ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:52:02.529 INFO:journalctl@ceph.mds.j.smithi107.stdout:May 09 08:52:02 smithi107 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-j[68123]: 2024-05-09T08:52:02.072+0000 7feee22ee700 -1 mds.j *** got signal Terminated *** 2024-05-09T08:52:05.229 INFO:journalctl@ceph.mds.j.smithi107.stdout:May 09 08:52:04 smithi107 podman[116828]: 2024-05-09 08:52:04.925325136 +0000 UTC m=+2.952354470 container died 908a53818100522105bc2473dbec051e3bd15b13bf736889db8cd3680c99db13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5) 2024-05-09T08:52:05.514 INFO:journalctl@ceph.mds.j.smithi107.stdout:May 09 08:52:05 smithi107 podman[116828]: 2024-05-09 08:52:05.331187776 +0000 UTC m=+3.358217109 container cleanup 908a53818100522105bc2473dbec051e3bd15b13bf736889db8cd3680c99db13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux ) 2024-05-09T08:52:05.514 INFO:journalctl@ceph.mds.j.smithi107.stdout:May 09 08:52:05 smithi107 bash[116828]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-j 2024-05-09T08:52:05.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.j.service' 2024-05-09T08:52:05.674 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:52:05.674 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-05-09T08:52:05.674 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-05-09T08:52:05.674 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.b 2024-05-09T08:52:06.121 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:52:05 smithi110 systemd[1]: Stopping Ceph mds.b for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:52:06.121 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:52:05 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-b[63783]: 2024-05-09T08:52:05.951+0000 7f9dd07b4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:52:06.121 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:52:05 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-b[63783]: 2024-05-09T08:52:05.951+0000 7f9dd07b4700 -1 mds.b *** got signal Terminated *** 2024-05-09T08:52:09.976 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:52:09 smithi110 podman[70946]: 2024-05-09 08:52:09.7006073 +0000 UTC m=+3.829859656 container died 100c0de814566a37ab7fa9734ef7b111ef30fecf568430216c225f2189cbc0dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-09T08:52:10.327 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:52:10 smithi110 podman[70946]: 2024-05-09 08:52:10.068748956 +0000 UTC m=+4.198001312 container cleanup 100c0de814566a37ab7fa9734ef7b111ef30fecf568430216c225f2189cbc0dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-05-09T08:52:10.327 INFO:journalctl@ceph.mds.b.smithi110.stdout:May 09 08:52:10 smithi110 bash[70946]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-b 2024-05-09T08:52:10.434 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.b.service' 2024-05-09T08:52:10.471 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:52:10.472 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-05-09T08:52:10.472 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-05-09T08:52:10.472 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.e 2024-05-09T08:52:10.620 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:52:10 smithi110 systemd[1]: Stopping Ceph mds.e for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:52:11.120 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:52:10 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-e[63163]: 2024-05-09T08:52:10.813+0000 7ff739205700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:52:11.121 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:52:10 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-e[63163]: 2024-05-09T08:52:10.813+0000 7ff739205700 -1 mds.e *** got signal Terminated *** 2024-05-09T08:52:13.997 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:52:13 smithi110 podman[71079]: 2024-05-09 08:52:13.702031023 +0000 UTC m=+3.008689060 container stop 388cf819a5548bc3cfdb6162ff11453b5e2a9a1efddd7344dd3988aa86c3897a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-e, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-09T08:52:13.997 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:52:13 smithi110 podman[71079]: 2024-05-09 08:52:13.727650432 +0000 UTC m=+3.034308471 container died 388cf819a5548bc3cfdb6162ff11453b5e2a9a1efddd7344dd3988aa86c3897a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-e, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:52:14.247 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:52:14 smithi110 podman[71079]: 2024-05-09 08:52:14.072580025 +0000 UTC m=+3.379238062 container cleanup 388cf819a5548bc3cfdb6162ff11453b5e2a9a1efddd7344dd3988aa86c3897a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-e, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-09T08:52:14.247 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:52:14 smithi110 bash[71079]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-e 2024-05-09T08:52:14.496 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.e.service' 2024-05-09T08:52:14.518 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:52:14 smithi110 podman[71093]: 2024-05-09 08:52:14.248851107 +0000 UTC m=+0.539746212 container remove 388cf819a5548bc3cfdb6162ff11453b5e2a9a1efddd7344dd3988aa86c3897a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.license=GPLv2) 2024-05-09T08:52:14.518 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:52:14 smithi110 systemd[1]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.e.service: Deactivated successfully. 2024-05-09T08:52:14.518 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:52:14 smithi110 systemd[1]: Stopped Ceph mds.e for 3954c88c-0ddd-11ef-bc98-c7b262605968. 2024-05-09T08:52:14.518 INFO:journalctl@ceph.mds.e.smithi110.stdout:May 09 08:52:14 smithi110 systemd[1]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.e.service: Consumed 2.623s CPU time. 2024-05-09T08:52:14.530 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:52:14.530 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-05-09T08:52:14.530 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-05-09T08:52:14.530 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.h 2024-05-09T08:52:14.871 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:52:14 smithi110 systemd[1]: Stopping Ceph mds.h for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:52:15.371 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:52:14 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-h[63480]: 2024-05-09T08:52:14.928+0000 7f0f8c04a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:52:15.371 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:52:14 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-h[63480]: 2024-05-09T08:52:14.928+0000 7f0f8c04a700 -1 mds.h *** got signal Terminated *** 2024-05-09T08:52:18.000 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:52:17 smithi110 podman[71199]: 2024-05-09 08:52:17.704214897 +0000 UTC m=+2.948097785 container died a262e31f23219fc51cba0d90f5df685df74b314db73ff1a24744a0959b11540f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-h, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-09T08:52:18.000 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:52:18 smithi110 podman[71199]: 2024-05-09 08:52:18.001392125 +0000 UTC m=+3.245275023 container cleanup a262e31f23219fc51cba0d90f5df685df74b314db73ff1a24744a0959b11540f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:52:18.370 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:52:18 smithi110 bash[71199]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-h 2024-05-09T08:52:18.371 INFO:journalctl@ceph.mds.h.smithi110.stdout:May 09 08:52:18 smithi110 podman[71214]: 2024-05-09 08:52:18.194566537 +0000 UTC m=+0.480782764 container remove a262e31f23219fc51cba0d90f5df685df74b314db73ff1a24744a0959b11540f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T08:52:18.418 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.h.service' 2024-05-09T08:52:18.450 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:52:18.450 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-05-09T08:52:18.450 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-05-09T08:52:18.450 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.k 2024-05-09T08:52:18.771 INFO:journalctl@ceph.mds.k.smithi110.stdout:May 09 08:52:18 smithi110 systemd[1]: Stopping Ceph mds.k for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:52:19.120 INFO:journalctl@ceph.mds.k.smithi110.stdout:May 09 08:52:18 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-k[62843]: 2024-05-09T08:52:18.771+0000 7f42d298c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:52:19.121 INFO:journalctl@ceph.mds.k.smithi110.stdout:May 09 08:52:18 smithi110 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-k[62843]: 2024-05-09T08:52:18.771+0000 7f42d298c700 -1 mds.k *** got signal Terminated *** 2024-05-09T08:52:22.371 INFO:journalctl@ceph.mds.k.smithi110.stdout:May 09 08:52:22 smithi110 podman[71322]: 2024-05-09 08:52:22.100662559 +0000 UTC m=+3.421461484 container died d83b79ffa2675cca1b6ec266bfb7e995760b3fb965a4900a2b5b935d7f7ddf88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-k, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-09T08:52:22.730 INFO:journalctl@ceph.mds.k.smithi110.stdout:May 09 08:52:22 smithi110 podman[71322]: 2024-05-09 08:52:22.455388512 +0000 UTC m=+3.776187489 container cleanup d83b79ffa2675cca1b6ec266bfb7e995760b3fb965a4900a2b5b935d7f7ddf88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-k, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T08:52:22.730 INFO:journalctl@ceph.mds.k.smithi110.stdout:May 09 08:52:22 smithi110 bash[71322]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-k 2024-05-09T08:52:22.875 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.k.service' 2024-05-09T08:52:22.907 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:52:22.907 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-05-09T08:52:22.907 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-05-09T08:52:22.907 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.c 2024-05-09T08:52:23.187 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:52:22 smithi157 systemd[1]: Stopping Ceph mds.c for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:52:23.445 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:52:23 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-c[64054]: 2024-05-09T08:52:23.188+0000 7f968719f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:52:23.445 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:52:23 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-c[64054]: 2024-05-09T08:52:23.188+0000 7f968719f700 -1 mds.c *** got signal Terminated *** 2024-05-09T08:52:26.088 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:52:25 smithi157 podman[72341]: 2024-05-09 08:52:25.694915617 +0000 UTC m=+2.595921362 container died 02e19a76ad1ccd4b735985d83fb6d375bcf51cb4254fcb50ad9ee47186401402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-c, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T08:52:26.343 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:52:26 smithi157 podman[72341]: 2024-05-09 08:52:26.22934508 +0000 UTC m=+3.130350868 container cleanup 02e19a76ad1ccd4b735985d83fb6d375bcf51cb4254fcb50ad9ee47186401402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-c, org.label-schema.build-date=20240507, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-09T08:52:26.344 INFO:journalctl@ceph.mds.c.smithi157.stdout:May 09 08:52:26 smithi157 bash[72341]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-c 2024-05-09T08:52:26.669 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.c.service' 2024-05-09T08:52:26.704 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:52:26.705 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-05-09T08:52:26.705 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-05-09T08:52:26.705 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.f 2024-05-09T08:52:27.032 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:52:26 smithi157 systemd[1]: Stopping Ceph mds.f for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:52:27.343 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:52:27 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[70169]: 2024-05-09T08:52:27.033+0000 7f28c9e20700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:52:27.343 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:52:27 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f[70169]: 2024-05-09T08:52:27.033+0000 7f28c9e20700 -1 mds.f *** got signal Terminated *** 2024-05-09T08:52:31.344 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:52:31 smithi157 podman[72473]: 2024-05-09 08:52:31.04578338 +0000 UTC m=+4.101650417 container died 58e5bc088347613f3902d75d4513c5831081760ad57ae0d15501d15e81f9f040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-09T08:52:31.690 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:52:31 smithi157 podman[72473]: 2024-05-09 08:52:31.44950141 +0000 UTC m=+4.505368446 container cleanup 58e5bc088347613f3902d75d4513c5831081760ad57ae0d15501d15e81f9f040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-09T08:52:31.691 INFO:journalctl@ceph.mds.f.smithi157.stdout:May 09 08:52:31 smithi157 bash[72473]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-f 2024-05-09T08:52:31.796 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.f.service' 2024-05-09T08:52:31.830 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:52:31.830 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-05-09T08:52:31.830 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-05-09T08:52:31.830 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.i 2024-05-09T08:52:32.093 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:52:31 smithi157 systemd[1]: Stopping Ceph mds.i for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:52:32.593 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:52:32 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-i[64360]: 2024-05-09T08:52:32.186+0000 7f64be179700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:52:32.594 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:52:32 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-i[64360]: 2024-05-09T08:52:32.186+0000 7f64be179700 -1 mds.i *** got signal Terminated *** 2024-05-09T08:52:33.977 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:52:33 smithi157 podman[72593]: 2024-05-09 08:52:33.702540628 +0000 UTC m=+1.650663705 container died 60aa0a0934a421f4e376727412f162b148d2c9645c1cb16ef92fea95bff8acbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-09T08:52:34.303 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:52:34 smithi157 podman[72593]: 2024-05-09 08:52:34.086555056 +0000 UTC m=+2.034678141 container cleanup 60aa0a0934a421f4e376727412f162b148d2c9645c1cb16ef92fea95bff8acbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-09T08:52:34.303 INFO:journalctl@ceph.mds.i.smithi157.stdout:May 09 08:52:34 smithi157 bash[72593]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-i 2024-05-09T08:52:34.426 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.i.service' 2024-05-09T08:52:34.460 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:52:34.460 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-05-09T08:52:34.460 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-05-09T08:52:34.460 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.l 2024-05-09T08:52:34.593 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:52:34 smithi157 systemd[1]: Stopping Ceph mds.l for 3954c88c-0ddd-11ef-bc98-c7b262605968... 2024-05-09T08:52:35.093 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:52:34 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-l[63415]: 2024-05-09T08:52:34.773+0000 7fea44811700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T08:52:35.093 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:52:34 smithi157 ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-l[63415]: 2024-05-09T08:52:34.773+0000 7fea44811700 -1 mds.l *** got signal Terminated *** 2024-05-09T08:52:37.980 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:52:37 smithi157 podman[72713]: 2024-05-09 08:52:37.703185809 +0000 UTC m=+3.026689780 container died c7aa358e7a80e8fd0b4dca39033facfa2c77813bffd1aacc9f54605f7eb8e587 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, ceph=True) 2024-05-09T08:52:38.246 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:52:38 smithi157 podman[72713]: 2024-05-09 08:52:38.106453928 +0000 UTC m=+3.429957940 container cleanup c7aa358e7a80e8fd0b4dca39033facfa2c77813bffd1aacc9f54605f7eb8e587 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-09T08:52:38.246 INFO:journalctl@ceph.mds.l.smithi157.stdout:May 09 08:52:38 smithi157 bash[72713]: ceph-3954c88c-0ddd-11ef-bc98-c7b262605968-mds-l 2024-05-09T08:52:38.444 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3954c88c-0ddd-11ef-bc98-c7b262605968@mds.l.service' 2024-05-09T08:52:38.476 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T08:52:38.476 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-05-09T08:52:38.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-09T08:52:38.667 INFO:teuthology.orchestra.run.smithi107.stdout:Deleting cluster with fsid: 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:52:41.244 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-09T08:52:41.436 INFO:teuthology.orchestra.run.smithi110.stdout:Deleting cluster with fsid: 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:52:43.833 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-09T08:52:44.028 INFO:teuthology.orchestra.run.smithi157.stdout:Deleting cluster with fsid: 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T08:52:46.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:52:46.505 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:52:46.538 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T08:52:46.566 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-09T08:52:46.568 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698666/remote/smithi107/crash 2024-05-09T08:52:46.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/crash -- . 2024-05-09T08:52:46.611 INFO:teuthology.orchestra.run.smithi107.stderr:tar: /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-09T08:52:46.612 INFO:teuthology.orchestra.run.smithi107.stderr:tar: Error is not recoverable: exiting now 2024-05-09T08:52:46.614 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698666/remote/smithi110/crash 2024-05-09T08:52:46.615 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/crash -- . 2024-05-09T08:52:46.675 INFO:teuthology.orchestra.run.smithi110.stderr:tar: /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-09T08:52:46.675 INFO:teuthology.orchestra.run.smithi110.stderr:tar: Error is not recoverable: exiting now 2024-05-09T08:52:46.677 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698666/remote/smithi157/crash 2024-05-09T08:52:46.678 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/crash -- . 2024-05-09T08:52:46.715 INFO:teuthology.orchestra.run.smithi157.stderr:tar: /var/lib/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-09T08:52:46.715 INFO:teuthology.orchestra.run.smithi157.stderr:tar: Error is not recoverable: exiting now 2024-05-09T08:52:46.716 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-09T08:52:46.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-09T08:52:46.760 INFO:tasks.cephadm:Compressing logs... 2024-05-09T08:52:46.760 DEBUG:teuthology.orchestra.run.smithi107:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T08:52:46.802 DEBUG:teuthology.orchestra.run.smithi110:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T08:52:46.805 DEBUG:teuthology.orchestra.run.smithi157:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T08:52:46.831 INFO:teuthology.orchestra.run.smithi157.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-09T08:52:46.831 INFO:teuthology.orchestra.run.smithi157.stderr:: No such file or directory 2024-05-09T08:52:46.832 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.log 2024-05-09T08:52:46.832 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.2.log 2024-05-09T08:52:46.832 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.f.log 2024-05-09T08:52:46.832 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.log: /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-09T08:52:46.832 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.5.log 2024-05-09T08:52:46.832 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.c.log 2024-05-09T08:52:46.834 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mon.c.log 2024-05-09T08:52:46.835 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mgr.z.log 2024-05-09T08:52:46.835 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.i.log 2024-05-09T08:52:46.835 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-09T08:52:46.835 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mgr.z.log: /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.11.log 2024-05-09T08:52:46.835 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.8.log 2024-05-09T08:52:46.835 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-09T08:52:46.835 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.l.log 2024-05-09T08:52:46.839 INFO:teuthology.orchestra.run.smithi110.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-09T08:52:46.845 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-09T08:52:46.845 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.log 2024-05-09T08:52:46.845 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-09T08:52:46.845 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.7.log 2024-05-09T08:52:46.845 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.b.log 2024-05-09T08:52:46.845 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.4.log 2024-05-09T08:52:46.845 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.1.log 2024-05-09T08:52:46.846 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mgr.y.log 2024-05-09T08:52:46.846 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.h.log 2024-05-09T08:52:46.846 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.10.log 2024-05-09T08:52:46.846 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-09T08:52:46.846 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.e.log 2024-05-09T08:52:46.846 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-09T08:52:46.846 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mon.b.log 2024-05-09T08:52:46.846 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.k.log 2024-05-09T08:52:46.854 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/var/log/rbd-target-api’gzip -5: No such file or directory --verbose 2024-05-09T08:52:46.854 INFO:teuthology.orchestra.run.smithi107.stderr: -- /var/log/ceph/ceph-client.0.log 2024-05-09T08:52:46.854 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-09T08:52:46.855 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.log 2024-05-09T08:52:46.855 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-09T08:52:46.857 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.g.log 2024-05-09T08:52:46.857 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.log: /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.audit.log: /var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.j.log 2024-05-09T08:52:46.857 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.3.log 2024-05-09T08:52:46.857 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.0.log 2024-05-09T08:52:46.857 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mgr.x.log 2024-05-09T08:52:46.857 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mon.a.log 2024-05-09T08:52:46.857 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.6.log 2024-05-09T08:52:46.859 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-09T08:52:46.859 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.a.log 2024-05-09T08:52:46.859 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-09T08:52:46.859 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.d.log 2024-05-09T08:52:46.859 INFO:teuthology.orchestra.run.smithi107.stderr: 84.2% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-09T08:52:46.859 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-volume.log: 87.6% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-09T08:52:46.860 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.9.log 2024-05-09T08:52:46.860 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-05-09T08:52:46.958 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.9.log: /var/log/ceph/ceph-client.admin.log: 90.1% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-09T08:52:46.977 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mon.b.log: /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.k.log: 83.9% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-09T08:52:46.996 INFO:teuthology.orchestra.run.smithi110.stderr: 94.6% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.k.log.gz 2024-05-09T08:52:47.004 INFO:teuthology.orchestra.run.smithi110.stderr: 92.2% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mgr.y.log.gz 2024-05-09T08:52:47.016 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-volume.log: /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.l.log: 92.3% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mgr.z.log.gz 2024-05-09T08:52:47.021 INFO:teuthology.orchestra.run.smithi157.stderr: 93.3% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.c.log.gz 2024-05-09T08:52:47.023 INFO:teuthology.orchestra.run.smithi110.stderr: 93.1% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.b.log.gz 2024-05-09T08:52:47.027 INFO:teuthology.orchestra.run.smithi110.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-09T08:52:47.036 INFO:teuthology.orchestra.run.smithi157.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-09T08:52:47.045 INFO:teuthology.orchestra.run.smithi110.stderr: 91.1% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-09T08:52:47.057 INFO:teuthology.orchestra.run.smithi157.stderr: 83.9% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-09T08:52:47.061 INFO:teuthology.orchestra.run.smithi110.stderr: 90.2% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-09T08:52:47.084 INFO:teuthology.orchestra.run.smithi107.stderr: 91.0% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-09T08:52:47.088 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-09T08:52:47.120 INFO:teuthology.orchestra.run.smithi107.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-05-09T08:52:47.120 INFO:teuthology.orchestra.run.smithi107.stderr: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-09T08:52:47.190 INFO:teuthology.orchestra.run.smithi157.stderr: 91.0% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-09T08:52:47.210 INFO:teuthology.orchestra.run.smithi157.stderr: 94.3% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.l.log.gz 2024-05-09T08:52:47.222 INFO:teuthology.orchestra.run.smithi157.stderr: 90.2% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-09T08:52:47.265 INFO:teuthology.orchestra.run.smithi157.stderr: 87.6% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-09T08:52:47.308 INFO:teuthology.orchestra.run.smithi107.stderr: 94.6% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.a.log.gz 2024-05-09T08:52:47.314 INFO:teuthology.orchestra.run.smithi107.stderr: 94.7% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.j.log.gz 2024-05-09T08:52:47.317 INFO:teuthology.orchestra.run.smithi107.stderr: 94.7% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.g.log.gz 2024-05-09T08:52:47.341 INFO:teuthology.orchestra.run.smithi107.stderr: 94.7% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.d.log.gz 2024-05-09T08:52:47.901 INFO:teuthology.orchestra.run.smithi110.stderr: 94.2% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.h.log.gz 2024-05-09T08:52:48.003 INFO:teuthology.orchestra.run.smithi110.stderr: 94.1% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.e.log.gz 2024-05-09T08:52:48.146 INFO:teuthology.orchestra.run.smithi107.stderr: 90.5% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mon.a.log.gz 2024-05-09T08:52:50.682 INFO:teuthology.orchestra.run.smithi157.stderr: 92.0% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mon.c.log.gz 2024-05-09T08:52:52.358 INFO:teuthology.orchestra.run.smithi107.stderr: 90.2% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mgr.x.log.gz 2024-05-09T08:52:53.141 INFO:teuthology.orchestra.run.smithi110.stderr: 91.9% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mon.b.log.gz 2024-05-09T08:53:00.255 INFO:teuthology.orchestra.run.smithi157.stderr: 93.6% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.f.log.gz 2024-05-09T08:53:25.605 INFO:teuthology.orchestra.run.smithi157.stderr: 93.5% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-mds.i.log.gz 2024-05-09T08:54:21.855 INFO:teuthology.orchestra.run.smithi110.stderr: 92.0% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.10.log.gz 2024-05-09T08:55:50.312 INFO:teuthology.orchestra.run.smithi157.stderr: 91.8% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.2.log.gz 2024-05-09T08:56:57.421 INFO:teuthology.orchestra.run.smithi110.stderr: 91.7% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.4.log.gz 2024-05-09T08:57:43.430 INFO:teuthology.orchestra.run.smithi157.stderr: 91.8% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.8.log.gz 2024-05-09T08:58:16.737 INFO:teuthology.orchestra.run.smithi157.stderr: 91.9% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.11.log.gz 2024-05-09T08:59:08.699 INFO:teuthology.orchestra.run.smithi110.stderr: 91.9% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.1.log.gz 2024-05-09T08:59:37.186 INFO:teuthology.orchestra.run.smithi110.stderr: 91.8% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.7.log.gz 2024-05-09T08:59:37.187 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-05-09T08:59:37.187 INFO:teuthology.orchestra.run.smithi110.stderr:real 6m50.368s 2024-05-09T08:59:37.188 INFO:teuthology.orchestra.run.smithi110.stderr:user 6m19.122s 2024-05-09T08:59:37.188 INFO:teuthology.orchestra.run.smithi110.stderr:sys 0m13.755s 2024-05-09T08:59:43.961 INFO:teuthology.orchestra.run.smithi157.stderr: 91.6% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.5.log.gz 2024-05-09T08:59:43.963 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-05-09T08:59:43.963 INFO:teuthology.orchestra.run.smithi157.stderr:real 6m57.143s 2024-05-09T08:59:43.964 INFO:teuthology.orchestra.run.smithi157.stderr:user 7m3.526s 2024-05-09T08:59:43.964 INFO:teuthology.orchestra.run.smithi157.stderr:sys 0m15.700s 2024-05-09T09:10:45.058 INFO:teuthology.orchestra.run.smithi107.stderr: 91.6% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.0.log.gz 2024-05-09T09:13:53.335 INFO:teuthology.orchestra.run.smithi107.stderr: 91.7% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.6.log.gz 2024-05-09T09:18:12.554 INFO:teuthology.orchestra.run.smithi107.stderr: 91.7% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.3.log.gz 2024-05-09T09:22:05.326 INFO:teuthology.orchestra.run.smithi107.stderr: 91.0% -- replaced with /var/log/ceph/3954c88c-0ddd-11ef-bc98-c7b262605968/ceph-osd.9.log.gz 2024-05-09T09:23:03.719 INFO:teuthology.orchestra.run.smithi107.stderr: 96.5% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-05-09T09:23:03.721 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-05-09T09:23:03.721 INFO:teuthology.orchestra.run.smithi107.stderr:real 30m16.904s 2024-05-09T09:23:03.721 INFO:teuthology.orchestra.run.smithi107.stderr:user 10m31.666s 2024-05-09T09:23:03.721 INFO:teuthology.orchestra.run.smithi107.stderr:sys 0m29.902s 2024-05-09T09:23:03.722 INFO:tasks.cephadm:Archiving logs... 2024-05-09T09:23:03.724 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698666/remote/smithi107/log 2024-05-09T09:23:03.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T09:24:40.187 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698666/remote/smithi110/log 2024-05-09T09:24:40.189 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T09:25:41.297 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698666/remote/smithi157/log 2024-05-09T09:25:41.299 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T09:26:52.986 INFO:tasks.cephadm:Removing cluster... 2024-05-09T09:26:53.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 --force 2024-05-09T09:26:53.462 INFO:teuthology.orchestra.run.smithi107.stdout:Deleting cluster with fsid: 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T09:26:54.248 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 --force 2024-05-09T09:26:54.454 INFO:teuthology.orchestra.run.smithi110.stdout:Deleting cluster with fsid: 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T09:26:55.138 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3954c88c-0ddd-11ef-bc98-c7b262605968 --force 2024-05-09T09:26:55.351 INFO:teuthology.orchestra.run.smithi157.stdout:Deleting cluster with fsid: 3954c88c-0ddd-11ef-bc98-c7b262605968 2024-05-09T09:26:56.111 INFO:tasks.cephadm:Removing cephadm ... 2024-05-09T09:26:56.112 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-09T09:26:56.131 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-09T09:26:56.150 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-09T09:26:56.173 INFO:tasks.cephadm:Teardown complete 2024-05-09T09:26:56.174 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-09T09:26:56.187 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-09T09:26:56.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T09:26:56.190 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T09:26:56.193 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T09:26:56.395 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-09T09:26:56.406 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-09T09:26:56.406 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T09:26:56.409 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T09:26:56.412 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T09:26:56.424 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-05-09T09:26:56.426 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-05-09T09:26:56.427 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-05-09T09:26:56.439 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T09:26:56.439 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-05-09T09:26:56.440 INFO:teuthology.orchestra.run.smithi110.stdout:^* hv01.front.sepia.ceph.com 3 6 377 2 +426us[ +413us] +/- 46ms 2024-05-09T09:26:56.440 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 3 -793us[ -806us] +/- 49ms 2024-05-09T09:26:56.440 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 +383us[ +383us] +/- 67ms 2024-05-09T09:26:56.440 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:26:56.447 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T09:26:56.447 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-05-09T09:26:56.447 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 4 +428us[ +420us] +/- 46ms 2024-05-09T09:26:56.447 INFO:teuthology.orchestra.run.smithi157.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -799us[ -807us] +/- 49ms 2024-05-09T09:26:56.447 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +386us[ +378us] +/- 67ms 2024-05-09T09:26:56.447 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:26:56.471 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T09:26:56.472 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-05-09T09:26:56.472 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 9 +422us[ +404us] +/- 46ms 2024-05-09T09:26:56.472 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -800us[ -818us] +/- 49ms 2024-05-09T09:26:56.472 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +374us[ +356us] +/- 67ms 2024-05-09T09:26:56.472 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:26:56.473 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-09T09:26:56.484 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-09T09:26:56.485 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-09T09:26:56.512 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T09:26:56.682 DEBUG:teuthology.orchestra.run.smithi110:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T09:26:56.817 DEBUG:teuthology.orchestra.run.smithi157:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T09:26:56.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T09:26:56.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:26:56.934 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T09:26:56.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:26:56.966 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T09:26:56.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:26:56.994 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-09T09:26:57.006 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-09T09:26:57.028 INFO:teuthology.task.internal:Duration was 4274.881922 seconds 2024-05-09T09:26:57.029 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-09T09:26:57.070 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-09T09:26:57.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T09:26:57.074 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T09:26:57.076 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T09:26:57.126 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T09:26:57.128 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T09:26:57.129 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T09:26:57.637 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-09T09:26:57.638 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2024-05-09T09:26:57.638 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T09:26:57.676 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2024-05-09T09:26:57.676 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T09:26:57.704 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2024-05-09T09:26:57.704 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T09:26:57.740 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-09T09:26:57.741 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T09:26:57.743 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T09:26:57.747 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T09:26:57.809 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-09T09:26:57.809 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T09:26:57.822 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T09:26:57.824 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T09:26:58.211 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-09T09:26:58.223 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-09T09:26:58.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T09:26:58.266 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T09:26:58.293 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T09:26:58.320 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-09T09:26:58.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:26:58.334 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:26:58.336 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:26:58.357 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2024-05-09T09:26:58.368 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2024-05-09T09:26:58.383 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2024-05-09T09:26:58.460 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:26:58.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:26:58.475 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:26:58.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:26:58.489 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T09:26:58.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:26:58.515 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-09T09:26:58.526 INFO:teuthology.task.internal:Transferring archived files... 2024-05-09T09:26:58.528 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698666/remote/smithi107 2024-05-09T09:26:58.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T09:26:58.572 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698666/remote/smithi110 2024-05-09T09:26:58.572 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T09:26:58.609 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698666/remote/smithi157 2024-05-09T09:26:58.610 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T09:26:58.647 INFO:teuthology.task.internal:Removing archive directory... 2024-05-09T09:26:58.647 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T09:26:58.650 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T09:26:58.652 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T09:26:58.695 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-09T09:26:58.709 INFO:teuthology.task.internal:Not uploading archives. 2024-05-09T09:26:58.709 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-09T09:26:58.737 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-09T09:26:58.738 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T09:26:58.740 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T09:26:58.742 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T09:26:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 09:26 /home/ubuntu/cephtest 2024-05-09T09:26:58.755 INFO:teuthology.orchestra.run.smithi110.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 09:26 /home/ubuntu/cephtest 2024-05-09T09:26:58.757 INFO:teuthology.orchestra.run.smithi157.stdout: 265295 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 09:26 /home/ubuntu/cephtest 2024-05-09T09:26:58.758 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-09T09:26:58.781 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-09T09:26:58.865 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} duration: 4274.88192152977 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-05-09T09:26:58.866 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T09:26:58.960 INFO:teuthology.run:pass